2024-09-21T12:03:34.409 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-21T12:03:34.427 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T12:03:34.517 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2024-09-21_10:07:29-orch:cephadm-wip-guits-main-2024-09-21-0813-distro-default-smithi/7913913 branch: wip-guits-main-2024-09-21-0813 description: orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} email: gabrioux@ibm.com first_in_suite: false job_id: '7913913' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: gabrioux-2024-09-21_10:07:29-orch:cephadm-wip-guits-main-2024-09-21-0813-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-guits-main-2024-09-21-0813 ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - CEPHADM_STRAY_DAEMON - CEPHADM_FAILED_DAEMON - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) log-only-match: - CEPHADM_ sha1: 5c1cdc3c470dae78e91a0ed2bcf13509a046d301 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 5c1cdc3c470dae78e91a0ed2bcf13509a046d301 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-guits-main-2024-09-21-0813 sha1: 5c1cdc3c470dae78e91a0ed2bcf13509a046d301 owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 3041 sha1: 5c1cdc3c470dae78e91a0ed2bcf13509a046d301 sleep_before_teardown: 0 subset: 1/8 suite: orch:cephadm suite_branch: wip-guits-main-2024-09-21-0813 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_5c1cdc3c470dae78e91a0ed2bcf13509a046d301/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 5c1cdc3c470dae78e91a0ed2bcf13509a046d301 targets: smithi114.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ2TsNYi4rltLPKzb3P2ATvN1E9JMUJTPGQWczSgNfuBtwGtRikXcRfA7GMOg1bG9R3lEzsi0Np/4TQCp6Q+RWw= smithi183.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIC9Ug/cOPNtriVO08LKXURppi9A2NpNV1GEg4Reb4SgR7FSHCpZyshxChUuscGG7GpOgJP32Y+sDRTdFR9AWzc= tasks: - pexec: all: - sudo dnf remove nvme-cli -y - sudo dnf install runc nvmetcli nvme-cli -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-21_10:07:29 tube: smithi user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-21T12:03:34.518 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_5c1cdc3c470dae78e91a0ed2bcf13509a046d301/qa; will attempt to use it 2024-09-21T12:03:34.518 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_5c1cdc3c470dae78e91a0ed2bcf13509a046d301/qa/tasks 2024-09-21T12:03:34.518 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-21T12:03:34.520 INFO:teuthology.task.internal:Checking packages... 2024-09-21T12:03:34.539 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5c1cdc3c470dae78e91a0ed2bcf13509a046d301' 2024-09-21T12:03:34.539 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T12:03:34.539 INFO:teuthology.packaging:ref: None 2024-09-21T12:03:34.539 INFO:teuthology.packaging:tag: None 2024-09-21T12:03:34.539 INFO:teuthology.packaging:branch: wip-guits-main-2024-09-21-0813 2024-09-21T12:03:34.539 INFO:teuthology.packaging:sha1: 5c1cdc3c470dae78e91a0ed2bcf13509a046d301 2024-09-21T12:03:34.539 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-guits-main-2024-09-21-0813 2024-09-21T12:03:34.676 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-5103.g5c1cdc3c 2024-09-21T12:03:34.677 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-21T12:03:34.690 INFO:teuthology.task.internal:no buildpackages task found 2024-09-21T12:03:34.690 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-21T12:03:34.726 INFO:teuthology.task.internal:Saving configuration 2024-09-21T12:03:34.738 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-21T12:03:34.743 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-21T12:03:34.768 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-09-21_10:07:29-orch:cephadm-wip-guits-main-2024-09-21-0813-distro-default-smithi/7913913', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 11:58:40.173273', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ2TsNYi4rltLPKzb3P2ATvN1E9JMUJTPGQWczSgNfuBtwGtRikXcRfA7GMOg1bG9R3lEzsi0Np/4TQCp6Q+RWw='} 2024-09-21T12:03:34.788 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-09-21_10:07:29-orch:cephadm-wip-guits-main-2024-09-21-0813-distro-default-smithi/7913913', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 11:58:40.171357', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIC9Ug/cOPNtriVO08LKXURppi9A2NpNV1GEg4Reb4SgR7FSHCpZyshxChUuscGG7GpOgJP32Y+sDRTdFR9AWzc='} 2024-09-21T12:03:34.788 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-21T12:03:34.794 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-09-21T12:03:34.795 INFO:teuthology.task.internal:roles: ubuntu@smithi183.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-09-21T12:03:34.795 INFO:teuthology.run_tasks:Running task console_log... 2024-09-21T12:03:34.902 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f405c064280>, signals=[15]) 2024-09-21T12:03:34.902 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-21T12:03:34.908 INFO:teuthology.task.internal:Opening connections... 2024-09-21T12:03:34.908 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi114.front.sepia.ceph.com 2024-09-21T12:03:34.909 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T12:03:34.985 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi183.front.sepia.ceph.com 2024-09-21T12:03:34.986 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T12:03:35.057 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-21T12:03:35.064 DEBUG:teuthology.orchestra.run.smithi114:> uname -m 2024-09-21T12:03:35.083 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64 2024-09-21T12:03:35.083 DEBUG:teuthology.orchestra.run.smithi114:> cat /etc/os-release 2024-09-21T12:03:35.141 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="CentOS Stream" 2024-09-21T12:03:35.142 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="9" 2024-09-21T12:03:35.142 INFO:teuthology.orchestra.run.smithi114.stdout:ID="centos" 2024-09-21T12:03:35.142 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE="rhel fedora" 2024-09-21T12:03:35.142 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="9" 2024-09-21T12:03:35.142 INFO:teuthology.orchestra.run.smithi114.stdout:PLATFORM_ID="platform:el9" 2024-09-21T12:03:35.142 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T12:03:35.142 INFO:teuthology.orchestra.run.smithi114.stdout:ANSI_COLOR="0;31" 2024-09-21T12:03:35.142 INFO:teuthology.orchestra.run.smithi114.stdout:LOGO="fedora-logo-icon" 2024-09-21T12:03:35.142 INFO:teuthology.orchestra.run.smithi114.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T12:03:35.142 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://centos.org/" 2024-09-21T12:03:35.142 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T12:03:35.142 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T12:03:35.142 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T12:03:35.143 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2024-09-21T12:03:35.170 DEBUG:teuthology.orchestra.run.smithi183:> uname -m 2024-09-21T12:03:35.188 INFO:teuthology.orchestra.run.smithi183.stdout:x86_64 2024-09-21T12:03:35.189 DEBUG:teuthology.orchestra.run.smithi183:> cat /etc/os-release 2024-09-21T12:03:35.246 INFO:teuthology.orchestra.run.smithi183.stdout:NAME="CentOS Stream" 2024-09-21T12:03:35.246 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION="9" 2024-09-21T12:03:35.246 INFO:teuthology.orchestra.run.smithi183.stdout:ID="centos" 2024-09-21T12:03:35.246 INFO:teuthology.orchestra.run.smithi183.stdout:ID_LIKE="rhel fedora" 2024-09-21T12:03:35.246 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_ID="9" 2024-09-21T12:03:35.247 INFO:teuthology.orchestra.run.smithi183.stdout:PLATFORM_ID="platform:el9" 2024-09-21T12:03:35.247 INFO:teuthology.orchestra.run.smithi183.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T12:03:35.247 INFO:teuthology.orchestra.run.smithi183.stdout:ANSI_COLOR="0;31" 2024-09-21T12:03:35.247 INFO:teuthology.orchestra.run.smithi183.stdout:LOGO="fedora-logo-icon" 2024-09-21T12:03:35.247 INFO:teuthology.orchestra.run.smithi183.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T12:03:35.247 INFO:teuthology.orchestra.run.smithi183.stdout:HOME_URL="https://centos.org/" 2024-09-21T12:03:35.247 INFO:teuthology.orchestra.run.smithi183.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T12:03:35.247 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T12:03:35.247 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T12:03:35.247 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2024-09-21T12:03:35.270 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-21T12:03:35.279 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-21T12:03:35.296 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-21T12:03:35.296 DEBUG:teuthology.orchestra.run.smithi114:> test '!' -e /home/ubuntu/cephtest 2024-09-21T12:03:35.298 DEBUG:teuthology.orchestra.run.smithi183:> test '!' -e /home/ubuntu/cephtest 2024-09-21T12:03:35.315 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-21T12:03:35.321 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-21T12:03:35.321 DEBUG:teuthology.orchestra.run.smithi114:> test -z $(ls -A /var/lib/ceph) 2024-09-21T12:03:35.356 DEBUG:teuthology.orchestra.run.smithi183:> test -z $(ls -A /var/lib/ceph) 2024-09-21T12:03:35.381 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-21T12:03:35.434 INFO:teuthology.run_tasks:Running task kernel... 2024-09-21T12:03:35.531 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-21T12:03:35.531 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-21T12:03:35.531 DEBUG:teuthology.orchestra.run.smithi114:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T12:03:35.531 DEBUG:teuthology.orchestra.run.smithi183:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T12:03:35.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:03:35.549 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2024-09-21T12:03:35.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:03:35.550 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-09-21T12:03:35.606 INFO:teuthology.orchestra.run.smithi183.stdout:5.14.0-503.el9.x86_64 2024-09-21T12:03:35.606 INFO:teuthology.task.kernel:Running kernel on smithi183: 5.14.0-503.el9.x86_64 2024-09-21T12:03:35.606 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum install -y kernel 2024-09-21T12:03:35.607 INFO:teuthology.orchestra.run.smithi114.stdout:5.14.0-503.el9.x86_64 2024-09-21T12:03:35.607 INFO:teuthology.task.kernel:Running kernel on smithi114: 5.14.0-503.el9.x86_64 2024-09-21T12:03:35.608 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum install -y kernel 2024-09-21T12:03:38.046 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.3 MB 00:00 2024-09-21T12:03:38.998 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.3 MB 00:01 2024-09-21T12:03:40.732 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 30 MB/s | 20 MB 00:00 2024-09-21T12:03:44.958 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-21T12:03:46.971 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - AppStream 2.6 MB/s | 20 MB 00:07 2024-09-21T12:03:47.284 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 17 kB/s | 19 kB 00:01 2024-09-21T12:03:48.355 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-21T12:03:51.313 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-21T12:03:52.730 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - Extras packages 90 kB/s | 19 kB 00:00 2024-09-21T12:03:53.526 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 34 MB/s | 23 MB 00:00 2024-09-21T12:03:53.777 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-21T12:03:55.633 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T12:03:55.634 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T12:03:55.721 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-21T12:03:55.726 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:03:55.726 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-09-21T12:03:55.726 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:03:55.727 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-21T12:03:55.727 INFO:teuthology.orchestra.run.smithi114.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T12:03:55.727 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-21T12:03:55.727 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T12:03:55.727 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T12:03:55.727 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T12:03:55.727 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:03:55.727 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-21T12:03:55.727 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:03:55.727 INFO:teuthology.orchestra.run.smithi114.stdout:Install 4 Packages 2024-09-21T12:03:55.727 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:03:55.728 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 79 M 2024-09-21T12:03:55.728 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 126 M 2024-09-21T12:03:55.728 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-21T12:03:55.929 INFO:teuthology.orchestra.run.smithi114.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 374 kB/s | 36 kB 00:00 2024-09-21T12:03:56.212 INFO:teuthology.orchestra.run.smithi114.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-09-21T12:03:56.529 INFO:teuthology.orchestra.run.smithi114.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 48 MB/s | 29 MB 00:00 2024-09-21T12:03:57.279 INFO:teuthology.orchestra.run.smithi114.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 24 MB/s | 35 MB 00:01 2024-09-21T12:03:57.280 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:03:57.280 INFO:teuthology.orchestra.run.smithi114.stdout:Total 51 MB/s | 79 MB 00:01 2024-09-21T12:03:57.965 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-21T12:03:58.324 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-21T12:03:58.324 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-21T12:03:58.907 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-21T12:03:59.274 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-21T12:03:59.274 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-21T12:04:00.813 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T12:04:00.813 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T12:04:00.896 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-21T12:04:00.908 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:04:00.908 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-09-21T12:04:00.908 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:04:00.909 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-21T12:04:00.909 INFO:teuthology.orchestra.run.smithi183.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T12:04:00.909 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-09-21T12:04:00.909 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T12:04:00.909 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T12:04:00.909 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T12:04:00.909 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:04:00.909 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-21T12:04:00.909 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:04:00.909 INFO:teuthology.orchestra.run.smithi183.stdout:Install 4 Packages 2024-09-21T12:04:00.909 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:04:00.910 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 79 M 2024-09-21T12:04:00.910 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 126 M 2024-09-21T12:04:00.910 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-21T12:04:01.283 INFO:teuthology.orchestra.run.smithi183.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 313 kB/s | 36 kB 00:00 2024-09-21T12:04:01.666 INFO:teuthology.orchestra.run.smithi183.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-09-21T12:04:01.685 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-21T12:04:02.058 INFO:teuthology.orchestra.run.smithi183.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 37 MB/s | 29 MB 00:00 2024-09-21T12:04:02.340 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T12:04:02.359 INFO:teuthology.orchestra.run.smithi183.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 29 MB/s | 35 MB 00:01 2024-09-21T12:04:02.359 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:04:02.359 INFO:teuthology.orchestra.run.smithi183.stdout:Total 54 MB/s | 79 MB 00:01 2024-09-21T12:04:02.421 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T12:04:03.045 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-21T12:04:03.391 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-21T12:04:03.391 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-21T12:04:04.376 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-21T12:04:04.376 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-21T12:04:05.169 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T12:04:05.298 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T12:04:06.828 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-21T12:04:07.528 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T12:04:07.610 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T12:04:10.106 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T12:04:10.223 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T12:04:10.359 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T12:04:10.510 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T12:04:15.053 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T12:04:15.355 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T12:04:15.445 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T12:04:20.281 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T12:04:53.477 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T12:04:53.516 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T12:04:53.853 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T12:04:53.853 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T12:04:53.853 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T12:04:53.853 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T12:04:54.772 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T12:04:54.772 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:04:54.772 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-21T12:04:54.772 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T12:04:54.772 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T12:04:54.772 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T12:04:54.772 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T12:04:54.772 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:04:54.772 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-21T12:04:55.126 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T12:04:55.264 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T12:04:55.264 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T12:04:55.264 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T12:04:55.264 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T12:04:55.264 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T12:04:55.265 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-21T12:04:55.265 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi114.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T12:04:55.265 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T12:04:55.323 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T12:04:55.324 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T12:04:55.324 DEBUG:teuthology.orchestra.run.smithi114:> sudo rpm -qi grub2-tools 2024-09-21T12:04:55.365 INFO:teuthology.orchestra.run.smithi114.stdout:Name : grub2-tools 2024-09-21T12:04:55.365 INFO:teuthology.orchestra.run.smithi114.stdout:Epoch : 1 2024-09-21T12:04:55.365 INFO:teuthology.orchestra.run.smithi114.stdout:Version : 2.06 2024-09-21T12:04:55.365 INFO:teuthology.orchestra.run.smithi114.stdout:Release : 82.el9 2024-09-21T12:04:55.365 INFO:teuthology.orchestra.run.smithi114.stdout:Architecture: x86_64 2024-09-21T12:04:55.366 INFO:teuthology.orchestra.run.smithi114.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T12:04:55.366 INFO:teuthology.orchestra.run.smithi114.stdout:Group : Unspecified 2024-09-21T12:04:55.366 INFO:teuthology.orchestra.run.smithi114.stdout:Size : 8274670 2024-09-21T12:04:55.366 INFO:teuthology.orchestra.run.smithi114.stdout:License : GPLv3+ 2024-09-21T12:04:55.366 INFO:teuthology.orchestra.run.smithi114.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T12:04:55.366 INFO:teuthology.orchestra.run.smithi114.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T12:04:55.366 INFO:teuthology.orchestra.run.smithi114.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T12:04:55.366 INFO:teuthology.orchestra.run.smithi114.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T12:04:55.366 INFO:teuthology.orchestra.run.smithi114.stdout:Packager : builder@centos.org 2024-09-21T12:04:55.366 INFO:teuthology.orchestra.run.smithi114.stdout:Vendor : CentOS 2024-09-21T12:04:55.366 INFO:teuthology.orchestra.run.smithi114.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T12:04:55.366 INFO:teuthology.orchestra.run.smithi114.stdout:Summary : Support tools for GRUB. 2024-09-21T12:04:55.367 INFO:teuthology.orchestra.run.smithi114.stdout:Description : 2024-09-21T12:04:55.367 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:04:55.367 INFO:teuthology.orchestra.run.smithi114.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T12:04:55.367 INFO:teuthology.orchestra.run.smithi114.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T12:04:55.367 INFO:teuthology.orchestra.run.smithi114.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T12:04:55.367 INFO:teuthology.orchestra.run.smithi114.stdout:hardware devices. 2024-09-21T12:04:55.367 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:04:55.367 INFO:teuthology.orchestra.run.smithi114.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T12:04:55.368 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T12:04:55.368 INFO:teuthology.task.kernel:Updating grub on smithi114 to boot 5.14.0-508.el9.x86_64 2024-09-21T12:04:55.368 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T12:04:56.054 INFO:teuthology.orchestra.run.smithi114.stderr:Generating grub configuration file ... 2024-09-21T12:04:57.293 INFO:teuthology.orchestra.run.smithi114.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T12:04:57.336 INFO:teuthology.orchestra.run.smithi114.stderr:done 2024-09-21T12:04:57.339 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-09-21T12:04:57.355 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.58ZmDaj6Cw 2024-09-21T12:04:57.355 DEBUG:teuthology.orchestra.run.smithi114:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.58ZmDaj6Cw 2024-09-21T12:04:57.422 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.58ZmDaj6Cw 2024-09-21T12:04:57.555 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.58ZmDaj6Cw is 6KB 2024-09-21T12:04:57.603 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.58ZmDaj6Cw 2024-09-21T12:04:57.618 DEBUG:teuthology.orchestra.run.smithi114:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T12:04:57.681 INFO:teuthology.orchestra.run.smithi114.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T12:04:57.681 INFO:teuthology.orchestra.run.smithi114.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T12:04:57.681 INFO:teuthology.orchestra.run.smithi114.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T12:04:57.681 INFO:teuthology.orchestra.run.smithi114.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T12:04:57.683 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T12:04:57.834 DEBUG:teuthology.orchestra.run.smithi114:> sudo shutdown -r now 2024-09-21T12:04:58.432 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T12:04:58.472 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T12:04:58.825 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T12:04:58.825 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T12:04:58.825 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T12:04:58.825 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T12:04:59.713 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T12:04:59.713 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:04:59.714 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-21T12:04:59.714 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T12:04:59.714 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T12:04:59.714 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T12:04:59.714 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T12:04:59.714 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:04:59.714 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-21T12:05:00.059 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T12:05:00.141 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T12:05:00.142 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T12:05:00.142 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T12:05:00.142 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T12:05:00.142 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T12:05:00.142 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-21T12:05:00.142 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi183.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T12:05:00.142 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T12:05:00.201 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T12:05:00.202 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T12:05:00.202 DEBUG:teuthology.orchestra.run.smithi183:> sudo rpm -qi grub2-tools 2024-09-21T12:05:00.233 INFO:teuthology.orchestra.run.smithi183.stdout:Name : grub2-tools 2024-09-21T12:05:00.234 INFO:teuthology.orchestra.run.smithi183.stdout:Epoch : 1 2024-09-21T12:05:00.234 INFO:teuthology.orchestra.run.smithi183.stdout:Version : 2.06 2024-09-21T12:05:00.234 INFO:teuthology.orchestra.run.smithi183.stdout:Release : 82.el9 2024-09-21T12:05:00.234 INFO:teuthology.orchestra.run.smithi183.stdout:Architecture: x86_64 2024-09-21T12:05:00.234 INFO:teuthology.orchestra.run.smithi183.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T12:05:00.234 INFO:teuthology.orchestra.run.smithi183.stdout:Group : Unspecified 2024-09-21T12:05:00.234 INFO:teuthology.orchestra.run.smithi183.stdout:Size : 8274670 2024-09-21T12:05:00.234 INFO:teuthology.orchestra.run.smithi183.stdout:License : GPLv3+ 2024-09-21T12:05:00.234 INFO:teuthology.orchestra.run.smithi183.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T12:05:00.234 INFO:teuthology.orchestra.run.smithi183.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T12:05:00.234 INFO:teuthology.orchestra.run.smithi183.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T12:05:00.234 INFO:teuthology.orchestra.run.smithi183.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T12:05:00.234 INFO:teuthology.orchestra.run.smithi183.stdout:Packager : builder@centos.org 2024-09-21T12:05:00.234 INFO:teuthology.orchestra.run.smithi183.stdout:Vendor : CentOS 2024-09-21T12:05:00.234 INFO:teuthology.orchestra.run.smithi183.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T12:05:00.235 INFO:teuthology.orchestra.run.smithi183.stdout:Summary : Support tools for GRUB. 2024-09-21T12:05:00.235 INFO:teuthology.orchestra.run.smithi183.stdout:Description : 2024-09-21T12:05:00.235 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:05:00.235 INFO:teuthology.orchestra.run.smithi183.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T12:05:00.235 INFO:teuthology.orchestra.run.smithi183.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T12:05:00.235 INFO:teuthology.orchestra.run.smithi183.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T12:05:00.235 INFO:teuthology.orchestra.run.smithi183.stdout:hardware devices. 2024-09-21T12:05:00.235 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:05:00.235 INFO:teuthology.orchestra.run.smithi183.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T12:05:00.236 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T12:05:00.236 INFO:teuthology.task.kernel:Updating grub on smithi183 to boot 5.14.0-508.el9.x86_64 2024-09-21T12:05:00.236 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T12:05:00.870 INFO:teuthology.orchestra.run.smithi183.stderr:Generating grub configuration file ... 2024-09-21T12:05:02.272 INFO:teuthology.orchestra.run.smithi183.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T12:05:02.301 INFO:teuthology.orchestra.run.smithi183.stderr:done 2024-09-21T12:05:02.303 DEBUG:teuthology.orchestra.run.smithi183:> mktemp 2024-09-21T12:05:02.318 INFO:teuthology.orchestra.run.smithi183.stdout:/tmp/tmp.KQPAPbAIeo 2024-09-21T12:05:02.318 DEBUG:teuthology.orchestra.run.smithi183:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.KQPAPbAIeo 2024-09-21T12:05:02.384 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0666 /tmp/tmp.KQPAPbAIeo 2024-09-21T12:05:02.530 DEBUG:teuthology.orchestra.remote:smithi183:/tmp/tmp.KQPAPbAIeo is 6KB 2024-09-21T12:05:02.578 DEBUG:teuthology.orchestra.run.smithi183:> rm -fr /tmp/tmp.KQPAPbAIeo 2024-09-21T12:05:02.593 DEBUG:teuthology.orchestra.run.smithi183:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T12:05:02.661 INFO:teuthology.orchestra.run.smithi183.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T12:05:02.661 INFO:teuthology.orchestra.run.smithi183.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T12:05:02.661 INFO:teuthology.orchestra.run.smithi183.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T12:05:02.661 INFO:teuthology.orchestra.run.smithi183.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T12:05:02.662 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T12:05:02.815 DEBUG:teuthology.orchestra.run.smithi183:> sudo shutdown -r now 2024-09-21T12:05:27.863 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-21T12:05:27.863 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-21T12:05:27.863 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T12:05:32.822 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-21T12:05:32.823 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-09-21T12:05:32.823 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T12:05:46.355 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-09-21T12:05:51.219 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-09-21T12:05:55.358 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-21T12:05:55.359 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T12:05:58.419 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-09-21T12:06:00.222 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-09-21T12:06:00.222 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T12:06:10.429 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-21T12:06:10.429 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T12:06:28.847 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-09-21T12:06:43.859 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-21T12:06:43.859 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T12:06:44.274 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-09-21T12:06:44.659 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-21T12:06:44.659 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-21T12:06:44.660 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-09-21T12:06:44.673 INFO:teuthology.orchestra.run.smithi114.stdout:5.14.0-508.el9.x86_64 2024-09-21T12:06:44.674 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T12:06:44.674 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T12:06:44.674 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T12:06:45.675 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-21T12:06:45.675 DEBUG:teuthology.orchestra.run.smithi114:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T12:06:45.746 INFO:teuthology.orchestra.run.smithi114.stdout:ttyS1 2024-09-21T12:06:45.768 DEBUG:teuthology.parallel:result is None 2024-09-21T12:07:00.224 DEBUG:teuthology.orchestra.remote:timed out 2024-09-21T12:07:12.225 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-09-21T12:07:12.226 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T12:07:12.623 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-09-21T12:07:12.968 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-09-21T12:07:12.968 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-21T12:07:12.968 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2024-09-21T12:07:12.984 INFO:teuthology.orchestra.run.smithi183.stdout:5.14.0-508.el9.x86_64 2024-09-21T12:07:12.984 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T12:07:12.984 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T12:07:12.984 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T12:07:13.985 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-21T12:07:13.985 DEBUG:teuthology.orchestra.run.smithi183:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T12:07:14.069 INFO:teuthology.orchestra.run.smithi183.stdout:ttyS1 2024-09-21T12:07:14.090 DEBUG:teuthology.parallel:result is None 2024-09-21T12:07:14.091 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-21T12:07:14.101 INFO:teuthology.task.internal:Creating test directory... 2024-09-21T12:07:14.101 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T12:07:14.104 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T12:07:14.123 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-21T12:07:14.131 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-21T12:07:14.175 INFO:teuthology.task.internal:Creating archive directory... 2024-09-21T12:07:14.175 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T12:07:14.178 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T12:07:14.228 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-21T12:07:14.234 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-21T12:07:14.234 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T12:07:14.237 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T12:07:14.265 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T12:07:14.276 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T12:07:14.279 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T12:07:14.290 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T12:07:14.292 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-21T12:07:14.311 INFO:teuthology.task.internal:Configuring sudo... 2024-09-21T12:07:14.311 DEBUG:teuthology.orchestra.run.smithi114:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T12:07:14.321 DEBUG:teuthology.orchestra.run.smithi183:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T12:07:14.361 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-21T12:07:14.369 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-21T12:07:14.370 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T12:07:14.389 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T12:07:14.419 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T12:07:14.489 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T12:07:14.535 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T12:07:14.618 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T12:07:14.659 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-21T12:07:14.659 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T12:07:14.723 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T12:07:14.768 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T12:07:14.830 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T12:07:14.874 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T12:07:14.917 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-09-21T12:07:14.917 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T12:07:14.984 DEBUG:teuthology.orchestra.run.smithi114:> sudo service rsyslog restart 2024-09-21T12:07:14.987 DEBUG:teuthology.orchestra.run.smithi183:> sudo service rsyslog restart 2024-09-21T12:07:15.067 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T12:07:15.091 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T12:07:15.308 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-21T12:07:15.322 INFO:teuthology.task.internal:Starting timer... 2024-09-21T12:07:15.322 INFO:teuthology.run_tasks:Running task pcp... 2024-09-21T12:07:15.405 INFO:teuthology.run_tasks:Running task selinux... 2024-09-21T12:07:15.490 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-09-21T12:07:15.490 DEBUG:teuthology.orchestra.run.smithi114:> sudo service auditd rotate 2024-09-21T12:07:15.583 INFO:teuthology.orchestra.run.smithi114.stdout:Rotating logs: 2024-09-21T12:07:15.585 DEBUG:teuthology.orchestra.run.smithi183:> sudo service auditd rotate 2024-09-21T12:07:15.632 INFO:teuthology.orchestra.run.smithi183.stdout:Rotating logs: 2024-09-21T12:07:15.634 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-21T12:07:15.634 DEBUG:teuthology.orchestra.run.smithi114:> /usr/sbin/getenforce 2024-09-21T12:07:15.659 INFO:teuthology.orchestra.run.smithi114.stdout:Permissive 2024-09-21T12:07:15.660 DEBUG:teuthology.orchestra.run.smithi183:> /usr/sbin/getenforce 2024-09-21T12:07:15.686 INFO:teuthology.orchestra.run.smithi183.stdout:Permissive 2024-09-21T12:07:15.687 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi114.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi183.front.sepia.ceph.com': 'permissive'} 2024-09-21T12:07:15.687 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T12:07:15.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:07:15.725 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T12:07:15.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:07:15.753 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-21T12:07:15.753 DEBUG:teuthology.orchestra.run.smithi114:> sudo /usr/sbin/setenforce permissive 2024-09-21T12:07:15.801 DEBUG:teuthology.orchestra.run.smithi183:> sudo /usr/sbin/setenforce permissive 2024-09-21T12:07:15.842 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-21T12:07:15.852 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-21T12:07:15.862 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-21T12:07:15.863 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi114.front.sepia.ceph.com,smithi183.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-21T12:10:56.707 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi114.front.sepia.ceph.com'), Remote(name='ubuntu@smithi183.front.sepia.ceph.com')] 2024-09-21T12:10:56.708 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-21T12:10:56.708 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T12:10:56.783 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-09-21T12:10:56.866 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-21T12:10:56.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-09-21T12:10:56.867 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T12:10:56.938 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-09-21T12:10:57.013 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-09-21T12:10:57.013 INFO:teuthology.run_tasks:Running task clock... 2024-09-21T12:10:57.022 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-21T12:10:57.022 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T12:10:57.022 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T12:10:57.025 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T12:10:57.025 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T12:10:57.053 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T12:10:57.068 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T12:10:57.083 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T12:10:57.093 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: ntpd: command not found 2024-09-21T12:10:57.096 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T12:10:57.104 INFO:teuthology.orchestra.run.smithi114.stdout:506 Cannot talk to daemon 2024-09-21T12:10:57.118 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T12:10:57.121 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: ntpd: command not found 2024-09-21T12:10:57.132 INFO:teuthology.orchestra.run.smithi183.stdout:506 Cannot talk to daemon 2024-09-21T12:10:57.132 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T12:10:57.146 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T12:10:57.160 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T12:10:57.190 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-09-21T12:10:57.194 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T12:10:57.194 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-09-21T12:10:57.194 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T12:10:57.194 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T12:10:57.194 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T12:10:57.194 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T12:10:57.215 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2024-09-21T12:10:57.219 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T12:10:57.219 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2024-09-21T12:10:57.219 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T12:10:57.219 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T12:10:57.219 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T12:10:57.219 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T12:10:57.219 INFO:teuthology.run_tasks:Running task pexec... 2024-09-21T12:10:57.228 INFO:teuthology.task.pexec:Executing custom commands... 2024-09-21T12:10:57.228 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi114.front.sepia.ceph.com 2024-09-21T12:10:57.228 DEBUG:teuthology.orchestra.run.smithi114:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-21T12:10:57.228 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi183.front.sepia.ceph.com 2024-09-21T12:10:57.229 DEBUG:teuthology.orchestra.run.smithi183:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-21T12:10:57.584 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-21T12:10:57.586 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:10:57.586 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-09-21T12:10:57.586 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:10:57.586 INFO:teuthology.orchestra.run.smithi114.stdout:Removing: 2024-09-21T12:10:57.586 INFO:teuthology.orchestra.run.smithi114.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-21T12:10:57.586 INFO:teuthology.orchestra.run.smithi114.stdout:Removing unused dependencies: 2024-09-21T12:10:57.587 INFO:teuthology.orchestra.run.smithi114.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-21T12:10:57.587 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:10:57.587 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-21T12:10:57.587 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:10:57.587 INFO:teuthology.orchestra.run.smithi114.stdout:Remove 2 Packages 2024-09-21T12:10:57.587 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:10:57.587 INFO:teuthology.orchestra.run.smithi114.stdout:Freed space: 5.5 M 2024-09-21T12:10:57.587 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-21T12:10:57.591 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-21T12:10:57.591 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-21T12:10:57.609 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-21T12:10:57.611 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:10:57.611 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-09-21T12:10:57.611 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:10:57.611 INFO:teuthology.orchestra.run.smithi183.stdout:Removing: 2024-09-21T12:10:57.611 INFO:teuthology.orchestra.run.smithi183.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-21T12:10:57.611 INFO:teuthology.orchestra.run.smithi183.stdout:Removing unused dependencies: 2024-09-21T12:10:57.611 INFO:teuthology.orchestra.run.smithi183.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-21T12:10:57.611 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:10:57.611 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-21T12:10:57.611 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:10:57.611 INFO:teuthology.orchestra.run.smithi183.stdout:Remove 2 Packages 2024-09-21T12:10:57.612 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:10:57.612 INFO:teuthology.orchestra.run.smithi183.stdout:Freed space: 5.5 M 2024-09-21T12:10:57.612 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-21T12:10:57.615 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-21T12:10:57.615 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-21T12:10:57.662 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-21T12:10:57.662 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-21T12:10:57.692 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-21T12:10:57.692 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-21T12:10:57.803 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-21T12:10:57.832 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-21T12:10:57.903 INFO:teuthology.orchestra.run.smithi114.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-21T12:10:57.916 INFO:teuthology.orchestra.run.smithi183.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-21T12:10:57.962 INFO:teuthology.orchestra.run.smithi114.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T12:10:57.977 INFO:teuthology.orchestra.run.smithi183.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T12:10:58.487 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T12:10:58.488 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-21T12:10:58.562 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T12:10:58.563 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-21T12:10:58.747 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-21T12:10:58.747 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:10:58.747 INFO:teuthology.orchestra.run.smithi114.stdout:Removed: 2024-09-21T12:10:58.747 INFO:teuthology.orchestra.run.smithi114.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T12:10:58.747 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:10:58.747 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-21T12:10:58.821 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-21T12:10:58.822 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:10:58.822 INFO:teuthology.orchestra.run.smithi183.stdout:Removed: 2024-09-21T12:10:58.822 INFO:teuthology.orchestra.run.smithi183.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T12:10:58.822 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:10:58.822 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-21T12:10:59.585 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:46 ago on Sat 21 Sep 2024 12:10:13 PM UTC. 2024-09-21T12:10:59.712 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:46 ago on Sat 21 Sep 2024 12:10:13 PM UTC. 2024-09-21T12:10:59.721 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-21T12:10:59.722 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:10:59.722 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-21T12:10:59.722 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:10:59.722 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-21T12:10:59.722 INFO:teuthology.orchestra.run.smithi114.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-21T12:10:59.722 INFO:teuthology.orchestra.run.smithi114.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-21T12:10:59.722 INFO:teuthology.orchestra.run.smithi114.stdout: runc x86_64 4:1.1.13-4.el9 appstream 3.2 M 2024-09-21T12:10:59.722 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-21T12:10:59.722 INFO:teuthology.orchestra.run.smithi114.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-21T12:10:59.722 INFO:teuthology.orchestra.run.smithi114.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-21T12:10:59.722 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-21T12:10:59.722 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-21T12:10:59.722 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:10:59.722 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-21T12:10:59.723 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:10:59.723 INFO:teuthology.orchestra.run.smithi114.stdout:Install 7 Packages 2024-09-21T12:10:59.723 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:10:59.733 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 5.2 M 2024-09-21T12:10:59.733 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 19 M 2024-09-21T12:10:59.733 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-21T12:10:59.849 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-21T12:10:59.849 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:10:59.849 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-09-21T12:10:59.849 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:10:59.850 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-21T12:10:59.850 INFO:teuthology.orchestra.run.smithi183.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-21T12:10:59.850 INFO:teuthology.orchestra.run.smithi183.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-21T12:10:59.850 INFO:teuthology.orchestra.run.smithi183.stdout: runc x86_64 4:1.1.13-4.el9 appstream 3.2 M 2024-09-21T12:10:59.850 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-09-21T12:10:59.850 INFO:teuthology.orchestra.run.smithi183.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-21T12:10:59.850 INFO:teuthology.orchestra.run.smithi183.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-21T12:10:59.850 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-21T12:10:59.850 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-21T12:10:59.850 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:10:59.850 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-21T12:10:59.850 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:10:59.850 INFO:teuthology.orchestra.run.smithi183.stdout:Install 7 Packages 2024-09-21T12:10:59.850 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:10:59.864 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 5.2 M 2024-09-21T12:10:59.864 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 19 M 2024-09-21T12:10:59.864 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-21T12:10:59.967 INFO:teuthology.orchestra.run.smithi114.stdout:(1/7): nvmetcli-0.7-3.el9.noarch.rpm 512 kB/s | 44 kB 00:00 2024-09-21T12:10:59.993 INFO:teuthology.orchestra.run.smithi114.stdout:(2/7): libnvme-1.9-3.el9.x86_64.rpm 943 kB/s | 106 kB 00:00 2024-09-21T12:11:00.035 INFO:teuthology.orchestra.run.smithi114.stdout:(3/7): python3-kmod-0.9-32.el9.x86_64.rpm 2.0 MB/s | 84 kB 00:00 2024-09-21T12:11:00.060 INFO:teuthology.orchestra.run.smithi114.stdout:(4/7): python3-configshell-1.1.30-1.el9.noarch. 776 kB/s | 72 kB 00:00 2024-09-21T12:11:00.102 INFO:teuthology.orchestra.run.smithi114.stdout:(5/7): nvme-cli-2.9.1-6.el9.x86_64.rpm 4.0 MB/s | 906 kB 00:00 2024-09-21T12:11:00.133 INFO:teuthology.orchestra.run.smithi183.stdout:(1/7): nvmetcli-0.7-3.el9.noarch.rpm 468 kB/s | 44 kB 00:00 2024-09-21T12:11:00.158 INFO:teuthology.orchestra.run.smithi183.stdout:(2/7): libnvme-1.9-3.el9.x86_64.rpm 880 kB/s | 106 kB 00:00 2024-09-21T12:11:00.192 INFO:teuthology.orchestra.run.smithi183.stdout:(3/7): python3-configshell-1.1.30-1.el9.noarch. 1.2 MB/s | 72 kB 00:00 2024-09-21T12:11:00.210 INFO:teuthology.orchestra.run.smithi114.stdout:(6/7): python3-urwid-2.1.2-4.el9.x86_64.rpm 4.7 MB/s | 837 kB 00:00 2024-09-21T12:11:00.217 INFO:teuthology.orchestra.run.smithi183.stdout:(4/7): python3-kmod-0.9-32.el9.x86_64.rpm 1.4 MB/s | 84 kB 00:00 2024-09-21T12:11:00.259 INFO:teuthology.orchestra.run.smithi183.stdout:(5/7): nvme-cli-2.9.1-6.el9.x86_64.rpm 4.0 MB/s | 906 kB 00:00 2024-09-21T12:11:00.301 INFO:teuthology.orchestra.run.smithi183.stdout:(6/7): python3-urwid-2.1.2-4.el9.x86_64.rpm 7.5 MB/s | 837 kB 00:00 2024-09-21T12:11:00.485 INFO:teuthology.orchestra.run.smithi183.stdout:(7/7): runc-1.1.13-4.el9.x86_64.rpm 12 MB/s | 3.2 MB 00:00 2024-09-21T12:11:00.485 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:11:00.485 INFO:teuthology.orchestra.run.smithi183.stdout:Total 8.3 MB/s | 5.2 MB 00:00 2024-09-21T12:11:00.592 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-21T12:11:00.607 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-21T12:11:00.607 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-21T12:11:00.928 INFO:teuthology.orchestra.run.smithi114.stdout:(7/7): runc-1.1.13-4.el9.x86_64.rpm 3.6 MB/s | 3.2 MB 00:00 2024-09-21T12:11:00.928 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:11:00.928 INFO:teuthology.orchestra.run.smithi114.stdout:Total 4.3 MB/s | 5.2 MB 00:01 2024-09-21T12:11:00.968 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-21T12:11:00.968 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-21T12:11:01.038 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-21T12:11:01.055 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-21T12:11:01.055 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-21T12:11:01.337 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-21T12:11:01.338 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-21T12:11:01.450 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-21T12:11:01.554 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/7 2024-09-21T12:11:01.588 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/7 2024-09-21T12:11:01.617 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/7 2024-09-21T12:11:01.819 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-21T12:11:01.827 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/7 2024-09-21T12:11:01.856 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-21T12:11:01.914 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/7 2024-09-21T12:11:01.948 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/7 2024-09-21T12:11:01.976 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/7 2024-09-21T12:11:02.183 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/7 2024-09-21T12:11:02.216 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-21T12:11:02.313 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-21T12:11:02.343 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/7 2024-09-21T12:11:02.463 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/7 2024-09-21T12:11:02.529 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T12:11:02.658 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-21T12:11:02.687 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/7 2024-09-21T12:11:02.804 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/7 2024-09-21T12:11:02.864 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T12:11:03.438 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T12:11:03.438 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/7 2024-09-21T12:11:03.438 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/7 2024-09-21T12:11:03.438 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/7 2024-09-21T12:11:03.438 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/7 2024-09-21T12:11:03.438 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/7 2024-09-21T12:11:03.438 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/7 2024-09-21T12:11:03.834 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T12:11:03.835 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:11:03.835 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-21T12:11:03.835 INFO:teuthology.orchestra.run.smithi183.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T12:11:03.835 INFO:teuthology.orchestra.run.smithi183.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-21T12:11:03.835 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-21T12:11:03.835 INFO:teuthology.orchestra.run.smithi183.stdout: runc-4:1.1.13-4.el9.x86_64 2024-09-21T12:11:03.835 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:11:03.835 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-21T12:11:03.858 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T12:11:03.858 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/7 2024-09-21T12:11:03.859 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/7 2024-09-21T12:11:03.859 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/7 2024-09-21T12:11:03.859 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/7 2024-09-21T12:11:03.859 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/7 2024-09-21T12:11:03.859 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/7 2024-09-21T12:11:04.200 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T12:11:04.200 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:11:04.200 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-21T12:11:04.200 INFO:teuthology.orchestra.run.smithi114.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T12:11:04.200 INFO:teuthology.orchestra.run.smithi114.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-21T12:11:04.200 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-21T12:11:04.200 INFO:teuthology.orchestra.run.smithi114.stdout: runc-4:1.1.13-4.el9.x86_64 2024-09-21T12:11:04.200 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:11:04.200 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-21T12:11:04.210 DEBUG:teuthology.parallel:result is None 2024-09-21T12:11:04.580 DEBUG:teuthology.parallel:result is None 2024-09-21T12:11:04.580 INFO:teuthology.run_tasks:Running task install... 2024-09-21T12:11:04.594 DEBUG:teuthology.task.install:project ceph 2024-09-21T12:11:04.594 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5c1cdc3c470dae78e91a0ed2bcf13509a046d301'}} 2024-09-21T12:11:04.594 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5c1cdc3c470dae78e91a0ed2bcf13509a046d301'} 2024-09-21T12:11:04.594 INFO:teuthology.task.install:Using flavor: default 2024-09-21T12:11:04.599 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-21T12:11:04.599 INFO:teuthology.task.install:extra packages: [] 2024-09-21T12:11:04.599 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5c1cdc3c470dae78e91a0ed2bcf13509a046d301', 'tag': None, 'wait_for_package': False} 2024-09-21T12:11:04.599 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5c1cdc3c470dae78e91a0ed2bcf13509a046d301 2024-09-21T12:11:04.601 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5c1cdc3c470dae78e91a0ed2bcf13509a046d301', 'tag': None, 'wait_for_package': False} 2024-09-21T12:11:04.601 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5c1cdc3c470dae78e91a0ed2bcf13509a046d301 2024-09-21T12:11:04.749 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-21-0813/5c1cdc3c470dae78e91a0ed2bcf13509a046d301/centos/9/flavors/default/ 2024-09-21T12:11:04.750 INFO:teuthology.task.install.rpm:Package version is 19.3.0-5103.g5c1cdc3c 2024-09-21T12:11:04.753 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-21-0813/5c1cdc3c470dae78e91a0ed2bcf13509a046d301/centos/9/flavors/default/ 2024-09-21T12:11:04.753 INFO:teuthology.task.install.rpm:Package version is 19.3.0-5103.g5c1cdc3c 2024-09-21T12:11:04.858 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-21-0813/5c1cdc3c470dae78e91a0ed2bcf13509a046d301/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-21-0813/5c1cdc3c470dae78e91a0ed2bcf13509a046d301/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-21-0813/5c1cdc3c470dae78e91a0ed2bcf13509a046d301/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T12:11:04.859 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-09-21T12:11:04.859 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T12:11:04.866 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-21-0813/5c1cdc3c470dae78e91a0ed2bcf13509a046d301/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-21-0813/5c1cdc3c470dae78e91a0ed2bcf13509a046d301/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-21-0813/5c1cdc3c470dae78e91a0ed2bcf13509a046d301/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T12:11:04.866 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-21T12:11:04.866 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T12:11:04.890 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-21T12:11:04.890 DEBUG:teuthology.orchestra.run.smithi183:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5c1cdc3c470dae78e91a0ed2bcf13509a046d301/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T12:11:04.891 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-21T12:11:04.891 DEBUG:teuthology.orchestra.run.smithi114:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5c1cdc3c470dae78e91a0ed2bcf13509a046d301/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T12:11:04.958 DEBUG:teuthology.orchestra.run.smithi114:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T12:11:04.959 DEBUG:teuthology.orchestra.run.smithi183:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T12:11:05.032 DEBUG:teuthology.orchestra.run.smithi114:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T12:11:05.039 DEBUG:teuthology.orchestra.run.smithi183:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T12:11:05.096 INFO:teuthology.orchestra.run.smithi114.stdout:check_obsoletes = 1 2024-09-21T12:11:05.098 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum clean all 2024-09-21T12:11:05.106 INFO:teuthology.orchestra.run.smithi183.stdout:check_obsoletes = 1 2024-09-21T12:11:05.108 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum clean all 2024-09-21T12:11:05.328 INFO:teuthology.orchestra.run.smithi183.stdout:45 files removed 2024-09-21T12:11:05.338 INFO:teuthology.orchestra.run.smithi114.stdout:45 files removed 2024-09-21T12:11:05.355 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-radosgw 2024-09-21T12:11:05.364 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-radosgw 2024-09-21T12:11:05.908 INFO:teuthology.orchestra.run.smithi183.stdout:ceph packages for x86_64 310 kB/s | 84 kB 00:00 2024-09-21T12:11:05.926 INFO:teuthology.orchestra.run.smithi114.stdout:ceph packages for x86_64 302 kB/s | 84 kB 00:00 2024-09-21T12:11:06.209 INFO:teuthology.orchestra.run.smithi183.stdout:ceph noarch packages 94 kB/s | 24 kB 00:00 2024-09-21T12:11:06.218 INFO:teuthology.orchestra.run.smithi114.stdout:ceph noarch packages 97 kB/s | 24 kB 00:00 2024-09-21T12:11:06.492 INFO:teuthology.orchestra.run.smithi183.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-21T12:11:06.518 INFO:teuthology.orchestra.run.smithi114.stdout:ceph source packages 7.4 kB/s | 1.9 kB 00:00 2024-09-21T12:11:07.109 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.3 MB 00:00 2024-09-21T12:11:08.869 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - AppStream 29 MB/s | 20 MB 00:00 2024-09-21T12:11:09.196 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 3.1 MB/s | 8.3 MB 00:02 2024-09-21T12:11:11.714 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 14 MB/s | 20 MB 00:01 2024-09-21T12:11:13.303 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.5 MB 00:00 2024-09-21T12:11:14.937 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - Extras packages 65 kB/s | 19 kB 00:00 2024-09-21T12:11:15.866 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 28 MB/s | 23 MB 00:00 2024-09-21T12:11:16.947 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 4.6 MB/s | 6.5 MB 00:01 2024-09-21T12:11:18.732 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 19 kB 00:00 2024-09-21T12:11:19.861 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:01 2024-09-21T12:11:21.213 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-21T12:11:23.439 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-21T12:11:23.441 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:11:23.441 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-09-21T12:11:23.442 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:11:23.442 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-21T12:11:23.442 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 11 M 2024-09-21T12:11:23.442 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrading: 2024-09-21T12:11:23.442 INFO:teuthology.orchestra.run.smithi183.stdout: librados2 x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 3.5 M 2024-09-21T12:11:23.442 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1 x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 3.2 M 2024-09-21T12:11:23.442 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-09-21T12:11:23.442 INFO:teuthology.orchestra.run.smithi183.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T12:11:23.442 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 5.5 M 2024-09-21T12:11:23.442 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 22 M 2024-09-21T12:11:23.442 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 25 k 2024-09-21T12:11:23.442 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T12:11:23.442 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T12:11:23.443 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T12:11:23.443 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2 x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 755 k 2024-09-21T12:11:23.443 INFO:teuthology.orchestra.run.smithi183.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-21T12:11:23.443 INFO:teuthology.orchestra.run.smithi183.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T12:11:23.443 INFO:teuthology.orchestra.run.smithi183.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T12:11:23.443 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T12:11:23.443 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1 x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 509 k 2024-09-21T12:11:23.443 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T12:11:23.443 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2 x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 5.5 M 2024-09-21T12:11:23.443 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T12:11:23.443 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T12:11:23.443 INFO:teuthology.orchestra.run.smithi183.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T12:11:23.443 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 45 k 2024-09-21T12:11:23.443 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 148 k 2024-09-21T12:11:23.444 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 161 k 2024-09-21T12:11:23.444 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T12:11:23.444 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 321 k 2024-09-21T12:11:23.444 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 306 k 2024-09-21T12:11:23.444 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 99 k 2024-09-21T12:11:23.444 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-21T12:11:23.444 INFO:teuthology.orchestra.run.smithi183.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-21T12:11:23.444 INFO:teuthology.orchestra.run.smithi183.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T12:11:23.444 INFO:teuthology.orchestra.run.smithi183.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T12:11:23.444 INFO:teuthology.orchestra.run.smithi183.stdout:Installing weak dependencies: 2024-09-21T12:11:23.444 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-21T12:11:23.444 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:11:23.444 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-21T12:11:23.444 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:11:23.445 INFO:teuthology.orchestra.run.smithi183.stdout:Install 31 Packages 2024-09-21T12:11:23.445 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrade 2 Packages 2024-09-21T12:11:23.445 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:11:23.445 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 62 M 2024-09-21T12:11:23.445 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-21T12:11:24.415 INFO:teuthology.orchestra.run.smithi183.stdout:(1/33): ceph-base-19.3.0-5103.g5c1cdc3c.el9.x86 9.5 MB/s | 5.5 MB 00:00 2024-09-21T12:11:24.465 INFO:teuthology.orchestra.run.smithi183.stdout:(2/33): ceph-selinux-19.3.0-5103.g5c1cdc3c.el9. 505 kB/s | 25 kB 00:00 2024-09-21T12:11:24.582 INFO:teuthology.orchestra.run.smithi183.stdout:(3/33): ceph-radosgw-19.3.0-5103.g5c1cdc3c.el9. 15 MB/s | 11 MB 00:00 2024-09-21T12:11:24.615 INFO:teuthology.orchestra.run.smithi183.stdout:(4/33): libcephfs2-19.3.0-5103.g5c1cdc3c.el9.x8 4.9 MB/s | 755 kB 00:00 2024-09-21T12:11:24.640 INFO:teuthology.orchestra.run.smithi183.stdout:(5/33): libradosstriper1-19.3.0-5103.g5c1cdc3c. 8.5 MB/s | 509 kB 00:00 2024-09-21T12:11:24.699 INFO:teuthology.orchestra.run.smithi183.stdout:(6/33): python3-ceph-argparse-19.3.0-5103.g5c1c 773 kB/s | 45 kB 00:00 2024-09-21T12:11:24.766 INFO:teuthology.orchestra.run.smithi183.stdout:(7/33): python3-ceph-common-19.3.0-5103.g5c1cdc 2.2 MB/s | 148 kB 00:00 2024-09-21T12:11:24.875 INFO:teuthology.orchestra.run.smithi183.stdout:(8/33): librgw2-19.3.0-5103.g5c1cdc3c.el9.x86_6 21 MB/s | 5.5 MB 00:00 2024-09-21T12:11:24.900 INFO:teuthology.orchestra.run.smithi183.stdout:(9/33): python3-cephfs-19.3.0-5103.g5c1cdc3c.el 1.2 MB/s | 161 kB 00:00 2024-09-21T12:11:24.933 INFO:teuthology.orchestra.run.smithi183.stdout:(10/33): python3-rados-19.3.0-5103.g5c1cdc3c.el 5.4 MB/s | 321 kB 00:00 2024-09-21T12:11:24.959 INFO:teuthology.orchestra.run.smithi183.stdout:(11/33): python3-rbd-19.3.0-5103.g5c1cdc3c.el9. 5.1 MB/s | 306 kB 00:00 2024-09-21T12:11:24.984 INFO:teuthology.orchestra.run.smithi183.stdout:(12/33): python3-rgw-19.3.0-5103.g5c1cdc3c.el9. 1.9 MB/s | 99 kB 00:00 2024-09-21T12:11:25.150 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-21T12:11:25.184 INFO:teuthology.orchestra.run.smithi183.stdout:(13/33): ceph-common-19.3.0-5103.g5c1cdc3c.el9. 16 MB/s | 22 MB 00:01 2024-09-21T12:11:25.235 INFO:teuthology.orchestra.run.smithi183.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 388 kB/s | 107 kB 00:00 2024-09-21T12:11:25.293 INFO:teuthology.orchestra.run.smithi183.stdout:(15/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 776 kB/s | 45 kB 00:00 2024-09-21T12:11:25.352 INFO:teuthology.orchestra.run.smithi183.stdout:(16/33): libnbd-1.20.2-2.el9.x86_64.rpm 464 kB/s | 171 kB 00:00 2024-09-21T12:11:25.377 INFO:teuthology.orchestra.run.smithi183.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 834 kB/s | 160 kB 00:00 2024-09-21T12:11:25.444 INFO:teuthology.orchestra.run.smithi183.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 633 kB/s | 42 kB 00:00 2024-09-21T12:11:25.469 INFO:teuthology.orchestra.run.smithi183.stdout:(19/33): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 662 kB 00:00 2024-09-21T12:11:25.503 INFO:teuthology.orchestra.run.smithi183.stdout:(20/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.9 MB/s | 292 kB 00:00 2024-09-21T12:11:25.528 INFO:teuthology.orchestra.run.smithi183.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 611 kB/s | 36 kB 00:00 2024-09-21T12:11:25.553 INFO:teuthology.orchestra.run.smithi183.stdout:(22/33): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-09-21T12:11:25.571 INFO:teuthology.orchestra.run.smithi183.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 702 kB/s | 47 kB 00:00 2024-09-21T12:11:25.621 INFO:teuthology.orchestra.run.smithi183.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.2 MB/s | 308 kB 00:00 2024-09-21T12:11:25.638 INFO:teuthology.orchestra.run.smithi183.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 3.3 MB/s | 49 kB 00:00 2024-09-21T12:11:25.655 INFO:teuthology.orchestra.run.smithi183.stdout:(26/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 373 kB/s | 26 kB 00:00 2024-09-21T12:11:25.680 INFO:teuthology.orchestra.run.smithi183.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-09-21T12:11:25.697 INFO:teuthology.orchestra.run.smithi183.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 11 MB/s | 191 kB 00:00 2024-09-21T12:11:25.731 INFO:teuthology.orchestra.run.smithi183.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 11 MB/s | 839 kB 00:00 2024-09-21T12:11:25.781 INFO:teuthology.orchestra.run.smithi183.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-09-21T12:11:25.873 INFO:teuthology.orchestra.run.smithi183.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-09-21T12:11:25.973 INFO:teuthology.orchestra.run.smithi183.stdout:(32/33): librados2-19.3.0-5103.g5c1cdc3c.el9.x8 14 MB/s | 3.5 MB 00:00 2024-09-21T12:11:26.056 INFO:teuthology.orchestra.run.smithi183.stdout:(33/33): librbd1-19.3.0-5103.g5c1cdc3c.el9.x86_ 12 MB/s | 3.2 MB 00:00 2024-09-21T12:11:26.061 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:11:26.061 INFO:teuthology.orchestra.run.smithi183.stdout:Total 24 MB/s | 62 MB 00:02 2024-09-21T12:11:26.166 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-21T12:11:26.201 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-21T12:11:26.202 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-21T12:11:26.714 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-21T12:11:26.715 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-21T12:11:27.378 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-21T12:11:27.381 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:11:27.381 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-21T12:11:27.381 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:11:27.381 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-21T12:11:27.381 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 11 M 2024-09-21T12:11:27.381 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrading: 2024-09-21T12:11:27.381 INFO:teuthology.orchestra.run.smithi114.stdout: librados2 x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 3.5 M 2024-09-21T12:11:27.381 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1 x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 3.2 M 2024-09-21T12:11:27.381 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-21T12:11:27.381 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T12:11:27.381 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 5.5 M 2024-09-21T12:11:27.381 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 22 M 2024-09-21T12:11:27.381 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 25 k 2024-09-21T12:11:27.381 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T12:11:27.381 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T12:11:27.382 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T12:11:27.382 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2 x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 755 k 2024-09-21T12:11:27.382 INFO:teuthology.orchestra.run.smithi114.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-21T12:11:27.382 INFO:teuthology.orchestra.run.smithi114.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T12:11:27.382 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T12:11:27.382 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T12:11:27.382 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1 x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 509 k 2024-09-21T12:11:27.382 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T12:11:27.382 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2 x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 5.5 M 2024-09-21T12:11:27.382 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T12:11:27.382 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T12:11:27.382 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T12:11:27.382 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 45 k 2024-09-21T12:11:27.383 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 148 k 2024-09-21T12:11:27.383 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 161 k 2024-09-21T12:11:27.383 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T12:11:27.383 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 321 k 2024-09-21T12:11:27.383 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 306 k 2024-09-21T12:11:27.383 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 99 k 2024-09-21T12:11:27.383 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-21T12:11:27.383 INFO:teuthology.orchestra.run.smithi114.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-21T12:11:27.383 INFO:teuthology.orchestra.run.smithi114.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T12:11:27.383 INFO:teuthology.orchestra.run.smithi114.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T12:11:27.383 INFO:teuthology.orchestra.run.smithi114.stdout:Installing weak dependencies: 2024-09-21T12:11:27.383 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-21T12:11:27.383 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:11:27.383 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-21T12:11:27.384 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:11:27.384 INFO:teuthology.orchestra.run.smithi114.stdout:Install 31 Packages 2024-09-21T12:11:27.384 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrade 2 Packages 2024-09-21T12:11:27.384 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:11:27.384 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 62 M 2024-09-21T12:11:27.384 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-21T12:11:28.193 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-21T12:11:28.276 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-21T12:11:28.313 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-21T12:11:28.338 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-21T12:11:28.361 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-21T12:11:28.381 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-21T12:11:28.468 INFO:teuthology.orchestra.run.smithi114.stdout:(1/33): ceph-base-19.3.0-5103.g5c1cdc3c.el9.x86 8.4 MB/s | 5.5 MB 00:00 2024-09-21T12:11:28.526 INFO:teuthology.orchestra.run.smithi114.stdout:(2/33): ceph-selinux-19.3.0-5103.g5c1cdc3c.el9. 433 kB/s | 25 kB 00:00 2024-09-21T12:11:28.562 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-21T12:11:28.616 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-21T12:11:28.652 INFO:teuthology.orchestra.run.smithi114.stdout:(3/33): ceph-radosgw-19.3.0-5103.g5c1cdc3c.el9. 13 MB/s | 11 MB 00:00 2024-09-21T12:11:28.712 INFO:teuthology.orchestra.run.smithi114.stdout:(4/33): libcephfs2-19.3.0-5103.g5c1cdc3c.el9.x8 4.7 MB/s | 755 kB 00:00 2024-09-21T12:11:28.712 INFO:teuthology.orchestra.run.smithi114.stdout:(5/33): libradosstriper1-19.3.0-5103.g5c1cdc3c. 8.5 MB/s | 509 kB 00:00 2024-09-21T12:11:28.761 INFO:teuthology.orchestra.run.smithi114.stdout:(6/33): python3-ceph-argparse-19.3.0-5103.g5c1c 901 kB/s | 45 kB 00:00 2024-09-21T12:11:28.836 INFO:teuthology.orchestra.run.smithi114.stdout:(7/33): python3-ceph-common-19.3.0-5103.g5c1cdc 1.9 MB/s | 148 kB 00:00 2024-09-21T12:11:28.954 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-21T12:11:29.037 INFO:teuthology.orchestra.run.smithi114.stdout:(8/33): ceph-common-19.3.0-5103.g5c1cdc3c.el9.x 18 MB/s | 22 MB 00:01 2024-09-21T12:11:29.062 INFO:teuthology.orchestra.run.smithi114.stdout:(9/33): python3-cephfs-19.3.0-5103.g5c1cdc3c.el 717 kB/s | 161 kB 00:00 2024-09-21T12:11:29.065 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librados2-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 7/35 2024-09-21T12:11:29.087 INFO:teuthology.orchestra.run.smithi114.stdout:(10/33): python3-rados-19.3.0-5103.g5c1cdc3c.el 6.3 MB/s | 321 kB 00:00 2024-09-21T12:11:29.112 INFO:teuthology.orchestra.run.smithi114.stdout:(11/33): python3-rbd-19.3.0-5103.g5c1cdc3c.el9. 6.0 MB/s | 306 kB 00:00 2024-09-21T12:11:29.137 INFO:teuthology.orchestra.run.smithi114.stdout:(12/33): python3-rgw-19.3.0-5103.g5c1cdc3c.el9. 1.9 MB/s | 99 kB 00:00 2024-09-21T12:11:29.158 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 7/35 2024-09-21T12:11:29.213 INFO:teuthology.orchestra.run.smithi114.stdout:(13/33): librgw2-19.3.0-5103.g5c1cdc3c.el9.x86_ 10 MB/s | 5.5 MB 00:00 2024-09-21T12:11:29.258 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rados-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 8/35 2024-09-21T12:11:29.283 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs2-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 9/35 2024-09-21T12:11:29.305 INFO:teuthology.orchestra.run.smithi114.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 556 kB/s | 107 kB 00:00 2024-09-21T12:11:29.346 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephfs2-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 9/35 2024-09-21T12:11:29.347 INFO:teuthology.orchestra.run.smithi114.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 816 kB/s | 171 kB 00:00 2024-09-21T12:11:29.372 INFO:teuthology.orchestra.run.smithi114.stdout:(16/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 678 kB/s | 45 kB 00:00 2024-09-21T12:11:29.383 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-21T12:11:29.414 INFO:teuthology.orchestra.run.smithi114.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 798 kB/s | 160 kB 00:00 2024-09-21T12:11:29.418 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-argparse-2:19.3.0-5103.g5c1cdc3c.el9. 11/35 2024-09-21T12:11:29.456 INFO:teuthology.orchestra.run.smithi114.stdout:(18/33): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 662 kB 00:00 2024-09-21T12:11:29.481 INFO:teuthology.orchestra.run.smithi114.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-09-21T12:11:29.489 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cephfs-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 12/35 2024-09-21T12:11:29.506 INFO:teuthology.orchestra.run.smithi114.stdout:(20/33): python3-prettytable-0.7.2-27.el9.noarc 459 kB/s | 42 kB 00:00 2024-09-21T12:11:29.518 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libradosstriper1-2:19.3.0-5103.g5c1cdc3c.el9.x86_6 13/35 2024-09-21T12:11:29.531 INFO:teuthology.orchestra.run.smithi114.stdout:(21/33): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 221 kB 00:00 2024-09-21T12:11:29.556 INFO:teuthology.orchestra.run.smithi114.stdout:(22/33): qatlib-service-24.02.0-1.el9.x86_64.rp 476 kB/s | 36 kB 00:00 2024-09-21T12:11:29.582 INFO:teuthology.orchestra.run.smithi114.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 628 kB/s | 47 kB 00:00 2024-09-21T12:11:29.582 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libradosstriper1-2:19.3.0-5103.g5c1cdc3c.el9.x86_6 13/35 2024-09-21T12:11:29.617 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-21T12:11:29.640 INFO:teuthology.orchestra.run.smithi114.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-21T12:11:29.666 INFO:teuthology.orchestra.run.smithi114.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 2.0 MB/s | 49 kB 00:00 2024-09-21T12:11:29.683 INFO:teuthology.orchestra.run.smithi114.stdout:(26/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 304 kB/s | 26 kB 00:00 2024-09-21T12:11:29.708 INFO:teuthology.orchestra.run.smithi114.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-09-21T12:11:29.728 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-21T12:11:29.741 INFO:teuthology.orchestra.run.smithi114.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-21T12:11:29.777 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-21T12:11:29.800 INFO:teuthology.orchestra.run.smithi114.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-09-21T12:11:29.817 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-21T12:11:29.842 INFO:teuthology.orchestra.run.smithi114.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-21T12:11:29.850 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-21T12:11:29.879 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-21T12:11:29.919 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-21T12:11:29.926 INFO:teuthology.orchestra.run.smithi114.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-21T12:11:29.965 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-21T12:11:30.034 INFO:teuthology.orchestra.run.smithi114.stdout:(32/33): librados2-19.3.0-5103.g5c1cdc3c.el9.x8 15 MB/s | 3.5 MB 00:00 2024-09-21T12:11:30.140 INFO:teuthology.orchestra.run.smithi114.stdout:(33/33): librbd1-19.3.0-5103.g5c1cdc3c.el9.x86_ 11 MB/s | 3.2 MB 00:00 2024-09-21T12:11:30.145 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:11:30.146 INFO:teuthology.orchestra.run.smithi114.stdout:Total 22 MB/s | 62 MB 00:02 2024-09-21T12:11:30.255 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-21T12:11:30.289 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-21T12:11:30.292 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-21T12:11:30.292 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-21T12:11:30.334 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librbd1-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 23/35 2024-09-21T12:11:30.432 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 23/35 2024-09-21T12:11:30.455 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rbd-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 24/35 2024-09-21T12:11:30.492 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-21T12:11:30.783 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-21T12:11:30.784 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-21T12:11:30.985 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-21T12:11:31.011 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librgw2-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 27/35 2024-09-21T12:11:31.074 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librgw2-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 27/35 2024-09-21T12:11:31.157 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rgw-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 28/35 2024-09-21T12:11:31.197 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-common-2:19.3.0-5103.g5c1cdc3c.el9.x8 29/35 2024-09-21T12:11:32.244 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-21T12:11:32.327 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-21T12:11:32.356 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-21T12:11:32.382 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-21T12:11:32.404 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-21T12:11:32.427 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-21T12:11:32.570 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-21T12:11:32.646 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-21T12:11:33.096 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-21T12:11:33.142 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librados2-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 7/35 2024-09-21T12:11:33.235 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 7/35 2024-09-21T12:11:33.325 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rados-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 8/35 2024-09-21T12:11:33.352 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs2-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 9/35 2024-09-21T12:11:33.415 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephfs2-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 9/35 2024-09-21T12:11:33.443 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-21T12:11:33.487 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-argparse-2:19.3.0-5103.g5c1cdc3c.el9. 11/35 2024-09-21T12:11:33.544 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 30/35 2024-09-21T12:11:33.558 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cephfs-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 12/35 2024-09-21T12:11:33.584 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-common-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 30/35 2024-09-21T12:11:33.587 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libradosstriper1-2:19.3.0-5103.g5c1cdc3c.el9.x86_6 13/35 2024-09-21T12:11:33.635 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libradosstriper1-2:19.3.0-5103.g5c1cdc3c.el9.x86_6 13/35 2024-09-21T12:11:33.669 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-21T12:11:33.787 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-21T12:11:33.820 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-21T12:11:33.861 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-21T12:11:33.894 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-21T12:11:33.923 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-21T12:11:33.956 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-21T12:11:33.994 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-21T12:11:34.194 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 30/35 2024-09-21T12:11:34.229 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-base-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 31/35 2024-09-21T12:11:34.325 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-21T12:11:34.325 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-base-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 31/35 2024-09-21T12:11:34.325 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T12:11:34.326 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T12:11:34.326 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:11:34.361 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librbd1-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 23/35 2024-09-21T12:11:34.364 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-selinux-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 32/35 2024-09-21T12:11:34.456 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 23/35 2024-09-21T12:11:34.491 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rbd-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 24/35 2024-09-21T12:11:34.535 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-21T12:11:35.028 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-21T12:11:35.055 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librgw2-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 27/35 2024-09-21T12:11:35.127 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librgw2-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 27/35 2024-09-21T12:11:35.207 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rgw-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 28/35 2024-09-21T12:11:35.241 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-common-2:19.3.0-5103.g5c1cdc3c.el9.x8 29/35 2024-09-21T12:11:37.574 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 30/35 2024-09-21T12:11:37.604 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-common-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 30/35 2024-09-21T12:11:38.190 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 30/35 2024-09-21T12:11:38.224 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-base-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 31/35 2024-09-21T12:11:38.313 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-base-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 31/35 2024-09-21T12:11:38.313 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T12:11:38.313 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T12:11:38.313 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:11:38.350 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-selinux-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 32/35 2024-09-21T12:11:44.607 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-selinux-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 32/35 2024-09-21T12:11:44.607 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /sys 2024-09-21T12:11:44.608 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /proc 2024-09-21T12:11:44.608 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /mnt 2024-09-21T12:11:44.608 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /var/tmp 2024-09-21T12:11:44.608 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /home 2024-09-21T12:11:44.608 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /root 2024-09-21T12:11:44.608 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /tmp 2024-09-21T12:11:44.608 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:11:45.639 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-radosgw-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 33/35 2024-09-21T12:11:45.665 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 33/35 2024-09-21T12:11:45.665 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T12:11:45.665 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T12:11:45.665 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T12:11:45.665 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T12:11:45.665 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:11:45.667 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-21T12:11:45.742 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-21T12:11:45.742 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T12:11:46.953 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T12:11:46.953 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-base-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/35 2024-09-21T12:11:46.953 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-common-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2/35 2024-09-21T12:11:46.953 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-radosgw-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 3/35 2024-09-21T12:11:46.953 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-selinux-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 4/35 2024-09-21T12:11:46.954 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs2-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 5/35 2024-09-21T12:11:46.954 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libradosstriper1-2:19.3.0-5103.g5c1cdc3c.el9.x86_6 6/35 2024-09-21T12:11:46.954 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librgw2-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 7/35 2024-09-21T12:11:46.954 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-argparse-2:19.3.0-5103.g5c1cdc3c.el9. 8/35 2024-09-21T12:11:46.954 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-common-2:19.3.0-5103.g5c1cdc3c.el9.x8 9/35 2024-09-21T12:11:46.954 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cephfs-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 10/35 2024-09-21T12:11:46.954 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rados-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 11/35 2024-09-21T12:11:46.954 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rbd-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 12/35 2024-09-21T12:11:46.954 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rgw-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 13/35 2024-09-21T12:11:46.954 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-21T12:11:46.954 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-21T12:11:46.954 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-21T12:11:46.954 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-21T12:11:46.955 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-21T12:11:46.955 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-21T12:11:46.955 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-21T12:11:46.955 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-21T12:11:46.955 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-21T12:11:46.955 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-21T12:11:46.955 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-21T12:11:46.955 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-21T12:11:46.955 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-21T12:11:46.955 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-21T12:11:46.956 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-21T12:11:46.956 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-21T12:11:46.956 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-21T12:11:46.956 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-21T12:11:46.956 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 32/35 2024-09-21T12:11:46.956 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-21T12:11:46.957 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 34/35 2024-09-21T12:11:47.934 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T12:11:47.934 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:11:47.934 INFO:teuthology.orchestra.run.smithi183.stdout:Upgraded: 2024-09-21T12:11:47.934 INFO:teuthology.orchestra.run.smithi183.stdout: librados2-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:47.934 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:47.934 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-21T12:11:47.935 INFO:teuthology.orchestra.run.smithi183.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T12:11:47.935 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:47.935 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:47.935 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:47.935 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:47.935 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T12:11:47.935 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T12:11:47.935 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T12:11:47.935 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:47.935 INFO:teuthology.orchestra.run.smithi183.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-21T12:11:47.935 INFO:teuthology.orchestra.run.smithi183.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T12:11:47.935 INFO:teuthology.orchestra.run.smithi183.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T12:11:47.935 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T12:11:47.936 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:47.936 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T12:11:47.936 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:47.936 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T12:11:47.936 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T12:11:47.936 INFO:teuthology.orchestra.run.smithi183.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T12:11:47.936 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:47.936 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:47.936 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:47.936 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T12:11:47.936 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:47.936 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:47.937 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:47.937 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-21T12:11:47.937 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-21T12:11:47.937 INFO:teuthology.orchestra.run.smithi183.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-21T12:11:47.937 INFO:teuthology.orchestra.run.smithi183.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T12:11:47.937 INFO:teuthology.orchestra.run.smithi183.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T12:11:47.937 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:11:47.937 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-21T12:11:48.268 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-selinux-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 32/35 2024-09-21T12:11:48.268 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /sys 2024-09-21T12:11:48.268 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /proc 2024-09-21T12:11:48.268 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /mnt 2024-09-21T12:11:48.268 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /var/tmp 2024-09-21T12:11:48.268 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /home 2024-09-21T12:11:48.268 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /root 2024-09-21T12:11:48.268 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /tmp 2024-09-21T12:11:48.268 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:11:48.313 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-test 2024-09-21T12:11:48.905 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:27 ago on Sat 21 Sep 2024 12:11:21 PM UTC. 2024-09-21T12:11:49.037 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-21T12:11:49.038 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:11:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-09-21T12:11:49.038 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:11:49.038 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-21T12:11:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 46 M 2024-09-21T12:11:49.038 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-09-21T12:11:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 161 k 2024-09-21T12:11:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T12:11:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T12:11:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:11:49.039 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-21T12:11:49.039 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:11:49.039 INFO:teuthology.orchestra.run.smithi183.stdout:Install 4 Packages 2024-09-21T12:11:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:11:49.039 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 46 M 2024-09-21T12:11:49.039 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 201 M 2024-09-21T12:11:49.040 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-21T12:11:49.292 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-radosgw-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 33/35 2024-09-21T12:11:49.307 INFO:teuthology.orchestra.run.smithi183.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.1 MB/s | 304 kB 00:00 2024-09-21T12:11:49.317 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 33/35 2024-09-21T12:11:49.317 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T12:11:49.317 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T12:11:49.317 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T12:11:49.317 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T12:11:49.318 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:11:49.319 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-21T12:11:49.374 INFO:teuthology.orchestra.run.smithi183.stdout:(2/4): libcephsqlite-19.3.0-5103.g5c1cdc3c.el9. 785 kB/s | 161 kB 00:00 2024-09-21T12:11:49.399 INFO:teuthology.orchestra.run.smithi183.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 694 kB/s | 64 kB 00:00 2024-09-21T12:11:49.403 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-21T12:11:49.403 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T12:11:50.583 INFO:teuthology.orchestra.run.smithi183.stdout:(4/4): ceph-test-19.3.0-5103.g5c1cdc3c.el9.x86_ 32 MB/s | 46 MB 00:01 2024-09-21T12:11:50.585 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:11:50.585 INFO:teuthology.orchestra.run.smithi183.stdout:Total 30 MB/s | 46 MB 00:01 2024-09-21T12:11:50.613 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-21T12:11:50.622 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T12:11:50.622 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-base-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/35 2024-09-21T12:11:50.622 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-common-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2/35 2024-09-21T12:11:50.622 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-radosgw-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 3/35 2024-09-21T12:11:50.622 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-selinux-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 4/35 2024-09-21T12:11:50.622 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs2-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 5/35 2024-09-21T12:11:50.622 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libradosstriper1-2:19.3.0-5103.g5c1cdc3c.el9.x86_6 6/35 2024-09-21T12:11:50.622 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librgw2-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 7/35 2024-09-21T12:11:50.622 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-argparse-2:19.3.0-5103.g5c1cdc3c.el9. 8/35 2024-09-21T12:11:50.622 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-common-2:19.3.0-5103.g5c1cdc3c.el9.x8 9/35 2024-09-21T12:11:50.622 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cephfs-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 10/35 2024-09-21T12:11:50.622 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rados-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 11/35 2024-09-21T12:11:50.623 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rbd-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 12/35 2024-09-21T12:11:50.623 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rgw-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 13/35 2024-09-21T12:11:50.623 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-21T12:11:50.623 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-21T12:11:50.623 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-21T12:11:50.623 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-21T12:11:50.623 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-21T12:11:50.623 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-21T12:11:50.623 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-21T12:11:50.623 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-21T12:11:50.623 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-21T12:11:50.623 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-21T12:11:50.623 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-21T12:11:50.623 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-21T12:11:50.624 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-21T12:11:50.625 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-21T12:11:50.625 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-21T12:11:50.625 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-21T12:11:50.625 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-21T12:11:50.625 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-21T12:11:50.625 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 32/35 2024-09-21T12:11:50.625 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-21T12:11:50.625 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 34/35 2024-09-21T12:11:50.633 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-21T12:11:50.633 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-21T12:11:51.011 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-21T12:11:51.011 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-21T12:11:51.540 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T12:11:51.540 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:11:51.540 INFO:teuthology.orchestra.run.smithi114.stdout:Upgraded: 2024-09-21T12:11:51.540 INFO:teuthology.orchestra.run.smithi114.stdout: librados2-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:51.540 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:51.541 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-21T12:11:51.541 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T12:11:51.541 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:51.541 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:51.541 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:51.541 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:51.541 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T12:11:51.541 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T12:11:51.541 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T12:11:51.541 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:51.541 INFO:teuthology.orchestra.run.smithi114.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-21T12:11:51.541 INFO:teuthology.orchestra.run.smithi114.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T12:11:51.541 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T12:11:51.541 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T12:11:51.541 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:51.542 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T12:11:51.542 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:51.542 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T12:11:51.542 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T12:11:51.542 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T12:11:51.542 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:51.542 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:51.542 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:51.542 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T12:11:51.542 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:51.542 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:51.542 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:51.542 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-21T12:11:51.542 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-21T12:11:51.543 INFO:teuthology.orchestra.run.smithi114.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-21T12:11:51.543 INFO:teuthology.orchestra.run.smithi114.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T12:11:51.543 INFO:teuthology.orchestra.run.smithi114.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T12:11:51.543 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:11:51.543 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-21T12:11:51.582 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-21T12:11:51.661 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-21T12:11:51.699 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-21T12:11:51.726 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephsqlite-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 3/4 2024-09-21T12:11:51.926 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-test 2024-09-21T12:11:52.510 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:27 ago on Sat 21 Sep 2024 12:11:25 PM UTC. 2024-09-21T12:11:52.641 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-21T12:11:52.641 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:11:52.641 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-21T12:11:52.641 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:11:52.642 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-21T12:11:52.642 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 46 M 2024-09-21T12:11:52.642 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-21T12:11:52.642 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 161 k 2024-09-21T12:11:52.642 INFO:teuthology.orchestra.run.smithi114.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T12:11:52.642 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T12:11:52.642 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:11:52.642 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-21T12:11:52.642 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:11:52.642 INFO:teuthology.orchestra.run.smithi114.stdout:Install 4 Packages 2024-09-21T12:11:52.642 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:11:52.643 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 46 M 2024-09-21T12:11:52.643 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 201 M 2024-09-21T12:11:52.643 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-21T12:11:52.993 INFO:teuthology.orchestra.run.smithi114.stdout:(1/4): libcephsqlite-19.3.0-5103.g5c1cdc3c.el9. 752 kB/s | 161 kB 00:00 2024-09-21T12:11:53.027 INFO:teuthology.orchestra.run.smithi114.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.2 MB/s | 304 kB 00:00 2024-09-21T12:11:53.052 INFO:teuthology.orchestra.run.smithi114.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-09-21T12:11:54.236 INFO:teuthology.orchestra.run.smithi114.stdout:(4/4): ceph-test-19.3.0-5103.g5c1cdc3c.el9.x86_ 31 MB/s | 46 MB 00:01 2024-09-21T12:11:54.238 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:11:54.238 INFO:teuthology.orchestra.run.smithi114.stdout:Total 29 MB/s | 46 MB 00:01 2024-09-21T12:11:54.266 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-21T12:11:54.287 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-21T12:11:54.287 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-21T12:11:54.670 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-21T12:11:54.671 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-21T12:11:55.233 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-21T12:11:55.313 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-21T12:11:55.352 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-21T12:11:55.379 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephsqlite-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 3/4 2024-09-21T12:11:56.099 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephsqlite-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 3/4 2024-09-21T12:11:56.174 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-test-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 4/4 2024-09-21T12:11:58.413 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-test-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 4/4 2024-09-21T12:11:58.414 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-test-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/4 2024-09-21T12:11:58.414 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephsqlite-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2/4 2024-09-21T12:11:58.414 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-21T12:11:58.701 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-21T12:11:58.701 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:11:58.701 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-21T12:11:58.701 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:58.701 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:11:58.701 INFO:teuthology.orchestra.run.smithi183.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T12:11:58.701 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T12:11:58.701 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:11:58.701 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-21T12:11:58.978 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph 2024-09-21T12:11:59.584 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:38 ago on Sat 21 Sep 2024 12:11:21 PM UTC. 2024-09-21T12:11:59.727 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-21T12:11:59.730 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:11:59.730 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-09-21T12:11:59.731 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:11:59.731 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-21T12:11:59.731 INFO:teuthology.orchestra.run.smithi183.stdout: ceph x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 6.6 k 2024-09-21T12:11:59.731 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-09-21T12:11:59.731 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 2.4 M 2024-09-21T12:11:59.731 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 1.5 M 2024-09-21T12:11:59.731 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core noarch 2:19.3.0-5103.g5c1cdc3c.el9 ceph-noarch 292 k 2024-09-21T12:11:59.732 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 4.8 M 2024-09-21T12:11:59.732 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 17 M 2024-09-21T12:11:59.732 INFO:teuthology.orchestra.run.smithi183.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-21T12:11:59.732 INFO:teuthology.orchestra.run.smithi183.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-21T12:11:59.732 INFO:teuthology.orchestra.run.smithi183.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-21T12:11:59.732 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T12:11:59.732 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T12:11:59.732 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T12:11:59.732 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T12:11:59.732 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T12:11:59.733 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T12:11:59.733 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T12:11:59.733 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-collections 2024-09-21T12:11:59.733 INFO:teuthology.orchestra.run.smithi183.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-21T12:11:59.733 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T12:11:59.733 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T12:11:59.733 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T12:11:59.733 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T12:11:59.733 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T12:11:59.734 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T12:11:59.734 INFO:teuthology.orchestra.run.smithi183.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T12:11:59.734 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T12:11:59.734 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T12:11:59.734 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T12:11:59.734 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T12:11:59.734 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T12:11:59.734 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T12:11:59.734 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T12:11:59.734 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T12:11:59.734 INFO:teuthology.orchestra.run.smithi183.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T12:11:59.735 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T12:11:59.735 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T12:11:59.735 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T12:11:59.735 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T12:11:59.735 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:11:59.735 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-21T12:11:59.735 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:11:59.735 INFO:teuthology.orchestra.run.smithi183.stdout:Install 37 Packages 2024-09-21T12:11:59.735 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:11:59.736 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 31 M 2024-09-21T12:11:59.736 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 109 M 2024-09-21T12:11:59.736 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-21T12:11:59.758 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephsqlite-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 3/4 2024-09-21T12:11:59.844 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-test-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 4/4 2024-09-21T12:12:00.137 INFO:teuthology.orchestra.run.smithi183.stdout:(1/37): ceph-19.3.0-5103.g5c1cdc3c.el9.x86_64.r 53 kB/s | 6.6 kB 00:00 2024-09-21T12:12:00.321 INFO:teuthology.orchestra.run.smithi183.stdout:(2/37): ceph-mgr-19.3.0-5103.g5c1cdc3c.el9.x86_ 5.0 MB/s | 1.5 MB 00:00 2024-09-21T12:12:00.380 INFO:teuthology.orchestra.run.smithi183.stdout:(3/37): ceph-mds-19.3.0-5103.g5c1cdc3c.el9.x86_ 6.5 MB/s | 2.4 MB 00:00 2024-09-21T12:12:00.489 INFO:teuthology.orchestra.run.smithi183.stdout:(4/37): ceph-mgr-modules-core-19.3.0-5103.g5c1c 2.6 MB/s | 292 kB 00:00 2024-09-21T12:12:00.564 INFO:teuthology.orchestra.run.smithi183.stdout:(5/37): ceph-mon-19.3.0-5103.g5c1cdc3c.el9.x86_ 11 MB/s | 4.8 MB 00:00 2024-09-21T12:12:00.906 INFO:teuthology.orchestra.run.smithi183.stdout:(6/37): ceph-osd-19.3.0-5103.g5c1cdc3c.el9.x86_ 29 MB/s | 17 MB 00:00 2024-09-21T12:12:00.931 INFO:teuthology.orchestra.run.smithi183.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 573 kB/s | 253 kB 00:00 2024-09-21T12:12:01.089 INFO:teuthology.orchestra.run.smithi183.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 854 kB/s | 135 kB 00:00 2024-09-21T12:12:01.115 INFO:teuthology.orchestra.run.smithi183.stdout:(9/37): python3-ply-3.11-14.el9.noarch.rpm 510 kB/s | 106 kB 00:00 2024-09-21T12:12:01.148 INFO:teuthology.orchestra.run.smithi183.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 2.1 MB/s | 1.2 MB 00:00 2024-09-21T12:12:01.199 INFO:teuthology.orchestra.run.smithi183.stdout:(11/37): python3-requests-2.25.1-8.el9.noarch.r 1.1 MB/s | 125 kB 00:00 2024-09-21T12:12:01.224 INFO:teuthology.orchestra.run.smithi183.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.9 MB/s | 215 kB 00:00 2024-09-21T12:12:01.332 INFO:teuthology.orchestra.run.smithi183.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 321 kB/s | 35 kB 00:00 2024-09-21T12:12:01.358 INFO:teuthology.orchestra.run.smithi183.stdout:(14/37): lua-5.4.4-4.el9.x86_64.rpm 902 kB/s | 188 kB 00:00 2024-09-21T12:12:01.408 INFO:teuthology.orchestra.run.smithi183.stdout:(15/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-21T12:12:01.425 INFO:teuthology.orchestra.run.smithi183.stdout:(16/37): python3-mako-1.1.4-6.el9.noarch.rpm 762 kB/s | 172 kB 00:00 2024-09-21T12:12:01.442 INFO:teuthology.orchestra.run.smithi183.stdout:(17/37): python3-packaging-20.9-5.el9.noarch.rp 706 kB/s | 77 kB 00:00 2024-09-21T12:12:01.467 INFO:teuthology.orchestra.run.smithi183.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 707 kB/s | 42 kB 00:00 2024-09-21T12:12:01.492 INFO:teuthology.orchestra.run.smithi183.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 332 kB/s | 22 kB 00:00 2024-09-21T12:12:01.543 INFO:teuthology.orchestra.run.smithi183.stdout:(20/37): luarocks-3.9.2-5.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-09-21T12:12:01.568 INFO:teuthology.orchestra.run.smithi183.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 433 kB/s | 43 kB 00:00 2024-09-21T12:12:01.593 INFO:teuthology.orchestra.run.smithi183.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 428 kB/s | 11 kB 00:00 2024-09-21T12:12:01.619 INFO:teuthology.orchestra.run.smithi183.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 706 kB/s | 18 kB 00:00 2024-09-21T12:12:01.644 INFO:teuthology.orchestra.run.smithi183.stdout:(24/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-09-21T12:12:01.669 INFO:teuthology.orchestra.run.smithi183.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-09-21T12:12:01.694 INFO:teuthology.orchestra.run.smithi183.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-21T12:12:01.719 INFO:teuthology.orchestra.run.smithi183.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-21T12:12:01.745 INFO:teuthology.orchestra.run.smithi183.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 920 kB/s | 46 kB 00:00 2024-09-21T12:12:01.778 INFO:teuthology.orchestra.run.smithi183.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-09-21T12:12:01.795 INFO:teuthology.orchestra.run.smithi183.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-21T12:12:01.821 INFO:teuthology.orchestra.run.smithi183.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 392 kB/s | 16 kB 00:00 2024-09-21T12:12:01.846 INFO:teuthology.orchestra.run.smithi183.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-21T12:12:01.863 INFO:teuthology.orchestra.run.smithi183.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-09-21T12:12:01.896 INFO:teuthology.orchestra.run.smithi183.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-21T12:12:01.914 INFO:teuthology.orchestra.run.smithi183.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-21T12:12:01.939 INFO:teuthology.orchestra.run.smithi183.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-09-21T12:12:01.972 INFO:teuthology.orchestra.run.smithi183.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-21T12:12:01.977 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:12:01.977 INFO:teuthology.orchestra.run.smithi183.stdout:Total 14 MB/s | 31 MB 00:02 2024-09-21T12:12:02.129 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-21T12:12:02.165 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-test-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 4/4 2024-09-21T12:12:02.165 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-test-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/4 2024-09-21T12:12:02.165 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephsqlite-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2/4 2024-09-21T12:12:02.165 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-21T12:12:02.171 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-21T12:12:02.171 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-21T12:12:02.466 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-21T12:12:02.466 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:02.466 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-21T12:12:02.467 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:12:02.467 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:12:02.467 INFO:teuthology.orchestra.run.smithi114.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T12:12:02.467 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T12:12:02.467 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:02.467 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-21T12:12:02.637 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-21T12:12:02.637 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-21T12:12:02.734 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph 2024-09-21T12:12:03.324 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:38 ago on Sat 21 Sep 2024 12:11:25 PM UTC. 2024-09-21T12:12:03.463 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-21T12:12:03.466 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:12:03.466 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-21T12:12:03.466 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:12:03.466 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-21T12:12:03.466 INFO:teuthology.orchestra.run.smithi114.stdout: ceph x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 6.6 k 2024-09-21T12:12:03.466 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-21T12:12:03.466 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 2.4 M 2024-09-21T12:12:03.467 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 1.5 M 2024-09-21T12:12:03.467 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core noarch 2:19.3.0-5103.g5c1cdc3c.el9 ceph-noarch 292 k 2024-09-21T12:12:03.467 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 4.8 M 2024-09-21T12:12:03.467 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 17 M 2024-09-21T12:12:03.467 INFO:teuthology.orchestra.run.smithi114.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-21T12:12:03.467 INFO:teuthology.orchestra.run.smithi114.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-21T12:12:03.467 INFO:teuthology.orchestra.run.smithi114.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-21T12:12:03.467 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T12:12:03.467 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T12:12:03.467 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T12:12:03.467 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T12:12:03.467 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T12:12:03.467 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T12:12:03.467 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T12:12:03.467 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections 2024-09-21T12:12:03.468 INFO:teuthology.orchestra.run.smithi114.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-21T12:12:03.468 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T12:12:03.468 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T12:12:03.468 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T12:12:03.468 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T12:12:03.468 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T12:12:03.468 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T12:12:03.468 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T12:12:03.468 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T12:12:03.468 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T12:12:03.468 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T12:12:03.468 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T12:12:03.468 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T12:12:03.468 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T12:12:03.469 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T12:12:03.469 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T12:12:03.469 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T12:12:03.469 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T12:12:03.469 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T12:12:03.469 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T12:12:03.469 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T12:12:03.469 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:03.469 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-21T12:12:03.469 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:12:03.469 INFO:teuthology.orchestra.run.smithi114.stdout:Install 37 Packages 2024-09-21T12:12:03.469 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:03.470 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 31 M 2024-09-21T12:12:03.470 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 109 M 2024-09-21T12:12:03.470 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-21T12:12:03.909 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-21T12:12:03.975 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-21T12:12:04.008 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-21T12:12:04.036 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-21T12:12:04.046 INFO:teuthology.orchestra.run.smithi114.stdout:(1/37): ceph-19.3.0-5103.g5c1cdc3c.el9.x86_64.r 47 kB/s | 6.6 kB 00:00 2024-09-21T12:12:04.109 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-21T12:12:04.160 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-21T12:12:04.192 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-21T12:12:04.213 INFO:teuthology.orchestra.run.smithi114.stdout:(2/37): ceph-mgr-19.3.0-5103.g5c1cdc3c.el9.x86_ 5.0 MB/s | 1.5 MB 00:00 2024-09-21T12:12:04.271 INFO:teuthology.orchestra.run.smithi114.stdout:(3/37): ceph-mds-19.3.0-5103.g5c1cdc3c.el9.x86_ 6.5 MB/s | 2.4 MB 00:00 2024-09-21T12:12:04.286 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-21T12:12:04.338 INFO:teuthology.orchestra.run.smithi114.stdout:(4/37): ceph-mgr-modules-core-19.3.0-5103.g5c1c 4.3 MB/s | 292 kB 00:00 2024-09-21T12:12:04.372 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-21T12:12:04.412 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-21T12:12:04.447 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-21T12:12:04.477 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-21T12:12:04.480 INFO:teuthology.orchestra.run.smithi114.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.7 MB/s | 253 kB 00:00 2024-09-21T12:12:04.511 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-21T12:12:04.545 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-21T12:12:04.585 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-21T12:12:04.589 INFO:teuthology.orchestra.run.smithi114.stdout:(6/37): python3-cryptography-36.0.1-4.el9.x86_6 11 MB/s | 1.2 MB 00:00 2024-09-21T12:12:04.628 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-21T12:12:04.663 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-21T12:12:04.712 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-21T12:12:04.752 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-21T12:12:04.756 INFO:teuthology.orchestra.run.smithi114.stdout:(7/37): ceph-osd-19.3.0-5103.g5c1cdc3c.el9.x86_ 32 MB/s | 17 MB 00:00 2024-09-21T12:12:04.781 INFO:teuthology.orchestra.run.smithi114.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 555 kB/s | 106 kB 00:00 2024-09-21T12:12:04.826 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-21T12:12:04.865 INFO:teuthology.orchestra.run.smithi114.stdout:(9/37): ceph-mon-19.3.0-5103.g5c1cdc3c.el9.x86_ 5.9 MB/s | 4.8 MB 00:00 2024-09-21T12:12:04.890 INFO:teuthology.orchestra.run.smithi114.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 1.1 MB/s | 125 kB 00:00 2024-09-21T12:12:04.932 INFO:teuthology.orchestra.run.smithi114.stdout:(11/37): python3-pycparser-2.20-6.el9.noarch.rp 770 kB/s | 135 kB 00:00 2024-09-21T12:12:04.965 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-21T12:12:04.966 INFO:teuthology.orchestra.run.smithi114.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 2.5 MB/s | 188 kB 00:00 2024-09-21T12:12:05.008 INFO:teuthology.orchestra.run.smithi114.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-09-21T12:12:05.024 INFO:teuthology.orchestra.run.smithi114.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 592 kB/s | 35 kB 00:00 2024-09-21T12:12:05.044 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-21T12:12:05.058 INFO:teuthology.orchestra.run.smithi114.stdout:(15/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.1 MB/s | 215 kB 00:00 2024-09-21T12:12:05.083 INFO:teuthology.orchestra.run.smithi114.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 1.0 MB/s | 77 kB 00:00 2024-09-21T12:12:05.097 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-21T12:12:05.100 INFO:teuthology.orchestra.run.smithi114.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 675 kB/s | 51 kB 00:00 2024-09-21T12:12:05.125 INFO:teuthology.orchestra.run.smithi114.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 620 kB/s | 42 kB 00:00 2024-09-21T12:12:05.135 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-21T12:12:05.159 INFO:teuthology.orchestra.run.smithi114.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-09-21T12:12:05.184 INFO:teuthology.orchestra.run.smithi114.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 7.2 MB/s | 172 kB 00:00 2024-09-21T12:12:05.192 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-21T12:12:05.201 INFO:teuthology.orchestra.run.smithi114.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 589 kB/s | 43 kB 00:00 2024-09-21T12:12:05.227 INFO:teuthology.orchestra.run.smithi114.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 427 kB/s | 11 kB 00:00 2024-09-21T12:12:05.243 INFO:teuthology.orchestra.run.smithi114.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 1.0 MB/s | 18 kB 00:00 2024-09-21T12:12:05.248 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-21T12:12:05.277 INFO:teuthology.orchestra.run.smithi114.stdout:(24/37): python3-cherrypy-18.6.1-2.el9.noarch.r 3.8 MB/s | 358 kB 00:00 2024-09-21T12:12:05.294 INFO:teuthology.orchestra.run.smithi114.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-09-21T12:12:05.311 INFO:teuthology.orchestra.run.smithi114.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 579 kB/s | 19 kB 00:00 2024-09-21T12:12:05.328 INFO:teuthology.orchestra.run.smithi114.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 583 kB/s | 20 kB 00:00 2024-09-21T12:12:05.345 INFO:teuthology.orchestra.run.smithi114.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 1.3 MB/s | 46 kB 00:00 2024-09-21T12:12:05.370 INFO:teuthology.orchestra.run.smithi114.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-09-21T12:12:05.395 INFO:teuthology.orchestra.run.smithi114.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 660 kB/s | 16 kB 00:00 2024-09-21T12:12:05.421 INFO:teuthology.orchestra.run.smithi114.stdout:(31/37): lua-devel-5.4.4-4.el9.x86_64.rpm 66 kB/s | 22 kB 00:00 2024-09-21T12:12:05.437 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-21T12:12:05.455 INFO:teuthology.orchestra.run.smithi114.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-09-21T12:12:05.481 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-21T12:12:05.496 INFO:teuthology.orchestra.run.smithi114.stdout:(33/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-21T12:12:05.521 INFO:teuthology.orchestra.run.smithi114.stdout:(34/37): python3-tempora-5.0.0-2.el9.noarch.rpm 356 kB/s | 36 kB 00:00 2024-09-21T12:12:05.547 INFO:teuthology.orchestra.run.smithi114.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 799 kB/s | 20 kB 00:00 2024-09-21T12:12:05.559 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-21T12:12:05.572 INFO:teuthology.orchestra.run.smithi114.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-09-21T12:12:05.597 INFO:teuthology.orchestra.run.smithi114.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-21T12:12:05.602 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:12:05.602 INFO:teuthology.orchestra.run.smithi114.stdout:Total 14 MB/s | 31 MB 00:02 2024-09-21T12:12:05.648 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-21T12:12:05.706 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-21T12:12:05.750 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-21T12:12:05.789 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-21T12:12:05.789 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-21T12:12:05.843 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-21T12:12:06.003 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-5103.g5c1cdc3c.el9. 32/37 2024-09-21T12:12:06.031 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 33/37 2024-09-21T12:12:06.058 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 33/37 2024-09-21T12:12:06.058 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T12:12:06.059 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T12:12:06.059 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T12:12:06.059 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T12:12:06.059 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:06.236 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-21T12:12:06.237 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-21T12:12:07.506 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-21T12:12:07.524 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-osd-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 34/37 2024-09-21T12:12:07.551 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-osd-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 34/37 2024-09-21T12:12:07.551 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T12:12:07.552 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T12:12:07.552 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T12:12:07.552 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T12:12:07.552 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:07.567 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-21T12:12:07.600 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-21T12:12:07.627 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-21T12:12:07.699 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-21T12:12:07.743 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-21T12:12:07.776 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-21T12:12:07.877 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-21T12:12:07.963 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-21T12:12:07.999 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mon-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 35/37 2024-09-21T12:12:08.003 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-21T12:12:08.024 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mon-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 35/37 2024-09-21T12:12:08.024 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T12:12:08.024 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T12:12:08.024 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T12:12:08.024 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T12:12:08.024 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:08.038 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-21T12:12:08.076 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-21T12:12:08.111 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-21T12:12:08.144 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-21T12:12:08.185 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-21T12:12:08.228 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-21T12:12:08.259 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mds-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 36/37 2024-09-21T12:12:08.262 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-21T12:12:08.282 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mds-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 36/37 2024-09-21T12:12:08.282 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T12:12:08.282 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T12:12:08.283 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T12:12:08.283 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T12:12:08.283 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:08.311 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-21T12:12:08.343 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-21T12:12:08.347 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 37/37 2024-09-21T12:12:08.425 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-21T12:12:08.573 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-21T12:12:08.651 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-21T12:12:08.704 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-21T12:12:08.743 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-21T12:12:08.800 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-21T12:12:08.855 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-21T12:12:09.044 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-21T12:12:09.088 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-21T12:12:09.167 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-21T12:12:09.255 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-21T12:12:09.299 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-21T12:12:09.434 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-21T12:12:09.601 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-5103.g5c1cdc3c.el9. 32/37 2024-09-21T12:12:09.631 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 33/37 2024-09-21T12:12:09.657 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 33/37 2024-09-21T12:12:09.657 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T12:12:09.657 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T12:12:09.657 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T12:12:09.657 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T12:12:09.657 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:10.120 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 37/37 2024-09-21T12:12:10.120 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/37 2024-09-21T12:12:10.120 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mds-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2/37 2024-09-21T12:12:10.120 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 3/37 2024-09-21T12:12:10.120 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mon-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 4/37 2024-09-21T12:12:10.120 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-osd-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 5/37 2024-09-21T12:12:10.120 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-5103.g5c1cdc3c.el9. 6/37 2024-09-21T12:12:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-21T12:12:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-21T12:12:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-21T12:12:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-21T12:12:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-21T12:12:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-21T12:12:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-21T12:12:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-21T12:12:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-21T12:12:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-21T12:12:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-21T12:12:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-21T12:12:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-21T12:12:10.122 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-21T12:12:10.122 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-21T12:12:10.122 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-21T12:12:10.122 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-21T12:12:10.122 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-21T12:12:10.122 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-21T12:12:10.122 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-21T12:12:10.122 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-21T12:12:10.122 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-21T12:12:10.122 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-21T12:12:10.122 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-21T12:12:10.123 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-21T12:12:10.123 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-21T12:12:10.123 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-21T12:12:10.123 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-21T12:12:10.124 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-21T12:12:10.124 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-21T12:12:11.046 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-21T12:12:11.046 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:11.047 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-21T12:12:11.047 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:12:11.047 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:12:11.047 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:12:11.047 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core-2:19.3.0-5103.g5c1cdc3c.el9.noarch 2024-09-21T12:12:11.047 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:12:11.047 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:12:11.047 INFO:teuthology.orchestra.run.smithi183.stdout: lua-5.4.4-4.el9.x86_64 2024-09-21T12:12:11.048 INFO:teuthology.orchestra.run.smithi183.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-21T12:12:11.048 INFO:teuthology.orchestra.run.smithi183.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-21T12:12:11.048 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T12:12:11.048 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T12:12:11.048 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T12:12:11.048 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T12:12:11.048 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T12:12:11.048 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T12:12:11.048 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T12:12:11.048 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T12:12:11.049 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T12:12:11.049 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T12:12:11.049 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T12:12:11.049 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T12:12:11.049 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T12:12:11.049 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T12:12:11.049 INFO:teuthology.orchestra.run.smithi183.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T12:12:11.049 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T12:12:11.049 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T12:12:11.049 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T12:12:11.049 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T12:12:11.049 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T12:12:11.049 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T12:12:11.049 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T12:12:11.050 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T12:12:11.050 INFO:teuthology.orchestra.run.smithi183.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T12:12:11.050 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T12:12:11.050 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T12:12:11.050 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T12:12:11.050 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T12:12:11.050 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:11.050 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-21T12:12:11.131 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-osd-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 34/37 2024-09-21T12:12:11.159 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-osd-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 34/37 2024-09-21T12:12:11.159 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T12:12:11.159 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T12:12:11.159 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T12:12:11.159 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T12:12:11.159 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:11.445 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-base 2024-09-21T12:12:11.607 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mon-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 35/37 2024-09-21T12:12:11.630 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mon-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 35/37 2024-09-21T12:12:11.631 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T12:12:11.631 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T12:12:11.631 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T12:12:11.631 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T12:12:11.631 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:11.858 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mds-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 36/37 2024-09-21T12:12:11.881 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mds-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 36/37 2024-09-21T12:12:11.881 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T12:12:11.881 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T12:12:11.881 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T12:12:11.882 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T12:12:11.882 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:11.945 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 37/37 2024-09-21T12:12:12.037 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:51 ago on Sat 21 Sep 2024 12:11:21 PM UTC. 2024-09-21T12:12:12.127 INFO:teuthology.orchestra.run.smithi183.stdout:Package ceph-base-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 is already installed. 2024-09-21T12:12:12.169 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-21T12:12:12.170 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-09-21T12:12:12.170 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-21T12:12:12.239 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install cephadm 2024-09-21T12:12:12.821 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:51 ago on Sat 21 Sep 2024 12:11:21 PM UTC. 2024-09-21T12:12:12.952 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-21T12:12:12.953 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:12:12.953 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-09-21T12:12:12.953 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:12:12.953 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-21T12:12:12.953 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm noarch 2:19.3.0-5103.g5c1cdc3c.el9 ceph-noarch 784 k 2024-09-21T12:12:12.953 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:12.953 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-21T12:12:12.953 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:12:12.953 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-09-21T12:12:12.953 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:12.954 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 784 k 2024-09-21T12:12:12.954 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 791 k 2024-09-21T12:12:12.954 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-21T12:12:13.234 INFO:teuthology.orchestra.run.smithi183.stdout:cephadm-19.3.0-5103.g5c1cdc3c.el9.noarch.rpm 2.7 MB/s | 784 kB 00:00 2024-09-21T12:12:13.234 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:12:13.235 INFO:teuthology.orchestra.run.smithi183.stdout:Total 2.7 MB/s | 784 kB 00:00 2024-09-21T12:12:13.235 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-21T12:12:13.240 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-21T12:12:13.240 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-21T12:12:13.246 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-21T12:12:13.246 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-21T12:12:13.321 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-21T12:12:13.658 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 37/37 2024-09-21T12:12:13.658 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/37 2024-09-21T12:12:13.658 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mds-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2/37 2024-09-21T12:12:13.658 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 3/37 2024-09-21T12:12:13.658 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mon-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 4/37 2024-09-21T12:12:13.658 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-osd-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 5/37 2024-09-21T12:12:13.658 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-5103.g5c1cdc3c.el9. 6/37 2024-09-21T12:12:13.658 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-21T12:12:13.658 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-21T12:12:13.658 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-21T12:12:13.658 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-21T12:12:13.659 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-21T12:12:13.659 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-21T12:12:13.659 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-21T12:12:13.659 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-21T12:12:13.659 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-21T12:12:13.659 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-21T12:12:13.659 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-21T12:12:13.659 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-21T12:12:13.659 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-21T12:12:13.659 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-21T12:12:13.659 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-21T12:12:13.659 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-21T12:12:13.659 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-21T12:12:13.659 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-21T12:12:13.660 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-21T12:12:13.660 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-21T12:12:13.660 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-21T12:12:13.660 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-21T12:12:13.660 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-21T12:12:13.660 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-21T12:12:13.660 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-21T12:12:13.660 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-21T12:12:13.660 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-21T12:12:13.660 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-21T12:12:13.660 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-21T12:12:13.660 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-21T12:12:13.746 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephadm-2:19.3.0-5103.g5c1cdc3c.el9.noarch 1/1 2024-09-21T12:12:13.838 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephadm-2:19.3.0-5103.g5c1cdc3c.el9.noarch 1/1 2024-09-21T12:12:14.207 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephadm-2:19.3.0-5103.g5c1cdc3c.el9.noarch 1/1 2024-09-21T12:12:14.457 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephadm-2:19.3.0-5103.g5c1cdc3c.el9.noarch 1/1 2024-09-21T12:12:14.457 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:14.457 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-21T12:12:14.457 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm-2:19.3.0-5103.g5c1cdc3c.el9.noarch 2024-09-21T12:12:14.458 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:14.458 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-21T12:12:14.555 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-21T12:12:14.555 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:14.555 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-21T12:12:14.555 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:12:14.555 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:12:14.555 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:12:14.555 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core-2:19.3.0-5103.g5c1cdc3c.el9.noarch 2024-09-21T12:12:14.555 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:12:14.555 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:12:14.556 INFO:teuthology.orchestra.run.smithi114.stdout: lua-5.4.4-4.el9.x86_64 2024-09-21T12:12:14.556 INFO:teuthology.orchestra.run.smithi114.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-21T12:12:14.556 INFO:teuthology.orchestra.run.smithi114.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-21T12:12:14.556 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T12:12:14.556 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T12:12:14.556 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T12:12:14.556 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T12:12:14.556 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T12:12:14.556 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T12:12:14.556 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T12:12:14.556 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T12:12:14.556 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T12:12:14.556 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T12:12:14.556 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T12:12:14.557 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T12:12:14.557 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T12:12:14.557 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T12:12:14.557 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T12:12:14.557 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T12:12:14.557 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T12:12:14.557 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T12:12:14.557 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T12:12:14.557 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T12:12:14.558 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T12:12:14.558 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T12:12:14.558 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T12:12:14.558 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T12:12:14.558 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T12:12:14.558 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T12:12:14.558 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T12:12:14.558 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T12:12:14.558 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:14.558 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-21T12:12:14.680 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-immutable-object-cache 2024-09-21T12:12:14.923 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-base 2024-09-21T12:12:15.267 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:54 ago on Sat 21 Sep 2024 12:11:21 PM UTC. 2024-09-21T12:12:15.401 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-21T12:12:15.402 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:12:15.402 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-09-21T12:12:15.402 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:12:15.402 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-21T12:12:15.402 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 145 k 2024-09-21T12:12:15.402 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:15.402 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-21T12:12:15.402 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:12:15.403 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-09-21T12:12:15.403 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:15.403 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 145 k 2024-09-21T12:12:15.403 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 432 k 2024-09-21T12:12:15.403 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-21T12:12:15.507 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:50 ago on Sat 21 Sep 2024 12:11:25 PM UTC. 2024-09-21T12:12:15.596 INFO:teuthology.orchestra.run.smithi114.stdout:Package ceph-base-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 is already installed. 2024-09-21T12:12:15.631 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-immutable-object-cache-19.3.0-5103.g5c1cdc 634 kB/s | 145 kB 00:00 2024-09-21T12:12:15.632 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:12:15.632 INFO:teuthology.orchestra.run.smithi183.stdout:Total 630 kB/s | 145 kB 00:00 2024-09-21T12:12:15.632 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-21T12:12:15.637 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-21T12:12:15.639 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-21T12:12:15.639 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-21T12:12:15.640 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-21T12:12:15.640 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-21T12:12:15.697 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-21T12:12:15.697 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-21T12:12:15.710 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install cephadm 2024-09-21T12:12:15.839 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-21T12:12:15.899 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-5103.g5c1cdc3c. 1/1 2024-09-21T12:12:15.924 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-5103.g5c1cdc3c. 1/1 2024-09-21T12:12:15.924 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T12:12:15.925 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T12:12:15.925 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:16.288 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:51 ago on Sat 21 Sep 2024 12:11:25 PM UTC. 2024-09-21T12:12:16.419 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-21T12:12:16.420 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:12:16.420 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-21T12:12:16.420 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:12:16.420 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-21T12:12:16.421 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm noarch 2:19.3.0-5103.g5c1cdc3c.el9 ceph-noarch 784 k 2024-09-21T12:12:16.421 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:16.421 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-21T12:12:16.421 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:12:16.421 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-21T12:12:16.421 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:16.421 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 784 k 2024-09-21T12:12:16.421 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 791 k 2024-09-21T12:12:16.421 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-21T12:12:16.707 INFO:teuthology.orchestra.run.smithi114.stdout:cephadm-19.3.0-5103.g5c1cdc3c.el9.noarch.rpm 2.7 MB/s | 784 kB 00:00 2024-09-21T12:12:16.707 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:12:16.708 INFO:teuthology.orchestra.run.smithi114.stdout:Total 2.7 MB/s | 784 kB 00:00 2024-09-21T12:12:16.708 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-21T12:12:16.713 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-21T12:12:16.713 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-21T12:12:16.719 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-21T12:12:16.719 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-21T12:12:16.796 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-21T12:12:16.911 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-5103.g5c1cdc3c. 1/1 2024-09-21T12:12:16.912 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:16.912 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-21T12:12:16.912 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:12:16.912 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:16.912 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-21T12:12:17.155 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr 2024-09-21T12:12:17.302 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephadm-2:19.3.0-5103.g5c1cdc3c.el9.noarch 1/1 2024-09-21T12:12:17.402 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephadm-2:19.3.0-5103.g5c1cdc3c.el9.noarch 1/1 2024-09-21T12:12:17.757 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephadm-2:19.3.0-5103.g5c1cdc3c.el9.noarch 1/1 2024-09-21T12:12:17.759 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:56 ago on Sat 21 Sep 2024 12:11:21 PM UTC. 2024-09-21T12:12:17.848 INFO:teuthology.orchestra.run.smithi183.stdout:Package ceph-mgr-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 is already installed. 2024-09-21T12:12:17.890 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-21T12:12:17.891 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-09-21T12:12:17.892 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-21T12:12:17.967 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-dashboard 2024-09-21T12:12:17.999 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephadm-2:19.3.0-5103.g5c1cdc3c.el9.noarch 1/1 2024-09-21T12:12:17.999 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:17.999 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-21T12:12:17.999 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm-2:19.3.0-5103.g5c1cdc3c.el9.noarch 2024-09-21T12:12:17.999 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:17.999 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-21T12:12:18.204 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-immutable-object-cache 2024-09-21T12:12:18.551 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:57 ago on Sat 21 Sep 2024 12:11:21 PM UTC. 2024-09-21T12:12:18.683 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-21T12:12:18.684 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:12:18.684 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-09-21T12:12:18.684 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:12:18.685 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-21T12:12:18.685 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard noarch 2:19.3.0-5103.g5c1cdc3c.el9 ceph-noarch 14 M 2024-09-21T12:12:18.685 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-09-21T12:12:18.685 INFO:teuthology.orchestra.run.smithi183.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-21T12:12:18.685 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards noarch 2:19.3.0-5103.g5c1cdc3c.el9 ceph-noarch 34 k 2024-09-21T12:12:18.685 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts noarch 2:19.3.0-5103.g5c1cdc3c.el9 ceph-noarch 16 k 2024-09-21T12:12:18.685 INFO:teuthology.orchestra.run.smithi183.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-21T12:12:18.685 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-21T12:12:18.685 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-21T12:12:18.685 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-21T12:12:18.685 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-21T12:12:18.685 INFO:teuthology.orchestra.run.smithi183.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-21T12:12:18.685 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T12:12:18.686 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T12:12:18.686 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:18.686 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-21T12:12:18.686 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:12:18.686 INFO:teuthology.orchestra.run.smithi183.stdout:Install 12 Packages 2024-09-21T12:12:18.686 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:18.686 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 19 M 2024-09-21T12:12:18.687 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 159 M 2024-09-21T12:12:18.687 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-21T12:12:18.786 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:53 ago on Sat 21 Sep 2024 12:11:25 PM UTC. 2024-09-21T12:12:18.923 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-21T12:12:18.924 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:12:18.924 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-09-21T12:12:18.924 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:12:18.924 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-21T12:12:18.924 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 145 k 2024-09-21T12:12:18.924 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:18.925 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-21T12:12:18.925 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:12:18.925 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-21T12:12:18.925 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:18.925 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 145 k 2024-09-21T12:12:18.925 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 432 k 2024-09-21T12:12:18.925 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-21T12:12:19.138 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-immutable-object-cache-19.3.0-5103.g5c1cdc 681 kB/s | 145 kB 00:00 2024-09-21T12:12:19.139 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:12:19.139 INFO:teuthology.orchestra.run.smithi114.stdout:Total 676 kB/s | 145 kB 00:00 2024-09-21T12:12:19.139 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-21T12:12:19.146 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-21T12:12:19.147 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-21T12:12:19.204 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-21T12:12:19.205 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-21T12:12:19.348 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-21T12:12:19.405 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-5103.g5c1cdc3c. 1/1 2024-09-21T12:12:19.431 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-5103.g5c1cdc3c. 1/1 2024-09-21T12:12:19.431 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T12:12:19.431 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T12:12:19.431 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:19.569 INFO:teuthology.orchestra.run.smithi183.stdout:(1/12): ceph-prometheus-alerts-19.3.0-5103.g5c1 114 kB/s | 16 kB 00:00 2024-09-21T12:12:19.594 INFO:teuthology.orchestra.run.smithi183.stdout:(2/12): ceph-grafana-dashboards-19.3.0-5103.g5c 201 kB/s | 34 kB 00:00 2024-09-21T12:12:20.028 INFO:teuthology.orchestra.run.smithi183.stdout:(3/12): ceph-mgr-dashboard-19.3.0-5103.g5c1cdc3 23 MB/s | 14 MB 00:00 2024-09-21T12:12:20.236 INFO:teuthology.orchestra.run.smithi183.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 420 kB/s | 269 kB 00:00 2024-09-21T12:12:20.286 INFO:teuthology.orchestra.run.smithi183.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-21T12:12:20.311 INFO:teuthology.orchestra.run.smithi183.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 780 kB/s | 19 kB 00:00 2024-09-21T12:12:20.378 INFO:teuthology.orchestra.run.smithi183.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 31 MB/s | 2.0 MB 00:00 2024-09-21T12:12:20.404 INFO:teuthology.orchestra.run.smithi183.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.6 MB/s | 144 kB 00:00 2024-09-21T12:12:20.429 INFO:teuthology.orchestra.run.smithi183.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-21T12:12:20.449 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-5103.g5c1cdc3c. 1/1 2024-09-21T12:12:20.449 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:20.449 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-21T12:12:20.449 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:12:20.449 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:20.449 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-21T12:12:20.471 INFO:teuthology.orchestra.run.smithi183.stdout:(10/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.1 MB/s | 1.0 MB 00:00 2024-09-21T12:12:20.497 INFO:teuthology.orchestra.run.smithi183.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-09-21T12:12:20.644 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr 2024-09-21T12:12:20.838 INFO:teuthology.orchestra.run.smithi183.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.0 MB/s | 863 kB 00:00 2024-09-21T12:12:20.842 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:12:20.843 INFO:teuthology.orchestra.run.smithi183.stdout:Total 8.9 MB/s | 19 MB 00:02 2024-09-21T12:12:20.895 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-21T12:12:20.914 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-21T12:12:20.915 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-21T12:12:21.199 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-21T12:12:21.199 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-21T12:12:21.236 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:56 ago on Sat 21 Sep 2024 12:11:25 PM UTC. 2024-09-21T12:12:21.325 INFO:teuthology.orchestra.run.smithi114.stdout:Package ceph-mgr-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 is already installed. 2024-09-21T12:12:21.368 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-21T12:12:21.369 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-21T12:12:21.369 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-21T12:12:21.439 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-dashboard 2024-09-21T12:12:21.801 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-21T12:12:21.889 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-21T12:12:21.932 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-21T12:12:21.979 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-21T12:12:22.010 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-21T12:12:22.014 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:57 ago on Sat 21 Sep 2024 12:11:25 PM UTC. 2024-09-21T12:12:22.100 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-21T12:12:22.146 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-21T12:12:22.147 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:12:22.147 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-21T12:12:22.148 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:12:22.148 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-21T12:12:22.148 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard noarch 2:19.3.0-5103.g5c1cdc3c.el9 ceph-noarch 14 M 2024-09-21T12:12:22.148 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-21T12:12:22.148 INFO:teuthology.orchestra.run.smithi114.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-21T12:12:22.148 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards noarch 2:19.3.0-5103.g5c1cdc3c.el9 ceph-noarch 34 k 2024-09-21T12:12:22.148 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts noarch 2:19.3.0-5103.g5c1cdc3c.el9 ceph-noarch 16 k 2024-09-21T12:12:22.148 INFO:teuthology.orchestra.run.smithi114.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-21T12:12:22.148 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-21T12:12:22.148 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-21T12:12:22.148 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-21T12:12:22.149 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-21T12:12:22.149 INFO:teuthology.orchestra.run.smithi114.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-21T12:12:22.149 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T12:12:22.149 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T12:12:22.149 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:22.149 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-21T12:12:22.149 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:12:22.149 INFO:teuthology.orchestra.run.smithi114.stdout:Install 12 Packages 2024-09-21T12:12:22.149 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:22.150 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 19 M 2024-09-21T12:12:22.150 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 159 M 2024-09-21T12:12:22.150 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-21T12:12:22.243 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-21T12:12:22.347 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-21T12:12:22.389 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-21T12:12:22.418 INFO:teuthology.orchestra.run.smithi114.stdout:(1/12): ceph-grafana-dashboards-19.3.0-5103.g5c 237 kB/s | 34 kB 00:00 2024-09-21T12:12:22.426 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-21T12:12:22.443 INFO:teuthology.orchestra.run.smithi114.stdout:(2/12): ceph-prometheus-alerts-19.3.0-5103.g5c1 97 kB/s | 16 kB 00:00 2024-09-21T12:12:22.459 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-5103.g5c1cdc3c.el9 10/12 2024-09-21T12:12:22.943 INFO:teuthology.orchestra.run.smithi114.stdout:(3/12): ceph-mgr-dashboard-19.3.0-5103.g5c1cdc3 21 MB/s | 14 MB 00:00 2024-09-21T12:12:22.968 INFO:teuthology.orchestra.run.smithi114.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 513 kB/s | 269 kB 00:00 2024-09-21T12:12:23.018 INFO:teuthology.orchestra.run.smithi114.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-21T12:12:23.044 INFO:teuthology.orchestra.run.smithi114.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 775 kB/s | 19 kB 00:00 2024-09-21T12:12:23.111 INFO:teuthology.orchestra.run.smithi114.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 31 MB/s | 2.0 MB 00:00 2024-09-21T12:12:23.136 INFO:teuthology.orchestra.run.smithi114.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.7 MB/s | 144 kB 00:00 2024-09-21T12:12:23.161 INFO:teuthology.orchestra.run.smithi114.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-21T12:12:23.195 INFO:teuthology.orchestra.run.smithi114.stdout:(10/12): protobuf-compiler-3.14.0-13.el9.x86_64 3.4 MB/s | 863 kB 00:00 2024-09-21T12:12:23.229 INFO:teuthology.orchestra.run.smithi114.stdout:(11/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.2 MB/s | 1.0 MB 00:00 2024-09-21T12:12:23.262 INFO:teuthology.orchestra.run.smithi114.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.8 MB/s | 188 kB 00:00 2024-09-21T12:12:23.265 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:12:23.265 INFO:teuthology.orchestra.run.smithi114.stdout:Total 17 MB/s | 19 MB 00:01 2024-09-21T12:12:23.316 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-21T12:12:23.334 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-21T12:12:23.334 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-21T12:12:23.607 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-21T12:12:23.608 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-21T12:12:24.190 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-21T12:12:24.271 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-21T12:12:24.314 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-21T12:12:24.361 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-21T12:12:24.392 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-21T12:12:24.484 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-21T12:12:24.616 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-21T12:12:24.701 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-21T12:12:24.746 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-21T12:12:24.775 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-21T12:12:24.807 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-5103.g5c1cdc3c.el9 10/12 2024-09-21T12:12:25.651 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-5103.g5c1cdc3c.el 11/12 2024-09-21T12:12:25.722 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-5103.g5c1cdc3c.el9.noa 12/12 2024-09-21T12:12:27.497 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-5103.g5c1cdc3c.el9.noa 12/12 2024-09-21T12:12:27.497 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-5103.g5c1cdc3c.el 1/12 2024-09-21T12:12:27.497 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-5103.g5c1cdc3c.el9.noa 2/12 2024-09-21T12:12:27.497 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-5103.g5c1cdc3c.el9 3/12 2024-09-21T12:12:27.497 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-21T12:12:27.497 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-21T12:12:27.497 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-21T12:12:27.497 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-21T12:12:27.497 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-21T12:12:27.497 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-21T12:12:27.498 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-21T12:12:27.498 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-21T12:12:27.813 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-21T12:12:27.813 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:27.813 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-21T12:12:27.813 INFO:teuthology.orchestra.run.smithi183.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-21T12:12:27.813 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards-2:19.3.0-5103.g5c1cdc3c.el9.noarch 2024-09-21T12:12:27.813 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard-2:19.3.0-5103.g5c1cdc3c.el9.noarch 2024-09-21T12:12:27.813 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts-2:19.3.0-5103.g5c1cdc3c.el9.noarch 2024-09-21T12:12:27.813 INFO:teuthology.orchestra.run.smithi183.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-21T12:12:27.814 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-21T12:12:27.814 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-21T12:12:27.814 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-21T12:12:27.814 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-21T12:12:27.814 INFO:teuthology.orchestra.run.smithi183.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-21T12:12:27.814 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T12:12:27.814 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T12:12:27.814 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:27.814 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-21T12:12:27.964 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-5103.g5c1cdc3c.el 11/12 2024-09-21T12:12:28.019 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-5103.g5c1cdc3c.el9.noa 12/12 2024-09-21T12:12:28.082 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-21T12:12:28.664 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:07 ago on Sat 21 Sep 2024 12:11:21 PM UTC. 2024-09-21T12:12:28.797 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-21T12:12:28.798 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================= 2024-09-21T12:12:28.798 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-09-21T12:12:28.798 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================= 2024-09-21T12:12:28.798 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-21T12:12:28.798 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-5103.g5c1cdc3c.el9 ceph-noarch 7.4 M 2024-09-21T12:12:28.798 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-09-21T12:12:28.798 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T12:12:28.798 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T12:12:28.799 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T12:12:28.799 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T12:12:28.799 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T12:12:28.799 INFO:teuthology.orchestra.run.smithi183.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T12:12:28.799 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T12:12:28.799 INFO:teuthology.orchestra.run.smithi183.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T12:12:28.799 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T12:12:28.799 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T12:12:28.799 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T12:12:28.799 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:28.799 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-21T12:12:28.799 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================= 2024-09-21T12:12:28.799 INFO:teuthology.orchestra.run.smithi183.stdout:Install 12 Packages 2024-09-21T12:12:28.799 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:28.800 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 38 M 2024-09-21T12:12:28.801 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 212 M 2024-09-21T12:12:28.802 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-21T12:12:29.062 INFO:teuthology.orchestra.run.smithi183.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-21T12:12:29.095 INFO:teuthology.orchestra.run.smithi183.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.8 MB/s | 801 kB 00:00 2024-09-21T12:12:29.154 INFO:teuthology.orchestra.run.smithi183.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 357 kB/s | 33 kB 00:00 2024-09-21T12:12:29.196 INFO:teuthology.orchestra.run.smithi183.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 437 kB/s | 18 kB 00:00 2024-09-21T12:12:29.238 INFO:teuthology.orchestra.run.smithi183.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 922 kB/s | 38 kB 00:00 2024-09-21T12:12:29.372 INFO:teuthology.orchestra.run.smithi183.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-51 15 MB/s | 7.4 MB 00:00 2024-09-21T12:12:29.505 INFO:teuthology.orchestra.run.smithi183.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.8 MB/s | 245 kB 00:00 2024-09-21T12:12:29.797 INFO:teuthology.orchestra.run.smithi183.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.3 MB/s | 3.0 MB 00:00 2024-09-21T12:12:29.889 INFO:teuthology.orchestra.run.smithi183.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.7 MB/s | 250 kB 00:00 2024-09-21T12:12:29.976 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-5103.g5c1cdc3c.el9.noa 12/12 2024-09-21T12:12:29.976 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-5103.g5c1cdc3c.el 1/12 2024-09-21T12:12:29.976 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-5103.g5c1cdc3c.el9.noa 2/12 2024-09-21T12:12:29.977 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-5103.g5c1cdc3c.el9 3/12 2024-09-21T12:12:29.977 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-21T12:12:29.977 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-21T12:12:29.977 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-21T12:12:29.977 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-21T12:12:29.977 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-21T12:12:29.977 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-21T12:12:29.977 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-21T12:12:29.977 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-21T12:12:30.297 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-21T12:12:30.298 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:30.298 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-21T12:12:30.298 INFO:teuthology.orchestra.run.smithi114.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-21T12:12:30.298 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards-2:19.3.0-5103.g5c1cdc3c.el9.noarch 2024-09-21T12:12:30.298 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard-2:19.3.0-5103.g5c1cdc3c.el9.noarch 2024-09-21T12:12:30.298 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts-2:19.3.0-5103.g5c1cdc3c.el9.noarch 2024-09-21T12:12:30.298 INFO:teuthology.orchestra.run.smithi114.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-21T12:12:30.298 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-21T12:12:30.298 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-21T12:12:30.298 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-21T12:12:30.298 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-21T12:12:30.298 INFO:teuthology.orchestra.run.smithi114.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-21T12:12:30.299 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T12:12:30.299 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T12:12:30.299 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:30.299 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-21T12:12:30.456 INFO:teuthology.orchestra.run.smithi183.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.1 MB/s | 4.9 MB 00:01 2024-09-21T12:12:30.574 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-21T12:12:30.789 INFO:teuthology.orchestra.run.smithi183.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.0 MB/s | 5.1 MB 00:01 2024-09-21T12:12:31.155 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:06 ago on Sat 21 Sep 2024 12:11:25 PM UTC. 2024-09-21T12:12:31.287 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-21T12:12:31.288 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================= 2024-09-21T12:12:31.288 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-21T12:12:31.288 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================= 2024-09-21T12:12:31.288 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-21T12:12:31.288 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-5103.g5c1cdc3c.el9 ceph-noarch 7.4 M 2024-09-21T12:12:31.288 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-21T12:12:31.289 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T12:12:31.289 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T12:12:31.289 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T12:12:31.289 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T12:12:31.289 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T12:12:31.289 INFO:teuthology.orchestra.run.smithi114.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T12:12:31.289 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T12:12:31.289 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T12:12:31.289 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T12:12:31.289 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T12:12:31.289 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T12:12:31.289 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:31.289 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-21T12:12:31.289 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================= 2024-09-21T12:12:31.289 INFO:teuthology.orchestra.run.smithi114.stdout:Install 12 Packages 2024-09-21T12:12:31.290 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:31.290 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 38 M 2024-09-21T12:12:31.290 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 212 M 2024-09-21T12:12:31.290 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-21T12:12:31.760 INFO:teuthology.orchestra.run.smithi114.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 947 kB/s | 191 kB 00:00 2024-09-21T12:12:31.827 INFO:teuthology.orchestra.run.smithi114.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 491 kB/s | 33 kB 00:00 2024-09-21T12:12:31.885 INFO:teuthology.orchestra.run.smithi114.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.4 MB/s | 801 kB 00:00 2024-09-21T12:12:32.002 INFO:teuthology.orchestra.run.smithi114.stdout:(4/12): ceph-mgr-diskprediction-local-19.3.0-51 17 MB/s | 7.4 MB 00:00 2024-09-21T12:12:32.028 INFO:teuthology.orchestra.run.smithi114.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 128 kB/s | 18 kB 00:00 2024-09-21T12:12:32.086 INFO:teuthology.orchestra.run.smithi114.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 461 kB/s | 38 kB 00:00 2024-09-21T12:12:32.136 INFO:teuthology.orchestra.run.smithi114.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.7 MB/s | 3.0 MB 00:00 2024-09-21T12:12:32.228 INFO:teuthology.orchestra.run.smithi114.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-09-21T12:12:32.312 INFO:teuthology.orchestra.run.smithi114.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 17 MB/s | 4.9 MB 00:00 2024-09-21T12:12:32.337 INFO:teuthology.orchestra.run.smithi114.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.3 MB/s | 250 kB 00:00 2024-09-21T12:12:32.521 INFO:teuthology.orchestra.run.smithi114.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 5.1 MB 00:00 2024-09-21T12:12:32.713 INFO:teuthology.orchestra.run.smithi114.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-09-21T12:12:32.714 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:12:32.714 INFO:teuthology.orchestra.run.smithi114.stdout:Total 27 MB/s | 38 MB 00:01 2024-09-21T12:12:33.071 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-21T12:12:33.097 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-21T12:12:33.097 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-21T12:12:33.531 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-21T12:12:33.531 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-21T12:12:34.264 INFO:teuthology.orchestra.run.smithi183.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 3.7 MB/s | 16 MB 00:04 2024-09-21T12:12:34.266 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:12:34.266 INFO:teuthology.orchestra.run.smithi183.stdout:Total 7.0 MB/s | 38 MB 00:05 2024-09-21T12:12:34.300 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-21T12:12:34.369 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-21T12:12:34.411 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-21T12:12:34.510 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-21T12:12:34.546 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-21T12:12:34.630 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-21T12:12:34.656 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-21T12:12:34.656 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-21T12:12:34.730 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-21T12:12:34.761 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-21T12:12:34.837 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-21T12:12:35.090 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-21T12:12:35.090 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-21T12:12:35.500 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-21T12:12:35.617 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-21T12:12:35.845 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-21T12:12:35.915 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-21T12:12:35.954 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-21T12:12:36.053 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-21T12:12:36.088 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-21T12:12:36.284 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-21T12:12:36.329 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-21T12:12:36.405 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-21T12:12:36.849 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T12:12:37.082 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-21T12:12:37.192 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-21T12:12:37.883 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-21T12:12:37.910 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-5103.g5c1cd 12/12 2024-09-21T12:12:38.404 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T12:12:39.647 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-21T12:12:39.679 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-5103.g5c1cd 12/12 2024-09-21T12:12:40.409 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-5103.g5c1cd 12/12 2024-09-21T12:12:40.409 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-5103.g5c1cd 1/12 2024-09-21T12:12:40.409 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-21T12:12:40.409 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-21T12:12:40.409 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-21T12:12:40.409 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-21T12:12:40.409 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-21T12:12:40.410 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-21T12:12:40.410 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-21T12:12:40.410 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-21T12:12:40.410 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T12:12:40.410 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-21T12:12:40.798 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-21T12:12:40.798 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:40.798 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-21T12:12:40.798 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local-2:19.3.0-5103.g5c1cdc3c.el9.noarch 2024-09-21T12:12:40.798 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T12:12:40.798 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T12:12:40.798 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T12:12:40.798 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T12:12:40.798 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T12:12:40.799 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T12:12:40.799 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T12:12:40.799 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T12:12:40.799 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T12:12:40.799 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T12:12:40.799 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T12:12:40.799 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:40.799 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-21T12:12:41.131 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-rook 2024-09-21T12:12:41.726 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:16 ago on Sat 21 Sep 2024 12:11:25 PM UTC. 2024-09-21T12:12:41.858 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-21T12:12:41.859 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:12:41.860 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-21T12:12:41.860 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:12:41.860 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-21T12:12:41.860 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook noarch 2:19.3.0-5103.g5c1cdc3c.el9 ceph-noarch 49 k 2024-09-21T12:12:41.860 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-21T12:12:41.860 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T12:12:41.860 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T12:12:41.860 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T12:12:41.860 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T12:12:41.860 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T12:12:41.860 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T12:12:41.860 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T12:12:41.861 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T12:12:41.861 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T12:12:41.861 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T12:12:41.861 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T12:12:41.861 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T12:12:41.861 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:41.861 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-21T12:12:41.861 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:12:41.861 INFO:teuthology.orchestra.run.smithi114.stdout:Install 13 Packages 2024-09-21T12:12:41.861 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:41.862 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 2.2 M 2024-09-21T12:12:41.862 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 26 M 2024-09-21T12:12:41.862 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-21T12:12:42.159 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-5103.g5c1cd 12/12 2024-09-21T12:12:42.160 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-5103.g5c1cd 1/12 2024-09-21T12:12:42.160 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-21T12:12:42.160 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-21T12:12:42.160 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-21T12:12:42.160 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-21T12:12:42.160 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-21T12:12:42.160 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-21T12:12:42.160 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-21T12:12:42.160 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-21T12:12:42.160 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T12:12:42.160 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-21T12:12:42.361 INFO:teuthology.orchestra.run.smithi114.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 214 kB/s | 19 kB 00:00 2024-09-21T12:12:42.395 INFO:teuthology.orchestra.run.smithi114.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 208 kB/s | 26 kB 00:00 2024-09-21T12:12:42.445 INFO:teuthology.orchestra.run.smithi114.stdout:(3/13): ceph-mgr-rook-19.3.0-5103.g5c1cdc3c.el9 284 kB/s | 49 kB 00:00 2024-09-21T12:12:42.479 INFO:teuthology.orchestra.run.smithi114.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 222 kB 00:00 2024-09-21T12:12:42.512 INFO:teuthology.orchestra.run.smithi114.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-21T12:12:42.546 INFO:teuthology.orchestra.run.smithi114.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 803 kB/s | 54 kB 00:00 2024-09-21T12:12:42.588 INFO:teuthology.orchestra.run.smithi114.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 427 kB/s | 32 kB 00:00 2024-09-21T12:12:42.596 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-21T12:12:42.596 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:42.596 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-21T12:12:42.596 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local-2:19.3.0-5103.g5c1cdc3c.el9.noarch 2024-09-21T12:12:42.596 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T12:12:42.596 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T12:12:42.596 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T12:12:42.596 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T12:12:42.596 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T12:12:42.596 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T12:12:42.596 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T12:12:42.596 INFO:teuthology.orchestra.run.smithi183.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T12:12:42.596 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T12:12:42.597 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T12:12:42.597 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T12:12:42.597 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:42.597 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-21T12:12:42.613 INFO:teuthology.orchestra.run.smithi114.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 211 kB/s | 14 kB 00:00 2024-09-21T12:12:42.639 INFO:teuthology.orchestra.run.smithi114.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 4.4 MB/s | 220 kB 00:00 2024-09-21T12:12:42.673 INFO:teuthology.orchestra.run.smithi114.stdout:(10/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.2 MB/s | 279 kB 00:00 2024-09-21T12:12:42.698 INFO:teuthology.orchestra.run.smithi114.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-09-21T12:12:42.748 INFO:teuthology.orchestra.run.smithi114.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 7.6 MB/s | 1.0 MB 00:00 2024-09-21T12:12:42.781 INFO:teuthology.orchestra.run.smithi114.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 826 kB/s | 90 kB 00:00 2024-09-21T12:12:42.784 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:12:42.784 INFO:teuthology.orchestra.run.smithi114.stdout:Total 2.4 MB/s | 2.2 MB 00:00 2024-09-21T12:12:42.850 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-21T12:12:42.865 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-21T12:12:42.865 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-21T12:12:42.955 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-21T12:12:42.957 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-21T12:12:42.972 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-rook 2024-09-21T12:12:43.465 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-21T12:12:43.560 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:22 ago on Sat 21 Sep 2024 12:11:21 PM UTC. 2024-09-21T12:12:43.645 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-21T12:12:43.694 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-21T12:12:43.696 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:12:43.696 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-09-21T12:12:43.696 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:12:43.696 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-21T12:12:43.696 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook noarch 2:19.3.0-5103.g5c1cdc3c.el9 ceph-noarch 49 k 2024-09-21T12:12:43.696 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-09-21T12:12:43.696 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T12:12:43.696 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T12:12:43.697 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T12:12:43.697 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T12:12:43.697 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T12:12:43.697 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T12:12:43.697 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T12:12:43.697 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T12:12:43.697 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T12:12:43.697 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T12:12:43.697 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T12:12:43.697 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T12:12:43.697 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:43.697 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-21T12:12:43.697 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:12:43.697 INFO:teuthology.orchestra.run.smithi183.stdout:Install 13 Packages 2024-09-21T12:12:43.698 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:43.698 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 2.2 M 2024-09-21T12:12:43.698 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 26 M 2024-09-21T12:12:43.698 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-21T12:12:43.715 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-21T12:12:43.780 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-21T12:12:43.817 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-21T12:12:43.859 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-21T12:12:43.960 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-21T12:12:44.063 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-21T12:12:44.110 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-21T12:12:44.439 INFO:teuthology.orchestra.run.smithi183.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 225 kB/s | 26 kB 00:00 2024-09-21T12:12:44.456 INFO:teuthology.orchestra.run.smithi183.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 147 kB/s | 19 kB 00:00 2024-09-21T12:12:44.481 INFO:teuthology.orchestra.run.smithi183.stdout:(3/13): ceph-mgr-rook-19.3.0-5103.g5c1cdc3c.el9 316 kB/s | 49 kB 00:00 2024-09-21T12:12:44.573 INFO:teuthology.orchestra.run.smithi183.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-21T12:12:44.598 INFO:teuthology.orchestra.run.smithi183.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 222 kB 00:00 2024-09-21T12:12:44.632 INFO:teuthology.orchestra.run.smithi183.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 917 kB/s | 54 kB 00:00 2024-09-21T12:12:44.666 INFO:teuthology.orchestra.run.smithi183.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 481 kB/s | 32 kB 00:00 2024-09-21T12:12:44.691 INFO:teuthology.orchestra.run.smithi183.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.3 MB/s | 279 kB 00:00 2024-09-21T12:12:44.716 INFO:teuthology.orchestra.run.smithi183.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 172 kB/s | 14 kB 00:00 2024-09-21T12:12:44.750 INFO:teuthology.orchestra.run.smithi183.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.2 MB/s | 59 kB 00:00 2024-09-21T12:12:44.775 INFO:teuthology.orchestra.run.smithi183.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 2.0 MB/s | 220 kB 00:00 2024-09-21T12:12:44.809 INFO:teuthology.orchestra.run.smithi183.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-09-21T12:12:44.831 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-21T12:12:44.851 INFO:teuthology.orchestra.run.smithi183.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.4 MB/s | 1.0 MB 00:00 2024-09-21T12:12:44.854 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:12:44.854 INFO:teuthology.orchestra.run.smithi183.stdout:Total 1.9 MB/s | 2.2 MB 00:01 2024-09-21T12:12:44.874 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-21T12:12:44.908 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-21T12:12:44.918 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-21T12:12:44.934 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-21T12:12:44.934 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-21T12:12:44.956 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-21T12:12:44.985 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-rook-2:19.3.0-5103.g5c1cdc3c.el9.noarch 13/13 2024-09-21T12:12:45.024 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-21T12:12:45.025 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-21T12:12:45.546 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-21T12:12:45.729 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-21T12:12:45.733 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-5103.g5c1cdc3c.el9.noarch 13/13 2024-09-21T12:12:45.734 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-rook-2:19.3.0-5103.g5c1cdc3c.el9.noarch 1/13 2024-09-21T12:12:45.734 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-21T12:12:45.734 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-21T12:12:45.734 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-21T12:12:45.734 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-21T12:12:45.734 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-21T12:12:45.734 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-21T12:12:45.734 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-21T12:12:45.734 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-21T12:12:45.734 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-21T12:12:45.734 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-21T12:12:45.734 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-21T12:12:45.785 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-21T12:12:45.840 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-21T12:12:45.877 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-21T12:12:45.919 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-21T12:12:46.020 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-21T12:12:46.070 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-21T12:12:46.070 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:46.070 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-21T12:12:46.070 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook-2:19.3.0-5103.g5c1cdc3c.el9.noarch 2024-09-21T12:12:46.070 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T12:12:46.070 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T12:12:46.070 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T12:12:46.070 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T12:12:46.070 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T12:12:46.070 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T12:12:46.070 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T12:12:46.070 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T12:12:46.070 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T12:12:46.071 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T12:12:46.071 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T12:12:46.071 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T12:12:46.071 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:46.071 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-21T12:12:46.121 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-21T12:12:46.190 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-21T12:12:46.354 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-cephadm 2024-09-21T12:12:46.897 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-21T12:12:46.941 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:21 ago on Sat 21 Sep 2024 12:11:25 PM UTC. 2024-09-21T12:12:46.943 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-21T12:12:46.977 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-21T12:12:47.025 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-21T12:12:47.047 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-rook-2:19.3.0-5103.g5c1cdc3c.el9.noarch 13/13 2024-09-21T12:12:47.076 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-21T12:12:47.077 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:12:47.077 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-21T12:12:47.077 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:12:47.077 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-21T12:12:47.077 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm noarch 2:19.3.0-5103.g5c1cdc3c.el9 ceph-noarch 154 k 2024-09-21T12:12:47.077 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-21T12:12:47.078 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T12:12:47.078 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T12:12:47.078 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T12:12:47.078 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T12:12:47.078 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T12:12:47.078 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:47.078 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-21T12:12:47.078 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:12:47.078 INFO:teuthology.orchestra.run.smithi114.stdout:Install 6 Packages 2024-09-21T12:12:47.078 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:47.079 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 7.0 M 2024-09-21T12:12:47.079 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 33 M 2024-09-21T12:12:47.079 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-21T12:12:47.592 INFO:teuthology.orchestra.run.smithi114.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.6 MB/s | 248 kB 00:00 2024-09-21T12:12:47.642 INFO:teuthology.orchestra.run.smithi114.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-21T12:12:47.667 INFO:teuthology.orchestra.run.smithi114.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-21T12:12:47.734 INFO:teuthology.orchestra.run.smithi114.stdout:(4/6): ceph-mgr-cephadm-19.3.0-5103.g5c1cdc3c.e 529 kB/s | 154 kB 00:00 2024-09-21T12:12:47.759 INFO:teuthology.orchestra.run.smithi114.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 811 kB/s | 74 kB 00:00 2024-09-21T12:12:47.793 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-5103.g5c1cdc3c.el9.noarch 13/13 2024-09-21T12:12:47.793 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-rook-2:19.3.0-5103.g5c1cdc3c.el9.noarch 1/13 2024-09-21T12:12:47.793 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-21T12:12:47.793 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-21T12:12:47.793 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-21T12:12:47.794 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-21T12:12:47.794 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-21T12:12:47.794 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-21T12:12:47.794 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-21T12:12:47.794 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-21T12:12:47.794 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-21T12:12:47.794 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-21T12:12:47.794 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-21T12:12:47.885 INFO:teuthology.orchestra.run.smithi114.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 6.0 MB 00:00 2024-09-21T12:12:47.887 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:12:47.887 INFO:teuthology.orchestra.run.smithi114.stdout:Total 8.7 MB/s | 7.0 MB 00:00 2024-09-21T12:12:47.958 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-21T12:12:47.969 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-21T12:12:47.969 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-21T12:12:48.069 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-21T12:12:48.070 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-21T12:12:48.135 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-21T12:12:48.135 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:48.135 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-21T12:12:48.135 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook-2:19.3.0-5103.g5c1cdc3c.el9.noarch 2024-09-21T12:12:48.135 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T12:12:48.135 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T12:12:48.135 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T12:12:48.135 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T12:12:48.135 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T12:12:48.135 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T12:12:48.135 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T12:12:48.136 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T12:12:48.136 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T12:12:48.136 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T12:12:48.136 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T12:12:48.136 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T12:12:48.136 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:48.136 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-21T12:12:48.385 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-21T12:12:48.401 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-cephadm 2024-09-21T12:12:48.560 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-21T12:12:48.611 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-21T12:12:48.984 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:27 ago on Sat 21 Sep 2024 12:11:21 PM UTC. 2024-09-21T12:12:49.119 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-21T12:12:49.119 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:12:49.119 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-09-21T12:12:49.119 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:12:49.119 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-21T12:12:49.120 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm noarch 2:19.3.0-5103.g5c1cdc3c.el9 ceph-noarch 154 k 2024-09-21T12:12:49.120 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-09-21T12:12:49.120 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T12:12:49.120 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T12:12:49.120 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T12:12:49.120 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T12:12:49.120 INFO:teuthology.orchestra.run.smithi183.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T12:12:49.120 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:49.120 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-21T12:12:49.120 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:12:49.120 INFO:teuthology.orchestra.run.smithi183.stdout:Install 6 Packages 2024-09-21T12:12:49.120 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:49.121 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 7.0 M 2024-09-21T12:12:49.121 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 33 M 2024-09-21T12:12:49.121 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-21T12:12:49.141 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-21T12:12:49.232 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-21T12:12:49.299 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-21T12:12:49.323 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-5103.g5c1cdc3c.el9.noarch 6/6 2024-09-21T12:12:49.911 INFO:teuthology.orchestra.run.smithi183.stdout:(1/6): ceph-mgr-cephadm-19.3.0-5103.g5c1cdc3c.e 744 kB/s | 154 kB 00:00 2024-09-21T12:12:49.936 INFO:teuthology.orchestra.run.smithi183.stdout:(2/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.0 MB/s | 248 kB 00:00 2024-09-21T12:12:49.986 INFO:teuthology.orchestra.run.smithi183.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.1 MB/s | 548 kB 00:00 2024-09-21T12:12:50.012 INFO:teuthology.orchestra.run.smithi183.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 767 kB/s | 58 kB 00:00 2024-09-21T12:12:50.037 INFO:teuthology.orchestra.run.smithi183.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.5 MB/s | 74 kB 00:00 2024-09-21T12:12:50.217 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-5103.g5c1cdc3c.el9.noarch 6/6 2024-09-21T12:12:50.217 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-5103.g5c1cdc3c.el9.noarch 1/6 2024-09-21T12:12:50.217 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-21T12:12:50.218 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-21T12:12:50.218 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-21T12:12:50.218 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-21T12:12:50.503 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-21T12:12:50.503 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:50.503 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-21T12:12:50.503 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm-2:19.3.0-5103.g5c1cdc3c.el9.noarch 2024-09-21T12:12:50.503 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T12:12:50.503 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T12:12:50.503 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T12:12:50.503 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T12:12:50.503 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T12:12:50.503 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:50.503 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-21T12:12:50.749 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-fuse 2024-09-21T12:12:51.112 INFO:teuthology.orchestra.run.smithi183.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.2 MB/s | 6.0 MB 00:01 2024-09-21T12:12:51.114 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:12:51.114 INFO:teuthology.orchestra.run.smithi183.stdout:Total 3.5 MB/s | 7.0 MB 00:01 2024-09-21T12:12:51.185 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-21T12:12:51.196 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-21T12:12:51.196 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-21T12:12:51.302 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-21T12:12:51.302 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-21T12:12:51.342 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:26 ago on Sat 21 Sep 2024 12:11:25 PM UTC. 2024-09-21T12:12:51.475 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-21T12:12:51.476 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:12:51.476 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-21T12:12:51.476 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:12:51.476 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-21T12:12:51.476 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 899 k 2024-09-21T12:12:51.476 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-21T12:12:51.476 INFO:teuthology.orchestra.run.smithi114.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T12:12:51.476 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:51.476 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-21T12:12:51.476 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:12:51.477 INFO:teuthology.orchestra.run.smithi114.stdout:Install 2 Packages 2024-09-21T12:12:51.477 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:51.477 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 979 k 2024-09-21T12:12:51.477 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 2.9 M 2024-09-21T12:12:51.477 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-21T12:12:51.610 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-21T12:12:51.712 INFO:teuthology.orchestra.run.smithi114.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 790 kB/s | 80 kB 00:00 2024-09-21T12:12:51.786 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-21T12:12:51.847 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-21T12:12:51.887 INFO:teuthology.orchestra.run.smithi114.stdout:(2/2): ceph-fuse-19.3.0-5103.g5c1cdc3c.el9.x86_ 3.2 MB/s | 899 kB 00:00 2024-09-21T12:12:51.887 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:12:51.887 INFO:teuthology.orchestra.run.smithi114.stdout:Total 2.3 MB/s | 979 kB 00:00 2024-09-21T12:12:51.899 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-21T12:12:51.922 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-21T12:12:51.922 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-21T12:12:52.004 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-21T12:12:52.005 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-21T12:12:52.190 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-21T12:12:52.331 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-21T12:12:52.383 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-21T12:12:52.403 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-fuse-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2/2 2024-09-21T12:12:52.475 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-21T12:12:52.543 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-21T12:12:52.576 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-5103.g5c1cdc3c.el9.noarch 6/6 2024-09-21T12:12:53.094 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-fuse-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2/2 2024-09-21T12:12:53.094 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-fuse-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/2 2024-09-21T12:12:53.394 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-21T12:12:53.395 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:53.395 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-21T12:12:53.395 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-21T12:12:53.395 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:53.395 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-21T12:12:53.415 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-5103.g5c1cdc3c.el9.noarch 6/6 2024-09-21T12:12:53.415 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-5103.g5c1cdc3c.el9.noarch 1/6 2024-09-21T12:12:53.415 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-21T12:12:53.415 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-21T12:12:53.415 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-21T12:12:53.415 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-21T12:12:53.700 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-21T12:12:53.700 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:53.701 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-21T12:12:53.701 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm-2:19.3.0-5103.g5c1cdc3c.el9.noarch 2024-09-21T12:12:53.701 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T12:12:53.701 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T12:12:53.701 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T12:12:53.701 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T12:12:53.701 INFO:teuthology.orchestra.run.smithi183.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T12:12:53.701 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:53.701 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-21T12:12:53.749 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-volume 2024-09-21T12:12:53.967 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-fuse 2024-09-21T12:12:54.340 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:29 ago on Sat 21 Sep 2024 12:11:25 PM UTC. 2024-09-21T12:12:54.473 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-21T12:12:54.474 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:12:54.474 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-21T12:12:54.474 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:12:54.474 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-21T12:12:54.474 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-volume noarch 2:19.3.0-5103.g5c1cdc3c.el9 ceph-noarch 287 k 2024-09-21T12:12:54.475 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:54.475 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-21T12:12:54.475 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:12:54.475 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-21T12:12:54.475 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:54.475 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 287 k 2024-09-21T12:12:54.475 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 1.4 M 2024-09-21T12:12:54.475 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-21T12:12:54.555 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:33 ago on Sat 21 Sep 2024 12:11:21 PM UTC. 2024-09-21T12:12:54.689 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-21T12:12:54.689 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:12:54.689 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-09-21T12:12:54.690 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:12:54.690 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-21T12:12:54.690 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 899 k 2024-09-21T12:12:54.690 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-09-21T12:12:54.690 INFO:teuthology.orchestra.run.smithi183.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T12:12:54.690 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:54.690 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-21T12:12:54.690 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:12:54.690 INFO:teuthology.orchestra.run.smithi183.stdout:Install 2 Packages 2024-09-21T12:12:54.690 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:54.691 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 979 k 2024-09-21T12:12:54.691 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 2.9 M 2024-09-21T12:12:54.691 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-21T12:12:54.699 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-volume-19.3.0-5103.g5c1cdc3c.el9.noarch.rp 1.3 MB/s | 287 kB 00:00 2024-09-21T12:12:54.700 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:12:54.700 INFO:teuthology.orchestra.run.smithi114.stdout:Total 1.2 MB/s | 287 kB 00:00 2024-09-21T12:12:54.700 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-21T12:12:54.709 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-21T12:12:54.709 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-21T12:12:54.736 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-21T12:12:54.736 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-21T12:12:54.856 INFO:teuthology.orchestra.run.smithi183.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 553 kB/s | 80 kB 00:00 2024-09-21T12:12:54.952 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-21T12:12:54.998 INFO:teuthology.orchestra.run.smithi183.stdout:(2/2): ceph-fuse-19.3.0-5103.g5c1cdc3c.el9.x86_ 3.1 MB/s | 899 kB 00:00 2024-09-21T12:12:54.998 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:12:54.998 INFO:teuthology.orchestra.run.smithi183.stdout:Total 3.1 MB/s | 979 kB 00:00 2024-09-21T12:12:55.009 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-21T12:12:55.018 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-volume-2:19.3.0-5103.g5c1cdc3c.el9.noarch 1/1 2024-09-21T12:12:55.031 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-21T12:12:55.031 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-21T12:12:55.034 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-volume-2:19.3.0-5103.g5c1cdc3c.el9.noarch 1/1 2024-09-21T12:12:55.034 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T12:12:55.034 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-21T12:12:55.034 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:55.112 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-21T12:12:55.113 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-21T12:12:55.292 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-21T12:12:55.425 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-21T12:12:55.488 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-fuse-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2/2 2024-09-21T12:12:56.045 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-volume-2:19.3.0-5103.g5c1cdc3c.el9.noarch 1/1 2024-09-21T12:12:56.045 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:56.045 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-21T12:12:56.045 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-volume-2:19.3.0-5103.g5c1cdc3c.el9.noarch 2024-09-21T12:12:56.045 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:56.046 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-21T12:12:56.192 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-fuse-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2/2 2024-09-21T12:12:56.192 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-fuse-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/2 2024-09-21T12:12:56.387 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install librados-devel 2024-09-21T12:12:56.484 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-21T12:12:56.484 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:56.484 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-21T12:12:56.484 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-21T12:12:56.484 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:56.484 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-21T12:12:56.776 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-volume 2024-09-21T12:12:56.977 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:31 ago on Sat 21 Sep 2024 12:11:25 PM UTC. 2024-09-21T12:12:57.111 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-21T12:12:57.112 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:12:57.112 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-09-21T12:12:57.112 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:12:57.112 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-21T12:12:57.112 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 126 k 2024-09-21T12:12:57.112 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:57.112 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-21T12:12:57.112 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:12:57.113 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-21T12:12:57.113 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:57.113 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 126 k 2024-09-21T12:12:57.113 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 456 k 2024-09-21T12:12:57.113 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-21T12:12:57.321 INFO:teuthology.orchestra.run.smithi114.stdout:librados-devel-19.3.0-5103.g5c1cdc3c.el9.x86_64 609 kB/s | 126 kB 00:00 2024-09-21T12:12:57.322 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:12:57.322 INFO:teuthology.orchestra.run.smithi114.stdout:Total 604 kB/s | 126 kB 00:00 2024-09-21T12:12:57.322 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-21T12:12:57.328 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-21T12:12:57.328 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-21T12:12:57.363 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:36 ago on Sat 21 Sep 2024 12:11:21 PM UTC. 2024-09-21T12:12:57.391 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-21T12:12:57.392 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-21T12:12:57.496 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-21T12:12:57.496 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:12:57.496 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-09-21T12:12:57.496 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:12:57.496 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-21T12:12:57.497 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-volume noarch 2:19.3.0-5103.g5c1cdc3c.el9 ceph-noarch 287 k 2024-09-21T12:12:57.497 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:57.497 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-21T12:12:57.497 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:12:57.497 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-09-21T12:12:57.497 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:57.497 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 287 k 2024-09-21T12:12:57.498 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 1.4 M 2024-09-21T12:12:57.498 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-21T12:12:57.544 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-21T12:12:57.660 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librados-devel-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/1 2024-09-21T12:12:57.719 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-volume-19.3.0-5103.g5c1cdc3c.el9.noarch.rp 1.3 MB/s | 287 kB 00:00 2024-09-21T12:12:57.719 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:12:57.720 INFO:teuthology.orchestra.run.smithi183.stdout:Total 1.3 MB/s | 287 kB 00:00 2024-09-21T12:12:57.720 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-21T12:12:57.728 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-21T12:12:57.728 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-21T12:12:57.752 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-21T12:12:57.753 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-21T12:12:57.968 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-21T12:12:58.029 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-volume-2:19.3.0-5103.g5c1cdc3c.el9.noarch 1/1 2024-09-21T12:12:58.045 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-volume-2:19.3.0-5103.g5c1cdc3c.el9.noarch 1/1 2024-09-21T12:12:58.045 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T12:12:58.045 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-21T12:12:58.045 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:58.081 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados-devel-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/1 2024-09-21T12:12:58.327 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados-devel-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/1 2024-09-21T12:12:58.327 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:58.327 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-21T12:12:58.327 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:12:58.327 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:12:58.327 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-21T12:12:58.631 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install libcephfs2 2024-09-21T12:12:59.074 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-volume-2:19.3.0-5103.g5c1cdc3c.el9.noarch 1/1 2024-09-21T12:12:59.074 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:59.075 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-21T12:12:59.075 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-volume-2:19.3.0-5103.g5c1cdc3c.el9.noarch 2024-09-21T12:12:59.075 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:12:59.075 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-21T12:12:59.228 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:34 ago on Sat 21 Sep 2024 12:11:25 PM UTC. 2024-09-21T12:12:59.321 INFO:teuthology.orchestra.run.smithi114.stdout:Package libcephfs2-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 is already installed. 2024-09-21T12:12:59.366 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-21T12:12:59.367 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-21T12:12:59.367 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-21T12:12:59.401 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librados-devel 2024-09-21T12:12:59.435 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install libcephfs-devel 2024-09-21T12:13:00.000 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:38 ago on Sat 21 Sep 2024 12:11:21 PM UTC. 2024-09-21T12:13:00.015 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:35 ago on Sat 21 Sep 2024 12:11:25 PM UTC. 2024-09-21T12:13:00.134 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-21T12:13:00.134 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:13:00.134 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-09-21T12:13:00.135 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:13:00.135 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-21T12:13:00.135 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 126 k 2024-09-21T12:13:00.135 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:13:00.135 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-21T12:13:00.135 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:13:00.135 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-09-21T12:13:00.135 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:13:00.135 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 126 k 2024-09-21T12:13:00.135 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 456 k 2024-09-21T12:13:00.135 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-21T12:13:00.148 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-21T12:13:00.148 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:13:00.148 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-09-21T12:13:00.148 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:13:00.149 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-21T12:13:00.149 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 32 k 2024-09-21T12:13:00.149 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:13:00.149 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-21T12:13:00.149 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:13:00.149 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-21T12:13:00.149 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:13:00.150 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 32 k 2024-09-21T12:13:00.150 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 142 k 2024-09-21T12:13:00.150 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-21T12:13:00.291 INFO:teuthology.orchestra.run.smithi114.stdout:libcephfs-devel-19.3.0-5103.g5c1cdc3c.el9.x86_6 224 kB/s | 32 kB 00:00 2024-09-21T12:13:00.292 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:13:00.292 INFO:teuthology.orchestra.run.smithi114.stdout:Total 222 kB/s | 32 kB 00:00 2024-09-21T12:13:00.292 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-21T12:13:00.296 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-21T12:13:00.296 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-21T12:13:00.318 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-21T12:13:00.318 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-21T12:13:00.341 INFO:teuthology.orchestra.run.smithi183.stdout:librados-devel-19.3.0-5103.g5c1cdc3c.el9.x86_64 617 kB/s | 126 kB 00:00 2024-09-21T12:13:00.341 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:13:00.342 INFO:teuthology.orchestra.run.smithi183.stdout:Total 612 kB/s | 126 kB 00:00 2024-09-21T12:13:00.342 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-21T12:13:00.347 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-21T12:13:00.347 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-21T12:13:00.409 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-21T12:13:00.411 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-21T12:13:00.412 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-21T12:13:00.505 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs-devel-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/1 2024-09-21T12:13:00.559 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-21T12:13:00.664 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librados-devel-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/1 2024-09-21T12:13:00.852 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/1 2024-09-21T12:13:01.092 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs-devel-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/1 2024-09-21T12:13:01.092 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:13:01.092 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-21T12:13:01.092 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:13:01.092 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:13:01.092 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-21T12:13:01.140 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados-devel-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/1 2024-09-21T12:13:01.287 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install librados2 2024-09-21T12:13:01.408 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados-devel-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/1 2024-09-21T12:13:01.408 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:13:01.408 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-21T12:13:01.408 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:13:01.408 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:13:01.408 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-21T12:13:01.709 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install libcephfs2 2024-09-21T12:13:01.881 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:36 ago on Sat 21 Sep 2024 12:11:25 PM UTC. 2024-09-21T12:13:01.971 INFO:teuthology.orchestra.run.smithi114.stdout:Package librados2-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 is already installed. 2024-09-21T12:13:02.014 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-21T12:13:02.016 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-21T12:13:02.016 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-21T12:13:02.083 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install librbd1 2024-09-21T12:13:02.298 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:41 ago on Sat 21 Sep 2024 12:11:21 PM UTC. 2024-09-21T12:13:02.388 INFO:teuthology.orchestra.run.smithi183.stdout:Package libcephfs2-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 is already installed. 2024-09-21T12:13:02.435 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-21T12:13:02.436 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-09-21T12:13:02.436 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-21T12:13:02.523 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install libcephfs-devel 2024-09-21T12:13:02.664 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:37 ago on Sat 21 Sep 2024 12:11:25 PM UTC. 2024-09-21T12:13:02.755 INFO:teuthology.orchestra.run.smithi114.stdout:Package librbd1-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 is already installed. 2024-09-21T12:13:02.799 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-21T12:13:02.800 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-21T12:13:02.800 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-21T12:13:02.866 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-rados 2024-09-21T12:13:03.112 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:42 ago on Sat 21 Sep 2024 12:11:21 PM UTC. 2024-09-21T12:13:03.246 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-21T12:13:03.247 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:13:03.247 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-09-21T12:13:03.247 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:13:03.247 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-21T12:13:03.247 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 32 k 2024-09-21T12:13:03.247 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:13:03.248 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-21T12:13:03.248 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:13:03.248 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-09-21T12:13:03.248 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:13:03.248 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 32 k 2024-09-21T12:13:03.248 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 142 k 2024-09-21T12:13:03.248 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-21T12:13:03.395 INFO:teuthology.orchestra.run.smithi183.stdout:libcephfs-devel-19.3.0-5103.g5c1cdc3c.el9.x86_6 216 kB/s | 32 kB 00:00 2024-09-21T12:13:03.395 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:13:03.396 INFO:teuthology.orchestra.run.smithi183.stdout:Total 214 kB/s | 32 kB 00:00 2024-09-21T12:13:03.396 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-21T12:13:03.399 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-21T12:13:03.399 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-21T12:13:03.418 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-21T12:13:03.418 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-21T12:13:03.445 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:38 ago on Sat 21 Sep 2024 12:11:25 PM UTC. 2024-09-21T12:13:03.497 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-21T12:13:03.534 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-rados-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 is already installed. 2024-09-21T12:13:03.578 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-21T12:13:03.579 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-21T12:13:03.579 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-21T12:13:03.600 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs-devel-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/1 2024-09-21T12:13:03.653 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-rgw 2024-09-21T12:13:03.994 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/1 2024-09-21T12:13:04.235 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:39 ago on Sat 21 Sep 2024 12:11:25 PM UTC. 2024-09-21T12:13:04.238 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs-devel-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/1 2024-09-21T12:13:04.239 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:13:04.239 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-21T12:13:04.239 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:13:04.239 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:13:04.239 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-21T12:13:04.324 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-rgw-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 is already installed. 2024-09-21T12:13:04.368 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-21T12:13:04.369 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-21T12:13:04.369 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-21T12:13:04.445 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-cephfs 2024-09-21T12:13:04.460 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librados2 2024-09-21T12:13:05.030 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:40 ago on Sat 21 Sep 2024 12:11:25 PM UTC. 2024-09-21T12:13:05.053 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:44 ago on Sat 21 Sep 2024 12:11:21 PM UTC. 2024-09-21T12:13:05.120 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-cephfs-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 is already installed. 2024-09-21T12:13:05.143 INFO:teuthology.orchestra.run.smithi183.stdout:Package librados2-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 is already installed. 2024-09-21T12:13:05.163 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-21T12:13:05.164 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-21T12:13:05.164 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-21T12:13:05.186 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-21T12:13:05.187 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-09-21T12:13:05.188 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-21T12:13:05.244 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-rbd 2024-09-21T12:13:05.267 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librbd1 2024-09-21T12:13:05.825 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:40 ago on Sat 21 Sep 2024 12:11:25 PM UTC. 2024-09-21T12:13:05.857 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:44 ago on Sat 21 Sep 2024 12:11:21 PM UTC. 2024-09-21T12:13:05.914 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-rbd-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 is already installed. 2024-09-21T12:13:05.947 INFO:teuthology.orchestra.run.smithi183.stdout:Package librbd1-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 is already installed. 2024-09-21T12:13:05.957 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-21T12:13:05.958 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-21T12:13:05.958 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-21T12:13:05.991 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-21T12:13:05.992 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-09-21T12:13:05.992 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-21T12:13:06.038 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install rbd-fuse 2024-09-21T12:13:06.066 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rados 2024-09-21T12:13:06.632 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:41 ago on Sat 21 Sep 2024 12:11:25 PM UTC. 2024-09-21T12:13:06.655 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:45 ago on Sat 21 Sep 2024 12:11:21 PM UTC. 2024-09-21T12:13:06.747 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-rados-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 is already installed. 2024-09-21T12:13:06.767 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-21T12:13:06.767 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:13:06.767 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-09-21T12:13:06.768 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:13:06.768 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-21T12:13:06.768 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 85 k 2024-09-21T12:13:06.768 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:13:06.768 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-21T12:13:06.768 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:13:06.768 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-21T12:13:06.768 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:13:06.769 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 85 k 2024-09-21T12:13:06.769 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 231 k 2024-09-21T12:13:06.769 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-21T12:13:06.791 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-21T12:13:06.792 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-09-21T12:13:06.792 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-21T12:13:06.868 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rgw 2024-09-21T12:13:06.970 INFO:teuthology.orchestra.run.smithi114.stdout:rbd-fuse-19.3.0-5103.g5c1cdc3c.el9.x86_64.rpm 424 kB/s | 85 kB 00:00 2024-09-21T12:13:06.971 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:13:06.971 INFO:teuthology.orchestra.run.smithi114.stdout:Total 422 kB/s | 85 kB 00:00 2024-09-21T12:13:06.971 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-21T12:13:06.977 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-21T12:13:06.977 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-21T12:13:07.043 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-21T12:13:07.044 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-21T12:13:07.230 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-21T12:13:07.326 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-fuse-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/1 2024-09-21T12:13:07.458 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:46 ago on Sat 21 Sep 2024 12:11:21 PM UTC. 2024-09-21T12:13:07.548 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-rgw-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 is already installed. 2024-09-21T12:13:07.592 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-21T12:13:07.592 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-09-21T12:13:07.593 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-21T12:13:07.670 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-cephfs 2024-09-21T12:13:07.709 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-fuse-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/1 2024-09-21T12:13:07.995 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-fuse-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/1 2024-09-21T12:13:07.995 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:13:07.995 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-21T12:13:07.995 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:13:07.995 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:13:07.995 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-21T12:13:08.206 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install rbd-mirror 2024-09-21T12:13:08.252 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:47 ago on Sat 21 Sep 2024 12:11:21 PM UTC. 2024-09-21T12:13:08.341 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-cephfs-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 is already installed. 2024-09-21T12:13:08.385 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-21T12:13:08.386 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-09-21T12:13:08.386 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-21T12:13:08.461 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rbd 2024-09-21T12:13:08.813 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:43 ago on Sat 21 Sep 2024 12:11:25 PM UTC. 2024-09-21T12:13:08.954 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-21T12:13:08.954 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:13:08.955 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-09-21T12:13:08.955 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:13:08.955 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-21T12:13:08.955 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 3.2 M 2024-09-21T12:13:08.955 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:13:08.955 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-21T12:13:08.955 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:13:08.955 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-21T12:13:08.955 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:13:08.955 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 3.2 M 2024-09-21T12:13:08.955 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 14 M 2024-09-21T12:13:08.955 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-21T12:13:09.043 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:48 ago on Sat 21 Sep 2024 12:11:21 PM UTC. 2024-09-21T12:13:09.132 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-rbd-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 is already installed. 2024-09-21T12:13:09.176 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-21T12:13:09.177 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-09-21T12:13:09.177 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-21T12:13:09.251 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-fuse 2024-09-21T12:13:09.289 INFO:teuthology.orchestra.run.smithi114.stdout:rbd-mirror-19.3.0-5103.g5c1cdc3c.el9.x86_64.rpm 9.6 MB/s | 3.2 MB 00:00 2024-09-21T12:13:09.290 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:13:09.291 INFO:teuthology.orchestra.run.smithi114.stdout:Total 9.5 MB/s | 3.2 MB 00:00 2024-09-21T12:13:09.291 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-21T12:13:09.300 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-21T12:13:09.300 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-21T12:13:09.386 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-21T12:13:09.386 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-21T12:13:09.849 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:48 ago on Sat 21 Sep 2024 12:11:21 PM UTC. 2024-09-21T12:13:09.849 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-21T12:13:09.916 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-mirror-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/1 2024-09-21T12:13:09.941 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-mirror-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/1 2024-09-21T12:13:09.941 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T12:13:09.941 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T12:13:09.942 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T12:13:09.942 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T12:13:09.942 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:13:09.989 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-21T12:13:09.990 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:13:09.990 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-09-21T12:13:09.990 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:13:09.990 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-21T12:13:09.990 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 85 k 2024-09-21T12:13:09.990 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:13:09.990 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-21T12:13:09.990 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:13:09.991 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-09-21T12:13:09.991 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:13:09.991 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 85 k 2024-09-21T12:13:09.991 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 231 k 2024-09-21T12:13:09.991 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-21T12:13:10.181 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-fuse-19.3.0-5103.g5c1cdc3c.el9.x86_64.rpm 448 kB/s | 85 kB 00:00 2024-09-21T12:13:10.182 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:13:10.182 INFO:teuthology.orchestra.run.smithi183.stdout:Total 444 kB/s | 85 kB 00:00 2024-09-21T12:13:10.183 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-21T12:13:10.190 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-21T12:13:10.190 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-21T12:13:10.256 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-21T12:13:10.257 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-21T12:13:10.446 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-21T12:13:10.556 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-fuse-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/1 2024-09-21T12:13:10.984 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-mirror-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/1 2024-09-21T12:13:10.984 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:13:10.984 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-21T12:13:10.984 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:13:10.984 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:13:10.985 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-21T12:13:11.001 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-fuse-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/1 2024-09-21T12:13:11.218 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install rbd-nbd 2024-09-21T12:13:11.257 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-fuse-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/1 2024-09-21T12:13:11.258 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:13:11.258 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-21T12:13:11.258 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:13:11.258 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:13:11.258 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-21T12:13:11.491 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-mirror 2024-09-21T12:13:11.823 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:46 ago on Sat 21 Sep 2024 12:11:25 PM UTC. 2024-09-21T12:13:11.957 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-21T12:13:11.957 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:13:11.957 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-09-21T12:13:11.958 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:13:11.958 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-21T12:13:11.958 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 170 k 2024-09-21T12:13:11.958 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:13:11.958 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-21T12:13:11.958 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-21T12:13:11.958 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-21T12:13:11.958 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:13:11.959 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 170 k 2024-09-21T12:13:11.959 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 490 k 2024-09-21T12:13:11.959 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-21T12:13:12.088 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:51 ago on Sat 21 Sep 2024 12:11:21 PM UTC. 2024-09-21T12:13:12.160 INFO:teuthology.orchestra.run.smithi114.stdout:rbd-nbd-19.3.0-5103.g5c1cdc3c.el9.x86_64.rpm 850 kB/s | 170 kB 00:00 2024-09-21T12:13:12.160 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:13:12.160 INFO:teuthology.orchestra.run.smithi114.stdout:Total 843 kB/s | 170 kB 00:00 2024-09-21T12:13:12.160 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-21T12:13:12.168 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-21T12:13:12.168 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-21T12:13:12.223 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-21T12:13:12.223 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:13:12.223 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-09-21T12:13:12.223 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:13:12.224 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-21T12:13:12.224 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 3.2 M 2024-09-21T12:13:12.224 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:13:12.224 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-21T12:13:12.224 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:13:12.224 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-09-21T12:13:12.224 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:13:12.224 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 3.2 M 2024-09-21T12:13:12.225 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 14 M 2024-09-21T12:13:12.225 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-21T12:13:12.233 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-21T12:13:12.233 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-21T12:13:12.385 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-21T12:13:12.490 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-nbd-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/1 2024-09-21T12:13:12.535 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-mirror-19.3.0-5103.g5c1cdc3c.el9.x86_64.rpm 10 MB/s | 3.2 MB 00:00 2024-09-21T12:13:12.536 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:13:12.536 INFO:teuthology.orchestra.run.smithi183.stdout:Total 10 MB/s | 3.2 MB 00:00 2024-09-21T12:13:12.536 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-21T12:13:12.546 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-21T12:13:12.546 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-21T12:13:12.628 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-21T12:13:12.629 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-21T12:13:12.924 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-nbd-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/1 2024-09-21T12:13:13.094 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-21T12:13:13.152 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-mirror-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/1 2024-09-21T12:13:13.179 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-mirror-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/1 2024-09-21T12:13:13.179 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T12:13:13.179 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T12:13:13.179 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T12:13:13.179 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T12:13:13.179 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:13:13.210 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-nbd-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/1 2024-09-21T12:13:13.210 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:13:13.210 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-21T12:13:13.210 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:13:13.210 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:13:13.210 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-21T12:13:13.455 DEBUG:teuthology.parallel:result is None 2024-09-21T12:13:14.287 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-mirror-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/1 2024-09-21T12:13:14.287 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:13:14.287 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-21T12:13:14.287 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:13:14.287 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:13:14.287 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-21T12:13:14.555 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-nbd 2024-09-21T12:13:15.155 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:54 ago on Sat 21 Sep 2024 12:11:21 PM UTC. 2024-09-21T12:13:15.287 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-21T12:13:15.288 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:13:15.288 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-09-21T12:13:15.288 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:13:15.288 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-21T12:13:15.288 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd x86_64 2:19.3.0-5103.g5c1cdc3c.el9 ceph 170 k 2024-09-21T12:13:15.289 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:13:15.289 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-21T12:13:15.289 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-21T12:13:15.289 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-09-21T12:13:15.289 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:13:15.289 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 170 k 2024-09-21T12:13:15.289 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 490 k 2024-09-21T12:13:15.289 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-21T12:13:15.518 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-nbd-19.3.0-5103.g5c1cdc3c.el9.x86_64.rpm 744 kB/s | 170 kB 00:00 2024-09-21T12:13:15.518 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:13:15.519 INFO:teuthology.orchestra.run.smithi183.stdout:Total 739 kB/s | 170 kB 00:00 2024-09-21T12:13:15.519 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-21T12:13:15.526 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-21T12:13:15.527 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-21T12:13:15.591 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-21T12:13:15.591 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-21T12:13:15.746 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-21T12:13:15.854 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-nbd-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/1 2024-09-21T12:13:16.334 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-nbd-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/1 2024-09-21T12:13:16.629 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-nbd-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 1/1 2024-09-21T12:13:16.630 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:13:16.630 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-21T12:13:16.630 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd-2:19.3.0-5103.g5c1cdc3c.el9.x86_64 2024-09-21T12:13:16.630 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:13:16.630 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-21T12:13:16.895 DEBUG:teuthology.parallel:result is None 2024-09-21T12:13:16.895 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5c1cdc3c470dae78e91a0ed2bcf13509a046d301 2024-09-21T12:13:17.038 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T12:13:17.064 INFO:teuthology.orchestra.run.smithi114.stdout:19.3.0-5103.g5c1cdc3c.el9 2024-09-21T12:13:17.064 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5103.g5c1cdc3c.el9 2024-09-21T12:13:17.064 INFO:teuthology.task.install:The correct ceph version 19.3.0-5103.g5c1cdc3c is installed. 2024-09-21T12:13:17.066 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5c1cdc3c470dae78e91a0ed2bcf13509a046d301 2024-09-21T12:13:17.205 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T12:13:17.229 INFO:teuthology.orchestra.run.smithi183.stdout:19.3.0-5103.g5c1cdc3c.el9 2024-09-21T12:13:17.229 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5103.g5c1cdc3c.el9 2024-09-21T12:13:17.229 INFO:teuthology.task.install:The correct ceph version 19.3.0-5103.g5c1cdc3c is installed. 2024-09-21T12:13:17.231 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-21T12:13:17.231 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-21T12:13:17.231 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T12:13:17.262 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-09-21T12:13:17.263 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T12:13:17.297 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-21T12:13:17.297 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-21T12:13:17.297 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T12:13:17.331 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T12:13:17.398 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-09-21T12:13:17.398 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T12:13:17.425 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T12:13:17.491 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-21T12:13:17.492 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-21T12:13:17.492 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T12:13:17.520 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T12:13:17.586 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-09-21T12:13:17.587 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T12:13:17.612 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T12:13:17.676 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-21T12:13:17.676 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-21T12:13:17.676 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T12:13:17.703 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T12:13:17.769 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-09-21T12:13:17.769 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T12:13:17.794 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T12:13:17.859 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-21T12:13:17.954 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'log-only-match': ['CEPHADM_'], 'sha1': '5c1cdc3c470dae78e91a0ed2bcf13509a046d301', 'cephadm_mode': 'root'} 2024-09-21T12:13:17.954 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 2024-09-21T12:13:17.954 INFO:tasks.cephadm:Cluster fsid is e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T12:13:17.954 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-21T12:13:17.955 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.114', 'mon.c': '[v2:172.21.15.114:3301,v1:172.21.15.114:6790]', 'mon.b': '172.21.15.183'} 2024-09-21T12:13:17.955 INFO:tasks.cephadm:First mon is mon.a on smithi114 2024-09-21T12:13:17.955 INFO:tasks.cephadm:First mgr is y 2024-09-21T12:13:17.955 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-21T12:13:17.955 DEBUG:teuthology.orchestra.run.smithi114:> sudo hostname $(hostname -s) 2024-09-21T12:13:17.982 DEBUG:teuthology.orchestra.run.smithi183:> sudo hostname $(hostname -s) 2024-09-21T12:13:18.007 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-21T12:13:18.007 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5c1cdc3c470dae78e91a0ed2bcf13509a046d301 2024-09-21T12:13:18.146 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '5c1cdc3c470dae78e91a0ed2bcf13509a046d301', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/83200/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-5103-g5c1cdc3c', 'node_name': '172.21.2.5+braggi05', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-5103.g5c1cdc3c'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-21-0813/5c1cdc3c470dae78e91a0ed2bcf13509a046d301/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-21 10:00:04.055809', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-guits-main-2024-09-21-0813', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-guits-main-2024-09-21-0813/5c1cdc3c470dae78e91a0ed2bcf13509a046d301/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-09-21T12:13:18.283 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-guits-main-2024-09-21-0813, sha1 5c1cdc3c470dae78e91a0ed2bcf13509a046d301 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=5c1cdc3c470dae78e91a0ed2bcf13509a046d301 2024-09-21T12:13:18.284 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-guits-main-2024-09-21-0813/5c1cdc3c470dae78e91a0ed2bcf13509a046d301/centos/9/x86_64/flavors/default/cephadm 2024-09-21T12:13:18.284 INFO:tasks.cephadm:Downloading cephadm from url: https://2.chacra.ceph.com/binaries/ceph/wip-guits-main-2024-09-21-0813/5c1cdc3c470dae78e91a0ed2bcf13509a046d301/centos/9/x86_64/flavors/default/cephadm 2024-09-21T12:13:18.285 DEBUG:teuthology.orchestra.run.smithi114:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-guits-main-2024-09-21-0813/5c1cdc3c470dae78e91a0ed2bcf13509a046d301/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T12:13:18.617 INFO:teuthology.orchestra.run.smithi114.stdout:-rw-r--r--. 1 ubuntu ubuntu 804189 Sep 21 12:13 /home/ubuntu/cephtest/cephadm 2024-09-21T12:13:18.618 DEBUG:teuthology.orchestra.run.smithi183:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-guits-main-2024-09-21-0813/5c1cdc3c470dae78e91a0ed2bcf13509a046d301/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T12:13:18.965 INFO:teuthology.orchestra.run.smithi183.stdout:-rw-r--r--. 1 ubuntu ubuntu 804189 Sep 21 12:13 /home/ubuntu/cephtest/cephadm 2024-09-21T12:13:18.965 DEBUG:teuthology.orchestra.run.smithi114:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T12:13:18.985 DEBUG:teuthology.orchestra.run.smithi183:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T12:13:19.018 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 on all hosts... 2024-09-21T12:13:19.018 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 pull 2024-09-21T12:13:19.029 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 pull 2024-09-21T12:13:20.582 INFO:teuthology.orchestra.run.smithi114.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301... 2024-09-21T12:13:20.719 INFO:teuthology.orchestra.run.smithi183.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301... 2024-09-21T12:14:08.009 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:14:08.009 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 19.3.0-5103-g5c1cdc3c (5c1cdc3c470dae78e91a0ed2bcf13509a046d301) squid (dev)", 2024-09-21T12:14:08.009 INFO:teuthology.orchestra.run.smithi114.stdout: "image_id": "8f4d013db793795fef09ff5edf96e6b42ba8cd2c9158b2f0b7a5d7c1e657c063", 2024-09-21T12:14:08.009 INFO:teuthology.orchestra.run.smithi114.stdout: "repo_digests": [ 2024-09-21T12:14:08.009 INFO:teuthology.orchestra.run.smithi114.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472" 2024-09-21T12:14:08.010 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:14:08.010 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:14:12.082 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:14:12.083 INFO:teuthology.orchestra.run.smithi183.stdout: "ceph_version": "ceph version 19.3.0-5103-g5c1cdc3c (5c1cdc3c470dae78e91a0ed2bcf13509a046d301) squid (dev)", 2024-09-21T12:14:12.083 INFO:teuthology.orchestra.run.smithi183.stdout: "image_id": "8f4d013db793795fef09ff5edf96e6b42ba8cd2c9158b2f0b7a5d7c1e657c063", 2024-09-21T12:14:12.083 INFO:teuthology.orchestra.run.smithi183.stdout: "repo_digests": [ 2024-09-21T12:14:12.083 INFO:teuthology.orchestra.run.smithi183.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472" 2024-09-21T12:14:12.083 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:14:12.083 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:14:12.109 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph 2024-09-21T12:14:12.143 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /etc/ceph 2024-09-21T12:14:12.170 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 777 /etc/ceph 2024-09-21T12:14:12.211 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 777 /etc/ceph 2024-09-21T12:14:12.235 INFO:tasks.cephadm:Writing seed config... 2024-09-21T12:14:12.236 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-21T12:14:12.236 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-21T12:14:12.236 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-09-21T12:14:12.236 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-09-21T12:14:12.236 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-09-21T12:14:12.237 INFO:tasks.cephadm: override: [global] ms type = async 2024-09-21T12:14:12.237 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-21T12:14:12.237 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-21T12:14:12.237 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-21T12:14:12.237 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-09-21T12:14:12.237 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-09-21T12:14:12.237 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-09-21T12:14:12.237 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-09-21T12:14:12.237 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-09-21T12:14:12.237 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-21T12:14:12.237 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-21T12:14:12.237 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-09-21T12:14:12.237 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-09-21T12:14:12.237 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-09-21T12:14:12.238 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-09-21T12:14:12.238 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-09-21T12:14:12.238 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-21T12:14:12.238 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-09-21T12:14:12.238 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-21T12:14:12.238 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-21T12:14:12.269 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = e2ad975c-7812-11ef-baf6-efdc52797490 ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-09-21T12:14:12.270 DEBUG:teuthology.orchestra.run.smithi114:mon.a> sudo journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@mon.a.service 2024-09-21T12:14:12.312 DEBUG:teuthology.orchestra.run.smithi114:mgr.y> sudo journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@mgr.y.service 2024-09-21T12:14:12.355 INFO:tasks.cephadm:Bootstrapping... 2024-09-21T12:14:12.356 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 -v bootstrap --fsid e2ad975c-7812-11ef-baf6-efdc52797490 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.114 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-21T12:14:12.563 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:14:12.563 INFO:teuthology.orchestra.run.smithi114.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301', '-v', 'bootstrap', '--fsid', 'e2ad975c-7812-11ef-baf6-efdc52797490', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.114', '--skip-admin-label'] 2024-09-21T12:14:12.563 INFO:teuthology.orchestra.run.smithi114.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-21T12:14:12.563 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying podman|docker is present... 2024-09-21T12:14:12.587 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stdout 5.2.2 2024-09-21T12:14:12.587 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying lvm2 is present... 2024-09-21T12:14:12.587 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying time synchronization is in place... 2024-09-21T12:14:12.595 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-21T12:14:12.595 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-21T12:14:12.601 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-21T12:14:12.601 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-09-21T12:14:12.608 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout enabled 2024-09-21T12:14:12.614 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout active 2024-09-21T12:14:12.614 INFO:teuthology.orchestra.run.smithi114.stdout:Unit chronyd.service is enabled and running 2024-09-21T12:14:12.614 INFO:teuthology.orchestra.run.smithi114.stdout:Repeating the final host check... 2024-09-21T12:14:12.636 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stdout 5.2.2 2024-09-21T12:14:12.636 INFO:teuthology.orchestra.run.smithi114.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-21T12:14:12.636 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl is present 2024-09-21T12:14:12.636 INFO:teuthology.orchestra.run.smithi114.stdout:lvcreate is present 2024-09-21T12:14:12.643 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-21T12:14:12.643 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-21T12:14:12.649 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-21T12:14:12.649 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-09-21T12:14:12.656 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout enabled 2024-09-21T12:14:12.662 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout active 2024-09-21T12:14:12.662 INFO:teuthology.orchestra.run.smithi114.stdout:Unit chronyd.service is enabled and running 2024-09-21T12:14:12.662 INFO:teuthology.orchestra.run.smithi114.stdout:Host looks OK 2024-09-21T12:14:12.662 INFO:teuthology.orchestra.run.smithi114.stdout:Cluster fsid: e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T12:14:12.663 INFO:teuthology.orchestra.run.smithi114.stdout:Acquiring lock 140650119305584 on /run/cephadm/e2ad975c-7812-11ef-baf6-efdc52797490.lock 2024-09-21T12:14:12.663 INFO:teuthology.orchestra.run.smithi114.stdout:Lock 140650119305584 acquired on /run/cephadm/e2ad975c-7812-11ef-baf6-efdc52797490.lock 2024-09-21T12:14:12.663 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying IP 172.21.15.114 port 3300 ... 2024-09-21T12:14:12.663 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying IP 172.21.15.114 port 6789 ... 2024-09-21T12:14:12.663 INFO:teuthology.orchestra.run.smithi114.stdout:Base mon IP(s) is [172.21.15.114:3300, 172.21.15.114:6789], mon addrv is [v2:172.21.15.114:3300,v1:172.21.15.114:6789] 2024-09-21T12:14:12.666 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.114 metric 100 2024-09-21T12:14:12.666 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.114 metric 100 2024-09-21T12:14:12.669 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-21T12:14:12.669 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-21T12:14:12.669 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-21T12:14:12.671 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-21T12:14:12.671 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-21T12:14:12.671 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-21T12:14:12.672 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-21T12:14:12.672 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link noprefixroute 2024-09-21T12:14:12.672 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-21T12:14:12.672 INFO:teuthology.orchestra.run.smithi114.stdout:Mon IP `172.21.15.114` is in CIDR network `172.21.0.0/20` 2024-09-21T12:14:12.672 INFO:teuthology.orchestra.run.smithi114.stdout:Mon IP `172.21.15.114` is in CIDR network `172.21.0.0/20` 2024-09-21T12:14:12.672 INFO:teuthology.orchestra.run.smithi114.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-21T12:14:12.673 INFO:teuthology.orchestra.run.smithi114.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-21T12:14:12.673 INFO:teuthology.orchestra.run.smithi114.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301... 2024-09-21T12:14:13.161 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stdout 8f4d013db793795fef09ff5edf96e6b42ba8cd2c9158b2f0b7a5d7c1e657c063 2024-09-21T12:14:13.161 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301... 2024-09-21T12:14:13.162 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Getting image source signatures 2024-09-21T12:14:13.162 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Copying blob sha256:832cb702be6ace10bd8c779318aa247eb0f26450f5c31f0789905499e59c463e 2024-09-21T12:14:13.162 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-21T12:14:13.162 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Copying config sha256:8f4d013db793795fef09ff5edf96e6b42ba8cd2c9158b2f0b7a5d7c1e657c063 2024-09-21T12:14:13.162 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-21T12:14:14.124 INFO:teuthology.orchestra.run.smithi114.stdout:ceph: stdout ceph version 19.3.0-5103-g5c1cdc3c (5c1cdc3c470dae78e91a0ed2bcf13509a046d301) squid (dev) 2024-09-21T12:14:14.124 INFO:teuthology.orchestra.run.smithi114.stdout:Ceph version: ceph version 19.3.0-5103-g5c1cdc3c (5c1cdc3c470dae78e91a0ed2bcf13509a046d301) squid (dev) 2024-09-21T12:14:14.124 INFO:teuthology.orchestra.run.smithi114.stdout:Extracting ceph user uid/gid from container image... 2024-09-21T12:14:15.039 INFO:teuthology.orchestra.run.smithi114.stdout:stat: stdout 167 167 2024-09-21T12:14:15.039 INFO:teuthology.orchestra.run.smithi114.stdout:Creating initial keys... 2024-09-21T12:14:15.970 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph-authtool: stdout AQCXuO5mstw8GRAAkDNCelaoAmVXqeiXXzLb7g== 2024-09-21T12:14:16.967 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph-authtool: stdout AQCYuO5myhEWExAA7KO9bAoY7sllus+O34dK3w== 2024-09-21T12:14:17.857 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph-authtool: stdout AQCZuO5mdpRyEBAANMHCB2yfpibtvxBLIAidyw== 2024-09-21T12:14:17.857 INFO:teuthology.orchestra.run.smithi114.stdout:Creating initial monmap... 2024-09-21T12:14:18.730 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-21T12:14:18.731 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-21T12:14:18.731 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T12:14:18.731 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-21T12:14:18.731 INFO:teuthology.orchestra.run.smithi114.stdout:monmaptool for a [v2:172.21.15.114:3300,v1:172.21.15.114:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-21T12:14:18.731 INFO:teuthology.orchestra.run.smithi114.stdout:setting min_mon_release = quincy 2024-09-21T12:14:18.731 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: set fsid to e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T12:14:18.731 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-21T12:14:18.731 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:14:18.731 INFO:teuthology.orchestra.run.smithi114.stdout:Creating mon... 2024-09-21T12:14:19.694 INFO:teuthology.orchestra.run.smithi114.stdout:create mon.a on 2024-09-21T12:14:19.906 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-21T12:14:20.051 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-21T12:14:20.233 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-e2ad975c-7812-11ef-baf6-efdc52797490.target → /etc/systemd/system/ceph-e2ad975c-7812-11ef-baf6-efdc52797490.target. 2024-09-21T12:14:20.233 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-e2ad975c-7812-11ef-baf6-efdc52797490.target → /etc/systemd/system/ceph-e2ad975c-7812-11ef-baf6-efdc52797490.target. 2024-09-21T12:14:20.464 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e2ad975c-7812-11ef-baf6-efdc52797490@mon.a 2024-09-21T12:14:20.464 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e2ad975c-7812-11ef-baf6-efdc52797490@mon.a.service: Unit ceph-e2ad975c-7812-11ef-baf6-efdc52797490@mon.a.service not loaded. 2024-09-21T12:14:20.619 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e2ad975c-7812-11ef-baf6-efdc52797490.target.wants/ceph-e2ad975c-7812-11ef-baf6-efdc52797490@mon.a.service → /etc/systemd/system/ceph-e2ad975c-7812-11ef-baf6-efdc52797490@.service. 2024-09-21T12:14:21.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:20 smithi114 podman[29816]: 2024-09-21 12:14:20.757054962 +0000 UTC m=+0.015245558 image pull 8f4d013db793795fef09ff5edf96e6b42ba8cd2c9158b2f0b7a5d7c1e657c063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 2024-09-21T12:14:22.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:22 smithi114 podman[29816]: 2024-09-21 12:14:22.114874502 +0000 UTC m=+1.373065135 container create 567edb8782b98ddf5c3384e630a6c69209264803ea65beb28e2f028b904f1400 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-a, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T12:14:24.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:24 smithi114 podman[29816]: 2024-09-21 12:14:24.132577657 +0000 UTC m=+3.390768254 container init 567edb8782b98ddf5c3384e630a6c69209264803ea65beb28e2f028b904f1400 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3) 2024-09-21T12:14:24.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:24 smithi114 podman[29816]: 2024-09-21 12:14:24.138402965 +0000 UTC m=+3.396593553 container start 567edb8782b98ddf5c3384e630a6c69209264803ea65beb28e2f028b904f1400 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-21T12:14:24.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:24 smithi114 ceph-mon[29853]: mkfs e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T12:14:24.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:24 smithi114 ceph-mon[29853]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T12:14:25.003 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T12:14:25.004 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-09-21T12:14:25.011 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T12:14:25.012 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-09-21T12:14:25.012 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-09-21T12:14:25.012 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to enable service . firewalld.service is not available 2024-09-21T12:14:25.012 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mon to start... 2024-09-21T12:14:25.012 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mon... 2024-09-21T12:14:25.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:24 smithi114 bash[29816]: 567edb8782b98ddf5c3384e630a6c69209264803ea65beb28e2f028b904f1400 2024-09-21T12:14:25.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:24 smithi114 systemd[1]: Started Ceph mon.a for e2ad975c-7812-11ef-baf6-efdc52797490. 2024-09-21T12:14:25.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:25 smithi114 ceph-mon[29853]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T12:14:25.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:25 smithi114 ceph-mon[29853]: monmap epoch 1 2024-09-21T12:14:25.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:25 smithi114 ceph-mon[29853]: fsid e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T12:14:25.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:25 smithi114 ceph-mon[29853]: last_changed 2024-09-21T12:14:18.247419+0000 2024-09-21T12:14:25.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:25 smithi114 ceph-mon[29853]: created 2024-09-21T12:14:18.247419+0000 2024-09-21T12:14:25.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:25 smithi114 ceph-mon[29853]: min_mon_release 19 (squid) 2024-09-21T12:14:25.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:25 smithi114 ceph-mon[29853]: election_strategy: 1 2024-09-21T12:14:25.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:25 smithi114 ceph-mon[29853]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-21T12:14:25.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:25 smithi114 ceph-mon[29853]: fsmap 2024-09-21T12:14:25.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:25 smithi114 ceph-mon[29853]: osdmap e1: 0 total, 0 up, 0 in 2024-09-21T12:14:25.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:25 smithi114 ceph-mon[29853]: mgrmap e1: no daemons active 2024-09-21T12:14:28.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:28 smithi114 ceph-mon[29853]: from='client.? 172.21.15.114:0/2452779624' entity='client.admin' cmd={"prefix": "status"} : dispatch 2024-09-21T12:14:32.900 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout cluster: 2024-09-21T12:14:32.900 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout id: e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T12:14:32.900 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-21T12:14:32.900 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-21T12:14:32.900 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout services: 2024-09-21T12:14:32.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 4s) 2024-09-21T12:14:32.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-21T12:14:32.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-21T12:14:32.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-21T12:14:32.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout data: 2024-09-21T12:14:32.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-21T12:14:32.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-21T12:14:32.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-21T12:14:32.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout pgs: 2024-09-21T12:14:32.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-21T12:14:32.901 INFO:teuthology.orchestra.run.smithi114.stdout:mon is available 2024-09-21T12:14:32.901 INFO:teuthology.orchestra.run.smithi114.stdout:Assimilating anything we can from ceph.conf... 2024-09-21T12:14:33.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:33 smithi114 ceph-mon[29853]: from='client.? 172.21.15.114:0/2814039023' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-21T12:14:33.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:33 smithi114 ceph-mon[29853]: from='client.? 172.21.15.114:0/2814039023' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-21T12:14:34.013 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-21T12:14:34.014 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [global] 2024-09-21T12:14:34.014 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout fsid = e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T12:14:34.014 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-21T12:14:34.014 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.114:3300,v1:172.21.15.114:6789] 2024-09-21T12:14:34.014 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-21T12:14:34.014 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-21T12:14:34.014 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-21T12:14:34.014 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-21T12:14:34.014 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-21T12:14:34.014 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-21T12:14:34.014 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-21T12:14:34.014 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-21T12:14:34.015 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [osd] 2024-09-21T12:14:34.015 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-21T12:14:34.015 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-21T12:14:34.015 INFO:teuthology.orchestra.run.smithi114.stdout:Generating new minimal ceph.conf... 2024-09-21T12:14:34.732 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:34 smithi114 ceph-mon[29853]: from='client.? 172.21.15.114:0/2335296726' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:14:34.961 INFO:teuthology.orchestra.run.smithi114.stdout:Restarting the monitor... 2024-09-21T12:14:35.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:34 smithi114 systemd[1]: Stopping Ceph mon.a for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:14:35.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:35 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-a[29830]: 2024-09-21T12:14:35.236+0000 7feea6101640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T12:14:35.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:35 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-a[29830]: 2024-09-21T12:14:35.236+0000 7feea6101640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-21T12:14:35.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:35 smithi114 podman[30172]: 2024-09-21 12:14:35.492662717 +0000 UTC m=+0.361630066 container died 567edb8782b98ddf5c3384e630a6c69209264803ea65beb28e2f028b904f1400 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-21T12:14:36.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:35 smithi114 podman[30172]: 2024-09-21 12:14:35.851258398 +0000 UTC m=+0.720225735 container cleanup 567edb8782b98ddf5c3384e630a6c69209264803ea65beb28e2f028b904f1400 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3) 2024-09-21T12:14:36.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:35 smithi114 bash[30172]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-a 2024-09-21T12:14:36.462 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 podman[30188]: 2024-09-21 12:14:36.110801306 +0000 UTC m=+0.615421846 container remove 567edb8782b98ddf5c3384e630a6c69209264803ea65beb28e2f028b904f1400 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-a, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=) 2024-09-21T12:14:36.463 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 systemd[1]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-21T12:14:36.463 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 systemd[1]: Stopped Ceph mon.a for e2ad975c-7812-11ef-baf6-efdc52797490. 2024-09-21T12:14:36.463 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 systemd[1]: Starting Ceph mon.a for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:14:36.723 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 podman[30286]: 2024-09-21 12:14:36.461564301 +0000 UTC m=+0.103872914 container create 19957648bd4ed923f5f31e20868af18b01a8a96b64d2d0418433545927aa7586 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-21T12:14:36.723 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 podman[30286]: 2024-09-21 12:14:36.37329125 +0000 UTC m=+0.015599884 image pull 8f4d013db793795fef09ff5edf96e6b42ba8cd2c9158b2f0b7a5d7c1e657c063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 2024-09-21T12:14:36.723 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 podman[30286]: 2024-09-21 12:14:36.647220285 +0000 UTC m=+0.289528909 container init 19957648bd4ed923f5f31e20868af18b01a8a96b64d2d0418433545927aa7586 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-a, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2) 2024-09-21T12:14:36.724 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 podman[30286]: 2024-09-21 12:14:36.652093853 +0000 UTC m=+0.294402470 container start 19957648bd4ed923f5f31e20868af18b01a8a96b64d2d0418433545927aa7586 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-a, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True) 2024-09-21T12:14:36.724 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T12:14:36.724 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: ceph version 19.3.0-5103-g5c1cdc3c (5c1cdc3c470dae78e91a0ed2bcf13509a046d301) squid (dev), process ceph-mon, pid 7 2024-09-21T12:14:36.724 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: pidfile_write: ignore empty --pid-file 2024-09-21T12:14:36.724 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: load: jerasure load: lrc 2024-09-21T12:14:36.724 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: RocksDB version: 7.9.2 2024-09-21T12:14:36.724 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Git sha 0 2024-09-21T12:14:36.724 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Compile date 2024-09-21 08:36:07 2024-09-21T12:14:36.724 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: DB SUMMARY 2024-09-21T12:14:36.724 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: DB Session ID: 17TZY0OVIBPBOS3MGHXK 2024-09-21T12:14:36.724 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: CURRENT file: CURRENT 2024-09-21T12:14:36.724 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T12:14:36.724 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-21T12:14:36.725 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-21T12:14:36.725 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 95331 ; 2024-09-21T12:14:36.725 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.error_if_exists: 0 2024-09-21T12:14:36.725 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.create_if_missing: 0 2024-09-21T12:14:36.725 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.paranoid_checks: 1 2024-09-21T12:14:36.725 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T12:14:36.725 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T12:14:36.725 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T12:14:36.725 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.env: 0x556d9cda5120 2024-09-21T12:14:36.725 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T12:14:36.725 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.info_log: 0x556d9ddc7700 2024-09-21T12:14:36.725 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T12:14:36.725 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.statistics: (nil) 2024-09-21T12:14:36.726 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.use_fsync: 0 2024-09-21T12:14:36.726 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.max_log_file_size: 0 2024-09-21T12:14:36.726 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T12:14:36.726 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T12:14:36.726 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T12:14:36.726 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T12:14:36.726 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.allow_fallocate: 1 2024-09-21T12:14:36.726 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T12:14:36.726 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T12:14:36.726 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.use_direct_reads: 0 2024-09-21T12:14:36.726 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T12:14:36.726 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T12:14:36.726 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.db_log_dir: 2024-09-21T12:14:36.727 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.wal_dir: 2024-09-21T12:14:36.727 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T12:14:36.728 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T12:14:36.728 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T12:14:36.728 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T12:14:36.728 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T12:14:36.728 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T12:14:36.728 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T12:14:36.728 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T12:14:36.728 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.write_buffer_manager: 0x556d9ddcbb80 2024-09-21T12:14:36.728 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T12:14:36.729 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T12:14:36.729 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T12:14:36.729 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T12:14:36.729 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T12:14:36.730 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T12:14:36.730 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T12:14:36.730 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T12:14:36.730 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.unordered_write: 0 2024-09-21T12:14:36.730 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T12:14:36.730 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T12:14:36.730 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T12:14:36.730 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T12:14:36.730 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.row_cache: None 2024-09-21T12:14:36.731 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.wal_filter: None 2024-09-21T12:14:36.731 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T12:14:36.731 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T12:14:36.731 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.two_write_queues: 0 2024-09-21T12:14:36.731 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T12:14:36.731 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.wal_compression: 0 2024-09-21T12:14:36.731 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.atomic_flush: 0 2024-09-21T12:14:36.731 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T12:14:36.731 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T12:14:36.731 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T12:14:36.731 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.log_readahead_size: 0 2024-09-21T12:14:36.731 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T12:14:36.731 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T12:14:36.732 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T12:14:36.732 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T12:14:36.732 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T12:14:36.732 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T12:14:36.732 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T12:14:36.732 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.max_background_jobs: 2 2024-09-21T12:14:36.732 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.max_background_compactions: -1 2024-09-21T12:14:36.732 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.max_subcompactions: 1 2024-09-21T12:14:36.732 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T12:14:36.732 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T12:14:36.732 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T12:14:36.732 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T12:14:36.732 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T12:14:36.732 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T12:14:36.732 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T12:14:36.733 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T12:14:36.733 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.max_open_files: -1 2024-09-21T12:14:36.733 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T12:14:36.733 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T12:14:36.733 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T12:14:36.733 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T12:14:36.733 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.max_background_flushes: -1 2024-09-21T12:14:36.733 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Compression algorithms supported: 2024-09-21T12:14:36.733 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: kZSTD supported: 0 2024-09-21T12:14:36.733 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: kXpressCompression supported: 0 2024-09-21T12:14:36.733 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: kBZip2Compression supported: 0 2024-09-21T12:14:36.733 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T12:14:36.733 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: kLZ4Compression supported: 1 2024-09-21T12:14:36.733 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: kZlibCompression supported: 1 2024-09-21T12:14:36.734 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T12:14:36.734 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: kSnappyCompression supported: 1 2024-09-21T12:14:36.734 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T12:14:36.734 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T12:14:36.734 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-21T12:14:36.734 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T12:14:36.734 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T12:14:36.734 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.merge_operator: 2024-09-21T12:14:36.734 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.compaction_filter: None 2024-09-21T12:14:36.734 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T12:14:36.734 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T12:14:36.734 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T12:14:36.734 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T12:14:36.734 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556d9ddc7320) 2024-09-21T12:14:36.734 INFO:journalctl@ceph.mon.a.smithi114.stdout: cache_index_and_filter_blocks: 1 2024-09-21T12:14:36.735 INFO:journalctl@ceph.mon.a.smithi114.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T12:14:36.735 INFO:journalctl@ceph.mon.a.smithi114.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T12:14:36.735 INFO:journalctl@ceph.mon.a.smithi114.stdout: pin_top_level_index_and_filter: 1 2024-09-21T12:14:36.735 INFO:journalctl@ceph.mon.a.smithi114.stdout: index_type: 0 2024-09-21T12:14:36.735 INFO:journalctl@ceph.mon.a.smithi114.stdout: data_block_index_type: 0 2024-09-21T12:14:36.735 INFO:journalctl@ceph.mon.a.smithi114.stdout: index_shortening: 1 2024-09-21T12:14:36.735 INFO:journalctl@ceph.mon.a.smithi114.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T12:14:36.735 INFO:journalctl@ceph.mon.a.smithi114.stdout: checksum: 4 2024-09-21T12:14:36.735 INFO:journalctl@ceph.mon.a.smithi114.stdout: no_block_cache: 0 2024-09-21T12:14:36.735 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_cache: 0x556d9ddbb610 2024-09-21T12:14:36.735 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_cache_name: BinnedLRUCache 2024-09-21T12:14:36.735 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_cache_options: 2024-09-21T12:14:36.735 INFO:journalctl@ceph.mon.a.smithi114.stdout: capacity : 536870912 2024-09-21T12:14:36.735 INFO:journalctl@ceph.mon.a.smithi114.stdout: num_shard_bits : 4 2024-09-21T12:14:36.736 INFO:journalctl@ceph.mon.a.smithi114.stdout: strict_capacity_limit : 0 2024-09-21T12:14:36.736 INFO:journalctl@ceph.mon.a.smithi114.stdout: high_pri_pool_ratio: 0.000 2024-09-21T12:14:36.736 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_cache_compressed: (nil) 2024-09-21T12:14:36.736 INFO:journalctl@ceph.mon.a.smithi114.stdout: persistent_cache: (nil) 2024-09-21T12:14:36.736 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_size: 4096 2024-09-21T12:14:36.736 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_size_deviation: 10 2024-09-21T12:14:36.736 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_restart_interval: 16 2024-09-21T12:14:36.736 INFO:journalctl@ceph.mon.a.smithi114.stdout: index_block_restart_interval: 1 2024-09-21T12:14:36.736 INFO:journalctl@ceph.mon.a.smithi114.stdout: metadata_block_size: 4096 2024-09-21T12:14:36.736 INFO:journalctl@ceph.mon.a.smithi114.stdout: partition_filters: 0 2024-09-21T12:14:36.736 INFO:journalctl@ceph.mon.a.smithi114.stdout: use_delta_encoding: 1 2024-09-21T12:14:36.736 INFO:journalctl@ceph.mon.a.smithi114.stdout: filter_policy: bloomfilter 2024-09-21T12:14:36.736 INFO:journalctl@ceph.mon.a.smithi114.stdout: whole_key_filtering: 1 2024-09-21T12:14:36.736 INFO:journalctl@ceph.mon.a.smithi114.stdout: verify_compression: 0 2024-09-21T12:14:36.736 INFO:journalctl@ceph.mon.a.smithi114.stdout: read_amp_bytes_per_bit: 0 2024-09-21T12:14:36.737 INFO:journalctl@ceph.mon.a.smithi114.stdout: format_version: 5 2024-09-21T12:14:36.737 INFO:journalctl@ceph.mon.a.smithi114.stdout: enable_index_compression: 1 2024-09-21T12:14:36.737 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_align: 0 2024-09-21T12:14:36.737 INFO:journalctl@ceph.mon.a.smithi114.stdout: max_auto_readahead_size: 262144 2024-09-21T12:14:36.737 INFO:journalctl@ceph.mon.a.smithi114.stdout: prepopulate_block_cache: 0 2024-09-21T12:14:36.737 INFO:journalctl@ceph.mon.a.smithi114.stdout: initial_auto_readahead_size: 8192 2024-09-21T12:14:36.737 INFO:journalctl@ceph.mon.a.smithi114.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T12:14:36.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T12:14:36.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T12:14:36.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.compression: NoCompression 2024-09-21T12:14:36.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T12:14:36.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T12:14:36.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T12:14:36.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.num_levels: 7 2024-09-21T12:14:36.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T12:14:36.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T12:14:36.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T12:14:36.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T12:14:36.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T12:14:36.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T12:14:36.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T12:14:36.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T12:14:36.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T12:14:36.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T12:14:36.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T12:14:36.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T12:14:36.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T12:14:36.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T12:14:36.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T12:14:36.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T12:14:36.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T12:14:36.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T12:14:36.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T12:14:36.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T12:14:36.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T12:14:36.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T12:14:36.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T12:14:36.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T12:14:36.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T12:14:36.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T12:14:36.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T12:14:36.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T12:14:36.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T12:14:36.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T12:14:36.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T12:14:36.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T12:14:36.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T12:14:36.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T12:14:36.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T12:14:36.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T12:14:36.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T12:14:36.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T12:14:36.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T12:14:36.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T12:14:36.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T12:14:36.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T12:14:36.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T12:14:36.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T12:14:36.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T12:14:36.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T12:14:36.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T12:14:36.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T12:14:36.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T12:14:36.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T12:14:36.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T12:14:36.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T12:14:36.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T12:14:36.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T12:14:36.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.inplace_update_support: 0 2024-09-21T12:14:36.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T12:14:36.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T12:14:36.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T12:14:36.741 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T12:14:36.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.bloom_locality: 0 2024-09-21T12:14:36.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.max_successive_merges: 0 2024-09-21T12:14:36.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T12:14:36.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T12:14:36.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T12:14:36.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T12:14:36.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.ttl: 2592000 2024-09-21T12:14:36.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T12:14:36.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T12:14:36.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T12:14:36.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.enable_blob_files: false 2024-09-21T12:14:36.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.min_blob_size: 0 2024-09-21T12:14:36.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T12:14:36.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T12:14:36.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T12:14:36.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T12:14:36.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T12:14:36.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T12:14:36.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T12:14:36.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T12:14:36.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-21T12:14:36.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-21T12:14:36.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f65224bc-d271-42fb-8516-2bae4931118d 2024-09-21T12:14:36.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726920876677631, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-21T12:14:36.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-21T12:14:36.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726920876678537, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 91240, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 291, "table_properties": {"data_size": 89285, "index_size": 275, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11728, "raw_average_key_size": 47, "raw_value_size": 82766, "raw_average_value_size": 336, "num_data_blocks": 12, "num_entries": 246, "num_filter_entries": 246, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726920876, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f65224bc-d271-42fb-8516-2bae4931118d", "db_session_id": "17TZY0OVIBPBOS3MGHXK", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-21T12:14:36.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726920876678629, "job": 1, "event": "recovery_finished"} 2024-09-21T12:14:36.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-21T12:14:36.743 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T12:14:36.744 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556d9ddeae00 2024-09-21T12:14:36.744 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: DB pointer 0x556d9deec000 2024-09-21T12:14:36.744 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T12:14:36.744 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T12:14:36.744 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** DB Stats ** 2024-09-21T12:14:36.744 INFO:journalctl@ceph.mon.a.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T12:14:36.744 INFO:journalctl@ceph.mon.a.smithi114.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T12:14:36.744 INFO:journalctl@ceph.mon.a.smithi114.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T12:14:36.744 INFO:journalctl@ceph.mon.a.smithi114.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T12:14:36.744 INFO:journalctl@ceph.mon.a.smithi114.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T12:14:36.744 INFO:journalctl@ceph.mon.a.smithi114.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T12:14:36.744 INFO:journalctl@ceph.mon.a.smithi114.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T12:14:36.744 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-09-21T12:14:36.744 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** Compaction Stats [default] ** 2024-09-21T12:14:36.745 INFO:journalctl@ceph.mon.a.smithi114.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T12:14:36.745 INFO:journalctl@ceph.mon.a.smithi114.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T12:14:36.745 INFO:journalctl@ceph.mon.a.smithi114.stdout: L0 2/0 90.96 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T12:14:36.745 INFO:journalctl@ceph.mon.a.smithi114.stdout: Sum 2/0 90.96 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T12:14:36.745 INFO:journalctl@ceph.mon.a.smithi114.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T12:14:36.745 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-09-21T12:14:36.745 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** Compaction Stats [default] ** 2024-09-21T12:14:36.745 INFO:journalctl@ceph.mon.a.smithi114.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T12:14:36.745 INFO:journalctl@ceph.mon.a.smithi114.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T12:14:36.745 INFO:journalctl@ceph.mon.a.smithi114.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 149.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T12:14:36.745 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-09-21T12:14:36.745 INFO:journalctl@ceph.mon.a.smithi114.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T12:14:36.745 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-09-21T12:14:36.745 INFO:journalctl@ceph.mon.a.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T12:14:36.745 INFO:journalctl@ceph.mon.a.smithi114.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T12:14:36.746 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T12:14:36.746 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T12:14:36.746 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T12:14:36.746 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T12:14:36.746 INFO:journalctl@ceph.mon.a.smithi114.stdout: Cumulative compaction: 0.00 GB write, 28.77 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T12:14:36.746 INFO:journalctl@ceph.mon.a.smithi114.stdout: Interval compaction: 0.00 GB write, 28.77 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T12:14:36.746 INFO:journalctl@ceph.mon.a.smithi114.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T12:14:36.746 INFO:journalctl@ceph.mon.a.smithi114.stdout: Block cache BinnedLRUCache@0x556d9ddbb610#7 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-21T12:14:36.746 INFO:journalctl@ceph.mon.a.smithi114.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-09-21T12:14:36.746 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-09-21T12:14:36.746 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T12:14:36.746 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: starting mon.a rank 0 at public addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] at bind addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T12:14:36.746 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: mon.a@-1(???) e1 preinit fsid e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T12:14:36.746 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: mon.a@-1(???).mds e1 new map 2024-09-21T12:14:36.746 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: mon.a@-1(???).mds e1 print_map 2024-09-21T12:14:36.747 INFO:journalctl@ceph.mon.a.smithi114.stdout: e1 2024-09-21T12:14:36.747 INFO:journalctl@ceph.mon.a.smithi114.stdout: btime 2024-09-21T12:14:24:181010+0000 2024-09-21T12:14:36.747 INFO:journalctl@ceph.mon.a.smithi114.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T12:14:36.747 INFO:journalctl@ceph.mon.a.smithi114.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-21T12:14:36.747 INFO:journalctl@ceph.mon.a.smithi114.stdout: legacy client fscid: -1 2024-09-21T12:14:36.747 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-09-21T12:14:36.747 INFO:journalctl@ceph.mon.a.smithi114.stdout: No filesystems configured 2024-09-21T12:14:36.747 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T12:14:36.747 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T12:14:36.747 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T12:14:36.747 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T12:14:36.747 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-21T12:14:36.747 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 bash[30286]: 19957648bd4ed923f5f31e20868af18b01a8a96b64d2d0418433545927aa7586 2024-09-21T12:14:36.747 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 systemd[1]: Started Ceph mon.a for e2ad975c-7812-11ef-baf6-efdc52797490. 2024-09-21T12:14:36.748 INFO:teuthology.orchestra.run.smithi114.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-21T12:14:36.997 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T12:14:36.997 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: monmap epoch 1 2024-09-21T12:14:36.997 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: fsid e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T12:14:36.997 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: last_changed 2024-09-21T12:14:18.247419+0000 2024-09-21T12:14:36.997 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: created 2024-09-21T12:14:18.247419+0000 2024-09-21T12:14:36.997 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: min_mon_release 19 (squid) 2024-09-21T12:14:36.998 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: election_strategy: 1 2024-09-21T12:14:36.998 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-21T12:14:36.998 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: fsmap 2024-09-21T12:14:36.998 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: osdmap e1: 0 total, 0 up, 0 in 2024-09-21T12:14:36.998 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:36 smithi114 ceph-mon[30324]: mgrmap e1: no daemons active 2024-09-21T12:14:37.765 INFO:teuthology.orchestra.run.smithi114.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-21T12:14:37.779 INFO:teuthology.orchestra.run.smithi114.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-21T12:14:37.779 INFO:teuthology.orchestra.run.smithi114.stdout:Creating mgr... 2024-09-21T12:14:37.779 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-21T12:14:37.779 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-21T12:14:37.999 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e2ad975c-7812-11ef-baf6-efdc52797490@mgr.y 2024-09-21T12:14:37.999 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e2ad975c-7812-11ef-baf6-efdc52797490@mgr.y.service: Unit ceph-e2ad975c-7812-11ef-baf6-efdc52797490@mgr.y.service not loaded. 2024-09-21T12:14:38.271 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e2ad975c-7812-11ef-baf6-efdc52797490.target.wants/ceph-e2ad975c-7812-11ef-baf6-efdc52797490@mgr.y.service → /etc/systemd/system/ceph-e2ad975c-7812-11ef-baf6-efdc52797490@.service. 2024-09-21T12:14:38.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:38 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4014126729' entity='client.admin' 2024-09-21T12:14:38.525 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:14:38 smithi114 systemd[1]: Starting Ceph mgr.y for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:14:38.868 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:14:38 smithi114 podman[30561]: 2024-09-21 12:14:38.468361006 +0000 UTC m=+0.015597309 image pull 8f4d013db793795fef09ff5edf96e6b42ba8cd2c9158b2f0b7a5d7c1e657c063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 2024-09-21T12:14:38.868 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:14:38 smithi114 podman[30561]: 2024-09-21 12:14:38.759689484 +0000 UTC m=+0.306925826 container create b5b5207883523b18469b1bbaa814349423722e255898bc59245c63c90d67c322 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-21T12:14:39.097 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T12:14:39.097 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-09-21T12:14:39.104 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T12:14:39.104 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-09-21T12:14:39.104 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-09-21T12:14:39.104 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to enable service . firewalld.service is not available 2024-09-21T12:14:39.114 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T12:14:39.114 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-09-21T12:14:39.122 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T12:14:39.122 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-09-21T12:14:39.122 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-09-21T12:14:39.122 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-21T12:14:39.122 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr to start... 2024-09-21T12:14:39.122 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr... 2024-09-21T12:14:39.248 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:14:38 smithi114 podman[30561]: 2024-09-21 12:14:38.97032504 +0000 UTC m=+0.517561343 container init b5b5207883523b18469b1bbaa814349423722e255898bc59245c63c90d67c322 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-21T12:14:39.248 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:14:38 smithi114 podman[30561]: 2024-09-21 12:14:38.975551442 +0000 UTC m=+0.522787740 container start b5b5207883523b18469b1bbaa814349423722e255898bc59245c63c90d67c322 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-21T12:14:39.248 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:14:39 smithi114 bash[30561]: b5b5207883523b18469b1bbaa814349423722e255898bc59245c63c90d67c322 2024-09-21T12:14:39.248 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:14:39 smithi114 systemd[1]: Started Ceph mgr.y for e2ad975c-7812-11ef-baf6-efdc52797490. 2024-09-21T12:14:39.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:39 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2528432439' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-21T12:14:40.248 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-21T12:14:40.253 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-09-21T12:14:40.253 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsid": "e2ad975c-7812-11ef-baf6-efdc52797490", 2024-09-21T12:14:40.253 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T12:14:40.253 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T12:14:40.253 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T12:14:40.253 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T12:14:40.253 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:14:40.254 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T12:14:40.254 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T12:14:40.254 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 0 2024-09-21T12:14:40.254 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-21T12:14:40.254 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T12:14:40.254 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T12:14:40.254 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-21T12:14:40.254 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-21T12:14:40.254 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T12:14:40.254 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T12:14:40.254 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-21T12:14:40.254 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T12:14:40.254 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:14:40.254 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T12:14:40.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T12:14:40.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T12:14:40.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T12:14:40.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T12:14:40.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T12:14:40.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T12:14:40.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T12:14:40.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:14:40.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T12:14:40.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T12:14:40.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T12:14:40.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T12:14:40.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T12:14:40.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T12:14:40.256 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T12:14:40.256 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T12:14:40.256 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T12:14:40.256 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:14:40.256 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T12:14:40.256 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T12:14:40.256 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "btime": "2024-09-21T12:14:24:181010+0000", 2024-09-21T12:14:40.256 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T12:14:40.256 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T12:14:40.256 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:14:40.256 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T12:14:40.256 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-21T12:14:40.256 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T12:14:40.256 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T12:14:40.256 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T12:14:40.256 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T12:14:40.257 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T12:14:40.257 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-21T12:14:40.257 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T12:14:40.257 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:14:40.257 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T12:14:40.257 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T12:14:40.258 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T12:14:24.181726+0000", 2024-09-21T12:14:40.258 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T12:14:40.258 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:14:40.258 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T12:14:40.258 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-09-21T12:14:40.258 INFO:teuthology.orchestra.run.smithi114.stdout:mgr not available, waiting (1/15)... 2024-09-21T12:14:43.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:42 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2494757700' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-21T12:14:43.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-21T12:14:43.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-09-21T12:14:43.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsid": "e2ad975c-7812-11ef-baf6-efdc52797490", 2024-09-21T12:14:43.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T12:14:43.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T12:14:43.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T12:14:43.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T12:14:43.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:14:43.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T12:14:43.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T12:14:43.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 0 2024-09-21T12:14:43.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-21T12:14:43.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T12:14:43.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T12:14:43.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-21T12:14:43.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-09-21T12:14:43.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T12:14:43.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T12:14:43.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-21T12:14:43.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T12:14:43.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:14:43.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T12:14:43.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T12:14:43.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T12:14:43.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T12:14:43.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T12:14:43.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T12:14:43.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T12:14:43.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T12:14:43.396 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:14:43.396 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T12:14:43.396 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T12:14:43.396 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T12:14:43.396 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T12:14:43.396 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T12:14:43.397 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T12:14:43.397 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T12:14:43.397 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T12:14:43.397 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T12:14:43.397 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:14:43.397 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T12:14:43.397 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T12:14:43.397 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "btime": "2024-09-21T12:14:24:181010+0000", 2024-09-21T12:14:43.397 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T12:14:43.397 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T12:14:43.397 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:14:43.397 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T12:14:43.397 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-21T12:14:43.397 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T12:14:43.398 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T12:14:43.398 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T12:14:43.398 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T12:14:43.398 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T12:14:43.398 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-21T12:14:43.398 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T12:14:43.398 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:14:43.398 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T12:14:43.398 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T12:14:43.398 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T12:14:24.181726+0000", 2024-09-21T12:14:43.399 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T12:14:43.399 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:14:43.399 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T12:14:43.399 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-09-21T12:14:43.399 INFO:teuthology.orchestra.run.smithi114.stdout:mgr not available, waiting (2/15)... 2024-09-21T12:14:44.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:44 smithi114 ceph-mon[30324]: Activating manager daemon y 2024-09-21T12:14:44.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:44 smithi114 ceph-mon[30324]: mgrmap e2: y(active, starting, since 0.00337444s) 2024-09-21T12:14:44.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:44 smithi114 ceph-mon[30324]: from='mgr.14100 172.21.15.114:0/3336567903' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T12:14:44.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:44 smithi114 ceph-mon[30324]: from='mgr.14100 172.21.15.114:0/3336567903' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T12:14:44.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:44 smithi114 ceph-mon[30324]: from='mgr.14100 172.21.15.114:0/3336567903' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T12:14:44.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:44 smithi114 ceph-mon[30324]: from='mgr.14100 172.21.15.114:0/3336567903' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T12:14:44.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:44 smithi114 ceph-mon[30324]: from='mgr.14100 172.21.15.114:0/3336567903' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T12:14:44.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:44 smithi114 ceph-mon[30324]: Manager daemon y is now available 2024-09-21T12:14:44.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:44 smithi114 ceph-mon[30324]: from='mgr.14100 172.21.15.114:0/3336567903' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T12:14:44.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:44 smithi114 ceph-mon[30324]: from='mgr.14100 172.21.15.114:0/3336567903' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T12:14:44.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:44 smithi114 ceph-mon[30324]: from='mgr.14100 172.21.15.114:0/3336567903' entity='mgr.y' 2024-09-21T12:14:44.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:44 smithi114 ceph-mon[30324]: from='mgr.14100 172.21.15.114:0/3336567903' entity='mgr.y' 2024-09-21T12:14:44.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:44 smithi114 ceph-mon[30324]: from='mgr.14100 172.21.15.114:0/3336567903' entity='mgr.y' 2024-09-21T12:14:46.061 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:45 smithi114 ceph-mon[30324]: mgrmap e3: y(active, since 1.00734s) 2024-09-21T12:14:46.512 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-21T12:14:46.513 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-09-21T12:14:46.513 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsid": "e2ad975c-7812-11ef-baf6-efdc52797490", 2024-09-21T12:14:46.513 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T12:14:46.513 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T12:14:46.513 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T12:14:46.513 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T12:14:46.513 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:14:46.513 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T12:14:46.513 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T12:14:46.513 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 0 2024-09-21T12:14:46.513 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-21T12:14:46.513 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T12:14:46.513 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T12:14:46.514 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-21T12:14:46.514 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-09-21T12:14:46.514 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T12:14:46.514 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T12:14:46.514 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-21T12:14:46.514 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T12:14:46.514 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:14:46.514 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T12:14:46.514 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T12:14:46.514 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T12:14:46.514 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T12:14:46.514 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T12:14:46.514 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T12:14:46.514 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T12:14:46.515 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T12:14:46.518 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:14:46.518 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T12:14:46.518 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T12:14:46.518 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T12:14:46.518 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T12:14:46.518 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T12:14:46.518 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T12:14:46.518 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T12:14:46.518 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T12:14:46.518 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T12:14:46.519 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:14:46.519 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T12:14:46.519 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T12:14:46.519 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "btime": "2024-09-21T12:14:24:181010+0000", 2024-09-21T12:14:46.519 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T12:14:46.519 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T12:14:46.519 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:14:46.519 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T12:14:46.519 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T12:14:46.519 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T12:14:46.519 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T12:14:46.519 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T12:14:46.519 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T12:14:46.520 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T12:14:46.520 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-21T12:14:46.520 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T12:14:46.520 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:14:46.520 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T12:14:46.520 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T12:14:46.520 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T12:14:24.181726+0000", 2024-09-21T12:14:46.520 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T12:14:46.520 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:14:46.520 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T12:14:46.520 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-09-21T12:14:46.520 INFO:teuthology.orchestra.run.smithi114.stdout:mgr is available 2024-09-21T12:14:47.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:46 smithi114 ceph-mon[30324]: mgrmap e4: y(active, since 2s) 2024-09-21T12:14:47.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:46 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/261369073' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-21T12:14:47.685 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-21T12:14:47.685 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [global] 2024-09-21T12:14:47.685 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout fsid = e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T12:14:47.685 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-21T12:14:47.685 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.114:3300,v1:172.21.15.114:6789] 2024-09-21T12:14:47.685 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-21T12:14:47.685 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-21T12:14:47.685 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-21T12:14:47.685 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-21T12:14:47.686 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-21T12:14:47.686 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-21T12:14:47.686 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-21T12:14:47.686 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-21T12:14:47.686 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [osd] 2024-09-21T12:14:47.686 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-21T12:14:47.686 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-21T12:14:47.686 INFO:teuthology.orchestra.run.smithi114.stdout:Enabling cephadm module... 2024-09-21T12:14:48.116 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:47 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/628993260' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-21T12:14:49.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:48 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/433097116' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "cephadm"} : dispatch 2024-09-21T12:14:49.275 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:14:49 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ignoring --setuser ceph since I am not root 2024-09-21T12:14:49.275 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:14:49 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ignoring --setgroup ceph since I am not root 2024-09-21T12:14:50.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:49 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/433097116' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-21T12:14:50.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:49 smithi114 ceph-mon[30324]: mgrmap e5: y(active, since 5s) 2024-09-21T12:14:50.833 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-09-21T12:14:50.833 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-21T12:14:50.833 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T12:14:50.833 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-21T12:14:50.833 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-21T12:14:50.833 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-09-21T12:14:50.833 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for the mgr to restart... 2024-09-21T12:14:50.833 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr epoch 5... 2024-09-21T12:14:51.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:50 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/306966359' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-21T12:14:54.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:54 smithi114 ceph-mon[30324]: Active manager daemon y restarted 2024-09-21T12:14:54.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:54 smithi114 ceph-mon[30324]: Activating manager daemon y 2024-09-21T12:14:54.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:54 smithi114 ceph-mon[30324]: osdmap e2: 0 total, 0 up, 0 in 2024-09-21T12:14:54.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:54 smithi114 ceph-mon[30324]: mgrmap e6: y(active, starting, since 0.0033829s) 2024-09-21T12:14:54.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:54 smithi114 ceph-mon[30324]: from='mgr.14118 172.21.15.114:0/1482034044' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T12:14:54.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:54 smithi114 ceph-mon[30324]: from='mgr.14118 172.21.15.114:0/1482034044' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T12:14:54.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:54 smithi114 ceph-mon[30324]: from='mgr.14118 172.21.15.114:0/1482034044' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T12:14:54.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:54 smithi114 ceph-mon[30324]: from='mgr.14118 172.21.15.114:0/1482034044' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T12:14:54.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:54 smithi114 ceph-mon[30324]: from='mgr.14118 172.21.15.114:0/1482034044' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T12:14:54.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:54 smithi114 ceph-mon[30324]: Manager daemon y is now available 2024-09-21T12:14:55.588 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-09-21T12:14:55.588 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-21T12:14:55.588 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-21T12:14:55.588 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-09-21T12:14:55.588 INFO:teuthology.orchestra.run.smithi114.stdout:mgr epoch 5 is available 2024-09-21T12:14:55.588 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying orchestrator module is enabled... 2024-09-21T12:14:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:55 smithi114 ceph-mon[30324]: from='mgr.14118 172.21.15.114:0/1482034044' entity='mgr.y' 2024-09-21T12:14:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:55 smithi114 ceph-mon[30324]: from='mgr.14118 172.21.15.114:0/1482034044' entity='mgr.y' 2024-09-21T12:14:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:55 smithi114 ceph-mon[30324]: Found migration_current of "None". Setting to last migration. 2024-09-21T12:14:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:55 smithi114 ceph-mon[30324]: from='mgr.14118 172.21.15.114:0/1482034044' entity='mgr.y' 2024-09-21T12:14:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:55 smithi114 ceph-mon[30324]: from='mgr.14118 172.21.15.114:0/1482034044' entity='mgr.y' 2024-09-21T12:14:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:55 smithi114 ceph-mon[30324]: from='mgr.14118 172.21.15.114:0/1482034044' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:14:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:55 smithi114 ceph-mon[30324]: from='mgr.14118 172.21.15.114:0/1482034044' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:14:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:55 smithi114 ceph-mon[30324]: from='mgr.14118 172.21.15.114:0/1482034044' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T12:14:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:55 smithi114 ceph-mon[30324]: from='mgr.14118 172.21.15.114:0/1482034044' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T12:14:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:55 smithi114 ceph-mon[30324]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-21T12:14:56.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:55 smithi114 ceph-mon[30324]: mgrmap e7: y(active, since 1.00603s) 2024-09-21T12:14:56.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:55 smithi114 ceph-mon[30324]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-21T12:14:57.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:56 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3029346890' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "orchestrator"} : dispatch 2024-09-21T12:14:57.275 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:14:57 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: 2024-09-21T12:14:57.112+0000 7f79f9516640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:12:14:57] ENGINE Error in HTTPServer.serve 2024-09-21T12:14:57.275 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:14:57 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: Traceback (most recent call last): 2024-09-21T12:14:57.275 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:14:57 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T12:14:57.275 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:14:57 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: self._connections.run(self.expiration_interval) 2024-09-21T12:14:57.276 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:14:57 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T12:14:57.276 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:14:57 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: self._run(expiration_interval) 2024-09-21T12:14:57.276 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:14:57 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T12:14:57.276 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:14:57 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T12:14:57.276 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:14:57 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T12:14:57.276 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:14:57 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T12:14:57.276 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:14:57 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T12:14:57.276 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:14:57 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: s = self.context.wrap_socket( 2024-09-21T12:14:57.276 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:14:57 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T12:14:57.276 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:14:57 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: return self.sslsocket_class._create( 2024-09-21T12:14:57.276 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:14:57 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T12:14:57.277 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:14:57 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: self.do_handshake() 2024-09-21T12:14:57.277 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:14:57 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T12:14:57.277 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:14:57 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: self._sslobj.do_handshake() 2024-09-21T12:14:57.277 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:14:57 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T12:14:57.277 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:14:57 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: 2024-09-21T12:14:57.578 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr module 'orchestrator' is already enabled (always-on) 2024-09-21T12:14:57.578 INFO:teuthology.orchestra.run.smithi114.stdout:Setting orchestrator backend to cephadm... 2024-09-21T12:14:58.190 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:57 smithi114 ceph-mon[30324]: [21/Sep/2024:12:14:56] ENGINE Bus STARTING 2024-09-21T12:14:58.190 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:57 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3029346890' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "orchestrator"}]': finished 2024-09-21T12:14:58.190 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:57 smithi114 ceph-mon[30324]: mgrmap e8: y(active, since 2s) 2024-09-21T12:14:58.190 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:57 smithi114 ceph-mon[30324]: [21/Sep/2024:12:14:56] ENGINE Serving on http://172.21.15.114:8765 2024-09-21T12:14:58.190 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:57 smithi114 ceph-mon[30324]: from='mgr.14118 172.21.15.114:0/1482034044' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:14:59.117 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:58 smithi114 ceph-mon[30324]: [21/Sep/2024:12:14:57] ENGINE Serving on https://172.21.15.114:7150 2024-09-21T12:14:59.117 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:58 smithi114 ceph-mon[30324]: [21/Sep/2024:12:14:57] ENGINE Bus STARTED 2024-09-21T12:14:59.117 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:58 smithi114 ceph-mon[30324]: [21/Sep/2024:12:14:57] ENGINE Error in HTTPServer.serve 2024-09-21T12:14:59.117 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-21T12:14:59.117 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T12:14:59.117 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._connections.run(self.expiration_interval) 2024-09-21T12:14:59.117 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T12:14:59.117 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._run(expiration_interval) 2024-09-21T12:14:59.118 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T12:14:59.118 INFO:journalctl@ceph.mon.a.smithi114.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T12:14:59.118 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T12:14:59.118 INFO:journalctl@ceph.mon.a.smithi114.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T12:14:59.118 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T12:14:59.118 INFO:journalctl@ceph.mon.a.smithi114.stdout: s = self.context.wrap_socket( 2024-09-21T12:14:59.118 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T12:14:59.118 INFO:journalctl@ceph.mon.a.smithi114.stdout: return self.sslsocket_class._create( 2024-09-21T12:14:59.118 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T12:14:59.118 INFO:journalctl@ceph.mon.a.smithi114.stdout: self.do_handshake() 2024-09-21T12:14:59.118 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T12:14:59.119 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._sslobj.do_handshake() 2024-09-21T12:14:59.119 INFO:journalctl@ceph.mon.a.smithi114.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T12:14:59.119 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:58 smithi114 ceph-mon[30324]: from='mgr.14118 172.21.15.114:0/1482034044' entity='mgr.y' 2024-09-21T12:14:59.119 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:58 smithi114 ceph-mon[30324]: from='mgr.14118 172.21.15.114:0/1482034044' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:14:59.856 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-21T12:14:59.856 INFO:teuthology.orchestra.run.smithi114.stdout:Generating ssh key... 2024-09-21T12:15:00.115 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:14:59 smithi114 ceph-mon[30324]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:15:01.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:00 smithi114 ceph-mon[30324]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:15:01.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:00 smithi114 ceph-mon[30324]: from='mgr.14118 172.21.15.114:0/1482034044' entity='mgr.y' 2024-09-21T12:15:01.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:00 smithi114 ceph-mon[30324]: from='mgr.14118 172.21.15.114:0/1482034044' entity='mgr.y' 2024-09-21T12:15:01.025 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:00 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: Generating public/private rsa key pair. 2024-09-21T12:15:01.025 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:00 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: Your identification has been saved in /tmp/tmpn91ufmxk/key 2024-09-21T12:15:01.025 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:00 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: Your public key has been saved in /tmp/tmpn91ufmxk/key.pub 2024-09-21T12:15:01.025 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:00 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: The key fingerprint is: 2024-09-21T12:15:01.025 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:00 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: SHA256:qD4uwT9BDckUBubfPizOrGz2SzBMCFVp6DCwceEA6ms ceph-e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T12:15:01.025 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:00 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: The key's randomart image is: 2024-09-21T12:15:01.025 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:00 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: +---[RSA 3072]----+ 2024-09-21T12:15:01.025 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:00 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: |O.*B=+ | 2024-09-21T12:15:01.026 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:00 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: |=X..* | 2024-09-21T12:15:01.026 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:00 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: |+++. o | 2024-09-21T12:15:01.026 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:00 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: |.o...... | 2024-09-21T12:15:01.026 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:00 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: | o+.. o S | 2024-09-21T12:15:01.026 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:00 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: | +o.+ | 2024-09-21T12:15:01.026 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:00 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: | E o+.+ | 2024-09-21T12:15:01.026 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:00 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: |..+*+. . | 2024-09-21T12:15:01.026 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:00 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: | oo=O+ | 2024-09-21T12:15:01.026 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:00 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: +----[SHA256]-----+ 2024-09-21T12:15:02.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:01 smithi114 ceph-mon[30324]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:15:02.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:01 smithi114 ceph-mon[30324]: Generating ssh key... 2024-09-21T12:15:02.461 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9ogCSiC+ZqN2puX3b0ZhQJaU1adN0uyjexZ9ywbZ7TkeZ5ul0PW99ixFUQxvdgKOHJWaTL3rsLk+gio5ZfcMZ47xyZbahitD48R6un2h/iqmqsrcLwgA3wdy+5I0cylKWFMG6TbHcgyY+zDC9kgvs8Enxc48HM1LUnwKZpKLXeROcLhC3s7MYRoTbxuEw4A/T9mrBu23MqRZqHisDPM9znNsB7fdpR8QQJYhiER4OwWNO4kJAEejd2zTUP5TYg1LTy97e6ShXUgmA/trZyfbE2Y2AKtQVhHqiqrptQ8jH6kovf0g8sPtjdzbcoWI/2CdIGp4XoZDl9lTBUeDuWW7oUI3NHWr2FhKySW/M8Bs62Cj1QnqInptP6Tet8YNyOoF6I1zXCBrhsvKnVWk+yK2tE+61Cin2v+VVK3WIKrxWHEc9YcjgJWP9nf7kHfm1bQTF8ZadtLub/+1fu3EmRBYgUgCyNGOlka4ITr3Xel4H+pqRC3+UkmnY9z+B9QhP8PM= ceph-e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T12:15:02.461 INFO:teuthology.orchestra.run.smithi114.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-21T12:15:02.462 INFO:teuthology.orchestra.run.smithi114.stdout:Adding key to root@localhost authorized_keys... 2024-09-21T12:15:02.469 INFO:teuthology.orchestra.run.smithi114.stdout:Adding host smithi114... 2024-09-21T12:15:03.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:02 smithi114 ceph-mon[30324]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:15:04.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:04 smithi114 ceph-mon[30324]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi114", "addr": "172.21.15.114", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:15:04.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:04 smithi114 ceph-mon[30324]: Deploying cephadm binary to smithi114 2024-09-21T12:15:06.911 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Added host 'smithi114' with addr '172.21.15.114' 2024-09-21T12:15:06.912 INFO:teuthology.orchestra.run.smithi114.stdout:Deploying unmanaged mon service... 2024-09-21T12:15:07.461 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:07 smithi114 ceph-mon[30324]: from='mgr.14118 172.21.15.114:0/1482034044' entity='mgr.y' 2024-09-21T12:15:07.461 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:07 smithi114 ceph-mon[30324]: Added host smithi114 2024-09-21T12:15:07.461 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:07 smithi114 ceph-mon[30324]: from='mgr.14118 172.21.15.114:0/1482034044' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:15:08.917 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-21T12:15:08.917 INFO:teuthology.orchestra.run.smithi114.stdout:Deploying unmanaged mgr service... 2024-09-21T12:15:09.128 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:08 smithi114 ceph-mon[30324]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:15:09.128 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:08 smithi114 ceph-mon[30324]: Saving service mon spec with placement count:5 2024-09-21T12:15:09.128 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:08 smithi114 ceph-mon[30324]: from='mgr.14118 172.21.15.114:0/1482034044' entity='mgr.y' 2024-09-21T12:15:09.128 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:08 smithi114 ceph-mon[30324]: from='mgr.14118 172.21.15.114:0/1482034044' entity='mgr.y' 2024-09-21T12:15:09.128 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:08 smithi114 ceph-mon[30324]: from='mgr.14118 172.21.15.114:0/1482034044' entity='mgr.y' 2024-09-21T12:15:10.576 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-21T12:15:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:10 smithi114 ceph-mon[30324]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:15:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:10 smithi114 ceph-mon[30324]: Saving service mgr spec with placement count:2 2024-09-21T12:15:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:10 smithi114 ceph-mon[30324]: from='mgr.14118 172.21.15.114:0/1482034044' entity='mgr.y' 2024-09-21T12:15:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:10 smithi114 ceph-mon[30324]: from='mgr.14118 172.21.15.114:0/1482034044' entity='mgr.y' 2024-09-21T12:15:12.367 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:12 smithi114 ceph-mon[30324]: from='mgr.14118 172.21.15.114:0/1482034044' entity='mgr.y' 2024-09-21T12:15:12.367 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:12 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1580314131' entity='client.admin' 2024-09-21T12:15:13.359 INFO:teuthology.orchestra.run.smithi114.stdout:Enabling the dashboard module... 2024-09-21T12:15:13.480 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:13 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3633851266' entity='client.admin' 2024-09-21T12:15:15.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:15 smithi114 ceph-mon[30324]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:15:15.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:15 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1125359405' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "dashboard"} : dispatch 2024-09-21T12:15:15.275 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:15 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ignoring --setuser ceph since I am not root 2024-09-21T12:15:15.275 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:15 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ignoring --setgroup ceph since I am not root 2024-09-21T12:15:16.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:16 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1125359405' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-21T12:15:16.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:16 smithi114 ceph-mon[30324]: mgrmap e9: y(active, since 21s) 2024-09-21T12:15:16.925 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-09-21T12:15:16.926 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-21T12:15:16.926 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T12:15:16.926 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-21T12:15:16.926 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-21T12:15:16.926 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-09-21T12:15:16.926 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for the mgr to restart... 2024-09-21T12:15:16.926 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr epoch 9... 2024-09-21T12:15:17.266 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:17 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3979836742' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-21T12:15:20.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:20 smithi114 ceph-mon[30324]: Active manager daemon y restarted 2024-09-21T12:15:20.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:20 smithi114 ceph-mon[30324]: Activating manager daemon y 2024-09-21T12:15:20.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:20 smithi114 ceph-mon[30324]: osdmap e3: 0 total, 0 up, 0 in 2024-09-21T12:15:20.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:20 smithi114 ceph-mon[30324]: mgrmap e10: y(active, starting, since 0.0040139s) 2024-09-21T12:15:20.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:20 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T12:15:20.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:20 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T12:15:20.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:20 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T12:15:20.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:20 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T12:15:20.428 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:20 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T12:15:20.428 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:20 smithi114 ceph-mon[30324]: Manager daemon y is now available 2024-09-21T12:15:21.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:21 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:15:21.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:21 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T12:15:21.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:21 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T12:15:21.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:21 smithi114 ceph-mon[30324]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-21T12:15:21.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:21 smithi114 ceph-mon[30324]: mgrmap e11: y(active, since 1.00617s) 2024-09-21T12:15:21.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:21 smithi114 ceph-mon[30324]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-21T12:15:21.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-09-21T12:15:21.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-21T12:15:21.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-21T12:15:21.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-09-21T12:15:21.725 INFO:teuthology.orchestra.run.smithi114.stdout:mgr epoch 9 is available 2024-09-21T12:15:21.725 INFO:teuthology.orchestra.run.smithi114.stdout:Using certmgr to generate dashboard self-signed certificate... 2024-09-21T12:15:21.981 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:21 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: 2024-09-21T12:15:21.762+0000 7f7dc5ef4640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:12:15:21] ENGINE Error in HTTPServer.serve 2024-09-21T12:15:21.981 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:21 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: Traceback (most recent call last): 2024-09-21T12:15:21.981 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:21 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T12:15:21.981 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:21 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: self._connections.run(self.expiration_interval) 2024-09-21T12:15:21.981 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:21 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T12:15:21.981 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:21 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: self._run(expiration_interval) 2024-09-21T12:15:21.981 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:21 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T12:15:21.981 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:21 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T12:15:21.981 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:21 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T12:15:21.981 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:21 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T12:15:21.982 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:21 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T12:15:21.982 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:21 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: s = self.context.wrap_socket( 2024-09-21T12:15:21.982 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:21 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T12:15:21.982 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:21 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: return self.sslsocket_class._create( 2024-09-21T12:15:21.982 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:21 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T12:15:21.982 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:21 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: self.do_handshake() 2024-09-21T12:15:21.982 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:21 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T12:15:21.982 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:21 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: self._sslobj.do_handshake() 2024-09-21T12:15:21.982 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:21 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T12:15:21.982 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:21 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: 2024-09-21T12:15:22.385 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:22 smithi114 ceph-mon[30324]: [21/Sep/2024:12:15:21] ENGINE Bus STARTING 2024-09-21T12:15:22.385 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:22 smithi114 ceph-mon[30324]: [21/Sep/2024:12:15:21] ENGINE Serving on https://172.21.15.114:7150 2024-09-21T12:15:22.385 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:22 smithi114 ceph-mon[30324]: [21/Sep/2024:12:15:21] ENGINE Error in HTTPServer.serve 2024-09-21T12:15:22.385 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-21T12:15:22.385 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T12:15:22.385 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._connections.run(self.expiration_interval) 2024-09-21T12:15:22.385 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T12:15:22.386 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._run(expiration_interval) 2024-09-21T12:15:22.386 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T12:15:22.386 INFO:journalctl@ceph.mon.a.smithi114.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T12:15:22.386 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T12:15:22.386 INFO:journalctl@ceph.mon.a.smithi114.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T12:15:22.386 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T12:15:22.386 INFO:journalctl@ceph.mon.a.smithi114.stdout: s = self.context.wrap_socket( 2024-09-21T12:15:22.386 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T12:15:22.386 INFO:journalctl@ceph.mon.a.smithi114.stdout: return self.sslsocket_class._create( 2024-09-21T12:15:22.386 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T12:15:22.386 INFO:journalctl@ceph.mon.a.smithi114.stdout: self.do_handshake() 2024-09-21T12:15:22.386 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T12:15:22.386 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._sslobj.do_handshake() 2024-09-21T12:15:22.386 INFO:journalctl@ceph.mon.a.smithi114.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T12:15:22.386 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:22 smithi114 ceph-mon[30324]: [21/Sep/2024:12:15:21] ENGINE Serving on http://172.21.15.114:8765 2024-09-21T12:15:22.387 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:22 smithi114 ceph-mon[30324]: [21/Sep/2024:12:15:21] ENGINE Bus STARTED 2024-09-21T12:15:23.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:23 smithi114 ceph-mon[30324]: mgrmap e12: y(active, since 2s) 2024-09-21T12:15:23.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:23 smithi114 ceph-mon[30324]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "orch certmgr generate-certificates", "module_name": "dashboard", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:15:23.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:23 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:23.516 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout {"cert": "-----BEGIN CERTIFICATE-----\nMIIE/DCCAuSgAwIBAgIUXpdrM+qUNb+ksOTauF+4oHZbJfowDQYJKoZIhvcNAQEL\nBQAwFzEVMBMGA1UEAwwMY2VwaGFkbS1yb290MB4XDTI0MDkyMTEyMTUyMloXDTM0\nMDkyMjEyMTUyMlowGDEWMBQGA1UEAwwNMTcyLjIxLjE1LjExNDCCAiIwDQYJKoZI\nhvcNAQEBBQADggIPADCCAgoCggIBALqaNXR8mIhXWQzLJGd214IqKBUe1L2Ygy9n\npycwUF1lDQ28jWQ1UtDSA86MHoZdWvYxo7XXrKLfRWdAXhWRXJY272Jilz+FU+oz\n5NdhZuJffAVFbsNRivYCR3RDr/W/7CDA2MPT1gmGdwfMeizZB/Xw4pV6TgNaLnny\n+OOnq5NvX/GcYSlP8bFserBaTL9PpELChut5h5Chq22PbSCXhCEvujD4h62mEOls\nedB5EoDwKtSyim8n4sC940RN90We0U7KekL7DRzJ3iC+RyiIvUboimDs/u6Yn3qc\nIdDjwEVzUS2s3eSYyZp+N7XAoTZpOWcoUDySGXKf7ooQBFY+NBwWYk719NN8ml+J\nvgwl7uku3ck3GGx+3Pn8ayL2qdspCzja8K7TXaAWqcB/+PUzUJY91nvj8XHzRdKw\nvK/gfo2fzdvagZcJHTbojD2wXMVX6+2MSDB39mJB29REzFPJL4GPNjgvHwiQ0l5G\npxBEd8M/xWrfwCjapkMrpgpQl2l9UFJ0JHRyx3QiCMervR/bksIYVHL1gYxqPLhN\nJdAeHAS35gRTvpaa1qIE/84f6gHffY2l387Ag/ZRwIdHyVHbcL8ARM8Kx5oTMjaH\nh3XQkODw1IEERIbksHvaz/+Tki78sziadESZRd1/slYonrJt30dl43vGQIMNIv7t\n2VwHja3bAgMBAAGjPzA9MC0GA1UdEQQmMCSCCXNtaXRoaTExNIIRZGFzaGJvYXJk\nX3NlcnZlcnOHBKwVD3IwDAYDVR0TAQH/BAIwADANBgkqhkiG9w0BAQsFAAOCAgEA\nOMSohR6vTnkFkujxUUXd6VYHj5Xue3T73EdOz+jls/vXibKnmi0ngtNqiCr5pBDV\nmdPFyVJbAHF+p9qJDRCrsGtD+yBblJpFFMes0JaI8S2ZnLrYzxfcQYKgo2aRmFcf\n8HI1YOiJ87/I7BmBWqgvkcVbdcKUxEi2bfE01C5Hbnq0oggEqFPaDWb0XfSaggBm\nDtAiDsplNXxwQ7TcWPdRkKPKum6Cgg/4teXgaM5W7b3pHdcBX4238bsXltuPdDUq\nZ07vuGnpaDze1uWTBhssUaveFDHJasRjXWmOhvvD2wxP86VsicfMZ0cuZqEEVxHW\ne6TKDs9R8ok37vr83+qOaEzctupYqDi5/x6pH6xM49TZnz7dWWdaKZnTYHyyjDse\n0cS1Qe2h2jnPbB3ypRadw5VwbVSaWMS+Iy5B/jlpvJQUaIh5g8OI2bZ6FskPVhyW\nVf/WYBok2FUH9Sr01s5+voMUoY8mfSlRNMFN+4BGiNWdiMSBt3MIoqS59KqteS+H\nAqnLNt2MZURQn65Clj/0j6N1Ilj1dq6MqtW/4kPTf4xxKt1BADj/Fi8lu3VklQri\nmUdIhpWXMgSKX0pQNXt75gFU52cXjQlUaXEaiN5Av8jDnlThb3NYbiKj3u40CVai\n/vOFJO4xRJAKktyD25efXprGpNsWOQZlOHY2FeI6tSk=\n-----END CERTIFICATE-----\n", "key": "-----BEGIN RSA PRIVATE KEY-----\nMIIJKQIBAAKCAgEAupo1dHyYiFdZDMskZ3bXgiooFR7UvZiDL2enJzBQXWUNDbyN\nZDVS0NIDzowehl1a9jGjtdesot9FZ0BeFZFcljbvYmKXP4VT6jPk12Fm4l98BUVu\nw1GK9gJHdEOv9b/sIMDYw9PWCYZ3B8x6LNkH9fDilXpOA1ouefL446erk29f8Zxh\nKU/xsWx6sFpMv0+kQsKG63mHkKGrbY9tIJeEIS+6MPiHraYQ6Wx50HkSgPAq1LKK\nbyfiwL3jRE33RZ7RTsp6QvsNHMneIL5HKIi9RuiKYOz+7pifepwh0OPARXNRLazd\n5JjJmn43tcChNmk5ZyhQPJIZcp/uihAEVj40HBZiTvX003yaX4m+DCXu6S7dyTcY\nbH7c+fxrIvap2ykLONrwrtNdoBapwH/49TNQlj3We+PxcfNF0rC8r+B+jZ/N29qB\nlwkdNuiMPbBcxVfr7YxIMHf2YkHb1ETMU8kvgY82OC8fCJDSXkanEER3wz/Fat/A\nKNqmQyumClCXaX1QUnQkdHLHdCIIx6u9H9uSwhhUcvWBjGo8uE0l0B4cBLfmBFO+\nlprWogT/zh/qAd99jaXfzsCD9lHAh0fJUdtwvwBEzwrHmhMyNoeHddCQ4PDUgQRE\nhuSwe9rP/5OSLvyzOJp0RJlF3X+yViiesm3fR2Xje8ZAgw0i/u3ZXAeNrdsCAwEA\nAQKCAgArF5qZnH5jGfzUEgOFqzGoexDHk9egMxy0cVTuHV+pucDSd8UVK6VE2QMN\nC1mvQNbVPPBn+XcCI9p0ti4j/+4oOGZ+y3LPNfyDfNlvjaxDQbazxu2FyIp1NmCG\n7myvrVMb2z0TuBNZHsbVM7jRK4uFlcvVpilm58gM3HnypNFgGlvHQYxEjJQtm3YX\nzLqlgYBW9HJQ5b9TPKqIghM68BKQ0WNHlNVMWinUpfrZlGQcB5p0hQioE8dvKErY\nBsNAPiVq8yJZ3n0QNWQJHrfzGeTP/7qZGJHkyrUljPHYOgg4YaGRETcq58fC2DwI\nSoRQOuuJMYuZ+UlMKbfeBUXjHrlMkDxADdtOdN8+BB6y3+V6FU/932DTOdAdzIb6\nxzXvccZa8dy/bdQ93Sz1BSiVoQEidW/y4z3yq4NRHvQwH6l57W90awlFYal2dWf7\nNszG4NShfydwj9uKeAHBjVv23fbJtI5+w2O+aAQLEz5LP0xMI9+G0KXDSpTMiZBm\nteYTVoQbr68HBC+eI3wePdoO81QxeE/K4XL51XcKYXqOfT3fQSWFbiRfbrO0E4qx\n0LLdf8O/k7bsU3Xabntb9Mpp79WbLcd4rah77KHOyF5qDWZRZYFwvkSyNTuorey4\niTXfaPle9OhV3991ugvW4gaSiTgGNGFREuJKCVvwC2ZZjRYfxQKCAQEA30b7bi34\nbZN136hmAnQVBfyPJJSj6bH8EKBC9/5ejgvnHI05oy7Wue2B1x7t07BtabSIT8Bw\nfMNzeCrVIl47/vz3f5Nc5bQIaCKfBTf8Br2I/JDPZuT1XvDptbh71aRA5f6A44K8\ndNhEMWk0V5BfI9GIdwSc4ArYoiBCHddGlGqdXjOClaFUJy3PDZMQM3elTS+BqKxo\nJNtspXF7nn4wjRPoXCgWKTxfUGKHVDuYcOZjhAHWHRpKx6tB/HYEUJs3mpPIB/CF\nkI9dP+cn+Dqu67OiUNg+VD1gwYvrK85NMjKGXTzaVCYJevMjxAkSV/J5p8lqDlbi\nHUKHIG1+71NwfQKCAQEA1fM9xqB2ObM+n05bScLNagbROhevvZ1GsgXnNLFYXIzx\n4fuclyQ2IIeIyZ9p2eQmYhoL8z1UbFXKS/l3SPim1JyX5B/5uibHoHeuvdFozF97\nqZbBRAOo0umJY4VoMfdT/TBRmIxbviimgaTC+Wc+fMNI0WjkiBUklmmwpcJbT66j\nHfa3xi8+vPNZrTSCb24GVJDU5VCzcMcRNHsDn3PwWOr7QDLnrLoFsGGLLSJNNQZK\nUnjpc1zZr7BMM5Q12UJ+vRsNIdNf1K/VG3UykqHXuxPKWT3gqBlbOwEsUNr4C5hn\nXl+VXau409yFKzFRgIoJzzGu7nLpf/WN3MGuxiT/NwKCAQEA1W5t9FGfU2eNWuIH\nuTqryhA16PrpgQAxetcnjRYr3s8D+mgJe7LPceXWowOexie1P0FiuBG1tZ3EKE+R\nc4vu//1ivSHpwG4LRzgpGHz+vpa5du0aRTqCT2Itlhcm0P91RiW1oUWcznjZHv55\nGWcZewzVzSesqgHUOY5qhUn+3urXSXAl2NDht53zef9lkj1B13mdENbVqr3RqojD\naaso0AMCr+1l2b8rQZUd1egt/WZOJKb4vNMgPqO3rRRHZaIqUeoIYYaKtrhEpExc\nCc5W3VrWwz8ivbX6YqlUgYhwhJaFmGUOMzg1AcGgI8mkLkvL/Rbs+Y4WEXBB6T6i\nAzaYpQKCAQARJ8LxVVaFg+jmjzrkrDkaYleOh+2wj5bmmXCTmhDSwv52N120BNRO\nQW/nct/i99dYEGya4XUOhsaE1wkIkd+WtsO7KBE3Kvrg1Bfv4oW7tYBN9eejU4PK\n/azljQHw17slVyWDzOViVxIa3SJBK1UoXeRN3lJM0x0u7mSatf3qYr9znJ/ZxVoL\nfpvXNpmq47CgQUjA5nqs9Vlc0uxUcdB44MaqvCo1gGkKXpfxU4nLUajN+iIF/RD7\ndHB3xk+XDyzobu3bo35QqJS28evPNCj/ODQGed60u+vDzHFIPnabP2oyeAmbDDTy\n0ADhmL/HRcpJkdtMBWzRtNfeNq2T9817AoIBAQCwESD+78sdp5L8KwYzgpgNH5gV\nez7jUDg/haV07/utABeVmSh4KYtZu2WUHYiNFtM1OYU8vapmxfdUMmyNHAPhuM5F\nJm1sTkd14rCGoDjTPvrULQhfLQwPrjja4gEp6NgTapTu3X+nUKZI1BhrIHqTgJpK\nJE+rSqplDxwRtqAh9f068cVcW7x5G38tOtu1ilYgwi8dxQncxHtrfBn7W6a272gx\nwDQrsb8c+mNA6tI7CSh6TWorrqm68H/Rc9rdk/+EqXQFzUbnlMbtFJLlqC0dqRwD\nM1OL5sk+3KdEo0i+ZdWTCoXnExbjBm9l5LUEgtMvIF7PnFyaizvZGeRMdQcX\n-----END RSA PRIVATE KEY-----\n"} 2024-09-21T12:15:24.814 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:24 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:24.814 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:24 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:24.816 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout SSL certificate updated 2024-09-21T12:15:25.836 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:25 smithi114 ceph-mon[30324]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:15:26.744 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout SSL certificate key updated 2024-09-21T12:15:26.744 INFO:teuthology.orchestra.run.smithi114.stdout:Creating initial admin user... 2024-09-21T12:15:27.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:27 smithi114 ceph-mon[30324]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:15:27.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:27 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:28.209 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$zcMVRtd7.RA4i/TsOCNVHugOVfi8vktf0YXwu/O6LkiQGIZ7QS6vi", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726920927, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-21T12:15:28.209 INFO:teuthology.orchestra.run.smithi114.stdout:Fetching dashboard port number... 2024-09-21T12:15:28.821 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:28 smithi114 ceph-mon[30324]: from='client.14170 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:15:28.821 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:28 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:29.406 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 8443 2024-09-21T12:15:29.415 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T12:15:29.416 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-09-21T12:15:29.423 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T12:15:29.423 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-09-21T12:15:29.423 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-09-21T12:15:29.423 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-21T12:15:29.424 INFO:teuthology.orchestra.run.smithi114.stdout:Ceph Dashboard is now available at: 2024-09-21T12:15:29.425 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:15:29.425 INFO:teuthology.orchestra.run.smithi114.stdout: URL: https://smithi114.front.sepia.ceph.com:8443/ 2024-09-21T12:15:29.425 INFO:teuthology.orchestra.run.smithi114.stdout: User: admin 2024-09-21T12:15:29.425 INFO:teuthology.orchestra.run.smithi114.stdout: Password: 0q9tfoc40g 2024-09-21T12:15:29.425 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:15:29.425 INFO:teuthology.orchestra.run.smithi114.stdout:Saving cluster configuration to /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/config directory 2024-09-21T12:15:29.764 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:29 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3770974711' entity='client.admin' cmd={"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} : dispatch 2024-09-21T12:15:30.705 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-21T12:15:30.705 INFO:teuthology.orchestra.run.smithi114.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-21T12:15:30.705 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:15:30.705 INFO:teuthology.orchestra.run.smithi114.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-21T12:15:30.705 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:15:30.706 INFO:teuthology.orchestra.run.smithi114.stdout:Or, if you are only running a single cluster on this host: 2024-09-21T12:15:30.706 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:15:30.706 INFO:teuthology.orchestra.run.smithi114.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-21T12:15:30.706 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:15:30.706 INFO:teuthology.orchestra.run.smithi114.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-21T12:15:30.706 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:15:30.707 INFO:teuthology.orchestra.run.smithi114.stdout: ceph telemetry on 2024-09-21T12:15:30.707 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:15:30.707 INFO:teuthology.orchestra.run.smithi114.stdout:For more information see: 2024-09-21T12:15:30.707 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:15:30.707 INFO:teuthology.orchestra.run.smithi114.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-21T12:15:30.707 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:15:30.707 INFO:teuthology.orchestra.run.smithi114.stdout:Bootstrap complete. 2024-09-21T12:15:30.745 INFO:tasks.cephadm:Fetching config... 2024-09-21T12:15:30.745 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-21T12:15:30.745 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-21T12:15:30.761 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-21T12:15:30.762 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-21T12:15:30.762 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-21T12:15:30.817 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-21T12:15:30.818 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-21T12:15:30.818 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/keyring of=/dev/stdout 2024-09-21T12:15:30.883 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-21T12:15:30.884 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-21T12:15:30.884 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-21T12:15:30.939 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-21T12:15:30.939 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9ogCSiC+ZqN2puX3b0ZhQJaU1adN0uyjexZ9ywbZ7TkeZ5ul0PW99ixFUQxvdgKOHJWaTL3rsLk+gio5ZfcMZ47xyZbahitD48R6un2h/iqmqsrcLwgA3wdy+5I0cylKWFMG6TbHcgyY+zDC9kgvs8Enxc48HM1LUnwKZpKLXeROcLhC3s7MYRoTbxuEw4A/T9mrBu23MqRZqHisDPM9znNsB7fdpR8QQJYhiER4OwWNO4kJAEejd2zTUP5TYg1LTy97e6ShXUgmA/trZyfbE2Y2AKtQVhHqiqrptQ8jH6kovf0g8sPtjdzbcoWI/2CdIGp4XoZDl9lTBUeDuWW7oUI3NHWr2FhKySW/M8Bs62Cj1QnqInptP6Tet8YNyOoF6I1zXCBrhsvKnVWk+yK2tE+61Cin2v+VVK3WIKrxWHEc9YcjgJWP9nf7kHfm1bQTF8ZadtLub/+1fu3EmRBYgUgCyNGOlka4ITr3Xel4H+pqRC3+UkmnY9z+B9QhP8PM= ceph-e2ad975c-7812-11ef-baf6-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T12:15:31.014 INFO:teuthology.orchestra.run.smithi114.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9ogCSiC+ZqN2puX3b0ZhQJaU1adN0uyjexZ9ywbZ7TkeZ5ul0PW99ixFUQxvdgKOHJWaTL3rsLk+gio5ZfcMZ47xyZbahitD48R6un2h/iqmqsrcLwgA3wdy+5I0cylKWFMG6TbHcgyY+zDC9kgvs8Enxc48HM1LUnwKZpKLXeROcLhC3s7MYRoTbxuEw4A/T9mrBu23MqRZqHisDPM9znNsB7fdpR8QQJYhiER4OwWNO4kJAEejd2zTUP5TYg1LTy97e6ShXUgmA/trZyfbE2Y2AKtQVhHqiqrptQ8jH6kovf0g8sPtjdzbcoWI/2CdIGp4XoZDl9lTBUeDuWW7oUI3NHWr2FhKySW/M8Bs62Cj1QnqInptP6Tet8YNyOoF6I1zXCBrhsvKnVWk+yK2tE+61Cin2v+VVK3WIKrxWHEc9YcjgJWP9nf7kHfm1bQTF8ZadtLub/+1fu3EmRBYgUgCyNGOlka4ITr3Xel4H+pqRC3+UkmnY9z+B9QhP8PM= ceph-e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T12:15:31.025 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9ogCSiC+ZqN2puX3b0ZhQJaU1adN0uyjexZ9ywbZ7TkeZ5ul0PW99ixFUQxvdgKOHJWaTL3rsLk+gio5ZfcMZ47xyZbahitD48R6un2h/iqmqsrcLwgA3wdy+5I0cylKWFMG6TbHcgyY+zDC9kgvs8Enxc48HM1LUnwKZpKLXeROcLhC3s7MYRoTbxuEw4A/T9mrBu23MqRZqHisDPM9znNsB7fdpR8QQJYhiER4OwWNO4kJAEejd2zTUP5TYg1LTy97e6ShXUgmA/trZyfbE2Y2AKtQVhHqiqrptQ8jH6kovf0g8sPtjdzbcoWI/2CdIGp4XoZDl9lTBUeDuWW7oUI3NHWr2FhKySW/M8Bs62Cj1QnqInptP6Tet8YNyOoF6I1zXCBrhsvKnVWk+yK2tE+61Cin2v+VVK3WIKrxWHEc9YcjgJWP9nf7kHfm1bQTF8ZadtLub/+1fu3EmRBYgUgCyNGOlka4ITr3Xel4H+pqRC3+UkmnY9z+B9QhP8PM= ceph-e2ad975c-7812-11ef-baf6-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T12:15:31.162 INFO:teuthology.orchestra.run.smithi183.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9ogCSiC+ZqN2puX3b0ZhQJaU1adN0uyjexZ9ywbZ7TkeZ5ul0PW99ixFUQxvdgKOHJWaTL3rsLk+gio5ZfcMZ47xyZbahitD48R6un2h/iqmqsrcLwgA3wdy+5I0cylKWFMG6TbHcgyY+zDC9kgvs8Enxc48HM1LUnwKZpKLXeROcLhC3s7MYRoTbxuEw4A/T9mrBu23MqRZqHisDPM9znNsB7fdpR8QQJYhiER4OwWNO4kJAEejd2zTUP5TYg1LTy97e6ShXUgmA/trZyfbE2Y2AKtQVhHqiqrptQ8jH6kovf0g8sPtjdzbcoWI/2CdIGp4XoZDl9lTBUeDuWW7oUI3NHWr2FhKySW/M8Bs62Cj1QnqInptP6Tet8YNyOoF6I1zXCBrhsvKnVWk+yK2tE+61Cin2v+VVK3WIKrxWHEc9YcjgJWP9nf7kHfm1bQTF8ZadtLub/+1fu3EmRBYgUgCyNGOlka4ITr3Xel4H+pqRC3+UkmnY9z+B9QhP8PM= ceph-e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T12:15:31.175 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-21T12:15:31.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:31 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/828520523' entity='client.admin' 2024-09-21T12:15:31.422 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:15:32.675 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-21T12:15:32.675 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-21T12:15:33.400 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:33 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1035318486' entity='client.admin' 2024-09-21T12:15:33.400 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:33 smithi114 ceph-mon[30324]: mgrmap e13: y(active, since 12s) 2024-09-21T12:15:33.640 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:15:34.892 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi183 2024-09-21T12:15:34.892 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-09-21T12:15:34.892 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/etc/ceph/ceph.conf 2024-09-21T12:15:34.911 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-09-21T12:15:34.911 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:15:34.969 INFO:tasks.cephadm:Adding host smithi183 to orchestrator... 2024-09-21T12:15:34.969 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph orch host add smithi183 2024-09-21T12:15:35.344 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:15:35.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:35 smithi114 ceph-mon[30324]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:15:35.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:35 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:35.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:35 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:35.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:35 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:35.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:35 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"} : dispatch 2024-09-21T12:15:35.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:35 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:15:35.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:35 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:15:37.176 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:36 smithi114 ceph-mon[30324]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-21T12:15:37.176 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:36 smithi114 ceph-mon[30324]: Updating smithi114:/var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:15:37.176 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:36 smithi114 ceph-mon[30324]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:15:37.176 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:36 smithi114 ceph-mon[30324]: Updating smithi114:/var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:15:37.176 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:36 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:37.176 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:36 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:37.176 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:36 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:37.176 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:36 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:15:37.177 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:36 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:15:37.177 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:36 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:15:37.177 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:36 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:38.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:37 smithi114 ceph-mon[30324]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi183", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:15:38.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:37 smithi114 ceph-mon[30324]: Deploying cephadm binary to smithi183 2024-09-21T12:15:39.141 INFO:teuthology.orchestra.run.smithi114.stdout:Added host 'smithi183' with addr '172.21.15.183' 2024-09-21T12:15:39.731 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph orch host ls --format=json 2024-09-21T12:15:39.973 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:15:40.383 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:40 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:40.383 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:40 smithi114 ceph-mon[30324]: Added host smithi183 2024-09-21T12:15:40.383 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:40 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:15:40.383 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:40 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:40.383 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:40 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:40.383 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:40 smithi114 ceph-mon[30324]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:15:40.660 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:15:40.660 INFO:teuthology.orchestra.run.smithi114.stdout:[{"addr": "172.21.15.114", "hostname": "smithi114", "labels": [], "status": ""}, {"addr": "172.21.15.183", "hostname": "smithi183", "labels": [], "status": ""}] 2024-09-21T12:15:41.274 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-21T12:15:41.275 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd crush tunables default 2024-09-21T12:15:41.514 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:15:41.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:41 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:41.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:41 smithi114 ceph-mon[30324]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:15:42.466 INFO:teuthology.orchestra.run.smithi114.stderr:adjusted tunables profile to default 2024-09-21T12:15:42.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:42 smithi114 ceph-mon[30324]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:15:42.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:42 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/477307236' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-21T12:15:43.070 INFO:tasks.cephadm:Adding mon.a on smithi114 2024-09-21T12:15:43.070 INFO:tasks.cephadm:Adding mon.c on smithi114 2024-09-21T12:15:43.070 INFO:tasks.cephadm:Adding mon.b on smithi183 2024-09-21T12:15:43.070 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph orch apply mon '3;smithi114:172.21.15.114=a;smithi114:[v2:172.21.15.114:3301,v1:172.21.15.114:6790]=c;smithi183:172.21.15.183=b' 2024-09-21T12:15:43.398 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T12:15:43.442 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T12:15:43.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:43 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/477307236' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T12:15:43.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:43 smithi114 ceph-mon[30324]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T12:15:43.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:43 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:43.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:43 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:43.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:43 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:43.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:43 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:43.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:43 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"} : dispatch 2024-09-21T12:15:43.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:43 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:15:43.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:43 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:15:44.138 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled mon update... 2024-09-21T12:15:44.714 DEBUG:teuthology.orchestra.run.smithi114:mon.c> sudo journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@mon.c.service 2024-09-21T12:15:44.716 DEBUG:teuthology.orchestra.run.smithi183:mon.b> sudo journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@mon.b.service 2024-09-21T12:15:44.720 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T12:15:44.720 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph mon dump -f json 2024-09-21T12:15:45.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:44 smithi114 ceph-mon[30324]: Updating smithi183:/etc/ceph/ceph.conf 2024-09-21T12:15:45.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:44 smithi114 ceph-mon[30324]: Updating smithi183:/var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:15:45.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:44 smithi114 ceph-mon[30324]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:15:45.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:44 smithi114 ceph-mon[30324]: Updating smithi183:/var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:15:45.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:44 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:45.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:44 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:45.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:44 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:45.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:44 smithi114 ceph-mon[30324]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:15:45.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:44 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:45.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:44 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:15:45.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:44 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:15:45.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:44 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:15:45.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:44 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:45.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:44 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T12:15:45.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:44 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:15:45.693 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:15:46.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:45 smithi114 ceph-mon[30324]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi114:172.21.15.114=a;smithi114:[v2:172.21.15.114:3301,v1:172.21.15.114:6790]=c;smithi183:172.21.15.183=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:15:46.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:45 smithi114 ceph-mon[30324]: Saving service mon spec with placement smithi114:172.21.15.114=a;smithi114:[v2:172.21.15.114:3301,v1:172.21.15.114:6790]=c;smithi183:172.21.15.183=b;count:3 2024-09-21T12:15:46.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:45 smithi114 ceph-mon[30324]: Deploying daemon mon.b on smithi183 2024-09-21T12:15:47.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:46 smithi114 ceph-mon[30324]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:15:47.383 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:15:47.383 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":1,"fsid":"e2ad975c-7812-11ef-baf6-efdc52797490","modified":"2024-09-21T12:14:18.247419Z","created":"2024-09-21T12:14:18.247419Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T12:15:47.384 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 1 2024-09-21T12:15:48.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:47 smithi114 ceph-mon[30324]: from='client.? 172.21.15.183:0/828193708' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T12:15:48.584 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-21T12:15:48.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-21T12:15:48.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-21T12:15:48.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-21T12:15:48.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-21T12:15:48.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-21T12:15:48.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T12:15:48.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T12:15:48.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T12:15:48.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T12:15:48.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:15:48.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/477307236' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-21T12:15:48.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/477307236' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T12:15:48.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T12:15:48.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:48.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:48.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:48.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:48.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"} : dispatch 2024-09-21T12:15:48.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:15:48.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:15:48.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: Updating smithi183:/etc/ceph/ceph.conf 2024-09-21T12:15:48.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: Updating smithi183:/var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:15:48.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:15:48.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: Updating smithi183:/var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:15:48.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:48.991 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:48.991 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:48.991 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:15:48.991 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:48.991 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:15:48.991 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:15:48.991 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:15:48.991 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:48.991 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T12:15:48.992 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:15:48.992 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi114:172.21.15.114=a;smithi114:[v2:172.21.15.114:3301,v1:172.21.15.114:6790]=c;smithi183:172.21.15.183=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:15:48.992 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: Saving service mon spec with placement smithi114:172.21.15.114=a;smithi114:[v2:172.21.15.114:3301,v1:172.21.15.114:6790]=c;smithi183:172.21.15.183=b;count:3 2024-09-21T12:15:48.992 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: Deploying daemon mon.b on smithi183 2024-09-21T12:15:48.992 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:15:48.992 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: from='client.? 172.21.15.183:0/828193708' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T12:15:48.992 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 ceph-mon[33277]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-21T12:15:48.992 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 bash[33238]: 91b61105f05cb8bea9357a5f3059bf8731eaba160867238f99ca65557b992694 2024-09-21T12:15:48.992 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:48 smithi183 systemd[1]: Started Ceph mon.b for e2ad975c-7812-11ef-baf6-efdc52797490. 2024-09-21T12:15:49.116 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T12:15:49.116 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph mon dump -f json 2024-09-21T12:15:49.381 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:15:53.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:53 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T12:15:53.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:53 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:15:53.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:53 smithi183 ceph-mon[33277]: mon.a calling monitor election 2024-09-21T12:15:53.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:53 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:15:53.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:53 smithi183 ceph-mon[33277]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:15:53.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:53 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:15:53.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:53 smithi183 ceph-mon[33277]: mon.b calling monitor election 2024-09-21T12:15:53.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:53 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:15:53.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:53 smithi183 ceph-mon[33277]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:15:53.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:53 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:15:53.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:53 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:15:53.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:53 smithi183 ceph-mon[33277]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T12:15:53.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:53 smithi183 ceph-mon[33277]: monmap epoch 2 2024-09-21T12:15:53.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:53 smithi183 ceph-mon[33277]: fsid e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T12:15:53.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:53 smithi183 ceph-mon[33277]: last_changed 2024-09-21T12:15:48.587882+0000 2024-09-21T12:15:53.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:53 smithi183 ceph-mon[33277]: created 2024-09-21T12:14:18.247419+0000 2024-09-21T12:15:53.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:53 smithi183 ceph-mon[33277]: min_mon_release 19 (squid) 2024-09-21T12:15:53.991 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:53 smithi183 ceph-mon[33277]: election_strategy: 1 2024-09-21T12:15:53.991 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:53 smithi183 ceph-mon[33277]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-21T12:15:53.991 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:53 smithi183 ceph-mon[33277]: 1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2024-09-21T12:15:53.991 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:53 smithi183 ceph-mon[33277]: fsmap 2024-09-21T12:15:53.991 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:53 smithi183 ceph-mon[33277]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T12:15:53.991 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:53 smithi183 ceph-mon[33277]: mgrmap e13: y(active, since 33s) 2024-09-21T12:15:53.991 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:53 smithi183 ceph-mon[33277]: overall HEALTH_OK 2024-09-21T12:15:53.991 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:53 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:53.991 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:53 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:53.991 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:53 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:53.991 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:53 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T12:15:53.992 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:53 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:15:54.008 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:53 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T12:15:54.008 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:53 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:15:54.008 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:53 smithi114 ceph-mon[30324]: mon.a calling monitor election 2024-09-21T12:15:54.009 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:53 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:15:54.009 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:53 smithi114 ceph-mon[30324]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:15:54.009 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:53 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:15:54.009 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:53 smithi114 ceph-mon[30324]: mon.b calling monitor election 2024-09-21T12:15:54.009 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:53 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:15:54.009 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:53 smithi114 ceph-mon[30324]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:15:54.009 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:53 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:15:54.009 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:53 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:15:54.009 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:53 smithi114 ceph-mon[30324]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T12:15:54.009 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:53 smithi114 ceph-mon[30324]: monmap epoch 2 2024-09-21T12:15:54.009 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:53 smithi114 ceph-mon[30324]: fsid e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T12:15:54.009 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:53 smithi114 ceph-mon[30324]: last_changed 2024-09-21T12:15:48.587882+0000 2024-09-21T12:15:54.009 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:53 smithi114 ceph-mon[30324]: created 2024-09-21T12:14:18.247419+0000 2024-09-21T12:15:54.010 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:53 smithi114 ceph-mon[30324]: min_mon_release 19 (squid) 2024-09-21T12:15:54.010 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:53 smithi114 ceph-mon[30324]: election_strategy: 1 2024-09-21T12:15:54.010 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:53 smithi114 ceph-mon[30324]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-21T12:15:54.010 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:53 smithi114 ceph-mon[30324]: 1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2024-09-21T12:15:54.010 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:53 smithi114 ceph-mon[30324]: fsmap 2024-09-21T12:15:54.010 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:53 smithi114 ceph-mon[30324]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T12:15:54.010 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:53 smithi114 ceph-mon[30324]: mgrmap e13: y(active, since 33s) 2024-09-21T12:15:54.010 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:53 smithi114 ceph-mon[30324]: overall HEALTH_OK 2024-09-21T12:15:54.010 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:53 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:54.010 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:53 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:54.010 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:53 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:54.010 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:53 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T12:15:54.010 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:53 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:15:54.468 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:15:54.468 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":2,"fsid":"e2ad975c-7812-11ef-baf6-efdc52797490","modified":"2024-09-21T12:15:48.587882Z","created":"2024-09-21T12:14:18.247419Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:3300","nonce":0},{"type":"v1","addr":"172.21.15.183:6789","nonce":0}]},"addr":"172.21.15.183:6789/0","public_addr":"172.21.15.183:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-21T12:15:54.468 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 2 2024-09-21T12:15:54.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:54 smithi183 ceph-mon[33277]: Deploying daemon mon.c on smithi114 2024-09-21T12:15:54.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:54 smithi183 ceph-mon[33277]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:15:54.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:54 smithi183 ceph-mon[33277]: from='client.? 172.21.15.183:0/3034655812' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T12:15:54.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:54 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:15:54.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:54 smithi114 ceph-mon[30324]: Deploying daemon mon.c on smithi114 2024-09-21T12:15:54.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:54 smithi114 ceph-mon[30324]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:15:54.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:54 smithi114 ceph-mon[30324]: from='client.? 172.21.15.183:0/3034655812' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T12:15:54.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:54 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:15:56.025 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:15:55 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: 2024-09-21T12:15:55.587+0000 7f7df9929640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-21T12:15:56.084 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T12:15:56.084 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph mon dump -f json 2024-09-21T12:15:56.339 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[30324]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:15:56.347 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:15:56.467 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:15:56 smithi183 ceph-mon[33277]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:15:56.717 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 systemd[1]: Starting Ceph mon.c for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:15:56.968 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 podman[40829]: 2024-09-21 12:15:56.618881189 +0000 UTC m=+0.015685005 image pull 8f4d013db793795fef09ff5edf96e6b42ba8cd2c9158b2f0b7a5d7c1e657c063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472 2024-09-21T12:15:56.968 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 podman[40829]: 2024-09-21 12:15:56.723108744 +0000 UTC m=+0.119912632 container create 78eecbc43ac75f9fce635b95e97306f07684f0aff5df76da86fe71daf4b30d47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-c, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-21T12:15:56.968 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 podman[40829]: 2024-09-21 12:15:56.933803404 +0000 UTC m=+0.330607229 container init 78eecbc43ac75f9fce635b95e97306f07684f0aff5df76da86fe71daf4b30d47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-c, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T12:15:56.968 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 podman[40829]: 2024-09-21 12:15:56.938222606 +0000 UTC m=+0.335026418 container start 78eecbc43ac75f9fce635b95e97306f07684f0aff5df76da86fe71daf4b30d47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-c, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-21T12:15:56.968 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T12:15:56.969 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: ceph version 19.3.0-5103-g5c1cdc3c (5c1cdc3c470dae78e91a0ed2bcf13509a046d301) squid (dev), process ceph-mon, pid 7 2024-09-21T12:15:56.969 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: pidfile_write: ignore empty --pid-file 2024-09-21T12:15:56.969 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: load: jerasure load: lrc 2024-09-21T12:15:56.969 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: RocksDB version: 7.9.2 2024-09-21T12:15:56.969 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Git sha 0 2024-09-21T12:15:56.969 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Compile date 2024-09-21 08:36:07 2024-09-21T12:15:56.969 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: DB SUMMARY 2024-09-21T12:15:56.969 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: DB Session ID: 47U1WDVMGKNIT4X50USJ 2024-09-21T12:15:56.969 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: CURRENT file: CURRENT 2024-09-21T12:15:56.969 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T12:15:56.969 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-21T12:15:56.969 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-21T12:15:56.970 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-09-21T12:15:56.970 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.error_if_exists: 0 2024-09-21T12:15:56.970 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.create_if_missing: 0 2024-09-21T12:15:56.970 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.paranoid_checks: 1 2024-09-21T12:15:56.970 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T12:15:56.971 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T12:15:56.971 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T12:15:56.971 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.env: 0x55fd257af120 2024-09-21T12:15:56.971 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T12:15:56.971 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.info_log: 0x55fd2649d700 2024-09-21T12:15:56.971 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T12:15:56.971 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.statistics: (nil) 2024-09-21T12:15:56.971 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.use_fsync: 0 2024-09-21T12:15:56.971 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.max_log_file_size: 0 2024-09-21T12:15:56.971 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T12:15:56.971 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T12:15:56.971 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T12:15:56.972 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T12:15:56.972 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.allow_fallocate: 1 2024-09-21T12:15:56.972 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T12:15:56.972 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T12:15:56.972 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.use_direct_reads: 0 2024-09-21T12:15:56.972 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T12:15:56.972 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T12:15:56.972 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.db_log_dir: 2024-09-21T12:15:56.972 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.wal_dir: 2024-09-21T12:15:56.972 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T12:15:56.972 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T12:15:56.972 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T12:15:56.973 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T12:15:56.973 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T12:15:56.973 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T12:15:56.973 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T12:15:56.973 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T12:15:56.973 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.write_buffer_manager: 0x55fd264a1b80 2024-09-21T12:15:56.973 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T12:15:56.973 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T12:15:56.973 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T12:15:56.973 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T12:15:56.973 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T12:15:56.973 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T12:15:56.973 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T12:15:56.973 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T12:15:56.973 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.unordered_write: 0 2024-09-21T12:15:56.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T12:15:56.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T12:15:56.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T12:15:56.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T12:15:56.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.row_cache: None 2024-09-21T12:15:56.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.wal_filter: None 2024-09-21T12:15:56.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T12:15:56.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T12:15:56.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.two_write_queues: 0 2024-09-21T12:15:56.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T12:15:56.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.wal_compression: 0 2024-09-21T12:15:56.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.atomic_flush: 0 2024-09-21T12:15:56.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T12:15:56.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T12:15:56.975 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T12:15:56.975 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.log_readahead_size: 0 2024-09-21T12:15:56.975 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T12:15:56.975 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T12:15:56.975 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T12:15:56.975 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T12:15:56.975 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T12:15:56.975 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T12:15:56.975 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T12:15:56.975 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.max_background_jobs: 2 2024-09-21T12:15:56.975 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.max_background_compactions: -1 2024-09-21T12:15:56.975 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.max_subcompactions: 1 2024-09-21T12:15:56.975 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T12:15:56.975 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T12:15:56.975 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T12:15:56.975 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T12:15:56.976 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T12:15:56.976 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T12:15:56.976 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T12:15:56.976 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T12:15:56.976 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.max_open_files: -1 2024-09-21T12:15:56.976 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T12:15:56.976 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T12:15:56.976 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T12:15:56.976 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T12:15:56.976 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.max_background_flushes: -1 2024-09-21T12:15:56.976 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Compression algorithms supported: 2024-09-21T12:15:56.976 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: kZSTD supported: 0 2024-09-21T12:15:56.976 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: kXpressCompression supported: 0 2024-09-21T12:15:56.977 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: kBZip2Compression supported: 0 2024-09-21T12:15:56.977 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T12:15:56.977 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: kLZ4Compression supported: 1 2024-09-21T12:15:56.977 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: kZlibCompression supported: 1 2024-09-21T12:15:56.977 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T12:15:56.977 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: kSnappyCompression supported: 1 2024-09-21T12:15:56.977 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T12:15:56.977 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T12:15:56.977 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-09-21T12:15:56.977 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T12:15:56.977 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T12:15:56.977 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.merge_operator: 2024-09-21T12:15:56.977 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.compaction_filter: None 2024-09-21T12:15:56.978 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T12:15:56.978 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T12:15:56.978 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T12:15:56.978 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T12:15:56.978 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fd2649d320) 2024-09-21T12:15:56.978 INFO:journalctl@ceph.mon.c.smithi114.stdout: cache_index_and_filter_blocks: 1 2024-09-21T12:15:56.978 INFO:journalctl@ceph.mon.c.smithi114.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T12:15:56.978 INFO:journalctl@ceph.mon.c.smithi114.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T12:15:56.978 INFO:journalctl@ceph.mon.c.smithi114.stdout: pin_top_level_index_and_filter: 1 2024-09-21T12:15:56.978 INFO:journalctl@ceph.mon.c.smithi114.stdout: index_type: 0 2024-09-21T12:15:56.978 INFO:journalctl@ceph.mon.c.smithi114.stdout: data_block_index_type: 0 2024-09-21T12:15:56.978 INFO:journalctl@ceph.mon.c.smithi114.stdout: index_shortening: 1 2024-09-21T12:15:56.978 INFO:journalctl@ceph.mon.c.smithi114.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T12:15:56.979 INFO:journalctl@ceph.mon.c.smithi114.stdout: checksum: 4 2024-09-21T12:15:56.979 INFO:journalctl@ceph.mon.c.smithi114.stdout: no_block_cache: 0 2024-09-21T12:15:56.979 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_cache: 0x55fd26491610 2024-09-21T12:15:56.979 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_cache_name: BinnedLRUCache 2024-09-21T12:15:56.979 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_cache_options: 2024-09-21T12:15:56.979 INFO:journalctl@ceph.mon.c.smithi114.stdout: capacity : 536870912 2024-09-21T12:15:56.979 INFO:journalctl@ceph.mon.c.smithi114.stdout: num_shard_bits : 4 2024-09-21T12:15:56.979 INFO:journalctl@ceph.mon.c.smithi114.stdout: strict_capacity_limit : 0 2024-09-21T12:15:56.979 INFO:journalctl@ceph.mon.c.smithi114.stdout: high_pri_pool_ratio: 0.000 2024-09-21T12:15:56.979 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_cache_compressed: (nil) 2024-09-21T12:15:56.979 INFO:journalctl@ceph.mon.c.smithi114.stdout: persistent_cache: (nil) 2024-09-21T12:15:56.979 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_size: 4096 2024-09-21T12:15:56.980 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_size_deviation: 10 2024-09-21T12:15:56.980 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_restart_interval: 16 2024-09-21T12:15:56.980 INFO:journalctl@ceph.mon.c.smithi114.stdout: index_block_restart_interval: 1 2024-09-21T12:15:56.980 INFO:journalctl@ceph.mon.c.smithi114.stdout: metadata_block_size: 4096 2024-09-21T12:15:56.981 INFO:journalctl@ceph.mon.c.smithi114.stdout: partition_filters: 0 2024-09-21T12:15:56.981 INFO:journalctl@ceph.mon.c.smithi114.stdout: use_delta_encoding: 1 2024-09-21T12:15:56.981 INFO:journalctl@ceph.mon.c.smithi114.stdout: filter_policy: bloomfilter 2024-09-21T12:15:56.981 INFO:journalctl@ceph.mon.c.smithi114.stdout: whole_key_filtering: 1 2024-09-21T12:15:56.981 INFO:journalctl@ceph.mon.c.smithi114.stdout: verify_compression: 0 2024-09-21T12:15:56.981 INFO:journalctl@ceph.mon.c.smithi114.stdout: read_amp_bytes_per_bit: 0 2024-09-21T12:15:56.981 INFO:journalctl@ceph.mon.c.smithi114.stdout: format_version: 5 2024-09-21T12:15:56.981 INFO:journalctl@ceph.mon.c.smithi114.stdout: enable_index_compression: 1 2024-09-21T12:15:56.981 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_align: 0 2024-09-21T12:15:56.981 INFO:journalctl@ceph.mon.c.smithi114.stdout: max_auto_readahead_size: 262144 2024-09-21T12:15:56.981 INFO:journalctl@ceph.mon.c.smithi114.stdout: prepopulate_block_cache: 0 2024-09-21T12:15:56.981 INFO:journalctl@ceph.mon.c.smithi114.stdout: initial_auto_readahead_size: 8192 2024-09-21T12:15:56.981 INFO:journalctl@ceph.mon.c.smithi114.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T12:15:56.982 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T12:15:56.982 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T12:15:56.982 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.compression: NoCompression 2024-09-21T12:15:56.982 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T12:15:56.982 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T12:15:56.982 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T12:15:56.982 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.num_levels: 7 2024-09-21T12:15:56.982 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T12:15:56.982 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T12:15:56.982 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T12:15:56.982 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T12:15:56.982 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T12:15:56.982 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T12:15:56.983 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T12:15:56.983 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T12:15:56.983 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T12:15:56.983 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T12:15:56.983 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T12:15:56.983 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T12:15:56.983 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T12:15:56.983 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T12:15:56.983 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T12:15:56.983 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T12:15:56.983 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T12:15:56.983 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T12:15:56.983 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T12:15:56.983 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T12:15:56.983 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T12:15:56.984 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T12:15:56.984 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T12:15:56.984 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T12:15:56.984 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T12:15:56.984 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T12:15:56.984 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T12:15:56.984 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T12:15:56.984 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T12:15:56.984 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T12:15:56.984 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T12:15:56.984 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T12:15:56.984 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T12:15:56.984 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T12:15:56.984 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T12:15:56.985 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T12:15:56.985 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T12:15:56.985 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T12:15:56.985 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T12:15:56.985 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T12:15:56.985 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T12:15:56.985 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T12:15:56.985 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T12:15:56.985 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T12:15:56.985 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T12:15:56.985 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T12:15:56.985 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T12:15:56.985 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T12:15:56.985 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T12:15:56.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T12:15:56.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T12:15:56.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T12:15:56.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T12:15:56.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T12:15:56.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.inplace_update_support: 0 2024-09-21T12:15:56.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T12:15:56.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T12:15:56.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T12:15:56.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T12:15:56.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.bloom_locality: 0 2024-09-21T12:15:56.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.max_successive_merges: 0 2024-09-21T12:15:56.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T12:15:56.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T12:15:56.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T12:15:56.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T12:15:56.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.ttl: 2592000 2024-09-21T12:15:56.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T12:15:56.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T12:15:56.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T12:15:56.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.enable_blob_files: false 2024-09-21T12:15:56.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.min_blob_size: 0 2024-09-21T12:15:56.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T12:15:56.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T12:15:56.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T12:15:56.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T12:15:56.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T12:15:56.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T12:15:56.987 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T12:15:56.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T12:15:56.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T12:15:56.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-21T12:15:56.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 630371b5-4178-4850-8a47-5bcae214f968 2024-09-21T12:15:56.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726920956964855, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-21T12:15:56.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-21T12:15:56.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726920956965248, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726920956, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "630371b5-4178-4850-8a47-5bcae214f968", "db_session_id": "47U1WDVMGKNIT4X50USJ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-21T12:15:56.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726920956965343, "job": 1, "event": "recovery_finished"} 2024-09-21T12:15:56.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-21T12:15:56.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T12:15:56.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55fd264c0e00 2024-09-21T12:15:56.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: DB pointer 0x55fd265c2000 2024-09-21T12:15:56.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T12:15:56.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T12:15:56.989 INFO:journalctl@ceph.mon.c.smithi114.stdout: ** DB Stats ** 2024-09-21T12:15:56.989 INFO:journalctl@ceph.mon.c.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T12:15:56.989 INFO:journalctl@ceph.mon.c.smithi114.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T12:15:56.989 INFO:journalctl@ceph.mon.c.smithi114.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T12:15:56.989 INFO:journalctl@ceph.mon.c.smithi114.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T12:15:56.989 INFO:journalctl@ceph.mon.c.smithi114.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T12:15:56.989 INFO:journalctl@ceph.mon.c.smithi114.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T12:15:56.989 INFO:journalctl@ceph.mon.c.smithi114.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T12:15:56.989 INFO:journalctl@ceph.mon.c.smithi114.stdout: 2024-09-21T12:15:56.989 INFO:journalctl@ceph.mon.c.smithi114.stdout: ** Compaction Stats [default] ** 2024-09-21T12:15:56.989 INFO:journalctl@ceph.mon.c.smithi114.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T12:15:56.989 INFO:journalctl@ceph.mon.c.smithi114.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T12:15:56.989 INFO:journalctl@ceph.mon.c.smithi114.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T12:15:56.989 INFO:journalctl@ceph.mon.c.smithi114.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T12:15:56.989 INFO:journalctl@ceph.mon.c.smithi114.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T12:15:56.989 INFO:journalctl@ceph.mon.c.smithi114.stdout: 2024-09-21T12:15:56.990 INFO:journalctl@ceph.mon.c.smithi114.stdout: ** Compaction Stats [default] ** 2024-09-21T12:15:56.990 INFO:journalctl@ceph.mon.c.smithi114.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T12:15:56.990 INFO:journalctl@ceph.mon.c.smithi114.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T12:15:56.990 INFO:journalctl@ceph.mon.c.smithi114.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T12:15:56.990 INFO:journalctl@ceph.mon.c.smithi114.stdout: 2024-09-21T12:15:56.990 INFO:journalctl@ceph.mon.c.smithi114.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T12:15:56.990 INFO:journalctl@ceph.mon.c.smithi114.stdout: 2024-09-21T12:15:56.990 INFO:journalctl@ceph.mon.c.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T12:15:56.990 INFO:journalctl@ceph.mon.c.smithi114.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T12:15:56.990 INFO:journalctl@ceph.mon.c.smithi114.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T12:15:56.990 INFO:journalctl@ceph.mon.c.smithi114.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T12:15:56.990 INFO:journalctl@ceph.mon.c.smithi114.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T12:15:56.990 INFO:journalctl@ceph.mon.c.smithi114.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T12:15:56.991 INFO:journalctl@ceph.mon.c.smithi114.stdout: Cumulative compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T12:15:56.991 INFO:journalctl@ceph.mon.c.smithi114.stdout: Interval compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T12:15:56.991 INFO:journalctl@ceph.mon.c.smithi114.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T12:15:56.991 INFO:journalctl@ceph.mon.c.smithi114.stdout: Block cache BinnedLRUCache@0x55fd26491610#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-21T12:15:56.991 INFO:journalctl@ceph.mon.c.smithi114.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-21T12:15:56.991 INFO:journalctl@ceph.mon.c.smithi114.stdout: 2024-09-21T12:15:56.991 INFO:journalctl@ceph.mon.c.smithi114.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T12:15:56.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-21T12:15:56.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: using public_addrv [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-21T12:15:56.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: starting mon.c rank -1 at public addrs [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] at bind addrs [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T12:15:56.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: mon.c@-1(???) e0 preinit fsid e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T12:15:57.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: mon.c@-1(synchronizing).mds e1 new map 2024-09-21T12:15:57.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-21T12:15:57.275 INFO:journalctl@ceph.mon.c.smithi114.stdout: e1 2024-09-21T12:15:57.276 INFO:journalctl@ceph.mon.c.smithi114.stdout: btime 2024-09-21T12:14:24:181010+0000 2024-09-21T12:15:57.276 INFO:journalctl@ceph.mon.c.smithi114.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T12:15:57.276 INFO:journalctl@ceph.mon.c.smithi114.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-21T12:15:57.276 INFO:journalctl@ceph.mon.c.smithi114.stdout: legacy client fscid: -1 2024-09-21T12:15:57.276 INFO:journalctl@ceph.mon.c.smithi114.stdout: 2024-09-21T12:15:57.276 INFO:journalctl@ceph.mon.c.smithi114.stdout: No filesystems configured 2024-09-21T12:15:57.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-21T12:15:57.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-21T12:15:57.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-21T12:15:57.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-21T12:15:57.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-21T12:15:57.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-21T12:15:57.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T12:15:57.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T12:15:57.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T12:15:57.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T12:15:57.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:15:57.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/477307236' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-21T12:15:57.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/477307236' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T12:15:57.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T12:15:57.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:57.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:57.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:57.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:57.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"} : dispatch 2024-09-21T12:15:57.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:15:57.278 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:15:57.278 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: Updating smithi183:/etc/ceph/ceph.conf 2024-09-21T12:15:57.278 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: Updating smithi183:/var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:15:57.278 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:15:57.278 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: Updating smithi183:/var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:15:57.278 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:57.278 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:57.278 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:57.278 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:15:57.278 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:57.278 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:15:57.278 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:15:57.279 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:15:57.279 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:57.279 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T12:15:57.279 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:15:57.279 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi114:172.21.15.114=a;smithi114:[v2:172.21.15.114:3301,v1:172.21.15.114:6790]=c;smithi183:172.21.15.183=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:15:57.279 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: Saving service mon spec with placement smithi114:172.21.15.114=a;smithi114:[v2:172.21.15.114:3301,v1:172.21.15.114:6790]=c;smithi183:172.21.15.183=b;count:3 2024-09-21T12:15:57.279 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: Deploying daemon mon.b on smithi183 2024-09-21T12:15:57.279 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:15:57.279 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='client.? 172.21.15.183:0/828193708' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T12:15:57.279 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T12:15:57.279 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:15:57.279 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: mon.a calling monitor election 2024-09-21T12:15:57.279 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:15:57.280 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:15:57.280 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:15:57.280 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: mon.b calling monitor election 2024-09-21T12:15:57.280 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:15:57.280 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:15:57.280 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:15:57.280 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:15:57.280 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T12:15:57.280 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: monmap epoch 2 2024-09-21T12:15:57.280 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: fsid e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T12:15:57.280 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: last_changed 2024-09-21T12:15:48.587882+0000 2024-09-21T12:15:57.280 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: created 2024-09-21T12:14:18.247419+0000 2024-09-21T12:15:57.280 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: min_mon_release 19 (squid) 2024-09-21T12:15:57.280 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: election_strategy: 1 2024-09-21T12:15:57.281 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-21T12:15:57.281 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: 1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2024-09-21T12:15:57.281 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: fsmap 2024-09-21T12:15:57.281 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T12:15:57.281 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: mgrmap e13: y(active, since 33s) 2024-09-21T12:15:57.281 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: overall HEALTH_OK 2024-09-21T12:15:57.281 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:57.281 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:57.281 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:15:57.281 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T12:15:57.281 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:15:57.281 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: Deploying daemon mon.c on smithi114 2024-09-21T12:15:57.281 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:15:57.281 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='client.? 172.21.15.183:0/3034655812' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T12:15:57.282 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:15:57.282 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:15:57.282 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:56 smithi114 ceph-mon[40867]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-21T12:15:57.282 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:57 smithi114 bash[40829]: 78eecbc43ac75f9fce635b95e97306f07684f0aff5df76da86fe71daf4b30d47 2024-09-21T12:15:57.282 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:15:57 smithi114 systemd[1]: Started Ceph mon.c for e2ad975c-7812-11ef-baf6-efdc52797490. 2024-09-21T12:16:02.002 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:16:02.002 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":3,"fsid":"e2ad975c-7812-11ef-baf6-efdc52797490","modified":"2024-09-21T12:15:56.990774Z","created":"2024-09-21T12:14:18.247419Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:3300","nonce":0},{"type":"v1","addr":"172.21.15.183:6789","nonce":0}]},"addr":"172.21.15.183:6789/0","public_addr":"172.21.15.183:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-21T12:16:02.003 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 3 2024-09-21T12:16:02.461 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T12:16:02.462 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: mon.a calling monitor election 2024-09-21T12:16:02.462 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:16:02.462 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:16:02.462 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: mon.b calling monitor election 2024-09-21T12:16:02.462 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: from='client.? 172.21.15.183:0/1820848226' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T12:16:02.462 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:16:02.462 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:02.462 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:16:02.462 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: mon.c calling monitor election 2024-09-21T12:16:02.462 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:16:02.462 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:02.462 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:16:02.463 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:16:02.463 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T12:16:02.463 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: monmap epoch 3 2024-09-21T12:16:02.463 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: fsid e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T12:16:02.463 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: last_changed 2024-09-21T12:15:56.990774+0000 2024-09-21T12:16:02.463 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: created 2024-09-21T12:14:18.247419+0000 2024-09-21T12:16:02.463 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: min_mon_release 19 (squid) 2024-09-21T12:16:02.463 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: election_strategy: 1 2024-09-21T12:16:02.463 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-21T12:16:02.463 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: 1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2024-09-21T12:16:02.463 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: 2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-21T12:16:02.463 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: fsmap 2024-09-21T12:16:02.463 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T12:16:02.463 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: mgrmap e13: y(active, since 41s) 2024-09-21T12:16:02.463 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: overall HEALTH_OK 2024-09-21T12:16:02.464 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:02.464 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:02.464 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:02.464 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:02 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:02.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T12:16:02.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: mon.a calling monitor election 2024-09-21T12:16:02.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:16:02.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:16:02.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: mon.b calling monitor election 2024-09-21T12:16:02.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: from='client.? 172.21.15.183:0/1820848226' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T12:16:02.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:16:02.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:02.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:16:02.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: mon.c calling monitor election 2024-09-21T12:16:02.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:16:02.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:02.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:16:02.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:16:02.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T12:16:02.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: monmap epoch 3 2024-09-21T12:16:02.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: fsid e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T12:16:02.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: last_changed 2024-09-21T12:15:56.990774+0000 2024-09-21T12:16:02.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: created 2024-09-21T12:14:18.247419+0000 2024-09-21T12:16:02.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: min_mon_release 19 (squid) 2024-09-21T12:16:02.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: election_strategy: 1 2024-09-21T12:16:02.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-21T12:16:02.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: 1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2024-09-21T12:16:02.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: 2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-21T12:16:02.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: fsmap 2024-09-21T12:16:02.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T12:16:02.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: mgrmap e13: y(active, since 41s) 2024-09-21T12:16:02.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: overall HEALTH_OK 2024-09-21T12:16:02.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:02.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:02.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:02.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:02.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T12:16:02.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: mon.a calling monitor election 2024-09-21T12:16:02.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:16:02.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:16:02.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: mon.b calling monitor election 2024-09-21T12:16:02.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: from='client.? 172.21.15.183:0/1820848226' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T12:16:02.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:16:02.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:02.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:16:02.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: mon.c calling monitor election 2024-09-21T12:16:02.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:16:02.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:02.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:16:02.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:16:02.529 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T12:16:02.529 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: monmap epoch 3 2024-09-21T12:16:02.529 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: fsid e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T12:16:02.529 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: last_changed 2024-09-21T12:15:56.990774+0000 2024-09-21T12:16:02.529 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: created 2024-09-21T12:14:18.247419+0000 2024-09-21T12:16:02.529 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: min_mon_release 19 (squid) 2024-09-21T12:16:02.529 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: election_strategy: 1 2024-09-21T12:16:02.529 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-21T12:16:02.529 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: 1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2024-09-21T12:16:02.529 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: 2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-21T12:16:02.529 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: fsmap 2024-09-21T12:16:02.529 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T12:16:02.529 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: mgrmap e13: y(active, since 41s) 2024-09-21T12:16:02.529 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: overall HEALTH_OK 2024-09-21T12:16:02.529 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:02.530 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:02.530 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:02.530 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:02 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:02.790 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-21T12:16:02.790 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph config generate-minimal-conf 2024-09-21T12:16:03.143 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:16:03.383 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:03 smithi114 ceph-mon[30324]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:03.383 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:03 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:16:03.383 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:03 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:16:03.383 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:03 smithi114 ceph-mon[40867]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:03.383 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:03 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:16:03.383 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:03 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:16:03.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:03 smithi183 ceph-mon[33277]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:03.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:03 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:16:03.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:03 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:16:03.788 INFO:teuthology.orchestra.run.smithi114.stdout:# minimal ceph.conf for e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T12:16:03.788 INFO:teuthology.orchestra.run.smithi114.stdout:[global] 2024-09-21T12:16:03.788 INFO:teuthology.orchestra.run.smithi114.stdout: fsid = e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T12:16:03.788 INFO:teuthology.orchestra.run.smithi114.stdout: mon_host = [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-21T12:16:04.380 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:04.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:16:04.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[30324]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-21T12:16:04.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[30324]: Updating smithi183:/etc/ceph/ceph.conf 2024-09-21T12:16:04.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[30324]: Updating smithi183:/var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:16:04.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[30324]: Updating smithi114:/var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:16:04.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/300488382' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:04.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[30324]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T12:16:04.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T12:16:04.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T12:16:04.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:04.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[30324]: Reconfiguring daemon mon.a on smithi114 2024-09-21T12:16:04.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[30324]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:04.383 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.383 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.383 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.383 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.383 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:04.383 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:16:04.383 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[40867]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-21T12:16:04.383 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[40867]: Updating smithi183:/etc/ceph/ceph.conf 2024-09-21T12:16:04.383 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[40867]: Updating smithi183:/var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:16:04.383 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[40867]: Updating smithi114:/var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:16:04.383 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/300488382' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:04.383 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.384 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.384 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.384 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.384 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.385 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.386 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.386 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.386 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.386 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[40867]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T12:16:04.386 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T12:16:04.386 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T12:16:04.386 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:04.386 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[40867]: Reconfiguring daemon mon.a on smithi114 2024-09-21T12:16:04.386 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:04 smithi114 ceph-mon[40867]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:04.406 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-21T12:16:04.406 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-21T12:16:04.406 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T12:16:04.434 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-21T12:16:04.434 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:16:04.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:04 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:04 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:04 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:04 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:04 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:04.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:04 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:16:04.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:04 smithi183 ceph-mon[33277]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-21T12:16:04.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:04 smithi183 ceph-mon[33277]: Updating smithi183:/etc/ceph/ceph.conf 2024-09-21T12:16:04.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:04 smithi183 ceph-mon[33277]: Updating smithi183:/var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:16:04.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:04 smithi183 ceph-mon[33277]: Updating smithi114:/var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:16:04.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:04 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/300488382' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:04.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:04 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:04 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:04 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:04 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:04 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:04 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:04 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:04 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:04 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:04.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:04 smithi183 ceph-mon[33277]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T12:16:04.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:04 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T12:16:04.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:04 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T12:16:04.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:04 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:04.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:04 smithi183 ceph-mon[33277]: Reconfiguring daemon mon.a on smithi114 2024-09-21T12:16:04.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:04 smithi183 ceph-mon[33277]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:04.499 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-09-21T12:16:04.499 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T12:16:04.528 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-09-21T12:16:04.528 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:16:04.592 INFO:tasks.cephadm:Adding mgr.y on smithi114 2024-09-21T12:16:04.593 INFO:tasks.cephadm:Adding mgr.x on smithi183 2024-09-21T12:16:04.593 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph orch apply mgr '2;smithi114=y;smithi183=x' 2024-09-21T12:16:04.880 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:16:05.530 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled mgr update... 2024-09-21T12:16:06.174 DEBUG:teuthology.orchestra.run.smithi183:mgr.x> sudo journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@mgr.x.service 2024-09-21T12:16:06.176 DEBUG:tasks.cephadm:set 0 configs 2024-09-21T12:16:06.176 INFO:tasks.cephadm:Deploying OSDs... 2024-09-21T12:16:06.176 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-21T12:16:06.176 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T12:16:06.192 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T12:16:06.192 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_1 2024-09-21T12:16:06.246 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T12:16:06.247 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T12:16:06.247 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 971 Links: 1 2024-09-21T12:16:06.247 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T12:16:06.247 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T12:16:06.247 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-09-21 12:15:56.448873965 +0000 2024-09-21T12:16:06.247 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-09-21 12:11:02.766400195 +0000 2024-09-21T12:16:06.247 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-09-21 12:11:02.766400195 +0000 2024-09-21T12:16:06.247 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-09-21 12:11:02.766400195 +0000 2024-09-21T12:16:06.247 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T12:16:06.310 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-09-21T12:16:06.310 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-09-21T12:16:06.310 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000188243 s, 2.7 MB/s 2024-09-21T12:16:06.311 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T12:16:06.368 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_2 2024-09-21T12:16:06.422 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T12:16:06.422 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T12:16:06.423 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 982 Links: 1 2024-09-21T12:16:06.423 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T12:16:06.423 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T12:16:06.423 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-09-21 12:15:56.448873965 +0000 2024-09-21T12:16:06.423 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-09-21 12:11:02.768400015 +0000 2024-09-21T12:16:06.423 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-09-21 12:11:02.768400015 +0000 2024-09-21T12:16:06.423 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-09-21 12:11:02.768400015 +0000 2024-09-21T12:16:06.423 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T12:16:06.483 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:06 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:06.483 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:06 smithi114 ceph-mon[30324]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi114=y;smithi183=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:16:06.484 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:06 smithi114 ceph-mon[30324]: Saving service mgr spec with placement smithi114=y;smithi183=x;count:2 2024-09-21T12:16:06.484 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:06 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:06.484 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:06 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:06.484 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:06 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:06.484 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:06 smithi114 ceph-mon[30324]: Reconfiguring mgr.y (unknown last config time)... 2024-09-21T12:16:06.484 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:06 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T12:16:06.484 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:06 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T12:16:06.484 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:06 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:06.484 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:06 smithi114 ceph-mon[30324]: Reconfiguring daemon mgr.y on smithi114 2024-09-21T12:16:06.484 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:06 smithi114 ceph-mon[30324]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:06.484 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:06 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:06.484 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:06 smithi114 ceph-mon[40867]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi114=y;smithi183=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:16:06.485 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:06 smithi114 ceph-mon[40867]: Saving service mgr spec with placement smithi114=y;smithi183=x;count:2 2024-09-21T12:16:06.485 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:06 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:06.485 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:06 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:06.485 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:06 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:06.485 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:06 smithi114 ceph-mon[40867]: Reconfiguring mgr.y (unknown last config time)... 2024-09-21T12:16:06.485 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:06 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T12:16:06.485 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:06 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T12:16:06.485 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:06 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:06.485 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:06 smithi114 ceph-mon[40867]: Reconfiguring daemon mgr.y on smithi114 2024-09-21T12:16:06.485 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:06 smithi114 ceph-mon[40867]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:06.485 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-09-21T12:16:06.486 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-09-21T12:16:06.486 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000218192 s, 2.3 MB/s 2024-09-21T12:16:06.487 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T12:16:06.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:06 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:06.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:06 smithi183 ceph-mon[33277]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi114=y;smithi183=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:16:06.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:06 smithi183 ceph-mon[33277]: Saving service mgr spec with placement smithi114=y;smithi183=x;count:2 2024-09-21T12:16:06.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:06 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:06.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:06 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:06.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:06 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:06.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:06 smithi183 ceph-mon[33277]: Reconfiguring mgr.y (unknown last config time)... 2024-09-21T12:16:06.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:06 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T12:16:06.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:06 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T12:16:06.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:06 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:06.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:06 smithi183 ceph-mon[33277]: Reconfiguring daemon mgr.y on smithi114 2024-09-21T12:16:06.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:06 smithi183 ceph-mon[33277]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:06.544 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_3 2024-09-21T12:16:06.600 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T12:16:06.600 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T12:16:06.600 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 983 Links: 1 2024-09-21T12:16:06.600 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T12:16:06.600 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T12:16:06.600 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-09-21 12:15:56.449873943 +0000 2024-09-21T12:16:06.600 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-09-21 12:11:02.768400015 +0000 2024-09-21T12:16:06.600 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-09-21 12:11:02.768400015 +0000 2024-09-21T12:16:06.600 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-09-21 12:11:02.768400015 +0000 2024-09-21T12:16:06.601 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T12:16:06.664 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-09-21T12:16:06.664 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-09-21T12:16:06.664 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000198338 s, 2.6 MB/s 2024-09-21T12:16:06.665 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T12:16:06.722 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_4 2024-09-21T12:16:06.776 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T12:16:06.776 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T12:16:06.777 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 966 Links: 1 2024-09-21T12:16:06.777 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T12:16:06.777 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T12:16:06.777 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-09-21 12:15:56.449873943 +0000 2024-09-21T12:16:06.777 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-09-21 12:11:02.766400195 +0000 2024-09-21T12:16:06.777 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-09-21 12:11:02.766400195 +0000 2024-09-21T12:16:06.777 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-09-21 12:11:02.766400195 +0000 2024-09-21T12:16:06.777 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T12:16:06.840 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-09-21T12:16:06.840 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-09-21T12:16:06.840 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000191047 s, 2.7 MB/s 2024-09-21T12:16:06.841 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T12:16:06.897 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-09-21T12:16:06.897 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T12:16:06.915 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T12:16:06.915 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_1 2024-09-21T12:16:06.972 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T12:16:06.972 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T12:16:06.972 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 978 Links: 1 2024-09-21T12:16:06.972 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T12:16:06.972 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T12:16:06.972 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-09-21 12:15:47.847617079 +0000 2024-09-21T12:16:06.972 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-09-21 12:11:02.427294205 +0000 2024-09-21T12:16:06.972 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-09-21 12:11:02.427294205 +0000 2024-09-21T12:16:06.972 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-09-21 12:11:02.427294205 +0000 2024-09-21T12:16:06.973 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T12:16:07.038 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-09-21T12:16:07.038 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-09-21T12:16:07.038 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000219462 s, 2.3 MB/s 2024-09-21T12:16:07.039 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T12:16:07.096 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_2 2024-09-21T12:16:07.152 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T12:16:07.152 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T12:16:07.153 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 970 Links: 1 2024-09-21T12:16:07.153 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T12:16:07.153 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T12:16:07.153 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-09-21 12:15:47.848617053 +0000 2024-09-21T12:16:07.153 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-09-21 12:11:02.427294205 +0000 2024-09-21T12:16:07.153 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-09-21 12:11:02.427294205 +0000 2024-09-21T12:16:07.153 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-09-21 12:11:02.427294205 +0000 2024-09-21T12:16:07.153 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T12:16:07.217 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-09-21T12:16:07.217 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-09-21T12:16:07.217 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000234581 s, 2.2 MB/s 2024-09-21T12:16:07.218 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T12:16:07.275 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_3 2024-09-21T12:16:07.330 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T12:16:07.331 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T12:16:07.331 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 975 Links: 1 2024-09-21T12:16:07.331 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T12:16:07.331 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T12:16:07.331 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-09-21 12:15:47.848617053 +0000 2024-09-21T12:16:07.331 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-09-21 12:11:02.427294205 +0000 2024-09-21T12:16:07.331 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-09-21 12:11:02.427294205 +0000 2024-09-21T12:16:07.331 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-09-21 12:11:02.427294205 +0000 2024-09-21T12:16:07.331 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T12:16:07.394 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-09-21T12:16:07.394 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-09-21T12:16:07.394 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000190613 s, 2.7 MB/s 2024-09-21T12:16:07.395 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T12:16:07.451 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_4 2024-09-21T12:16:07.507 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T12:16:07.507 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T12:16:07.507 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 980 Links: 1 2024-09-21T12:16:07.507 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T12:16:07.507 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T12:16:07.507 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-09-21 12:15:47.848617053 +0000 2024-09-21T12:16:07.507 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-09-21 12:11:02.428294154 +0000 2024-09-21T12:16:07.507 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-09-21 12:11:02.428294154 +0000 2024-09-21T12:16:07.507 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-09-21 12:11:02.428294154 +0000 2024-09-21T12:16:07.508 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T12:16:07.571 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-09-21T12:16:07.571 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-09-21T12:16:07.571 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000183898 s, 2.8 MB/s 2024-09-21T12:16:07.572 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T12:16:07.628 INFO:tasks.cephadm:Deploying osd.0 on smithi114 with /dev/vg_nvme/lv_4... 2024-09-21T12:16:07.629 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T12:16:07.870 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:16:08.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:07 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:08.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:07 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:08.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:07 smithi114 ceph-mon[30324]: Reconfiguring mon.c (monmap changed)... 2024-09-21T12:16:08.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:07 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T12:16:08.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:07 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T12:16:08.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:07 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:08.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:07 smithi114 ceph-mon[30324]: Reconfiguring daemon mon.c on smithi114 2024-09-21T12:16:08.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:07 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:08.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:07 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:08.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:07 smithi114 ceph-mon[40867]: Reconfiguring mon.c (monmap changed)... 2024-09-21T12:16:08.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:07 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T12:16:08.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:07 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T12:16:08.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:07 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:08.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:07 smithi114 ceph-mon[40867]: Reconfiguring daemon mon.c on smithi114 2024-09-21T12:16:08.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:07 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:08.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:07 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:08.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:07 smithi183 ceph-mon[33277]: Reconfiguring mon.c (monmap changed)... 2024-09-21T12:16:08.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:07 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T12:16:08.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:07 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T12:16:08.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:07 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:08.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:07 smithi183 ceph-mon[33277]: Reconfiguring daemon mon.c on smithi114 2024-09-21T12:16:09.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:08 smithi183 ceph-mon[33277]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:09.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:08 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:09.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:08 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:09.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:08 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T12:16:09.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:08 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T12:16:09.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:08 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:09.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:08 smithi114 ceph-mon[30324]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:09.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:08 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:09.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:08 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:09.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:08 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T12:16:09.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:08 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T12:16:09.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:08 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:09.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:08 smithi114 ceph-mon[40867]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:09.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:08 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:09.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:08 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:09.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:08 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T12:16:09.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:08 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T12:16:09.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:08 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:10.253 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:09 smithi183 ceph-mon[33277]: Reconfiguring mon.b (monmap changed)... 2024-09-21T12:16:10.254 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:09 smithi183 ceph-mon[33277]: Reconfiguring daemon mon.b on smithi183 2024-09-21T12:16:10.254 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:10.254 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:10.254 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:16:10.254 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:10.254 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:16:10.254 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:10.255 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T12:16:10.255 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T12:16:10.255 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T12:16:10.255 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:10.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:09 smithi114 ceph-mon[30324]: Reconfiguring mon.b (monmap changed)... 2024-09-21T12:16:10.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:09 smithi114 ceph-mon[30324]: Reconfiguring daemon mon.b on smithi183 2024-09-21T12:16:10.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:10.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:10.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:16:10.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:10.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:16:10.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:10.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T12:16:10.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T12:16:10.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T12:16:10.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:10.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:09 smithi114 ceph-mon[40867]: Reconfiguring mon.b (monmap changed)... 2024-09-21T12:16:10.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:09 smithi114 ceph-mon[40867]: Reconfiguring daemon mon.b on smithi183 2024-09-21T12:16:10.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:10.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:10.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:16:10.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:10.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:16:10.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:10.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T12:16:10.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T12:16:10.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T12:16:10.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:10.985 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:16:11.012 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi114:vg_nvme/lv_4 2024-09-21T12:16:11.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:11 smithi183 ceph-mon[33277]: Deploying daemon mgr.x on smithi183 2024-09-21T12:16:11.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:11 smithi183 ceph-mon[33277]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:11.255 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:16:11.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:10 smithi114 ceph-mon[40867]: Deploying daemon mgr.x on smithi183 2024-09-21T12:16:11.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:10 smithi114 ceph-mon[40867]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:11.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:10 smithi114 ceph-mon[30324]: Deploying daemon mgr.x on smithi183 2024-09-21T12:16:11.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:10 smithi114 ceph-mon[30324]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:11.869 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Sep 21 12:16:11 smithi183 systemd[1]: Starting Ceph mgr.x for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:16:12.188 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:12 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T12:16:12.188 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:12 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T12:16:12.188 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:12 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:12.189 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Sep 21 12:16:11 smithi183 podman[34788]: 2024-09-21 12:16:11.868140282 +0000 UTC m=+0.103175314 container create 4b8441eb92187ba7467a7e43f79f2558574cb864b77a4eb795b06bc4d05f2bfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-x, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-21T12:16:12.189 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Sep 21 12:16:11 smithi183 podman[34788]: 2024-09-21 12:16:11.780229301 +0000 UTC m=+0.015264338 image pull 8f4d013db793795fef09ff5edf96e6b42ba8cd2c9158b2f0b7a5d7c1e657c063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472 2024-09-21T12:16:12.189 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Sep 21 12:16:12 smithi183 podman[34788]: 2024-09-21 12:16:12.07880581 +0000 UTC m=+0.313840856 container init 4b8441eb92187ba7467a7e43f79f2558574cb864b77a4eb795b06bc4d05f2bfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, GIT_BRANCH=HEAD) 2024-09-21T12:16:12.189 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Sep 21 12:16:12 smithi183 podman[34788]: 2024-09-21 12:16:12.083715338 +0000 UTC m=+0.318750370 container start 4b8441eb92187ba7467a7e43f79f2558574cb864b77a4eb795b06bc4d05f2bfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-x, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, io.buildah.version=1.37.2) 2024-09-21T12:16:12.189 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Sep 21 12:16:12 smithi183 ceph-mgr[34826]: -- 172.21.15.183:0/3892906340 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564d656150e0 con 0x564d655f3000 2024-09-21T12:16:12.189 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Sep 21 12:16:12 smithi183 bash[34788]: 4b8441eb92187ba7467a7e43f79f2558574cb864b77a4eb795b06bc4d05f2bfb 2024-09-21T12:16:12.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:12 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T12:16:12.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:12 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T12:16:12.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:12 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:12.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:12 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T12:16:12.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:12 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T12:16:12.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:12 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:12.488 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Sep 21 12:16:12 smithi183 systemd[1]: Started Ceph mgr.x for e2ad975c-7812-11ef-baf6-efdc52797490. 2024-09-21T12:16:13.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:13 smithi183 ceph-mon[33277]: from='client.24107 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:16:13.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:13 smithi183 ceph-mon[33277]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:13.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:13 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:13.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:13 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:13.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:13 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:13.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:13 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:13.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:13 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:16:13.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:13 smithi114 ceph-mon[30324]: from='client.24107 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:16:13.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:13 smithi114 ceph-mon[30324]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:13.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:13 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:13.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:13 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:13.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:13 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:13.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:13 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:13.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:13 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:16:13.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:13 smithi114 ceph-mon[40867]: from='client.24107 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:16:13.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:13 smithi114 ceph-mon[40867]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:13.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:13 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:13.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:13 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:13.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:13 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:13.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:13 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:13.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:13 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:16:14.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:14 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:14.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:14 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:14.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:14 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:14.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:14 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:16:14.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:14 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:14.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:14 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:16:14.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:14 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:14.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:14 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:16:14.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:14 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:14.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:14 smithi183 ceph-mon[33277]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:14.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:14 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2213399096' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "5715e521-c7dc-41dd-a075-5c96a88fa9b3"} : dispatch 2024-09-21T12:16:14.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:14 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2213399096' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5715e521-c7dc-41dd-a075-5c96a88fa9b3"}]': finished 2024-09-21T12:16:14.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:14 smithi183 ceph-mon[33277]: osdmap e5: 1 total, 0 up, 1 in 2024-09-21T12:16:14.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:14 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T12:16:14.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:14 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:14.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:14 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:14.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:14 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:14.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:14 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:16:14.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:14 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:14.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:14 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:16:14.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:14 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:14.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:14 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:16:14.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:14 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:14.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:14 smithi114 ceph-mon[30324]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:14.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:14 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2213399096' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "5715e521-c7dc-41dd-a075-5c96a88fa9b3"} : dispatch 2024-09-21T12:16:14.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:14 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2213399096' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5715e521-c7dc-41dd-a075-5c96a88fa9b3"}]': finished 2024-09-21T12:16:14.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:14 smithi114 ceph-mon[30324]: osdmap e5: 1 total, 0 up, 1 in 2024-09-21T12:16:14.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:14 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T12:16:14.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:14 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:14.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:14 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:14.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:14 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:14.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:14 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:16:14.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:14 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:14.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:14 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:16:14.777 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:14 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:14.777 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:14 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:16:14.777 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:14 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:14.777 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:14 smithi114 ceph-mon[40867]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:14.777 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:14 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2213399096' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "5715e521-c7dc-41dd-a075-5c96a88fa9b3"} : dispatch 2024-09-21T12:16:14.777 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:14 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2213399096' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5715e521-c7dc-41dd-a075-5c96a88fa9b3"}]': finished 2024-09-21T12:16:14.777 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:14 smithi114 ceph-mon[40867]: osdmap e5: 1 total, 0 up, 1 in 2024-09-21T12:16:14.777 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:14 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T12:16:15.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:15 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1594618292' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T12:16:15.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:15 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:15.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:15 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1594618292' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T12:16:15.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:15 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:15.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:15 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1594618292' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T12:16:15.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:15 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:16.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:16 smithi183 ceph-mon[33277]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:16.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:16 smithi114 ceph-mon[30324]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:16.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:16 smithi114 ceph-mon[40867]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:17.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:17 smithi183 ceph-mon[33277]: Standby manager daemon x started 2024-09-21T12:16:17.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:17 smithi183 ceph-mon[33277]: from='mgr.? 172.21.15.183:0/2193762591' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-21T12:16:17.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:17 smithi183 ceph-mon[33277]: from='mgr.? 172.21.15.183:0/2193762591' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T12:16:17.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:17 smithi183 ceph-mon[33277]: from='mgr.? 172.21.15.183:0/2193762591' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-21T12:16:17.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:17 smithi183 ceph-mon[33277]: from='mgr.? 172.21.15.183:0/2193762591' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T12:16:18.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:17 smithi114 ceph-mon[30324]: Standby manager daemon x started 2024-09-21T12:16:18.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:17 smithi114 ceph-mon[30324]: from='mgr.? 172.21.15.183:0/2193762591' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-21T12:16:18.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:17 smithi114 ceph-mon[30324]: from='mgr.? 172.21.15.183:0/2193762591' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T12:16:18.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:17 smithi114 ceph-mon[30324]: from='mgr.? 172.21.15.183:0/2193762591' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-21T12:16:18.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:17 smithi114 ceph-mon[30324]: from='mgr.? 172.21.15.183:0/2193762591' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T12:16:18.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:17 smithi114 ceph-mon[40867]: Standby manager daemon x started 2024-09-21T12:16:18.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:17 smithi114 ceph-mon[40867]: from='mgr.? 172.21.15.183:0/2193762591' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-21T12:16:18.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:17 smithi114 ceph-mon[40867]: from='mgr.? 172.21.15.183:0/2193762591' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T12:16:18.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:17 smithi114 ceph-mon[40867]: from='mgr.? 172.21.15.183:0/2193762591' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-21T12:16:18.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:17 smithi114 ceph-mon[40867]: from='mgr.? 172.21.15.183:0/2193762591' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T12:16:18.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:18 smithi183 ceph-mon[33277]: mgrmap e14: y(active, since 57s), standbys: x 2024-09-21T12:16:18.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:18 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T12:16:18.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:18 smithi183 ceph-mon[33277]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:19.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:18 smithi114 ceph-mon[40867]: mgrmap e14: y(active, since 57s), standbys: x 2024-09-21T12:16:19.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:18 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T12:16:19.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:18 smithi114 ceph-mon[40867]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:19.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:18 smithi114 ceph-mon[30324]: mgrmap e14: y(active, since 57s), standbys: x 2024-09-21T12:16:19.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:18 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T12:16:19.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:18 smithi114 ceph-mon[30324]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:20.349 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:20 smithi114 ceph-mon[30324]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:20.350 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:20 smithi114 ceph-mon[40867]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:20 smithi183 ceph-mon[33277]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:21.722 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:21 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T12:16:21.722 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:21 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:21.722 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:21 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T12:16:21.722 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:21 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:21.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:21 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T12:16:21.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:21 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:22.661 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:22 smithi114 ceph-mon[30324]: Deploying daemon osd.0 on smithi114 2024-09-21T12:16:22.661 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:22 smithi114 ceph-mon[30324]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:22.661 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:22 smithi114 ceph-mon[40867]: Deploying daemon osd.0 on smithi114 2024-09-21T12:16:22.661 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:22 smithi114 ceph-mon[40867]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:22.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:22 smithi183 ceph-mon[33277]: Deploying daemon osd.0 on smithi114 2024-09-21T12:16:22.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:22 smithi183 ceph-mon[33277]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:24.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:24 smithi183 ceph-mon[33277]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:24.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:24 smithi114 ceph-mon[30324]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:24.506 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:24 smithi114 ceph-mon[40867]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:26.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:26 smithi183 ceph-mon[33277]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:26.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:26 smithi114 ceph-mon[30324]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:26.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:26 smithi114 ceph-mon[40867]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:28.449 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:28 smithi114 ceph-mon[40867]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:28.449 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:28 smithi114 ceph-mon[30324]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:28.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:28 smithi183 ceph-mon[33277]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:29.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:29 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:16:29.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:29 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:29.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:29 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:29.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:29 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:16:29.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:29 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:29.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:29 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:29.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:29 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:16:29.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:29 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:29.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:29 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:30.769 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:30 smithi114 ceph-mon[30324]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:30.769 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:30 smithi114 ceph-mon[40867]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:30.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:30 smithi183 ceph-mon[33277]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:33.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:32 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:33.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:32 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:33.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:32 smithi183 ceph-mon[33277]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:33.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:32 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:33.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:32 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:16:33.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:32 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:33.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:32 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:33.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:32 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:33.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:32 smithi114 ceph-mon[30324]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:33.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:32 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:33.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:32 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:16:33.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:32 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:33.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:32 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:33.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:32 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:33.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:32 smithi114 ceph-mon[40867]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:33.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:32 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:33.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:32 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:16:33.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:32 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:34.479 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:34 smithi183 ceph-mon[33277]: from='osd.0 [v2:172.21.15.114:6802/3555869816,v1:172.21.15.114:6803/3555869816]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T12:16:34.479 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:34 smithi183 ceph-mon[33277]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T12:16:34.479 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:34 smithi183 ceph-mon[33277]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:34.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:34 smithi114 ceph-mon[30324]: from='osd.0 [v2:172.21.15.114:6802/3555869816,v1:172.21.15.114:6803/3555869816]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T12:16:34.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:34 smithi114 ceph-mon[30324]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T12:16:34.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:34 smithi114 ceph-mon[30324]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:34.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:34 smithi114 ceph-mon[40867]: from='osd.0 [v2:172.21.15.114:6802/3555869816,v1:172.21.15.114:6803/3555869816]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T12:16:34.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:34 smithi114 ceph-mon[40867]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T12:16:34.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:34 smithi114 ceph-mon[40867]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:34.897 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 0 on host 'smithi114' 2024-09-21T12:16:35.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:35 smithi183 ceph-mon[33277]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T12:16:35.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:35 smithi183 ceph-mon[33277]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T12:16:35.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:35 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T12:16:35.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:35 smithi183 ceph-mon[33277]: from='osd.0 [v2:172.21.15.114:6802/3555869816,v1:172.21.15.114:6803/3555869816]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-21T12:16:35.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:35 smithi183 ceph-mon[33277]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-21T12:16:35.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:35 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:16:35.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:35 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:35.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:35 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:35.499 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:35 smithi114 ceph-mon[30324]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T12:16:35.499 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:35 smithi114 ceph-mon[30324]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T12:16:35.499 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:35 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T12:16:35.499 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:35 smithi114 ceph-mon[30324]: from='osd.0 [v2:172.21.15.114:6802/3555869816,v1:172.21.15.114:6803/3555869816]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-21T12:16:35.499 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:35 smithi114 ceph-mon[30324]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-21T12:16:35.499 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:35 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:16:35.499 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:35 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:35.499 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:35 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:35.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:35 smithi114 ceph-mon[40867]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T12:16:35.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:35 smithi114 ceph-mon[40867]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T12:16:35.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:35 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T12:16:35.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:35 smithi114 ceph-mon[40867]: from='osd.0 [v2:172.21.15.114:6802/3555869816,v1:172.21.15.114:6803/3555869816]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-21T12:16:35.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:35 smithi114 ceph-mon[40867]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-21T12:16:35.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:35 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:16:35.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:35 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:35.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:35 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:35.528 DEBUG:teuthology.orchestra.run.smithi114:osd.0> sudo journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@osd.0.service 2024-09-21T12:16:35.530 INFO:tasks.cephadm:Deploying osd.1 on smithi114 with /dev/vg_nvme/lv_3... 2024-09-21T12:16:35.531 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T12:16:36.136 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:36 smithi114 ceph-mon[40867]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-21T12:16:36.136 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:36 smithi114 ceph-mon[40867]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T12:16:36.136 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:36 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T12:16:36.136 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:36 smithi114 ceph-mon[40867]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:36.137 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:36 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T12:16:36.137 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:36 smithi114 ceph-mon[30324]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-21T12:16:36.137 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:36 smithi114 ceph-mon[30324]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T12:16:36.137 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:36 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T12:16:36.137 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:36 smithi114 ceph-mon[30324]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:36.137 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:36 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T12:16:36.376 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:16:36.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:36 smithi183 ceph-mon[33277]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-21T12:16:36.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:36 smithi183 ceph-mon[33277]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T12:16:36.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:36 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T12:16:36.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:36 smithi183 ceph-mon[33277]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:16:36.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:36 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T12:16:37.403 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:37 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:16:37.403 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:37 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:16:37.404 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:37 smithi114 ceph-mon[30324]: osd.0 [v2:172.21.15.114:6802/3555869816,v1:172.21.15.114:6803/3555869816] boot 2024-09-21T12:16:37.404 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:37 smithi114 ceph-mon[30324]: osdmap e8: 1 total, 1 up, 1 in 2024-09-21T12:16:37.404 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:37 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T12:16:37.404 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:37 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:16:37.404 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:37 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:16:37.404 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:37 smithi114 ceph-mon[40867]: osd.0 [v2:172.21.15.114:6802/3555869816,v1:172.21.15.114:6803/3555869816] boot 2024-09-21T12:16:37.404 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:37 smithi114 ceph-mon[40867]: osdmap e8: 1 total, 1 up, 1 in 2024-09-21T12:16:37.404 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:37 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T12:16:37.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:37 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:16:37.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:37 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:16:37.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:37 smithi183 ceph-mon[33277]: osd.0 [v2:172.21.15.114:6802/3555869816,v1:172.21.15.114:6803/3555869816] boot 2024-09-21T12:16:37.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:37 smithi183 ceph-mon[33277]: osdmap e8: 1 total, 1 up, 1 in 2024-09-21T12:16:37.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:37 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T12:16:38.944 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:38 smithi114 ceph-mon[40867]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T12:16:38.944 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:38 smithi114 ceph-mon[40867]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:38.944 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:38 smithi114 ceph-mon[30324]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T12:16:38.944 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:38 smithi114 ceph-mon[30324]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:38.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:38 smithi183 ceph-mon[33277]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T12:16:38.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:38 smithi183 ceph-mon[33277]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:40.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:39 smithi183 ceph-mon[33277]: Detected new or changed devices on smithi114 2024-09-21T12:16:40.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:39 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:40.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:39 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:40.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:39 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T12:16:40.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:39 smithi183 ceph-mon[33277]: Adjusting osd_memory_target on smithi114 to 16064M 2024-09-21T12:16:40.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:39 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:40.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:39 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:40.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:39 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:16:40.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:39 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:40.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:39 smithi114 ceph-mon[40867]: Detected new or changed devices on smithi114 2024-09-21T12:16:40.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:39 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:40.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:39 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:40.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:39 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T12:16:40.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:39 smithi114 ceph-mon[40867]: Adjusting osd_memory_target on smithi114 to 16064M 2024-09-21T12:16:40.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:39 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:40.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:39 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:40.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:39 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:16:40.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:39 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:40.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:39 smithi114 ceph-mon[30324]: Detected new or changed devices on smithi114 2024-09-21T12:16:40.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:39 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:40.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:39 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:40.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:39 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T12:16:40.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:39 smithi114 ceph-mon[30324]: Adjusting osd_memory_target on smithi114 to 16064M 2024-09-21T12:16:40.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:39 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:40.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:39 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:40.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:39 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:16:40.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:39 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:40.645 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:16:40.674 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi114:vg_nvme/lv_3 2024-09-21T12:16:40.918 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:16:41.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:40 smithi183 ceph-mon[33277]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:41.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:40 smithi114 ceph-mon[30324]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:41.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:40 smithi114 ceph-mon[40867]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:42.191 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:41 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T12:16:42.191 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:41 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T12:16:42.191 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:41 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:42.191 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:41 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T12:16:42.191 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:41 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T12:16:42.191 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:41 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:42.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:41 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T12:16:42.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:41 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T12:16:42.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:41 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:43.158 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:42 smithi114 ceph-mon[30324]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:16:43.158 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:42 smithi114 ceph-mon[30324]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:43.158 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:42 smithi114 ceph-mon[40867]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:16:43.158 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:42 smithi114 ceph-mon[40867]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:43.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:42 smithi183 ceph-mon[33277]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:16:43.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:42 smithi183 ceph-mon[33277]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:44.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:43 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2263399221' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1fe160a7-5972-4e3e-99ce-15e2311ba4c5"} : dispatch 2024-09-21T12:16:44.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:43 smithi114 ceph-mon[30324]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1fe160a7-5972-4e3e-99ce-15e2311ba4c5"} : dispatch 2024-09-21T12:16:44.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:43 smithi114 ceph-mon[30324]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1fe160a7-5972-4e3e-99ce-15e2311ba4c5"}]': finished 2024-09-21T12:16:44.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:43 smithi114 ceph-mon[30324]: osdmap e10: 2 total, 1 up, 2 in 2024-09-21T12:16:44.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:43 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T12:16:44.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:43 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2263399221' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1fe160a7-5972-4e3e-99ce-15e2311ba4c5"} : dispatch 2024-09-21T12:16:44.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:43 smithi114 ceph-mon[40867]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1fe160a7-5972-4e3e-99ce-15e2311ba4c5"} : dispatch 2024-09-21T12:16:44.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:43 smithi114 ceph-mon[40867]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1fe160a7-5972-4e3e-99ce-15e2311ba4c5"}]': finished 2024-09-21T12:16:44.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:43 smithi114 ceph-mon[40867]: osdmap e10: 2 total, 1 up, 2 in 2024-09-21T12:16:44.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:43 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T12:16:44.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:43 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2263399221' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1fe160a7-5972-4e3e-99ce-15e2311ba4c5"} : dispatch 2024-09-21T12:16:44.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:43 smithi183 ceph-mon[33277]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1fe160a7-5972-4e3e-99ce-15e2311ba4c5"} : dispatch 2024-09-21T12:16:44.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:43 smithi183 ceph-mon[33277]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1fe160a7-5972-4e3e-99ce-15e2311ba4c5"}]': finished 2024-09-21T12:16:44.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:43 smithi183 ceph-mon[33277]: osdmap e10: 2 total, 1 up, 2 in 2024-09-21T12:16:44.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:43 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T12:16:45.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:44 smithi114 ceph-mon[40867]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:45.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:44 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2809920179' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T12:16:45.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:44 smithi114 ceph-mon[30324]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:45.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:44 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2809920179' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T12:16:45.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:44 smithi183 ceph-mon[33277]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:45.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:44 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2809920179' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T12:16:46.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:46 smithi183 ceph-mon[33277]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:46.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:46 smithi114 ceph-mon[30324]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:46.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:46 smithi114 ceph-mon[40867]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:48.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:48 smithi114 ceph-mon[30324]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:48.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:48 smithi114 ceph-mon[40867]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:48.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:48 smithi183 ceph-mon[33277]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:50.431 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:50 smithi114 ceph-mon[30324]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:50.431 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:50 smithi114 ceph-mon[40867]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:50.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:50 smithi183 ceph-mon[33277]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:51.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:51 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T12:16:51.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:51 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:51.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:51 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T12:16:51.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:51 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:51.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:51 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T12:16:51.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:51 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:16:52.647 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:52 smithi114 ceph-mon[30324]: Deploying daemon osd.1 on smithi114 2024-09-21T12:16:52.647 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:52 smithi114 ceph-mon[30324]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:52.647 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:52 smithi114 ceph-mon[40867]: Deploying daemon osd.1 on smithi114 2024-09-21T12:16:52.647 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:52 smithi114 ceph-mon[40867]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:52.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:52 smithi183 ceph-mon[33277]: Deploying daemon osd.1 on smithi114 2024-09-21T12:16:52.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:52 smithi183 ceph-mon[33277]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:54.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:54 smithi183 ceph-mon[33277]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:54.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:54 smithi114 ceph-mon[30324]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:54.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:54 smithi114 ceph-mon[40867]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:56.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:56 smithi183 ceph-mon[33277]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:56.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:56 smithi114 ceph-mon[30324]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:56.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:56 smithi114 ceph-mon[40867]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:58.486 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:58 smithi114 ceph-mon[30324]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:58.486 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:58 smithi114 ceph-mon[40867]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:58.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:58 smithi183 ceph-mon[33277]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:16:59.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:59 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:16:59.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:59 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:59.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:16:59 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:59.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:59 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:16:59.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:59 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:59.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:16:59 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:59.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:59 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:16:59.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:59 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:16:59.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:16:59 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:00.702 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:00 smithi114 ceph-mon[30324]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:17:00.703 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:00 smithi114 ceph-mon[40867]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:17:00.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:00 smithi183 ceph-mon[33277]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:17:03.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:03 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:03.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:03 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:03.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:03 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:17:03.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:03 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:17:03.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:03 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:03.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:03 smithi114 ceph-mon[30324]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:17:03.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:03 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:03.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:03 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:03.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:03 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:17:03.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:03 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:17:03.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:03 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:03.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:03 smithi114 ceph-mon[40867]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:17:03.441 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:03 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:03.441 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:03 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:03.441 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:03 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:17:03.441 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:03 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:17:03.441 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:03 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:03.441 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:03 smithi183 ceph-mon[33277]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:17:04.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:04 smithi183 ceph-mon[33277]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:17:04.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:04 smithi183 ceph-mon[33277]: from='osd.1 [v2:172.21.15.114:6810/2744663863,v1:172.21.15.114:6811/2744663863]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T12:17:04.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:04 smithi114 ceph-mon[40867]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:17:04.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:04 smithi114 ceph-mon[40867]: from='osd.1 [v2:172.21.15.114:6810/2744663863,v1:172.21.15.114:6811/2744663863]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T12:17:04.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:04 smithi114 ceph-mon[30324]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:17:04.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:04 smithi114 ceph-mon[30324]: from='osd.1 [v2:172.21.15.114:6810/2744663863,v1:172.21.15.114:6811/2744663863]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T12:17:04.847 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 1 on host 'smithi114' 2024-09-21T12:17:05.412 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:05 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:17:05.412 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:05 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:05.412 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:05 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:05.412 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:05 smithi114 ceph-mon[40867]: from='osd.1 [v2:172.21.15.114:6810/2744663863,v1:172.21.15.114:6811/2744663863]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T12:17:05.412 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:05 smithi114 ceph-mon[40867]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T12:17:05.412 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:05 smithi114 ceph-mon[40867]: from='osd.1 [v2:172.21.15.114:6810/2744663863,v1:172.21.15.114:6811/2744663863]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-21T12:17:05.412 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:05 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T12:17:05.413 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:05 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:17:05.413 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:05 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:05.413 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:05 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:05.413 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:05 smithi114 ceph-mon[30324]: from='osd.1 [v2:172.21.15.114:6810/2744663863,v1:172.21.15.114:6811/2744663863]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T12:17:05.413 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:05 smithi114 ceph-mon[30324]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T12:17:05.413 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:05 smithi114 ceph-mon[30324]: from='osd.1 [v2:172.21.15.114:6810/2744663863,v1:172.21.15.114:6811/2744663863]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-21T12:17:05.413 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:05 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T12:17:05.414 DEBUG:teuthology.orchestra.run.smithi114:osd.1> sudo journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@osd.1.service 2024-09-21T12:17:05.416 INFO:tasks.cephadm:Deploying osd.2 on smithi114 with /dev/vg_nvme/lv_2... 2024-09-21T12:17:05.416 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T12:17:05.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:05 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:17:05.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:05 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:05.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:05 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:05.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:05 smithi183 ceph-mon[33277]: from='osd.1 [v2:172.21.15.114:6810/2744663863,v1:172.21.15.114:6811/2744663863]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T12:17:05.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:05 smithi183 ceph-mon[33277]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T12:17:05.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:05 smithi183 ceph-mon[33277]: from='osd.1 [v2:172.21.15.114:6810/2744663863,v1:172.21.15.114:6811/2744663863]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-21T12:17:05.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:05 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T12:17:06.151 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:17:06.512 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:06 smithi114 ceph-mon[30324]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:17:06.512 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:06 smithi114 ceph-mon[30324]: from='osd.1 [v2:172.21.15.114:6810/2744663863,v1:172.21.15.114:6811/2744663863]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-21T12:17:06.512 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:06 smithi114 ceph-mon[30324]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T12:17:06.512 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:06 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T12:17:06.512 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:06 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T12:17:06.513 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:06 smithi114 ceph-mon[40867]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:17:06.513 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:06 smithi114 ceph-mon[40867]: from='osd.1 [v2:172.21.15.114:6810/2744663863,v1:172.21.15.114:6811/2744663863]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-21T12:17:06.513 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:06 smithi114 ceph-mon[40867]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T12:17:06.513 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:06 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T12:17:06.513 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:06 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T12:17:06.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:06 smithi183 ceph-mon[33277]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:17:06.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:06 smithi183 ceph-mon[33277]: from='osd.1 [v2:172.21.15.114:6810/2744663863,v1:172.21.15.114:6811/2744663863]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-21T12:17:06.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:06 smithi183 ceph-mon[33277]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T12:17:06.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:06 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T12:17:06.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:06 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T12:17:06.775 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 21 12:17:06 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-1[50411]: 2024-09-21T12:17:06.571+0000 7f8466c22640 -1 osd.1 0 waiting for initial osdmap 2024-09-21T12:17:06.775 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 21 12:17:06 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-1[50411]: 2024-09-21T12:17:06.574+0000 7f845ea41640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T12:17:07.700 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:07 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:17:07.700 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:07 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:17:07.700 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:07 smithi114 ceph-mon[30324]: from='osd.1 [v2:172.21.15.114:6810/2744663863,v1:172.21.15.114:6811/2744663863]' entity='osd.1' 2024-09-21T12:17:07.700 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:07 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T12:17:07.700 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:07 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:17:07.700 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:07 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:17:07.701 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:07 smithi114 ceph-mon[40867]: from='osd.1 [v2:172.21.15.114:6810/2744663863,v1:172.21.15.114:6811/2744663863]' entity='osd.1' 2024-09-21T12:17:07.701 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:07 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T12:17:07.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:07 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:17:07.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:07 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:17:07.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:07 smithi183 ceph-mon[33277]: from='osd.1 [v2:172.21.15.114:6810/2744663863,v1:172.21.15.114:6811/2744663863]' entity='osd.1' 2024-09-21T12:17:07.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:07 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T12:17:08.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:08 smithi114 ceph-mon[30324]: osd.1 [v2:172.21.15.114:6810/2744663863,v1:172.21.15.114:6811/2744663863] boot 2024-09-21T12:17:08.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:08 smithi114 ceph-mon[30324]: osdmap e13: 2 total, 2 up, 2 in 2024-09-21T12:17:08.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:08 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T12:17:08.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:08 smithi114 ceph-mon[30324]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:17:08.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:08 smithi114 ceph-mon[40867]: osd.1 [v2:172.21.15.114:6810/2744663863,v1:172.21.15.114:6811/2744663863] boot 2024-09-21T12:17:08.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:08 smithi114 ceph-mon[40867]: osdmap e13: 2 total, 2 up, 2 in 2024-09-21T12:17:08.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:08 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T12:17:08.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:08 smithi114 ceph-mon[40867]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:17:08.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:08 smithi183 ceph-mon[33277]: osd.1 [v2:172.21.15.114:6810/2744663863,v1:172.21.15.114:6811/2744663863] boot 2024-09-21T12:17:08.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:08 smithi183 ceph-mon[33277]: osdmap e13: 2 total, 2 up, 2 in 2024-09-21T12:17:08.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:08 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T12:17:08.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:08 smithi183 ceph-mon[33277]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:17:09.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:09 smithi114 ceph-mon[40867]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T12:17:09.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:09 smithi114 ceph-mon[40867]: Detected new or changed devices on smithi114 2024-09-21T12:17:09.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:09.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:09.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T12:17:09.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T12:17:09.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:09 smithi114 ceph-mon[40867]: Adjusting osd_memory_target on smithi114 to 8032M 2024-09-21T12:17:09.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:09.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:17:09.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:17:09.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:09.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:09 smithi114 ceph-mon[30324]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T12:17:09.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:09 smithi114 ceph-mon[30324]: Detected new or changed devices on smithi114 2024-09-21T12:17:09.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:09.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:09.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T12:17:09.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T12:17:09.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:09 smithi114 ceph-mon[30324]: Adjusting osd_memory_target on smithi114 to 8032M 2024-09-21T12:17:09.777 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:09.777 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:17:09.777 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:17:09.777 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:09.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:09 smithi183 ceph-mon[33277]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T12:17:09.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:09 smithi183 ceph-mon[33277]: Detected new or changed devices on smithi114 2024-09-21T12:17:09.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:09.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:09.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T12:17:09.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T12:17:09.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:09 smithi183 ceph-mon[33277]: Adjusting osd_memory_target on smithi114 to 8032M 2024-09-21T12:17:09.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:09.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:17:09.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:17:09.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:10.586 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:17:10.617 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi114:vg_nvme/lv_2 2024-09-21T12:17:10.864 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:17:10.984 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:10 smithi114 ceph-mon[30324]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:10.984 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:10 smithi114 ceph-mon[40867]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:10.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:10 smithi183 ceph-mon[33277]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:11.884 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:11 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T12:17:11.884 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:11 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T12:17:11.884 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:11 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:17:11.884 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:11 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T12:17:11.885 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:11 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T12:17:11.885 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:11 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:17:11.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:11 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T12:17:11.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:11 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T12:17:11.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:11 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:17:12.893 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:12 smithi114 ceph-mon[40867]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:17:12.893 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:12 smithi114 ceph-mon[40867]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:12.894 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:12 smithi114 ceph-mon[30324]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:17:12.894 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:12 smithi114 ceph-mon[30324]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:12.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:12 smithi183 ceph-mon[33277]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:17:12.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:12 smithi183 ceph-mon[33277]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:14.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:13 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3810177886' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6a63581f-7603-49ae-9ab3-b37e7c91ab91"} : dispatch 2024-09-21T12:17:14.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:13 smithi183 ceph-mon[33277]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6a63581f-7603-49ae-9ab3-b37e7c91ab91"} : dispatch 2024-09-21T12:17:14.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:13 smithi183 ceph-mon[33277]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6a63581f-7603-49ae-9ab3-b37e7c91ab91"}]': finished 2024-09-21T12:17:14.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:13 smithi183 ceph-mon[33277]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T12:17:14.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:13 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T12:17:14.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:13 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3810177886' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6a63581f-7603-49ae-9ab3-b37e7c91ab91"} : dispatch 2024-09-21T12:17:14.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:13 smithi114 ceph-mon[30324]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6a63581f-7603-49ae-9ab3-b37e7c91ab91"} : dispatch 2024-09-21T12:17:14.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:13 smithi114 ceph-mon[30324]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6a63581f-7603-49ae-9ab3-b37e7c91ab91"}]': finished 2024-09-21T12:17:14.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:13 smithi114 ceph-mon[30324]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T12:17:14.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:13 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T12:17:14.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:13 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3810177886' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6a63581f-7603-49ae-9ab3-b37e7c91ab91"} : dispatch 2024-09-21T12:17:14.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:13 smithi114 ceph-mon[40867]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6a63581f-7603-49ae-9ab3-b37e7c91ab91"} : dispatch 2024-09-21T12:17:14.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:13 smithi114 ceph-mon[40867]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6a63581f-7603-49ae-9ab3-b37e7c91ab91"}]': finished 2024-09-21T12:17:14.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:13 smithi114 ceph-mon[40867]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T12:17:14.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:13 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T12:17:15.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:14 smithi183 ceph-mon[33277]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:15.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:14 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/500470391' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T12:17:15.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:14 smithi114 ceph-mon[30324]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:15.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:14 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/500470391' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T12:17:15.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:14 smithi114 ceph-mon[40867]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:15.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:14 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/500470391' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T12:17:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:16 smithi183 ceph-mon[33277]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:16.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:16 smithi114 ceph-mon[30324]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:16.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:16 smithi114 ceph-mon[40867]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:18.378 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:18 smithi114 ceph-mon[30324]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:18.378 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:18 smithi114 ceph-mon[40867]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:18.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:18 smithi183 ceph-mon[33277]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:20.221 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:20 smithi114 ceph-mon[30324]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:20.221 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:20 smithi114 ceph-mon[40867]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:20 smithi183 ceph-mon[33277]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:21.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:21 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T12:17:21.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:21 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:17:21.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:21 smithi114 ceph-mon[30324]: Deploying daemon osd.2 on smithi114 2024-09-21T12:17:21.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:21 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T12:17:21.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:21 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:17:21.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:21 smithi114 ceph-mon[40867]: Deploying daemon osd.2 on smithi114 2024-09-21T12:17:21.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:21 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T12:17:21.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:21 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:17:21.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:21 smithi183 ceph-mon[33277]: Deploying daemon osd.2 on smithi114 2024-09-21T12:17:22.306 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:22 smithi114 ceph-mon[30324]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:22.306 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:22 smithi114 ceph-mon[40867]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:22.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:22 smithi183 ceph-mon[33277]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:24.468 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:24 smithi114 ceph-mon[30324]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:24.468 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:24 smithi114 ceph-mon[40867]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:24.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:24 smithi183 ceph-mon[33277]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:26.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:26 smithi183 ceph-mon[33277]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:26.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:26 smithi114 ceph-mon[30324]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:26.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:26 smithi114 ceph-mon[40867]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:28.404 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:28 smithi114 ceph-mon[30324]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:28.405 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:28 smithi114 ceph-mon[40867]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:28.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:28 smithi183 ceph-mon[33277]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:29.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:29 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:17:29.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:29 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:29.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:29 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:29.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:29 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:17:29.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:29 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:29.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:29 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:29 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:17:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:29 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:29 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:30.566 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:30 smithi114 ceph-mon[30324]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:30.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:30 smithi114 ceph-mon[40867]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:30.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:30 smithi183 ceph-mon[33277]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:33.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:32 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:33.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:32 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:33.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:32 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:17:33.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:32 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:17:33.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:32 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:33.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:32 smithi183 ceph-mon[33277]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:33.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:32 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:33.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:32 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:33.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:32 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:17:33.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:32 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:17:33.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:32 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:33.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:32 smithi114 ceph-mon[30324]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:33.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:32 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:33.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:32 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:33.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:32 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:17:33.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:32 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:17:33.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:32 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:33.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:32 smithi114 ceph-mon[40867]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:34.443 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:34 smithi114 ceph-mon[30324]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:34.443 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:34 smithi114 ceph-mon[40867]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:34.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:34 smithi183 ceph-mon[33277]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:34.803 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 2 on host 'smithi114' 2024-09-21T12:17:35.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:35 smithi114 ceph-mon[30324]: from='osd.2 [v2:172.21.15.114:6818/3740045173,v1:172.21.15.114:6819/3740045173]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T12:17:35.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:35 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:17:35.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:35 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:35.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:35 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:35.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:35 smithi114 ceph-mon[40867]: from='osd.2 [v2:172.21.15.114:6818/3740045173,v1:172.21.15.114:6819/3740045173]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T12:17:35.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:35 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:17:35.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:35 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:35.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:35 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:35.403 DEBUG:teuthology.orchestra.run.smithi114:osd.2> sudo journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@osd.2.service 2024-09-21T12:17:35.405 INFO:tasks.cephadm:Deploying osd.3 on smithi114 with /dev/vg_nvme/lv_1... 2024-09-21T12:17:35.405 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T12:17:35.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:35 smithi183 ceph-mon[33277]: from='osd.2 [v2:172.21.15.114:6818/3740045173,v1:172.21.15.114:6819/3740045173]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T12:17:35.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:35 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:17:35.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:35 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:35.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:35 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:36.141 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:17:36.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:36 smithi183 ceph-mon[33277]: from='osd.2 [v2:172.21.15.114:6818/3740045173,v1:172.21.15.114:6819/3740045173]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T12:17:36.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:36 smithi183 ceph-mon[33277]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T12:17:36.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:36 smithi183 ceph-mon[33277]: from='osd.2 [v2:172.21.15.114:6818/3740045173,v1:172.21.15.114:6819/3740045173]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-21T12:17:36.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:36 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T12:17:36.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:36 smithi183 ceph-mon[33277]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:36.497 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:36 smithi114 ceph-mon[30324]: from='osd.2 [v2:172.21.15.114:6818/3740045173,v1:172.21.15.114:6819/3740045173]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T12:17:36.497 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:36 smithi114 ceph-mon[30324]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T12:17:36.497 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:36 smithi114 ceph-mon[30324]: from='osd.2 [v2:172.21.15.114:6818/3740045173,v1:172.21.15.114:6819/3740045173]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-21T12:17:36.497 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:36 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T12:17:36.497 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:36 smithi114 ceph-mon[30324]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:36.497 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:36 smithi114 ceph-mon[40867]: from='osd.2 [v2:172.21.15.114:6818/3740045173,v1:172.21.15.114:6819/3740045173]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T12:17:36.497 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:36 smithi114 ceph-mon[40867]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T12:17:36.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:36 smithi114 ceph-mon[40867]: from='osd.2 [v2:172.21.15.114:6818/3740045173,v1:172.21.15.114:6819/3740045173]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-21T12:17:36.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:36 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T12:17:36.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:36 smithi114 ceph-mon[40867]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:36.765 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:17:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2[56527]: 2024-09-21T12:17:36.572+0000 7fc6b5050640 -1 osd.2 0 waiting for initial osdmap 2024-09-21T12:17:36.765 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:17:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2[56527]: 2024-09-21T12:17:36.574+0000 7fc6ace6f640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T12:17:37.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:37 smithi114 ceph-mon[30324]: from='osd.2 [v2:172.21.15.114:6818/3740045173,v1:172.21.15.114:6819/3740045173]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-21T12:17:37.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:37 smithi114 ceph-mon[30324]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T12:17:37.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:37 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T12:17:37.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:37 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T12:17:37.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:37 smithi114 ceph-mon[30324]: from='osd.2 [v2:172.21.15.114:6818/3740045173,v1:172.21.15.114:6819/3740045173]' entity='osd.2' 2024-09-21T12:17:37.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:37 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T12:17:37.302 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:37 smithi114 ceph-mon[40867]: from='osd.2 [v2:172.21.15.114:6818/3740045173,v1:172.21.15.114:6819/3740045173]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-21T12:17:37.302 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:37 smithi114 ceph-mon[40867]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T12:17:37.303 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:37 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T12:17:37.303 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:37 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T12:17:37.303 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:37 smithi114 ceph-mon[40867]: from='osd.2 [v2:172.21.15.114:6818/3740045173,v1:172.21.15.114:6819/3740045173]' entity='osd.2' 2024-09-21T12:17:37.303 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:37 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T12:17:37.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:37 smithi183 ceph-mon[33277]: from='osd.2 [v2:172.21.15.114:6818/3740045173,v1:172.21.15.114:6819/3740045173]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-21T12:17:37.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:37 smithi183 ceph-mon[33277]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T12:17:37.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:37 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T12:17:37.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:37 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T12:17:37.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:37 smithi183 ceph-mon[33277]: from='osd.2 [v2:172.21.15.114:6818/3740045173,v1:172.21.15.114:6819/3740045173]' entity='osd.2' 2024-09-21T12:17:37.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:37 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T12:17:38.446 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:38 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:17:38.446 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:38 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:17:38.446 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:38 smithi114 ceph-mon[40867]: osd.2 [v2:172.21.15.114:6818/3740045173,v1:172.21.15.114:6819/3740045173] boot 2024-09-21T12:17:38.446 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:38 smithi114 ceph-mon[40867]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T12:17:38.446 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:38 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T12:17:38.446 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:38 smithi114 ceph-mon[40867]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:38.446 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:38 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:17:38.446 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:38 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:17:38.446 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:38 smithi114 ceph-mon[30324]: osd.2 [v2:172.21.15.114:6818/3740045173,v1:172.21.15.114:6819/3740045173] boot 2024-09-21T12:17:38.447 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:38 smithi114 ceph-mon[30324]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T12:17:38.447 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:38 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T12:17:38.447 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:38 smithi114 ceph-mon[30324]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:38.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:38 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:17:38.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:38 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:17:38.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:38 smithi183 ceph-mon[33277]: osd.2 [v2:172.21.15.114:6818/3740045173,v1:172.21.15.114:6819/3740045173] boot 2024-09-21T12:17:38.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:38 smithi183 ceph-mon[33277]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T12:17:38.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:38 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T12:17:38.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:38 smithi183 ceph-mon[33277]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:17:39.434 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:39 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:17:39.434 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:39 smithi114 ceph-mon[40867]: Detected new or changed devices on smithi114 2024-09-21T12:17:39.434 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:39 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:39.434 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:39 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:39.434 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:39 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T12:17:39.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:39 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T12:17:39.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:39 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T12:17:39.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:39 smithi114 ceph-mon[40867]: Adjusting osd_memory_target on smithi114 to 5354M 2024-09-21T12:17:39.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:39 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:39.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:39 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:17:39.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:39 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:17:39.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:39 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:39.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:39 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:17:39.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:39 smithi114 ceph-mon[30324]: Detected new or changed devices on smithi114 2024-09-21T12:17:39.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:39 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:39.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:39 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:39.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:39 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T12:17:39.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:39 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T12:17:39.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:39 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T12:17:39.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:39 smithi114 ceph-mon[30324]: Adjusting osd_memory_target on smithi114 to 5354M 2024-09-21T12:17:39.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:39 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:39.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:39 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:17:39.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:39 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:17:39.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:39 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:39.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:39 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:17:39.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:39 smithi183 ceph-mon[33277]: Detected new or changed devices on smithi114 2024-09-21T12:17:39.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:39 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:39.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:39 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:39.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:39 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T12:17:39.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:39 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T12:17:39.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:39 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T12:17:39.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:39 smithi183 ceph-mon[33277]: Adjusting osd_memory_target on smithi114 to 5354M 2024-09-21T12:17:39.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:39 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:39.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:39 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:17:39.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:39 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:17:39.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:39 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:40.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:40 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T12:17:40.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:40 smithi114 ceph-mon[40867]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T12:17:40.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:40 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:17:40.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:40 smithi114 ceph-mon[40867]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:17:40.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:40 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T12:17:40.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:40 smithi114 ceph-mon[30324]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T12:17:40.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:40 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:17:40.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:40 smithi114 ceph-mon[30324]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:17:40.441 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:40 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T12:17:40.441 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:40 smithi183 ceph-mon[33277]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T12:17:40.441 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:40 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:17:40.441 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:40 smithi183 ceph-mon[33277]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:17:40.441 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:40 smithi183 sudo[35265]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T12:17:40.441 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:40 smithi183 sudo[35265]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T12:17:40.441 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:40 smithi183 sudo[35265]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T12:17:40.441 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:40 smithi183 sudo[35265]: pam_unix(sudo:session): session closed for user root 2024-09-21T12:17:40.502 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60775]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T12:17:40.502 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60775]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T12:17:40.502 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60775]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T12:17:40.503 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60763]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T12:17:40.503 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60763]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T12:17:40.503 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60763]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T12:17:40.503 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60763]: pam_unix(sudo:session): session closed for user root 2024-09-21T12:17:40.503 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60766]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T12:17:40.503 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60766]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T12:17:40.503 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60766]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T12:17:40.503 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60766]: pam_unix(sudo:session): session closed for user root 2024-09-21T12:17:40.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60769]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T12:17:40.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60769]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T12:17:40.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60769]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T12:17:40.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60769]: pam_unix(sudo:session): session closed for user root 2024-09-21T12:17:40.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60772]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T12:17:40.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60772]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T12:17:40.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60772]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T12:17:40.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60772]: pam_unix(sudo:session): session closed for user root 2024-09-21T12:17:40.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60757]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T12:17:40.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60757]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T12:17:40.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60757]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T12:17:40.504 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60757]: pam_unix(sudo:session): session closed for user root 2024-09-21T12:17:40.505 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60760]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T12:17:40.505 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60760]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T12:17:40.505 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60760]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T12:17:40.505 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60760]: pam_unix(sudo:session): session closed for user root 2024-09-21T12:17:40.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60742]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T12:17:40.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60742]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T12:17:40.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60742]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T12:17:40.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60742]: pam_unix(sudo:session): session closed for user root 2024-09-21T12:17:40.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60754]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T12:17:40.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60754]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T12:17:40.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60754]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T12:17:40.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60754]: pam_unix(sudo:session): session closed for user root 2024-09-21T12:17:40.731 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:17:40.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:40 smithi183 sudo[35268]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T12:17:40.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:40 smithi183 sudo[35268]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T12:17:40.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:40 smithi183 sudo[35268]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T12:17:40.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:40 smithi183 sudo[35268]: pam_unix(sudo:session): session closed for user root 2024-09-21T12:17:40.754 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60775]: pam_unix(sudo:session): session closed for user root 2024-09-21T12:17:40.755 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60778]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T12:17:40.755 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60778]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T12:17:40.755 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60778]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T12:17:40.755 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:40 smithi114 sudo[60778]: pam_unix(sudo:session): session closed for user root 2024-09-21T12:17:40.756 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi114:vg_nvme/lv_1 2024-09-21T12:17:41.002 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:17:41.379 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[40867]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T12:17:41.379 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T12:17:41.379 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[40867]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T12:17:41.379 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:41.379 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[40867]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T12:17:41.379 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[40867]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T12:17:41.379 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T12:17:41.379 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:17:41.379 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:17:41.380 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[40867]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T12:17:41.380 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T12:17:41.380 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:17:41.380 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:17:41.380 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[40867]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T12:17:41.380 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:41.380 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[40867]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T12:17:41.380 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T12:17:41.380 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:17:41.380 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:17:41.380 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[40867]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T12:17:41.380 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[30324]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T12:17:41.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T12:17:41.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[30324]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T12:17:41.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:41.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[30324]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T12:17:41.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[30324]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T12:17:41.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T12:17:41.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:17:41.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:17:41.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[30324]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T12:17:41.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T12:17:41.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:17:41.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:17:41.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[30324]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T12:17:41.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:41.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[30324]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T12:17:41.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T12:17:41.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:17:41.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:17:41.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:41 smithi114 ceph-mon[30324]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T12:17:41.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:41 smithi183 ceph-mon[33277]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T12:17:41.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:41 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T12:17:41.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:41 smithi183 ceph-mon[33277]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T12:17:41.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:41 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:41.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:41 smithi183 ceph-mon[33277]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T12:17:41.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:41 smithi183 ceph-mon[33277]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T12:17:41.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:41 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T12:17:41.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:41 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:17:41.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:41 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:17:41.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:41 smithi183 ceph-mon[33277]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T12:17:41.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:41 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T12:17:41.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:41 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:17:41.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:41 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:17:41.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:41 smithi183 ceph-mon[33277]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T12:17:41.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:41 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:41.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:41 smithi183 ceph-mon[33277]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T12:17:41.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:41 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T12:17:41.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:41 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:17:41.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:41 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:17:41.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:41 smithi183 ceph-mon[33277]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T12:17:42.285 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:42 smithi114 ceph-mon[40867]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T12:17:42.285 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:42 smithi114 ceph-mon[40867]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:17:42.285 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:42 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T12:17:42.285 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:42 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T12:17:42.285 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:42 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:17:42.285 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:42 smithi114 ceph-mon[40867]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:17:42.285 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:42 smithi114 ceph-mon[30324]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T12:17:42.286 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:42 smithi114 ceph-mon[30324]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:17:42.286 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:42 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T12:17:42.286 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:42 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T12:17:42.286 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:42 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:17:42.286 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:42 smithi114 ceph-mon[30324]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:17:42.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:42 smithi183 ceph-mon[33277]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T12:17:42.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:42 smithi183 ceph-mon[33277]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:17:42.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:42 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T12:17:42.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:42 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T12:17:42.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:42 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:17:42.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:42 smithi183 ceph-mon[33277]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:17:43.326 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:43 smithi114 ceph-mon[30324]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T12:17:43.326 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:43 smithi114 ceph-mon[30324]: Cluster is now healthy 2024-09-21T12:17:43.326 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:43 smithi114 ceph-mon[30324]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-21T12:17:43.326 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:43 smithi114 ceph-mon[40867]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T12:17:43.326 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:43 smithi114 ceph-mon[40867]: Cluster is now healthy 2024-09-21T12:17:43.326 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:43 smithi114 ceph-mon[40867]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-21T12:17:43.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:43 smithi183 ceph-mon[33277]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T12:17:43.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:43 smithi183 ceph-mon[33277]: Cluster is now healthy 2024-09-21T12:17:43.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:43 smithi183 ceph-mon[33277]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-21T12:17:44.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:44 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/546506319' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0ad68cdb-ae5c-4090-8a8d-b67c1177ae42"} : dispatch 2024-09-21T12:17:44.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:44 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/546506319' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0ad68cdb-ae5c-4090-8a8d-b67c1177ae42"}]': finished 2024-09-21T12:17:44.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:44 smithi183 ceph-mon[33277]: osdmap e22: 4 total, 3 up, 4 in 2024-09-21T12:17:44.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:44 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T12:17:44.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:44 smithi183 ceph-mon[33277]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:17:44.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:44 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/546506319' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0ad68cdb-ae5c-4090-8a8d-b67c1177ae42"} : dispatch 2024-09-21T12:17:44.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:44 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/546506319' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0ad68cdb-ae5c-4090-8a8d-b67c1177ae42"}]': finished 2024-09-21T12:17:44.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:44 smithi114 ceph-mon[30324]: osdmap e22: 4 total, 3 up, 4 in 2024-09-21T12:17:44.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:44 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T12:17:44.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:44 smithi114 ceph-mon[30324]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:17:44.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:44 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/546506319' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0ad68cdb-ae5c-4090-8a8d-b67c1177ae42"} : dispatch 2024-09-21T12:17:44.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:44 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/546506319' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0ad68cdb-ae5c-4090-8a8d-b67c1177ae42"}]': finished 2024-09-21T12:17:44.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:44 smithi114 ceph-mon[40867]: osdmap e22: 4 total, 3 up, 4 in 2024-09-21T12:17:44.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:44 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T12:17:44.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:44 smithi114 ceph-mon[40867]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:17:45.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:45 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1523251331' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T12:17:45.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:45 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1523251331' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T12:17:45.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:45 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1523251331' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T12:17:46.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:46 smithi183 ceph-mon[33277]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:17:46.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:46 smithi114 ceph-mon[30324]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:17:46.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:46 smithi114 ceph-mon[40867]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:17:48.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:48 smithi114 ceph-mon[30324]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:17:48.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:48 smithi114 ceph-mon[40867]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:17:48.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:48 smithi183 ceph-mon[33277]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:17:50.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:50 smithi183 ceph-mon[33277]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:17:50.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:50 smithi114 ceph-mon[30324]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:17:50.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:50 smithi114 ceph-mon[40867]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:17:51.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:51 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T12:17:51.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:51 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:17:51.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:51 smithi114 ceph-mon[40867]: Deploying daemon osd.3 on smithi114 2024-09-21T12:17:51.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:51 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T12:17:51.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:51 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:17:51.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:51 smithi114 ceph-mon[30324]: Deploying daemon osd.3 on smithi114 2024-09-21T12:17:51.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:51 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T12:17:51.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:51 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:17:51.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:51 smithi183 ceph-mon[33277]: Deploying daemon osd.3 on smithi114 2024-09-21T12:17:52.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:52 smithi114 ceph-mon[40867]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:17:52.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:52 smithi114 ceph-mon[30324]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:17:52.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:52 smithi183 ceph-mon[33277]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:17:54.464 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:54 smithi114 ceph-mon[30324]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:17:54.464 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:54 smithi114 ceph-mon[40867]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:17:54.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:54 smithi183 ceph-mon[33277]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:17:55.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:55 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:17:55.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:55 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:17:55.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:55 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:17:55.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:55 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:17:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:55 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:17:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:55 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:17:56.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:56 smithi183 ceph-mon[33277]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T12:17:56.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:56 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T12:17:56.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:56 smithi183 ceph-mon[33277]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:17:56.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:56 smithi114 ceph-mon[40867]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T12:17:56.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:56 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T12:17:56.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:56 smithi114 ceph-mon[40867]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:17:56.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:56 smithi114 ceph-mon[30324]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T12:17:56.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:56 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T12:17:56.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:56 smithi114 ceph-mon[30324]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:17:58.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:58 smithi183 ceph-mon[33277]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:17:58.501 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:58 smithi114 ceph-mon[30324]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:17:58.501 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:58 smithi114 ceph-mon[40867]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:17:59.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:59 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:17:59.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:59 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:59.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:17:59 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:59 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:17:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:59 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:17:59 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:59 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:17:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:59 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:17:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:17:59 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:00.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:00 smithi183 ceph-mon[33277]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:18:00.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:00 smithi114 ceph-mon[30324]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:18:00.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:00 smithi114 ceph-mon[40867]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:18:03.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:03 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:03.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:03 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:03.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:03 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:18:03.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:03 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:18:03.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:03 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:03.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:03 smithi114 ceph-mon[30324]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:18:03.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:03 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:03.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:03 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:03.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:03 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:18:03.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:03 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:18:03.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:03 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:03.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:03 smithi114 ceph-mon[40867]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:18:03.460 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:03 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:03.460 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:03 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:03.460 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:03 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:18:03.460 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:03 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:18:03.460 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:03 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:03.460 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:03 smithi183 ceph-mon[33277]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:18:04.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:04 smithi114 ceph-mon[30324]: from='osd.3 [v2:172.21.15.114:6826/1312590318,v1:172.21.15.114:6827/1312590318]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T12:18:04.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:04 smithi114 ceph-mon[40867]: from='osd.3 [v2:172.21.15.114:6826/1312590318,v1:172.21.15.114:6827/1312590318]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T12:18:04.414 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:04 smithi183 ceph-mon[33277]: from='osd.3 [v2:172.21.15.114:6826/1312590318,v1:172.21.15.114:6827/1312590318]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T12:18:04.607 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 3 on host 'smithi114' 2024-09-21T12:18:05.252 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:05 smithi114 ceph-mon[40867]: from='osd.3 [v2:172.21.15.114:6826/1312590318,v1:172.21.15.114:6827/1312590318]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T12:18:05.252 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:05 smithi114 ceph-mon[40867]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T12:18:05.252 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:05 smithi114 ceph-mon[40867]: from='osd.3 [v2:172.21.15.114:6826/1312590318,v1:172.21.15.114:6827/1312590318]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-21T12:18:05.252 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:05 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T12:18:05.252 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:05 smithi114 ceph-mon[40867]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:18:05.252 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:05 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:18:05.252 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:05 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:05.252 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:05 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:05.252 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:05 smithi114 ceph-mon[30324]: from='osd.3 [v2:172.21.15.114:6826/1312590318,v1:172.21.15.114:6827/1312590318]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T12:18:05.252 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:05 smithi114 ceph-mon[30324]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T12:18:05.252 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:05 smithi114 ceph-mon[30324]: from='osd.3 [v2:172.21.15.114:6826/1312590318,v1:172.21.15.114:6827/1312590318]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-21T12:18:05.253 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:05 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T12:18:05.253 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:05 smithi114 ceph-mon[30324]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:18:05.253 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:05 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:18:05.253 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:05 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:05.253 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:05 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:05.280 DEBUG:teuthology.orchestra.run.smithi114:osd.3> sudo journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@osd.3.service 2024-09-21T12:18:05.283 INFO:tasks.cephadm:Deploying osd.4 on smithi183 with /dev/vg_nvme/lv_4... 2024-09-21T12:18:05.283 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T12:18:05.306 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:05 smithi183 ceph-mon[33277]: from='osd.3 [v2:172.21.15.114:6826/1312590318,v1:172.21.15.114:6827/1312590318]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T12:18:05.306 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:05 smithi183 ceph-mon[33277]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T12:18:05.307 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:05 smithi183 ceph-mon[33277]: from='osd.3 [v2:172.21.15.114:6826/1312590318,v1:172.21.15.114:6827/1312590318]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-21T12:18:05.307 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:05 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T12:18:05.307 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:05 smithi183 ceph-mon[33277]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:18:05.307 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:05 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:18:05.307 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:05 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:05.307 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:05 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:05.523 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:18:06.166 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:06 smithi114 ceph-mon[40867]: from='osd.3 [v2:172.21.15.114:6826/1312590318,v1:172.21.15.114:6827/1312590318]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-21T12:18:06.166 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:06 smithi114 ceph-mon[40867]: osdmap e25: 4 total, 3 up, 4 in 2024-09-21T12:18:06.166 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:06 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T12:18:06.166 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:06 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T12:18:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:06 smithi114 ceph-mon[30324]: from='osd.3 [v2:172.21.15.114:6826/1312590318,v1:172.21.15.114:6827/1312590318]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-21T12:18:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:06 smithi114 ceph-mon[30324]: osdmap e25: 4 total, 3 up, 4 in 2024-09-21T12:18:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:06 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T12:18:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:06 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T12:18:06.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:06 smithi183 ceph-mon[33277]: from='osd.3 [v2:172.21.15.114:6826/1312590318,v1:172.21.15.114:6827/1312590318]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-21T12:18:06.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:06 smithi183 ceph-mon[33277]: osdmap e25: 4 total, 3 up, 4 in 2024-09-21T12:18:06.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:06 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T12:18:06.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:06 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T12:18:07.209 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:07 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:18:07.209 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:07 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:18:07.209 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:07 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T12:18:07.209 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:07 smithi183 ceph-mon[33277]: osd.3 [v2:172.21.15.114:6826/1312590318,v1:172.21.15.114:6827/1312590318] boot 2024-09-21T12:18:07.210 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:07 smithi183 ceph-mon[33277]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T12:18:07.210 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:07 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T12:18:07.210 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:07 smithi183 ceph-mon[33277]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:18:07.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:07 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:18:07.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:07 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:18:07.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:07 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T12:18:07.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:07 smithi114 ceph-mon[40867]: osd.3 [v2:172.21.15.114:6826/1312590318,v1:172.21.15.114:6827/1312590318] boot 2024-09-21T12:18:07.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:07 smithi114 ceph-mon[40867]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T12:18:07.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:07 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T12:18:07.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:07 smithi114 ceph-mon[40867]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:18:07.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:07 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:18:07.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:07 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:18:07.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:07 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T12:18:07.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:07 smithi114 ceph-mon[30324]: osd.3 [v2:172.21.15.114:6826/1312590318,v1:172.21.15.114:6827/1312590318] boot 2024-09-21T12:18:07.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:07 smithi114 ceph-mon[30324]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T12:18:07.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:07 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T12:18:07.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:07 smithi114 ceph-mon[30324]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:18:08.184 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:08 smithi183 ceph-mon[33277]: osdmap e27: 4 total, 4 up, 4 in 2024-09-21T12:18:08.187 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:18:08.214 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi183:vg_nvme/lv_4 2024-09-21T12:18:08.498 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:18:08.507 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:08 smithi114 ceph-mon[30324]: osdmap e27: 4 total, 4 up, 4 in 2024-09-21T12:18:08.507 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:08 smithi114 ceph-mon[40867]: osdmap e27: 4 total, 4 up, 4 in 2024-09-21T12:18:09.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:09 smithi183 ceph-mon[33277]: Detected new or changed devices on smithi114 2024-09-21T12:18:09.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:09.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:09.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T12:18:09.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T12:18:09.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T12:18:09.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T12:18:09.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:09 smithi183 ceph-mon[33277]: Adjusting osd_memory_target on smithi114 to 4016M 2024-09-21T12:18:09.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:09 smithi183 ceph-mon[33277]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:09.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:09.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:18:09.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:18:09.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:09.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:09 smithi114 ceph-mon[30324]: Detected new or changed devices on smithi114 2024-09-21T12:18:09.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:09.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:09.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T12:18:09.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T12:18:09.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T12:18:09.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T12:18:09.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:09 smithi114 ceph-mon[30324]: Adjusting osd_memory_target on smithi114 to 4016M 2024-09-21T12:18:09.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:09 smithi114 ceph-mon[30324]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:09.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:09.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:18:09.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:18:09.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:09.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:09 smithi114 ceph-mon[40867]: Detected new or changed devices on smithi114 2024-09-21T12:18:09.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:09.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:09.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T12:18:09.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T12:18:09.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T12:18:09.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T12:18:09.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:09 smithi114 ceph-mon[40867]: Adjusting osd_memory_target on smithi114 to 4016M 2024-09-21T12:18:09.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:09 smithi114 ceph-mon[40867]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:09.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:09.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:18:09.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:18:09.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:10.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:10 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T12:18:10.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:10 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T12:18:10.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:10 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:18:10.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:10 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T12:18:10.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:10 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T12:18:10.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:10 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:18:10.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:10 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T12:18:10.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:10 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T12:18:10.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:10 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:18:11.216 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:11 smithi183 ceph-mon[33277]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:18:11.216 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:11 smithi183 ceph-mon[33277]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:11.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:11 smithi114 ceph-mon[30324]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:18:11.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:11 smithi114 ceph-mon[30324]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:11.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:11 smithi114 ceph-mon[40867]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:18:11.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:11 smithi114 ceph-mon[40867]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:12 smithi183 ceph-mon[33277]: from='client.? 172.21.15.183:0/1714648798' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "26bb8a84-e41d-4425-8491-2dea9764b7cf"} : dispatch 2024-09-21T12:18:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:12 smithi183 ceph-mon[33277]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "26bb8a84-e41d-4425-8491-2dea9764b7cf"} : dispatch 2024-09-21T12:18:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:12 smithi183 ceph-mon[33277]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "26bb8a84-e41d-4425-8491-2dea9764b7cf"}]': finished 2024-09-21T12:18:12.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:12 smithi183 ceph-mon[33277]: osdmap e28: 5 total, 4 up, 5 in 2024-09-21T12:18:12.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:12 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T12:18:12.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:12 smithi114 ceph-mon[30324]: from='client.? 172.21.15.183:0/1714648798' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "26bb8a84-e41d-4425-8491-2dea9764b7cf"} : dispatch 2024-09-21T12:18:12.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:12 smithi114 ceph-mon[30324]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "26bb8a84-e41d-4425-8491-2dea9764b7cf"} : dispatch 2024-09-21T12:18:12.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:12 smithi114 ceph-mon[30324]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "26bb8a84-e41d-4425-8491-2dea9764b7cf"}]': finished 2024-09-21T12:18:12.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:12 smithi114 ceph-mon[30324]: osdmap e28: 5 total, 4 up, 5 in 2024-09-21T12:18:12.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:12 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T12:18:12.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:12 smithi114 ceph-mon[40867]: from='client.? 172.21.15.183:0/1714648798' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "26bb8a84-e41d-4425-8491-2dea9764b7cf"} : dispatch 2024-09-21T12:18:12.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:12 smithi114 ceph-mon[40867]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "26bb8a84-e41d-4425-8491-2dea9764b7cf"} : dispatch 2024-09-21T12:18:12.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:12 smithi114 ceph-mon[40867]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "26bb8a84-e41d-4425-8491-2dea9764b7cf"}]': finished 2024-09-21T12:18:12.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:12 smithi114 ceph-mon[40867]: osdmap e28: 5 total, 4 up, 5 in 2024-09-21T12:18:12.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:12 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T12:18:13.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:13 smithi183 ceph-mon[33277]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:13.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:13 smithi183 ceph-mon[33277]: from='client.? 172.21.15.183:0/3856655315' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T12:18:13.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:13 smithi114 ceph-mon[30324]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:13.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:13 smithi114 ceph-mon[30324]: from='client.? 172.21.15.183:0/3856655315' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T12:18:13.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:13 smithi114 ceph-mon[40867]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:13.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:13 smithi114 ceph-mon[40867]: from='client.? 172.21.15.183:0/3856655315' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T12:18:15.352 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:15 smithi183 ceph-mon[33277]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:15.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:15 smithi114 ceph-mon[40867]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:15.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:15 smithi114 ceph-mon[30324]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:16.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:16 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:18:16.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:16 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:18:16.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:16 smithi183 ceph-mon[33277]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T12:18:16.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:16 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T12:18:16.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:16 smithi183 ceph-mon[33277]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:16 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:18:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:16 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:18:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:16 smithi114 ceph-mon[30324]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T12:18:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:16 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T12:18:16.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:16 smithi114 ceph-mon[30324]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:16.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:16 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:18:16.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:16 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:18:16.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:16 smithi114 ceph-mon[40867]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T12:18:16.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:16 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T12:18:16.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:16 smithi114 ceph-mon[40867]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:18.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:18 smithi114 ceph-mon[40867]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:18.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:18 smithi114 ceph-mon[30324]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:18.412 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:18 smithi183 ceph-mon[33277]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:19.339 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:19 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T12:18:19.339 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:19 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:18:19.339 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:19 smithi183 ceph-mon[33277]: Deploying daemon osd.4 on smithi183 2024-09-21T12:18:19.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:19 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T12:18:19.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:19 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:18:19.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:19 smithi114 ceph-mon[40867]: Deploying daemon osd.4 on smithi183 2024-09-21T12:18:19.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:19 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T12:18:19.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:19 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:18:19.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:19 smithi114 ceph-mon[30324]: Deploying daemon osd.4 on smithi183 2024-09-21T12:18:20.330 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:20 smithi183 ceph-mon[33277]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:20.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:20 smithi114 ceph-mon[30324]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:20.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:20 smithi114 ceph-mon[40867]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:22.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:22 smithi183 ceph-mon[33277]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:22.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:22 smithi114 ceph-mon[30324]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:22.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:22 smithi114 ceph-mon[40867]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:24.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:24 smithi183 ceph-mon[33277]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:24.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:24 smithi114 ceph-mon[30324]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:24.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:24 smithi114 ceph-mon[40867]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:26.431 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:26 smithi183 ceph-mon[33277]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:26.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:26 smithi114 ceph-mon[30324]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:26.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:26 smithi114 ceph-mon[40867]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:27.421 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:27 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:18:27.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:27 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:27.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:27 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:27.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:27 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:18:27.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:27 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:27.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:27 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:27.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:27 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:18:27.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:27 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:27.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:27 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:28.436 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:28 smithi183 ceph-mon[33277]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:28.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:28 smithi114 ceph-mon[30324]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:28.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:28 smithi114 ceph-mon[40867]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:30.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:29 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:30.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:29 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:30.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:29 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:18:30.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:29 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:18:30.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:29 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:30.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:29 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:30.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:29 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:30.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:29 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:18:30.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:29 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:18:30.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:29 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:30.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:29 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:30.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:29 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:30.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:29 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:18:30.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:29 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:18:30.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:29 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:31.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:30 smithi183 ceph-mon[33277]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:31.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:30 smithi114 ceph-mon[30324]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:31.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:30 smithi114 ceph-mon[40867]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:31.662 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 4 on host 'smithi183' 2024-09-21T12:18:32.212 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:31 smithi183 ceph-mon[33277]: from='osd.4 [v2:172.21.15.183:6800/1391644836,v1:172.21.15.183:6801/1391644836]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T12:18:32.213 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:31 smithi183 ceph-mon[33277]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T12:18:32.213 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:31 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:18:32.213 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:31 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:32.213 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:31 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:32.247 DEBUG:teuthology.orchestra.run.smithi183:osd.4> sudo journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@osd.4.service 2024-09-21T12:18:32.249 INFO:tasks.cephadm:Deploying osd.5 on smithi183 with /dev/vg_nvme/lv_3... 2024-09-21T12:18:32.249 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T12:18:32.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:31 smithi114 ceph-mon[30324]: from='osd.4 [v2:172.21.15.183:6800/1391644836,v1:172.21.15.183:6801/1391644836]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T12:18:32.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:31 smithi114 ceph-mon[30324]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T12:18:32.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:31 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:18:32.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:31 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:32.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:31 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:32.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:31 smithi114 ceph-mon[40867]: from='osd.4 [v2:172.21.15.183:6800/1391644836,v1:172.21.15.183:6801/1391644836]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T12:18:32.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:31 smithi114 ceph-mon[40867]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T12:18:32.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:31 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:18:32.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:31 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:32.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:31 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:33.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:32 smithi114 ceph-mon[30324]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T12:18:33.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:32 smithi114 ceph-mon[30324]: from='osd.4 [v2:172.21.15.183:6800/1391644836,v1:172.21.15.183:6801/1391644836]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]} : dispatch 2024-09-21T12:18:33.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:32 smithi114 ceph-mon[30324]: osdmap e30: 5 total, 4 up, 5 in 2024-09-21T12:18:33.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:32 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T12:18:33.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:32 smithi114 ceph-mon[30324]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]} : dispatch 2024-09-21T12:18:33.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:32 smithi114 ceph-mon[30324]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:33.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:32 smithi114 ceph-mon[40867]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T12:18:33.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:32 smithi114 ceph-mon[40867]: from='osd.4 [v2:172.21.15.183:6800/1391644836,v1:172.21.15.183:6801/1391644836]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]} : dispatch 2024-09-21T12:18:33.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:32 smithi114 ceph-mon[40867]: osdmap e30: 5 total, 4 up, 5 in 2024-09-21T12:18:33.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:32 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T12:18:33.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:32 smithi114 ceph-mon[40867]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]} : dispatch 2024-09-21T12:18:33.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:32 smithi114 ceph-mon[40867]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:33.285 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:32 smithi183 ceph-mon[33277]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T12:18:33.285 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:32 smithi183 ceph-mon[33277]: from='osd.4 [v2:172.21.15.183:6800/1391644836,v1:172.21.15.183:6801/1391644836]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]} : dispatch 2024-09-21T12:18:33.285 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:32 smithi183 ceph-mon[33277]: osdmap e30: 5 total, 4 up, 5 in 2024-09-21T12:18:33.286 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:32 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T12:18:33.286 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:32 smithi183 ceph-mon[33277]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]} : dispatch 2024-09-21T12:18:33.286 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:32 smithi183 ceph-mon[33277]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:18:33.361 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:18:33.566 INFO:journalctl@ceph.osd.4.smithi183.stdout:Sep 21 12:18:33 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-4[37819]: 2024-09-21T12:18:33.442+0000 7f57d6d1f640 -1 osd.4 0 waiting for initial osdmap 2024-09-21T12:18:33.566 INFO:journalctl@ceph.osd.4.smithi183.stdout:Sep 21 12:18:33 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-4[37819]: 2024-09-21T12:18:33.445+0000 7f57ceb3e640 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T12:18:34.175 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:34 smithi183 ceph-mon[33277]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-09-21T12:18:34.175 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:34 smithi183 ceph-mon[33277]: osdmap e31: 5 total, 4 up, 5 in 2024-09-21T12:18:34.175 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:34 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T12:18:34.175 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:34 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T12:18:34.175 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:34 smithi183 ceph-mon[33277]: from='osd.4 ' entity='osd.4' 2024-09-21T12:18:34.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:34 smithi114 ceph-mon[30324]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-09-21T12:18:34.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:34 smithi114 ceph-mon[30324]: osdmap e31: 5 total, 4 up, 5 in 2024-09-21T12:18:34.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:34 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T12:18:34.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:34 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T12:18:34.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:34 smithi114 ceph-mon[30324]: from='osd.4 ' entity='osd.4' 2024-09-21T12:18:34.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:34 smithi114 ceph-mon[40867]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-09-21T12:18:34.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:34 smithi114 ceph-mon[40867]: osdmap e31: 5 total, 4 up, 5 in 2024-09-21T12:18:34.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:34 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T12:18:34.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:34 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T12:18:34.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:34 smithi114 ceph-mon[40867]: from='osd.4 ' entity='osd.4' 2024-09-21T12:18:35.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:35 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:18:35.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:35 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:18:35.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:35 smithi183 ceph-mon[33277]: osd.4 [v2:172.21.15.183:6800/1391644836,v1:172.21.15.183:6801/1391644836] boot 2024-09-21T12:18:35.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:35 smithi183 ceph-mon[33277]: osdmap e32: 5 total, 5 up, 5 in 2024-09-21T12:18:35.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:35 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T12:18:35.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:35 smithi183 ceph-mon[33277]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 347 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:18:35.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:35 smithi183 ceph-mon[33277]: osdmap e33: 5 total, 5 up, 5 in 2024-09-21T12:18:35.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:35 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:18:35.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:35 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:18:35.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:35 smithi114 ceph-mon[30324]: osd.4 [v2:172.21.15.183:6800/1391644836,v1:172.21.15.183:6801/1391644836] boot 2024-09-21T12:18:35.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:35 smithi114 ceph-mon[30324]: osdmap e32: 5 total, 5 up, 5 in 2024-09-21T12:18:35.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:35 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T12:18:35.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:35 smithi114 ceph-mon[30324]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 347 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:18:35.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:35 smithi114 ceph-mon[30324]: osdmap e33: 5 total, 5 up, 5 in 2024-09-21T12:18:35.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:35 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:18:35.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:35 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:18:35.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:35 smithi114 ceph-mon[40867]: osd.4 [v2:172.21.15.183:6800/1391644836,v1:172.21.15.183:6801/1391644836] boot 2024-09-21T12:18:35.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:35 smithi114 ceph-mon[40867]: osdmap e32: 5 total, 5 up, 5 in 2024-09-21T12:18:35.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:35 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T12:18:35.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:35 smithi114 ceph-mon[40867]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 347 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:18:35.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:35 smithi114 ceph-mon[40867]: osdmap e33: 5 total, 5 up, 5 in 2024-09-21T12:18:36.592 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:36 smithi183 ceph-mon[33277]: Detected new or changed devices on smithi183 2024-09-21T12:18:36.592 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:36 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:36.592 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:36 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:36.592 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:36 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T12:18:36.592 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:36 smithi183 ceph-mon[33277]: Adjusting osd_memory_target on smithi183 to 17088M 2024-09-21T12:18:36.592 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:36 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:36.593 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:36 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:18:36.593 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:36 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:18:36.593 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:36 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:36.593 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:36 smithi183 ceph-mon[33277]: osdmap e34: 5 total, 5 up, 5 in 2024-09-21T12:18:36.593 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:36 smithi183 ceph-mon[33277]: pgmap v122: 1 pgs: 1 peering; 577 KiB data, 347 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:18:37.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:36 smithi114 ceph-mon[40867]: Detected new or changed devices on smithi183 2024-09-21T12:18:37.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:36 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:37.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:36 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:37.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:36 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T12:18:37.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:36 smithi114 ceph-mon[40867]: Adjusting osd_memory_target on smithi183 to 17088M 2024-09-21T12:18:37.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:36 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:37.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:36 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:18:37.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:36 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:18:37.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:36 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:37.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:36 smithi114 ceph-mon[40867]: osdmap e34: 5 total, 5 up, 5 in 2024-09-21T12:18:37.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:36 smithi114 ceph-mon[40867]: pgmap v122: 1 pgs: 1 peering; 577 KiB data, 347 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:18:37.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:36 smithi114 ceph-mon[30324]: Detected new or changed devices on smithi183 2024-09-21T12:18:37.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:36 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:37.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:36 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:37.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:36 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T12:18:37.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:36 smithi114 ceph-mon[30324]: Adjusting osd_memory_target on smithi183 to 17088M 2024-09-21T12:18:37.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:36 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:37.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:36 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:18:37.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:36 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:18:37.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:36 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:37.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:36 smithi114 ceph-mon[30324]: osdmap e34: 5 total, 5 up, 5 in 2024-09-21T12:18:37.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:36 smithi114 ceph-mon[30324]: pgmap v122: 1 pgs: 1 peering; 577 KiB data, 347 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:18:37.234 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:18:37.259 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi183:vg_nvme/lv_3 2024-09-21T12:18:37.504 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:18:38.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:38 smithi183 ceph-mon[33277]: pgmap v123: 1 pgs: 1 peering; 577 KiB data, 347 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:18:38.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:38 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T12:18:38.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:38 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T12:18:38.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:38 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:18:38.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:38 smithi114 ceph-mon[40867]: pgmap v123: 1 pgs: 1 peering; 577 KiB data, 347 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:18:38.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:38 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T12:18:38.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:38 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T12:18:38.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:38 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:18:38.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:38 smithi114 ceph-mon[30324]: pgmap v123: 1 pgs: 1 peering; 577 KiB data, 347 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:18:38.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:38 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T12:18:38.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:38 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T12:18:38.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:38 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:18:39.482 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:39 smithi183 ceph-mon[33277]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:18:39.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:39 smithi114 ceph-mon[40867]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:18:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:39 smithi114 ceph-mon[30324]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:18:40.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:40 smithi183 ceph-mon[33277]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-09-21T12:18:40.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:40 smithi114 ceph-mon[30324]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-09-21T12:18:40.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:40 smithi114 ceph-mon[40867]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-09-21T12:18:41.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:41 smithi183 ceph-mon[33277]: from='client.? 172.21.15.183:0/4076905705' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "33f96871-0869-4281-82b9-bd518987c551"} : dispatch 2024-09-21T12:18:41.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:41 smithi183 ceph-mon[33277]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "33f96871-0869-4281-82b9-bd518987c551"} : dispatch 2024-09-21T12:18:41.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:41 smithi183 ceph-mon[33277]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33f96871-0869-4281-82b9-bd518987c551"}]': finished 2024-09-21T12:18:41.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:41 smithi183 ceph-mon[33277]: osdmap e35: 6 total, 5 up, 6 in 2024-09-21T12:18:41.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:41 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T12:18:41.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:41 smithi183 ceph-mon[33277]: from='client.? 172.21.15.183:0/3421320437' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T12:18:41.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:41 smithi114 ceph-mon[40867]: from='client.? 172.21.15.183:0/4076905705' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "33f96871-0869-4281-82b9-bd518987c551"} : dispatch 2024-09-21T12:18:41.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:41 smithi114 ceph-mon[40867]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "33f96871-0869-4281-82b9-bd518987c551"} : dispatch 2024-09-21T12:18:41.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:41 smithi114 ceph-mon[40867]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33f96871-0869-4281-82b9-bd518987c551"}]': finished 2024-09-21T12:18:41.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:41 smithi114 ceph-mon[40867]: osdmap e35: 6 total, 5 up, 6 in 2024-09-21T12:18:41.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:41 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T12:18:41.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:41 smithi114 ceph-mon[40867]: from='client.? 172.21.15.183:0/3421320437' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T12:18:41.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:41 smithi114 ceph-mon[30324]: from='client.? 172.21.15.183:0/4076905705' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "33f96871-0869-4281-82b9-bd518987c551"} : dispatch 2024-09-21T12:18:41.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:41 smithi114 ceph-mon[30324]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "33f96871-0869-4281-82b9-bd518987c551"} : dispatch 2024-09-21T12:18:41.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:41 smithi114 ceph-mon[30324]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33f96871-0869-4281-82b9-bd518987c551"}]': finished 2024-09-21T12:18:41.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:41 smithi114 ceph-mon[30324]: osdmap e35: 6 total, 5 up, 6 in 2024-09-21T12:18:41.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:41 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T12:18:41.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:41 smithi114 ceph-mon[30324]: from='client.? 172.21.15.183:0/3421320437' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T12:18:42.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:42 smithi183 ceph-mon[33277]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-09-21T12:18:43.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:42 smithi114 ceph-mon[40867]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-09-21T12:18:43.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:42 smithi114 ceph-mon[30324]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-09-21T12:18:44.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:44 smithi183 ceph-mon[33277]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-09-21T12:18:44.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:44 smithi114 ceph-mon[40867]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-09-21T12:18:44.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:44 smithi114 ceph-mon[30324]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-09-21T12:18:46.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:46 smithi183 ceph-mon[33277]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T12:18:46.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:46 smithi114 ceph-mon[40867]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T12:18:46.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:46 smithi114 ceph-mon[30324]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T12:18:47.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:47 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T12:18:47.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:47 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:18:48.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:47 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T12:18:48.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:47 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:18:48.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:47 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T12:18:48.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:47 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:18:49.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:48 smithi183 ceph-mon[33277]: Deploying daemon osd.5 on smithi183 2024-09-21T12:18:49.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:48 smithi183 ceph-mon[33277]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T12:18:49.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:48 smithi114 ceph-mon[30324]: Deploying daemon osd.5 on smithi183 2024-09-21T12:18:49.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:48 smithi114 ceph-mon[30324]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T12:18:49.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:48 smithi114 ceph-mon[40867]: Deploying daemon osd.5 on smithi183 2024-09-21T12:18:49.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:48 smithi114 ceph-mon[40867]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T12:18:50.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:50 smithi183 ceph-mon[33277]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:18:50.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:50 smithi114 ceph-mon[30324]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:18:50.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:50 smithi114 ceph-mon[40867]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:18:52.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:52 smithi183 ceph-mon[33277]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:18:52.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:52 smithi114 ceph-mon[30324]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:18:52.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:52 smithi114 ceph-mon[40867]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:18:54.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:54 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:18:54.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:54 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:18:54.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:54 smithi183 ceph-mon[33277]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T12:18:54.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:54 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T12:18:54.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:54 smithi183 ceph-mon[33277]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:18:55.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:54 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:18:55.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:54 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:18:55.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:54 smithi114 ceph-mon[30324]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T12:18:55.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:54 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T12:18:55.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:54 smithi114 ceph-mon[30324]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:18:55.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:54 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:18:55.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:54 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:18:55.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:54 smithi114 ceph-mon[40867]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T12:18:55.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:54 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T12:18:55.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:54 smithi114 ceph-mon[40867]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:18:56.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:55 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:18:56.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:55 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:56.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:55 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:55 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:18:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:55 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:55 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:56.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:55 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:18:56.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:55 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:56.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:55 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:56.964 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:56 smithi183 ceph-mon[33277]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:18:57.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:56 smithi114 ceph-mon[30324]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:18:57.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:56 smithi114 ceph-mon[40867]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:18:58.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:58 smithi183 ceph-mon[33277]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:18:58.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:58 smithi114 ceph-mon[30324]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:18:58.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:58 smithi114 ceph-mon[40867]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:18:59.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:59 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:59.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:59 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:59.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:59 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:18:59.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:59 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:18:59.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:18:59 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:59.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:59 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:59.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:59 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:59.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:59 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:18:59.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:59 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:18:59.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:18:59 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:59.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:59 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:59.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:59 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:18:59.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:59 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:18:59.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:59 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:18:59.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:18:59 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:00.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:00 smithi183 ceph-mon[33277]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:19:00.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:00 smithi114 ceph-mon[30324]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:19:00.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:00 smithi114 ceph-mon[40867]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:19:01.231 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 5 on host 'smithi183' 2024-09-21T12:19:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:01 smithi183 ceph-mon[33277]: from='osd.5 [v2:172.21.15.183:6808/3187643473,v1:172.21.15.183:6809/3187643473]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T12:19:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:01 smithi183 ceph-mon[33277]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T12:19:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:01 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:19:01.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:01 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:01.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:01 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:01.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:01 smithi114 ceph-mon[30324]: from='osd.5 [v2:172.21.15.183:6808/3187643473,v1:172.21.15.183:6809/3187643473]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T12:19:01.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:01 smithi114 ceph-mon[30324]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T12:19:01.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:01 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:19:01.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:01 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:01.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:01 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:01.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:01 smithi114 ceph-mon[40867]: from='osd.5 [v2:172.21.15.183:6808/3187643473,v1:172.21.15.183:6809/3187643473]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T12:19:01.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:01 smithi114 ceph-mon[40867]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T12:19:01.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:01 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:19:01.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:01 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:01.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:01 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:01.838 DEBUG:teuthology.orchestra.run.smithi183:osd.5> sudo journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@osd.5.service 2024-09-21T12:19:01.841 INFO:tasks.cephadm:Deploying osd.6 on smithi183 with /dev/vg_nvme/lv_2... 2024-09-21T12:19:01.841 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T12:19:02.614 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:19:02.719 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:02 smithi183 ceph-mon[33277]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T12:19:02.719 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:02 smithi183 ceph-mon[33277]: osdmap e37: 6 total, 5 up, 6 in 2024-09-21T12:19:02.719 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:02 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T12:19:02.719 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:02 smithi183 ceph-mon[33277]: from='osd.5 [v2:172.21.15.183:6808/3187643473,v1:172.21.15.183:6809/3187643473]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]} : dispatch 2024-09-21T12:19:02.719 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:02 smithi183 ceph-mon[33277]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]} : dispatch 2024-09-21T12:19:02.719 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:02 smithi183 ceph-mon[33277]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:19:02.719 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:19:02 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:19:02.425+0000 7fa16bc70640 -1 osd.5 0 waiting for initial osdmap 2024-09-21T12:19:02.719 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:19:02 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:19:02.431+0000 7fa167a97640 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T12:19:02.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:02 smithi114 ceph-mon[30324]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T12:19:02.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:02 smithi114 ceph-mon[30324]: osdmap e37: 6 total, 5 up, 6 in 2024-09-21T12:19:02.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:02 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T12:19:02.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:02 smithi114 ceph-mon[30324]: from='osd.5 [v2:172.21.15.183:6808/3187643473,v1:172.21.15.183:6809/3187643473]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]} : dispatch 2024-09-21T12:19:02.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:02 smithi114 ceph-mon[30324]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]} : dispatch 2024-09-21T12:19:02.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:02 smithi114 ceph-mon[30324]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:19:02.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:02 smithi114 ceph-mon[40867]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T12:19:02.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:02 smithi114 ceph-mon[40867]: osdmap e37: 6 total, 5 up, 6 in 2024-09-21T12:19:02.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:02 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T12:19:02.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:02 smithi114 ceph-mon[40867]: from='osd.5 [v2:172.21.15.183:6808/3187643473,v1:172.21.15.183:6809/3187643473]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]} : dispatch 2024-09-21T12:19:02.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:02 smithi114 ceph-mon[40867]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]} : dispatch 2024-09-21T12:19:02.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:02 smithi114 ceph-mon[40867]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:19:03.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:03 smithi183 ceph-mon[33277]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-09-21T12:19:03.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:03 smithi183 ceph-mon[33277]: osdmap e38: 6 total, 5 up, 6 in 2024-09-21T12:19:03.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:03 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T12:19:03.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:03 smithi183 ceph-mon[33277]: osd.5 [v2:172.21.15.183:6808/3187643473,v1:172.21.15.183:6809/3187643473] boot 2024-09-21T12:19:03.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:03 smithi183 ceph-mon[33277]: osdmap e39: 6 total, 6 up, 6 in 2024-09-21T12:19:03.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:03 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T12:19:03.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:03 smithi114 ceph-mon[30324]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-09-21T12:19:03.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:03 smithi114 ceph-mon[30324]: osdmap e38: 6 total, 5 up, 6 in 2024-09-21T12:19:03.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:03 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T12:19:03.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:03 smithi114 ceph-mon[30324]: osd.5 [v2:172.21.15.183:6808/3187643473,v1:172.21.15.183:6809/3187643473] boot 2024-09-21T12:19:03.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:03 smithi114 ceph-mon[30324]: osdmap e39: 6 total, 6 up, 6 in 2024-09-21T12:19:03.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:03 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T12:19:03.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:03 smithi114 ceph-mon[40867]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-09-21T12:19:03.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:03 smithi114 ceph-mon[40867]: osdmap e38: 6 total, 5 up, 6 in 2024-09-21T12:19:03.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:03 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T12:19:03.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:03 smithi114 ceph-mon[40867]: osd.5 [v2:172.21.15.183:6808/3187643473,v1:172.21.15.183:6809/3187643473] boot 2024-09-21T12:19:03.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:03 smithi114 ceph-mon[40867]: osdmap e39: 6 total, 6 up, 6 in 2024-09-21T12:19:03.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:03 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T12:19:04.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:04 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:19:04.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:04 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:19:04.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:04 smithi183 ceph-mon[33277]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:04.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:04 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:19:04.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:04 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:19:04.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:04 smithi114 ceph-mon[30324]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:04.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:04 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:19:04.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:04 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:19:04.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:04 smithi114 ceph-mon[40867]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:05.608 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:05 smithi183 ceph-mon[33277]: osdmap e40: 6 total, 6 up, 6 in 2024-09-21T12:19:05.608 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:05 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:19:05.608 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:05 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:19:05.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:05 smithi114 ceph-mon[30324]: osdmap e40: 6 total, 6 up, 6 in 2024-09-21T12:19:05.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:05 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:19:05.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:05 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:19:05.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:05 smithi114 ceph-mon[40867]: osdmap e40: 6 total, 6 up, 6 in 2024-09-21T12:19:05.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:05 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:19:05.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:05 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:19:06.668 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:06 smithi183 ceph-mon[33277]: osdmap e41: 6 total, 6 up, 6 in 2024-09-21T12:19:06.668 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:06 smithi183 ceph-mon[33277]: Detected new or changed devices on smithi183 2024-09-21T12:19:06.668 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:06 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:06.668 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:06 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:06.668 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:06 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T12:19:06.668 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:06 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T12:19:06.669 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:06 smithi183 ceph-mon[33277]: Adjusting osd_memory_target on smithi183 to 8544M 2024-09-21T12:19:06.669 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:06 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:06.669 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:06 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:19:06.669 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:06 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:19:06.669 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:06 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:06.669 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:06 smithi183 ceph-mon[33277]: 1.0 scrub starts 2024-09-21T12:19:06.669 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:06 smithi183 ceph-mon[33277]: 1.0 scrub ok 2024-09-21T12:19:06.669 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:06 smithi183 ceph-mon[33277]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:06.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:06 smithi114 ceph-mon[40867]: osdmap e41: 6 total, 6 up, 6 in 2024-09-21T12:19:06.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:06 smithi114 ceph-mon[40867]: Detected new or changed devices on smithi183 2024-09-21T12:19:06.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:06 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:06.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:06 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:06.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:06 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T12:19:06.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:06 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T12:19:06.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:06 smithi114 ceph-mon[40867]: Adjusting osd_memory_target on smithi183 to 8544M 2024-09-21T12:19:06.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:06 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:06.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:06 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:19:06.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:06 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:19:06.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:06 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:06.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:06 smithi114 ceph-mon[40867]: 1.0 scrub starts 2024-09-21T12:19:06.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:06 smithi114 ceph-mon[40867]: 1.0 scrub ok 2024-09-21T12:19:06.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:06 smithi114 ceph-mon[40867]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:06.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:06 smithi114 ceph-mon[30324]: osdmap e41: 6 total, 6 up, 6 in 2024-09-21T12:19:06.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:06 smithi114 ceph-mon[30324]: Detected new or changed devices on smithi183 2024-09-21T12:19:06.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:06 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:06.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:06 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:06.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:06 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T12:19:06.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:06 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T12:19:06.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:06 smithi114 ceph-mon[30324]: Adjusting osd_memory_target on smithi183 to 8544M 2024-09-21T12:19:06.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:06 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:06.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:06 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:19:06.777 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:06 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:19:06.777 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:06 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:06.777 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:06 smithi114 ceph-mon[30324]: 1.0 scrub starts 2024-09-21T12:19:06.777 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:06 smithi114 ceph-mon[30324]: 1.0 scrub ok 2024-09-21T12:19:06.777 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:06 smithi114 ceph-mon[30324]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:07.285 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:19:07.311 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi183:vg_nvme/lv_2 2024-09-21T12:19:07.557 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:19:08.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:08 smithi183 ceph-mon[33277]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:08.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:08 smithi114 ceph-mon[30324]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:08.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:08 smithi114 ceph-mon[40867]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:09.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:09 smithi183 ceph-mon[33277]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:19:09.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T12:19:09.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T12:19:09.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:09 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:19:09.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:09 smithi114 ceph-mon[30324]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:19:09.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T12:19:09.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T12:19:09.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:09 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:19:09.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:09 smithi114 ceph-mon[40867]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:19:09.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T12:19:09.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T12:19:09.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:09 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:19:10.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:10 smithi183 ceph-mon[33277]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:10.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:10 smithi114 ceph-mon[30324]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:10.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:10 smithi114 ceph-mon[40867]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:11.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:11 smithi183 ceph-mon[33277]: from='client.? 172.21.15.183:0/10822253' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e09c72b8-ad5a-4779-a874-b149fce6ed70"} : dispatch 2024-09-21T12:19:11.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:11 smithi183 ceph-mon[33277]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e09c72b8-ad5a-4779-a874-b149fce6ed70"} : dispatch 2024-09-21T12:19:11.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:11 smithi183 ceph-mon[33277]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e09c72b8-ad5a-4779-a874-b149fce6ed70"}]': finished 2024-09-21T12:19:11.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:11 smithi183 ceph-mon[33277]: osdmap e42: 7 total, 6 up, 7 in 2024-09-21T12:19:11.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:11 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T12:19:11.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:11 smithi183 ceph-mon[33277]: from='client.? 172.21.15.183:0/3899309717' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T12:19:11.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:11 smithi114 ceph-mon[30324]: from='client.? 172.21.15.183:0/10822253' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e09c72b8-ad5a-4779-a874-b149fce6ed70"} : dispatch 2024-09-21T12:19:11.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:11 smithi114 ceph-mon[30324]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e09c72b8-ad5a-4779-a874-b149fce6ed70"} : dispatch 2024-09-21T12:19:11.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:11 smithi114 ceph-mon[30324]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e09c72b8-ad5a-4779-a874-b149fce6ed70"}]': finished 2024-09-21T12:19:11.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:11 smithi114 ceph-mon[30324]: osdmap e42: 7 total, 6 up, 7 in 2024-09-21T12:19:11.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:11 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T12:19:11.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:11 smithi114 ceph-mon[30324]: from='client.? 172.21.15.183:0/3899309717' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T12:19:11.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:11 smithi114 ceph-mon[40867]: from='client.? 172.21.15.183:0/10822253' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e09c72b8-ad5a-4779-a874-b149fce6ed70"} : dispatch 2024-09-21T12:19:11.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:11 smithi114 ceph-mon[40867]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e09c72b8-ad5a-4779-a874-b149fce6ed70"} : dispatch 2024-09-21T12:19:11.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:11 smithi114 ceph-mon[40867]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e09c72b8-ad5a-4779-a874-b149fce6ed70"}]': finished 2024-09-21T12:19:11.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:11 smithi114 ceph-mon[40867]: osdmap e42: 7 total, 6 up, 7 in 2024-09-21T12:19:11.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:11 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T12:19:11.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:11 smithi114 ceph-mon[40867]: from='client.? 172.21.15.183:0/3899309717' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T12:19:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:12 smithi183 ceph-mon[33277]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:12.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:12 smithi114 ceph-mon[30324]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:12.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:12 smithi114 ceph-mon[40867]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:14.352 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:14 smithi183 ceph-mon[33277]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:14.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:14 smithi114 ceph-mon[30324]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:14.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:14 smithi114 ceph-mon[40867]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:16 smithi183 ceph-mon[33277]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:16.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:16 smithi114 ceph-mon[30324]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:16.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:16 smithi114 ceph-mon[40867]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:17.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:17 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T12:19:17.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:17 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:19:18.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:17 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T12:19:18.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:17 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:19:18.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:17 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T12:19:18.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:17 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:19:18.959 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:18 smithi183 ceph-mon[33277]: Deploying daemon osd.6 on smithi183 2024-09-21T12:19:18.960 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:18 smithi183 ceph-mon[33277]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:19.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:18 smithi114 ceph-mon[30324]: Deploying daemon osd.6 on smithi183 2024-09-21T12:19:19.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:18 smithi114 ceph-mon[30324]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:19.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:18 smithi114 ceph-mon[40867]: Deploying daemon osd.6 on smithi183 2024-09-21T12:19:19.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:18 smithi114 ceph-mon[40867]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:20 smithi183 ceph-mon[33277]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:20.537 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:20 smithi114 ceph-mon[30324]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:20.538 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:20 smithi114 ceph-mon[40867]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:23.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:22 smithi114 ceph-mon[40867]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:23.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:22 smithi114 ceph-mon[30324]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:23.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:22 smithi183 ceph-mon[33277]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:24.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:24 smithi183 ceph-mon[33277]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:24.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:24 smithi114 ceph-mon[30324]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:24.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:24 smithi114 ceph-mon[40867]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:25.242 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:25 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:19:25.242 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:25 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:19:25.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:25 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:19:25.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:25 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:19:25.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:25 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:19:25.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:25 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:19:26.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:26 smithi183 ceph-mon[33277]: osdmap e43: 7 total, 6 up, 7 in 2024-09-21T12:19:26.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:26 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T12:19:26.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:26 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:19:26.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:26 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:26.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:26 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:26.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:26 smithi183 ceph-mon[33277]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:26.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:26 smithi114 ceph-mon[40867]: osdmap e43: 7 total, 6 up, 7 in 2024-09-21T12:19:26.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:26 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T12:19:26.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:26 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:19:26.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:26 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:26.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:26 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:26.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:26 smithi114 ceph-mon[40867]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:26.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:26 smithi114 ceph-mon[30324]: osdmap e43: 7 total, 6 up, 7 in 2024-09-21T12:19:26.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:26 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T12:19:26.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:26 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:19:26.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:26 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:26.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:26 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:26.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:26 smithi114 ceph-mon[30324]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:28.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:28 smithi183 ceph-mon[33277]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:28.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:28 smithi114 ceph-mon[30324]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:28.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:28 smithi114 ceph-mon[40867]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:29 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:29 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:29 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:19:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:29 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:19:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:29 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:29.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:29 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:29.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:29 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:29.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:29 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:19:29.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:29 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:19:29.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:29 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:29.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:29 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:29.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:29 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:29.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:29 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:19:29.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:29 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:19:29.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:29 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:30.690 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:30 smithi183 ceph-mon[33277]: from='osd.6 [v2:172.21.15.183:6816/3025199631,v1:172.21.15.183:6817/3025199631]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T12:19:30.691 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:30 smithi183 ceph-mon[33277]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T12:19:30.691 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:30 smithi183 ceph-mon[33277]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:30.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:30 smithi114 ceph-mon[30324]: from='osd.6 [v2:172.21.15.183:6816/3025199631,v1:172.21.15.183:6817/3025199631]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T12:19:30.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:30 smithi114 ceph-mon[30324]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T12:19:30.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:30 smithi114 ceph-mon[30324]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:30.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:30 smithi114 ceph-mon[40867]: from='osd.6 [v2:172.21.15.183:6816/3025199631,v1:172.21.15.183:6817/3025199631]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T12:19:30.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:30 smithi114 ceph-mon[40867]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T12:19:30.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:30 smithi114 ceph-mon[40867]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:31.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:31 smithi183 ceph-mon[33277]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T12:19:31.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:31 smithi183 ceph-mon[33277]: osdmap e44: 7 total, 6 up, 7 in 2024-09-21T12:19:31.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:31 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T12:19:31.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:31 smithi183 ceph-mon[33277]: from='osd.6 [v2:172.21.15.183:6816/3025199631,v1:172.21.15.183:6817/3025199631]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi183", "root=default"]} : dispatch 2024-09-21T12:19:31.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:31 smithi183 ceph-mon[33277]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi183", "root=default"]} : dispatch 2024-09-21T12:19:31.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:31 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:19:31.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:31 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:19:31.548 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 6 on host 'smithi183' 2024-09-21T12:19:31.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:31 smithi114 ceph-mon[30324]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T12:19:31.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:31 smithi114 ceph-mon[30324]: osdmap e44: 7 total, 6 up, 7 in 2024-09-21T12:19:31.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:31 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T12:19:31.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:31 smithi114 ceph-mon[30324]: from='osd.6 [v2:172.21.15.183:6816/3025199631,v1:172.21.15.183:6817/3025199631]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi183", "root=default"]} : dispatch 2024-09-21T12:19:31.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:31 smithi114 ceph-mon[30324]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi183", "root=default"]} : dispatch 2024-09-21T12:19:31.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:31 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:19:31.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:31 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:19:31.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:31 smithi114 ceph-mon[40867]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T12:19:31.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:31 smithi114 ceph-mon[40867]: osdmap e44: 7 total, 6 up, 7 in 2024-09-21T12:19:31.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:31 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T12:19:31.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:31 smithi114 ceph-mon[40867]: from='osd.6 [v2:172.21.15.183:6816/3025199631,v1:172.21.15.183:6817/3025199631]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi183", "root=default"]} : dispatch 2024-09-21T12:19:31.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:31 smithi114 ceph-mon[40867]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi183", "root=default"]} : dispatch 2024-09-21T12:19:31.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:31 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:19:31.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:31 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:19:32.140 DEBUG:teuthology.orchestra.run.smithi183:osd.6> sudo journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@osd.6.service 2024-09-21T12:19:32.142 INFO:tasks.cephadm:Deploying osd.7 on smithi183 with /dev/vg_nvme/lv_1... 2024-09-21T12:19:32.142 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T12:19:32.701 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:32 smithi183 ceph-mon[33277]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-09-21T12:19:32.701 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:32 smithi183 ceph-mon[33277]: osdmap e45: 7 total, 6 up, 7 in 2024-09-21T12:19:32.701 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:32 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T12:19:32.701 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:32 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T12:19:32.701 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:32 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:19:32.701 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:32 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:32.702 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:32 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:32.702 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:32 smithi183 ceph-mon[33277]: osdmap e46: 7 total, 6 up, 7 in 2024-09-21T12:19:32.702 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:32 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T12:19:32.702 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:32 smithi183 ceph-mon[33277]: from='osd.6 ' entity='osd.6' 2024-09-21T12:19:32.702 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:32 smithi183 ceph-mon[33277]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:32.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:32 smithi114 ceph-mon[30324]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-09-21T12:19:32.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:32 smithi114 ceph-mon[30324]: osdmap e45: 7 total, 6 up, 7 in 2024-09-21T12:19:32.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:32 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T12:19:32.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:32 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T12:19:32.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:32 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:19:32.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:32 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:32.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:32 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:32.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:32 smithi114 ceph-mon[30324]: osdmap e46: 7 total, 6 up, 7 in 2024-09-21T12:19:32.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:32 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T12:19:32.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:32 smithi114 ceph-mon[30324]: from='osd.6 ' entity='osd.6' 2024-09-21T12:19:32.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:32 smithi114 ceph-mon[30324]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:32.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:32 smithi114 ceph-mon[40867]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-09-21T12:19:32.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:32 smithi114 ceph-mon[40867]: osdmap e45: 7 total, 6 up, 7 in 2024-09-21T12:19:32.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:32 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T12:19:32.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:32 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T12:19:32.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:32 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:19:32.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:32 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:32.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:32 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:32.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:32 smithi114 ceph-mon[40867]: osdmap e46: 7 total, 6 up, 7 in 2024-09-21T12:19:32.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:32 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T12:19:32.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:32 smithi114 ceph-mon[40867]: from='osd.6 ' entity='osd.6' 2024-09-21T12:19:32.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:32 smithi114 ceph-mon[40867]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:19:32.925 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:19:33.679 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:33 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:19:33.680 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:33 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:19:33.680 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:33 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T12:19:33.680 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:33 smithi183 ceph-mon[33277]: osd.6 [v2:172.21.15.183:6816/3025199631,v1:172.21.15.183:6817/3025199631] boot 2024-09-21T12:19:33.680 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:33 smithi183 ceph-mon[33277]: osdmap e47: 7 total, 7 up, 7 in 2024-09-21T12:19:33.680 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:33 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T12:19:33.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:33 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:19:33.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:33 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:19:33.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:33 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T12:19:33.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:33 smithi114 ceph-mon[30324]: osd.6 [v2:172.21.15.183:6816/3025199631,v1:172.21.15.183:6817/3025199631] boot 2024-09-21T12:19:33.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:33 smithi114 ceph-mon[30324]: osdmap e47: 7 total, 7 up, 7 in 2024-09-21T12:19:33.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:33 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T12:19:33.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:33 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:19:33.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:33 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:19:33.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:33 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T12:19:33.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:33 smithi114 ceph-mon[40867]: osd.6 [v2:172.21.15.183:6816/3025199631,v1:172.21.15.183:6817/3025199631] boot 2024-09-21T12:19:33.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:33 smithi114 ceph-mon[40867]: osdmap e47: 7 total, 7 up, 7 in 2024-09-21T12:19:33.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:33 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T12:19:34.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:34 smithi183 ceph-mon[33277]: osdmap e48: 7 total, 7 up, 7 in 2024-09-21T12:19:34.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:34 smithi183 ceph-mon[33277]: pgmap v165: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:19:35.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:34 smithi114 ceph-mon[30324]: osdmap e48: 7 total, 7 up, 7 in 2024-09-21T12:19:35.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:34 smithi114 ceph-mon[30324]: pgmap v165: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:19:35.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:34 smithi114 ceph-mon[40867]: osdmap e48: 7 total, 7 up, 7 in 2024-09-21T12:19:35.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:34 smithi114 ceph-mon[40867]: pgmap v165: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:19:36.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:35 smithi114 ceph-mon[30324]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T12:19:36.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:35 smithi114 ceph-mon[30324]: osdmap e49: 7 total, 7 up, 7 in 2024-09-21T12:19:36.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:35 smithi114 ceph-mon[40867]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T12:19:36.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:35 smithi114 ceph-mon[40867]: osdmap e49: 7 total, 7 up, 7 in 2024-09-21T12:19:36.088 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:35 smithi183 ceph-mon[33277]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T12:19:36.088 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:35 smithi183 ceph-mon[33277]: osdmap e49: 7 total, 7 up, 7 in 2024-09-21T12:19:37.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:36 smithi183 ceph-mon[33277]: Detected new or changed devices on smithi183 2024-09-21T12:19:37.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:36 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:37.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:36 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:37.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:36 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T12:19:37.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:36 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T12:19:37.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:36 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T12:19:37.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:36 smithi183 ceph-mon[33277]: Adjusting osd_memory_target on smithi183 to 5696M 2024-09-21T12:19:37.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:36 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:37.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:36 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:19:37.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:36 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:19:37.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:36 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:37.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:36 smithi183 ceph-mon[33277]: pgmap v167: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:19:37.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:36 smithi114 ceph-mon[30324]: Detected new or changed devices on smithi183 2024-09-21T12:19:37.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:36 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:37.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:36 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:37.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:36 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T12:19:37.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:36 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T12:19:37.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:36 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T12:19:37.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:36 smithi114 ceph-mon[30324]: Adjusting osd_memory_target on smithi183 to 5696M 2024-09-21T12:19:37.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:36 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:37.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:36 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:19:37.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:36 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:19:37.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:36 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:37.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:36 smithi114 ceph-mon[30324]: pgmap v167: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:19:37.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:36 smithi114 ceph-mon[40867]: Detected new or changed devices on smithi183 2024-09-21T12:19:37.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:36 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:37.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:36 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:37.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:36 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T12:19:37.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:36 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T12:19:37.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:36 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T12:19:37.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:36 smithi114 ceph-mon[40867]: Adjusting osd_memory_target on smithi183 to 5696M 2024-09-21T12:19:37.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:36 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:37.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:36 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:19:37.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:36 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:19:37.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:36 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:37.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:36 smithi114 ceph-mon[40867]: pgmap v167: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:19:37.645 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-21T12:19:37.677 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi183:vg_nvme/lv_1 2024-09-21T12:19:37.928 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:19:38.194 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:37 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:19:38.194 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:37 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:19:38.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:37 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:19:38.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:37 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:19:38.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:37 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:19:38.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:37 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:19:39.232 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:38 smithi183 ceph-mon[33277]: osdmap e50: 7 total, 7 up, 7 in 2024-09-21T12:19:39.232 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:38 smithi183 ceph-mon[33277]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:19:39.232 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:38 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T12:19:39.232 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:38 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T12:19:39.232 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:38 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:19:39.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:38 smithi114 ceph-mon[30324]: osdmap e50: 7 total, 7 up, 7 in 2024-09-21T12:19:39.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:38 smithi114 ceph-mon[30324]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:19:39.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:38 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T12:19:39.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:38 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T12:19:39.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:38 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:19:39.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:38 smithi114 ceph-mon[40867]: osdmap e50: 7 total, 7 up, 7 in 2024-09-21T12:19:39.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:38 smithi114 ceph-mon[40867]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:19:39.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:38 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T12:19:39.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:38 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T12:19:39.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:38 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:19:39.985 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:39 smithi183 ceph-mon[33277]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:19:40.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:39 smithi114 ceph-mon[30324]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:19:40.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:39 smithi114 ceph-mon[40867]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:19:41.223 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:40 smithi183 ceph-mon[33277]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-09-21T12:19:41.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:40 smithi114 ceph-mon[30324]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-09-21T12:19:41.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:40 smithi114 ceph-mon[40867]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-09-21T12:19:42.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:41 smithi183 ceph-mon[33277]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T12:19:42.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:41 smithi183 ceph-mon[33277]: Cluster is now healthy 2024-09-21T12:19:42.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:41 smithi183 ceph-mon[33277]: from='client.? 172.21.15.183:0/1272464753' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1a24194d-1af6-485f-8856-98844a02dd9d"} : dispatch 2024-09-21T12:19:42.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:41 smithi183 ceph-mon[33277]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1a24194d-1af6-485f-8856-98844a02dd9d"} : dispatch 2024-09-21T12:19:42.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:41 smithi183 ceph-mon[33277]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a24194d-1af6-485f-8856-98844a02dd9d"}]': finished 2024-09-21T12:19:42.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:41 smithi183 ceph-mon[33277]: osdmap e51: 8 total, 7 up, 8 in 2024-09-21T12:19:42.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:41 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T12:19:42.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:41 smithi183 ceph-mon[33277]: from='client.? 172.21.15.183:0/505106710' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T12:19:42.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:41 smithi114 ceph-mon[30324]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T12:19:42.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:41 smithi114 ceph-mon[30324]: Cluster is now healthy 2024-09-21T12:19:42.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:41 smithi114 ceph-mon[30324]: from='client.? 172.21.15.183:0/1272464753' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1a24194d-1af6-485f-8856-98844a02dd9d"} : dispatch 2024-09-21T12:19:42.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:41 smithi114 ceph-mon[30324]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1a24194d-1af6-485f-8856-98844a02dd9d"} : dispatch 2024-09-21T12:19:42.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:41 smithi114 ceph-mon[30324]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a24194d-1af6-485f-8856-98844a02dd9d"}]': finished 2024-09-21T12:19:42.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:41 smithi114 ceph-mon[30324]: osdmap e51: 8 total, 7 up, 8 in 2024-09-21T12:19:42.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:41 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T12:19:42.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:41 smithi114 ceph-mon[30324]: from='client.? 172.21.15.183:0/505106710' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T12:19:42.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:41 smithi114 ceph-mon[40867]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T12:19:42.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:41 smithi114 ceph-mon[40867]: Cluster is now healthy 2024-09-21T12:19:42.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:41 smithi114 ceph-mon[40867]: from='client.? 172.21.15.183:0/1272464753' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1a24194d-1af6-485f-8856-98844a02dd9d"} : dispatch 2024-09-21T12:19:42.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:41 smithi114 ceph-mon[40867]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1a24194d-1af6-485f-8856-98844a02dd9d"} : dispatch 2024-09-21T12:19:42.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:41 smithi114 ceph-mon[40867]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a24194d-1af6-485f-8856-98844a02dd9d"}]': finished 2024-09-21T12:19:42.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:41 smithi114 ceph-mon[40867]: osdmap e51: 8 total, 7 up, 8 in 2024-09-21T12:19:42.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:41 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T12:19:42.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:41 smithi114 ceph-mon[40867]: from='client.? 172.21.15.183:0/505106710' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T12:19:43.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:42 smithi183 ceph-mon[33277]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-21T12:19:43.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:42 smithi114 ceph-mon[30324]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-21T12:19:43.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:42 smithi114 ceph-mon[40867]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-21T12:19:44.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:44 smithi183 ceph-mon[33277]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T12:19:44.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:44 smithi114 ceph-mon[30324]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T12:19:44.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:44 smithi114 ceph-mon[40867]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T12:19:46.271 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:46 smithi183 ceph-mon[33277]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-09-21T12:19:46.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:46 smithi114 ceph-mon[30324]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-09-21T12:19:46.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:46 smithi114 ceph-mon[40867]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-09-21T12:19:48.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:48 smithi183 ceph-mon[33277]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T12:19:48.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:48 smithi114 ceph-mon[40867]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T12:19:48.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:48 smithi114 ceph-mon[30324]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T12:19:49.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:49 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T12:19:49.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:49 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:19:49.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:49 smithi183 ceph-mon[33277]: Deploying daemon osd.7 on smithi183 2024-09-21T12:19:49.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:49 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T12:19:49.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:49 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:19:49.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:49 smithi114 ceph-mon[40867]: Deploying daemon osd.7 on smithi183 2024-09-21T12:19:49.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:49 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T12:19:49.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:49 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:19:49.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:49 smithi114 ceph-mon[30324]: Deploying daemon osd.7 on smithi183 2024-09-21T12:19:50.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:50 smithi183 ceph-mon[33277]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:19:50.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:50 smithi114 ceph-mon[30324]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:19:50.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:50 smithi114 ceph-mon[40867]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:19:52.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:52 smithi183 ceph-mon[33277]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:19:52.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:52 smithi114 ceph-mon[30324]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:19:52.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:52 smithi114 ceph-mon[40867]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:19:54.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:54 smithi183 ceph-mon[33277]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:19:54.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:54 smithi114 ceph-mon[30324]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:19:54.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:54 smithi114 ceph-mon[40867]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:19:56.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:56 smithi183 ceph-mon[33277]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:19:56.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:56 smithi114 ceph-mon[30324]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:19:56.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:56 smithi114 ceph-mon[40867]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:19:57.375 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:57 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:19:57.376 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:57 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:57.376 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:57 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:57.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:57 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:19:57.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:57 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:57.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:57 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:57.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:57 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:19:57.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:57 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:57.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:57 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:19:58.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:19:58 smithi183 ceph-mon[33277]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:19:58.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:19:58 smithi114 ceph-mon[30324]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:19:58.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:19:58 smithi114 ceph-mon[40867]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:20:00.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:00 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:00.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:00 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:00.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:00 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:20:00.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:00 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:20:00.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:00 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:00.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:00 smithi183 ceph-mon[33277]: overall HEALTH_OK 2024-09-21T12:20:00.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:00 smithi183 ceph-mon[33277]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:20:00.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:00 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:00.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:00 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:00.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:00 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:20:00.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:00 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:20:00.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:00 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:00.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:00 smithi114 ceph-mon[30324]: overall HEALTH_OK 2024-09-21T12:20:00.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:00 smithi114 ceph-mon[30324]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:20:00.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:00 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:00.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:00 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:00.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:00 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:20:00.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:00 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:20:00.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:00 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:00.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:00 smithi114 ceph-mon[40867]: overall HEALTH_OK 2024-09-21T12:20:00.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:00 smithi114 ceph-mon[40867]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:20:01.697 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 7 on host 'smithi183' 2024-09-21T12:20:01.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:01 smithi183 ceph-mon[33277]: from='osd.7 [v2:172.21.15.183:6824/3203337442,v1:172.21.15.183:6825/3203337442]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T12:20:01.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:01 smithi183 ceph-mon[33277]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T12:20:01.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:01 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:20:01.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:01 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:01.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:01 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:02.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:01 smithi114 ceph-mon[40867]: from='osd.7 [v2:172.21.15.183:6824/3203337442,v1:172.21.15.183:6825/3203337442]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T12:20:02.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:01 smithi114 ceph-mon[40867]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T12:20:02.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:01 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:20:02.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:01 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:02.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:01 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:02.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:01 smithi114 ceph-mon[30324]: from='osd.7 [v2:172.21.15.183:6824/3203337442,v1:172.21.15.183:6825/3203337442]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T12:20:02.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:01 smithi114 ceph-mon[30324]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T12:20:02.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:01 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:20:02.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:01 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:02.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:01 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:02.562 DEBUG:teuthology.orchestra.run.smithi183:osd.7> sudo journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@osd.7.service 2024-09-21T12:20:02.565 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-21T12:20:02.565 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd stat -f json 2024-09-21T12:20:02.829 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:20:02.846 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:02 smithi183 ceph-mon[33277]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T12:20:02.847 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:02 smithi183 ceph-mon[33277]: from='osd.7 [v2:172.21.15.183:6824/3203337442,v1:172.21.15.183:6825/3203337442]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi183", "root=default"]} : dispatch 2024-09-21T12:20:02.847 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:02 smithi183 ceph-mon[33277]: osdmap e52: 8 total, 7 up, 8 in 2024-09-21T12:20:02.847 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:02 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T12:20:02.847 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:02 smithi183 ceph-mon[33277]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi183", "root=default"]} : dispatch 2024-09-21T12:20:02.847 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:02 smithi183 ceph-mon[33277]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:20:02.847 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:02 smithi183 ceph-mon[33277]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-09-21T12:20:02.847 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:02 smithi183 ceph-mon[33277]: osdmap e53: 8 total, 7 up, 8 in 2024-09-21T12:20:02.950 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:02 smithi114 ceph-mon[30324]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T12:20:02.950 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:02 smithi114 ceph-mon[30324]: from='osd.7 [v2:172.21.15.183:6824/3203337442,v1:172.21.15.183:6825/3203337442]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi183", "root=default"]} : dispatch 2024-09-21T12:20:02.950 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:02 smithi114 ceph-mon[30324]: osdmap e52: 8 total, 7 up, 8 in 2024-09-21T12:20:02.950 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:02 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T12:20:02.950 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:02 smithi114 ceph-mon[30324]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi183", "root=default"]} : dispatch 2024-09-21T12:20:02.951 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:02 smithi114 ceph-mon[30324]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:20:02.951 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:02 smithi114 ceph-mon[30324]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-09-21T12:20:02.951 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:02 smithi114 ceph-mon[30324]: osdmap e53: 8 total, 7 up, 8 in 2024-09-21T12:20:02.951 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:02 smithi114 ceph-mon[40867]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T12:20:02.951 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:02 smithi114 ceph-mon[40867]: from='osd.7 [v2:172.21.15.183:6824/3203337442,v1:172.21.15.183:6825/3203337442]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi183", "root=default"]} : dispatch 2024-09-21T12:20:02.951 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:02 smithi114 ceph-mon[40867]: osdmap e52: 8 total, 7 up, 8 in 2024-09-21T12:20:02.951 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:02 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T12:20:02.951 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:02 smithi114 ceph-mon[40867]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi183", "root=default"]} : dispatch 2024-09-21T12:20:02.951 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:02 smithi114 ceph-mon[40867]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:20:02.951 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:02 smithi114 ceph-mon[40867]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-09-21T12:20:02.951 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:02 smithi114 ceph-mon[40867]: osdmap e53: 8 total, 7 up, 8 in 2024-09-21T12:20:03.441 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:20:03 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:20:03.157+0000 7f2398dd6640 -1 osd.7 0 waiting for initial osdmap 2024-09-21T12:20:03.441 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:20:03 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:20:03.163+0000 7f2390bf5640 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T12:20:03.524 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:20:03.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:03 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T12:20:03.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:03 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T12:20:03.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:03 smithi114 ceph-mon[30324]: from='osd.7 ' entity='osd.7' 2024-09-21T12:20:03.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:03 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1187187020' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T12:20:03.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:03 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T12:20:03.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:03 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T12:20:03.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:03 smithi114 ceph-mon[40867]: from='osd.7 ' entity='osd.7' 2024-09-21T12:20:03.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:03 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1187187020' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T12:20:03.968 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:03 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T12:20:03.968 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:03 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T12:20:03.969 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:03 smithi183 ceph-mon[33277]: from='osd.7 ' entity='osd.7' 2024-09-21T12:20:03.969 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:03 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1187187020' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T12:20:04.309 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":53,"num_osds":8,"num_up_osds":7,"osd_up_since":1726921172,"num_in_osds":8,"osd_in_since":1726921180,"num_remapped_pgs":0} 2024-09-21T12:20:04.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:04 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:20:04.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:04 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:20:04.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:04 smithi183 ceph-mon[33277]: osd.7 [v2:172.21.15.183:6824/3203337442,v1:172.21.15.183:6825/3203337442] boot 2024-09-21T12:20:04.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:04 smithi183 ceph-mon[33277]: osdmap e54: 8 total, 8 up, 8 in 2024-09-21T12:20:04.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:04 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T12:20:04.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:04 smithi183 ceph-mon[33277]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:20:04.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:04 smithi183 ceph-mon[33277]: osdmap e55: 8 total, 8 up, 8 in 2024-09-21T12:20:05.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:04 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:20:05.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:04 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:20:05.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:04 smithi114 ceph-mon[30324]: osd.7 [v2:172.21.15.183:6824/3203337442,v1:172.21.15.183:6825/3203337442] boot 2024-09-21T12:20:05.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:04 smithi114 ceph-mon[30324]: osdmap e54: 8 total, 8 up, 8 in 2024-09-21T12:20:05.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:04 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T12:20:05.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:04 smithi114 ceph-mon[30324]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:20:05.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:04 smithi114 ceph-mon[30324]: osdmap e55: 8 total, 8 up, 8 in 2024-09-21T12:20:05.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:04 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:20:05.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:04 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:20:05.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:04 smithi114 ceph-mon[40867]: osd.7 [v2:172.21.15.183:6824/3203337442,v1:172.21.15.183:6825/3203337442] boot 2024-09-21T12:20:05.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:04 smithi114 ceph-mon[40867]: osdmap e54: 8 total, 8 up, 8 in 2024-09-21T12:20:05.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:04 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T12:20:05.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:04 smithi114 ceph-mon[40867]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:20:05.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:04 smithi114 ceph-mon[40867]: osdmap e55: 8 total, 8 up, 8 in 2024-09-21T12:20:05.310 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd stat -f json 2024-09-21T12:20:05.568 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:20:06.215 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:20:06.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:06 smithi114 ceph-mon[30324]: Detected new or changed devices on smithi183 2024-09-21T12:20:06.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:06 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:06.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:06 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:06.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:06 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:06.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:06 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:06.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:06 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:06.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:06 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:06.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:06 smithi114 ceph-mon[30324]: Adjusting osd_memory_target on smithi183 to 4272M 2024-09-21T12:20:06.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:06 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:06.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:06 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:20:06.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:06 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:20:06.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:06 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:06.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:06 smithi114 ceph-mon[30324]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:20:06.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:06 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1158114793' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T12:20:06.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:06 smithi114 ceph-mon[40867]: Detected new or changed devices on smithi183 2024-09-21T12:20:06.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:06 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:06.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:06 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:06.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:06 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:06.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:06 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:06.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:06 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:06.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:06 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:06.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:06 smithi114 ceph-mon[40867]: Adjusting osd_memory_target on smithi183 to 4272M 2024-09-21T12:20:06.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:06 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:06.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:06 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:20:06.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:06 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:20:06.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:06 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:06.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:06 smithi114 ceph-mon[40867]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:20:06.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:06 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1158114793' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T12:20:06.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:06 smithi183 ceph-mon[33277]: Detected new or changed devices on smithi183 2024-09-21T12:20:06.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:06 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:06.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:06 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:06.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:06 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:06.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:06 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:06.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:06 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:06.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:06 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:06.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:06 smithi183 ceph-mon[33277]: Adjusting osd_memory_target on smithi183 to 4272M 2024-09-21T12:20:06.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:06 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:06.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:06 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:20:06.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:06 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:20:06.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:06 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:06.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:06 smithi183 ceph-mon[33277]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:20:06.740 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:06 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1158114793' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T12:20:07.013 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":55,"num_osds":8,"num_up_osds":8,"osd_up_since":1726921203,"num_in_osds":8,"osd_in_since":1726921180,"num_remapped_pgs":0} 2024-09-21T12:20:07.013 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T12:20:07.257 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:20:07.619 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:07 smithi114 ceph-mon[40867]: osdmap e56: 8 total, 8 up, 8 in 2024-09-21T12:20:07.619 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:07 smithi114 ceph-mon[30324]: osdmap e56: 8 total, 8 up, 8 in 2024-09-21T12:20:07.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:07 smithi183 ceph-mon[33277]: osdmap e56: 8 total, 8 up, 8 in 2024-09-21T12:20:07.886 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:20:07.887 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":56,"fsid":"e2ad975c-7812-11ef-baf6-efdc52797490","created":"2024-09-21T12:14:24.181227+0000","modified":"2024-09-21T12:20:06.334396+0000","last_up_change":"2024-09-21T12:20:03.702987+0000","last_in_change":"2024-09-21T12:19:40.981543+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T12:17:38.357647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5715e521-c7dc-41dd-a075-5c96a88fa9b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3555869816},{"type":"v1","addr":"172.21.15.114:6803","nonce":3555869816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3555869816},{"type":"v1","addr":"172.21.15.114:6805","nonce":3555869816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3555869816},{"type":"v1","addr":"172.21.15.114:6809","nonce":3555869816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3555869816},{"type":"v1","addr":"172.21.15.114:6807","nonce":3555869816}]},"public_addr":"172.21.15.114:6803/3555869816","cluster_addr":"172.21.15.114:6805/3555869816","heartbeat_back_addr":"172.21.15.114:6809/3555869816","heartbeat_front_addr":"172.21.15.114:6807/3555869816","state":["exists","up"]},{"osd":1,"uuid":"1fe160a7-5972-4e3e-99ce-15e2311ba4c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":2744663863},{"type":"v1","addr":"172.21.15.114:6811","nonce":2744663863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":2744663863},{"type":"v1","addr":"172.21.15.114:6813","nonce":2744663863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":2744663863},{"type":"v1","addr":"172.21.15.114:6817","nonce":2744663863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":2744663863},{"type":"v1","addr":"172.21.15.114:6815","nonce":2744663863}]},"public_addr":"172.21.15.114:6811/2744663863","cluster_addr":"172.21.15.114:6813/2744663863","heartbeat_back_addr":"172.21.15.114:6817/2744663863","heartbeat_front_addr":"172.21.15.114:6815/2744663863","state":["exists","up"]},{"osd":2,"uuid":"6a63581f-7603-49ae-9ab3-b37e7c91ab91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3740045173},{"type":"v1","addr":"172.21.15.114:6819","nonce":3740045173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3740045173},{"type":"v1","addr":"172.21.15.114:6821","nonce":3740045173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":3740045173},{"type":"v1","addr":"172.21.15.114:6825","nonce":3740045173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3740045173},{"type":"v1","addr":"172.21.15.114:6823","nonce":3740045173}]},"public_addr":"172.21.15.114:6819/3740045173","cluster_addr":"172.21.15.114:6821/3740045173","heartbeat_back_addr":"172.21.15.114:6825/3740045173","heartbeat_front_addr":"172.21.15.114:6823/3740045173","state":["exists","up"]},{"osd":3,"uuid":"0ad68cdb-ae5c-4090-8a8d-b67c1177ae42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":1312590318},{"type":"v1","addr":"172.21.15.114:6827","nonce":1312590318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":1312590318},{"type":"v1","addr":"172.21.15.114:6829","nonce":1312590318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1312590318},{"type":"v1","addr":"172.21.15.114:6833","nonce":1312590318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":1312590318},{"type":"v1","addr":"172.21.15.114:6831","nonce":1312590318}]},"public_addr":"172.21.15.114:6827/1312590318","cluster_addr":"172.21.15.114:6829/1312590318","heartbeat_back_addr":"172.21.15.114:6833/1312590318","heartbeat_front_addr":"172.21.15.114:6831/1312590318","state":["exists","up"]},{"osd":4,"uuid":"26bb8a84-e41d-4425-8491-2dea9764b7cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":1391644836},{"type":"v1","addr":"172.21.15.183:6801","nonce":1391644836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":1391644836},{"type":"v1","addr":"172.21.15.183:6803","nonce":1391644836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":1391644836},{"type":"v1","addr":"172.21.15.183:6807","nonce":1391644836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":1391644836},{"type":"v1","addr":"172.21.15.183:6805","nonce":1391644836}]},"public_addr":"172.21.15.183:6801/1391644836","cluster_addr":"172.21.15.183:6803/1391644836","heartbeat_back_addr":"172.21.15.183:6807/1391644836","heartbeat_front_addr":"172.21.15.183:6805/1391644836","state":["exists","up"]},{"osd":5,"uuid":"33f96871-0869-4281-82b9-bd518987c551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":3187643473},{"type":"v1","addr":"172.21.15.183:6809","nonce":3187643473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":3187643473},{"type":"v1","addr":"172.21.15.183:6811","nonce":3187643473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":3187643473},{"type":"v1","addr":"172.21.15.183:6815","nonce":3187643473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":3187643473},{"type":"v1","addr":"172.21.15.183:6813","nonce":3187643473}]},"public_addr":"172.21.15.183:6809/3187643473","cluster_addr":"172.21.15.183:6811/3187643473","heartbeat_back_addr":"172.21.15.183:6815/3187643473","heartbeat_front_addr":"172.21.15.183:6813/3187643473","state":["exists","up"]},{"osd":6,"uuid":"e09c72b8-ad5a-4779-a874-b149fce6ed70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3025199631},{"type":"v1","addr":"172.21.15.183:6817","nonce":3025199631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3025199631},{"type":"v1","addr":"172.21.15.183:6819","nonce":3025199631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":3025199631},{"type":"v1","addr":"172.21.15.183:6823","nonce":3025199631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3025199631},{"type":"v1","addr":"172.21.15.183:6821","nonce":3025199631}]},"public_addr":"172.21.15.183:6817/3025199631","cluster_addr":"172.21.15.183:6819/3025199631","heartbeat_back_addr":"172.21.15.183:6823/3025199631","heartbeat_front_addr":"172.21.15.183:6821/3025199631","state":["exists","up"]},{"osd":7,"uuid":"1a24194d-1af6-485f-8856-98844a02dd9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3203337442},{"type":"v1","addr":"172.21.15.183:6825","nonce":3203337442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":3203337442},{"type":"v1","addr":"172.21.15.183:6827","nonce":3203337442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3203337442},{"type":"v1","addr":"172.21.15.183:6831","nonce":3203337442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3203337442},{"type":"v1","addr":"172.21.15.183:6829","nonce":3203337442}]},"public_addr":"172.21.15.183:6825/3203337442","cluster_addr":"172.21.15.183:6827/3203337442","heartbeat_back_addr":"172.21.15.183:6831/3203337442","heartbeat_front_addr":"172.21.15.183:6829/3203337442","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:19:04.941476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:19:30.681663+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:18:53.611331+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:19:24.512765+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:19:37.016906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:19:01.831283+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:19:31.112105+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:20:02.689705+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6801/770932721":"2024-09-22T12:15:20.045217+0000","172.21.15.114:6800/770932721":"2024-09-22T12:15:20.045217+0000","172.21.15.114:0/139803357":"2024-09-22T12:15:20.045217+0000","172.21.15.114:0/2106467614":"2024-09-22T12:15:20.045217+0000","172.21.15.114:6801/1725148595":"2024-09-22T12:14:53.998653+0000","172.21.15.114:0/442404805":"2024-09-22T12:14:53.998653+0000","172.21.15.114:6800/1725148595":"2024-09-22T12:14:53.998653+0000","172.21.15.114:0/2620151541":"2024-09-22T12:15:20.045217+0000","172.21.15.114:0/3416259228":"2024-09-22T12:14:53.998653+0000","172.21.15.114:0/898990943":"2024-09-22T12:14:53.998653+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T12:20:08.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:08 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4081931690' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T12:20:08.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:08 smithi114 ceph-mon[30324]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:20:08.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:08 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/4081931690' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T12:20:08.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:08 smithi114 ceph-mon[40867]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:20:08.671 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T12:17:38.357647+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-21T12:20:08.672 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pool get .mgr pg_num 2024-09-21T12:20:08.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:08 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/4081931690' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T12:20:08.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:08 smithi183 ceph-mon[33277]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:20:08.919 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:20:09.564 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-09-21T12:20:09.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:09 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3542237687' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T12:20:09.873 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:09 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3542237687' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T12:20:09.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:09 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3542237687' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T12:20:10.368 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi114 2024-09-21T12:20:10.368 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph orch apply rgw foo.a --placement '1;smithi114=foo.a' 2024-09-21T12:20:10.629 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:20:10.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:10 smithi183 ceph-mon[33277]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:20:11.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:10 smithi114 ceph-mon[30324]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:20:11.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:10 smithi114 ceph-mon[40867]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:20:11.258 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled rgw.foo.a update... 2024-09-21T12:20:11.900 DEBUG:teuthology.orchestra.run.smithi114:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@rgw.foo.a.service 2024-09-21T12:20:11.903 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi183 2024-09-21T12:20:11.903 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pool create datapool 3 3 replicated 2024-09-21T12:20:12.147 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:20:12.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:12 smithi114 ceph-mon[30324]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi114=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:20:12.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:12 smithi114 ceph-mon[30324]: Saving service rgw.foo.a spec with placement smithi114=foo.a;count:1 2024-09-21T12:20:12.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:12 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:12.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:12 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:20:12.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:12 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:20:12.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:12 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:20:12.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:12 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:12.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:12 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-21T12:20:12.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:12 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-21T12:20:12.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:12 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:12.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:12 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:20:12.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:12 smithi114 ceph-mon[30324]: Deploying daemon rgw.foo.a on smithi114 2024-09-21T12:20:12.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:12 smithi114 ceph-mon[30324]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:20:12.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:12 smithi114 ceph-mon[40867]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi114=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:20:12.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:12 smithi114 ceph-mon[40867]: Saving service rgw.foo.a spec with placement smithi114=foo.a;count:1 2024-09-21T12:20:12.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:12 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:12.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:12 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:20:12.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:12 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:20:12.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:12 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:20:12.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:12 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:12.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:12 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-21T12:20:12.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:12 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-21T12:20:12.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:12 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:12.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:12 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:20:12.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:12 smithi114 ceph-mon[40867]: Deploying daemon rgw.foo.a on smithi114 2024-09-21T12:20:12.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:12 smithi114 ceph-mon[40867]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:20:12.591 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:12 smithi183 ceph-mon[33277]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi114=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:20:12.591 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:12 smithi183 ceph-mon[33277]: Saving service rgw.foo.a spec with placement smithi114=foo.a;count:1 2024-09-21T12:20:12.591 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:12 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:12.591 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:12 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:20:12.592 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:12 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:20:12.592 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:12 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:20:12.592 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:12 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:12.592 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:12 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-21T12:20:12.592 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:12 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-21T12:20:12.592 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:12 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:12.592 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:12 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:20:12.592 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:12 smithi183 ceph-mon[33277]: Deploying daemon rgw.foo.a on smithi114 2024-09-21T12:20:12.592 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:12 smithi183 ceph-mon[33277]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:20:13.281 INFO:teuthology.orchestra.run.smithi183.stderr:pool 'datapool' created 2024-09-21T12:20:13.443 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:13 smithi114 ceph-mon[40867]: from='client.? 172.21.15.183:0/2470226272' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-21T12:20:13.443 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:13 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-21T12:20:13.443 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:13 smithi114 ceph-mon[30324]: from='client.? 172.21.15.183:0/2470226272' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-21T12:20:13.443 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:13 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-21T12:20:13.578 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:13 smithi183 ceph-mon[33277]: from='client.? 172.21.15.183:0/2470226272' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-21T12:20:13.578 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:13 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-21T12:20:13.762 INFO:journalctl@ceph.rgw.foo.a.smithi114.stdout:Sep 21 12:20:13 smithi114 podman[67486]: 2024-09-21 12:20:13.674521025 +0000 UTC m=+0.340164908 container start c3e2161c3c5d822b12fc83776b098fd2bedb1a447c0de82cf6be9a569b0416f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-rgw-foo-a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3) 2024-09-21T12:20:13.763 INFO:journalctl@ceph.rgw.foo.a.smithi114.stdout:Sep 21 12:20:13 smithi114 bash[67486]: c3e2161c3c5d822b12fc83776b098fd2bedb1a447c0de82cf6be9a569b0416f3 2024-09-21T12:20:13.872 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- rbd pool init datapool 2024-09-21T12:20:14.024 INFO:journalctl@ceph.rgw.foo.a.smithi114.stdout:Sep 21 12:20:13 smithi114 systemd[1]: Started Ceph rgw.foo.a for e2ad975c-7812-11ef-baf6-efdc52797490. 2024-09-21T12:20:14.116 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:20:14.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:14 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-21T12:20:14.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:14 smithi183 ceph-mon[33277]: osdmap e57: 8 total, 8 up, 8 in 2024-09-21T12:20:14.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:14 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:14.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:14 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:14.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:14 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:14.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:14 smithi183 ceph-mon[33277]: Saving service rgw.foo.a spec with placement smithi114=foo.a;count:1 2024-09-21T12:20:14.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:14 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:14.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:14 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:14.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:14 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:20:14.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:14 smithi183 ceph-mon[33277]: pgmap v194: 4 pgs: 1 creating+peering, 1 active+clean, 2 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:20:14.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:14 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:20:14.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:14 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:20:14.617 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:14 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-21T12:20:14.617 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:14 smithi114 ceph-mon[40867]: osdmap e57: 8 total, 8 up, 8 in 2024-09-21T12:20:14.617 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:14 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:14.618 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:14 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:14.618 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:14 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:14.618 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:14 smithi114 ceph-mon[40867]: Saving service rgw.foo.a spec with placement smithi114=foo.a;count:1 2024-09-21T12:20:14.618 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:14 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:14.618 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:14 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:14.618 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:14 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:20:14.618 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:14 smithi114 ceph-mon[40867]: pgmap v194: 4 pgs: 1 creating+peering, 1 active+clean, 2 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:20:14.618 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:14 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:20:14.618 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:14 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:20:14.618 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:14 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-21T12:20:14.618 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:14 smithi114 ceph-mon[30324]: osdmap e57: 8 total, 8 up, 8 in 2024-09-21T12:20:14.618 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:14 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:14.619 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:14 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:14.619 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:14 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:14.619 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:14 smithi114 ceph-mon[30324]: Saving service rgw.foo.a spec with placement smithi114=foo.a;count:1 2024-09-21T12:20:14.619 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:14 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:14.619 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:14 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:14.619 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:14 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:20:14.619 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:14 smithi114 ceph-mon[30324]: pgmap v194: 4 pgs: 1 creating+peering, 1 active+clean, 2 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:20:14.619 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:14 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:20:14.619 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:14 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:20:15.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:15 smithi183 ceph-mon[33277]: osdmap e58: 8 total, 8 up, 8 in 2024-09-21T12:20:15.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:15 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3744219554' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-21T12:20:15.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:15 smithi183 ceph-mon[33277]: from='client.? 172.21.15.183:0/619235967' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-21T12:20:15.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:15 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-21T12:20:15.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:15 smithi183 ceph-mon[33277]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T12:20:15.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:15 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:15.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:15 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:15.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:15 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:20:15.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:15 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:20:15.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:15 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3744219554' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-21T12:20:15.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:15 smithi114 ceph-mon[40867]: osdmap e58: 8 total, 8 up, 8 in 2024-09-21T12:20:15.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:15 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3744219554' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-21T12:20:15.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:15 smithi114 ceph-mon[40867]: from='client.? 172.21.15.183:0/619235967' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-21T12:20:15.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:15 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-21T12:20:15.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:15 smithi114 ceph-mon[40867]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T12:20:15.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:15 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:15.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:15 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:15.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:15 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:20:15.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:15 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:20:15.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:15 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3744219554' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-21T12:20:15.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:15 smithi114 ceph-mon[30324]: osdmap e58: 8 total, 8 up, 8 in 2024-09-21T12:20:15.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:15 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3744219554' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-21T12:20:15.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:15 smithi114 ceph-mon[30324]: from='client.? 172.21.15.183:0/619235967' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-21T12:20:15.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:15 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-21T12:20:15.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:15 smithi114 ceph-mon[30324]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T12:20:15.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:15 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:15.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:15 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:15.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:15 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:20:15.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:15 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:20:15.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:15 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3744219554' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-21T12:20:16.591 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:16 smithi183 ceph-mon[33277]: Checking dashboard <-> RGW credentials 2024-09-21T12:20:16.592 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:16 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-21T12:20:16.592 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:16 smithi183 ceph-mon[33277]: osdmap e59: 8 total, 8 up, 8 in 2024-09-21T12:20:16.592 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:16 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:16.592 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:16 smithi183 ceph-mon[33277]: pgmap v197: 36 pgs: 1 creating+peering, 1 active+clean, 34 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:20:16.592 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:16 smithi183 ceph-mon[33277]: osdmap e60: 8 total, 8 up, 8 in 2024-09-21T12:20:16.592 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:16 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1949624628' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-21T12:20:16.592 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:16 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1371625176' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-21T12:20:16.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:16 smithi114 ceph-mon[30324]: Checking dashboard <-> RGW credentials 2024-09-21T12:20:16.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:16 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-21T12:20:16.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:16 smithi114 ceph-mon[30324]: osdmap e59: 8 total, 8 up, 8 in 2024-09-21T12:20:16.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:16 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:16.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:16 smithi114 ceph-mon[30324]: pgmap v197: 36 pgs: 1 creating+peering, 1 active+clean, 34 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:20:16.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:16 smithi114 ceph-mon[30324]: osdmap e60: 8 total, 8 up, 8 in 2024-09-21T12:20:16.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:16 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1949624628' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-21T12:20:16.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:16 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1371625176' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-21T12:20:16.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:16 smithi114 ceph-mon[40867]: Checking dashboard <-> RGW credentials 2024-09-21T12:20:16.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:16 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-21T12:20:16.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:16 smithi114 ceph-mon[40867]: osdmap e59: 8 total, 8 up, 8 in 2024-09-21T12:20:16.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:16 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:16.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:16 smithi114 ceph-mon[40867]: pgmap v197: 36 pgs: 1 creating+peering, 1 active+clean, 34 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:20:16.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:16 smithi114 ceph-mon[40867]: osdmap e60: 8 total, 8 up, 8 in 2024-09-21T12:20:16.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:16 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1949624628' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-21T12:20:16.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:16 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1371625176' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-21T12:20:17.603 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:17 smithi183 ceph-mon[33277]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-21T12:20:17.603 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:17 smithi183 ceph-mon[33277]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T12:20:17.603 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:17 smithi183 ceph-mon[33277]: Cluster is now healthy 2024-09-21T12:20:17.603 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:17 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1371625176' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-21T12:20:17.603 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:17 smithi183 ceph-mon[33277]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-21T12:20:17.603 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:17 smithi183 ceph-mon[33277]: osdmap e61: 8 total, 8 up, 8 in 2024-09-21T12:20:17.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:17 smithi114 ceph-mon[30324]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-21T12:20:17.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:17 smithi114 ceph-mon[30324]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T12:20:17.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:17 smithi114 ceph-mon[30324]: Cluster is now healthy 2024-09-21T12:20:17.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:17 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1371625176' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-21T12:20:17.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:17 smithi114 ceph-mon[30324]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-21T12:20:17.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:17 smithi114 ceph-mon[30324]: osdmap e61: 8 total, 8 up, 8 in 2024-09-21T12:20:17.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:17 smithi114 ceph-mon[40867]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-21T12:20:17.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:17 smithi114 ceph-mon[40867]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T12:20:17.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:17 smithi114 ceph-mon[40867]: Cluster is now healthy 2024-09-21T12:20:17.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:17 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1371625176' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-21T12:20:17.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:17 smithi114 ceph-mon[40867]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-21T12:20:17.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:17 smithi114 ceph-mon[40867]: osdmap e61: 8 total, 8 up, 8 in 2024-09-21T12:20:17.988 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.183 --placement '1;smithi183=iscsi.a' 2024-09-21T12:20:18.239 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:20:18.573 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:18 smithi183 ceph-mon[33277]: pgmap v200: 68 pgs: 13 creating+peering, 13 active+clean, 42 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-21T12:20:18.573 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:18 smithi183 ceph-mon[33277]: osdmap e62: 8 total, 8 up, 8 in 2024-09-21T12:20:18.573 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:18 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1371625176' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-21T12:20:18.573 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:18 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1949624628' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-21T12:20:18.574 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:18 smithi183 ceph-mon[33277]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-21T12:20:18.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:18 smithi114 ceph-mon[30324]: pgmap v200: 68 pgs: 13 creating+peering, 13 active+clean, 42 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-21T12:20:18.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:18 smithi114 ceph-mon[30324]: osdmap e62: 8 total, 8 up, 8 in 2024-09-21T12:20:18.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:18 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1371625176' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-21T12:20:18.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:18 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1949624628' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-21T12:20:18.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:18 smithi114 ceph-mon[30324]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-21T12:20:18.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:18 smithi114 ceph-mon[40867]: pgmap v200: 68 pgs: 13 creating+peering, 13 active+clean, 42 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-21T12:20:18.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:18 smithi114 ceph-mon[40867]: osdmap e62: 8 total, 8 up, 8 in 2024-09-21T12:20:18.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:18 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1371625176' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-21T12:20:18.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:18 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1949624628' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-21T12:20:18.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:18 smithi114 ceph-mon[40867]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-21T12:20:18.869 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled iscsi.datapool update... 2024-09-21T12:20:19.538 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-09-21T12:20:19.539 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-21T12:20:19.539 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-21T12:20:19.569 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-09-21T12:20:19.570 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-21T12:20:19.595 DEBUG:teuthology.orchestra.run.smithi183:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@iscsi.iscsi.a.service 2024-09-21T12:20:19.637 INFO:tasks.cephadm:Adding prometheus.a on smithi183 2024-09-21T12:20:19.638 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph orch apply prometheus '1;smithi183=a' 2024-09-21T12:20:19.917 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:20:20.146 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:19 smithi183 ceph-mon[33277]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.183", "placement": "1;smithi183=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:20:20.146 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:19 smithi183 ceph-mon[33277]: Saving service iscsi.datapool spec with placement smithi183=iscsi.a;count:1 2024-09-21T12:20:20.146 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:19 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:20.146 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:19 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1371625176' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-21T12:20:20.146 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:19 smithi183 ceph-mon[33277]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-21T12:20:20.146 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:19 smithi183 ceph-mon[33277]: osdmap e63: 8 total, 8 up, 8 in 2024-09-21T12:20:20.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:19 smithi114 ceph-mon[40867]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.183", "placement": "1;smithi183=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:20:20.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:19 smithi114 ceph-mon[40867]: Saving service iscsi.datapool spec with placement smithi183=iscsi.a;count:1 2024-09-21T12:20:20.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:19 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:20.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:19 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1371625176' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-21T12:20:20.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:19 smithi114 ceph-mon[40867]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-21T12:20:20.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:19 smithi114 ceph-mon[40867]: osdmap e63: 8 total, 8 up, 8 in 2024-09-21T12:20:20.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:19 smithi114 ceph-mon[30324]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.183", "placement": "1;smithi183=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:20:20.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:19 smithi114 ceph-mon[30324]: Saving service iscsi.datapool spec with placement smithi183=iscsi.a;count:1 2024-09-21T12:20:20.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:19 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:20.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:19 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1371625176' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-21T12:20:20.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:19 smithi114 ceph-mon[30324]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-21T12:20:20.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:19 smithi114 ceph-mon[30324]: osdmap e63: 8 total, 8 up, 8 in 2024-09-21T12:20:20.583 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled prometheus update... 2024-09-21T12:20:21.128 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:20 smithi183 ceph-mon[33277]: pgmap v203: 100 pgs: 28 creating+peering, 56 active+clean, 16 unknown; 578 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-09-21T12:20:21.128 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:20 smithi183 ceph-mon[33277]: osdmap e64: 8 total, 8 up, 8 in 2024-09-21T12:20:21.128 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:20 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1949624628' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-21T12:20:21.128 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:20 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1371625176' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-21T12:20:21.128 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:20 smithi183 ceph-mon[33277]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-21T12:20:21.129 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:20 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:21.155 DEBUG:teuthology.orchestra.run.smithi183:prometheus.a> sudo journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@prometheus.a.service 2024-09-21T12:20:21.157 INFO:tasks.cephadm:Adding node-exporter.a on smithi114 2024-09-21T12:20:21.157 INFO:tasks.cephadm:Adding node-exporter.b on smithi183 2024-09-21T12:20:21.157 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph orch apply node-exporter '2;smithi114=a;smithi183=b' 2024-09-21T12:20:21.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:20 smithi114 ceph-mon[40867]: pgmap v203: 100 pgs: 28 creating+peering, 56 active+clean, 16 unknown; 578 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-09-21T12:20:21.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:20 smithi114 ceph-mon[40867]: osdmap e64: 8 total, 8 up, 8 in 2024-09-21T12:20:21.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:20 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1949624628' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-21T12:20:21.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:20 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1371625176' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-21T12:20:21.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:20 smithi114 ceph-mon[40867]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-21T12:20:21.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:20 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:21.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:20 smithi114 ceph-mon[30324]: pgmap v203: 100 pgs: 28 creating+peering, 56 active+clean, 16 unknown; 578 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-09-21T12:20:21.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:20 smithi114 ceph-mon[30324]: osdmap e64: 8 total, 8 up, 8 in 2024-09-21T12:20:21.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:20 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1949624628' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-21T12:20:21.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:20 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1371625176' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-21T12:20:21.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:20 smithi114 ceph-mon[30324]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-21T12:20:21.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:20 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:21.445 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:20:22.178 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled node-exporter update... 2024-09-21T12:20:22.178 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:21 smithi183 ceph-mon[33277]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi183=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:20:22.179 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:21 smithi183 ceph-mon[33277]: Saving service prometheus spec with placement smithi183=a;count:1 2024-09-21T12:20:22.179 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:21 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1371625176' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-21T12:20:22.179 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:21 smithi183 ceph-mon[33277]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-21T12:20:22.179 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:21 smithi183 ceph-mon[33277]: osdmap e65: 8 total, 8 up, 8 in 2024-09-21T12:20:22.179 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:21 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1371625176' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-21T12:20:22.179 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:21 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1949624628' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-21T12:20:22.179 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:21 smithi183 ceph-mon[33277]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-21T12:20:22.179 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:21 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1371625176' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-21T12:20:22.179 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:21 smithi183 ceph-mon[33277]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-21T12:20:22.179 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:21 smithi183 ceph-mon[33277]: osdmap e66: 8 total, 8 up, 8 in 2024-09-21T12:20:22.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:21 smithi114 ceph-mon[30324]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi183=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:20:22.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:21 smithi114 ceph-mon[30324]: Saving service prometheus spec with placement smithi183=a;count:1 2024-09-21T12:20:22.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:21 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1371625176' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-21T12:20:22.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:21 smithi114 ceph-mon[30324]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-21T12:20:22.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:21 smithi114 ceph-mon[30324]: osdmap e65: 8 total, 8 up, 8 in 2024-09-21T12:20:22.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:21 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1371625176' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-21T12:20:22.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:21 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1949624628' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-21T12:20:22.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:21 smithi114 ceph-mon[30324]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-21T12:20:22.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:21 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1371625176' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-21T12:20:22.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:21 smithi114 ceph-mon[30324]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-21T12:20:22.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:21 smithi114 ceph-mon[30324]: osdmap e66: 8 total, 8 up, 8 in 2024-09-21T12:20:22.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:21 smithi114 ceph-mon[40867]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi183=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:20:22.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:21 smithi114 ceph-mon[40867]: Saving service prometheus spec with placement smithi183=a;count:1 2024-09-21T12:20:22.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:21 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1371625176' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-21T12:20:22.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:21 smithi114 ceph-mon[40867]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-21T12:20:22.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:21 smithi114 ceph-mon[40867]: osdmap e65: 8 total, 8 up, 8 in 2024-09-21T12:20:22.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:21 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1371625176' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-21T12:20:22.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:21 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1949624628' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-21T12:20:22.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:21 smithi114 ceph-mon[40867]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-21T12:20:22.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:21 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1371625176' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-21T12:20:22.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:21 smithi114 ceph-mon[40867]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-21T12:20:22.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:21 smithi114 ceph-mon[40867]: osdmap e66: 8 total, 8 up, 8 in 2024-09-21T12:20:22.753 DEBUG:teuthology.orchestra.run.smithi114:node-exporter.a> sudo journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@node-exporter.a.service 2024-09-21T12:20:22.756 DEBUG:teuthology.orchestra.run.smithi183:node-exporter.b> sudo journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@node-exporter.b.service 2024-09-21T12:20:22.758 INFO:tasks.cephadm:Adding alertmanager.a on smithi114 2024-09-21T12:20:22.759 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph orch apply alertmanager '1;smithi114=a' 2024-09-21T12:20:23.033 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:22 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:23.034 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:22 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:23.034 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:22 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:23.034 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:22 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-21T12:20:23.034 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:22 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-21T12:20:23.034 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:22 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:20:23.034 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:22 smithi183 ceph-mon[33277]: Deploying daemon iscsi.iscsi.a on smithi183 2024-09-21T12:20:23.034 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:22 smithi183 ceph-mon[33277]: pgmap v207: 132 pgs: 21 creating+peering, 74 active+clean, 37 unknown; 579 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 2.6 KiB/s wr, 44 op/s 2024-09-21T12:20:23.034 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:22 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:23.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:22 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:23.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:22 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:23.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:22 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:23.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:22 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-21T12:20:23.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:22 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-21T12:20:23.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:22 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:20:23.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:22 smithi114 ceph-mon[40867]: Deploying daemon iscsi.iscsi.a on smithi183 2024-09-21T12:20:23.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:22 smithi114 ceph-mon[40867]: pgmap v207: 132 pgs: 21 creating+peering, 74 active+clean, 37 unknown; 579 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 2.6 KiB/s wr, 44 op/s 2024-09-21T12:20:23.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:22 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:23.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:22 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:23.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:22 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:23.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:22 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:23.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:22 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-21T12:20:23.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:22 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-21T12:20:23.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:22 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:20:23.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:22 smithi114 ceph-mon[30324]: Deploying daemon iscsi.iscsi.a on smithi183 2024-09-21T12:20:23.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:22 smithi114 ceph-mon[30324]: pgmap v207: 132 pgs: 21 creating+peering, 74 active+clean, 37 unknown; 579 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 2.6 KiB/s wr, 44 op/s 2024-09-21T12:20:23.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:22 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:23.502 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:20:24.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:23 smithi114 ceph-mon[30324]: from='client.14526 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi114=a;smithi183=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:20:24.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:23 smithi114 ceph-mon[30324]: Saving service node-exporter spec with placement smithi114=a;smithi183=b;count:2 2024-09-21T12:20:24.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:23 smithi114 ceph-mon[40867]: from='client.14526 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi114=a;smithi183=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:20:24.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:23 smithi114 ceph-mon[40867]: Saving service node-exporter spec with placement smithi114=a;smithi183=b;count:2 2024-09-21T12:20:24.317 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:23 smithi183 ceph-mon[33277]: from='client.14526 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi114=a;smithi183=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:20:24.317 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:23 smithi183 ceph-mon[33277]: Saving service node-exporter spec with placement smithi114=a;smithi183=b;count:2 2024-09-21T12:20:24.651 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled alertmanager update... 2024-09-21T12:20:24.948 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:20:24 smithi183 systemd[1]: Starting Ceph iscsi.iscsi.a for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:20:25.231 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:24 smithi183 ceph-mon[33277]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 7.5 KiB/s wr, 221 op/s 2024-09-21T12:20:25.231 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:24 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:25.259 DEBUG:teuthology.orchestra.run.smithi114:alertmanager.a> sudo journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@alertmanager.a.service 2024-09-21T12:20:25.262 INFO:tasks.cephadm:Adding grafana.a on smithi183 2024-09-21T12:20:25.262 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph orch apply grafana '1;smithi183=a' 2024-09-21T12:20:25.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:24 smithi114 ceph-mon[30324]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 7.5 KiB/s wr, 221 op/s 2024-09-21T12:20:25.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:24 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:25.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:24 smithi114 ceph-mon[40867]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 7.5 KiB/s wr, 221 op/s 2024-09-21T12:20:25.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:24 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:25.488 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:20:25 smithi183 podman[60343]: 2024-09-21 12:20:25.456423365 +0000 UTC m=+0.099380397 container create a66333b3bcc4f3e5c8f2081bf5087d59688d55c891deb99605273d0c03cfd592 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a, ceph=True, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-21T12:20:25.488 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:20:25 smithi183 podman[60343]: 2024-09-21 12:20:25.372788167 +0000 UTC m=+0.015745214 image pull 8f4d013db793795fef09ff5edf96e6b42ba8cd2c9158b2f0b7a5d7c1e657c063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472 2024-09-21T12:20:25.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:20:25 smithi183 podman[60343]: 2024-09-21 12:20:25.708866029 +0000 UTC m=+0.351823062 container init a66333b3bcc4f3e5c8f2081bf5087d59688d55c891deb99605273d0c03cfd592 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-21T12:20:25.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:20:25 smithi183 podman[60343]: 2024-09-21 12:20:25.713829932 +0000 UTC m=+0.356786964 container start a66333b3bcc4f3e5c8f2081bf5087d59688d55c891deb99605273d0c03cfd592 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-21T12:20:25.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:20:25 smithi183 bash[60343]: a66333b3bcc4f3e5c8f2081bf5087d59688d55c891deb99605273d0c03cfd592 2024-09-21T12:20:25.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:20:25 smithi183 systemd[1]: Started Ceph iscsi.iscsi.a for e2ad975c-7812-11ef-baf6-efdc52797490. 2024-09-21T12:20:26.255 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:20:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug Started the configuration object watcher 2024-09-21T12:20:26.255 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:20:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug Checking for config object changes every 1s 2024-09-21T12:20:26.255 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:20:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug Processing osd blocklist entries for this node 2024-09-21T12:20:26.255 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:26 smithi183 ceph-mon[33277]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi114=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:20:26.255 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:26 smithi183 ceph-mon[33277]: Saving service alertmanager spec with placement smithi114=a;count:1 2024-09-21T12:20:26.255 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:26 smithi183 ceph-mon[33277]: 1.0 scrub starts 2024-09-21T12:20:26.255 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:26 smithi183 ceph-mon[33277]: 1.0 scrub ok 2024-09-21T12:20:26.255 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:26 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:20:26.256 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:26 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:20:26.256 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:26 smithi183 ceph-mon[33277]: osdmap e67: 8 total, 8 up, 8 in 2024-09-21T12:20:26.256 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:26 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:26.256 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:26 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:26.256 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:26 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:26.256 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:26 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:26.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:26 smithi114 ceph-mon[30324]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi114=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:20:26.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:26 smithi114 ceph-mon[30324]: Saving service alertmanager spec with placement smithi114=a;count:1 2024-09-21T12:20:26.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:26 smithi114 ceph-mon[30324]: 1.0 scrub starts 2024-09-21T12:20:26.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:26 smithi114 ceph-mon[30324]: 1.0 scrub ok 2024-09-21T12:20:26.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:26 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:20:26.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:26 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:20:26.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:26 smithi114 ceph-mon[30324]: osdmap e67: 8 total, 8 up, 8 in 2024-09-21T12:20:26.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:26 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:26.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:26 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:26.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:26 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:26.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:26 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:26.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:26 smithi114 ceph-mon[40867]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi114=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:20:26.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:26 smithi114 ceph-mon[40867]: Saving service alertmanager spec with placement smithi114=a;count:1 2024-09-21T12:20:26.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:26 smithi114 ceph-mon[40867]: 1.0 scrub starts 2024-09-21T12:20:26.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:26 smithi114 ceph-mon[40867]: 1.0 scrub ok 2024-09-21T12:20:26.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:26 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:20:26.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:26 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:20:26.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:26 smithi114 ceph-mon[40867]: osdmap e67: 8 total, 8 up, 8 in 2024-09-21T12:20:26.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:26 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:26.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:26 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:26.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:26 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:26.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:26 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:26.467 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:20:26.581 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:20:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug Reading the configuration object to update local LIO configuration 2024-09-21T12:20:26.581 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:20:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug Configuration does not have an entry for this host(smithi183.front.sepia.ceph.com) - nothing to define to LIO 2024-09-21T12:20:26.581 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:20:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: * Serving Flask app 'rbd-target-api' (lazy loading) 2024-09-21T12:20:26.581 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:20:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: * Environment: production 2024-09-21T12:20:26.581 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:20:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-21T12:20:26.581 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:20:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: Use a production WSGI server instead. 2024-09-21T12:20:26.581 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:20:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: * Debug mode: off 2024-09-21T12:20:26.582 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:20:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug * Running on all addresses. 2024-09-21T12:20:26.582 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:20:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-21T12:20:26.582 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:20:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: * Running on all addresses. 2024-09-21T12:20:26.582 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:20:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-21T12:20:26.582 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:20:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug * Running on http://[fe80::ec4:7aff:fe88:8085]:5000/ (Press CTRL+C to quit) 2024-09-21T12:20:26.582 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:20:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: * Running on http://[fe80::ec4:7aff:fe88:8085]:5000/ (Press CTRL+C to quit) 2024-09-21T12:20:27.110 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:27 smithi183 ceph-mon[33277]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-21T12:20:27.110 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:27 smithi183 ceph-mon[33277]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 6.2 KiB/s wr, 183 op/s 2024-09-21T12:20:27.110 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:27 smithi183 ceph-mon[33277]: from='client.? 172.21.15.183:0/3764647832' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T12:20:27.110 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:27 smithi183 ceph-mon[33277]: osdmap e68: 8 total, 8 up, 8 in 2024-09-21T12:20:27.184 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled grafana update... 2024-09-21T12:20:27.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:27 smithi114 ceph-mon[40867]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-21T12:20:27.343 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:27 smithi114 ceph-mon[40867]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 6.2 KiB/s wr, 183 op/s 2024-09-21T12:20:27.343 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:27 smithi114 ceph-mon[40867]: from='client.? 172.21.15.183:0/3764647832' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T12:20:27.343 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:27 smithi114 ceph-mon[40867]: osdmap e68: 8 total, 8 up, 8 in 2024-09-21T12:20:27.344 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:27 smithi114 ceph-mon[30324]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-21T12:20:27.344 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:27 smithi114 ceph-mon[30324]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 6.2 KiB/s wr, 183 op/s 2024-09-21T12:20:27.344 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:27 smithi114 ceph-mon[30324]: from='client.? 172.21.15.183:0/3764647832' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T12:20:27.344 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:27 smithi114 ceph-mon[30324]: osdmap e68: 8 total, 8 up, 8 in 2024-09-21T12:20:28.225 DEBUG:teuthology.orchestra.run.smithi183:grafana.a> sudo journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@grafana.a.service 2024-09-21T12:20:28.228 INFO:tasks.cephadm:Setting up client nodes... 2024-09-21T12:20:28.228 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-21T12:20:28.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:28 smithi114 ceph-mon[30324]: Deploying daemon prometheus.a on smithi183 2024-09-21T12:20:28.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:28 smithi114 ceph-mon[30324]: mgrmap e16: y(active, since 5m), standbys: x 2024-09-21T12:20:28.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:28 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:28.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:28 smithi114 ceph-mon[40867]: Deploying daemon prometheus.a on smithi183 2024-09-21T12:20:28.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:28 smithi114 ceph-mon[40867]: mgrmap e16: y(active, since 5m), standbys: x 2024-09-21T12:20:28.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:28 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:28.483 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:20:28.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:28 smithi183 ceph-mon[33277]: Deploying daemon prometheus.a on smithi183 2024-09-21T12:20:28.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:28 smithi183 ceph-mon[33277]: mgrmap e16: y(active, since 5m), standbys: x 2024-09-21T12:20:28.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:28 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:29.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:29 smithi114 ceph-mon[30324]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi183=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:20:29.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:29 smithi114 ceph-mon[30324]: Saving service grafana spec with placement smithi183=a;count:1 2024-09-21T12:20:29.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:29 smithi114 ceph-mon[30324]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 5.7 KiB/s wr, 177 op/s 2024-09-21T12:20:29.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:29 smithi114 ceph-mon[40867]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi183=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:20:29.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:29 smithi114 ceph-mon[40867]: Saving service grafana spec with placement smithi183=a;count:1 2024-09-21T12:20:29.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:29 smithi114 ceph-mon[40867]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 5.7 KiB/s wr, 177 op/s 2024-09-21T12:20:29.277 INFO:teuthology.orchestra.run.smithi114.stdout:[client.0] 2024-09-21T12:20:29.277 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQANuu5mOOk/EBAA7DcZCF+zODgj+8v8VK05FA== 2024-09-21T12:20:29.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:29 smithi183 ceph-mon[33277]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi183=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:20:29.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:29 smithi183 ceph-mon[33277]: Saving service grafana spec with placement smithi183=a;count:1 2024-09-21T12:20:29.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:29 smithi183 ceph-mon[33277]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 5.7 KiB/s wr, 177 op/s 2024-09-21T12:20:30.011 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-21T12:20:30.011 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-21T12:20:30.011 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-21T12:20:30.048 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-21T12:20:30.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:30 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2259431365' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T12:20:30.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:30 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2259431365' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T12:20:30.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:30 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2259431365' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T12:20:30.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:30 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2259431365' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T12:20:30.303 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:20:30.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:30 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2259431365' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T12:20:30.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:30 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2259431365' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T12:20:31.222 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:31 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:20:31.222 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:31 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:20:31.222 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:31 smithi183 ceph-mon[33277]: pgmap v213: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 3.9 KiB/s wr, 122 op/s 2024-09-21T12:20:31.222 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:31 smithi183 ceph-mon[33277]: osdmap e69: 8 total, 8 up, 8 in 2024-09-21T12:20:31.222 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:31 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:31.223 INFO:teuthology.orchestra.run.smithi183.stdout:[client.1] 2024-09-21T12:20:31.223 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQAPuu5mPf+9DBAAoEAPxXL5mc6swjkFHBJplQ== 2024-09-21T12:20:31.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:31 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:20:31.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:31 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:20:31.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:31 smithi114 ceph-mon[30324]: pgmap v213: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 3.9 KiB/s wr, 122 op/s 2024-09-21T12:20:31.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:31 smithi114 ceph-mon[30324]: osdmap e69: 8 total, 8 up, 8 in 2024-09-21T12:20:31.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:31 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:31.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:31 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:20:31.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:31 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:20:31.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:31 smithi114 ceph-mon[40867]: pgmap v213: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s rd, 3.9 KiB/s wr, 122 op/s 2024-09-21T12:20:31.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:31 smithi114 ceph-mon[40867]: osdmap e69: 8 total, 8 up, 8 in 2024-09-21T12:20:31.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:31 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:32.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:32 smithi183 ceph-mon[33277]: from='client.? 172.21.15.183:0/1188503016' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T12:20:32.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:32 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T12:20:32.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:32 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T12:20:32.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:32 smithi183 ceph-mon[33277]: osdmap e70: 8 total, 8 up, 8 in 2024-09-21T12:20:32.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:32 smithi114 ceph-mon[40867]: from='client.? 172.21.15.183:0/1188503016' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T12:20:32.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:32 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T12:20:32.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:32 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T12:20:32.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:32 smithi114 ceph-mon[40867]: osdmap e70: 8 total, 8 up, 8 in 2024-09-21T12:20:32.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:32 smithi114 ceph-mon[30324]: from='client.? 172.21.15.183:0/1188503016' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T12:20:32.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:32 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T12:20:32.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:32 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T12:20:32.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:32 smithi114 ceph-mon[30324]: osdmap e70: 8 total, 8 up, 8 in 2024-09-21T12:20:33.298 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:33 smithi183 ceph-mon[33277]: pgmap v216: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 170 B/s wr, 12 op/s 2024-09-21T12:20:33.325 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-09-21T12:20:33.325 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-21T12:20:33.325 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-21T12:20:33.364 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph config log 1 --format=json 2024-09-21T12:20:33.390 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:33 smithi114 ceph-mon[30324]: pgmap v216: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 170 B/s wr, 12 op/s 2024-09-21T12:20:33.390 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:33 smithi114 ceph-mon[40867]: pgmap v216: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 170 B/s wr, 12 op/s 2024-09-21T12:20:33.618 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:20:34.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:34 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:20:34.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:34 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:20:34.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:34 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:20:34.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:34 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:20:34.391 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:20:34.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:34 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:20:34.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:34 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:20:35.111 INFO:teuthology.orchestra.run.smithi114.stdout:[{"version":25,"timestamp":"2024-09-21T12:20:21.974174+0000","name":"","changes":[{"name":"mgr/mgr/dashboard/RGW_API_SECRET_KEY","new_value":"MBWspdS2uqGUbWxmot9xxgj2mN2pCeJfA7Ld3hga"}]}] 2024-09-21T12:20:35.111 INFO:tasks.ceph_manager:config epoch is 25 2024-09-21T12:20:35.112 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T12:20:35.112 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-21T12:20:35.112 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph mgr dump --format=json 2024-09-21T12:20:35.311 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:35 smithi183 ceph-mon[33277]: osdmap e71: 8 total, 8 up, 8 in 2024-09-21T12:20:35.311 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:35 smithi183 ceph-mon[33277]: pgmap v218: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2024-09-21T12:20:35.312 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:35 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2128185613' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-21T12:20:35.398 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:20:35.519 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:35 smithi114 ceph-mon[40867]: osdmap e71: 8 total, 8 up, 8 in 2024-09-21T12:20:35.519 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:35 smithi114 ceph-mon[40867]: pgmap v218: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2024-09-21T12:20:35.519 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:35 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2128185613' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-21T12:20:35.519 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:35 smithi114 ceph-mon[30324]: osdmap e71: 8 total, 8 up, 8 in 2024-09-21T12:20:35.519 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:35 smithi114 ceph-mon[30324]: pgmap v218: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2024-09-21T12:20:35.519 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:35 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2128185613' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-21T12:20:36.081 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:20:36.401 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:20:36 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:20:36.401 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:36 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:20:36.401 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:36 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3442571997' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T12:20:36.401 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:36 smithi183 ceph-mon[33277]: pgmap v219: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:20:36.451 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:36 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:20:36.451 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:36 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3442571997' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T12:20:36.451 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:36 smithi114 ceph-mon[30324]: pgmap v219: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:20:36.451 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:36 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:20:36.451 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:36 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3442571997' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T12:20:36.451 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:36 smithi114 ceph-mon[40867]: pgmap v219: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:20:36.729 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":16,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":237768393},{"type":"v1","addr":"172.21.15.114:6801","nonce":237768393}]},"active_addr":"172.21.15.114:6801/237768393","active_change":"2024-09-21T12:15:20.045322+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":24112,"name":"x","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.114:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":3598782088}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":4173909019}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":3866305956}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":1457547294}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":995404371}]}]} 2024-09-21T12:20:36.733 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-21T12:20:36.733 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-21T12:20:36.733 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T12:20:36.980 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:20:37.647 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:20:37.647 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":72,"fsid":"e2ad975c-7812-11ef-baf6-efdc52797490","created":"2024-09-21T12:14:24.181227+0000","modified":"2024-09-21T12:20:36.750128+0000","last_up_change":"2024-09-21T12:20:03.702987+0000","last_in_change":"2024-09-21T12:19:40.981543+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T12:17:38.357647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-21T12:20:12.761046+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":61,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-21T12:20:13.713971+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-21T12:20:15.368765+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-21T12:20:17.314275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-21T12:20:19.370218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5715e521-c7dc-41dd-a075-5c96a88fa9b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3555869816},{"type":"v1","addr":"172.21.15.114:6803","nonce":3555869816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3555869816},{"type":"v1","addr":"172.21.15.114:6805","nonce":3555869816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3555869816},{"type":"v1","addr":"172.21.15.114:6809","nonce":3555869816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3555869816},{"type":"v1","addr":"172.21.15.114:6807","nonce":3555869816}]},"public_addr":"172.21.15.114:6803/3555869816","cluster_addr":"172.21.15.114:6805/3555869816","heartbeat_back_addr":"172.21.15.114:6809/3555869816","heartbeat_front_addr":"172.21.15.114:6807/3555869816","state":["exists","up"]},{"osd":1,"uuid":"1fe160a7-5972-4e3e-99ce-15e2311ba4c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":2744663863},{"type":"v1","addr":"172.21.15.114:6811","nonce":2744663863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":2744663863},{"type":"v1","addr":"172.21.15.114:6813","nonce":2744663863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":2744663863},{"type":"v1","addr":"172.21.15.114:6817","nonce":2744663863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":2744663863},{"type":"v1","addr":"172.21.15.114:6815","nonce":2744663863}]},"public_addr":"172.21.15.114:6811/2744663863","cluster_addr":"172.21.15.114:6813/2744663863","heartbeat_back_addr":"172.21.15.114:6817/2744663863","heartbeat_front_addr":"172.21.15.114:6815/2744663863","state":["exists","up"]},{"osd":2,"uuid":"6a63581f-7603-49ae-9ab3-b37e7c91ab91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3740045173},{"type":"v1","addr":"172.21.15.114:6819","nonce":3740045173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3740045173},{"type":"v1","addr":"172.21.15.114:6821","nonce":3740045173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":3740045173},{"type":"v1","addr":"172.21.15.114:6825","nonce":3740045173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3740045173},{"type":"v1","addr":"172.21.15.114:6823","nonce":3740045173}]},"public_addr":"172.21.15.114:6819/3740045173","cluster_addr":"172.21.15.114:6821/3740045173","heartbeat_back_addr":"172.21.15.114:6825/3740045173","heartbeat_front_addr":"172.21.15.114:6823/3740045173","state":["exists","up"]},{"osd":3,"uuid":"0ad68cdb-ae5c-4090-8a8d-b67c1177ae42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":1312590318},{"type":"v1","addr":"172.21.15.114:6827","nonce":1312590318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":1312590318},{"type":"v1","addr":"172.21.15.114:6829","nonce":1312590318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1312590318},{"type":"v1","addr":"172.21.15.114:6833","nonce":1312590318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":1312590318},{"type":"v1","addr":"172.21.15.114:6831","nonce":1312590318}]},"public_addr":"172.21.15.114:6827/1312590318","cluster_addr":"172.21.15.114:6829/1312590318","heartbeat_back_addr":"172.21.15.114:6833/1312590318","heartbeat_front_addr":"172.21.15.114:6831/1312590318","state":["exists","up"]},{"osd":4,"uuid":"26bb8a84-e41d-4425-8491-2dea9764b7cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":1391644836},{"type":"v1","addr":"172.21.15.183:6801","nonce":1391644836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":1391644836},{"type":"v1","addr":"172.21.15.183:6803","nonce":1391644836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":1391644836},{"type":"v1","addr":"172.21.15.183:6807","nonce":1391644836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":1391644836},{"type":"v1","addr":"172.21.15.183:6805","nonce":1391644836}]},"public_addr":"172.21.15.183:6801/1391644836","cluster_addr":"172.21.15.183:6803/1391644836","heartbeat_back_addr":"172.21.15.183:6807/1391644836","heartbeat_front_addr":"172.21.15.183:6805/1391644836","state":["exists","up"]},{"osd":5,"uuid":"33f96871-0869-4281-82b9-bd518987c551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":3187643473},{"type":"v1","addr":"172.21.15.183:6809","nonce":3187643473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":3187643473},{"type":"v1","addr":"172.21.15.183:6811","nonce":3187643473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":3187643473},{"type":"v1","addr":"172.21.15.183:6815","nonce":3187643473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":3187643473},{"type":"v1","addr":"172.21.15.183:6813","nonce":3187643473}]},"public_addr":"172.21.15.183:6809/3187643473","cluster_addr":"172.21.15.183:6811/3187643473","heartbeat_back_addr":"172.21.15.183:6815/3187643473","heartbeat_front_addr":"172.21.15.183:6813/3187643473","state":["exists","up"]},{"osd":6,"uuid":"e09c72b8-ad5a-4779-a874-b149fce6ed70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3025199631},{"type":"v1","addr":"172.21.15.183:6817","nonce":3025199631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3025199631},{"type":"v1","addr":"172.21.15.183:6819","nonce":3025199631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":3025199631},{"type":"v1","addr":"172.21.15.183:6823","nonce":3025199631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3025199631},{"type":"v1","addr":"172.21.15.183:6821","nonce":3025199631}]},"public_addr":"172.21.15.183:6817/3025199631","cluster_addr":"172.21.15.183:6819/3025199631","heartbeat_back_addr":"172.21.15.183:6823/3025199631","heartbeat_front_addr":"172.21.15.183:6821/3025199631","state":["exists","up"]},{"osd":7,"uuid":"1a24194d-1af6-485f-8856-98844a02dd9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3203337442},{"type":"v1","addr":"172.21.15.183:6825","nonce":3203337442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":3203337442},{"type":"v1","addr":"172.21.15.183:6827","nonce":3203337442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3203337442},{"type":"v1","addr":"172.21.15.183:6831","nonce":3203337442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3203337442},{"type":"v1","addr":"172.21.15.183:6829","nonce":3203337442}]},"public_addr":"172.21.15.183:6825/3203337442","cluster_addr":"172.21.15.183:6827/3203337442","heartbeat_back_addr":"172.21.15.183:6831/3203337442","heartbeat_front_addr":"172.21.15.183:6829/3203337442","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:20:29.879765+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:19:30.681663+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:20:14.175512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:20:33.194446+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:19:37.016906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:20:24.938106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:19:31.112105+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:20:02.689705+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6801/770932721":"2024-09-22T12:15:20.045217+0000","172.21.15.114:6800/770932721":"2024-09-22T12:15:20.045217+0000","172.21.15.114:0/139803357":"2024-09-22T12:15:20.045217+0000","172.21.15.114:0/2106467614":"2024-09-22T12:15:20.045217+0000","172.21.15.114:6801/1725148595":"2024-09-22T12:14:53.998653+0000","172.21.15.114:0/442404805":"2024-09-22T12:14:53.998653+0000","172.21.15.114:6800/1725148595":"2024-09-22T12:14:53.998653+0000","172.21.15.114:0/2620151541":"2024-09-22T12:15:20.045217+0000","172.21.15.114:0/3416259228":"2024-09-22T12:14:53.998653+0000","172.21.15.114:0/898990943":"2024-09-22T12:14:53.998653+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T12:20:37.858 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:37 smithi183 ceph-mon[33277]: osdmap e72: 8 total, 8 up, 8 in 2024-09-21T12:20:37.858 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:37 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/4081308142' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T12:20:37.957 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:37 smithi114 ceph-mon[40867]: osdmap e72: 8 total, 8 up, 8 in 2024-09-21T12:20:37.957 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:37 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/4081308142' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T12:20:37.957 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:37 smithi114 ceph-mon[30324]: osdmap e72: 8 total, 8 up, 8 in 2024-09-21T12:20:37.957 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:37 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4081308142' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T12:20:38.311 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-21T12:20:38.312 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T12:20:38.488 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:20:38 smithi183 systemd[1]: Starting Ceph prometheus.a for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:20:38.554 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:20:38.879 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:38 smithi183 ceph-mon[33277]: pgmap v221: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 805 B/s rd, 0 op/s 2024-09-21T12:20:38.880 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:20:38 smithi183 podman[61396]: 2024-09-21 12:20:38.618165864 +0000 UTC m=+0.107034140 container create bc19bc112d29a7438590de74c31d220bddf8e7c03f938ec6aaeb66f041dfaed9 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:20:38.880 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:20:38 smithi183 podman[61396]: 2024-09-21 12:20:38.527448534 +0000 UTC m=+0.016316817 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-21T12:20:38.880 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:20:38 smithi183 podman[61396]: 2024-09-21 12:20:38.878899138 +0000 UTC m=+0.367767426 container init bc19bc112d29a7438590de74c31d220bddf8e7c03f938ec6aaeb66f041dfaed9 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:20:39.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:38 smithi114 ceph-mon[40867]: pgmap v221: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 805 B/s rd, 0 op/s 2024-09-21T12:20:39.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:38 smithi114 ceph-mon[30324]: pgmap v221: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 805 B/s rd, 0 op/s 2024-09-21T12:20:39.180 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:20:39.180 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":72,"fsid":"e2ad975c-7812-11ef-baf6-efdc52797490","created":"2024-09-21T12:14:24.181227+0000","modified":"2024-09-21T12:20:36.750128+0000","last_up_change":"2024-09-21T12:20:03.702987+0000","last_in_change":"2024-09-21T12:19:40.981543+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T12:17:38.357647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-21T12:20:12.761046+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":61,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-21T12:20:13.713971+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-21T12:20:15.368765+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-21T12:20:17.314275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-21T12:20:19.370218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5715e521-c7dc-41dd-a075-5c96a88fa9b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3555869816},{"type":"v1","addr":"172.21.15.114:6803","nonce":3555869816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3555869816},{"type":"v1","addr":"172.21.15.114:6805","nonce":3555869816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3555869816},{"type":"v1","addr":"172.21.15.114:6809","nonce":3555869816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3555869816},{"type":"v1","addr":"172.21.15.114:6807","nonce":3555869816}]},"public_addr":"172.21.15.114:6803/3555869816","cluster_addr":"172.21.15.114:6805/3555869816","heartbeat_back_addr":"172.21.15.114:6809/3555869816","heartbeat_front_addr":"172.21.15.114:6807/3555869816","state":["exists","up"]},{"osd":1,"uuid":"1fe160a7-5972-4e3e-99ce-15e2311ba4c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":2744663863},{"type":"v1","addr":"172.21.15.114:6811","nonce":2744663863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":2744663863},{"type":"v1","addr":"172.21.15.114:6813","nonce":2744663863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":2744663863},{"type":"v1","addr":"172.21.15.114:6817","nonce":2744663863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":2744663863},{"type":"v1","addr":"172.21.15.114:6815","nonce":2744663863}]},"public_addr":"172.21.15.114:6811/2744663863","cluster_addr":"172.21.15.114:6813/2744663863","heartbeat_back_addr":"172.21.15.114:6817/2744663863","heartbeat_front_addr":"172.21.15.114:6815/2744663863","state":["exists","up"]},{"osd":2,"uuid":"6a63581f-7603-49ae-9ab3-b37e7c91ab91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3740045173},{"type":"v1","addr":"172.21.15.114:6819","nonce":3740045173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3740045173},{"type":"v1","addr":"172.21.15.114:6821","nonce":3740045173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":3740045173},{"type":"v1","addr":"172.21.15.114:6825","nonce":3740045173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3740045173},{"type":"v1","addr":"172.21.15.114:6823","nonce":3740045173}]},"public_addr":"172.21.15.114:6819/3740045173","cluster_addr":"172.21.15.114:6821/3740045173","heartbeat_back_addr":"172.21.15.114:6825/3740045173","heartbeat_front_addr":"172.21.15.114:6823/3740045173","state":["exists","up"]},{"osd":3,"uuid":"0ad68cdb-ae5c-4090-8a8d-b67c1177ae42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":1312590318},{"type":"v1","addr":"172.21.15.114:6827","nonce":1312590318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":1312590318},{"type":"v1","addr":"172.21.15.114:6829","nonce":1312590318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1312590318},{"type":"v1","addr":"172.21.15.114:6833","nonce":1312590318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":1312590318},{"type":"v1","addr":"172.21.15.114:6831","nonce":1312590318}]},"public_addr":"172.21.15.114:6827/1312590318","cluster_addr":"172.21.15.114:6829/1312590318","heartbeat_back_addr":"172.21.15.114:6833/1312590318","heartbeat_front_addr":"172.21.15.114:6831/1312590318","state":["exists","up"]},{"osd":4,"uuid":"26bb8a84-e41d-4425-8491-2dea9764b7cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":1391644836},{"type":"v1","addr":"172.21.15.183:6801","nonce":1391644836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":1391644836},{"type":"v1","addr":"172.21.15.183:6803","nonce":1391644836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":1391644836},{"type":"v1","addr":"172.21.15.183:6807","nonce":1391644836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":1391644836},{"type":"v1","addr":"172.21.15.183:6805","nonce":1391644836}]},"public_addr":"172.21.15.183:6801/1391644836","cluster_addr":"172.21.15.183:6803/1391644836","heartbeat_back_addr":"172.21.15.183:6807/1391644836","heartbeat_front_addr":"172.21.15.183:6805/1391644836","state":["exists","up"]},{"osd":5,"uuid":"33f96871-0869-4281-82b9-bd518987c551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":3187643473},{"type":"v1","addr":"172.21.15.183:6809","nonce":3187643473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":3187643473},{"type":"v1","addr":"172.21.15.183:6811","nonce":3187643473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":3187643473},{"type":"v1","addr":"172.21.15.183:6815","nonce":3187643473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":3187643473},{"type":"v1","addr":"172.21.15.183:6813","nonce":3187643473}]},"public_addr":"172.21.15.183:6809/3187643473","cluster_addr":"172.21.15.183:6811/3187643473","heartbeat_back_addr":"172.21.15.183:6815/3187643473","heartbeat_front_addr":"172.21.15.183:6813/3187643473","state":["exists","up"]},{"osd":6,"uuid":"e09c72b8-ad5a-4779-a874-b149fce6ed70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3025199631},{"type":"v1","addr":"172.21.15.183:6817","nonce":3025199631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3025199631},{"type":"v1","addr":"172.21.15.183:6819","nonce":3025199631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":3025199631},{"type":"v1","addr":"172.21.15.183:6823","nonce":3025199631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3025199631},{"type":"v1","addr":"172.21.15.183:6821","nonce":3025199631}]},"public_addr":"172.21.15.183:6817/3025199631","cluster_addr":"172.21.15.183:6819/3025199631","heartbeat_back_addr":"172.21.15.183:6823/3025199631","heartbeat_front_addr":"172.21.15.183:6821/3025199631","state":["exists","up"]},{"osd":7,"uuid":"1a24194d-1af6-485f-8856-98844a02dd9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3203337442},{"type":"v1","addr":"172.21.15.183:6825","nonce":3203337442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":3203337442},{"type":"v1","addr":"172.21.15.183:6827","nonce":3203337442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3203337442},{"type":"v1","addr":"172.21.15.183:6831","nonce":3203337442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3203337442},{"type":"v1","addr":"172.21.15.183:6829","nonce":3203337442}]},"public_addr":"172.21.15.183:6825/3203337442","cluster_addr":"172.21.15.183:6827/3203337442","heartbeat_back_addr":"172.21.15.183:6831/3203337442","heartbeat_front_addr":"172.21.15.183:6829/3203337442","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:20:29.879765+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:19:30.681663+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:20:14.175512+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:20:33.194446+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:19:37.016906+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:20:24.938106+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:19:31.112105+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:20:02.689705+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6801/770932721":"2024-09-22T12:15:20.045217+0000","172.21.15.114:6800/770932721":"2024-09-22T12:15:20.045217+0000","172.21.15.114:0/139803357":"2024-09-22T12:15:20.045217+0000","172.21.15.114:0/2106467614":"2024-09-22T12:15:20.045217+0000","172.21.15.114:6801/1725148595":"2024-09-22T12:14:53.998653+0000","172.21.15.114:0/442404805":"2024-09-22T12:14:53.998653+0000","172.21.15.114:6800/1725148595":"2024-09-22T12:14:53.998653+0000","172.21.15.114:0/2620151541":"2024-09-22T12:15:20.045217+0000","172.21.15.114:0/3416259228":"2024-09-22T12:14:53.998653+0000","172.21.15.114:0/898990943":"2024-09-22T12:14:53.998653+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T12:20:39.239 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:20:38 smithi183 podman[61396]: 2024-09-21 12:20:38.884067203 +0000 UTC m=+0.372935483 container start bc19bc112d29a7438590de74c31d220bddf8e7c03f938ec6aaeb66f041dfaed9 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:20:39.239 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:20:38 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:20:38.904Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-21T12:20:39.239 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:20:38 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:20:38.904Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-21T12:20:39.239 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:20:38 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:20:38.904Z caller=main.go:623 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi183 (none))" 2024-09-21T12:20:39.239 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:20:38 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:20:38.904Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-21T12:20:39.239 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:20:38 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:20:38.904Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-21T12:20:39.239 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:20:38 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:20:38.905Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-21T12:20:39.239 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:20:38 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:20:38.906Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-21T12:20:39.239 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:20:38 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:20:38.907Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-21T12:20:39.239 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:20:38 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:20:38.907Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-21T12:20:39.240 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:20:38 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:20:38.909Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-21T12:20:39.240 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:20:38 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:20:38.909Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.89µs 2024-09-21T12:20:39.240 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:20:38 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:20:38.909Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-21T12:20:39.240 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:20:38 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:20:38.909Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-09-21T12:20:39.240 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:20:38 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:20:38.909Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=23.38µs wal_replay_duration=187.437µs wbl_replay_duration=118ns total_replay_duration=230.212µs 2024-09-21T12:20:39.240 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:20:38 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:20:38.910Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-21T12:20:39.240 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:20:38 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:20:38.910Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-21T12:20:39.240 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:20:38 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:20:38.910Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-21T12:20:39.240 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:20:38 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:20:38.926Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.242936ms db_storage=676ns remote_storage=1.064µs web_handler=334ns query_engine=569ns scrape=386.249µs scrape_sd=92.415µs notify=706ns notify_sd=843ns rules=14.216741ms tracing=7.215µs 2024-09-21T12:20:39.240 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:20:38 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:20:38.926Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-21T12:20:39.240 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:20:38 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:20:38.926Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-21T12:20:39.240 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:20:39 smithi183 bash[61396]: bc19bc112d29a7438590de74c31d220bddf8e7c03f938ec6aaeb66f041dfaed9 2024-09-21T12:20:39.240 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:20:39 smithi183 systemd[1]: Started Ceph prometheus.a for e2ad975c-7812-11ef-baf6-efdc52797490. 2024-09-21T12:20:40.104 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-21T12:20:40.105 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-21T12:20:40.105 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-21T12:20:40.105 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-21T12:20:40.106 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-21T12:20:40.106 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-21T12:20:40.107 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-21T12:20:40.107 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-21T12:20:40.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:40 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:40.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:40 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:40.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:40 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:40.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:40 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-21T12:20:40.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:40 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2801359754' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T12:20:40.489 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Sep 21 12:20:40 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-x[34803]: ignoring --setuser ceph since I am not root 2024-09-21T12:20:40.489 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Sep 21 12:20:40 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-x[34803]: ignoring --setgroup ceph since I am not root 2024-09-21T12:20:40.489 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Sep 21 12:20:40 smithi183 ceph-mgr[34826]: -- 172.21.15.183:0/3600560253 <== mon.1 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560cf7d790e0 con 0x560cf7d56800 2024-09-21T12:20:40.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:40 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:40.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:40 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:40.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:40 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:40.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:40 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-21T12:20:40.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:40 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2801359754' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T12:20:40.525 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:20:40 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ignoring --setuser ceph since I am not root 2024-09-21T12:20:40.525 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:20:40 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ignoring --setgroup ceph since I am not root 2024-09-21T12:20:40.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:40 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:40.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:40 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:40.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:40 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' 2024-09-21T12:20:40.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:40 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-21T12:20:40.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:40 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2801359754' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T12:20:40.658 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:20:40.953 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:20:40.953 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:20:40.954 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:20:40.955 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:20:41.254 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:41 smithi114 ceph-mon[40867]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-21T12:20:41.254 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:41 smithi114 ceph-mon[40867]: mgrmap e17: y(active, since 5m), standbys: x 2024-09-21T12:20:41.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:41 smithi114 ceph-mon[30324]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-21T12:20:41.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:41 smithi114 ceph-mon[30324]: mgrmap e17: y(active, since 5m), standbys: x 2024-09-21T12:20:41.259 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:20:41.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:41 smithi183 ceph-mon[33277]: from='mgr.14152 172.21.15.114:0/2933742270' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-21T12:20:41.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:41 smithi183 ceph-mon[33277]: mgrmap e17: y(active, since 5m), standbys: x 2024-09-21T12:20:42.196 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:42 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:20:42.196 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:42 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:20:42.196 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:42 smithi114 ceph-mon[40867]: osdmap e73: 8 total, 8 up, 8 in 2024-09-21T12:20:42.196 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:42 smithi114 ceph-mon[40867]: osdmap e74: 8 total, 8 up, 8 in 2024-09-21T12:20:42.196 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:42 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:20:42.196 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:42 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:20:42.196 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:42 smithi114 ceph-mon[30324]: osdmap e73: 8 total, 8 up, 8 in 2024-09-21T12:20:42.196 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:42 smithi114 ceph-mon[30324]: osdmap e74: 8 total, 8 up, 8 in 2024-09-21T12:20:42.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:42 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:20:42.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:42 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:20:42.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:42 smithi183 ceph-mon[33277]: osdmap e73: 8 total, 8 up, 8 in 2024-09-21T12:20:42.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:42 smithi183 ceph-mon[33277]: osdmap e74: 8 total, 8 up, 8 in 2024-09-21T12:20:44.289 INFO:teuthology.orchestra.run.smithi114.stdout:55834574894 2024-09-21T12:20:44.290 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-21T12:20:45.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:45 smithi183 ceph-mon[33277]: Standby manager daemon x restarted 2024-09-21T12:20:45.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:45 smithi183 ceph-mon[33277]: Standby manager daemon x started 2024-09-21T12:20:45.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:45 smithi183 ceph-mon[33277]: from='mgr.? 172.21.15.183:0/3257868885' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-21T12:20:45.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:45 smithi183 ceph-mon[33277]: from='mgr.? 172.21.15.183:0/3257868885' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T12:20:45.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:45 smithi183 ceph-mon[33277]: from='mgr.? 172.21.15.183:0/3257868885' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-21T12:20:45.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:45 smithi183 ceph-mon[33277]: from='mgr.? 172.21.15.183:0/3257868885' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T12:20:45.489 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Sep 21 12:20:45 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-x[34803]: [21/Sep/2024:12:20:45] ENGINE Bus STARTING 2024-09-21T12:20:45.489 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Sep 21 12:20:45 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-x[34803]: CherryPy Checker: 2024-09-21T12:20:45.489 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Sep 21 12:20:45 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-x[34803]: The Application mounted at '' has an empty config. 2024-09-21T12:20:45.489 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Sep 21 12:20:45 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-x[34803]: 2024-09-21T12:20:45.489 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Sep 21 12:20:45 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-x[34803]: [21/Sep/2024:12:20:45] ENGINE Serving on http://:::9283 2024-09-21T12:20:45.489 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Sep 21 12:20:45 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-x[34803]: [21/Sep/2024:12:20:45] ENGINE Bus STARTED 2024-09-21T12:20:45.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:45 smithi114 ceph-mon[40867]: Standby manager daemon x restarted 2024-09-21T12:20:45.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:45 smithi114 ceph-mon[40867]: Standby manager daemon x started 2024-09-21T12:20:45.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:45 smithi114 ceph-mon[40867]: from='mgr.? 172.21.15.183:0/3257868885' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-21T12:20:45.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:45 smithi114 ceph-mon[40867]: from='mgr.? 172.21.15.183:0/3257868885' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T12:20:45.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:45 smithi114 ceph-mon[40867]: from='mgr.? 172.21.15.183:0/3257868885' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-21T12:20:45.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:45 smithi114 ceph-mon[40867]: from='mgr.? 172.21.15.183:0/3257868885' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T12:20:45.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:45 smithi114 ceph-mon[30324]: Standby manager daemon x restarted 2024-09-21T12:20:45.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:45 smithi114 ceph-mon[30324]: Standby manager daemon x started 2024-09-21T12:20:45.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:45 smithi114 ceph-mon[30324]: from='mgr.? 172.21.15.183:0/3257868885' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-21T12:20:45.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:45 smithi114 ceph-mon[30324]: from='mgr.? 172.21.15.183:0/3257868885' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T12:20:45.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:45 smithi114 ceph-mon[30324]: from='mgr.? 172.21.15.183:0/3257868885' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-21T12:20:45.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:45 smithi114 ceph-mon[30324]: from='mgr.? 172.21.15.183:0/3257868885' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T12:20:45.565 INFO:teuthology.orchestra.run.smithi114.stdout:111669149730 2024-09-21T12:20:45.566 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-21T12:20:45.898 INFO:teuthology.orchestra.run.smithi114.stdout:34359738420 2024-09-21T12:20:45.899 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-21T12:20:45.999 INFO:teuthology.orchestra.run.smithi114.stdout:231928233993 2024-09-21T12:20:46.000 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-21T12:20:46.399 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[30324]: mgrmap e18: y(active, since 5m), standbys: x 2024-09-21T12:20:46.399 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[30324]: Active manager daemon y restarted 2024-09-21T12:20:46.399 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[30324]: Activating manager daemon y 2024-09-21T12:20:46.400 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T12:20:46.400 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[30324]: osdmap e75: 8 total, 8 up, 8 in 2024-09-21T12:20:46.400 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:20:46.400 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:20:46.400 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[30324]: mgrmap e19: y(active, starting, since 0.0140443s), standbys: x 2024-09-21T12:20:46.400 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T12:20:46.400 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T12:20:46.400 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T12:20:46.400 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T12:20:46.400 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T12:20:46.400 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T12:20:46.400 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T12:20:46.400 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T12:20:46.401 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T12:20:46.401 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T12:20:46.401 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T12:20:46.401 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T12:20:46.401 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T12:20:46.401 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[30324]: Manager daemon y is now available 2024-09-21T12:20:46.401 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:20:46.401 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:20:46.401 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:20:46.401 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T12:20:46.401 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T12:20:46.401 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T12:20:46.401 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T12:20:46.401 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T12:20:46.402 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[40867]: mgrmap e18: y(active, since 5m), standbys: x 2024-09-21T12:20:46.402 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[40867]: Active manager daemon y restarted 2024-09-21T12:20:46.402 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[40867]: Activating manager daemon y 2024-09-21T12:20:46.402 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T12:20:46.402 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[40867]: osdmap e75: 8 total, 8 up, 8 in 2024-09-21T12:20:46.402 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:20:46.402 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:20:46.402 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[40867]: mgrmap e19: y(active, starting, since 0.0140443s), standbys: x 2024-09-21T12:20:46.402 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T12:20:46.402 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T12:20:46.403 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T12:20:46.403 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T12:20:46.403 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T12:20:46.403 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T12:20:46.403 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T12:20:46.403 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T12:20:46.403 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T12:20:46.403 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T12:20:46.403 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T12:20:46.403 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T12:20:46.403 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T12:20:46.403 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[40867]: Manager daemon y is now available 2024-09-21T12:20:46.403 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:20:46.404 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:20:46.404 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:20:46.404 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T12:20:46.404 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T12:20:46.404 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T12:20:46.404 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T12:20:46.404 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T12:20:46.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-mon[33277]: mgrmap e18: y(active, since 5m), standbys: x 2024-09-21T12:20:46.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-mon[33277]: Active manager daemon y restarted 2024-09-21T12:20:46.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-mon[33277]: Activating manager daemon y 2024-09-21T12:20:46.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T12:20:46.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-mon[33277]: osdmap e75: 8 total, 8 up, 8 in 2024-09-21T12:20:46.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T12:20:46.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T12:20:46.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-mon[33277]: mgrmap e19: y(active, starting, since 0.0140443s), standbys: x 2024-09-21T12:20:46.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T12:20:46.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T12:20:46.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T12:20:46.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T12:20:46.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T12:20:46.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T12:20:46.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T12:20:46.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T12:20:46.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T12:20:46.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T12:20:46.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T12:20:46.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T12:20:46.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T12:20:46.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-mon[33277]: Manager daemon y is now available 2024-09-21T12:20:46.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:20:46.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:20:46.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:20:46.491 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T12:20:46.491 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T12:20:46.491 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T12:20:46.491 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T12:20:46.491 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T12:20:46.660 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:20:46] ENGINE Bus STARTING 2024-09-21T12:20:46.660 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: CherryPy Checker: 2024-09-21T12:20:46.660 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: The Application mounted at '' has an empty config. 2024-09-21T12:20:46.660 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: 2024-09-21T12:20:46.819 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:20:46 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:20:47.025 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:20:46] ENGINE Serving on http://:::9283 2024-09-21T12:20:47.025 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:20:46] ENGINE Bus STARTED 2024-09-21T12:20:47.025 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: 2024-09-21T12:20:46.900+0000 7f0d60d85640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:12:20:46] ENGINE Error in HTTPServer.serve 2024-09-21T12:20:47.026 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: Traceback (most recent call last): 2024-09-21T12:20:47.026 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T12:20:47.026 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: self._connections.run(self.expiration_interval) 2024-09-21T12:20:47.026 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T12:20:47.026 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: self._run(expiration_interval) 2024-09-21T12:20:47.026 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T12:20:47.026 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T12:20:47.026 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T12:20:47.026 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T12:20:47.026 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T12:20:47.026 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: s = self.context.wrap_socket( 2024-09-21T12:20:47.026 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T12:20:47.026 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: return self.sslsocket_class._create( 2024-09-21T12:20:47.027 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T12:20:47.027 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: self.do_handshake() 2024-09-21T12:20:47.027 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T12:20:47.027 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: self._sslobj.do_handshake() 2024-09-21T12:20:47.027 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T12:20:47.027 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:20:46 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: 2024-09-21T12:20:47.189 INFO:teuthology.orchestra.run.smithi114.stdout:167503724564 2024-09-21T12:20:47.189 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-21T12:20:47.231 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:20:47.231 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:20:47.632 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:20:47.650 INFO:teuthology.orchestra.run.smithi114.stdout:77309411368 2024-09-21T12:20:47.650 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-21T12:20:47.651 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:20:47.819 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:47 smithi183 ceph-mon[33277]: mgrmap e20: y(active, since 1.01575s), standbys: x 2024-09-21T12:20:47.819 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:47 smithi183 ceph-mon[33277]: osdmap e76: 8 total, 8 up, 8 in 2024-09-21T12:20:47.820 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:47 smithi183 ceph-mon[33277]: [21/Sep/2024:12:20:46] ENGINE Bus STARTING 2024-09-21T12:20:47.820 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:47 smithi183 ceph-mon[33277]: [21/Sep/2024:12:20:46] ENGINE Serving on https://172.21.15.114:7150 2024-09-21T12:20:47.820 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:47 smithi183 ceph-mon[33277]: [21/Sep/2024:12:20:46] ENGINE Error in HTTPServer.serve 2024-09-21T12:20:47.820 INFO:journalctl@ceph.mon.b.smithi183.stdout: Traceback (most recent call last): 2024-09-21T12:20:47.820 INFO:journalctl@ceph.mon.b.smithi183.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T12:20:47.820 INFO:journalctl@ceph.mon.b.smithi183.stdout: self._connections.run(self.expiration_interval) 2024-09-21T12:20:47.820 INFO:journalctl@ceph.mon.b.smithi183.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T12:20:47.820 INFO:journalctl@ceph.mon.b.smithi183.stdout: self._run(expiration_interval) 2024-09-21T12:20:47.820 INFO:journalctl@ceph.mon.b.smithi183.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T12:20:47.820 INFO:journalctl@ceph.mon.b.smithi183.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T12:20:47.820 INFO:journalctl@ceph.mon.b.smithi183.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T12:20:47.820 INFO:journalctl@ceph.mon.b.smithi183.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T12:20:47.820 INFO:journalctl@ceph.mon.b.smithi183.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T12:20:47.821 INFO:journalctl@ceph.mon.b.smithi183.stdout: s = self.context.wrap_socket( 2024-09-21T12:20:47.821 INFO:journalctl@ceph.mon.b.smithi183.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T12:20:47.821 INFO:journalctl@ceph.mon.b.smithi183.stdout: return self.sslsocket_class._create( 2024-09-21T12:20:47.821 INFO:journalctl@ceph.mon.b.smithi183.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T12:20:47.821 INFO:journalctl@ceph.mon.b.smithi183.stdout: self.do_handshake() 2024-09-21T12:20:47.821 INFO:journalctl@ceph.mon.b.smithi183.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T12:20:47.821 INFO:journalctl@ceph.mon.b.smithi183.stdout: self._sslobj.do_handshake() 2024-09-21T12:20:47.821 INFO:journalctl@ceph.mon.b.smithi183.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T12:20:47.821 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:47 smithi183 ceph-mon[33277]: [21/Sep/2024:12:20:46] ENGINE Serving on http://172.21.15.114:8765 2024-09-21T12:20:47.821 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:47 smithi183 ceph-mon[33277]: [21/Sep/2024:12:20:46] ENGINE Bus STARTED 2024-09-21T12:20:47.913 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:47 smithi114 ceph-mon[30324]: mgrmap e20: y(active, since 1.01575s), standbys: x 2024-09-21T12:20:47.913 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:47 smithi114 ceph-mon[30324]: osdmap e76: 8 total, 8 up, 8 in 2024-09-21T12:20:47.913 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:47 smithi114 ceph-mon[30324]: [21/Sep/2024:12:20:46] ENGINE Bus STARTING 2024-09-21T12:20:47.913 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:47 smithi114 ceph-mon[30324]: [21/Sep/2024:12:20:46] ENGINE Serving on https://172.21.15.114:7150 2024-09-21T12:20:47.913 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:47 smithi114 ceph-mon[30324]: [21/Sep/2024:12:20:46] ENGINE Error in HTTPServer.serve 2024-09-21T12:20:47.913 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-21T12:20:47.913 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T12:20:47.913 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._connections.run(self.expiration_interval) 2024-09-21T12:20:47.914 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T12:20:47.914 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._run(expiration_interval) 2024-09-21T12:20:47.914 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T12:20:47.914 INFO:journalctl@ceph.mon.a.smithi114.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T12:20:47.914 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T12:20:47.914 INFO:journalctl@ceph.mon.a.smithi114.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T12:20:47.914 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T12:20:47.914 INFO:journalctl@ceph.mon.a.smithi114.stdout: s = self.context.wrap_socket( 2024-09-21T12:20:47.914 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T12:20:47.914 INFO:journalctl@ceph.mon.a.smithi114.stdout: return self.sslsocket_class._create( 2024-09-21T12:20:47.914 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T12:20:47.914 INFO:journalctl@ceph.mon.a.smithi114.stdout: self.do_handshake() 2024-09-21T12:20:47.914 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T12:20:47.914 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._sslobj.do_handshake() 2024-09-21T12:20:47.914 INFO:journalctl@ceph.mon.a.smithi114.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T12:20:47.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:47 smithi114 ceph-mon[30324]: [21/Sep/2024:12:20:46] ENGINE Serving on http://172.21.15.114:8765 2024-09-21T12:20:47.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:47 smithi114 ceph-mon[30324]: [21/Sep/2024:12:20:46] ENGINE Bus STARTED 2024-09-21T12:20:47.915 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:47 smithi114 ceph-mon[40867]: mgrmap e20: y(active, since 1.01575s), standbys: x 2024-09-21T12:20:47.915 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:47 smithi114 ceph-mon[40867]: osdmap e76: 8 total, 8 up, 8 in 2024-09-21T12:20:47.915 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:47 smithi114 ceph-mon[40867]: [21/Sep/2024:12:20:46] ENGINE Bus STARTING 2024-09-21T12:20:47.915 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:47 smithi114 ceph-mon[40867]: [21/Sep/2024:12:20:46] ENGINE Serving on https://172.21.15.114:7150 2024-09-21T12:20:47.915 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:47 smithi114 ceph-mon[40867]: [21/Sep/2024:12:20:46] ENGINE Error in HTTPServer.serve 2024-09-21T12:20:47.915 INFO:journalctl@ceph.mon.c.smithi114.stdout: Traceback (most recent call last): 2024-09-21T12:20:47.915 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T12:20:47.915 INFO:journalctl@ceph.mon.c.smithi114.stdout: self._connections.run(self.expiration_interval) 2024-09-21T12:20:47.915 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T12:20:47.915 INFO:journalctl@ceph.mon.c.smithi114.stdout: self._run(expiration_interval) 2024-09-21T12:20:47.916 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T12:20:47.916 INFO:journalctl@ceph.mon.c.smithi114.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T12:20:47.916 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T12:20:47.916 INFO:journalctl@ceph.mon.c.smithi114.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T12:20:47.916 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T12:20:47.916 INFO:journalctl@ceph.mon.c.smithi114.stdout: s = self.context.wrap_socket( 2024-09-21T12:20:47.916 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T12:20:47.916 INFO:journalctl@ceph.mon.c.smithi114.stdout: return self.sslsocket_class._create( 2024-09-21T12:20:47.916 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T12:20:47.916 INFO:journalctl@ceph.mon.c.smithi114.stdout: self.do_handshake() 2024-09-21T12:20:47.916 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T12:20:47.916 INFO:journalctl@ceph.mon.c.smithi114.stdout: self._sslobj.do_handshake() 2024-09-21T12:20:47.916 INFO:journalctl@ceph.mon.c.smithi114.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T12:20:47.916 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:47 smithi114 ceph-mon[40867]: [21/Sep/2024:12:20:46] ENGINE Serving on http://172.21.15.114:8765 2024-09-21T12:20:47.916 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:47 smithi114 ceph-mon[40867]: [21/Sep/2024:12:20:46] ENGINE Bus STARTED 2024-09-21T12:20:48.564 INFO:teuthology.orchestra.run.smithi114.stdout:34359738421 2024-09-21T12:20:48.901 INFO:teuthology.orchestra.run.smithi114.stdout:231928233994 2024-09-21T12:20:48.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:48 smithi183 ceph-mon[33277]: pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:20:48.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:48 smithi183 ceph-mon[33277]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-21T12:20:48.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:48 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1828096081' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T12:20:49.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:48 smithi114 ceph-mon[40867]: pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:20:49.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:48 smithi114 ceph-mon[40867]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-21T12:20:49.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:48 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1828096081' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T12:20:49.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:48 smithi114 ceph-mon[30324]: pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:20:49.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:48 smithi114 ceph-mon[30324]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-21T12:20:49.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:48 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1828096081' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T12:20:49.556 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233993 got 231928233994 for osd.7 2024-09-21T12:20:49.556 DEBUG:teuthology.parallel:result is None 2024-09-21T12:20:49.781 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:49 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2441301881' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T12:20:49.782 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:49 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:20:49.782 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:49 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:20:49.782 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:49 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:20:49.782 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:49 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:20:49.782 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:49 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:49.782 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:49 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:49.782 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:49 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:49.782 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:49 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:49.782 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:49 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:49.782 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:49 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:49.782 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:49 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:49.782 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:49 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:49.783 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:49 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:20:49.783 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:49 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2441301881' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T12:20:49.783 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:49 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:20:49.783 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:49 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:20:49.783 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:49 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:20:49.783 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:49 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:20:49.783 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:49 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:49.783 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:49 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:49.783 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:49 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:49.783 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:49 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:49.783 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:49 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:49.783 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:49 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:49.784 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:49 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:49.784 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:49 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:49.784 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:49 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:20:50.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:49 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2441301881' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T12:20:50.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:49 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:20:50.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:49 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:20:50.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:49 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:20:50.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:49 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:20:50.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:49 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:50.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:49 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:50.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:49 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:50.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:49 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:50.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:49 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:50.240 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:49 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:50.240 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:49 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:50.240 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:49 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T12:20:50.240 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:49 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:20:50.483 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:20:50.837 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:50 smithi114 ceph-mon[30324]: Adjusting osd_memory_target on smithi183 to 3760M 2024-09-21T12:20:50.838 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:50 smithi114 ceph-mon[30324]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:20:50.838 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:50 smithi114 ceph-mon[30324]: mgrmap e22: y(active, since 5s), standbys: x 2024-09-21T12:20:50.838 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:50 smithi114 ceph-mon[40867]: Adjusting osd_memory_target on smithi183 to 3760M 2024-09-21T12:20:50.838 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:50 smithi114 ceph-mon[40867]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:20:50.838 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:50 smithi114 ceph-mon[40867]: mgrmap e22: y(active, since 5s), standbys: x 2024-09-21T12:20:50.865 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738420 got 34359738421 for osd.0 2024-09-21T12:20:50.865 DEBUG:teuthology.parallel:result is None 2024-09-21T12:20:51.173 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:20:51.175 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:20:51.190 INFO:teuthology.orchestra.run.smithi114.stdout:201863462930 2024-09-21T12:20:51.190 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-21T12:20:51.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:50 smithi183 ceph-mon[33277]: Adjusting osd_memory_target on smithi183 to 3760M 2024-09-21T12:20:51.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:50 smithi183 ceph-mon[33277]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:20:51.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:50 smithi183 ceph-mon[33277]: mgrmap e22: y(active, since 5s), standbys: x 2024-09-21T12:20:52.382 INFO:teuthology.orchestra.run.smithi114.stdout:111669149731 2024-09-21T12:20:52.642 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:20:52 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:20:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:20:52.643 INFO:teuthology.orchestra.run.smithi114.stdout:137438953502 2024-09-21T12:20:52.643 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-21T12:20:52.870 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:20:52.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:52 smithi114 ceph-mon[30324]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-21T12:20:52.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:52 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1787183520' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T12:20:52.969 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:52 smithi114 ceph-mon[40867]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-21T12:20:52.969 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:52 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1787183520' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T12:20:53.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:52 smithi183 ceph-mon[33277]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-21T12:20:53.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:52 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1787183520' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T12:20:53.277 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149730 got 111669149731 for osd.3 2024-09-21T12:20:53.277 DEBUG:teuthology.parallel:result is None 2024-09-21T12:20:53.844 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:20:53.923 INFO:teuthology.orchestra.run.smithi114.stdout:55834574896 2024-09-21T12:20:54.624 INFO:teuthology.orchestra.run.smithi114.stdout:77309411370 2024-09-21T12:20:54.982 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:54 smithi114 ceph-mon[40867]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T12:20:54.982 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:54 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:20:54.982 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:54 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:20:54.982 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:54 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1339000973' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T12:20:54.982 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:54 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/513048322' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T12:20:54.982 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:54 smithi114 ceph-mon[30324]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T12:20:54.983 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:54 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:20:54.983 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:54 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:20:54.983 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:54 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1339000973' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T12:20:54.983 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:54 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/513048322' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T12:20:55.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:54 smithi183 ceph-mon[33277]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T12:20:55.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:54 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:20:55.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:54 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:20:55.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:54 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1339000973' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T12:20:55.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:54 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/513048322' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T12:20:55.520 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411368 got 77309411370 for osd.2 2024-09-21T12:20:55.520 DEBUG:teuthology.parallel:result is None 2024-09-21T12:20:55.922 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574894 got 55834574896 for osd.1 2024-09-21T12:20:55.922 DEBUG:teuthology.parallel:result is None 2024-09-21T12:20:56.046 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:55 smithi114 ceph-mon[40867]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T12:20:56.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:55 smithi114 ceph-mon[30324]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T12:20:56.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:55 smithi183 ceph-mon[33277]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T12:20:56.254 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:20:56.654 INFO:teuthology.orchestra.run.smithi114.stdout:201863462931 2024-09-21T12:20:56.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:56 smithi183 ceph-mon[33277]: pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T12:20:56.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:56 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3636472341' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T12:20:56.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:56 smithi183 ceph-mon[33277]: osdmap e78: 8 total, 8 up, 8 in 2024-09-21T12:20:56.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:20:56 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:20:57.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:56 smithi114 ceph-mon[40867]: pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T12:20:57.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:56 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3636472341' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T12:20:57.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:56 smithi114 ceph-mon[40867]: osdmap e78: 8 total, 8 up, 8 in 2024-09-21T12:20:57.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:56 smithi114 ceph-mon[30324]: pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T12:20:57.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:56 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3636472341' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T12:20:57.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:56 smithi114 ceph-mon[30324]: osdmap e78: 8 total, 8 up, 8 in 2024-09-21T12:20:57.547 INFO:teuthology.orchestra.run.smithi114.stdout:167503724566 2024-09-21T12:20:58.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:57 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:20:58.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:57 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1153082863' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T12:20:58.245 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:57 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:20:58.245 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:57 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1153082863' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T12:20:58.246 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:57 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:20:58.246 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:57 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1153082863' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T12:20:58.274 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724564 got 167503724566 for osd.5 2024-09-21T12:20:58.274 DEBUG:teuthology.parallel:result is None 2024-09-21T12:20:58.694 INFO:teuthology.orchestra.run.smithi114.stdout:137438953503 2024-09-21T12:20:58.936 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462930 got 201863462931 for osd.6 2024-09-21T12:20:58.936 DEBUG:teuthology.parallel:result is None 2024-09-21T12:20:59.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:58 smithi183 ceph-mon[33277]: pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T12:20:59.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:20:58 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/390976474' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T12:20:59.244 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:58 smithi114 ceph-mon[40867]: pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T12:20:59.244 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:20:58 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/390976474' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T12:20:59.245 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:58 smithi114 ceph-mon[30324]: pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T12:20:59.245 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:20:58 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/390976474' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T12:20:59.591 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953502 got 137438953503 for osd.4 2024-09-21T12:20:59.591 DEBUG:teuthology.parallel:result is None 2024-09-21T12:20:59.591 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T12:20:59.591 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:20:59.865 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:21:00.592 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:21:00.594 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:21:00.972 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[30324]: pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T12:21:00.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:00.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:00.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:00.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:00.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T12:21:00.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T12:21:00.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T12:21:00.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T12:21:00.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T12:21:00.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T12:21:00.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T12:21:00.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T12:21:00.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:00.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:21:00.974 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:21:00.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[40867]: pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T12:21:00.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:00.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:00.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:00.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:00.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T12:21:00.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T12:21:00.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T12:21:00.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T12:21:00.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T12:21:00.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T12:21:00.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T12:21:00.975 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T12:21:00.975 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:00.975 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:21:00.975 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:00 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:21:01.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:00 smithi183 ceph-mon[33277]: pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T12:21:01.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:00 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:01.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:00 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:01.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:00 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:01.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:00 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:01.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:00 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T12:21:01.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:00 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T12:21:01.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:00 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T12:21:01.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:00 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T12:21:01.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:00 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T12:21:01.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:00 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T12:21:01.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:00 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T12:21:01.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:00 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T12:21:01.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:00 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:01.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:00 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:21:01.240 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:00 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:21:01.344 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-09-21T12:20:59.547647+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":884,"num_read_kb":741,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":245564,"kb_used_data":8540,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749715652,"statfs":{"total":767960285184,"available":767708827648,"internally_reserved":0,"allocated":8744960,"data_stored":4791089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001652"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779313+0000","last_change":"2024-09-21T12:20:15.309744+0000","last_active":"2024-09-21T12:20:54.779313+0000","last_peered":"2024-09-21T12:20:54.779313+0000","last_clean":"2024-09-21T12:20:54.779313+0000","last_became_active":"2024-09-21T12:20:15.309431+0000","last_became_peered":"2024-09-21T12:20:15.309431+0000","last_unstale":"2024-09-21T12:20:54.779313+0000","last_undegraded":"2024-09-21T12:20:54.779313+0000","last_fullsized":"2024-09-21T12:20:54.779313+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:37.321280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":60,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779173+0000","last_change":"2024-09-21T12:20:17.304357+0000","last_active":"2024-09-21T12:20:54.779173+0000","last_peered":"2024-09-21T12:20:54.779173+0000","last_clean":"2024-09-21T12:20:54.779173+0000","last_became_active":"2024-09-21T12:20:17.303856+0000","last_became_peered":"2024-09-21T12:20:17.303856+0000","last_unstale":"2024-09-21T12:20:54.779173+0000","last_undegraded":"2024-09-21T12:20:54.779173+0000","last_fullsized":"2024-09-21T12:20:54.779173+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:40.716880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":40,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.666951+0000","last_change":"2024-09-21T12:20:19.315239+0000","last_active":"2024-09-21T12:20:55.666951+0000","last_peered":"2024-09-21T12:20:55.666951+0000","last_clean":"2024-09-21T12:20:55.666951+0000","last_became_active":"2024-09-21T12:20:19.314908+0000","last_became_peered":"2024-09-21T12:20:19.314908+0000","last_unstale":"2024-09-21T12:20:55.666951+0000","last_undegraded":"2024-09-21T12:20:55.666951+0000","last_fullsized":"2024-09-21T12:20:55.666951+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:33.386500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779386+0000","last_change":"2024-09-21T12:20:21.308871+0000","last_active":"2024-09-21T12:20:54.779386+0000","last_peered":"2024-09-21T12:20:54.779386+0000","last_clean":"2024-09-21T12:20:54.779386+0000","last_became_active":"2024-09-21T12:20:21.308577+0000","last_became_peered":"2024-09-21T12:20:21.308577+0000","last_unstale":"2024-09-21T12:20:54.779386+0000","last_undegraded":"2024-09-21T12:20:54.779386+0000","last_fullsized":"2024-09-21T12:20:54.779386+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:43.912920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":35,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667619+0000","last_change":"2024-09-21T12:20:21.595607+0000","last_active":"2024-09-21T12:20:55.667619+0000","last_peered":"2024-09-21T12:20:55.667619+0000","last_clean":"2024-09-21T12:20:55.667619+0000","last_became_active":"2024-09-21T12:20:21.595301+0000","last_became_peered":"2024-09-21T12:20:21.595301+0000","last_unstale":"2024-09-21T12:20:55.667619+0000","last_undegraded":"2024-09-21T12:20:55.667619+0000","last_fullsized":"2024-09-21T12:20:55.667619+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:27.801303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":47,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.666718+0000","last_change":"2024-09-21T12:20:15.308941+0000","last_active":"2024-09-21T12:20:55.666718+0000","last_peered":"2024-09-21T12:20:55.666718+0000","last_clean":"2024-09-21T12:20:55.666718+0000","last_became_active":"2024-09-21T12:20:15.308655+0000","last_became_peered":"2024-09-21T12:20:15.308655+0000","last_unstale":"2024-09-21T12:20:55.666718+0000","last_undegraded":"2024-09-21T12:20:55.666718+0000","last_fullsized":"2024-09-21T12:20:55.666718+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:31.202541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":68,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667582+0000","last_change":"2024-09-21T12:20:17.305499+0000","last_active":"2024-09-21T12:20:55.667582+0000","last_peered":"2024-09-21T12:20:55.667582+0000","last_clean":"2024-09-21T12:20:55.667582+0000","last_became_active":"2024-09-21T12:20:17.305220+0000","last_became_peered":"2024-09-21T12:20:17.305220+0000","last_unstale":"2024-09-21T12:20:55.667582+0000","last_undegraded":"2024-09-21T12:20:55.667582+0000","last_fullsized":"2024-09-21T12:20:55.667582+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.370654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":40,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779173+0000","last_change":"2024-09-21T12:20:19.311730+0000","last_active":"2024-09-21T12:20:54.779173+0000","last_peered":"2024-09-21T12:20:54.779173+0000","last_clean":"2024-09-21T12:20:54.779173+0000","last_became_active":"2024-09-21T12:20:19.311403+0000","last_became_peered":"2024-09-21T12:20:19.311403+0000","last_unstale":"2024-09-21T12:20:54.779173+0000","last_undegraded":"2024-09-21T12:20:54.779173+0000","last_fullsized":"2024-09-21T12:20:54.779173+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:40.685703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.666981+0000","last_change":"2024-09-21T12:20:15.302150+0000","last_active":"2024-09-21T12:20:55.666981+0000","last_peered":"2024-09-21T12:20:55.666981+0000","last_clean":"2024-09-21T12:20:55.666981+0000","last_became_active":"2024-09-21T12:20:15.301847+0000","last_became_peered":"2024-09-21T12:20:15.301847+0000","last_unstale":"2024-09-21T12:20:55.666981+0000","last_undegraded":"2024-09-21T12:20:55.666981+0000","last_fullsized":"2024-09-21T12:20:55.666981+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:26.753083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":60,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779510+0000","last_change":"2024-09-21T12:20:17.307650+0000","last_active":"2024-09-21T12:20:54.779510+0000","last_peered":"2024-09-21T12:20:54.779510+0000","last_clean":"2024-09-21T12:20:54.779510+0000","last_became_active":"2024-09-21T12:20:17.307280+0000","last_became_peered":"2024-09-21T12:20:17.307280+0000","last_unstale":"2024-09-21T12:20:54.779510+0000","last_undegraded":"2024-09-21T12:20:54.779510+0000","last_fullsized":"2024-09-21T12:20:54.779510+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:34.375479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":40,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667704+0000","last_change":"2024-09-21T12:20:19.308520+0000","last_active":"2024-09-21T12:20:55.667704+0000","last_peered":"2024-09-21T12:20:55.667704+0000","last_clean":"2024-09-21T12:20:55.667704+0000","last_became_active":"2024-09-21T12:20:19.308184+0000","last_became_peered":"2024-09-21T12:20:19.308184+0000","last_unstale":"2024-09-21T12:20:55.667704+0000","last_undegraded":"2024-09-21T12:20:55.667704+0000","last_fullsized":"2024-09-21T12:20:55.667704+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:39.981449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.778828+0000","last_change":"2024-09-21T12:20:21.309871+0000","last_active":"2024-09-21T12:20:54.778828+0000","last_peered":"2024-09-21T12:20:54.778828+0000","last_clean":"2024-09-21T12:20:54.778828+0000","last_became_active":"2024-09-21T12:20:21.309190+0000","last_became_peered":"2024-09-21T12:20:21.309190+0000","last_unstale":"2024-09-21T12:20:54.778828+0000","last_undegraded":"2024-09-21T12:20:54.778828+0000","last_fullsized":"2024-09-21T12:20:54.778828+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.813677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.666811+0000","last_change":"2024-09-21T12:20:15.300728+0000","last_active":"2024-09-21T12:20:55.666811+0000","last_peered":"2024-09-21T12:20:55.666811+0000","last_clean":"2024-09-21T12:20:55.666811+0000","last_became_active":"2024-09-21T12:20:15.300441+0000","last_became_peered":"2024-09-21T12:20:15.300441+0000","last_unstale":"2024-09-21T12:20:55.666811+0000","last_undegraded":"2024-09-21T12:20:55.666811+0000","last_fullsized":"2024-09-21T12:20:55.666811+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:34.282857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":54,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779447+0000","last_change":"2024-09-21T12:20:17.305973+0000","last_active":"2024-09-21T12:20:54.779447+0000","last_peered":"2024-09-21T12:20:54.779447+0000","last_clean":"2024-09-21T12:20:54.779447+0000","last_became_active":"2024-09-21T12:20:17.305663+0000","last_became_peered":"2024-09-21T12:20:17.305663+0000","last_unstale":"2024-09-21T12:20:54.779447+0000","last_undegraded":"2024-09-21T12:20:54.779447+0000","last_fullsized":"2024-09-21T12:20:54.779447+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.420641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":40,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667577+0000","last_change":"2024-09-21T12:20:19.312502+0000","last_active":"2024-09-21T12:20:55.667577+0000","last_peered":"2024-09-21T12:20:55.667577+0000","last_clean":"2024-09-21T12:20:55.667577+0000","last_became_active":"2024-09-21T12:20:19.312242+0000","last_became_peered":"2024-09-21T12:20:19.312242+0000","last_unstale":"2024-09-21T12:20:55.667577+0000","last_undegraded":"2024-09-21T12:20:55.667577+0000","last_fullsized":"2024-09-21T12:20:55.667577+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:32.544179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667399+0000","last_change":"2024-09-21T12:20:21.311536+0000","last_active":"2024-09-21T12:20:55.667399+0000","last_peered":"2024-09-21T12:20:55.667399+0000","last_clean":"2024-09-21T12:20:55.667399+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:20:55.667399+0000","last_undegraded":"2024-09-21T12:20:55.667399+0000","last_fullsized":"2024-09-21T12:20:55.667399+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:41.239506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779985+0000","last_change":"2024-09-21T12:20:21.592944+0000","last_active":"2024-09-21T12:20:54.779985+0000","last_peered":"2024-09-21T12:20:54.779985+0000","last_clean":"2024-09-21T12:20:54.779985+0000","last_became_active":"2024-09-21T12:20:21.592631+0000","last_became_peered":"2024-09-21T12:20:21.592631+0000","last_unstale":"2024-09-21T12:20:54.779985+0000","last_undegraded":"2024-09-21T12:20:54.779985+0000","last_fullsized":"2024-09-21T12:20:54.779985+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:39.384088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779792+0000","last_change":"2024-09-21T12:20:15.302079+0000","last_active":"2024-09-21T12:20:54.779792+0000","last_peered":"2024-09-21T12:20:54.779792+0000","last_clean":"2024-09-21T12:20:54.779792+0000","last_became_active":"2024-09-21T12:20:15.301793+0000","last_became_peered":"2024-09-21T12:20:15.301793+0000","last_unstale":"2024-09-21T12:20:54.779792+0000","last_undegraded":"2024-09-21T12:20:54.779792+0000","last_fullsized":"2024-09-21T12:20:54.779792+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:19.991370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":69,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779217+0000","last_change":"2024-09-21T12:20:17.308186+0000","last_active":"2024-09-21T12:20:54.779217+0000","last_peered":"2024-09-21T12:20:54.779217+0000","last_clean":"2024-09-21T12:20:54.779217+0000","last_became_active":"2024-09-21T12:20:17.307828+0000","last_became_peered":"2024-09-21T12:20:17.307828+0000","last_unstale":"2024-09-21T12:20:54.779217+0000","last_undegraded":"2024-09-21T12:20:54.779217+0000","last_fullsized":"2024-09-21T12:20:54.779217+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:41.845334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":40,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667621+0000","last_change":"2024-09-21T12:20:19.316360+0000","last_active":"2024-09-21T12:20:55.667621+0000","last_peered":"2024-09-21T12:20:55.667621+0000","last_clean":"2024-09-21T12:20:55.667621+0000","last_became_active":"2024-09-21T12:20:19.315977+0000","last_became_peered":"2024-09-21T12:20:19.315977+0000","last_unstale":"2024-09-21T12:20:55.667621+0000","last_undegraded":"2024-09-21T12:20:55.667621+0000","last_fullsized":"2024-09-21T12:20:55.667621+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:37.515718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":35,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.668113+0000","last_change":"2024-09-21T12:20:21.593129+0000","last_active":"2024-09-21T12:20:55.668113+0000","last_peered":"2024-09-21T12:20:55.668113+0000","last_clean":"2024-09-21T12:20:55.668113+0000","last_became_active":"2024-09-21T12:20:21.592796+0000","last_became_peered":"2024-09-21T12:20:21.592796+0000","last_unstale":"2024-09-21T12:20:55.668113+0000","last_undegraded":"2024-09-21T12:20:55.668113+0000","last_fullsized":"2024-09-21T12:20:55.668113+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:47.880164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779676+0000","last_change":"2024-09-21T12:20:15.309390+0000","last_active":"2024-09-21T12:20:54.779676+0000","last_peered":"2024-09-21T12:20:54.779676+0000","last_clean":"2024-09-21T12:20:54.779676+0000","last_became_active":"2024-09-21T12:20:15.309023+0000","last_became_peered":"2024-09-21T12:20:15.309023+0000","last_unstale":"2024-09-21T12:20:54.779676+0000","last_undegraded":"2024-09-21T12:20:54.779676+0000","last_fullsized":"2024-09-21T12:20:54.779676+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:25.611326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":66,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667655+0000","last_change":"2024-09-21T12:20:17.310727+0000","last_active":"2024-09-21T12:20:55.667655+0000","last_peered":"2024-09-21T12:20:55.667655+0000","last_clean":"2024-09-21T12:20:55.667655+0000","last_became_active":"2024-09-21T12:20:17.310369+0000","last_became_peered":"2024-09-21T12:20:17.310369+0000","last_unstale":"2024-09-21T12:20:55.667655+0000","last_undegraded":"2024-09-21T12:20:55.667655+0000","last_fullsized":"2024-09-21T12:20:55.667655+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:22.563871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":40,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779717+0000","last_change":"2024-09-21T12:20:19.310859+0000","last_active":"2024-09-21T12:20:54.779717+0000","last_peered":"2024-09-21T12:20:54.779717+0000","last_clean":"2024-09-21T12:20:54.779717+0000","last_became_active":"2024-09-21T12:20:19.310575+0000","last_became_peered":"2024-09-21T12:20:19.310575+0000","last_unstale":"2024-09-21T12:20:54.779717+0000","last_undegraded":"2024-09-21T12:20:54.779717+0000","last_fullsized":"2024-09-21T12:20:54.779717+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.082897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":37,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667774+0000","last_change":"2024-09-21T12:20:21.313476+0000","last_active":"2024-09-21T12:20:55.667774+0000","last_peered":"2024-09-21T12:20:55.667774+0000","last_clean":"2024-09-21T12:20:55.667774+0000","last_became_active":"2024-09-21T12:20:21.312976+0000","last_became_peered":"2024-09-21T12:20:21.312976+0000","last_unstale":"2024-09-21T12:20:55.667774+0000","last_undegraded":"2024-09-21T12:20:55.667774+0000","last_fullsized":"2024-09-21T12:20:55.667774+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.367953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":49,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.666540+0000","last_change":"2024-09-21T12:20:15.299812+0000","last_active":"2024-09-21T12:20:55.666540+0000","last_peered":"2024-09-21T12:20:55.666540+0000","last_clean":"2024-09-21T12:20:55.666540+0000","last_became_active":"2024-09-21T12:20:15.299459+0000","last_became_peered":"2024-09-21T12:20:15.299459+0000","last_unstale":"2024-09-21T12:20:55.666540+0000","last_undegraded":"2024-09-21T12:20:55.666540+0000","last_fullsized":"2024-09-21T12:20:55.666540+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.458091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":59,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779313+0000","last_change":"2024-09-21T12:20:17.696423+0000","last_active":"2024-09-21T12:20:54.779313+0000","last_peered":"2024-09-21T12:20:54.779313+0000","last_clean":"2024-09-21T12:20:54.779313+0000","last_became_active":"2024-09-21T12:20:17.695977+0000","last_became_peered":"2024-09-21T12:20:17.695977+0000","last_unstale":"2024-09-21T12:20:54.779313+0000","last_undegraded":"2024-09-21T12:20:54.779313+0000","last_fullsized":"2024-09-21T12:20:54.779313+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:44.740918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":60,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.397014+0000","last_change":"2024-09-21T12:20:19.313262+0000","last_active":"2024-09-21T12:20:55.397014+0000","last_peered":"2024-09-21T12:20:55.397014+0000","last_clean":"2024-09-21T12:20:55.397014+0000","last_became_active":"2024-09-21T12:20:19.312403+0000","last_became_peered":"2024-09-21T12:20:19.312403+0000","last_unstale":"2024-09-21T12:20:55.397014+0000","last_undegraded":"2024-09-21T12:20:55.397014+0000","last_fullsized":"2024-09-21T12:20:55.397014+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:43.181150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":69,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667634+0000","last_change":"2024-09-21T12:20:17.305947+0000","last_active":"2024-09-21T12:20:55.667634+0000","last_peered":"2024-09-21T12:20:55.667634+0000","last_clean":"2024-09-21T12:20:55.667634+0000","last_became_active":"2024-09-21T12:20:17.305666+0000","last_became_peered":"2024-09-21T12:20:17.305666+0000","last_unstale":"2024-09-21T12:20:55.667634+0000","last_undegraded":"2024-09-21T12:20:55.667634+0000","last_fullsized":"2024-09-21T12:20:55.667634+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.503390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":47,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667036+0000","last_change":"2024-09-21T12:20:15.296251+0000","last_active":"2024-09-21T12:20:55.667036+0000","last_peered":"2024-09-21T12:20:55.667036+0000","last_clean":"2024-09-21T12:20:55.667036+0000","last_became_active":"2024-09-21T12:20:15.295774+0000","last_became_peered":"2024-09-21T12:20:15.295774+0000","last_unstale":"2024-09-21T12:20:55.667036+0000","last_undegraded":"2024-09-21T12:20:55.667036+0000","last_fullsized":"2024-09-21T12:20:55.667036+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:27.341994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":57,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.780160+0000","last_change":"2024-09-21T12:20:19.309874+0000","last_active":"2024-09-21T12:20:54.780160+0000","last_peered":"2024-09-21T12:20:54.780160+0000","last_clean":"2024-09-21T12:20:54.780160+0000","last_became_active":"2024-09-21T12:20:19.309606+0000","last_became_peered":"2024-09-21T12:20:19.309606+0000","last_unstale":"2024-09-21T12:20:54.780160+0000","last_undegraded":"2024-09-21T12:20:54.780160+0000","last_fullsized":"2024-09-21T12:20:54.780160+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:37.621763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":37,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.668109+0000","last_change":"2024-09-21T12:20:21.310295+0000","last_active":"2024-09-21T12:20:55.668109+0000","last_peered":"2024-09-21T12:20:55.668109+0000","last_clean":"2024-09-21T12:20:55.668109+0000","last_became_active":"2024-09-21T12:20:21.310022+0000","last_became_peered":"2024-09-21T12:20:21.310022+0000","last_unstale":"2024-09-21T12:20:55.668109+0000","last_undegraded":"2024-09-21T12:20:55.668109+0000","last_fullsized":"2024-09-21T12:20:55.668109+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:37.912528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":75,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.666879+0000","last_change":"2024-09-21T12:20:17.693550+0000","last_active":"2024-09-21T12:20:55.666879+0000","last_peered":"2024-09-21T12:20:55.666879+0000","last_clean":"2024-09-21T12:20:55.666879+0000","last_became_active":"2024-09-21T12:20:17.693179+0000","last_became_peered":"2024-09-21T12:20:17.693179+0000","last_unstale":"2024-09-21T12:20:55.666879+0000","last_undegraded":"2024-09-21T12:20:55.666879+0000","last_fullsized":"2024-09-21T12:20:55.666879+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:45.358986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":47,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.666575+0000","last_change":"2024-09-21T12:20:15.297621+0000","last_active":"2024-09-21T12:20:55.666575+0000","last_peered":"2024-09-21T12:20:55.666575+0000","last_clean":"2024-09-21T12:20:55.666575+0000","last_became_active":"2024-09-21T12:20:15.297340+0000","last_became_peered":"2024-09-21T12:20:15.297340+0000","last_unstale":"2024-09-21T12:20:55.666575+0000","last_undegraded":"2024-09-21T12:20:55.666575+0000","last_fullsized":"2024-09-21T12:20:55.666575+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:17.208372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":40,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779206+0000","last_change":"2024-09-21T12:20:19.315065+0000","last_active":"2024-09-21T12:20:54.779206+0000","last_peered":"2024-09-21T12:20:54.779206+0000","last_clean":"2024-09-21T12:20:54.779206+0000","last_became_active":"2024-09-21T12:20:19.314775+0000","last_became_peered":"2024-09-21T12:20:19.314775+0000","last_unstale":"2024-09-21T12:20:54.779206+0000","last_undegraded":"2024-09-21T12:20:54.779206+0000","last_fullsized":"2024-09-21T12:20:54.779206+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:26.384029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779516+0000","last_change":"2024-09-21T12:20:21.314735+0000","last_active":"2024-09-21T12:20:54.779516+0000","last_peered":"2024-09-21T12:20:54.779516+0000","last_clean":"2024-09-21T12:20:54.779516+0000","last_became_active":"2024-09-21T12:20:21.314433+0000","last_became_peered":"2024-09-21T12:20:21.314433+0000","last_unstale":"2024-09-21T12:20:54.779516+0000","last_undegraded":"2024-09-21T12:20:54.779516+0000","last_fullsized":"2024-09-21T12:20:54.779516+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:39.652862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":65,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779342+0000","last_change":"2024-09-21T12:20:17.301460+0000","last_active":"2024-09-21T12:20:54.779342+0000","last_peered":"2024-09-21T12:20:54.779342+0000","last_clean":"2024-09-21T12:20:54.779342+0000","last_became_active":"2024-09-21T12:20:17.301090+0000","last_became_peered":"2024-09-21T12:20:17.301090+0000","last_unstale":"2024-09-21T12:20:54.779342+0000","last_undegraded":"2024-09-21T12:20:54.779342+0000","last_fullsized":"2024-09-21T12:20:54.779342+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:34.835982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":49,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779715+0000","last_change":"2024-09-21T12:20:15.302238+0000","last_active":"2024-09-21T12:20:54.779715+0000","last_peered":"2024-09-21T12:20:54.779715+0000","last_clean":"2024-09-21T12:20:54.779715+0000","last_became_active":"2024-09-21T12:20:15.301904+0000","last_became_peered":"2024-09-21T12:20:15.301904+0000","last_unstale":"2024-09-21T12:20:54.779715+0000","last_undegraded":"2024-09-21T12:20:54.779715+0000","last_fullsized":"2024-09-21T12:20:54.779715+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:33.679116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":56,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.666634+0000","last_change":"2024-09-21T12:20:19.311362+0000","last_active":"2024-09-21T12:20:55.666634+0000","last_peered":"2024-09-21T12:20:55.666634+0000","last_clean":"2024-09-21T12:20:55.666634+0000","last_became_active":"2024-09-21T12:20:19.311056+0000","last_became_peered":"2024-09-21T12:20:19.311056+0000","last_unstale":"2024-09-21T12:20:55.666634+0000","last_undegraded":"2024-09-21T12:20:55.666634+0000","last_fullsized":"2024-09-21T12:20:55.666634+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:22.628895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667600+0000","last_change":"2024-09-21T12:20:21.593620+0000","last_active":"2024-09-21T12:20:55.667600+0000","last_peered":"2024-09-21T12:20:55.667600+0000","last_clean":"2024-09-21T12:20:55.667600+0000","last_became_active":"2024-09-21T12:20:21.593367+0000","last_became_peered":"2024-09-21T12:20:21.593367+0000","last_unstale":"2024-09-21T12:20:55.667600+0000","last_undegraded":"2024-09-21T12:20:55.667600+0000","last_fullsized":"2024-09-21T12:20:55.667600+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:20.553670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":59,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667210+0000","last_change":"2024-09-21T12:20:17.303245+0000","last_active":"2024-09-21T12:20:55.667210+0000","last_peered":"2024-09-21T12:20:55.667210+0000","last_clean":"2024-09-21T12:20:55.667210+0000","last_became_active":"2024-09-21T12:20:17.302871+0000","last_became_peered":"2024-09-21T12:20:17.302871+0000","last_unstale":"2024-09-21T12:20:55.667210+0000","last_undegraded":"2024-09-21T12:20:55.667210+0000","last_fullsized":"2024-09-21T12:20:55.667210+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.942911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.666850+0000","last_change":"2024-09-21T12:20:15.308877+0000","last_active":"2024-09-21T12:20:55.666850+0000","last_peered":"2024-09-21T12:20:55.666850+0000","last_clean":"2024-09-21T12:20:55.666850+0000","last_became_active":"2024-09-21T12:20:15.308518+0000","last_became_peered":"2024-09-21T12:20:15.308518+0000","last_unstale":"2024-09-21T12:20:55.666850+0000","last_undegraded":"2024-09-21T12:20:55.666850+0000","last_fullsized":"2024-09-21T12:20:55.666850+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:23.801832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":60,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779550+0000","last_change":"2024-09-21T12:20:19.307467+0000","last_active":"2024-09-21T12:20:54.779550+0000","last_peered":"2024-09-21T12:20:54.779550+0000","last_clean":"2024-09-21T12:20:54.779550+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:20:54.779550+0000","last_undegraded":"2024-09-21T12:20:54.779550+0000","last_fullsized":"2024-09-21T12:20:54.779550+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:21.225955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.778966+0000","last_change":"2024-09-21T12:20:21.316127+0000","last_active":"2024-09-21T12:20:54.778966+0000","last_peered":"2024-09-21T12:20:54.778966+0000","last_clean":"2024-09-21T12:20:54.778966+0000","last_became_active":"2024-09-21T12:20:21.315801+0000","last_became_peered":"2024-09-21T12:20:21.315801+0000","last_unstale":"2024-09-21T12:20:54.778966+0000","last_undegraded":"2024-09-21T12:20:54.778966+0000","last_fullsized":"2024-09-21T12:20:54.778966+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:28.234335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":80,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779629+0000","last_change":"2024-09-21T12:20:17.693585+0000","last_active":"2024-09-21T12:20:54.779629+0000","last_peered":"2024-09-21T12:20:54.779629+0000","last_clean":"2024-09-21T12:20:54.779629+0000","last_became_active":"2024-09-21T12:20:17.693067+0000","last_became_peered":"2024-09-21T12:20:17.693067+0000","last_unstale":"2024-09-21T12:20:54.779629+0000","last_undegraded":"2024-09-21T12:20:54.779629+0000","last_fullsized":"2024-09-21T12:20:54.779629+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:37.693764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.666707+0000","last_change":"2024-09-21T12:20:15.308741+0000","last_active":"2024-09-21T12:20:55.666707+0000","last_peered":"2024-09-21T12:20:55.666707+0000","last_clean":"2024-09-21T12:20:55.666707+0000","last_became_active":"2024-09-21T12:20:15.308441+0000","last_became_peered":"2024-09-21T12:20:15.308441+0000","last_unstale":"2024-09-21T12:20:55.666707+0000","last_undegraded":"2024-09-21T12:20:55.666707+0000","last_fullsized":"2024-09-21T12:20:55.666707+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:29.626513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":40,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.396383+0000","last_change":"2024-09-21T12:20:19.309945+0000","last_active":"2024-09-21T12:20:55.396383+0000","last_peered":"2024-09-21T12:20:55.396383+0000","last_clean":"2024-09-21T12:20:55.396383+0000","last_became_active":"2024-09-21T12:20:19.309598+0000","last_became_peered":"2024-09-21T12:20:19.309598+0000","last_unstale":"2024-09-21T12:20:55.396383+0000","last_undegraded":"2024-09-21T12:20:55.396383+0000","last_fullsized":"2024-09-21T12:20:55.396383+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:31.055740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.666869+0000","last_change":"2024-09-21T12:20:21.593173+0000","last_active":"2024-09-21T12:20:55.666869+0000","last_peered":"2024-09-21T12:20:55.666869+0000","last_clean":"2024-09-21T12:20:55.666869+0000","last_became_active":"2024-09-21T12:20:21.592828+0000","last_became_peered":"2024-09-21T12:20:21.592828+0000","last_unstale":"2024-09-21T12:20:55.666869+0000","last_undegraded":"2024-09-21T12:20:55.666869+0000","last_fullsized":"2024-09-21T12:20:55.666869+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.351266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":101,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667810+0000","last_change":"2024-09-21T12:20:17.308049+0000","last_active":"2024-09-21T12:20:55.667810+0000","last_peered":"2024-09-21T12:20:55.667810+0000","last_clean":"2024-09-21T12:20:55.667810+0000","last_became_active":"2024-09-21T12:20:17.307769+0000","last_became_peered":"2024-09-21T12:20:17.307769+0000","last_unstale":"2024-09-21T12:20:55.667810+0000","last_undegraded":"2024-09-21T12:20:55.667810+0000","last_fullsized":"2024-09-21T12:20:55.667810+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:29.183963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779774+0000","last_change":"2024-09-21T12:20:15.299429+0000","last_active":"2024-09-21T12:20:54.779774+0000","last_peered":"2024-09-21T12:20:54.779774+0000","last_clean":"2024-09-21T12:20:54.779774+0000","last_became_active":"2024-09-21T12:20:15.299143+0000","last_became_peered":"2024-09-21T12:20:15.299143+0000","last_unstale":"2024-09-21T12:20:54.779774+0000","last_undegraded":"2024-09-21T12:20:54.779774+0000","last_fullsized":"2024-09-21T12:20:54.779774+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:42.581678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":58,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.668084+0000","last_change":"2024-09-21T12:20:17.298211+0000","last_active":"2024-09-21T12:20:55.668084+0000","last_peered":"2024-09-21T12:20:55.668084+0000","last_clean":"2024-09-21T12:20:55.668084+0000","last_became_active":"2024-09-21T12:20:14.287998+0000","last_became_peered":"2024-09-21T12:20:14.287998+0000","last_unstale":"2024-09-21T12:20:55.668084+0000","last_undegraded":"2024-09-21T12:20:55.668084+0000","last_fullsized":"2024-09-21T12:20:55.668084+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:20:13.272075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:19.171111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011011149999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":40,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779436+0000","last_change":"2024-09-21T12:20:19.313478+0000","last_active":"2024-09-21T12:20:54.779436+0000","last_peered":"2024-09-21T12:20:54.779436+0000","last_clean":"2024-09-21T12:20:54.779436+0000","last_became_active":"2024-09-21T12:20:19.312170+0000","last_became_peered":"2024-09-21T12:20:19.312170+0000","last_unstale":"2024-09-21T12:20:54.779436+0000","last_undegraded":"2024-09-21T12:20:54.779436+0000","last_fullsized":"2024-09-21T12:20:54.779436+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.170409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":35,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667988+0000","last_change":"2024-09-21T12:20:21.314452+0000","last_active":"2024-09-21T12:20:55.667988+0000","last_peered":"2024-09-21T12:20:55.667988+0000","last_clean":"2024-09-21T12:20:55.667988+0000","last_became_active":"2024-09-21T12:20:21.314126+0000","last_became_peered":"2024-09-21T12:20:21.314126+0000","last_unstale":"2024-09-21T12:20:55.667988+0000","last_undegraded":"2024-09-21T12:20:55.667988+0000","last_fullsized":"2024-09-21T12:20:55.667988+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:44.311699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":71,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667644+0000","last_change":"2024-09-21T12:20:17.297678+0000","last_active":"2024-09-21T12:20:55.667644+0000","last_peered":"2024-09-21T12:20:55.667644+0000","last_clean":"2024-09-21T12:20:55.667644+0000","last_became_active":"2024-09-21T12:20:17.297374+0000","last_became_peered":"2024-09-21T12:20:17.297374+0000","last_unstale":"2024-09-21T12:20:55.667644+0000","last_undegraded":"2024-09-21T12:20:55.667644+0000","last_fullsized":"2024-09-21T12:20:55.667644+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:29.314219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.666830+0000","last_change":"2024-09-21T12:20:15.308648+0000","last_active":"2024-09-21T12:20:55.666830+0000","last_peered":"2024-09-21T12:20:55.666830+0000","last_clean":"2024-09-21T12:20:55.666830+0000","last_became_active":"2024-09-21T12:20:15.308356+0000","last_became_peered":"2024-09-21T12:20:15.308356+0000","last_unstale":"2024-09-21T12:20:55.666830+0000","last_undegraded":"2024-09-21T12:20:55.666830+0000","last_fullsized":"2024-09-21T12:20:55.666830+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:25.741492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":57,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.778808+0000","last_change":"2024-09-21T12:20:17.297432+0000","last_active":"2024-09-21T12:20:54.778808+0000","last_peered":"2024-09-21T12:20:54.778808+0000","last_clean":"2024-09-21T12:20:54.778808+0000","last_became_active":"2024-09-21T12:20:14.288358+0000","last_became_peered":"2024-09-21T12:20:14.288358+0000","last_unstale":"2024-09-21T12:20:54.778808+0000","last_undegraded":"2024-09-21T12:20:54.778808+0000","last_fullsized":"2024-09-21T12:20:54.778808+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:20:13.272075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:21.220031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019042449999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":40,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779877+0000","last_change":"2024-09-21T12:20:19.310164+0000","last_active":"2024-09-21T12:20:54.779877+0000","last_peered":"2024-09-21T12:20:54.779877+0000","last_clean":"2024-09-21T12:20:54.779877+0000","last_became_active":"2024-09-21T12:20:19.309724+0000","last_became_peered":"2024-09-21T12:20:19.309724+0000","last_unstale":"2024-09-21T12:20:54.779877+0000","last_undegraded":"2024-09-21T12:20:54.779877+0000","last_fullsized":"2024-09-21T12:20:54.779877+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:43.623687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.668569+0000","last_change":"2024-09-21T12:20:21.593267+0000","last_active":"2024-09-21T12:20:55.668569+0000","last_peered":"2024-09-21T12:20:55.668569+0000","last_clean":"2024-09-21T12:20:55.668569+0000","last_became_active":"2024-09-21T12:20:21.592982+0000","last_became_peered":"2024-09-21T12:20:21.592982+0000","last_unstale":"2024-09-21T12:20:55.668569+0000","last_undegraded":"2024-09-21T12:20:55.668569+0000","last_fullsized":"2024-09-21T12:20:55.668569+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:43.824339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":62,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779539+0000","last_change":"2024-09-21T12:20:17.299646+0000","last_active":"2024-09-21T12:20:54.779539+0000","last_peered":"2024-09-21T12:20:54.779539+0000","last_clean":"2024-09-21T12:20:54.779539+0000","last_became_active":"2024-09-21T12:20:17.299325+0000","last_became_peered":"2024-09-21T12:20:17.299325+0000","last_unstale":"2024-09-21T12:20:54.779539+0000","last_undegraded":"2024-09-21T12:20:54.779539+0000","last_fullsized":"2024-09-21T12:20:54.779539+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:22.183786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.778915+0000","last_change":"2024-09-21T12:20:15.302177+0000","last_active":"2024-09-21T12:20:54.778915+0000","last_peered":"2024-09-21T12:20:54.778915+0000","last_clean":"2024-09-21T12:20:54.778915+0000","last_became_active":"2024-09-21T12:20:15.301859+0000","last_became_peered":"2024-09-21T12:20:15.301859+0000","last_unstale":"2024-09-21T12:20:54.778915+0000","last_undegraded":"2024-09-21T12:20:54.778915+0000","last_fullsized":"2024-09-21T12:20:54.778915+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.505232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":95,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.668040+0000","last_change":"2024-09-21T12:20:17.693865+0000","last_active":"2024-09-21T12:20:55.668040+0000","last_peered":"2024-09-21T12:20:55.668040+0000","last_clean":"2024-09-21T12:20:55.668040+0000","last_became_active":"2024-09-21T12:20:14.287543+0000","last_became_peered":"2024-09-21T12:20:14.287543+0000","last_unstale":"2024-09-21T12:20:55.668040+0000","last_undegraded":"2024-09-21T12:20:55.668040+0000","last_fullsized":"2024-09-21T12:20:55.668040+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:20:13.272075+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:21.111464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":36,"num_read_kb":31,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":40,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.668576+0000","last_change":"2024-09-21T12:20:19.314574+0000","last_active":"2024-09-21T12:20:55.668576+0000","last_peered":"2024-09-21T12:20:55.668576+0000","last_clean":"2024-09-21T12:20:55.668576+0000","last_became_active":"2024-09-21T12:20:19.314217+0000","last_became_peered":"2024-09-21T12:20:19.314217+0000","last_unstale":"2024-09-21T12:20:55.668576+0000","last_undegraded":"2024-09-21T12:20:55.668576+0000","last_fullsized":"2024-09-21T12:20:55.668576+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:23.172823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667390+0000","last_change":"2024-09-21T12:20:21.311501+0000","last_active":"2024-09-21T12:20:55.667390+0000","last_peered":"2024-09-21T12:20:55.667390+0000","last_clean":"2024-09-21T12:20:55.667390+0000","last_became_active":"2024-09-21T12:20:21.311188+0000","last_became_peered":"2024-09-21T12:20:21.311188+0000","last_unstale":"2024-09-21T12:20:55.667390+0000","last_undegraded":"2024-09-21T12:20:55.667390+0000","last_fullsized":"2024-09-21T12:20:55.667390+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:27.755252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"66'16","reported_seq":75,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.396859+0000","last_change":"2024-09-21T12:20:17.696268+0000","last_active":"2024-09-21T12:20:55.396859+0000","last_peered":"2024-09-21T12:20:55.396859+0000","last_clean":"2024-09-21T12:20:55.396859+0000","last_became_active":"2024-09-21T12:20:17.695940+0000","last_became_peered":"2024-09-21T12:20:17.695940+0000","last_unstale":"2024-09-21T12:20:55.396859+0000","last_undegraded":"2024-09-21T12:20:55.396859+0000","last_fullsized":"2024-09-21T12:20:55.396859+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:39.599877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":53,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667855+0000","last_change":"2024-09-21T12:20:15.299151+0000","last_active":"2024-09-21T12:20:55.667855+0000","last_peered":"2024-09-21T12:20:55.667855+0000","last_clean":"2024-09-21T12:20:55.667855+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:20:55.667855+0000","last_undegraded":"2024-09-21T12:20:55.667855+0000","last_fullsized":"2024-09-21T12:20:55.667855+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:16.205349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":93,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.668195+0000","last_change":"2024-09-21T12:20:24.850239+0000","last_active":"2024-09-21T12:20:55.668195+0000","last_peered":"2024-09-21T12:20:55.668195+0000","last_clean":"2024-09-21T12:20:55.668195+0000","last_became_active":"2024-09-21T12:20:06.349670+0000","last_became_peered":"2024-09-21T12:20:06.349670+0000","last_unstale":"2024-09-21T12:20:55.668195+0000","last_undegraded":"2024-09-21T12:20:55.668195+0000","last_fullsized":"2024-09-21T12:20:55.668195+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T12:20:24.850157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:20:24.850157+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:53.046478+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":40,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667966+0000","last_change":"2024-09-21T12:20:19.310280+0000","last_active":"2024-09-21T12:20:55.667966+0000","last_peered":"2024-09-21T12:20:55.667966+0000","last_clean":"2024-09-21T12:20:55.667966+0000","last_became_active":"2024-09-21T12:20:19.309859+0000","last_became_peered":"2024-09-21T12:20:19.309859+0000","last_unstale":"2024-09-21T12:20:55.667966+0000","last_undegraded":"2024-09-21T12:20:55.667966+0000","last_fullsized":"2024-09-21T12:20:55.667966+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:43.850206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.668269+0000","last_change":"2024-09-21T12:20:21.311690+0000","last_active":"2024-09-21T12:20:55.668269+0000","last_peered":"2024-09-21T12:20:55.668269+0000","last_clean":"2024-09-21T12:20:55.668269+0000","last_became_active":"2024-09-21T12:20:21.311194+0000","last_became_peered":"2024-09-21T12:20:21.311194+0000","last_unstale":"2024-09-21T12:20:55.668269+0000","last_undegraded":"2024-09-21T12:20:55.668269+0000","last_fullsized":"2024-09-21T12:20:55.668269+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:25.752002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":63,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.780164+0000","last_change":"2024-09-21T12:20:17.310299+0000","last_active":"2024-09-21T12:20:54.780164+0000","last_peered":"2024-09-21T12:20:54.780164+0000","last_clean":"2024-09-21T12:20:54.780164+0000","last_became_active":"2024-09-21T12:20:17.309930+0000","last_became_peered":"2024-09-21T12:20:17.309930+0000","last_unstale":"2024-09-21T12:20:54.780164+0000","last_undegraded":"2024-09-21T12:20:54.780164+0000","last_fullsized":"2024-09-21T12:20:54.780164+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:17.883258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779545+0000","last_change":"2024-09-21T12:20:15.310346+0000","last_active":"2024-09-21T12:20:54.779545+0000","last_peered":"2024-09-21T12:20:54.779545+0000","last_clean":"2024-09-21T12:20:54.779545+0000","last_became_active":"2024-09-21T12:20:15.310048+0000","last_became_peered":"2024-09-21T12:20:15.310048+0000","last_unstale":"2024-09-21T12:20:54.779545+0000","last_undegraded":"2024-09-21T12:20:54.779545+0000","last_fullsized":"2024-09-21T12:20:54.779545+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:31.701321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":40,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.668227+0000","last_change":"2024-09-21T12:20:19.309554+0000","last_active":"2024-09-21T12:20:55.668227+0000","last_peered":"2024-09-21T12:20:55.668227+0000","last_clean":"2024-09-21T12:20:55.668227+0000","last_became_active":"2024-09-21T12:20:19.309258+0000","last_became_peered":"2024-09-21T12:20:19.309258+0000","last_unstale":"2024-09-21T12:20:55.668227+0000","last_undegraded":"2024-09-21T12:20:55.668227+0000","last_fullsized":"2024-09-21T12:20:55.668227+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:44.940960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":35,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667816+0000","last_change":"2024-09-21T12:20:21.593145+0000","last_active":"2024-09-21T12:20:55.667816+0000","last_peered":"2024-09-21T12:20:55.667816+0000","last_clean":"2024-09-21T12:20:55.667816+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:20:55.667816+0000","last_undegraded":"2024-09-21T12:20:55.667816+0000","last_fullsized":"2024-09-21T12:20:55.667816+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.236407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":72,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.780095+0000","last_change":"2024-09-21T12:20:17.310292+0000","last_active":"2024-09-21T12:20:54.780095+0000","last_peered":"2024-09-21T12:20:54.780095+0000","last_clean":"2024-09-21T12:20:54.780095+0000","last_became_active":"2024-09-21T12:20:17.309828+0000","last_became_peered":"2024-09-21T12:20:17.309828+0000","last_unstale":"2024-09-21T12:20:54.780095+0000","last_undegraded":"2024-09-21T12:20:54.780095+0000","last_fullsized":"2024-09-21T12:20:54.780095+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:44.016276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.396032+0000","last_change":"2024-09-21T12:20:15.302137+0000","last_active":"2024-09-21T12:20:55.396032+0000","last_peered":"2024-09-21T12:20:55.396032+0000","last_clean":"2024-09-21T12:20:55.396032+0000","last_became_active":"2024-09-21T12:20:15.301851+0000","last_became_peered":"2024-09-21T12:20:15.301851+0000","last_unstale":"2024-09-21T12:20:55.396032+0000","last_undegraded":"2024-09-21T12:20:55.396032+0000","last_fullsized":"2024-09-21T12:20:55.396032+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.661202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":40,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667472+0000","last_change":"2024-09-21T12:20:19.312912+0000","last_active":"2024-09-21T12:20:55.667472+0000","last_peered":"2024-09-21T12:20:55.667472+0000","last_clean":"2024-09-21T12:20:55.667472+0000","last_became_active":"2024-09-21T12:20:19.312301+0000","last_became_peered":"2024-09-21T12:20:19.312301+0000","last_unstale":"2024-09-21T12:20:55.667472+0000","last_undegraded":"2024-09-21T12:20:55.667472+0000","last_fullsized":"2024-09-21T12:20:55.667472+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:42.511905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779876+0000","last_change":"2024-09-21T12:20:21.313585+0000","last_active":"2024-09-21T12:20:54.779876+0000","last_peered":"2024-09-21T12:20:54.779876+0000","last_clean":"2024-09-21T12:20:54.779876+0000","last_became_active":"2024-09-21T12:20:21.313254+0000","last_became_peered":"2024-09-21T12:20:21.313254+0000","last_unstale":"2024-09-21T12:20:54.779876+0000","last_undegraded":"2024-09-21T12:20:54.779876+0000","last_fullsized":"2024-09-21T12:20:54.779876+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.442191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":59,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779774+0000","last_change":"2024-09-21T12:20:17.307651+0000","last_active":"2024-09-21T12:20:54.779774+0000","last_peered":"2024-09-21T12:20:54.779774+0000","last_clean":"2024-09-21T12:20:54.779774+0000","last_became_active":"2024-09-21T12:20:17.307317+0000","last_became_peered":"2024-09-21T12:20:17.307317+0000","last_unstale":"2024-09-21T12:20:54.779774+0000","last_undegraded":"2024-09-21T12:20:54.779774+0000","last_fullsized":"2024-09-21T12:20:54.779774+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:37.610294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":47,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.666464+0000","last_change":"2024-09-21T12:20:15.298107+0000","last_active":"2024-09-21T12:20:55.666464+0000","last_peered":"2024-09-21T12:20:55.666464+0000","last_clean":"2024-09-21T12:20:55.666464+0000","last_became_active":"2024-09-21T12:20:15.297819+0000","last_became_peered":"2024-09-21T12:20:15.297819+0000","last_unstale":"2024-09-21T12:20:55.666464+0000","last_undegraded":"2024-09-21T12:20:55.666464+0000","last_fullsized":"2024-09-21T12:20:55.666464+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:38.788978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":57,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779536+0000","last_change":"2024-09-21T12:20:19.308246+0000","last_active":"2024-09-21T12:20:54.779536+0000","last_peered":"2024-09-21T12:20:54.779536+0000","last_clean":"2024-09-21T12:20:54.779536+0000","last_became_active":"2024-09-21T12:20:19.307924+0000","last_became_peered":"2024-09-21T12:20:19.307924+0000","last_unstale":"2024-09-21T12:20:54.779536+0000","last_undegraded":"2024-09-21T12:20:54.779536+0000","last_fullsized":"2024-09-21T12:20:54.779536+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:33.476030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779942+0000","last_change":"2024-09-21T12:20:21.313250+0000","last_active":"2024-09-21T12:20:54.779942+0000","last_peered":"2024-09-21T12:20:54.779942+0000","last_clean":"2024-09-21T12:20:54.779942+0000","last_became_active":"2024-09-21T12:20:21.312954+0000","last_became_peered":"2024-09-21T12:20:21.312954+0000","last_unstale":"2024-09-21T12:20:54.779942+0000","last_undegraded":"2024-09-21T12:20:54.779942+0000","last_fullsized":"2024-09-21T12:20:54.779942+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.513060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":60,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779521+0000","last_change":"2024-09-21T12:20:17.299948+0000","last_active":"2024-09-21T12:20:54.779521+0000","last_peered":"2024-09-21T12:20:54.779521+0000","last_clean":"2024-09-21T12:20:54.779521+0000","last_became_active":"2024-09-21T12:20:17.299653+0000","last_became_peered":"2024-09-21T12:20:17.299653+0000","last_unstale":"2024-09-21T12:20:54.779521+0000","last_undegraded":"2024-09-21T12:20:54.779521+0000","last_fullsized":"2024-09-21T12:20:54.779521+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:16.539847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.668002+0000","last_change":"2024-09-21T12:20:15.302587+0000","last_active":"2024-09-21T12:20:55.668002+0000","last_peered":"2024-09-21T12:20:55.668002+0000","last_clean":"2024-09-21T12:20:55.668002+0000","last_became_active":"2024-09-21T12:20:15.302262+0000","last_became_peered":"2024-09-21T12:20:15.302262+0000","last_unstale":"2024-09-21T12:20:55.668002+0000","last_undegraded":"2024-09-21T12:20:55.668002+0000","last_fullsized":"2024-09-21T12:20:55.668002+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:32.093775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":40,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779198+0000","last_change":"2024-09-21T12:20:19.308484+0000","last_active":"2024-09-21T12:20:54.779198+0000","last_peered":"2024-09-21T12:20:54.779198+0000","last_clean":"2024-09-21T12:20:54.779198+0000","last_became_active":"2024-09-21T12:20:19.308048+0000","last_became_peered":"2024-09-21T12:20:19.308048+0000","last_unstale":"2024-09-21T12:20:54.779198+0000","last_undegraded":"2024-09-21T12:20:54.779198+0000","last_fullsized":"2024-09-21T12:20:54.779198+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:34.470562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.778834+0000","last_change":"2024-09-21T12:20:21.311799+0000","last_active":"2024-09-21T12:20:54.778834+0000","last_peered":"2024-09-21T12:20:54.778834+0000","last_clean":"2024-09-21T12:20:54.778834+0000","last_became_active":"2024-09-21T12:20:21.311493+0000","last_became_peered":"2024-09-21T12:20:21.311493+0000","last_unstale":"2024-09-21T12:20:54.778834+0000","last_undegraded":"2024-09-21T12:20:54.778834+0000","last_fullsized":"2024-09-21T12:20:54.778834+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:40.676620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":75,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.396459+0000","last_change":"2024-09-21T12:20:17.693677+0000","last_active":"2024-09-21T12:20:55.396459+0000","last_peered":"2024-09-21T12:20:55.396459+0000","last_clean":"2024-09-21T12:20:55.396459+0000","last_became_active":"2024-09-21T12:20:17.692867+0000","last_became_peered":"2024-09-21T12:20:17.692867+0000","last_unstale":"2024-09-21T12:20:55.396459+0000","last_undegraded":"2024-09-21T12:20:55.396459+0000","last_fullsized":"2024-09-21T12:20:55.396459+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.747644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.668723+0000","last_change":"2024-09-21T12:20:15.300854+0000","last_active":"2024-09-21T12:20:55.668723+0000","last_peered":"2024-09-21T12:20:55.668723+0000","last_clean":"2024-09-21T12:20:55.668723+0000","last_became_active":"2024-09-21T12:20:15.299333+0000","last_became_peered":"2024-09-21T12:20:15.299333+0000","last_unstale":"2024-09-21T12:20:55.668723+0000","last_undegraded":"2024-09-21T12:20:55.668723+0000","last_fullsized":"2024-09-21T12:20:55.668723+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:34.912500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":40,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779348+0000","last_change":"2024-09-21T12:20:19.307316+0000","last_active":"2024-09-21T12:20:54.779348+0000","last_peered":"2024-09-21T12:20:54.779348+0000","last_clean":"2024-09-21T12:20:54.779348+0000","last_became_active":"2024-09-21T12:20:19.306952+0000","last_became_peered":"2024-09-21T12:20:19.306952+0000","last_unstale":"2024-09-21T12:20:54.779348+0000","last_undegraded":"2024-09-21T12:20:54.779348+0000","last_fullsized":"2024-09-21T12:20:54.779348+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:47.444300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.668092+0000","last_change":"2024-09-21T12:20:21.314891+0000","last_active":"2024-09-21T12:20:55.668092+0000","last_peered":"2024-09-21T12:20:55.668092+0000","last_clean":"2024-09-21T12:20:55.668092+0000","last_became_active":"2024-09-21T12:20:21.314545+0000","last_became_peered":"2024-09-21T12:20:21.314545+0000","last_unstale":"2024-09-21T12:20:55.668092+0000","last_undegraded":"2024-09-21T12:20:55.668092+0000","last_fullsized":"2024-09-21T12:20:55.668092+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:21.854833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":67,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779402+0000","last_change":"2024-09-21T12:20:17.306090+0000","last_active":"2024-09-21T12:20:54.779402+0000","last_peered":"2024-09-21T12:20:54.779402+0000","last_clean":"2024-09-21T12:20:54.779402+0000","last_became_active":"2024-09-21T12:20:17.305661+0000","last_became_peered":"2024-09-21T12:20:17.305661+0000","last_unstale":"2024-09-21T12:20:54.779402+0000","last_undegraded":"2024-09-21T12:20:54.779402+0000","last_fullsized":"2024-09-21T12:20:54.779402+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:31.017952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667191+0000","last_change":"2024-09-21T12:20:15.305876+0000","last_active":"2024-09-21T12:20:55.667191+0000","last_peered":"2024-09-21T12:20:55.667191+0000","last_clean":"2024-09-21T12:20:55.667191+0000","last_became_active":"2024-09-21T12:20:15.305545+0000","last_became_peered":"2024-09-21T12:20:15.305545+0000","last_unstale":"2024-09-21T12:20:55.667191+0000","last_undegraded":"2024-09-21T12:20:55.667191+0000","last_fullsized":"2024-09-21T12:20:55.667191+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:19.227777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":40,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.778582+0000","last_change":"2024-09-21T12:20:19.312986+0000","last_active":"2024-09-21T12:20:54.778582+0000","last_peered":"2024-09-21T12:20:54.778582+0000","last_clean":"2024-09-21T12:20:54.778582+0000","last_became_active":"2024-09-21T12:20:19.312572+0000","last_became_peered":"2024-09-21T12:20:19.312572+0000","last_unstale":"2024-09-21T12:20:54.778582+0000","last_undegraded":"2024-09-21T12:20:54.778582+0000","last_fullsized":"2024-09-21T12:20:54.778582+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:20.301869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":35,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667664+0000","last_change":"2024-09-21T12:20:21.317550+0000","last_active":"2024-09-21T12:20:55.667664+0000","last_peered":"2024-09-21T12:20:55.667664+0000","last_clean":"2024-09-21T12:20:55.667664+0000","last_became_active":"2024-09-21T12:20:21.317223+0000","last_became_peered":"2024-09-21T12:20:21.317223+0000","last_unstale":"2024-09-21T12:20:55.667664+0000","last_undegraded":"2024-09-21T12:20:55.667664+0000","last_fullsized":"2024-09-21T12:20:55.667664+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:25.406157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":69,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667244+0000","last_change":"2024-09-21T12:20:17.693302+0000","last_active":"2024-09-21T12:20:55.667244+0000","last_peered":"2024-09-21T12:20:55.667244+0000","last_clean":"2024-09-21T12:20:55.667244+0000","last_became_active":"2024-09-21T12:20:17.692870+0000","last_became_peered":"2024-09-21T12:20:17.692870+0000","last_unstale":"2024-09-21T12:20:55.667244+0000","last_undegraded":"2024-09-21T12:20:55.667244+0000","last_fullsized":"2024-09-21T12:20:55.667244+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.102470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":60,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667348+0000","last_change":"2024-09-21T12:20:15.302366+0000","last_active":"2024-09-21T12:20:55.667348+0000","last_peered":"2024-09-21T12:20:55.667348+0000","last_clean":"2024-09-21T12:20:55.667348+0000","last_became_active":"2024-09-21T12:20:15.302038+0000","last_became_peered":"2024-09-21T12:20:15.302038+0000","last_unstale":"2024-09-21T12:20:55.667348+0000","last_undegraded":"2024-09-21T12:20:55.667348+0000","last_fullsized":"2024-09-21T12:20:55.667348+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:29.184795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":57,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.668196+0000","last_change":"2024-09-21T12:20:19.310188+0000","last_active":"2024-09-21T12:20:55.668196+0000","last_peered":"2024-09-21T12:20:55.668196+0000","last_clean":"2024-09-21T12:20:55.668196+0000","last_became_active":"2024-09-21T12:20:19.309708+0000","last_became_peered":"2024-09-21T12:20:19.309708+0000","last_unstale":"2024-09-21T12:20:55.668196+0000","last_undegraded":"2024-09-21T12:20:55.668196+0000","last_fullsized":"2024-09-21T12:20:55.668196+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:44.922935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667330+0000","last_change":"2024-09-21T12:20:21.592893+0000","last_active":"2024-09-21T12:20:55.667330+0000","last_peered":"2024-09-21T12:20:55.667330+0000","last_clean":"2024-09-21T12:20:55.667330+0000","last_became_active":"2024-09-21T12:20:21.592576+0000","last_became_peered":"2024-09-21T12:20:21.592576+0000","last_unstale":"2024-09-21T12:20:55.667330+0000","last_undegraded":"2024-09-21T12:20:55.667330+0000","last_fullsized":"2024-09-21T12:20:55.667330+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:45.792911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.396026+0000","last_change":"2024-09-21T12:20:15.303770+0000","last_active":"2024-09-21T12:20:55.396026+0000","last_peered":"2024-09-21T12:20:55.396026+0000","last_clean":"2024-09-21T12:20:55.396026+0000","last_became_active":"2024-09-21T12:20:15.303480+0000","last_became_peered":"2024-09-21T12:20:15.303480+0000","last_unstale":"2024-09-21T12:20:55.396026+0000","last_undegraded":"2024-09-21T12:20:55.396026+0000","last_fullsized":"2024-09-21T12:20:55.396026+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:14.876724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":52,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.668223+0000","last_change":"2024-09-21T12:20:17.694977+0000","last_active":"2024-09-21T12:20:55.668223+0000","last_peered":"2024-09-21T12:20:55.668223+0000","last_clean":"2024-09-21T12:20:55.668223+0000","last_became_active":"2024-09-21T12:20:17.694628+0000","last_became_peered":"2024-09-21T12:20:17.694628+0000","last_unstale":"2024-09-21T12:20:55.668223+0000","last_undegraded":"2024-09-21T12:20:55.668223+0000","last_fullsized":"2024-09-21T12:20:55.668223+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.288937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":40,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667863+0000","last_change":"2024-09-21T12:20:19.314432+0000","last_active":"2024-09-21T12:20:55.667863+0000","last_peered":"2024-09-21T12:20:55.667863+0000","last_clean":"2024-09-21T12:20:55.667863+0000","last_became_active":"2024-09-21T12:20:19.313882+0000","last_became_peered":"2024-09-21T12:20:19.313882+0000","last_unstale":"2024-09-21T12:20:55.667863+0000","last_undegraded":"2024-09-21T12:20:55.667863+0000","last_fullsized":"2024-09-21T12:20:55.667863+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.453804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.668400+0000","last_change":"2024-09-21T12:20:21.594311+0000","last_active":"2024-09-21T12:20:55.668400+0000","last_peered":"2024-09-21T12:20:55.668400+0000","last_clean":"2024-09-21T12:20:55.668400+0000","last_became_active":"2024-09-21T12:20:21.594048+0000","last_became_peered":"2024-09-21T12:20:21.594048+0000","last_unstale":"2024-09-21T12:20:55.668400+0000","last_undegraded":"2024-09-21T12:20:55.668400+0000","last_fullsized":"2024-09-21T12:20:55.668400+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:38.347489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":60,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779623+0000","last_change":"2024-09-21T12:20:17.696279+0000","last_active":"2024-09-21T12:20:54.779623+0000","last_peered":"2024-09-21T12:20:54.779623+0000","last_clean":"2024-09-21T12:20:54.779623+0000","last_became_active":"2024-09-21T12:20:17.695898+0000","last_became_peered":"2024-09-21T12:20:17.695898+0000","last_unstale":"2024-09-21T12:20:54.779623+0000","last_undegraded":"2024-09-21T12:20:54.779623+0000","last_fullsized":"2024-09-21T12:20:54.779623+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:21.926785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.666845+0000","last_change":"2024-09-21T12:20:15.301560+0000","last_active":"2024-09-21T12:20:55.666845+0000","last_peered":"2024-09-21T12:20:55.666845+0000","last_clean":"2024-09-21T12:20:55.666845+0000","last_became_active":"2024-09-21T12:20:15.300722+0000","last_became_peered":"2024-09-21T12:20:15.300722+0000","last_unstale":"2024-09-21T12:20:55.666845+0000","last_undegraded":"2024-09-21T12:20:55.666845+0000","last_fullsized":"2024-09-21T12:20:55.666845+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.142850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":39,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.668160+0000","last_change":"2024-09-21T12:20:19.312248+0000","last_active":"2024-09-21T12:20:55.668160+0000","last_peered":"2024-09-21T12:20:55.668160+0000","last_clean":"2024-09-21T12:20:55.668160+0000","last_became_active":"2024-09-21T12:20:19.311947+0000","last_became_peered":"2024-09-21T12:20:19.311947+0000","last_unstale":"2024-09-21T12:20:55.668160+0000","last_undegraded":"2024-09-21T12:20:55.668160+0000","last_fullsized":"2024-09-21T12:20:55.668160+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.388400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.778864+0000","last_change":"2024-09-21T12:20:21.314554+0000","last_active":"2024-09-21T12:20:54.778864+0000","last_peered":"2024-09-21T12:20:54.778864+0000","last_clean":"2024-09-21T12:20:54.778864+0000","last_became_active":"2024-09-21T12:20:21.314277+0000","last_became_peered":"2024-09-21T12:20:21.314277+0000","last_unstale":"2024-09-21T12:20:54.778864+0000","last_undegraded":"2024-09-21T12:20:54.778864+0000","last_fullsized":"2024-09-21T12:20:54.778864+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.219191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":60,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.668391+0000","last_change":"2024-09-21T12:20:17.694904+0000","last_active":"2024-09-21T12:20:55.668391+0000","last_peered":"2024-09-21T12:20:55.668391+0000","last_clean":"2024-09-21T12:20:55.668391+0000","last_became_active":"2024-09-21T12:20:17.694568+0000","last_became_peered":"2024-09-21T12:20:17.694568+0000","last_unstale":"2024-09-21T12:20:55.668391+0000","last_undegraded":"2024-09-21T12:20:55.668391+0000","last_fullsized":"2024-09-21T12:20:55.668391+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:24.206067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779008+0000","last_change":"2024-09-21T12:20:15.297853+0000","last_active":"2024-09-21T12:20:54.779008+0000","last_peered":"2024-09-21T12:20:54.779008+0000","last_clean":"2024-09-21T12:20:54.779008+0000","last_became_active":"2024-09-21T12:20:15.297519+0000","last_became_peered":"2024-09-21T12:20:15.297519+0000","last_unstale":"2024-09-21T12:20:54.779008+0000","last_undegraded":"2024-09-21T12:20:54.779008+0000","last_fullsized":"2024-09-21T12:20:54.779008+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:29.814404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":56,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667962+0000","last_change":"2024-09-21T12:20:19.305689+0000","last_active":"2024-09-21T12:20:55.667962+0000","last_peered":"2024-09-21T12:20:55.667962+0000","last_clean":"2024-09-21T12:20:55.667962+0000","last_became_active":"2024-09-21T12:20:19.305364+0000","last_became_peered":"2024-09-21T12:20:19.305364+0000","last_unstale":"2024-09-21T12:20:55.667962+0000","last_undegraded":"2024-09-21T12:20:55.667962+0000","last_fullsized":"2024-09-21T12:20:55.667962+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.650981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.780157+0000","last_change":"2024-09-21T12:20:21.315079+0000","last_active":"2024-09-21T12:20:54.780157+0000","last_peered":"2024-09-21T12:20:54.780157+0000","last_clean":"2024-09-21T12:20:54.780157+0000","last_became_active":"2024-09-21T12:20:21.314066+0000","last_became_peered":"2024-09-21T12:20:21.314066+0000","last_unstale":"2024-09-21T12:20:54.780157+0000","last_undegraded":"2024-09-21T12:20:54.780157+0000","last_fullsized":"2024-09-21T12:20:54.780157+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:27.528692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":58,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.668183+0000","last_change":"2024-09-21T12:20:17.693562+0000","last_active":"2024-09-21T12:20:55.668183+0000","last_peered":"2024-09-21T12:20:55.668183+0000","last_clean":"2024-09-21T12:20:55.668183+0000","last_became_active":"2024-09-21T12:20:17.693209+0000","last_became_peered":"2024-09-21T12:20:17.693209+0000","last_unstale":"2024-09-21T12:20:55.668183+0000","last_undegraded":"2024-09-21T12:20:55.668183+0000","last_fullsized":"2024-09-21T12:20:55.668183+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:25.923701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.666864+0000","last_change":"2024-09-21T12:20:15.308767+0000","last_active":"2024-09-21T12:20:55.666864+0000","last_peered":"2024-09-21T12:20:55.666864+0000","last_clean":"2024-09-21T12:20:55.666864+0000","last_became_active":"2024-09-21T12:20:15.308429+0000","last_became_peered":"2024-09-21T12:20:15.308429+0000","last_unstale":"2024-09-21T12:20:55.666864+0000","last_undegraded":"2024-09-21T12:20:55.666864+0000","last_fullsized":"2024-09-21T12:20:55.666864+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:15.014581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":57,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.668226+0000","last_change":"2024-09-21T12:20:19.314884+0000","last_active":"2024-09-21T12:20:55.668226+0000","last_peered":"2024-09-21T12:20:55.668226+0000","last_clean":"2024-09-21T12:20:55.668226+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:20:55.668226+0000","last_undegraded":"2024-09-21T12:20:55.668226+0000","last_fullsized":"2024-09-21T12:20:55.668226+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:29.894782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779115+0000","last_change":"2024-09-21T12:20:21.309936+0000","last_active":"2024-09-21T12:20:54.779115+0000","last_peered":"2024-09-21T12:20:54.779115+0000","last_clean":"2024-09-21T12:20:54.779115+0000","last_became_active":"2024-09-21T12:20:21.309385+0000","last_became_peered":"2024-09-21T12:20:21.309385+0000","last_unstale":"2024-09-21T12:20:54.779115+0000","last_undegraded":"2024-09-21T12:20:54.779115+0000","last_fullsized":"2024-09-21T12:20:54.779115+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:47.499218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":63,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779601+0000","last_change":"2024-09-21T12:20:17.304948+0000","last_active":"2024-09-21T12:20:54.779601+0000","last_peered":"2024-09-21T12:20:54.779601+0000","last_clean":"2024-09-21T12:20:54.779601+0000","last_became_active":"2024-09-21T12:20:17.303763+0000","last_became_peered":"2024-09-21T12:20:17.303763+0000","last_unstale":"2024-09-21T12:20:54.779601+0000","last_undegraded":"2024-09-21T12:20:54.779601+0000","last_fullsized":"2024-09-21T12:20:54.779601+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:31.523086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779939+0000","last_change":"2024-09-21T12:20:15.298500+0000","last_active":"2024-09-21T12:20:54.779939+0000","last_peered":"2024-09-21T12:20:54.779939+0000","last_clean":"2024-09-21T12:20:54.779939+0000","last_became_active":"2024-09-21T12:20:15.298198+0000","last_became_peered":"2024-09-21T12:20:15.298198+0000","last_unstale":"2024-09-21T12:20:54.779939+0000","last_undegraded":"2024-09-21T12:20:54.779939+0000","last_fullsized":"2024-09-21T12:20:54.779939+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:36.549556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":40,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667088+0000","last_change":"2024-09-21T12:20:19.315714+0000","last_active":"2024-09-21T12:20:55.667088+0000","last_peered":"2024-09-21T12:20:55.667088+0000","last_clean":"2024-09-21T12:20:55.667088+0000","last_became_active":"2024-09-21T12:20:19.315418+0000","last_became_peered":"2024-09-21T12:20:19.315418+0000","last_unstale":"2024-09-21T12:20:55.667088+0000","last_undegraded":"2024-09-21T12:20:55.667088+0000","last_fullsized":"2024-09-21T12:20:55.667088+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:38.005096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":35,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667229+0000","last_change":"2024-09-21T12:20:21.310344+0000","last_active":"2024-09-21T12:20:55.667229+0000","last_peered":"2024-09-21T12:20:55.667229+0000","last_clean":"2024-09-21T12:20:55.667229+0000","last_became_active":"2024-09-21T12:20:21.310053+0000","last_became_peered":"2024-09-21T12:20:21.310053+0000","last_unstale":"2024-09-21T12:20:55.667229+0000","last_undegraded":"2024-09-21T12:20:55.667229+0000","last_fullsized":"2024-09-21T12:20:55.667229+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:42.963350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":60,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667235+0000","last_change":"2024-09-21T12:20:17.299367+0000","last_active":"2024-09-21T12:20:55.667235+0000","last_peered":"2024-09-21T12:20:55.667235+0000","last_clean":"2024-09-21T12:20:55.667235+0000","last_became_active":"2024-09-21T12:20:17.299015+0000","last_became_peered":"2024-09-21T12:20:17.299015+0000","last_unstale":"2024-09-21T12:20:55.667235+0000","last_undegraded":"2024-09-21T12:20:55.667235+0000","last_fullsized":"2024-09-21T12:20:55.667235+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:42.174386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667129+0000","last_change":"2024-09-21T12:20:15.303462+0000","last_active":"2024-09-21T12:20:55.667129+0000","last_peered":"2024-09-21T12:20:55.667129+0000","last_clean":"2024-09-21T12:20:55.667129+0000","last_became_active":"2024-09-21T12:20:15.300554+0000","last_became_peered":"2024-09-21T12:20:15.300554+0000","last_unstale":"2024-09-21T12:20:55.667129+0000","last_undegraded":"2024-09-21T12:20:55.667129+0000","last_fullsized":"2024-09-21T12:20:55.667129+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:22.850479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":39,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667721+0000","last_change":"2024-09-21T12:20:19.311327+0000","last_active":"2024-09-21T12:20:55.667721+0000","last_peered":"2024-09-21T12:20:55.667721+0000","last_clean":"2024-09-21T12:20:55.667721+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:20:55.667721+0000","last_undegraded":"2024-09-21T12:20:55.667721+0000","last_fullsized":"2024-09-21T12:20:55.667721+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:20.603464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.778689+0000","last_change":"2024-09-21T12:20:21.308003+0000","last_active":"2024-09-21T12:20:54.778689+0000","last_peered":"2024-09-21T12:20:54.778689+0000","last_clean":"2024-09-21T12:20:54.778689+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:20:54.778689+0000","last_undegraded":"2024-09-21T12:20:54.778689+0000","last_fullsized":"2024-09-21T12:20:54.778689+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:41.631005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":62,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667354+0000","last_change":"2024-09-21T12:20:17.695358+0000","last_active":"2024-09-21T12:20:55.667354+0000","last_peered":"2024-09-21T12:20:55.667354+0000","last_clean":"2024-09-21T12:20:55.667354+0000","last_became_active":"2024-09-21T12:20:17.694794+0000","last_became_peered":"2024-09-21T12:20:17.694794+0000","last_unstale":"2024-09-21T12:20:55.667354+0000","last_undegraded":"2024-09-21T12:20:55.667354+0000","last_fullsized":"2024-09-21T12:20:55.667354+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:25.305944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667953+0000","last_change":"2024-09-21T12:20:15.296285+0000","last_active":"2024-09-21T12:20:55.667953+0000","last_peered":"2024-09-21T12:20:55.667953+0000","last_clean":"2024-09-21T12:20:55.667953+0000","last_became_active":"2024-09-21T12:20:15.295968+0000","last_became_peered":"2024-09-21T12:20:15.295968+0000","last_unstale":"2024-09-21T12:20:55.667953+0000","last_undegraded":"2024-09-21T12:20:55.667953+0000","last_fullsized":"2024-09-21T12:20:55.667953+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:33.273850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":40,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667279+0000","last_change":"2024-09-21T12:20:19.309473+0000","last_active":"2024-09-21T12:20:55.667279+0000","last_peered":"2024-09-21T12:20:55.667279+0000","last_clean":"2024-09-21T12:20:55.667279+0000","last_became_active":"2024-09-21T12:20:19.309195+0000","last_became_peered":"2024-09-21T12:20:19.309195+0000","last_unstale":"2024-09-21T12:20:55.667279+0000","last_undegraded":"2024-09-21T12:20:55.667279+0000","last_fullsized":"2024-09-21T12:20:55.667279+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:39.580857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":35,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667940+0000","last_change":"2024-09-21T12:20:21.595660+0000","last_active":"2024-09-21T12:20:55.667940+0000","last_peered":"2024-09-21T12:20:55.667940+0000","last_clean":"2024-09-21T12:20:55.667940+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:20:55.667940+0000","last_undegraded":"2024-09-21T12:20:55.667940+0000","last_fullsized":"2024-09-21T12:20:55.667940+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:22.428602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":49,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.666791+0000","last_change":"2024-09-21T12:20:17.304075+0000","last_active":"2024-09-21T12:20:55.666791+0000","last_peered":"2024-09-21T12:20:55.666791+0000","last_clean":"2024-09-21T12:20:55.666791+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:20:55.666791+0000","last_undegraded":"2024-09-21T12:20:55.666791+0000","last_fullsized":"2024-09-21T12:20:55.666791+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.024324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":54,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779398+0000","last_change":"2024-09-21T12:20:15.298443+0000","last_active":"2024-09-21T12:20:54.779398+0000","last_peered":"2024-09-21T12:20:54.779398+0000","last_clean":"2024-09-21T12:20:54.779398+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:20:54.779398+0000","last_undegraded":"2024-09-21T12:20:54.779398+0000","last_fullsized":"2024-09-21T12:20:54.779398+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:36.774153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":40,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.396315+0000","last_change":"2024-09-21T12:20:19.312978+0000","last_active":"2024-09-21T12:20:55.396315+0000","last_peered":"2024-09-21T12:20:55.396315+0000","last_clean":"2024-09-21T12:20:55.396315+0000","last_became_active":"2024-09-21T12:20:19.312512+0000","last_became_peered":"2024-09-21T12:20:19.312512+0000","last_unstale":"2024-09-21T12:20:55.396315+0000","last_undegraded":"2024-09-21T12:20:55.396315+0000","last_fullsized":"2024-09-21T12:20:55.396315+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:41.253296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667478+0000","last_change":"2024-09-21T12:20:21.313523+0000","last_active":"2024-09-21T12:20:55.667478+0000","last_peered":"2024-09-21T12:20:55.667478+0000","last_clean":"2024-09-21T12:20:55.667478+0000","last_became_active":"2024-09-21T12:20:21.313228+0000","last_became_peered":"2024-09-21T12:20:21.313228+0000","last_unstale":"2024-09-21T12:20:55.667478+0000","last_undegraded":"2024-09-21T12:20:55.667478+0000","last_fullsized":"2024-09-21T12:20:55.667478+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:47.112414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.667168+0000","last_change":"2024-09-21T12:20:21.309169+0000","last_active":"2024-09-21T12:20:55.667168+0000","last_peered":"2024-09-21T12:20:55.667168+0000","last_clean":"2024-09-21T12:20:55.667168+0000","last_became_active":"2024-09-21T12:20:21.308865+0000","last_became_peered":"2024-09-21T12:20:21.308865+0000","last_unstale":"2024-09-21T12:20:55.667168+0000","last_undegraded":"2024-09-21T12:20:55.667168+0000","last_fullsized":"2024-09-21T12:20:55.667168+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:26.841000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":47,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.666460+0000","last_change":"2024-09-21T12:20:15.298322+0000","last_active":"2024-09-21T12:20:55.666460+0000","last_peered":"2024-09-21T12:20:55.666460+0000","last_clean":"2024-09-21T12:20:55.666460+0000","last_became_active":"2024-09-21T12:20:15.297993+0000","last_became_peered":"2024-09-21T12:20:15.297993+0000","last_unstale":"2024-09-21T12:20:55.666460+0000","last_undegraded":"2024-09-21T12:20:55.666460+0000","last_fullsized":"2024-09-21T12:20:55.666460+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:34.289061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":63,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:55.397044+0000","last_change":"2024-09-21T12:20:17.299134+0000","last_active":"2024-09-21T12:20:55.397044+0000","last_peered":"2024-09-21T12:20:55.397044+0000","last_clean":"2024-09-21T12:20:55.397044+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:20:55.397044+0000","last_undegraded":"2024-09-21T12:20:55.397044+0000","last_fullsized":"2024-09-21T12:20:55.397044+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.792664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":40,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T12:20:54.779241+0000","last_change":"2024-09-21T12:20:19.308586+0000","last_active":"2024-09-21T12:20:54.779241+0000","last_peered":"2024-09-21T12:20:54.779241+0000","last_clean":"2024-09-21T12:20:54.779241+0000","last_became_active":"2024-09-21T12:20:19.308247+0000","last_became_peered":"2024-09-21T12:20:19.308247+0000","last_unstale":"2024-09-21T12:20:54.779241+0000","last_undegraded":"2024-09-21T12:20:54.779241+0000","last_fullsized":"2024-09-21T12:20:54.779241+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:42.632946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":36,"num_read_kb":31,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928233996,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32612,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712540,"statfs":{"total":95995035648,"available":95961640960,"internally_reserved":0,"allocated":1458176,"data_stored":967702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462932,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32596,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712556,"statfs":{"total":95995035648,"available":95961657344,"internally_reserved":0,"allocated":1441792,"data_stored":967053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724567,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27868,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717284,"statfs":{"total":95995035648,"available":95966498816,"internally_reserved":0,"allocated":860160,"data_stored":378050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953503,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32040,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713112,"statfs":{"total":95995035648,"available":95962226688,"internally_reserved":0,"allocated":872448,"data_stored":376854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149733,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":888832,"data_stored":378569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411371,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32008,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713144,"statfs":{"total":95995035648,"available":95962259456,"internally_reserved":0,"allocated":839680,"data_stored":376397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":913408,"data_stored":378047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738423,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28464,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716688,"statfs":{"total":95995035648,"available":95965888512,"internally_reserved":0,"allocated":1470464,"data_stored":968417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:21:01.351 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:21:01.618 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:21:02.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:21:02.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:21:02.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[30324]: Adjusting osd_memory_target on smithi114 to 3760M 2024-09-21T12:21:02.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[30324]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-21T12:21:02.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[30324]: Updating smithi183:/etc/ceph/ceph.conf 2024-09-21T12:21:02.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[30324]: from='client.24596 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:21:02.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[30324]: Updating smithi114:/var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:21:02.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[30324]: Updating smithi183:/var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:21:02.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[30324]: osdmap e79: 8 total, 8 up, 8 in 2024-09-21T12:21:02.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[30324]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:21:02.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[30324]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:21:02.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:21:02.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[30324]: Updating smithi114:/var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:21:02.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[30324]: Updating smithi183:/var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:21:02.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:02.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:02.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:02.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:02.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:02.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[30324]: osdmap e80: 8 total, 8 up, 8 in 2024-09-21T12:21:02.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:21:02.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:21:02.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[40867]: Adjusting osd_memory_target on smithi114 to 3760M 2024-09-21T12:21:02.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[40867]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-21T12:21:02.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[40867]: Updating smithi183:/etc/ceph/ceph.conf 2024-09-21T12:21:02.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[40867]: from='client.24596 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:21:02.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[40867]: Updating smithi114:/var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:21:02.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[40867]: Updating smithi183:/var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:21:02.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[40867]: osdmap e79: 8 total, 8 up, 8 in 2024-09-21T12:21:02.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[40867]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:21:02.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[40867]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:21:02.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:21:02.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[40867]: Updating smithi114:/var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:21:02.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[40867]: Updating smithi183:/var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:21:02.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:02.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:02.028 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:02.028 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:02.028 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:02.028 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:01 smithi114 ceph-mon[40867]: osdmap e80: 8 total, 8 up, 8 in 2024-09-21T12:21:02.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:01 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:21:02.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:01 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:21:02.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:01 smithi183 ceph-mon[33277]: Adjusting osd_memory_target on smithi114 to 3760M 2024-09-21T12:21:02.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:01 smithi183 ceph-mon[33277]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-21T12:21:02.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:01 smithi183 ceph-mon[33277]: Updating smithi183:/etc/ceph/ceph.conf 2024-09-21T12:21:02.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:01 smithi183 ceph-mon[33277]: from='client.24596 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:21:02.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:01 smithi183 ceph-mon[33277]: Updating smithi114:/var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:21:02.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:01 smithi183 ceph-mon[33277]: Updating smithi183:/var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:21:02.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:01 smithi183 ceph-mon[33277]: osdmap e79: 8 total, 8 up, 8 in 2024-09-21T12:21:02.240 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:01 smithi183 ceph-mon[33277]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:21:02.240 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:01 smithi183 ceph-mon[33277]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:21:02.240 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:01 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:21:02.240 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:01 smithi183 ceph-mon[33277]: Updating smithi114:/var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:21:02.240 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:01 smithi183 ceph-mon[33277]: Updating smithi183:/var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:21:02.240 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:01 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:02.240 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:01 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:02.240 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:01 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:02.240 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:01 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:02.240 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:01 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:02.240 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:01 smithi183 ceph-mon[33277]: osdmap e80: 8 total, 8 up, 8 in 2024-09-21T12:21:02.343 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:21:02.345 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:21:02.774 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:21:02 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:21:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:21:03.140 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:02 smithi183 ceph-mon[33277]: Deploying daemon alertmanager.a on smithi114 2024-09-21T12:21:03.140 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:02 smithi183 ceph-mon[33277]: pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T12:21:03.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:02 smithi114 ceph-mon[40867]: Deploying daemon alertmanager.a on smithi114 2024-09-21T12:21:03.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:02 smithi114 ceph-mon[40867]: pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T12:21:03.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:02 smithi114 ceph-mon[30324]: Deploying daemon alertmanager.a on smithi114 2024-09-21T12:21:03.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:02 smithi114 ceph-mon[30324]: pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T12:21:03.606 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-09-21T12:21:01.759532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":889,"num_read_kb":746,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":245708,"kb_used_data":8672,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749715508,"statfs":{"total":767960285184,"available":767708680192,"internally_reserved":0,"allocated":8880128,"data_stored":4875070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.212591"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":52,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.795407+0000","last_change":"2024-09-21T12:20:15.309744+0000","last_active":"2024-09-21T12:21:00.795407+0000","last_peered":"2024-09-21T12:21:00.795407+0000","last_clean":"2024-09-21T12:21:00.795407+0000","last_became_active":"2024-09-21T12:20:15.309431+0000","last_became_peered":"2024-09-21T12:20:15.309431+0000","last_unstale":"2024-09-21T12:21:00.795407+0000","last_undegraded":"2024-09-21T12:21:00.795407+0000","last_fullsized":"2024-09-21T12:21:00.795407+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:37.321280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":62,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:56.761164+0000","last_change":"2024-09-21T12:20:17.304357+0000","last_active":"2024-09-21T12:20:56.761164+0000","last_peered":"2024-09-21T12:20:56.761164+0000","last_clean":"2024-09-21T12:20:56.761164+0000","last_became_active":"2024-09-21T12:20:17.303856+0000","last_became_peered":"2024-09-21T12:20:17.303856+0000","last_unstale":"2024-09-21T12:20:56.761164+0000","last_undegraded":"2024-09-21T12:20:56.761164+0000","last_fullsized":"2024-09-21T12:20:56.761164+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:40.716880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":44,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.794028+0000","last_change":"2024-09-21T12:20:19.315239+0000","last_active":"2024-09-21T12:21:00.794028+0000","last_peered":"2024-09-21T12:21:00.794028+0000","last_clean":"2024-09-21T12:21:00.794028+0000","last_became_active":"2024-09-21T12:20:19.314908+0000","last_became_peered":"2024-09-21T12:20:19.314908+0000","last_unstale":"2024-09-21T12:21:00.794028+0000","last_undegraded":"2024-09-21T12:21:00.794028+0000","last_fullsized":"2024-09-21T12:21:00.794028+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:33.386500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:56.761336+0000","last_change":"2024-09-21T12:20:21.308871+0000","last_active":"2024-09-21T12:20:56.761336+0000","last_peered":"2024-09-21T12:20:56.761336+0000","last_clean":"2024-09-21T12:20:56.761336+0000","last_became_active":"2024-09-21T12:20:21.308577+0000","last_became_peered":"2024-09-21T12:20:21.308577+0000","last_unstale":"2024-09-21T12:20:56.761336+0000","last_undegraded":"2024-09-21T12:20:56.761336+0000","last_fullsized":"2024-09-21T12:20:56.761336+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:43.912920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":37,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.963128+0000","last_change":"2024-09-21T12:20:21.595607+0000","last_active":"2024-09-21T12:20:57.963128+0000","last_peered":"2024-09-21T12:20:57.963128+0000","last_clean":"2024-09-21T12:20:57.963128+0000","last_became_active":"2024-09-21T12:20:21.595301+0000","last_became_peered":"2024-09-21T12:20:21.595301+0000","last_unstale":"2024-09-21T12:20:57.963128+0000","last_undegraded":"2024-09-21T12:20:57.963128+0000","last_fullsized":"2024-09-21T12:20:57.963128+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:27.801303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":49,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.963504+0000","last_change":"2024-09-21T12:20:15.308941+0000","last_active":"2024-09-21T12:20:57.963504+0000","last_peered":"2024-09-21T12:20:57.963504+0000","last_clean":"2024-09-21T12:20:57.963504+0000","last_became_active":"2024-09-21T12:20:15.308655+0000","last_became_peered":"2024-09-21T12:20:15.308655+0000","last_unstale":"2024-09-21T12:20:57.963504+0000","last_undegraded":"2024-09-21T12:20:57.963504+0000","last_fullsized":"2024-09-21T12:20:57.963504+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:31.202541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":70,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.962339+0000","last_change":"2024-09-21T12:20:17.305499+0000","last_active":"2024-09-21T12:20:57.962339+0000","last_peered":"2024-09-21T12:20:57.962339+0000","last_clean":"2024-09-21T12:20:57.962339+0000","last_became_active":"2024-09-21T12:20:17.305220+0000","last_became_peered":"2024-09-21T12:20:17.305220+0000","last_unstale":"2024-09-21T12:20:57.962339+0000","last_undegraded":"2024-09-21T12:20:57.962339+0000","last_fullsized":"2024-09-21T12:20:57.962339+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.370654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":42,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:56.761123+0000","last_change":"2024-09-21T12:20:19.311730+0000","last_active":"2024-09-21T12:20:56.761123+0000","last_peered":"2024-09-21T12:20:56.761123+0000","last_clean":"2024-09-21T12:20:56.761123+0000","last_became_active":"2024-09-21T12:20:19.311403+0000","last_became_peered":"2024-09-21T12:20:19.311403+0000","last_unstale":"2024-09-21T12:20:56.761123+0000","last_undegraded":"2024-09-21T12:20:56.761123+0000","last_fullsized":"2024-09-21T12:20:56.761123+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:40.685703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":52,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.794219+0000","last_change":"2024-09-21T12:20:15.302150+0000","last_active":"2024-09-21T12:21:00.794219+0000","last_peered":"2024-09-21T12:21:00.794219+0000","last_clean":"2024-09-21T12:21:00.794219+0000","last_became_active":"2024-09-21T12:20:15.301847+0000","last_became_peered":"2024-09-21T12:20:15.301847+0000","last_unstale":"2024-09-21T12:21:00.794219+0000","last_undegraded":"2024-09-21T12:21:00.794219+0000","last_fullsized":"2024-09-21T12:21:00.794219+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:26.753083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":62,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:56.761530+0000","last_change":"2024-09-21T12:20:17.307650+0000","last_active":"2024-09-21T12:20:56.761530+0000","last_peered":"2024-09-21T12:20:56.761530+0000","last_clean":"2024-09-21T12:20:56.761530+0000","last_became_active":"2024-09-21T12:20:17.307280+0000","last_became_peered":"2024-09-21T12:20:17.307280+0000","last_unstale":"2024-09-21T12:20:56.761530+0000","last_undegraded":"2024-09-21T12:20:56.761530+0000","last_fullsized":"2024-09-21T12:20:56.761530+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:34.375479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":44,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.793811+0000","last_change":"2024-09-21T12:20:19.308520+0000","last_active":"2024-09-21T12:21:00.793811+0000","last_peered":"2024-09-21T12:21:00.793811+0000","last_clean":"2024-09-21T12:21:00.793811+0000","last_became_active":"2024-09-21T12:20:19.308184+0000","last_became_peered":"2024-09-21T12:20:19.308184+0000","last_unstale":"2024-09-21T12:21:00.793811+0000","last_undegraded":"2024-09-21T12:21:00.793811+0000","last_fullsized":"2024-09-21T12:21:00.793811+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:39.981449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.794009+0000","last_change":"2024-09-21T12:20:21.309871+0000","last_active":"2024-09-21T12:21:00.794009+0000","last_peered":"2024-09-21T12:21:00.794009+0000","last_clean":"2024-09-21T12:21:00.794009+0000","last_became_active":"2024-09-21T12:20:21.309190+0000","last_became_peered":"2024-09-21T12:20:21.309190+0000","last_unstale":"2024-09-21T12:21:00.794009+0000","last_undegraded":"2024-09-21T12:21:00.794009+0000","last_fullsized":"2024-09-21T12:21:00.794009+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.813677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":52,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.794000+0000","last_change":"2024-09-21T12:20:15.300728+0000","last_active":"2024-09-21T12:21:00.794000+0000","last_peered":"2024-09-21T12:21:00.794000+0000","last_clean":"2024-09-21T12:21:00.794000+0000","last_became_active":"2024-09-21T12:20:15.300441+0000","last_became_peered":"2024-09-21T12:20:15.300441+0000","last_unstale":"2024-09-21T12:21:00.794000+0000","last_undegraded":"2024-09-21T12:21:00.794000+0000","last_fullsized":"2024-09-21T12:21:00.794000+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:34.282857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:56.761478+0000","last_change":"2024-09-21T12:20:17.305973+0000","last_active":"2024-09-21T12:20:56.761478+0000","last_peered":"2024-09-21T12:20:56.761478+0000","last_clean":"2024-09-21T12:20:56.761478+0000","last_became_active":"2024-09-21T12:20:17.305663+0000","last_became_peered":"2024-09-21T12:20:17.305663+0000","last_unstale":"2024-09-21T12:20:56.761478+0000","last_undegraded":"2024-09-21T12:20:56.761478+0000","last_fullsized":"2024-09-21T12:20:56.761478+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.420641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":42,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.055936+0000","last_change":"2024-09-21T12:20:19.312502+0000","last_active":"2024-09-21T12:20:57.055936+0000","last_peered":"2024-09-21T12:20:57.055936+0000","last_clean":"2024-09-21T12:20:57.055936+0000","last_became_active":"2024-09-21T12:20:19.312242+0000","last_became_peered":"2024-09-21T12:20:19.312242+0000","last_unstale":"2024-09-21T12:20:57.055936+0000","last_undegraded":"2024-09-21T12:20:57.055936+0000","last_fullsized":"2024-09-21T12:20:57.055936+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:32.544179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.793633+0000","last_change":"2024-09-21T12:20:21.311536+0000","last_active":"2024-09-21T12:21:00.793633+0000","last_peered":"2024-09-21T12:21:00.793633+0000","last_clean":"2024-09-21T12:21:00.793633+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:21:00.793633+0000","last_undegraded":"2024-09-21T12:21:00.793633+0000","last_fullsized":"2024-09-21T12:21:00.793633+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:41.239506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:56.761792+0000","last_change":"2024-09-21T12:20:21.592944+0000","last_active":"2024-09-21T12:20:56.761792+0000","last_peered":"2024-09-21T12:20:56.761792+0000","last_clean":"2024-09-21T12:20:56.761792+0000","last_became_active":"2024-09-21T12:20:21.592631+0000","last_became_peered":"2024-09-21T12:20:21.592631+0000","last_unstale":"2024-09-21T12:20:56.761792+0000","last_undegraded":"2024-09-21T12:20:56.761792+0000","last_fullsized":"2024-09-21T12:20:56.761792+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:39.384088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":52,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.795932+0000","last_change":"2024-09-21T12:20:15.302079+0000","last_active":"2024-09-21T12:21:00.795932+0000","last_peered":"2024-09-21T12:21:00.795932+0000","last_clean":"2024-09-21T12:21:00.795932+0000","last_became_active":"2024-09-21T12:20:15.301793+0000","last_became_peered":"2024-09-21T12:20:15.301793+0000","last_unstale":"2024-09-21T12:21:00.795932+0000","last_undegraded":"2024-09-21T12:21:00.795932+0000","last_fullsized":"2024-09-21T12:21:00.795932+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:19.991370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":73,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.455608+0000","last_change":"2024-09-21T12:20:17.308186+0000","last_active":"2024-09-21T12:21:01.455608+0000","last_peered":"2024-09-21T12:21:01.455608+0000","last_clean":"2024-09-21T12:21:01.455608+0000","last_became_active":"2024-09-21T12:20:17.307828+0000","last_became_peered":"2024-09-21T12:20:17.307828+0000","last_unstale":"2024-09-21T12:21:01.455608+0000","last_undegraded":"2024-09-21T12:21:01.455608+0000","last_fullsized":"2024-09-21T12:21:01.455608+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:41.845334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":44,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.794027+0000","last_change":"2024-09-21T12:20:19.316360+0000","last_active":"2024-09-21T12:21:00.794027+0000","last_peered":"2024-09-21T12:21:00.794027+0000","last_clean":"2024-09-21T12:21:00.794027+0000","last_became_active":"2024-09-21T12:20:19.315977+0000","last_became_peered":"2024-09-21T12:20:19.315977+0000","last_unstale":"2024-09-21T12:21:00.794027+0000","last_undegraded":"2024-09-21T12:21:00.794027+0000","last_fullsized":"2024-09-21T12:21:00.794027+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:37.515718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":37,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.963492+0000","last_change":"2024-09-21T12:20:21.593129+0000","last_active":"2024-09-21T12:20:57.963492+0000","last_peered":"2024-09-21T12:20:57.963492+0000","last_clean":"2024-09-21T12:20:57.963492+0000","last_became_active":"2024-09-21T12:20:21.592796+0000","last_became_peered":"2024-09-21T12:20:21.592796+0000","last_unstale":"2024-09-21T12:20:57.963492+0000","last_undegraded":"2024-09-21T12:20:57.963492+0000","last_fullsized":"2024-09-21T12:20:57.963492+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:47.880164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:56.761773+0000","last_change":"2024-09-21T12:20:15.309390+0000","last_active":"2024-09-21T12:20:56.761773+0000","last_peered":"2024-09-21T12:20:56.761773+0000","last_clean":"2024-09-21T12:20:56.761773+0000","last_became_active":"2024-09-21T12:20:15.309023+0000","last_became_peered":"2024-09-21T12:20:15.309023+0000","last_unstale":"2024-09-21T12:20:56.761773+0000","last_undegraded":"2024-09-21T12:20:56.761773+0000","last_fullsized":"2024-09-21T12:20:56.761773+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:25.611326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":68,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.962521+0000","last_change":"2024-09-21T12:20:17.310727+0000","last_active":"2024-09-21T12:20:57.962521+0000","last_peered":"2024-09-21T12:20:57.962521+0000","last_clean":"2024-09-21T12:20:57.962521+0000","last_became_active":"2024-09-21T12:20:17.310369+0000","last_became_peered":"2024-09-21T12:20:17.310369+0000","last_unstale":"2024-09-21T12:20:57.962521+0000","last_undegraded":"2024-09-21T12:20:57.962521+0000","last_fullsized":"2024-09-21T12:20:57.962521+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:22.563871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":42,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:56.761537+0000","last_change":"2024-09-21T12:20:19.310859+0000","last_active":"2024-09-21T12:20:56.761537+0000","last_peered":"2024-09-21T12:20:56.761537+0000","last_clean":"2024-09-21T12:20:56.761537+0000","last_became_active":"2024-09-21T12:20:19.310575+0000","last_became_peered":"2024-09-21T12:20:19.310575+0000","last_unstale":"2024-09-21T12:20:56.761537+0000","last_undegraded":"2024-09-21T12:20:56.761537+0000","last_fullsized":"2024-09-21T12:20:56.761537+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.082897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":39,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.056223+0000","last_change":"2024-09-21T12:20:21.313476+0000","last_active":"2024-09-21T12:20:57.056223+0000","last_peered":"2024-09-21T12:20:57.056223+0000","last_clean":"2024-09-21T12:20:57.056223+0000","last_became_active":"2024-09-21T12:20:21.312976+0000","last_became_peered":"2024-09-21T12:20:21.312976+0000","last_unstale":"2024-09-21T12:20:57.056223+0000","last_undegraded":"2024-09-21T12:20:57.056223+0000","last_fullsized":"2024-09-21T12:20:57.056223+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.367953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":53,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.794224+0000","last_change":"2024-09-21T12:20:15.299812+0000","last_active":"2024-09-21T12:21:00.794224+0000","last_peered":"2024-09-21T12:21:00.794224+0000","last_clean":"2024-09-21T12:21:00.794224+0000","last_became_active":"2024-09-21T12:20:15.299459+0000","last_became_peered":"2024-09-21T12:20:15.299459+0000","last_unstale":"2024-09-21T12:21:00.794224+0000","last_undegraded":"2024-09-21T12:21:00.794224+0000","last_fullsized":"2024-09-21T12:21:00.794224+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.458091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":63,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.795659+0000","last_change":"2024-09-21T12:20:17.696423+0000","last_active":"2024-09-21T12:21:00.795659+0000","last_peered":"2024-09-21T12:21:00.795659+0000","last_clean":"2024-09-21T12:21:00.795659+0000","last_became_active":"2024-09-21T12:20:17.695977+0000","last_became_peered":"2024-09-21T12:20:17.695977+0000","last_unstale":"2024-09-21T12:21:00.795659+0000","last_undegraded":"2024-09-21T12:21:00.795659+0000","last_fullsized":"2024-09-21T12:21:00.795659+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:44.740918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":63,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.673747+0000","last_change":"2024-09-21T12:20:19.313262+0000","last_active":"2024-09-21T12:20:57.673747+0000","last_peered":"2024-09-21T12:20:57.673747+0000","last_clean":"2024-09-21T12:20:57.673747+0000","last_became_active":"2024-09-21T12:20:19.312403+0000","last_became_peered":"2024-09-21T12:20:19.312403+0000","last_unstale":"2024-09-21T12:20:57.673747+0000","last_undegraded":"2024-09-21T12:20:57.673747+0000","last_fullsized":"2024-09-21T12:20:57.673747+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:43.181150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":73,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.794798+0000","last_change":"2024-09-21T12:20:17.305947+0000","last_active":"2024-09-21T12:21:00.794798+0000","last_peered":"2024-09-21T12:21:00.794798+0000","last_clean":"2024-09-21T12:21:00.794798+0000","last_became_active":"2024-09-21T12:20:17.305666+0000","last_became_peered":"2024-09-21T12:20:17.305666+0000","last_unstale":"2024-09-21T12:21:00.794798+0000","last_undegraded":"2024-09-21T12:21:00.794798+0000","last_fullsized":"2024-09-21T12:21:00.794798+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.503390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":49,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.962804+0000","last_change":"2024-09-21T12:20:15.296251+0000","last_active":"2024-09-21T12:20:57.962804+0000","last_peered":"2024-09-21T12:20:57.962804+0000","last_clean":"2024-09-21T12:20:57.962804+0000","last_became_active":"2024-09-21T12:20:15.295774+0000","last_became_peered":"2024-09-21T12:20:15.295774+0000","last_unstale":"2024-09-21T12:20:57.962804+0000","last_undegraded":"2024-09-21T12:20:57.962804+0000","last_fullsized":"2024-09-21T12:20:57.962804+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:27.341994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":60,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:56.761977+0000","last_change":"2024-09-21T12:20:19.309874+0000","last_active":"2024-09-21T12:20:56.761977+0000","last_peered":"2024-09-21T12:20:56.761977+0000","last_clean":"2024-09-21T12:20:56.761977+0000","last_became_active":"2024-09-21T12:20:19.309606+0000","last_became_peered":"2024-09-21T12:20:19.309606+0000","last_unstale":"2024-09-21T12:20:56.761977+0000","last_undegraded":"2024-09-21T12:20:56.761977+0000","last_fullsized":"2024-09-21T12:20:56.761977+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:37.621763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":41,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.793923+0000","last_change":"2024-09-21T12:20:21.310295+0000","last_active":"2024-09-21T12:21:00.793923+0000","last_peered":"2024-09-21T12:21:00.793923+0000","last_clean":"2024-09-21T12:21:00.793923+0000","last_became_active":"2024-09-21T12:20:21.310022+0000","last_became_peered":"2024-09-21T12:20:21.310022+0000","last_unstale":"2024-09-21T12:21:00.793923+0000","last_undegraded":"2024-09-21T12:21:00.793923+0000","last_fullsized":"2024-09-21T12:21:00.793923+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:37.912528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":77,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.962335+0000","last_change":"2024-09-21T12:20:17.693550+0000","last_active":"2024-09-21T12:20:57.962335+0000","last_peered":"2024-09-21T12:20:57.962335+0000","last_clean":"2024-09-21T12:20:57.962335+0000","last_became_active":"2024-09-21T12:20:17.693179+0000","last_became_peered":"2024-09-21T12:20:17.693179+0000","last_unstale":"2024-09-21T12:20:57.962335+0000","last_undegraded":"2024-09-21T12:20:57.962335+0000","last_fullsized":"2024-09-21T12:20:57.962335+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:45.358986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":49,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.963575+0000","last_change":"2024-09-21T12:20:15.297621+0000","last_active":"2024-09-21T12:20:57.963575+0000","last_peered":"2024-09-21T12:20:57.963575+0000","last_clean":"2024-09-21T12:20:57.963575+0000","last_became_active":"2024-09-21T12:20:15.297340+0000","last_became_peered":"2024-09-21T12:20:15.297340+0000","last_unstale":"2024-09-21T12:20:57.963575+0000","last_undegraded":"2024-09-21T12:20:57.963575+0000","last_fullsized":"2024-09-21T12:20:57.963575+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:17.208372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":42,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:56.761201+0000","last_change":"2024-09-21T12:20:19.315065+0000","last_active":"2024-09-21T12:20:56.761201+0000","last_peered":"2024-09-21T12:20:56.761201+0000","last_clean":"2024-09-21T12:20:56.761201+0000","last_became_active":"2024-09-21T12:20:19.314775+0000","last_became_peered":"2024-09-21T12:20:19.314775+0000","last_unstale":"2024-09-21T12:20:56.761201+0000","last_undegraded":"2024-09-21T12:20:56.761201+0000","last_fullsized":"2024-09-21T12:20:56.761201+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:26.384029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:56.761562+0000","last_change":"2024-09-21T12:20:21.314735+0000","last_active":"2024-09-21T12:20:56.761562+0000","last_peered":"2024-09-21T12:20:56.761562+0000","last_clean":"2024-09-21T12:20:56.761562+0000","last_became_active":"2024-09-21T12:20:21.314433+0000","last_became_peered":"2024-09-21T12:20:21.314433+0000","last_unstale":"2024-09-21T12:20:56.761562+0000","last_undegraded":"2024-09-21T12:20:56.761562+0000","last_fullsized":"2024-09-21T12:20:56.761562+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:39.652862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":67,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:56.761281+0000","last_change":"2024-09-21T12:20:17.301460+0000","last_active":"2024-09-21T12:20:56.761281+0000","last_peered":"2024-09-21T12:20:56.761281+0000","last_clean":"2024-09-21T12:20:56.761281+0000","last_became_active":"2024-09-21T12:20:17.301090+0000","last_became_peered":"2024-09-21T12:20:17.301090+0000","last_unstale":"2024-09-21T12:20:56.761281+0000","last_undegraded":"2024-09-21T12:20:56.761281+0000","last_fullsized":"2024-09-21T12:20:56.761281+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:34.835982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":53,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.795839+0000","last_change":"2024-09-21T12:20:15.302238+0000","last_active":"2024-09-21T12:21:00.795839+0000","last_peered":"2024-09-21T12:21:00.795839+0000","last_clean":"2024-09-21T12:21:00.795839+0000","last_became_active":"2024-09-21T12:20:15.301904+0000","last_became_peered":"2024-09-21T12:20:15.301904+0000","last_unstale":"2024-09-21T12:21:00.795839+0000","last_undegraded":"2024-09-21T12:21:00.795839+0000","last_fullsized":"2024-09-21T12:21:00.795839+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:33.679116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.962636+0000","last_change":"2024-09-21T12:20:19.311362+0000","last_active":"2024-09-21T12:20:57.962636+0000","last_peered":"2024-09-21T12:20:57.962636+0000","last_clean":"2024-09-21T12:20:57.962636+0000","last_became_active":"2024-09-21T12:20:19.311056+0000","last_became_peered":"2024-09-21T12:20:19.311056+0000","last_unstale":"2024-09-21T12:20:57.962636+0000","last_undegraded":"2024-09-21T12:20:57.962636+0000","last_fullsized":"2024-09-21T12:20:57.962636+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:22.628895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.056071+0000","last_change":"2024-09-21T12:20:21.593620+0000","last_active":"2024-09-21T12:20:57.056071+0000","last_peered":"2024-09-21T12:20:57.056071+0000","last_clean":"2024-09-21T12:20:57.056071+0000","last_became_active":"2024-09-21T12:20:21.593367+0000","last_became_peered":"2024-09-21T12:20:21.593367+0000","last_unstale":"2024-09-21T12:20:57.056071+0000","last_undegraded":"2024-09-21T12:20:57.056071+0000","last_fullsized":"2024-09-21T12:20:57.056071+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:20.553670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":63,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.794316+0000","last_change":"2024-09-21T12:20:17.303245+0000","last_active":"2024-09-21T12:21:00.794316+0000","last_peered":"2024-09-21T12:21:00.794316+0000","last_clean":"2024-09-21T12:21:00.794316+0000","last_became_active":"2024-09-21T12:20:17.302871+0000","last_became_peered":"2024-09-21T12:20:17.302871+0000","last_unstale":"2024-09-21T12:21:00.794316+0000","last_undegraded":"2024-09-21T12:21:00.794316+0000","last_fullsized":"2024-09-21T12:21:00.794316+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.942911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":52,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.794245+0000","last_change":"2024-09-21T12:20:15.308877+0000","last_active":"2024-09-21T12:21:00.794245+0000","last_peered":"2024-09-21T12:21:00.794245+0000","last_clean":"2024-09-21T12:21:00.794245+0000","last_became_active":"2024-09-21T12:20:15.308518+0000","last_became_peered":"2024-09-21T12:20:15.308518+0000","last_unstale":"2024-09-21T12:21:00.794245+0000","last_undegraded":"2024-09-21T12:21:00.794245+0000","last_fullsized":"2024-09-21T12:21:00.794245+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:23.801832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":65,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.795692+0000","last_change":"2024-09-21T12:20:19.307467+0000","last_active":"2024-09-21T12:21:00.795692+0000","last_peered":"2024-09-21T12:21:00.795692+0000","last_clean":"2024-09-21T12:21:00.795692+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:21:00.795692+0000","last_undegraded":"2024-09-21T12:21:00.795692+0000","last_fullsized":"2024-09-21T12:21:00.795692+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:21.225955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.794163+0000","last_change":"2024-09-21T12:20:21.316127+0000","last_active":"2024-09-21T12:21:00.794163+0000","last_peered":"2024-09-21T12:21:00.794163+0000","last_clean":"2024-09-21T12:21:00.794163+0000","last_became_active":"2024-09-21T12:20:21.315801+0000","last_became_peered":"2024-09-21T12:20:21.315801+0000","last_unstale":"2024-09-21T12:21:00.794163+0000","last_undegraded":"2024-09-21T12:21:00.794163+0000","last_fullsized":"2024-09-21T12:21:00.794163+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:28.234335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":84,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.795770+0000","last_change":"2024-09-21T12:20:17.693585+0000","last_active":"2024-09-21T12:21:00.795770+0000","last_peered":"2024-09-21T12:21:00.795770+0000","last_clean":"2024-09-21T12:21:00.795770+0000","last_became_active":"2024-09-21T12:20:17.693067+0000","last_became_peered":"2024-09-21T12:20:17.693067+0000","last_unstale":"2024-09-21T12:21:00.795770+0000","last_undegraded":"2024-09-21T12:21:00.795770+0000","last_fullsized":"2024-09-21T12:21:00.795770+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:37.693764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":52,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.794789+0000","last_change":"2024-09-21T12:20:15.308741+0000","last_active":"2024-09-21T12:21:00.794789+0000","last_peered":"2024-09-21T12:21:00.794789+0000","last_clean":"2024-09-21T12:21:00.794789+0000","last_became_active":"2024-09-21T12:20:15.308441+0000","last_became_peered":"2024-09-21T12:20:15.308441+0000","last_unstale":"2024-09-21T12:21:00.794789+0000","last_undegraded":"2024-09-21T12:21:00.794789+0000","last_fullsized":"2024-09-21T12:21:00.794789+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:29.626513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":42,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.673147+0000","last_change":"2024-09-21T12:20:19.309945+0000","last_active":"2024-09-21T12:20:57.673147+0000","last_peered":"2024-09-21T12:20:57.673147+0000","last_clean":"2024-09-21T12:20:57.673147+0000","last_became_active":"2024-09-21T12:20:19.309598+0000","last_became_peered":"2024-09-21T12:20:19.309598+0000","last_unstale":"2024-09-21T12:20:57.673147+0000","last_undegraded":"2024-09-21T12:20:57.673147+0000","last_fullsized":"2024-09-21T12:20:57.673147+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:31.055740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.793928+0000","last_change":"2024-09-21T12:20:21.593173+0000","last_active":"2024-09-21T12:21:00.793928+0000","last_peered":"2024-09-21T12:21:00.793928+0000","last_clean":"2024-09-21T12:21:00.793928+0000","last_became_active":"2024-09-21T12:20:21.592828+0000","last_became_peered":"2024-09-21T12:20:21.592828+0000","last_unstale":"2024-09-21T12:21:00.793928+0000","last_undegraded":"2024-09-21T12:21:00.793928+0000","last_fullsized":"2024-09-21T12:21:00.793928+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.351266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":106,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.794656+0000","last_change":"2024-09-21T12:20:17.308049+0000","last_active":"2024-09-21T12:21:00.794656+0000","last_peered":"2024-09-21T12:21:00.794656+0000","last_clean":"2024-09-21T12:21:00.794656+0000","last_became_active":"2024-09-21T12:20:17.307769+0000","last_became_peered":"2024-09-21T12:20:17.307769+0000","last_unstale":"2024-09-21T12:21:00.794656+0000","last_undegraded":"2024-09-21T12:21:00.794656+0000","last_fullsized":"2024-09-21T12:21:00.794656+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:29.183963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:56.761780+0000","last_change":"2024-09-21T12:20:15.299429+0000","last_active":"2024-09-21T12:20:56.761780+0000","last_peered":"2024-09-21T12:20:56.761780+0000","last_clean":"2024-09-21T12:20:56.761780+0000","last_became_active":"2024-09-21T12:20:15.299143+0000","last_became_peered":"2024-09-21T12:20:15.299143+0000","last_unstale":"2024-09-21T12:20:56.761780+0000","last_undegraded":"2024-09-21T12:20:56.761780+0000","last_fullsized":"2024-09-21T12:20:56.761780+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:42.581678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":62,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.793846+0000","last_change":"2024-09-21T12:20:17.298211+0000","last_active":"2024-09-21T12:21:00.793846+0000","last_peered":"2024-09-21T12:21:00.793846+0000","last_clean":"2024-09-21T12:21:00.793846+0000","last_became_active":"2024-09-21T12:20:14.287998+0000","last_became_peered":"2024-09-21T12:20:14.287998+0000","last_unstale":"2024-09-21T12:21:00.793846+0000","last_undegraded":"2024-09-21T12:21:00.793846+0000","last_fullsized":"2024-09-21T12:21:00.793846+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:20:13.272075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:19.171111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011011149999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":44,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.795240+0000","last_change":"2024-09-21T12:20:19.313478+0000","last_active":"2024-09-21T12:21:00.795240+0000","last_peered":"2024-09-21T12:21:00.795240+0000","last_clean":"2024-09-21T12:21:00.795240+0000","last_became_active":"2024-09-21T12:20:19.312170+0000","last_became_peered":"2024-09-21T12:20:19.312170+0000","last_unstale":"2024-09-21T12:21:00.795240+0000","last_undegraded":"2024-09-21T12:21:00.795240+0000","last_fullsized":"2024-09-21T12:21:00.795240+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.170409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":37,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.963400+0000","last_change":"2024-09-21T12:20:21.314452+0000","last_active":"2024-09-21T12:20:57.963400+0000","last_peered":"2024-09-21T12:20:57.963400+0000","last_clean":"2024-09-21T12:20:57.963400+0000","last_became_active":"2024-09-21T12:20:21.314126+0000","last_became_peered":"2024-09-21T12:20:21.314126+0000","last_unstale":"2024-09-21T12:20:57.963400+0000","last_undegraded":"2024-09-21T12:20:57.963400+0000","last_fullsized":"2024-09-21T12:20:57.963400+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:44.311699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":75,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.794825+0000","last_change":"2024-09-21T12:20:17.297678+0000","last_active":"2024-09-21T12:21:00.794825+0000","last_peered":"2024-09-21T12:21:00.794825+0000","last_clean":"2024-09-21T12:21:00.794825+0000","last_became_active":"2024-09-21T12:20:17.297374+0000","last_became_peered":"2024-09-21T12:20:17.297374+0000","last_unstale":"2024-09-21T12:21:00.794825+0000","last_undegraded":"2024-09-21T12:21:00.794825+0000","last_fullsized":"2024-09-21T12:21:00.794825+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:29.314219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":52,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.793930+0000","last_change":"2024-09-21T12:20:15.308648+0000","last_active":"2024-09-21T12:21:00.793930+0000","last_peered":"2024-09-21T12:21:00.793930+0000","last_clean":"2024-09-21T12:21:00.793930+0000","last_became_active":"2024-09-21T12:20:15.308356+0000","last_became_peered":"2024-09-21T12:20:15.308356+0000","last_unstale":"2024-09-21T12:21:00.793930+0000","last_undegraded":"2024-09-21T12:21:00.793930+0000","last_fullsized":"2024-09-21T12:21:00.793930+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:25.741492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":61,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.455148+0000","last_change":"2024-09-21T12:20:17.297432+0000","last_active":"2024-09-21T12:21:01.455148+0000","last_peered":"2024-09-21T12:21:01.455148+0000","last_clean":"2024-09-21T12:21:01.455148+0000","last_became_active":"2024-09-21T12:20:14.288358+0000","last_became_peered":"2024-09-21T12:20:14.288358+0000","last_unstale":"2024-09-21T12:21:01.455148+0000","last_undegraded":"2024-09-21T12:21:01.455148+0000","last_fullsized":"2024-09-21T12:21:01.455148+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:20:13.272075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:21.220031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019042449999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":44,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.455442+0000","last_change":"2024-09-21T12:20:19.310164+0000","last_active":"2024-09-21T12:21:01.455442+0000","last_peered":"2024-09-21T12:21:01.455442+0000","last_clean":"2024-09-21T12:21:01.455442+0000","last_became_active":"2024-09-21T12:20:19.309724+0000","last_became_peered":"2024-09-21T12:20:19.309724+0000","last_unstale":"2024-09-21T12:21:01.455442+0000","last_undegraded":"2024-09-21T12:21:01.455442+0000","last_fullsized":"2024-09-21T12:21:01.455442+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:43.623687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.056405+0000","last_change":"2024-09-21T12:20:21.593267+0000","last_active":"2024-09-21T12:20:57.056405+0000","last_peered":"2024-09-21T12:20:57.056405+0000","last_clean":"2024-09-21T12:20:57.056405+0000","last_became_active":"2024-09-21T12:20:21.592982+0000","last_became_peered":"2024-09-21T12:20:21.592982+0000","last_unstale":"2024-09-21T12:20:57.056405+0000","last_undegraded":"2024-09-21T12:20:57.056405+0000","last_fullsized":"2024-09-21T12:20:57.056405+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:43.824339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":66,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.795741+0000","last_change":"2024-09-21T12:20:17.299646+0000","last_active":"2024-09-21T12:21:00.795741+0000","last_peered":"2024-09-21T12:21:00.795741+0000","last_clean":"2024-09-21T12:21:00.795741+0000","last_became_active":"2024-09-21T12:20:17.299325+0000","last_became_peered":"2024-09-21T12:20:17.299325+0000","last_unstale":"2024-09-21T12:21:00.795741+0000","last_undegraded":"2024-09-21T12:21:00.795741+0000","last_fullsized":"2024-09-21T12:21:00.795741+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:22.183786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":52,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.794696+0000","last_change":"2024-09-21T12:20:15.302177+0000","last_active":"2024-09-21T12:21:00.794696+0000","last_peered":"2024-09-21T12:21:00.794696+0000","last_clean":"2024-09-21T12:21:00.794696+0000","last_became_active":"2024-09-21T12:20:15.301859+0000","last_became_peered":"2024-09-21T12:20:15.301859+0000","last_unstale":"2024-09-21T12:21:00.794696+0000","last_undegraded":"2024-09-21T12:21:00.794696+0000","last_fullsized":"2024-09-21T12:21:00.794696+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.505232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":102,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.084294+0000","last_change":"2024-09-21T12:20:17.693865+0000","last_active":"2024-09-21T12:21:00.084294+0000","last_peered":"2024-09-21T12:21:00.084294+0000","last_clean":"2024-09-21T12:21:00.084294+0000","last_became_active":"2024-09-21T12:20:14.287543+0000","last_became_peered":"2024-09-21T12:20:14.287543+0000","last_unstale":"2024-09-21T12:21:00.084294+0000","last_undegraded":"2024-09-21T12:21:00.084294+0000","last_fullsized":"2024-09-21T12:21:00.084294+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:20:13.272075+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:21.111464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":41,"num_read_kb":36,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":44,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.794404+0000","last_change":"2024-09-21T12:20:19.314574+0000","last_active":"2024-09-21T12:21:00.794404+0000","last_peered":"2024-09-21T12:21:00.794404+0000","last_clean":"2024-09-21T12:21:00.794404+0000","last_became_active":"2024-09-21T12:20:19.314217+0000","last_became_peered":"2024-09-21T12:20:19.314217+0000","last_unstale":"2024-09-21T12:21:00.794404+0000","last_undegraded":"2024-09-21T12:21:00.794404+0000","last_fullsized":"2024-09-21T12:21:00.794404+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:23.172823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.794525+0000","last_change":"2024-09-21T12:20:21.311501+0000","last_active":"2024-09-21T12:21:00.794525+0000","last_peered":"2024-09-21T12:21:00.794525+0000","last_clean":"2024-09-21T12:21:00.794525+0000","last_became_active":"2024-09-21T12:20:21.311188+0000","last_became_peered":"2024-09-21T12:20:21.311188+0000","last_unstale":"2024-09-21T12:21:00.794525+0000","last_undegraded":"2024-09-21T12:21:00.794525+0000","last_fullsized":"2024-09-21T12:21:00.794525+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:27.755252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"66'16","reported_seq":78,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.673699+0000","last_change":"2024-09-21T12:20:17.696268+0000","last_active":"2024-09-21T12:20:57.673699+0000","last_peered":"2024-09-21T12:20:57.673699+0000","last_clean":"2024-09-21T12:20:57.673699+0000","last_became_active":"2024-09-21T12:20:17.695940+0000","last_became_peered":"2024-09-21T12:20:17.695940+0000","last_unstale":"2024-09-21T12:20:57.673699+0000","last_undegraded":"2024-09-21T12:20:57.673699+0000","last_fullsized":"2024-09-21T12:20:57.673699+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:39.599877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":55,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.963009+0000","last_change":"2024-09-21T12:20:15.299151+0000","last_active":"2024-09-21T12:20:57.963009+0000","last_peered":"2024-09-21T12:20:57.963009+0000","last_clean":"2024-09-21T12:20:57.963009+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:20:57.963009+0000","last_undegraded":"2024-09-21T12:20:57.963009+0000","last_fullsized":"2024-09-21T12:20:57.963009+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:16.205349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":95,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.056111+0000","last_change":"2024-09-21T12:20:24.850239+0000","last_active":"2024-09-21T12:20:57.056111+0000","last_peered":"2024-09-21T12:20:57.056111+0000","last_clean":"2024-09-21T12:20:57.056111+0000","last_became_active":"2024-09-21T12:20:06.349670+0000","last_became_peered":"2024-09-21T12:20:06.349670+0000","last_unstale":"2024-09-21T12:20:57.056111+0000","last_undegraded":"2024-09-21T12:20:57.056111+0000","last_fullsized":"2024-09-21T12:20:57.056111+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T12:20:24.850157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:20:24.850157+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:53.046478+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":42,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.056347+0000","last_change":"2024-09-21T12:20:19.310280+0000","last_active":"2024-09-21T12:20:57.056347+0000","last_peered":"2024-09-21T12:20:57.056347+0000","last_clean":"2024-09-21T12:20:57.056347+0000","last_became_active":"2024-09-21T12:20:19.309859+0000","last_became_peered":"2024-09-21T12:20:19.309859+0000","last_unstale":"2024-09-21T12:20:57.056347+0000","last_undegraded":"2024-09-21T12:20:57.056347+0000","last_fullsized":"2024-09-21T12:20:57.056347+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:43.850206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.794325+0000","last_change":"2024-09-21T12:20:21.311690+0000","last_active":"2024-09-21T12:21:00.794325+0000","last_peered":"2024-09-21T12:21:00.794325+0000","last_clean":"2024-09-21T12:21:00.794325+0000","last_became_active":"2024-09-21T12:20:21.311194+0000","last_became_peered":"2024-09-21T12:20:21.311194+0000","last_unstale":"2024-09-21T12:21:00.794325+0000","last_undegraded":"2024-09-21T12:21:00.794325+0000","last_fullsized":"2024-09-21T12:21:00.794325+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:25.752002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":65,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:56.762083+0000","last_change":"2024-09-21T12:20:17.310299+0000","last_active":"2024-09-21T12:20:56.762083+0000","last_peered":"2024-09-21T12:20:56.762083+0000","last_clean":"2024-09-21T12:20:56.762083+0000","last_became_active":"2024-09-21T12:20:17.309930+0000","last_became_peered":"2024-09-21T12:20:17.309930+0000","last_unstale":"2024-09-21T12:20:56.762083+0000","last_undegraded":"2024-09-21T12:20:56.762083+0000","last_fullsized":"2024-09-21T12:20:56.762083+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:17.883258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:56.761478+0000","last_change":"2024-09-21T12:20:15.310346+0000","last_active":"2024-09-21T12:20:56.761478+0000","last_peered":"2024-09-21T12:20:56.761478+0000","last_clean":"2024-09-21T12:20:56.761478+0000","last_became_active":"2024-09-21T12:20:15.310048+0000","last_became_peered":"2024-09-21T12:20:15.310048+0000","last_unstale":"2024-09-21T12:20:56.761478+0000","last_undegraded":"2024-09-21T12:20:56.761478+0000","last_fullsized":"2024-09-21T12:20:56.761478+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:31.701321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":44,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.794226+0000","last_change":"2024-09-21T12:20:19.309554+0000","last_active":"2024-09-21T12:21:00.794226+0000","last_peered":"2024-09-21T12:21:00.794226+0000","last_clean":"2024-09-21T12:21:00.794226+0000","last_became_active":"2024-09-21T12:20:19.309258+0000","last_became_peered":"2024-09-21T12:20:19.309258+0000","last_unstale":"2024-09-21T12:21:00.794226+0000","last_undegraded":"2024-09-21T12:21:00.794226+0000","last_fullsized":"2024-09-21T12:21:00.794226+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:44.940960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":37,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.963212+0000","last_change":"2024-09-21T12:20:21.593145+0000","last_active":"2024-09-21T12:20:57.963212+0000","last_peered":"2024-09-21T12:20:57.963212+0000","last_clean":"2024-09-21T12:20:57.963212+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:20:57.963212+0000","last_undegraded":"2024-09-21T12:20:57.963212+0000","last_fullsized":"2024-09-21T12:20:57.963212+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.236407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":74,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:56.762124+0000","last_change":"2024-09-21T12:20:17.310292+0000","last_active":"2024-09-21T12:20:56.762124+0000","last_peered":"2024-09-21T12:20:56.762124+0000","last_clean":"2024-09-21T12:20:56.762124+0000","last_became_active":"2024-09-21T12:20:17.309828+0000","last_became_peered":"2024-09-21T12:20:17.309828+0000","last_unstale":"2024-09-21T12:20:56.762124+0000","last_undegraded":"2024-09-21T12:20:56.762124+0000","last_fullsized":"2024-09-21T12:20:56.762124+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:44.016276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.673418+0000","last_change":"2024-09-21T12:20:15.302137+0000","last_active":"2024-09-21T12:20:57.673418+0000","last_peered":"2024-09-21T12:20:57.673418+0000","last_clean":"2024-09-21T12:20:57.673418+0000","last_became_active":"2024-09-21T12:20:15.301851+0000","last_became_peered":"2024-09-21T12:20:15.301851+0000","last_unstale":"2024-09-21T12:20:57.673418+0000","last_undegraded":"2024-09-21T12:20:57.673418+0000","last_fullsized":"2024-09-21T12:20:57.673418+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.661202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":44,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.794566+0000","last_change":"2024-09-21T12:20:19.312912+0000","last_active":"2024-09-21T12:21:00.794566+0000","last_peered":"2024-09-21T12:21:00.794566+0000","last_clean":"2024-09-21T12:21:00.794566+0000","last_became_active":"2024-09-21T12:20:19.312301+0000","last_became_peered":"2024-09-21T12:20:19.312301+0000","last_unstale":"2024-09-21T12:21:00.794566+0000","last_undegraded":"2024-09-21T12:21:00.794566+0000","last_fullsized":"2024-09-21T12:21:00.794566+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:42.511905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.455385+0000","last_change":"2024-09-21T12:20:21.313585+0000","last_active":"2024-09-21T12:21:01.455385+0000","last_peered":"2024-09-21T12:21:01.455385+0000","last_clean":"2024-09-21T12:21:01.455385+0000","last_became_active":"2024-09-21T12:20:21.313254+0000","last_became_peered":"2024-09-21T12:20:21.313254+0000","last_unstale":"2024-09-21T12:21:01.455385+0000","last_undegraded":"2024-09-21T12:21:01.455385+0000","last_fullsized":"2024-09-21T12:21:01.455385+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.442191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:56.761646+0000","last_change":"2024-09-21T12:20:17.307651+0000","last_active":"2024-09-21T12:20:56.761646+0000","last_peered":"2024-09-21T12:20:56.761646+0000","last_clean":"2024-09-21T12:20:56.761646+0000","last_became_active":"2024-09-21T12:20:17.307317+0000","last_became_peered":"2024-09-21T12:20:17.307317+0000","last_unstale":"2024-09-21T12:20:56.761646+0000","last_undegraded":"2024-09-21T12:20:56.761646+0000","last_fullsized":"2024-09-21T12:20:56.761646+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:37.610294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":49,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.963312+0000","last_change":"2024-09-21T12:20:15.298107+0000","last_active":"2024-09-21T12:20:57.963312+0000","last_peered":"2024-09-21T12:20:57.963312+0000","last_clean":"2024-09-21T12:20:57.963312+0000","last_became_active":"2024-09-21T12:20:15.297819+0000","last_became_peered":"2024-09-21T12:20:15.297819+0000","last_unstale":"2024-09-21T12:20:57.963312+0000","last_undegraded":"2024-09-21T12:20:57.963312+0000","last_fullsized":"2024-09-21T12:20:57.963312+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:38.788978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":62,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.455124+0000","last_change":"2024-09-21T12:20:19.308246+0000","last_active":"2024-09-21T12:21:01.455124+0000","last_peered":"2024-09-21T12:21:01.455124+0000","last_clean":"2024-09-21T12:21:01.455124+0000","last_became_active":"2024-09-21T12:20:19.307924+0000","last_became_peered":"2024-09-21T12:20:19.307924+0000","last_unstale":"2024-09-21T12:21:01.455124+0000","last_undegraded":"2024-09-21T12:21:01.455124+0000","last_fullsized":"2024-09-21T12:21:01.455124+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:33.476030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:56.761789+0000","last_change":"2024-09-21T12:20:21.313250+0000","last_active":"2024-09-21T12:20:56.761789+0000","last_peered":"2024-09-21T12:20:56.761789+0000","last_clean":"2024-09-21T12:20:56.761789+0000","last_became_active":"2024-09-21T12:20:21.312954+0000","last_became_peered":"2024-09-21T12:20:21.312954+0000","last_unstale":"2024-09-21T12:20:56.761789+0000","last_undegraded":"2024-09-21T12:20:56.761789+0000","last_fullsized":"2024-09-21T12:20:56.761789+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.513060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":64,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.795709+0000","last_change":"2024-09-21T12:20:17.299948+0000","last_active":"2024-09-21T12:21:00.795709+0000","last_peered":"2024-09-21T12:21:00.795709+0000","last_clean":"2024-09-21T12:21:00.795709+0000","last_became_active":"2024-09-21T12:20:17.299653+0000","last_became_peered":"2024-09-21T12:20:17.299653+0000","last_unstale":"2024-09-21T12:21:00.795709+0000","last_undegraded":"2024-09-21T12:21:00.795709+0000","last_fullsized":"2024-09-21T12:21:00.795709+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:16.539847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":52,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.794066+0000","last_change":"2024-09-21T12:20:15.302587+0000","last_active":"2024-09-21T12:21:00.794066+0000","last_peered":"2024-09-21T12:21:00.794066+0000","last_clean":"2024-09-21T12:21:00.794066+0000","last_became_active":"2024-09-21T12:20:15.302262+0000","last_became_peered":"2024-09-21T12:20:15.302262+0000","last_unstale":"2024-09-21T12:21:00.794066+0000","last_undegraded":"2024-09-21T12:21:00.794066+0000","last_fullsized":"2024-09-21T12:21:00.794066+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:32.093775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":44,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.454968+0000","last_change":"2024-09-21T12:20:19.308484+0000","last_active":"2024-09-21T12:21:01.454968+0000","last_peered":"2024-09-21T12:21:01.454968+0000","last_clean":"2024-09-21T12:21:01.454968+0000","last_became_active":"2024-09-21T12:20:19.308048+0000","last_became_peered":"2024-09-21T12:20:19.308048+0000","last_unstale":"2024-09-21T12:21:01.454968+0000","last_undegraded":"2024-09-21T12:21:01.454968+0000","last_fullsized":"2024-09-21T12:21:01.454968+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:34.470562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.794613+0000","last_change":"2024-09-21T12:20:21.311799+0000","last_active":"2024-09-21T12:21:00.794613+0000","last_peered":"2024-09-21T12:21:00.794613+0000","last_clean":"2024-09-21T12:21:00.794613+0000","last_became_active":"2024-09-21T12:20:21.311493+0000","last_became_peered":"2024-09-21T12:20:21.311493+0000","last_unstale":"2024-09-21T12:21:00.794613+0000","last_undegraded":"2024-09-21T12:21:00.794613+0000","last_fullsized":"2024-09-21T12:21:00.794613+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:40.676620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":77,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.673278+0000","last_change":"2024-09-21T12:20:17.693677+0000","last_active":"2024-09-21T12:20:57.673278+0000","last_peered":"2024-09-21T12:20:57.673278+0000","last_clean":"2024-09-21T12:20:57.673278+0000","last_became_active":"2024-09-21T12:20:17.692867+0000","last_became_peered":"2024-09-21T12:20:17.692867+0000","last_unstale":"2024-09-21T12:20:57.673278+0000","last_undegraded":"2024-09-21T12:20:57.673278+0000","last_fullsized":"2024-09-21T12:20:57.673278+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.747644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.056669+0000","last_change":"2024-09-21T12:20:15.300854+0000","last_active":"2024-09-21T12:20:57.056669+0000","last_peered":"2024-09-21T12:20:57.056669+0000","last_clean":"2024-09-21T12:20:57.056669+0000","last_became_active":"2024-09-21T12:20:15.299333+0000","last_became_peered":"2024-09-21T12:20:15.299333+0000","last_unstale":"2024-09-21T12:20:57.056669+0000","last_undegraded":"2024-09-21T12:20:57.056669+0000","last_fullsized":"2024-09-21T12:20:57.056669+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:34.912500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":44,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.454896+0000","last_change":"2024-09-21T12:20:19.307316+0000","last_active":"2024-09-21T12:21:01.454896+0000","last_peered":"2024-09-21T12:21:01.454896+0000","last_clean":"2024-09-21T12:21:01.454896+0000","last_became_active":"2024-09-21T12:20:19.306952+0000","last_became_peered":"2024-09-21T12:20:19.306952+0000","last_unstale":"2024-09-21T12:21:01.454896+0000","last_undegraded":"2024-09-21T12:21:01.454896+0000","last_fullsized":"2024-09-21T12:21:01.454896+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:47.444300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.056180+0000","last_change":"2024-09-21T12:20:21.314891+0000","last_active":"2024-09-21T12:20:57.056180+0000","last_peered":"2024-09-21T12:20:57.056180+0000","last_clean":"2024-09-21T12:20:57.056180+0000","last_became_active":"2024-09-21T12:20:21.314545+0000","last_became_peered":"2024-09-21T12:20:21.314545+0000","last_unstale":"2024-09-21T12:20:57.056180+0000","last_undegraded":"2024-09-21T12:20:57.056180+0000","last_fullsized":"2024-09-21T12:20:57.056180+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:21.854833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":69,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:56.761381+0000","last_change":"2024-09-21T12:20:17.306090+0000","last_active":"2024-09-21T12:20:56.761381+0000","last_peered":"2024-09-21T12:20:56.761381+0000","last_clean":"2024-09-21T12:20:56.761381+0000","last_became_active":"2024-09-21T12:20:17.305661+0000","last_became_peered":"2024-09-21T12:20:17.305661+0000","last_unstale":"2024-09-21T12:20:56.761381+0000","last_undegraded":"2024-09-21T12:20:56.761381+0000","last_fullsized":"2024-09-21T12:20:56.761381+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:31.017952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.056744+0000","last_change":"2024-09-21T12:20:15.305876+0000","last_active":"2024-09-21T12:20:57.056744+0000","last_peered":"2024-09-21T12:20:57.056744+0000","last_clean":"2024-09-21T12:20:57.056744+0000","last_became_active":"2024-09-21T12:20:15.305545+0000","last_became_peered":"2024-09-21T12:20:15.305545+0000","last_unstale":"2024-09-21T12:20:57.056744+0000","last_undegraded":"2024-09-21T12:20:57.056744+0000","last_fullsized":"2024-09-21T12:20:57.056744+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:19.227777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":44,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.454862+0000","last_change":"2024-09-21T12:20:19.312986+0000","last_active":"2024-09-21T12:21:01.454862+0000","last_peered":"2024-09-21T12:21:01.454862+0000","last_clean":"2024-09-21T12:21:01.454862+0000","last_became_active":"2024-09-21T12:20:19.312572+0000","last_became_peered":"2024-09-21T12:20:19.312572+0000","last_unstale":"2024-09-21T12:21:01.454862+0000","last_undegraded":"2024-09-21T12:21:01.454862+0000","last_fullsized":"2024-09-21T12:21:01.454862+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:20.301869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":37,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.963153+0000","last_change":"2024-09-21T12:20:21.317550+0000","last_active":"2024-09-21T12:20:57.963153+0000","last_peered":"2024-09-21T12:20:57.963153+0000","last_clean":"2024-09-21T12:20:57.963153+0000","last_became_active":"2024-09-21T12:20:21.317223+0000","last_became_peered":"2024-09-21T12:20:21.317223+0000","last_unstale":"2024-09-21T12:20:57.963153+0000","last_undegraded":"2024-09-21T12:20:57.963153+0000","last_fullsized":"2024-09-21T12:20:57.963153+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:25.406157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":73,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.793556+0000","last_change":"2024-09-21T12:20:17.693302+0000","last_active":"2024-09-21T12:21:00.793556+0000","last_peered":"2024-09-21T12:21:00.793556+0000","last_clean":"2024-09-21T12:21:00.793556+0000","last_became_active":"2024-09-21T12:20:17.692870+0000","last_became_peered":"2024-09-21T12:20:17.692870+0000","last_unstale":"2024-09-21T12:21:00.793556+0000","last_undegraded":"2024-09-21T12:21:00.793556+0000","last_fullsized":"2024-09-21T12:21:00.793556+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.102470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":62,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.056735+0000","last_change":"2024-09-21T12:20:15.302366+0000","last_active":"2024-09-21T12:20:57.056735+0000","last_peered":"2024-09-21T12:20:57.056735+0000","last_clean":"2024-09-21T12:20:57.056735+0000","last_became_active":"2024-09-21T12:20:15.302038+0000","last_became_peered":"2024-09-21T12:20:15.302038+0000","last_unstale":"2024-09-21T12:20:57.056735+0000","last_undegraded":"2024-09-21T12:20:57.056735+0000","last_fullsized":"2024-09-21T12:20:57.056735+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:29.184795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":60,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.056128+0000","last_change":"2024-09-21T12:20:19.310188+0000","last_active":"2024-09-21T12:20:57.056128+0000","last_peered":"2024-09-21T12:20:57.056128+0000","last_clean":"2024-09-21T12:20:57.056128+0000","last_became_active":"2024-09-21T12:20:19.309708+0000","last_became_peered":"2024-09-21T12:20:19.309708+0000","last_unstale":"2024-09-21T12:20:57.056128+0000","last_undegraded":"2024-09-21T12:20:57.056128+0000","last_fullsized":"2024-09-21T12:20:57.056128+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:44.922935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.793638+0000","last_change":"2024-09-21T12:20:21.592893+0000","last_active":"2024-09-21T12:21:00.793638+0000","last_peered":"2024-09-21T12:21:00.793638+0000","last_clean":"2024-09-21T12:21:00.793638+0000","last_became_active":"2024-09-21T12:20:21.592576+0000","last_became_peered":"2024-09-21T12:20:21.592576+0000","last_unstale":"2024-09-21T12:21:00.793638+0000","last_undegraded":"2024-09-21T12:21:00.793638+0000","last_fullsized":"2024-09-21T12:21:00.793638+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:45.792911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.672984+0000","last_change":"2024-09-21T12:20:15.303770+0000","last_active":"2024-09-21T12:20:57.672984+0000","last_peered":"2024-09-21T12:20:57.672984+0000","last_clean":"2024-09-21T12:20:57.672984+0000","last_became_active":"2024-09-21T12:20:15.303480+0000","last_became_peered":"2024-09-21T12:20:15.303480+0000","last_unstale":"2024-09-21T12:20:57.672984+0000","last_undegraded":"2024-09-21T12:20:57.672984+0000","last_fullsized":"2024-09-21T12:20:57.672984+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:14.876724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":54,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.962459+0000","last_change":"2024-09-21T12:20:17.694977+0000","last_active":"2024-09-21T12:20:57.962459+0000","last_peered":"2024-09-21T12:20:57.962459+0000","last_clean":"2024-09-21T12:20:57.962459+0000","last_became_active":"2024-09-21T12:20:17.694628+0000","last_became_peered":"2024-09-21T12:20:17.694628+0000","last_unstale":"2024-09-21T12:20:57.962459+0000","last_undegraded":"2024-09-21T12:20:57.962459+0000","last_fullsized":"2024-09-21T12:20:57.962459+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.288937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":44,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.794195+0000","last_change":"2024-09-21T12:20:19.314432+0000","last_active":"2024-09-21T12:21:00.794195+0000","last_peered":"2024-09-21T12:21:00.794195+0000","last_clean":"2024-09-21T12:21:00.794195+0000","last_became_active":"2024-09-21T12:20:19.313882+0000","last_became_peered":"2024-09-21T12:20:19.313882+0000","last_unstale":"2024-09-21T12:21:00.794195+0000","last_undegraded":"2024-09-21T12:21:00.794195+0000","last_fullsized":"2024-09-21T12:21:00.794195+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.453804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.056385+0000","last_change":"2024-09-21T12:20:21.594311+0000","last_active":"2024-09-21T12:20:57.056385+0000","last_peered":"2024-09-21T12:20:57.056385+0000","last_clean":"2024-09-21T12:20:57.056385+0000","last_became_active":"2024-09-21T12:20:21.594048+0000","last_became_peered":"2024-09-21T12:20:21.594048+0000","last_unstale":"2024-09-21T12:20:57.056385+0000","last_undegraded":"2024-09-21T12:20:57.056385+0000","last_fullsized":"2024-09-21T12:20:57.056385+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:38.347489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":64,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.795769+0000","last_change":"2024-09-21T12:20:17.696279+0000","last_active":"2024-09-21T12:21:00.795769+0000","last_peered":"2024-09-21T12:21:00.795769+0000","last_clean":"2024-09-21T12:21:00.795769+0000","last_became_active":"2024-09-21T12:20:17.695898+0000","last_became_peered":"2024-09-21T12:20:17.695898+0000","last_unstale":"2024-09-21T12:21:00.795769+0000","last_undegraded":"2024-09-21T12:21:00.795769+0000","last_fullsized":"2024-09-21T12:21:00.795769+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:21.926785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.056258+0000","last_change":"2024-09-21T12:20:15.301560+0000","last_active":"2024-09-21T12:20:57.056258+0000","last_peered":"2024-09-21T12:20:57.056258+0000","last_clean":"2024-09-21T12:20:57.056258+0000","last_became_active":"2024-09-21T12:20:15.300722+0000","last_became_peered":"2024-09-21T12:20:15.300722+0000","last_unstale":"2024-09-21T12:20:57.056258+0000","last_undegraded":"2024-09-21T12:20:57.056258+0000","last_fullsized":"2024-09-21T12:20:57.056258+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.142850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":41,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.963487+0000","last_change":"2024-09-21T12:20:19.312248+0000","last_active":"2024-09-21T12:20:57.963487+0000","last_peered":"2024-09-21T12:20:57.963487+0000","last_clean":"2024-09-21T12:20:57.963487+0000","last_became_active":"2024-09-21T12:20:19.311947+0000","last_became_peered":"2024-09-21T12:20:19.311947+0000","last_unstale":"2024-09-21T12:20:57.963487+0000","last_undegraded":"2024-09-21T12:20:57.963487+0000","last_fullsized":"2024-09-21T12:20:57.963487+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.388400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.455309+0000","last_change":"2024-09-21T12:20:21.314554+0000","last_active":"2024-09-21T12:21:01.455309+0000","last_peered":"2024-09-21T12:21:01.455309+0000","last_clean":"2024-09-21T12:21:01.455309+0000","last_became_active":"2024-09-21T12:20:21.314277+0000","last_became_peered":"2024-09-21T12:20:21.314277+0000","last_unstale":"2024-09-21T12:21:01.455309+0000","last_undegraded":"2024-09-21T12:21:01.455309+0000","last_fullsized":"2024-09-21T12:21:01.455309+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.219191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":64,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.794150+0000","last_change":"2024-09-21T12:20:17.694904+0000","last_active":"2024-09-21T12:21:00.794150+0000","last_peered":"2024-09-21T12:21:00.794150+0000","last_clean":"2024-09-21T12:21:00.794150+0000","last_became_active":"2024-09-21T12:20:17.694568+0000","last_became_peered":"2024-09-21T12:20:17.694568+0000","last_unstale":"2024-09-21T12:21:00.794150+0000","last_undegraded":"2024-09-21T12:21:00.794150+0000","last_fullsized":"2024-09-21T12:21:00.794150+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:24.206067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":52,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.794938+0000","last_change":"2024-09-21T12:20:15.297853+0000","last_active":"2024-09-21T12:21:00.794938+0000","last_peered":"2024-09-21T12:21:00.794938+0000","last_clean":"2024-09-21T12:21:00.794938+0000","last_became_active":"2024-09-21T12:20:15.297519+0000","last_became_peered":"2024-09-21T12:20:15.297519+0000","last_unstale":"2024-09-21T12:21:00.794938+0000","last_undegraded":"2024-09-21T12:21:00.794938+0000","last_fullsized":"2024-09-21T12:21:00.794938+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:29.814404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.963311+0000","last_change":"2024-09-21T12:20:19.305689+0000","last_active":"2024-09-21T12:20:57.963311+0000","last_peered":"2024-09-21T12:20:57.963311+0000","last_clean":"2024-09-21T12:20:57.963311+0000","last_became_active":"2024-09-21T12:20:19.305364+0000","last_became_peered":"2024-09-21T12:20:19.305364+0000","last_unstale":"2024-09-21T12:20:57.963311+0000","last_undegraded":"2024-09-21T12:20:57.963311+0000","last_fullsized":"2024-09-21T12:20:57.963311+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.650981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:56.762081+0000","last_change":"2024-09-21T12:20:21.315079+0000","last_active":"2024-09-21T12:20:56.762081+0000","last_peered":"2024-09-21T12:20:56.762081+0000","last_clean":"2024-09-21T12:20:56.762081+0000","last_became_active":"2024-09-21T12:20:21.314066+0000","last_became_peered":"2024-09-21T12:20:21.314066+0000","last_unstale":"2024-09-21T12:20:56.762081+0000","last_undegraded":"2024-09-21T12:20:56.762081+0000","last_fullsized":"2024-09-21T12:20:56.762081+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:27.528692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":60,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.962612+0000","last_change":"2024-09-21T12:20:17.693562+0000","last_active":"2024-09-21T12:20:57.962612+0000","last_peered":"2024-09-21T12:20:57.962612+0000","last_clean":"2024-09-21T12:20:57.962612+0000","last_became_active":"2024-09-21T12:20:17.693209+0000","last_became_peered":"2024-09-21T12:20:17.693209+0000","last_unstale":"2024-09-21T12:20:57.962612+0000","last_undegraded":"2024-09-21T12:20:57.962612+0000","last_fullsized":"2024-09-21T12:20:57.962612+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:25.923701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.056322+0000","last_change":"2024-09-21T12:20:15.308767+0000","last_active":"2024-09-21T12:20:57.056322+0000","last_peered":"2024-09-21T12:20:57.056322+0000","last_clean":"2024-09-21T12:20:57.056322+0000","last_became_active":"2024-09-21T12:20:15.308429+0000","last_became_peered":"2024-09-21T12:20:15.308429+0000","last_unstale":"2024-09-21T12:20:57.056322+0000","last_undegraded":"2024-09-21T12:20:57.056322+0000","last_fullsized":"2024-09-21T12:20:57.056322+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:15.014581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":62,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.794021+0000","last_change":"2024-09-21T12:20:19.314884+0000","last_active":"2024-09-21T12:21:00.794021+0000","last_peered":"2024-09-21T12:21:00.794021+0000","last_clean":"2024-09-21T12:21:00.794021+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:21:00.794021+0000","last_undegraded":"2024-09-21T12:21:00.794021+0000","last_fullsized":"2024-09-21T12:21:00.794021+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:29.894782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.795566+0000","last_change":"2024-09-21T12:20:21.309936+0000","last_active":"2024-09-21T12:21:00.795566+0000","last_peered":"2024-09-21T12:21:00.795566+0000","last_clean":"2024-09-21T12:21:00.795566+0000","last_became_active":"2024-09-21T12:20:21.309385+0000","last_became_peered":"2024-09-21T12:20:21.309385+0000","last_unstale":"2024-09-21T12:21:00.795566+0000","last_undegraded":"2024-09-21T12:21:00.795566+0000","last_fullsized":"2024-09-21T12:21:00.795566+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:47.499218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":65,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:56.761500+0000","last_change":"2024-09-21T12:20:17.304948+0000","last_active":"2024-09-21T12:20:56.761500+0000","last_peered":"2024-09-21T12:20:56.761500+0000","last_clean":"2024-09-21T12:20:56.761500+0000","last_became_active":"2024-09-21T12:20:17.303763+0000","last_became_peered":"2024-09-21T12:20:17.303763+0000","last_unstale":"2024-09-21T12:20:56.761500+0000","last_undegraded":"2024-09-21T12:20:56.761500+0000","last_fullsized":"2024-09-21T12:20:56.761500+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:31.523086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:56.761849+0000","last_change":"2024-09-21T12:20:15.298500+0000","last_active":"2024-09-21T12:20:56.761849+0000","last_peered":"2024-09-21T12:20:56.761849+0000","last_clean":"2024-09-21T12:20:56.761849+0000","last_became_active":"2024-09-21T12:20:15.298198+0000","last_became_peered":"2024-09-21T12:20:15.298198+0000","last_unstale":"2024-09-21T12:20:56.761849+0000","last_undegraded":"2024-09-21T12:20:56.761849+0000","last_fullsized":"2024-09-21T12:20:56.761849+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:36.549556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":44,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.794220+0000","last_change":"2024-09-21T12:20:19.315714+0000","last_active":"2024-09-21T12:21:00.794220+0000","last_peered":"2024-09-21T12:21:00.794220+0000","last_clean":"2024-09-21T12:21:00.794220+0000","last_became_active":"2024-09-21T12:20:19.315418+0000","last_became_peered":"2024-09-21T12:20:19.315418+0000","last_unstale":"2024-09-21T12:21:00.794220+0000","last_undegraded":"2024-09-21T12:21:00.794220+0000","last_fullsized":"2024-09-21T12:21:00.794220+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:38.005096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":37,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.962779+0000","last_change":"2024-09-21T12:20:21.310344+0000","last_active":"2024-09-21T12:20:57.962779+0000","last_peered":"2024-09-21T12:20:57.962779+0000","last_clean":"2024-09-21T12:20:57.962779+0000","last_became_active":"2024-09-21T12:20:21.310053+0000","last_became_peered":"2024-09-21T12:20:21.310053+0000","last_unstale":"2024-09-21T12:20:57.962779+0000","last_undegraded":"2024-09-21T12:20:57.962779+0000","last_fullsized":"2024-09-21T12:20:57.962779+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:42.963350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":64,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.794360+0000","last_change":"2024-09-21T12:20:17.299367+0000","last_active":"2024-09-21T12:21:00.794360+0000","last_peered":"2024-09-21T12:21:00.794360+0000","last_clean":"2024-09-21T12:21:00.794360+0000","last_became_active":"2024-09-21T12:20:17.299015+0000","last_became_peered":"2024-09-21T12:20:17.299015+0000","last_unstale":"2024-09-21T12:21:00.794360+0000","last_undegraded":"2024-09-21T12:21:00.794360+0000","last_fullsized":"2024-09-21T12:21:00.794360+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:42.174386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":50,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.056548+0000","last_change":"2024-09-21T12:20:15.303462+0000","last_active":"2024-09-21T12:20:57.056548+0000","last_peered":"2024-09-21T12:20:57.056548+0000","last_clean":"2024-09-21T12:20:57.056548+0000","last_became_active":"2024-09-21T12:20:15.300554+0000","last_became_peered":"2024-09-21T12:20:15.300554+0000","last_unstale":"2024-09-21T12:20:57.056548+0000","last_undegraded":"2024-09-21T12:20:57.056548+0000","last_fullsized":"2024-09-21T12:20:57.056548+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:22.850479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":41,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.963215+0000","last_change":"2024-09-21T12:20:19.311327+0000","last_active":"2024-09-21T12:20:57.963215+0000","last_peered":"2024-09-21T12:20:57.963215+0000","last_clean":"2024-09-21T12:20:57.963215+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:20:57.963215+0000","last_undegraded":"2024-09-21T12:20:57.963215+0000","last_fullsized":"2024-09-21T12:20:57.963215+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:20.603464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.793857+0000","last_change":"2024-09-21T12:20:21.308003+0000","last_active":"2024-09-21T12:21:00.793857+0000","last_peered":"2024-09-21T12:21:00.793857+0000","last_clean":"2024-09-21T12:21:00.793857+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:21:00.793857+0000","last_undegraded":"2024-09-21T12:21:00.793857+0000","last_fullsized":"2024-09-21T12:21:00.793857+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:41.631005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":64,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.962491+0000","last_change":"2024-09-21T12:20:17.695358+0000","last_active":"2024-09-21T12:20:57.962491+0000","last_peered":"2024-09-21T12:20:57.962491+0000","last_clean":"2024-09-21T12:20:57.962491+0000","last_became_active":"2024-09-21T12:20:17.694794+0000","last_became_peered":"2024-09-21T12:20:17.694794+0000","last_unstale":"2024-09-21T12:20:57.962491+0000","last_undegraded":"2024-09-21T12:20:57.962491+0000","last_fullsized":"2024-09-21T12:20:57.962491+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:25.305944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":52,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.794285+0000","last_change":"2024-09-21T12:20:15.296285+0000","last_active":"2024-09-21T12:21:00.794285+0000","last_peered":"2024-09-21T12:21:00.794285+0000","last_clean":"2024-09-21T12:21:00.794285+0000","last_became_active":"2024-09-21T12:20:15.295968+0000","last_became_peered":"2024-09-21T12:20:15.295968+0000","last_unstale":"2024-09-21T12:21:00.794285+0000","last_undegraded":"2024-09-21T12:21:00.794285+0000","last_fullsized":"2024-09-21T12:21:00.794285+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:33.273850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":42,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.055699+0000","last_change":"2024-09-21T12:20:19.309473+0000","last_active":"2024-09-21T12:20:57.055699+0000","last_peered":"2024-09-21T12:20:57.055699+0000","last_clean":"2024-09-21T12:20:57.055699+0000","last_became_active":"2024-09-21T12:20:19.309195+0000","last_became_peered":"2024-09-21T12:20:19.309195+0000","last_unstale":"2024-09-21T12:20:57.055699+0000","last_undegraded":"2024-09-21T12:20:57.055699+0000","last_fullsized":"2024-09-21T12:20:57.055699+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:39.580857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":37,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.963266+0000","last_change":"2024-09-21T12:20:21.595660+0000","last_active":"2024-09-21T12:20:57.963266+0000","last_peered":"2024-09-21T12:20:57.963266+0000","last_clean":"2024-09-21T12:20:57.963266+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:20:57.963266+0000","last_undegraded":"2024-09-21T12:20:57.963266+0000","last_fullsized":"2024-09-21T12:20:57.963266+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:22.428602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":51,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.962482+0000","last_change":"2024-09-21T12:20:17.304075+0000","last_active":"2024-09-21T12:20:57.962482+0000","last_peered":"2024-09-21T12:20:57.962482+0000","last_clean":"2024-09-21T12:20:57.962482+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:20:57.962482+0000","last_undegraded":"2024-09-21T12:20:57.962482+0000","last_fullsized":"2024-09-21T12:20:57.962482+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.024324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":58,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.795423+0000","last_change":"2024-09-21T12:20:15.298443+0000","last_active":"2024-09-21T12:21:00.795423+0000","last_peered":"2024-09-21T12:21:00.795423+0000","last_clean":"2024-09-21T12:21:00.795423+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:21:00.795423+0000","last_undegraded":"2024-09-21T12:21:00.795423+0000","last_fullsized":"2024-09-21T12:21:00.795423+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:36.774153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":42,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.673146+0000","last_change":"2024-09-21T12:20:19.312978+0000","last_active":"2024-09-21T12:20:57.673146+0000","last_peered":"2024-09-21T12:20:57.673146+0000","last_clean":"2024-09-21T12:20:57.673146+0000","last_became_active":"2024-09-21T12:20:19.312512+0000","last_became_peered":"2024-09-21T12:20:19.312512+0000","last_unstale":"2024-09-21T12:20:57.673146+0000","last_undegraded":"2024-09-21T12:20:57.673146+0000","last_fullsized":"2024-09-21T12:20:57.673146+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:41.253296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.055936+0000","last_change":"2024-09-21T12:20:21.313523+0000","last_active":"2024-09-21T12:20:57.055936+0000","last_peered":"2024-09-21T12:20:57.055936+0000","last_clean":"2024-09-21T12:20:57.055936+0000","last_became_active":"2024-09-21T12:20:21.313228+0000","last_became_peered":"2024-09-21T12:20:21.313228+0000","last_unstale":"2024-09-21T12:20:57.055936+0000","last_undegraded":"2024-09-21T12:20:57.055936+0000","last_fullsized":"2024-09-21T12:20:57.055936+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:47.112414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.793857+0000","last_change":"2024-09-21T12:20:21.309169+0000","last_active":"2024-09-21T12:21:00.793857+0000","last_peered":"2024-09-21T12:21:00.793857+0000","last_clean":"2024-09-21T12:21:00.793857+0000","last_became_active":"2024-09-21T12:20:21.308865+0000","last_became_peered":"2024-09-21T12:20:21.308865+0000","last_unstale":"2024-09-21T12:21:00.793857+0000","last_undegraded":"2024-09-21T12:21:00.793857+0000","last_fullsized":"2024-09-21T12:21:00.793857+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:26.841000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":49,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.962719+0000","last_change":"2024-09-21T12:20:15.298322+0000","last_active":"2024-09-21T12:20:57.962719+0000","last_peered":"2024-09-21T12:20:57.962719+0000","last_clean":"2024-09-21T12:20:57.962719+0000","last_became_active":"2024-09-21T12:20:15.297993+0000","last_became_peered":"2024-09-21T12:20:15.297993+0000","last_unstale":"2024-09-21T12:20:57.962719+0000","last_undegraded":"2024-09-21T12:20:57.962719+0000","last_fullsized":"2024-09-21T12:20:57.962719+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:34.289061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":65,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T12:20:57.672703+0000","last_change":"2024-09-21T12:20:17.299134+0000","last_active":"2024-09-21T12:20:57.672703+0000","last_peered":"2024-09-21T12:20:57.672703+0000","last_clean":"2024-09-21T12:20:57.672703+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:20:57.672703+0000","last_undegraded":"2024-09-21T12:20:57.672703+0000","last_fullsized":"2024-09-21T12:20:57.672703+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.792664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":44,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-21T12:21:00.795654+0000","last_change":"2024-09-21T12:20:19.308586+0000","last_active":"2024-09-21T12:21:00.795654+0000","last_peered":"2024-09-21T12:21:00.795654+0000","last_clean":"2024-09-21T12:21:00.795654+0000","last_became_active":"2024-09-21T12:20:19.308247+0000","last_became_peered":"2024-09-21T12:20:19.308247+0000","last_unstale":"2024-09-21T12:21:00.795654+0000","last_undegraded":"2024-09-21T12:21:00.795654+0000","last_fullsized":"2024-09-21T12:21:00.795654+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:42.632946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":41,"num_read_kb":36,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928233997,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32628,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712524,"statfs":{"total":95995035648,"available":95961624576,"internally_reserved":0,"allocated":1470464,"data_stored":975323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462933,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32608,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712544,"statfs":{"total":95995035648,"available":95961645056,"internally_reserved":0,"allocated":1454080,"data_stored":974674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724568,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":884736,"data_stored":393342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953504,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32056,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713096,"statfs":{"total":95995035648,"available":95962210304,"internally_reserved":0,"allocated":884736,"data_stored":384475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":901120,"data_stored":386190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411372,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32020,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713132,"statfs":{"total":95995035648,"available":95962247168,"internally_reserved":0,"allocated":851968,"data_stored":384018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":937984,"data_stored":393339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28488,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716664,"statfs":{"total":95995035648,"available":95965863936,"internally_reserved":0,"allocated":1495040,"data_stored":983709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:21:03.611 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T12:21:03.611 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T12:21:03.611 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-21T12:21:03.612 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph health --format=json 2024-09-21T12:21:03.859 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:21:03.980 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:03 smithi114 ceph-mon[40867]: from='client.24602 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:21:03.980 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:03 smithi114 ceph-mon[30324]: from='client.24602 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:21:04.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:03 smithi183 ceph-mon[33277]: from='client.24602 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:21:04.660 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:21:04.660 INFO:teuthology.orchestra.run.smithi114.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T12:21:05.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:04 smithi114 ceph-mon[40867]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 754 B/s rd, 0 op/s 2024-09-21T12:21:05.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:04 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3619216841' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T12:21:05.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:04 smithi114 ceph-mon[30324]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 754 B/s rd, 0 op/s 2024-09-21T12:21:05.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:04 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3619216841' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T12:21:05.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:04 smithi183 ceph-mon[33277]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 754 B/s rd, 0 op/s 2024-09-21T12:21:05.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:04 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3619216841' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T12:21:06.033 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-21T12:21:06.033 INFO:tasks.cephadm:Setup complete, yielding 2024-09-21T12:21:06.033 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-21T12:21:06.044 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-21T12:21:06.044 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-guits-main-2024-09-21-0813'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'log-only-match': ['CEPHADM_'], 'sha1': '5c1cdc3c470dae78e91a0ed2bcf13509a046d301'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5c1cdc3c470dae78e91a0ed2bcf13509a046d301'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-guits-main-2024-09-21-0813', 'sha1': '5c1cdc3c470dae78e91a0ed2bcf13509a046d301'}} 2024-09-21T12:21:06.044 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-21T12:21:06.044 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-21T12:21:06.510 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:21:06.898 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:06 smithi114 ceph-mon[30324]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T12:21:06.898 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:06 smithi114 ceph-mon[40867]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T12:21:06.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:06 smithi183 ceph-mon[33277]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T12:21:06.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:21:06 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:21:07.524 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:21:07 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:21:07] ENGINE Bus STOPPING 2024-09-21T12:21:07.805 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:21:07 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:21:07] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T12:21:07.805 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:21:07 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:21:07] ENGINE Bus STOPPED 2024-09-21T12:21:07.805 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:21:07 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:21:07] ENGINE Bus STARTING 2024-09-21T12:21:07.805 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:21:07 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:21:07] ENGINE Serving on http://:::9283 2024-09-21T12:21:07.805 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:21:07 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:21:07] ENGINE Bus STARTED 2024-09-21T12:21:08.089 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:07 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:21:08.089 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:07 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3720371371' entity='client.admin' 2024-09-21T12:21:08.089 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:07 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T12:21:08.090 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:07 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:21:08.090 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:07 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3720371371' entity='client.admin' 2024-09-21T12:21:08.090 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:07 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T12:21:08.117 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-21T12:21:08.117 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T12:21:08.117 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:21:08.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:07 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:21:08.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:07 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3720371371' entity='client.admin' 2024-09-21T12:21:08.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:07 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T12:21:09.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:08 smithi183 ceph-mon[33277]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:21:09.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:08 smithi183 ceph-mon[33277]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:21:09.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:08 smithi114 ceph-mon[40867]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:21:09.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:08 smithi114 ceph-mon[40867]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:21:09.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:08 smithi114 ceph-mon[30324]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:21:09.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:08 smithi114 ceph-mon[30324]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:21:10.374 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:21:11.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:10 smithi114 ceph-mon[30324]: pgmap v20: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T12:21:11.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:10 smithi114 ceph-mon[40867]: pgmap v20: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T12:21:11.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:10 smithi183 ceph-mon[33277]: pgmap v20: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T12:21:11.722 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:21:11.724 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:21:12.812 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:21:12 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:21:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:21:12.977 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-21T12:21:11.550509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":899,"num_read_kb":756,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":245868,"kb_used_data":8828,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749715348,"statfs":{"total":767960285184,"available":767708516352,"internally_reserved":0,"allocated":9039872,"data_stored":4974393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002549"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761030+0000","last_change":"2024-09-21T12:20:15.309744+0000","last_active":"2024-09-21T12:21:01.761030+0000","last_peered":"2024-09-21T12:21:01.761030+0000","last_clean":"2024-09-21T12:21:01.761030+0000","last_became_active":"2024-09-21T12:20:15.309431+0000","last_became_peered":"2024-09-21T12:20:15.309431+0000","last_unstale":"2024-09-21T12:21:01.761030+0000","last_undegraded":"2024-09-21T12:21:01.761030+0000","last_fullsized":"2024-09-21T12:21:01.761030+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:37.321280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798853+0000","last_change":"2024-09-21T12:20:17.304357+0000","last_active":"2024-09-21T12:21:01.798853+0000","last_peered":"2024-09-21T12:21:01.798853+0000","last_clean":"2024-09-21T12:21:01.798853+0000","last_became_active":"2024-09-21T12:20:17.303856+0000","last_became_peered":"2024-09-21T12:20:17.303856+0000","last_unstale":"2024-09-21T12:21:01.798853+0000","last_undegraded":"2024-09-21T12:21:01.798853+0000","last_fullsized":"2024-09-21T12:21:01.798853+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:40.716880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760469+0000","last_change":"2024-09-21T12:20:19.315239+0000","last_active":"2024-09-21T12:21:01.760469+0000","last_peered":"2024-09-21T12:21:01.760469+0000","last_clean":"2024-09-21T12:21:01.760469+0000","last_became_active":"2024-09-21T12:20:19.314908+0000","last_became_peered":"2024-09-21T12:20:19.314908+0000","last_unstale":"2024-09-21T12:21:01.760469+0000","last_undegraded":"2024-09-21T12:21:01.760469+0000","last_fullsized":"2024-09-21T12:21:01.760469+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:33.386500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797572+0000","last_change":"2024-09-21T12:20:21.308871+0000","last_active":"2024-09-21T12:21:01.797572+0000","last_peered":"2024-09-21T12:21:01.797572+0000","last_clean":"2024-09-21T12:21:01.797572+0000","last_became_active":"2024-09-21T12:20:21.308577+0000","last_became_peered":"2024-09-21T12:20:21.308577+0000","last_unstale":"2024-09-21T12:21:01.797572+0000","last_undegraded":"2024-09-21T12:21:01.797572+0000","last_fullsized":"2024-09-21T12:21:01.797572+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:43.912920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798215+0000","last_change":"2024-09-21T12:20:21.595607+0000","last_active":"2024-09-21T12:21:01.798215+0000","last_peered":"2024-09-21T12:21:01.798215+0000","last_clean":"2024-09-21T12:21:01.798215+0000","last_became_active":"2024-09-21T12:20:21.595301+0000","last_became_peered":"2024-09-21T12:20:21.595301+0000","last_unstale":"2024-09-21T12:21:01.798215+0000","last_undegraded":"2024-09-21T12:21:01.798215+0000","last_fullsized":"2024-09-21T12:21:01.798215+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:27.801303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":52,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799325+0000","last_change":"2024-09-21T12:20:15.308941+0000","last_active":"2024-09-21T12:21:01.799325+0000","last_peered":"2024-09-21T12:21:01.799325+0000","last_clean":"2024-09-21T12:21:01.799325+0000","last_became_active":"2024-09-21T12:20:15.308655+0000","last_became_peered":"2024-09-21T12:20:15.308655+0000","last_unstale":"2024-09-21T12:21:01.799325+0000","last_undegraded":"2024-09-21T12:21:01.799325+0000","last_fullsized":"2024-09-21T12:21:01.799325+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:31.202541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":73,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798236+0000","last_change":"2024-09-21T12:20:17.305499+0000","last_active":"2024-09-21T12:21:01.798236+0000","last_peered":"2024-09-21T12:21:01.798236+0000","last_clean":"2024-09-21T12:21:01.798236+0000","last_became_active":"2024-09-21T12:20:17.305220+0000","last_became_peered":"2024-09-21T12:20:17.305220+0000","last_unstale":"2024-09-21T12:21:01.798236+0000","last_undegraded":"2024-09-21T12:21:01.798236+0000","last_fullsized":"2024-09-21T12:21:01.798236+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.370654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798939+0000","last_change":"2024-09-21T12:20:19.311730+0000","last_active":"2024-09-21T12:21:01.798939+0000","last_peered":"2024-09-21T12:21:01.798939+0000","last_clean":"2024-09-21T12:21:01.798939+0000","last_became_active":"2024-09-21T12:20:19.311403+0000","last_became_peered":"2024-09-21T12:20:19.311403+0000","last_unstale":"2024-09-21T12:21:01.798939+0000","last_undegraded":"2024-09-21T12:21:01.798939+0000","last_fullsized":"2024-09-21T12:21:01.798939+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:40.685703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799371+0000","last_change":"2024-09-21T12:20:15.302150+0000","last_active":"2024-09-21T12:21:01.799371+0000","last_peered":"2024-09-21T12:21:01.799371+0000","last_clean":"2024-09-21T12:21:01.799371+0000","last_became_active":"2024-09-21T12:20:15.301847+0000","last_became_peered":"2024-09-21T12:20:15.301847+0000","last_unstale":"2024-09-21T12:21:01.799371+0000","last_undegraded":"2024-09-21T12:21:01.799371+0000","last_fullsized":"2024-09-21T12:21:01.799371+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:26.753083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799636+0000","last_change":"2024-09-21T12:20:17.307650+0000","last_active":"2024-09-21T12:21:01.799636+0000","last_peered":"2024-09-21T12:21:01.799636+0000","last_clean":"2024-09-21T12:21:01.799636+0000","last_became_active":"2024-09-21T12:20:17.307280+0000","last_became_peered":"2024-09-21T12:20:17.307280+0000","last_unstale":"2024-09-21T12:21:01.799636+0000","last_undegraded":"2024-09-21T12:21:01.799636+0000","last_fullsized":"2024-09-21T12:21:01.799636+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:34.375479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798060+0000","last_change":"2024-09-21T12:20:19.308520+0000","last_active":"2024-09-21T12:21:01.798060+0000","last_peered":"2024-09-21T12:21:01.798060+0000","last_clean":"2024-09-21T12:21:01.798060+0000","last_became_active":"2024-09-21T12:20:19.308184+0000","last_became_peered":"2024-09-21T12:20:19.308184+0000","last_unstale":"2024-09-21T12:21:01.798060+0000","last_undegraded":"2024-09-21T12:21:01.798060+0000","last_fullsized":"2024-09-21T12:21:01.798060+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:39.981449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760366+0000","last_change":"2024-09-21T12:20:21.309871+0000","last_active":"2024-09-21T12:21:01.760366+0000","last_peered":"2024-09-21T12:21:01.760366+0000","last_clean":"2024-09-21T12:21:01.760366+0000","last_became_active":"2024-09-21T12:20:21.309190+0000","last_became_peered":"2024-09-21T12:20:21.309190+0000","last_unstale":"2024-09-21T12:21:01.760366+0000","last_undegraded":"2024-09-21T12:21:01.760366+0000","last_fullsized":"2024-09-21T12:21:01.760366+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.813677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798210+0000","last_change":"2024-09-21T12:20:15.300728+0000","last_active":"2024-09-21T12:21:01.798210+0000","last_peered":"2024-09-21T12:21:01.798210+0000","last_clean":"2024-09-21T12:21:01.798210+0000","last_became_active":"2024-09-21T12:20:15.300441+0000","last_became_peered":"2024-09-21T12:20:15.300441+0000","last_unstale":"2024-09-21T12:21:01.798210+0000","last_undegraded":"2024-09-21T12:21:01.798210+0000","last_fullsized":"2024-09-21T12:21:01.798210+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:34.282857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":59,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799359+0000","last_change":"2024-09-21T12:20:17.305973+0000","last_active":"2024-09-21T12:21:01.799359+0000","last_peered":"2024-09-21T12:21:01.799359+0000","last_clean":"2024-09-21T12:21:01.799359+0000","last_became_active":"2024-09-21T12:20:17.305663+0000","last_became_peered":"2024-09-21T12:20:17.305663+0000","last_unstale":"2024-09-21T12:21:01.799359+0000","last_undegraded":"2024-09-21T12:21:01.799359+0000","last_fullsized":"2024-09-21T12:21:01.799359+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.420641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798568+0000","last_change":"2024-09-21T12:20:19.312502+0000","last_active":"2024-09-21T12:21:01.798568+0000","last_peered":"2024-09-21T12:21:01.798568+0000","last_clean":"2024-09-21T12:21:01.798568+0000","last_became_active":"2024-09-21T12:20:19.312242+0000","last_became_peered":"2024-09-21T12:20:19.312242+0000","last_unstale":"2024-09-21T12:21:01.798568+0000","last_undegraded":"2024-09-21T12:21:01.798568+0000","last_fullsized":"2024-09-21T12:21:01.798568+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:32.544179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797726+0000","last_change":"2024-09-21T12:20:21.311536+0000","last_active":"2024-09-21T12:21:01.797726+0000","last_peered":"2024-09-21T12:21:01.797726+0000","last_clean":"2024-09-21T12:21:01.797726+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:21:01.797726+0000","last_undegraded":"2024-09-21T12:21:01.797726+0000","last_fullsized":"2024-09-21T12:21:01.797726+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:41.239506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797876+0000","last_change":"2024-09-21T12:20:21.592944+0000","last_active":"2024-09-21T12:21:01.797876+0000","last_peered":"2024-09-21T12:21:01.797876+0000","last_clean":"2024-09-21T12:21:01.797876+0000","last_became_active":"2024-09-21T12:20:21.592631+0000","last_became_peered":"2024-09-21T12:20:21.592631+0000","last_unstale":"2024-09-21T12:21:01.797876+0000","last_undegraded":"2024-09-21T12:21:01.797876+0000","last_fullsized":"2024-09-21T12:21:01.797876+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:39.384088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761697+0000","last_change":"2024-09-21T12:20:15.302079+0000","last_active":"2024-09-21T12:21:01.761697+0000","last_peered":"2024-09-21T12:21:01.761697+0000","last_clean":"2024-09-21T12:21:01.761697+0000","last_became_active":"2024-09-21T12:20:15.301793+0000","last_became_peered":"2024-09-21T12:20:15.301793+0000","last_unstale":"2024-09-21T12:21:01.761697+0000","last_undegraded":"2024-09-21T12:21:01.761697+0000","last_fullsized":"2024-09-21T12:21:01.761697+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:19.991370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":75,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797700+0000","last_change":"2024-09-21T12:20:17.308186+0000","last_active":"2024-09-21T12:21:01.797700+0000","last_peered":"2024-09-21T12:21:01.797700+0000","last_clean":"2024-09-21T12:21:01.797700+0000","last_became_active":"2024-09-21T12:20:17.307828+0000","last_became_peered":"2024-09-21T12:20:17.307828+0000","last_unstale":"2024-09-21T12:21:01.797700+0000","last_undegraded":"2024-09-21T12:21:01.797700+0000","last_fullsized":"2024-09-21T12:21:01.797700+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:41.845334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761354+0000","last_change":"2024-09-21T12:20:19.316360+0000","last_active":"2024-09-21T12:21:01.761354+0000","last_peered":"2024-09-21T12:21:01.761354+0000","last_clean":"2024-09-21T12:21:01.761354+0000","last_became_active":"2024-09-21T12:20:19.315977+0000","last_became_peered":"2024-09-21T12:20:19.315977+0000","last_unstale":"2024-09-21T12:21:01.761354+0000","last_undegraded":"2024-09-21T12:21:01.761354+0000","last_fullsized":"2024-09-21T12:21:01.761354+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:37.515718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797418+0000","last_change":"2024-09-21T12:20:21.593129+0000","last_active":"2024-09-21T12:21:01.797418+0000","last_peered":"2024-09-21T12:21:01.797418+0000","last_clean":"2024-09-21T12:21:01.797418+0000","last_became_active":"2024-09-21T12:20:21.592796+0000","last_became_peered":"2024-09-21T12:20:21.592796+0000","last_unstale":"2024-09-21T12:21:01.797418+0000","last_undegraded":"2024-09-21T12:21:01.797418+0000","last_fullsized":"2024-09-21T12:21:01.797418+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:47.880164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799779+0000","last_change":"2024-09-21T12:20:15.309390+0000","last_active":"2024-09-21T12:21:01.799779+0000","last_peered":"2024-09-21T12:21:01.799779+0000","last_clean":"2024-09-21T12:21:01.799779+0000","last_became_active":"2024-09-21T12:20:15.309023+0000","last_became_peered":"2024-09-21T12:20:15.309023+0000","last_unstale":"2024-09-21T12:21:01.799779+0000","last_undegraded":"2024-09-21T12:21:01.799779+0000","last_fullsized":"2024-09-21T12:21:01.799779+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:25.611326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":71,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798284+0000","last_change":"2024-09-21T12:20:17.310727+0000","last_active":"2024-09-21T12:21:01.798284+0000","last_peered":"2024-09-21T12:21:01.798284+0000","last_clean":"2024-09-21T12:21:01.798284+0000","last_became_active":"2024-09-21T12:20:17.310369+0000","last_became_peered":"2024-09-21T12:20:17.310369+0000","last_unstale":"2024-09-21T12:21:01.798284+0000","last_undegraded":"2024-09-21T12:21:01.798284+0000","last_fullsized":"2024-09-21T12:21:01.798284+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:22.563871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.800078+0000","last_change":"2024-09-21T12:20:19.310859+0000","last_active":"2024-09-21T12:21:01.800078+0000","last_peered":"2024-09-21T12:21:01.800078+0000","last_clean":"2024-09-21T12:21:01.800078+0000","last_became_active":"2024-09-21T12:20:19.310575+0000","last_became_peered":"2024-09-21T12:20:19.310575+0000","last_unstale":"2024-09-21T12:21:01.800078+0000","last_undegraded":"2024-09-21T12:21:01.800078+0000","last_fullsized":"2024-09-21T12:21:01.800078+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.082897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797401+0000","last_change":"2024-09-21T12:20:21.313476+0000","last_active":"2024-09-21T12:21:01.797401+0000","last_peered":"2024-09-21T12:21:01.797401+0000","last_clean":"2024-09-21T12:21:01.797401+0000","last_became_active":"2024-09-21T12:20:21.312976+0000","last_became_peered":"2024-09-21T12:20:21.312976+0000","last_unstale":"2024-09-21T12:21:01.797401+0000","last_undegraded":"2024-09-21T12:21:01.797401+0000","last_fullsized":"2024-09-21T12:21:01.797401+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.367953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":55,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760594+0000","last_change":"2024-09-21T12:20:15.299812+0000","last_active":"2024-09-21T12:21:01.760594+0000","last_peered":"2024-09-21T12:21:01.760594+0000","last_clean":"2024-09-21T12:21:01.760594+0000","last_became_active":"2024-09-21T12:20:15.299459+0000","last_became_peered":"2024-09-21T12:20:15.299459+0000","last_unstale":"2024-09-21T12:21:01.760594+0000","last_undegraded":"2024-09-21T12:21:01.760594+0000","last_fullsized":"2024-09-21T12:21:01.760594+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.458091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761351+0000","last_change":"2024-09-21T12:20:17.696423+0000","last_active":"2024-09-21T12:21:01.761351+0000","last_peered":"2024-09-21T12:21:01.761351+0000","last_clean":"2024-09-21T12:21:01.761351+0000","last_became_active":"2024-09-21T12:20:17.695977+0000","last_became_peered":"2024-09-21T12:20:17.695977+0000","last_unstale":"2024-09-21T12:21:01.761351+0000","last_undegraded":"2024-09-21T12:21:01.761351+0000","last_fullsized":"2024-09-21T12:21:01.761351+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:44.740918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":68,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761113+0000","last_change":"2024-09-21T12:20:19.313262+0000","last_active":"2024-09-21T12:21:01.761113+0000","last_peered":"2024-09-21T12:21:01.761113+0000","last_clean":"2024-09-21T12:21:01.761113+0000","last_became_active":"2024-09-21T12:20:19.312403+0000","last_became_peered":"2024-09-21T12:20:19.312403+0000","last_unstale":"2024-09-21T12:21:01.761113+0000","last_undegraded":"2024-09-21T12:21:01.761113+0000","last_fullsized":"2024-09-21T12:21:01.761113+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:43.181150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":75,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761866+0000","last_change":"2024-09-21T12:20:17.305947+0000","last_active":"2024-09-21T12:21:01.761866+0000","last_peered":"2024-09-21T12:21:01.761866+0000","last_clean":"2024-09-21T12:21:01.761866+0000","last_became_active":"2024-09-21T12:20:17.305666+0000","last_became_peered":"2024-09-21T12:20:17.305666+0000","last_unstale":"2024-09-21T12:21:01.761866+0000","last_undegraded":"2024-09-21T12:21:01.761866+0000","last_fullsized":"2024-09-21T12:21:01.761866+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.503390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":52,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798247+0000","last_change":"2024-09-21T12:20:15.296251+0000","last_active":"2024-09-21T12:21:01.798247+0000","last_peered":"2024-09-21T12:21:01.798247+0000","last_clean":"2024-09-21T12:21:01.798247+0000","last_became_active":"2024-09-21T12:20:15.295774+0000","last_became_peered":"2024-09-21T12:20:15.295774+0000","last_unstale":"2024-09-21T12:21:01.798247+0000","last_undegraded":"2024-09-21T12:21:01.798247+0000","last_fullsized":"2024-09-21T12:21:01.798247+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:27.341994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:05.357999+0000","last_change":"2024-09-21T12:20:19.309874+0000","last_active":"2024-09-21T12:21:05.357999+0000","last_peered":"2024-09-21T12:21:05.357999+0000","last_clean":"2024-09-21T12:21:05.357999+0000","last_became_active":"2024-09-21T12:20:19.309606+0000","last_became_peered":"2024-09-21T12:20:19.309606+0000","last_unstale":"2024-09-21T12:21:05.357999+0000","last_undegraded":"2024-09-21T12:21:05.357999+0000","last_fullsized":"2024-09-21T12:21:05.357999+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:37.621763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":43,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798874+0000","last_change":"2024-09-21T12:20:21.310295+0000","last_active":"2024-09-21T12:21:01.798874+0000","last_peered":"2024-09-21T12:21:01.798874+0000","last_clean":"2024-09-21T12:21:01.798874+0000","last_became_active":"2024-09-21T12:20:21.310022+0000","last_became_peered":"2024-09-21T12:20:21.310022+0000","last_unstale":"2024-09-21T12:21:01.798874+0000","last_undegraded":"2024-09-21T12:21:01.798874+0000","last_fullsized":"2024-09-21T12:21:01.798874+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:37.912528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":80,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798048+0000","last_change":"2024-09-21T12:20:17.693550+0000","last_active":"2024-09-21T12:21:01.798048+0000","last_peered":"2024-09-21T12:21:01.798048+0000","last_clean":"2024-09-21T12:21:01.798048+0000","last_became_active":"2024-09-21T12:20:17.693179+0000","last_became_peered":"2024-09-21T12:20:17.693179+0000","last_unstale":"2024-09-21T12:21:01.798048+0000","last_undegraded":"2024-09-21T12:21:01.798048+0000","last_fullsized":"2024-09-21T12:21:01.798048+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:45.358986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":52,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799611+0000","last_change":"2024-09-21T12:20:15.297621+0000","last_active":"2024-09-21T12:21:01.799611+0000","last_peered":"2024-09-21T12:21:01.799611+0000","last_clean":"2024-09-21T12:21:01.799611+0000","last_became_active":"2024-09-21T12:20:15.297340+0000","last_became_peered":"2024-09-21T12:20:15.297340+0000","last_unstale":"2024-09-21T12:21:01.799611+0000","last_undegraded":"2024-09-21T12:21:01.799611+0000","last_fullsized":"2024-09-21T12:21:01.799611+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:17.208372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798975+0000","last_change":"2024-09-21T12:20:19.315065+0000","last_active":"2024-09-21T12:21:01.798975+0000","last_peered":"2024-09-21T12:21:01.798975+0000","last_clean":"2024-09-21T12:21:01.798975+0000","last_became_active":"2024-09-21T12:20:19.314775+0000","last_became_peered":"2024-09-21T12:20:19.314775+0000","last_unstale":"2024-09-21T12:21:01.798975+0000","last_undegraded":"2024-09-21T12:21:01.798975+0000","last_fullsized":"2024-09-21T12:21:01.798975+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:26.384029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799583+0000","last_change":"2024-09-21T12:20:21.314735+0000","last_active":"2024-09-21T12:21:01.799583+0000","last_peered":"2024-09-21T12:21:01.799583+0000","last_clean":"2024-09-21T12:21:01.799583+0000","last_became_active":"2024-09-21T12:20:21.314433+0000","last_became_peered":"2024-09-21T12:20:21.314433+0000","last_unstale":"2024-09-21T12:21:01.799583+0000","last_undegraded":"2024-09-21T12:21:01.799583+0000","last_fullsized":"2024-09-21T12:21:01.799583+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:39.652862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":70,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799651+0000","last_change":"2024-09-21T12:20:17.301460+0000","last_active":"2024-09-21T12:21:01.799651+0000","last_peered":"2024-09-21T12:21:01.799651+0000","last_clean":"2024-09-21T12:21:01.799651+0000","last_became_active":"2024-09-21T12:20:17.301090+0000","last_became_peered":"2024-09-21T12:20:17.301090+0000","last_unstale":"2024-09-21T12:21:01.799651+0000","last_undegraded":"2024-09-21T12:21:01.799651+0000","last_fullsized":"2024-09-21T12:21:01.799651+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:34.835982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":55,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761630+0000","last_change":"2024-09-21T12:20:15.302238+0000","last_active":"2024-09-21T12:21:01.761630+0000","last_peered":"2024-09-21T12:21:01.761630+0000","last_clean":"2024-09-21T12:21:01.761630+0000","last_became_active":"2024-09-21T12:20:15.301904+0000","last_became_peered":"2024-09-21T12:20:15.301904+0000","last_unstale":"2024-09-21T12:21:01.761630+0000","last_undegraded":"2024-09-21T12:21:01.761630+0000","last_fullsized":"2024-09-21T12:21:01.761630+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:33.679116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":64,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:05.357777+0000","last_change":"2024-09-21T12:20:19.311362+0000","last_active":"2024-09-21T12:21:05.357777+0000","last_peered":"2024-09-21T12:21:05.357777+0000","last_clean":"2024-09-21T12:21:05.357777+0000","last_became_active":"2024-09-21T12:20:19.311056+0000","last_became_peered":"2024-09-21T12:20:19.311056+0000","last_unstale":"2024-09-21T12:21:05.357777+0000","last_undegraded":"2024-09-21T12:21:05.357777+0000","last_fullsized":"2024-09-21T12:21:05.357777+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:22.628895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797588+0000","last_change":"2024-09-21T12:20:21.593620+0000","last_active":"2024-09-21T12:21:01.797588+0000","last_peered":"2024-09-21T12:21:01.797588+0000","last_clean":"2024-09-21T12:21:01.797588+0000","last_became_active":"2024-09-21T12:20:21.593367+0000","last_became_peered":"2024-09-21T12:20:21.593367+0000","last_unstale":"2024-09-21T12:21:01.797588+0000","last_undegraded":"2024-09-21T12:21:01.797588+0000","last_fullsized":"2024-09-21T12:21:01.797588+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:20.553670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760808+0000","last_change":"2024-09-21T12:20:17.303245+0000","last_active":"2024-09-21T12:21:01.760808+0000","last_peered":"2024-09-21T12:21:01.760808+0000","last_clean":"2024-09-21T12:21:01.760808+0000","last_became_active":"2024-09-21T12:20:17.302871+0000","last_became_peered":"2024-09-21T12:20:17.302871+0000","last_unstale":"2024-09-21T12:21:01.760808+0000","last_undegraded":"2024-09-21T12:21:01.760808+0000","last_fullsized":"2024-09-21T12:21:01.760808+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.942911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799448+0000","last_change":"2024-09-21T12:20:15.308877+0000","last_active":"2024-09-21T12:21:01.799448+0000","last_peered":"2024-09-21T12:21:01.799448+0000","last_clean":"2024-09-21T12:21:01.799448+0000","last_became_active":"2024-09-21T12:20:15.308518+0000","last_became_peered":"2024-09-21T12:20:15.308518+0000","last_unstale":"2024-09-21T12:21:01.799448+0000","last_undegraded":"2024-09-21T12:21:01.799448+0000","last_fullsized":"2024-09-21T12:21:01.799448+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:23.801832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":69,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:05.357854+0000","last_change":"2024-09-21T12:20:19.307467+0000","last_active":"2024-09-21T12:21:05.357854+0000","last_peered":"2024-09-21T12:21:05.357854+0000","last_clean":"2024-09-21T12:21:05.357854+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:21:05.357854+0000","last_undegraded":"2024-09-21T12:21:05.357854+0000","last_fullsized":"2024-09-21T12:21:05.357854+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:21.225955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760501+0000","last_change":"2024-09-21T12:20:21.316127+0000","last_active":"2024-09-21T12:21:01.760501+0000","last_peered":"2024-09-21T12:21:01.760501+0000","last_clean":"2024-09-21T12:21:01.760501+0000","last_became_active":"2024-09-21T12:20:21.315801+0000","last_became_peered":"2024-09-21T12:20:21.315801+0000","last_unstale":"2024-09-21T12:21:01.760501+0000","last_undegraded":"2024-09-21T12:21:01.760501+0000","last_fullsized":"2024-09-21T12:21:01.760501+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:28.234335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":86,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761471+0000","last_change":"2024-09-21T12:20:17.693585+0000","last_active":"2024-09-21T12:21:01.761471+0000","last_peered":"2024-09-21T12:21:01.761471+0000","last_clean":"2024-09-21T12:21:01.761471+0000","last_became_active":"2024-09-21T12:20:17.693067+0000","last_became_peered":"2024-09-21T12:20:17.693067+0000","last_unstale":"2024-09-21T12:21:01.761471+0000","last_undegraded":"2024-09-21T12:21:01.761471+0000","last_fullsized":"2024-09-21T12:21:01.761471+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:37.693764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761357+0000","last_change":"2024-09-21T12:20:15.308741+0000","last_active":"2024-09-21T12:21:01.761357+0000","last_peered":"2024-09-21T12:21:01.761357+0000","last_clean":"2024-09-21T12:21:01.761357+0000","last_became_active":"2024-09-21T12:20:15.308441+0000","last_became_peered":"2024-09-21T12:20:15.308441+0000","last_unstale":"2024-09-21T12:21:01.761357+0000","last_undegraded":"2024-09-21T12:21:01.761357+0000","last_fullsized":"2024-09-21T12:21:01.761357+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:29.626513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760392+0000","last_change":"2024-09-21T12:20:19.309945+0000","last_active":"2024-09-21T12:21:01.760392+0000","last_peered":"2024-09-21T12:21:01.760392+0000","last_clean":"2024-09-21T12:21:01.760392+0000","last_became_active":"2024-09-21T12:20:19.309598+0000","last_became_peered":"2024-09-21T12:20:19.309598+0000","last_unstale":"2024-09-21T12:21:01.760392+0000","last_undegraded":"2024-09-21T12:21:01.760392+0000","last_fullsized":"2024-09-21T12:21:01.760392+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:31.055740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760388+0000","last_change":"2024-09-21T12:20:21.593173+0000","last_active":"2024-09-21T12:21:01.760388+0000","last_peered":"2024-09-21T12:21:01.760388+0000","last_clean":"2024-09-21T12:21:01.760388+0000","last_became_active":"2024-09-21T12:20:21.592828+0000","last_became_peered":"2024-09-21T12:20:21.592828+0000","last_unstale":"2024-09-21T12:21:01.760388+0000","last_undegraded":"2024-09-21T12:21:01.760388+0000","last_fullsized":"2024-09-21T12:21:01.760388+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.351266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":110,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:05.357795+0000","last_change":"2024-09-21T12:20:17.308049+0000","last_active":"2024-09-21T12:21:05.357795+0000","last_peered":"2024-09-21T12:21:05.357795+0000","last_clean":"2024-09-21T12:21:05.357795+0000","last_became_active":"2024-09-21T12:20:17.307769+0000","last_became_peered":"2024-09-21T12:20:17.307769+0000","last_unstale":"2024-09-21T12:21:05.357795+0000","last_undegraded":"2024-09-21T12:21:05.357795+0000","last_fullsized":"2024-09-21T12:21:05.357795+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:29.183963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.800293+0000","last_change":"2024-09-21T12:20:15.299429+0000","last_active":"2024-09-21T12:21:01.800293+0000","last_peered":"2024-09-21T12:21:01.800293+0000","last_clean":"2024-09-21T12:21:01.800293+0000","last_became_active":"2024-09-21T12:20:15.299143+0000","last_became_peered":"2024-09-21T12:20:15.299143+0000","last_unstale":"2024-09-21T12:21:01.800293+0000","last_undegraded":"2024-09-21T12:21:01.800293+0000","last_fullsized":"2024-09-21T12:21:01.800293+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:42.581678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":64,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798714+0000","last_change":"2024-09-21T12:20:17.298211+0000","last_active":"2024-09-21T12:21:01.798714+0000","last_peered":"2024-09-21T12:21:01.798714+0000","last_clean":"2024-09-21T12:21:01.798714+0000","last_became_active":"2024-09-21T12:20:14.287998+0000","last_became_peered":"2024-09-21T12:20:14.287998+0000","last_unstale":"2024-09-21T12:21:01.798714+0000","last_undegraded":"2024-09-21T12:21:01.798714+0000","last_fullsized":"2024-09-21T12:21:01.798714+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:20:13.272075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:19.171111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011011149999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761160+0000","last_change":"2024-09-21T12:20:19.313478+0000","last_active":"2024-09-21T12:21:01.761160+0000","last_peered":"2024-09-21T12:21:01.761160+0000","last_clean":"2024-09-21T12:21:01.761160+0000","last_became_active":"2024-09-21T12:20:19.312170+0000","last_became_peered":"2024-09-21T12:20:19.312170+0000","last_unstale":"2024-09-21T12:21:01.761160+0000","last_undegraded":"2024-09-21T12:21:01.761160+0000","last_fullsized":"2024-09-21T12:21:01.761160+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.170409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798740+0000","last_change":"2024-09-21T12:20:21.314452+0000","last_active":"2024-09-21T12:21:01.798740+0000","last_peered":"2024-09-21T12:21:01.798740+0000","last_clean":"2024-09-21T12:21:01.798740+0000","last_became_active":"2024-09-21T12:20:21.314126+0000","last_became_peered":"2024-09-21T12:20:21.314126+0000","last_unstale":"2024-09-21T12:21:01.798740+0000","last_undegraded":"2024-09-21T12:21:01.798740+0000","last_fullsized":"2024-09-21T12:21:01.798740+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:44.311699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":77,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761871+0000","last_change":"2024-09-21T12:20:17.297678+0000","last_active":"2024-09-21T12:21:01.761871+0000","last_peered":"2024-09-21T12:21:01.761871+0000","last_clean":"2024-09-21T12:21:01.761871+0000","last_became_active":"2024-09-21T12:20:17.297374+0000","last_became_peered":"2024-09-21T12:20:17.297374+0000","last_unstale":"2024-09-21T12:21:01.761871+0000","last_undegraded":"2024-09-21T12:21:01.761871+0000","last_fullsized":"2024-09-21T12:21:01.761871+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:29.314219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760509+0000","last_change":"2024-09-21T12:20:15.308648+0000","last_active":"2024-09-21T12:21:01.760509+0000","last_peered":"2024-09-21T12:21:01.760509+0000","last_clean":"2024-09-21T12:21:01.760509+0000","last_became_active":"2024-09-21T12:20:15.308356+0000","last_became_peered":"2024-09-21T12:20:15.308356+0000","last_unstale":"2024-09-21T12:21:01.760509+0000","last_undegraded":"2024-09-21T12:21:01.760509+0000","last_fullsized":"2024-09-21T12:21:01.760509+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:25.741492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":63,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797588+0000","last_change":"2024-09-21T12:20:17.297432+0000","last_active":"2024-09-21T12:21:01.797588+0000","last_peered":"2024-09-21T12:21:01.797588+0000","last_clean":"2024-09-21T12:21:01.797588+0000","last_became_active":"2024-09-21T12:20:14.288358+0000","last_became_peered":"2024-09-21T12:20:14.288358+0000","last_unstale":"2024-09-21T12:21:01.797588+0000","last_undegraded":"2024-09-21T12:21:01.797588+0000","last_fullsized":"2024-09-21T12:21:01.797588+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:20:13.272075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:21.220031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019042449999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.796989+0000","last_change":"2024-09-21T12:20:19.310164+0000","last_active":"2024-09-21T12:21:01.796989+0000","last_peered":"2024-09-21T12:21:01.796989+0000","last_clean":"2024-09-21T12:21:01.796989+0000","last_became_active":"2024-09-21T12:20:19.309724+0000","last_became_peered":"2024-09-21T12:20:19.309724+0000","last_unstale":"2024-09-21T12:21:01.796989+0000","last_undegraded":"2024-09-21T12:21:01.796989+0000","last_fullsized":"2024-09-21T12:21:01.796989+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:43.623687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.800070+0000","last_change":"2024-09-21T12:20:21.593267+0000","last_active":"2024-09-21T12:21:01.800070+0000","last_peered":"2024-09-21T12:21:01.800070+0000","last_clean":"2024-09-21T12:21:01.800070+0000","last_became_active":"2024-09-21T12:20:21.592982+0000","last_became_peered":"2024-09-21T12:20:21.592982+0000","last_unstale":"2024-09-21T12:21:01.800070+0000","last_undegraded":"2024-09-21T12:21:01.800070+0000","last_fullsized":"2024-09-21T12:21:01.800070+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:43.824339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":68,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761450+0000","last_change":"2024-09-21T12:20:17.299646+0000","last_active":"2024-09-21T12:21:01.761450+0000","last_peered":"2024-09-21T12:21:01.761450+0000","last_clean":"2024-09-21T12:21:01.761450+0000","last_became_active":"2024-09-21T12:20:17.299325+0000","last_became_peered":"2024-09-21T12:20:17.299325+0000","last_unstale":"2024-09-21T12:21:01.761450+0000","last_undegraded":"2024-09-21T12:21:01.761450+0000","last_fullsized":"2024-09-21T12:21:01.761450+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:22.183786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760648+0000","last_change":"2024-09-21T12:20:15.302177+0000","last_active":"2024-09-21T12:21:01.760648+0000","last_peered":"2024-09-21T12:21:01.760648+0000","last_clean":"2024-09-21T12:21:01.760648+0000","last_became_active":"2024-09-21T12:20:15.301859+0000","last_became_peered":"2024-09-21T12:20:15.301859+0000","last_unstale":"2024-09-21T12:21:01.760648+0000","last_undegraded":"2024-09-21T12:21:01.760648+0000","last_fullsized":"2024-09-21T12:21:01.760648+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.505232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":115,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:10.100059+0000","last_change":"2024-09-21T12:20:17.693865+0000","last_active":"2024-09-21T12:21:10.100059+0000","last_peered":"2024-09-21T12:21:10.100059+0000","last_clean":"2024-09-21T12:21:10.100059+0000","last_became_active":"2024-09-21T12:20:14.287543+0000","last_became_peered":"2024-09-21T12:20:14.287543+0000","last_unstale":"2024-09-21T12:21:10.100059+0000","last_undegraded":"2024-09-21T12:21:10.100059+0000","last_fullsized":"2024-09-21T12:21:10.100059+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:20:13.272075+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:21.111464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":51,"num_read_kb":46,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799083+0000","last_change":"2024-09-21T12:20:19.314574+0000","last_active":"2024-09-21T12:21:01.799083+0000","last_peered":"2024-09-21T12:21:01.799083+0000","last_clean":"2024-09-21T12:21:01.799083+0000","last_became_active":"2024-09-21T12:20:19.314217+0000","last_became_peered":"2024-09-21T12:20:19.314217+0000","last_unstale":"2024-09-21T12:21:01.799083+0000","last_undegraded":"2024-09-21T12:21:01.799083+0000","last_fullsized":"2024-09-21T12:21:01.799083+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:23.172823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761038+0000","last_change":"2024-09-21T12:20:21.311501+0000","last_active":"2024-09-21T12:21:01.761038+0000","last_peered":"2024-09-21T12:21:01.761038+0000","last_clean":"2024-09-21T12:21:01.761038+0000","last_became_active":"2024-09-21T12:20:21.311188+0000","last_became_peered":"2024-09-21T12:20:21.311188+0000","last_unstale":"2024-09-21T12:21:01.761038+0000","last_undegraded":"2024-09-21T12:21:01.761038+0000","last_fullsized":"2024-09-21T12:21:01.761038+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:27.755252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"66'16","reported_seq":83,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761025+0000","last_change":"2024-09-21T12:20:17.696268+0000","last_active":"2024-09-21T12:21:01.761025+0000","last_peered":"2024-09-21T12:21:01.761025+0000","last_clean":"2024-09-21T12:21:01.761025+0000","last_became_active":"2024-09-21T12:20:17.695940+0000","last_became_peered":"2024-09-21T12:20:17.695940+0000","last_unstale":"2024-09-21T12:21:01.761025+0000","last_undegraded":"2024-09-21T12:21:01.761025+0000","last_fullsized":"2024-09-21T12:21:01.761025+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:39.599877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":58,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799244+0000","last_change":"2024-09-21T12:20:15.299151+0000","last_active":"2024-09-21T12:21:01.799244+0000","last_peered":"2024-09-21T12:21:01.799244+0000","last_clean":"2024-09-21T12:21:01.799244+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:21:01.799244+0000","last_undegraded":"2024-09-21T12:21:01.799244+0000","last_fullsized":"2024-09-21T12:21:01.799244+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:16.205349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":98,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799314+0000","last_change":"2024-09-21T12:20:24.850239+0000","last_active":"2024-09-21T12:21:01.799314+0000","last_peered":"2024-09-21T12:21:01.799314+0000","last_clean":"2024-09-21T12:21:01.799314+0000","last_became_active":"2024-09-21T12:20:06.349670+0000","last_became_peered":"2024-09-21T12:20:06.349670+0000","last_unstale":"2024-09-21T12:21:01.799314+0000","last_undegraded":"2024-09-21T12:21:01.799314+0000","last_fullsized":"2024-09-21T12:21:01.799314+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T12:20:24.850157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:20:24.850157+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:53.046478+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799933+0000","last_change":"2024-09-21T12:20:19.310280+0000","last_active":"2024-09-21T12:21:01.799933+0000","last_peered":"2024-09-21T12:21:01.799933+0000","last_clean":"2024-09-21T12:21:01.799933+0000","last_became_active":"2024-09-21T12:20:19.309859+0000","last_became_peered":"2024-09-21T12:20:19.309859+0000","last_unstale":"2024-09-21T12:21:01.799933+0000","last_undegraded":"2024-09-21T12:21:01.799933+0000","last_fullsized":"2024-09-21T12:21:01.799933+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:43.850206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797865+0000","last_change":"2024-09-21T12:20:21.311690+0000","last_active":"2024-09-21T12:21:01.797865+0000","last_peered":"2024-09-21T12:21:01.797865+0000","last_clean":"2024-09-21T12:21:01.797865+0000","last_became_active":"2024-09-21T12:20:21.311194+0000","last_became_peered":"2024-09-21T12:20:21.311194+0000","last_unstale":"2024-09-21T12:21:01.797865+0000","last_undegraded":"2024-09-21T12:21:01.797865+0000","last_fullsized":"2024-09-21T12:21:01.797865+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:25.752002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":68,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.800772+0000","last_change":"2024-09-21T12:20:17.310299+0000","last_active":"2024-09-21T12:21:01.800772+0000","last_peered":"2024-09-21T12:21:01.800772+0000","last_clean":"2024-09-21T12:21:01.800772+0000","last_became_active":"2024-09-21T12:20:17.309930+0000","last_became_peered":"2024-09-21T12:20:17.309930+0000","last_unstale":"2024-09-21T12:21:01.800772+0000","last_undegraded":"2024-09-21T12:21:01.800772+0000","last_fullsized":"2024-09-21T12:21:01.800772+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:17.883258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.800123+0000","last_change":"2024-09-21T12:20:15.310346+0000","last_active":"2024-09-21T12:21:01.800123+0000","last_peered":"2024-09-21T12:21:01.800123+0000","last_clean":"2024-09-21T12:21:01.800123+0000","last_became_active":"2024-09-21T12:20:15.310048+0000","last_became_peered":"2024-09-21T12:20:15.310048+0000","last_unstale":"2024-09-21T12:21:01.800123+0000","last_undegraded":"2024-09-21T12:21:01.800123+0000","last_fullsized":"2024-09-21T12:21:01.800123+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:31.701321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797742+0000","last_change":"2024-09-21T12:20:19.309554+0000","last_active":"2024-09-21T12:21:01.797742+0000","last_peered":"2024-09-21T12:21:01.797742+0000","last_clean":"2024-09-21T12:21:01.797742+0000","last_became_active":"2024-09-21T12:20:19.309258+0000","last_became_peered":"2024-09-21T12:20:19.309258+0000","last_unstale":"2024-09-21T12:21:01.797742+0000","last_undegraded":"2024-09-21T12:21:01.797742+0000","last_fullsized":"2024-09-21T12:21:01.797742+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:44.940960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797367+0000","last_change":"2024-09-21T12:20:21.593145+0000","last_active":"2024-09-21T12:21:01.797367+0000","last_peered":"2024-09-21T12:21:01.797367+0000","last_clean":"2024-09-21T12:21:01.797367+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:21:01.797367+0000","last_undegraded":"2024-09-21T12:21:01.797367+0000","last_fullsized":"2024-09-21T12:21:01.797367+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.236407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":77,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.800972+0000","last_change":"2024-09-21T12:20:17.310292+0000","last_active":"2024-09-21T12:21:01.800972+0000","last_peered":"2024-09-21T12:21:01.800972+0000","last_clean":"2024-09-21T12:21:01.800972+0000","last_became_active":"2024-09-21T12:20:17.309828+0000","last_became_peered":"2024-09-21T12:20:17.309828+0000","last_unstale":"2024-09-21T12:21:01.800972+0000","last_undegraded":"2024-09-21T12:21:01.800972+0000","last_fullsized":"2024-09-21T12:21:01.800972+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:44.016276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760930+0000","last_change":"2024-09-21T12:20:15.302137+0000","last_active":"2024-09-21T12:21:01.760930+0000","last_peered":"2024-09-21T12:21:01.760930+0000","last_clean":"2024-09-21T12:21:01.760930+0000","last_became_active":"2024-09-21T12:20:15.301851+0000","last_became_peered":"2024-09-21T12:20:15.301851+0000","last_unstale":"2024-09-21T12:21:01.760930+0000","last_undegraded":"2024-09-21T12:21:01.760930+0000","last_fullsized":"2024-09-21T12:21:01.760930+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.661202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761056+0000","last_change":"2024-09-21T12:20:19.312912+0000","last_active":"2024-09-21T12:21:01.761056+0000","last_peered":"2024-09-21T12:21:01.761056+0000","last_clean":"2024-09-21T12:21:01.761056+0000","last_became_active":"2024-09-21T12:20:19.312301+0000","last_became_peered":"2024-09-21T12:20:19.312301+0000","last_unstale":"2024-09-21T12:21:01.761056+0000","last_undegraded":"2024-09-21T12:21:01.761056+0000","last_fullsized":"2024-09-21T12:21:01.761056+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:42.511905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797148+0000","last_change":"2024-09-21T12:20:21.313585+0000","last_active":"2024-09-21T12:21:01.797148+0000","last_peered":"2024-09-21T12:21:01.797148+0000","last_clean":"2024-09-21T12:21:01.797148+0000","last_became_active":"2024-09-21T12:20:21.313254+0000","last_became_peered":"2024-09-21T12:20:21.313254+0000","last_unstale":"2024-09-21T12:21:01.797148+0000","last_undegraded":"2024-09-21T12:21:01.797148+0000","last_fullsized":"2024-09-21T12:21:01.797148+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.442191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":64,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.800202+0000","last_change":"2024-09-21T12:20:17.307651+0000","last_active":"2024-09-21T12:21:01.800202+0000","last_peered":"2024-09-21T12:21:01.800202+0000","last_clean":"2024-09-21T12:21:01.800202+0000","last_became_active":"2024-09-21T12:20:17.307317+0000","last_became_peered":"2024-09-21T12:20:17.307317+0000","last_unstale":"2024-09-21T12:21:01.800202+0000","last_undegraded":"2024-09-21T12:21:01.800202+0000","last_fullsized":"2024-09-21T12:21:01.800202+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:37.610294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":52,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798603+0000","last_change":"2024-09-21T12:20:15.298107+0000","last_active":"2024-09-21T12:21:01.798603+0000","last_peered":"2024-09-21T12:21:01.798603+0000","last_clean":"2024-09-21T12:21:01.798603+0000","last_became_active":"2024-09-21T12:20:15.297819+0000","last_became_peered":"2024-09-21T12:20:15.297819+0000","last_unstale":"2024-09-21T12:21:01.798603+0000","last_undegraded":"2024-09-21T12:21:01.798603+0000","last_fullsized":"2024-09-21T12:21:01.798603+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:38.788978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":66,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:05.357778+0000","last_change":"2024-09-21T12:20:19.308246+0000","last_active":"2024-09-21T12:21:05.357778+0000","last_peered":"2024-09-21T12:21:05.357778+0000","last_clean":"2024-09-21T12:21:05.357778+0000","last_became_active":"2024-09-21T12:20:19.307924+0000","last_became_peered":"2024-09-21T12:20:19.307924+0000","last_unstale":"2024-09-21T12:21:05.357778+0000","last_undegraded":"2024-09-21T12:21:05.357778+0000","last_fullsized":"2024-09-21T12:21:05.357778+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:33.476030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798009+0000","last_change":"2024-09-21T12:20:21.313250+0000","last_active":"2024-09-21T12:21:01.798009+0000","last_peered":"2024-09-21T12:21:01.798009+0000","last_clean":"2024-09-21T12:21:01.798009+0000","last_became_active":"2024-09-21T12:20:21.312954+0000","last_became_peered":"2024-09-21T12:20:21.312954+0000","last_unstale":"2024-09-21T12:21:01.798009+0000","last_undegraded":"2024-09-21T12:21:01.798009+0000","last_fullsized":"2024-09-21T12:21:01.798009+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.513060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":66,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761383+0000","last_change":"2024-09-21T12:20:17.299948+0000","last_active":"2024-09-21T12:21:01.761383+0000","last_peered":"2024-09-21T12:21:01.761383+0000","last_clean":"2024-09-21T12:21:01.761383+0000","last_became_active":"2024-09-21T12:20:17.299653+0000","last_became_peered":"2024-09-21T12:20:17.299653+0000","last_unstale":"2024-09-21T12:21:01.761383+0000","last_undegraded":"2024-09-21T12:21:01.761383+0000","last_fullsized":"2024-09-21T12:21:01.761383+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:16.539847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798378+0000","last_change":"2024-09-21T12:20:15.302587+0000","last_active":"2024-09-21T12:21:01.798378+0000","last_peered":"2024-09-21T12:21:01.798378+0000","last_clean":"2024-09-21T12:21:01.798378+0000","last_became_active":"2024-09-21T12:20:15.302262+0000","last_became_peered":"2024-09-21T12:20:15.302262+0000","last_unstale":"2024-09-21T12:21:01.798378+0000","last_undegraded":"2024-09-21T12:21:01.798378+0000","last_fullsized":"2024-09-21T12:21:01.798378+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:32.093775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.796917+0000","last_change":"2024-09-21T12:20:19.308484+0000","last_active":"2024-09-21T12:21:01.796917+0000","last_peered":"2024-09-21T12:21:01.796917+0000","last_clean":"2024-09-21T12:21:01.796917+0000","last_became_active":"2024-09-21T12:20:19.308048+0000","last_became_peered":"2024-09-21T12:20:19.308048+0000","last_unstale":"2024-09-21T12:21:01.796917+0000","last_undegraded":"2024-09-21T12:21:01.796917+0000","last_fullsized":"2024-09-21T12:21:01.796917+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:34.470562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760554+0000","last_change":"2024-09-21T12:20:21.311799+0000","last_active":"2024-09-21T12:21:01.760554+0000","last_peered":"2024-09-21T12:21:01.760554+0000","last_clean":"2024-09-21T12:21:01.760554+0000","last_became_active":"2024-09-21T12:20:21.311493+0000","last_became_peered":"2024-09-21T12:20:21.311493+0000","last_unstale":"2024-09-21T12:21:01.760554+0000","last_undegraded":"2024-09-21T12:21:01.760554+0000","last_fullsized":"2024-09-21T12:21:01.760554+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:40.676620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":81,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760599+0000","last_change":"2024-09-21T12:20:17.693677+0000","last_active":"2024-09-21T12:21:01.760599+0000","last_peered":"2024-09-21T12:21:01.760599+0000","last_clean":"2024-09-21T12:21:01.760599+0000","last_became_active":"2024-09-21T12:20:17.692867+0000","last_became_peered":"2024-09-21T12:20:17.692867+0000","last_unstale":"2024-09-21T12:21:01.760599+0000","last_undegraded":"2024-09-21T12:21:01.760599+0000","last_fullsized":"2024-09-21T12:21:01.760599+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.747644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.800431+0000","last_change":"2024-09-21T12:20:15.300854+0000","last_active":"2024-09-21T12:21:01.800431+0000","last_peered":"2024-09-21T12:21:01.800431+0000","last_clean":"2024-09-21T12:21:01.800431+0000","last_became_active":"2024-09-21T12:20:15.299333+0000","last_became_peered":"2024-09-21T12:20:15.299333+0000","last_unstale":"2024-09-21T12:21:01.800431+0000","last_undegraded":"2024-09-21T12:21:01.800431+0000","last_fullsized":"2024-09-21T12:21:01.800431+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:34.912500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797216+0000","last_change":"2024-09-21T12:20:19.307316+0000","last_active":"2024-09-21T12:21:01.797216+0000","last_peered":"2024-09-21T12:21:01.797216+0000","last_clean":"2024-09-21T12:21:01.797216+0000","last_became_active":"2024-09-21T12:20:19.306952+0000","last_became_peered":"2024-09-21T12:20:19.306952+0000","last_unstale":"2024-09-21T12:21:01.797216+0000","last_undegraded":"2024-09-21T12:21:01.797216+0000","last_fullsized":"2024-09-21T12:21:01.797216+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:47.444300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799177+0000","last_change":"2024-09-21T12:20:21.314891+0000","last_active":"2024-09-21T12:21:01.799177+0000","last_peered":"2024-09-21T12:21:01.799177+0000","last_clean":"2024-09-21T12:21:01.799177+0000","last_became_active":"2024-09-21T12:20:21.314545+0000","last_became_peered":"2024-09-21T12:20:21.314545+0000","last_unstale":"2024-09-21T12:21:01.799177+0000","last_undegraded":"2024-09-21T12:21:01.799177+0000","last_fullsized":"2024-09-21T12:21:01.799177+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:21.854833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":72,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799889+0000","last_change":"2024-09-21T12:20:17.306090+0000","last_active":"2024-09-21T12:21:01.799889+0000","last_peered":"2024-09-21T12:21:01.799889+0000","last_clean":"2024-09-21T12:21:01.799889+0000","last_became_active":"2024-09-21T12:20:17.305661+0000","last_became_peered":"2024-09-21T12:20:17.305661+0000","last_unstale":"2024-09-21T12:21:01.799889+0000","last_undegraded":"2024-09-21T12:21:01.799889+0000","last_fullsized":"2024-09-21T12:21:01.799889+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:31.017952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.800776+0000","last_change":"2024-09-21T12:20:15.305876+0000","last_active":"2024-09-21T12:21:01.800776+0000","last_peered":"2024-09-21T12:21:01.800776+0000","last_clean":"2024-09-21T12:21:01.800776+0000","last_became_active":"2024-09-21T12:20:15.305545+0000","last_became_peered":"2024-09-21T12:20:15.305545+0000","last_unstale":"2024-09-21T12:21:01.800776+0000","last_undegraded":"2024-09-21T12:21:01.800776+0000","last_fullsized":"2024-09-21T12:21:01.800776+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:19.227777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.796678+0000","last_change":"2024-09-21T12:20:19.312986+0000","last_active":"2024-09-21T12:21:01.796678+0000","last_peered":"2024-09-21T12:21:01.796678+0000","last_clean":"2024-09-21T12:21:01.796678+0000","last_became_active":"2024-09-21T12:20:19.312572+0000","last_became_peered":"2024-09-21T12:20:19.312572+0000","last_unstale":"2024-09-21T12:21:01.796678+0000","last_undegraded":"2024-09-21T12:21:01.796678+0000","last_fullsized":"2024-09-21T12:21:01.796678+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:20.301869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797401+0000","last_change":"2024-09-21T12:20:21.317550+0000","last_active":"2024-09-21T12:21:01.797401+0000","last_peered":"2024-09-21T12:21:01.797401+0000","last_clean":"2024-09-21T12:21:01.797401+0000","last_became_active":"2024-09-21T12:20:21.317223+0000","last_became_peered":"2024-09-21T12:20:21.317223+0000","last_unstale":"2024-09-21T12:21:01.797401+0000","last_undegraded":"2024-09-21T12:21:01.797401+0000","last_fullsized":"2024-09-21T12:21:01.797401+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:25.406157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":75,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797194+0000","last_change":"2024-09-21T12:20:17.693302+0000","last_active":"2024-09-21T12:21:01.797194+0000","last_peered":"2024-09-21T12:21:01.797194+0000","last_clean":"2024-09-21T12:21:01.797194+0000","last_became_active":"2024-09-21T12:20:17.692870+0000","last_became_peered":"2024-09-21T12:20:17.692870+0000","last_unstale":"2024-09-21T12:21:01.797194+0000","last_undegraded":"2024-09-21T12:21:01.797194+0000","last_fullsized":"2024-09-21T12:21:01.797194+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.102470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.800720+0000","last_change":"2024-09-21T12:20:15.302366+0000","last_active":"2024-09-21T12:21:01.800720+0000","last_peered":"2024-09-21T12:21:01.800720+0000","last_clean":"2024-09-21T12:21:01.800720+0000","last_became_active":"2024-09-21T12:20:15.302038+0000","last_became_peered":"2024-09-21T12:20:15.302038+0000","last_unstale":"2024-09-21T12:21:01.800720+0000","last_undegraded":"2024-09-21T12:21:01.800720+0000","last_fullsized":"2024-09-21T12:21:01.800720+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:29.184795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:05.358056+0000","last_change":"2024-09-21T12:20:19.310188+0000","last_active":"2024-09-21T12:21:05.358056+0000","last_peered":"2024-09-21T12:21:05.358056+0000","last_clean":"2024-09-21T12:21:05.358056+0000","last_became_active":"2024-09-21T12:20:19.309708+0000","last_became_peered":"2024-09-21T12:20:19.309708+0000","last_unstale":"2024-09-21T12:21:05.358056+0000","last_undegraded":"2024-09-21T12:21:05.358056+0000","last_fullsized":"2024-09-21T12:21:05.358056+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:44.922935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798423+0000","last_change":"2024-09-21T12:20:21.592893+0000","last_active":"2024-09-21T12:21:01.798423+0000","last_peered":"2024-09-21T12:21:01.798423+0000","last_clean":"2024-09-21T12:21:01.798423+0000","last_became_active":"2024-09-21T12:20:21.592576+0000","last_became_peered":"2024-09-21T12:20:21.592576+0000","last_unstale":"2024-09-21T12:21:01.798423+0000","last_undegraded":"2024-09-21T12:21:01.798423+0000","last_fullsized":"2024-09-21T12:21:01.798423+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:45.792911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760521+0000","last_change":"2024-09-21T12:20:15.303770+0000","last_active":"2024-09-21T12:21:01.760521+0000","last_peered":"2024-09-21T12:21:01.760521+0000","last_clean":"2024-09-21T12:21:01.760521+0000","last_became_active":"2024-09-21T12:20:15.303480+0000","last_became_peered":"2024-09-21T12:20:15.303480+0000","last_unstale":"2024-09-21T12:21:01.760521+0000","last_undegraded":"2024-09-21T12:21:01.760521+0000","last_fullsized":"2024-09-21T12:21:01.760521+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:14.876724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":57,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799528+0000","last_change":"2024-09-21T12:20:17.694977+0000","last_active":"2024-09-21T12:21:01.799528+0000","last_peered":"2024-09-21T12:21:01.799528+0000","last_clean":"2024-09-21T12:21:01.799528+0000","last_became_active":"2024-09-21T12:20:17.694628+0000","last_became_peered":"2024-09-21T12:20:17.694628+0000","last_unstale":"2024-09-21T12:21:01.799528+0000","last_undegraded":"2024-09-21T12:21:01.799528+0000","last_fullsized":"2024-09-21T12:21:01.799528+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.288937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797934+0000","last_change":"2024-09-21T12:20:19.314432+0000","last_active":"2024-09-21T12:21:01.797934+0000","last_peered":"2024-09-21T12:21:01.797934+0000","last_clean":"2024-09-21T12:21:01.797934+0000","last_became_active":"2024-09-21T12:20:19.313882+0000","last_became_peered":"2024-09-21T12:20:19.313882+0000","last_unstale":"2024-09-21T12:21:01.797934+0000","last_undegraded":"2024-09-21T12:21:01.797934+0000","last_fullsized":"2024-09-21T12:21:01.797934+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.453804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.800106+0000","last_change":"2024-09-21T12:20:21.594311+0000","last_active":"2024-09-21T12:21:01.800106+0000","last_peered":"2024-09-21T12:21:01.800106+0000","last_clean":"2024-09-21T12:21:01.800106+0000","last_became_active":"2024-09-21T12:20:21.594048+0000","last_became_peered":"2024-09-21T12:20:21.594048+0000","last_unstale":"2024-09-21T12:21:01.800106+0000","last_undegraded":"2024-09-21T12:21:01.800106+0000","last_fullsized":"2024-09-21T12:21:01.800106+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:38.347489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":66,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761540+0000","last_change":"2024-09-21T12:20:17.696279+0000","last_active":"2024-09-21T12:21:01.761540+0000","last_peered":"2024-09-21T12:21:01.761540+0000","last_clean":"2024-09-21T12:21:01.761540+0000","last_became_active":"2024-09-21T12:20:17.695898+0000","last_became_peered":"2024-09-21T12:20:17.695898+0000","last_unstale":"2024-09-21T12:21:01.761540+0000","last_undegraded":"2024-09-21T12:21:01.761540+0000","last_fullsized":"2024-09-21T12:21:01.761540+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:21.926785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798290+0000","last_change":"2024-09-21T12:20:15.301560+0000","last_active":"2024-09-21T12:21:01.798290+0000","last_peered":"2024-09-21T12:21:01.798290+0000","last_clean":"2024-09-21T12:21:01.798290+0000","last_became_active":"2024-09-21T12:20:15.300722+0000","last_became_peered":"2024-09-21T12:20:15.300722+0000","last_unstale":"2024-09-21T12:21:01.798290+0000","last_undegraded":"2024-09-21T12:21:01.798290+0000","last_fullsized":"2024-09-21T12:21:01.798290+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.142850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":44,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799393+0000","last_change":"2024-09-21T12:20:19.312248+0000","last_active":"2024-09-21T12:21:01.799393+0000","last_peered":"2024-09-21T12:21:01.799393+0000","last_clean":"2024-09-21T12:21:01.799393+0000","last_became_active":"2024-09-21T12:20:19.311947+0000","last_became_peered":"2024-09-21T12:20:19.311947+0000","last_unstale":"2024-09-21T12:21:01.799393+0000","last_undegraded":"2024-09-21T12:21:01.799393+0000","last_fullsized":"2024-09-21T12:21:01.799393+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.388400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797355+0000","last_change":"2024-09-21T12:20:21.314554+0000","last_active":"2024-09-21T12:21:01.797355+0000","last_peered":"2024-09-21T12:21:01.797355+0000","last_clean":"2024-09-21T12:21:01.797355+0000","last_became_active":"2024-09-21T12:20:21.314277+0000","last_became_peered":"2024-09-21T12:20:21.314277+0000","last_unstale":"2024-09-21T12:21:01.797355+0000","last_undegraded":"2024-09-21T12:21:01.797355+0000","last_fullsized":"2024-09-21T12:21:01.797355+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.219191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":66,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799278+0000","last_change":"2024-09-21T12:20:17.694904+0000","last_active":"2024-09-21T12:21:01.799278+0000","last_peered":"2024-09-21T12:21:01.799278+0000","last_clean":"2024-09-21T12:21:01.799278+0000","last_became_active":"2024-09-21T12:20:17.694568+0000","last_became_peered":"2024-09-21T12:20:17.694568+0000","last_unstale":"2024-09-21T12:21:01.799278+0000","last_undegraded":"2024-09-21T12:21:01.799278+0000","last_fullsized":"2024-09-21T12:21:01.799278+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:24.206067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761045+0000","last_change":"2024-09-21T12:20:15.297853+0000","last_active":"2024-09-21T12:21:01.761045+0000","last_peered":"2024-09-21T12:21:01.761045+0000","last_clean":"2024-09-21T12:21:01.761045+0000","last_became_active":"2024-09-21T12:20:15.297519+0000","last_became_peered":"2024-09-21T12:20:15.297519+0000","last_unstale":"2024-09-21T12:21:01.761045+0000","last_undegraded":"2024-09-21T12:21:01.761045+0000","last_fullsized":"2024-09-21T12:21:01.761045+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:29.814404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":64,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:05.357792+0000","last_change":"2024-09-21T12:20:19.305689+0000","last_active":"2024-09-21T12:21:05.357792+0000","last_peered":"2024-09-21T12:21:05.357792+0000","last_clean":"2024-09-21T12:21:05.357792+0000","last_became_active":"2024-09-21T12:20:19.305364+0000","last_became_peered":"2024-09-21T12:20:19.305364+0000","last_unstale":"2024-09-21T12:21:05.357792+0000","last_undegraded":"2024-09-21T12:21:05.357792+0000","last_fullsized":"2024-09-21T12:21:05.357792+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.650981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798482+0000","last_change":"2024-09-21T12:20:21.315079+0000","last_active":"2024-09-21T12:21:01.798482+0000","last_peered":"2024-09-21T12:21:01.798482+0000","last_clean":"2024-09-21T12:21:01.798482+0000","last_became_active":"2024-09-21T12:20:21.314066+0000","last_became_peered":"2024-09-21T12:20:21.314066+0000","last_unstale":"2024-09-21T12:21:01.798482+0000","last_undegraded":"2024-09-21T12:21:01.798482+0000","last_fullsized":"2024-09-21T12:21:01.798482+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:27.528692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":63,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799346+0000","last_change":"2024-09-21T12:20:17.693562+0000","last_active":"2024-09-21T12:21:01.799346+0000","last_peered":"2024-09-21T12:21:01.799346+0000","last_clean":"2024-09-21T12:21:01.799346+0000","last_became_active":"2024-09-21T12:20:17.693209+0000","last_became_peered":"2024-09-21T12:20:17.693209+0000","last_unstale":"2024-09-21T12:21:01.799346+0000","last_undegraded":"2024-09-21T12:21:01.799346+0000","last_fullsized":"2024-09-21T12:21:01.799346+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:25.923701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799435+0000","last_change":"2024-09-21T12:20:15.308767+0000","last_active":"2024-09-21T12:21:01.799435+0000","last_peered":"2024-09-21T12:21:01.799435+0000","last_clean":"2024-09-21T12:21:01.799435+0000","last_became_active":"2024-09-21T12:20:15.308429+0000","last_became_peered":"2024-09-21T12:20:15.308429+0000","last_unstale":"2024-09-21T12:21:01.799435+0000","last_undegraded":"2024-09-21T12:21:01.799435+0000","last_fullsized":"2024-09-21T12:21:01.799435+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:15.014581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":66,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:05.358290+0000","last_change":"2024-09-21T12:20:19.314884+0000","last_active":"2024-09-21T12:21:05.358290+0000","last_peered":"2024-09-21T12:21:05.358290+0000","last_clean":"2024-09-21T12:21:05.358290+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:21:05.358290+0000","last_undegraded":"2024-09-21T12:21:05.358290+0000","last_fullsized":"2024-09-21T12:21:05.358290+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:29.894782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761225+0000","last_change":"2024-09-21T12:20:21.309936+0000","last_active":"2024-09-21T12:21:01.761225+0000","last_peered":"2024-09-21T12:21:01.761225+0000","last_clean":"2024-09-21T12:21:01.761225+0000","last_became_active":"2024-09-21T12:20:21.309385+0000","last_became_peered":"2024-09-21T12:20:21.309385+0000","last_unstale":"2024-09-21T12:21:01.761225+0000","last_undegraded":"2024-09-21T12:21:01.761225+0000","last_fullsized":"2024-09-21T12:21:01.761225+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:47.499218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":68,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.800003+0000","last_change":"2024-09-21T12:20:17.304948+0000","last_active":"2024-09-21T12:21:01.800003+0000","last_peered":"2024-09-21T12:21:01.800003+0000","last_clean":"2024-09-21T12:21:01.800003+0000","last_became_active":"2024-09-21T12:20:17.303763+0000","last_became_peered":"2024-09-21T12:20:17.303763+0000","last_unstale":"2024-09-21T12:21:01.800003+0000","last_undegraded":"2024-09-21T12:21:01.800003+0000","last_fullsized":"2024-09-21T12:21:01.800003+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:31.523086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.800458+0000","last_change":"2024-09-21T12:20:15.298500+0000","last_active":"2024-09-21T12:21:01.800458+0000","last_peered":"2024-09-21T12:21:01.800458+0000","last_clean":"2024-09-21T12:21:01.800458+0000","last_became_active":"2024-09-21T12:20:15.298198+0000","last_became_peered":"2024-09-21T12:20:15.298198+0000","last_unstale":"2024-09-21T12:21:01.800458+0000","last_undegraded":"2024-09-21T12:21:01.800458+0000","last_fullsized":"2024-09-21T12:21:01.800458+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:36.549556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760710+0000","last_change":"2024-09-21T12:20:19.315714+0000","last_active":"2024-09-21T12:21:01.760710+0000","last_peered":"2024-09-21T12:21:01.760710+0000","last_clean":"2024-09-21T12:21:01.760710+0000","last_became_active":"2024-09-21T12:20:19.315418+0000","last_became_peered":"2024-09-21T12:20:19.315418+0000","last_unstale":"2024-09-21T12:21:01.760710+0000","last_undegraded":"2024-09-21T12:21:01.760710+0000","last_fullsized":"2024-09-21T12:21:01.760710+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:38.005096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797490+0000","last_change":"2024-09-21T12:20:21.310344+0000","last_active":"2024-09-21T12:21:01.797490+0000","last_peered":"2024-09-21T12:21:01.797490+0000","last_clean":"2024-09-21T12:21:01.797490+0000","last_became_active":"2024-09-21T12:20:21.310053+0000","last_became_peered":"2024-09-21T12:20:21.310053+0000","last_unstale":"2024-09-21T12:21:01.797490+0000","last_undegraded":"2024-09-21T12:21:01.797490+0000","last_fullsized":"2024-09-21T12:21:01.797490+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:42.963350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":66,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760846+0000","last_change":"2024-09-21T12:20:17.299367+0000","last_active":"2024-09-21T12:21:01.760846+0000","last_peered":"2024-09-21T12:21:01.760846+0000","last_clean":"2024-09-21T12:21:01.760846+0000","last_became_active":"2024-09-21T12:20:17.299015+0000","last_became_peered":"2024-09-21T12:20:17.299015+0000","last_unstale":"2024-09-21T12:21:01.760846+0000","last_undegraded":"2024-09-21T12:21:01.760846+0000","last_fullsized":"2024-09-21T12:21:01.760846+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:42.174386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.800658+0000","last_change":"2024-09-21T12:20:15.303462+0000","last_active":"2024-09-21T12:21:01.800658+0000","last_peered":"2024-09-21T12:21:01.800658+0000","last_clean":"2024-09-21T12:21:01.800658+0000","last_became_active":"2024-09-21T12:20:15.300554+0000","last_became_peered":"2024-09-21T12:20:15.300554+0000","last_unstale":"2024-09-21T12:21:01.800658+0000","last_undegraded":"2024-09-21T12:21:01.800658+0000","last_fullsized":"2024-09-21T12:21:01.800658+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:22.850479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":44,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798353+0000","last_change":"2024-09-21T12:20:19.311327+0000","last_active":"2024-09-21T12:21:01.798353+0000","last_peered":"2024-09-21T12:21:01.798353+0000","last_clean":"2024-09-21T12:21:01.798353+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:21:01.798353+0000","last_undegraded":"2024-09-21T12:21:01.798353+0000","last_fullsized":"2024-09-21T12:21:01.798353+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:20.603464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760220+0000","last_change":"2024-09-21T12:20:21.308003+0000","last_active":"2024-09-21T12:21:01.760220+0000","last_peered":"2024-09-21T12:21:01.760220+0000","last_clean":"2024-09-21T12:21:01.760220+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:21:01.760220+0000","last_undegraded":"2024-09-21T12:21:01.760220+0000","last_fullsized":"2024-09-21T12:21:01.760220+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:41.631005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":67,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798016+0000","last_change":"2024-09-21T12:20:17.695358+0000","last_active":"2024-09-21T12:21:01.798016+0000","last_peered":"2024-09-21T12:21:01.798016+0000","last_clean":"2024-09-21T12:21:01.798016+0000","last_became_active":"2024-09-21T12:20:17.694794+0000","last_became_peered":"2024-09-21T12:20:17.694794+0000","last_unstale":"2024-09-21T12:21:01.798016+0000","last_undegraded":"2024-09-21T12:21:01.798016+0000","last_fullsized":"2024-09-21T12:21:01.798016+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:25.305944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798031+0000","last_change":"2024-09-21T12:20:15.296285+0000","last_active":"2024-09-21T12:21:01.798031+0000","last_peered":"2024-09-21T12:21:01.798031+0000","last_clean":"2024-09-21T12:21:01.798031+0000","last_became_active":"2024-09-21T12:20:15.295968+0000","last_became_peered":"2024-09-21T12:20:15.295968+0000","last_unstale":"2024-09-21T12:21:01.798031+0000","last_undegraded":"2024-09-21T12:21:01.798031+0000","last_fullsized":"2024-09-21T12:21:01.798031+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:33.273850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797528+0000","last_change":"2024-09-21T12:20:19.309473+0000","last_active":"2024-09-21T12:21:01.797528+0000","last_peered":"2024-09-21T12:21:01.797528+0000","last_clean":"2024-09-21T12:21:01.797528+0000","last_became_active":"2024-09-21T12:20:19.309195+0000","last_became_peered":"2024-09-21T12:20:19.309195+0000","last_unstale":"2024-09-21T12:21:01.797528+0000","last_undegraded":"2024-09-21T12:21:01.797528+0000","last_fullsized":"2024-09-21T12:21:01.797528+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:39.580857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797258+0000","last_change":"2024-09-21T12:20:21.595660+0000","last_active":"2024-09-21T12:21:01.797258+0000","last_peered":"2024-09-21T12:21:01.797258+0000","last_clean":"2024-09-21T12:21:01.797258+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:21:01.797258+0000","last_undegraded":"2024-09-21T12:21:01.797258+0000","last_fullsized":"2024-09-21T12:21:01.797258+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:22.428602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797916+0000","last_change":"2024-09-21T12:20:17.304075+0000","last_active":"2024-09-21T12:21:01.797916+0000","last_peered":"2024-09-21T12:21:01.797916+0000","last_clean":"2024-09-21T12:21:01.797916+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:21:01.797916+0000","last_undegraded":"2024-09-21T12:21:01.797916+0000","last_fullsized":"2024-09-21T12:21:01.797916+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.024324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":60,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761119+0000","last_change":"2024-09-21T12:20:15.298443+0000","last_active":"2024-09-21T12:21:01.761119+0000","last_peered":"2024-09-21T12:21:01.761119+0000","last_clean":"2024-09-21T12:21:01.761119+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:21:01.761119+0000","last_undegraded":"2024-09-21T12:21:01.761119+0000","last_fullsized":"2024-09-21T12:21:01.761119+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:36.774153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760492+0000","last_change":"2024-09-21T12:20:19.312978+0000","last_active":"2024-09-21T12:21:01.760492+0000","last_peered":"2024-09-21T12:21:01.760492+0000","last_clean":"2024-09-21T12:21:01.760492+0000","last_became_active":"2024-09-21T12:20:19.312512+0000","last_became_peered":"2024-09-21T12:20:19.312512+0000","last_unstale":"2024-09-21T12:21:01.760492+0000","last_undegraded":"2024-09-21T12:21:01.760492+0000","last_fullsized":"2024-09-21T12:21:01.760492+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:41.253296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797974+0000","last_change":"2024-09-21T12:20:21.313523+0000","last_active":"2024-09-21T12:21:01.797974+0000","last_peered":"2024-09-21T12:21:01.797974+0000","last_clean":"2024-09-21T12:21:01.797974+0000","last_became_active":"2024-09-21T12:20:21.313228+0000","last_became_peered":"2024-09-21T12:20:21.313228+0000","last_unstale":"2024-09-21T12:21:01.797974+0000","last_undegraded":"2024-09-21T12:21:01.797974+0000","last_fullsized":"2024-09-21T12:21:01.797974+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:47.112414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761201+0000","last_change":"2024-09-21T12:20:21.309169+0000","last_active":"2024-09-21T12:21:01.761201+0000","last_peered":"2024-09-21T12:21:01.761201+0000","last_clean":"2024-09-21T12:21:01.761201+0000","last_became_active":"2024-09-21T12:20:21.308865+0000","last_became_peered":"2024-09-21T12:20:21.308865+0000","last_unstale":"2024-09-21T12:21:01.761201+0000","last_undegraded":"2024-09-21T12:21:01.761201+0000","last_fullsized":"2024-09-21T12:21:01.761201+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:26.841000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":52,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798231+0000","last_change":"2024-09-21T12:20:15.298322+0000","last_active":"2024-09-21T12:21:01.798231+0000","last_peered":"2024-09-21T12:21:01.798231+0000","last_clean":"2024-09-21T12:21:01.798231+0000","last_became_active":"2024-09-21T12:20:15.297993+0000","last_became_peered":"2024-09-21T12:20:15.297993+0000","last_unstale":"2024-09-21T12:21:01.798231+0000","last_undegraded":"2024-09-21T12:21:01.798231+0000","last_fullsized":"2024-09-21T12:21:01.798231+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:34.289061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":69,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761092+0000","last_change":"2024-09-21T12:20:17.299134+0000","last_active":"2024-09-21T12:21:01.761092+0000","last_peered":"2024-09-21T12:21:01.761092+0000","last_clean":"2024-09-21T12:21:01.761092+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:21:01.761092+0000","last_undegraded":"2024-09-21T12:21:01.761092+0000","last_fullsized":"2024-09-21T12:21:01.761092+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.792664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761320+0000","last_change":"2024-09-21T12:20:19.308586+0000","last_active":"2024-09-21T12:21:01.761320+0000","last_peered":"2024-09-21T12:21:01.761320+0000","last_clean":"2024-09-21T12:21:01.761320+0000","last_became_active":"2024-09-21T12:20:19.308247+0000","last_became_peered":"2024-09-21T12:20:19.308247+0000","last_unstale":"2024-09-21T12:21:01.761320+0000","last_undegraded":"2024-09-21T12:21:01.761320+0000","last_fullsized":"2024-09-21T12:21:01.761320+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:42.632946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":51,"num_read_kb":46,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928233999,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32648,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712504,"statfs":{"total":95995035648,"available":95961604096,"internally_reserved":0,"allocated":1495040,"data_stored":990615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462934,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32636,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712516,"statfs":{"total":95995035648,"available":95961616384,"internally_reserved":0,"allocated":1478656,"data_stored":989966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724570,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":897024,"data_stored":400963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953506,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":909312,"data_stored":399767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149736,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":925696,"data_stored":401482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32048,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713104,"statfs":{"total":95995035648,"available":95962218496,"internally_reserved":0,"allocated":876544,"data_stored":399310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":950272,"data_stored":400960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28504,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716648,"statfs":{"total":95995035648,"available":95965847552,"internally_reserved":0,"allocated":1507328,"data_stored":991330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:21:12.983 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:21:13.195 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:12 smithi114 ceph-mon[40867]: pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T12:21:13.196 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:12 smithi114 ceph-mon[40867]: from='client.24620 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:21:13.196 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:12 smithi114 ceph-mon[30324]: pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T12:21:13.196 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:12 smithi114 ceph-mon[30324]: from='client.24620 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:21:13.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:12 smithi183 ceph-mon[33277]: pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T12:21:13.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:12 smithi183 ceph-mon[33277]: from='client.24620 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:21:13.524 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:21:13 smithi114 podman[75383]: 2024-09-21 12:21:13.311334679 +0000 UTC m=+0.222326692 container create 5dcc38d0a6d785f370bec3180c131bb70fdc8cdeb567de62e720e4f8bb1c7006 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T12:21:13.788 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:21:13.911 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:21:13 smithi114 podman[75383]: 2024-09-21 12:21:13.647693393 +0000 UTC m=+0.558685420 container init 5dcc38d0a6d785f370bec3180c131bb70fdc8cdeb567de62e720e4f8bb1c7006 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T12:21:13.911 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:21:13 smithi114 podman[75383]: 2024-09-21 12:21:13.652321401 +0000 UTC m=+0.563313422 container start 5dcc38d0a6d785f370bec3180c131bb70fdc8cdeb567de62e720e4f8bb1c7006 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T12:21:13.911 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:21:13 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a[75423]: ts=2024-09-21T12:21:13.663Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2024-09-21T12:21:13.911 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:21:13 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a[75423]: ts=2024-09-21T12:21:13.663Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)" 2024-09-21T12:21:13.911 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:21:13 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a[75423]: ts=2024-09-21T12:21:13.664Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.114 port=9094 2024-09-21T12:21:13.912 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:21:13 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a[75423]: ts=2024-09-21T12:21:13.665Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-21T12:21:13.912 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:21:13 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a[75423]: ts=2024-09-21T12:21:13.687Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T12:21:13.912 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:21:13 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a[75423]: ts=2024-09-21T12:21:13.687Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T12:21:13.912 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:21:13 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a[75423]: ts=2024-09-21T12:21:13.689Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093 2024-09-21T12:21:13.912 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:21:13 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a[75423]: ts=2024-09-21T12:21:13.689Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-21T12:21:13.912 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:21:13 smithi114 bash[75383]: 5dcc38d0a6d785f370bec3180c131bb70fdc8cdeb567de62e720e4f8bb1c7006 2024-09-21T12:21:13.912 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:21:13 smithi114 systemd[1]: Started Ceph alertmanager.a for e2ad975c-7812-11ef-baf6-efdc52797490. 2024-09-21T12:21:14.489 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:21:14.490 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:21:14.775 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:21:14] ENGINE Bus STOPPING 2024-09-21T12:21:14.775 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:21:14] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T12:21:14.775 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:21:14] ENGINE Bus STOPPED 2024-09-21T12:21:14.775 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:21:14] ENGINE Bus STARTING 2024-09-21T12:21:14.775 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:21:14] ENGINE Serving on http://:::9283 2024-09-21T12:21:14.775 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:21:14] ENGINE Bus STARTED 2024-09-21T12:21:15.083 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-mon[40867]: pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 868 B/s rd, 0 op/s 2024-09-21T12:21:15.083 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:15.083 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:15.083 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:15.083 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:15.083 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-mon[40867]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-21T12:21:15.083 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:21:15.084 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:21:15.084 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:15.084 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:15.084 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-21T12:21:15.084 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:15.084 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T12:21:15.084 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-mon[30324]: pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 868 B/s rd, 0 op/s 2024-09-21T12:21:15.084 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:15.084 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:15.084 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:15.084 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:15.085 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-mon[30324]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-21T12:21:15.085 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:21:15.085 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:21:15.085 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:15.085 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:15.085 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-21T12:21:15.085 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:15.085 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:14 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T12:21:15.109 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-21T12:21:13.551012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":899,"num_read_kb":756,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":245868,"kb_used_data":8828,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749715348,"statfs":{"total":767960285184,"available":767708516352,"internally_reserved":0,"allocated":9039872,"data_stored":4974393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.791479"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761030+0000","last_change":"2024-09-21T12:20:15.309744+0000","last_active":"2024-09-21T12:21:01.761030+0000","last_peered":"2024-09-21T12:21:01.761030+0000","last_clean":"2024-09-21T12:21:01.761030+0000","last_became_active":"2024-09-21T12:20:15.309431+0000","last_became_peered":"2024-09-21T12:20:15.309431+0000","last_unstale":"2024-09-21T12:21:01.761030+0000","last_undegraded":"2024-09-21T12:21:01.761030+0000","last_fullsized":"2024-09-21T12:21:01.761030+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:37.321280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798853+0000","last_change":"2024-09-21T12:20:17.304357+0000","last_active":"2024-09-21T12:21:01.798853+0000","last_peered":"2024-09-21T12:21:01.798853+0000","last_clean":"2024-09-21T12:21:01.798853+0000","last_became_active":"2024-09-21T12:20:17.303856+0000","last_became_peered":"2024-09-21T12:20:17.303856+0000","last_unstale":"2024-09-21T12:21:01.798853+0000","last_undegraded":"2024-09-21T12:21:01.798853+0000","last_fullsized":"2024-09-21T12:21:01.798853+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:40.716880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760469+0000","last_change":"2024-09-21T12:20:19.315239+0000","last_active":"2024-09-21T12:21:01.760469+0000","last_peered":"2024-09-21T12:21:01.760469+0000","last_clean":"2024-09-21T12:21:01.760469+0000","last_became_active":"2024-09-21T12:20:19.314908+0000","last_became_peered":"2024-09-21T12:20:19.314908+0000","last_unstale":"2024-09-21T12:21:01.760469+0000","last_undegraded":"2024-09-21T12:21:01.760469+0000","last_fullsized":"2024-09-21T12:21:01.760469+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:33.386500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797572+0000","last_change":"2024-09-21T12:20:21.308871+0000","last_active":"2024-09-21T12:21:01.797572+0000","last_peered":"2024-09-21T12:21:01.797572+0000","last_clean":"2024-09-21T12:21:01.797572+0000","last_became_active":"2024-09-21T12:20:21.308577+0000","last_became_peered":"2024-09-21T12:20:21.308577+0000","last_unstale":"2024-09-21T12:21:01.797572+0000","last_undegraded":"2024-09-21T12:21:01.797572+0000","last_fullsized":"2024-09-21T12:21:01.797572+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:43.912920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798215+0000","last_change":"2024-09-21T12:20:21.595607+0000","last_active":"2024-09-21T12:21:01.798215+0000","last_peered":"2024-09-21T12:21:01.798215+0000","last_clean":"2024-09-21T12:21:01.798215+0000","last_became_active":"2024-09-21T12:20:21.595301+0000","last_became_peered":"2024-09-21T12:20:21.595301+0000","last_unstale":"2024-09-21T12:21:01.798215+0000","last_undegraded":"2024-09-21T12:21:01.798215+0000","last_fullsized":"2024-09-21T12:21:01.798215+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:27.801303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":52,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799325+0000","last_change":"2024-09-21T12:20:15.308941+0000","last_active":"2024-09-21T12:21:01.799325+0000","last_peered":"2024-09-21T12:21:01.799325+0000","last_clean":"2024-09-21T12:21:01.799325+0000","last_became_active":"2024-09-21T12:20:15.308655+0000","last_became_peered":"2024-09-21T12:20:15.308655+0000","last_unstale":"2024-09-21T12:21:01.799325+0000","last_undegraded":"2024-09-21T12:21:01.799325+0000","last_fullsized":"2024-09-21T12:21:01.799325+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:31.202541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":73,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798236+0000","last_change":"2024-09-21T12:20:17.305499+0000","last_active":"2024-09-21T12:21:01.798236+0000","last_peered":"2024-09-21T12:21:01.798236+0000","last_clean":"2024-09-21T12:21:01.798236+0000","last_became_active":"2024-09-21T12:20:17.305220+0000","last_became_peered":"2024-09-21T12:20:17.305220+0000","last_unstale":"2024-09-21T12:21:01.798236+0000","last_undegraded":"2024-09-21T12:21:01.798236+0000","last_fullsized":"2024-09-21T12:21:01.798236+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.370654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798939+0000","last_change":"2024-09-21T12:20:19.311730+0000","last_active":"2024-09-21T12:21:01.798939+0000","last_peered":"2024-09-21T12:21:01.798939+0000","last_clean":"2024-09-21T12:21:01.798939+0000","last_became_active":"2024-09-21T12:20:19.311403+0000","last_became_peered":"2024-09-21T12:20:19.311403+0000","last_unstale":"2024-09-21T12:21:01.798939+0000","last_undegraded":"2024-09-21T12:21:01.798939+0000","last_fullsized":"2024-09-21T12:21:01.798939+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:40.685703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799371+0000","last_change":"2024-09-21T12:20:15.302150+0000","last_active":"2024-09-21T12:21:01.799371+0000","last_peered":"2024-09-21T12:21:01.799371+0000","last_clean":"2024-09-21T12:21:01.799371+0000","last_became_active":"2024-09-21T12:20:15.301847+0000","last_became_peered":"2024-09-21T12:20:15.301847+0000","last_unstale":"2024-09-21T12:21:01.799371+0000","last_undegraded":"2024-09-21T12:21:01.799371+0000","last_fullsized":"2024-09-21T12:21:01.799371+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:26.753083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799636+0000","last_change":"2024-09-21T12:20:17.307650+0000","last_active":"2024-09-21T12:21:01.799636+0000","last_peered":"2024-09-21T12:21:01.799636+0000","last_clean":"2024-09-21T12:21:01.799636+0000","last_became_active":"2024-09-21T12:20:17.307280+0000","last_became_peered":"2024-09-21T12:20:17.307280+0000","last_unstale":"2024-09-21T12:21:01.799636+0000","last_undegraded":"2024-09-21T12:21:01.799636+0000","last_fullsized":"2024-09-21T12:21:01.799636+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:34.375479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798060+0000","last_change":"2024-09-21T12:20:19.308520+0000","last_active":"2024-09-21T12:21:01.798060+0000","last_peered":"2024-09-21T12:21:01.798060+0000","last_clean":"2024-09-21T12:21:01.798060+0000","last_became_active":"2024-09-21T12:20:19.308184+0000","last_became_peered":"2024-09-21T12:20:19.308184+0000","last_unstale":"2024-09-21T12:21:01.798060+0000","last_undegraded":"2024-09-21T12:21:01.798060+0000","last_fullsized":"2024-09-21T12:21:01.798060+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:39.981449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760366+0000","last_change":"2024-09-21T12:20:21.309871+0000","last_active":"2024-09-21T12:21:01.760366+0000","last_peered":"2024-09-21T12:21:01.760366+0000","last_clean":"2024-09-21T12:21:01.760366+0000","last_became_active":"2024-09-21T12:20:21.309190+0000","last_became_peered":"2024-09-21T12:20:21.309190+0000","last_unstale":"2024-09-21T12:21:01.760366+0000","last_undegraded":"2024-09-21T12:21:01.760366+0000","last_fullsized":"2024-09-21T12:21:01.760366+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.813677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798210+0000","last_change":"2024-09-21T12:20:15.300728+0000","last_active":"2024-09-21T12:21:01.798210+0000","last_peered":"2024-09-21T12:21:01.798210+0000","last_clean":"2024-09-21T12:21:01.798210+0000","last_became_active":"2024-09-21T12:20:15.300441+0000","last_became_peered":"2024-09-21T12:20:15.300441+0000","last_unstale":"2024-09-21T12:21:01.798210+0000","last_undegraded":"2024-09-21T12:21:01.798210+0000","last_fullsized":"2024-09-21T12:21:01.798210+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:34.282857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":59,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799359+0000","last_change":"2024-09-21T12:20:17.305973+0000","last_active":"2024-09-21T12:21:01.799359+0000","last_peered":"2024-09-21T12:21:01.799359+0000","last_clean":"2024-09-21T12:21:01.799359+0000","last_became_active":"2024-09-21T12:20:17.305663+0000","last_became_peered":"2024-09-21T12:20:17.305663+0000","last_unstale":"2024-09-21T12:21:01.799359+0000","last_undegraded":"2024-09-21T12:21:01.799359+0000","last_fullsized":"2024-09-21T12:21:01.799359+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.420641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798568+0000","last_change":"2024-09-21T12:20:19.312502+0000","last_active":"2024-09-21T12:21:01.798568+0000","last_peered":"2024-09-21T12:21:01.798568+0000","last_clean":"2024-09-21T12:21:01.798568+0000","last_became_active":"2024-09-21T12:20:19.312242+0000","last_became_peered":"2024-09-21T12:20:19.312242+0000","last_unstale":"2024-09-21T12:21:01.798568+0000","last_undegraded":"2024-09-21T12:21:01.798568+0000","last_fullsized":"2024-09-21T12:21:01.798568+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:32.544179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797726+0000","last_change":"2024-09-21T12:20:21.311536+0000","last_active":"2024-09-21T12:21:01.797726+0000","last_peered":"2024-09-21T12:21:01.797726+0000","last_clean":"2024-09-21T12:21:01.797726+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:21:01.797726+0000","last_undegraded":"2024-09-21T12:21:01.797726+0000","last_fullsized":"2024-09-21T12:21:01.797726+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:41.239506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797876+0000","last_change":"2024-09-21T12:20:21.592944+0000","last_active":"2024-09-21T12:21:01.797876+0000","last_peered":"2024-09-21T12:21:01.797876+0000","last_clean":"2024-09-21T12:21:01.797876+0000","last_became_active":"2024-09-21T12:20:21.592631+0000","last_became_peered":"2024-09-21T12:20:21.592631+0000","last_unstale":"2024-09-21T12:21:01.797876+0000","last_undegraded":"2024-09-21T12:21:01.797876+0000","last_fullsized":"2024-09-21T12:21:01.797876+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:39.384088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761697+0000","last_change":"2024-09-21T12:20:15.302079+0000","last_active":"2024-09-21T12:21:01.761697+0000","last_peered":"2024-09-21T12:21:01.761697+0000","last_clean":"2024-09-21T12:21:01.761697+0000","last_became_active":"2024-09-21T12:20:15.301793+0000","last_became_peered":"2024-09-21T12:20:15.301793+0000","last_unstale":"2024-09-21T12:21:01.761697+0000","last_undegraded":"2024-09-21T12:21:01.761697+0000","last_fullsized":"2024-09-21T12:21:01.761697+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:19.991370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":75,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797700+0000","last_change":"2024-09-21T12:20:17.308186+0000","last_active":"2024-09-21T12:21:01.797700+0000","last_peered":"2024-09-21T12:21:01.797700+0000","last_clean":"2024-09-21T12:21:01.797700+0000","last_became_active":"2024-09-21T12:20:17.307828+0000","last_became_peered":"2024-09-21T12:20:17.307828+0000","last_unstale":"2024-09-21T12:21:01.797700+0000","last_undegraded":"2024-09-21T12:21:01.797700+0000","last_fullsized":"2024-09-21T12:21:01.797700+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:41.845334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761354+0000","last_change":"2024-09-21T12:20:19.316360+0000","last_active":"2024-09-21T12:21:01.761354+0000","last_peered":"2024-09-21T12:21:01.761354+0000","last_clean":"2024-09-21T12:21:01.761354+0000","last_became_active":"2024-09-21T12:20:19.315977+0000","last_became_peered":"2024-09-21T12:20:19.315977+0000","last_unstale":"2024-09-21T12:21:01.761354+0000","last_undegraded":"2024-09-21T12:21:01.761354+0000","last_fullsized":"2024-09-21T12:21:01.761354+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:37.515718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797418+0000","last_change":"2024-09-21T12:20:21.593129+0000","last_active":"2024-09-21T12:21:01.797418+0000","last_peered":"2024-09-21T12:21:01.797418+0000","last_clean":"2024-09-21T12:21:01.797418+0000","last_became_active":"2024-09-21T12:20:21.592796+0000","last_became_peered":"2024-09-21T12:20:21.592796+0000","last_unstale":"2024-09-21T12:21:01.797418+0000","last_undegraded":"2024-09-21T12:21:01.797418+0000","last_fullsized":"2024-09-21T12:21:01.797418+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:47.880164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799779+0000","last_change":"2024-09-21T12:20:15.309390+0000","last_active":"2024-09-21T12:21:01.799779+0000","last_peered":"2024-09-21T12:21:01.799779+0000","last_clean":"2024-09-21T12:21:01.799779+0000","last_became_active":"2024-09-21T12:20:15.309023+0000","last_became_peered":"2024-09-21T12:20:15.309023+0000","last_unstale":"2024-09-21T12:21:01.799779+0000","last_undegraded":"2024-09-21T12:21:01.799779+0000","last_fullsized":"2024-09-21T12:21:01.799779+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:25.611326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":71,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798284+0000","last_change":"2024-09-21T12:20:17.310727+0000","last_active":"2024-09-21T12:21:01.798284+0000","last_peered":"2024-09-21T12:21:01.798284+0000","last_clean":"2024-09-21T12:21:01.798284+0000","last_became_active":"2024-09-21T12:20:17.310369+0000","last_became_peered":"2024-09-21T12:20:17.310369+0000","last_unstale":"2024-09-21T12:21:01.798284+0000","last_undegraded":"2024-09-21T12:21:01.798284+0000","last_fullsized":"2024-09-21T12:21:01.798284+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:22.563871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.800078+0000","last_change":"2024-09-21T12:20:19.310859+0000","last_active":"2024-09-21T12:21:01.800078+0000","last_peered":"2024-09-21T12:21:01.800078+0000","last_clean":"2024-09-21T12:21:01.800078+0000","last_became_active":"2024-09-21T12:20:19.310575+0000","last_became_peered":"2024-09-21T12:20:19.310575+0000","last_unstale":"2024-09-21T12:21:01.800078+0000","last_undegraded":"2024-09-21T12:21:01.800078+0000","last_fullsized":"2024-09-21T12:21:01.800078+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.082897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797401+0000","last_change":"2024-09-21T12:20:21.313476+0000","last_active":"2024-09-21T12:21:01.797401+0000","last_peered":"2024-09-21T12:21:01.797401+0000","last_clean":"2024-09-21T12:21:01.797401+0000","last_became_active":"2024-09-21T12:20:21.312976+0000","last_became_peered":"2024-09-21T12:20:21.312976+0000","last_unstale":"2024-09-21T12:21:01.797401+0000","last_undegraded":"2024-09-21T12:21:01.797401+0000","last_fullsized":"2024-09-21T12:21:01.797401+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.367953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":55,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760594+0000","last_change":"2024-09-21T12:20:15.299812+0000","last_active":"2024-09-21T12:21:01.760594+0000","last_peered":"2024-09-21T12:21:01.760594+0000","last_clean":"2024-09-21T12:21:01.760594+0000","last_became_active":"2024-09-21T12:20:15.299459+0000","last_became_peered":"2024-09-21T12:20:15.299459+0000","last_unstale":"2024-09-21T12:21:01.760594+0000","last_undegraded":"2024-09-21T12:21:01.760594+0000","last_fullsized":"2024-09-21T12:21:01.760594+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.458091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761351+0000","last_change":"2024-09-21T12:20:17.696423+0000","last_active":"2024-09-21T12:21:01.761351+0000","last_peered":"2024-09-21T12:21:01.761351+0000","last_clean":"2024-09-21T12:21:01.761351+0000","last_became_active":"2024-09-21T12:20:17.695977+0000","last_became_peered":"2024-09-21T12:20:17.695977+0000","last_unstale":"2024-09-21T12:21:01.761351+0000","last_undegraded":"2024-09-21T12:21:01.761351+0000","last_fullsized":"2024-09-21T12:21:01.761351+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:44.740918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":69,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:05.358315+0000","last_change":"2024-09-21T12:20:19.313262+0000","last_active":"2024-09-21T12:21:05.358315+0000","last_peered":"2024-09-21T12:21:05.358315+0000","last_clean":"2024-09-21T12:21:05.358315+0000","last_became_active":"2024-09-21T12:20:19.312403+0000","last_became_peered":"2024-09-21T12:20:19.312403+0000","last_unstale":"2024-09-21T12:21:05.358315+0000","last_undegraded":"2024-09-21T12:21:05.358315+0000","last_fullsized":"2024-09-21T12:21:05.358315+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:43.181150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":75,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761866+0000","last_change":"2024-09-21T12:20:17.305947+0000","last_active":"2024-09-21T12:21:01.761866+0000","last_peered":"2024-09-21T12:21:01.761866+0000","last_clean":"2024-09-21T12:21:01.761866+0000","last_became_active":"2024-09-21T12:20:17.305666+0000","last_became_peered":"2024-09-21T12:20:17.305666+0000","last_unstale":"2024-09-21T12:21:01.761866+0000","last_undegraded":"2024-09-21T12:21:01.761866+0000","last_fullsized":"2024-09-21T12:21:01.761866+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.503390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":52,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798247+0000","last_change":"2024-09-21T12:20:15.296251+0000","last_active":"2024-09-21T12:21:01.798247+0000","last_peered":"2024-09-21T12:21:01.798247+0000","last_clean":"2024-09-21T12:21:01.798247+0000","last_became_active":"2024-09-21T12:20:15.295774+0000","last_became_peered":"2024-09-21T12:20:15.295774+0000","last_unstale":"2024-09-21T12:21:01.798247+0000","last_undegraded":"2024-09-21T12:21:01.798247+0000","last_fullsized":"2024-09-21T12:21:01.798247+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:27.341994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:05.357999+0000","last_change":"2024-09-21T12:20:19.309874+0000","last_active":"2024-09-21T12:21:05.357999+0000","last_peered":"2024-09-21T12:21:05.357999+0000","last_clean":"2024-09-21T12:21:05.357999+0000","last_became_active":"2024-09-21T12:20:19.309606+0000","last_became_peered":"2024-09-21T12:20:19.309606+0000","last_unstale":"2024-09-21T12:21:05.357999+0000","last_undegraded":"2024-09-21T12:21:05.357999+0000","last_fullsized":"2024-09-21T12:21:05.357999+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:37.621763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":43,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798874+0000","last_change":"2024-09-21T12:20:21.310295+0000","last_active":"2024-09-21T12:21:01.798874+0000","last_peered":"2024-09-21T12:21:01.798874+0000","last_clean":"2024-09-21T12:21:01.798874+0000","last_became_active":"2024-09-21T12:20:21.310022+0000","last_became_peered":"2024-09-21T12:20:21.310022+0000","last_unstale":"2024-09-21T12:21:01.798874+0000","last_undegraded":"2024-09-21T12:21:01.798874+0000","last_fullsized":"2024-09-21T12:21:01.798874+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:37.912528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":80,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798048+0000","last_change":"2024-09-21T12:20:17.693550+0000","last_active":"2024-09-21T12:21:01.798048+0000","last_peered":"2024-09-21T12:21:01.798048+0000","last_clean":"2024-09-21T12:21:01.798048+0000","last_became_active":"2024-09-21T12:20:17.693179+0000","last_became_peered":"2024-09-21T12:20:17.693179+0000","last_unstale":"2024-09-21T12:21:01.798048+0000","last_undegraded":"2024-09-21T12:21:01.798048+0000","last_fullsized":"2024-09-21T12:21:01.798048+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:45.358986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":52,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799611+0000","last_change":"2024-09-21T12:20:15.297621+0000","last_active":"2024-09-21T12:21:01.799611+0000","last_peered":"2024-09-21T12:21:01.799611+0000","last_clean":"2024-09-21T12:21:01.799611+0000","last_became_active":"2024-09-21T12:20:15.297340+0000","last_became_peered":"2024-09-21T12:20:15.297340+0000","last_unstale":"2024-09-21T12:21:01.799611+0000","last_undegraded":"2024-09-21T12:21:01.799611+0000","last_fullsized":"2024-09-21T12:21:01.799611+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:17.208372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798975+0000","last_change":"2024-09-21T12:20:19.315065+0000","last_active":"2024-09-21T12:21:01.798975+0000","last_peered":"2024-09-21T12:21:01.798975+0000","last_clean":"2024-09-21T12:21:01.798975+0000","last_became_active":"2024-09-21T12:20:19.314775+0000","last_became_peered":"2024-09-21T12:20:19.314775+0000","last_unstale":"2024-09-21T12:21:01.798975+0000","last_undegraded":"2024-09-21T12:21:01.798975+0000","last_fullsized":"2024-09-21T12:21:01.798975+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:26.384029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799583+0000","last_change":"2024-09-21T12:20:21.314735+0000","last_active":"2024-09-21T12:21:01.799583+0000","last_peered":"2024-09-21T12:21:01.799583+0000","last_clean":"2024-09-21T12:21:01.799583+0000","last_became_active":"2024-09-21T12:20:21.314433+0000","last_became_peered":"2024-09-21T12:20:21.314433+0000","last_unstale":"2024-09-21T12:21:01.799583+0000","last_undegraded":"2024-09-21T12:21:01.799583+0000","last_fullsized":"2024-09-21T12:21:01.799583+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:39.652862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":70,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799651+0000","last_change":"2024-09-21T12:20:17.301460+0000","last_active":"2024-09-21T12:21:01.799651+0000","last_peered":"2024-09-21T12:21:01.799651+0000","last_clean":"2024-09-21T12:21:01.799651+0000","last_became_active":"2024-09-21T12:20:17.301090+0000","last_became_peered":"2024-09-21T12:20:17.301090+0000","last_unstale":"2024-09-21T12:21:01.799651+0000","last_undegraded":"2024-09-21T12:21:01.799651+0000","last_fullsized":"2024-09-21T12:21:01.799651+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:34.835982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":55,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761630+0000","last_change":"2024-09-21T12:20:15.302238+0000","last_active":"2024-09-21T12:21:01.761630+0000","last_peered":"2024-09-21T12:21:01.761630+0000","last_clean":"2024-09-21T12:21:01.761630+0000","last_became_active":"2024-09-21T12:20:15.301904+0000","last_became_peered":"2024-09-21T12:20:15.301904+0000","last_unstale":"2024-09-21T12:21:01.761630+0000","last_undegraded":"2024-09-21T12:21:01.761630+0000","last_fullsized":"2024-09-21T12:21:01.761630+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:33.679116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":64,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:05.357777+0000","last_change":"2024-09-21T12:20:19.311362+0000","last_active":"2024-09-21T12:21:05.357777+0000","last_peered":"2024-09-21T12:21:05.357777+0000","last_clean":"2024-09-21T12:21:05.357777+0000","last_became_active":"2024-09-21T12:20:19.311056+0000","last_became_peered":"2024-09-21T12:20:19.311056+0000","last_unstale":"2024-09-21T12:21:05.357777+0000","last_undegraded":"2024-09-21T12:21:05.357777+0000","last_fullsized":"2024-09-21T12:21:05.357777+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:22.628895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797588+0000","last_change":"2024-09-21T12:20:21.593620+0000","last_active":"2024-09-21T12:21:01.797588+0000","last_peered":"2024-09-21T12:21:01.797588+0000","last_clean":"2024-09-21T12:21:01.797588+0000","last_became_active":"2024-09-21T12:20:21.593367+0000","last_became_peered":"2024-09-21T12:20:21.593367+0000","last_unstale":"2024-09-21T12:21:01.797588+0000","last_undegraded":"2024-09-21T12:21:01.797588+0000","last_fullsized":"2024-09-21T12:21:01.797588+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:20.553670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760808+0000","last_change":"2024-09-21T12:20:17.303245+0000","last_active":"2024-09-21T12:21:01.760808+0000","last_peered":"2024-09-21T12:21:01.760808+0000","last_clean":"2024-09-21T12:21:01.760808+0000","last_became_active":"2024-09-21T12:20:17.302871+0000","last_became_peered":"2024-09-21T12:20:17.302871+0000","last_unstale":"2024-09-21T12:21:01.760808+0000","last_undegraded":"2024-09-21T12:21:01.760808+0000","last_fullsized":"2024-09-21T12:21:01.760808+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.942911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799448+0000","last_change":"2024-09-21T12:20:15.308877+0000","last_active":"2024-09-21T12:21:01.799448+0000","last_peered":"2024-09-21T12:21:01.799448+0000","last_clean":"2024-09-21T12:21:01.799448+0000","last_became_active":"2024-09-21T12:20:15.308518+0000","last_became_peered":"2024-09-21T12:20:15.308518+0000","last_unstale":"2024-09-21T12:21:01.799448+0000","last_undegraded":"2024-09-21T12:21:01.799448+0000","last_fullsized":"2024-09-21T12:21:01.799448+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:23.801832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":69,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:05.357854+0000","last_change":"2024-09-21T12:20:19.307467+0000","last_active":"2024-09-21T12:21:05.357854+0000","last_peered":"2024-09-21T12:21:05.357854+0000","last_clean":"2024-09-21T12:21:05.357854+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:21:05.357854+0000","last_undegraded":"2024-09-21T12:21:05.357854+0000","last_fullsized":"2024-09-21T12:21:05.357854+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:21.225955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760501+0000","last_change":"2024-09-21T12:20:21.316127+0000","last_active":"2024-09-21T12:21:01.760501+0000","last_peered":"2024-09-21T12:21:01.760501+0000","last_clean":"2024-09-21T12:21:01.760501+0000","last_became_active":"2024-09-21T12:20:21.315801+0000","last_became_peered":"2024-09-21T12:20:21.315801+0000","last_unstale":"2024-09-21T12:21:01.760501+0000","last_undegraded":"2024-09-21T12:21:01.760501+0000","last_fullsized":"2024-09-21T12:21:01.760501+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:28.234335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":86,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761471+0000","last_change":"2024-09-21T12:20:17.693585+0000","last_active":"2024-09-21T12:21:01.761471+0000","last_peered":"2024-09-21T12:21:01.761471+0000","last_clean":"2024-09-21T12:21:01.761471+0000","last_became_active":"2024-09-21T12:20:17.693067+0000","last_became_peered":"2024-09-21T12:20:17.693067+0000","last_unstale":"2024-09-21T12:21:01.761471+0000","last_undegraded":"2024-09-21T12:21:01.761471+0000","last_fullsized":"2024-09-21T12:21:01.761471+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:37.693764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761357+0000","last_change":"2024-09-21T12:20:15.308741+0000","last_active":"2024-09-21T12:21:01.761357+0000","last_peered":"2024-09-21T12:21:01.761357+0000","last_clean":"2024-09-21T12:21:01.761357+0000","last_became_active":"2024-09-21T12:20:15.308441+0000","last_became_peered":"2024-09-21T12:20:15.308441+0000","last_unstale":"2024-09-21T12:21:01.761357+0000","last_undegraded":"2024-09-21T12:21:01.761357+0000","last_fullsized":"2024-09-21T12:21:01.761357+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:29.626513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760392+0000","last_change":"2024-09-21T12:20:19.309945+0000","last_active":"2024-09-21T12:21:01.760392+0000","last_peered":"2024-09-21T12:21:01.760392+0000","last_clean":"2024-09-21T12:21:01.760392+0000","last_became_active":"2024-09-21T12:20:19.309598+0000","last_became_peered":"2024-09-21T12:20:19.309598+0000","last_unstale":"2024-09-21T12:21:01.760392+0000","last_undegraded":"2024-09-21T12:21:01.760392+0000","last_fullsized":"2024-09-21T12:21:01.760392+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:31.055740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760388+0000","last_change":"2024-09-21T12:20:21.593173+0000","last_active":"2024-09-21T12:21:01.760388+0000","last_peered":"2024-09-21T12:21:01.760388+0000","last_clean":"2024-09-21T12:21:01.760388+0000","last_became_active":"2024-09-21T12:20:21.592828+0000","last_became_peered":"2024-09-21T12:20:21.592828+0000","last_unstale":"2024-09-21T12:21:01.760388+0000","last_undegraded":"2024-09-21T12:21:01.760388+0000","last_fullsized":"2024-09-21T12:21:01.760388+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.351266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":110,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:05.357795+0000","last_change":"2024-09-21T12:20:17.308049+0000","last_active":"2024-09-21T12:21:05.357795+0000","last_peered":"2024-09-21T12:21:05.357795+0000","last_clean":"2024-09-21T12:21:05.357795+0000","last_became_active":"2024-09-21T12:20:17.307769+0000","last_became_peered":"2024-09-21T12:20:17.307769+0000","last_unstale":"2024-09-21T12:21:05.357795+0000","last_undegraded":"2024-09-21T12:21:05.357795+0000","last_fullsized":"2024-09-21T12:21:05.357795+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:29.183963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.800293+0000","last_change":"2024-09-21T12:20:15.299429+0000","last_active":"2024-09-21T12:21:01.800293+0000","last_peered":"2024-09-21T12:21:01.800293+0000","last_clean":"2024-09-21T12:21:01.800293+0000","last_became_active":"2024-09-21T12:20:15.299143+0000","last_became_peered":"2024-09-21T12:20:15.299143+0000","last_unstale":"2024-09-21T12:21:01.800293+0000","last_undegraded":"2024-09-21T12:21:01.800293+0000","last_fullsized":"2024-09-21T12:21:01.800293+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:42.581678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":64,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798714+0000","last_change":"2024-09-21T12:20:17.298211+0000","last_active":"2024-09-21T12:21:01.798714+0000","last_peered":"2024-09-21T12:21:01.798714+0000","last_clean":"2024-09-21T12:21:01.798714+0000","last_became_active":"2024-09-21T12:20:14.287998+0000","last_became_peered":"2024-09-21T12:20:14.287998+0000","last_unstale":"2024-09-21T12:21:01.798714+0000","last_undegraded":"2024-09-21T12:21:01.798714+0000","last_fullsized":"2024-09-21T12:21:01.798714+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:20:13.272075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:19.171111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011011149999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761160+0000","last_change":"2024-09-21T12:20:19.313478+0000","last_active":"2024-09-21T12:21:01.761160+0000","last_peered":"2024-09-21T12:21:01.761160+0000","last_clean":"2024-09-21T12:21:01.761160+0000","last_became_active":"2024-09-21T12:20:19.312170+0000","last_became_peered":"2024-09-21T12:20:19.312170+0000","last_unstale":"2024-09-21T12:21:01.761160+0000","last_undegraded":"2024-09-21T12:21:01.761160+0000","last_fullsized":"2024-09-21T12:21:01.761160+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.170409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798740+0000","last_change":"2024-09-21T12:20:21.314452+0000","last_active":"2024-09-21T12:21:01.798740+0000","last_peered":"2024-09-21T12:21:01.798740+0000","last_clean":"2024-09-21T12:21:01.798740+0000","last_became_active":"2024-09-21T12:20:21.314126+0000","last_became_peered":"2024-09-21T12:20:21.314126+0000","last_unstale":"2024-09-21T12:21:01.798740+0000","last_undegraded":"2024-09-21T12:21:01.798740+0000","last_fullsized":"2024-09-21T12:21:01.798740+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:44.311699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":77,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761871+0000","last_change":"2024-09-21T12:20:17.297678+0000","last_active":"2024-09-21T12:21:01.761871+0000","last_peered":"2024-09-21T12:21:01.761871+0000","last_clean":"2024-09-21T12:21:01.761871+0000","last_became_active":"2024-09-21T12:20:17.297374+0000","last_became_peered":"2024-09-21T12:20:17.297374+0000","last_unstale":"2024-09-21T12:21:01.761871+0000","last_undegraded":"2024-09-21T12:21:01.761871+0000","last_fullsized":"2024-09-21T12:21:01.761871+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:29.314219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760509+0000","last_change":"2024-09-21T12:20:15.308648+0000","last_active":"2024-09-21T12:21:01.760509+0000","last_peered":"2024-09-21T12:21:01.760509+0000","last_clean":"2024-09-21T12:21:01.760509+0000","last_became_active":"2024-09-21T12:20:15.308356+0000","last_became_peered":"2024-09-21T12:20:15.308356+0000","last_unstale":"2024-09-21T12:21:01.760509+0000","last_undegraded":"2024-09-21T12:21:01.760509+0000","last_fullsized":"2024-09-21T12:21:01.760509+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:25.741492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":63,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797588+0000","last_change":"2024-09-21T12:20:17.297432+0000","last_active":"2024-09-21T12:21:01.797588+0000","last_peered":"2024-09-21T12:21:01.797588+0000","last_clean":"2024-09-21T12:21:01.797588+0000","last_became_active":"2024-09-21T12:20:14.288358+0000","last_became_peered":"2024-09-21T12:20:14.288358+0000","last_unstale":"2024-09-21T12:21:01.797588+0000","last_undegraded":"2024-09-21T12:21:01.797588+0000","last_fullsized":"2024-09-21T12:21:01.797588+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:20:13.272075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:21.220031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019042449999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.796989+0000","last_change":"2024-09-21T12:20:19.310164+0000","last_active":"2024-09-21T12:21:01.796989+0000","last_peered":"2024-09-21T12:21:01.796989+0000","last_clean":"2024-09-21T12:21:01.796989+0000","last_became_active":"2024-09-21T12:20:19.309724+0000","last_became_peered":"2024-09-21T12:20:19.309724+0000","last_unstale":"2024-09-21T12:21:01.796989+0000","last_undegraded":"2024-09-21T12:21:01.796989+0000","last_fullsized":"2024-09-21T12:21:01.796989+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:43.623687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.800070+0000","last_change":"2024-09-21T12:20:21.593267+0000","last_active":"2024-09-21T12:21:01.800070+0000","last_peered":"2024-09-21T12:21:01.800070+0000","last_clean":"2024-09-21T12:21:01.800070+0000","last_became_active":"2024-09-21T12:20:21.592982+0000","last_became_peered":"2024-09-21T12:20:21.592982+0000","last_unstale":"2024-09-21T12:21:01.800070+0000","last_undegraded":"2024-09-21T12:21:01.800070+0000","last_fullsized":"2024-09-21T12:21:01.800070+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:43.824339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":68,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761450+0000","last_change":"2024-09-21T12:20:17.299646+0000","last_active":"2024-09-21T12:21:01.761450+0000","last_peered":"2024-09-21T12:21:01.761450+0000","last_clean":"2024-09-21T12:21:01.761450+0000","last_became_active":"2024-09-21T12:20:17.299325+0000","last_became_peered":"2024-09-21T12:20:17.299325+0000","last_unstale":"2024-09-21T12:21:01.761450+0000","last_undegraded":"2024-09-21T12:21:01.761450+0000","last_fullsized":"2024-09-21T12:21:01.761450+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:22.183786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760648+0000","last_change":"2024-09-21T12:20:15.302177+0000","last_active":"2024-09-21T12:21:01.760648+0000","last_peered":"2024-09-21T12:21:01.760648+0000","last_clean":"2024-09-21T12:21:01.760648+0000","last_became_active":"2024-09-21T12:20:15.301859+0000","last_became_peered":"2024-09-21T12:20:15.301859+0000","last_unstale":"2024-09-21T12:21:01.760648+0000","last_undegraded":"2024-09-21T12:21:01.760648+0000","last_fullsized":"2024-09-21T12:21:01.760648+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.505232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":115,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:10.100059+0000","last_change":"2024-09-21T12:20:17.693865+0000","last_active":"2024-09-21T12:21:10.100059+0000","last_peered":"2024-09-21T12:21:10.100059+0000","last_clean":"2024-09-21T12:21:10.100059+0000","last_became_active":"2024-09-21T12:20:14.287543+0000","last_became_peered":"2024-09-21T12:20:14.287543+0000","last_unstale":"2024-09-21T12:21:10.100059+0000","last_undegraded":"2024-09-21T12:21:10.100059+0000","last_fullsized":"2024-09-21T12:21:10.100059+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:20:13.272075+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:21.111464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":51,"num_read_kb":46,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799083+0000","last_change":"2024-09-21T12:20:19.314574+0000","last_active":"2024-09-21T12:21:01.799083+0000","last_peered":"2024-09-21T12:21:01.799083+0000","last_clean":"2024-09-21T12:21:01.799083+0000","last_became_active":"2024-09-21T12:20:19.314217+0000","last_became_peered":"2024-09-21T12:20:19.314217+0000","last_unstale":"2024-09-21T12:21:01.799083+0000","last_undegraded":"2024-09-21T12:21:01.799083+0000","last_fullsized":"2024-09-21T12:21:01.799083+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:23.172823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761038+0000","last_change":"2024-09-21T12:20:21.311501+0000","last_active":"2024-09-21T12:21:01.761038+0000","last_peered":"2024-09-21T12:21:01.761038+0000","last_clean":"2024-09-21T12:21:01.761038+0000","last_became_active":"2024-09-21T12:20:21.311188+0000","last_became_peered":"2024-09-21T12:20:21.311188+0000","last_unstale":"2024-09-21T12:21:01.761038+0000","last_undegraded":"2024-09-21T12:21:01.761038+0000","last_fullsized":"2024-09-21T12:21:01.761038+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:27.755252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"66'16","reported_seq":84,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:05.358329+0000","last_change":"2024-09-21T12:20:17.696268+0000","last_active":"2024-09-21T12:21:05.358329+0000","last_peered":"2024-09-21T12:21:05.358329+0000","last_clean":"2024-09-21T12:21:05.358329+0000","last_became_active":"2024-09-21T12:20:17.695940+0000","last_became_peered":"2024-09-21T12:20:17.695940+0000","last_unstale":"2024-09-21T12:21:05.358329+0000","last_undegraded":"2024-09-21T12:21:05.358329+0000","last_fullsized":"2024-09-21T12:21:05.358329+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:39.599877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":58,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799244+0000","last_change":"2024-09-21T12:20:15.299151+0000","last_active":"2024-09-21T12:21:01.799244+0000","last_peered":"2024-09-21T12:21:01.799244+0000","last_clean":"2024-09-21T12:21:01.799244+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:21:01.799244+0000","last_undegraded":"2024-09-21T12:21:01.799244+0000","last_fullsized":"2024-09-21T12:21:01.799244+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:16.205349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":98,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799314+0000","last_change":"2024-09-21T12:20:24.850239+0000","last_active":"2024-09-21T12:21:01.799314+0000","last_peered":"2024-09-21T12:21:01.799314+0000","last_clean":"2024-09-21T12:21:01.799314+0000","last_became_active":"2024-09-21T12:20:06.349670+0000","last_became_peered":"2024-09-21T12:20:06.349670+0000","last_unstale":"2024-09-21T12:21:01.799314+0000","last_undegraded":"2024-09-21T12:21:01.799314+0000","last_fullsized":"2024-09-21T12:21:01.799314+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T12:20:24.850157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:20:24.850157+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:53.046478+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799933+0000","last_change":"2024-09-21T12:20:19.310280+0000","last_active":"2024-09-21T12:21:01.799933+0000","last_peered":"2024-09-21T12:21:01.799933+0000","last_clean":"2024-09-21T12:21:01.799933+0000","last_became_active":"2024-09-21T12:20:19.309859+0000","last_became_peered":"2024-09-21T12:20:19.309859+0000","last_unstale":"2024-09-21T12:21:01.799933+0000","last_undegraded":"2024-09-21T12:21:01.799933+0000","last_fullsized":"2024-09-21T12:21:01.799933+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:43.850206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797865+0000","last_change":"2024-09-21T12:20:21.311690+0000","last_active":"2024-09-21T12:21:01.797865+0000","last_peered":"2024-09-21T12:21:01.797865+0000","last_clean":"2024-09-21T12:21:01.797865+0000","last_became_active":"2024-09-21T12:20:21.311194+0000","last_became_peered":"2024-09-21T12:20:21.311194+0000","last_unstale":"2024-09-21T12:21:01.797865+0000","last_undegraded":"2024-09-21T12:21:01.797865+0000","last_fullsized":"2024-09-21T12:21:01.797865+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:25.752002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":68,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.800772+0000","last_change":"2024-09-21T12:20:17.310299+0000","last_active":"2024-09-21T12:21:01.800772+0000","last_peered":"2024-09-21T12:21:01.800772+0000","last_clean":"2024-09-21T12:21:01.800772+0000","last_became_active":"2024-09-21T12:20:17.309930+0000","last_became_peered":"2024-09-21T12:20:17.309930+0000","last_unstale":"2024-09-21T12:21:01.800772+0000","last_undegraded":"2024-09-21T12:21:01.800772+0000","last_fullsized":"2024-09-21T12:21:01.800772+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:17.883258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.800123+0000","last_change":"2024-09-21T12:20:15.310346+0000","last_active":"2024-09-21T12:21:01.800123+0000","last_peered":"2024-09-21T12:21:01.800123+0000","last_clean":"2024-09-21T12:21:01.800123+0000","last_became_active":"2024-09-21T12:20:15.310048+0000","last_became_peered":"2024-09-21T12:20:15.310048+0000","last_unstale":"2024-09-21T12:21:01.800123+0000","last_undegraded":"2024-09-21T12:21:01.800123+0000","last_fullsized":"2024-09-21T12:21:01.800123+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:31.701321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797742+0000","last_change":"2024-09-21T12:20:19.309554+0000","last_active":"2024-09-21T12:21:01.797742+0000","last_peered":"2024-09-21T12:21:01.797742+0000","last_clean":"2024-09-21T12:21:01.797742+0000","last_became_active":"2024-09-21T12:20:19.309258+0000","last_became_peered":"2024-09-21T12:20:19.309258+0000","last_unstale":"2024-09-21T12:21:01.797742+0000","last_undegraded":"2024-09-21T12:21:01.797742+0000","last_fullsized":"2024-09-21T12:21:01.797742+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:44.940960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797367+0000","last_change":"2024-09-21T12:20:21.593145+0000","last_active":"2024-09-21T12:21:01.797367+0000","last_peered":"2024-09-21T12:21:01.797367+0000","last_clean":"2024-09-21T12:21:01.797367+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:21:01.797367+0000","last_undegraded":"2024-09-21T12:21:01.797367+0000","last_fullsized":"2024-09-21T12:21:01.797367+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.236407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":77,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.800972+0000","last_change":"2024-09-21T12:20:17.310292+0000","last_active":"2024-09-21T12:21:01.800972+0000","last_peered":"2024-09-21T12:21:01.800972+0000","last_clean":"2024-09-21T12:21:01.800972+0000","last_became_active":"2024-09-21T12:20:17.309828+0000","last_became_peered":"2024-09-21T12:20:17.309828+0000","last_unstale":"2024-09-21T12:21:01.800972+0000","last_undegraded":"2024-09-21T12:21:01.800972+0000","last_fullsized":"2024-09-21T12:21:01.800972+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:44.016276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760930+0000","last_change":"2024-09-21T12:20:15.302137+0000","last_active":"2024-09-21T12:21:01.760930+0000","last_peered":"2024-09-21T12:21:01.760930+0000","last_clean":"2024-09-21T12:21:01.760930+0000","last_became_active":"2024-09-21T12:20:15.301851+0000","last_became_peered":"2024-09-21T12:20:15.301851+0000","last_unstale":"2024-09-21T12:21:01.760930+0000","last_undegraded":"2024-09-21T12:21:01.760930+0000","last_fullsized":"2024-09-21T12:21:01.760930+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.661202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761056+0000","last_change":"2024-09-21T12:20:19.312912+0000","last_active":"2024-09-21T12:21:01.761056+0000","last_peered":"2024-09-21T12:21:01.761056+0000","last_clean":"2024-09-21T12:21:01.761056+0000","last_became_active":"2024-09-21T12:20:19.312301+0000","last_became_peered":"2024-09-21T12:20:19.312301+0000","last_unstale":"2024-09-21T12:21:01.761056+0000","last_undegraded":"2024-09-21T12:21:01.761056+0000","last_fullsized":"2024-09-21T12:21:01.761056+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:42.511905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797148+0000","last_change":"2024-09-21T12:20:21.313585+0000","last_active":"2024-09-21T12:21:01.797148+0000","last_peered":"2024-09-21T12:21:01.797148+0000","last_clean":"2024-09-21T12:21:01.797148+0000","last_became_active":"2024-09-21T12:20:21.313254+0000","last_became_peered":"2024-09-21T12:20:21.313254+0000","last_unstale":"2024-09-21T12:21:01.797148+0000","last_undegraded":"2024-09-21T12:21:01.797148+0000","last_fullsized":"2024-09-21T12:21:01.797148+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.442191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":64,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.800202+0000","last_change":"2024-09-21T12:20:17.307651+0000","last_active":"2024-09-21T12:21:01.800202+0000","last_peered":"2024-09-21T12:21:01.800202+0000","last_clean":"2024-09-21T12:21:01.800202+0000","last_became_active":"2024-09-21T12:20:17.307317+0000","last_became_peered":"2024-09-21T12:20:17.307317+0000","last_unstale":"2024-09-21T12:21:01.800202+0000","last_undegraded":"2024-09-21T12:21:01.800202+0000","last_fullsized":"2024-09-21T12:21:01.800202+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:37.610294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":52,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798603+0000","last_change":"2024-09-21T12:20:15.298107+0000","last_active":"2024-09-21T12:21:01.798603+0000","last_peered":"2024-09-21T12:21:01.798603+0000","last_clean":"2024-09-21T12:21:01.798603+0000","last_became_active":"2024-09-21T12:20:15.297819+0000","last_became_peered":"2024-09-21T12:20:15.297819+0000","last_unstale":"2024-09-21T12:21:01.798603+0000","last_undegraded":"2024-09-21T12:21:01.798603+0000","last_fullsized":"2024-09-21T12:21:01.798603+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:38.788978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":66,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:05.357778+0000","last_change":"2024-09-21T12:20:19.308246+0000","last_active":"2024-09-21T12:21:05.357778+0000","last_peered":"2024-09-21T12:21:05.357778+0000","last_clean":"2024-09-21T12:21:05.357778+0000","last_became_active":"2024-09-21T12:20:19.307924+0000","last_became_peered":"2024-09-21T12:20:19.307924+0000","last_unstale":"2024-09-21T12:21:05.357778+0000","last_undegraded":"2024-09-21T12:21:05.357778+0000","last_fullsized":"2024-09-21T12:21:05.357778+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:33.476030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798009+0000","last_change":"2024-09-21T12:20:21.313250+0000","last_active":"2024-09-21T12:21:01.798009+0000","last_peered":"2024-09-21T12:21:01.798009+0000","last_clean":"2024-09-21T12:21:01.798009+0000","last_became_active":"2024-09-21T12:20:21.312954+0000","last_became_peered":"2024-09-21T12:20:21.312954+0000","last_unstale":"2024-09-21T12:21:01.798009+0000","last_undegraded":"2024-09-21T12:21:01.798009+0000","last_fullsized":"2024-09-21T12:21:01.798009+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.513060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":66,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761383+0000","last_change":"2024-09-21T12:20:17.299948+0000","last_active":"2024-09-21T12:21:01.761383+0000","last_peered":"2024-09-21T12:21:01.761383+0000","last_clean":"2024-09-21T12:21:01.761383+0000","last_became_active":"2024-09-21T12:20:17.299653+0000","last_became_peered":"2024-09-21T12:20:17.299653+0000","last_unstale":"2024-09-21T12:21:01.761383+0000","last_undegraded":"2024-09-21T12:21:01.761383+0000","last_fullsized":"2024-09-21T12:21:01.761383+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:16.539847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798378+0000","last_change":"2024-09-21T12:20:15.302587+0000","last_active":"2024-09-21T12:21:01.798378+0000","last_peered":"2024-09-21T12:21:01.798378+0000","last_clean":"2024-09-21T12:21:01.798378+0000","last_became_active":"2024-09-21T12:20:15.302262+0000","last_became_peered":"2024-09-21T12:20:15.302262+0000","last_unstale":"2024-09-21T12:21:01.798378+0000","last_undegraded":"2024-09-21T12:21:01.798378+0000","last_fullsized":"2024-09-21T12:21:01.798378+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:32.093775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.796917+0000","last_change":"2024-09-21T12:20:19.308484+0000","last_active":"2024-09-21T12:21:01.796917+0000","last_peered":"2024-09-21T12:21:01.796917+0000","last_clean":"2024-09-21T12:21:01.796917+0000","last_became_active":"2024-09-21T12:20:19.308048+0000","last_became_peered":"2024-09-21T12:20:19.308048+0000","last_unstale":"2024-09-21T12:21:01.796917+0000","last_undegraded":"2024-09-21T12:21:01.796917+0000","last_fullsized":"2024-09-21T12:21:01.796917+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:34.470562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760554+0000","last_change":"2024-09-21T12:20:21.311799+0000","last_active":"2024-09-21T12:21:01.760554+0000","last_peered":"2024-09-21T12:21:01.760554+0000","last_clean":"2024-09-21T12:21:01.760554+0000","last_became_active":"2024-09-21T12:20:21.311493+0000","last_became_peered":"2024-09-21T12:20:21.311493+0000","last_unstale":"2024-09-21T12:21:01.760554+0000","last_undegraded":"2024-09-21T12:21:01.760554+0000","last_fullsized":"2024-09-21T12:21:01.760554+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:40.676620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":81,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760599+0000","last_change":"2024-09-21T12:20:17.693677+0000","last_active":"2024-09-21T12:21:01.760599+0000","last_peered":"2024-09-21T12:21:01.760599+0000","last_clean":"2024-09-21T12:21:01.760599+0000","last_became_active":"2024-09-21T12:20:17.692867+0000","last_became_peered":"2024-09-21T12:20:17.692867+0000","last_unstale":"2024-09-21T12:21:01.760599+0000","last_undegraded":"2024-09-21T12:21:01.760599+0000","last_fullsized":"2024-09-21T12:21:01.760599+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.747644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.800431+0000","last_change":"2024-09-21T12:20:15.300854+0000","last_active":"2024-09-21T12:21:01.800431+0000","last_peered":"2024-09-21T12:21:01.800431+0000","last_clean":"2024-09-21T12:21:01.800431+0000","last_became_active":"2024-09-21T12:20:15.299333+0000","last_became_peered":"2024-09-21T12:20:15.299333+0000","last_unstale":"2024-09-21T12:21:01.800431+0000","last_undegraded":"2024-09-21T12:21:01.800431+0000","last_fullsized":"2024-09-21T12:21:01.800431+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:34.912500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797216+0000","last_change":"2024-09-21T12:20:19.307316+0000","last_active":"2024-09-21T12:21:01.797216+0000","last_peered":"2024-09-21T12:21:01.797216+0000","last_clean":"2024-09-21T12:21:01.797216+0000","last_became_active":"2024-09-21T12:20:19.306952+0000","last_became_peered":"2024-09-21T12:20:19.306952+0000","last_unstale":"2024-09-21T12:21:01.797216+0000","last_undegraded":"2024-09-21T12:21:01.797216+0000","last_fullsized":"2024-09-21T12:21:01.797216+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:47.444300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799177+0000","last_change":"2024-09-21T12:20:21.314891+0000","last_active":"2024-09-21T12:21:01.799177+0000","last_peered":"2024-09-21T12:21:01.799177+0000","last_clean":"2024-09-21T12:21:01.799177+0000","last_became_active":"2024-09-21T12:20:21.314545+0000","last_became_peered":"2024-09-21T12:20:21.314545+0000","last_unstale":"2024-09-21T12:21:01.799177+0000","last_undegraded":"2024-09-21T12:21:01.799177+0000","last_fullsized":"2024-09-21T12:21:01.799177+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:21.854833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":72,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799889+0000","last_change":"2024-09-21T12:20:17.306090+0000","last_active":"2024-09-21T12:21:01.799889+0000","last_peered":"2024-09-21T12:21:01.799889+0000","last_clean":"2024-09-21T12:21:01.799889+0000","last_became_active":"2024-09-21T12:20:17.305661+0000","last_became_peered":"2024-09-21T12:20:17.305661+0000","last_unstale":"2024-09-21T12:21:01.799889+0000","last_undegraded":"2024-09-21T12:21:01.799889+0000","last_fullsized":"2024-09-21T12:21:01.799889+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:31.017952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.800776+0000","last_change":"2024-09-21T12:20:15.305876+0000","last_active":"2024-09-21T12:21:01.800776+0000","last_peered":"2024-09-21T12:21:01.800776+0000","last_clean":"2024-09-21T12:21:01.800776+0000","last_became_active":"2024-09-21T12:20:15.305545+0000","last_became_peered":"2024-09-21T12:20:15.305545+0000","last_unstale":"2024-09-21T12:21:01.800776+0000","last_undegraded":"2024-09-21T12:21:01.800776+0000","last_fullsized":"2024-09-21T12:21:01.800776+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:19.227777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.796678+0000","last_change":"2024-09-21T12:20:19.312986+0000","last_active":"2024-09-21T12:21:01.796678+0000","last_peered":"2024-09-21T12:21:01.796678+0000","last_clean":"2024-09-21T12:21:01.796678+0000","last_became_active":"2024-09-21T12:20:19.312572+0000","last_became_peered":"2024-09-21T12:20:19.312572+0000","last_unstale":"2024-09-21T12:21:01.796678+0000","last_undegraded":"2024-09-21T12:21:01.796678+0000","last_fullsized":"2024-09-21T12:21:01.796678+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:20.301869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797401+0000","last_change":"2024-09-21T12:20:21.317550+0000","last_active":"2024-09-21T12:21:01.797401+0000","last_peered":"2024-09-21T12:21:01.797401+0000","last_clean":"2024-09-21T12:21:01.797401+0000","last_became_active":"2024-09-21T12:20:21.317223+0000","last_became_peered":"2024-09-21T12:20:21.317223+0000","last_unstale":"2024-09-21T12:21:01.797401+0000","last_undegraded":"2024-09-21T12:21:01.797401+0000","last_fullsized":"2024-09-21T12:21:01.797401+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:25.406157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":75,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797194+0000","last_change":"2024-09-21T12:20:17.693302+0000","last_active":"2024-09-21T12:21:01.797194+0000","last_peered":"2024-09-21T12:21:01.797194+0000","last_clean":"2024-09-21T12:21:01.797194+0000","last_became_active":"2024-09-21T12:20:17.692870+0000","last_became_peered":"2024-09-21T12:20:17.692870+0000","last_unstale":"2024-09-21T12:21:01.797194+0000","last_undegraded":"2024-09-21T12:21:01.797194+0000","last_fullsized":"2024-09-21T12:21:01.797194+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.102470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.800720+0000","last_change":"2024-09-21T12:20:15.302366+0000","last_active":"2024-09-21T12:21:01.800720+0000","last_peered":"2024-09-21T12:21:01.800720+0000","last_clean":"2024-09-21T12:21:01.800720+0000","last_became_active":"2024-09-21T12:20:15.302038+0000","last_became_peered":"2024-09-21T12:20:15.302038+0000","last_unstale":"2024-09-21T12:21:01.800720+0000","last_undegraded":"2024-09-21T12:21:01.800720+0000","last_fullsized":"2024-09-21T12:21:01.800720+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:29.184795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:05.358056+0000","last_change":"2024-09-21T12:20:19.310188+0000","last_active":"2024-09-21T12:21:05.358056+0000","last_peered":"2024-09-21T12:21:05.358056+0000","last_clean":"2024-09-21T12:21:05.358056+0000","last_became_active":"2024-09-21T12:20:19.309708+0000","last_became_peered":"2024-09-21T12:20:19.309708+0000","last_unstale":"2024-09-21T12:21:05.358056+0000","last_undegraded":"2024-09-21T12:21:05.358056+0000","last_fullsized":"2024-09-21T12:21:05.358056+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:44.922935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798423+0000","last_change":"2024-09-21T12:20:21.592893+0000","last_active":"2024-09-21T12:21:01.798423+0000","last_peered":"2024-09-21T12:21:01.798423+0000","last_clean":"2024-09-21T12:21:01.798423+0000","last_became_active":"2024-09-21T12:20:21.592576+0000","last_became_peered":"2024-09-21T12:20:21.592576+0000","last_unstale":"2024-09-21T12:21:01.798423+0000","last_undegraded":"2024-09-21T12:21:01.798423+0000","last_fullsized":"2024-09-21T12:21:01.798423+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:45.792911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760521+0000","last_change":"2024-09-21T12:20:15.303770+0000","last_active":"2024-09-21T12:21:01.760521+0000","last_peered":"2024-09-21T12:21:01.760521+0000","last_clean":"2024-09-21T12:21:01.760521+0000","last_became_active":"2024-09-21T12:20:15.303480+0000","last_became_peered":"2024-09-21T12:20:15.303480+0000","last_unstale":"2024-09-21T12:21:01.760521+0000","last_undegraded":"2024-09-21T12:21:01.760521+0000","last_fullsized":"2024-09-21T12:21:01.760521+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:14.876724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":57,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799528+0000","last_change":"2024-09-21T12:20:17.694977+0000","last_active":"2024-09-21T12:21:01.799528+0000","last_peered":"2024-09-21T12:21:01.799528+0000","last_clean":"2024-09-21T12:21:01.799528+0000","last_became_active":"2024-09-21T12:20:17.694628+0000","last_became_peered":"2024-09-21T12:20:17.694628+0000","last_unstale":"2024-09-21T12:21:01.799528+0000","last_undegraded":"2024-09-21T12:21:01.799528+0000","last_fullsized":"2024-09-21T12:21:01.799528+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.288937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797934+0000","last_change":"2024-09-21T12:20:19.314432+0000","last_active":"2024-09-21T12:21:01.797934+0000","last_peered":"2024-09-21T12:21:01.797934+0000","last_clean":"2024-09-21T12:21:01.797934+0000","last_became_active":"2024-09-21T12:20:19.313882+0000","last_became_peered":"2024-09-21T12:20:19.313882+0000","last_unstale":"2024-09-21T12:21:01.797934+0000","last_undegraded":"2024-09-21T12:21:01.797934+0000","last_fullsized":"2024-09-21T12:21:01.797934+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:35.453804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.800106+0000","last_change":"2024-09-21T12:20:21.594311+0000","last_active":"2024-09-21T12:21:01.800106+0000","last_peered":"2024-09-21T12:21:01.800106+0000","last_clean":"2024-09-21T12:21:01.800106+0000","last_became_active":"2024-09-21T12:20:21.594048+0000","last_became_peered":"2024-09-21T12:20:21.594048+0000","last_unstale":"2024-09-21T12:21:01.800106+0000","last_undegraded":"2024-09-21T12:21:01.800106+0000","last_fullsized":"2024-09-21T12:21:01.800106+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:38.347489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":66,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761540+0000","last_change":"2024-09-21T12:20:17.696279+0000","last_active":"2024-09-21T12:21:01.761540+0000","last_peered":"2024-09-21T12:21:01.761540+0000","last_clean":"2024-09-21T12:21:01.761540+0000","last_became_active":"2024-09-21T12:20:17.695898+0000","last_became_peered":"2024-09-21T12:20:17.695898+0000","last_unstale":"2024-09-21T12:21:01.761540+0000","last_undegraded":"2024-09-21T12:21:01.761540+0000","last_fullsized":"2024-09-21T12:21:01.761540+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:21.926785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798290+0000","last_change":"2024-09-21T12:20:15.301560+0000","last_active":"2024-09-21T12:21:01.798290+0000","last_peered":"2024-09-21T12:21:01.798290+0000","last_clean":"2024-09-21T12:21:01.798290+0000","last_became_active":"2024-09-21T12:20:15.300722+0000","last_became_peered":"2024-09-21T12:20:15.300722+0000","last_unstale":"2024-09-21T12:21:01.798290+0000","last_undegraded":"2024-09-21T12:21:01.798290+0000","last_fullsized":"2024-09-21T12:21:01.798290+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.142850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":44,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799393+0000","last_change":"2024-09-21T12:20:19.312248+0000","last_active":"2024-09-21T12:21:01.799393+0000","last_peered":"2024-09-21T12:21:01.799393+0000","last_clean":"2024-09-21T12:21:01.799393+0000","last_became_active":"2024-09-21T12:20:19.311947+0000","last_became_peered":"2024-09-21T12:20:19.311947+0000","last_unstale":"2024-09-21T12:21:01.799393+0000","last_undegraded":"2024-09-21T12:21:01.799393+0000","last_fullsized":"2024-09-21T12:21:01.799393+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.388400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797355+0000","last_change":"2024-09-21T12:20:21.314554+0000","last_active":"2024-09-21T12:21:01.797355+0000","last_peered":"2024-09-21T12:21:01.797355+0000","last_clean":"2024-09-21T12:21:01.797355+0000","last_became_active":"2024-09-21T12:20:21.314277+0000","last_became_peered":"2024-09-21T12:20:21.314277+0000","last_unstale":"2024-09-21T12:21:01.797355+0000","last_undegraded":"2024-09-21T12:21:01.797355+0000","last_fullsized":"2024-09-21T12:21:01.797355+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.219191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":66,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799278+0000","last_change":"2024-09-21T12:20:17.694904+0000","last_active":"2024-09-21T12:21:01.799278+0000","last_peered":"2024-09-21T12:21:01.799278+0000","last_clean":"2024-09-21T12:21:01.799278+0000","last_became_active":"2024-09-21T12:20:17.694568+0000","last_became_peered":"2024-09-21T12:20:17.694568+0000","last_unstale":"2024-09-21T12:21:01.799278+0000","last_undegraded":"2024-09-21T12:21:01.799278+0000","last_fullsized":"2024-09-21T12:21:01.799278+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:24.206067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761045+0000","last_change":"2024-09-21T12:20:15.297853+0000","last_active":"2024-09-21T12:21:01.761045+0000","last_peered":"2024-09-21T12:21:01.761045+0000","last_clean":"2024-09-21T12:21:01.761045+0000","last_became_active":"2024-09-21T12:20:15.297519+0000","last_became_peered":"2024-09-21T12:20:15.297519+0000","last_unstale":"2024-09-21T12:21:01.761045+0000","last_undegraded":"2024-09-21T12:21:01.761045+0000","last_fullsized":"2024-09-21T12:21:01.761045+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:29.814404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":64,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:05.357792+0000","last_change":"2024-09-21T12:20:19.305689+0000","last_active":"2024-09-21T12:21:05.357792+0000","last_peered":"2024-09-21T12:21:05.357792+0000","last_clean":"2024-09-21T12:21:05.357792+0000","last_became_active":"2024-09-21T12:20:19.305364+0000","last_became_peered":"2024-09-21T12:20:19.305364+0000","last_unstale":"2024-09-21T12:21:05.357792+0000","last_undegraded":"2024-09-21T12:21:05.357792+0000","last_fullsized":"2024-09-21T12:21:05.357792+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.650981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798482+0000","last_change":"2024-09-21T12:20:21.315079+0000","last_active":"2024-09-21T12:21:01.798482+0000","last_peered":"2024-09-21T12:21:01.798482+0000","last_clean":"2024-09-21T12:21:01.798482+0000","last_became_active":"2024-09-21T12:20:21.314066+0000","last_became_peered":"2024-09-21T12:20:21.314066+0000","last_unstale":"2024-09-21T12:21:01.798482+0000","last_undegraded":"2024-09-21T12:21:01.798482+0000","last_fullsized":"2024-09-21T12:21:01.798482+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:27.528692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":63,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799346+0000","last_change":"2024-09-21T12:20:17.693562+0000","last_active":"2024-09-21T12:21:01.799346+0000","last_peered":"2024-09-21T12:21:01.799346+0000","last_clean":"2024-09-21T12:21:01.799346+0000","last_became_active":"2024-09-21T12:20:17.693209+0000","last_became_peered":"2024-09-21T12:20:17.693209+0000","last_unstale":"2024-09-21T12:21:01.799346+0000","last_undegraded":"2024-09-21T12:21:01.799346+0000","last_fullsized":"2024-09-21T12:21:01.799346+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:25.923701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.799435+0000","last_change":"2024-09-21T12:20:15.308767+0000","last_active":"2024-09-21T12:21:01.799435+0000","last_peered":"2024-09-21T12:21:01.799435+0000","last_clean":"2024-09-21T12:21:01.799435+0000","last_became_active":"2024-09-21T12:20:15.308429+0000","last_became_peered":"2024-09-21T12:20:15.308429+0000","last_unstale":"2024-09-21T12:21:01.799435+0000","last_undegraded":"2024-09-21T12:21:01.799435+0000","last_fullsized":"2024-09-21T12:21:01.799435+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:15.014581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":66,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:05.358290+0000","last_change":"2024-09-21T12:20:19.314884+0000","last_active":"2024-09-21T12:21:05.358290+0000","last_peered":"2024-09-21T12:21:05.358290+0000","last_clean":"2024-09-21T12:21:05.358290+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:21:05.358290+0000","last_undegraded":"2024-09-21T12:21:05.358290+0000","last_fullsized":"2024-09-21T12:21:05.358290+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:29.894782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761225+0000","last_change":"2024-09-21T12:20:21.309936+0000","last_active":"2024-09-21T12:21:01.761225+0000","last_peered":"2024-09-21T12:21:01.761225+0000","last_clean":"2024-09-21T12:21:01.761225+0000","last_became_active":"2024-09-21T12:20:21.309385+0000","last_became_peered":"2024-09-21T12:20:21.309385+0000","last_unstale":"2024-09-21T12:21:01.761225+0000","last_undegraded":"2024-09-21T12:21:01.761225+0000","last_fullsized":"2024-09-21T12:21:01.761225+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:47.499218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":68,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.800003+0000","last_change":"2024-09-21T12:20:17.304948+0000","last_active":"2024-09-21T12:21:01.800003+0000","last_peered":"2024-09-21T12:21:01.800003+0000","last_clean":"2024-09-21T12:21:01.800003+0000","last_became_active":"2024-09-21T12:20:17.303763+0000","last_became_peered":"2024-09-21T12:20:17.303763+0000","last_unstale":"2024-09-21T12:21:01.800003+0000","last_undegraded":"2024-09-21T12:21:01.800003+0000","last_fullsized":"2024-09-21T12:21:01.800003+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:31.523086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.800458+0000","last_change":"2024-09-21T12:20:15.298500+0000","last_active":"2024-09-21T12:21:01.800458+0000","last_peered":"2024-09-21T12:21:01.800458+0000","last_clean":"2024-09-21T12:21:01.800458+0000","last_became_active":"2024-09-21T12:20:15.298198+0000","last_became_peered":"2024-09-21T12:20:15.298198+0000","last_unstale":"2024-09-21T12:21:01.800458+0000","last_undegraded":"2024-09-21T12:21:01.800458+0000","last_fullsized":"2024-09-21T12:21:01.800458+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:36.549556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760710+0000","last_change":"2024-09-21T12:20:19.315714+0000","last_active":"2024-09-21T12:21:01.760710+0000","last_peered":"2024-09-21T12:21:01.760710+0000","last_clean":"2024-09-21T12:21:01.760710+0000","last_became_active":"2024-09-21T12:20:19.315418+0000","last_became_peered":"2024-09-21T12:20:19.315418+0000","last_unstale":"2024-09-21T12:21:01.760710+0000","last_undegraded":"2024-09-21T12:21:01.760710+0000","last_fullsized":"2024-09-21T12:21:01.760710+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:38.005096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797490+0000","last_change":"2024-09-21T12:20:21.310344+0000","last_active":"2024-09-21T12:21:01.797490+0000","last_peered":"2024-09-21T12:21:01.797490+0000","last_clean":"2024-09-21T12:21:01.797490+0000","last_became_active":"2024-09-21T12:20:21.310053+0000","last_became_peered":"2024-09-21T12:20:21.310053+0000","last_unstale":"2024-09-21T12:21:01.797490+0000","last_undegraded":"2024-09-21T12:21:01.797490+0000","last_fullsized":"2024-09-21T12:21:01.797490+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:42.963350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":66,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760846+0000","last_change":"2024-09-21T12:20:17.299367+0000","last_active":"2024-09-21T12:21:01.760846+0000","last_peered":"2024-09-21T12:21:01.760846+0000","last_clean":"2024-09-21T12:21:01.760846+0000","last_became_active":"2024-09-21T12:20:17.299015+0000","last_became_peered":"2024-09-21T12:20:17.299015+0000","last_unstale":"2024-09-21T12:21:01.760846+0000","last_undegraded":"2024-09-21T12:21:01.760846+0000","last_fullsized":"2024-09-21T12:21:01.760846+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:42.174386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.800658+0000","last_change":"2024-09-21T12:20:15.303462+0000","last_active":"2024-09-21T12:21:01.800658+0000","last_peered":"2024-09-21T12:21:01.800658+0000","last_clean":"2024-09-21T12:21:01.800658+0000","last_became_active":"2024-09-21T12:20:15.300554+0000","last_became_peered":"2024-09-21T12:20:15.300554+0000","last_unstale":"2024-09-21T12:21:01.800658+0000","last_undegraded":"2024-09-21T12:21:01.800658+0000","last_fullsized":"2024-09-21T12:21:01.800658+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:22.850479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":44,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798353+0000","last_change":"2024-09-21T12:20:19.311327+0000","last_active":"2024-09-21T12:21:01.798353+0000","last_peered":"2024-09-21T12:21:01.798353+0000","last_clean":"2024-09-21T12:21:01.798353+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:21:01.798353+0000","last_undegraded":"2024-09-21T12:21:01.798353+0000","last_fullsized":"2024-09-21T12:21:01.798353+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:20.603464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760220+0000","last_change":"2024-09-21T12:20:21.308003+0000","last_active":"2024-09-21T12:21:01.760220+0000","last_peered":"2024-09-21T12:21:01.760220+0000","last_clean":"2024-09-21T12:21:01.760220+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:21:01.760220+0000","last_undegraded":"2024-09-21T12:21:01.760220+0000","last_fullsized":"2024-09-21T12:21:01.760220+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:41.631005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":67,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798016+0000","last_change":"2024-09-21T12:20:17.695358+0000","last_active":"2024-09-21T12:21:01.798016+0000","last_peered":"2024-09-21T12:21:01.798016+0000","last_clean":"2024-09-21T12:21:01.798016+0000","last_became_active":"2024-09-21T12:20:17.694794+0000","last_became_peered":"2024-09-21T12:20:17.694794+0000","last_unstale":"2024-09-21T12:21:01.798016+0000","last_undegraded":"2024-09-21T12:21:01.798016+0000","last_fullsized":"2024-09-21T12:21:01.798016+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:25.305944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798031+0000","last_change":"2024-09-21T12:20:15.296285+0000","last_active":"2024-09-21T12:21:01.798031+0000","last_peered":"2024-09-21T12:21:01.798031+0000","last_clean":"2024-09-21T12:21:01.798031+0000","last_became_active":"2024-09-21T12:20:15.295968+0000","last_became_peered":"2024-09-21T12:20:15.295968+0000","last_unstale":"2024-09-21T12:21:01.798031+0000","last_undegraded":"2024-09-21T12:21:01.798031+0000","last_fullsized":"2024-09-21T12:21:01.798031+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:33.273850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797528+0000","last_change":"2024-09-21T12:20:19.309473+0000","last_active":"2024-09-21T12:21:01.797528+0000","last_peered":"2024-09-21T12:21:01.797528+0000","last_clean":"2024-09-21T12:21:01.797528+0000","last_became_active":"2024-09-21T12:20:19.309195+0000","last_became_peered":"2024-09-21T12:20:19.309195+0000","last_unstale":"2024-09-21T12:21:01.797528+0000","last_undegraded":"2024-09-21T12:21:01.797528+0000","last_fullsized":"2024-09-21T12:21:01.797528+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:39.580857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797258+0000","last_change":"2024-09-21T12:20:21.595660+0000","last_active":"2024-09-21T12:21:01.797258+0000","last_peered":"2024-09-21T12:21:01.797258+0000","last_clean":"2024-09-21T12:21:01.797258+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:21:01.797258+0000","last_undegraded":"2024-09-21T12:21:01.797258+0000","last_fullsized":"2024-09-21T12:21:01.797258+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:22.428602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797916+0000","last_change":"2024-09-21T12:20:17.304075+0000","last_active":"2024-09-21T12:21:01.797916+0000","last_peered":"2024-09-21T12:21:01.797916+0000","last_clean":"2024-09-21T12:21:01.797916+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:21:01.797916+0000","last_undegraded":"2024-09-21T12:21:01.797916+0000","last_fullsized":"2024-09-21T12:21:01.797916+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:46.024324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":60,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761119+0000","last_change":"2024-09-21T12:20:15.298443+0000","last_active":"2024-09-21T12:21:01.761119+0000","last_peered":"2024-09-21T12:21:01.761119+0000","last_clean":"2024-09-21T12:21:01.761119+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:21:01.761119+0000","last_undegraded":"2024-09-21T12:21:01.761119+0000","last_fullsized":"2024-09-21T12:21:01.761119+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:36.774153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.760492+0000","last_change":"2024-09-21T12:20:19.312978+0000","last_active":"2024-09-21T12:21:01.760492+0000","last_peered":"2024-09-21T12:21:01.760492+0000","last_clean":"2024-09-21T12:21:01.760492+0000","last_became_active":"2024-09-21T12:20:19.312512+0000","last_became_peered":"2024-09-21T12:20:19.312512+0000","last_unstale":"2024-09-21T12:21:01.760492+0000","last_undegraded":"2024-09-21T12:21:01.760492+0000","last_fullsized":"2024-09-21T12:21:01.760492+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:41.253296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.797974+0000","last_change":"2024-09-21T12:20:21.313523+0000","last_active":"2024-09-21T12:21:01.797974+0000","last_peered":"2024-09-21T12:21:01.797974+0000","last_clean":"2024-09-21T12:21:01.797974+0000","last_became_active":"2024-09-21T12:20:21.313228+0000","last_became_peered":"2024-09-21T12:20:21.313228+0000","last_unstale":"2024-09-21T12:21:01.797974+0000","last_undegraded":"2024-09-21T12:21:01.797974+0000","last_fullsized":"2024-09-21T12:21:01.797974+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:47.112414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761201+0000","last_change":"2024-09-21T12:20:21.309169+0000","last_active":"2024-09-21T12:21:01.761201+0000","last_peered":"2024-09-21T12:21:01.761201+0000","last_clean":"2024-09-21T12:21:01.761201+0000","last_became_active":"2024-09-21T12:20:21.308865+0000","last_became_peered":"2024-09-21T12:20:21.308865+0000","last_unstale":"2024-09-21T12:21:01.761201+0000","last_undegraded":"2024-09-21T12:21:01.761201+0000","last_fullsized":"2024-09-21T12:21:01.761201+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:26.841000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":52,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.798231+0000","last_change":"2024-09-21T12:20:15.298322+0000","last_active":"2024-09-21T12:21:01.798231+0000","last_peered":"2024-09-21T12:21:01.798231+0000","last_clean":"2024-09-21T12:21:01.798231+0000","last_became_active":"2024-09-21T12:20:15.297993+0000","last_became_peered":"2024-09-21T12:20:15.297993+0000","last_unstale":"2024-09-21T12:21:01.798231+0000","last_undegraded":"2024-09-21T12:21:01.798231+0000","last_fullsized":"2024-09-21T12:21:01.798231+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:34.289061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":69,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761092+0000","last_change":"2024-09-21T12:20:17.299134+0000","last_active":"2024-09-21T12:21:01.761092+0000","last_peered":"2024-09-21T12:21:01.761092+0000","last_clean":"2024-09-21T12:21:01.761092+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:21:01.761092+0000","last_undegraded":"2024-09-21T12:21:01.761092+0000","last_fullsized":"2024-09-21T12:21:01.761092+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:30.792664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-21T12:21:01.761320+0000","last_change":"2024-09-21T12:20:19.308586+0000","last_active":"2024-09-21T12:21:01.761320+0000","last_peered":"2024-09-21T12:21:01.761320+0000","last_clean":"2024-09-21T12:21:01.761320+0000","last_became_active":"2024-09-21T12:20:19.308247+0000","last_became_peered":"2024-09-21T12:20:19.308247+0000","last_unstale":"2024-09-21T12:21:01.761320+0000","last_undegraded":"2024-09-21T12:21:01.761320+0000","last_fullsized":"2024-09-21T12:21:01.761320+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:42.632946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":51,"num_read_kb":46,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928233999,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32648,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712504,"statfs":{"total":95995035648,"available":95961604096,"internally_reserved":0,"allocated":1495040,"data_stored":990615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462935,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32636,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712516,"statfs":{"total":95995035648,"available":95961616384,"internally_reserved":0,"allocated":1478656,"data_stored":989966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724570,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":897024,"data_stored":400963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953506,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":909312,"data_stored":399767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149736,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":925696,"data_stored":401482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32048,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713104,"statfs":{"total":95995035648,"available":95962218496,"internally_reserved":0,"allocated":876544,"data_stored":399310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":950272,"data_stored":400960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28504,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716648,"statfs":{"total":95995035648,"available":95965847552,"internally_reserved":0,"allocated":1507328,"data_stored":991330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:21:15.115 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T12:21:15.115 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd dump 2024-09-21T12:21:15.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:14 smithi183 ceph-mon[33277]: pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 868 B/s rd, 0 op/s 2024-09-21T12:21:15.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:14 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:15.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:14 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:15.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:14 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:15.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:14 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:15.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:14 smithi183 ceph-mon[33277]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-21T12:21:15.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:14 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:21:15.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:14 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:21:15.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:14 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:15.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:14 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:15.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:14 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-21T12:21:15.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:14 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:15.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:14 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T12:21:15.365 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:21:15.746 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:21:15 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a[75423]: ts=2024-09-21T12:21:15.665Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000330428s 2024-09-21T12:21:16.017 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 81 2024-09-21T12:21:16.018 INFO:teuthology.orchestra.run.smithi114.stdout:fsid e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T12:21:16.018 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-09-21T12:14:24.181227+0000 2024-09-21T12:21:16.018 INFO:teuthology.orchestra.run.smithi114.stdout:modified 2024-09-21T12:21:14.831304+0000 2024-09-21T12:21:16.018 INFO:teuthology.orchestra.run.smithi114.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-21T12:21:16.018 INFO:teuthology.orchestra.run.smithi114.stdout:crush_version 18 2024-09-21T12:21:16.018 INFO:teuthology.orchestra.run.smithi114.stdout:full_ratio 0.95 2024-09-21T12:21:16.018 INFO:teuthology.orchestra.run.smithi114.stdout:backfillfull_ratio 0.9 2024-09-21T12:21:16.018 INFO:teuthology.orchestra.run.smithi114.stdout:nearfull_ratio 0.85 2024-09-21T12:21:16.018 INFO:teuthology.orchestra.run.smithi114.stdout:require_min_compat_client luminous 2024-09-21T12:21:16.018 INFO:teuthology.orchestra.run.smithi114.stdout:min_compat_client jewel 2024-09-21T12:21:16.018 INFO:teuthology.orchestra.run.smithi114.stdout:require_osd_release squid 2024-09-21T12:21:16.018 INFO:teuthology.orchestra.run.smithi114.stdout:stretch_mode_enabled false 2024-09-21T12:21:16.018 INFO:teuthology.orchestra.run.smithi114.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-09-21T12:21:16.019 INFO:teuthology.orchestra.run.smithi114.stdout:pool 2 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 61 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-09-21T12:21:16.019 INFO:teuthology.orchestra.run.smithi114.stdout:pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 60 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-09-21T12:21:16.019 INFO:teuthology.orchestra.run.smithi114.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 62 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-09-21T12:21:16.019 INFO:teuthology.orchestra.run.smithi114.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 64 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-09-21T12:21:16.019 INFO:teuthology.orchestra.run.smithi114.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 66 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-09-21T12:21:16.019 INFO:teuthology.orchestra.run.smithi114.stdout:max_osd 8 2024-09-21T12:21:16.019 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 up in weight 1 up_from 8 up_thru 64 down_at 0 last_clean_interval [0,0) [v2:172.21.15.114:6802/3555869816,v1:172.21.15.114:6803/3555869816] [v2:172.21.15.114:6804/3555869816,v1:172.21.15.114:6805/3555869816] exists,up 5715e521-c7dc-41dd-a075-5c96a88fa9b3 2024-09-21T12:21:16.019 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 up in weight 1 up_from 13 up_thru 64 down_at 0 last_clean_interval [0,0) [v2:172.21.15.114:6810/2744663863,v1:172.21.15.114:6811/2744663863] [v2:172.21.15.114:6812/2744663863,v1:172.21.15.114:6813/2744663863] exists,up 1fe160a7-5972-4e3e-99ce-15e2311ba4c5 2024-09-21T12:21:16.019 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 up in weight 1 up_from 18 up_thru 64 down_at 0 last_clean_interval [0,0) [v2:172.21.15.114:6818/3740045173,v1:172.21.15.114:6819/3740045173] [v2:172.21.15.114:6820/3740045173,v1:172.21.15.114:6821/3740045173] exists,up 6a63581f-7603-49ae-9ab3-b37e7c91ab91 2024-09-21T12:21:16.019 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 up in weight 1 up_from 26 up_thru 64 down_at 0 last_clean_interval [0,0) [v2:172.21.15.114:6826/1312590318,v1:172.21.15.114:6827/1312590318] [v2:172.21.15.114:6828/1312590318,v1:172.21.15.114:6829/1312590318] exists,up 0ad68cdb-ae5c-4090-8a8d-b67c1177ae42 2024-09-21T12:21:16.019 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 up in weight 1 up_from 32 up_thru 64 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6800/1391644836,v1:172.21.15.183:6801/1391644836] [v2:172.21.15.183:6802/1391644836,v1:172.21.15.183:6803/1391644836] exists,up 26bb8a84-e41d-4425-8491-2dea9764b7cf 2024-09-21T12:21:16.019 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 up in weight 1 up_from 39 up_thru 64 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6808/3187643473,v1:172.21.15.183:6809/3187643473] [v2:172.21.15.183:6810/3187643473,v1:172.21.15.183:6811/3187643473] exists,up 33f96871-0869-4281-82b9-bd518987c551 2024-09-21T12:21:16.019 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6 up in weight 1 up_from 47 up_thru 62 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6816/3025199631,v1:172.21.15.183:6817/3025199631] [v2:172.21.15.183:6818/3025199631,v1:172.21.15.183:6819/3025199631] exists,up e09c72b8-ad5a-4779-a874-b149fce6ed70 2024-09-21T12:21:16.020 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7 up in weight 1 up_from 54 up_thru 64 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6824/3203337442,v1:172.21.15.183:6825/3203337442] [v2:172.21.15.183:6826/3203337442,v1:172.21.15.183:6827/3203337442] exists,up 1a24194d-1af6-485f-8856-98844a02dd9d 2024-09-21T12:21:16.020 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:6801/237768393 expires 2024-09-22T12:20:45.528856+0000 2024-09-21T12:21:16.020 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:6800/237768393 expires 2024-09-22T12:20:45.528856+0000 2024-09-21T12:21:16.020 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/898990943 expires 2024-09-22T12:14:53.998653+0000 2024-09-21T12:21:16.020 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/3866305956 expires 2024-09-22T12:20:45.528856+0000 2024-09-21T12:21:16.020 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/3598782088 expires 2024-09-22T12:20:45.528856+0000 2024-09-21T12:21:16.020 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/2620151541 expires 2024-09-22T12:15:20.045217+0000 2024-09-21T12:21:16.020 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:6800/1725148595 expires 2024-09-22T12:14:53.998653+0000 2024-09-21T12:21:16.020 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/442404805 expires 2024-09-22T12:14:53.998653+0000 2024-09-21T12:21:16.020 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:6801/1725148595 expires 2024-09-22T12:14:53.998653+0000 2024-09-21T12:21:16.020 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/4173909019 expires 2024-09-22T12:20:45.528856+0000 2024-09-21T12:21:16.020 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/2106467614 expires 2024-09-22T12:15:20.045217+0000 2024-09-21T12:21:16.021 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/139803357 expires 2024-09-22T12:15:20.045217+0000 2024-09-21T12:21:16.021 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:6801/770932721 expires 2024-09-22T12:15:20.045217+0000 2024-09-21T12:21:16.021 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:6800/770932721 expires 2024-09-22T12:15:20.045217+0000 2024-09-21T12:21:16.021 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/1457547294 expires 2024-09-22T12:20:45.528856+0000 2024-09-21T12:21:16.021 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/3416259228 expires 2024-09-22T12:14:53.998653+0000 2024-09-21T12:21:16.021 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/995404371 expires 2024-09-22T12:20:45.528856+0000 2024-09-21T12:21:16.021 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:15 smithi114 ceph-mon[30324]: from='client.24626 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:21:16.021 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:15 smithi114 ceph-mon[30324]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T12:21:16.021 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:15 smithi114 ceph-mon[30324]: Deploying daemon grafana.a on smithi183 2024-09-21T12:21:16.021 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:15 smithi114 ceph-mon[30324]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:21:16.022 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:15 smithi114 ceph-mon[30324]: osdmap e81: 8 total, 8 up, 8 in 2024-09-21T12:21:16.022 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:15 smithi114 ceph-mon[40867]: from='client.24626 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:21:16.022 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:15 smithi114 ceph-mon[40867]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T12:21:16.022 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:15 smithi114 ceph-mon[40867]: Deploying daemon grafana.a on smithi183 2024-09-21T12:21:16.022 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:15 smithi114 ceph-mon[40867]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:21:16.022 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:15 smithi114 ceph-mon[40867]: osdmap e81: 8 total, 8 up, 8 in 2024-09-21T12:21:16.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:15 smithi183 ceph-mon[33277]: from='client.24626 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:21:16.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:15 smithi183 ceph-mon[33277]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T12:21:16.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:15 smithi183 ceph-mon[33277]: Deploying daemon grafana.a on smithi183 2024-09-21T12:21:16.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:15 smithi183 ceph-mon[33277]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:21:16.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:15 smithi183 ceph-mon[33277]: osdmap e81: 8 total, 8 up, 8 in 2024-09-21T12:21:16.773 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 64 down_at 0 last_clean_interval [0,0) [v2:172.21.15.114:6802/3555869816,v1:172.21.15.114:6803/3555869816] [v2:172.21.15.114:6804/3555869816,v1:172.21.15.114:6805/3555869816] exists,up 5715e521-c7dc-41dd-a075-5c96a88fa9b3', 'osd.1 up in weight 1 up_from 13 up_thru 64 down_at 0 last_clean_interval [0,0) [v2:172.21.15.114:6810/2744663863,v1:172.21.15.114:6811/2744663863] [v2:172.21.15.114:6812/2744663863,v1:172.21.15.114:6813/2744663863] exists,up 1fe160a7-5972-4e3e-99ce-15e2311ba4c5', 'osd.2 up in weight 1 up_from 18 up_thru 64 down_at 0 last_clean_interval [0,0) [v2:172.21.15.114:6818/3740045173,v1:172.21.15.114:6819/3740045173] [v2:172.21.15.114:6820/3740045173,v1:172.21.15.114:6821/3740045173] exists,up 6a63581f-7603-49ae-9ab3-b37e7c91ab91', 'osd.3 up in weight 1 up_from 26 up_thru 64 down_at 0 last_clean_interval [0,0) [v2:172.21.15.114:6826/1312590318,v1:172.21.15.114:6827/1312590318] [v2:172.21.15.114:6828/1312590318,v1:172.21.15.114:6829/1312590318] exists,up 0ad68cdb-ae5c-4090-8a8d-b67c1177ae42', 'osd.4 up in weight 1 up_from 32 up_thru 64 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6800/1391644836,v1:172.21.15.183:6801/1391644836] [v2:172.21.15.183:6802/1391644836,v1:172.21.15.183:6803/1391644836] exists,up 26bb8a84-e41d-4425-8491-2dea9764b7cf', 'osd.5 up in weight 1 up_from 39 up_thru 64 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6808/3187643473,v1:172.21.15.183:6809/3187643473] [v2:172.21.15.183:6810/3187643473,v1:172.21.15.183:6811/3187643473] exists,up 33f96871-0869-4281-82b9-bd518987c551', 'osd.6 up in weight 1 up_from 47 up_thru 62 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6816/3025199631,v1:172.21.15.183:6817/3025199631] [v2:172.21.15.183:6818/3025199631,v1:172.21.15.183:6819/3025199631] exists,up e09c72b8-ad5a-4779-a874-b149fce6ed70', 'osd.7 up in weight 1 up_from 54 up_thru 64 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6824/3203337442,v1:172.21.15.183:6825/3203337442] [v2:172.21.15.183:6826/3203337442,v1:172.21.15.183:6827/3203337442] exists,up 1a24194d-1af6-485f-8856-98844a02dd9d'] 2024-09-21T12:21:16.774 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon mon.a config show 2024-09-21T12:21:16.847 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:21:16 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:21:17.018 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:21:17.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:16 smithi114 ceph-mon[30324]: pgmap v24: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T12:21:17.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:16 smithi114 ceph-mon[30324]: 3.10 scrub starts 2024-09-21T12:21:17.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:16 smithi114 ceph-mon[30324]: 3.10 scrub ok 2024-09-21T12:21:17.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:16 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:17.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:16 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:21:17.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:16 smithi114 ceph-mon[30324]: 3.13 scrub starts 2024-09-21T12:21:17.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:16 smithi114 ceph-mon[30324]: 3.13 scrub ok 2024-09-21T12:21:17.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:16 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1610753532' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T12:21:17.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:16 smithi114 ceph-mon[30324]: osdmap e82: 8 total, 8 up, 8 in 2024-09-21T12:21:17.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:16 smithi114 ceph-mon[40867]: pgmap v24: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T12:21:17.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:16 smithi114 ceph-mon[40867]: 3.10 scrub starts 2024-09-21T12:21:17.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:16 smithi114 ceph-mon[40867]: 3.10 scrub ok 2024-09-21T12:21:17.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:16 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:17.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:16 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:21:17.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:16 smithi114 ceph-mon[40867]: 3.13 scrub starts 2024-09-21T12:21:17.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:16 smithi114 ceph-mon[40867]: 3.13 scrub ok 2024-09-21T12:21:17.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:16 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1610753532' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T12:21:17.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:16 smithi114 ceph-mon[40867]: osdmap e82: 8 total, 8 up, 8 in 2024-09-21T12:21:17.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:16 smithi183 ceph-mon[33277]: pgmap v24: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T12:21:17.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:16 smithi183 ceph-mon[33277]: 3.10 scrub starts 2024-09-21T12:21:17.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:16 smithi183 ceph-mon[33277]: 3.10 scrub ok 2024-09-21T12:21:17.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:16 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:16 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:21:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:16 smithi183 ceph-mon[33277]: 3.13 scrub starts 2024-09-21T12:21:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:16 smithi183 ceph-mon[33277]: 3.13 scrub ok 2024-09-21T12:21:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:16 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1610753532' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T12:21:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:16 smithi183 ceph-mon[33277]: osdmap e82: 8 total, 8 up, 8 in 2024-09-21T12:21:17.495 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:21:17.495 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mon.a", 2024-09-21T12:21:17.495 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster": "ceph", 2024-09-21T12:21:17.496 INFO:teuthology.orchestra.run.smithi114.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-21T12:21:17.496 INFO:teuthology.orchestra.run.smithi114.stdout: "admin_socket_mode": "", 2024-09-21T12:21:17.496 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-09-21T12:21:17.496 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_client_required": "cephx, none", 2024-09-21T12:21:17.496 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_cluster_required": "cephx", 2024-09-21T12:21:17.496 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_debug": "true", 2024-09-21T12:21:17.496 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-21T12:21:17.496 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-21T12:21:17.496 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_service_required": "cephx", 2024-09-21T12:21:17.496 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-21T12:21:17.496 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_supported": "", 2024-09-21T12:21:17.496 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_aio": "true", 2024-09-21T12:21:17.496 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-21T12:21:17.497 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_aio_poll_ms": "250", 2024-09-21T12:21:17.497 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_aio_reap_max": "16", 2024-09-21T12:21:17.497 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_async_discard_threads": "0", 2024-09-21T12:21:17.497 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_block_size": "4096", 2024-09-21T12:21:17.497 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_debug_aio": "false", 2024-09-21T12:21:17.497 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-21T12:21:17.497 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-21T12:21:17.497 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_debug_inflight_ios": "false", 2024-09-21T12:21:17.497 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_enable_discard": "false", 2024-09-21T12:21:17.497 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_flock_retry": "3", 2024-09-21T12:21:17.497 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-21T12:21:17.497 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_inject_crash": "0", 2024-09-21T12:21:17.497 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-21T12:21:17.497 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_ioring": "false", 2024-09-21T12:21:17.498 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_ioring_hipri": "false", 2024-09-21T12:21:17.498 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-21T12:21:17.498 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-21T12:21:17.498 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-21T12:21:17.498 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-21T12:21:17.498 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-21T12:21:17.498 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-21T12:21:17.498 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_type": "", 2024-09-21T12:21:17.498 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_alloc_size": "1048576", 2024-09-21T12:21:17.498 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_allocator": "hybrid", 2024-09-21T12:21:17.498 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_buffered_io": "true", 2024-09-21T12:21:17.498 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_check_for_zeros": "false", 2024-09-21T12:21:17.498 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-21T12:21:17.499 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-21T12:21:17.499 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_compact_log_sync": "false", 2024-09-21T12:21:17.499 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-21T12:21:17.499 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-21T12:21:17.499 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-21T12:21:17.499 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-21T12:21:17.499 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_max_log_runway": "4194304", 2024-09-21T12:21:17.499 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_max_prefetch": "1048576", 2024-09-21T12:21:17.499 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_min_flush_size": "524288", 2024-09-21T12:21:17.499 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_min_log_runway": "1048576", 2024-09-21T12:21:17.499 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_replay_recovery": "false", 2024-09-21T12:21:17.499 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-21T12:21:17.499 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-21T12:21:17.499 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_sync_write": "false", 2024-09-21T12:21:17.500 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-21T12:21:17.500 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-21T12:21:17.500 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-21T12:21:17.500 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_allocation_from_file": "true", 2024-09-21T12:21:17.500 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_allocator": "hybrid", 2024-09-21T12:21:17.500 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_async_db_compaction": "true", 2024-09-21T12:21:17.500 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-21T12:21:17.500 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-21T12:21:17.500 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-21T12:21:17.500 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-21T12:21:17.500 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_bdev_label_multi": "true", 2024-09-21T12:21:17.500 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-21T12:21:17.500 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-21T12:21:17.500 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-21T12:21:17.501 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-21T12:21:17.501 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-21T12:21:17.501 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_block_create": "true", 2024-09-21T12:21:17.501 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_block_db_create": "false", 2024-09-21T12:21:17.501 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_block_db_path": "", 2024-09-21T12:21:17.501 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_block_db_size": "0", 2024-09-21T12:21:17.501 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_block_path": "", 2024-09-21T12:21:17.501 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_block_preallocate_file": "false", 2024-09-21T12:21:17.501 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_block_size": "107374182400", 2024-09-21T12:21:17.501 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_block_wal_create": "false", 2024-09-21T12:21:17.501 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_block_wal_path": "", 2024-09-21T12:21:17.501 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_block_wal_size": "100663296", 2024-09-21T12:21:17.501 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_bluefs": "true", 2024-09-21T12:21:17.501 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-21T12:21:17.502 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-21T12:21:17.502 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-21T12:21:17.502 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-09-21T12:21:17.502 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-21T12:21:17.502 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T12:21:17.502 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T12:21:17.502 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-21T12:21:17.502 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T12:21:17.502 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_autotune": "true", 2024-09-21T12:21:17.502 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-21T12:21:17.502 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-21T12:21:17.502 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-21T12:21:17.502 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-21T12:21:17.502 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_size": "0", 2024-09-21T12:21:17.503 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-21T12:21:17.503 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-21T12:21:17.503 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-21T12:21:17.503 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-21T12:21:17.503 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_type": "2q", 2024-09-21T12:21:17.503 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-21T12:21:17.503 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_clone_cow": "true", 2024-09-21T12:21:17.503 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-21T12:21:17.503 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-21T12:21:17.503 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-21T12:21:17.503 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-21T12:21:17.503 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-21T12:21:17.503 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-21T12:21:17.503 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-21T12:21:17.504 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_compression_mode": "none", 2024-09-21T12:21:17.504 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-21T12:21:17.504 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_csum_type": "crc32c", 2024-09-21T12:21:17.504 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-09-21T12:21:17.504 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-21T12:21:17.504 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_freelist": "false", 2024-09-21T12:21:17.504 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-21T12:21:17.504 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-21T12:21:17.504 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-21T12:21:17.504 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-21T12:21:17.504 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-21T12:21:17.504 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-21T12:21:17.504 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-21T12:21:17.504 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-21T12:21:17.504 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-21T12:21:17.505 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-21T12:21:17.505 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-21T12:21:17.505 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-21T12:21:17.505 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_small_allocations": "0", 2024-09-21T12:21:17.505 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-21T12:21:17.505 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_default_buffered_read": "true", 2024-09-21T12:21:17.505 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_default_buffered_write": "false", 2024-09-21T12:21:17.505 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-21T12:21:17.505 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-21T12:21:17.505 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-21T12:21:17.505 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-21T12:21:17.505 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-21T12:21:17.505 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-21T12:21:17.506 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-21T12:21:17.506 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-21T12:21:17.506 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-21T12:21:17.506 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fail_eio": "false", 2024-09-21T12:21:17.506 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-21T12:21:17.506 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-21T12:21:17.506 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-21T12:21:17.506 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-21T12:21:17.506 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-21T12:21:17.506 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-21T12:21:17.506 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_on_mount": "false", 2024-09-21T12:21:17.506 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-21T12:21:17.506 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_on_umount": "false", 2024-09-21T12:21:17.506 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-21T12:21:17.506 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-21T12:21:17.507 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-21T12:21:17.507 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-21T12:21:17.507 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-21T12:21:17.507 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-21T12:21:17.507 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-21T12:21:17.507 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-21T12:21:17.507 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_ignore_data_csum": "false", 2024-09-21T12:21:17.507 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-21T12:21:17.507 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-21T12:21:17.507 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-21T12:21:17.507 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-21T12:21:17.507 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_log_op_age": "5.000000", 2024-09-21T12:21:17.507 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_max_alloc_size": "0", 2024-09-21T12:21:17.507 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_max_blob_size": "0", 2024-09-21T12:21:17.507 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-21T12:21:17.508 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-21T12:21:17.508 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-21T12:21:17.508 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_max_deferred_txc": "32", 2024-09-21T12:21:17.508 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_min_alloc_size": "0", 2024-09-21T12:21:17.508 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-21T12:21:17.508 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-21T12:21:17.508 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_nid_prealloc": "1024", 2024-09-21T12:21:17.508 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-21T12:21:17.508 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-21T12:21:17.508 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-21T12:21:17.508 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-21T12:21:17.508 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_retry_disk_reads": "3", 2024-09-21T12:21:17.508 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_rocksdb_cf": "true", 2024-09-21T12:21:17.508 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-21T12:21:17.509 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-21T12:21:17.509 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-21T12:21:17.509 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-21T12:21:17.509 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-21T12:21:17.509 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-21T12:21:17.509 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-21T12:21:17.509 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-21T12:21:17.509 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_spdk_mem": "512", 2024-09-21T12:21:17.509 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-21T12:21:17.509 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-21T12:21:17.509 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-21T12:21:17.509 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-21T12:21:17.509 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-21T12:21:17.509 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-21T12:21:17.510 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-21T12:21:17.510 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_tracing": "false", 2024-09-21T12:21:17.510 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-21T12:21:17.510 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-21T12:21:17.510 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-21T12:21:17.510 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-21T12:21:17.510 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-21T12:21:17.510 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-21T12:21:17.510 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-21T12:21:17.510 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-21T12:21:17.510 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-21T12:21:17.510 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_write_v2": "false", 2024-09-21T12:21:17.510 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_write_v2_random": "false", 2024-09-21T12:21:17.510 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_zero_block_detection": "false", 2024-09-21T12:21:17.511 INFO:teuthology.orchestra.run.smithi114.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-21T12:21:17.511 INFO:teuthology.orchestra.run.smithi114.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-21T12:21:17.511 INFO:teuthology.orchestra.run.smithi114.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-21T12:21:17.511 INFO:teuthology.orchestra.run.smithi114.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-21T12:21:17.511 INFO:teuthology.orchestra.run.smithi114.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-21T12:21:17.511 INFO:teuthology.orchestra.run.smithi114.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-21T12:21:17.511 INFO:teuthology.orchestra.run.smithi114.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-21T12:21:17.511 INFO:teuthology.orchestra.run.smithi114.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-21T12:21:17.511 INFO:teuthology.orchestra.run.smithi114.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-21T12:21:17.511 INFO:teuthology.orchestra.run.smithi114.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-21T12:21:17.511 INFO:teuthology.orchestra.run.smithi114.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-21T12:21:17.511 INFO:teuthology.orchestra.run.smithi114.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-21T12:21:17.511 INFO:teuthology.orchestra.run.smithi114.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-21T12:21:17.511 INFO:teuthology.orchestra.run.smithi114.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-21T12:21:17.512 INFO:teuthology.orchestra.run.smithi114.stdout: "cephx_cluster_require_signatures": "false", 2024-09-21T12:21:17.512 INFO:teuthology.orchestra.run.smithi114.stdout: "cephx_cluster_require_version": "2", 2024-09-21T12:21:17.512 INFO:teuthology.orchestra.run.smithi114.stdout: "cephx_require_signatures": "false", 2024-09-21T12:21:17.512 INFO:teuthology.orchestra.run.smithi114.stdout: "cephx_require_version": "2", 2024-09-21T12:21:17.512 INFO:teuthology.orchestra.run.smithi114.stdout: "cephx_service_require_signatures": "false", 2024-09-21T12:21:17.512 INFO:teuthology.orchestra.run.smithi114.stdout: "cephx_service_require_version": "2", 2024-09-21T12:21:17.512 INFO:teuthology.orchestra.run.smithi114.stdout: "cephx_sign_messages": "true", 2024-09-21T12:21:17.512 INFO:teuthology.orchestra.run.smithi114.stdout: "chdir": "", 2024-09-21T12:21:17.512 INFO:teuthology.orchestra.run.smithi114.stdout: "client_acl_type": "", 2024-09-21T12:21:17.512 INFO:teuthology.orchestra.run.smithi114.stdout: "client_asio_thread_count": "2", 2024-09-21T12:21:17.512 INFO:teuthology.orchestra.run.smithi114.stdout: "client_cache_mid": "0.750000", 2024-09-21T12:21:17.512 INFO:teuthology.orchestra.run.smithi114.stdout: "client_cache_size": "16384", 2024-09-21T12:21:17.512 INFO:teuthology.orchestra.run.smithi114.stdout: "client_caps_release_delay": "5", 2024-09-21T12:21:17.512 INFO:teuthology.orchestra.run.smithi114.stdout: "client_check_pool_perm": "true", 2024-09-21T12:21:17.512 INFO:teuthology.orchestra.run.smithi114.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-21T12:21:17.512 INFO:teuthology.orchestra.run.smithi114.stdout: "client_debug_force_sync_read": "false", 2024-09-21T12:21:17.513 INFO:teuthology.orchestra.run.smithi114.stdout: "client_debug_getattr_caps": "false", 2024-09-21T12:21:17.513 INFO:teuthology.orchestra.run.smithi114.stdout: "client_debug_inject_features": "", 2024-09-21T12:21:17.513 INFO:teuthology.orchestra.run.smithi114.stdout: "client_debug_inject_tick_delay": "0", 2024-09-21T12:21:17.513 INFO:teuthology.orchestra.run.smithi114.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-21T12:21:17.513 INFO:teuthology.orchestra.run.smithi114.stdout: "client_die_on_failed_remount": "false", 2024-09-21T12:21:17.513 INFO:teuthology.orchestra.run.smithi114.stdout: "client_dirsize_rbytes": "true", 2024-09-21T12:21:17.513 INFO:teuthology.orchestra.run.smithi114.stdout: "client_force_lazyio": "false", 2024-09-21T12:21:17.513 INFO:teuthology.orchestra.run.smithi114.stdout: "client_fs": "", 2024-09-21T12:21:17.513 INFO:teuthology.orchestra.run.smithi114.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-21T12:21:17.513 INFO:teuthology.orchestra.run.smithi114.stdout: "client_inject_release_failure": "false", 2024-09-21T12:21:17.513 INFO:teuthology.orchestra.run.smithi114.stdout: "client_max_inline_size": "4096", 2024-09-21T12:21:17.513 INFO:teuthology.orchestra.run.smithi114.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-21T12:21:17.513 INFO:teuthology.orchestra.run.smithi114.stdout: "client_mds_namespace": "", 2024-09-21T12:21:17.513 INFO:teuthology.orchestra.run.smithi114.stdout: "client_metadata": "", 2024-09-21T12:21:17.513 INFO:teuthology.orchestra.run.smithi114.stdout: "client_mount_gid": "-1", 2024-09-21T12:21:17.513 INFO:teuthology.orchestra.run.smithi114.stdout: "client_mount_timeout": "300", 2024-09-21T12:21:17.514 INFO:teuthology.orchestra.run.smithi114.stdout: "client_mount_uid": "-1", 2024-09-21T12:21:17.514 INFO:teuthology.orchestra.run.smithi114.stdout: "client_mountpoint": "/", 2024-09-21T12:21:17.514 INFO:teuthology.orchestra.run.smithi114.stdout: "client_notify_timeout": "10", 2024-09-21T12:21:17.514 INFO:teuthology.orchestra.run.smithi114.stdout: "client_oc": "true", 2024-09-21T12:21:17.514 INFO:teuthology.orchestra.run.smithi114.stdout: "client_oc_max_dirty": "104857600", 2024-09-21T12:21:17.514 INFO:teuthology.orchestra.run.smithi114.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-21T12:21:17.514 INFO:teuthology.orchestra.run.smithi114.stdout: "client_oc_max_objects": "1000", 2024-09-21T12:21:17.514 INFO:teuthology.orchestra.run.smithi114.stdout: "client_oc_size": "209715200", 2024-09-21T12:21:17.514 INFO:teuthology.orchestra.run.smithi114.stdout: "client_oc_target_dirty": "8388608", 2024-09-21T12:21:17.514 INFO:teuthology.orchestra.run.smithi114.stdout: "client_permissions": "true", 2024-09-21T12:21:17.514 INFO:teuthology.orchestra.run.smithi114.stdout: "client_quota": "true", 2024-09-21T12:21:17.514 INFO:teuthology.orchestra.run.smithi114.stdout: "client_quota_df": "true", 2024-09-21T12:21:17.514 INFO:teuthology.orchestra.run.smithi114.stdout: "client_readahead_max_bytes": "0", 2024-09-21T12:21:17.514 INFO:teuthology.orchestra.run.smithi114.stdout: "client_readahead_max_periods": "4", 2024-09-21T12:21:17.515 INFO:teuthology.orchestra.run.smithi114.stdout: "client_readahead_min": "131072", 2024-09-21T12:21:17.515 INFO:teuthology.orchestra.run.smithi114.stdout: "client_reconnect_stale": "false", 2024-09-21T12:21:17.515 INFO:teuthology.orchestra.run.smithi114.stdout: "client_shutdown_timeout": "30", 2024-09-21T12:21:17.515 INFO:teuthology.orchestra.run.smithi114.stdout: "client_snapdir": ".snap", 2024-09-21T12:21:17.515 INFO:teuthology.orchestra.run.smithi114.stdout: "client_tick_interval": "1", 2024-09-21T12:21:17.515 INFO:teuthology.orchestra.run.smithi114.stdout: "client_trace": "", 2024-09-21T12:21:17.515 INFO:teuthology.orchestra.run.smithi114.stdout: "client_try_dentry_invalidate": "false", 2024-09-21T12:21:17.515 INFO:teuthology.orchestra.run.smithi114.stdout: "client_use_faked_inos": "false", 2024-09-21T12:21:17.515 INFO:teuthology.orchestra.run.smithi114.stdout: "client_use_random_mds": "false", 2024-09-21T12:21:17.515 INFO:teuthology.orchestra.run.smithi114.stdout: "clog_to_graylog": "false", 2024-09-21T12:21:17.515 INFO:teuthology.orchestra.run.smithi114.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-21T12:21:17.515 INFO:teuthology.orchestra.run.smithi114.stdout: "clog_to_graylog_port": "12201", 2024-09-21T12:21:17.515 INFO:teuthology.orchestra.run.smithi114.stdout: "clog_to_monitors": "default=true", 2024-09-21T12:21:17.515 INFO:teuthology.orchestra.run.smithi114.stdout: "clog_to_syslog": "false", 2024-09-21T12:21:17.516 INFO:teuthology.orchestra.run.smithi114.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-21T12:21:17.516 INFO:teuthology.orchestra.run.smithi114.stdout: "clog_to_syslog_level": "info", 2024-09-21T12:21:17.516 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "-", 2024-09-21T12:21:17.516 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_network": "", 2024-09-21T12:21:17.516 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_network_interface": "", 2024-09-21T12:21:17.516 INFO:teuthology.orchestra.run.smithi114.stdout: "compressor_zlib_isal": "false", 2024-09-21T12:21:17.516 INFO:teuthology.orchestra.run.smithi114.stdout: "compressor_zlib_level": "5", 2024-09-21T12:21:17.516 INFO:teuthology.orchestra.run.smithi114.stdout: "compressor_zlib_winsize": "-15", 2024-09-21T12:21:17.516 INFO:teuthology.orchestra.run.smithi114.stdout: "compressor_zstd_level": "1", 2024-09-21T12:21:17.516 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472", 2024-09-21T12:21:17.516 INFO:teuthology.orchestra.run.smithi114.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-21T12:21:17.516 INFO:teuthology.orchestra.run.smithi114.stdout: "crimson_alien_op_num_threads": "6", 2024-09-21T12:21:17.516 INFO:teuthology.orchestra.run.smithi114.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-21T12:21:17.516 INFO:teuthology.orchestra.run.smithi114.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-21T12:21:17.516 INFO:teuthology.orchestra.run.smithi114.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-21T12:21:17.516 INFO:teuthology.orchestra.run.smithi114.stdout: "crimson_osd_stat_interval": "0", 2024-09-21T12:21:17.516 INFO:teuthology.orchestra.run.smithi114.stdout: "crimson_seastar_cpu_cores": "", 2024-09-21T12:21:17.517 INFO:teuthology.orchestra.run.smithi114.stdout: "crimson_seastar_num_threads": "0", 2024-09-21T12:21:17.517 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "", 2024-09-21T12:21:17.517 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location_hook": "", 2024-09-21T12:21:17.517 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location_hook_timeout": "10", 2024-09-21T12:21:17.517 INFO:teuthology.orchestra.run.smithi114.stdout: "daemonize": "false", 2024-09-21T12:21:17.517 INFO:teuthology.orchestra.run.smithi114.stdout: "daos_pool": "tank", 2024-09-21T12:21:17.517 INFO:teuthology.orchestra.run.smithi114.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-21T12:21:17.517 INFO:teuthology.orchestra.run.smithi114.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-21T12:21:17.517 INFO:teuthology.orchestra.run.smithi114.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-21T12:21:17.517 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_alienstore": "0/5", 2024-09-21T12:21:17.517 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_allow_any_pool_priority": "false", 2024-09-21T12:21:17.517 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_asok": "1/5", 2024-09-21T12:21:17.517 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_asok_assert_abort": "false", 2024-09-21T12:21:17.517 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_asserts_on_shutdown": "true", 2024-09-21T12:21:17.517 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_auth": "1/5", 2024-09-21T12:21:17.518 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_bdev": "1/3", 2024-09-21T12:21:17.518 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_bluefs": "1/5", 2024-09-21T12:21:17.518 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_bluestore": "1/5", 2024-09-21T12:21:17.518 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_buffer": "0/1", 2024-09-21T12:21:17.518 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_ceph_dedup": "0/5", 2024-09-21T12:21:17.518 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_ceph_exporter": "1/5", 2024-09-21T12:21:17.518 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_cephfs_mirror": "0/5", 2024-09-21T12:21:17.518 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_cephsqlite": "0/5", 2024-09-21T12:21:17.518 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_client": "0/5", 2024-09-21T12:21:17.518 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_compressor": "1/5", 2024-09-21T12:21:17.518 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_context": "0/1", 2024-09-21T12:21:17.518 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_crimson_interrupt": "0/5", 2024-09-21T12:21:17.518 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_crush": "1/1", 2024-09-21T12:21:17.518 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_crypto": "1/5", 2024-09-21T12:21:17.518 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_cyanstore": "0/5", 2024-09-21T12:21:17.519 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_deliberately_leak_memory": "false", 2024-09-21T12:21:17.519 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_disable_randomized_ping": "false", 2024-09-21T12:21:17.519 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_dpdk": "1/5", 2024-09-21T12:21:17.519 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_eventtrace": "1/5", 2024-09-21T12:21:17.519 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_filer": "0/1", 2024-09-21T12:21:17.519 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_filestore": "1/3", 2024-09-21T12:21:17.519 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_finisher": "1/1", 2024-09-21T12:21:17.519 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_fuse": "1/5", 2024-09-21T12:21:17.519 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_heartbeat_testing_span": "0", 2024-09-21T12:21:17.519 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_heartbeatmap": "1/5", 2024-09-21T12:21:17.519 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-21T12:21:17.519 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_javaclient": "1/5", 2024-09-21T12:21:17.519 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_journal": "1/3", 2024-09-21T12:21:17.519 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_journaler": "0/5", 2024-09-21T12:21:17.519 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_kstore": "1/5", 2024-09-21T12:21:17.520 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_lockdep": "0/1", 2024-09-21T12:21:17.520 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_mclock": "1/5", 2024-09-21T12:21:17.520 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_mds": "1/5", 2024-09-21T12:21:17.520 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_mds_balancer": "1/5", 2024-09-21T12:21:17.520 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_mds_locker": "1/5", 2024-09-21T12:21:17.520 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_mds_log": "1/5", 2024-09-21T12:21:17.520 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_mds_log_expire": "1/5", 2024-09-21T12:21:17.520 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_mds_migrator": "1/5", 2024-09-21T12:21:17.520 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_mds_quiesce": "3/5", 2024-09-21T12:21:17.520 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_memstore": "1/5", 2024-09-21T12:21:17.520 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_mgr": "2/5", 2024-09-21T12:21:17.520 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_mgrc": "1/5", 2024-09-21T12:21:17.520 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_mon": "20/20", 2024-09-21T12:21:17.520 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_monc": "0/10", 2024-09-21T12:21:17.520 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_ms": "1/1", 2024-09-21T12:21:17.521 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_none": "0/5", 2024-09-21T12:21:17.521 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_objclass": "0/5", 2024-09-21T12:21:17.521 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_objectcacher": "0/5", 2024-09-21T12:21:17.521 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_objecter": "0/1", 2024-09-21T12:21:17.521 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_optracker": "0/5", 2024-09-21T12:21:17.521 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_osd": "1/5", 2024-09-21T12:21:17.521 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_paxos": "20/20", 2024-09-21T12:21:17.521 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_perfcounter": "1/5", 2024-09-21T12:21:17.521 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_prioritycache": "1/5", 2024-09-21T12:21:17.521 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_rados": "0/5", 2024-09-21T12:21:17.521 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_rbd": "0/5", 2024-09-21T12:21:17.521 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_rbd_mirror": "0/5", 2024-09-21T12:21:17.521 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_rbd_pwl": "0/5", 2024-09-21T12:21:17.521 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_rbd_replay": "0/5", 2024-09-21T12:21:17.521 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_refs": "0/0", 2024-09-21T12:21:17.522 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_reserver": "1/1", 2024-09-21T12:21:17.522 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_rgw": "1/5", 2024-09-21T12:21:17.522 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_rgw_access": "1/5", 2024-09-21T12:21:17.522 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_rgw_datacache": "1/5", 2024-09-21T12:21:17.522 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_rgw_dbstore": "1/5", 2024-09-21T12:21:17.522 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_rgw_flight": "1/5", 2024-09-21T12:21:17.522 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-21T12:21:17.522 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_rgw_notification": "1/5", 2024-09-21T12:21:17.522 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_rgw_sync": "1/5", 2024-09-21T12:21:17.522 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_rocksdb": "4/5", 2024-09-21T12:21:17.522 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_seastore": "0/5", 2024-09-21T12:21:17.522 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_seastore_backref": "0/5", 2024-09-21T12:21:17.522 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_seastore_cache": "0/5", 2024-09-21T12:21:17.522 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_seastore_cleaner": "0/5", 2024-09-21T12:21:17.522 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_seastore_device": "0/5", 2024-09-21T12:21:17.523 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_seastore_epm": "0/5", 2024-09-21T12:21:17.523 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-21T12:21:17.523 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_seastore_journal": "0/5", 2024-09-21T12:21:17.523 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_seastore_lba": "0/5", 2024-09-21T12:21:17.523 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_seastore_odata": "0/5", 2024-09-21T12:21:17.523 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_seastore_omap": "0/5", 2024-09-21T12:21:17.523 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_seastore_onode": "0/5", 2024-09-21T12:21:17.523 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_seastore_t": "0/5", 2024-09-21T12:21:17.523 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_seastore_tm": "0/5", 2024-09-21T12:21:17.523 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_striper": "0/1", 2024-09-21T12:21:17.523 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_test": "0/5", 2024-09-21T12:21:17.523 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_throttle": "1/1", 2024-09-21T12:21:17.523 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_timer": "0/1", 2024-09-21T12:21:17.523 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_tp": "0/5", 2024-09-21T12:21:17.523 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_trace": "1/5", 2024-09-21T12:21:17.524 INFO:teuthology.orchestra.run.smithi114.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-21T12:21:17.524 INFO:teuthology.orchestra.run.smithi114.stdout: "device_failure_prediction_mode": "none", 2024-09-21T12:21:17.524 INFO:teuthology.orchestra.run.smithi114.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-21T12:21:17.524 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-21T12:21:17.524 INFO:teuthology.orchestra.run.smithi114.stdout: "err_to_graylog": "false", 2024-09-21T12:21:17.524 INFO:teuthology.orchestra.run.smithi114.stdout: "err_to_journald": "false", 2024-09-21T12:21:17.524 INFO:teuthology.orchestra.run.smithi114.stdout: "err_to_stderr": "true", 2024-09-21T12:21:17.524 INFO:teuthology.orchestra.run.smithi114.stdout: "err_to_syslog": "false", 2024-09-21T12:21:17.524 INFO:teuthology.orchestra.run.smithi114.stdout: "event_tracing": "false", 2024-09-21T12:21:17.524 INFO:teuthology.orchestra.run.smithi114.stdout: "exporter_addr": "0.0.0.0", 2024-09-21T12:21:17.524 INFO:teuthology.orchestra.run.smithi114.stdout: "exporter_cert_file": "", 2024-09-21T12:21:17.524 INFO:teuthology.orchestra.run.smithi114.stdout: "exporter_http_port": "9926", 2024-09-21T12:21:17.524 INFO:teuthology.orchestra.run.smithi114.stdout: "exporter_key_file": "", 2024-09-21T12:21:17.524 INFO:teuthology.orchestra.run.smithi114.stdout: "exporter_prio_limit": "5", 2024-09-21T12:21:17.524 INFO:teuthology.orchestra.run.smithi114.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-21T12:21:17.525 INFO:teuthology.orchestra.run.smithi114.stdout: "exporter_sort_metrics": "true", 2024-09-21T12:21:17.525 INFO:teuthology.orchestra.run.smithi114.stdout: "exporter_stats_period": "5", 2024-09-21T12:21:17.525 INFO:teuthology.orchestra.run.smithi114.stdout: "fake_statfs_for_testing": "0", 2024-09-21T12:21:17.525 INFO:teuthology.orchestra.run.smithi114.stdout: "fatal_signal_handlers": "true", 2024-09-21T12:21:17.525 INFO:teuthology.orchestra.run.smithi114.stdout: "filer_max_purge_ops": "10", 2024-09-21T12:21:17.525 INFO:teuthology.orchestra.run.smithi114.stdout: "filer_max_truncate_ops": "128", 2024-09-21T12:21:17.525 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_apply_finisher_threads": "1", 2024-09-21T12:21:17.525 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_blackhole": "false", 2024-09-21T12:21:17.525 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_btrfs_clone_range": "true", 2024-09-21T12:21:17.525 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_btrfs_snap": "true", 2024-09-21T12:21:17.525 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_caller_concurrency": "10", 2024-09-21T12:21:17.525 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_collect_device_partition_information": "true", 2024-09-21T12:21:17.525 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_commit_timeout": "600.000000", 2024-09-21T12:21:17.525 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_debug_inject_read_err": "false", 2024-09-21T12:21:17.526 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_debug_omap_check": "false", 2024-09-21T12:21:17.526 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_debug_verify_split": "false", 2024-09-21T12:21:17.526 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_dump_file": "", 2024-09-21T12:21:17.526 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-21T12:21:17.526 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-21T12:21:17.526 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_fadvise": "true", 2024-09-21T12:21:17.526 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_fail_eio": "true", 2024-09-21T12:21:17.526 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_fd_cache_shards": "16", 2024-09-21T12:21:17.526 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_fd_cache_size": "128", 2024-09-21T12:21:17.526 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_fiemap": "false", 2024-09-21T12:21:17.526 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_fiemap_threshold": "4096", 2024-09-21T12:21:17.526 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-21T12:21:17.526 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-21T12:21:17.526 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_inject_stall": "0", 2024-09-21T12:21:17.526 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_journal_parallel": "false", 2024-09-21T12:21:17.527 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_journal_trailing": "false", 2024-09-21T12:21:17.527 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_journal_writeahead": "false", 2024-09-21T12:21:17.527 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_kill_at": "0", 2024-09-21T12:21:17.527 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-21T12:21:17.527 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-21T12:21:17.527 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-21T12:21:17.527 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-21T12:21:17.527 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-21T12:21:17.527 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_max_inline_xattrs": "0", 2024-09-21T12:21:17.527 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-21T12:21:17.527 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-21T12:21:17.527 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-21T12:21:17.527 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-21T12:21:17.527 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_max_xattr_value_size": "0", 2024-09-21T12:21:17.527 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-21T12:21:17.528 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-21T12:21:17.528 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-21T12:21:17.528 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_merge_threshold": "-10", 2024-09-21T12:21:17.528 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-21T12:21:17.528 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_odsync_write": "false", 2024-09-21T12:21:17.528 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_omap_backend": "rocksdb", 2024-09-21T12:21:17.528 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_omap_backend_path": "", 2024-09-21T12:21:17.528 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-21T12:21:17.528 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-21T12:21:17.528 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-21T12:21:17.528 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_op_thread_timeout": "60", 2024-09-21T12:21:17.528 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_op_threads": "2", 2024-09-21T12:21:17.528 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_punch_hole": "false", 2024-09-21T12:21:17.528 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-21T12:21:17.529 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-21T12:21:17.529 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-21T12:21:17.529 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-21T12:21:17.529 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-21T12:21:17.529 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-21T12:21:17.529 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-21T12:21:17.529 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-21T12:21:17.529 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-21T12:21:17.529 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_queue_max_ops": "50", 2024-09-21T12:21:17.529 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-21T12:21:17.529 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_seek_data_hole": "false", 2024-09-21T12:21:17.529 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_sloppy_crc": "false", 2024-09-21T12:21:17.529 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-21T12:21:17.529 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_splice": "false", 2024-09-21T12:21:17.530 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_split_multiple": "2", 2024-09-21T12:21:17.530 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_split_rand_factor": "20", 2024-09-21T12:21:17.530 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_update_to": "1000", 2024-09-21T12:21:17.530 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-21T12:21:17.530 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-21T12:21:17.530 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-21T12:21:17.530 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-21T12:21:17.530 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-21T12:21:17.530 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-21T12:21:17.530 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_wbthrottle_enable": "true", 2024-09-21T12:21:17.530 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-21T12:21:17.530 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-21T12:21:17.530 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-21T12:21:17.530 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-21T12:21:17.531 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-21T12:21:17.531 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-21T12:21:17.531 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_xfs_extsize": "false", 2024-09-21T12:21:17.531 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_zfs_snap": "false", 2024-09-21T12:21:17.531 INFO:teuthology.orchestra.run.smithi114.stdout: "fio_dir": "/tmp/fio", 2024-09-21T12:21:17.531 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "e2ad975c-7812-11ef-baf6-efdc52797490", 2024-09-21T12:21:17.531 INFO:teuthology.orchestra.run.smithi114.stdout: "fuse_allow_other": "true", 2024-09-21T12:21:17.531 INFO:teuthology.orchestra.run.smithi114.stdout: "fuse_atomic_o_trunc": "true", 2024-09-21T12:21:17.531 INFO:teuthology.orchestra.run.smithi114.stdout: "fuse_big_writes": "true", 2024-09-21T12:21:17.531 INFO:teuthology.orchestra.run.smithi114.stdout: "fuse_debug": "false", 2024-09-21T12:21:17.531 INFO:teuthology.orchestra.run.smithi114.stdout: "fuse_default_permissions": "false", 2024-09-21T12:21:17.531 INFO:teuthology.orchestra.run.smithi114.stdout: "fuse_disable_pagecache": "false", 2024-09-21T12:21:17.531 INFO:teuthology.orchestra.run.smithi114.stdout: "fuse_max_write": "0", 2024-09-21T12:21:17.531 INFO:teuthology.orchestra.run.smithi114.stdout: "fuse_multithreaded": "true", 2024-09-21T12:21:17.532 INFO:teuthology.orchestra.run.smithi114.stdout: "fuse_require_active_mds": "true", 2024-09-21T12:21:17.532 INFO:teuthology.orchestra.run.smithi114.stdout: "fuse_set_user_groups": "true", 2024-09-21T12:21:17.532 INFO:teuthology.orchestra.run.smithi114.stdout: "fuse_splice_move": "true", 2024-09-21T12:21:17.532 INFO:teuthology.orchestra.run.smithi114.stdout: "fuse_splice_read": "true", 2024-09-21T12:21:17.532 INFO:teuthology.orchestra.run.smithi114.stdout: "fuse_splice_write": "true", 2024-09-21T12:21:17.532 INFO:teuthology.orchestra.run.smithi114.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-21T12:21:17.532 INFO:teuthology.orchestra.run.smithi114.stdout: "fuse_use_invalidate_cb": "true", 2024-09-21T12:21:17.532 INFO:teuthology.orchestra.run.smithi114.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-21T12:21:17.532 INFO:teuthology.orchestra.run.smithi114.stdout: "gss_target_name": "ceph", 2024-09-21T12:21:17.532 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_file": "", 2024-09-21T12:21:17.532 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_inject_failure": "0", 2024-09-21T12:21:17.532 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_interval": "5", 2024-09-21T12:21:17.532 INFO:teuthology.orchestra.run.smithi114.stdout: "host": "smithi114", 2024-09-21T12:21:17.532 INFO:teuthology.orchestra.run.smithi114.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-21T12:21:17.532 INFO:teuthology.orchestra.run.smithi114.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-21T12:21:17.533 INFO:teuthology.orchestra.run.smithi114.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-21T12:21:17.533 INFO:teuthology.orchestra.run.smithi114.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-21T12:21:17.533 INFO:teuthology.orchestra.run.smithi114.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-21T12:21:17.533 INFO:teuthology.orchestra.run.smithi114.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-21T12:21:17.533 INFO:teuthology.orchestra.run.smithi114.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-21T12:21:17.533 INFO:teuthology.orchestra.run.smithi114.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-21T12:21:17.533 INFO:teuthology.orchestra.run.smithi114.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-21T12:21:17.533 INFO:teuthology.orchestra.run.smithi114.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-21T12:21:17.533 INFO:teuthology.orchestra.run.smithi114.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-21T12:21:17.533 INFO:teuthology.orchestra.run.smithi114.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-21T12:21:17.533 INFO:teuthology.orchestra.run.smithi114.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-21T12:21:17.533 INFO:teuthology.orchestra.run.smithi114.stdout: "inject_early_sigterm": "false", 2024-09-21T12:21:17.533 INFO:teuthology.orchestra.run.smithi114.stdout: "jaeger_agent_port": "6799", 2024-09-21T12:21:17.533 INFO:teuthology.orchestra.run.smithi114.stdout: "jaeger_tracing_enable": "false", 2024-09-21T12:21:17.533 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_aio": "true", 2024-09-21T12:21:17.534 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_align_min_size": "65536", 2024-09-21T12:21:17.534 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_block_align": "true", 2024-09-21T12:21:17.534 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_block_size": "4096", 2024-09-21T12:21:17.534 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_dio": "true", 2024-09-21T12:21:17.534 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_discard": "false", 2024-09-21T12:21:17.534 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_force_aio": "false", 2024-09-21T12:21:17.534 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_ignore_corruption": "false", 2024-09-21T12:21:17.534 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_max_write_bytes": "10485760", 2024-09-21T12:21:17.534 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_max_write_entries": "100", 2024-09-21T12:21:17.534 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_replay_from": "0", 2024-09-21T12:21:17.534 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-21T12:21:17.534 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-21T12:21:17.534 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-21T12:21:17.534 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-21T12:21:17.535 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_write_header_frequency": "0", 2024-09-21T12:21:17.535 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_zero_on_create": "false", 2024-09-21T12:21:17.535 INFO:teuthology.orchestra.run.smithi114.stdout: "journaler_prefetch_periods": "10", 2024-09-21T12:21:17.535 INFO:teuthology.orchestra.run.smithi114.stdout: "journaler_prezero_periods": "5", 2024-09-21T12:21:17.535 INFO:teuthology.orchestra.run.smithi114.stdout: "journaler_write_head_interval": "15", 2024-09-21T12:21:17.535 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "", 2024-09-21T12:21:17.535 INFO:teuthology.orchestra.run.smithi114.stdout: "keyfile": "", 2024-09-21T12:21:17.535 INFO:teuthology.orchestra.run.smithi114.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-21T12:21:17.535 INFO:teuthology.orchestra.run.smithi114.stdout: "kstore_backend": "rocksdb", 2024-09-21T12:21:17.535 INFO:teuthology.orchestra.run.smithi114.stdout: "kstore_default_stripe_size": "65536", 2024-09-21T12:21:17.535 INFO:teuthology.orchestra.run.smithi114.stdout: "kstore_fsck_on_mount": "false", 2024-09-21T12:21:17.535 INFO:teuthology.orchestra.run.smithi114.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-21T12:21:17.535 INFO:teuthology.orchestra.run.smithi114.stdout: "kstore_max_bytes": "67108864", 2024-09-21T12:21:17.535 INFO:teuthology.orchestra.run.smithi114.stdout: "kstore_max_ops": "512", 2024-09-21T12:21:17.536 INFO:teuthology.orchestra.run.smithi114.stdout: "kstore_nid_prealloc": "1024", 2024-09-21T12:21:17.536 INFO:teuthology.orchestra.run.smithi114.stdout: "kstore_onode_map_size": "1024", 2024-09-21T12:21:17.536 INFO:teuthology.orchestra.run.smithi114.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-21T12:21:17.536 INFO:teuthology.orchestra.run.smithi114.stdout: "kstore_sync_submit_transaction": "false", 2024-09-21T12:21:17.536 INFO:teuthology.orchestra.run.smithi114.stdout: "kstore_sync_transaction": "false", 2024-09-21T12:21:17.536 INFO:teuthology.orchestra.run.smithi114.stdout: "librados_thread_count": "2", 2024-09-21T12:21:17.536 INFO:teuthology.orchestra.run.smithi114.stdout: "lockdep": "false", 2024-09-21T12:21:17.536 INFO:teuthology.orchestra.run.smithi114.stdout: "lockdep_force_backtrace": "false", 2024-09-21T12:21:17.536 INFO:teuthology.orchestra.run.smithi114.stdout: "log_coarse_timestamps": "true", 2024-09-21T12:21:17.536 INFO:teuthology.orchestra.run.smithi114.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-21T12:21:17.536 INFO:teuthology.orchestra.run.smithi114.stdout: "log_flush_on_exit": "false", 2024-09-21T12:21:17.536 INFO:teuthology.orchestra.run.smithi114.stdout: "log_graylog_host": "127.0.0.1", 2024-09-21T12:21:17.536 INFO:teuthology.orchestra.run.smithi114.stdout: "log_graylog_port": "12201", 2024-09-21T12:21:17.536 INFO:teuthology.orchestra.run.smithi114.stdout: "log_max_new": "1000", 2024-09-21T12:21:17.537 INFO:teuthology.orchestra.run.smithi114.stdout: "log_max_recent": "10000", 2024-09-21T12:21:17.537 INFO:teuthology.orchestra.run.smithi114.stdout: "log_stderr_prefix": "", 2024-09-21T12:21:17.537 INFO:teuthology.orchestra.run.smithi114.stdout: "log_stop_at_utilization": "0.970000", 2024-09-21T12:21:17.537 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": "true", 2024-09-21T12:21:17.537 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_graylog": "false", 2024-09-21T12:21:17.537 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_journald": "false", 2024-09-21T12:21:17.537 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_stderr": "false", 2024-09-21T12:21:17.537 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_syslog": "false", 2024-09-21T12:21:17.537 INFO:teuthology.orchestra.run.smithi114.stdout: "max_rotating_auth_attempts": "10", 2024-09-21T12:21:17.537 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-21T12:21:17.537 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_action_on_write_error": "1", 2024-09-21T12:21:17.537 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_alternate_name_max": "8192", 2024-09-21T12:21:17.537 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_asio_thread_count": "2", 2024-09-21T12:21:17.537 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_export_pin": "true", 2024-09-21T12:21:17.538 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_fragment_dirs": "true", 2024-09-21T12:21:17.538 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-21T12:21:17.538 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_fragment_interval": "5", 2024-09-21T12:21:17.538 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-21T12:21:17.538 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-21T12:21:17.538 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_interval": "10", 2024-09-21T12:21:17.538 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_max": "-1", 2024-09-21T12:21:17.538 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_max_until": "-1", 2024-09-21T12:21:17.538 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_merge_size": "50", 2024-09-21T12:21:17.538 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_midchunk": "0.300000", 2024-09-21T12:21:17.538 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-21T12:21:17.538 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_min_start": "0.200000", 2024-09-21T12:21:17.538 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_minchunk": "0.001000", 2024-09-21T12:21:17.539 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_mode": "0", 2024-09-21T12:21:17.539 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_need_max": "1.200000", 2024-09-21T12:21:17.539 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_need_min": "0.800000", 2024-09-21T12:21:17.539 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_overload_epochs": "2", 2024-09-21T12:21:17.539 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-21T12:21:17.539 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-21T12:21:17.539 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_split_bits": "3", 2024-09-21T12:21:17.539 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-21T12:21:17.539 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_split_size": "10000", 2024-09-21T12:21:17.539 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-21T12:21:17.539 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_target_decay": "10.000000", 2024-09-21T12:21:17.539 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-21T12:21:17.539 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_beacon_grace": "15.000000", 2024-09-21T12:21:17.539 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_beacon_interval": "4.000000", 2024-09-21T12:21:17.540 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-21T12:21:17.540 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-21T12:21:17.540 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache_mid": "0.700000", 2024-09-21T12:21:17.540 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-21T12:21:17.540 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache_quiesce_delay": "0", 2024-09-21T12:21:17.540 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-21T12:21:17.540 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-21T12:21:17.540 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-21T12:21:17.540 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache_release_free_interval": "10", 2024-09-21T12:21:17.540 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache_reservation": "0.050000", 2024-09-21T12:21:17.540 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-21T12:21:17.540 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache_trim_interval": "1", 2024-09-21T12:21:17.540 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache_trim_threshold": "262144", 2024-09-21T12:21:17.540 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-21T12:21:17.541 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-21T12:21:17.541 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-21T12:21:17.541 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_client_prealloc_inos": "1000", 2024-09-21T12:21:17.541 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-21T12:21:17.541 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_connect_bootstrapping": "false", 2024-09-21T12:21:17.541 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_damage_table_max_entries": "10000", 2024-09-21T12:21:17.541 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-21T12:21:17.541 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_debug_auth_pins": "false", 2024-09-21T12:21:17.541 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_debug_frag": "false", 2024-09-21T12:21:17.541 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_debug_scatterstat": "false", 2024-09-21T12:21:17.541 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_debug_subtrees": "false", 2024-09-21T12:21:17.541 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_decay_halflife": "5.000000", 2024-09-21T12:21:17.541 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_default_dir_hash": "2", 2024-09-21T12:21:17.541 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_defer_session_stale": "true", 2024-09-21T12:21:17.541 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_deny_all_reconnect": "false", 2024-09-21T12:21:17.541 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_dir_keys_per_op": "16384", 2024-09-21T12:21:17.542 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_dir_max_commit_size": "10", 2024-09-21T12:21:17.542 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_dir_max_entries": "0", 2024-09-21T12:21:17.542 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_dir_prefetch": "true", 2024-09-21T12:21:17.542 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-21T12:21:17.542 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-21T12:21:17.542 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_dump_cache_on_map": "false", 2024-09-21T12:21:17.542 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-21T12:21:17.542 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-21T12:21:17.542 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_early_reply": "true", 2024-09-21T12:21:17.542 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_enable_op_tracker": "true", 2024-09-21T12:21:17.542 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_enforce_unique_name": "true", 2024-09-21T12:21:17.542 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-21T12:21:17.542 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-21T12:21:17.542 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_export_ephemeral_random": "true", 2024-09-21T12:21:17.542 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-21T12:21:17.543 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-21T12:21:17.543 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-21T12:21:17.543 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-21T12:21:17.543 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-21T12:21:17.543 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-21T12:21:17.543 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-21T12:21:17.543 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-21T12:21:17.543 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_health_summarize_threshold": "10", 2024-09-21T12:21:17.543 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-21T12:21:17.543 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-21T12:21:17.543 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_inject_health_dummy": "false", 2024-09-21T12:21:17.543 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-21T12:21:17.543 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_inject_migrator_session_race": "false", 2024-09-21T12:21:17.543 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-21T12:21:17.544 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-21T12:21:17.544 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-21T12:21:17.544 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_join_fs": "", 2024-09-21T12:21:17.544 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_journal_format": "1", 2024-09-21T12:21:17.544 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-21T12:21:17.544 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_kill_create_at": "0", 2024-09-21T12:21:17.544 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_kill_dirfrag_at": "0", 2024-09-21T12:21:17.544 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_kill_export_at": "0", 2024-09-21T12:21:17.544 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_kill_import_at": "0", 2024-09-21T12:21:17.544 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_kill_journal_at": "0", 2024-09-21T12:21:17.544 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_kill_journal_expire_at": "0", 2024-09-21T12:21:17.544 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_kill_journal_replay_at": "0", 2024-09-21T12:21:17.544 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_kill_link_at": "0", 2024-09-21T12:21:17.544 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_kill_mdstable_at": "0", 2024-09-21T12:21:17.544 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_kill_openc_at": "0", 2024-09-21T12:21:17.544 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_kill_rename_at": "0", 2024-09-21T12:21:17.544 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_kill_shutdown_at": "0", 2024-09-21T12:21:17.545 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log_event_large_threshold": "524288", 2024-09-21T12:21:17.545 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log_events_per_segment": "1024", 2024-09-21T12:21:17.545 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-21T12:21:17.545 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log_max_events": "-1", 2024-09-21T12:21:17.545 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log_max_segments": "128", 2024-09-21T12:21:17.545 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log_pause": "false", 2024-09-21T12:21:17.545 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log_segment_size": "0", 2024-09-21T12:21:17.545 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-21T12:21:17.545 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-21T12:21:17.545 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-21T12:21:17.545 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log_trim_threshold": "128", 2024-09-21T12:21:17.545 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-21T12:21:17.545 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log_warn_factor": "2.000000", 2024-09-21T12:21:17.545 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_max_caps_per_client": "1048576", 2024-09-21T12:21:17.545 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_max_completed_flushes": "100000", 2024-09-21T12:21:17.546 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_max_completed_requests": "100000", 2024-09-21T12:21:17.546 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_max_export_size": "20971520", 2024-09-21T12:21:17.546 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_max_file_recover": "32", 2024-09-21T12:21:17.546 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_max_purge_files": "64", 2024-09-21T12:21:17.546 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_max_purge_ops": "8192", 2024-09-21T12:21:17.546 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-21T12:21:17.546 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-21T12:21:17.546 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_max_snaps_per_dir": "100", 2024-09-21T12:21:17.546 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_metrics_update_interval": "2", 2024-09-21T12:21:17.546 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_min_caps_per_client": "100", 2024-09-21T12:21:17.546 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_min_caps_working_set": "10000", 2024-09-21T12:21:17.546 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-21T12:21:17.546 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_numa_node": "-1", 2024-09-21T12:21:17.546 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-21T12:21:17.547 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_op_complaint_time": "30.000000", 2024-09-21T12:21:17.547 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_op_history_duration": "600", 2024-09-21T12:21:17.547 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_op_history_size": "20", 2024-09-21T12:21:17.547 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_op_history_slow_op_size": "20", 2024-09-21T12:21:17.547 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-21T12:21:17.547 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_op_log_threshold": "5", 2024-09-21T12:21:17.547 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_ping_grace": "15", 2024-09-21T12:21:17.547 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_ping_interval": "5", 2024-09-21T12:21:17.547 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-21T12:21:17.547 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-21T12:21:17.547 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_recall_max_caps": "30000", 2024-09-21T12:21:17.547 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-21T12:21:17.547 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-21T12:21:17.547 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-21T12:21:17.547 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_recall_warning_threshold": "262144", 2024-09-21T12:21:17.547 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-21T12:21:17.548 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_replay_interval": "1.000000", 2024-09-21T12:21:17.548 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-21T12:21:17.548 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-21T12:21:17.548 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_root_ino_gid": "0", 2024-09-21T12:21:17.548 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_root_ino_uid": "0", 2024-09-21T12:21:17.548 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-21T12:21:17.548 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_scrub_stats_review_period": "1", 2024-09-21T12:21:17.548 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-09-21T12:21:17.548 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-09-21T12:21:17.548 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-21T12:21:17.548 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-21T12:21:17.548 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-21T12:21:17.548 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-21T12:21:17.548 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-21T12:21:17.548 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-21T12:21:17.548 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-21T12:21:17.549 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-21T12:21:17.549 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-21T12:21:17.549 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_shutdown_check": "0", 2024-09-21T12:21:17.549 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_skip_ino": "0", 2024-09-21T12:21:17.549 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-21T12:21:17.549 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_snap_max_uid": "4294967294", 2024-09-21T12:21:17.549 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_snap_min_uid": "0", 2024-09-21T12:21:17.549 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_snap_rstat": "false", 2024-09-21T12:21:17.549 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_standby_replay_damaged": "false", 2024-09-21T12:21:17.549 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_symlink_recovery": "true", 2024-09-21T12:21:17.549 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-21T12:21:17.549 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_thrash_exports": "0", 2024-09-21T12:21:17.549 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_thrash_fragments": "0", 2024-09-21T12:21:17.549 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_tick_interval": "5.000000", 2024-09-21T12:21:17.549 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_valgrind_exit": "false", 2024-09-21T12:21:17.550 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_verify_backtrace": "1", 2024-09-21T12:21:17.550 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_verify_scatter": "false", 2024-09-21T12:21:17.550 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-21T12:21:17.550 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_wipe_sessions": "false", 2024-09-21T12:21:17.550 INFO:teuthology.orchestra.run.smithi114.stdout: "mempool_debug": "false", 2024-09-21T12:21:17.550 INFO:teuthology.orchestra.run.smithi114.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-21T12:21:17.550 INFO:teuthology.orchestra.run.smithi114.stdout: "memstore_device_bytes": "1073741824", 2024-09-21T12:21:17.550 INFO:teuthology.orchestra.run.smithi114.stdout: "memstore_page_set": "false", 2024-09-21T12:21:17.550 INFO:teuthology.orchestra.run.smithi114.stdout: "memstore_page_size": "65536", 2024-09-21T12:21:17.550 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_client_bytes": "134217728", 2024-09-21T12:21:17.550 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_client_messages": "512", 2024-09-21T12:21:17.550 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-21T12:21:17.550 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-21T12:21:17.550 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-21T12:21:17.551 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-21T12:21:17.551 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_disabled_modules": "", 2024-09-21T12:21:17.551 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_enable_op_tracker": "true", 2024-09-21T12:21:17.551 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-21T12:21:17.551 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_max_pg_creating": "1024", 2024-09-21T12:21:17.551 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_max_pg_num_change": "128", 2024-09-21T12:21:17.551 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_mds_bytes": "134217728", 2024-09-21T12:21:17.551 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_mds_messages": "128", 2024-09-21T12:21:17.551 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-21T12:21:17.551 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_mon_bytes": "134217728", 2024-09-21T12:21:17.551 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_mon_messages": "128", 2024-09-21T12:21:17.551 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-21T12:21:17.551 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-21T12:21:17.551 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_op_history_duration": "600", 2024-09-21T12:21:17.552 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_op_history_size": "20", 2024-09-21T12:21:17.552 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-21T12:21:17.552 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-21T12:21:17.552 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_op_log_threshold": "5", 2024-09-21T12:21:17.552 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_osd_bytes": "536870912", 2024-09-21T12:21:17.552 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_osd_messages": "8192", 2024-09-21T12:21:17.552 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_pool": "true", 2024-09-21T12:21:17.552 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-21T12:21:17.552 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_standby_modules": "true", 2024-09-21T12:21:17.552 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_stats_period": "5", 2024-09-21T12:21:17.552 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_stats_threshold": "5", 2024-09-21T12:21:17.552 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_tick_period": "2", 2024-09-21T12:21:17.552 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-21T12:21:17.552 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-21T12:21:17.552 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_allow_pool_delete": "true", 2024-09-21T12:21:17.553 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_allow_pool_size_one": "false", 2024-09-21T12:21:17.553 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_auth_validate_all_caps": "true", 2024-09-21T12:21:17.553 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-21T12:21:17.553 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-21T12:21:17.553 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_client_bytes": "104857600", 2024-09-21T12:21:17.553 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_client_directed_command_retry": "2", 2024-09-21T12:21:17.553 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-21T12:21:17.553 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-21T12:21:17.553 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-21T12:21:17.553 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-21T12:21:17.553 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_client_hunt_parallel": "3", 2024-09-21T12:21:17.553 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_client_log_interval": "1.000000", 2024-09-21T12:21:17.553 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-21T12:21:17.553 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_client_ping_interval": "10.000000", 2024-09-21T12:21:17.553 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-21T12:21:17.554 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_client_target_rank": "-1", 2024-09-21T12:21:17.554 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-21T12:21:17.554 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-21T12:21:17.554 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-21T12:21:17.554 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_cluster_log_level": "debug", 2024-09-21T12:21:17.554 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_cluster_log_to_file": "true", 2024-09-21T12:21:17.554 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-21T12:21:17.554 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-21T12:21:17.554 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-21T12:21:17.554 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_cluster_log_to_journald": "true", 2024-09-21T12:21:17.554 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-21T12:21:17.554 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-21T12:21:17.554 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-21T12:21:17.554 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_compact_on_bootstrap": "false", 2024-09-21T12:21:17.555 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_compact_on_start": "false", 2024-09-21T12:21:17.555 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_compact_on_trim": "true", 2024-09-21T12:21:17.555 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-21T12:21:17.555 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-21T12:21:17.555 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-21T12:21:17.555 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_cpu_threads": "4", 2024-09-21T12:21:17.555 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_crush_min_required_version": "hammer", 2024-09-21T12:21:17.555 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_daemon_bytes": "419430400", 2024-09-21T12:21:17.555 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-21T12:21:17.555 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_data_avail_crit": "5", 2024-09-21T12:21:17.555 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_data_avail_warn": "5", 2024-09-21T12:21:17.555 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_data_size_warn": "16106127360", 2024-09-21T12:21:17.555 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-21T12:21:17.555 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-21T12:21:17.555 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_debug_dump_json": "false", 2024-09-21T12:21:17.555 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-21T12:21:17.556 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_debug_dump_transactions": "false", 2024-09-21T12:21:17.556 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_debug_extra_checks": "false", 2024-09-21T12:21:17.556 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-21T12:21:17.556 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-21T12:21:17.556 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_debug_no_require_reef": "false", 2024-09-21T12:21:17.556 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_debug_no_require_squid": "false", 2024-09-21T12:21:17.556 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-21T12:21:17.556 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-21T12:21:17.556 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-21T12:21:17.556 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_down_mkfs_grace": "60", 2024-09-21T12:21:17.556 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_down_uptime_grace": "60", 2024-09-21T12:21:17.556 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_election_default_strategy": "1", 2024-09-21T12:21:17.556 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_election_timeout": "5.000000", 2024-09-21T12:21:17.556 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-21T12:21:17.556 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_elector_ping_divisor": "2", 2024-09-21T12:21:17.556 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-21T12:21:17.557 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_enable_op_tracker": "true", 2024-09-21T12:21:17.557 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_fake_pool_delete": "false", 2024-09-21T12:21:17.557 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_force_quorum_join": "false", 2024-09-21T12:21:17.557 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-21T12:21:17.557 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_globalid_prealloc": "10000", 2024-09-21T12:21:17.557 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_health_detail_to_clog": "true", 2024-09-21T12:21:17.557 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_health_log_update_period": "5", 2024-09-21T12:21:17.557 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_health_max_detail": "50", 2024-09-21T12:21:17.557 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_health_to_clog": "true", 2024-09-21T12:21:17.557 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_health_to_clog_interval": "600", 2024-09-21T12:21:17.557 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-21T12:21:17.557 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_host": "[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0]", 2024-09-21T12:21:17.557 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_host_override": "", 2024-09-21T12:21:17.557 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_initial_members": "", 2024-09-21T12:21:17.557 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-21T12:21:17.557 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-21T12:21:17.558 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-21T12:21:17.558 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-21T12:21:17.558 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-21T12:21:17.558 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_lease": "5.000000", 2024-09-21T12:21:17.558 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-21T12:21:17.558 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-21T12:21:17.558 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_log_full_interval": "50", 2024-09-21T12:21:17.558 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_log_max": "10000", 2024-09-21T12:21:17.558 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_log_max_summary": "50", 2024-09-21T12:21:17.558 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-21T12:21:17.558 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_max_log_epochs": "500", 2024-09-21T12:21:17.558 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-21T12:21:17.558 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-21T12:21:17.558 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_max_nvmeof_epochs": "500", 2024-09-21T12:21:17.558 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_max_osd": "10000", 2024-09-21T12:21:17.559 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_max_pg_per_osd": "10000", 2024-09-21T12:21:17.559 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_max_pool_pg_num": "65536", 2024-09-21T12:21:17.559 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-21T12:21:17.559 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-21T12:21:17.559 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_mds_force_trim_to": "0", 2024-09-21T12:21:17.559 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_mds_skip_sanity": "false", 2024-09-21T12:21:17.559 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_memory_autotune": "true", 2024-09-21T12:21:17.559 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_memory_target": "2147483648", 2024-09-21T12:21:17.559 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_mgr_beacon_grace": "30", 2024-09-21T12:21:17.559 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-21T12:21:17.559 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_mgr_digest_period": "5", 2024-09-21T12:21:17.559 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_mgr_inactive_grace": "60", 2024-09-21T12:21:17.559 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-21T12:21:17.559 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-21T12:21:17.559 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_min_osdmap_epochs": "50", 2024-09-21T12:21:17.560 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-09-21T12:21:17.560 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-09-21T12:21:17.560 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_op_complaint_time": "30", 2024-09-21T12:21:17.560 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_op_history_duration": "600", 2024-09-21T12:21:17.560 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_op_history_size": "20", 2024-09-21T12:21:17.560 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_op_history_slow_op_size": "20", 2024-09-21T12:21:17.560 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-21T12:21:17.560 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_op_log_threshold": "5", 2024-09-21T12:21:17.560 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-21T12:21:17.560 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-21T12:21:17.560 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-21T12:21:17.560 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_auto_mark_in": "false", 2024-09-21T12:21:17.560 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-21T12:21:17.560 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-21T12:21:17.560 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-21T12:21:17.560 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_cache_size": "500", 2024-09-21T12:21:17.560 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-21T12:21:17.561 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-21T12:21:17.561 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-21T12:21:17.561 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_down_out_interval": "600", 2024-09-21T12:21:17.561 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-21T12:21:17.561 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-21T12:21:17.561 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_force_trim_to": "0", 2024-09-21T12:21:17.561 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-21T12:21:17.561 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-21T12:21:17.561 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-21T12:21:17.561 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-21T12:21:17.561 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-21T12:21:17.561 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-21T12:21:17.561 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-21T12:21:17.561 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_min_down_reporters": "2", 2024-09-21T12:21:17.562 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-21T12:21:17.562 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-21T12:21:17.562 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-21T12:21:17.562 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-21T12:21:17.562 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-21T12:21:17.562 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-21T12:21:17.562 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_report_timeout": "900", 2024-09-21T12:21:17.562 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-21T12:21:17.562 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-21T12:21:17.563 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-21T12:21:17.563 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-21T12:21:17.563 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-21T12:21:17.564 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osdmap_full_prune_interval": "2", 2024-09-21T12:21:17.564 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osdmap_full_prune_min": "15", 2024-09-21T12:21:17.564 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-09-21T12:21:17.564 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-21T12:21:17.564 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_pg_stuck_threshold": "60", 2024-09-21T12:21:17.564 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-21T12:21:17.564 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-21T12:21:17.564 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-21T12:21:17.564 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-21T12:21:17.564 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-21T12:21:17.564 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-21T12:21:17.564 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_probe_timeout": "2.000000", 2024-09-21T12:21:17.565 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_reweight_max_change": "0.050000", 2024-09-21T12:21:17.565 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_reweight_max_osds": "4", 2024-09-21T12:21:17.565 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-21T12:21:17.565 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-21T12:21:17.565 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-21T12:21:17.565 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-21T12:21:17.565 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-21T12:21:17.565 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_scrub_interval": "86400", 2024-09-21T12:21:17.565 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_scrub_max_keys": "100", 2024-09-21T12:21:17.565 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_scrub_timeout": "300", 2024-09-21T12:21:17.565 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_session_timeout": "300", 2024-09-21T12:21:17.565 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_smart_report_timeout": "5", 2024-09-21T12:21:17.565 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_stat_smooth_intervals": "6", 2024-09-21T12:21:17.565 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-21T12:21:17.566 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_stretch_pool_min_size": "2", 2024-09-21T12:21:17.566 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_stretch_pool_size": "4", 2024-09-21T12:21:17.566 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-21T12:21:17.566 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-21T12:21:17.566 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_sync_debug": "false", 2024-09-21T12:21:17.566 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-21T12:21:17.566 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-21T12:21:17.566 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_sync_provider_kill_at": "0", 2024-09-21T12:21:17.566 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_sync_requester_kill_at": "0", 2024-09-21T12:21:17.566 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_sync_timeout": "60.000000", 2024-09-21T12:21:17.566 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_target_pg_per_osd": "100", 2024-09-21T12:21:17.566 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_tick_interval": "5", 2024-09-21T12:21:17.566 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_timecheck_interval": "300.000000", 2024-09-21T12:21:17.567 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-21T12:21:17.567 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_use_min_delay_socket": "false", 2024-09-21T12:21:17.567 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_older_version_delay": "604800", 2024-09-21T12:21:17.567 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-21T12:21:17.567 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-21T12:21:17.567 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-21T12:21:17.567 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_filestore_osds": "true", 2024-09-21T12:21:17.567 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-21T12:21:17.567 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-21T12:21:17.567 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-21T12:21:17.567 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_misplaced": "false", 2024-09-21T12:21:17.567 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-21T12:21:17.567 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_older_version": "true", 2024-09-21T12:21:17.567 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-21T12:21:17.568 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-21T12:21:17.568 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-09-21T12:21:17.568 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-21T12:21:17.568 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-21T12:21:17.568 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-21T12:21:17.568 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-21T12:21:17.568 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-21T12:21:17.568 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-21T12:21:17.568 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": "", 2024-09-21T12:21:17.568 INFO:teuthology.orchestra.run.smithi114.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-21T12:21:17.568 INFO:teuthology.orchestra.run.smithi114.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-21T12:21:17.568 INFO:teuthology.orchestra.run.smithi114.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-21T12:21:17.568 INFO:teuthology.orchestra.run.smithi114.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-21T12:21:17.568 INFO:teuthology.orchestra.run.smithi114.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-21T12:21:17.568 INFO:teuthology.orchestra.run.smithi114.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-21T12:21:17.569 INFO:teuthology.orchestra.run.smithi114.stdout: "motr_tracing_enabled": "false", 2024-09-21T12:21:17.569 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_op_threads": "3", 2024-09-21T12:21:17.569 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-21T12:21:17.569 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_cm": "false", 2024-09-21T12:21:17.569 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_device_name": "", 2024-09-21T12:21:17.569 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_dscp": "96", 2024-09-21T12:21:17.569 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-21T12:21:17.569 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-21T12:21:17.569 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_local_gid": "", 2024-09-21T12:21:17.569 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-21T12:21:17.569 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_port_num": "1", 2024-09-21T12:21:17.569 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-21T12:21:17.569 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-21T12:21:17.569 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-21T12:21:17.570 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-21T12:21:17.570 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_sl": "3", 2024-09-21T12:21:17.570 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_support_srq": "true", 2024-09-21T12:21:17.570 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_type": "ib", 2024-09-21T12:21:17.570 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_reap_threshold": "5", 2024-09-21T12:21:17.570 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_bind_before_connect": "false", 2024-09-21T12:21:17.570 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_bind_ipv4": "true", 2024-09-21T12:21:17.570 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_bind_ipv6": "false", 2024-09-21T12:21:17.570 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_bind_msgr1": "true", 2024-09-21T12:21:17.570 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_bind_msgr2": "true", 2024-09-21T12:21:17.570 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_bind_port_max": "7568", 2024-09-21T12:21:17.570 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_bind_port_min": "6800", 2024-09-21T12:21:17.570 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-21T12:21:17.570 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_bind_retry_count": "3", 2024-09-21T12:21:17.571 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_bind_retry_delay": "5", 2024-09-21T12:21:17.571 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_blackhole_client": "false", 2024-09-21T12:21:17.571 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_blackhole_mds": "false", 2024-09-21T12:21:17.571 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_blackhole_mgr": "false", 2024-09-21T12:21:17.571 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_blackhole_mon": "false", 2024-09-21T12:21:17.571 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_blackhole_osd": "false", 2024-09-21T12:21:17.571 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_client_mode": "crc secure", 2024-09-21T12:21:17.571 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-09-21T12:21:17.571 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_cluster_mode": "crc secure", 2024-09-21T12:21:17.571 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_cluster_type": "", 2024-09-21T12:21:17.571 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_compress_secure": "false", 2024-09-21T12:21:17.571 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_connection_idle_timeout": "900", 2024-09-21T12:21:17.571 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_connection_ready_timeout": "10", 2024-09-21T12:21:17.572 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_crc_data": "true", 2024-09-21T12:21:17.572 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_crc_header": "true", 2024-09-21T12:21:17.572 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_die_on_bad_msg": "false", 2024-09-21T12:21:17.572 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_die_on_bug": "true", 2024-09-21T12:21:17.572 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_die_on_old_message": "true", 2024-09-21T12:21:17.572 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_die_on_skipped_message": "false", 2024-09-21T12:21:17.572 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-21T12:21:17.572 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-21T12:21:17.572 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dpdk_coremask": "0xF", 2024-09-21T12:21:17.572 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-21T12:21:17.572 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-21T12:21:17.572 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dpdk_enable_tso": "true", 2024-09-21T12:21:17.572 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-21T12:21:17.572 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-21T12:21:17.572 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dpdk_hugepages": "", 2024-09-21T12:21:17.573 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-21T12:21:17.573 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-21T12:21:17.573 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dpdk_lro": "true", 2024-09-21T12:21:17.573 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dpdk_memory_channel": "4", 2024-09-21T12:21:17.573 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-21T12:21:17.573 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dpdk_pmd": "", 2024-09-21T12:21:17.573 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dpdk_port_id": "0", 2024-09-21T12:21:17.573 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-21T12:21:17.573 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-21T12:21:17.573 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dump_on_send": "false", 2024-09-21T12:21:17.573 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_initial_backoff": "0.200000", 2024-09-21T12:21:17.573 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_inject_delay_max": "1.000000", 2024-09-21T12:21:17.573 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-21T12:21:17.573 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_inject_delay_type": "", 2024-09-21T12:21:17.574 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-21T12:21:17.574 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_inject_network_congestion": "0", 2024-09-21T12:21:17.574 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_inject_socket_failures": "0", 2024-09-21T12:21:17.574 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_learn_addr_from_peer": "true", 2024-09-21T12:21:17.574 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_max_accept_failures": "4", 2024-09-21T12:21:17.574 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_max_backoff": "15.000000", 2024-09-21T12:21:17.574 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_mon_client_mode": "secure crc", 2024-09-21T12:21:17.574 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-21T12:21:17.574 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_mon_service_mode": "secure crc", 2024-09-21T12:21:17.574 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_osd_compress_min_size": "1024", 2024-09-21T12:21:17.574 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_osd_compress_mode": "none", 2024-09-21T12:21:17.574 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-21T12:21:17.574 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-21T12:21:17.575 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_pq_min_cost": "65536", 2024-09-21T12:21:17.575 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_public_type": "", 2024-09-21T12:21:17.575 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_service_mode": "crc secure", 2024-09-21T12:21:17.575 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_tcp_listen_backlog": "512", 2024-09-21T12:21:17.575 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_tcp_nodelay": "true", 2024-09-21T12:21:17.575 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-21T12:21:17.575 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_tcp_rcvbuf": "0", 2024-09-21T12:21:17.575 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_time_events_min_wait_interval": "1000", 2024-09-21T12:21:17.575 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_type": "async+posix", 2024-09-21T12:21:17.575 INFO:teuthology.orchestra.run.smithi114.stdout: "no_config_file": "false", 2024-09-21T12:21:17.575 INFO:teuthology.orchestra.run.smithi114.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-09-21T12:21:17.575 INFO:teuthology.orchestra.run.smithi114.stdout: "nvmeof_mon_client_tick_period": "2", 2024-09-21T12:21:17.575 INFO:teuthology.orchestra.run.smithi114.stdout: "objecter_completion_locks_per_session": "32", 2024-09-21T12:21:17.575 INFO:teuthology.orchestra.run.smithi114.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-21T12:21:17.575 INFO:teuthology.orchestra.run.smithi114.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-21T12:21:17.576 INFO:teuthology.orchestra.run.smithi114.stdout: "objecter_inflight_ops": "1024", 2024-09-21T12:21:17.576 INFO:teuthology.orchestra.run.smithi114.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-21T12:21:17.576 INFO:teuthology.orchestra.run.smithi114.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-21T12:21:17.576 INFO:teuthology.orchestra.run.smithi114.stdout: "objecter_tick_interval": "5.000000", 2024-09-21T12:21:17.576 INFO:teuthology.orchestra.run.smithi114.stdout: "objecter_timeout": "10.000000", 2024-09-21T12:21:17.576 INFO:teuthology.orchestra.run.smithi114.stdout: "objectstore_blackhole": "false", 2024-09-21T12:21:17.576 INFO:teuthology.orchestra.run.smithi114.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-21T12:21:17.576 INFO:teuthology.orchestra.run.smithi114.stdout: "openssl_engine_opts": "", 2024-09-21T12:21:17.576 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_agent_delay_time": "5.000000", 2024-09-21T12:21:17.576 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_agent_hist_halflife": "1000", 2024-09-21T12:21:17.576 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_agent_max_low_ops": "2", 2024-09-21T12:21:17.576 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_agent_max_ops": "4", 2024-09-21T12:21:17.576 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-21T12:21:17.576 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-21T12:21:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_agent_slop": "0.020000", 2024-09-21T12:21:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-21T12:21:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-21T12:21:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_asio_thread_count": "2", 2024-09-21T12:21:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_async_recovery_min_cost": "100", 2024-09-21T12:21:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-21T12:21:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-21T12:21:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_backfill_scan_max": "512", 2024-09-21T12:21:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_backfill_scan_min": "64", 2024-09-21T12:21:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_backoff_on_degraded": "false", 2024-09-21T12:21:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_backoff_on_peering": "false", 2024-09-21T12:21:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_backoff_on_unfound": "true", 2024-09-21T12:21:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_beacon_report_interval": "300", 2024-09-21T12:21:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_bench_duration": "30", 2024-09-21T12:21:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-21T12:21:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_bench_max_block_size": "67108864", 2024-09-21T12:21:17.578 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-21T12:21:17.578 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_blkin_trace_all": "false", 2024-09-21T12:21:17.578 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-21T12:21:17.578 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-21T12:21:17.578 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-21T12:21:17.578 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-21T12:21:17.578 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_check_for_log_corruption": "false", 2024-09-21T12:21:17.578 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-21T12:21:17.578 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-21T12:21:17.578 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-21T12:21:17.578 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-21T12:21:17.578 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_class_update_on_start": "true", 2024-09-21T12:21:17.578 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_client_message_cap": "256", 2024-09-21T12:21:17.578 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_client_message_size_cap": "524288000", 2024-09-21T12:21:17.578 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_client_op_priority": "63", 2024-09-21T12:21:17.579 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_client_watch_timeout": "30", 2024-09-21T12:21:17.579 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_command_max_records": "256", 2024-09-21T12:21:17.579 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_compact_on_start": "false", 2024-09-21T12:21:17.579 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-21T12:21:17.579 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_crush_chooseleaf_type": "1", 2024-09-21T12:21:17.579 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-21T12:21:17.579 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_crush_update_on_start": "true", 2024-09-21T12:21:17.579 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_crush_update_weight_set": "true", 2024-09-21T12:21:17.579 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-21T12:21:17.579 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-21T12:21:17.579 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-21T12:21:17.579 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-21T12:21:17.579 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-21T12:21:17.579 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_feed_pullee": "-1", 2024-09-21T12:21:17.579 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-21T12:21:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-21T12:21:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-21T12:21:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_misdirected_ops": "false", 2024-09-21T12:21:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_no_acting_change": "false", 2024-09-21T12:21:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_no_purge_strays": "false", 2024-09-21T12:21:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_op_order": "false", 2024-09-21T12:21:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-21T12:21:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-21T12:21:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-21T12:21:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-21T12:21:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_shutdown": "false", 2024-09-21T12:21:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-21T12:21:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-21T12:21:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_trim_objects": "false", 2024-09-21T12:21:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-21T12:21:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-21T12:21:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_verify_snaps": "false", 2024-09-21T12:21:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-21T12:21:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-21T12:21:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_deep_scrub_keys": "1024", 2024-09-21T12:21:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-21T12:21:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-21T12:21:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-21T12:21:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_deep_scrub_stride": "524288", 2024-09-21T12:21:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-21T12:21:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_default_data_pool_replay_window": "45", 2024-09-21T12:21:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_default_notify_timeout": "30", 2024-09-21T12:21:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_delete_sleep": "0.000000", 2024-09-21T12:21:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-21T12:21:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-21T12:21:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-21T12:21:17.582 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_discard_disconnected_ops": "true", 2024-09-21T12:21:17.582 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_ec_partial_reads": "true", 2024-09-21T12:21:17.582 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_enable_op_tracker": "true", 2024-09-21T12:21:17.582 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-21T12:21:17.582 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-21T12:21:17.582 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-21T12:21:17.582 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-21T12:21:17.582 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_fast_info": "true", 2024-09-21T12:21:17.582 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_fast_shutdown": "true", 2024-09-21T12:21:17.582 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-21T12:21:17.582 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-21T12:21:17.582 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-21T12:21:17.582 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-21T12:21:17.582 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-21T12:21:17.582 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_function_tracing": "false", 2024-09-21T12:21:17.582 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_heartbeat_grace": "20", 2024-09-21T12:21:17.583 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_heartbeat_interval": "6", 2024-09-21T12:21:17.583 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-21T12:21:17.583 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_heartbeat_min_peers": "10", 2024-09-21T12:21:17.583 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_heartbeat_min_size": "2000", 2024-09-21T12:21:17.583 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_heartbeat_stale": "600", 2024-09-21T12:21:17.583 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-21T12:21:17.583 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_hit_set_max_size": "100000", 2024-09-21T12:21:17.583 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_hit_set_min_size": "1000", 2024-09-21T12:21:17.583 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-21T12:21:17.583 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-21T12:21:17.583 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-21T12:21:17.583 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-21T12:21:17.583 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-21T12:21:17.583 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-21T12:21:17.583 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_journal_size": "5120", 2024-09-21T12:21:17.584 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_kill_backfill_at": "0", 2024-09-21T12:21:17.584 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-21T12:21:17.584 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_map_cache_size": "50", 2024-09-21T12:21:17.584 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_map_dedup": "true", 2024-09-21T12:21:17.584 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_map_message_max": "40", 2024-09-21T12:21:17.584 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-21T12:21:17.584 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_map_share_max_epochs": "40", 2024-09-21T12:21:17.584 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_attr_name_len": "100", 2024-09-21T12:21:17.584 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_attr_size": "0", 2024-09-21T12:21:17.584 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_backfills": "1", 2024-09-21T12:21:17.584 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_markdown_count": "5", 2024-09-21T12:21:17.584 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_markdown_period": "600", 2024-09-21T12:21:17.584 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_object_name_len": "2048", 2024-09-21T12:21:17.584 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_object_namespace_len": "256", 2024-09-21T12:21:17.584 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_object_size": "134217728", 2024-09-21T12:21:17.584 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-21T12:21:17.584 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-21T12:21:17.585 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_pg_blocked_by": "16", 2024-09-21T12:21:17.585 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_pg_log_entries": "10000", 2024-09-21T12:21:17.585 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-21T12:21:17.585 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_pgls": "1024", 2024-09-21T12:21:17.585 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_push_cost": "8388608", 2024-09-21T12:21:17.585 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_push_objects": "10", 2024-09-21T12:21:17.585 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_scrubs": "3", 2024-09-21T12:21:17.585 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-21T12:21:17.585 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_trimming_pgs": "2", 2024-09-21T12:21:17.585 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_write_op_reply_len": "64", 2024-09-21T12:21:17.585 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_write_size": "90", 2024-09-21T12:21:17.585 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-21T12:21:17.585 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-21T12:21:17.585 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-21T12:21:17.585 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-21T12:21:17.585 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-21T12:21:17.586 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-21T12:21:17.586 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-21T12:21:17.586 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-21T12:21:17.586 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_profile": "balanced", 2024-09-21T12:21:17.586 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-21T12:21:17.586 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-21T12:21:17.586 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-21T12:21:17.586 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-21T12:21:17.586 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-21T12:21:17.586 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-21T12:21:17.586 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-21T12:21:17.586 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-21T12:21:17.586 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-21T12:21:17.586 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-21T12:21:17.586 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-21T12:21:17.587 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_memory_base": "805306368", 2024-09-21T12:21:17.587 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_memory_cache_min": "134217728", 2024-09-21T12:21:17.587 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-21T12:21:17.587 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-21T12:21:17.587 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_memory_target": "4294967296", 2024-09-21T12:21:17.587 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_memory_target_autotune": "false", 2024-09-21T12:21:17.587 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-21T12:21:17.587 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_min_pg_log_entries": "250", 2024-09-21T12:21:17.587 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_min_recovery_priority": "0", 2024-09-21T12:21:17.587 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-21T12:21:17.587 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-21T12:21:17.587 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mon_report_interval": "5", 2024-09-21T12:21:17.587 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-21T12:21:17.587 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-21T12:21:17.587 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_num_cache_shards": "32", 2024-09-21T12:21:17.588 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_num_op_tracker_shard": "32", 2024-09-21T12:21:17.588 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_numa_auto_affinity": "true", 2024-09-21T12:21:17.588 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_numa_node": "-1", 2024-09-21T12:21:17.588 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_numa_prefer_iface": "true", 2024-09-21T12:21:17.588 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-21T12:21:17.588 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_objecter_finishers": "1", 2024-09-21T12:21:17.588 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_objectstore": "bluestore", 2024-09-21T12:21:17.588 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_objectstore_fuse": "false", 2024-09-21T12:21:17.588 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_objectstore_ideal_list_max": "64", 2024-09-21T12:21:17.588 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_objectstore_tracing": "false", 2024-09-21T12:21:17.588 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_complaint_time": "30.000000", 2024-09-21T12:21:17.588 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_history_duration": "600", 2024-09-21T12:21:17.588 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_history_size": "20", 2024-09-21T12:21:17.588 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_history_slow_op_size": "20", 2024-09-21T12:21:17.588 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-21T12:21:17.589 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_log_threshold": "5", 2024-09-21T12:21:17.589 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_num_shards": "0", 2024-09-21T12:21:17.589 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_num_shards_hdd": "5", 2024-09-21T12:21:17.589 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_num_shards_ssd": "8", 2024-09-21T12:21:17.589 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-21T12:21:17.589 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-21T12:21:17.589 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-21T12:21:17.589 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-21T12:21:17.589 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_pq_min_cost": "65536", 2024-09-21T12:21:17.589 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-21T12:21:17.589 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_queue_cut_off": "high", 2024-09-21T12:21:17.589 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-21T12:21:17.589 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_thread_timeout": "15", 2024-09-21T12:21:17.589 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_open_classes_on_start": "true", 2024-09-21T12:21:17.589 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_os_flags": "0", 2024-09-21T12:21:17.589 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_peering_op_priority": "255", 2024-09-21T12:21:17.589 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pg_delete_cost": "1048576", 2024-09-21T12:21:17.590 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pg_delete_priority": "5", 2024-09-21T12:21:17.590 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-21T12:21:17.590 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-21T12:21:17.590 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-21T12:21:17.590 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pg_log_trim_max": "10000", 2024-09-21T12:21:17.590 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pg_log_trim_min": "100", 2024-09-21T12:21:17.590 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-21T12:21:17.590 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-21T12:21:17.590 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-21T12:21:17.590 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-21T12:21:17.590 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-21T12:21:17.590 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-21T12:21:17.590 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-21T12:21:17.590 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-21T12:21:17.590 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-21T12:21:17.591 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-21T12:21:17.591 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_crimson": "false", 2024-09-21T12:21:17.591 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-21T12:21:17.591 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-21T12:21:17.591 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-21T12:21:17.591 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-21T12:21:17.591 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-21T12:21:17.591 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-21T12:21:17.591 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-21T12:21:17.591 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-21T12:21:17.591 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_flags": "0", 2024-09-21T12:21:17.591 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-21T12:21:17.591 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_min_size": "0", 2024-09-21T12:21:17.591 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-21T12:21:17.591 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_pg_num": "32", 2024-09-21T12:21:17.591 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_pgp_num": "0", 2024-09-21T12:21:17.592 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-21T12:21:17.592 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_read_ratio": "70", 2024-09-21T12:21:17.592 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_size": "3", 2024-09-21T12:21:17.592 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_type": "replicated", 2024-09-21T12:21:17.592 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-21T12:21:17.592 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-21T12:21:17.592 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_push_per_object_cost": "1000", 2024-09-21T12:21:17.592 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-21T12:21:17.592 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recover_clone_overlap": "true", 2024-09-21T12:21:17.592 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-21T12:21:17.592 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_cost": "20971520", 2024-09-21T12:21:17.592 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-21T12:21:17.592 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_max_active": "0", 2024-09-21T12:21:17.592 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-21T12:21:17.592 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-21T12:21:17.593 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-21T12:21:17.593 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-21T12:21:17.593 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_max_single_start": "1", 2024-09-21T12:21:17.593 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_op_priority": "3", 2024-09-21T12:21:17.593 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-21T12:21:17.593 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_priority": "5", 2024-09-21T12:21:17.593 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-21T12:21:17.593 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_sleep": "0.000000", 2024-09-21T12:21:17.593 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-21T12:21:17.593 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-21T12:21:17.593 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-21T12:21:17.593 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_repair_during_recovery": "false", 2024-09-21T12:21:17.593 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_requested_scrub_priority": "5", 2024-09-21T12:21:17.593 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-21T12:21:17.594 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-21T12:21:17.594 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_auto_repair": "false", 2024-09-21T12:21:17.594 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-21T12:21:17.594 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-21T12:21:17.594 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_begin_hour": "0", 2024-09-21T12:21:17.594 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_begin_week_day": "0", 2024-09-21T12:21:17.594 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_chunk_max": "15", 2024-09-21T12:21:17.594 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_chunk_min": "5", 2024-09-21T12:21:17.594 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_cost": "52428800", 2024-09-21T12:21:17.594 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-21T12:21:17.594 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_during_recovery": "false", 2024-09-21T12:21:17.594 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_end_hour": "0", 2024-09-21T12:21:17.594 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_end_week_day": "0", 2024-09-21T12:21:17.594 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_event_cost": "4096", 2024-09-21T12:21:17.594 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-21T12:21:17.595 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-21T12:21:17.595 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_invalid_stats": "true", 2024-09-21T12:21:17.595 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-21T12:21:17.595 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-21T12:21:17.595 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_max_preemptions": "5", 2024-09-21T12:21:17.595 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-21T12:21:17.595 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_priority": "5", 2024-09-21T12:21:17.595 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_retry_after_noscrub": "60", 2024-09-21T12:21:17.595 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_retry_delay": "30", 2024-09-21T12:21:17.595 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_retry_new_interval": "10", 2024-09-21T12:21:17.595 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_retry_pg_state": "60", 2024-09-21T12:21:17.595 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_retry_trimming": "10", 2024-09-21T12:21:17.595 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_sleep": "0.000000", 2024-09-21T12:21:17.595 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-21T12:21:17.596 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-21T12:21:17.596 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-21T12:21:17.596 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-09-21T12:21:17.596 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_skip_data_digest": "false", 2024-09-21T12:21:17.596 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_smart_report_timeout": "5", 2024-09-21T12:21:17.596 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_snap_trim_cost": "1048576", 2024-09-21T12:21:17.596 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_snap_trim_priority": "5", 2024-09-21T12:21:17.596 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-21T12:21:17.596 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-21T12:21:17.596 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-21T12:21:17.596 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-21T12:21:17.596 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-21T12:21:17.596 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-21T12:21:17.596 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-21T12:21:17.596 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_target_transaction_size": "30", 2024-09-21T12:21:17.597 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-21T12:21:17.597 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-21T12:21:17.597 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-21T12:21:17.597 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-21T12:21:17.597 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-21T12:21:17.597 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-21T12:21:17.597 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-21T12:21:17.597 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-21T12:21:17.597 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-21T12:21:17.597 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-21T12:21:17.597 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_tracing": "false", 2024-09-21T12:21:17.597 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_use_stale_snap": "false", 2024-09-21T12:21:17.597 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-21T12:21:17.597 INFO:teuthology.orchestra.run.smithi114.stdout: "osdc_blkin_trace_all": "false", 2024-09-21T12:21:17.598 INFO:teuthology.orchestra.run.smithi114.stdout: "paxos_kill_at": "0", 2024-09-21T12:21:17.598 INFO:teuthology.orchestra.run.smithi114.stdout: "paxos_max_join_drift": "10", 2024-09-21T12:21:17.598 INFO:teuthology.orchestra.run.smithi114.stdout: "paxos_min": "500", 2024-09-21T12:21:17.598 INFO:teuthology.orchestra.run.smithi114.stdout: "paxos_min_wait": "0.050000", 2024-09-21T12:21:17.598 INFO:teuthology.orchestra.run.smithi114.stdout: "paxos_propose_interval": "1.000000", 2024-09-21T12:21:17.598 INFO:teuthology.orchestra.run.smithi114.stdout: "paxos_service_trim_max": "500", 2024-09-21T12:21:17.598 INFO:teuthology.orchestra.run.smithi114.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-21T12:21:17.598 INFO:teuthology.orchestra.run.smithi114.stdout: "paxos_service_trim_min": "10", 2024-09-21T12:21:17.598 INFO:teuthology.orchestra.run.smithi114.stdout: "paxos_stash_full_interval": "25", 2024-09-21T12:21:17.598 INFO:teuthology.orchestra.run.smithi114.stdout: "paxos_trim_max": "500", 2024-09-21T12:21:17.598 INFO:teuthology.orchestra.run.smithi114.stdout: "paxos_trim_min": "250", 2024-09-21T12:21:17.598 INFO:teuthology.orchestra.run.smithi114.stdout: "perf": "true", 2024-09-21T12:21:17.598 INFO:teuthology.orchestra.run.smithi114.stdout: "pid_file": "", 2024-09-21T12:21:17.598 INFO:teuthology.orchestra.run.smithi114.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-21T12:21:17.599 INFO:teuthology.orchestra.run.smithi114.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-21T12:21:17.599 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "-", 2024-09-21T12:21:17.599 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrv": "", 2024-09-21T12:21:17.599 INFO:teuthology.orchestra.run.smithi114.stdout: "public_bind_addr": "-", 2024-09-21T12:21:17.599 INFO:teuthology.orchestra.run.smithi114.stdout: "public_network": "", 2024-09-21T12:21:17.599 INFO:teuthology.orchestra.run.smithi114.stdout: "public_network_interface": "", 2024-09-21T12:21:17.599 INFO:teuthology.orchestra.run.smithi114.stdout: "qat_compressor_busy_polling": "false", 2024-09-21T12:21:17.599 INFO:teuthology.orchestra.run.smithi114.stdout: "qat_compressor_enabled": "false", 2024-09-21T12:21:17.599 INFO:teuthology.orchestra.run.smithi114.stdout: "qat_compressor_session_max_number": "256", 2024-09-21T12:21:17.599 INFO:teuthology.orchestra.run.smithi114.stdout: "rados_mon_op_timeout": "0", 2024-09-21T12:21:17.599 INFO:teuthology.orchestra.run.smithi114.stdout: "rados_osd_op_timeout": "0", 2024-09-21T12:21:17.599 INFO:teuthology.orchestra.run.smithi114.stdout: "rados_tracing": "false", 2024-09-21T12:21:17.599 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_atime_update_interval": "60", 2024-09-21T12:21:17.599 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-21T12:21:17.600 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_balance_parent_reads": "false", 2024-09-21T12:21:17.600 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_balance_snap_reads": "false", 2024-09-21T12:21:17.600 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_blkin_trace_all": "false", 2024-09-21T12:21:17.600 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-21T12:21:17.600 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-21T12:21:17.600 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_cache": "true", 2024-09-21T12:21:17.600 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-21T12:21:17.600 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-21T12:21:17.600 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-21T12:21:17.600 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-21T12:21:17.600 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_cache_policy": "writearound", 2024-09-21T12:21:17.600 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_cache_size": "33554432", 2024-09-21T12:21:17.600 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-21T12:21:17.600 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-21T12:21:17.601 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_clone_copy_on_read": "false", 2024-09-21T12:21:17.601 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_compression_hint": "none", 2024-09-21T12:21:17.601 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_concurrent_management_ops": "10", 2024-09-21T12:21:17.601 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-21T12:21:17.601 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_default_clone_format": "auto", 2024-09-21T12:21:17.601 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_default_data_pool": "", 2024-09-21T12:21:17.601 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_default_features": "61", 2024-09-21T12:21:17.601 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_default_format": "2", 2024-09-21T12:21:17.601 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_default_map_options": "", 2024-09-21T12:21:17.601 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_default_order": "22", 2024-09-21T12:21:17.601 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_default_pool": "rbd", 2024-09-21T12:21:17.601 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-21T12:21:17.601 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_default_stripe_count": "0", 2024-09-21T12:21:17.601 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_default_stripe_unit": "0", 2024-09-21T12:21:17.601 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-21T12:21:17.602 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-21T12:21:17.602 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-21T12:21:17.602 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_enable_alloc_hint": "true", 2024-09-21T12:21:17.602 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-21T12:21:17.602 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_io_scheduler": "simple", 2024-09-21T12:21:17.602 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-21T12:21:17.602 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-21T12:21:17.602 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-21T12:21:17.602 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-21T12:21:17.602 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-21T12:21:17.602 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-21T12:21:17.602 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-21T12:21:17.602 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-21T12:21:17.602 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-21T12:21:17.602 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_journal_order": "24", 2024-09-21T12:21:17.603 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_journal_pool": "", 2024-09-21T12:21:17.603 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_journal_splay_width": "4", 2024-09-21T12:21:17.603 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_localize_parent_reads": "false", 2024-09-21T12:21:17.603 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_localize_snap_reads": "false", 2024-09-21T12:21:17.603 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-21T12:21:17.603 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-21T12:21:17.603 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-21T12:21:17.603 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-21T12:21:17.603 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-21T12:21:17.603 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-21T12:21:17.603 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-21T12:21:17.603 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-21T12:21:17.603 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-21T12:21:17.604 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-21T12:21:17.604 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-21T12:21:17.604 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-21T12:21:17.604 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-21T12:21:17.604 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-21T12:21:17.604 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-21T12:21:17.604 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-21T12:21:17.604 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-21T12:21:17.604 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-21T12:21:17.604 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-21T12:21:17.604 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-21T12:21:17.604 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-21T12:21:17.604 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-21T12:21:17.604 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-21T12:21:17.604 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-21T12:21:17.605 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-21T12:21:17.605 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-21T12:21:17.605 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-21T12:21:17.605 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-21T12:21:17.605 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-21T12:21:17.605 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-21T12:21:17.605 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-21T12:21:17.605 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mtime_update_interval": "60", 2024-09-21T12:21:17.605 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_non_blocking_aio": "true", 2024-09-21T12:21:17.605 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_op_thread_timeout": "60", 2024-09-21T12:21:17.605 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_op_threads": "1", 2024-09-21T12:21:17.605 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_parent_cache_enabled": "false", 2024-09-21T12:21:17.605 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-21T12:21:17.605 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-21T12:21:17.606 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-21T12:21:17.606 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_plugins": "", 2024-09-21T12:21:17.606 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_bps_burst": "0", 2024-09-21T12:21:17.606 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-21T12:21:17.606 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_bps_limit": "0", 2024-09-21T12:21:17.606 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_exclude_ops": "0", 2024-09-21T12:21:17.606 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_iops_burst": "0", 2024-09-21T12:21:17.606 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-21T12:21:17.606 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_iops_limit": "0", 2024-09-21T12:21:17.606 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-21T12:21:17.606 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-21T12:21:17.606 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-21T12:21:17.606 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-21T12:21:17.606 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-21T12:21:17.607 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-21T12:21:17.607 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-21T12:21:17.607 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-21T12:21:17.607 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-21T12:21:17.607 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-21T12:21:17.607 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-21T12:21:17.607 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-21T12:21:17.607 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-21T12:21:17.607 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-21T12:21:17.607 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_read_from_replica_policy": "default", 2024-09-21T12:21:17.607 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-21T12:21:17.607 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-21T12:21:17.607 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-21T12:21:17.607 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-21T12:21:17.607 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_skip_partial_discard": "true", 2024-09-21T12:21:17.608 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-21T12:21:17.608 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_tracing": "false", 2024-09-21T12:21:17.608 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_validate_names": "true", 2024-09-21T12:21:17.608 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_validate_pool": "true", 2024-09-21T12:21:17.608 INFO:teuthology.orchestra.run.smithi114.stdout: "restapi_base_url": "", 2024-09-21T12:21:17.608 INFO:teuthology.orchestra.run.smithi114.stdout: "restapi_log_level": "", 2024-09-21T12:21:17.608 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-21T12:21:17.608 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-21T12:21:17.608 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_acl_grants_max_num": "100", 2024-09-21T12:21:17.608 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_admin_entry": "admin", 2024-09-21T12:21:17.608 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-21T12:21:17.608 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_asio_assert_yielding": "false", 2024-09-21T12:21:17.608 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_backend_store": "rados", 2024-09-21T12:21:17.608 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_barbican_url": "", 2024-09-21T12:21:17.608 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_beast_enable_async": "true", 2024-09-21T12:21:17.609 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_bucket_counters_cache": "false", 2024-09-21T12:21:17.609 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-21T12:21:17.609 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-21T12:21:17.609 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-21T12:21:17.609 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-21T12:21:17.609 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-21T12:21:17.609 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-21T12:21:17.609 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-21T12:21:17.609 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-21T12:21:17.609 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_cache_enabled": "true", 2024-09-21T12:21:17.609 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_cache_expiry_interval": "900", 2024-09-21T12:21:17.609 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_cache_lru_size": "25000", 2024-09-21T12:21:17.609 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_config_store": "rados", 2024-09-21T12:21:17.609 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_content_length_compat": "false", 2024-09-21T12:21:17.609 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_copy_obj_progress": "true", 2024-09-21T12:21:17.609 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-21T12:21:17.610 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_cors_rules_max_num": "100", 2024-09-21T12:21:17.610 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_cross_domain_policy": "", 2024-09-21T12:21:17.610 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-21T12:21:17.610 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_kmip_addr": "", 2024-09-21T12:21:17.610 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-21T12:21:17.610 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-21T12:21:17.610 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-21T12:21:17.610 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-21T12:21:17.610 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_kmip_password": "", 2024-09-21T12:21:17.610 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-21T12:21:17.610 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_kmip_username": "", 2024-09-21T12:21:17.610 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_require_ssl": "true", 2024-09-21T12:21:17.610 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-21T12:21:17.610 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-21T12:21:17.611 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-21T12:21:17.611 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-21T12:21:17.611 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-21T12:21:17.611 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-21T12:21:17.611 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-21T12:21:17.611 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-21T12:21:17.611 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-21T12:21:17.611 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-21T12:21:17.611 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-21T12:21:17.611 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-21T12:21:17.611 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-21T12:21:17.611 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-21T12:21:17.611 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-21T12:21:17.611 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_vault_addr": "", 2024-09-21T12:21:17.612 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_vault_auth": "token", 2024-09-21T12:21:17.612 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_vault_namespace": "", 2024-09-21T12:21:17.612 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_vault_prefix": "", 2024-09-21T12:21:17.612 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-21T12:21:17.612 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-21T12:21:17.612 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-21T12:21:17.612 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-21T12:21:17.612 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_vault_token_file": "", 2024-09-21T12:21:17.612 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-21T12:21:17.612 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_curl_buffersize": "524288", 2024-09-21T12:21:17.612 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-21T12:21:17.612 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_curl_low_speed_time": "300", 2024-09-21T12:21:17.612 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-21T12:21:17.612 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-21T12:21:17.612 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-21T12:21:17.612 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-21T12:21:17.613 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-21T12:21:17.613 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-21T12:21:17.613 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-21T12:21:17.613 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-21T12:21:17.613 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-21T12:21:17.613 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-21T12:21:17.613 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-09-21T12:21:17.613 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-09-21T12:21:17.613 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-09-21T12:21:17.613 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-09-21T12:21:17.613 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-09-21T12:21:17.613 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d4n_l1_fadvise": "4", 2024-09-21T12:21:17.613 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-09-21T12:21:17.613 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-09-21T12:21:17.613 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-21T12:21:17.613 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_data_log_changes_size": "1000", 2024-09-21T12:21:17.614 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_data_log_num_shards": "128", 2024-09-21T12:21:17.614 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-21T12:21:17.614 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_data_log_window": "30", 2024-09-21T12:21:17.614 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-21T12:21:17.614 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-21T12:21:17.614 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-21T12:21:17.614 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-21T12:21:17.614 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-21T12:21:17.614 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-21T12:21:17.614 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-21T12:21:17.614 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-21T12:21:17.614 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-21T12:21:17.614 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-21T12:21:17.614 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-21T12:21:17.614 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-21T12:21:17.615 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-21T12:21:17.615 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_disable_s3select": "false", 2024-09-21T12:21:17.615 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-21T12:21:17.615 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-21T12:21:17.615 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-21T12:21:17.615 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-21T12:21:17.615 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-21T12:21:17.615 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-21T12:21:17.615 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-21T12:21:17.615 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-21T12:21:17.615 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-21T12:21:17.615 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-21T12:21:17.615 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-21T12:21:17.615 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-21T12:21:17.615 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dns_name": "", 2024-09-21T12:21:17.616 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dns_s3website_name": "", 2024-09-21T12:21:17.616 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dynamic_resharding": "true", 2024-09-21T12:21:17.616 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-09-21T12:21:17.616 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-09-21T12:21:17.616 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-21T12:21:17.616 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_enable_gc_threads": "true", 2024-09-21T12:21:17.616 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_enable_lc_threads": "true", 2024-09-21T12:21:17.616 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_enable_mdsearch": "true", 2024-09-21T12:21:17.616 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_enable_ops_log": "false", 2024-09-21T12:21:17.616 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_enable_quota_threads": "true", 2024-09-21T12:21:17.616 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_enable_static_website": "false", 2024-09-21T12:21:17.616 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_enable_usage_log": "false", 2024-09-21T12:21:17.616 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_enforce_swift_acls": "true", 2024-09-21T12:21:17.616 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_exit_timeout_secs": "120", 2024-09-21T12:21:17.617 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_expose_bucket": "false", 2024-09-21T12:21:17.617 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_extended_http_attrs": "", 2024-09-21T12:21:17.617 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_filter": "none", 2024-09-21T12:21:17.617 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-21T12:21:17.617 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_frontends": "beast port=7480", 2024-09-21T12:21:17.617 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-21T12:21:17.617 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_gc_max_deferred": "50", 2024-09-21T12:21:17.617 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-21T12:21:17.617 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_gc_max_objs": "32", 2024-09-21T12:21:17.617 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-21T12:21:17.617 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-21T12:21:17.617 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-21T12:21:17.617 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-21T12:21:17.617 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_gc_processor_period": "3600", 2024-09-21T12:21:17.617 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-21T12:21:17.618 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-21T12:21:17.618 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-21T12:21:17.618 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-21T12:21:17.618 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_init_timeout": "300", 2024-09-21T12:21:17.618 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-21T12:21:17.618 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-21T12:21:17.618 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_kafka_connection_idle": "300", 2024-09-21T12:21:17.618 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-21T12:21:17.618 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-21T12:21:17.618 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-21T12:21:17.618 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-21T12:21:17.618 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-21T12:21:17.618 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_admin_domain": "", 2024-09-21T12:21:17.618 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_admin_password": "", 2024-09-21T12:21:17.619 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_admin_password_path": "", 2024-09-21T12:21:17.619 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_admin_project": "", 2024-09-21T12:21:17.619 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_admin_tenant": "", 2024-09-21T12:21:17.619 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_admin_token": "", 2024-09-21T12:21:17.619 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_admin_token_path": "", 2024-09-21T12:21:17.619 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_admin_user": "", 2024-09-21T12:21:17.619 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_barbican_domain": "", 2024-09-21T12:21:17.619 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_barbican_password": "", 2024-09-21T12:21:17.619 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_barbican_project": "", 2024-09-21T12:21:17.619 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-21T12:21:17.619 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_barbican_user": "", 2024-09-21T12:21:17.619 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-21T12:21:17.619 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-21T12:21:17.619 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-21T12:21:17.619 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-21T12:21:17.620 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-21T12:21:17.620 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_url": "", 2024-09-21T12:21:17.620 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-21T12:21:17.620 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_lc_debug_interval": "-1", 2024-09-21T12:21:17.620 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_lc_lock_max_time": "90", 2024-09-21T12:21:17.620 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_lc_max_objs": "32", 2024-09-21T12:21:17.620 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_lc_max_rules": "1000", 2024-09-21T12:21:17.620 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_lc_max_worker": "3", 2024-09-21T12:21:17.620 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-21T12:21:17.620 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_lc_thread_delay": "0", 2024-09-21T12:21:17.620 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-21T12:21:17.620 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_ldap_dnattr": "uid", 2024-09-21T12:21:17.620 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-21T12:21:17.620 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_ldap_searchfilter": "", 2024-09-21T12:21:17.621 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-21T12:21:17.621 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_ldap_uri": "", 2024-09-21T12:21:17.621 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_lfuda_sync_frequency": "60", 2024-09-21T12:21:17.621 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-21T12:21:17.621 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-21T12:21:17.621 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-21T12:21:17.621 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_log_http_headers": "", 2024-09-21T12:21:17.621 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-21T12:21:17.621 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-21T12:21:17.621 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_log_object_name_utc": "false", 2024-09-21T12:21:17.621 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-21T12:21:17.621 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-21T12:21:17.621 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_max_attr_name_len": "0", 2024-09-21T12:21:17.621 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_max_attr_size": "0", 2024-09-21T12:21:17.621 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-21T12:21:17.622 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_max_chunk_size": "4194304", 2024-09-21T12:21:17.622 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-21T12:21:17.622 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-21T12:21:17.622 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-21T12:21:17.622 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_max_listing_results": "5000", 2024-09-21T12:21:17.622 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_max_notify_retries": "10", 2024-09-21T12:21:17.622 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-21T12:21:17.622 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_max_put_param_size": "1048576", 2024-09-21T12:21:17.622 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_max_put_size": "5368709120", 2024-09-21T12:21:17.622 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_max_slo_entries": "1000", 2024-09-21T12:21:17.622 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_md_log_max_shards": "64", 2024-09-21T12:21:17.622 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-21T12:21:17.622 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-21T12:21:17.622 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-21T12:21:17.623 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-21T12:21:17.623 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_mp_lock_max_time": "600", 2024-09-21T12:21:17.623 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-21T12:21:17.623 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-21T12:21:17.623 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-21T12:21:17.623 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-21T12:21:17.623 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-21T12:21:17.623 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-21T12:21:17.623 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-21T12:21:17.623 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-21T12:21:17.623 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_nfs_max_gc": "300", 2024-09-21T12:21:17.623 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-21T12:21:17.623 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-21T12:21:17.623 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-21T12:21:17.624 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-21T12:21:17.624 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-21T12:21:17.624 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-21T12:21:17.624 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-21T12:21:17.624 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_num_async_rados_threads": "32", 2024-09-21T12:21:17.624 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_num_control_oids": "8", 2024-09-21T12:21:17.624 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_numa_node": "-1", 2024-09-21T12:21:17.624 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-21T12:21:17.624 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-21T12:21:17.624 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_objexp_chunk_size": "100", 2024-09-21T12:21:17.624 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_objexp_gc_interval": "600", 2024-09-21T12:21:17.624 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-21T12:21:17.624 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-21T12:21:17.624 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-21T12:21:17.624 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_op_thread_timeout": "600", 2024-09-21T12:21:17.625 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_op_tracing": "false", 2024-09-21T12:21:17.625 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_opa_token": "", 2024-09-21T12:21:17.625 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_opa_url": "", 2024-09-21T12:21:17.625 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_opa_verify_ssl": "true", 2024-09-21T12:21:17.625 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-21T12:21:17.625 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-21T12:21:17.625 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_ops_log_rados": "false", 2024-09-21T12:21:17.625 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_ops_log_socket_path": "", 2024-09-21T12:21:17.625 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-21T12:21:17.625 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-21T12:21:17.625 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-21T12:21:17.625 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_period_push_interval": "2.000000", 2024-09-21T12:21:17.625 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-21T12:21:17.625 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-21T12:21:17.625 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-21T12:21:17.626 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-21T12:21:17.626 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_posix_cache_lanes": "3", 2024-09-21T12:21:17.626 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-21T12:21:17.626 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-21T12:21:17.626 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_posix_cache_partitions": "3", 2024-09-21T12:21:17.626 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-21T12:21:17.626 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_print_continue": "true", 2024-09-21T12:21:17.626 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-21T12:21:17.626 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-21T12:21:17.626 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-21T12:21:17.626 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-21T12:21:17.626 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-21T12:21:17.626 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_rados_tracing": "false", 2024-09-21T12:21:17.626 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_realm": "", 2024-09-21T12:21:17.627 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_realm_id": "", 2024-09-21T12:21:17.627 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-21T12:21:17.627 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_region": "", 2024-09-21T12:21:17.627 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-21T12:21:17.627 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-21T12:21:17.627 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-21T12:21:17.627 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-21T12:21:17.627 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_request_uri": "", 2024-09-21T12:21:17.627 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_reshard_batch_size": "64", 2024-09-21T12:21:17.627 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-21T12:21:17.627 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_reshard_debug_interval": "-1", 2024-09-21T12:21:17.627 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_reshard_max_aio": "128", 2024-09-21T12:21:17.627 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_reshard_num_logs": "16", 2024-09-21T12:21:17.627 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_reshard_progress_judge_interval": "120", 2024-09-21T12:21:17.627 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_reshard_progress_judge_ratio": "0.500000", 2024-09-21T12:21:17.628 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_reshard_thread_interval": "600", 2024-09-21T12:21:17.628 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_reshardlog_threshold": "30000", 2024-09-21T12:21:17.628 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_resolve_cname": "false", 2024-09-21T12:21:17.628 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-21T12:21:17.628 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_run_sync_thread": "true", 2024-09-21T12:21:17.628 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-21T12:21:17.628 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-21T12:21:17.628 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-21T12:21:17.628 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-21T12:21:17.628 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-21T12:21:17.628 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-21T12:21:17.628 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-21T12:21:17.628 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-21T12:21:17.628 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-21T12:21:17.628 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_scheduler_type": "throttler", 2024-09-21T12:21:17.629 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_script_uri": "", 2024-09-21T12:21:17.629 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_service_provider_name": "", 2024-09-21T12:21:17.629 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-21T12:21:17.629 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sts_client_id": "", 2024-09-21T12:21:17.629 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sts_client_secret": "", 2024-09-21T12:21:17.629 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sts_entry": "sts", 2024-09-21T12:21:17.629 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sts_key": "sts", 2024-09-21T12:21:17.629 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-21T12:21:17.629 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sts_min_session_duration": "900", 2024-09-21T12:21:17.629 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sts_token_introspection_url": "", 2024-09-21T12:21:17.629 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_swift_account_in_url": "false", 2024-09-21T12:21:17.629 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_swift_auth_entry": "auth", 2024-09-21T12:21:17.629 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_swift_auth_url": "", 2024-09-21T12:21:17.629 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_swift_custom_header": "", 2024-09-21T12:21:17.629 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-21T12:21:17.630 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_swift_need_stats": "true", 2024-09-21T12:21:17.630 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_swift_tenant_name": "", 2024-09-21T12:21:17.630 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_swift_token_expiration": "86400", 2024-09-21T12:21:17.630 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_swift_url": "", 2024-09-21T12:21:17.630 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_swift_url_prefix": "swift", 2024-09-21T12:21:17.630 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-21T12:21:17.630 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-21T12:21:17.630 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-21T12:21:17.630 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sync_lease_period": "120", 2024-09-21T12:21:17.630 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-21T12:21:17.630 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-21T12:21:17.630 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-21T12:21:17.630 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-21T12:21:17.630 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-21T12:21:17.631 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-21T12:21:17.631 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-21T12:21:17.631 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-21T12:21:17.631 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-21T12:21:17.631 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_thread_pool_size": "512", 2024-09-21T12:21:17.631 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-21T12:21:17.631 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-21T12:21:17.631 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-21T12:21:17.631 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-21T12:21:17.631 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_torrent_comment": "", 2024-09-21T12:21:17.631 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_torrent_createby": "", 2024-09-21T12:21:17.631 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_torrent_encoding": "", 2024-09-21T12:21:17.631 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_torrent_flag": "false", 2024-09-21T12:21:17.631 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-21T12:21:17.631 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_torrent_origin": "", 2024-09-21T12:21:17.632 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-21T12:21:17.632 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_torrent_tracker": "", 2024-09-21T12:21:17.632 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_trust_forwarded_https": "false", 2024-09-21T12:21:17.632 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-21T12:21:17.632 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-21T12:21:17.632 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_usage_max_shards": "32", 2024-09-21T12:21:17.632 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_usage_max_user_shards": "1", 2024-09-21T12:21:17.632 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_use_opa_authz": "false", 2024-09-21T12:21:17.632 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_user_counters_cache": "false", 2024-09-21T12:21:17.632 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-21T12:21:17.632 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-21T12:21:17.632 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-21T12:21:17.632 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_user_max_buckets": "1000", 2024-09-21T12:21:17.632 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_user_policies_max_num": "100", 2024-09-21T12:21:17.633 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-21T12:21:17.633 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-21T12:21:17.633 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-21T12:21:17.633 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-21T12:21:17.633 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_user_unique_email": "true", 2024-09-21T12:21:17.633 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_verify_ssl": "true", 2024-09-21T12:21:17.633 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-21T12:21:17.633 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_zone": "", 2024-09-21T12:21:17.633 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_zone_id": "", 2024-09-21T12:21:17.633 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-21T12:21:17.633 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_zonegroup": "", 2024-09-21T12:21:17.633 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_zonegroup_id": "", 2024-09-21T12:21:17.633 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-21T12:21:17.633 INFO:teuthology.orchestra.run.smithi114.stdout: "rgwlc_auto_session_clear": "true", 2024-09-21T12:21:17.633 INFO:teuthology.orchestra.run.smithi114.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-21T12:21:17.634 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_block_size": "4096", 2024-09-21T12:21:17.634 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-21T12:21:17.634 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-21T12:21:17.634 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-21T12:21:17.634 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-21T12:21:17.634 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-21T12:21:17.634 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_cache_size": "536870912", 2024-09-21T12:21:17.634 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-21T12:21:17.634 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-21T12:21:17.634 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-21T12:21:17.634 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-21T12:21:17.634 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-21T12:21:17.634 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-21T12:21:17.634 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-21T12:21:17.635 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-21T12:21:17.635 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_index_type": "binary_search", 2024-09-21T12:21:17.635 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-21T12:21:17.635 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-21T12:21:17.635 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_partition_filters": "false", 2024-09-21T12:21:17.635 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_perf": "false", 2024-09-21T12:21:17.635 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-21T12:21:17.635 INFO:teuthology.orchestra.run.smithi114.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-21T12:21:17.635 INFO:teuthology.orchestra.run.smithi114.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-21T12:21:17.635 INFO:teuthology.orchestra.run.smithi114.stdout: "run_dir": "/var/run/ceph", 2024-09-21T12:21:17.635 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_block_create": "true", 2024-09-21T12:21:17.635 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_cache_lru_size": "67108864", 2024-09-21T12:21:17.635 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-21T12:21:17.635 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-21T12:21:17.635 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_default_max_object_size": "16777216", 2024-09-21T12:21:17.636 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-21T12:21:17.636 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_device_size": "53687091200", 2024-09-21T12:21:17.636 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-21T12:21:17.636 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_full_integrity_check": "false", 2024-09-21T12:21:17.636 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_journal_batch_capacity": "16", 2024-09-21T12:21:17.636 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-21T12:21:17.636 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-21T12:21:17.636 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-21T12:21:17.636 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_main_device_type": "SSD", 2024-09-21T12:21:17.636 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-21T12:21:17.636 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-21T12:21:17.636 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-21T12:21:17.636 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-21T12:21:17.636 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-21T12:21:17.636 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-21T12:21:17.637 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_segment_size": "67108864", 2024-09-21T12:21:17.637 INFO:teuthology.orchestra.run.smithi114.stdout: "set_keepcaps": "false", 2024-09-21T12:21:17.637 INFO:teuthology.orchestra.run.smithi114.stdout: "setgroup": "ceph", 2024-09-21T12:21:17.637 INFO:teuthology.orchestra.run.smithi114.stdout: "setuser": "ceph", 2024-09-21T12:21:17.637 INFO:teuthology.orchestra.run.smithi114.stdout: "setuser_match_path": "", 2024-09-21T12:21:17.637 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-21T12:21:17.637 INFO:teuthology.orchestra.run.smithi114.stdout: "thp": "false", 2024-09-21T12:21:17.637 INFO:teuthology.orchestra.run.smithi114.stdout: "threadpool_default_timeout": "60", 2024-09-21T12:21:17.637 INFO:teuthology.orchestra.run.smithi114.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-21T12:21:17.637 INFO:teuthology.orchestra.run.smithi114.stdout: "throttler_perf_counter": "true", 2024-09-21T12:21:17.637 INFO:teuthology.orchestra.run.smithi114.stdout: "tmp_dir": "/tmp", 2024-09-21T12:21:17.637 INFO:teuthology.orchestra.run.smithi114.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-09-21T12:21:17.637 INFO:teuthology.orchestra.run.smithi114.stdout: "uadk_compressor_enabled": "false", 2024-09-21T12:21:17.637 INFO:teuthology.orchestra.run.smithi114.stdout: "uadk_wd_sync_ctx_num": "2" 2024-09-21T12:21:17.637 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:21:18.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:17 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:21:18.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:17 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:21:18.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:17 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:21:18.315 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-21T12:21:18.557 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:21:19.013 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:18 smithi114 ceph-mon[40867]: 3.2 scrub starts 2024-09-21T12:21:19.013 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:18 smithi114 ceph-mon[40867]: 3.2 scrub ok 2024-09-21T12:21:19.013 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:18 smithi114 ceph-mon[40867]: pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:21:19.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:18 smithi114 ceph-mon[30324]: 3.2 scrub starts 2024-09-21T12:21:19.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:18 smithi114 ceph-mon[30324]: 3.2 scrub ok 2024-09-21T12:21:19.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:18 smithi114 ceph-mon[30324]: pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:21:19.014 INFO:teuthology.orchestra.run.smithi114.stdout:mon.a: {} 2024-09-21T12:21:19.014 INFO:teuthology.orchestra.run.smithi114.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T12:21:19.022 INFO:teuthology.orchestra.run.smithi114.stdout:mon.b: {} 2024-09-21T12:21:19.022 INFO:teuthology.orchestra.run.smithi114.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T12:21:19.030 INFO:teuthology.orchestra.run.smithi114.stdout:mon.c: {} 2024-09-21T12:21:19.030 INFO:teuthology.orchestra.run.smithi114.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T12:21:19.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:18 smithi183 ceph-mon[33277]: 3.2 scrub starts 2024-09-21T12:21:19.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:18 smithi183 ceph-mon[33277]: 3.2 scrub ok 2024-09-21T12:21:19.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:18 smithi183 ceph-mon[33277]: pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:21:19.771 INFO:teuthology.run_tasks:Running task full_sequential... 2024-09-21T12:21:19.786 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T12:21:19.787 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T12:21:19.787 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-21T12:21:19.788 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_0 16 2024-09-21T12:21:19.788 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-21T12:21:19.788 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T12:21:19.788 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:21:19.788 INFO:tasks.thrashosds.thrasher:Setting full ratio to .001 2024-09-21T12:21:19.789 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set-full-ratio .001 2024-09-21T12:21:19.789 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-21T12:21:19.789 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:19.789 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-21T12:21:19.790 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:21:19.790 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-21T12:21:19.790 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T12:21:19.790 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-21T12:21:19.791 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T12:21:19.819 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:19.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:19.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:19.921 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:19.953 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:19.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:19.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:20.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:19 smithi114 ceph-mon[40867]: 3.7 scrub starts 2024-09-21T12:21:20.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:19 smithi114 ceph-mon[40867]: 3.7 scrub ok 2024-09-21T12:21:20.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:19 smithi114 ceph-mon[40867]: 4.b scrub starts 2024-09-21T12:21:20.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:19 smithi114 ceph-mon[40867]: 4.b scrub ok 2024-09-21T12:21:20.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:19 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/4280777965' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T12:21:20.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:19 smithi114 ceph-mon[30324]: 3.7 scrub starts 2024-09-21T12:21:20.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:19 smithi114 ceph-mon[30324]: 3.7 scrub ok 2024-09-21T12:21:20.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:19 smithi114 ceph-mon[30324]: 4.b scrub starts 2024-09-21T12:21:20.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:19 smithi114 ceph-mon[30324]: 4.b scrub ok 2024-09-21T12:21:20.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:19 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4280777965' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T12:21:20.055 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:21:20.080 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:20.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:20.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:21:20.116 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:21:20.117 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:21:20.118 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:21:20.164 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:21:20.181 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:20.202 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:20.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:20.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:20.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:19 smithi183 ceph-mon[33277]: 3.7 scrub starts 2024-09-21T12:21:20.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:19 smithi183 ceph-mon[33277]: 3.7 scrub ok 2024-09-21T12:21:20.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:19 smithi183 ceph-mon[33277]: 4.b scrub starts 2024-09-21T12:21:20.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:19 smithi183 ceph-mon[33277]: 4.b scrub ok 2024-09-21T12:21:20.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:19 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/4280777965' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T12:21:20.304 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:20.326 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:20.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:20.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:20.428 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:20.451 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:20.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:20.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:20.552 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:20.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:20.574 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:20.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:20.675 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:20.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:20.697 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:20.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:20.798 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:20.822 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:20.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:20.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:20.924 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:20.947 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:20.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:20.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:21.049 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:21.071 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:21.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:21.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:21.103 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:20 smithi114 ceph-mon[40867]: 4.e scrub starts 2024-09-21T12:21:21.104 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:20 smithi114 ceph-mon[40867]: 4.e scrub ok 2024-09-21T12:21:21.104 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:20 smithi114 ceph-mon[40867]: pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T12:21:21.104 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:20 smithi114 ceph-mon[40867]: 2.2 scrub starts 2024-09-21T12:21:21.104 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:20 smithi114 ceph-mon[40867]: 2.2 scrub ok 2024-09-21T12:21:21.104 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:20 smithi114 ceph-mon[30324]: 4.e scrub starts 2024-09-21T12:21:21.104 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:20 smithi114 ceph-mon[30324]: 4.e scrub ok 2024-09-21T12:21:21.104 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:20 smithi114 ceph-mon[30324]: pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T12:21:21.104 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:20 smithi114 ceph-mon[30324]: 2.2 scrub starts 2024-09-21T12:21:21.104 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:20 smithi114 ceph-mon[30324]: 2.2 scrub ok 2024-09-21T12:21:21.173 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:21.194 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:21.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:21.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:21.205 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:21:21.205 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:21:21.205 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 0 2024-09-21T12:21:21.205 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:21:21.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:20 smithi183 ceph-mon[33277]: 4.e scrub starts 2024-09-21T12:21:21.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:20 smithi183 ceph-mon[33277]: 4.e scrub ok 2024-09-21T12:21:21.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:20 smithi183 ceph-mon[33277]: pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T12:21:21.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:20 smithi183 ceph-mon[33277]: 2.2 scrub starts 2024-09-21T12:21:21.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:20 smithi183 ceph-mon[33277]: 2.2 scrub ok 2024-09-21T12:21:21.296 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:21.318 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:21.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:21.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:21.420 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:21.441 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:21.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:21.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:21.542 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:21:21.563 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:21.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:21.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:21:21.664 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:21.685 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:21.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:21.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:21.787 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:21:21.809 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:21.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:21.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:21:21.862 INFO:teuthology.orchestra.run.smithi114.stderr:osd set-full-ratio 0.001 2024-09-21T12:21:21.910 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:21.932 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:21.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:21.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:21.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:21 smithi114 ceph-mon[40867]: 3.1b scrub starts 2024-09-21T12:21:21.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:21 smithi114 ceph-mon[40867]: 3.1b scrub ok 2024-09-21T12:21:21.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:21 smithi114 ceph-mon[40867]: 3.e scrub starts 2024-09-21T12:21:21.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:21 smithi114 ceph-mon[40867]: 3.e scrub ok 2024-09-21T12:21:21.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:21 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/175026445' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.001} : dispatch 2024-09-21T12:21:21.986 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:21 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-a[30301]: 2024-09-21T12:21:21.858+0000 7f4a00e8a640 -1 log_channel(cluster) log [ERR] : Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL) 2024-09-21T12:21:21.986 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:21 smithi114 ceph-mon[30324]: 3.1b scrub starts 2024-09-21T12:21:21.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:21 smithi114 ceph-mon[30324]: 3.1b scrub ok 2024-09-21T12:21:21.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:21 smithi114 ceph-mon[30324]: 3.e scrub starts 2024-09-21T12:21:21.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:21 smithi114 ceph-mon[30324]: 3.e scrub ok 2024-09-21T12:21:21.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:21 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/175026445' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.001} : dispatch 2024-09-21T12:21:21.992 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:21:21.992 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:21:22.003 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:21:22.003 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:21:22.019 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:21:22.020 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:21:22.035 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:22.035 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:21:22.036 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:21:22.052 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:21:22.052 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:21:22.056 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:22.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:22.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:22.069 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:21:22.069 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:21:22.086 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:21:22.086 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:21:22.102 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:21:22.103 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:21:22.157 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:22.179 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:22.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:22.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:22.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:21 smithi183 ceph-mon[33277]: 3.1b scrub starts 2024-09-21T12:21:22.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:21 smithi183 ceph-mon[33277]: 3.1b scrub ok 2024-09-21T12:21:22.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:21 smithi183 ceph-mon[33277]: 3.e scrub starts 2024-09-21T12:21:22.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:21 smithi183 ceph-mon[33277]: 3.e scrub ok 2024-09-21T12:21:22.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:21 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/175026445' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.001} : dispatch 2024-09-21T12:21:22.280 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:22.302 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:22.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:22.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:22.404 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:22.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:22.426 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:22.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:22.527 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:22.550 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:22.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:22.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:22.652 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:21:22.677 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:22.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:22.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:21:22.779 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:22.801 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:22.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:22.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:22.868 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:21:22 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:21:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:21:22.903 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:21:22.925 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:22.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:22.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:21:22.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:22 smithi183 ceph-mon[33277]: 5.13 scrub starts 2024-09-21T12:21:22.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:22 smithi183 ceph-mon[33277]: 5.13 scrub ok 2024-09-21T12:21:22.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:22 smithi183 ceph-mon[33277]: pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:21:22.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:22 smithi183 ceph-mon[33277]: 6.3 scrub starts 2024-09-21T12:21:22.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:22 smithi183 ceph-mon[33277]: 6.3 scrub ok 2024-09-21T12:21:22.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:22 smithi183 ceph-mon[33277]: Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL) 2024-09-21T12:21:22.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:22 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/175026445' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.001}]': finished 2024-09-21T12:21:22.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:22 smithi183 ceph-mon[33277]: osdmap e83: 8 total, 8 up, 8 in 2024-09-21T12:21:22.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:22 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3725483290' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:21:23.027 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:23.048 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:23.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:23.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:23.126 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:22 smithi114 ceph-mon[30324]: 5.13 scrub starts 2024-09-21T12:21:23.126 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:22 smithi114 ceph-mon[30324]: 5.13 scrub ok 2024-09-21T12:21:23.126 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:22 smithi114 ceph-mon[30324]: pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:21:23.127 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:22 smithi114 ceph-mon[30324]: 6.3 scrub starts 2024-09-21T12:21:23.127 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:22 smithi114 ceph-mon[30324]: 6.3 scrub ok 2024-09-21T12:21:23.127 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:22 smithi114 ceph-mon[30324]: Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL) 2024-09-21T12:21:23.127 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:22 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/175026445' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.001}]': finished 2024-09-21T12:21:23.127 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:22 smithi114 ceph-mon[30324]: osdmap e83: 8 total, 8 up, 8 in 2024-09-21T12:21:23.127 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:22 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3725483290' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:21:23.127 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:22 smithi114 ceph-mon[40867]: 5.13 scrub starts 2024-09-21T12:21:23.127 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:22 smithi114 ceph-mon[40867]: 5.13 scrub ok 2024-09-21T12:21:23.127 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:22 smithi114 ceph-mon[40867]: pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:21:23.127 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:22 smithi114 ceph-mon[40867]: 6.3 scrub starts 2024-09-21T12:21:23.127 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:22 smithi114 ceph-mon[40867]: 6.3 scrub ok 2024-09-21T12:21:23.127 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:22 smithi114 ceph-mon[40867]: Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL) 2024-09-21T12:21:23.127 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:22 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/175026445' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.001}]': finished 2024-09-21T12:21:23.128 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:22 smithi114 ceph-mon[40867]: osdmap e83: 8 total, 8 up, 8 in 2024-09-21T12:21:23.128 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:22 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3725483290' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:21:23.132 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:21:23.149 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:21:23.171 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:23.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:23.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:21:23.272 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:23.293 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:23.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:23.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:23.395 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:23.417 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:23.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:23.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:23.519 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:23.540 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:23.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:23.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:23.642 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:23.663 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:23.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:23.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:23.765 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:23.786 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:23.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:23.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:23.872 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'unique_pool_0' created 2024-09-21T12:21:23.888 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:23.909 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:23.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:23.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:23.927 INFO:tasks.thrashosds.thrasher:Setting full ratio back to .95 2024-09-21T12:21:23.927 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set-full-ratio .95 2024-09-21T12:21:23.972 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:21:23 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a[75423]: ts=2024-09-21T12:21:23.666Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.001410234s 2024-09-21T12:21:23.972 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:23 smithi114 ceph-mon[40867]: 2.0 scrub starts 2024-09-21T12:21:23.972 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:23 smithi114 ceph-mon[40867]: 2.0 scrub ok 2024-09-21T12:21:23.972 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:23 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/660003339' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-21T12:21:23.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:23 smithi114 ceph-mon[30324]: 2.0 scrub starts 2024-09-21T12:21:23.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:23 smithi114 ceph-mon[30324]: 2.0 scrub ok 2024-09-21T12:21:23.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:23 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/660003339' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-21T12:21:24.011 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:24.032 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:24.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:24.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:24.134 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:24.156 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:24.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:24.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:24.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:23 smithi183 ceph-mon[33277]: 2.0 scrub starts 2024-09-21T12:21:24.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:23 smithi183 ceph-mon[33277]: 2.0 scrub ok 2024-09-21T12:21:24.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:23 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/660003339' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-21T12:21:24.258 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:24.280 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:24.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:24.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:24.382 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:21:24.404 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:24.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:24.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:21:24.506 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:24.527 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:24.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:24.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:24.597 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T12:21:24.628 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:24.661 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:24.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:24.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:24.762 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:24.784 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:24.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:24.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:24.827 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:21:24.856 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:21:24.882 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-21T12:21:24.886 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:24.887 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:24 smithi114 ceph-mon[40867]: 5.3 scrub starts 2024-09-21T12:21:24.887 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:24 smithi114 ceph-mon[40867]: 5.3 scrub ok 2024-09-21T12:21:24.888 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:24 smithi114 ceph-mon[40867]: 5.8 scrub starts 2024-09-21T12:21:24.888 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:24 smithi114 ceph-mon[40867]: 5.8 scrub ok 2024-09-21T12:21:24.888 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:24 smithi114 ceph-mon[40867]: pgmap v30: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:21:24.888 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:24 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/660003339' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-21T12:21:24.888 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:24 smithi114 ceph-mon[40867]: osdmap e84: 8 total, 8 up, 8 in 2024-09-21T12:21:24.888 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:24 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/853846098' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:21:24.888 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:24 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:21:24.912 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:24.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:24.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:25.014 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:25.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:25.035 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:25.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:25.136 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:25.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:25.158 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:25.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:25.225 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:24 smithi183 ceph-mon[33277]: 5.3 scrub starts 2024-09-21T12:21:25.225 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:24 smithi183 ceph-mon[33277]: 5.3 scrub ok 2024-09-21T12:21:25.225 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:24 smithi183 ceph-mon[33277]: 5.8 scrub starts 2024-09-21T12:21:25.225 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:24 smithi183 ceph-mon[33277]: 5.8 scrub ok 2024-09-21T12:21:25.225 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:24 smithi183 ceph-mon[33277]: pgmap v30: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:21:25.225 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:24 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/660003339' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-21T12:21:25.225 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:24 smithi183 ceph-mon[33277]: osdmap e84: 8 total, 8 up, 8 in 2024-09-21T12:21:25.225 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:24 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/853846098' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:21:25.225 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:24 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:21:25.225 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 systemd[1]: Starting Ceph grafana.a for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:21:25.259 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:25.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:24 smithi114 ceph-mon[30324]: 5.3 scrub starts 2024-09-21T12:21:25.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:24 smithi114 ceph-mon[30324]: 5.3 scrub ok 2024-09-21T12:21:25.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:24 smithi114 ceph-mon[30324]: 5.8 scrub starts 2024-09-21T12:21:25.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:24 smithi114 ceph-mon[30324]: 5.8 scrub ok 2024-09-21T12:21:25.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:24 smithi114 ceph-mon[30324]: pgmap v30: 132 pgs: 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:21:25.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:24 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/660003339' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-21T12:21:25.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:24 smithi114 ceph-mon[30324]: osdmap e84: 8 total, 8 up, 8 in 2024-09-21T12:21:25.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:24 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/853846098' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:21:25.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:24 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:21:25.280 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:25.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:25.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:25.302 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-21T12:21:25.382 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:25.406 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:25.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:25.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:25.488 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 podman[63252]: 2024-09-21 12:21:25.126886164 +0000 UTC m=+0.016689567 image pull 5aad1d7cf004c80a4acb170e1c80fb85781b022f00f08c6bcd22608882b4c748 quay.io/ceph/grafana:10.4.8 2024-09-21T12:21:25.488 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 podman[63252]: 2024-09-21 12:21:25.233360304 +0000 UTC m=+0.123163751 container create 439a4d10dbdea341607cd280b3b6402e0f43128eef27b63c09047fb0224a8d09 (image=quay.io/ceph/grafana:10.4.8, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a, maintainer=Grafana Labs ) 2024-09-21T12:21:25.489 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 podman[63252]: 2024-09-21 12:21:25.45454027 +0000 UTC m=+0.344343675 container init 439a4d10dbdea341607cd280b3b6402e0f43128eef27b63c09047fb0224a8d09 (image=quay.io/ceph/grafana:10.4.8, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a, maintainer=Grafana Labs ) 2024-09-21T12:21:25.489 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 podman[63252]: 2024-09-21 12:21:25.459502798 +0000 UTC m=+0.349306201 container start 439a4d10dbdea341607cd280b3b6402e0f43128eef27b63c09047fb0224a8d09 (image=quay.io/ceph/grafana:10.4.8, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a, maintainer=Grafana Labs ) 2024-09-21T12:21:25.508 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:25.532 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:25.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:25.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:25.633 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:25.655 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:25.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:25.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:25.756 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:21:25.778 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:25.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:25.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:21:25.825 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=settings t=2024-09-21T12:21:25.573606143Z level=info msg="Starting Grafana" version=10.4.8 commit=8c384b71bde142a25ddc01ed5a183ca572b2b7f7 branch=v10.4.x compiled=2024-09-21T12:21:25Z 2024-09-21T12:21:25.825 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=settings t=2024-09-21T12:21:25.573833471Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-21T12:21:25.825 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=settings t=2024-09-21T12:21:25.573842069Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-21T12:21:25.825 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=settings t=2024-09-21T12:21:25.573846729Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-09-21T12:21:25.825 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=settings t=2024-09-21T12:21:25.573850611Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-09-21T12:21:25.825 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=settings t=2024-09-21T12:21:25.573853145Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-09-21T12:21:25.825 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=settings t=2024-09-21T12:21:25.573855638Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-09-21T12:21:25.825 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=settings t=2024-09-21T12:21:25.573858107Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-09-21T12:21:25.825 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=settings t=2024-09-21T12:21:25.573860816Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-21T12:21:25.825 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=settings t=2024-09-21T12:21:25.573863391Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-21T12:21:25.826 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=settings t=2024-09-21T12:21:25.573865887Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-21T12:21:25.826 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=settings t=2024-09-21T12:21:25.573868378Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-21T12:21:25.826 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=settings t=2024-09-21T12:21:25.573870924Z level=info msg=Target target=[all] 2024-09-21T12:21:25.826 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=settings t=2024-09-21T12:21:25.573875863Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-21T12:21:25.826 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=settings t=2024-09-21T12:21:25.573878278Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-21T12:21:25.826 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=settings t=2024-09-21T12:21:25.573880636Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-21T12:21:25.826 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=settings t=2024-09-21T12:21:25.57388287Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-21T12:21:25.826 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=settings t=2024-09-21T12:21:25.57388531Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-21T12:21:25.827 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=settings t=2024-09-21T12:21:25.573888785Z level=info msg="App mode production" 2024-09-21T12:21:25.827 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=sqlstore t=2024-09-21T12:21:25.574091839Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-21T12:21:25.827 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=sqlstore t=2024-09-21T12:21:25.574103438Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-21T12:21:25.827 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.574682377Z level=info msg="Starting DB migrations" 2024-09-21T12:21:25.827 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.57551985Z level=info msg="Executing migration" id="create migration_log table" 2024-09-21T12:21:25.827 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.576294944Z level=info msg="Migration successfully executed" id="create migration_log table" duration=774.82µs 2024-09-21T12:21:25.827 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.57677159Z level=info msg="Executing migration" id="create user table" 2024-09-21T12:21:25.827 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.57730083Z level=info msg="Migration successfully executed" id="create user table" duration=529.228µs 2024-09-21T12:21:25.827 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.577708039Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-21T12:21:25.828 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.578234213Z level=info msg="Migration successfully executed" id="add unique index user.login" duration=526.248µs 2024-09-21T12:21:25.828 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.578637346Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-21T12:21:25.828 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.579281299Z level=info msg="Migration successfully executed" id="add unique index user.email" duration=643.809µs 2024-09-21T12:21:25.828 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.579674423Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-21T12:21:25.828 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.580181726Z level=info msg="Migration successfully executed" id="drop index UQE_user_login - v1" duration=507.102µs 2024-09-21T12:21:25.828 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.580548113Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-21T12:21:25.828 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.581007113Z level=info msg="Migration successfully executed" id="drop index UQE_user_email - v1" duration=458.694µs 2024-09-21T12:21:25.828 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.581372682Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-21T12:21:25.828 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.582995758Z level=info msg="Migration successfully executed" id="Rename table user to user_v1 - v1" duration=1.621136ms 2024-09-21T12:21:25.828 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.583405477Z level=info msg="Executing migration" id="create user table v2" 2024-09-21T12:21:25.828 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.583944648Z level=info msg="Migration successfully executed" id="create user table v2" duration=538.955µs 2024-09-21T12:21:25.828 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.584339588Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-21T12:21:25.828 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.584792754Z level=info msg="Migration successfully executed" id="create index UQE_user_login - v2" duration=452.968µs 2024-09-21T12:21:25.828 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.585188589Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-21T12:21:25.828 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.585638619Z level=info msg="Migration successfully executed" id="create index UQE_user_email - v2" duration=448.16µs 2024-09-21T12:21:25.829 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.586052268Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-21T12:21:25.829 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.586330386Z level=info msg="Migration successfully executed" id="copy data_source v1 to v2" duration=278.347µs 2024-09-21T12:21:25.829 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.586788529Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-21T12:21:25.829 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.587177778Z level=info msg="Migration successfully executed" id="Drop old table user_v1" duration=387.336µs 2024-09-21T12:21:25.829 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.587767538Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-21T12:21:25.829 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 bash[63252]: 439a4d10dbdea341607cd280b3b6402e0f43128eef27b63c09047fb0224a8d09 2024-09-21T12:21:25.829 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.58876297Z level=info msg="Migration successfully executed" id="Add column help_flags1 to user table" duration=996.406µs 2024-09-21T12:21:25.829 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.589203296Z level=info msg="Executing migration" id="Update user table charset" 2024-09-21T12:21:25.829 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.589224843Z level=info msg="Migration successfully executed" id="Update user table charset" duration=22.213µs 2024-09-21T12:21:25.829 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.589985428Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-21T12:21:25.829 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.59068719Z level=info msg="Migration successfully executed" id="Add last_seen_at column to user" duration=699.745µs 2024-09-21T12:21:25.829 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 systemd[1]: Started Ceph grafana.a for e2ad975c-7812-11ef-baf6-efdc52797490. 2024-09-21T12:21:25.829 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.591370067Z level=info msg="Executing migration" id="Add missing user data" 2024-09-21T12:21:25.829 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.591649914Z level=info msg="Migration successfully executed" id="Add missing user data" duration=279.418µs 2024-09-21T12:21:25.830 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.593625712Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-21T12:21:25.830 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.595206862Z level=info msg="Migration successfully executed" id="Add is_disabled column to user" duration=1.579757ms 2024-09-21T12:21:25.830 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.596149683Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-21T12:21:25.830 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.596954837Z level=info msg="Migration successfully executed" id="Add index user.login/user.email" duration=805.851µs 2024-09-21T12:21:25.830 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.597356177Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-21T12:21:25.830 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.598133154Z level=info msg="Migration successfully executed" id="Add is_service_account column to user" duration=776.506µs 2024-09-21T12:21:25.830 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.598487016Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-21T12:21:25.830 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.60365775Z level=info msg="Migration successfully executed" id="Update is_service_account column to nullable" duration=5.170017ms 2024-09-21T12:21:25.830 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.604060467Z level=info msg="Executing migration" id="Add uid column to user" 2024-09-21T12:21:25.830 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.604791309Z level=info msg="Migration successfully executed" id="Add uid column to user" duration=730.785µs 2024-09-21T12:21:25.830 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.60515499Z level=info msg="Executing migration" id="Update uid column values for users" 2024-09-21T12:21:25.830 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.60529618Z level=info msg="Migration successfully executed" id="Update uid column values for users" duration=141.242µs 2024-09-21T12:21:25.830 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.605956632Z level=info msg="Executing migration" id="Add unique index user_uid" 2024-09-21T12:21:25.830 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.606458228Z level=info msg="Migration successfully executed" id="Add unique index user_uid" duration=501.633µs 2024-09-21T12:21:25.830 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.606870155Z level=info msg="Executing migration" id="update login field with orgid to allow for multiple service accounts with same name across orgs" 2024-09-21T12:21:25.831 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.607070595Z level=info msg="Migration successfully executed" id="update login field with orgid to allow for multiple service accounts with same name across orgs" duration=200.394µs 2024-09-21T12:21:25.831 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.607483168Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-21T12:21:25.831 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.607999417Z level=info msg="Migration successfully executed" id="create temp user table v1-7" duration=516.06µs 2024-09-21T12:21:25.831 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.60839742Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-21T12:21:25.831 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.608855738Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_email - v1-7" duration=458.187µs 2024-09-21T12:21:25.831 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.609212085Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-21T12:21:25.831 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.609657997Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_org_id - v1-7" duration=444.458µs 2024-09-21T12:21:25.831 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.610051883Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-21T12:21:25.831 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.610499681Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_code - v1-7" duration=447.508µs 2024-09-21T12:21:25.831 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.610893659Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-21T12:21:25.831 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.611387148Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_status - v1-7" duration=493.329µs 2024-09-21T12:21:25.831 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.611739338Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-21T12:21:25.831 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.611758218Z level=info msg="Migration successfully executed" id="Update temp_user table charset" duration=18.972µs 2024-09-21T12:21:25.831 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.61254499Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-21T12:21:25.832 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.613055617Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_email - v1" duration=510.311µs 2024-09-21T12:21:25.832 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.615332163Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-21T12:21:25.832 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.615931194Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_org_id - v1" duration=599.801µs 2024-09-21T12:21:25.832 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.617946501Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-21T12:21:25.832 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.618483496Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_code - v1" duration=537.452µs 2024-09-21T12:21:25.832 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.618850404Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-21T12:21:25.832 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.619304624Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_status - v1" duration=454.453µs 2024-09-21T12:21:25.832 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.621621962Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-21T12:21:25.832 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.623728725Z level=info msg="Migration successfully executed" id="Rename table temp_user to temp_user_tmp_qwerty - v1" duration=2.106124ms 2024-09-21T12:21:25.832 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.624157993Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-21T12:21:25.832 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.624704232Z level=info msg="Migration successfully executed" id="create temp_user v2" duration=546.945µs 2024-09-21T12:21:25.832 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.626842006Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-21T12:21:25.832 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.627420492Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_email - v2" duration=578.467µs 2024-09-21T12:21:25.832 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.627770383Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-21T12:21:25.833 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.628257877Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_org_id - v2" duration=487.669µs 2024-09-21T12:21:25.833 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.630549852Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-21T12:21:25.833 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.631053868Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_code - v2" duration=506.235µs 2024-09-21T12:21:25.833 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.633142149Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-21T12:21:25.833 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.633653817Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_status - v2" duration=511.574µs 2024-09-21T12:21:25.833 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.635853051Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-21T12:21:25.833 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.636160124Z level=info msg="Migration successfully executed" id="copy temp_user v1 to v2" duration=307.525µs 2024-09-21T12:21:25.833 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.636496675Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-21T12:21:25.833 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.636896772Z level=info msg="Migration successfully executed" id="drop temp_user_tmp_qwerty" duration=400.413µs 2024-09-21T12:21:25.833 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.6395083Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-21T12:21:25.833 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.639859156Z level=info msg="Migration successfully executed" id="Set created for temp users that will otherwise prematurely expire" duration=354.718µs 2024-09-21T12:21:25.835 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.641918883Z level=info msg="Executing migration" id="create star table" 2024-09-21T12:21:25.835 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.642394242Z level=info msg="Migration successfully executed" id="create star table" duration=475.258µs 2024-09-21T12:21:25.835 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.644617035Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-21T12:21:25.835 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.645164449Z level=info msg="Migration successfully executed" id="add unique index star.user_id_dashboard_id" duration=547.576µs 2024-09-21T12:21:25.835 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.647346585Z level=info msg="Executing migration" id="create org table v1" 2024-09-21T12:21:25.835 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.647949966Z level=info msg="Migration successfully executed" id="create org table v1" duration=605.569µs 2024-09-21T12:21:25.835 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.649799145Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-21T12:21:25.835 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.650313396Z level=info msg="Migration successfully executed" id="create index UQE_org_name - v1" duration=512.817µs 2024-09-21T12:21:25.835 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.652580752Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-21T12:21:25.836 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.653083636Z level=info msg="Migration successfully executed" id="create org_user table v1" duration=502.734µs 2024-09-21T12:21:25.836 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.655290142Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-21T12:21:25.836 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.655796654Z level=info msg="Migration successfully executed" id="create index IDX_org_user_org_id - v1" duration=506.296µs 2024-09-21T12:21:25.836 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.657932985Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-21T12:21:25.836 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.658478375Z level=info msg="Migration successfully executed" id="create index UQE_org_user_org_id_user_id - v1" duration=543.646µs 2024-09-21T12:21:25.836 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.660773465Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-21T12:21:25.836 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.6613305Z level=info msg="Migration successfully executed" id="create index IDX_org_user_user_id - v1" duration=558.953µs 2024-09-21T12:21:25.836 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.663284444Z level=info msg="Executing migration" id="Update org table charset" 2024-09-21T12:21:25.836 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.663323053Z level=info msg="Migration successfully executed" id="Update org table charset" duration=40.066µs 2024-09-21T12:21:25.836 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.663841263Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-21T12:21:25.836 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.663856428Z level=info msg="Migration successfully executed" id="Update org_user table charset" duration=15.794µs 2024-09-21T12:21:25.836 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.666418172Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-21T12:21:25.836 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.666549144Z level=info msg="Migration successfully executed" id="Migrate all Read Only Viewers to Viewers" duration=131.226µs 2024-09-21T12:21:25.837 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.669076783Z level=info msg="Executing migration" id="create dashboard table" 2024-09-21T12:21:25.837 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.669633729Z level=info msg="Migration successfully executed" id="create dashboard table" duration=555.426µs 2024-09-21T12:21:25.837 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.671808225Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-21T12:21:25.837 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.67237945Z level=info msg="Migration successfully executed" id="add index dashboard.account_id" duration=573.29µs 2024-09-21T12:21:25.837 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.672747818Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-21T12:21:25.837 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.673287969Z level=info msg="Migration successfully executed" id="add unique index dashboard_account_id_slug" duration=539.865µs 2024-09-21T12:21:25.837 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.673664603Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-21T12:21:25.837 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.674124977Z level=info msg="Migration successfully executed" id="create dashboard_tag table" duration=460.158µs 2024-09-21T12:21:25.837 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.674504265Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-21T12:21:25.837 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.675012741Z level=info msg="Migration successfully executed" id="add unique index dashboard_tag.dasboard_id_term" duration=508.306µs 2024-09-21T12:21:25.837 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.675408123Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-21T12:21:25.837 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.675944523Z level=info msg="Migration successfully executed" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" duration=535.056µs 2024-09-21T12:21:25.837 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.676361446Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-21T12:21:25.837 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.679865226Z level=info msg="Migration successfully executed" id="Rename table dashboard to dashboard_v1 - v1" duration=3.503582ms 2024-09-21T12:21:25.837 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.680244628Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-21T12:21:25.838 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.680747902Z level=info msg="Migration successfully executed" id="create dashboard v2" duration=502.998µs 2024-09-21T12:21:25.838 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.681110951Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-21T12:21:25.838 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.681585758Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_org_id - v2" duration=474.659µs 2024-09-21T12:21:25.838 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.68197258Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-21T12:21:25.838 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.682492382Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_org_id_slug - v2" duration=519.687µs 2024-09-21T12:21:25.838 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.682867636Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-21T12:21:25.838 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.683097289Z level=info msg="Migration successfully executed" id="copy dashboard v1 to v2" duration=229.466µs 2024-09-21T12:21:25.838 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.683735624Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-21T12:21:25.838 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.684328471Z level=info msg="Migration successfully executed" id="drop table dashboard_v1" duration=593.515µs 2024-09-21T12:21:25.838 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.684701312Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-21T12:21:25.838 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.684739878Z level=info msg="Migration successfully executed" id="alter dashboard.data to mediumtext v1" duration=38.899µs 2024-09-21T12:21:25.838 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.685587386Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-21T12:21:25.838 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.686667353Z level=info msg="Migration successfully executed" id="Add column updated_by in dashboard - v2" duration=1.079431ms 2024-09-21T12:21:25.839 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.687042534Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-21T12:21:25.839 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.688142307Z level=info msg="Migration successfully executed" id="Add column created_by in dashboard - v2" duration=1.10009ms 2024-09-21T12:21:25.839 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.68851174Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-21T12:21:25.839 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.68957516Z level=info msg="Migration successfully executed" id="Add column gnetId in dashboard" duration=1.063406ms 2024-09-21T12:21:25.839 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.689945122Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-21T12:21:25.839 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.690598317Z level=info msg="Migration successfully executed" id="Add index for gnetId in dashboard" duration=653.304µs 2024-09-21T12:21:25.839 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.692906323Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-21T12:21:25.839 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.693996837Z level=info msg="Migration successfully executed" id="Add column plugin_id in dashboard" duration=1.090093ms 2024-09-21T12:21:25.839 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.695869541Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-21T12:21:25.839 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.696502419Z level=info msg="Migration successfully executed" id="Add index for plugin_id in dashboard" duration=634.526µs 2024-09-21T12:21:25.839 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.69843899Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-21T12:21:25.839 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.698934584Z level=info msg="Migration successfully executed" id="Add index for dashboard_id in dashboard_tag" duration=495.29µs 2024-09-21T12:21:25.839 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.701292479Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-21T12:21:25.839 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.70131135Z level=info msg="Migration successfully executed" id="Update dashboard table charset" duration=19.87µs 2024-09-21T12:21:25.840 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.701761811Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-21T12:21:25.840 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.701781514Z level=info msg="Migration successfully executed" id="Update dashboard_tag table charset" duration=20.582µs 2024-09-21T12:21:25.840 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.703988914Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-21T12:21:25.840 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.705179823Z level=info msg="Migration successfully executed" id="Add column folder_id in dashboard" duration=1.191052ms 2024-09-21T12:21:25.840 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.706942116Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-21T12:21:25.840 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.708217944Z level=info msg="Migration successfully executed" id="Add column isFolder in dashboard" duration=1.278906ms 2024-09-21T12:21:25.840 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.709416038Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-21T12:21:25.840 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.710560857Z level=info msg="Migration successfully executed" id="Add column has_acl in dashboard" duration=1.144672ms 2024-09-21T12:21:25.840 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.712137016Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-21T12:21:25.840 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.713299064Z level=info msg="Migration successfully executed" id="Add column uid in dashboard" duration=1.16239ms 2024-09-21T12:21:25.840 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.71502149Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-21T12:21:25.840 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.715159222Z level=info msg="Migration successfully executed" id="Update uid column values in dashboard" duration=136.612µs 2024-09-21T12:21:25.840 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.717742537Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-21T12:21:25.840 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.718230372Z level=info msg="Migration successfully executed" id="Add unique index dashboard_org_id_uid" duration=487.436µs 2024-09-21T12:21:25.841 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.720586157Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-21T12:21:25.841 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.72112703Z level=info msg="Migration successfully executed" id="Remove unique index org_id_slug" duration=540.928µs 2024-09-21T12:21:25.841 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.723178649Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-21T12:21:25.841 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.723218292Z level=info msg="Migration successfully executed" id="Update dashboard title length" duration=41.442µs 2024-09-21T12:21:25.841 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.725722409Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-21T12:21:25.841 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.726328635Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_title_folder_id" duration=606.122µs 2024-09-21T12:21:25.841 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.728623591Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-21T12:21:25.841 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.729149154Z level=info msg="Migration successfully executed" id="create dashboard_provisioning" duration=525.324µs 2024-09-21T12:21:25.841 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.731316587Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-21T12:21:25.841 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.734538099Z level=info msg="Migration successfully executed" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" duration=3.220371ms 2024-09-21T12:21:25.841 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.734949437Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-21T12:21:25.841 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.73542957Z level=info msg="Migration successfully executed" id="create dashboard_provisioning v2" duration=479.906µs 2024-09-21T12:21:25.841 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.737893226Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-21T12:21:25.841 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.738411285Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_provisioning_dashboard_id - v2" duration=518.16µs 2024-09-21T12:21:25.842 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.740360692Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-21T12:21:25.842 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.740867573Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" duration=505.371µs 2024-09-21T12:21:25.842 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.743201184Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-21T12:21:25.842 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.743453357Z level=info msg="Migration successfully executed" id="copy dashboard_provisioning v1 to v2" duration=252.298µs 2024-09-21T12:21:25.842 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.74591061Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-21T12:21:25.842 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.746428155Z level=info msg="Migration successfully executed" id="drop dashboard_provisioning_tmp_qwerty" duration=519.493µs 2024-09-21T12:21:25.842 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.748557139Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-21T12:21:25.842 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.749790874Z level=info msg="Migration successfully executed" id="Add check_sum column" duration=1.232443ms 2024-09-21T12:21:25.842 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.751311623Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-21T12:21:25.842 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.751804413Z level=info msg="Migration successfully executed" id="Add index for dashboard_title" duration=492.776µs 2024-09-21T12:21:25.842 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.753951121Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-21T12:21:25.842 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.754080331Z level=info msg="Migration successfully executed" id="delete tags for deleted dashboards" duration=129.262µs 2024-09-21T12:21:25.842 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.756540527Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-21T12:21:25.843 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.756664042Z level=info msg="Migration successfully executed" id="delete stars for deleted dashboards" duration=123.598µs 2024-09-21T12:21:25.843 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.759204727Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-21T12:21:25.843 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.759743995Z level=info msg="Migration successfully executed" id="Add index for dashboard_is_folder" duration=539.462µs 2024-09-21T12:21:25.843 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.761832897Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-21T12:21:25.843 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.76321352Z level=info msg="Migration successfully executed" id="Add isPublic for dashboard" duration=1.380474ms 2024-09-21T12:21:25.843 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.764367834Z level=info msg="Executing migration" id="create data_source table" 2024-09-21T12:21:25.843 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.765110576Z level=info msg="Migration successfully executed" id="create data_source table" duration=765.532µs 2024-09-21T12:21:25.843 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.765537202Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-21T12:21:25.843 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.766065866Z level=info msg="Migration successfully executed" id="add index data_source.account_id" duration=528.382µs 2024-09-21T12:21:25.843 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.768019668Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-21T12:21:25.843 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.768589092Z level=info msg="Migration successfully executed" id="add unique index data_source.account_id_name" duration=571.12µs 2024-09-21T12:21:25.843 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.770698196Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-21T12:21:25.843 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.771225035Z level=info msg="Migration successfully executed" id="drop index IDX_data_source_account_id - v1" duration=526.87µs 2024-09-21T12:21:25.844 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.773218588Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-21T12:21:25.844 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.773720809Z level=info msg="Migration successfully executed" id="drop index UQE_data_source_account_id_name - v1" duration=500.668µs 2024-09-21T12:21:25.844 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.77414156Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-21T12:21:25.844 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.777759396Z level=info msg="Migration successfully executed" id="Rename table data_source to data_source_v1 - v1" duration=3.61477ms 2024-09-21T12:21:25.844 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.778214892Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-21T12:21:25.844 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.778877008Z level=info msg="Migration successfully executed" id="create data_source table v2" duration=662.015µs 2024-09-21T12:21:25.844 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.780867435Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-21T12:21:25.844 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.781410566Z level=info msg="Migration successfully executed" id="create index IDX_data_source_org_id - v2" duration=542.903µs 2024-09-21T12:21:25.844 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.783561825Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-21T12:21:25.844 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.78411985Z level=info msg="Migration successfully executed" id="create index UQE_data_source_org_id_name - v2" duration=557.959µs 2024-09-21T12:21:25.844 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.784498011Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-21T12:21:25.844 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.78491187Z level=info msg="Migration successfully executed" id="Drop old table data_source_v1 #2" duration=413.519µs 2024-09-21T12:21:25.844 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.785327364Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-21T12:21:25.844 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.786835586Z level=info msg="Migration successfully executed" id="Add column with_credentials" duration=1.50794ms 2024-09-21T12:21:25.845 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.787208865Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-21T12:21:25.845 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.788645178Z level=info msg="Migration successfully executed" id="Add secure json data column" duration=1.436056ms 2024-09-21T12:21:25.845 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.789053042Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-21T12:21:25.845 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.789068975Z level=info msg="Migration successfully executed" id="Update data_source table charset" duration=17.924µs 2024-09-21T12:21:25.845 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.789529207Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-21T12:21:25.845 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.789671311Z level=info msg="Migration successfully executed" id="Update initial version to 1" duration=142.291µs 2024-09-21T12:21:25.845 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.790378647Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-21T12:21:25.845 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.791799637Z level=info msg="Migration successfully executed" id="Add read_only data column" duration=1.42082ms 2024-09-21T12:21:25.845 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.792186387Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-21T12:21:25.845 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.792333764Z level=info msg="Migration successfully executed" id="Migrate logging ds to loki ds" duration=147.373µs 2024-09-21T12:21:25.845 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.79300041Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-21T12:21:25.845 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.793137501Z level=info msg="Migration successfully executed" id="Update json_data with nulls" duration=137.047µs 2024-09-21T12:21:25.845 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.793559043Z level=info msg="Executing migration" id="Add uid column" 2024-09-21T12:21:25.845 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.794983531Z level=info msg="Migration successfully executed" id="Add uid column" duration=1.424101ms 2024-09-21T12:21:25.845 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.795353017Z level=info msg="Executing migration" id="Update uid value" 2024-09-21T12:21:25.846 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.795499054Z level=info msg="Migration successfully executed" id="Update uid value" duration=145.326µs 2024-09-21T12:21:25.846 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.796151403Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-21T12:21:25.846 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.796700531Z level=info msg="Migration successfully executed" id="Add unique index datasource_org_id_uid" duration=548.865µs 2024-09-21T12:21:25.846 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.797078062Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-21T12:21:25.846 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.797584087Z level=info msg="Migration successfully executed" id="add unique index datasource_org_id_is_default" duration=506.38µs 2024-09-21T12:21:25.846 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.797960785Z level=info msg="Executing migration" id="create api_key table" 2024-09-21T12:21:25.846 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.798518175Z level=info msg="Migration successfully executed" id="create api_key table" duration=557.214µs 2024-09-21T12:21:25.846 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.798909891Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-21T12:21:25.846 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.799435336Z level=info msg="Migration successfully executed" id="add index api_key.account_id" duration=525.279µs 2024-09-21T12:21:25.846 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.799821365Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-21T12:21:25.846 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.800325422Z level=info msg="Migration successfully executed" id="add index api_key.key" duration=504.452µs 2024-09-21T12:21:25.846 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.800715539Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-21T12:21:25.846 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.801272213Z level=info msg="Migration successfully executed" id="add index api_key.account_id_name" duration=556.474µs 2024-09-21T12:21:25.846 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.801672933Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-21T12:21:25.847 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.802177306Z level=info msg="Migration successfully executed" id="drop index IDX_api_key_account_id - v1" duration=504.365µs 2024-09-21T12:21:25.847 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.802538058Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-21T12:21:25.847 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.803053511Z level=info msg="Migration successfully executed" id="drop index UQE_api_key_key - v1" duration=515.273µs 2024-09-21T12:21:25.847 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.803409615Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-21T12:21:25.847 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.803916893Z level=info msg="Migration successfully executed" id="drop index UQE_api_key_account_id_name - v1" duration=506.948µs 2024-09-21T12:21:25.847 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.804330674Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-21T12:21:25.847 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.808129245Z level=info msg="Migration successfully executed" id="Rename table api_key to api_key_v1 - v1" duration=3.79827ms 2024-09-21T12:21:25.847 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.808514301Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-21T12:21:25.847 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.80904762Z level=info msg="Migration successfully executed" id="create api_key table v2" duration=533.002µs 2024-09-21T12:21:25.847 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.809461369Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-21T12:21:25.847 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.809984641Z level=info msg="Migration successfully executed" id="create index IDX_api_key_org_id - v2" duration=523.157µs 2024-09-21T12:21:25.847 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.810397881Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-21T12:21:25.847 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.810908456Z level=info msg="Migration successfully executed" id="create index UQE_api_key_key - v2" duration=510.402µs 2024-09-21T12:21:25.848 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.811286359Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-21T12:21:25.848 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.811791613Z level=info msg="Migration successfully executed" id="create index UQE_api_key_org_id_name - v2" duration=505.056µs 2024-09-21T12:21:25.848 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.812165429Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-21T12:21:25.848 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.812426142Z level=info msg="Migration successfully executed" id="copy api_key v1 to v2" duration=260.457µs 2024-09-21T12:21:25.848 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.812899278Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-21T12:21:25.848 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.813309693Z level=info msg="Migration successfully executed" id="Drop old table api_key_v1" duration=410.228µs 2024-09-21T12:21:25.848 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.813718206Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-21T12:21:25.848 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.813732775Z level=info msg="Migration successfully executed" id="Update api_key table charset" duration=15.229µs 2024-09-21T12:21:25.848 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.814184607Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-21T12:21:25.848 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.815644579Z level=info msg="Migration successfully executed" id="Add expires to api_key table" duration=1.461197ms 2024-09-21T12:21:25.848 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.816031426Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-21T12:21:25.848 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.817529445Z level=info msg="Migration successfully executed" id="Add service account foreign key" duration=1.497774ms 2024-09-21T12:21:25.848 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.817944716Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-21T12:21:25.848 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.818065376Z level=info msg="Migration successfully executed" id="set service account foreign key to nil if 0" duration=120.897µs 2024-09-21T12:21:25.848 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.818473312Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-21T12:21:25.848 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.820042259Z level=info msg="Migration successfully executed" id="Add last_used_at to api_key table" duration=1.568591ms 2024-09-21T12:21:25.849 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.820406449Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-21T12:21:25.849 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.821941878Z level=info msg="Migration successfully executed" id="Add is_revoked column to api_key table" duration=1.535158ms 2024-09-21T12:21:25.849 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.822346404Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-21T12:21:25.849 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.822908868Z level=info msg="Migration successfully executed" id="create dashboard_snapshot table v4" duration=562.198µs 2024-09-21T12:21:25.849 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.823288579Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-21T12:21:25.849 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.823695047Z level=info msg="Migration successfully executed" id="drop table dashboard_snapshot_v4 #1" duration=406.152µs 2024-09-21T12:21:25.849 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.824085915Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-21T12:21:25.849 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.824848766Z level=info msg="Migration successfully executed" id="create dashboard_snapshot table v5 #2" duration=762.296µs 2024-09-21T12:21:25.849 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.825195607Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-21T12:21:25.849 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.825718793Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_snapshot_key - v5" duration=522.83µs 2024-09-21T12:21:25.849 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.826066766Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-21T12:21:25.849 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.826610228Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_snapshot_delete_key - v5" duration=542.706µs 2024-09-21T12:21:25.849 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.826950644Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-21T12:21:25.849 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.827446447Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_snapshot_user_id - v5" duration=495.537µs 2024-09-21T12:21:25.850 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.827781666Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-21T12:21:25.850 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.827826682Z level=info msg="Migration successfully executed" id="alter dashboard_snapshot to mediumtext v2" duration=45.23µs 2024-09-21T12:21:25.850 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.8282247Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-21T12:21:25.850 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.828238709Z level=info msg="Migration successfully executed" id="Update dashboard_snapshot table charset" duration=14.638µs 2024-09-21T12:21:25.850 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.828601298Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-21T12:21:25.850 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.830218097Z level=info msg="Migration successfully executed" id="Add column external_delete_url to dashboard_snapshots table" duration=1.616508ms 2024-09-21T12:21:25.850 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.830554655Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-21T12:21:25.880 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:25.902 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:25.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:25.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:26.004 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:26.025 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:26.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:26.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:26.075 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-mon[33277]: 4.16 scrub starts 2024-09-21T12:21:26.075 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-mon[33277]: 4.16 scrub ok 2024-09-21T12:21:26.075 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-mon[33277]: 2.1 scrub starts 2024-09-21T12:21:26.076 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-mon[33277]: 2.1 scrub ok 2024-09-21T12:21:26.076 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-mon[33277]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:21:26.076 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:21:26.076 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-mon[33277]: osdmap e85: 8 total, 8 up, 8 in 2024-09-21T12:21:26.076 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:26.076 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:26.076 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:26.076 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:26.076 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:26.077 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.832200141Z level=info msg="Migration successfully executed" id="Add encrypted dashboard json column" duration=1.645037ms 2024-09-21T12:21:26.077 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.832590592Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-21T12:21:26.077 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.832630166Z level=info msg="Migration successfully executed" id="Change dashboard_encrypted column to MEDIUMBLOB" duration=39.915µs 2024-09-21T12:21:26.077 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.833045153Z level=info msg="Executing migration" id="create quota table v1" 2024-09-21T12:21:26.077 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.833500543Z level=info msg="Migration successfully executed" id="create quota table v1" duration=453.48µs 2024-09-21T12:21:26.077 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.83386419Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-21T12:21:26.077 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.834557286Z level=info msg="Migration successfully executed" id="create index UQE_quota_org_id_user_id_target - v1" duration=692.091µs 2024-09-21T12:21:26.077 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.835023808Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-21T12:21:26.077 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.83504551Z level=info msg="Migration successfully executed" id="Update quota table charset" duration=22.544µs 2024-09-21T12:21:26.077 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.835572546Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-21T12:21:26.077 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.836297049Z level=info msg="Migration successfully executed" id="create plugin_setting table" duration=724.271µs 2024-09-21T12:21:26.077 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.836734213Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-21T12:21:26.077 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.837478058Z level=info msg="Migration successfully executed" id="create index UQE_plugin_setting_org_id_plugin_id - v1" duration=742.605µs 2024-09-21T12:21:26.078 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.837934582Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-21T12:21:26.078 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.840940473Z level=info msg="Migration successfully executed" id="Add column plugin_version to plugin_settings" duration=3.005572ms 2024-09-21T12:21:26.078 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.841412965Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-21T12:21:26.078 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.841435674Z level=info msg="Migration successfully executed" id="Update plugin_setting table charset" duration=24.397µs 2024-09-21T12:21:26.078 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.841970563Z level=info msg="Executing migration" id="create session table" 2024-09-21T12:21:26.078 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.842782796Z level=info msg="Migration successfully executed" id="create session table" duration=812.641µs 2024-09-21T12:21:26.078 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.843258363Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-21T12:21:26.078 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.843339268Z level=info msg="Migration successfully executed" id="Drop old table playlist table" duration=81.204µs 2024-09-21T12:21:26.078 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.843871863Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-21T12:21:26.078 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.843951895Z level=info msg="Migration successfully executed" id="Drop old table playlist_item table" duration=81.436µs 2024-09-21T12:21:26.078 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.844492033Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-21T12:21:26.078 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.845166047Z level=info msg="Migration successfully executed" id="create playlist table v2" duration=673.813µs 2024-09-21T12:21:26.078 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.845630022Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-21T12:21:26.078 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.846329316Z level=info msg="Migration successfully executed" id="create playlist item table v2" duration=699µs 2024-09-21T12:21:26.079 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.846780217Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-21T12:21:26.079 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.846801111Z level=info msg="Migration successfully executed" id="Update playlist table charset" duration=21.858µs 2024-09-21T12:21:26.079 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.847343899Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-21T12:21:26.079 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.847364706Z level=info msg="Migration successfully executed" id="Update playlist_item table charset" duration=21.414µs 2024-09-21T12:21:26.079 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.847886522Z level=info msg="Executing migration" id="Add playlist column created_at" 2024-09-21T12:21:26.079 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.850745948Z level=info msg="Migration successfully executed" id="Add playlist column created_at" duration=2.860649ms 2024-09-21T12:21:26.079 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.851192557Z level=info msg="Executing migration" id="Add playlist column updated_at" 2024-09-21T12:21:26.079 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.854311167Z level=info msg="Migration successfully executed" id="Add playlist column updated_at" duration=3.117528ms 2024-09-21T12:21:26.079 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.85474094Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-21T12:21:26.079 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.854839264Z level=info msg="Migration successfully executed" id="drop preferences table v2" duration=95.425µs 2024-09-21T12:21:26.079 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.855326811Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-21T12:21:26.079 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.855407318Z level=info msg="Migration successfully executed" id="drop preferences table v3" duration=80.639µs 2024-09-21T12:21:26.079 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.855914914Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-21T12:21:26.080 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.85668011Z level=info msg="Migration successfully executed" id="create preferences table v3" duration=764.886µs 2024-09-21T12:21:26.080 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.857151043Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-21T12:21:26.080 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.85720749Z level=info msg="Migration successfully executed" id="Update preferences table charset" duration=57.034µs 2024-09-21T12:21:26.080 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.857806746Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-21T12:21:26.080 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.860962036Z level=info msg="Migration successfully executed" id="Add column team_id in preferences" duration=3.154954ms 2024-09-21T12:21:26.080 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.861408682Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-21T12:21:26.080 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.861536511Z level=info msg="Migration successfully executed" id="Update team_id column values in preferences" duration=129.875µs 2024-09-21T12:21:26.080 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.862063369Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-21T12:21:26.080 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.864692314Z level=info msg="Migration successfully executed" id="Add column week_start in preferences" duration=2.628781ms 2024-09-21T12:21:26.080 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.865098045Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-21T12:21:26.080 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.867068618Z level=info msg="Migration successfully executed" id="Add column preferences.json_data" duration=1.970194ms 2024-09-21T12:21:26.080 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.867411507Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-21T12:21:26.080 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.867452748Z level=info msg="Migration successfully executed" id="alter preferences.json_data to mediumtext v1" duration=41.624µs 2024-09-21T12:21:26.080 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.867905413Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-21T12:21:26.081 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.868499038Z level=info msg="Migration successfully executed" id="Add preferences index org_id" duration=593.264µs 2024-09-21T12:21:26.081 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.868847052Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-21T12:21:26.081 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.869366246Z level=info msg="Migration successfully executed" id="Add preferences index user_id" duration=518.922µs 2024-09-21T12:21:26.081 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.869723328Z level=info msg="Executing migration" id="create alert table v1" 2024-09-21T12:21:26.081 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.870451794Z level=info msg="Migration successfully executed" id="create alert table v1" duration=728.155µs 2024-09-21T12:21:26.081 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.870801635Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-21T12:21:26.081 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.871389961Z level=info msg="Migration successfully executed" id="add index alert org_id & id " duration=588.033µs 2024-09-21T12:21:26.081 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.871729845Z level=info msg="Executing migration" id="add index alert state" 2024-09-21T12:21:26.081 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.87224374Z level=info msg="Migration successfully executed" id="add index alert state" duration=513.637µs 2024-09-21T12:21:26.081 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.872564944Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-21T12:21:26.081 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.873103367Z level=info msg="Migration successfully executed" id="add index alert dashboard_id" duration=538.98µs 2024-09-21T12:21:26.081 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.87344876Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-21T12:21:26.081 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.873887811Z level=info msg="Migration successfully executed" id="Create alert_rule_tag table v1" duration=437.346µs 2024-09-21T12:21:26.081 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.874219257Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-21T12:21:26.081 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.874762712Z level=info msg="Migration successfully executed" id="Add unique index alert_rule_tag.alert_id_tag_id" duration=543.277µs 2024-09-21T12:21:26.082 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.877099988Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-21T12:21:26.082 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.877690581Z level=info msg="Migration successfully executed" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" duration=592.738µs 2024-09-21T12:21:26.082 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.879420056Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-21T12:21:26.083 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.88569657Z level=info msg="Migration successfully executed" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" duration=6.273839ms 2024-09-21T12:21:26.083 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.88640425Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-21T12:21:26.083 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.886925052Z level=info msg="Migration successfully executed" id="Create alert_rule_tag table v2" duration=517.27µs 2024-09-21T12:21:26.083 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.887251471Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-21T12:21:26.083 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.887777698Z level=info msg="Migration successfully executed" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" duration=524.437µs 2024-09-21T12:21:26.084 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.890015628Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-21T12:21:26.084 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.890218956Z level=info msg="Migration successfully executed" id="copy alert_rule_tag v1 to v2" duration=203.668µs 2024-09-21T12:21:26.084 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.892445001Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-21T12:21:26.084 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.892825022Z level=info msg="Migration successfully executed" id="drop table alert_rule_tag_v1" duration=379.733µs 2024-09-21T12:21:26.084 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.894984006Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-21T12:21:26.084 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.895510023Z level=info msg="Migration successfully executed" id="create alert_notification table v1" duration=527.147µs 2024-09-21T12:21:26.084 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.89756584Z level=info msg="Executing migration" id="Add column is_default" 2024-09-21T12:21:26.084 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.900933922Z level=info msg="Migration successfully executed" id="Add column is_default" duration=3.366588ms 2024-09-21T12:21:26.084 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.901385082Z level=info msg="Executing migration" id="Add column frequency" 2024-09-21T12:21:26.084 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.903711989Z level=info msg="Migration successfully executed" id="Add column frequency" duration=2.326464ms 2024-09-21T12:21:26.084 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.904121864Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-21T12:21:26.084 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.906431623Z level=info msg="Migration successfully executed" id="Add column send_reminder" duration=2.311282ms 2024-09-21T12:21:26.084 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.90676332Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-21T12:21:26.084 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.908809704Z level=info msg="Migration successfully executed" id="Add column disable_resolve_message" duration=2.046074ms 2024-09-21T12:21:26.084 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.909169369Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-21T12:21:26.085 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.909707908Z level=info msg="Migration successfully executed" id="add index alert_notification org_id & name" duration=538.191µs 2024-09-21T12:21:26.085 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.911861238Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-21T12:21:26.085 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.911879816Z level=info msg="Migration successfully executed" id="Update alert table charset" duration=19.507µs 2024-09-21T12:21:26.085 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.914261893Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-21T12:21:26.085 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.914309168Z level=info msg="Migration successfully executed" id="Update alert_notification table charset" duration=50.112µs 2024-09-21T12:21:26.085 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.91655456Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-21T12:21:26.085 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.917058051Z level=info msg="Migration successfully executed" id="create notification_journal table v1" duration=503.885µs 2024-09-21T12:21:26.085 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.91921275Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-21T12:21:26.085 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.919835663Z level=info msg="Migration successfully executed" id="add index notification_journal org_id & alert_id & notifier_id" duration=625.733µs 2024-09-21T12:21:26.085 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.921587464Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-21T12:21:26.085 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.922142439Z level=info msg="Migration successfully executed" id="drop alert_notification_journal" duration=554.662µs 2024-09-21T12:21:26.085 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.92421952Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-21T12:21:26.085 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.924740434Z level=info msg="Migration successfully executed" id="create alert_notification_state table v1" duration=520.654µs 2024-09-21T12:21:26.085 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.925105161Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-21T12:21:26.086 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.925643166Z level=info msg="Migration successfully executed" id="add index alert_notification_state org_id & alert_id & notifier_id" duration=537.658µs 2024-09-21T12:21:26.086 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.927701253Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-21T12:21:26.086 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.929974117Z level=info msg="Migration successfully executed" id="Add for to alert table" duration=2.276124ms 2024-09-21T12:21:26.086 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.93033622Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-21T12:21:26.086 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.932596751Z level=info msg="Migration successfully executed" id="Add column uid in alert_notification" duration=2.26017ms 2024-09-21T12:21:26.086 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.932980256Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-21T12:21:26.086 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.933101317Z level=info msg="Migration successfully executed" id="Update uid column values in alert_notification" duration=121.757µs 2024-09-21T12:21:26.086 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.933509637Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-21T12:21:26.086 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.93403933Z level=info msg="Migration successfully executed" id="Add unique index alert_notification_org_id_uid" duration=529.737µs 2024-09-21T12:21:26.086 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.934437338Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-21T12:21:26.086 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.934969655Z level=info msg="Migration successfully executed" id="Remove unique index org_id_name" duration=532.161µs 2024-09-21T12:21:26.086 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.935359515Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-21T12:21:26.086 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.937489441Z level=info msg="Migration successfully executed" id="Add column secure_settings in alert_notification" duration=2.129677ms 2024-09-21T12:21:26.086 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.937851843Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-21T12:21:26.087 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.937889193Z level=info msg="Migration successfully executed" id="alter alert.settings to mediumtext" duration=37.82µs 2024-09-21T12:21:26.087 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.938347737Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-21T12:21:26.087 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.938879181Z level=info msg="Migration successfully executed" id="Add non-unique index alert_notification_state_alert_id" duration=530.695µs 2024-09-21T12:21:26.087 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.939239088Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-21T12:21:26.087 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.939841881Z level=info msg="Migration successfully executed" id="Add non-unique index alert_rule_tag_alert_id" duration=602.493µs 2024-09-21T12:21:26.087 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.940220929Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-21T12:21:26.087 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.940280959Z level=info msg="Migration successfully executed" id="Drop old annotation table v4" duration=57.743µs 2024-09-21T12:21:26.087 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.940996643Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-21T12:21:26.087 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.941564767Z level=info msg="Migration successfully executed" id="create annotation table v5" duration=568.471µs 2024-09-21T12:21:26.087 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.941938252Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-21T12:21:26.087 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.942472364Z level=info msg="Migration successfully executed" id="add index annotation 0 v3" duration=535.249µs 2024-09-21T12:21:26.087 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.942846023Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-21T12:21:26.087 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.943378088Z level=info msg="Migration successfully executed" id="add index annotation 1 v3" duration=532.187µs 2024-09-21T12:21:26.087 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.943763089Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-21T12:21:26.087 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.944314102Z level=info msg="Migration successfully executed" id="add index annotation 2 v3" duration=551.636µs 2024-09-21T12:21:26.087 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.944680321Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-21T12:21:26.088 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.94528233Z level=info msg="Migration successfully executed" id="add index annotation 3 v3" duration=602.295µs 2024-09-21T12:21:26.088 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.945649391Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-21T12:21:26.088 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.946273043Z level=info msg="Migration successfully executed" id="add index annotation 4 v3" duration=623.483µs 2024-09-21T12:21:26.088 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.946663422Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-21T12:21:26.088 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.946683298Z level=info msg="Migration successfully executed" id="Update annotation table charset" duration=20.077µs 2024-09-21T12:21:26.088 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.947353485Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-21T12:21:26.088 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.949708995Z level=info msg="Migration successfully executed" id="Add column region_id to annotation table" duration=2.355727ms 2024-09-21T12:21:26.088 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.950100971Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-21T12:21:26.088 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.950648534Z level=info msg="Migration successfully executed" id="Drop category_id index" duration=547.961µs 2024-09-21T12:21:26.088 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.951029656Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-21T12:21:26.088 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.953221944Z level=info msg="Migration successfully executed" id="Add column tags to annotation table" duration=2.192656ms 2024-09-21T12:21:26.088 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.95359516Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-21T12:21:26.088 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.95403991Z level=info msg="Migration successfully executed" id="Create annotation_tag table v2" duration=444.606µs 2024-09-21T12:21:26.088 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.954434412Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-21T12:21:26.088 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.955011187Z level=info msg="Migration successfully executed" id="Add unique index annotation_tag.annotation_id_tag_id" duration=576.111µs 2024-09-21T12:21:26.089 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.955382692Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-21T12:21:26.089 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.955934962Z level=info msg="Migration successfully executed" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" duration=552.413µs 2024-09-21T12:21:26.089 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.956302917Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-21T12:21:26.089 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.962652757Z level=info msg="Migration successfully executed" id="Rename table annotation_tag to annotation_tag_v2 - v2" duration=6.348515ms 2024-09-21T12:21:26.089 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.963025762Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-21T12:21:26.089 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.963483897Z level=info msg="Migration successfully executed" id="Create annotation_tag table v3" duration=457.883µs 2024-09-21T12:21:26.089 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.963876254Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-21T12:21:26.089 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.964426439Z level=info msg="Migration successfully executed" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" duration=550.171µs 2024-09-21T12:21:26.089 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.964807099Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-21T12:21:26.089 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.965011292Z level=info msg="Migration successfully executed" id="copy annotation_tag v2 to v3" duration=204.65µs 2024-09-21T12:21:26.089 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.965389144Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-21T12:21:26.089 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.965750062Z level=info msg="Migration successfully executed" id="drop table annotation_tag_v2" duration=361.117µs 2024-09-21T12:21:26.089 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.966162484Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-21T12:21:26.089 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.966295688Z level=info msg="Migration successfully executed" id="Update alert annotations and set TEXT to empty" duration=149.351µs 2024-09-21T12:21:26.089 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.966934296Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-21T12:21:26.090 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.969241711Z level=info msg="Migration successfully executed" id="Add created time to annotation table" duration=2.307639ms 2024-09-21T12:21:26.090 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.969606937Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-21T12:21:26.090 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.971899865Z level=info msg="Migration successfully executed" id="Add updated time to annotation table" duration=2.292643ms 2024-09-21T12:21:26.090 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.972265141Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-21T12:21:26.090 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.972804208Z level=info msg="Migration successfully executed" id="Add index for created in annotation table" duration=538.954µs 2024-09-21T12:21:26.090 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.973177814Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-21T12:21:26.090 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.973711586Z level=info msg="Migration successfully executed" id="Add index for updated in annotation table" duration=534.06µs 2024-09-21T12:21:26.090 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.974098074Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-21T12:21:26.090 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.974247354Z level=info msg="Migration successfully executed" id="Convert existing annotations from seconds to milliseconds" duration=146.67µs 2024-09-21T12:21:26.090 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.974901698Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-21T12:21:26.090 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.977248087Z level=info msg="Migration successfully executed" id="Add epoch_end column" duration=2.346629ms 2024-09-21T12:21:26.090 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.977608005Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-21T12:21:26.090 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.978172709Z level=info msg="Migration successfully executed" id="Add index for epoch_end" duration=564.389µs 2024-09-21T12:21:26.090 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.978553972Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-21T12:21:26.091 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.978660989Z level=info msg="Migration successfully executed" id="Make epoch_end the same as epoch" duration=106.926µs 2024-09-21T12:21:26.091 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.979320193Z level=info msg="Executing migration" id="Move region to single row" 2024-09-21T12:21:26.091 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.979548881Z level=info msg="Migration successfully executed" id="Move region to single row" duration=229.128µs 2024-09-21T12:21:26.091 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.980081703Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-21T12:21:26.091 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.980609359Z level=info msg="Migration successfully executed" id="Remove index org_id_epoch from annotation table" duration=527.414µs 2024-09-21T12:21:26.091 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.980945806Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-21T12:21:26.091 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.981495735Z level=info msg="Migration successfully executed" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" duration=549.941µs 2024-09-21T12:21:26.091 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.983767365Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-21T12:21:26.091 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.98433916Z level=info msg="Migration successfully executed" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" duration=571.596µs 2024-09-21T12:21:26.091 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.986631834Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-21T12:21:26.091 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.987230266Z level=info msg="Migration successfully executed" id="Add index for org_id_epoch_end_epoch on annotation table" duration=598.165µs 2024-09-21T12:21:26.091 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.989362223Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-21T12:21:26.091 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.989912943Z level=info msg="Migration successfully executed" id="Remove index org_id_epoch_epoch_end from annotation table" duration=550.455µs 2024-09-21T12:21:26.091 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.990269467Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-21T12:21:26.092 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.990812994Z level=info msg="Migration successfully executed" id="Add index for alert_id on annotation table" duration=544.635µs 2024-09-21T12:21:26.092 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.993049786Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-21T12:21:26.092 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.993158718Z level=info msg="Migration successfully executed" id="Increase tags column to length 4096" duration=114.9µs 2024-09-21T12:21:26.092 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.995520469Z level=info msg="Executing migration" id="create test_data table" 2024-09-21T12:21:26.092 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.996075551Z level=info msg="Migration successfully executed" id="create test_data table" duration=555.214µs 2024-09-21T12:21:26.092 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.998362204Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-21T12:21:26.092 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.999022796Z level=info msg="Migration successfully executed" id="create dashboard_version table v1" duration=664.311µs 2024-09-21T12:21:26.092 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.99940087Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-21T12:21:26.092 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:25.99997549Z level=info msg="Migration successfully executed" id="add index dashboard_version.dashboard_id" duration=574.748µs 2024-09-21T12:21:26.092 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.001912931Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-21T12:21:26.092 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.002484302Z level=info msg="Migration successfully executed" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" duration=571.113µs 2024-09-21T12:21:26.092 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.004658297Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-21T12:21:26.092 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.004856863Z level=info msg="Migration successfully executed" id="Set dashboard version to 1 where 0" duration=199.918µs 2024-09-21T12:21:26.092 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.007287951Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-21T12:21:26.093 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.007518347Z level=info msg="Migration successfully executed" id="save existing dashboard data in dashboard_version table v1" duration=230.34µs 2024-09-21T12:21:26.093 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.007850426Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-21T12:21:26.093 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.007887062Z level=info msg="Migration successfully executed" id="alter dashboard_version.data to mediumtext v1" duration=37.038µs 2024-09-21T12:21:26.093 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.010735079Z level=info msg="Executing migration" id="create team table" 2024-09-21T12:21:26.093 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.011313152Z level=info msg="Migration successfully executed" id="create team table" duration=581.416µs 2024-09-21T12:21:26.093 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.013554871Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-21T12:21:26.093 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.014250962Z level=info msg="Migration successfully executed" id="add index team.org_id" duration=711.843µs 2024-09-21T12:21:26.093 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.016065913Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-21T12:21:26.093 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.016714269Z level=info msg="Migration successfully executed" id="add unique index team_org_id_name" duration=650.232µs 2024-09-21T12:21:26.093 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.017078044Z level=info msg="Executing migration" id="Add column uid in team" 2024-09-21T12:21:26.093 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.019643271Z level=info msg="Migration successfully executed" id="Add column uid in team" duration=2.565606ms 2024-09-21T12:21:26.093 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.020012697Z level=info msg="Executing migration" id="Update uid column values in team" 2024-09-21T12:21:26.093 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.020124279Z level=info msg="Migration successfully executed" id="Update uid column values in team" duration=111.648µs 2024-09-21T12:21:26.094 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.022594765Z level=info msg="Executing migration" id="Add unique index team_org_id_uid" 2024-09-21T12:21:26.094 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.023196043Z level=info msg="Migration successfully executed" id="Add unique index team_org_id_uid" duration=602.425µs 2024-09-21T12:21:26.094 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.024875416Z level=info msg="Executing migration" id="create team member table" 2024-09-21T12:21:26.094 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.0253677Z level=info msg="Migration successfully executed" id="create team member table" duration=492.024µs 2024-09-21T12:21:26.094 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.025730994Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-21T12:21:26.094 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.026289026Z level=info msg="Migration successfully executed" id="add index team_member.org_id" duration=559.018µs 2024-09-21T12:21:26.094 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.028208109Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-21T12:21:26.094 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.028781489Z level=info msg="Migration successfully executed" id="add unique index team_member_org_id_team_id_user_id" duration=573.103µs 2024-09-21T12:21:26.094 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.030615545Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-21T12:21:26.094 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.031188341Z level=info msg="Migration successfully executed" id="add index team_member.team_id" duration=572.654µs 2024-09-21T12:21:26.094 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.033019628Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-21T12:21:26.094 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.035779978Z level=info msg="Migration successfully executed" id="Add column email to team table" duration=2.760196ms 2024-09-21T12:21:26.094 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.036149341Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-21T12:21:26.094 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.038975337Z level=info msg="Migration successfully executed" id="Add column external to team_member table" duration=2.825792ms 2024-09-21T12:21:26.095 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.039336444Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-21T12:21:26.095 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.041917556Z level=info msg="Migration successfully executed" id="Add column permission to team_member table" duration=2.579802ms 2024-09-21T12:21:26.095 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.042306002Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-21T12:21:26.095 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.042945187Z level=info msg="Migration successfully executed" id="create dashboard acl table" duration=639.014µs 2024-09-21T12:21:26.095 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.044853183Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-21T12:21:26.095 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.045437607Z level=info msg="Migration successfully executed" id="add index dashboard_acl_dashboard_id" duration=584.194µs 2024-09-21T12:21:26.095 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.04578851Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-21T12:21:26.095 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.046430175Z level=info msg="Migration successfully executed" id="add unique index dashboard_acl_dashboard_id_user_id" duration=641.507µs 2024-09-21T12:21:26.095 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.048526922Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-21T12:21:26.095 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.049152152Z level=info msg="Migration successfully executed" id="add unique index dashboard_acl_dashboard_id_team_id" duration=625.023µs 2024-09-21T12:21:26.095 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.051038754Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-21T12:21:26.095 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.051607081Z level=info msg="Migration successfully executed" id="add index dashboard_acl_user_id" duration=568.161µs 2024-09-21T12:21:26.095 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.053603266Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-21T12:21:26.096 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.054193631Z level=info msg="Migration successfully executed" id="add index dashboard_acl_team_id" duration=590.338µs 2024-09-21T12:21:26.096 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.055924752Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-21T12:21:26.096 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.056522794Z level=info msg="Migration successfully executed" id="add index dashboard_acl_org_id_role" duration=597.902µs 2024-09-21T12:21:26.096 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.058399079Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-21T12:21:26.096 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.058998151Z level=info msg="Migration successfully executed" id="add index dashboard_permission" duration=598.915µs 2024-09-21T12:21:26.096 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.061011941Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-21T12:21:26.096 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.061338071Z level=info msg="Migration successfully executed" id="save default acl rules in dashboard_acl table" duration=327.111µs 2024-09-21T12:21:26.096 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.061713637Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-21T12:21:26.096 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.061858366Z level=info msg="Migration successfully executed" id="delete acl rules for deleted dashboards and folders" duration=144.646µs 2024-09-21T12:21:26.096 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.062456721Z level=info msg="Executing migration" id="create tag table" 2024-09-21T12:21:26.096 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.062954694Z level=info msg="Migration successfully executed" id="create tag table" duration=498.338µs 2024-09-21T12:21:26.096 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.063334278Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-21T12:21:26.096 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.063904836Z level=info msg="Migration successfully executed" id="add index tag.key_value" duration=570.274µs 2024-09-21T12:21:26.096 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.064262109Z level=info msg="Executing migration" id="create login attempt table" 2024-09-21T12:21:26.097 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.064744416Z level=info msg="Migration successfully executed" id="create login attempt table" duration=481.98µs 2024-09-21T12:21:26.097 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.065114767Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-21T12:21:26.097 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.065687839Z level=info msg="Migration successfully executed" id="add index login_attempt.username" duration=572.92µs 2024-09-21T12:21:26.097 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.066048168Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-21T12:21:26.097 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.066617127Z level=info msg="Migration successfully executed" id="drop index IDX_login_attempt_username - v1" duration=568.965µs 2024-09-21T12:21:26.097 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.066958388Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-21T12:21:26.097 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.074686742Z level=info msg="Migration successfully executed" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" duration=7.728193ms 2024-09-21T12:21:26.097 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.075113214Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-21T12:21:26.097 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.075585978Z level=info msg="Migration successfully executed" id="create login_attempt v2" duration=472.676µs 2024-09-21T12:21:26.097 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.075929974Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-21T12:21:26.097 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.076493689Z level=info msg="Migration successfully executed" id="create index IDX_login_attempt_username - v2" duration=563.949µs 2024-09-21T12:21:26.097 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.076851475Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-21T12:21:26.097 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.07706358Z level=info msg="Migration successfully executed" id="copy login_attempt v1 to v2" duration=211.992µs 2024-09-21T12:21:26.098 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.07761513Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-21T12:21:26.098 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.078033291Z level=info msg="Migration successfully executed" id="drop login_attempt_tmp_qwerty" duration=418.431µs 2024-09-21T12:21:26.098 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.078440946Z level=info msg="Executing migration" id="create user auth table" 2024-09-21T12:21:26.098 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.078933858Z level=info msg="Migration successfully executed" id="create user auth table" duration=492.818µs 2024-09-21T12:21:26.098 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.079282465Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-21T12:21:26.098 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.079862609Z level=info msg="Migration successfully executed" id="create index IDX_user_auth_auth_module_auth_id - v1" duration=580.004µs 2024-09-21T12:21:26.098 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.080216014Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-21T12:21:26.098 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.080262801Z level=info msg="Migration successfully executed" id="alter user_auth.auth_id to length 190" duration=47.739µs 2024-09-21T12:21:26.098 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.081008318Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-21T12:21:26.126 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:26.147 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:26.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:26.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:26.249 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:26.270 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:26.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:26.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:26.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:25 smithi114 ceph-mon[40867]: 4.16 scrub starts 2024-09-21T12:21:26.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:25 smithi114 ceph-mon[40867]: 4.16 scrub ok 2024-09-21T12:21:26.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:25 smithi114 ceph-mon[40867]: 2.1 scrub starts 2024-09-21T12:21:26.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:25 smithi114 ceph-mon[40867]: 2.1 scrub ok 2024-09-21T12:21:26.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:25 smithi114 ceph-mon[40867]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:21:26.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:25 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:21:26.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:25 smithi114 ceph-mon[40867]: osdmap e85: 8 total, 8 up, 8 in 2024-09-21T12:21:26.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:25 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:26.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:25 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:26.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:25 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:26.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:25 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:26.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:25 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:26.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:25 smithi114 ceph-mon[30324]: 4.16 scrub starts 2024-09-21T12:21:26.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:25 smithi114 ceph-mon[30324]: 4.16 scrub ok 2024-09-21T12:21:26.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:25 smithi114 ceph-mon[30324]: 2.1 scrub starts 2024-09-21T12:21:26.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:25 smithi114 ceph-mon[30324]: 2.1 scrub ok 2024-09-21T12:21:26.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:25 smithi114 ceph-mon[30324]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:21:26.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:25 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:21:26.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:25 smithi114 ceph-mon[30324]: osdmap e85: 8 total, 8 up, 8 in 2024-09-21T12:21:26.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:25 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:26.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:25 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:26.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:25 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:26.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:25 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:26.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:25 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:26.285 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:21:26.327 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.086239592Z level=info msg="Migration successfully executed" id="Add OAuth access token to user_auth" duration=5.231351ms 2024-09-21T12:21:26.327 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.086656373Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-21T12:21:26.327 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.089522694Z level=info msg="Migration successfully executed" id="Add OAuth refresh token to user_auth" duration=2.867161ms 2024-09-21T12:21:26.327 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.08988271Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-21T12:21:26.327 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.092714174Z level=info msg="Migration successfully executed" id="Add OAuth token type to user_auth" duration=2.83141ms 2024-09-21T12:21:26.327 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.093073519Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-21T12:21:26.327 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.09604532Z level=info msg="Migration successfully executed" id="Add OAuth expiry to user_auth" duration=2.971462ms 2024-09-21T12:21:26.327 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.096378151Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-21T12:21:26.327 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.096935518Z level=info msg="Migration successfully executed" id="Add index to user_id column in user_auth" duration=557.295µs 2024-09-21T12:21:26.327 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.097369715Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-21T12:21:26.328 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.102589145Z level=info msg="Migration successfully executed" id="Add OAuth ID token to user_auth" duration=5.218978ms 2024-09-21T12:21:26.328 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.103025525Z level=info msg="Executing migration" id="create server_lock table" 2024-09-21T12:21:26.328 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.103751441Z level=info msg="Migration successfully executed" id="create server_lock table" duration=726.647µs 2024-09-21T12:21:26.328 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.104217727Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-21T12:21:26.328 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.104961675Z level=info msg="Migration successfully executed" id="add index server_lock.operation_uid" duration=744.357µs 2024-09-21T12:21:26.328 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.105301636Z level=info msg="Executing migration" id="create user auth token table" 2024-09-21T12:21:26.328 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.10584925Z level=info msg="Migration successfully executed" id="create user auth token table" duration=548.476µs 2024-09-21T12:21:26.328 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.106184522Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-21T12:21:26.329 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.106736771Z level=info msg="Migration successfully executed" id="add unique index user_auth_token.auth_token" duration=552.101µs 2024-09-21T12:21:26.329 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.107100672Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-21T12:21:26.329 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.107658778Z level=info msg="Migration successfully executed" id="add unique index user_auth_token.prev_auth_token" duration=557.926µs 2024-09-21T12:21:26.329 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.108004557Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-21T12:21:26.329 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.108620232Z level=info msg="Migration successfully executed" id="add index user_auth_token.user_id" duration=616.516µs 2024-09-21T12:21:26.329 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.108959684Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-21T12:21:26.329 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.112115958Z level=info msg="Migration successfully executed" id="Add revoked_at to the user auth token" duration=3.156043ms 2024-09-21T12:21:26.329 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.112459071Z level=info msg="Executing migration" id="add index user_auth_token.revoked_at" 2024-09-21T12:21:26.330 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.113048809Z level=info msg="Migration successfully executed" id="add index user_auth_token.revoked_at" duration=588.42µs 2024-09-21T12:21:26.330 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.113413736Z level=info msg="Executing migration" id="create cache_data table" 2024-09-21T12:21:26.330 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.11398192Z level=info msg="Migration successfully executed" id="create cache_data table" duration=567.996µs 2024-09-21T12:21:26.330 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.114355188Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-21T12:21:26.330 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.114923529Z level=info msg="Migration successfully executed" id="add unique index cache_data.cache_key" duration=568.031µs 2024-09-21T12:21:26.330 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.115257389Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-21T12:21:26.330 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.115811741Z level=info msg="Migration successfully executed" id="create short_url table v1" duration=554.041µs 2024-09-21T12:21:26.330 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.116227012Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-21T12:21:26.330 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.116872237Z level=info msg="Migration successfully executed" id="add index short_url.org_id-uid" duration=645.361µs 2024-09-21T12:21:26.330 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.117212498Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-21T12:21:26.330 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.117251977Z level=info msg="Migration successfully executed" id="alter table short_url alter column created_by type to bigint" duration=39.748µs 2024-09-21T12:21:26.330 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.117633343Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-21T12:21:26.330 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.117685894Z level=info msg="Migration successfully executed" id="delete alert_definition table" duration=52.698µs 2024-09-21T12:21:26.331 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.118075971Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-21T12:21:26.331 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.118621088Z level=info msg="Migration successfully executed" id="recreate alert_definition table" duration=544.767µs 2024-09-21T12:21:26.331 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.118961333Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-21T12:21:26.331 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.119569467Z level=info msg="Migration successfully executed" id="add index in alert_definition on org_id and title columns" duration=606.726µs 2024-09-21T12:21:26.331 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.119990963Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-21T12:21:26.331 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.120588898Z level=info msg="Migration successfully executed" id="add index in alert_definition on org_id and uid columns" duration=596.764µs 2024-09-21T12:21:26.331 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.120907304Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-21T12:21:26.331 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.120944429Z level=info msg="Migration successfully executed" id="alter alert_definition table data column to mediumtext in mysql" duration=37.479µs 2024-09-21T12:21:26.331 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.121324583Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-21T12:21:26.331 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.121888012Z level=info msg="Migration successfully executed" id="drop index in alert_definition on org_id and title columns" duration=563.16µs 2024-09-21T12:21:26.331 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.122212317Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-21T12:21:26.331 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.122768714Z level=info msg="Migration successfully executed" id="drop index in alert_definition on org_id and uid columns" duration=556.065µs 2024-09-21T12:21:26.331 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.123106612Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-21T12:21:26.331 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.123698129Z level=info msg="Migration successfully executed" id="add unique index in alert_definition on org_id and title columns" duration=591.129µs 2024-09-21T12:21:26.331 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.125955741Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-21T12:21:26.332 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.126560933Z level=info msg="Migration successfully executed" id="add unique index in alert_definition on org_id and uid columns" duration=605.971µs 2024-09-21T12:21:26.332 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.128623357Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-21T12:21:26.332 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.132067393Z level=info msg="Migration successfully executed" id="Add column paused in alert_definition" duration=3.445832ms 2024-09-21T12:21:26.332 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.132435679Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-21T12:21:26.332 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.133151739Z level=info msg="Migration successfully executed" id="drop alert_definition table" duration=715.906µs 2024-09-21T12:21:26.332 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.133511819Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-21T12:21:26.332 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.133569732Z level=info msg="Migration successfully executed" id="delete alert_definition_version table" duration=58.009µs 2024-09-21T12:21:26.332 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.136259594Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-21T12:21:26.332 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.136841148Z level=info msg="Migration successfully executed" id="recreate alert_definition_version table" duration=581.712µs 2024-09-21T12:21:26.332 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.137194368Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-21T12:21:26.332 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.137863479Z level=info msg="Migration successfully executed" id="add index in alert_definition_version table on alert_definition_id and version columns" duration=667.663µs 2024-09-21T12:21:26.332 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.140008712Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-21T12:21:26.332 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.140643424Z level=info msg="Migration successfully executed" id="add index in alert_definition_version table on alert_definition_uid and version columns" duration=634.42µs 2024-09-21T12:21:26.332 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.142596159Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-21T12:21:26.332 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.142637362Z level=info msg="Migration successfully executed" id="alter alert_definition_version table data column to mediumtext in mysql" duration=42.043µs 2024-09-21T12:21:26.332 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.145259714Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-21T12:21:26.333 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.145908204Z level=info msg="Migration successfully executed" id="drop alert_definition_version table" duration=648.909µs 2024-09-21T12:21:26.333 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.146276506Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-21T12:21:26.333 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.146952783Z level=info msg="Migration successfully executed" id="create alert_instance table" duration=676.462µs 2024-09-21T12:21:26.333 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.149261611Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-21T12:21:26.333 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.149929303Z level=info msg="Migration successfully executed" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" duration=667.853µs 2024-09-21T12:21:26.333 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.151997619Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-21T12:21:26.333 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.152611727Z level=info msg="Migration successfully executed" id="add index in alert_instance table on def_org_id, current_state columns" duration=612.992µs 2024-09-21T12:21:26.333 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.154707322Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-21T12:21:26.333 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.158035602Z level=info msg="Migration successfully executed" id="add column current_state_end to alert_instance" duration=3.329039ms 2024-09-21T12:21:26.333 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.158378972Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-21T12:21:26.333 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.158972406Z level=info msg="Migration successfully executed" id="remove index def_org_id, def_uid, current_state on alert_instance" duration=593.66µs 2024-09-21T12:21:26.333 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.15934287Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-21T12:21:26.333 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.159914827Z level=info msg="Migration successfully executed" id="remove index def_org_id, current_state on alert_instance" duration=571.774µs 2024-09-21T12:21:26.333 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.162041022Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-21T12:21:26.333 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.179257141Z level=info msg="Migration successfully executed" id="rename def_org_id to rule_org_id in alert_instance" duration=17.21585ms 2024-09-21T12:21:26.333 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.179633179Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-21T12:21:26.334 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.195556575Z level=info msg="Migration successfully executed" id="rename def_uid to rule_uid in alert_instance" duration=15.925238ms 2024-09-21T12:21:26.334 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.195933556Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-21T12:21:26.334 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.196549498Z level=info msg="Migration successfully executed" id="add index rule_org_id, rule_uid, current_state on alert_instance" duration=615.751µs 2024-09-21T12:21:26.334 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.198673661Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-21T12:21:26.334 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.199301425Z level=info msg="Migration successfully executed" id="add index rule_org_id, current_state on alert_instance" duration=627.583µs 2024-09-21T12:21:26.334 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.201232521Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-21T12:21:26.334 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.204530672Z level=info msg="Migration successfully executed" id="add current_reason column related to current_state" duration=3.299998ms 2024-09-21T12:21:26.334 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.204873045Z level=info msg="Executing migration" id="add result_fingerprint column to alert_instance" 2024-09-21T12:21:26.334 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.208118808Z level=info msg="Migration successfully executed" id="add result_fingerprint column to alert_instance" duration=3.245425ms 2024-09-21T12:21:26.334 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.208487817Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-21T12:21:26.334 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.209122916Z level=info msg="Migration successfully executed" id="create alert_rule table" duration=634.833µs 2024-09-21T12:21:26.334 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.211326806Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-21T12:21:26.334 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.211986724Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id and title columns" duration=660.03µs 2024-09-21T12:21:26.334 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.21407066Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-21T12:21:26.335 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.214694854Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id and uid columns" duration=625.78µs 2024-09-21T12:21:26.335 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.2167535Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-21T12:21:26.335 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.217443722Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" duration=690.022µs 2024-09-21T12:21:26.335 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.219200095Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-21T12:21:26.335 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.21925538Z level=info msg="Migration successfully executed" id="alter alert_rule table data column to mediumtext in mysql" duration=54.625µs 2024-09-21T12:21:26.335 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.221860359Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-21T12:21:26.335 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.22531438Z level=info msg="Migration successfully executed" id="add column for to alert_rule" duration=3.453682ms 2024-09-21T12:21:26.335 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.225674253Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-21T12:21:26.335 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.229014227Z level=info msg="Migration successfully executed" id="add column annotations to alert_rule" duration=3.339682ms 2024-09-21T12:21:26.335 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.229412352Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-21T12:21:26.335 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.232727714Z level=info msg="Migration successfully executed" id="add column labels to alert_rule" duration=3.313449ms 2024-09-21T12:21:26.335 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.233094808Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-21T12:21:26.335 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.233674203Z level=info msg="Migration successfully executed" id="remove unique index from alert_rule on org_id, title columns" duration=579.289µs 2024-09-21T12:21:26.335 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.235910897Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-21T12:21:26.335 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.236581305Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, namespase_uid and title columns" duration=672.018µs 2024-09-21T12:21:26.336 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.238591437Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-21T12:21:26.336 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.241849884Z level=info msg="Migration successfully executed" id="add dashboard_uid column to alert_rule" duration=3.260166ms 2024-09-21T12:21:26.336 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.242208248Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-21T12:21:26.336 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.245407032Z level=info msg="Migration successfully executed" id="add panel_id column to alert_rule" duration=3.198557ms 2024-09-21T12:21:26.336 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.245801653Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-21T12:21:26.337 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.246450924Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" duration=648.991µs 2024-09-21T12:21:26.337 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.248635121Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-21T12:21:26.337 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.251995592Z level=info msg="Migration successfully executed" id="add rule_group_idx column to alert_rule" duration=3.358517ms 2024-09-21T12:21:26.337 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.252344963Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-21T12:21:26.337 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.255662626Z level=info msg="Migration successfully executed" id="add is_paused column to alert_rule table" duration=3.317354ms 2024-09-21T12:21:26.337 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.256028777Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-21T12:21:26.337 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.256066079Z level=info msg="Migration successfully executed" id="fix is_paused column for alert_rule table" duration=37.588µs 2024-09-21T12:21:26.337 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.256861687Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-21T12:21:26.337 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.257583473Z level=info msg="Migration successfully executed" id="create alert_rule_version table" duration=721.366µs 2024-09-21T12:21:26.337 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.257933882Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-21T12:21:26.337 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.258563342Z level=info msg="Migration successfully executed" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" duration=629.545µs 2024-09-21T12:21:26.337 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.258922021Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-21T12:21:26.337 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.259584949Z level=info msg="Migration successfully executed" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" duration=662.765µs 2024-09-21T12:21:26.338 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.259948319Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-21T12:21:26.338 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.259985392Z level=info msg="Migration successfully executed" id="alter alert_rule_version table data column to mediumtext in mysql" duration=38.528µs 2024-09-21T12:21:26.338 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.260755133Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-21T12:21:26.338 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.264177389Z level=info msg="Migration successfully executed" id="add column for to alert_rule_version" duration=3.424106ms 2024-09-21T12:21:26.338 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.264532595Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-21T12:21:26.338 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.26794927Z level=info msg="Migration successfully executed" id="add column annotations to alert_rule_version" duration=3.417003ms 2024-09-21T12:21:26.338 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.268303021Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-21T12:21:26.338 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.271708304Z level=info msg="Migration successfully executed" id="add column labels to alert_rule_version" duration=3.405419ms 2024-09-21T12:21:26.338 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.272068578Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-21T12:21:26.338 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.275509403Z level=info msg="Migration successfully executed" id="add rule_group_idx column to alert_rule_version" duration=3.441109ms 2024-09-21T12:21:26.338 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.275892861Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-21T12:21:26.338 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.279419566Z level=info msg="Migration successfully executed" id="add is_paused column to alert_rule_versions table" duration=3.526982ms 2024-09-21T12:21:26.338 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.279779948Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-21T12:21:26.338 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.279840191Z level=info msg="Migration successfully executed" id="fix is_paused column for alert_rule_version table" duration=60.932µs 2024-09-21T12:21:26.338 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.280796357Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-21T12:21:26.339 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.281414012Z level=info msg="Migration successfully executed" id=create_alert_configuration_table duration=620.924µs 2024-09-21T12:21:26.339 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.281777578Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-21T12:21:26.339 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.285324576Z level=info msg="Migration successfully executed" id="Add column default in alert_configuration" duration=3.547061ms 2024-09-21T12:21:26.339 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.285705187Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-21T12:21:26.339 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.285747698Z level=info msg="Migration successfully executed" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" duration=43.336µs 2024-09-21T12:21:26.339 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.286470235Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-21T12:21:26.339 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.290207866Z level=info msg="Migration successfully executed" id="add column org_id in alert_configuration" duration=3.735629ms 2024-09-21T12:21:26.339 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.290561538Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-21T12:21:26.339 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.291174373Z level=info msg="Migration successfully executed" id="add index in alert_configuration table on org_id column" duration=613.21µs 2024-09-21T12:21:26.339 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.291520716Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-21T12:21:26.339 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.295052431Z level=info msg="Migration successfully executed" id="add configuration_hash column to alert_configuration" duration=3.531443ms 2024-09-21T12:21:26.339 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.295470173Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-21T12:21:26.339 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.29597415Z level=info msg="Migration successfully executed" id=create_ngalert_configuration_table duration=502.716µs 2024-09-21T12:21:26.340 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.296356811Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-21T12:21:26.340 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.296973499Z level=info msg="Migration successfully executed" id="add index in ngalert_configuration on org_id column" duration=616.425µs 2024-09-21T12:21:26.340 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.297387974Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-21T12:21:26.340 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.301101862Z level=info msg="Migration successfully executed" id="add column send_alerts_to in ngalert_configuration" duration=3.713925ms 2024-09-21T12:21:26.340 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.301474026Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-21T12:21:26.340 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.302007577Z level=info msg="Migration successfully executed" id="create provenance_type table" duration=533.42µs 2024-09-21T12:21:26.340 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.302408827Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-21T12:21:26.340 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.303035323Z level=info msg="Migration successfully executed" id="add index to uniquify (record_key, record_type, org_id) columns" duration=626.248µs 2024-09-21T12:21:26.340 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.303387432Z level=info msg="Executing migration" id="create alert_image table" 2024-09-21T12:21:26.340 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.303897328Z level=info msg="Migration successfully executed" id="create alert_image table" duration=509.562µs 2024-09-21T12:21:26.340 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.304239613Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-21T12:21:26.340 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.304833175Z level=info msg="Migration successfully executed" id="add unique index on token to alert_image table" duration=593.43µs 2024-09-21T12:21:26.340 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.30517287Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-21T12:21:26.340 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.305209406Z level=info msg="Migration successfully executed" id="support longer URLs in alert_image table" duration=36.905µs 2024-09-21T12:21:26.341 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.306010889Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-21T12:21:26.341 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.306605624Z level=info msg="Migration successfully executed" id=create_alert_configuration_history_table duration=594.555µs 2024-09-21T12:21:26.341 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.306959473Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-21T12:21:26.341 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.307545449Z level=info msg="Migration successfully executed" id="drop non-unique orgID index on alert_configuration" duration=584.384µs 2024-09-21T12:21:26.341 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.30787927Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-21T12:21:26.341 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.30809949Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-21T12:21:26.341 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.308769125Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-21T12:21:26.341 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.309053265Z level=info msg="Migration successfully executed" id="extract alertmanager configuration history to separate table" duration=284.135µs 2024-09-21T12:21:26.341 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.312372391Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-21T12:21:26.341 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.313006415Z level=info msg="Migration successfully executed" id="add unique index on orgID to alert_configuration" duration=638.176µs 2024-09-21T12:21:26.341 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.313340425Z level=info msg="Executing migration" id="add last_applied column to alert_configuration_history" 2024-09-21T12:21:26.341 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.31706462Z level=info msg="Migration successfully executed" id="add last_applied column to alert_configuration_history" duration=3.724498ms 2024-09-21T12:21:26.341 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.317458107Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-21T12:21:26.341 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.318153019Z level=info msg="Migration successfully executed" id="create library_element table v1" duration=694.66µs 2024-09-21T12:21:26.342 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.31857822Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-21T12:21:26.342 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.31923558Z level=info msg="Migration successfully executed" id="add index library_element org_id-folder_id-name-kind" duration=657.189µs 2024-09-21T12:21:26.342 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.319588429Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-21T12:21:26.342 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.32013586Z level=info msg="Migration successfully executed" id="create library_element_connection table v1" duration=547.398µs 2024-09-21T12:21:26.342 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.320495699Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-21T12:21:26.342 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.32116008Z level=info msg="Migration successfully executed" id="add index library_element_connection element_id-kind-connection_id" duration=664.181µs 2024-09-21T12:21:26.342 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.323515747Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-21T12:21:26.342 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.324267057Z level=info msg="Migration successfully executed" id="add unique index library_element org_id_uid" duration=753.045µs 2024-09-21T12:21:26.342 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.326127684Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-21T12:21:26.342 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.326142005Z level=info msg="Migration successfully executed" id="increase max description length to 2048" duration=14.912µs 2024-09-21T12:21:26.342 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.3289823Z level=info msg="Executing migration" id="alter library_element model to mediumtext" 2024-09-21T12:21:26.342 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.329037904Z level=info msg="Migration successfully executed" id="alter library_element model to mediumtext" duration=56.243µs 2024-09-21T12:21:26.372 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:26.394 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:26.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:26.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:26.496 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:26.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:26.518 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:26.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:26.581 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.331775546Z level=info msg="Executing migration" id="add library_element folder uid" 2024-09-21T12:21:26.581 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.335994073Z level=info msg="Migration successfully executed" id="add library_element folder uid" duration=4.218159ms 2024-09-21T12:21:26.581 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.336355916Z level=info msg="Executing migration" id="populate library_element folder_uid" 2024-09-21T12:21:26.582 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.336604737Z level=info msg="Migration successfully executed" id="populate library_element folder_uid" duration=248.734µs 2024-09-21T12:21:26.582 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.339095702Z level=info msg="Executing migration" id="add index library_element org_id-folder_uid-name-kind" 2024-09-21T12:21:26.582 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.339774255Z level=info msg="Migration successfully executed" id="add index library_element org_id-folder_uid-name-kind" duration=678.321µs 2024-09-21T12:21:26.582 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.340136314Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-21T12:21:26.582 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.340314354Z level=info msg="Migration successfully executed" id="clone move dashboard alerts to unified alerting" duration=177.965µs 2024-09-21T12:21:26.582 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.342876832Z level=info msg="Executing migration" id="create data_keys table" 2024-09-21T12:21:26.582 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.343528769Z level=info msg="Migration successfully executed" id="create data_keys table" duration=652.57µs 2024-09-21T12:21:26.582 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.345680039Z level=info msg="Executing migration" id="create secrets table" 2024-09-21T12:21:26.582 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.34629734Z level=info msg="Migration successfully executed" id="create secrets table" duration=618.091µs 2024-09-21T12:21:26.582 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.348568891Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-21T12:21:26.582 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.371926466Z level=info msg="Migration successfully executed" id="rename data_keys name column to id" duration=23.360585ms 2024-09-21T12:21:26.583 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.372298832Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-21T12:21:26.583 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.376427235Z level=info msg="Migration successfully executed" id="add name column into data_keys" duration=4.128456ms 2024-09-21T12:21:26.583 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.376786695Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-21T12:21:26.583 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.37689629Z level=info msg="Migration successfully executed" id="copy data_keys id column values into name" duration=110.157µs 2024-09-21T12:21:26.583 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.379554462Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-21T12:21:26.583 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.400268721Z level=info msg="Migration successfully executed" id="rename data_keys name column to label" duration=20.714376ms 2024-09-21T12:21:26.583 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.400642231Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-21T12:21:26.583 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.421866188Z level=info msg="Migration successfully executed" id="rename data_keys id column back to name" duration=21.225638ms 2024-09-21T12:21:26.583 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.422226566Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-21T12:21:26.583 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.422783443Z level=info msg="Migration successfully executed" id="create kv_store table v1" duration=557.125µs 2024-09-21T12:21:26.583 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.425077416Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-21T12:21:26.583 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.425789782Z level=info msg="Migration successfully executed" id="add index kv_store.org_id-namespace-key" duration=712.489µs 2024-09-21T12:21:26.583 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.427744666Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-21T12:21:26.584 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.427888761Z level=info msg="Migration successfully executed" id="update dashboard_uid and panel_id from existing annotations" duration=144.215µs 2024-09-21T12:21:26.584 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.430628323Z level=info msg="Executing migration" id="create permission table" 2024-09-21T12:21:26.584 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.431206188Z level=info msg="Migration successfully executed" id="create permission table" duration=577.569µs 2024-09-21T12:21:26.584 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.433235077Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-21T12:21:26.584 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.433871778Z level=info msg="Migration successfully executed" id="add unique index permission.role_id" duration=636.526µs 2024-09-21T12:21:26.584 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.435909739Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-21T12:21:26.584 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.436601783Z level=info msg="Migration successfully executed" id="add unique index role_id_action_scope" duration=690.235µs 2024-09-21T12:21:26.584 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.438304341Z level=info msg="Executing migration" id="create role table" 2024-09-21T12:21:26.585 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.438905505Z level=info msg="Migration successfully executed" id="create role table" duration=600.901µs 2024-09-21T12:21:26.585 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.440948869Z level=info msg="Executing migration" id="add column display_name" 2024-09-21T12:21:26.585 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.445200092Z level=info msg="Migration successfully executed" id="add column display_name" duration=4.25093ms 2024-09-21T12:21:26.585 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.445577996Z level=info msg="Executing migration" id="add column group_name" 2024-09-21T12:21:26.585 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.449881744Z level=info msg="Migration successfully executed" id="add column group_name" duration=4.303376ms 2024-09-21T12:21:26.585 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.450239519Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-21T12:21:26.586 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.450881864Z level=info msg="Migration successfully executed" id="add index role.org_id" duration=642.414µs 2024-09-21T12:21:26.586 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.453104613Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-21T12:21:26.586 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.453810384Z level=info msg="Migration successfully executed" id="add unique index role_org_id_name" duration=707.388µs 2024-09-21T12:21:26.586 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.455623111Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-21T12:21:26.586 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.456322151Z level=info msg="Migration successfully executed" id="add index role_org_id_uid" duration=698.833µs 2024-09-21T12:21:26.586 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.458393859Z level=info msg="Executing migration" id="create team role table" 2024-09-21T12:21:26.586 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.459154259Z level=info msg="Migration successfully executed" id="create team role table" duration=762.511µs 2024-09-21T12:21:26.586 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.460910702Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-21T12:21:26.586 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.461589793Z level=info msg="Migration successfully executed" id="add index team_role.org_id" duration=678.913µs 2024-09-21T12:21:26.586 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.461952413Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-21T12:21:26.586 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.462631616Z level=info msg="Migration successfully executed" id="add unique index team_role_org_id_team_id_role_id" duration=680.046µs 2024-09-21T12:21:26.586 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.46300364Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-21T12:21:26.586 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.463672218Z level=info msg="Migration successfully executed" id="add index team_role.team_id" duration=668.256µs 2024-09-21T12:21:26.587 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.464044269Z level=info msg="Executing migration" id="create user role table" 2024-09-21T12:21:26.587 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.464628074Z level=info msg="Migration successfully executed" id="create user role table" duration=583.552µs 2024-09-21T12:21:26.587 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.465003804Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-21T12:21:26.587 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.465672243Z level=info msg="Migration successfully executed" id="add index user_role.org_id" duration=668.306µs 2024-09-21T12:21:26.587 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.466048789Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-21T12:21:26.587 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.466732705Z level=info msg="Migration successfully executed" id="add unique index user_role_org_id_user_id_role_id" duration=685.36µs 2024-09-21T12:21:26.587 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.467108487Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-21T12:21:26.587 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.467778196Z level=info msg="Migration successfully executed" id="add index user_role.user_id" duration=669.401µs 2024-09-21T12:21:26.587 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.46816475Z level=info msg="Executing migration" id="create builtin role table" 2024-09-21T12:21:26.587 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.468740749Z level=info msg="Migration successfully executed" id="create builtin role table" duration=575.725µs 2024-09-21T12:21:26.587 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.4691182Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-21T12:21:26.587 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.469808544Z level=info msg="Migration successfully executed" id="add index builtin_role.role_id" duration=689.903µs 2024-09-21T12:21:26.587 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.470184395Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-21T12:21:26.587 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.470883502Z level=info msg="Migration successfully executed" id="add index builtin_role.name" duration=698.695µs 2024-09-21T12:21:26.587 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.471233873Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-21T12:21:26.588 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.475645749Z level=info msg="Migration successfully executed" id="Add column org_id to builtin_role table" duration=4.410549ms 2024-09-21T12:21:26.588 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.476016595Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-21T12:21:26.588 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.476655758Z level=info msg="Migration successfully executed" id="add index builtin_role.org_id" duration=638.99µs 2024-09-21T12:21:26.588 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.477019102Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-21T12:21:26.588 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.477677256Z level=info msg="Migration successfully executed" id="add unique index builtin_role_org_id_role_id_role" duration=657.886µs 2024-09-21T12:21:26.588 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.478016053Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-21T12:21:26.588 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.478660814Z level=info msg="Migration successfully executed" id="Remove unique index role_org_id_uid" duration=645.07µs 2024-09-21T12:21:26.588 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.478998899Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-21T12:21:26.588 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.479638948Z level=info msg="Migration successfully executed" id="add unique index role.uid" duration=639.827µs 2024-09-21T12:21:26.588 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.479965152Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-21T12:21:26.588 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.480502856Z level=info msg="Migration successfully executed" id="create seed assignment table" duration=537.425µs 2024-09-21T12:21:26.589 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.480937024Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-21T12:21:26.589 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.482024728Z level=info msg="Migration successfully executed" id="add unique index builtin_role_role_name" duration=1.086857ms 2024-09-21T12:21:26.589 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.48303971Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-21T12:21:26.589 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.48803489Z level=info msg="Migration successfully executed" id="add column hidden to role table" duration=4.995395ms 2024-09-21T12:21:26.589 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.488394333Z level=info msg="Executing migration" id="permission kind migration" 2024-09-21T12:21:26.589 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.493097328Z level=info msg="Migration successfully executed" id="permission kind migration" duration=4.702629ms 2024-09-21T12:21:26.589 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.493463241Z level=info msg="Executing migration" id="permission attribute migration" 2024-09-21T12:21:26.589 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.498034273Z level=info msg="Migration successfully executed" id="permission attribute migration" duration=4.570735ms 2024-09-21T12:21:26.589 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.4983678Z level=info msg="Executing migration" id="permission identifier migration" 2024-09-21T12:21:26.589 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.502889509Z level=info msg="Migration successfully executed" id="permission identifier migration" duration=4.521344ms 2024-09-21T12:21:26.589 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.503236089Z level=info msg="Executing migration" id="add permission identifier index" 2024-09-21T12:21:26.589 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.503925328Z level=info msg="Migration successfully executed" id="add permission identifier index" duration=688.973µs 2024-09-21T12:21:26.589 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.504264563Z level=info msg="Executing migration" id="add permission action scope role_id index" 2024-09-21T12:21:26.589 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.504919605Z level=info msg="Migration successfully executed" id="add permission action scope role_id index" duration=654.53µs 2024-09-21T12:21:26.589 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.505248505Z level=info msg="Executing migration" id="remove permission role_id action scope index" 2024-09-21T12:21:26.590 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.505885017Z level=info msg="Migration successfully executed" id="remove permission role_id action scope index" duration=636.205µs 2024-09-21T12:21:26.590 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.506213279Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-21T12:21:26.590 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.506774343Z level=info msg="Migration successfully executed" id="create query_history table v1" duration=560.889µs 2024-09-21T12:21:26.590 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.507136403Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-21T12:21:26.590 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.50777743Z level=info msg="Migration successfully executed" id="add index query_history.org_id-created_by-datasource_uid" duration=640.725µs 2024-09-21T12:21:26.590 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.508114066Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-21T12:21:26.590 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.508151249Z level=info msg="Migration successfully executed" id="alter table query_history alter column created_by type to bigint" duration=37.53µs 2024-09-21T12:21:26.590 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.508917032Z level=info msg="Executing migration" id="rbac disabled migrator" 2024-09-21T12:21:26.590 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.508936538Z level=info msg="Migration successfully executed" id="rbac disabled migrator" duration=20.145µs 2024-09-21T12:21:26.590 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.509646203Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-21T12:21:26.590 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.509898543Z level=info msg="Migration successfully executed" id="teams permissions migration" duration=252.309µs 2024-09-21T12:21:26.590 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.510493806Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-21T12:21:26.590 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.510810192Z level=info msg="Migration successfully executed" id="dashboard permissions" duration=316.888µs 2024-09-21T12:21:26.591 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.511202131Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-21T12:21:26.591 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.511590797Z level=info msg="Migration successfully executed" id="dashboard permissions uid scopes" duration=389.5µs 2024-09-21T12:21:26.591 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.511966292Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-21T12:21:26.591 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.512092513Z level=info msg="Migration successfully executed" id="drop managed folder create actions" duration=126.204µs 2024-09-21T12:21:26.591 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.512770616Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-21T12:21:26.591 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.513054851Z level=info msg="Migration successfully executed" id="alerting notification permissions" duration=284.215µs 2024-09-21T12:21:26.591 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.513685255Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-21T12:21:26.591 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.514367358Z level=info msg="Migration successfully executed" id="create query_history_star table v1" duration=684.143µs 2024-09-21T12:21:26.591 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.514711938Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-21T12:21:26.591 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.515433089Z level=info msg="Migration successfully executed" id="add index query_history.user_id-query_uid" duration=720.843µs 2024-09-21T12:21:26.591 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.515790989Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-21T12:21:26.591 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.52043574Z level=info msg="Migration successfully executed" id="add column org_id in query_history_star" duration=4.64447ms 2024-09-21T12:21:26.591 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.520783183Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-21T12:21:26.591 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.520829515Z level=info msg="Migration successfully executed" id="alter table query_history_star_mig column user_id type to bigint" duration=45.339µs 2024-09-21T12:21:26.592 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.523414139Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-21T12:21:26.592 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.524044168Z level=info msg="Migration successfully executed" id="create correlation table v1" duration=629.614µs 2024-09-21T12:21:26.592 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.526062002Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-21T12:21:26.592 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.526751924Z level=info msg="Migration successfully executed" id="add index correlations.uid" duration=689.725µs 2024-09-21T12:21:26.592 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.528908985Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-21T12:21:26.592 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.529654098Z level=info msg="Migration successfully executed" id="add index correlations.source_uid" duration=747.034µs 2024-09-21T12:21:26.592 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.531211284Z level=info msg="Executing migration" id="add correlation config column" 2024-09-21T12:21:26.592 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.535939184Z level=info msg="Migration successfully executed" id="add correlation config column" duration=4.727634ms 2024-09-21T12:21:26.592 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.536268578Z level=info msg="Executing migration" id="drop index IDX_correlation_uid - v1" 2024-09-21T12:21:26.592 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.536899927Z level=info msg="Migration successfully executed" id="drop index IDX_correlation_uid - v1" duration=631.177µs 2024-09-21T12:21:26.592 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.539155898Z level=info msg="Executing migration" id="drop index IDX_correlation_source_uid - v1" 2024-09-21T12:21:26.592 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.539902762Z level=info msg="Migration successfully executed" id="drop index IDX_correlation_source_uid - v1" duration=748.837µs 2024-09-21T12:21:26.592 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.541618296Z level=info msg="Executing migration" id="Rename table correlation to correlation_tmp_qwerty - v1" 2024-09-21T12:21:26.592 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.558897778Z level=info msg="Migration successfully executed" id="Rename table correlation to correlation_tmp_qwerty - v1" duration=17.278452ms 2024-09-21T12:21:26.592 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.559277546Z level=info msg="Executing migration" id="create correlation v2" 2024-09-21T12:21:26.593 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.560132624Z level=info msg="Migration successfully executed" id="create correlation v2" duration=854.567µs 2024-09-21T12:21:26.593 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.561960179Z level=info msg="Executing migration" id="create index IDX_correlation_uid - v2" 2024-09-21T12:21:26.593 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.562631144Z level=info msg="Migration successfully executed" id="create index IDX_correlation_uid - v2" duration=671.701µs 2024-09-21T12:21:26.593 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.564653718Z level=info msg="Executing migration" id="create index IDX_correlation_source_uid - v2" 2024-09-21T12:21:26.593 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.565379357Z level=info msg="Migration successfully executed" id="create index IDX_correlation_source_uid - v2" duration=725.274µs 2024-09-21T12:21:26.593 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.567357103Z level=info msg="Executing migration" id="create index IDX_correlation_org_id - v2" 2024-09-21T12:21:26.593 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.568039916Z level=info msg="Migration successfully executed" id="create index IDX_correlation_org_id - v2" duration=682.581µs 2024-09-21T12:21:26.593 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.569987936Z level=info msg="Executing migration" id="copy correlation v1 to v2" 2024-09-21T12:21:26.593 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.570141813Z level=info msg="Migration successfully executed" id="copy correlation v1 to v2" duration=154.791µs 2024-09-21T12:21:26.593 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.572813771Z level=info msg="Executing migration" id="drop correlation_tmp_qwerty" 2024-09-21T12:21:26.593 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.573371619Z level=info msg="Migration successfully executed" id="drop correlation_tmp_qwerty" duration=556.431µs 2024-09-21T12:21:26.593 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.5753868Z level=info msg="Executing migration" id="add provisioning column" 2024-09-21T12:21:26.593 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.580140683Z level=info msg="Migration successfully executed" id="add provisioning column" duration=4.753695ms 2024-09-21T12:21:26.593 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.580530119Z level=info msg="Executing migration" id="create entity_events table" 2024-09-21T12:21:26.594 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.581097556Z level=info msg="Migration successfully executed" id="create entity_events table" duration=567.316µs 2024-09-21T12:21:26.594 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.583166297Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-21T12:21:26.619 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:26.643 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:26.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:26.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:26.654 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:21:26.654 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:21:26.745 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:26.767 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:26.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:26.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:26.833 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:21:26.834 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.583915582Z level=info msg="Migration successfully executed" id="create dashboard public config v1" duration=749.067µs 2024-09-21T12:21:26.834 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.585800164Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-21T12:21:26.834 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.586089285Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-21T12:21:26.834 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.588211124Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-21T12:21:26.834 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.588502881Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-21T12:21:26.834 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.590654117Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-21T12:21:26.835 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.591218747Z level=info msg="Migration successfully executed" id="Drop old dashboard public config table" duration=564.554µs 2024-09-21T12:21:26.835 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.593189374Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-21T12:21:26.835 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.593855247Z level=info msg="Migration successfully executed" id="recreate dashboard public config v1" duration=665.511µs 2024-09-21T12:21:26.835 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.595653922Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-21T12:21:26.835 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.596463381Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_uid - v1" duration=811.545µs 2024-09-21T12:21:26.835 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.597916271Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-21T12:21:26.835 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.598586384Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" duration=670.062µs 2024-09-21T12:21:26.835 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.600357776Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-21T12:21:26.835 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.60103932Z level=info msg="Migration successfully executed" id="drop index UQE_dashboard_public_config_uid - v2" duration=681.324µs 2024-09-21T12:21:26.835 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.603037376Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-21T12:21:26.835 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.603700151Z level=info msg="Migration successfully executed" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" duration=663.203µs 2024-09-21T12:21:26.835 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.605519112Z level=info msg="Executing migration" id="Drop public config table" 2024-09-21T12:21:26.835 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.606103725Z level=info msg="Migration successfully executed" id="Drop public config table" duration=584.852µs 2024-09-21T12:21:26.835 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.607953369Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-21T12:21:26.836 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.608666992Z level=info msg="Migration successfully executed" id="Recreate dashboard public config v2" duration=714.041µs 2024-09-21T12:21:26.836 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.610395197Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-21T12:21:26.836 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.611117736Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_uid - v2" duration=722.888µs 2024-09-21T12:21:26.836 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.612840159Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-21T12:21:26.836 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.613558903Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" duration=718.623µs 2024-09-21T12:21:26.836 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.615527588Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-21T12:21:26.836 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.616217536Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_access_token - v2" duration=689.884µs 2024-09-21T12:21:26.836 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.617919327Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-21T12:21:26.836 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.632448846Z level=info msg="Migration successfully executed" id="Rename table dashboard_public_config to dashboard_public - v2" duration=14.529326ms 2024-09-21T12:21:26.836 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.632792948Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-21T12:21:26.836 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.637994757Z level=info msg="Migration successfully executed" id="add annotations_enabled column" duration=5.201686ms 2024-09-21T12:21:26.836 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.638387137Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-21T12:21:26.836 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.643383926Z level=info msg="Migration successfully executed" id="add time_selection_enabled column" duration=4.996883ms 2024-09-21T12:21:26.837 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.643740066Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-21T12:21:26.838 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.64390663Z level=info msg="Migration successfully executed" id="delete orphaned public dashboards" duration=167.028µs 2024-09-21T12:21:26.838 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.646489627Z level=info msg="Executing migration" id="add share column" 2024-09-21T12:21:26.838 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.651697361Z level=info msg="Migration successfully executed" id="add share column" duration=5.207398ms 2024-09-21T12:21:26.838 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.65206854Z level=info msg="Executing migration" id="backfill empty share column fields with default of public" 2024-09-21T12:21:26.838 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.652199967Z level=info msg="Migration successfully executed" id="backfill empty share column fields with default of public" duration=131.342µs 2024-09-21T12:21:26.838 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.652811601Z level=info msg="Executing migration" id="create file table" 2024-09-21T12:21:26.838 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.653464855Z level=info msg="Migration successfully executed" id="create file table" duration=653.493µs 2024-09-21T12:21:26.838 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.65383283Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-21T12:21:26.838 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.654521113Z level=info msg="Migration successfully executed" id="file table idx: path natural pk" duration=688.591µs 2024-09-21T12:21:26.838 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.654883675Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-21T12:21:26.838 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.655603468Z level=info msg="Migration successfully executed" id="file table idx: parent_folder_path_hash fast folder retrieval" duration=719.984µs 2024-09-21T12:21:26.838 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.655966098Z level=info msg="Executing migration" id="create file_meta table" 2024-09-21T12:21:26.839 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.656550309Z level=info msg="Migration successfully executed" id="create file_meta table" duration=584.408µs 2024-09-21T12:21:26.839 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.656935624Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-21T12:21:26.839 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.657674784Z level=info msg="Migration successfully executed" id="file table idx: path key" duration=738.985µs 2024-09-21T12:21:26.839 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.65813745Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-21T12:21:26.839 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.658214879Z level=info msg="Migration successfully executed" id="set path collation in file table" duration=77.591µs 2024-09-21T12:21:26.839 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.658863733Z level=info msg="Executing migration" id="migrate contents column to mediumblob for MySQL" 2024-09-21T12:21:26.839 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.65890407Z level=info msg="Migration successfully executed" id="migrate contents column to mediumblob for MySQL" duration=41.071µs 2024-09-21T12:21:26.839 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.659601084Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-21T12:21:26.839 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.659947529Z level=info msg="Migration successfully executed" id="managed permissions migration" duration=346.107µs 2024-09-21T12:21:26.839 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.66034155Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-21T12:21:26.839 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.660473116Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions migration" duration=131.542µs 2024-09-21T12:21:26.839 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.661101323Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-21T12:21:26.839 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.661947309Z level=info msg="Migration successfully executed" id="RBAC action name migrator" duration=846.092µs 2024-09-21T12:21:26.840 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.662371608Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-21T12:21:26.840 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.667546309Z level=info msg="Migration successfully executed" id="Add UID column to playlist" duration=5.176214ms 2024-09-21T12:21:26.840 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.667937696Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-21T12:21:26.840 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.668043029Z level=info msg="Migration successfully executed" id="Update uid column values in playlist" duration=106.638µs 2024-09-21T12:21:26.840 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.668702261Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-21T12:21:26.840 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.669393858Z level=info msg="Migration successfully executed" id="Add index for uid in playlist" duration=691.28µs 2024-09-21T12:21:26.840 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.669752247Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-21T12:21:26.840 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.66998667Z level=info msg="Migration successfully executed" id="update group index for alert rules" duration=234.94µs 2024-09-21T12:21:26.840 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.670464321Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-21T12:21:26.840 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.670599455Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions repeated migration" duration=134.849µs 2024-09-21T12:21:26.840 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.671208671Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-21T12:21:26.840 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.671503033Z level=info msg="Migration successfully executed" id="admin only folder/dashboard permission" duration=293.444µs 2024-09-21T12:21:26.840 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.671904693Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-21T12:21:26.840 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.676800165Z level=info msg="Migration successfully executed" id="add action column to seed_assignment" duration=4.895111ms 2024-09-21T12:21:26.841 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.677174987Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-21T12:21:26.841 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.682655997Z level=info msg="Migration successfully executed" id="add scope column to seed_assignment" duration=5.480765ms 2024-09-21T12:21:26.841 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.683012635Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-21T12:21:26.841 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.683668868Z level=info msg="Migration successfully executed" id="remove unique index builtin_role_role_name before nullable update" duration=656.043µs 2024-09-21T12:21:26.841 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.684014914Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-21T12:21:26.841 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.739494158Z level=info msg="Migration successfully executed" id="update seed_assignment role_name column to nullable" duration=55.478744ms 2024-09-21T12:21:26.841 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.739880727Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-21T12:21:26.841 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.74056268Z level=info msg="Migration successfully executed" id="add unique index builtin_role_name back" duration=681.852µs 2024-09-21T12:21:26.841 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.740920452Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-21T12:21:26.841 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.741606132Z level=info msg="Migration successfully executed" id="add unique index builtin_role_action_scope" duration=685.415µs 2024-09-21T12:21:26.841 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.741946213Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-21T12:21:26.841 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.758870634Z level=info msg="Migration successfully executed" id="add primary key to seed_assigment" duration=16.924119ms 2024-09-21T12:21:26.841 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.759247878Z level=info msg="Executing migration" id="add origin column to seed_assignment" 2024-09-21T12:21:26.841 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.764409267Z level=info msg="Migration successfully executed" id="add origin column to seed_assignment" duration=5.161143ms 2024-09-21T12:21:26.842 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.764777137Z level=info msg="Executing migration" id="add origin to plugin seed_assignment" 2024-09-21T12:21:26.842 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.764979646Z level=info msg="Migration successfully executed" id="add origin to plugin seed_assignment" duration=202.564µs 2024-09-21T12:21:26.842 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.765398365Z level=info msg="Executing migration" id="prevent seeding OnCall access" 2024-09-21T12:21:26.842 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.765510498Z level=info msg="Migration successfully executed" id="prevent seeding OnCall access" duration=112.709µs 2024-09-21T12:21:26.842 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.766149794Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-21T12:21:26.842 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.766291598Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions repeated fixed migration" duration=142.777µs 2024-09-21T12:21:26.842 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.766950285Z level=info msg="Executing migration" id="managed folder permissions library panel actions migration" 2024-09-21T12:21:26.842 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.767085811Z level=info msg="Migration successfully executed" id="managed folder permissions library panel actions migration" duration=134.907µs 2024-09-21T12:21:26.842 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.767741537Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-21T12:21:26.842 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.767881549Z level=info msg="Migration successfully executed" id="migrate external alertmanagers to datsourcse" duration=140.055µs 2024-09-21T12:21:26.842 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.768273913Z level=info msg="Executing migration" id="create folder table" 2024-09-21T12:21:26.842 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.768991898Z level=info msg="Migration successfully executed" id="create folder table" duration=717.505µs 2024-09-21T12:21:26.842 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.769381283Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-21T12:21:26.842 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.77010871Z level=info msg="Migration successfully executed" id="Add index for parent_uid" duration=727.129µs 2024-09-21T12:21:26.843 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.770470665Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-21T12:21:26.843 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.771185902Z level=info msg="Migration successfully executed" id="Add unique index for folder.uid and folder.org_id" duration=715.035µs 2024-09-21T12:21:26.843 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.773161039Z level=info msg="Executing migration" id="Update folder title length" 2024-09-21T12:21:26.843 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.773179541Z level=info msg="Migration successfully executed" id="Update folder title length" duration=19.627µs 2024-09-21T12:21:26.843 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.781290934Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-21T12:21:26.843 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.782191355Z level=info msg="Migration successfully executed" id="Add unique index for folder.title and folder.parent_uid" duration=900.409µs 2024-09-21T12:21:26.843 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.78253097Z level=info msg="Executing migration" id="Remove unique index for folder.title and folder.parent_uid" 2024-09-21T12:21:26.843 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.783256821Z level=info msg="Migration successfully executed" id="Remove unique index for folder.title and folder.parent_uid" duration=726.38µs 2024-09-21T12:21:26.843 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.7836191Z level=info msg="Executing migration" id="Add unique index for title, parent_uid, and org_id" 2024-09-21T12:21:26.843 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.784329434Z level=info msg="Migration successfully executed" id="Add unique index for title, parent_uid, and org_id" duration=710.094µs 2024-09-21T12:21:26.843 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.784711504Z level=info msg="Executing migration" id="Sync dashboard and folder table" 2024-09-21T12:21:26.843 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.785024229Z level=info msg="Migration successfully executed" id="Sync dashboard and folder table" duration=314.742µs 2024-09-21T12:21:26.843 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.785411042Z level=info msg="Executing migration" id="Remove ghost folders from the folder table" 2024-09-21T12:21:26.844 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.785587907Z level=info msg="Migration successfully executed" id="Remove ghost folders from the folder table" duration=177.871µs 2024-09-21T12:21:26.844 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.786243399Z level=info msg="Executing migration" id="Remove unique index UQE_folder_uid_org_id" 2024-09-21T12:21:26.844 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.786977711Z level=info msg="Migration successfully executed" id="Remove unique index UQE_folder_uid_org_id" duration=734.093µs 2024-09-21T12:21:26.844 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.787370879Z level=info msg="Executing migration" id="Add unique index UQE_folder_org_id_uid" 2024-09-21T12:21:26.844 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.788089509Z level=info msg="Migration successfully executed" id="Add unique index UQE_folder_org_id_uid" duration=718.953µs 2024-09-21T12:21:26.844 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.788461736Z level=info msg="Executing migration" id="Remove unique index UQE_folder_title_parent_uid_org_id" 2024-09-21T12:21:26.844 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.789168161Z level=info msg="Migration successfully executed" id="Remove unique index UQE_folder_title_parent_uid_org_id" duration=706.052µs 2024-09-21T12:21:26.844 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.789535145Z level=info msg="Executing migration" id="Add unique index UQE_folder_org_id_parent_uid_title" 2024-09-21T12:21:26.844 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.790235774Z level=info msg="Migration successfully executed" id="Add unique index UQE_folder_org_id_parent_uid_title" duration=700.802µs 2024-09-21T12:21:26.844 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.790574046Z level=info msg="Executing migration" id="Remove index IDX_folder_parent_uid_org_id" 2024-09-21T12:21:26.844 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.791253121Z level=info msg="Migration successfully executed" id="Remove index IDX_folder_parent_uid_org_id" duration=679.723µs 2024-09-21T12:21:26.844 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.791593802Z level=info msg="Executing migration" id="create anon_device table" 2024-09-21T12:21:26.844 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.792188668Z level=info msg="Migration successfully executed" id="create anon_device table" duration=594.689µs 2024-09-21T12:21:26.844 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.792548065Z level=info msg="Executing migration" id="add unique index anon_device.device_id" 2024-09-21T12:21:26.844 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.793304912Z level=info msg="Migration successfully executed" id="add unique index anon_device.device_id" duration=756.72µs 2024-09-21T12:21:26.845 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.79372382Z level=info msg="Executing migration" id="add index anon_device.updated_at" 2024-09-21T12:21:26.845 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.794461263Z level=info msg="Migration successfully executed" id="add index anon_device.updated_at" duration=737.349µs 2024-09-21T12:21:26.845 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.794853969Z level=info msg="Executing migration" id="create signing_key table" 2024-09-21T12:21:26.845 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.795405676Z level=info msg="Migration successfully executed" id="create signing_key table" duration=552.011µs 2024-09-21T12:21:26.845 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.795779611Z level=info msg="Executing migration" id="add unique index signing_key.key_id" 2024-09-21T12:21:26.845 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.796476994Z level=info msg="Migration successfully executed" id="add unique index signing_key.key_id" duration=697.236µs 2024-09-21T12:21:26.845 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.796848264Z level=info msg="Executing migration" id="set legacy alert migration status in kvstore" 2024-09-21T12:21:26.845 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.797563604Z level=info msg="Migration successfully executed" id="set legacy alert migration status in kvstore" duration=716.018µs 2024-09-21T12:21:26.845 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.797934676Z level=info msg="Executing migration" id="migrate record of created folders during legacy migration to kvstore" 2024-09-21T12:21:26.845 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.798118184Z level=info msg="Migration successfully executed" id="migrate record of created folders during legacy migration to kvstore" duration=184.738µs 2024-09-21T12:21:26.845 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.800611925Z level=info msg="Executing migration" id="Add folder_uid for dashboard" 2024-09-21T12:21:26.845 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.806137763Z level=info msg="Migration successfully executed" id="Add folder_uid for dashboard" duration=5.525174ms 2024-09-21T12:21:26.845 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.806516852Z level=info msg="Executing migration" id="Populate dashboard folder_uid column" 2024-09-21T12:21:26.845 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.806985511Z level=info msg="Migration successfully executed" id="Populate dashboard folder_uid column" duration=469.612µs 2024-09-21T12:21:26.846 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.809114685Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_folder_uid_title" 2024-09-21T12:21:26.846 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.809144772Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_folder_uid_title" duration=14.042µs 2024-09-21T12:21:26.846 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.80957672Z level=info msg="Executing migration" id="Delete unique index for dashboard_org_id_folder_id_title" 2024-09-21T12:21:26.846 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.810270115Z level=info msg="Migration successfully executed" id="Delete unique index for dashboard_org_id_folder_id_title" duration=695.699µs 2024-09-21T12:21:26.846 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.812280431Z level=info msg="Executing migration" id="Delete unique index for dashboard_org_id_folder_uid_title" 2024-09-21T12:21:26.846 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.812306521Z level=info msg="Migration successfully executed" id="Delete unique index for dashboard_org_id_folder_uid_title" duration=26.682µs 2024-09-21T12:21:26.846 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.814907772Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_folder_uid_title_is_folder" 2024-09-21T12:21:26.846 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.815637876Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_folder_uid_title_is_folder" duration=730.258µs 2024-09-21T12:21:26.846 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.815990527Z level=info msg="Executing migration" id="Restore index for dashboard_org_id_folder_id_title" 2024-09-21T12:21:26.846 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.816716548Z level=info msg="Migration successfully executed" id="Restore index for dashboard_org_id_folder_id_title" duration=726.314µs 2024-09-21T12:21:26.846 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.817092919Z level=info msg="Executing migration" id="create sso_setting table" 2024-09-21T12:21:26.846 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.817749907Z level=info msg="Migration successfully executed" id="create sso_setting table" duration=657.148µs 2024-09-21T12:21:26.846 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.819838773Z level=info msg="Executing migration" id="copy kvstore migration status to each org" 2024-09-21T12:21:26.847 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.820469304Z level=info msg="Migration successfully executed" id="copy kvstore migration status to each org" duration=634.449µs 2024-09-21T12:21:26.847 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.822361758Z level=info msg="Executing migration" id="add back entry for orgid=0 migrated status" 2024-09-21T12:21:26.847 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.822544698Z level=info msg="Migration successfully executed" id="add back entry for orgid=0 migrated status" duration=183.851µs 2024-09-21T12:21:26.847 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.824947951Z level=info msg="Executing migration" id="alter kv_store.value to longtext" 2024-09-21T12:21:26.847 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.824990309Z level=info msg="Migration successfully executed" id="alter kv_store.value to longtext" duration=42.632µs 2024-09-21T12:21:26.847 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.827860514Z level=info msg="Executing migration" id="add notification_settings column to alert_rule table" 2024-09-21T12:21:26.847 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.833071829Z level=info msg="Migration successfully executed" id="add notification_settings column to alert_rule table" duration=5.21493ms 2024-09-21T12:21:26.847 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.83351877Z level=info msg="Executing migration" id="add notification_settings column to alert_rule_version table" 2024-09-21T12:21:26.868 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:26.890 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:26.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:26.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:26.992 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:27.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:27.016 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:27.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:27.116 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:21:27.131 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-mon[33277]: pgmap v33: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:21:27.131 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-mon[33277]: Deploying daemon node-exporter.a on smithi114 2024-09-21T12:21:27.131 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-mon[33277]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T12:21:27.131 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-mon[33277]: osdmap e86: 8 total, 8 up, 8 in 2024-09-21T12:21:27.132 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.843176775Z level=info msg="Migration successfully executed" id="add notification_settings column to alert_rule_version table" duration=9.657691ms 2024-09-21T12:21:27.132 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.843621948Z level=info msg="Executing migration" id="removing scope from alert.instances:read action migration" 2024-09-21T12:21:27.132 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.84392Z level=info msg="Migration successfully executed" id="removing scope from alert.instances:read action migration" duration=297.123µs 2024-09-21T12:21:27.132 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=migrator t=2024-09-21T12:21:26.846204741Z level=info msg="migrations completed" performed=551 skipped=0 duration=1.270712904s 2024-09-21T12:21:27.132 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=sqlstore t=2024-09-21T12:21:26.847358765Z level=info msg="Created default organization" 2024-09-21T12:21:27.132 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=secrets t=2024-09-21T12:21:26.849421488Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-21T12:21:27.133 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=plugin.store t=2024-09-21T12:21:26.863489523Z level=info msg="Loading plugins..." 2024-09-21T12:21:27.133 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=local.finder t=2024-09-21T12:21:26.910912033Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-09-21T12:21:27.133 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=plugin.store t=2024-09-21T12:21:26.910930678Z level=info msg="Plugins loaded" count=55 duration=47.441799ms 2024-09-21T12:21:27.133 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=query_data t=2024-09-21T12:21:26.912894729Z level=info msg="Query Service initialization" 2024-09-21T12:21:27.133 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=live.push_http t=2024-09-21T12:21:26.915167711Z level=info msg="Live Push Gateway initialization" 2024-09-21T12:21:27.133 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=ngalert.migration t=2024-09-21T12:21:26.916156472Z level=info msg=Starting 2024-09-21T12:21:27.133 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=ngalert.migration t=2024-09-21T12:21:26.916422326Z level=info msg="Applying transition" currentType=Legacy desiredType=UnifiedAlerting cleanOnDowngrade=false cleanOnUpgrade=false 2024-09-21T12:21:27.133 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=ngalert.migration orgID=1 t=2024-09-21T12:21:26.916681283Z level=info msg="Migrating alerts for organisation" 2024-09-21T12:21:27.133 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=ngalert.migration orgID=1 t=2024-09-21T12:21:26.917159949Z level=info msg="Alerts found to migrate" alerts=0 2024-09-21T12:21:27.133 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=ngalert.migration t=2024-09-21T12:21:26.918260981Z level=info msg="Completed alerting migration" 2024-09-21T12:21:27.133 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=ngalert.state.manager t=2024-09-21T12:21:26.92850533Z level=info msg="Running in alternative execution of Error/NoData mode" 2024-09-21T12:21:27.133 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=infra.usagestats.collector t=2024-09-21T12:21:26.92974617Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-21T12:21:27.133 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=provisioning.datasources t=2024-09-21T12:21:26.931322459Z level=info msg="inserting datasource from configuration" name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-21T12:21:27.133 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=provisioning.datasources t=2024-09-21T12:21:26.943395545Z level=info msg="inserting datasource from configuration" name=Loki uid=P8E80F9AEF21F6940 2024-09-21T12:21:27.134 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=provisioning.alerting t=2024-09-21T12:21:26.957903644Z level=info msg="starting to provision alerting" 2024-09-21T12:21:27.134 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=provisioning.alerting t=2024-09-21T12:21:26.957922129Z level=info msg="finished to provision alerting" 2024-09-21T12:21:27.134 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=grafanaStorageLogger t=2024-09-21T12:21:26.958700869Z level=info msg="Storage starting" 2024-09-21T12:21:27.134 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=ngalert.state.manager t=2024-09-21T12:21:26.958866445Z level=info msg="Warming state cache for startup" 2024-09-21T12:21:27.134 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=ngalert.multiorg.alertmanager t=2024-09-21T12:21:26.959035166Z level=info msg="Starting MultiOrg Alertmanager" 2024-09-21T12:21:27.134 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=http.server t=2024-09-21T12:21:26.960459079Z level=info msg="HTTP Server TLS settings" MinTLSVersion=TLS1.2 configuredciphers=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA 2024-09-21T12:21:27.134 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=http.server t=2024-09-21T12:21:26.960710851Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-21T12:21:27.134 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=ngalert.state.manager t=2024-09-21T12:21:26.978023036Z level=info msg="State cache has been initialized" states=0 duration=19.155319ms 2024-09-21T12:21:27.134 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=ngalert.scheduler t=2024-09-21T12:21:26.97804839Z level=info msg="Starting scheduler" tickInterval=10s maxAttempts=1 2024-09-21T12:21:27.134 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=ticker t=2024-09-21T12:21:26.97809034Z level=info msg=starting first_tick=2024-09-21T12:21:30Z 2024-09-21T12:21:27.134 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=provisioning.dashboard t=2024-09-21T12:21:26.978596057Z level=info msg="starting to provision dashboards" 2024-09-21T12:21:27.134 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:27 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=sqlstore.transactions t=2024-09-21T12:21:27.009194827Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T12:21:27.134 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:27 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=grafana.update.checker t=2024-09-21T12:21:27.031306265Z level=info msg="Update check succeeded" duration=73.266178ms 2024-09-21T12:21:27.135 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:27 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=plugins.update.checker t=2024-09-21T12:21:27.036540397Z level=info msg="Update check succeeded" duration=78.450781ms 2024-09-21T12:21:27.135 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:27 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=provisioning.dashboard t=2024-09-21T12:21:27.130859919Z level=info msg="finished to provision dashboards" 2024-09-21T12:21:27.137 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:27.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:27.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:21:27.239 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:21:27.260 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:27.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:27.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:21:27.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:26 smithi114 ceph-mon[30324]: pgmap v33: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:21:27.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:26 smithi114 ceph-mon[30324]: Deploying daemon node-exporter.a on smithi114 2024-09-21T12:21:27.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:26 smithi114 ceph-mon[30324]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T12:21:27.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:26 smithi114 ceph-mon[30324]: osdmap e86: 8 total, 8 up, 8 in 2024-09-21T12:21:27.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:26 smithi114 ceph-mon[40867]: pgmap v33: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:21:27.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:26 smithi114 ceph-mon[40867]: Deploying daemon node-exporter.a on smithi114 2024-09-21T12:21:27.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:26 smithi114 ceph-mon[40867]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T12:21:27.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:26 smithi114 ceph-mon[40867]: osdmap e86: 8 total, 8 up, 8 in 2024-09-21T12:21:27.361 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:27.382 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:27.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:27.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:27.484 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:27.488 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:27 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=grafana-apiserver t=2024-09-21T12:21:27.181551907Z level=info msg="Adding GroupVersion playlist.grafana.app v0alpha1 to ResourceManager" 2024-09-21T12:21:27.488 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:21:27 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=grafana-apiserver t=2024-09-21T12:21:27.181932828Z level=info msg="Adding GroupVersion featuretoggle.grafana.app v0alpha1 to ResourceManager" 2024-09-21T12:21:27.506 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:27.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:27.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:27.607 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:21:27.629 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:27.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:27.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:21:27.649 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:21:27.658 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T12:21:27.731 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:27.753 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:27.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:27.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:27.854 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:27.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:27.877 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:27.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:27.895 INFO:teuthology.orchestra.run.smithi114.stderr:osd set-full-ratio 0.95 2024-09-21T12:21:27.978 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:28.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:28.001 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:28.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:28.102 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:28.124 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:28.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:28.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:28.226 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:28.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:27 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:21:28.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:27 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/4194006403' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.95} : dispatch 2024-09-21T12:21:28.247 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:28.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:28.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:28.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:27 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:21:28.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:27 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4194006403' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.95} : dispatch 2024-09-21T12:21:28.265 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:27 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:21:28.265 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:27 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/4194006403' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.95} : dispatch 2024-09-21T12:21:28.349 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:28.371 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:28.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:28.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:28.410 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T12:21:28.473 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:28.496 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:28.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:28.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:28.598 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:28.619 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:28.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:28.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:28.721 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:21:28.746 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:28.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:28.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:21:28.847 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:21:28.873 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:28.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:28.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:21:28.909 INFO:teuthology.orchestra.run.smithi114.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-21T12:21:28.974 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:28.997 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:28.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:28.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:29.100 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:29.126 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:29.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:29.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:29.227 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:29.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:28 smithi183 ceph-mon[33277]: pgmap v35: 148 pgs: 148 active+clean; 583 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 899 B/s rd, 0 op/s 2024-09-21T12:21:29.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:28 smithi183 ceph-mon[33277]: Health check cleared: OSD_OUT_OF_ORDER_FULL (was: full ratio(s) out of order) 2024-09-21T12:21:29.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:28 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/4194006403' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.95}]': finished 2024-09-21T12:21:29.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:28 smithi183 ceph-mon[33277]: osdmap e87: 8 total, 8 up, 8 in 2024-09-21T12:21:29.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:28 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1818050181' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:21:29.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:28 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:21:29.249 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:29.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:29.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:29.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:28 smithi114 ceph-mon[40867]: pgmap v35: 148 pgs: 148 active+clean; 583 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 899 B/s rd, 0 op/s 2024-09-21T12:21:29.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:28 smithi114 ceph-mon[40867]: Health check cleared: OSD_OUT_OF_ORDER_FULL (was: full ratio(s) out of order) 2024-09-21T12:21:29.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:28 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/4194006403' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.95}]': finished 2024-09-21T12:21:29.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:28 smithi114 ceph-mon[40867]: osdmap e87: 8 total, 8 up, 8 in 2024-09-21T12:21:29.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:28 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1818050181' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:21:29.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:28 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:21:29.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:28 smithi114 ceph-mon[30324]: pgmap v35: 148 pgs: 148 active+clean; 583 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 899 B/s rd, 0 op/s 2024-09-21T12:21:29.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:28 smithi114 ceph-mon[30324]: Health check cleared: OSD_OUT_OF_ORDER_FULL (was: full ratio(s) out of order) 2024-09-21T12:21:29.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:28 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4194006403' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.95}]': finished 2024-09-21T12:21:29.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:28 smithi114 ceph-mon[30324]: osdmap e87: 8 total, 8 up, 8 in 2024-09-21T12:21:29.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:28 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1818050181' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:21:29.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:28 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:21:29.351 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:21:29.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:29.376 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:29.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:21:29.477 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:29.499 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:29.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:29.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:29.578 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:21:29.579 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:21:29.601 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:29.611 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:21:29.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:29.627 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:29.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:29.728 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:29.754 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:29.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:29.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:29.855 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:29.881 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:29.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:29.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:29.982 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:30.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:30.008 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:30.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:30.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:29 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T12:21:30.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:29 smithi114 ceph-mon[40867]: osdmap e88: 8 total, 8 up, 8 in 2024-09-21T12:21:30.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:29 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T12:21:30.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:29 smithi114 ceph-mon[30324]: osdmap e88: 8 total, 8 up, 8 in 2024-09-21T12:21:30.109 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:30.131 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:30.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:30.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:30.163 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:21:30.163 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:21:30.180 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:21:30.180 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:21:30.196 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:21:30.196 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:21:30.212 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:21:30.212 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:21:30.229 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:21:30.229 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:21:30.233 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:30.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:29 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T12:21:30.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:29 smithi183 ceph-mon[33277]: osdmap e88: 8 total, 8 up, 8 in 2024-09-21T12:21:30.246 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:21:30.246 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:21:30.253 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:30.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:30.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:30.263 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:21:30.263 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:21:30.280 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:21:30.280 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:21:30.355 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:30.376 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:30.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:30.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:30.478 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:30.499 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:30.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:30.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:30.541 DEBUG:teuthology.orchestra.run.smithi114:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-09-21T12:21:30.543 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T12:21:30.543 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T12:21:30.600 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:30.618 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:hints = 1 2024-09-21T12:21:30.618 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T12:21:30.618 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Object prefix: benchmark_data_smithi114_78432 2024-09-21T12:21:30.619 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T12:21:30.619 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 0 0 0 0 0 0 - 0 2024-09-21T12:21:30.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:30.629 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:30.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:30.730 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:30.767 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:30.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:30.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:30.869 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:30.907 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:30.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:30.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:31.009 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:31.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:30 smithi114 ceph-mon[30324]: pgmap v38: 148 pgs: 148 active+clean; 583 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T12:21:31.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:30 smithi114 ceph-mon[30324]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T12:21:31.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:30 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/380508771' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:21:31.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:30 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:31.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:30 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:21:31.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:30 smithi114 ceph-mon[40867]: pgmap v38: 148 pgs: 148 active+clean; 583 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T12:21:31.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:30 smithi114 ceph-mon[40867]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T12:21:31.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:30 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/380508771' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:21:31.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:30 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:31.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:30 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:21:31.047 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:31.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:31.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:31.148 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:31.179 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:31.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:31.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:31.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:30 smithi183 ceph-mon[33277]: pgmap v38: 148 pgs: 148 active+clean; 583 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T12:21:31.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:30 smithi183 ceph-mon[33277]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T12:21:31.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:30 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/380508771' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:21:31.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:30 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:31.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:30 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:21:31.281 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:31.317 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:31.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:31.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:31.418 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:31.447 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:31.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:31.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:31.549 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:31.588 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:31.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:31.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:31.618 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 1 16 3440 3424 213.988 214 0.00361592 0.00464575 2024-09-21T12:21:31.690 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:31.718 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:31.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:31.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:31.820 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:21:31.848 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:31.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:31.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:21:31.950 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:21:31.978 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:31.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:31.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:21:32.077 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:21:32.078 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 20, 2024-09-21T12:21:32.078 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 600, 2024-09-21T12:21:32.078 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-21T12:21:32.078 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.078 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.14460.0:1 3.12 3:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-21T12:21:32.078 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:20:15.187258+0000", 2024-09-21T12:21:32.078 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 76.886126586000003, 2024-09-21T12:21:32.078 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.110949246, 2024-09-21T12:21:32.078 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:21:32.078 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:21:32.078 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:21:32.078 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:21:32.079 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.14460", 2024-09-21T12:21:32.079 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/950760115", 2024-09-21T12:21:32.079 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 1 2024-09-21T12:21:32.079 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.079 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:21:32.079 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.079 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:21:32.079 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:20:15.187258+0000", 2024-09-21T12:21:32.079 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.079 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.079 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.079 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:21:32.079 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:20:15.187258+0000", 2024-09-21T12:21:32.079 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.079 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.080 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.080 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:21:32.080 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:20:15.187266+0000", 2024-09-21T12:21:32.080 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.738e-06 2024-09-21T12:21:32.080 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.080 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.080 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:21:32.080 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:20:15.187271+0000", 2024-09-21T12:21:32.080 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8409999999999999e-06 2024-09-21T12:21:32.080 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.080 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.080 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:21:32.080 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:20:15.187281+0000", 2024-09-21T12:21:32.081 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7529999999999996e-06 2024-09-21T12:21:32.081 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.081 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.081 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:21:32.081 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:20:15.187291+0000", 2024-09-21T12:21:32.081 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0232e-05 2024-09-21T12:21:32.081 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.081 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.081 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:21:32.081 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:20:15.187355+0000", 2024-09-21T12:21:32.081 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3812999999999994e-05 2024-09-21T12:21:32.081 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.081 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.081 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:21:32.082 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:20:15.187363+0000", 2024-09-21T12:21:32.082 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7340000000000008e-06 2024-09-21T12:21:32.082 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.082 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.082 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:21:32.082 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:20:15.298146+0000", 2024-09-21T12:21:32.082 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.11078271200000001 2024-09-21T12:21:32.082 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.082 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.082 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:21:32.082 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:20:15.298207+0000", 2024-09-21T12:21:32.082 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1422999999999998e-05 2024-09-21T12:21:32.082 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.083 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:21:32.083 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.083 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.083 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:3402 7.f 7:fa7c6c79:::benchmark_data_smithi114_78432_object3401:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-21T12:21:32.083 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:31.605570+0000", 2024-09-21T12:21:32.083 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.46781464299999997, 2024-09-21T12:21:32.083 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.013164294999999999, 2024-09-21T12:21:32.083 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:21:32.083 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:21:32.083 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:21:32.083 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:21:32.083 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:21:32.083 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:21:32.084 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 3402 2024-09-21T12:21:32.084 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.084 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:21:32.084 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.084 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:21:32.084 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.605570+0000", 2024-09-21T12:21:32.084 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.084 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.084 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.084 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:21:32.084 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.605570+0000", 2024-09-21T12:21:32.084 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.084 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.084 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.084 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:21:32.085 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.605571+0000", 2024-09-21T12:21:32.085 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.553e-06 2024-09-21T12:21:32.085 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.085 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.085 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:21:32.085 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.605594+0000", 2024-09-21T12:21:32.085 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3105e-05 2024-09-21T12:21:32.085 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.085 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.085 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:21:32.085 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.605610+0000", 2024-09-21T12:21:32.085 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5916000000000001e-05 2024-09-21T12:21:32.085 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.085 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.086 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:21:32.086 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.605618+0000", 2024-09-21T12:21:32.086 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9030000000000004e-06 2024-09-21T12:21:32.086 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.086 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.086 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:21:32.086 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.605680+0000", 2024-09-21T12:21:32.086 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2082999999999995e-05 2024-09-21T12:21:32.086 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.086 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.086 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:21:32.086 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.605800+0000", 2024-09-21T12:21:32.087 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000119535 2024-09-21T12:21:32.087 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.087 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.087 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:21:32.087 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.605922+0000", 2024-09-21T12:21:32.087 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000122015 2024-09-21T12:21:32.087 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.087 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.087 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:21:32.087 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.607270+0000", 2024-09-21T12:21:32.087 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001348288 2024-09-21T12:21:32.087 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.087 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.087 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.087 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.614820+0000", 2024-09-21T12:21:32.088 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0075495170000000004 2024-09-21T12:21:32.088 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.088 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.088 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.088 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.618648+0000", 2024-09-21T12:21:32.088 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0038286689999999998 2024-09-21T12:21:32.088 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.088 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.088 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:21:32.088 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.618699+0000", 2024-09-21T12:21:32.088 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0382e-05 2024-09-21T12:21:32.088 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.088 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.088 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:21:32.088 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.618734+0000", 2024-09-21T12:21:32.089 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5329e-05 2024-09-21T12:21:32.089 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.089 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:21:32.089 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.089 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.089 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.089 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:3687 7.6 7:6f740cea:::benchmark_data_smithi114_78432_object3686:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-21T12:21:32.089 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:31.699342+0000", 2024-09-21T12:21:32.089 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.37404219399999999, 2024-09-21T12:21:32.089 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.013209655000000001, 2024-09-21T12:21:32.089 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:21:32.089 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:21:32.089 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:21:32.089 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:21:32.089 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:21:32.090 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:21:32.090 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 3687 2024-09-21T12:21:32.090 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.090 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:21:32.090 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.090 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:21:32.090 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.699342+0000", 2024-09-21T12:21:32.090 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.090 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.090 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.090 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:21:32.090 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.699342+0000", 2024-09-21T12:21:32.090 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.090 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.091 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.091 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:21:32.091 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.699343+0000", 2024-09-21T12:21:32.091 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7100000000000002e-07 2024-09-21T12:21:32.091 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.091 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.091 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:21:32.091 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.699364+0000", 2024-09-21T12:21:32.091 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1687000000000001e-05 2024-09-21T12:21:32.091 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.091 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:32.092 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.092 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:21:32.093 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.699372+0000", 2024-09-21T12:21:32.093 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7020000000000001e-06 2024-09-21T12:21:32.093 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.093 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.093 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:21:32.093 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.699378+0000", 2024-09-21T12:21:32.093 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4920000000000001e-06 2024-09-21T12:21:32.093 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.093 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.093 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:21:32.093 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.702124+0000", 2024-09-21T12:21:32.093 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0027462509999999999 2024-09-21T12:21:32.094 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.094 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.094 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:21:32.094 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.702233+0000", 2024-09-21T12:21:32.094 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000108782 2024-09-21T12:21:32.094 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.094 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.094 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:21:32.094 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.702365+0000", 2024-09-21T12:21:32.094 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000131944 2024-09-21T12:21:32.094 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.094 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.095 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.095 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.704399+0000", 2024-09-21T12:21:32.095 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0020340620000000001 2024-09-21T12:21:32.095 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.095 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.095 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:21:32.095 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.704565+0000", 2024-09-21T12:21:32.095 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000166787 2024-09-21T12:21:32.095 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.095 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.095 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.095 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.712466+0000", 2024-09-21T12:21:32.095 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0079010590000000002 2024-09-21T12:21:32.095 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.096 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.096 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:21:32.096 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.712516+0000", 2024-09-21T12:21:32.096 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9341000000000001e-05 2024-09-21T12:21:32.096 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.096 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.096 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:21:32.096 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.712552+0000", 2024-09-21T12:21:32.096 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5976999999999998e-05 2024-09-21T12:21:32.096 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.096 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:21:32.096 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.096 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.096 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.096 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:3731 7.f 7:fcf4b4ac:::benchmark_data_smithi114_78432_object3730:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-21T12:21:32.097 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:31.715386+0000", 2024-09-21T12:21:32.097 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.357998331, 2024-09-21T12:21:32.097 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.012526035, 2024-09-21T12:21:32.097 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:21:32.097 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:21:32.097 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:21:32.097 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:21:32.097 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:21:32.097 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:21:32.097 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 3731 2024-09-21T12:21:32.097 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.097 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:21:32.097 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.097 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:21:32.098 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.715386+0000", 2024-09-21T12:21:32.098 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.098 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.098 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.098 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:21:32.098 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.715386+0000", 2024-09-21T12:21:32.098 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.098 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.098 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.098 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:21:32.098 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.715387+0000", 2024-09-21T12:21:32.098 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5700000000000001e-07 2024-09-21T12:21:32.098 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.099 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.099 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:21:32.099 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.715413+0000", 2024-09-21T12:21:32.099 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.6214e-05 2024-09-21T12:21:32.099 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.099 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.099 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:21:32.099 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.715450+0000", 2024-09-21T12:21:32.099 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.6551999999999998e-05 2024-09-21T12:21:32.099 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.099 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.099 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:21:32.099 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.715480+0000", 2024-09-21T12:21:32.099 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.0537999999999998e-05 2024-09-21T12:21:32.099 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.100 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.100 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:21:32.100 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.722899+0000", 2024-09-21T12:21:32.100 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0074193899999999997 2024-09-21T12:21:32.100 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.100 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.100 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:21:32.100 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.723386+0000", 2024-09-21T12:21:32.100 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000487002 2024-09-21T12:21:32.100 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.100 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.100 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:21:32.100 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.723556+0000", 2024-09-21T12:21:32.100 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000169066 2024-09-21T12:21:32.101 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.101 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.101 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:21:32.101 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.724459+0000", 2024-09-21T12:21:32.101 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00090370000000000001 2024-09-21T12:21:32.101 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.101 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.101 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.101 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.725362+0000", 2024-09-21T12:21:32.101 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000902887 2024-09-21T12:21:32.101 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.101 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.101 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.101 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.727829+0000", 2024-09-21T12:21:32.102 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0024671990000000002 2024-09-21T12:21:32.102 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.102 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.102 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:21:32.102 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.727874+0000", 2024-09-21T12:21:32.102 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5067000000000001e-05 2024-09-21T12:21:32.102 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.102 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.102 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:21:32.102 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.727912+0000", 2024-09-21T12:21:32.102 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.7562999999999999e-05 2024-09-21T12:21:32.102 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.102 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:21:32.102 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.103 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.103 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.103 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:3806 7.f 7:ff1a304c:::benchmark_data_smithi114_78432_object3805:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-21T12:21:32.103 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:31.751511+0000", 2024-09-21T12:21:32.103 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.32187368300000002, 2024-09-21T12:21:32.103 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.012173191, 2024-09-21T12:21:32.103 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:21:32.103 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:21:32.103 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:21:32.103 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:21:32.103 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:21:32.103 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:21:32.103 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 3806 2024-09-21T12:21:32.103 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.103 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:21:32.104 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.104 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:21:32.104 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.751511+0000", 2024-09-21T12:21:32.104 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.104 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.104 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.104 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:21:32.104 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.751511+0000", 2024-09-21T12:21:32.104 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.104 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.104 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.104 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:21:32.104 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.751512+0000", 2024-09-21T12:21:32.104 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5039999999999999e-06 2024-09-21T12:21:32.104 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.105 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.105 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:21:32.105 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.751537+0000", 2024-09-21T12:21:32.105 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.4590999999999999e-05 2024-09-21T12:21:32.105 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.105 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.105 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:21:32.105 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.751549+0000", 2024-09-21T12:21:32.105 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2014999999999999e-05 2024-09-21T12:21:32.105 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.105 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.105 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:21:32.105 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.751556+0000", 2024-09-21T12:21:32.105 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9940000000000003e-06 2024-09-21T12:21:32.106 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.106 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.106 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:21:32.106 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.754147+0000", 2024-09-21T12:21:32.106 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0025913619999999998 2024-09-21T12:21:32.106 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.106 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.106 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:21:32.106 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.754263+0000", 2024-09-21T12:21:32.106 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000115773 2024-09-21T12:21:32.106 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.106 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.106 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:21:32.106 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.754387+0000", 2024-09-21T12:21:32.107 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012453799999999999 2024-09-21T12:21:32.107 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.107 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.107 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:21:32.107 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.754894+0000", 2024-09-21T12:21:32.107 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00050670299999999995 2024-09-21T12:21:32.107 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.107 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.107 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.107 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.759106+0000", 2024-09-21T12:21:32.107 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0042116039999999999 2024-09-21T12:21:32.107 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.107 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.107 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.108 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.763595+0000", 2024-09-21T12:21:32.108 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0044893229999999999 2024-09-21T12:21:32.108 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.108 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.108 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:21:32.108 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.763643+0000", 2024-09-21T12:21:32.108 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8162999999999999e-05 2024-09-21T12:21:32.108 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.108 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.108 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:21:32.108 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.763684+0000", 2024-09-21T12:21:32.108 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0621000000000003e-05 2024-09-21T12:21:32.108 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.108 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:21:32.108 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.109 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.109 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.109 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:3838 7.f 7:f81917aa:::benchmark_data_smithi114_78432_object3837:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-21T12:21:32.109 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:31.765314+0000", 2024-09-21T12:21:32.109 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.30807041800000001, 2024-09-21T12:21:32.109 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.011817563, 2024-09-21T12:21:32.109 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:21:32.109 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:21:32.109 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:21:32.109 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:21:32.109 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:21:32.109 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:21:32.109 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 3838 2024-09-21T12:21:32.109 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.109 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:21:32.110 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.110 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:21:32.110 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.765314+0000", 2024-09-21T12:21:32.110 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.110 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.110 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.110 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:21:32.110 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.765314+0000", 2024-09-21T12:21:32.110 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.110 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.110 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.110 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:21:32.110 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.765315+0000", 2024-09-21T12:21:32.110 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3340000000000001e-06 2024-09-21T12:21:32.111 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.111 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.111 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:21:32.111 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.765338+0000", 2024-09-21T12:21:32.111 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2379999999999999e-05 2024-09-21T12:21:32.111 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.111 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.111 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:21:32.111 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.765348+0000", 2024-09-21T12:21:32.111 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0076e-05 2024-09-21T12:21:32.111 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.111 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.111 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:21:32.111 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.765354+0000", 2024-09-21T12:21:32.112 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5649999999999998e-06 2024-09-21T12:21:32.112 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.112 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.112 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:21:32.112 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.765660+0000", 2024-09-21T12:21:32.112 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00030624299999999999 2024-09-21T12:21:32.112 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.112 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.112 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:21:32.112 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.765783+0000", 2024-09-21T12:21:32.112 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012216899999999999 2024-09-21T12:21:32.112 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.112 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.112 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:21:32.113 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.765920+0000", 2024-09-21T12:21:32.113 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013778299999999999 2024-09-21T12:21:32.113 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.113 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.113 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:21:32.113 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.771824+0000", 2024-09-21T12:21:32.113 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0059034100000000004 2024-09-21T12:21:32.113 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.113 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.113 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.113 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.771856+0000", 2024-09-21T12:21:32.113 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.1795000000000003e-05 2024-09-21T12:21:32.113 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.113 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.113 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.114 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.777047+0000", 2024-09-21T12:21:32.114 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0051913410000000004 2024-09-21T12:21:32.114 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.114 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.114 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:21:32.114 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.777095+0000", 2024-09-21T12:21:32.114 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7663000000000001e-05 2024-09-21T12:21:32.114 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.114 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.114 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:21:32.114 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.777131+0000", 2024-09-21T12:21:32.114 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.6804000000000002e-05 2024-09-21T12:21:32.114 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.114 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:21:32.115 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.115 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.115 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.115 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:3845 7.9 7:933f6c45:::benchmark_data_smithi114_78432_object3844:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-21T12:21:32.115 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:31.767842+0000", 2024-09-21T12:21:32.115 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.30554219399999999, 2024-09-21T12:21:32.115 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.014516375, 2024-09-21T12:21:32.115 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:21:32.115 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:21:32.115 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:21:32.115 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:21:32.115 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:21:32.115 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:21:32.116 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 3845 2024-09-21T12:21:32.116 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.116 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:21:32.116 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.116 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:21:32.116 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.767842+0000", 2024-09-21T12:21:32.116 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.116 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.116 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.116 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:21:32.116 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.767842+0000", 2024-09-21T12:21:32.116 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.116 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.116 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.117 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:21:32.117 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.767844+0000", 2024-09-21T12:21:32.117 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.486e-06 2024-09-21T12:21:32.117 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.117 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.117 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:21:32.117 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.767869+0000", 2024-09-21T12:21:32.117 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5510000000000001e-05 2024-09-21T12:21:32.117 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.117 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.117 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:21:32.117 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.767881+0000", 2024-09-21T12:21:32.117 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2152e-05 2024-09-21T12:21:32.117 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.117 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.117 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:21:32.118 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.767889+0000", 2024-09-21T12:21:32.118 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4819999999999997e-06 2024-09-21T12:21:32.118 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.118 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.118 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:21:32.118 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.777776+0000", 2024-09-21T12:21:32.118 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0098870690000000001 2024-09-21T12:21:32.118 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.118 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.118 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:21:32.118 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.777880+0000", 2024-09-21T12:21:32.118 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010470499999999999 2024-09-21T12:21:32.118 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.118 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.119 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,4", 2024-09-21T12:21:32.119 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.777999+0000", 2024-09-21T12:21:32.119 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011848199999999999 2024-09-21T12:21:32.119 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.119 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.119 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:21:32.119 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.782045+0000", 2024-09-21T12:21:32.120 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0040455739999999997 2024-09-21T12:21:32.120 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.120 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.120 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.120 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.782096+0000", 2024-09-21T12:21:32.120 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1177999999999997e-05 2024-09-21T12:21:32.120 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.120 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.121 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.121 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.782266+0000", 2024-09-21T12:21:32.121 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017044299999999999 2024-09-21T12:21:32.121 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.121 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.121 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:21:32.121 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.782315+0000", 2024-09-21T12:21:32.121 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8875999999999998e-05 2024-09-21T12:21:32.121 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.121 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.121 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:21:32.121 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.782358+0000", 2024-09-21T12:21:32.121 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3417999999999997e-05 2024-09-21T12:21:32.121 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.122 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:21:32.122 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.122 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.122 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.122 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:3852 7.6 7:6efb763d:::benchmark_data_smithi114_78432_object3851:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-21T12:21:32.122 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:31.770921+0000", 2024-09-21T12:21:32.122 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.30246318799999999, 2024-09-21T12:21:32.122 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.013717216000000001, 2024-09-21T12:21:32.122 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:21:32.122 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:21:32.122 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:21:32.122 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:21:32.122 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:21:32.122 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:21:32.123 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 3852 2024-09-21T12:21:32.123 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.123 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:21:32.123 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.123 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:21:32.123 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.770921+0000", 2024-09-21T12:21:32.123 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.123 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.123 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.123 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:21:32.123 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.770921+0000", 2024-09-21T12:21:32.123 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.123 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.123 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.123 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:21:32.124 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.770923+0000", 2024-09-21T12:21:32.124 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.443e-06 2024-09-21T12:21:32.124 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.124 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.124 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:21:32.124 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.770944+0000", 2024-09-21T12:21:32.124 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1545000000000001e-05 2024-09-21T12:21:32.124 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.124 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.124 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:21:32.124 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.770952+0000", 2024-09-21T12:21:32.124 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.225e-06 2024-09-21T12:21:32.124 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.124 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.125 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:21:32.125 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.770958+0000", 2024-09-21T12:21:32.125 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2229999999999998e-06 2024-09-21T12:21:32.125 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.125 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.125 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:21:32.125 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.771027+0000", 2024-09-21T12:21:32.125 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9542000000000006e-05 2024-09-21T12:21:32.125 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.125 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.125 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:21:32.125 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.771130+0000", 2024-09-21T12:21:32.125 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010289900000000001 2024-09-21T12:21:32.125 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.125 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.126 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:21:32.126 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.771241+0000", 2024-09-21T12:21:32.126 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000110907 2024-09-21T12:21:32.126 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.126 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.126 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:21:32.126 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.778437+0000", 2024-09-21T12:21:32.126 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0071958730000000002 2024-09-21T12:21:32.126 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.126 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.126 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.126 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.782690+0000", 2024-09-21T12:21:32.126 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0042533839999999998 2024-09-21T12:21:32.126 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.126 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.127 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.127 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.784529+0000", 2024-09-21T12:21:32.127 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0018389840000000001 2024-09-21T12:21:32.127 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.127 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.127 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:21:32.127 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.784580+0000", 2024-09-21T12:21:32.127 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0832999999999999e-05 2024-09-21T12:21:32.127 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.127 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.127 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:21:32.127 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.784638+0000", 2024-09-21T12:21:32.127 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8357999999999999e-05 2024-09-21T12:21:32.127 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.127 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:21:32.128 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.128 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.128 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.128 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:3860 7.f 7:f5989534:::benchmark_data_smithi114_78432_object3859:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-21T12:21:32.128 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:31.777505+0000", 2024-09-21T12:21:32.128 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.295879691, 2024-09-21T12:21:32.128 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.012867585000000001, 2024-09-21T12:21:32.128 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:21:32.128 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:21:32.128 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:21:32.128 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:21:32.128 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:21:32.128 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:21:32.128 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 3860 2024-09-21T12:21:32.129 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.129 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:21:32.129 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.129 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:21:32.129 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.777505+0000", 2024-09-21T12:21:32.129 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.129 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.129 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.129 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:21:32.129 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.777505+0000", 2024-09-21T12:21:32.129 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.129 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.129 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.129 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:21:32.130 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.777506+0000", 2024-09-21T12:21:32.130 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.299e-06 2024-09-21T12:21:32.130 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.130 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.130 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:21:32.130 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.777531+0000", 2024-09-21T12:21:32.130 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5028000000000002e-05 2024-09-21T12:21:32.130 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.130 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.130 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:21:32.130 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.777543+0000", 2024-09-21T12:21:32.130 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1815e-05 2024-09-21T12:21:32.130 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.130 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.130 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:21:32.131 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.777550+0000", 2024-09-21T12:21:32.131 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3799999999999996e-06 2024-09-21T12:21:32.131 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.131 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.131 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:21:32.131 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.777604+0000", 2024-09-21T12:21:32.131 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3943000000000002e-05 2024-09-21T12:21:32.131 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.131 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.131 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:21:32.131 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.777753+0000", 2024-09-21T12:21:32.131 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014940400000000001 2024-09-21T12:21:32.131 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.131 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.131 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:21:32.132 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.777931+0000", 2024-09-21T12:21:32.132 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017724400000000001 2024-09-21T12:21:32.132 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.132 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.132 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:21:32.132 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.781792+0000", 2024-09-21T12:21:32.132 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0038615609999999999 2024-09-21T12:21:32.132 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.132 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.132 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.132 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.787844+0000", 2024-09-21T12:21:32.132 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0060514549999999999 2024-09-21T12:21:32.132 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.133 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.133 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.133 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.790295+0000", 2024-09-21T12:21:32.133 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002450856 2024-09-21T12:21:32.133 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.133 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.133 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:21:32.133 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.790339+0000", 2024-09-21T12:21:32.133 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.464e-05 2024-09-21T12:21:32.133 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.133 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.133 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:21:32.133 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.790372+0000", 2024-09-21T12:21:32.133 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2960000000000003e-05 2024-09-21T12:21:32.133 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.134 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:21:32.134 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.134 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.134 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.134 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:3872 7.6 7:616ac780:::benchmark_data_smithi114_78432_object3871:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-21T12:21:32.134 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:31.782440+0000", 2024-09-21T12:21:32.134 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.29094449999999999, 2024-09-21T12:21:32.134 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.019032276000000001, 2024-09-21T12:21:32.134 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:21:32.134 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:21:32.134 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:21:32.134 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:21:32.134 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:21:32.134 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:21:32.135 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 3872 2024-09-21T12:21:32.135 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.135 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:21:32.135 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.135 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:21:32.135 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.782440+0000", 2024-09-21T12:21:32.135 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.135 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.135 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.135 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:21:32.135 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.782440+0000", 2024-09-21T12:21:32.135 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.135 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.135 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.136 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:21:32.136 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.782441+0000", 2024-09-21T12:21:32.136 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6700000000000002e-07 2024-09-21T12:21:32.136 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.136 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.136 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:21:32.136 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.782466+0000", 2024-09-21T12:21:32.136 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5023000000000001e-05 2024-09-21T12:21:32.136 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.136 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.136 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:21:32.136 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.782474+0000", 2024-09-21T12:21:32.136 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9559999999999997e-06 2024-09-21T12:21:32.136 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.136 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.137 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:21:32.137 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.782479+0000", 2024-09-21T12:21:32.137 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7150000000000003e-06 2024-09-21T12:21:32.137 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.137 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.137 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:21:32.137 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.782718+0000", 2024-09-21T12:21:32.137 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00023859200000000001 2024-09-21T12:21:32.137 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.137 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.137 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:21:32.137 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.782815+0000", 2024-09-21T12:21:32.137 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6905999999999997e-05 2024-09-21T12:21:32.137 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.137 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.138 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:21:32.138 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.782932+0000", 2024-09-21T12:21:32.138 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011689 2024-09-21T12:21:32.138 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.138 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.138 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:21:32.138 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.784648+0000", 2024-09-21T12:21:32.138 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001716548 2024-09-21T12:21:32.138 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.138 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.138 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.138 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.788889+0000", 2024-09-21T12:21:32.138 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0042407260000000002 2024-09-21T12:21:32.138 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.139 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.139 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.139 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.801381+0000", 2024-09-21T12:21:32.139 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.012491412 2024-09-21T12:21:32.139 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.139 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.139 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:21:32.139 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.801434+0000", 2024-09-21T12:21:32.139 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3720000000000001e-05 2024-09-21T12:21:32.139 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.139 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.139 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:21:32.139 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.801472+0000", 2024-09-21T12:21:32.139 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.7821000000000003e-05 2024-09-21T12:21:32.140 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.140 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:21:32.140 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.140 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.140 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.140 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:3876 7.6 7:68722ebf:::benchmark_data_smithi114_78432_object3875:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-21T12:21:32.140 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:31.786218+0000", 2024-09-21T12:21:32.140 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.28716668299999998, 2024-09-21T12:21:32.140 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.015643351999999999, 2024-09-21T12:21:32.140 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:21:32.140 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:21:32.140 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:21:32.140 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:21:32.140 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:21:32.141 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:21:32.141 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 3876 2024-09-21T12:21:32.141 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.141 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:21:32.141 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.141 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:21:32.141 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.786218+0000", 2024-09-21T12:21:32.141 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.141 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.141 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.141 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:21:32.141 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.786218+0000", 2024-09-21T12:21:32.141 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.141 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.142 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.142 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:21:32.142 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.786219+0000", 2024-09-21T12:21:32.142 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5570000000000001e-06 2024-09-21T12:21:32.142 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.142 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.142 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:21:32.142 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.786246+0000", 2024-09-21T12:21:32.142 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.6917e-05 2024-09-21T12:21:32.142 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.142 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.142 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:21:32.142 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.786274+0000", 2024-09-21T12:21:32.142 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.7572999999999998e-05 2024-09-21T12:21:32.143 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.143 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.143 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:21:32.143 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.786283+0000", 2024-09-21T12:21:32.143 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.645e-06 2024-09-21T12:21:32.143 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.143 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.143 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:21:32.143 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.786391+0000", 2024-09-21T12:21:32.143 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000108175 2024-09-21T12:21:32.143 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.143 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.143 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:21:32.143 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.786525+0000", 2024-09-21T12:21:32.144 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000133974 2024-09-21T12:21:32.144 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.144 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.144 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:21:32.144 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.786870+0000", 2024-09-21T12:21:32.144 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00034482600000000001 2024-09-21T12:21:32.144 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.144 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.144 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.144 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.791383+0000", 2024-09-21T12:21:32.144 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0045130939999999996 2024-09-21T12:21:32.144 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.144 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.144 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:21:32.144 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.799976+0000", 2024-09-21T12:21:32.145 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0085928879999999999 2024-09-21T12:21:32.145 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.145 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.145 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.145 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.801510+0000", 2024-09-21T12:21:32.145 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001533576 2024-09-21T12:21:32.145 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.145 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.145 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:21:32.145 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.801692+0000", 2024-09-21T12:21:32.145 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00018258 2024-09-21T12:21:32.145 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.145 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.145 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:21:32.146 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.801861+0000", 2024-09-21T12:21:32.146 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016854700000000001 2024-09-21T12:21:32.146 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.146 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:21:32.146 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.146 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.146 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.147 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:3877 7.9 7:94774e28:::benchmark_data_smithi114_78432_object3876:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-21T12:21:32.147 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:31.786303+0000", 2024-09-21T12:21:32.147 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.28708150100000002, 2024-09-21T12:21:32.147 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.013895527, 2024-09-21T12:21:32.147 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:21:32.147 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:21:32.147 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:21:32.147 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:21:32.148 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:21:32.148 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:21:32.148 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 3877 2024-09-21T12:21:32.148 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.148 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:21:32.148 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.148 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:21:32.148 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.786303+0000", 2024-09-21T12:21:32.148 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.148 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.148 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.148 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:21:32.148 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.786303+0000", 2024-09-21T12:21:32.148 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.149 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.149 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.149 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:21:32.149 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.786303+0000", 2024-09-21T12:21:32.149 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5000000000000003e-07 2024-09-21T12:21:32.149 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.149 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.149 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:21:32.149 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.786326+0000", 2024-09-21T12:21:32.149 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3153999999999999e-05 2024-09-21T12:21:32.149 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.149 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.149 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:21:32.149 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.786335+0000", 2024-09-21T12:21:32.149 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1000000000000004e-06 2024-09-21T12:21:32.150 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.150 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.150 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:21:32.150 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.786340+0000", 2024-09-21T12:21:32.150 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9109999999999998e-06 2024-09-21T12:21:32.150 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.150 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.150 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:21:32.150 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.787011+0000", 2024-09-21T12:21:32.150 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00067064299999999996 2024-09-21T12:21:32.150 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.150 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.150 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:21:32.150 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.787121+0000", 2024-09-21T12:21:32.151 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000110331 2024-09-21T12:21:32.151 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.151 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.151 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,4", 2024-09-21T12:21:32.151 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.787241+0000", 2024-09-21T12:21:32.151 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011931500000000001 2024-09-21T12:21:32.151 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.151 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.151 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.151 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.790397+0000", 2024-09-21T12:21:32.151 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0031562370000000001 2024-09-21T12:21:32.151 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.151 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.151 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.152 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.796334+0000", 2024-09-21T12:21:32.152 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0059372679999999999 2024-09-21T12:21:32.152 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.152 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.152 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:21:32.152 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.800103+0000", 2024-09-21T12:21:32.152 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0037685829999999998 2024-09-21T12:21:32.152 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.152 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.152 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:21:32.152 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.800155+0000", 2024-09-21T12:21:32.152 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1722999999999999e-05 2024-09-21T12:21:32.152 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.152 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.153 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:21:32.153 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.800198+0000", 2024-09-21T12:21:32.153 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3711999999999998e-05 2024-09-21T12:21:32.153 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.153 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:21:32.153 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.153 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.153 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.153 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.24695.0:3879 7.e e88/84 7:789d18a8:::benchmark_data_smithi114_78432_object3878:head v 88'244, mlcod=88'242)", 2024-09-21T12:21:32.153 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:31.787551+0000", 2024-09-21T12:21:32.153 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.28583314999999998, 2024-09-21T12:21:32.153 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.012479618, 2024-09-21T12:21:32.153 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:21:32.153 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:21:32.153 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:21:32.154 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:21:32.154 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.154 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:21:32.154 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.787551+0000", 2024-09-21T12:21:32.154 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.154 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.154 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.154 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:21:32.154 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.787551+0000", 2024-09-21T12:21:32.154 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.154 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.154 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.154 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:21:32.154 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.787552+0000", 2024-09-21T12:21:32.155 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.08e-06 2024-09-21T12:21:32.155 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.155 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.155 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:21:32.155 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.787577+0000", 2024-09-21T12:21:32.155 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.4989000000000001e-05 2024-09-21T12:21:32.155 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.155 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.155 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:21:32.155 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.787590+0000", 2024-09-21T12:21:32.155 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2595999999999999e-05 2024-09-21T12:21:32.155 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.155 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.155 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:21:32.155 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.787597+0000", 2024-09-21T12:21:32.156 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6009999999999997e-06 2024-09-21T12:21:32.156 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.156 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.156 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:21:32.156 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.787747+0000", 2024-09-21T12:21:32.156 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015010100000000001 2024-09-21T12:21:32.156 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.156 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.156 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:21:32.156 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.787777+0000", 2024-09-21T12:21:32.156 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.9805e-05 2024-09-21T12:21:32.156 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.156 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.156 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:21:32.156 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.799979+0000", 2024-09-21T12:21:32.156 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.012201236000000001 2024-09-21T12:21:32.157 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.157 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.157 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:21:32.157 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.800031+0000", 2024-09-21T12:21:32.157 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2210000000000003e-05 2024-09-21T12:21:32.157 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.157 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:21:32.157 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.157 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.157 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.157 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:4030 7.f 7:febe03a3:::benchmark_data_smithi114_78432_object4029:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-21T12:21:32.157 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:31.850638+0000", 2024-09-21T12:21:32.157 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.222746004, 2024-09-21T12:21:32.157 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.011949545000000001, 2024-09-21T12:21:32.158 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:21:32.158 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:21:32.158 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:21:32.158 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:21:32.158 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:21:32.158 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:21:32.158 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 4030 2024-09-21T12:21:32.158 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.158 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:21:32.158 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.158 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:21:32.158 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.850638+0000", 2024-09-21T12:21:32.158 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.158 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.159 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.159 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:21:32.159 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.850638+0000", 2024-09-21T12:21:32.159 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.159 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.159 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.159 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:21:32.159 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.850639+0000", 2024-09-21T12:21:32.159 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8199999999999999e-07 2024-09-21T12:21:32.159 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.159 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.159 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:21:32.159 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.850662+0000", 2024-09-21T12:21:32.159 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2464e-05 2024-09-21T12:21:32.159 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.160 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.160 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:21:32.160 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.850670+0000", 2024-09-21T12:21:32.160 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5439999999999999e-06 2024-09-21T12:21:32.160 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.160 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.160 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:21:32.160 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.850676+0000", 2024-09-21T12:21:32.160 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0599999999999996e-06 2024-09-21T12:21:32.160 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.160 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.160 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:21:32.160 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.850738+0000", 2024-09-21T12:21:32.160 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1396999999999995e-05 2024-09-21T12:21:32.160 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.160 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.161 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:21:32.161 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.851482+0000", 2024-09-21T12:21:32.161 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00074480200000000003 2024-09-21T12:21:32.161 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.161 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.161 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:21:32.161 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.851612+0000", 2024-09-21T12:21:32.161 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012957299999999999 2024-09-21T12:21:32.161 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.161 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.161 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:21:32.161 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.852188+0000", 2024-09-21T12:21:32.161 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00057633599999999999 2024-09-21T12:21:32.161 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.161 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.162 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.162 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.856554+0000", 2024-09-21T12:21:32.162 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0043658239999999999 2024-09-21T12:21:32.162 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.162 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.162 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.162 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.862489+0000", 2024-09-21T12:21:32.162 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0059350080000000003 2024-09-21T12:21:32.162 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.162 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.162 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:21:32.162 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.862545+0000", 2024-09-21T12:21:32.162 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5605e-05 2024-09-21T12:21:32.162 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.163 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.163 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:21:32.163 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.862588+0000", 2024-09-21T12:21:32.163 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3149999999999999e-05 2024-09-21T12:21:32.163 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.163 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:21:32.163 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.163 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.163 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.163 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:4043 7.9 7:900473b2:::benchmark_data_smithi114_78432_object4042:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-21T12:21:32.163 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:31.857533+0000", 2024-09-21T12:21:32.163 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.21585167399999999, 2024-09-21T12:21:32.163 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.011345587000000001, 2024-09-21T12:21:32.163 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:21:32.164 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:21:32.164 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:21:32.164 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:21:32.164 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:21:32.164 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:21:32.164 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 4043 2024-09-21T12:21:32.164 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.164 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:21:32.164 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.164 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:21:32.164 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.857533+0000", 2024-09-21T12:21:32.164 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.164 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.164 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.164 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:21:32.165 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.857533+0000", 2024-09-21T12:21:32.165 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.165 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.165 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.165 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:21:32.165 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.857534+0000", 2024-09-21T12:21:32.165 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3069999999999999e-06 2024-09-21T12:21:32.165 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.165 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.165 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:21:32.165 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.857557+0000", 2024-09-21T12:21:32.165 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2639000000000001e-05 2024-09-21T12:21:32.165 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.166 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.166 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:21:32.166 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.857574+0000", 2024-09-21T12:21:32.166 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7501000000000001e-05 2024-09-21T12:21:32.166 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.166 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.166 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:21:32.166 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.857582+0000", 2024-09-21T12:21:32.166 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.0120000000000005e-06 2024-09-21T12:21:32.166 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.166 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.166 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:21:32.166 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.857690+0000", 2024-09-21T12:21:32.166 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000107808 2024-09-21T12:21:32.167 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.167 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.167 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:21:32.167 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.857875+0000", 2024-09-21T12:21:32.167 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00018550899999999999 2024-09-21T12:21:32.167 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.167 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.167 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,4", 2024-09-21T12:21:32.167 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.858001+0000", 2024-09-21T12:21:32.167 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012554100000000001 2024-09-21T12:21:32.167 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.167 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.167 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:21:32.167 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.859002+0000", 2024-09-21T12:21:32.167 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0010008549999999999 2024-09-21T12:21:32.168 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.168 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.168 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.168 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.868278+0000", 2024-09-21T12:21:32.168 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0092767240000000001 2024-09-21T12:21:32.168 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.168 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.168 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.168 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.868729+0000", 2024-09-21T12:21:32.168 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00045098799999999999 2024-09-21T12:21:32.168 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.168 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.168 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:21:32.168 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.868839+0000", 2024-09-21T12:21:32.169 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000109137 2024-09-21T12:21:32.169 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.169 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.169 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:21:32.169 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.868878+0000", 2024-09-21T12:21:32.169 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.9566e-05 2024-09-21T12:21:32.169 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.169 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:21:32.169 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.169 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.169 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.169 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:4134 7.6 7:61f1eb57:::benchmark_data_smithi114_78432_object4133:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-21T12:21:32.169 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:31.887620+0000", 2024-09-21T12:21:32.169 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.185764078, 2024-09-21T12:21:32.169 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.012269581999999999, 2024-09-21T12:21:32.170 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:21:32.170 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:21:32.170 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:21:32.170 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:21:32.170 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:21:32.170 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:21:32.170 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 4134 2024-09-21T12:21:32.170 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.170 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:21:32.170 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.170 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:21:32.170 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.887620+0000", 2024-09-21T12:21:32.170 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.170 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.171 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.171 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:21:32.171 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.887620+0000", 2024-09-21T12:21:32.171 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.171 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.171 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.171 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:21:32.171 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.887621+0000", 2024-09-21T12:21:32.171 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5299999999999999e-07 2024-09-21T12:21:32.171 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.171 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.171 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:21:32.171 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.887640+0000", 2024-09-21T12:21:32.171 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9562e-05 2024-09-21T12:21:32.172 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.172 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.172 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:21:32.172 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.887647+0000", 2024-09-21T12:21:32.172 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.579e-06 2024-09-21T12:21:32.172 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.172 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.172 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:21:32.172 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.887652+0000", 2024-09-21T12:21:32.172 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9710000000000003e-06 2024-09-21T12:21:32.172 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.172 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.172 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:21:32.172 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.888840+0000", 2024-09-21T12:21:32.172 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0011886500000000001 2024-09-21T12:21:32.173 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.173 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.173 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:21:32.173 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.888941+0000", 2024-09-21T12:21:32.173 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000100987 2024-09-21T12:21:32.173 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.173 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.173 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:21:32.173 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.889073+0000", 2024-09-21T12:21:32.173 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013179500000000001 2024-09-21T12:21:32.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:32.174 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:32.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:32.174 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.174 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.174 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.175 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.894751+0000", 2024-09-21T12:21:32.175 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.005677975 2024-09-21T12:21:32.175 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.175 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.175 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:21:32.175 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.895211+0000", 2024-09-21T12:21:32.175 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00045992 2024-09-21T12:21:32.175 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.175 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.175 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.175 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.899790+0000", 2024-09-21T12:21:32.175 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0045791210000000002 2024-09-21T12:21:32.175 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.175 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.176 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:21:32.176 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.899845+0000", 2024-09-21T12:21:32.176 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5047999999999999e-05 2024-09-21T12:21:32.176 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.176 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.176 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:21:32.176 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.899890+0000", 2024-09-21T12:21:32.176 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4521000000000003e-05 2024-09-21T12:21:32.176 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.176 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:21:32.176 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.176 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.176 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.176 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:4165 7.4 7:2a3e48cb:::benchmark_data_smithi114_78432_object4164:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-21T12:21:32.177 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:31.903934+0000", 2024-09-21T12:21:32.177 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.169450552, 2024-09-21T12:21:32.177 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.011475740999999999, 2024-09-21T12:21:32.177 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:21:32.177 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:21:32.177 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:21:32.177 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:21:32.177 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:21:32.177 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:21:32.177 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 4165 2024-09-21T12:21:32.177 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.177 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:21:32.177 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.177 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:21:32.177 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.903934+0000", 2024-09-21T12:21:32.178 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.178 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.178 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.178 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:21:32.178 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.903934+0000", 2024-09-21T12:21:32.178 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.178 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.178 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.178 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:21:32.178 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.903934+0000", 2024-09-21T12:21:32.178 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.61e-07 2024-09-21T12:21:32.178 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.178 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.178 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:21:32.179 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.903958+0000", 2024-09-21T12:21:32.179 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.4019000000000002e-05 2024-09-21T12:21:32.179 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.179 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.179 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:21:32.179 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.903968+0000", 2024-09-21T12:21:32.179 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6379999999999999e-06 2024-09-21T12:21:32.179 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.179 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.179 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:21:32.179 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.903975+0000", 2024-09-21T12:21:32.179 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8129999999999998e-06 2024-09-21T12:21:32.179 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.179 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.179 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:21:32.180 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.904036+0000", 2024-09-21T12:21:32.180 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0584999999999997e-05 2024-09-21T12:21:32.180 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.180 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.180 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:21:32.180 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.904143+0000", 2024-09-21T12:21:32.180 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000107311 2024-09-21T12:21:32.180 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.180 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.180 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,4", 2024-09-21T12:21:32.180 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.904261+0000", 2024-09-21T12:21:32.180 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000117871 2024-09-21T12:21:32.180 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.180 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.180 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:21:32.181 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.906382+0000", 2024-09-21T12:21:32.181 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0021208680000000001 2024-09-21T12:21:32.181 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.181 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.181 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.181 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.907138+0000", 2024-09-21T12:21:32.181 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00075597000000000001 2024-09-21T12:21:32.181 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.181 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.181 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.181 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.914129+0000", 2024-09-21T12:21:32.181 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0069914510000000001 2024-09-21T12:21:32.181 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.181 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.181 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:21:32.182 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.914983+0000", 2024-09-21T12:21:32.182 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00085419999999999995 2024-09-21T12:21:32.182 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.182 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.182 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:21:32.182 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.915409+0000", 2024-09-21T12:21:32.182 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00042625399999999999 2024-09-21T12:21:32.182 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.182 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:21:32.182 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.182 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.182 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.182 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:4173 7.f 7:f8781bfc:::benchmark_data_smithi114_78432_object4172:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-21T12:21:32.182 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:31.905827+0000", 2024-09-21T12:21:32.183 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.16755686, 2024-09-21T12:21:32.183 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.012839562000000001, 2024-09-21T12:21:32.183 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:21:32.183 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:21:32.183 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:21:32.183 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:21:32.183 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:21:32.183 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:21:32.183 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 4173 2024-09-21T12:21:32.183 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.183 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:21:32.183 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.183 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:21:32.183 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.905827+0000", 2024-09-21T12:21:32.183 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.184 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.184 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.184 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:21:32.184 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.905827+0000", 2024-09-21T12:21:32.184 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.184 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.184 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.184 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:21:32.184 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.905829+0000", 2024-09-21T12:21:32.184 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.313e-06 2024-09-21T12:21:32.184 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.184 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.184 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:21:32.184 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.905851+0000", 2024-09-21T12:21:32.185 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2425e-05 2024-09-21T12:21:32.185 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.185 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.185 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:21:32.185 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.905864+0000", 2024-09-21T12:21:32.185 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2364999999999999e-05 2024-09-21T12:21:32.185 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.185 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.185 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:21:32.185 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.905872+0000", 2024-09-21T12:21:32.185 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.2190000000000004e-06 2024-09-21T12:21:32.185 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.185 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.185 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:21:32.185 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.906129+0000", 2024-09-21T12:21:32.186 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00025757699999999999 2024-09-21T12:21:32.186 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.186 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.186 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:21:32.186 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.906250+0000", 2024-09-21T12:21:32.186 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000120846 2024-09-21T12:21:32.186 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.186 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.186 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:21:32.186 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.906457+0000", 2024-09-21T12:21:32.186 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00020687599999999999 2024-09-21T12:21:32.186 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.186 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.186 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.186 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.907885+0000", 2024-09-21T12:21:32.187 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001428035 2024-09-21T12:21:32.187 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.187 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.187 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:21:32.187 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.907905+0000", 2024-09-21T12:21:32.187 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0015999999999999e-05 2024-09-21T12:21:32.187 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.187 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.187 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.187 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.918562+0000", 2024-09-21T12:21:32.187 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010656580000000001 2024-09-21T12:21:32.187 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.187 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.187 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:21:32.187 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.918621+0000", 2024-09-21T12:21:32.188 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9341e-05 2024-09-21T12:21:32.188 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.188 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.188 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:21:32.188 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.918667+0000", 2024-09-21T12:21:32.188 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5969e-05 2024-09-21T12:21:32.188 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.188 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:21:32.188 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.188 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.188 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.188 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:4175 7.9 7:9da9e1b5:::benchmark_data_smithi114_78432_object4174:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-21T12:21:32.188 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:31.906521+0000", 2024-09-21T12:21:32.188 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.16686295300000001, 2024-09-21T12:21:32.189 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.013647197999999999, 2024-09-21T12:21:32.189 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:21:32.189 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:21:32.189 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:21:32.189 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:21:32.189 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:21:32.189 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:21:32.189 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 4175 2024-09-21T12:21:32.189 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.189 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:21:32.189 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.189 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:21:32.189 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.906521+0000", 2024-09-21T12:21:32.189 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.189 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.190 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.190 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:21:32.190 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.906521+0000", 2024-09-21T12:21:32.190 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.190 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.190 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.190 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:21:32.190 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.906522+0000", 2024-09-21T12:21:32.190 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.099e-06 2024-09-21T12:21:32.190 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.190 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.190 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:21:32.190 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.906549+0000", 2024-09-21T12:21:32.190 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.6341000000000001e-05 2024-09-21T12:21:32.190 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.191 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.191 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:21:32.191 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.906561+0000", 2024-09-21T12:21:32.191 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.204e-05 2024-09-21T12:21:32.191 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.191 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.191 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:21:32.191 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.906568+0000", 2024-09-21T12:21:32.191 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5680000000000002e-06 2024-09-21T12:21:32.191 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.191 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.191 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:21:32.191 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.906622+0000", 2024-09-21T12:21:32.191 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3551000000000003e-05 2024-09-21T12:21:32.192 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.192 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.192 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:21:32.192 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.906718+0000", 2024-09-21T12:21:32.192 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6356999999999999e-05 2024-09-21T12:21:32.192 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.192 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.192 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,4", 2024-09-21T12:21:32.192 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.906838+0000", 2024-09-21T12:21:32.192 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011982800000000001 2024-09-21T12:21:32.192 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.192 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.192 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:21:32.192 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.907781+0000", 2024-09-21T12:21:32.192 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00094262800000000004 2024-09-21T12:21:32.193 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.193 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.193 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.193 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.908717+0000", 2024-09-21T12:21:32.193 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00093601100000000005 2024-09-21T12:21:32.193 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.193 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.193 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.193 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.920070+0000", 2024-09-21T12:21:32.193 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.011353243000000001 2024-09-21T12:21:32.193 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.193 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.193 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:21:32.194 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.920126+0000", 2024-09-21T12:21:32.194 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5837000000000001e-05 2024-09-21T12:21:32.194 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.194 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.194 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:21:32.194 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.920169+0000", 2024-09-21T12:21:32.194 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.2694999999999997e-05 2024-09-21T12:21:32.194 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.194 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:21:32.194 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.194 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.194 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.194 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:4237 7.6 7:6abf58f9:::benchmark_data_smithi114_78432_object4236:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-21T12:21:32.194 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:31.933684+0000", 2024-09-21T12:21:32.194 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.13970060100000001, 2024-09-21T12:21:32.195 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.013471449999999999, 2024-09-21T12:21:32.195 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:21:32.195 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:21:32.195 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:21:32.195 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:21:32.195 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:21:32.195 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:21:32.195 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 4237 2024-09-21T12:21:32.195 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.195 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:21:32.195 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.195 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:21:32.195 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.933684+0000", 2024-09-21T12:21:32.195 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.195 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.196 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.196 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:21:32.196 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.933684+0000", 2024-09-21T12:21:32.196 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:32.196 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.196 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.196 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:21:32.196 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.933685+0000", 2024-09-21T12:21:32.196 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0669999999999999e-06 2024-09-21T12:21:32.196 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.196 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.196 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:21:32.196 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.933711+0000", 2024-09-21T12:21:32.196 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5890000000000001e-05 2024-09-21T12:21:32.196 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.197 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.197 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:21:32.197 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.933720+0000", 2024-09-21T12:21:32.197 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.3319999999999998e-06 2024-09-21T12:21:32.197 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.197 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.197 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:21:32.197 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.933726+0000", 2024-09-21T12:21:32.197 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4529999999999999e-06 2024-09-21T12:21:32.197 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.197 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.197 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:21:32.197 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.934746+0000", 2024-09-21T12:21:32.197 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0010196440000000001 2024-09-21T12:21:32.197 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.197 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.198 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:21:32.198 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.934849+0000", 2024-09-21T12:21:32.198 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000103057 2024-09-21T12:21:32.198 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.198 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.198 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:21:32.198 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.934970+0000", 2024-09-21T12:21:32.198 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012079 2024-09-21T12:21:32.198 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.198 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.198 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:21:32.198 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.939694+0000", 2024-09-21T12:21:32.198 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0047238649999999998 2024-09-21T12:21:32.198 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.199 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.199 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.199 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.939725+0000", 2024-09-21T12:21:32.199 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.1440999999999999e-05 2024-09-21T12:21:32.199 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.199 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.199 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:32.199 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.947073+0000", 2024-09-21T12:21:32.199 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0073473820000000004 2024-09-21T12:21:32.199 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.199 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.199 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:21:32.199 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.947118+0000", 2024-09-21T12:21:32.199 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5495999999999999e-05 2024-09-21T12:21:32.199 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:32.199 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:32.199 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:21:32.200 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:31.947155+0000", 2024-09-21T12:21:32.200 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.7033000000000003e-05 2024-09-21T12:21:32.200 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.200 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:21:32.200 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.200 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:32.200 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:21:32.200 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:21:32.274 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:32.310 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:32.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:32.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:32.412 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:32.449 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:32.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:32.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:32.525 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:32 smithi114 bash[78604]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-21T12:21:32.551 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:32.580 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:32.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:32.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:32.619 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 2 16 6418 6402 200.047 186.125 0.00308776 0.00498499 2024-09-21T12:21:32.682 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:32.723 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:32.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:32.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:32.825 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:21:32.854 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:32.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:32.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:21:32.915 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:21:32 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:21:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:21:32.933 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-21T12:21:32.956 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:32.996 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:32.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:32.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:33.098 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:33.129 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:33.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:33.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:33.231 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:33.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:32 smithi183 ceph-mon[33277]: pgmap v39: 148 pgs: 148 active+clean; 203 MiB data, 515 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 34 MiB/s wr, 541 op/s 2024-09-21T12:21:33.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:32 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1157051036' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:21:33.251 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:21:33.269 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:33.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:33.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:33.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:32 smithi114 ceph-mon[30324]: pgmap v39: 148 pgs: 148 active+clean; 203 MiB data, 515 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 34 MiB/s wr, 541 op/s 2024-09-21T12:21:33.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:32 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1157051036' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:21:33.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:32 smithi114 ceph-mon[40867]: pgmap v39: 148 pgs: 148 active+clean; 203 MiB data, 515 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 34 MiB/s wr, 541 op/s 2024-09-21T12:21:33.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:32 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1157051036' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:21:33.371 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:33.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:33.418 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:33.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:33.519 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:33.564 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:33.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:33.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:33.619 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 3 16 9170 9154 190.685 172 0.00294436 0.00523611 2024-09-21T12:21:33.667 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:33.703 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:33.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:33.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:33.805 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:21:33.838 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:33.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:33.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:21:33.940 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:33.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:33.975 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:33.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:33.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:33 smithi183 ceph-mon[33277]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:21:33.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:33 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1157051036' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:21:33.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:33 smithi183 ceph-mon[33277]: osdmap e89: 8 total, 8 up, 8 in 2024-09-21T12:21:34.006 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T12:21:34.007 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:21:34.007 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T12:21:34.007 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd out 4 2024-09-21T12:21:34.076 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:34.109 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:34.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:34.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:34.211 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:21:34.241 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:34.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:34.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:21:34.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:33 smithi114 ceph-mon[30324]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:21:34.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:33 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1157051036' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:21:34.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:33 smithi114 ceph-mon[30324]: osdmap e89: 8 total, 8 up, 8 in 2024-09-21T12:21:34.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:33 smithi114 ceph-mon[40867]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:21:34.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:33 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1157051036' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:21:34.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:33 smithi114 ceph-mon[40867]: osdmap e89: 8 total, 8 up, 8 in 2024-09-21T12:21:34.343 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:21:34.374 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:34.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:34.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:21:34.476 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:34.515 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:34.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:34.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:34.616 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:34.620 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 4 16 12112 12096 188.969 183.875 0.00942232 0.00528459 2024-09-21T12:21:34.658 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:34.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:34.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:34.760 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:21:34.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:34.792 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:34.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:21:34.893 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:34.929 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:34.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:34.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:35.030 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:35.075 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:35.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:35.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:35.177 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:35.205 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:35.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:35.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:35.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:34 smithi183 ceph-mon[33277]: pgmap v41: 148 pgs: 148 active+clean; 203 MiB data, 515 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 34 MiB/s wr, 540 op/s 2024-09-21T12:21:35.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:34 smithi114 ceph-mon[30324]: pgmap v41: 148 pgs: 148 active+clean; 203 MiB data, 515 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 34 MiB/s wr, 540 op/s 2024-09-21T12:21:35.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:34 smithi114 ceph-mon[40867]: pgmap v41: 148 pgs: 148 active+clean; 203 MiB data, 515 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 34 MiB/s wr, 540 op/s 2024-09-21T12:21:35.307 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:35.346 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:35.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:35.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:35.448 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:35.486 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:35.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:35.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:35.569 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T12:21:35.588 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:21:35.615 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:35.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:35.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:21:35.619 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 5 15 14796 14781 184.73 167.812 0.00749322 0.00540568 2024-09-21T12:21:35.718 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:21:35.747 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:35.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:35.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:21:35.775 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:35 smithi114 bash[78604]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-21T12:21:35.849 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:35.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:35.878 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:35.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:35.980 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:36.018 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:36.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:36.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:36.120 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:21:36.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:36.151 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:36.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:21:36.252 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:36.274 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 bash[78604]: Writing manifest to image destination 2024-09-21T12:21:36.294 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:36.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:36.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:36.396 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:36.439 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:36.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:36.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:36.541 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:36.569 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:36.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:36.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:36.596 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 podman[78604]: 2024-09-21 12:21:36.313326379 +0000 UTC m=+4.726041474 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-21T12:21:36.597 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 podman[78604]: 2024-09-21 12:21:36.450799951 +0000 UTC m=+4.863515041 container create e900873487e17a09d8531d17b30f0fbb191b9f26ab1a5a399d2b6eb73ebcc78f (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T12:21:36.620 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 6 16 17401 17385 181.035 162.75 0.00497676 0.00551724 2024-09-21T12:21:36.671 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:36.711 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:36.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:36.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:36.813 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:36.843 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:36.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:36.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:36.849 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 podman[78604]: 2024-09-21 12:21:36.737293747 +0000 UTC m=+5.150008848 container init e900873487e17a09d8531d17b30f0fbb191b9f26ab1a5a399d2b6eb73ebcc78f (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T12:21:36.849 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 podman[78604]: 2024-09-21 12:21:36.746054627 +0000 UTC m=+5.158769721 container start e900873487e17a09d8531d17b30f0fbb191b9f26ab1a5a399d2b6eb73ebcc78f (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T12:21:36.849 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.753Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-21T12:21:36.849 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.753Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-21T12:21:36.849 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.754Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-21T12:21:36.849 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.754Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-21T12:21:36.849 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.754Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-21T12:21:36.849 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.754Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-21T12:21:36.849 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.755Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-21T12:21:36.849 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.755Z caller=node_exporter.go:117 level=info collector=arp 2024-09-21T12:21:36.850 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.755Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-21T12:21:36.850 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.755Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-21T12:21:36.850 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.755Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-21T12:21:36.850 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.756Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-21T12:21:36.850 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.756Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-21T12:21:36.850 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.756Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-21T12:21:36.850 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.756Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-21T12:21:36.850 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.756Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-21T12:21:36.850 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.756Z caller=node_exporter.go:117 level=info collector=edac 2024-09-21T12:21:36.850 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.756Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-21T12:21:36.850 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.756Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-21T12:21:36.850 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.756Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-21T12:21:36.850 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.756Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-21T12:21:36.851 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.756Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-21T12:21:36.851 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.756Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-21T12:21:36.851 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.756Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-21T12:21:36.851 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.756Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-21T12:21:36.851 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.756Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-21T12:21:36.851 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.756Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-21T12:21:36.851 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.756Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-21T12:21:36.851 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.756Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-21T12:21:36.851 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.756Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-21T12:21:36.851 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.756Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-21T12:21:36.851 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.756Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-21T12:21:36.851 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.756Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-21T12:21:36.851 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.756Z caller=node_exporter.go:117 level=info collector=os 2024-09-21T12:21:36.851 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.756Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-21T12:21:36.851 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.756Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-21T12:21:36.852 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.756Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-21T12:21:36.852 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.756Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-21T12:21:36.852 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.756Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-21T12:21:36.852 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.756Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-21T12:21:36.852 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.756Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-21T12:21:36.852 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.757Z caller=node_exporter.go:117 level=info collector=stat 2024-09-21T12:21:36.852 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.757Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-21T12:21:36.852 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.757Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-21T12:21:36.852 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.757Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-21T12:21:36.852 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.757Z caller=node_exporter.go:117 level=info collector=time 2024-09-21T12:21:36.852 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.757Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-21T12:21:36.852 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.757Z caller=node_exporter.go:117 level=info collector=uname 2024-09-21T12:21:36.852 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.757Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-21T12:21:36.852 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.757Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-21T12:21:36.852 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.757Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-21T12:21:36.853 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.757Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-21T12:21:36.853 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a[79166]: ts=2024-09-21T12:21:36.757Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-21T12:21:36.853 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 bash[78604]: e900873487e17a09d8531d17b30f0fbb191b9f26ab1a5a399d2b6eb73ebcc78f 2024-09-21T12:21:36.872 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:21:36 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:21:36.897 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:21:36.897 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:21:36.945 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:36.968 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:21:37.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:37.017 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:37.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:37.108 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-mon[30324]: pgmap v42: 148 pgs: 148 active+clean; 203 MiB data, 515 MiB used, 715 GiB / 715 GiB avail; 668 B/s rd, 26 MiB/s wr, 423 op/s 2024-09-21T12:21:37.108 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:37.108 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-mon[30324]: osdmap e90: 8 total, 8 up, 8 in 2024-09-21T12:21:37.109 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-mon[40867]: pgmap v42: 148 pgs: 148 active+clean; 203 MiB data, 515 MiB used, 715 GiB / 715 GiB avail; 668 B/s rd, 26 MiB/s wr, 423 op/s 2024-09-21T12:21:37.109 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:37.109 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:36 smithi114 ceph-mon[40867]: osdmap e90: 8 total, 8 up, 8 in 2024-09-21T12:21:37.109 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:21:36 smithi114 systemd[1]: Started Ceph node-exporter.a for e2ad975c-7812-11ef-baf6-efdc52797490. 2024-09-21T12:21:37.118 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:37.159 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:37.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:37.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:37.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:36 smithi183 ceph-mon[33277]: pgmap v42: 148 pgs: 148 active+clean; 203 MiB data, 515 MiB used, 715 GiB / 715 GiB avail; 668 B/s rd, 26 MiB/s wr, 423 op/s 2024-09-21T12:21:37.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:36 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:37.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:36 smithi183 ceph-mon[33277]: osdmap e90: 8 total, 8 up, 8 in 2024-09-21T12:21:37.261 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:37.300 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:37.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:37.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:37.402 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:21:37.433 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:37.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:37.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:21:37.534 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:37.566 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:37.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:37.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:37.620 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 7 16 19845 19829 176.994 152.75 0.00303744 0.00564324 2024-09-21T12:21:37.668 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:21:37.698 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:37.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:37.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:21:37.799 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:21:37.834 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:37.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:37.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:21:37.936 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:37.958 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:37.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:37.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:38.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:37 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:21:38.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:37 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:38.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:37 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:38.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:37 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:38.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:37 smithi183 ceph-mon[33277]: Deploying daemon node-exporter.b on smithi183 2024-09-21T12:21:38.039 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:38 smithi183 systemd[1]: Starting Ceph node-exporter.b for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:21:38.060 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:38.095 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:38.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:38.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:38.197 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:38.218 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:38.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:38.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:38.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:37 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:21:38.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:37 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:38.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:37 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:38.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:37 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:38.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:37 smithi114 ceph-mon[30324]: Deploying daemon node-exporter.b on smithi183 2024-09-21T12:21:38.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:37 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:21:38.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:37 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:38.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:37 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:38.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:37 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:38.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:37 smithi114 ceph-mon[40867]: Deploying daemon node-exporter.b on smithi183 2024-09-21T12:21:38.320 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:38.348 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T12:21:38.352 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:38.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:38.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:38.454 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:38.496 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:38.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:38.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:38.598 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:38.621 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 8 16 22012 21996 171.799 135.438 0.0123408 0.00581411 2024-09-21T12:21:38.624 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:38.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:38.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:38.642 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:38 smithi183 bash[64278]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-09-21T12:21:38.642 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:38 smithi183 bash[64278]: Getting image source signatures 2024-09-21T12:21:38.642 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:38 smithi183 bash[64278]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-09-21T12:21:38.726 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:38.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:38.771 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:38.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:38.872 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:21:38.901 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:38.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:38.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:21:38.981 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:38 smithi183 bash[64278]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-21T12:21:38.982 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:38 smithi183 bash[64278]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-21T12:21:39.003 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:39.030 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:39.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:39.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:39.132 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:21:39.162 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:39.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:39.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:21:39.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:38 smithi183 ceph-mon[33277]: pgmap v44: 148 pgs: 148 active+clean; 1.1 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 134 MiB/s wr, 2.15k op/s 2024-09-21T12:21:39.264 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:21:39.270 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:38 smithi114 ceph-mon[30324]: pgmap v44: 148 pgs: 148 active+clean; 1.1 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 134 MiB/s wr, 2.15k op/s 2024-09-21T12:21:39.271 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:38 smithi114 ceph-mon[40867]: pgmap v44: 148 pgs: 148 active+clean; 1.1 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 134 MiB/s wr, 2.15k op/s 2024-09-21T12:21:39.292 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:39.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:39.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:21:39.393 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:39.422 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:39.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:39.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:39.524 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:39 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-a[30301]: 2024-09-21T12:21:39.267+0000 7f49fe685640 -1 mon.a@0(leader).osd e90 definitely_dead 0 2024-09-21T12:21:39.564 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:39.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:39.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:39.621 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 9 16 23716 23700 164.543 106.5 0.0178991 0.00606985 2024-09-21T12:21:39.666 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:39.700 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:39.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:39.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:39.802 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:39.842 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:39.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:39.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:39.944 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:39.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:39.995 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:39.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:40.096 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:40.117 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:40.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:40.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:40.219 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:21:40.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:39 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1416793655' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T12:21:40.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:39 smithi183 ceph-mon[33277]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-21T12:21:40.240 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:40.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:40.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:21:40.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:39 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1416793655' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T12:21:40.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:39 smithi114 ceph-mon[40867]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-21T12:21:40.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:39 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1416793655' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T12:21:40.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:39 smithi114 ceph-mon[30324]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-21T12:21:40.341 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:40.362 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:40.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:40.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:40.463 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:40.484 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:40.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:40.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:40.586 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:40.608 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:40.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:40.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:40.620 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 10 16 24423 24407 152.509 44.1875 0.00242914 0.00616415 2024-09-21T12:21:40.709 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:40.732 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:40.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:40.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:40.738 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:40 smithi183 bash[64278]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-21T12:21:40.738 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:40 smithi183 bash[64278]: Writing manifest to image destination 2024-09-21T12:21:40.834 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:40.860 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:40.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:40.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:40.962 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:40.988 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:40.988 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:21:40.988 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:21:40.988 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 0 2024-09-21T12:21:40.988 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:21:40.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:40.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:41.090 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:41.113 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:41.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:41.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:41.214 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:41.236 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:41.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:41.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:41.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:41 smithi114 ceph-mon[40867]: pgmap v45: 148 pgs: 148 active+clean; 1.1 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 109 MiB/s wr, 1.75k op/s 2024-09-21T12:21:41.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:41 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1416793655' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-21T12:21:41.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:41 smithi114 ceph-mon[40867]: osdmap e91: 8 total, 8 up, 7 in 2024-09-21T12:21:41.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:41 smithi114 ceph-mon[30324]: pgmap v45: 148 pgs: 148 active+clean; 1.1 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 109 MiB/s wr, 1.75k op/s 2024-09-21T12:21:41.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:41 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1416793655' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-21T12:21:41.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:41 smithi114 ceph-mon[30324]: osdmap e91: 8 total, 8 up, 7 in 2024-09-21T12:21:41.339 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:41.361 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:41.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:41.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:41.462 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:41.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:41.486 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:41.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:41.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-mon[33277]: pgmap v45: 148 pgs: 148 active+clean; 1.1 GiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 109 MiB/s wr, 1.75k op/s 2024-09-21T12:21:41.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1416793655' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-21T12:21:41.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-mon[33277]: osdmap e91: 8 total, 8 up, 7 in 2024-09-21T12:21:41.587 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:41.610 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:41.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:41.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:41.621 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 11 16 24423 24407 138.646 0 - 0.00616415 2024-09-21T12:21:41.711 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:41.737 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:41.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:41.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:41.839 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:41.864 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:41.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:41.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:41.868 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 podman[64278]: 2024-09-21 12:21:41.560338393 +0000 UTC m=+3.375370785 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-21T12:21:41.868 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 podman[64278]: 2024-09-21 12:21:41.650903424 +0000 UTC m=+3.465935861 container create 03eac822cd8878e9b4cff3808345c4315c1205d493e36c65894ba389e69633c4 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T12:21:41.868 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 podman[64278]: 2024-09-21 12:21:41.861445913 +0000 UTC m=+3.676478309 container init 03eac822cd8878e9b4cff3808345c4315c1205d493e36c65894ba389e69633c4 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T12:21:41.868 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 podman[64278]: 2024-09-21 12:21:41.866865023 +0000 UTC m=+3.681897415 container start 03eac822cd8878e9b4cff3808345c4315c1205d493e36c65894ba389e69633c4 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T12:21:41.965 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:41.991 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:41.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:41.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:42.093 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:42.116 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:42.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:42.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:42.217 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:42.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:42 smithi183 ceph-mon[33277]: osdmap e92: 8 total, 8 up, 7 in 2024-09-21T12:21:42.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:42 smithi183 ceph-mon[33277]: osdmap e93: 8 total, 8 up, 7 in 2024-09-21T12:21:42.239 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.870Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-21T12:21:42.239 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.870Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-21T12:21:42.239 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.871Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-21T12:21:42.239 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.871Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-21T12:21:42.239 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.871Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-21T12:21:42.239 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.871Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-21T12:21:42.239 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-21T12:21:42.239 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=arp 2024-09-21T12:21:42.239 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-21T12:21:42.239 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-21T12:21:42.240 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-21T12:21:42.240 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-21T12:21:42.240 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-21T12:21:42.240 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-21T12:21:42.240 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-21T12:21:42.240 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-21T12:21:42.240 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=edac 2024-09-21T12:21:42.240 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-21T12:21:42.241 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-21T12:21:42.241 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-21T12:21:42.241 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-21T12:21:42.241 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-21T12:21:42.241 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-21T12:21:42.241 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-21T12:21:42.241 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-21T12:21:42.241 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-21T12:21:42.241 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-21T12:21:42.241 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-21T12:21:42.241 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-21T12:21:42.241 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-21T12:21:42.241 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-21T12:21:42.242 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-21T12:21:42.242 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-21T12:21:42.242 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=os 2024-09-21T12:21:42.242 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-21T12:21:42.242 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-21T12:21:42.242 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-21T12:21:42.242 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-21T12:21:42.242 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-21T12:21:42.242 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-21T12:21:42.242 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-21T12:21:42.242 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=stat 2024-09-21T12:21:42.242 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-21T12:21:42.242 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-21T12:21:42.243 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-21T12:21:42.243 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=time 2024-09-21T12:21:42.243 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-21T12:21:42.243 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=uname 2024-09-21T12:21:42.243 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-21T12:21:42.243 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-21T12:21:42.243 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-21T12:21:42.243 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-21T12:21:42.243 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b[64538]: ts=2024-09-21T12:21:41.872Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-21T12:21:42.243 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 bash[64278]: 03eac822cd8878e9b4cff3808345c4315c1205d493e36c65894ba389e69633c4 2024-09-21T12:21:42.243 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:21:41 smithi183 systemd[1]: Started Ceph node-exporter.b for e2ad975c-7812-11ef-baf6-efdc52797490. 2024-09-21T12:21:42.244 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:42.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:42.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:42.346 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:21:42.348 INFO:teuthology.orchestra.run.smithi114.stderr:marked out osd.4. 2024-09-21T12:21:42.349 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:42 smithi114 ceph-mon[30324]: osdmap e92: 8 total, 8 up, 7 in 2024-09-21T12:21:42.349 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:42 smithi114 ceph-mon[30324]: osdmap e93: 8 total, 8 up, 7 in 2024-09-21T12:21:42.349 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:42 smithi114 ceph-mon[40867]: osdmap e92: 8 total, 8 up, 7 in 2024-09-21T12:21:42.350 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:42 smithi114 ceph-mon[40867]: osdmap e93: 8 total, 8 up, 7 in 2024-09-21T12:21:42.375 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:42.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:42.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:21:42.477 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:42.498 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:42.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:42.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:42.599 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:42.622 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 12 16 24856 24840 129.348 13.5312 0.022071 0.0077226 2024-09-21T12:21:42.635 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:42.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:42.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:42.737 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:42.770 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:42.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:42.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:42.774 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:21:42 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:21:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:21:42.872 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:21:42.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:42.902 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:42.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:21:43.003 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:43.028 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:43.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:43.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:43.130 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:43.155 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:43.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:43.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:43.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:43 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:21:43.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:43 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:21:43.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:43 smithi183 ceph-mon[33277]: pgmap v48: 148 pgs: 5 remapped+peering, 143 active+clean; 1.5 GiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 341 B/s rd, 220 MiB/s wr, 3.53k op/s; 276 B/s, 13 objects/s recovering 2024-09-21T12:21:43.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:43 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:43.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:43 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:43.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:43 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:43.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:43 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:43.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:43 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:43.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:43 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:21:43.257 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:43.287 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:43.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:43.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:43.388 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:43.432 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:43.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:43.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:43.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:43 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:21:43.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:43 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:21:43.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:43 smithi114 ceph-mon[40867]: pgmap v48: 148 pgs: 5 remapped+peering, 143 active+clean; 1.5 GiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 341 B/s rd, 220 MiB/s wr, 3.53k op/s; 276 B/s, 13 objects/s recovering 2024-09-21T12:21:43.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:43 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:43.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:43 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:43.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:43 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:43.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:43 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:43.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:43 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:43.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:43 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:21:43.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:43 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:21:43.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:43 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:21:43.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:43 smithi114 ceph-mon[30324]: pgmap v48: 148 pgs: 5 remapped+peering, 143 active+clean; 1.5 GiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 341 B/s rd, 220 MiB/s wr, 3.53k op/s; 276 B/s, 13 objects/s recovering 2024-09-21T12:21:43.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:43 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:43.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:43 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:43.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:43 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:43.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:43 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:43.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:43 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:43.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:43 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:21:43.534 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:43.562 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:43.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:43.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:43.621 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 13 16 26343 26327 126.547 92.9375 0.00323247 0.00789482 2024-09-21T12:21:43.665 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:43.690 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:43.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:43.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:43.792 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:43.817 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:43.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:43.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:43.918 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:43.959 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:43.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:43.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:44.051 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T12:21:44.061 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:44.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:44.134 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:44.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:44.235 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:21:44.258 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:44.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:44.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:21:44.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:44 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:21:44.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:44 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:21:44.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:44 smithi114 ceph-mon[40867]: osdmap e94: 8 total, 8 up, 7 in 2024-09-21T12:21:44.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:44 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:21:44.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:44 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:21:44.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:44 smithi114 ceph-mon[30324]: osdmap e94: 8 total, 8 up, 7 in 2024-09-21T12:21:44.360 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:44.407 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:44.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:44.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:44.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:44 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:21:44.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:44 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:21:44.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:44 smithi183 ceph-mon[33277]: osdmap e94: 8 total, 8 up, 7 in 2024-09-21T12:21:44.510 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:21:44.538 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:44.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:44.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:21:44.621 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 14 16 27581 27565 123.035 77.375 0.0088911 0.00812019 2024-09-21T12:21:44.640 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:44.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:44.692 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:44.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:44.768 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:21:44.768 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:21:44.793 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:21:44.807 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:21:44.807 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:21:44.818 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:44.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:44.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:21:44.830 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:21:44.837 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:21:44.852 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:21:44.853 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:21:44.892 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:21:44.893 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:21:44.919 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:21:44.925 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:21:44.925 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:21:44.945 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:44.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:44.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:21:44.959 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:21:44.959 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:21:44.994 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:21:44.994 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:21:45.025 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:21:45.026 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:21:45.047 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:45.105 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:45.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:45.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:45.207 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:45.232 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:45.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:45.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:45.334 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:45.379 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:45.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:45.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:45.480 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:45.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:45 smithi114 ceph-mon[40867]: pgmap v51: 148 pgs: 5 remapped+peering, 143 active+clean; 1.5 GiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 112 MiB/s wr, 1.80k op/s; 414 B/s, 20 objects/s recovering 2024-09-21T12:21:45.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:45 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1013885047' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:21:45.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:45 smithi114 ceph-mon[30324]: pgmap v51: 148 pgs: 5 remapped+peering, 143 active+clean; 1.5 GiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 112 MiB/s wr, 1.80k op/s; 414 B/s, 20 objects/s recovering 2024-09-21T12:21:45.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:45 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1013885047' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:21:45.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:45 smithi183 ceph-mon[33277]: pgmap v51: 148 pgs: 5 remapped+peering, 143 active+clean; 1.5 GiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 112 MiB/s wr, 1.80k op/s; 414 B/s, 20 objects/s recovering 2024-09-21T12:21:45.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:45 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1013885047' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:21:45.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:45.505 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:45.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:45.607 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:45.621 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 15 16 28796 28780 119.894 75.9375 0.0114181 0.00833128 2024-09-21T12:21:45.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:45.670 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:45.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:45.771 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:45.811 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:45.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:45.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:45.913 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:45.969 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:45.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:45.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:46.070 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:46.118 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:46.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:46.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:46.220 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:46.221 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:21:46.221 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:21:46.261 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:46.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:46.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:46.363 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:46.406 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:46.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:46.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:46.508 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:46.535 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:46.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:46.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:46.621 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 16 15 30104 30089 117.514 81.8125 0.0252193 0.0085025 2024-09-21T12:21:46.637 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:46.674 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:46.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:46.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:46.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:21:46 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:21:46.775 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:21:46.801 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:46.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:46.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:21:46.903 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:46.945 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T12:21:46.960 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:46.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:46.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:47.062 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:47.108 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:47.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:47.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:47.210 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:47.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:46 smithi183 ceph-mon[33277]: pgmap v52: 148 pgs: 5 remapped+peering, 143 active+clean; 1.5 GiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 81 MiB/s wr, 1.29k op/s; 298 B/s, 14 objects/s recovering 2024-09-21T12:21:47.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:46 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:47.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:21:47.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:46 smithi114 ceph-mon[30324]: pgmap v52: 148 pgs: 5 remapped+peering, 143 active+clean; 1.5 GiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 81 MiB/s wr, 1.29k op/s; 298 B/s, 14 objects/s recovering 2024-09-21T12:21:47.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:46 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:47.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:21:47.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:46 smithi114 ceph-mon[40867]: pgmap v52: 148 pgs: 5 remapped+peering, 143 active+clean; 1.5 GiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 81 MiB/s wr, 1.29k op/s; 298 B/s, 14 objects/s recovering 2024-09-21T12:21:47.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:46 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:47.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:21:47.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:47.289 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:47.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:47.390 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:47.418 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:47.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:47.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:47.520 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:47.547 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:47.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:47.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:47.621 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 17 16 31413 31397 115.41 81.75 0.00383132 0.0086568 2024-09-21T12:21:47.649 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:47.673 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:47.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:47.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:47.775 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:21:47.800 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:47.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:47.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:21:47.902 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:47.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:47.956 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:47.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:48.057 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:48.121 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:48.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:48.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:48.183 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:21:48.223 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:21:48.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:47 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:21:48.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:47 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:48.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:47 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:48.249 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:48.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:48.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:21:48.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:47 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:21:48.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:47 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:48.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:47 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:48.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:47 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:21:48.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:47 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:48.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:47 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:48.350 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:48.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:48.403 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:48.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:48.504 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:48.550 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:48.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:48.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:48.622 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 18 16 32566 32550 113.001 72.0625 0.0180976 0.00884374 2024-09-21T12:21:48.655 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:21:48.681 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:48.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:48.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:21:48.782 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:48.808 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:48.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:48.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:48.871 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:21:48.871 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:21:48.910 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:48.911 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-21T12:21:48.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:48.960 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:48.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:49.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:48 smithi114 ceph-mon[40867]: pgmap v53: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 1.8 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 312 B/s rd, 122 MiB/s wr, 1.95k op/s; 2/90621 objects degraded (0.002%); 6244/90621 objects misplaced (6.890%); 13 MiB/s, 225 objects/s recovering 2024-09-21T12:21:49.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:48 smithi114 ceph-mon[40867]: Health check failed: Degraded data redundancy: 2/90621 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T12:21:49.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:48 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2223239994' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:21:49.027 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:48 smithi114 ceph-mon[30324]: pgmap v53: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 1.8 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 312 B/s rd, 122 MiB/s wr, 1.95k op/s; 2/90621 objects degraded (0.002%); 6244/90621 objects misplaced (6.890%); 13 MiB/s, 225 objects/s recovering 2024-09-21T12:21:49.027 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:48 smithi114 ceph-mon[30324]: Health check failed: Degraded data redundancy: 2/90621 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T12:21:49.027 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:48 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2223239994' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:21:49.061 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:49.101 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:49.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:49.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:49.202 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:49.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:48 smithi183 ceph-mon[33277]: pgmap v53: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 1.8 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 312 B/s rd, 122 MiB/s wr, 1.95k op/s; 2/90621 objects degraded (0.002%); 6244/90621 objects misplaced (6.890%); 13 MiB/s, 225 objects/s recovering 2024-09-21T12:21:49.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:48 smithi183 ceph-mon[33277]: Health check failed: Degraded data redundancy: 2/90621 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T12:21:49.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:48 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2223239994' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:21:49.247 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:49.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:49.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:49.349 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:49.397 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:49.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:49.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:49.480 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T12:21:49.480 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:21:49.481 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-21T12:21:49.481 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:21:49.499 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:49.523 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:49.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:49.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:49.623 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 19 16 33832 33816 111.209 79.125 0.0258019 0.00898006 2024-09-21T12:21:49.625 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:49.722 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:49.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:49.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:49.824 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:49.871 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:49.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:49.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:49.896 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:49 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:21:49.896 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:49 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:21:49.896 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:49 smithi114 ceph-mon[30324]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:21:49.896 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:49 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2223239994' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:21:49.955 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:21:49.972 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:50.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:50.052 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:50.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:50.153 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:50.161 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:49 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:21:50.161 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:49 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:21:50.162 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:49 smithi114 ceph-mon[40867]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:21:50.162 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:49 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2223239994' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:21:50.162 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:49 smithi114 ceph-mon[40867]: osdmap e95: 8 total, 8 up, 7 in 2024-09-21T12:21:50.162 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:49 smithi114 ceph-mon[30324]: osdmap e95: 8 total, 8 up, 7 in 2024-09-21T12:21:50.204 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:50.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:50.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:50.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:49 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:21:50.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:49 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:21:50.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:49 smithi183 ceph-mon[33277]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:21:50.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:49 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2223239994' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:21:50.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:49 smithi183 ceph-mon[33277]: osdmap e95: 8 total, 8 up, 7 in 2024-09-21T12:21:50.306 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:50.332 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:50.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:50.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:50.434 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:50.472 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:50.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:50.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:50.575 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:21:50.599 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:50.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:50.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:21:50.623 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-21T12:21:50.623018+0000 min lat: 0.00135771 max lat: 2.08441 avg lat: 0.00913313 2024-09-21T12:21:50.623 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T12:21:50.623 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 20 15 35032 35017 109.402 75.0625 0.0116156 0.00913313 2024-09-21T12:21:50.625 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T12:21:50.702 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:50.725 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:50.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:50.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:50.827 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:50.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:50.873 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:50.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:50.974 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:50.996 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:50.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:50.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:51.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:50 smithi114 ceph-mon[30324]: pgmap v55: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 1.8 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 677 B/s rd, 46 MiB/s wr, 741 op/s; 2/90621 objects degraded (0.002%); 6244/90621 objects misplaced (6.890%); 12 MiB/s, 185 objects/s recovering 2024-09-21T12:21:51.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:50 smithi114 ceph-mon[40867]: pgmap v55: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 1.8 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 677 B/s rd, 46 MiB/s wr, 741 op/s; 2/90621 objects degraded (0.002%); 6244/90621 objects misplaced (6.890%); 12 MiB/s, 185 objects/s recovering 2024-09-21T12:21:51.098 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:51.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:51.152 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:51.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:51.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:50 smithi183 ceph-mon[33277]: pgmap v55: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 1.8 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 677 B/s rd, 46 MiB/s wr, 741 op/s; 2/90621 objects degraded (0.002%); 6244/90621 objects misplaced (6.890%); 12 MiB/s, 185 objects/s recovering 2024-09-21T12:21:51.254 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:51.279 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:51.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:51.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:51.381 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:51.422 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:51.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:51.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:51.524 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:51.551 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:21:51.576 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:21:51.577 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:21:51.608 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:51.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:51.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:51.624 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 21 16 36177 36161 107.594 71.5 0.0139163 0.0092888 2024-09-21T12:21:51.710 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:51.750 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:51.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:51.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:51.852 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:51.891 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:51.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:51.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:51.993 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:52.038 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:52.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:52.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:52.090 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T12:21:52.139 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:21:52.164 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:52.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:52.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:21:52.265 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:52.289 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:52.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:52.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:52.391 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:52.414 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:52.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:52.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:52.516 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:52.564 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:52.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:52.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:52.627 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 22 16 37347 37331 106.026 73.125 0.0109124 0.00942596 2024-09-21T12:21:52.660 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:21:52.660 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:21:52.667 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:52.691 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:21:52 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:21:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:21:52.692 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:52.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:52.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:52.694 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:21:52.694 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:21:52.732 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:21:52.732 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:21:52.765 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:21:52.765 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:21:52.794 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:52.796 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:21:52.796 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:21:52.820 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:52.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:52.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:52.830 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:21:52.830 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:21:52.866 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:21:52.866 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:21:52.901 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:21:52.901 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:21:52.921 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:52.943 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:52 smithi114 ceph-mon[40867]: pgmap v56: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 2.0 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 2/99975 objects degraded (0.002%); 5958/99975 objects misplaced (5.959%); 12 MiB/s, 198 objects/s recovering 2024-09-21T12:21:52.943 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:52 smithi114 ceph-mon[40867]: osdmap e96: 8 total, 8 up, 7 in 2024-09-21T12:21:52.944 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:52 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1090449211' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:21:52.944 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:52 smithi114 ceph-mon[30324]: pgmap v56: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 2.0 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 2/99975 objects degraded (0.002%); 5958/99975 objects misplaced (5.959%); 12 MiB/s, 198 objects/s recovering 2024-09-21T12:21:52.944 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:52 smithi114 ceph-mon[30324]: osdmap e96: 8 total, 8 up, 7 in 2024-09-21T12:21:52.944 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:52 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1090449211' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:21:52.963 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:52.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:52.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:53.065 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:53.112 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:53.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:53.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:53.213 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:53.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:52 smithi183 ceph-mon[33277]: pgmap v56: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 2.0 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 2/99975 objects degraded (0.002%); 5958/99975 objects misplaced (5.959%); 12 MiB/s, 198 objects/s recovering 2024-09-21T12:21:53.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:52 smithi183 ceph-mon[33277]: osdmap e96: 8 total, 8 up, 7 in 2024-09-21T12:21:53.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:52 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1090449211' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:21:53.257 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:53.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:53.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:53.359 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:53.383 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:53.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:53.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:53.485 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:53.529 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:53.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:53.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:53.624 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 23 16 38609 38593 104.845 78.875 0.00817294 0.00953029 2024-09-21T12:21:53.631 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:53.657 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:53.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:53.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:53.759 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:53.783 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:53.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:53.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:53.885 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:53.935 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:53.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:53.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:54.028 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:53 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:21:54.028 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:53 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:21:54.028 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:53 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:54.028 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:53 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:54.028 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:53 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:21:54.028 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:53 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:21:54.028 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:53 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:54.029 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:53 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:21:54.029 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:53 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:21:54.029 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:53 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:54.029 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:53 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:54.029 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:53 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:21:54.029 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:53 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:21:54.029 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:53 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:54.037 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:54.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:54.091 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:54.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:54.192 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:54.233 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:21:54.234 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-21T12:21:54.234 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:54.234 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:39215 7.e 7:79e59aca:::benchmark_data_smithi114_78432_object39214:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-21T12:21:54.234 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:54.222909+0000", 2024-09-21T12:21:54.234 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0032883669999999999, 2024-09-21T12:21:54.234 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0033242879999999999, 2024-09-21T12:21:54.234 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:21:54.234 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:21:54.234 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:21:54.234 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:21:54.234 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:21:54.234 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:21:54.234 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 39215 2024-09-21T12:21:54.235 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:54.235 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:21:54.235 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:54.235 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:21:54.235 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:54.222909+0000", 2024-09-21T12:21:54.235 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:54.235 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:54.235 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:54.235 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:21:54.235 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:54.222909+0000", 2024-09-21T12:21:54.235 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:21:54.235 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:54.235 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:54.235 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:21:54.235 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:54.222911+0000", 2024-09-21T12:21:54.236 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.618e-06 2024-09-21T12:21:54.236 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:54.236 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:54.236 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:21:54.236 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:54.222937+0000", 2024-09-21T12:21:54.236 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5854e-05 2024-09-21T12:21:54.236 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:54.236 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:54.236 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:21:54.236 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:54.222949+0000", 2024-09-21T12:21:54.236 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2041e-05 2024-09-21T12:21:54.236 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:54.236 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:54.236 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:21:54.237 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:54.222961+0000", 2024-09-21T12:21:54.237 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1843e-05 2024-09-21T12:21:54.237 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:54.237 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:54.237 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:21:54.237 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:54.223257+0000", 2024-09-21T12:21:54.237 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00029570699999999998 2024-09-21T12:21:54.237 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:54.237 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:54.237 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:21:54.237 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:54.223379+0000", 2024-09-21T12:21:54.237 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012294699999999999 2024-09-21T12:21:54.237 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:54.237 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:54.237 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,5", 2024-09-21T12:21:54.238 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:54.223522+0000", 2024-09-21T12:21:54.238 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000142886 2024-09-21T12:21:54.238 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:54.238 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:54.238 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:21:54.238 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:54.225125+0000", 2024-09-21T12:21:54.238 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001602839 2024-09-21T12:21:54.238 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:21:54.238 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:21:54.238 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:21:54.238 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:54.225159+0000", 2024-09-21T12:21:54.238 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.3488999999999997e-05 2024-09-21T12:21:54.238 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:54.238 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:21:54.238 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:54.239 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:21:54.239 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:21:54.239 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 1 2024-09-21T12:21:54.239 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:21:54.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:53 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:21:54.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:53 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:21:54.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:53 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:54.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:53 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:54.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:53 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:21:54.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:53 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:21:54.240 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:53 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:21:54.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:54.250 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:54.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:54.328 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:21:54.337 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:21:54.352 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:54.394 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:54.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:54.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:54.496 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:54.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:54.541 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:54.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:54.624 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 24 16 39713 39697 103.352 69 0.00847053 0.0096697 2024-09-21T12:21:54.642 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:54.667 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:54.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:54.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:54.769 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:21:54.798 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:54.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:54.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:21:54.900 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:21:54.926 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:54.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:54.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:21:54.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:54 smithi114 ceph-mon[40867]: pgmap v58: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 2.2 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.47k op/s; 2/108975 objects degraded (0.002%); 4467/108975 objects misplaced (4.099%); 25 MiB/s, 396 objects/s recovering 2024-09-21T12:21:54.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:54 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 2/108975 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T12:21:54.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:54 smithi114 ceph-mon[40867]: osdmap e97: 8 total, 8 up, 7 in 2024-09-21T12:21:54.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:54 smithi114 ceph-mon[40867]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T12:21:54.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:54 smithi114 ceph-mon[40867]: Reconfiguring daemon alertmanager.a on smithi114 2024-09-21T12:21:54.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:54 smithi114 ceph-mon[30324]: pgmap v58: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 2.2 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.47k op/s; 2/108975 objects degraded (0.002%); 4467/108975 objects misplaced (4.099%); 25 MiB/s, 396 objects/s recovering 2024-09-21T12:21:54.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:54 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 2/108975 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T12:21:54.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:54 smithi114 ceph-mon[30324]: osdmap e97: 8 total, 8 up, 7 in 2024-09-21T12:21:54.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:54 smithi114 ceph-mon[30324]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T12:21:54.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:54 smithi114 ceph-mon[30324]: Reconfiguring daemon alertmanager.a on smithi114 2024-09-21T12:21:55.028 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:21:55.055 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:55.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:55.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:21:55.157 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:55.202 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:55.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:55.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:55.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:54 smithi183 ceph-mon[33277]: pgmap v58: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 2.2 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.47k op/s; 2/108975 objects degraded (0.002%); 4467/108975 objects misplaced (4.099%); 25 MiB/s, 396 objects/s recovering 2024-09-21T12:21:55.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:54 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 2/108975 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T12:21:55.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:54 smithi183 ceph-mon[33277]: osdmap e97: 8 total, 8 up, 7 in 2024-09-21T12:21:55.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:54 smithi183 ceph-mon[33277]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T12:21:55.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:54 smithi183 ceph-mon[33277]: Reconfiguring daemon alertmanager.a on smithi114 2024-09-21T12:21:55.304 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:55.330 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:55.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:55.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:55.432 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:55.471 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:55.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:55.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:55.573 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:55.612 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:55.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:55.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:55.624 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 25 16 41058 41042 102.58 84.0625 0.0124007 0.00974266 2024-09-21T12:21:55.714 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:55.754 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:55.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:55.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:55.856 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:55.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:55.878 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:55.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:55.979 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:21:56.007 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:56.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:56.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:21:56.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:55 smithi114 ceph-mon[40867]: from='client.24610 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:21:56.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:55 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:21:56.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:55 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:21:56.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:55 smithi114 ceph-mon[30324]: from='client.24610 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:21:56.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:55 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:21:56.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:55 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:21:56.108 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:56.136 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:56.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:56.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:56.237 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:56.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:55 smithi183 ceph-mon[33277]: from='client.24610 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:21:56.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:55 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:21:56.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:55 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:21:56.275 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:56.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:56.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:56.377 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:56.416 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:56.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:56.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:56.518 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:21:56.545 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:56.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:56.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:21:56.625 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 26 16 42199 42183 101.377 71.3125 0.0198112 0.00985845 2024-09-21T12:21:56.646 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:56.690 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:56.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:56.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:56.792 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:21:56.817 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:56.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:56.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:21:56.919 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:56.961 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:56.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:56.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:56.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:21:56 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:21:56.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:56 smithi183 ceph-mon[33277]: 3.16 scrub starts 2024-09-21T12:21:56.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:56 smithi183 ceph-mon[33277]: 3.16 scrub ok 2024-09-21T12:21:56.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:56 smithi183 ceph-mon[33277]: pgmap v60: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 2.2 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 769 B/s rd, 57 MiB/s wr, 919 op/s; 2/108975 objects degraded (0.002%); 4467/108975 objects misplaced (4.099%); 17 MiB/s, 266 objects/s recovering 2024-09-21T12:21:56.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:56 smithi183 ceph-mon[33277]: osdmap e98: 8 total, 8 up, 7 in 2024-09-21T12:21:57.063 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:57.089 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:57.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:57.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:57.191 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:57.231 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:57.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:57.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:57.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:56 smithi114 ceph-mon[40867]: 3.16 scrub starts 2024-09-21T12:21:57.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:56 smithi114 ceph-mon[40867]: 3.16 scrub ok 2024-09-21T12:21:57.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:56 smithi114 ceph-mon[40867]: pgmap v60: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 2.2 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 769 B/s rd, 57 MiB/s wr, 919 op/s; 2/108975 objects degraded (0.002%); 4467/108975 objects misplaced (4.099%); 17 MiB/s, 266 objects/s recovering 2024-09-21T12:21:57.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:56 smithi114 ceph-mon[40867]: osdmap e98: 8 total, 8 up, 7 in 2024-09-21T12:21:57.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:56 smithi114 ceph-mon[30324]: 3.16 scrub starts 2024-09-21T12:21:57.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:56 smithi114 ceph-mon[30324]: 3.16 scrub ok 2024-09-21T12:21:57.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:56 smithi114 ceph-mon[30324]: pgmap v60: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 2.2 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 769 B/s rd, 57 MiB/s wr, 919 op/s; 2/108975 objects degraded (0.002%); 4467/108975 objects misplaced (4.099%); 17 MiB/s, 266 objects/s recovering 2024-09-21T12:21:57.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:56 smithi114 ceph-mon[30324]: osdmap e98: 8 total, 8 up, 7 in 2024-09-21T12:21:57.333 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:21:57.361 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:57.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:57.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:21:57.417 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":59,"stamp":"2024-09-21T12:21:53.799011+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2368150043,"num_objects":36325,"num_object_clones":0,"num_object_copies":108975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":4467,"num_objects_unfound":0,"num_objects_dirty":36325,"num_whiteouts":0,"num_read":931,"num_read_kb":788,"num_write":36675,"num_write_kb":2313737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3260,"num_bytes_recovered":208734362,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36683,"ondisk_log_size":36683,"up":444,"acting":439,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":6802160,"kb_used_data":6490868,"kb_used_omap":10,"kb_used_meta":311157,"kb_avail":649413904,"statfs":{"total":671965249536,"available":664999837696,"internally_reserved":0,"allocated":6646648832,"data_stored":6642589608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":318624875},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":400949248,"num_objects":6118,"num_object_clones":0,"num_object_copies":18354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1777,"num_objects_unfound":0,"num_objects_dirty":6118,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6118,"num_write_kb":391552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1777,"num_bytes_recovered":116457472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.237198"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":84,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.917374+0000","last_change":"2024-09-21T12:20:15.309744+0000","last_active":"2024-09-21T12:21:48.917374+0000","last_peered":"2024-09-21T12:21:48.917374+0000","last_clean":"2024-09-21T12:21:48.917374+0000","last_became_active":"2024-09-21T12:20:15.309431+0000","last_became_peered":"2024-09-21T12:20:15.309431+0000","last_unstale":"2024-09-21T12:21:48.917374+0000","last_undegraded":"2024-09-21T12:21:48.917374+0000","last_fullsized":"2024-09-21T12:21:48.917374+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":64,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.907937+0000","last_change":"2024-09-21T12:21:41.159374+0000","last_active":"2024-09-21T12:21:48.907937+0000","last_peered":"2024-09-21T12:21:48.907937+0000","last_clean":"2024-09-21T12:21:48.907937+0000","last_became_active":"2024-09-21T12:21:41.141268+0000","last_became_peered":"2024-09-21T12:21:41.141268+0000","last_unstale":"2024-09-21T12:21:48.907937+0000","last_undegraded":"2024-09-21T12:21:48.907937+0000","last_fullsized":"2024-09-21T12:21:48.907937+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":76,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.926094+0000","last_change":"2024-09-21T12:20:19.315239+0000","last_active":"2024-09-21T12:21:48.926094+0000","last_peered":"2024-09-21T12:21:48.926094+0000","last_clean":"2024-09-21T12:21:48.926094+0000","last_became_active":"2024-09-21T12:20:19.314908+0000","last_became_peered":"2024-09-21T12:20:19.314908+0000","last_unstale":"2024-09-21T12:21:48.926094+0000","last_undegraded":"2024-09-21T12:21:48.926094+0000","last_fullsized":"2024-09-21T12:21:48.926094+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":19,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.930410+0000","last_change":"2024-09-21T12:21:41.015912+0000","last_active":"2024-09-21T12:21:48.930410+0000","last_peered":"2024-09-21T12:21:48.930410+0000","last_clean":"2024-09-21T12:21:48.930410+0000","last_became_active":"2024-09-21T12:21:41.015469+0000","last_became_peered":"2024-09-21T12:21:41.015469+0000","last_unstale":"2024-09-21T12:21:48.930410+0000","last_undegraded":"2024-09-21T12:21:48.930410+0000","last_fullsized":"2024-09-21T12:21:48.930410+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":70,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.925987+0000","last_change":"2024-09-21T12:20:21.595607+0000","last_active":"2024-09-21T12:21:48.925987+0000","last_peered":"2024-09-21T12:21:48.925987+0000","last_clean":"2024-09-21T12:21:48.925987+0000","last_became_active":"2024-09-21T12:20:21.595301+0000","last_became_peered":"2024-09-21T12:20:21.595301+0000","last_unstale":"2024-09-21T12:21:48.925987+0000","last_undegraded":"2024-09-21T12:21:48.925987+0000","last_fullsized":"2024-09-21T12:21:48.925987+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":82,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.921369+0000","last_change":"2024-09-21T12:20:15.308941+0000","last_active":"2024-09-21T12:21:48.921369+0000","last_peered":"2024-09-21T12:21:48.921369+0000","last_clean":"2024-09-21T12:21:48.921369+0000","last_became_active":"2024-09-21T12:20:15.308655+0000","last_became_peered":"2024-09-21T12:20:15.308655+0000","last_unstale":"2024-09-21T12:21:48.921369+0000","last_undegraded":"2024-09-21T12:21:48.921369+0000","last_fullsized":"2024-09-21T12:21:48.921369+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":103,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.929631+0000","last_change":"2024-09-21T12:20:17.305499+0000","last_active":"2024-09-21T12:21:48.929631+0000","last_peered":"2024-09-21T12:21:48.929631+0000","last_clean":"2024-09-21T12:21:48.929631+0000","last_became_active":"2024-09-21T12:20:17.305220+0000","last_became_peered":"2024-09-21T12:20:17.305220+0000","last_unstale":"2024-09-21T12:21:48.929631+0000","last_undegraded":"2024-09-21T12:21:48.929631+0000","last_fullsized":"2024-09-21T12:21:48.929631+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":20,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.908094+0000","last_change":"2024-09-21T12:21:41.142145+0000","last_active":"2024-09-21T12:21:48.908094+0000","last_peered":"2024-09-21T12:21:48.908094+0000","last_clean":"2024-09-21T12:21:48.908094+0000","last_became_active":"2024-09-21T12:21:41.141539+0000","last_became_peered":"2024-09-21T12:21:41.141539+0000","last_unstale":"2024-09-21T12:21:48.908094+0000","last_undegraded":"2024-09-21T12:21:48.908094+0000","last_fullsized":"2024-09-21T12:21:48.908094+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":90,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.931194+0000","last_change":"2024-09-21T12:21:41.135435+0000","last_active":"2024-09-21T12:21:48.931194+0000","last_peered":"2024-09-21T12:21:48.931194+0000","last_clean":"2024-09-21T12:21:48.931194+0000","last_became_active":"2024-09-21T12:21:41.135116+0000","last_became_peered":"2024-09-21T12:21:41.135116+0000","last_unstale":"2024-09-21T12:21:48.931194+0000","last_undegraded":"2024-09-21T12:21:48.931194+0000","last_fullsized":"2024-09-21T12:21:48.931194+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:42.163650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":65,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.920280+0000","last_change":"2024-09-21T12:21:41.039594+0000","last_active":"2024-09-21T12:21:48.920280+0000","last_peered":"2024-09-21T12:21:48.920280+0000","last_clean":"2024-09-21T12:21:48.920280+0000","last_became_active":"2024-09-21T12:21:41.030715+0000","last_became_peered":"2024-09-21T12:21:41.030715+0000","last_unstale":"2024-09-21T12:21:48.920280+0000","last_undegraded":"2024-09-21T12:21:48.920280+0000","last_fullsized":"2024-09-21T12:21:48.920280+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":75,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.930851+0000","last_change":"2024-09-21T12:20:19.308520+0000","last_active":"2024-09-21T12:21:48.930851+0000","last_peered":"2024-09-21T12:21:48.930851+0000","last_clean":"2024-09-21T12:21:48.930851+0000","last_became_active":"2024-09-21T12:20:19.308184+0000","last_became_peered":"2024-09-21T12:20:19.308184+0000","last_unstale":"2024-09-21T12:21:48.930851+0000","last_undegraded":"2024-09-21T12:21:48.930851+0000","last_fullsized":"2024-09-21T12:21:48.930851+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":72,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.927057+0000","last_change":"2024-09-21T12:20:21.309871+0000","last_active":"2024-09-21T12:21:48.927057+0000","last_peered":"2024-09-21T12:21:48.927057+0000","last_clean":"2024-09-21T12:21:48.927057+0000","last_became_active":"2024-09-21T12:20:21.309190+0000","last_became_peered":"2024-09-21T12:20:21.309190+0000","last_unstale":"2024-09-21T12:21:48.927057+0000","last_undegraded":"2024-09-21T12:21:48.927057+0000","last_fullsized":"2024-09-21T12:21:48.927057+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":90,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.930734+0000","last_change":"2024-09-21T12:21:41.138596+0000","last_active":"2024-09-21T12:21:48.930734+0000","last_peered":"2024-09-21T12:21:48.930734+0000","last_clean":"2024-09-21T12:21:48.930734+0000","last_became_active":"2024-09-21T12:21:41.138284+0000","last_became_peered":"2024-09-21T12:21:41.138284+0000","last_unstale":"2024-09-21T12:21:48.930734+0000","last_undegraded":"2024-09-21T12:21:48.930734+0000","last_fullsized":"2024-09-21T12:21:48.930734+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":55,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.931159+0000","last_change":"2024-09-21T12:21:41.036394+0000","last_active":"2024-09-21T12:21:48.931159+0000","last_peered":"2024-09-21T12:21:48.931159+0000","last_clean":"2024-09-21T12:21:48.931159+0000","last_became_active":"2024-09-21T12:21:41.030147+0000","last_became_peered":"2024-09-21T12:21:41.030147+0000","last_unstale":"2024-09-21T12:21:48.931159+0000","last_undegraded":"2024-09-21T12:21:48.931159+0000","last_fullsized":"2024-09-21T12:21:48.931159+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":82,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.919498+0000","last_change":"2024-09-21T12:21:41.018481+0000","last_active":"2024-09-21T12:21:48.919498+0000","last_peered":"2024-09-21T12:21:48.919498+0000","last_clean":"2024-09-21T12:21:48.919498+0000","last_became_active":"2024-09-21T12:21:41.018140+0000","last_became_peered":"2024-09-21T12:21:41.018140+0000","last_unstale":"2024-09-21T12:21:48.919498+0000","last_undegraded":"2024-09-21T12:21:48.919498+0000","last_fullsized":"2024-09-21T12:21:48.919498+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":71,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.930443+0000","last_change":"2024-09-21T12:20:21.311536+0000","last_active":"2024-09-21T12:21:48.930443+0000","last_peered":"2024-09-21T12:21:48.930443+0000","last_clean":"2024-09-21T12:21:48.930443+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:21:48.930443+0000","last_undegraded":"2024-09-21T12:21:48.930443+0000","last_fullsized":"2024-09-21T12:21:48.930443+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":19,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.908671+0000","last_change":"2024-09-21T12:21:41.139887+0000","last_active":"2024-09-21T12:21:48.908671+0000","last_peered":"2024-09-21T12:21:48.908671+0000","last_clean":"2024-09-21T12:21:48.908671+0000","last_became_active":"2024-09-21T12:21:41.134693+0000","last_became_peered":"2024-09-21T12:21:41.134693+0000","last_unstale":"2024-09-21T12:21:48.908671+0000","last_undegraded":"2024-09-21T12:21:48.908671+0000","last_fullsized":"2024-09-21T12:21:48.908671+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":99,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.915598+0000","last_change":"2024-09-21T12:21:41.136366+0000","last_active":"2024-09-21T12:21:48.915598+0000","last_peered":"2024-09-21T12:21:48.915598+0000","last_clean":"2024-09-21T12:21:48.915598+0000","last_became_active":"2024-09-21T12:21:41.135835+0000","last_became_peered":"2024-09-21T12:21:41.135835+0000","last_unstale":"2024-09-21T12:21:48.915598+0000","last_undegraded":"2024-09-21T12:21:48.915598+0000","last_fullsized":"2024-09-21T12:21:48.915598+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:20.217841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.217841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:41.794380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":105,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.933176+0000","last_change":"2024-09-21T12:20:17.308186+0000","last_active":"2024-09-21T12:21:48.933176+0000","last_peered":"2024-09-21T12:21:48.933176+0000","last_clean":"2024-09-21T12:21:48.933176+0000","last_became_active":"2024-09-21T12:20:17.307828+0000","last_became_peered":"2024-09-21T12:20:17.307828+0000","last_unstale":"2024-09-21T12:21:48.933176+0000","last_undegraded":"2024-09-21T12:21:48.933176+0000","last_fullsized":"2024-09-21T12:21:48.933176+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":83,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.923868+0000","last_change":"2024-09-21T12:21:41.022451+0000","last_active":"2024-09-21T12:21:48.923868+0000","last_peered":"2024-09-21T12:21:48.923868+0000","last_clean":"2024-09-21T12:21:48.923868+0000","last_became_active":"2024-09-21T12:21:41.021947+0000","last_became_peered":"2024-09-21T12:21:41.021947+0000","last_unstale":"2024-09-21T12:21:48.923868+0000","last_undegraded":"2024-09-21T12:21:48.923868+0000","last_fullsized":"2024-09-21T12:21:48.923868+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":70,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.921747+0000","last_change":"2024-09-21T12:20:21.593129+0000","last_active":"2024-09-21T12:21:48.921747+0000","last_peered":"2024-09-21T12:21:48.921747+0000","last_clean":"2024-09-21T12:21:48.921747+0000","last_became_active":"2024-09-21T12:20:21.592796+0000","last_became_peered":"2024-09-21T12:20:21.592796+0000","last_unstale":"2024-09-21T12:21:48.921747+0000","last_undegraded":"2024-09-21T12:21:48.921747+0000","last_fullsized":"2024-09-21T12:21:48.921747+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.928568+0000","last_change":"2024-09-21T12:21:41.022965+0000","last_active":"2024-09-21T12:21:48.928568+0000","last_peered":"2024-09-21T12:21:48.928568+0000","last_clean":"2024-09-21T12:21:48.928568+0000","last_became_active":"2024-09-21T12:21:41.022589+0000","last_became_peered":"2024-09-21T12:21:41.022589+0000","last_unstale":"2024-09-21T12:21:48.928568+0000","last_undegraded":"2024-09-21T12:21:48.928568+0000","last_fullsized":"2024-09-21T12:21:48.928568+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":120,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.919201+0000","last_change":"2024-09-21T12:21:41.153978+0000","last_active":"2024-09-21T12:21:48.919201+0000","last_peered":"2024-09-21T12:21:48.919201+0000","last_clean":"2024-09-21T12:21:48.919201+0000","last_became_active":"2024-09-21T12:21:41.142296+0000","last_became_peered":"2024-09-21T12:21:41.142296+0000","last_unstale":"2024-09-21T12:21:48.919201+0000","last_undegraded":"2024-09-21T12:21:48.919201+0000","last_fullsized":"2024-09-21T12:21:48.919201+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":20,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.919536+0000","last_change":"2024-09-21T12:21:41.139452+0000","last_active":"2024-09-21T12:21:48.919536+0000","last_peered":"2024-09-21T12:21:48.919536+0000","last_clean":"2024-09-21T12:21:48.919536+0000","last_became_active":"2024-09-21T12:21:41.139055+0000","last_became_peered":"2024-09-21T12:21:41.139055+0000","last_unstale":"2024-09-21T12:21:48.919536+0000","last_undegraded":"2024-09-21T12:21:48.919536+0000","last_fullsized":"2024-09-21T12:21:48.919536+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":72,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.919951+0000","last_change":"2024-09-21T12:20:21.313476+0000","last_active":"2024-09-21T12:21:48.919951+0000","last_peered":"2024-09-21T12:21:48.919951+0000","last_clean":"2024-09-21T12:21:48.919951+0000","last_became_active":"2024-09-21T12:20:21.312976+0000","last_became_peered":"2024-09-21T12:20:21.312976+0000","last_unstale":"2024-09-21T12:21:48.919951+0000","last_undegraded":"2024-09-21T12:21:48.919951+0000","last_fullsized":"2024-09-21T12:21:48.919951+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:35.977937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":96,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.926246+0000","last_change":"2024-09-21T12:21:41.149510+0000","last_active":"2024-09-21T12:21:48.926246+0000","last_peered":"2024-09-21T12:21:48.926246+0000","last_clean":"2024-09-21T12:21:48.926246+0000","last_became_active":"2024-09-21T12:21:41.140897+0000","last_became_peered":"2024-09-21T12:21:41.140897+0000","last_unstale":"2024-09-21T12:21:48.926246+0000","last_undegraded":"2024-09-21T12:21:48.926246+0000","last_fullsized":"2024-09-21T12:21:48.926246+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":95,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.918517+0000","last_change":"2024-09-21T12:20:17.696423+0000","last_active":"2024-09-21T12:21:48.918517+0000","last_peered":"2024-09-21T12:21:48.918517+0000","last_clean":"2024-09-21T12:21:48.918517+0000","last_became_active":"2024-09-21T12:20:17.695977+0000","last_became_peered":"2024-09-21T12:20:17.695977+0000","last_unstale":"2024-09-21T12:21:48.918517+0000","last_undegraded":"2024-09-21T12:21:48.918517+0000","last_fullsized":"2024-09-21T12:21:48.918517+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"92'9","reported_seq":117,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.909107+0000","last_change":"2024-09-21T12:21:41.157000+0000","last_active":"2024-09-21T12:21:48.909107+0000","last_peered":"2024-09-21T12:21:48.909107+0000","last_clean":"2024-09-21T12:21:48.909107+0000","last_became_active":"2024-09-21T12:21:41.144198+0000","last_became_peered":"2024-09-21T12:21:41.144198+0000","last_unstale":"2024-09-21T12:21:48.909107+0000","last_undegraded":"2024-09-21T12:21:48.909107+0000","last_fullsized":"2024-09-21T12:21:48.909107+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"95'2258","reported_seq":2290,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:51.484464+0000","last_change":"2024-09-21T12:21:24.886463+0000","last_active":"2024-09-21T12:21:51.484464+0000","last_peered":"2024-09-21T12:21:51.484464+0000","last_clean":"2024-09-21T12:21:51.484464+0000","last_became_active":"2024-09-21T12:21:24.886130+0000","last_became_peered":"2024-09-21T12:21:24.886130+0000","last_unstale":"2024-09-21T12:21:51.484464+0000","last_undegraded":"2024-09-21T12:21:51.484464+0000","last_fullsized":"2024-09-21T12:21:51.484464+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":2258,"log_dups_size":0,"ondisk_log_size":2258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:37.307039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147980288,"num_objects":2258,"num_object_clones":0,"num_object_copies":6774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2258,"num_write_kb":144512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":128,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.924848+0000","last_change":"2024-09-21T12:21:41.151018+0000","last_active":"2024-09-21T12:21:48.924848+0000","last_peered":"2024-09-21T12:21:48.924848+0000","last_clean":"2024-09-21T12:21:48.924848+0000","last_became_active":"2024-09-21T12:21:41.029573+0000","last_became_peered":"2024-09-21T12:21:41.029573+0000","last_unstale":"2024-09-21T12:21:48.924848+0000","last_undegraded":"2024-09-21T12:21:48.924848+0000","last_fullsized":"2024-09-21T12:21:48.924848+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":82,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.926696+0000","last_change":"2024-09-21T12:20:15.296251+0000","last_active":"2024-09-21T12:21:48.926696+0000","last_peered":"2024-09-21T12:21:48.926696+0000","last_clean":"2024-09-21T12:21:48.926696+0000","last_became_active":"2024-09-21T12:20:15.295774+0000","last_became_peered":"2024-09-21T12:20:15.295774+0000","last_unstale":"2024-09-21T12:21:48.926696+0000","last_undegraded":"2024-09-21T12:21:48.926696+0000","last_fullsized":"2024-09-21T12:21:48.926696+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"92'9","reported_seq":52,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.931370+0000","last_change":"2024-09-21T12:21:41.131633+0000","last_active":"2024-09-21T12:21:48.931370+0000","last_peered":"2024-09-21T12:21:48.931370+0000","last_clean":"2024-09-21T12:21:48.931370+0000","last_became_active":"2024-09-21T12:21:41.023562+0000","last_became_peered":"2024-09-21T12:21:41.023562+0000","last_unstale":"2024-09-21T12:21:48.931370+0000","last_undegraded":"2024-09-21T12:21:48.931370+0000","last_fullsized":"2024-09-21T12:21:48.931370+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:44.099993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":72,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.930847+0000","last_change":"2024-09-21T12:20:21.310295+0000","last_active":"2024-09-21T12:21:48.930847+0000","last_peered":"2024-09-21T12:21:48.930847+0000","last_clean":"2024-09-21T12:21:48.930847+0000","last_became_active":"2024-09-21T12:20:21.310022+0000","last_became_peered":"2024-09-21T12:20:21.310022+0000","last_unstale":"2024-09-21T12:21:48.930847+0000","last_undegraded":"2024-09-21T12:21:48.930847+0000","last_fullsized":"2024-09-21T12:21:48.930847+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"95'2245","reported_seq":3678,"reported_epoch":95,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-21T12:21:51.580712+0000","last_change":"2024-09-21T12:21:42.090911+0000","last_active":"2024-09-21T12:21:51.580712+0000","last_peered":"2024-09-21T12:21:51.580712+0000","last_clean":"2024-09-21T12:21:39.995286+0000","last_became_active":"2024-09-21T12:21:42.066440+0000","last_became_peered":"2024-09-21T12:21:42.066440+0000","last_unstale":"2024-09-21T12:21:51.580712+0000","last_undegraded":"2024-09-21T12:21:51.580712+0000","last_fullsized":"2024-09-21T12:21:42.007091+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":2245,"log_dups_size":0,"ondisk_log_size":2245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147128320,"num_objects":2245,"num_object_clones":0,"num_object_copies":6735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":838,"num_objects_unfound":0,"num_objects_dirty":2245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2245,"num_write_kb":143680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":698,"num_bytes_recovered":45744128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2],"avail_no_missing":["3","2","4"],"object_location_counts":[{"shards":"2,3","objects":1407},{"shards":"2,3,4","objects":838}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":110,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.926494+0000","last_change":"2024-09-21T12:20:17.693550+0000","last_active":"2024-09-21T12:21:48.926494+0000","last_peered":"2024-09-21T12:21:48.926494+0000","last_clean":"2024-09-21T12:21:48.926494+0000","last_became_active":"2024-09-21T12:20:17.693179+0000","last_became_peered":"2024-09-21T12:20:17.693179+0000","last_unstale":"2024-09-21T12:21:48.926494+0000","last_undegraded":"2024-09-21T12:21:48.926494+0000","last_fullsized":"2024-09-21T12:21:48.926494+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":90,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.922095+0000","last_change":"2024-09-21T12:21:18.056305+0000","last_active":"2024-09-21T12:21:48.922095+0000","last_peered":"2024-09-21T12:21:48.922095+0000","last_clean":"2024-09-21T12:21:48.922095+0000","last_became_active":"2024-09-21T12:20:15.297340+0000","last_became_peered":"2024-09-21T12:20:15.297340+0000","last_unstale":"2024-09-21T12:21:48.922095+0000","last_undegraded":"2024-09-21T12:21:48.922095+0000","last_fullsized":"2024-09-21T12:21:48.922095+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:18.056261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:18.056261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:31.683295+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":20,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.929346+0000","last_change":"2024-09-21T12:21:41.137418+0000","last_active":"2024-09-21T12:21:48.929346+0000","last_peered":"2024-09-21T12:21:48.929346+0000","last_clean":"2024-09-21T12:21:48.929346+0000","last_became_active":"2024-09-21T12:21:41.137080+0000","last_became_peered":"2024-09-21T12:21:41.137080+0000","last_unstale":"2024-09-21T12:21:48.929346+0000","last_undegraded":"2024-09-21T12:21:48.929346+0000","last_fullsized":"2024-09-21T12:21:48.929346+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":19,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.920200+0000","last_change":"2024-09-21T12:21:41.144137+0000","last_active":"2024-09-21T12:21:48.920200+0000","last_peered":"2024-09-21T12:21:48.920200+0000","last_clean":"2024-09-21T12:21:48.920200+0000","last_became_active":"2024-09-21T12:21:41.143312+0000","last_became_peered":"2024-09-21T12:21:41.143312+0000","last_unstale":"2024-09-21T12:21:48.920200+0000","last_undegraded":"2024-09-21T12:21:48.920200+0000","last_fullsized":"2024-09-21T12:21:48.920200+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"95'2336","reported_seq":2369,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:51.548489+0000","last_change":"2024-09-21T12:21:24.889433+0000","last_active":"2024-09-21T12:21:51.548489+0000","last_peered":"2024-09-21T12:21:51.548489+0000","last_clean":"2024-09-21T12:21:51.548489+0000","last_became_active":"2024-09-21T12:21:24.889133+0000","last_became_peered":"2024-09-21T12:21:24.889133+0000","last_unstale":"2024-09-21T12:21:51.548489+0000","last_undegraded":"2024-09-21T12:21:51.548489+0000","last_fullsized":"2024-09-21T12:21:51.548489+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":2336,"log_dups_size":0,"ondisk_log_size":2336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:48.105823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153092096,"num_objects":2336,"num_object_clones":0,"num_object_copies":7008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2336,"num_write_kb":149504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":76,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.930468+0000","last_change":"2024-09-21T12:21:41.154425+0000","last_active":"2024-09-21T12:21:48.930468+0000","last_peered":"2024-09-21T12:21:48.930468+0000","last_clean":"2024-09-21T12:21:48.930468+0000","last_became_active":"2024-09-21T12:21:41.134620+0000","last_became_peered":"2024-09-21T12:21:41.134620+0000","last_unstale":"2024-09-21T12:21:48.930468+0000","last_undegraded":"2024-09-21T12:21:48.930468+0000","last_fullsized":"2024-09-21T12:21:48.930468+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":96,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.918746+0000","last_change":"2024-09-21T12:21:41.141533+0000","last_active":"2024-09-21T12:21:48.918746+0000","last_peered":"2024-09-21T12:21:48.918746+0000","last_clean":"2024-09-21T12:21:48.918746+0000","last_became_active":"2024-09-21T12:21:41.134498+0000","last_became_peered":"2024-09-21T12:21:41.134498+0000","last_unstale":"2024-09-21T12:21:48.918746+0000","last_undegraded":"2024-09-21T12:21:48.918746+0000","last_fullsized":"2024-09-21T12:21:48.918746+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"92'9","reported_seq":113,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.926319+0000","last_change":"2024-09-21T12:21:41.034012+0000","last_active":"2024-09-21T12:21:48.926319+0000","last_peered":"2024-09-21T12:21:48.926319+0000","last_clean":"2024-09-21T12:21:48.926319+0000","last_became_active":"2024-09-21T12:21:41.029821+0000","last_became_peered":"2024-09-21T12:21:41.029821+0000","last_unstale":"2024-09-21T12:21:48.926319+0000","last_undegraded":"2024-09-21T12:21:48.926319+0000","last_fullsized":"2024-09-21T12:21:48.926319+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":79,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.919554+0000","last_change":"2024-09-21T12:21:21.835669+0000","last_active":"2024-09-21T12:21:48.919554+0000","last_peered":"2024-09-21T12:21:48.919554+0000","last_clean":"2024-09-21T12:21:48.919554+0000","last_became_active":"2024-09-21T12:20:21.593367+0000","last_became_peered":"2024-09-21T12:20:21.593367+0000","last_unstale":"2024-09-21T12:21:48.919554+0000","last_undegraded":"2024-09-21T12:21:48.919554+0000","last_fullsized":"2024-09-21T12:21:48.919554+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:21.835623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:21:21.835623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:26.007747+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"95'2294","reported_seq":3438,"reported_epoch":95,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-21T12:21:51.577419+0000","last_change":"2024-09-21T12:21:42.097942+0000","last_active":"2024-09-21T12:21:51.577419+0000","last_peered":"2024-09-21T12:21:51.577419+0000","last_clean":"2024-09-21T12:21:39.956377+0000","last_became_active":"2024-09-21T12:21:42.073046+0000","last_became_peered":"2024-09-21T12:21:42.073046+0000","last_unstale":"2024-09-21T12:21:51.577419+0000","last_undegraded":"2024-09-21T12:21:51.577419+0000","last_fullsized":"2024-09-21T12:21:42.006858+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":2294,"log_dups_size":0,"ondisk_log_size":2294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150339584,"num_objects":2294,"num_object_clones":0,"num_object_copies":6882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":999,"num_objects_unfound":0,"num_objects_dirty":2294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2294,"num_write_kb":146816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":554,"num_bytes_recovered":36306944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":999},{"shards":"3,6","objects":1295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":114,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.928329+0000","last_change":"2024-09-21T12:21:41.144421+0000","last_active":"2024-09-21T12:21:48.928329+0000","last_peered":"2024-09-21T12:21:48.928329+0000","last_clean":"2024-09-21T12:21:48.928329+0000","last_became_active":"2024-09-21T12:21:41.021857+0000","last_became_peered":"2024-09-21T12:21:41.021857+0000","last_unstale":"2024-09-21T12:21:48.928329+0000","last_undegraded":"2024-09-21T12:21:48.928329+0000","last_fullsized":"2024-09-21T12:21:48.928329+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":83,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.931294+0000","last_change":"2024-09-21T12:20:15.308877+0000","last_active":"2024-09-21T12:21:48.931294+0000","last_peered":"2024-09-21T12:21:48.931294+0000","last_clean":"2024-09-21T12:21:48.931294+0000","last_became_active":"2024-09-21T12:20:15.308518+0000","last_became_peered":"2024-09-21T12:20:15.308518+0000","last_unstale":"2024-09-21T12:21:48.931294+0000","last_undegraded":"2024-09-21T12:21:48.931294+0000","last_fullsized":"2024-09-21T12:21:48.931294+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:26.986669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":115,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.915267+0000","last_change":"2024-09-21T12:21:23.139205+0000","last_active":"2024-09-21T12:21:48.915267+0000","last_peered":"2024-09-21T12:21:48.915267+0000","last_clean":"2024-09-21T12:21:48.915267+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:21:48.915267+0000","last_undegraded":"2024-09-21T12:21:48.915267+0000","last_fullsized":"2024-09-21T12:21:48.915267+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:21:23.139124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.139124+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:48.708731+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":72,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.927190+0000","last_change":"2024-09-21T12:20:21.316127+0000","last_active":"2024-09-21T12:21:48.927190+0000","last_peered":"2024-09-21T12:21:48.927190+0000","last_clean":"2024-09-21T12:21:48.927190+0000","last_became_active":"2024-09-21T12:20:21.315801+0000","last_became_peered":"2024-09-21T12:20:21.315801+0000","last_unstale":"2024-09-21T12:21:48.927190+0000","last_undegraded":"2024-09-21T12:21:48.927190+0000","last_fullsized":"2024-09-21T12:21:48.927190+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:33.077380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"95'2214","reported_seq":2246,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:51.504989+0000","last_change":"2024-09-21T12:21:24.884531+0000","last_active":"2024-09-21T12:21:51.504989+0000","last_peered":"2024-09-21T12:21:51.504989+0000","last_clean":"2024-09-21T12:21:51.504989+0000","last_became_active":"2024-09-21T12:21:24.884183+0000","last_became_peered":"2024-09-21T12:21:24.884183+0000","last_unstale":"2024-09-21T12:21:51.504989+0000","last_undegraded":"2024-09-21T12:21:51.504989+0000","last_fullsized":"2024-09-21T12:21:51.504989+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":2214,"log_dups_size":0,"ondisk_log_size":2214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:30.526110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145096704,"num_objects":2214,"num_object_clones":0,"num_object_copies":6642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2214,"num_write_kb":141696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":116,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.915397+0000","last_change":"2024-09-21T12:20:17.693585+0000","last_active":"2024-09-21T12:21:48.915397+0000","last_peered":"2024-09-21T12:21:48.915397+0000","last_clean":"2024-09-21T12:21:48.915397+0000","last_became_active":"2024-09-21T12:20:17.693067+0000","last_became_peered":"2024-09-21T12:20:17.693067+0000","last_unstale":"2024-09-21T12:21:48.915397+0000","last_undegraded":"2024-09-21T12:21:48.915397+0000","last_fullsized":"2024-09-21T12:21:48.915397+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":84,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.923993+0000","last_change":"2024-09-21T12:20:15.308741+0000","last_active":"2024-09-21T12:21:48.923993+0000","last_peered":"2024-09-21T12:21:48.923993+0000","last_clean":"2024-09-21T12:21:48.923993+0000","last_became_active":"2024-09-21T12:20:15.308441+0000","last_became_peered":"2024-09-21T12:20:15.308441+0000","last_unstale":"2024-09-21T12:21:48.923993+0000","last_undegraded":"2024-09-21T12:21:48.923993+0000","last_fullsized":"2024-09-21T12:21:48.923993+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":75,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.908054+0000","last_change":"2024-09-21T12:20:19.309945+0000","last_active":"2024-09-21T12:21:48.908054+0000","last_peered":"2024-09-21T12:21:48.908054+0000","last_clean":"2024-09-21T12:21:48.908054+0000","last_became_active":"2024-09-21T12:20:19.309598+0000","last_became_peered":"2024-09-21T12:20:19.309598+0000","last_unstale":"2024-09-21T12:21:48.908054+0000","last_undegraded":"2024-09-21T12:21:48.908054+0000","last_fullsized":"2024-09-21T12:21:48.908054+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:32.718869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":72,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.926008+0000","last_change":"2024-09-21T12:20:21.593173+0000","last_active":"2024-09-21T12:21:48.926008+0000","last_peered":"2024-09-21T12:21:48.926008+0000","last_clean":"2024-09-21T12:21:48.926008+0000","last_became_active":"2024-09-21T12:20:21.592828+0000","last_became_peered":"2024-09-21T12:20:21.592828+0000","last_unstale":"2024-09-21T12:21:48.926008+0000","last_undegraded":"2024-09-21T12:21:48.926008+0000","last_fullsized":"2024-09-21T12:21:48.926008+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"95'2358","reported_seq":2390,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:51.516157+0000","last_change":"2024-09-21T12:21:24.887083+0000","last_active":"2024-09-21T12:21:51.516157+0000","last_peered":"2024-09-21T12:21:51.516157+0000","last_clean":"2024-09-21T12:21:51.516157+0000","last_became_active":"2024-09-21T12:21:24.886854+0000","last_became_peered":"2024-09-21T12:21:24.886854+0000","last_unstale":"2024-09-21T12:21:51.516157+0000","last_undegraded":"2024-09-21T12:21:51.516157+0000","last_fullsized":"2024-09-21T12:21:51.516157+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":2358,"log_dups_size":0,"ondisk_log_size":2358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:44.381461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154533888,"num_objects":2358,"num_object_clones":0,"num_object_copies":7074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2358,"num_write_kb":150912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":148,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.923898+0000","last_change":"2024-09-21T12:20:17.308049+0000","last_active":"2024-09-21T12:21:48.923898+0000","last_peered":"2024-09-21T12:21:48.923898+0000","last_clean":"2024-09-21T12:21:48.923898+0000","last_became_active":"2024-09-21T12:20:17.307769+0000","last_became_peered":"2024-09-21T12:20:17.307769+0000","last_unstale":"2024-09-21T12:21:48.923898+0000","last_undegraded":"2024-09-21T12:21:48.923898+0000","last_fullsized":"2024-09-21T12:21:48.923898+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:35.762419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.915575+0000","last_change":"2024-09-21T12:21:41.141257+0000","last_active":"2024-09-21T12:21:48.915575+0000","last_peered":"2024-09-21T12:21:48.915575+0000","last_clean":"2024-09-21T12:21:48.915575+0000","last_became_active":"2024-09-21T12:21:41.140886+0000","last_became_peered":"2024-09-21T12:21:41.140886+0000","last_unstale":"2024-09-21T12:21:48.915575+0000","last_undegraded":"2024-09-21T12:21:48.915575+0000","last_fullsized":"2024-09-21T12:21:48.915575+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":101,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.930804+0000","last_change":"2024-09-21T12:21:20.045824+0000","last_active":"2024-09-21T12:21:48.930804+0000","last_peered":"2024-09-21T12:21:48.930804+0000","last_clean":"2024-09-21T12:21:48.930804+0000","last_became_active":"2024-09-21T12:20:14.287998+0000","last_became_peered":"2024-09-21T12:20:14.287998+0000","last_unstale":"2024-09-21T12:21:48.930804+0000","last_undegraded":"2024-09-21T12:21:48.930804+0000","last_fullsized":"2024-09-21T12:21:48.930804+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T12:21:20.045755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.045755+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:31.010568+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011011149999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":83,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.915340+0000","last_change":"2024-09-21T12:21:41.147785+0000","last_active":"2024-09-21T12:21:48.915340+0000","last_peered":"2024-09-21T12:21:48.915340+0000","last_clean":"2024-09-21T12:21:48.915340+0000","last_became_active":"2024-09-21T12:21:41.147355+0000","last_became_peered":"2024-09-21T12:21:41.147355+0000","last_unstale":"2024-09-21T12:21:48.915340+0000","last_undegraded":"2024-09-21T12:21:48.915340+0000","last_fullsized":"2024-09-21T12:21:48.915340+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":77,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.921095+0000","last_change":"2024-09-21T12:21:41.143026+0000","last_active":"2024-09-21T12:21:48.921095+0000","last_peered":"2024-09-21T12:21:48.921095+0000","last_clean":"2024-09-21T12:21:48.921095+0000","last_became_active":"2024-09-21T12:21:41.142698+0000","last_became_peered":"2024-09-21T12:21:41.142698+0000","last_unstale":"2024-09-21T12:21:48.921095+0000","last_undegraded":"2024-09-21T12:21:48.921095+0000","last_fullsized":"2024-09-21T12:21:48.921095+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"95'2163","reported_seq":3609,"reported_epoch":95,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T12:21:51.569315+0000","last_change":"2024-09-21T12:21:42.079223+0000","last_active":"2024-09-21T12:21:51.569315+0000","last_peered":"2024-09-21T12:21:51.569315+0000","last_clean":"2024-09-21T12:21:39.999416+0000","last_became_active":"2024-09-21T12:21:42.068815+0000","last_became_peered":"2024-09-21T12:21:42.068815+0000","last_unstale":"2024-09-21T12:21:51.569315+0000","last_undegraded":"2024-09-21T12:21:42.025478+0000","last_fullsized":"2024-09-21T12:21:42.010383+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":2163,"log_dups_size":0,"ondisk_log_size":2163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141754368,"num_objects":2163,"num_object_clones":0,"num_object_copies":6489,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":768,"num_objects_unfound":0,"num_objects_dirty":2163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2163,"num_write_kb":138432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":705,"num_bytes_recovered":46202880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1","objects":1394},{"shards":"0,1,4","objects":768}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":107,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.924942+0000","last_change":"2024-09-21T12:20:17.297678+0000","last_active":"2024-09-21T12:21:48.924942+0000","last_peered":"2024-09-21T12:21:48.924942+0000","last_clean":"2024-09-21T12:21:48.924942+0000","last_became_active":"2024-09-21T12:20:17.297374+0000","last_became_peered":"2024-09-21T12:20:17.297374+0000","last_unstale":"2024-09-21T12:21:48.924942+0000","last_undegraded":"2024-09-21T12:21:48.924942+0000","last_fullsized":"2024-09-21T12:21:48.924942+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":84,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.923840+0000","last_change":"2024-09-21T12:20:15.308648+0000","last_active":"2024-09-21T12:21:48.923840+0000","last_peered":"2024-09-21T12:21:48.923840+0000","last_clean":"2024-09-21T12:21:48.923840+0000","last_became_active":"2024-09-21T12:20:15.308356+0000","last_became_peered":"2024-09-21T12:20:15.308356+0000","last_unstale":"2024-09-21T12:21:48.923840+0000","last_undegraded":"2024-09-21T12:21:48.923840+0000","last_fullsized":"2024-09-21T12:21:48.923840+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:27.637162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":101,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.932664+0000","last_change":"2024-09-21T12:21:24.390178+0000","last_active":"2024-09-21T12:21:48.932664+0000","last_peered":"2024-09-21T12:21:48.932664+0000","last_clean":"2024-09-21T12:21:48.932664+0000","last_became_active":"2024-09-21T12:20:14.288358+0000","last_became_peered":"2024-09-21T12:20:14.288358+0000","last_unstale":"2024-09-21T12:21:48.932664+0000","last_undegraded":"2024-09-21T12:21:48.932664+0000","last_fullsized":"2024-09-21T12:21:48.932664+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:21:24.390129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:21:24.390129+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:52.729420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019042449999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":76,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.933734+0000","last_change":"2024-09-21T12:20:19.310164+0000","last_active":"2024-09-21T12:21:48.933734+0000","last_peered":"2024-09-21T12:21:48.933734+0000","last_clean":"2024-09-21T12:21:48.933734+0000","last_became_active":"2024-09-21T12:20:19.309724+0000","last_became_peered":"2024-09-21T12:20:19.309724+0000","last_unstale":"2024-09-21T12:21:48.933734+0000","last_undegraded":"2024-09-21T12:21:48.933734+0000","last_fullsized":"2024-09-21T12:21:48.933734+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":71,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.919856+0000","last_change":"2024-09-21T12:20:21.593267+0000","last_active":"2024-09-21T12:21:48.919856+0000","last_peered":"2024-09-21T12:21:48.919856+0000","last_clean":"2024-09-21T12:21:48.919856+0000","last_became_active":"2024-09-21T12:20:21.592982+0000","last_became_peered":"2024-09-21T12:20:21.592982+0000","last_unstale":"2024-09-21T12:21:48.919856+0000","last_undegraded":"2024-09-21T12:21:48.919856+0000","last_fullsized":"2024-09-21T12:21:48.919856+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"95'2216","reported_seq":2249,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:51.545075+0000","last_change":"2024-09-21T12:21:24.889134+0000","last_active":"2024-09-21T12:21:51.545075+0000","last_peered":"2024-09-21T12:21:51.545075+0000","last_clean":"2024-09-21T12:21:51.545075+0000","last_became_active":"2024-09-21T12:21:24.888897+0000","last_became_peered":"2024-09-21T12:21:24.888897+0000","last_unstale":"2024-09-21T12:21:51.545075+0000","last_undegraded":"2024-09-21T12:21:51.545075+0000","last_fullsized":"2024-09-21T12:21:51.545075+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":2216,"log_dups_size":0,"ondisk_log_size":2216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:46.053995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145227776,"num_objects":2216,"num_object_clones":0,"num_object_copies":6648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2216,"num_write_kb":141824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":98,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.918592+0000","last_change":"2024-09-21T12:20:17.299646+0000","last_active":"2024-09-21T12:21:48.918592+0000","last_peered":"2024-09-21T12:21:48.918592+0000","last_clean":"2024-09-21T12:21:48.918592+0000","last_became_active":"2024-09-21T12:20:17.299325+0000","last_became_peered":"2024-09-21T12:20:17.299325+0000","last_unstale":"2024-09-21T12:21:48.918592+0000","last_undegraded":"2024-09-21T12:21:48.918592+0000","last_fullsized":"2024-09-21T12:21:48.918592+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:26.090123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":91,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.917104+0000","last_change":"2024-09-21T12:21:41.028810+0000","last_active":"2024-09-21T12:21:48.917104+0000","last_peered":"2024-09-21T12:21:48.917104+0000","last_clean":"2024-09-21T12:21:48.917104+0000","last_became_active":"2024-09-21T12:21:41.028524+0000","last_became_peered":"2024-09-21T12:21:41.028524+0000","last_unstale":"2024-09-21T12:21:48.917104+0000","last_undegraded":"2024-09-21T12:21:48.917104+0000","last_fullsized":"2024-09-21T12:21:48.917104+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":193,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:50.148166+0000","last_change":"2024-09-21T12:21:22.834972+0000","last_active":"2024-09-21T12:21:50.148166+0000","last_peered":"2024-09-21T12:21:50.148166+0000","last_clean":"2024-09-21T12:21:50.148166+0000","last_became_active":"2024-09-21T12:20:14.287543+0000","last_became_peered":"2024-09-21T12:20:14.287543+0000","last_unstale":"2024-09-21T12:21:50.148166+0000","last_undegraded":"2024-09-21T12:21:50.148166+0000","last_fullsized":"2024-09-21T12:21:50.148166+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-21T12:21:22.834917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:21:22.834917+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:29.301829+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":91,"num_read_kb":86,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":75,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.931271+0000","last_change":"2024-09-21T12:20:19.314574+0000","last_active":"2024-09-21T12:21:48.931271+0000","last_peered":"2024-09-21T12:21:48.931271+0000","last_clean":"2024-09-21T12:21:48.931271+0000","last_became_active":"2024-09-21T12:20:19.314217+0000","last_became_peered":"2024-09-21T12:20:19.314217+0000","last_unstale":"2024-09-21T12:21:48.931271+0000","last_undegraded":"2024-09-21T12:21:48.931271+0000","last_fullsized":"2024-09-21T12:21:48.931271+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:25.012476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":72,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.923395+0000","last_change":"2024-09-21T12:20:21.311501+0000","last_active":"2024-09-21T12:21:48.923395+0000","last_peered":"2024-09-21T12:21:48.923395+0000","last_clean":"2024-09-21T12:21:48.923395+0000","last_became_active":"2024-09-21T12:20:21.311188+0000","last_became_peered":"2024-09-21T12:20:21.311188+0000","last_unstale":"2024-09-21T12:21:48.923395+0000","last_undegraded":"2024-09-21T12:21:48.923395+0000","last_fullsized":"2024-09-21T12:21:48.923395+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:30.646996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"95'2230","reported_seq":2263,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:51.505629+0000","last_change":"2024-09-21T12:21:24.886290+0000","last_active":"2024-09-21T12:21:51.505629+0000","last_peered":"2024-09-21T12:21:51.505629+0000","last_clean":"2024-09-21T12:21:51.505629+0000","last_became_active":"2024-09-21T12:21:24.885986+0000","last_became_peered":"2024-09-21T12:21:24.885986+0000","last_unstale":"2024-09-21T12:21:51.505629+0000","last_undegraded":"2024-09-21T12:21:51.505629+0000","last_fullsized":"2024-09-21T12:21:51.505629+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":2230,"log_dups_size":0,"ondisk_log_size":2230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:42.327997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146145280,"num_objects":2230,"num_object_clones":0,"num_object_copies":6690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2230,"num_write_kb":142720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"92'17","reported_seq":146,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.908851+0000","last_change":"2024-09-21T12:21:41.164861+0000","last_active":"2024-09-21T12:21:48.908851+0000","last_peered":"2024-09-21T12:21:48.908851+0000","last_clean":"2024-09-21T12:21:48.908851+0000","last_became_active":"2024-09-21T12:21:41.146130+0000","last_became_peered":"2024-09-21T12:21:41.146130+0000","last_unstale":"2024-09-21T12:21:48.908851+0000","last_undegraded":"2024-09-21T12:21:48.908851+0000","last_fullsized":"2024-09-21T12:21:48.908851+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":96,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.920406+0000","last_change":"2024-09-21T12:21:17.107378+0000","last_active":"2024-09-21T12:21:48.920406+0000","last_peered":"2024-09-21T12:21:48.920406+0000","last_clean":"2024-09-21T12:21:48.920406+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:21:48.920406+0000","last_undegraded":"2024-09-21T12:21:48.920406+0000","last_fullsized":"2024-09-21T12:21:48.920406+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:21:17.107309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:17.107309+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:44.755672+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":128,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.919563+0000","last_change":"2024-09-21T12:20:24.850239+0000","last_active":"2024-09-21T12:21:48.919563+0000","last_peered":"2024-09-21T12:21:48.919563+0000","last_clean":"2024-09-21T12:21:48.919563+0000","last_became_active":"2024-09-21T12:20:06.349670+0000","last_became_peered":"2024-09-21T12:20:06.349670+0000","last_unstale":"2024-09-21T12:21:48.919563+0000","last_undegraded":"2024-09-21T12:21:48.919563+0000","last_fullsized":"2024-09-21T12:21:48.919563+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T12:20:24.850157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:20:24.850157+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:21:53.046478+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":75,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.920028+0000","last_change":"2024-09-21T12:20:19.310280+0000","last_active":"2024-09-21T12:21:48.920028+0000","last_peered":"2024-09-21T12:21:48.920028+0000","last_clean":"2024-09-21T12:21:48.920028+0000","last_became_active":"2024-09-21T12:20:19.309859+0000","last_became_peered":"2024-09-21T12:20:19.309859+0000","last_unstale":"2024-09-21T12:21:48.920028+0000","last_undegraded":"2024-09-21T12:21:48.920028+0000","last_fullsized":"2024-09-21T12:21:48.920028+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":78,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.931002+0000","last_change":"2024-09-21T12:21:41.133498+0000","last_active":"2024-09-21T12:21:48.931002+0000","last_peered":"2024-09-21T12:21:48.931002+0000","last_clean":"2024-09-21T12:21:48.931002+0000","last_became_active":"2024-09-21T12:21:41.133203+0000","last_became_peered":"2024-09-21T12:21:41.133203+0000","last_unstale":"2024-09-21T12:21:48.931002+0000","last_undegraded":"2024-09-21T12:21:48.931002+0000","last_fullsized":"2024-09-21T12:21:48.931002+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"95'2264","reported_seq":2296,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:51.499239+0000","last_change":"2024-09-21T12:21:24.888106+0000","last_active":"2024-09-21T12:21:51.499239+0000","last_peered":"2024-09-21T12:21:51.499239+0000","last_clean":"2024-09-21T12:21:51.499239+0000","last_became_active":"2024-09-21T12:21:24.887845+0000","last_became_peered":"2024-09-21T12:21:24.887845+0000","last_unstale":"2024-09-21T12:21:51.499239+0000","last_undegraded":"2024-09-21T12:21:51.499239+0000","last_fullsized":"2024-09-21T12:21:51.499239+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":2264,"log_dups_size":0,"ondisk_log_size":2264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:35.680437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148373504,"num_objects":2264,"num_object_clones":0,"num_object_copies":6792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2264,"num_write_kb":144896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":70,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.908831+0000","last_change":"2024-09-21T12:21:41.155757+0000","last_active":"2024-09-21T12:21:48.908831+0000","last_peered":"2024-09-21T12:21:48.908831+0000","last_clean":"2024-09-21T12:21:48.908831+0000","last_became_active":"2024-09-21T12:21:41.141402+0000","last_became_peered":"2024-09-21T12:21:41.141402+0000","last_unstale":"2024-09-21T12:21:48.908831+0000","last_undegraded":"2024-09-21T12:21:48.908831+0000","last_fullsized":"2024-09-21T12:21:48.908831+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:21:19.327640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:21:19.327640+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:31.090630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.932555+0000","last_change":"2024-09-21T12:21:41.013376+0000","last_active":"2024-09-21T12:21:48.932555+0000","last_peered":"2024-09-21T12:21:48.932555+0000","last_clean":"2024-09-21T12:21:48.932555+0000","last_became_active":"2024-09-21T12:21:41.013031+0000","last_became_peered":"2024-09-21T12:21:41.013031+0000","last_unstale":"2024-09-21T12:21:48.932555+0000","last_undegraded":"2024-09-21T12:21:48.932555+0000","last_fullsized":"2024-09-21T12:21:48.932555+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":82,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.930946+0000","last_change":"2024-09-21T12:21:41.136591+0000","last_active":"2024-09-21T12:21:48.930946+0000","last_peered":"2024-09-21T12:21:48.930946+0000","last_clean":"2024-09-21T12:21:48.930946+0000","last_became_active":"2024-09-21T12:21:41.136178+0000","last_became_peered":"2024-09-21T12:21:41.136178+0000","last_unstale":"2024-09-21T12:21:48.930946+0000","last_undegraded":"2024-09-21T12:21:48.930946+0000","last_fullsized":"2024-09-21T12:21:48.930946+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":70,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.919458+0000","last_change":"2024-09-21T12:20:21.593145+0000","last_active":"2024-09-21T12:21:48.919458+0000","last_peered":"2024-09-21T12:21:48.919458+0000","last_clean":"2024-09-21T12:21:48.919458+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:21:48.919458+0000","last_undegraded":"2024-09-21T12:21:48.919458+0000","last_fullsized":"2024-09-21T12:21:48.919458+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"95'2246","reported_seq":2279,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:51.546976+0000","last_change":"2024-09-21T12:21:24.888960+0000","last_active":"2024-09-21T12:21:51.546976+0000","last_peered":"2024-09-21T12:21:51.546976+0000","last_clean":"2024-09-21T12:21:51.546976+0000","last_became_active":"2024-09-21T12:21:24.888670+0000","last_became_peered":"2024-09-21T12:21:24.888670+0000","last_unstale":"2024-09-21T12:21:51.546976+0000","last_undegraded":"2024-09-21T12:21:51.546976+0000","last_fullsized":"2024-09-21T12:21:51.546976+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":2246,"log_dups_size":0,"ondisk_log_size":2246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:42.169378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147193856,"num_objects":2246,"num_object_clones":0,"num_object_copies":6738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2246,"num_write_kb":143744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":85,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.933151+0000","last_change":"2024-09-21T12:21:41.167474+0000","last_active":"2024-09-21T12:21:48.933151+0000","last_peered":"2024-09-21T12:21:48.933151+0000","last_clean":"2024-09-21T12:21:48.933151+0000","last_became_active":"2024-09-21T12:21:41.148194+0000","last_became_peered":"2024-09-21T12:21:41.148194+0000","last_unstale":"2024-09-21T12:21:48.933151+0000","last_undegraded":"2024-09-21T12:21:48.933151+0000","last_fullsized":"2024-09-21T12:21:48.933151+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":90,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.908302+0000","last_change":"2024-09-21T12:21:41.142387+0000","last_active":"2024-09-21T12:21:48.908302+0000","last_peered":"2024-09-21T12:21:48.908302+0000","last_clean":"2024-09-21T12:21:48.908302+0000","last_became_active":"2024-09-21T12:21:41.142065+0000","last_became_peered":"2024-09-21T12:21:41.142065+0000","last_unstale":"2024-09-21T12:21:48.908302+0000","last_undegraded":"2024-09-21T12:21:48.908302+0000","last_fullsized":"2024-09-21T12:21:48.908302+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":83,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.923586+0000","last_change":"2024-09-21T12:21:41.022060+0000","last_active":"2024-09-21T12:21:48.923586+0000","last_peered":"2024-09-21T12:21:48.923586+0000","last_clean":"2024-09-21T12:21:48.923586+0000","last_became_active":"2024-09-21T12:21:41.021194+0000","last_became_peered":"2024-09-21T12:21:41.021194+0000","last_unstale":"2024-09-21T12:21:48.923586+0000","last_undegraded":"2024-09-21T12:21:48.923586+0000","last_fullsized":"2024-09-21T12:21:48.923586+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":79,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.933570+0000","last_change":"2024-09-21T12:21:41.146426+0000","last_active":"2024-09-21T12:21:48.933570+0000","last_peered":"2024-09-21T12:21:48.933570+0000","last_clean":"2024-09-21T12:21:48.933570+0000","last_became_active":"2024-09-21T12:21:41.145902+0000","last_became_peered":"2024-09-21T12:21:41.145902+0000","last_unstale":"2024-09-21T12:21:48.933570+0000","last_undegraded":"2024-09-21T12:21:48.933570+0000","last_fullsized":"2024-09-21T12:21:48.933570+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"95'2269","reported_seq":2302,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:51.535002+0000","last_change":"2024-09-21T12:21:24.887418+0000","last_active":"2024-09-21T12:21:51.535002+0000","last_peered":"2024-09-21T12:21:51.535002+0000","last_clean":"2024-09-21T12:21:51.535002+0000","last_became_active":"2024-09-21T12:21:24.887000+0000","last_became_peered":"2024-09-21T12:21:24.887000+0000","last_unstale":"2024-09-21T12:21:51.535002+0000","last_undegraded":"2024-09-21T12:21:51.535002+0000","last_fullsized":"2024-09-21T12:21:51.535002+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":2269,"log_dups_size":0,"ondisk_log_size":2269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:29.571773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148701184,"num_objects":2269,"num_object_clones":0,"num_object_copies":6807,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2269,"num_write_kb":145216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":66,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.919691+0000","last_change":"2024-09-21T12:21:41.159063+0000","last_active":"2024-09-21T12:21:48.919691+0000","last_peered":"2024-09-21T12:21:48.919691+0000","last_clean":"2024-09-21T12:21:48.919691+0000","last_became_active":"2024-09-21T12:21:41.142910+0000","last_became_peered":"2024-09-21T12:21:41.142910+0000","last_unstale":"2024-09-21T12:21:48.919691+0000","last_undegraded":"2024-09-21T12:21:48.919691+0000","last_fullsized":"2024-09-21T12:21:48.919691+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":89,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.931380+0000","last_change":"2024-09-21T12:21:41.032297+0000","last_active":"2024-09-21T12:21:48.931380+0000","last_peered":"2024-09-21T12:21:48.931380+0000","last_clean":"2024-09-21T12:21:48.931380+0000","last_became_active":"2024-09-21T12:21:41.030825+0000","last_became_peered":"2024-09-21T12:21:41.030825+0000","last_unstale":"2024-09-21T12:21:48.931380+0000","last_undegraded":"2024-09-21T12:21:48.931380+0000","last_fullsized":"2024-09-21T12:21:48.931380+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":104,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.933103+0000","last_change":"2024-09-21T12:20:19.308246+0000","last_active":"2024-09-21T12:21:48.933103+0000","last_peered":"2024-09-21T12:21:48.933103+0000","last_clean":"2024-09-21T12:21:48.933103+0000","last_became_active":"2024-09-21T12:20:19.307924+0000","last_became_peered":"2024-09-21T12:20:19.307924+0000","last_unstale":"2024-09-21T12:21:48.933103+0000","last_undegraded":"2024-09-21T12:21:48.933103+0000","last_fullsized":"2024-09-21T12:21:48.933103+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:40.633298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":19,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.924543+0000","last_change":"2024-09-21T12:21:41.018351+0000","last_active":"2024-09-21T12:21:48.924543+0000","last_peered":"2024-09-21T12:21:48.924543+0000","last_clean":"2024-09-21T12:21:48.924543+0000","last_became_active":"2024-09-21T12:21:41.016316+0000","last_became_peered":"2024-09-21T12:21:41.016316+0000","last_unstale":"2024-09-21T12:21:48.924543+0000","last_undegraded":"2024-09-21T12:21:48.924543+0000","last_fullsized":"2024-09-21T12:21:48.924543+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"95'2335","reported_seq":3467,"reported_epoch":95,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T12:21:51.524013+0000","last_change":"2024-09-21T12:21:42.073850+0000","last_active":"2024-09-21T12:21:51.524013+0000","last_peered":"2024-09-21T12:21:51.524013+0000","last_clean":"2024-09-21T12:21:39.984999+0000","last_became_active":"2024-09-21T12:21:42.062776+0000","last_became_peered":"2024-09-21T12:21:42.062776+0000","last_unstale":"2024-09-21T12:21:51.524013+0000","last_undegraded":"2024-09-21T12:21:42.015189+0000","last_fullsized":"2024-09-21T12:21:42.005386+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":2335,"log_dups_size":0,"ondisk_log_size":2335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153026560,"num_objects":2335,"num_object_clones":0,"num_object_copies":7005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":976,"num_objects_unfound":0,"num_objects_dirty":2335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2335,"num_write_kb":149440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":549,"num_bytes_recovered":35979264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7],"avail_no_missing":["2","4","7"],"object_location_counts":[{"shards":"2,4,7","objects":976},{"shards":"2,7","objects":1358}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":121,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.915314+0000","last_change":"2024-09-21T12:21:41.038134+0000","last_active":"2024-09-21T12:21:48.915314+0000","last_peered":"2024-09-21T12:21:48.915314+0000","last_clean":"2024-09-21T12:21:48.915314+0000","last_became_active":"2024-09-21T12:21:41.031748+0000","last_became_peered":"2024-09-21T12:21:41.031748+0000","last_unstale":"2024-09-21T12:21:48.915314+0000","last_undegraded":"2024-09-21T12:21:48.915314+0000","last_fullsized":"2024-09-21T12:21:48.915314+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:21:18.243868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:21:18.243868+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:24.545912+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":83,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.930791+0000","last_change":"2024-09-21T12:20:15.302587+0000","last_active":"2024-09-21T12:21:48.930791+0000","last_peered":"2024-09-21T12:21:48.930791+0000","last_clean":"2024-09-21T12:21:48.930791+0000","last_became_active":"2024-09-21T12:20:15.302262+0000","last_became_peered":"2024-09-21T12:20:15.302262+0000","last_unstale":"2024-09-21T12:21:48.930791+0000","last_undegraded":"2024-09-21T12:21:48.930791+0000","last_fullsized":"2024-09-21T12:21:48.930791+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:48.085958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":83,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.933147+0000","last_change":"2024-09-21T12:21:41.030100+0000","last_active":"2024-09-21T12:21:48.933147+0000","last_peered":"2024-09-21T12:21:48.933147+0000","last_clean":"2024-09-21T12:21:48.933147+0000","last_became_active":"2024-09-21T12:21:41.029681+0000","last_became_peered":"2024-09-21T12:21:41.029681+0000","last_unstale":"2024-09-21T12:21:48.933147+0000","last_undegraded":"2024-09-21T12:21:48.933147+0000","last_fullsized":"2024-09-21T12:21:48.933147+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":72,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.916996+0000","last_change":"2024-09-21T12:20:21.311799+0000","last_active":"2024-09-21T12:21:48.916996+0000","last_peered":"2024-09-21T12:21:48.916996+0000","last_clean":"2024-09-21T12:21:48.916996+0000","last_became_active":"2024-09-21T12:20:21.311493+0000","last_became_peered":"2024-09-21T12:20:21.311493+0000","last_unstale":"2024-09-21T12:21:48.916996+0000","last_undegraded":"2024-09-21T12:21:48.916996+0000","last_fullsized":"2024-09-21T12:21:48.916996+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"95'2261","reported_seq":3635,"reported_epoch":95,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-21T12:21:51.580749+0000","last_change":"2024-09-21T12:21:42.103001+0000","last_active":"2024-09-21T12:21:51.580749+0000","last_peered":"2024-09-21T12:21:51.580749+0000","last_clean":"2024-09-21T12:21:40.002980+0000","last_became_active":"2024-09-21T12:21:42.071596+0000","last_became_peered":"2024-09-21T12:21:42.071596+0000","last_unstale":"2024-09-21T12:21:51.580749+0000","last_undegraded":"2024-09-21T12:21:51.580749+0000","last_fullsized":"2024-09-21T12:21:42.010404+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":2261,"log_dups_size":0,"ondisk_log_size":2261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148176896,"num_objects":2261,"num_object_clones":0,"num_object_copies":6783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":886,"num_objects_unfound":0,"num_objects_dirty":2261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2261,"num_write_kb":144704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":670,"num_bytes_recovered":43909120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1","objects":1375},{"shards":"0,1,4","objects":886}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":110,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.908114+0000","last_change":"2024-09-21T12:20:17.693677+0000","last_active":"2024-09-21T12:21:48.908114+0000","last_peered":"2024-09-21T12:21:48.908114+0000","last_clean":"2024-09-21T12:21:48.908114+0000","last_became_active":"2024-09-21T12:20:17.692867+0000","last_became_peered":"2024-09-21T12:20:17.692867+0000","last_unstale":"2024-09-21T12:21:48.908114+0000","last_undegraded":"2024-09-21T12:21:48.908114+0000","last_fullsized":"2024-09-21T12:21:48.908114+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:40.775254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":83,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.920070+0000","last_change":"2024-09-21T12:20:15.300854+0000","last_active":"2024-09-21T12:21:48.920070+0000","last_peered":"2024-09-21T12:21:48.920070+0000","last_clean":"2024-09-21T12:21:48.920070+0000","last_became_active":"2024-09-21T12:20:15.299333+0000","last_became_peered":"2024-09-21T12:20:15.299333+0000","last_unstale":"2024-09-21T12:21:48.920070+0000","last_undegraded":"2024-09-21T12:21:48.920070+0000","last_fullsized":"2024-09-21T12:21:48.920070+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":76,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.932880+0000","last_change":"2024-09-21T12:20:19.307316+0000","last_active":"2024-09-21T12:21:48.932880+0000","last_peered":"2024-09-21T12:21:48.932880+0000","last_clean":"2024-09-21T12:21:48.932880+0000","last_became_active":"2024-09-21T12:20:19.306952+0000","last_became_peered":"2024-09-21T12:20:19.306952+0000","last_unstale":"2024-09-21T12:21:48.932880+0000","last_undegraded":"2024-09-21T12:21:48.932880+0000","last_fullsized":"2024-09-21T12:21:48.932880+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":71,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.919509+0000","last_change":"2024-09-21T12:20:21.314891+0000","last_active":"2024-09-21T12:21:48.919509+0000","last_peered":"2024-09-21T12:21:48.919509+0000","last_clean":"2024-09-21T12:21:48.919509+0000","last_became_active":"2024-09-21T12:20:21.314545+0000","last_became_peered":"2024-09-21T12:20:21.314545+0000","last_unstale":"2024-09-21T12:21:48.919509+0000","last_undegraded":"2024-09-21T12:21:48.919509+0000","last_fullsized":"2024-09-21T12:21:48.919509+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:24.899720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"95'2211","reported_seq":2244,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:51.569037+0000","last_change":"2024-09-21T12:21:24.888714+0000","last_active":"2024-09-21T12:21:51.569037+0000","last_peered":"2024-09-21T12:21:51.569037+0000","last_clean":"2024-09-21T12:21:51.569037+0000","last_became_active":"2024-09-21T12:21:24.888445+0000","last_became_peered":"2024-09-21T12:21:24.888445+0000","last_unstale":"2024-09-21T12:21:51.569037+0000","last_undegraded":"2024-09-21T12:21:51.569037+0000","last_fullsized":"2024-09-21T12:21:51.569037+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":2211,"log_dups_size":0,"ondisk_log_size":2211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:53.110945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144900096,"num_objects":2211,"num_object_clones":0,"num_object_copies":6633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2211,"num_write_kb":141504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":74,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.926166+0000","last_change":"2024-09-21T12:21:41.155880+0000","last_active":"2024-09-21T12:21:48.926166+0000","last_peered":"2024-09-21T12:21:48.926166+0000","last_clean":"2024-09-21T12:21:48.926166+0000","last_became_active":"2024-09-21T12:21:41.140972+0000","last_became_peered":"2024-09-21T12:21:41.140972+0000","last_unstale":"2024-09-21T12:21:48.926166+0000","last_undegraded":"2024-09-21T12:21:48.926166+0000","last_fullsized":"2024-09-21T12:21:48.926166+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":98,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.920709+0000","last_change":"2024-09-21T12:21:41.131873+0000","last_active":"2024-09-21T12:21:48.920709+0000","last_peered":"2024-09-21T12:21:48.920709+0000","last_clean":"2024-09-21T12:21:48.920709+0000","last_became_active":"2024-09-21T12:21:41.019983+0000","last_became_peered":"2024-09-21T12:21:41.019983+0000","last_unstale":"2024-09-21T12:21:48.920709+0000","last_undegraded":"2024-09-21T12:21:48.920709+0000","last_fullsized":"2024-09-21T12:21:48.920709+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:20.865749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.865749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:31.198847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":84,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.939027+0000","last_change":"2024-09-21T12:21:23.393320+0000","last_active":"2024-09-21T12:21:48.939027+0000","last_peered":"2024-09-21T12:21:48.939027+0000","last_clean":"2024-09-21T12:21:48.939027+0000","last_became_active":"2024-09-21T12:20:19.312572+0000","last_became_peered":"2024-09-21T12:20:19.312572+0000","last_unstale":"2024-09-21T12:21:48.939027+0000","last_undegraded":"2024-09-21T12:21:48.939027+0000","last_fullsized":"2024-09-21T12:21:48.939027+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.393270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.393270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:28.198881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":70,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.930964+0000","last_change":"2024-09-21T12:20:21.317550+0000","last_active":"2024-09-21T12:21:48.930964+0000","last_peered":"2024-09-21T12:21:48.930964+0000","last_clean":"2024-09-21T12:21:48.930964+0000","last_became_active":"2024-09-21T12:20:21.317223+0000","last_became_peered":"2024-09-21T12:20:21.317223+0000","last_unstale":"2024-09-21T12:21:48.930964+0000","last_undegraded":"2024-09-21T12:21:48.930964+0000","last_fullsized":"2024-09-21T12:21:48.930964+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:34.865119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"95'2226","reported_seq":2258,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:51.542707+0000","last_change":"2024-09-21T12:21:24.888120+0000","last_active":"2024-09-21T12:21:51.542707+0000","last_peered":"2024-09-21T12:21:51.542707+0000","last_clean":"2024-09-21T12:21:51.542707+0000","last_became_active":"2024-09-21T12:21:24.887488+0000","last_became_peered":"2024-09-21T12:21:24.887488+0000","last_unstale":"2024-09-21T12:21:51.542707+0000","last_undegraded":"2024-09-21T12:21:51.542707+0000","last_fullsized":"2024-09-21T12:21:51.542707+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":2226,"log_dups_size":0,"ondisk_log_size":2226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:52.700127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145883136,"num_objects":2226,"num_object_clones":0,"num_object_copies":6678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2226,"num_write_kb":142464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":104,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.930352+0000","last_change":"2024-09-21T12:20:17.693302+0000","last_active":"2024-09-21T12:21:48.930352+0000","last_peered":"2024-09-21T12:21:48.930352+0000","last_clean":"2024-09-21T12:21:48.930352+0000","last_became_active":"2024-09-21T12:20:17.692870+0000","last_became_peered":"2024-09-21T12:20:17.692870+0000","last_unstale":"2024-09-21T12:21:48.930352+0000","last_undegraded":"2024-09-21T12:21:48.930352+0000","last_fullsized":"2024-09-21T12:21:48.930352+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":108,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.920043+0000","last_change":"2024-09-21T12:21:41.143474+0000","last_active":"2024-09-21T12:21:48.920043+0000","last_peered":"2024-09-21T12:21:48.920043+0000","last_clean":"2024-09-21T12:21:48.920043+0000","last_became_active":"2024-09-21T12:21:41.134524+0000","last_became_peered":"2024-09-21T12:21:41.134524+0000","last_unstale":"2024-09-21T12:21:48.920043+0000","last_undegraded":"2024-09-21T12:21:48.920043+0000","last_fullsized":"2024-09-21T12:21:48.920043+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"92'9","reported_seq":114,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.919742+0000","last_change":"2024-09-21T12:21:41.146163+0000","last_active":"2024-09-21T12:21:48.919742+0000","last_peered":"2024-09-21T12:21:48.919742+0000","last_clean":"2024-09-21T12:21:48.919742+0000","last_became_active":"2024-09-21T12:21:41.133979+0000","last_became_peered":"2024-09-21T12:21:41.133979+0000","last_unstale":"2024-09-21T12:21:48.919742+0000","last_undegraded":"2024-09-21T12:21:48.919742+0000","last_fullsized":"2024-09-21T12:21:48.919742+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":71,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.930586+0000","last_change":"2024-09-21T12:20:21.592893+0000","last_active":"2024-09-21T12:21:48.930586+0000","last_peered":"2024-09-21T12:21:48.930586+0000","last_clean":"2024-09-21T12:21:48.930586+0000","last_became_active":"2024-09-21T12:20:21.592576+0000","last_became_peered":"2024-09-21T12:20:21.592576+0000","last_unstale":"2024-09-21T12:21:48.930586+0000","last_undegraded":"2024-09-21T12:21:48.930586+0000","last_fullsized":"2024-09-21T12:21:48.930586+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":91,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.908197+0000","last_change":"2024-09-21T12:21:15.764309+0000","last_active":"2024-09-21T12:21:48.908197+0000","last_peered":"2024-09-21T12:21:48.908197+0000","last_clean":"2024-09-21T12:21:48.908197+0000","last_became_active":"2024-09-21T12:20:15.303480+0000","last_became_peered":"2024-09-21T12:20:15.303480+0000","last_unstale":"2024-09-21T12:21:48.908197+0000","last_undegraded":"2024-09-21T12:21:48.908197+0000","last_fullsized":"2024-09-21T12:21:48.908197+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:15.764254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:15.764254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:41.145874+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":87,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.921949+0000","last_change":"2024-09-21T12:20:17.694977+0000","last_active":"2024-09-21T12:21:48.921949+0000","last_peered":"2024-09-21T12:21:48.921949+0000","last_clean":"2024-09-21T12:21:48.921949+0000","last_became_active":"2024-09-21T12:20:17.694628+0000","last_became_peered":"2024-09-21T12:20:17.694628+0000","last_unstale":"2024-09-21T12:21:48.921949+0000","last_undegraded":"2024-09-21T12:21:48.921949+0000","last_fullsized":"2024-09-21T12:21:48.921949+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":75,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.931413+0000","last_change":"2024-09-21T12:20:19.314432+0000","last_active":"2024-09-21T12:21:48.931413+0000","last_peered":"2024-09-21T12:21:48.931413+0000","last_clean":"2024-09-21T12:21:48.931413+0000","last_became_active":"2024-09-21T12:20:19.313882+0000","last_became_peered":"2024-09-21T12:20:19.313882+0000","last_unstale":"2024-09-21T12:21:48.931413+0000","last_undegraded":"2024-09-21T12:21:48.931413+0000","last_fullsized":"2024-09-21T12:21:48.931413+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":78,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.919735+0000","last_change":"2024-09-21T12:21:41.134196+0000","last_active":"2024-09-21T12:21:48.919735+0000","last_peered":"2024-09-21T12:21:48.919735+0000","last_clean":"2024-09-21T12:21:48.919735+0000","last_became_active":"2024-09-21T12:21:41.133715+0000","last_became_peered":"2024-09-21T12:21:41.133715+0000","last_unstale":"2024-09-21T12:21:48.919735+0000","last_undegraded":"2024-09-21T12:21:48.919735+0000","last_fullsized":"2024-09-21T12:21:48.919735+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":104,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.918669+0000","last_change":"2024-09-21T12:21:24.093710+0000","last_active":"2024-09-21T12:21:48.918669+0000","last_peered":"2024-09-21T12:21:48.918669+0000","last_clean":"2024-09-21T12:21:48.918669+0000","last_became_active":"2024-09-21T12:20:17.695898+0000","last_became_peered":"2024-09-21T12:20:17.695898+0000","last_unstale":"2024-09-21T12:21:48.918669+0000","last_undegraded":"2024-09-21T12:21:48.918669+0000","last_fullsized":"2024-09-21T12:21:48.918669+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:21:24.093656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:21:24.093656+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:32.120147+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":90,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.919677+0000","last_change":"2024-09-21T12:21:41.020333+0000","last_active":"2024-09-21T12:21:48.919677+0000","last_peered":"2024-09-21T12:21:48.919677+0000","last_clean":"2024-09-21T12:21:48.919677+0000","last_became_active":"2024-09-21T12:21:41.019970+0000","last_became_peered":"2024-09-21T12:21:41.019970+0000","last_unstale":"2024-09-21T12:21:48.919677+0000","last_undegraded":"2024-09-21T12:21:48.919677+0000","last_fullsized":"2024-09-21T12:21:48.919677+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":74,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.921827+0000","last_change":"2024-09-21T12:20:19.312248+0000","last_active":"2024-09-21T12:21:48.921827+0000","last_peered":"2024-09-21T12:21:48.921827+0000","last_clean":"2024-09-21T12:21:48.921827+0000","last_became_active":"2024-09-21T12:20:19.311947+0000","last_became_peered":"2024-09-21T12:20:19.311947+0000","last_unstale":"2024-09-21T12:21:48.921827+0000","last_undegraded":"2024-09-21T12:21:48.921827+0000","last_fullsized":"2024-09-21T12:21:48.921827+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":79,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.932843+0000","last_change":"2024-09-21T12:21:41.022595+0000","last_active":"2024-09-21T12:21:48.932843+0000","last_peered":"2024-09-21T12:21:48.932843+0000","last_clean":"2024-09-21T12:21:48.932843+0000","last_became_active":"2024-09-21T12:21:41.022271+0000","last_became_peered":"2024-09-21T12:21:41.022271+0000","last_unstale":"2024-09-21T12:21:48.932843+0000","last_undegraded":"2024-09-21T12:21:48.932843+0000","last_fullsized":"2024-09-21T12:21:48.932843+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":95,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.931150+0000","last_change":"2024-09-21T12:20:17.694904+0000","last_active":"2024-09-21T12:21:48.931150+0000","last_peered":"2024-09-21T12:21:48.931150+0000","last_clean":"2024-09-21T12:21:48.931150+0000","last_became_active":"2024-09-21T12:20:17.694568+0000","last_became_peered":"2024-09-21T12:20:17.694568+0000","last_unstale":"2024-09-21T12:21:48.931150+0000","last_undegraded":"2024-09-21T12:21:48.931150+0000","last_fullsized":"2024-09-21T12:21:48.931150+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:30.021433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":84,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.915327+0000","last_change":"2024-09-21T12:20:15.297853+0000","last_active":"2024-09-21T12:21:48.915327+0000","last_peered":"2024-09-21T12:21:48.915327+0000","last_clean":"2024-09-21T12:21:48.915327+0000","last_became_active":"2024-09-21T12:20:15.297519+0000","last_became_peered":"2024-09-21T12:20:15.297519+0000","last_unstale":"2024-09-21T12:21:48.915327+0000","last_undegraded":"2024-09-21T12:21:48.915327+0000","last_fullsized":"2024-09-21T12:21:48.915327+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:36.133863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":102,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.919646+0000","last_change":"2024-09-21T12:20:19.305689+0000","last_active":"2024-09-21T12:21:48.919646+0000","last_peered":"2024-09-21T12:21:48.919646+0000","last_clean":"2024-09-21T12:21:48.919646+0000","last_became_active":"2024-09-21T12:20:19.305364+0000","last_became_peered":"2024-09-21T12:20:19.305364+0000","last_unstale":"2024-09-21T12:21:48.919646+0000","last_undegraded":"2024-09-21T12:21:48.919646+0000","last_fullsized":"2024-09-21T12:21:48.919646+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":19,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.933659+0000","last_change":"2024-09-21T12:21:41.024480+0000","last_active":"2024-09-21T12:21:48.933659+0000","last_peered":"2024-09-21T12:21:48.933659+0000","last_clean":"2024-09-21T12:21:48.933659+0000","last_became_active":"2024-09-21T12:21:41.023834+0000","last_became_peered":"2024-09-21T12:21:41.023834+0000","last_unstale":"2024-09-21T12:21:48.933659+0000","last_undegraded":"2024-09-21T12:21:48.933659+0000","last_fullsized":"2024-09-21T12:21:48.933659+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":93,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.919931+0000","last_change":"2024-09-21T12:20:17.693562+0000","last_active":"2024-09-21T12:21:48.919931+0000","last_peered":"2024-09-21T12:21:48.919931+0000","last_clean":"2024-09-21T12:21:48.919931+0000","last_became_active":"2024-09-21T12:20:17.693209+0000","last_became_peered":"2024-09-21T12:20:17.693209+0000","last_unstale":"2024-09-21T12:21:48.919931+0000","last_undegraded":"2024-09-21T12:21:48.919931+0000","last_fullsized":"2024-09-21T12:21:48.919931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:37.813752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":98,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.919805+0000","last_change":"2024-09-21T12:21:41.134037+0000","last_active":"2024-09-21T12:21:48.919805+0000","last_peered":"2024-09-21T12:21:48.919805+0000","last_clean":"2024-09-21T12:21:48.919805+0000","last_became_active":"2024-09-21T12:21:41.133683+0000","last_became_peered":"2024-09-21T12:21:41.133683+0000","last_unstale":"2024-09-21T12:21:48.919805+0000","last_undegraded":"2024-09-21T12:21:48.919805+0000","last_fullsized":"2024-09-21T12:21:48.919805+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:15.938093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:15.938093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:37.185549+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":103,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.931091+0000","last_change":"2024-09-21T12:20:19.314884+0000","last_active":"2024-09-21T12:21:48.931091+0000","last_peered":"2024-09-21T12:21:48.931091+0000","last_clean":"2024-09-21T12:21:48.931091+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:21:48.931091+0000","last_undegraded":"2024-09-21T12:21:48.931091+0000","last_fullsized":"2024-09-21T12:21:48.931091+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:46.062119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":72,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.918164+0000","last_change":"2024-09-21T12:20:21.309936+0000","last_active":"2024-09-21T12:21:48.918164+0000","last_peered":"2024-09-21T12:21:48.918164+0000","last_clean":"2024-09-21T12:21:48.918164+0000","last_became_active":"2024-09-21T12:20:21.309385+0000","last_became_peered":"2024-09-21T12:20:21.309385+0000","last_unstale":"2024-09-21T12:21:48.918164+0000","last_undegraded":"2024-09-21T12:21:48.918164+0000","last_fullsized":"2024-09-21T12:21:48.918164+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":70,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.908852+0000","last_change":"2024-09-21T12:21:41.161724+0000","last_active":"2024-09-21T12:21:48.908852+0000","last_peered":"2024-09-21T12:21:48.908852+0000","last_clean":"2024-09-21T12:21:48.908852+0000","last_became_active":"2024-09-21T12:21:41.141269+0000","last_became_peered":"2024-09-21T12:21:41.141269+0000","last_unstale":"2024-09-21T12:21:48.908852+0000","last_undegraded":"2024-09-21T12:21:48.908852+0000","last_fullsized":"2024-09-21T12:21:48.908852+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.920127+0000","last_change":"2024-09-21T12:21:41.141823+0000","last_active":"2024-09-21T12:21:48.920127+0000","last_peered":"2024-09-21T12:21:48.920127+0000","last_clean":"2024-09-21T12:21:48.920127+0000","last_became_active":"2024-09-21T12:21:41.141507+0000","last_became_peered":"2024-09-21T12:21:41.141507+0000","last_unstale":"2024-09-21T12:21:48.920127+0000","last_undegraded":"2024-09-21T12:21:48.920127+0000","last_fullsized":"2024-09-21T12:21:48.920127+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":76,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.928168+0000","last_change":"2024-09-21T12:20:19.315714+0000","last_active":"2024-09-21T12:21:48.928168+0000","last_peered":"2024-09-21T12:21:48.928168+0000","last_clean":"2024-09-21T12:21:48.928168+0000","last_became_active":"2024-09-21T12:20:19.315418+0000","last_became_peered":"2024-09-21T12:20:19.315418+0000","last_unstale":"2024-09-21T12:21:48.928168+0000","last_undegraded":"2024-09-21T12:21:48.928168+0000","last_fullsized":"2024-09-21T12:21:48.928168+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":70,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.929263+0000","last_change":"2024-09-21T12:20:21.310344+0000","last_active":"2024-09-21T12:21:48.929263+0000","last_peered":"2024-09-21T12:21:48.929263+0000","last_clean":"2024-09-21T12:21:48.929263+0000","last_became_active":"2024-09-21T12:20:21.310053+0000","last_became_peered":"2024-09-21T12:20:21.310053+0000","last_unstale":"2024-09-21T12:21:48.929263+0000","last_undegraded":"2024-09-21T12:21:48.929263+0000","last_fullsized":"2024-09-21T12:21:48.929263+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":96,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.928419+0000","last_change":"2024-09-21T12:20:17.299367+0000","last_active":"2024-09-21T12:21:48.928419+0000","last_peered":"2024-09-21T12:21:48.928419+0000","last_clean":"2024-09-21T12:21:48.928419+0000","last_became_active":"2024-09-21T12:20:17.299015+0000","last_became_peered":"2024-09-21T12:20:17.299015+0000","last_unstale":"2024-09-21T12:21:48.928419+0000","last_undegraded":"2024-09-21T12:21:48.928419+0000","last_fullsized":"2024-09-21T12:21:48.928419+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":90,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.919978+0000","last_change":"2024-09-21T12:21:41.137259+0000","last_active":"2024-09-21T12:21:48.919978+0000","last_peered":"2024-09-21T12:21:48.919978+0000","last_clean":"2024-09-21T12:21:48.919978+0000","last_became_active":"2024-09-21T12:21:41.135204+0000","last_became_peered":"2024-09-21T12:21:41.135204+0000","last_unstale":"2024-09-21T12:21:48.919978+0000","last_undegraded":"2024-09-21T12:21:48.919978+0000","last_fullsized":"2024-09-21T12:21:48.919978+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":82,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.931070+0000","last_change":"2024-09-21T12:21:20.931077+0000","last_active":"2024-09-21T12:21:48.931070+0000","last_peered":"2024-09-21T12:21:48.931070+0000","last_clean":"2024-09-21T12:21:48.931070+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:21:48.931070+0000","last_undegraded":"2024-09-21T12:21:48.931070+0000","last_fullsized":"2024-09-21T12:21:48.931070+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:20.931025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.931025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:50.767841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":72,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.926914+0000","last_change":"2024-09-21T12:20:21.308003+0000","last_active":"2024-09-21T12:21:48.926914+0000","last_peered":"2024-09-21T12:21:48.926914+0000","last_clean":"2024-09-21T12:21:48.926914+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:21:48.926914+0000","last_undegraded":"2024-09-21T12:21:48.926914+0000","last_fullsized":"2024-09-21T12:21:48.926914+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":97,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.929418+0000","last_change":"2024-09-21T12:20:17.695358+0000","last_active":"2024-09-21T12:21:48.929418+0000","last_peered":"2024-09-21T12:21:48.929418+0000","last_clean":"2024-09-21T12:21:48.929418+0000","last_became_active":"2024-09-21T12:20:17.694794+0000","last_became_peered":"2024-09-21T12:20:17.694794+0000","last_unstale":"2024-09-21T12:21:48.929418+0000","last_undegraded":"2024-09-21T12:21:48.929418+0000","last_fullsized":"2024-09-21T12:21:48.929418+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:32.861223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":83,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.931479+0000","last_change":"2024-09-21T12:20:15.296285+0000","last_active":"2024-09-21T12:21:48.931479+0000","last_peered":"2024-09-21T12:21:48.931479+0000","last_clean":"2024-09-21T12:21:48.931479+0000","last_became_active":"2024-09-21T12:20:15.295968+0000","last_became_peered":"2024-09-21T12:20:15.295968+0000","last_unstale":"2024-09-21T12:21:48.931479+0000","last_undegraded":"2024-09-21T12:21:48.931479+0000","last_fullsized":"2024-09-21T12:21:48.931479+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":82,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.919173+0000","last_change":"2024-09-21T12:21:41.131529+0000","last_active":"2024-09-21T12:21:48.919173+0000","last_peered":"2024-09-21T12:21:48.919173+0000","last_clean":"2024-09-21T12:21:48.919173+0000","last_became_active":"2024-09-21T12:21:41.020208+0000","last_became_peered":"2024-09-21T12:21:41.020208+0000","last_unstale":"2024-09-21T12:21:48.919173+0000","last_undegraded":"2024-09-21T12:21:48.919173+0000","last_fullsized":"2024-09-21T12:21:48.919173+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":70,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.931252+0000","last_change":"2024-09-21T12:20:21.595660+0000","last_active":"2024-09-21T12:21:48.931252+0000","last_peered":"2024-09-21T12:21:48.931252+0000","last_clean":"2024-09-21T12:21:48.931252+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:21:48.931252+0000","last_undegraded":"2024-09-21T12:21:48.931252+0000","last_fullsized":"2024-09-21T12:21:48.931252+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:27.917656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":84,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.926414+0000","last_change":"2024-09-21T12:20:17.304075+0000","last_active":"2024-09-21T12:21:48.926414+0000","last_peered":"2024-09-21T12:21:48.926414+0000","last_clean":"2024-09-21T12:21:48.926414+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:21:48.926414+0000","last_undegraded":"2024-09-21T12:21:48.926414+0000","last_fullsized":"2024-09-21T12:21:48.926414+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":90,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.917445+0000","last_change":"2024-09-21T12:20:15.298443+0000","last_active":"2024-09-21T12:21:48.917445+0000","last_peered":"2024-09-21T12:21:48.917445+0000","last_clean":"2024-09-21T12:21:48.917445+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:21:48.917445+0000","last_undegraded":"2024-09-21T12:21:48.917445+0000","last_fullsized":"2024-09-21T12:21:48.917445+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":82,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.909061+0000","last_change":"2024-09-21T12:21:41.142531+0000","last_active":"2024-09-21T12:21:48.909061+0000","last_peered":"2024-09-21T12:21:48.909061+0000","last_clean":"2024-09-21T12:21:48.909061+0000","last_became_active":"2024-09-21T12:21:41.142213+0000","last_became_peered":"2024-09-21T12:21:41.142213+0000","last_unstale":"2024-09-21T12:21:48.909061+0000","last_undegraded":"2024-09-21T12:21:48.909061+0000","last_fullsized":"2024-09-21T12:21:48.909061+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":78,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.919421+0000","last_change":"2024-09-21T12:21:41.136222+0000","last_active":"2024-09-21T12:21:48.919421+0000","last_peered":"2024-09-21T12:21:48.919421+0000","last_clean":"2024-09-21T12:21:48.919421+0000","last_became_active":"2024-09-21T12:21:41.135842+0000","last_became_peered":"2024-09-21T12:21:41.135842+0000","last_unstale":"2024-09-21T12:21:48.919421+0000","last_undegraded":"2024-09-21T12:21:48.919421+0000","last_fullsized":"2024-09-21T12:21:48.919421+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":79,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.923730+0000","last_change":"2024-09-21T12:21:41.014986+0000","last_active":"2024-09-21T12:21:48.923730+0000","last_peered":"2024-09-21T12:21:48.923730+0000","last_clean":"2024-09-21T12:21:48.923730+0000","last_became_active":"2024-09-21T12:21:41.014363+0000","last_became_peered":"2024-09-21T12:21:41.014363+0000","last_unstale":"2024-09-21T12:21:48.923730+0000","last_undegraded":"2024-09-21T12:21:48.923730+0000","last_fullsized":"2024-09-21T12:21:48.923730+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":82,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.926621+0000","last_change":"2024-09-21T12:20:15.298322+0000","last_active":"2024-09-21T12:21:48.926621+0000","last_peered":"2024-09-21T12:21:48.926621+0000","last_clean":"2024-09-21T12:21:48.926621+0000","last_became_active":"2024-09-21T12:20:15.297993+0000","last_became_peered":"2024-09-21T12:20:15.297993+0000","last_unstale":"2024-09-21T12:21:48.926621+0000","last_undegraded":"2024-09-21T12:21:48.926621+0000","last_fullsized":"2024-09-21T12:21:48.926621+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":98,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.909093+0000","last_change":"2024-09-21T12:20:17.299134+0000","last_active":"2024-09-21T12:21:48.909093+0000","last_peered":"2024-09-21T12:21:48.909093+0000","last_clean":"2024-09-21T12:21:48.909093+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:21:48.909093+0000","last_undegraded":"2024-09-21T12:21:48.909093+0000","last_fullsized":"2024-09-21T12:21:48.909093+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:31.737370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":76,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-21T12:21:48.918401+0000","last_change":"2024-09-21T12:20:19.308586+0000","last_active":"2024-09-21T12:21:48.918401+0000","last_peered":"2024-09-21T12:21:48.918401+0000","last_clean":"2024-09-21T12:21:48.918401+0000","last_became_active":"2024-09-21T12:20:19.308247+0000","last_became_peered":"2024-09-21T12:20:19.308247+0000","last_unstale":"2024-09-21T12:21:48.918401+0000","last_undegraded":"2024-09-21T12:21:48.918401+0000","last_fullsized":"2024-09-21T12:21:48.918401+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2367553536,"num_objects":36126,"num_object_clones":0,"num_object_copies":108378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":4467,"num_objects_unfound":0,"num_objects_dirty":36126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36126,"num_write_kb":2312064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3176,"num_bytes_recovered":208142336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7310082048,"data_stored":7310082048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36126,"ondisk_log_size":36126,"up":48,"acting":43,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":690,"num_read_kb":447,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":1474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":394,"ondisk_log_size":394,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":5023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":91,"num_read_kb":86,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234007,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":830276,"kb_used_data":799088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":92914876,"statfs":{"total":95995035648,"available":95144833024,"internally_reserved":0,"allocated":818266112,"data_stored":817691463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462943,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":926720,"kb_used_data":873328,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92818432,"statfs":{"total":95995035648,"available":95046074368,"internally_reserved":0,"allocated":894287872,"data_stored":893713014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724578,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":593524,"kb_used_data":562340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93151628,"statfs":{"total":95995035648,"available":95387267072,"internally_reserved":0,"allocated":575836160,"data_stored":575273925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953514,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149744,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1407832,"kb_used_data":1354440,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92337320,"statfs":{"total":95995035648,"available":94553415680,"internally_reserved":0,"allocated":1386946560,"data_stored":1386348200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411382,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":891176,"kb_used_data":847508,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":92853976,"statfs":{"total":95995035648,"available":95082471424,"internally_reserved":0,"allocated":867848192,"data_stored":867300798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574908,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":961392,"kb_used_data":916316,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":92783760,"statfs":{"total":95995035648,"available":95010570240,"internally_reserved":0,"allocated":938307584,"data_stored":937687984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738434,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1191240,"kb_used_data":1137848,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92553912,"statfs":{"total":95995035648,"available":94775205888,"internally_reserved":0,"allocated":1165156352,"data_stored":1164574224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1180565504,"data_stored":1180565504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":966656000,"data_stored":966656000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":890372096,"data_stored":890372096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1427701760,"data_stored":1427701760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":500891648,"data_stored":500891648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":582877184,"data_stored":582877184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":930283520,"data_stored":930283520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":830734336,"data_stored":830734336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:21:57.426 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T12:21:57.426 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pg_num 11 2024-09-21T12:21:57.463 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:21:57.487 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:57.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:57.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:21:57.589 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:57.628 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 27 16 43554 43538 100.758 84.6875 0.0108344 0.00991828 2024-09-21T12:21:57.651 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:57.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:57.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:57.754 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:57.803 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:57.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:57.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:57.905 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:21:57.925 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:57.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:57.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:21:58.027 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:21:58.052 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:58.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:58.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:21:58.154 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:58.182 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:58.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:58.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:58.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:57 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:21:58.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:57 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:21:58.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:57 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:21:58.283 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:21:58.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:58.306 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:58.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:21:58.407 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:58.435 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:58.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:58.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:58.537 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:58.558 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:58.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:58.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:58.625 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 28 16 44658 44642 99.6236 69 0.0111555 0.0100305 2024-09-21T12:21:58.659 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:21:58.699 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:58.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:58.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:21:58.800 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:21:58.826 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:58.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:58.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:21:58.928 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:58.967 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:58.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:58.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:59.069 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:21:59.094 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:59.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:59.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:21:59.195 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:21:59.233 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:59.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:59.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:21:59.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:58 smithi183 ceph-mon[33277]: pgmap v62: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 2.6 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 852 B/s rd, 94 MiB/s wr, 1.50k op/s; 2/126954 objects degraded (0.002%); 2435/126954 objects misplaced (1.918%); 37 MiB/s, 586 objects/s recovering 2024-09-21T12:21:59.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:58 smithi114 ceph-mon[30324]: pgmap v62: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 2.6 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 852 B/s rd, 94 MiB/s wr, 1.50k op/s; 2/126954 objects degraded (0.002%); 2435/126954 objects misplaced (1.918%); 37 MiB/s, 586 objects/s recovering 2024-09-21T12:21:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:58 smithi114 ceph-mon[40867]: pgmap v62: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 2.6 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 852 B/s rd, 94 MiB/s wr, 1.50k op/s; 2/126954 objects degraded (0.002%); 2435/126954 objects misplaced (1.918%); 37 MiB/s, 586 objects/s recovering 2024-09-21T12:21:59.335 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:59.357 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:59.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:59.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:59.458 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:21:59.487 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:59.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:59.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:21:59.590 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:21:59.625 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 29 16 45770 45754 98.5848 69.5 0.0166208 0.0101374 2024-09-21T12:21:59.631 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:21:59.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:59.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:21:59.733 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:21:59.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:59.762 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:59.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:21:59.863 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:21:59.889 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:21:59.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:21:59.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:21:59.991 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:00.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:00.019 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:00.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:00.120 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:00.142 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:00.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:00.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:00.188 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:59 smithi114 ceph-mon[40867]: 6.1a scrub starts 2024-09-21T12:22:00.188 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:21:59 smithi114 ceph-mon[40867]: 6.1a scrub ok 2024-09-21T12:22:00.188 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:59 smithi114 ceph-mon[30324]: 6.1a scrub starts 2024-09-21T12:22:00.188 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:21:59 smithi114 ceph-mon[30324]: 6.1a scrub ok 2024-09-21T12:22:00.231 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T12:22:00.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:59 smithi183 ceph-mon[33277]: 6.1a scrub starts 2024-09-21T12:22:00.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:21:59 smithi183 ceph-mon[33277]: 6.1a scrub ok 2024-09-21T12:22:00.243 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:00.264 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:00.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:00.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:00.366 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:00.389 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:00.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:00.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:00.491 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:00.514 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:00.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:00.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:00.616 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:00.625 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 30 16 46253 46237 96.3052 30.1875 0.00237659 0.0101337 2024-09-21T12:22:00.638 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:00.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:00.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:00.740 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:00.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:00.762 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:00.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:00.863 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:00.892 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:00.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:00.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:00.994 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:01.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:01.017 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:01.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:01.119 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:01.142 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:01.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:01.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:01.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:00 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:22:01.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:00 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:22:01.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:00 smithi183 ceph-mon[33277]: pgmap v63: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 2.6 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 657 B/s rd, 48 MiB/s wr, 769 op/s; 2/126954 objects degraded (0.002%); 2435/126954 objects misplaced (1.918%); 16 MiB/s, 260 objects/s recovering 2024-09-21T12:22:01.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:00 smithi183 ceph-mon[33277]: osdmap e99: 8 total, 8 up, 7 in 2024-09-21T12:22:01.244 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:01.267 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:01.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:01.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:01.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:00 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:22:01.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:00 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:22:01.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:00 smithi114 ceph-mon[30324]: pgmap v63: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 2.6 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 657 B/s rd, 48 MiB/s wr, 769 op/s; 2/126954 objects degraded (0.002%); 2435/126954 objects misplaced (1.918%); 16 MiB/s, 260 objects/s recovering 2024-09-21T12:22:01.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:00 smithi114 ceph-mon[30324]: osdmap e99: 8 total, 8 up, 7 in 2024-09-21T12:22:01.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:00 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:22:01.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:00 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:22:01.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:00 smithi114 ceph-mon[40867]: pgmap v63: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 2.6 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 657 B/s rd, 48 MiB/s wr, 769 op/s; 2/126954 objects degraded (0.002%); 2435/126954 objects misplaced (1.918%); 16 MiB/s, 260 objects/s recovering 2024-09-21T12:22:01.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:00 smithi114 ceph-mon[40867]: osdmap e99: 8 total, 8 up, 7 in 2024-09-21T12:22:01.369 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:01.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:01.391 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:01.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:01.492 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:01.522 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:01.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:01.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:01.527 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:22:01.529 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:22:01.624 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:01.626 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 31 16 46521 46505 93.7392 16.75 0.00237484 0.010464 2024-09-21T12:22:01.647 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:01.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:01.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:01.749 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:01.770 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:01.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:01.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:01.871 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:01.894 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:01.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:01.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:01.996 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:02.016 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:02.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:02.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:02.118 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:02.139 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:02.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:02.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:02.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:01 smithi183 ceph-mon[33277]: 4.8 scrub starts 2024-09-21T12:22:02.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:01 smithi183 ceph-mon[33277]: 4.8 scrub ok 2024-09-21T12:22:02.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:01 smithi183 ceph-mon[33277]: osdmap e100: 8 total, 8 up, 7 in 2024-09-21T12:22:02.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:01 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:02.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:01 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:22:02.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:01 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 2/126954 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T12:22:02.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:01 smithi183 ceph-mon[33277]: osdmap e101: 8 total, 8 up, 7 in 2024-09-21T12:22:02.241 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:02.262 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:02.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:02.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:02.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:01 smithi114 ceph-mon[30324]: 4.8 scrub starts 2024-09-21T12:22:02.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:01 smithi114 ceph-mon[30324]: 4.8 scrub ok 2024-09-21T12:22:02.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:01 smithi114 ceph-mon[30324]: osdmap e100: 8 total, 8 up, 7 in 2024-09-21T12:22:02.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:01 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:02.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:01 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:22:02.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:01 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 2/126954 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T12:22:02.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:01 smithi114 ceph-mon[30324]: osdmap e101: 8 total, 8 up, 7 in 2024-09-21T12:22:02.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:01 smithi114 ceph-mon[40867]: 4.8 scrub starts 2024-09-21T12:22:02.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:01 smithi114 ceph-mon[40867]: 4.8 scrub ok 2024-09-21T12:22:02.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:01 smithi114 ceph-mon[40867]: osdmap e100: 8 total, 8 up, 7 in 2024-09-21T12:22:02.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:01 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:02.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:01 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:22:02.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:01 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 2/126954 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T12:22:02.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:01 smithi114 ceph-mon[40867]: osdmap e101: 8 total, 8 up, 7 in 2024-09-21T12:22:02.363 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:02.388 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:02.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:02.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:02.489 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:02.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:02.515 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:02.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:02.616 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:02.625 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 32 16 46657 46641 91.0756 8.5 0.00234166 0.0107057 2024-09-21T12:22:02.639 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:02.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:02.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:02.740 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:02.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:02.767 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:02.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:02.868 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:02.908 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:02.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:02.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:02.936 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:22:02 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:22:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:22:03.011 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:03.047 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:03.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:03.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:03.148 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:03.179 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:03.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:03.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:03.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:02 smithi183 ceph-mon[33277]: pgmap v66: 148 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 2.7 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1.7 KiB/s rd, 85 MiB/s wr, 1.37k op/s; 1/133575 objects degraded (0.001%); 1777/133575 objects misplaced (1.330%); 28 MiB/s, 448 objects/s recovering 2024-09-21T12:22:03.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:02 smithi183 ceph-mon[33277]: 4.d scrub starts 2024-09-21T12:22:03.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:02 smithi183 ceph-mon[33277]: 4.d scrub ok 2024-09-21T12:22:03.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:02 smithi183 ceph-mon[33277]: 5.1 scrub starts 2024-09-21T12:22:03.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:02 smithi183 ceph-mon[33277]: 5.1 scrub ok 2024-09-21T12:22:03.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:02 smithi183 ceph-mon[33277]: osdmap e102: 8 total, 8 up, 7 in 2024-09-21T12:22:03.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:02 smithi114 ceph-mon[30324]: pgmap v66: 148 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 2.7 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1.7 KiB/s rd, 85 MiB/s wr, 1.37k op/s; 1/133575 objects degraded (0.001%); 1777/133575 objects misplaced (1.330%); 28 MiB/s, 448 objects/s recovering 2024-09-21T12:22:03.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:02 smithi114 ceph-mon[30324]: 4.d scrub starts 2024-09-21T12:22:03.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:02 smithi114 ceph-mon[30324]: 4.d scrub ok 2024-09-21T12:22:03.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:02 smithi114 ceph-mon[30324]: 5.1 scrub starts 2024-09-21T12:22:03.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:02 smithi114 ceph-mon[30324]: 5.1 scrub ok 2024-09-21T12:22:03.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:02 smithi114 ceph-mon[30324]: osdmap e102: 8 total, 8 up, 7 in 2024-09-21T12:22:03.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:02 smithi114 ceph-mon[40867]: pgmap v66: 148 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 143 active+clean; 2.7 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1.7 KiB/s rd, 85 MiB/s wr, 1.37k op/s; 1/133575 objects degraded (0.001%); 1777/133575 objects misplaced (1.330%); 28 MiB/s, 448 objects/s recovering 2024-09-21T12:22:03.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:02 smithi114 ceph-mon[40867]: 4.d scrub starts 2024-09-21T12:22:03.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:02 smithi114 ceph-mon[40867]: 4.d scrub ok 2024-09-21T12:22:03.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:02 smithi114 ceph-mon[40867]: 5.1 scrub starts 2024-09-21T12:22:03.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:02 smithi114 ceph-mon[40867]: 5.1 scrub ok 2024-09-21T12:22:03.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:02 smithi114 ceph-mon[40867]: osdmap e102: 8 total, 8 up, 7 in 2024-09-21T12:22:03.281 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:03.305 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:03.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:03.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:03.407 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:03.433 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:03.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:03.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:03.517 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:22:03.535 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:03.603 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:03.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:03.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:03.626 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 33 16 47886 47870 90.6432 76.8125 0.00442492 0.0110261 2024-09-21T12:22:03.705 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:03.732 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:03.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:03.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:03.834 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:03.860 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:03.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:03.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:03.961 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:03.987 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:03.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:03.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:04.089 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:04.110 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:04.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:04.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:04.212 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:04.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:03 smithi183 ceph-mon[33277]: 6.1e scrub starts 2024-09-21T12:22:04.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:03 smithi183 ceph-mon[33277]: 6.1e scrub ok 2024-09-21T12:22:04.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:03 smithi183 ceph-mon[33277]: 4.1a scrub starts 2024-09-21T12:22:04.249 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:04.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:04.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:04.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:03 smithi114 ceph-mon[30324]: 6.1e scrub starts 2024-09-21T12:22:04.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:03 smithi114 ceph-mon[30324]: 6.1e scrub ok 2024-09-21T12:22:04.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:03 smithi114 ceph-mon[30324]: 4.1a scrub starts 2024-09-21T12:22:04.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:03 smithi114 ceph-mon[40867]: 6.1e scrub starts 2024-09-21T12:22:04.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:03 smithi114 ceph-mon[40867]: 6.1e scrub ok 2024-09-21T12:22:04.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:03 smithi114 ceph-mon[40867]: 4.1a scrub starts 2024-09-21T12:22:04.351 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:04.389 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:04.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:04.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:04.491 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:04.532 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:04.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:04.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:04.626 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 34 16 49240 49224 90.4659 84.625 0.00271267 0.0110324 2024-09-21T12:22:04.634 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:04.659 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:04.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:04.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:04.761 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:04.787 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:04.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:04.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:04.889 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:04.910 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:04.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:04.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:04.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:04 smithi183 ceph-mon[33277]: 4.1a scrub ok 2024-09-21T12:22:04.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:04 smithi183 ceph-mon[33277]: pgmap v69: 148 pgs: 1 peering, 2 active+undersized+remapped, 145 active+clean; 2.8 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.10k op/s; 38 MiB/s, 609 objects/s recovering 2024-09-21T12:22:04.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:04 smithi183 ceph-mon[33277]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/133575 objects degraded (0.001%), 1 pg degraded) 2024-09-21T12:22:05.012 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:05.055 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:05.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:05.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:05.156 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:05.192 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:05.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:05.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:05.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:04 smithi114 ceph-mon[30324]: 4.1a scrub ok 2024-09-21T12:22:05.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:04 smithi114 ceph-mon[30324]: pgmap v69: 148 pgs: 1 peering, 2 active+undersized+remapped, 145 active+clean; 2.8 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.10k op/s; 38 MiB/s, 609 objects/s recovering 2024-09-21T12:22:05.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:04 smithi114 ceph-mon[30324]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/133575 objects degraded (0.001%), 1 pg degraded) 2024-09-21T12:22:05.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:04 smithi114 ceph-mon[40867]: 4.1a scrub ok 2024-09-21T12:22:05.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:04 smithi114 ceph-mon[40867]: pgmap v69: 148 pgs: 1 peering, 2 active+undersized+remapped, 145 active+clean; 2.8 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.10k op/s; 38 MiB/s, 609 objects/s recovering 2024-09-21T12:22:05.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:04 smithi114 ceph-mon[40867]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/133575 objects degraded (0.001%), 1 pg degraded) 2024-09-21T12:22:05.294 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:05.315 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:05.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:05.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:05.417 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:05.443 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:05.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:05.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:05.545 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:05.572 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:05.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:05.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:05.626 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 35 16 50355 50339 89.8722 69.6875 0.00949531 0.0111211 2024-09-21T12:22:05.673 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:05.697 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:05.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:05.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:05.799 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:05.834 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:05.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:05.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:05.936 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:05.962 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:05.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:05.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:06.064 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:06.090 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:06.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:06.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:06.192 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:06.217 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:06.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:06.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:06.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:05 smithi183 ceph-mon[33277]: 3.15 scrub starts 2024-09-21T12:22:06.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:05 smithi183 ceph-mon[33277]: 3.15 scrub ok 2024-09-21T12:22:06.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:05 smithi114 ceph-mon[30324]: 3.15 scrub starts 2024-09-21T12:22:06.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:05 smithi114 ceph-mon[30324]: 3.15 scrub ok 2024-09-21T12:22:06.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:05 smithi114 ceph-mon[40867]: 3.15 scrub starts 2024-09-21T12:22:06.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:05 smithi114 ceph-mon[40867]: 3.15 scrub ok 2024-09-21T12:22:06.318 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:06.344 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:06.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:06.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:06.446 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:06.471 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:06.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:06.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:06.573 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:06.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:06.606 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:06.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:06.626 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 36 16 51724 51708 89.7522 85.5625 0.0045698 0.0111277 2024-09-21T12:22:06.708 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:06.744 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:06.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:06.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:06.845 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:06.870 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:06.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:06.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:06.961 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:22:06 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:22:06.972 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:06.997 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:06.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:06.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:07.098 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:07.128 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:07.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:07.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:07.230 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:07.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:06 smithi183 ceph-mon[33277]: 3.3 scrub starts 2024-09-21T12:22:07.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:06 smithi183 ceph-mon[33277]: 3.3 scrub ok 2024-09-21T12:22:07.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:06 smithi183 ceph-mon[33277]: pgmap v70: 148 pgs: 1 peering, 2 active+undersized+remapped, 145 active+clean; 2.8 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 894 B/s rd, 48 MiB/s wr, 767 op/s; 27 MiB/s, 425 objects/s recovering 2024-09-21T12:22:07.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:06 smithi183 ceph-mon[33277]: 3.9 scrub starts 2024-09-21T12:22:07.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:06 smithi183 ceph-mon[33277]: 3.9 scrub ok 2024-09-21T12:22:07.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:06 smithi183 ceph-mon[33277]: 5.16 scrub starts 2024-09-21T12:22:07.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:06 smithi183 ceph-mon[33277]: 5.16 scrub ok 2024-09-21T12:22:07.274 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:07.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:06 smithi114 ceph-mon[40867]: 3.3 scrub starts 2024-09-21T12:22:07.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:06 smithi114 ceph-mon[40867]: 3.3 scrub ok 2024-09-21T12:22:07.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:06 smithi114 ceph-mon[40867]: pgmap v70: 148 pgs: 1 peering, 2 active+undersized+remapped, 145 active+clean; 2.8 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 894 B/s rd, 48 MiB/s wr, 767 op/s; 27 MiB/s, 425 objects/s recovering 2024-09-21T12:22:07.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:06 smithi114 ceph-mon[40867]: 3.9 scrub starts 2024-09-21T12:22:07.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:06 smithi114 ceph-mon[40867]: 3.9 scrub ok 2024-09-21T12:22:07.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:06 smithi114 ceph-mon[40867]: 5.16 scrub starts 2024-09-21T12:22:07.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:06 smithi114 ceph-mon[40867]: 5.16 scrub ok 2024-09-21T12:22:07.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:06 smithi114 ceph-mon[30324]: 3.3 scrub starts 2024-09-21T12:22:07.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:06 smithi114 ceph-mon[30324]: 3.3 scrub ok 2024-09-21T12:22:07.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:06 smithi114 ceph-mon[30324]: pgmap v70: 148 pgs: 1 peering, 2 active+undersized+remapped, 145 active+clean; 2.8 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 894 B/s rd, 48 MiB/s wr, 767 op/s; 27 MiB/s, 425 objects/s recovering 2024-09-21T12:22:07.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:06 smithi114 ceph-mon[30324]: 3.9 scrub starts 2024-09-21T12:22:07.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:06 smithi114 ceph-mon[30324]: 3.9 scrub ok 2024-09-21T12:22:07.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:06 smithi114 ceph-mon[30324]: 5.16 scrub starts 2024-09-21T12:22:07.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:06 smithi114 ceph-mon[30324]: 5.16 scrub ok 2024-09-21T12:22:07.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:07.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:07.377 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:07.401 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:07.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:07.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:07.504 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:07.530 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:07.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:07.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:07.626 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 37 15 53005 52990 89.4918 80.125 0.0125694 0.0111688 2024-09-21T12:22:07.632 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:07.668 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:07.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:07.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:07.770 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:07.791 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:07.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:07.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:07.892 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:07.919 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:07.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:07.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:08.021 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:08.046 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:08.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:08.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:08.147 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:08.174 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:08.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:08.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:08.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:07 smithi183 ceph-mon[33277]: 5.5 scrub starts 2024-09-21T12:22:08.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:07 smithi183 ceph-mon[33277]: 5.5 scrub ok 2024-09-21T12:22:08.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:07 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:22:08.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:07 smithi183 ceph-mon[33277]: 5.18 scrub starts 2024-09-21T12:22:08.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:07 smithi183 ceph-mon[33277]: 5.18 scrub ok 2024-09-21T12:22:08.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:07 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:22:08.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:07 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:22:08.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:07 smithi183 ceph-mon[33277]: 5.f scrub starts 2024-09-21T12:22:08.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:07 smithi183 ceph-mon[33277]: 5.f scrub ok 2024-09-21T12:22:08.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:07 smithi183 ceph-mon[33277]: 3.11 scrub starts 2024-09-21T12:22:08.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:07 smithi183 ceph-mon[33277]: 3.11 scrub ok 2024-09-21T12:22:08.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:07 smithi114 ceph-mon[40867]: 5.5 scrub starts 2024-09-21T12:22:08.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:07 smithi114 ceph-mon[40867]: 5.5 scrub ok 2024-09-21T12:22:08.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:07 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:22:08.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:07 smithi114 ceph-mon[40867]: 5.18 scrub starts 2024-09-21T12:22:08.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:07 smithi114 ceph-mon[40867]: 5.18 scrub ok 2024-09-21T12:22:08.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:07 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:22:08.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:07 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:22:08.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:07 smithi114 ceph-mon[40867]: 5.f scrub starts 2024-09-21T12:22:08.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:07 smithi114 ceph-mon[40867]: 5.f scrub ok 2024-09-21T12:22:08.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:07 smithi114 ceph-mon[40867]: 3.11 scrub starts 2024-09-21T12:22:08.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:07 smithi114 ceph-mon[40867]: 3.11 scrub ok 2024-09-21T12:22:08.275 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:08.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:07 smithi114 ceph-mon[30324]: 5.5 scrub starts 2024-09-21T12:22:08.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:07 smithi114 ceph-mon[30324]: 5.5 scrub ok 2024-09-21T12:22:08.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:07 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:22:08.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:07 smithi114 ceph-mon[30324]: 5.18 scrub starts 2024-09-21T12:22:08.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:07 smithi114 ceph-mon[30324]: 5.18 scrub ok 2024-09-21T12:22:08.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:07 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:22:08.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:07 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:22:08.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:07 smithi114 ceph-mon[30324]: 5.f scrub starts 2024-09-21T12:22:08.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:07 smithi114 ceph-mon[30324]: 5.f scrub ok 2024-09-21T12:22:08.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:07 smithi114 ceph-mon[30324]: 3.11 scrub starts 2024-09-21T12:22:08.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:07 smithi114 ceph-mon[30324]: 3.11 scrub ok 2024-09-21T12:22:08.301 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:08.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:08.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:08.402 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:08.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:08.442 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:08.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:08.543 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:08.578 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:08.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:08.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:08.626 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 38 16 54327 54311 89.3093 82.5625 0.00873403 0.0111915 2024-09-21T12:22:08.679 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:08.707 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:08.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:08.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:08.809 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:08.840 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:08.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:08.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:08.942 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:08.980 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:08.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:08.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:09.082 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:09.107 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:09.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:09.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:09.208 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:09.234 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:09.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:09.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:09.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:08 smithi183 ceph-mon[33277]: pgmap v71: 148 pgs: 148 active+clean; 3.2 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.5 KiB/s rd, 89 MiB/s wr, 1.43k op/s; 23 MiB/s, 362 objects/s recovering 2024-09-21T12:22:09.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:08 smithi183 ceph-mon[33277]: 3.6 scrub starts 2024-09-21T12:22:09.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:08 smithi183 ceph-mon[33277]: 3.6 scrub ok 2024-09-21T12:22:09.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:08 smithi183 ceph-mon[33277]: 3.1a scrub starts 2024-09-21T12:22:09.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:08 smithi183 ceph-mon[33277]: 3.1a scrub ok 2024-09-21T12:22:09.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:08 smithi183 ceph-mon[33277]: osdmap e103: 8 total, 8 up, 7 in 2024-09-21T12:22:09.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:08 smithi114 ceph-mon[40867]: pgmap v71: 148 pgs: 148 active+clean; 3.2 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.5 KiB/s rd, 89 MiB/s wr, 1.43k op/s; 23 MiB/s, 362 objects/s recovering 2024-09-21T12:22:09.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:08 smithi114 ceph-mon[40867]: 3.6 scrub starts 2024-09-21T12:22:09.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:08 smithi114 ceph-mon[40867]: 3.6 scrub ok 2024-09-21T12:22:09.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:08 smithi114 ceph-mon[40867]: 3.1a scrub starts 2024-09-21T12:22:09.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:08 smithi114 ceph-mon[40867]: 3.1a scrub ok 2024-09-21T12:22:09.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:08 smithi114 ceph-mon[40867]: osdmap e103: 8 total, 8 up, 7 in 2024-09-21T12:22:09.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:08 smithi114 ceph-mon[30324]: pgmap v71: 148 pgs: 148 active+clean; 3.2 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.5 KiB/s rd, 89 MiB/s wr, 1.43k op/s; 23 MiB/s, 362 objects/s recovering 2024-09-21T12:22:09.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:08 smithi114 ceph-mon[30324]: 3.6 scrub starts 2024-09-21T12:22:09.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:08 smithi114 ceph-mon[30324]: 3.6 scrub ok 2024-09-21T12:22:09.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:08 smithi114 ceph-mon[30324]: 3.1a scrub starts 2024-09-21T12:22:09.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:08 smithi114 ceph-mon[30324]: 3.1a scrub ok 2024-09-21T12:22:09.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:08 smithi114 ceph-mon[30324]: osdmap e103: 8 total, 8 up, 7 in 2024-09-21T12:22:09.336 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:09.358 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:09.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:09.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:09.460 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:09.493 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:09.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:09.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:09.595 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:09.620 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:09.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:09.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:09.626 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 39 16 55502 55486 88.9022 73.4375 0.00836163 0.0112423 2024-09-21T12:22:09.722 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:09.762 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:09.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:09.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:09.864 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:09.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:09.900 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:09.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:10.001 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:10.025 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:10.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:10.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:10.126 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:10.167 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:10.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:10.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:10.197 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:09 smithi114 ceph-mon[40867]: 5.1f scrub starts 2024-09-21T12:22:10.197 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:09 smithi114 ceph-mon[40867]: 5.1f scrub ok 2024-09-21T12:22:10.197 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:09 smithi114 ceph-mon[40867]: 3.b scrub starts 2024-09-21T12:22:10.197 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:09 smithi114 ceph-mon[40867]: 3.b scrub ok 2024-09-21T12:22:10.197 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:09 smithi114 ceph-mon[30324]: 5.1f scrub starts 2024-09-21T12:22:10.197 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:09 smithi114 ceph-mon[30324]: 5.1f scrub ok 2024-09-21T12:22:10.198 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:09 smithi114 ceph-mon[30324]: 3.b scrub starts 2024-09-21T12:22:10.198 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:09 smithi114 ceph-mon[30324]: 3.b scrub ok 2024-09-21T12:22:10.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:09 smithi183 ceph-mon[33277]: 5.1f scrub starts 2024-09-21T12:22:10.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:09 smithi183 ceph-mon[33277]: 5.1f scrub ok 2024-09-21T12:22:10.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:09 smithi183 ceph-mon[33277]: 3.b scrub starts 2024-09-21T12:22:10.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:09 smithi183 ceph-mon[33277]: 3.b scrub ok 2024-09-21T12:22:10.269 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:10.294 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:10.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:10.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:10.396 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:10.441 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:10.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:10.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:10.543 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:10.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:10.582 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:10.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:10.626 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-21T12:22:10.626016+0000 min lat: 0.00135771 max lat: 2.08441 avg lat: 0.011283 2024-09-21T12:22:10.626 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T12:22:10.626 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 40 16 56724 56708 88.5889 76.375 0.0112575 0.011283 2024-09-21T12:22:10.683 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:10.707 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:10.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:10.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:10.809 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:10.851 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:10.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:10.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:10.953 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:10.987 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:10.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:10.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:11.088 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:11.125 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:11.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:11.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:11.227 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:11.263 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:11.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:11.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:11.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:10 smithi114 ceph-mon[30324]: pgmap v73: 148 pgs: 148 active+clean; 3.2 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 656 B/s rd, 42 MiB/s wr, 665 op/s 2024-09-21T12:22:11.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:10 smithi114 ceph-mon[30324]: 5.11 scrub starts 2024-09-21T12:22:11.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:10 smithi114 ceph-mon[30324]: 5.11 scrub ok 2024-09-21T12:22:11.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:10 smithi114 ceph-mon[30324]: 4.9 scrub starts 2024-09-21T12:22:11.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:10 smithi114 ceph-mon[30324]: 4.9 scrub ok 2024-09-21T12:22:11.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:10 smithi114 ceph-mon[30324]: 4.1b scrub starts 2024-09-21T12:22:11.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:10 smithi114 ceph-mon[30324]: 4.1b scrub ok 2024-09-21T12:22:11.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:10 smithi114 ceph-mon[40867]: pgmap v73: 148 pgs: 148 active+clean; 3.2 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 656 B/s rd, 42 MiB/s wr, 665 op/s 2024-09-21T12:22:11.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:10 smithi114 ceph-mon[40867]: 5.11 scrub starts 2024-09-21T12:22:11.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:10 smithi114 ceph-mon[40867]: 5.11 scrub ok 2024-09-21T12:22:11.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:10 smithi114 ceph-mon[40867]: 4.9 scrub starts 2024-09-21T12:22:11.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:10 smithi114 ceph-mon[40867]: 4.9 scrub ok 2024-09-21T12:22:11.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:10 smithi114 ceph-mon[40867]: 4.1b scrub starts 2024-09-21T12:22:11.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:10 smithi114 ceph-mon[40867]: 4.1b scrub ok 2024-09-21T12:22:11.366 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:11.403 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:11.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:11.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:11.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:10 smithi183 ceph-mon[33277]: pgmap v73: 148 pgs: 148 active+clean; 3.2 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 656 B/s rd, 42 MiB/s wr, 665 op/s 2024-09-21T12:22:11.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:10 smithi183 ceph-mon[33277]: 5.11 scrub starts 2024-09-21T12:22:11.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:10 smithi183 ceph-mon[33277]: 5.11 scrub ok 2024-09-21T12:22:11.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:10 smithi183 ceph-mon[33277]: 4.9 scrub starts 2024-09-21T12:22:11.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:10 smithi183 ceph-mon[33277]: 4.9 scrub ok 2024-09-21T12:22:11.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:10 smithi183 ceph-mon[33277]: 4.1b scrub starts 2024-09-21T12:22:11.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:10 smithi183 ceph-mon[33277]: 4.1b scrub ok 2024-09-21T12:22:11.505 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:11.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:11.540 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:11.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:11.626 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 41 16 58107 58091 88.5362 86.4375 0.0069225 0.0112893 2024-09-21T12:22:11.642 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:11.678 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:11.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:11.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:11.780 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:11.806 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:11.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:11.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:11.908 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:11.952 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:11.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:11.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:12.054 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:12.078 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:12.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:12.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:12.180 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:12.236 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:12.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:12.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:12.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:12 smithi114 ceph-mon[30324]: 5.1d scrub starts 2024-09-21T12:22:12.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:12 smithi114 ceph-mon[30324]: 5.1d scrub ok 2024-09-21T12:22:12.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:12 smithi114 ceph-mon[30324]: osdmap e104: 8 total, 8 up, 7 in 2024-09-21T12:22:12.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:12 smithi114 ceph-mon[40867]: 5.1d scrub starts 2024-09-21T12:22:12.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:12 smithi114 ceph-mon[40867]: 5.1d scrub ok 2024-09-21T12:22:12.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:12 smithi114 ceph-mon[40867]: osdmap e104: 8 total, 8 up, 7 in 2024-09-21T12:22:12.310 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:22:12.338 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:12.365 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:12.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:12.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:12.467 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:11 smithi183 ceph-mon[33277]: 5.1d scrub starts 2024-09-21T12:22:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:11 smithi183 ceph-mon[33277]: 5.1d scrub ok 2024-09-21T12:22:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:11 smithi183 ceph-mon[33277]: osdmap e104: 8 total, 8 up, 7 in 2024-09-21T12:22:12.494 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:12.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:12.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:12.596 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:12.618 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:12.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:12.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:12.626 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 42 16 59163 59147 87.9995 66 0.0192683 0.0113517 2024-09-21T12:22:12.720 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:12.747 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:12.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:12.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:12.848 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:12.891 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:12.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:12.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:12.993 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:13.003 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:22:12 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:22:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:22:13.027 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-21T12:22:13.030 INFO:teuthology.orchestra.run.smithi114.stderr:set pool 1 pg_num to 11 2024-09-21T12:22:13.041 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:13.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:13.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:13.143 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:13.169 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:13.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:13.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:13.271 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:13.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:13 smithi114 ceph-mon[40867]: pgmap v74: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.1 KiB/s rd, 56 MiB/s wr, 902 op/s 2024-09-21T12:22:13.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:13 smithi114 ceph-mon[40867]: 5.a scrub starts 2024-09-21T12:22:13.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:13 smithi114 ceph-mon[40867]: 5.a scrub ok 2024-09-21T12:22:13.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:13 smithi114 ceph-mon[40867]: 6.d scrub starts 2024-09-21T12:22:13.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:13 smithi114 ceph-mon[40867]: 6.d scrub ok 2024-09-21T12:22:13.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:13 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2921511525' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T12:22:13.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:13 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T12:22:13.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:13 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/639393539' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:22:13.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:13 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:22:13.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:13 smithi114 ceph-mon[30324]: pgmap v74: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.1 KiB/s rd, 56 MiB/s wr, 902 op/s 2024-09-21T12:22:13.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:13 smithi114 ceph-mon[30324]: 5.a scrub starts 2024-09-21T12:22:13.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:13 smithi114 ceph-mon[30324]: 5.a scrub ok 2024-09-21T12:22:13.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:13 smithi114 ceph-mon[30324]: 6.d scrub starts 2024-09-21T12:22:13.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:13 smithi114 ceph-mon[30324]: 6.d scrub ok 2024-09-21T12:22:13.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:13 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2921511525' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T12:22:13.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:13 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T12:22:13.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:13 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/639393539' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:22:13.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:13 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:22:13.297 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:13.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:13.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:13.399 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:13.441 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:13.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:13.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:13.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:13 smithi183 ceph-mon[33277]: pgmap v74: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.1 KiB/s rd, 56 MiB/s wr, 902 op/s 2024-09-21T12:22:13.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:13 smithi183 ceph-mon[33277]: 5.a scrub starts 2024-09-21T12:22:13.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:13 smithi183 ceph-mon[33277]: 5.a scrub ok 2024-09-21T12:22:13.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:13 smithi183 ceph-mon[33277]: 6.d scrub starts 2024-09-21T12:22:13.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:13 smithi183 ceph-mon[33277]: 6.d scrub ok 2024-09-21T12:22:13.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:13 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2921511525' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T12:22:13.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:13 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T12:22:13.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:13 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/639393539' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:22:13.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:13 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:22:13.543 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:13.594 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:13.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:13.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:13.626 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 43 16 60464 60448 87.8439 81.3125 0.0137605 0.0113784 2024-09-21T12:22:13.696 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:13.723 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:13.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:13.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:13.824 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:13.864 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:13.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:13.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:13.966 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:13.994 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:13.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:13.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:14.096 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:14.124 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:14.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:14.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:14.226 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:14.251 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:14.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:14.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:14.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:14 smithi114 ceph-mon[40867]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:22:14.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:14 smithi114 ceph-mon[40867]: Cluster is now healthy 2024-09-21T12:22:14.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:14 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-21T12:22:14.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:14 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:22:14.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:14 smithi114 ceph-mon[40867]: osdmap e105: 8 total, 8 up, 7 in 2024-09-21T12:22:14.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:14 smithi114 ceph-mon[40867]: 6.2 scrub starts 2024-09-21T12:22:14.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:14 smithi114 ceph-mon[40867]: 6.2 scrub ok 2024-09-21T12:22:14.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:14 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-21T12:22:14.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:14 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-21T12:22:14.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:14 smithi114 ceph-mon[30324]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:22:14.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:14 smithi114 ceph-mon[30324]: Cluster is now healthy 2024-09-21T12:22:14.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:14 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-21T12:22:14.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:14 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:22:14.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:14 smithi114 ceph-mon[30324]: osdmap e105: 8 total, 8 up, 7 in 2024-09-21T12:22:14.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:14 smithi114 ceph-mon[30324]: 6.2 scrub starts 2024-09-21T12:22:14.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:14 smithi114 ceph-mon[30324]: 6.2 scrub ok 2024-09-21T12:22:14.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:14 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-21T12:22:14.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:14 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-21T12:22:14.353 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:14.378 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:14.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:14.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:14.479 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:14.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:14 smithi183 ceph-mon[33277]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:22:14.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:14 smithi183 ceph-mon[33277]: Cluster is now healthy 2024-09-21T12:22:14.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:14 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-21T12:22:14.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:14 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:22:14.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:14 smithi183 ceph-mon[33277]: osdmap e105: 8 total, 8 up, 7 in 2024-09-21T12:22:14.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:14 smithi183 ceph-mon[33277]: 6.2 scrub starts 2024-09-21T12:22:14.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:14 smithi183 ceph-mon[33277]: 6.2 scrub ok 2024-09-21T12:22:14.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:14 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-21T12:22:14.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:14 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-21T12:22:14.507 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:14.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:14.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:14.608 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:14.627 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 44 16 61801 61785 87.7464 83.5625 0.00492331 0.0113911 2024-09-21T12:22:14.636 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:14.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:14.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:14.738 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:14.777 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:14.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:14.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:14.878 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:14.917 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:14.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:14.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:15.019 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:15.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:15.048 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:15.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:15.149 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:15.176 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:15.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:15.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:15.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:15 smithi183 ceph-mon[33277]: pgmap v77: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-21T12:22:15.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:15 smithi183 ceph-mon[33277]: 3.1 scrub starts 2024-09-21T12:22:15.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:15 smithi183 ceph-mon[33277]: 3.1 scrub ok 2024-09-21T12:22:15.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:15 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-21T12:22:15.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:15 smithi183 ceph-mon[33277]: osdmap e106: 8 total, 8 up, 7 in 2024-09-21T12:22:15.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:15 smithi183 ceph-mon[33277]: 3.8 scrub starts 2024-09-21T12:22:15.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:15 smithi183 ceph-mon[33277]: 3.8 scrub ok 2024-09-21T12:22:15.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:15 smithi183 ceph-mon[33277]: 4.1 scrub starts 2024-09-21T12:22:15.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:15 smithi183 ceph-mon[33277]: 4.1 scrub ok 2024-09-21T12:22:15.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:15 smithi114 ceph-mon[40867]: pgmap v77: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-21T12:22:15.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:15 smithi114 ceph-mon[40867]: 3.1 scrub starts 2024-09-21T12:22:15.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:15 smithi114 ceph-mon[40867]: 3.1 scrub ok 2024-09-21T12:22:15.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:15 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-21T12:22:15.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:15 smithi114 ceph-mon[40867]: osdmap e106: 8 total, 8 up, 7 in 2024-09-21T12:22:15.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:15 smithi114 ceph-mon[40867]: 3.8 scrub starts 2024-09-21T12:22:15.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:15 smithi114 ceph-mon[40867]: 3.8 scrub ok 2024-09-21T12:22:15.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:15 smithi114 ceph-mon[40867]: 4.1 scrub starts 2024-09-21T12:22:15.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:15 smithi114 ceph-mon[40867]: 4.1 scrub ok 2024-09-21T12:22:15.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:15 smithi114 ceph-mon[30324]: pgmap v77: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-21T12:22:15.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:15 smithi114 ceph-mon[30324]: 3.1 scrub starts 2024-09-21T12:22:15.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:15 smithi114 ceph-mon[30324]: 3.1 scrub ok 2024-09-21T12:22:15.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:15 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-21T12:22:15.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:15 smithi114 ceph-mon[30324]: osdmap e106: 8 total, 8 up, 7 in 2024-09-21T12:22:15.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:15 smithi114 ceph-mon[30324]: 3.8 scrub starts 2024-09-21T12:22:15.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:15 smithi114 ceph-mon[30324]: 3.8 scrub ok 2024-09-21T12:22:15.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:15 smithi114 ceph-mon[30324]: 4.1 scrub starts 2024-09-21T12:22:15.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:15 smithi114 ceph-mon[30324]: 4.1 scrub ok 2024-09-21T12:22:15.278 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:15.304 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:15.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:15.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:15.406 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:15.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:15.441 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:15.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:15.543 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:15.579 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:15.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:15.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:15.627 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 45 16 63103 63087 87.6046 81.375 0.00795702 0.0114023 2024-09-21T12:22:15.681 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:15.713 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:15.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:15.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:15.816 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:15.841 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:15.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:15.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:15.942 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:15.982 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:15.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:15.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:16.084 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:16.110 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:16.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:16.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:16.213 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:16.248 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:16.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:16.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:16.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:16 smithi114 ceph-mon[30324]: 3.17 scrub starts 2024-09-21T12:22:16.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:16 smithi114 ceph-mon[30324]: 3.17 scrub ok 2024-09-21T12:22:16.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:16 smithi114 ceph-mon[30324]: 6.e scrub starts 2024-09-21T12:22:16.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:16 smithi114 ceph-mon[30324]: 6.e scrub ok 2024-09-21T12:22:16.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:16 smithi114 ceph-mon[30324]: 3.f scrub starts 2024-09-21T12:22:16.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:16 smithi114 ceph-mon[30324]: 3.f scrub ok 2024-09-21T12:22:16.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:16 smithi114 ceph-mon[30324]: 5.1b scrub starts 2024-09-21T12:22:16.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:16 smithi114 ceph-mon[30324]: 5.1b scrub ok 2024-09-21T12:22:16.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:16 smithi114 ceph-mon[30324]: osdmap e107: 8 total, 8 up, 7 in 2024-09-21T12:22:16.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:16 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-21T12:22:16.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:16 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-21T12:22:16.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:16 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:16.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:16 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:22:16.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:16 smithi114 ceph-mon[40867]: 3.17 scrub starts 2024-09-21T12:22:16.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:16 smithi114 ceph-mon[40867]: 3.17 scrub ok 2024-09-21T12:22:16.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:16 smithi114 ceph-mon[40867]: 6.e scrub starts 2024-09-21T12:22:16.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:16 smithi114 ceph-mon[40867]: 6.e scrub ok 2024-09-21T12:22:16.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:16 smithi114 ceph-mon[40867]: 3.f scrub starts 2024-09-21T12:22:16.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:16 smithi114 ceph-mon[40867]: 3.f scrub ok 2024-09-21T12:22:16.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:16 smithi114 ceph-mon[40867]: 5.1b scrub starts 2024-09-21T12:22:16.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:16 smithi114 ceph-mon[40867]: 5.1b scrub ok 2024-09-21T12:22:16.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:16 smithi114 ceph-mon[40867]: osdmap e107: 8 total, 8 up, 7 in 2024-09-21T12:22:16.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:16 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-21T12:22:16.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:16 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-21T12:22:16.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:16 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:16.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:16 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:22:16.350 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:16.374 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:16.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:16.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:16.392 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:22:16.477 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:16 smithi183 ceph-mon[33277]: 3.17 scrub starts 2024-09-21T12:22:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:16 smithi183 ceph-mon[33277]: 3.17 scrub ok 2024-09-21T12:22:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:16 smithi183 ceph-mon[33277]: 6.e scrub starts 2024-09-21T12:22:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:16 smithi183 ceph-mon[33277]: 6.e scrub ok 2024-09-21T12:22:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:16 smithi183 ceph-mon[33277]: 3.f scrub starts 2024-09-21T12:22:16.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:16 smithi183 ceph-mon[33277]: 3.f scrub ok 2024-09-21T12:22:16.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:16 smithi183 ceph-mon[33277]: 5.1b scrub starts 2024-09-21T12:22:16.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:16 smithi183 ceph-mon[33277]: 5.1b scrub ok 2024-09-21T12:22:16.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:16 smithi183 ceph-mon[33277]: osdmap e107: 8 total, 8 up, 7 in 2024-09-21T12:22:16.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:16 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-21T12:22:16.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:16 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-21T12:22:16.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:16 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:16.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:16 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:22:16.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:16.523 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:16.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:16.624 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:16.627 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 46 15 64353 64338 87.3989 78.1875 0.0186963 0.0114364 2024-09-21T12:22:16.672 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:16.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:16.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:16.774 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:16.815 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:16.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:16.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:16.916 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:16.919 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T12:22:16.957 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:16.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:16.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:16.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:22:16 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:22:17.059 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:17.087 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:17.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:17.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:17.188 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:17.211 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:17.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:17.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:17.231 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:17 smithi114 ceph-mon[30324]: pgmap v80: 158 pgs: 10 unknown, 148 active+clean; 3.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 57 MiB/s wr, 913 op/s 2024-09-21T12:22:17.231 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:17 smithi114 ceph-mon[30324]: 6.19 scrub starts 2024-09-21T12:22:17.231 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:17 smithi114 ceph-mon[30324]: 6.19 scrub ok 2024-09-21T12:22:17.232 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:17 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-21T12:22:17.232 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:17 smithi114 ceph-mon[30324]: osdmap e108: 8 total, 8 up, 7 in 2024-09-21T12:22:17.232 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:17 smithi114 ceph-mon[40867]: pgmap v80: 158 pgs: 10 unknown, 148 active+clean; 3.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 57 MiB/s wr, 913 op/s 2024-09-21T12:22:17.232 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:17 smithi114 ceph-mon[40867]: 6.19 scrub starts 2024-09-21T12:22:17.232 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:17 smithi114 ceph-mon[40867]: 6.19 scrub ok 2024-09-21T12:22:17.232 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:17 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-21T12:22:17.232 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:17 smithi114 ceph-mon[40867]: osdmap e108: 8 total, 8 up, 7 in 2024-09-21T12:22:17.232 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:22:17 smithi114 systemd[1]: Stopping Ceph alertmanager.a for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:22:17.313 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:17.359 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:17.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:17.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:17.460 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:17.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:17 smithi183 ceph-mon[33277]: pgmap v80: 158 pgs: 10 unknown, 148 active+clean; 3.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 57 MiB/s wr, 913 op/s 2024-09-21T12:22:17.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:17 smithi183 ceph-mon[33277]: 6.19 scrub starts 2024-09-21T12:22:17.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:17 smithi183 ceph-mon[33277]: 6.19 scrub ok 2024-09-21T12:22:17.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:17 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-21T12:22:17.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:17 smithi183 ceph-mon[33277]: osdmap e108: 8 total, 8 up, 7 in 2024-09-21T12:22:17.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:17.508 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:17.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:17.522 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:22:17.522 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:22:17.522 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:22:17.522 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:22:17.522 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:22:17.609 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:17.627 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 47 16 65549 65533 87.1284 74.6875 0.0092762 0.011472 2024-09-21T12:22:17.649 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:17.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:17.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:17.751 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:17.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:17.799 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:17.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:17.900 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:17.953 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:17.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:17.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:18.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:18.100 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:18.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:18.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:18.202 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:18.240 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:18.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:18.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:18.341 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:18.370 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:18.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:18.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:18.472 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:18.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:18 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:22:18.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:18 smithi114 ceph-mon[30324]: 3.a scrub starts 2024-09-21T12:22:18.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:18 smithi114 ceph-mon[30324]: 3.a scrub ok 2024-09-21T12:22:18.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:18 smithi114 ceph-mon[30324]: 3.1f scrub starts 2024-09-21T12:22:18.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:18 smithi114 ceph-mon[30324]: 3.1f scrub ok 2024-09-21T12:22:18.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:18 smithi114 ceph-mon[30324]: 1.0 scrub starts 2024-09-21T12:22:18.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:18 smithi114 ceph-mon[30324]: 1.0 scrub ok 2024-09-21T12:22:18.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:18 smithi114 ceph-mon[30324]: osdmap e109: 8 total, 8 up, 7 in 2024-09-21T12:22:18.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:18 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:22:18.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:18 smithi114 ceph-mon[40867]: 3.a scrub starts 2024-09-21T12:22:18.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:18 smithi114 ceph-mon[40867]: 3.a scrub ok 2024-09-21T12:22:18.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:18 smithi114 ceph-mon[40867]: 3.1f scrub starts 2024-09-21T12:22:18.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:18 smithi114 ceph-mon[40867]: 3.1f scrub ok 2024-09-21T12:22:18.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:18 smithi114 ceph-mon[40867]: 1.0 scrub starts 2024-09-21T12:22:18.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:18 smithi114 ceph-mon[40867]: 1.0 scrub ok 2024-09-21T12:22:18.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:18 smithi114 ceph-mon[40867]: osdmap e109: 8 total, 8 up, 7 in 2024-09-21T12:22:18.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:18 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:22:18.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:18 smithi183 ceph-mon[33277]: 3.a scrub starts 2024-09-21T12:22:18.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:18 smithi183 ceph-mon[33277]: 3.a scrub ok 2024-09-21T12:22:18.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:18 smithi183 ceph-mon[33277]: 3.1f scrub starts 2024-09-21T12:22:18.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:18 smithi183 ceph-mon[33277]: 3.1f scrub ok 2024-09-21T12:22:18.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:18 smithi183 ceph-mon[33277]: 1.0 scrub starts 2024-09-21T12:22:18.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:18 smithi183 ceph-mon[33277]: 1.0 scrub ok 2024-09-21T12:22:18.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:18 smithi183 ceph-mon[33277]: osdmap e109: 8 total, 8 up, 7 in 2024-09-21T12:22:18.518 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:18.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:18.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:18.538 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T12:22:18.621 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:18.629 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 48 16 66851 66835 87.0048 81.375 0.0206668 0.0114873 2024-09-21T12:22:18.643 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:18.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:18.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:18.745 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:18.806 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:18.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:18.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:18.818 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:22:18.818 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:22:18.849 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:22:18.859 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:22:18.860 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:22:18.900 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:22:18.900 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:22:18.908 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:18.930 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:18.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:18.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:18.933 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:22:18.934 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:22:18.966 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:22:18.966 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:22:18.999 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:22:18.999 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:22:19.030 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:22:19.030 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:22:19.032 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:19.063 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:22:19.063 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:22:19.079 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:19.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:19.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:19.181 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:19.204 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:19.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:19.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:19.306 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:19.349 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:19.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:19.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:19.441 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:19 smithi114 ceph-mon[30324]: pgmap v83: 158 pgs: 10 peering, 148 active+clean; 3.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 97 MiB/s wr, 1.56k op/s 2024-09-21T12:22:19.441 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:19 smithi114 ceph-mon[30324]: 1.5 scrub starts 2024-09-21T12:22:19.441 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:19 smithi114 ceph-mon[30324]: 1.5 scrub ok 2024-09-21T12:22:19.441 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:19 smithi114 ceph-mon[30324]: 1.a scrub starts 2024-09-21T12:22:19.441 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:19 smithi114 ceph-mon[30324]: 1.a scrub ok 2024-09-21T12:22:19.441 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:19 smithi114 ceph-mon[30324]: Health check failed: Reduced data availability: 8 pgs peering (PG_AVAILABILITY) 2024-09-21T12:22:19.442 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:19 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3431622113' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:22:19.442 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:19 smithi114 ceph-mon[40867]: pgmap v83: 158 pgs: 10 peering, 148 active+clean; 3.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 97 MiB/s wr, 1.56k op/s 2024-09-21T12:22:19.442 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:19 smithi114 ceph-mon[40867]: 1.5 scrub starts 2024-09-21T12:22:19.442 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:19 smithi114 ceph-mon[40867]: 1.5 scrub ok 2024-09-21T12:22:19.442 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:19 smithi114 ceph-mon[40867]: 1.a scrub starts 2024-09-21T12:22:19.442 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:19 smithi114 ceph-mon[40867]: 1.a scrub ok 2024-09-21T12:22:19.442 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:19 smithi114 ceph-mon[40867]: Health check failed: Reduced data availability: 8 pgs peering (PG_AVAILABILITY) 2024-09-21T12:22:19.442 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:19 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3431622113' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:22:19.450 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:19.474 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:19.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:19.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:19 smithi183 ceph-mon[33277]: pgmap v83: 158 pgs: 10 peering, 148 active+clean; 3.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 97 MiB/s wr, 1.56k op/s 2024-09-21T12:22:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:19 smithi183 ceph-mon[33277]: 1.5 scrub starts 2024-09-21T12:22:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:19 smithi183 ceph-mon[33277]: 1.5 scrub ok 2024-09-21T12:22:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:19 smithi183 ceph-mon[33277]: 1.a scrub starts 2024-09-21T12:22:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:19 smithi183 ceph-mon[33277]: 1.a scrub ok 2024-09-21T12:22:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:19 smithi183 ceph-mon[33277]: Health check failed: Reduced data availability: 8 pgs peering (PG_AVAILABILITY) 2024-09-21T12:22:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:19 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3431622113' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:22:19.576 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:19.624 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:19.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:19.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:19.632 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 49 16 67975 67959 86.6584 70.25 0.00908985 0.0115342 2024-09-21T12:22:19.727 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:19.751 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:19.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:19.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:19.853 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:19.877 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:19.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:19.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:19.979 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:20.241 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-21T12:22:20.325 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:20.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:20.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:20.426 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:20.452 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:20.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:20.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:20 smithi183 ceph-mon[33277]: 6.14 scrub starts 2024-09-21T12:22:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:20 smithi183 ceph-mon[33277]: 6.14 scrub ok 2024-09-21T12:22:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:20 smithi183 ceph-mon[33277]: 1.3 scrub starts 2024-09-21T12:22:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:20 smithi183 ceph-mon[33277]: 1.3 scrub ok 2024-09-21T12:22:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:20 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/534534881' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:22:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:20 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:22:20.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:20 smithi114 ceph-mon[30324]: 6.14 scrub starts 2024-09-21T12:22:20.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:20 smithi114 ceph-mon[30324]: 6.14 scrub ok 2024-09-21T12:22:20.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:20 smithi114 ceph-mon[30324]: 1.3 scrub starts 2024-09-21T12:22:20.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:20 smithi114 ceph-mon[30324]: 1.3 scrub ok 2024-09-21T12:22:20.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:20 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/534534881' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:22:20.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:20 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:22:20.504 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:20 smithi114 ceph-mon[40867]: 6.14 scrub starts 2024-09-21T12:22:20.504 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:20 smithi114 ceph-mon[40867]: 6.14 scrub ok 2024-09-21T12:22:20.504 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:20 smithi114 ceph-mon[40867]: 1.3 scrub starts 2024-09-21T12:22:20.504 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:20 smithi114 ceph-mon[40867]: 1.3 scrub ok 2024-09-21T12:22:20.504 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:20 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/534534881' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:22:20.504 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:20 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:22:20.554 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:20.592 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:20.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:20.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:20.632 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 50 16 69239 69223 86.5051 79 0.0139828 0.0115548 2024-09-21T12:22:20.694 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:20.720 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:20.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:20.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:20.822 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:20.847 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:20.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:20.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:20.949 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:20.977 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:20.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:20.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:21.053 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T12:22:21.053 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:22:21.054 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-09-21T12:22:21.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd in 4 2024-09-21T12:22:21.079 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:21.102 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:21.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:21.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:21.204 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:21.246 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:21.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:21.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:21.347 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:21.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:21.395 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:21.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:21.422 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:21 smithi114 ceph-mon[40867]: pgmap v84: 158 pgs: 10 peering, 148 active+clean; 3.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-21T12:22:21.423 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:21 smithi114 ceph-mon[40867]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:22:21.423 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:21 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:22:21.423 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:21 smithi114 ceph-mon[40867]: osdmap e110: 8 total, 8 up, 7 in 2024-09-21T12:22:21.423 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:21 smithi114 ceph-mon[30324]: pgmap v84: 158 pgs: 10 peering, 148 active+clean; 3.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-21T12:22:21.423 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:21 smithi114 ceph-mon[30324]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:22:21.423 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:21 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:22:21.423 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:21 smithi114 ceph-mon[30324]: osdmap e110: 8 total, 8 up, 7 in 2024-09-21T12:22:21.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:21 smithi183 ceph-mon[33277]: pgmap v84: 158 pgs: 10 peering, 148 active+clean; 3.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-21T12:22:21.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:21 smithi183 ceph-mon[33277]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:22:21.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:21 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:22:21.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:21 smithi183 ceph-mon[33277]: osdmap e110: 8 total, 8 up, 7 in 2024-09-21T12:22:21.497 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:21.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:21.567 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:21.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:21.632 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 51 15 70390 70375 86.2205 72 0.018005 0.0115927 2024-09-21T12:22:21.668 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:21.719 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:21.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:21.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:21.774 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:22:21 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a[75423]: ts=2024-09-21T12:22:21.548Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-21T12:22:21.775 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:22:21 smithi114 podman[84208]: 2024-09-21 12:22:21.591572703 +0000 UTC m=+2.712811728 container died 5dcc38d0a6d785f370bec3180c131bb70fdc8cdeb567de62e720e4f8bb1c7006 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T12:22:21.821 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:21.847 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:21.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:21.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:21.949 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:21.974 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:21.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:21.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:22.076 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:22.102 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:22.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:22.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:22.203 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:22.228 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:22.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:22.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:22.242 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:22:22.275 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:22:22 smithi114 podman[84208]: 2024-09-21 12:22:22.099751765 +0000 UTC m=+3.220990790 container cleanup 5dcc38d0a6d785f370bec3180c131bb70fdc8cdeb567de62e720e4f8bb1c7006 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T12:22:22.275 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:22:22 smithi114 bash[84208]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a 2024-09-21T12:22:22.330 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:22.352 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:22.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:22.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:22.454 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:22.497 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:22.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:22.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:22.523 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:22:22.534 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:22:22.552 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:22:22 smithi114 podman[84519]: 2024-09-21 12:22:22.369731874 +0000 UTC m=+0.770608252 container remove 5dcc38d0a6d785f370bec3180c131bb70fdc8cdeb567de62e720e4f8bb1c7006 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T12:22:22.552 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:22:22 smithi114 podman[84519]: 2024-09-21 12:22:22.461640176 +0000 UTC m=+0.862516572 volume remove f62c7ad5218cc74221120ba59697d2b7c20925ab69f7c0ac76a174250b6f9996 2024-09-21T12:22:22.554 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:22:22 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:22:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:22:22.599 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:22.623 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:22.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:22.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:22.634 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 52 16 71565 71549 85.9702 73.375 0.00778358 0.0116268 2024-09-21T12:22:22.725 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:22.749 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:22.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:22.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:22.850 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:22.895 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:22.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:22.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:22.997 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:23.022 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:23.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:23.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:23.124 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:23.149 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:23.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:23.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:23.251 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:23.274 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:23.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:23.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:23.356 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:23 smithi114 ceph-mon[40867]: pgmap v86: 158 pgs: 2 peering, 156 active+clean; 4.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 93 MiB/s wr, 1.48k op/s; 17 KiB/s, 0 objects/s recovering 2024-09-21T12:22:23.356 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:23 smithi114 ceph-mon[30324]: pgmap v86: 158 pgs: 2 peering, 156 active+clean; 4.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 93 MiB/s wr, 1.48k op/s; 17 KiB/s, 0 objects/s recovering 2024-09-21T12:22:23.375 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:23.424 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:23.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:23.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:23.476 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T12:22:23.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:23 smithi183 ceph-mon[33277]: pgmap v86: 158 pgs: 2 peering, 156 active+clean; 4.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 93 MiB/s wr, 1.48k op/s; 17 KiB/s, 0 objects/s recovering 2024-09-21T12:22:23.527 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:23.547 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:23.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:23.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:23.634 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 53 16 72677 72661 85.6594 69.5 0.0203968 0.0116686 2024-09-21T12:22:23.649 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:23.664 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:22:23.664 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:22:23.672 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:23.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:23.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:23.774 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:23.799 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:23.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:23.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:23.901 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:23.928 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:23.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:23.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:23.977 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:23 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-a[30301]: 2024-09-21T12:22:23.753+0000 7f49fe685640 -1 mon.a@0(leader).osd e110 definitely_dead 0 2024-09-21T12:22:24.030 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:24.055 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:24.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:24.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:24.112 INFO:teuthology.orchestra.run.smithi114.stderr:marked in osd.4. 2024-09-21T12:22:24.157 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:24.189 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:24.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:24.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:24.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:24 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/677358063' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T12:22:24.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:24 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T12:22:24.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:24 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/677358063' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T12:22:24.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:24 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T12:22:24.275 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:22:23 smithi114 systemd[1]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-21T12:22:24.275 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:22:23 smithi114 systemd[1]: Stopped Ceph alertmanager.a for e2ad975c-7812-11ef-baf6-efdc52797490. 2024-09-21T12:22:24.275 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:22:23 smithi114 systemd[1]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490@alertmanager.a.service: Consumed 1.274s CPU time. 2024-09-21T12:22:24.275 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:22:23 smithi114 systemd[1]: Starting Ceph alertmanager.a for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:22:24.291 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:24.312 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:24.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:24.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:24.414 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:24.436 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:24.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:24.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:24.443 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T12:22:24.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:24 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/677358063' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T12:22:24.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:24 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T12:22:24.538 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:24.560 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:24.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:24.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:24.634 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 54 16 73309 73293 84.8047 39.5 0.00237689 0.0116858 2024-09-21T12:22:24.661 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:24.670 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:22:24.675 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:22:24.683 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:24.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:24.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:24.720 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:22:24.786 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:24.807 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:24.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:24.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:24.909 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:24.930 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:24.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:24.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:25.032 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:25.053 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:25.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:25.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:25.155 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:25.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:25.183 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:25.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:25.284 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:25.305 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:25.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:25.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:25.312 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:25 smithi114 ceph-mon[40867]: pgmap v87: 158 pgs: 158 active+clean; 4.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 102 MiB/s wr, 1.63k op/s; 49 KiB/s, 0 objects/s recovering 2024-09-21T12:22:25.312 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:25 smithi114 ceph-mon[40867]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-21T12:22:25.312 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:25 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-21T12:22:25.312 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:25 smithi114 ceph-mon[40867]: osdmap e111: 8 total, 8 up, 8 in 2024-09-21T12:22:25.312 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:25 smithi114 ceph-mon[30324]: pgmap v87: 158 pgs: 158 active+clean; 4.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 102 MiB/s wr, 1.63k op/s; 49 KiB/s, 0 objects/s recovering 2024-09-21T12:22:25.312 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:25 smithi114 ceph-mon[30324]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-21T12:22:25.312 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:25 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-21T12:22:25.312 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:25 smithi114 ceph-mon[30324]: osdmap e111: 8 total, 8 up, 8 in 2024-09-21T12:22:25.407 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:25.429 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:25.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:25.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:25.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:25 smithi183 ceph-mon[33277]: pgmap v87: 158 pgs: 158 active+clean; 4.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 102 MiB/s wr, 1.63k op/s; 49 KiB/s, 0 objects/s recovering 2024-09-21T12:22:25.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:25 smithi183 ceph-mon[33277]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-21T12:22:25.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:25 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-21T12:22:25.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:25 smithi183 ceph-mon[33277]: osdmap e111: 8 total, 8 up, 8 in 2024-09-21T12:22:25.531 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:25.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:25.553 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:25.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:25.635 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 55 16 73309 73293 83.263 0 - 0.0116858 2024-09-21T12:22:25.654 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:25.676 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:25.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:25.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:25.778 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:25.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:25.803 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:25.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:25.851 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-09-21T12:22:25.904 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:25.924 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:25.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:25.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:26.026 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:26.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:26.053 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:26.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:26.115 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:22:25 smithi114 podman[85087]: 2024-09-21 12:22:25.906727856 +0000 UTC m=+0.726013775 volume create c6e2b695ba50f3c7f023453a430b0d38c59732691ca126ab94c3cf9ac39bd566 2024-09-21T12:22:26.116 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:22:25 smithi114 podman[85087]: 2024-09-21 12:22:25.829714465 +0000 UTC m=+0.649000398 image pull 11f11916f8cdfb17c1da74b20deb2075d778357b4d60474d82453303f8d461f5 quay.io/prometheus/alertmanager:v0.27.0 2024-09-21T12:22:26.116 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:22:25 smithi114 podman[85087]: 2024-09-21 12:22:25.998970263 +0000 UTC m=+0.818256181 container create 9a9b92f1127acee778994c000cc07647c2820e8abac5f18d3ca6d6209e105fd1 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T12:22:26.119 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-21T12:22:26.154 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:26.180 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:26.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:26.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:26.282 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:26.319 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:26.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:26.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:26.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:26.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:26.463 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:26.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:26.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:26 smithi183 ceph-mon[33277]: osdmap e112: 8 total, 8 up, 8 in 2024-09-21T12:22:26.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:26 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2253842040' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:22:26.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:26 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:22:26.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:26 smithi114 ceph-mon[40867]: osdmap e112: 8 total, 8 up, 8 in 2024-09-21T12:22:26.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:26 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2253842040' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:22:26.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:26 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:22:26.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:26 smithi114 ceph-mon[30324]: osdmap e112: 8 total, 8 up, 8 in 2024-09-21T12:22:26.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:26 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2253842040' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:22:26.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:26 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:22:26.564 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:26.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:26.608 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:26.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:26.635 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 56 16 73765 73749 82.2853 14.25 0.00507613 0.0121478 2024-09-21T12:22:26.708 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:26.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:26.739 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:26.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:26.840 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:26.885 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:26.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:26.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:26.987 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:26.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:22:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:22:27.015 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:27.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:27.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:27.080 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:22:27 smithi114 podman[85087]: 2024-09-21 12:22:27.034071911 +0000 UTC m=+1.853357847 container init 9a9b92f1127acee778994c000cc07647c2820e8abac5f18d3ca6d6209e105fd1 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T12:22:27.080 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:22:27 smithi114 podman[85087]: 2024-09-21 12:22:27.045855823 +0000 UTC m=+1.865141750 container start 9a9b92f1127acee778994c000cc07647c2820e8abac5f18d3ca6d6209e105fd1 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T12:22:27.117 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:27.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:27.146 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:27.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:27.155 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:22:27.155 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:22:27.248 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:27.276 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:27.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:27.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:27.378 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:27.416 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:27.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:27.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:27.455 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:27 smithi114 ceph-mon[40867]: pgmap v90: 158 pgs: 158 active+clean; 4.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 63 MiB/s wr, 1.01k op/s; 61 KiB/s, 0 objects/s recovering 2024-09-21T12:22:27.455 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:27 smithi114 ceph-mon[40867]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:22:27.455 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:27 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:22:27.455 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:27 smithi114 ceph-mon[40867]: 7.9 starting backfill to osd.4 from (0'0,0'0] MAX to 110'4628 2024-09-21T12:22:27.455 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:27 smithi114 ceph-mon[40867]: 7.4 starting backfill to osd.4 from (0'0,0'0] MAX to 110'4491 2024-09-21T12:22:27.455 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:27 smithi114 ceph-mon[40867]: osdmap e113: 8 total, 8 up, 8 in 2024-09-21T12:22:27.455 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:27 smithi114 ceph-mon[40867]: 7.3 starting backfill to osd.4 from (0'0,0'0] MAX to 110'4500 2024-09-21T12:22:27.455 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:27 smithi114 ceph-mon[40867]: 7.1 starting backfill to osd.4 from (0'0,0'0] MAX to 110'4642 2024-09-21T12:22:27.455 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:27 smithi114 ceph-mon[40867]: osdmap e114: 8 total, 8 up, 8 in 2024-09-21T12:22:27.456 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:22:27 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a[85236]: ts=2024-09-21T12:22:27.078Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2024-09-21T12:22:27.456 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:22:27 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a[85236]: ts=2024-09-21T12:22:27.079Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)" 2024-09-21T12:22:27.456 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:22:27 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a[85236]: ts=2024-09-21T12:22:27.081Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.114 port=9094 2024-09-21T12:22:27.456 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:22:27 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a[85236]: ts=2024-09-21T12:22:27.085Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-21T12:22:27.456 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:22:27 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a[85236]: ts=2024-09-21T12:22:27.124Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T12:22:27.456 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:22:27 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a[85236]: ts=2024-09-21T12:22:27.125Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T12:22:27.456 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:22:27 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a[85236]: ts=2024-09-21T12:22:27.127Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093 2024-09-21T12:22:27.456 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:22:27 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a[85236]: ts=2024-09-21T12:22:27.131Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-21T12:22:27.456 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:27 smithi114 ceph-mon[30324]: pgmap v90: 158 pgs: 158 active+clean; 4.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 63 MiB/s wr, 1.01k op/s; 61 KiB/s, 0 objects/s recovering 2024-09-21T12:22:27.456 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:27 smithi114 ceph-mon[30324]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:22:27.457 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:27 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:22:27.457 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:27 smithi114 ceph-mon[30324]: 7.9 starting backfill to osd.4 from (0'0,0'0] MAX to 110'4628 2024-09-21T12:22:27.457 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:27 smithi114 ceph-mon[30324]: 7.4 starting backfill to osd.4 from (0'0,0'0] MAX to 110'4491 2024-09-21T12:22:27.457 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:27 smithi114 ceph-mon[30324]: osdmap e113: 8 total, 8 up, 8 in 2024-09-21T12:22:27.457 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:27 smithi114 ceph-mon[30324]: 7.3 starting backfill to osd.4 from (0'0,0'0] MAX to 110'4500 2024-09-21T12:22:27.457 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:27 smithi114 ceph-mon[30324]: 7.1 starting backfill to osd.4 from (0'0,0'0] MAX to 110'4642 2024-09-21T12:22:27.457 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:27 smithi114 ceph-mon[30324]: osdmap e114: 8 total, 8 up, 8 in 2024-09-21T12:22:27.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:27 smithi183 ceph-mon[33277]: pgmap v90: 158 pgs: 158 active+clean; 4.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 63 MiB/s wr, 1.01k op/s; 61 KiB/s, 0 objects/s recovering 2024-09-21T12:22:27.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:27 smithi183 ceph-mon[33277]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:22:27.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:27 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:22:27.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:27 smithi183 ceph-mon[33277]: 7.9 starting backfill to osd.4 from (0'0,0'0] MAX to 110'4628 2024-09-21T12:22:27.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:27 smithi183 ceph-mon[33277]: 7.4 starting backfill to osd.4 from (0'0,0'0] MAX to 110'4491 2024-09-21T12:22:27.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:27 smithi183 ceph-mon[33277]: osdmap e113: 8 total, 8 up, 8 in 2024-09-21T12:22:27.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:27 smithi183 ceph-mon[33277]: 7.3 starting backfill to osd.4 from (0'0,0'0] MAX to 110'4500 2024-09-21T12:22:27.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:27 smithi183 ceph-mon[33277]: 7.1 starting backfill to osd.4 from (0'0,0'0] MAX to 110'4642 2024-09-21T12:22:27.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:27 smithi183 ceph-mon[33277]: osdmap e114: 8 total, 8 up, 8 in 2024-09-21T12:22:27.518 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:27.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:27.562 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:27.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:27.635 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 57 16 74815 74799 81.993 65.625 0.0194576 0.0121902 2024-09-21T12:22:27.664 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:27.707 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:27.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:27.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:27.809 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:27.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:27.839 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:27.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:27.940 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:27.967 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:27.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:27.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:28.069 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:28.107 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:28.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:28.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:28.209 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:28.215 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T12:22:28.248 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:28.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:28.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:28.349 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:28.388 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:28 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:22:28.388 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:28 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:22:28.388 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:22:28 smithi114 bash[85087]: 9a9b92f1127acee778994c000cc07647c2820e8abac5f18d3ca6d6209e105fd1 2024-09-21T12:22:28.389 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:22:28 smithi114 systemd[1]: Started Ceph alertmanager.a for e2ad975c-7812-11ef-baf6-efdc52797490. 2024-09-21T12:22:28.413 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:28.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:28.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:28.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:28 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:22:28.515 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:28.540 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:28.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:28.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:28.612 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:22:28.612 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:22:28.636 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 58 16 75779 75763 81.6169 60.25 0.0191078 0.0122453 2024-09-21T12:22:28.642 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:28.655 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:22:28.655 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:22:28.672 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:28.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:28.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:28.697 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:22:28.697 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:22:28.736 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:22:28.736 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:22:28.756 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:22:28.773 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:22:28.774 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:22:28.774 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:28.810 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:22:28.810 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:22:28.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:28.825 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:28.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:28.842 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:22:28.842 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:22:28.866 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T12:22:28.877 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:22:28.877 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:22:28.926 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:28.973 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:28.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:28.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:29.074 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:29.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:29.147 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:29.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:29.196 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:22:29 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a[85236]: ts=2024-09-21T12:22:29.086Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000764939s 2024-09-21T12:22:29.196 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:29 smithi114 ceph-mon[30324]: pgmap v93: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 839 op/s; 22911/221886 objects misplaced (10.326%); 2.0 MiB/s, 41 objects/s recovering 2024-09-21T12:22:29.197 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:29 smithi114 ceph-mon[30324]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:22:29.197 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:29 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:29.197 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:29 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:29.197 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:29 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1364361777' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:22:29.197 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:29 smithi114 ceph-mon[40867]: pgmap v93: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 839 op/s; 22911/221886 objects misplaced (10.326%); 2.0 MiB/s, 41 objects/s recovering 2024-09-21T12:22:29.197 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:29 smithi114 ceph-mon[40867]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:22:29.197 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:29 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:29.197 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:29 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:29.197 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:29 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1364361777' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:22:29.249 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:29.295 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:29.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:29.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:29.397 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:29.401 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:29 smithi183 ceph-mon[33277]: pgmap v93: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 839 op/s; 22911/221886 objects misplaced (10.326%); 2.0 MiB/s, 41 objects/s recovering 2024-09-21T12:22:29.401 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:29 smithi183 ceph-mon[33277]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:22:29.401 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:29 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:29.401 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:29 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:29.401 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:29 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1364361777' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:22:29.436 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:29.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:29.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:29.538 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:29.579 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:29.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:29.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:29.636 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 59 16 76679 76663 81.187 56.25 0.0337759 0.0123115 2024-09-21T12:22:29.681 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:29.723 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:29.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:29.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:29.825 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:29.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:29.863 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:29.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:29.964 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:29.995 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:29.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:29.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:30.096 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:30.133 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:30.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:30.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:30.235 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:30.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:30.267 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:30.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:30.368 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:30.370 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:30 smithi183 ceph-mon[33277]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T12:22:30.370 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:30 smithi183 ceph-mon[33277]: Reconfiguring daemon prometheus.a on smithi183 2024-09-21T12:22:30.411 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:30.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:30.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:30.513 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:30.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:30 smithi114 ceph-mon[40867]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T12:22:30.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:30 smithi114 ceph-mon[40867]: Reconfiguring daemon prometheus.a on smithi183 2024-09-21T12:22:30.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:30 smithi114 ceph-mon[30324]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T12:22:30.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:30 smithi114 ceph-mon[30324]: Reconfiguring daemon prometheus.a on smithi183 2024-09-21T12:22:30.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:30.553 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:30.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:30.636 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-21T12:22:30.635614+0000 min lat: 0.00135771 max lat: 2.12184 avg lat: 0.0123572 2024-09-21T12:22:30.636 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T12:22:30.636 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 60 16 77694 77678 80.8911 63.4375 0.0109202 0.0123572 2024-09-21T12:22:30.654 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:30.694 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:30.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:30.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:30.795 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:30.823 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:30.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:30.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:30.852 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T12:22:30.852 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:22:30.852 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2024-09-21T12:22:30.852 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 8 2024-09-21T12:22:30.852 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:22:30.925 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:30.979 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:30.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:30.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:31.082 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:31.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:31.107 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:31.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:31.201 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:22:31.202 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:22:31.202 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:22:31.202 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:22:31.202 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:22:31.208 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:31.235 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:31 smithi183 ceph-mon[33277]: pgmap v94: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 935 B/s rd, 38 MiB/s wr, 613 op/s; 22911/221886 objects misplaced (10.326%); 1.4 MiB/s, 30 objects/s recovering 2024-09-21T12:22:31.235 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:31 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:31.235 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:31 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:22:31.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:31.259 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:31.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:31.360 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:31.390 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:31.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:31.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:31.492 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:31.516 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:31 smithi114 ceph-mon[40867]: pgmap v94: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 935 B/s rd, 38 MiB/s wr, 613 op/s; 22911/221886 objects misplaced (10.326%); 1.4 MiB/s, 30 objects/s recovering 2024-09-21T12:22:31.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:31 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:31.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:31 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:22:31.517 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:31 smithi114 ceph-mon[30324]: pgmap v94: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 935 B/s rd, 38 MiB/s wr, 613 op/s; 22911/221886 objects misplaced (10.326%); 1.4 MiB/s, 30 objects/s recovering 2024-09-21T12:22:31.517 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:31 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:31.517 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:31 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:22:31.539 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:31.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:31.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:31.636 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 61 16 78692 78676 80.5875 62.375 0.0141038 0.0124031 2024-09-21T12:22:31.641 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:31.671 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:31.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:31.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:31.773 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:31.807 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:31.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:31.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:31.875 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T12:22:31.910 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:31.935 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:31.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:31.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:31.947 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:22:31.975 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:22:32.038 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:32.082 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:32.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:32.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:32.184 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:32.211 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:32.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:32.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:32.313 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:32.347 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:32.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:32.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:32.450 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:32.486 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:32.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:32.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:32.488 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:32 smithi183 systemd[1]: Stopping Ceph prometheus.a for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:22:32.588 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:32.620 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:32.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:32.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:32.636 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 62 16 79622 79606 80.2249 58.125 0.0153284 0.012459 2024-09-21T12:22:32.721 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:32.740 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:22:32 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:22:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:22:32.765 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:32.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:32.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:32.867 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:32.892 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:32.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:32.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:32.994 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:33.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:33.019 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:33.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:33.040 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:32 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:22:32.739Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-21T12:22:33.040 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:32 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:22:32.740Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-21T12:22:33.041 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:32 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:22:32.740Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-21T12:22:33.041 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:32 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:22:32.740Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-21T12:22:33.041 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:32 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:22:32.740Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-21T12:22:33.041 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:32 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:22:32.740Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-21T12:22:33.041 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:32 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:22:32.740Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-21T12:22:33.041 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:32 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:22:32.740Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-21T12:22:33.041 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:32 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:22:32.740Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-21T12:22:33.041 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:32 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:22:32.741Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-21T12:22:33.041 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:32 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:22:32.741Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-21T12:22:33.041 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:32 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[61410]: ts=2024-09-21T12:22:32.741Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-21T12:22:33.041 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:32 smithi183 podman[68429]: 2024-09-21 12:22:32.750264833 +0000 UTC m=+0.139750501 container died bc19bc112d29a7438590de74c31d220bddf8e7c03f938ec6aaeb66f041dfaed9 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:22:33.058 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:22:33.121 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:33.140 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:22:33.175 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:33.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:33.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:33.278 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:33.306 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:33.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:33.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:33.408 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:33.414 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:33 smithi114 ceph-mon[30324]: pgmap v95: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.5 KiB/s rd, 47 MiB/s wr, 752 op/s; 22911/226407 objects misplaced (10.119%); 1.2 MiB/s, 25 objects/s recovering 2024-09-21T12:22:33.415 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:33 smithi114 ceph-mon[40867]: pgmap v95: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.5 KiB/s rd, 47 MiB/s wr, 752 op/s; 22911/226407 objects misplaced (10.119%); 1.2 MiB/s, 25 objects/s recovering 2024-09-21T12:22:33.421 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:22:33.423 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:22:33.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:33 smithi183 ceph-mon[33277]: pgmap v95: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.5 KiB/s rd, 47 MiB/s wr, 752 op/s; 22911/226407 objects misplaced (10.119%); 1.2 MiB/s, 25 objects/s recovering 2024-09-21T12:22:33.425 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:33 smithi183 podman[68429]: 2024-09-21 12:22:33.153672656 +0000 UTC m=+0.543158312 container cleanup bc19bc112d29a7438590de74c31d220bddf8e7c03f938ec6aaeb66f041dfaed9 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:22:33.425 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:33 smithi183 bash[68429]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a 2024-09-21T12:22:33.434 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:33.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:33.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:33.535 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:33.566 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:33.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:33.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:33.636 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 63 16 80442 80426 79.765 51.25 0.012441 0.0125318 2024-09-21T12:22:33.668 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:33.700 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:33.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:33.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:33.738 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:33 smithi183 podman[68448]: 2024-09-21 12:22:33.422244922 +0000 UTC m=+0.671100117 container remove bc19bc112d29a7438590de74c31d220bddf8e7c03f938ec6aaeb66f041dfaed9 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:22:33.738 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:33 smithi183 systemd[1]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-21T12:22:33.739 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:33 smithi183 systemd[1]: Stopped Ceph prometheus.a for e2ad975c-7812-11ef-baf6-efdc52797490. 2024-09-21T12:22:33.739 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:33 smithi183 systemd[1]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490@prometheus.a.service: Consumed 1.284s CPU time. 2024-09-21T12:22:33.739 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:33 smithi183 systemd[1]: Starting Ceph prometheus.a for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:22:33.802 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:33.829 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:33.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:33.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:33.931 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:33.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:33.975 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:33.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:34.076 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:34.112 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:34.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:34.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:34.175 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:33 smithi183 podman[68643]: 2024-09-21 12:22:33.780772683 +0000 UTC m=+0.021658332 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-21T12:22:34.175 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:33 smithi183 podman[68643]: 2024-09-21 12:22:33.898890523 +0000 UTC m=+0.139776153 container create cb6f35f47c5b55547c82f9315d527a12248fa8f6c0293d11c3a09cfce41966e4 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:22:34.176 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-21T12:22:34.214 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:34.247 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:34.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:34.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:34.293 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:22:34.293 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 20, 2024-09-21T12:22:34.293 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 600, 2024-09-21T12:22:34.293 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-21T12:22:34.294 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.294 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:24382 7.3 7:c60f87f6:::benchmark_data_smithi114_78432_object24381:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T12:22:34.294 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:40.027851+0000", 2024-09-21T12:22:34.294 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 53.731138303000002, 2024-09-21T12:22:34.294 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0543905850000002, 2024-09-21T12:22:34.294 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:22:34.294 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:22:34.294 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:34.294 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:22:34.294 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:22:34.294 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:34.294 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 24382 2024-09-21T12:22:34.295 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.295 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:22:34.295 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.295 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:22:34.295 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.027851+0000", 2024-09-21T12:22:34.295 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.295 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.295 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.295 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:22:34.295 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.027851+0000", 2024-09-21T12:22:34.295 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.295 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.295 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.295 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:22:34.296 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.027853+0000", 2024-09-21T12:22:34.296 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6729999999999999e-06 2024-09-21T12:22:34.296 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.296 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.296 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:22:34.296 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.027885+0000", 2024-09-21T12:22:34.296 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2155e-05 2024-09-21T12:22:34.296 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.296 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.296 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:22:34.296 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.027897+0000", 2024-09-21T12:22:34.296 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2371000000000001e-05 2024-09-21T12:22:34.296 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.296 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.297 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:22:34.297 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.027907+0000", 2024-09-21T12:22:34.297 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.9580000000000002e-06 2024-09-21T12:22:34.297 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.297 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.297 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.297 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.031303+0000", 2024-09-21T12:22:34.297 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003395749 2024-09-21T12:22:34.297 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.297 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.297 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:22:34.297 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.031311+0000", 2024-09-21T12:22:34.297 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.3499999999999997e-06 2024-09-21T12:22:34.297 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.298 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.298 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.298 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.026611+0000", 2024-09-21T12:22:34.298 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.995300132 2024-09-21T12:22:34.298 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.298 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.298 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:22:34.298 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.026622+0000", 2024-09-21T12:22:34.298 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0735e-05 2024-09-21T12:22:34.298 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.298 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.298 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.298 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.070786+0000", 2024-09-21T12:22:34.299 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0441637530000001 2024-09-21T12:22:34.299 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.299 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.299 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:22:34.299 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.070935+0000", 2024-09-21T12:22:34.299 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014934 2024-09-21T12:22:34.299 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.299 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.299 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:22:34.299 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.071175+0000", 2024-09-21T12:22:34.299 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000239409 2024-09-21T12:22:34.299 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.299 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.299 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.300 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.074999+0000", 2024-09-21T12:22:34.300 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0038247229999999999 2024-09-21T12:22:34.300 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.300 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.300 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.300 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.075051+0000", 2024-09-21T12:22:34.300 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1588000000000001e-05 2024-09-21T12:22:34.300 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.300 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.300 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:22:34.300 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.081204+0000", 2024-09-21T12:22:34.300 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0061526209999999996 2024-09-21T12:22:34.300 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.300 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.301 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:22:34.301 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.081254+0000", 2024-09-21T12:22:34.301 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.024e-05 2024-09-21T12:22:34.301 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.301 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.301 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:22:34.301 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.082242+0000", 2024-09-21T12:22:34.301 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00098778800000000008 2024-09-21T12:22:34.301 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.301 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:34.301 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.301 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.301 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.301 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:24399 7.3 7:cc1424c5:::benchmark_data_smithi114_78432_object24398:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:22:34.302 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:40.028127+0000", 2024-09-21T12:22:34.302 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 53.730862731999999, 2024-09-21T12:22:34.302 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0647246259999998, 2024-09-21T12:22:34.302 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:22:34.302 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:22:34.302 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:34.302 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:22:34.302 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:22:34.302 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:34.302 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 24399 2024-09-21T12:22:34.302 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.302 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:22:34.302 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.303 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:22:34.303 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.028127+0000", 2024-09-21T12:22:34.303 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.303 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.303 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.303 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:22:34.303 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.028127+0000", 2024-09-21T12:22:34.303 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.303 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.303 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.303 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:22:34.303 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.028127+0000", 2024-09-21T12:22:34.303 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5000000000000003e-07 2024-09-21T12:22:34.303 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.304 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.304 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:22:34.304 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.028150+0000", 2024-09-21T12:22:34.304 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2653000000000001e-05 2024-09-21T12:22:34.304 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.304 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.304 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:22:34.304 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.028161+0000", 2024-09-21T12:22:34.304 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0939e-05 2024-09-21T12:22:34.304 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.304 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.304 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:22:34.304 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.028166+0000", 2024-09-21T12:22:34.304 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1970000000000004e-06 2024-09-21T12:22:34.305 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.305 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.305 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.305 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.031332+0000", 2024-09-21T12:22:34.305 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003165883 2024-09-21T12:22:34.305 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.305 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.305 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:22:34.305 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.031338+0000", 2024-09-21T12:22:34.305 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8850000000000004e-06 2024-09-21T12:22:34.305 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.305 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.305 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.305 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.032668+0000", 2024-09-21T12:22:34.305 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.001330367 2024-09-21T12:22:34.306 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.306 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.306 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:22:34.306 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.032679+0000", 2024-09-21T12:22:34.306 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0912e-05 2024-09-21T12:22:34.306 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.306 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.306 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.306 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.073338+0000", 2024-09-21T12:22:34.306 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.040658632 2024-09-21T12:22:34.306 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.306 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.306 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:22:34.306 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.073479+0000", 2024-09-21T12:22:34.307 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014109800000000001 2024-09-21T12:22:34.307 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.307 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.307 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:22:34.307 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.073642+0000", 2024-09-21T12:22:34.307 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016357500000000001 2024-09-21T12:22:34.307 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.307 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.307 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.307 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.079581+0000", 2024-09-21T12:22:34.307 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0059387479999999998 2024-09-21T12:22:34.307 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.307 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.308 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:22:34.308 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.082251+0000", 2024-09-21T12:22:34.308 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0026700399999999998 2024-09-21T12:22:34.308 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.308 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.308 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.308 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.090796+0000", 2024-09-21T12:22:34.308 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0085451050000000008 2024-09-21T12:22:34.308 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.308 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.308 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:22:34.308 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.090909+0000", 2024-09-21T12:22:34.308 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011303600000000001 2024-09-21T12:22:34.308 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.308 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.309 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:22:34.309 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.092851+0000", 2024-09-21T12:22:34.309 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001942006 2024-09-21T12:22:34.309 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.309 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:34.309 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.309 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.309 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.309 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:24403 7.1 7:8d4347eb:::benchmark_data_smithi114_78432_object24402:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:22:34.309 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:40.038076+0000", 2024-09-21T12:22:34.309 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 53.720913703999997, 2024-09-21T12:22:34.309 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0479541729999999, 2024-09-21T12:22:34.309 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:22:34.310 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:22:34.310 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:34.310 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:22:34.310 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:22:34.310 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:34.310 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 24403 2024-09-21T12:22:34.310 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.310 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:22:34.310 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.310 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:22:34.310 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.038076+0000", 2024-09-21T12:22:34.310 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.310 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.310 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.311 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:22:34.311 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.038076+0000", 2024-09-21T12:22:34.311 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.311 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.311 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.311 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:22:34.311 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.038077+0000", 2024-09-21T12:22:34.311 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.641e-06 2024-09-21T12:22:34.311 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.311 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.311 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:22:34.311 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.038103+0000", 2024-09-21T12:22:34.311 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5689000000000001e-05 2024-09-21T12:22:34.311 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.311 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.312 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:22:34.312 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.038112+0000", 2024-09-21T12:22:34.312 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.3409999999999992e-06 2024-09-21T12:22:34.312 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.312 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.312 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:22:34.312 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.038119+0000", 2024-09-21T12:22:34.312 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1829999999999999e-06 2024-09-21T12:22:34.312 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.312 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.312 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.312 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.038160+0000", 2024-09-21T12:22:34.312 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1597000000000002e-05 2024-09-21T12:22:34.312 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.313 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.313 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:22:34.313 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.038170+0000", 2024-09-21T12:22:34.313 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.4520000000000003e-06 2024-09-21T12:22:34.313 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.313 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.313 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.313 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.021190+0000", 2024-09-21T12:22:34.313 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98302043100000003 2024-09-21T12:22:34.313 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.313 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.313 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:22:34.313 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.021201+0000", 2024-09-21T12:22:34.313 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1e-05 2024-09-21T12:22:34.313 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.314 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.314 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.314 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.076308+0000", 2024-09-21T12:22:34.314 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0551071169999999 2024-09-21T12:22:34.314 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.314 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.314 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:22:34.314 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.077219+0000", 2024-09-21T12:22:34.314 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00091087299999999998 2024-09-21T12:22:34.314 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.314 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.314 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:22:34.314 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.077364+0000", 2024-09-21T12:22:34.314 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014518300000000001 2024-09-21T12:22:34.315 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.315 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.315 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.315 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.080655+0000", 2024-09-21T12:22:34.315 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0032908130000000001 2024-09-21T12:22:34.315 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.315 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.315 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:22:34.315 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.082504+0000", 2024-09-21T12:22:34.315 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001849301 2024-09-21T12:22:34.315 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.315 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.315 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.315 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.085101+0000", 2024-09-21T12:22:34.315 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002597079 2024-09-21T12:22:34.315 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.316 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.316 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:22:34.316 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.085142+0000", 2024-09-21T12:22:34.316 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0460000000000002e-05 2024-09-21T12:22:34.316 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.316 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.316 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:22:34.316 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.086030+0000", 2024-09-21T12:22:34.316 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000888013 2024-09-21T12:22:34.316 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.316 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:34.316 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.316 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.316 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.317 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:24412 7.3 7:c33a628a:::benchmark_data_smithi114_78432_object24411:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:22:34.317 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:40.043570+0000", 2024-09-21T12:22:34.317 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 53.715419820000001, 2024-09-21T12:22:34.317 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0510890659999998, 2024-09-21T12:22:34.317 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:22:34.317 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:22:34.317 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:34.317 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:22:34.317 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:22:34.317 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:34.317 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 24412 2024-09-21T12:22:34.317 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.317 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:22:34.317 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.318 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:22:34.318 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.043570+0000", 2024-09-21T12:22:34.318 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.318 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.318 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.318 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:22:34.318 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.043570+0000", 2024-09-21T12:22:34.318 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.318 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.318 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.318 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:22:34.318 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.043571+0000", 2024-09-21T12:22:34.318 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0690000000000001e-06 2024-09-21T12:22:34.318 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.319 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.319 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:22:34.319 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.043592+0000", 2024-09-21T12:22:34.319 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1237e-05 2024-09-21T12:22:34.319 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.319 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.319 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:22:34.319 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.043602+0000", 2024-09-21T12:22:34.319 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6630000000000005e-06 2024-09-21T12:22:34.319 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.319 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.319 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:22:34.319 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.043608+0000", 2024-09-21T12:22:34.319 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1179999999999997e-06 2024-09-21T12:22:34.319 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.320 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.320 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.320 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.043667+0000", 2024-09-21T12:22:34.320 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9574000000000003e-05 2024-09-21T12:22:34.320 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.320 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.320 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:22:34.320 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.043677+0000", 2024-09-21T12:22:34.320 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.3899999999999999e-06 2024-09-21T12:22:34.320 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.320 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.320 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.320 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.032706+0000", 2024-09-21T12:22:34.321 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98902957700000005 2024-09-21T12:22:34.321 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.321 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.321 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:22:34.321 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.032714+0000", 2024-09-21T12:22:34.321 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9379999999999992e-06 2024-09-21T12:22:34.321 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.321 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.321 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.321 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.075083+0000", 2024-09-21T12:22:34.321 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0423692630000001 2024-09-21T12:22:34.321 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.321 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.322 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:22:34.322 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.075221+0000", 2024-09-21T12:22:34.322 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013774800000000001 2024-09-21T12:22:34.322 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.322 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.322 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:22:34.322 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.075379+0000", 2024-09-21T12:22:34.322 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000157577 2024-09-21T12:22:34.322 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.322 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.322 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.322 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.079711+0000", 2024-09-21T12:22:34.322 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0043323260000000001 2024-09-21T12:22:34.323 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.323 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.323 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:22:34.323 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.082257+0000", 2024-09-21T12:22:34.323 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0025462739999999998 2024-09-21T12:22:34.323 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.323 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.323 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.323 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.092913+0000", 2024-09-21T12:22:34.323 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010656025 2024-09-21T12:22:34.323 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.323 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.323 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:22:34.323 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.092966+0000", 2024-09-21T12:22:34.324 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2296000000000002e-05 2024-09-21T12:22:34.324 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.324 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.324 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:22:34.324 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.094659+0000", 2024-09-21T12:22:34.324 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001692991 2024-09-21T12:22:34.324 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.324 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:34.324 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.324 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.324 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.324 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:24418 7.3 7:c9c1f723:::benchmark_data_smithi114_78432_object24417:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:22:34.324 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:40.048102+0000", 2024-09-21T12:22:34.324 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 53.710887348999997, 2024-09-21T12:22:34.325 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0488444459999999, 2024-09-21T12:22:34.325 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:22:34.325 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:22:34.325 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:34.325 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:22:34.325 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:22:34.325 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:34.325 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 24418 2024-09-21T12:22:34.325 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.325 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:22:34.325 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.325 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:22:34.325 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.048102+0000", 2024-09-21T12:22:34.325 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.326 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.326 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.326 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:22:34.326 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.048102+0000", 2024-09-21T12:22:34.326 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.326 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.326 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.326 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:22:34.326 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.048104+0000", 2024-09-21T12:22:34.326 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4729999999999999e-06 2024-09-21T12:22:34.326 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.326 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.326 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:22:34.326 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.048127+0000", 2024-09-21T12:22:34.327 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3088999999999999e-05 2024-09-21T12:22:34.327 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.327 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.327 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:22:34.327 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.048143+0000", 2024-09-21T12:22:34.327 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6314e-05 2024-09-21T12:22:34.327 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.327 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.327 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:22:34.327 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.048149+0000", 2024-09-21T12:22:34.327 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5579999999999997e-06 2024-09-21T12:22:34.327 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.327 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.327 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.328 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.048224+0000", 2024-09-21T12:22:34.328 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4055999999999997e-05 2024-09-21T12:22:34.328 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.328 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.328 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:22:34.328 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.048235+0000", 2024-09-21T12:22:34.328 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1354e-05 2024-09-21T12:22:34.328 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.328 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.329 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.329 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.032739+0000", 2024-09-21T12:22:34.330 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98450402000000004 2024-09-21T12:22:34.330 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.330 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.330 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:22:34.330 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.032746+0000", 2024-09-21T12:22:34.330 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5510000000000002e-06 2024-09-21T12:22:34.330 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.330 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.330 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.330 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.076754+0000", 2024-09-21T12:22:34.330 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0440078779999999 2024-09-21T12:22:34.330 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.330 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.331 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:22:34.331 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.076899+0000", 2024-09-21T12:22:34.331 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000145122 2024-09-21T12:22:34.331 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.331 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.331 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:22:34.331 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.078384+0000", 2024-09-21T12:22:34.331 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001484527 2024-09-21T12:22:34.331 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.331 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.331 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:22:34.331 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.083050+0000", 2024-09-21T12:22:34.331 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0046663069999999997 2024-09-21T12:22:34.331 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.332 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.332 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.332 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.083099+0000", 2024-09-21T12:22:34.332 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9089999999999999e-05 2024-09-21T12:22:34.332 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.332 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.332 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.332 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.094756+0000", 2024-09-21T12:22:34.332 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.011656992999999999 2024-09-21T12:22:34.332 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.332 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.332 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:22:34.332 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.094807+0000", 2024-09-21T12:22:34.332 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.109e-05 2024-09-21T12:22:34.332 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.333 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.333 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:22:34.333 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.096946+0000", 2024-09-21T12:22:34.333 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0021390239999999998 2024-09-21T12:22:34.333 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.333 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:34.333 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.333 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.333 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.333 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:24423 7.1 7:82b639fa:::benchmark_data_smithi114_78432_object24422:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:22:34.333 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:40.057046+0000", 2024-09-21T12:22:34.333 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 53.701943372999999, 2024-09-21T12:22:34.333 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0306053990000001, 2024-09-21T12:22:34.334 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:22:34.334 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:22:34.334 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:34.334 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:22:34.334 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:22:34.334 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:34.334 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 24423 2024-09-21T12:22:34.334 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.334 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:22:34.334 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.334 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:22:34.334 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.057046+0000", 2024-09-21T12:22:34.334 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.334 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.335 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.335 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:22:34.335 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.057046+0000", 2024-09-21T12:22:34.335 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.335 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.335 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.335 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:22:34.335 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.057048+0000", 2024-09-21T12:22:34.335 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6169999999999999e-06 2024-09-21T12:22:34.335 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.335 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.335 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:22:34.335 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.057070+0000", 2024-09-21T12:22:34.336 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2320999999999999e-05 2024-09-21T12:22:34.336 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.336 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.336 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:22:34.336 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.057083+0000", 2024-09-21T12:22:34.336 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2917e-05 2024-09-21T12:22:34.336 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.336 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.336 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:22:34.336 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.057091+0000", 2024-09-21T12:22:34.336 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.661e-06 2024-09-21T12:22:34.336 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.336 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.336 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.337 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.057161+0000", 2024-09-21T12:22:34.337 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0339000000000005e-05 2024-09-21T12:22:34.337 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.337 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.337 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:22:34.337 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.057173+0000", 2024-09-21T12:22:34.337 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2061e-05 2024-09-21T12:22:34.337 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.337 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.337 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.337 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.024342+0000", 2024-09-21T12:22:34.337 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.96716935900000001 2024-09-21T12:22:34.337 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.337 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.338 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:22:34.338 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.024353+0000", 2024-09-21T12:22:34.338 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0808999999999999e-05 2024-09-21T12:22:34.338 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.338 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.338 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.338 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.079289+0000", 2024-09-21T12:22:34.338 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0549363009999999 2024-09-21T12:22:34.338 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.338 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.338 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:22:34.338 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.079427+0000", 2024-09-21T12:22:34.338 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013735599999999999 2024-09-21T12:22:34.338 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.339 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.339 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:22:34.339 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.079592+0000", 2024-09-21T12:22:34.339 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016532 2024-09-21T12:22:34.339 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.339 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.339 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.339 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.082335+0000", 2024-09-21T12:22:34.339 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0027429020000000002 2024-09-21T12:22:34.339 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.339 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.339 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:22:34.339 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.082510+0000", 2024-09-21T12:22:34.339 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017516 2024-09-21T12:22:34.340 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.340 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.340 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.340 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.086126+0000", 2024-09-21T12:22:34.340 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0036157950000000002 2024-09-21T12:22:34.340 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.340 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.340 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:22:34.340 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.086183+0000", 2024-09-21T12:22:34.340 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7157000000000002e-05 2024-09-21T12:22:34.340 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.340 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.340 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:22:34.341 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.087651+0000", 2024-09-21T12:22:34.341 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001468324 2024-09-21T12:22:34.341 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.341 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:34.341 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.341 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.341 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.341 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.14478.0:1115 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 94620848893312] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-21T12:22:34.341 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:40.364223+0000", 2024-09-21T12:22:34.341 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 53.394766853999997, 2024-09-21T12:22:34.341 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.66897492700000005, 2024-09-21T12:22:34.341 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:22:34.341 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:22:34.341 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:22:34.342 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:22:34.342 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.14478", 2024-09-21T12:22:34.342 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:22:34.342 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 1115 2024-09-21T12:22:34.342 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.342 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:22:34.342 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.342 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:22:34.342 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364223+0000", 2024-09-21T12:22:34.342 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.342 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.342 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.342 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:22:34.342 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364223+0000", 2024-09-21T12:22:34.343 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.343 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.343 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.343 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:22:34.343 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364225+0000", 2024-09-21T12:22:34.343 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3410000000000001e-06 2024-09-21T12:22:34.343 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.343 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.343 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:22:34.343 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364232+0000", 2024-09-21T12:22:34.343 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0879999999999997e-06 2024-09-21T12:22:34.343 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.343 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.343 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:22:34.344 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364254+0000", 2024-09-21T12:22:34.344 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1945000000000001e-05 2024-09-21T12:22:34.344 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.344 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.344 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:22:34.344 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364264+0000", 2024-09-21T12:22:34.344 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.781e-06 2024-09-21T12:22:34.344 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.344 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.344 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.344 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364516+0000", 2024-09-21T12:22:34.344 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00025249199999999999 2024-09-21T12:22:34.344 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.344 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.344 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:22:34.345 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364547+0000", 2024-09-21T12:22:34.345 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.0797999999999999e-05 2024-09-21T12:22:34.345 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.345 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.345 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.345 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.030239+0000", 2024-09-21T12:22:34.345 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.66569158500000003 2024-09-21T12:22:34.345 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.345 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.345 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for degraded object", 2024-09-21T12:22:34.345 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.030551+0000", 2024-09-21T12:22:34.345 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00031200999999999999 2024-09-21T12:22:34.345 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.345 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.346 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.346 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.032993+0000", 2024-09-21T12:22:34.346 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0024427419999999999 2024-09-21T12:22:34.346 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.346 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.346 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:22:34.346 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.033084+0000", 2024-09-21T12:22:34.346 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1180000000000005e-05 2024-09-21T12:22:34.346 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.346 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.346 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:22:34.346 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.033197+0000", 2024-09-21T12:22:34.346 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000112965 2024-09-21T12:22:34.347 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.347 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:34.347 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.347 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.347 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.347 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.14478.0:1122 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94620848893312 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:22:34.347 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:40.364792+0000", 2024-09-21T12:22:34.347 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 53.394197370000001, 2024-09-21T12:22:34.347 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.66963756900000004, 2024-09-21T12:22:34.347 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:22:34.347 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:22:34.347 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:22:34.347 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:22:34.347 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.14478", 2024-09-21T12:22:34.347 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:22:34.348 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 1122 2024-09-21T12:22:34.348 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.348 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:22:34.348 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.348 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:22:34.348 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364792+0000", 2024-09-21T12:22:34.348 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.348 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.348 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.348 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:22:34.348 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364792+0000", 2024-09-21T12:22:34.348 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.348 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.348 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.349 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:22:34.349 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364793+0000", 2024-09-21T12:22:34.349 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.7400000000000002e-07 2024-09-21T12:22:34.349 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.349 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.349 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:22:34.349 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364797+0000", 2024-09-21T12:22:34.349 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8360000000000002e-06 2024-09-21T12:22:34.349 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.349 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.349 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:22:34.349 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364809+0000", 2024-09-21T12:22:34.349 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1849e-05 2024-09-21T12:22:34.349 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.350 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.350 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:22:34.350 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364814+0000", 2024-09-21T12:22:34.350 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1370000000000001e-06 2024-09-21T12:22:34.350 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.350 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.350 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.350 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364855+0000", 2024-09-21T12:22:34.350 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1736999999999997e-05 2024-09-21T12:22:34.350 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.350 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.350 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:22:34.350 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364862+0000", 2024-09-21T12:22:34.350 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4049999999999997e-06 2024-09-21T12:22:34.351 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.351 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.351 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.351 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.030691+0000", 2024-09-21T12:22:34.351 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.66582926799999997 2024-09-21T12:22:34.351 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.351 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.351 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for degraded object", 2024-09-21T12:22:34.351 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.030745+0000", 2024-09-21T12:22:34.351 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3947999999999999e-05 2024-09-21T12:22:34.351 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.351 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.351 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.351 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.033242+0000", 2024-09-21T12:22:34.352 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0024973220000000002 2024-09-21T12:22:34.352 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.352 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.352 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:22:34.352 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.033302+0000", 2024-09-21T12:22:34.352 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9738000000000003e-05 2024-09-21T12:22:34.352 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.352 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.352 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:22:34.352 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.034430+0000", 2024-09-21T12:22:34.352 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0011274550000000001 2024-09-21T12:22:34.352 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.352 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:34.353 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.353 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.353 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.353 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46120 7.3 7:cc2d5e26:::benchmark_data_smithi114_78432_object46119:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-21T12:22:34.353 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:59.864654+0000", 2024-09-21T12:22:34.353 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 33.894335243999997, 2024-09-21T12:22:34.353 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.006362438, 2024-09-21T12:22:34.353 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:22:34.353 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:22:34.353 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:34.353 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:22:34.353 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:22:34.353 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:34.353 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46120 2024-09-21T12:22:34.354 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.354 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:22:34.354 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.354 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:22:34.354 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.864654+0000", 2024-09-21T12:22:34.354 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.354 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.354 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.354 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:22:34.354 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.864654+0000", 2024-09-21T12:22:34.354 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.354 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.354 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.354 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:22:34.355 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.864656+0000", 2024-09-21T12:22:34.355 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5999999999999999e-06 2024-09-21T12:22:34.355 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.355 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.355 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:22:34.355 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.864678+0000", 2024-09-21T12:22:34.355 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2048999999999998e-05 2024-09-21T12:22:34.355 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.355 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.355 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:22:34.355 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.864690+0000", 2024-09-21T12:22:34.355 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2533e-05 2024-09-21T12:22:34.355 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.356 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.356 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:22:34.356 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.864698+0000", 2024-09-21T12:22:34.356 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4950000000000002e-06 2024-09-21T12:22:34.356 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.356 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.356 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.356 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.864757+0000", 2024-09-21T12:22:34.356 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9276e-05 2024-09-21T12:22:34.356 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.356 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.356 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:22:34.356 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.864767+0000", 2024-09-21T12:22:34.357 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0134e-05 2024-09-21T12:22:34.357 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.357 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.357 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.357 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.865518+0000", 2024-09-21T12:22:34.357 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0007512750000001 2024-09-21T12:22:34.357 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.357 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.357 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:22:34.357 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.865632+0000", 2024-09-21T12:22:34.357 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000113299 2024-09-21T12:22:34.357 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.357 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.357 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:22:34.358 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.865781+0000", 2024-09-21T12:22:34.358 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000149386 2024-09-21T12:22:34.358 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.358 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.358 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.358 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.867635+0000", 2024-09-21T12:22:34.358 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001853873 2024-09-21T12:22:34.358 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.358 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.358 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:22:34.358 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.870752+0000", 2024-09-21T12:22:34.358 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0031166980000000002 2024-09-21T12:22:34.358 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.358 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.359 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.359 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.870919+0000", 2024-09-21T12:22:34.359 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016756599999999999 2024-09-21T12:22:34.359 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.359 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.359 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:22:34.359 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.870978+0000", 2024-09-21T12:22:34.359 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8734999999999999e-05 2024-09-21T12:22:34.359 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.359 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.359 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:22:34.359 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.871017+0000", 2024-09-21T12:22:34.359 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8519000000000002e-05 2024-09-21T12:22:34.359 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.360 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:34.360 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.360 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.360 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.360 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46176 7.3 7:cbc365d5:::benchmark_data_smithi114_78432_object46175:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-21T12:22:34.360 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:59.906213+0000", 2024-09-21T12:22:34.360 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 33.852776194, 2024-09-21T12:22:34.360 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.96654458399999998, 2024-09-21T12:22:34.360 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:22:34.360 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:22:34.360 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:34.360 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:22:34.360 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:22:34.360 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:34.361 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46176 2024-09-21T12:22:34.361 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.361 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:22:34.361 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.361 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:22:34.361 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.906213+0000", 2024-09-21T12:22:34.361 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.361 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.361 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.361 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:22:34.361 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.906213+0000", 2024-09-21T12:22:34.361 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.361 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.361 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.362 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:22:34.362 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.906215+0000", 2024-09-21T12:22:34.362 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6139999999999999e-06 2024-09-21T12:22:34.362 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.362 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.362 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:22:34.362 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.906236+0000", 2024-09-21T12:22:34.362 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1457000000000001e-05 2024-09-21T12:22:34.362 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.362 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.362 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:22:34.362 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.906249+0000", 2024-09-21T12:22:34.362 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2252999999999999e-05 2024-09-21T12:22:34.362 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.363 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.363 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:22:34.363 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.906256+0000", 2024-09-21T12:22:34.363 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4839999999999998e-06 2024-09-21T12:22:34.363 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.363 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.363 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.363 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.906306+0000", 2024-09-21T12:22:34.363 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9719000000000003e-05 2024-09-21T12:22:34.363 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.363 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.363 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:22:34.363 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.906314+0000", 2024-09-21T12:22:34.363 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9289999999999998e-06 2024-09-21T12:22:34.364 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.364 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.364 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.364 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.867653+0000", 2024-09-21T12:22:34.364 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.96133975599999999 2024-09-21T12:22:34.364 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.364 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.364 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:22:34.364 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.867717+0000", 2024-09-21T12:22:34.364 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3133999999999996e-05 2024-09-21T12:22:34.364 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.364 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.364 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:22:34.364 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.867796+0000", 2024-09-21T12:22:34.365 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9532999999999999e-05 2024-09-21T12:22:34.365 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.365 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.365 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.365 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.870868+0000", 2024-09-21T12:22:34.365 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0030720370000000001 2024-09-21T12:22:34.365 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.365 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.365 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:22:34.365 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.870880+0000", 2024-09-21T12:22:34.365 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1610999999999999e-05 2024-09-21T12:22:34.365 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.365 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.365 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.366 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.872686+0000", 2024-09-21T12:22:34.366 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0018067280000000001 2024-09-21T12:22:34.366 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.366 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.366 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:22:34.366 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.872719+0000", 2024-09-21T12:22:34.366 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2938999999999998e-05 2024-09-21T12:22:34.366 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.366 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.366 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:22:34.366 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.872758+0000", 2024-09-21T12:22:34.366 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8389999999999997e-05 2024-09-21T12:22:34.366 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.366 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:34.367 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.367 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.367 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.367 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46206 7.3 7:cc04789a:::benchmark_data_smithi114_78432_object46205:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-21T12:22:34.367 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:59.928010+0000", 2024-09-21T12:22:34.367 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 33.830979526, 2024-09-21T12:22:34.367 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.94644473799999995, 2024-09-21T12:22:34.367 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:22:34.367 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:22:34.367 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:34.367 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:22:34.367 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:22:34.367 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:34.368 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46206 2024-09-21T12:22:34.368 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.368 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:22:34.368 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.368 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:22:34.368 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.928010+0000", 2024-09-21T12:22:34.368 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.368 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.368 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.368 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:22:34.368 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.928010+0000", 2024-09-21T12:22:34.368 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.368 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.368 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.369 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:22:34.369 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.928011+0000", 2024-09-21T12:22:34.369 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6050000000000001e-06 2024-09-21T12:22:34.369 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.369 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.369 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:22:34.369 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.928039+0000", 2024-09-21T12:22:34.369 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.7185000000000001e-05 2024-09-21T12:22:34.369 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.369 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.369 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:22:34.369 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.928051+0000", 2024-09-21T12:22:34.369 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2622e-05 2024-09-21T12:22:34.369 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.370 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.370 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:22:34.370 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.928059+0000", 2024-09-21T12:22:34.370 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2409999999999998e-06 2024-09-21T12:22:34.370 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.370 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.370 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.370 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.928121+0000", 2024-09-21T12:22:34.370 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2769999999999997e-05 2024-09-21T12:22:34.370 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.370 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.370 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:22:34.370 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.928131+0000", 2024-09-21T12:22:34.370 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.5459999999999997e-06 2024-09-21T12:22:34.371 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.371 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.371 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.371 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.869100+0000", 2024-09-21T12:22:34.371 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.94096877300000004 2024-09-21T12:22:34.371 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.371 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.371 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:22:34.371 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.869242+0000", 2024-09-21T12:22:34.371 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014223799999999999 2024-09-21T12:22:34.371 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.371 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.371 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:22:34.372 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.869383+0000", 2024-09-21T12:22:34.372 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000140755 2024-09-21T12:22:34.372 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.372 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.372 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.372 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.871074+0000", 2024-09-21T12:22:34.372 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0016910969999999999 2024-09-21T12:22:34.372 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.372 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.372 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:22:34.372 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.874287+0000", 2024-09-21T12:22:34.372 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0032134260000000001 2024-09-21T12:22:34.372 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.372 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.372 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.373 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.874360+0000", 2024-09-21T12:22:34.373 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2896000000000001e-05 2024-09-21T12:22:34.373 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.373 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.373 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:22:34.373 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.874406+0000", 2024-09-21T12:22:34.373 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5509000000000001e-05 2024-09-21T12:22:34.373 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.373 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.373 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:22:34.373 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.874455+0000", 2024-09-21T12:22:34.373 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9075e-05 2024-09-21T12:22:34.373 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.373 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:34.374 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.374 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.374 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.374 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46253 7.3 7:cea9647b:::benchmark_data_smithi114_78432_object46252:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-21T12:22:34.374 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:00.004002+0000", 2024-09-21T12:22:34.374 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 33.754987120999999, 2024-09-21T12:22:34.374 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.87111806700000005, 2024-09-21T12:22:34.374 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:22:34.374 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:22:34.374 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:34.374 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:22:34.374 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:22:34.374 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:34.374 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46253 2024-09-21T12:22:34.375 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.375 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:22:34.375 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.375 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:22:34.375 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.004002+0000", 2024-09-21T12:22:34.375 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.375 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.375 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.375 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:22:34.375 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.004002+0000", 2024-09-21T12:22:34.375 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.375 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.375 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.375 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:22:34.376 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.004004+0000", 2024-09-21T12:22:34.376 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.451e-06 2024-09-21T12:22:34.376 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.376 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.376 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:22:34.376 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.004036+0000", 2024-09-21T12:22:34.376 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2790000000000003e-05 2024-09-21T12:22:34.376 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.376 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.376 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:22:34.376 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.004051+0000", 2024-09-21T12:22:34.376 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4138000000000001e-05 2024-09-21T12:22:34.376 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.377 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.377 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:22:34.377 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.004057+0000", 2024-09-21T12:22:34.377 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9569999999999999e-06 2024-09-21T12:22:34.377 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.377 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.377 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.377 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.004117+0000", 2024-09-21T12:22:34.377 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0891999999999999e-05 2024-09-21T12:22:34.377 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.377 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.377 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:22:34.377 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.004128+0000", 2024-09-21T12:22:34.377 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0939e-05 2024-09-21T12:22:34.378 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.378 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.378 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.378 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.871110+0000", 2024-09-21T12:22:34.378 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.86698113200000004 2024-09-21T12:22:34.378 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.378 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.378 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:22:34.378 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.871213+0000", 2024-09-21T12:22:34.378 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010368400000000001 2024-09-21T12:22:34.378 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.378 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.378 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:22:34.378 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.871319+0000", 2024-09-21T12:22:34.379 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010538 2024-09-21T12:22:34.379 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.379 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.379 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.379 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.874322+0000", 2024-09-21T12:22:34.379 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0030033320000000001 2024-09-21T12:22:34.379 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.379 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.379 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:22:34.379 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.874332+0000", 2024-09-21T12:22:34.379 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6569999999999992e-06 2024-09-21T12:22:34.379 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.379 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.380 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.380 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.875030+0000", 2024-09-21T12:22:34.380 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00069842500000000002 2024-09-21T12:22:34.380 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.380 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.380 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:22:34.380 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.875083+0000", 2024-09-21T12:22:34.380 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3387999999999998e-05 2024-09-21T12:22:34.380 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.380 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.380 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:22:34.380 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.875120+0000", 2024-09-21T12:22:34.380 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.6902e-05 2024-09-21T12:22:34.381 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.381 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:34.381 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.381 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.381 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.381 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46526 7.1 7:859c7591:::benchmark_data_smithi114_78432_object46525:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:22:34.381 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.797069+0000", 2024-09-21T12:22:34.381 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 31.961920620000001, 2024-09-21T12:22:34.381 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.020480762, 2024-09-21T12:22:34.381 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:22:34.381 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:22:34.381 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:34.381 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:22:34.381 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:22:34.382 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:34.382 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46526 2024-09-21T12:22:34.382 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.382 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:22:34.382 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.382 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:22:34.382 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.797069+0000", 2024-09-21T12:22:34.382 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.382 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.382 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.382 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:22:34.382 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.797069+0000", 2024-09-21T12:22:34.382 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.382 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.383 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.383 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:22:34.383 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.797070+0000", 2024-09-21T12:22:34.383 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6339999999999999e-06 2024-09-21T12:22:34.383 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.383 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.383 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:22:34.383 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.797094+0000", 2024-09-21T12:22:34.383 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3870999999999999e-05 2024-09-21T12:22:34.383 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.383 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.383 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:22:34.384 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.797111+0000", 2024-09-21T12:22:34.384 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6495999999999999e-05 2024-09-21T12:22:34.384 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.384 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.384 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:22:34.384 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.797119+0000", 2024-09-21T12:22:34.384 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.3440000000000001e-06 2024-09-21T12:22:34.384 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.384 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.384 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.384 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.797189+0000", 2024-09-21T12:22:34.384 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9733999999999993e-05 2024-09-21T12:22:34.384 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.385 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.385 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:22:34.385 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.797201+0000", 2024-09-21T12:22:34.385 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1803e-05 2024-09-21T12:22:34.385 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.385 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.385 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.385 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.811507+0000", 2024-09-21T12:22:34.385 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0143067020000001 2024-09-21T12:22:34.385 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.385 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.385 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:22:34.385 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.811633+0000", 2024-09-21T12:22:34.385 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012554600000000001 2024-09-21T12:22:34.386 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.386 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.386 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:22:34.386 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.811788+0000", 2024-09-21T12:22:34.386 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015533499999999999 2024-09-21T12:22:34.386 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.386 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.386 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.386 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.815343+0000", 2024-09-21T12:22:34.386 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0035549879999999998 2024-09-21T12:22:34.386 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.386 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.386 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.387 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.815402+0000", 2024-09-21T12:22:34.387 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8409999999999998e-05 2024-09-21T12:22:34.387 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.387 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.387 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:22:34.387 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.817386+0000", 2024-09-21T12:22:34.387 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001984478 2024-09-21T12:22:34.387 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.387 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.387 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:22:34.387 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.817452+0000", 2024-09-21T12:22:34.387 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5679000000000006e-05 2024-09-21T12:22:34.387 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.388 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.388 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:22:34.388 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.817550+0000", 2024-09-21T12:22:34.388 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7742e-05 2024-09-21T12:22:34.388 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.388 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:34.388 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.388 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.388 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.388 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46536 7.1 7:84370686:::benchmark_data_smithi114_78432_object46535:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:22:34.388 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.810616+0000", 2024-09-21T12:22:34.388 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 31.948373051000001, 2024-09-21T12:22:34.389 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0092227149999999, 2024-09-21T12:22:34.389 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:22:34.389 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:22:34.389 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:34.389 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:22:34.389 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:22:34.389 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:34.389 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46536 2024-09-21T12:22:34.389 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.389 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:22:34.389 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.389 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:22:34.389 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.810616+0000", 2024-09-21T12:22:34.389 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.390 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.390 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.390 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:22:34.390 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.810616+0000", 2024-09-21T12:22:34.390 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.390 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.390 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.390 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:22:34.390 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.810617+0000", 2024-09-21T12:22:34.390 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0999999999999998e-07 2024-09-21T12:22:34.390 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.390 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.390 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:22:34.391 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.810641+0000", 2024-09-21T12:22:34.391 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3589000000000001e-05 2024-09-21T12:22:34.391 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.391 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.391 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:22:34.391 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.810660+0000", 2024-09-21T12:22:34.391 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9089e-05 2024-09-21T12:22:34.391 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.391 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.391 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:22:34.391 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.810670+0000", 2024-09-21T12:22:34.391 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0694e-05 2024-09-21T12:22:34.391 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.392 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.392 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.392 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.813144+0000", 2024-09-21T12:22:34.392 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002473598 2024-09-21T12:22:34.392 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.392 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.392 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:22:34.392 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.813155+0000", 2024-09-21T12:22:34.392 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0781999999999999e-05 2024-09-21T12:22:34.392 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.392 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.392 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.392 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.813819+0000", 2024-09-21T12:22:34.392 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0006642649999999 2024-09-21T12:22:34.392 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.393 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.393 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:22:34.393 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.813931+0000", 2024-09-21T12:22:34.393 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000112076 2024-09-21T12:22:34.393 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.393 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.393 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:22:34.393 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.814058+0000", 2024-09-21T12:22:34.393 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000127174 2024-09-21T12:22:34.393 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.393 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.393 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.393 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.819658+0000", 2024-09-21T12:22:34.393 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0056001870000000004 2024-09-21T12:22:34.394 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.394 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.394 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.394 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.819725+0000", 2024-09-21T12:22:34.394 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6711999999999994e-05 2024-09-21T12:22:34.394 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.394 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.394 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:22:34.394 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.819741+0000", 2024-09-21T12:22:34.394 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5341000000000001e-05 2024-09-21T12:22:34.394 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.394 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.394 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:22:34.395 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.819792+0000", 2024-09-21T12:22:34.395 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1418999999999997e-05 2024-09-21T12:22:34.395 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.395 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.395 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:22:34.395 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.819839+0000", 2024-09-21T12:22:34.395 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7079000000000001e-05 2024-09-21T12:22:34.395 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.395 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:34.395 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.395 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.395 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.395 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46544 7.1 7:87ac3b5d:::benchmark_data_smithi114_78432_object46543:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:22:34.395 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.817944+0000", 2024-09-21T12:22:34.395 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 31.941045756000001, 2024-09-21T12:22:34.396 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.002154743, 2024-09-21T12:22:34.396 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:22:34.396 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:22:34.396 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:34.396 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:22:34.396 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:22:34.396 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:34.396 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46544 2024-09-21T12:22:34.396 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.396 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:22:34.396 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.396 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:22:34.396 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.817944+0000", 2024-09-21T12:22:34.396 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.397 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.397 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.397 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:22:34.397 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.817944+0000", 2024-09-21T12:22:34.397 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.397 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.397 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.397 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:22:34.397 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.817945+0000", 2024-09-21T12:22:34.397 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.424e-06 2024-09-21T12:22:34.397 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.397 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.397 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:22:34.397 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.817965+0000", 2024-09-21T12:22:34.398 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0428000000000002e-05 2024-09-21T12:22:34.398 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.398 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.398 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:22:34.398 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.817975+0000", 2024-09-21T12:22:34.398 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1120000000000002e-06 2024-09-21T12:22:34.398 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.398 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.398 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:22:34.398 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.817980+0000", 2024-09-21T12:22:34.398 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6829999999999996e-06 2024-09-21T12:22:34.398 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.398 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.398 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.399 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.818106+0000", 2024-09-21T12:22:34.399 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012569799999999999 2024-09-21T12:22:34.399 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.399 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.399 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:22:34.399 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.818116+0000", 2024-09-21T12:22:34.399 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7110000000000007e-06 2024-09-21T12:22:34.399 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.399 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.399 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.399 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.815448+0000", 2024-09-21T12:22:34.399 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99733251899999997 2024-09-21T12:22:34.399 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.399 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.399 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:22:34.400 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.815583+0000", 2024-09-21T12:22:34.400 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013521900000000001 2024-09-21T12:22:34.400 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.400 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.400 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:22:34.400 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.815750+0000", 2024-09-21T12:22:34.400 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016642600000000001 2024-09-21T12:22:34.400 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.400 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.400 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:22:34.400 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.819850+0000", 2024-09-21T12:22:34.400 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0041004489999999999 2024-09-21T12:22:34.400 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.400 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.401 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.401 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.819916+0000", 2024-09-21T12:22:34.401 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6057999999999997e-05 2024-09-21T12:22:34.401 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.401 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.401 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.401 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.819981+0000", 2024-09-21T12:22:34.401 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4624999999999999e-05 2024-09-21T12:22:34.401 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.401 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.401 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:22:34.401 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.820064+0000", 2024-09-21T12:22:34.401 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.3144000000000002e-05 2024-09-21T12:22:34.401 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.402 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.402 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:22:34.402 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.820098+0000", 2024-09-21T12:22:34.402 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4246999999999999e-05 2024-09-21T12:22:34.402 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.402 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:34.402 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.402 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.402 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.402 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46548 7.1 7:86537d3f:::benchmark_data_smithi114_78432_object46547:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:22:34.402 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.822112+0000", 2024-09-21T12:22:34.402 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 31.936877652, 2024-09-21T12:22:34.402 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.002139297, 2024-09-21T12:22:34.402 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:22:34.403 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:22:34.403 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:34.403 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:22:34.403 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:22:34.403 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:34.403 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46548 2024-09-21T12:22:34.403 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.403 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:22:34.403 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.403 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:22:34.403 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.822112+0000", 2024-09-21T12:22:34.403 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.403 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.403 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.404 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:22:34.404 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.822112+0000", 2024-09-21T12:22:34.404 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.404 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.404 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.404 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:22:34.404 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.822113+0000", 2024-09-21T12:22:34.404 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.299e-06 2024-09-21T12:22:34.404 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.404 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.404 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:22:34.404 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.822143+0000", 2024-09-21T12:22:34.404 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.9869000000000002e-05 2024-09-21T12:22:34.404 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.405 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.405 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:22:34.405 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.822160+0000", 2024-09-21T12:22:34.405 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7450000000000001e-05 2024-09-21T12:22:34.405 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.405 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.405 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:22:34.405 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.822168+0000", 2024-09-21T12:22:34.405 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.0669999999999992e-06 2024-09-21T12:22:34.405 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.405 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.405 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.405 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.822328+0000", 2024-09-21T12:22:34.406 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015988999999999999 2024-09-21T12:22:34.406 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.406 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.406 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:22:34.406 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.822338+0000", 2024-09-21T12:22:34.406 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7799999999999995e-06 2024-09-21T12:22:34.406 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.406 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.406 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.406 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.817706+0000", 2024-09-21T12:22:34.406 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99536773599999995 2024-09-21T12:22:34.406 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.406 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.406 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:22:34.407 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.817826+0000", 2024-09-21T12:22:34.407 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000120677 2024-09-21T12:22:34.407 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.407 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.407 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:22:34.407 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.817957+0000", 2024-09-21T12:22:34.407 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013050099999999999 2024-09-21T12:22:34.407 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.407 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.407 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.407 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.820152+0000", 2024-09-21T12:22:34.407 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0021950199999999998 2024-09-21T12:22:34.407 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.407 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.408 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:22:34.408 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.820162+0000", 2024-09-21T12:22:34.408 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0032e-05 2024-09-21T12:22:34.408 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.408 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.408 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.408 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.824158+0000", 2024-09-21T12:22:34.408 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0039958099999999998 2024-09-21T12:22:34.408 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.408 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.408 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:22:34.408 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.824209+0000", 2024-09-21T12:22:34.408 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0997e-05 2024-09-21T12:22:34.408 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.409 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.409 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:22:34.409 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.824251+0000", 2024-09-21T12:22:34.409 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.2169000000000002e-05 2024-09-21T12:22:34.409 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.409 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:34.409 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.409 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.409 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.409 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46555 7.1 7:86dd50c3:::benchmark_data_smithi114_78432_object46554:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:22:34.409 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.826502+0000", 2024-09-21T12:22:34.409 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 31.932487452, 2024-09-21T12:22:34.409 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.002009758, 2024-09-21T12:22:34.410 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:22:34.410 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:22:34.410 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:34.410 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:22:34.410 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:22:34.410 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:34.410 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46555 2024-09-21T12:22:34.410 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.410 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:22:34.410 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.410 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:22:34.410 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.826502+0000", 2024-09-21T12:22:34.410 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.410 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.411 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.411 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:22:34.411 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.826502+0000", 2024-09-21T12:22:34.411 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.411 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.411 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.411 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:22:34.411 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.826503+0000", 2024-09-21T12:22:34.411 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.5600000000000004e-07 2024-09-21T12:22:34.411 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.411 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.411 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:22:34.411 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.826527+0000", 2024-09-21T12:22:34.412 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3975999999999999e-05 2024-09-21T12:22:34.412 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.412 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.412 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:22:34.412 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.826543+0000", 2024-09-21T12:22:34.412 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6008e-05 2024-09-21T12:22:34.412 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.412 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.412 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:22:34.412 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.826551+0000", 2024-09-21T12:22:34.412 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6720000000000004e-06 2024-09-21T12:22:34.412 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.412 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.412 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.412 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.826673+0000", 2024-09-21T12:22:34.413 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000122774 2024-09-21T12:22:34.413 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.413 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.413 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:22:34.413 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.826682+0000", 2024-09-21T12:22:34.413 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1379999999999996e-06 2024-09-21T12:22:34.413 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.413 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.413 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.413 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.820189+0000", 2024-09-21T12:22:34.413 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99350612699999996 2024-09-21T12:22:34.413 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.413 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.414 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:22:34.414 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.820310+0000", 2024-09-21T12:22:34.414 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000121172 2024-09-21T12:22:34.414 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.414 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.414 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:22:34.414 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.820483+0000", 2024-09-21T12:22:34.414 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000173079 2024-09-21T12:22:34.414 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.414 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.414 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:22:34.414 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.824267+0000", 2024-09-21T12:22:34.414 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0037840059999999999 2024-09-21T12:22:34.414 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.415 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.415 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.415 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.828060+0000", 2024-09-21T12:22:34.415 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00379314 2024-09-21T12:22:34.415 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.415 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.415 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.415 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.828424+0000", 2024-09-21T12:22:34.415 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00036447200000000002 2024-09-21T12:22:34.415 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.415 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.415 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:22:34.415 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.828474+0000", 2024-09-21T12:22:34.415 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9218999999999998e-05 2024-09-21T12:22:34.416 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.416 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.416 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:22:34.416 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.828512+0000", 2024-09-21T12:22:34.416 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8019000000000003e-05 2024-09-21T12:22:34.416 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.416 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:34.416 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.416 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.416 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.416 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46559 7.1 7:8b7c05c1:::benchmark_data_smithi114_78432_object46558:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:22:34.416 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.827486+0000", 2024-09-21T12:22:34.416 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 31.931503143, 2024-09-21T12:22:34.416 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.001462729, 2024-09-21T12:22:34.416 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:22:34.417 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:22:34.417 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:34.417 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:22:34.417 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:22:34.417 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:34.417 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46559 2024-09-21T12:22:34.417 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.417 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:22:34.417 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.417 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:22:34.417 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.827486+0000", 2024-09-21T12:22:34.417 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.417 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.418 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.418 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:22:34.418 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.827486+0000", 2024-09-21T12:22:34.418 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.418 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.418 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.418 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:22:34.418 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.827488+0000", 2024-09-21T12:22:34.418 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.694e-06 2024-09-21T12:22:34.418 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.418 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.418 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:22:34.418 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.827510+0000", 2024-09-21T12:22:34.418 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2019e-05 2024-09-21T12:22:34.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.419 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.419 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:22:34.419 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.827524+0000", 2024-09-21T12:22:34.419 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3789e-05 2024-09-21T12:22:34.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.419 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.419 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:22:34.419 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.827531+0000", 2024-09-21T12:22:34.419 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4680000000000003e-06 2024-09-21T12:22:34.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.419 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.419 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.419 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.827592+0000", 2024-09-21T12:22:34.420 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0396000000000002e-05 2024-09-21T12:22:34.420 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.420 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.420 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:22:34.420 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.827601+0000", 2024-09-21T12:22:34.420 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0389999999999993e-06 2024-09-21T12:22:34.420 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.420 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.420 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.420 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.822243+0000", 2024-09-21T12:22:34.420 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99464195399999999 2024-09-21T12:22:34.420 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.420 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.420 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:22:34.421 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.822347+0000", 2024-09-21T12:22:34.421 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000104283 2024-09-21T12:22:34.421 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.421 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.421 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:22:34.421 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.822495+0000", 2024-09-21T12:22:34.421 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000147737 2024-09-21T12:22:34.421 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.421 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.421 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:22:34.421 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.828006+0000", 2024-09-21T12:22:34.421 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0055110910000000001 2024-09-21T12:22:34.421 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.421 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.422 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.422 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.828124+0000", 2024-09-21T12:22:34.422 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000118528 2024-09-21T12:22:34.422 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.422 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.422 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.422 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.828548+0000", 2024-09-21T12:22:34.422 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00042400199999999998 2024-09-21T12:22:34.422 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.422 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.422 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:22:34.422 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.828593+0000", 2024-09-21T12:22:34.422 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4336999999999999e-05 2024-09-21T12:22:34.422 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.423 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.423 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:22:34.423 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.828949+0000", 2024-09-21T12:22:34.423 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00035639200000000003 2024-09-21T12:22:34.423 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.423 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:34.423 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.423 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.423 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.423 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46583 7.1 7:889d1627:::benchmark_data_smithi114_78432_object46582:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:22:34.423 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.845169+0000", 2024-09-21T12:22:34.423 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 31.913820003000001, 2024-09-21T12:22:34.423 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98807372699999996, 2024-09-21T12:22:34.423 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:22:34.424 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:22:34.424 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:34.424 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:22:34.424 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:22:34.424 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:34.424 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46583 2024-09-21T12:22:34.424 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.424 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:22:34.424 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.424 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:22:34.424 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.845169+0000", 2024-09-21T12:22:34.424 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.424 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.424 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.425 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:22:34.425 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.845169+0000", 2024-09-21T12:22:34.425 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.425 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.425 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.425 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:22:34.425 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.845170+0000", 2024-09-21T12:22:34.425 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.850000000000001e-07 2024-09-21T12:22:34.425 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.425 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.425 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:22:34.425 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.845191+0000", 2024-09-21T12:22:34.425 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1087e-05 2024-09-21T12:22:34.425 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.426 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.426 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:22:34.426 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.845203+0000", 2024-09-21T12:22:34.426 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1428e-05 2024-09-21T12:22:34.426 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.426 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.426 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:22:34.426 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.845210+0000", 2024-09-21T12:22:34.426 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7649999999999996e-06 2024-09-21T12:22:34.426 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.426 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.426 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.426 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.845254+0000", 2024-09-21T12:22:34.426 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4338000000000001e-05 2024-09-21T12:22:34.427 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.427 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.427 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:22:34.427 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.845260+0000", 2024-09-21T12:22:34.427 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0909999999999998e-06 2024-09-21T12:22:34.427 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.427 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.427 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.427 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.824294+0000", 2024-09-21T12:22:34.427 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97903364199999998 2024-09-21T12:22:34.427 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.427 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.427 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:22:34.427 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.824397+0000", 2024-09-21T12:22:34.428 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000103479 2024-09-21T12:22:34.428 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.428 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.428 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:22:34.428 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.824523+0000", 2024-09-21T12:22:34.428 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012569400000000001 2024-09-21T12:22:34.428 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.428 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.428 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.428 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.829568+0000", 2024-09-21T12:22:34.428 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0050454330000000002 2024-09-21T12:22:34.428 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.428 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.428 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:22:34.428 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.830733+0000", 2024-09-21T12:22:34.429 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001165063 2024-09-21T12:22:34.429 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.429 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.429 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.429 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.833137+0000", 2024-09-21T12:22:34.429 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0024032519999999998 2024-09-21T12:22:34.429 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.429 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.429 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:22:34.429 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.833199+0000", 2024-09-21T12:22:34.429 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2794999999999998e-05 2024-09-21T12:22:34.429 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.429 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.429 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:22:34.430 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.833243+0000", 2024-09-21T12:22:34.430 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3674999999999998e-05 2024-09-21T12:22:34.430 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.430 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:34.430 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.430 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.430 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.430 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46587 7.1 7:8649b0f8:::benchmark_data_smithi114_78432_object46586:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:22:34.430 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.847506+0000", 2024-09-21T12:22:34.430 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 31.911483591, 2024-09-21T12:22:34.430 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.985956316, 2024-09-21T12:22:34.430 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:22:34.430 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:22:34.431 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:34.431 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:22:34.431 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:22:34.431 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:34.431 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46587 2024-09-21T12:22:34.431 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.431 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:22:34.431 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.431 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:22:34.431 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.847506+0000", 2024-09-21T12:22:34.431 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.431 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.431 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.431 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:22:34.431 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.847506+0000", 2024-09-21T12:22:34.432 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:22:34.432 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.432 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.432 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:22:34.432 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.847507+0000", 2024-09-21T12:22:34.432 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5000000000000001e-07 2024-09-21T12:22:34.432 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.432 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.432 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:22:34.432 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.847525+0000", 2024-09-21T12:22:34.432 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8082999999999999e-05 2024-09-21T12:22:34.432 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.432 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.432 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:22:34.433 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.847536+0000", 2024-09-21T12:22:34.433 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1270000000000001e-05 2024-09-21T12:22:34.433 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.433 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.433 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:22:34.433 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.847541+0000", 2024-09-21T12:22:34.433 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9799999999999998e-06 2024-09-21T12:22:34.433 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.433 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.433 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.433 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.847595+0000", 2024-09-21T12:22:34.433 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3600999999999997e-05 2024-09-21T12:22:34.433 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.434 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.434 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:22:34.434 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.847600+0000", 2024-09-21T12:22:34.434 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6699999999999999e-06 2024-09-21T12:22:34.434 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.434 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.434 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:22:34.434 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.826230+0000", 2024-09-21T12:22:34.434 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97862941199999998 2024-09-21T12:22:34.434 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.434 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.434 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:22:34.434 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.826353+0000", 2024-09-21T12:22:34.434 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000123433 2024-09-21T12:22:34.435 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.435 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.435 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:22:34.435 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.826485+0000", 2024-09-21T12:22:34.435 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013211899999999999 2024-09-21T12:22:34.435 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.435 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.435 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:22:34.435 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.830740+0000", 2024-09-21T12:22:34.435 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.004254903 2024-09-21T12:22:34.435 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.435 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.435 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.436 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.833294+0000", 2024-09-21T12:22:34.436 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0025538900000000001 2024-09-21T12:22:34.436 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.436 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.436 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:34.436 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.833378+0000", 2024-09-21T12:22:34.436 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.4357999999999994e-05 2024-09-21T12:22:34.436 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.436 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.436 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:22:34.436 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.833426+0000", 2024-09-21T12:22:34.436 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7429000000000003e-05 2024-09-21T12:22:34.436 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:34.437 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:34.437 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:22:34.437 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.833462+0000", 2024-09-21T12:22:34.437 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.6318e-05 2024-09-21T12:22:34.437 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.437 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:34.437 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.437 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:34.437 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:34.437 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:22:34.437 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:34.482 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:34.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:34.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:34.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:34 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3063093644' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:22:34.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:34 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:22:34.489 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:34 smithi183 podman[68643]: 2024-09-21 12:22:34.195984012 +0000 UTC m=+0.436869652 container init cb6f35f47c5b55547c82f9315d527a12248fa8f6c0293d11c3a09cfce41966e4 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:22:34.489 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:34 smithi183 podman[68643]: 2024-09-21 12:22:34.203454928 +0000 UTC m=+0.444340559 container start cb6f35f47c5b55547c82f9315d527a12248fa8f6c0293d11c3a09cfce41966e4 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:22:34.489 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:22:34.239Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-21T12:22:34.489 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:22:34.239Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-21T12:22:34.489 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:22:34.239Z caller=main.go:623 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi183 (none))" 2024-09-21T12:22:34.489 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:22:34.241Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-21T12:22:34.489 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:22:34.242Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-21T12:22:34.489 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:22:34.245Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-21T12:22:34.489 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:22:34.246Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-21T12:22:34.489 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:22:34.247Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-21T12:22:34.489 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:22:34.247Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-21T12:22:34.489 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:22:34.254Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-21T12:22:34.490 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:22:34.254Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.664µs 2024-09-21T12:22:34.490 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:22:34.254Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-21T12:22:34.490 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:22:34.255Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-21T12:22:34.490 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:22:34.255Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-21T12:22:34.490 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:22:34.255Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=44.404µs wal_replay_duration=698.683µs wbl_replay_duration=86ns total_replay_duration=775.607µs 2024-09-21T12:22:34.490 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:22:34.257Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-21T12:22:34.490 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:22:34.257Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-21T12:22:34.490 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:22:34.257Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-21T12:22:34.490 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:22:34.285Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=27.991974ms db_storage=926ns remote_storage=1.721µs web_handler=492ns query_engine=1.027µs scrape=557.394µs scrape_sd=147.942µs notify=17.249µs notify_sd=14.699µs rules=26.64623ms tracing=9.03µs 2024-09-21T12:22:34.490 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:22:34.285Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-21T12:22:34.490 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:22:34.285Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-21T12:22:34.490 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:34 smithi183 bash[68643]: cb6f35f47c5b55547c82f9315d527a12248fa8f6c0293d11c3a09cfce41966e4 2024-09-21T12:22:34.490 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:22:34 smithi183 systemd[1]: Started Ceph prometheus.a for e2ad975c-7812-11ef-baf6-efdc52797490. 2024-09-21T12:22:34.524 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:22:34 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:22:34] ENGINE Bus STOPPING 2024-09-21T12:22:34.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:34 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3063093644' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:22:34.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:34 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:22:34.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:34 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3063093644' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:22:34.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:34 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:22:34.585 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:34.612 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:34.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:34.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:34.636 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 64 15 81366 81351 79.422 57.8125 0.0180625 0.0125858 2024-09-21T12:22:34.714 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:34.759 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:34.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:34.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:34.861 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:34.903 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:34.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:34.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:35.005 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:35.025 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:22:34 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:22:34] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T12:22:35.025 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:22:34 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:22:34] ENGINE Bus STOPPED 2024-09-21T12:22:35.025 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:22:34 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:22:34] ENGINE Bus STARTING 2024-09-21T12:22:35.025 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:22:34 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:22:34] ENGINE Serving on http://:::9283 2024-09-21T12:22:35.025 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:22:34 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:22:34] ENGINE Bus STARTED 2024-09-21T12:22:35.025 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:22:34 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:22:34] ENGINE Bus STOPPING 2024-09-21T12:22:35.052 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:35.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:35.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:35.154 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:35.190 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:35.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:35.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:35.292 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:35.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:35.339 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:35.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:35.440 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:35.451 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:22:35] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T12:22:35.451 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:22:35] ENGINE Bus STOPPED 2024-09-21T12:22:35.451 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:22:35] ENGINE Bus STARTING 2024-09-21T12:22:35.452 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[30324]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:22:35.452 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[30324]: pgmap v96: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 65 MiB/s wr, 1.04k op/s; 22723/236682 objects misplaced (9.601%); 13 MiB/s, 207 objects/s recovering 2024-09-21T12:22:35.452 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[30324]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:22:35.452 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:22:35.452 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[30324]: osdmap e115: 8 total, 8 up, 8 in 2024-09-21T12:22:35.452 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:35.452 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:35.453 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-21T12:22:35.453 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi114.front.sepia.ceph.com:9093"} : dispatch 2024-09-21T12:22:35.453 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:35.453 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-21T12:22:35.453 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi183.front.sepia.ceph.com:9095"} : dispatch 2024-09-21T12:22:35.453 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:35.453 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-21T12:22:35.453 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi183.front.sepia.ceph.com:3000"} : dispatch 2024-09-21T12:22:35.453 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:35.453 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[30324]: 5.2 scrub starts 2024-09-21T12:22:35.453 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[30324]: 5.2 scrub ok 2024-09-21T12:22:35.453 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T12:22:35.453 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:22:35.453 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[30324]: 4.11 scrub starts 2024-09-21T12:22:35.453 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[30324]: 4.11 scrub ok 2024-09-21T12:22:35.454 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[40867]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:22:35.454 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[40867]: pgmap v96: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 65 MiB/s wr, 1.04k op/s; 22723/236682 objects misplaced (9.601%); 13 MiB/s, 207 objects/s recovering 2024-09-21T12:22:35.454 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[40867]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:22:35.454 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:22:35.454 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[40867]: osdmap e115: 8 total, 8 up, 8 in 2024-09-21T12:22:35.454 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:35.454 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:35.454 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-21T12:22:35.454 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi114.front.sepia.ceph.com:9093"} : dispatch 2024-09-21T12:22:35.454 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:35.455 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-21T12:22:35.455 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi183.front.sepia.ceph.com:9095"} : dispatch 2024-09-21T12:22:35.455 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:35.455 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-21T12:22:35.455 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi183.front.sepia.ceph.com:3000"} : dispatch 2024-09-21T12:22:35.455 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:35.455 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[40867]: 5.2 scrub starts 2024-09-21T12:22:35.455 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[40867]: 5.2 scrub ok 2024-09-21T12:22:35.455 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T12:22:35.456 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:22:35.456 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[40867]: 4.11 scrub starts 2024-09-21T12:22:35.456 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-mon[40867]: 4.11 scrub ok 2024-09-21T12:22:35.457 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T12:22:35.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:35 smithi183 ceph-mon[33277]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:22:35.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:35 smithi183 ceph-mon[33277]: pgmap v96: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 65 MiB/s wr, 1.04k op/s; 22723/236682 objects misplaced (9.601%); 13 MiB/s, 207 objects/s recovering 2024-09-21T12:22:35.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:35 smithi183 ceph-mon[33277]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:22:35.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:35 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:22:35.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:35 smithi183 ceph-mon[33277]: osdmap e115: 8 total, 8 up, 8 in 2024-09-21T12:22:35.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:35 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:35.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:35 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:35.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:35 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-21T12:22:35.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:35 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi114.front.sepia.ceph.com:9093"} : dispatch 2024-09-21T12:22:35.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:35 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:35.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:35 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-21T12:22:35.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:35 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi183.front.sepia.ceph.com:9095"} : dispatch 2024-09-21T12:22:35.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:35 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:35.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:35 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-21T12:22:35.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:35 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi183.front.sepia.ceph.com:3000"} : dispatch 2024-09-21T12:22:35.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:35 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:35.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:35 smithi183 ceph-mon[33277]: 5.2 scrub starts 2024-09-21T12:22:35.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:35 smithi183 ceph-mon[33277]: 5.2 scrub ok 2024-09-21T12:22:35.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:35 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T12:22:35.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:35 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:22:35.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:35 smithi183 ceph-mon[33277]: 4.11 scrub starts 2024-09-21T12:22:35.491 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:35 smithi183 ceph-mon[33277]: 4.11 scrub ok 2024-09-21T12:22:35.495 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:35.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:35.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:35.597 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:35.637 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 65 16 82283 82267 79.0808 57.25 0.0136072 0.0126398 2024-09-21T12:22:35.639 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:35.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:35.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:35.741 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:35.775 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:22:35] ENGINE Serving on http://:::9283 2024-09-21T12:22:35.775 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:22:35] ENGINE Bus STARTED 2024-09-21T12:22:35.775 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:22:35] ENGINE Bus STOPPING 2024-09-21T12:22:35.775 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:22:35] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T12:22:35.775 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:22:35] ENGINE Bus STOPPED 2024-09-21T12:22:35.775 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:22:35] ENGINE Bus STARTING 2024-09-21T12:22:35.775 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:22:35] ENGINE Serving on http://:::9283 2024-09-21T12:22:35.775 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:22:35 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:22:35] ENGINE Bus STARTED 2024-09-21T12:22:35.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:35.790 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:35.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:35.890 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:35.934 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:35.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:35.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:36.035 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:36.077 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:36.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:36.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:36.178 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:36.207 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:36.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:36.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:36.309 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:36.342 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:22:36.356 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:36.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:36.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:36.460 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:36.477 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:36 smithi183 ceph-mon[33277]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T12:22:36.477 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:36 smithi183 ceph-mon[33277]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi114.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T12:22:36.477 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:36 smithi183 ceph-mon[33277]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T12:22:36.477 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:36 smithi183 ceph-mon[33277]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi183.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T12:22:36.478 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:36 smithi183 ceph-mon[33277]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T12:22:36.478 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:36 smithi183 ceph-mon[33277]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi183.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T12:22:36.478 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:36 smithi183 ceph-mon[33277]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:22:36.478 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:36 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T12:22:36.478 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:36 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T12:22:36.478 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:36 smithi183 ceph-mon[33277]: 6.b scrub starts 2024-09-21T12:22:36.478 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:36 smithi183 ceph-mon[33277]: 6.b scrub ok 2024-09-21T12:22:36.490 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:36.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:36.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:36.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:36 smithi114 ceph-mon[40867]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T12:22:36.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:36 smithi114 ceph-mon[40867]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi114.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T12:22:36.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:36 smithi114 ceph-mon[40867]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T12:22:36.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:36 smithi114 ceph-mon[40867]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi183.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T12:22:36.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:36 smithi114 ceph-mon[40867]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T12:22:36.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:36 smithi114 ceph-mon[40867]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi183.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T12:22:36.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:36 smithi114 ceph-mon[40867]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:22:36.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:36 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T12:22:36.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:36 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T12:22:36.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:36 smithi114 ceph-mon[40867]: 6.b scrub starts 2024-09-21T12:22:36.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:36 smithi114 ceph-mon[40867]: 6.b scrub ok 2024-09-21T12:22:36.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:36 smithi114 ceph-mon[30324]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T12:22:36.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:36 smithi114 ceph-mon[30324]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi114.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T12:22:36.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:36 smithi114 ceph-mon[30324]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T12:22:36.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:36 smithi114 ceph-mon[30324]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi183.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T12:22:36.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:36 smithi114 ceph-mon[30324]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T12:22:36.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:36 smithi114 ceph-mon[30324]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi183.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T12:22:36.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:36 smithi114 ceph-mon[30324]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:22:36.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:36 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T12:22:36.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:36 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T12:22:36.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:36 smithi114 ceph-mon[30324]: 6.b scrub starts 2024-09-21T12:22:36.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:36 smithi114 ceph-mon[30324]: 6.b scrub ok 2024-09-21T12:22:36.592 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:36.623 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:36.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:36.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:36.637 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 66 16 83286 83270 78.8324 62.6875 0.0109487 0.0126789 2024-09-21T12:22:36.725 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:36.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:22:36 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:22:36.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:36.761 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:36.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:36.862 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:36.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:36.899 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:36.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:36.962 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:22:36.962 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-21T12:22:36.963 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 0 2024-09-21T12:22:36.963 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:22:37.000 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:37.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:37.039 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:37.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:37.140 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:37.170 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:37.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:37.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:37.272 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:37.302 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:37.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:37.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:37.405 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:37.446 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:37.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:37.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:37.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:37 smithi183 ceph-mon[33277]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:22:37.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:37 smithi183 ceph-mon[33277]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:22:37.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:37 smithi183 ceph-mon[33277]: pgmap v98: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 582 B/s rd, 35 MiB/s wr, 561 op/s; 22723/236682 objects misplaced (9.601%); 11 MiB/s, 169 objects/s recovering 2024-09-21T12:22:37.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:37 smithi183 ceph-mon[33277]: 5.7 scrub starts 2024-09-21T12:22:37.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:37 smithi183 ceph-mon[33277]: 5.7 scrub ok 2024-09-21T12:22:37.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:37 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:22:37.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:37 smithi183 ceph-mon[33277]: osdmap e116: 8 total, 8 up, 8 in 2024-09-21T12:22:37.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:37 smithi183 ceph-mon[33277]: 4.17 scrub starts 2024-09-21T12:22:37.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:37 smithi183 ceph-mon[33277]: 4.17 scrub ok 2024-09-21T12:22:37.524 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:22:37 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a[85236]: ts=2024-09-21T12:22:37.089Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.004209151s 2024-09-21T12:22:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:37 smithi114 ceph-mon[40867]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:22:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:37 smithi114 ceph-mon[40867]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:22:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:37 smithi114 ceph-mon[40867]: pgmap v98: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 582 B/s rd, 35 MiB/s wr, 561 op/s; 22723/236682 objects misplaced (9.601%); 11 MiB/s, 169 objects/s recovering 2024-09-21T12:22:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:37 smithi114 ceph-mon[40867]: 5.7 scrub starts 2024-09-21T12:22:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:37 smithi114 ceph-mon[40867]: 5.7 scrub ok 2024-09-21T12:22:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:37 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:22:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:37 smithi114 ceph-mon[40867]: osdmap e116: 8 total, 8 up, 8 in 2024-09-21T12:22:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:37 smithi114 ceph-mon[40867]: 4.17 scrub starts 2024-09-21T12:22:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:37 smithi114 ceph-mon[40867]: 4.17 scrub ok 2024-09-21T12:22:37.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:37 smithi114 ceph-mon[30324]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:22:37.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:37 smithi114 ceph-mon[30324]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:22:37.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:37 smithi114 ceph-mon[30324]: pgmap v98: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 582 B/s rd, 35 MiB/s wr, 561 op/s; 22723/236682 objects misplaced (9.601%); 11 MiB/s, 169 objects/s recovering 2024-09-21T12:22:37.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:37 smithi114 ceph-mon[30324]: 5.7 scrub starts 2024-09-21T12:22:37.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:37 smithi114 ceph-mon[30324]: 5.7 scrub ok 2024-09-21T12:22:37.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:37 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:22:37.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:37 smithi114 ceph-mon[30324]: osdmap e116: 8 total, 8 up, 8 in 2024-09-21T12:22:37.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:37 smithi114 ceph-mon[30324]: 4.17 scrub starts 2024-09-21T12:22:37.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:37 smithi114 ceph-mon[30324]: 4.17 scrub ok 2024-09-21T12:22:37.547 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:37.576 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:37.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:37.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:37.637 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 67 16 84202 84186 78.5103 57.25 0.0195189 0.0127315 2024-09-21T12:22:37.677 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:37.710 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:37.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:37.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:37.812 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:37.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:37.842 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:37.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:37.943 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:37.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:37.977 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:37.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:38.038 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T12:22:38.078 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:38.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:38.118 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:38.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:38.219 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:38.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:38 smithi183 ceph-mon[33277]: 3.5 scrub starts 2024-09-21T12:22:38.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:38 smithi183 ceph-mon[33277]: 3.5 scrub ok 2024-09-21T12:22:38.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:38 smithi183 ceph-mon[33277]: pgmap v100: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.19k op/s; 22337/250524 objects misplaced (8.916%); 23 MiB/s, 366 objects/s recovering 2024-09-21T12:22:38.257 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:38.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:38.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:38.358 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:38.402 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:38.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:38.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:38.469 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:38 smithi114 ceph-mon[40867]: 3.5 scrub starts 2024-09-21T12:22:38.469 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:38 smithi114 ceph-mon[40867]: 3.5 scrub ok 2024-09-21T12:22:38.469 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:38 smithi114 ceph-mon[40867]: pgmap v100: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.19k op/s; 22337/250524 objects misplaced (8.916%); 23 MiB/s, 366 objects/s recovering 2024-09-21T12:22:38.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:38 smithi114 ceph-mon[30324]: 3.5 scrub starts 2024-09-21T12:22:38.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:38 smithi114 ceph-mon[30324]: 3.5 scrub ok 2024-09-21T12:22:38.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:38 smithi114 ceph-mon[30324]: pgmap v100: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.19k op/s; 22337/250524 objects misplaced (8.916%); 23 MiB/s, 366 objects/s recovering 2024-09-21T12:22:38.504 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:38.555 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:38.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:38.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:38.638 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 68 16 85254 85238 78.3226 65.75 0.0160304 0.0127619 2024-09-21T12:22:38.657 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:38.695 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:38.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:38.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:38.799 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:38.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:38.842 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:38.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:38.872 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:22:38.943 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:38.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:38.973 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:38.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:39.074 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:39.122 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2024-09-21T12:22:31.581312+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4933276549,"num_objects":75469,"num_object_clones":0,"num_object_copies":226407,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22911,"num_objects_unfound":0,"num_objects_dirty":75469,"num_whiteouts":0,"num_read":919,"num_read_kb":739,"num_write":75778,"num_write_kb":4818361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7898,"num_bytes_recovered":510248970,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71958,"ondisk_log_size":71958,"up":474,"acting":473,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":500,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15092464,"kb_used_data":14537564,"kb_used_omap":12,"kb_used_meta":554739,"kb_avail":734868752,"statfs":{"total":767960285184,"available":752505602048,"internally_reserved":0,"allocated":14886465536,"data_stored":14881316169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":568053296},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":318570496,"num_objects":4861,"num_object_clones":0,"num_object_copies":14583,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22911,"num_objects_unfound":0,"num_objects_dirty":4861,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4861,"num_write_kb":311104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":8258534,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.471358"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":129,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.116877+0000","last_change":"2024-09-21T12:22:16.705200+0000","last_active":"2024-09-21T12:22:26.116877+0000","last_peered":"2024-09-21T12:22:26.116877+0000","last_clean":"2024-09-21T12:22:26.116877+0000","last_became_active":"2024-09-21T12:20:15.309431+0000","last_became_peered":"2024-09-21T12:20:15.309431+0000","last_unstale":"2024-09-21T12:22:26.116877+0000","last_undegraded":"2024-09-21T12:22:26.116877+0000","last_fullsized":"2024-09-21T12:22:26.116877+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:16.705088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:16.705088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:28.011157+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":26,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.789004+0000","last_change":"2024-09-21T12:22:25.164591+0000","last_active":"2024-09-21T12:22:26.789004+0000","last_peered":"2024-09-21T12:22:26.789004+0000","last_clean":"2024-09-21T12:22:26.789004+0000","last_became_active":"2024-09-21T12:22:25.132980+0000","last_became_peered":"2024-09-21T12:22:25.132980+0000","last_unstale":"2024-09-21T12:22:26.789004+0000","last_undegraded":"2024-09-21T12:22:26.789004+0000","last_fullsized":"2024-09-21T12:22:26.789004+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":112,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.135981+0000","last_change":"2024-09-21T12:20:19.315239+0000","last_active":"2024-09-21T12:22:26.135981+0000","last_peered":"2024-09-21T12:22:26.135981+0000","last_clean":"2024-09-21T12:22:26.135981+0000","last_became_active":"2024-09-21T12:20:19.314908+0000","last_became_peered":"2024-09-21T12:20:19.314908+0000","last_unstale":"2024-09-21T12:22:26.135981+0000","last_undegraded":"2024-09-21T12:22:26.135981+0000","last_fullsized":"2024-09-21T12:22:26.135981+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.791191+0000","last_change":"2024-09-21T12:22:25.131083+0000","last_active":"2024-09-21T12:22:26.791191+0000","last_peered":"2024-09-21T12:22:26.791191+0000","last_clean":"2024-09-21T12:22:26.791191+0000","last_became_active":"2024-09-21T12:22:25.126469+0000","last_became_peered":"2024-09-21T12:22:25.126469+0000","last_unstale":"2024-09-21T12:22:26.791191+0000","last_undegraded":"2024-09-21T12:22:26.791191+0000","last_fullsized":"2024-09-21T12:22:26.791191+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:58.189665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:21:58.189665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:05.779547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":106,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.130959+0000","last_change":"2024-09-21T12:20:21.595607+0000","last_active":"2024-09-21T12:22:26.130959+0000","last_peered":"2024-09-21T12:22:26.130959+0000","last_clean":"2024-09-21T12:22:26.130959+0000","last_became_active":"2024-09-21T12:20:21.595301+0000","last_became_peered":"2024-09-21T12:20:21.595301+0000","last_unstale":"2024-09-21T12:22:26.130959+0000","last_undegraded":"2024-09-21T12:22:26.130959+0000","last_fullsized":"2024-09-21T12:22:26.130959+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:25.025934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":119,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.124058+0000","last_change":"2024-09-21T12:20:15.308941+0000","last_active":"2024-09-21T12:22:26.124058+0000","last_peered":"2024-09-21T12:22:26.124058+0000","last_clean":"2024-09-21T12:22:26.124058+0000","last_became_active":"2024-09-21T12:20:15.308655+0000","last_became_peered":"2024-09-21T12:20:15.308655+0000","last_unstale":"2024-09-21T12:22:26.124058+0000","last_undegraded":"2024-09-21T12:22:26.124058+0000","last_fullsized":"2024-09-21T12:22:26.124058+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":139,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.123651+0000","last_change":"2024-09-21T12:20:17.305499+0000","last_active":"2024-09-21T12:22:26.123651+0000","last_peered":"2024-09-21T12:22:26.123651+0000","last_clean":"2024-09-21T12:22:26.123651+0000","last_became_active":"2024-09-21T12:20:17.305220+0000","last_became_peered":"2024-09-21T12:20:17.305220+0000","last_unstale":"2024-09-21T12:22:26.123651+0000","last_undegraded":"2024-09-21T12:22:26.123651+0000","last_fullsized":"2024-09-21T12:22:26.123651+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.788847+0000","last_change":"2024-09-21T12:22:25.131090+0000","last_active":"2024-09-21T12:22:26.788847+0000","last_peered":"2024-09-21T12:22:26.788847+0000","last_clean":"2024-09-21T12:22:26.788847+0000","last_became_active":"2024-09-21T12:22:25.129985+0000","last_became_peered":"2024-09-21T12:22:25.129985+0000","last_unstale":"2024-09-21T12:22:26.788847+0000","last_undegraded":"2024-09-21T12:22:26.788847+0000","last_fullsized":"2024-09-21T12:22:26.788847+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:06.670835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:06.670835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:12.735053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":135,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.801575+0000","last_change":"2024-09-21T12:22:25.127057+0000","last_active":"2024-09-21T12:22:26.801575+0000","last_peered":"2024-09-21T12:22:26.801575+0000","last_clean":"2024-09-21T12:22:26.801575+0000","last_became_active":"2024-09-21T12:22:25.126737+0000","last_became_peered":"2024-09-21T12:22:25.126737+0000","last_unstale":"2024-09-21T12:22:26.801575+0000","last_undegraded":"2024-09-21T12:22:26.801575+0000","last_fullsized":"2024-09-21T12:22:26.801575+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":26,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.789643+0000","last_change":"2024-09-21T12:22:25.174256+0000","last_active":"2024-09-21T12:22:26.789643+0000","last_peered":"2024-09-21T12:22:26.789643+0000","last_clean":"2024-09-21T12:22:26.789643+0000","last_became_active":"2024-09-21T12:22:25.135915+0000","last_became_peered":"2024-09-21T12:22:25.135915+0000","last_unstale":"2024-09-21T12:22:26.789643+0000","last_undegraded":"2024-09-21T12:22:26.789643+0000","last_fullsized":"2024-09-21T12:22:26.789643+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:22:03.017467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:03.017467+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:17.534555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":121,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.802158+0000","last_change":"2024-09-21T12:22:15.018306+0000","last_active":"2024-09-21T12:22:26.802158+0000","last_peered":"2024-09-21T12:22:26.802158+0000","last_clean":"2024-09-21T12:22:26.802158+0000","last_became_active":"2024-09-21T12:20:19.308184+0000","last_became_peered":"2024-09-21T12:20:19.308184+0000","last_unstale":"2024-09-21T12:22:26.802158+0000","last_undegraded":"2024-09-21T12:22:26.802158+0000","last_fullsized":"2024-09-21T12:22:26.802158+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:15.018264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:15.018264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:44.133250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":108,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.115875+0000","last_change":"2024-09-21T12:20:21.309871+0000","last_active":"2024-09-21T12:22:26.115875+0000","last_peered":"2024-09-21T12:22:26.115875+0000","last_clean":"2024-09-21T12:22:26.115875+0000","last_became_active":"2024-09-21T12:20:21.309190+0000","last_became_peered":"2024-09-21T12:20:21.309190+0000","last_unstale":"2024-09-21T12:22:26.115875+0000","last_undegraded":"2024-09-21T12:22:26.115875+0000","last_fullsized":"2024-09-21T12:22:26.115875+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":135,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.801200+0000","last_change":"2024-09-21T12:22:25.127080+0000","last_active":"2024-09-21T12:22:26.801200+0000","last_peered":"2024-09-21T12:22:26.801200+0000","last_clean":"2024-09-21T12:22:26.801200+0000","last_became_active":"2024-09-21T12:22:25.126752+0000","last_became_peered":"2024-09-21T12:22:25.126752+0000","last_unstale":"2024-09-21T12:22:26.801200+0000","last_undegraded":"2024-09-21T12:22:26.801200+0000","last_fullsized":"2024-09-21T12:22:26.801200+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":22,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.790111+0000","last_change":"2024-09-21T12:22:25.157833+0000","last_active":"2024-09-21T12:22:26.790111+0000","last_peered":"2024-09-21T12:22:26.790111+0000","last_clean":"2024-09-21T12:22:26.790111+0000","last_became_active":"2024-09-21T12:22:25.132488+0000","last_became_peered":"2024-09-21T12:22:25.132488+0000","last_unstale":"2024-09-21T12:22:26.790111+0000","last_undegraded":"2024-09-21T12:22:26.790111+0000","last_fullsized":"2024-09-21T12:22:26.790111+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-21T12:22:10.011633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:10.011633+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:27.239547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":127,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.803383+0000","last_change":"2024-09-21T12:22:25.140766+0000","last_active":"2024-09-21T12:22:26.803383+0000","last_peered":"2024-09-21T12:22:26.803383+0000","last_clean":"2024-09-21T12:22:26.803383+0000","last_became_active":"2024-09-21T12:22:25.140042+0000","last_became_peered":"2024-09-21T12:22:25.140042+0000","last_unstale":"2024-09-21T12:22:26.803383+0000","last_undegraded":"2024-09-21T12:22:26.803383+0000","last_fullsized":"2024-09-21T12:22:26.803383+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":117,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.800865+0000","last_change":"2024-09-21T12:22:15.977619+0000","last_active":"2024-09-21T12:22:26.800865+0000","last_peered":"2024-09-21T12:22:26.800865+0000","last_clean":"2024-09-21T12:22:26.800865+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:22:26.800865+0000","last_undegraded":"2024-09-21T12:22:26.800865+0000","last_fullsized":"2024-09-21T12:22:26.800865+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:15.977543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:15.977543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:21.175259+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.790759+0000","last_change":"2024-09-21T12:22:25.131639+0000","last_active":"2024-09-21T12:22:26.790759+0000","last_peered":"2024-09-21T12:22:26.790759+0000","last_clean":"2024-09-21T12:22:26.790759+0000","last_became_active":"2024-09-21T12:22:25.131336+0000","last_became_peered":"2024-09-21T12:22:25.131336+0000","last_unstale":"2024-09-21T12:22:26.790759+0000","last_undegraded":"2024-09-21T12:22:26.790759+0000","last_fullsized":"2024-09-21T12:22:26.790759+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:02.712009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:02.712009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:14.438496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":142,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.118517+0000","last_change":"2024-09-21T12:22:25.143132+0000","last_active":"2024-09-21T12:22:26.118517+0000","last_peered":"2024-09-21T12:22:26.118517+0000","last_clean":"2024-09-21T12:22:26.118517+0000","last_became_active":"2024-09-21T12:22:25.142745+0000","last_became_peered":"2024-09-21T12:22:25.142745+0000","last_unstale":"2024-09-21T12:22:26.118517+0000","last_undegraded":"2024-09-21T12:22:26.118517+0000","last_fullsized":"2024-09-21T12:22:26.118517+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:20.217841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.217841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":141,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.117103+0000","last_change":"2024-09-21T12:20:17.308186+0000","last_active":"2024-09-21T12:22:26.117103+0000","last_peered":"2024-09-21T12:22:26.117103+0000","last_clean":"2024-09-21T12:22:26.117103+0000","last_became_active":"2024-09-21T12:20:17.307828+0000","last_became_peered":"2024-09-21T12:20:17.307828+0000","last_unstale":"2024-09-21T12:22:26.117103+0000","last_undegraded":"2024-09-21T12:22:26.117103+0000","last_fullsized":"2024-09-21T12:22:26.117103+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":134,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.128002+0000","last_change":"2024-09-21T12:22:25.136298+0000","last_active":"2024-09-21T12:22:26.128002+0000","last_peered":"2024-09-21T12:22:26.128002+0000","last_clean":"2024-09-21T12:22:26.128002+0000","last_became_active":"2024-09-21T12:22:25.135015+0000","last_became_peered":"2024-09-21T12:22:25.135015+0000","last_unstale":"2024-09-21T12:22:26.128002+0000","last_undegraded":"2024-09-21T12:22:26.128002+0000","last_fullsized":"2024-09-21T12:22:26.128002+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:10.823915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:10.823915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:29.799573+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":107,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.124092+0000","last_change":"2024-09-21T12:20:21.593129+0000","last_active":"2024-09-21T12:22:26.124092+0000","last_peered":"2024-09-21T12:22:26.124092+0000","last_clean":"2024-09-21T12:22:26.124092+0000","last_became_active":"2024-09-21T12:20:21.592796+0000","last_became_peered":"2024-09-21T12:20:21.592796+0000","last_unstale":"2024-09-21T12:22:26.124092+0000","last_undegraded":"2024-09-21T12:22:26.124092+0000","last_fullsized":"2024-09-21T12:22:26.124092+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.789371+0000","last_change":"2024-09-21T12:22:25.130148+0000","last_active":"2024-09-21T12:22:26.789371+0000","last_peered":"2024-09-21T12:22:26.789371+0000","last_clean":"2024-09-21T12:22:26.789371+0000","last_became_active":"2024-09-21T12:22:25.129449+0000","last_became_peered":"2024-09-21T12:22:25.129449+0000","last_unstale":"2024-09-21T12:22:26.789371+0000","last_undegraded":"2024-09-21T12:22:26.789371+0000","last_fullsized":"2024-09-21T12:22:26.789371+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:07.846732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:07.846732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:27.282055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":175,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.123318+0000","last_change":"2024-09-21T12:22:25.148877+0000","last_active":"2024-09-21T12:22:26.123318+0000","last_peered":"2024-09-21T12:22:26.123318+0000","last_clean":"2024-09-21T12:22:26.123318+0000","last_became_active":"2024-09-21T12:22:25.126163+0000","last_became_peered":"2024-09-21T12:22:25.126163+0000","last_unstale":"2024-09-21T12:22:26.123318+0000","last_undegraded":"2024-09-21T12:22:26.123318+0000","last_fullsized":"2024-09-21T12:22:26.123318+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.791056+0000","last_change":"2024-09-21T12:22:25.124501+0000","last_active":"2024-09-21T12:22:26.791056+0000","last_peered":"2024-09-21T12:22:26.791056+0000","last_clean":"2024-09-21T12:22:26.791056+0000","last_became_active":"2024-09-21T12:22:25.124004+0000","last_became_peered":"2024-09-21T12:22:25.124004+0000","last_unstale":"2024-09-21T12:22:26.791056+0000","last_undegraded":"2024-09-21T12:22:26.791056+0000","last_fullsized":"2024-09-21T12:22:26.791056+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":110,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.803341+0000","last_change":"2024-09-21T12:20:21.313476+0000","last_active":"2024-09-21T12:22:26.803341+0000","last_peered":"2024-09-21T12:22:26.803341+0000","last_clean":"2024-09-21T12:22:26.803341+0000","last_became_active":"2024-09-21T12:20:21.312976+0000","last_became_peered":"2024-09-21T12:20:21.312976+0000","last_unstale":"2024-09-21T12:22:26.803341+0000","last_undegraded":"2024-09-21T12:22:26.803341+0000","last_fullsized":"2024-09-21T12:22:26.803341+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:35.977937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":143,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.136159+0000","last_change":"2024-09-21T12:22:25.152911+0000","last_active":"2024-09-21T12:22:26.136159+0000","last_peered":"2024-09-21T12:22:26.136159+0000","last_clean":"2024-09-21T12:22:26.136159+0000","last_became_active":"2024-09-21T12:22:25.134995+0000","last_became_peered":"2024-09-21T12:22:25.134995+0000","last_unstale":"2024-09-21T12:22:26.136159+0000","last_undegraded":"2024-09-21T12:22:26.136159+0000","last_fullsized":"2024-09-21T12:22:26.136159+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":132,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.121607+0000","last_change":"2024-09-21T12:20:17.696423+0000","last_active":"2024-09-21T12:22:26.121607+0000","last_peered":"2024-09-21T12:22:26.121607+0000","last_clean":"2024-09-21T12:22:26.121607+0000","last_became_active":"2024-09-21T12:20:17.695977+0000","last_became_peered":"2024-09-21T12:20:17.695977+0000","last_unstale":"2024-09-21T12:22:26.121607+0000","last_undegraded":"2024-09-21T12:22:26.121607+0000","last_fullsized":"2024-09-21T12:22:26.121607+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"112'10","reported_seq":183,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.795862+0000","last_change":"2024-09-21T12:22:25.144862+0000","last_active":"2024-09-21T12:22:26.795862+0000","last_peered":"2024-09-21T12:22:26.795862+0000","last_clean":"2024-09-21T12:22:26.795862+0000","last_became_active":"2024-09-21T12:22:25.127896+0000","last_became_peered":"2024-09-21T12:22:25.127896+0000","last_unstale":"2024-09-21T12:22:26.795862+0000","last_undegraded":"2024-09-21T12:22:26.795862+0000","last_fullsized":"2024-09-21T12:22:26.795862+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'9","last_scrub_stamp":"2024-09-21T12:22:08.647997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:08.647997+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:17.161042+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"76'79","reported_seq":30,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.800971+0000","last_change":"2024-09-21T12:22:18.000752+0000","last_active":"2024-09-21T12:22:26.800971+0000","last_peered":"2024-09-21T12:22:26.800971+0000","last_clean":"2024-09-21T12:22:26.800971+0000","last_became_active":"2024-09-21T12:22:17.074721+0000","last_became_peered":"2024-09-21T12:22:17.074721+0000","last_unstale":"2024-09-21T12:22:26.800971+0000","last_undegraded":"2024-09-21T12:22:26.800971+0000","last_fullsized":"2024-09-21T12:22:26.800971+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:22:18.000705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:22:18.000705+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:29.821832+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"114'4885","reported_seq":4955,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:31.505677+0000","last_change":"2024-09-21T12:21:24.886463+0000","last_active":"2024-09-21T12:22:31.505677+0000","last_peered":"2024-09-21T12:22:31.505677+0000","last_clean":"2024-09-21T12:22:31.505677+0000","last_became_active":"2024-09-21T12:21:24.886130+0000","last_became_peered":"2024-09-21T12:21:24.886130+0000","last_unstale":"2024-09-21T12:22:31.505677+0000","last_undegraded":"2024-09-21T12:22:31.505677+0000","last_fullsized":"2024-09-21T12:22:31.505677+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4885,"log_dups_size":0,"ondisk_log_size":4885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:37.307039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320143360,"num_objects":4885,"num_object_clones":0,"num_object_copies":14655,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4885,"num_write_kb":312640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":187,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.125494+0000","last_change":"2024-09-21T12:22:25.152279+0000","last_active":"2024-09-21T12:22:26.125494+0000","last_peered":"2024-09-21T12:22:26.125494+0000","last_clean":"2024-09-21T12:22:26.125494+0000","last_became_active":"2024-09-21T12:22:25.134113+0000","last_became_peered":"2024-09-21T12:22:25.134113+0000","last_unstale":"2024-09-21T12:22:26.125494+0000","last_undegraded":"2024-09-21T12:22:26.125494+0000","last_fullsized":"2024-09-21T12:22:26.125494+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":127,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.123016+0000","last_change":"2024-09-21T12:22:14.036290+0000","last_active":"2024-09-21T12:22:26.123016+0000","last_peered":"2024-09-21T12:22:26.123016+0000","last_clean":"2024-09-21T12:22:26.123016+0000","last_became_active":"2024-09-21T12:20:15.295774+0000","last_became_peered":"2024-09-21T12:20:15.295774+0000","last_unstale":"2024-09-21T12:22:26.123016+0000","last_undegraded":"2024-09-21T12:22:26.123016+0000","last_fullsized":"2024-09-21T12:22:26.123016+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:14.036236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.036236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:39.734223+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"112'10","reported_seq":22,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.790714+0000","last_change":"2024-09-21T12:22:25.169509+0000","last_active":"2024-09-21T12:22:26.790714+0000","last_peered":"2024-09-21T12:22:26.790714+0000","last_clean":"2024-09-21T12:22:26.790714+0000","last_became_active":"2024-09-21T12:22:25.135473+0000","last_became_peered":"2024-09-21T12:22:25.135473+0000","last_unstale":"2024-09-21T12:22:26.790714+0000","last_undegraded":"2024-09-21T12:22:26.790714+0000","last_fullsized":"2024-09-21T12:22:26.790714+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":118,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.801107+0000","last_change":"2024-09-21T12:22:12.145637+0000","last_active":"2024-09-21T12:22:26.801107+0000","last_peered":"2024-09-21T12:22:26.801107+0000","last_clean":"2024-09-21T12:22:26.801107+0000","last_became_active":"2024-09-21T12:20:21.310022+0000","last_became_peered":"2024-09-21T12:20:21.310022+0000","last_unstale":"2024-09-21T12:22:26.801107+0000","last_undegraded":"2024-09-21T12:22:26.801107+0000","last_fullsized":"2024-09-21T12:22:26.801107+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:22:12.145550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:12.145550+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:41.591945+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"76'79","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.790676+0000","last_change":"2024-09-21T12:22:25.132943+0000","last_active":"2024-09-21T12:22:26.790676+0000","last_peered":"2024-09-21T12:22:26.790676+0000","last_clean":"2024-09-21T12:22:26.790676+0000","last_became_active":"2024-09-21T12:22:25.132605+0000","last_became_peered":"2024-09-21T12:22:25.132605+0000","last_unstale":"2024-09-21T12:22:26.790676+0000","last_undegraded":"2024-09-21T12:22:26.790676+0000","last_fullsized":"2024-09-21T12:22:26.790676+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:22:17.824342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:22:17.824342+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:29.552825+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"113'4533","reported_seq":7692,"reported_epoch":113,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-21T12:22:26.718262+0000","last_change":"2024-09-21T12:22:26.209691+0000","last_active":"2024-09-21T12:22:26.718262+0000","last_peered":"2024-09-21T12:22:26.718262+0000","last_clean":"2024-09-21T12:22:24.098938+0000","last_became_active":"2024-09-21T12:22:26.200969+0000","last_became_peered":"2024-09-21T12:22:26.200969+0000","last_unstale":"2024-09-21T12:22:26.718262+0000","last_undegraded":"2024-09-21T12:22:26.718262+0000","last_fullsized":"2024-09-21T12:22:26.718262+0000","mapping_epoch":112,"log_start":"89'600","ondisk_log_start":"89'600","created":84,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":3933,"log_dups_size":0,"ondisk_log_size":3933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297074688,"num_objects":4533,"num_object_clones":0,"num_object_copies":13599,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4533,"num_objects_unfound":0,"num_objects_dirty":4533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4533,"num_write_kb":290112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1536,"num_bytes_recovered":100663296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,6],"avail_no_missing":["3","2","6"],"object_location_counts":[{"shards":"2,3,6","objects":4533}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":147,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.122768+0000","last_change":"2024-09-21T12:20:17.693550+0000","last_active":"2024-09-21T12:22:26.122768+0000","last_peered":"2024-09-21T12:22:26.122768+0000","last_clean":"2024-09-21T12:22:26.122768+0000","last_became_active":"2024-09-21T12:20:17.693179+0000","last_became_peered":"2024-09-21T12:20:17.693179+0000","last_unstale":"2024-09-21T12:22:26.122768+0000","last_undegraded":"2024-09-21T12:22:26.122768+0000","last_fullsized":"2024-09-21T12:22:26.122768+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":127,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.124510+0000","last_change":"2024-09-21T12:21:18.056305+0000","last_active":"2024-09-21T12:22:26.124510+0000","last_peered":"2024-09-21T12:22:26.124510+0000","last_clean":"2024-09-21T12:22:26.124510+0000","last_became_active":"2024-09-21T12:20:15.297340+0000","last_became_peered":"2024-09-21T12:20:15.297340+0000","last_unstale":"2024-09-21T12:22:26.124510+0000","last_undegraded":"2024-09-21T12:22:26.124510+0000","last_fullsized":"2024-09-21T12:22:26.124510+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:18.056261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:18.056261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:31.683295+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.789485+0000","last_change":"2024-09-21T12:22:25.141903+0000","last_active":"2024-09-21T12:22:26.789485+0000","last_peered":"2024-09-21T12:22:26.789485+0000","last_clean":"2024-09-21T12:22:26.789485+0000","last_became_active":"2024-09-21T12:22:25.139695+0000","last_became_peered":"2024-09-21T12:22:25.139695+0000","last_unstale":"2024-09-21T12:22:26.789485+0000","last_undegraded":"2024-09-21T12:22:26.789485+0000","last_fullsized":"2024-09-21T12:22:26.789485+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:01.971812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:01.971812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:18.541280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.789553+0000","last_change":"2024-09-21T12:22:25.131238+0000","last_active":"2024-09-21T12:22:26.789553+0000","last_peered":"2024-09-21T12:22:26.789553+0000","last_clean":"2024-09-21T12:22:26.789553+0000","last_became_active":"2024-09-21T12:22:25.129554+0000","last_became_peered":"2024-09-21T12:22:25.129554+0000","last_unstale":"2024-09-21T12:22:26.789553+0000","last_undegraded":"2024-09-21T12:22:26.789553+0000","last_fullsized":"2024-09-21T12:22:26.789553+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:13.050383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:13.050383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:20.963999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"76'79","reported_seq":25,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.122405+0000","last_change":"2024-09-21T12:22:17.098015+0000","last_active":"2024-09-21T12:22:26.122405+0000","last_peered":"2024-09-21T12:22:26.122405+0000","last_clean":"2024-09-21T12:22:26.122405+0000","last_became_active":"2024-09-21T12:22:17.069042+0000","last_became_peered":"2024-09-21T12:22:17.069042+0000","last_unstale":"2024-09-21T12:22:26.122405+0000","last_undegraded":"2024-09-21T12:22:26.122405+0000","last_fullsized":"2024-09-21T12:22:26.122405+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T12:20:24.850157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:20:24.850157+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:23:21.985894+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":381350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"113'4682","reported_seq":4751,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.713430+0000","last_change":"2024-09-21T12:21:24.889433+0000","last_active":"2024-09-21T12:22:26.713430+0000","last_peered":"2024-09-21T12:22:26.713430+0000","last_clean":"2024-09-21T12:22:26.713430+0000","last_became_active":"2024-09-21T12:21:24.889133+0000","last_became_peered":"2024-09-21T12:21:24.889133+0000","last_unstale":"2024-09-21T12:22:26.713430+0000","last_undegraded":"2024-09-21T12:22:26.713430+0000","last_fullsized":"2024-09-21T12:22:26.713430+0000","mapping_epoch":84,"log_start":"89'725","ondisk_log_start":"89'725","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":3957,"log_dups_size":0,"ondisk_log_size":3957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:48.105823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306839552,"num_objects":4682,"num_object_clones":0,"num_object_copies":14046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4682,"num_write_kb":299648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":32,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.789397+0000","last_change":"2024-09-21T12:22:25.172157+0000","last_active":"2024-09-21T12:22:26.789397+0000","last_peered":"2024-09-21T12:22:26.789397+0000","last_clean":"2024-09-21T12:22:26.789397+0000","last_became_active":"2024-09-21T12:22:25.131932+0000","last_became_peered":"2024-09-21T12:22:25.131932+0000","last_unstale":"2024-09-21T12:22:26.789397+0000","last_undegraded":"2024-09-21T12:22:26.789397+0000","last_fullsized":"2024-09-21T12:22:26.789397+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-21T12:22:14.062464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.062464+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:39.348288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":152,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.131800+0000","last_change":"2024-09-21T12:22:25.154069+0000","last_active":"2024-09-21T12:22:26.131800+0000","last_peered":"2024-09-21T12:22:26.131800+0000","last_clean":"2024-09-21T12:22:26.131800+0000","last_became_active":"2024-09-21T12:22:25.133731+0000","last_became_peered":"2024-09-21T12:22:25.133731+0000","last_unstale":"2024-09-21T12:22:26.131800+0000","last_undegraded":"2024-09-21T12:22:26.131800+0000","last_fullsized":"2024-09-21T12:22:26.131800+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:22:07.582969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:07.582969+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:07.642254+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"112'10","reported_seq":168,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.122256+0000","last_change":"2024-09-21T12:22:25.157711+0000","last_active":"2024-09-21T12:22:26.122256+0000","last_peered":"2024-09-21T12:22:26.122256+0000","last_clean":"2024-09-21T12:22:26.122256+0000","last_became_active":"2024-09-21T12:22:25.136379+0000","last_became_peered":"2024-09-21T12:22:25.136379+0000","last_unstale":"2024-09-21T12:22:26.122256+0000","last_undegraded":"2024-09-21T12:22:26.122256+0000","last_fullsized":"2024-09-21T12:22:26.122256+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:26.026625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":118,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.803640+0000","last_change":"2024-09-21T12:21:21.835669+0000","last_active":"2024-09-21T12:22:26.803640+0000","last_peered":"2024-09-21T12:22:26.803640+0000","last_clean":"2024-09-21T12:22:26.803640+0000","last_became_active":"2024-09-21T12:20:21.593367+0000","last_became_peered":"2024-09-21T12:20:21.593367+0000","last_unstale":"2024-09-21T12:22:26.803640+0000","last_undegraded":"2024-09-21T12:22:26.803640+0000","last_fullsized":"2024-09-21T12:22:26.803640+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:21.835623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:21:21.835623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"76'79","reported_seq":112,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.795444+0000","last_change":"2024-09-21T12:22:17.093539+0000","last_active":"2024-09-21T12:22:26.795444+0000","last_peered":"2024-09-21T12:22:26.795444+0000","last_clean":"2024-09-21T12:22:26.795444+0000","last_became_active":"2024-09-21T12:22:17.093298+0000","last_became_peered":"2024-09-21T12:22:17.093298+0000","last_unstale":"2024-09-21T12:22:26.795444+0000","last_undegraded":"2024-09-21T12:22:26.795444+0000","last_fullsized":"2024-09-21T12:22:26.795444+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T12:20:24.850157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:20:24.850157+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:23:23.614841+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"113'4675","reported_seq":7963,"reported_epoch":113,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T12:22:26.583559+0000","last_change":"2024-09-21T12:22:26.237926+0000","last_active":"2024-09-21T12:22:26.583559+0000","last_peered":"2024-09-21T12:22:26.583559+0000","last_clean":"2024-09-21T12:22:24.105498+0000","last_became_active":"2024-09-21T12:22:26.228337+0000","last_became_peered":"2024-09-21T12:22:26.228337+0000","last_unstale":"2024-09-21T12:22:26.583559+0000","last_undegraded":"2024-09-21T12:22:26.583559+0000","last_fullsized":"2024-09-21T12:22:26.583559+0000","mapping_epoch":112,"log_start":"89'710","ondisk_log_start":"89'710","created":84,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":3965,"log_dups_size":0,"ondisk_log_size":3965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306380800,"num_objects":4675,"num_object_clones":0,"num_object_copies":14025,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4626,"num_objects_unfound":0,"num_objects_dirty":4675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4675,"num_write_kb":299200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1602,"num_bytes_recovered":104988672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[1,3,6],"avail_no_missing":["1","3","6"],"object_location_counts":[{"shards":"1,3,6","objects":4675}],"blocked_by":[],"up_primary":4,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":170,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.126664+0000","last_change":"2024-09-21T12:22:25.151084+0000","last_active":"2024-09-21T12:22:26.126664+0000","last_peered":"2024-09-21T12:22:26.126664+0000","last_clean":"2024-09-21T12:22:26.126664+0000","last_became_active":"2024-09-21T12:22:25.135698+0000","last_became_peered":"2024-09-21T12:22:25.135698+0000","last_unstale":"2024-09-21T12:22:26.126664+0000","last_undegraded":"2024-09-21T12:22:26.126664+0000","last_fullsized":"2024-09-21T12:22:26.126664+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":122,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.801656+0000","last_change":"2024-09-21T12:20:15.308877+0000","last_active":"2024-09-21T12:22:26.801656+0000","last_peered":"2024-09-21T12:22:26.801656+0000","last_clean":"2024-09-21T12:22:26.801656+0000","last_became_active":"2024-09-21T12:20:15.308518+0000","last_became_peered":"2024-09-21T12:20:15.308518+0000","last_unstale":"2024-09-21T12:22:26.801656+0000","last_undegraded":"2024-09-21T12:22:26.801656+0000","last_fullsized":"2024-09-21T12:22:26.801656+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:26.986669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":159,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.118075+0000","last_change":"2024-09-21T12:21:23.139205+0000","last_active":"2024-09-21T12:22:26.118075+0000","last_peered":"2024-09-21T12:22:26.118075+0000","last_clean":"2024-09-21T12:22:26.118075+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:22:26.118075+0000","last_undegraded":"2024-09-21T12:22:26.118075+0000","last_fullsized":"2024-09-21T12:22:26.118075+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:21:23.139124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.139124+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:48.708731+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":109,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.116073+0000","last_change":"2024-09-21T12:20:21.316127+0000","last_active":"2024-09-21T12:22:26.116073+0000","last_peered":"2024-09-21T12:22:26.116073+0000","last_clean":"2024-09-21T12:22:26.116073+0000","last_became_active":"2024-09-21T12:20:21.315801+0000","last_became_peered":"2024-09-21T12:20:21.315801+0000","last_unstale":"2024-09-21T12:22:26.116073+0000","last_undegraded":"2024-09-21T12:22:26.116073+0000","last_fullsized":"2024-09-21T12:22:26.116073+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:33.077380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"76'79","reported_seq":112,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.795486+0000","last_change":"2024-09-21T12:22:17.083917+0000","last_active":"2024-09-21T12:22:26.795486+0000","last_peered":"2024-09-21T12:22:26.795486+0000","last_clean":"2024-09-21T12:22:26.795486+0000","last_became_active":"2024-09-21T12:22:17.083629+0000","last_became_peered":"2024-09-21T12:22:17.083629+0000","last_unstale":"2024-09-21T12:22:26.795486+0000","last_undegraded":"2024-09-21T12:22:26.795486+0000","last_fullsized":"2024-09-21T12:22:26.795486+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T12:20:24.850157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:20:24.850157+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:23:22.611693+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"114'4838","reported_seq":4908,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:31.538017+0000","last_change":"2024-09-21T12:21:24.884531+0000","last_active":"2024-09-21T12:22:31.538017+0000","last_peered":"2024-09-21T12:22:31.538017+0000","last_clean":"2024-09-21T12:22:31.538017+0000","last_became_active":"2024-09-21T12:21:24.884183+0000","last_became_peered":"2024-09-21T12:21:24.884183+0000","last_unstale":"2024-09-21T12:22:31.538017+0000","last_undegraded":"2024-09-21T12:22:31.538017+0000","last_fullsized":"2024-09-21T12:22:31.538017+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4838,"log_dups_size":0,"ondisk_log_size":4838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317063168,"num_objects":4838,"num_object_clones":0,"num_object_copies":14514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4838,"num_write_kb":309632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":153,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.118158+0000","last_change":"2024-09-21T12:20:17.693585+0000","last_active":"2024-09-21T12:22:26.118158+0000","last_peered":"2024-09-21T12:22:26.118158+0000","last_clean":"2024-09-21T12:22:26.118158+0000","last_became_active":"2024-09-21T12:20:17.693067+0000","last_became_peered":"2024-09-21T12:20:17.693067+0000","last_unstale":"2024-09-21T12:22:26.118158+0000","last_undegraded":"2024-09-21T12:22:26.118158+0000","last_fullsized":"2024-09-21T12:22:26.118158+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:20.707330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":121,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.127744+0000","last_change":"2024-09-21T12:20:15.308741+0000","last_active":"2024-09-21T12:22:26.127744+0000","last_peered":"2024-09-21T12:22:26.127744+0000","last_clean":"2024-09-21T12:22:26.127744+0000","last_became_active":"2024-09-21T12:20:15.308441+0000","last_became_peered":"2024-09-21T12:20:15.308441+0000","last_unstale":"2024-09-21T12:22:26.127744+0000","last_undegraded":"2024-09-21T12:22:26.127744+0000","last_fullsized":"2024-09-21T12:22:26.127744+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":114,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.795147+0000","last_change":"2024-09-21T12:20:19.309945+0000","last_active":"2024-09-21T12:22:26.795147+0000","last_peered":"2024-09-21T12:22:26.795147+0000","last_clean":"2024-09-21T12:22:26.795147+0000","last_became_active":"2024-09-21T12:20:19.309598+0000","last_became_peered":"2024-09-21T12:20:19.309598+0000","last_unstale":"2024-09-21T12:22:26.795147+0000","last_undegraded":"2024-09-21T12:22:26.795147+0000","last_fullsized":"2024-09-21T12:22:26.795147+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:32.718869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":109,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.136001+0000","last_change":"2024-09-21T12:20:21.593173+0000","last_active":"2024-09-21T12:22:26.136001+0000","last_peered":"2024-09-21T12:22:26.136001+0000","last_clean":"2024-09-21T12:22:26.136001+0000","last_became_active":"2024-09-21T12:20:21.592828+0000","last_became_peered":"2024-09-21T12:20:21.592828+0000","last_unstale":"2024-09-21T12:22:26.136001+0000","last_undegraded":"2024-09-21T12:22:26.136001+0000","last_fullsized":"2024-09-21T12:22:26.136001+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"76'79","reported_seq":22,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.800762+0000","last_change":"2024-09-21T12:22:17.085155+0000","last_active":"2024-09-21T12:22:26.800762+0000","last_peered":"2024-09-21T12:22:26.800762+0000","last_clean":"2024-09-21T12:22:26.800762+0000","last_became_active":"2024-09-21T12:22:17.084732+0000","last_became_peered":"2024-09-21T12:22:17.084732+0000","last_unstale":"2024-09-21T12:22:26.800762+0000","last_undegraded":"2024-09-21T12:22:26.800762+0000","last_fullsized":"2024-09-21T12:22:26.800762+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T12:20:24.850157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:20:24.850157+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:23:20.982807+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"114'5023","reported_seq":5093,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:31.542588+0000","last_change":"2024-09-21T12:21:24.887083+0000","last_active":"2024-09-21T12:22:31.542588+0000","last_peered":"2024-09-21T12:22:31.542588+0000","last_clean":"2024-09-21T12:22:31.542588+0000","last_became_active":"2024-09-21T12:21:24.886854+0000","last_became_peered":"2024-09-21T12:21:24.886854+0000","last_unstale":"2024-09-21T12:22:31.542588+0000","last_undegraded":"2024-09-21T12:22:31.542588+0000","last_fullsized":"2024-09-21T12:22:31.542588+0000","mapping_epoch":84,"log_start":"88'100","ondisk_log_start":"88'100","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4923,"log_dups_size":0,"ondisk_log_size":4923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:44.381461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329187328,"num_objects":5023,"num_object_clones":0,"num_object_copies":15069,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5023,"num_write_kb":321472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":192,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.127618+0000","last_change":"2024-09-21T12:20:17.308049+0000","last_active":"2024-09-21T12:22:26.127618+0000","last_peered":"2024-09-21T12:22:26.127618+0000","last_clean":"2024-09-21T12:22:26.127618+0000","last_became_active":"2024-09-21T12:20:17.307769+0000","last_became_peered":"2024-09-21T12:20:17.307769+0000","last_unstale":"2024-09-21T12:22:26.127618+0000","last_undegraded":"2024-09-21T12:22:26.127618+0000","last_fullsized":"2024-09-21T12:22:26.127618+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:35.762419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.790162+0000","last_change":"2024-09-21T12:22:25.136492+0000","last_active":"2024-09-21T12:22:26.790162+0000","last_peered":"2024-09-21T12:22:26.790162+0000","last_clean":"2024-09-21T12:22:26.790162+0000","last_became_active":"2024-09-21T12:22:25.135285+0000","last_became_peered":"2024-09-21T12:22:25.135285+0000","last_unstale":"2024-09-21T12:22:26.790162+0000","last_undegraded":"2024-09-21T12:22:26.790162+0000","last_fullsized":"2024-09-21T12:22:26.790162+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:05.507899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:05.507899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:24.798782+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":140,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.801333+0000","last_change":"2024-09-21T12:21:20.045824+0000","last_active":"2024-09-21T12:22:26.801333+0000","last_peered":"2024-09-21T12:22:26.801333+0000","last_clean":"2024-09-21T12:22:26.801333+0000","last_became_active":"2024-09-21T12:20:14.287998+0000","last_became_peered":"2024-09-21T12:20:14.287998+0000","last_unstale":"2024-09-21T12:22:26.801333+0000","last_undegraded":"2024-09-21T12:22:26.801333+0000","last_fullsized":"2024-09-21T12:22:26.801333+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T12:21:20.045755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.045755+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:31.010568+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011011149999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":134,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.121068+0000","last_change":"2024-09-21T12:22:25.134934+0000","last_active":"2024-09-21T12:22:26.121068+0000","last_peered":"2024-09-21T12:22:26.121068+0000","last_clean":"2024-09-21T12:22:26.121068+0000","last_became_active":"2024-09-21T12:22:25.133710+0000","last_became_peered":"2024-09-21T12:22:25.133710+0000","last_unstale":"2024-09-21T12:22:26.121068+0000","last_undegraded":"2024-09-21T12:22:26.121068+0000","last_fullsized":"2024-09-21T12:22:26.121068+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:06.563104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:06.563104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:29.971323+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":120,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.123898+0000","last_change":"2024-09-21T12:22:25.129213+0000","last_active":"2024-09-21T12:22:26.123898+0000","last_peered":"2024-09-21T12:22:26.123898+0000","last_clean":"2024-09-21T12:22:26.123898+0000","last_became_active":"2024-09-21T12:22:25.128864+0000","last_became_peered":"2024-09-21T12:22:25.128864+0000","last_unstale":"2024-09-21T12:22:26.123898+0000","last_undegraded":"2024-09-21T12:22:26.123898+0000","last_fullsized":"2024-09-21T12:22:26.123898+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'79","reported_seq":190,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.803194+0000","last_change":"2024-09-21T12:22:25.140854+0000","last_active":"2024-09-21T12:22:26.803194+0000","last_peered":"2024-09-21T12:22:26.803194+0000","last_clean":"2024-09-21T12:22:26.803194+0000","last_became_active":"2024-09-21T12:22:25.140473+0000","last_became_peered":"2024-09-21T12:22:25.140473+0000","last_unstale":"2024-09-21T12:22:26.803194+0000","last_undegraded":"2024-09-21T12:22:26.803194+0000","last_fullsized":"2024-09-21T12:22:26.803194+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T12:20:24.850157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:20:24.850157+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":21,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"113'4521","reported_seq":7556,"reported_epoch":113,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-21T12:22:26.688194+0000","last_change":"2024-09-21T12:22:26.219014+0000","last_active":"2024-09-21T12:22:26.688194+0000","last_peered":"2024-09-21T12:22:26.688194+0000","last_clean":"2024-09-21T12:22:24.115328+0000","last_became_active":"2024-09-21T12:22:26.207650+0000","last_became_peered":"2024-09-21T12:22:26.207650+0000","last_unstale":"2024-09-21T12:22:26.688194+0000","last_undegraded":"2024-09-21T12:22:26.688194+0000","last_fullsized":"2024-09-21T12:22:26.688194+0000","mapping_epoch":112,"log_start":"88'300","ondisk_log_start":"88'300","created":84,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4221,"log_dups_size":0,"ondisk_log_size":4221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296288256,"num_objects":4521,"num_object_clones":0,"num_object_copies":13563,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4521,"num_objects_unfound":0,"num_objects_dirty":4521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4521,"num_write_kb":289344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1474,"num_bytes_recovered":96600064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,1,6],"avail_no_missing":["0","1","6"],"object_location_counts":[{"shards":"0,1,6","objects":4521}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":144,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.125628+0000","last_change":"2024-09-21T12:20:17.297678+0000","last_active":"2024-09-21T12:22:26.125628+0000","last_peered":"2024-09-21T12:22:26.125628+0000","last_clean":"2024-09-21T12:22:26.125628+0000","last_became_active":"2024-09-21T12:20:17.297374+0000","last_became_peered":"2024-09-21T12:20:17.297374+0000","last_unstale":"2024-09-21T12:22:26.125628+0000","last_undegraded":"2024-09-21T12:22:26.125628+0000","last_fullsized":"2024-09-21T12:22:26.125628+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:22.813197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":121,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.126242+0000","last_change":"2024-09-21T12:20:15.308648+0000","last_active":"2024-09-21T12:22:26.126242+0000","last_peered":"2024-09-21T12:22:26.126242+0000","last_clean":"2024-09-21T12:22:26.126242+0000","last_became_active":"2024-09-21T12:20:15.308356+0000","last_became_peered":"2024-09-21T12:20:15.308356+0000","last_unstale":"2024-09-21T12:22:26.126242+0000","last_undegraded":"2024-09-21T12:22:26.126242+0000","last_fullsized":"2024-09-21T12:22:26.126242+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:27.637162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":138,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.116208+0000","last_change":"2024-09-21T12:21:24.390178+0000","last_active":"2024-09-21T12:22:26.116208+0000","last_peered":"2024-09-21T12:22:26.116208+0000","last_clean":"2024-09-21T12:22:26.116208+0000","last_became_active":"2024-09-21T12:20:14.288358+0000","last_became_peered":"2024-09-21T12:20:14.288358+0000","last_unstale":"2024-09-21T12:22:26.116208+0000","last_undegraded":"2024-09-21T12:22:26.116208+0000","last_fullsized":"2024-09-21T12:22:26.116208+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:21:24.390129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:21:24.390129+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:52.729420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019042449999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":112,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.116610+0000","last_change":"2024-09-21T12:20:19.310164+0000","last_active":"2024-09-21T12:22:26.116610+0000","last_peered":"2024-09-21T12:22:26.116610+0000","last_clean":"2024-09-21T12:22:26.116610+0000","last_became_active":"2024-09-21T12:20:19.309724+0000","last_became_peered":"2024-09-21T12:20:19.309724+0000","last_unstale":"2024-09-21T12:22:26.116610+0000","last_undegraded":"2024-09-21T12:22:26.116610+0000","last_fullsized":"2024-09-21T12:22:26.116610+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":110,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.804022+0000","last_change":"2024-09-21T12:20:21.593267+0000","last_active":"2024-09-21T12:22:26.804022+0000","last_peered":"2024-09-21T12:22:26.804022+0000","last_clean":"2024-09-21T12:22:26.804022+0000","last_became_active":"2024-09-21T12:20:21.592982+0000","last_became_peered":"2024-09-21T12:20:21.592982+0000","last_unstale":"2024-09-21T12:22:26.804022+0000","last_undegraded":"2024-09-21T12:22:26.804022+0000","last_fullsized":"2024-09-21T12:22:26.804022+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"76'79","reported_seq":35,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.128607+0000","last_change":"2024-09-21T12:22:25.137721+0000","last_active":"2024-09-21T12:22:26.128607+0000","last_peered":"2024-09-21T12:22:26.128607+0000","last_clean":"2024-09-21T12:22:26.128607+0000","last_became_active":"2024-09-21T12:22:25.135738+0000","last_became_peered":"2024-09-21T12:22:25.135738+0000","last_unstale":"2024-09-21T12:22:26.128607+0000","last_undegraded":"2024-09-21T12:22:26.128607+0000","last_fullsized":"2024-09-21T12:22:26.128607+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:22:18.810791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:22:18.810791+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:39.764730+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"114'4752","reported_seq":4823,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:31.541140+0000","last_change":"2024-09-21T12:21:24.889134+0000","last_active":"2024-09-21T12:22:31.541140+0000","last_peered":"2024-09-21T12:22:31.541140+0000","last_clean":"2024-09-21T12:22:31.541140+0000","last_became_active":"2024-09-21T12:21:24.888897+0000","last_became_peered":"2024-09-21T12:21:24.888897+0000","last_unstale":"2024-09-21T12:22:31.541140+0000","last_undegraded":"2024-09-21T12:22:31.541140+0000","last_fullsized":"2024-09-21T12:22:31.541140+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4752,"log_dups_size":0,"ondisk_log_size":4752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:46.053995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311427072,"num_objects":4752,"num_object_clones":0,"num_object_copies":14256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4752,"num_write_kb":304128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":135,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.133496+0000","last_change":"2024-09-21T12:20:17.299646+0000","last_active":"2024-09-21T12:22:26.133496+0000","last_peered":"2024-09-21T12:22:26.133496+0000","last_clean":"2024-09-21T12:22:26.133496+0000","last_became_active":"2024-09-21T12:20:17.299325+0000","last_became_peered":"2024-09-21T12:20:17.299325+0000","last_unstale":"2024-09-21T12:22:26.133496+0000","last_undegraded":"2024-09-21T12:22:26.133496+0000","last_fullsized":"2024-09-21T12:22:26.133496+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":143,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.117378+0000","last_change":"2024-09-21T12:22:25.134309+0000","last_active":"2024-09-21T12:22:26.117378+0000","last_peered":"2024-09-21T12:22:26.117378+0000","last_clean":"2024-09-21T12:22:26.117378+0000","last_became_active":"2024-09-21T12:22:25.133800+0000","last_became_peered":"2024-09-21T12:22:25.133800+0000","last_unstale":"2024-09-21T12:22:26.117378+0000","last_undegraded":"2024-09-21T12:22:26.117378+0000","last_fullsized":"2024-09-21T12:22:26.117378+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:13.623395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:13.623395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:22.574375+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":272,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:30.192221+0000","last_change":"2024-09-21T12:21:22.834972+0000","last_active":"2024-09-21T12:22:30.192221+0000","last_peered":"2024-09-21T12:22:30.192221+0000","last_clean":"2024-09-21T12:22:30.192221+0000","last_became_active":"2024-09-21T12:20:14.287543+0000","last_became_peered":"2024-09-21T12:20:14.287543+0000","last_unstale":"2024-09-21T12:22:30.192221+0000","last_undegraded":"2024-09-21T12:22:30.192221+0000","last_fullsized":"2024-09-21T12:22:30.192221+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-21T12:21:22.834917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:21:22.834917+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":131,"num_read_kb":126,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":114,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.801715+0000","last_change":"2024-09-21T12:20:19.314574+0000","last_active":"2024-09-21T12:22:26.801715+0000","last_peered":"2024-09-21T12:22:26.801715+0000","last_clean":"2024-09-21T12:22:26.801715+0000","last_became_active":"2024-09-21T12:20:19.314217+0000","last_became_peered":"2024-09-21T12:20:19.314217+0000","last_unstale":"2024-09-21T12:22:26.801715+0000","last_undegraded":"2024-09-21T12:22:26.801715+0000","last_fullsized":"2024-09-21T12:22:26.801715+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":108,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.127013+0000","last_change":"2024-09-21T12:20:21.311501+0000","last_active":"2024-09-21T12:22:26.127013+0000","last_peered":"2024-09-21T12:22:26.127013+0000","last_clean":"2024-09-21T12:22:26.127013+0000","last_became_active":"2024-09-21T12:20:21.311188+0000","last_became_peered":"2024-09-21T12:20:21.311188+0000","last_unstale":"2024-09-21T12:22:26.127013+0000","last_undegraded":"2024-09-21T12:22:26.127013+0000","last_fullsized":"2024-09-21T12:22:26.127013+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:30.646996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"113'4568","reported_seq":4637,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.683781+0000","last_change":"2024-09-21T12:21:24.886290+0000","last_active":"2024-09-21T12:22:26.683781+0000","last_peered":"2024-09-21T12:22:26.683781+0000","last_clean":"2024-09-21T12:22:26.683781+0000","last_became_active":"2024-09-21T12:21:24.885986+0000","last_became_peered":"2024-09-21T12:21:24.885986+0000","last_unstale":"2024-09-21T12:22:26.683781+0000","last_undegraded":"2024-09-21T12:22:26.683781+0000","last_fullsized":"2024-09-21T12:22:26.683781+0000","mapping_epoch":84,"log_start":"88'300","ondisk_log_start":"88'300","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4268,"log_dups_size":0,"ondisk_log_size":4268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:42.327997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299368448,"num_objects":4568,"num_object_clones":0,"num_object_copies":13704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4568,"num_write_kb":292352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"112'18","reported_seq":218,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.795439+0000","last_change":"2024-09-21T12:22:25.158776+0000","last_active":"2024-09-21T12:22:26.795439+0000","last_peered":"2024-09-21T12:22:26.795439+0000","last_clean":"2024-09-21T12:22:26.795439+0000","last_became_active":"2024-09-21T12:22:25.137394+0000","last_became_peered":"2024-09-21T12:22:25.137394+0000","last_unstale":"2024-09-21T12:22:26.795439+0000","last_undegraded":"2024-09-21T12:22:26.795439+0000","last_fullsized":"2024-09-21T12:22:26.795439+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:28.549685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":133,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.122800+0000","last_change":"2024-09-21T12:21:17.107378+0000","last_active":"2024-09-21T12:22:26.122800+0000","last_peered":"2024-09-21T12:22:26.122800+0000","last_clean":"2024-09-21T12:22:26.122800+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:22:26.122800+0000","last_undegraded":"2024-09-21T12:22:26.122800+0000","last_fullsized":"2024-09-21T12:22:26.122800+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:21:17.107309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:17.107309+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:44.755672+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":185,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.804416+0000","last_change":"2024-09-21T12:22:16.936605+0000","last_active":"2024-09-21T12:22:26.804416+0000","last_peered":"2024-09-21T12:22:26.804416+0000","last_clean":"2024-09-21T12:22:26.804416+0000","last_became_active":"2024-09-21T12:22:15.049592+0000","last_became_peered":"2024-09-21T12:22:15.049592+0000","last_unstale":"2024-09-21T12:22:26.804416+0000","last_undegraded":"2024-09-21T12:22:26.804416+0000","last_fullsized":"2024-09-21T12:22:26.804416+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:22:16.936548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:22:16.936548+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:36.347732+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":114,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.803480+0000","last_change":"2024-09-21T12:20:19.310280+0000","last_active":"2024-09-21T12:22:26.803480+0000","last_peered":"2024-09-21T12:22:26.803480+0000","last_clean":"2024-09-21T12:22:26.803480+0000","last_became_active":"2024-09-21T12:20:19.309859+0000","last_became_peered":"2024-09-21T12:20:19.309859+0000","last_unstale":"2024-09-21T12:22:26.803480+0000","last_undegraded":"2024-09-21T12:22:26.803480+0000","last_fullsized":"2024-09-21T12:22:26.803480+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":124,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.801508+0000","last_change":"2024-09-21T12:22:25.138480+0000","last_active":"2024-09-21T12:22:26.801508+0000","last_peered":"2024-09-21T12:22:26.801508+0000","last_clean":"2024-09-21T12:22:26.801508+0000","last_became_active":"2024-09-21T12:22:25.138137+0000","last_became_peered":"2024-09-21T12:22:25.138137+0000","last_unstale":"2024-09-21T12:22:26.801508+0000","last_undegraded":"2024-09-21T12:22:26.801508+0000","last_fullsized":"2024-09-21T12:22:26.801508+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"114'4962","reported_seq":5032,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:31.551133+0000","last_change":"2024-09-21T12:21:24.888106+0000","last_active":"2024-09-21T12:22:31.551133+0000","last_peered":"2024-09-21T12:22:31.551133+0000","last_clean":"2024-09-21T12:22:31.551133+0000","last_became_active":"2024-09-21T12:21:24.887845+0000","last_became_peered":"2024-09-21T12:21:24.887845+0000","last_unstale":"2024-09-21T12:22:31.551133+0000","last_undegraded":"2024-09-21T12:22:31.551133+0000","last_fullsized":"2024-09-21T12:22:31.551133+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4962,"log_dups_size":0,"ondisk_log_size":4962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:35.680437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325189632,"num_objects":4962,"num_object_clones":0,"num_object_copies":14886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4962,"num_write_kb":317568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":28,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.790600+0000","last_change":"2024-09-21T12:22:25.170642+0000","last_active":"2024-09-21T12:22:26.790600+0000","last_peered":"2024-09-21T12:22:26.790600+0000","last_clean":"2024-09-21T12:22:26.790600+0000","last_became_active":"2024-09-21T12:22:25.133846+0000","last_became_peered":"2024-09-21T12:22:25.133846+0000","last_unstale":"2024-09-21T12:22:26.790600+0000","last_undegraded":"2024-09-21T12:22:26.790600+0000","last_fullsized":"2024-09-21T12:22:26.790600+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:21:19.327640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:21:19.327640+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:48.937488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.789147+0000","last_change":"2024-09-21T12:22:25.139631+0000","last_active":"2024-09-21T12:22:26.789147+0000","last_peered":"2024-09-21T12:22:26.789147+0000","last_clean":"2024-09-21T12:22:26.789147+0000","last_became_active":"2024-09-21T12:22:25.138035+0000","last_became_peered":"2024-09-21T12:22:25.138035+0000","last_unstale":"2024-09-21T12:22:26.789147+0000","last_undegraded":"2024-09-21T12:22:26.789147+0000","last_fullsized":"2024-09-21T12:22:26.789147+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:05.892153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:05.892153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:33.098727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":135,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.801385+0000","last_change":"2024-09-21T12:22:25.138889+0000","last_active":"2024-09-21T12:22:26.801385+0000","last_peered":"2024-09-21T12:22:26.801385+0000","last_clean":"2024-09-21T12:22:26.801385+0000","last_became_active":"2024-09-21T12:22:25.137740+0000","last_became_peered":"2024-09-21T12:22:25.137740+0000","last_unstale":"2024-09-21T12:22:26.801385+0000","last_undegraded":"2024-09-21T12:22:26.801385+0000","last_fullsized":"2024-09-21T12:22:26.801385+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:07.050620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:07.050620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:22.323908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":106,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.123191+0000","last_change":"2024-09-21T12:20:21.593145+0000","last_active":"2024-09-21T12:22:26.123191+0000","last_peered":"2024-09-21T12:22:26.123191+0000","last_clean":"2024-09-21T12:22:26.123191+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:22:26.123191+0000","last_undegraded":"2024-09-21T12:22:26.123191+0000","last_fullsized":"2024-09-21T12:22:26.123191+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"76'79","reported_seq":29,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.800722+0000","last_change":"2024-09-21T12:22:17.114985+0000","last_active":"2024-09-21T12:22:26.800722+0000","last_peered":"2024-09-21T12:22:26.800722+0000","last_clean":"2024-09-21T12:22:26.800722+0000","last_became_active":"2024-09-21T12:22:17.087245+0000","last_became_peered":"2024-09-21T12:22:17.087245+0000","last_unstale":"2024-09-21T12:22:26.800722+0000","last_undegraded":"2024-09-21T12:22:26.800722+0000","last_fullsized":"2024-09-21T12:22:26.800722+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T12:20:24.850157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:20:24.850157+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:23:22.031869+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":316358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"113'4614","reported_seq":4683,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.554864+0000","last_change":"2024-09-21T12:21:24.888960+0000","last_active":"2024-09-21T12:22:26.554864+0000","last_peered":"2024-09-21T12:22:26.554864+0000","last_clean":"2024-09-21T12:22:26.554864+0000","last_became_active":"2024-09-21T12:21:24.888670+0000","last_became_peered":"2024-09-21T12:21:24.888670+0000","last_unstale":"2024-09-21T12:22:26.554864+0000","last_undegraded":"2024-09-21T12:22:26.554864+0000","last_fullsized":"2024-09-21T12:22:26.554864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4614,"log_dups_size":0,"ondisk_log_size":4614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:42.169378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302383104,"num_objects":4614,"num_object_clones":0,"num_object_copies":13842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4614,"num_write_kb":295296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":34,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.790764+0000","last_change":"2024-09-21T12:22:25.167516+0000","last_active":"2024-09-21T12:22:26.790764+0000","last_peered":"2024-09-21T12:22:26.790764+0000","last_clean":"2024-09-21T12:22:26.790764+0000","last_became_active":"2024-09-21T12:22:25.132834+0000","last_became_peered":"2024-09-21T12:22:25.132834+0000","last_unstale":"2024-09-21T12:22:26.790764+0000","last_undegraded":"2024-09-21T12:22:26.790764+0000","last_fullsized":"2024-09-21T12:22:26.790764+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-21T12:22:01.738653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:01.738653+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:06.199017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":144,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.795421+0000","last_change":"2024-09-21T12:22:25.127342+0000","last_active":"2024-09-21T12:22:26.795421+0000","last_peered":"2024-09-21T12:22:26.795421+0000","last_clean":"2024-09-21T12:22:26.795421+0000","last_became_active":"2024-09-21T12:22:25.126299+0000","last_became_peered":"2024-09-21T12:22:25.126299+0000","last_unstale":"2024-09-21T12:22:26.795421+0000","last_undegraded":"2024-09-21T12:22:26.795421+0000","last_fullsized":"2024-09-21T12:22:26.795421+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:16.670500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:16.670500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:45.812828+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":126,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.127105+0000","last_change":"2024-09-21T12:22:25.135236+0000","last_active":"2024-09-21T12:22:26.127105+0000","last_peered":"2024-09-21T12:22:26.127105+0000","last_clean":"2024-09-21T12:22:26.127105+0000","last_became_active":"2024-09-21T12:22:25.134712+0000","last_became_peered":"2024-09-21T12:22:25.134712+0000","last_unstale":"2024-09-21T12:22:26.127105+0000","last_undegraded":"2024-09-21T12:22:26.127105+0000","last_fullsized":"2024-09-21T12:22:26.127105+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":122,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.116255+0000","last_change":"2024-09-21T12:22:25.126872+0000","last_active":"2024-09-21T12:22:26.116255+0000","last_peered":"2024-09-21T12:22:26.116255+0000","last_clean":"2024-09-21T12:22:26.116255+0000","last_became_active":"2024-09-21T12:22:25.126377+0000","last_became_peered":"2024-09-21T12:22:25.126377+0000","last_unstale":"2024-09-21T12:22:26.116255+0000","last_undegraded":"2024-09-21T12:22:26.116255+0000","last_fullsized":"2024-09-21T12:22:26.116255+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:25.773691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"76'79","reported_seq":183,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.803213+0000","last_change":"2024-09-21T12:22:17.073301+0000","last_active":"2024-09-21T12:22:26.803213+0000","last_peered":"2024-09-21T12:22:26.803213+0000","last_clean":"2024-09-21T12:22:26.803213+0000","last_became_active":"2024-09-21T12:22:17.072872+0000","last_became_peered":"2024-09-21T12:22:17.072872+0000","last_unstale":"2024-09-21T12:22:26.803213+0000","last_undegraded":"2024-09-21T12:22:26.803213+0000","last_fullsized":"2024-09-21T12:22:26.803213+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T12:20:24.850157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:20:24.850157+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:23:20.982847+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":21,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.f","version":"113'4717","reported_seq":4784,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.702327+0000","last_change":"2024-09-21T12:21:24.887418+0000","last_active":"2024-09-21T12:22:26.702327+0000","last_peered":"2024-09-21T12:22:26.702327+0000","last_clean":"2024-09-21T12:22:26.702327+0000","last_became_active":"2024-09-21T12:21:24.887000+0000","last_became_peered":"2024-09-21T12:21:24.887000+0000","last_unstale":"2024-09-21T12:22:26.702327+0000","last_undegraded":"2024-09-21T12:22:26.702327+0000","last_fullsized":"2024-09-21T12:22:26.702327+0000","mapping_epoch":84,"log_start":"89'500","ondisk_log_start":"89'500","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4217,"log_dups_size":0,"ondisk_log_size":4217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:29.571773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309133312,"num_objects":4717,"num_object_clones":0,"num_object_copies":14151,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4717,"num_write_kb":301888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":28,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.790979+0000","last_change":"2024-09-21T12:22:25.164024+0000","last_active":"2024-09-21T12:22:26.790979+0000","last_peered":"2024-09-21T12:22:26.790979+0000","last_clean":"2024-09-21T12:22:26.790979+0000","last_became_active":"2024-09-21T12:22:25.132917+0000","last_became_peered":"2024-09-21T12:22:25.132917+0000","last_unstale":"2024-09-21T12:22:26.790979+0000","last_undegraded":"2024-09-21T12:22:26.790979+0000","last_fullsized":"2024-09-21T12:22:26.790979+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":140,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.131288+0000","last_change":"2024-09-21T12:22:25.126340+0000","last_active":"2024-09-21T12:22:26.131288+0000","last_peered":"2024-09-21T12:22:26.131288+0000","last_clean":"2024-09-21T12:22:26.131288+0000","last_became_active":"2024-09-21T12:22:25.126002+0000","last_became_peered":"2024-09-21T12:22:25.126002+0000","last_unstale":"2024-09-21T12:22:26.131288+0000","last_undegraded":"2024-09-21T12:22:26.131288+0000","last_fullsized":"2024-09-21T12:22:26.131288+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:08.999015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:08.999015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:16.362381+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":147,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.117381+0000","last_change":"2024-09-21T12:20:19.308246+0000","last_active":"2024-09-21T12:22:26.117381+0000","last_peered":"2024-09-21T12:22:26.117381+0000","last_clean":"2024-09-21T12:22:26.117381+0000","last_became_active":"2024-09-21T12:20:19.307924+0000","last_became_peered":"2024-09-21T12:20:19.307924+0000","last_unstale":"2024-09-21T12:22:26.117381+0000","last_undegraded":"2024-09-21T12:22:26.117381+0000","last_fullsized":"2024-09-21T12:22:26.117381+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:40.633298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.790523+0000","last_change":"2024-09-21T12:22:25.125880+0000","last_active":"2024-09-21T12:22:26.790523+0000","last_peered":"2024-09-21T12:22:26.790523+0000","last_clean":"2024-09-21T12:22:26.790523+0000","last_became_active":"2024-09-21T12:22:25.125581+0000","last_became_peered":"2024-09-21T12:22:25.125581+0000","last_unstale":"2024-09-21T12:22:26.790523+0000","last_undegraded":"2024-09-21T12:22:26.790523+0000","last_fullsized":"2024-09-21T12:22:26.790523+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:14.809555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.809555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:24.311280+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"113'4683","reported_seq":7829,"reported_epoch":113,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-21T12:22:26.552088+0000","last_change":"2024-09-21T12:22:26.311133+0000","last_active":"2024-09-21T12:22:26.552088+0000","last_peered":"2024-09-21T12:22:26.552088+0000","last_clean":"2024-09-21T12:22:24.099457+0000","last_became_active":"2024-09-21T12:22:26.247763+0000","last_became_peered":"2024-09-21T12:22:26.247763+0000","last_unstale":"2024-09-21T12:22:26.552088+0000","last_undegraded":"2024-09-21T12:22:26.552088+0000","last_fullsized":"2024-09-21T12:22:26.115900+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4683,"log_dups_size":0,"ondisk_log_size":4683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306905088,"num_objects":4683,"num_object_clones":0,"num_object_copies":14049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4650,"num_objects_unfound":0,"num_objects_dirty":4683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4683,"num_write_kb":299712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1532,"num_bytes_recovered":100401152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7],"avail_no_missing":["2","3","7"],"object_location_counts":[{"shards":"2,3,7","objects":4650},{"shards":"2,7","objects":33}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":174,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.118274+0000","last_change":"2024-09-21T12:22:25.162846+0000","last_active":"2024-09-21T12:22:26.118274+0000","last_peered":"2024-09-21T12:22:26.118274+0000","last_clean":"2024-09-21T12:22:26.118274+0000","last_became_active":"2024-09-21T12:22:25.134045+0000","last_became_peered":"2024-09-21T12:22:25.134045+0000","last_unstale":"2024-09-21T12:22:26.118274+0000","last_undegraded":"2024-09-21T12:22:26.118274+0000","last_fullsized":"2024-09-21T12:22:26.118274+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:21:18.243868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:21:18.243868+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:43.413490+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":122,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.801311+0000","last_change":"2024-09-21T12:20:15.302587+0000","last_active":"2024-09-21T12:22:26.801311+0000","last_peered":"2024-09-21T12:22:26.801311+0000","last_clean":"2024-09-21T12:22:26.801311+0000","last_became_active":"2024-09-21T12:20:15.302262+0000","last_became_peered":"2024-09-21T12:20:15.302262+0000","last_unstale":"2024-09-21T12:22:26.801311+0000","last_undegraded":"2024-09-21T12:22:26.801311+0000","last_fullsized":"2024-09-21T12:22:26.801311+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:48.085958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":134,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.116791+0000","last_change":"2024-09-21T12:22:25.126971+0000","last_active":"2024-09-21T12:22:26.116791+0000","last_peered":"2024-09-21T12:22:26.116791+0000","last_clean":"2024-09-21T12:22:26.116791+0000","last_became_active":"2024-09-21T12:22:25.126079+0000","last_became_peered":"2024-09-21T12:22:25.126079+0000","last_unstale":"2024-09-21T12:22:26.116791+0000","last_undegraded":"2024-09-21T12:22:26.116791+0000","last_fullsized":"2024-09-21T12:22:26.116791+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:11.892850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:11.892850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:13.586482+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":108,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.117294+0000","last_change":"2024-09-21T12:20:21.311799+0000","last_active":"2024-09-21T12:22:26.117294+0000","last_peered":"2024-09-21T12:22:26.117294+0000","last_clean":"2024-09-21T12:22:26.117294+0000","last_became_active":"2024-09-21T12:20:21.311493+0000","last_became_peered":"2024-09-21T12:20:21.311493+0000","last_unstale":"2024-09-21T12:22:26.117294+0000","last_undegraded":"2024-09-21T12:22:26.117294+0000","last_fullsized":"2024-09-21T12:22:26.117294+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:23.727080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"113'4652","reported_seq":7993,"reported_epoch":113,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T12:22:26.695863+0000","last_change":"2024-09-21T12:22:26.239084+0000","last_active":"2024-09-21T12:22:26.695863+0000","last_peered":"2024-09-21T12:22:26.695863+0000","last_clean":"2024-09-21T12:22:24.114109+0000","last_became_active":"2024-09-21T12:22:26.228288+0000","last_became_peered":"2024-09-21T12:22:26.228288+0000","last_unstale":"2024-09-21T12:22:26.695863+0000","last_undegraded":"2024-09-21T12:22:26.695863+0000","last_fullsized":"2024-09-21T12:22:26.695863+0000","mapping_epoch":112,"log_start":"88'400","ondisk_log_start":"88'400","created":84,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4252,"log_dups_size":0,"ondisk_log_size":4252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304873472,"num_objects":4652,"num_object_clones":0,"num_object_copies":13956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4581,"num_objects_unfound":0,"num_objects_dirty":4652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4652,"num_write_kb":297728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1627,"num_bytes_recovered":106627072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,7],"avail_no_missing":["0","1","7"],"object_location_counts":[{"shards":"0,1,7","objects":4652}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":149,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.795258+0000","last_change":"2024-09-21T12:20:17.693677+0000","last_active":"2024-09-21T12:22:26.795258+0000","last_peered":"2024-09-21T12:22:26.795258+0000","last_clean":"2024-09-21T12:22:26.795258+0000","last_became_active":"2024-09-21T12:20:17.692867+0000","last_became_peered":"2024-09-21T12:20:17.692867+0000","last_unstale":"2024-09-21T12:22:26.795258+0000","last_undegraded":"2024-09-21T12:22:26.795258+0000","last_fullsized":"2024-09-21T12:22:26.795258+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:40.775254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":121,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.804156+0000","last_change":"2024-09-21T12:20:15.300854+0000","last_active":"2024-09-21T12:22:26.804156+0000","last_peered":"2024-09-21T12:22:26.804156+0000","last_clean":"2024-09-21T12:22:26.804156+0000","last_became_active":"2024-09-21T12:20:15.299333+0000","last_became_peered":"2024-09-21T12:20:15.299333+0000","last_unstale":"2024-09-21T12:22:26.804156+0000","last_undegraded":"2024-09-21T12:22:26.804156+0000","last_fullsized":"2024-09-21T12:22:26.804156+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":112,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.116477+0000","last_change":"2024-09-21T12:20:19.307316+0000","last_active":"2024-09-21T12:22:26.116477+0000","last_peered":"2024-09-21T12:22:26.116477+0000","last_clean":"2024-09-21T12:22:26.116477+0000","last_became_active":"2024-09-21T12:20:19.306952+0000","last_became_peered":"2024-09-21T12:20:19.306952+0000","last_unstale":"2024-09-21T12:22:26.116477+0000","last_undegraded":"2024-09-21T12:22:26.116477+0000","last_fullsized":"2024-09-21T12:22:26.116477+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":109,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.804374+0000","last_change":"2024-09-21T12:20:21.314891+0000","last_active":"2024-09-21T12:22:26.804374+0000","last_peered":"2024-09-21T12:22:26.804374+0000","last_clean":"2024-09-21T12:22:26.804374+0000","last_became_active":"2024-09-21T12:20:21.314545+0000","last_became_peered":"2024-09-21T12:20:21.314545+0000","last_unstale":"2024-09-21T12:22:26.804374+0000","last_undegraded":"2024-09-21T12:22:26.804374+0000","last_fullsized":"2024-09-21T12:22:26.804374+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"113'4644","reported_seq":4713,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.707778+0000","last_change":"2024-09-21T12:21:24.888714+0000","last_active":"2024-09-21T12:22:26.707778+0000","last_peered":"2024-09-21T12:22:26.707778+0000","last_clean":"2024-09-21T12:22:26.707778+0000","last_became_active":"2024-09-21T12:21:24.888445+0000","last_became_peered":"2024-09-21T12:21:24.888445+0000","last_unstale":"2024-09-21T12:22:26.707778+0000","last_undegraded":"2024-09-21T12:22:26.707778+0000","last_fullsized":"2024-09-21T12:22:26.707778+0000","mapping_epoch":84,"log_start":"89'629","ondisk_log_start":"89'629","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:53.110945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304349184,"num_objects":4644,"num_object_clones":0,"num_object_copies":13932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4644,"num_write_kb":297216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":28,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.789275+0000","last_change":"2024-09-21T12:22:25.150434+0000","last_active":"2024-09-21T12:22:26.789275+0000","last_peered":"2024-09-21T12:22:26.789275+0000","last_clean":"2024-09-21T12:22:26.789275+0000","last_became_active":"2024-09-21T12:22:25.124164+0000","last_became_peered":"2024-09-21T12:22:25.124164+0000","last_unstale":"2024-09-21T12:22:26.789275+0000","last_undegraded":"2024-09-21T12:22:26.789275+0000","last_fullsized":"2024-09-21T12:22:26.789275+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:22:09.782437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:09.782437+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:21.816564+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":143,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.804351+0000","last_change":"2024-09-21T12:22:25.143524+0000","last_active":"2024-09-21T12:22:26.804351+0000","last_peered":"2024-09-21T12:22:26.804351+0000","last_clean":"2024-09-21T12:22:26.804351+0000","last_became_active":"2024-09-21T12:22:25.143215+0000","last_became_peered":"2024-09-21T12:22:25.143215+0000","last_unstale":"2024-09-21T12:22:26.804351+0000","last_undegraded":"2024-09-21T12:22:26.804351+0000","last_fullsized":"2024-09-21T12:22:26.804351+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:20.865749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.865749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":120,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.116275+0000","last_change":"2024-09-21T12:21:23.393320+0000","last_active":"2024-09-21T12:22:26.116275+0000","last_peered":"2024-09-21T12:22:26.116275+0000","last_clean":"2024-09-21T12:22:26.116275+0000","last_became_active":"2024-09-21T12:20:19.312572+0000","last_became_peered":"2024-09-21T12:20:19.312572+0000","last_unstale":"2024-09-21T12:22:26.116275+0000","last_undegraded":"2024-09-21T12:22:26.116275+0000","last_fullsized":"2024-09-21T12:22:26.116275+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.393270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.393270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:28.198881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":106,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.131094+0000","last_change":"2024-09-21T12:20:21.317550+0000","last_active":"2024-09-21T12:22:26.131094+0000","last_peered":"2024-09-21T12:22:26.131094+0000","last_clean":"2024-09-21T12:22:26.131094+0000","last_became_active":"2024-09-21T12:20:21.317223+0000","last_became_peered":"2024-09-21T12:20:21.317223+0000","last_unstale":"2024-09-21T12:22:26.131094+0000","last_undegraded":"2024-09-21T12:22:26.131094+0000","last_fullsized":"2024-09-21T12:22:26.131094+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:34.865119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"113'4521","reported_seq":4590,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.570923+0000","last_change":"2024-09-21T12:21:24.888120+0000","last_active":"2024-09-21T12:22:26.570923+0000","last_peered":"2024-09-21T12:22:26.570923+0000","last_clean":"2024-09-21T12:22:26.570923+0000","last_became_active":"2024-09-21T12:21:24.887488+0000","last_became_peered":"2024-09-21T12:21:24.887488+0000","last_unstale":"2024-09-21T12:22:26.570923+0000","last_undegraded":"2024-09-21T12:22:26.570923+0000","last_fullsized":"2024-09-21T12:22:26.570923+0000","mapping_epoch":84,"log_start":"88'400","ondisk_log_start":"88'400","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4121,"log_dups_size":0,"ondisk_log_size":4121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:52.700127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296288256,"num_objects":4521,"num_object_clones":0,"num_object_copies":13563,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4521,"num_write_kb":289344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":151,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.800858+0000","last_change":"2024-09-21T12:22:00.134130+0000","last_active":"2024-09-21T12:22:26.800858+0000","last_peered":"2024-09-21T12:22:26.800858+0000","last_clean":"2024-09-21T12:22:26.800858+0000","last_became_active":"2024-09-21T12:20:17.692870+0000","last_became_peered":"2024-09-21T12:20:17.692870+0000","last_unstale":"2024-09-21T12:22:26.800858+0000","last_undegraded":"2024-09-21T12:22:26.800858+0000","last_fullsized":"2024-09-21T12:22:26.800858+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:22:00.134062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:00.134062+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:12.498238+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":167,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.804278+0000","last_change":"2024-09-21T12:22:25.156057+0000","last_active":"2024-09-21T12:22:26.804278+0000","last_peered":"2024-09-21T12:22:26.804278+0000","last_clean":"2024-09-21T12:22:26.804278+0000","last_became_active":"2024-09-21T12:22:25.141257+0000","last_became_peered":"2024-09-21T12:22:25.141257+0000","last_unstale":"2024-09-21T12:22:26.804278+0000","last_undegraded":"2024-09-21T12:22:26.804278+0000","last_fullsized":"2024-09-21T12:22:26.804278+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-21T12:22:14.920932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.920932+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:31.036837+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"112'10","reported_seq":172,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:30.376653+0000","last_change":"2024-09-21T12:22:25.156691+0000","last_active":"2024-09-21T12:22:30.376653+0000","last_peered":"2024-09-21T12:22:30.376653+0000","last_clean":"2024-09-21T12:22:30.376653+0000","last_became_active":"2024-09-21T12:22:25.143340+0000","last_became_peered":"2024-09-21T12:22:25.143340+0000","last_unstale":"2024-09-21T12:22:30.376653+0000","last_undegraded":"2024-09-21T12:22:30.376653+0000","last_fullsized":"2024-09-21T12:22:30.376653+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":109,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.801034+0000","last_change":"2024-09-21T12:20:21.592893+0000","last_active":"2024-09-21T12:22:26.801034+0000","last_peered":"2024-09-21T12:22:26.801034+0000","last_clean":"2024-09-21T12:22:26.801034+0000","last_became_active":"2024-09-21T12:20:21.592576+0000","last_became_peered":"2024-09-21T12:20:21.592576+0000","last_unstale":"2024-09-21T12:22:26.801034+0000","last_undegraded":"2024-09-21T12:22:26.801034+0000","last_fullsized":"2024-09-21T12:22:26.801034+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":129,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.794933+0000","last_change":"2024-09-21T12:21:15.764309+0000","last_active":"2024-09-21T12:22:26.794933+0000","last_peered":"2024-09-21T12:22:26.794933+0000","last_clean":"2024-09-21T12:22:26.794933+0000","last_became_active":"2024-09-21T12:20:15.303480+0000","last_became_peered":"2024-09-21T12:20:15.303480+0000","last_unstale":"2024-09-21T12:22:26.794933+0000","last_undegraded":"2024-09-21T12:22:26.794933+0000","last_fullsized":"2024-09-21T12:22:26.794933+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:15.764254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:15.764254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:41.145874+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":123,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.124340+0000","last_change":"2024-09-21T12:20:17.694977+0000","last_active":"2024-09-21T12:22:26.124340+0000","last_peered":"2024-09-21T12:22:26.124340+0000","last_clean":"2024-09-21T12:22:26.124340+0000","last_became_active":"2024-09-21T12:20:17.694628+0000","last_became_peered":"2024-09-21T12:20:17.694628+0000","last_unstale":"2024-09-21T12:22:26.124340+0000","last_undegraded":"2024-09-21T12:22:26.124340+0000","last_fullsized":"2024-09-21T12:22:26.124340+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":121,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.801431+0000","last_change":"2024-09-21T12:22:06.061142+0000","last_active":"2024-09-21T12:22:26.801431+0000","last_peered":"2024-09-21T12:22:26.801431+0000","last_clean":"2024-09-21T12:22:26.801431+0000","last_became_active":"2024-09-21T12:20:19.313882+0000","last_became_peered":"2024-09-21T12:20:19.313882+0000","last_unstale":"2024-09-21T12:22:26.801431+0000","last_undegraded":"2024-09-21T12:22:26.801431+0000","last_fullsized":"2024-09-21T12:22:26.801431+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:06.061092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:06.061092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:11.282607+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":123,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.803855+0000","last_change":"2024-09-21T12:22:25.140291+0000","last_active":"2024-09-21T12:22:26.803855+0000","last_peered":"2024-09-21T12:22:26.803855+0000","last_clean":"2024-09-21T12:22:26.803855+0000","last_became_active":"2024-09-21T12:22:25.139393+0000","last_became_peered":"2024-09-21T12:22:25.139393+0000","last_unstale":"2024-09-21T12:22:26.803855+0000","last_undegraded":"2024-09-21T12:22:26.803855+0000","last_fullsized":"2024-09-21T12:22:26.803855+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":140,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.130905+0000","last_change":"2024-09-21T12:21:24.093710+0000","last_active":"2024-09-21T12:22:26.130905+0000","last_peered":"2024-09-21T12:22:26.130905+0000","last_clean":"2024-09-21T12:22:26.130905+0000","last_became_active":"2024-09-21T12:20:17.695898+0000","last_became_peered":"2024-09-21T12:20:17.695898+0000","last_unstale":"2024-09-21T12:22:26.130905+0000","last_undegraded":"2024-09-21T12:22:26.130905+0000","last_fullsized":"2024-09-21T12:22:26.130905+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:21:24.093656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:21:24.093656+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:32.120147+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":143,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.803397+0000","last_change":"2024-09-21T12:22:25.144271+0000","last_active":"2024-09-21T12:22:26.803397+0000","last_peered":"2024-09-21T12:22:26.803397+0000","last_clean":"2024-09-21T12:22:26.803397+0000","last_became_active":"2024-09-21T12:22:25.143910+0000","last_became_peered":"2024-09-21T12:22:25.143910+0000","last_unstale":"2024-09-21T12:22:26.803397+0000","last_undegraded":"2024-09-21T12:22:26.803397+0000","last_fullsized":"2024-09-21T12:22:26.803397+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:07.084008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:07.084008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:16.731053+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":110,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.124138+0000","last_change":"2024-09-21T12:20:19.312248+0000","last_active":"2024-09-21T12:22:26.124138+0000","last_peered":"2024-09-21T12:22:26.124138+0000","last_clean":"2024-09-21T12:22:26.124138+0000","last_became_active":"2024-09-21T12:20:19.311947+0000","last_became_peered":"2024-09-21T12:20:19.311947+0000","last_unstale":"2024-09-21T12:22:26.124138+0000","last_undegraded":"2024-09-21T12:22:26.124138+0000","last_fullsized":"2024-09-21T12:22:26.124138+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":130,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.116676+0000","last_change":"2024-09-21T12:22:25.127384+0000","last_active":"2024-09-21T12:22:26.116676+0000","last_peered":"2024-09-21T12:22:26.116676+0000","last_clean":"2024-09-21T12:22:26.116676+0000","last_became_active":"2024-09-21T12:22:25.127041+0000","last_became_peered":"2024-09-21T12:22:25.127041+0000","last_unstale":"2024-09-21T12:22:26.116676+0000","last_undegraded":"2024-09-21T12:22:26.116676+0000","last_fullsized":"2024-09-21T12:22:26.116676+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:18.797578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:18.797578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:38.329948+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":133,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.801493+0000","last_change":"2024-09-21T12:20:17.694904+0000","last_active":"2024-09-21T12:22:26.801493+0000","last_peered":"2024-09-21T12:22:26.801493+0000","last_clean":"2024-09-21T12:22:26.801493+0000","last_became_active":"2024-09-21T12:20:17.694568+0000","last_became_peered":"2024-09-21T12:20:17.694568+0000","last_unstale":"2024-09-21T12:22:26.801493+0000","last_undegraded":"2024-09-21T12:22:26.801493+0000","last_fullsized":"2024-09-21T12:22:26.801493+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:30.021433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":120,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.117730+0000","last_change":"2024-09-21T12:20:15.297853+0000","last_active":"2024-09-21T12:22:26.117730+0000","last_peered":"2024-09-21T12:22:26.117730+0000","last_clean":"2024-09-21T12:22:26.117730+0000","last_became_active":"2024-09-21T12:20:15.297519+0000","last_became_peered":"2024-09-21T12:20:15.297519+0000","last_unstale":"2024-09-21T12:22:26.117730+0000","last_undegraded":"2024-09-21T12:22:26.117730+0000","last_fullsized":"2024-09-21T12:22:26.117730+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:36.133863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":145,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.123328+0000","last_change":"2024-09-21T12:20:19.305689+0000","last_active":"2024-09-21T12:22:26.123328+0000","last_peered":"2024-09-21T12:22:26.123328+0000","last_clean":"2024-09-21T12:22:26.123328+0000","last_became_active":"2024-09-21T12:20:19.305364+0000","last_became_peered":"2024-09-21T12:20:19.305364+0000","last_unstale":"2024-09-21T12:22:26.123328+0000","last_undegraded":"2024-09-21T12:22:26.123328+0000","last_fullsized":"2024-09-21T12:22:26.123328+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.791117+0000","last_change":"2024-09-21T12:22:25.131553+0000","last_active":"2024-09-21T12:22:26.791117+0000","last_peered":"2024-09-21T12:22:26.791117+0000","last_clean":"2024-09-21T12:22:26.791117+0000","last_became_active":"2024-09-21T12:22:25.129468+0000","last_became_peered":"2024-09-21T12:22:25.129468+0000","last_unstale":"2024-09-21T12:22:26.791117+0000","last_undegraded":"2024-09-21T12:22:26.791117+0000","last_fullsized":"2024-09-21T12:22:26.791117+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":129,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.123610+0000","last_change":"2024-09-21T12:20:17.693562+0000","last_active":"2024-09-21T12:22:26.123610+0000","last_peered":"2024-09-21T12:22:26.123610+0000","last_clean":"2024-09-21T12:22:26.123610+0000","last_became_active":"2024-09-21T12:20:17.693209+0000","last_became_peered":"2024-09-21T12:20:17.693209+0000","last_unstale":"2024-09-21T12:22:26.123610+0000","last_undegraded":"2024-09-21T12:22:26.123610+0000","last_fullsized":"2024-09-21T12:22:26.123610+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:37.813752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":143,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.803737+0000","last_change":"2024-09-21T12:22:25.141416+0000","last_active":"2024-09-21T12:22:26.803737+0000","last_peered":"2024-09-21T12:22:26.803737+0000","last_clean":"2024-09-21T12:22:26.803737+0000","last_became_active":"2024-09-21T12:22:25.141109+0000","last_became_peered":"2024-09-21T12:22:25.141109+0000","last_unstale":"2024-09-21T12:22:26.803737+0000","last_undegraded":"2024-09-21T12:22:26.803737+0000","last_fullsized":"2024-09-21T12:22:26.803737+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:15.938093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:15.938093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:31.140232+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":149,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.801330+0000","last_change":"2024-09-21T12:20:19.314884+0000","last_active":"2024-09-21T12:22:26.801330+0000","last_peered":"2024-09-21T12:22:26.801330+0000","last_clean":"2024-09-21T12:22:26.801330+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:22:26.801330+0000","last_undegraded":"2024-09-21T12:22:26.801330+0000","last_fullsized":"2024-09-21T12:22:26.801330+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:46.062119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":108,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.121293+0000","last_change":"2024-09-21T12:20:21.309936+0000","last_active":"2024-09-21T12:22:26.121293+0000","last_peered":"2024-09-21T12:22:26.121293+0000","last_clean":"2024-09-21T12:22:26.121293+0000","last_became_active":"2024-09-21T12:20:21.309385+0000","last_became_peered":"2024-09-21T12:20:21.309385+0000","last_unstale":"2024-09-21T12:22:26.121293+0000","last_undegraded":"2024-09-21T12:22:26.121293+0000","last_fullsized":"2024-09-21T12:22:26.121293+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":28,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.790192+0000","last_change":"2024-09-21T12:22:25.159316+0000","last_active":"2024-09-21T12:22:26.790192+0000","last_peered":"2024-09-21T12:22:26.790192+0000","last_clean":"2024-09-21T12:22:26.790192+0000","last_became_active":"2024-09-21T12:22:25.132640+0000","last_became_peered":"2024-09-21T12:22:25.132640+0000","last_unstale":"2024-09-21T12:22:26.790192+0000","last_undegraded":"2024-09-21T12:22:26.790192+0000","last_fullsized":"2024-09-21T12:22:26.790192+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.790902+0000","last_change":"2024-09-21T12:22:25.137274+0000","last_active":"2024-09-21T12:22:26.790902+0000","last_peered":"2024-09-21T12:22:26.790902+0000","last_clean":"2024-09-21T12:22:26.790902+0000","last_became_active":"2024-09-21T12:22:25.136954+0000","last_became_peered":"2024-09-21T12:22:25.136954+0000","last_unstale":"2024-09-21T12:22:26.790902+0000","last_undegraded":"2024-09-21T12:22:26.790902+0000","last_fullsized":"2024-09-21T12:22:26.790902+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":112,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.126505+0000","last_change":"2024-09-21T12:20:19.315714+0000","last_active":"2024-09-21T12:22:26.126505+0000","last_peered":"2024-09-21T12:22:26.126505+0000","last_clean":"2024-09-21T12:22:26.126505+0000","last_became_active":"2024-09-21T12:20:19.315418+0000","last_became_peered":"2024-09-21T12:20:19.315418+0000","last_unstale":"2024-09-21T12:22:26.126505+0000","last_undegraded":"2024-09-21T12:22:26.126505+0000","last_fullsized":"2024-09-21T12:22:26.126505+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":106,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.122470+0000","last_change":"2024-09-21T12:20:21.310344+0000","last_active":"2024-09-21T12:22:26.122470+0000","last_peered":"2024-09-21T12:22:26.122470+0000","last_clean":"2024-09-21T12:22:26.122470+0000","last_became_active":"2024-09-21T12:20:21.310053+0000","last_became_peered":"2024-09-21T12:20:21.310053+0000","last_unstale":"2024-09-21T12:22:26.122470+0000","last_undegraded":"2024-09-21T12:22:26.122470+0000","last_fullsized":"2024-09-21T12:22:26.122470+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":132,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.126750+0000","last_change":"2024-09-21T12:20:17.299367+0000","last_active":"2024-09-21T12:22:26.126750+0000","last_peered":"2024-09-21T12:22:26.126750+0000","last_clean":"2024-09-21T12:22:26.126750+0000","last_became_active":"2024-09-21T12:20:17.299015+0000","last_became_peered":"2024-09-21T12:20:17.299015+0000","last_unstale":"2024-09-21T12:22:26.126750+0000","last_undegraded":"2024-09-21T12:22:26.126750+0000","last_fullsized":"2024-09-21T12:22:26.126750+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":143,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.804077+0000","last_change":"2024-09-21T12:22:25.141534+0000","last_active":"2024-09-21T12:22:26.804077+0000","last_peered":"2024-09-21T12:22:26.804077+0000","last_clean":"2024-09-21T12:22:26.804077+0000","last_became_active":"2024-09-21T12:22:25.140748+0000","last_became_peered":"2024-09-21T12:22:25.140748+0000","last_unstale":"2024-09-21T12:22:26.804077+0000","last_undegraded":"2024-09-21T12:22:26.804077+0000","last_fullsized":"2024-09-21T12:22:26.804077+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:05.067280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:05.067280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:05.852388+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":118,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.131168+0000","last_change":"2024-09-21T12:21:20.931077+0000","last_active":"2024-09-21T12:22:26.131168+0000","last_peered":"2024-09-21T12:22:26.131168+0000","last_clean":"2024-09-21T12:22:26.131168+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:22:26.131168+0000","last_undegraded":"2024-09-21T12:22:26.131168+0000","last_fullsized":"2024-09-21T12:22:26.131168+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:20.931025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.931025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:50.767841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":108,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.115654+0000","last_change":"2024-09-21T12:20:21.308003+0000","last_active":"2024-09-21T12:22:26.115654+0000","last_peered":"2024-09-21T12:22:26.115654+0000","last_clean":"2024-09-21T12:22:26.115654+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:22:26.115654+0000","last_undegraded":"2024-09-21T12:22:26.115654+0000","last_fullsized":"2024-09-21T12:22:26.115654+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:24.738749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":133,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.122750+0000","last_change":"2024-09-21T12:20:17.695358+0000","last_active":"2024-09-21T12:22:26.122750+0000","last_peered":"2024-09-21T12:22:26.122750+0000","last_clean":"2024-09-21T12:22:26.122750+0000","last_became_active":"2024-09-21T12:20:17.694794+0000","last_became_peered":"2024-09-21T12:20:17.694794+0000","last_unstale":"2024-09-21T12:22:26.122750+0000","last_undegraded":"2024-09-21T12:22:26.122750+0000","last_fullsized":"2024-09-21T12:22:26.122750+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:32.861223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":129,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.801505+0000","last_change":"2024-09-21T12:21:55.172399+0000","last_active":"2024-09-21T12:22:26.801505+0000","last_peered":"2024-09-21T12:22:26.801505+0000","last_clean":"2024-09-21T12:22:26.801505+0000","last_became_active":"2024-09-21T12:20:15.295968+0000","last_became_peered":"2024-09-21T12:20:15.295968+0000","last_unstale":"2024-09-21T12:22:26.801505+0000","last_undegraded":"2024-09-21T12:22:26.801505+0000","last_fullsized":"2024-09-21T12:22:26.801505+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:55.172344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:55.172344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:22.388105+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":127,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.803257+0000","last_change":"2024-09-21T12:22:25.141061+0000","last_active":"2024-09-21T12:22:26.803257+0000","last_peered":"2024-09-21T12:22:26.803257+0000","last_clean":"2024-09-21T12:22:26.803257+0000","last_became_active":"2024-09-21T12:22:25.139938+0000","last_became_peered":"2024-09-21T12:22:25.139938+0000","last_unstale":"2024-09-21T12:22:26.803257+0000","last_undegraded":"2024-09-21T12:22:26.803257+0000","last_fullsized":"2024-09-21T12:22:26.803257+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":106,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.131287+0000","last_change":"2024-09-21T12:20:21.595660+0000","last_active":"2024-09-21T12:22:26.131287+0000","last_peered":"2024-09-21T12:22:26.131287+0000","last_clean":"2024-09-21T12:22:26.131287+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:22:26.131287+0000","last_undegraded":"2024-09-21T12:22:26.131287+0000","last_fullsized":"2024-09-21T12:22:26.131287+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:27.917656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":120,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.123244+0000","last_change":"2024-09-21T12:20:17.304075+0000","last_active":"2024-09-21T12:22:26.123244+0000","last_peered":"2024-09-21T12:22:26.123244+0000","last_clean":"2024-09-21T12:22:26.123244+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:22:26.123244+0000","last_undegraded":"2024-09-21T12:22:26.123244+0000","last_fullsized":"2024-09-21T12:22:26.123244+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":134,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.116963+0000","last_change":"2024-09-21T12:22:14.635397+0000","last_active":"2024-09-21T12:22:26.116963+0000","last_peered":"2024-09-21T12:22:26.116963+0000","last_clean":"2024-09-21T12:22:26.116963+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:22:26.116963+0000","last_undegraded":"2024-09-21T12:22:26.116963+0000","last_fullsized":"2024-09-21T12:22:26.116963+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:22:14.635191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.635191+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:18.354894+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":135,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.794892+0000","last_change":"2024-09-21T12:22:25.128726+0000","last_active":"2024-09-21T12:22:26.794892+0000","last_peered":"2024-09-21T12:22:26.794892+0000","last_clean":"2024-09-21T12:22:26.794892+0000","last_became_active":"2024-09-21T12:22:25.128394+0000","last_became_peered":"2024-09-21T12:22:25.128394+0000","last_unstale":"2024-09-21T12:22:26.794892+0000","last_undegraded":"2024-09-21T12:22:26.794892+0000","last_fullsized":"2024-09-21T12:22:26.794892+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:09.649109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:09.649109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:37.578912+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":123,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.803292+0000","last_change":"2024-09-21T12:22:25.140405+0000","last_active":"2024-09-21T12:22:26.803292+0000","last_peered":"2024-09-21T12:22:26.803292+0000","last_clean":"2024-09-21T12:22:26.803292+0000","last_became_active":"2024-09-21T12:22:25.139475+0000","last_became_peered":"2024-09-21T12:22:25.139475+0000","last_unstale":"2024-09-21T12:22:26.803292+0000","last_undegraded":"2024-09-21T12:22:26.803292+0000","last_fullsized":"2024-09-21T12:22:26.803292+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":122,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.127866+0000","last_change":"2024-09-21T12:22:25.136271+0000","last_active":"2024-09-21T12:22:26.127866+0000","last_peered":"2024-09-21T12:22:26.127866+0000","last_clean":"2024-09-21T12:22:26.127866+0000","last_became_active":"2024-09-21T12:22:25.134105+0000","last_became_peered":"2024-09-21T12:22:25.134105+0000","last_unstale":"2024-09-21T12:22:26.127866+0000","last_undegraded":"2024-09-21T12:22:26.127866+0000","last_fullsized":"2024-09-21T12:22:26.127866+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":118,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.122908+0000","last_change":"2024-09-21T12:20:15.298322+0000","last_active":"2024-09-21T12:22:26.122908+0000","last_peered":"2024-09-21T12:22:26.122908+0000","last_clean":"2024-09-21T12:22:26.122908+0000","last_became_active":"2024-09-21T12:20:15.297993+0000","last_became_peered":"2024-09-21T12:20:15.297993+0000","last_unstale":"2024-09-21T12:22:26.122908+0000","last_undegraded":"2024-09-21T12:22:26.122908+0000","last_fullsized":"2024-09-21T12:22:26.122908+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":137,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.795716+0000","last_change":"2024-09-21T12:20:17.299134+0000","last_active":"2024-09-21T12:22:26.795716+0000","last_peered":"2024-09-21T12:22:26.795716+0000","last_clean":"2024-09-21T12:22:26.795716+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:22:26.795716+0000","last_undegraded":"2024-09-21T12:22:26.795716+0000","last_fullsized":"2024-09-21T12:22:26.795716+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:22:31.737370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":113,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-21T12:22:26.121561+0000","last_change":"2024-09-21T12:20:19.308586+0000","last_active":"2024-09-21T12:22:26.121561+0000","last_peered":"2024-09-21T12:22:26.121561+0000","last_clean":"2024-09-21T12:22:26.121561+0000","last_became_active":"2024-09-21T12:20:19.308247+0000","last_became_peered":"2024-09-21T12:20:19.308247+0000","last_unstale":"2024-09-21T12:22:26.121561+0000","last_undegraded":"2024-09-21T12:22:26.121561+0000","last_fullsized":"2024-09-21T12:22:26.121561+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":4932894720,"num_objects":75270,"num_object_clones":0,"num_object_copies":225810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22911,"num_objects_unfound":0,"num_objects_dirty":75270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75270,"num_write_kb":4817280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7771,"num_bytes_recovered":509280256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14968225792,"data_stored":14968225792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70606,"ondisk_log_size":70606,"up":48,"acting":47,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":690,"num_read_kb":447,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":2288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":131,"num_read_kb":126,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":375690,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":142,"num_write":72,"num_write_kb":1036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":966058,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234015,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1938712,"kb_used_data":1871432,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91806440,"statfs":{"total":95995035648,"available":94009794560,"internally_reserved":0,"allocated":1916346368,"data_stored":1915705884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462951,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2274248,"kb_used_data":2202808,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91470904,"statfs":{"total":95995035648,"available":93666205696,"internally_reserved":0,"allocated":2255675392,"data_stored":2255050643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724586,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1300184,"kb_used_data":1242632,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":92444968,"statfs":{"total":95995035648,"available":94663647232,"internally_reserved":0,"allocated":1272455168,"data_stored":1271815944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953522,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":183584,"kb_used_data":116304,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93561568,"statfs":{"total":95995035648,"available":95807045632,"internally_reserved":0,"allocated":119095296,"data_stored":118446140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149751,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3033188,"kb_used_data":2947852,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":90711964,"statfs":{"total":95995035648,"available":92889051136,"internally_reserved":0,"allocated":3018600448,"data_stored":3017941321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":87357895},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1820656,"kb_used_data":1757532,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":91924496,"statfs":{"total":95995035648,"available":94130683904,"internally_reserved":0,"allocated":1799712768,"data_stored":1799100621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2126900,"kb_used_data":2051296,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91618252,"statfs":{"total":95995035648,"available":93817090048,"internally_reserved":0,"allocated":2100527104,"data_stored":2099846431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2414992,"kb_used_data":2347708,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91330160,"statfs":{"total":95995035648,"available":93522083840,"internally_reserved":0,"allocated":2404052992,"data_stored":2403409185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2420506624,"data_stored":2420506624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2110128128,"data_stored":2110128128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1808138240,"data_stored":1808138240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3038969856,"data_stored":3038969856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":128581632,"data_stored":128581632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1277624320,"data_stored":1277624320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2254700544,"data_stored":2254700544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1929576448,"data_stored":1929576448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:22:39.129 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-09-21T12:22:39.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:39.130 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:39.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:39.207 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:39 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:22:39.232 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:39.259 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:39.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:39.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:39.332 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:22:39.360 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:39 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:22:39.360 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:39 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:22:39.360 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:39 smithi183 ceph-mon[33277]: osdmap e117: 8 total, 8 up, 8 in 2024-09-21T12:22:39.360 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:39 smithi183 ceph-mon[33277]: 7.d scrub starts 2024-09-21T12:22:39.361 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:39.419 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:39.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:39.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:39.521 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:39.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:39 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:22:39.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:39 smithi114 ceph-mon[30324]: osdmap e117: 8 total, 8 up, 8 in 2024-09-21T12:22:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:39 smithi114 ceph-mon[30324]: 7.d scrub starts 2024-09-21T12:22:39.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:39 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:22:39.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:39 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:22:39.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:39 smithi114 ceph-mon[40867]: osdmap e117: 8 total, 8 up, 8 in 2024-09-21T12:22:39.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:39 smithi114 ceph-mon[40867]: 7.d scrub starts 2024-09-21T12:22:39.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:39.570 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:39.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:39.637 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 69 16 85963 85947 77.8297 44.3125 0.0179271 0.0128426 2024-09-21T12:22:39.671 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:39.702 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:39.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:39.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:39.782 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:22:39.782 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-21T12:22:39.782 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-21T12:22:39.783 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-21T12:22:39.783 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:22:39.804 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:39.837 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:39.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:39.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:39.939 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:39.972 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:39.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:39.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:40.074 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:40.103 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:40.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:40.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:40.205 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:40.229 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:40.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:40.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:40.331 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:40.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:40.366 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:40.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:40.403 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:40 smithi183 ceph-mon[33277]: 4.15 scrub starts 2024-09-21T12:22:40.403 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:40 smithi183 ceph-mon[33277]: 4.15 scrub ok 2024-09-21T12:22:40.403 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:40 smithi183 ceph-mon[33277]: pgmap v102: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 48 MiB/s wr, 769 op/s; 22337/250524 objects misplaced (8.916%); 15 MiB/s, 240 objects/s recovering 2024-09-21T12:22:40.403 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:40 smithi183 ceph-mon[33277]: 4.14 scrub starts 2024-09-21T12:22:40.403 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:40 smithi183 ceph-mon[33277]: 4.14 scrub ok 2024-09-21T12:22:40.451 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T12:22:40.466 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:40.487 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:40 smithi114 ceph-mon[40867]: 4.15 scrub starts 2024-09-21T12:22:40.487 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:40 smithi114 ceph-mon[40867]: 4.15 scrub ok 2024-09-21T12:22:40.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:40 smithi114 ceph-mon[40867]: pgmap v102: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 48 MiB/s wr, 769 op/s; 22337/250524 objects misplaced (8.916%); 15 MiB/s, 240 objects/s recovering 2024-09-21T12:22:40.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:40 smithi114 ceph-mon[40867]: 4.14 scrub starts 2024-09-21T12:22:40.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:40 smithi114 ceph-mon[40867]: 4.14 scrub ok 2024-09-21T12:22:40.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:40 smithi114 ceph-mon[30324]: 4.15 scrub starts 2024-09-21T12:22:40.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:40 smithi114 ceph-mon[30324]: 4.15 scrub ok 2024-09-21T12:22:40.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:40 smithi114 ceph-mon[30324]: pgmap v102: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 48 MiB/s wr, 769 op/s; 22337/250524 objects misplaced (8.916%); 15 MiB/s, 240 objects/s recovering 2024-09-21T12:22:40.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:40 smithi114 ceph-mon[30324]: 4.14 scrub starts 2024-09-21T12:22:40.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:40 smithi114 ceph-mon[30324]: 4.14 scrub ok 2024-09-21T12:22:40.526 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:40.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:40.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:40.628 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:40.637 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 70 16 86760 86744 77.4295 49.8125 0.0444792 0.0129082 2024-09-21T12:22:40.668 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:40.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:40.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:40.770 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:40.801 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:40.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:40.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:40.903 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:40.928 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:40.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:40.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:40.944 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T12:22:41.030 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:41.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:41.069 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:41.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:41.170 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:41.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:41.201 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:41.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:41.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:41 smithi114 ceph-mon[40867]: 7.d scrub ok 2024-09-21T12:22:41.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:41 smithi114 ceph-mon[40867]: 7.c scrub starts 2024-09-21T12:22:41.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:41 smithi114 ceph-mon[30324]: 7.d scrub ok 2024-09-21T12:22:41.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:41 smithi114 ceph-mon[30324]: 7.c scrub starts 2024-09-21T12:22:41.302 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:41.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:41.345 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:41.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:41.446 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:41.480 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:41.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:41.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:41.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:41 smithi183 ceph-mon[33277]: 7.d scrub ok 2024-09-21T12:22:41.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:41 smithi183 ceph-mon[33277]: 7.c scrub starts 2024-09-21T12:22:41.582 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:41.616 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:41.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:41.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:41.637 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 71 16 87602 87586 77.0797 52.625 0.0329646 0.0129669 2024-09-21T12:22:41.718 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:41.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:41.772 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:41.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:41.874 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:41.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:41.915 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:41.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:42.016 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:42.065 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:42.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:42.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:42.167 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:42.200 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:42.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:42.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:42.302 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:42.342 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:42.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:42.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:42.444 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:42.479 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:42.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:42.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:42.581 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:42.608 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:42.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:42.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:42.637 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 72 16 88496 88480 76.7852 55.875 0.00996229 0.0130177 2024-09-21T12:22:42.710 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:42.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:42.752 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:42.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:42.780 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:22:42.801 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:22:42 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:22:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:22:42.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:42 smithi114 ceph-mon[30324]: 6.7 scrub starts 2024-09-21T12:22:42.802 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:42 smithi183 ceph-mon[33277]: 6.7 scrub starts 2024-09-21T12:22:42.803 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:42 smithi183 ceph-mon[33277]: 6.7 scrub ok 2024-09-21T12:22:42.803 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:42 smithi183 ceph-mon[33277]: pgmap v103: 158 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 152 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 50 MiB/s wr, 800 op/s; 22337/254466 objects misplaced (8.778%); 12 MiB/s, 194 objects/s recovering 2024-09-21T12:22:42.853 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:42.898 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:42.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:42.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:43.000 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:43.028 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:43.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:43.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:43.068 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:42 smithi183 ceph-mon[33277]: osdmap e118: 8 total, 8 up, 8 in 2024-09-21T12:22:43.069 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:42 smithi183 ceph-mon[33277]: 3.1e scrub starts 2024-09-21T12:22:43.069 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:42 smithi183 ceph-mon[33277]: 3.1e scrub ok 2024-09-21T12:22:43.069 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:42 smithi183 ceph-mon[33277]: 7.c scrub ok 2024-09-21T12:22:43.130 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:43.166 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:43.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:43.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:43.268 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:43.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:42 smithi114 ceph-mon[40867]: 6.7 scrub starts 2024-09-21T12:22:43.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:42 smithi114 ceph-mon[40867]: 6.7 scrub ok 2024-09-21T12:22:43.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:42 smithi114 ceph-mon[40867]: pgmap v103: 158 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 152 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 50 MiB/s wr, 800 op/s; 22337/254466 objects misplaced (8.778%); 12 MiB/s, 194 objects/s recovering 2024-09-21T12:22:43.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:42 smithi114 ceph-mon[40867]: osdmap e118: 8 total, 8 up, 8 in 2024-09-21T12:22:43.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:42 smithi114 ceph-mon[40867]: 3.1e scrub starts 2024-09-21T12:22:43.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:42 smithi114 ceph-mon[40867]: 3.1e scrub ok 2024-09-21T12:22:43.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:42 smithi114 ceph-mon[40867]: 7.c scrub ok 2024-09-21T12:22:43.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:42 smithi114 ceph-mon[30324]: 6.7 scrub ok 2024-09-21T12:22:43.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:42 smithi114 ceph-mon[30324]: pgmap v103: 158 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 152 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 50 MiB/s wr, 800 op/s; 22337/254466 objects misplaced (8.778%); 12 MiB/s, 194 objects/s recovering 2024-09-21T12:22:43.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:42 smithi114 ceph-mon[30324]: osdmap e118: 8 total, 8 up, 8 in 2024-09-21T12:22:43.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:42 smithi114 ceph-mon[30324]: 3.1e scrub starts 2024-09-21T12:22:43.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:42 smithi114 ceph-mon[30324]: 3.1e scrub ok 2024-09-21T12:22:43.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:42 smithi114 ceph-mon[30324]: 7.c scrub ok 2024-09-21T12:22:43.294 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:43.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:43.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:43.396 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:43.426 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:43.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:43.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:43.528 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:43.588 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:43.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:43.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:43.615 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:22:43.615 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-21T12:22:43.615 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-21T12:22:43.615 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:22:43.615 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.616 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:23884 7.1 7:8f148689:::benchmark_data_smithi114_78432_object23883:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T12:22:43.616 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:39.695870+0000", 2024-09-21T12:22:43.616 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 63.916541961, 2024-09-21T12:22:43.616 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.039559285999999999, 2024-09-21T12:22:43.616 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:43.616 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:43.616 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:43.616 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:22:43.616 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:22:43.616 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:43.616 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 23884 2024-09-21T12:22:43.616 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.616 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:43.616 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.617 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:43.617 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695870+0000", 2024-09-21T12:22:43.617 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.617 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.617 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.617 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:43.617 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695870+0000", 2024-09-21T12:22:43.617 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.617 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.617 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.617 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:43.618 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695873+0000", 2024-09-21T12:22:43.618 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0090000000000002e-06 2024-09-21T12:22:43.618 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.618 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.618 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:43.618 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695927+0000", 2024-09-21T12:22:43.618 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4941000000000001e-05 2024-09-21T12:22:43.618 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.618 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.618 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:43.618 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695955+0000", 2024-09-21T12:22:43.618 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7120000000000001e-05 2024-09-21T12:22:43.618 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.618 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.619 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:43.619 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695963+0000", 2024-09-21T12:22:43.619 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7760000000000003e-06 2024-09-21T12:22:43.619 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.619 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.619 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:43.619 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696022+0000", 2024-09-21T12:22:43.619 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8941999999999999e-05 2024-09-21T12:22:43.619 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.619 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.619 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:43.619 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696107+0000", 2024-09-21T12:22:43.619 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4853999999999998e-05 2024-09-21T12:22:43.619 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.620 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.620 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,6", 2024-09-21T12:22:43.620 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696202+0000", 2024-09-21T12:22:43.620 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4889999999999994e-05 2024-09-21T12:22:43.620 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.620 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.620 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:22:43.620 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.698132+0000", 2024-09-21T12:22:43.620 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001929998 2024-09-21T12:22:43.620 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.620 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.620 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:43.620 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.698167+0000", 2024-09-21T12:22:43.620 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5247000000000003e-05 2024-09-21T12:22:43.620 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.621 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:43.621 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.735322+0000", 2024-09-21T12:22:43.621 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.037154514 2024-09-21T12:22:43.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.621 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:43.621 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.735382+0000", 2024-09-21T12:22:43.621 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0371000000000002e-05 2024-09-21T12:22:43.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.621 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:43.621 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.735430+0000", 2024-09-21T12:22:43.621 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7623999999999997e-05 2024-09-21T12:22:43.622 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.622 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:43.622 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.622 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.622 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.622 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:23885 7.1 7:81ccbda1:::benchmark_data_smithi114_78432_object23884:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T12:22:43.622 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:39.696116+0000", 2024-09-21T12:22:43.622 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 63.916296131999999, 2024-09-21T12:22:43.622 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.040219576, 2024-09-21T12:22:43.622 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:43.622 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:43.622 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:43.622 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:22:43.622 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:22:43.623 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:43.623 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 23885 2024-09-21T12:22:43.623 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.623 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:43.623 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.623 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:43.623 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696116+0000", 2024-09-21T12:22:43.623 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.623 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.623 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.623 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:43.623 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696116+0000", 2024-09-21T12:22:43.623 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.623 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.623 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.624 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:43.624 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696117+0000", 2024-09-21T12:22:43.624 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0109999999999999e-06 2024-09-21T12:22:43.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.624 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:43.624 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696178+0000", 2024-09-21T12:22:43.624 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1060000000000002e-05 2024-09-21T12:22:43.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.624 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:43.624 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696188+0000", 2024-09-21T12:22:43.624 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0067e-05 2024-09-21T12:22:43.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:43.625 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696196+0000", 2024-09-21T12:22:43.625 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0740000000000004e-06 2024-09-21T12:22:43.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:43.625 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.697126+0000", 2024-09-21T12:22:43.625 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00093067599999999996 2024-09-21T12:22:43.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:43.625 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.697270+0000", 2024-09-21T12:22:43.625 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000143502 2024-09-21T12:22:43.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.626 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,6", 2024-09-21T12:22:43.626 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.697418+0000", 2024-09-21T12:22:43.626 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000147892 2024-09-21T12:22:43.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.626 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:22:43.626 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.698242+0000", 2024-09-21T12:22:43.626 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00082411799999999999 2024-09-21T12:22:43.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.626 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:43.626 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.698485+0000", 2024-09-21T12:22:43.626 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00024341099999999999 2024-09-21T12:22:43.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.627 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.627 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:43.627 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.736250+0000", 2024-09-21T12:22:43.627 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.037765265999999999 2024-09-21T12:22:43.627 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.627 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.627 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:43.627 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.736300+0000", 2024-09-21T12:22:43.627 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9886000000000003e-05 2024-09-21T12:22:43.627 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.627 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.627 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:43.627 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.736336+0000", 2024-09-21T12:22:43.627 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5612999999999999e-05 2024-09-21T12:22:43.627 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.627 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:43.628 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.628 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.628 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.628 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:81802 7.8 e115/112 7:110c0c7a:::benchmark_data_smithi114_78432_object81801:head v 115'5198, mlcod=88'242)", 2024-09-21T12:22:43.628 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.137865+0000", 2024-09-21T12:22:43.628 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 8.4745472450000001, 2024-09-21T12:22:43.628 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.051160639000000001, 2024-09-21T12:22:43.628 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:43.628 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:43.628 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:43.628 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:43.628 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.628 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:43.629 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137865+0000", 2024-09-21T12:22:43.629 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.629 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.629 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:43.629 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137865+0000", 2024-09-21T12:22:43.629 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.630 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.630 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:43.630 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137866+0000", 2024-09-21T12:22:43.630 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5099999999999998e-07 2024-09-21T12:22:43.630 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.630 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.630 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:43.630 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137909+0000", 2024-09-21T12:22:43.630 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2545e-05 2024-09-21T12:22:43.630 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.630 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.630 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:43.630 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137938+0000", 2024-09-21T12:22:43.630 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9737999999999999e-05 2024-09-21T12:22:43.631 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.631 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.631 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:43.631 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137961+0000", 2024-09-21T12:22:43.631 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2320999999999999e-05 2024-09-21T12:22:43.631 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.631 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.631 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:43.631 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.138026+0000", 2024-09-21T12:22:43.631 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5571e-05 2024-09-21T12:22:43.631 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.631 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.631 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:43.631 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.138111+0000", 2024-09-21T12:22:43.631 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4682e-05 2024-09-21T12:22:43.632 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.632 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.632 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:43.632 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.188940+0000", 2024-09-21T12:22:43.632 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.050828867 2024-09-21T12:22:43.632 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.632 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.632 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:43.632 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189026+0000", 2024-09-21T12:22:43.632 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6063999999999995e-05 2024-09-21T12:22:43.632 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.632 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:43.632 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.632 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.632 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.633 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:81809 7.8 e115/112 7:1042b23e:::benchmark_data_smithi114_78432_object81808:head v 115'5199, mlcod=88'242)", 2024-09-21T12:22:43.633 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.140023+0000", 2024-09-21T12:22:43.633 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 8.4723896740000004, 2024-09-21T12:22:43.633 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.049109005999999997, 2024-09-21T12:22:43.633 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:43.633 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:43.633 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:43.633 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:43.633 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.633 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:43.633 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140023+0000", 2024-09-21T12:22:43.633 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.633 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.633 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.633 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:43.634 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140023+0000", 2024-09-21T12:22:43.634 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.634 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.634 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:43.634 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140023+0000", 2024-09-21T12:22:43.634 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6700000000000003e-07 2024-09-21T12:22:43.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.634 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.634 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:43.634 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140136+0000", 2024-09-21T12:22:43.634 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000112623 2024-09-21T12:22:43.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.634 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.634 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:43.635 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140146+0000", 2024-09-21T12:22:43.635 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0098999999999999e-05 2024-09-21T12:22:43.635 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.635 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.635 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:43.635 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140161+0000", 2024-09-21T12:22:43.635 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5288e-05 2024-09-21T12:22:43.635 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.635 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.635 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:43.635 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146087+0000", 2024-09-21T12:22:43.635 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0059259070000000002 2024-09-21T12:22:43.635 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.636 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:43.636 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146109+0000", 2024-09-21T12:22:43.636 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2045999999999999e-05 2024-09-21T12:22:43.636 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.636 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:43.636 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189077+0000", 2024-09-21T12:22:43.636 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.042967639000000002 2024-09-21T12:22:43.636 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.636 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:43.636 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189132+0000", 2024-09-21T12:22:43.636 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4737000000000001e-05 2024-09-21T12:22:43.636 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.637 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:43.637 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.637 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.637 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.637 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:81811 7.8 e115/112 7:10044f1a:::benchmark_data_smithi114_78432_object81810:head v 115'5200, mlcod=88'242)", 2024-09-21T12:22:43.637 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.146782+0000", 2024-09-21T12:22:43.637 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 8.4656301890000005, 2024-09-21T12:22:43.637 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.042406829, 2024-09-21T12:22:43.637 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:43.637 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:43.637 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:43.638 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:43.638 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.638 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:43.638 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146782+0000", 2024-09-21T12:22:43.638 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.638 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.638 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.638 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:43.638 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146782+0000", 2024-09-21T12:22:43.638 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.638 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.638 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.638 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:43.638 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146783+0000", 2024-09-21T12:22:43.638 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2999999999999999e-07 2024-09-21T12:22:43.639 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.639 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.639 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:43.639 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146826+0000", 2024-09-21T12:22:43.639 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2781000000000003e-05 2024-09-21T12:22:43.639 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.639 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.639 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:43.639 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146837+0000", 2024-09-21T12:22:43.639 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1126e-05 2024-09-21T12:22:43.639 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.639 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.639 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:43.639 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146843+0000", 2024-09-21T12:22:43.639 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4710000000000004e-06 2024-09-21T12:22:43.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.640 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:43.640 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.152740+0000", 2024-09-21T12:22:43.640 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0058961209999999998 2024-09-21T12:22:43.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.640 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:43.640 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.152763+0000", 2024-09-21T12:22:43.640 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3101000000000001e-05 2024-09-21T12:22:43.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.640 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:43.640 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189134+0000", 2024-09-21T12:22:43.641 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.036371542999999999 2024-09-21T12:22:43.641 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.641 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.641 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:43.641 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189189+0000", 2024-09-21T12:22:43.641 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4855999999999998e-05 2024-09-21T12:22:43.641 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.641 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:43.641 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.641 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.641 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.641 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:81821 7.8 e115/112 7:1c79dadc:::benchmark_data_smithi114_78432_object81820:head v 115'5201, mlcod=88'242)", 2024-09-21T12:22:43.641 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.148399+0000", 2024-09-21T12:22:43.641 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 8.4640131140000001, 2024-09-21T12:22:43.641 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.040843988999999997, 2024-09-21T12:22:43.642 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:43.642 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:43.642 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:43.642 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:43.642 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.642 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:43.642 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148399+0000", 2024-09-21T12:22:43.642 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.642 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.642 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.642 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:43.642 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148399+0000", 2024-09-21T12:22:43.642 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.642 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.643 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.643 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:43.643 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148400+0000", 2024-09-21T12:22:43.643 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.049e-06 2024-09-21T12:22:43.643 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.643 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.643 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:43.643 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148502+0000", 2024-09-21T12:22:43.643 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000101542 2024-09-21T12:22:43.643 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.643 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.643 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:43.644 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148512+0000", 2024-09-21T12:22:43.644 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0386e-05 2024-09-21T12:22:43.644 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.644 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.644 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:43.644 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148519+0000", 2024-09-21T12:22:43.644 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3589999999999996e-06 2024-09-21T12:22:43.644 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.644 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.644 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:43.644 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.158203+0000", 2024-09-21T12:22:43.644 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0096844710000000001 2024-09-21T12:22:43.644 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.644 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.644 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:43.645 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.158229+0000", 2024-09-21T12:22:43.645 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6078999999999999e-05 2024-09-21T12:22:43.645 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.645 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.645 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:43.645 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189192+0000", 2024-09-21T12:22:43.645 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.030962455 2024-09-21T12:22:43.645 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.645 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.645 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:43.645 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189243+0000", 2024-09-21T12:22:43.645 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1647999999999997e-05 2024-09-21T12:22:43.645 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.645 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:43.645 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.646 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.646 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.646 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82430 7.8 e115/112 7:1762e17c:::benchmark_data_smithi114_78432_object82429:head v 115'5247, mlcod=88'255)", 2024-09-21T12:22:43.646 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.787623+0000", 2024-09-21T12:22:43.646 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 7.8247893260000003, 2024-09-21T12:22:43.646 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.048511400000000003, 2024-09-21T12:22:43.646 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:43.646 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:43.646 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:43.646 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:43.646 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.646 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:43.646 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787623+0000", 2024-09-21T12:22:43.646 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:43.647 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787623+0000", 2024-09-21T12:22:43.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:43.647 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787625+0000", 2024-09-21T12:22:43.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5319999999999999e-06 2024-09-21T12:22:43.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:43.648 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787667+0000", 2024-09-21T12:22:43.648 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1876999999999999e-05 2024-09-21T12:22:43.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:43.648 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787688+0000", 2024-09-21T12:22:43.648 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1046000000000001e-05 2024-09-21T12:22:43.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:43.648 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787697+0000", 2024-09-21T12:22:43.648 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4930000000000004e-06 2024-09-21T12:22:43.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:43.649 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.788378+0000", 2024-09-21T12:22:43.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00068115999999999999 2024-09-21T12:22:43.649 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.649 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:43.649 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.788427+0000", 2024-09-21T12:22:43.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8553000000000001e-05 2024-09-21T12:22:43.649 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.649 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:43.649 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.836011+0000", 2024-09-21T12:22:43.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.047583978999999998 2024-09-21T12:22:43.649 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.649 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:43.650 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.836135+0000", 2024-09-21T12:22:43.650 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012375999999999999 2024-09-21T12:22:43.650 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.650 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:43.650 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.650 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.650 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.650 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDPGPush(7.8 115/112 [PushOp(7:10df046b:::benchmark_data_smithi114_78432_object3988:head, version: 88'257, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:10df046b:::benchmark_data_smithi114_78432_object3988:head@88'257, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T12:22:43.650 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.787722+0000", 2024-09-21T12:22:43.650 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 7.8246902509999998, 2024-09-21T12:22:43.650 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.074165650999999999, 2024-09-21T12:22:43.650 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:43.650 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:43.650 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:22:43.651 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:43.651 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.651 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:43.651 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787722+0000", 2024-09-21T12:22:43.651 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.651 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.651 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.651 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:43.651 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787722+0000", 2024-09-21T12:22:43.651 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.651 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.652 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.652 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:43.652 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787723+0000", 2024-09-21T12:22:43.652 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0499999999999999e-07 2024-09-21T12:22:43.652 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.652 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.652 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:43.652 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787779+0000", 2024-09-21T12:22:43.652 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6053e-05 2024-09-21T12:22:43.652 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.652 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.652 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:43.652 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787803+0000", 2024-09-21T12:22:43.652 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3980000000000001e-05 2024-09-21T12:22:43.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.653 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.653 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:43.653 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787832+0000", 2024-09-21T12:22:43.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.942e-05 2024-09-21T12:22:43.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.653 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.653 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:43.653 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.859697+0000", 2024-09-21T12:22:43.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.071864649000000003 2024-09-21T12:22:43.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.653 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.653 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:43.653 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.859710+0000", 2024-09-21T12:22:43.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3740000000000001e-05 2024-09-21T12:22:43.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.654 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:43.654 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.861888+0000", 2024-09-21T12:22:43.654 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002177404 2024-09-21T12:22:43.654 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.654 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:43.654 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.654 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82442 7.8 e115/112 7:1035456a:::benchmark_data_smithi114_78432_object82441:head v 115'5248, mlcod=88'255)", 2024-09-21T12:22:43.654 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.791117+0000", 2024-09-21T12:22:43.654 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 7.8212953479999996, 2024-09-21T12:22:43.654 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.051248759999999997, 2024-09-21T12:22:43.654 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:43.655 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:43.655 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:43.655 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:43.655 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.655 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:43.655 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791117+0000", 2024-09-21T12:22:43.655 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.655 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.655 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.655 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:43.655 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791117+0000", 2024-09-21T12:22:43.655 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.656 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.656 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:43.656 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791118+0000", 2024-09-21T12:22:43.656 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9999999999999998e-07 2024-09-21T12:22:43.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.656 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.656 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:43.656 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791158+0000", 2024-09-21T12:22:43.656 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0025999999999999e-05 2024-09-21T12:22:43.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.656 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.656 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:43.656 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791167+0000", 2024-09-21T12:22:43.657 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9910000000000008e-06 2024-09-21T12:22:43.657 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.657 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.657 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:43.657 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791176+0000", 2024-09-21T12:22:43.657 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0329999999999997e-06 2024-09-21T12:22:43.657 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.657 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.657 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:43.657 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.794825+0000", 2024-09-21T12:22:43.657 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0036494700000000001 2024-09-21T12:22:43.657 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.657 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.657 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:43.657 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.794878+0000", 2024-09-21T12:22:43.658 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3292999999999999e-05 2024-09-21T12:22:43.658 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.658 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.658 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:43.658 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842289+0000", 2024-09-21T12:22:43.658 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.047410256999999997 2024-09-21T12:22:43.658 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.658 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.658 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:43.658 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842366+0000", 2024-09-21T12:22:43.658 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7189999999999998e-05 2024-09-21T12:22:43.658 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.658 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:43.658 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.658 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.659 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.659 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82447 7.8 e115/112 7:174332ca:::benchmark_data_smithi114_78432_object82446:head v 115'5249, mlcod=88'255)", 2024-09-21T12:22:43.659 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.793874+0000", 2024-09-21T12:22:43.659 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 7.8185388969999998, 2024-09-21T12:22:43.659 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.048541015999999999, 2024-09-21T12:22:43.659 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:43.659 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:43.659 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:43.659 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:43.659 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.659 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:43.659 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793874+0000", 2024-09-21T12:22:43.659 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.659 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.659 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:43.660 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793874+0000", 2024-09-21T12:22:43.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:43.660 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793875+0000", 2024-09-21T12:22:43.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0890000000000001e-06 2024-09-21T12:22:43.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:43.660 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793910+0000", 2024-09-21T12:22:43.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5080000000000003e-05 2024-09-21T12:22:43.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.661 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:43.661 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793921+0000", 2024-09-21T12:22:43.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1284000000000001e-05 2024-09-21T12:22:43.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.661 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:43.661 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793930+0000", 2024-09-21T12:22:43.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5429999999999994e-06 2024-09-21T12:22:43.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.661 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:43.661 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.801012+0000", 2024-09-21T12:22:43.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0070825219999999999 2024-09-21T12:22:43.662 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.662 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:43.662 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.801039+0000", 2024-09-21T12:22:43.662 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7248999999999999e-05 2024-09-21T12:22:43.662 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.662 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:43.662 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842368+0000", 2024-09-21T12:22:43.662 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.041328831000000003 2024-09-21T12:22:43.662 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.662 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:43.662 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842415+0000", 2024-09-21T12:22:43.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6418000000000002e-05 2024-09-21T12:22:43.663 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.663 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:43.663 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.663 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.663 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82448 7.8 e115/112 7:11700dea:::benchmark_data_smithi114_78432_object82447:head v 115'5250, mlcod=88'255)", 2024-09-21T12:22:43.663 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.796531+0000", 2024-09-21T12:22:43.663 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 7.8158811530000003, 2024-09-21T12:22:43.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.045928499999999997, 2024-09-21T12:22:43.663 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:43.663 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:43.663 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:43.663 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:43.664 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.664 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:43.664 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796531+0000", 2024-09-21T12:22:43.664 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.664 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.664 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.664 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:43.664 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796531+0000", 2024-09-21T12:22:43.664 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.664 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.664 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.664 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:43.664 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796532+0000", 2024-09-21T12:22:43.664 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7199999999999999e-07 2024-09-21T12:22:43.664 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.664 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.665 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:43.665 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796647+0000", 2024-09-21T12:22:43.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000115097 2024-09-21T12:22:43.665 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.665 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.665 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:43.665 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796660+0000", 2024-09-21T12:22:43.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3271999999999999e-05 2024-09-21T12:22:43.665 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.665 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.665 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:43.665 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796666+0000", 2024-09-21T12:22:43.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0170000000000002e-06 2024-09-21T12:22:43.665 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.665 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:43.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.809350+0000", 2024-09-21T12:22:43.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.012683379 2024-09-21T12:22:43.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:43.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.809374+0000", 2024-09-21T12:22:43.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4417e-05 2024-09-21T12:22:43.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:43.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842417+0000", 2024-09-21T12:22:43.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.033042646000000002 2024-09-21T12:22:43.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:43.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842460+0000", 2024-09-21T12:22:43.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3099999999999997e-05 2024-09-21T12:22:43.667 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.667 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:43.667 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.667 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82473 7.8 e115/112 7:15016882:::benchmark_data_smithi114_78432_object82472:head v 115'5254, mlcod=88'255)", 2024-09-21T12:22:43.667 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.818965+0000", 2024-09-21T12:22:43.667 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 7.7934475599999997, 2024-09-21T12:22:43.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.045024287000000003, 2024-09-21T12:22:43.667 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:43.667 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:43.667 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:43.668 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:43.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:43.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.818965+0000", 2024-09-21T12:22:43.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:43.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.818965+0000", 2024-09-21T12:22:43.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:43.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.818966+0000", 2024-09-21T12:22:43.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9199999999999999e-07 2024-09-21T12:22:43.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:43.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.819001+0000", 2024-09-21T12:22:43.669 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 73 15 89465 89450 76.5638 60.625 0.0141546 0.0130559 2024-09-21T12:22:43.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5117999999999998e-05 2024-09-21T12:22:43.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:43.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.819014+0000", 2024-09-21T12:22:43.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2869e-05 2024-09-21T12:22:43.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:43.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.819020+0000", 2024-09-21T12:22:43.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6959999999999999e-06 2024-09-21T12:22:43.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:43.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.836164+0000", 2024-09-21T12:22:43.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.017143617 2024-09-21T12:22:43.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:43.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.836188+0000", 2024-09-21T12:22:43.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4074e-05 2024-09-21T12:22:43.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:43.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.863879+0000", 2024-09-21T12:22:43.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.027690392000000001 2024-09-21T12:22:43.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:43.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.863989+0000", 2024-09-21T12:22:43.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000110629 2024-09-21T12:22:43.672 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.672 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:43.672 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.672 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82539 7.8 e115/112 7:1cd4c60d:::benchmark_data_smithi114_78432_object82538:head v 115'5258, mlcod=88'256)", 2024-09-21T12:22:43.673 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.889845+0000", 2024-09-21T12:22:43.673 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 7.7225671570000003, 2024-09-21T12:22:43.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.046325071000000002, 2024-09-21T12:22:43.673 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:43.673 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:43.673 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:43.673 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:43.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:43.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889845+0000", 2024-09-21T12:22:43.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:43.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889845+0000", 2024-09-21T12:22:43.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:43.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889846+0000", 2024-09-21T12:22:43.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1800000000000004e-07 2024-09-21T12:22:43.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:43.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889878+0000", 2024-09-21T12:22:43.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1619999999999999e-05 2024-09-21T12:22:43.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:43.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889893+0000", 2024-09-21T12:22:43.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4819e-05 2024-09-21T12:22:43.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:43.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889900+0000", 2024-09-21T12:22:43.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8510000000000001e-06 2024-09-21T12:22:43.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:43.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.898101+0000", 2024-09-21T12:22:43.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0082018119999999993 2024-09-21T12:22:43.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:43.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.898164+0000", 2024-09-21T12:22:43.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2756999999999996e-05 2024-09-21T12:22:43.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:43.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.936000+0000", 2024-09-21T12:22:43.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.037835720000000003 2024-09-21T12:22:43.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:43.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.936170+0000", 2024-09-21T12:22:43.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000170574 2024-09-21T12:22:43.676 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.676 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:43.676 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.677 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82545 7.8 e115/112 7:1ddea6d1:::benchmark_data_smithi114_78432_object82544:head v 115'5259, mlcod=88'256)", 2024-09-21T12:22:43.677 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.897417+0000", 2024-09-21T12:22:43.677 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 7.7149956199999998, 2024-09-21T12:22:43.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.038827027, 2024-09-21T12:22:43.677 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:43.677 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:43.677 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:43.677 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:43.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:43.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897417+0000", 2024-09-21T12:22:43.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:43.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897417+0000", 2024-09-21T12:22:43.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:43.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897418+0000", 2024-09-21T12:22:43.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3500000000000005e-07 2024-09-21T12:22:43.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:43.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897462+0000", 2024-09-21T12:22:43.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4036999999999999e-05 2024-09-21T12:22:43.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:43.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897472+0000", 2024-09-21T12:22:43.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0669999999999999e-05 2024-09-21T12:22:43.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:43.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897479+0000", 2024-09-21T12:22:43.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9190000000000002e-06 2024-09-21T12:22:43.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:43.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.905524+0000", 2024-09-21T12:22:43.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0080444899999999996 2024-09-21T12:22:43.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:43.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.905597+0000", 2024-09-21T12:22:43.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3539000000000002e-05 2024-09-21T12:22:43.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:43.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.936176+0000", 2024-09-21T12:22:43.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.030578893999999999 2024-09-21T12:22:43.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:43.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.936244+0000", 2024-09-21T12:22:43.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7543e-05 2024-09-21T12:22:43.680 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.680 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:43.681 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.681 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:83346 7.8 e115/112 7:1517ab05:::benchmark_data_smithi114_78432_object83345:head v 115'5319, mlcod=88'282)", 2024-09-21T12:22:43.681 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:36.700133+0000", 2024-09-21T12:22:43.681 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.9122797189999998, 2024-09-21T12:22:43.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.044693733999999999, 2024-09-21T12:22:43.681 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:43.681 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:43.681 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:43.681 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:43.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:43.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700133+0000", 2024-09-21T12:22:43.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:43.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700133+0000", 2024-09-21T12:22:43.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:43.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700133+0000", 2024-09-21T12:22:43.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4300000000000003e-07 2024-09-21T12:22:43.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:43.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700182+0000", 2024-09-21T12:22:43.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8433000000000002e-05 2024-09-21T12:22:43.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:43.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700192+0000", 2024-09-21T12:22:43.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0382e-05 2024-09-21T12:22:43.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:43.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700201+0000", 2024-09-21T12:22:43.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9819999999999997e-06 2024-09-21T12:22:43.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:43.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.703063+0000", 2024-09-21T12:22:43.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002862121 2024-09-21T12:22:43.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:43.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.703087+0000", 2024-09-21T12:22:43.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3730000000000001e-05 2024-09-21T12:22:43.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:43.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.744677+0000", 2024-09-21T12:22:43.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.041589785999999997 2024-09-21T12:22:43.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:43.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.744826+0000", 2024-09-21T12:22:43.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014965700000000001 2024-09-21T12:22:43.685 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.685 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:43.685 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.685 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:87464 7.8 e117/112 7:17a0a89d:::benchmark_data_smithi114_78432_object87463:head v 117'5593, mlcod=88'417)", 2024-09-21T12:22:43.685 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:41.463661+0000", 2024-09-21T12:22:43.685 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.1487517860000001, 2024-09-21T12:22:43.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.067076070000000002, 2024-09-21T12:22:43.685 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:43.685 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:43.685 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:43.685 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:43.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:43.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463661+0000", 2024-09-21T12:22:43.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:43.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463661+0000", 2024-09-21T12:22:43.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:43.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463661+0000", 2024-09-21T12:22:43.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2799999999999995e-07 2024-09-21T12:22:43.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:43.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463845+0000", 2024-09-21T12:22:43.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000183692 2024-09-21T12:22:43.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:43.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463854+0000", 2024-09-21T12:22:43.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0750000000000004e-06 2024-09-21T12:22:43.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:43.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463860+0000", 2024-09-21T12:22:43.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.015e-06 2024-09-21T12:22:43.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:43.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.480218+0000", 2024-09-21T12:22:43.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.016357289000000001 2024-09-21T12:22:43.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:43.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.480239+0000", 2024-09-21T12:22:43.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1691999999999998e-05 2024-09-21T12:22:43.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:43.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.530695+0000", 2024-09-21T12:22:43.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.050456050000000002 2024-09-21T12:22:43.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:43.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.530737+0000", 2024-09-21T12:22:43.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1428999999999999e-05 2024-09-21T12:22:43.689 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.689 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:43.689 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.689 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:87470 7.8 e117/112 7:1f3b224f:::benchmark_data_smithi114_78432_object87469:head v 117'5594, mlcod=88'417)", 2024-09-21T12:22:43.689 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:41.472523+0000", 2024-09-21T12:22:43.689 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.1398897880000001, 2024-09-21T12:22:43.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.06343451, 2024-09-21T12:22:43.690 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:43.690 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:43.690 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:43.690 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:43.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:43.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472523+0000", 2024-09-21T12:22:43.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:43.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472523+0000", 2024-09-21T12:22:43.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.691 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:43.691 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472524+0000", 2024-09-21T12:22:43.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7399999999999991e-07 2024-09-21T12:22:43.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.691 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:43.691 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472596+0000", 2024-09-21T12:22:43.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2152999999999998e-05 2024-09-21T12:22:43.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.691 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:43.691 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472606+0000", 2024-09-21T12:22:43.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0485e-05 2024-09-21T12:22:43.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:43.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472613+0000", 2024-09-21T12:22:43.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5590000000000002e-06 2024-09-21T12:22:43.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:43.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.484029+0000", 2024-09-21T12:22:43.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.01141587 2024-09-21T12:22:43.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:43.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.484053+0000", 2024-09-21T12:22:43.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4559999999999999e-05 2024-09-21T12:22:43.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:43.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.535915+0000", 2024-09-21T12:22:43.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.051861431 2024-09-21T12:22:43.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:43.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.535957+0000", 2024-09-21T12:22:43.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2478000000000002e-05 2024-09-21T12:22:43.693 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.693 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:43.693 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.693 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:87486 7.8 e117/112 7:14e05f7b:::benchmark_data_smithi114_78432_object87485:head v 117'5595, mlcod=88'417)", 2024-09-21T12:22:43.694 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:41.475017+0000", 2024-09-21T12:22:43.694 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.1373954519999998, 2024-09-21T12:22:43.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.060970067000000003, 2024-09-21T12:22:43.694 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:43.694 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:43.694 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:43.694 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:43.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:43.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475017+0000", 2024-09-21T12:22:43.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:43.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475017+0000", 2024-09-21T12:22:43.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:43.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475018+0000", 2024-09-21T12:22:43.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5799999999999999e-07 2024-09-21T12:22:43.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:43.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475150+0000", 2024-09-21T12:22:43.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013201700000000001 2024-09-21T12:22:43.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:43.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475159+0000", 2024-09-21T12:22:43.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5840000000000001e-06 2024-09-21T12:22:43.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:43.696 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475166+0000", 2024-09-21T12:22:43.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3589999999999996e-06 2024-09-21T12:22:43.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:43.696 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.525196+0000", 2024-09-21T12:22:43.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.050030507000000002 2024-09-21T12:22:43.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:43.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.525214+0000", 2024-09-21T12:22:43.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7884e-05 2024-09-21T12:22:43.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:43.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.535960+0000", 2024-09-21T12:22:43.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.010746143999999999 2024-09-21T12:22:43.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:43.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.535987+0000", 2024-09-21T12:22:43.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6914e-05 2024-09-21T12:22:43.697 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.697 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:43.698 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.698 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDPGPush(7.8 117/112 [PushOp(7:1de1c2d4:::benchmark_data_smithi114_78432_object6336:head, version: 88'418, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:1de1c2d4:::benchmark_data_smithi114_78432_object6336:head@88'418, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T12:22:43.698 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:41.485571+0000", 2024-09-21T12:22:43.698 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.1268415570000001, 2024-09-21T12:22:43.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.046712901000000001, 2024-09-21T12:22:43.698 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:43.698 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:43.698 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:22:43.698 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:43.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:43.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485571+0000", 2024-09-21T12:22:43.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:43.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485571+0000", 2024-09-21T12:22:43.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:43.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485572+0000", 2024-09-21T12:22:43.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7299999999999995e-07 2024-09-21T12:22:43.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:43.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485643+0000", 2024-09-21T12:22:43.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0976999999999995e-05 2024-09-21T12:22:43.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:43.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485655+0000", 2024-09-21T12:22:43.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2904e-05 2024-09-21T12:22:43.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:43.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485665+0000", 2024-09-21T12:22:43.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6709999999999994e-06 2024-09-21T12:22:43.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:43.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.530794+0000", 2024-09-21T12:22:43.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.045129295 2024-09-21T12:22:43.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:43.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.530802+0000", 2024-09-21T12:22:43.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5059999999999998e-06 2024-09-21T12:22:43.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:43.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.532284+0000", 2024-09-21T12:22:43.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014818749999999999 2024-09-21T12:22:43.701 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.701 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:43.702 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.702 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:87497 7.8 e117/112 7:19475f57:::benchmark_data_smithi114_78432_object87496:head v 117'5596, mlcod=88'417)", 2024-09-21T12:22:43.702 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:41.494916+0000", 2024-09-21T12:22:43.702 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 2.1174960779999998, 2024-09-21T12:22:43.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.041122084000000003, 2024-09-21T12:22:43.702 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:43.702 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:43.702 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:43.702 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:43.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:43.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.494916+0000", 2024-09-21T12:22:43.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:43.703 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.494916+0000", 2024-09-21T12:22:43.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:43.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:43.703 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.494918+0000", 2024-09-21T12:22:43.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.719e-06 2024-09-21T12:22:43.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:43.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.494975+0000", 2024-09-21T12:22:43.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6981999999999998e-05 2024-09-21T12:22:43.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:43.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.525059+0000", 2024-09-21T12:22:43.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.030083526999999999 2024-09-21T12:22:43.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:43.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.525071+0000", 2024-09-21T12:22:43.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2438e-05 2024-09-21T12:22:43.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:43.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.532312+0000", 2024-09-21T12:22:43.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0072404909999999999 2024-09-21T12:22:43.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:43.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.532336+0000", 2024-09-21T12:22:43.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4090000000000001e-05 2024-09-21T12:22:43.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:43.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.536013+0000", 2024-09-21T12:22:43.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003677825 2024-09-21T12:22:43.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:43.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:43.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:43.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.536038+0000", 2024-09-21T12:22:43.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5012e-05 2024-09-21T12:22:43.706 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.706 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:43.706 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.706 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:43.706 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:43.706 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:22:43.706 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:43.727 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:43.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:43.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:43.829 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:43.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:43.871 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:43.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:43.972 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:43.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:43 smithi183 ceph-mon[33277]: 4.a scrub starts 2024-09-21T12:22:43.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:43 smithi183 ceph-mon[33277]: 4.a scrub ok 2024-09-21T12:22:44.003 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:44.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:44.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:44.105 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:44.130 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T12:22:44.130 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:22:44.130 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-21T12:22:44.130 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd out 4 2024-09-21T12:22:44.132 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:44.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:44.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:44.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:43 smithi114 ceph-mon[30324]: 4.a scrub starts 2024-09-21T12:22:44.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:43 smithi114 ceph-mon[30324]: 4.a scrub ok 2024-09-21T12:22:44.169 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:43 smithi114 ceph-mon[40867]: 4.a scrub starts 2024-09-21T12:22:44.169 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:43 smithi114 ceph-mon[40867]: 4.a scrub ok 2024-09-21T12:22:44.233 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:44.259 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:44.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:44.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:44.361 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:44.384 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:44.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:44.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:44.485 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:44.515 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:44.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:44.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:44.617 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:44.637 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 74 16 90310 90294 76.242 52.75 0.0114739 0.01311 2024-09-21T12:22:44.644 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:44.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:44.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:44.682 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:22:44.746 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:44.763 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:22:44.763 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:22:44.771 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:44.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:44.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:44.806 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:22:44.806 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:22:44.839 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:22:44.839 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:22:44.871 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:22:44.871 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:22:44.872 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:44.900 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:44.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:44.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:44.908 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:22:44.908 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:22:44.943 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:22:44.943 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:22:44.978 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:22:44.979 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:22:45.002 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:45.014 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:22:45.014 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:22:45.021 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:44 smithi114 ceph-mon[40867]: 3.1d scrub starts 2024-09-21T12:22:45.021 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:44 smithi114 ceph-mon[40867]: 3.1d scrub ok 2024-09-21T12:22:45.021 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:44 smithi114 ceph-mon[40867]: pgmap v105: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 152 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 753 B/s rd, 40 MiB/s wr, 634 op/s; 21858/263436 objects misplaced (8.297%); 13 MiB/s, 203 objects/s recovering 2024-09-21T12:22:45.021 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:44 smithi114 ceph-mon[40867]: 3.18 scrub starts 2024-09-21T12:22:45.021 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:44 smithi114 ceph-mon[40867]: 3.18 scrub ok 2024-09-21T12:22:45.021 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:44 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/588851678' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:22:45.022 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:44 smithi114 ceph-mon[30324]: 3.1d scrub starts 2024-09-21T12:22:45.022 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:44 smithi114 ceph-mon[30324]: 3.1d scrub ok 2024-09-21T12:22:45.023 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:44 smithi114 ceph-mon[30324]: pgmap v105: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 152 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 753 B/s rd, 40 MiB/s wr, 634 op/s; 21858/263436 objects misplaced (8.297%); 13 MiB/s, 203 objects/s recovering 2024-09-21T12:22:45.023 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:44 smithi114 ceph-mon[30324]: 3.18 scrub starts 2024-09-21T12:22:45.023 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:44 smithi114 ceph-mon[30324]: 3.18 scrub ok 2024-09-21T12:22:45.023 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:44 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/588851678' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:22:45.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:45.071 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:45.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:45.080 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:44 smithi183 ceph-mon[33277]: 3.1d scrub starts 2024-09-21T12:22:45.081 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:44 smithi183 ceph-mon[33277]: 3.1d scrub ok 2024-09-21T12:22:45.081 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:44 smithi183 ceph-mon[33277]: pgmap v105: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 152 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 753 B/s rd, 40 MiB/s wr, 634 op/s; 21858/263436 objects misplaced (8.297%); 13 MiB/s, 203 objects/s recovering 2024-09-21T12:22:45.081 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:44 smithi183 ceph-mon[33277]: 3.18 scrub starts 2024-09-21T12:22:45.081 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:44 smithi183 ceph-mon[33277]: 3.18 scrub ok 2024-09-21T12:22:45.081 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:44 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/588851678' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:22:45.172 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:45.206 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:45.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:45.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:45.308 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:45.346 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:45.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:45.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:45.447 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:45.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:45.503 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:45.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:45.604 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:45.639 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 75 16 91189 91173 75.9563 54.9375 0.0138295 0.0131597 2024-09-21T12:22:45.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:45.650 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:45.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:45.732 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T12:22:45.751 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:45.805 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:45.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:45.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:45.907 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:45.948 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:45.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:45.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:45.979 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:45 smithi183 ceph-mon[33277]: 4.19 scrub starts 2024-09-21T12:22:45.979 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:45 smithi183 ceph-mon[33277]: 4.19 scrub ok 2024-09-21T12:22:46.050 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:46.089 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:46.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:46.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:46.191 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:46.216 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:46.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:46.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:46.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:45 smithi114 ceph-mon[40867]: 4.19 scrub starts 2024-09-21T12:22:46.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:45 smithi114 ceph-mon[40867]: 4.19 scrub ok 2024-09-21T12:22:46.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:45 smithi114 ceph-mon[30324]: 4.19 scrub starts 2024-09-21T12:22:46.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:45 smithi114 ceph-mon[30324]: 4.19 scrub ok 2024-09-21T12:22:46.317 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:46.344 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:46.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:46.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:46.446 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:46.484 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:46.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:46.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:46.516 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:22:46.586 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:46.614 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:46.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:46.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:46.639 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 76 16 92089 92073 75.697 56.25 0.013008 0.0132053 2024-09-21T12:22:46.716 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:46.748 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:46.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:46.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:46.850 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:46.889 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:46.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:46.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:46.956 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:22:46 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:22:46.956 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:46 smithi183 ceph-mon[33277]: 6.a scrub starts 2024-09-21T12:22:46.956 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:46 smithi183 ceph-mon[33277]: 6.a scrub ok 2024-09-21T12:22:46.956 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:46 smithi183 ceph-mon[33277]: pgmap v106: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 152 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 34 MiB/s wr, 538 op/s; 21858/263436 objects misplaced (8.297%); 11 MiB/s, 173 objects/s recovering 2024-09-21T12:22:46.956 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:46 smithi183 ceph-mon[33277]: 3.10 scrub starts 2024-09-21T12:22:46.956 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:46 smithi183 ceph-mon[33277]: 3.10 scrub ok 2024-09-21T12:22:46.956 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:46 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1795986957' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:22:46.956 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:46 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:22:46.957 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:22:46.957 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:46 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:46.957 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:46 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:46.991 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:47.031 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:47.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:47.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:47.099 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-21T12:22:47.116 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:22:47.116 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:22:47.133 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:47.184 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:47.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:47.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:47.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:46 smithi114 ceph-mon[30324]: 6.a scrub starts 2024-09-21T12:22:47.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:46 smithi114 ceph-mon[30324]: 6.a scrub ok 2024-09-21T12:22:47.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:46 smithi114 ceph-mon[30324]: pgmap v106: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 152 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 34 MiB/s wr, 538 op/s; 21858/263436 objects misplaced (8.297%); 11 MiB/s, 173 objects/s recovering 2024-09-21T12:22:47.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:46 smithi114 ceph-mon[30324]: 3.10 scrub starts 2024-09-21T12:22:47.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:46 smithi114 ceph-mon[30324]: 3.10 scrub ok 2024-09-21T12:22:47.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:46 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1795986957' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:22:47.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:46 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:22:47.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:22:47.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:46 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:47.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:46 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:47.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:46 smithi114 ceph-mon[40867]: 6.a scrub starts 2024-09-21T12:22:47.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:46 smithi114 ceph-mon[40867]: 6.a scrub ok 2024-09-21T12:22:47.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:46 smithi114 ceph-mon[40867]: pgmap v106: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 152 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 34 MiB/s wr, 538 op/s; 21858/263436 objects misplaced (8.297%); 11 MiB/s, 173 objects/s recovering 2024-09-21T12:22:47.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:46 smithi114 ceph-mon[40867]: 3.10 scrub starts 2024-09-21T12:22:47.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:46 smithi114 ceph-mon[40867]: 3.10 scrub ok 2024-09-21T12:22:47.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:46 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1795986957' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:22:47.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:46 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:22:47.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:22:47.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:46 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:47.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:46 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:47.286 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:47.326 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:47.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:47.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:47.428 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:47.454 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:47.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:47.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:47.558 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:47.597 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:47.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:47.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:47.639 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 77 16 92962 92946 75.4225 54.5625 0.019479 0.0132541 2024-09-21T12:22:47.669 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:22:47.698 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:47.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:47.746 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:47.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:47.847 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:47.885 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T12:22:47.886 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:47.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:47.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:47.988 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:48.034 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:48.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:48.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:48.066 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:22:48.093 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:47 smithi114 ceph-mon[30324]: 5.15 scrub starts 2024-09-21T12:22:48.093 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:47 smithi114 ceph-mon[30324]: 5.15 scrub ok 2024-09-21T12:22:48.093 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:47 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:22:48.093 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:47 smithi114 ceph-mon[30324]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:22:48.093 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:47 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:22:48.093 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:47 smithi114 ceph-mon[30324]: osdmap e119: 8 total, 8 up, 8 in 2024-09-21T12:22:48.094 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:47 smithi114 ceph-mon[40867]: 5.15 scrub starts 2024-09-21T12:22:48.094 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:47 smithi114 ceph-mon[40867]: 5.15 scrub ok 2024-09-21T12:22:48.094 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:47 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:22:48.094 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:47 smithi114 ceph-mon[40867]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:22:48.094 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:47 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:22:48.094 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:47 smithi114 ceph-mon[40867]: osdmap e119: 8 total, 8 up, 8 in 2024-09-21T12:22:48.137 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:48.198 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:48.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:48.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:48.232 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:22:48.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:47 smithi183 ceph-mon[33277]: 5.15 scrub starts 2024-09-21T12:22:48.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:47 smithi183 ceph-mon[33277]: 5.15 scrub ok 2024-09-21T12:22:48.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:47 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:22:48.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:47 smithi183 ceph-mon[33277]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:22:48.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:47 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:22:48.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:47 smithi183 ceph-mon[33277]: osdmap e119: 8 total, 8 up, 8 in 2024-09-21T12:22:48.300 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:48.329 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:48.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:48.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:48.430 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:48.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:48.452 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:48.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:48.553 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:48.583 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:48.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:48.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:48.639 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 78 16 93794 93778 75.1222 52 0.0107626 0.0133065 2024-09-21T12:22:48.685 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:48.726 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:48.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:48.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:48.828 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:48.857 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:48.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:48.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:48.958 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:48.994 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:48.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:48.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:49.084 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:48 smithi183 ceph-mon[33277]: 7.0 scrub starts 2024-09-21T12:22:49.084 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:48 smithi183 ceph-mon[33277]: pgmap v108: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.10k op/s; 21310/276891 objects misplaced (7.696%); 22 MiB/s, 352 objects/s recovering 2024-09-21T12:22:49.096 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:49.131 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:49.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:49.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:49.157 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:48 smithi114 ceph-mon[30324]: 7.0 scrub starts 2024-09-21T12:22:49.157 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:48 smithi114 ceph-mon[30324]: pgmap v108: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.10k op/s; 21310/276891 objects misplaced (7.696%); 22 MiB/s, 352 objects/s recovering 2024-09-21T12:22:49.157 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:48 smithi114 ceph-mon[40867]: 7.0 scrub starts 2024-09-21T12:22:49.157 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:48 smithi114 ceph-mon[40867]: pgmap v108: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.10k op/s; 21310/276891 objects misplaced (7.696%); 22 MiB/s, 352 objects/s recovering 2024-09-21T12:22:49.234 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:49.234 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:22:49.234 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:22:49.279 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:49.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:49.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:49.380 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:49.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:49.406 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:49.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:49.507 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:49.533 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:49.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:49.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:49.635 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:49.640 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 79 16 94650 94634 74.8485 53.5 0.0293149 0.0133547 2024-09-21T12:22:49.665 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:49.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:49.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:49.767 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:49.812 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:49.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:49.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:49.884 INFO:teuthology.orchestra.run.smithi114.stderr:marked out osd.4. 2024-09-21T12:22:49.914 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:49.940 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:49.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:49.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:50.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:49 smithi114 ceph-mon[40867]: 3.c scrub starts 2024-09-21T12:22:50.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:49 smithi114 ceph-mon[40867]: 3.c scrub ok 2024-09-21T12:22:50.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:49 smithi114 ceph-mon[40867]: 7.0 scrub ok 2024-09-21T12:22:50.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:49 smithi114 ceph-mon[40867]: 7.7 scrub starts 2024-09-21T12:22:50.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:49 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2623138783' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T12:22:50.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:49 smithi114 ceph-mon[40867]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-21T12:22:50.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:49 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-a[30301]: 2024-09-21T12:22:49.667+0000 7f49fe685640 -1 mon.a@0(leader).osd e119 definitely_dead 0 2024-09-21T12:22:50.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:49 smithi114 ceph-mon[30324]: 3.c scrub starts 2024-09-21T12:22:50.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:49 smithi114 ceph-mon[30324]: 3.c scrub ok 2024-09-21T12:22:50.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:49 smithi114 ceph-mon[30324]: 7.0 scrub ok 2024-09-21T12:22:50.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:49 smithi114 ceph-mon[30324]: 7.7 scrub starts 2024-09-21T12:22:50.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:49 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2623138783' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T12:22:50.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:49 smithi114 ceph-mon[30324]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-21T12:22:50.042 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:50.063 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:50.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:50.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:50.165 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:50.187 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:50.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:50.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:50.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:49 smithi183 ceph-mon[33277]: 3.c scrub starts 2024-09-21T12:22:50.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:49 smithi183 ceph-mon[33277]: 3.c scrub ok 2024-09-21T12:22:50.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:49 smithi183 ceph-mon[33277]: 7.0 scrub ok 2024-09-21T12:22:50.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:49 smithi183 ceph-mon[33277]: 7.7 scrub starts 2024-09-21T12:22:50.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:49 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2623138783' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T12:22:50.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:49 smithi183 ceph-mon[33277]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-21T12:22:50.288 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:50.309 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:50.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:50.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:50.320 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T12:22:50.410 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:50.432 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:50.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:50.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:50.533 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:50.555 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:50.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:50.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:50.568 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:22:50.640 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-21T12:22:50.639617+0000 min lat: 0.00135771 max lat: 2.12184 avg lat: 0.0133646 2024-09-21T12:22:50.655 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T12:22:50.655 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 80 16 94915 94899 74.12 16.5625 0.00610126 0.0133646 2024-09-21T12:22:50.657 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:50.678 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:50.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:50.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:50.780 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:50.801 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:50.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:50.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:50.903 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:50.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:50.947 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:50.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:50.951 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:50 smithi183 ceph-mon[33277]: pgmap v109: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 58 MiB/s wr, 934 op/s; 21310/276891 objects misplaced (7.696%); 22 MiB/s, 352 objects/s recovering 2024-09-21T12:22:50.951 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:50 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2623138783' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-21T12:22:50.951 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:50 smithi183 ceph-mon[33277]: osdmap e120: 8 total, 8 up, 7 in 2024-09-21T12:22:50.951 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:50 smithi183 ceph-mon[33277]: 4.6 scrub starts 2024-09-21T12:22:50.951 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:50 smithi183 ceph-mon[33277]: 4.6 scrub ok 2024-09-21T12:22:50.951 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:50 smithi183 ceph-mon[33277]: 7.7 scrub ok 2024-09-21T12:22:50.952 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:50 smithi183 ceph-mon[33277]: 6.11 scrub starts 2024-09-21T12:22:50.952 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:50 smithi183 ceph-mon[33277]: 6.11 scrub ok 2024-09-21T12:22:51.048 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:51.077 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:51.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:51.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:51.078 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:22:51.088 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:22:51.088 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:22:51.179 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:51.199 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:51.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:51.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:51.216 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:50 smithi114 ceph-mon[40867]: pgmap v109: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 58 MiB/s wr, 934 op/s; 21310/276891 objects misplaced (7.696%); 22 MiB/s, 352 objects/s recovering 2024-09-21T12:22:51.217 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:50 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2623138783' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-21T12:22:51.217 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:50 smithi114 ceph-mon[40867]: osdmap e120: 8 total, 8 up, 7 in 2024-09-21T12:22:51.217 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:50 smithi114 ceph-mon[40867]: 4.6 scrub starts 2024-09-21T12:22:51.217 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:50 smithi114 ceph-mon[40867]: 4.6 scrub ok 2024-09-21T12:22:51.217 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:50 smithi114 ceph-mon[40867]: 7.7 scrub ok 2024-09-21T12:22:51.217 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:50 smithi114 ceph-mon[40867]: 6.11 scrub starts 2024-09-21T12:22:51.217 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:50 smithi114 ceph-mon[40867]: 6.11 scrub ok 2024-09-21T12:22:51.217 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:50 smithi114 ceph-mon[30324]: pgmap v109: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 58 MiB/s wr, 934 op/s; 21310/276891 objects misplaced (7.696%); 22 MiB/s, 352 objects/s recovering 2024-09-21T12:22:51.217 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:50 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2623138783' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-21T12:22:51.217 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:50 smithi114 ceph-mon[30324]: osdmap e120: 8 total, 8 up, 7 in 2024-09-21T12:22:51.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:50 smithi114 ceph-mon[30324]: 4.6 scrub starts 2024-09-21T12:22:51.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:50 smithi114 ceph-mon[30324]: 4.6 scrub ok 2024-09-21T12:22:51.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:50 smithi114 ceph-mon[30324]: 7.7 scrub ok 2024-09-21T12:22:51.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:50 smithi114 ceph-mon[30324]: 6.11 scrub starts 2024-09-21T12:22:51.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:50 smithi114 ceph-mon[30324]: 6.11 scrub ok 2024-09-21T12:22:51.301 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:51.323 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:51.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:51.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:51.425 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:51.446 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:51.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:51.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:51.548 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:51.570 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:51.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:51.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:51.640 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 81 16 95066 95050 73.3215 9.4375 0.00296366 0.0135212 2024-09-21T12:22:51.672 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:51.693 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:51.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:51.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:51.733 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T12:22:51.794 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:51.840 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:51.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:51.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:51.943 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:51.959 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:22:51.959 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:22:51.960 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:51 smithi114 ceph-mon[30324]: osdmap e121: 8 total, 8 up, 7 in 2024-09-21T12:22:51.960 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:51 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:22:51.960 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:51 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:22:51.960 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:51 smithi114 ceph-mon[30324]: osdmap e122: 8 total, 8 up, 7 in 2024-09-21T12:22:51.960 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:51 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/323004956' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:22:51.960 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:51 smithi114 ceph-mon[40867]: osdmap e121: 8 total, 8 up, 7 in 2024-09-21T12:22:51.960 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:51 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:22:51.960 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:51 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:22:51.960 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:51 smithi114 ceph-mon[40867]: osdmap e122: 8 total, 8 up, 7 in 2024-09-21T12:22:51.961 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:51 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/323004956' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:22:51.967 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:22:51.968 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:22:51.971 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:51.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:51.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:51.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:51 smithi183 ceph-mon[33277]: osdmap e121: 8 total, 8 up, 7 in 2024-09-21T12:22:51.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:51 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:22:51.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:51 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:22:51.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:51 smithi183 ceph-mon[33277]: osdmap e122: 8 total, 8 up, 7 in 2024-09-21T12:22:51.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:51 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/323004956' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:22:52.005 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:22:52.006 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:22:52.035 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:22:52.036 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:22:52.068 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:22:52.068 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:22:52.073 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:52.099 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:22:52.099 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:22:52.103 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:22:52.113 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:52.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:52.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:52.140 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:22:52.140 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:22:52.175 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:22:52.176 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:22:52.215 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:52.243 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:52.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:52.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:52.280 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T12:22:52.344 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:52.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:52.396 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:52.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:52.496 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:52.543 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:52.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:52.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:52.640 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 82 16 95894 95878 73.0583 51.75 0.0156639 0.013682 2024-09-21T12:22:52.645 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:52.668 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:22:52.668 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:22:52.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.669 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:95904 7.7 7:ef6f8cb3:::benchmark_data_smithi114_78432_object95903:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-21T12:22:52.669 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:52.647664+0000", 2024-09-21T12:22:52.669 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.018637509999999999, 2024-09-21T12:22:52.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.01866021, 2024-09-21T12:22:52.669 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:52.669 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:52.669 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:22:52.669 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:22:52.669 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:22:52.669 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:52.669 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 95904 2024-09-21T12:22:52.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.669 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:52.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:52.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.647664+0000", 2024-09-21T12:22:52.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:52.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:52.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.647664+0000", 2024-09-21T12:22:52.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:52.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:52.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.647665+0000", 2024-09-21T12:22:52.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7379999999999999e-06 2024-09-21T12:22:52.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:52.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.647723+0000", 2024-09-21T12:22:52.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7717000000000003e-05 2024-09-21T12:22:52.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:52.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.647733+0000", 2024-09-21T12:22:52.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9089999999999995e-06 2024-09-21T12:22:52.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:52.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.647738+0000", 2024-09-21T12:22:52.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6460000000000001e-06 2024-09-21T12:22:52.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:52.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.647779+0000", 2024-09-21T12:22:52.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1198000000000001e-05 2024-09-21T12:22:52.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:52.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.647869+0000", 2024-09-21T12:22:52.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9808999999999993e-05 2024-09-21T12:22:52.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,3", 2024-09-21T12:22:52.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.647955+0000", 2024-09-21T12:22:52.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6403000000000006e-05 2024-09-21T12:22:52.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:22:52.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.650014+0000", 2024-09-21T12:22:52.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020585880000000001 2024-09-21T12:22:52.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:52.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.653960+0000", 2024-09-21T12:22:52.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0039459389999999999 2024-09-21T12:22:52.673 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:52.673 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:52.673 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:52.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.674 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:95915 7.7 7:e5842e20:::benchmark_data_smithi114_78432_object95914:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-21T12:22:52.674 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:52.655639+0000", 2024-09-21T12:22:52.674 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.010662058, 2024-09-21T12:22:52.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0107315, 2024-09-21T12:22:52.674 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:52.674 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:52.674 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:22:52.674 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:22:52.674 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:22:52.674 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:52.674 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 95915 2024-09-21T12:22:52.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.675 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:52.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:52.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.655639+0000", 2024-09-21T12:22:52.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:52.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:52.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.655639+0000", 2024-09-21T12:22:52.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:52.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:52.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.655640+0000", 2024-09-21T12:22:52.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.158e-06 2024-09-21T12:22:52.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:52.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.655695+0000", 2024-09-21T12:22:52.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4951000000000003e-05 2024-09-21T12:22:52.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:52.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.655707+0000", 2024-09-21T12:22:52.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1888e-05 2024-09-21T12:22:52.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:52.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.655714+0000", 2024-09-21T12:22:52.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8369999999999999e-06 2024-09-21T12:22:52.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:52.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.655767+0000", 2024-09-21T12:22:52.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3068000000000001e-05 2024-09-21T12:22:52.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:52.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.655901+0000", 2024-09-21T12:22:52.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013386999999999999 2024-09-21T12:22:52.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,3", 2024-09-21T12:22:52.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.656056+0000", 2024-09-21T12:22:52.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015469799999999999 2024-09-21T12:22:52.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:22:52.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.658102+0000", 2024-09-21T12:22:52.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020458640000000001 2024-09-21T12:22:52.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:52.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.662700+0000", 2024-09-21T12:22:52.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0045980420000000001 2024-09-21T12:22:52.678 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:52.678 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:52.678 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:52.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.679 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:95922 7.d 7:bf8f19ac:::benchmark_data_smithi114_78432_object95921:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-21T12:22:52.679 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:52.663218+0000", 2024-09-21T12:22:52.679 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0030832500000000001, 2024-09-21T12:22:52.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0031932190000000002, 2024-09-21T12:22:52.679 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:52.679 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:52.679 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:22:52.679 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:22:52.679 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:22:52.679 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:52.679 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 95922 2024-09-21T12:22:52.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.679 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:52.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:52.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.663218+0000", 2024-09-21T12:22:52.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:52.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:52.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.663218+0000", 2024-09-21T12:22:52.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:52.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:52.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.663219+0000", 2024-09-21T12:22:52.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.209e-06 2024-09-21T12:22:52.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:52.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.663263+0000", 2024-09-21T12:22:52.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3334999999999998e-05 2024-09-21T12:22:52.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:52.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.663274+0000", 2024-09-21T12:22:52.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1844e-05 2024-09-21T12:22:52.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:52.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.663280+0000", 2024-09-21T12:22:52.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3929999999999999e-06 2024-09-21T12:22:52.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:52.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.663329+0000", 2024-09-21T12:22:52.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9305999999999999e-05 2024-09-21T12:22:52.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:52.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.663447+0000", 2024-09-21T12:22:52.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011799900000000001 2024-09-21T12:22:52.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1", 2024-09-21T12:22:52.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.663573+0000", 2024-09-21T12:22:52.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012576800000000001 2024-09-21T12:22:52.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:52.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:52.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:22:52.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:52.665843+0000", 2024-09-21T12:22:52.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002270233 2024-09-21T12:22:52.683 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:52.683 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:52.683 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:52.683 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:52.683 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-21T12:22:52.683 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 3 2024-09-21T12:22:52.683 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:22:52.690 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:52.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:52.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:52.791 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:52.834 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:52.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:52.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:52.903 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:22:52 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:22:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:22:52.937 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:52.978 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:52.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:52.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:53.080 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:53.105 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:53.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:53.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:53.206 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:53.232 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:53.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:53.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:53.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:52 smithi183 ceph-mon[33277]: pgmap v112: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 5.7 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.7 KiB/s rd, 56 MiB/s wr, 897 op/s; 21310/279564 objects misplaced (7.623%); 15 MiB/s, 244 objects/s recovering 2024-09-21T12:22:53.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:52 smithi183 ceph-mon[33277]: 6.0 scrub starts 2024-09-21T12:22:53.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:52 smithi183 ceph-mon[33277]: 6.0 scrub ok 2024-09-21T12:22:53.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:52 smithi183 ceph-mon[33277]: 3.19 scrub starts 2024-09-21T12:22:53.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:52 smithi183 ceph-mon[33277]: 3.19 scrub ok 2024-09-21T12:22:53.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:52 smithi114 ceph-mon[30324]: pgmap v112: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 5.7 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.7 KiB/s rd, 56 MiB/s wr, 897 op/s; 21310/279564 objects misplaced (7.623%); 15 MiB/s, 244 objects/s recovering 2024-09-21T12:22:53.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:52 smithi114 ceph-mon[30324]: 6.0 scrub starts 2024-09-21T12:22:53.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:52 smithi114 ceph-mon[30324]: 6.0 scrub ok 2024-09-21T12:22:53.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:52 smithi114 ceph-mon[30324]: 3.19 scrub starts 2024-09-21T12:22:53.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:52 smithi114 ceph-mon[30324]: 3.19 scrub ok 2024-09-21T12:22:53.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:52 smithi114 ceph-mon[40867]: pgmap v112: 158 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 5.7 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.7 KiB/s rd, 56 MiB/s wr, 897 op/s; 21310/279564 objects misplaced (7.623%); 15 MiB/s, 244 objects/s recovering 2024-09-21T12:22:53.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:52 smithi114 ceph-mon[40867]: 6.0 scrub starts 2024-09-21T12:22:53.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:52 smithi114 ceph-mon[40867]: 6.0 scrub ok 2024-09-21T12:22:53.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:52 smithi114 ceph-mon[40867]: 3.19 scrub starts 2024-09-21T12:22:53.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:52 smithi114 ceph-mon[40867]: 3.19 scrub ok 2024-09-21T12:22:53.335 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:53.372 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:53.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:53.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:53.405 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T12:22:53.474 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:53.516 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:53.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:53.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:53.578 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:22:53.619 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:53.640 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 83 16 97012 96996 73.0197 69.875 0.0130499 0.0136892 2024-09-21T12:22:53.680 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:53.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:53.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:53.772 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:22:53.782 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:53.820 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:53.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:53.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:53.921 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:53.948 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:53.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:53.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:54.050 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:54.098 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:54.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:54.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:54.188 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:53 smithi183 ceph-mon[33277]: 3.0 scrub starts 2024-09-21T12:22:54.188 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:53 smithi183 ceph-mon[33277]: 3.0 scrub ok 2024-09-21T12:22:54.200 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:54.226 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:54.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:54.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:54.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:53 smithi114 ceph-mon[30324]: 3.0 scrub starts 2024-09-21T12:22:54.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:53 smithi114 ceph-mon[30324]: 3.0 scrub ok 2024-09-21T12:22:54.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:53 smithi114 ceph-mon[40867]: 3.0 scrub starts 2024-09-21T12:22:54.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:53 smithi114 ceph-mon[40867]: 3.0 scrub ok 2024-09-21T12:22:54.303 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:22:54.303 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-21T12:22:54.303 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-21T12:22:54.303 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-21T12:22:54.303 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:22:54.329 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:54.376 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:54.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:54.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:54.478 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:54.504 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:54.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:54.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:54.606 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:54.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:54.630 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:54.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:54.640 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 84 16 97968 97952 72.8617 59.75 0.111917 0.0137081 2024-09-21T12:22:54.731 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:54.758 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:54.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:54.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:54.860 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:54.899 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:54.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:54.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:54.981 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T12:22:55.000 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:55.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:55.023 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:55.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:55.091 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:22:55.124 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:55.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:55.174 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:55.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:55.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:54 smithi183 ceph-mon[33277]: pgmap v114: 158 pgs: 1 active+recovering+undersized+remapped, 1 peering, 156 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 31 MiB/s wr, 492 op/s; 1294/285747 objects misplaced (0.453%); 10 MiB/s, 178 objects/s recovering 2024-09-21T12:22:55.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:54 smithi183 ceph-mon[33277]: 4.5 scrub starts 2024-09-21T12:22:55.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:54 smithi183 ceph-mon[33277]: 4.5 scrub ok 2024-09-21T12:22:55.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:54 smithi183 ceph-mon[33277]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-21T12:22:55.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:54 smithi183 ceph-mon[33277]: Cluster is now healthy 2024-09-21T12:22:55.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:54 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:55.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:54 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:55.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:54 smithi183 ceph-mon[33277]: 3.12 scrub starts 2024-09-21T12:22:55.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:54 smithi183 ceph-mon[33277]: 3.12 scrub ok 2024-09-21T12:22:55.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:54 smithi183 ceph-mon[33277]: 3.1c scrub starts 2024-09-21T12:22:55.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:54 smithi183 ceph-mon[33277]: 3.1c scrub ok 2024-09-21T12:22:55.265 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:22:55.274 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:55.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:54 smithi114 ceph-mon[30324]: pgmap v114: 158 pgs: 1 active+recovering+undersized+remapped, 1 peering, 156 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 31 MiB/s wr, 492 op/s; 1294/285747 objects misplaced (0.453%); 10 MiB/s, 178 objects/s recovering 2024-09-21T12:22:55.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:54 smithi114 ceph-mon[30324]: 4.5 scrub starts 2024-09-21T12:22:55.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:54 smithi114 ceph-mon[30324]: 4.5 scrub ok 2024-09-21T12:22:55.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:54 smithi114 ceph-mon[30324]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-21T12:22:55.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:54 smithi114 ceph-mon[30324]: Cluster is now healthy 2024-09-21T12:22:55.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:54 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:55.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:54 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:55.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:54 smithi114 ceph-mon[30324]: 3.12 scrub starts 2024-09-21T12:22:55.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:54 smithi114 ceph-mon[30324]: 3.12 scrub ok 2024-09-21T12:22:55.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:54 smithi114 ceph-mon[30324]: 3.1c scrub starts 2024-09-21T12:22:55.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:54 smithi114 ceph-mon[30324]: 3.1c scrub ok 2024-09-21T12:22:55.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:54 smithi114 ceph-mon[40867]: pgmap v114: 158 pgs: 1 active+recovering+undersized+remapped, 1 peering, 156 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 31 MiB/s wr, 492 op/s; 1294/285747 objects misplaced (0.453%); 10 MiB/s, 178 objects/s recovering 2024-09-21T12:22:55.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:54 smithi114 ceph-mon[40867]: 4.5 scrub starts 2024-09-21T12:22:55.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:54 smithi114 ceph-mon[40867]: 4.5 scrub ok 2024-09-21T12:22:55.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:54 smithi114 ceph-mon[40867]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-21T12:22:55.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:54 smithi114 ceph-mon[40867]: Cluster is now healthy 2024-09-21T12:22:55.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:54 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:55.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:54 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:55.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:54 smithi114 ceph-mon[40867]: 3.12 scrub starts 2024-09-21T12:22:55.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:54 smithi114 ceph-mon[40867]: 3.12 scrub ok 2024-09-21T12:22:55.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:54 smithi114 ceph-mon[40867]: 3.1c scrub starts 2024-09-21T12:22:55.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:54 smithi114 ceph-mon[40867]: 3.1c scrub ok 2024-09-21T12:22:55.304 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:55.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:55.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:55.406 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:55.427 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:55.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:55.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:55.529 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:55.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:55.573 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:55.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:55.641 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 85 16 98899 98883 72.689 58.1875 0.0135656 0.0137521 2024-09-21T12:22:55.674 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:55.698 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:55.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:55.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:55.800 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:55.834 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:55.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:55.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:55.936 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:55.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:55.976 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:55.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:56.004 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T12:22:56.004 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:22:56.004 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-21T12:22:56.004 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-21T12:22:56.078 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:56.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:56.123 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:56.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:56.192 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:55 smithi183 ceph-mon[33277]: 5.1e scrub starts 2024-09-21T12:22:56.192 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:55 smithi183 ceph-mon[33277]: 5.1e scrub ok 2024-09-21T12:22:56.192 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:55 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:22:56.193 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:55 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:22:56.193 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:55 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:56.224 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:56.251 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:56.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:56.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:56.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:55 smithi114 ceph-mon[40867]: 5.1e scrub starts 2024-09-21T12:22:56.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:55 smithi114 ceph-mon[40867]: 5.1e scrub ok 2024-09-21T12:22:56.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:55 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:22:56.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:55 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:22:56.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:55 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:55 smithi114 ceph-mon[30324]: 5.1e scrub starts 2024-09-21T12:22:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:55 smithi114 ceph-mon[30324]: 5.1e scrub ok 2024-09-21T12:22:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:55 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:22:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:55 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:22:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:55 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:22:56.353 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:56.383 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:56.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:56.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:56.443 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:22:56.443 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-21T12:22:56.443 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-21T12:22:56.443 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:22:56.444 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.444 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14478.0:1119 5.1f 5:f95f44c2:::notify.0:head [watch ping cookie 94620848892160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-21T12:22:56.444 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:40.364371+0000", 2024-09-21T12:22:56.444 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 76.076172051, 2024-09-21T12:22:56.444 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78371777300000001, 2024-09-21T12:22:56.444 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:56.444 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:56.444 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:22:56.444 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:22:56.444 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14478", 2024-09-21T12:22:56.444 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:22:56.444 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 1119 2024-09-21T12:22:56.444 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.445 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:56.445 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.445 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:56.445 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364371+0000", 2024-09-21T12:22:56.445 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.445 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.445 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.445 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:56.445 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364371+0000", 2024-09-21T12:22:56.445 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.445 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.445 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.445 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:56.445 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364372+0000", 2024-09-21T12:22:56.445 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6199999999999997e-07 2024-09-21T12:22:56.445 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.446 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.446 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:56.446 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364376+0000", 2024-09-21T12:22:56.446 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8099999999999999e-06 2024-09-21T12:22:56.446 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.446 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.446 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:56.446 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364384+0000", 2024-09-21T12:22:56.446 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2870000000000004e-06 2024-09-21T12:22:56.446 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.446 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.446 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:56.446 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364393+0000", 2024-09-21T12:22:56.446 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5820000000000002e-06 2024-09-21T12:22:56.447 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.447 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.447 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:56.447 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364557+0000", 2024-09-21T12:22:56.447 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016425399999999999 2024-09-21T12:22:56.447 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.447 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.447 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:22:56.447 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364573+0000", 2024-09-21T12:22:56.447 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6286999999999998e-05 2024-09-21T12:22:56.447 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.447 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.447 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:56.447 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.144878+0000", 2024-09-21T12:22:56.447 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78030514600000001 2024-09-21T12:22:56.447 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.448 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.448 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for degraded object", 2024-09-21T12:22:56.448 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.145417+0000", 2024-09-21T12:22:56.448 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00053866500000000002 2024-09-21T12:22:56.448 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.448 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.448 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:56.448 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.147869+0000", 2024-09-21T12:22:56.448 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024518779999999998 2024-09-21T12:22:56.448 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.448 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.448 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:56.448 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.147939+0000", 2024-09-21T12:22:56.448 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0574000000000006e-05 2024-09-21T12:22:56.448 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.448 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.449 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:56.449 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.148089+0000", 2024-09-21T12:22:56.449 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014952800000000001 2024-09-21T12:22:56.449 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.449 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:56.449 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.449 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.449 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.449 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14478.0:1120 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch ping cookie 94620852177152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-21T12:22:56.449 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:40.364419+0000", 2024-09-21T12:22:56.449 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 76.076124375000006, 2024-09-21T12:22:56.449 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78472206099999997, 2024-09-21T12:22:56.449 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:56.450 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:56.450 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:22:56.450 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:22:56.450 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14478", 2024-09-21T12:22:56.450 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:22:56.450 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 1120 2024-09-21T12:22:56.450 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.450 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:56.450 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.450 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:56.450 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364419+0000", 2024-09-21T12:22:56.450 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.450 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.450 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.450 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:56.451 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364419+0000", 2024-09-21T12:22:56.451 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.451 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.451 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.451 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:56.451 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364419+0000", 2024-09-21T12:22:56.451 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5400000000000002e-07 2024-09-21T12:22:56.451 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.451 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.451 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:56.451 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364426+0000", 2024-09-21T12:22:56.451 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8650000000000003e-06 2024-09-21T12:22:56.451 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.452 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.452 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:56.452 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364434+0000", 2024-09-21T12:22:56.452 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2309999999999996e-06 2024-09-21T12:22:56.452 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.452 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.452 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:56.452 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364441+0000", 2024-09-21T12:22:56.452 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0929999999999999e-06 2024-09-21T12:22:56.452 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.452 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.452 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:56.452 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364520+0000", 2024-09-21T12:22:56.453 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9487999999999995e-05 2024-09-21T12:22:56.453 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.453 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.453 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:22:56.453 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364543+0000", 2024-09-21T12:22:56.453 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.296e-05 2024-09-21T12:22:56.453 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.453 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.453 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:56.453 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.147074+0000", 2024-09-21T12:22:56.453 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78253067600000004 2024-09-21T12:22:56.453 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.453 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.453 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for degraded object", 2024-09-21T12:22:56.453 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.147381+0000", 2024-09-21T12:22:56.453 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00030698200000000002 2024-09-21T12:22:56.454 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.454 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.454 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:56.454 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.148992+0000", 2024-09-21T12:22:56.454 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001611267 2024-09-21T12:22:56.454 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.454 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.454 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:56.454 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.149064+0000", 2024-09-21T12:22:56.454 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2058999999999996e-05 2024-09-21T12:22:56.454 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.454 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.454 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:56.454 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.149141+0000", 2024-09-21T12:22:56.455 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6985999999999998e-05 2024-09-21T12:22:56.455 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.455 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:56.455 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.455 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.455 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.455 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14478.0:1124 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94620848892160 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:22:56.455 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:40.364832+0000", 2024-09-21T12:22:56.455 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 76.075711323999997, 2024-09-21T12:22:56.455 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78486121600000003, 2024-09-21T12:22:56.455 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:56.455 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:56.455 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:22:56.455 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:22:56.455 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14478", 2024-09-21T12:22:56.456 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:22:56.456 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 1124 2024-09-21T12:22:56.456 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.456 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:56.456 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.456 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:56.456 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364832+0000", 2024-09-21T12:22:56.456 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.456 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.456 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.456 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:56.456 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364832+0000", 2024-09-21T12:22:56.456 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.456 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.456 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.457 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:56.457 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364833+0000", 2024-09-21T12:22:56.457 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3900000000000004e-07 2024-09-21T12:22:56.457 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.457 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.457 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:56.457 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364838+0000", 2024-09-21T12:22:56.458 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1909999999999999e-06 2024-09-21T12:22:56.458 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.458 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.458 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:56.458 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364846+0000", 2024-09-21T12:22:56.458 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7000000000000008e-06 2024-09-21T12:22:56.458 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.458 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.458 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:56.458 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364850+0000", 2024-09-21T12:22:56.458 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.921e-06 2024-09-21T12:22:56.458 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.458 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.458 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:56.459 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364911+0000", 2024-09-21T12:22:56.459 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0636e-05 2024-09-21T12:22:56.459 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.459 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.459 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:22:56.459 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364922+0000", 2024-09-21T12:22:56.459 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0964e-05 2024-09-21T12:22:56.459 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.459 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.459 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:56.459 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.145447+0000", 2024-09-21T12:22:56.459 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78052487599999998 2024-09-21T12:22:56.459 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.459 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.460 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for degraded object", 2024-09-21T12:22:56.460 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.145493+0000", 2024-09-21T12:22:56.460 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5745000000000003e-05 2024-09-21T12:22:56.460 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.460 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.460 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:56.460 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.148124+0000", 2024-09-21T12:22:56.460 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0026310719999999999 2024-09-21T12:22:56.460 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.460 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.460 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:56.460 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.148191+0000", 2024-09-21T12:22:56.460 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7447999999999995e-05 2024-09-21T12:22:56.460 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.460 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.460 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:56.461 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.149693+0000", 2024-09-21T12:22:56.461 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001501924 2024-09-21T12:22:56.461 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.461 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:56.461 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.461 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.461 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.461 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14478.0:1125 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94620852177152 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:22:56.461 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:40.364865+0000", 2024-09-21T12:22:56.461 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 76.075677811000006, 2024-09-21T12:22:56.461 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78551680400000001, 2024-09-21T12:22:56.461 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:56.461 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:56.461 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:22:56.461 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:22:56.462 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14478", 2024-09-21T12:22:56.462 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:22:56.462 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 1125 2024-09-21T12:22:56.462 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.462 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:56.462 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.462 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:56.462 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364865+0000", 2024-09-21T12:22:56.462 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.462 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.462 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.462 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:56.462 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364865+0000", 2024-09-21T12:22:56.462 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.463 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.463 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.463 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:56.463 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364866+0000", 2024-09-21T12:22:56.463 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6499999999999999e-07 2024-09-21T12:22:56.463 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.463 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.463 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:56.463 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364868+0000", 2024-09-21T12:22:56.463 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.39e-06 2024-09-21T12:22:56.463 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.463 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.463 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:56.463 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364882+0000", 2024-09-21T12:22:56.464 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3890000000000001e-05 2024-09-21T12:22:56.464 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.464 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.464 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:56.464 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364888+0000", 2024-09-21T12:22:56.464 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3940000000000004e-06 2024-09-21T12:22:56.464 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.464 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.464 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:56.464 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364953+0000", 2024-09-21T12:22:56.464 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5877000000000006e-05 2024-09-21T12:22:56.464 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.464 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.464 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:22:56.464 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364963+0000", 2024-09-21T12:22:56.465 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5349999999999993e-06 2024-09-21T12:22:56.465 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.465 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.465 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:56.465 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.147420+0000", 2024-09-21T12:22:56.465 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78245673400000004 2024-09-21T12:22:56.465 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.465 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.465 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for degraded object", 2024-09-21T12:22:56.465 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.147469+0000", 2024-09-21T12:22:56.465 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9699e-05 2024-09-21T12:22:56.465 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.465 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.465 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:56.465 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.149190+0000", 2024-09-21T12:22:56.465 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001720606 2024-09-21T12:22:56.465 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.466 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.466 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:56.466 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.149267+0000", 2024-09-21T12:22:56.466 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7194000000000007e-05 2024-09-21T12:22:56.466 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.466 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.466 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:56.466 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.150382+0000", 2024-09-21T12:22:56.466 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00111502 2024-09-21T12:22:56.466 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.466 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:56.466 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.466 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.466 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.467 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:24811 7.7 7:ec8774ff:::benchmark_data_smithi114_78432_object24810:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T12:22:56.467 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:42.445492+0000", 2024-09-21T12:22:56.467 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 73.995050879999994, 2024-09-21T12:22:56.467 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.145669727, 2024-09-21T12:22:56.467 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:56.467 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:56.467 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:56.467 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:22:56.467 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:22:56.467 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:56.467 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 24811 2024-09-21T12:22:56.467 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.467 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:56.467 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.467 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:56.468 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445492+0000", 2024-09-21T12:22:56.468 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.468 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.468 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.468 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:56.468 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445492+0000", 2024-09-21T12:22:56.468 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.468 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.468 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.468 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:56.468 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445494+0000", 2024-09-21T12:22:56.468 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5460000000000001e-06 2024-09-21T12:22:56.468 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.468 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.468 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:56.468 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445541+0000", 2024-09-21T12:22:56.469 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6687999999999998e-05 2024-09-21T12:22:56.469 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.469 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.469 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:56.469 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445553+0000", 2024-09-21T12:22:56.469 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2517999999999999e-05 2024-09-21T12:22:56.469 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.469 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.469 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:56.469 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445560+0000", 2024-09-21T12:22:56.469 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6579999999999996e-06 2024-09-21T12:22:56.469 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.469 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.469 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:56.469 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445625+0000", 2024-09-21T12:22:56.469 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5092000000000006e-05 2024-09-21T12:22:56.469 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.470 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.470 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:56.470 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445776+0000", 2024-09-21T12:22:56.470 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0001507 2024-09-21T12:22:56.470 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.470 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.470 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,3", 2024-09-21T12:22:56.470 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445923+0000", 2024-09-21T12:22:56.470 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014760199999999999 2024-09-21T12:22:56.470 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.470 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.470 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:22:56.471 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.446851+0000", 2024-09-21T12:22:56.471 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00092788099999999997 2024-09-21T12:22:56.471 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.471 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.471 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.471 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.590740+0000", 2024-09-21T12:22:56.471 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14388880500000001 2024-09-21T12:22:56.471 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.471 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.471 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.471 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.591050+0000", 2024-09-21T12:22:56.471 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000310217 2024-09-21T12:22:56.471 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.471 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.472 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:56.472 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.591109+0000", 2024-09-21T12:22:56.472 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9188000000000003e-05 2024-09-21T12:22:56.472 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.472 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.472 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:56.472 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.591162+0000", 2024-09-21T12:22:56.472 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2831999999999998e-05 2024-09-21T12:22:56.472 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.472 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:56.472 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.472 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.472 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.472 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:45498 7.7 7:e1336a67:::benchmark_data_smithi114_78432_object45497:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T12:22:56.473 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:59.293850+0000", 2024-09-21T12:22:56.473 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 57.146692956999999, 2024-09-21T12:22:56.473 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15681044, 2024-09-21T12:22:56.473 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:56.473 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:56.473 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:56.473 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:22:56.473 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:22:56.473 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:56.473 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 45498 2024-09-21T12:22:56.473 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.473 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:56.473 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.474 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:56.474 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293850+0000", 2024-09-21T12:22:56.474 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.474 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.474 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.474 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:56.474 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293850+0000", 2024-09-21T12:22:56.474 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.474 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.474 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.474 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:56.474 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293851+0000", 2024-09-21T12:22:56.474 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.096e-06 2024-09-21T12:22:56.474 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.475 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.475 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:56.475 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293907+0000", 2024-09-21T12:22:56.475 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5126e-05 2024-09-21T12:22:56.475 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.475 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.475 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:56.475 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293915+0000", 2024-09-21T12:22:56.475 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8419999999999994e-06 2024-09-21T12:22:56.475 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.475 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.475 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:56.475 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293920+0000", 2024-09-21T12:22:56.475 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5900000000000001e-06 2024-09-21T12:22:56.475 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.475 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.476 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:56.476 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293959+0000", 2024-09-21T12:22:56.476 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9447999999999999e-05 2024-09-21T12:22:56.476 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.476 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.476 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:56.476 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294060+0000", 2024-09-21T12:22:56.476 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100247 2024-09-21T12:22:56.476 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.476 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.476 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,3", 2024-09-21T12:22:56.476 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294145+0000", 2024-09-21T12:22:56.476 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5099999999999995e-05 2024-09-21T12:22:56.476 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.477 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.477 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:22:56.477 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.295566+0000", 2024-09-21T12:22:56.477 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014215429999999999 2024-09-21T12:22:56.477 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.477 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.477 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.477 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.449406+0000", 2024-09-21T12:22:56.477 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15383943999999999 2024-09-21T12:22:56.477 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.477 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.477 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.477 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.450563+0000", 2024-09-21T12:22:56.477 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00115749 2024-09-21T12:22:56.477 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.478 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.478 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:56.478 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.450621+0000", 2024-09-21T12:22:56.478 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8161000000000001e-05 2024-09-21T12:22:56.478 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.478 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.478 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:56.478 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.450661+0000", 2024-09-21T12:22:56.478 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9357000000000003e-05 2024-09-21T12:22:56.478 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.478 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:56.478 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.478 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.478 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.478 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:45514 7.c 7:3b027556:::benchmark_data_smithi114_78432_object45513:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T12:22:56.479 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:59.302539+0000", 2024-09-21T12:22:56.479 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 57.138004064, 2024-09-21T12:22:56.479 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.150159925, 2024-09-21T12:22:56.479 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:56.479 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:56.479 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:56.479 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:22:56.479 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:22:56.479 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:56.479 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 45514 2024-09-21T12:22:56.479 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.479 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:56.479 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.479 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:56.479 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302539+0000", 2024-09-21T12:22:56.479 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.480 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.480 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.480 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:56.480 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302539+0000", 2024-09-21T12:22:56.480 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.480 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.480 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.480 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:56.480 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302541+0000", 2024-09-21T12:22:56.480 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.376e-06 2024-09-21T12:22:56.480 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.480 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.480 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:56.480 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302592+0000", 2024-09-21T12:22:56.481 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1054999999999998e-05 2024-09-21T12:22:56.481 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.481 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.481 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:56.481 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302603+0000", 2024-09-21T12:22:56.481 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0883000000000001e-05 2024-09-21T12:22:56.481 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.481 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.481 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:56.481 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302608+0000", 2024-09-21T12:22:56.481 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5339999999999999e-06 2024-09-21T12:22:56.481 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.481 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.482 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:56.482 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302655+0000", 2024-09-21T12:22:56.482 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7125e-05 2024-09-21T12:22:56.482 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.482 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.482 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:56.482 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302784+0000", 2024-09-21T12:22:56.482 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012886700000000001 2024-09-21T12:22:56.482 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.482 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.482 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:22:56.482 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.303003+0000", 2024-09-21T12:22:56.482 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00021902099999999999 2024-09-21T12:22:56.483 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.483 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.483 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:22:56.483 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.304304+0000", 2024-09-21T12:22:56.483 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0013008060000000001 2024-09-21T12:22:56.483 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.483 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.483 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.483 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.304689+0000", 2024-09-21T12:22:56.483 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000385208 2024-09-21T12:22:56.483 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.483 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.483 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.483 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.452633+0000", 2024-09-21T12:22:56.483 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14794360000000001 2024-09-21T12:22:56.484 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.484 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.484 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:56.484 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.452669+0000", 2024-09-21T12:22:56.484 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6202000000000003e-05 2024-09-21T12:22:56.484 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.484 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.484 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:56.484 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.452699+0000", 2024-09-21T12:22:56.484 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0247999999999999e-05 2024-09-21T12:22:56.484 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.484 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:56.484 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.484 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.484 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.485 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:49226 7.d 7:bcebfbbf:::benchmark_data_smithi114_78432_object49225:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:22:56.485 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:04.569255+0000", 2024-09-21T12:22:56.485 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 51.871288077000003, 2024-09-21T12:22:56.485 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14786647, 2024-09-21T12:22:56.485 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:56.485 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:56.485 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:56.485 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:22:56.485 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:22:56.485 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:56.485 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 49226 2024-09-21T12:22:56.485 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.485 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:56.485 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.485 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:56.486 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569255+0000", 2024-09-21T12:22:56.486 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.486 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.486 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.486 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:56.486 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569255+0000", 2024-09-21T12:22:56.486 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.486 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.486 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.486 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:56.486 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569257+0000", 2024-09-21T12:22:56.486 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3480000000000001e-06 2024-09-21T12:22:56.486 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.486 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.486 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:56.487 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569332+0000", 2024-09-21T12:22:56.487 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4969000000000007e-05 2024-09-21T12:22:56.487 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.487 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.487 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:56.487 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569344+0000", 2024-09-21T12:22:56.487 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2700999999999999e-05 2024-09-21T12:22:56.487 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.487 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.487 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:56.487 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569351+0000", 2024-09-21T12:22:56.487 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0369999999999997e-06 2024-09-21T12:22:56.487 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.487 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.488 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:56.488 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569427+0000", 2024-09-21T12:22:56.488 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6206999999999997e-05 2024-09-21T12:22:56.488 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.488 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.488 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:56.488 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569517+0000", 2024-09-21T12:22:56.488 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9875000000000003e-05 2024-09-21T12:22:56.488 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.488 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.488 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1", 2024-09-21T12:22:56.488 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569601+0000", 2024-09-21T12:22:56.488 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3849000000000003e-05 2024-09-21T12:22:56.488 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.488 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.488 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:22:56.489 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.573956+0000", 2024-09-21T12:22:56.489 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0043544689999999997 2024-09-21T12:22:56.489 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.489 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.489 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.489 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.575786+0000", 2024-09-21T12:22:56.489 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0018307480000000001 2024-09-21T12:22:56.489 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.489 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.489 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.489 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.717016+0000", 2024-09-21T12:22:56.489 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14122955000000001 2024-09-21T12:22:56.489 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.490 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.490 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:56.490 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.717072+0000", 2024-09-21T12:22:56.490 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6063000000000002e-05 2024-09-21T12:22:56.490 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.490 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.490 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:56.490 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.717122+0000", 2024-09-21T12:22:56.490 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9654000000000003e-05 2024-09-21T12:22:56.490 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.490 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:56.490 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.490 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.490 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.490 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:49229 7.d 7:b095dac2:::benchmark_data_smithi114_78432_object49228:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:22:56.491 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:04.569388+0000", 2024-09-21T12:22:56.491 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 51.871155543, 2024-09-21T12:22:56.491 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.147855664, 2024-09-21T12:22:56.491 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:56.491 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:56.491 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:56.491 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:22:56.491 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:22:56.491 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:56.491 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 49229 2024-09-21T12:22:56.491 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.491 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:56.491 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.491 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:56.491 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569388+0000", 2024-09-21T12:22:56.491 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.492 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.492 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.492 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:56.492 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569388+0000", 2024-09-21T12:22:56.492 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.492 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.492 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.492 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:56.492 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569388+0000", 2024-09-21T12:22:56.492 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.37e-07 2024-09-21T12:22:56.492 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.492 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.492 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:56.492 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569462+0000", 2024-09-21T12:22:56.492 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3244000000000005e-05 2024-09-21T12:22:56.492 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.493 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.493 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:56.493 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569471+0000", 2024-09-21T12:22:56.493 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2499999999999995e-06 2024-09-21T12:22:56.493 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.493 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.493 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:56.493 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569477+0000", 2024-09-21T12:22:56.493 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.494e-06 2024-09-21T12:22:56.493 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.493 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.493 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:56.493 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.572570+0000", 2024-09-21T12:22:56.493 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0030923930000000001 2024-09-21T12:22:56.493 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.493 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.494 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:56.494 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.572664+0000", 2024-09-21T12:22:56.494 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4609000000000002e-05 2024-09-21T12:22:56.494 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.494 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.494 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1", 2024-09-21T12:22:56.494 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.572751+0000", 2024-09-21T12:22:56.494 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6725999999999996e-05 2024-09-21T12:22:56.494 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.494 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.494 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:22:56.494 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.574094+0000", 2024-09-21T12:22:56.494 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00134326 2024-09-21T12:22:56.494 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.494 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.495 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.495 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.576231+0000", 2024-09-21T12:22:56.495 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021369510000000002 2024-09-21T12:22:56.495 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.495 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.495 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.495 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.717166+0000", 2024-09-21T12:22:56.495 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14093446000000001 2024-09-21T12:22:56.495 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.495 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.495 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:56.495 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.717210+0000", 2024-09-21T12:22:56.495 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4239000000000001e-05 2024-09-21T12:22:56.495 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.495 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.496 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:56.497 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:56.497 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.717243+0000", 2024-09-21T12:22:56.497 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3401000000000001e-05 2024-09-21T12:22:56.497 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.497 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:56.497 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.497 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.498 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.498 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:49936 7.7 7:ee2fb7d7:::benchmark_data_smithi114_78432_object49935:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:22:56.498 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:05.207697+0000", 2024-09-21T12:22:56.498 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 51.232846672000001, 2024-09-21T12:22:56.498 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.180234797, 2024-09-21T12:22:56.498 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:56.498 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:56.498 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:56.498 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:22:56.498 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:22:56.498 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:56.498 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 49936 2024-09-21T12:22:56.498 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.498 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:56.499 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.499 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:56.499 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207697+0000", 2024-09-21T12:22:56.499 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.499 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.499 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.499 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:56.499 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207697+0000", 2024-09-21T12:22:56.499 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.499 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.499 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.499 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:56.499 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207698+0000", 2024-09-21T12:22:56.499 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6449999999999999e-06 2024-09-21T12:22:56.499 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.500 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.500 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:56.500 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207744+0000", 2024-09-21T12:22:56.500 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6171999999999998e-05 2024-09-21T12:22:56.500 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.500 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.500 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:56.500 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207755+0000", 2024-09-21T12:22:56.500 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0998999999999999e-05 2024-09-21T12:22:56.500 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.500 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.500 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:56.500 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207761+0000", 2024-09-21T12:22:56.500 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9789999999999998e-06 2024-09-21T12:22:56.500 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.501 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.501 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:56.501 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207822+0000", 2024-09-21T12:22:56.501 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0504000000000002e-05 2024-09-21T12:22:56.501 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.501 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.501 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:56.501 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207921+0000", 2024-09-21T12:22:56.501 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9524000000000005e-05 2024-09-21T12:22:56.501 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.501 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.501 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,3", 2024-09-21T12:22:56.501 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.208068+0000", 2024-09-21T12:22:56.502 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014648299999999999 2024-09-21T12:22:56.502 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.502 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.502 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:22:56.502 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.209637+0000", 2024-09-21T12:22:56.502 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001569555 2024-09-21T12:22:56.502 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.502 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.502 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.502 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.214355+0000", 2024-09-21T12:22:56.502 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0047178810000000002 2024-09-21T12:22:56.502 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.502 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.503 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.503 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.387868+0000", 2024-09-21T12:22:56.503 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.173512203 2024-09-21T12:22:56.503 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.503 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.503 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:56.503 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.387905+0000", 2024-09-21T12:22:56.503 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7678000000000001e-05 2024-09-21T12:22:56.503 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.503 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.503 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:56.503 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.387931+0000", 2024-09-21T12:22:56.503 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6174e-05 2024-09-21T12:22:56.504 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.504 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:56.504 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.504 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.504 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.504 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:49949 7.d 7:b8e0b394:::benchmark_data_smithi114_78432_object49948:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:22:56.504 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:05.215908+0000", 2024-09-21T12:22:56.504 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 51.224634954999999, 2024-09-21T12:22:56.504 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.17212050400000001, 2024-09-21T12:22:56.504 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:56.504 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:56.504 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:56.504 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:22:56.504 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:22:56.505 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:56.505 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 49949 2024-09-21T12:22:56.505 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.505 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:56.505 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.505 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:56.505 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215908+0000", 2024-09-21T12:22:56.505 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.505 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.505 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.505 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:56.505 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215908+0000", 2024-09-21T12:22:56.506 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.506 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.506 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.506 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:56.506 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215910+0000", 2024-09-21T12:22:56.506 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.389e-06 2024-09-21T12:22:56.506 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.506 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.506 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:56.506 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215966+0000", 2024-09-21T12:22:56.506 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6189e-05 2024-09-21T12:22:56.506 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.507 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.507 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:56.507 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215976+0000", 2024-09-21T12:22:56.507 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0115999999999999e-05 2024-09-21T12:22:56.507 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.507 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.507 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:56.507 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215983+0000", 2024-09-21T12:22:56.507 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9240000000000001e-06 2024-09-21T12:22:56.507 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.507 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.507 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:56.507 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.216032+0000", 2024-09-21T12:22:56.508 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9237000000000003e-05 2024-09-21T12:22:56.508 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.508 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.508 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:56.508 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.216109+0000", 2024-09-21T12:22:56.508 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6538999999999993e-05 2024-09-21T12:22:56.508 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.508 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.508 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1", 2024-09-21T12:22:56.508 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.216208+0000", 2024-09-21T12:22:56.508 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8979000000000002e-05 2024-09-21T12:22:56.508 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.508 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.508 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:22:56.509 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217649+0000", 2024-09-21T12:22:56.509 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014417500000000001 2024-09-21T12:22:56.509 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.509 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.509 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.509 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.219275+0000", 2024-09-21T12:22:56.509 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0016254749999999999 2024-09-21T12:22:56.509 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.509 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.509 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.509 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.387917+0000", 2024-09-21T12:22:56.509 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.16864237100000001 2024-09-21T12:22:56.509 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.509 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.509 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:56.510 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.387990+0000", 2024-09-21T12:22:56.510 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2249999999999994e-05 2024-09-21T12:22:56.510 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.510 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.510 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:56.510 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.388029+0000", 2024-09-21T12:22:56.510 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9285000000000001e-05 2024-09-21T12:22:56.510 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.510 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:56.510 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.510 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.510 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.510 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:49953 7.7 7:e1a3875f:::benchmark_data_smithi114_78432_object49952:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:22:56.510 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:05.217615+0000", 2024-09-21T12:22:56.510 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 51.222928033999999, 2024-09-21T12:22:56.511 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.17616359000000001, 2024-09-21T12:22:56.511 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:56.511 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:56.511 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:56.511 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:22:56.511 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:22:56.511 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:56.511 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 49953 2024-09-21T12:22:56.511 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.511 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:56.511 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.511 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:56.511 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217615+0000", 2024-09-21T12:22:56.511 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.512 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.512 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.512 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:56.512 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217615+0000", 2024-09-21T12:22:56.512 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.512 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.512 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.512 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:56.512 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217616+0000", 2024-09-21T12:22:56.512 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.209e-06 2024-09-21T12:22:56.512 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.512 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.512 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:56.512 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217687+0000", 2024-09-21T12:22:56.512 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0570999999999999e-05 2024-09-21T12:22:56.513 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.513 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.513 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:56.513 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217701+0000", 2024-09-21T12:22:56.513 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3828e-05 2024-09-21T12:22:56.513 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.513 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.513 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:56.513 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217709+0000", 2024-09-21T12:22:56.513 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3240000000000003e-06 2024-09-21T12:22:56.513 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.513 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.513 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:56.514 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217763+0000", 2024-09-21T12:22:56.514 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3804999999999997e-05 2024-09-21T12:22:56.514 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.514 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.514 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:56.514 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217854+0000", 2024-09-21T12:22:56.514 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0771000000000002e-05 2024-09-21T12:22:56.514 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.514 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.514 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,3", 2024-09-21T12:22:56.514 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217935+0000", 2024-09-21T12:22:56.514 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0951000000000005e-05 2024-09-21T12:22:56.514 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.514 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.514 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:22:56.515 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.219237+0000", 2024-09-21T12:22:56.515 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0013019679999999999 2024-09-21T12:22:56.515 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.515 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.515 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.515 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.220275+0000", 2024-09-21T12:22:56.515 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010378609999999999 2024-09-21T12:22:56.515 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.515 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.515 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.515 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.393564+0000", 2024-09-21T12:22:56.515 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.17328943899999999 2024-09-21T12:22:56.515 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.515 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.515 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:56.515 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.393734+0000", 2024-09-21T12:22:56.516 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017039999999999999 2024-09-21T12:22:56.516 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.516 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.516 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:56.516 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.393779+0000", 2024-09-21T12:22:56.516 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4462999999999998e-05 2024-09-21T12:22:56.516 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.516 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:56.516 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.516 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.516 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.516 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:49961 7.7 7:e82f0c80:::benchmark_data_smithi114_78432_object49960:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:22:56.516 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:05.226839+0000", 2024-09-21T12:22:56.516 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 51.213704038000003, 2024-09-21T12:22:56.516 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.170311145, 2024-09-21T12:22:56.517 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:56.517 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:56.517 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:56.517 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:22:56.517 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:22:56.517 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:56.517 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 49961 2024-09-21T12:22:56.517 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.517 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:56.517 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.517 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:56.517 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.226839+0000", 2024-09-21T12:22:56.517 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.517 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.517 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.517 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:56.518 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.226839+0000", 2024-09-21T12:22:56.518 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.518 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.518 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.518 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:56.518 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.226840+0000", 2024-09-21T12:22:56.518 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8400000000000003e-07 2024-09-21T12:22:56.518 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.518 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.518 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:56.518 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.226896+0000", 2024-09-21T12:22:56.518 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5517999999999999e-05 2024-09-21T12:22:56.518 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.518 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.518 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:56.519 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.226906+0000", 2024-09-21T12:22:56.519 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0589e-05 2024-09-21T12:22:56.519 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.519 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.519 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:56.519 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.226914+0000", 2024-09-21T12:22:56.519 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2779999999999996e-06 2024-09-21T12:22:56.519 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.519 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.519 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:56.519 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.226963+0000", 2024-09-21T12:22:56.519 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9021999999999999e-05 2024-09-21T12:22:56.519 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.519 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.519 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:56.520 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.227045+0000", 2024-09-21T12:22:56.520 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2800999999999995e-05 2024-09-21T12:22:56.520 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.520 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.520 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,3", 2024-09-21T12:22:56.520 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.227126+0000", 2024-09-21T12:22:56.520 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0467000000000001e-05 2024-09-21T12:22:56.520 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.520 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.520 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:22:56.520 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.228347+0000", 2024-09-21T12:22:56.520 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001220907 2024-09-21T12:22:56.520 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.520 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.521 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.521 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.393832+0000", 2024-09-21T12:22:56.521 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.165485038 2024-09-21T12:22:56.521 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.521 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.521 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.521 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.397086+0000", 2024-09-21T12:22:56.521 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0032546799999999998 2024-09-21T12:22:56.521 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.521 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.521 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:56.521 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.397121+0000", 2024-09-21T12:22:56.521 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4584e-05 2024-09-21T12:22:56.521 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.521 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.522 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:56.522 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.397150+0000", 2024-09-21T12:22:56.522 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9377e-05 2024-09-21T12:22:56.522 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.522 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:56.522 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.522 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.522 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.522 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:52202 7.7 7:e71a4b9c:::benchmark_data_smithi114_78432_object52201:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:22:56.522 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:06.968679+0000", 2024-09-21T12:22:56.522 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 49.471863894000002, 2024-09-21T12:22:56.522 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.11371431799999999, 2024-09-21T12:22:56.522 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:56.523 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:56.523 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:56.523 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:22:56.523 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:22:56.523 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:56.523 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 52202 2024-09-21T12:22:56.523 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.523 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:56.523 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.523 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:56.523 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.968679+0000", 2024-09-21T12:22:56.523 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.523 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.523 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.524 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:56.524 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.968679+0000", 2024-09-21T12:22:56.524 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.524 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.524 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.524 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:56.524 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.968681+0000", 2024-09-21T12:22:56.524 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3090000000000001e-06 2024-09-21T12:22:56.524 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.524 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.524 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:56.524 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.968707+0000", 2024-09-21T12:22:56.524 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6582999999999999e-05 2024-09-21T12:22:56.524 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.524 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.525 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:56.525 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.968719+0000", 2024-09-21T12:22:56.525 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1637000000000001e-05 2024-09-21T12:22:56.525 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.525 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.525 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:56.525 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.968724+0000", 2024-09-21T12:22:56.525 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5770000000000001e-06 2024-09-21T12:22:56.525 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.525 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.525 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:56.525 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.968771+0000", 2024-09-21T12:22:56.525 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6641000000000003e-05 2024-09-21T12:22:56.526 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.526 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.526 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:56.526 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.968861+0000", 2024-09-21T12:22:56.526 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9710999999999995e-05 2024-09-21T12:22:56.526 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.526 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.526 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,3", 2024-09-21T12:22:56.526 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.969036+0000", 2024-09-21T12:22:56.526 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017536 2024-09-21T12:22:56.526 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.526 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.526 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:22:56.527 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.970822+0000", 2024-09-21T12:22:56.527 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001785476 2024-09-21T12:22:56.527 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.527 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.527 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.527 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.977126+0000", 2024-09-21T12:22:56.527 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.006304736 2024-09-21T12:22:56.527 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.527 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.527 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.527 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.082331+0000", 2024-09-21T12:22:56.527 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.105204697 2024-09-21T12:22:56.527 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.527 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.527 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:56.528 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.082367+0000", 2024-09-21T12:22:56.528 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5639999999999998e-05 2024-09-21T12:22:56.528 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.528 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.528 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:56.528 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.082394+0000", 2024-09-21T12:22:56.528 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6951e-05 2024-09-21T12:22:56.528 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.528 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:56.528 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.528 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.528 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.528 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:53121 7.d 7:b55e55a7:::benchmark_data_smithi114_78432_object53120:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:22:56.528 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:07.710802+0000", 2024-09-21T12:22:56.528 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 48.729741140999998, 2024-09-21T12:22:56.529 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.12673653600000001, 2024-09-21T12:22:56.529 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:56.529 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:56.529 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:56.529 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:22:56.529 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:22:56.529 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:56.529 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 53121 2024-09-21T12:22:56.529 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.529 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:56.529 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.529 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:56.529 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.710802+0000", 2024-09-21T12:22:56.529 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.530 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.530 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.530 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:56.530 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.710802+0000", 2024-09-21T12:22:56.530 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.530 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.530 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.530 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:56.530 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.710803+0000", 2024-09-21T12:22:56.530 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3e-07 2024-09-21T12:22:56.530 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.530 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.530 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:56.530 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.710848+0000", 2024-09-21T12:22:56.530 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5630000000000002e-05 2024-09-21T12:22:56.531 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.531 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.531 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:56.531 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.710857+0000", 2024-09-21T12:22:56.531 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2269999999999993e-06 2024-09-21T12:22:56.531 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.531 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.531 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:56.531 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.710863+0000", 2024-09-21T12:22:56.531 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2639999999999997e-06 2024-09-21T12:22:56.531 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.531 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.531 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:56.531 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.710944+0000", 2024-09-21T12:22:56.532 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1353999999999994e-05 2024-09-21T12:22:56.532 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.532 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.532 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:56.532 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.711207+0000", 2024-09-21T12:22:56.532 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00026285399999999997 2024-09-21T12:22:56.532 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.532 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.532 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1", 2024-09-21T12:22:56.532 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.711337+0000", 2024-09-21T12:22:56.532 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000129827 2024-09-21T12:22:56.532 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.532 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.532 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:22:56.532 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.714512+0000", 2024-09-21T12:22:56.533 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0031746750000000001 2024-09-21T12:22:56.533 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.533 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.533 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.533 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.716416+0000", 2024-09-21T12:22:56.533 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0019048299999999999 2024-09-21T12:22:56.533 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.533 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.533 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.533 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.837462+0000", 2024-09-21T12:22:56.533 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.121045347 2024-09-21T12:22:56.533 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.533 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.533 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:56.533 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.837500+0000", 2024-09-21T12:22:56.534 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8059000000000003e-05 2024-09-21T12:22:56.534 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.534 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.534 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:56.534 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.837539+0000", 2024-09-21T12:22:56.534 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8838999999999999e-05 2024-09-21T12:22:56.534 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.534 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:56.534 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.534 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.534 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.534 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:53122 7.d 7:b6d0f6a6:::benchmark_data_smithi114_78432_object53121:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:22:56.534 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:07.710882+0000", 2024-09-21T12:22:56.535 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 48.729661182999997, 2024-09-21T12:22:56.535 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.126757757, 2024-09-21T12:22:56.535 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:56.535 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:56.535 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:56.535 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:22:56.535 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:22:56.535 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:56.535 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 53122 2024-09-21T12:22:56.535 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.535 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:56.535 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.535 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:56.535 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.710882+0000", 2024-09-21T12:22:56.535 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.536 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.536 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.536 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:56.536 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.710882+0000", 2024-09-21T12:22:56.536 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.536 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.536 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.536 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:56.536 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.710883+0000", 2024-09-21T12:22:56.536 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8500000000000001e-07 2024-09-21T12:22:56.536 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.536 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.536 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:56.536 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.710929+0000", 2024-09-21T12:22:56.537 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5867e-05 2024-09-21T12:22:56.537 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.537 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.537 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:56.537 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.710937+0000", 2024-09-21T12:22:56.537 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.619e-06 2024-09-21T12:22:56.537 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.537 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.537 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:56.537 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.710945+0000", 2024-09-21T12:22:56.537 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6920000000000002e-06 2024-09-21T12:22:56.537 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.537 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.537 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:56.538 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.713008+0000", 2024-09-21T12:22:56.538 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002063442 2024-09-21T12:22:56.538 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.538 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.538 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:56.538 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.713116+0000", 2024-09-21T12:22:56.538 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000107886 2024-09-21T12:22:56.538 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.538 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.538 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1", 2024-09-21T12:22:56.538 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.713198+0000", 2024-09-21T12:22:56.538 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1842999999999996e-05 2024-09-21T12:22:56.538 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.539 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.539 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:22:56.539 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.714595+0000", 2024-09-21T12:22:56.539 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0013966779999999999 2024-09-21T12:22:56.539 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.539 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.539 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.539 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.716453+0000", 2024-09-21T12:22:56.539 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0018580230000000001 2024-09-21T12:22:56.539 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.539 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.539 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.539 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.837570+0000", 2024-09-21T12:22:56.539 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.121117483 2024-09-21T12:22:56.540 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.540 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.540 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:56.540 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.837606+0000", 2024-09-21T12:22:56.540 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5225000000000002e-05 2024-09-21T12:22:56.540 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.540 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.540 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:56.540 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.837640+0000", 2024-09-21T12:22:56.540 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4313999999999997e-05 2024-09-21T12:22:56.540 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.540 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:56.540 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.540 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.541 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.541 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:53140 7.d 7:b58871fb:::benchmark_data_smithi114_78432_object53139:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:22:56.541 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:07.725303+0000", 2024-09-21T12:22:56.541 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 48.715240641999998, 2024-09-21T12:22:56.541 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.114967558, 2024-09-21T12:22:56.541 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:56.541 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:56.541 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:56.541 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:22:56.541 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:22:56.541 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:56.541 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 53140 2024-09-21T12:22:56.541 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.541 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:56.542 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.542 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:56.542 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.725303+0000", 2024-09-21T12:22:56.542 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.542 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.542 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.542 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:56.542 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.725303+0000", 2024-09-21T12:22:56.542 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.542 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.542 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.542 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:56.542 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.725304+0000", 2024-09-21T12:22:56.543 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.85e-07 2024-09-21T12:22:56.543 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.543 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.543 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:56.543 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.725371+0000", 2024-09-21T12:22:56.543 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7945000000000001e-05 2024-09-21T12:22:56.543 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.543 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.543 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:56.543 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.725380+0000", 2024-09-21T12:22:56.543 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5229999999999995e-06 2024-09-21T12:22:56.543 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.543 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.543 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:56.543 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.725389+0000", 2024-09-21T12:22:56.544 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4190000000000008e-06 2024-09-21T12:22:56.544 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.544 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.544 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:56.544 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.725437+0000", 2024-09-21T12:22:56.544 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7429000000000003e-05 2024-09-21T12:22:56.544 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.544 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.544 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:56.544 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.725520+0000", 2024-09-21T12:22:56.544 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3072999999999996e-05 2024-09-21T12:22:56.544 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.544 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.544 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1", 2024-09-21T12:22:56.544 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.725602+0000", 2024-09-21T12:22:56.545 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1823000000000006e-05 2024-09-21T12:22:56.545 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.545 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.545 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:22:56.545 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.726927+0000", 2024-09-21T12:22:56.545 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0013257329999999999 2024-09-21T12:22:56.545 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.545 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.545 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.545 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.727549+0000", 2024-09-21T12:22:56.545 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00062129199999999996 2024-09-21T12:22:56.545 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.545 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.545 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.545 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.840175+0000", 2024-09-21T12:22:56.545 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.112626243 2024-09-21T12:22:56.546 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.546 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.546 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:56.546 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.840233+0000", 2024-09-21T12:22:56.546 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7822999999999998e-05 2024-09-21T12:22:56.546 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.546 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.546 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:56.546 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.840270+0000", 2024-09-21T12:22:56.546 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7370000000000003e-05 2024-09-21T12:22:56.546 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.546 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:56.546 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.547 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.547 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.547 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:97938 7.c 7:3c17c87d:::benchmark_data_smithi114_78432_object97937:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-21T12:22:56.547 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:54.522207+0000", 2024-09-21T12:22:56.547 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.918336204, 2024-09-21T12:22:56.547 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.11778506599999999, 2024-09-21T12:22:56.547 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:56.547 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:56.547 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:56.547 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:22:56.547 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:22:56.547 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:56.547 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 97938 2024-09-21T12:22:56.547 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.547 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:56.548 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.548 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:56.548 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.522207+0000", 2024-09-21T12:22:56.548 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.548 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.548 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.548 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:56.548 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.522207+0000", 2024-09-21T12:22:56.548 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.548 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.548 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.548 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:56.548 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.522218+0000", 2024-09-21T12:22:56.548 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0737000000000001e-05 2024-09-21T12:22:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.522292+0000", 2024-09-21T12:22:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4431000000000006e-05 2024-09-21T12:22:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.522366+0000", 2024-09-21T12:22:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3817000000000002e-05 2024-09-21T12:22:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.522433+0000", 2024-09-21T12:22:56.549 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6839000000000002e-05 2024-09-21T12:22:56.550 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.550 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.550 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:56.550 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.526487+0000", 2024-09-21T12:22:56.550 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0040536909999999999 2024-09-21T12:22:56.551 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.551 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.551 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:56.552 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.526601+0000", 2024-09-21T12:22:56.552 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000114646 2024-09-21T12:22:56.552 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.552 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.552 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:22:56.552 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.526732+0000", 2024-09-21T12:22:56.552 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000130538 2024-09-21T12:22:56.552 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.552 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.552 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.552 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.534059+0000", 2024-09-21T12:22:56.552 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0073272729999999996 2024-09-21T12:22:56.552 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.552 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.553 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:22:56.553 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.534217+0000", 2024-09-21T12:22:56.553 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000157787 2024-09-21T12:22:56.553 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.553 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.553 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.553 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.639803+0000", 2024-09-21T12:22:56.553 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.105585767 2024-09-21T12:22:56.553 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.553 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.553 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:56.553 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.639944+0000", 2024-09-21T12:22:56.553 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000141366 2024-09-21T12:22:56.553 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.553 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.554 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:56.554 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.639992+0000", 2024-09-21T12:22:56.554 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8174000000000003e-05 2024-09-21T12:22:56.554 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.554 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:56.554 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.554 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.554 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.554 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:97943 7.c 7:32777e6f:::benchmark_data_smithi114_78432_object97942:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-21T12:22:56.554 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:54.525666+0000", 2024-09-21T12:22:56.554 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.914877114, 2024-09-21T12:22:56.554 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.11443463099999999, 2024-09-21T12:22:56.554 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:56.555 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:56.555 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:56.555 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:22:56.555 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:22:56.555 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:56.555 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 97943 2024-09-21T12:22:56.555 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.555 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:56.555 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.555 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:56.555 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.525666+0000", 2024-09-21T12:22:56.555 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.555 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.555 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.555 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:56.556 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.525666+0000", 2024-09-21T12:22:56.556 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.556 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.556 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.556 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:56.556 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.525668+0000", 2024-09-21T12:22:56.556 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.728e-06 2024-09-21T12:22:56.556 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.556 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.556 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:56.556 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.525700+0000", 2024-09-21T12:22:56.556 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2007e-05 2024-09-21T12:22:56.556 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.556 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.556 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:56.556 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.525714+0000", 2024-09-21T12:22:56.556 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4231e-05 2024-09-21T12:22:56.557 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.557 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.557 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:56.557 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.525722+0000", 2024-09-21T12:22:56.557 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2719999999999997e-06 2024-09-21T12:22:56.557 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.557 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.557 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:56.557 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.528993+0000", 2024-09-21T12:22:56.557 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0032704399999999999 2024-09-21T12:22:56.557 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.557 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.557 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:56.557 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.529122+0000", 2024-09-21T12:22:56.557 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012888999999999999 2024-09-21T12:22:56.558 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.558 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.558 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:22:56.558 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.529291+0000", 2024-09-21T12:22:56.558 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000169252 2024-09-21T12:22:56.558 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.558 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.558 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:22:56.558 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.534223+0000", 2024-09-21T12:22:56.558 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0049318280000000001 2024-09-21T12:22:56.558 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.558 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.558 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.559 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.534864+0000", 2024-09-21T12:22:56.559 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00064124199999999998 2024-09-21T12:22:56.559 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.559 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.559 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.559 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.640044+0000", 2024-09-21T12:22:56.559 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.105179833 2024-09-21T12:22:56.559 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.559 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.559 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:56.559 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.640077+0000", 2024-09-21T12:22:56.559 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3540000000000001e-05 2024-09-21T12:22:56.559 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.559 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.560 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:56.560 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.640101+0000", 2024-09-21T12:22:56.560 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3368000000000001e-05 2024-09-21T12:22:56.560 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.560 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:56.560 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.560 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.560 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.560 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:97959 7.7 7:ef75ea0c:::benchmark_data_smithi114_78432_object97958:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-21T12:22:56.560 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:54.537270+0000", 2024-09-21T12:22:56.560 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.9032731890000001, 2024-09-21T12:22:56.560 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.116963444, 2024-09-21T12:22:56.560 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:56.560 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:56.560 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:22:56.560 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:22:56.561 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:22:56.561 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:22:56.561 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 97959 2024-09-21T12:22:56.561 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.561 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:56.561 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.561 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:56.561 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.537270+0000", 2024-09-21T12:22:56.561 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.561 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.561 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.561 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:56.561 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.537270+0000", 2024-09-21T12:22:56.561 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:56.561 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.561 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.562 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:56.562 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.537271+0000", 2024-09-21T12:22:56.562 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.311e-06 2024-09-21T12:22:56.562 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.562 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.562 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:56.562 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.537320+0000", 2024-09-21T12:22:56.562 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8365999999999997e-05 2024-09-21T12:22:56.562 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.562 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.562 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:56.562 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.537330+0000", 2024-09-21T12:22:56.562 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0294e-05 2024-09-21T12:22:56.563 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.563 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.563 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:56.563 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.537336+0000", 2024-09-21T12:22:56.563 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5060000000000003e-06 2024-09-21T12:22:56.563 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.563 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.563 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:56.563 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.537381+0000", 2024-09-21T12:22:56.563 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5698000000000002e-05 2024-09-21T12:22:56.563 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.563 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.563 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:56.563 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.537473+0000", 2024-09-21T12:22:56.563 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2173e-05 2024-09-21T12:22:56.563 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.564 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.564 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,3", 2024-09-21T12:22:56.564 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.537570+0000", 2024-09-21T12:22:56.564 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6144999999999995e-05 2024-09-21T12:22:56.564 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.564 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.564 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:22:56.564 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.539566+0000", 2024-09-21T12:22:56.564 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0019962249999999999 2024-09-21T12:22:56.564 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.564 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.564 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.564 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.540245+0000", 2024-09-21T12:22:56.564 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00067934899999999999 2024-09-21T12:22:56.564 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.565 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.565 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:22:56.565 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.654137+0000", 2024-09-21T12:22:56.565 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.113891699 2024-09-21T12:22:56.565 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.565 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.565 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:22:56.565 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.654191+0000", 2024-09-21T12:22:56.565 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3977000000000002e-05 2024-09-21T12:22:56.565 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:56.565 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:56.565 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:22:56.565 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:54.654234+0000", 2024-09-21T12:22:56.565 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2701000000000003e-05 2024-09-21T12:22:56.565 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.566 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:56.566 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.566 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:56.566 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:56.566 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:22:56.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:56.567 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:56.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:56.641 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 86 16 99802 99786 72.5 56.4375 0.00928179 0.0137872 2024-09-21T12:22:56.667 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:56.706 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:56.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:56.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:56.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:22:56 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:22:56.807 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:56.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:56.848 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:56.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:56.949 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:56.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:56.977 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:56.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:57.078 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:57.103 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:57.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:57.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:57.157 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:22:57.172 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:22:57.204 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:57.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:56 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:22:57.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:56 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:22:57.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:56 smithi183 ceph-mon[33277]: pgmap v115: 158 pgs: 1 active+recovering+undersized+remapped, 1 peering, 156 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 31 MiB/s wr, 492 op/s; 1294/285747 objects misplaced (0.453%); 10 MiB/s, 178 objects/s recovering 2024-09-21T12:22:57.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:56 smithi183 ceph-mon[33277]: osdmap e123: 8 total, 8 up, 7 in 2024-09-21T12:22:57.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:56 smithi183 ceph-mon[33277]: osdmap e124: 8 total, 8 up, 7 in 2024-09-21T12:22:57.239 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:22:56 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=infra.usagestats t=2024-09-21T12:22:56.966496243Z level=info msg="Usage stats are ready to report" 2024-09-21T12:22:57.249 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:57.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:57.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:57.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:56 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:22:57.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:56 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:22:57.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:56 smithi114 ceph-mon[40867]: pgmap v115: 158 pgs: 1 active+recovering+undersized+remapped, 1 peering, 156 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 31 MiB/s wr, 492 op/s; 1294/285747 objects misplaced (0.453%); 10 MiB/s, 178 objects/s recovering 2024-09-21T12:22:57.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:56 smithi114 ceph-mon[40867]: osdmap e123: 8 total, 8 up, 7 in 2024-09-21T12:22:57.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:56 smithi114 ceph-mon[40867]: osdmap e124: 8 total, 8 up, 7 in 2024-09-21T12:22:57.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:56 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:22:57.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:56 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:22:57.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:56 smithi114 ceph-mon[30324]: pgmap v115: 158 pgs: 1 active+recovering+undersized+remapped, 1 peering, 156 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 31 MiB/s wr, 492 op/s; 1294/285747 objects misplaced (0.453%); 10 MiB/s, 178 objects/s recovering 2024-09-21T12:22:57.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:56 smithi114 ceph-mon[30324]: osdmap e123: 8 total, 8 up, 7 in 2024-09-21T12:22:57.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:56 smithi114 ceph-mon[30324]: osdmap e124: 8 total, 8 up, 7 in 2024-09-21T12:22:57.351 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:57.355 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T12:22:57.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:57.376 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:57.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:57.478 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:57.519 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:57.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:57.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:57.621 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:57.641 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 87 16 100819 100803 72.3973 63.5625 0.0090174 0.0138079 2024-09-21T12:22:57.651 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:57.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:57.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:57.702 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:22:57.753 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:57.779 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:57.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:57.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:57.881 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:22:57.907 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:57.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:57.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:22:57.965 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-21T12:22:58.009 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:58.018 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:22:58.018 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:22:58.038 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:58.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:58.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:58.081 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:22:58.081 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:22:58.122 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:22:58.122 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:22:58.140 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:22:58.168 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:57 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:22:58.168 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:57 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1071914905' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:22:58.168 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:57 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:22:58.168 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:57 smithi183 ceph-mon[33277]: 6.4 scrub starts 2024-09-21T12:22:58.168 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:57 smithi183 ceph-mon[33277]: 6.4 scrub ok 2024-09-21T12:22:58.170 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:22:58.170 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:22:58.170 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:57 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:22:58.170 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:57 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1071914905' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:22:58.170 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:57 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:22:58.170 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:57 smithi114 ceph-mon[40867]: 6.4 scrub starts 2024-09-21T12:22:58.170 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:57 smithi114 ceph-mon[40867]: 6.4 scrub ok 2024-09-21T12:22:58.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:57 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:22:58.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:57 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1071914905' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:22:58.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:57 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:22:58.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:57 smithi114 ceph-mon[30324]: 6.4 scrub starts 2024-09-21T12:22:58.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:57 smithi114 ceph-mon[30324]: 6.4 scrub ok 2024-09-21T12:22:58.183 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:58.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:58.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:22:58.209 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:22:58.209 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:22:58.251 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:22:58.251 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:22:58.285 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:58.290 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:22:58.290 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:22:58.291 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:22:58.291 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:22:58.291 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:58.291 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:101475 7.6 e125/84 7:6806897f:::benchmark_data_smithi114_78432_object101474:head v 125'6256, mlcod=125'6254)", 2024-09-21T12:22:58.291 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:58.287030+0000", 2024-09-21T12:22:58.291 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.002281206, 2024-09-21T12:22:58.291 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00230847, 2024-09-21T12:22:58.291 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:58.291 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:58.291 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:22:58.291 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:58.292 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:58.292 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:58.292 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:58.287030+0000", 2024-09-21T12:22:58.292 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:58.292 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:58.292 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:58.292 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:58.292 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:58.287030+0000", 2024-09-21T12:22:58.292 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:58.292 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:58.292 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:58.292 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:58.292 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:58.287031+0000", 2024-09-21T12:22:58.293 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1480000000000001e-06 2024-09-21T12:22:58.293 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:58.293 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:58.293 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:58.293 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:58.287080+0000", 2024-09-21T12:22:58.293 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8634000000000002e-05 2024-09-21T12:22:58.293 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:58.293 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:58.293 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:58.293 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:58.287092+0000", 2024-09-21T12:22:58.293 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2396e-05 2024-09-21T12:22:58.293 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:58.293 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:58.293 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:58.293 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:58.287099+0000", 2024-09-21T12:22:58.293 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5209999999999999e-06 2024-09-21T12:22:58.294 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:58.294 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:58.294 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:58.294 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:58.287151+0000", 2024-09-21T12:22:58.294 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2568000000000002e-05 2024-09-21T12:22:58.294 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:58.294 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:58.294 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:58.294 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:58.287172+0000", 2024-09-21T12:22:58.294 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1267000000000002e-05 2024-09-21T12:22:58.294 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:58.294 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:58.294 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:58.294 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:58.294 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:58.295 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:101476 7.8 e125/120 7:19acc498:::benchmark_data_smithi114_78432_object101475:head v 125'6448, mlcod=114'5053)", 2024-09-21T12:22:58.295 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:58.287519+0000", 2024-09-21T12:22:58.295 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.001792166, 2024-09-21T12:22:58.295 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0018732029999999999, 2024-09-21T12:22:58.295 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:22:58.295 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:22:58.295 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:22:58.295 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:22:58.295 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:58.295 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:22:58.295 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:58.287519+0000", 2024-09-21T12:22:58.295 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:58.295 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:58.295 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:58.295 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:22:58.296 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:58.287519+0000", 2024-09-21T12:22:58.296 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:22:58.296 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:58.296 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:58.296 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:22:58.296 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:58.287520+0000", 2024-09-21T12:22:58.296 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9100000000000003e-07 2024-09-21T12:22:58.296 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:58.296 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:58.296 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:22:58.296 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:58.287588+0000", 2024-09-21T12:22:58.296 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8273000000000002e-05 2024-09-21T12:22:58.296 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:58.296 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:58.296 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:22:58.296 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:58.287600+0000", 2024-09-21T12:22:58.296 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2393e-05 2024-09-21T12:22:58.297 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:58.297 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:58.297 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:22:58.297 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:58.287607+0000", 2024-09-21T12:22:58.297 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.657e-06 2024-09-21T12:22:58.297 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:58.297 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:58.297 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:22:58.297 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:58.287671+0000", 2024-09-21T12:22:58.297 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3808999999999998e-05 2024-09-21T12:22:58.297 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:22:58.297 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:22:58.297 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:22:58.297 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:58.287697+0000", 2024-09-21T12:22:58.297 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6588e-05 2024-09-21T12:22:58.297 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:58.298 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:22:58.298 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:58.298 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:22:58.298 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-21T12:22:58.298 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 2 2024-09-21T12:22:58.298 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:22:58.316 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:58.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:58.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:58.327 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:22:58.327 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:22:58.418 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:58.444 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:58.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:58.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:58.546 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:58.569 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:58.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:58.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:58.641 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 88 16 101825 101809 72.289 62.875 0.0181232 0.0138284 2024-09-21T12:22:58.670 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:58.677 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:22:58.677 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:22:58.677 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 125, 2024-09-21T12:22:58.677 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "e2ad975c-7812-11ef-baf6-efdc52797490", 2024-09-21T12:22:58.677 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-21T12:14:24.181227+0000", 2024-09-21T12:22:58.677 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-21T12:22:57.934494+0000", 2024-09-21T12:22:58.677 INFO:teuthology.orchestra.run.smithi114.stdout: "last_up_change": "2024-09-21T12:20:03.702987+0000", 2024-09-21T12:22:58.677 INFO:teuthology.orchestra.run.smithi114.stdout: "last_in_change": "2024-09-21T12:22:49.868269+0000", 2024-09-21T12:22:58.677 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T12:22:58.678 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_num": 5801984, 2024-09-21T12:22:58.678 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_set": [ 2024-09-21T12:22:58.678 INFO:teuthology.orchestra.run.smithi114.stdout: "noscrub", 2024-09-21T12:22:58.678 INFO:teuthology.orchestra.run.smithi114.stdout: "pglog_hardlimit", 2024-09-21T12:22:58.678 INFO:teuthology.orchestra.run.smithi114.stdout: "purged_snapdirs", 2024-09-21T12:22:58.678 INFO:teuthology.orchestra.run.smithi114.stdout: "recovery_deletes", 2024-09-21T12:22:58.678 INFO:teuthology.orchestra.run.smithi114.stdout: "sortbitwise" 2024-09-21T12:22:58.678 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:22:58.678 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_version": 18, 2024-09-21T12:22:58.678 INFO:teuthology.orchestra.run.smithi114.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T12:22:58.678 INFO:teuthology.orchestra.run.smithi114.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T12:22:58.678 INFO:teuthology.orchestra.run.smithi114.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T12:22:58.679 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_snapshot": "", 2024-09-21T12:22:58.679 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_max": 7, 2024-09-21T12:22:58.679 INFO:teuthology.orchestra.run.smithi114.stdout: "max_osd": 8, 2024-09-21T12:22:58.679 INFO:teuthology.orchestra.run.smithi114.stdout: "require_min_compat_client": "luminous", 2024-09-21T12:22:58.679 INFO:teuthology.orchestra.run.smithi114.stdout: "min_compat_client": "jewel", 2024-09-21T12:22:58.679 INFO:teuthology.orchestra.run.smithi114.stdout: "require_osd_release": "squid", 2024-09-21T12:22:58.679 INFO:teuthology.orchestra.run.smithi114.stdout: "allow_crimson": false, 2024-09-21T12:22:58.679 INFO:teuthology.orchestra.run.smithi114.stdout: "pools": [ 2024-09-21T12:22:58.679 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.679 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 1, 2024-09-21T12:22:58.679 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": ".mgr", 2024-09-21T12:22:58.679 INFO:teuthology.orchestra.run.smithi114.stdout: "create_time": "2024-09-21T12:17:38.357647+0000", 2024-09-21T12:22:58.679 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-09-21T12:22:58.680 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_names": "hashpspool", 2024-09-21T12:22:58.680 INFO:teuthology.orchestra.run.smithi114.stdout: "type": 1, 2024-09-21T12:22:58.680 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 3, 2024-09-21T12:22:58.680 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": 2, 2024-09-21T12:22:58.680 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_rule": 0, 2024-09-21T12:22:58.680 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_count": 0, 2024-09-21T12:22:58.680 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_target": 0, 2024-09-21T12:22:58.680 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T12:22:58.680 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T12:22:58.680 INFO:teuthology.orchestra.run.smithi114.stdout: "is_stretch_pool": false, 2024-09-21T12:22:58.680 INFO:teuthology.orchestra.run.smithi114.stdout: "object_hash": 2, 2024-09-21T12:22:58.680 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_mode": "off", 2024-09-21T12:22:58.680 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": 11, 2024-09-21T12:22:58.680 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num": 11, 2024-09-21T12:22:58.681 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num_target": 11, 2024-09-21T12:22:58.681 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_target": 11, 2024-09-21T12:22:58.681 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_pending": 11, 2024-09-21T12:22:58.681 INFO:teuthology.orchestra.run.smithi114.stdout: "last_pg_merge_meta": { 2024-09-21T12:22:58.681 INFO:teuthology.orchestra.run.smithi114.stdout: "source_pgid": "0.0", 2024-09-21T12:22:58.681 INFO:teuthology.orchestra.run.smithi114.stdout: "ready_epoch": 0, 2024-09-21T12:22:58.681 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_started": 0, 2024-09-21T12:22:58.681 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_clean": 0, 2024-09-21T12:22:58.681 INFO:teuthology.orchestra.run.smithi114.stdout: "source_version": "0'0", 2024-09-21T12:22:58.681 INFO:teuthology.orchestra.run.smithi114.stdout: "target_version": "0'0" 2024-09-21T12:22:58.681 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.681 INFO:teuthology.orchestra.run.smithi114.stdout: "last_change": "108", 2024-09-21T12:22:58.681 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend": "0", 2024-09-21T12:22:58.681 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T12:22:58.681 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_preluminous": "106", 2024-09-21T12:22:58.682 INFO:teuthology.orchestra.run.smithi114.stdout: "auid": 0, 2024-09-21T12:22:58.682 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_mode": "selfmanaged", 2024-09-21T12:22:58.682 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_seq": 0, 2024-09-21T12:22:58.682 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_epoch": 0, 2024-09-21T12:22:58.682 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_snaps": [], 2024-09-21T12:22:58.682 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps": "[]", 2024-09-21T12:22:58.682 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_bytes": 0, 2024-09-21T12:22:58.682 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_objects": 0, 2024-09-21T12:22:58.682 INFO:teuthology.orchestra.run.smithi114.stdout: "tiers": [], 2024-09-21T12:22:58.682 INFO:teuthology.orchestra.run.smithi114.stdout: "tier_of": -1, 2024-09-21T12:22:58.682 INFO:teuthology.orchestra.run.smithi114.stdout: "read_tier": -1, 2024-09-21T12:22:58.682 INFO:teuthology.orchestra.run.smithi114.stdout: "write_tier": -1, 2024-09-21T12:22:58.682 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_mode": "none", 2024-09-21T12:22:58.683 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_bytes": 0, 2024-09-21T12:22:58.683 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_objects": 0, 2024-09-21T12:22:58.683 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T12:22:58.683 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T12:22:58.683 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T12:22:58.683 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_flush_age": 0, 2024-09-21T12:22:58.683 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_evict_age": 0, 2024-09-21T12:22:58.683 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profile": "", 2024-09-21T12:22:58.683 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_params": { 2024-09-21T12:22:58.683 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "none" 2024-09-21T12:22:58.683 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.683 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_period": 0, 2024-09-21T12:22:58.683 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_count": 0, 2024-09-21T12:22:58.684 INFO:teuthology.orchestra.run.smithi114.stdout: "use_gmt_hitset": true, 2024-09-21T12:22:58.684 INFO:teuthology.orchestra.run.smithi114.stdout: "min_read_recency_for_promote": 0, 2024-09-21T12:22:58.684 INFO:teuthology.orchestra.run.smithi114.stdout: "min_write_recency_for_promote": 0, 2024-09-21T12:22:58.684 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T12:22:58.684 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_search_last_n": 0, 2024-09-21T12:22:58.684 INFO:teuthology.orchestra.run.smithi114.stdout: "grade_table": [], 2024-09-21T12:22:58.684 INFO:teuthology.orchestra.run.smithi114.stdout: "stripe_width": 0, 2024-09-21T12:22:58.684 INFO:teuthology.orchestra.run.smithi114.stdout: "expected_num_objects": 0, 2024-09-21T12:22:58.684 INFO:teuthology.orchestra.run.smithi114.stdout: "fast_read": false, 2024-09-21T12:22:58.684 INFO:teuthology.orchestra.run.smithi114.stdout: "options": { 2024-09-21T12:22:58.684 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_max": 32, 2024-09-21T12:22:58.684 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_min": 1 2024-09-21T12:22:58.684 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.684 INFO:teuthology.orchestra.run.smithi114.stdout: "application_metadata": { 2024-09-21T12:22:58.684 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": {} 2024-09-21T12:22:58.685 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.685 INFO:teuthology.orchestra.run.smithi114.stdout: "read_balance": { 2024-09-21T12:22:58.685 INFO:teuthology.orchestra.run.smithi114.stdout: "score_type": "Fair distribution", 2024-09-21T12:22:58.685 INFO:teuthology.orchestra.run.smithi114.stdout: "score_acting": 2.1700000762939453, 2024-09-21T12:22:58.685 INFO:teuthology.orchestra.run.smithi114.stdout: "score_stable": 2.1700000762939453, 2024-09-21T12:22:58.685 INFO:teuthology.orchestra.run.smithi114.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T12:22:58.685 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_acting": 1.9099999666213989, 2024-09-21T12:22:58.685 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_stable": 1.9099999666213989, 2024-09-21T12:22:58.685 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity_weighted": 1, 2024-09-21T12:22:58.685 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity": 1, 2024-09-21T12:22:58.685 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity_weighted": 1 2024-09-21T12:22:58.685 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.685 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.685 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.686 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 2, 2024-09-21T12:22:58.686 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": "datapool", 2024-09-21T12:22:58.686 INFO:teuthology.orchestra.run.smithi114.stdout: "create_time": "2024-09-21T12:20:12.761046+0000", 2024-09-21T12:22:58.686 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 8193, 2024-09-21T12:22:58.686 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T12:22:58.686 INFO:teuthology.orchestra.run.smithi114.stdout: "type": 1, 2024-09-21T12:22:58.686 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 3, 2024-09-21T12:22:58.686 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": 2, 2024-09-21T12:22:58.686 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_rule": 0, 2024-09-21T12:22:58.686 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_count": 0, 2024-09-21T12:22:58.686 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_target": 0, 2024-09-21T12:22:58.686 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T12:22:58.686 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T12:22:58.686 INFO:teuthology.orchestra.run.smithi114.stdout: "is_stretch_pool": false, 2024-09-21T12:22:58.687 INFO:teuthology.orchestra.run.smithi114.stdout: "object_hash": 2, 2024-09-21T12:22:58.687 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_mode": "off", 2024-09-21T12:22:58.687 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": 3, 2024-09-21T12:22:58.687 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num": 3, 2024-09-21T12:22:58.687 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num_target": 3, 2024-09-21T12:22:58.687 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_target": 3, 2024-09-21T12:22:58.687 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_pending": 3, 2024-09-21T12:22:58.687 INFO:teuthology.orchestra.run.smithi114.stdout: "last_pg_merge_meta": { 2024-09-21T12:22:58.687 INFO:teuthology.orchestra.run.smithi114.stdout: "source_pgid": "0.0", 2024-09-21T12:22:58.687 INFO:teuthology.orchestra.run.smithi114.stdout: "ready_epoch": 0, 2024-09-21T12:22:58.687 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_started": 0, 2024-09-21T12:22:58.687 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_clean": 0, 2024-09-21T12:22:58.687 INFO:teuthology.orchestra.run.smithi114.stdout: "source_version": "0'0", 2024-09-21T12:22:58.687 INFO:teuthology.orchestra.run.smithi114.stdout: "target_version": "0'0" 2024-09-21T12:22:58.687 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.688 INFO:teuthology.orchestra.run.smithi114.stdout: "last_change": "61", 2024-09-21T12:22:58.688 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend": "0", 2024-09-21T12:22:58.688 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T12:22:58.688 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T12:22:58.688 INFO:teuthology.orchestra.run.smithi114.stdout: "auid": 0, 2024-09-21T12:22:58.688 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_mode": "selfmanaged", 2024-09-21T12:22:58.688 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_seq": 2, 2024-09-21T12:22:58.688 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_epoch": 61, 2024-09-21T12:22:58.688 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_snaps": [], 2024-09-21T12:22:58.688 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps": "[]", 2024-09-21T12:22:58.688 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_bytes": 0, 2024-09-21T12:22:58.688 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_objects": 0, 2024-09-21T12:22:58.688 INFO:teuthology.orchestra.run.smithi114.stdout: "tiers": [], 2024-09-21T12:22:58.688 INFO:teuthology.orchestra.run.smithi114.stdout: "tier_of": -1, 2024-09-21T12:22:58.689 INFO:teuthology.orchestra.run.smithi114.stdout: "read_tier": -1, 2024-09-21T12:22:58.689 INFO:teuthology.orchestra.run.smithi114.stdout: "write_tier": -1, 2024-09-21T12:22:58.689 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_mode": "none", 2024-09-21T12:22:58.689 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_bytes": 0, 2024-09-21T12:22:58.689 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_objects": 0, 2024-09-21T12:22:58.689 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T12:22:58.689 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T12:22:58.689 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T12:22:58.689 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_flush_age": 0, 2024-09-21T12:22:58.689 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_evict_age": 0, 2024-09-21T12:22:58.689 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profile": "", 2024-09-21T12:22:58.689 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_params": { 2024-09-21T12:22:58.689 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "none" 2024-09-21T12:22:58.689 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.689 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_period": 0, 2024-09-21T12:22:58.690 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_count": 0, 2024-09-21T12:22:58.690 INFO:teuthology.orchestra.run.smithi114.stdout: "use_gmt_hitset": true, 2024-09-21T12:22:58.690 INFO:teuthology.orchestra.run.smithi114.stdout: "min_read_recency_for_promote": 0, 2024-09-21T12:22:58.690 INFO:teuthology.orchestra.run.smithi114.stdout: "min_write_recency_for_promote": 0, 2024-09-21T12:22:58.690 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T12:22:58.690 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_search_last_n": 0, 2024-09-21T12:22:58.690 INFO:teuthology.orchestra.run.smithi114.stdout: "grade_table": [], 2024-09-21T12:22:58.690 INFO:teuthology.orchestra.run.smithi114.stdout: "stripe_width": 0, 2024-09-21T12:22:58.690 INFO:teuthology.orchestra.run.smithi114.stdout: "expected_num_objects": 0, 2024-09-21T12:22:58.690 INFO:teuthology.orchestra.run.smithi114.stdout: "fast_read": false, 2024-09-21T12:22:58.690 INFO:teuthology.orchestra.run.smithi114.stdout: "options": {}, 2024-09-21T12:22:58.690 INFO:teuthology.orchestra.run.smithi114.stdout: "application_metadata": { 2024-09-21T12:22:58.690 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd": {} 2024-09-21T12:22:58.690 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.691 INFO:teuthology.orchestra.run.smithi114.stdout: "read_balance": { 2024-09-21T12:22:58.691 INFO:teuthology.orchestra.run.smithi114.stdout: "score_type": "Fair distribution", 2024-09-21T12:22:58.691 INFO:teuthology.orchestra.run.smithi114.stdout: "score_acting": 2.6500000953674316, 2024-09-21T12:22:58.691 INFO:teuthology.orchestra.run.smithi114.stdout: "score_stable": 2.6500000953674316, 2024-09-21T12:22:58.691 INFO:teuthology.orchestra.run.smithi114.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T12:22:58.691 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-21T12:22:58.691 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-21T12:22:58.691 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity_weighted": 1, 2024-09-21T12:22:58.691 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity": 1, 2024-09-21T12:22:58.691 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity_weighted": 1 2024-09-21T12:22:58.691 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.691 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.691 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.692 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 3, 2024-09-21T12:22:58.692 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": ".rgw.root", 2024-09-21T12:22:58.692 INFO:teuthology.orchestra.run.smithi114.stdout: "create_time": "2024-09-21T12:20:13.713971+0000", 2024-09-21T12:22:58.692 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-09-21T12:22:58.692 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_names": "hashpspool", 2024-09-21T12:22:58.692 INFO:teuthology.orchestra.run.smithi114.stdout: "type": 1, 2024-09-21T12:22:58.692 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 3, 2024-09-21T12:22:58.692 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": 2, 2024-09-21T12:22:58.692 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_rule": 0, 2024-09-21T12:22:58.692 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_count": 0, 2024-09-21T12:22:58.692 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_target": 0, 2024-09-21T12:22:58.692 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T12:22:58.692 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T12:22:58.693 INFO:teuthology.orchestra.run.smithi114.stdout: "is_stretch_pool": false, 2024-09-21T12:22:58.693 INFO:teuthology.orchestra.run.smithi114.stdout: "object_hash": 2, 2024-09-21T12:22:58.693 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_mode": "off", 2024-09-21T12:22:58.693 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": 32, 2024-09-21T12:22:58.693 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num": 32, 2024-09-21T12:22:58.693 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num_target": 32, 2024-09-21T12:22:58.693 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_target": 32, 2024-09-21T12:22:58.693 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_pending": 32, 2024-09-21T12:22:58.693 INFO:teuthology.orchestra.run.smithi114.stdout: "last_pg_merge_meta": { 2024-09-21T12:22:58.693 INFO:teuthology.orchestra.run.smithi114.stdout: "source_pgid": "0.0", 2024-09-21T12:22:58.693 INFO:teuthology.orchestra.run.smithi114.stdout: "ready_epoch": 0, 2024-09-21T12:22:58.693 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_started": 0, 2024-09-21T12:22:58.693 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_clean": 0, 2024-09-21T12:22:58.694 INFO:teuthology.orchestra.run.smithi114.stdout: "source_version": "0'0", 2024-09-21T12:22:58.694 INFO:teuthology.orchestra.run.smithi114.stdout: "target_version": "0'0" 2024-09-21T12:22:58.694 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.694 INFO:teuthology.orchestra.run.smithi114.stdout: "last_change": "60", 2024-09-21T12:22:58.694 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend": "0", 2024-09-21T12:22:58.694 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T12:22:58.694 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T12:22:58.694 INFO:teuthology.orchestra.run.smithi114.stdout: "auid": 0, 2024-09-21T12:22:58.694 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_mode": "selfmanaged", 2024-09-21T12:22:58.694 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_seq": 0, 2024-09-21T12:22:58.694 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_epoch": 0, 2024-09-21T12:22:58.694 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_snaps": [], 2024-09-21T12:22:58.694 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps": "[]", 2024-09-21T12:22:58.694 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_bytes": 0, 2024-09-21T12:22:58.694 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_objects": 0, 2024-09-21T12:22:58.695 INFO:teuthology.orchestra.run.smithi114.stdout: "tiers": [], 2024-09-21T12:22:58.695 INFO:teuthology.orchestra.run.smithi114.stdout: "tier_of": -1, 2024-09-21T12:22:58.695 INFO:teuthology.orchestra.run.smithi114.stdout: "read_tier": -1, 2024-09-21T12:22:58.695 INFO:teuthology.orchestra.run.smithi114.stdout: "write_tier": -1, 2024-09-21T12:22:58.695 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_mode": "none", 2024-09-21T12:22:58.695 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_bytes": 0, 2024-09-21T12:22:58.695 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_objects": 0, 2024-09-21T12:22:58.695 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T12:22:58.695 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T12:22:58.695 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T12:22:58.695 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_flush_age": 0, 2024-09-21T12:22:58.695 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_evict_age": 0, 2024-09-21T12:22:58.695 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profile": "", 2024-09-21T12:22:58.695 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_params": { 2024-09-21T12:22:58.695 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "none" 2024-09-21T12:22:58.696 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.696 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_period": 0, 2024-09-21T12:22:58.696 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_count": 0, 2024-09-21T12:22:58.696 INFO:teuthology.orchestra.run.smithi114.stdout: "use_gmt_hitset": true, 2024-09-21T12:22:58.696 INFO:teuthology.orchestra.run.smithi114.stdout: "min_read_recency_for_promote": 0, 2024-09-21T12:22:58.696 INFO:teuthology.orchestra.run.smithi114.stdout: "min_write_recency_for_promote": 0, 2024-09-21T12:22:58.696 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T12:22:58.696 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_search_last_n": 0, 2024-09-21T12:22:58.696 INFO:teuthology.orchestra.run.smithi114.stdout: "grade_table": [], 2024-09-21T12:22:58.696 INFO:teuthology.orchestra.run.smithi114.stdout: "stripe_width": 0, 2024-09-21T12:22:58.696 INFO:teuthology.orchestra.run.smithi114.stdout: "expected_num_objects": 0, 2024-09-21T12:22:58.696 INFO:teuthology.orchestra.run.smithi114.stdout: "fast_read": false, 2024-09-21T12:22:58.696 INFO:teuthology.orchestra.run.smithi114.stdout: "options": {}, 2024-09-21T12:22:58.696 INFO:teuthology.orchestra.run.smithi114.stdout: "application_metadata": { 2024-09-21T12:22:58.696 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw": {} 2024-09-21T12:22:58.696 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.697 INFO:teuthology.orchestra.run.smithi114.stdout: "read_balance": { 2024-09-21T12:22:58.697 INFO:teuthology.orchestra.run.smithi114.stdout: "score_type": "Fair distribution", 2024-09-21T12:22:58.697 INFO:teuthology.orchestra.run.smithi114.stdout: "score_acting": 1.7400000095367432, 2024-09-21T12:22:58.697 INFO:teuthology.orchestra.run.smithi114.stdout: "score_stable": 1.7400000095367432, 2024-09-21T12:22:58.697 INFO:teuthology.orchestra.run.smithi114.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T12:22:58.697 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_acting": 1.5299999713897705, 2024-09-21T12:22:58.697 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_stable": 1.5299999713897705, 2024-09-21T12:22:58.697 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity_weighted": 1, 2024-09-21T12:22:58.697 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity": 1, 2024-09-21T12:22:58.697 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity_weighted": 1 2024-09-21T12:22:58.697 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.697 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.697 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.697 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 4, 2024-09-21T12:22:58.697 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": "default.rgw.log", 2024-09-21T12:22:58.698 INFO:teuthology.orchestra.run.smithi114.stdout: "create_time": "2024-09-21T12:20:15.368765+0000", 2024-09-21T12:22:58.698 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-09-21T12:22:58.698 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_names": "hashpspool", 2024-09-21T12:22:58.698 INFO:teuthology.orchestra.run.smithi114.stdout: "type": 1, 2024-09-21T12:22:58.698 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 3, 2024-09-21T12:22:58.698 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": 2, 2024-09-21T12:22:58.698 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_rule": 0, 2024-09-21T12:22:58.698 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_count": 0, 2024-09-21T12:22:58.698 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_target": 0, 2024-09-21T12:22:58.698 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T12:22:58.698 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T12:22:58.698 INFO:teuthology.orchestra.run.smithi114.stdout: "is_stretch_pool": false, 2024-09-21T12:22:58.698 INFO:teuthology.orchestra.run.smithi114.stdout: "object_hash": 2, 2024-09-21T12:22:58.698 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_mode": "off", 2024-09-21T12:22:58.699 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": 32, 2024-09-21T12:22:58.699 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num": 32, 2024-09-21T12:22:58.699 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num_target": 32, 2024-09-21T12:22:58.699 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_target": 32, 2024-09-21T12:22:58.699 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_pending": 32, 2024-09-21T12:22:58.699 INFO:teuthology.orchestra.run.smithi114.stdout: "last_pg_merge_meta": { 2024-09-21T12:22:58.699 INFO:teuthology.orchestra.run.smithi114.stdout: "source_pgid": "0.0", 2024-09-21T12:22:58.699 INFO:teuthology.orchestra.run.smithi114.stdout: "ready_epoch": 0, 2024-09-21T12:22:58.699 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_started": 0, 2024-09-21T12:22:58.699 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_clean": 0, 2024-09-21T12:22:58.699 INFO:teuthology.orchestra.run.smithi114.stdout: "source_version": "0'0", 2024-09-21T12:22:58.699 INFO:teuthology.orchestra.run.smithi114.stdout: "target_version": "0'0" 2024-09-21T12:22:58.699 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.700 INFO:teuthology.orchestra.run.smithi114.stdout: "last_change": "62", 2024-09-21T12:22:58.700 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend": "0", 2024-09-21T12:22:58.700 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T12:22:58.700 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T12:22:58.700 INFO:teuthology.orchestra.run.smithi114.stdout: "auid": 0, 2024-09-21T12:22:58.700 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_mode": "selfmanaged", 2024-09-21T12:22:58.700 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_seq": 0, 2024-09-21T12:22:58.700 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_epoch": 0, 2024-09-21T12:22:58.700 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_snaps": [], 2024-09-21T12:22:58.700 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps": "[]", 2024-09-21T12:22:58.700 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_bytes": 0, 2024-09-21T12:22:58.700 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_objects": 0, 2024-09-21T12:22:58.700 INFO:teuthology.orchestra.run.smithi114.stdout: "tiers": [], 2024-09-21T12:22:58.700 INFO:teuthology.orchestra.run.smithi114.stdout: "tier_of": -1, 2024-09-21T12:22:58.700 INFO:teuthology.orchestra.run.smithi114.stdout: "read_tier": -1, 2024-09-21T12:22:58.701 INFO:teuthology.orchestra.run.smithi114.stdout: "write_tier": -1, 2024-09-21T12:22:58.701 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_mode": "none", 2024-09-21T12:22:58.701 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_bytes": 0, 2024-09-21T12:22:58.701 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_objects": 0, 2024-09-21T12:22:58.701 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T12:22:58.701 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T12:22:58.701 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T12:22:58.701 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_flush_age": 0, 2024-09-21T12:22:58.701 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_evict_age": 0, 2024-09-21T12:22:58.701 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profile": "", 2024-09-21T12:22:58.701 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_params": { 2024-09-21T12:22:58.701 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "none" 2024-09-21T12:22:58.701 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.701 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_period": 0, 2024-09-21T12:22:58.701 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_count": 0, 2024-09-21T12:22:58.702 INFO:teuthology.orchestra.run.smithi114.stdout: "use_gmt_hitset": true, 2024-09-21T12:22:58.702 INFO:teuthology.orchestra.run.smithi114.stdout: "min_read_recency_for_promote": 0, 2024-09-21T12:22:58.702 INFO:teuthology.orchestra.run.smithi114.stdout: "min_write_recency_for_promote": 0, 2024-09-21T12:22:58.702 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T12:22:58.702 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_search_last_n": 0, 2024-09-21T12:22:58.702 INFO:teuthology.orchestra.run.smithi114.stdout: "grade_table": [], 2024-09-21T12:22:58.702 INFO:teuthology.orchestra.run.smithi114.stdout: "stripe_width": 0, 2024-09-21T12:22:58.702 INFO:teuthology.orchestra.run.smithi114.stdout: "expected_num_objects": 0, 2024-09-21T12:22:58.702 INFO:teuthology.orchestra.run.smithi114.stdout: "fast_read": false, 2024-09-21T12:22:58.702 INFO:teuthology.orchestra.run.smithi114.stdout: "options": {}, 2024-09-21T12:22:58.702 INFO:teuthology.orchestra.run.smithi114.stdout: "application_metadata": { 2024-09-21T12:22:58.702 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw": {} 2024-09-21T12:22:58.702 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.703 INFO:teuthology.orchestra.run.smithi114.stdout: "read_balance": { 2024-09-21T12:22:58.703 INFO:teuthology.orchestra.run.smithi114.stdout: "score_type": "Fair distribution", 2024-09-21T12:22:58.703 INFO:teuthology.orchestra.run.smithi114.stdout: "score_acting": 2.4900000095367432, 2024-09-21T12:22:58.703 INFO:teuthology.orchestra.run.smithi114.stdout: "score_stable": 2.4900000095367432, 2024-09-21T12:22:58.703 INFO:teuthology.orchestra.run.smithi114.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T12:22:58.703 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_acting": 2.190000057220459, 2024-09-21T12:22:58.703 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_stable": 2.190000057220459, 2024-09-21T12:22:58.703 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity_weighted": 1, 2024-09-21T12:22:58.703 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity": 1, 2024-09-21T12:22:58.703 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity_weighted": 1 2024-09-21T12:22:58.703 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.703 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.703 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.704 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 5, 2024-09-21T12:22:58.704 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": "default.rgw.control", 2024-09-21T12:22:58.704 INFO:teuthology.orchestra.run.smithi114.stdout: "create_time": "2024-09-21T12:20:17.314275+0000", 2024-09-21T12:22:58.704 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-09-21T12:22:58.704 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_names": "hashpspool", 2024-09-21T12:22:58.704 INFO:teuthology.orchestra.run.smithi114.stdout: "type": 1, 2024-09-21T12:22:58.704 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 3, 2024-09-21T12:22:58.704 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": 2, 2024-09-21T12:22:58.704 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_rule": 0, 2024-09-21T12:22:58.704 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_count": 0, 2024-09-21T12:22:58.704 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_target": 0, 2024-09-21T12:22:58.704 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T12:22:58.704 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T12:22:58.705 INFO:teuthology.orchestra.run.smithi114.stdout: "is_stretch_pool": false, 2024-09-21T12:22:58.705 INFO:teuthology.orchestra.run.smithi114.stdout: "object_hash": 2, 2024-09-21T12:22:58.705 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_mode": "off", 2024-09-21T12:22:58.705 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": 32, 2024-09-21T12:22:58.705 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num": 32, 2024-09-21T12:22:58.705 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num_target": 32, 2024-09-21T12:22:58.705 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_target": 32, 2024-09-21T12:22:58.705 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_pending": 32, 2024-09-21T12:22:58.705 INFO:teuthology.orchestra.run.smithi114.stdout: "last_pg_merge_meta": { 2024-09-21T12:22:58.705 INFO:teuthology.orchestra.run.smithi114.stdout: "source_pgid": "0.0", 2024-09-21T12:22:58.705 INFO:teuthology.orchestra.run.smithi114.stdout: "ready_epoch": 0, 2024-09-21T12:22:58.705 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_started": 0, 2024-09-21T12:22:58.705 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_clean": 0, 2024-09-21T12:22:58.705 INFO:teuthology.orchestra.run.smithi114.stdout: "source_version": "0'0", 2024-09-21T12:22:58.706 INFO:teuthology.orchestra.run.smithi114.stdout: "target_version": "0'0" 2024-09-21T12:22:58.706 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.706 INFO:teuthology.orchestra.run.smithi114.stdout: "last_change": "64", 2024-09-21T12:22:58.706 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend": "0", 2024-09-21T12:22:58.706 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T12:22:58.706 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T12:22:58.706 INFO:teuthology.orchestra.run.smithi114.stdout: "auid": 0, 2024-09-21T12:22:58.706 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_mode": "selfmanaged", 2024-09-21T12:22:58.706 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_seq": 0, 2024-09-21T12:22:58.706 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_epoch": 0, 2024-09-21T12:22:58.706 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_snaps": [], 2024-09-21T12:22:58.706 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps": "[]", 2024-09-21T12:22:58.706 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_bytes": 0, 2024-09-21T12:22:58.706 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_objects": 0, 2024-09-21T12:22:58.707 INFO:teuthology.orchestra.run.smithi114.stdout: "tiers": [], 2024-09-21T12:22:58.707 INFO:teuthology.orchestra.run.smithi114.stdout: "tier_of": -1, 2024-09-21T12:22:58.707 INFO:teuthology.orchestra.run.smithi114.stdout: "read_tier": -1, 2024-09-21T12:22:58.707 INFO:teuthology.orchestra.run.smithi114.stdout: "write_tier": -1, 2024-09-21T12:22:58.707 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_mode": "none", 2024-09-21T12:22:58.707 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_bytes": 0, 2024-09-21T12:22:58.707 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_objects": 0, 2024-09-21T12:22:58.707 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T12:22:58.707 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T12:22:58.707 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T12:22:58.707 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_flush_age": 0, 2024-09-21T12:22:58.707 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_evict_age": 0, 2024-09-21T12:22:58.707 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profile": "", 2024-09-21T12:22:58.707 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_params": { 2024-09-21T12:22:58.708 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "none" 2024-09-21T12:22:58.708 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.708 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_period": 0, 2024-09-21T12:22:58.708 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_count": 0, 2024-09-21T12:22:58.708 INFO:teuthology.orchestra.run.smithi114.stdout: "use_gmt_hitset": true, 2024-09-21T12:22:58.708 INFO:teuthology.orchestra.run.smithi114.stdout: "min_read_recency_for_promote": 0, 2024-09-21T12:22:58.708 INFO:teuthology.orchestra.run.smithi114.stdout: "min_write_recency_for_promote": 0, 2024-09-21T12:22:58.708 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T12:22:58.708 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_search_last_n": 0, 2024-09-21T12:22:58.708 INFO:teuthology.orchestra.run.smithi114.stdout: "grade_table": [], 2024-09-21T12:22:58.708 INFO:teuthology.orchestra.run.smithi114.stdout: "stripe_width": 0, 2024-09-21T12:22:58.708 INFO:teuthology.orchestra.run.smithi114.stdout: "expected_num_objects": 0, 2024-09-21T12:22:58.708 INFO:teuthology.orchestra.run.smithi114.stdout: "fast_read": false, 2024-09-21T12:22:58.708 INFO:teuthology.orchestra.run.smithi114.stdout: "options": {}, 2024-09-21T12:22:58.708 INFO:teuthology.orchestra.run.smithi114.stdout: "application_metadata": { 2024-09-21T12:22:58.709 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw": {} 2024-09-21T12:22:58.709 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.709 INFO:teuthology.orchestra.run.smithi114.stdout: "read_balance": { 2024-09-21T12:22:58.709 INFO:teuthology.orchestra.run.smithi114.stdout: "score_type": "Fair distribution", 2024-09-21T12:22:58.709 INFO:teuthology.orchestra.run.smithi114.stdout: "score_acting": 1.4900000095367432, 2024-09-21T12:22:58.709 INFO:teuthology.orchestra.run.smithi114.stdout: "score_stable": 1.4900000095367432, 2024-09-21T12:22:58.709 INFO:teuthology.orchestra.run.smithi114.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T12:22:58.709 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_acting": 1.309999942779541, 2024-09-21T12:22:58.709 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_stable": 1.309999942779541, 2024-09-21T12:22:58.709 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity_weighted": 1, 2024-09-21T12:22:58.709 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity": 1, 2024-09-21T12:22:58.709 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity_weighted": 1 2024-09-21T12:22:58.709 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.709 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.710 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.710 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 6, 2024-09-21T12:22:58.710 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": "default.rgw.meta", 2024-09-21T12:22:58.710 INFO:teuthology.orchestra.run.smithi114.stdout: "create_time": "2024-09-21T12:20:19.370218+0000", 2024-09-21T12:22:58.710 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-09-21T12:22:58.710 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_names": "hashpspool", 2024-09-21T12:22:58.710 INFO:teuthology.orchestra.run.smithi114.stdout: "type": 1, 2024-09-21T12:22:58.710 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 3, 2024-09-21T12:22:58.710 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": 2, 2024-09-21T12:22:58.710 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_rule": 0, 2024-09-21T12:22:58.710 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_count": 0, 2024-09-21T12:22:58.710 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_target": 0, 2024-09-21T12:22:58.710 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T12:22:58.710 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T12:22:58.710 INFO:teuthology.orchestra.run.smithi114.stdout: "is_stretch_pool": false, 2024-09-21T12:22:58.711 INFO:teuthology.orchestra.run.smithi114.stdout: "object_hash": 2, 2024-09-21T12:22:58.711 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_mode": "off", 2024-09-21T12:22:58.711 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": 32, 2024-09-21T12:22:58.711 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num": 32, 2024-09-21T12:22:58.711 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num_target": 32, 2024-09-21T12:22:58.711 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_target": 32, 2024-09-21T12:22:58.711 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_pending": 32, 2024-09-21T12:22:58.711 INFO:teuthology.orchestra.run.smithi114.stdout: "last_pg_merge_meta": { 2024-09-21T12:22:58.711 INFO:teuthology.orchestra.run.smithi114.stdout: "source_pgid": "0.0", 2024-09-21T12:22:58.711 INFO:teuthology.orchestra.run.smithi114.stdout: "ready_epoch": 0, 2024-09-21T12:22:58.711 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_started": 0, 2024-09-21T12:22:58.711 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_clean": 0, 2024-09-21T12:22:58.711 INFO:teuthology.orchestra.run.smithi114.stdout: "source_version": "0'0", 2024-09-21T12:22:58.711 INFO:teuthology.orchestra.run.smithi114.stdout: "target_version": "0'0" 2024-09-21T12:22:58.711 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.712 INFO:teuthology.orchestra.run.smithi114.stdout: "last_change": "66", 2024-09-21T12:22:58.712 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend": "0", 2024-09-21T12:22:58.712 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T12:22:58.712 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T12:22:58.712 INFO:teuthology.orchestra.run.smithi114.stdout: "auid": 0, 2024-09-21T12:22:58.712 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_mode": "selfmanaged", 2024-09-21T12:22:58.712 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_seq": 0, 2024-09-21T12:22:58.712 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_epoch": 0, 2024-09-21T12:22:58.712 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_snaps": [], 2024-09-21T12:22:58.712 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps": "[]", 2024-09-21T12:22:58.712 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_bytes": 0, 2024-09-21T12:22:58.712 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_objects": 0, 2024-09-21T12:22:58.712 INFO:teuthology.orchestra.run.smithi114.stdout: "tiers": [], 2024-09-21T12:22:58.713 INFO:teuthology.orchestra.run.smithi114.stdout: "tier_of": -1, 2024-09-21T12:22:58.713 INFO:teuthology.orchestra.run.smithi114.stdout: "read_tier": -1, 2024-09-21T12:22:58.713 INFO:teuthology.orchestra.run.smithi114.stdout: "write_tier": -1, 2024-09-21T12:22:58.713 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_mode": "none", 2024-09-21T12:22:58.713 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_bytes": 0, 2024-09-21T12:22:58.713 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_objects": 0, 2024-09-21T12:22:58.713 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T12:22:58.713 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T12:22:58.713 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T12:22:58.713 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_flush_age": 0, 2024-09-21T12:22:58.713 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_evict_age": 0, 2024-09-21T12:22:58.713 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profile": "", 2024-09-21T12:22:58.713 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_params": { 2024-09-21T12:22:58.713 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "none" 2024-09-21T12:22:58.713 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.714 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_period": 0, 2024-09-21T12:22:58.714 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_count": 0, 2024-09-21T12:22:58.714 INFO:teuthology.orchestra.run.smithi114.stdout: "use_gmt_hitset": true, 2024-09-21T12:22:58.714 INFO:teuthology.orchestra.run.smithi114.stdout: "min_read_recency_for_promote": 0, 2024-09-21T12:22:58.714 INFO:teuthology.orchestra.run.smithi114.stdout: "min_write_recency_for_promote": 0, 2024-09-21T12:22:58.714 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T12:22:58.714 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_search_last_n": 0, 2024-09-21T12:22:58.714 INFO:teuthology.orchestra.run.smithi114.stdout: "grade_table": [], 2024-09-21T12:22:58.714 INFO:teuthology.orchestra.run.smithi114.stdout: "stripe_width": 0, 2024-09-21T12:22:58.714 INFO:teuthology.orchestra.run.smithi114.stdout: "expected_num_objects": 0, 2024-09-21T12:22:58.714 INFO:teuthology.orchestra.run.smithi114.stdout: "fast_read": false, 2024-09-21T12:22:58.714 INFO:teuthology.orchestra.run.smithi114.stdout: "options": { 2024-09-21T12:22:58.714 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_bias": 4 2024-09-21T12:22:58.714 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.715 INFO:teuthology.orchestra.run.smithi114.stdout: "application_metadata": { 2024-09-21T12:22:58.715 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw": {} 2024-09-21T12:22:58.715 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.715 INFO:teuthology.orchestra.run.smithi114.stdout: "read_balance": { 2024-09-21T12:22:58.715 INFO:teuthology.orchestra.run.smithi114.stdout: "score_type": "Fair distribution", 2024-09-21T12:22:58.715 INFO:teuthology.orchestra.run.smithi114.stdout: "score_acting": 1.9900000095367432, 2024-09-21T12:22:58.715 INFO:teuthology.orchestra.run.smithi114.stdout: "score_stable": 1.9900000095367432, 2024-09-21T12:22:58.715 INFO:teuthology.orchestra.run.smithi114.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T12:22:58.715 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_acting": 1.75, 2024-09-21T12:22:58.715 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_stable": 1.75, 2024-09-21T12:22:58.715 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity_weighted": 1, 2024-09-21T12:22:58.715 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity": 1, 2024-09-21T12:22:58.715 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity_weighted": 1 2024-09-21T12:22:58.715 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.716 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.716 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.716 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 7, 2024-09-21T12:22:58.716 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": "unique_pool_0", 2024-09-21T12:22:58.716 INFO:teuthology.orchestra.run.smithi114.stdout: "create_time": "2024-09-21T12:21:23.302481+0000", 2024-09-21T12:22:58.716 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-09-21T12:22:58.716 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_names": "hashpspool", 2024-09-21T12:22:58.716 INFO:teuthology.orchestra.run.smithi114.stdout: "type": 1, 2024-09-21T12:22:58.716 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 3, 2024-09-21T12:22:58.716 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": 2, 2024-09-21T12:22:58.716 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_rule": 0, 2024-09-21T12:22:58.716 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_count": 0, 2024-09-21T12:22:58.716 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_target": 0, 2024-09-21T12:22:58.716 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T12:22:58.716 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T12:22:58.716 INFO:teuthology.orchestra.run.smithi114.stdout: "is_stretch_pool": false, 2024-09-21T12:22:58.717 INFO:teuthology.orchestra.run.smithi114.stdout: "object_hash": 2, 2024-09-21T12:22:58.717 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_mode": "off", 2024-09-21T12:22:58.717 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": 16, 2024-09-21T12:22:58.717 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num": 16, 2024-09-21T12:22:58.717 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num_target": 16, 2024-09-21T12:22:58.717 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_target": 16, 2024-09-21T12:22:58.717 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_pending": 16, 2024-09-21T12:22:58.717 INFO:teuthology.orchestra.run.smithi114.stdout: "last_pg_merge_meta": { 2024-09-21T12:22:58.717 INFO:teuthology.orchestra.run.smithi114.stdout: "source_pgid": "0.0", 2024-09-21T12:22:58.717 INFO:teuthology.orchestra.run.smithi114.stdout: "ready_epoch": 0, 2024-09-21T12:22:58.717 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_started": 0, 2024-09-21T12:22:58.717 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_clean": 0, 2024-09-21T12:22:58.717 INFO:teuthology.orchestra.run.smithi114.stdout: "source_version": "0'0", 2024-09-21T12:22:58.717 INFO:teuthology.orchestra.run.smithi114.stdout: "target_version": "0'0" 2024-09-21T12:22:58.717 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.718 INFO:teuthology.orchestra.run.smithi114.stdout: "last_change": "88", 2024-09-21T12:22:58.718 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend": "0", 2024-09-21T12:22:58.718 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T12:22:58.718 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T12:22:58.718 INFO:teuthology.orchestra.run.smithi114.stdout: "auid": 0, 2024-09-21T12:22:58.718 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_mode": "selfmanaged", 2024-09-21T12:22:58.718 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_seq": 0, 2024-09-21T12:22:58.718 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_epoch": 0, 2024-09-21T12:22:58.718 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_snaps": [], 2024-09-21T12:22:58.718 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps": "[]", 2024-09-21T12:22:58.718 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_bytes": 0, 2024-09-21T12:22:58.718 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_objects": 0, 2024-09-21T12:22:58.718 INFO:teuthology.orchestra.run.smithi114.stdout: "tiers": [], 2024-09-21T12:22:58.718 INFO:teuthology.orchestra.run.smithi114.stdout: "tier_of": -1, 2024-09-21T12:22:58.718 INFO:teuthology.orchestra.run.smithi114.stdout: "read_tier": -1, 2024-09-21T12:22:58.719 INFO:teuthology.orchestra.run.smithi114.stdout: "write_tier": -1, 2024-09-21T12:22:58.719 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_mode": "none", 2024-09-21T12:22:58.719 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_bytes": 0, 2024-09-21T12:22:58.719 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_objects": 0, 2024-09-21T12:22:58.719 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T12:22:58.719 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T12:22:58.719 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T12:22:58.719 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_flush_age": 0, 2024-09-21T12:22:58.719 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_evict_age": 0, 2024-09-21T12:22:58.719 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profile": "", 2024-09-21T12:22:58.719 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_params": { 2024-09-21T12:22:58.719 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "none" 2024-09-21T12:22:58.719 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.719 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_period": 0, 2024-09-21T12:22:58.720 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_count": 0, 2024-09-21T12:22:58.720 INFO:teuthology.orchestra.run.smithi114.stdout: "use_gmt_hitset": true, 2024-09-21T12:22:58.720 INFO:teuthology.orchestra.run.smithi114.stdout: "min_read_recency_for_promote": 0, 2024-09-21T12:22:58.720 INFO:teuthology.orchestra.run.smithi114.stdout: "min_write_recency_for_promote": 0, 2024-09-21T12:22:58.720 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T12:22:58.720 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_search_last_n": 0, 2024-09-21T12:22:58.720 INFO:teuthology.orchestra.run.smithi114.stdout: "grade_table": [], 2024-09-21T12:22:58.720 INFO:teuthology.orchestra.run.smithi114.stdout: "stripe_width": 0, 2024-09-21T12:22:58.720 INFO:teuthology.orchestra.run.smithi114.stdout: "expected_num_objects": 0, 2024-09-21T12:22:58.720 INFO:teuthology.orchestra.run.smithi114.stdout: "fast_read": false, 2024-09-21T12:22:58.720 INFO:teuthology.orchestra.run.smithi114.stdout: "options": {}, 2024-09-21T12:22:58.720 INFO:teuthology.orchestra.run.smithi114.stdout: "application_metadata": { 2024-09-21T12:22:58.720 INFO:teuthology.orchestra.run.smithi114.stdout: "rados": {} 2024-09-21T12:22:58.720 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.721 INFO:teuthology.orchestra.run.smithi114.stdout: "read_balance": { 2024-09-21T12:22:58.721 INFO:teuthology.orchestra.run.smithi114.stdout: "score_type": "Fair distribution", 2024-09-21T12:22:58.721 INFO:teuthology.orchestra.run.smithi114.stdout: "score_acting": 1.9900000095367432, 2024-09-21T12:22:58.721 INFO:teuthology.orchestra.run.smithi114.stdout: "score_stable": 1.9900000095367432, 2024-09-21T12:22:58.721 INFO:teuthology.orchestra.run.smithi114.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T12:22:58.721 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_acting": 1.75, 2024-09-21T12:22:58.721 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_stable": 1.75, 2024-09-21T12:22:58.721 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity_weighted": 1, 2024-09-21T12:22:58.721 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity": 1, 2024-09-21T12:22:58.721 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity_weighted": 1 2024-09-21T12:22:58.721 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.721 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.721 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:22:58.721 INFO:teuthology.orchestra.run.smithi114.stdout: "osds": [ 2024-09-21T12:22:58.722 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.722 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 0, 2024-09-21T12:22:58.722 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "5715e521-c7dc-41dd-a075-5c96a88fa9b3", 2024-09-21T12:22:58.722 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:22:58.722 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 1, 2024-09-21T12:22:58.722 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 1, 2024-09-21T12:22:58.722 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 1, 2024-09-21T12:22:58.722 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:22:58.722 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:22:58.722 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 8, 2024-09-21T12:22:58.722 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 120, 2024-09-21T12:22:58.723 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:22:58.723 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:22:58.723 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:22:58.723 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.723 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.723 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.723 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6802", 2024-09-21T12:22:58.723 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:22:58.723 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.723 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.723 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.723 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6803", 2024-09-21T12:22:58.723 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:22:58.723 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.724 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.724 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.724 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:22:58.724 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.724 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.724 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.724 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6804", 2024-09-21T12:22:58.724 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:22:58.724 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.724 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.724 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.724 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6805", 2024-09-21T12:22:58.724 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:22:58.724 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.724 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.724 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.725 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:22:58.725 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.725 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.725 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.725 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6808", 2024-09-21T12:22:58.725 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:22:58.725 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.725 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.725 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.725 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6809", 2024-09-21T12:22:58.725 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:22:58.725 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.725 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.725 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.725 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:22:58.726 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.726 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.726 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.726 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6806", 2024-09-21T12:22:58.726 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:22:58.726 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.726 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.726 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.726 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6807", 2024-09-21T12:22:58.726 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:22:58.726 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.726 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.726 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.726 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6803/3555869816", 2024-09-21T12:22:58.727 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.114:6805/3555869816", 2024-09-21T12:22:58.727 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.114:6809/3555869816", 2024-09-21T12:22:58.727 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.114:6807/3555869816", 2024-09-21T12:22:58.727 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:22:58.727 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:22:58.727 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:22:58.727 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.727 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.727 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.727 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 1, 2024-09-21T12:22:58.727 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "1fe160a7-5972-4e3e-99ce-15e2311ba4c5", 2024-09-21T12:22:58.727 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:22:58.727 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 1, 2024-09-21T12:22:58.728 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 1, 2024-09-21T12:22:58.728 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 1, 2024-09-21T12:22:58.728 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:22:58.728 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:22:58.728 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 13, 2024-09-21T12:22:58.728 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 120, 2024-09-21T12:22:58.728 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:22:58.728 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:22:58.728 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:22:58.728 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.728 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.728 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.728 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6810", 2024-09-21T12:22:58.728 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:22:58.728 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.729 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.729 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.729 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6811", 2024-09-21T12:22:58.729 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:22:58.729 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.729 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.729 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.729 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:22:58.729 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.729 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.729 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.729 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6812", 2024-09-21T12:22:58.729 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:22:58.729 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.729 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.730 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.730 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6813", 2024-09-21T12:22:58.730 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:22:58.730 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.730 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.730 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.730 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:22:58.730 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.730 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.730 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.730 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6816", 2024-09-21T12:22:58.730 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:22:58.730 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.730 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.731 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.731 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6817", 2024-09-21T12:22:58.731 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:22:58.731 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.731 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.731 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.731 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:22:58.731 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.731 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.731 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.731 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6814", 2024-09-21T12:22:58.731 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:22:58.731 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.731 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.732 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.732 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6815", 2024-09-21T12:22:58.732 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:22:58.732 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.732 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.732 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.732 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6811/2744663863", 2024-09-21T12:22:58.732 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.114:6813/2744663863", 2024-09-21T12:22:58.732 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.114:6817/2744663863", 2024-09-21T12:22:58.732 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.114:6815/2744663863", 2024-09-21T12:22:58.732 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:22:58.732 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:22:58.732 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:22:58.732 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.732 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.733 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.733 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 2, 2024-09-21T12:22:58.733 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "6a63581f-7603-49ae-9ab3-b37e7c91ab91", 2024-09-21T12:22:58.733 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:22:58.733 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 1, 2024-09-21T12:22:58.733 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 1, 2024-09-21T12:22:58.733 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 1, 2024-09-21T12:22:58.733 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:22:58.733 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:22:58.733 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 18, 2024-09-21T12:22:58.733 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 120, 2024-09-21T12:22:58.733 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:22:58.733 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:22:58.733 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:22:58.734 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.734 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.734 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.734 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6818", 2024-09-21T12:22:58.734 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:22:58.734 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.734 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.734 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.734 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6819", 2024-09-21T12:22:58.734 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:22:58.734 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.734 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.734 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.734 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:22:58.735 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.735 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.735 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.735 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6820", 2024-09-21T12:22:58.735 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:22:58.735 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.735 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.735 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.735 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6821", 2024-09-21T12:22:58.735 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:22:58.735 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.735 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.735 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.735 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:22:58.735 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.736 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.736 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.736 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6824", 2024-09-21T12:22:58.736 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:22:58.736 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.736 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.736 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.736 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6825", 2024-09-21T12:22:58.736 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:22:58.736 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.736 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.736 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.736 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:22:58.736 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.736 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.736 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.737 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6822", 2024-09-21T12:22:58.737 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:22:58.737 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.737 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.737 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.737 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6823", 2024-09-21T12:22:58.737 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:22:58.737 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.737 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.737 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.737 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6819/3740045173", 2024-09-21T12:22:58.737 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.114:6821/3740045173", 2024-09-21T12:22:58.737 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.114:6825/3740045173", 2024-09-21T12:22:58.738 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.114:6823/3740045173", 2024-09-21T12:22:58.738 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:22:58.738 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:22:58.738 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:22:58.738 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.738 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.738 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.738 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 3, 2024-09-21T12:22:58.738 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "0ad68cdb-ae5c-4090-8a8d-b67c1177ae42", 2024-09-21T12:22:58.738 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:22:58.738 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 1, 2024-09-21T12:22:58.738 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 1, 2024-09-21T12:22:58.738 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 1, 2024-09-21T12:22:58.739 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:22:58.739 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:22:58.739 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 26, 2024-09-21T12:22:58.739 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 121, 2024-09-21T12:22:58.739 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:22:58.739 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:22:58.739 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:22:58.739 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.739 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.739 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.739 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6826", 2024-09-21T12:22:58.739 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:22:58.739 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.739 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.739 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.739 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6827", 2024-09-21T12:22:58.740 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:22:58.740 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.740 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.740 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.740 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:22:58.740 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.740 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.740 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.740 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6828", 2024-09-21T12:22:58.740 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:22:58.740 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.740 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.740 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.740 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6829", 2024-09-21T12:22:58.740 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:22:58.741 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.741 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.741 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.741 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:22:58.741 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.741 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.741 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.741 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6832", 2024-09-21T12:22:58.741 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:22:58.741 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.741 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.741 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.741 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6833", 2024-09-21T12:22:58.741 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:22:58.742 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.742 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.742 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.742 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:22:58.742 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.742 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.742 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.742 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6830", 2024-09-21T12:22:58.742 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:22:58.742 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.742 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.742 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.742 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6831", 2024-09-21T12:22:58.742 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:22:58.743 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.743 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.743 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.743 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6827/1312590318", 2024-09-21T12:22:58.743 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.114:6829/1312590318", 2024-09-21T12:22:58.743 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.114:6833/1312590318", 2024-09-21T12:22:58.743 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.114:6831/1312590318", 2024-09-21T12:22:58.743 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:22:58.743 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:22:58.743 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:22:58.743 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.743 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.743 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.744 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 4, 2024-09-21T12:22:58.744 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "26bb8a84-e41d-4425-8491-2dea9764b7cf", 2024-09-21T12:22:58.744 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:22:58.744 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 0, 2024-09-21T12:22:58.744 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-21T12:22:58.744 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 1, 2024-09-21T12:22:58.744 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:22:58.744 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:22:58.744 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 32, 2024-09-21T12:22:58.744 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 111, 2024-09-21T12:22:58.744 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:22:58.744 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:22:58.744 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:22:58.744 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.744 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.745 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.745 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6800", 2024-09-21T12:22:58.745 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:22:58.745 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.745 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.745 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.745 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6801", 2024-09-21T12:22:58.745 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:22:58.745 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.745 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.745 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.745 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:22:58.745 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.746 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.746 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.746 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6802", 2024-09-21T12:22:58.746 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:22:58.746 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.746 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.746 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.746 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6803", 2024-09-21T12:22:58.746 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:22:58.746 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.746 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.746 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.746 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:22:58.746 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.747 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.747 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.747 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6806", 2024-09-21T12:22:58.747 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:22:58.747 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.747 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.747 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.747 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6807", 2024-09-21T12:22:58.747 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:22:58.747 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.747 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.747 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.747 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:22:58.748 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.748 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.748 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.748 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6804", 2024-09-21T12:22:58.748 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:22:58.748 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.748 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.748 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.748 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6805", 2024-09-21T12:22:58.748 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:22:58.748 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.748 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.748 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.748 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.183:6801/1391644836", 2024-09-21T12:22:58.749 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.183:6803/1391644836", 2024-09-21T12:22:58.749 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.183:6807/1391644836", 2024-09-21T12:22:58.749 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.183:6805/1391644836", 2024-09-21T12:22:58.749 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:22:58.749 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:22:58.749 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:22:58.749 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.749 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.749 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.749 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 5, 2024-09-21T12:22:58.749 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "33f96871-0869-4281-82b9-bd518987c551", 2024-09-21T12:22:58.749 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:22:58.749 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 1, 2024-09-21T12:22:58.750 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 1, 2024-09-21T12:22:58.750 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 1, 2024-09-21T12:22:58.750 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:22:58.750 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:22:58.750 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 39, 2024-09-21T12:22:58.750 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 120, 2024-09-21T12:22:58.750 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:22:58.750 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:22:58.750 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:22:58.750 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.750 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.750 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.750 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6808", 2024-09-21T12:22:58.750 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:22:58.751 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.751 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.751 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.751 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6809", 2024-09-21T12:22:58.751 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:22:58.751 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.751 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.751 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.751 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:22:58.751 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.751 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.751 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.751 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6810", 2024-09-21T12:22:58.751 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:22:58.752 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.752 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.752 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.752 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6811", 2024-09-21T12:22:58.752 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:22:58.752 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.752 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.752 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.752 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:22:58.752 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.752 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.752 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.752 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6814", 2024-09-21T12:22:58.752 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:22:58.752 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.753 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.753 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.753 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6815", 2024-09-21T12:22:58.753 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:22:58.753 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.753 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.753 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.753 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:22:58.753 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.753 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.753 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.753 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6812", 2024-09-21T12:22:58.753 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:22:58.753 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.754 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.754 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.754 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6813", 2024-09-21T12:22:58.754 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:22:58.754 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.755 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.755 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.755 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.183:6809/3187643473", 2024-09-21T12:22:58.755 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.183:6811/3187643473", 2024-09-21T12:22:58.755 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.183:6815/3187643473", 2024-09-21T12:22:58.755 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.183:6813/3187643473", 2024-09-21T12:22:58.755 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:22:58.755 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:22:58.755 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:22:58.755 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.755 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.755 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.755 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 6, 2024-09-21T12:22:58.755 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "e09c72b8-ad5a-4779-a874-b149fce6ed70", 2024-09-21T12:22:58.756 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:22:58.756 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 1, 2024-09-21T12:22:58.756 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 1, 2024-09-21T12:22:58.756 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 1, 2024-09-21T12:22:58.756 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:22:58.756 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:22:58.756 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 47, 2024-09-21T12:22:58.756 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 120, 2024-09-21T12:22:58.756 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:22:58.756 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:22:58.756 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:22:58.756 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.756 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.756 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.756 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6816", 2024-09-21T12:22:58.756 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:22:58.757 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.757 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.757 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.757 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6817", 2024-09-21T12:22:58.757 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:22:58.757 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.757 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.757 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.757 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:22:58.757 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.757 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.757 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.757 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6818", 2024-09-21T12:22:58.757 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:22:58.758 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.758 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.758 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.758 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6819", 2024-09-21T12:22:58.758 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:22:58.758 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.758 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.758 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.758 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:22:58.758 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.758 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.758 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.758 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6822", 2024-09-21T12:22:58.758 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:22:58.759 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.759 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.759 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.759 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6823", 2024-09-21T12:22:58.759 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:22:58.759 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.759 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.759 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.759 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:22:58.759 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.759 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.759 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.759 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6820", 2024-09-21T12:22:58.759 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:22:58.759 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.759 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.760 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.760 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6821", 2024-09-21T12:22:58.760 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:22:58.760 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.760 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.760 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.760 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.183:6817/3025199631", 2024-09-21T12:22:58.760 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.183:6819/3025199631", 2024-09-21T12:22:58.760 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.183:6823/3025199631", 2024-09-21T12:22:58.760 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.183:6821/3025199631", 2024-09-21T12:22:58.760 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:22:58.760 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:22:58.760 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:22:58.760 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.760 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.760 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.761 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 7, 2024-09-21T12:22:58.761 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "1a24194d-1af6-485f-8856-98844a02dd9d", 2024-09-21T12:22:58.761 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:22:58.761 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 1, 2024-09-21T12:22:58.761 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 1, 2024-09-21T12:22:58.761 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 1, 2024-09-21T12:22:58.761 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:22:58.761 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:22:58.761 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 54, 2024-09-21T12:22:58.761 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 120, 2024-09-21T12:22:58.761 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:22:58.761 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:22:58.761 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:22:58.761 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.761 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.762 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.762 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6824", 2024-09-21T12:22:58.762 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:22:58.762 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.762 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.762 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.762 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6825", 2024-09-21T12:22:58.762 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:22:58.762 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.762 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.762 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.762 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:22:58.762 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.762 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.763 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.763 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6826", 2024-09-21T12:22:58.763 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:22:58.763 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.763 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.763 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.763 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6827", 2024-09-21T12:22:58.763 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:22:58.763 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.763 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.763 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.763 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:22:58.764 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.764 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.764 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.764 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6830", 2024-09-21T12:22:58.764 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:22:58.764 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.764 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.764 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.764 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6831", 2024-09-21T12:22:58.764 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:22:58.764 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.764 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.764 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.764 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:22:58.764 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:22:58.765 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.765 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:22:58.765 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6828", 2024-09-21T12:22:58.765 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:22:58.765 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.765 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.765 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:22:58.765 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6829", 2024-09-21T12:22:58.765 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:22:58.765 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.765 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.765 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.765 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.183:6825/3203337442", 2024-09-21T12:22:58.765 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.183:6827/3203337442", 2024-09-21T12:22:58.765 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.183:6831/3203337442", 2024-09-21T12:22:58.766 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.183:6829/3203337442", 2024-09-21T12:22:58.766 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:22:58.766 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:22:58.766 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:22:58.766 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.766 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.766 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:22:58.766 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_xinfo": [ 2024-09-21T12:22:58.766 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.766 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 0, 2024-09-21T12:22:58.766 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:22:58.766 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:22:58.766 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:22:58.766 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:22:58.766 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 0, 2024-09-21T12:22:58.767 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:22:55.420292+0000", 2024-09-21T12:22:58.767 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:22:58.767 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.767 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.767 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 1, 2024-09-21T12:22:58.767 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:22:58.767 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:22:58.767 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:22:58.767 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:22:58.767 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 0, 2024-09-21T12:22:58.767 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:21:53.011311+0000", 2024-09-21T12:22:58.767 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:22:58.767 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.767 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.767 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 2, 2024-09-21T12:22:58.768 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:22:58.768 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:22:58.768 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:22:58.768 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:22:58.768 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 0, 2024-09-21T12:22:58.768 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:21:41.356755+0000", 2024-09-21T12:22:58.768 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:22:58.768 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.768 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.768 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 3, 2024-09-21T12:22:58.768 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:22:58.768 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:22:58.768 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:22:58.768 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:22:58.768 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 0, 2024-09-21T12:22:58.768 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:21:59.257732+0000", 2024-09-21T12:22:58.769 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:22:58.769 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.769 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.769 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 4, 2024-09-21T12:22:58.769 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:22:58.769 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:22:58.769 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:22:58.769 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:22:58.769 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 65536, 2024-09-21T12:22:58.769 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:21:54.884054+0000", 2024-09-21T12:22:58.769 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:22:58.769 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.769 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.769 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 5, 2024-09-21T12:22:58.770 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:22:58.770 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:22:58.770 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:22:58.770 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:22:58.770 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 0, 2024-09-21T12:22:58.770 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:22:51.008666+0000", 2024-09-21T12:22:58.770 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:22:58.770 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.770 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.770 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 6, 2024-09-21T12:22:58.770 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:22:58.770 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:22:58.770 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:22:58.770 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:22:58.770 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 0, 2024-09-21T12:22:58.771 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:22:06.964619+0000", 2024-09-21T12:22:58.771 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:22:58.771 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.771 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.771 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 7, 2024-09-21T12:22:58.771 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:22:58.771 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:22:58.771 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:22:58.771 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:22:58.771 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 0, 2024-09-21T12:22:58.771 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:22:37.990847+0000", 2024-09-21T12:22:58.771 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:22:58.771 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.771 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:22:58.772 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_upmap": [], 2024-09-21T12:22:58.772 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_upmap_items": [], 2024-09-21T12:22:58.772 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_upmap_primaries": [], 2024-09-21T12:22:58.772 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_temp": [ 2024-09-21T12:22:58.772 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:22:58.772 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "7.8", 2024-09-21T12:22:58.772 INFO:teuthology.orchestra.run.smithi114.stdout: "osds": [ 2024-09-21T12:22:58.772 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-09-21T12:22:58.772 INFO:teuthology.orchestra.run.smithi114.stdout: 7 2024-09-21T12:22:58.772 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:22:58.772 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.772 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:22:58.772 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_temp": [], 2024-09-21T12:22:58.772 INFO:teuthology.orchestra.run.smithi114.stdout: "blocklist": { 2024-09-21T12:22:58.773 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:6801/237768393": "2024-09-22T12:20:45.528856+0000", 2024-09-21T12:22:58.773 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:6800/237768393": "2024-09-22T12:20:45.528856+0000", 2024-09-21T12:22:58.773 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/898990943": "2024-09-22T12:14:53.998653+0000", 2024-09-21T12:22:58.773 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/3866305956": "2024-09-22T12:20:45.528856+0000", 2024-09-21T12:22:58.773 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/3598782088": "2024-09-22T12:20:45.528856+0000", 2024-09-21T12:22:58.773 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/2620151541": "2024-09-22T12:15:20.045217+0000", 2024-09-21T12:22:58.773 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:6800/1725148595": "2024-09-22T12:14:53.998653+0000", 2024-09-21T12:22:58.773 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/442404805": "2024-09-22T12:14:53.998653+0000", 2024-09-21T12:22:58.773 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:6801/1725148595": "2024-09-22T12:14:53.998653+0000", 2024-09-21T12:22:58.773 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/4173909019": "2024-09-22T12:20:45.528856+0000", 2024-09-21T12:22:58.773 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/2106467614": "2024-09-22T12:15:20.045217+0000", 2024-09-21T12:22:58.773 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/139803357": "2024-09-22T12:15:20.045217+0000", 2024-09-21T12:22:58.773 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:6801/770932721": "2024-09-22T12:15:20.045217+0000", 2024-09-21T12:22:58.773 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:6800/770932721": "2024-09-22T12:15:20.045217+0000", 2024-09-21T12:22:58.774 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/1457547294": "2024-09-22T12:20:45.528856+0000", 2024-09-21T12:22:58.774 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/3416259228": "2024-09-22T12:14:53.998653+0000", 2024-09-21T12:22:58.774 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/995404371": "2024-09-22T12:20:45.528856+0000" 2024-09-21T12:22:58.774 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.774 INFO:teuthology.orchestra.run.smithi114.stdout: "range_blocklist": {}, 2024-09-21T12:22:58.774 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profiles": { 2024-09-21T12:22:58.774 INFO:teuthology.orchestra.run.smithi114.stdout: "default": { 2024-09-21T12:22:58.774 INFO:teuthology.orchestra.run.smithi114.stdout: "crush-failure-domain": "osd", 2024-09-21T12:22:58.774 INFO:teuthology.orchestra.run.smithi114.stdout: "k": "2", 2024-09-21T12:22:58.774 INFO:teuthology.orchestra.run.smithi114.stdout: "m": "1", 2024-09-21T12:22:58.774 INFO:teuthology.orchestra.run.smithi114.stdout: "plugin": "jerasure", 2024-09-21T12:22:58.774 INFO:teuthology.orchestra.run.smithi114.stdout: "technique": "reed_sol_van" 2024-09-21T12:22:58.774 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.774 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:22:58.774 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps_queue": [], 2024-09-21T12:22:58.775 INFO:teuthology.orchestra.run.smithi114.stdout: "new_removed_snaps": [], 2024-09-21T12:22:58.775 INFO:teuthology.orchestra.run.smithi114.stdout: "new_purged_snaps": [], 2024-09-21T12:22:58.775 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_node_flags": {}, 2024-09-21T12:22:58.775 INFO:teuthology.orchestra.run.smithi114.stdout: "device_class_flags": {}, 2024-09-21T12:22:58.775 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": { 2024-09-21T12:22:58.775 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode_enabled": false, 2024-09-21T12:22:58.775 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_bucket_count": 0, 2024-09-21T12:22:58.775 INFO:teuthology.orchestra.run.smithi114.stdout: "degraded_stretch_mode": 0, 2024-09-21T12:22:58.775 INFO:teuthology.orchestra.run.smithi114.stdout: "recovering_stretch_mode": 0, 2024-09-21T12:22:58.775 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode_bucket": 0 2024-09-21T12:22:58.775 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:22:58.775 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:22:58.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:58.776 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:58.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:58.877 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:58.902 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:58.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:58.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:22:58.973 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T12:22:59.003 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:59.048 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:59.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:59.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:59.150 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:22:59.191 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:59.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:59.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:22:59.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:58 smithi183 ceph-mon[33277]: pgmap v118: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 71 MiB/s wr, 1.14k op/s; 1077/300117 objects misplaced (0.359%); 13 MiB/s, 208 objects/s recovering 2024-09-21T12:22:59.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:58 smithi183 ceph-mon[33277]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:22:59.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:58 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:22:59.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:58 smithi183 ceph-mon[33277]: osdmap e125: 8 total, 8 up, 7 in 2024-09-21T12:22:59.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:58 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/337254602' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:22:59.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:22:58 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3306023993' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T12:22:59.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:58 smithi114 ceph-mon[30324]: pgmap v118: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 71 MiB/s wr, 1.14k op/s; 1077/300117 objects misplaced (0.359%); 13 MiB/s, 208 objects/s recovering 2024-09-21T12:22:59.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:58 smithi114 ceph-mon[30324]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:22:59.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:58 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:22:59.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:58 smithi114 ceph-mon[30324]: osdmap e125: 8 total, 8 up, 7 in 2024-09-21T12:22:59.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:58 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/337254602' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:22:59.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:22:58 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3306023993' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T12:22:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:58 smithi114 ceph-mon[40867]: pgmap v118: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 71 MiB/s wr, 1.14k op/s; 1077/300117 objects misplaced (0.359%); 13 MiB/s, 208 objects/s recovering 2024-09-21T12:22:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:58 smithi114 ceph-mon[40867]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:22:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:58 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:22:59.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:58 smithi114 ceph-mon[40867]: osdmap e125: 8 total, 8 up, 7 in 2024-09-21T12:22:59.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:58 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/337254602' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:22:59.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:22:58 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3306023993' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T12:22:59.293 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:59.302 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:22:59.321 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:59.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:59.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:59.423 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:22:59.461 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:59.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:59.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:22:59.563 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:22:59.600 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:22:59.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:59.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:22:59.641 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 89 16 102994 102978 72.2976 73.0625 0.0128923 0.0138263 2024-09-21T12:22:59.702 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:22:59.731 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:59.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:59.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:22:59.832 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:22:59.860 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:59.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:59.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:22:59.874 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:22:59.874 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:22:59.962 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:22:59.989 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:22:59.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:22:59.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:00.091 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:00.116 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:00.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:00.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:00.218 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:00.244 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:00.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:00.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:00.345 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:00.381 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:00.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:00.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:00.483 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:00.521 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:00.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:00.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:00.623 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:00.641 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Total time run: 90.0063 2024-09-21T12:23:00.641 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Total writes made: 104143 2024-09-21T12:23:00.641 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Write size: 65536 2024-09-21T12:23:00.641 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Object size: 65536 2024-09-21T12:23:00.641 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Bandwidth (MB/sec): 72.3165 2024-09-21T12:23:00.641 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev Bandwidth: 38.3123 2024-09-21T12:23:00.642 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max bandwidth (MB/sec): 214 2024-09-21T12:23:00.642 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min bandwidth (MB/sec): 0 2024-09-21T12:23:00.642 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Average IOPS: 1157 2024-09-21T12:23:00.642 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev IOPS: 613.006 2024-09-21T12:23:00.642 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max IOPS: 3424 2024-09-21T12:23:00.642 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min IOPS: 0 2024-09-21T12:23:00.642 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Average Latency(s): 0.013824 2024-09-21T12:23:00.642 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev Latency(s): 0.0455261 2024-09-21T12:23:00.642 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max latency(s): 2.12184 2024-09-21T12:23:00.642 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min latency(s): 0.00135771 2024-09-21T12:23:00.645 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:00.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:00.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:00.648 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T12:23:00.672 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T12:23:00.747 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:00.772 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:00.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:00.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:00.830 INFO:tasks.thrashosds.thrasher:j is {'epoch': 125, 'fsid': 'e2ad975c-7812-11ef-baf6-efdc52797490', 'created': '2024-09-21T12:14:24.181227+0000', 'modified': '2024-09-21T12:22:57.934494+0000', 'last_up_change': '2024-09-21T12:20:03.702987+0000', 'last_in_change': '2024-09-21T12:22:49.868269+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T12:17:38.357647+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '106', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.1700000762939453, 'score_stable': 2.1700000762939453, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.909999966621399, 'raw_score_stable': 1.909999966621399, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-21T12:20:12.761046+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 61, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-21T12:20:13.713971+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-21T12:20:15.368765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.490000009536743, 'score_stable': 2.490000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-21T12:20:17.314275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-21T12:20:19.370218+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-21T12:21:23.302481+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '5715e521-c7dc-41dd-a075-5c96a88fa9b3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 120, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6802', 'nonce': 3555869816}, {'type': 'v1', 'addr': '172.21.15.114:6803', 'nonce': 3555869816}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6804', 'nonce': 3555869816}, {'type': 'v1', 'addr': '172.21.15.114:6805', 'nonce': 3555869816}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6808', 'nonce': 3555869816}, {'type': 'v1', 'addr': '172.21.15.114:6809', 'nonce': 3555869816}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6806', 'nonce': 3555869816}, {'type': 'v1', 'addr': '172.21.15.114:6807', 'nonce': 3555869816}]}, 'public_addr': '172.21.15.114:6803/3555869816', 'cluster_addr': '172.21.15.114:6805/3555869816', 'heartbeat_back_addr': '172.21.15.114:6809/3555869816', 'heartbeat_front_addr': '172.21.15.114:6807/3555869816', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '1fe160a7-5972-4e3e-99ce-15e2311ba4c5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 120, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6810', 'nonce': 2744663863}, {'type': 'v1', 'addr': '172.21.15.114:6811', 'nonce': 2744663863}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6812', 'nonce': 2744663863}, {'type': 'v1', 'addr': '172.21.15.114:6813', 'nonce': 2744663863}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6816', 'nonce': 2744663863}, {'type': 'v1', 'addr': '172.21.15.114:6817', 'nonce': 2744663863}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6814', 'nonce': 2744663863}, {'type': 'v1', 'addr': '172.21.15.114:6815', 'nonce': 2744663863}]}, 'public_addr': '172.21.15.114:6811/2744663863', 'cluster_addr': '172.21.15.114:6813/2744663863', 'heartbeat_back_addr': '172.21.15.114:6817/2744663863', 'heartbeat_front_addr': '172.21.15.114:6815/2744663863', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '6a63581f-7603-49ae-9ab3-b37e7c91ab91', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 120, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6818', 'nonce': 3740045173}, {'type': 'v1', 'addr': '172.21.15.114:6819', 'nonce': 3740045173}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6820', 'nonce': 3740045173}, {'type': 'v1', 'addr': '172.21.15.114:6821', 'nonce': 3740045173}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6824', 'nonce': 3740045173}, {'type': 'v1', 'addr': '172.21.15.114:6825', 'nonce': 3740045173}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6822', 'nonce': 3740045173}, {'type': 'v1', 'addr': '172.21.15.114:6823', 'nonce': 3740045173}]}, 'public_addr': '172.21.15.114:6819/3740045173', 'cluster_addr': '172.21.15.114:6821/3740045173', 'heartbeat_back_addr': '172.21.15.114:6825/3740045173', 'heartbeat_front_addr': '172.21.15.114:6823/3740045173', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '0ad68cdb-ae5c-4090-8a8d-b67c1177ae42', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 121, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6826', 'nonce': 1312590318}, {'type': 'v1', 'addr': '172.21.15.114:6827', 'nonce': 1312590318}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6828', 'nonce': 1312590318}, {'type': 'v1', 'addr': '172.21.15.114:6829', 'nonce': 1312590318}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6832', 'nonce': 1312590318}, {'type': 'v1', 'addr': '172.21.15.114:6833', 'nonce': 1312590318}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6830', 'nonce': 1312590318}, {'type': 'v1', 'addr': '172.21.15.114:6831', 'nonce': 1312590318}]}, 'public_addr': '172.21.15.114:6827/1312590318', 'cluster_addr': '172.21.15.114:6829/1312590318', 'heartbeat_back_addr': '172.21.15.114:6833/1312590318', 'heartbeat_front_addr': '172.21.15.114:6831/1312590318', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '26bb8a84-e41d-4425-8491-2dea9764b7cf', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 111, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6800', 'nonce': 1391644836}, {'type': 'v1', 'addr': '172.21.15.183:6801', 'nonce': 1391644836}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6802', 'nonce': 1391644836}, {'type': 'v1', 'addr': '172.21.15.183:6803', 'nonce': 1391644836}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6806', 'nonce': 1391644836}, {'type': 'v1', 'addr': '172.21.15.183:6807', 'nonce': 1391644836}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6804', 'nonce': 1391644836}, {'type': 'v1', 'addr': '172.21.15.183:6805', 'nonce': 1391644836}]}, 'public_addr': '172.21.15.183:6801/1391644836', 'cluster_addr': '172.21.15.183:6803/1391644836', 'heartbeat_back_addr': '172.21.15.183:6807/1391644836', 'heartbeat_front_addr': '172.21.15.183:6805/1391644836', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '33f96871-0869-4281-82b9-bd518987c551', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 120, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6808', 'nonce': 3187643473}, {'type': 'v1', 'addr': '172.21.15.183:6809', 'nonce': 3187643473}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6810', 'nonce': 3187643473}, {'type': 'v1', 'addr': '172.21.15.183:6811', 'nonce': 3187643473}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6814', 'nonce': 3187643473}, {'type': 'v1', 'addr': '172.21.15.183:6815', 'nonce': 3187643473}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6812', 'nonce': 3187643473}, {'type': 'v1', 'addr': '172.21.15.183:6813', 'nonce': 3187643473}]}, 'public_addr': '172.21.15.183:6809/3187643473', 'cluster_addr': '172.21.15.183:6811/3187643473', 'heartbeat_back_addr': '172.21.15.183:6815/3187643473', 'heartbeat_front_addr': '172.21.15.183:6813/3187643473', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e09c72b8-ad5a-4779-a874-b149fce6ed70', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 120, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6816', 'nonce': 3025199631}, {'type': 'v1', 'addr': '172.21.15.183:6817', 'nonce': 3025199631}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6818', 'nonce': 3025199631}, {'type': 'v1', 'addr': '172.21.15.183:6819', 'nonce': 3025199631}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6822', 'nonce': 3025199631}, {'type': 'v1', 'addr': '172.21.15.183:6823', 'nonce': 3025199631}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6820', 'nonce': 3025199631}, {'type': 'v1', 'addr': '172.21.15.183:6821', 'nonce': 3025199631}]}, 'public_addr': '172.21.15.183:6817/3025199631', 'cluster_addr': '172.21.15.183:6819/3025199631', 'heartbeat_back_addr': '172.21.15.183:6823/3025199631', 'heartbeat_front_addr': '172.21.15.183:6821/3025199631', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1a24194d-1af6-485f-8856-98844a02dd9d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 120, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6824', 'nonce': 3203337442}, {'type': 'v1', 'addr': '172.21.15.183:6825', 'nonce': 3203337442}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6826', 'nonce': 3203337442}, {'type': 'v1', 'addr': '172.21.15.183:6827', 'nonce': 3203337442}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6830', 'nonce': 3203337442}, {'type': 'v1', 'addr': '172.21.15.183:6831', 'nonce': 3203337442}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6828', 'nonce': 3203337442}, {'type': 'v1', 'addr': '172.21.15.183:6829', 'nonce': 3203337442}]}, 'public_addr': '172.21.15.183:6825/3203337442', 'cluster_addr': '172.21.15.183:6827/3203337442', 'heartbeat_back_addr': '172.21.15.183:6831/3203337442', 'heartbeat_front_addr': '172.21.15.183:6829/3203337442', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T12:22:55.420292+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T12:21:53.011311+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T12:21:41.356755+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T12:21:59.257732+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-21T12:21:54.884054+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T12:22:51.008666+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T12:22:06.964619+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T12:22:37.990847+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '7.8', 'osds': [2, 7]}], 'primary_temp': [], 'blocklist': {'172.21.15.114:6801/237768393': '2024-09-22T12:20:45.528856+0000', '172.21.15.114:6800/237768393': '2024-09-22T12:20:45.528856+0000', '172.21.15.114:0/898990943': '2024-09-22T12:14:53.998653+0000', '172.21.15.114:0/3866305956': '2024-09-22T12:20:45.528856+0000', '172.21.15.114:0/3598782088': '2024-09-22T12:20:45.528856+0000', '172.21.15.114:0/2620151541': '2024-09-22T12:15:20.045217+0000', '172.21.15.114:6800/1725148595': '2024-09-22T12:14:53.998653+0000', '172.21.15.114:0/442404805': '2024-09-22T12:14:53.998653+0000', '172.21.15.114:6801/1725148595': '2024-09-22T12:14:53.998653+0000', '172.21.15.114:0/4173909019': '2024-09-22T12:20:45.528856+0000', '172.21.15.114:0/2106467614': '2024-09-22T12:15:20.045217+0000', '172.21.15.114:0/139803357': '2024-09-22T12:15:20.045217+0000', '172.21.15.114:6801/770932721': '2024-09-22T12:15:20.045217+0000', '172.21.15.114:6800/770932721': '2024-09-22T12:15:20.045217+0000', '172.21.15.114:0/1457547294': '2024-09-22T12:20:45.528856+0000', '172.21.15.114:0/3416259228': '2024-09-22T12:14:53.998653+0000', '172.21.15.114:0/995404371': '2024-09-22T12:20:45.528856+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T12:23:00.830 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:23:00.873 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:00.898 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:00.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:00.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:00.988 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:23:01.000 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:01.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:01.061 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:01.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:01.162 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:01.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:01.206 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:01.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:01.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:00 smithi183 ceph-mon[33277]: pgmap v120: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 50 MiB/s wr, 798 op/s; 1077/300117 objects misplaced (0.359%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T12:23:01.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:00 smithi114 ceph-mon[30324]: pgmap v120: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 50 MiB/s wr, 798 op/s; 1077/300117 objects misplaced (0.359%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T12:23:01.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:00 smithi114 ceph-mon[40867]: pgmap v120: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 50 MiB/s wr, 798 op/s; 1077/300117 objects misplaced (0.359%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T12:23:01.307 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:01.328 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:01.387 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:01.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:01.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:01.489 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:01.527 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:01.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:01.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:01.561 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:23:01.561 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:23:01.629 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:01.652 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:01.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:01.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:01.693 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T12:23:01.754 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:01.777 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:01.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:01.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:01.879 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:01.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:01.925 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:01.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:02.026 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:02.048 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:02.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:02.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:02.150 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:02.171 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:02 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:23:02.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:02 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:23:02.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:02 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:23:02.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:02 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:23:02.172 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:02 smithi183 ceph-mon[33277]: osdmap e126: 8 total, 8 up, 7 in 2024-09-21T12:23:02.173 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:02.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:02.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:02.187 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:23:02.208 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T12:23:02.275 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:02.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:02 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:23:02.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:02 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:23:02.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:02 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:23:02.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:02 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:23:02.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:02 smithi114 ceph-mon[30324]: osdmap e126: 8 total, 8 up, 7 in 2024-09-21T12:23:02.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:02 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:23:02.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:02 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:23:02.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:02 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:23:02.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:02 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:23:02.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:02 smithi114 ceph-mon[40867]: osdmap e126: 8 total, 8 up, 7 in 2024-09-21T12:23:02.298 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:02.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:02.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:02.389 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:02.401 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:02.481 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:02.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:02.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:02.583 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:02.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:02.631 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:02.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:02.732 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:02.743 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:23:02.748 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:23:02.754 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:02.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:02.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:02.807 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:23:02 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:23:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:23:02.856 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:02.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:02.897 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:02.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:02.998 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:03.022 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:03.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:03.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:03.124 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:03.161 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:03.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:03.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:03.264 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:03.290 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:03.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:03.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:03.389 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:03 smithi183 ceph-mon[33277]: pgmap v121: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 6.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.8 KiB/s rd, 64 MiB/s wr, 1.09k op/s; 1077/303279 objects misplaced (0.355%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T12:23:03.392 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:03.420 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:03.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:03.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:03.442 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:03 smithi114 ceph-mon[30324]: pgmap v121: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 6.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.8 KiB/s rd, 64 MiB/s wr, 1.09k op/s; 1077/303279 objects misplaced (0.355%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T12:23:03.443 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:03 smithi114 ceph-mon[40867]: pgmap v121: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 6.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.8 KiB/s rd, 64 MiB/s wr, 1.09k op/s; 1077/303279 objects misplaced (0.355%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T12:23:03.521 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:03.547 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:03.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:03.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:03.649 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:03.701 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:03.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:03.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:03.803 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:03.843 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:03.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:03.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:03.945 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:03.970 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:03.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:03.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:04.072 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:04.081 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-21T12:23:04.115 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:04.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:04.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:04.216 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:04.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:04 smithi183 ceph-mon[33277]: from='client.24658 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:23:04.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:04 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3047871698' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:23:04.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:04 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:23:04.253 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:04.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:04.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:04.355 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:04.391 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:04.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:04.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:04.493 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:04.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:04 smithi114 ceph-mon[30324]: from='client.24658 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:23:04.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:04 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3047871698' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:23:04.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:04 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:23:04.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:04 smithi114 ceph-mon[40867]: from='client.24658 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:23:04.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:04 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3047871698' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:23:04.494 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:04 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:23:04.535 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:04.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:04.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:04.542 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2024-09-21T12:23:01.808063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6749606813,"num_objects":103185,"num_object_clones":0,"num_object_copies":309555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":851,"num_objects_unfound":0,"num_objects_dirty":103185,"num_whiteouts":0,"num_read":950,"num_read_kb":770,"num_write":105810,"num_write_kb":6666234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15165,"num_bytes_recovered":981454248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72122,"ondisk_log_size":72122,"up":474,"acting":473,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":20508616,"kb_used_data":19855512,"kb_used_omap":10,"kb_used_meta":652661,"kb_avail":635707448,"statfs":{"total":671965249536,"available":650964426752,"internally_reserved":0,"allocated":20332044288,"data_stored":20327118176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":668324971},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,3,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":520028184,"num_objects":7936,"num_object_clones":0,"num_object_copies":23808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-443,"num_objects_unfound":0,"num_objects_dirty":7936,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":10252,"num_write_kb":581953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":443,"num_bytes_recovered":29032448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.872355"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":153,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.955897+0000","last_change":"2024-09-21T12:22:16.705200+0000","last_active":"2024-09-21T12:22:57.955897+0000","last_peered":"2024-09-21T12:22:57.955897+0000","last_clean":"2024-09-21T12:22:57.955897+0000","last_became_active":"2024-09-21T12:20:15.309431+0000","last_became_peered":"2024-09-21T12:20:15.309431+0000","last_unstale":"2024-09-21T12:22:57.955897+0000","last_undegraded":"2024-09-21T12:22:57.955897+0000","last_fullsized":"2024-09-21T12:22:57.955897+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:16.705088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:16.705088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:28.011157+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":123,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.957262+0000","last_change":"2024-09-21T12:22:51.040744+0000","last_active":"2024-09-21T12:22:57.957262+0000","last_peered":"2024-09-21T12:22:57.957262+0000","last_clean":"2024-09-21T12:22:57.957262+0000","last_became_active":"2024-09-21T12:22:50.909699+0000","last_became_peered":"2024-09-21T12:22:50.909699+0000","last_unstale":"2024-09-21T12:22:57.957262+0000","last_undegraded":"2024-09-21T12:22:57.957262+0000","last_fullsized":"2024-09-21T12:22:57.957262+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":137,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.972247+0000","last_change":"2024-09-21T12:20:19.315239+0000","last_active":"2024-09-21T12:22:57.972247+0000","last_peered":"2024-09-21T12:22:57.972247+0000","last_clean":"2024-09-21T12:22:57.972247+0000","last_became_active":"2024-09-21T12:20:19.314908+0000","last_became_peered":"2024-09-21T12:20:19.314908+0000","last_unstale":"2024-09-21T12:22:57.972247+0000","last_undegraded":"2024-09-21T12:22:57.972247+0000","last_fullsized":"2024-09-21T12:22:57.972247+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":76,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.964181+0000","last_change":"2024-09-21T12:22:50.914064+0000","last_active":"2024-09-21T12:22:57.964181+0000","last_peered":"2024-09-21T12:22:57.964181+0000","last_clean":"2024-09-21T12:22:57.964181+0000","last_became_active":"2024-09-21T12:22:50.913743+0000","last_became_peered":"2024-09-21T12:22:50.913743+0000","last_unstale":"2024-09-21T12:22:57.964181+0000","last_undegraded":"2024-09-21T12:22:57.964181+0000","last_fullsized":"2024-09-21T12:22:57.964181+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:58.189665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:21:58.189665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:05.853011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":131,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.963295+0000","last_change":"2024-09-21T12:20:21.595607+0000","last_active":"2024-09-21T12:22:57.963295+0000","last_peered":"2024-09-21T12:22:57.963295+0000","last_clean":"2024-09-21T12:22:57.963295+0000","last_became_active":"2024-09-21T12:20:21.595301+0000","last_became_peered":"2024-09-21T12:20:21.595301+0000","last_unstale":"2024-09-21T12:22:57.963295+0000","last_undegraded":"2024-09-21T12:22:57.963295+0000","last_fullsized":"2024-09-21T12:22:57.963295+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:25.025934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":151,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.962891+0000","last_change":"2024-09-21T12:22:42.026488+0000","last_active":"2024-09-21T12:22:57.962891+0000","last_peered":"2024-09-21T12:22:57.962891+0000","last_clean":"2024-09-21T12:22:57.962891+0000","last_became_active":"2024-09-21T12:20:15.308655+0000","last_became_peered":"2024-09-21T12:20:15.308655+0000","last_unstale":"2024-09-21T12:22:57.962891+0000","last_undegraded":"2024-09-21T12:22:57.962891+0000","last_fullsized":"2024-09-21T12:22:57.962891+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:42.026416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:42.026416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:48.816231+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":172,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.962646+0000","last_change":"2024-09-21T12:22:45.081280+0000","last_active":"2024-09-21T12:22:57.962646+0000","last_peered":"2024-09-21T12:22:57.962646+0000","last_clean":"2024-09-21T12:22:57.962646+0000","last_became_active":"2024-09-21T12:20:17.305220+0000","last_became_peered":"2024-09-21T12:20:17.305220+0000","last_unstale":"2024-09-21T12:22:57.962646+0000","last_undegraded":"2024-09-21T12:22:57.962646+0000","last_fullsized":"2024-09-21T12:22:57.962646+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:22:45.081089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:45.081089+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:11.317199+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":77,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.957308+0000","last_change":"2024-09-21T12:22:50.926564+0000","last_active":"2024-09-21T12:22:57.957308+0000","last_peered":"2024-09-21T12:22:57.957308+0000","last_clean":"2024-09-21T12:22:57.957308+0000","last_became_active":"2024-09-21T12:22:50.926240+0000","last_became_peered":"2024-09-21T12:22:50.926240+0000","last_unstale":"2024-09-21T12:22:57.957308+0000","last_undegraded":"2024-09-21T12:22:57.957308+0000","last_fullsized":"2024-09-21T12:22:57.957308+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:06.670835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:06.670835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:35.841421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":173,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.964960+0000","last_change":"2024-09-21T12:22:50.912964+0000","last_active":"2024-09-21T12:22:57.964960+0000","last_peered":"2024-09-21T12:22:57.964960+0000","last_clean":"2024-09-21T12:22:57.964960+0000","last_became_active":"2024-09-21T12:22:50.912481+0000","last_became_peered":"2024-09-21T12:22:50.912481+0000","last_unstale":"2024-09-21T12:22:57.964960+0000","last_undegraded":"2024-09-21T12:22:57.964960+0000","last_fullsized":"2024-09-21T12:22:57.964960+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:43.207441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:43.207441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:04.703202+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.964266+0000","last_change":"2024-09-21T12:22:51.018933+0000","last_active":"2024-09-21T12:22:57.964266+0000","last_peered":"2024-09-21T12:22:57.964266+0000","last_clean":"2024-09-21T12:22:57.964266+0000","last_became_active":"2024-09-21T12:22:50.927804+0000","last_became_peered":"2024-09-21T12:22:50.927804+0000","last_unstale":"2024-09-21T12:22:57.964266+0000","last_undegraded":"2024-09-21T12:22:57.964266+0000","last_fullsized":"2024-09-21T12:22:57.964266+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:22:03.017467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:03.017467+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:23.788098+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":144,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.964550+0000","last_change":"2024-09-21T12:22:15.018306+0000","last_active":"2024-09-21T12:22:57.964550+0000","last_peered":"2024-09-21T12:22:57.964550+0000","last_clean":"2024-09-21T12:22:57.964550+0000","last_became_active":"2024-09-21T12:20:19.308184+0000","last_became_peered":"2024-09-21T12:20:19.308184+0000","last_unstale":"2024-09-21T12:22:57.964550+0000","last_undegraded":"2024-09-21T12:22:57.964550+0000","last_fullsized":"2024-09-21T12:22:57.964550+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:15.018264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:15.018264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:44.133250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":133,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.954040+0000","last_change":"2024-09-21T12:20:21.309871+0000","last_active":"2024-09-21T12:22:57.954040+0000","last_peered":"2024-09-21T12:22:57.954040+0000","last_clean":"2024-09-21T12:22:57.954040+0000","last_became_active":"2024-09-21T12:20:21.309190+0000","last_became_peered":"2024-09-21T12:20:21.309190+0000","last_unstale":"2024-09-21T12:22:57.954040+0000","last_undegraded":"2024-09-21T12:22:57.954040+0000","last_fullsized":"2024-09-21T12:22:57.954040+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":173,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.964573+0000","last_change":"2024-09-21T12:22:54.174099+0000","last_active":"2024-09-21T12:22:57.964573+0000","last_peered":"2024-09-21T12:22:57.964573+0000","last_clean":"2024-09-21T12:22:57.964573+0000","last_became_active":"2024-09-21T12:22:50.911737+0000","last_became_peered":"2024-09-21T12:22:50.911737+0000","last_unstale":"2024-09-21T12:22:57.964573+0000","last_undegraded":"2024-09-21T12:22:57.964573+0000","last_fullsized":"2024-09-21T12:22:57.964573+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:54.174034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:54.174034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:20.695067+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":118,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.963550+0000","last_change":"2024-09-21T12:22:51.038838+0000","last_active":"2024-09-21T12:22:57.963550+0000","last_peered":"2024-09-21T12:22:57.963550+0000","last_clean":"2024-09-21T12:22:57.963550+0000","last_became_active":"2024-09-21T12:22:50.918794+0000","last_became_peered":"2024-09-21T12:22:50.918794+0000","last_unstale":"2024-09-21T12:22:57.963550+0000","last_undegraded":"2024-09-21T12:22:57.963550+0000","last_fullsized":"2024-09-21T12:22:57.963550+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-21T12:22:10.011633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:10.011633+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:24.501905+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":157,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.959408+0000","last_change":"2024-09-21T12:22:50.922087+0000","last_active":"2024-09-21T12:22:57.959408+0000","last_peered":"2024-09-21T12:22:57.959408+0000","last_clean":"2024-09-21T12:22:57.959408+0000","last_became_active":"2024-09-21T12:22:50.921780+0000","last_became_peered":"2024-09-21T12:22:50.921780+0000","last_unstale":"2024-09-21T12:22:57.959408+0000","last_undegraded":"2024-09-21T12:22:57.959408+0000","last_fullsized":"2024-09-21T12:22:57.959408+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":140,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.964254+0000","last_change":"2024-09-21T12:22:15.977619+0000","last_active":"2024-09-21T12:22:57.964254+0000","last_peered":"2024-09-21T12:22:57.964254+0000","last_clean":"2024-09-21T12:22:57.964254+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:22:57.964254+0000","last_undegraded":"2024-09-21T12:22:57.964254+0000","last_fullsized":"2024-09-21T12:22:57.964254+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:15.977543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:15.977543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:21.175259+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":76,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.957808+0000","last_change":"2024-09-21T12:22:50.915136+0000","last_active":"2024-09-21T12:22:57.957808+0000","last_peered":"2024-09-21T12:22:57.957808+0000","last_clean":"2024-09-21T12:22:57.957808+0000","last_became_active":"2024-09-21T12:22:50.914792+0000","last_became_peered":"2024-09-21T12:22:50.914792+0000","last_unstale":"2024-09-21T12:22:57.957808+0000","last_undegraded":"2024-09-21T12:22:57.957808+0000","last_fullsized":"2024-09-21T12:22:57.957808+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:02.712009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:02.712009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:28.145066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":174,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.955562+0000","last_change":"2024-09-21T12:22:50.922692+0000","last_active":"2024-09-21T12:22:57.955562+0000","last_peered":"2024-09-21T12:22:57.955562+0000","last_clean":"2024-09-21T12:22:57.955562+0000","last_became_active":"2024-09-21T12:22:50.921576+0000","last_became_peered":"2024-09-21T12:22:50.921576+0000","last_unstale":"2024-09-21T12:22:57.955562+0000","last_undegraded":"2024-09-21T12:22:57.955562+0000","last_fullsized":"2024-09-21T12:22:57.955562+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:20.217841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.217841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":166,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.955821+0000","last_change":"2024-09-21T12:20:17.308186+0000","last_active":"2024-09-21T12:22:57.955821+0000","last_peered":"2024-09-21T12:22:57.955821+0000","last_clean":"2024-09-21T12:22:57.955821+0000","last_became_active":"2024-09-21T12:20:17.307828+0000","last_became_peered":"2024-09-21T12:20:17.307828+0000","last_unstale":"2024-09-21T12:22:57.955821+0000","last_undegraded":"2024-09-21T12:22:57.955821+0000","last_fullsized":"2024-09-21T12:22:57.955821+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":166,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.976049+0000","last_change":"2024-09-21T12:22:50.930597+0000","last_active":"2024-09-21T12:22:57.976049+0000","last_peered":"2024-09-21T12:22:57.976049+0000","last_clean":"2024-09-21T12:22:57.976049+0000","last_became_active":"2024-09-21T12:22:50.928972+0000","last_became_peered":"2024-09-21T12:22:50.928972+0000","last_unstale":"2024-09-21T12:22:57.976049+0000","last_undegraded":"2024-09-21T12:22:57.976049+0000","last_fullsized":"2024-09-21T12:22:57.976049+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:10.823915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:10.823915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:31.452808+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":131,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.971554+0000","last_change":"2024-09-21T12:20:21.593129+0000","last_active":"2024-09-21T12:22:57.971554+0000","last_peered":"2024-09-21T12:22:57.971554+0000","last_clean":"2024-09-21T12:22:57.971554+0000","last_became_active":"2024-09-21T12:20:21.592796+0000","last_became_peered":"2024-09-21T12:20:21.592796+0000","last_unstale":"2024-09-21T12:22:57.971554+0000","last_undegraded":"2024-09-21T12:22:57.971554+0000","last_fullsized":"2024-09-21T12:22:57.971554+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":76,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.966823+0000","last_change":"2024-09-21T12:22:50.926169+0000","last_active":"2024-09-21T12:22:57.966823+0000","last_peered":"2024-09-21T12:22:57.966823+0000","last_clean":"2024-09-21T12:22:57.966823+0000","last_became_active":"2024-09-21T12:22:50.925403+0000","last_became_peered":"2024-09-21T12:22:50.925403+0000","last_unstale":"2024-09-21T12:22:57.966823+0000","last_undegraded":"2024-09-21T12:22:57.966823+0000","last_fullsized":"2024-09-21T12:22:57.966823+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:07.846732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:07.846732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:35.648599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":219,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.962158+0000","last_change":"2024-09-21T12:22:50.983054+0000","last_active":"2024-09-21T12:22:57.962158+0000","last_peered":"2024-09-21T12:22:57.962158+0000","last_clean":"2024-09-21T12:22:57.962158+0000","last_became_active":"2024-09-21T12:22:50.912144+0000","last_became_peered":"2024-09-21T12:22:50.912144+0000","last_unstale":"2024-09-21T12:22:57.962158+0000","last_undegraded":"2024-09-21T12:22:57.962158+0000","last_fullsized":"2024-09-21T12:22:57.962158+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":69,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.968573+0000","last_change":"2024-09-21T12:22:50.920178+0000","last_active":"2024-09-21T12:22:57.968573+0000","last_peered":"2024-09-21T12:22:57.968573+0000","last_clean":"2024-09-21T12:22:57.968573+0000","last_became_active":"2024-09-21T12:22:50.918326+0000","last_became_peered":"2024-09-21T12:22:50.918326+0000","last_unstale":"2024-09-21T12:22:57.968573+0000","last_undegraded":"2024-09-21T12:22:57.968573+0000","last_fullsized":"2024-09-21T12:22:57.968573+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":133,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.959722+0000","last_change":"2024-09-21T12:20:21.313476+0000","last_active":"2024-09-21T12:22:57.959722+0000","last_peered":"2024-09-21T12:22:57.959722+0000","last_clean":"2024-09-21T12:22:57.959722+0000","last_became_active":"2024-09-21T12:20:21.312976+0000","last_became_peered":"2024-09-21T12:20:21.312976+0000","last_unstale":"2024-09-21T12:22:57.959722+0000","last_undegraded":"2024-09-21T12:22:57.959722+0000","last_fullsized":"2024-09-21T12:22:57.959722+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":187,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.972336+0000","last_change":"2024-09-21T12:22:52.070589+0000","last_active":"2024-09-21T12:22:57.972336+0000","last_peered":"2024-09-21T12:22:57.972336+0000","last_clean":"2024-09-21T12:22:57.972336+0000","last_became_active":"2024-09-21T12:22:50.910933+0000","last_became_peered":"2024-09-21T12:22:50.910933+0000","last_unstale":"2024-09-21T12:22:57.972336+0000","last_undegraded":"2024-09-21T12:22:57.972336+0000","last_fullsized":"2024-09-21T12:22:57.972336+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:22:52.070513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:52.070513+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:07.437026+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":156,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.954417+0000","last_change":"2024-09-21T12:20:17.696423+0000","last_active":"2024-09-21T12:22:57.954417+0000","last_peered":"2024-09-21T12:22:57.954417+0000","last_clean":"2024-09-21T12:22:57.954417+0000","last_became_active":"2024-09-21T12:20:17.695977+0000","last_became_peered":"2024-09-21T12:20:17.695977+0000","last_unstale":"2024-09-21T12:22:57.954417+0000","last_undegraded":"2024-09-21T12:22:57.954417+0000","last_fullsized":"2024-09-21T12:22:57.954417+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"121'11","reported_seq":222,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.958235+0000","last_change":"2024-09-21T12:22:50.926408+0000","last_active":"2024-09-21T12:22:57.958235+0000","last_peered":"2024-09-21T12:22:57.958235+0000","last_clean":"2024-09-21T12:22:57.958235+0000","last_became_active":"2024-09-21T12:22:50.916683+0000","last_became_peered":"2024-09-21T12:22:50.916683+0000","last_unstale":"2024-09-21T12:22:57.958235+0000","last_undegraded":"2024-09-21T12:22:57.958235+0000","last_fullsized":"2024-09-21T12:22:57.958235+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'9","last_scrub_stamp":"2024-09-21T12:22:08.647997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:08.647997+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:21.181826+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"76'79","reported_seq":52,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.964151+0000","last_change":"2024-09-21T12:22:18.000752+0000","last_active":"2024-09-21T12:22:57.964151+0000","last_peered":"2024-09-21T12:22:57.964151+0000","last_clean":"2024-09-21T12:22:57.964151+0000","last_became_active":"2024-09-21T12:22:17.074721+0000","last_became_peered":"2024-09-21T12:22:17.074721+0000","last_unstale":"2024-09-21T12:22:57.964151+0000","last_undegraded":"2024-09-21T12:22:57.964151+0000","last_fullsized":"2024-09-21T12:22:57.964151+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:22:18.000705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:22:18.000705+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:29.821832+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"125'6517","reported_seq":6673,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:23:01.553329+0000","last_change":"2024-09-21T12:22:42.121876+0000","last_active":"2024-09-21T12:23:01.553329+0000","last_peered":"2024-09-21T12:23:01.553329+0000","last_clean":"2024-09-21T12:23:01.553329+0000","last_became_active":"2024-09-21T12:21:24.886130+0000","last_became_peered":"2024-09-21T12:21:24.886130+0000","last_unstale":"2024-09-21T12:23:01.553329+0000","last_undegraded":"2024-09-21T12:23:01.553329+0000","last_fullsized":"2024-09-21T12:23:01.553329+0000","mapping_epoch":84,"log_start":"94'1947","ondisk_log_start":"94'1947","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5485","last_scrub_stamp":"2024-09-21T12:22:42.121763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:22:42.121763+0000","objects_scrubbed":5450,"log_size":4570,"log_dups_size":0,"ondisk_log_size":4570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:11.078251+0000","scrub_duration":1486,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418971648,"num_objects":6393,"num_object_clones":0,"num_object_copies":19179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6517,"num_write_kb":413120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":235,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.970577+0000","last_change":"2024-09-21T12:22:50.956017+0000","last_active":"2024-09-21T12:22:57.970577+0000","last_peered":"2024-09-21T12:22:57.970577+0000","last_clean":"2024-09-21T12:22:57.970577+0000","last_became_active":"2024-09-21T12:22:50.906153+0000","last_became_peered":"2024-09-21T12:22:50.906153+0000","last_unstale":"2024-09-21T12:22:57.970577+0000","last_undegraded":"2024-09-21T12:22:57.970577+0000","last_fullsized":"2024-09-21T12:22:57.970577+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":151,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.966215+0000","last_change":"2024-09-21T12:22:14.036290+0000","last_active":"2024-09-21T12:22:57.966215+0000","last_peered":"2024-09-21T12:22:57.966215+0000","last_clean":"2024-09-21T12:22:57.966215+0000","last_became_active":"2024-09-21T12:20:15.295774+0000","last_became_peered":"2024-09-21T12:20:15.295774+0000","last_unstale":"2024-09-21T12:22:57.966215+0000","last_undegraded":"2024-09-21T12:22:57.966215+0000","last_fullsized":"2024-09-21T12:22:57.966215+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:14.036236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.036236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:39.734223+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"121'11","reported_seq":114,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.965710+0000","last_change":"2024-09-21T12:22:50.955810+0000","last_active":"2024-09-21T12:22:57.965710+0000","last_peered":"2024-09-21T12:22:57.965710+0000","last_clean":"2024-09-21T12:22:57.965710+0000","last_became_active":"2024-09-21T12:22:50.929426+0000","last_became_peered":"2024-09-21T12:22:50.929426+0000","last_unstale":"2024-09-21T12:22:57.965710+0000","last_undegraded":"2024-09-21T12:22:57.965710+0000","last_fullsized":"2024-09-21T12:22:57.965710+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":141,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.964726+0000","last_change":"2024-09-21T12:22:12.145637+0000","last_active":"2024-09-21T12:22:57.964726+0000","last_peered":"2024-09-21T12:22:57.964726+0000","last_clean":"2024-09-21T12:22:57.964726+0000","last_became_active":"2024-09-21T12:20:21.310022+0000","last_became_peered":"2024-09-21T12:20:21.310022+0000","last_unstale":"2024-09-21T12:22:57.964726+0000","last_undegraded":"2024-09-21T12:22:57.964726+0000","last_fullsized":"2024-09-21T12:22:57.964726+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:22:12.145550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:12.145550+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:41.591945+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"76'79","reported_seq":41,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.956756+0000","last_change":"2024-09-21T12:22:50.916239+0000","last_active":"2024-09-21T12:22:57.956756+0000","last_peered":"2024-09-21T12:22:57.956756+0000","last_clean":"2024-09-21T12:22:57.956756+0000","last_became_active":"2024-09-21T12:22:50.914885+0000","last_became_peered":"2024-09-21T12:22:50.914885+0000","last_unstale":"2024-09-21T12:22:57.956756+0000","last_undegraded":"2024-09-21T12:22:57.956756+0000","last_fullsized":"2024-09-21T12:22:57.956756+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:22:17.824342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:22:17.824342+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:29.442279+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.3","version":"125'6494","reported_seq":9684,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:23:01.723907+0000","last_change":"2024-09-21T12:22:50.925842+0000","last_active":"2024-09-21T12:23:01.723907+0000","last_peered":"2024-09-21T12:23:01.723907+0000","last_clean":"2024-09-21T12:23:01.723907+0000","last_became_active":"2024-09-21T12:22:50.919145+0000","last_became_peered":"2024-09-21T12:22:50.919145+0000","last_unstale":"2024-09-21T12:23:01.723907+0000","last_undegraded":"2024-09-21T12:23:01.723907+0000","last_fullsized":"2024-09-21T12:23:01.723907+0000","mapping_epoch":120,"log_start":"98'2566","ondisk_log_start":"98'2566","created":84,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":3928,"log_dups_size":0,"ondisk_log_size":3928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415367168,"num_objects":6338,"num_object_clones":0,"num_object_copies":19014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6494,"num_write_kb":410624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1536,"num_bytes_recovered":100663296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":171,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.966085+0000","last_change":"2024-09-21T12:20:17.693550+0000","last_active":"2024-09-21T12:22:57.966085+0000","last_peered":"2024-09-21T12:22:57.966085+0000","last_clean":"2024-09-21T12:22:57.966085+0000","last_became_active":"2024-09-21T12:20:17.693179+0000","last_became_peered":"2024-09-21T12:20:17.693179+0000","last_unstale":"2024-09-21T12:22:57.966085+0000","last_undegraded":"2024-09-21T12:22:57.966085+0000","last_fullsized":"2024-09-21T12:22:57.966085+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":151,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.971934+0000","last_change":"2024-09-21T12:21:18.056305+0000","last_active":"2024-09-21T12:22:57.971934+0000","last_peered":"2024-09-21T12:22:57.971934+0000","last_clean":"2024-09-21T12:22:57.971934+0000","last_became_active":"2024-09-21T12:20:15.297340+0000","last_became_peered":"2024-09-21T12:20:15.297340+0000","last_unstale":"2024-09-21T12:22:57.971934+0000","last_undegraded":"2024-09-21T12:22:57.971934+0000","last_fullsized":"2024-09-21T12:22:57.971934+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:18.056261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:18.056261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":77,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.962230+0000","last_change":"2024-09-21T12:22:50.931233+0000","last_active":"2024-09-21T12:22:57.962230+0000","last_peered":"2024-09-21T12:22:57.962230+0000","last_clean":"2024-09-21T12:22:57.962230+0000","last_became_active":"2024-09-21T12:22:50.930858+0000","last_became_peered":"2024-09-21T12:22:50.930858+0000","last_unstale":"2024-09-21T12:22:57.962230+0000","last_undegraded":"2024-09-21T12:22:57.962230+0000","last_fullsized":"2024-09-21T12:22:57.962230+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:01.971812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:01.971812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:16.343884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":76,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.964182+0000","last_change":"2024-09-21T12:22:50.927366+0000","last_active":"2024-09-21T12:22:57.964182+0000","last_peered":"2024-09-21T12:22:57.964182+0000","last_clean":"2024-09-21T12:22:57.964182+0000","last_became_active":"2024-09-21T12:22:50.918940+0000","last_became_peered":"2024-09-21T12:22:50.918940+0000","last_unstale":"2024-09-21T12:22:57.964182+0000","last_undegraded":"2024-09-21T12:22:57.964182+0000","last_fullsized":"2024-09-21T12:22:57.964182+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:13.050383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:13.050383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:31.135674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"76'79","reported_seq":49,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.968336+0000","last_change":"2024-09-21T12:22:17.098015+0000","last_active":"2024-09-21T12:22:57.968336+0000","last_peered":"2024-09-21T12:22:57.968336+0000","last_clean":"2024-09-21T12:22:57.968336+0000","last_became_active":"2024-09-21T12:22:17.069042+0000","last_became_peered":"2024-09-21T12:22:17.069042+0000","last_unstale":"2024-09-21T12:22:57.968336+0000","last_undegraded":"2024-09-21T12:22:57.968336+0000","last_fullsized":"2024-09-21T12:22:57.968336+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T12:20:24.850157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:20:24.850157+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:23:21.985894+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":381350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"125'6678","reported_seq":6771,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:23:01.769910+0000","last_change":"2024-09-21T12:21:24.889433+0000","last_active":"2024-09-21T12:23:01.769910+0000","last_peered":"2024-09-21T12:23:01.769910+0000","last_clean":"2024-09-21T12:23:01.769910+0000","last_became_active":"2024-09-21T12:21:24.889133+0000","last_became_peered":"2024-09-21T12:21:24.889133+0000","last_unstale":"2024-09-21T12:23:01.769910+0000","last_undegraded":"2024-09-21T12:23:01.769910+0000","last_fullsized":"2024-09-21T12:23:01.769910+0000","mapping_epoch":84,"log_start":"98'2713","ondisk_log_start":"98'2713","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":3965,"log_dups_size":0,"ondisk_log_size":3965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427556864,"num_objects":6524,"num_object_clones":0,"num_object_copies":19572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6678,"num_write_kb":422464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":149,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.964311+0000","last_change":"2024-09-21T12:22:50.978102+0000","last_active":"2024-09-21T12:22:57.964311+0000","last_peered":"2024-09-21T12:22:57.964311+0000","last_clean":"2024-09-21T12:22:57.964311+0000","last_became_active":"2024-09-21T12:22:50.926008+0000","last_became_peered":"2024-09-21T12:22:50.926008+0000","last_unstale":"2024-09-21T12:22:57.964311+0000","last_undegraded":"2024-09-21T12:22:57.964311+0000","last_fullsized":"2024-09-21T12:22:57.964311+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-21T12:22:14.062464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.062464+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:38.500422+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":187,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.954719+0000","last_change":"2024-09-21T12:22:51.003768+0000","last_active":"2024-09-21T12:22:57.954719+0000","last_peered":"2024-09-21T12:22:57.954719+0000","last_clean":"2024-09-21T12:22:57.954719+0000","last_became_active":"2024-09-21T12:22:50.923284+0000","last_became_peered":"2024-09-21T12:22:50.923284+0000","last_unstale":"2024-09-21T12:22:57.954719+0000","last_undegraded":"2024-09-21T12:22:57.954719+0000","last_fullsized":"2024-09-21T12:22:57.954719+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:22:07.582969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:07.582969+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:20.480647+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"121'11","reported_seq":210,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.965883+0000","last_change":"2024-09-21T12:22:51.062896+0000","last_active":"2024-09-21T12:22:57.965883+0000","last_peered":"2024-09-21T12:22:57.965883+0000","last_clean":"2024-09-21T12:22:57.965883+0000","last_became_active":"2024-09-21T12:22:50.930342+0000","last_became_peered":"2024-09-21T12:22:50.930342+0000","last_unstale":"2024-09-21T12:22:57.965883+0000","last_undegraded":"2024-09-21T12:22:57.965883+0000","last_fullsized":"2024-09-21T12:22:57.965883+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":140,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.959275+0000","last_change":"2024-09-21T12:21:21.835669+0000","last_active":"2024-09-21T12:22:57.959275+0000","last_peered":"2024-09-21T12:22:57.959275+0000","last_clean":"2024-09-21T12:22:57.959275+0000","last_became_active":"2024-09-21T12:20:21.593367+0000","last_became_peered":"2024-09-21T12:20:21.593367+0000","last_unstale":"2024-09-21T12:22:57.959275+0000","last_undegraded":"2024-09-21T12:22:57.959275+0000","last_fullsized":"2024-09-21T12:22:57.959275+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:21.835623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:21:21.835623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"76'79","reported_seq":134,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.958108+0000","last_change":"2024-09-21T12:22:17.093539+0000","last_active":"2024-09-21T12:22:57.958108+0000","last_peered":"2024-09-21T12:22:57.958108+0000","last_clean":"2024-09-21T12:22:57.958108+0000","last_became_active":"2024-09-21T12:22:17.093298+0000","last_became_peered":"2024-09-21T12:22:17.093298+0000","last_unstale":"2024-09-21T12:22:57.958108+0000","last_undegraded":"2024-09-21T12:22:57.958108+0000","last_fullsized":"2024-09-21T12:22:57.958108+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T12:20:24.850157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:20:24.850157+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:23:23.614841+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"125'6766","reported_seq":15910,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:23:01.758848+0000","last_change":"2024-09-21T12:22:51.819626+0000","last_active":"2024-09-21T12:23:01.758848+0000","last_peered":"2024-09-21T12:23:01.758848+0000","last_clean":"2024-09-21T12:23:01.758848+0000","last_became_active":"2024-09-21T12:22:51.808221+0000","last_became_peered":"2024-09-21T12:22:51.808221+0000","last_unstale":"2024-09-21T12:23:01.758848+0000","last_undegraded":"2024-09-21T12:23:01.758848+0000","last_fullsized":"2024-09-21T12:23:01.758848+0000","mapping_epoch":121,"log_start":"98'2788","ondisk_log_start":"98'2788","created":84,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":3978,"log_dups_size":0,"ondisk_log_size":3978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432668672,"num_objects":6602,"num_object_clones":0,"num_object_copies":19806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6766,"num_write_kb":427776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4516,"num_bytes_recovered":295960576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":213,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.966713+0000","last_change":"2024-09-21T12:22:51.006819+0000","last_active":"2024-09-21T12:22:57.966713+0000","last_peered":"2024-09-21T12:22:57.966713+0000","last_clean":"2024-09-21T12:22:57.966713+0000","last_became_active":"2024-09-21T12:22:50.925840+0000","last_became_peered":"2024-09-21T12:22:50.925840+0000","last_unstale":"2024-09-21T12:22:57.966713+0000","last_undegraded":"2024-09-21T12:22:57.966713+0000","last_fullsized":"2024-09-21T12:22:57.966713+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":152,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.965062+0000","last_change":"2024-09-21T12:22:37.176483+0000","last_active":"2024-09-21T12:22:57.965062+0000","last_peered":"2024-09-21T12:22:57.965062+0000","last_clean":"2024-09-21T12:22:57.965062+0000","last_became_active":"2024-09-21T12:20:15.308518+0000","last_became_peered":"2024-09-21T12:20:15.308518+0000","last_unstale":"2024-09-21T12:22:57.965062+0000","last_undegraded":"2024-09-21T12:22:57.965062+0000","last_fullsized":"2024-09-21T12:22:57.965062+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:37.176413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:37.176413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:01.193747+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":190,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.955678+0000","last_change":"2024-09-21T12:21:23.139205+0000","last_active":"2024-09-21T12:22:57.955678+0000","last_peered":"2024-09-21T12:22:57.955678+0000","last_clean":"2024-09-21T12:22:57.955678+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:22:57.955678+0000","last_undegraded":"2024-09-21T12:22:57.955678+0000","last_fullsized":"2024-09-21T12:22:57.955678+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:21:23.139124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.139124+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":141,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.956348+0000","last_change":"2024-09-21T12:22:51.987032+0000","last_active":"2024-09-21T12:22:57.956348+0000","last_peered":"2024-09-21T12:22:57.956348+0000","last_clean":"2024-09-21T12:22:57.956348+0000","last_became_active":"2024-09-21T12:20:21.315801+0000","last_became_peered":"2024-09-21T12:20:21.315801+0000","last_unstale":"2024-09-21T12:22:57.956348+0000","last_undegraded":"2024-09-21T12:22:57.956348+0000","last_fullsized":"2024-09-21T12:22:57.956348+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:51.986953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:51.986953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:58.750058+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"76'79","reported_seq":134,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.957697+0000","last_change":"2024-09-21T12:22:17.083917+0000","last_active":"2024-09-21T12:22:57.957697+0000","last_peered":"2024-09-21T12:22:57.957697+0000","last_clean":"2024-09-21T12:22:57.957697+0000","last_became_active":"2024-09-21T12:22:17.083629+0000","last_became_peered":"2024-09-21T12:22:17.083629+0000","last_unstale":"2024-09-21T12:22:57.957697+0000","last_undegraded":"2024-09-21T12:22:57.957697+0000","last_fullsized":"2024-09-21T12:22:57.957697+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T12:20:24.850157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:20:24.850157+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:23:22.611693+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"125'6460","reported_seq":6618,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:23:01.529793+0000","last_change":"2024-09-21T12:22:48.664422+0000","last_active":"2024-09-21T12:23:01.529793+0000","last_peered":"2024-09-21T12:23:01.529793+0000","last_clean":"2024-09-21T12:23:01.529793+0000","last_became_active":"2024-09-21T12:21:24.884183+0000","last_became_peered":"2024-09-21T12:21:24.884183+0000","last_unstale":"2024-09-21T12:23:01.529793+0000","last_undegraded":"2024-09-21T12:23:01.529793+0000","last_fullsized":"2024-09-21T12:23:01.529793+0000","mapping_epoch":84,"log_start":"94'1591","ondisk_log_start":"94'1591","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'5758","last_scrub_stamp":"2024-09-21T12:22:48.664234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:22:48.664234+0000","objects_scrubbed":5726,"log_size":4869,"log_dups_size":0,"ondisk_log_size":4869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:12.546965+0000","scrub_duration":1463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415760384,"num_objects":6344,"num_object_clones":0,"num_object_copies":19032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6460,"num_write_kb":409728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":177,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.955280+0000","last_change":"2024-09-21T12:20:17.693585+0000","last_active":"2024-09-21T12:22:57.955280+0000","last_peered":"2024-09-21T12:22:57.955280+0000","last_clean":"2024-09-21T12:22:57.955280+0000","last_became_active":"2024-09-21T12:20:17.693067+0000","last_became_peered":"2024-09-21T12:20:17.693067+0000","last_unstale":"2024-09-21T12:22:57.955280+0000","last_undegraded":"2024-09-21T12:22:57.955280+0000","last_fullsized":"2024-09-21T12:22:57.955280+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:20.707330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":145,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.971770+0000","last_change":"2024-09-21T12:20:15.308741+0000","last_active":"2024-09-21T12:22:57.971770+0000","last_peered":"2024-09-21T12:22:57.971770+0000","last_clean":"2024-09-21T12:22:57.971770+0000","last_became_active":"2024-09-21T12:20:15.308441+0000","last_became_peered":"2024-09-21T12:20:15.308441+0000","last_unstale":"2024-09-21T12:22:57.971770+0000","last_undegraded":"2024-09-21T12:22:57.971770+0000","last_fullsized":"2024-09-21T12:22:57.971770+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":144,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.957481+0000","last_change":"2024-09-21T12:22:34.563365+0000","last_active":"2024-09-21T12:22:57.957481+0000","last_peered":"2024-09-21T12:22:57.957481+0000","last_clean":"2024-09-21T12:22:57.957481+0000","last_became_active":"2024-09-21T12:20:19.309598+0000","last_became_peered":"2024-09-21T12:20:19.309598+0000","last_unstale":"2024-09-21T12:22:57.957481+0000","last_undegraded":"2024-09-21T12:22:57.957481+0000","last_fullsized":"2024-09-21T12:22:57.957481+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:34.563282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:34.563282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:55.460915+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":133,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.972209+0000","last_change":"2024-09-21T12:20:21.593173+0000","last_active":"2024-09-21T12:22:57.972209+0000","last_peered":"2024-09-21T12:22:57.972209+0000","last_clean":"2024-09-21T12:22:57.972209+0000","last_became_active":"2024-09-21T12:20:21.592828+0000","last_became_peered":"2024-09-21T12:20:21.592828+0000","last_unstale":"2024-09-21T12:22:57.972209+0000","last_undegraded":"2024-09-21T12:22:57.972209+0000","last_fullsized":"2024-09-21T12:22:57.972209+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"76'79","reported_seq":44,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.964107+0000","last_change":"2024-09-21T12:22:17.085155+0000","last_active":"2024-09-21T12:22:57.964107+0000","last_peered":"2024-09-21T12:22:57.964107+0000","last_clean":"2024-09-21T12:22:57.964107+0000","last_became_active":"2024-09-21T12:22:17.084732+0000","last_became_peered":"2024-09-21T12:22:17.084732+0000","last_unstale":"2024-09-21T12:22:57.964107+0000","last_undegraded":"2024-09-21T12:22:57.964107+0000","last_fullsized":"2024-09-21T12:22:57.964107+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T12:20:24.850157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:20:24.850157+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:23:20.982807+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"125'6694","reported_seq":6854,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:23:01.552715+0000","last_change":"2024-09-21T12:22:50.126449+0000","last_active":"2024-09-21T12:23:01.552715+0000","last_peered":"2024-09-21T12:23:01.552715+0000","last_clean":"2024-09-21T12:23:01.552715+0000","last_became_active":"2024-09-21T12:21:24.886854+0000","last_became_peered":"2024-09-21T12:21:24.886854+0000","last_unstale":"2024-09-21T12:23:01.552715+0000","last_undegraded":"2024-09-21T12:23:01.552715+0000","last_fullsized":"2024-09-21T12:23:01.552715+0000","mapping_epoch":84,"log_start":"94'2100","ondisk_log_start":"94'2100","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'6045","last_scrub_stamp":"2024-09-21T12:22:50.126349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:22:50.126349+0000","objects_scrubbed":6025,"log_size":4594,"log_dups_size":0,"ondisk_log_size":4594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:09.057671+0000","scrub_duration":1386,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428605440,"num_objects":6540,"num_object_clones":0,"num_object_copies":19620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6694,"num_write_kb":423488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":223,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.971674+0000","last_change":"2024-09-21T12:20:17.308049+0000","last_active":"2024-09-21T12:22:57.971674+0000","last_peered":"2024-09-21T12:22:57.971674+0000","last_clean":"2024-09-21T12:22:57.971674+0000","last_became_active":"2024-09-21T12:20:17.307769+0000","last_became_peered":"2024-09-21T12:20:17.307769+0000","last_unstale":"2024-09-21T12:22:57.971674+0000","last_undegraded":"2024-09-21T12:22:57.971674+0000","last_fullsized":"2024-09-21T12:22:57.971674+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":76,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.955480+0000","last_change":"2024-09-21T12:22:50.922961+0000","last_active":"2024-09-21T12:22:57.955480+0000","last_peered":"2024-09-21T12:22:57.955480+0000","last_clean":"2024-09-21T12:22:57.955480+0000","last_became_active":"2024-09-21T12:22:50.922002+0000","last_became_peered":"2024-09-21T12:22:50.922002+0000","last_unstale":"2024-09-21T12:22:57.955480+0000","last_undegraded":"2024-09-21T12:22:57.955480+0000","last_fullsized":"2024-09-21T12:22:57.955480+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:05.507899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:05.507899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:20.115972+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":162,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.964517+0000","last_change":"2024-09-21T12:21:20.045824+0000","last_active":"2024-09-21T12:22:57.964517+0000","last_peered":"2024-09-21T12:22:57.964517+0000","last_clean":"2024-09-21T12:22:57.964517+0000","last_became_active":"2024-09-21T12:20:14.287998+0000","last_became_peered":"2024-09-21T12:20:14.287998+0000","last_unstale":"2024-09-21T12:22:57.964517+0000","last_undegraded":"2024-09-21T12:22:57.964517+0000","last_fullsized":"2024-09-21T12:22:57.964517+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T12:21:20.045755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.045755+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011011149999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":166,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.957014+0000","last_change":"2024-09-21T12:22:50.921798+0000","last_active":"2024-09-21T12:22:57.957014+0000","last_peered":"2024-09-21T12:22:57.957014+0000","last_clean":"2024-09-21T12:22:57.957014+0000","last_became_active":"2024-09-21T12:22:50.921488+0000","last_became_peered":"2024-09-21T12:22:50.921488+0000","last_unstale":"2024-09-21T12:22:57.957014+0000","last_undegraded":"2024-09-21T12:22:57.957014+0000","last_fullsized":"2024-09-21T12:22:57.957014+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:06.563104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:06.563104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:28.835407+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":152,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.962700+0000","last_change":"2024-09-21T12:22:50.920543+0000","last_active":"2024-09-21T12:22:57.962700+0000","last_peered":"2024-09-21T12:22:57.962700+0000","last_clean":"2024-09-21T12:22:57.962700+0000","last_became_active":"2024-09-21T12:22:50.917558+0000","last_became_peered":"2024-09-21T12:22:50.917558+0000","last_unstale":"2024-09-21T12:22:57.962700+0000","last_undegraded":"2024-09-21T12:22:57.962700+0000","last_fullsized":"2024-09-21T12:22:57.962700+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'79","reported_seq":219,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.959289+0000","last_change":"2024-09-21T12:22:50.922077+0000","last_active":"2024-09-21T12:22:57.959289+0000","last_peered":"2024-09-21T12:22:57.959289+0000","last_clean":"2024-09-21T12:22:57.959289+0000","last_became_active":"2024-09-21T12:22:50.921709+0000","last_became_peered":"2024-09-21T12:22:50.921709+0000","last_unstale":"2024-09-21T12:22:57.959289+0000","last_undegraded":"2024-09-21T12:22:57.959289+0000","last_fullsized":"2024-09-21T12:22:57.959289+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T12:20:24.850157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:20:24.850157+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":21,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"125'6513","reported_seq":9583,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:23:01.732993+0000","last_change":"2024-09-21T12:22:51.045889+0000","last_active":"2024-09-21T12:23:01.732993+0000","last_peered":"2024-09-21T12:23:01.732993+0000","last_clean":"2024-09-21T12:23:01.732993+0000","last_became_active":"2024-09-21T12:22:50.936833+0000","last_became_peered":"2024-09-21T12:22:50.936833+0000","last_unstale":"2024-09-21T12:23:01.732993+0000","last_undegraded":"2024-09-21T12:23:01.732993+0000","last_fullsized":"2024-09-21T12:23:01.732993+0000","mapping_epoch":120,"log_start":"96'2253","ondisk_log_start":"96'2253","created":84,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4260,"log_dups_size":0,"ondisk_log_size":4260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417398784,"num_objects":6369,"num_object_clones":0,"num_object_copies":19107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6513,"num_write_kb":412224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1475,"num_bytes_recovered":96665600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":168,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.970903+0000","last_change":"2024-09-21T12:20:17.297678+0000","last_active":"2024-09-21T12:22:57.970903+0000","last_peered":"2024-09-21T12:22:57.970903+0000","last_clean":"2024-09-21T12:22:57.970903+0000","last_became_active":"2024-09-21T12:20:17.297374+0000","last_became_peered":"2024-09-21T12:20:17.297374+0000","last_unstale":"2024-09-21T12:22:57.970903+0000","last_undegraded":"2024-09-21T12:22:57.970903+0000","last_fullsized":"2024-09-21T12:22:57.970903+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:22.813197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":153,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.973605+0000","last_change":"2024-09-21T12:22:53.116442+0000","last_active":"2024-09-21T12:22:57.973605+0000","last_peered":"2024-09-21T12:22:57.973605+0000","last_clean":"2024-09-21T12:22:57.973605+0000","last_became_active":"2024-09-21T12:20:15.308356+0000","last_became_peered":"2024-09-21T12:20:15.308356+0000","last_unstale":"2024-09-21T12:22:57.973605+0000","last_undegraded":"2024-09-21T12:22:57.973605+0000","last_fullsized":"2024-09-21T12:22:57.973605+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:53.116377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:53.116377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:54.003540+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":162,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.955597+0000","last_change":"2024-09-21T12:21:24.390178+0000","last_active":"2024-09-21T12:22:57.955597+0000","last_peered":"2024-09-21T12:22:57.955597+0000","last_clean":"2024-09-21T12:22:57.955597+0000","last_became_active":"2024-09-21T12:20:14.288358+0000","last_became_peered":"2024-09-21T12:20:14.288358+0000","last_unstale":"2024-09-21T12:22:57.955597+0000","last_undegraded":"2024-09-21T12:22:57.955597+0000","last_fullsized":"2024-09-21T12:22:57.955597+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:21:24.390129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:21:24.390129+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019042449999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":137,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.956581+0000","last_change":"2024-09-21T12:20:19.310164+0000","last_active":"2024-09-21T12:22:57.956581+0000","last_peered":"2024-09-21T12:22:57.956581+0000","last_clean":"2024-09-21T12:22:57.956581+0000","last_became_active":"2024-09-21T12:20:19.309724+0000","last_became_peered":"2024-09-21T12:20:19.309724+0000","last_unstale":"2024-09-21T12:22:57.956581+0000","last_undegraded":"2024-09-21T12:22:57.956581+0000","last_fullsized":"2024-09-21T12:22:57.956581+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.959856+0000","last_change":"2024-09-21T12:20:21.593267+0000","last_active":"2024-09-21T12:22:57.959856+0000","last_peered":"2024-09-21T12:22:57.959856+0000","last_clean":"2024-09-21T12:22:57.959856+0000","last_became_active":"2024-09-21T12:20:21.592982+0000","last_became_peered":"2024-09-21T12:20:21.592982+0000","last_unstale":"2024-09-21T12:22:57.959856+0000","last_undegraded":"2024-09-21T12:22:57.959856+0000","last_fullsized":"2024-09-21T12:22:57.959856+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"76'79","reported_seq":66,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.966579+0000","last_change":"2024-09-21T12:22:50.912481+0000","last_active":"2024-09-21T12:22:57.966579+0000","last_peered":"2024-09-21T12:22:57.966579+0000","last_clean":"2024-09-21T12:22:57.966579+0000","last_became_active":"2024-09-21T12:22:50.912088+0000","last_became_peered":"2024-09-21T12:22:50.912088+0000","last_unstale":"2024-09-21T12:22:57.966579+0000","last_undegraded":"2024-09-21T12:22:57.966579+0000","last_fullsized":"2024-09-21T12:22:57.966579+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:22:18.810791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:22:18.810791+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:19.441983+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"125'6355","reported_seq":6448,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:23:01.524672+0000","last_change":"2024-09-21T12:21:24.889134+0000","last_active":"2024-09-21T12:23:01.524672+0000","last_peered":"2024-09-21T12:23:01.524672+0000","last_clean":"2024-09-21T12:23:01.524672+0000","last_became_active":"2024-09-21T12:21:24.888897+0000","last_became_peered":"2024-09-21T12:21:24.888897+0000","last_unstale":"2024-09-21T12:23:01.524672+0000","last_undegraded":"2024-09-21T12:23:01.524672+0000","last_fullsized":"2024-09-21T12:23:01.524672+0000","mapping_epoch":84,"log_start":"94'1836","ondisk_log_start":"94'1836","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4519,"log_dups_size":0,"ondisk_log_size":4519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407175168,"num_objects":6213,"num_object_clones":0,"num_object_copies":18639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6355,"num_write_kb":402176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":167,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.954506+0000","last_change":"2024-09-21T12:22:49.933621+0000","last_active":"2024-09-21T12:22:57.954506+0000","last_peered":"2024-09-21T12:22:57.954506+0000","last_clean":"2024-09-21T12:22:57.954506+0000","last_became_active":"2024-09-21T12:20:17.299325+0000","last_became_peered":"2024-09-21T12:20:17.299325+0000","last_unstale":"2024-09-21T12:22:57.954506+0000","last_undegraded":"2024-09-21T12:22:57.954506+0000","last_fullsized":"2024-09-21T12:22:57.954506+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:22:49.933382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:49.933382+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:00.818963+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":174,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.953873+0000","last_change":"2024-09-21T12:22:50.929055+0000","last_active":"2024-09-21T12:22:57.953873+0000","last_peered":"2024-09-21T12:22:57.953873+0000","last_clean":"2024-09-21T12:22:57.953873+0000","last_became_active":"2024-09-21T12:22:50.923796+0000","last_became_peered":"2024-09-21T12:22:50.923796+0000","last_unstale":"2024-09-21T12:22:57.953873+0000","last_undegraded":"2024-09-21T12:22:57.953873+0000","last_fullsized":"2024-09-21T12:22:57.953873+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:13.623395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:13.623395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:25.209353+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":324,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:23:00.228144+0000","last_change":"2024-09-21T12:21:22.834972+0000","last_active":"2024-09-21T12:23:00.228144+0000","last_peered":"2024-09-21T12:23:00.228144+0000","last_clean":"2024-09-21T12:23:00.228144+0000","last_became_active":"2024-09-21T12:20:14.287543+0000","last_became_peered":"2024-09-21T12:20:14.287543+0000","last_unstale":"2024-09-21T12:23:00.228144+0000","last_undegraded":"2024-09-21T12:23:00.228144+0000","last_fullsized":"2024-09-21T12:23:00.228144+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-21T12:21:22.834917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:21:22.834917+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":161,"num_read_kb":156,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":144,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.965135+0000","last_change":"2024-09-21T12:22:36.135464+0000","last_active":"2024-09-21T12:22:57.965135+0000","last_peered":"2024-09-21T12:22:57.965135+0000","last_clean":"2024-09-21T12:22:57.965135+0000","last_became_active":"2024-09-21T12:20:19.314217+0000","last_became_peered":"2024-09-21T12:20:19.314217+0000","last_unstale":"2024-09-21T12:22:57.965135+0000","last_undegraded":"2024-09-21T12:22:57.965135+0000","last_fullsized":"2024-09-21T12:22:57.965135+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:36.135390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:36.135390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:57.811533+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":141,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.971280+0000","last_change":"2024-09-21T12:22:57.265262+0000","last_active":"2024-09-21T12:22:57.971280+0000","last_peered":"2024-09-21T12:22:57.971280+0000","last_clean":"2024-09-21T12:22:57.971280+0000","last_became_active":"2024-09-21T12:20:21.311188+0000","last_became_peered":"2024-09-21T12:20:21.311188+0000","last_unstale":"2024-09-21T12:22:57.971280+0000","last_undegraded":"2024-09-21T12:22:57.971280+0000","last_fullsized":"2024-09-21T12:22:57.971280+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:57.265191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:57.265191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:24.950692+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"125'6482","reported_seq":6575,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:23:01.756447+0000","last_change":"2024-09-21T12:21:24.886290+0000","last_active":"2024-09-21T12:23:01.756447+0000","last_peered":"2024-09-21T12:23:01.756447+0000","last_clean":"2024-09-21T12:23:01.756447+0000","last_became_active":"2024-09-21T12:21:24.885986+0000","last_became_peered":"2024-09-21T12:21:24.885986+0000","last_unstale":"2024-09-21T12:23:01.756447+0000","last_undegraded":"2024-09-21T12:23:01.756447+0000","last_fullsized":"2024-09-21T12:23:01.756447+0000","mapping_epoch":84,"log_start":"96'2278","ondisk_log_start":"96'2278","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4204,"log_dups_size":0,"ondisk_log_size":4204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415760384,"num_objects":6344,"num_object_clones":0,"num_object_copies":19032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6482,"num_write_kb":410432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"121'19","reported_seq":279,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.958012+0000","last_change":"2024-09-21T12:22:53.756501+0000","last_active":"2024-09-21T12:22:57.958012+0000","last_peered":"2024-09-21T12:22:57.958012+0000","last_clean":"2024-09-21T12:22:57.958012+0000","last_became_active":"2024-09-21T12:22:50.921690+0000","last_became_peered":"2024-09-21T12:22:50.921690+0000","last_unstale":"2024-09-21T12:22:57.958012+0000","last_undegraded":"2024-09-21T12:22:57.958012+0000","last_fullsized":"2024-09-21T12:22:57.958012+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'19","last_scrub_stamp":"2024-09-21T12:22:53.756438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:53.756438+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:17.096246+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":157,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.964396+0000","last_change":"2024-09-21T12:21:17.107378+0000","last_active":"2024-09-21T12:22:57.964396+0000","last_peered":"2024-09-21T12:22:57.964396+0000","last_clean":"2024-09-21T12:22:57.964396+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:22:57.964396+0000","last_undegraded":"2024-09-21T12:22:57.964396+0000","last_fullsized":"2024-09-21T12:22:57.964396+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:21:17.107309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:17.107309+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":207,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.959516+0000","last_change":"2024-09-21T12:22:16.936605+0000","last_active":"2024-09-21T12:22:57.959516+0000","last_peered":"2024-09-21T12:22:57.959516+0000","last_clean":"2024-09-21T12:22:57.959516+0000","last_became_active":"2024-09-21T12:22:15.049592+0000","last_became_peered":"2024-09-21T12:22:15.049592+0000","last_unstale":"2024-09-21T12:22:57.959516+0000","last_undegraded":"2024-09-21T12:22:57.959516+0000","last_fullsized":"2024-09-21T12:22:57.959516+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:22:16.936548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:22:16.936548+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:36.347732+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.959787+0000","last_change":"2024-09-21T12:20:19.310280+0000","last_active":"2024-09-21T12:22:57.959787+0000","last_peered":"2024-09-21T12:22:57.959787+0000","last_clean":"2024-09-21T12:22:57.959787+0000","last_became_active":"2024-09-21T12:20:19.309859+0000","last_became_peered":"2024-09-21T12:20:19.309859+0000","last_unstale":"2024-09-21T12:22:57.959787+0000","last_undegraded":"2024-09-21T12:22:57.959787+0000","last_fullsized":"2024-09-21T12:22:57.959787+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":161,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.965013+0000","last_change":"2024-09-21T12:22:50.913944+0000","last_active":"2024-09-21T12:22:57.965013+0000","last_peered":"2024-09-21T12:22:57.965013+0000","last_clean":"2024-09-21T12:22:57.965013+0000","last_became_active":"2024-09-21T12:22:50.913493+0000","last_became_peered":"2024-09-21T12:22:50.913493+0000","last_unstale":"2024-09-21T12:22:57.965013+0000","last_undegraded":"2024-09-21T12:22:57.965013+0000","last_fullsized":"2024-09-21T12:22:57.965013+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:41.210247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:41.210247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:55.372874+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"125'6631","reported_seq":6788,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:23:01.553377+0000","last_change":"2024-09-21T12:22:40.358709+0000","last_active":"2024-09-21T12:23:01.553377+0000","last_peered":"2024-09-21T12:23:01.553377+0000","last_clean":"2024-09-21T12:23:01.553377+0000","last_became_active":"2024-09-21T12:21:24.887845+0000","last_became_peered":"2024-09-21T12:21:24.887845+0000","last_unstale":"2024-09-21T12:23:01.553377+0000","last_undegraded":"2024-09-21T12:23:01.553377+0000","last_fullsized":"2024-09-21T12:23:01.553377+0000","mapping_epoch":84,"log_start":"94'2002","ondisk_log_start":"94'2002","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'5424","last_scrub_stamp":"2024-09-21T12:22:40.358604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:22:40.358604+0000","objects_scrubbed":5391,"log_size":4629,"log_dups_size":0,"ondisk_log_size":4629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:53.778779+0000","scrub_duration":1743,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423886872,"num_objects":6469,"num_object_clones":0,"num_object_copies":19407,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6469,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6631,"num_write_kb":419137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":131,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.957926+0000","last_change":"2024-09-21T12:22:50.966517+0000","last_active":"2024-09-21T12:22:57.957926+0000","last_peered":"2024-09-21T12:22:57.957926+0000","last_clean":"2024-09-21T12:22:57.957926+0000","last_became_active":"2024-09-21T12:22:50.914810+0000","last_became_peered":"2024-09-21T12:22:50.914810+0000","last_unstale":"2024-09-21T12:22:57.957926+0000","last_undegraded":"2024-09-21T12:22:57.957926+0000","last_fullsized":"2024-09-21T12:22:57.957926+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:21:19.327640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:21:19.327640+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":76,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.955444+0000","last_change":"2024-09-21T12:22:50.908380+0000","last_active":"2024-09-21T12:22:57.955444+0000","last_peered":"2024-09-21T12:22:57.955444+0000","last_clean":"2024-09-21T12:22:57.955444+0000","last_became_active":"2024-09-21T12:22:50.908060+0000","last_became_peered":"2024-09-21T12:22:50.908060+0000","last_unstale":"2024-09-21T12:22:57.955444+0000","last_undegraded":"2024-09-21T12:22:57.955444+0000","last_fullsized":"2024-09-21T12:22:57.955444+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:05.892153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:05.892153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:32.732340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":165,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.964991+0000","last_change":"2024-09-21T12:22:50.915357+0000","last_active":"2024-09-21T12:22:57.964991+0000","last_peered":"2024-09-21T12:22:57.964991+0000","last_clean":"2024-09-21T12:22:57.964991+0000","last_became_active":"2024-09-21T12:22:50.915081+0000","last_became_peered":"2024-09-21T12:22:50.915081+0000","last_unstale":"2024-09-21T12:22:57.964991+0000","last_undegraded":"2024-09-21T12:22:57.964991+0000","last_fullsized":"2024-09-21T12:22:57.964991+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:07.050620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:07.050620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:17.285948+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":131,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.968489+0000","last_change":"2024-09-21T12:20:21.593145+0000","last_active":"2024-09-21T12:22:57.968489+0000","last_peered":"2024-09-21T12:22:57.968489+0000","last_clean":"2024-09-21T12:22:57.968489+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:22:57.968489+0000","last_undegraded":"2024-09-21T12:22:57.968489+0000","last_fullsized":"2024-09-21T12:22:57.968489+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:31.963984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"76'79","reported_seq":51,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.964057+0000","last_change":"2024-09-21T12:22:17.114985+0000","last_active":"2024-09-21T12:22:57.964057+0000","last_peered":"2024-09-21T12:22:57.964057+0000","last_clean":"2024-09-21T12:22:57.964057+0000","last_became_active":"2024-09-21T12:22:17.087245+0000","last_became_peered":"2024-09-21T12:22:17.087245+0000","last_unstale":"2024-09-21T12:22:57.964057+0000","last_undegraded":"2024-09-21T12:22:57.964057+0000","last_fullsized":"2024-09-21T12:22:57.964057+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T12:20:24.850157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:20:24.850157+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:23:22.031869+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":316358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"125'6573","reported_seq":6666,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:23:01.632575+0000","last_change":"2024-09-21T12:21:24.888960+0000","last_active":"2024-09-21T12:23:01.632575+0000","last_peered":"2024-09-21T12:23:01.632575+0000","last_clean":"2024-09-21T12:23:01.632575+0000","last_became_active":"2024-09-21T12:21:24.888670+0000","last_became_peered":"2024-09-21T12:21:24.888670+0000","last_unstale":"2024-09-21T12:23:01.632575+0000","last_undegraded":"2024-09-21T12:23:01.632575+0000","last_fullsized":"2024-09-21T12:23:01.632575+0000","mapping_epoch":84,"log_start":"90'1189","ondisk_log_start":"90'1189","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":5384,"log_dups_size":0,"ondisk_log_size":5384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":421068800,"num_objects":6425,"num_object_clones":0,"num_object_copies":19275,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6573,"num_write_kb":415936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":160,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.956964+0000","last_change":"2024-09-21T12:22:50.989705+0000","last_active":"2024-09-21T12:22:57.956964+0000","last_peered":"2024-09-21T12:22:57.956964+0000","last_clean":"2024-09-21T12:22:57.956964+0000","last_became_active":"2024-09-21T12:22:50.915202+0000","last_became_peered":"2024-09-21T12:22:50.915202+0000","last_unstale":"2024-09-21T12:22:57.956964+0000","last_undegraded":"2024-09-21T12:22:57.956964+0000","last_fullsized":"2024-09-21T12:22:57.956964+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-21T12:22:01.738653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:01.738653+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:01.755748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":173,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.957893+0000","last_change":"2024-09-21T12:22:50.907175+0000","last_active":"2024-09-21T12:22:57.957893+0000","last_peered":"2024-09-21T12:22:57.957893+0000","last_clean":"2024-09-21T12:22:57.957893+0000","last_became_active":"2024-09-21T12:22:50.906886+0000","last_became_peered":"2024-09-21T12:22:50.906886+0000","last_unstale":"2024-09-21T12:22:57.957893+0000","last_undegraded":"2024-09-21T12:22:57.957893+0000","last_fullsized":"2024-09-21T12:22:57.957893+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:16.670500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:16.670500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:25.660662+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":158,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.971371+0000","last_change":"2024-09-21T12:22:50.928575+0000","last_active":"2024-09-21T12:22:57.971371+0000","last_peered":"2024-09-21T12:22:57.971371+0000","last_clean":"2024-09-21T12:22:57.971371+0000","last_became_active":"2024-09-21T12:22:50.927438+0000","last_became_peered":"2024-09-21T12:22:50.927438+0000","last_unstale":"2024-09-21T12:22:57.971371+0000","last_undegraded":"2024-09-21T12:22:57.971371+0000","last_fullsized":"2024-09-21T12:22:57.971371+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":154,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.955879+0000","last_change":"2024-09-21T12:22:50.914246+0000","last_active":"2024-09-21T12:22:57.955879+0000","last_peered":"2024-09-21T12:22:57.955879+0000","last_clean":"2024-09-21T12:22:57.955879+0000","last_became_active":"2024-09-21T12:22:50.913898+0000","last_became_peered":"2024-09-21T12:22:50.913898+0000","last_unstale":"2024-09-21T12:22:57.955879+0000","last_undegraded":"2024-09-21T12:22:57.955879+0000","last_fullsized":"2024-09-21T12:22:57.955879+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"76'79","reported_seq":205,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.960929+0000","last_change":"2024-09-21T12:22:17.073301+0000","last_active":"2024-09-21T12:22:57.960929+0000","last_peered":"2024-09-21T12:22:57.960929+0000","last_clean":"2024-09-21T12:22:57.960929+0000","last_became_active":"2024-09-21T12:22:17.072872+0000","last_became_peered":"2024-09-21T12:22:17.072872+0000","last_unstale":"2024-09-21T12:22:57.960929+0000","last_undegraded":"2024-09-21T12:22:57.960929+0000","last_fullsized":"2024-09-21T12:22:57.960929+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T12:20:24.850157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:20:24.850157+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:23:20.982847+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":21,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.f","version":"125'6649","reported_seq":6742,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:23:01.774413+0000","last_change":"2024-09-21T12:21:24.887418+0000","last_active":"2024-09-21T12:23:01.774413+0000","last_peered":"2024-09-21T12:23:01.774413+0000","last_clean":"2024-09-21T12:23:01.774413+0000","last_became_active":"2024-09-21T12:21:24.887000+0000","last_became_peered":"2024-09-21T12:21:24.887000+0000","last_unstale":"2024-09-21T12:23:01.774413+0000","last_undegraded":"2024-09-21T12:23:01.774413+0000","last_fullsized":"2024-09-21T12:23:01.774413+0000","mapping_epoch":84,"log_start":"96'2451","ondisk_log_start":"96'2451","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4198,"log_dups_size":0,"ondisk_log_size":4198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426967040,"num_objects":6515,"num_object_clones":0,"num_object_copies":19545,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6649,"num_write_kb":421248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":127,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.968801+0000","last_change":"2024-09-21T12:22:50.962856+0000","last_active":"2024-09-21T12:22:57.968801+0000","last_peered":"2024-09-21T12:22:57.968801+0000","last_clean":"2024-09-21T12:22:57.968801+0000","last_became_active":"2024-09-21T12:22:50.909221+0000","last_became_peered":"2024-09-21T12:22:50.909221+0000","last_unstale":"2024-09-21T12:22:57.968801+0000","last_undegraded":"2024-09-21T12:22:57.968801+0000","last_fullsized":"2024-09-21T12:22:57.968801+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":172,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.963721+0000","last_change":"2024-09-21T12:22:50.928240+0000","last_active":"2024-09-21T12:22:57.963721+0000","last_peered":"2024-09-21T12:22:57.963721+0000","last_clean":"2024-09-21T12:22:57.963721+0000","last_became_active":"2024-09-21T12:22:50.927946+0000","last_became_peered":"2024-09-21T12:22:50.927946+0000","last_unstale":"2024-09-21T12:22:57.963721+0000","last_undegraded":"2024-09-21T12:22:57.963721+0000","last_fullsized":"2024-09-21T12:22:57.963721+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:08.999015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:08.999015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:10.244238+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":179,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.956919+0000","last_change":"2024-09-21T12:20:19.308246+0000","last_active":"2024-09-21T12:22:57.956919+0000","last_peered":"2024-09-21T12:22:57.956919+0000","last_clean":"2024-09-21T12:22:57.956919+0000","last_became_active":"2024-09-21T12:20:19.307924+0000","last_became_peered":"2024-09-21T12:20:19.307924+0000","last_unstale":"2024-09-21T12:22:57.956919+0000","last_undegraded":"2024-09-21T12:22:57.956919+0000","last_fullsized":"2024-09-21T12:22:57.956919+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":76,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.969815+0000","last_change":"2024-09-21T12:22:50.905752+0000","last_active":"2024-09-21T12:22:57.969815+0000","last_peered":"2024-09-21T12:22:57.969815+0000","last_clean":"2024-09-21T12:22:57.969815+0000","last_became_active":"2024-09-21T12:22:50.905470+0000","last_became_peered":"2024-09-21T12:22:50.905470+0000","last_unstale":"2024-09-21T12:22:57.969815+0000","last_undegraded":"2024-09-21T12:22:57.969815+0000","last_fullsized":"2024-09-21T12:22:57.969815+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:14.809555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.809555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:35.716649+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"125'6690","reported_seq":12339,"reported_epoch":125,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-21T12:23:01.614430+0000","last_change":"2024-09-21T12:22:51.067180+0000","last_active":"2024-09-21T12:23:01.614430+0000","last_peered":"2024-09-21T12:23:01.614430+0000","last_clean":"2024-09-21T12:22:24.099457+0000","last_became_active":"2024-09-21T12:22:50.996040+0000","last_became_peered":"2024-09-21T12:22:50.996040+0000","last_unstale":"2024-09-21T12:23:01.614430+0000","last_undegraded":"2024-09-21T12:23:01.614430+0000","last_fullsized":"2024-09-21T12:22:50.894596+0000","mapping_epoch":120,"log_start":"90'1301","ondisk_log_start":"90'1301","created":84,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":5389,"log_dups_size":0,"ondisk_log_size":5389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429654016,"num_objects":6556,"num_object_clones":0,"num_object_copies":19668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":851,"num_objects_unfound":0,"num_objects_dirty":6556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6690,"num_write_kb":423872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2770,"num_bytes_recovered":181534720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,4,7","objects":851},{"shards":"2,7","objects":5705}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":216,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.955182+0000","last_change":"2024-09-21T12:22:51.052784+0000","last_active":"2024-09-21T12:22:57.955182+0000","last_peered":"2024-09-21T12:22:57.955182+0000","last_clean":"2024-09-21T12:22:57.955182+0000","last_became_active":"2024-09-21T12:22:50.922343+0000","last_became_peered":"2024-09-21T12:22:50.922343+0000","last_unstale":"2024-09-21T12:22:57.955182+0000","last_undegraded":"2024-09-21T12:22:57.955182+0000","last_fullsized":"2024-09-21T12:22:57.955182+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:21:18.243868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:21:18.243868+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":152,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.964629+0000","last_change":"2024-09-21T12:22:48.173073+0000","last_active":"2024-09-21T12:22:57.964629+0000","last_peered":"2024-09-21T12:22:57.964629+0000","last_clean":"2024-09-21T12:22:57.964629+0000","last_became_active":"2024-09-21T12:20:15.302262+0000","last_became_peered":"2024-09-21T12:20:15.302262+0000","last_unstale":"2024-09-21T12:22:57.964629+0000","last_undegraded":"2024-09-21T12:22:57.964629+0000","last_fullsized":"2024-09-21T12:22:57.964629+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:48.173022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:48.173022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:48.961679+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":166,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.956507+0000","last_change":"2024-09-21T12:22:50.926450+0000","last_active":"2024-09-21T12:22:57.956507+0000","last_peered":"2024-09-21T12:22:57.956507+0000","last_clean":"2024-09-21T12:22:57.956507+0000","last_became_active":"2024-09-21T12:22:50.925974+0000","last_became_peered":"2024-09-21T12:22:50.925974+0000","last_unstale":"2024-09-21T12:22:57.956507+0000","last_undegraded":"2024-09-21T12:22:57.956507+0000","last_fullsized":"2024-09-21T12:22:57.956507+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:11.892850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:11.892850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:27.007736+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":133,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.953785+0000","last_change":"2024-09-21T12:20:21.311799+0000","last_active":"2024-09-21T12:22:57.953785+0000","last_peered":"2024-09-21T12:22:57.953785+0000","last_clean":"2024-09-21T12:22:57.953785+0000","last_became_active":"2024-09-21T12:20:21.311493+0000","last_became_peered":"2024-09-21T12:20:21.311493+0000","last_unstale":"2024-09-21T12:22:57.953785+0000","last_undegraded":"2024-09-21T12:22:57.953785+0000","last_fullsized":"2024-09-21T12:22:57.953785+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:23.727080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"125'6690","reported_seq":16136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:23:01.767605+0000","last_change":"2024-09-21T12:22:50.949717+0000","last_active":"2024-09-21T12:23:01.767605+0000","last_peered":"2024-09-21T12:23:01.767605+0000","last_clean":"2024-09-21T12:23:01.767605+0000","last_became_active":"2024-09-21T12:22:50.924585+0000","last_became_peered":"2024-09-21T12:22:50.924585+0000","last_unstale":"2024-09-21T12:23:01.767605+0000","last_undegraded":"2024-09-21T12:23:01.767605+0000","last_fullsized":"2024-09-21T12:23:01.767605+0000","mapping_epoch":120,"log_start":"97'2480","ondisk_log_start":"97'2480","created":84,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4210,"log_dups_size":0,"ondisk_log_size":4210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427294720,"num_objects":6520,"num_object_clones":0,"num_object_copies":19560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6690,"num_write_kb":422720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4664,"num_bytes_recovered":305659904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":179,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.957574+0000","last_change":"2024-09-21T12:22:42.707274+0000","last_active":"2024-09-21T12:22:57.957574+0000","last_peered":"2024-09-21T12:22:57.957574+0000","last_clean":"2024-09-21T12:22:57.957574+0000","last_became_active":"2024-09-21T12:20:17.692867+0000","last_became_peered":"2024-09-21T12:20:17.692867+0000","last_unstale":"2024-09-21T12:22:57.957574+0000","last_undegraded":"2024-09-21T12:22:57.957574+0000","last_fullsized":"2024-09-21T12:22:57.957574+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-21T12:22:42.707175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:42.707175+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:46.335757+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":144,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.960132+0000","last_change":"2024-09-21T12:20:15.300854+0000","last_active":"2024-09-21T12:22:57.960132+0000","last_peered":"2024-09-21T12:22:57.960132+0000","last_clean":"2024-09-21T12:22:57.960132+0000","last_became_active":"2024-09-21T12:20:15.299333+0000","last_became_peered":"2024-09-21T12:20:15.299333+0000","last_unstale":"2024-09-21T12:22:57.960132+0000","last_undegraded":"2024-09-21T12:22:57.960132+0000","last_fullsized":"2024-09-21T12:22:57.960132+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":137,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.955952+0000","last_change":"2024-09-21T12:20:19.307316+0000","last_active":"2024-09-21T12:22:57.955952+0000","last_peered":"2024-09-21T12:22:57.955952+0000","last_clean":"2024-09-21T12:22:57.955952+0000","last_became_active":"2024-09-21T12:20:19.306952+0000","last_became_peered":"2024-09-21T12:20:19.306952+0000","last_unstale":"2024-09-21T12:22:57.955952+0000","last_undegraded":"2024-09-21T12:22:57.955952+0000","last_fullsized":"2024-09-21T12:22:57.955952+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.959466+0000","last_change":"2024-09-21T12:20:21.314891+0000","last_active":"2024-09-21T12:22:57.959466+0000","last_peered":"2024-09-21T12:22:57.959466+0000","last_clean":"2024-09-21T12:22:57.959466+0000","last_became_active":"2024-09-21T12:20:21.314545+0000","last_became_peered":"2024-09-21T12:20:21.314545+0000","last_unstale":"2024-09-21T12:22:57.959466+0000","last_undegraded":"2024-09-21T12:22:57.959466+0000","last_fullsized":"2024-09-21T12:22:57.959466+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"125'6599","reported_seq":6692,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:23:01.770002+0000","last_change":"2024-09-21T12:21:24.888714+0000","last_active":"2024-09-21T12:23:01.770002+0000","last_peered":"2024-09-21T12:23:01.770002+0000","last_clean":"2024-09-21T12:23:01.770002+0000","last_became_active":"2024-09-21T12:21:24.888445+0000","last_became_peered":"2024-09-21T12:21:24.888445+0000","last_unstale":"2024-09-21T12:23:01.770002+0000","last_undegraded":"2024-09-21T12:23:01.770002+0000","last_fullsized":"2024-09-21T12:23:01.770002+0000","mapping_epoch":84,"log_start":"98'2660","ondisk_log_start":"98'2660","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":3939,"log_dups_size":0,"ondisk_log_size":3939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423821312,"num_objects":6467,"num_object_clones":0,"num_object_copies":19401,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6599,"num_write_kb":418112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":143,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.972289+0000","last_change":"2024-09-21T12:22:50.947094+0000","last_active":"2024-09-21T12:22:57.972289+0000","last_peered":"2024-09-21T12:22:57.972289+0000","last_clean":"2024-09-21T12:22:57.972289+0000","last_became_active":"2024-09-21T12:22:50.910973+0000","last_became_peered":"2024-09-21T12:22:50.910973+0000","last_unstale":"2024-09-21T12:22:57.972289+0000","last_undegraded":"2024-09-21T12:22:57.972289+0000","last_fullsized":"2024-09-21T12:22:57.972289+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:22:09.782437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:09.782437+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:24.496058+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":173,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.960283+0000","last_change":"2024-09-21T12:22:50.928139+0000","last_active":"2024-09-21T12:22:57.960283+0000","last_peered":"2024-09-21T12:22:57.960283+0000","last_clean":"2024-09-21T12:22:57.960283+0000","last_became_active":"2024-09-21T12:22:50.927859+0000","last_became_peered":"2024-09-21T12:22:50.927859+0000","last_unstale":"2024-09-21T12:22:57.960283+0000","last_undegraded":"2024-09-21T12:22:57.960283+0000","last_fullsized":"2024-09-21T12:22:57.960283+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:20.865749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.865749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":145,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.960566+0000","last_change":"2024-09-21T12:21:23.393320+0000","last_active":"2024-09-21T12:22:57.960566+0000","last_peered":"2024-09-21T12:22:57.960566+0000","last_clean":"2024-09-21T12:22:57.960566+0000","last_became_active":"2024-09-21T12:20:19.312572+0000","last_became_peered":"2024-09-21T12:20:19.312572+0000","last_unstale":"2024-09-21T12:22:57.960566+0000","last_undegraded":"2024-09-21T12:22:57.960566+0000","last_fullsized":"2024-09-21T12:22:57.960566+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.393270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.393270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":139,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.963376+0000","last_change":"2024-09-21T12:22:36.006177+0000","last_active":"2024-09-21T12:22:57.963376+0000","last_peered":"2024-09-21T12:22:57.963376+0000","last_clean":"2024-09-21T12:22:57.963376+0000","last_became_active":"2024-09-21T12:20:21.317223+0000","last_became_peered":"2024-09-21T12:20:21.317223+0000","last_unstale":"2024-09-21T12:22:57.963376+0000","last_undegraded":"2024-09-21T12:22:57.963376+0000","last_fullsized":"2024-09-21T12:22:57.963376+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:36.006109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:36.006109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:38.949250+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"125'6511","reported_seq":6604,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:23:01.606096+0000","last_change":"2024-09-21T12:21:24.888120+0000","last_active":"2024-09-21T12:23:01.606096+0000","last_peered":"2024-09-21T12:23:01.606096+0000","last_clean":"2024-09-21T12:23:01.606096+0000","last_became_active":"2024-09-21T12:21:24.887488+0000","last_became_peered":"2024-09-21T12:21:24.887488+0000","last_unstale":"2024-09-21T12:23:01.606096+0000","last_undegraded":"2024-09-21T12:23:01.606096+0000","last_fullsized":"2024-09-21T12:23:01.606096+0000","mapping_epoch":84,"log_start":"96'2382","ondisk_log_start":"96'2382","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4129,"log_dups_size":0,"ondisk_log_size":4129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417267712,"num_objects":6367,"num_object_clones":0,"num_object_copies":19101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6511,"num_write_kb":412096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":173,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.964257+0000","last_change":"2024-09-21T12:22:00.134130+0000","last_active":"2024-09-21T12:22:57.964257+0000","last_peered":"2024-09-21T12:22:57.964257+0000","last_clean":"2024-09-21T12:22:57.964257+0000","last_became_active":"2024-09-21T12:20:17.692870+0000","last_became_peered":"2024-09-21T12:20:17.692870+0000","last_unstale":"2024-09-21T12:22:57.964257+0000","last_undegraded":"2024-09-21T12:22:57.964257+0000","last_fullsized":"2024-09-21T12:22:57.964257+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:22:00.134062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:00.134062+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:12.498238+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":203,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.960182+0000","last_change":"2024-09-21T12:22:51.052311+0000","last_active":"2024-09-21T12:22:57.960182+0000","last_peered":"2024-09-21T12:22:57.960182+0000","last_clean":"2024-09-21T12:22:57.960182+0000","last_became_active":"2024-09-21T12:22:50.927141+0000","last_became_peered":"2024-09-21T12:22:50.927141+0000","last_unstale":"2024-09-21T12:22:57.960182+0000","last_undegraded":"2024-09-21T12:22:57.960182+0000","last_fullsized":"2024-09-21T12:22:57.960182+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-21T12:22:14.920932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.920932+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:17.225412+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"121'11","reported_seq":211,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.960969+0000","last_change":"2024-09-21T12:22:50.915762+0000","last_active":"2024-09-21T12:22:57.960969+0000","last_peered":"2024-09-21T12:22:57.960969+0000","last_clean":"2024-09-21T12:22:57.960969+0000","last_became_active":"2024-09-21T12:22:50.908371+0000","last_became_peered":"2024-09-21T12:22:50.908371+0000","last_unstale":"2024-09-21T12:22:57.960969+0000","last_undegraded":"2024-09-21T12:22:57.960969+0000","last_fullsized":"2024-09-21T12:22:57.960969+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":140,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.964270+0000","last_change":"2024-09-21T12:22:45.223570+0000","last_active":"2024-09-21T12:22:57.964270+0000","last_peered":"2024-09-21T12:22:57.964270+0000","last_clean":"2024-09-21T12:22:57.964270+0000","last_became_active":"2024-09-21T12:20:21.592576+0000","last_became_peered":"2024-09-21T12:20:21.592576+0000","last_unstale":"2024-09-21T12:22:57.964270+0000","last_undegraded":"2024-09-21T12:22:57.964270+0000","last_fullsized":"2024-09-21T12:22:57.964270+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:45.223396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:45.223396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:05.217553+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":160,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.957416+0000","last_change":"2024-09-21T12:22:45.726349+0000","last_active":"2024-09-21T12:22:57.957416+0000","last_peered":"2024-09-21T12:22:57.957416+0000","last_clean":"2024-09-21T12:22:57.957416+0000","last_became_active":"2024-09-21T12:20:15.303480+0000","last_became_peered":"2024-09-21T12:20:15.303480+0000","last_unstale":"2024-09-21T12:22:57.957416+0000","last_undegraded":"2024-09-21T12:22:57.957416+0000","last_fullsized":"2024-09-21T12:22:57.957416+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:45.726268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:45.726268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:10.634112+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":156,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.971811+0000","last_change":"2024-09-21T12:22:36.999995+0000","last_active":"2024-09-21T12:22:57.971811+0000","last_peered":"2024-09-21T12:22:57.971811+0000","last_clean":"2024-09-21T12:22:57.971811+0000","last_became_active":"2024-09-21T12:20:17.694628+0000","last_became_peered":"2024-09-21T12:20:17.694628+0000","last_unstale":"2024-09-21T12:22:57.971811+0000","last_undegraded":"2024-09-21T12:22:57.971811+0000","last_fullsized":"2024-09-21T12:22:57.971811+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-21T12:22:36.999705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:36.999705+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:37.754032+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":144,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.965739+0000","last_change":"2024-09-21T12:22:06.061142+0000","last_active":"2024-09-21T12:22:57.965739+0000","last_peered":"2024-09-21T12:22:57.965739+0000","last_clean":"2024-09-21T12:22:57.965739+0000","last_became_active":"2024-09-21T12:20:19.313882+0000","last_became_peered":"2024-09-21T12:20:19.313882+0000","last_unstale":"2024-09-21T12:22:57.965739+0000","last_undegraded":"2024-09-21T12:22:57.965739+0000","last_fullsized":"2024-09-21T12:22:57.965739+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:06.061092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:06.061092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:11.282607+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":153,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.959723+0000","last_change":"2024-09-21T12:22:50.904801+0000","last_active":"2024-09-21T12:22:57.959723+0000","last_peered":"2024-09-21T12:22:57.959723+0000","last_clean":"2024-09-21T12:22:57.959723+0000","last_became_active":"2024-09-21T12:22:50.904404+0000","last_became_peered":"2024-09-21T12:22:50.904404+0000","last_unstale":"2024-09-21T12:22:57.959723+0000","last_undegraded":"2024-09-21T12:22:57.959723+0000","last_fullsized":"2024-09-21T12:22:57.959723+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":165,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.954591+0000","last_change":"2024-09-21T12:21:24.093710+0000","last_active":"2024-09-21T12:22:57.954591+0000","last_peered":"2024-09-21T12:22:57.954591+0000","last_clean":"2024-09-21T12:22:57.954591+0000","last_became_active":"2024-09-21T12:20:17.695898+0000","last_became_peered":"2024-09-21T12:20:17.695898+0000","last_unstale":"2024-09-21T12:22:57.954591+0000","last_undegraded":"2024-09-21T12:22:57.954591+0000","last_fullsized":"2024-09-21T12:22:57.954591+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:21:24.093656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:21:24.093656+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":173,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.961043+0000","last_change":"2024-09-21T12:22:50.928059+0000","last_active":"2024-09-21T12:22:57.961043+0000","last_peered":"2024-09-21T12:22:57.961043+0000","last_clean":"2024-09-21T12:22:57.961043+0000","last_became_active":"2024-09-21T12:22:50.927649+0000","last_became_peered":"2024-09-21T12:22:50.927649+0000","last_unstale":"2024-09-21T12:22:57.961043+0000","last_undegraded":"2024-09-21T12:22:57.961043+0000","last_fullsized":"2024-09-21T12:22:57.961043+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:07.084008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:07.084008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:16.013845+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":135,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.971640+0000","last_change":"2024-09-21T12:20:19.312248+0000","last_active":"2024-09-21T12:22:57.971640+0000","last_peered":"2024-09-21T12:22:57.971640+0000","last_clean":"2024-09-21T12:22:57.971640+0000","last_became_active":"2024-09-21T12:20:19.311947+0000","last_became_peered":"2024-09-21T12:20:19.311947+0000","last_unstale":"2024-09-21T12:22:57.971640+0000","last_undegraded":"2024-09-21T12:22:57.971640+0000","last_fullsized":"2024-09-21T12:22:57.971640+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":162,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.955496+0000","last_change":"2024-09-21T12:22:50.925237+0000","last_active":"2024-09-21T12:22:57.955496+0000","last_peered":"2024-09-21T12:22:57.955496+0000","last_clean":"2024-09-21T12:22:57.955496+0000","last_became_active":"2024-09-21T12:22:50.924931+0000","last_became_peered":"2024-09-21T12:22:50.924931+0000","last_unstale":"2024-09-21T12:22:57.955496+0000","last_undegraded":"2024-09-21T12:22:57.955496+0000","last_fullsized":"2024-09-21T12:22:57.955496+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:18.797578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:18.797578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:25.968048+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":164,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.964914+0000","last_change":"2024-09-21T12:22:39.201717+0000","last_active":"2024-09-21T12:22:57.964914+0000","last_peered":"2024-09-21T12:22:57.964914+0000","last_clean":"2024-09-21T12:22:57.964914+0000","last_became_active":"2024-09-21T12:20:17.694568+0000","last_became_peered":"2024-09-21T12:20:17.694568+0000","last_unstale":"2024-09-21T12:22:57.964914+0000","last_undegraded":"2024-09-21T12:22:57.964914+0000","last_fullsized":"2024-09-21T12:22:57.964914+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:22:39.201667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:39.201667+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:07.627492+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":153,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.956776+0000","last_change":"2024-09-21T12:22:53.989409+0000","last_active":"2024-09-21T12:22:57.956776+0000","last_peered":"2024-09-21T12:22:57.956776+0000","last_clean":"2024-09-21T12:22:57.956776+0000","last_became_active":"2024-09-21T12:20:15.297519+0000","last_became_peered":"2024-09-21T12:20:15.297519+0000","last_unstale":"2024-09-21T12:22:57.956776+0000","last_undegraded":"2024-09-21T12:22:57.956776+0000","last_fullsized":"2024-09-21T12:22:57.956776+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:53.989318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:53.989318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:15.527166+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":177,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.968730+0000","last_change":"2024-09-21T12:20:19.305689+0000","last_active":"2024-09-21T12:22:57.968730+0000","last_peered":"2024-09-21T12:22:57.968730+0000","last_clean":"2024-09-21T12:22:57.968730+0000","last_became_active":"2024-09-21T12:20:19.305364+0000","last_became_peered":"2024-09-21T12:20:19.305364+0000","last_unstale":"2024-09-21T12:22:57.968730+0000","last_undegraded":"2024-09-21T12:22:57.968730+0000","last_fullsized":"2024-09-21T12:22:57.968730+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":68,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.955964+0000","last_change":"2024-09-21T12:22:50.925108+0000","last_active":"2024-09-21T12:22:57.955964+0000","last_peered":"2024-09-21T12:22:57.955964+0000","last_clean":"2024-09-21T12:22:57.955964+0000","last_became_active":"2024-09-21T12:22:50.924770+0000","last_became_peered":"2024-09-21T12:22:50.924770+0000","last_unstale":"2024-09-21T12:22:57.955964+0000","last_undegraded":"2024-09-21T12:22:57.955964+0000","last_fullsized":"2024-09-21T12:22:57.955964+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":162,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.972734+0000","last_change":"2024-09-21T12:22:40.026248+0000","last_active":"2024-09-21T12:22:57.972734+0000","last_peered":"2024-09-21T12:22:57.972734+0000","last_clean":"2024-09-21T12:22:57.972734+0000","last_became_active":"2024-09-21T12:20:17.693209+0000","last_became_peered":"2024-09-21T12:20:17.693209+0000","last_unstale":"2024-09-21T12:22:57.972734+0000","last_undegraded":"2024-09-21T12:22:57.972734+0000","last_fullsized":"2024-09-21T12:22:57.972734+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:22:40.026037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:40.026037+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:52.898127+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":173,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.959693+0000","last_change":"2024-09-21T12:22:50.908560+0000","last_active":"2024-09-21T12:22:57.959693+0000","last_peered":"2024-09-21T12:22:57.959693+0000","last_clean":"2024-09-21T12:22:57.959693+0000","last_became_active":"2024-09-21T12:22:50.908238+0000","last_became_peered":"2024-09-21T12:22:50.908238+0000","last_unstale":"2024-09-21T12:22:57.959693+0000","last_undegraded":"2024-09-21T12:22:57.959693+0000","last_fullsized":"2024-09-21T12:22:57.959693+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:15.938093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:15.938093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":186,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.964692+0000","last_change":"2024-09-21T12:22:46.216242+0000","last_active":"2024-09-21T12:22:57.964692+0000","last_peered":"2024-09-21T12:22:57.964692+0000","last_clean":"2024-09-21T12:22:57.964692+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:22:57.964692+0000","last_undegraded":"2024-09-21T12:22:57.964692+0000","last_fullsized":"2024-09-21T12:22:57.964692+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:22:46.216153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:46.216153+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:08.790143+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":133,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.954006+0000","last_change":"2024-09-21T12:20:21.309936+0000","last_active":"2024-09-21T12:22:57.954006+0000","last_peered":"2024-09-21T12:22:57.954006+0000","last_clean":"2024-09-21T12:22:57.954006+0000","last_became_active":"2024-09-21T12:20:21.309385+0000","last_became_peered":"2024-09-21T12:20:21.309385+0000","last_unstale":"2024-09-21T12:22:57.954006+0000","last_undegraded":"2024-09-21T12:22:57.954006+0000","last_fullsized":"2024-09-21T12:22:57.954006+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":131,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.958073+0000","last_change":"2024-09-21T12:22:50.922549+0000","last_active":"2024-09-21T12:22:57.958073+0000","last_peered":"2024-09-21T12:22:57.958073+0000","last_clean":"2024-09-21T12:22:57.958073+0000","last_became_active":"2024-09-21T12:22:50.906473+0000","last_became_peered":"2024-09-21T12:22:50.906473+0000","last_unstale":"2024-09-21T12:22:57.958073+0000","last_undegraded":"2024-09-21T12:22:57.958073+0000","last_fullsized":"2024-09-21T12:22:57.958073+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":68,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.959878+0000","last_change":"2024-09-21T12:22:50.904173+0000","last_active":"2024-09-21T12:22:57.959878+0000","last_peered":"2024-09-21T12:22:57.959878+0000","last_clean":"2024-09-21T12:22:57.959878+0000","last_became_active":"2024-09-21T12:22:50.903705+0000","last_became_peered":"2024-09-21T12:22:50.903705+0000","last_unstale":"2024-09-21T12:22:57.959878+0000","last_undegraded":"2024-09-21T12:22:57.959878+0000","last_fullsized":"2024-09-21T12:22:57.959878+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":137,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.966618+0000","last_change":"2024-09-21T12:20:19.315714+0000","last_active":"2024-09-21T12:22:57.966618+0000","last_peered":"2024-09-21T12:22:57.966618+0000","last_clean":"2024-09-21T12:22:57.966618+0000","last_became_active":"2024-09-21T12:20:19.315418+0000","last_became_peered":"2024-09-21T12:20:19.315418+0000","last_unstale":"2024-09-21T12:22:57.966618+0000","last_undegraded":"2024-09-21T12:22:57.966618+0000","last_fullsized":"2024-09-21T12:22:57.966618+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":139,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.962139+0000","last_change":"2024-09-21T12:22:50.143669+0000","last_active":"2024-09-21T12:22:57.962139+0000","last_peered":"2024-09-21T12:22:57.962139+0000","last_clean":"2024-09-21T12:22:57.962139+0000","last_became_active":"2024-09-21T12:20:21.310053+0000","last_became_peered":"2024-09-21T12:20:21.310053+0000","last_unstale":"2024-09-21T12:22:57.962139+0000","last_undegraded":"2024-09-21T12:22:57.962139+0000","last_fullsized":"2024-09-21T12:22:57.962139+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:50.143628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:50.143628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:00.552274+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":157,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.966737+0000","last_change":"2024-09-21T12:20:17.299367+0000","last_active":"2024-09-21T12:22:57.966737+0000","last_peered":"2024-09-21T12:22:57.966737+0000","last_clean":"2024-09-21T12:22:57.966737+0000","last_became_active":"2024-09-21T12:20:17.299015+0000","last_became_peered":"2024-09-21T12:20:17.299015+0000","last_unstale":"2024-09-21T12:22:57.966737+0000","last_undegraded":"2024-09-21T12:22:57.966737+0000","last_fullsized":"2024-09-21T12:22:57.966737+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":173,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.960036+0000","last_change":"2024-09-21T12:22:50.922328+0000","last_active":"2024-09-21T12:22:57.960036+0000","last_peered":"2024-09-21T12:22:57.960036+0000","last_clean":"2024-09-21T12:22:57.960036+0000","last_became_active":"2024-09-21T12:22:50.922019+0000","last_became_peered":"2024-09-21T12:22:50.922019+0000","last_unstale":"2024-09-21T12:22:57.960036+0000","last_undegraded":"2024-09-21T12:22:57.960036+0000","last_fullsized":"2024-09-21T12:22:57.960036+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:05.067280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:05.067280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:20.660344+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":143,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.963464+0000","last_change":"2024-09-21T12:21:20.931077+0000","last_active":"2024-09-21T12:22:57.963464+0000","last_peered":"2024-09-21T12:22:57.963464+0000","last_clean":"2024-09-21T12:22:57.963464+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:22:57.963464+0000","last_undegraded":"2024-09-21T12:22:57.963464+0000","last_fullsized":"2024-09-21T12:22:57.963464+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:20.931025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.931025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":133,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.953880+0000","last_change":"2024-09-21T12:20:21.308003+0000","last_active":"2024-09-21T12:22:57.953880+0000","last_peered":"2024-09-21T12:22:57.953880+0000","last_clean":"2024-09-21T12:22:57.953880+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:22:57.953880+0000","last_undegraded":"2024-09-21T12:22:57.953880+0000","last_fullsized":"2024-09-21T12:22:57.953880+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:24.738749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":166,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.962378+0000","last_change":"2024-09-21T12:22:35.026113+0000","last_active":"2024-09-21T12:22:57.962378+0000","last_peered":"2024-09-21T12:22:57.962378+0000","last_clean":"2024-09-21T12:22:57.962378+0000","last_became_active":"2024-09-21T12:20:17.694794+0000","last_became_peered":"2024-09-21T12:20:17.694794+0000","last_unstale":"2024-09-21T12:22:57.962378+0000","last_undegraded":"2024-09-21T12:22:57.962378+0000","last_fullsized":"2024-09-21T12:22:57.962378+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:22:35.026007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:35.026007+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:01.068751+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":152,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.965781+0000","last_change":"2024-09-21T12:21:55.172399+0000","last_active":"2024-09-21T12:22:57.965781+0000","last_peered":"2024-09-21T12:22:57.965781+0000","last_clean":"2024-09-21T12:22:57.965781+0000","last_became_active":"2024-09-21T12:20:15.295968+0000","last_became_peered":"2024-09-21T12:20:15.295968+0000","last_unstale":"2024-09-21T12:22:57.965781+0000","last_undegraded":"2024-09-21T12:22:57.965781+0000","last_fullsized":"2024-09-21T12:22:57.965781+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:55.172344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:55.172344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:22.388105+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":157,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.959087+0000","last_change":"2024-09-21T12:22:50.921970+0000","last_active":"2024-09-21T12:22:57.959087+0000","last_peered":"2024-09-21T12:22:57.959087+0000","last_clean":"2024-09-21T12:22:57.959087+0000","last_became_active":"2024-09-21T12:22:50.921604+0000","last_became_peered":"2024-09-21T12:22:50.921604+0000","last_unstale":"2024-09-21T12:22:57.959087+0000","last_undegraded":"2024-09-21T12:22:57.959087+0000","last_fullsized":"2024-09-21T12:22:57.959087+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":131,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.963635+0000","last_change":"2024-09-21T12:20:21.595660+0000","last_active":"2024-09-21T12:22:57.963635+0000","last_peered":"2024-09-21T12:22:57.963635+0000","last_clean":"2024-09-21T12:22:57.963635+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:22:57.963635+0000","last_undegraded":"2024-09-21T12:22:57.963635+0000","last_fullsized":"2024-09-21T12:22:57.963635+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:28.985337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":145,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.965990+0000","last_change":"2024-09-21T12:20:17.304075+0000","last_active":"2024-09-21T12:22:57.965990+0000","last_peered":"2024-09-21T12:22:57.965990+0000","last_clean":"2024-09-21T12:22:57.965990+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:22:57.965990+0000","last_undegraded":"2024-09-21T12:22:57.965990+0000","last_fullsized":"2024-09-21T12:22:57.965990+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":159,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.955973+0000","last_change":"2024-09-21T12:22:14.635397+0000","last_active":"2024-09-21T12:22:57.955973+0000","last_peered":"2024-09-21T12:22:57.955973+0000","last_clean":"2024-09-21T12:22:57.955973+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:22:57.955973+0000","last_undegraded":"2024-09-21T12:22:57.955973+0000","last_fullsized":"2024-09-21T12:22:57.955973+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:22:14.635191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.635191+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:18.354894+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":165,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.957433+0000","last_change":"2024-09-21T12:22:50.931202+0000","last_active":"2024-09-21T12:22:57.957433+0000","last_peered":"2024-09-21T12:22:57.957433+0000","last_clean":"2024-09-21T12:22:57.957433+0000","last_became_active":"2024-09-21T12:22:50.930615+0000","last_became_peered":"2024-09-21T12:22:50.930615+0000","last_unstale":"2024-09-21T12:22:57.957433+0000","last_undegraded":"2024-09-21T12:22:57.957433+0000","last_fullsized":"2024-09-21T12:22:57.957433+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:09.649109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:09.649109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:09.781859+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":153,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.959376+0000","last_change":"2024-09-21T12:22:50.917883+0000","last_active":"2024-09-21T12:22:57.959376+0000","last_peered":"2024-09-21T12:22:57.959376+0000","last_clean":"2024-09-21T12:22:57.959376+0000","last_became_active":"2024-09-21T12:22:50.917408+0000","last_became_peered":"2024-09-21T12:22:50.917408+0000","last_unstale":"2024-09-21T12:22:57.959376+0000","last_undegraded":"2024-09-21T12:22:57.959376+0000","last_fullsized":"2024-09-21T12:22:57.959376+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":154,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.975897+0000","last_change":"2024-09-21T12:22:50.914633+0000","last_active":"2024-09-21T12:22:57.975897+0000","last_peered":"2024-09-21T12:22:57.975897+0000","last_clean":"2024-09-21T12:22:57.975897+0000","last_became_active":"2024-09-21T12:22:50.914221+0000","last_became_peered":"2024-09-21T12:22:50.914221+0000","last_unstale":"2024-09-21T12:22:57.975897+0000","last_undegraded":"2024-09-21T12:22:57.975897+0000","last_fullsized":"2024-09-21T12:22:57.975897+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":151,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.966269+0000","last_change":"2024-09-21T12:22:44.080149+0000","last_active":"2024-09-21T12:22:57.966269+0000","last_peered":"2024-09-21T12:22:57.966269+0000","last_clean":"2024-09-21T12:22:57.966269+0000","last_became_active":"2024-09-21T12:20:15.297993+0000","last_became_peered":"2024-09-21T12:20:15.297993+0000","last_unstale":"2024-09-21T12:22:57.966269+0000","last_undegraded":"2024-09-21T12:22:57.966269+0000","last_fullsized":"2024-09-21T12:22:57.966269+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:44.080070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:44.080070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:01.088731+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":159,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.958271+0000","last_change":"2024-09-21T12:20:17.299134+0000","last_active":"2024-09-21T12:22:57.958271+0000","last_peered":"2024-09-21T12:22:57.958271+0000","last_clean":"2024-09-21T12:22:57.958271+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:22:57.958271+0000","last_undegraded":"2024-09-21T12:22:57.958271+0000","last_fullsized":"2024-09-21T12:22:57.958271+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:33.593130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":145,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-21T12:22:57.954283+0000","last_change":"2024-09-21T12:22:55.065505+0000","last_active":"2024-09-21T12:22:57.954283+0000","last_peered":"2024-09-21T12:22:57.954283+0000","last_clean":"2024-09-21T12:22:57.954283+0000","last_became_active":"2024-09-21T12:20:19.308247+0000","last_became_peered":"2024-09-21T12:20:19.308247+0000","last_unstale":"2024-09-21T12:22:57.954283+0000","last_undegraded":"2024-09-21T12:22:57.954283+0000","last_fullsized":"2024-09-21T12:22:57.954283+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:55.065444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:55.065444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:15.087826+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6749224984,"num_objects":102986,"num_object_clones":0,"num_object_copies":308958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":851,"num_objects_unfound":0,"num_objects_dirty":102986,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105302,"num_write_kb":6665153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14961,"num_bytes_recovered":980484096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20330786816,"data_stored":20330774600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70765,"ondisk_log_size":70765,"up":48,"acting":47,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":690,"num_read_kb":447,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":3542,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":396,"ondisk_log_size":396,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":5023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":161,"num_read_kb":156,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":375690,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":142,"num_write":72,"num_write_kb":1036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":966058,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234021,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2580196,"kb_used_data":2486548,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":91164956,"statfs":{"total":95995035648,"available":93352914944,"internally_reserved":0,"allocated":2546225152,"data_stored":2545531959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":95877578},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462957,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3017320,"kb_used_data":2913944,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":90727832,"statfs":{"total":95995035648,"available":92905299968,"internally_reserved":0,"allocated":2983878656,"data_stored":2983181182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724592,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1700896,"kb_used_data":1647504,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92044256,"statfs":{"total":95995035648,"available":94253318144,"internally_reserved":0,"allocated":1687044096,"data_stored":1686356213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953528,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149758,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4220932,"kb_used_data":4104820,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":89524220,"statfs":{"total":95995035648,"available":91672801280,"internally_reserved":0,"allocated":4203335680,"data_stored":4202611672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":118618567},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411396,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2555408,"kb_used_data":2483904,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91189744,"statfs":{"total":95995035648,"available":93378297856,"internally_reserved":0,"allocated":2543517696,"data_stored":2542844654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574922,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3019864,"kb_used_data":2908168,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":90725288,"statfs":{"total":95995035648,"available":92902694912,"internally_reserved":0,"allocated":2977964032,"data_stored":2977218264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738448,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3414000,"kb_used_data":3310624,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":90331152,"statfs":{"total":95995035648,"available":92499099648,"internally_reserved":0,"allocated":3390078976,"data_stored":3389374232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3365736448,"data_stored":3365732376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2962755584,"data_stored":2962751512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2525036544,"data_stored":2525036544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4177002496,"data_stored":4177002496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":137297920,"data_stored":137297920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1679556608,"data_stored":1679556608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2955415552,"data_stored":2955411480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2527985664,"data_stored":2527985664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:23:04.548 INFO:tasks.thrashosds.thrasher:Setting 5.e to [7, 2, 1] 2024-09-21T12:23:04.548 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '5.e', '7', '2', '1'] 2024-09-21T12:23:04.548 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pg-upmap 5.e 7 2 1 2024-09-21T12:23:04.560 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:04.573 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:04.637 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:04.661 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:04.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:04.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:04.763 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:04.789 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:04.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:04.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:04.891 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:04.917 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:04.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:04.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:05.019 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:05.045 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:05.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:05.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:05.146 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:05.181 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:05.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:05.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:05.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:05 smithi183 ceph-mon[33277]: pgmap v123: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 904 B/s rd, 40 MiB/s wr, 805 op/s; 851/309555 objects misplaced (0.275%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T12:23:05.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:05 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:23:05.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:05 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:23:05.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:05 smithi183 ceph-mon[33277]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:23:05.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:05 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:23:05.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:05 smithi183 ceph-mon[33277]: osdmap e127: 8 total, 8 up, 7 in 2024-09-21T12:23:05.283 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:05.308 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:05.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:05.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:05.407 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:05 smithi114 ceph-mon[30324]: pgmap v123: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 904 B/s rd, 40 MiB/s wr, 805 op/s; 851/309555 objects misplaced (0.275%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T12:23:05.407 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:05 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:23:05.407 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:05 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:23:05.407 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:05 smithi114 ceph-mon[30324]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:23:05.407 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:05 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:23:05.407 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:05 smithi114 ceph-mon[30324]: osdmap e127: 8 total, 8 up, 7 in 2024-09-21T12:23:05.407 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:05 smithi114 ceph-mon[40867]: pgmap v123: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 904 B/s rd, 40 MiB/s wr, 805 op/s; 851/309555 objects misplaced (0.275%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T12:23:05.407 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:05 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:23:05.407 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:05 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:23:05.408 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:05 smithi114 ceph-mon[40867]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:23:05.408 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:05 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:23:05.408 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:05 smithi114 ceph-mon[40867]: osdmap e127: 8 total, 8 up, 7 in 2024-09-21T12:23:05.410 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:05.489 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:05.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:05.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:05.591 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:05.615 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:05.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:05.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:05.717 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:05.754 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:05.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:05.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:05.855 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:05.878 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:05.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:05.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:05.979 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:06.137 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:06.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:06.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:06.239 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:06.263 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:06.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:06.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:06.330 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:23:06.331 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:23:06.350 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:23:06.350 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:23:06.365 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:06.387 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:06.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:06.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:06.390 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:23:06.390 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:23:06.437 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:23:06.437 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:23:06.464 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:23:06.464 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-21T12:23:06.465 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.465 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.24695.0:110532 7.e e127/84 7:7dc8381b:::benchmark_data_smithi114_78432_object6386:head v 127'6893, mlcod=127'6889)", 2024-09-21T12:23:06.465 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:23:06.348372+0000", 2024-09-21T12:23:06.465 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.027185918, 2024-09-21T12:23:06.465 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.027210285000000001, 2024-09-21T12:23:06.465 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:06.465 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:06.465 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:23:06.465 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:06.465 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.465 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:06.465 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.348372+0000", 2024-09-21T12:23:06.465 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:06.465 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.466 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.466 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:06.466 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.348372+0000", 2024-09-21T12:23:06.466 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:06.466 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.466 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.466 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:06.466 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.348373+0000", 2024-09-21T12:23:06.466 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.8400000000000003e-07 2024-09-21T12:23:06.466 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.466 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.466 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:06.466 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.348379+0000", 2024-09-21T12:23:06.466 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.195e-06 2024-09-21T12:23:06.466 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.467 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.467 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:06.467 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.348390+0000", 2024-09-21T12:23:06.467 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0467e-05 2024-09-21T12:23:06.467 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.467 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.467 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:06.467 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.348396+0000", 2024-09-21T12:23:06.467 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2899999999999999e-06 2024-09-21T12:23:06.467 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.467 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.467 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:06.467 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.352706+0000", 2024-09-21T12:23:06.467 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0043099030000000003 2024-09-21T12:23:06.467 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.468 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.468 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:06.468 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.352723+0000", 2024-09-21T12:23:06.468 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7311000000000001e-05 2024-09-21T12:23:06.468 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:06.468 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:06.468 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:06.468 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.468 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.468 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.24695.0:110544 7.e e127/84 7:71ca6f71:::benchmark_data_smithi114_78432_object6398:head v 127'6894, mlcod=127'6889)", 2024-09-21T12:23:06.468 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:23:06.354216+0000", 2024-09-21T12:23:06.468 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.021341859000000001, 2024-09-21T12:23:06.468 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.021432634999999998, 2024-09-21T12:23:06.468 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:06.468 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:06.469 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:23:06.469 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:06.469 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.469 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:06.469 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.354216+0000", 2024-09-21T12:23:06.469 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:06.469 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.469 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.469 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:06.469 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.354216+0000", 2024-09-21T12:23:06.469 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:06.469 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.469 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.469 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:06.469 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.354217+0000", 2024-09-21T12:23:06.469 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6699999999999999e-07 2024-09-21T12:23:06.470 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.470 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.470 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:06.470 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.354221+0000", 2024-09-21T12:23:06.470 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.6849999999999999e-06 2024-09-21T12:23:06.470 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.470 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.470 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:06.470 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.354227+0000", 2024-09-21T12:23:06.470 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2600000000000002e-06 2024-09-21T12:23:06.470 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.470 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.470 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:06.470 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.354231+0000", 2024-09-21T12:23:06.471 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6750000000000001e-06 2024-09-21T12:23:06.471 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.471 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.471 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:06.471 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.359371+0000", 2024-09-21T12:23:06.471 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0051394819999999999 2024-09-21T12:23:06.471 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.471 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.471 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:06.471 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.359388+0000", 2024-09-21T12:23:06.471 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6803000000000002e-05 2024-09-21T12:23:06.471 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:06.471 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:06.471 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:06.472 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.472 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.472 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:110547 7.6 7:63d79939:::benchmark_data_smithi114_78432_object6401:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-21T12:23:06.472 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:23:06.354958+0000", 2024-09-21T12:23:06.472 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.020600131000000001, 2024-09-21T12:23:06.472 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.020807108000000001, 2024-09-21T12:23:06.472 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:06.472 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:06.472 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:23:06.472 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:23:06.472 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:23:06.472 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:23:06.472 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 110547 2024-09-21T12:23:06.472 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.472 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:06.473 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.473 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:06.473 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.354958+0000", 2024-09-21T12:23:06.473 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:06.473 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.473 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.473 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:06.473 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.354958+0000", 2024-09-21T12:23:06.473 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:06.473 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.473 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.473 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:06.473 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.354960+0000", 2024-09-21T12:23:06.473 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.559e-06 2024-09-21T12:23:06.473 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.473 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.474 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:06.474 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.354965+0000", 2024-09-21T12:23:06.474 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4759999999999997e-06 2024-09-21T12:23:06.474 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.474 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.474 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:06.474 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.354981+0000", 2024-09-21T12:23:06.474 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5481e-05 2024-09-21T12:23:06.474 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.474 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.474 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:06.474 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.354988+0000", 2024-09-21T12:23:06.474 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3320000000000003e-06 2024-09-21T12:23:06.474 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.474 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.475 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:06.475 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.364376+0000", 2024-09-21T12:23:06.475 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0093878859999999998 2024-09-21T12:23:06.475 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.475 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.475 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:06.475 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.364479+0000", 2024-09-21T12:23:06.475 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010365 2024-09-21T12:23:06.475 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.475 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.475 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:23:06.475 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.364581+0000", 2024-09-21T12:23:06.475 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000101078 2024-09-21T12:23:06.475 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.475 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.476 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:06.476 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.373472+0000", 2024-09-21T12:23:06.476 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0088913529999999994 2024-09-21T12:23:06.476 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:06.476 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:06.476 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:06.476 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.476 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.476 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:110548 7.6 7.da0a2876 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-21T12:23:06.476 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:23:06.355422+0000", 2024-09-21T12:23:06.476 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.020136202999999998, 2024-09-21T12:23:06.476 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.020409996999999999, 2024-09-21T12:23:06.476 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:06.476 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:06.476 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "queued for pg", 2024-09-21T12:23:06.476 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:23:06.477 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:23:06.477 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:23:06.477 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 110548 2024-09-21T12:23:06.477 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.477 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:06.477 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.477 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:06.477 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.355422+0000", 2024-09-21T12:23:06.477 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:06.477 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.477 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.477 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:06.477 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.355422+0000", 2024-09-21T12:23:06.477 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:06.477 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.477 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.478 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:06.478 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.355423+0000", 2024-09-21T12:23:06.478 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6799999999999999e-07 2024-09-21T12:23:06.478 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.478 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.478 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:06.478 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.355427+0000", 2024-09-21T12:23:06.478 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0219999999999998e-06 2024-09-21T12:23:06.478 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.478 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.478 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:06.478 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.355438+0000", 2024-09-21T12:23:06.478 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0937999999999999e-05 2024-09-21T12:23:06.478 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.478 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.479 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:06.479 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.355445+0000", 2024-09-21T12:23:06.479 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.883e-06 2024-09-21T12:23:06.479 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:06.479 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:06.479 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:06.479 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.479 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.479 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.24695.0:110551 7.e e127/84 7:703a4b3c:::benchmark_data_smithi114_78432_object6405:head v 127'6895, mlcod=127'6889)", 2024-09-21T12:23:06.479 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:23:06.357936+0000", 2024-09-21T12:23:06.479 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.017622694000000001, 2024-09-21T12:23:06.479 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.017941325000000001, 2024-09-21T12:23:06.479 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:06.479 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:06.480 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:23:06.480 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:06.480 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.480 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:06.480 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.357936+0000", 2024-09-21T12:23:06.480 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:06.480 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.480 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.480 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:06.480 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.357936+0000", 2024-09-21T12:23:06.480 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:06.480 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.480 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.480 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:06.481 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.357936+0000", 2024-09-21T12:23:06.481 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8299999999999996e-07 2024-09-21T12:23:06.481 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.481 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.481 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:06.481 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.357942+0000", 2024-09-21T12:23:06.481 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5779999999999998e-06 2024-09-21T12:23:06.481 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.481 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.481 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:06.481 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.357958+0000", 2024-09-21T12:23:06.481 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.632e-05 2024-09-21T12:23:06.481 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.481 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.481 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:06.482 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.357964+0000", 2024-09-21T12:23:06.482 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.088e-06 2024-09-21T12:23:06.482 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.482 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.482 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:06.482 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.367052+0000", 2024-09-21T12:23:06.482 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0090876480000000003 2024-09-21T12:23:06.482 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.482 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.482 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:06.482 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.367074+0000", 2024-09-21T12:23:06.482 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2540000000000001e-05 2024-09-21T12:23:06.482 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:06.483 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:06.483 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:06.483 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.483 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.483 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.24695.0:110556 7.e e127/84 7:7b88e97f:::benchmark_data_smithi114_78432_object6410:head v 127'6896, mlcod=127'6890)", 2024-09-21T12:23:06.483 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:23:06.367408+0000", 2024-09-21T12:23:06.483 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0081507049999999994, 2024-09-21T12:23:06.483 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0085135660000000002, 2024-09-21T12:23:06.483 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:06.483 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:06.483 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:23:06.483 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:06.483 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.483 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:06.484 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.367408+0000", 2024-09-21T12:23:06.484 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:06.484 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.484 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.484 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:06.484 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.367408+0000", 2024-09-21T12:23:06.484 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:06.484 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.484 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.484 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:06.484 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.367408+0000", 2024-09-21T12:23:06.484 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1999999999999998e-07 2024-09-21T12:23:06.484 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.484 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.485 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:06.485 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.367415+0000", 2024-09-21T12:23:06.485 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4989999999999999e-06 2024-09-21T12:23:06.485 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.485 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.485 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:06.485 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.367426+0000", 2024-09-21T12:23:06.485 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0740000000000001e-05 2024-09-21T12:23:06.485 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.485 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.485 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:06.485 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.367433+0000", 2024-09-21T12:23:06.485 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.917e-06 2024-09-21T12:23:06.485 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.485 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.486 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:06.486 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.373504+0000", 2024-09-21T12:23:06.486 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0060714799999999998 2024-09-21T12:23:06.486 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.486 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.486 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:06.486 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.373523+0000", 2024-09-21T12:23:06.486 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8947e-05 2024-09-21T12:23:06.486 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:06.486 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:06.486 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:06.486 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.486 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.486 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop_reply(client.24695.0:110547 7.6 e127/84)", 2024-09-21T12:23:06.487 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:23:06.373554+0000", 2024-09-21T12:23:06.487 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0020039369999999999, 2024-09-21T12:23:06.487 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0024080400000000002, 2024-09-21T12:23:06.487 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:06.487 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:06.487 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "queued for pg", 2024-09-21T12:23:06.487 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:06.487 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.487 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:06.487 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.373554+0000", 2024-09-21T12:23:06.487 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:06.487 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.487 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.487 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:06.488 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.373554+0000", 2024-09-21T12:23:06.488 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:06.488 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.488 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.488 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:06.488 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.373555+0000", 2024-09-21T12:23:06.488 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6500000000000008e-07 2024-09-21T12:23:06.488 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.488 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.488 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:06.488 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.373560+0000", 2024-09-21T12:23:06.488 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5000000000000001e-06 2024-09-21T12:23:06.488 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.488 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.488 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:06.489 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.373571+0000", 2024-09-21T12:23:06.489 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1262e-05 2024-09-21T12:23:06.489 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:06.489 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:06.489 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:06.489 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:06.373578+0000", 2024-09-21T12:23:06.489 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4680000000000003e-06 2024-09-21T12:23:06.489 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:06.489 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:06.489 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:06.489 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:06.489 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:23:06.489 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 7 2024-09-21T12:23:06.489 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:23:06.490 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:06.491 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:23:06.491 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:23:06.500 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:23:06.500 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:23:06.514 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:06.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:06.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:06.532 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:23:06.532 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:23:06.563 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:23:06.564 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:23:06.615 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:06.639 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:06.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:06.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:06.741 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:06.786 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:06.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:06.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:06.887 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:06.913 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:06.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:06.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:06.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:23:06 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:23:07.014 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:07.039 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:07.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:07.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:07.140 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:07.164 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:07.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:07.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:07.266 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:07.307 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:07.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:07.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:07.409 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:07.443 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:07.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:07.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:07.484 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T12:23:07.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:07 smithi183 ceph-mon[33277]: pgmap v125: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 803 B/s rd, 35 MiB/s wr, 714 op/s; 851/309555 objects misplaced (0.275%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T12:23:07.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:07 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3371902438' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:23:07.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:07 smithi183 ceph-mon[33277]: osdmap e128: 8 total, 8 up, 7 in 2024-09-21T12:23:07.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:07 smithi114 ceph-mon[30324]: pgmap v125: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 803 B/s rd, 35 MiB/s wr, 714 op/s; 851/309555 objects misplaced (0.275%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T12:23:07.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:07 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3371902438' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:23:07.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:07 smithi114 ceph-mon[30324]: osdmap e128: 8 total, 8 up, 7 in 2024-09-21T12:23:07.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:07 smithi114 ceph-mon[40867]: pgmap v125: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 803 B/s rd, 35 MiB/s wr, 714 op/s; 851/309555 objects misplaced (0.275%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T12:23:07.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:07 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3371902438' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:23:07.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:07 smithi114 ceph-mon[40867]: osdmap e128: 8 total, 8 up, 7 in 2024-09-21T12:23:07.544 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:07.566 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:07.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:07.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:07.668 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:07.692 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:07.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:07.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:07.793 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:07.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:07.848 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:07.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:07.949 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:07.974 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:07.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:07.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:08.076 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:08.099 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:08.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:08.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:08.103 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T12:23:08.200 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:08.216 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:08.217 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:08.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:08.266 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:08.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:08.367 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:08.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:08 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:23:08.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:08 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:23:08.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:08 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:23:08.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:08 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:23:08.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:08 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:23:08.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:08 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:23:08.391 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:08.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:08.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:08.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:08 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:23:08.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:08 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:23:08.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:08 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:23:08.494 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:08.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:08.515 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:08.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:08.616 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:08.649 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:08.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:08.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:08.751 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:08.774 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:08.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:08.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:08.876 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:08.907 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:08.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:08.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:09.009 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:09.032 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:09.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:09.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:09.134 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:09.155 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:09.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:09.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:09.256 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:09.286 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:09.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:09.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:09.358 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:09 smithi114 ceph-mon[30324]: pgmap v127: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 30 MiB/s wr, 1.55k op/s; 624/292692 objects misplaced (0.213%); 4.7 MiB/s, 75 objects/s recovering 2024-09-21T12:23:09.358 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:09 smithi114 ceph-mon[40867]: pgmap v127: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 30 MiB/s wr, 1.55k op/s; 624/292692 objects misplaced (0.213%); 4.7 MiB/s, 75 objects/s recovering 2024-09-21T12:23:09.388 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:09.410 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:09.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:09.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:09.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:09 smithi183 ceph-mon[33277]: pgmap v127: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 30 MiB/s wr, 1.55k op/s; 624/292692 objects misplaced (0.213%); 4.7 MiB/s, 75 objects/s recovering 2024-09-21T12:23:09.512 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:09.536 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:09.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:09.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:09.579 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:23:09.638 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:09.659 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:09.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:09.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:09.760 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:09.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:09.802 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:09.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:09.903 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:09.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:09.971 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:09.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:10.072 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:10.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:10.121 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:10.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:10.222 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:10.229 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:10.252 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:10.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:10.293 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:10.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:10.394 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:10.417 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:10.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:10.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:10.519 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:10.544 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:10.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:10.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:10.646 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:10.666 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:10.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:10.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:10.768 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:10.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:10.794 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:10.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:10.895 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:10.920 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:10.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:10.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:11.021 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:11.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:11.061 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:11.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:11.120 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-21T12:23:11.122 INFO:teuthology.orchestra.run.smithi114.stderr:set 5.e pg_upmap mapping to [7,2,1] 2024-09-21T12:23:11.162 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:11.202 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:11.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:11.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:11.304 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:11.342 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:11.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:11.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:11.443 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:11.483 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:11.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:11.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:11.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:11 smithi183 ceph-mon[33277]: pgmap v128: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 657 B/s rd, 0 B/s wr, 722 op/s; 624/292692 objects misplaced (0.213%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T12:23:11.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:11 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1404184389' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:23:11.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:11 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:23:11.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:11 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3735728097' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "5.e", "id": [7, 2, 1]} : dispatch 2024-09-21T12:23:11.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:11 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "5.e", "id": [7, 2, 1]} : dispatch 2024-09-21T12:23:11.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:11 smithi114 ceph-mon[40867]: pgmap v128: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 657 B/s rd, 0 B/s wr, 722 op/s; 624/292692 objects misplaced (0.213%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T12:23:11.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:11 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1404184389' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:23:11.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:11 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:23:11.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:11 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3735728097' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "5.e", "id": [7, 2, 1]} : dispatch 2024-09-21T12:23:11.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:11 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "5.e", "id": [7, 2, 1]} : dispatch 2024-09-21T12:23:11.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:11 smithi114 ceph-mon[30324]: pgmap v128: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 657 B/s rd, 0 B/s wr, 722 op/s; 624/292692 objects misplaced (0.213%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T12:23:11.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:11 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1404184389' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:23:11.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:11 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:23:11.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:11 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3735728097' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "5.e", "id": [7, 2, 1]} : dispatch 2024-09-21T12:23:11.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:11 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "5.e", "id": [7, 2, 1]} : dispatch 2024-09-21T12:23:11.585 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:11.619 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:11.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:11.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:11.720 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:11.745 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:11.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:11.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:11.846 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:11.869 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:11.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:11.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:11.971 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:11.997 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:11.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:11.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:12.098 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:12.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:12.123 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:12.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:12.224 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:12.249 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:12.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:12.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:12.350 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:12.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:12.386 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:12.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:12.487 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:12 smithi183 ceph-mon[33277]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:23:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:12 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:23:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:12 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.e", "id": [7, 2, 1]}]': finished 2024-09-21T12:23:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:12 smithi183 ceph-mon[33277]: osdmap e129: 8 total, 8 up, 7 in 2024-09-21T12:23:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:12 smithi183 ceph-mon[33277]: osdmap e130: 8 total, 8 up, 7 in 2024-09-21T12:23:12.508 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:12.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:12.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:12.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:12 smithi114 ceph-mon[40867]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:23:12.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:12 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:23:12.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:12 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.e", "id": [7, 2, 1]}]': finished 2024-09-21T12:23:12.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:12 smithi114 ceph-mon[40867]: osdmap e129: 8 total, 8 up, 7 in 2024-09-21T12:23:12.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:12 smithi114 ceph-mon[40867]: osdmap e130: 8 total, 8 up, 7 in 2024-09-21T12:23:12.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:12 smithi114 ceph-mon[30324]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:23:12.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:12 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:23:12.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:12 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.e", "id": [7, 2, 1]}]': finished 2024-09-21T12:23:12.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:12 smithi114 ceph-mon[30324]: osdmap e129: 8 total, 8 up, 7 in 2024-09-21T12:23:12.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:12 smithi114 ceph-mon[30324]: osdmap e130: 8 total, 8 up, 7 in 2024-09-21T12:23:12.610 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:12.634 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:12.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:12.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:12.736 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:12.758 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:12.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:12.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:12.860 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:12.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:12.897 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:12.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:12.998 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:13.024 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:23:12 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:23:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:23:13.031 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:13.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:13.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:13.132 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:13.170 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:13.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:13.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:13.271 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:13.298 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:13.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:13.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:13.400 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:13.424 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:13.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:13.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:13.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:13 smithi183 ceph-mon[33277]: pgmap v130: 158 pgs: 1 peering, 1 active+recovering+undersized+remapped, 156 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 959 op/s; 624/287931 objects misplaced (0.217%); 1.9 MiB/s, 29 objects/s recovering 2024-09-21T12:23:13.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:13 smithi183 ceph-mon[33277]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T12:23:13.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:13 smithi114 ceph-mon[40867]: pgmap v130: 158 pgs: 1 peering, 1 active+recovering+undersized+remapped, 156 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 959 op/s; 624/287931 objects misplaced (0.217%); 1.9 MiB/s, 29 objects/s recovering 2024-09-21T12:23:13.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:13 smithi114 ceph-mon[40867]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T12:23:13.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:13 smithi114 ceph-mon[30324]: pgmap v130: 158 pgs: 1 peering, 1 active+recovering+undersized+remapped, 156 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 959 op/s; 624/287931 objects misplaced (0.217%); 1.9 MiB/s, 29 objects/s recovering 2024-09-21T12:23:13.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:13 smithi114 ceph-mon[30324]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T12:23:13.525 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:13.558 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:13.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:13.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:13.660 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:13.691 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:13.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:13.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:13.793 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:13.832 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:13.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:13.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:13.933 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:13.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:13.971 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:13.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:14.072 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:14.109 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:14.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:14.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:14.211 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:14.236 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:14.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:14.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:14.338 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:14.383 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:14.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:14.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:14.485 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:14.508 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:14.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:14.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:14.610 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:14.650 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:14.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:14.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:14.752 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:14.777 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:14.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:14.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:14.878 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:14.914 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:14.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:14.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:14.934 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:23:14.935 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:23:14.980 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:23:14.980 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:23:15.012 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:23:15.012 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:23:15.015 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:15.040 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:15.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:15.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:15.046 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:23:15.046 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:23:15.079 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:23:15.079 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:23:15.112 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:23:15.112 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:23:15.142 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:15.145 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:23:15.146 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:23:15.179 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:23:15.179 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:23:15.183 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:15.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:15.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:15.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:15 smithi114 ceph-mon[40867]: pgmap v132: 158 pgs: 1 active+recovering+undersized+remapped, 1 peering, 156 active+clean; 5.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 754 B/s rd, 0 B/s wr, 756 op/s; 402/277302 objects misplaced (0.145%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T12:23:15.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:15 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3557048933' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:23:15.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:15 smithi114 ceph-mon[30324]: pgmap v132: 158 pgs: 1 active+recovering+undersized+remapped, 1 peering, 156 active+clean; 5.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 754 B/s rd, 0 B/s wr, 756 op/s; 402/277302 objects misplaced (0.145%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T12:23:15.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:15 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3557048933' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:23:15.285 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:15.311 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:15.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:15.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:15.413 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:15.444 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:15.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:15.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:15.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:15 smithi183 ceph-mon[33277]: pgmap v132: 158 pgs: 1 active+recovering+undersized+remapped, 1 peering, 156 active+clean; 5.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 754 B/s rd, 0 B/s wr, 756 op/s; 402/277302 objects misplaced (0.145%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T12:23:15.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:15 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3557048933' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:23:15.506 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:23:15.506 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:23:15.506 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:23:15.506 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:23:15.507 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:23:15.546 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:15.588 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:15.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:15.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:15.689 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:15.714 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:15.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:15.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:15.816 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:15.855 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:15.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:15.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:15.957 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:15.982 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:15.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:15.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:16.084 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:16.124 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:16.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:16.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:16.162 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T12:23:16.226 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:16.251 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:16.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:16.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:16.353 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:16.402 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:16.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:16.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:16 smithi183 ceph-mon[33277]: 4.b scrub starts 2024-09-21T12:23:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:16 smithi183 ceph-mon[33277]: 4.b scrub ok 2024-09-21T12:23:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:16 smithi183 ceph-mon[33277]: 5.16 scrub starts 2024-09-21T12:23:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:16 smithi183 ceph-mon[33277]: 5.16 scrub ok 2024-09-21T12:23:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:16 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:23:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:16 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:23:16.504 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:16.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:16 smithi114 ceph-mon[30324]: 4.b scrub starts 2024-09-21T12:23:16.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:16 smithi114 ceph-mon[30324]: 4.b scrub ok 2024-09-21T12:23:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:16 smithi114 ceph-mon[30324]: 5.16 scrub starts 2024-09-21T12:23:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:16 smithi114 ceph-mon[30324]: 5.16 scrub ok 2024-09-21T12:23:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:16 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:23:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:16 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:23:16.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:16 smithi114 ceph-mon[40867]: 4.b scrub starts 2024-09-21T12:23:16.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:16 smithi114 ceph-mon[40867]: 4.b scrub ok 2024-09-21T12:23:16.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:16 smithi114 ceph-mon[40867]: 5.16 scrub starts 2024-09-21T12:23:16.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:16 smithi114 ceph-mon[40867]: 5.16 scrub ok 2024-09-21T12:23:16.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:16 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:23:16.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:16 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:23:16.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:16.530 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:16.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:16.631 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:16.656 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:16.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:16.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:16.758 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:16.783 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:16.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:16.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:16.885 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:16.911 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:16.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:16.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:16.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:23:16 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:23:17.013 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:17.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:17.055 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:17.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:17.156 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:17.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:17.206 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:17.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:17.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:17 smithi114 ceph-mon[30324]: pgmap v133: 158 pgs: 1 active+recovering+undersized+remapped, 1 peering, 156 active+clean; 5.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 641 op/s; 402/277302 objects misplaced (0.145%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T12:23:17.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:17 smithi114 ceph-mon[40867]: pgmap v133: 158 pgs: 1 active+recovering+undersized+remapped, 1 peering, 156 active+clean; 5.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 641 op/s; 402/277302 objects misplaced (0.145%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T12:23:17.307 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:17.333 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:17.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:17.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:17.434 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:17.469 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:17.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:17.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:17.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:17 smithi183 ceph-mon[33277]: pgmap v133: 158 pgs: 1 active+recovering+undersized+remapped, 1 peering, 156 active+clean; 5.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 641 op/s; 402/277302 objects misplaced (0.145%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T12:23:17.571 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:17.601 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:17.626 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:17.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:17.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:17.728 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:17.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:17.774 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:17.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:17.875 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:17.901 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:17.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:17.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:18.003 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:18.046 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:18.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:18.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:18.148 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:18.169 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:18.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:18.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:18.270 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:18.305 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:18.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:18.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:18.407 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:18.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:18.435 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:18.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:18.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:18 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:23:18.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:18 smithi183 ceph-mon[33277]: 7.6 scrub starts 2024-09-21T12:23:18.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:18 smithi183 ceph-mon[33277]: 4.8 scrub starts 2024-09-21T12:23:18.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:18 smithi183 ceph-mon[33277]: 4.8 scrub ok 2024-09-21T12:23:18.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:18 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:23:18.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:18 smithi114 ceph-mon[40867]: 7.6 scrub starts 2024-09-21T12:23:18.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:18 smithi114 ceph-mon[40867]: 4.8 scrub starts 2024-09-21T12:23:18.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:18 smithi114 ceph-mon[40867]: 4.8 scrub ok 2024-09-21T12:23:18.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:18 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:23:18.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:18 smithi114 ceph-mon[30324]: 7.6 scrub starts 2024-09-21T12:23:18.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:18 smithi114 ceph-mon[30324]: 4.8 scrub starts 2024-09-21T12:23:18.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:18 smithi114 ceph-mon[30324]: 4.8 scrub ok 2024-09-21T12:23:18.535 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:18.560 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:18.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:18.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:18.609 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T12:23:18.662 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:18.697 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:18.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:18.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:18.798 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:18.841 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:18.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:18.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:18.943 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:18.979 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:18.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:18.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:19.082 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:19.106 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:19.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:19.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:19.208 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:19.247 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:19.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:19.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:19.349 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:19.370 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:19.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:19.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:19.432 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:19 smithi114 ceph-mon[30324]: 5.c scrub starts 2024-09-21T12:23:19.432 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:19 smithi114 ceph-mon[30324]: 5.c scrub ok 2024-09-21T12:23:19.432 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:19 smithi114 ceph-mon[30324]: pgmap v134: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 124/260220 objects misplaced (0.048%); 3.9 MiB/s, 62 objects/s recovering 2024-09-21T12:23:19.432 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:19 smithi114 ceph-mon[30324]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T12:23:19.432 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:19 smithi114 ceph-mon[40867]: 5.c scrub starts 2024-09-21T12:23:19.433 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:19 smithi114 ceph-mon[40867]: 5.c scrub ok 2024-09-21T12:23:19.433 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:19 smithi114 ceph-mon[40867]: pgmap v134: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 124/260220 objects misplaced (0.048%); 3.9 MiB/s, 62 objects/s recovering 2024-09-21T12:23:19.433 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:19 smithi114 ceph-mon[40867]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T12:23:19.442 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:23:19.442 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:23:19.472 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:19 smithi183 ceph-mon[33277]: 5.c scrub starts 2024-09-21T12:23:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:19 smithi183 ceph-mon[33277]: 5.c scrub ok 2024-09-21T12:23:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:19 smithi183 ceph-mon[33277]: pgmap v134: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 124/260220 objects misplaced (0.048%); 3.9 MiB/s, 62 objects/s recovering 2024-09-21T12:23:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:19 smithi183 ceph-mon[33277]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T12:23:19.493 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:19.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:19.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:19.522 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:23:19.595 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:19.617 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:19.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:19.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:19.697 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:19.719 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:19.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:19.741 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:19.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:19.842 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:19.864 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:19.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:19.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:19.966 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:19.988 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:19.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:19.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:20.089 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:20.112 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:20.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:20.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:20.213 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:20.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:20.243 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:20.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:20.343 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:20.368 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:20.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:20.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:20.470 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:20 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:23:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:20 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:23:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:20 smithi183 ceph-mon[33277]: 3.1b scrub starts 2024-09-21T12:23:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:20 smithi183 ceph-mon[33277]: 3.1b scrub ok 2024-09-21T12:23:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:20 smithi183 ceph-mon[33277]: osdmap e131: 8 total, 8 up, 7 in 2024-09-21T12:23:20.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:20 smithi183 ceph-mon[33277]: 7.6 scrub ok 2024-09-21T12:23:20.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:20 smithi183 ceph-mon[33277]: 5.f scrub starts 2024-09-21T12:23:20.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:20 smithi183 ceph-mon[33277]: 5.f scrub ok 2024-09-21T12:23:20.507 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:20.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:20.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:20.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:20 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:23:20.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:20 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:23:20.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:20 smithi114 ceph-mon[40867]: 3.1b scrub starts 2024-09-21T12:23:20.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:20 smithi114 ceph-mon[40867]: 3.1b scrub ok 2024-09-21T12:23:20.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:20 smithi114 ceph-mon[40867]: osdmap e131: 8 total, 8 up, 7 in 2024-09-21T12:23:20.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:20 smithi114 ceph-mon[40867]: 7.6 scrub ok 2024-09-21T12:23:20.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:20 smithi114 ceph-mon[40867]: 5.f scrub starts 2024-09-21T12:23:20.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:20 smithi114 ceph-mon[40867]: 5.f scrub ok 2024-09-21T12:23:20.516 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:20 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:23:20.516 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:20 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:23:20.516 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:20 smithi114 ceph-mon[30324]: 3.1b scrub starts 2024-09-21T12:23:20.516 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:20 smithi114 ceph-mon[30324]: 3.1b scrub ok 2024-09-21T12:23:20.516 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:20 smithi114 ceph-mon[30324]: osdmap e131: 8 total, 8 up, 7 in 2024-09-21T12:23:20.516 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:20 smithi114 ceph-mon[30324]: 7.6 scrub ok 2024-09-21T12:23:20.516 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:20 smithi114 ceph-mon[30324]: 5.f scrub starts 2024-09-21T12:23:20.516 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:20 smithi114 ceph-mon[30324]: 5.f scrub ok 2024-09-21T12:23:20.572 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T12:23:20.609 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:20.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:20.671 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:20.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:20.772 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:20.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:20.808 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:20.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:20.909 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:20.932 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:20.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:20.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:21.034 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:21.075 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:21.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:21.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:21.177 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:21.199 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:21.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:21.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:21.203 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:21.301 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:21.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:21.341 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:21.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:21.386 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:21 smithi114 ceph-mon[40867]: pgmap v136: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.15k op/s; 124/260220 objects misplaced (0.048%); 3.9 MiB/s, 62 objects/s recovering 2024-09-21T12:23:21.386 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:21 smithi114 ceph-mon[40867]: 1.2 scrub starts 2024-09-21T12:23:21.386 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:21 smithi114 ceph-mon[40867]: 1.2 scrub ok 2024-09-21T12:23:21.386 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:21 smithi114 ceph-mon[40867]: osdmap e132: 8 total, 8 up, 7 in 2024-09-21T12:23:21.386 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:21 smithi114 ceph-mon[40867]: 6.6 scrub starts 2024-09-21T12:23:21.386 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:21 smithi114 ceph-mon[40867]: 6.6 scrub ok 2024-09-21T12:23:21.386 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:21 smithi114 ceph-mon[30324]: pgmap v136: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.15k op/s; 124/260220 objects misplaced (0.048%); 3.9 MiB/s, 62 objects/s recovering 2024-09-21T12:23:21.386 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:21 smithi114 ceph-mon[30324]: 1.2 scrub starts 2024-09-21T12:23:21.387 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:21 smithi114 ceph-mon[30324]: 1.2 scrub ok 2024-09-21T12:23:21.387 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:21 smithi114 ceph-mon[30324]: osdmap e132: 8 total, 8 up, 7 in 2024-09-21T12:23:21.387 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:21 smithi114 ceph-mon[30324]: 6.6 scrub starts 2024-09-21T12:23:21.387 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:21 smithi114 ceph-mon[30324]: 6.6 scrub ok 2024-09-21T12:23:21.442 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:21.482 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:21.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:21.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:21.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:21 smithi183 ceph-mon[33277]: pgmap v136: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.15k op/s; 124/260220 objects misplaced (0.048%); 3.9 MiB/s, 62 objects/s recovering 2024-09-21T12:23:21.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:21 smithi183 ceph-mon[33277]: 1.2 scrub starts 2024-09-21T12:23:21.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:21 smithi183 ceph-mon[33277]: 1.2 scrub ok 2024-09-21T12:23:21.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:21 smithi183 ceph-mon[33277]: osdmap e132: 8 total, 8 up, 7 in 2024-09-21T12:23:21.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:21 smithi183 ceph-mon[33277]: 6.6 scrub starts 2024-09-21T12:23:21.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:21 smithi183 ceph-mon[33277]: 6.6 scrub ok 2024-09-21T12:23:21.584 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:21.610 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:21.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:21.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:21.711 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:21.737 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:21.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:21.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:21.838 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:21.874 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:21.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:21.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:21.884 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:21.908 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:23:21.908 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:23:21.945 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:23:21.945 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:23:21.975 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:23:21.976 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:23:21.976 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:22.000 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:22.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:22.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:22.008 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:23:22.008 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:23:22.011 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T12:23:22.012 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:23:22.012 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-21T12:23:22.012 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:23:22.040 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:23:22.040 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:23:22.075 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:23:22.076 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:23:22.101 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:22.111 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:23:22.111 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:23:22.125 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:22.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:22.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:22.150 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:23:22.150 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:23:22.216 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-21T12:23:22.227 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:22.231 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:22 smithi114 ceph-mon[30324]: 5.19 scrub starts 2024-09-21T12:23:22.231 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:22 smithi114 ceph-mon[30324]: 5.19 scrub ok 2024-09-21T12:23:22.231 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:22 smithi114 ceph-mon[30324]: 6.17 scrub starts 2024-09-21T12:23:22.231 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:22 smithi114 ceph-mon[30324]: 6.17 scrub ok 2024-09-21T12:23:22.231 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:22 smithi114 ceph-mon[30324]: 1.1 scrub starts 2024-09-21T12:23:22.231 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:22 smithi114 ceph-mon[30324]: 1.1 scrub ok 2024-09-21T12:23:22.232 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:22 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2483425505' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:23:22.232 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:22 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3996556895' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:23:22.232 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:22 smithi114 ceph-mon[40867]: 5.19 scrub starts 2024-09-21T12:23:22.232 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:22 smithi114 ceph-mon[40867]: 5.19 scrub ok 2024-09-21T12:23:22.232 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:22 smithi114 ceph-mon[40867]: 6.17 scrub starts 2024-09-21T12:23:22.232 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:22 smithi114 ceph-mon[40867]: 6.17 scrub ok 2024-09-21T12:23:22.232 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:22 smithi114 ceph-mon[40867]: 1.1 scrub starts 2024-09-21T12:23:22.232 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:22 smithi114 ceph-mon[40867]: 1.1 scrub ok 2024-09-21T12:23:22.232 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:22 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2483425505' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:23:22.232 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:22 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3996556895' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:23:22.250 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:22.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:22.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:22.352 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:22.392 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:22.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:22.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:22.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:22 smithi183 ceph-mon[33277]: 5.19 scrub starts 2024-09-21T12:23:22.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:22 smithi183 ceph-mon[33277]: 5.19 scrub ok 2024-09-21T12:23:22.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:22 smithi183 ceph-mon[33277]: 6.17 scrub starts 2024-09-21T12:23:22.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:22 smithi183 ceph-mon[33277]: 6.17 scrub ok 2024-09-21T12:23:22.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:22 smithi183 ceph-mon[33277]: 1.1 scrub starts 2024-09-21T12:23:22.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:22 smithi183 ceph-mon[33277]: 1.1 scrub ok 2024-09-21T12:23:22.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:22 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2483425505' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:23:22.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:22 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3996556895' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:23:22.495 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:22.544 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:22.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:22.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:22.646 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:22.687 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:22.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:22.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:22.789 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:22.816 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:22.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:22.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:22.918 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:22.943 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:22.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:22.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:23.024 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:23:22 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:23:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:23:23.045 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:23.069 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:23.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:23.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:23.171 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:23.197 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:23.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:23.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:23.299 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:23.333 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:23.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:23.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:23.435 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:23.460 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:23.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:23.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:23.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:23 smithi183 ceph-mon[33277]: pgmap v138: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 5.2 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 896 op/s; 124/255693 objects misplaced (0.048%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T12:23:23.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:23 smithi183 ceph-mon[33277]: 4.3 scrub starts 2024-09-21T12:23:23.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:23 smithi183 ceph-mon[33277]: 4.3 scrub ok 2024-09-21T12:23:23.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:23 smithi183 ceph-mon[33277]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:23:23.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:23 smithi183 ceph-mon[33277]: Cluster is now healthy 2024-09-21T12:23:23.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:23 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2483425505' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:23:23.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:23 smithi183 ceph-mon[33277]: osdmap e133: 8 total, 8 up, 7 in 2024-09-21T12:23:23.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:23 smithi183 ceph-mon[33277]: 5.d scrub starts 2024-09-21T12:23:23.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:23 smithi183 ceph-mon[33277]: 5.d scrub ok 2024-09-21T12:23:23.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:23 smithi183 ceph-mon[33277]: 1.8 scrub starts 2024-09-21T12:23:23.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:23 smithi183 ceph-mon[33277]: 1.8 scrub ok 2024-09-21T12:23:23.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:23 smithi114 ceph-mon[40867]: pgmap v138: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 5.2 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 896 op/s; 124/255693 objects misplaced (0.048%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T12:23:23.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:23 smithi114 ceph-mon[40867]: 4.3 scrub starts 2024-09-21T12:23:23.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:23 smithi114 ceph-mon[40867]: 4.3 scrub ok 2024-09-21T12:23:23.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:23 smithi114 ceph-mon[40867]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:23:23.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:23 smithi114 ceph-mon[40867]: Cluster is now healthy 2024-09-21T12:23:23.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:23 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2483425505' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:23:23.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:23 smithi114 ceph-mon[40867]: osdmap e133: 8 total, 8 up, 7 in 2024-09-21T12:23:23.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:23 smithi114 ceph-mon[40867]: 5.d scrub starts 2024-09-21T12:23:23.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:23 smithi114 ceph-mon[40867]: 5.d scrub ok 2024-09-21T12:23:23.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:23 smithi114 ceph-mon[40867]: 1.8 scrub starts 2024-09-21T12:23:23.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:23 smithi114 ceph-mon[40867]: 1.8 scrub ok 2024-09-21T12:23:23.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:23 smithi114 ceph-mon[30324]: pgmap v138: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 5.2 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 896 op/s; 124/255693 objects misplaced (0.048%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T12:23:23.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:23 smithi114 ceph-mon[30324]: 4.3 scrub starts 2024-09-21T12:23:23.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:23 smithi114 ceph-mon[30324]: 4.3 scrub ok 2024-09-21T12:23:23.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:23 smithi114 ceph-mon[30324]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:23:23.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:23 smithi114 ceph-mon[30324]: Cluster is now healthy 2024-09-21T12:23:23.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:23 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2483425505' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:23:23.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:23 smithi114 ceph-mon[30324]: osdmap e133: 8 total, 8 up, 7 in 2024-09-21T12:23:23.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:23 smithi114 ceph-mon[30324]: 5.d scrub starts 2024-09-21T12:23:23.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:23 smithi114 ceph-mon[30324]: 5.d scrub ok 2024-09-21T12:23:23.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:23 smithi114 ceph-mon[30324]: 1.8 scrub starts 2024-09-21T12:23:23.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:23 smithi114 ceph-mon[30324]: 1.8 scrub ok 2024-09-21T12:23:23.562 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:23.604 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:23.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:23.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:23.706 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:23.723 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:23:23.723 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-21T12:23:23.723 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:23.723 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.24695.0:128794 7.0 e133/84 7:0f9974d6:::benchmark_data_smithi114_78432_object24648:head v 133'7887, mlcod=133'7884)", 2024-09-21T12:23:23.723 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:23:23.653740+0000", 2024-09-21T12:23:23.723 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.001311078, 2024-09-21T12:23:23.723 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0013389529999999999, 2024-09-21T12:23:23.723 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:23.723 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:23.723 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:23:23.723 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:23.723 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:23.724 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:23.724 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:23.653740+0000", 2024-09-21T12:23:23.724 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:23.724 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:23.724 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:23.724 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:23.724 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:23.653740+0000", 2024-09-21T12:23:23.724 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:23.724 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:23.724 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:23.724 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:23.725 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:23.653741+0000", 2024-09-21T12:23:23.725 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.2699999999999998e-07 2024-09-21T12:23:23.725 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:23.725 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:23.725 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:23.725 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:23.653747+0000", 2024-09-21T12:23:23.725 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6980000000000003e-06 2024-09-21T12:23:23.725 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:23.725 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:23.725 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:23.725 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:23.653760+0000", 2024-09-21T12:23:23.725 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3380999999999999e-05 2024-09-21T12:23:23.725 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:23.725 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:23.726 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:23.726 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:23.653767+0000", 2024-09-21T12:23:23.726 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6850000000000004e-06 2024-09-21T12:23:23.726 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:23.726 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:23.726 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:23.726 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:23.653826+0000", 2024-09-21T12:23:23.726 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9122000000000001e-05 2024-09-21T12:23:23.726 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:23.726 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:23.726 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:23.727 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:23.653845+0000", 2024-09-21T12:23:23.727 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9083999999999999e-05 2024-09-21T12:23:23.727 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:23.727 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:23.727 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:23.727 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:23.727 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:23.727 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:128795 7.b 7:d16e54ac:::benchmark_data_smithi114_78432_object24649:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-21T12:23:23.727 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:23:23.654169+0000", 2024-09-21T12:23:23.727 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.00088226799999999996, 2024-09-21T12:23:23.727 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00094889400000000004, 2024-09-21T12:23:23.727 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:23.727 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:23.728 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:23:23.728 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:23:23.728 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:23:23.728 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:23:23.728 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 128795 2024-09-21T12:23:23.728 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:23.728 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:23.728 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:23.728 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:23.728 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:23.654169+0000", 2024-09-21T12:23:23.728 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:23.728 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:23.728 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:23.728 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:23.729 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:23.654169+0000", 2024-09-21T12:23:23.729 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:23.729 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:23.729 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:23.729 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:23.729 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:23.654171+0000", 2024-09-21T12:23:23.729 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.956e-06 2024-09-21T12:23:23.729 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:23.729 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:23.729 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:23.729 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:23.654175+0000", 2024-09-21T12:23:23.729 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5000000000000001e-06 2024-09-21T12:23:23.729 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:23.729 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:23.729 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:23.730 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:23.654193+0000", 2024-09-21T12:23:23.730 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7510999999999999e-05 2024-09-21T12:23:23.730 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:23.730 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:23.730 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:23.730 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:23.654201+0000", 2024-09-21T12:23:23.730 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.994e-06 2024-09-21T12:23:23.730 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:23.730 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:23.730 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:23.730 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:23.654663+0000", 2024-09-21T12:23:23.730 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00046224999999999999 2024-09-21T12:23:23.730 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:23.730 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:23.731 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:23.731 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:23.654783+0000", 2024-09-21T12:23:23.731 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000120565 2024-09-21T12:23:23.731 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:23.731 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:23.731 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 5,6", 2024-09-21T12:23:23.731 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:23.654894+0000", 2024-09-21T12:23:23.731 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011076100000000001 2024-09-21T12:23:23.731 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:23.731 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:23.731 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:23.731 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:23.731 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:23:23.731 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 2 2024-09-21T12:23:23.731 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:23:23.745 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:23.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:23.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:23.847 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:23.872 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:23.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:23.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:23.974 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:24.011 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:24.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:24.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:24.113 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:24.147 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:24.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:24.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:24.249 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:24.284 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:24.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:24.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:24.386 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:24.409 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:24.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:24.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:24.446 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:24 smithi114 ceph-mon[30324]: 1.4 scrub starts 2024-09-21T12:23:24.447 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:24 smithi114 ceph-mon[30324]: 1.4 scrub ok 2024-09-21T12:23:24.447 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:24 smithi114 ceph-mon[30324]: pgmap v140: 158 pgs: 158 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 776 op/s; 1.3 MiB/s, 20 objects/s recovering 2024-09-21T12:23:24.447 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:24 smithi114 ceph-mon[40867]: 1.4 scrub starts 2024-09-21T12:23:24.447 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:24 smithi114 ceph-mon[40867]: 1.4 scrub ok 2024-09-21T12:23:24.447 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:24 smithi114 ceph-mon[40867]: pgmap v140: 158 pgs: 158 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 776 op/s; 1.3 MiB/s, 20 objects/s recovering 2024-09-21T12:23:24.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:24 smithi183 ceph-mon[33277]: 1.4 scrub starts 2024-09-21T12:23:24.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:24 smithi183 ceph-mon[33277]: 1.4 scrub ok 2024-09-21T12:23:24.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:24 smithi183 ceph-mon[33277]: pgmap v140: 158 pgs: 158 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 776 op/s; 1.3 MiB/s, 20 objects/s recovering 2024-09-21T12:23:24.497 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T12:23:24.511 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:24.535 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:24.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:24.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:24.546 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:24.636 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:24.691 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:24.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:24.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:24.793 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:24.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:24.849 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:24.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:24.950 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:24.998 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:24.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:24.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:25.094 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T12:23:25.100 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:25.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:25.181 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:25.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:25.282 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:25.305 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:25.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:25.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:25.334 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:25.407 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:25.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:25.461 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:25.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:25.476 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:25 smithi114 ceph-mon[30324]: 1.6 scrub starts 2024-09-21T12:23:25.476 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:25 smithi114 ceph-mon[30324]: 1.6 scrub ok 2024-09-21T12:23:25.477 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:25 smithi114 ceph-mon[40867]: 1.6 scrub starts 2024-09-21T12:23:25.477 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:25 smithi114 ceph-mon[40867]: 1.6 scrub ok 2024-09-21T12:23:25.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:25 smithi183 ceph-mon[33277]: 1.6 scrub starts 2024-09-21T12:23:25.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:25 smithi183 ceph-mon[33277]: 1.6 scrub ok 2024-09-21T12:23:25.561 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:25.599 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:23:25.601 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:23:25.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:25.603 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:25.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:25.668 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:23:25.704 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:25.726 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:25.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:25.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:25.827 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:25.879 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:25.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:25.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:25.980 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:26.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:26.038 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:26.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:26.139 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:26.153 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:23:26.153 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:23:26.154 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:23:26.154 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:23:26.154 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:23:26.200 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:26.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:26.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:26.301 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:26.325 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:26.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:26.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:26.390 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:26.427 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:26.480 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:26.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:26.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:26.523 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:26 smithi114 ceph-mon[40867]: from='client.24821 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:23:26.523 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:26 smithi114 ceph-mon[40867]: pgmap v141: 158 pgs: 158 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 726 op/s; 1.2 MiB/s, 19 objects/s recovering 2024-09-21T12:23:26.523 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:26 smithi114 ceph-mon[40867]: 1.7 scrub starts 2024-09-21T12:23:26.523 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:26 smithi114 ceph-mon[40867]: 1.7 scrub ok 2024-09-21T12:23:26.523 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:26 smithi114 ceph-mon[30324]: from='client.24821 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:23:26.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:26 smithi114 ceph-mon[30324]: pgmap v141: 158 pgs: 158 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 726 op/s; 1.2 MiB/s, 19 objects/s recovering 2024-09-21T12:23:26.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:26 smithi114 ceph-mon[30324]: 1.7 scrub starts 2024-09-21T12:23:26.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:26 smithi114 ceph-mon[30324]: 1.7 scrub ok 2024-09-21T12:23:26.582 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:26.619 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:26.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:26.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:26.718 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:26 smithi183 ceph-mon[33277]: from='client.24821 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:23:26.718 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:26 smithi183 ceph-mon[33277]: pgmap v141: 158 pgs: 158 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 726 op/s; 1.2 MiB/s, 19 objects/s recovering 2024-09-21T12:23:26.718 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:26 smithi183 ceph-mon[33277]: 1.7 scrub starts 2024-09-21T12:23:26.718 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:26 smithi183 ceph-mon[33277]: 1.7 scrub ok 2024-09-21T12:23:26.721 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:26.767 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:26.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:26.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:26.832 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":140,"stamp":"2024-09-21T12:23:23.604081+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5364920401,"num_objects":82056,"num_object_clones":0,"num_object_copies":246168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82056,"num_whiteouts":0,"num_read":947,"num_read_kb":749,"num_write":126915,"num_write_kb":6665938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16013,"num_bytes_recovered":1037118044,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72222,"ondisk_log_size":72222,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":16583020,"kb_used_data":15843044,"kb_used_omap":10,"kb_used_meta":738741,"kb_avail":639633044,"statfs":{"total":671965249536,"available":654984237056,"internally_reserved":0,"allocated":16223277056,"data_stored":16218144031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":756470891},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-306946892,"num_objects":-4684,"num_object_clones":0,"num_object_copies":-14052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-124,"num_objects_unfound":0,"num_objects_dirty":-4684,"num_whiteouts":0,"num_read":-18,"num_read_kb":-36,"num_write":4660,"num_write_kb":-296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":8019124,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.004637"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":167,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.198799+0000","last_change":"2024-09-21T12:22:16.705200+0000","last_active":"2024-09-21T12:23:20.198799+0000","last_peered":"2024-09-21T12:23:20.198799+0000","last_clean":"2024-09-21T12:23:20.198799+0000","last_became_active":"2024-09-21T12:20:15.309431+0000","last_became_peered":"2024-09-21T12:20:15.309431+0000","last_unstale":"2024-09-21T12:23:20.198799+0000","last_undegraded":"2024-09-21T12:23:20.198799+0000","last_fullsized":"2024-09-21T12:23:20.198799+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:16.705088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:16.705088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:28.011157+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":137,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.228920+0000","last_change":"2024-09-21T12:22:51.040744+0000","last_active":"2024-09-21T12:23:20.228920+0000","last_peered":"2024-09-21T12:23:20.228920+0000","last_clean":"2024-09-21T12:23:20.228920+0000","last_became_active":"2024-09-21T12:22:50.909699+0000","last_became_peered":"2024-09-21T12:22:50.909699+0000","last_unstale":"2024-09-21T12:23:20.228920+0000","last_undegraded":"2024-09-21T12:23:20.228920+0000","last_fullsized":"2024-09-21T12:23:20.228920+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:04.846478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":159,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.433685+0000","last_change":"2024-09-21T12:23:20.433685+0000","last_active":"2024-09-21T12:23:20.433685+0000","last_peered":"2024-09-21T12:23:20.433685+0000","last_clean":"2024-09-21T12:23:20.433685+0000","last_became_active":"2024-09-21T12:20:19.314908+0000","last_became_peered":"2024-09-21T12:20:19.314908+0000","last_unstale":"2024-09-21T12:23:20.433685+0000","last_undegraded":"2024-09-21T12:23:20.433685+0000","last_fullsized":"2024-09-21T12:23:20.433685+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:20.433534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:20.433534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:24.249151+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":90,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.221153+0000","last_change":"2024-09-21T12:22:50.914064+0000","last_active":"2024-09-21T12:23:20.221153+0000","last_peered":"2024-09-21T12:23:20.221153+0000","last_clean":"2024-09-21T12:23:20.221153+0000","last_became_active":"2024-09-21T12:22:50.913743+0000","last_became_peered":"2024-09-21T12:22:50.913743+0000","last_unstale":"2024-09-21T12:23:20.221153+0000","last_undegraded":"2024-09-21T12:23:20.221153+0000","last_fullsized":"2024-09-21T12:23:20.221153+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:58.189665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:21:58.189665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:06.279838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":145,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.202361+0000","last_change":"2024-09-21T12:20:21.595607+0000","last_active":"2024-09-21T12:23:20.202361+0000","last_peered":"2024-09-21T12:23:20.202361+0000","last_clean":"2024-09-21T12:23:20.202361+0000","last_became_active":"2024-09-21T12:20:21.595301+0000","last_became_peered":"2024-09-21T12:20:21.595301+0000","last_unstale":"2024-09-21T12:23:20.202361+0000","last_undegraded":"2024-09-21T12:23:20.202361+0000","last_fullsized":"2024-09-21T12:23:20.202361+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:25.025934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":165,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.203094+0000","last_change":"2024-09-21T12:22:42.026488+0000","last_active":"2024-09-21T12:23:20.203094+0000","last_peered":"2024-09-21T12:23:20.203094+0000","last_clean":"2024-09-21T12:23:20.203094+0000","last_became_active":"2024-09-21T12:20:15.308655+0000","last_became_peered":"2024-09-21T12:20:15.308655+0000","last_unstale":"2024-09-21T12:23:20.203094+0000","last_undegraded":"2024-09-21T12:23:20.203094+0000","last_fullsized":"2024-09-21T12:23:20.203094+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:42.026416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:42.026416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:48.816231+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":186,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.202209+0000","last_change":"2024-09-21T12:22:45.081280+0000","last_active":"2024-09-21T12:23:20.202209+0000","last_peered":"2024-09-21T12:23:20.202209+0000","last_clean":"2024-09-21T12:23:20.202209+0000","last_became_active":"2024-09-21T12:20:17.305220+0000","last_became_peered":"2024-09-21T12:20:17.305220+0000","last_unstale":"2024-09-21T12:23:20.202209+0000","last_undegraded":"2024-09-21T12:23:20.202209+0000","last_fullsized":"2024-09-21T12:23:20.202209+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:22:45.081089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:45.081089+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:11.317199+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":91,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.229012+0000","last_change":"2024-09-21T12:22:50.926564+0000","last_active":"2024-09-21T12:23:20.229012+0000","last_peered":"2024-09-21T12:23:20.229012+0000","last_clean":"2024-09-21T12:23:20.229012+0000","last_became_active":"2024-09-21T12:22:50.926240+0000","last_became_peered":"2024-09-21T12:22:50.926240+0000","last_unstale":"2024-09-21T12:23:20.229012+0000","last_undegraded":"2024-09-21T12:23:20.229012+0000","last_fullsized":"2024-09-21T12:23:20.229012+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:06.670835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:06.670835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:35.841421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":187,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.221090+0000","last_change":"2024-09-21T12:22:50.912964+0000","last_active":"2024-09-21T12:23:20.221090+0000","last_peered":"2024-09-21T12:23:20.221090+0000","last_clean":"2024-09-21T12:23:20.221090+0000","last_became_active":"2024-09-21T12:22:50.912481+0000","last_became_peered":"2024-09-21T12:22:50.912481+0000","last_unstale":"2024-09-21T12:23:20.221090+0000","last_undegraded":"2024-09-21T12:23:20.221090+0000","last_fullsized":"2024-09-21T12:23:20.221090+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:43.207441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:43.207441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:04.703202+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":146,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.202633+0000","last_change":"2024-09-21T12:22:51.018933+0000","last_active":"2024-09-21T12:23:20.202633+0000","last_peered":"2024-09-21T12:23:20.202633+0000","last_clean":"2024-09-21T12:23:20.202633+0000","last_became_active":"2024-09-21T12:22:50.927804+0000","last_became_peered":"2024-09-21T12:22:50.927804+0000","last_unstale":"2024-09-21T12:23:20.202633+0000","last_undegraded":"2024-09-21T12:23:20.202633+0000","last_fullsized":"2024-09-21T12:23:20.202633+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:22:03.017467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:03.017467+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:23.788098+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":158,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.220696+0000","last_change":"2024-09-21T12:22:15.018306+0000","last_active":"2024-09-21T12:23:20.220696+0000","last_peered":"2024-09-21T12:23:20.220696+0000","last_clean":"2024-09-21T12:23:20.220696+0000","last_became_active":"2024-09-21T12:20:19.308184+0000","last_became_peered":"2024-09-21T12:20:19.308184+0000","last_unstale":"2024-09-21T12:23:20.220696+0000","last_undegraded":"2024-09-21T12:23:20.220696+0000","last_fullsized":"2024-09-21T12:23:20.220696+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:15.018264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:15.018264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:44.133250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.196805+0000","last_change":"2024-09-21T12:20:21.309871+0000","last_active":"2024-09-21T12:23:20.196805+0000","last_peered":"2024-09-21T12:23:20.196805+0000","last_clean":"2024-09-21T12:23:20.196805+0000","last_became_active":"2024-09-21T12:20:21.309190+0000","last_became_peered":"2024-09-21T12:20:21.309190+0000","last_unstale":"2024-09-21T12:23:20.196805+0000","last_undegraded":"2024-09-21T12:23:20.196805+0000","last_fullsized":"2024-09-21T12:23:20.196805+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:00.126733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":187,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.220838+0000","last_change":"2024-09-21T12:22:54.174099+0000","last_active":"2024-09-21T12:23:20.220838+0000","last_peered":"2024-09-21T12:23:20.220838+0000","last_clean":"2024-09-21T12:23:20.220838+0000","last_became_active":"2024-09-21T12:22:50.911737+0000","last_became_peered":"2024-09-21T12:22:50.911737+0000","last_unstale":"2024-09-21T12:23:20.220838+0000","last_undegraded":"2024-09-21T12:23:20.220838+0000","last_fullsized":"2024-09-21T12:23:20.220838+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:54.174034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:54.174034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:20.695067+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":132,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.202657+0000","last_change":"2024-09-21T12:22:51.038838+0000","last_active":"2024-09-21T12:23:20.202657+0000","last_peered":"2024-09-21T12:23:20.202657+0000","last_clean":"2024-09-21T12:23:20.202657+0000","last_became_active":"2024-09-21T12:22:50.918794+0000","last_became_peered":"2024-09-21T12:22:50.918794+0000","last_unstale":"2024-09-21T12:23:20.202657+0000","last_undegraded":"2024-09-21T12:23:20.202657+0000","last_fullsized":"2024-09-21T12:23:20.202657+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-21T12:22:10.011633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:10.011633+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:24.501905+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":171,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.200983+0000","last_change":"2024-09-21T12:22:50.922087+0000","last_active":"2024-09-21T12:23:20.200983+0000","last_peered":"2024-09-21T12:23:20.200983+0000","last_clean":"2024-09-21T12:23:20.200983+0000","last_became_active":"2024-09-21T12:22:50.921780+0000","last_became_peered":"2024-09-21T12:22:50.921780+0000","last_unstale":"2024-09-21T12:23:20.200983+0000","last_undegraded":"2024-09-21T12:23:20.200983+0000","last_fullsized":"2024-09-21T12:23:20.200983+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":154,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.220417+0000","last_change":"2024-09-21T12:22:15.977619+0000","last_active":"2024-09-21T12:23:20.220417+0000","last_peered":"2024-09-21T12:23:20.220417+0000","last_clean":"2024-09-21T12:23:20.220417+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:23:20.220417+0000","last_undegraded":"2024-09-21T12:23:20.220417+0000","last_fullsized":"2024-09-21T12:23:20.220417+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:15.977543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:15.977543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:21.175259+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":90,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.230228+0000","last_change":"2024-09-21T12:22:50.915136+0000","last_active":"2024-09-21T12:23:20.230228+0000","last_peered":"2024-09-21T12:23:20.230228+0000","last_clean":"2024-09-21T12:23:20.230228+0000","last_became_active":"2024-09-21T12:22:50.914792+0000","last_became_peered":"2024-09-21T12:22:50.914792+0000","last_unstale":"2024-09-21T12:23:20.230228+0000","last_undegraded":"2024-09-21T12:23:20.230228+0000","last_fullsized":"2024-09-21T12:23:20.230228+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:02.712009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:02.712009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:28.145066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":196,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.197921+0000","last_change":"2024-09-21T12:23:19.156525+0000","last_active":"2024-09-21T12:23:20.197921+0000","last_peered":"2024-09-21T12:23:20.197921+0000","last_clean":"2024-09-21T12:23:20.197921+0000","last_became_active":"2024-09-21T12:22:50.921576+0000","last_became_peered":"2024-09-21T12:22:50.921576+0000","last_unstale":"2024-09-21T12:23:20.197921+0000","last_undegraded":"2024-09-21T12:23:20.197921+0000","last_fullsized":"2024-09-21T12:23:20.197921+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:19.156456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:23:19.156456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:24.315189+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":180,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.200648+0000","last_change":"2024-09-21T12:20:17.308186+0000","last_active":"2024-09-21T12:23:20.200648+0000","last_peered":"2024-09-21T12:23:20.200648+0000","last_clean":"2024-09-21T12:23:20.200648+0000","last_became_active":"2024-09-21T12:20:17.307828+0000","last_became_peered":"2024-09-21T12:20:17.307828+0000","last_unstale":"2024-09-21T12:23:20.200648+0000","last_undegraded":"2024-09-21T12:23:20.200648+0000","last_fullsized":"2024-09-21T12:23:20.200648+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":180,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.197048+0000","last_change":"2024-09-21T12:22:50.930597+0000","last_active":"2024-09-21T12:23:20.197048+0000","last_peered":"2024-09-21T12:23:20.197048+0000","last_clean":"2024-09-21T12:23:20.197048+0000","last_became_active":"2024-09-21T12:22:50.928972+0000","last_became_peered":"2024-09-21T12:22:50.928972+0000","last_unstale":"2024-09-21T12:23:20.197048+0000","last_undegraded":"2024-09-21T12:23:20.197048+0000","last_fullsized":"2024-09-21T12:23:20.197048+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:10.823915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:10.823915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:31.452808+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":145,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.202982+0000","last_change":"2024-09-21T12:20:21.593129+0000","last_active":"2024-09-21T12:23:20.202982+0000","last_peered":"2024-09-21T12:23:20.202982+0000","last_clean":"2024-09-21T12:23:20.202982+0000","last_became_active":"2024-09-21T12:20:21.592796+0000","last_became_peered":"2024-09-21T12:20:21.592796+0000","last_unstale":"2024-09-21T12:23:20.202982+0000","last_undegraded":"2024-09-21T12:23:20.202982+0000","last_fullsized":"2024-09-21T12:23:20.202982+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":90,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.197751+0000","last_change":"2024-09-21T12:22:50.926169+0000","last_active":"2024-09-21T12:23:20.197751+0000","last_peered":"2024-09-21T12:23:20.197751+0000","last_clean":"2024-09-21T12:23:20.197751+0000","last_became_active":"2024-09-21T12:22:50.925403+0000","last_became_peered":"2024-09-21T12:22:50.925403+0000","last_unstale":"2024-09-21T12:23:20.197751+0000","last_undegraded":"2024-09-21T12:23:20.197751+0000","last_fullsized":"2024-09-21T12:23:20.197751+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:07.846732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:07.846732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:35.648599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":233,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.202716+0000","last_change":"2024-09-21T12:22:50.983054+0000","last_active":"2024-09-21T12:23:20.202716+0000","last_peered":"2024-09-21T12:23:20.202716+0000","last_clean":"2024-09-21T12:23:20.202716+0000","last_became_active":"2024-09-21T12:22:50.912144+0000","last_became_peered":"2024-09-21T12:22:50.912144+0000","last_unstale":"2024-09-21T12:23:20.202716+0000","last_undegraded":"2024-09-21T12:23:20.202716+0000","last_fullsized":"2024-09-21T12:23:20.202716+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":83,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.202046+0000","last_change":"2024-09-21T12:22:50.920178+0000","last_active":"2024-09-21T12:23:20.202046+0000","last_peered":"2024-09-21T12:23:20.202046+0000","last_clean":"2024-09-21T12:23:20.202046+0000","last_became_active":"2024-09-21T12:22:50.918326+0000","last_became_peered":"2024-09-21T12:22:50.918326+0000","last_unstale":"2024-09-21T12:23:20.202046+0000","last_undegraded":"2024-09-21T12:23:20.202046+0000","last_fullsized":"2024-09-21T12:23:20.202046+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.201321+0000","last_change":"2024-09-21T12:20:21.313476+0000","last_active":"2024-09-21T12:23:20.201321+0000","last_peered":"2024-09-21T12:23:20.201321+0000","last_clean":"2024-09-21T12:23:20.201321+0000","last_became_active":"2024-09-21T12:20:21.312976+0000","last_became_peered":"2024-09-21T12:20:21.312976+0000","last_unstale":"2024-09-21T12:23:20.201321+0000","last_undegraded":"2024-09-21T12:23:20.201321+0000","last_fullsized":"2024-09-21T12:23:20.201321+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":201,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.196682+0000","last_change":"2024-09-21T12:22:52.070589+0000","last_active":"2024-09-21T12:23:20.196682+0000","last_peered":"2024-09-21T12:23:20.196682+0000","last_clean":"2024-09-21T12:23:20.196682+0000","last_became_active":"2024-09-21T12:22:50.910933+0000","last_became_peered":"2024-09-21T12:22:50.910933+0000","last_unstale":"2024-09-21T12:23:20.196682+0000","last_undegraded":"2024-09-21T12:23:20.196682+0000","last_fullsized":"2024-09-21T12:23:20.196682+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:22:52.070513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:52.070513+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:07.437026+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":170,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.198521+0000","last_change":"2024-09-21T12:20:17.696423+0000","last_active":"2024-09-21T12:23:20.198521+0000","last_peered":"2024-09-21T12:23:20.198521+0000","last_clean":"2024-09-21T12:23:20.198521+0000","last_became_active":"2024-09-21T12:20:17.695977+0000","last_became_peered":"2024-09-21T12:20:17.695977+0000","last_unstale":"2024-09-21T12:23:20.198521+0000","last_undegraded":"2024-09-21T12:23:20.198521+0000","last_fullsized":"2024-09-21T12:23:20.198521+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:59.091497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"121'11","reported_seq":240,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.230611+0000","last_change":"2024-09-21T12:22:50.926408+0000","last_active":"2024-09-21T12:23:20.230611+0000","last_peered":"2024-09-21T12:23:20.230611+0000","last_clean":"2024-09-21T12:23:20.230611+0000","last_became_active":"2024-09-21T12:22:50.916683+0000","last_became_peered":"2024-09-21T12:22:50.916683+0000","last_unstale":"2024-09-21T12:23:20.230611+0000","last_undegraded":"2024-09-21T12:23:20.230611+0000","last_fullsized":"2024-09-21T12:23:20.230611+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'9","last_scrub_stamp":"2024-09-21T12:22:08.647997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:08.647997+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:21.181826+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"76'79","reported_seq":66,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.221574+0000","last_change":"2024-09-21T12:22:18.000752+0000","last_active":"2024-09-21T12:23:20.221574+0000","last_peered":"2024-09-21T12:23:20.221574+0000","last_clean":"2024-09-21T12:23:20.221574+0000","last_became_active":"2024-09-21T12:22:17.074721+0000","last_became_peered":"2024-09-21T12:22:17.074721+0000","last_unstale":"2024-09-21T12:23:20.221574+0000","last_undegraded":"2024-09-21T12:23:20.221574+0000","last_fullsized":"2024-09-21T12:23:20.221574+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:22:18.000705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:22:18.000705+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:29.821832+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"132'7815","reported_seq":7985,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:21.579720+0000","last_change":"2024-09-21T12:22:42.121876+0000","last_active":"2024-09-21T12:23:21.579720+0000","last_peered":"2024-09-21T12:23:21.579720+0000","last_clean":"2024-09-21T12:23:21.579720+0000","last_became_active":"2024-09-21T12:21:24.886130+0000","last_became_peered":"2024-09-21T12:21:24.886130+0000","last_unstale":"2024-09-21T12:23:21.579720+0000","last_undegraded":"2024-09-21T12:23:21.579720+0000","last_fullsized":"2024-09-21T12:23:21.579720+0000","mapping_epoch":84,"log_start":"102'3247","ondisk_log_start":"102'3247","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5485","last_scrub_stamp":"2024-09-21T12:22:42.121763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:22:42.121763+0000","objects_scrubbed":5450,"log_size":4568,"log_dups_size":0,"ondisk_log_size":4568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:11.078251+0000","scrub_duration":1486,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333905920,"num_objects":5095,"num_object_clones":0,"num_object_copies":15285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7815,"num_write_kb":413120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":249,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.196692+0000","last_change":"2024-09-21T12:22:50.956017+0000","last_active":"2024-09-21T12:23:20.196692+0000","last_peered":"2024-09-21T12:23:20.196692+0000","last_clean":"2024-09-21T12:23:20.196692+0000","last_became_active":"2024-09-21T12:22:50.906153+0000","last_became_peered":"2024-09-21T12:22:50.906153+0000","last_unstale":"2024-09-21T12:23:20.196692+0000","last_undegraded":"2024-09-21T12:23:20.196692+0000","last_fullsized":"2024-09-21T12:23:20.196692+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":165,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.202307+0000","last_change":"2024-09-21T12:22:14.036290+0000","last_active":"2024-09-21T12:23:20.202307+0000","last_peered":"2024-09-21T12:23:20.202307+0000","last_clean":"2024-09-21T12:23:20.202307+0000","last_became_active":"2024-09-21T12:20:15.295774+0000","last_became_peered":"2024-09-21T12:20:15.295774+0000","last_unstale":"2024-09-21T12:23:20.202307+0000","last_undegraded":"2024-09-21T12:23:20.202307+0000","last_fullsized":"2024-09-21T12:23:20.202307+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:14.036236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.036236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:39.734223+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"130'12","reported_seq":23,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.201664+0000","last_change":"2024-09-21T12:23:11.832151+0000","last_active":"2024-09-21T12:23:20.201664+0000","last_peered":"2024-09-21T12:23:20.201664+0000","last_clean":"2024-09-21T12:23:20.201664+0000","last_became_active":"2024-09-21T12:23:11.827128+0000","last_became_peered":"2024-09-21T12:23:11.827128+0000","last_unstale":"2024-09-21T12:23:20.201664+0000","last_undegraded":"2024-09-21T12:23:20.201664+0000","last_fullsized":"2024-09-21T12:23:20.201664+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":155,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.220594+0000","last_change":"2024-09-21T12:22:12.145637+0000","last_active":"2024-09-21T12:23:20.220594+0000","last_peered":"2024-09-21T12:23:20.220594+0000","last_clean":"2024-09-21T12:23:20.220594+0000","last_became_active":"2024-09-21T12:20:21.310022+0000","last_became_peered":"2024-09-21T12:20:21.310022+0000","last_unstale":"2024-09-21T12:23:20.220594+0000","last_undegraded":"2024-09-21T12:23:20.220594+0000","last_fullsized":"2024-09-21T12:23:20.220594+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:22:12.145550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:12.145550+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:41.591945+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"76'79","reported_seq":55,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.200270+0000","last_change":"2024-09-21T12:22:50.916239+0000","last_active":"2024-09-21T12:23:20.200270+0000","last_peered":"2024-09-21T12:23:20.200270+0000","last_clean":"2024-09-21T12:23:20.200270+0000","last_became_active":"2024-09-21T12:22:50.914885+0000","last_became_peered":"2024-09-21T12:22:50.914885+0000","last_unstale":"2024-09-21T12:23:20.200270+0000","last_undegraded":"2024-09-21T12:23:20.200270+0000","last_fullsized":"2024-09-21T12:23:20.200270+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:22:17.824342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:22:17.824342+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:29.442279+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.3","version":"132'7816","reported_seq":11020,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:21.815883+0000","last_change":"2024-09-21T12:22:50.925842+0000","last_active":"2024-09-21T12:23:21.815883+0000","last_peered":"2024-09-21T12:23:21.815883+0000","last_clean":"2024-09-21T12:23:21.815883+0000","last_became_active":"2024-09-21T12:22:50.919145+0000","last_became_peered":"2024-09-21T12:22:50.919145+0000","last_unstale":"2024-09-21T12:23:21.815883+0000","last_undegraded":"2024-09-21T12:23:21.815883+0000","last_fullsized":"2024-09-21T12:23:21.815883+0000","mapping_epoch":120,"log_start":"107'3866","ondisk_log_start":"107'3866","created":84,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":3950,"log_dups_size":0,"ondisk_log_size":3950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328728576,"num_objects":5016,"num_object_clones":0,"num_object_copies":15048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7816,"num_write_kb":410624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1536,"num_bytes_recovered":100663296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":185,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.202058+0000","last_change":"2024-09-21T12:20:17.693550+0000","last_active":"2024-09-21T12:23:20.202058+0000","last_peered":"2024-09-21T12:23:20.202058+0000","last_clean":"2024-09-21T12:23:20.202058+0000","last_became_active":"2024-09-21T12:20:17.693179+0000","last_became_peered":"2024-09-21T12:20:17.693179+0000","last_unstale":"2024-09-21T12:23:20.202058+0000","last_undegraded":"2024-09-21T12:23:20.202058+0000","last_fullsized":"2024-09-21T12:23:20.202058+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":165,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.203193+0000","last_change":"2024-09-21T12:21:18.056305+0000","last_active":"2024-09-21T12:23:20.203193+0000","last_peered":"2024-09-21T12:23:20.203193+0000","last_clean":"2024-09-21T12:23:20.203193+0000","last_became_active":"2024-09-21T12:20:15.297340+0000","last_became_peered":"2024-09-21T12:20:15.297340+0000","last_unstale":"2024-09-21T12:23:20.203193+0000","last_undegraded":"2024-09-21T12:23:20.203193+0000","last_fullsized":"2024-09-21T12:23:20.203193+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:18.056261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:18.056261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":91,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.201942+0000","last_change":"2024-09-21T12:22:50.931233+0000","last_active":"2024-09-21T12:23:20.201942+0000","last_peered":"2024-09-21T12:23:20.201942+0000","last_clean":"2024-09-21T12:23:20.201942+0000","last_became_active":"2024-09-21T12:22:50.930858+0000","last_became_peered":"2024-09-21T12:22:50.930858+0000","last_unstale":"2024-09-21T12:23:20.201942+0000","last_undegraded":"2024-09-21T12:23:20.201942+0000","last_fullsized":"2024-09-21T12:23:20.201942+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:01.971812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:01.971812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":90,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.202528+0000","last_change":"2024-09-21T12:22:50.927366+0000","last_active":"2024-09-21T12:23:20.202528+0000","last_peered":"2024-09-21T12:23:20.202528+0000","last_clean":"2024-09-21T12:23:20.202528+0000","last_became_active":"2024-09-21T12:22:50.918940+0000","last_became_peered":"2024-09-21T12:22:50.918940+0000","last_unstale":"2024-09-21T12:23:20.202528+0000","last_undegraded":"2024-09-21T12:23:20.202528+0000","last_fullsized":"2024-09-21T12:23:20.202528+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:13.050383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:13.050383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:31.135674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"76'79","reported_seq":63,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.201823+0000","last_change":"2024-09-21T12:22:17.098015+0000","last_active":"2024-09-21T12:23:20.201823+0000","last_peered":"2024-09-21T12:23:20.201823+0000","last_clean":"2024-09-21T12:23:20.201823+0000","last_became_active":"2024-09-21T12:22:17.069042+0000","last_became_peered":"2024-09-21T12:22:17.069042+0000","last_unstale":"2024-09-21T12:23:20.201823+0000","last_undegraded":"2024-09-21T12:23:20.201823+0000","last_fullsized":"2024-09-21T12:23:20.201823+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T12:20:24.850157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:20:24.850157+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:23:21.985894+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":381350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"132'8105","reported_seq":8212,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:21.803982+0000","last_change":"2024-09-21T12:21:24.889433+0000","last_active":"2024-09-21T12:23:21.803982+0000","last_peered":"2024-09-21T12:23:21.803982+0000","last_clean":"2024-09-21T12:23:21.803982+0000","last_became_active":"2024-09-21T12:21:24.889133+0000","last_became_peered":"2024-09-21T12:21:24.889133+0000","last_unstale":"2024-09-21T12:23:21.803982+0000","last_undegraded":"2024-09-21T12:23:21.803982+0000","last_fullsized":"2024-09-21T12:23:21.803982+0000","mapping_epoch":84,"log_start":"108'4113","ondisk_log_start":"108'4113","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":3992,"log_dups_size":0,"ondisk_log_size":3992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334036992,"num_objects":5097,"num_object_clones":0,"num_object_copies":15291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8105,"num_write_kb":422464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":163,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.220411+0000","last_change":"2024-09-21T12:22:50.978102+0000","last_active":"2024-09-21T12:23:20.220411+0000","last_peered":"2024-09-21T12:23:20.220411+0000","last_clean":"2024-09-21T12:23:20.220411+0000","last_became_active":"2024-09-21T12:22:50.926008+0000","last_became_peered":"2024-09-21T12:22:50.926008+0000","last_unstale":"2024-09-21T12:23:20.220411+0000","last_undegraded":"2024-09-21T12:23:20.220411+0000","last_fullsized":"2024-09-21T12:23:20.220411+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-21T12:22:14.062464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.062464+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:38.500422+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":201,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.198691+0000","last_change":"2024-09-21T12:22:51.003768+0000","last_active":"2024-09-21T12:23:20.198691+0000","last_peered":"2024-09-21T12:23:20.198691+0000","last_clean":"2024-09-21T12:23:20.198691+0000","last_became_active":"2024-09-21T12:22:50.923284+0000","last_became_peered":"2024-09-21T12:22:50.923284+0000","last_unstale":"2024-09-21T12:23:20.198691+0000","last_undegraded":"2024-09-21T12:23:20.198691+0000","last_fullsized":"2024-09-21T12:23:20.198691+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:22:07.582969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:07.582969+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:20.480647+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"121'11","reported_seq":228,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.202399+0000","last_change":"2024-09-21T12:22:51.062896+0000","last_active":"2024-09-21T12:23:20.202399+0000","last_peered":"2024-09-21T12:23:20.202399+0000","last_clean":"2024-09-21T12:23:20.202399+0000","last_became_active":"2024-09-21T12:22:50.930342+0000","last_became_peered":"2024-09-21T12:22:50.930342+0000","last_unstale":"2024-09-21T12:23:20.202399+0000","last_undegraded":"2024-09-21T12:23:20.202399+0000","last_fullsized":"2024-09-21T12:23:20.202399+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":154,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.200969+0000","last_change":"2024-09-21T12:21:21.835669+0000","last_active":"2024-09-21T12:23:20.200969+0000","last_peered":"2024-09-21T12:23:20.200969+0000","last_clean":"2024-09-21T12:23:20.200969+0000","last_became_active":"2024-09-21T12:20:21.593367+0000","last_became_peered":"2024-09-21T12:20:21.593367+0000","last_unstale":"2024-09-21T12:23:20.200969+0000","last_undegraded":"2024-09-21T12:23:20.200969+0000","last_fullsized":"2024-09-21T12:23:20.200969+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:21.835623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:21:21.835623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"76'79","reported_seq":148,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.230416+0000","last_change":"2024-09-21T12:22:17.093539+0000","last_active":"2024-09-21T12:23:20.230416+0000","last_peered":"2024-09-21T12:23:20.230416+0000","last_clean":"2024-09-21T12:23:20.230416+0000","last_became_active":"2024-09-21T12:22:17.093298+0000","last_became_peered":"2024-09-21T12:22:17.093298+0000","last_unstale":"2024-09-21T12:23:20.230416+0000","last_undegraded":"2024-09-21T12:23:20.230416+0000","last_fullsized":"2024-09-21T12:23:20.230416+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T12:20:24.850157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:20:24.850157+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:23:23.614841+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"132'8120","reported_seq":17278,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:21.799121+0000","last_change":"2024-09-21T12:22:51.819626+0000","last_active":"2024-09-21T12:23:21.799121+0000","last_peered":"2024-09-21T12:23:21.799121+0000","last_clean":"2024-09-21T12:23:21.799121+0000","last_became_active":"2024-09-21T12:22:51.808221+0000","last_became_peered":"2024-09-21T12:22:51.808221+0000","last_unstale":"2024-09-21T12:23:21.799121+0000","last_undegraded":"2024-09-21T12:23:21.799121+0000","last_fullsized":"2024-09-21T12:23:21.799121+0000","mapping_epoch":121,"log_start":"109'4188","ondisk_log_start":"109'4188","created":84,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":3932,"log_dups_size":0,"ondisk_log_size":3932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343932928,"num_objects":5248,"num_object_clones":0,"num_object_copies":15744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8120,"num_write_kb":427776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4516,"num_bytes_recovered":295960576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":227,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.197684+0000","last_change":"2024-09-21T12:22:51.006819+0000","last_active":"2024-09-21T12:23:20.197684+0000","last_peered":"2024-09-21T12:23:20.197684+0000","last_clean":"2024-09-21T12:23:20.197684+0000","last_became_active":"2024-09-21T12:22:50.925840+0000","last_became_peered":"2024-09-21T12:22:50.925840+0000","last_unstale":"2024-09-21T12:23:20.197684+0000","last_undegraded":"2024-09-21T12:23:20.197684+0000","last_fullsized":"2024-09-21T12:23:20.197684+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:01.233288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":166,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.221294+0000","last_change":"2024-09-21T12:22:37.176483+0000","last_active":"2024-09-21T12:23:20.221294+0000","last_peered":"2024-09-21T12:23:20.221294+0000","last_clean":"2024-09-21T12:23:20.221294+0000","last_became_active":"2024-09-21T12:20:15.308518+0000","last_became_peered":"2024-09-21T12:20:15.308518+0000","last_unstale":"2024-09-21T12:23:20.221294+0000","last_undegraded":"2024-09-21T12:23:20.221294+0000","last_fullsized":"2024-09-21T12:23:20.221294+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:37.176413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:37.176413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:01.193747+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":208,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.197466+0000","last_change":"2024-09-21T12:21:23.139205+0000","last_active":"2024-09-21T12:23:20.197466+0000","last_peered":"2024-09-21T12:23:20.197466+0000","last_clean":"2024-09-21T12:23:20.197466+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:23:20.197466+0000","last_undegraded":"2024-09-21T12:23:20.197466+0000","last_fullsized":"2024-09-21T12:23:20.197466+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:21:23.139124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.139124+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":155,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.196930+0000","last_change":"2024-09-21T12:22:51.987032+0000","last_active":"2024-09-21T12:23:20.196930+0000","last_peered":"2024-09-21T12:23:20.196930+0000","last_clean":"2024-09-21T12:23:20.196930+0000","last_became_active":"2024-09-21T12:20:21.315801+0000","last_became_peered":"2024-09-21T12:20:21.315801+0000","last_unstale":"2024-09-21T12:23:20.196930+0000","last_undegraded":"2024-09-21T12:23:20.196930+0000","last_fullsized":"2024-09-21T12:23:20.196930+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:51.986953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:51.986953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:58.750058+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"76'79","reported_seq":148,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.230205+0000","last_change":"2024-09-21T12:22:17.083917+0000","last_active":"2024-09-21T12:23:20.230205+0000","last_peered":"2024-09-21T12:23:20.230205+0000","last_clean":"2024-09-21T12:23:20.230205+0000","last_became_active":"2024-09-21T12:22:17.083629+0000","last_became_peered":"2024-09-21T12:22:17.083629+0000","last_unstale":"2024-09-21T12:23:20.230205+0000","last_undegraded":"2024-09-21T12:23:20.230205+0000","last_fullsized":"2024-09-21T12:23:20.230205+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T12:20:24.850157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:20:24.850157+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:23:22.611693+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"132'7727","reported_seq":7899,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:21.548080+0000","last_change":"2024-09-21T12:22:48.664422+0000","last_active":"2024-09-21T12:23:21.548080+0000","last_peered":"2024-09-21T12:23:21.548080+0000","last_clean":"2024-09-21T12:23:21.548080+0000","last_became_active":"2024-09-21T12:21:24.884183+0000","last_became_peered":"2024-09-21T12:21:24.884183+0000","last_unstale":"2024-09-21T12:23:21.548080+0000","last_undegraded":"2024-09-21T12:23:21.548080+0000","last_fullsized":"2024-09-21T12:23:21.548080+0000","mapping_epoch":84,"log_start":"98'2791","ondisk_log_start":"98'2791","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'5758","last_scrub_stamp":"2024-09-21T12:22:48.664234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:22:48.664234+0000","objects_scrubbed":5726,"log_size":4936,"log_dups_size":0,"ondisk_log_size":4936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:12.546965+0000","scrub_duration":1463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332726272,"num_objects":5077,"num_object_clones":0,"num_object_copies":15231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7727,"num_write_kb":409728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":191,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.197633+0000","last_change":"2024-09-21T12:20:17.693585+0000","last_active":"2024-09-21T12:23:20.197633+0000","last_peered":"2024-09-21T12:23:20.197633+0000","last_clean":"2024-09-21T12:23:20.197633+0000","last_became_active":"2024-09-21T12:20:17.693067+0000","last_became_peered":"2024-09-21T12:20:17.693067+0000","last_unstale":"2024-09-21T12:23:20.197633+0000","last_undegraded":"2024-09-21T12:23:20.197633+0000","last_fullsized":"2024-09-21T12:23:20.197633+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:20.707330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":159,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.197342+0000","last_change":"2024-09-21T12:20:15.308741+0000","last_active":"2024-09-21T12:23:20.197342+0000","last_peered":"2024-09-21T12:23:20.197342+0000","last_clean":"2024-09-21T12:23:20.197342+0000","last_became_active":"2024-09-21T12:20:15.308441+0000","last_became_peered":"2024-09-21T12:20:15.308441+0000","last_unstale":"2024-09-21T12:23:20.197342+0000","last_undegraded":"2024-09-21T12:23:20.197342+0000","last_fullsized":"2024-09-21T12:23:20.197342+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:05.332981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":158,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.229217+0000","last_change":"2024-09-21T12:22:34.563365+0000","last_active":"2024-09-21T12:23:20.229217+0000","last_peered":"2024-09-21T12:23:20.229217+0000","last_clean":"2024-09-21T12:23:20.229217+0000","last_became_active":"2024-09-21T12:20:19.309598+0000","last_became_peered":"2024-09-21T12:20:19.309598+0000","last_unstale":"2024-09-21T12:23:20.229217+0000","last_undegraded":"2024-09-21T12:23:20.229217+0000","last_fullsized":"2024-09-21T12:23:20.229217+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:34.563282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:34.563282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:55.460915+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.196303+0000","last_change":"2024-09-21T12:20:21.593173+0000","last_active":"2024-09-21T12:23:20.196303+0000","last_peered":"2024-09-21T12:23:20.196303+0000","last_clean":"2024-09-21T12:23:20.196303+0000","last_became_active":"2024-09-21T12:20:21.592828+0000","last_became_peered":"2024-09-21T12:20:21.592828+0000","last_unstale":"2024-09-21T12:23:20.196303+0000","last_undegraded":"2024-09-21T12:23:20.196303+0000","last_fullsized":"2024-09-21T12:23:20.196303+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"76'79","reported_seq":66,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:21.346117+0000","last_change":"2024-09-21T12:23:21.346117+0000","last_active":"2024-09-21T12:23:21.346117+0000","last_peered":"2024-09-21T12:23:21.346117+0000","last_clean":"2024-09-21T12:23:21.346117+0000","last_became_active":"2024-09-21T12:22:17.084732+0000","last_became_peered":"2024-09-21T12:22:17.084732+0000","last_unstale":"2024-09-21T12:23:21.346117+0000","last_undegraded":"2024-09-21T12:23:21.346117+0000","last_fullsized":"2024-09-21T12:23:21.346117+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:23:21.346046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:23:21.346046+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:22.689177+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"132'8111","reported_seq":8285,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:21.562992+0000","last_change":"2024-09-21T12:22:50.126449+0000","last_active":"2024-09-21T12:23:21.562992+0000","last_peered":"2024-09-21T12:23:21.562992+0000","last_clean":"2024-09-21T12:23:21.562992+0000","last_became_active":"2024-09-21T12:21:24.886854+0000","last_became_peered":"2024-09-21T12:21:24.886854+0000","last_unstale":"2024-09-21T12:23:21.562992+0000","last_undegraded":"2024-09-21T12:23:21.562992+0000","last_fullsized":"2024-09-21T12:23:21.562992+0000","mapping_epoch":84,"log_start":"103'3500","ondisk_log_start":"103'3500","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'6045","last_scrub_stamp":"2024-09-21T12:22:50.126349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:22:50.126349+0000","objects_scrubbed":6025,"log_size":4611,"log_dups_size":0,"ondisk_log_size":4611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:09.057671+0000","scrub_duration":1386,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335740928,"num_objects":5123,"num_object_clones":0,"num_object_copies":15369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8111,"num_write_kb":423488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":241,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.197245+0000","last_change":"2024-09-21T12:20:17.308049+0000","last_active":"2024-09-21T12:23:20.197245+0000","last_peered":"2024-09-21T12:23:20.197245+0000","last_clean":"2024-09-21T12:23:20.197245+0000","last_became_active":"2024-09-21T12:20:17.307769+0000","last_became_peered":"2024-09-21T12:20:17.307769+0000","last_unstale":"2024-09-21T12:23:20.197245+0000","last_undegraded":"2024-09-21T12:23:20.197245+0000","last_fullsized":"2024-09-21T12:23:20.197245+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:03.262693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":90,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.197845+0000","last_change":"2024-09-21T12:22:50.922961+0000","last_active":"2024-09-21T12:23:20.197845+0000","last_peered":"2024-09-21T12:23:20.197845+0000","last_clean":"2024-09-21T12:23:20.197845+0000","last_became_active":"2024-09-21T12:22:50.922002+0000","last_became_peered":"2024-09-21T12:22:50.922002+0000","last_unstale":"2024-09-21T12:23:20.197845+0000","last_undegraded":"2024-09-21T12:23:20.197845+0000","last_fullsized":"2024-09-21T12:23:20.197845+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:05.507899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:05.507899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":176,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.221626+0000","last_change":"2024-09-21T12:21:20.045824+0000","last_active":"2024-09-21T12:23:20.221626+0000","last_peered":"2024-09-21T12:23:20.221626+0000","last_clean":"2024-09-21T12:23:20.221626+0000","last_became_active":"2024-09-21T12:20:14.287998+0000","last_became_peered":"2024-09-21T12:20:14.287998+0000","last_unstale":"2024-09-21T12:23:20.221626+0000","last_undegraded":"2024-09-21T12:23:20.221626+0000","last_fullsized":"2024-09-21T12:23:20.221626+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T12:21:20.045755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.045755+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:05.288744+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011011149999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":180,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.198337+0000","last_change":"2024-09-21T12:22:50.921798+0000","last_active":"2024-09-21T12:23:20.198337+0000","last_peered":"2024-09-21T12:23:20.198337+0000","last_clean":"2024-09-21T12:23:20.198337+0000","last_became_active":"2024-09-21T12:22:50.921488+0000","last_became_peered":"2024-09-21T12:22:50.921488+0000","last_unstale":"2024-09-21T12:23:20.198337+0000","last_undegraded":"2024-09-21T12:23:20.198337+0000","last_fullsized":"2024-09-21T12:23:20.198337+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:06.563104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:06.563104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:28.835407+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":174,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.237788+0000","last_change":"2024-09-21T12:23:20.237788+0000","last_active":"2024-09-21T12:23:20.237788+0000","last_peered":"2024-09-21T12:23:20.237788+0000","last_clean":"2024-09-21T12:23:20.237788+0000","last_became_active":"2024-09-21T12:22:50.917558+0000","last_became_peered":"2024-09-21T12:22:50.917558+0000","last_unstale":"2024-09-21T12:23:20.237788+0000","last_undegraded":"2024-09-21T12:23:20.237788+0000","last_fullsized":"2024-09-21T12:23:20.237788+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:20.237715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:20.237715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:40.331198+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'79","reported_seq":241,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.200852+0000","last_change":"2024-09-21T12:23:19.867918+0000","last_active":"2024-09-21T12:23:20.200852+0000","last_peered":"2024-09-21T12:23:20.200852+0000","last_clean":"2024-09-21T12:23:20.200852+0000","last_became_active":"2024-09-21T12:22:50.921709+0000","last_became_peered":"2024-09-21T12:22:50.921709+0000","last_unstale":"2024-09-21T12:23:20.200852+0000","last_undegraded":"2024-09-21T12:23:20.200852+0000","last_fullsized":"2024-09-21T12:23:20.200852+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:23:19.867868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:23:19.867868+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:49.599838+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"132'7798","reported_seq":10882,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:21.780062+0000","last_change":"2024-09-21T12:22:51.045889+0000","last_active":"2024-09-21T12:23:21.780062+0000","last_peered":"2024-09-21T12:23:21.780062+0000","last_clean":"2024-09-21T12:23:21.780062+0000","last_became_active":"2024-09-21T12:22:50.936833+0000","last_became_peered":"2024-09-21T12:22:50.936833+0000","last_unstale":"2024-09-21T12:23:21.780062+0000","last_undegraded":"2024-09-21T12:23:21.780062+0000","last_fullsized":"2024-09-21T12:23:21.780062+0000","mapping_epoch":120,"log_start":"103'3553","ondisk_log_start":"103'3553","created":84,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4245,"log_dups_size":0,"ondisk_log_size":4245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:03.176515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333185024,"num_objects":5084,"num_object_clones":0,"num_object_copies":15252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7798,"num_write_kb":412224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1475,"num_bytes_recovered":96665600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":182,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.196789+0000","last_change":"2024-09-21T12:20:17.297678+0000","last_active":"2024-09-21T12:23:20.196789+0000","last_peered":"2024-09-21T12:23:20.196789+0000","last_clean":"2024-09-21T12:23:20.196789+0000","last_became_active":"2024-09-21T12:20:17.297374+0000","last_became_peered":"2024-09-21T12:20:17.297374+0000","last_unstale":"2024-09-21T12:23:20.196789+0000","last_undegraded":"2024-09-21T12:23:20.196789+0000","last_fullsized":"2024-09-21T12:23:20.196789+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:22.813197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":167,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.196494+0000","last_change":"2024-09-21T12:22:53.116442+0000","last_active":"2024-09-21T12:23:20.196494+0000","last_peered":"2024-09-21T12:23:20.196494+0000","last_clean":"2024-09-21T12:23:20.196494+0000","last_became_active":"2024-09-21T12:20:15.308356+0000","last_became_peered":"2024-09-21T12:20:15.308356+0000","last_unstale":"2024-09-21T12:23:20.196494+0000","last_undegraded":"2024-09-21T12:23:20.196494+0000","last_fullsized":"2024-09-21T12:23:20.196494+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:53.116377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:53.116377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:54.003540+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":176,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.199700+0000","last_change":"2024-09-21T12:21:24.390178+0000","last_active":"2024-09-21T12:23:20.199700+0000","last_peered":"2024-09-21T12:23:20.199700+0000","last_clean":"2024-09-21T12:23:20.199700+0000","last_became_active":"2024-09-21T12:20:14.288358+0000","last_became_peered":"2024-09-21T12:20:14.288358+0000","last_unstale":"2024-09-21T12:23:20.199700+0000","last_undegraded":"2024-09-21T12:23:20.199700+0000","last_fullsized":"2024-09-21T12:23:20.199700+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:21:24.390129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:21:24.390129+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019042449999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":151,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.200984+0000","last_change":"2024-09-21T12:20:19.310164+0000","last_active":"2024-09-21T12:23:20.200984+0000","last_peered":"2024-09-21T12:23:20.200984+0000","last_clean":"2024-09-21T12:23:20.200984+0000","last_became_active":"2024-09-21T12:20:19.309724+0000","last_became_peered":"2024-09-21T12:20:19.309724+0000","last_unstale":"2024-09-21T12:23:20.200984+0000","last_undegraded":"2024-09-21T12:23:20.200984+0000","last_fullsized":"2024-09-21T12:23:20.200984+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":146,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.201447+0000","last_change":"2024-09-21T12:20:21.593267+0000","last_active":"2024-09-21T12:23:20.201447+0000","last_peered":"2024-09-21T12:23:20.201447+0000","last_clean":"2024-09-21T12:23:20.201447+0000","last_became_active":"2024-09-21T12:20:21.592982+0000","last_became_peered":"2024-09-21T12:20:21.592982+0000","last_unstale":"2024-09-21T12:23:20.201447+0000","last_undegraded":"2024-09-21T12:23:20.201447+0000","last_fullsized":"2024-09-21T12:23:20.201447+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"76'79","reported_seq":80,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.197614+0000","last_change":"2024-09-21T12:22:50.912481+0000","last_active":"2024-09-21T12:23:20.197614+0000","last_peered":"2024-09-21T12:23:20.197614+0000","last_clean":"2024-09-21T12:23:20.197614+0000","last_became_active":"2024-09-21T12:22:50.912088+0000","last_became_peered":"2024-09-21T12:22:50.912088+0000","last_unstale":"2024-09-21T12:23:20.197614+0000","last_undegraded":"2024-09-21T12:23:20.197614+0000","last_fullsized":"2024-09-21T12:23:20.197614+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:22:18.810791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:22:18.810791+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"132'7629","reported_seq":7736,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:21.524553+0000","last_change":"2024-09-21T12:21:24.889134+0000","last_active":"2024-09-21T12:23:21.524553+0000","last_peered":"2024-09-21T12:23:21.524553+0000","last_clean":"2024-09-21T12:23:21.524553+0000","last_became_active":"2024-09-21T12:21:24.888897+0000","last_became_peered":"2024-09-21T12:21:24.888897+0000","last_unstale":"2024-09-21T12:23:21.524553+0000","last_undegraded":"2024-09-21T12:23:21.524553+0000","last_fullsized":"2024-09-21T12:23:21.524553+0000","mapping_epoch":84,"log_start":"102'3148","ondisk_log_start":"102'3148","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4481,"log_dups_size":0,"ondisk_log_size":4481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323682304,"num_objects":4939,"num_object_clones":0,"num_object_copies":14817,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7629,"num_write_kb":402176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":181,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.198618+0000","last_change":"2024-09-21T12:22:49.933621+0000","last_active":"2024-09-21T12:23:20.198618+0000","last_peered":"2024-09-21T12:23:20.198618+0000","last_clean":"2024-09-21T12:23:20.198618+0000","last_became_active":"2024-09-21T12:20:17.299325+0000","last_became_peered":"2024-09-21T12:20:17.299325+0000","last_unstale":"2024-09-21T12:23:20.198618+0000","last_undegraded":"2024-09-21T12:23:20.198618+0000","last_fullsized":"2024-09-21T12:23:20.198618+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:22:49.933382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:49.933382+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:00.818963+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":188,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.197674+0000","last_change":"2024-09-21T12:22:50.929055+0000","last_active":"2024-09-21T12:23:20.197674+0000","last_peered":"2024-09-21T12:23:20.197674+0000","last_clean":"2024-09-21T12:23:20.197674+0000","last_became_active":"2024-09-21T12:22:50.923796+0000","last_became_peered":"2024-09-21T12:22:50.923796+0000","last_unstale":"2024-09-21T12:23:20.197674+0000","last_undegraded":"2024-09-21T12:23:20.197674+0000","last_fullsized":"2024-09-21T12:23:20.197674+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:13.623395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:13.623395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:25.209353+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":358,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.248911+0000","last_change":"2024-09-21T12:21:22.834972+0000","last_active":"2024-09-21T12:23:20.248911+0000","last_peered":"2024-09-21T12:23:20.248911+0000","last_clean":"2024-09-21T12:23:20.248911+0000","last_became_active":"2024-09-21T12:20:14.287543+0000","last_became_peered":"2024-09-21T12:20:14.287543+0000","last_unstale":"2024-09-21T12:23:20.248911+0000","last_undegraded":"2024-09-21T12:23:20.248911+0000","last_fullsized":"2024-09-21T12:23:20.248911+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-21T12:21:22.834917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:21:22.834917+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":181,"num_read_kb":176,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":158,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.221468+0000","last_change":"2024-09-21T12:22:36.135464+0000","last_active":"2024-09-21T12:23:20.221468+0000","last_peered":"2024-09-21T12:23:20.221468+0000","last_clean":"2024-09-21T12:23:20.221468+0000","last_became_active":"2024-09-21T12:20:19.314217+0000","last_became_peered":"2024-09-21T12:20:19.314217+0000","last_unstale":"2024-09-21T12:23:20.221468+0000","last_undegraded":"2024-09-21T12:23:20.221468+0000","last_fullsized":"2024-09-21T12:23:20.221468+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:36.135390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:36.135390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:57.811533+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":155,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.196859+0000","last_change":"2024-09-21T12:22:57.265262+0000","last_active":"2024-09-21T12:23:20.196859+0000","last_peered":"2024-09-21T12:23:20.196859+0000","last_clean":"2024-09-21T12:23:20.196859+0000","last_became_active":"2024-09-21T12:20:21.311188+0000","last_became_peered":"2024-09-21T12:20:21.311188+0000","last_unstale":"2024-09-21T12:23:20.196859+0000","last_undegraded":"2024-09-21T12:23:20.196859+0000","last_fullsized":"2024-09-21T12:23:20.196859+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:57.265191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:57.265191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:24.950692+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"132'7731","reported_seq":7896,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:21.800543+0000","last_change":"2024-09-21T12:23:19.231036+0000","last_active":"2024-09-21T12:23:21.800543+0000","last_peered":"2024-09-21T12:23:21.800543+0000","last_clean":"2024-09-21T12:23:21.800543+0000","last_became_active":"2024-09-21T12:21:24.885986+0000","last_became_peered":"2024-09-21T12:21:24.885986+0000","last_unstale":"2024-09-21T12:23:21.800543+0000","last_undegraded":"2024-09-21T12:23:21.800543+0000","last_fullsized":"2024-09-21T12:23:21.800543+0000","mapping_epoch":84,"log_start":"103'3478","ondisk_log_start":"103'3478","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'7605","last_scrub_stamp":"2024-09-21T12:23:19.230513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:23:19.230513+0000","objects_scrubbed":5276,"log_size":4253,"log_dups_size":0,"ondisk_log_size":4253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:27.706332+0000","scrub_duration":2077,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333905920,"num_objects":5095,"num_object_clones":0,"num_object_copies":15285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7731,"num_write_kb":410432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"121'19","reported_seq":297,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.230476+0000","last_change":"2024-09-21T12:22:53.756501+0000","last_active":"2024-09-21T12:23:20.230476+0000","last_peered":"2024-09-21T12:23:20.230476+0000","last_clean":"2024-09-21T12:23:20.230476+0000","last_became_active":"2024-09-21T12:22:50.921690+0000","last_became_peered":"2024-09-21T12:22:50.921690+0000","last_unstale":"2024-09-21T12:23:20.230476+0000","last_undegraded":"2024-09-21T12:23:20.230476+0000","last_fullsized":"2024-09-21T12:23:20.230476+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'19","last_scrub_stamp":"2024-09-21T12:22:53.756438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:53.756438+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:17.096246+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":171,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.202883+0000","last_change":"2024-09-21T12:21:17.107378+0000","last_active":"2024-09-21T12:23:20.202883+0000","last_peered":"2024-09-21T12:23:20.202883+0000","last_clean":"2024-09-21T12:23:20.202883+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:23:20.202883+0000","last_undegraded":"2024-09-21T12:23:20.202883+0000","last_fullsized":"2024-09-21T12:23:20.202883+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:21:17.107309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:17.107309+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":221,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.201373+0000","last_change":"2024-09-21T12:22:16.936605+0000","last_active":"2024-09-21T12:23:20.201373+0000","last_peered":"2024-09-21T12:23:20.201373+0000","last_clean":"2024-09-21T12:23:20.201373+0000","last_became_active":"2024-09-21T12:22:15.049592+0000","last_became_peered":"2024-09-21T12:22:15.049592+0000","last_unstale":"2024-09-21T12:23:20.201373+0000","last_undegraded":"2024-09-21T12:23:20.201373+0000","last_fullsized":"2024-09-21T12:23:20.201373+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:22:16.936548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:22:16.936548+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:36.347732+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":150,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.201493+0000","last_change":"2024-09-21T12:20:19.310280+0000","last_active":"2024-09-21T12:23:20.201493+0000","last_peered":"2024-09-21T12:23:20.201493+0000","last_clean":"2024-09-21T12:23:20.201493+0000","last_became_active":"2024-09-21T12:20:19.309859+0000","last_became_peered":"2024-09-21T12:20:19.309859+0000","last_unstale":"2024-09-21T12:23:20.201493+0000","last_undegraded":"2024-09-21T12:23:20.201493+0000","last_fullsized":"2024-09-21T12:23:20.201493+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":175,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.221360+0000","last_change":"2024-09-21T12:22:50.913944+0000","last_active":"2024-09-21T12:23:20.221360+0000","last_peered":"2024-09-21T12:23:20.221360+0000","last_clean":"2024-09-21T12:23:20.221360+0000","last_became_active":"2024-09-21T12:22:50.913493+0000","last_became_peered":"2024-09-21T12:22:50.913493+0000","last_unstale":"2024-09-21T12:23:20.221360+0000","last_undegraded":"2024-09-21T12:23:20.221360+0000","last_fullsized":"2024-09-21T12:23:20.221360+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:41.210247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:41.210247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:55.372874+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"132'7939","reported_seq":8110,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:21.570647+0000","last_change":"2024-09-21T12:22:40.358709+0000","last_active":"2024-09-21T12:23:21.570647+0000","last_peered":"2024-09-21T12:23:21.570647+0000","last_clean":"2024-09-21T12:23:21.570647+0000","last_became_active":"2024-09-21T12:21:24.887845+0000","last_became_peered":"2024-09-21T12:21:24.887845+0000","last_unstale":"2024-09-21T12:23:21.570647+0000","last_undegraded":"2024-09-21T12:23:21.570647+0000","last_fullsized":"2024-09-21T12:23:21.570647+0000","mapping_epoch":84,"log_start":"102'3302","ondisk_log_start":"102'3302","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'5424","last_scrub_stamp":"2024-09-21T12:22:40.358604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:22:40.358604+0000","objects_scrubbed":5391,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:53.778779+0000","scrub_duration":1743,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338165784,"num_objects":5161,"num_object_clones":0,"num_object_copies":15483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5161,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7939,"num_write_kb":419137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":145,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.230328+0000","last_change":"2024-09-21T12:22:50.966517+0000","last_active":"2024-09-21T12:23:20.230328+0000","last_peered":"2024-09-21T12:23:20.230328+0000","last_clean":"2024-09-21T12:23:20.230328+0000","last_became_active":"2024-09-21T12:22:50.914810+0000","last_became_peered":"2024-09-21T12:22:50.914810+0000","last_unstale":"2024-09-21T12:23:20.230328+0000","last_undegraded":"2024-09-21T12:23:20.230328+0000","last_fullsized":"2024-09-21T12:23:20.230328+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:21:19.327640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:21:19.327640+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:05.800457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":90,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.199560+0000","last_change":"2024-09-21T12:22:50.908380+0000","last_active":"2024-09-21T12:23:20.199560+0000","last_peered":"2024-09-21T12:23:20.199560+0000","last_clean":"2024-09-21T12:23:20.199560+0000","last_became_active":"2024-09-21T12:22:50.908060+0000","last_became_peered":"2024-09-21T12:22:50.908060+0000","last_unstale":"2024-09-21T12:23:20.199560+0000","last_undegraded":"2024-09-21T12:23:20.199560+0000","last_fullsized":"2024-09-21T12:23:20.199560+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:05.892153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:05.892153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:32.732340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":187,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.221273+0000","last_change":"2024-09-21T12:23:19.372762+0000","last_active":"2024-09-21T12:23:20.221273+0000","last_peered":"2024-09-21T12:23:20.221273+0000","last_clean":"2024-09-21T12:23:20.221273+0000","last_became_active":"2024-09-21T12:22:50.915081+0000","last_became_peered":"2024-09-21T12:22:50.915081+0000","last_unstale":"2024-09-21T12:23:20.221273+0000","last_undegraded":"2024-09-21T12:23:20.221273+0000","last_fullsized":"2024-09-21T12:23:20.221273+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:19.372719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:19.372719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:32.884588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":145,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.201948+0000","last_change":"2024-09-21T12:20:21.593145+0000","last_active":"2024-09-21T12:23:20.201948+0000","last_peered":"2024-09-21T12:23:20.201948+0000","last_clean":"2024-09-21T12:23:20.201948+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:23:20.201948+0000","last_undegraded":"2024-09-21T12:23:20.201948+0000","last_fullsized":"2024-09-21T12:23:20.201948+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:31.963984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"76'79","reported_seq":65,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.220197+0000","last_change":"2024-09-21T12:22:17.114985+0000","last_active":"2024-09-21T12:23:20.220197+0000","last_peered":"2024-09-21T12:23:20.220197+0000","last_clean":"2024-09-21T12:23:20.220197+0000","last_became_active":"2024-09-21T12:22:17.087245+0000","last_became_peered":"2024-09-21T12:22:17.087245+0000","last_unstale":"2024-09-21T12:23:20.220197+0000","last_undegraded":"2024-09-21T12:23:20.220197+0000","last_fullsized":"2024-09-21T12:23:20.220197+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T12:20:24.850157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:20:24.850157+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:23:22.031869+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":316358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"132'7876","reported_seq":7983,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:21.690386+0000","last_change":"2024-09-21T12:21:24.888960+0000","last_active":"2024-09-21T12:23:21.690386+0000","last_peered":"2024-09-21T12:23:21.690386+0000","last_clean":"2024-09-21T12:23:21.690386+0000","last_became_active":"2024-09-21T12:21:24.888670+0000","last_became_peered":"2024-09-21T12:21:24.888670+0000","last_unstale":"2024-09-21T12:23:21.690386+0000","last_undegraded":"2024-09-21T12:23:21.690386+0000","last_fullsized":"2024-09-21T12:23:21.690386+0000","mapping_epoch":84,"log_start":"97'2489","ondisk_log_start":"97'2489","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":5387,"log_dups_size":0,"ondisk_log_size":5387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335675392,"num_objects":5122,"num_object_clones":0,"num_object_copies":15366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7876,"num_write_kb":415936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":174,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.200589+0000","last_change":"2024-09-21T12:22:50.989705+0000","last_active":"2024-09-21T12:23:20.200589+0000","last_peered":"2024-09-21T12:23:20.200589+0000","last_clean":"2024-09-21T12:23:20.200589+0000","last_became_active":"2024-09-21T12:22:50.915202+0000","last_became_peered":"2024-09-21T12:22:50.915202+0000","last_unstale":"2024-09-21T12:23:20.200589+0000","last_undegraded":"2024-09-21T12:23:20.200589+0000","last_fullsized":"2024-09-21T12:23:20.200589+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-21T12:22:01.738653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:01.738653+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":187,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.229545+0000","last_change":"2024-09-21T12:22:50.907175+0000","last_active":"2024-09-21T12:23:20.229545+0000","last_peered":"2024-09-21T12:23:20.229545+0000","last_clean":"2024-09-21T12:23:20.229545+0000","last_became_active":"2024-09-21T12:22:50.906886+0000","last_became_peered":"2024-09-21T12:22:50.906886+0000","last_unstale":"2024-09-21T12:23:20.229545+0000","last_undegraded":"2024-09-21T12:23:20.229545+0000","last_fullsized":"2024-09-21T12:23:20.229545+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:16.670500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:16.670500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:25.660662+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":180,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.196944+0000","last_change":"2024-09-21T12:23:17.434377+0000","last_active":"2024-09-21T12:23:20.196944+0000","last_peered":"2024-09-21T12:23:20.196944+0000","last_clean":"2024-09-21T12:23:20.196944+0000","last_became_active":"2024-09-21T12:22:50.927438+0000","last_became_peered":"2024-09-21T12:22:50.927438+0000","last_unstale":"2024-09-21T12:23:20.196944+0000","last_undegraded":"2024-09-21T12:23:20.196944+0000","last_fullsized":"2024-09-21T12:23:20.196944+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:17.434261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:17.434261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:43.736130+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":168,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.200648+0000","last_change":"2024-09-21T12:22:50.914246+0000","last_active":"2024-09-21T12:23:20.200648+0000","last_peered":"2024-09-21T12:23:20.200648+0000","last_clean":"2024-09-21T12:23:20.200648+0000","last_became_active":"2024-09-21T12:22:50.913898+0000","last_became_peered":"2024-09-21T12:22:50.913898+0000","last_unstale":"2024-09-21T12:23:20.200648+0000","last_undegraded":"2024-09-21T12:23:20.200648+0000","last_fullsized":"2024-09-21T12:23:20.200648+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"76'79","reported_seq":219,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.201045+0000","last_change":"2024-09-21T12:22:17.073301+0000","last_active":"2024-09-21T12:23:20.201045+0000","last_peered":"2024-09-21T12:23:20.201045+0000","last_clean":"2024-09-21T12:23:20.201045+0000","last_became_active":"2024-09-21T12:22:17.072872+0000","last_became_peered":"2024-09-21T12:22:17.072872+0000","last_unstale":"2024-09-21T12:23:20.201045+0000","last_undegraded":"2024-09-21T12:23:20.201045+0000","last_fullsized":"2024-09-21T12:23:20.201045+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T12:20:24.850157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:20:24.850157+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:23:20.982847+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":21,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.f","version":"132'8012","reported_seq":8119,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:21.811896+0000","last_change":"2024-09-21T12:21:24.887418+0000","last_active":"2024-09-21T12:23:21.811896+0000","last_peered":"2024-09-21T12:23:21.811896+0000","last_clean":"2024-09-21T12:23:21.811896+0000","last_became_active":"2024-09-21T12:21:24.887000+0000","last_became_peered":"2024-09-21T12:21:24.887000+0000","last_unstale":"2024-09-21T12:23:21.811896+0000","last_undegraded":"2024-09-21T12:23:21.811896+0000","last_fullsized":"2024-09-21T12:23:21.811896+0000","mapping_epoch":84,"log_start":"104'3751","ondisk_log_start":"104'3751","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4261,"log_dups_size":0,"ondisk_log_size":4261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:05.142226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337641472,"num_objects":5152,"num_object_clones":0,"num_object_copies":15456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8012,"num_write_kb":421248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":141,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.202234+0000","last_change":"2024-09-21T12:22:50.962856+0000","last_active":"2024-09-21T12:23:20.202234+0000","last_peered":"2024-09-21T12:23:20.202234+0000","last_clean":"2024-09-21T12:23:20.202234+0000","last_became_active":"2024-09-21T12:22:50.909221+0000","last_became_peered":"2024-09-21T12:22:50.909221+0000","last_unstale":"2024-09-21T12:23:20.202234+0000","last_undegraded":"2024-09-21T12:23:20.202234+0000","last_fullsized":"2024-09-21T12:23:20.202234+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":186,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.202804+0000","last_change":"2024-09-21T12:22:50.928240+0000","last_active":"2024-09-21T12:23:20.202804+0000","last_peered":"2024-09-21T12:23:20.202804+0000","last_clean":"2024-09-21T12:23:20.202804+0000","last_became_active":"2024-09-21T12:22:50.927946+0000","last_became_peered":"2024-09-21T12:22:50.927946+0000","last_unstale":"2024-09-21T12:23:20.202804+0000","last_undegraded":"2024-09-21T12:23:20.202804+0000","last_fullsized":"2024-09-21T12:23:20.202804+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:08.999015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:08.999015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":197,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.200535+0000","last_change":"2024-09-21T12:20:19.308246+0000","last_active":"2024-09-21T12:23:20.200535+0000","last_peered":"2024-09-21T12:23:20.200535+0000","last_clean":"2024-09-21T12:23:20.200535+0000","last_became_active":"2024-09-21T12:20:19.307924+0000","last_became_peered":"2024-09-21T12:20:19.307924+0000","last_unstale":"2024-09-21T12:23:20.200535+0000","last_undegraded":"2024-09-21T12:23:20.200535+0000","last_fullsized":"2024-09-21T12:23:20.200535+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":90,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.197234+0000","last_change":"2024-09-21T12:22:50.905752+0000","last_active":"2024-09-21T12:23:20.197234+0000","last_peered":"2024-09-21T12:23:20.197234+0000","last_clean":"2024-09-21T12:23:20.197234+0000","last_became_active":"2024-09-21T12:22:50.905470+0000","last_became_peered":"2024-09-21T12:22:50.905470+0000","last_unstale":"2024-09-21T12:23:20.197234+0000","last_undegraded":"2024-09-21T12:23:20.197234+0000","last_fullsized":"2024-09-21T12:23:20.197234+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:14.809555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.809555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:35.716649+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"132'8015","reported_seq":15389,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:21.674754+0000","last_change":"2024-09-21T12:23:20.213767+0000","last_active":"2024-09-21T12:23:21.674754+0000","last_peered":"2024-09-21T12:23:21.674754+0000","last_clean":"2024-09-21T12:23:21.674754+0000","last_became_active":"2024-09-21T12:23:20.203417+0000","last_became_peered":"2024-09-21T12:23:20.203417+0000","last_unstale":"2024-09-21T12:23:21.674754+0000","last_undegraded":"2024-09-21T12:23:21.674754+0000","last_fullsized":"2024-09-21T12:23:21.674754+0000","mapping_epoch":131,"log_start":"97'2601","ondisk_log_start":"97'2601","created":84,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":5414,"log_dups_size":0,"ondisk_log_size":5414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342818816,"num_objects":5231,"num_object_clones":0,"num_object_copies":15693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8015,"num_write_kb":423872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3621,"num_bytes_recovered":237305856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":238,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.197517+0000","last_change":"2024-09-21T12:23:15.165398+0000","last_active":"2024-09-21T12:23:20.197517+0000","last_peered":"2024-09-21T12:23:20.197517+0000","last_clean":"2024-09-21T12:23:20.197517+0000","last_became_active":"2024-09-21T12:22:50.922343+0000","last_became_peered":"2024-09-21T12:22:50.922343+0000","last_unstale":"2024-09-21T12:23:20.197517+0000","last_undegraded":"2024-09-21T12:23:20.197517+0000","last_fullsized":"2024-09-21T12:23:20.197517+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:23:15.158428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:15.158428+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:27.015422+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":166,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.221022+0000","last_change":"2024-09-21T12:22:48.173073+0000","last_active":"2024-09-21T12:23:20.221022+0000","last_peered":"2024-09-21T12:23:20.221022+0000","last_clean":"2024-09-21T12:23:20.221022+0000","last_became_active":"2024-09-21T12:20:15.302262+0000","last_became_peered":"2024-09-21T12:20:15.302262+0000","last_unstale":"2024-09-21T12:23:20.221022+0000","last_undegraded":"2024-09-21T12:23:20.221022+0000","last_fullsized":"2024-09-21T12:23:20.221022+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:48.173022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:48.173022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:48.961679+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":180,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.199855+0000","last_change":"2024-09-21T12:22:50.926450+0000","last_active":"2024-09-21T12:23:20.199855+0000","last_peered":"2024-09-21T12:23:20.199855+0000","last_clean":"2024-09-21T12:23:20.199855+0000","last_became_active":"2024-09-21T12:22:50.925974+0000","last_became_peered":"2024-09-21T12:22:50.925974+0000","last_unstale":"2024-09-21T12:23:20.199855+0000","last_undegraded":"2024-09-21T12:23:20.199855+0000","last_fullsized":"2024-09-21T12:23:20.199855+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:11.892850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:11.892850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:27.007736+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.197607+0000","last_change":"2024-09-21T12:20:21.311799+0000","last_active":"2024-09-21T12:23:20.197607+0000","last_peered":"2024-09-21T12:23:20.197607+0000","last_clean":"2024-09-21T12:23:20.197607+0000","last_became_active":"2024-09-21T12:20:21.311493+0000","last_became_peered":"2024-09-21T12:20:21.311493+0000","last_unstale":"2024-09-21T12:23:20.197607+0000","last_undegraded":"2024-09-21T12:23:20.197607+0000","last_fullsized":"2024-09-21T12:23:20.197607+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:23.727080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"132'8034","reported_seq":17494,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:21.803838+0000","last_change":"2024-09-21T12:22:50.949717+0000","last_active":"2024-09-21T12:23:21.803838+0000","last_peered":"2024-09-21T12:23:21.803838+0000","last_clean":"2024-09-21T12:23:21.803838+0000","last_became_active":"2024-09-21T12:22:50.924585+0000","last_became_peered":"2024-09-21T12:22:50.924585+0000","last_unstale":"2024-09-21T12:23:21.803838+0000","last_undegraded":"2024-09-21T12:23:21.803838+0000","last_fullsized":"2024-09-21T12:23:21.803838+0000","mapping_epoch":120,"log_start":"104'3780","ondisk_log_start":"104'3780","created":84,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4254,"log_dups_size":0,"ondisk_log_size":4254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:06.115911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339214336,"num_objects":5176,"num_object_clones":0,"num_object_copies":15528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8034,"num_write_kb":422720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4664,"num_bytes_recovered":305659904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":193,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.229304+0000","last_change":"2024-09-21T12:22:42.707274+0000","last_active":"2024-09-21T12:23:20.229304+0000","last_peered":"2024-09-21T12:23:20.229304+0000","last_clean":"2024-09-21T12:23:20.229304+0000","last_became_active":"2024-09-21T12:20:17.692867+0000","last_became_peered":"2024-09-21T12:20:17.692867+0000","last_unstale":"2024-09-21T12:23:20.229304+0000","last_undegraded":"2024-09-21T12:23:20.229304+0000","last_fullsized":"2024-09-21T12:23:20.229304+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-21T12:22:42.707175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:42.707175+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:46.335757+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":158,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.201748+0000","last_change":"2024-09-21T12:20:15.300854+0000","last_active":"2024-09-21T12:23:20.201748+0000","last_peered":"2024-09-21T12:23:20.201748+0000","last_clean":"2024-09-21T12:23:20.201748+0000","last_became_active":"2024-09-21T12:20:15.299333+0000","last_became_peered":"2024-09-21T12:20:15.299333+0000","last_unstale":"2024-09-21T12:23:20.201748+0000","last_undegraded":"2024-09-21T12:23:20.201748+0000","last_fullsized":"2024-09-21T12:23:20.201748+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":151,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.200351+0000","last_change":"2024-09-21T12:20:19.307316+0000","last_active":"2024-09-21T12:23:20.200351+0000","last_peered":"2024-09-21T12:23:20.200351+0000","last_clean":"2024-09-21T12:23:20.200351+0000","last_became_active":"2024-09-21T12:20:19.306952+0000","last_became_peered":"2024-09-21T12:20:19.306952+0000","last_unstale":"2024-09-21T12:23:20.200351+0000","last_undegraded":"2024-09-21T12:23:20.200351+0000","last_fullsized":"2024-09-21T12:23:20.200351+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":146,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.201260+0000","last_change":"2024-09-21T12:20:21.314891+0000","last_active":"2024-09-21T12:23:20.201260+0000","last_peered":"2024-09-21T12:23:20.201260+0000","last_clean":"2024-09-21T12:23:20.201260+0000","last_became_active":"2024-09-21T12:20:21.314545+0000","last_became_peered":"2024-09-21T12:20:21.314545+0000","last_unstale":"2024-09-21T12:23:20.201260+0000","last_undegraded":"2024-09-21T12:23:20.201260+0000","last_fullsized":"2024-09-21T12:23:20.201260+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"132'7892","reported_seq":7999,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:21.796983+0000","last_change":"2024-09-21T12:21:24.888714+0000","last_active":"2024-09-21T12:23:21.796983+0000","last_peered":"2024-09-21T12:23:21.796983+0000","last_clean":"2024-09-21T12:23:21.796983+0000","last_became_active":"2024-09-21T12:21:24.888445+0000","last_became_peered":"2024-09-21T12:21:24.888445+0000","last_unstale":"2024-09-21T12:23:21.796983+0000","last_undegraded":"2024-09-21T12:23:21.796983+0000","last_fullsized":"2024-09-21T12:23:21.796983+0000","mapping_epoch":84,"log_start":"107'3960","ondisk_log_start":"107'3960","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":3932,"log_dups_size":0,"ondisk_log_size":3932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339083264,"num_objects":5174,"num_object_clones":0,"num_object_copies":15522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7892,"num_write_kb":418112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":157,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.196495+0000","last_change":"2024-09-21T12:22:50.947094+0000","last_active":"2024-09-21T12:23:20.196495+0000","last_peered":"2024-09-21T12:23:20.196495+0000","last_clean":"2024-09-21T12:23:20.196495+0000","last_became_active":"2024-09-21T12:22:50.910973+0000","last_became_peered":"2024-09-21T12:22:50.910973+0000","last_unstale":"2024-09-21T12:23:20.196495+0000","last_undegraded":"2024-09-21T12:23:20.196495+0000","last_fullsized":"2024-09-21T12:23:20.196495+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:22:09.782437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:09.782437+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:24.496058+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":187,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.201890+0000","last_change":"2024-09-21T12:22:50.928139+0000","last_active":"2024-09-21T12:23:20.201890+0000","last_peered":"2024-09-21T12:23:20.201890+0000","last_clean":"2024-09-21T12:23:20.201890+0000","last_became_active":"2024-09-21T12:22:50.927859+0000","last_became_peered":"2024-09-21T12:22:50.927859+0000","last_unstale":"2024-09-21T12:23:20.201890+0000","last_undegraded":"2024-09-21T12:23:20.201890+0000","last_fullsized":"2024-09-21T12:23:20.201890+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:20.865749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.865749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":159,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.199642+0000","last_change":"2024-09-21T12:21:23.393320+0000","last_active":"2024-09-21T12:23:20.199642+0000","last_peered":"2024-09-21T12:23:20.199642+0000","last_clean":"2024-09-21T12:23:20.199642+0000","last_became_active":"2024-09-21T12:20:19.312572+0000","last_became_peered":"2024-09-21T12:20:19.312572+0000","last_unstale":"2024-09-21T12:23:20.199642+0000","last_undegraded":"2024-09-21T12:23:20.199642+0000","last_fullsized":"2024-09-21T12:23:20.199642+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.393270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.393270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":153,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.202468+0000","last_change":"2024-09-21T12:22:36.006177+0000","last_active":"2024-09-21T12:23:20.202468+0000","last_peered":"2024-09-21T12:23:20.202468+0000","last_clean":"2024-09-21T12:23:20.202468+0000","last_became_active":"2024-09-21T12:20:21.317223+0000","last_became_peered":"2024-09-21T12:20:21.317223+0000","last_unstale":"2024-09-21T12:23:20.202468+0000","last_undegraded":"2024-09-21T12:23:20.202468+0000","last_fullsized":"2024-09-21T12:23:20.202468+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:36.006109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:36.006109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:38.949250+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"132'7809","reported_seq":7915,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:21.614302+0000","last_change":"2024-09-21T12:21:24.888120+0000","last_active":"2024-09-21T12:23:21.614302+0000","last_peered":"2024-09-21T12:23:21.614302+0000","last_clean":"2024-09-21T12:23:21.614302+0000","last_became_active":"2024-09-21T12:21:24.887488+0000","last_became_peered":"2024-09-21T12:21:24.887488+0000","last_unstale":"2024-09-21T12:23:21.614302+0000","last_undegraded":"2024-09-21T12:23:21.614302+0000","last_fullsized":"2024-09-21T12:23:21.614302+0000","mapping_epoch":84,"log_start":"106'3798","ondisk_log_start":"106'3798","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4011,"log_dups_size":0,"ondisk_log_size":4011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332201984,"num_objects":5069,"num_object_clones":0,"num_object_copies":15207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7809,"num_write_kb":412096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":195,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.220390+0000","last_change":"2024-09-21T12:23:17.286227+0000","last_active":"2024-09-21T12:23:20.220390+0000","last_peered":"2024-09-21T12:23:20.220390+0000","last_clean":"2024-09-21T12:23:20.220390+0000","last_became_active":"2024-09-21T12:20:17.692870+0000","last_became_peered":"2024-09-21T12:20:17.692870+0000","last_unstale":"2024-09-21T12:23:20.220390+0000","last_undegraded":"2024-09-21T12:23:20.220390+0000","last_fullsized":"2024-09-21T12:23:20.220390+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:23:17.286166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:17.286166+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:45.336901+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":217,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.201859+0000","last_change":"2024-09-21T12:22:51.052311+0000","last_active":"2024-09-21T12:23:20.201859+0000","last_peered":"2024-09-21T12:23:20.201859+0000","last_clean":"2024-09-21T12:23:20.201859+0000","last_became_active":"2024-09-21T12:22:50.927141+0000","last_became_peered":"2024-09-21T12:22:50.927141+0000","last_unstale":"2024-09-21T12:23:20.201859+0000","last_undegraded":"2024-09-21T12:23:20.201859+0000","last_fullsized":"2024-09-21T12:23:20.201859+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-21T12:22:14.920932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.920932+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"121'11","reported_seq":229,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.200981+0000","last_change":"2024-09-21T12:22:50.915762+0000","last_active":"2024-09-21T12:23:20.200981+0000","last_peered":"2024-09-21T12:23:20.200981+0000","last_clean":"2024-09-21T12:23:20.200981+0000","last_became_active":"2024-09-21T12:22:50.908371+0000","last_became_peered":"2024-09-21T12:22:50.908371+0000","last_unstale":"2024-09-21T12:23:20.200981+0000","last_undegraded":"2024-09-21T12:23:20.200981+0000","last_fullsized":"2024-09-21T12:23:20.200981+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":154,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.221260+0000","last_change":"2024-09-21T12:22:45.223570+0000","last_active":"2024-09-21T12:23:20.221260+0000","last_peered":"2024-09-21T12:23:20.221260+0000","last_clean":"2024-09-21T12:23:20.221260+0000","last_became_active":"2024-09-21T12:20:21.592576+0000","last_became_peered":"2024-09-21T12:20:21.592576+0000","last_unstale":"2024-09-21T12:23:20.221260+0000","last_undegraded":"2024-09-21T12:23:20.221260+0000","last_fullsized":"2024-09-21T12:23:20.221260+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:45.223396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:45.223396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:05.217553+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":174,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.229087+0000","last_change":"2024-09-21T12:22:45.726349+0000","last_active":"2024-09-21T12:23:20.229087+0000","last_peered":"2024-09-21T12:23:20.229087+0000","last_clean":"2024-09-21T12:23:20.229087+0000","last_became_active":"2024-09-21T12:20:15.303480+0000","last_became_peered":"2024-09-21T12:20:15.303480+0000","last_unstale":"2024-09-21T12:23:20.229087+0000","last_undegraded":"2024-09-21T12:23:20.229087+0000","last_fullsized":"2024-09-21T12:23:20.229087+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:45.726268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:45.726268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:10.634112+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":170,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.203103+0000","last_change":"2024-09-21T12:22:36.999995+0000","last_active":"2024-09-21T12:23:20.203103+0000","last_peered":"2024-09-21T12:23:20.203103+0000","last_clean":"2024-09-21T12:23:20.203103+0000","last_became_active":"2024-09-21T12:20:17.694628+0000","last_became_peered":"2024-09-21T12:20:17.694628+0000","last_unstale":"2024-09-21T12:23:20.203103+0000","last_undegraded":"2024-09-21T12:23:20.203103+0000","last_fullsized":"2024-09-21T12:23:20.203103+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-21T12:22:36.999705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:36.999705+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:37.754032+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":166,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.221172+0000","last_change":"2024-09-21T12:23:15.291669+0000","last_active":"2024-09-21T12:23:20.221172+0000","last_peered":"2024-09-21T12:23:20.221172+0000","last_clean":"2024-09-21T12:23:20.221172+0000","last_became_active":"2024-09-21T12:20:19.313882+0000","last_became_peered":"2024-09-21T12:20:19.313882+0000","last_unstale":"2024-09-21T12:23:20.221172+0000","last_undegraded":"2024-09-21T12:23:20.221172+0000","last_fullsized":"2024-09-21T12:23:20.221172+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:15.291602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:15.291602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:36.660286+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":167,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.201438+0000","last_change":"2024-09-21T12:22:50.904801+0000","last_active":"2024-09-21T12:23:20.201438+0000","last_peered":"2024-09-21T12:23:20.201438+0000","last_clean":"2024-09-21T12:23:20.201438+0000","last_became_active":"2024-09-21T12:22:50.904404+0000","last_became_peered":"2024-09-21T12:22:50.904404+0000","last_unstale":"2024-09-21T12:23:20.201438+0000","last_undegraded":"2024-09-21T12:23:20.201438+0000","last_fullsized":"2024-09-21T12:23:20.201438+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":179,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.198684+0000","last_change":"2024-09-21T12:21:24.093710+0000","last_active":"2024-09-21T12:23:20.198684+0000","last_peered":"2024-09-21T12:23:20.198684+0000","last_clean":"2024-09-21T12:23:20.198684+0000","last_became_active":"2024-09-21T12:20:17.695898+0000","last_became_peered":"2024-09-21T12:20:17.695898+0000","last_unstale":"2024-09-21T12:23:20.198684+0000","last_undegraded":"2024-09-21T12:23:20.198684+0000","last_fullsized":"2024-09-21T12:23:20.198684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:21:24.093656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:21:24.093656+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:07.151578+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":187,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.201314+0000","last_change":"2024-09-21T12:22:50.928059+0000","last_active":"2024-09-21T12:23:20.201314+0000","last_peered":"2024-09-21T12:23:20.201314+0000","last_clean":"2024-09-21T12:23:20.201314+0000","last_became_active":"2024-09-21T12:22:50.927649+0000","last_became_peered":"2024-09-21T12:22:50.927649+0000","last_unstale":"2024-09-21T12:23:20.201314+0000","last_undegraded":"2024-09-21T12:23:20.201314+0000","last_fullsized":"2024-09-21T12:23:20.201314+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:07.084008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:07.084008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":149,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.202993+0000","last_change":"2024-09-21T12:20:19.312248+0000","last_active":"2024-09-21T12:23:20.202993+0000","last_peered":"2024-09-21T12:23:20.202993+0000","last_clean":"2024-09-21T12:23:20.202993+0000","last_became_active":"2024-09-21T12:20:19.311947+0000","last_became_peered":"2024-09-21T12:20:19.311947+0000","last_unstale":"2024-09-21T12:23:20.202993+0000","last_undegraded":"2024-09-21T12:23:20.202993+0000","last_fullsized":"2024-09-21T12:23:20.202993+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":176,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.200301+0000","last_change":"2024-09-21T12:22:50.925237+0000","last_active":"2024-09-21T12:23:20.200301+0000","last_peered":"2024-09-21T12:23:20.200301+0000","last_clean":"2024-09-21T12:23:20.200301+0000","last_became_active":"2024-09-21T12:22:50.924931+0000","last_became_peered":"2024-09-21T12:22:50.924931+0000","last_unstale":"2024-09-21T12:23:20.200301+0000","last_undegraded":"2024-09-21T12:23:20.200301+0000","last_fullsized":"2024-09-21T12:23:20.200301+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:18.797578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:18.797578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:25.968048+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":178,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.221054+0000","last_change":"2024-09-21T12:22:39.201717+0000","last_active":"2024-09-21T12:23:20.221054+0000","last_peered":"2024-09-21T12:23:20.221054+0000","last_clean":"2024-09-21T12:23:20.221054+0000","last_became_active":"2024-09-21T12:20:17.694568+0000","last_became_peered":"2024-09-21T12:20:17.694568+0000","last_unstale":"2024-09-21T12:23:20.221054+0000","last_undegraded":"2024-09-21T12:23:20.221054+0000","last_fullsized":"2024-09-21T12:23:20.221054+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:22:39.201667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:39.201667+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:07.627492+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":167,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.198003+0000","last_change":"2024-09-21T12:22:53.989409+0000","last_active":"2024-09-21T12:23:20.198003+0000","last_peered":"2024-09-21T12:23:20.198003+0000","last_clean":"2024-09-21T12:23:20.198003+0000","last_became_active":"2024-09-21T12:20:15.297519+0000","last_became_peered":"2024-09-21T12:20:15.297519+0000","last_unstale":"2024-09-21T12:23:20.198003+0000","last_undegraded":"2024-09-21T12:23:20.198003+0000","last_fullsized":"2024-09-21T12:23:20.198003+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:53.989318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:53.989318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:15.527166+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":195,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.202182+0000","last_change":"2024-09-21T12:20:19.305689+0000","last_active":"2024-09-21T12:23:20.202182+0000","last_peered":"2024-09-21T12:23:20.202182+0000","last_clean":"2024-09-21T12:23:20.202182+0000","last_became_active":"2024-09-21T12:20:19.305364+0000","last_became_peered":"2024-09-21T12:20:19.305364+0000","last_unstale":"2024-09-21T12:23:20.202182+0000","last_undegraded":"2024-09-21T12:23:20.202182+0000","last_fullsized":"2024-09-21T12:23:20.202182+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":90,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:21.250547+0000","last_change":"2024-09-21T12:23:21.250547+0000","last_active":"2024-09-21T12:23:21.250547+0000","last_peered":"2024-09-21T12:23:21.250547+0000","last_clean":"2024-09-21T12:23:21.250547+0000","last_became_active":"2024-09-21T12:22:50.924770+0000","last_became_peered":"2024-09-21T12:22:50.924770+0000","last_unstale":"2024-09-21T12:23:21.250547+0000","last_undegraded":"2024-09-21T12:23:21.250547+0000","last_fullsized":"2024-09-21T12:23:21.250547+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:21.250464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:21.250464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:40.125523+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":176,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.202447+0000","last_change":"2024-09-21T12:22:40.026248+0000","last_active":"2024-09-21T12:23:20.202447+0000","last_peered":"2024-09-21T12:23:20.202447+0000","last_clean":"2024-09-21T12:23:20.202447+0000","last_became_active":"2024-09-21T12:20:17.693209+0000","last_became_peered":"2024-09-21T12:20:17.693209+0000","last_unstale":"2024-09-21T12:23:20.202447+0000","last_undegraded":"2024-09-21T12:23:20.202447+0000","last_fullsized":"2024-09-21T12:23:20.202447+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:22:40.026037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:40.026037+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:52.898127+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":187,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.201266+0000","last_change":"2024-09-21T12:22:50.908560+0000","last_active":"2024-09-21T12:23:20.201266+0000","last_peered":"2024-09-21T12:23:20.201266+0000","last_clean":"2024-09-21T12:23:20.201266+0000","last_became_active":"2024-09-21T12:22:50.908238+0000","last_became_peered":"2024-09-21T12:22:50.908238+0000","last_unstale":"2024-09-21T12:23:20.201266+0000","last_undegraded":"2024-09-21T12:23:20.201266+0000","last_fullsized":"2024-09-21T12:23:20.201266+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:15.938093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:15.938093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":204,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.220796+0000","last_change":"2024-09-21T12:22:46.216242+0000","last_active":"2024-09-21T12:23:20.220796+0000","last_peered":"2024-09-21T12:23:20.220796+0000","last_clean":"2024-09-21T12:23:20.220796+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:23:20.220796+0000","last_undegraded":"2024-09-21T12:23:20.220796+0000","last_fullsized":"2024-09-21T12:23:20.220796+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:22:46.216153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:46.216153+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:08.790143+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.196692+0000","last_change":"2024-09-21T12:20:21.309936+0000","last_active":"2024-09-21T12:23:20.196692+0000","last_peered":"2024-09-21T12:23:20.196692+0000","last_clean":"2024-09-21T12:23:20.196692+0000","last_became_active":"2024-09-21T12:20:21.309385+0000","last_became_peered":"2024-09-21T12:20:21.309385+0000","last_unstale":"2024-09-21T12:23:20.196692+0000","last_undegraded":"2024-09-21T12:23:20.196692+0000","last_fullsized":"2024-09-21T12:23:20.196692+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:02.136673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":145,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.229827+0000","last_change":"2024-09-21T12:22:50.922549+0000","last_active":"2024-09-21T12:23:20.229827+0000","last_peered":"2024-09-21T12:23:20.229827+0000","last_clean":"2024-09-21T12:23:20.229827+0000","last_became_active":"2024-09-21T12:22:50.906473+0000","last_became_peered":"2024-09-21T12:22:50.906473+0000","last_unstale":"2024-09-21T12:23:20.229827+0000","last_undegraded":"2024-09-21T12:23:20.229827+0000","last_fullsized":"2024-09-21T12:23:20.229827+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:58.841100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":82,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.201625+0000","last_change":"2024-09-21T12:22:50.904173+0000","last_active":"2024-09-21T12:23:20.201625+0000","last_peered":"2024-09-21T12:23:20.201625+0000","last_clean":"2024-09-21T12:23:20.201625+0000","last_became_active":"2024-09-21T12:22:50.903705+0000","last_became_peered":"2024-09-21T12:22:50.903705+0000","last_unstale":"2024-09-21T12:23:20.201625+0000","last_undegraded":"2024-09-21T12:23:20.201625+0000","last_fullsized":"2024-09-21T12:23:20.201625+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":151,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.197489+0000","last_change":"2024-09-21T12:20:19.315714+0000","last_active":"2024-09-21T12:23:20.197489+0000","last_peered":"2024-09-21T12:23:20.197489+0000","last_clean":"2024-09-21T12:23:20.197489+0000","last_became_active":"2024-09-21T12:20:19.315418+0000","last_became_peered":"2024-09-21T12:20:19.315418+0000","last_unstale":"2024-09-21T12:23:20.197489+0000","last_undegraded":"2024-09-21T12:23:20.197489+0000","last_fullsized":"2024-09-21T12:23:20.197489+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":153,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.201706+0000","last_change":"2024-09-21T12:22:50.143669+0000","last_active":"2024-09-21T12:23:20.201706+0000","last_peered":"2024-09-21T12:23:20.201706+0000","last_clean":"2024-09-21T12:23:20.201706+0000","last_became_active":"2024-09-21T12:20:21.310053+0000","last_became_peered":"2024-09-21T12:20:21.310053+0000","last_unstale":"2024-09-21T12:23:20.201706+0000","last_undegraded":"2024-09-21T12:23:20.201706+0000","last_fullsized":"2024-09-21T12:23:20.201706+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:50.143628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:50.143628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:00.552274+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":171,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.197607+0000","last_change":"2024-09-21T12:20:17.299367+0000","last_active":"2024-09-21T12:23:20.197607+0000","last_peered":"2024-09-21T12:23:20.197607+0000","last_clean":"2024-09-21T12:23:20.197607+0000","last_became_active":"2024-09-21T12:20:17.299015+0000","last_became_peered":"2024-09-21T12:20:17.299015+0000","last_unstale":"2024-09-21T12:23:20.197607+0000","last_undegraded":"2024-09-21T12:23:20.197607+0000","last_fullsized":"2024-09-21T12:23:20.197607+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":187,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.201646+0000","last_change":"2024-09-21T12:22:50.922328+0000","last_active":"2024-09-21T12:23:20.201646+0000","last_peered":"2024-09-21T12:23:20.201646+0000","last_clean":"2024-09-21T12:23:20.201646+0000","last_became_active":"2024-09-21T12:22:50.922019+0000","last_became_peered":"2024-09-21T12:22:50.922019+0000","last_unstale":"2024-09-21T12:23:20.201646+0000","last_undegraded":"2024-09-21T12:23:20.201646+0000","last_fullsized":"2024-09-21T12:23:20.201646+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:05.067280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:05.067280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:20.660344+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":157,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.202546+0000","last_change":"2024-09-21T12:21:20.931077+0000","last_active":"2024-09-21T12:23:20.202546+0000","last_peered":"2024-09-21T12:23:20.202546+0000","last_clean":"2024-09-21T12:23:20.202546+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:23:20.202546+0000","last_undegraded":"2024-09-21T12:23:20.202546+0000","last_fullsized":"2024-09-21T12:23:20.202546+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:20.931025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.931025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":147,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.196673+0000","last_change":"2024-09-21T12:20:21.308003+0000","last_active":"2024-09-21T12:23:20.196673+0000","last_peered":"2024-09-21T12:23:20.196673+0000","last_clean":"2024-09-21T12:23:20.196673+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:23:20.196673+0000","last_undegraded":"2024-09-21T12:23:20.196673+0000","last_fullsized":"2024-09-21T12:23:20.196673+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:24.738749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":180,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.202040+0000","last_change":"2024-09-21T12:22:35.026113+0000","last_active":"2024-09-21T12:23:20.202040+0000","last_peered":"2024-09-21T12:23:20.202040+0000","last_clean":"2024-09-21T12:23:20.202040+0000","last_became_active":"2024-09-21T12:20:17.694794+0000","last_became_peered":"2024-09-21T12:20:17.694794+0000","last_unstale":"2024-09-21T12:23:20.202040+0000","last_undegraded":"2024-09-21T12:23:20.202040+0000","last_fullsized":"2024-09-21T12:23:20.202040+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:22:35.026007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:35.026007+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:01.068751+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":166,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.221286+0000","last_change":"2024-09-21T12:21:55.172399+0000","last_active":"2024-09-21T12:23:20.221286+0000","last_peered":"2024-09-21T12:23:20.221286+0000","last_clean":"2024-09-21T12:23:20.221286+0000","last_became_active":"2024-09-21T12:20:15.295968+0000","last_became_peered":"2024-09-21T12:20:15.295968+0000","last_unstale":"2024-09-21T12:23:20.221286+0000","last_undegraded":"2024-09-21T12:23:20.221286+0000","last_fullsized":"2024-09-21T12:23:20.221286+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:55.172344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:55.172344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:22.388105+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":171,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.200668+0000","last_change":"2024-09-21T12:22:50.921970+0000","last_active":"2024-09-21T12:23:20.200668+0000","last_peered":"2024-09-21T12:23:20.200668+0000","last_clean":"2024-09-21T12:23:20.200668+0000","last_became_active":"2024-09-21T12:22:50.921604+0000","last_became_peered":"2024-09-21T12:22:50.921604+0000","last_unstale":"2024-09-21T12:23:20.200668+0000","last_undegraded":"2024-09-21T12:23:20.200668+0000","last_fullsized":"2024-09-21T12:23:20.200668+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":145,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.202713+0000","last_change":"2024-09-21T12:20:21.595660+0000","last_active":"2024-09-21T12:23:20.202713+0000","last_peered":"2024-09-21T12:23:20.202713+0000","last_clean":"2024-09-21T12:23:20.202713+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:23:20.202713+0000","last_undegraded":"2024-09-21T12:23:20.202713+0000","last_fullsized":"2024-09-21T12:23:20.202713+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:28.985337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":159,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.201948+0000","last_change":"2024-09-21T12:20:17.304075+0000","last_active":"2024-09-21T12:23:20.201948+0000","last_peered":"2024-09-21T12:23:20.201948+0000","last_clean":"2024-09-21T12:23:20.201948+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:23:20.201948+0000","last_undegraded":"2024-09-21T12:23:20.201948+0000","last_fullsized":"2024-09-21T12:23:20.201948+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":173,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.198830+0000","last_change":"2024-09-21T12:22:14.635397+0000","last_active":"2024-09-21T12:23:20.198830+0000","last_peered":"2024-09-21T12:23:20.198830+0000","last_clean":"2024-09-21T12:23:20.198830+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:23:20.198830+0000","last_undegraded":"2024-09-21T12:23:20.198830+0000","last_fullsized":"2024-09-21T12:23:20.198830+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:22:14.635191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.635191+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":179,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.229103+0000","last_change":"2024-09-21T12:22:50.931202+0000","last_active":"2024-09-21T12:23:20.229103+0000","last_peered":"2024-09-21T12:23:20.229103+0000","last_clean":"2024-09-21T12:23:20.229103+0000","last_became_active":"2024-09-21T12:22:50.930615+0000","last_became_peered":"2024-09-21T12:22:50.930615+0000","last_unstale":"2024-09-21T12:23:20.229103+0000","last_undegraded":"2024-09-21T12:23:20.229103+0000","last_fullsized":"2024-09-21T12:23:20.229103+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:09.649109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:09.649109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:10.864999+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":167,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.200908+0000","last_change":"2024-09-21T12:22:50.917883+0000","last_active":"2024-09-21T12:23:20.200908+0000","last_peered":"2024-09-21T12:23:20.200908+0000","last_clean":"2024-09-21T12:23:20.200908+0000","last_became_active":"2024-09-21T12:22:50.917408+0000","last_became_peered":"2024-09-21T12:22:50.917408+0000","last_unstale":"2024-09-21T12:23:20.200908+0000","last_undegraded":"2024-09-21T12:23:20.200908+0000","last_fullsized":"2024-09-21T12:23:20.200908+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":168,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.196892+0000","last_change":"2024-09-21T12:22:50.914633+0000","last_active":"2024-09-21T12:23:20.196892+0000","last_peered":"2024-09-21T12:23:20.196892+0000","last_clean":"2024-09-21T12:23:20.196892+0000","last_became_active":"2024-09-21T12:22:50.914221+0000","last_became_peered":"2024-09-21T12:22:50.914221+0000","last_unstale":"2024-09-21T12:23:20.196892+0000","last_undegraded":"2024-09-21T12:23:20.196892+0000","last_fullsized":"2024-09-21T12:23:20.196892+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:08.353389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":165,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.202185+0000","last_change":"2024-09-21T12:22:44.080149+0000","last_active":"2024-09-21T12:23:20.202185+0000","last_peered":"2024-09-21T12:23:20.202185+0000","last_clean":"2024-09-21T12:23:20.202185+0000","last_became_active":"2024-09-21T12:20:15.297993+0000","last_became_peered":"2024-09-21T12:20:15.297993+0000","last_unstale":"2024-09-21T12:23:20.202185+0000","last_undegraded":"2024-09-21T12:23:20.202185+0000","last_fullsized":"2024-09-21T12:23:20.202185+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:44.080070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:44.080070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:01.088731+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":173,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.230648+0000","last_change":"2024-09-21T12:20:17.299134+0000","last_active":"2024-09-21T12:23:20.230648+0000","last_peered":"2024-09-21T12:23:20.230648+0000","last_clean":"2024-09-21T12:23:20.230648+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:23:20.230648+0000","last_undegraded":"2024-09-21T12:23:20.230648+0000","last_fullsized":"2024-09-21T12:23:20.230648+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:23:33.593130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":159,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-21T12:23:20.198422+0000","last_change":"2024-09-21T12:22:55.065505+0000","last_active":"2024-09-21T12:23:20.198422+0000","last_peered":"2024-09-21T12:23:20.198422+0000","last_clean":"2024-09-21T12:23:20.198422+0000","last_became_active":"2024-09-21T12:20:19.308247+0000","last_became_peered":"2024-09-21T12:20:19.308247+0000","last_unstale":"2024-09-21T12:23:20.198422+0000","last_undegraded":"2024-09-21T12:23:20.198422+0000","last_fullsized":"2024-09-21T12:23:20.198422+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:55.065444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:55.065444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:15.087826+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":5364645912,"num_objects":81859,"num_object_clones":0,"num_object_copies":245577,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81859,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126429,"num_write_kb":6665153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15812,"num_bytes_recovered":1036255232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16237932544,"data_stored":16237920328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70864,"ondisk_log_size":70864,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":690,"num_read_kb":447,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":3542,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":396,"ondisk_log_size":396,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":5023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":181,"num_read_kb":176,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":268350,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":51,"num_read_kb":101,"num_write":50,"num_write_kb":740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":858718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234025,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2094144,"kb_used_data":1986600,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":91651008,"statfs":{"total":95995035648,"available":93850632192,"internally_reserved":0,"allocated":2034278400,"data_stored":2033555648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462961,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2420796,"kb_used_data":2313260,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":91324356,"statfs":{"total":95995035648,"available":93516140544,"internally_reserved":0,"allocated":2368778240,"data_stored":2368051207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110098886},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724596,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1392632,"kb_used_data":1317028,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92352520,"statfs":{"total":95995035648,"available":94568980480,"internally_reserved":0,"allocated":1348636672,"data_stored":1347919230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953532,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3451928,"kb_used_data":3308104,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":90293224,"statfs":{"total":95995035648,"available":92460261376,"internally_reserved":0,"allocated":3387498496,"data_stored":3386744929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":147061191},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2057576,"kb_used_data":1972244,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":91687576,"statfs":{"total":95995035648,"available":93888077824,"internally_reserved":0,"allocated":2019577856,"data_stored":2018875255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2420784,"kb_used_data":2308764,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":91324368,"statfs":{"total":95995035648,"available":93516152832,"internally_reserved":0,"allocated":2364174336,"data_stored":2363399009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738452,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2745160,"kb_used_data":2637044,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":90999992,"statfs":{"total":95995035648,"available":93183991808,"internally_reserved":0,"allocated":2700333056,"data_stored":2699598753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2673086464,"data_stored":2673082392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2356482048,"data_stored":2356477976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2005270528,"data_stored":2005270528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3358392320,"data_stored":3358392320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":137297920,"data_stored":137297920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1342242816,"data_stored":1342242816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2349993984,"data_stored":2349989912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2015166464,"data_stored":2015166464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:23:26.838 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pgp_num 11 2024-09-21T12:23:26.869 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:26.895 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:26.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:26.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:26.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:23:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:23:26.996 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:27.021 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:27.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:27.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:27.122 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:27.148 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:27.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:27.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:27.249 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:27.292 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:27.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:27.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:27.325 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:27.383 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T12:23:27.393 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:27.469 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:27.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:27.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:27.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:27 smithi114 ceph-mon[30324]: 4.1c scrub starts 2024-09-21T12:23:27.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:27 smithi114 ceph-mon[30324]: 4.1c scrub ok 2024-09-21T12:23:27.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:27 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:23:27.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:27 smithi114 ceph-mon[30324]: osdmap e134: 8 total, 8 up, 7 in 2024-09-21T12:23:27.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:27 smithi114 ceph-mon[40867]: 4.1c scrub starts 2024-09-21T12:23:27.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:27 smithi114 ceph-mon[40867]: 4.1c scrub ok 2024-09-21T12:23:27.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:27 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:23:27.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:27 smithi114 ceph-mon[40867]: osdmap e134: 8 total, 8 up, 7 in 2024-09-21T12:23:27.570 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:27.623 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:27.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:27.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:27.724 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:27.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:27 smithi183 ceph-mon[33277]: 4.1c scrub starts 2024-09-21T12:23:27.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:27 smithi183 ceph-mon[33277]: 4.1c scrub ok 2024-09-21T12:23:27.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:27 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:23:27.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:27 smithi183 ceph-mon[33277]: osdmap e134: 8 total, 8 up, 7 in 2024-09-21T12:23:27.772 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:27.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:27.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:27.873 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:27.898 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:27.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:27.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:28.000 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:28.024 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:28.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:28.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:28.126 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:28.166 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:28.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:28.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:28.268 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:28.293 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:28.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:28.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:28.395 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:28.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:28 smithi114 ceph-mon[30324]: 5.0 scrub starts 2024-09-21T12:23:28.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:28 smithi114 ceph-mon[30324]: 5.0 scrub ok 2024-09-21T12:23:28.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:28 smithi114 ceph-mon[30324]: 6.19 scrub starts 2024-09-21T12:23:28.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:28 smithi114 ceph-mon[30324]: 6.19 scrub ok 2024-09-21T12:23:28.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:28 smithi114 ceph-mon[30324]: pgmap v143: 158 pgs: 158 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.42k op/s; 950 KiB/s, 16 objects/s recovering 2024-09-21T12:23:28.420 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:28 smithi114 ceph-mon[40867]: 5.0 scrub starts 2024-09-21T12:23:28.421 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:28 smithi114 ceph-mon[40867]: 5.0 scrub ok 2024-09-21T12:23:28.421 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:28 smithi114 ceph-mon[40867]: 6.19 scrub starts 2024-09-21T12:23:28.421 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:28 smithi114 ceph-mon[40867]: 6.19 scrub ok 2024-09-21T12:23:28.421 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:28 smithi114 ceph-mon[40867]: pgmap v143: 158 pgs: 158 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.42k op/s; 950 KiB/s, 16 objects/s recovering 2024-09-21T12:23:28.421 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:23:28.421 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:23:28.432 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:28.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:28.448 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:28.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:28.458 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:23:28.458 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:23:28.504 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:23:28.505 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:23:28.549 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:28.552 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:23:28.552 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:23:28.588 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:28.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:28.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:28.590 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:23:28.590 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:23:28.630 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:23:28.630 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:23:28.663 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:23:28.663 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:23:28.690 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:28.698 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:23:28.698 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:23:28.734 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:28.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:28.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:28.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:28 smithi183 ceph-mon[33277]: 5.0 scrub starts 2024-09-21T12:23:28.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:28 smithi183 ceph-mon[33277]: 5.0 scrub ok 2024-09-21T12:23:28.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:28 smithi183 ceph-mon[33277]: 6.19 scrub starts 2024-09-21T12:23:28.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:28 smithi183 ceph-mon[33277]: 6.19 scrub ok 2024-09-21T12:23:28.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:28 smithi183 ceph-mon[33277]: pgmap v143: 158 pgs: 158 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.42k op/s; 950 KiB/s, 16 objects/s recovering 2024-09-21T12:23:28.836 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:28.867 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:28.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:28.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:28.969 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:29.004 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:29.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:29.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:29.106 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:29.132 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:29.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:29.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:29.234 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:29.258 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:29.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:29.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:29.271 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-21T12:23:29.321 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:29 smithi114 ceph-mon[30324]: 6.9 scrub starts 2024-09-21T12:23:29.321 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:29 smithi114 ceph-mon[30324]: 6.9 scrub ok 2024-09-21T12:23:29.321 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:29 smithi114 ceph-mon[30324]: 3.16 scrub starts 2024-09-21T12:23:29.321 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:29 smithi114 ceph-mon[30324]: 3.16 scrub ok 2024-09-21T12:23:29.321 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:29 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1740515495' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:23:29.321 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:29 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/177665228' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:23:29.321 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:29 smithi114 ceph-mon[40867]: 6.9 scrub starts 2024-09-21T12:23:29.321 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:29 smithi114 ceph-mon[40867]: 6.9 scrub ok 2024-09-21T12:23:29.321 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:29 smithi114 ceph-mon[40867]: 3.16 scrub starts 2024-09-21T12:23:29.321 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:29 smithi114 ceph-mon[40867]: 3.16 scrub ok 2024-09-21T12:23:29.322 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:29 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1740515495' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:23:29.322 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:29 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/177665228' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:23:29.360 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:29.386 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:29.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:29.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:29.488 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:29.510 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:29.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:29.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:29.611 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:29.645 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:29.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:29.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:29 smithi183 ceph-mon[33277]: 6.9 scrub starts 2024-09-21T12:23:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:29 smithi183 ceph-mon[33277]: 6.9 scrub ok 2024-09-21T12:23:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:29 smithi183 ceph-mon[33277]: 3.16 scrub starts 2024-09-21T12:23:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:29 smithi183 ceph-mon[33277]: 3.16 scrub ok 2024-09-21T12:23:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:29 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1740515495' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:23:29.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:29 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/177665228' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:23:29.747 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:29.771 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:29.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:29.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:29.873 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:29.916 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:29.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:29.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:30.017 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:30.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:30.055 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:30.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:30.155 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:30.181 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:30.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:30.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:30.282 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:30.306 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:30.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:30.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:30.408 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:30.435 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:30.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:30.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:30.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:30 smithi114 ceph-mon[40867]: 6.10 scrub starts 2024-09-21T12:23:30.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:30 smithi114 ceph-mon[40867]: 6.10 scrub ok 2024-09-21T12:23:30.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:30 smithi114 ceph-mon[40867]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:23:30.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:30 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/177665228' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:23:30.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:30 smithi114 ceph-mon[40867]: 5.6 scrub starts 2024-09-21T12:23:30.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:30 smithi114 ceph-mon[40867]: 5.6 scrub ok 2024-09-21T12:23:30.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:30 smithi114 ceph-mon[40867]: osdmap e135: 8 total, 8 up, 7 in 2024-09-21T12:23:30.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:30 smithi114 ceph-mon[40867]: pgmap v145: 158 pgs: 158 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 796 op/s 2024-09-21T12:23:30.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:30 smithi114 ceph-mon[30324]: 6.10 scrub starts 2024-09-21T12:23:30.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:30 smithi114 ceph-mon[30324]: 6.10 scrub ok 2024-09-21T12:23:30.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:30 smithi114 ceph-mon[30324]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:23:30.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:30 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/177665228' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:23:30.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:30 smithi114 ceph-mon[30324]: 5.6 scrub starts 2024-09-21T12:23:30.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:30 smithi114 ceph-mon[30324]: 5.6 scrub ok 2024-09-21T12:23:30.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:30 smithi114 ceph-mon[30324]: osdmap e135: 8 total, 8 up, 7 in 2024-09-21T12:23:30.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:30 smithi114 ceph-mon[30324]: pgmap v145: 158 pgs: 158 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 796 op/s 2024-09-21T12:23:30.536 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:30.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:30.583 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:30.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:30.663 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:30.684 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:30.735 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:30.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:30.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:30.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:30 smithi183 ceph-mon[33277]: 6.10 scrub starts 2024-09-21T12:23:30.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:30 smithi183 ceph-mon[33277]: 6.10 scrub ok 2024-09-21T12:23:30.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:30 smithi183 ceph-mon[33277]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:23:30.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:30 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/177665228' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:23:30.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:30 smithi183 ceph-mon[33277]: 5.6 scrub starts 2024-09-21T12:23:30.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:30 smithi183 ceph-mon[33277]: 5.6 scrub ok 2024-09-21T12:23:30.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:30 smithi183 ceph-mon[33277]: osdmap e135: 8 total, 8 up, 7 in 2024-09-21T12:23:30.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:30 smithi183 ceph-mon[33277]: pgmap v145: 158 pgs: 158 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 796 op/s 2024-09-21T12:23:30.837 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:30.883 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:30.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:30.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:30.985 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:31.007 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:31.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:31.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:31.108 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:31.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:31.159 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:31.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:31.260 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:31.276 INFO:teuthology.orchestra.run.smithi114.stderr:set pool 1 pgp_num to 11 2024-09-21T12:23:31.307 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:23:31.307 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:23:31.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:31.313 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:31.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:31.414 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:31.440 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:31.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:31.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:31.502 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:31 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:23:31.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:31 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:23:31.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:31 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/634805312' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-21T12:23:31.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:31 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:23:31.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:31 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:23:31.503 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:31 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:23:31.503 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:31 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:23:31.503 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:31 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/634805312' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-21T12:23:31.503 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:31 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:23:31.503 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:31 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:23:31.542 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:31.567 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:31.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:31.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:31.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:31 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:23:31.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:31 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:23:31.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:31 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/634805312' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-21T12:23:31.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:31 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:23:31.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:31 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:23:31.802 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:31.847 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:31.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:31.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:31.905 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:23:31.949 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:31.973 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:31.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:31.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:32.074 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:32.119 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:32.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:32.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:32.221 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:32.245 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:32.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:32.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:32.333 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T12:23:32.346 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:32.438 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:32.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:32.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:32.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:32 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/634805312' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-21T12:23:32.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:32 smithi114 ceph-mon[30324]: osdmap e136: 8 total, 8 up, 7 in 2024-09-21T12:23:32.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:32 smithi114 ceph-mon[30324]: pgmap v147: 158 pgs: 158 active+clean; 4.5 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.29k op/s 2024-09-21T12:23:32.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:32 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/634805312' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-21T12:23:32.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:32 smithi114 ceph-mon[40867]: osdmap e136: 8 total, 8 up, 7 in 2024-09-21T12:23:32.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:32 smithi114 ceph-mon[40867]: pgmap v147: 158 pgs: 158 active+clean; 4.5 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.29k op/s 2024-09-21T12:23:32.540 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:32.588 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:32.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:32.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:32.690 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:32.715 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:32.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:32.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:32.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:32 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/634805312' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-21T12:23:32.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:32 smithi183 ceph-mon[33277]: osdmap e136: 8 total, 8 up, 7 in 2024-09-21T12:23:32.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:32 smithi183 ceph-mon[33277]: pgmap v147: 158 pgs: 158 active+clean; 4.5 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.29k op/s 2024-09-21T12:23:32.795 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:32.817 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:32.824 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T12:23:32.824 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:23:32 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:23:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:23:32.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:32.880 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:32.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:32.980 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:33.004 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:33.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:33.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:33.108 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:33.126 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:33.133 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:33.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:33.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:33.234 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:33.258 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:33.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:33.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:33.359 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:33.384 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:33.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:33.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:33.486 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:33.510 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:33.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:33.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:33.614 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:33.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:33.661 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:33.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:33.705 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:33 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3749423228' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:23:33.705 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:33 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3749423228' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:23:33.762 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:33.788 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:33.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:33.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:33.889 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:33.910 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:23:33.910 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:23:33.910 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:23:33.910 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:23:33.910 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:23:33.910 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:23:33.910 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:23:33.911 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:23:33.911 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:23:33.911 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:23:33.911 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:23:33.911 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:23:33.911 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:23:33.911 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:23:33.911 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:23:33.911 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:23:33.928 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:33.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:33.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:33.971 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:33 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3749423228' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:23:34.029 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:34.065 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:34.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:34.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:34.167 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:34.198 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:34.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:34.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:34.300 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:34.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:34.329 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:34.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:34.430 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:34.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:34.478 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:34.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:34.578 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:34.604 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:34.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:34.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:34.706 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:34.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:34.735 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:34.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:34.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:34 smithi114 ceph-mon[40867]: pgmap v148: 158 pgs: 158 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 1.30k op/s 2024-09-21T12:23:34.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:34 smithi114 ceph-mon[30324]: pgmap v148: 158 pgs: 158 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 1.30k op/s 2024-09-21T12:23:34.837 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:34.871 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:34.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:34.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:34.973 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:34.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:34 smithi183 ceph-mon[33277]: pgmap v148: 158 pgs: 158 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 1.30k op/s 2024-09-21T12:23:35.010 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:35.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:35.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:35.112 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:35.149 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:35.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:35.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:35.250 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:35.276 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:35.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:35.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:35.378 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:35.402 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:35.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:35.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:35.503 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:35.534 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:35.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:35.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:35.536 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-21T12:23:35.637 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:35.681 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:35.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:35.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:35.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:35 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2266055523' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:23:35.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:35 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:23:35.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:35 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2266055523' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:23:35.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:35 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:23:35.783 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:35.822 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:35.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:35.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:35.924 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:35.950 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:35.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:35.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:35.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:35 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2266055523' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:23:35.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:35 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:23:36.052 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:36.081 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:36.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:36.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:36.183 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:36.208 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:36.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:36.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:36.310 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:36.346 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:36.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:36.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:36.448 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:36.473 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:36.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:36.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:36.574 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:36.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:36.609 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:36.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:36.710 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:36.742 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:36.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:36.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:36.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:36 smithi114 ceph-mon[40867]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:23:36.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:36 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:23:36.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:36 smithi114 ceph-mon[40867]: osdmap e137: 8 total, 8 up, 7 in 2024-09-21T12:23:36.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:36 smithi114 ceph-mon[40867]: pgmap v150: 158 pgs: 158 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 807 B/s rd, 0 B/s wr, 928 op/s 2024-09-21T12:23:36.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:36 smithi114 ceph-mon[30324]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:23:36.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:36 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:23:36.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:36 smithi114 ceph-mon[30324]: osdmap e137: 8 total, 8 up, 7 in 2024-09-21T12:23:36.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:36 smithi114 ceph-mon[30324]: pgmap v150: 158 pgs: 158 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 807 B/s rd, 0 B/s wr, 928 op/s 2024-09-21T12:23:36.844 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:36.866 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:36.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:36.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:36.968 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:36.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:23:36 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:23:36.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:36 smithi183 ceph-mon[33277]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:23:36.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:36 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:23:36.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:36 smithi183 ceph-mon[33277]: osdmap e137: 8 total, 8 up, 7 in 2024-09-21T12:23:36.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:36 smithi183 ceph-mon[33277]: pgmap v150: 158 pgs: 158 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 807 B/s rd, 0 B/s wr, 928 op/s 2024-09-21T12:23:37.005 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:37.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:37.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:37.106 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:37.133 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:37.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:37.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:37.235 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:37.260 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:37.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:37.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:37.284 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T12:23:37.284 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T12:23:37.362 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:37.381 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:37.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:37.386 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:37.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:37.487 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:37.511 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:37.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:37.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:37.612 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:37.662 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:37.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:37.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:37.686 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:23:37.748 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:37 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:23:37.748 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:37 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:23:37.764 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:37.808 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:37.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:37.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:37.910 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:37.935 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:37.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:37.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:37.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:37 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:23:38.037 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:38.084 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:38.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:38.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:38.185 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:38.208 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:38.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:38.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:38.309 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:38.347 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:38.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:38.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:38.448 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:38.470 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:38.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:38.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:38.494 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:23:38.494 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-21T12:23:38.494 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.495 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:146006 7.8 7:1e724381:::benchmark_data_smithi114_78432_object41860:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-21T12:23:38.495 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:23:38.475324+0000", 2024-09-21T12:23:38.495 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.016415101000000001, 2024-09-21T12:23:38.495 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.016442938000000001, 2024-09-21T12:23:38.495 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:38.495 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:38.495 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:23:38.495 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:23:38.495 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:23:38.495 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:23:38.495 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 146006 2024-09-21T12:23:38.495 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.496 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:38.496 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.496 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:38.496 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.475324+0000", 2024-09-21T12:23:38.496 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:38.496 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.496 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.496 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:38.496 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.475324+0000", 2024-09-21T12:23:38.496 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:38.496 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.496 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.496 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:38.497 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.475325+0000", 2024-09-21T12:23:38.497 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.39e-06 2024-09-21T12:23:38.497 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.497 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.497 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:38.497 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.475331+0000", 2024-09-21T12:23:38.497 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6550000000000001e-06 2024-09-21T12:23:38.497 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.497 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.497 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:38.497 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.475343+0000", 2024-09-21T12:23:38.497 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2347000000000001e-05 2024-09-21T12:23:38.497 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.497 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.498 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:38.498 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.475351+0000", 2024-09-21T12:23:38.498 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.396e-06 2024-09-21T12:23:38.498 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.498 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.498 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:38.498 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.475408+0000", 2024-09-21T12:23:38.498 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7549000000000001e-05 2024-09-21T12:23:38.498 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.498 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.498 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:38.498 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.475540+0000", 2024-09-21T12:23:38.499 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013131800000000001 2024-09-21T12:23:38.499 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.499 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.499 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:23:38.499 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.475666+0000", 2024-09-21T12:23:38.499 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012591300000000001 2024-09-21T12:23:38.499 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.499 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.499 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:38.499 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.481205+0000", 2024-09-21T12:23:38.499 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0055389640000000004 2024-09-21T12:23:38.499 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.499 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.500 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:23:38.500 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.482678+0000", 2024-09-21T12:23:38.500 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0014737350000000001 2024-09-21T12:23:38.500 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:38.500 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:38.500 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:38.500 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.500 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.500 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.24695.0:146004 7.3 e137/120 7:cd6336d6:::benchmark_data_smithi114_78432_object41858:head v 137'9010, mlcod=137'9008)", 2024-09-21T12:23:38.500 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:23:38.488241+0000", 2024-09-21T12:23:38.500 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0034980829999999999, 2024-09-21T12:23:38.500 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0035989609999999999, 2024-09-21T12:23:38.500 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:38.500 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:38.500 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:23:38.501 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:38.501 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.501 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:38.501 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.488241+0000", 2024-09-21T12:23:38.501 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:38.501 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.501 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.501 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:38.501 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.488241+0000", 2024-09-21T12:23:38.501 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:38.501 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.501 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.501 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:38.501 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.488242+0000", 2024-09-21T12:23:38.501 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.074e-06 2024-09-21T12:23:38.502 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.502 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.502 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:38.502 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.488253+0000", 2024-09-21T12:23:38.502 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0576e-05 2024-09-21T12:23:38.502 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.502 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.502 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:38.502 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.488272+0000", 2024-09-21T12:23:38.502 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9052e-05 2024-09-21T12:23:38.502 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.502 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.502 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:38.502 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.488281+0000", 2024-09-21T12:23:38.502 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0629999999999995e-06 2024-09-21T12:23:38.503 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.503 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.503 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:38.503 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.488409+0000", 2024-09-21T12:23:38.503 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012798099999999999 2024-09-21T12:23:38.503 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.503 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.503 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:38.503 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.488426+0000", 2024-09-21T12:23:38.503 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7113999999999999e-05 2024-09-21T12:23:38.503 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:38.503 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:38.503 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:38.503 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.503 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.503 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.24695.0:146011 7.3 e137/120 7:cd23760f:::benchmark_data_smithi114_78432_object41865:head v 137'9011, mlcod=137'9008)", 2024-09-21T12:23:38.504 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:23:38.488340+0000", 2024-09-21T12:23:38.504 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0033987219999999999, 2024-09-21T12:23:38.504 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003546741, 2024-09-21T12:23:38.504 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:38.504 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:38.504 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:23:38.504 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:38.504 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.504 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:38.504 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.488340+0000", 2024-09-21T12:23:38.504 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:38.504 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.504 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.504 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:38.504 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.488340+0000", 2024-09-21T12:23:38.504 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:38.505 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.505 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.505 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:38.505 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.488341+0000", 2024-09-21T12:23:38.505 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.6300000000000001e-07 2024-09-21T12:23:38.505 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.505 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.505 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:38.505 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.488348+0000", 2024-09-21T12:23:38.505 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8170000000000001e-06 2024-09-21T12:23:38.505 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.505 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.505 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:38.505 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.488354+0000", 2024-09-21T12:23:38.505 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0499999999999997e-06 2024-09-21T12:23:38.506 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.506 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.506 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:38.506 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.488358+0000", 2024-09-21T12:23:38.506 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8099999999999997e-06 2024-09-21T12:23:38.506 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.506 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.506 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:38.506 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.490298+0000", 2024-09-21T12:23:38.506 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0019395899999999999 2024-09-21T12:23:38.506 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.506 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.506 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:38.506 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.490315+0000", 2024-09-21T12:23:38.506 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6646999999999998e-05 2024-09-21T12:23:38.507 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:38.507 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:38.507 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:38.507 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.507 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.507 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:146012 7.8 7:106d83f9:::benchmark_data_smithi114_78432_object41866:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-21T12:23:38.507 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:23:38.490282+0000", 2024-09-21T12:23:38.507 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.001457189, 2024-09-21T12:23:38.507 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0016510629999999999, 2024-09-21T12:23:38.507 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:38.507 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:38.507 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:23:38.507 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:23:38.507 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:23:38.508 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:23:38.508 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 146012 2024-09-21T12:23:38.508 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.508 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:38.508 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.508 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:38.508 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.490282+0000", 2024-09-21T12:23:38.508 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:38.508 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.509 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.509 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:38.509 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.490282+0000", 2024-09-21T12:23:38.509 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:38.509 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.509 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.509 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:38.509 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.490284+0000", 2024-09-21T12:23:38.509 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8479999999999999e-06 2024-09-21T12:23:38.509 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.510 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.510 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:38.510 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.490291+0000", 2024-09-21T12:23:38.510 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3690000000000001e-06 2024-09-21T12:23:38.510 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.510 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.510 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:38.510 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.490307+0000", 2024-09-21T12:23:38.510 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5517000000000001e-05 2024-09-21T12:23:38.510 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.510 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.510 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:38.510 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.490315+0000", 2024-09-21T12:23:38.510 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.3510000000000002e-06 2024-09-21T12:23:38.511 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.511 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.511 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:38.511 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.490384+0000", 2024-09-21T12:23:38.511 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8817000000000002e-05 2024-09-21T12:23:38.511 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.511 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.511 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:38.511 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.490503+0000", 2024-09-21T12:23:38.511 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000118942 2024-09-21T12:23:38.511 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:38.511 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:38.511 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:23:38.511 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:38.490621+0000", 2024-09-21T12:23:38.512 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011861799999999999 2024-09-21T12:23:38.512 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:38.512 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:38.512 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:38.512 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:38.512 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:23:38.512 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 4 2024-09-21T12:23:38.512 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:23:38.512 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:38.572 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:38.610 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:38.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:38.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:38.712 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:38.735 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:38.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:38.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:38.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:38 smithi114 ceph-mon[40867]: pgmap v151: 158 pgs: 158 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s 2024-09-21T12:23:38.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:38 smithi114 ceph-mon[30324]: pgmap v151: 158 pgs: 158 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s 2024-09-21T12:23:38.837 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:38.878 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:38.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:38.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:38.980 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:38.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:38 smithi183 ceph-mon[33277]: pgmap v151: 158 pgs: 158 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s 2024-09-21T12:23:39.017 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:39.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:39.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:39.119 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:39.144 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:39.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:39.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:39.245 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:39.271 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:39.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:39.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:39.373 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:39.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:39.413 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:39.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:39.514 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:39.538 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:39.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:39.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:39.639 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:39.663 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:39.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:39.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:39.765 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:39.773 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T12:23:39.806 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:39.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:39.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:39.899 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T12:23:39.908 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:39.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:39.988 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:39.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:40.089 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:40.146 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:40.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:40.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:40.247 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:40.286 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:40.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:40.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:40.388 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:40.428 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:40.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:40.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:40.530 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:40.570 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:40.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:40.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:40.672 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:40.698 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:40.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:40.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:40.800 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:40.823 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:40.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:40.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:40.923 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:40 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:23:40.923 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:40 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:23:40.923 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:40 smithi114 ceph-mon[30324]: osdmap e138: 8 total, 8 up, 7 in 2024-09-21T12:23:40.923 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:40 smithi114 ceph-mon[30324]: pgmap v153: 158 pgs: 158 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.23k op/s 2024-09-21T12:23:40.923 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:40 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:23:40.923 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:40 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:23:40.924 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:40 smithi114 ceph-mon[40867]: osdmap e138: 8 total, 8 up, 7 in 2024-09-21T12:23:40.924 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:40 smithi114 ceph-mon[40867]: pgmap v153: 158 pgs: 158 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.23k op/s 2024-09-21T12:23:40.925 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:40.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:40.963 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:40.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:40.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:40 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:23:40.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:40 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:23:40.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:40 smithi183 ceph-mon[33277]: osdmap e138: 8 total, 8 up, 7 in 2024-09-21T12:23:40.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:40 smithi183 ceph-mon[33277]: pgmap v153: 158 pgs: 158 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.23k op/s 2024-09-21T12:23:41.065 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:41.085 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:41.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:41.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:41.187 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:41.212 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:41.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:41.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:41.314 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:41.339 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:41.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:41.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:41.441 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:41.476 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:41.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:41.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:41.578 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:41.617 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:41.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:41.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:41.719 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:41.753 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:41.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:41.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:41.855 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:41.924 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:41.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:41.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:41.945 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:42.026 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:42.047 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:42.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:42.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:42.149 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:42.173 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:42.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:42.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:42.275 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:42.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:42.309 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:42.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:42.410 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:42.435 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:42.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:42.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:42.537 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:42.562 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:42.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:42.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:42.664 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:42.685 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:42.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:42.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:42.787 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:42.813 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:42.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:42.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:42.820 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:23:42 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:23:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:23:42.915 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:42.940 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:42.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:42.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:43.042 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:43.067 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:43.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:43.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:43.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:42 smithi114 ceph-mon[30324]: pgmap v154: 158 pgs: 158 active+clean; 3.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 945 op/s 2024-09-21T12:23:43.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:42 smithi114 ceph-mon[40867]: pgmap v154: 158 pgs: 158 active+clean; 3.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 945 op/s 2024-09-21T12:23:43.168 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:43.191 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:43.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:43.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:43.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:42 smithi183 ceph-mon[33277]: pgmap v154: 158 pgs: 158 active+clean; 3.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 945 op/s 2024-09-21T12:23:43.293 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:43.302 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:43.322 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:23:43.323 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:23:43.342 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:43.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:43.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:43.366 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:23:43.366 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:23:43.399 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:23:43.399 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:23:43.431 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:23:43.431 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:23:43.443 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:43.465 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:23:43.465 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:23:43.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:43.494 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:43.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:43.501 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:23:43.502 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:23:43.533 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:23:43.534 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:23:43.565 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:23:43.566 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:23:43.595 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:43.618 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:43.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:43.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:43.719 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:43.743 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:43.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:43.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:43.845 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:43.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:43.888 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:43.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:43.928 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:43 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3122819556' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:23:43.928 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:43 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3122819556' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:23:43.989 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:44.015 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:44.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:44.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:44.117 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:44.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:44.160 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:44.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:44.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:43 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3122819556' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:23:44.261 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:44.302 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:44.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:44.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:44.404 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:44.427 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:44.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:44.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:44.529 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:44.562 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:44.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:44.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:44.664 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:44.701 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:44.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:44.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:44.802 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:44.827 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:44.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:44.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:44.838 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T12:23:44.929 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:44.969 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:44.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:44.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:45.071 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:45.098 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:44 smithi114 ceph-mon[40867]: pgmap v155: 158 pgs: 158 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-09-21T12:23:45.098 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:44 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/752894473' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:23:45.098 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:44 smithi114 ceph-mon[30324]: pgmap v155: 158 pgs: 158 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-09-21T12:23:45.098 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:44 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/752894473' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:23:45.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:45.106 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:45.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:45.207 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:45.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:44 smithi183 ceph-mon[33277]: pgmap v155: 158 pgs: 158 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-09-21T12:23:45.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:44 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/752894473' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:23:45.246 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:45.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:45.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:45.348 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:45.372 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:45.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:45.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:45.474 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:45.508 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:45.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:45.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:45.548 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:45.610 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:45.673 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:45.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:45.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:45.774 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:45.810 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:45.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:45.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:45.865 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-21T12:23:45.912 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:45.938 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:45.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:45.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:46.036 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T12:23:46.040 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:46.063 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:46.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:46.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:46.164 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:46.188 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:46.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:46.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:46.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:45 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/752894473' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T12:23:46.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:45 smithi183 ceph-mon[33277]: osdmap e139: 8 total, 8 up, 7 in 2024-09-21T12:23:46.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:45 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/783952747' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:23:46.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:45 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 2]} : dispatch 2024-09-21T12:23:46.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:45 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]} : dispatch 2024-09-21T12:23:46.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:45 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]} : dispatch 2024-09-21T12:23:46.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:45 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]} : dispatch 2024-09-21T12:23:46.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:45 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]} : dispatch 2024-09-21T12:23:46.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:45 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 2]} : dispatch 2024-09-21T12:23:46.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:45 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]} : dispatch 2024-09-21T12:23:46.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:45 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]} : dispatch 2024-09-21T12:23:46.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:45 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]} : dispatch 2024-09-21T12:23:46.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:45 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]} : dispatch 2024-09-21T12:23:46.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:45 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/752894473' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T12:23:46.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:45 smithi114 ceph-mon[30324]: osdmap e139: 8 total, 8 up, 7 in 2024-09-21T12:23:46.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:45 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/783952747' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:23:46.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:45 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 2]} : dispatch 2024-09-21T12:23:46.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:45 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]} : dispatch 2024-09-21T12:23:46.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:45 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]} : dispatch 2024-09-21T12:23:46.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:45 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]} : dispatch 2024-09-21T12:23:46.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:45 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]} : dispatch 2024-09-21T12:23:46.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:45 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 2]} : dispatch 2024-09-21T12:23:46.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:45 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]} : dispatch 2024-09-21T12:23:46.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:45 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]} : dispatch 2024-09-21T12:23:46.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:45 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]} : dispatch 2024-09-21T12:23:46.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:45 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]} : dispatch 2024-09-21T12:23:46.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:45 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/752894473' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T12:23:46.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:45 smithi114 ceph-mon[40867]: osdmap e139: 8 total, 8 up, 7 in 2024-09-21T12:23:46.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:45 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/783952747' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:23:46.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:45 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 2]} : dispatch 2024-09-21T12:23:46.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:45 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]} : dispatch 2024-09-21T12:23:46.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:45 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]} : dispatch 2024-09-21T12:23:46.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:45 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]} : dispatch 2024-09-21T12:23:46.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:45 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]} : dispatch 2024-09-21T12:23:46.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:45 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 2]} : dispatch 2024-09-21T12:23:46.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:45 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]} : dispatch 2024-09-21T12:23:46.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:45 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]} : dispatch 2024-09-21T12:23:46.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:45 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]} : dispatch 2024-09-21T12:23:46.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:45 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]} : dispatch 2024-09-21T12:23:46.290 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:46.314 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:46.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:46.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:46.416 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:46.440 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:46.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:46.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:46.542 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:46.566 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:46.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:46.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:46.667 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:46.702 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:46.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:46.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:46.804 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:46.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:46.843 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:46.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:46.944 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:46.968 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:46.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:46.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:47.069 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:47.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:47.113 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:47.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:47.199 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:46 smithi114 ceph-mon[30324]: pgmap v157: 158 pgs: 158 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 678 op/s 2024-09-21T12:23:47.199 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:46 smithi114 ceph-mon[30324]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:23:47.199 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:46 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/783952747' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:23:47.199 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:46 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 2]}]': finished 2024-09-21T12:23:47.199 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:46 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-09-21T12:23:47.200 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:46 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]': finished 2024-09-21T12:23:47.200 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:46 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-09-21T12:23:47.200 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:46 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]': finished 2024-09-21T12:23:47.200 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:46 smithi114 ceph-mon[30324]: osdmap e140: 8 total, 8 up, 7 in 2024-09-21T12:23:47.200 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:23:47.200 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:46 smithi114 ceph-mon[30324]: 6.c scrub starts 2024-09-21T12:23:47.200 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:46 smithi114 ceph-mon[30324]: 6.c scrub ok 2024-09-21T12:23:47.200 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:46 smithi114 ceph-mon[40867]: pgmap v157: 158 pgs: 158 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 678 op/s 2024-09-21T12:23:47.200 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:46 smithi114 ceph-mon[40867]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:23:47.200 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:46 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/783952747' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:23:47.200 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:46 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 2]}]': finished 2024-09-21T12:23:47.200 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:46 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-09-21T12:23:47.200 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:46 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]': finished 2024-09-21T12:23:47.201 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:46 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-09-21T12:23:47.201 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:46 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]': finished 2024-09-21T12:23:47.201 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:46 smithi114 ceph-mon[40867]: osdmap e140: 8 total, 8 up, 7 in 2024-09-21T12:23:47.201 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:23:47.201 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:46 smithi114 ceph-mon[40867]: 6.c scrub starts 2024-09-21T12:23:47.201 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:46 smithi114 ceph-mon[40867]: 6.c scrub ok 2024-09-21T12:23:47.214 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:47.238 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:47.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:23:46 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:23:47.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:46 smithi183 ceph-mon[33277]: pgmap v157: 158 pgs: 158 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 678 op/s 2024-09-21T12:23:47.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:46 smithi183 ceph-mon[33277]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:23:47.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:46 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/783952747' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:23:47.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:46 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [3, 2]}]': finished 2024-09-21T12:23:47.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:46 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-09-21T12:23:47.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:46 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]': finished 2024-09-21T12:23:47.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:46 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-09-21T12:23:47.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:46 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]': finished 2024-09-21T12:23:47.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:46 smithi183 ceph-mon[33277]: osdmap e140: 8 total, 8 up, 7 in 2024-09-21T12:23:47.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:23:47.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:46 smithi183 ceph-mon[33277]: 6.c scrub starts 2024-09-21T12:23:47.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:46 smithi183 ceph-mon[33277]: 6.c scrub ok 2024-09-21T12:23:47.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:47.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:47.258 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:23:47.258 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:23:47.261 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:47.343 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:47.368 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:47.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:47.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:47.470 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:47.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:47.518 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:47.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:47.593 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:23:47.619 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:47.642 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:47.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:47.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:47.744 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:47.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:47.782 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:47.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:47.883 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:47.905 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:47.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:47.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:48.006 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:48.046 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:48.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:48.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:48.148 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:48.195 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:48.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:48.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:48.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:47 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:23:48.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:47 smithi183 ceph-mon[33277]: osdmap e141: 8 total, 8 up, 7 in 2024-09-21T12:23:48.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:47 smithi183 ceph-mon[33277]: 4.1d scrub starts 2024-09-21T12:23:48.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:47 smithi183 ceph-mon[33277]: 4.1d scrub ok 2024-09-21T12:23:48.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:47 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:23:48.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:47 smithi114 ceph-mon[30324]: osdmap e141: 8 total, 8 up, 7 in 2024-09-21T12:23:48.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:47 smithi114 ceph-mon[30324]: 4.1d scrub starts 2024-09-21T12:23:48.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:47 smithi114 ceph-mon[30324]: 4.1d scrub ok 2024-09-21T12:23:48.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:47 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:23:48.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:47 smithi114 ceph-mon[40867]: osdmap e141: 8 total, 8 up, 7 in 2024-09-21T12:23:48.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:47 smithi114 ceph-mon[40867]: 4.1d scrub starts 2024-09-21T12:23:48.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:47 smithi114 ceph-mon[40867]: 4.1d scrub ok 2024-09-21T12:23:48.297 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:48.323 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:48.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:48.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:48.425 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:48.449 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:48.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:48.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:48.551 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:48.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:48.594 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:48.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:48.695 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:48.729 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:48.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:48.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:48.831 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:48.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:48.872 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:48.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:48.943 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T12:23:48.973 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:49.060 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:49.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:49.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:49.162 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:49.207 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:49.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:49.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:49.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:48 smithi183 ceph-mon[33277]: 4.9 scrub starts 2024-09-21T12:23:49.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:48 smithi183 ceph-mon[33277]: 4.9 scrub ok 2024-09-21T12:23:49.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:48 smithi183 ceph-mon[33277]: 4.f scrub starts 2024-09-21T12:23:49.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:48 smithi183 ceph-mon[33277]: 4.f scrub ok 2024-09-21T12:23:49.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:48 smithi183 ceph-mon[33277]: pgmap v160: 158 pgs: 5 peering, 153 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.55k op/s 2024-09-21T12:23:49.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:48 smithi183 ceph-mon[33277]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T12:23:49.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:48 smithi114 ceph-mon[30324]: 4.9 scrub starts 2024-09-21T12:23:49.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:48 smithi114 ceph-mon[30324]: 4.9 scrub ok 2024-09-21T12:23:49.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:48 smithi114 ceph-mon[30324]: 4.f scrub starts 2024-09-21T12:23:49.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:48 smithi114 ceph-mon[30324]: 4.f scrub ok 2024-09-21T12:23:49.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:48 smithi114 ceph-mon[30324]: pgmap v160: 158 pgs: 5 peering, 153 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.55k op/s 2024-09-21T12:23:49.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:48 smithi114 ceph-mon[30324]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T12:23:49.257 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:48 smithi114 ceph-mon[40867]: 4.9 scrub starts 2024-09-21T12:23:49.257 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:48 smithi114 ceph-mon[40867]: 4.9 scrub ok 2024-09-21T12:23:49.257 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:48 smithi114 ceph-mon[40867]: 4.f scrub starts 2024-09-21T12:23:49.257 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:48 smithi114 ceph-mon[40867]: 4.f scrub ok 2024-09-21T12:23:49.257 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:48 smithi114 ceph-mon[40867]: pgmap v160: 158 pgs: 5 peering, 153 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.55k op/s 2024-09-21T12:23:49.257 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:48 smithi114 ceph-mon[40867]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T12:23:49.310 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:49.335 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T12:23:49.362 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:49.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:49.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:49.435 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:49.464 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:49.488 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:49.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:49.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:49.589 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:49.615 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:49.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:49.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:49.714 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:49.717 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:49.740 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:49.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:49.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:49.841 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:49.883 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:49.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:49.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:49.902 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T12:23:49.985 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:50.011 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:50.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:50.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:50.113 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:50.136 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:50.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:50.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:50.161 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:49 smithi114 ceph-mon[40867]: 5.1d scrub starts 2024-09-21T12:23:50.161 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:49 smithi114 ceph-mon[40867]: 5.1d scrub ok 2024-09-21T12:23:50.161 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:49 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/4233783491' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:23:50.161 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:49 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:23:50.161 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:49 smithi114 ceph-mon[40867]: 4.c scrub starts 2024-09-21T12:23:50.161 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:49 smithi114 ceph-mon[40867]: 4.c scrub ok 2024-09-21T12:23:50.162 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:49 smithi114 ceph-mon[30324]: 5.1d scrub starts 2024-09-21T12:23:50.162 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:49 smithi114 ceph-mon[30324]: 5.1d scrub ok 2024-09-21T12:23:50.162 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:49 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4233783491' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:23:50.163 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:49 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:23:50.163 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:49 smithi114 ceph-mon[30324]: 4.c scrub starts 2024-09-21T12:23:50.163 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:49 smithi114 ceph-mon[30324]: 4.c scrub ok 2024-09-21T12:23:50.238 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:50.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:49 smithi183 ceph-mon[33277]: 5.1d scrub starts 2024-09-21T12:23:50.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:49 smithi183 ceph-mon[33277]: 5.1d scrub ok 2024-09-21T12:23:50.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:49 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/4233783491' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:23:50.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:49 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:23:50.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:49 smithi183 ceph-mon[33277]: 4.c scrub starts 2024-09-21T12:23:50.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:49 smithi183 ceph-mon[33277]: 4.c scrub ok 2024-09-21T12:23:50.263 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:50.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:50.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:50.366 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:50.390 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:50.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:50.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:50.492 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:50.517 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:50.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:50.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:50.618 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:50.644 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:50.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:50.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:50.705 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T12:23:50.745 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:50.786 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:50.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:50.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:50.888 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:50.911 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:50.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:50.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:51.013 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:51.059 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:51.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:51.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:51.064 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:23:51.065 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:23:51.079 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:23:51.079 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:23:51.126 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:23:51.126 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:23:51.162 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:51.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:50 smithi114 ceph-mon[30324]: 4.0 scrub starts 2024-09-21T12:23:51.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:50 smithi114 ceph-mon[30324]: 4.0 scrub ok 2024-09-21T12:23:51.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:50 smithi114 ceph-mon[30324]: 3.c scrub starts 2024-09-21T12:23:51.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:50 smithi114 ceph-mon[30324]: 3.c scrub ok 2024-09-21T12:23:51.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:50 smithi114 ceph-mon[30324]: pgmap v161: 158 pgs: 5 peering, 153 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 929 op/s 2024-09-21T12:23:51.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:50 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T12:23:51.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:50 smithi114 ceph-mon[30324]: osdmap e142: 8 total, 8 up, 7 in 2024-09-21T12:23:51.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:50 smithi114 ceph-mon[30324]: 4.d scrub starts 2024-09-21T12:23:51.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:50 smithi114 ceph-mon[30324]: 4.d scrub ok 2024-09-21T12:23:51.171 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:50 smithi114 ceph-mon[40867]: 4.0 scrub starts 2024-09-21T12:23:51.171 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:50 smithi114 ceph-mon[40867]: 4.0 scrub ok 2024-09-21T12:23:51.171 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:50 smithi114 ceph-mon[40867]: 3.c scrub starts 2024-09-21T12:23:51.171 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:50 smithi114 ceph-mon[40867]: 3.c scrub ok 2024-09-21T12:23:51.172 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:50 smithi114 ceph-mon[40867]: pgmap v161: 158 pgs: 5 peering, 153 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 929 op/s 2024-09-21T12:23:51.172 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:50 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T12:23:51.172 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:50 smithi114 ceph-mon[40867]: osdmap e142: 8 total, 8 up, 7 in 2024-09-21T12:23:51.172 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:50 smithi114 ceph-mon[40867]: 4.d scrub starts 2024-09-21T12:23:51.172 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:50 smithi114 ceph-mon[40867]: 4.d scrub ok 2024-09-21T12:23:51.172 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:23:51.173 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:23:51.187 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:51.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:51.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:51.218 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:23:51.218 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:23:51.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:50 smithi183 ceph-mon[33277]: 4.0 scrub starts 2024-09-21T12:23:51.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:50 smithi183 ceph-mon[33277]: 4.0 scrub ok 2024-09-21T12:23:51.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:50 smithi183 ceph-mon[33277]: 3.c scrub starts 2024-09-21T12:23:51.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:50 smithi183 ceph-mon[33277]: 3.c scrub ok 2024-09-21T12:23:51.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:50 smithi183 ceph-mon[33277]: pgmap v161: 158 pgs: 5 peering, 153 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 929 op/s 2024-09-21T12:23:51.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:50 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T12:23:51.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:50 smithi183 ceph-mon[33277]: osdmap e142: 8 total, 8 up, 7 in 2024-09-21T12:23:51.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:50 smithi183 ceph-mon[33277]: 4.d scrub starts 2024-09-21T12:23:51.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:50 smithi183 ceph-mon[33277]: 4.d scrub ok 2024-09-21T12:23:51.257 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:23:51.258 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:23:51.289 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:51.293 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:51.301 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:51.307 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:23:51.307 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:23:51.360 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:23:51.361 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:23:51.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:51.361 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:51.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:51.462 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:51.485 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:51.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:51.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:51.587 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:51.625 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:51.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:51.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:51.726 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:51.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:51.760 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:51.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:51.861 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:51.902 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:51.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:51.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:51.908 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-21T12:23:52.003 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:52.028 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:52.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:52.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:52.116 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:51 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3741196601' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:23:52.117 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:51 smithi114 ceph-mon[30324]: 4.e scrub starts 2024-09-21T12:23:52.117 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:51 smithi114 ceph-mon[30324]: 4.e scrub ok 2024-09-21T12:23:52.117 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:51 smithi114 ceph-mon[30324]: 6.1c scrub starts 2024-09-21T12:23:52.117 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:51 smithi114 ceph-mon[30324]: 6.1c scrub ok 2024-09-21T12:23:52.117 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:51 smithi114 ceph-mon[30324]: 6.14 scrub starts 2024-09-21T12:23:52.117 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:51 smithi114 ceph-mon[30324]: 6.14 scrub ok 2024-09-21T12:23:52.117 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:51 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1852010414' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:23:52.117 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:51 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3741196601' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:23:52.117 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:51 smithi114 ceph-mon[40867]: 4.e scrub starts 2024-09-21T12:23:52.117 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:51 smithi114 ceph-mon[40867]: 4.e scrub ok 2024-09-21T12:23:52.117 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:51 smithi114 ceph-mon[40867]: 6.1c scrub starts 2024-09-21T12:23:52.118 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:51 smithi114 ceph-mon[40867]: 6.1c scrub ok 2024-09-21T12:23:52.118 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:51 smithi114 ceph-mon[40867]: 6.14 scrub starts 2024-09-21T12:23:52.118 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:51 smithi114 ceph-mon[40867]: 6.14 scrub ok 2024-09-21T12:23:52.118 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:51 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1852010414' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:23:52.130 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:52.154 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:52.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:52.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:52.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:51 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3741196601' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:23:52.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:51 smithi183 ceph-mon[33277]: 4.e scrub starts 2024-09-21T12:23:52.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:51 smithi183 ceph-mon[33277]: 4.e scrub ok 2024-09-21T12:23:52.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:51 smithi183 ceph-mon[33277]: 6.1c scrub starts 2024-09-21T12:23:52.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:51 smithi183 ceph-mon[33277]: 6.1c scrub ok 2024-09-21T12:23:52.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:51 smithi183 ceph-mon[33277]: 6.14 scrub starts 2024-09-21T12:23:52.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:51 smithi183 ceph-mon[33277]: 6.14 scrub ok 2024-09-21T12:23:52.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:51 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1852010414' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:23:52.256 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:52.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:52.295 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:52.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:52.396 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:52.433 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:52.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:52.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:52.535 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:52.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:52.573 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:52.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:52.674 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:52.689 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:23:52 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:23:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:23:52.698 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:52.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:52.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:52.800 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:52.835 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:52.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:52.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:52.937 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:52.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:52 smithi114 ceph-mon[30324]: 4.10 scrub starts 2024-09-21T12:23:52.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:52 smithi114 ceph-mon[30324]: 4.10 scrub ok 2024-09-21T12:23:52.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:52 smithi114 ceph-mon[30324]: pgmap v163: 158 pgs: 3 peering, 155 active+clean; 3.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.23k op/s; 36 B/s, 0 objects/s recovering 2024-09-21T12:23:52.974 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:52 smithi114 ceph-mon[30324]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:23:52.974 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:52 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1852010414' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:23:52.974 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:52 smithi114 ceph-mon[30324]: osdmap e143: 8 total, 8 up, 7 in 2024-09-21T12:23:52.974 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:52 smithi114 ceph-mon[30324]: 3.a scrub starts 2024-09-21T12:23:52.974 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:52 smithi114 ceph-mon[30324]: 3.a scrub ok 2024-09-21T12:23:52.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:52 smithi114 ceph-mon[40867]: 4.10 scrub starts 2024-09-21T12:23:52.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:52 smithi114 ceph-mon[40867]: 4.10 scrub ok 2024-09-21T12:23:52.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:52 smithi114 ceph-mon[40867]: pgmap v163: 158 pgs: 3 peering, 155 active+clean; 3.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.23k op/s; 36 B/s, 0 objects/s recovering 2024-09-21T12:23:52.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:52 smithi114 ceph-mon[40867]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:23:52.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:52 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1852010414' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:23:52.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:52 smithi114 ceph-mon[40867]: osdmap e143: 8 total, 8 up, 7 in 2024-09-21T12:23:52.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:52 smithi114 ceph-mon[40867]: 3.a scrub starts 2024-09-21T12:23:52.975 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:52 smithi114 ceph-mon[40867]: 3.a scrub ok 2024-09-21T12:23:52.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:52.985 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:52.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:53.085 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:53.108 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:53.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:53.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:53.210 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:53.238 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:53.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:52 smithi183 ceph-mon[33277]: 4.10 scrub starts 2024-09-21T12:23:53.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:52 smithi183 ceph-mon[33277]: 4.10 scrub ok 2024-09-21T12:23:53.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:52 smithi183 ceph-mon[33277]: pgmap v163: 158 pgs: 3 peering, 155 active+clean; 3.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.23k op/s; 36 B/s, 0 objects/s recovering 2024-09-21T12:23:53.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:52 smithi183 ceph-mon[33277]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:23:53.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:52 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1852010414' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:23:53.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:52 smithi183 ceph-mon[33277]: osdmap e143: 8 total, 8 up, 7 in 2024-09-21T12:23:53.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:52 smithi183 ceph-mon[33277]: 3.a scrub starts 2024-09-21T12:23:53.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:52 smithi183 ceph-mon[33277]: 3.a scrub ok 2024-09-21T12:23:53.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:53.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:53.340 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:53.367 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:53.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:53.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:53.468 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:53.492 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:53.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:53.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:53.594 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:53.633 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:53.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:53.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:53.735 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:53.766 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:53.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:53.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:53.868 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:53.891 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:53.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:53.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:53.921 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T12:23:53.992 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:54.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:54.026 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:54.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:54.127 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:54.170 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:54.171 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:23:54.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:54.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:54.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:53 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2929004597' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:23:54.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:53 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:23:54.252 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:53 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2929004597' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:23:54.253 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:53 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:23:54.253 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:53 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2929004597' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:23:54.253 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:53 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:23:54.273 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:54.297 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:54.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:54.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:54.325 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:23:54.325 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 20, 2024-09-21T12:23:54.325 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 600, 2024-09-21T12:23:54.325 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-21T12:23:54.326 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.326 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:24383 7.8 7:1f298c15:::benchmark_data_smithi114_78432_object24382:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T12:23:54.326 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:39.998832+0000", 2024-09-21T12:23:54.326 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 134.305840905, 2024-09-21T12:23:54.326 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0658084639999998, 2024-09-21T12:23:54.326 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:54.326 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:54.326 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:23:54.326 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:23:54.326 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:23:54.326 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:23:54.326 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 24383 2024-09-21T12:23:54.326 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.327 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:54.327 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.327 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:54.327 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.998832+0000", 2024-09-21T12:23:54.327 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.327 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.327 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.327 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:54.327 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.998832+0000", 2024-09-21T12:23:54.327 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.327 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.327 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.327 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:54.327 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.998833+0000", 2024-09-21T12:23:54.328 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.37e-06 2024-09-21T12:23:54.328 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.328 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.328 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:54.328 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.998855+0000", 2024-09-21T12:23:54.328 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1846999999999999e-05 2024-09-21T12:23:54.328 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.328 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.328 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:54.328 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.998866+0000", 2024-09-21T12:23:54.328 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0363999999999999e-05 2024-09-21T12:23:54.328 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.328 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.328 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:54.329 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.998873+0000", 2024-09-21T12:23:54.329 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2110000000000001e-06 2024-09-21T12:23:54.329 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.329 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.329 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.329 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.998970+0000", 2024-09-21T12:23:54.329 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6693000000000004e-05 2024-09-21T12:23:54.329 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.329 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.329 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:54.329 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.999115+0000", 2024-09-21T12:23:54.329 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000145048 2024-09-21T12:23:54.329 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.329 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.329 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,7", 2024-09-21T12:23:54.330 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.999246+0000", 2024-09-21T12:23:54.330 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000131068 2024-09-21T12:23:54.330 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.330 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.330 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:23:54.330 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.002347+0000", 2024-09-21T12:23:54.330 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0031012959999999999 2024-09-21T12:23:54.330 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.330 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.330 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.330 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.002577+0000", 2024-09-21T12:23:54.330 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000229769 2024-09-21T12:23:54.330 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.330 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.331 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.331 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.010579+0000", 2024-09-21T12:23:54.331 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0080019189999999997 2024-09-21T12:23:54.331 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.331 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.331 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:23:54.331 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.010595+0000", 2024-09-21T12:23:54.331 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6161e-05 2024-09-21T12:23:54.331 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.331 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.331 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.331 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.020743+0000", 2024-09-21T12:23:54.331 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0101476890000001 2024-09-21T12:23:54.331 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.332 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.332 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:23:54.332 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.020791+0000", 2024-09-21T12:23:54.332 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.795e-05 2024-09-21T12:23:54.332 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.332 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.332 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.332 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.064509+0000", 2024-09-21T12:23:54.332 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0437187219999999 2024-09-21T12:23:54.332 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.332 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.332 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:23:54.332 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.064641+0000", 2024-09-21T12:23:54.332 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000131357 2024-09-21T12:23:54.332 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.333 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:54.333 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.333 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.333 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.333 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:24385 7.8 7:16dcbae5:::benchmark_data_smithi114_78432_object24384:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T12:23:54.333 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:39.998892+0000", 2024-09-21T12:23:54.333 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 134.30578058500001, 2024-09-21T12:23:54.333 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0668990809999999, 2024-09-21T12:23:54.333 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:54.333 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:54.333 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:23:54.333 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:23:54.333 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:23:54.333 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:23:54.334 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 24385 2024-09-21T12:23:54.334 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.334 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:54.334 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.334 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:54.334 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.998892+0000", 2024-09-21T12:23:54.334 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.334 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.334 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.334 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:54.334 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.998892+0000", 2024-09-21T12:23:54.334 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.334 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.335 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.335 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:54.335 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.998893+0000", 2024-09-21T12:23:54.335 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.06e-07 2024-09-21T12:23:54.335 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.335 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.335 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:54.335 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.998918+0000", 2024-09-21T12:23:54.335 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5378999999999999e-05 2024-09-21T12:23:54.335 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.335 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.335 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:54.335 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.998927+0000", 2024-09-21T12:23:54.335 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.3909999999999998e-06 2024-09-21T12:23:54.335 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.336 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.336 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:54.336 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.998932+0000", 2024-09-21T12:23:54.336 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6860000000000003e-06 2024-09-21T12:23:54.336 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.336 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.336 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.336 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.000469+0000", 2024-09-21T12:23:54.336 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001536716 2024-09-21T12:23:54.336 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.336 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.336 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:54.336 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.000583+0000", 2024-09-21T12:23:54.336 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000113354 2024-09-21T12:23:54.336 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.337 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.337 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,7", 2024-09-21T12:23:54.337 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.000713+0000", 2024-09-21T12:23:54.337 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000130932 2024-09-21T12:23:54.337 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.337 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.337 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:23:54.337 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.002357+0000", 2024-09-21T12:23:54.337 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001643207 2024-09-21T12:23:54.337 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.337 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.337 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.337 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.006431+0000", 2024-09-21T12:23:54.337 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0040742169999999998 2024-09-21T12:23:54.338 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.338 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.338 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.338 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.010634+0000", 2024-09-21T12:23:54.338 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0042035079999999999 2024-09-21T12:23:54.338 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.338 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.338 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:23:54.338 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.010644+0000", 2024-09-21T12:23:54.338 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1249999999999999e-06 2024-09-21T12:23:54.338 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.338 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.338 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.338 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.020852+0000", 2024-09-21T12:23:54.339 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0102084520000001 2024-09-21T12:23:54.339 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.339 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.339 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:23:54.339 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.020866+0000", 2024-09-21T12:23:54.339 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4199000000000001e-05 2024-09-21T12:23:54.339 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.339 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.339 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.339 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.065675+0000", 2024-09-21T12:23:54.339 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0448089519999999 2024-09-21T12:23:54.339 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.339 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.339 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:23:54.340 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.065792+0000", 2024-09-21T12:23:54.340 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000116357 2024-09-21T12:23:54.340 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.340 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:54.340 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.340 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.340 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.340 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:24410 7.8 7:1e9f2e1e:::benchmark_data_smithi114_78432_object24409:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:23:54.340 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:40.041873+0000", 2024-09-21T12:23:54.340 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 134.26279979399999, 2024-09-21T12:23:54.340 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0306425940000001, 2024-09-21T12:23:54.340 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:54.341 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:54.341 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:23:54.341 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:23:54.341 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:23:54.341 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:23:54.341 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 24410 2024-09-21T12:23:54.341 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.341 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:54.341 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.341 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:54.341 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.041873+0000", 2024-09-21T12:23:54.341 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.341 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.342 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.342 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:54.342 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.041873+0000", 2024-09-21T12:23:54.342 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.342 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.342 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.342 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:54.342 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.041875+0000", 2024-09-21T12:23:54.342 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.42e-06 2024-09-21T12:23:54.342 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.342 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.342 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:54.342 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.041896+0000", 2024-09-21T12:23:54.342 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1631e-05 2024-09-21T12:23:54.343 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.343 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.343 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:54.343 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.041928+0000", 2024-09-21T12:23:54.343 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.1977000000000002e-05 2024-09-21T12:23:54.343 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.343 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.343 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:54.343 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.041937+0000", 2024-09-21T12:23:54.343 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1649999999999995e-06 2024-09-21T12:23:54.343 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.343 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.343 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.343 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.042006+0000", 2024-09-21T12:23:54.344 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8824000000000004e-05 2024-09-21T12:23:54.344 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.344 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.344 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:23:54.344 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.042015+0000", 2024-09-21T12:23:54.344 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0909999999999999e-06 2024-09-21T12:23:54.344 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.344 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.344 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.344 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.020897+0000", 2024-09-21T12:23:54.344 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97888188200000004 2024-09-21T12:23:54.344 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.344 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.344 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:23:54.344 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.020908+0000", 2024-09-21T12:23:54.344 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1007e-05 2024-09-21T12:23:54.345 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.345 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.345 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.345 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.066247+0000", 2024-09-21T12:23:54.345 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.045338791 2024-09-21T12:23:54.345 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.345 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.345 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:54.345 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.066399+0000", 2024-09-21T12:23:54.345 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000151681 2024-09-21T12:23:54.345 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.345 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.345 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:23:54.345 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.066635+0000", 2024-09-21T12:23:54.346 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00023616899999999999 2024-09-21T12:23:54.346 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.346 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.346 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.346 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.070382+0000", 2024-09-21T12:23:54.346 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003747205 2024-09-21T12:23:54.346 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.346 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.346 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:23:54.346 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.070401+0000", 2024-09-21T12:23:54.346 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8635e-05 2024-09-21T12:23:54.346 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.346 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.346 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.347 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.071452+0000", 2024-09-21T12:23:54.347 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0010513250000000001 2024-09-21T12:23:54.347 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.347 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.347 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:23:54.347 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.071575+0000", 2024-09-21T12:23:54.347 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000122526 2024-09-21T12:23:54.347 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.347 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.347 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:23:54.347 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.072516+0000", 2024-09-21T12:23:54.347 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00094126500000000005 2024-09-21T12:23:54.347 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.347 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:54.348 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.348 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.348 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.348 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:24415 7.8 7:16bdf497:::benchmark_data_smithi114_78432_object24414:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:23:54.348 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:40.045617+0000", 2024-09-21T12:23:54.348 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 134.25905645500001, 2024-09-21T12:23:54.348 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0278312839999999, 2024-09-21T12:23:54.348 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:54.348 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:54.348 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:23:54.348 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:23:54.348 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:23:54.348 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:23:54.348 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 24415 2024-09-21T12:23:54.348 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.349 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:54.349 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.349 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:54.349 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.045617+0000", 2024-09-21T12:23:54.349 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.349 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.349 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.349 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:54.349 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.045617+0000", 2024-09-21T12:23:54.349 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.349 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.349 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.349 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:54.349 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.045618+0000", 2024-09-21T12:23:54.350 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.578e-06 2024-09-21T12:23:54.350 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.350 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.350 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:54.350 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.045639+0000", 2024-09-21T12:23:54.350 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1047999999999999e-05 2024-09-21T12:23:54.350 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.350 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.350 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:54.350 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.045653+0000", 2024-09-21T12:23:54.350 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3849000000000001e-05 2024-09-21T12:23:54.350 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.350 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.351 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:54.351 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.045660+0000", 2024-09-21T12:23:54.351 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8839999999999997e-06 2024-09-21T12:23:54.351 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.351 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.351 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.351 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.045723+0000", 2024-09-21T12:23:54.351 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2618000000000003e-05 2024-09-21T12:23:54.351 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.351 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.351 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:23:54.351 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.045732+0000", 2024-09-21T12:23:54.351 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7669999999999998e-06 2024-09-21T12:23:54.351 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.352 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.352 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.352 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.022590+0000", 2024-09-21T12:23:54.352 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97685747999999994 2024-09-21T12:23:54.352 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.352 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.352 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:23:54.352 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.022599+0000", 2024-09-21T12:23:54.352 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.4669999999999993e-06 2024-09-21T12:23:54.352 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.352 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.352 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.352 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.068684+0000", 2024-09-21T12:23:54.352 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0460851920000001 2024-09-21T12:23:54.352 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.353 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.353 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:54.353 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.068838+0000", 2024-09-21T12:23:54.353 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015381600000000001 2024-09-21T12:23:54.353 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.353 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.353 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:23:54.353 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.069029+0000", 2024-09-21T12:23:54.353 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000191227 2024-09-21T12:23:54.353 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.353 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.353 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:23:54.353 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.070406+0000", 2024-09-21T12:23:54.353 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0013769710000000001 2024-09-21T12:23:54.354 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.354 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.354 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.354 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.072571+0000", 2024-09-21T12:23:54.354 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0021641730000000001 2024-09-21T12:23:54.354 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.354 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.354 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.354 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.072668+0000", 2024-09-21T12:23:54.354 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7110999999999999e-05 2024-09-21T12:23:54.354 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.354 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.354 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:23:54.354 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.072724+0000", 2024-09-21T12:23:54.355 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6276000000000001e-05 2024-09-21T12:23:54.355 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.355 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.355 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:23:54.355 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.073448+0000", 2024-09-21T12:23:54.355 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00072382700000000002 2024-09-21T12:23:54.355 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.355 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:54.355 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.355 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.355 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.355 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:24791 7.8 7:153a621d:::benchmark_data_smithi114_78432_object24790:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T12:23:54.355 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:42.432155+0000", 2024-09-21T12:23:54.355 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 131.87251783299999, 2024-09-21T12:23:54.355 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.159980077, 2024-09-21T12:23:54.356 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:54.356 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:54.356 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:23:54.356 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:23:54.356 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:23:54.356 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:23:54.356 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 24791 2024-09-21T12:23:54.356 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.356 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:54.356 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.356 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:54.356 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.432155+0000", 2024-09-21T12:23:54.356 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.356 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.356 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.357 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:54.357 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.432155+0000", 2024-09-21T12:23:54.357 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.357 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.357 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.357 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:54.357 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.432157+0000", 2024-09-21T12:23:54.357 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8959999999999999e-06 2024-09-21T12:23:54.357 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.357 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.357 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:54.357 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.432179+0000", 2024-09-21T12:23:54.357 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1662999999999999e-05 2024-09-21T12:23:54.357 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.358 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.358 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:54.358 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.432189+0000", 2024-09-21T12:23:54.358 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.061e-05 2024-09-21T12:23:54.358 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.358 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.358 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:54.358 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.432196+0000", 2024-09-21T12:23:54.358 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3679999999999998e-06 2024-09-21T12:23:54.358 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.358 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.358 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.358 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.442074+0000", 2024-09-21T12:23:54.358 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0098782109999999996 2024-09-21T12:23:54.358 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.359 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.359 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:54.359 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.442218+0000", 2024-09-21T12:23:54.359 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014386499999999999 2024-09-21T12:23:54.359 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.359 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.359 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:23:54.359 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.442384+0000", 2024-09-21T12:23:54.359 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016588099999999999 2024-09-21T12:23:54.359 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.359 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.360 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.360 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.443960+0000", 2024-09-21T12:23:54.360 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0015763730000000001 2024-09-21T12:23:54.360 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.360 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.360 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:23:54.360 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.444343+0000", 2024-09-21T12:23:54.360 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00038294799999999998 2024-09-21T12:23:54.360 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.360 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.360 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.360 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.591068+0000", 2024-09-21T12:23:54.361 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.146724875 2024-09-21T12:23:54.361 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.361 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.361 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:23:54.361 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.591137+0000", 2024-09-21T12:23:54.361 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9438999999999997e-05 2024-09-21T12:23:54.361 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.361 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.361 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:23:54.361 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.592135+0000", 2024-09-21T12:23:54.361 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00099794800000000002 2024-09-21T12:23:54.361 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.361 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:54.361 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.362 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.362 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.362 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:45502 7.e 7:71c90ed0:::benchmark_data_smithi114_78432_object45501:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T12:23:54.362 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:59.294060+0000", 2024-09-21T12:23:54.362 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 115.010613061, 2024-09-21T12:23:54.362 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.16001768, 2024-09-21T12:23:54.362 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:54.362 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:54.362 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:23:54.362 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:23:54.362 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:23:54.362 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:23:54.362 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 45502 2024-09-21T12:23:54.362 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.362 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:54.363 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.363 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:54.363 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.294060+0000", 2024-09-21T12:23:54.363 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.363 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.363 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.363 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:54.363 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.294060+0000", 2024-09-21T12:23:54.363 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.363 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.363 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.363 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:54.363 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.294061+0000", 2024-09-21T12:23:54.363 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4440000000000001e-06 2024-09-21T12:23:54.363 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.364 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.364 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:54.364 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.294082+0000", 2024-09-21T12:23:54.364 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0598000000000002e-05 2024-09-21T12:23:54.364 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.364 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.364 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:54.364 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.294093+0000", 2024-09-21T12:23:54.364 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1158000000000001e-05 2024-09-21T12:23:54.364 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.364 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.364 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:54.365 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.294101+0000", 2024-09-21T12:23:54.365 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3499999999999999e-06 2024-09-21T12:23:54.365 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.365 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.365 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.365 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.294153+0000", 2024-09-21T12:23:54.365 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2098000000000002e-05 2024-09-21T12:23:54.365 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.365 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.365 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:54.365 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.294271+0000", 2024-09-21T12:23:54.366 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000118664 2024-09-21T12:23:54.366 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.366 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.366 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,5", 2024-09-21T12:23:54.366 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.294397+0000", 2024-09-21T12:23:54.366 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000125542 2024-09-21T12:23:54.367 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.367 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.367 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.367 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.296815+0000", 2024-09-21T12:23:54.367 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002417874 2024-09-21T12:23:54.367 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.367 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.368 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:23:54.368 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.296826+0000", 2024-09-21T12:23:54.368 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1321999999999999e-05 2024-09-21T12:23:54.368 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.368 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.368 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.368 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.453830+0000", 2024-09-21T12:23:54.368 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.15700447300000001 2024-09-21T12:23:54.368 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.368 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.368 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:23:54.368 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.454033+0000", 2024-09-21T12:23:54.368 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000202664 2024-09-21T12:23:54.368 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.369 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.369 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:23:54.369 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.454078+0000", 2024-09-21T12:23:54.369 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4493000000000002e-05 2024-09-21T12:23:54.369 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.369 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:54.369 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.369 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.369 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.369 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:45506 7.8 7:1157a2d2:::benchmark_data_smithi114_78432_object45505:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T12:23:54.369 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:59.297399+0000", 2024-09-21T12:23:54.369 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 115.007273525, 2024-09-21T12:23:54.369 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.160065606, 2024-09-21T12:23:54.370 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:54.370 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:54.370 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:23:54.370 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:23:54.370 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:23:54.370 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:23:54.370 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 45506 2024-09-21T12:23:54.370 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.370 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:54.370 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.370 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:54.370 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.297399+0000", 2024-09-21T12:23:54.370 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.370 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.371 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.371 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:54.371 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.297399+0000", 2024-09-21T12:23:54.371 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.371 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.371 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.371 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:54.371 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.297401+0000", 2024-09-21T12:23:54.371 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7969999999999999e-06 2024-09-21T12:23:54.371 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.371 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.371 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:54.371 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.297427+0000", 2024-09-21T12:23:54.371 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.6075e-05 2024-09-21T12:23:54.372 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.372 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.372 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:54.372 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.297440+0000", 2024-09-21T12:23:54.372 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.252e-05 2024-09-21T12:23:54.372 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.372 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.372 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:54.372 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.297447+0000", 2024-09-21T12:23:54.372 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2980000000000003e-06 2024-09-21T12:23:54.372 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.372 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.372 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.372 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.297599+0000", 2024-09-21T12:23:54.372 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015153599999999999 2024-09-21T12:23:54.373 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.373 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.373 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:54.373 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.297724+0000", 2024-09-21T12:23:54.373 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000125148 2024-09-21T12:23:54.373 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.373 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.373 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:23:54.373 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.297871+0000", 2024-09-21T12:23:54.373 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000147505 2024-09-21T12:23:54.373 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.373 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.373 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.374 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.301568+0000", 2024-09-21T12:23:54.374 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0036965230000000002 2024-09-21T12:23:54.374 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.374 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.374 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:23:54.374 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.301639+0000", 2024-09-21T12:23:54.374 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0963999999999993e-05 2024-09-21T12:23:54.374 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.374 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.374 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.374 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.456770+0000", 2024-09-21T12:23:54.374 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.15513142099999999 2024-09-21T12:23:54.374 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.374 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.374 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:23:54.375 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.456833+0000", 2024-09-21T12:23:54.375 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2525000000000002e-05 2024-09-21T12:23:54.375 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.375 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.375 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:23:54.375 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.457465+0000", 2024-09-21T12:23:54.375 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00063229400000000004 2024-09-21T12:23:54.375 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.375 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:54.375 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.375 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.375 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.375 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:45509 7.8 7:1989e657:::benchmark_data_smithi114_78432_object45508:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T12:23:54.375 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:59.297470+0000", 2024-09-21T12:23:54.376 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 115.007203327, 2024-09-21T12:23:54.376 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.164477763, 2024-09-21T12:23:54.376 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:54.376 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:54.376 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:23:54.376 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:23:54.376 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:23:54.376 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:23:54.376 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 45509 2024-09-21T12:23:54.376 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.376 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:54.376 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.376 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:54.377 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.297470+0000", 2024-09-21T12:23:54.377 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.377 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.377 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.377 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:54.377 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.297470+0000", 2024-09-21T12:23:54.377 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.377 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.377 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.377 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:54.377 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.297470+0000", 2024-09-21T12:23:54.377 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3700000000000001e-07 2024-09-21T12:23:54.377 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.377 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.377 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:54.377 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.297493+0000", 2024-09-21T12:23:54.378 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2374e-05 2024-09-21T12:23:54.378 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.378 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.378 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:54.378 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.297499+0000", 2024-09-21T12:23:54.378 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9870000000000002e-06 2024-09-21T12:23:54.378 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.378 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.378 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:54.378 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.297505+0000", 2024-09-21T12:23:54.378 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.074e-06 2024-09-21T12:23:54.378 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.378 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.379 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.379 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.299566+0000", 2024-09-21T12:23:54.379 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0020616940000000002 2024-09-21T12:23:54.379 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.379 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.379 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:54.379 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.299708+0000", 2024-09-21T12:23:54.379 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014215500000000001 2024-09-21T12:23:54.379 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.379 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.379 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:23:54.379 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.299865+0000", 2024-09-21T12:23:54.379 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000156704 2024-09-21T12:23:54.379 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.379 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.380 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.380 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.302117+0000", 2024-09-21T12:23:54.380 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002251739 2024-09-21T12:23:54.380 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.380 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.380 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:23:54.380 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.302132+0000", 2024-09-21T12:23:54.380 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5095e-05 2024-09-21T12:23:54.380 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.380 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.380 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.380 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.461162+0000", 2024-09-21T12:23:54.380 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.159030112 2024-09-21T12:23:54.380 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.381 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.381 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:23:54.381 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.461226+0000", 2024-09-21T12:23:54.381 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4228999999999998e-05 2024-09-21T12:23:54.381 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.381 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.381 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:23:54.381 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.461947+0000", 2024-09-21T12:23:54.381 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00072116300000000003 2024-09-21T12:23:54.381 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.381 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:54.381 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.381 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.381 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.382 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:45516 7.e 7:78799872:::benchmark_data_smithi114_78432_object45515:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T12:23:54.382 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:59.304654+0000", 2024-09-21T12:23:54.382 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 115.000018676, 2024-09-21T12:23:54.382 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.15534616300000001, 2024-09-21T12:23:54.382 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:54.382 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:54.382 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:23:54.382 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:23:54.382 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:23:54.382 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:23:54.382 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 45516 2024-09-21T12:23:54.382 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.382 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:54.383 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.383 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:54.383 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.304654+0000", 2024-09-21T12:23:54.383 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.383 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.383 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.383 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:54.383 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.304654+0000", 2024-09-21T12:23:54.383 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.383 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.383 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.383 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:54.383 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.304656+0000", 2024-09-21T12:23:54.384 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9139999999999998e-06 2024-09-21T12:23:54.384 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.384 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.384 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:54.384 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.304674+0000", 2024-09-21T12:23:54.384 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7822999999999999e-05 2024-09-21T12:23:54.384 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.384 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.384 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:54.384 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.304684+0000", 2024-09-21T12:23:54.384 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.8830000000000001e-06 2024-09-21T12:23:54.384 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.384 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.384 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:54.385 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.304689+0000", 2024-09-21T12:23:54.385 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.446e-06 2024-09-21T12:23:54.385 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.385 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.385 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.385 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.304743+0000", 2024-09-21T12:23:54.385 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3708000000000001e-05 2024-09-21T12:23:54.385 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.385 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.385 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:54.385 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.304821+0000", 2024-09-21T12:23:54.385 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7923000000000006e-05 2024-09-21T12:23:54.385 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.385 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.386 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,5", 2024-09-21T12:23:54.386 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.304900+0000", 2024-09-21T12:23:54.386 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8699e-05 2024-09-21T12:23:54.386 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.386 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.386 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:23:54.386 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.306078+0000", 2024-09-21T12:23:54.386 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00117793 2024-09-21T12:23:54.386 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.386 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.386 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.386 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.306379+0000", 2024-09-21T12:23:54.386 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00030152999999999999 2024-09-21T12:23:54.386 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.387 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.387 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.387 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.459763+0000", 2024-09-21T12:23:54.387 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.153383995 2024-09-21T12:23:54.387 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.387 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.387 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:23:54.387 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.459954+0000", 2024-09-21T12:23:54.387 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00019092100000000001 2024-09-21T12:23:54.387 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.387 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.387 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:23:54.387 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.460001+0000", 2024-09-21T12:23:54.387 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6390999999999997e-05 2024-09-21T12:23:54.388 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.388 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:54.388 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.388 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.388 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.388 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46523 7.8 7:11c4ba2e:::benchmark_data_smithi114_78432_object46522:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:23:54.388 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.793498+0000", 2024-09-21T12:23:54.388 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 112.511174719, 2024-09-21T12:23:54.388 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0224531560000001, 2024-09-21T12:23:54.388 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:54.388 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:54.388 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:23:54.388 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:23:54.388 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:23:54.388 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:23:54.389 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46523 2024-09-21T12:23:54.389 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.389 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:54.389 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.389 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:54.389 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.793498+0000", 2024-09-21T12:23:54.389 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.389 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.389 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.389 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:54.389 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.793498+0000", 2024-09-21T12:23:54.389 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.389 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.389 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.390 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:54.390 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.793500+0000", 2024-09-21T12:23:54.390 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9379999999999999e-06 2024-09-21T12:23:54.390 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.390 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.390 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:54.390 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.793526+0000", 2024-09-21T12:23:54.390 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5816000000000002e-05 2024-09-21T12:23:54.390 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.390 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.390 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:54.391 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.793544+0000", 2024-09-21T12:23:54.391 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7889000000000001e-05 2024-09-21T12:23:54.391 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.391 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.391 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:54.391 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.793552+0000", 2024-09-21T12:23:54.391 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.3159999999999997e-06 2024-09-21T12:23:54.391 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.391 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.391 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.392 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.793610+0000", 2024-09-21T12:23:54.392 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8180999999999997e-05 2024-09-21T12:23:54.392 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.392 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.392 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:23:54.392 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.793617+0000", 2024-09-21T12:23:54.392 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.691e-06 2024-09-21T12:23:54.392 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.392 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.392 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.392 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.811363+0000", 2024-09-21T12:23:54.393 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.01774579 2024-09-21T12:23:54.393 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.393 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.393 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:54.393 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.811475+0000", 2024-09-21T12:23:54.393 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000111846 2024-09-21T12:23:54.393 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.393 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.393 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:23:54.393 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.811637+0000", 2024-09-21T12:23:54.393 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000162268 2024-09-21T12:23:54.393 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.393 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.393 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.393 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.815751+0000", 2024-09-21T12:23:54.394 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0041139089999999998 2024-09-21T12:23:54.394 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.394 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.394 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.394 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.815826+0000", 2024-09-21T12:23:54.394 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4887000000000003e-05 2024-09-21T12:23:54.394 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.394 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.394 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:23:54.394 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.815842+0000", 2024-09-21T12:23:54.394 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6518999999999999e-05 2024-09-21T12:23:54.394 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.394 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.394 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:23:54.395 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.815905+0000", 2024-09-21T12:23:54.395 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2515999999999996e-05 2024-09-21T12:23:54.395 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.395 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.395 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:23:54.395 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.815951+0000", 2024-09-21T12:23:54.395 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6589999999999999e-05 2024-09-21T12:23:54.395 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.395 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:54.395 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.395 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.395 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.395 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46531 7.8 7:1bc837da:::benchmark_data_smithi114_78432_object46530:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:23:54.396 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.804524+0000", 2024-09-21T12:23:54.396 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 112.50014907400001, 2024-09-21T12:23:54.396 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0155765649999999, 2024-09-21T12:23:54.396 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:54.396 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:54.396 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:23:54.396 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:23:54.396 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:23:54.396 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:23:54.396 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46531 2024-09-21T12:23:54.396 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.396 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:54.396 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.396 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:54.396 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.804524+0000", 2024-09-21T12:23:54.397 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.397 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.397 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.397 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:54.397 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.804524+0000", 2024-09-21T12:23:54.397 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.397 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.397 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.397 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:54.397 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.804526+0000", 2024-09-21T12:23:54.397 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6139999999999999e-06 2024-09-21T12:23:54.397 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.397 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.397 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:54.398 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.804558+0000", 2024-09-21T12:23:54.398 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2119000000000002e-05 2024-09-21T12:23:54.398 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.398 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.398 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:54.398 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.804578+0000", 2024-09-21T12:23:54.398 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.001e-05 2024-09-21T12:23:54.398 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.398 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.398 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:54.398 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.804586+0000", 2024-09-21T12:23:54.398 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1210000000000007e-06 2024-09-21T12:23:54.398 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.398 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.399 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.399 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.804675+0000", 2024-09-21T12:23:54.399 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9492000000000003e-05 2024-09-21T12:23:54.399 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.399 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.399 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:23:54.399 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.804685+0000", 2024-09-21T12:23:54.399 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.9529999999999994e-06 2024-09-21T12:23:54.399 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.399 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.399 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.399 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.813729+0000", 2024-09-21T12:23:54.399 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0090438020000001 2024-09-21T12:23:54.399 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.399 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.400 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:54.400 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.813870+0000", 2024-09-21T12:23:54.400 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014118099999999999 2024-09-21T12:23:54.400 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.400 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.400 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:23:54.400 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.814037+0000", 2024-09-21T12:23:54.400 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016691199999999999 2024-09-21T12:23:54.400 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.400 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.400 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.400 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.816120+0000", 2024-09-21T12:23:54.400 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0020824039999999999 2024-09-21T12:23:54.401 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.401 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.401 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:23:54.401 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.817843+0000", 2024-09-21T12:23:54.401 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0017235390000000001 2024-09-21T12:23:54.401 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.401 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.401 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.401 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.819999+0000", 2024-09-21T12:23:54.401 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0021563659999999998 2024-09-21T12:23:54.401 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.401 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.401 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:23:54.402 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.820050+0000", 2024-09-21T12:23:54.402 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0439999999999998e-05 2024-09-21T12:23:54.402 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.402 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.402 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:23:54.402 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.820101+0000", 2024-09-21T12:23:54.402 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0612000000000003e-05 2024-09-21T12:23:54.402 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.402 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:54.402 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.402 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.403 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.403 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46554 7.8 7:1ef36561:::benchmark_data_smithi114_78432_object46553:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:23:54.403 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.823773+0000", 2024-09-21T12:23:54.403 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 112.480900428, 2024-09-21T12:23:54.403 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99858891699999996, 2024-09-21T12:23:54.403 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:54.403 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:54.403 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:23:54.403 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:23:54.403 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:23:54.403 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:23:54.403 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46554 2024-09-21T12:23:54.404 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.404 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:54.404 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.404 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:54.404 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.823773+0000", 2024-09-21T12:23:54.404 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.404 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.404 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.404 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:54.404 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.823773+0000", 2024-09-21T12:23:54.404 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.404 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.404 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.404 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:54.404 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.823774+0000", 2024-09-21T12:23:54.405 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.077e-06 2024-09-21T12:23:54.405 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.405 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.405 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:54.405 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.823797+0000", 2024-09-21T12:23:54.405 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3478000000000001e-05 2024-09-21T12:23:54.405 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.405 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.405 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:54.405 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.823811+0000", 2024-09-21T12:23:54.405 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3878e-05 2024-09-21T12:23:54.405 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.405 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.405 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:54.405 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.823819+0000", 2024-09-21T12:23:54.405 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9810000000000003e-06 2024-09-21T12:23:54.405 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.406 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.406 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.406 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.823933+0000", 2024-09-21T12:23:54.406 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000113807 2024-09-21T12:23:54.406 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.406 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.406 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:23:54.406 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.823942+0000", 2024-09-21T12:23:54.406 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.5459999999999997e-06 2024-09-21T12:23:54.406 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.406 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.406 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.406 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.816164+0000", 2024-09-21T12:23:54.406 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99222121799999996 2024-09-21T12:23:54.406 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.407 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.407 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:54.407 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.816287+0000", 2024-09-21T12:23:54.407 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012309699999999999 2024-09-21T12:23:54.407 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.407 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.407 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:23:54.407 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.816419+0000", 2024-09-21T12:23:54.407 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013215199999999999 2024-09-21T12:23:54.407 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.407 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.407 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.407 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.820187+0000", 2024-09-21T12:23:54.407 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003768633 2024-09-21T12:23:54.407 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.407 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.408 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.408 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.820256+0000", 2024-09-21T12:23:54.408 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8388999999999999e-05 2024-09-21T12:23:54.408 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.408 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.408 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:23:54.408 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.822263+0000", 2024-09-21T12:23:54.408 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0020074030000000001 2024-09-21T12:23:54.408 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.408 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.408 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:23:54.408 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.822317+0000", 2024-09-21T12:23:54.408 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4095999999999998e-05 2024-09-21T12:23:54.409 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.409 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.409 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:23:54.409 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.822362+0000", 2024-09-21T12:23:54.409 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4162000000000002e-05 2024-09-21T12:23:54.409 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.409 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:54.409 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.409 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.409 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.409 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46568 7.8 7:1389cba8:::benchmark_data_smithi114_78432_object46567:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:23:54.409 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.834184+0000", 2024-09-21T12:23:54.409 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 112.470488702, 2024-09-21T12:23:54.409 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99022050800000005, 2024-09-21T12:23:54.410 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:54.410 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:54.410 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:23:54.410 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:23:54.410 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:23:54.410 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:23:54.410 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46568 2024-09-21T12:23:54.410 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.410 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:54.410 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.410 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:54.410 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.834184+0000", 2024-09-21T12:23:54.410 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.410 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.411 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.411 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:54.411 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.834184+0000", 2024-09-21T12:23:54.411 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.411 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.411 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.411 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:54.411 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.834186+0000", 2024-09-21T12:23:54.411 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9410000000000001e-06 2024-09-21T12:23:54.411 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.411 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.411 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:54.411 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.834208+0000", 2024-09-21T12:23:54.411 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2039999999999999e-05 2024-09-21T12:23:54.412 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.412 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.412 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:54.412 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.834225+0000", 2024-09-21T12:23:54.412 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6875999999999999e-05 2024-09-21T12:23:54.412 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.412 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.412 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:54.412 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.834233+0000", 2024-09-21T12:23:54.412 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8199999999999997e-06 2024-09-21T12:23:54.412 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.412 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.413 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.413 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.834320+0000", 2024-09-21T12:23:54.413 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.7083000000000006e-05 2024-09-21T12:23:54.413 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.413 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.413 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:23:54.413 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.834331+0000", 2024-09-21T12:23:54.413 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1090000000000001e-05 2024-09-21T12:23:54.413 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.413 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.413 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.413 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.817892+0000", 2024-09-21T12:23:54.413 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98356091400000001 2024-09-21T12:23:54.413 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.414 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.414 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:54.414 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.818008+0000", 2024-09-21T12:23:54.414 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000115606 2024-09-21T12:23:54.414 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.414 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.414 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:23:54.414 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.818251+0000", 2024-09-21T12:23:54.414 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00024378099999999999 2024-09-21T12:23:54.414 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.415 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.415 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:23:54.415 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.822369+0000", 2024-09-21T12:23:54.415 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0041175159999999999 2024-09-21T12:23:54.415 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.415 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.415 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.415 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.824256+0000", 2024-09-21T12:23:54.415 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0018866620000000001 2024-09-21T12:23:54.415 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.415 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.415 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.415 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.824304+0000", 2024-09-21T12:23:54.415 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8281000000000001e-05 2024-09-21T12:23:54.416 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.416 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.416 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:23:54.416 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.824361+0000", 2024-09-21T12:23:54.416 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6783000000000002e-05 2024-09-21T12:23:54.416 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.416 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.416 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:23:54.417 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.824405+0000", 2024-09-21T12:23:54.417 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4115e-05 2024-09-21T12:23:54.417 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.417 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:54.417 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.417 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.417 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.417 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46574 7.8 7:11204ae1:::benchmark_data_smithi114_78432_object46573:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:23:54.417 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.838311+0000", 2024-09-21T12:23:54.417 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 112.466361582, 2024-09-21T12:23:54.417 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99371615499999999, 2024-09-21T12:23:54.417 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:54.417 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:54.417 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:23:54.417 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:23:54.418 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:23:54.418 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:23:54.418 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46574 2024-09-21T12:23:54.418 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.418 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:54.418 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.418 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:54.418 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.838311+0000", 2024-09-21T12:23:54.418 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.418 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.418 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.418 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:54.418 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.838311+0000", 2024-09-21T12:23:54.418 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.419 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.419 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:54.419 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.838313+0000", 2024-09-21T12:23:54.419 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.854e-06 2024-09-21T12:23:54.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.419 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.419 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:54.419 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.838337+0000", 2024-09-21T12:23:54.419 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3574000000000002e-05 2024-09-21T12:23:54.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.419 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.419 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:54.419 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.838354+0000", 2024-09-21T12:23:54.419 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.702e-05 2024-09-21T12:23:54.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.420 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.420 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:54.420 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.838363+0000", 2024-09-21T12:23:54.420 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9150000000000002e-06 2024-09-21T12:23:54.420 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.420 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.420 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.420 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.838438+0000", 2024-09-21T12:23:54.420 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5680999999999996e-05 2024-09-21T12:23:54.421 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.421 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.421 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:23:54.421 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.838448+0000", 2024-09-21T12:23:54.421 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1409999999999994e-06 2024-09-21T12:23:54.421 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.421 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.421 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.421 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.820308+0000", 2024-09-21T12:23:54.421 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98186004500000001 2024-09-21T12:23:54.421 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.421 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.421 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:54.421 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.820415+0000", 2024-09-21T12:23:54.422 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010713 2024-09-21T12:23:54.422 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.422 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.422 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:23:54.422 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.820568+0000", 2024-09-21T12:23:54.422 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000152907 2024-09-21T12:23:54.422 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.422 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.422 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:23:54.422 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.824267+0000", 2024-09-21T12:23:54.423 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0036996249999999998 2024-09-21T12:23:54.423 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.423 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.423 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.423 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:54.425 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.828377+0000", 2024-09-21T12:23:54.425 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0041097740000000001 2024-09-21T12:23:54.425 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.425 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.425 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.426 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.831764+0000", 2024-09-21T12:23:54.426 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0033872970000000001 2024-09-21T12:23:54.426 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.426 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.426 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:23:54.426 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.831972+0000", 2024-09-21T12:23:54.426 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000207512 2024-09-21T12:23:54.426 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.426 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.426 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:23:54.427 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.832028+0000", 2024-09-21T12:23:54.427 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5680000000000002e-05 2024-09-21T12:23:54.427 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.427 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:54.427 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.427 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.427 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.427 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46597 7.8 7:113cfed4:::benchmark_data_smithi114_78432_object46596:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:23:54.427 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.859457+0000", 2024-09-21T12:23:54.427 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 112.44521622000001, 2024-09-21T12:23:54.427 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97467767900000002, 2024-09-21T12:23:54.427 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:54.428 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:54.428 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:23:54.428 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:23:54.428 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:23:54.428 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:23:54.428 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46597 2024-09-21T12:23:54.428 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.428 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:54.428 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.428 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:54.428 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.859457+0000", 2024-09-21T12:23:54.428 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.428 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.429 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.429 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:54.429 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.859457+0000", 2024-09-21T12:23:54.429 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.429 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.429 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.429 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:54.429 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.859458+0000", 2024-09-21T12:23:54.429 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.618e-06 2024-09-21T12:23:54.429 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.429 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.429 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:54.429 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.859479+0000", 2024-09-21T12:23:54.429 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0678000000000001e-05 2024-09-21T12:23:54.430 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.430 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.430 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:54.430 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.859501+0000", 2024-09-21T12:23:54.430 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1908000000000001e-05 2024-09-21T12:23:54.430 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.430 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.430 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:54.430 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.859508+0000", 2024-09-21T12:23:54.430 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4610000000000002e-06 2024-09-21T12:23:54.430 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.430 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.430 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.431 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.859569+0000", 2024-09-21T12:23:54.431 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0092e-05 2024-09-21T12:23:54.431 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.431 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.431 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:23:54.431 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.859575+0000", 2024-09-21T12:23:54.431 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8390000000000001e-06 2024-09-21T12:23:54.431 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.431 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.431 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.431 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.822397+0000", 2024-09-21T12:23:54.431 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.96282112500000006 2024-09-21T12:23:54.431 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.431 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.432 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:54.432 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.822504+0000", 2024-09-21T12:23:54.432 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010725700000000001 2024-09-21T12:23:54.432 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.432 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.432 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:23:54.432 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.822635+0000", 2024-09-21T12:23:54.432 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013144299999999999 2024-09-21T12:23:54.432 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.432 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.432 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:23:54.432 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.826386+0000", 2024-09-21T12:23:54.432 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0037504970000000002 2024-09-21T12:23:54.432 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.432 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.433 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.433 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.833905+0000", 2024-09-21T12:23:54.433 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0075191850000000003 2024-09-21T12:23:54.433 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.433 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.433 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.433 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.834031+0000", 2024-09-21T12:23:54.433 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000126091 2024-09-21T12:23:54.433 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.433 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.433 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:23:54.433 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.834088+0000", 2024-09-21T12:23:54.433 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6740999999999998e-05 2024-09-21T12:23:54.433 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.434 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.434 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:23:54.434 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.834134+0000", 2024-09-21T12:23:54.434 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6743999999999998e-05 2024-09-21T12:23:54.434 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.434 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:54.434 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.434 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.434 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.434 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46610 7.8 7:1a279a8a:::benchmark_data_smithi114_78432_object46609:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:23:54.434 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.873647+0000", 2024-09-21T12:23:54.435 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 112.431026067, 2024-09-21T12:23:54.435 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.96062798800000004, 2024-09-21T12:23:54.435 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:54.435 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:54.435 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:23:54.435 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:23:54.435 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:23:54.435 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:23:54.435 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46610 2024-09-21T12:23:54.435 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.435 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:54.435 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.435 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:54.435 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.873647+0000", 2024-09-21T12:23:54.435 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.436 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.436 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.436 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:54.436 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.873647+0000", 2024-09-21T12:23:54.436 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.436 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.436 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.436 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:54.436 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.873649+0000", 2024-09-21T12:23:54.436 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5999999999999999e-06 2024-09-21T12:23:54.436 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.436 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.436 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:54.436 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.873670+0000", 2024-09-21T12:23:54.436 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1336e-05 2024-09-21T12:23:54.437 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.437 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.437 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:54.437 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.873684+0000", 2024-09-21T12:23:54.437 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4185000000000001e-05 2024-09-21T12:23:54.437 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.437 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.437 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:54.437 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.873691+0000", 2024-09-21T12:23:54.437 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7429999999999996e-06 2024-09-21T12:23:54.437 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.437 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.437 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.437 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.873748+0000", 2024-09-21T12:23:54.437 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6804e-05 2024-09-21T12:23:54.438 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.438 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.438 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:23:54.438 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.873755+0000", 2024-09-21T12:23:54.438 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0700000000000001e-06 2024-09-21T12:23:54.438 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.438 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.438 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.438 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.824447+0000", 2024-09-21T12:23:54.438 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.95069197299999997 2024-09-21T12:23:54.438 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.438 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.438 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:54.438 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.824554+0000", 2024-09-21T12:23:54.438 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000107299 2024-09-21T12:23:54.439 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.439 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.439 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:23:54.439 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.824691+0000", 2024-09-21T12:23:54.439 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013733500000000001 2024-09-21T12:23:54.439 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.439 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.439 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:23:54.439 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.833924+0000", 2024-09-21T12:23:54.439 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0092329390000000008 2024-09-21T12:23:54.440 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.440 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.440 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.440 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.833971+0000", 2024-09-21T12:23:54.440 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6675000000000003e-05 2024-09-21T12:23:54.440 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.440 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.440 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.440 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.834188+0000", 2024-09-21T12:23:54.440 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000216987 2024-09-21T12:23:54.440 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.440 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.440 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:23:54.440 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.834240+0000", 2024-09-21T12:23:54.441 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2129999999999997e-05 2024-09-21T12:23:54.441 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.441 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.441 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:23:54.441 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.834275+0000", 2024-09-21T12:23:54.441 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4912e-05 2024-09-21T12:23:54.441 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.441 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:54.441 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.441 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.441 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.441 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46657 7.8 7:1a3a6588:::benchmark_data_smithi114_78432_object46656:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:23:54.441 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.970902+0000", 2024-09-21T12:23:54.441 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 112.333771376, 2024-09-21T12:23:54.441 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.86373514500000004, 2024-09-21T12:23:54.442 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:54.442 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:54.442 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:23:54.442 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:23:54.442 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:23:54.442 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:23:54.442 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46657 2024-09-21T12:23:54.442 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.442 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:54.442 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.442 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:54.442 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.970902+0000", 2024-09-21T12:23:54.442 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.443 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.443 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.443 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:54.443 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.970902+0000", 2024-09-21T12:23:54.443 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.443 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.443 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.443 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:54.443 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.970904+0000", 2024-09-21T12:23:54.443 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8700000000000001e-06 2024-09-21T12:23:54.443 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.443 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.443 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:54.443 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.970925+0000", 2024-09-21T12:23:54.444 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1316999999999999e-05 2024-09-21T12:23:54.444 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.444 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.444 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:54.444 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.970942+0000", 2024-09-21T12:23:54.444 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7574000000000001e-05 2024-09-21T12:23:54.444 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.444 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.444 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:54.444 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.970950+0000", 2024-09-21T12:23:54.444 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2080000000000003e-06 2024-09-21T12:23:54.444 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.444 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.445 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.445 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.971012+0000", 2024-09-21T12:23:54.445 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2797000000000002e-05 2024-09-21T12:23:54.445 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.445 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.445 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:23:54.445 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.971026+0000", 2024-09-21T12:23:54.445 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.314e-05 2024-09-21T12:23:54.445 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.445 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.445 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.445 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.826416+0000", 2024-09-21T12:23:54.446 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.85538999699999996 2024-09-21T12:23:54.446 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.446 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.446 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:54.446 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.826517+0000", 2024-09-21T12:23:54.446 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000101659 2024-09-21T12:23:54.446 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.446 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.446 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:23:54.446 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.826643+0000", 2024-09-21T12:23:54.446 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000126198 2024-09-21T12:23:54.446 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.446 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.446 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:23:54.447 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.833933+0000", 2024-09-21T12:23:54.447 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0072896000000000002 2024-09-21T12:23:54.447 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.447 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.447 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.447 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.834316+0000", 2024-09-21T12:23:54.447 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00038313900000000001 2024-09-21T12:23:54.447 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.447 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.447 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.447 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.834533+0000", 2024-09-21T12:23:54.447 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00021702000000000001 2024-09-21T12:23:54.447 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.447 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.448 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:23:54.448 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.834591+0000", 2024-09-21T12:23:54.448 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7979999999999997e-05 2024-09-21T12:23:54.448 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.448 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.448 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:23:54.448 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.834637+0000", 2024-09-21T12:23:54.448 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5646000000000003e-05 2024-09-21T12:23:54.448 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.448 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:54.448 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.448 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.448 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.448 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:49225 7.e 7:78039099:::benchmark_data_smithi114_78432_object49224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:23:54.448 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:04.569229+0000", 2024-09-21T12:23:54.449 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 109.735443628, 2024-09-21T12:23:54.449 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.14412950999999999, 2024-09-21T12:23:54.449 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:54.449 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:54.449 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:23:54.449 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:23:54.449 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:23:54.449 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:23:54.449 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 49225 2024-09-21T12:23:54.449 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.449 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:54.449 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.449 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:54.449 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.569229+0000", 2024-09-21T12:23:54.450 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.450 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.450 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.450 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:54.450 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.569229+0000", 2024-09-21T12:23:54.450 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.450 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.450 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.450 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:54.450 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.569235+0000", 2024-09-21T12:23:54.450 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1109999999999999e-06 2024-09-21T12:23:54.450 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.450 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.450 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:54.450 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.569276+0000", 2024-09-21T12:23:54.451 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1179e-05 2024-09-21T12:23:54.451 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.451 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.451 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:54.451 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.569292+0000", 2024-09-21T12:23:54.451 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6422999999999998e-05 2024-09-21T12:23:54.451 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.451 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.451 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:54.452 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.569309+0000", 2024-09-21T12:23:54.452 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6631000000000001e-05 2024-09-21T12:23:54.452 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.452 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.452 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.452 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.569383+0000", 2024-09-21T12:23:54.452 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4114000000000002e-05 2024-09-21T12:23:54.452 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.452 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.452 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:54.452 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.569522+0000", 2024-09-21T12:23:54.452 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013892 2024-09-21T12:23:54.452 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.453 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.453 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,5", 2024-09-21T12:23:54.453 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.569657+0000", 2024-09-21T12:23:54.453 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013499900000000001 2024-09-21T12:23:54.453 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.453 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.453 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.453 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.572520+0000", 2024-09-21T12:23:54.453 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0028635930000000002 2024-09-21T12:23:54.453 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.454 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.454 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:23:54.454 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.572542+0000", 2024-09-21T12:23:54.454 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1373999999999999e-05 2024-09-21T12:23:54.454 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.454 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.454 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.454 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.713243+0000", 2024-09-21T12:23:54.454 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.140700941 2024-09-21T12:23:54.454 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.454 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.454 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:23:54.454 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.713309+0000", 2024-09-21T12:23:54.455 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6767999999999995e-05 2024-09-21T12:23:54.455 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.455 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.455 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:23:54.455 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.713359+0000", 2024-09-21T12:23:54.455 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9456999999999998e-05 2024-09-21T12:23:54.455 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.455 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:54.455 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.455 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.455 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.456 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:49235 7.8 7:1f0dee80:::benchmark_data_smithi114_78432_object49234:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:23:54.456 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:04.574390+0000", 2024-09-21T12:23:54.456 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 109.73028345100001, 2024-09-21T12:23:54.456 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.141800871, 2024-09-21T12:23:54.456 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:54.456 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:54.456 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:23:54.456 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:23:54.456 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:23:54.456 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:23:54.456 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 49235 2024-09-21T12:23:54.456 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.457 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:54.457 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.457 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:54.457 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.574390+0000", 2024-09-21T12:23:54.457 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.457 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.457 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.457 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:54.457 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.574390+0000", 2024-09-21T12:23:54.457 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.457 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.457 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.457 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:54.457 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.574391+0000", 2024-09-21T12:23:54.458 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.753e-06 2024-09-21T12:23:54.458 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.458 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.458 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:54.458 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.574414+0000", 2024-09-21T12:23:54.458 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3071e-05 2024-09-21T12:23:54.458 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.458 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.458 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:54.459 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.574428+0000", 2024-09-21T12:23:54.459 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3309000000000001e-05 2024-09-21T12:23:54.459 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.459 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.459 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:54.459 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.574435+0000", 2024-09-21T12:23:54.459 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4050000000000003e-06 2024-09-21T12:23:54.459 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.459 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.459 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.460 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.574494+0000", 2024-09-21T12:23:54.460 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8749000000000003e-05 2024-09-21T12:23:54.460 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.460 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.460 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:54.460 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.574623+0000", 2024-09-21T12:23:54.460 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012902700000000001 2024-09-21T12:23:54.460 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.460 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.460 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:23:54.460 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.574749+0000", 2024-09-21T12:23:54.460 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000125897 2024-09-21T12:23:54.461 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.461 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.461 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.461 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.576774+0000", 2024-09-21T12:23:54.461 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002025542 2024-09-21T12:23:54.461 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.461 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.461 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:23:54.461 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.576879+0000", 2024-09-21T12:23:54.461 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000105079 2024-09-21T12:23:54.461 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.461 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.461 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.461 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.716089+0000", 2024-09-21T12:23:54.461 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.139209902 2024-09-21T12:23:54.461 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.462 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.462 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:23:54.462 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.716149+0000", 2024-09-21T12:23:54.462 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9413000000000002e-05 2024-09-21T12:23:54.462 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.462 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.462 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:23:54.462 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.716190+0000", 2024-09-21T12:23:54.462 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1724000000000002e-05 2024-09-21T12:23:54.462 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.462 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:54.462 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.462 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.462 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.462 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:116825 7.8 7:1dbf5839:::benchmark_data_smithi114_78432_object12679:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T12:23:54.463 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:23:12.374864+0000", 2024-09-21T12:23:54.463 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 41.929809341999999, 2024-09-21T12:23:54.463 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.16173959600000001, 2024-09-21T12:23:54.463 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:23:54.463 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:23:54.463 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:23:54.463 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:23:54.463 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:23:54.463 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:23:54.463 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 116825 2024-09-21T12:23:54.463 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.463 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:23:54.463 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.463 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:23:54.463 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:12.374864+0000", 2024-09-21T12:23:54.463 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.464 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.464 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.464 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:23:54.464 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:12.374864+0000", 2024-09-21T12:23:54.464 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:23:54.464 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.464 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.464 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:23:54.464 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:12.374865+0000", 2024-09-21T12:23:54.464 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.373e-06 2024-09-21T12:23:54.464 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.464 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.464 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:23:54.464 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:12.374870+0000", 2024-09-21T12:23:54.464 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4040000000000003e-06 2024-09-21T12:23:54.464 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.464 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.465 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:23:54.465 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:12.374883+0000", 2024-09-21T12:23:54.465 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2486999999999999e-05 2024-09-21T12:23:54.465 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.465 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.465 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:23:54.465 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:12.374890+0000", 2024-09-21T12:23:54.465 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1049999999999997e-06 2024-09-21T12:23:54.465 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.465 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.465 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:23:54.465 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:12.374951+0000", 2024-09-21T12:23:54.465 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0841999999999998e-05 2024-09-21T12:23:54.465 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.466 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.466 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:23:54.466 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:12.375093+0000", 2024-09-21T12:23:54.466 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014165599999999999 2024-09-21T12:23:54.466 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.466 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.466 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:23:54.466 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:12.375225+0000", 2024-09-21T12:23:54.466 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013289500000000001 2024-09-21T12:23:54.466 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.466 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.466 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:23:54.466 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:12.377645+0000", 2024-09-21T12:23:54.466 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0024194889999999999 2024-09-21T12:23:54.466 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.466 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.467 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.467 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:12.378041+0000", 2024-09-21T12:23:54.467 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00039582700000000002 2024-09-21T12:23:54.467 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.467 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.467 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:23:54.467 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:12.536116+0000", 2024-09-21T12:23:54.467 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.158075304 2024-09-21T12:23:54.467 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.467 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.467 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:23:54.467 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:12.536183+0000", 2024-09-21T12:23:54.467 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6668999999999995e-05 2024-09-21T12:23:54.467 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:23:54.467 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:23:54.467 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:23:54.468 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:12.536603+0000", 2024-09-21T12:23:54.468 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00042054499999999998 2024-09-21T12:23:54.468 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.468 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:54.468 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.468 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:23:54.468 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:23:54.468 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:23:54.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:54.469 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:54.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:54.570 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:54.595 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:54.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:54.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:54.697 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:54.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:54.745 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:54.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:54.846 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:54.871 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:54.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:54.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:54.973 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:23:54.996 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:54.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:54.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:23:55.098 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:55.121 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:55.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:55.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:55.223 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:55.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:54 smithi183 ceph-mon[33277]: 5.17 scrub starts 2024-09-21T12:23:55.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:54 smithi183 ceph-mon[33277]: 5.17 scrub ok 2024-09-21T12:23:55.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:54 smithi183 ceph-mon[33277]: 3.1a scrub starts 2024-09-21T12:23:55.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:54 smithi183 ceph-mon[33277]: 3.1a scrub ok 2024-09-21T12:23:55.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:54 smithi183 ceph-mon[33277]: pgmap v165: 158 pgs: 158 active+clean; 2.9 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 758 B/s rd, 0 B/s wr, 1.05k op/s; 65 B/s, 4 objects/s recovering 2024-09-21T12:23:55.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:54 smithi183 ceph-mon[33277]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T12:23:55.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:54 smithi183 ceph-mon[33277]: Cluster is now healthy 2024-09-21T12:23:55.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:54 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T12:23:55.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:54 smithi183 ceph-mon[33277]: osdmap e144: 8 total, 8 up, 7 in 2024-09-21T12:23:55.248 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:55.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:55.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:55.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:54 smithi114 ceph-mon[40867]: 5.17 scrub starts 2024-09-21T12:23:55.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:54 smithi114 ceph-mon[40867]: 5.17 scrub ok 2024-09-21T12:23:55.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:54 smithi114 ceph-mon[40867]: 3.1a scrub starts 2024-09-21T12:23:55.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:54 smithi114 ceph-mon[40867]: 3.1a scrub ok 2024-09-21T12:23:55.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:54 smithi114 ceph-mon[40867]: pgmap v165: 158 pgs: 158 active+clean; 2.9 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 758 B/s rd, 0 B/s wr, 1.05k op/s; 65 B/s, 4 objects/s recovering 2024-09-21T12:23:55.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:54 smithi114 ceph-mon[40867]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T12:23:55.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:54 smithi114 ceph-mon[40867]: Cluster is now healthy 2024-09-21T12:23:55.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:54 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T12:23:55.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:54 smithi114 ceph-mon[40867]: osdmap e144: 8 total, 8 up, 7 in 2024-09-21T12:23:55.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:54 smithi114 ceph-mon[30324]: 5.17 scrub starts 2024-09-21T12:23:55.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:54 smithi114 ceph-mon[30324]: 5.17 scrub ok 2024-09-21T12:23:55.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:54 smithi114 ceph-mon[30324]: 3.1a scrub starts 2024-09-21T12:23:55.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:54 smithi114 ceph-mon[30324]: 3.1a scrub ok 2024-09-21T12:23:55.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:54 smithi114 ceph-mon[30324]: pgmap v165: 158 pgs: 158 active+clean; 2.9 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 758 B/s rd, 0 B/s wr, 1.05k op/s; 65 B/s, 4 objects/s recovering 2024-09-21T12:23:55.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:54 smithi114 ceph-mon[30324]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T12:23:55.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:54 smithi114 ceph-mon[30324]: Cluster is now healthy 2024-09-21T12:23:55.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:54 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T12:23:55.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:54 smithi114 ceph-mon[30324]: osdmap e144: 8 total, 8 up, 7 in 2024-09-21T12:23:55.350 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:55.392 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:55.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:55.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:55.493 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:55.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:55.532 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:55.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:55.633 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:55.657 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:55.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:55.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:55.666 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T12:23:55.759 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:55.785 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:55.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:55.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:55.887 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:55.938 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:55.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:55.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:56.041 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:56.065 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:56.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:56.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:56.167 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:56.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:56.189 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:56.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:56.290 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:56.317 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:56.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:56.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:56.419 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:56.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:56.450 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:56.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:56.551 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:56.575 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:56.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:56.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:56.665 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T12:23:56.676 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:56.714 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T12:23:56.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:56.769 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:56.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:56.771 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:56.870 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:56.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:56.951 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:56.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:57.053 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:57.081 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:57.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:57.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:57.126 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:57.182 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:57.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:57.227 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:57.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:57.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:23:56 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:23:57.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:56 smithi183 ceph-mon[33277]: pgmap v167: 158 pgs: 158 active+clean; 2.9 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 953 op/s; 73 B/s, 4 objects/s recovering 2024-09-21T12:23:57.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:56 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:23:57.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:56 smithi183 ceph-mon[33277]: 6.1e scrub starts 2024-09-21T12:23:57.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:56 smithi183 ceph-mon[33277]: 6.1e scrub ok 2024-09-21T12:23:57.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:56 smithi183 ceph-mon[33277]: 6.7 scrub starts 2024-09-21T12:23:57.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:56 smithi183 ceph-mon[33277]: 6.7 scrub ok 2024-09-21T12:23:57.262 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:56 smithi114 ceph-mon[40867]: pgmap v167: 158 pgs: 158 active+clean; 2.9 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 953 op/s; 73 B/s, 4 objects/s recovering 2024-09-21T12:23:57.263 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:56 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:23:57.263 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:56 smithi114 ceph-mon[40867]: 6.1e scrub starts 2024-09-21T12:23:57.263 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:56 smithi114 ceph-mon[40867]: 6.1e scrub ok 2024-09-21T12:23:57.263 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:56 smithi114 ceph-mon[40867]: 6.7 scrub starts 2024-09-21T12:23:57.263 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:56 smithi114 ceph-mon[40867]: 6.7 scrub ok 2024-09-21T12:23:57.263 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:56 smithi114 ceph-mon[30324]: pgmap v167: 158 pgs: 158 active+clean; 2.9 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 953 op/s; 73 B/s, 4 objects/s recovering 2024-09-21T12:23:57.263 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:56 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:23:57.263 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:56 smithi114 ceph-mon[30324]: 6.1e scrub starts 2024-09-21T12:23:57.263 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:56 smithi114 ceph-mon[30324]: 6.1e scrub ok 2024-09-21T12:23:57.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:56 smithi114 ceph-mon[30324]: 6.7 scrub starts 2024-09-21T12:23:57.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:56 smithi114 ceph-mon[30324]: 6.7 scrub ok 2024-09-21T12:23:57.328 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:57.364 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:57.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:57.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:57.466 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:57.496 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:23:57.496 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:23:57.504 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:57.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:57.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:57.536 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:23:57.536 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:23:57.580 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:23:57.581 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:23:57.606 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:57.618 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:23:57.618 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:23:57.644 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:57.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:57.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:57.647 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:23:57.647 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:23:57.681 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:23:57.682 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:23:57.713 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:23:57.713 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:23:57.746 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:57.749 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:23:57.749 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:23:57.774 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:57.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:57.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:57.876 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:57.900 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:57.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:57.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:58.002 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:58.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:58.047 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:58.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:58.147 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:58.192 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:58.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:58.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:58.212 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:57 smithi114 ceph-mon[40867]: 6.e scrub starts 2024-09-21T12:23:58.212 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:57 smithi114 ceph-mon[40867]: 6.e scrub ok 2024-09-21T12:23:58.212 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:57 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:23:58.212 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:57 smithi114 ceph-mon[40867]: 5.10 scrub starts 2024-09-21T12:23:58.212 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:57 smithi114 ceph-mon[40867]: 5.10 scrub ok 2024-09-21T12:23:58.212 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:57 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/4855702' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:23:58.212 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:57 smithi114 ceph-mon[30324]: 6.e scrub starts 2024-09-21T12:23:58.212 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:57 smithi114 ceph-mon[30324]: 6.e scrub ok 2024-09-21T12:23:58.212 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:57 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:23:58.212 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:57 smithi114 ceph-mon[30324]: 5.10 scrub starts 2024-09-21T12:23:58.212 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:57 smithi114 ceph-mon[30324]: 5.10 scrub ok 2024-09-21T12:23:58.213 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:57 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4855702' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:23:58.225 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:23:58.231 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:57 smithi183 ceph-mon[33277]: 6.e scrub starts 2024-09-21T12:23:58.231 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:57 smithi183 ceph-mon[33277]: 6.e scrub ok 2024-09-21T12:23:58.231 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:57 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:23:58.232 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:57 smithi183 ceph-mon[33277]: 5.10 scrub starts 2024-09-21T12:23:58.232 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:57 smithi183 ceph-mon[33277]: 5.10 scrub ok 2024-09-21T12:23:58.232 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:57 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/4855702' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:23:58.294 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:58.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:58.343 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:58.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:58.444 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:23:58.486 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:58.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:58.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:23:58.588 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:23:58.617 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:58.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:58.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:23:58.719 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:58.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:58.743 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:58.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:58.844 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:58.881 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:58.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:58.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:23:58.956 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T12:23:58.983 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:59.020 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:59.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:59.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:59.122 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:59.149 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:59.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:59.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:59.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:58 smithi183 ceph-mon[33277]: 5.14 scrub starts 2024-09-21T12:23:59.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:58 smithi183 ceph-mon[33277]: 5.14 scrub ok 2024-09-21T12:23:59.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:58 smithi183 ceph-mon[33277]: 1.3 scrub starts 2024-09-21T12:23:59.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:58 smithi183 ceph-mon[33277]: 1.3 scrub ok 2024-09-21T12:23:59.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:58 smithi183 ceph-mon[33277]: pgmap v168: 158 pgs: 158 active+clean; 2.5 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.50k op/s; 56 B/s, 3 objects/s recovering 2024-09-21T12:23:59.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:58 smithi183 ceph-mon[33277]: 5.a scrub starts 2024-09-21T12:23:59.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:58 smithi183 ceph-mon[33277]: 5.a scrub ok 2024-09-21T12:23:59.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:58 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1209548162' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:23:59.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:58 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:23:59.251 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:23:59.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:58 smithi114 ceph-mon[40867]: 5.14 scrub starts 2024-09-21T12:23:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:58 smithi114 ceph-mon[40867]: 5.14 scrub ok 2024-09-21T12:23:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:58 smithi114 ceph-mon[40867]: 1.3 scrub starts 2024-09-21T12:23:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:58 smithi114 ceph-mon[40867]: 1.3 scrub ok 2024-09-21T12:23:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:58 smithi114 ceph-mon[40867]: pgmap v168: 158 pgs: 158 active+clean; 2.5 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.50k op/s; 56 B/s, 3 objects/s recovering 2024-09-21T12:23:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:58 smithi114 ceph-mon[40867]: 5.a scrub starts 2024-09-21T12:23:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:58 smithi114 ceph-mon[40867]: 5.a scrub ok 2024-09-21T12:23:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:58 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1209548162' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:23:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:58 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:23:59.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:58 smithi114 ceph-mon[30324]: 5.14 scrub starts 2024-09-21T12:23:59.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:58 smithi114 ceph-mon[30324]: 5.14 scrub ok 2024-09-21T12:23:59.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:58 smithi114 ceph-mon[30324]: 1.3 scrub starts 2024-09-21T12:23:59.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:58 smithi114 ceph-mon[30324]: 1.3 scrub ok 2024-09-21T12:23:59.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:58 smithi114 ceph-mon[30324]: pgmap v168: 158 pgs: 158 active+clean; 2.5 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.50k op/s; 56 B/s, 3 objects/s recovering 2024-09-21T12:23:59.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:58 smithi114 ceph-mon[30324]: 5.a scrub starts 2024-09-21T12:23:59.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:58 smithi114 ceph-mon[30324]: 5.a scrub ok 2024-09-21T12:23:59.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:58 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1209548162' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:23:59.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:58 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:23:59.286 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:59.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:59.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:23:59.388 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:23:59.424 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:59.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:59.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:23:59.526 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:23:59.555 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:59.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:59.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:23:59.657 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:59.682 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:59.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:59.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:59.784 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:23:59.810 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:23:59.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:59.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:23:59.912 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:23:59.956 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:23:59.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:23:59.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:00.058 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:00.097 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:00.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:00.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:00.199 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:00.224 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:00.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:00.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:00.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:59 smithi183 ceph-mon[33277]: 6.1f scrub starts 2024-09-21T12:24:00.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:59 smithi183 ceph-mon[33277]: 6.1f scrub ok 2024-09-21T12:24:00.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:59 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T12:24:00.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:59 smithi183 ceph-mon[33277]: osdmap e145: 8 total, 8 up, 7 in 2024-09-21T12:24:00.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:59 smithi183 ceph-mon[33277]: 4.13 scrub starts 2024-09-21T12:24:00.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:59 smithi183 ceph-mon[33277]: 3.d scrub starts 2024-09-21T12:24:00.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:59 smithi183 ceph-mon[33277]: 3.d scrub ok 2024-09-21T12:24:00.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:59 smithi183 ceph-mon[33277]: 4.13 scrub ok 2024-09-21T12:24:00.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:59 smithi183 ceph-mon[33277]: 3.0 scrub starts 2024-09-21T12:24:00.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:23:59 smithi183 ceph-mon[33277]: 3.0 scrub ok 2024-09-21T12:24:00.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:59 smithi114 ceph-mon[40867]: 6.1f scrub starts 2024-09-21T12:24:00.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:59 smithi114 ceph-mon[40867]: 6.1f scrub ok 2024-09-21T12:24:00.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:59 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T12:24:00.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:59 smithi114 ceph-mon[40867]: osdmap e145: 8 total, 8 up, 7 in 2024-09-21T12:24:00.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:59 smithi114 ceph-mon[40867]: 4.13 scrub starts 2024-09-21T12:24:00.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:59 smithi114 ceph-mon[40867]: 3.d scrub starts 2024-09-21T12:24:00.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:59 smithi114 ceph-mon[40867]: 3.d scrub ok 2024-09-21T12:24:00.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:59 smithi114 ceph-mon[40867]: 4.13 scrub ok 2024-09-21T12:24:00.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:59 smithi114 ceph-mon[40867]: 3.0 scrub starts 2024-09-21T12:24:00.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:23:59 smithi114 ceph-mon[40867]: 3.0 scrub ok 2024-09-21T12:24:00.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:59 smithi114 ceph-mon[30324]: 6.1f scrub starts 2024-09-21T12:24:00.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:59 smithi114 ceph-mon[30324]: 6.1f scrub ok 2024-09-21T12:24:00.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:59 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T12:24:00.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:59 smithi114 ceph-mon[30324]: osdmap e145: 8 total, 8 up, 7 in 2024-09-21T12:24:00.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:59 smithi114 ceph-mon[30324]: 4.13 scrub starts 2024-09-21T12:24:00.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:59 smithi114 ceph-mon[30324]: 3.d scrub starts 2024-09-21T12:24:00.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:59 smithi114 ceph-mon[30324]: 3.d scrub ok 2024-09-21T12:24:00.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:59 smithi114 ceph-mon[30324]: 4.13 scrub ok 2024-09-21T12:24:00.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:59 smithi114 ceph-mon[30324]: 3.0 scrub starts 2024-09-21T12:24:00.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:23:59 smithi114 ceph-mon[30324]: 3.0 scrub ok 2024-09-21T12:24:00.326 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:00.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:00.366 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:00.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:00.467 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:00.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:00.513 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:00.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:00.614 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:00.637 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:00.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:00.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:00.739 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:00.765 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:00.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:00.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:00.866 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:00.897 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:00.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:00.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:00.999 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:01.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:00 smithi114 ceph-mon[40867]: pgmap v170: 158 pgs: 158 active+clean; 2.5 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 663 B/s rd, 0 B/s wr, 940 op/s 2024-09-21T12:24:01.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:00 smithi114 ceph-mon[40867]: 6.15 scrub starts 2024-09-21T12:24:01.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:00 smithi114 ceph-mon[40867]: 6.15 scrub ok 2024-09-21T12:24:01.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:00 smithi114 ceph-mon[30324]: pgmap v170: 158 pgs: 158 active+clean; 2.5 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 663 B/s rd, 0 B/s wr, 940 op/s 2024-09-21T12:24:01.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:00 smithi114 ceph-mon[30324]: 6.15 scrub starts 2024-09-21T12:24:01.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:00 smithi114 ceph-mon[30324]: 6.15 scrub ok 2024-09-21T12:24:01.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:01.026 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:01.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:01.127 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:01.167 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:01.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:01.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:01.235 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T12:24:01.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:00 smithi183 ceph-mon[33277]: pgmap v170: 158 pgs: 158 active+clean; 2.5 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 663 B/s rd, 0 B/s wr, 940 op/s 2024-09-21T12:24:01.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:00 smithi183 ceph-mon[33277]: 6.15 scrub starts 2024-09-21T12:24:01.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:00 smithi183 ceph-mon[33277]: 6.15 scrub ok 2024-09-21T12:24:01.270 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:01.294 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:01.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:01.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:01.310 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:24:01.310 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:24:01.396 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:01.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:01.457 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:01.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:01.553 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:01.558 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:01.579 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:01.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:01.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:01.681 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:01.737 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:01.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:01.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:01.838 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:01.864 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:01.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:01.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:01.965 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:01.991 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:01.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:01.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:02.077 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:24:02.093 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:02.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:02.181 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:02.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:02.282 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:02.327 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:02.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:02.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:02.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:02 smithi183 ceph-mon[33277]: 5.e scrub starts 2024-09-21T12:24:02.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:02 smithi183 ceph-mon[33277]: 5.e scrub ok 2024-09-21T12:24:02.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:02 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:24:02.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:02 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:24:02.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:02 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:24:02.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:02 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:24:02.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:02 smithi183 ceph-mon[33277]: 5.18 scrub starts 2024-09-21T12:24:02.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:02 smithi183 ceph-mon[33277]: 5.18 scrub ok 2024-09-21T12:24:02.489 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:02.523 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:02.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:02.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:02.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:02 smithi114 ceph-mon[40867]: 5.e scrub starts 2024-09-21T12:24:02.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:02 smithi114 ceph-mon[40867]: 5.e scrub ok 2024-09-21T12:24:02.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:02 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:24:02.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:02 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:24:02.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:02 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:24:02.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:02 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:24:02.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:02 smithi114 ceph-mon[40867]: 5.18 scrub starts 2024-09-21T12:24:02.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:02 smithi114 ceph-mon[40867]: 5.18 scrub ok 2024-09-21T12:24:02.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:02 smithi114 ceph-mon[30324]: 5.e scrub starts 2024-09-21T12:24:02.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:02 smithi114 ceph-mon[30324]: 5.e scrub ok 2024-09-21T12:24:02.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:02 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:24:02.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:02 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:24:02.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:02 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:24:02.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:02 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:24:02.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:02 smithi114 ceph-mon[30324]: 5.18 scrub starts 2024-09-21T12:24:02.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:02 smithi114 ceph-mon[30324]: 5.18 scrub ok 2024-09-21T12:24:02.625 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:02.648 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:02.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:02.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:02.749 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:02.775 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:02.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:02.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:02.876 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:02.877 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:24:02 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:24:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:24:02.913 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:02.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:02.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:03.015 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:03.050 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:03.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:03.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:03.152 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:03.177 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:03.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:03.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:03.234 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T12:24:03.279 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:03.337 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:03.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:03.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:03.437 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:03 smithi114 ceph-mon[40867]: 4.2 scrub starts 2024-09-21T12:24:03.437 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:03 smithi114 ceph-mon[40867]: 4.2 scrub ok 2024-09-21T12:24:03.437 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:03 smithi114 ceph-mon[40867]: pgmap v171: 158 pgs: 158 active+clean; 2.4 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 947 op/s 2024-09-21T12:24:03.438 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:03 smithi114 ceph-mon[40867]: 4.a scrub starts 2024-09-21T12:24:03.438 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:03 smithi114 ceph-mon[40867]: 4.a scrub ok 2024-09-21T12:24:03.438 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:03 smithi114 ceph-mon[30324]: 4.2 scrub starts 2024-09-21T12:24:03.438 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:03 smithi114 ceph-mon[30324]: 4.2 scrub ok 2024-09-21T12:24:03.438 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:03 smithi114 ceph-mon[30324]: pgmap v171: 158 pgs: 158 active+clean; 2.4 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 947 op/s 2024-09-21T12:24:03.438 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:03 smithi114 ceph-mon[30324]: 4.a scrub starts 2024-09-21T12:24:03.438 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:03 smithi114 ceph-mon[30324]: 4.a scrub ok 2024-09-21T12:24:03.439 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:03.463 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:03.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:03.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:03.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:03 smithi183 ceph-mon[33277]: 4.2 scrub starts 2024-09-21T12:24:03.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:03 smithi183 ceph-mon[33277]: 4.2 scrub ok 2024-09-21T12:24:03.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:03 smithi183 ceph-mon[33277]: pgmap v171: 158 pgs: 158 active+clean; 2.4 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 947 op/s 2024-09-21T12:24:03.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:03 smithi183 ceph-mon[33277]: 4.a scrub starts 2024-09-21T12:24:03.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:03 smithi183 ceph-mon[33277]: 4.a scrub ok 2024-09-21T12:24:03.565 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:03.590 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:03.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:03.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:03.600 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:03.600 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:03.692 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:03.743 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:03.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:03.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:03.845 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:03.872 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:03.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:03.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:03.974 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:03.998 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:03.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:03.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:04.100 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:04.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:04.141 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:04.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:04.192 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-21T12:24:04.242 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:04.269 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:04.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:04.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:04.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:04 smithi114 ceph-mon[40867]: 4.1e scrub starts 2024-09-21T12:24:04.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:04 smithi114 ceph-mon[40867]: 4.1e scrub ok 2024-09-21T12:24:04.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:04 smithi114 ceph-mon[40867]: 7.3 scrub starts 2024-09-21T12:24:04.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:04 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2041367721' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:24:04.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:04 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:24:04.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:04 smithi114 ceph-mon[40867]: 7.d scrub starts 2024-09-21T12:24:04.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:04 smithi114 ceph-mon[30324]: 4.1e scrub starts 2024-09-21T12:24:04.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:04 smithi114 ceph-mon[30324]: 4.1e scrub ok 2024-09-21T12:24:04.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:04 smithi114 ceph-mon[30324]: 7.3 scrub starts 2024-09-21T12:24:04.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:04 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2041367721' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:24:04.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:04 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:24:04.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:04 smithi114 ceph-mon[30324]: 7.d scrub starts 2024-09-21T12:24:04.371 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:04.408 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:04.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:04.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:04.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:04 smithi183 ceph-mon[33277]: 4.1e scrub starts 2024-09-21T12:24:04.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:04 smithi183 ceph-mon[33277]: 4.1e scrub ok 2024-09-21T12:24:04.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:04 smithi183 ceph-mon[33277]: 7.3 scrub starts 2024-09-21T12:24:04.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:04 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2041367721' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:24:04.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:04 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:24:04.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:04 smithi183 ceph-mon[33277]: 7.d scrub starts 2024-09-21T12:24:04.510 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:04.537 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:04.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:04.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:04.638 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:04.663 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:04.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:04.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:04.764 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:04.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:04.803 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:04.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:04.904 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:04.945 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:04.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:04.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:05.047 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:05.069 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:05.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:05.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:05.170 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:05.197 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:05.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:05.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:05.298 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:05.330 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:05.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:05.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:05.432 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:05.470 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:05.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:05.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:05.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:05 smithi183 ceph-mon[33277]: pgmap v172: 158 pgs: 158 active+clean; 2.2 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-09-21T12:24:05.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:05 smithi183 ceph-mon[33277]: 7.3 scrub ok 2024-09-21T12:24:05.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:05 smithi183 ceph-mon[33277]: 6.18 scrub starts 2024-09-21T12:24:05.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:05 smithi183 ceph-mon[33277]: 6.18 scrub ok 2024-09-21T12:24:05.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:05 smithi183 ceph-mon[33277]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:24:05.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:05 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:24:05.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:05 smithi183 ceph-mon[33277]: osdmap e146: 8 total, 8 up, 7 in 2024-09-21T12:24:05.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:05 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/907281906' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:24:05.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:05 smithi114 ceph-mon[40867]: pgmap v172: 158 pgs: 158 active+clean; 2.2 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-09-21T12:24:05.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:05 smithi114 ceph-mon[40867]: 7.3 scrub ok 2024-09-21T12:24:05.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:05 smithi114 ceph-mon[40867]: 6.18 scrub starts 2024-09-21T12:24:05.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:05 smithi114 ceph-mon[40867]: 6.18 scrub ok 2024-09-21T12:24:05.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:05 smithi114 ceph-mon[40867]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:24:05.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:05 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:24:05.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:05 smithi114 ceph-mon[40867]: osdmap e146: 8 total, 8 up, 7 in 2024-09-21T12:24:05.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:05 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/907281906' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:24:05.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:05 smithi114 ceph-mon[30324]: pgmap v172: 158 pgs: 158 active+clean; 2.2 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-09-21T12:24:05.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:05 smithi114 ceph-mon[30324]: 7.3 scrub ok 2024-09-21T12:24:05.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:05 smithi114 ceph-mon[30324]: 6.18 scrub starts 2024-09-21T12:24:05.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:05 smithi114 ceph-mon[30324]: 6.18 scrub ok 2024-09-21T12:24:05.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:05 smithi114 ceph-mon[30324]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:24:05.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:05 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:24:05.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:05 smithi114 ceph-mon[30324]: osdmap e146: 8 total, 8 up, 7 in 2024-09-21T12:24:05.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:05 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/907281906' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:24:05.572 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:05.597 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:05.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:05.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:05.698 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:05.725 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:05.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:05.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:05.826 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:05.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:05.866 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:05.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:05.967 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:05.988 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:05.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:05.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:06.090 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:06.115 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:06.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:06.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:06.217 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:06.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:06.252 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:06.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:06.353 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:06.379 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:06.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:06.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:06.441 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:24:06.441 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:24:06.441 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:24:06.441 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:24:06.441 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:24:06.442 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:24:06.442 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:24:06.442 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:24:06.442 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:24:06.442 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:24:06.442 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:24:06.442 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:24:06.442 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:24:06.442 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:24:06.442 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:24:06.442 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:24:06.481 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:06.505 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:06.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:06.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:06.607 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:06.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:06.647 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:06.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:06.651 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:06.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:06.793 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:06.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:06.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:06.895 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:06.930 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:06.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:06.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:07.032 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:07.057 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:07.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:07.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:07.158 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:07.202 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:07.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:07.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:07.205 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:24:06 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:24:07.304 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:07.342 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:07 smithi114 ceph-mon[30324]: pgmap v174: 158 pgs: 158 active+clean; 2.2 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 723 op/s 2024-09-21T12:24:07.343 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:07 smithi114 ceph-mon[40867]: pgmap v174: 158 pgs: 158 active+clean; 2.2 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 723 op/s 2024-09-21T12:24:07.357 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:07.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:07.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:07.458 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:07.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:07 smithi183 ceph-mon[33277]: pgmap v174: 158 pgs: 158 active+clean; 2.2 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 723 op/s 2024-09-21T12:24:07.497 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:07.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:07.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:07.599 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:07.624 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:07.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:07.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:07.725 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:07.764 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:07.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:07.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:07.865 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:07.891 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:07.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:07.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:07.992 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:08.016 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:08.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:08.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:08.117 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:08.158 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:08.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:08.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:08.260 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:08.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:08 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:24:08.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:08 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:24:08.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:08 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:24:08.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:08 smithi114 ceph-mon[40867]: osdmap e147: 8 total, 8 up, 7 in 2024-09-21T12:24:08.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:08 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:24:08.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:08 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:24:08.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:08 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:24:08.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:08 smithi114 ceph-mon[30324]: osdmap e147: 8 total, 8 up, 7 in 2024-09-21T12:24:08.284 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:08.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:08.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:08.386 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:08.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:08.429 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:08.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:08.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:08 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:24:08.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:08 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:24:08.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:08 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:24:08.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:08 smithi183 ceph-mon[33277]: osdmap e147: 8 total, 8 up, 7 in 2024-09-21T12:24:08.529 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:08.552 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:08.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:08.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:08.654 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:08.676 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:08.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:08.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:08.716 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:24:08.716 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:24:08.716 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:24:08.716 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:24:08.716 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:24:08.777 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:08.813 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:08.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:08.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:08.915 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:08.939 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:08.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:08.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:09.040 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:09.078 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:09.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:09.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:09.123 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:24:09.179 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:09.231 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:09.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:09.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:09.232 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T12:24:09.333 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:09.355 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:09.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:09.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:09.456 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:09.479 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:09.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:09.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:09.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:09 smithi114 ceph-mon[40867]: pgmap v176: 158 pgs: 158 active+clean; 1.9 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-09-21T12:24:09.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:09 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1953635769' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:24:09.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:09 smithi114 ceph-mon[30324]: pgmap v176: 158 pgs: 158 active+clean; 1.9 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-09-21T12:24:09.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:09 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1953635769' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:24:09.580 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:09.604 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:09.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:09.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:09.668 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T12:24:09.705 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:09.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:09.732 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:09.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:09.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:09 smithi183 ceph-mon[33277]: pgmap v176: 158 pgs: 158 active+clean; 1.9 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-09-21T12:24:09.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:09 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1953635769' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:24:09.832 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:09.856 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:09.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:09.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:09.957 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:09.980 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T12:24:09.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:09.982 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:09.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:10.083 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:10.106 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:10.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:10.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:10.208 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:10.232 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:10.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:10.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:10.334 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:10.358 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:10.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:10.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:10.459 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:10.497 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:10.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:10.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:10.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:10 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1953635769' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T12:24:10.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:10 smithi114 ceph-mon[40867]: osdmap e148: 8 total, 8 up, 7 in 2024-09-21T12:24:10.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:10 smithi114 ceph-mon[40867]: pgmap v178: 158 pgs: 158 active+clean; 1.9 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 915 op/s 2024-09-21T12:24:10.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:10 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1953635769' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T12:24:10.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:10 smithi114 ceph-mon[30324]: osdmap e148: 8 total, 8 up, 7 in 2024-09-21T12:24:10.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:10 smithi114 ceph-mon[30324]: pgmap v178: 158 pgs: 158 active+clean; 1.9 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 915 op/s 2024-09-21T12:24:10.600 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:10.638 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:10.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:10.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:10.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:10 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1953635769' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T12:24:10.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:10 smithi183 ceph-mon[33277]: osdmap e148: 8 total, 8 up, 7 in 2024-09-21T12:24:10.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:10 smithi183 ceph-mon[33277]: pgmap v178: 158 pgs: 158 active+clean; 1.9 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 915 op/s 2024-09-21T12:24:10.740 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:10.770 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:10.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:10.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:10.872 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:10.899 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:10.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:10.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:11.001 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:11.026 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:11.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:11.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:11.052 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T12:24:11.128 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:11.176 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:11.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:11.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:11.278 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:11.303 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:11.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:11.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:11.404 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:11.447 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:11.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:11.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:11.549 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:11.588 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:11.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:11.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:11.690 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:11.730 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:11.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:11.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:11.832 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:11.858 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:11.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:11.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:11.960 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:11.985 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:11.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:11.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:12.087 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:12.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:12.123 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:12.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:12.224 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:12.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:12.256 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:12.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:12.357 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:12.382 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:12.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:12.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:12.483 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:12.507 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:12.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:12.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:12.609 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:12.643 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:12.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:12.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:12.744 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:12.766 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:12.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:12.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:12.830 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:24:12 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:24:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:24:12.868 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:12.893 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:12.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:12.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:12.995 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:13.043 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:13.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:13.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:13.145 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:13.177 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:13.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:13.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:13.207 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:12 smithi114 ceph-mon[30324]: pgmap v179: 158 pgs: 158 active+clean; 1.8 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 959 op/s 2024-09-21T12:24:13.207 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:12 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:24:13.207 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:12 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:24:13.207 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:12 smithi114 ceph-mon[30324]: osdmap e149: 8 total, 8 up, 7 in 2024-09-21T12:24:13.207 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:12 smithi114 ceph-mon[40867]: pgmap v179: 158 pgs: 158 active+clean; 1.8 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 959 op/s 2024-09-21T12:24:13.208 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:12 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:24:13.208 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:12 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:24:13.208 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:12 smithi114 ceph-mon[40867]: osdmap e149: 8 total, 8 up, 7 in 2024-09-21T12:24:13.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:12 smithi183 ceph-mon[33277]: pgmap v179: 158 pgs: 158 active+clean; 1.8 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 959 op/s 2024-09-21T12:24:13.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:12 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:24:13.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:12 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:24:13.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:12 smithi183 ceph-mon[33277]: osdmap e149: 8 total, 8 up, 7 in 2024-09-21T12:24:13.280 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:13.305 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:13.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:13.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:13.387 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:13.387 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:13.391 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:13.407 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:13.430 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:13.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:13.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:13.532 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:13.558 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:13.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:13.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:13.660 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:13.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:13.704 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:13.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:13.798 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:13.805 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:13.828 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:13.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:13.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:13.930 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:13.954 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:13.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:13.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:14.056 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:14.078 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:14.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:14.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:14.180 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:14.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:14.237 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:14.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:14.338 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:14.388 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:14.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:14.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:14.490 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:14.515 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:14.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:14.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:14.616 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:14.640 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:14.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:14.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:14.742 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:14.786 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:14.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:14.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:14.809 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:14 smithi114 ceph-mon[30324]: pgmap v181: 158 pgs: 158 active+clean; 1.5 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 794 B/s rd, 0 B/s wr, 825 op/s 2024-09-21T12:24:14.809 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:14 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:24:14.809 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:14 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:24:14.810 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:14 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:24:14.810 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:14 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:24:14.810 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:14 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:24:14.810 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:14 smithi114 ceph-mon[40867]: pgmap v181: 158 pgs: 158 active+clean; 1.5 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 794 B/s rd, 0 B/s wr, 825 op/s 2024-09-21T12:24:14.810 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:14 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:24:14.810 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:14 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:24:14.810 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:14 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:24:14.810 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:14 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:24:14.810 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:14 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:24:14.889 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:14.914 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:14.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:14.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:15.016 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:15.038 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:15.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:15.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:15.140 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:15.165 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:15.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:15.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:15.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:14 smithi183 ceph-mon[33277]: pgmap v181: 158 pgs: 158 active+clean; 1.5 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 794 B/s rd, 0 B/s wr, 825 op/s 2024-09-21T12:24:15.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:14 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:24:15.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:14 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:24:15.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:14 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:24:15.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:14 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:24:15.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:14 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:24:15.267 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:15.291 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:15.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:15.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:15.393 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:15.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:15.418 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:15.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:15.519 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:15.543 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:15.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:15.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:15.645 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:15.669 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:15.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:15.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:15.770 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:15.785 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-21T12:24:15.790 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T12:24:15.807 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:24:15.807 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 20, 2024-09-21T12:24:15.807 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 600, 2024-09-21T12:24:15.807 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-21T12:24:15.807 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.808 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:24382 7.3 7:c60f87f6:::benchmark_data_smithi114_78432_object24381:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T12:24:15.808 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:40.027851+0000", 2024-09-21T12:24:15.808 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 155.61206451499999, 2024-09-21T12:24:15.808 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0543905850000002, 2024-09-21T12:24:15.808 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:24:15.808 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:24:15.808 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:15.808 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:24:15.808 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:24:15.808 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:15.808 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 24382 2024-09-21T12:24:15.808 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.808 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:24:15.808 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.809 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:24:15.809 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.027851+0000", 2024-09-21T12:24:15.809 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.809 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.809 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.809 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:24:15.809 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.027851+0000", 2024-09-21T12:24:15.809 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.809 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.809 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.809 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:24:15.809 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.027853+0000", 2024-09-21T12:24:15.809 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6729999999999999e-06 2024-09-21T12:24:15.809 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.809 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.810 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:24:15.810 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.027885+0000", 2024-09-21T12:24:15.810 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2155e-05 2024-09-21T12:24:15.810 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.810 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.810 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:24:15.810 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.027897+0000", 2024-09-21T12:24:15.810 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2371000000000001e-05 2024-09-21T12:24:15.810 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.810 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.810 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:24:15.810 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.027907+0000", 2024-09-21T12:24:15.811 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.9580000000000002e-06 2024-09-21T12:24:15.811 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.811 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.811 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.811 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.031303+0000", 2024-09-21T12:24:15.811 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003395749 2024-09-21T12:24:15.811 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.811 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.811 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:24:15.811 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.031311+0000", 2024-09-21T12:24:15.811 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.3499999999999997e-06 2024-09-21T12:24:15.811 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.812 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.812 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.812 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.026611+0000", 2024-09-21T12:24:15.812 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.995300132 2024-09-21T12:24:15.812 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.812 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.812 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:24:15.812 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.026622+0000", 2024-09-21T12:24:15.812 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0735e-05 2024-09-21T12:24:15.812 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.812 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.812 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.812 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.070786+0000", 2024-09-21T12:24:15.812 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0441637530000001 2024-09-21T12:24:15.813 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.813 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.813 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:24:15.813 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.070935+0000", 2024-09-21T12:24:15.813 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014934 2024-09-21T12:24:15.813 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.813 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.813 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:24:15.813 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.071175+0000", 2024-09-21T12:24:15.813 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000239409 2024-09-21T12:24:15.813 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.813 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.813 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.813 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.074999+0000", 2024-09-21T12:24:15.814 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0038247229999999999 2024-09-21T12:24:15.814 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.814 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.814 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.814 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.075051+0000", 2024-09-21T12:24:15.814 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1588000000000001e-05 2024-09-21T12:24:15.814 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.814 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.814 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:24:15.815 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.081204+0000", 2024-09-21T12:24:15.815 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0061526209999999996 2024-09-21T12:24:15.815 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.815 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.815 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:24:15.815 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.081254+0000", 2024-09-21T12:24:15.815 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.024e-05 2024-09-21T12:24:15.815 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.815 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.815 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:24:15.815 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.082242+0000", 2024-09-21T12:24:15.815 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00098778800000000008 2024-09-21T12:24:15.815 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.815 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:15.816 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.816 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.816 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.816 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:24399 7.3 7:cc1424c5:::benchmark_data_smithi114_78432_object24398:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:24:15.816 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:40.028127+0000", 2024-09-21T12:24:15.816 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 155.61178894400001, 2024-09-21T12:24:15.816 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0647246259999998, 2024-09-21T12:24:15.816 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:24:15.816 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:24:15.816 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:15.816 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:24:15.816 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:24:15.816 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:15.817 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 24399 2024-09-21T12:24:15.817 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.817 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:24:15.817 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.817 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:24:15.817 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.028127+0000", 2024-09-21T12:24:15.817 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.817 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.817 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.817 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:24:15.817 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.028127+0000", 2024-09-21T12:24:15.817 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.817 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.817 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.818 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:24:15.818 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.028127+0000", 2024-09-21T12:24:15.818 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5000000000000003e-07 2024-09-21T12:24:15.818 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.818 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.818 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:24:15.818 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.028150+0000", 2024-09-21T12:24:15.818 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2653000000000001e-05 2024-09-21T12:24:15.818 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.818 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.818 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:24:15.818 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.028161+0000", 2024-09-21T12:24:15.818 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0939e-05 2024-09-21T12:24:15.818 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.818 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.819 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:24:15.819 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.028166+0000", 2024-09-21T12:24:15.819 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1970000000000004e-06 2024-09-21T12:24:15.819 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.819 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.819 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.819 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.031332+0000", 2024-09-21T12:24:15.819 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003165883 2024-09-21T12:24:15.819 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.819 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.819 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:24:15.819 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.031338+0000", 2024-09-21T12:24:15.819 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8850000000000004e-06 2024-09-21T12:24:15.819 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.819 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.820 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.820 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.032668+0000", 2024-09-21T12:24:15.820 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.001330367 2024-09-21T12:24:15.820 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.820 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.820 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:24:15.820 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.032679+0000", 2024-09-21T12:24:15.820 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0912e-05 2024-09-21T12:24:15.820 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.820 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.820 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.821 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.073338+0000", 2024-09-21T12:24:15.821 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.040658632 2024-09-21T12:24:15.821 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.821 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.821 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:24:15.821 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.073479+0000", 2024-09-21T12:24:15.821 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014109800000000001 2024-09-21T12:24:15.821 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.821 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.821 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:24:15.821 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.073642+0000", 2024-09-21T12:24:15.821 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016357500000000001 2024-09-21T12:24:15.821 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.822 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.822 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.822 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.079581+0000", 2024-09-21T12:24:15.822 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0059387479999999998 2024-09-21T12:24:15.822 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.822 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.822 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:24:15.822 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.082251+0000", 2024-09-21T12:24:15.822 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0026700399999999998 2024-09-21T12:24:15.822 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.822 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.822 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.822 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.090796+0000", 2024-09-21T12:24:15.823 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0085451050000000008 2024-09-21T12:24:15.823 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.823 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.823 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:24:15.823 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.090909+0000", 2024-09-21T12:24:15.823 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011303600000000001 2024-09-21T12:24:15.823 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.823 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.823 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:24:15.823 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.092851+0000", 2024-09-21T12:24:15.823 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001942006 2024-09-21T12:24:15.823 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.823 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:15.824 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.824 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.824 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.824 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:24403 7.1 7:8d4347eb:::benchmark_data_smithi114_78432_object24402:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:24:15.824 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:40.038076+0000", 2024-09-21T12:24:15.824 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 155.60183991599999, 2024-09-21T12:24:15.824 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0479541729999999, 2024-09-21T12:24:15.824 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:24:15.824 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:24:15.824 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:15.824 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:24:15.824 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:24:15.824 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:15.824 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 24403 2024-09-21T12:24:15.825 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.825 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:24:15.825 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.825 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:24:15.825 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.038076+0000", 2024-09-21T12:24:15.825 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.825 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.825 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.825 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:24:15.825 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.038076+0000", 2024-09-21T12:24:15.825 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.825 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.825 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.825 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:24:15.826 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.038077+0000", 2024-09-21T12:24:15.826 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.641e-06 2024-09-21T12:24:15.826 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.826 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.826 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:24:15.826 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.038103+0000", 2024-09-21T12:24:15.826 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5689000000000001e-05 2024-09-21T12:24:15.826 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.826 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.826 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:24:15.826 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.038112+0000", 2024-09-21T12:24:15.826 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.3409999999999992e-06 2024-09-21T12:24:15.826 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.826 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.827 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:24:15.827 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.038119+0000", 2024-09-21T12:24:15.827 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1829999999999999e-06 2024-09-21T12:24:15.827 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.827 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.827 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.827 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.038160+0000", 2024-09-21T12:24:15.827 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1597000000000002e-05 2024-09-21T12:24:15.827 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.827 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.827 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:24:15.827 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.038170+0000", 2024-09-21T12:24:15.827 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.4520000000000003e-06 2024-09-21T12:24:15.827 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.828 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.828 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.828 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.021190+0000", 2024-09-21T12:24:15.828 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98302043100000003 2024-09-21T12:24:15.828 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.828 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.828 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:24:15.828 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.021201+0000", 2024-09-21T12:24:15.828 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1e-05 2024-09-21T12:24:15.828 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.828 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.828 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.828 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.076308+0000", 2024-09-21T12:24:15.828 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0551071169999999 2024-09-21T12:24:15.829 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.829 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.829 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:24:15.829 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.077219+0000", 2024-09-21T12:24:15.829 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00091087299999999998 2024-09-21T12:24:15.829 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.829 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.829 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:24:15.829 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.077364+0000", 2024-09-21T12:24:15.829 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014518300000000001 2024-09-21T12:24:15.829 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.829 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.829 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.829 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.080655+0000", 2024-09-21T12:24:15.829 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0032908130000000001 2024-09-21T12:24:15.830 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.830 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.830 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:24:15.830 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.082504+0000", 2024-09-21T12:24:15.830 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001849301 2024-09-21T12:24:15.830 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.830 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.830 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.830 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.085101+0000", 2024-09-21T12:24:15.830 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002597079 2024-09-21T12:24:15.830 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.830 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.830 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:24:15.830 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.085142+0000", 2024-09-21T12:24:15.831 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0460000000000002e-05 2024-09-21T12:24:15.831 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.831 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.831 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:24:15.831 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.086030+0000", 2024-09-21T12:24:15.831 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000888013 2024-09-21T12:24:15.831 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.831 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:15.831 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.831 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.831 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.831 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:24412 7.3 7:c33a628a:::benchmark_data_smithi114_78432_object24411:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:24:15.831 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:40.043570+0000", 2024-09-21T12:24:15.831 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 155.59634603200001, 2024-09-21T12:24:15.832 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0510890659999998, 2024-09-21T12:24:15.832 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:24:15.832 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:24:15.832 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:15.832 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:24:15.832 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:24:15.832 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:15.832 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 24412 2024-09-21T12:24:15.832 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.832 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:24:15.832 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.832 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:24:15.832 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.043570+0000", 2024-09-21T12:24:15.832 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.833 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.833 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.833 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:24:15.833 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.043570+0000", 2024-09-21T12:24:15.833 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.833 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.833 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.833 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:24:15.833 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.043571+0000", 2024-09-21T12:24:15.833 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0690000000000001e-06 2024-09-21T12:24:15.833 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.833 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.833 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:24:15.833 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.043592+0000", 2024-09-21T12:24:15.833 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1237e-05 2024-09-21T12:24:15.834 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.834 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.834 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:24:15.834 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.043602+0000", 2024-09-21T12:24:15.834 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6630000000000005e-06 2024-09-21T12:24:15.834 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.834 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.834 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:24:15.834 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.043608+0000", 2024-09-21T12:24:15.834 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1179999999999997e-06 2024-09-21T12:24:15.834 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.834 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.834 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.834 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.043667+0000", 2024-09-21T12:24:15.834 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9574000000000003e-05 2024-09-21T12:24:15.835 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.835 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.835 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:24:15.835 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.043677+0000", 2024-09-21T12:24:15.835 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.3899999999999999e-06 2024-09-21T12:24:15.835 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.835 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.835 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.835 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.032706+0000", 2024-09-21T12:24:15.835 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98902957700000005 2024-09-21T12:24:15.835 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.835 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.835 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:24:15.835 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.032714+0000", 2024-09-21T12:24:15.835 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9379999999999992e-06 2024-09-21T12:24:15.836 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.836 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.836 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.836 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.075083+0000", 2024-09-21T12:24:15.836 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0423692630000001 2024-09-21T12:24:15.836 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.836 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.836 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:24:15.836 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.075221+0000", 2024-09-21T12:24:15.836 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013774800000000001 2024-09-21T12:24:15.836 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.836 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.836 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:24:15.836 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.075379+0000", 2024-09-21T12:24:15.836 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000157577 2024-09-21T12:24:15.836 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.836 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.837 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.837 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.079711+0000", 2024-09-21T12:24:15.837 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0043323260000000001 2024-09-21T12:24:15.837 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.837 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.837 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:24:15.837 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.082257+0000", 2024-09-21T12:24:15.837 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0025462739999999998 2024-09-21T12:24:15.837 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.837 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.837 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.837 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.092913+0000", 2024-09-21T12:24:15.837 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010656025 2024-09-21T12:24:15.837 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.837 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.837 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:24:15.838 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.092966+0000", 2024-09-21T12:24:15.838 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2296000000000002e-05 2024-09-21T12:24:15.838 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.838 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.838 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:24:15.838 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.094659+0000", 2024-09-21T12:24:15.838 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001692991 2024-09-21T12:24:15.838 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.838 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:15.838 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.838 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.838 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.838 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:24418 7.3 7:c9c1f723:::benchmark_data_smithi114_78432_object24417:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:24:15.838 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:40.048102+0000", 2024-09-21T12:24:15.838 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 155.59181356100001, 2024-09-21T12:24:15.839 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0488444459999999, 2024-09-21T12:24:15.839 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:24:15.839 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:24:15.839 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:15.839 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:24:15.839 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:24:15.839 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:15.839 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 24418 2024-09-21T12:24:15.839 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.839 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:24:15.839 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.839 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:24:15.839 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.048102+0000", 2024-09-21T12:24:15.839 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.839 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.839 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.840 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:24:15.840 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.048102+0000", 2024-09-21T12:24:15.840 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.840 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.840 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.840 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:24:15.840 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.048104+0000", 2024-09-21T12:24:15.840 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4729999999999999e-06 2024-09-21T12:24:15.840 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.840 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.840 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:24:15.840 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.048127+0000", 2024-09-21T12:24:15.840 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3088999999999999e-05 2024-09-21T12:24:15.840 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.840 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.840 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:24:15.840 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.048143+0000", 2024-09-21T12:24:15.841 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6314e-05 2024-09-21T12:24:15.841 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.841 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.841 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:24:15.841 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.048149+0000", 2024-09-21T12:24:15.841 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5579999999999997e-06 2024-09-21T12:24:15.841 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.841 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.841 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.841 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.048224+0000", 2024-09-21T12:24:15.841 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4055999999999997e-05 2024-09-21T12:24:15.841 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.841 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.841 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:24:15.841 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.048235+0000", 2024-09-21T12:24:15.841 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1354e-05 2024-09-21T12:24:15.842 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.842 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.842 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.842 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.032739+0000", 2024-09-21T12:24:15.842 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98450402000000004 2024-09-21T12:24:15.842 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.842 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.842 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:24:15.842 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.032746+0000", 2024-09-21T12:24:15.842 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5510000000000002e-06 2024-09-21T12:24:15.842 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.842 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.842 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.842 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.076754+0000", 2024-09-21T12:24:15.843 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0440078779999999 2024-09-21T12:24:15.843 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.843 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.843 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:24:15.843 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.076899+0000", 2024-09-21T12:24:15.843 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000145122 2024-09-21T12:24:15.843 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.843 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.843 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:24:15.843 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.078384+0000", 2024-09-21T12:24:15.843 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001484527 2024-09-21T12:24:15.843 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.843 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.843 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:24:15.843 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.083050+0000", 2024-09-21T12:24:15.843 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0046663069999999997 2024-09-21T12:24:15.843 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.844 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.844 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.844 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.083099+0000", 2024-09-21T12:24:15.844 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9089999999999999e-05 2024-09-21T12:24:15.844 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.844 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.844 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.844 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.094756+0000", 2024-09-21T12:24:15.844 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.011656992999999999 2024-09-21T12:24:15.844 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.844 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.844 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:24:15.844 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.094807+0000", 2024-09-21T12:24:15.844 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.109e-05 2024-09-21T12:24:15.844 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.844 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.845 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:24:15.845 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.096946+0000", 2024-09-21T12:24:15.845 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0021390239999999998 2024-09-21T12:24:15.845 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.845 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:15.845 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.845 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.845 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.845 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:24423 7.1 7:82b639fa:::benchmark_data_smithi114_78432_object24422:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:24:15.845 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:40.057046+0000", 2024-09-21T12:24:15.845 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 155.582869585, 2024-09-21T12:24:15.845 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0306053990000001, 2024-09-21T12:24:15.845 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:24:15.845 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:24:15.845 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:15.846 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:24:15.846 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:24:15.846 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:15.846 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 24423 2024-09-21T12:24:15.846 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.846 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:24:15.846 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.846 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:24:15.846 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.057046+0000", 2024-09-21T12:24:15.846 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.846 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.846 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.846 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:24:15.846 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.057046+0000", 2024-09-21T12:24:15.846 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.847 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.847 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.847 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:24:15.847 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.057048+0000", 2024-09-21T12:24:15.847 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6169999999999999e-06 2024-09-21T12:24:15.847 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.847 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.847 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:24:15.847 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.057070+0000", 2024-09-21T12:24:15.847 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2320999999999999e-05 2024-09-21T12:24:15.847 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.847 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.847 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:24:15.848 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.057083+0000", 2024-09-21T12:24:15.848 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2917e-05 2024-09-21T12:24:15.848 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.848 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.848 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:24:15.848 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.057091+0000", 2024-09-21T12:24:15.848 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.661e-06 2024-09-21T12:24:15.848 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.848 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.848 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.848 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.057161+0000", 2024-09-21T12:24:15.848 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0339000000000005e-05 2024-09-21T12:24:15.848 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.848 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.848 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:24:15.849 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.057173+0000", 2024-09-21T12:24:15.849 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2061e-05 2024-09-21T12:24:15.849 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.849 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.849 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.849 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.024342+0000", 2024-09-21T12:24:15.849 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.96716935900000001 2024-09-21T12:24:15.849 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.849 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.849 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:24:15.849 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.024353+0000", 2024-09-21T12:24:15.849 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0808999999999999e-05 2024-09-21T12:24:15.849 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.850 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.850 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.850 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.079289+0000", 2024-09-21T12:24:15.850 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0549363009999999 2024-09-21T12:24:15.850 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.850 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.850 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:24:15.850 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.079427+0000", 2024-09-21T12:24:15.850 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013735599999999999 2024-09-21T12:24:15.850 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.850 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.850 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:24:15.850 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.079592+0000", 2024-09-21T12:24:15.851 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016532 2024-09-21T12:24:15.851 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.851 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.851 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.851 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.082335+0000", 2024-09-21T12:24:15.851 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0027429020000000002 2024-09-21T12:24:15.851 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.851 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.851 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:24:15.851 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.082510+0000", 2024-09-21T12:24:15.851 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017516 2024-09-21T12:24:15.851 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.851 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.851 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.852 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.086126+0000", 2024-09-21T12:24:15.852 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0036157950000000002 2024-09-21T12:24:15.852 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.852 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.852 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:24:15.852 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.086183+0000", 2024-09-21T12:24:15.852 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7157000000000002e-05 2024-09-21T12:24:15.852 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.852 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.852 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:24:15.852 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.087651+0000", 2024-09-21T12:24:15.852 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001468324 2024-09-21T12:24:15.853 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.853 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:15.853 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.853 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.853 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.853 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.14478.0:1115 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 94620848893312] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-21T12:24:15.853 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:40.364223+0000", 2024-09-21T12:24:15.853 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 155.275693066, 2024-09-21T12:24:15.853 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.66897492700000005, 2024-09-21T12:24:15.853 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:24:15.853 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:24:15.853 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:24:15.853 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:24:15.853 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.14478", 2024-09-21T12:24:15.854 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:24:15.854 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 1115 2024-09-21T12:24:15.854 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.854 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:24:15.854 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.854 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:24:15.854 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364223+0000", 2024-09-21T12:24:15.854 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.854 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.854 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.854 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:24:15.854 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364223+0000", 2024-09-21T12:24:15.854 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.854 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.855 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.855 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:24:15.855 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364225+0000", 2024-09-21T12:24:15.855 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3410000000000001e-06 2024-09-21T12:24:15.855 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.855 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.855 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:24:15.855 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364232+0000", 2024-09-21T12:24:15.855 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0879999999999997e-06 2024-09-21T12:24:15.855 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.855 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.855 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:24:15.855 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364254+0000", 2024-09-21T12:24:15.855 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1945000000000001e-05 2024-09-21T12:24:15.855 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.856 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.856 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:24:15.856 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364264+0000", 2024-09-21T12:24:15.856 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.781e-06 2024-09-21T12:24:15.856 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.856 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.856 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.856 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364516+0000", 2024-09-21T12:24:15.856 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00025249199999999999 2024-09-21T12:24:15.856 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.856 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.856 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:24:15.857 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364547+0000", 2024-09-21T12:24:15.857 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.0797999999999999e-05 2024-09-21T12:24:15.857 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.857 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.857 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.857 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.030239+0000", 2024-09-21T12:24:15.857 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.66569158500000003 2024-09-21T12:24:15.857 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.857 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.857 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for degraded object", 2024-09-21T12:24:15.857 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.030551+0000", 2024-09-21T12:24:15.857 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00031200999999999999 2024-09-21T12:24:15.857 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.857 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.858 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.858 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.032993+0000", 2024-09-21T12:24:15.858 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0024427419999999999 2024-09-21T12:24:15.858 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.858 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.858 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:24:15.858 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.033084+0000", 2024-09-21T12:24:15.858 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1180000000000005e-05 2024-09-21T12:24:15.858 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.858 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.858 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:24:15.859 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.033197+0000", 2024-09-21T12:24:15.859 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000112965 2024-09-21T12:24:15.859 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.859 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:15.859 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.859 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.859 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.859 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.14478.0:1122 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94620848893312 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:24:15.859 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:40.364792+0000", 2024-09-21T12:24:15.859 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 155.27512358199999, 2024-09-21T12:24:15.859 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.66963756900000004, 2024-09-21T12:24:15.859 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:24:15.859 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:24:15.859 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:24:15.859 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:24:15.860 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.14478", 2024-09-21T12:24:15.860 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:24:15.860 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 1122 2024-09-21T12:24:15.860 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.860 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:24:15.860 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.860 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:24:15.860 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364792+0000", 2024-09-21T12:24:15.860 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.860 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.860 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.860 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:24:15.860 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364792+0000", 2024-09-21T12:24:15.860 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.860 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.861 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.861 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:24:15.861 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364793+0000", 2024-09-21T12:24:15.861 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.7400000000000002e-07 2024-09-21T12:24:15.861 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.861 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.861 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:24:15.861 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364797+0000", 2024-09-21T12:24:15.861 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8360000000000002e-06 2024-09-21T12:24:15.861 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.861 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.861 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:24:15.861 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364809+0000", 2024-09-21T12:24:15.861 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1849e-05 2024-09-21T12:24:15.862 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.862 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.862 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:24:15.862 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364814+0000", 2024-09-21T12:24:15.862 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1370000000000001e-06 2024-09-21T12:24:15.862 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.862 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.862 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.862 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364855+0000", 2024-09-21T12:24:15.862 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1736999999999997e-05 2024-09-21T12:24:15.862 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.862 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.862 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:24:15.862 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.364862+0000", 2024-09-21T12:24:15.863 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4049999999999997e-06 2024-09-21T12:24:15.863 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.863 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.863 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.863 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.030691+0000", 2024-09-21T12:24:15.863 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.66582926799999997 2024-09-21T12:24:15.863 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.863 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.863 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for degraded object", 2024-09-21T12:24:15.863 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.030745+0000", 2024-09-21T12:24:15.863 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3947999999999999e-05 2024-09-21T12:24:15.863 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.863 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.863 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.864 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.033242+0000", 2024-09-21T12:24:15.864 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0024973220000000002 2024-09-21T12:24:15.864 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.864 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.864 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:24:15.864 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.033302+0000", 2024-09-21T12:24:15.864 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9738000000000003e-05 2024-09-21T12:24:15.864 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.864 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.864 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:24:15.864 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.034430+0000", 2024-09-21T12:24:15.864 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0011274550000000001 2024-09-21T12:24:15.864 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.865 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:15.865 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.865 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.865 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.865 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46120 7.3 7:cc2d5e26:::benchmark_data_smithi114_78432_object46119:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-21T12:24:15.865 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:59.864654+0000", 2024-09-21T12:24:15.865 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 135.77526145600001, 2024-09-21T12:24:15.865 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.006362438, 2024-09-21T12:24:15.865 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:24:15.865 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:24:15.865 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:15.865 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:24:15.865 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:24:15.865 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:15.865 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46120 2024-09-21T12:24:15.865 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.866 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:24:15.866 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.866 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:24:15.866 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.864654+0000", 2024-09-21T12:24:15.866 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.866 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.866 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.866 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:24:15.866 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.864654+0000", 2024-09-21T12:24:15.866 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.866 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.866 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.866 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:24:15.866 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.864656+0000", 2024-09-21T12:24:15.866 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5999999999999999e-06 2024-09-21T12:24:15.867 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.867 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.867 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:24:15.867 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.864678+0000", 2024-09-21T12:24:15.867 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2048999999999998e-05 2024-09-21T12:24:15.867 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.867 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.867 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:24:15.867 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.864690+0000", 2024-09-21T12:24:15.867 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2533e-05 2024-09-21T12:24:15.867 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.867 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.867 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:24:15.867 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.864698+0000", 2024-09-21T12:24:15.867 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4950000000000002e-06 2024-09-21T12:24:15.868 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.868 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.868 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.868 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.864757+0000", 2024-09-21T12:24:15.868 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9276e-05 2024-09-21T12:24:15.868 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.868 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.868 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:24:15.868 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.864767+0000", 2024-09-21T12:24:15.868 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0134e-05 2024-09-21T12:24:15.868 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.868 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.868 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.868 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.865518+0000", 2024-09-21T12:24:15.868 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0007512750000001 2024-09-21T12:24:15.869 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.869 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.869 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:24:15.869 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.865632+0000", 2024-09-21T12:24:15.869 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000113299 2024-09-21T12:24:15.869 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.869 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.869 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:24:15.869 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.865781+0000", 2024-09-21T12:24:15.869 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000149386 2024-09-21T12:24:15.869 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.869 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.869 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.869 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.867635+0000", 2024-09-21T12:24:15.869 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001853873 2024-09-21T12:24:15.869 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.870 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.870 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:24:15.870 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.870752+0000", 2024-09-21T12:24:15.870 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0031166980000000002 2024-09-21T12:24:15.870 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.870 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.870 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.870 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.870919+0000", 2024-09-21T12:24:15.870 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016756599999999999 2024-09-21T12:24:15.870 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.870 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.870 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:24:15.870 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.870978+0000", 2024-09-21T12:24:15.870 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8734999999999999e-05 2024-09-21T12:24:15.870 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.871 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.871 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:24:15.871 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.871017+0000", 2024-09-21T12:24:15.871 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8519000000000002e-05 2024-09-21T12:24:15.871 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.871 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:15.871 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.871 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.871 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.871 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46176 7.3 7:cbc365d5:::benchmark_data_smithi114_78432_object46175:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-21T12:24:15.871 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:59.906213+0000", 2024-09-21T12:24:15.871 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 135.73370240599999, 2024-09-21T12:24:15.871 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.96654458399999998, 2024-09-21T12:24:15.871 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:24:15.871 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:24:15.872 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:15.872 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:24:15.872 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:24:15.872 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:15.872 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46176 2024-09-21T12:24:15.872 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.872 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:24:15.872 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.872 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:24:15.872 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.906213+0000", 2024-09-21T12:24:15.872 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.872 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.872 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.872 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:24:15.873 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.906213+0000", 2024-09-21T12:24:15.873 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.873 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.873 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.873 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:24:15.873 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.906215+0000", 2024-09-21T12:24:15.873 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6139999999999999e-06 2024-09-21T12:24:15.873 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.873 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.873 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:24:15.873 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.906236+0000", 2024-09-21T12:24:15.873 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1457000000000001e-05 2024-09-21T12:24:15.873 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.873 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.874 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:24:15.874 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.906249+0000", 2024-09-21T12:24:15.874 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2252999999999999e-05 2024-09-21T12:24:15.874 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.874 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.874 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:24:15.874 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.906256+0000", 2024-09-21T12:24:15.874 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4839999999999998e-06 2024-09-21T12:24:15.874 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.874 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.874 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.874 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.906306+0000", 2024-09-21T12:24:15.874 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9719000000000003e-05 2024-09-21T12:24:15.874 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.875 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.875 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:24:15.875 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.906314+0000", 2024-09-21T12:24:15.875 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9289999999999998e-06 2024-09-21T12:24:15.875 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.875 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.875 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.875 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.867653+0000", 2024-09-21T12:24:15.875 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.96133975599999999 2024-09-21T12:24:15.875 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.875 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.875 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:24:15.875 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.867717+0000", 2024-09-21T12:24:15.875 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3133999999999996e-05 2024-09-21T12:24:15.875 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.876 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.876 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:24:15.876 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.867796+0000", 2024-09-21T12:24:15.876 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9532999999999999e-05 2024-09-21T12:24:15.876 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.876 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.876 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.876 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.870868+0000", 2024-09-21T12:24:15.876 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0030720370000000001 2024-09-21T12:24:15.876 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.876 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.876 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:24:15.876 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.870880+0000", 2024-09-21T12:24:15.876 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1610999999999999e-05 2024-09-21T12:24:15.876 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.877 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.877 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.877 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.872686+0000", 2024-09-21T12:24:15.877 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0018067280000000001 2024-09-21T12:24:15.877 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.877 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.877 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:24:15.877 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.872719+0000", 2024-09-21T12:24:15.877 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2938999999999998e-05 2024-09-21T12:24:15.877 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.877 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.877 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:24:15.877 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.872758+0000", 2024-09-21T12:24:15.877 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8389999999999997e-05 2024-09-21T12:24:15.878 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.878 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:15.878 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.878 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.878 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.878 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46206 7.3 7:cc04789a:::benchmark_data_smithi114_78432_object46205:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-21T12:24:15.878 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:59.928010+0000", 2024-09-21T12:24:15.878 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 135.71190573800001, 2024-09-21T12:24:15.878 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.94644473799999995, 2024-09-21T12:24:15.878 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:24:15.878 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:24:15.878 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:15.878 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:24:15.878 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:24:15.879 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:15.879 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46206 2024-09-21T12:24:15.879 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.879 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:24:15.879 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.879 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:24:15.879 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.928010+0000", 2024-09-21T12:24:15.879 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.879 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.879 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.879 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:24:15.879 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.928010+0000", 2024-09-21T12:24:15.879 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.879 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.879 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.880 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:24:15.880 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.928011+0000", 2024-09-21T12:24:15.880 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6050000000000001e-06 2024-09-21T12:24:15.880 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.880 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.880 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:24:15.880 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.928039+0000", 2024-09-21T12:24:15.880 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.7185000000000001e-05 2024-09-21T12:24:15.880 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.880 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.880 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:24:15.880 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.928051+0000", 2024-09-21T12:24:15.880 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2622e-05 2024-09-21T12:24:15.880 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.880 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.880 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:24:15.881 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.928059+0000", 2024-09-21T12:24:15.881 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2409999999999998e-06 2024-09-21T12:24:15.881 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.881 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.881 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.881 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.928121+0000", 2024-09-21T12:24:15.881 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2769999999999997e-05 2024-09-21T12:24:15.881 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.881 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.881 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:24:15.881 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.928131+0000", 2024-09-21T12:24:15.881 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.5459999999999997e-06 2024-09-21T12:24:15.881 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.881 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.882 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.882 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.869100+0000", 2024-09-21T12:24:15.882 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.94096877300000004 2024-09-21T12:24:15.882 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.882 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.882 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:24:15.882 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.869242+0000", 2024-09-21T12:24:15.882 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014223799999999999 2024-09-21T12:24:15.882 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.882 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.882 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:24:15.882 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.869383+0000", 2024-09-21T12:24:15.882 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000140755 2024-09-21T12:24:15.882 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.883 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.883 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.883 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.871074+0000", 2024-09-21T12:24:15.883 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0016910969999999999 2024-09-21T12:24:15.883 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.883 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.883 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:24:15.883 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.874287+0000", 2024-09-21T12:24:15.883 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0032134260000000001 2024-09-21T12:24:15.883 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.883 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.883 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.883 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.874360+0000", 2024-09-21T12:24:15.883 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2896000000000001e-05 2024-09-21T12:24:15.883 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.884 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.884 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:24:15.884 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.874406+0000", 2024-09-21T12:24:15.884 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5509000000000001e-05 2024-09-21T12:24:15.884 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.884 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.884 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:24:15.884 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.874455+0000", 2024-09-21T12:24:15.884 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9075e-05 2024-09-21T12:24:15.884 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.884 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:15.884 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.884 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.884 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.884 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46253 7.3 7:cea9647b:::benchmark_data_smithi114_78432_object46252:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-21T12:24:15.885 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:00.004002+0000", 2024-09-21T12:24:15.885 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 135.63591333299999, 2024-09-21T12:24:15.885 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.87111806700000005, 2024-09-21T12:24:15.885 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:24:15.885 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:24:15.885 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:15.885 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:24:15.885 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:24:15.885 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:15.885 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46253 2024-09-21T12:24:15.885 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.885 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:24:15.885 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.885 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:24:15.885 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.004002+0000", 2024-09-21T12:24:15.886 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.886 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.886 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.886 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:24:15.886 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.004002+0000", 2024-09-21T12:24:15.886 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.886 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.886 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.886 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:24:15.886 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.004004+0000", 2024-09-21T12:24:15.886 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.451e-06 2024-09-21T12:24:15.886 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.886 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.886 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:24:15.886 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.004036+0000", 2024-09-21T12:24:15.887 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2790000000000003e-05 2024-09-21T12:24:15.887 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.887 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.887 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:24:15.887 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.004051+0000", 2024-09-21T12:24:15.887 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4138000000000001e-05 2024-09-21T12:24:15.887 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.887 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.887 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:24:15.887 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.004057+0000", 2024-09-21T12:24:15.887 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9569999999999999e-06 2024-09-21T12:24:15.887 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.887 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.887 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.887 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.004117+0000", 2024-09-21T12:24:15.888 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0891999999999999e-05 2024-09-21T12:24:15.888 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.888 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.888 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:24:15.888 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.004128+0000", 2024-09-21T12:24:15.888 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0939e-05 2024-09-21T12:24:15.888 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.888 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.888 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.888 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.871110+0000", 2024-09-21T12:24:15.888 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.86698113200000004 2024-09-21T12:24:15.888 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.888 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.888 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:24:15.888 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.871213+0000", 2024-09-21T12:24:15.889 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010368400000000001 2024-09-21T12:24:15.889 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.889 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.889 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:24:15.889 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.871319+0000", 2024-09-21T12:24:15.889 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010538 2024-09-21T12:24:15.889 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.889 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.889 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.889 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.874322+0000", 2024-09-21T12:24:15.889 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0030033320000000001 2024-09-21T12:24:15.889 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.889 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.889 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:24:15.890 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.874332+0000", 2024-09-21T12:24:15.890 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6569999999999992e-06 2024-09-21T12:24:15.890 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.890 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.890 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.890 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.875030+0000", 2024-09-21T12:24:15.890 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00069842500000000002 2024-09-21T12:24:15.890 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.890 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.890 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:24:15.890 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.875083+0000", 2024-09-21T12:24:15.890 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3387999999999998e-05 2024-09-21T12:24:15.890 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.890 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.891 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:24:15.891 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.875120+0000", 2024-09-21T12:24:15.891 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.6902e-05 2024-09-21T12:24:15.891 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.891 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:15.891 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.891 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.891 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.891 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46526 7.1 7:859c7591:::benchmark_data_smithi114_78432_object46525:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:24:15.891 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.797069+0000", 2024-09-21T12:24:15.891 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 133.84284683199999, 2024-09-21T12:24:15.891 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.020480762, 2024-09-21T12:24:15.891 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:24:15.891 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:24:15.891 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:15.892 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:24:15.892 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:24:15.892 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:15.892 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46526 2024-09-21T12:24:15.892 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.892 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:24:15.892 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.892 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:24:15.892 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.797069+0000", 2024-09-21T12:24:15.892 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.892 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.892 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.892 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:24:15.892 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.797069+0000", 2024-09-21T12:24:15.893 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.893 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.893 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.893 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:24:15.893 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.797070+0000", 2024-09-21T12:24:15.893 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6339999999999999e-06 2024-09-21T12:24:15.893 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.893 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.893 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:24:15.893 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.797094+0000", 2024-09-21T12:24:15.893 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3870999999999999e-05 2024-09-21T12:24:15.893 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.893 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.893 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:24:15.893 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.797111+0000", 2024-09-21T12:24:15.893 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6495999999999999e-05 2024-09-21T12:24:15.894 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.894 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.894 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:24:15.894 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.797119+0000", 2024-09-21T12:24:15.894 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.3440000000000001e-06 2024-09-21T12:24:15.894 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.894 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.894 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.894 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.797189+0000", 2024-09-21T12:24:15.894 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9733999999999993e-05 2024-09-21T12:24:15.894 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.894 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.894 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:24:15.895 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.797201+0000", 2024-09-21T12:24:15.895 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1803e-05 2024-09-21T12:24:15.895 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.895 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.895 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.895 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.811507+0000", 2024-09-21T12:24:15.895 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0143067020000001 2024-09-21T12:24:15.895 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.895 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.895 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:24:15.895 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.811633+0000", 2024-09-21T12:24:15.895 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012554600000000001 2024-09-21T12:24:15.895 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.895 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.895 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:24:15.896 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.811788+0000", 2024-09-21T12:24:15.896 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015533499999999999 2024-09-21T12:24:15.896 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.896 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.896 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.896 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.815343+0000", 2024-09-21T12:24:15.896 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0035549879999999998 2024-09-21T12:24:15.896 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.896 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.896 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.896 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.815402+0000", 2024-09-21T12:24:15.896 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8409999999999998e-05 2024-09-21T12:24:15.896 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.896 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.896 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:24:15.897 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.817386+0000", 2024-09-21T12:24:15.897 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001984478 2024-09-21T12:24:15.897 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.897 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.897 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:24:15.897 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.817452+0000", 2024-09-21T12:24:15.897 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5679000000000006e-05 2024-09-21T12:24:15.897 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.897 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.897 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:24:15.897 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.817550+0000", 2024-09-21T12:24:15.897 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7742e-05 2024-09-21T12:24:15.897 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.897 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:15.897 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.898 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.898 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.898 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46536 7.1 7:84370686:::benchmark_data_smithi114_78432_object46535:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:24:15.898 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.810616+0000", 2024-09-21T12:24:15.898 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 133.829299263, 2024-09-21T12:24:15.898 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0092227149999999, 2024-09-21T12:24:15.898 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:24:15.898 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:24:15.898 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:15.898 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:24:15.898 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:24:15.898 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:15.898 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46536 2024-09-21T12:24:15.898 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.898 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:24:15.899 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.899 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:24:15.899 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.810616+0000", 2024-09-21T12:24:15.899 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.899 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.899 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.899 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:24:15.899 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.810616+0000", 2024-09-21T12:24:15.899 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.899 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.899 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.899 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:24:15.899 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.810617+0000", 2024-09-21T12:24:15.899 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0999999999999998e-07 2024-09-21T12:24:15.900 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.900 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.900 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:24:15.900 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.810641+0000", 2024-09-21T12:24:15.900 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3589000000000001e-05 2024-09-21T12:24:15.900 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.900 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.900 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:24:15.900 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.810660+0000", 2024-09-21T12:24:15.900 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9089e-05 2024-09-21T12:24:15.900 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.901 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.901 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:24:15.901 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.810670+0000", 2024-09-21T12:24:15.901 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0694e-05 2024-09-21T12:24:15.901 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.901 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.901 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.901 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.813144+0000", 2024-09-21T12:24:15.901 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002473598 2024-09-21T12:24:15.901 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.901 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.901 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:24:15.901 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.813155+0000", 2024-09-21T12:24:15.901 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0781999999999999e-05 2024-09-21T12:24:15.902 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.902 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.902 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.902 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.813819+0000", 2024-09-21T12:24:15.902 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0006642649999999 2024-09-21T12:24:15.902 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.902 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.902 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:24:15.902 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.813931+0000", 2024-09-21T12:24:15.902 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000112076 2024-09-21T12:24:15.902 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.902 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.902 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:24:15.902 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.814058+0000", 2024-09-21T12:24:15.903 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000127174 2024-09-21T12:24:15.903 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.903 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.903 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.903 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.819658+0000", 2024-09-21T12:24:15.903 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0056001870000000004 2024-09-21T12:24:15.903 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.903 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.903 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.903 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.819725+0000", 2024-09-21T12:24:15.903 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6711999999999994e-05 2024-09-21T12:24:15.903 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.903 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.904 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:24:15.904 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.819741+0000", 2024-09-21T12:24:15.904 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5341000000000001e-05 2024-09-21T12:24:15.904 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.904 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.904 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:24:15.904 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.819792+0000", 2024-09-21T12:24:15.904 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1418999999999997e-05 2024-09-21T12:24:15.904 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.904 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.904 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:24:15.904 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.819839+0000", 2024-09-21T12:24:15.904 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7079000000000001e-05 2024-09-21T12:24:15.904 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.905 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:15.905 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.905 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.905 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.905 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46544 7.1 7:87ac3b5d:::benchmark_data_smithi114_78432_object46543:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:24:15.905 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.817944+0000", 2024-09-21T12:24:15.905 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 133.82197196800001, 2024-09-21T12:24:15.905 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.002154743, 2024-09-21T12:24:15.905 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:24:15.905 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:24:15.905 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:15.905 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:24:15.905 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:24:15.905 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:15.905 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46544 2024-09-21T12:24:15.905 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.906 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:24:15.906 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.906 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:24:15.906 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.817944+0000", 2024-09-21T12:24:15.906 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.906 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.906 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.906 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:24:15.906 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.817944+0000", 2024-09-21T12:24:15.906 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.906 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.906 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.907 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:24:15.907 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.817945+0000", 2024-09-21T12:24:15.907 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.424e-06 2024-09-21T12:24:15.907 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.907 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.907 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:24:15.907 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.817965+0000", 2024-09-21T12:24:15.907 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0428000000000002e-05 2024-09-21T12:24:15.907 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.907 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.907 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:24:15.907 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.817975+0000", 2024-09-21T12:24:15.907 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1120000000000002e-06 2024-09-21T12:24:15.907 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.908 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.908 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:24:15.908 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.817980+0000", 2024-09-21T12:24:15.908 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6829999999999996e-06 2024-09-21T12:24:15.908 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.908 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.908 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.908 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.818106+0000", 2024-09-21T12:24:15.908 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012569799999999999 2024-09-21T12:24:15.908 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.908 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.908 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:24:15.908 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.818116+0000", 2024-09-21T12:24:15.908 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7110000000000007e-06 2024-09-21T12:24:15.908 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.909 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.909 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.909 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.815448+0000", 2024-09-21T12:24:15.909 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99733251899999997 2024-09-21T12:24:15.909 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.909 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.909 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:24:15.909 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.815583+0000", 2024-09-21T12:24:15.909 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013521900000000001 2024-09-21T12:24:15.909 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.909 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.909 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:24:15.909 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.815750+0000", 2024-09-21T12:24:15.909 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016642600000000001 2024-09-21T12:24:15.910 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.910 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.910 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:24:15.910 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.819850+0000", 2024-09-21T12:24:15.910 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0041004489999999999 2024-09-21T12:24:15.910 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.910 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.910 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.910 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.819916+0000", 2024-09-21T12:24:15.910 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6057999999999997e-05 2024-09-21T12:24:15.910 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.910 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.910 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.910 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.819981+0000", 2024-09-21T12:24:15.910 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4624999999999999e-05 2024-09-21T12:24:15.911 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.911 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.911 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:24:15.911 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.820064+0000", 2024-09-21T12:24:15.911 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.3144000000000002e-05 2024-09-21T12:24:15.911 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.911 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.911 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:24:15.911 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.820098+0000", 2024-09-21T12:24:15.911 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4246999999999999e-05 2024-09-21T12:24:15.911 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.911 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:15.911 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.911 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.911 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.912 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46548 7.1 7:86537d3f:::benchmark_data_smithi114_78432_object46547:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:24:15.912 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.822112+0000", 2024-09-21T12:24:15.912 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 133.81780386400001, 2024-09-21T12:24:15.912 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.002139297, 2024-09-21T12:24:15.912 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:24:15.912 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:24:15.912 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:15.912 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:24:15.912 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:24:15.912 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:15.912 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46548 2024-09-21T12:24:15.912 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.912 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:24:15.912 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.913 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:24:15.913 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.822112+0000", 2024-09-21T12:24:15.913 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.913 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.913 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.913 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:24:15.913 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.822112+0000", 2024-09-21T12:24:15.913 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.913 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.913 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.913 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:24:15.913 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.822113+0000", 2024-09-21T12:24:15.913 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.299e-06 2024-09-21T12:24:15.913 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.914 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.914 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:24:15.914 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.822143+0000", 2024-09-21T12:24:15.914 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.9869000000000002e-05 2024-09-21T12:24:15.914 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.914 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.914 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:24:15.914 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.822160+0000", 2024-09-21T12:24:15.914 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7450000000000001e-05 2024-09-21T12:24:15.914 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.914 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.914 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:24:15.914 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.822168+0000", 2024-09-21T12:24:15.915 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.0669999999999992e-06 2024-09-21T12:24:15.915 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.915 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.915 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.915 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.822328+0000", 2024-09-21T12:24:15.915 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015988999999999999 2024-09-21T12:24:15.915 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.915 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.915 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:24:15.915 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.822338+0000", 2024-09-21T12:24:15.915 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7799999999999995e-06 2024-09-21T12:24:15.915 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.915 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.915 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.915 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.817706+0000", 2024-09-21T12:24:15.916 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99536773599999995 2024-09-21T12:24:15.916 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.916 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.916 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:24:15.916 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.817826+0000", 2024-09-21T12:24:15.916 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000120677 2024-09-21T12:24:15.916 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.916 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.916 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:24:15.916 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.817957+0000", 2024-09-21T12:24:15.916 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013050099999999999 2024-09-21T12:24:15.916 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.916 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.916 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.917 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.820152+0000", 2024-09-21T12:24:15.917 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0021950199999999998 2024-09-21T12:24:15.917 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.917 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.917 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:24:15.917 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.820162+0000", 2024-09-21T12:24:15.917 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0032e-05 2024-09-21T12:24:15.917 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.917 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.917 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.917 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.824158+0000", 2024-09-21T12:24:15.917 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0039958099999999998 2024-09-21T12:24:15.917 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.917 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.917 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:24:15.917 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.824209+0000", 2024-09-21T12:24:15.918 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0997e-05 2024-09-21T12:24:15.918 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.918 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.918 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:24:15.918 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.824251+0000", 2024-09-21T12:24:15.918 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.2169000000000002e-05 2024-09-21T12:24:15.918 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.918 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:15.918 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.918 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.918 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.918 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46555 7.1 7:86dd50c3:::benchmark_data_smithi114_78432_object46554:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:24:15.918 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.826502+0000", 2024-09-21T12:24:15.918 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 133.813413664, 2024-09-21T12:24:15.918 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.002009758, 2024-09-21T12:24:15.919 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:24:15.919 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:24:15.919 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:15.919 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:24:15.919 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:24:15.919 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:15.919 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46555 2024-09-21T12:24:15.919 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.919 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:24:15.919 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.919 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:24:15.919 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.826502+0000", 2024-09-21T12:24:15.919 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.919 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.919 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.919 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:24:15.920 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.826502+0000", 2024-09-21T12:24:15.920 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.920 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.920 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.920 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:24:15.920 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.826503+0000", 2024-09-21T12:24:15.920 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.5600000000000004e-07 2024-09-21T12:24:15.920 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.920 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.920 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:24:15.920 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.826527+0000", 2024-09-21T12:24:15.920 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3975999999999999e-05 2024-09-21T12:24:15.920 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.920 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.920 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:24:15.921 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.826543+0000", 2024-09-21T12:24:15.921 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6008e-05 2024-09-21T12:24:15.921 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.921 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.921 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:24:15.921 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.826551+0000", 2024-09-21T12:24:15.921 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6720000000000004e-06 2024-09-21T12:24:15.921 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.921 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.921 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.921 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.826673+0000", 2024-09-21T12:24:15.921 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000122774 2024-09-21T12:24:15.921 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.921 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.921 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:24:15.921 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.826682+0000", 2024-09-21T12:24:15.922 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1379999999999996e-06 2024-09-21T12:24:15.922 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.922 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.922 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.922 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.820189+0000", 2024-09-21T12:24:15.922 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99350612699999996 2024-09-21T12:24:15.922 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.922 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.922 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:24:15.922 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.820310+0000", 2024-09-21T12:24:15.922 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000121172 2024-09-21T12:24:15.922 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.922 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.922 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:24:15.922 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.820483+0000", 2024-09-21T12:24:15.923 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000173079 2024-09-21T12:24:15.923 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.923 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.923 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:24:15.923 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.824267+0000", 2024-09-21T12:24:15.923 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0037840059999999999 2024-09-21T12:24:15.923 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.923 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.923 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.923 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.828060+0000", 2024-09-21T12:24:15.923 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00379314 2024-09-21T12:24:15.923 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.923 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.923 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.923 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.828424+0000", 2024-09-21T12:24:15.923 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00036447200000000002 2024-09-21T12:24:15.924 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.924 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.924 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:24:15.924 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.828474+0000", 2024-09-21T12:24:15.924 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9218999999999998e-05 2024-09-21T12:24:15.924 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.924 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.924 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:24:15.924 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.828512+0000", 2024-09-21T12:24:15.924 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8019000000000003e-05 2024-09-21T12:24:15.924 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.924 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:15.924 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.924 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.924 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.925 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46559 7.1 7:8b7c05c1:::benchmark_data_smithi114_78432_object46558:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:24:15.925 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.827486+0000", 2024-09-21T12:24:15.925 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 133.81242935500001, 2024-09-21T12:24:15.925 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.001462729, 2024-09-21T12:24:15.925 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:24:15.925 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:24:15.925 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:15.925 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:24:15.925 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:24:15.925 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:15.925 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46559 2024-09-21T12:24:15.925 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.925 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:24:15.925 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.925 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:24:15.925 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.827486+0000", 2024-09-21T12:24:15.925 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.926 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.926 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.926 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:24:15.926 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.827486+0000", 2024-09-21T12:24:15.926 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.926 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.926 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.926 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:24:15.926 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.827488+0000", 2024-09-21T12:24:15.926 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.694e-06 2024-09-21T12:24:15.926 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.926 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.926 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:24:15.926 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.827510+0000", 2024-09-21T12:24:15.926 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2019e-05 2024-09-21T12:24:15.927 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.927 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.927 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:24:15.927 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.827524+0000", 2024-09-21T12:24:15.927 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3789e-05 2024-09-21T12:24:15.927 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.927 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.927 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:24:15.927 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.827531+0000", 2024-09-21T12:24:15.927 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4680000000000003e-06 2024-09-21T12:24:15.927 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.927 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.927 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.927 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.827592+0000", 2024-09-21T12:24:15.927 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0396000000000002e-05 2024-09-21T12:24:15.927 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.928 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.928 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:24:15.928 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.827601+0000", 2024-09-21T12:24:15.928 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0389999999999993e-06 2024-09-21T12:24:15.928 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.928 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.928 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.928 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.822243+0000", 2024-09-21T12:24:15.928 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99464195399999999 2024-09-21T12:24:15.928 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.928 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.928 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:24:15.928 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.822347+0000", 2024-09-21T12:24:15.928 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000104283 2024-09-21T12:24:15.928 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.928 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.929 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:24:15.929 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.822495+0000", 2024-09-21T12:24:15.929 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000147737 2024-09-21T12:24:15.929 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.929 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.929 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:24:15.929 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.828006+0000", 2024-09-21T12:24:15.929 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0055110910000000001 2024-09-21T12:24:15.929 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.929 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.929 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.929 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.828124+0000", 2024-09-21T12:24:15.929 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000118528 2024-09-21T12:24:15.929 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.929 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.929 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.930 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.828548+0000", 2024-09-21T12:24:15.930 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00042400199999999998 2024-09-21T12:24:15.930 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.930 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.930 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:24:15.930 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.828593+0000", 2024-09-21T12:24:15.930 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4336999999999999e-05 2024-09-21T12:24:15.930 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.930 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.930 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:24:15.930 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.828949+0000", 2024-09-21T12:24:15.930 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00035639200000000003 2024-09-21T12:24:15.930 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.930 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:15.930 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.931 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.931 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.931 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46583 7.1 7:889d1627:::benchmark_data_smithi114_78432_object46582:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:24:15.931 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.845169+0000", 2024-09-21T12:24:15.931 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 133.794746215, 2024-09-21T12:24:15.931 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98807372699999996, 2024-09-21T12:24:15.931 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:24:15.931 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:24:15.931 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:15.931 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:24:15.931 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:24:15.931 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:15.931 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46583 2024-09-21T12:24:15.931 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.931 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:24:15.932 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.932 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:24:15.932 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.845169+0000", 2024-09-21T12:24:15.932 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.932 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.932 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.932 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:24:15.932 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.845169+0000", 2024-09-21T12:24:15.932 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.932 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.932 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.932 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:24:15.932 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.845170+0000", 2024-09-21T12:24:15.932 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.850000000000001e-07 2024-09-21T12:24:15.932 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.932 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.933 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:24:15.933 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.845191+0000", 2024-09-21T12:24:15.933 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1087e-05 2024-09-21T12:24:15.933 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.933 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.933 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:24:15.933 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.845203+0000", 2024-09-21T12:24:15.933 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1428e-05 2024-09-21T12:24:15.933 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.933 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.933 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:24:15.933 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.845210+0000", 2024-09-21T12:24:15.933 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7649999999999996e-06 2024-09-21T12:24:15.933 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.933 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.933 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.934 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.845254+0000", 2024-09-21T12:24:15.934 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4338000000000001e-05 2024-09-21T12:24:15.934 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.934 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.934 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:24:15.934 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.845260+0000", 2024-09-21T12:24:15.934 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0909999999999998e-06 2024-09-21T12:24:15.934 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.934 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.934 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.934 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.824294+0000", 2024-09-21T12:24:15.934 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97903364199999998 2024-09-21T12:24:15.934 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.934 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.934 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:24:15.935 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.824397+0000", 2024-09-21T12:24:15.935 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000103479 2024-09-21T12:24:15.935 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.935 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.935 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:24:15.935 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.824523+0000", 2024-09-21T12:24:15.935 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012569400000000001 2024-09-21T12:24:15.935 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.935 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.935 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.935 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.829568+0000", 2024-09-21T12:24:15.935 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0050454330000000002 2024-09-21T12:24:15.935 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.935 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.935 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:24:15.936 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.830733+0000", 2024-09-21T12:24:15.936 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001165063 2024-09-21T12:24:15.936 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.936 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.936 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.936 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.833137+0000", 2024-09-21T12:24:15.936 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0024032519999999998 2024-09-21T12:24:15.936 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.936 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.936 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:24:15.936 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.833199+0000", 2024-09-21T12:24:15.936 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2794999999999998e-05 2024-09-21T12:24:15.936 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.936 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.936 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:24:15.936 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.833243+0000", 2024-09-21T12:24:15.936 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3674999999999998e-05 2024-09-21T12:24:15.937 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.937 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:15.937 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.937 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.937 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.937 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46587 7.1 7:8649b0f8:::benchmark_data_smithi114_78432_object46586:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:24:15.937 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.847506+0000", 2024-09-21T12:24:15.937 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 133.792409803, 2024-09-21T12:24:15.937 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.985956316, 2024-09-21T12:24:15.937 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:24:15.937 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:24:15.937 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:15.937 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:24:15.937 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:24:15.937 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:15.938 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46587 2024-09-21T12:24:15.938 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.938 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:24:15.938 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.938 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:24:15.938 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.847506+0000", 2024-09-21T12:24:15.938 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.938 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.938 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.938 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:24:15.938 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.847506+0000", 2024-09-21T12:24:15.938 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:24:15.938 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.938 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.938 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:24:15.939 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.847507+0000", 2024-09-21T12:24:15.939 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5000000000000001e-07 2024-09-21T12:24:15.939 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.939 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.939 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:24:15.939 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.847525+0000", 2024-09-21T12:24:15.939 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8082999999999999e-05 2024-09-21T12:24:15.939 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.939 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.939 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:24:15.939 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.847536+0000", 2024-09-21T12:24:15.939 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1270000000000001e-05 2024-09-21T12:24:15.939 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.939 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.939 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:24:15.940 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.847541+0000", 2024-09-21T12:24:15.940 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9799999999999998e-06 2024-09-21T12:24:15.940 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.940 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.940 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.940 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.847595+0000", 2024-09-21T12:24:15.940 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3600999999999997e-05 2024-09-21T12:24:15.940 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.940 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.940 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:24:15.940 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.847600+0000", 2024-09-21T12:24:15.940 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6699999999999999e-06 2024-09-21T12:24:15.940 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.940 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.940 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:24:15.940 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.826230+0000", 2024-09-21T12:24:15.940 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97862941199999998 2024-09-21T12:24:15.941 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.941 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.941 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:24:15.941 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.826353+0000", 2024-09-21T12:24:15.941 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000123433 2024-09-21T12:24:15.941 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.941 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.941 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:24:15.941 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.826485+0000", 2024-09-21T12:24:15.941 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013211899999999999 2024-09-21T12:24:15.941 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.941 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.941 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:24:15.941 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.830740+0000", 2024-09-21T12:24:15.941 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.004254903 2024-09-21T12:24:15.941 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.941 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.942 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.942 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.833294+0000", 2024-09-21T12:24:15.942 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0025538900000000001 2024-09-21T12:24:15.942 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.942 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.942 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:15.942 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.833378+0000", 2024-09-21T12:24:15.942 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.4357999999999994e-05 2024-09-21T12:24:15.942 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.942 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.942 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:24:15.942 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.833426+0000", 2024-09-21T12:24:15.942 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7429000000000003e-05 2024-09-21T12:24:15.943 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:15.943 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:15.943 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:24:15.943 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.833462+0000", 2024-09-21T12:24:15.943 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.6318e-05 2024-09-21T12:24:15.943 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.943 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:15.943 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.943 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:15.943 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:15.943 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:24:15.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:15.944 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:15.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:16.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:15 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1362256750' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:24:16.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:15 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2469306356' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:24:16.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:15 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:24:16.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:15 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2230313461' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:24:16.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:15 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1362256750' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:24:16.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:15 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2469306356' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:24:16.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:15 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:24:16.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:15 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2230313461' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:24:16.045 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:16.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:16.084 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:16.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:16.184 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:16.221 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:16.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:16.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:16.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:15 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1362256750' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:24:16.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:15 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2469306356' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:24:16.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:15 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:24:16.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:15 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2230313461' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:24:16.295 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:24:16.295 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:24:16.295 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:24:16.295 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:24:16.295 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:24:16.295 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:24:16.295 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:24:16.295 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:24:16.295 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:24:16.296 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:24:16.296 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:24:16.296 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:24:16.296 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:24:16.296 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:24:16.296 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:24:16.296 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:24:16.323 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:16.349 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:16.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:16.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:16.451 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:16.475 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:16.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:16.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:16.577 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:16.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:16.612 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:16.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:16.713 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:16.738 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:16.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:16.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:16.840 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:16.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:16.873 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:16.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:16.974 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:17.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:17.016 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:17.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:17.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:16 smithi114 ceph-mon[30324]: pgmap v182: 158 pgs: 158 active+clean; 1.5 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 664 op/s 2024-09-21T12:24:17.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:16 smithi114 ceph-mon[30324]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:24:17.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:16 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1362256750' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:24:17.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:16 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T12:24:17.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:16 smithi114 ceph-mon[30324]: osdmap e150: 8 total, 8 up, 7 in 2024-09-21T12:24:17.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:16 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:24:17.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:16 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:24:17.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:16 smithi114 ceph-mon[40867]: pgmap v182: 158 pgs: 158 active+clean; 1.5 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 664 op/s 2024-09-21T12:24:17.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:16 smithi114 ceph-mon[40867]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:24:17.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:16 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1362256750' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:24:17.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:16 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T12:24:17.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:16 smithi114 ceph-mon[40867]: osdmap e150: 8 total, 8 up, 7 in 2024-09-21T12:24:17.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:16 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:24:17.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:16 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:24:17.117 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:17.142 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:17.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:17.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:17.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:24:16 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:24:17.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:16 smithi183 ceph-mon[33277]: pgmap v182: 158 pgs: 158 active+clean; 1.5 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 664 op/s 2024-09-21T12:24:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:16 smithi183 ceph-mon[33277]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:24:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:16 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1362256750' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:24:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:16 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T12:24:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:16 smithi183 ceph-mon[33277]: osdmap e150: 8 total, 8 up, 7 in 2024-09-21T12:24:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:16 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:24:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:16 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:24:17.245 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:17.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:17.290 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:17.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:17.391 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:17.417 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:17.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:17.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:17.519 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:17.555 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:17.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:17.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:17.657 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:17.685 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:17.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:17.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:17.787 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:17.812 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:17.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:17.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:17.914 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:17.938 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:17.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:17.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:18.039 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:18.075 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:18.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:18.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:18.177 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:18.198 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:18.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:18.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:18.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:17 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:24:18.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:17 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:24:18.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:17 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:24:18.300 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:18.326 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:18.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:18.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:18.427 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:18.469 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:18.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:18.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:18.570 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:18.603 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:18.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:18.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:18.705 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:18.739 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:18.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:18.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:18.841 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:18.880 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:18.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:18.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:18.982 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:19.003 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:19.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:19.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:19.105 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:19.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:19.147 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:19.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:19.204 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:18 smithi114 ceph-mon[30324]: pgmap v184: 158 pgs: 158 active+clean; 1.2 GiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-09-21T12:24:19.205 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:18 smithi114 ceph-mon[40867]: pgmap v184: 158 pgs: 158 active+clean; 1.2 GiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-09-21T12:24:19.213 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:24:19.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:18 smithi183 ceph-mon[33277]: pgmap v184: 158 pgs: 158 active+clean; 1.2 GiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-09-21T12:24:19.248 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:19.271 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:19.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:19.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:19.373 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:19.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:19.419 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:19.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:19.520 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:19.571 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:19.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:19.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:19.582 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T12:24:19.673 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:19.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:19.698 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:19.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:19.799 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:19.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:19.840 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:19.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:19.940 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:19.945 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T12:24:19.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:19.982 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:19.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:20.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:19 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:24:20.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:19 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:24:20.067 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:19 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:24:20.067 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:19 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:24:20.083 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:20.108 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:20.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:20.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:20.210 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:20.221 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T12:24:20.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:19 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:24:20.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:19 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:24:20.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:20.285 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:20.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:20.319 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:20.353 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:20.386 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:20.435 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:20.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:20.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:20.532 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:24:20.537 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:20.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:20.581 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:20.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:20.681 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:20.728 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:20.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:20.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:20.830 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:20.862 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:20.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:20.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:20.964 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:20.970 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:20 smithi183 ceph-mon[33277]: pgmap v185: 158 pgs: 158 active+clean; 1.2 GiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.15k op/s 2024-09-21T12:24:20.970 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:20 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:24:20.970 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:20 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:24:20.970 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:20 smithi183 ceph-mon[33277]: osdmap e151: 8 total, 8 up, 7 in 2024-09-21T12:24:20.993 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:20.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:20.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:20.994 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:20 smithi114 ceph-mon[40867]: pgmap v185: 158 pgs: 158 active+clean; 1.2 GiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.15k op/s 2024-09-21T12:24:20.994 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:20 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:24:20.994 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:20 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:24:20.994 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:20 smithi114 ceph-mon[40867]: osdmap e151: 8 total, 8 up, 7 in 2024-09-21T12:24:20.995 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:20 smithi114 ceph-mon[30324]: pgmap v185: 158 pgs: 158 active+clean; 1.2 GiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.15k op/s 2024-09-21T12:24:20.995 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:20 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:24:20.995 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:20 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:24:20.995 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:20 smithi114 ceph-mon[30324]: osdmap e151: 8 total, 8 up, 7 in 2024-09-21T12:24:21.094 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:21.103 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:24:21.103 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:24:21.120 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:21.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:21.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:21.222 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:21.237 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:24:21.237 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:24:21.238 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:24:21.238 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:24:21.238 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:24:21.238 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:24:21.238 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:24:21.238 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:24:21.253 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:24:21.253 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:24:21.266 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:21.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:21.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:21.288 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:24:21.288 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:24:21.320 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:24:21.320 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:24:21.354 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:24:21.354 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:24:21.368 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:21.391 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:21.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:21.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:21.493 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:21.517 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:21.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:21.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:21.618 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:21.660 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:21.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:21.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:21.731 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:21.762 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:21.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:21.818 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:21.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:21.919 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:21.952 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:21.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:21.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:21.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:21 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/806607314' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:24:22.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:22.088 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:22.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:22.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:22.165 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T12:24:22.190 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:22.233 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:22.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:22.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:22.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:21 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/806607314' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:24:22.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:21 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/806607314' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:24:22.335 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:22.360 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:22.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:22.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:22.462 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:22.493 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:22.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:22.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:22.495 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:24:22.595 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:22.632 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:22.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:22.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:22.696 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:24:22 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:24:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:24:22.734 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:22.777 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:22.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:22.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:22.879 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:22.881 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T12:24:22.936 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:22.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:22.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:22.952 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:22 smithi114 ceph-mon[30324]: pgmap v187: 158 pgs: 158 active+clean; 1.1 GiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 912 op/s 2024-09-21T12:24:22.952 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:22 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1039714832' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:24:22.952 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:22 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:24:22.952 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:22 smithi114 ceph-mon[40867]: pgmap v187: 158 pgs: 158 active+clean; 1.1 GiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 912 op/s 2024-09-21T12:24:22.952 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:22 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1039714832' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:24:22.953 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:22 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:24:22.999 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:24:23.000 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-21T12:24:23.000 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-21T12:24:23.000 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-21T12:24:23.000 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:24:23.038 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:23.067 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:23.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:23.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:23.169 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:23.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:23.196 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:23.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:23.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:22 smithi183 ceph-mon[33277]: pgmap v187: 158 pgs: 158 active+clean; 1.1 GiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 912 op/s 2024-09-21T12:24:23.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:22 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1039714832' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:24:23.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:22 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:24:23.297 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:23.323 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:23.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:23.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:23.426 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:23.451 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:23.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:23.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:23.552 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:23.585 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:23.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:23.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:23.686 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:23.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:23.711 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:23.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:23.784 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T12:24:23.812 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:23.849 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:23.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:23.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:23.904 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-21T12:24:23.951 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:23.972 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:23.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:23.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:24.074 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:24.092 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:24:24.112 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:24.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:24.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:24.157 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:23 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T12:24:24.158 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:23 smithi114 ceph-mon[40867]: osdmap e152: 8 total, 8 up, 7 in 2024-09-21T12:24:24.158 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:23 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1543219917' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:24:24.158 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:23 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T12:24:24.158 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:23 smithi114 ceph-mon[30324]: osdmap e152: 8 total, 8 up, 7 in 2024-09-21T12:24:24.158 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:23 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1543219917' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:24:24.214 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:24.217 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:23 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T12:24:24.217 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:23 smithi183 ceph-mon[33277]: osdmap e152: 8 total, 8 up, 7 in 2024-09-21T12:24:24.217 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:23 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1543219917' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:24:24.239 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:24.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:24.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:24.340 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:24.376 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:24.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:24.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:24.404 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:24:24.478 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:24.503 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:24.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:24.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:24.536 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T12:24:24.536 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:24:24.606 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:24.616 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:24:24.616 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-21T12:24:24.616 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-21T12:24:24.617 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:24:24.617 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.617 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:23884 7.1 7:8f148689:::benchmark_data_smithi114_78432_object23883:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T12:24:24.617 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:39.695870+0000", 2024-09-21T12:24:24.617 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 164.91762089700001, 2024-09-21T12:24:24.617 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.039559285999999999, 2024-09-21T12:24:24.617 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:24.617 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:24.617 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:24.617 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:24:24.617 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:24:24.617 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:24.617 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 23884 2024-09-21T12:24:24.618 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.618 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:24.618 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.618 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:24.618 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695870+0000", 2024-09-21T12:24:24.618 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.618 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.618 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.618 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:24.618 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695870+0000", 2024-09-21T12:24:24.618 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.618 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.618 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.618 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:24.619 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695873+0000", 2024-09-21T12:24:24.619 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0090000000000002e-06 2024-09-21T12:24:24.619 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.619 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.619 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:24.619 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695927+0000", 2024-09-21T12:24:24.619 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4941000000000001e-05 2024-09-21T12:24:24.619 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.619 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.619 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:24.619 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695955+0000", 2024-09-21T12:24:24.619 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7120000000000001e-05 2024-09-21T12:24:24.619 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.619 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.620 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:24.620 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695963+0000", 2024-09-21T12:24:24.620 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7760000000000003e-06 2024-09-21T12:24:24.620 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.620 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.620 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:24.620 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696022+0000", 2024-09-21T12:24:24.620 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8941999999999999e-05 2024-09-21T12:24:24.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.621 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:24.621 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696107+0000", 2024-09-21T12:24:24.621 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4853999999999998e-05 2024-09-21T12:24:24.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.621 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,6", 2024-09-21T12:24:24.621 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696202+0000", 2024-09-21T12:24:24.621 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4889999999999994e-05 2024-09-21T12:24:24.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.621 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:24:24.621 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.698132+0000", 2024-09-21T12:24:24.622 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001929998 2024-09-21T12:24:24.622 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.622 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.622 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:24.622 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.698167+0000", 2024-09-21T12:24:24.622 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5247000000000003e-05 2024-09-21T12:24:24.622 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.622 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.622 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:24.622 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.735322+0000", 2024-09-21T12:24:24.622 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.037154514 2024-09-21T12:24:24.622 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.622 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.623 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:24.623 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.735382+0000", 2024-09-21T12:24:24.623 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0371000000000002e-05 2024-09-21T12:24:24.623 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.623 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.623 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:24.623 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.735430+0000", 2024-09-21T12:24:24.623 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7623999999999997e-05 2024-09-21T12:24:24.623 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.623 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:24.624 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.624 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:23885 7.1 7:81ccbda1:::benchmark_data_smithi114_78432_object23884:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T12:24:24.624 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:39.696116+0000", 2024-09-21T12:24:24.624 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 164.91737506800001, 2024-09-21T12:24:24.624 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.040219576, 2024-09-21T12:24:24.624 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:24.624 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:24.624 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:24.624 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:24:24.624 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:24:24.624 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:24.624 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 23885 2024-09-21T12:24:24.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.625 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:24.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:24.625 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696116+0000", 2024-09-21T12:24:24.625 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:24.625 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696116+0000", 2024-09-21T12:24:24.625 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:24.626 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696117+0000", 2024-09-21T12:24:24.626 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0109999999999999e-06 2024-09-21T12:24:24.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.626 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:24.626 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696178+0000", 2024-09-21T12:24:24.626 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1060000000000002e-05 2024-09-21T12:24:24.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.627 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.627 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:24.627 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696188+0000", 2024-09-21T12:24:24.627 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0067e-05 2024-09-21T12:24:24.627 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.627 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.627 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:24.627 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696196+0000", 2024-09-21T12:24:24.627 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0740000000000004e-06 2024-09-21T12:24:24.627 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.627 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.627 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:24.627 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.697126+0000", 2024-09-21T12:24:24.627 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00093067599999999996 2024-09-21T12:24:24.628 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.628 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.628 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:24.628 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.697270+0000", 2024-09-21T12:24:24.628 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000143502 2024-09-21T12:24:24.628 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.628 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.628 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,6", 2024-09-21T12:24:24.628 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.697418+0000", 2024-09-21T12:24:24.628 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000147892 2024-09-21T12:24:24.628 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.628 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.628 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:24:24.628 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.698242+0000", 2024-09-21T12:24:24.628 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00082411799999999999 2024-09-21T12:24:24.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.629 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.629 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:24.629 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.698485+0000", 2024-09-21T12:24:24.629 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00024341099999999999 2024-09-21T12:24:24.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.629 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.629 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:24.629 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.736250+0000", 2024-09-21T12:24:24.629 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.037765265999999999 2024-09-21T12:24:24.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.629 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.629 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:24.629 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.736300+0000", 2024-09-21T12:24:24.630 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9886000000000003e-05 2024-09-21T12:24:24.630 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.630 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.630 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:24.630 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.736336+0000", 2024-09-21T12:24:24.630 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5612999999999999e-05 2024-09-21T12:24:24.630 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.630 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:24.630 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.630 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.630 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.631 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:81802 7.8 e115/112 7:110c0c7a:::benchmark_data_smithi114_78432_object81801:head v 115'5198, mlcod=88'242)", 2024-09-21T12:24:24.631 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.137865+0000", 2024-09-21T12:24:24.631 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 109.475626181, 2024-09-21T12:24:24.631 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.051160639000000001, 2024-09-21T12:24:24.631 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:24.631 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:24.631 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:24.631 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:24.631 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.631 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:24.632 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137865+0000", 2024-09-21T12:24:24.632 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.632 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.633 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.633 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:24.633 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137865+0000", 2024-09-21T12:24:24.633 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.633 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.633 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.633 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:24.633 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137866+0000", 2024-09-21T12:24:24.634 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5099999999999998e-07 2024-09-21T12:24:24.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.634 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.634 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:24.634 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137909+0000", 2024-09-21T12:24:24.634 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2545e-05 2024-09-21T12:24:24.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.634 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.634 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:24.634 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137938+0000", 2024-09-21T12:24:24.634 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9737999999999999e-05 2024-09-21T12:24:24.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.634 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.634 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:24.635 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137961+0000", 2024-09-21T12:24:24.635 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2320999999999999e-05 2024-09-21T12:24:24.635 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.635 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.635 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:24.635 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.138026+0000", 2024-09-21T12:24:24.635 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5571e-05 2024-09-21T12:24:24.635 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.635 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.635 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:24.636 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.138111+0000", 2024-09-21T12:24:24.636 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4682e-05 2024-09-21T12:24:24.636 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.636 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:24.636 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.188940+0000", 2024-09-21T12:24:24.636 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.050828867 2024-09-21T12:24:24.636 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.636 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:24.636 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189026+0000", 2024-09-21T12:24:24.636 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6063999999999995e-05 2024-09-21T12:24:24.636 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.636 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:24.637 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.637 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.637 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.637 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:81809 7.8 e115/112 7:1042b23e:::benchmark_data_smithi114_78432_object81808:head v 115'5199, mlcod=88'242)", 2024-09-21T12:24:24.637 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.140023+0000", 2024-09-21T12:24:24.637 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 109.47346861, 2024-09-21T12:24:24.637 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.049109005999999997, 2024-09-21T12:24:24.637 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:24.637 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:24.637 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:24.637 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:24.637 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.637 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:24.637 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140023+0000", 2024-09-21T12:24:24.638 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.638 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.638 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.638 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:24.638 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140023+0000", 2024-09-21T12:24:24.638 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.638 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.638 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.638 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:24.638 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140023+0000", 2024-09-21T12:24:24.638 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6700000000000003e-07 2024-09-21T12:24:24.638 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.638 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.639 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:24.639 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140136+0000", 2024-09-21T12:24:24.639 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000112623 2024-09-21T12:24:24.639 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.639 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.639 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:24.639 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140146+0000", 2024-09-21T12:24:24.639 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0098999999999999e-05 2024-09-21T12:24:24.639 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.639 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.639 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:24.639 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140161+0000", 2024-09-21T12:24:24.639 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5288e-05 2024-09-21T12:24:24.639 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.640 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:24.640 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146087+0000", 2024-09-21T12:24:24.640 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0059259070000000002 2024-09-21T12:24:24.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.640 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:24.640 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146109+0000", 2024-09-21T12:24:24.640 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2045999999999999e-05 2024-09-21T12:24:24.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.641 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:24.641 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189077+0000", 2024-09-21T12:24:24.641 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.042967639000000002 2024-09-21T12:24:24.641 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.641 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.641 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:24.641 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189132+0000", 2024-09-21T12:24:24.641 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4737000000000001e-05 2024-09-21T12:24:24.641 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.641 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:24.641 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.641 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.641 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.641 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:81811 7.8 e115/112 7:10044f1a:::benchmark_data_smithi114_78432_object81810:head v 115'5200, mlcod=88'242)", 2024-09-21T12:24:24.642 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.146782+0000", 2024-09-21T12:24:24.642 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 109.46670912499999, 2024-09-21T12:24:24.642 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.042406829, 2024-09-21T12:24:24.642 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:24.642 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:24.642 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:24.642 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:24.642 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.642 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:24.642 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146782+0000", 2024-09-21T12:24:24.642 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.642 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.642 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.642 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:24.643 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146782+0000", 2024-09-21T12:24:24.643 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.643 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.643 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.643 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:24.643 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146783+0000", 2024-09-21T12:24:24.643 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2999999999999999e-07 2024-09-21T12:24:24.643 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.643 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.643 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:24.643 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146826+0000", 2024-09-21T12:24:24.644 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2781000000000003e-05 2024-09-21T12:24:24.644 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.644 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.644 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:24.644 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146837+0000", 2024-09-21T12:24:24.644 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1126e-05 2024-09-21T12:24:24.644 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.644 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.644 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:24.644 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146843+0000", 2024-09-21T12:24:24.644 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4710000000000004e-06 2024-09-21T12:24:24.644 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.644 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.644 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:24.645 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.152740+0000", 2024-09-21T12:24:24.645 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0058961209999999998 2024-09-21T12:24:24.645 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.645 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.645 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:24.645 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.152763+0000", 2024-09-21T12:24:24.645 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3101000000000001e-05 2024-09-21T12:24:24.645 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.645 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.645 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:24.645 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189134+0000", 2024-09-21T12:24:24.645 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.036371542999999999 2024-09-21T12:24:24.645 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.645 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.646 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:24.646 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189189+0000", 2024-09-21T12:24:24.646 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4855999999999998e-05 2024-09-21T12:24:24.646 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.646 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:24.646 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.646 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.646 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.646 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:81821 7.8 e115/112 7:1c79dadc:::benchmark_data_smithi114_78432_object81820:head v 115'5201, mlcod=88'242)", 2024-09-21T12:24:24.646 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.148399+0000", 2024-09-21T12:24:24.646 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 109.46509205, 2024-09-21T12:24:24.646 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.040843988999999997, 2024-09-21T12:24:24.646 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:24.646 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:24.647 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:24.647 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:24.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:24.647 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148399+0000", 2024-09-21T12:24:24.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:24.647 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148399+0000", 2024-09-21T12:24:24.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:24.648 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148400+0000", 2024-09-21T12:24:24.648 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.049e-06 2024-09-21T12:24:24.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:24.648 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148502+0000", 2024-09-21T12:24:24.648 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000101542 2024-09-21T12:24:24.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:24.649 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148512+0000", 2024-09-21T12:24:24.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0386e-05 2024-09-21T12:24:24.649 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.649 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:24.649 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148519+0000", 2024-09-21T12:24:24.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3589999999999996e-06 2024-09-21T12:24:24.649 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.649 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:24.649 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.158203+0000", 2024-09-21T12:24:24.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0096844710000000001 2024-09-21T12:24:24.649 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.649 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:24.650 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.158229+0000", 2024-09-21T12:24:24.650 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6078999999999999e-05 2024-09-21T12:24:24.650 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.650 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.650 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:24.650 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189192+0000", 2024-09-21T12:24:24.650 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.030962455 2024-09-21T12:24:24.650 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.650 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.650 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:24.650 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189243+0000", 2024-09-21T12:24:24.650 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1647999999999997e-05 2024-09-21T12:24:24.651 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.651 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:24.651 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.651 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.651 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.651 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82430 7.8 e115/112 7:1762e17c:::benchmark_data_smithi114_78432_object82429:head v 115'5247, mlcod=88'255)", 2024-09-21T12:24:24.651 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.787623+0000", 2024-09-21T12:24:24.651 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 108.825868262, 2024-09-21T12:24:24.651 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.048511400000000003, 2024-09-21T12:24:24.651 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:24.651 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:24.651 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:24.651 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:24.652 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.652 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:24.652 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787623+0000", 2024-09-21T12:24:24.652 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.652 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.652 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.652 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:24.652 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787623+0000", 2024-09-21T12:24:24.652 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.652 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.652 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.652 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:24.652 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787625+0000", 2024-09-21T12:24:24.652 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5319999999999999e-06 2024-09-21T12:24:24.652 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.653 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.653 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:24.653 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787667+0000", 2024-09-21T12:24:24.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1876999999999999e-05 2024-09-21T12:24:24.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.653 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.653 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:24.653 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787688+0000", 2024-09-21T12:24:24.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1046000000000001e-05 2024-09-21T12:24:24.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.653 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.653 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:24.653 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787697+0000", 2024-09-21T12:24:24.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4930000000000004e-06 2024-09-21T12:24:24.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.654 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:24.654 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.788378+0000", 2024-09-21T12:24:24.654 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00068115999999999999 2024-09-21T12:24:24.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.654 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:24.654 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.788427+0000", 2024-09-21T12:24:24.654 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8553000000000001e-05 2024-09-21T12:24:24.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.654 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:24.654 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.836011+0000", 2024-09-21T12:24:24.655 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.047583978999999998 2024-09-21T12:24:24.655 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.655 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.655 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:24.655 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.836135+0000", 2024-09-21T12:24:24.655 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012375999999999999 2024-09-21T12:24:24.655 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.655 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:24.655 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.655 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.655 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.655 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDPGPush(7.8 115/112 [PushOp(7:10df046b:::benchmark_data_smithi114_78432_object3988:head, version: 88'257, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:10df046b:::benchmark_data_smithi114_78432_object3988:head@88'257, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T12:24:24.655 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.787722+0000", 2024-09-21T12:24:24.655 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 108.82576918700001, 2024-09-21T12:24:24.655 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.074165650999999999, 2024-09-21T12:24:24.656 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:24.656 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:24.656 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:24:24.656 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:24.656 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.656 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:24.656 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787722+0000", 2024-09-21T12:24:24.656 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.656 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.656 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:24.656 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787722+0000", 2024-09-21T12:24:24.656 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.656 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.657 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:24.657 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787723+0000", 2024-09-21T12:24:24.657 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0499999999999999e-07 2024-09-21T12:24:24.657 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.657 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.657 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:24.657 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787779+0000", 2024-09-21T12:24:24.657 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6053e-05 2024-09-21T12:24:24.657 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.657 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.657 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:24.657 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787803+0000", 2024-09-21T12:24:24.657 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3980000000000001e-05 2024-09-21T12:24:24.657 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.657 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.658 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:24.658 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787832+0000", 2024-09-21T12:24:24.658 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.942e-05 2024-09-21T12:24:24.658 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.658 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.658 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:24.658 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.859697+0000", 2024-09-21T12:24:24.658 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.071864649000000003 2024-09-21T12:24:24.658 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.658 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.658 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:24.658 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.859710+0000", 2024-09-21T12:24:24.658 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3740000000000001e-05 2024-09-21T12:24:24.658 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.659 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.659 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:24.659 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.861888+0000", 2024-09-21T12:24:24.659 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002177404 2024-09-21T12:24:24.659 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.659 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:24.659 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.659 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.659 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.659 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82442 7.8 e115/112 7:1035456a:::benchmark_data_smithi114_78432_object82441:head v 115'5248, mlcod=88'255)", 2024-09-21T12:24:24.659 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.791117+0000", 2024-09-21T12:24:24.659 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 108.82237428400001, 2024-09-21T12:24:24.659 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.051248759999999997, 2024-09-21T12:24:24.659 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:24.659 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:24.660 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:24.660 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:24.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:24.660 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791117+0000", 2024-09-21T12:24:24.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:24.660 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791117+0000", 2024-09-21T12:24:24.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:24.661 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791118+0000", 2024-09-21T12:24:24.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9999999999999998e-07 2024-09-21T12:24:24.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.661 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:24.661 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791158+0000", 2024-09-21T12:24:24.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0025999999999999e-05 2024-09-21T12:24:24.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.661 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:24.661 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791167+0000", 2024-09-21T12:24:24.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9910000000000008e-06 2024-09-21T12:24:24.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:24.662 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791176+0000", 2024-09-21T12:24:24.662 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0329999999999997e-06 2024-09-21T12:24:24.662 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.662 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:24.662 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.794825+0000", 2024-09-21T12:24:24.662 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0036494700000000001 2024-09-21T12:24:24.662 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.662 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:24.662 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.794878+0000", 2024-09-21T12:24:24.662 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3292999999999999e-05 2024-09-21T12:24:24.662 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.663 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.663 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:24.663 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842289+0000", 2024-09-21T12:24:24.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.047410256999999997 2024-09-21T12:24:24.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.663 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.663 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:24.663 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842366+0000", 2024-09-21T12:24:24.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7189999999999998e-05 2024-09-21T12:24:24.663 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.663 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:24.663 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.664 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.664 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82447 7.8 e115/112 7:174332ca:::benchmark_data_smithi114_78432_object82446:head v 115'5249, mlcod=88'255)", 2024-09-21T12:24:24.664 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.793874+0000", 2024-09-21T12:24:24.664 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 108.819617833, 2024-09-21T12:24:24.664 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.048541015999999999, 2024-09-21T12:24:24.664 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:24.664 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:24.664 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:24.664 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:24.664 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.664 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:24.664 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793874+0000", 2024-09-21T12:24:24.664 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.664 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.665 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.665 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:24.665 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793874+0000", 2024-09-21T12:24:24.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.665 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.665 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.665 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:24.665 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793875+0000", 2024-09-21T12:24:24.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0890000000000001e-06 2024-09-21T12:24:24.665 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.665 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.665 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:24.665 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793910+0000", 2024-09-21T12:24:24.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5080000000000003e-05 2024-09-21T12:24:24.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:24.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793921+0000", 2024-09-21T12:24:24.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1284000000000001e-05 2024-09-21T12:24:24.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:24.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793930+0000", 2024-09-21T12:24:24.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5429999999999994e-06 2024-09-21T12:24:24.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:24.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.801012+0000", 2024-09-21T12:24:24.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0070825219999999999 2024-09-21T12:24:24.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:24.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.801039+0000", 2024-09-21T12:24:24.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7248999999999999e-05 2024-09-21T12:24:24.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:24.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842368+0000", 2024-09-21T12:24:24.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.041328831000000003 2024-09-21T12:24:24.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:24.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842415+0000", 2024-09-21T12:24:24.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6418000000000002e-05 2024-09-21T12:24:24.668 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.668 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:24.668 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.668 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82448 7.8 e115/112 7:11700dea:::benchmark_data_smithi114_78432_object82447:head v 115'5250, mlcod=88'255)", 2024-09-21T12:24:24.668 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.796531+0000", 2024-09-21T12:24:24.668 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 108.81696008900001, 2024-09-21T12:24:24.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.045928499999999997, 2024-09-21T12:24:24.668 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:24.668 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:24.668 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:24.668 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:24.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:24.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796531+0000", 2024-09-21T12:24:24.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:24.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796531+0000", 2024-09-21T12:24:24.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:24.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796532+0000", 2024-09-21T12:24:24.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7199999999999999e-07 2024-09-21T12:24:24.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:24.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796647+0000", 2024-09-21T12:24:24.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000115097 2024-09-21T12:24:24.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:24.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796660+0000", 2024-09-21T12:24:24.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3271999999999999e-05 2024-09-21T12:24:24.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:24.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796666+0000", 2024-09-21T12:24:24.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0170000000000002e-06 2024-09-21T12:24:24.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:24.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.809350+0000", 2024-09-21T12:24:24.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.012683379 2024-09-21T12:24:24.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:24.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.809374+0000", 2024-09-21T12:24:24.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4417e-05 2024-09-21T12:24:24.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:24.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842417+0000", 2024-09-21T12:24:24.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.033042646000000002 2024-09-21T12:24:24.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:24.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842460+0000", 2024-09-21T12:24:24.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3099999999999997e-05 2024-09-21T12:24:24.672 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.672 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:24.672 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.672 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82473 7.8 e115/112 7:15016882:::benchmark_data_smithi114_78432_object82472:head v 115'5254, mlcod=88'255)", 2024-09-21T12:24:24.673 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.818965+0000", 2024-09-21T12:24:24.673 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 108.794526496, 2024-09-21T12:24:24.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.045024287000000003, 2024-09-21T12:24:24.673 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:24.673 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:24.673 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:24.673 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:24.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:24.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.818965+0000", 2024-09-21T12:24:24.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:24.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.818965+0000", 2024-09-21T12:24:24.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:24.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.818966+0000", 2024-09-21T12:24:24.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9199999999999999e-07 2024-09-21T12:24:24.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:24.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.819001+0000", 2024-09-21T12:24:24.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:24.675 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:24.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:24.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5117999999999998e-05 2024-09-21T12:24:24.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:24.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.819014+0000", 2024-09-21T12:24:24.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2869e-05 2024-09-21T12:24:24.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:24.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.819020+0000", 2024-09-21T12:24:24.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6959999999999999e-06 2024-09-21T12:24:24.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:24.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.836164+0000", 2024-09-21T12:24:24.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.017143617 2024-09-21T12:24:24.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:24.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.836188+0000", 2024-09-21T12:24:24.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4074e-05 2024-09-21T12:24:24.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:24.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.863879+0000", 2024-09-21T12:24:24.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.027690392000000001 2024-09-21T12:24:24.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:24.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.863989+0000", 2024-09-21T12:24:24.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000110629 2024-09-21T12:24:24.679 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.679 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:24.679 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.679 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82539 7.8 e115/112 7:1cd4c60d:::benchmark_data_smithi114_78432_object82538:head v 115'5258, mlcod=88'256)", 2024-09-21T12:24:24.679 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.889845+0000", 2024-09-21T12:24:24.679 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 108.723646093, 2024-09-21T12:24:24.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.046325071000000002, 2024-09-21T12:24:24.679 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:24.679 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:24.679 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:24.680 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:24.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:24.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889845+0000", 2024-09-21T12:24:24.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:24.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889845+0000", 2024-09-21T12:24:24.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:24.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889846+0000", 2024-09-21T12:24:24.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1800000000000004e-07 2024-09-21T12:24:24.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:24.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889878+0000", 2024-09-21T12:24:24.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1619999999999999e-05 2024-09-21T12:24:24.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:24.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889893+0000", 2024-09-21T12:24:24.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4819e-05 2024-09-21T12:24:24.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:24.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889900+0000", 2024-09-21T12:24:24.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8510000000000001e-06 2024-09-21T12:24:24.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:24.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.898101+0000", 2024-09-21T12:24:24.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0082018119999999993 2024-09-21T12:24:24.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:24.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.898164+0000", 2024-09-21T12:24:24.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2756999999999996e-05 2024-09-21T12:24:24.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:24.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.936000+0000", 2024-09-21T12:24:24.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.037835720000000003 2024-09-21T12:24:24.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:24.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.936170+0000", 2024-09-21T12:24:24.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000170574 2024-09-21T12:24:24.683 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.683 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:24.683 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.683 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82545 7.8 e115/112 7:1ddea6d1:::benchmark_data_smithi114_78432_object82544:head v 115'5259, mlcod=88'256)", 2024-09-21T12:24:24.684 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.897417+0000", 2024-09-21T12:24:24.684 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 108.716074556, 2024-09-21T12:24:24.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.038827027, 2024-09-21T12:24:24.684 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:24.684 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:24.684 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:24.684 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:24.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:24.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897417+0000", 2024-09-21T12:24:24.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:24.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897417+0000", 2024-09-21T12:24:24.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:24.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897418+0000", 2024-09-21T12:24:24.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3500000000000005e-07 2024-09-21T12:24:24.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:24.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897462+0000", 2024-09-21T12:24:24.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4036999999999999e-05 2024-09-21T12:24:24.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:24.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897472+0000", 2024-09-21T12:24:24.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0669999999999999e-05 2024-09-21T12:24:24.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:24.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897479+0000", 2024-09-21T12:24:24.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9190000000000002e-06 2024-09-21T12:24:24.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:24.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.905524+0000", 2024-09-21T12:24:24.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0080444899999999996 2024-09-21T12:24:24.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:24.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.905597+0000", 2024-09-21T12:24:24.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3539000000000002e-05 2024-09-21T12:24:24.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.691 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:24.691 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.936176+0000", 2024-09-21T12:24:24.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.030578893999999999 2024-09-21T12:24:24.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.691 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:24.691 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.936244+0000", 2024-09-21T12:24:24.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7543e-05 2024-09-21T12:24:24.691 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.691 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:24.691 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.692 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:83346 7.8 e115/112 7:1517ab05:::benchmark_data_smithi114_78432_object83345:head v 115'5319, mlcod=88'282)", 2024-09-21T12:24:24.692 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:36.700133+0000", 2024-09-21T12:24:24.692 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 107.913358655, 2024-09-21T12:24:24.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.044693733999999999, 2024-09-21T12:24:24.692 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:24.692 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:24.692 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:24.692 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:24.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:24.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700133+0000", 2024-09-21T12:24:24.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:24.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700133+0000", 2024-09-21T12:24:24.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:24.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700133+0000", 2024-09-21T12:24:24.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4300000000000003e-07 2024-09-21T12:24:24.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:24.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700182+0000", 2024-09-21T12:24:24.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8433000000000002e-05 2024-09-21T12:24:24.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:24.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700192+0000", 2024-09-21T12:24:24.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0382e-05 2024-09-21T12:24:24.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:24.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700201+0000", 2024-09-21T12:24:24.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9819999999999997e-06 2024-09-21T12:24:24.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:24.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.703063+0000", 2024-09-21T12:24:24.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002862121 2024-09-21T12:24:24.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:24.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.703087+0000", 2024-09-21T12:24:24.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3730000000000001e-05 2024-09-21T12:24:24.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:24.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.744677+0000", 2024-09-21T12:24:24.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.041589785999999997 2024-09-21T12:24:24.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:24.696 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.744826+0000", 2024-09-21T12:24:24.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014965700000000001 2024-09-21T12:24:24.696 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.696 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:24.696 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.696 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:87464 7.8 e117/112 7:17a0a89d:::benchmark_data_smithi114_78432_object87463:head v 117'5593, mlcod=88'417)", 2024-09-21T12:24:24.696 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:41.463661+0000", 2024-09-21T12:24:24.696 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 103.149830722, 2024-09-21T12:24:24.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.067076070000000002, 2024-09-21T12:24:24.696 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:24.696 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:24.697 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:24.697 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:24.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:24.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463661+0000", 2024-09-21T12:24:24.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:24.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463661+0000", 2024-09-21T12:24:24.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:24.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463661+0000", 2024-09-21T12:24:24.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2799999999999995e-07 2024-09-21T12:24:24.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:24.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463845+0000", 2024-09-21T12:24:24.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000183692 2024-09-21T12:24:24.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:24.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463854+0000", 2024-09-21T12:24:24.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0750000000000004e-06 2024-09-21T12:24:24.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:24.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463860+0000", 2024-09-21T12:24:24.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.015e-06 2024-09-21T12:24:24.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:24.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.480218+0000", 2024-09-21T12:24:24.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.016357289000000001 2024-09-21T12:24:24.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:24.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.480239+0000", 2024-09-21T12:24:24.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1691999999999998e-05 2024-09-21T12:24:24.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:24.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.530695+0000", 2024-09-21T12:24:24.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.050456050000000002 2024-09-21T12:24:24.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:24.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.530737+0000", 2024-09-21T12:24:24.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1428999999999999e-05 2024-09-21T12:24:24.700 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.700 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:24.700 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.700 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:87470 7.8 e117/112 7:1f3b224f:::benchmark_data_smithi114_78432_object87469:head v 117'5594, mlcod=88'417)", 2024-09-21T12:24:24.700 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:41.472523+0000", 2024-09-21T12:24:24.701 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 103.140968724, 2024-09-21T12:24:24.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.06343451, 2024-09-21T12:24:24.701 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:24.701 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:24.701 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:24.701 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:24.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:24.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472523+0000", 2024-09-21T12:24:24.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:24.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472523+0000", 2024-09-21T12:24:24.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:24.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472524+0000", 2024-09-21T12:24:24.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7399999999999991e-07 2024-09-21T12:24:24.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:24.703 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472596+0000", 2024-09-21T12:24:24.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2152999999999998e-05 2024-09-21T12:24:24.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:24.703 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472606+0000", 2024-09-21T12:24:24.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0485e-05 2024-09-21T12:24:24.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:24.703 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472613+0000", 2024-09-21T12:24:24.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5590000000000002e-06 2024-09-21T12:24:24.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:24.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.484029+0000", 2024-09-21T12:24:24.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.01141587 2024-09-21T12:24:24.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:24.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.484053+0000", 2024-09-21T12:24:24.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4559999999999999e-05 2024-09-21T12:24:24.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:24.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.535915+0000", 2024-09-21T12:24:24.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.051861431 2024-09-21T12:24:24.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:24.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.535957+0000", 2024-09-21T12:24:24.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2478000000000002e-05 2024-09-21T12:24:24.705 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.705 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:24.705 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.705 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:87486 7.8 e117/112 7:14e05f7b:::benchmark_data_smithi114_78432_object87485:head v 117'5595, mlcod=88'417)", 2024-09-21T12:24:24.705 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:41.475017+0000", 2024-09-21T12:24:24.705 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 103.13847438800001, 2024-09-21T12:24:24.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.060970067000000003, 2024-09-21T12:24:24.705 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:24.705 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:24.705 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:24.705 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:24.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:24.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475017+0000", 2024-09-21T12:24:24.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:24.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475017+0000", 2024-09-21T12:24:24.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:24.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475018+0000", 2024-09-21T12:24:24.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5799999999999999e-07 2024-09-21T12:24:24.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:24.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475150+0000", 2024-09-21T12:24:24.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013201700000000001 2024-09-21T12:24:24.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:24.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475159+0000", 2024-09-21T12:24:24.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5840000000000001e-06 2024-09-21T12:24:24.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:24.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475166+0000", 2024-09-21T12:24:24.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3589999999999996e-06 2024-09-21T12:24:24.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:24.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.525196+0000", 2024-09-21T12:24:24.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.050030507000000002 2024-09-21T12:24:24.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:24.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.525214+0000", 2024-09-21T12:24:24.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7884e-05 2024-09-21T12:24:24.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:24.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.535960+0000", 2024-09-21T12:24:24.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.010746143999999999 2024-09-21T12:24:24.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:24.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.535987+0000", 2024-09-21T12:24:24.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6914e-05 2024-09-21T12:24:24.709 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.709 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:24.709 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.709 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDPGPush(7.8 117/112 [PushOp(7:1de1c2d4:::benchmark_data_smithi114_78432_object6336:head, version: 88'418, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:1de1c2d4:::benchmark_data_smithi114_78432_object6336:head@88'418, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T12:24:24.709 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:41.485571+0000", 2024-09-21T12:24:24.709 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 103.127920493, 2024-09-21T12:24:24.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.046712901000000001, 2024-09-21T12:24:24.709 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:24.710 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:24.710 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:24:24.710 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:24.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.710 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:24.710 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485571+0000", 2024-09-21T12:24:24.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.710 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:24.710 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485571+0000", 2024-09-21T12:24:24.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:24.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485572+0000", 2024-09-21T12:24:24.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7299999999999995e-07 2024-09-21T12:24:24.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:24.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485643+0000", 2024-09-21T12:24:24.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0976999999999995e-05 2024-09-21T12:24:24.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:24.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485655+0000", 2024-09-21T12:24:24.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2904e-05 2024-09-21T12:24:24.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.712 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:24.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485665+0000", 2024-09-21T12:24:24.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6709999999999994e-06 2024-09-21T12:24:24.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.712 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:24.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.530794+0000", 2024-09-21T12:24:24.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.045129295 2024-09-21T12:24:24.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:24.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.530802+0000", 2024-09-21T12:24:24.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5059999999999998e-06 2024-09-21T12:24:24.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:24.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.532284+0000", 2024-09-21T12:24:24.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014818749999999999 2024-09-21T12:24:24.713 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.713 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:24.713 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.714 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:87497 7.8 e117/112 7:19475f57:::benchmark_data_smithi114_78432_object87496:head v 117'5596, mlcod=88'417)", 2024-09-21T12:24:24.714 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:41.494916+0000", 2024-09-21T12:24:24.714 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 103.118575014, 2024-09-21T12:24:24.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.041122084000000003, 2024-09-21T12:24:24.714 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:24.714 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:24.714 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:24.714 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:24.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:24.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.494916+0000", 2024-09-21T12:24:24.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:24.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.494916+0000", 2024-09-21T12:24:24.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:24.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:24.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.494918+0000", 2024-09-21T12:24:24.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.719e-06 2024-09-21T12:24:24.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:24.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.494975+0000", 2024-09-21T12:24:24.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6981999999999998e-05 2024-09-21T12:24:24.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.716 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.716 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:24.716 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.525059+0000", 2024-09-21T12:24:24.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.030083526999999999 2024-09-21T12:24:24.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.716 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.716 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:24.716 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.525071+0000", 2024-09-21T12:24:24.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2438e-05 2024-09-21T12:24:24.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.716 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.716 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:24.716 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.532312+0000", 2024-09-21T12:24:24.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0072404909999999999 2024-09-21T12:24:24.717 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:24.717 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.532336+0000", 2024-09-21T12:24:24.717 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4090000000000001e-05 2024-09-21T12:24:24.717 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:24.717 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.536013+0000", 2024-09-21T12:24:24.717 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003677825 2024-09-21T12:24:24.717 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:24.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:24.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:24.717 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.536038+0000", 2024-09-21T12:24:24.717 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5012e-05 2024-09-21T12:24:24.718 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.718 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:24.718 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.718 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:24.718 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:24.718 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:24:24.776 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:24.801 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:24.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:24.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:24.902 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:24.929 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:24.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:24.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:25.031 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:25.057 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:25.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:25.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:25.159 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:25.199 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:25.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:25.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:25.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:24 smithi183 ceph-mon[33277]: pgmap v189: 158 pgs: 158 active+clean; 861 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-09-21T12:24:25.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:24 smithi183 ceph-mon[33277]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:24:25.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:24 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1543219917' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:24:25.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:24 smithi183 ceph-mon[33277]: osdmap e153: 8 total, 8 up, 7 in 2024-09-21T12:24:25.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:24 smithi183 ceph-mon[33277]: 7.5 scrub starts 2024-09-21T12:24:25.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:24 smithi183 ceph-mon[33277]: 7.5 scrub ok 2024-09-21T12:24:25.271 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:24 smithi114 ceph-mon[40867]: pgmap v189: 158 pgs: 158 active+clean; 861 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-09-21T12:24:25.271 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:24 smithi114 ceph-mon[40867]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:24:25.271 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:24 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1543219917' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:24:25.271 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:24 smithi114 ceph-mon[40867]: osdmap e153: 8 total, 8 up, 7 in 2024-09-21T12:24:25.271 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:24 smithi114 ceph-mon[40867]: 7.5 scrub starts 2024-09-21T12:24:25.271 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:24 smithi114 ceph-mon[40867]: 7.5 scrub ok 2024-09-21T12:24:25.271 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:24 smithi114 ceph-mon[30324]: pgmap v189: 158 pgs: 158 active+clean; 861 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-09-21T12:24:25.271 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:24 smithi114 ceph-mon[30324]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:24:25.271 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:24 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1543219917' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:24:25.272 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:24 smithi114 ceph-mon[30324]: osdmap e153: 8 total, 8 up, 7 in 2024-09-21T12:24:25.272 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:24 smithi114 ceph-mon[30324]: 7.5 scrub starts 2024-09-21T12:24:25.272 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:24 smithi114 ceph-mon[30324]: 7.5 scrub ok 2024-09-21T12:24:25.301 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:25.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:25.356 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:25.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:25.425 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:25.443 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:25.457 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:25.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:25.483 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:25.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:25.584 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:25.611 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:25.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:25.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:25.713 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:25.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:25.758 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:25.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:25.858 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:25.883 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:25.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:25.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:25.984 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:26.009 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:26.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:26.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:26.111 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:26.115 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T12:24:26.130 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:25 smithi114 ceph-mon[30324]: 5.1 scrub starts 2024-09-21T12:24:26.130 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:25 smithi114 ceph-mon[30324]: 5.1 scrub ok 2024-09-21T12:24:26.131 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:25 smithi114 ceph-mon[30324]: 7.7 scrub starts 2024-09-21T12:24:26.131 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:25 smithi114 ceph-mon[30324]: 5.1e scrub starts 2024-09-21T12:24:26.131 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:25 smithi114 ceph-mon[30324]: 5.1e scrub ok 2024-09-21T12:24:26.131 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:25 smithi114 ceph-mon[30324]: 7.7 scrub ok 2024-09-21T12:24:26.131 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:25 smithi114 ceph-mon[40867]: 5.1 scrub starts 2024-09-21T12:24:26.131 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:25 smithi114 ceph-mon[40867]: 5.1 scrub ok 2024-09-21T12:24:26.131 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:25 smithi114 ceph-mon[40867]: 7.7 scrub starts 2024-09-21T12:24:26.131 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:25 smithi114 ceph-mon[40867]: 5.1e scrub starts 2024-09-21T12:24:26.131 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:25 smithi114 ceph-mon[40867]: 5.1e scrub ok 2024-09-21T12:24:26.131 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:25 smithi114 ceph-mon[40867]: 7.7 scrub ok 2024-09-21T12:24:26.151 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:26.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:26.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:26.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:25 smithi183 ceph-mon[33277]: 5.1 scrub starts 2024-09-21T12:24:26.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:25 smithi183 ceph-mon[33277]: 5.1 scrub ok 2024-09-21T12:24:26.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:25 smithi183 ceph-mon[33277]: 7.7 scrub starts 2024-09-21T12:24:26.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:25 smithi183 ceph-mon[33277]: 5.1e scrub starts 2024-09-21T12:24:26.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:25 smithi183 ceph-mon[33277]: 5.1e scrub ok 2024-09-21T12:24:26.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:25 smithi183 ceph-mon[33277]: 7.7 scrub ok 2024-09-21T12:24:26.252 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:26.289 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:26.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:26.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:26.306 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:24:26.306 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:24:26.347 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:24:26.348 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:24:26.387 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:24:26.387 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:24:26.391 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:26.422 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:24:26.428 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:24:26.428 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:24:26.431 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:26.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:26.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:26.464 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:24:26.464 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:24:26.505 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:24:26.505 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:24:26.533 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:26.545 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:24:26.546 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:24:26.556 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:26.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:26.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:26.579 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:24:26.579 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:24:26.658 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:26.693 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:26.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:26.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:26.795 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:26.835 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:26.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:26.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:26.838 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:24:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:24:26.936 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:24:26.939 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:24:26.940 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:26.968 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:26.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:26.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:26.970 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:24:26.970 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:24:27.070 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:27.102 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:27.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:27.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:27.204 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:27.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:26 smithi183 ceph-mon[33277]: 4.1a scrub starts 2024-09-21T12:24:27.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:26 smithi183 ceph-mon[33277]: pgmap v191: 158 pgs: 158 active+clean; 861 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 960 op/s 2024-09-21T12:24:27.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:26 smithi183 ceph-mon[33277]: 4.1a scrub ok 2024-09-21T12:24:27.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:26 smithi183 ceph-mon[33277]: 3.10 scrub starts 2024-09-21T12:24:27.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:26 smithi183 ceph-mon[33277]: 3.10 scrub ok 2024-09-21T12:24:27.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:26 smithi183 ceph-mon[33277]: 3.12 scrub starts 2024-09-21T12:24:27.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:26 smithi183 ceph-mon[33277]: 3.12 scrub ok 2024-09-21T12:24:27.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:26 smithi183 ceph-mon[33277]: 3.11 scrub starts 2024-09-21T12:24:27.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:26 smithi183 ceph-mon[33277]: 3.11 scrub ok 2024-09-21T12:24:27.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:26 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/157649022' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:24:27.245 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:27.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:27.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:27.247 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:26 smithi114 ceph-mon[30324]: 4.1a scrub starts 2024-09-21T12:24:27.247 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:26 smithi114 ceph-mon[30324]: pgmap v191: 158 pgs: 158 active+clean; 861 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 960 op/s 2024-09-21T12:24:27.248 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:26 smithi114 ceph-mon[30324]: 4.1a scrub ok 2024-09-21T12:24:27.248 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:26 smithi114 ceph-mon[30324]: 3.10 scrub starts 2024-09-21T12:24:27.248 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:26 smithi114 ceph-mon[30324]: 3.10 scrub ok 2024-09-21T12:24:27.248 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:26 smithi114 ceph-mon[30324]: 3.12 scrub starts 2024-09-21T12:24:27.248 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:26 smithi114 ceph-mon[30324]: 3.12 scrub ok 2024-09-21T12:24:27.248 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:26 smithi114 ceph-mon[30324]: 3.11 scrub starts 2024-09-21T12:24:27.248 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:26 smithi114 ceph-mon[30324]: 3.11 scrub ok 2024-09-21T12:24:27.248 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:26 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/157649022' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:24:27.248 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:26 smithi114 ceph-mon[40867]: 4.1a scrub starts 2024-09-21T12:24:27.248 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:26 smithi114 ceph-mon[40867]: pgmap v191: 158 pgs: 158 active+clean; 861 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 960 op/s 2024-09-21T12:24:27.248 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:26 smithi114 ceph-mon[40867]: 4.1a scrub ok 2024-09-21T12:24:27.248 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:26 smithi114 ceph-mon[40867]: 3.10 scrub starts 2024-09-21T12:24:27.249 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:26 smithi114 ceph-mon[40867]: 3.10 scrub ok 2024-09-21T12:24:27.249 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:26 smithi114 ceph-mon[40867]: 3.12 scrub starts 2024-09-21T12:24:27.249 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:26 smithi114 ceph-mon[40867]: 3.12 scrub ok 2024-09-21T12:24:27.249 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:26 smithi114 ceph-mon[40867]: 3.11 scrub starts 2024-09-21T12:24:27.249 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:26 smithi114 ceph-mon[40867]: 3.11 scrub ok 2024-09-21T12:24:27.249 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:26 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/157649022' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:24:27.315 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T12:24:27.347 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:27.372 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:27.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:27.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:27.474 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:27.496 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:27.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:27.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:27.598 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:27.625 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:27.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:27.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:27.726 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:27.751 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T12:24:27.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:27.767 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:27.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:27.868 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:27.901 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:27.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:27.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:27.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:27 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:24:27.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:27 smithi183 ceph-mon[33277]: 7.c scrub starts 2024-09-21T12:24:27.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:27 smithi183 ceph-mon[33277]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:24:27.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:27 smithi183 ceph-mon[33277]: 7.c scrub ok 2024-09-21T12:24:28.003 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:28.038 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:28.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:28.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:28.063 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:24:28.140 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:28.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:28.175 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:28.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:28.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:27 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:24:28.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:27 smithi114 ceph-mon[30324]: 7.c scrub starts 2024-09-21T12:24:28.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:27 smithi114 ceph-mon[30324]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:24:28.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:27 smithi114 ceph-mon[30324]: 7.c scrub ok 2024-09-21T12:24:28.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:27 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:24:28.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:27 smithi114 ceph-mon[40867]: 7.c scrub starts 2024-09-21T12:24:28.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:27 smithi114 ceph-mon[40867]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:24:28.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:27 smithi114 ceph-mon[40867]: 7.c scrub ok 2024-09-21T12:24:28.276 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:28.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:28.302 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:28.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:28.403 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:28.430 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:28.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:28.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:28.531 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:28.561 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:28.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:28.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:28.641 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:24:28.641 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:24:28.663 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:28.690 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:28.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:28.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:28.707 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:28.742 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2024-09-21T12:24:25.637817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":902622169,"num_objects":13963,"num_object_clones":0,"num_object_copies":41889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13963,"num_whiteouts":0,"num_read":967,"num_read_kb":729,"num_write":194970,"num_write_kb":6665346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16035,"num_bytes_recovered":1036313436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72299,"ondisk_log_size":72299,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":3664348,"kb_used_data":2768352,"kb_used_omap":10,"kb_used_meta":895605,"kb_avail":652551716,"statfs":{"total":671965249536,"available":668212957184,"internally_reserved":0,"allocated":2834792448,"data_stored":2832362172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":917099627},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,6,2,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-377487360,"num_objects":-5760,"num_object_clones":0,"num_object_copies":-17280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5760,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5760,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.003616"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":205,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.875807+0000","last_change":"2024-09-21T12:22:16.705200+0000","last_active":"2024-09-21T12:24:19.875807+0000","last_peered":"2024-09-21T12:24:19.875807+0000","last_clean":"2024-09-21T12:24:19.875807+0000","last_became_active":"2024-09-21T12:20:15.309431+0000","last_became_peered":"2024-09-21T12:20:15.309431+0000","last_unstale":"2024-09-21T12:24:19.875807+0000","last_undegraded":"2024-09-21T12:24:19.875807+0000","last_fullsized":"2024-09-21T12:24:19.875807+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:16.705088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:16.705088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:38.137043+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":176,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.868023+0000","last_change":"2024-09-21T12:22:51.040744+0000","last_active":"2024-09-21T12:24:19.868023+0000","last_peered":"2024-09-21T12:24:19.868023+0000","last_clean":"2024-09-21T12:24:19.868023+0000","last_became_active":"2024-09-21T12:22:50.909699+0000","last_became_peered":"2024-09-21T12:22:50.909699+0000","last_unstale":"2024-09-21T12:24:19.868023+0000","last_undegraded":"2024-09-21T12:24:19.868023+0000","last_fullsized":"2024-09-21T12:24:19.868023+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:05.017454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":197,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.880191+0000","last_change":"2024-09-21T12:23:20.433685+0000","last_active":"2024-09-21T12:24:19.880191+0000","last_peered":"2024-09-21T12:24:19.880191+0000","last_clean":"2024-09-21T12:24:19.880191+0000","last_became_active":"2024-09-21T12:20:19.314908+0000","last_became_peered":"2024-09-21T12:20:19.314908+0000","last_unstale":"2024-09-21T12:24:19.880191+0000","last_undegraded":"2024-09-21T12:24:19.880191+0000","last_fullsized":"2024-09-21T12:24:19.880191+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:20.433534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:20.433534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:24.249151+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.883810+0000","last_change":"2024-09-21T12:22:50.914064+0000","last_active":"2024-09-21T12:24:19.883810+0000","last_peered":"2024-09-21T12:24:19.883810+0000","last_clean":"2024-09-21T12:24:19.883810+0000","last_became_active":"2024-09-21T12:22:50.913743+0000","last_became_peered":"2024-09-21T12:22:50.913743+0000","last_unstale":"2024-09-21T12:24:19.883810+0000","last_undegraded":"2024-09-21T12:24:19.883810+0000","last_fullsized":"2024-09-21T12:24:19.883810+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:58.189665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:21:58.189665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:06.335216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":183,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.876062+0000","last_change":"2024-09-21T12:20:21.595607+0000","last_active":"2024-09-21T12:24:19.876062+0000","last_peered":"2024-09-21T12:24:19.876062+0000","last_clean":"2024-09-21T12:24:19.876062+0000","last_became_active":"2024-09-21T12:20:21.595301+0000","last_became_peered":"2024-09-21T12:20:21.595301+0000","last_unstale":"2024-09-21T12:24:19.876062+0000","last_undegraded":"2024-09-21T12:24:19.876062+0000","last_fullsized":"2024-09-21T12:24:19.876062+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:34.199328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":204,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.873358+0000","last_change":"2024-09-21T12:22:42.026488+0000","last_active":"2024-09-21T12:24:19.873358+0000","last_peered":"2024-09-21T12:24:19.873358+0000","last_clean":"2024-09-21T12:24:19.873358+0000","last_became_active":"2024-09-21T12:20:15.308655+0000","last_became_peered":"2024-09-21T12:20:15.308655+0000","last_unstale":"2024-09-21T12:24:19.873358+0000","last_undegraded":"2024-09-21T12:24:19.873358+0000","last_fullsized":"2024-09-21T12:24:19.873358+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:42.026416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:42.026416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":224,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.872612+0000","last_change":"2024-09-21T12:22:45.081280+0000","last_active":"2024-09-21T12:24:19.872612+0000","last_peered":"2024-09-21T12:24:19.872612+0000","last_clean":"2024-09-21T12:24:19.872612+0000","last_became_active":"2024-09-21T12:20:17.305220+0000","last_became_peered":"2024-09-21T12:20:17.305220+0000","last_unstale":"2024-09-21T12:24:19.872612+0000","last_undegraded":"2024-09-21T12:24:19.872612+0000","last_fullsized":"2024-09-21T12:24:19.872612+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:22:45.081089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:45.081089+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":138,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.868104+0000","last_change":"2024-09-21T12:24:01.949534+0000","last_active":"2024-09-21T12:24:19.868104+0000","last_peered":"2024-09-21T12:24:19.868104+0000","last_clean":"2024-09-21T12:24:19.868104+0000","last_became_active":"2024-09-21T12:22:50.926240+0000","last_became_peered":"2024-09-21T12:22:50.926240+0000","last_unstale":"2024-09-21T12:24:19.868104+0000","last_undegraded":"2024-09-21T12:24:19.868104+0000","last_fullsized":"2024-09-21T12:24:19.868104+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:01.949475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:01.949475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:04.277027+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":225,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.884611+0000","last_change":"2024-09-21T12:22:50.912964+0000","last_active":"2024-09-21T12:24:19.884611+0000","last_peered":"2024-09-21T12:24:19.884611+0000","last_clean":"2024-09-21T12:24:19.884611+0000","last_became_active":"2024-09-21T12:22:50.912481+0000","last_became_peered":"2024-09-21T12:22:50.912481+0000","last_unstale":"2024-09-21T12:24:19.884611+0000","last_undegraded":"2024-09-21T12:24:19.884611+0000","last_fullsized":"2024-09-21T12:24:19.884611+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:43.207441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:43.207441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:14.462493+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":185,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.874456+0000","last_change":"2024-09-21T12:22:51.018933+0000","last_active":"2024-09-21T12:24:19.874456+0000","last_peered":"2024-09-21T12:24:19.874456+0000","last_clean":"2024-09-21T12:24:19.874456+0000","last_became_active":"2024-09-21T12:22:50.927804+0000","last_became_peered":"2024-09-21T12:22:50.927804+0000","last_unstale":"2024-09-21T12:24:19.874456+0000","last_undegraded":"2024-09-21T12:24:19.874456+0000","last_fullsized":"2024-09-21T12:24:19.874456+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:22:03.017467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:03.017467+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":196,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.884228+0000","last_change":"2024-09-21T12:22:15.018306+0000","last_active":"2024-09-21T12:24:19.884228+0000","last_peered":"2024-09-21T12:24:19.884228+0000","last_clean":"2024-09-21T12:24:19.884228+0000","last_became_active":"2024-09-21T12:20:19.308184+0000","last_became_peered":"2024-09-21T12:20:19.308184+0000","last_unstale":"2024-09-21T12:24:19.884228+0000","last_undegraded":"2024-09-21T12:24:19.884228+0000","last_fullsized":"2024-09-21T12:24:19.884228+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:15.018264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:15.018264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:45.395051+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":193,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.868059+0000","last_change":"2024-09-21T12:24:04.091870+0000","last_active":"2024-09-21T12:24:19.868059+0000","last_peered":"2024-09-21T12:24:19.868059+0000","last_clean":"2024-09-21T12:24:19.868059+0000","last_became_active":"2024-09-21T12:20:21.309190+0000","last_became_peered":"2024-09-21T12:20:21.309190+0000","last_unstale":"2024-09-21T12:24:19.868059+0000","last_undegraded":"2024-09-21T12:24:19.868059+0000","last_fullsized":"2024-09-21T12:24:19.868059+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:04.091807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:04.091807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:21.906035+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":225,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.884144+0000","last_change":"2024-09-21T12:22:54.174099+0000","last_active":"2024-09-21T12:24:19.884144+0000","last_peered":"2024-09-21T12:24:19.884144+0000","last_clean":"2024-09-21T12:24:19.884144+0000","last_became_active":"2024-09-21T12:22:50.911737+0000","last_became_peered":"2024-09-21T12:22:50.911737+0000","last_unstale":"2024-09-21T12:24:19.884144+0000","last_undegraded":"2024-09-21T12:24:19.884144+0000","last_fullsized":"2024-09-21T12:24:19.884144+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:54.174034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:54.174034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:20.695067+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":171,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.876473+0000","last_change":"2024-09-21T12:22:51.038838+0000","last_active":"2024-09-21T12:24:19.876473+0000","last_peered":"2024-09-21T12:24:19.876473+0000","last_clean":"2024-09-21T12:24:19.876473+0000","last_became_active":"2024-09-21T12:22:50.918794+0000","last_became_peered":"2024-09-21T12:22:50.918794+0000","last_unstale":"2024-09-21T12:24:19.876473+0000","last_undegraded":"2024-09-21T12:24:19.876473+0000","last_fullsized":"2024-09-21T12:24:19.876473+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-21T12:22:10.011633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:10.011633+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":209,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.865458+0000","last_change":"2024-09-21T12:22:50.922087+0000","last_active":"2024-09-21T12:24:19.865458+0000","last_peered":"2024-09-21T12:24:19.865458+0000","last_clean":"2024-09-21T12:24:19.865458+0000","last_became_active":"2024-09-21T12:22:50.921780+0000","last_became_peered":"2024-09-21T12:22:50.921780+0000","last_unstale":"2024-09-21T12:24:19.865458+0000","last_undegraded":"2024-09-21T12:24:19.865458+0000","last_fullsized":"2024-09-21T12:24:19.865458+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:42.023401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":200,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.883812+0000","last_change":"2024-09-21T12:23:27.383183+0000","last_active":"2024-09-21T12:24:19.883812+0000","last_peered":"2024-09-21T12:24:19.883812+0000","last_clean":"2024-09-21T12:24:19.883812+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:24:19.883812+0000","last_undegraded":"2024-09-21T12:24:19.883812+0000","last_fullsized":"2024-09-21T12:24:19.883812+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:27.383128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:27.383128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:32.310862+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":137,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.868809+0000","last_change":"2024-09-21T12:23:56.060558+0000","last_active":"2024-09-21T12:24:19.868809+0000","last_peered":"2024-09-21T12:24:19.868809+0000","last_clean":"2024-09-21T12:24:19.868809+0000","last_became_active":"2024-09-21T12:22:50.914792+0000","last_became_peered":"2024-09-21T12:22:50.914792+0000","last_unstale":"2024-09-21T12:24:19.868809+0000","last_undegraded":"2024-09-21T12:24:19.868809+0000","last_fullsized":"2024-09-21T12:24:19.868809+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:56.060488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:56.060488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:20.735686+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":234,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.869511+0000","last_change":"2024-09-21T12:23:19.156525+0000","last_active":"2024-09-21T12:24:19.869511+0000","last_peered":"2024-09-21T12:24:19.869511+0000","last_clean":"2024-09-21T12:24:19.869511+0000","last_became_active":"2024-09-21T12:22:50.921576+0000","last_became_peered":"2024-09-21T12:22:50.921576+0000","last_unstale":"2024-09-21T12:24:19.869511+0000","last_undegraded":"2024-09-21T12:24:19.869511+0000","last_fullsized":"2024-09-21T12:24:19.869511+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:19.156456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:23:19.156456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:24.315189+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":226,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.882480+0000","last_change":"2024-09-21T12:23:26.320324+0000","last_active":"2024-09-21T12:24:19.882480+0000","last_peered":"2024-09-21T12:24:19.882480+0000","last_clean":"2024-09-21T12:24:19.882480+0000","last_became_active":"2024-09-21T12:20:17.307828+0000","last_became_peered":"2024-09-21T12:20:17.307828+0000","last_unstale":"2024-09-21T12:24:19.882480+0000","last_undegraded":"2024-09-21T12:24:19.882480+0000","last_fullsized":"2024-09-21T12:24:19.882480+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:23:26.320231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:26.320231+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:38.921474+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":226,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.883885+0000","last_change":"2024-09-21T12:23:48.435833+0000","last_active":"2024-09-21T12:24:19.883885+0000","last_peered":"2024-09-21T12:24:19.883885+0000","last_clean":"2024-09-21T12:24:19.883885+0000","last_became_active":"2024-09-21T12:22:50.928972+0000","last_became_peered":"2024-09-21T12:22:50.928972+0000","last_unstale":"2024-09-21T12:24:19.883885+0000","last_undegraded":"2024-09-21T12:24:19.883885+0000","last_fullsized":"2024-09-21T12:24:19.883885+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:48.435769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:48.435769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:05.008777+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":191,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.875360+0000","last_change":"2024-09-21T12:23:58.354628+0000","last_active":"2024-09-21T12:24:19.875360+0000","last_peered":"2024-09-21T12:24:19.875360+0000","last_clean":"2024-09-21T12:24:19.875360+0000","last_became_active":"2024-09-21T12:20:21.592796+0000","last_became_peered":"2024-09-21T12:20:21.592796+0000","last_unstale":"2024-09-21T12:24:19.875360+0000","last_undegraded":"2024-09-21T12:24:19.875360+0000","last_fullsized":"2024-09-21T12:24:19.875360+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:58.354570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:58.354570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:05.826983+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":137,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.887122+0000","last_change":"2024-09-21T12:23:53.441120+0000","last_active":"2024-09-21T12:24:19.887122+0000","last_peered":"2024-09-21T12:24:19.887122+0000","last_clean":"2024-09-21T12:24:19.887122+0000","last_became_active":"2024-09-21T12:22:50.925403+0000","last_became_peered":"2024-09-21T12:22:50.925403+0000","last_unstale":"2024-09-21T12:24:19.887122+0000","last_undegraded":"2024-09-21T12:24:19.887122+0000","last_fullsized":"2024-09-21T12:24:19.887122+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:53.440795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:23:53.440795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:54.533332+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":279,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.872713+0000","last_change":"2024-09-21T12:23:47.285620+0000","last_active":"2024-09-21T12:24:19.872713+0000","last_peered":"2024-09-21T12:24:19.872713+0000","last_clean":"2024-09-21T12:24:19.872713+0000","last_became_active":"2024-09-21T12:22:50.912144+0000","last_became_peered":"2024-09-21T12:22:50.912144+0000","last_unstale":"2024-09-21T12:24:19.872713+0000","last_undegraded":"2024-09-21T12:24:19.872713+0000","last_fullsized":"2024-09-21T12:24:19.872713+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:23:47.285537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:47.285537+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:11.927780+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":122,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.879254+0000","last_change":"2024-09-21T12:22:50.920178+0000","last_active":"2024-09-21T12:24:19.879254+0000","last_peered":"2024-09-21T12:24:19.879254+0000","last_clean":"2024-09-21T12:24:19.879254+0000","last_became_active":"2024-09-21T12:22:50.918326+0000","last_became_peered":"2024-09-21T12:22:50.918326+0000","last_unstale":"2024-09-21T12:24:19.879254+0000","last_undegraded":"2024-09-21T12:24:19.879254+0000","last_fullsized":"2024-09-21T12:24:19.879254+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:41.091819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":193,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.865963+0000","last_change":"2024-09-21T12:23:51.065780+0000","last_active":"2024-09-21T12:24:19.865963+0000","last_peered":"2024-09-21T12:24:19.865963+0000","last_clean":"2024-09-21T12:24:19.865963+0000","last_became_active":"2024-09-21T12:20:21.312976+0000","last_became_peered":"2024-09-21T12:20:21.312976+0000","last_unstale":"2024-09-21T12:24:19.865963+0000","last_undegraded":"2024-09-21T12:24:19.865963+0000","last_fullsized":"2024-09-21T12:24:19.865963+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:23:51.065733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:51.065733+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:51.472146+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":239,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.880350+0000","last_change":"2024-09-21T12:22:52.070589+0000","last_active":"2024-09-21T12:24:19.880350+0000","last_peered":"2024-09-21T12:24:19.880350+0000","last_clean":"2024-09-21T12:24:19.880350+0000","last_became_active":"2024-09-21T12:22:50.910933+0000","last_became_peered":"2024-09-21T12:22:50.910933+0000","last_unstale":"2024-09-21T12:24:19.880350+0000","last_undegraded":"2024-09-21T12:24:19.880350+0000","last_fullsized":"2024-09-21T12:24:19.880350+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:22:52.070513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:52.070513+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:12.242749+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":217,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.874036+0000","last_change":"2024-09-21T12:24:03.059782+0000","last_active":"2024-09-21T12:24:19.874036+0000","last_peered":"2024-09-21T12:24:19.874036+0000","last_clean":"2024-09-21T12:24:19.874036+0000","last_became_active":"2024-09-21T12:20:17.695977+0000","last_became_peered":"2024-09-21T12:20:17.695977+0000","last_unstale":"2024-09-21T12:24:19.874036+0000","last_undegraded":"2024-09-21T12:24:19.874036+0000","last_fullsized":"2024-09-21T12:24:19.874036+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:24:03.059549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:03.059549+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:08.313061+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"121'11","reported_seq":290,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.869136+0000","last_change":"2024-09-21T12:22:50.926408+0000","last_active":"2024-09-21T12:24:19.869136+0000","last_peered":"2024-09-21T12:24:19.869136+0000","last_clean":"2024-09-21T12:24:19.869136+0000","last_became_active":"2024-09-21T12:22:50.916683+0000","last_became_peered":"2024-09-21T12:22:50.916683+0000","last_unstale":"2024-09-21T12:24:19.869136+0000","last_undegraded":"2024-09-21T12:24:19.869136+0000","last_fullsized":"2024-09-21T12:24:19.869136+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'9","last_scrub_stamp":"2024-09-21T12:22:08.647997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:08.647997+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:43.003218+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"76'79","reported_seq":105,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.883696+0000","last_change":"2024-09-21T12:22:18.000752+0000","last_active":"2024-09-21T12:24:19.883696+0000","last_peered":"2024-09-21T12:24:19.883696+0000","last_clean":"2024-09-21T12:24:19.883696+0000","last_became_active":"2024-09-21T12:22:17.074721+0000","last_became_peered":"2024-09-21T12:22:17.074721+0000","last_unstale":"2024-09-21T12:24:19.883696+0000","last_undegraded":"2024-09-21T12:24:19.883696+0000","last_fullsized":"2024-09-21T12:24:19.883696+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:22:18.000705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:22:18.000705+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:39.443289+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"151'12072","reported_seq":12280,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:21.574591+0000","last_change":"2024-09-21T12:22:42.121876+0000","last_active":"2024-09-21T12:24:21.574591+0000","last_peered":"2024-09-21T12:24:21.574591+0000","last_clean":"2024-09-21T12:24:21.574591+0000","last_became_active":"2024-09-21T12:21:24.886130+0000","last_became_peered":"2024-09-21T12:21:24.886130+0000","last_unstale":"2024-09-21T12:24:21.574591+0000","last_undegraded":"2024-09-21T12:24:21.574591+0000","last_fullsized":"2024-09-21T12:24:21.574591+0000","mapping_epoch":84,"log_start":"130'7447","ondisk_log_start":"130'7447","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5485","last_scrub_stamp":"2024-09-21T12:22:42.121763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:22:42.121763+0000","objects_scrubbed":5450,"log_size":4625,"log_dups_size":0,"ondisk_log_size":4625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1486,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54919168,"num_objects":838,"num_object_clones":0,"num_object_copies":2514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12072,"num_write_kb":413120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":56,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.881727+0000","last_change":"2024-09-21T12:23:47.449186+0000","last_active":"2024-09-21T12:24:19.881727+0000","last_peered":"2024-09-21T12:24:19.881727+0000","last_clean":"2024-09-21T12:24:19.881727+0000","last_became_active":"2024-09-21T12:23:46.910112+0000","last_became_peered":"2024-09-21T12:23:46.910112+0000","last_unstale":"2024-09-21T12:24:19.881727+0000","last_undegraded":"2024-09-21T12:24:19.881727+0000","last_fullsized":"2024-09-21T12:24:19.881727+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:23:47.449113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:47.449113+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:59.572023+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":203,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.875754+0000","last_change":"2024-09-21T12:22:14.036290+0000","last_active":"2024-09-21T12:24:19.875754+0000","last_peered":"2024-09-21T12:24:19.875754+0000","last_clean":"2024-09-21T12:24:19.875754+0000","last_became_active":"2024-09-21T12:20:15.295774+0000","last_became_peered":"2024-09-21T12:20:15.295774+0000","last_unstale":"2024-09-21T12:24:19.875754+0000","last_undegraded":"2024-09-21T12:24:19.875754+0000","last_fullsized":"2024-09-21T12:24:19.875754+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:14.036236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.036236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"130'12","reported_seq":81,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.866476+0000","last_change":"2024-09-21T12:24:01.051310+0000","last_active":"2024-09-21T12:24:19.866476+0000","last_peered":"2024-09-21T12:24:19.866476+0000","last_clean":"2024-09-21T12:24:19.866476+0000","last_became_active":"2024-09-21T12:23:11.827128+0000","last_became_peered":"2024-09-21T12:23:11.827128+0000","last_unstale":"2024-09-21T12:24:19.866476+0000","last_undegraded":"2024-09-21T12:24:19.866476+0000","last_fullsized":"2024-09-21T12:24:19.866476+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'12","last_scrub_stamp":"2024-09-21T12:24:01.051257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:01.051257+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:20.741040+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":193,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.884321+0000","last_change":"2024-09-21T12:22:12.145637+0000","last_active":"2024-09-21T12:24:19.884321+0000","last_peered":"2024-09-21T12:24:19.884321+0000","last_clean":"2024-09-21T12:24:19.884321+0000","last_became_active":"2024-09-21T12:20:21.310022+0000","last_became_peered":"2024-09-21T12:20:21.310022+0000","last_unstale":"2024-09-21T12:24:19.884321+0000","last_undegraded":"2024-09-21T12:24:19.884321+0000","last_fullsized":"2024-09-21T12:24:19.884321+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:22:12.145550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:12.145550+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:44.423363+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"76'79","reported_seq":95,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.881838+0000","last_change":"2024-09-21T12:22:50.916239+0000","last_active":"2024-09-21T12:24:19.881838+0000","last_peered":"2024-09-21T12:24:19.881838+0000","last_clean":"2024-09-21T12:24:19.881838+0000","last_became_active":"2024-09-21T12:22:50.914885+0000","last_became_peered":"2024-09-21T12:22:50.914885+0000","last_unstale":"2024-09-21T12:24:19.881838+0000","last_undegraded":"2024-09-21T12:24:19.881838+0000","last_fullsized":"2024-09-21T12:24:19.881838+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:22:17.824342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:22:17.824342+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:07.348734+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.3","version":"151'12021","reported_seq":15293,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:21.962050+0000","last_change":"2024-09-21T12:24:04.025445+0000","last_active":"2024-09-21T12:24:21.962050+0000","last_peered":"2024-09-21T12:24:21.962050+0000","last_clean":"2024-09-21T12:24:21.962050+0000","last_became_active":"2024-09-21T12:22:50.919145+0000","last_became_peered":"2024-09-21T12:22:50.919145+0000","last_unstale":"2024-09-21T12:24:21.962050+0000","last_undegraded":"2024-09-21T12:24:21.962050+0000","last_fullsized":"2024-09-21T12:24:21.962050+0000","mapping_epoch":120,"log_start":"133'7966","ondisk_log_start":"133'7966","created":84,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'10775","last_scrub_stamp":"2024-09-21T12:24:04.025071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:24:04.025071+0000","objects_scrubbed":2081,"log_size":4055,"log_dups_size":0,"ondisk_log_size":4055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:15.894620+0000","scrub_duration":745,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53149696,"num_objects":811,"num_object_clones":0,"num_object_copies":2433,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12021,"num_write_kb":410624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1536,"num_bytes_recovered":100663296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":232,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.875547+0000","last_change":"2024-09-21T12:23:49.326670+0000","last_active":"2024-09-21T12:24:19.875547+0000","last_peered":"2024-09-21T12:24:19.875547+0000","last_clean":"2024-09-21T12:24:19.875547+0000","last_became_active":"2024-09-21T12:20:17.693179+0000","last_became_peered":"2024-09-21T12:20:17.693179+0000","last_unstale":"2024-09-21T12:24:19.875547+0000","last_undegraded":"2024-09-21T12:24:19.875547+0000","last_fullsized":"2024-09-21T12:24:19.875547+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-21T12:23:49.326428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:49.326428+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:15.058038+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":203,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.877068+0000","last_change":"2024-09-21T12:21:18.056305+0000","last_active":"2024-09-21T12:24:19.877068+0000","last_peered":"2024-09-21T12:24:19.877068+0000","last_clean":"2024-09-21T12:24:19.877068+0000","last_became_active":"2024-09-21T12:20:15.297340+0000","last_became_peered":"2024-09-21T12:20:15.297340+0000","last_unstale":"2024-09-21T12:24:19.877068+0000","last_undegraded":"2024-09-21T12:24:19.877068+0000","last_fullsized":"2024-09-21T12:24:19.877068+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:18.056261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:18.056261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:04.326707+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":130,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.872277+0000","last_change":"2024-09-21T12:22:50.931233+0000","last_active":"2024-09-21T12:24:19.872277+0000","last_peered":"2024-09-21T12:24:19.872277+0000","last_clean":"2024-09-21T12:24:19.872277+0000","last_became_active":"2024-09-21T12:22:50.930858+0000","last_became_peered":"2024-09-21T12:22:50.930858+0000","last_unstale":"2024-09-21T12:24:19.872277+0000","last_undegraded":"2024-09-21T12:24:19.872277+0000","last_fullsized":"2024-09-21T12:24:19.872277+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:01.971812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:01.971812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.874420+0000","last_change":"2024-09-21T12:22:50.927366+0000","last_active":"2024-09-21T12:24:19.874420+0000","last_peered":"2024-09-21T12:24:19.874420+0000","last_clean":"2024-09-21T12:24:19.874420+0000","last_became_active":"2024-09-21T12:22:50.918940+0000","last_became_peered":"2024-09-21T12:22:50.918940+0000","last_unstale":"2024-09-21T12:24:19.874420+0000","last_undegraded":"2024-09-21T12:24:19.874420+0000","last_fullsized":"2024-09-21T12:24:19.874420+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:13.050383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:13.050383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"76'79","reported_seq":111,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.879115+0000","last_change":"2024-09-21T12:23:23.217945+0000","last_active":"2024-09-21T12:24:19.879115+0000","last_peered":"2024-09-21T12:24:19.879115+0000","last_clean":"2024-09-21T12:24:19.879115+0000","last_became_active":"2024-09-21T12:22:17.069042+0000","last_became_peered":"2024-09-21T12:22:17.069042+0000","last_unstale":"2024-09-21T12:24:19.879115+0000","last_undegraded":"2024-09-21T12:24:19.879115+0000","last_fullsized":"2024-09-21T12:24:19.879115+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:23:23.217863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:23:23.217863+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:33.585014+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"151'12332","reported_seq":12476,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:21.996456+0000","last_change":"2024-09-21T12:21:24.889433+0000","last_active":"2024-09-21T12:24:21.996456+0000","last_peered":"2024-09-21T12:24:21.996456+0000","last_clean":"2024-09-21T12:24:21.996456+0000","last_became_active":"2024-09-21T12:21:24.889133+0000","last_became_peered":"2024-09-21T12:21:24.889133+0000","last_unstale":"2024-09-21T12:24:21.996456+0000","last_undegraded":"2024-09-21T12:24:21.996456+0000","last_fullsized":"2024-09-21T12:24:21.996456+0000","mapping_epoch":84,"log_start":"133'8313","ondisk_log_start":"133'8313","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4019,"log_dups_size":0,"ondisk_log_size":4019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:11.396154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57016320,"num_objects":870,"num_object_clones":0,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12332,"num_write_kb":422464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":202,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.883949+0000","last_change":"2024-09-21T12:22:50.978102+0000","last_active":"2024-09-21T12:24:19.883949+0000","last_peered":"2024-09-21T12:24:19.883949+0000","last_clean":"2024-09-21T12:24:19.883949+0000","last_became_active":"2024-09-21T12:22:50.926008+0000","last_became_peered":"2024-09-21T12:22:50.926008+0000","last_unstale":"2024-09-21T12:24:19.883949+0000","last_undegraded":"2024-09-21T12:24:19.883949+0000","last_fullsized":"2024-09-21T12:24:19.883949+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-21T12:22:14.062464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.062464+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:41.411251+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":239,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.874235+0000","last_change":"2024-09-21T12:22:51.003768+0000","last_active":"2024-09-21T12:24:19.874235+0000","last_peered":"2024-09-21T12:24:19.874235+0000","last_clean":"2024-09-21T12:24:19.874235+0000","last_became_active":"2024-09-21T12:22:50.923284+0000","last_became_peered":"2024-09-21T12:22:50.923284+0000","last_unstale":"2024-09-21T12:24:19.874235+0000","last_undegraded":"2024-09-21T12:24:19.874235+0000","last_fullsized":"2024-09-21T12:24:19.874235+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:22:07.582969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:07.582969+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:35.130245+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"121'11","reported_seq":286,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.875324+0000","last_change":"2024-09-21T12:23:27.244152+0000","last_active":"2024-09-21T12:24:19.875324+0000","last_peered":"2024-09-21T12:24:19.875324+0000","last_clean":"2024-09-21T12:24:19.875324+0000","last_became_active":"2024-09-21T12:22:50.930342+0000","last_became_peered":"2024-09-21T12:22:50.930342+0000","last_unstale":"2024-09-21T12:24:19.875324+0000","last_undegraded":"2024-09-21T12:24:19.875324+0000","last_fullsized":"2024-09-21T12:24:19.875324+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:23:27.243903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:27.243903+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:54.351518+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":192,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.865686+0000","last_change":"2024-09-21T12:21:21.835669+0000","last_active":"2024-09-21T12:24:19.865686+0000","last_peered":"2024-09-21T12:24:19.865686+0000","last_clean":"2024-09-21T12:24:19.865686+0000","last_became_active":"2024-09-21T12:20:21.593367+0000","last_became_peered":"2024-09-21T12:20:21.593367+0000","last_unstale":"2024-09-21T12:24:19.865686+0000","last_undegraded":"2024-09-21T12:24:19.865686+0000","last_fullsized":"2024-09-21T12:24:19.865686+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:21.835623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:21:21.835623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:17.066934+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"76'79","reported_seq":195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.868772+0000","last_change":"2024-09-21T12:23:26.056027+0000","last_active":"2024-09-21T12:24:19.868772+0000","last_peered":"2024-09-21T12:24:19.868772+0000","last_clean":"2024-09-21T12:24:19.868772+0000","last_became_active":"2024-09-21T12:22:17.093298+0000","last_became_peered":"2024-09-21T12:22:17.093298+0000","last_unstale":"2024-09-21T12:24:19.868772+0000","last_undegraded":"2024-09-21T12:24:19.868772+0000","last_fullsized":"2024-09-21T12:24:19.868772+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:23:26.055979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:23:26.055979+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:34.028294+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"151'12476","reported_seq":21672,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:22.003733+0000","last_change":"2024-09-21T12:22:51.819626+0000","last_active":"2024-09-21T12:24:22.003733+0000","last_peered":"2024-09-21T12:24:22.003733+0000","last_clean":"2024-09-21T12:24:22.003733+0000","last_became_active":"2024-09-21T12:22:51.808221+0000","last_became_peered":"2024-09-21T12:22:51.808221+0000","last_unstale":"2024-09-21T12:24:22.003733+0000","last_undegraded":"2024-09-21T12:24:22.003733+0000","last_fullsized":"2024-09-21T12:24:22.003733+0000","mapping_epoch":121,"log_start":"133'8388","ondisk_log_start":"133'8388","created":84,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4088,"log_dups_size":0,"ondisk_log_size":4088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:19.598845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58458112,"num_objects":892,"num_object_clones":0,"num_object_copies":2676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12476,"num_write_kb":427776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4516,"num_bytes_recovered":295960576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":273,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.886612+0000","last_change":"2024-09-21T12:24:01.388987+0000","last_active":"2024-09-21T12:24:19.886612+0000","last_peered":"2024-09-21T12:24:19.886612+0000","last_clean":"2024-09-21T12:24:19.886612+0000","last_became_active":"2024-09-21T12:22:50.925840+0000","last_became_peered":"2024-09-21T12:22:50.925840+0000","last_unstale":"2024-09-21T12:24:19.886612+0000","last_undegraded":"2024-09-21T12:24:19.886612+0000","last_fullsized":"2024-09-21T12:24:19.886612+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:24:01.388905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:01.388905+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:06.868982+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":204,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.884741+0000","last_change":"2024-09-21T12:22:37.176483+0000","last_active":"2024-09-21T12:24:19.884741+0000","last_peered":"2024-09-21T12:24:19.884741+0000","last_clean":"2024-09-21T12:24:19.884741+0000","last_became_active":"2024-09-21T12:20:15.308518+0000","last_became_peered":"2024-09-21T12:20:15.308518+0000","last_unstale":"2024-09-21T12:24:19.884741+0000","last_undegraded":"2024-09-21T12:24:19.884741+0000","last_fullsized":"2024-09-21T12:24:19.884741+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:37.176413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:37.176413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:08.307375+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":258,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.869135+0000","last_change":"2024-09-21T12:21:23.139205+0000","last_active":"2024-09-21T12:24:19.869135+0000","last_peered":"2024-09-21T12:24:19.869135+0000","last_clean":"2024-09-21T12:24:19.869135+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:24:19.869135+0000","last_undegraded":"2024-09-21T12:24:19.869135+0000","last_fullsized":"2024-09-21T12:24:19.869135+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:21:23.139124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.139124+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:30.139833+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":193,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.868188+0000","last_change":"2024-09-21T12:22:51.987032+0000","last_active":"2024-09-21T12:24:19.868188+0000","last_peered":"2024-09-21T12:24:19.868188+0000","last_clean":"2024-09-21T12:24:19.868188+0000","last_became_active":"2024-09-21T12:20:21.315801+0000","last_became_peered":"2024-09-21T12:20:21.315801+0000","last_unstale":"2024-09-21T12:24:19.868188+0000","last_undegraded":"2024-09-21T12:24:19.868188+0000","last_fullsized":"2024-09-21T12:24:19.868188+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:51.986953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:51.986953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:19.008252+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"76'79","reported_seq":195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.868714+0000","last_change":"2024-09-21T12:23:25.039495+0000","last_active":"2024-09-21T12:24:19.868714+0000","last_peered":"2024-09-21T12:24:19.868714+0000","last_clean":"2024-09-21T12:24:19.868714+0000","last_became_active":"2024-09-21T12:22:17.083629+0000","last_became_peered":"2024-09-21T12:22:17.083629+0000","last_unstale":"2024-09-21T12:24:19.868714+0000","last_undegraded":"2024-09-21T12:24:19.868714+0000","last_fullsized":"2024-09-21T12:24:19.868714+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:23:25.039448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:23:25.039448+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:48.971864+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"151'11939","reported_seq":12150,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:21.550948+0000","last_change":"2024-09-21T12:22:48.664422+0000","last_active":"2024-09-21T12:24:21.550948+0000","last_peered":"2024-09-21T12:24:21.550948+0000","last_clean":"2024-09-21T12:24:21.550948+0000","last_became_active":"2024-09-21T12:21:24.884183+0000","last_became_peered":"2024-09-21T12:21:24.884183+0000","last_unstale":"2024-09-21T12:24:21.550948+0000","last_undegraded":"2024-09-21T12:24:21.550948+0000","last_fullsized":"2024-09-21T12:24:21.550948+0000","mapping_epoch":84,"log_start":"129'7105","ondisk_log_start":"129'7105","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'5758","last_scrub_stamp":"2024-09-21T12:22:48.664234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:22:48.664234+0000","objects_scrubbed":5726,"log_size":4834,"log_dups_size":0,"ondisk_log_size":4834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:17.405538+0000","scrub_duration":1463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56688640,"num_objects":865,"num_object_clones":0,"num_object_copies":2595,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11939,"num_write_kb":409728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":237,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.869256+0000","last_change":"2024-09-21T12:23:22.124368+0000","last_active":"2024-09-21T12:24:19.869256+0000","last_peered":"2024-09-21T12:24:19.869256+0000","last_clean":"2024-09-21T12:24:19.869256+0000","last_became_active":"2024-09-21T12:20:17.693067+0000","last_became_peered":"2024-09-21T12:20:17.693067+0000","last_unstale":"2024-09-21T12:24:19.869256+0000","last_undegraded":"2024-09-21T12:24:19.869256+0000","last_fullsized":"2024-09-21T12:24:19.869256+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-21T12:23:22.124283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:22.124283+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:29.366927+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":197,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.881785+0000","last_change":"2024-09-21T12:20:15.308741+0000","last_active":"2024-09-21T12:24:19.881785+0000","last_peered":"2024-09-21T12:24:19.881785+0000","last_clean":"2024-09-21T12:24:19.881785+0000","last_became_active":"2024-09-21T12:20:15.308441+0000","last_became_peered":"2024-09-21T12:20:15.308441+0000","last_unstale":"2024-09-21T12:24:19.881785+0000","last_undegraded":"2024-09-21T12:24:19.881785+0000","last_fullsized":"2024-09-21T12:24:19.881785+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:06.290912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":196,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.868299+0000","last_change":"2024-09-21T12:22:34.563365+0000","last_active":"2024-09-21T12:24:19.868299+0000","last_peered":"2024-09-21T12:24:19.868299+0000","last_clean":"2024-09-21T12:24:19.868299+0000","last_became_active":"2024-09-21T12:20:19.309598+0000","last_became_peered":"2024-09-21T12:20:19.309598+0000","last_unstale":"2024-09-21T12:24:19.868299+0000","last_undegraded":"2024-09-21T12:24:19.868299+0000","last_fullsized":"2024-09-21T12:24:19.868299+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:34.563282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:34.563282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:06.982730+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":185,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.880109+0000","last_change":"2024-09-21T12:20:21.593173+0000","last_active":"2024-09-21T12:24:19.880109+0000","last_peered":"2024-09-21T12:24:19.880109+0000","last_clean":"2024-09-21T12:24:19.880109+0000","last_became_active":"2024-09-21T12:20:21.592828+0000","last_became_peered":"2024-09-21T12:20:21.592828+0000","last_unstale":"2024-09-21T12:24:19.880109+0000","last_undegraded":"2024-09-21T12:24:19.880109+0000","last_fullsized":"2024-09-21T12:24:19.880109+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:39.438547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"76'79","reported_seq":106,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.883698+0000","last_change":"2024-09-21T12:23:21.346117+0000","last_active":"2024-09-21T12:24:19.883698+0000","last_peered":"2024-09-21T12:24:19.883698+0000","last_clean":"2024-09-21T12:24:19.883698+0000","last_became_active":"2024-09-21T12:22:17.084732+0000","last_became_peered":"2024-09-21T12:22:17.084732+0000","last_unstale":"2024-09-21T12:24:19.883698+0000","last_undegraded":"2024-09-21T12:24:19.883698+0000","last_fullsized":"2024-09-21T12:24:19.883698+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:23:21.346046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:23:21.346046+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:35.936936+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"151'12370","reported_seq":12582,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:21.574035+0000","last_change":"2024-09-21T12:22:50.126449+0000","last_active":"2024-09-21T12:24:21.574035+0000","last_peered":"2024-09-21T12:24:21.574035+0000","last_clean":"2024-09-21T12:24:21.574035+0000","last_became_active":"2024-09-21T12:21:24.886854+0000","last_became_peered":"2024-09-21T12:21:24.886854+0000","last_unstale":"2024-09-21T12:24:21.574035+0000","last_undegraded":"2024-09-21T12:24:21.574035+0000","last_fullsized":"2024-09-21T12:24:21.574035+0000","mapping_epoch":84,"log_start":"130'7800","ondisk_log_start":"130'7800","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'6045","last_scrub_stamp":"2024-09-21T12:22:50.126349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:22:50.126349+0000","objects_scrubbed":6025,"log_size":4570,"log_dups_size":0,"ondisk_log_size":4570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1386,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56623104,"num_objects":864,"num_object_clones":0,"num_object_copies":2592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12370,"num_write_kb":423488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":291,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.881110+0000","last_change":"2024-09-21T12:20:17.308049+0000","last_active":"2024-09-21T12:24:19.881110+0000","last_peered":"2024-09-21T12:24:19.881110+0000","last_clean":"2024-09-21T12:24:19.881110+0000","last_became_active":"2024-09-21T12:20:17.307769+0000","last_became_peered":"2024-09-21T12:20:17.307769+0000","last_unstale":"2024-09-21T12:24:19.881110+0000","last_undegraded":"2024-09-21T12:24:19.881110+0000","last_fullsized":"2024-09-21T12:24:19.881110+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:05.312793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.869443+0000","last_change":"2024-09-21T12:22:50.922961+0000","last_active":"2024-09-21T12:24:19.869443+0000","last_peered":"2024-09-21T12:24:19.869443+0000","last_clean":"2024-09-21T12:24:19.869443+0000","last_became_active":"2024-09-21T12:22:50.922002+0000","last_became_peered":"2024-09-21T12:22:50.922002+0000","last_unstale":"2024-09-21T12:24:19.869443+0000","last_undegraded":"2024-09-21T12:24:19.869443+0000","last_fullsized":"2024-09-21T12:24:19.869443+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:05.507899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:05.507899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:34.159504+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":214,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.884150+0000","last_change":"2024-09-21T12:21:20.045824+0000","last_active":"2024-09-21T12:24:19.884150+0000","last_peered":"2024-09-21T12:24:19.884150+0000","last_clean":"2024-09-21T12:24:19.884150+0000","last_became_active":"2024-09-21T12:20:14.287998+0000","last_became_peered":"2024-09-21T12:20:14.287998+0000","last_unstale":"2024-09-21T12:24:19.884150+0000","last_undegraded":"2024-09-21T12:24:19.884150+0000","last_fullsized":"2024-09-21T12:24:19.884150+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T12:21:20.045755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.045755+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:05.292532+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011011149999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":218,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.874695+0000","last_change":"2024-09-21T12:22:50.921798+0000","last_active":"2024-09-21T12:24:19.874695+0000","last_peered":"2024-09-21T12:24:19.874695+0000","last_clean":"2024-09-21T12:24:19.874695+0000","last_became_active":"2024-09-21T12:22:50.921488+0000","last_became_peered":"2024-09-21T12:22:50.921488+0000","last_unstale":"2024-09-21T12:24:19.874695+0000","last_undegraded":"2024-09-21T12:24:19.874695+0000","last_fullsized":"2024-09-21T12:24:19.874695+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:06.563104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:06.563104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:41.170121+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":212,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.873182+0000","last_change":"2024-09-21T12:23:20.237788+0000","last_active":"2024-09-21T12:24:19.873182+0000","last_peered":"2024-09-21T12:24:19.873182+0000","last_clean":"2024-09-21T12:24:19.873182+0000","last_became_active":"2024-09-21T12:22:50.917558+0000","last_became_peered":"2024-09-21T12:22:50.917558+0000","last_unstale":"2024-09-21T12:24:19.873182+0000","last_undegraded":"2024-09-21T12:24:19.873182+0000","last_fullsized":"2024-09-21T12:24:19.873182+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:20.237715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:20.237715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:40.331198+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'79","reported_seq":281,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.864643+0000","last_change":"2024-09-21T12:23:19.867918+0000","last_active":"2024-09-21T12:24:19.864643+0000","last_peered":"2024-09-21T12:24:19.864643+0000","last_clean":"2024-09-21T12:24:19.864643+0000","last_became_active":"2024-09-21T12:22:50.921709+0000","last_became_peered":"2024-09-21T12:22:50.921709+0000","last_unstale":"2024-09-21T12:24:19.864643+0000","last_undegraded":"2024-09-21T12:24:19.864643+0000","last_fullsized":"2024-09-21T12:24:19.864643+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:23:19.867868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:23:19.867868+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:37.496791+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"151'12000","reported_seq":15123,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:21.975982+0000","last_change":"2024-09-21T12:22:51.045889+0000","last_active":"2024-09-21T12:24:21.975982+0000","last_peered":"2024-09-21T12:24:21.975982+0000","last_clean":"2024-09-21T12:24:21.975982+0000","last_became_active":"2024-09-21T12:22:50.936833+0000","last_became_peered":"2024-09-21T12:22:50.936833+0000","last_unstale":"2024-09-21T12:24:21.975982+0000","last_undegraded":"2024-09-21T12:24:21.975982+0000","last_fullsized":"2024-09-21T12:24:21.975982+0000","mapping_epoch":120,"log_start":"132'7753","ondisk_log_start":"132'7753","created":84,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4247,"log_dups_size":0,"ondisk_log_size":4247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:07.096316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57802752,"num_objects":882,"num_object_clones":0,"num_object_copies":2646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12000,"num_write_kb":412224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1475,"num_bytes_recovered":96665600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":220,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.880712+0000","last_change":"2024-09-21T12:20:17.297678+0000","last_active":"2024-09-21T12:24:19.880712+0000","last_peered":"2024-09-21T12:24:19.880712+0000","last_clean":"2024-09-21T12:24:19.880712+0000","last_became_active":"2024-09-21T12:20:17.297374+0000","last_became_peered":"2024-09-21T12:20:17.297374+0000","last_unstale":"2024-09-21T12:24:19.880712+0000","last_undegraded":"2024-09-21T12:24:19.880712+0000","last_fullsized":"2024-09-21T12:24:19.880712+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:31.489358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":214,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.881075+0000","last_change":"2024-09-21T12:23:59.367953+0000","last_active":"2024-09-21T12:24:19.881075+0000","last_peered":"2024-09-21T12:24:19.881075+0000","last_clean":"2024-09-21T12:24:19.881075+0000","last_became_active":"2024-09-21T12:20:15.308356+0000","last_became_peered":"2024-09-21T12:20:15.308356+0000","last_unstale":"2024-09-21T12:24:19.881075+0000","last_undegraded":"2024-09-21T12:24:19.881075+0000","last_fullsized":"2024-09-21T12:24:19.881075+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:59.367778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:23:59.367778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:12.420396+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":214,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.872857+0000","last_change":"2024-09-21T12:21:24.390178+0000","last_active":"2024-09-21T12:24:19.872857+0000","last_peered":"2024-09-21T12:24:19.872857+0000","last_clean":"2024-09-21T12:24:19.872857+0000","last_became_active":"2024-09-21T12:20:14.288358+0000","last_became_peered":"2024-09-21T12:20:14.288358+0000","last_unstale":"2024-09-21T12:24:19.872857+0000","last_undegraded":"2024-09-21T12:24:19.872857+0000","last_fullsized":"2024-09-21T12:24:19.872857+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:21:24.390129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:21:24.390129+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:45.413201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019042449999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":197,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.882476+0000","last_change":"2024-09-21T12:23:29.274584+0000","last_active":"2024-09-21T12:24:19.882476+0000","last_peered":"2024-09-21T12:24:19.882476+0000","last_clean":"2024-09-21T12:24:19.882476+0000","last_became_active":"2024-09-21T12:20:19.309724+0000","last_became_peered":"2024-09-21T12:20:19.309724+0000","last_unstale":"2024-09-21T12:24:19.882476+0000","last_undegraded":"2024-09-21T12:24:19.882476+0000","last_fullsized":"2024-09-21T12:24:19.882476+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:29.274316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:29.274316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:31.465077+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":184,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.866281+0000","last_change":"2024-09-21T12:20:21.593267+0000","last_active":"2024-09-21T12:24:19.866281+0000","last_peered":"2024-09-21T12:24:19.866281+0000","last_clean":"2024-09-21T12:24:19.866281+0000","last_became_active":"2024-09-21T12:20:21.592982+0000","last_became_peered":"2024-09-21T12:20:21.592982+0000","last_unstale":"2024-09-21T12:24:19.866281+0000","last_undegraded":"2024-09-21T12:24:19.866281+0000","last_fullsized":"2024-09-21T12:24:19.866281+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:14.059177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"76'79","reported_seq":128,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.883091+0000","last_change":"2024-09-21T12:23:57.442430+0000","last_active":"2024-09-21T12:24:19.883091+0000","last_peered":"2024-09-21T12:24:19.883091+0000","last_clean":"2024-09-21T12:24:19.883091+0000","last_became_active":"2024-09-21T12:22:50.912088+0000","last_became_peered":"2024-09-21T12:22:50.912088+0000","last_unstale":"2024-09-21T12:24:19.883091+0000","last_undegraded":"2024-09-21T12:24:19.883091+0000","last_fullsized":"2024-09-21T12:24:19.883091+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:23:57.442284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:23:57.442284+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:17.628364+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"151'11714","reported_seq":11860,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:21.544650+0000","last_change":"2024-09-21T12:21:24.889134+0000","last_active":"2024-09-21T12:24:21.544650+0000","last_peered":"2024-09-21T12:24:21.544650+0000","last_clean":"2024-09-21T12:24:21.544650+0000","last_became_active":"2024-09-21T12:21:24.888897+0000","last_became_peered":"2024-09-21T12:21:24.888897+0000","last_unstale":"2024-09-21T12:24:21.544650+0000","last_undegraded":"2024-09-21T12:24:21.544650+0000","last_fullsized":"2024-09-21T12:24:21.544650+0000","mapping_epoch":84,"log_start":"130'7248","ondisk_log_start":"130'7248","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4466,"log_dups_size":0,"ondisk_log_size":4466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55967744,"num_objects":854,"num_object_clones":0,"num_object_copies":2562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11714,"num_write_kb":402176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":219,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.874079+0000","last_change":"2024-09-21T12:22:49.933621+0000","last_active":"2024-09-21T12:24:19.874079+0000","last_peered":"2024-09-21T12:24:19.874079+0000","last_clean":"2024-09-21T12:24:19.874079+0000","last_became_active":"2024-09-21T12:20:17.299325+0000","last_became_peered":"2024-09-21T12:20:17.299325+0000","last_unstale":"2024-09-21T12:24:19.874079+0000","last_undegraded":"2024-09-21T12:24:19.874079+0000","last_fullsized":"2024-09-21T12:24:19.874079+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:22:49.933382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:49.933382+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":226,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.868308+0000","last_change":"2024-09-21T12:22:50.929055+0000","last_active":"2024-09-21T12:24:19.868308+0000","last_peered":"2024-09-21T12:24:19.868308+0000","last_clean":"2024-09-21T12:24:19.868308+0000","last_became_active":"2024-09-21T12:22:50.923796+0000","last_became_peered":"2024-09-21T12:22:50.923796+0000","last_unstale":"2024-09-21T12:24:19.868308+0000","last_undegraded":"2024-09-21T12:24:19.868308+0000","last_fullsized":"2024-09-21T12:24:19.868308+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:13.623395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:13.623395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:37.094556+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":457,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:20.310201+0000","last_change":"2024-09-21T12:21:22.834972+0000","last_active":"2024-09-21T12:24:20.310201+0000","last_peered":"2024-09-21T12:24:20.310201+0000","last_clean":"2024-09-21T12:24:20.310201+0000","last_became_active":"2024-09-21T12:20:14.287543+0000","last_became_peered":"2024-09-21T12:20:14.287543+0000","last_unstale":"2024-09-21T12:24:20.310201+0000","last_undegraded":"2024-09-21T12:24:20.310201+0000","last_fullsized":"2024-09-21T12:24:20.310201+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-21T12:21:22.834917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:21:22.834917+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:21.087210+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":241,"num_read_kb":236,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":196,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.884664+0000","last_change":"2024-09-21T12:22:36.135464+0000","last_active":"2024-09-21T12:24:19.884664+0000","last_peered":"2024-09-21T12:24:19.884664+0000","last_clean":"2024-09-21T12:24:19.884664+0000","last_became_active":"2024-09-21T12:20:19.314217+0000","last_became_peered":"2024-09-21T12:20:19.314217+0000","last_unstale":"2024-09-21T12:24:19.884664+0000","last_undegraded":"2024-09-21T12:24:19.884664+0000","last_fullsized":"2024-09-21T12:24:19.884664+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:36.135390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:36.135390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:07.332958+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":193,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.880707+0000","last_change":"2024-09-21T12:22:57.265262+0000","last_active":"2024-09-21T12:24:19.880707+0000","last_peered":"2024-09-21T12:24:19.880707+0000","last_clean":"2024-09-21T12:24:19.880707+0000","last_became_active":"2024-09-21T12:20:21.311188+0000","last_became_peered":"2024-09-21T12:20:21.311188+0000","last_unstale":"2024-09-21T12:24:19.880707+0000","last_undegraded":"2024-09-21T12:24:19.880707+0000","last_fullsized":"2024-09-21T12:24:19.880707+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:57.265191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:57.265191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:24.950692+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"151'11988","reported_seq":12194,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:21.960080+0000","last_change":"2024-09-21T12:23:19.231036+0000","last_active":"2024-09-21T12:24:21.960080+0000","last_peered":"2024-09-21T12:24:21.960080+0000","last_clean":"2024-09-21T12:24:21.960080+0000","last_became_active":"2024-09-21T12:21:24.885986+0000","last_became_peered":"2024-09-21T12:21:24.885986+0000","last_unstale":"2024-09-21T12:24:21.960080+0000","last_undegraded":"2024-09-21T12:24:21.960080+0000","last_fullsized":"2024-09-21T12:24:21.960080+0000","mapping_epoch":84,"log_start":"132'7678","ondisk_log_start":"132'7678","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'7605","last_scrub_stamp":"2024-09-21T12:23:19.230513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:23:19.230513+0000","objects_scrubbed":5276,"log_size":4310,"log_dups_size":0,"ondisk_log_size":4310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:27.706332+0000","scrub_duration":2077,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54919168,"num_objects":838,"num_object_clones":0,"num_object_copies":2514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11988,"num_write_kb":410432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"121'19","reported_seq":347,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.869176+0000","last_change":"2024-09-21T12:22:53.756501+0000","last_active":"2024-09-21T12:24:19.869176+0000","last_peered":"2024-09-21T12:24:19.869176+0000","last_clean":"2024-09-21T12:24:19.869176+0000","last_became_active":"2024-09-21T12:22:50.921690+0000","last_became_peered":"2024-09-21T12:22:50.921690+0000","last_unstale":"2024-09-21T12:24:19.869176+0000","last_undegraded":"2024-09-21T12:24:19.869176+0000","last_fullsized":"2024-09-21T12:24:19.869176+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'19","last_scrub_stamp":"2024-09-21T12:22:53.756438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:53.756438+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":209,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.874563+0000","last_change":"2024-09-21T12:21:17.107378+0000","last_active":"2024-09-21T12:24:19.874563+0000","last_peered":"2024-09-21T12:24:19.874563+0000","last_clean":"2024-09-21T12:24:19.874563+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:24:19.874563+0000","last_undegraded":"2024-09-21T12:24:19.874563+0000","last_fullsized":"2024-09-21T12:24:19.874563+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:21:17.107309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:17.107309+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:09.385398+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":261,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.865080+0000","last_change":"2024-09-21T12:22:16.936605+0000","last_active":"2024-09-21T12:24:19.865080+0000","last_peered":"2024-09-21T12:24:19.865080+0000","last_clean":"2024-09-21T12:24:19.865080+0000","last_became_active":"2024-09-21T12:22:15.049592+0000","last_became_peered":"2024-09-21T12:22:15.049592+0000","last_unstale":"2024-09-21T12:24:19.865080+0000","last_undegraded":"2024-09-21T12:24:19.865080+0000","last_fullsized":"2024-09-21T12:24:19.865080+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:22:16.936548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:22:16.936548+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.866057+0000","last_change":"2024-09-21T12:20:19.310280+0000","last_active":"2024-09-21T12:24:19.866057+0000","last_peered":"2024-09-21T12:24:19.866057+0000","last_clean":"2024-09-21T12:24:19.866057+0000","last_became_active":"2024-09-21T12:20:19.309859+0000","last_became_peered":"2024-09-21T12:20:19.309859+0000","last_unstale":"2024-09-21T12:24:19.866057+0000","last_undegraded":"2024-09-21T12:24:19.866057+0000","last_fullsized":"2024-09-21T12:24:19.866057+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:15.053195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":221,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.884481+0000","last_change":"2024-09-21T12:23:56.364599+0000","last_active":"2024-09-21T12:24:19.884481+0000","last_peered":"2024-09-21T12:24:19.884481+0000","last_clean":"2024-09-21T12:24:19.884481+0000","last_became_active":"2024-09-21T12:22:50.913493+0000","last_became_peered":"2024-09-21T12:22:50.913493+0000","last_unstale":"2024-09-21T12:24:19.884481+0000","last_undegraded":"2024-09-21T12:24:19.884481+0000","last_fullsized":"2024-09-21T12:24:19.884481+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:56.364537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:56.364537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:25.702501+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"151'12198","reported_seq":12420,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:21.599962+0000","last_change":"2024-09-21T12:24:04.221374+0000","last_active":"2024-09-21T12:24:21.599962+0000","last_peered":"2024-09-21T12:24:21.599962+0000","last_clean":"2024-09-21T12:24:21.599962+0000","last_became_active":"2024-09-21T12:21:24.887845+0000","last_became_peered":"2024-09-21T12:21:24.887845+0000","last_unstale":"2024-09-21T12:24:21.599962+0000","last_undegraded":"2024-09-21T12:24:21.599962+0000","last_fullsized":"2024-09-21T12:24:21.599962+0000","mapping_epoch":84,"log_start":"130'7602","ondisk_log_start":"130'7602","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'5424","last_scrub_stamp":"2024-09-21T12:22:40.358604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:22:40.358604+0000","objects_scrubbed":899,"log_size":4596,"log_dups_size":0,"ondisk_log_size":4596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:34.191565+0000","scrub_duration":1743,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59047960,"num_objects":902,"num_object_clones":0,"num_object_copies":2706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":902,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12198,"num_write_kb":419137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":211,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.868852+0000","last_change":"2024-09-21T12:23:51.061783+0000","last_active":"2024-09-21T12:24:19.868852+0000","last_peered":"2024-09-21T12:24:19.868852+0000","last_clean":"2024-09-21T12:24:19.868852+0000","last_became_active":"2024-09-21T12:23:46.885802+0000","last_became_peered":"2024-09-21T12:23:46.885802+0000","last_unstale":"2024-09-21T12:24:19.868852+0000","last_undegraded":"2024-09-21T12:24:19.868852+0000","last_fullsized":"2024-09-21T12:24:19.868852+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:23:51.061706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:51.061706+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:09.333060+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.872711+0000","last_change":"2024-09-21T12:22:50.908380+0000","last_active":"2024-09-21T12:24:19.872711+0000","last_peered":"2024-09-21T12:24:19.872711+0000","last_clean":"2024-09-21T12:24:19.872711+0000","last_became_active":"2024-09-21T12:22:50.908060+0000","last_became_peered":"2024-09-21T12:22:50.908060+0000","last_unstale":"2024-09-21T12:24:19.872711+0000","last_undegraded":"2024-09-21T12:24:19.872711+0000","last_fullsized":"2024-09-21T12:24:19.872711+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:05.892153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:05.892153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:08.389121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":225,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.884398+0000","last_change":"2024-09-21T12:23:19.372762+0000","last_active":"2024-09-21T12:24:19.884398+0000","last_peered":"2024-09-21T12:24:19.884398+0000","last_clean":"2024-09-21T12:24:19.884398+0000","last_became_active":"2024-09-21T12:22:50.915081+0000","last_became_peered":"2024-09-21T12:22:50.915081+0000","last_unstale":"2024-09-21T12:24:19.884398+0000","last_undegraded":"2024-09-21T12:24:19.884398+0000","last_fullsized":"2024-09-21T12:24:19.884398+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:19.372719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:19.372719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:32.884588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":191,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.879167+0000","last_change":"2024-09-21T12:23:46.282792+0000","last_active":"2024-09-21T12:24:19.879167+0000","last_peered":"2024-09-21T12:24:19.879167+0000","last_clean":"2024-09-21T12:24:19.879167+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:24:19.879167+0000","last_undegraded":"2024-09-21T12:24:19.879167+0000","last_fullsized":"2024-09-21T12:24:19.879167+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:46.282661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:46.282661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:00.482338+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"76'79","reported_seq":112,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.883612+0000","last_change":"2024-09-21T12:23:22.321968+0000","last_active":"2024-09-21T12:24:19.883612+0000","last_peered":"2024-09-21T12:24:19.883612+0000","last_clean":"2024-09-21T12:24:19.883612+0000","last_became_active":"2024-09-21T12:22:17.087245+0000","last_became_peered":"2024-09-21T12:22:17.087245+0000","last_unstale":"2024-09-21T12:24:19.883612+0000","last_undegraded":"2024-09-21T12:24:19.883612+0000","last_fullsized":"2024-09-21T12:24:19.883612+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:23:22.321892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:23:22.321892+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:25.284854+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"151'12154","reported_seq":12300,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:21.697410+0000","last_change":"2024-09-21T12:21:24.888960+0000","last_active":"2024-09-21T12:24:21.697410+0000","last_peered":"2024-09-21T12:24:21.697410+0000","last_clean":"2024-09-21T12:24:21.697410+0000","last_became_active":"2024-09-21T12:21:24.888670+0000","last_became_peered":"2024-09-21T12:21:24.888670+0000","last_unstale":"2024-09-21T12:24:21.697410+0000","last_undegraded":"2024-09-21T12:24:21.697410+0000","last_fullsized":"2024-09-21T12:24:21.697410+0000","mapping_epoch":84,"log_start":"128'6984","ondisk_log_start":"128'6984","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":5170,"log_dups_size":0,"ondisk_log_size":5170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:43.402696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55312384,"num_objects":844,"num_object_clones":0,"num_object_copies":2532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12154,"num_write_kb":415936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":246,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.882034+0000","last_change":"2024-09-21T12:23:50.328004+0000","last_active":"2024-09-21T12:24:19.882034+0000","last_peered":"2024-09-21T12:24:19.882034+0000","last_clean":"2024-09-21T12:24:19.882034+0000","last_became_active":"2024-09-21T12:23:46.897197+0000","last_became_peered":"2024-09-21T12:23:46.897197+0000","last_unstale":"2024-09-21T12:24:19.882034+0000","last_undegraded":"2024-09-21T12:24:19.882034+0000","last_fullsized":"2024-09-21T12:24:19.882034+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-21T12:23:50.327918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:50.327918+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:08.093826+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":233,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.868502+0000","last_change":"2024-09-21T12:23:52.067111+0000","last_active":"2024-09-21T12:24:19.868502+0000","last_peered":"2024-09-21T12:24:19.868502+0000","last_clean":"2024-09-21T12:24:19.868502+0000","last_became_active":"2024-09-21T12:22:50.906886+0000","last_became_peered":"2024-09-21T12:22:50.906886+0000","last_unstale":"2024-09-21T12:24:19.868502+0000","last_undegraded":"2024-09-21T12:24:19.868502+0000","last_fullsized":"2024-09-21T12:24:19.868502+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:52.067065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:23:52.067065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:06.846290+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":218,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.880808+0000","last_change":"2024-09-21T12:23:17.434377+0000","last_active":"2024-09-21T12:24:19.880808+0000","last_peered":"2024-09-21T12:24:19.880808+0000","last_clean":"2024-09-21T12:24:19.880808+0000","last_became_active":"2024-09-21T12:22:50.927438+0000","last_became_peered":"2024-09-21T12:22:50.927438+0000","last_unstale":"2024-09-21T12:24:19.880808+0000","last_undegraded":"2024-09-21T12:24:19.880808+0000","last_fullsized":"2024-09-21T12:24:19.880808+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:17.434261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:17.434261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:43.736130+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":206,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.881929+0000","last_change":"2024-09-21T12:22:50.914246+0000","last_active":"2024-09-21T12:24:19.881929+0000","last_peered":"2024-09-21T12:24:19.881929+0000","last_clean":"2024-09-21T12:24:19.881929+0000","last_became_active":"2024-09-21T12:22:50.913898+0000","last_became_peered":"2024-09-21T12:22:50.913898+0000","last_unstale":"2024-09-21T12:24:19.881929+0000","last_undegraded":"2024-09-21T12:24:19.881929+0000","last_fullsized":"2024-09-21T12:24:19.881929+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:35.313521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"76'79","reported_seq":258,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.864764+0000","last_change":"2024-09-21T12:22:17.073301+0000","last_active":"2024-09-21T12:24:19.864764+0000","last_peered":"2024-09-21T12:24:19.864764+0000","last_clean":"2024-09-21T12:24:19.864764+0000","last_became_active":"2024-09-21T12:22:17.072872+0000","last_became_peered":"2024-09-21T12:22:17.072872+0000","last_unstale":"2024-09-21T12:24:19.864764+0000","last_undegraded":"2024-09-21T12:24:19.864764+0000","last_fullsized":"2024-09-21T12:24:19.864764+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T12:20:24.850157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:20:24.850157+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:24:33.931607+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":21,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.f","version":"151'12345","reported_seq":12491,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:21.964219+0000","last_change":"2024-09-21T12:21:24.887418+0000","last_active":"2024-09-21T12:24:21.964219+0000","last_peered":"2024-09-21T12:24:21.964219+0000","last_clean":"2024-09-21T12:24:21.964219+0000","last_became_active":"2024-09-21T12:21:24.887000+0000","last_became_peered":"2024-09-21T12:21:24.887000+0000","last_unstale":"2024-09-21T12:24:21.964219+0000","last_undegraded":"2024-09-21T12:24:21.964219+0000","last_fullsized":"2024-09-21T12:24:21.964219+0000","mapping_epoch":84,"log_start":"133'8051","ondisk_log_start":"133'8051","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4294,"log_dups_size":0,"ondisk_log_size":4294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:12.027728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53673984,"num_objects":819,"num_object_clones":0,"num_object_copies":2457,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12345,"num_write_kb":421248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":52,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.881301+0000","last_change":"2024-09-21T12:23:49.348056+0000","last_active":"2024-09-21T12:24:19.881301+0000","last_peered":"2024-09-21T12:24:19.881301+0000","last_clean":"2024-09-21T12:24:19.881301+0000","last_became_active":"2024-09-21T12:23:46.902260+0000","last_became_peered":"2024-09-21T12:23:46.902260+0000","last_unstale":"2024-09-21T12:24:19.881301+0000","last_undegraded":"2024-09-21T12:24:19.881301+0000","last_fullsized":"2024-09-21T12:24:19.881301+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:23:49.347964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:49.347964+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:02.497058+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":224,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.876670+0000","last_change":"2024-09-21T12:22:50.928240+0000","last_active":"2024-09-21T12:24:19.876670+0000","last_peered":"2024-09-21T12:24:19.876670+0000","last_clean":"2024-09-21T12:24:19.876670+0000","last_became_active":"2024-09-21T12:22:50.927946+0000","last_became_peered":"2024-09-21T12:22:50.927946+0000","last_unstale":"2024-09-21T12:24:19.876670+0000","last_undegraded":"2024-09-21T12:24:19.876670+0000","last_fullsized":"2024-09-21T12:24:19.876670+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:08.999015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:08.999015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":255,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.881976+0000","last_change":"2024-09-21T12:23:22.274636+0000","last_active":"2024-09-21T12:24:19.881976+0000","last_peered":"2024-09-21T12:24:19.881976+0000","last_clean":"2024-09-21T12:24:19.881976+0000","last_became_active":"2024-09-21T12:20:19.307924+0000","last_became_peered":"2024-09-21T12:20:19.307924+0000","last_unstale":"2024-09-21T12:24:19.881976+0000","last_undegraded":"2024-09-21T12:24:19.881976+0000","last_fullsized":"2024-09-21T12:24:19.881976+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:23:22.274558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:22.274558+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:43.175576+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":137,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.880256+0000","last_change":"2024-09-21T12:23:56.471960+0000","last_active":"2024-09-21T12:24:19.880256+0000","last_peered":"2024-09-21T12:24:19.880256+0000","last_clean":"2024-09-21T12:24:19.880256+0000","last_became_active":"2024-09-21T12:22:50.905470+0000","last_became_peered":"2024-09-21T12:22:50.905470+0000","last_unstale":"2024-09-21T12:24:19.880256+0000","last_undegraded":"2024-09-21T12:24:19.880256+0000","last_fullsized":"2024-09-21T12:24:19.880256+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:56.471116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:56.471116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:13.266560+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"151'12388","reported_seq":19801,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:21.743056+0000","last_change":"2024-09-21T12:23:20.213767+0000","last_active":"2024-09-21T12:24:21.743056+0000","last_peered":"2024-09-21T12:24:21.743056+0000","last_clean":"2024-09-21T12:24:21.743056+0000","last_became_active":"2024-09-21T12:23:20.203417+0000","last_became_peered":"2024-09-21T12:23:20.203417+0000","last_unstale":"2024-09-21T12:24:21.743056+0000","last_undegraded":"2024-09-21T12:24:21.743056+0000","last_fullsized":"2024-09-21T12:24:21.743056+0000","mapping_epoch":131,"log_start":"128'7210","ondisk_log_start":"128'7210","created":84,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":5178,"log_dups_size":0,"ondisk_log_size":5178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:42.358912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56229888,"num_objects":858,"num_object_clones":0,"num_object_copies":2574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12388,"num_write_kb":423872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3621,"num_bytes_recovered":237305856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":276,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.869177+0000","last_change":"2024-09-21T12:23:15.165398+0000","last_active":"2024-09-21T12:24:19.869177+0000","last_peered":"2024-09-21T12:24:19.869177+0000","last_clean":"2024-09-21T12:24:19.869177+0000","last_became_active":"2024-09-21T12:22:50.922343+0000","last_became_peered":"2024-09-21T12:22:50.922343+0000","last_unstale":"2024-09-21T12:24:19.869177+0000","last_undegraded":"2024-09-21T12:24:19.869177+0000","last_fullsized":"2024-09-21T12:24:19.869177+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:23:15.158428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:15.158428+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:27.015422+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":212,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.884302+0000","last_change":"2024-09-21T12:23:49.455863+0000","last_active":"2024-09-21T12:24:19.884302+0000","last_peered":"2024-09-21T12:24:19.884302+0000","last_clean":"2024-09-21T12:24:19.884302+0000","last_became_active":"2024-09-21T12:20:15.302262+0000","last_became_peered":"2024-09-21T12:20:15.302262+0000","last_unstale":"2024-09-21T12:24:19.884302+0000","last_undegraded":"2024-09-21T12:24:19.884302+0000","last_fullsized":"2024-09-21T12:24:19.884302+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:49.455808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:23:49.455808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:13.983105+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":226,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.884304+0000","last_change":"2024-09-21T12:23:58.277901+0000","last_active":"2024-09-21T12:24:19.884304+0000","last_peered":"2024-09-21T12:24:19.884304+0000","last_clean":"2024-09-21T12:24:19.884304+0000","last_became_active":"2024-09-21T12:22:50.925974+0000","last_became_peered":"2024-09-21T12:22:50.925974+0000","last_unstale":"2024-09-21T12:24:19.884304+0000","last_undegraded":"2024-09-21T12:24:19.884304+0000","last_fullsized":"2024-09-21T12:24:19.884304+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:58.277829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:58.277829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:06.555793+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":193,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.868257+0000","last_change":"2024-09-21T12:23:28.121344+0000","last_active":"2024-09-21T12:24:19.868257+0000","last_peered":"2024-09-21T12:24:19.868257+0000","last_clean":"2024-09-21T12:24:19.868257+0000","last_became_active":"2024-09-21T12:20:21.311493+0000","last_became_peered":"2024-09-21T12:20:21.311493+0000","last_unstale":"2024-09-21T12:24:19.868257+0000","last_undegraded":"2024-09-21T12:24:19.868257+0000","last_fullsized":"2024-09-21T12:24:19.868257+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:28.121286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:28.121286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:54.388322+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"151'12365","reported_seq":21864,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:21.953201+0000","last_change":"2024-09-21T12:22:50.949717+0000","last_active":"2024-09-21T12:24:21.953201+0000","last_peered":"2024-09-21T12:24:21.953201+0000","last_clean":"2024-09-21T12:24:21.953201+0000","last_became_active":"2024-09-21T12:22:50.924585+0000","last_became_peered":"2024-09-21T12:22:50.924585+0000","last_unstale":"2024-09-21T12:24:21.953201+0000","last_undegraded":"2024-09-21T12:24:21.953201+0000","last_fullsized":"2024-09-21T12:24:21.953201+0000","mapping_epoch":120,"log_start":"133'8080","ondisk_log_start":"133'8080","created":84,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4285,"log_dups_size":0,"ondisk_log_size":4285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:13.043881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55377920,"num_objects":845,"num_object_clones":0,"num_object_copies":2535,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12365,"num_write_kb":422720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4664,"num_bytes_recovered":305659904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":239,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.868308+0000","last_change":"2024-09-21T12:24:02.960807+0000","last_active":"2024-09-21T12:24:19.868308+0000","last_peered":"2024-09-21T12:24:19.868308+0000","last_clean":"2024-09-21T12:24:19.868308+0000","last_became_active":"2024-09-21T12:20:17.692867+0000","last_became_peered":"2024-09-21T12:20:17.692867+0000","last_unstale":"2024-09-21T12:24:19.868308+0000","last_undegraded":"2024-09-21T12:24:19.868308+0000","last_fullsized":"2024-09-21T12:24:19.868308+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-21T12:24:02.960733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:02.960733+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:17.768958+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":204,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.866548+0000","last_change":"2024-09-21T12:23:59.046000+0000","last_active":"2024-09-21T12:24:19.866548+0000","last_peered":"2024-09-21T12:24:19.866548+0000","last_clean":"2024-09-21T12:24:19.866548+0000","last_became_active":"2024-09-21T12:20:15.299333+0000","last_became_peered":"2024-09-21T12:20:15.299333+0000","last_unstale":"2024-09-21T12:24:19.866548+0000","last_undegraded":"2024-09-21T12:24:19.866548+0000","last_fullsized":"2024-09-21T12:24:19.866548+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:59.045927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:23:59.045927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:26.188346+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":189,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.881555+0000","last_change":"2024-09-21T12:20:19.307316+0000","last_active":"2024-09-21T12:24:19.881555+0000","last_peered":"2024-09-21T12:24:19.881555+0000","last_clean":"2024-09-21T12:24:19.881555+0000","last_became_active":"2024-09-21T12:20:19.306952+0000","last_became_peered":"2024-09-21T12:20:19.306952+0000","last_unstale":"2024-09-21T12:24:19.881555+0000","last_undegraded":"2024-09-21T12:24:19.881555+0000","last_fullsized":"2024-09-21T12:24:19.881555+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:33.300129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":184,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.864905+0000","last_change":"2024-09-21T12:20:21.314891+0000","last_active":"2024-09-21T12:24:19.864905+0000","last_peered":"2024-09-21T12:24:19.864905+0000","last_clean":"2024-09-21T12:24:19.864905+0000","last_became_active":"2024-09-21T12:20:21.314545+0000","last_became_peered":"2024-09-21T12:20:21.314545+0000","last_unstale":"2024-09-21T12:24:19.864905+0000","last_undegraded":"2024-09-21T12:24:19.864905+0000","last_fullsized":"2024-09-21T12:24:19.864905+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:37.032668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"151'12191","reported_seq":12337,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:21.999967+0000","last_change":"2024-09-21T12:21:24.888714+0000","last_active":"2024-09-21T12:24:21.999967+0000","last_peered":"2024-09-21T12:24:21.999967+0000","last_clean":"2024-09-21T12:24:21.999967+0000","last_became_active":"2024-09-21T12:21:24.888445+0000","last_became_peered":"2024-09-21T12:21:24.888445+0000","last_unstale":"2024-09-21T12:24:21.999967+0000","last_undegraded":"2024-09-21T12:24:21.999967+0000","last_fullsized":"2024-09-21T12:24:21.999967+0000","mapping_epoch":84,"log_start":"133'8160","ondisk_log_start":"133'8160","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4031,"log_dups_size":0,"ondisk_log_size":4031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:20.598292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57344000,"num_objects":875,"num_object_clones":0,"num_object_copies":2625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12191,"num_write_kb":418112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":204,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.880267+0000","last_change":"2024-09-21T12:23:47.400086+0000","last_active":"2024-09-21T12:24:19.880267+0000","last_peered":"2024-09-21T12:24:19.880267+0000","last_clean":"2024-09-21T12:24:19.880267+0000","last_became_active":"2024-09-21T12:22:50.910973+0000","last_became_peered":"2024-09-21T12:22:50.910973+0000","last_unstale":"2024-09-21T12:24:19.880267+0000","last_undegraded":"2024-09-21T12:24:19.880267+0000","last_fullsized":"2024-09-21T12:24:19.880267+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:23:47.400000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:47.400000+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:51.328592+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":225,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.866835+0000","last_change":"2024-09-21T12:22:50.928139+0000","last_active":"2024-09-21T12:24:19.866835+0000","last_peered":"2024-09-21T12:24:19.866835+0000","last_clean":"2024-09-21T12:24:19.866835+0000","last_became_active":"2024-09-21T12:22:50.927859+0000","last_became_peered":"2024-09-21T12:22:50.927859+0000","last_unstale":"2024-09-21T12:24:19.866835+0000","last_undegraded":"2024-09-21T12:24:19.866835+0000","last_fullsized":"2024-09-21T12:24:19.866835+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:20.865749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.865749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":197,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.872530+0000","last_change":"2024-09-21T12:21:23.393320+0000","last_active":"2024-09-21T12:24:19.872530+0000","last_peered":"2024-09-21T12:24:19.872530+0000","last_clean":"2024-09-21T12:24:19.872530+0000","last_became_active":"2024-09-21T12:20:19.312572+0000","last_became_peered":"2024-09-21T12:20:19.312572+0000","last_unstale":"2024-09-21T12:24:19.872530+0000","last_undegraded":"2024-09-21T12:24:19.872530+0000","last_fullsized":"2024-09-21T12:24:19.872530+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.393270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.393270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:37.312138+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":191,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.876280+0000","last_change":"2024-09-21T12:22:36.006177+0000","last_active":"2024-09-21T12:24:19.876280+0000","last_peered":"2024-09-21T12:24:19.876280+0000","last_clean":"2024-09-21T12:24:19.876280+0000","last_became_active":"2024-09-21T12:20:21.317223+0000","last_became_peered":"2024-09-21T12:20:21.317223+0000","last_unstale":"2024-09-21T12:24:19.876280+0000","last_undegraded":"2024-09-21T12:24:19.876280+0000","last_fullsized":"2024-09-21T12:24:19.876280+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:36.006109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:36.006109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"151'11971","reported_seq":12117,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:21.677359+0000","last_change":"2024-09-21T12:21:24.888120+0000","last_active":"2024-09-21T12:24:21.677359+0000","last_peered":"2024-09-21T12:24:21.677359+0000","last_clean":"2024-09-21T12:24:21.677359+0000","last_became_active":"2024-09-21T12:21:24.887488+0000","last_became_peered":"2024-09-21T12:21:24.887488+0000","last_unstale":"2024-09-21T12:24:21.677359+0000","last_undegraded":"2024-09-21T12:24:21.677359+0000","last_fullsized":"2024-09-21T12:24:21.677359+0000","mapping_epoch":84,"log_start":"132'7798","ondisk_log_start":"132'7798","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4173,"log_dups_size":0,"ondisk_log_size":4173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:45.414846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59441152,"num_objects":907,"num_object_clones":0,"num_object_copies":2721,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11971,"num_write_kb":412096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":233,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.883896+0000","last_change":"2024-09-21T12:23:17.286227+0000","last_active":"2024-09-21T12:24:19.883896+0000","last_peered":"2024-09-21T12:24:19.883896+0000","last_clean":"2024-09-21T12:24:19.883896+0000","last_became_active":"2024-09-21T12:20:17.692870+0000","last_became_peered":"2024-09-21T12:20:17.692870+0000","last_unstale":"2024-09-21T12:24:19.883896+0000","last_undegraded":"2024-09-21T12:24:19.883896+0000","last_fullsized":"2024-09-21T12:24:19.883896+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:23:17.286166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:17.286166+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:45.336901+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":255,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.867145+0000","last_change":"2024-09-21T12:22:51.052311+0000","last_active":"2024-09-21T12:24:19.867145+0000","last_peered":"2024-09-21T12:24:19.867145+0000","last_clean":"2024-09-21T12:24:19.867145+0000","last_became_active":"2024-09-21T12:22:50.927141+0000","last_became_peered":"2024-09-21T12:22:50.927141+0000","last_unstale":"2024-09-21T12:24:19.867145+0000","last_undegraded":"2024-09-21T12:24:19.867145+0000","last_fullsized":"2024-09-21T12:24:19.867145+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-21T12:22:14.920932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.920932+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"121'11","reported_seq":279,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.864718+0000","last_change":"2024-09-21T12:22:50.915762+0000","last_active":"2024-09-21T12:24:19.864718+0000","last_peered":"2024-09-21T12:24:19.864718+0000","last_clean":"2024-09-21T12:24:19.864718+0000","last_became_active":"2024-09-21T12:22:50.908371+0000","last_became_peered":"2024-09-21T12:22:50.908371+0000","last_unstale":"2024-09-21T12:24:19.864718+0000","last_undegraded":"2024-09-21T12:24:19.864718+0000","last_fullsized":"2024-09-21T12:24:19.864718+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:45.000445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":192,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.883825+0000","last_change":"2024-09-21T12:22:45.223570+0000","last_active":"2024-09-21T12:24:19.883825+0000","last_peered":"2024-09-21T12:24:19.883825+0000","last_clean":"2024-09-21T12:24:19.883825+0000","last_became_active":"2024-09-21T12:20:21.592576+0000","last_became_peered":"2024-09-21T12:20:21.592576+0000","last_unstale":"2024-09-21T12:24:19.883825+0000","last_undegraded":"2024-09-21T12:24:19.883825+0000","last_fullsized":"2024-09-21T12:24:19.883825+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:45.223396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:45.223396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:15.441394+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":212,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.868190+0000","last_change":"2024-09-21T12:22:45.726349+0000","last_active":"2024-09-21T12:24:19.868190+0000","last_peered":"2024-09-21T12:24:19.868190+0000","last_clean":"2024-09-21T12:24:19.868190+0000","last_became_active":"2024-09-21T12:20:15.303480+0000","last_became_peered":"2024-09-21T12:20:15.303480+0000","last_unstale":"2024-09-21T12:24:19.868190+0000","last_undegraded":"2024-09-21T12:24:19.868190+0000","last_fullsized":"2024-09-21T12:24:19.868190+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:45.726268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:45.726268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":208,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.875661+0000","last_change":"2024-09-21T12:22:36.999995+0000","last_active":"2024-09-21T12:24:19.875661+0000","last_peered":"2024-09-21T12:24:19.875661+0000","last_clean":"2024-09-21T12:24:19.875661+0000","last_became_active":"2024-09-21T12:20:17.694628+0000","last_became_peered":"2024-09-21T12:20:17.694628+0000","last_unstale":"2024-09-21T12:24:19.875661+0000","last_undegraded":"2024-09-21T12:24:19.875661+0000","last_fullsized":"2024-09-21T12:24:19.875661+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-21T12:22:36.999705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:36.999705+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":204,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.885527+0000","last_change":"2024-09-21T12:23:15.291669+0000","last_active":"2024-09-21T12:24:19.885527+0000","last_peered":"2024-09-21T12:24:19.885527+0000","last_clean":"2024-09-21T12:24:19.885527+0000","last_became_active":"2024-09-21T12:20:19.313882+0000","last_became_peered":"2024-09-21T12:20:19.313882+0000","last_unstale":"2024-09-21T12:24:19.885527+0000","last_undegraded":"2024-09-21T12:24:19.885527+0000","last_fullsized":"2024-09-21T12:24:19.885527+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:15.291602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:15.291602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:36.660286+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":213,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.866273+0000","last_change":"2024-09-21T12:24:00.066522+0000","last_active":"2024-09-21T12:24:19.866273+0000","last_peered":"2024-09-21T12:24:19.866273+0000","last_clean":"2024-09-21T12:24:19.866273+0000","last_became_active":"2024-09-21T12:22:50.904404+0000","last_became_peered":"2024-09-21T12:22:50.904404+0000","last_unstale":"2024-09-21T12:24:19.866273+0000","last_undegraded":"2024-09-21T12:24:19.866273+0000","last_fullsized":"2024-09-21T12:24:19.866273+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:00.066451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:00.066451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:27.003864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":217,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.874157+0000","last_change":"2024-09-21T12:21:24.093710+0000","last_active":"2024-09-21T12:24:19.874157+0000","last_peered":"2024-09-21T12:24:19.874157+0000","last_clean":"2024-09-21T12:24:19.874157+0000","last_became_active":"2024-09-21T12:20:17.695898+0000","last_became_peered":"2024-09-21T12:20:17.695898+0000","last_unstale":"2024-09-21T12:24:19.874157+0000","last_undegraded":"2024-09-21T12:24:19.874157+0000","last_fullsized":"2024-09-21T12:24:19.874157+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:21:24.093656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:21:24.093656+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:15.049499+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":225,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.864950+0000","last_change":"2024-09-21T12:22:50.928059+0000","last_active":"2024-09-21T12:24:19.864950+0000","last_peered":"2024-09-21T12:24:19.864950+0000","last_clean":"2024-09-21T12:24:19.864950+0000","last_became_active":"2024-09-21T12:22:50.927649+0000","last_became_peered":"2024-09-21T12:22:50.927649+0000","last_unstale":"2024-09-21T12:24:19.864950+0000","last_undegraded":"2024-09-21T12:24:19.864950+0000","last_fullsized":"2024-09-21T12:24:19.864950+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:07.084008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:07.084008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.875451+0000","last_change":"2024-09-21T12:23:53.356954+0000","last_active":"2024-09-21T12:24:19.875451+0000","last_peered":"2024-09-21T12:24:19.875451+0000","last_clean":"2024-09-21T12:24:19.875451+0000","last_became_active":"2024-09-21T12:20:19.311947+0000","last_became_peered":"2024-09-21T12:20:19.311947+0000","last_unstale":"2024-09-21T12:24:19.875451+0000","last_undegraded":"2024-09-21T12:24:19.875451+0000","last_fullsized":"2024-09-21T12:24:19.875451+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:53.356896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:53.356896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:11.617911+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":222,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.881473+0000","last_change":"2024-09-21T12:23:51.313630+0000","last_active":"2024-09-21T12:24:19.881473+0000","last_peered":"2024-09-21T12:24:19.881473+0000","last_clean":"2024-09-21T12:24:19.881473+0000","last_became_active":"2024-09-21T12:22:50.924931+0000","last_became_peered":"2024-09-21T12:22:50.924931+0000","last_unstale":"2024-09-21T12:24:19.881473+0000","last_undegraded":"2024-09-21T12:24:19.881473+0000","last_fullsized":"2024-09-21T12:24:19.881473+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:51.313528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:51.313528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:55.518605+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":233,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.884553+0000","last_change":"2024-09-21T12:23:46.925752+0000","last_active":"2024-09-21T12:24:19.884553+0000","last_peered":"2024-09-21T12:24:19.884553+0000","last_clean":"2024-09-21T12:24:19.884553+0000","last_became_active":"2024-09-21T12:23:46.898523+0000","last_became_peered":"2024-09-21T12:23:46.898523+0000","last_unstale":"2024-09-21T12:24:19.884553+0000","last_undegraded":"2024-09-21T12:24:19.884553+0000","last_fullsized":"2024-09-21T12:24:19.884553+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:22:39.201667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:39.201667+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:12.377811+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":205,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.868863+0000","last_change":"2024-09-21T12:22:53.989409+0000","last_active":"2024-09-21T12:24:19.868863+0000","last_peered":"2024-09-21T12:24:19.868863+0000","last_clean":"2024-09-21T12:24:19.868863+0000","last_became_active":"2024-09-21T12:20:15.297519+0000","last_became_peered":"2024-09-21T12:20:15.297519+0000","last_unstale":"2024-09-21T12:24:19.868863+0000","last_undegraded":"2024-09-21T12:24:19.868863+0000","last_fullsized":"2024-09-21T12:24:19.868863+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:53.989318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:53.989318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":253,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.879393+0000","last_change":"2024-09-21T12:23:57.379033+0000","last_active":"2024-09-21T12:24:19.879393+0000","last_peered":"2024-09-21T12:24:19.879393+0000","last_clean":"2024-09-21T12:24:19.879393+0000","last_became_active":"2024-09-21T12:20:19.305364+0000","last_became_peered":"2024-09-21T12:20:19.305364+0000","last_unstale":"2024-09-21T12:24:19.879393+0000","last_undegraded":"2024-09-21T12:24:19.879393+0000","last_fullsized":"2024-09-21T12:24:19.879393+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:23:57.378959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:57.378959+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:04.919603+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":129,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.882052+0000","last_change":"2024-09-21T12:23:21.250547+0000","last_active":"2024-09-21T12:24:19.882052+0000","last_peered":"2024-09-21T12:24:19.882052+0000","last_clean":"2024-09-21T12:24:19.882052+0000","last_became_active":"2024-09-21T12:22:50.924770+0000","last_became_peered":"2024-09-21T12:22:50.924770+0000","last_unstale":"2024-09-21T12:24:19.882052+0000","last_undegraded":"2024-09-21T12:24:19.882052+0000","last_fullsized":"2024-09-21T12:24:19.882052+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:21.250464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:21.250464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:40.125523+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":214,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.879635+0000","last_change":"2024-09-21T12:22:40.026248+0000","last_active":"2024-09-21T12:24:19.879635+0000","last_peered":"2024-09-21T12:24:19.879635+0000","last_clean":"2024-09-21T12:24:19.879635+0000","last_became_active":"2024-09-21T12:20:17.693209+0000","last_became_peered":"2024-09-21T12:20:17.693209+0000","last_unstale":"2024-09-21T12:24:19.879635+0000","last_undegraded":"2024-09-21T12:24:19.879635+0000","last_fullsized":"2024-09-21T12:24:19.879635+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:22:40.026037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:40.026037+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":225,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.866041+0000","last_change":"2024-09-21T12:22:50.908560+0000","last_active":"2024-09-21T12:24:19.866041+0000","last_peered":"2024-09-21T12:24:19.866041+0000","last_clean":"2024-09-21T12:24:19.866041+0000","last_became_active":"2024-09-21T12:22:50.908238+0000","last_became_peered":"2024-09-21T12:22:50.908238+0000","last_unstale":"2024-09-21T12:24:19.866041+0000","last_undegraded":"2024-09-21T12:24:19.866041+0000","last_fullsized":"2024-09-21T12:24:19.866041+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:15.938093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:15.938093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":254,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.884471+0000","last_change":"2024-09-21T12:22:46.216242+0000","last_active":"2024-09-21T12:24:19.884471+0000","last_peered":"2024-09-21T12:24:19.884471+0000","last_clean":"2024-09-21T12:24:19.884471+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:24:19.884471+0000","last_undegraded":"2024-09-21T12:24:19.884471+0000","last_fullsized":"2024-09-21T12:24:19.884471+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:22:46.216153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:46.216153+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:16.423966+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":185,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.873713+0000","last_change":"2024-09-21T12:20:21.309936+0000","last_active":"2024-09-21T12:24:19.873713+0000","last_peered":"2024-09-21T12:24:19.873713+0000","last_clean":"2024-09-21T12:24:19.873713+0000","last_became_active":"2024-09-21T12:20:21.309385+0000","last_became_peered":"2024-09-21T12:20:21.309385+0000","last_unstale":"2024-09-21T12:24:19.873713+0000","last_undegraded":"2024-09-21T12:24:19.873713+0000","last_fullsized":"2024-09-21T12:24:19.873713+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:06.127285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":192,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.868708+0000","last_change":"2024-09-21T12:23:59.046758+0000","last_active":"2024-09-21T12:24:19.868708+0000","last_peered":"2024-09-21T12:24:19.868708+0000","last_clean":"2024-09-21T12:24:19.868708+0000","last_became_active":"2024-09-21T12:22:50.906473+0000","last_became_peered":"2024-09-21T12:22:50.906473+0000","last_unstale":"2024-09-21T12:24:19.868708+0000","last_undegraded":"2024-09-21T12:24:19.868708+0000","last_fullsized":"2024-09-21T12:24:19.868708+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:23:59.046704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:59.046704+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:11.682375+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":121,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.866153+0000","last_change":"2024-09-21T12:22:50.904173+0000","last_active":"2024-09-21T12:24:19.866153+0000","last_peered":"2024-09-21T12:24:19.866153+0000","last_clean":"2024-09-21T12:24:19.866153+0000","last_became_active":"2024-09-21T12:22:50.903705+0000","last_became_peered":"2024-09-21T12:22:50.903705+0000","last_unstale":"2024-09-21T12:24:19.866153+0000","last_undegraded":"2024-09-21T12:24:19.866153+0000","last_fullsized":"2024-09-21T12:24:19.866153+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:40.056927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":189,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.886466+0000","last_change":"2024-09-21T12:20:19.315714+0000","last_active":"2024-09-21T12:24:19.886466+0000","last_peered":"2024-09-21T12:24:19.886466+0000","last_clean":"2024-09-21T12:24:19.886466+0000","last_became_active":"2024-09-21T12:20:19.315418+0000","last_became_peered":"2024-09-21T12:20:19.315418+0000","last_unstale":"2024-09-21T12:24:19.886466+0000","last_undegraded":"2024-09-21T12:24:19.886466+0000","last_fullsized":"2024-09-21T12:24:19.886466+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:35.447262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":191,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.872189+0000","last_change":"2024-09-21T12:22:50.143669+0000","last_active":"2024-09-21T12:24:19.872189+0000","last_peered":"2024-09-21T12:24:19.872189+0000","last_clean":"2024-09-21T12:24:19.872189+0000","last_became_active":"2024-09-21T12:20:21.310053+0000","last_became_peered":"2024-09-21T12:20:21.310053+0000","last_unstale":"2024-09-21T12:24:19.872189+0000","last_undegraded":"2024-09-21T12:24:19.872189+0000","last_fullsized":"2024-09-21T12:24:19.872189+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:50.143628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:50.143628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":209,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.886692+0000","last_change":"2024-09-21T12:20:17.299367+0000","last_active":"2024-09-21T12:24:19.886692+0000","last_peered":"2024-09-21T12:24:19.886692+0000","last_clean":"2024-09-21T12:24:19.886692+0000","last_became_active":"2024-09-21T12:20:17.299015+0000","last_became_peered":"2024-09-21T12:20:17.299015+0000","last_unstale":"2024-09-21T12:24:19.886692+0000","last_undegraded":"2024-09-21T12:24:19.886692+0000","last_fullsized":"2024-09-21T12:24:19.886692+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:37.431706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":225,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.866430+0000","last_change":"2024-09-21T12:22:50.922328+0000","last_active":"2024-09-21T12:24:19.866430+0000","last_peered":"2024-09-21T12:24:19.866430+0000","last_clean":"2024-09-21T12:24:19.866430+0000","last_became_active":"2024-09-21T12:22:50.922019+0000","last_became_peered":"2024-09-21T12:22:50.922019+0000","last_unstale":"2024-09-21T12:24:19.866430+0000","last_undegraded":"2024-09-21T12:24:19.866430+0000","last_fullsized":"2024-09-21T12:24:19.866430+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:05.067280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:05.067280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.876382+0000","last_change":"2024-09-21T12:21:20.931077+0000","last_active":"2024-09-21T12:24:19.876382+0000","last_peered":"2024-09-21T12:24:19.876382+0000","last_clean":"2024-09-21T12:24:19.876382+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:24:19.876382+0000","last_undegraded":"2024-09-21T12:24:19.876382+0000","last_fullsized":"2024-09-21T12:24:19.876382+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:20.931025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.931025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:16.543441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":193,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.867912+0000","last_change":"2024-09-21T12:23:29.134829+0000","last_active":"2024-09-21T12:24:19.867912+0000","last_peered":"2024-09-21T12:24:19.867912+0000","last_clean":"2024-09-21T12:24:19.867912+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:24:19.867912+0000","last_undegraded":"2024-09-21T12:24:19.867912+0000","last_fullsized":"2024-09-21T12:24:19.867912+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:29.134767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:29.134767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:36.055630+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":218,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.872357+0000","last_change":"2024-09-21T12:22:35.026113+0000","last_active":"2024-09-21T12:24:19.872357+0000","last_peered":"2024-09-21T12:24:19.872357+0000","last_clean":"2024-09-21T12:24:19.872357+0000","last_became_active":"2024-09-21T12:20:17.694794+0000","last_became_peered":"2024-09-21T12:20:17.694794+0000","last_unstale":"2024-09-21T12:24:19.872357+0000","last_undegraded":"2024-09-21T12:24:19.872357+0000","last_fullsized":"2024-09-21T12:24:19.872357+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:22:35.026007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:35.026007+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":212,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.885584+0000","last_change":"2024-09-21T12:23:28.334764+0000","last_active":"2024-09-21T12:24:19.885584+0000","last_peered":"2024-09-21T12:24:19.885584+0000","last_clean":"2024-09-21T12:24:19.885584+0000","last_became_active":"2024-09-21T12:20:15.295968+0000","last_became_peered":"2024-09-21T12:20:15.295968+0000","last_unstale":"2024-09-21T12:24:19.885584+0000","last_undegraded":"2024-09-21T12:24:19.885584+0000","last_fullsized":"2024-09-21T12:24:19.885584+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:28.334702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:23:28.334702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:31.853399+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":217,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.864519+0000","last_change":"2024-09-21T12:23:57.018797+0000","last_active":"2024-09-21T12:24:19.864519+0000","last_peered":"2024-09-21T12:24:19.864519+0000","last_clean":"2024-09-21T12:24:19.864519+0000","last_became_active":"2024-09-21T12:22:50.921604+0000","last_became_peered":"2024-09-21T12:22:50.921604+0000","last_unstale":"2024-09-21T12:24:19.864519+0000","last_undegraded":"2024-09-21T12:24:19.864519+0000","last_fullsized":"2024-09-21T12:24:19.864519+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:57.018749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:57.018749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:03.841967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":183,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.876581+0000","last_change":"2024-09-21T12:20:21.595660+0000","last_active":"2024-09-21T12:24:19.876581+0000","last_peered":"2024-09-21T12:24:19.876581+0000","last_clean":"2024-09-21T12:24:19.876581+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:24:19.876581+0000","last_undegraded":"2024-09-21T12:24:19.876581+0000","last_fullsized":"2024-09-21T12:24:19.876581+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:32.224446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":205,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.875424+0000","last_change":"2024-09-21T12:23:51.343568+0000","last_active":"2024-09-21T12:24:19.875424+0000","last_peered":"2024-09-21T12:24:19.875424+0000","last_clean":"2024-09-21T12:24:19.875424+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:24:19.875424+0000","last_undegraded":"2024-09-21T12:24:19.875424+0000","last_fullsized":"2024-09-21T12:24:19.875424+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-21T12:23:51.343491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:51.343491+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:08.087951+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":211,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.875889+0000","last_change":"2024-09-21T12:22:14.635397+0000","last_active":"2024-09-21T12:24:19.875889+0000","last_peered":"2024-09-21T12:24:19.875889+0000","last_clean":"2024-09-21T12:24:19.875889+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:24:19.875889+0000","last_undegraded":"2024-09-21T12:24:19.875889+0000","last_fullsized":"2024-09-21T12:24:19.875889+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:22:14.635191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.635191+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:33.123389+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":217,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.868253+0000","last_change":"2024-09-21T12:22:50.931202+0000","last_active":"2024-09-21T12:24:19.868253+0000","last_peered":"2024-09-21T12:24:19.868253+0000","last_clean":"2024-09-21T12:24:19.868253+0000","last_became_active":"2024-09-21T12:22:50.930615+0000","last_became_peered":"2024-09-21T12:22:50.930615+0000","last_unstale":"2024-09-21T12:24:19.868253+0000","last_undegraded":"2024-09-21T12:24:19.868253+0000","last_fullsized":"2024-09-21T12:24:19.868253+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:09.649109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:09.649109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":205,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.865328+0000","last_change":"2024-09-21T12:22:50.917883+0000","last_active":"2024-09-21T12:24:19.865328+0000","last_peered":"2024-09-21T12:24:19.865328+0000","last_clean":"2024-09-21T12:24:19.865328+0000","last_became_active":"2024-09-21T12:22:50.917408+0000","last_became_peered":"2024-09-21T12:22:50.917408+0000","last_unstale":"2024-09-21T12:24:19.865328+0000","last_undegraded":"2024-09-21T12:24:19.865328+0000","last_fullsized":"2024-09-21T12:24:19.865328+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:05.158628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":206,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.883733+0000","last_change":"2024-09-21T12:22:50.914633+0000","last_active":"2024-09-21T12:24:19.883733+0000","last_peered":"2024-09-21T12:24:19.883733+0000","last_clean":"2024-09-21T12:24:19.883733+0000","last_became_active":"2024-09-21T12:22:50.914221+0000","last_became_peered":"2024-09-21T12:22:50.914221+0000","last_unstale":"2024-09-21T12:24:19.883733+0000","last_undegraded":"2024-09-21T12:24:19.883733+0000","last_fullsized":"2024-09-21T12:24:19.883733+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:10.300219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":203,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.875662+0000","last_change":"2024-09-21T12:22:44.080149+0000","last_active":"2024-09-21T12:24:19.875662+0000","last_peered":"2024-09-21T12:24:19.875662+0000","last_clean":"2024-09-21T12:24:19.875662+0000","last_became_active":"2024-09-21T12:20:15.297993+0000","last_became_peered":"2024-09-21T12:20:15.297993+0000","last_unstale":"2024-09-21T12:24:19.875662+0000","last_undegraded":"2024-09-21T12:24:19.875662+0000","last_fullsized":"2024-09-21T12:24:19.875662+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:44.080070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:44.080070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":211,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.869178+0000","last_change":"2024-09-21T12:20:17.299134+0000","last_active":"2024-09-21T12:24:19.869178+0000","last_peered":"2024-09-21T12:24:19.869178+0000","last_clean":"2024-09-21T12:24:19.869178+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:24:19.869178+0000","last_undegraded":"2024-09-21T12:24:19.869178+0000","last_fullsized":"2024-09-21T12:24:19.869178+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:36.062051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":197,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-21T12:24:19.873879+0000","last_change":"2024-09-21T12:22:55.065505+0000","last_active":"2024-09-21T12:24:19.873879+0000","last_peered":"2024-09-21T12:24:19.873879+0000","last_clean":"2024-09-21T12:24:19.873879+0000","last_became_active":"2024-09-21T12:20:19.308247+0000","last_became_peered":"2024-09-21T12:20:19.308247+0000","last_unstale":"2024-09-21T12:24:19.873879+0000","last_undegraded":"2024-09-21T12:24:19.873879+0000","last_fullsized":"2024-09-21T12:24:19.873879+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:55.065444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:55.065444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":901971992,"num_objects":13764,"num_object_clones":0,"num_object_copies":41292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13764,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":194524,"num_write_kb":6665153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15812,"num_bytes_recovered":1036255232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2850435072,"data_stored":2850422856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70941,"ondisk_log_size":70941,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":690,"num_read_kb":447,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":201,"num_bytes_recovered":3982,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":396,"ondisk_log_size":396,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":5023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":241,"num_read_kb":236,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":644038,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":21,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234037,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":472736,"kb_used_data":342988,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93272416,"statfs":{"total":95995035648,"available":95510953984,"internally_reserved":0,"allocated":351219712,"data_stored":350883067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":132839882},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462973,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":564640,"kb_used_data":416848,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93180512,"statfs":{"total":95995035648,"available":95416844288,"internally_reserved":0,"allocated":426852352,"data_stored":426511426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151321030},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724608,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306212,"kb_used_data":230608,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93438940,"statfs":{"total":95995035648,"available":95681474560,"internally_reserved":0,"allocated":236142592,"data_stored":235803285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953544,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149774,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":723708,"kb_used_data":557864,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":93021444,"statfs":{"total":95995035648,"available":95253958656,"internally_reserved":0,"allocated":571252736,"data_stored":570889262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":169802183},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":455888,"kb_used_data":344192,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93289264,"statfs":{"total":95995035648,"available":95528206336,"internally_reserved":0,"allocated":352452608,"data_stored":352128142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574938,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":549508,"kb_used_data":410036,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93195644,"statfs":{"total":95995035648,"available":95432339456,"internally_reserved":0,"allocated":419876864,"data_stored":419499602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":591656,"kb_used_data":465816,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93153496,"statfs":{"total":95995035648,"available":95389179904,"internally_reserved":0,"allocated":476995584,"data_stored":476647388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":442044416,"data_stored":442040344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":409145344,"data_stored":409141272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":336855040,"data_stored":336855040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":551813120,"data_stored":551813120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":137297920,"data_stored":137297920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":231866368,"data_stored":231866368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":405278720,"data_stored":405274648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":336134144,"data_stored":336134144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:24:28.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:24:28.791 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:28.846 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:28.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:28.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:28.948 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:28.973 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:28.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:28.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:29.021 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:28 smithi114 ceph-mon[30324]: pgmap v192: 158 pgs: 158 active+clean; 498 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.49k op/s 2024-09-21T12:24:29.021 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:28 smithi114 ceph-mon[30324]: 4.5 scrub starts 2024-09-21T12:24:29.021 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:28 smithi114 ceph-mon[30324]: 4.5 scrub ok 2024-09-21T12:24:29.021 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:28 smithi114 ceph-mon[30324]: 1.8 scrub starts 2024-09-21T12:24:29.021 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:28 smithi114 ceph-mon[30324]: 1.8 scrub ok 2024-09-21T12:24:29.021 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:28 smithi114 ceph-mon[40867]: pgmap v192: 158 pgs: 158 active+clean; 498 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.49k op/s 2024-09-21T12:24:29.021 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:28 smithi114 ceph-mon[40867]: 4.5 scrub starts 2024-09-21T12:24:29.021 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:28 smithi114 ceph-mon[40867]: 4.5 scrub ok 2024-09-21T12:24:29.021 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:28 smithi114 ceph-mon[40867]: 1.8 scrub starts 2024-09-21T12:24:29.022 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:28 smithi114 ceph-mon[40867]: 1.8 scrub ok 2024-09-21T12:24:29.075 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:29.098 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:29.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:29.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:29.200 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:29.226 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:29.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:29.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:29.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:28 smithi183 ceph-mon[33277]: pgmap v192: 158 pgs: 158 active+clean; 498 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.49k op/s 2024-09-21T12:24:29.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:28 smithi183 ceph-mon[33277]: 4.5 scrub starts 2024-09-21T12:24:29.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:28 smithi183 ceph-mon[33277]: 4.5 scrub ok 2024-09-21T12:24:29.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:28 smithi183 ceph-mon[33277]: 1.8 scrub starts 2024-09-21T12:24:29.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:28 smithi183 ceph-mon[33277]: 1.8 scrub ok 2024-09-21T12:24:29.300 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:24:29.328 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:29.383 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:29.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:29.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:29.452 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:29.485 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:29.508 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T12:24:29.510 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:29.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:29.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:29.612 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:29.638 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:29.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:29.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:29.740 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:29.767 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:29.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:29.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:29.823 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:24:29.868 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:29.904 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:29.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:29.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:29.944 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-21T12:24:30.006 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:30.031 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:30.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:30.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:30.132 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:30.168 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:30.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:30.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:30.226 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:29 smithi183 ceph-mon[33277]: 4.1b scrub starts 2024-09-21T12:24:30.226 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:29 smithi183 ceph-mon[33277]: 4.1b scrub ok 2024-09-21T12:24:30.226 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:29 smithi183 ceph-mon[33277]: 3.f scrub starts 2024-09-21T12:24:30.227 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:29 smithi183 ceph-mon[33277]: 3.f scrub ok 2024-09-21T12:24:30.227 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:29 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:24:30.227 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:29 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:24:30.227 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:29 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2721751611' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:24:30.269 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:30.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:29 smithi114 ceph-mon[30324]: 4.1b scrub starts 2024-09-21T12:24:30.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:29 smithi114 ceph-mon[30324]: 4.1b scrub ok 2024-09-21T12:24:30.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:29 smithi114 ceph-mon[30324]: 3.f scrub starts 2024-09-21T12:24:30.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:29 smithi114 ceph-mon[30324]: 3.f scrub ok 2024-09-21T12:24:30.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:29 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:24:30.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:29 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:24:30.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:29 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2721751611' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:24:30.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:29 smithi114 ceph-mon[40867]: 4.1b scrub starts 2024-09-21T12:24:30.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:29 smithi114 ceph-mon[40867]: 4.1b scrub ok 2024-09-21T12:24:30.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:29 smithi114 ceph-mon[40867]: 3.f scrub starts 2024-09-21T12:24:30.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:29 smithi114 ceph-mon[40867]: 3.f scrub ok 2024-09-21T12:24:30.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:29 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:24:30.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:29 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:24:30.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:29 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2721751611' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:24:30.296 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:30.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:30.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:30.352 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:24:30.352 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:24:30.398 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:30.421 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:30.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:30.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:30.486 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:24:30.489 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:24:30.523 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:30.549 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:30.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:30.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:30.651 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:30.678 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:30.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:30.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:30.779 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:30.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:30.820 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:30.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:30.920 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:30.946 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:30.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:30.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:31.047 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:31.082 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:31.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:31.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:31.184 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:31.188 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T12:24:31.193 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:30 smithi114 ceph-mon[40867]: 6.2 scrub starts 2024-09-21T12:24:31.193 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:30 smithi114 ceph-mon[40867]: 6.2 scrub ok 2024-09-21T12:24:31.193 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:30 smithi114 ceph-mon[40867]: pgmap v193: 158 pgs: 158 active+clean; 498 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.23k op/s 2024-09-21T12:24:31.193 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:30 smithi114 ceph-mon[40867]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:24:31.194 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:30 smithi114 ceph-mon[40867]: Cluster is now healthy 2024-09-21T12:24:31.194 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:30 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2721751611' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:24:31.194 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:30 smithi114 ceph-mon[40867]: osdmap e154: 8 total, 8 up, 7 in 2024-09-21T12:24:31.194 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:30 smithi114 ceph-mon[40867]: 3.1b scrub starts 2024-09-21T12:24:31.194 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:30 smithi114 ceph-mon[40867]: 3.1b scrub ok 2024-09-21T12:24:31.194 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:30 smithi114 ceph-mon[40867]: 3.15 scrub starts 2024-09-21T12:24:31.194 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:30 smithi114 ceph-mon[40867]: 3.15 scrub ok 2024-09-21T12:24:31.194 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:30 smithi114 ceph-mon[30324]: 6.2 scrub starts 2024-09-21T12:24:31.194 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:30 smithi114 ceph-mon[30324]: 6.2 scrub ok 2024-09-21T12:24:31.194 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:30 smithi114 ceph-mon[30324]: pgmap v193: 158 pgs: 158 active+clean; 498 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.23k op/s 2024-09-21T12:24:31.194 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:30 smithi114 ceph-mon[30324]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:24:31.194 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:30 smithi114 ceph-mon[30324]: Cluster is now healthy 2024-09-21T12:24:31.195 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:30 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2721751611' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:24:31.195 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:30 smithi114 ceph-mon[30324]: osdmap e154: 8 total, 8 up, 7 in 2024-09-21T12:24:31.195 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:30 smithi114 ceph-mon[30324]: 3.1b scrub starts 2024-09-21T12:24:31.195 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:30 smithi114 ceph-mon[30324]: 3.1b scrub ok 2024-09-21T12:24:31.195 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:30 smithi114 ceph-mon[30324]: 3.15 scrub starts 2024-09-21T12:24:31.195 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:30 smithi114 ceph-mon[30324]: 3.15 scrub ok 2024-09-21T12:24:31.209 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:31.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:31.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:31.233 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:30 smithi183 ceph-mon[33277]: 6.2 scrub starts 2024-09-21T12:24:31.233 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:30 smithi183 ceph-mon[33277]: 6.2 scrub ok 2024-09-21T12:24:31.233 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:30 smithi183 ceph-mon[33277]: pgmap v193: 158 pgs: 158 active+clean; 498 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.23k op/s 2024-09-21T12:24:31.233 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:30 smithi183 ceph-mon[33277]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:24:31.233 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:30 smithi183 ceph-mon[33277]: Cluster is now healthy 2024-09-21T12:24:31.233 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:30 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2721751611' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:24:31.233 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:30 smithi183 ceph-mon[33277]: osdmap e154: 8 total, 8 up, 7 in 2024-09-21T12:24:31.233 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:30 smithi183 ceph-mon[33277]: 3.1b scrub starts 2024-09-21T12:24:31.234 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:30 smithi183 ceph-mon[33277]: 3.1b scrub ok 2024-09-21T12:24:31.234 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:30 smithi183 ceph-mon[33277]: 3.15 scrub starts 2024-09-21T12:24:31.234 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:30 smithi183 ceph-mon[33277]: 3.15 scrub ok 2024-09-21T12:24:31.311 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:31.352 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:31.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:31.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:31.454 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:31.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:31.483 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:31.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:31.523 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:31.532 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:24:31.563 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-09-21T12:24:29.940728+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":521726937,"num_objects":8151,"num_object_clones":0,"num_object_copies":24453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8151,"num_whiteouts":0,"num_read":972,"num_read_kb":734,"num_write":200782,"num_write_kb":6665346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16035,"num_bytes_recovered":1036313436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72411,"ondisk_log_size":72411,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2502012,"kb_used_data":1625284,"kb_used_omap":10,"kb_used_meta":876213,"kb_avail":653714052,"statfs":{"total":671965249536,"available":669403189248,"internally_reserved":0,"allocated":1664290816,"data_stored":1662169835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":897242219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,6,6,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-380895232,"num_objects":-5812,"num_object_clones":0,"num_object_copies":-17436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5812,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5812,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.066569"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":210,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.920527+0000","last_change":"2024-09-21T12:22:16.705200+0000","last_active":"2024-09-21T12:24:23.920527+0000","last_peered":"2024-09-21T12:24:23.920527+0000","last_clean":"2024-09-21T12:24:23.920527+0000","last_became_active":"2024-09-21T12:20:15.309431+0000","last_became_peered":"2024-09-21T12:20:15.309431+0000","last_unstale":"2024-09-21T12:24:23.920527+0000","last_undegraded":"2024-09-21T12:24:23.920527+0000","last_fullsized":"2024-09-21T12:24:23.920527+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:16.705088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:16.705088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:38.137043+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":180,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.924087+0000","last_change":"2024-09-21T12:22:51.040744+0000","last_active":"2024-09-21T12:24:23.924087+0000","last_peered":"2024-09-21T12:24:23.924087+0000","last_clean":"2024-09-21T12:24:23.924087+0000","last_became_active":"2024-09-21T12:22:50.909699+0000","last_became_peered":"2024-09-21T12:22:50.909699+0000","last_unstale":"2024-09-21T12:24:23.924087+0000","last_undegraded":"2024-09-21T12:24:23.924087+0000","last_fullsized":"2024-09-21T12:24:23.924087+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:05.017454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":201,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.928705+0000","last_change":"2024-09-21T12:23:20.433685+0000","last_active":"2024-09-21T12:24:23.928705+0000","last_peered":"2024-09-21T12:24:23.928705+0000","last_clean":"2024-09-21T12:24:23.928705+0000","last_became_active":"2024-09-21T12:20:19.314908+0000","last_became_peered":"2024-09-21T12:20:19.314908+0000","last_unstale":"2024-09-21T12:24:23.928705+0000","last_undegraded":"2024-09-21T12:24:23.928705+0000","last_fullsized":"2024-09-21T12:24:23.928705+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:20.433534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:20.433534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:24.249151+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":133,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.932777+0000","last_change":"2024-09-21T12:22:50.914064+0000","last_active":"2024-09-21T12:24:23.932777+0000","last_peered":"2024-09-21T12:24:23.932777+0000","last_clean":"2024-09-21T12:24:23.932777+0000","last_became_active":"2024-09-21T12:22:50.913743+0000","last_became_peered":"2024-09-21T12:22:50.913743+0000","last_unstale":"2024-09-21T12:24:23.932777+0000","last_undegraded":"2024-09-21T12:24:23.932777+0000","last_fullsized":"2024-09-21T12:24:23.932777+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:58.189665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:21:58.189665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:06.335216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":187,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.978429+0000","last_change":"2024-09-21T12:20:21.595607+0000","last_active":"2024-09-21T12:24:23.978429+0000","last_peered":"2024-09-21T12:24:23.978429+0000","last_clean":"2024-09-21T12:24:23.978429+0000","last_became_active":"2024-09-21T12:20:21.595301+0000","last_became_peered":"2024-09-21T12:20:21.595301+0000","last_unstale":"2024-09-21T12:24:23.978429+0000","last_undegraded":"2024-09-21T12:24:23.978429+0000","last_fullsized":"2024-09-21T12:24:23.978429+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:34.199328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":208,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.978189+0000","last_change":"2024-09-21T12:22:42.026488+0000","last_active":"2024-09-21T12:24:23.978189+0000","last_peered":"2024-09-21T12:24:23.978189+0000","last_clean":"2024-09-21T12:24:23.978189+0000","last_became_active":"2024-09-21T12:20:15.308655+0000","last_became_peered":"2024-09-21T12:20:15.308655+0000","last_unstale":"2024-09-21T12:24:23.978189+0000","last_undegraded":"2024-09-21T12:24:23.978189+0000","last_fullsized":"2024-09-21T12:24:23.978189+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:42.026416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:42.026416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":228,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.979651+0000","last_change":"2024-09-21T12:22:45.081280+0000","last_active":"2024-09-21T12:24:23.979651+0000","last_peered":"2024-09-21T12:24:23.979651+0000","last_clean":"2024-09-21T12:24:23.979651+0000","last_became_active":"2024-09-21T12:20:17.305220+0000","last_became_peered":"2024-09-21T12:20:17.305220+0000","last_unstale":"2024-09-21T12:24:23.979651+0000","last_undegraded":"2024-09-21T12:24:23.979651+0000","last_fullsized":"2024-09-21T12:24:23.979651+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:22:45.081089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:45.081089+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":142,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.924190+0000","last_change":"2024-09-21T12:24:01.949534+0000","last_active":"2024-09-21T12:24:23.924190+0000","last_peered":"2024-09-21T12:24:23.924190+0000","last_clean":"2024-09-21T12:24:23.924190+0000","last_became_active":"2024-09-21T12:22:50.926240+0000","last_became_peered":"2024-09-21T12:22:50.926240+0000","last_unstale":"2024-09-21T12:24:23.924190+0000","last_undegraded":"2024-09-21T12:24:23.924190+0000","last_fullsized":"2024-09-21T12:24:23.924190+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:01.949475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:01.949475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:04.277027+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":229,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.932935+0000","last_change":"2024-09-21T12:22:50.912964+0000","last_active":"2024-09-21T12:24:23.932935+0000","last_peered":"2024-09-21T12:24:23.932935+0000","last_clean":"2024-09-21T12:24:23.932935+0000","last_became_active":"2024-09-21T12:22:50.912481+0000","last_became_peered":"2024-09-21T12:22:50.912481+0000","last_unstale":"2024-09-21T12:24:23.932935+0000","last_undegraded":"2024-09-21T12:24:23.932935+0000","last_fullsized":"2024-09-21T12:24:23.932935+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:43.207441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:43.207441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:14.462493+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":197,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:25.640253+0000","last_change":"2024-09-21T12:24:25.640253+0000","last_active":"2024-09-21T12:24:25.640253+0000","last_peered":"2024-09-21T12:24:25.640253+0000","last_clean":"2024-09-21T12:24:25.640253+0000","last_became_active":"2024-09-21T12:22:50.927804+0000","last_became_peered":"2024-09-21T12:22:50.927804+0000","last_unstale":"2024-09-21T12:24:25.640253+0000","last_undegraded":"2024-09-21T12:24:25.640253+0000","last_fullsized":"2024-09-21T12:24:25.640253+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:24:25.640169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:25.640169+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:45.636860+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":200,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.932882+0000","last_change":"2024-09-21T12:22:15.018306+0000","last_active":"2024-09-21T12:24:23.932882+0000","last_peered":"2024-09-21T12:24:23.932882+0000","last_clean":"2024-09-21T12:24:23.932882+0000","last_became_active":"2024-09-21T12:20:19.308184+0000","last_became_peered":"2024-09-21T12:20:19.308184+0000","last_unstale":"2024-09-21T12:24:23.932882+0000","last_undegraded":"2024-09-21T12:24:23.932882+0000","last_fullsized":"2024-09-21T12:24:23.932882+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:15.018264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:15.018264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:45.395051+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":197,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.918907+0000","last_change":"2024-09-21T12:24:04.091870+0000","last_active":"2024-09-21T12:24:23.918907+0000","last_peered":"2024-09-21T12:24:23.918907+0000","last_clean":"2024-09-21T12:24:23.918907+0000","last_became_active":"2024-09-21T12:20:21.309190+0000","last_became_peered":"2024-09-21T12:20:21.309190+0000","last_unstale":"2024-09-21T12:24:23.918907+0000","last_undegraded":"2024-09-21T12:24:23.918907+0000","last_fullsized":"2024-09-21T12:24:23.918907+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:04.091807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:04.091807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:21.906035+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":229,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.932424+0000","last_change":"2024-09-21T12:22:54.174099+0000","last_active":"2024-09-21T12:24:23.932424+0000","last_peered":"2024-09-21T12:24:23.932424+0000","last_clean":"2024-09-21T12:24:23.932424+0000","last_became_active":"2024-09-21T12:22:50.911737+0000","last_became_peered":"2024-09-21T12:22:50.911737+0000","last_unstale":"2024-09-21T12:24:23.932424+0000","last_undegraded":"2024-09-21T12:24:23.932424+0000","last_fullsized":"2024-09-21T12:24:23.932424+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:54.174034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:54.174034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:22.431422+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":175,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.978696+0000","last_change":"2024-09-21T12:22:51.038838+0000","last_active":"2024-09-21T12:24:23.978696+0000","last_peered":"2024-09-21T12:24:23.978696+0000","last_clean":"2024-09-21T12:24:23.978696+0000","last_became_active":"2024-09-21T12:22:50.918794+0000","last_became_peered":"2024-09-21T12:22:50.918794+0000","last_unstale":"2024-09-21T12:24:23.978696+0000","last_undegraded":"2024-09-21T12:24:23.978696+0000","last_fullsized":"2024-09-21T12:24:23.978696+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-21T12:22:10.011633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:10.011633+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":213,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.926144+0000","last_change":"2024-09-21T12:22:50.922087+0000","last_active":"2024-09-21T12:24:23.926144+0000","last_peered":"2024-09-21T12:24:23.926144+0000","last_clean":"2024-09-21T12:24:23.926144+0000","last_became_active":"2024-09-21T12:22:50.921780+0000","last_became_peered":"2024-09-21T12:22:50.921780+0000","last_unstale":"2024-09-21T12:24:23.926144+0000","last_undegraded":"2024-09-21T12:24:23.926144+0000","last_fullsized":"2024-09-21T12:24:23.926144+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:42.023401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":204,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.931981+0000","last_change":"2024-09-21T12:23:27.383183+0000","last_active":"2024-09-21T12:24:23.931981+0000","last_peered":"2024-09-21T12:24:23.931981+0000","last_clean":"2024-09-21T12:24:23.931981+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:24:23.931981+0000","last_undegraded":"2024-09-21T12:24:23.931981+0000","last_fullsized":"2024-09-21T12:24:23.931981+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:27.383128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:27.383128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:32.310862+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":141,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.924630+0000","last_change":"2024-09-21T12:23:56.060558+0000","last_active":"2024-09-21T12:24:23.924630+0000","last_peered":"2024-09-21T12:24:23.924630+0000","last_clean":"2024-09-21T12:24:23.924630+0000","last_became_active":"2024-09-21T12:22:50.914792+0000","last_became_peered":"2024-09-21T12:22:50.914792+0000","last_unstale":"2024-09-21T12:24:23.924630+0000","last_undegraded":"2024-09-21T12:24:23.924630+0000","last_fullsized":"2024-09-21T12:24:23.924630+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:56.060488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:56.060488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:20.735686+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":238,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.921603+0000","last_change":"2024-09-21T12:23:19.156525+0000","last_active":"2024-09-21T12:24:23.921603+0000","last_peered":"2024-09-21T12:24:23.921603+0000","last_clean":"2024-09-21T12:24:23.921603+0000","last_became_active":"2024-09-21T12:22:50.921576+0000","last_became_peered":"2024-09-21T12:22:50.921576+0000","last_unstale":"2024-09-21T12:24:23.921603+0000","last_undegraded":"2024-09-21T12:24:23.921603+0000","last_fullsized":"2024-09-21T12:24:23.921603+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:19.156456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:23:19.156456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:24.315189+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":230,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.925249+0000","last_change":"2024-09-21T12:23:26.320324+0000","last_active":"2024-09-21T12:24:23.925249+0000","last_peered":"2024-09-21T12:24:23.925249+0000","last_clean":"2024-09-21T12:24:23.925249+0000","last_became_active":"2024-09-21T12:20:17.307828+0000","last_became_peered":"2024-09-21T12:20:17.307828+0000","last_unstale":"2024-09-21T12:24:23.925249+0000","last_undegraded":"2024-09-21T12:24:23.925249+0000","last_fullsized":"2024-09-21T12:24:23.925249+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:23:26.320231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:26.320231+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:38.921474+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":230,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.928205+0000","last_change":"2024-09-21T12:23:48.435833+0000","last_active":"2024-09-21T12:24:23.928205+0000","last_peered":"2024-09-21T12:24:23.928205+0000","last_clean":"2024-09-21T12:24:23.928205+0000","last_became_active":"2024-09-21T12:22:50.928972+0000","last_became_peered":"2024-09-21T12:22:50.928972+0000","last_unstale":"2024-09-21T12:24:23.928205+0000","last_undegraded":"2024-09-21T12:24:23.928205+0000","last_fullsized":"2024-09-21T12:24:23.928205+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:48.435769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:48.435769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:05.008777+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":195,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.982754+0000","last_change":"2024-09-21T12:23:58.354628+0000","last_active":"2024-09-21T12:24:23.982754+0000","last_peered":"2024-09-21T12:24:23.982754+0000","last_clean":"2024-09-21T12:24:23.982754+0000","last_became_active":"2024-09-21T12:20:21.592796+0000","last_became_peered":"2024-09-21T12:20:21.592796+0000","last_unstale":"2024-09-21T12:24:23.982754+0000","last_undegraded":"2024-09-21T12:24:23.982754+0000","last_fullsized":"2024-09-21T12:24:23.982754+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:58.354570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:58.354570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:05.826983+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":141,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.928017+0000","last_change":"2024-09-21T12:23:53.441120+0000","last_active":"2024-09-21T12:24:23.928017+0000","last_peered":"2024-09-21T12:24:23.928017+0000","last_clean":"2024-09-21T12:24:23.928017+0000","last_became_active":"2024-09-21T12:22:50.925403+0000","last_became_peered":"2024-09-21T12:22:50.925403+0000","last_unstale":"2024-09-21T12:24:23.928017+0000","last_undegraded":"2024-09-21T12:24:23.928017+0000","last_fullsized":"2024-09-21T12:24:23.928017+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:53.440795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:23:53.440795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:54.533332+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":283,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.977722+0000","last_change":"2024-09-21T12:23:47.285620+0000","last_active":"2024-09-21T12:24:23.977722+0000","last_peered":"2024-09-21T12:24:23.977722+0000","last_clean":"2024-09-21T12:24:23.977722+0000","last_became_active":"2024-09-21T12:22:50.912144+0000","last_became_peered":"2024-09-21T12:22:50.912144+0000","last_unstale":"2024-09-21T12:24:23.977722+0000","last_undegraded":"2024-09-21T12:24:23.977722+0000","last_fullsized":"2024-09-21T12:24:23.977722+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:23:47.285537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:47.285537+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:11.927780+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":126,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.978121+0000","last_change":"2024-09-21T12:22:50.920178+0000","last_active":"2024-09-21T12:24:23.978121+0000","last_peered":"2024-09-21T12:24:23.978121+0000","last_clean":"2024-09-21T12:24:23.978121+0000","last_became_active":"2024-09-21T12:22:50.918326+0000","last_became_peered":"2024-09-21T12:22:50.918326+0000","last_unstale":"2024-09-21T12:24:23.978121+0000","last_undegraded":"2024-09-21T12:24:23.978121+0000","last_fullsized":"2024-09-21T12:24:23.978121+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:41.091819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":197,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.927014+0000","last_change":"2024-09-21T12:23:51.065780+0000","last_active":"2024-09-21T12:24:23.927014+0000","last_peered":"2024-09-21T12:24:23.927014+0000","last_clean":"2024-09-21T12:24:23.927014+0000","last_became_active":"2024-09-21T12:20:21.312976+0000","last_became_peered":"2024-09-21T12:20:21.312976+0000","last_unstale":"2024-09-21T12:24:23.927014+0000","last_undegraded":"2024-09-21T12:24:23.927014+0000","last_fullsized":"2024-09-21T12:24:23.927014+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:23:51.065733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:51.065733+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:51.472146+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":243,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.928857+0000","last_change":"2024-09-21T12:22:52.070589+0000","last_active":"2024-09-21T12:24:23.928857+0000","last_peered":"2024-09-21T12:24:23.928857+0000","last_clean":"2024-09-21T12:24:23.928857+0000","last_became_active":"2024-09-21T12:22:50.910933+0000","last_became_peered":"2024-09-21T12:22:50.910933+0000","last_unstale":"2024-09-21T12:24:23.928857+0000","last_undegraded":"2024-09-21T12:24:23.928857+0000","last_fullsized":"2024-09-21T12:24:23.928857+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:22:52.070513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:52.070513+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:12.242749+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":221,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.919590+0000","last_change":"2024-09-21T12:24:03.059782+0000","last_active":"2024-09-21T12:24:23.919590+0000","last_peered":"2024-09-21T12:24:23.919590+0000","last_clean":"2024-09-21T12:24:23.919590+0000","last_became_active":"2024-09-21T12:20:17.695977+0000","last_became_peered":"2024-09-21T12:20:17.695977+0000","last_unstale":"2024-09-21T12:24:23.919590+0000","last_undegraded":"2024-09-21T12:24:23.919590+0000","last_fullsized":"2024-09-21T12:24:23.919590+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:24:03.059549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:03.059549+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:08.313061+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"121'11","reported_seq":296,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:25.396530+0000","last_change":"2024-09-21T12:22:50.926408+0000","last_active":"2024-09-21T12:24:25.396530+0000","last_peered":"2024-09-21T12:24:25.396530+0000","last_clean":"2024-09-21T12:24:25.396530+0000","last_became_active":"2024-09-21T12:22:50.916683+0000","last_became_peered":"2024-09-21T12:22:50.916683+0000","last_unstale":"2024-09-21T12:24:25.396530+0000","last_undegraded":"2024-09-21T12:24:25.396530+0000","last_fullsized":"2024-09-21T12:24:25.396530+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'9","last_scrub_stamp":"2024-09-21T12:22:08.647997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:08.647997+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:43.003218+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"76'79","reported_seq":110,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.932217+0000","last_change":"2024-09-21T12:22:18.000752+0000","last_active":"2024-09-21T12:24:23.932217+0000","last_peered":"2024-09-21T12:24:23.932217+0000","last_clean":"2024-09-21T12:24:23.932217+0000","last_became_active":"2024-09-21T12:22:17.074721+0000","last_became_peered":"2024-09-21T12:22:17.074721+0000","last_unstale":"2024-09-21T12:24:23.932217+0000","last_undegraded":"2024-09-21T12:24:23.932217+0000","last_fullsized":"2024-09-21T12:24:23.932217+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:22:18.000705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:22:18.000705+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:39.443289+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"153'12417","reported_seq":12628,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:26.606356+0000","last_change":"2024-09-21T12:22:42.121876+0000","last_active":"2024-09-21T12:24:26.606356+0000","last_peered":"2024-09-21T12:24:26.606356+0000","last_clean":"2024-09-21T12:24:26.606356+0000","last_became_active":"2024-09-21T12:21:24.886130+0000","last_became_peered":"2024-09-21T12:21:24.886130+0000","last_unstale":"2024-09-21T12:24:26.606356+0000","last_undegraded":"2024-09-21T12:24:26.606356+0000","last_fullsized":"2024-09-21T12:24:26.606356+0000","mapping_epoch":84,"log_start":"132'7847","ondisk_log_start":"132'7847","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5485","last_scrub_stamp":"2024-09-21T12:22:42.121763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:22:42.121763+0000","objects_scrubbed":5450,"log_size":4570,"log_dups_size":0,"ondisk_log_size":4570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1486,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32309248,"num_objects":493,"num_object_clones":0,"num_object_copies":1479,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12417,"num_write_kb":413120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":60,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.925525+0000","last_change":"2024-09-21T12:23:47.449186+0000","last_active":"2024-09-21T12:24:23.925525+0000","last_peered":"2024-09-21T12:24:23.925525+0000","last_clean":"2024-09-21T12:24:23.925525+0000","last_became_active":"2024-09-21T12:23:46.910112+0000","last_became_peered":"2024-09-21T12:23:46.910112+0000","last_unstale":"2024-09-21T12:24:23.925525+0000","last_undegraded":"2024-09-21T12:24:23.925525+0000","last_fullsized":"2024-09-21T12:24:23.925525+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:23:47.449113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:47.449113+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:59.572023+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":207,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.978919+0000","last_change":"2024-09-21T12:22:14.036290+0000","last_active":"2024-09-21T12:24:23.978919+0000","last_peered":"2024-09-21T12:24:23.978919+0000","last_clean":"2024-09-21T12:24:23.978919+0000","last_became_active":"2024-09-21T12:20:15.295774+0000","last_became_peered":"2024-09-21T12:20:15.295774+0000","last_unstale":"2024-09-21T12:24:23.978919+0000","last_undegraded":"2024-09-21T12:24:23.978919+0000","last_fullsized":"2024-09-21T12:24:23.978919+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:14.036236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.036236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"130'12","reported_seq":86,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.926414+0000","last_change":"2024-09-21T12:24:01.051310+0000","last_active":"2024-09-21T12:24:23.926414+0000","last_peered":"2024-09-21T12:24:23.926414+0000","last_clean":"2024-09-21T12:24:23.926414+0000","last_became_active":"2024-09-21T12:23:11.827128+0000","last_became_peered":"2024-09-21T12:23:11.827128+0000","last_unstale":"2024-09-21T12:24:23.926414+0000","last_undegraded":"2024-09-21T12:24:23.926414+0000","last_fullsized":"2024-09-21T12:24:23.926414+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'12","last_scrub_stamp":"2024-09-21T12:24:01.051257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:01.051257+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:20.741040+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":197,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.932448+0000","last_change":"2024-09-21T12:22:12.145637+0000","last_active":"2024-09-21T12:24:23.932448+0000","last_peered":"2024-09-21T12:24:23.932448+0000","last_clean":"2024-09-21T12:24:23.932448+0000","last_became_active":"2024-09-21T12:20:21.310022+0000","last_became_peered":"2024-09-21T12:20:21.310022+0000","last_unstale":"2024-09-21T12:24:23.932448+0000","last_undegraded":"2024-09-21T12:24:23.932448+0000","last_fullsized":"2024-09-21T12:24:23.932448+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:22:12.145550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:12.145550+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:44.423363+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"76'79","reported_seq":99,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.925028+0000","last_change":"2024-09-21T12:22:50.916239+0000","last_active":"2024-09-21T12:24:23.925028+0000","last_peered":"2024-09-21T12:24:23.925028+0000","last_clean":"2024-09-21T12:24:23.925028+0000","last_became_active":"2024-09-21T12:22:50.914885+0000","last_became_peered":"2024-09-21T12:22:50.914885+0000","last_unstale":"2024-09-21T12:24:23.925028+0000","last_undegraded":"2024-09-21T12:24:23.925028+0000","last_fullsized":"2024-09-21T12:24:23.925028+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:22:17.824342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:22:17.824342+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:07.348734+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.3","version":"153'12357","reported_seq":15633,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:27.021084+0000","last_change":"2024-09-21T12:24:04.025445+0000","last_active":"2024-09-21T12:24:27.021084+0000","last_peered":"2024-09-21T12:24:27.021084+0000","last_clean":"2024-09-21T12:24:27.021084+0000","last_became_active":"2024-09-21T12:22:50.919145+0000","last_became_peered":"2024-09-21T12:22:50.919145+0000","last_unstale":"2024-09-21T12:24:27.021084+0000","last_undegraded":"2024-09-21T12:24:27.021084+0000","last_fullsized":"2024-09-21T12:24:27.021084+0000","mapping_epoch":120,"log_start":"134'8266","ondisk_log_start":"134'8266","created":84,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'10775","last_scrub_stamp":"2024-09-21T12:24:04.025071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:24:04.025071+0000","objects_scrubbed":2081,"log_size":4091,"log_dups_size":0,"ondisk_log_size":4091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:15.894620+0000","scrub_duration":745,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31129600,"num_objects":475,"num_object_clones":0,"num_object_copies":1425,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12357,"num_write_kb":410624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1536,"num_bytes_recovered":100663296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":236,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.978712+0000","last_change":"2024-09-21T12:23:49.326670+0000","last_active":"2024-09-21T12:24:23.978712+0000","last_peered":"2024-09-21T12:24:23.978712+0000","last_clean":"2024-09-21T12:24:23.978712+0000","last_became_active":"2024-09-21T12:20:17.693179+0000","last_became_peered":"2024-09-21T12:20:17.693179+0000","last_unstale":"2024-09-21T12:24:23.978712+0000","last_undegraded":"2024-09-21T12:24:23.978712+0000","last_fullsized":"2024-09-21T12:24:23.978712+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-21T12:23:49.326428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:49.326428+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:15.058038+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":207,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.983135+0000","last_change":"2024-09-21T12:21:18.056305+0000","last_active":"2024-09-21T12:24:23.983135+0000","last_peered":"2024-09-21T12:24:23.983135+0000","last_clean":"2024-09-21T12:24:23.983135+0000","last_became_active":"2024-09-21T12:20:15.297340+0000","last_became_peered":"2024-09-21T12:20:15.297340+0000","last_unstale":"2024-09-21T12:24:23.983135+0000","last_undegraded":"2024-09-21T12:24:23.983135+0000","last_fullsized":"2024-09-21T12:24:23.983135+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:18.056261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:18.056261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:04.326707+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":142,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:24.661109+0000","last_change":"2024-09-21T12:24:24.661109+0000","last_active":"2024-09-21T12:24:24.661109+0000","last_peered":"2024-09-21T12:24:24.661109+0000","last_clean":"2024-09-21T12:24:24.661109+0000","last_became_active":"2024-09-21T12:22:50.930858+0000","last_became_peered":"2024-09-21T12:22:50.930858+0000","last_unstale":"2024-09-21T12:24:24.661109+0000","last_undegraded":"2024-09-21T12:24:24.661109+0000","last_fullsized":"2024-09-21T12:24:24.661109+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:24.661038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:24.661038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:35.691262+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":133,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.929246+0000","last_change":"2024-09-21T12:22:50.927366+0000","last_active":"2024-09-21T12:24:23.929246+0000","last_peered":"2024-09-21T12:24:23.929246+0000","last_clean":"2024-09-21T12:24:23.929246+0000","last_became_active":"2024-09-21T12:22:50.918940+0000","last_became_peered":"2024-09-21T12:22:50.918940+0000","last_unstale":"2024-09-21T12:24:23.929246+0000","last_undegraded":"2024-09-21T12:24:23.929246+0000","last_fullsized":"2024-09-21T12:24:23.929246+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:13.050383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:13.050383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"76'79","reported_seq":115,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.977931+0000","last_change":"2024-09-21T12:23:23.217945+0000","last_active":"2024-09-21T12:24:23.977931+0000","last_peered":"2024-09-21T12:24:23.977931+0000","last_clean":"2024-09-21T12:24:23.977931+0000","last_became_active":"2024-09-21T12:22:17.069042+0000","last_became_peered":"2024-09-21T12:22:17.069042+0000","last_unstale":"2024-09-21T12:24:23.977931+0000","last_undegraded":"2024-09-21T12:24:23.977931+0000","last_fullsized":"2024-09-21T12:24:23.977931+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:23:23.217863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:23:23.217863+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:33.585014+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"153'12700","reported_seq":12850,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:26.972369+0000","last_change":"2024-09-21T12:21:24.889433+0000","last_active":"2024-09-21T12:24:26.972369+0000","last_peered":"2024-09-21T12:24:26.972369+0000","last_clean":"2024-09-21T12:24:26.972369+0000","last_became_active":"2024-09-21T12:21:24.889133+0000","last_became_peered":"2024-09-21T12:21:24.889133+0000","last_unstale":"2024-09-21T12:24:26.972369+0000","last_undegraded":"2024-09-21T12:24:26.972369+0000","last_fullsized":"2024-09-21T12:24:26.972369+0000","mapping_epoch":84,"log_start":"134'8613","ondisk_log_start":"134'8613","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4087,"log_dups_size":0,"ondisk_log_size":4087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:11.396154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32899072,"num_objects":502,"num_object_clones":0,"num_object_copies":1506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12700,"num_write_kb":422464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":206,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.932057+0000","last_change":"2024-09-21T12:22:50.978102+0000","last_active":"2024-09-21T12:24:23.932057+0000","last_peered":"2024-09-21T12:24:23.932057+0000","last_clean":"2024-09-21T12:24:23.932057+0000","last_became_active":"2024-09-21T12:22:50.926008+0000","last_became_peered":"2024-09-21T12:22:50.926008+0000","last_unstale":"2024-09-21T12:24:23.932057+0000","last_undegraded":"2024-09-21T12:24:23.932057+0000","last_fullsized":"2024-09-21T12:24:23.932057+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-21T12:22:14.062464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.062464+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:41.411251+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":243,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.919824+0000","last_change":"2024-09-21T12:22:51.003768+0000","last_active":"2024-09-21T12:24:23.919824+0000","last_peered":"2024-09-21T12:24:23.919824+0000","last_clean":"2024-09-21T12:24:23.919824+0000","last_became_active":"2024-09-21T12:22:50.923284+0000","last_became_peered":"2024-09-21T12:22:50.923284+0000","last_unstale":"2024-09-21T12:24:23.919824+0000","last_undegraded":"2024-09-21T12:24:23.919824+0000","last_fullsized":"2024-09-21T12:24:23.919824+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:22:07.582969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:07.582969+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:35.130245+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"121'11","reported_seq":292,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:25.401320+0000","last_change":"2024-09-21T12:23:27.244152+0000","last_active":"2024-09-21T12:24:25.401320+0000","last_peered":"2024-09-21T12:24:25.401320+0000","last_clean":"2024-09-21T12:24:25.401320+0000","last_became_active":"2024-09-21T12:22:50.930342+0000","last_became_peered":"2024-09-21T12:22:50.930342+0000","last_unstale":"2024-09-21T12:24:25.401320+0000","last_undegraded":"2024-09-21T12:24:25.401320+0000","last_fullsized":"2024-09-21T12:24:25.401320+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:23:27.243903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:27.243903+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:54.351518+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":196,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.926136+0000","last_change":"2024-09-21T12:21:21.835669+0000","last_active":"2024-09-21T12:24:23.926136+0000","last_peered":"2024-09-21T12:24:23.926136+0000","last_clean":"2024-09-21T12:24:23.926136+0000","last_became_active":"2024-09-21T12:20:21.593367+0000","last_became_peered":"2024-09-21T12:20:21.593367+0000","last_unstale":"2024-09-21T12:24:23.926136+0000","last_undegraded":"2024-09-21T12:24:23.926136+0000","last_fullsized":"2024-09-21T12:24:23.926136+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:21.835623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:21:21.835623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:17.066934+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"76'79","reported_seq":200,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.924721+0000","last_change":"2024-09-21T12:23:26.056027+0000","last_active":"2024-09-21T12:24:23.924721+0000","last_peered":"2024-09-21T12:24:23.924721+0000","last_clean":"2024-09-21T12:24:23.924721+0000","last_became_active":"2024-09-21T12:22:17.093298+0000","last_became_peered":"2024-09-21T12:22:17.093298+0000","last_unstale":"2024-09-21T12:24:23.924721+0000","last_undegraded":"2024-09-21T12:24:23.924721+0000","last_fullsized":"2024-09-21T12:24:23.924721+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:23:26.055979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:23:26.055979+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:34.028294+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"153'12870","reported_seq":22071,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:26.995854+0000","last_change":"2024-09-21T12:22:51.819626+0000","last_active":"2024-09-21T12:24:26.995854+0000","last_peered":"2024-09-21T12:24:26.995854+0000","last_clean":"2024-09-21T12:24:26.995854+0000","last_became_active":"2024-09-21T12:22:51.808221+0000","last_became_peered":"2024-09-21T12:22:51.808221+0000","last_unstale":"2024-09-21T12:24:26.995854+0000","last_undegraded":"2024-09-21T12:24:26.995854+0000","last_fullsized":"2024-09-21T12:24:26.995854+0000","mapping_epoch":121,"log_start":"135'8788","ondisk_log_start":"135'8788","created":84,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4082,"log_dups_size":0,"ondisk_log_size":4082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:19.598845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32636928,"num_objects":498,"num_object_clones":0,"num_object_copies":1494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12870,"num_write_kb":427776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4516,"num_bytes_recovered":295960576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":278,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.927312+0000","last_change":"2024-09-21T12:24:01.388987+0000","last_active":"2024-09-21T12:24:23.927312+0000","last_peered":"2024-09-21T12:24:23.927312+0000","last_clean":"2024-09-21T12:24:23.927312+0000","last_became_active":"2024-09-21T12:22:50.925840+0000","last_became_peered":"2024-09-21T12:22:50.925840+0000","last_unstale":"2024-09-21T12:24:23.927312+0000","last_undegraded":"2024-09-21T12:24:23.927312+0000","last_fullsized":"2024-09-21T12:24:23.927312+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:24:01.388905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:01.388905+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:06.868982+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":208,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.933058+0000","last_change":"2024-09-21T12:22:37.176483+0000","last_active":"2024-09-21T12:24:23.933058+0000","last_peered":"2024-09-21T12:24:23.933058+0000","last_clean":"2024-09-21T12:24:23.933058+0000","last_became_active":"2024-09-21T12:20:15.308518+0000","last_became_peered":"2024-09-21T12:20:15.308518+0000","last_unstale":"2024-09-21T12:24:23.933058+0000","last_undegraded":"2024-09-21T12:24:23.933058+0000","last_fullsized":"2024-09-21T12:24:23.933058+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:37.176413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:37.176413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:08.307375+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":263,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.921317+0000","last_change":"2024-09-21T12:21:23.139205+0000","last_active":"2024-09-21T12:24:23.921317+0000","last_peered":"2024-09-21T12:24:23.921317+0000","last_clean":"2024-09-21T12:24:23.921317+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:24:23.921317+0000","last_undegraded":"2024-09-21T12:24:23.921317+0000","last_fullsized":"2024-09-21T12:24:23.921317+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:21:23.139124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.139124+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:30.139833+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":198,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.919112+0000","last_change":"2024-09-21T12:22:51.987032+0000","last_active":"2024-09-21T12:24:23.919112+0000","last_peered":"2024-09-21T12:24:23.919112+0000","last_clean":"2024-09-21T12:24:23.919112+0000","last_became_active":"2024-09-21T12:20:21.315801+0000","last_became_peered":"2024-09-21T12:20:21.315801+0000","last_unstale":"2024-09-21T12:24:23.919112+0000","last_undegraded":"2024-09-21T12:24:23.919112+0000","last_fullsized":"2024-09-21T12:24:23.919112+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:51.986953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:51.986953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:19.008252+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"76'79","reported_seq":200,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.924921+0000","last_change":"2024-09-21T12:23:25.039495+0000","last_active":"2024-09-21T12:24:23.924921+0000","last_peered":"2024-09-21T12:24:23.924921+0000","last_clean":"2024-09-21T12:24:23.924921+0000","last_became_active":"2024-09-21T12:22:17.083629+0000","last_became_peered":"2024-09-21T12:22:17.083629+0000","last_unstale":"2024-09-21T12:24:23.924921+0000","last_undegraded":"2024-09-21T12:24:23.924921+0000","last_fullsized":"2024-09-21T12:24:23.924921+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:23:25.039448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:23:25.039448+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:48.971864+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"153'12295","reported_seq":12510,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:26.571748+0000","last_change":"2024-09-21T12:22:48.664422+0000","last_active":"2024-09-21T12:24:26.571748+0000","last_peered":"2024-09-21T12:24:26.571748+0000","last_clean":"2024-09-21T12:24:26.571748+0000","last_became_active":"2024-09-21T12:21:24.884183+0000","last_became_peered":"2024-09-21T12:21:24.884183+0000","last_unstale":"2024-09-21T12:24:26.571748+0000","last_undegraded":"2024-09-21T12:24:26.571748+0000","last_fullsized":"2024-09-21T12:24:26.571748+0000","mapping_epoch":84,"log_start":"130'7505","ondisk_log_start":"130'7505","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'5758","last_scrub_stamp":"2024-09-21T12:22:48.664234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:22:48.664234+0000","objects_scrubbed":5726,"log_size":4790,"log_dups_size":0,"ondisk_log_size":4790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:17.405538+0000","scrub_duration":1463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33357824,"num_objects":509,"num_object_clones":0,"num_object_copies":1527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12295,"num_write_kb":409728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":242,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.921492+0000","last_change":"2024-09-21T12:23:22.124368+0000","last_active":"2024-09-21T12:24:23.921492+0000","last_peered":"2024-09-21T12:24:23.921492+0000","last_clean":"2024-09-21T12:24:23.921492+0000","last_became_active":"2024-09-21T12:20:17.693067+0000","last_became_peered":"2024-09-21T12:20:17.693067+0000","last_unstale":"2024-09-21T12:24:23.921492+0000","last_undegraded":"2024-09-21T12:24:23.921492+0000","last_fullsized":"2024-09-21T12:24:23.921492+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-21T12:23:22.124283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:22.124283+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:29.366927+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":201,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.928292+0000","last_change":"2024-09-21T12:20:15.308741+0000","last_active":"2024-09-21T12:24:23.928292+0000","last_peered":"2024-09-21T12:24:23.928292+0000","last_clean":"2024-09-21T12:24:23.928292+0000","last_became_active":"2024-09-21T12:20:15.308441+0000","last_became_peered":"2024-09-21T12:20:15.308441+0000","last_unstale":"2024-09-21T12:24:23.928292+0000","last_undegraded":"2024-09-21T12:24:23.928292+0000","last_fullsized":"2024-09-21T12:24:23.928292+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:06.290912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":201,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.924331+0000","last_change":"2024-09-21T12:22:34.563365+0000","last_active":"2024-09-21T12:24:23.924331+0000","last_peered":"2024-09-21T12:24:23.924331+0000","last_clean":"2024-09-21T12:24:23.924331+0000","last_became_active":"2024-09-21T12:20:19.309598+0000","last_became_peered":"2024-09-21T12:20:19.309598+0000","last_unstale":"2024-09-21T12:24:23.924331+0000","last_undegraded":"2024-09-21T12:24:23.924331+0000","last_fullsized":"2024-09-21T12:24:23.924331+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:34.563282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:34.563282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:06.982730+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":189,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.928628+0000","last_change":"2024-09-21T12:20:21.593173+0000","last_active":"2024-09-21T12:24:23.928628+0000","last_peered":"2024-09-21T12:24:23.928628+0000","last_clean":"2024-09-21T12:24:23.928628+0000","last_became_active":"2024-09-21T12:20:21.592828+0000","last_became_peered":"2024-09-21T12:20:21.592828+0000","last_unstale":"2024-09-21T12:24:23.928628+0000","last_undegraded":"2024-09-21T12:24:23.928628+0000","last_fullsized":"2024-09-21T12:24:23.928628+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:39.438547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"76'79","reported_seq":110,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.931736+0000","last_change":"2024-09-21T12:23:21.346117+0000","last_active":"2024-09-21T12:24:23.931736+0000","last_peered":"2024-09-21T12:24:23.931736+0000","last_clean":"2024-09-21T12:24:23.931736+0000","last_became_active":"2024-09-21T12:22:17.084732+0000","last_became_peered":"2024-09-21T12:22:17.084732+0000","last_unstale":"2024-09-21T12:24:23.931736+0000","last_undegraded":"2024-09-21T12:24:23.931736+0000","last_fullsized":"2024-09-21T12:24:23.931736+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:23:21.346046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:23:21.346046+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:35.936936+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"153'12723","reported_seq":12955,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:26.578056+0000","last_change":"2024-09-21T12:24:25.147579+0000","last_active":"2024-09-21T12:24:26.578056+0000","last_peered":"2024-09-21T12:24:26.578056+0000","last_clean":"2024-09-21T12:24:26.578056+0000","last_became_active":"2024-09-21T12:21:24.886854+0000","last_became_peered":"2024-09-21T12:21:24.886854+0000","last_unstale":"2024-09-21T12:24:26.578056+0000","last_undegraded":"2024-09-21T12:24:26.578056+0000","last_fullsized":"2024-09-21T12:24:26.578056+0000","mapping_epoch":84,"log_start":"132'8100","ondisk_log_start":"132'8100","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'12621","last_scrub_stamp":"2024-09-21T12:24:25.147410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:24:25.147410+0000","objects_scrubbed":622,"log_size":4623,"log_dups_size":0,"ondisk_log_size":4623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:30.467053+0000","scrub_duration":216,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33488896,"num_objects":511,"num_object_clones":0,"num_object_copies":1533,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12723,"num_write_kb":423488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":296,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.928174+0000","last_change":"2024-09-21T12:20:17.308049+0000","last_active":"2024-09-21T12:24:23.928174+0000","last_peered":"2024-09-21T12:24:23.928174+0000","last_clean":"2024-09-21T12:24:23.928174+0000","last_became_active":"2024-09-21T12:20:17.307769+0000","last_became_peered":"2024-09-21T12:20:17.307769+0000","last_unstale":"2024-09-21T12:24:23.928174+0000","last_undegraded":"2024-09-21T12:24:23.928174+0000","last_fullsized":"2024-09-21T12:24:23.928174+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:05.312793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":133,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.921598+0000","last_change":"2024-09-21T12:22:50.922961+0000","last_active":"2024-09-21T12:24:23.921598+0000","last_peered":"2024-09-21T12:24:23.921598+0000","last_clean":"2024-09-21T12:24:23.921598+0000","last_became_active":"2024-09-21T12:22:50.922002+0000","last_became_peered":"2024-09-21T12:22:50.922002+0000","last_unstale":"2024-09-21T12:24:23.921598+0000","last_undegraded":"2024-09-21T12:24:23.921598+0000","last_fullsized":"2024-09-21T12:24:23.921598+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:05.507899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:05.507899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:34.159504+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":219,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.932730+0000","last_change":"2024-09-21T12:21:20.045824+0000","last_active":"2024-09-21T12:24:23.932730+0000","last_peered":"2024-09-21T12:24:23.932730+0000","last_clean":"2024-09-21T12:24:23.932730+0000","last_became_active":"2024-09-21T12:20:14.287998+0000","last_became_peered":"2024-09-21T12:20:14.287998+0000","last_unstale":"2024-09-21T12:24:23.932730+0000","last_undegraded":"2024-09-21T12:24:23.932730+0000","last_fullsized":"2024-09-21T12:24:23.932730+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T12:21:20.045755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.045755+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:05.292532+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011011149999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":222,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.920317+0000","last_change":"2024-09-21T12:22:50.921798+0000","last_active":"2024-09-21T12:24:23.920317+0000","last_peered":"2024-09-21T12:24:23.920317+0000","last_clean":"2024-09-21T12:24:23.920317+0000","last_became_active":"2024-09-21T12:22:50.921488+0000","last_became_peered":"2024-09-21T12:22:50.921488+0000","last_unstale":"2024-09-21T12:24:23.920317+0000","last_undegraded":"2024-09-21T12:24:23.920317+0000","last_fullsized":"2024-09-21T12:24:23.920317+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:06.563104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:06.563104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:41.170121+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":216,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.978042+0000","last_change":"2024-09-21T12:23:20.237788+0000","last_active":"2024-09-21T12:24:23.978042+0000","last_peered":"2024-09-21T12:24:23.978042+0000","last_clean":"2024-09-21T12:24:23.978042+0000","last_became_active":"2024-09-21T12:22:50.917558+0000","last_became_peered":"2024-09-21T12:22:50.917558+0000","last_unstale":"2024-09-21T12:24:23.978042+0000","last_undegraded":"2024-09-21T12:24:23.978042+0000","last_fullsized":"2024-09-21T12:24:23.978042+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:20.237715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:20.237715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:40.331198+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'79","reported_seq":285,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.926031+0000","last_change":"2024-09-21T12:23:19.867918+0000","last_active":"2024-09-21T12:24:23.926031+0000","last_peered":"2024-09-21T12:24:23.926031+0000","last_clean":"2024-09-21T12:24:23.926031+0000","last_became_active":"2024-09-21T12:22:50.921709+0000","last_became_peered":"2024-09-21T12:22:50.921709+0000","last_unstale":"2024-09-21T12:24:23.926031+0000","last_undegraded":"2024-09-21T12:24:23.926031+0000","last_fullsized":"2024-09-21T12:24:23.926031+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:23:19.867868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:23:19.867868+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:37.496791+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"153'12370","reported_seq":15497,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:26.931154+0000","last_change":"2024-09-21T12:22:51.045889+0000","last_active":"2024-09-21T12:24:26.931154+0000","last_peered":"2024-09-21T12:24:26.931154+0000","last_clean":"2024-09-21T12:24:26.931154+0000","last_became_active":"2024-09-21T12:22:50.936833+0000","last_became_peered":"2024-09-21T12:22:50.936833+0000","last_unstale":"2024-09-21T12:24:26.931154+0000","last_undegraded":"2024-09-21T12:24:26.931154+0000","last_fullsized":"2024-09-21T12:24:26.931154+0000","mapping_epoch":120,"log_start":"133'8053","ondisk_log_start":"133'8053","created":84,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4317,"log_dups_size":0,"ondisk_log_size":4317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:07.096316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":512,"num_object_clones":0,"num_object_copies":1536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12370,"num_write_kb":412224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1475,"num_bytes_recovered":96665600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":224,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.928154+0000","last_change":"2024-09-21T12:20:17.297678+0000","last_active":"2024-09-21T12:24:23.928154+0000","last_peered":"2024-09-21T12:24:23.928154+0000","last_clean":"2024-09-21T12:24:23.928154+0000","last_became_active":"2024-09-21T12:20:17.297374+0000","last_became_peered":"2024-09-21T12:20:17.297374+0000","last_unstale":"2024-09-21T12:24:23.928154+0000","last_undegraded":"2024-09-21T12:24:23.928154+0000","last_fullsized":"2024-09-21T12:24:23.928154+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:31.489358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":218,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.927756+0000","last_change":"2024-09-21T12:23:59.367953+0000","last_active":"2024-09-21T12:24:23.927756+0000","last_peered":"2024-09-21T12:24:23.927756+0000","last_clean":"2024-09-21T12:24:23.927756+0000","last_became_active":"2024-09-21T12:20:15.308356+0000","last_became_peered":"2024-09-21T12:20:15.308356+0000","last_unstale":"2024-09-21T12:24:23.927756+0000","last_undegraded":"2024-09-21T12:24:23.927756+0000","last_fullsized":"2024-09-21T12:24:23.927756+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:59.367778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:23:59.367778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:12.420396+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":219,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.923483+0000","last_change":"2024-09-21T12:21:24.390178+0000","last_active":"2024-09-21T12:24:23.923483+0000","last_peered":"2024-09-21T12:24:23.923483+0000","last_clean":"2024-09-21T12:24:23.923483+0000","last_became_active":"2024-09-21T12:20:14.288358+0000","last_became_peered":"2024-09-21T12:20:14.288358+0000","last_unstale":"2024-09-21T12:24:23.923483+0000","last_undegraded":"2024-09-21T12:24:23.923483+0000","last_fullsized":"2024-09-21T12:24:23.923483+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:21:24.390129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:21:24.390129+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:45.413201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019042449999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":201,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.979971+0000","last_change":"2024-09-21T12:23:29.274584+0000","last_active":"2024-09-21T12:24:23.979971+0000","last_peered":"2024-09-21T12:24:23.979971+0000","last_clean":"2024-09-21T12:24:23.979971+0000","last_became_active":"2024-09-21T12:20:19.309724+0000","last_became_peered":"2024-09-21T12:20:19.309724+0000","last_unstale":"2024-09-21T12:24:23.979971+0000","last_undegraded":"2024-09-21T12:24:23.979971+0000","last_fullsized":"2024-09-21T12:24:23.979971+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:29.274316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:29.274316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:31.465077+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":188,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.926810+0000","last_change":"2024-09-21T12:20:21.593267+0000","last_active":"2024-09-21T12:24:23.926810+0000","last_peered":"2024-09-21T12:24:23.926810+0000","last_clean":"2024-09-21T12:24:23.926810+0000","last_became_active":"2024-09-21T12:20:21.592982+0000","last_became_peered":"2024-09-21T12:20:21.592982+0000","last_unstale":"2024-09-21T12:24:23.926810+0000","last_undegraded":"2024-09-21T12:24:23.926810+0000","last_fullsized":"2024-09-21T12:24:23.926810+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:14.059177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"76'79","reported_seq":132,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.926102+0000","last_change":"2024-09-21T12:23:57.442430+0000","last_active":"2024-09-21T12:24:23.926102+0000","last_peered":"2024-09-21T12:24:23.926102+0000","last_clean":"2024-09-21T12:24:23.926102+0000","last_became_active":"2024-09-21T12:22:50.912088+0000","last_became_peered":"2024-09-21T12:22:50.912088+0000","last_unstale":"2024-09-21T12:24:23.926102+0000","last_undegraded":"2024-09-21T12:24:23.926102+0000","last_fullsized":"2024-09-21T12:24:23.926102+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:23:57.442284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:23:57.442284+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:17.628364+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"153'12053","reported_seq":12218,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:26.538526+0000","last_change":"2024-09-21T12:24:24.304763+0000","last_active":"2024-09-21T12:24:26.538526+0000","last_peered":"2024-09-21T12:24:26.538526+0000","last_clean":"2024-09-21T12:24:26.538526+0000","last_became_active":"2024-09-21T12:21:24.888897+0000","last_became_peered":"2024-09-21T12:21:24.888897+0000","last_unstale":"2024-09-21T12:24:26.538526+0000","last_undegraded":"2024-09-21T12:24:26.538526+0000","last_fullsized":"2024-09-21T12:24:26.538526+0000","mapping_epoch":84,"log_start":"132'7548","ondisk_log_start":"132'7548","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'11902","last_scrub_stamp":"2024-09-21T12:24:24.304594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:24:24.304594+0000","objects_scrubbed":673,"log_size":4505,"log_dups_size":0,"ondisk_log_size":4505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:48.515170+0000","scrub_duration":194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33751040,"num_objects":515,"num_object_clones":0,"num_object_copies":1545,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12053,"num_write_kb":402176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":223,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.919666+0000","last_change":"2024-09-21T12:22:49.933621+0000","last_active":"2024-09-21T12:24:23.919666+0000","last_peered":"2024-09-21T12:24:23.919666+0000","last_clean":"2024-09-21T12:24:23.919666+0000","last_became_active":"2024-09-21T12:20:17.299325+0000","last_became_peered":"2024-09-21T12:20:17.299325+0000","last_unstale":"2024-09-21T12:24:23.919666+0000","last_undegraded":"2024-09-21T12:24:23.919666+0000","last_fullsized":"2024-09-21T12:24:23.919666+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:22:49.933382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:49.933382+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:21.986368+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":231,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.920920+0000","last_change":"2024-09-21T12:22:50.929055+0000","last_active":"2024-09-21T12:24:23.920920+0000","last_peered":"2024-09-21T12:24:23.920920+0000","last_clean":"2024-09-21T12:24:23.920920+0000","last_became_active":"2024-09-21T12:22:50.923796+0000","last_became_peered":"2024-09-21T12:22:50.923796+0000","last_unstale":"2024-09-21T12:24:23.920920+0000","last_undegraded":"2024-09-21T12:24:23.920920+0000","last_fullsized":"2024-09-21T12:24:23.920920+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:13.623395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:13.623395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:37.094556+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":466,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:25.315085+0000","last_change":"2024-09-21T12:21:22.834972+0000","last_active":"2024-09-21T12:24:25.315085+0000","last_peered":"2024-09-21T12:24:25.315085+0000","last_clean":"2024-09-21T12:24:25.315085+0000","last_became_active":"2024-09-21T12:20:14.287543+0000","last_became_peered":"2024-09-21T12:20:14.287543+0000","last_unstale":"2024-09-21T12:24:25.315085+0000","last_undegraded":"2024-09-21T12:24:25.315085+0000","last_fullsized":"2024-09-21T12:24:25.315085+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-21T12:21:22.834917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:21:22.834917+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:21.087210+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":246,"num_read_kb":241,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":200,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.933438+0000","last_change":"2024-09-21T12:22:36.135464+0000","last_active":"2024-09-21T12:24:23.933438+0000","last_peered":"2024-09-21T12:24:23.933438+0000","last_clean":"2024-09-21T12:24:23.933438+0000","last_became_active":"2024-09-21T12:20:19.314217+0000","last_became_peered":"2024-09-21T12:20:19.314217+0000","last_unstale":"2024-09-21T12:24:23.933438+0000","last_undegraded":"2024-09-21T12:24:23.933438+0000","last_fullsized":"2024-09-21T12:24:23.933438+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:36.135390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:36.135390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:07.332958+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":197,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.926496+0000","last_change":"2024-09-21T12:22:57.265262+0000","last_active":"2024-09-21T12:24:23.926496+0000","last_peered":"2024-09-21T12:24:23.926496+0000","last_clean":"2024-09-21T12:24:23.926496+0000","last_became_active":"2024-09-21T12:20:21.311188+0000","last_became_peered":"2024-09-21T12:20:21.311188+0000","last_unstale":"2024-09-21T12:24:23.926496+0000","last_undegraded":"2024-09-21T12:24:23.926496+0000","last_fullsized":"2024-09-21T12:24:23.926496+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:57.265191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:57.265191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:24.950692+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"153'12356","reported_seq":12566,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:26.977698+0000","last_change":"2024-09-21T12:23:19.231036+0000","last_active":"2024-09-21T12:24:26.977698+0000","last_peered":"2024-09-21T12:24:26.977698+0000","last_clean":"2024-09-21T12:24:26.977698+0000","last_became_active":"2024-09-21T12:21:24.885986+0000","last_became_peered":"2024-09-21T12:21:24.885986+0000","last_unstale":"2024-09-21T12:24:26.977698+0000","last_undegraded":"2024-09-21T12:24:26.977698+0000","last_fullsized":"2024-09-21T12:24:26.977698+0000","mapping_epoch":84,"log_start":"133'8078","ondisk_log_start":"133'8078","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'7605","last_scrub_stamp":"2024-09-21T12:23:19.230513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:23:19.230513+0000","objects_scrubbed":5276,"log_size":4278,"log_dups_size":0,"ondisk_log_size":4278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:27.706332+0000","scrub_duration":2077,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30801920,"num_objects":470,"num_object_clones":0,"num_object_copies":1410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12356,"num_write_kb":410432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"121'19","reported_seq":352,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.925226+0000","last_change":"2024-09-21T12:22:53.756501+0000","last_active":"2024-09-21T12:24:23.925226+0000","last_peered":"2024-09-21T12:24:23.925226+0000","last_clean":"2024-09-21T12:24:23.925226+0000","last_became_active":"2024-09-21T12:22:50.921690+0000","last_became_peered":"2024-09-21T12:22:50.921690+0000","last_unstale":"2024-09-21T12:24:23.925226+0000","last_undegraded":"2024-09-21T12:24:23.925226+0000","last_fullsized":"2024-09-21T12:24:23.925226+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'19","last_scrub_stamp":"2024-09-21T12:22:53.756438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:53.756438+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":214,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.979365+0000","last_change":"2024-09-21T12:21:17.107378+0000","last_active":"2024-09-21T12:24:23.979365+0000","last_peered":"2024-09-21T12:24:23.979365+0000","last_clean":"2024-09-21T12:24:23.979365+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:24:23.979365+0000","last_undegraded":"2024-09-21T12:24:23.979365+0000","last_fullsized":"2024-09-21T12:24:23.979365+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:21:17.107309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:17.107309+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:09.385398+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":265,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.926180+0000","last_change":"2024-09-21T12:22:16.936605+0000","last_active":"2024-09-21T12:24:23.926180+0000","last_peered":"2024-09-21T12:24:23.926180+0000","last_clean":"2024-09-21T12:24:23.926180+0000","last_became_active":"2024-09-21T12:22:15.049592+0000","last_became_peered":"2024-09-21T12:22:15.049592+0000","last_unstale":"2024-09-21T12:24:23.926180+0000","last_undegraded":"2024-09-21T12:24:23.926180+0000","last_fullsized":"2024-09-21T12:24:23.926180+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:22:16.936548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:22:16.936548+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":192,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.927048+0000","last_change":"2024-09-21T12:20:19.310280+0000","last_active":"2024-09-21T12:24:23.927048+0000","last_peered":"2024-09-21T12:24:23.927048+0000","last_clean":"2024-09-21T12:24:23.927048+0000","last_became_active":"2024-09-21T12:20:19.309859+0000","last_became_peered":"2024-09-21T12:20:19.309859+0000","last_unstale":"2024-09-21T12:24:23.927048+0000","last_undegraded":"2024-09-21T12:24:23.927048+0000","last_fullsized":"2024-09-21T12:24:23.927048+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:15.053195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":225,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.933311+0000","last_change":"2024-09-21T12:23:56.364599+0000","last_active":"2024-09-21T12:24:23.933311+0000","last_peered":"2024-09-21T12:24:23.933311+0000","last_clean":"2024-09-21T12:24:23.933311+0000","last_became_active":"2024-09-21T12:22:50.913493+0000","last_became_peered":"2024-09-21T12:22:50.913493+0000","last_unstale":"2024-09-21T12:24:23.933311+0000","last_undegraded":"2024-09-21T12:24:23.933311+0000","last_fullsized":"2024-09-21T12:24:23.933311+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:56.364537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:56.364537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:25.702501+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"153'12556","reported_seq":12780,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:26.593780+0000","last_change":"2024-09-21T12:24:04.221374+0000","last_active":"2024-09-21T12:24:26.593780+0000","last_peered":"2024-09-21T12:24:26.593780+0000","last_clean":"2024-09-21T12:24:26.593780+0000","last_became_active":"2024-09-21T12:21:24.887845+0000","last_became_peered":"2024-09-21T12:21:24.887845+0000","last_unstale":"2024-09-21T12:24:26.593780+0000","last_undegraded":"2024-09-21T12:24:26.593780+0000","last_fullsized":"2024-09-21T12:24:26.593780+0000","mapping_epoch":84,"log_start":"133'8002","ondisk_log_start":"133'8002","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'5424","last_scrub_stamp":"2024-09-21T12:22:40.358604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:22:40.358604+0000","objects_scrubbed":899,"log_size":4554,"log_dups_size":0,"ondisk_log_size":4554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:34.191565+0000","scrub_duration":1743,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35586072,"num_objects":544,"num_object_clones":0,"num_object_copies":1632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":544,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12556,"num_write_kb":419137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":215,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.924728+0000","last_change":"2024-09-21T12:23:51.061783+0000","last_active":"2024-09-21T12:24:23.924728+0000","last_peered":"2024-09-21T12:24:23.924728+0000","last_clean":"2024-09-21T12:24:23.924728+0000","last_became_active":"2024-09-21T12:23:46.885802+0000","last_became_peered":"2024-09-21T12:23:46.885802+0000","last_unstale":"2024-09-21T12:24:23.924728+0000","last_undegraded":"2024-09-21T12:24:23.924728+0000","last_fullsized":"2024-09-21T12:24:23.924728+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:23:51.061706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:51.061706+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:09.333060+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":133,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.923345+0000","last_change":"2024-09-21T12:22:50.908380+0000","last_active":"2024-09-21T12:24:23.923345+0000","last_peered":"2024-09-21T12:24:23.923345+0000","last_clean":"2024-09-21T12:24:23.923345+0000","last_became_active":"2024-09-21T12:22:50.908060+0000","last_became_peered":"2024-09-21T12:22:50.908060+0000","last_unstale":"2024-09-21T12:24:23.923345+0000","last_undegraded":"2024-09-21T12:24:23.923345+0000","last_fullsized":"2024-09-21T12:24:23.923345+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:05.892153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:05.892153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:08.389121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":229,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.933306+0000","last_change":"2024-09-21T12:23:19.372762+0000","last_active":"2024-09-21T12:24:23.933306+0000","last_peered":"2024-09-21T12:24:23.933306+0000","last_clean":"2024-09-21T12:24:23.933306+0000","last_became_active":"2024-09-21T12:22:50.915081+0000","last_became_peered":"2024-09-21T12:22:50.915081+0000","last_unstale":"2024-09-21T12:24:23.933306+0000","last_undegraded":"2024-09-21T12:24:23.933306+0000","last_fullsized":"2024-09-21T12:24:23.933306+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:19.372719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:19.372719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:32.884588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":195,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.978033+0000","last_change":"2024-09-21T12:23:46.282792+0000","last_active":"2024-09-21T12:24:23.978033+0000","last_peered":"2024-09-21T12:24:23.978033+0000","last_clean":"2024-09-21T12:24:23.978033+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:24:23.978033+0000","last_undegraded":"2024-09-21T12:24:23.978033+0000","last_fullsized":"2024-09-21T12:24:23.978033+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:46.282661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:46.282661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:00.482338+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"76'79","reported_seq":117,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.931857+0000","last_change":"2024-09-21T12:23:22.321968+0000","last_active":"2024-09-21T12:24:23.931857+0000","last_peered":"2024-09-21T12:24:23.931857+0000","last_clean":"2024-09-21T12:24:23.931857+0000","last_became_active":"2024-09-21T12:22:17.087245+0000","last_became_peered":"2024-09-21T12:22:17.087245+0000","last_unstale":"2024-09-21T12:24:23.931857+0000","last_undegraded":"2024-09-21T12:24:23.931857+0000","last_fullsized":"2024-09-21T12:24:23.931857+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:23:22.321892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:23:22.321892+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"153'12538","reported_seq":12688,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:26.737932+0000","last_change":"2024-09-21T12:21:24.888960+0000","last_active":"2024-09-21T12:24:26.737932+0000","last_peered":"2024-09-21T12:24:26.737932+0000","last_clean":"2024-09-21T12:24:26.737932+0000","last_became_active":"2024-09-21T12:21:24.888670+0000","last_became_peered":"2024-09-21T12:21:24.888670+0000","last_unstale":"2024-09-21T12:24:26.737932+0000","last_undegraded":"2024-09-21T12:24:26.737932+0000","last_fullsized":"2024-09-21T12:24:26.737932+0000","mapping_epoch":84,"log_start":"130'7384","ondisk_log_start":"130'7384","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":5154,"log_dups_size":0,"ondisk_log_size":5154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:43.402696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30146560,"num_objects":460,"num_object_clones":0,"num_object_copies":1380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12538,"num_write_kb":415936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":250,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.925291+0000","last_change":"2024-09-21T12:23:50.328004+0000","last_active":"2024-09-21T12:24:23.925291+0000","last_peered":"2024-09-21T12:24:23.925291+0000","last_clean":"2024-09-21T12:24:23.925291+0000","last_became_active":"2024-09-21T12:23:46.897197+0000","last_became_peered":"2024-09-21T12:23:46.897197+0000","last_unstale":"2024-09-21T12:24:23.925291+0000","last_undegraded":"2024-09-21T12:24:23.925291+0000","last_fullsized":"2024-09-21T12:24:23.925291+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-21T12:23:50.327918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:50.327918+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:08.093826+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":237,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.924498+0000","last_change":"2024-09-21T12:23:52.067111+0000","last_active":"2024-09-21T12:24:23.924498+0000","last_peered":"2024-09-21T12:24:23.924498+0000","last_clean":"2024-09-21T12:24:23.924498+0000","last_became_active":"2024-09-21T12:22:50.906886+0000","last_became_peered":"2024-09-21T12:22:50.906886+0000","last_unstale":"2024-09-21T12:24:23.924498+0000","last_undegraded":"2024-09-21T12:24:23.924498+0000","last_fullsized":"2024-09-21T12:24:23.924498+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:52.067065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:23:52.067065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:06.846290+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":222,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.926596+0000","last_change":"2024-09-21T12:23:17.434377+0000","last_active":"2024-09-21T12:24:23.926596+0000","last_peered":"2024-09-21T12:24:23.926596+0000","last_clean":"2024-09-21T12:24:23.926596+0000","last_became_active":"2024-09-21T12:22:50.927438+0000","last_became_peered":"2024-09-21T12:22:50.927438+0000","last_unstale":"2024-09-21T12:24:23.926596+0000","last_undegraded":"2024-09-21T12:24:23.926596+0000","last_fullsized":"2024-09-21T12:24:23.926596+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:17.434261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:17.434261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:43.736130+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":210,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.925728+0000","last_change":"2024-09-21T12:22:50.914246+0000","last_active":"2024-09-21T12:24:23.925728+0000","last_peered":"2024-09-21T12:24:23.925728+0000","last_clean":"2024-09-21T12:24:23.925728+0000","last_became_active":"2024-09-21T12:22:50.913898+0000","last_became_peered":"2024-09-21T12:22:50.913898+0000","last_unstale":"2024-09-21T12:24:23.925728+0000","last_undegraded":"2024-09-21T12:24:23.925728+0000","last_fullsized":"2024-09-21T12:24:23.925728+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:35.313521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"76'79","reported_seq":263,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.926543+0000","last_change":"2024-09-21T12:22:17.073301+0000","last_active":"2024-09-21T12:24:23.926543+0000","last_peered":"2024-09-21T12:24:23.926543+0000","last_clean":"2024-09-21T12:24:23.926543+0000","last_became_active":"2024-09-21T12:22:17.072872+0000","last_became_peered":"2024-09-21T12:22:17.072872+0000","last_unstale":"2024-09-21T12:24:23.926543+0000","last_undegraded":"2024-09-21T12:24:23.926543+0000","last_fullsized":"2024-09-21T12:24:23.926543+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T12:20:24.850157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:20:24.850157+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:24:33.931607+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":21,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.f","version":"153'12674","reported_seq":12823,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:26.948756+0000","last_change":"2024-09-21T12:21:24.887418+0000","last_active":"2024-09-21T12:24:26.948756+0000","last_peered":"2024-09-21T12:24:26.948756+0000","last_clean":"2024-09-21T12:24:26.948756+0000","last_became_active":"2024-09-21T12:21:24.887000+0000","last_became_peered":"2024-09-21T12:21:24.887000+0000","last_unstale":"2024-09-21T12:24:26.948756+0000","last_undegraded":"2024-09-21T12:24:26.948756+0000","last_fullsized":"2024-09-21T12:24:26.948756+0000","mapping_epoch":84,"log_start":"133'8351","ondisk_log_start":"133'8351","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4323,"log_dups_size":0,"ondisk_log_size":4323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:12.027728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32112640,"num_objects":490,"num_object_clones":0,"num_object_copies":1470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12674,"num_write_kb":421248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":56,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.924760+0000","last_change":"2024-09-21T12:23:49.348056+0000","last_active":"2024-09-21T12:24:23.924760+0000","last_peered":"2024-09-21T12:24:23.924760+0000","last_clean":"2024-09-21T12:24:23.924760+0000","last_became_active":"2024-09-21T12:23:46.902260+0000","last_became_peered":"2024-09-21T12:23:46.902260+0000","last_unstale":"2024-09-21T12:24:23.924760+0000","last_undegraded":"2024-09-21T12:24:23.924760+0000","last_fullsized":"2024-09-21T12:24:23.924760+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:23:49.347964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:49.347964+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:02.497058+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":228,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.978874+0000","last_change":"2024-09-21T12:22:50.928240+0000","last_active":"2024-09-21T12:24:23.978874+0000","last_peered":"2024-09-21T12:24:23.978874+0000","last_clean":"2024-09-21T12:24:23.978874+0000","last_became_active":"2024-09-21T12:22:50.927946+0000","last_became_peered":"2024-09-21T12:22:50.927946+0000","last_unstale":"2024-09-21T12:24:23.978874+0000","last_undegraded":"2024-09-21T12:24:23.978874+0000","last_fullsized":"2024-09-21T12:24:23.978874+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:08.999015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:08.999015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:23.611546+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":260,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.925238+0000","last_change":"2024-09-21T12:23:22.274636+0000","last_active":"2024-09-21T12:24:23.925238+0000","last_peered":"2024-09-21T12:24:23.925238+0000","last_clean":"2024-09-21T12:24:23.925238+0000","last_became_active":"2024-09-21T12:20:19.307924+0000","last_became_peered":"2024-09-21T12:20:19.307924+0000","last_unstale":"2024-09-21T12:24:23.925238+0000","last_undegraded":"2024-09-21T12:24:23.925238+0000","last_fullsized":"2024-09-21T12:24:23.925238+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:23:22.274558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:22.274558+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:43.175576+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":141,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.926540+0000","last_change":"2024-09-21T12:23:56.471960+0000","last_active":"2024-09-21T12:24:23.926540+0000","last_peered":"2024-09-21T12:24:23.926540+0000","last_clean":"2024-09-21T12:24:23.926540+0000","last_became_active":"2024-09-21T12:22:50.905470+0000","last_became_peered":"2024-09-21T12:22:50.905470+0000","last_unstale":"2024-09-21T12:24:23.926540+0000","last_undegraded":"2024-09-21T12:24:23.926540+0000","last_fullsized":"2024-09-21T12:24:23.926540+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:56.471116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:56.471116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:13.266560+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"153'12751","reported_seq":20168,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:26.770536+0000","last_change":"2024-09-21T12:23:20.213767+0000","last_active":"2024-09-21T12:24:26.770536+0000","last_peered":"2024-09-21T12:24:26.770536+0000","last_clean":"2024-09-21T12:24:26.770536+0000","last_became_active":"2024-09-21T12:23:20.203417+0000","last_became_peered":"2024-09-21T12:23:20.203417+0000","last_unstale":"2024-09-21T12:24:26.770536+0000","last_undegraded":"2024-09-21T12:24:26.770536+0000","last_fullsized":"2024-09-21T12:24:26.770536+0000","mapping_epoch":131,"log_start":"130'7610","ondisk_log_start":"130'7610","created":84,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":5141,"log_dups_size":0,"ondisk_log_size":5141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:42.358912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32440320,"num_objects":495,"num_object_clones":0,"num_object_copies":1485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12751,"num_write_kb":423872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3621,"num_bytes_recovered":237305856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":280,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.921370+0000","last_change":"2024-09-21T12:23:15.165398+0000","last_active":"2024-09-21T12:24:23.921370+0000","last_peered":"2024-09-21T12:24:23.921370+0000","last_clean":"2024-09-21T12:24:23.921370+0000","last_became_active":"2024-09-21T12:22:50.922343+0000","last_became_peered":"2024-09-21T12:22:50.922343+0000","last_unstale":"2024-09-21T12:24:23.921370+0000","last_undegraded":"2024-09-21T12:24:23.921370+0000","last_fullsized":"2024-09-21T12:24:23.921370+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:23:15.158428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:15.158428+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:27.015422+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":216,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.932482+0000","last_change":"2024-09-21T12:23:49.455863+0000","last_active":"2024-09-21T12:24:23.932482+0000","last_peered":"2024-09-21T12:24:23.932482+0000","last_clean":"2024-09-21T12:24:23.932482+0000","last_became_active":"2024-09-21T12:20:15.302262+0000","last_became_peered":"2024-09-21T12:20:15.302262+0000","last_unstale":"2024-09-21T12:24:23.932482+0000","last_undegraded":"2024-09-21T12:24:23.932482+0000","last_fullsized":"2024-09-21T12:24:23.932482+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:49.455808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:23:49.455808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:13.983105+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":230,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.925579+0000","last_change":"2024-09-21T12:23:58.277901+0000","last_active":"2024-09-21T12:24:23.925579+0000","last_peered":"2024-09-21T12:24:23.925579+0000","last_clean":"2024-09-21T12:24:23.925579+0000","last_became_active":"2024-09-21T12:22:50.925974+0000","last_became_peered":"2024-09-21T12:22:50.925974+0000","last_unstale":"2024-09-21T12:24:23.925579+0000","last_undegraded":"2024-09-21T12:24:23.925579+0000","last_fullsized":"2024-09-21T12:24:23.925579+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:58.277829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:58.277829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:06.555793+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":197,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.920847+0000","last_change":"2024-09-21T12:23:28.121344+0000","last_active":"2024-09-21T12:24:23.920847+0000","last_peered":"2024-09-21T12:24:23.920847+0000","last_clean":"2024-09-21T12:24:23.920847+0000","last_became_active":"2024-09-21T12:20:21.311493+0000","last_became_peered":"2024-09-21T12:20:21.311493+0000","last_unstale":"2024-09-21T12:24:23.920847+0000","last_undegraded":"2024-09-21T12:24:23.920847+0000","last_fullsized":"2024-09-21T12:24:23.920847+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:28.121286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:28.121286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:54.388322+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"153'12742","reported_seq":22245,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:26.948883+0000","last_change":"2024-09-21T12:22:50.949717+0000","last_active":"2024-09-21T12:24:26.948883+0000","last_peered":"2024-09-21T12:24:26.948883+0000","last_clean":"2024-09-21T12:24:26.948883+0000","last_became_active":"2024-09-21T12:22:50.924585+0000","last_became_peered":"2024-09-21T12:22:50.924585+0000","last_unstale":"2024-09-21T12:24:26.948883+0000","last_undegraded":"2024-09-21T12:24:26.948883+0000","last_fullsized":"2024-09-21T12:24:26.948883+0000","mapping_epoch":120,"log_start":"134'8480","ondisk_log_start":"134'8480","created":84,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4262,"log_dups_size":0,"ondisk_log_size":4262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:13.043881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30670848,"num_objects":468,"num_object_clones":0,"num_object_copies":1404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12742,"num_write_kb":422720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4664,"num_bytes_recovered":305659904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":243,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.924377+0000","last_change":"2024-09-21T12:24:02.960807+0000","last_active":"2024-09-21T12:24:23.924377+0000","last_peered":"2024-09-21T12:24:23.924377+0000","last_clean":"2024-09-21T12:24:23.924377+0000","last_became_active":"2024-09-21T12:20:17.692867+0000","last_became_peered":"2024-09-21T12:20:17.692867+0000","last_unstale":"2024-09-21T12:24:23.924377+0000","last_undegraded":"2024-09-21T12:24:23.924377+0000","last_fullsized":"2024-09-21T12:24:23.924377+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-21T12:24:02.960733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:02.960733+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:17.768958+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":208,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.926729+0000","last_change":"2024-09-21T12:23:59.046000+0000","last_active":"2024-09-21T12:24:23.926729+0000","last_peered":"2024-09-21T12:24:23.926729+0000","last_clean":"2024-09-21T12:24:23.926729+0000","last_became_active":"2024-09-21T12:20:15.299333+0000","last_became_peered":"2024-09-21T12:20:15.299333+0000","last_unstale":"2024-09-21T12:24:23.926729+0000","last_undegraded":"2024-09-21T12:24:23.926729+0000","last_fullsized":"2024-09-21T12:24:23.926729+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:59.045927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:23:59.045927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:26.188346+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":193,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.924852+0000","last_change":"2024-09-21T12:20:19.307316+0000","last_active":"2024-09-21T12:24:23.924852+0000","last_peered":"2024-09-21T12:24:23.924852+0000","last_clean":"2024-09-21T12:24:23.924852+0000","last_became_active":"2024-09-21T12:20:19.306952+0000","last_became_peered":"2024-09-21T12:20:19.306952+0000","last_unstale":"2024-09-21T12:24:23.924852+0000","last_undegraded":"2024-09-21T12:24:23.924852+0000","last_fullsized":"2024-09-21T12:24:23.924852+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:33.300129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":188,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.926062+0000","last_change":"2024-09-21T12:20:21.314891+0000","last_active":"2024-09-21T12:24:23.926062+0000","last_peered":"2024-09-21T12:24:23.926062+0000","last_clean":"2024-09-21T12:24:23.926062+0000","last_became_active":"2024-09-21T12:20:21.314545+0000","last_became_peered":"2024-09-21T12:20:21.314545+0000","last_unstale":"2024-09-21T12:24:23.926062+0000","last_undegraded":"2024-09-21T12:24:23.926062+0000","last_fullsized":"2024-09-21T12:24:23.926062+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:37.032668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"153'12595","reported_seq":12742,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:26.987413+0000","last_change":"2024-09-21T12:21:24.888714+0000","last_active":"2024-09-21T12:24:26.987413+0000","last_peered":"2024-09-21T12:24:26.987413+0000","last_clean":"2024-09-21T12:24:26.987413+0000","last_became_active":"2024-09-21T12:21:24.888445+0000","last_became_peered":"2024-09-21T12:21:24.888445+0000","last_unstale":"2024-09-21T12:24:26.987413+0000","last_undegraded":"2024-09-21T12:24:26.987413+0000","last_fullsized":"2024-09-21T12:24:26.987413+0000","mapping_epoch":84,"log_start":"135'8560","ondisk_log_start":"135'8560","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4035,"log_dups_size":0,"ondisk_log_size":4035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:20.598292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30867456,"num_objects":471,"num_object_clones":0,"num_object_copies":1413,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12595,"num_write_kb":418112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":208,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.928779+0000","last_change":"2024-09-21T12:23:47.400086+0000","last_active":"2024-09-21T12:24:23.928779+0000","last_peered":"2024-09-21T12:24:23.928779+0000","last_clean":"2024-09-21T12:24:23.928779+0000","last_became_active":"2024-09-21T12:22:50.910973+0000","last_became_peered":"2024-09-21T12:22:50.910973+0000","last_unstale":"2024-09-21T12:24:23.928779+0000","last_undegraded":"2024-09-21T12:24:23.928779+0000","last_fullsized":"2024-09-21T12:24:23.928779+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:23:47.400000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:47.400000+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:51.328592+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":229,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.926913+0000","last_change":"2024-09-21T12:22:50.928139+0000","last_active":"2024-09-21T12:24:23.926913+0000","last_peered":"2024-09-21T12:24:23.926913+0000","last_clean":"2024-09-21T12:24:23.926913+0000","last_became_active":"2024-09-21T12:22:50.927859+0000","last_became_peered":"2024-09-21T12:22:50.927859+0000","last_unstale":"2024-09-21T12:24:23.926913+0000","last_undegraded":"2024-09-21T12:24:23.926913+0000","last_fullsized":"2024-09-21T12:24:23.926913+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:20.865749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.865749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:23.098052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":201,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.925661+0000","last_change":"2024-09-21T12:21:23.393320+0000","last_active":"2024-09-21T12:24:23.925661+0000","last_peered":"2024-09-21T12:24:23.925661+0000","last_clean":"2024-09-21T12:24:23.925661+0000","last_became_active":"2024-09-21T12:20:19.312572+0000","last_became_peered":"2024-09-21T12:20:19.312572+0000","last_unstale":"2024-09-21T12:24:23.925661+0000","last_undegraded":"2024-09-21T12:24:23.925661+0000","last_fullsized":"2024-09-21T12:24:23.925661+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.393270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.393270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:37.312138+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":195,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.978516+0000","last_change":"2024-09-21T12:22:36.006177+0000","last_active":"2024-09-21T12:24:23.978516+0000","last_peered":"2024-09-21T12:24:23.978516+0000","last_clean":"2024-09-21T12:24:23.978516+0000","last_became_active":"2024-09-21T12:20:21.317223+0000","last_became_peered":"2024-09-21T12:20:21.317223+0000","last_unstale":"2024-09-21T12:24:23.978516+0000","last_undegraded":"2024-09-21T12:24:23.978516+0000","last_fullsized":"2024-09-21T12:24:23.978516+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:36.006109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:36.006109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"153'12339","reported_seq":12489,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:26.678191+0000","last_change":"2024-09-21T12:21:24.888120+0000","last_active":"2024-09-21T12:24:26.678191+0000","last_peered":"2024-09-21T12:24:26.678191+0000","last_clean":"2024-09-21T12:24:26.678191+0000","last_became_active":"2024-09-21T12:21:24.887488+0000","last_became_peered":"2024-09-21T12:21:24.887488+0000","last_unstale":"2024-09-21T12:24:26.678191+0000","last_undegraded":"2024-09-21T12:24:26.678191+0000","last_fullsized":"2024-09-21T12:24:26.678191+0000","mapping_epoch":84,"log_start":"133'8098","ondisk_log_start":"133'8098","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:21:23.867302+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.867302+0000","objects_scrubbed":0,"log_size":4241,"log_dups_size":0,"ondisk_log_size":4241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:45.414846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35323904,"num_objects":539,"num_object_clones":0,"num_object_copies":1617,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12339,"num_write_kb":412096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":237,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.931909+0000","last_change":"2024-09-21T12:23:17.286227+0000","last_active":"2024-09-21T12:24:23.931909+0000","last_peered":"2024-09-21T12:24:23.931909+0000","last_clean":"2024-09-21T12:24:23.931909+0000","last_became_active":"2024-09-21T12:20:17.692870+0000","last_became_peered":"2024-09-21T12:20:17.692870+0000","last_unstale":"2024-09-21T12:24:23.931909+0000","last_undegraded":"2024-09-21T12:24:23.931909+0000","last_fullsized":"2024-09-21T12:24:23.931909+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:23:17.286166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:17.286166+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:45.336901+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":259,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.926720+0000","last_change":"2024-09-21T12:22:51.052311+0000","last_active":"2024-09-21T12:24:23.926720+0000","last_peered":"2024-09-21T12:24:23.926720+0000","last_clean":"2024-09-21T12:24:23.926720+0000","last_became_active":"2024-09-21T12:22:50.927141+0000","last_became_peered":"2024-09-21T12:22:50.927141+0000","last_unstale":"2024-09-21T12:24:23.926720+0000","last_undegraded":"2024-09-21T12:24:23.926720+0000","last_fullsized":"2024-09-21T12:24:23.926720+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-21T12:22:14.920932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.920932+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"121'11","reported_seq":284,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.926216+0000","last_change":"2024-09-21T12:22:50.915762+0000","last_active":"2024-09-21T12:24:23.926216+0000","last_peered":"2024-09-21T12:24:23.926216+0000","last_clean":"2024-09-21T12:24:23.926216+0000","last_became_active":"2024-09-21T12:22:50.908371+0000","last_became_peered":"2024-09-21T12:22:50.908371+0000","last_unstale":"2024-09-21T12:24:23.926216+0000","last_undegraded":"2024-09-21T12:24:23.926216+0000","last_fullsized":"2024-09-21T12:24:23.926216+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:45.000445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":196,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.932351+0000","last_change":"2024-09-21T12:22:45.223570+0000","last_active":"2024-09-21T12:24:23.932351+0000","last_peered":"2024-09-21T12:24:23.932351+0000","last_clean":"2024-09-21T12:24:23.932351+0000","last_became_active":"2024-09-21T12:20:21.592576+0000","last_became_peered":"2024-09-21T12:20:21.592576+0000","last_unstale":"2024-09-21T12:24:23.932351+0000","last_undegraded":"2024-09-21T12:24:23.932351+0000","last_fullsized":"2024-09-21T12:24:23.932351+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:45.223396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:45.223396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:15.441394+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":224,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:25.927008+0000","last_change":"2024-09-21T12:24:25.927008+0000","last_active":"2024-09-21T12:24:25.927008+0000","last_peered":"2024-09-21T12:24:25.927008+0000","last_clean":"2024-09-21T12:24:25.927008+0000","last_became_active":"2024-09-21T12:20:15.303480+0000","last_became_peered":"2024-09-21T12:20:15.303480+0000","last_unstale":"2024-09-21T12:24:25.927008+0000","last_undegraded":"2024-09-21T12:24:25.927008+0000","last_fullsized":"2024-09-21T12:24:25.927008+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:25.926922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:25.926922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:31.586807+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":212,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.983000+0000","last_change":"2024-09-21T12:22:36.999995+0000","last_active":"2024-09-21T12:24:23.983000+0000","last_peered":"2024-09-21T12:24:23.983000+0000","last_clean":"2024-09-21T12:24:23.983000+0000","last_became_active":"2024-09-21T12:20:17.694628+0000","last_became_peered":"2024-09-21T12:20:17.694628+0000","last_unstale":"2024-09-21T12:24:23.983000+0000","last_undegraded":"2024-09-21T12:24:23.983000+0000","last_fullsized":"2024-09-21T12:24:23.983000+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-21T12:22:36.999705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:36.999705+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":208,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.933254+0000","last_change":"2024-09-21T12:23:15.291669+0000","last_active":"2024-09-21T12:24:23.933254+0000","last_peered":"2024-09-21T12:24:23.933254+0000","last_clean":"2024-09-21T12:24:23.933254+0000","last_became_active":"2024-09-21T12:20:19.313882+0000","last_became_peered":"2024-09-21T12:20:19.313882+0000","last_unstale":"2024-09-21T12:24:23.933254+0000","last_undegraded":"2024-09-21T12:24:23.933254+0000","last_fullsized":"2024-09-21T12:24:23.933254+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:15.291602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:15.291602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:36.660286+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":217,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.926380+0000","last_change":"2024-09-21T12:24:00.066522+0000","last_active":"2024-09-21T12:24:23.926380+0000","last_peered":"2024-09-21T12:24:23.926380+0000","last_clean":"2024-09-21T12:24:23.926380+0000","last_became_active":"2024-09-21T12:22:50.904404+0000","last_became_peered":"2024-09-21T12:22:50.904404+0000","last_unstale":"2024-09-21T12:24:23.926380+0000","last_undegraded":"2024-09-21T12:24:23.926380+0000","last_fullsized":"2024-09-21T12:24:23.926380+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:00.066451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:00.066451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:27.003864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":221,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.919742+0000","last_change":"2024-09-21T12:21:24.093710+0000","last_active":"2024-09-21T12:24:23.919742+0000","last_peered":"2024-09-21T12:24:23.919742+0000","last_clean":"2024-09-21T12:24:23.919742+0000","last_became_active":"2024-09-21T12:20:17.695898+0000","last_became_peered":"2024-09-21T12:20:17.695898+0000","last_unstale":"2024-09-21T12:24:23.919742+0000","last_undegraded":"2024-09-21T12:24:23.919742+0000","last_fullsized":"2024-09-21T12:24:23.919742+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:21:24.093656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:21:24.093656+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:15.049499+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":237,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:26.078804+0000","last_change":"2024-09-21T12:24:26.078804+0000","last_active":"2024-09-21T12:24:26.078804+0000","last_peered":"2024-09-21T12:24:26.078804+0000","last_clean":"2024-09-21T12:24:26.078804+0000","last_became_active":"2024-09-21T12:22:50.927649+0000","last_became_peered":"2024-09-21T12:22:50.927649+0000","last_unstale":"2024-09-21T12:24:26.078804+0000","last_undegraded":"2024-09-21T12:24:26.078804+0000","last_fullsized":"2024-09-21T12:24:26.078804+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:26.078737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:26.078737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:51.918796+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":199,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.982832+0000","last_change":"2024-09-21T12:23:53.356954+0000","last_active":"2024-09-21T12:24:23.982832+0000","last_peered":"2024-09-21T12:24:23.982832+0000","last_clean":"2024-09-21T12:24:23.982832+0000","last_became_active":"2024-09-21T12:20:19.311947+0000","last_became_peered":"2024-09-21T12:20:19.311947+0000","last_unstale":"2024-09-21T12:24:23.982832+0000","last_undegraded":"2024-09-21T12:24:23.982832+0000","last_fullsized":"2024-09-21T12:24:23.982832+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:53.356896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:53.356896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:11.617911+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":226,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.924905+0000","last_change":"2024-09-21T12:23:51.313630+0000","last_active":"2024-09-21T12:24:23.924905+0000","last_peered":"2024-09-21T12:24:23.924905+0000","last_clean":"2024-09-21T12:24:23.924905+0000","last_became_active":"2024-09-21T12:22:50.924931+0000","last_became_peered":"2024-09-21T12:22:50.924931+0000","last_unstale":"2024-09-21T12:24:23.924905+0000","last_undegraded":"2024-09-21T12:24:23.924905+0000","last_fullsized":"2024-09-21T12:24:23.924905+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:51.313528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:51.313528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:55.518605+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":237,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.932845+0000","last_change":"2024-09-21T12:23:46.925752+0000","last_active":"2024-09-21T12:24:23.932845+0000","last_peered":"2024-09-21T12:24:23.932845+0000","last_clean":"2024-09-21T12:24:23.932845+0000","last_became_active":"2024-09-21T12:23:46.898523+0000","last_became_peered":"2024-09-21T12:23:46.898523+0000","last_unstale":"2024-09-21T12:24:23.932845+0000","last_undegraded":"2024-09-21T12:24:23.932845+0000","last_fullsized":"2024-09-21T12:24:23.932845+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:22:39.201667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:39.201667+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:12.377811+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":217,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:25.931617+0000","last_change":"2024-09-21T12:24:25.931617+0000","last_active":"2024-09-21T12:24:25.931617+0000","last_peered":"2024-09-21T12:24:25.931617+0000","last_clean":"2024-09-21T12:24:25.931617+0000","last_became_active":"2024-09-21T12:20:15.297519+0000","last_became_peered":"2024-09-21T12:20:15.297519+0000","last_unstale":"2024-09-21T12:24:25.931617+0000","last_undegraded":"2024-09-21T12:24:25.931617+0000","last_fullsized":"2024-09-21T12:24:25.931617+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:25.931561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:25.931561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:44.486853+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":258,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.978234+0000","last_change":"2024-09-21T12:23:57.379033+0000","last_active":"2024-09-21T12:24:23.978234+0000","last_peered":"2024-09-21T12:24:23.978234+0000","last_clean":"2024-09-21T12:24:23.978234+0000","last_became_active":"2024-09-21T12:20:19.305364+0000","last_became_peered":"2024-09-21T12:20:19.305364+0000","last_unstale":"2024-09-21T12:24:23.978234+0000","last_undegraded":"2024-09-21T12:24:23.978234+0000","last_fullsized":"2024-09-21T12:24:23.978234+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:23:57.378959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:57.378959+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:04.919603+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":133,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.925974+0000","last_change":"2024-09-21T12:23:21.250547+0000","last_active":"2024-09-21T12:24:23.925974+0000","last_peered":"2024-09-21T12:24:23.925974+0000","last_clean":"2024-09-21T12:24:23.925974+0000","last_became_active":"2024-09-21T12:22:50.924770+0000","last_became_peered":"2024-09-21T12:22:50.924770+0000","last_unstale":"2024-09-21T12:24:23.925974+0000","last_undegraded":"2024-09-21T12:24:23.925974+0000","last_fullsized":"2024-09-21T12:24:23.925974+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:21.250464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:21.250464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:40.125523+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":218,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.978437+0000","last_change":"2024-09-21T12:22:40.026248+0000","last_active":"2024-09-21T12:24:23.978437+0000","last_peered":"2024-09-21T12:24:23.978437+0000","last_clean":"2024-09-21T12:24:23.978437+0000","last_became_active":"2024-09-21T12:20:17.693209+0000","last_became_peered":"2024-09-21T12:20:17.693209+0000","last_unstale":"2024-09-21T12:24:23.978437+0000","last_undegraded":"2024-09-21T12:24:23.978437+0000","last_fullsized":"2024-09-21T12:24:23.978437+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:22:40.026037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:40.026037+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":229,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.926422+0000","last_change":"2024-09-21T12:22:50.908560+0000","last_active":"2024-09-21T12:24:23.926422+0000","last_peered":"2024-09-21T12:24:23.926422+0000","last_clean":"2024-09-21T12:24:23.926422+0000","last_became_active":"2024-09-21T12:22:50.908238+0000","last_became_peered":"2024-09-21T12:22:50.908238+0000","last_unstale":"2024-09-21T12:24:23.926422+0000","last_undegraded":"2024-09-21T12:24:23.926422+0000","last_fullsized":"2024-09-21T12:24:23.926422+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:15.938093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:15.938093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:22.088220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":259,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.932632+0000","last_change":"2024-09-21T12:22:46.216242+0000","last_active":"2024-09-21T12:24:23.932632+0000","last_peered":"2024-09-21T12:24:23.932632+0000","last_clean":"2024-09-21T12:24:23.932632+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:24:23.932632+0000","last_undegraded":"2024-09-21T12:24:23.932632+0000","last_fullsized":"2024-09-21T12:24:23.932632+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:22:46.216153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:46.216153+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:16.423966+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":189,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.919315+0000","last_change":"2024-09-21T12:20:21.309936+0000","last_active":"2024-09-21T12:24:23.919315+0000","last_peered":"2024-09-21T12:24:23.919315+0000","last_clean":"2024-09-21T12:24:23.919315+0000","last_became_active":"2024-09-21T12:20:21.309385+0000","last_became_peered":"2024-09-21T12:20:21.309385+0000","last_unstale":"2024-09-21T12:24:23.919315+0000","last_undegraded":"2024-09-21T12:24:23.919315+0000","last_fullsized":"2024-09-21T12:24:23.919315+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:06.127285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":196,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.925115+0000","last_change":"2024-09-21T12:23:59.046758+0000","last_active":"2024-09-21T12:24:23.925115+0000","last_peered":"2024-09-21T12:24:23.925115+0000","last_clean":"2024-09-21T12:24:23.925115+0000","last_became_active":"2024-09-21T12:22:50.906473+0000","last_became_peered":"2024-09-21T12:22:50.906473+0000","last_unstale":"2024-09-21T12:24:23.925115+0000","last_undegraded":"2024-09-21T12:24:23.925115+0000","last_fullsized":"2024-09-21T12:24:23.925115+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:23:59.046704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:59.046704+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:11.682375+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":125,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.927135+0000","last_change":"2024-09-21T12:22:50.904173+0000","last_active":"2024-09-21T12:24:23.927135+0000","last_peered":"2024-09-21T12:24:23.927135+0000","last_clean":"2024-09-21T12:24:23.927135+0000","last_became_active":"2024-09-21T12:22:50.903705+0000","last_became_peered":"2024-09-21T12:22:50.903705+0000","last_unstale":"2024-09-21T12:24:23.927135+0000","last_undegraded":"2024-09-21T12:24:23.927135+0000","last_fullsized":"2024-09-21T12:24:23.927135+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:40.056927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":193,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.926486+0000","last_change":"2024-09-21T12:20:19.315714+0000","last_active":"2024-09-21T12:24:23.926486+0000","last_peered":"2024-09-21T12:24:23.926486+0000","last_clean":"2024-09-21T12:24:23.926486+0000","last_became_active":"2024-09-21T12:20:19.315418+0000","last_became_peered":"2024-09-21T12:20:19.315418+0000","last_unstale":"2024-09-21T12:24:23.926486+0000","last_undegraded":"2024-09-21T12:24:23.926486+0000","last_fullsized":"2024-09-21T12:24:23.926486+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:35.447262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":195,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.929033+0000","last_change":"2024-09-21T12:22:50.143669+0000","last_active":"2024-09-21T12:24:23.929033+0000","last_peered":"2024-09-21T12:24:23.929033+0000","last_clean":"2024-09-21T12:24:23.929033+0000","last_became_active":"2024-09-21T12:20:21.310053+0000","last_became_peered":"2024-09-21T12:20:21.310053+0000","last_unstale":"2024-09-21T12:24:23.929033+0000","last_undegraded":"2024-09-21T12:24:23.929033+0000","last_fullsized":"2024-09-21T12:24:23.929033+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:50.143628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:50.143628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":213,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.927854+0000","last_change":"2024-09-21T12:20:17.299367+0000","last_active":"2024-09-21T12:24:23.927854+0000","last_peered":"2024-09-21T12:24:23.927854+0000","last_clean":"2024-09-21T12:24:23.927854+0000","last_became_active":"2024-09-21T12:20:17.299015+0000","last_became_peered":"2024-09-21T12:20:17.299015+0000","last_unstale":"2024-09-21T12:24:23.927854+0000","last_undegraded":"2024-09-21T12:24:23.927854+0000","last_fullsized":"2024-09-21T12:24:23.927854+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:37.431706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":229,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.926595+0000","last_change":"2024-09-21T12:22:50.922328+0000","last_active":"2024-09-21T12:24:23.926595+0000","last_peered":"2024-09-21T12:24:23.926595+0000","last_clean":"2024-09-21T12:24:23.926595+0000","last_became_active":"2024-09-21T12:22:50.922019+0000","last_became_peered":"2024-09-21T12:22:50.922019+0000","last_unstale":"2024-09-21T12:24:23.926595+0000","last_undegraded":"2024-09-21T12:24:23.926595+0000","last_fullsized":"2024-09-21T12:24:23.926595+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:05.067280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:05.067280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":199,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.978611+0000","last_change":"2024-09-21T12:21:20.931077+0000","last_active":"2024-09-21T12:24:23.978611+0000","last_peered":"2024-09-21T12:24:23.978611+0000","last_clean":"2024-09-21T12:24:23.978611+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:24:23.978611+0000","last_undegraded":"2024-09-21T12:24:23.978611+0000","last_fullsized":"2024-09-21T12:24:23.978611+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:20.931025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.931025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:16.543441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":197,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.918760+0000","last_change":"2024-09-21T12:23:29.134829+0000","last_active":"2024-09-21T12:24:23.918760+0000","last_peered":"2024-09-21T12:24:23.918760+0000","last_clean":"2024-09-21T12:24:23.918760+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:24:23.918760+0000","last_undegraded":"2024-09-21T12:24:23.918760+0000","last_fullsized":"2024-09-21T12:24:23.918760+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:29.134767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:29.134767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:36.055630+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":222,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.929270+0000","last_change":"2024-09-21T12:22:35.026113+0000","last_active":"2024-09-21T12:24:23.929270+0000","last_peered":"2024-09-21T12:24:23.929270+0000","last_clean":"2024-09-21T12:24:23.929270+0000","last_became_active":"2024-09-21T12:20:17.694794+0000","last_became_peered":"2024-09-21T12:20:17.694794+0000","last_unstale":"2024-09-21T12:24:23.929270+0000","last_undegraded":"2024-09-21T12:24:23.929270+0000","last_fullsized":"2024-09-21T12:24:23.929270+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:22:35.026007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:35.026007+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":216,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.933337+0000","last_change":"2024-09-21T12:23:28.334764+0000","last_active":"2024-09-21T12:24:23.933337+0000","last_peered":"2024-09-21T12:24:23.933337+0000","last_clean":"2024-09-21T12:24:23.933337+0000","last_became_active":"2024-09-21T12:20:15.295968+0000","last_became_peered":"2024-09-21T12:20:15.295968+0000","last_unstale":"2024-09-21T12:24:23.933337+0000","last_undegraded":"2024-09-21T12:24:23.933337+0000","last_fullsized":"2024-09-21T12:24:23.933337+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:28.334702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:23:28.334702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:31.853399+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":221,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.925823+0000","last_change":"2024-09-21T12:23:57.018797+0000","last_active":"2024-09-21T12:24:23.925823+0000","last_peered":"2024-09-21T12:24:23.925823+0000","last_clean":"2024-09-21T12:24:23.925823+0000","last_became_active":"2024-09-21T12:22:50.921604+0000","last_became_peered":"2024-09-21T12:22:50.921604+0000","last_unstale":"2024-09-21T12:24:23.925823+0000","last_undegraded":"2024-09-21T12:24:23.925823+0000","last_fullsized":"2024-09-21T12:24:23.925823+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:57.018749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:57.018749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:03.841967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":187,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.978787+0000","last_change":"2024-09-21T12:20:21.595660+0000","last_active":"2024-09-21T12:24:23.978787+0000","last_peered":"2024-09-21T12:24:23.978787+0000","last_clean":"2024-09-21T12:24:23.978787+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:24:23.978787+0000","last_undegraded":"2024-09-21T12:24:23.978787+0000","last_fullsized":"2024-09-21T12:24:23.978787+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:32.224446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":209,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.978620+0000","last_change":"2024-09-21T12:23:51.343568+0000","last_active":"2024-09-21T12:24:23.978620+0000","last_peered":"2024-09-21T12:24:23.978620+0000","last_clean":"2024-09-21T12:24:23.978620+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:24:23.978620+0000","last_undegraded":"2024-09-21T12:24:23.978620+0000","last_fullsized":"2024-09-21T12:24:23.978620+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-21T12:23:51.343491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:51.343491+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:08.087951+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":215,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.920570+0000","last_change":"2024-09-21T12:22:14.635397+0000","last_active":"2024-09-21T12:24:23.920570+0000","last_peered":"2024-09-21T12:24:23.920570+0000","last_clean":"2024-09-21T12:24:23.920570+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:24:23.920570+0000","last_undegraded":"2024-09-21T12:24:23.920570+0000","last_fullsized":"2024-09-21T12:24:23.920570+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:22:14.635191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.635191+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:33.123389+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":221,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.924660+0000","last_change":"2024-09-21T12:22:50.931202+0000","last_active":"2024-09-21T12:24:23.924660+0000","last_peered":"2024-09-21T12:24:23.924660+0000","last_clean":"2024-09-21T12:24:23.924660+0000","last_became_active":"2024-09-21T12:22:50.930615+0000","last_became_peered":"2024-09-21T12:22:50.930615+0000","last_unstale":"2024-09-21T12:24:23.924660+0000","last_undegraded":"2024-09-21T12:24:23.924660+0000","last_fullsized":"2024-09-21T12:24:23.924660+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:09.649109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:09.649109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:19.977825+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":209,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.926102+0000","last_change":"2024-09-21T12:22:50.917883+0000","last_active":"2024-09-21T12:24:23.926102+0000","last_peered":"2024-09-21T12:24:23.926102+0000","last_clean":"2024-09-21T12:24:23.926102+0000","last_became_active":"2024-09-21T12:22:50.917408+0000","last_became_peered":"2024-09-21T12:22:50.917408+0000","last_unstale":"2024-09-21T12:24:23.926102+0000","last_undegraded":"2024-09-21T12:24:23.926102+0000","last_fullsized":"2024-09-21T12:24:23.926102+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:05.158628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":210,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.928015+0000","last_change":"2024-09-21T12:22:50.914633+0000","last_active":"2024-09-21T12:24:23.928015+0000","last_peered":"2024-09-21T12:24:23.928015+0000","last_clean":"2024-09-21T12:24:23.928015+0000","last_became_active":"2024-09-21T12:22:50.914221+0000","last_became_peered":"2024-09-21T12:22:50.914221+0000","last_unstale":"2024-09-21T12:24:23.928015+0000","last_undegraded":"2024-09-21T12:24:23.928015+0000","last_fullsized":"2024-09-21T12:24:23.928015+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:10.300219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":207,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.978844+0000","last_change":"2024-09-21T12:22:44.080149+0000","last_active":"2024-09-21T12:24:23.978844+0000","last_peered":"2024-09-21T12:24:23.978844+0000","last_clean":"2024-09-21T12:24:23.978844+0000","last_became_active":"2024-09-21T12:20:15.297993+0000","last_became_peered":"2024-09-21T12:20:15.297993+0000","last_unstale":"2024-09-21T12:24:23.978844+0000","last_undegraded":"2024-09-21T12:24:23.978844+0000","last_fullsized":"2024-09-21T12:24:23.978844+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:44.080070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:44.080070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":216,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:23.924946+0000","last_change":"2024-09-21T12:20:17.299134+0000","last_active":"2024-09-21T12:24:23.924946+0000","last_peered":"2024-09-21T12:24:23.924946+0000","last_clean":"2024-09-21T12:24:23.924946+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:24:23.924946+0000","last_undegraded":"2024-09-21T12:24:23.924946+0000","last_fullsized":"2024-09-21T12:24:23.924946+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:24:36.062051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":210,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-21T12:24:24.973869+0000","last_change":"2024-09-21T12:24:24.973667+0000","last_active":"2024-09-21T12:24:24.973869+0000","last_peered":"2024-09-21T12:24:24.973869+0000","last_clean":"2024-09-21T12:24:24.973869+0000","last_became_active":"2024-09-21T12:20:19.308247+0000","last_became_peered":"2024-09-21T12:20:19.308247+0000","last_unstale":"2024-09-21T12:24:24.973869+0000","last_undegraded":"2024-09-21T12:24:24.973869+0000","last_fullsized":"2024-09-21T12:24:24.973869+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:24.973604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:24.973604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:25.684844+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":521076760,"num_objects":7952,"num_object_clones":0,"num_object_copies":23856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7952,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200336,"num_write_kb":6665153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15812,"num_bytes_recovered":1036255232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1710043136,"data_stored":1710030920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71053,"ondisk_log_size":71053,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":690,"num_read_kb":447,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":201,"num_bytes_recovered":3982,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":396,"ondisk_log_size":396,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":5023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":246,"num_read_kb":241,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":644038,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":21,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234038,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":328536,"kb_used_data":207112,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93416616,"statfs":{"total":95995035648,"available":95658614784,"internally_reserved":0,"allocated":212082688,"data_stored":211790228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":124320202},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462974,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":396384,"kb_used_data":256844,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93348768,"statfs":{"total":95995035648,"available":95589138432,"internally_reserved":0,"allocated":263008256,"data_stored":262711515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":142801350},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724609,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":201564,"kb_used_data":134284,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93543588,"statfs":{"total":95995035648,"available":95788634112,"internally_reserved":0,"allocated":137506816,"data_stored":137211694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953545,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149775,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":507060,"kb_used_data":327268,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93238092,"statfs":{"total":95995035648,"available":95475806208,"internally_reserved":0,"allocated":335122432,"data_stored":334803143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":184023495},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411413,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310224,"kb_used_data":206652,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93434928,"statfs":{"total":95995035648,"available":95677366272,"internally_reserved":0,"allocated":211611648,"data_stored":211331367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":390784,"kb_used_data":242928,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93354368,"statfs":{"total":95995035648,"available":95594872832,"internally_reserved":0,"allocated":248758272,"data_stored":248425195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":367460,"kb_used_data":250196,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93377692,"statfs":{"total":95995035648,"available":95618756608,"internally_reserved":0,"allocated":256200704,"data_stored":255896693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":254218240,"data_stored":254214168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":238686208,"data_stored":238682136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":193789952,"data_stored":193789952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":315424768,"data_stored":315424768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":137297920,"data_stored":137297920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":133038080,"data_stored":133038080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":239538176,"data_stored":239534104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":198049792,"data_stored":198049792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:24:31.569 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T12:24:31.569 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:24:31.569 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-21T12:24:31.569 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-21T12:24:31.584 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:31.608 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:31.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:31.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:31.710 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:31.733 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:31.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:31.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:31.835 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:31.857 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:31.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:31.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:31.959 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:31.984 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:31.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:31.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:32.086 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:32.110 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:32.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:32.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:32.184 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:24:32.184 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:24:32.189 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:32.212 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:32.218 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:24:32.218 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:24:32.236 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:32.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:32.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:32.252 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:24:32.252 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:24:32.282 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:24:32.282 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:24:32.314 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:24:32.314 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:24:32.337 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:32.351 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:24:32.351 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:24:32.352 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:32 smithi114 ceph-mon[30324]: from='client.15066 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:24:32.352 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:32 smithi114 ceph-mon[30324]: 1.0 scrub starts 2024-09-21T12:24:32.352 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:32 smithi114 ceph-mon[30324]: 1.0 scrub ok 2024-09-21T12:24:32.352 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:32 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:24:32.352 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:32 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:24:32.352 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:32 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/723919751' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:24:32.352 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:32 smithi114 ceph-mon[40867]: from='client.15066 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:24:32.353 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:32 smithi114 ceph-mon[40867]: 1.0 scrub starts 2024-09-21T12:24:32.353 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:32 smithi114 ceph-mon[40867]: 1.0 scrub ok 2024-09-21T12:24:32.353 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:32 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:24:32.353 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:32 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:24:32.353 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:32 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/723919751' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:24:32.362 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:32.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:32.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:32.388 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:24:32.388 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:24:32.415 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:24:32.415 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:24:32.464 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:32 smithi183 ceph-mon[33277]: from='client.15066 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:24:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:32 smithi183 ceph-mon[33277]: 1.0 scrub starts 2024-09-21T12:24:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:32 smithi183 ceph-mon[33277]: 1.0 scrub ok 2024-09-21T12:24:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:32 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:24:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:32 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:24:32.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:32 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/723919751' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:24:32.489 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:32.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:32.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:32.590 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:32.626 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:32.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:32.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:32.728 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:32.766 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:32.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:32.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:32.774 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:24:32 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:24:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:24:32.859 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T12:24:32.868 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:32.889 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:32.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:32.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:32.990 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:33.017 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:33.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:33.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:33.313 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:33.339 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:33.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:33.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:33.441 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:33.469 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:33.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:33.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:33.473 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:33 smithi114 ceph-mon[30324]: pgmap v195: 158 pgs: 158 active+clean; 383 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 956 op/s 2024-09-21T12:24:33.473 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:33 smithi114 ceph-mon[30324]: 4.7 scrub starts 2024-09-21T12:24:33.473 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:33 smithi114 ceph-mon[30324]: 4.7 scrub ok 2024-09-21T12:24:33.474 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:33 smithi114 ceph-mon[40867]: pgmap v195: 158 pgs: 158 active+clean; 383 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 956 op/s 2024-09-21T12:24:33.474 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:33 smithi114 ceph-mon[40867]: 4.7 scrub starts 2024-09-21T12:24:33.474 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:33 smithi114 ceph-mon[40867]: 4.7 scrub ok 2024-09-21T12:24:33.523 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:33 smithi183 ceph-mon[33277]: pgmap v195: 158 pgs: 158 active+clean; 383 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 956 op/s 2024-09-21T12:24:33.523 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:33 smithi183 ceph-mon[33277]: 4.7 scrub starts 2024-09-21T12:24:33.523 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:33 smithi183 ceph-mon[33277]: 4.7 scrub ok 2024-09-21T12:24:33.553 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:24:33.553 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:24:33.553 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.553 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:208120 7.c 7:3188cbd4:::benchmark_data_smithi114_78432_object103974:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-21T12:24:33.553 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:24:33.542309+0000", 2024-09-21T12:24:33.554 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0086267010000000005, 2024-09-21T12:24:33.554 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0086568429999999991, 2024-09-21T12:24:33.554 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:33.554 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:33.554 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:24:33.554 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:24:33.554 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:24:33.554 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:33.554 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 208120 2024-09-21T12:24:33.554 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.554 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:33.554 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.554 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:33.554 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:33.542309+0000", 2024-09-21T12:24:33.555 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:33.555 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.555 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.555 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:33.555 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:33.542309+0000", 2024-09-21T12:24:33.555 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:33.555 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.555 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.555 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:33.555 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:33.542310+0000", 2024-09-21T12:24:33.555 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5499999999999996e-07 2024-09-21T12:24:33.555 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.556 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.556 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:33.556 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:33.542314+0000", 2024-09-21T12:24:33.556 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9219999999999999e-06 2024-09-21T12:24:33.556 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.556 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.556 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:33.556 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:33.542324+0000", 2024-09-21T12:24:33.556 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6290000000000005e-06 2024-09-21T12:24:33.556 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.556 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.556 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:33.556 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:33.542329+0000", 2024-09-21T12:24:33.556 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1420000000000001e-06 2024-09-21T12:24:33.557 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.557 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.557 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:33.557 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:33.543602+0000", 2024-09-21T12:24:33.557 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012730300000000001 2024-09-21T12:24:33.557 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.557 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.557 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:33.557 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:33.543723+0000", 2024-09-21T12:24:33.557 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000120646 2024-09-21T12:24:33.557 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.557 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.557 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:24:33.557 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:33.543842+0000", 2024-09-21T12:24:33.557 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000119675 2024-09-21T12:24:33.558 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.558 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.558 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:33.558 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:33.549149+0000", 2024-09-21T12:24:33.558 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0053065120000000002 2024-09-21T12:24:33.558 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.558 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.558 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:33.558 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:33.549192+0000", 2024-09-21T12:24:33.558 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3257000000000003e-05 2024-09-21T12:24:33.558 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:33.558 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:33.558 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:33.558 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.558 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.559 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:208122 7.d 7:bf99f2f5:::benchmark_data_smithi114_78432_object103976:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-21T12:24:33.559 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:24:33.545568+0000", 2024-09-21T12:24:33.559 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0053678650000000003, 2024-09-21T12:24:33.559 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0054728429999999998, 2024-09-21T12:24:33.559 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:33.559 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:33.559 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:24:33.559 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:24:33.559 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:24:33.559 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:33.559 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 208122 2024-09-21T12:24:33.559 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.559 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:33.559 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.559 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:33.560 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:33.545568+0000", 2024-09-21T12:24:33.560 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:33.560 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.560 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.560 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:33.560 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:33.545568+0000", 2024-09-21T12:24:33.560 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:33.560 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.560 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.560 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:33.560 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:33.545570+0000", 2024-09-21T12:24:33.560 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.53e-06 2024-09-21T12:24:33.560 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.560 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.560 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:33.560 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:33.545575+0000", 2024-09-21T12:24:33.561 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.118e-06 2024-09-21T12:24:33.561 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.561 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.561 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:33.561 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:33.545590+0000", 2024-09-21T12:24:33.561 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4945e-05 2024-09-21T12:24:33.561 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.561 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.561 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:33.561 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:33.545597+0000", 2024-09-21T12:24:33.561 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7330000000000003e-06 2024-09-21T12:24:33.561 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.561 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.561 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:33.562 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:33.545887+0000", 2024-09-21T12:24:33.562 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00028963600000000002 2024-09-21T12:24:33.562 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.562 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.562 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:33.562 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:33.546020+0000", 2024-09-21T12:24:33.562 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000132927 2024-09-21T12:24:33.562 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.562 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.562 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1", 2024-09-21T12:24:33.562 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:33.546145+0000", 2024-09-21T12:24:33.562 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012542299999999999 2024-09-21T12:24:33.562 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.562 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.562 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:24:33.563 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:33.549287+0000", 2024-09-21T12:24:33.563 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0031410600000000002 2024-09-21T12:24:33.563 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:33.563 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:33.563 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:33.563 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.563 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.563 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:208126 7.c 7:34f90a85:::benchmark_data_smithi114_78432_object103980:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-21T12:24:33.563 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:24:33.547124+0000", 2024-09-21T12:24:33.563 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0038120860000000001, 2024-09-21T12:24:33.563 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0039783839999999997, 2024-09-21T12:24:33.563 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:33.563 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:33.563 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:24:33.563 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:24:33.564 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:24:33.564 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:33.564 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 208126 2024-09-21T12:24:33.564 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.564 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:33.564 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.564 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:33.564 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:33.547124+0000", 2024-09-21T12:24:33.564 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:33.564 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.564 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.564 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:33.564 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:33.547124+0000", 2024-09-21T12:24:33.564 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:33.565 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.565 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.565 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:33.565 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:33.547125+0000", 2024-09-21T12:24:33.565 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5370000000000001e-06 2024-09-21T12:24:33.565 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.565 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.565 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:33.565 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:33.547130+0000", 2024-09-21T12:24:33.565 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6439999999999999e-06 2024-09-21T12:24:33.565 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.565 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.565 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:33.565 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:33.547145+0000", 2024-09-21T12:24:33.565 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4544e-05 2024-09-21T12:24:33.565 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.565 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.566 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:33.566 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:33.547153+0000", 2024-09-21T12:24:33.566 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8240000000000006e-06 2024-09-21T12:24:33.566 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.566 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.566 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:33.566 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:33.549221+0000", 2024-09-21T12:24:33.566 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020673419999999998 2024-09-21T12:24:33.566 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.566 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.566 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:33.566 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:33.549344+0000", 2024-09-21T12:24:33.566 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000123132 2024-09-21T12:24:33.566 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:33.567 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:33.567 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:24:33.567 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:33.549467+0000", 2024-09-21T12:24:33.567 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012335199999999999 2024-09-21T12:24:33.567 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:33.567 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:33.567 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:33.567 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:33.567 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-21T12:24:33.567 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 3 2024-09-21T12:24:33.567 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:24:33.572 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:33.599 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:33.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:33.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:33.627 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:33.701 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:33.717 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Removed 104143 objects 2024-09-21T12:24:33.718 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Clean up completed and total clean up time :93.0694 2024-09-21T12:24:33.726 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:33.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:33.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:33.779 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:24:33.779 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:24:33.779 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 154, 2024-09-21T12:24:33.779 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "e2ad975c-7812-11ef-baf6-efdc52797490", 2024-09-21T12:24:33.779 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-21T12:14:24.181227+0000", 2024-09-21T12:24:33.780 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-21T12:24:29.930050+0000", 2024-09-21T12:24:33.780 INFO:teuthology.orchestra.run.smithi114.stdout: "last_up_change": "2024-09-21T12:20:03.702987+0000", 2024-09-21T12:24:33.780 INFO:teuthology.orchestra.run.smithi114.stdout: "last_in_change": "2024-09-21T12:22:49.868269+0000", 2024-09-21T12:24:33.780 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T12:24:33.780 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_num": 5799936, 2024-09-21T12:24:33.780 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_set": [ 2024-09-21T12:24:33.780 INFO:teuthology.orchestra.run.smithi114.stdout: "pglog_hardlimit", 2024-09-21T12:24:33.780 INFO:teuthology.orchestra.run.smithi114.stdout: "purged_snapdirs", 2024-09-21T12:24:33.780 INFO:teuthology.orchestra.run.smithi114.stdout: "recovery_deletes", 2024-09-21T12:24:33.780 INFO:teuthology.orchestra.run.smithi114.stdout: "sortbitwise" 2024-09-21T12:24:33.780 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:24:33.780 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_version": 18, 2024-09-21T12:24:33.780 INFO:teuthology.orchestra.run.smithi114.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T12:24:33.780 INFO:teuthology.orchestra.run.smithi114.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T12:24:33.781 INFO:teuthology.orchestra.run.smithi114.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T12:24:33.781 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_snapshot": "", 2024-09-21T12:24:33.781 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_max": 7, 2024-09-21T12:24:33.781 INFO:teuthology.orchestra.run.smithi114.stdout: "max_osd": 8, 2024-09-21T12:24:33.781 INFO:teuthology.orchestra.run.smithi114.stdout: "require_min_compat_client": "luminous", 2024-09-21T12:24:33.781 INFO:teuthology.orchestra.run.smithi114.stdout: "min_compat_client": "luminous", 2024-09-21T12:24:33.781 INFO:teuthology.orchestra.run.smithi114.stdout: "require_osd_release": "squid", 2024-09-21T12:24:33.781 INFO:teuthology.orchestra.run.smithi114.stdout: "allow_crimson": false, 2024-09-21T12:24:33.781 INFO:teuthology.orchestra.run.smithi114.stdout: "pools": [ 2024-09-21T12:24:33.781 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.781 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 1, 2024-09-21T12:24:33.781 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": ".mgr", 2024-09-21T12:24:33.781 INFO:teuthology.orchestra.run.smithi114.stdout: "create_time": "2024-09-21T12:17:38.357647+0000", 2024-09-21T12:24:33.782 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-09-21T12:24:33.782 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_names": "hashpspool", 2024-09-21T12:24:33.782 INFO:teuthology.orchestra.run.smithi114.stdout: "type": 1, 2024-09-21T12:24:33.782 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 3, 2024-09-21T12:24:33.782 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": 2, 2024-09-21T12:24:33.782 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_rule": 0, 2024-09-21T12:24:33.782 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_count": 0, 2024-09-21T12:24:33.782 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_target": 0, 2024-09-21T12:24:33.782 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T12:24:33.782 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T12:24:33.782 INFO:teuthology.orchestra.run.smithi114.stdout: "is_stretch_pool": false, 2024-09-21T12:24:33.782 INFO:teuthology.orchestra.run.smithi114.stdout: "object_hash": 2, 2024-09-21T12:24:33.782 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_mode": "off", 2024-09-21T12:24:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": 11, 2024-09-21T12:24:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num": 11, 2024-09-21T12:24:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num_target": 11, 2024-09-21T12:24:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_target": 11, 2024-09-21T12:24:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_pending": 11, 2024-09-21T12:24:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: "last_pg_merge_meta": { 2024-09-21T12:24:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: "source_pgid": "0.0", 2024-09-21T12:24:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: "ready_epoch": 0, 2024-09-21T12:24:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_started": 0, 2024-09-21T12:24:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_clean": 0, 2024-09-21T12:24:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: "source_version": "0'0", 2024-09-21T12:24:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: "target_version": "0'0" 2024-09-21T12:24:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: "last_change": "136", 2024-09-21T12:24:33.783 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend": "0", 2024-09-21T12:24:33.784 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T12:24:33.784 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_preluminous": "106", 2024-09-21T12:24:33.784 INFO:teuthology.orchestra.run.smithi114.stdout: "auid": 0, 2024-09-21T12:24:33.784 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_mode": "selfmanaged", 2024-09-21T12:24:33.784 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_seq": 0, 2024-09-21T12:24:33.784 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_epoch": 0, 2024-09-21T12:24:33.784 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_snaps": [], 2024-09-21T12:24:33.784 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps": "[]", 2024-09-21T12:24:33.784 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_bytes": 0, 2024-09-21T12:24:33.784 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_objects": 0, 2024-09-21T12:24:33.784 INFO:teuthology.orchestra.run.smithi114.stdout: "tiers": [], 2024-09-21T12:24:33.784 INFO:teuthology.orchestra.run.smithi114.stdout: "tier_of": -1, 2024-09-21T12:24:33.784 INFO:teuthology.orchestra.run.smithi114.stdout: "read_tier": -1, 2024-09-21T12:24:33.784 INFO:teuthology.orchestra.run.smithi114.stdout: "write_tier": -1, 2024-09-21T12:24:33.785 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_mode": "none", 2024-09-21T12:24:33.785 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_bytes": 0, 2024-09-21T12:24:33.785 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_objects": 0, 2024-09-21T12:24:33.785 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T12:24:33.785 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T12:24:33.785 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T12:24:33.785 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_flush_age": 0, 2024-09-21T12:24:33.785 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_evict_age": 0, 2024-09-21T12:24:33.785 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profile": "", 2024-09-21T12:24:33.785 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_params": { 2024-09-21T12:24:33.785 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "none" 2024-09-21T12:24:33.785 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.785 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_period": 0, 2024-09-21T12:24:33.786 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_count": 0, 2024-09-21T12:24:33.786 INFO:teuthology.orchestra.run.smithi114.stdout: "use_gmt_hitset": true, 2024-09-21T12:24:33.786 INFO:teuthology.orchestra.run.smithi114.stdout: "min_read_recency_for_promote": 0, 2024-09-21T12:24:33.786 INFO:teuthology.orchestra.run.smithi114.stdout: "min_write_recency_for_promote": 0, 2024-09-21T12:24:33.786 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T12:24:33.786 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_search_last_n": 0, 2024-09-21T12:24:33.786 INFO:teuthology.orchestra.run.smithi114.stdout: "grade_table": [], 2024-09-21T12:24:33.786 INFO:teuthology.orchestra.run.smithi114.stdout: "stripe_width": 0, 2024-09-21T12:24:33.786 INFO:teuthology.orchestra.run.smithi114.stdout: "expected_num_objects": 0, 2024-09-21T12:24:33.786 INFO:teuthology.orchestra.run.smithi114.stdout: "fast_read": false, 2024-09-21T12:24:33.786 INFO:teuthology.orchestra.run.smithi114.stdout: "options": { 2024-09-21T12:24:33.786 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_max": 32, 2024-09-21T12:24:33.786 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_min": 1 2024-09-21T12:24:33.787 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.787 INFO:teuthology.orchestra.run.smithi114.stdout: "application_metadata": { 2024-09-21T12:24:33.787 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": {} 2024-09-21T12:24:33.787 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.787 INFO:teuthology.orchestra.run.smithi114.stdout: "read_balance": { 2024-09-21T12:24:33.787 INFO:teuthology.orchestra.run.smithi114.stdout: "score_type": "Fair distribution", 2024-09-21T12:24:33.787 INFO:teuthology.orchestra.run.smithi114.stdout: "score_acting": 2.1700000762939453, 2024-09-21T12:24:33.787 INFO:teuthology.orchestra.run.smithi114.stdout: "score_stable": 2.1700000762939453, 2024-09-21T12:24:33.787 INFO:teuthology.orchestra.run.smithi114.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T12:24:33.787 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_acting": 1.9099999666213989, 2024-09-21T12:24:33.787 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_stable": 1.9099999666213989, 2024-09-21T12:24:33.787 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity_weighted": 1, 2024-09-21T12:24:33.787 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity": 1, 2024-09-21T12:24:33.787 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity_weighted": 1 2024-09-21T12:24:33.788 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.788 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.788 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.788 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 2, 2024-09-21T12:24:33.788 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": "datapool", 2024-09-21T12:24:33.788 INFO:teuthology.orchestra.run.smithi114.stdout: "create_time": "2024-09-21T12:20:12.761046+0000", 2024-09-21T12:24:33.788 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 8193, 2024-09-21T12:24:33.788 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T12:24:33.788 INFO:teuthology.orchestra.run.smithi114.stdout: "type": 1, 2024-09-21T12:24:33.788 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 3, 2024-09-21T12:24:33.788 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": 2, 2024-09-21T12:24:33.788 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_rule": 0, 2024-09-21T12:24:33.788 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_count": 0, 2024-09-21T12:24:33.788 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_target": 0, 2024-09-21T12:24:33.788 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T12:24:33.789 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T12:24:33.789 INFO:teuthology.orchestra.run.smithi114.stdout: "is_stretch_pool": false, 2024-09-21T12:24:33.789 INFO:teuthology.orchestra.run.smithi114.stdout: "object_hash": 2, 2024-09-21T12:24:33.789 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_mode": "off", 2024-09-21T12:24:33.789 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": 3, 2024-09-21T12:24:33.789 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num": 3, 2024-09-21T12:24:33.789 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num_target": 3, 2024-09-21T12:24:33.789 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_target": 3, 2024-09-21T12:24:33.789 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_pending": 3, 2024-09-21T12:24:33.789 INFO:teuthology.orchestra.run.smithi114.stdout: "last_pg_merge_meta": { 2024-09-21T12:24:33.789 INFO:teuthology.orchestra.run.smithi114.stdout: "source_pgid": "0.0", 2024-09-21T12:24:33.789 INFO:teuthology.orchestra.run.smithi114.stdout: "ready_epoch": 0, 2024-09-21T12:24:33.789 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_started": 0, 2024-09-21T12:24:33.789 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_clean": 0, 2024-09-21T12:24:33.790 INFO:teuthology.orchestra.run.smithi114.stdout: "source_version": "0'0", 2024-09-21T12:24:33.790 INFO:teuthology.orchestra.run.smithi114.stdout: "target_version": "0'0" 2024-09-21T12:24:33.790 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.790 INFO:teuthology.orchestra.run.smithi114.stdout: "last_change": "61", 2024-09-21T12:24:33.790 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend": "0", 2024-09-21T12:24:33.790 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T12:24:33.790 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T12:24:33.790 INFO:teuthology.orchestra.run.smithi114.stdout: "auid": 0, 2024-09-21T12:24:33.790 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_mode": "selfmanaged", 2024-09-21T12:24:33.790 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_seq": 2, 2024-09-21T12:24:33.790 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_epoch": 61, 2024-09-21T12:24:33.790 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_snaps": [], 2024-09-21T12:24:33.790 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps": "[]", 2024-09-21T12:24:33.790 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_bytes": 0, 2024-09-21T12:24:33.791 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_objects": 0, 2024-09-21T12:24:33.791 INFO:teuthology.orchestra.run.smithi114.stdout: "tiers": [], 2024-09-21T12:24:33.791 INFO:teuthology.orchestra.run.smithi114.stdout: "tier_of": -1, 2024-09-21T12:24:33.791 INFO:teuthology.orchestra.run.smithi114.stdout: "read_tier": -1, 2024-09-21T12:24:33.791 INFO:teuthology.orchestra.run.smithi114.stdout: "write_tier": -1, 2024-09-21T12:24:33.791 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_mode": "none", 2024-09-21T12:24:33.791 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_bytes": 0, 2024-09-21T12:24:33.791 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_objects": 0, 2024-09-21T12:24:33.791 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T12:24:33.791 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T12:24:33.791 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T12:24:33.791 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_flush_age": 0, 2024-09-21T12:24:33.791 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_evict_age": 0, 2024-09-21T12:24:33.791 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profile": "", 2024-09-21T12:24:33.792 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_params": { 2024-09-21T12:24:33.792 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "none" 2024-09-21T12:24:33.792 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.792 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_period": 0, 2024-09-21T12:24:33.792 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_count": 0, 2024-09-21T12:24:33.792 INFO:teuthology.orchestra.run.smithi114.stdout: "use_gmt_hitset": true, 2024-09-21T12:24:33.792 INFO:teuthology.orchestra.run.smithi114.stdout: "min_read_recency_for_promote": 0, 2024-09-21T12:24:33.792 INFO:teuthology.orchestra.run.smithi114.stdout: "min_write_recency_for_promote": 0, 2024-09-21T12:24:33.792 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T12:24:33.792 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_search_last_n": 0, 2024-09-21T12:24:33.792 INFO:teuthology.orchestra.run.smithi114.stdout: "grade_table": [], 2024-09-21T12:24:33.792 INFO:teuthology.orchestra.run.smithi114.stdout: "stripe_width": 0, 2024-09-21T12:24:33.792 INFO:teuthology.orchestra.run.smithi114.stdout: "expected_num_objects": 0, 2024-09-21T12:24:33.792 INFO:teuthology.orchestra.run.smithi114.stdout: "fast_read": false, 2024-09-21T12:24:33.792 INFO:teuthology.orchestra.run.smithi114.stdout: "options": {}, 2024-09-21T12:24:33.793 INFO:teuthology.orchestra.run.smithi114.stdout: "application_metadata": { 2024-09-21T12:24:33.793 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd": {} 2024-09-21T12:24:33.793 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.793 INFO:teuthology.orchestra.run.smithi114.stdout: "read_balance": { 2024-09-21T12:24:33.793 INFO:teuthology.orchestra.run.smithi114.stdout: "score_type": "Fair distribution", 2024-09-21T12:24:33.793 INFO:teuthology.orchestra.run.smithi114.stdout: "score_acting": 2.6500000953674316, 2024-09-21T12:24:33.793 INFO:teuthology.orchestra.run.smithi114.stdout: "score_stable": 2.6500000953674316, 2024-09-21T12:24:33.793 INFO:teuthology.orchestra.run.smithi114.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T12:24:33.793 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-21T12:24:33.793 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-21T12:24:33.793 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity_weighted": 1, 2024-09-21T12:24:33.793 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity": 1, 2024-09-21T12:24:33.793 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity_weighted": 1 2024-09-21T12:24:33.793 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.794 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.794 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.794 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 3, 2024-09-21T12:24:33.794 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": ".rgw.root", 2024-09-21T12:24:33.794 INFO:teuthology.orchestra.run.smithi114.stdout: "create_time": "2024-09-21T12:20:13.713971+0000", 2024-09-21T12:24:33.794 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-09-21T12:24:33.794 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_names": "hashpspool", 2024-09-21T12:24:33.794 INFO:teuthology.orchestra.run.smithi114.stdout: "type": 1, 2024-09-21T12:24:33.794 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 3, 2024-09-21T12:24:33.794 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": 2, 2024-09-21T12:24:33.794 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_rule": 0, 2024-09-21T12:24:33.794 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_count": 0, 2024-09-21T12:24:33.794 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_target": 0, 2024-09-21T12:24:33.794 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T12:24:33.795 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T12:24:33.795 INFO:teuthology.orchestra.run.smithi114.stdout: "is_stretch_pool": false, 2024-09-21T12:24:33.795 INFO:teuthology.orchestra.run.smithi114.stdout: "object_hash": 2, 2024-09-21T12:24:33.795 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_mode": "off", 2024-09-21T12:24:33.795 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": 32, 2024-09-21T12:24:33.795 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num": 32, 2024-09-21T12:24:33.795 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num_target": 32, 2024-09-21T12:24:33.795 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_target": 32, 2024-09-21T12:24:33.795 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_pending": 32, 2024-09-21T12:24:33.795 INFO:teuthology.orchestra.run.smithi114.stdout: "last_pg_merge_meta": { 2024-09-21T12:24:33.795 INFO:teuthology.orchestra.run.smithi114.stdout: "source_pgid": "0.0", 2024-09-21T12:24:33.795 INFO:teuthology.orchestra.run.smithi114.stdout: "ready_epoch": 0, 2024-09-21T12:24:33.795 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_started": 0, 2024-09-21T12:24:33.795 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_clean": 0, 2024-09-21T12:24:33.796 INFO:teuthology.orchestra.run.smithi114.stdout: "source_version": "0'0", 2024-09-21T12:24:33.796 INFO:teuthology.orchestra.run.smithi114.stdout: "target_version": "0'0" 2024-09-21T12:24:33.796 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.796 INFO:teuthology.orchestra.run.smithi114.stdout: "last_change": "60", 2024-09-21T12:24:33.796 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend": "0", 2024-09-21T12:24:33.796 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T12:24:33.796 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T12:24:33.796 INFO:teuthology.orchestra.run.smithi114.stdout: "auid": 0, 2024-09-21T12:24:33.796 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_mode": "selfmanaged", 2024-09-21T12:24:33.796 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_seq": 0, 2024-09-21T12:24:33.796 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_epoch": 0, 2024-09-21T12:24:33.796 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_snaps": [], 2024-09-21T12:24:33.796 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps": "[]", 2024-09-21T12:24:33.796 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_bytes": 0, 2024-09-21T12:24:33.797 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_objects": 0, 2024-09-21T12:24:33.797 INFO:teuthology.orchestra.run.smithi114.stdout: "tiers": [], 2024-09-21T12:24:33.797 INFO:teuthology.orchestra.run.smithi114.stdout: "tier_of": -1, 2024-09-21T12:24:33.797 INFO:teuthology.orchestra.run.smithi114.stdout: "read_tier": -1, 2024-09-21T12:24:33.797 INFO:teuthology.orchestra.run.smithi114.stdout: "write_tier": -1, 2024-09-21T12:24:33.797 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_mode": "none", 2024-09-21T12:24:33.797 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_bytes": 0, 2024-09-21T12:24:33.797 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_objects": 0, 2024-09-21T12:24:33.797 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T12:24:33.797 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T12:24:33.797 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T12:24:33.797 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_flush_age": 0, 2024-09-21T12:24:33.797 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_evict_age": 0, 2024-09-21T12:24:33.797 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profile": "", 2024-09-21T12:24:33.798 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_params": { 2024-09-21T12:24:33.798 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "none" 2024-09-21T12:24:33.798 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.798 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_period": 0, 2024-09-21T12:24:33.798 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_count": 0, 2024-09-21T12:24:33.798 INFO:teuthology.orchestra.run.smithi114.stdout: "use_gmt_hitset": true, 2024-09-21T12:24:33.798 INFO:teuthology.orchestra.run.smithi114.stdout: "min_read_recency_for_promote": 0, 2024-09-21T12:24:33.798 INFO:teuthology.orchestra.run.smithi114.stdout: "min_write_recency_for_promote": 0, 2024-09-21T12:24:33.798 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T12:24:33.798 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_search_last_n": 0, 2024-09-21T12:24:33.798 INFO:teuthology.orchestra.run.smithi114.stdout: "grade_table": [], 2024-09-21T12:24:33.798 INFO:teuthology.orchestra.run.smithi114.stdout: "stripe_width": 0, 2024-09-21T12:24:33.798 INFO:teuthology.orchestra.run.smithi114.stdout: "expected_num_objects": 0, 2024-09-21T12:24:33.798 INFO:teuthology.orchestra.run.smithi114.stdout: "fast_read": false, 2024-09-21T12:24:33.799 INFO:teuthology.orchestra.run.smithi114.stdout: "options": {}, 2024-09-21T12:24:33.799 INFO:teuthology.orchestra.run.smithi114.stdout: "application_metadata": { 2024-09-21T12:24:33.799 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw": {} 2024-09-21T12:24:33.799 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.799 INFO:teuthology.orchestra.run.smithi114.stdout: "read_balance": { 2024-09-21T12:24:33.799 INFO:teuthology.orchestra.run.smithi114.stdout: "score_type": "Fair distribution", 2024-09-21T12:24:33.799 INFO:teuthology.orchestra.run.smithi114.stdout: "score_acting": 1.7400000095367432, 2024-09-21T12:24:33.799 INFO:teuthology.orchestra.run.smithi114.stdout: "score_stable": 1.7400000095367432, 2024-09-21T12:24:33.799 INFO:teuthology.orchestra.run.smithi114.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T12:24:33.799 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_acting": 1.5299999713897705, 2024-09-21T12:24:33.799 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_stable": 1.5299999713897705, 2024-09-21T12:24:33.799 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity_weighted": 1, 2024-09-21T12:24:33.799 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity": 1, 2024-09-21T12:24:33.799 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity_weighted": 1 2024-09-21T12:24:33.800 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.800 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.800 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.800 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 4, 2024-09-21T12:24:33.800 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": "default.rgw.log", 2024-09-21T12:24:33.800 INFO:teuthology.orchestra.run.smithi114.stdout: "create_time": "2024-09-21T12:20:15.368765+0000", 2024-09-21T12:24:33.800 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-09-21T12:24:33.800 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_names": "hashpspool", 2024-09-21T12:24:33.800 INFO:teuthology.orchestra.run.smithi114.stdout: "type": 1, 2024-09-21T12:24:33.800 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 3, 2024-09-21T12:24:33.800 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": 2, 2024-09-21T12:24:33.800 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_rule": 0, 2024-09-21T12:24:33.800 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_count": 0, 2024-09-21T12:24:33.800 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_target": 0, 2024-09-21T12:24:33.801 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T12:24:33.801 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T12:24:33.801 INFO:teuthology.orchestra.run.smithi114.stdout: "is_stretch_pool": false, 2024-09-21T12:24:33.801 INFO:teuthology.orchestra.run.smithi114.stdout: "object_hash": 2, 2024-09-21T12:24:33.801 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_mode": "off", 2024-09-21T12:24:33.801 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": 32, 2024-09-21T12:24:33.801 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num": 32, 2024-09-21T12:24:33.801 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num_target": 32, 2024-09-21T12:24:33.801 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_target": 32, 2024-09-21T12:24:33.801 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_pending": 32, 2024-09-21T12:24:33.801 INFO:teuthology.orchestra.run.smithi114.stdout: "last_pg_merge_meta": { 2024-09-21T12:24:33.801 INFO:teuthology.orchestra.run.smithi114.stdout: "source_pgid": "0.0", 2024-09-21T12:24:33.801 INFO:teuthology.orchestra.run.smithi114.stdout: "ready_epoch": 0, 2024-09-21T12:24:33.801 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_started": 0, 2024-09-21T12:24:33.802 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_clean": 0, 2024-09-21T12:24:33.802 INFO:teuthology.orchestra.run.smithi114.stdout: "source_version": "0'0", 2024-09-21T12:24:33.802 INFO:teuthology.orchestra.run.smithi114.stdout: "target_version": "0'0" 2024-09-21T12:24:33.802 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.802 INFO:teuthology.orchestra.run.smithi114.stdout: "last_change": "62", 2024-09-21T12:24:33.802 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend": "0", 2024-09-21T12:24:33.802 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T12:24:33.802 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T12:24:33.802 INFO:teuthology.orchestra.run.smithi114.stdout: "auid": 0, 2024-09-21T12:24:33.802 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_mode": "selfmanaged", 2024-09-21T12:24:33.802 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_seq": 0, 2024-09-21T12:24:33.802 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_epoch": 0, 2024-09-21T12:24:33.802 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_snaps": [], 2024-09-21T12:24:33.802 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps": "[]", 2024-09-21T12:24:33.803 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_bytes": 0, 2024-09-21T12:24:33.803 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_objects": 0, 2024-09-21T12:24:33.803 INFO:teuthology.orchestra.run.smithi114.stdout: "tiers": [], 2024-09-21T12:24:33.803 INFO:teuthology.orchestra.run.smithi114.stdout: "tier_of": -1, 2024-09-21T12:24:33.803 INFO:teuthology.orchestra.run.smithi114.stdout: "read_tier": -1, 2024-09-21T12:24:33.803 INFO:teuthology.orchestra.run.smithi114.stdout: "write_tier": -1, 2024-09-21T12:24:33.803 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_mode": "none", 2024-09-21T12:24:33.803 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_bytes": 0, 2024-09-21T12:24:33.803 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_objects": 0, 2024-09-21T12:24:33.803 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T12:24:33.803 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T12:24:33.803 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T12:24:33.803 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_flush_age": 0, 2024-09-21T12:24:33.804 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_evict_age": 0, 2024-09-21T12:24:33.804 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profile": "", 2024-09-21T12:24:33.804 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_params": { 2024-09-21T12:24:33.804 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "none" 2024-09-21T12:24:33.804 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.804 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_period": 0, 2024-09-21T12:24:33.804 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_count": 0, 2024-09-21T12:24:33.804 INFO:teuthology.orchestra.run.smithi114.stdout: "use_gmt_hitset": true, 2024-09-21T12:24:33.804 INFO:teuthology.orchestra.run.smithi114.stdout: "min_read_recency_for_promote": 0, 2024-09-21T12:24:33.804 INFO:teuthology.orchestra.run.smithi114.stdout: "min_write_recency_for_promote": 0, 2024-09-21T12:24:33.804 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T12:24:33.804 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_search_last_n": 0, 2024-09-21T12:24:33.804 INFO:teuthology.orchestra.run.smithi114.stdout: "grade_table": [], 2024-09-21T12:24:33.804 INFO:teuthology.orchestra.run.smithi114.stdout: "stripe_width": 0, 2024-09-21T12:24:33.805 INFO:teuthology.orchestra.run.smithi114.stdout: "expected_num_objects": 0, 2024-09-21T12:24:33.805 INFO:teuthology.orchestra.run.smithi114.stdout: "fast_read": false, 2024-09-21T12:24:33.805 INFO:teuthology.orchestra.run.smithi114.stdout: "options": {}, 2024-09-21T12:24:33.805 INFO:teuthology.orchestra.run.smithi114.stdout: "application_metadata": { 2024-09-21T12:24:33.805 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw": {} 2024-09-21T12:24:33.805 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.805 INFO:teuthology.orchestra.run.smithi114.stdout: "read_balance": { 2024-09-21T12:24:33.805 INFO:teuthology.orchestra.run.smithi114.stdout: "score_type": "Fair distribution", 2024-09-21T12:24:33.805 INFO:teuthology.orchestra.run.smithi114.stdout: "score_acting": 2.2400000095367432, 2024-09-21T12:24:33.805 INFO:teuthology.orchestra.run.smithi114.stdout: "score_stable": 2.2400000095367432, 2024-09-21T12:24:33.805 INFO:teuthology.orchestra.run.smithi114.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T12:24:33.805 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_acting": 1.9700000286102295, 2024-09-21T12:24:33.805 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_stable": 1.9700000286102295, 2024-09-21T12:24:33.805 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity_weighted": 1, 2024-09-21T12:24:33.805 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity": 1, 2024-09-21T12:24:33.806 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity_weighted": 1 2024-09-21T12:24:33.806 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.806 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.806 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.806 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 5, 2024-09-21T12:24:33.806 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": "default.rgw.control", 2024-09-21T12:24:33.806 INFO:teuthology.orchestra.run.smithi114.stdout: "create_time": "2024-09-21T12:20:17.314275+0000", 2024-09-21T12:24:33.806 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-09-21T12:24:33.806 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_names": "hashpspool", 2024-09-21T12:24:33.806 INFO:teuthology.orchestra.run.smithi114.stdout: "type": 1, 2024-09-21T12:24:33.806 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 3, 2024-09-21T12:24:33.806 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": 2, 2024-09-21T12:24:33.806 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_rule": 0, 2024-09-21T12:24:33.807 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_count": 0, 2024-09-21T12:24:33.807 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_target": 0, 2024-09-21T12:24:33.807 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T12:24:33.807 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T12:24:33.807 INFO:teuthology.orchestra.run.smithi114.stdout: "is_stretch_pool": false, 2024-09-21T12:24:33.807 INFO:teuthology.orchestra.run.smithi114.stdout: "object_hash": 2, 2024-09-21T12:24:33.807 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_mode": "off", 2024-09-21T12:24:33.807 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": 32, 2024-09-21T12:24:33.807 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num": 32, 2024-09-21T12:24:33.807 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num_target": 32, 2024-09-21T12:24:33.807 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_target": 32, 2024-09-21T12:24:33.807 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_pending": 32, 2024-09-21T12:24:33.807 INFO:teuthology.orchestra.run.smithi114.stdout: "last_pg_merge_meta": { 2024-09-21T12:24:33.807 INFO:teuthology.orchestra.run.smithi114.stdout: "source_pgid": "0.0", 2024-09-21T12:24:33.807 INFO:teuthology.orchestra.run.smithi114.stdout: "ready_epoch": 0, 2024-09-21T12:24:33.808 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_started": 0, 2024-09-21T12:24:33.808 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_clean": 0, 2024-09-21T12:24:33.808 INFO:teuthology.orchestra.run.smithi114.stdout: "source_version": "0'0", 2024-09-21T12:24:33.808 INFO:teuthology.orchestra.run.smithi114.stdout: "target_version": "0'0" 2024-09-21T12:24:33.808 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.808 INFO:teuthology.orchestra.run.smithi114.stdout: "last_change": "64", 2024-09-21T12:24:33.808 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend": "0", 2024-09-21T12:24:33.808 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T12:24:33.808 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T12:24:33.808 INFO:teuthology.orchestra.run.smithi114.stdout: "auid": 0, 2024-09-21T12:24:33.808 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_mode": "selfmanaged", 2024-09-21T12:24:33.808 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_seq": 0, 2024-09-21T12:24:33.808 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_epoch": 0, 2024-09-21T12:24:33.808 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_snaps": [], 2024-09-21T12:24:33.809 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps": "[]", 2024-09-21T12:24:33.809 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_bytes": 0, 2024-09-21T12:24:33.809 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_objects": 0, 2024-09-21T12:24:33.809 INFO:teuthology.orchestra.run.smithi114.stdout: "tiers": [], 2024-09-21T12:24:33.809 INFO:teuthology.orchestra.run.smithi114.stdout: "tier_of": -1, 2024-09-21T12:24:33.809 INFO:teuthology.orchestra.run.smithi114.stdout: "read_tier": -1, 2024-09-21T12:24:33.809 INFO:teuthology.orchestra.run.smithi114.stdout: "write_tier": -1, 2024-09-21T12:24:33.809 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_mode": "none", 2024-09-21T12:24:33.809 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_bytes": 0, 2024-09-21T12:24:33.809 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_objects": 0, 2024-09-21T12:24:33.809 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T12:24:33.809 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T12:24:33.809 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T12:24:33.809 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_flush_age": 0, 2024-09-21T12:24:33.809 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_evict_age": 0, 2024-09-21T12:24:33.810 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profile": "", 2024-09-21T12:24:33.810 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_params": { 2024-09-21T12:24:33.810 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "none" 2024-09-21T12:24:33.810 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.810 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_period": 0, 2024-09-21T12:24:33.810 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_count": 0, 2024-09-21T12:24:33.810 INFO:teuthology.orchestra.run.smithi114.stdout: "use_gmt_hitset": true, 2024-09-21T12:24:33.810 INFO:teuthology.orchestra.run.smithi114.stdout: "min_read_recency_for_promote": 0, 2024-09-21T12:24:33.810 INFO:teuthology.orchestra.run.smithi114.stdout: "min_write_recency_for_promote": 0, 2024-09-21T12:24:33.810 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T12:24:33.810 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_search_last_n": 0, 2024-09-21T12:24:33.810 INFO:teuthology.orchestra.run.smithi114.stdout: "grade_table": [], 2024-09-21T12:24:33.810 INFO:teuthology.orchestra.run.smithi114.stdout: "stripe_width": 0, 2024-09-21T12:24:33.810 INFO:teuthology.orchestra.run.smithi114.stdout: "expected_num_objects": 0, 2024-09-21T12:24:33.811 INFO:teuthology.orchestra.run.smithi114.stdout: "fast_read": false, 2024-09-21T12:24:33.811 INFO:teuthology.orchestra.run.smithi114.stdout: "options": {}, 2024-09-21T12:24:33.811 INFO:teuthology.orchestra.run.smithi114.stdout: "application_metadata": { 2024-09-21T12:24:33.811 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw": {} 2024-09-21T12:24:33.811 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.811 INFO:teuthology.orchestra.run.smithi114.stdout: "read_balance": { 2024-09-21T12:24:33.811 INFO:teuthology.orchestra.run.smithi114.stdout: "score_type": "Fair distribution", 2024-09-21T12:24:33.811 INFO:teuthology.orchestra.run.smithi114.stdout: "score_acting": 1.4900000095367432, 2024-09-21T12:24:33.811 INFO:teuthology.orchestra.run.smithi114.stdout: "score_stable": 1.4900000095367432, 2024-09-21T12:24:33.811 INFO:teuthology.orchestra.run.smithi114.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T12:24:33.812 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_acting": 1.309999942779541, 2024-09-21T12:24:33.812 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_stable": 1.309999942779541, 2024-09-21T12:24:33.812 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity_weighted": 1, 2024-09-21T12:24:33.812 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity": 1, 2024-09-21T12:24:33.812 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity_weighted": 1 2024-09-21T12:24:33.812 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.813 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.813 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.813 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 6, 2024-09-21T12:24:33.813 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": "default.rgw.meta", 2024-09-21T12:24:33.813 INFO:teuthology.orchestra.run.smithi114.stdout: "create_time": "2024-09-21T12:20:19.370218+0000", 2024-09-21T12:24:33.813 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-09-21T12:24:33.813 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_names": "hashpspool", 2024-09-21T12:24:33.813 INFO:teuthology.orchestra.run.smithi114.stdout: "type": 1, 2024-09-21T12:24:33.813 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 3, 2024-09-21T12:24:33.813 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": 2, 2024-09-21T12:24:33.813 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_rule": 0, 2024-09-21T12:24:33.813 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_count": 0, 2024-09-21T12:24:33.813 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_target": 0, 2024-09-21T12:24:33.813 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T12:24:33.814 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T12:24:33.814 INFO:teuthology.orchestra.run.smithi114.stdout: "is_stretch_pool": false, 2024-09-21T12:24:33.814 INFO:teuthology.orchestra.run.smithi114.stdout: "object_hash": 2, 2024-09-21T12:24:33.814 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_mode": "off", 2024-09-21T12:24:33.814 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": 32, 2024-09-21T12:24:33.814 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num": 32, 2024-09-21T12:24:33.814 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num_target": 32, 2024-09-21T12:24:33.814 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_target": 32, 2024-09-21T12:24:33.814 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_pending": 32, 2024-09-21T12:24:33.814 INFO:teuthology.orchestra.run.smithi114.stdout: "last_pg_merge_meta": { 2024-09-21T12:24:33.814 INFO:teuthology.orchestra.run.smithi114.stdout: "source_pgid": "0.0", 2024-09-21T12:24:33.814 INFO:teuthology.orchestra.run.smithi114.stdout: "ready_epoch": 0, 2024-09-21T12:24:33.814 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_started": 0, 2024-09-21T12:24:33.815 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_clean": 0, 2024-09-21T12:24:33.815 INFO:teuthology.orchestra.run.smithi114.stdout: "source_version": "0'0", 2024-09-21T12:24:33.815 INFO:teuthology.orchestra.run.smithi114.stdout: "target_version": "0'0" 2024-09-21T12:24:33.815 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.815 INFO:teuthology.orchestra.run.smithi114.stdout: "last_change": "66", 2024-09-21T12:24:33.815 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend": "0", 2024-09-21T12:24:33.815 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T12:24:33.815 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T12:24:33.815 INFO:teuthology.orchestra.run.smithi114.stdout: "auid": 0, 2024-09-21T12:24:33.815 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_mode": "selfmanaged", 2024-09-21T12:24:33.815 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_seq": 0, 2024-09-21T12:24:33.815 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_epoch": 0, 2024-09-21T12:24:33.815 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_snaps": [], 2024-09-21T12:24:33.815 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps": "[]", 2024-09-21T12:24:33.816 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_bytes": 0, 2024-09-21T12:24:33.816 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_objects": 0, 2024-09-21T12:24:33.816 INFO:teuthology.orchestra.run.smithi114.stdout: "tiers": [], 2024-09-21T12:24:33.816 INFO:teuthology.orchestra.run.smithi114.stdout: "tier_of": -1, 2024-09-21T12:24:33.816 INFO:teuthology.orchestra.run.smithi114.stdout: "read_tier": -1, 2024-09-21T12:24:33.816 INFO:teuthology.orchestra.run.smithi114.stdout: "write_tier": -1, 2024-09-21T12:24:33.816 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_mode": "none", 2024-09-21T12:24:33.816 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_bytes": 0, 2024-09-21T12:24:33.816 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_objects": 0, 2024-09-21T12:24:33.816 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T12:24:33.816 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T12:24:33.816 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T12:24:33.816 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_flush_age": 0, 2024-09-21T12:24:33.816 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_evict_age": 0, 2024-09-21T12:24:33.817 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profile": "", 2024-09-21T12:24:33.817 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_params": { 2024-09-21T12:24:33.817 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "none" 2024-09-21T12:24:33.817 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.817 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_period": 0, 2024-09-21T12:24:33.817 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_count": 0, 2024-09-21T12:24:33.817 INFO:teuthology.orchestra.run.smithi114.stdout: "use_gmt_hitset": true, 2024-09-21T12:24:33.817 INFO:teuthology.orchestra.run.smithi114.stdout: "min_read_recency_for_promote": 0, 2024-09-21T12:24:33.817 INFO:teuthology.orchestra.run.smithi114.stdout: "min_write_recency_for_promote": 0, 2024-09-21T12:24:33.817 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T12:24:33.817 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_search_last_n": 0, 2024-09-21T12:24:33.817 INFO:teuthology.orchestra.run.smithi114.stdout: "grade_table": [], 2024-09-21T12:24:33.817 INFO:teuthology.orchestra.run.smithi114.stdout: "stripe_width": 0, 2024-09-21T12:24:33.817 INFO:teuthology.orchestra.run.smithi114.stdout: "expected_num_objects": 0, 2024-09-21T12:24:33.818 INFO:teuthology.orchestra.run.smithi114.stdout: "fast_read": false, 2024-09-21T12:24:33.818 INFO:teuthology.orchestra.run.smithi114.stdout: "options": { 2024-09-21T12:24:33.818 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_bias": 4 2024-09-21T12:24:33.818 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.818 INFO:teuthology.orchestra.run.smithi114.stdout: "application_metadata": { 2024-09-21T12:24:33.818 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw": {} 2024-09-21T12:24:33.818 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.818 INFO:teuthology.orchestra.run.smithi114.stdout: "read_balance": { 2024-09-21T12:24:33.818 INFO:teuthology.orchestra.run.smithi114.stdout: "score_type": "Fair distribution", 2024-09-21T12:24:33.818 INFO:teuthology.orchestra.run.smithi114.stdout: "score_acting": 1.9900000095367432, 2024-09-21T12:24:33.818 INFO:teuthology.orchestra.run.smithi114.stdout: "score_stable": 1.9900000095367432, 2024-09-21T12:24:33.818 INFO:teuthology.orchestra.run.smithi114.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T12:24:33.818 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_acting": 1.75, 2024-09-21T12:24:33.818 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_stable": 1.75, 2024-09-21T12:24:33.819 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity_weighted": 1, 2024-09-21T12:24:33.819 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity": 1, 2024-09-21T12:24:33.819 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity_weighted": 1 2024-09-21T12:24:33.819 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.819 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.819 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.819 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 7, 2024-09-21T12:24:33.819 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": "unique_pool_0", 2024-09-21T12:24:33.819 INFO:teuthology.orchestra.run.smithi114.stdout: "create_time": "2024-09-21T12:21:23.302481+0000", 2024-09-21T12:24:33.819 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-09-21T12:24:33.819 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_names": "hashpspool", 2024-09-21T12:24:33.819 INFO:teuthology.orchestra.run.smithi114.stdout: "type": 1, 2024-09-21T12:24:33.819 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 3, 2024-09-21T12:24:33.819 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": 2, 2024-09-21T12:24:33.819 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_rule": 0, 2024-09-21T12:24:33.820 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_count": 0, 2024-09-21T12:24:33.820 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_target": 0, 2024-09-21T12:24:33.820 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T12:24:33.820 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T12:24:33.820 INFO:teuthology.orchestra.run.smithi114.stdout: "is_stretch_pool": false, 2024-09-21T12:24:33.820 INFO:teuthology.orchestra.run.smithi114.stdout: "object_hash": 2, 2024-09-21T12:24:33.820 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_mode": "off", 2024-09-21T12:24:33.820 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": 16, 2024-09-21T12:24:33.820 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num": 16, 2024-09-21T12:24:33.820 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num_target": 16, 2024-09-21T12:24:33.820 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_target": 16, 2024-09-21T12:24:33.820 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_pending": 16, 2024-09-21T12:24:33.820 INFO:teuthology.orchestra.run.smithi114.stdout: "last_pg_merge_meta": { 2024-09-21T12:24:33.820 INFO:teuthology.orchestra.run.smithi114.stdout: "source_pgid": "0.0", 2024-09-21T12:24:33.821 INFO:teuthology.orchestra.run.smithi114.stdout: "ready_epoch": 0, 2024-09-21T12:24:33.821 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_started": 0, 2024-09-21T12:24:33.821 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_clean": 0, 2024-09-21T12:24:33.821 INFO:teuthology.orchestra.run.smithi114.stdout: "source_version": "0'0", 2024-09-21T12:24:33.821 INFO:teuthology.orchestra.run.smithi114.stdout: "target_version": "0'0" 2024-09-21T12:24:33.821 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.821 INFO:teuthology.orchestra.run.smithi114.stdout: "last_change": "88", 2024-09-21T12:24:33.821 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend": "0", 2024-09-21T12:24:33.821 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T12:24:33.821 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T12:24:33.821 INFO:teuthology.orchestra.run.smithi114.stdout: "auid": 0, 2024-09-21T12:24:33.821 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_mode": "selfmanaged", 2024-09-21T12:24:33.821 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_seq": 0, 2024-09-21T12:24:33.821 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_epoch": 0, 2024-09-21T12:24:33.821 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_snaps": [], 2024-09-21T12:24:33.822 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps": "[]", 2024-09-21T12:24:33.822 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_bytes": 0, 2024-09-21T12:24:33.822 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_objects": 0, 2024-09-21T12:24:33.822 INFO:teuthology.orchestra.run.smithi114.stdout: "tiers": [], 2024-09-21T12:24:33.822 INFO:teuthology.orchestra.run.smithi114.stdout: "tier_of": -1, 2024-09-21T12:24:33.822 INFO:teuthology.orchestra.run.smithi114.stdout: "read_tier": -1, 2024-09-21T12:24:33.822 INFO:teuthology.orchestra.run.smithi114.stdout: "write_tier": -1, 2024-09-21T12:24:33.822 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_mode": "none", 2024-09-21T12:24:33.822 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_bytes": 0, 2024-09-21T12:24:33.822 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_objects": 0, 2024-09-21T12:24:33.822 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T12:24:33.822 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T12:24:33.822 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T12:24:33.822 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_flush_age": 0, 2024-09-21T12:24:33.823 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_evict_age": 0, 2024-09-21T12:24:33.823 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profile": "", 2024-09-21T12:24:33.823 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_params": { 2024-09-21T12:24:33.823 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "none" 2024-09-21T12:24:33.823 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.823 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_period": 0, 2024-09-21T12:24:33.823 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_count": 0, 2024-09-21T12:24:33.823 INFO:teuthology.orchestra.run.smithi114.stdout: "use_gmt_hitset": true, 2024-09-21T12:24:33.824 INFO:teuthology.orchestra.run.smithi114.stdout: "min_read_recency_for_promote": 0, 2024-09-21T12:24:33.824 INFO:teuthology.orchestra.run.smithi114.stdout: "min_write_recency_for_promote": 0, 2024-09-21T12:24:33.824 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T12:24:33.824 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_search_last_n": 0, 2024-09-21T12:24:33.824 INFO:teuthology.orchestra.run.smithi114.stdout: "grade_table": [], 2024-09-21T12:24:33.824 INFO:teuthology.orchestra.run.smithi114.stdout: "stripe_width": 0, 2024-09-21T12:24:33.824 INFO:teuthology.orchestra.run.smithi114.stdout: "expected_num_objects": 0, 2024-09-21T12:24:33.824 INFO:teuthology.orchestra.run.smithi114.stdout: "fast_read": false, 2024-09-21T12:24:33.825 INFO:teuthology.orchestra.run.smithi114.stdout: "options": {}, 2024-09-21T12:24:33.825 INFO:teuthology.orchestra.run.smithi114.stdout: "application_metadata": { 2024-09-21T12:24:33.825 INFO:teuthology.orchestra.run.smithi114.stdout: "rados": {} 2024-09-21T12:24:33.825 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.825 INFO:teuthology.orchestra.run.smithi114.stdout: "read_balance": { 2024-09-21T12:24:33.825 INFO:teuthology.orchestra.run.smithi114.stdout: "score_type": "Fair distribution", 2024-09-21T12:24:33.825 INFO:teuthology.orchestra.run.smithi114.stdout: "score_acting": 1.9900000095367432, 2024-09-21T12:24:33.825 INFO:teuthology.orchestra.run.smithi114.stdout: "score_stable": 1.9900000095367432, 2024-09-21T12:24:33.825 INFO:teuthology.orchestra.run.smithi114.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T12:24:33.825 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_acting": 1.75, 2024-09-21T12:24:33.825 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_stable": 1.75, 2024-09-21T12:24:33.825 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity_weighted": 1, 2024-09-21T12:24:33.825 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity": 1, 2024-09-21T12:24:33.826 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity_weighted": 1 2024-09-21T12:24:33.826 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.826 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.826 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:24:33.826 INFO:teuthology.orchestra.run.smithi114.stdout: "osds": [ 2024-09-21T12:24:33.826 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.826 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 0, 2024-09-21T12:24:33.826 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "5715e521-c7dc-41dd-a075-5c96a88fa9b3", 2024-09-21T12:24:33.826 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:24:33.826 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 1, 2024-09-21T12:24:33.826 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 1, 2024-09-21T12:24:33.826 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 1, 2024-09-21T12:24:33.826 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:24:33.826 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:24:33.827 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 8, 2024-09-21T12:24:33.827 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 120, 2024-09-21T12:24:33.827 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:24:33.827 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:24:33.827 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:24:33.827 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.827 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.827 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.827 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6802", 2024-09-21T12:24:33.828 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:24:33.828 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.828 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.828 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.828 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6803", 2024-09-21T12:24:33.828 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:24:33.828 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.829 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.829 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.829 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:24:33.829 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.829 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.829 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.829 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6804", 2024-09-21T12:24:33.829 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:24:33.830 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.830 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.830 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.830 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6805", 2024-09-21T12:24:33.830 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:24:33.830 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.830 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.830 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.830 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:24:33.831 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.831 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.831 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.831 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6808", 2024-09-21T12:24:33.831 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:24:33.831 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.831 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.831 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.831 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6809", 2024-09-21T12:24:33.831 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:24:33.832 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.832 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.832 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.832 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:24:33.832 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.832 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.832 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.832 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6806", 2024-09-21T12:24:33.832 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:24:33.832 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.832 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.832 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.832 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6807", 2024-09-21T12:24:33.832 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:24:33.833 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.833 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.833 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.833 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6803/3555869816", 2024-09-21T12:24:33.833 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.114:6805/3555869816", 2024-09-21T12:24:33.833 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.114:6809/3555869816", 2024-09-21T12:24:33.833 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.114:6807/3555869816", 2024-09-21T12:24:33.833 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:24:33.833 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:24:33.833 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:24:33.833 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.833 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.833 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.833 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 1, 2024-09-21T12:24:33.834 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "1fe160a7-5972-4e3e-99ce-15e2311ba4c5", 2024-09-21T12:24:33.834 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:24:33.834 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 1, 2024-09-21T12:24:33.834 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 1, 2024-09-21T12:24:33.834 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 1, 2024-09-21T12:24:33.834 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:24:33.834 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:24:33.834 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 13, 2024-09-21T12:24:33.834 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 120, 2024-09-21T12:24:33.834 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:24:33.834 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:24:33.834 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:24:33.835 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.835 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.835 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.835 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6810", 2024-09-21T12:24:33.835 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:24:33.835 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.835 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.835 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.835 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6811", 2024-09-21T12:24:33.835 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:24:33.835 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.835 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.835 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.835 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:24:33.836 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.836 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.836 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.836 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6812", 2024-09-21T12:24:33.836 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:24:33.836 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.836 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.836 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.836 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6813", 2024-09-21T12:24:33.836 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:24:33.836 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.836 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.836 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.836 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:24:33.837 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.837 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.837 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.837 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6816", 2024-09-21T12:24:33.837 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:24:33.837 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.837 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.837 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.837 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6817", 2024-09-21T12:24:33.837 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:24:33.837 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.837 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.837 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.838 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:24:33.838 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.838 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.838 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.838 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6814", 2024-09-21T12:24:33.838 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:24:33.838 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.838 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.838 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.838 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6815", 2024-09-21T12:24:33.838 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:24:33.838 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.838 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.838 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.839 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6811/2744663863", 2024-09-21T12:24:33.839 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.114:6813/2744663863", 2024-09-21T12:24:33.839 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.114:6817/2744663863", 2024-09-21T12:24:33.839 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.114:6815/2744663863", 2024-09-21T12:24:33.839 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:24:33.839 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:24:33.839 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:24:33.839 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.839 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.839 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.839 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 2, 2024-09-21T12:24:33.839 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "6a63581f-7603-49ae-9ab3-b37e7c91ab91", 2024-09-21T12:24:33.839 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:24:33.839 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 1, 2024-09-21T12:24:33.839 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 1, 2024-09-21T12:24:33.840 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 1, 2024-09-21T12:24:33.840 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:24:33.840 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:24:33.840 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 18, 2024-09-21T12:24:33.840 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 140, 2024-09-21T12:24:33.840 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:24:33.840 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:24:33.840 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:24:33.840 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.840 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.840 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.840 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6818", 2024-09-21T12:24:33.840 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:24:33.840 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.841 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.841 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.841 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6819", 2024-09-21T12:24:33.841 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:24:33.841 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.841 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.841 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.841 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:24:33.841 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.841 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.841 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.841 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6820", 2024-09-21T12:24:33.841 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:24:33.841 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.842 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.842 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.842 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6821", 2024-09-21T12:24:33.842 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:24:33.842 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.842 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.842 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.842 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:24:33.842 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.842 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.842 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.842 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6824", 2024-09-21T12:24:33.842 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:24:33.842 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.843 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.843 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.843 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6825", 2024-09-21T12:24:33.843 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:24:33.843 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.843 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.843 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.843 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:24:33.843 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.843 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.843 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.843 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6822", 2024-09-21T12:24:33.843 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:24:33.844 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.844 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.844 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.844 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6823", 2024-09-21T12:24:33.844 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:24:33.844 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.844 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.844 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.844 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6819/3740045173", 2024-09-21T12:24:33.844 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.114:6821/3740045173", 2024-09-21T12:24:33.844 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.114:6825/3740045173", 2024-09-21T12:24:33.844 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.114:6823/3740045173", 2024-09-21T12:24:33.844 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:24:33.845 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:24:33.845 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:24:33.845 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.845 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.845 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.845 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 3, 2024-09-21T12:24:33.845 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "0ad68cdb-ae5c-4090-8a8d-b67c1177ae42", 2024-09-21T12:24:33.845 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:24:33.845 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 1, 2024-09-21T12:24:33.845 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 1, 2024-09-21T12:24:33.845 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 1, 2024-09-21T12:24:33.845 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:24:33.845 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:24:33.845 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 26, 2024-09-21T12:24:33.845 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 121, 2024-09-21T12:24:33.846 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:24:33.846 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:24:33.846 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:24:33.846 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.846 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.846 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.846 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6826", 2024-09-21T12:24:33.846 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:24:33.846 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.846 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.846 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.846 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6827", 2024-09-21T12:24:33.846 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:24:33.847 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.847 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.847 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.847 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:24:33.847 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.847 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.847 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.847 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6828", 2024-09-21T12:24:33.847 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:24:33.847 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.847 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.847 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.847 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6829", 2024-09-21T12:24:33.847 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:24:33.847 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.848 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.848 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.848 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:24:33.848 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.848 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.848 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.848 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6832", 2024-09-21T12:24:33.848 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:24:33.848 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.848 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.848 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.848 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6833", 2024-09-21T12:24:33.848 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:24:33.848 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.849 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.849 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.849 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:24:33.849 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.849 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.849 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.849 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6830", 2024-09-21T12:24:33.849 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:24:33.849 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.849 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.849 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.849 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6831", 2024-09-21T12:24:33.849 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:24:33.849 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.850 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.850 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.850 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6827/1312590318", 2024-09-21T12:24:33.850 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.114:6829/1312590318", 2024-09-21T12:24:33.850 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.114:6833/1312590318", 2024-09-21T12:24:33.850 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.114:6831/1312590318", 2024-09-21T12:24:33.850 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:24:33.850 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:24:33.850 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:24:33.850 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.850 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.850 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.850 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 4, 2024-09-21T12:24:33.851 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "26bb8a84-e41d-4425-8491-2dea9764b7cf", 2024-09-21T12:24:33.851 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:24:33.851 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 0, 2024-09-21T12:24:33.851 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 0, 2024-09-21T12:24:33.851 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 1, 2024-09-21T12:24:33.851 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:24:33.851 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:24:33.851 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 32, 2024-09-21T12:24:33.851 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 111, 2024-09-21T12:24:33.851 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:24:33.851 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:24:33.851 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:24:33.851 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.851 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.852 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.852 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6800", 2024-09-21T12:24:33.852 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:24:33.852 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.852 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.852 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.852 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6801", 2024-09-21T12:24:33.852 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:24:33.852 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.852 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.852 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.852 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:24:33.852 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.852 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.852 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.853 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6802", 2024-09-21T12:24:33.853 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:24:33.853 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.853 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.853 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.853 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6803", 2024-09-21T12:24:33.853 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:24:33.853 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.853 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.853 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.853 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:24:33.853 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.853 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.853 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.854 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6806", 2024-09-21T12:24:33.854 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:24:33.854 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.854 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.854 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.854 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6807", 2024-09-21T12:24:33.854 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:24:33.854 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.854 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.854 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.854 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:24:33.854 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.854 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.854 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.855 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6804", 2024-09-21T12:24:33.855 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:24:33.855 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.855 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.855 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.855 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6805", 2024-09-21T12:24:33.855 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:24:33.855 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.855 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.855 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.855 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.183:6801/1391644836", 2024-09-21T12:24:33.855 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.183:6803/1391644836", 2024-09-21T12:24:33.855 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.183:6807/1391644836", 2024-09-21T12:24:33.855 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.183:6805/1391644836", 2024-09-21T12:24:33.855 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:24:33.856 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:24:33.856 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:24:33.856 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.856 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.856 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.856 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 5, 2024-09-21T12:24:33.856 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "33f96871-0869-4281-82b9-bd518987c551", 2024-09-21T12:24:33.856 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:24:33.856 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 1, 2024-09-21T12:24:33.856 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 1, 2024-09-21T12:24:33.856 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 1, 2024-09-21T12:24:33.856 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:24:33.856 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:24:33.856 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 39, 2024-09-21T12:24:33.856 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 140, 2024-09-21T12:24:33.857 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:24:33.857 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:24:33.857 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:24:33.857 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.857 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.857 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.857 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6808", 2024-09-21T12:24:33.857 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:24:33.857 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.857 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.857 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.857 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6809", 2024-09-21T12:24:33.857 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:24:33.857 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.858 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.858 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.858 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:24:33.858 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.858 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.858 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.858 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6810", 2024-09-21T12:24:33.858 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:24:33.858 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.858 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.858 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.858 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6811", 2024-09-21T12:24:33.858 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:24:33.858 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.859 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.859 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.859 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:24:33.859 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.859 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.859 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.859 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6814", 2024-09-21T12:24:33.859 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:24:33.859 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.859 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.859 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.859 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6815", 2024-09-21T12:24:33.859 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:24:33.859 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.859 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.860 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.860 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:24:33.860 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.860 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.860 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.860 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6812", 2024-09-21T12:24:33.860 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:24:33.860 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.860 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.860 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.860 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6813", 2024-09-21T12:24:33.860 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:24:33.860 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.860 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.860 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.861 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.183:6809/3187643473", 2024-09-21T12:24:33.861 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.183:6811/3187643473", 2024-09-21T12:24:33.861 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.183:6815/3187643473", 2024-09-21T12:24:33.861 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.183:6813/3187643473", 2024-09-21T12:24:33.861 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:24:33.861 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:24:33.861 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:24:33.861 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.861 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.861 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.861 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 6, 2024-09-21T12:24:33.861 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "e09c72b8-ad5a-4779-a874-b149fce6ed70", 2024-09-21T12:24:33.861 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:24:33.861 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 1, 2024-09-21T12:24:33.862 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 1, 2024-09-21T12:24:33.862 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 1, 2024-09-21T12:24:33.862 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:24:33.862 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:24:33.862 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 47, 2024-09-21T12:24:33.862 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 140, 2024-09-21T12:24:33.862 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:24:33.862 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:24:33.862 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:24:33.862 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.862 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.862 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.862 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6816", 2024-09-21T12:24:33.862 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:24:33.862 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.863 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.863 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.863 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6817", 2024-09-21T12:24:33.863 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:24:33.863 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.863 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.863 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.863 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:24:33.863 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.863 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.863 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.863 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6818", 2024-09-21T12:24:33.863 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:24:33.863 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.863 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.864 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.864 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6819", 2024-09-21T12:24:33.864 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:24:33.864 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.864 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.864 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.864 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:24:33.864 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.864 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.864 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.864 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6822", 2024-09-21T12:24:33.864 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:24:33.864 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.864 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.864 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.865 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6823", 2024-09-21T12:24:33.865 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:24:33.865 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.865 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.865 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.865 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:24:33.865 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.865 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.865 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.865 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6820", 2024-09-21T12:24:33.865 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:24:33.865 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.865 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.865 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.865 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6821", 2024-09-21T12:24:33.866 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:24:33.866 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.866 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.866 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.866 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.183:6817/3025199631", 2024-09-21T12:24:33.866 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.183:6819/3025199631", 2024-09-21T12:24:33.866 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.183:6823/3025199631", 2024-09-21T12:24:33.866 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.183:6821/3025199631", 2024-09-21T12:24:33.866 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:24:33.866 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:24:33.866 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:24:33.866 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.866 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.866 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.867 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 7, 2024-09-21T12:24:33.867 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "1a24194d-1af6-485f-8856-98844a02dd9d", 2024-09-21T12:24:33.867 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:24:33.867 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 1, 2024-09-21T12:24:33.867 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 1, 2024-09-21T12:24:33.867 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 1, 2024-09-21T12:24:33.867 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:24:33.867 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:24:33.867 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 54, 2024-09-21T12:24:33.867 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 129, 2024-09-21T12:24:33.867 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:24:33.867 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:24:33.867 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:24:33.867 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.868 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.868 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.868 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6824", 2024-09-21T12:24:33.868 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:24:33.868 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.868 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.868 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.868 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6825", 2024-09-21T12:24:33.868 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:24:33.868 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.868 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.868 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.868 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:24:33.868 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.869 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.869 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.869 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6826", 2024-09-21T12:24:33.869 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:24:33.869 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.869 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.869 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.869 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6827", 2024-09-21T12:24:33.869 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:24:33.869 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.869 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.869 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.869 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:24:33.869 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.870 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.870 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.870 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6830", 2024-09-21T12:24:33.870 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:24:33.870 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.870 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.870 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.870 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6831", 2024-09-21T12:24:33.870 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:24:33.870 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.870 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.870 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.870 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:24:33.870 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:24:33.871 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.871 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:24:33.871 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6828", 2024-09-21T12:24:33.871 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:24:33.871 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.871 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.871 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:24:33.871 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6829", 2024-09-21T12:24:33.871 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:24:33.871 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.871 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.871 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.871 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.183:6825/3203337442", 2024-09-21T12:24:33.872 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.183:6827/3203337442", 2024-09-21T12:24:33.872 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.183:6831/3203337442", 2024-09-21T12:24:33.872 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.183:6829/3203337442", 2024-09-21T12:24:33.872 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:24:33.872 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:24:33.872 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:24:33.872 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.872 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.872 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:24:33.872 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_xinfo": [ 2024-09-21T12:24:33.872 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.872 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 0, 2024-09-21T12:24:33.872 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:24:33.872 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:24:33.873 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:24:33.873 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:24:33.873 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 0, 2024-09-21T12:24:33.873 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:24:11.699923+0000", 2024-09-21T12:24:33.873 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:24:33.873 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.873 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.873 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 1, 2024-09-21T12:24:33.873 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:24:33.873 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:24:33.873 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:24:33.873 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:24:33.873 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 0, 2024-09-21T12:24:33.873 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:23:18.359836+0000", 2024-09-21T12:24:33.873 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:24:33.874 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.874 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.874 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 2, 2024-09-21T12:24:33.874 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:24:33.874 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:24:33.874 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:24:33.874 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:24:33.874 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 0, 2024-09-21T12:24:33.874 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:24:29.296644+0000", 2024-09-21T12:24:33.874 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:24:33.874 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.874 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.874 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 3, 2024-09-21T12:24:33.874 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:24:33.875 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:24:33.875 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:24:33.875 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:24:33.875 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 0, 2024-09-21T12:24:33.875 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:24:19.107690+0000", 2024-09-21T12:24:33.875 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:24:33.875 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.875 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.875 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 4, 2024-09-21T12:24:33.875 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:24:33.875 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:24:33.875 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:24:33.875 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:24:33.875 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 65536, 2024-09-21T12:24:33.876 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:24:07.051670+0000", 2024-09-21T12:24:33.876 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:24:33.876 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.876 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.876 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 5, 2024-09-21T12:24:33.876 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:24:33.876 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:24:33.876 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:24:33.876 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:24:33.876 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 0, 2024-09-21T12:24:33.876 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:24:19.752788+0000", 2024-09-21T12:24:33.876 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:24:33.876 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.876 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.877 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 6, 2024-09-21T12:24:33.877 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:24:33.877 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:24:33.877 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:24:33.877 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:24:33.877 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 0, 2024-09-21T12:24:33.877 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:23:30.539958+0000", 2024-09-21T12:24:33.877 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:24:33.877 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.877 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.877 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 7, 2024-09-21T12:24:33.877 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:24:33.877 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:24:33.877 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:24:33.877 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:24:33.878 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 0, 2024-09-21T12:24:33.878 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:23:39.055575+0000", 2024-09-21T12:24:33.878 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:24:33.878 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.878 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:24:33.878 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_upmap": [ 2024-09-21T12:24:33.878 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.878 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "5.e", 2024-09-21T12:24:33.878 INFO:teuthology.orchestra.run.smithi114.stdout: "osds": [ 2024-09-21T12:24:33.878 INFO:teuthology.orchestra.run.smithi114.stdout: 7, 2024-09-21T12:24:33.878 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-09-21T12:24:33.878 INFO:teuthology.orchestra.run.smithi114.stdout: 1 2024-09-21T12:24:33.878 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.878 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.879 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:24:33.879 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_upmap_items": [ 2024-09-21T12:24:33.879 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.879 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "4.c", 2024-09-21T12:24:33.879 INFO:teuthology.orchestra.run.smithi114.stdout: "mappings": [ 2024-09-21T12:24:33.879 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.879 INFO:teuthology.orchestra.run.smithi114.stdout: "from": 3, 2024-09-21T12:24:33.879 INFO:teuthology.orchestra.run.smithi114.stdout: "to": 2 2024-09-21T12:24:33.879 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.879 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.879 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.879 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.879 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "4.d", 2024-09-21T12:24:33.879 INFO:teuthology.orchestra.run.smithi114.stdout: "mappings": [ 2024-09-21T12:24:33.879 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.880 INFO:teuthology.orchestra.run.smithi114.stdout: "from": 1, 2024-09-21T12:24:33.880 INFO:teuthology.orchestra.run.smithi114.stdout: "to": 5 2024-09-21T12:24:33.880 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.880 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.880 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.880 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.880 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "4.e", 2024-09-21T12:24:33.880 INFO:teuthology.orchestra.run.smithi114.stdout: "mappings": [ 2024-09-21T12:24:33.880 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.880 INFO:teuthology.orchestra.run.smithi114.stdout: "from": 1, 2024-09-21T12:24:33.880 INFO:teuthology.orchestra.run.smithi114.stdout: "to": 5 2024-09-21T12:24:33.880 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.880 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.880 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.881 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.881 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "4.f", 2024-09-21T12:24:33.881 INFO:teuthology.orchestra.run.smithi114.stdout: "mappings": [ 2024-09-21T12:24:33.881 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.881 INFO:teuthology.orchestra.run.smithi114.stdout: "from": 1, 2024-09-21T12:24:33.881 INFO:teuthology.orchestra.run.smithi114.stdout: "to": 2 2024-09-21T12:24:33.881 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.881 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.881 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.881 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.881 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "4.15", 2024-09-21T12:24:33.881 INFO:teuthology.orchestra.run.smithi114.stdout: "mappings": [ 2024-09-21T12:24:33.881 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:24:33.881 INFO:teuthology.orchestra.run.smithi114.stdout: "from": 3, 2024-09-21T12:24:33.881 INFO:teuthology.orchestra.run.smithi114.stdout: "to": 2 2024-09-21T12:24:33.882 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.882 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:24:33.882 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.882 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:24:33.882 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_upmap_primaries": [], 2024-09-21T12:24:33.882 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_temp": [], 2024-09-21T12:24:33.882 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_temp": [], 2024-09-21T12:24:33.882 INFO:teuthology.orchestra.run.smithi114.stdout: "blocklist": { 2024-09-21T12:24:33.882 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:6801/237768393": "2024-09-22T12:20:45.528856+0000", 2024-09-21T12:24:33.882 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:6800/237768393": "2024-09-22T12:20:45.528856+0000", 2024-09-21T12:24:33.882 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/898990943": "2024-09-22T12:14:53.998653+0000", 2024-09-21T12:24:33.882 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/3866305956": "2024-09-22T12:20:45.528856+0000", 2024-09-21T12:24:33.882 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/3598782088": "2024-09-22T12:20:45.528856+0000", 2024-09-21T12:24:33.882 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/2620151541": "2024-09-22T12:15:20.045217+0000", 2024-09-21T12:24:33.882 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:6800/1725148595": "2024-09-22T12:14:53.998653+0000", 2024-09-21T12:24:33.883 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/442404805": "2024-09-22T12:14:53.998653+0000", 2024-09-21T12:24:33.883 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:6801/1725148595": "2024-09-22T12:14:53.998653+0000", 2024-09-21T12:24:33.883 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/4173909019": "2024-09-22T12:20:45.528856+0000", 2024-09-21T12:24:33.883 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/2106467614": "2024-09-22T12:15:20.045217+0000", 2024-09-21T12:24:33.883 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/139803357": "2024-09-22T12:15:20.045217+0000", 2024-09-21T12:24:33.883 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:6801/770932721": "2024-09-22T12:15:20.045217+0000", 2024-09-21T12:24:33.883 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:6800/770932721": "2024-09-22T12:15:20.045217+0000", 2024-09-21T12:24:33.883 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/1457547294": "2024-09-22T12:20:45.528856+0000", 2024-09-21T12:24:33.883 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/3416259228": "2024-09-22T12:14:53.998653+0000", 2024-09-21T12:24:33.883 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/995404371": "2024-09-22T12:20:45.528856+0000" 2024-09-21T12:24:33.883 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.883 INFO:teuthology.orchestra.run.smithi114.stdout: "range_blocklist": {}, 2024-09-21T12:24:33.883 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profiles": { 2024-09-21T12:24:33.883 INFO:teuthology.orchestra.run.smithi114.stdout: "default": { 2024-09-21T12:24:33.883 INFO:teuthology.orchestra.run.smithi114.stdout: "crush-failure-domain": "osd", 2024-09-21T12:24:33.884 INFO:teuthology.orchestra.run.smithi114.stdout: "k": "2", 2024-09-21T12:24:33.884 INFO:teuthology.orchestra.run.smithi114.stdout: "m": "1", 2024-09-21T12:24:33.884 INFO:teuthology.orchestra.run.smithi114.stdout: "plugin": "jerasure", 2024-09-21T12:24:33.884 INFO:teuthology.orchestra.run.smithi114.stdout: "technique": "reed_sol_van" 2024-09-21T12:24:33.884 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.884 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:24:33.884 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps_queue": [], 2024-09-21T12:24:33.884 INFO:teuthology.orchestra.run.smithi114.stdout: "new_removed_snaps": [], 2024-09-21T12:24:33.884 INFO:teuthology.orchestra.run.smithi114.stdout: "new_purged_snaps": [], 2024-09-21T12:24:33.884 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_node_flags": {}, 2024-09-21T12:24:33.884 INFO:teuthology.orchestra.run.smithi114.stdout: "device_class_flags": {}, 2024-09-21T12:24:33.884 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": { 2024-09-21T12:24:33.884 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode_enabled": false, 2024-09-21T12:24:33.884 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_bucket_count": 0, 2024-09-21T12:24:33.884 INFO:teuthology.orchestra.run.smithi114.stdout: "degraded_stretch_mode": 0, 2024-09-21T12:24:33.885 INFO:teuthology.orchestra.run.smithi114.stdout: "recovering_stretch_mode": 0, 2024-09-21T12:24:33.885 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode_bucket": 0 2024-09-21T12:24:33.885 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:24:33.885 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:24:33.885 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:33.908 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:33.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:33.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:34.009 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:34.030 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:34.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:34.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:34.132 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:34.154 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:34.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:34.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:34.255 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:34.277 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:34.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:34.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:34.357 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T12:24:34.379 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:34.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:34.422 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:34.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:34.523 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:34.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:34 smithi114 ceph-mon[40867]: 3.16 scrub starts 2024-09-21T12:24:34.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:34 smithi114 ceph-mon[40867]: 3.16 scrub ok 2024-09-21T12:24:34.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:34 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2950254037' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T12:24:34.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:34 smithi114 ceph-mon[30324]: 3.16 scrub starts 2024-09-21T12:24:34.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:34 smithi114 ceph-mon[30324]: 3.16 scrub ok 2024-09-21T12:24:34.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:34 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2950254037' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T12:24:34.544 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:34.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:34.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:34.616 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:24:34.647 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:34.658 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:34 smithi183 ceph-mon[33277]: 3.16 scrub starts 2024-09-21T12:24:34.658 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:34 smithi183 ceph-mon[33277]: 3.16 scrub ok 2024-09-21T12:24:34.658 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:34 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2950254037' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T12:24:34.670 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:34.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:34.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:34.772 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:34.796 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:34.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:34.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:34.898 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:34.923 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:34.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:34.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:35.025 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:35.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:35.048 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:35.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:35.137 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:24:35.137 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-21T12:24:35.137 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-21T12:24:35.137 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-21T12:24:35.138 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:24:35.150 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:35.171 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:35.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:35.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:35.273 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:35.295 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:35.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:35.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:35.396 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:35.417 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:35.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:35.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:35.475 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:24:35.496 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:35 smithi183 ceph-mon[33277]: pgmap v196: 158 pgs: 158 active+clean; 132 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-09-21T12:24:35.497 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:35 smithi183 ceph-mon[33277]: 5.b scrub starts 2024-09-21T12:24:35.497 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:35 smithi183 ceph-mon[33277]: 5.b scrub ok 2024-09-21T12:24:35.497 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:35 smithi183 ceph-mon[33277]: 5.19 scrub starts 2024-09-21T12:24:35.497 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:35 smithi183 ceph-mon[33277]: 5.19 scrub ok 2024-09-21T12:24:35.497 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:35 smithi183 ceph-mon[33277]: 6.19 scrub starts 2024-09-21T12:24:35.497 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:35 smithi183 ceph-mon[33277]: 6.19 scrub ok 2024-09-21T12:24:35.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:35 smithi114 ceph-mon[40867]: pgmap v196: 158 pgs: 158 active+clean; 132 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-09-21T12:24:35.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:35 smithi114 ceph-mon[40867]: 5.b scrub starts 2024-09-21T12:24:35.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:35 smithi114 ceph-mon[40867]: 5.b scrub ok 2024-09-21T12:24:35.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:35 smithi114 ceph-mon[40867]: 5.19 scrub starts 2024-09-21T12:24:35.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:35 smithi114 ceph-mon[40867]: 5.19 scrub ok 2024-09-21T12:24:35.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:35 smithi114 ceph-mon[40867]: 6.19 scrub starts 2024-09-21T12:24:35.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:35 smithi114 ceph-mon[40867]: 6.19 scrub ok 2024-09-21T12:24:35.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:35 smithi114 ceph-mon[30324]: pgmap v196: 158 pgs: 158 active+clean; 132 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-09-21T12:24:35.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:35 smithi114 ceph-mon[30324]: 5.b scrub starts 2024-09-21T12:24:35.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:35 smithi114 ceph-mon[30324]: 5.b scrub ok 2024-09-21T12:24:35.501 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:35 smithi114 ceph-mon[30324]: 5.19 scrub starts 2024-09-21T12:24:35.501 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:35 smithi114 ceph-mon[30324]: 5.19 scrub ok 2024-09-21T12:24:35.501 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:35 smithi114 ceph-mon[30324]: 6.19 scrub starts 2024-09-21T12:24:35.501 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:35 smithi114 ceph-mon[30324]: 6.19 scrub ok 2024-09-21T12:24:35.519 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:35.542 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:35.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:35.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:35.644 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:35.666 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:35.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:35.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:35.768 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:35.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:35.791 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:35.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:35.892 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:35.914 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:35.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:35.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:35.942 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T12:24:36.016 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:36.047 INFO:tasks.thrashosds.thrasher:j is {'epoch': 154, 'fsid': 'e2ad975c-7812-11ef-baf6-efdc52797490', 'created': '2024-09-21T12:14:24.181227+0000', 'modified': '2024-09-21T12:24:29.930050+0000', 'last_up_change': '2024-09-21T12:20:03.702987+0000', 'last_in_change': '2024-09-21T12:22:49.868269+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T12:17:38.357647+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '136', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '106', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.1700000762939453, 'score_stable': 2.1700000762939453, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.909999966621399, 'raw_score_stable': 1.909999966621399, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-21T12:20:12.761046+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 61, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-21T12:20:13.713971+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-21T12:20:15.368765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-21T12:20:17.314275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-21T12:20:19.370218+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-21T12:21:23.302481+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '5715e521-c7dc-41dd-a075-5c96a88fa9b3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 120, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6802', 'nonce': 3555869816}, {'type': 'v1', 'addr': '172.21.15.114:6803', 'nonce': 3555869816}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6804', 'nonce': 3555869816}, {'type': 'v1', 'addr': '172.21.15.114:6805', 'nonce': 3555869816}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6808', 'nonce': 3555869816}, {'type': 'v1', 'addr': '172.21.15.114:6809', 'nonce': 3555869816}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6806', 'nonce': 3555869816}, {'type': 'v1', 'addr': '172.21.15.114:6807', 'nonce': 3555869816}]}, 'public_addr': '172.21.15.114:6803/3555869816', 'cluster_addr': '172.21.15.114:6805/3555869816', 'heartbeat_back_addr': '172.21.15.114:6809/3555869816', 'heartbeat_front_addr': '172.21.15.114:6807/3555869816', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '1fe160a7-5972-4e3e-99ce-15e2311ba4c5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 120, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6810', 'nonce': 2744663863}, {'type': 'v1', 'addr': '172.21.15.114:6811', 'nonce': 2744663863}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6812', 'nonce': 2744663863}, {'type': 'v1', 'addr': '172.21.15.114:6813', 'nonce': 2744663863}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6816', 'nonce': 2744663863}, {'type': 'v1', 'addr': '172.21.15.114:6817', 'nonce': 2744663863}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6814', 'nonce': 2744663863}, {'type': 'v1', 'addr': '172.21.15.114:6815', 'nonce': 2744663863}]}, 'public_addr': '172.21.15.114:6811/2744663863', 'cluster_addr': '172.21.15.114:6813/2744663863', 'heartbeat_back_addr': '172.21.15.114:6817/2744663863', 'heartbeat_front_addr': '172.21.15.114:6815/2744663863', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '6a63581f-7603-49ae-9ab3-b37e7c91ab91', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 140, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6818', 'nonce': 3740045173}, {'type': 'v1', 'addr': '172.21.15.114:6819', 'nonce': 3740045173}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6820', 'nonce': 3740045173}, {'type': 'v1', 'addr': '172.21.15.114:6821', 'nonce': 3740045173}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6824', 'nonce': 3740045173}, {'type': 'v1', 'addr': '172.21.15.114:6825', 'nonce': 3740045173}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6822', 'nonce': 3740045173}, {'type': 'v1', 'addr': '172.21.15.114:6823', 'nonce': 3740045173}]}, 'public_addr': '172.21.15.114:6819/3740045173', 'cluster_addr': '172.21.15.114:6821/3740045173', 'heartbeat_back_addr': '172.21.15.114:6825/3740045173', 'heartbeat_front_addr': '172.21.15.114:6823/3740045173', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '0ad68cdb-ae5c-4090-8a8d-b67c1177ae42', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 121, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6826', 'nonce': 1312590318}, {'type': 'v1', 'addr': '172.21.15.114:6827', 'nonce': 1312590318}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6828', 'nonce': 1312590318}, {'type': 'v1', 'addr': '172.21.15.114:6829', 'nonce': 1312590318}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6832', 'nonce': 1312590318}, {'type': 'v1', 'addr': '172.21.15.114:6833', 'nonce': 1312590318}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6830', 'nonce': 1312590318}, {'type': 'v1', 'addr': '172.21.15.114:6831', 'nonce': 1312590318}]}, 'public_addr': '172.21.15.114:6827/1312590318', 'cluster_addr': '172.21.15.114:6829/1312590318', 'heartbeat_back_addr': '172.21.15.114:6833/1312590318', 'heartbeat_front_addr': '172.21.15.114:6831/1312590318', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '26bb8a84-e41d-4425-8491-2dea9764b7cf', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 111, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6800', 'nonce': 1391644836}, {'type': 'v1', 'addr': '172.21.15.183:6801', 'nonce': 1391644836}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6802', 'nonce': 1391644836}, {'type': 'v1', 'addr': '172.21.15.183:6803', 'nonce': 1391644836}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6806', 'nonce': 1391644836}, {'type': 'v1', 'addr': '172.21.15.183:6807', 'nonce': 1391644836}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6804', 'nonce': 1391644836}, {'type': 'v1', 'addr': '172.21.15.183:6805', 'nonce': 1391644836}]}, 'public_addr': '172.21.15.183:6801/1391644836', 'cluster_addr': '172.21.15.183:6803/1391644836', 'heartbeat_back_addr': '172.21.15.183:6807/1391644836', 'heartbeat_front_addr': '172.21.15.183:6805/1391644836', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '33f96871-0869-4281-82b9-bd518987c551', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 140, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6808', 'nonce': 3187643473}, {'type': 'v1', 'addr': '172.21.15.183:6809', 'nonce': 3187643473}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6810', 'nonce': 3187643473}, {'type': 'v1', 'addr': '172.21.15.183:6811', 'nonce': 3187643473}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6814', 'nonce': 3187643473}, {'type': 'v1', 'addr': '172.21.15.183:6815', 'nonce': 3187643473}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6812', 'nonce': 3187643473}, {'type': 'v1', 'addr': '172.21.15.183:6813', 'nonce': 3187643473}]}, 'public_addr': '172.21.15.183:6809/3187643473', 'cluster_addr': '172.21.15.183:6811/3187643473', 'heartbeat_back_addr': '172.21.15.183:6815/3187643473', 'heartbeat_front_addr': '172.21.15.183:6813/3187643473', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e09c72b8-ad5a-4779-a874-b149fce6ed70', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 140, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6816', 'nonce': 3025199631}, {'type': 'v1', 'addr': '172.21.15.183:6817', 'nonce': 3025199631}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6818', 'nonce': 3025199631}, {'type': 'v1', 'addr': '172.21.15.183:6819', 'nonce': 3025199631}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6822', 'nonce': 3025199631}, {'type': 'v1', 'addr': '172.21.15.183:6823', 'nonce': 3025199631}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6820', 'nonce': 3025199631}, {'type': 'v1', 'addr': '172.21.15.183:6821', 'nonce': 3025199631}]}, 'public_addr': '172.21.15.183:6817/3025199631', 'cluster_addr': '172.21.15.183:6819/3025199631', 'heartbeat_back_addr': '172.21.15.183:6823/3025199631', 'heartbeat_front_addr': '172.21.15.183:6821/3025199631', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1a24194d-1af6-485f-8856-98844a02dd9d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 129, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6824', 'nonce': 3203337442}, {'type': 'v1', 'addr': '172.21.15.183:6825', 'nonce': 3203337442}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6826', 'nonce': 3203337442}, {'type': 'v1', 'addr': '172.21.15.183:6827', 'nonce': 3203337442}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6830', 'nonce': 3203337442}, {'type': 'v1', 'addr': '172.21.15.183:6831', 'nonce': 3203337442}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6828', 'nonce': 3203337442}, {'type': 'v1', 'addr': '172.21.15.183:6829', 'nonce': 3203337442}]}, 'public_addr': '172.21.15.183:6825/3203337442', 'cluster_addr': '172.21.15.183:6827/3203337442', 'heartbeat_back_addr': '172.21.15.183:6831/3203337442', 'heartbeat_front_addr': '172.21.15.183:6829/3203337442', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T12:24:11.699923+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T12:23:18.359836+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T12:24:29.296644+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T12:24:19.107690+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-21T12:24:07.051670+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T12:24:19.752788+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T12:23:30.539958+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T12:23:39.055575+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '5.e', 'osds': [7, 2, 1]}], 'pg_upmap_items': [{'pgid': '4.c', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.e', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.15', 'mappings': [{'from': 3, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.114:6801/237768393': '2024-09-22T12:20:45.528856+0000', '172.21.15.114:6800/237768393': '2024-09-22T12:20:45.528856+0000', '172.21.15.114:0/898990943': '2024-09-22T12:14:53.998653+0000', '172.21.15.114:0/3866305956': '2024-09-22T12:20:45.528856+0000', '172.21.15.114:0/3598782088': '2024-09-22T12:20:45.528856+0000', '172.21.15.114:0/2620151541': '2024-09-22T12:15:20.045217+0000', '172.21.15.114:6800/1725148595': '2024-09-22T12:14:53.998653+0000', '172.21.15.114:0/442404805': '2024-09-22T12:14:53.998653+0000', '172.21.15.114:6801/1725148595': '2024-09-22T12:14:53.998653+0000', '172.21.15.114:0/4173909019': '2024-09-22T12:20:45.528856+0000', '172.21.15.114:0/2106467614': '2024-09-22T12:15:20.045217+0000', '172.21.15.114:0/139803357': '2024-09-22T12:15:20.045217+0000', '172.21.15.114:6801/770932721': '2024-09-22T12:15:20.045217+0000', '172.21.15.114:6800/770932721': '2024-09-22T12:15:20.045217+0000', '172.21.15.114:0/1457547294': '2024-09-22T12:20:45.528856+0000', '172.21.15.114:0/3416259228': '2024-09-22T12:14:53.998653+0000', '172.21.15.114:0/995404371': '2024-09-22T12:20:45.528856+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T12:24:36.047 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 5.e 2024-09-21T12:24:36.047 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd rm-pg-upmap 5.e 2024-09-21T12:24:36.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:36.048 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:36.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:36.084 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:36.149 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:36.170 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:36.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:36.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:36.232 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:24:36.247 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-21T12:24:36.272 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:36.292 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:36.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:36.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:36.313 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:36 smithi114 ceph-mon[40867]: 6.4 scrub starts 2024-09-21T12:24:36.314 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:36 smithi114 ceph-mon[40867]: 6.4 scrub ok 2024-09-21T12:24:36.314 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:36 smithi114 ceph-mon[40867]: pgmap v197: 158 pgs: 158 active+clean; 132 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-09-21T12:24:36.314 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:36 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/503304042' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:24:36.314 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:36 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:24:36.314 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:36 smithi114 ceph-mon[30324]: 6.4 scrub starts 2024-09-21T12:24:36.314 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:36 smithi114 ceph-mon[30324]: 6.4 scrub ok 2024-09-21T12:24:36.314 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:36 smithi114 ceph-mon[30324]: pgmap v197: 158 pgs: 158 active+clean; 132 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-09-21T12:24:36.314 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:36 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/503304042' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:24:36.314 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:36 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:24:36.394 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:36.417 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:36.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:36.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:36.518 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:36.539 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:36.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:36.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:36.557 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:36 smithi183 ceph-mon[33277]: 6.4 scrub starts 2024-09-21T12:24:36.557 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:36 smithi183 ceph-mon[33277]: 6.4 scrub ok 2024-09-21T12:24:36.557 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:36 smithi183 ceph-mon[33277]: pgmap v197: 158 pgs: 158 active+clean; 132 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-09-21T12:24:36.557 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:36 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/503304042' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:24:36.558 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:36 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:24:36.641 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:36.662 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:36.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:36.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:36.764 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:36.785 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:36.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:36.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:36.876 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:24:36 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:24:36.887 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:36.902 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:24:36.902 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-21T12:24:36.902 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-21T12:24:36.903 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:24:36.903 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.903 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14478.0:1119 5.1f 5:f95f44c2:::notify.0:head [watch ping cookie 94620848892160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-21T12:24:36.903 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:40.364371+0000", 2024-09-21T12:24:36.903 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 176.535012322, 2024-09-21T12:24:36.903 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78371777300000001, 2024-09-21T12:24:36.903 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:36.903 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:36.903 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:24:36.903 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:24:36.903 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14478", 2024-09-21T12:24:36.903 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:24:36.904 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 1119 2024-09-21T12:24:36.904 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.904 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:36.904 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.904 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:36.904 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364371+0000", 2024-09-21T12:24:36.904 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:36.904 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.904 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.904 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:36.904 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364371+0000", 2024-09-21T12:24:36.904 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:36.904 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.905 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.905 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:36.905 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364372+0000", 2024-09-21T12:24:36.905 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6199999999999997e-07 2024-09-21T12:24:36.905 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.905 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.905 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:36.905 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364376+0000", 2024-09-21T12:24:36.905 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8099999999999999e-06 2024-09-21T12:24:36.905 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.905 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.905 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:36.905 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364384+0000", 2024-09-21T12:24:36.906 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2870000000000004e-06 2024-09-21T12:24:36.906 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.906 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.906 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:36.906 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364393+0000", 2024-09-21T12:24:36.906 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5820000000000002e-06 2024-09-21T12:24:36.906 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.906 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.906 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:36.906 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364557+0000", 2024-09-21T12:24:36.906 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016425399999999999 2024-09-21T12:24:36.906 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.906 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.906 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:24:36.907 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364573+0000", 2024-09-21T12:24:36.907 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6286999999999998e-05 2024-09-21T12:24:36.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.907 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:36.907 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.144878+0000", 2024-09-21T12:24:36.907 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78030514600000001 2024-09-21T12:24:36.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.907 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for degraded object", 2024-09-21T12:24:36.907 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.145417+0000", 2024-09-21T12:24:36.907 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00053866500000000002 2024-09-21T12:24:36.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.908 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:36.908 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.147869+0000", 2024-09-21T12:24:36.908 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024518779999999998 2024-09-21T12:24:36.908 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.908 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.908 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:36.908 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.147939+0000", 2024-09-21T12:24:36.908 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0574000000000006e-05 2024-09-21T12:24:36.908 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.908 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.908 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:36.908 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.148089+0000", 2024-09-21T12:24:36.908 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014952800000000001 2024-09-21T12:24:36.908 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:36.909 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:36.909 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:36.909 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.909 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.909 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14478.0:1120 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch ping cookie 94620852177152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-21T12:24:36.909 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:40.364419+0000", 2024-09-21T12:24:36.909 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 176.53496464599999, 2024-09-21T12:24:36.909 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78472206099999997, 2024-09-21T12:24:36.909 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:36.909 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:36.909 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:24:36.909 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:24:36.909 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14478", 2024-09-21T12:24:36.909 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:24:36.910 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 1120 2024-09-21T12:24:36.910 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.910 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:36.910 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.910 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:36.910 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364419+0000", 2024-09-21T12:24:36.910 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:36.910 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.910 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.910 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:36.910 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364419+0000", 2024-09-21T12:24:36.910 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:36.911 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.911 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.911 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:36.911 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364419+0000", 2024-09-21T12:24:36.911 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5400000000000002e-07 2024-09-21T12:24:36.911 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.911 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.911 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:36.911 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364426+0000", 2024-09-21T12:24:36.911 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8650000000000003e-06 2024-09-21T12:24:36.911 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.911 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.911 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:36.911 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364434+0000", 2024-09-21T12:24:36.912 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2309999999999996e-06 2024-09-21T12:24:36.912 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.912 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.912 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:36.912 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364441+0000", 2024-09-21T12:24:36.912 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0929999999999999e-06 2024-09-21T12:24:36.912 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.912 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.912 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:36.912 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364520+0000", 2024-09-21T12:24:36.912 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9487999999999995e-05 2024-09-21T12:24:36.912 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.912 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.912 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:24:36.913 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364543+0000", 2024-09-21T12:24:36.913 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.296e-05 2024-09-21T12:24:36.913 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.913 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.913 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:36.913 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.147074+0000", 2024-09-21T12:24:36.913 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78253067600000004 2024-09-21T12:24:36.913 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.913 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.913 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for degraded object", 2024-09-21T12:24:36.913 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.147381+0000", 2024-09-21T12:24:36.913 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00030698200000000002 2024-09-21T12:24:36.913 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.913 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.914 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:36.914 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.148992+0000", 2024-09-21T12:24:36.914 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001611267 2024-09-21T12:24:36.914 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.914 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.914 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:36.914 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.149064+0000", 2024-09-21T12:24:36.914 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2058999999999996e-05 2024-09-21T12:24:36.914 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.914 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.914 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:36.914 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.149141+0000", 2024-09-21T12:24:36.914 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6985999999999998e-05 2024-09-21T12:24:36.914 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:36.915 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:36.915 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:36.915 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.915 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.915 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14478.0:1124 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94620848892160 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:24:36.915 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:40.364832+0000", 2024-09-21T12:24:36.915 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 176.53455159500001, 2024-09-21T12:24:36.915 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78486121600000003, 2024-09-21T12:24:36.915 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:36.915 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:36.915 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:24:36.915 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:24:36.915 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14478", 2024-09-21T12:24:36.916 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:24:36.916 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 1124 2024-09-21T12:24:36.916 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.916 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:36.916 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.916 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:36.916 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364832+0000", 2024-09-21T12:24:36.916 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:36.916 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.916 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.916 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:36.916 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364832+0000", 2024-09-21T12:24:36.916 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:36.916 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.917 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.918 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:36.918 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364833+0000", 2024-09-21T12:24:36.918 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3900000000000004e-07 2024-09-21T12:24:36.918 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.918 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.918 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:36.918 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364838+0000", 2024-09-21T12:24:36.918 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1909999999999999e-06 2024-09-21T12:24:36.918 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.918 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.919 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:36.919 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364846+0000", 2024-09-21T12:24:36.919 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7000000000000008e-06 2024-09-21T12:24:36.919 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.919 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.919 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:36.919 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364850+0000", 2024-09-21T12:24:36.919 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.921e-06 2024-09-21T12:24:36.919 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.919 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.919 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:36.919 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364911+0000", 2024-09-21T12:24:36.919 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0636e-05 2024-09-21T12:24:36.920 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.920 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.920 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:24:36.920 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364922+0000", 2024-09-21T12:24:36.920 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0964e-05 2024-09-21T12:24:36.920 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.920 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.920 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:36.920 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.145447+0000", 2024-09-21T12:24:36.920 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78052487599999998 2024-09-21T12:24:36.920 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.920 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.920 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for degraded object", 2024-09-21T12:24:36.920 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.145493+0000", 2024-09-21T12:24:36.921 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5745000000000003e-05 2024-09-21T12:24:36.921 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.921 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.921 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:36.921 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.148124+0000", 2024-09-21T12:24:36.921 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0026310719999999999 2024-09-21T12:24:36.921 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.921 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.921 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:36.921 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.148191+0000", 2024-09-21T12:24:36.921 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7447999999999995e-05 2024-09-21T12:24:36.921 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.921 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.921 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:36.922 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.149693+0000", 2024-09-21T12:24:36.922 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001501924 2024-09-21T12:24:36.922 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:36.922 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:36.922 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:36.922 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.922 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.922 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14478.0:1125 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94620852177152 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:24:36.922 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:40.364865+0000", 2024-09-21T12:24:36.922 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 176.53451808200001, 2024-09-21T12:24:36.922 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78551680400000001, 2024-09-21T12:24:36.922 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:36.922 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:36.923 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:24:36.923 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:24:36.923 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14478", 2024-09-21T12:24:36.923 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:24:36.923 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 1125 2024-09-21T12:24:36.923 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.923 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:36.923 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.923 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:36.923 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364865+0000", 2024-09-21T12:24:36.923 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:36.923 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.923 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.924 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:36.924 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364865+0000", 2024-09-21T12:24:36.924 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:36.924 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.924 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.924 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:36.924 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364866+0000", 2024-09-21T12:24:36.924 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6499999999999999e-07 2024-09-21T12:24:36.924 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.924 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.924 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:36.924 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364868+0000", 2024-09-21T12:24:36.925 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.39e-06 2024-09-21T12:24:36.925 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.925 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.925 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:36.925 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364882+0000", 2024-09-21T12:24:36.925 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3890000000000001e-05 2024-09-21T12:24:36.925 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.925 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.925 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:36.925 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364888+0000", 2024-09-21T12:24:36.925 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3940000000000004e-06 2024-09-21T12:24:36.925 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.926 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.926 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:36.926 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364953+0000", 2024-09-21T12:24:36.926 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5877000000000006e-05 2024-09-21T12:24:36.926 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.926 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.926 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:24:36.926 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364963+0000", 2024-09-21T12:24:36.926 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5349999999999993e-06 2024-09-21T12:24:36.926 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.926 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.926 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:36.926 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.147420+0000", 2024-09-21T12:24:36.927 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78245673400000004 2024-09-21T12:24:36.927 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.927 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.927 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for degraded object", 2024-09-21T12:24:36.927 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.147469+0000", 2024-09-21T12:24:36.927 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9699e-05 2024-09-21T12:24:36.927 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.927 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.927 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:36.927 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.149190+0000", 2024-09-21T12:24:36.927 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001720606 2024-09-21T12:24:36.927 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.927 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.927 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:36.928 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.149267+0000", 2024-09-21T12:24:36.928 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7194000000000007e-05 2024-09-21T12:24:36.928 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.928 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.928 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:36.928 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.150382+0000", 2024-09-21T12:24:36.928 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00111502 2024-09-21T12:24:36.928 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:36.928 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:36.928 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:36.928 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.928 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.929 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:24811 7.7 7:ec8774ff:::benchmark_data_smithi114_78432_object24810:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T12:24:36.929 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:42.445492+0000", 2024-09-21T12:24:36.929 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 174.45389115099999, 2024-09-21T12:24:36.929 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.145669727, 2024-09-21T12:24:36.929 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:36.929 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:36.929 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:36.929 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:24:36.929 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:24:36.929 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:36.929 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 24811 2024-09-21T12:24:36.929 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.929 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:36.929 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.930 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:36.930 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445492+0000", 2024-09-21T12:24:36.930 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:36.930 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.930 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.930 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:36.930 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445492+0000", 2024-09-21T12:24:36.930 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:36.930 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.930 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.930 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:36.931 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445494+0000", 2024-09-21T12:24:36.931 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5460000000000001e-06 2024-09-21T12:24:36.931 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.931 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.931 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:36.931 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445541+0000", 2024-09-21T12:24:36.931 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6687999999999998e-05 2024-09-21T12:24:36.931 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.931 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.931 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:36.931 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445553+0000", 2024-09-21T12:24:36.931 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2517999999999999e-05 2024-09-21T12:24:36.931 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.932 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.932 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:36.932 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445560+0000", 2024-09-21T12:24:36.932 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6579999999999996e-06 2024-09-21T12:24:36.932 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.932 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.932 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:36.932 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445625+0000", 2024-09-21T12:24:36.932 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5092000000000006e-05 2024-09-21T12:24:36.932 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.932 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.932 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:36.932 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445776+0000", 2024-09-21T12:24:36.932 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0001507 2024-09-21T12:24:36.933 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.933 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.933 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,3", 2024-09-21T12:24:36.933 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445923+0000", 2024-09-21T12:24:36.933 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014760199999999999 2024-09-21T12:24:36.933 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.933 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.933 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:24:36.933 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.446851+0000", 2024-09-21T12:24:36.933 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00092788099999999997 2024-09-21T12:24:36.933 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.933 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.933 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:36.933 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.590740+0000", 2024-09-21T12:24:36.934 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14388880500000001 2024-09-21T12:24:36.934 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.934 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.934 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:36.934 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.591050+0000", 2024-09-21T12:24:36.934 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000310217 2024-09-21T12:24:36.934 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.934 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.934 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:36.934 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.591109+0000", 2024-09-21T12:24:36.934 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9188000000000003e-05 2024-09-21T12:24:36.934 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.934 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.934 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:36.935 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.591162+0000", 2024-09-21T12:24:36.935 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2831999999999998e-05 2024-09-21T12:24:36.935 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:36.935 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:36.935 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:36.935 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.935 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.935 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:45498 7.7 7:e1336a67:::benchmark_data_smithi114_78432_object45497:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T12:24:36.935 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:59.293850+0000", 2024-09-21T12:24:36.935 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 157.60553322800001, 2024-09-21T12:24:36.935 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15681044, 2024-09-21T12:24:36.935 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:36.935 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:36.935 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:36.936 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:24:36.936 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:24:36.936 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:36.936 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 45498 2024-09-21T12:24:36.936 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.936 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:36.936 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.936 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:36.936 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293850+0000", 2024-09-21T12:24:36.936 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:36.936 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.936 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.936 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:36.936 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293850+0000", 2024-09-21T12:24:36.936 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:36.937 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.937 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.937 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:36.937 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293851+0000", 2024-09-21T12:24:36.937 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.096e-06 2024-09-21T12:24:36.937 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.937 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.937 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:36.937 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293907+0000", 2024-09-21T12:24:36.937 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5126e-05 2024-09-21T12:24:36.937 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.937 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.937 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:36.937 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293915+0000", 2024-09-21T12:24:36.937 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8419999999999994e-06 2024-09-21T12:24:36.937 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.938 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.938 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:36.938 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293920+0000", 2024-09-21T12:24:36.938 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5900000000000001e-06 2024-09-21T12:24:36.938 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.938 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.938 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:36.938 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293959+0000", 2024-09-21T12:24:36.938 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9447999999999999e-05 2024-09-21T12:24:36.938 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.938 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.938 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:36.938 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294060+0000", 2024-09-21T12:24:36.939 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100247 2024-09-21T12:24:36.939 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.939 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.939 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,3", 2024-09-21T12:24:36.939 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294145+0000", 2024-09-21T12:24:36.939 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5099999999999995e-05 2024-09-21T12:24:36.939 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.939 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.939 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:24:36.939 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.295566+0000", 2024-09-21T12:24:36.939 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014215429999999999 2024-09-21T12:24:36.939 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.939 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.939 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:36.939 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.449406+0000", 2024-09-21T12:24:36.940 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15383943999999999 2024-09-21T12:24:36.940 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.940 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.940 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:36.940 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.450563+0000", 2024-09-21T12:24:36.940 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00115749 2024-09-21T12:24:36.940 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.940 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.940 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:36.940 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.450621+0000", 2024-09-21T12:24:36.940 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8161000000000001e-05 2024-09-21T12:24:36.940 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.940 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.940 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:36.940 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.450661+0000", 2024-09-21T12:24:36.941 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9357000000000003e-05 2024-09-21T12:24:36.941 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:36.941 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:36.941 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:36.941 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.941 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.941 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:45514 7.c 7:3b027556:::benchmark_data_smithi114_78432_object45513:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T12:24:36.941 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:59.302539+0000", 2024-09-21T12:24:36.941 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 157.59684433499999, 2024-09-21T12:24:36.941 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.150159925, 2024-09-21T12:24:36.941 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:36.941 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:36.941 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:36.941 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:24:36.942 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:24:36.942 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:36.942 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 45514 2024-09-21T12:24:36.942 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.942 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:36.942 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.942 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:36.942 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302539+0000", 2024-09-21T12:24:36.942 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:36.942 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.942 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.942 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:36.942 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302539+0000", 2024-09-21T12:24:36.942 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:36.943 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.943 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.943 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:36.943 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302541+0000", 2024-09-21T12:24:36.943 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.376e-06 2024-09-21T12:24:36.943 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.943 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.943 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:36.943 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302592+0000", 2024-09-21T12:24:36.943 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1054999999999998e-05 2024-09-21T12:24:36.943 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.943 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.943 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:36.944 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302603+0000", 2024-09-21T12:24:36.944 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0883000000000001e-05 2024-09-21T12:24:36.944 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.944 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.944 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:36.944 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302608+0000", 2024-09-21T12:24:36.944 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5339999999999999e-06 2024-09-21T12:24:36.944 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.944 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.944 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:36.944 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302655+0000", 2024-09-21T12:24:36.944 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7125e-05 2024-09-21T12:24:36.944 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.944 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.944 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:36.944 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302784+0000", 2024-09-21T12:24:36.945 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012886700000000001 2024-09-21T12:24:36.945 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.945 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.945 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:24:36.945 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.303003+0000", 2024-09-21T12:24:36.945 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00021902099999999999 2024-09-21T12:24:36.945 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.945 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.945 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:24:36.945 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.304304+0000", 2024-09-21T12:24:36.945 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0013008060000000001 2024-09-21T12:24:36.945 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.945 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.945 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:36.945 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.304689+0000", 2024-09-21T12:24:36.946 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000385208 2024-09-21T12:24:36.946 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.946 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.946 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:36.946 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.452633+0000", 2024-09-21T12:24:36.946 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14794360000000001 2024-09-21T12:24:36.946 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.946 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.946 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:36.946 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.452669+0000", 2024-09-21T12:24:36.946 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6202000000000003e-05 2024-09-21T12:24:36.946 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.946 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.947 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:36.947 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.452699+0000", 2024-09-21T12:24:36.947 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0247999999999999e-05 2024-09-21T12:24:36.947 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:36.947 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:36.947 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:36.947 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.947 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.947 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:49226 7.d 7:bcebfbbf:::benchmark_data_smithi114_78432_object49225:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:24:36.947 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:04.569255+0000", 2024-09-21T12:24:36.947 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 152.33012834799999, 2024-09-21T12:24:36.947 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14786647, 2024-09-21T12:24:36.947 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:36.947 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:36.947 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:36.948 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:24:36.948 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:24:36.948 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:36.948 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 49226 2024-09-21T12:24:36.948 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.948 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:36.948 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.948 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:36.948 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569255+0000", 2024-09-21T12:24:36.948 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:36.948 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.948 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.948 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:36.949 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569255+0000", 2024-09-21T12:24:36.949 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:36.949 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.949 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.949 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:36.949 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569257+0000", 2024-09-21T12:24:36.949 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3480000000000001e-06 2024-09-21T12:24:36.949 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.949 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.949 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:36.949 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569332+0000", 2024-09-21T12:24:36.949 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4969000000000007e-05 2024-09-21T12:24:36.949 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.949 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.949 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:36.950 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569344+0000", 2024-09-21T12:24:36.950 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2700999999999999e-05 2024-09-21T12:24:36.950 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.950 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.950 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:36.950 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569351+0000", 2024-09-21T12:24:36.950 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0369999999999997e-06 2024-09-21T12:24:36.950 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.950 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.950 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:36.950 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569427+0000", 2024-09-21T12:24:36.950 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6206999999999997e-05 2024-09-21T12:24:36.950 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.950 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.951 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:36.951 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569517+0000", 2024-09-21T12:24:36.951 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9875000000000003e-05 2024-09-21T12:24:36.951 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.951 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.951 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1", 2024-09-21T12:24:36.951 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569601+0000", 2024-09-21T12:24:36.951 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3849000000000003e-05 2024-09-21T12:24:36.951 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.951 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.951 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:24:36.951 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.573956+0000", 2024-09-21T12:24:36.951 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0043544689999999997 2024-09-21T12:24:36.951 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.952 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.952 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:36.952 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.575786+0000", 2024-09-21T12:24:36.952 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0018307480000000001 2024-09-21T12:24:36.952 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.952 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.952 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:36.952 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.717016+0000", 2024-09-21T12:24:36.952 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14122955000000001 2024-09-21T12:24:36.952 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.952 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.952 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:36.952 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.717072+0000", 2024-09-21T12:24:36.952 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6063000000000002e-05 2024-09-21T12:24:36.953 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.953 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.953 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:36.953 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.717122+0000", 2024-09-21T12:24:36.953 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9654000000000003e-05 2024-09-21T12:24:36.953 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:36.953 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:36.953 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:36.953 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.953 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.953 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:49229 7.d 7:b095dac2:::benchmark_data_smithi114_78432_object49228:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:24:36.953 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:04.569388+0000", 2024-09-21T12:24:36.953 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 152.329995814, 2024-09-21T12:24:36.953 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.147855664, 2024-09-21T12:24:36.954 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:36.954 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:36.954 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:36.954 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:24:36.954 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:24:36.954 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:36.954 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 49229 2024-09-21T12:24:36.954 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.954 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:36.954 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.954 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:36.954 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569388+0000", 2024-09-21T12:24:36.954 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:36.954 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.955 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.955 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:36.955 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569388+0000", 2024-09-21T12:24:36.955 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:36.955 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.955 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.955 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:36.955 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569388+0000", 2024-09-21T12:24:36.955 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.37e-07 2024-09-21T12:24:36.955 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.955 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.955 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:36.955 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569462+0000", 2024-09-21T12:24:36.955 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3244000000000005e-05 2024-09-21T12:24:36.955 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.956 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.956 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:36.956 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569471+0000", 2024-09-21T12:24:36.956 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2499999999999995e-06 2024-09-21T12:24:36.956 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.956 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.956 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:36.956 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569477+0000", 2024-09-21T12:24:36.956 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.494e-06 2024-09-21T12:24:36.956 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.956 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.956 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:36.956 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.572570+0000", 2024-09-21T12:24:36.956 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0030923930000000001 2024-09-21T12:24:36.957 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.957 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.957 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:36.957 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.572664+0000", 2024-09-21T12:24:36.957 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4609000000000002e-05 2024-09-21T12:24:36.957 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.957 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.957 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1", 2024-09-21T12:24:36.957 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.572751+0000", 2024-09-21T12:24:36.957 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6725999999999996e-05 2024-09-21T12:24:36.957 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.957 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.957 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:24:36.957 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.574094+0000", 2024-09-21T12:24:36.957 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00134326 2024-09-21T12:24:36.958 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.958 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.958 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:36.958 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.576231+0000", 2024-09-21T12:24:36.958 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021369510000000002 2024-09-21T12:24:36.958 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.958 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.958 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:36.958 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.717166+0000", 2024-09-21T12:24:36.958 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14093446000000001 2024-09-21T12:24:36.958 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.958 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.958 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:36.958 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.717210+0000", 2024-09-21T12:24:36.959 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4239000000000001e-05 2024-09-21T12:24:36.959 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:36.960 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:36.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:36.961 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.961 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:36.961 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.717243+0000", 2024-09-21T12:24:36.961 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3401000000000001e-05 2024-09-21T12:24:36.961 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:36.961 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:36.961 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:36.962 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.962 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.962 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:49936 7.7 7:ee2fb7d7:::benchmark_data_smithi114_78432_object49935:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:24:36.962 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:05.207697+0000", 2024-09-21T12:24:36.962 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 151.69168694300001, 2024-09-21T12:24:36.962 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.180234797, 2024-09-21T12:24:36.962 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:36.962 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:36.962 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:36.962 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:24:36.962 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:24:36.962 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:36.962 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 49936 2024-09-21T12:24:36.963 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.963 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:36.963 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.963 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:36.963 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207697+0000", 2024-09-21T12:24:36.963 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:36.963 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.963 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.963 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:36.963 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207697+0000", 2024-09-21T12:24:36.963 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:36.963 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.963 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.963 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:36.964 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207698+0000", 2024-09-21T12:24:36.964 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6449999999999999e-06 2024-09-21T12:24:36.964 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.964 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.964 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:36.964 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207744+0000", 2024-09-21T12:24:36.964 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6171999999999998e-05 2024-09-21T12:24:36.964 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.964 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.964 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:36.964 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207755+0000", 2024-09-21T12:24:36.964 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0998999999999999e-05 2024-09-21T12:24:36.964 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.964 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.965 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:36.965 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207761+0000", 2024-09-21T12:24:36.965 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9789999999999998e-06 2024-09-21T12:24:36.965 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.965 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.965 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:36.965 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207822+0000", 2024-09-21T12:24:36.965 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0504000000000002e-05 2024-09-21T12:24:36.965 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.965 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.965 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:36.965 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207921+0000", 2024-09-21T12:24:36.965 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9524000000000005e-05 2024-09-21T12:24:36.965 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.965 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.966 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,3", 2024-09-21T12:24:36.966 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.208068+0000", 2024-09-21T12:24:36.966 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014648299999999999 2024-09-21T12:24:36.966 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.966 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.966 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:24:36.966 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.209637+0000", 2024-09-21T12:24:36.966 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001569555 2024-09-21T12:24:36.966 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.966 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.966 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:36.966 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.214355+0000", 2024-09-21T12:24:36.966 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0047178810000000002 2024-09-21T12:24:36.966 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.967 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.967 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:36.967 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.387868+0000", 2024-09-21T12:24:36.967 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.173512203 2024-09-21T12:24:36.967 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.967 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.967 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:36.967 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.387905+0000", 2024-09-21T12:24:36.967 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7678000000000001e-05 2024-09-21T12:24:36.967 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.967 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.967 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:36.967 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.387931+0000", 2024-09-21T12:24:36.968 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6174e-05 2024-09-21T12:24:36.968 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:36.968 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:36.968 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:36.968 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.968 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.968 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:49949 7.d 7:b8e0b394:::benchmark_data_smithi114_78432_object49948:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:24:36.968 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:05.215908+0000", 2024-09-21T12:24:36.968 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 151.68347522600001, 2024-09-21T12:24:36.968 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.17212050400000001, 2024-09-21T12:24:36.968 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:36.968 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:36.968 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:36.968 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:24:36.969 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:24:36.969 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:36.969 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 49949 2024-09-21T12:24:36.969 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.969 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:36.969 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.969 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:36.969 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215908+0000", 2024-09-21T12:24:36.969 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:36.969 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.969 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.969 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:36.969 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215908+0000", 2024-09-21T12:24:36.969 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:36.970 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.970 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.970 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:36.970 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215910+0000", 2024-09-21T12:24:36.970 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.389e-06 2024-09-21T12:24:36.970 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.970 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.970 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:36.970 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215966+0000", 2024-09-21T12:24:36.970 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6189e-05 2024-09-21T12:24:36.970 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.970 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.970 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:36.970 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215976+0000", 2024-09-21T12:24:36.971 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0115999999999999e-05 2024-09-21T12:24:36.971 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.971 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.971 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:36.971 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215983+0000", 2024-09-21T12:24:36.971 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9240000000000001e-06 2024-09-21T12:24:36.971 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.971 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.971 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:36.971 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.216032+0000", 2024-09-21T12:24:36.971 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9237000000000003e-05 2024-09-21T12:24:36.971 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.971 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.971 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:36.972 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.216109+0000", 2024-09-21T12:24:36.972 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6538999999999993e-05 2024-09-21T12:24:36.972 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.972 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.972 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1", 2024-09-21T12:24:36.972 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.216208+0000", 2024-09-21T12:24:36.972 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8979000000000002e-05 2024-09-21T12:24:36.972 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.972 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.972 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:24:36.972 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217649+0000", 2024-09-21T12:24:36.972 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014417500000000001 2024-09-21T12:24:36.972 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.972 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.973 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:36.973 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.219275+0000", 2024-09-21T12:24:36.973 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0016254749999999999 2024-09-21T12:24:36.973 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.973 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.973 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:36.973 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.387917+0000", 2024-09-21T12:24:36.973 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.16864237100000001 2024-09-21T12:24:36.973 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.973 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.973 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:36.973 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.387990+0000", 2024-09-21T12:24:36.973 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2249999999999994e-05 2024-09-21T12:24:36.973 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.973 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.974 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:36.974 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.388029+0000", 2024-09-21T12:24:36.974 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9285000000000001e-05 2024-09-21T12:24:36.974 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:36.974 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:36.974 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:36.974 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.974 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.974 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:49953 7.7 7:e1a3875f:::benchmark_data_smithi114_78432_object49952:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:24:36.974 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:05.217615+0000", 2024-09-21T12:24:36.974 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 151.68176830499999, 2024-09-21T12:24:36.974 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.17616359000000001, 2024-09-21T12:24:36.974 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:36.974 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:36.975 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:36.975 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:24:36.975 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:24:36.975 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:36.975 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 49953 2024-09-21T12:24:36.975 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.975 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:36.975 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.975 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:36.975 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217615+0000", 2024-09-21T12:24:36.975 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:36.975 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.975 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.975 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:36.976 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217615+0000", 2024-09-21T12:24:36.976 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:36.976 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.976 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.976 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:36.976 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217616+0000", 2024-09-21T12:24:36.976 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.209e-06 2024-09-21T12:24:36.976 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.976 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.976 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:36.976 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217687+0000", 2024-09-21T12:24:36.976 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0570999999999999e-05 2024-09-21T12:24:36.976 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.976 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.976 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:36.977 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217701+0000", 2024-09-21T12:24:36.977 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3828e-05 2024-09-21T12:24:36.977 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.977 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.977 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:36.977 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217709+0000", 2024-09-21T12:24:36.977 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3240000000000003e-06 2024-09-21T12:24:36.977 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.977 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.977 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:36.977 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217763+0000", 2024-09-21T12:24:36.977 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3804999999999997e-05 2024-09-21T12:24:36.977 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.978 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.978 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:36.978 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217854+0000", 2024-09-21T12:24:36.978 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0771000000000002e-05 2024-09-21T12:24:36.978 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.978 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.978 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,3", 2024-09-21T12:24:36.978 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217935+0000", 2024-09-21T12:24:36.978 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0951000000000005e-05 2024-09-21T12:24:36.978 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.978 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.978 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:24:36.978 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.219237+0000", 2024-09-21T12:24:36.978 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0013019679999999999 2024-09-21T12:24:36.979 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.979 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.979 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:36.979 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.220275+0000", 2024-09-21T12:24:36.979 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010378609999999999 2024-09-21T12:24:36.979 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.979 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.979 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:36.979 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.393564+0000", 2024-09-21T12:24:36.979 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.17328943899999999 2024-09-21T12:24:36.979 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.979 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.979 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:36.979 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.393734+0000", 2024-09-21T12:24:36.980 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017039999999999999 2024-09-21T12:24:36.980 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.980 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.980 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:36.980 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.393779+0000", 2024-09-21T12:24:36.980 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4462999999999998e-05 2024-09-21T12:24:36.980 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:36.980 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:36.980 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:36.980 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.980 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.980 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:49961 7.7 7:e82f0c80:::benchmark_data_smithi114_78432_object49960:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:24:36.980 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:05.226839+0000", 2024-09-21T12:24:36.980 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 151.67254430899999, 2024-09-21T12:24:36.981 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.170311145, 2024-09-21T12:24:36.981 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:36.981 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:36.981 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:36.981 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:24:36.981 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:24:36.981 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:36.981 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 49961 2024-09-21T12:24:36.981 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.981 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:36.981 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.981 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:36.981 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.226839+0000", 2024-09-21T12:24:36.981 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:36.981 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.982 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.982 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:36.982 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.226839+0000", 2024-09-21T12:24:36.982 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:36.982 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.982 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.982 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:36.982 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.226840+0000", 2024-09-21T12:24:36.982 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8400000000000003e-07 2024-09-21T12:24:36.982 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.982 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.982 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:36.982 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.226896+0000", 2024-09-21T12:24:36.982 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5517999999999999e-05 2024-09-21T12:24:36.983 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.983 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.983 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:36.983 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.226906+0000", 2024-09-21T12:24:36.983 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0589e-05 2024-09-21T12:24:36.983 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.983 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.983 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:36.983 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.226914+0000", 2024-09-21T12:24:36.983 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2779999999999996e-06 2024-09-21T12:24:36.983 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.983 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.983 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:36.983 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.226963+0000", 2024-09-21T12:24:36.984 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9021999999999999e-05 2024-09-21T12:24:36.984 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.984 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.984 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:36.984 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.227045+0000", 2024-09-21T12:24:36.984 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2800999999999995e-05 2024-09-21T12:24:36.984 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.984 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.984 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,3", 2024-09-21T12:24:36.984 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.227126+0000", 2024-09-21T12:24:36.984 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0467000000000001e-05 2024-09-21T12:24:36.984 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.984 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.984 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:24:36.985 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.228347+0000", 2024-09-21T12:24:36.985 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001220907 2024-09-21T12:24:36.985 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.985 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.985 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:36.985 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.393832+0000", 2024-09-21T12:24:36.985 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.165485038 2024-09-21T12:24:36.985 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.985 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.985 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:36.985 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.397086+0000", 2024-09-21T12:24:36.985 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0032546799999999998 2024-09-21T12:24:36.985 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.985 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.985 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:36.986 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.397121+0000", 2024-09-21T12:24:36.986 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4584e-05 2024-09-21T12:24:36.986 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.986 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.986 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:36.986 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.397150+0000", 2024-09-21T12:24:36.986 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9377e-05 2024-09-21T12:24:36.986 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:36.986 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:36.986 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:36.986 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.986 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.986 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:53122 7.d 7:b6d0f6a6:::benchmark_data_smithi114_78432_object53121:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:24:36.986 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:07.710882+0000", 2024-09-21T12:24:36.987 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 149.188501454, 2024-09-21T12:24:36.987 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.126757757, 2024-09-21T12:24:36.987 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:36.987 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:36.987 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:36.987 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:24:36.987 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:24:36.987 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:36.987 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 53122 2024-09-21T12:24:36.987 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.987 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:36.987 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.987 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:36.987 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.710882+0000", 2024-09-21T12:24:36.988 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:36.988 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.988 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.988 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:36.988 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.710882+0000", 2024-09-21T12:24:36.988 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:36.988 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.988 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.988 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:36.988 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.710883+0000", 2024-09-21T12:24:36.988 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8500000000000001e-07 2024-09-21T12:24:36.988 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.988 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.989 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:36.989 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.710929+0000", 2024-09-21T12:24:36.989 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5867e-05 2024-09-21T12:24:36.989 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.989 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.989 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:36.989 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.710937+0000", 2024-09-21T12:24:36.989 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.619e-06 2024-09-21T12:24:36.989 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.989 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.989 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:36.989 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.710945+0000", 2024-09-21T12:24:36.989 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6920000000000002e-06 2024-09-21T12:24:36.989 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.989 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.989 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:36.990 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.713008+0000", 2024-09-21T12:24:36.990 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002063442 2024-09-21T12:24:36.990 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.990 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.990 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:36.990 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.713116+0000", 2024-09-21T12:24:36.990 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000107886 2024-09-21T12:24:36.990 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.990 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.990 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1", 2024-09-21T12:24:36.990 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.713198+0000", 2024-09-21T12:24:36.990 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1842999999999996e-05 2024-09-21T12:24:36.990 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.990 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.991 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:24:36.991 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.714595+0000", 2024-09-21T12:24:36.991 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0013966779999999999 2024-09-21T12:24:36.991 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.991 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.991 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:36.991 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.716453+0000", 2024-09-21T12:24:36.991 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0018580230000000001 2024-09-21T12:24:36.991 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.991 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.991 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:36.991 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.837570+0000", 2024-09-21T12:24:36.991 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.121117483 2024-09-21T12:24:36.991 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.991 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.992 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:36.992 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.837606+0000", 2024-09-21T12:24:36.992 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5225000000000002e-05 2024-09-21T12:24:36.992 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.992 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.992 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:36.992 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:07.837640+0000", 2024-09-21T12:24:36.992 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4313999999999997e-05 2024-09-21T12:24:36.992 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:36.992 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:36.992 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:36.992 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.992 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.992 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:116822 7.c 7:3b47e54f:::benchmark_data_smithi114_78432_object12676:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T12:24:36.992 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:23:12.373717+0000", 2024-09-21T12:24:36.992 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 84.525666635999997, 2024-09-21T12:24:36.993 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15758128199999999, 2024-09-21T12:24:36.993 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:36.993 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:36.993 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:36.993 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:24:36.993 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:24:36.993 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:36.993 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 116822 2024-09-21T12:24:36.993 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.993 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:36.993 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.993 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:36.993 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.373717+0000", 2024-09-21T12:24:36.993 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:36.993 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.994 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.994 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:36.994 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.373717+0000", 2024-09-21T12:24:36.994 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:36.994 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.994 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.994 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:36.994 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.373718+0000", 2024-09-21T12:24:36.994 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2190000000000001e-06 2024-09-21T12:24:36.994 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.994 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.994 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:36.994 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.373723+0000", 2024-09-21T12:24:36.994 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7400000000000004e-06 2024-09-21T12:24:36.995 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.995 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.995 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:36.995 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.373732+0000", 2024-09-21T12:24:36.995 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8319999999999995e-06 2024-09-21T12:24:36.995 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.995 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.995 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:36.995 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.373737+0000", 2024-09-21T12:24:36.995 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2789999999999998e-06 2024-09-21T12:24:36.995 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.995 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.995 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:36.996 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.373777+0000", 2024-09-21T12:24:36.996 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0123999999999997e-05 2024-09-21T12:24:36.996 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.996 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.996 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:36.996 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.373854+0000", 2024-09-21T12:24:36.996 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6743999999999996e-05 2024-09-21T12:24:36.996 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.996 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.996 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:24:36.996 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.373916+0000", 2024-09-21T12:24:36.996 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2451000000000002e-05 2024-09-21T12:24:36.996 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.996 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.996 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:24:36.997 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.375743+0000", 2024-09-21T12:24:36.997 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001826338 2024-09-21T12:24:36.997 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.997 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.997 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:36.997 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.375885+0000", 2024-09-21T12:24:36.997 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014278699999999999 2024-09-21T12:24:36.997 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.997 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.997 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:36.997 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.531196+0000", 2024-09-21T12:24:36.997 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.155311 2024-09-21T12:24:36.997 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.998 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.998 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:36.998 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.531247+0000", 2024-09-21T12:24:36.998 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0476000000000003e-05 2024-09-21T12:24:36.998 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.998 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.998 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:36.998 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.531298+0000", 2024-09-21T12:24:36.998 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1292000000000003e-05 2024-09-21T12:24:36.998 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:36.998 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:36.998 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:36.998 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.998 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.999 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:116828 7.c 7:31e7a7e6:::benchmark_data_smithi114_78432_object12682:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T12:24:36.999 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:23:12.377844+0000", 2024-09-21T12:24:36.999 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 84.521539736999998, 2024-09-21T12:24:36.999 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15356805100000001, 2024-09-21T12:24:36.999 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:36.999 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:36.999 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:36.999 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:24:36.999 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:24:36.999 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:36.999 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 116828 2024-09-21T12:24:36.999 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:36.999 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:36.999 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:36.999 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:37.000 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.377844+0000", 2024-09-21T12:24:37.000 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:37.000 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.000 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.000 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:37.000 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.377844+0000", 2024-09-21T12:24:37.000 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:37.000 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.000 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.000 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:37.000 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.377845+0000", 2024-09-21T12:24:37.000 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3600000000000002e-07 2024-09-21T12:24:37.000 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.000 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.001 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:37.001 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.377849+0000", 2024-09-21T12:24:37.001 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.622e-06 2024-09-21T12:24:37.001 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.001 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.001 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:37.001 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.377862+0000", 2024-09-21T12:24:37.001 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2989999999999999e-05 2024-09-21T12:24:37.001 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.001 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.001 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:37.001 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.377870+0000", 2024-09-21T12:24:37.002 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5780000000000001e-06 2024-09-21T12:24:37.002 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.002 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.002 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:37.002 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.377917+0000", 2024-09-21T12:24:37.002 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7178999999999997e-05 2024-09-21T12:24:37.002 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.002 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.002 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:37.002 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.377988+0000", 2024-09-21T12:24:37.002 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0890999999999996e-05 2024-09-21T12:24:37.002 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.002 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.002 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:24:37.003 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.378052+0000", 2024-09-21T12:24:37.003 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4194000000000002e-05 2024-09-21T12:24:37.003 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.003 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.003 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:24:37.003 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.379417+0000", 2024-09-21T12:24:37.003 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001365184 2024-09-21T12:24:37.003 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.003 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.003 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:37.003 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.379801+0000", 2024-09-21T12:24:37.003 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000383483 2024-09-21T12:24:37.003 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.003 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.003 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:37.004 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.531334+0000", 2024-09-21T12:24:37.004 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15153333499999999 2024-09-21T12:24:37.004 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.004 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.004 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:37.004 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.531378+0000", 2024-09-21T12:24:37.004 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3557999999999999e-05 2024-09-21T12:24:37.004 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.004 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.004 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:37.004 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.531412+0000", 2024-09-21T12:24:37.004 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4100999999999998e-05 2024-09-21T12:24:37.004 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:37.004 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:37.005 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:37.005 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.005 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.005 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:150329 7.7 7:ebd08636:::benchmark_data_smithi114_78432_object46183:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T12:24:37.005 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:23:42.610086+0000", 2024-09-21T12:24:37.005 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 54.289297484000002, 2024-09-21T12:24:37.005 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13913827400000001, 2024-09-21T12:24:37.005 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:37.005 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:37.005 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:37.005 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:24:37.005 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:24:37.005 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:37.005 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 150329 2024-09-21T12:24:37.006 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.006 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:37.006 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.006 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:37.006 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.610086+0000", 2024-09-21T12:24:37.006 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:37.006 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.006 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.006 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:37.006 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.610086+0000", 2024-09-21T12:24:37.006 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:37.006 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.006 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.006 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:37.007 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.610087+0000", 2024-09-21T12:24:37.007 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9699999999999994e-07 2024-09-21T12:24:37.007 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.007 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.007 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:37.007 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.610091+0000", 2024-09-21T12:24:37.007 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8139999999999998e-06 2024-09-21T12:24:37.007 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.007 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.007 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:37.007 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.610100+0000", 2024-09-21T12:24:37.007 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6239999999999997e-06 2024-09-21T12:24:37.007 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.007 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.007 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:37.008 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.610106+0000", 2024-09-21T12:24:37.008 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6350000000000003e-06 2024-09-21T12:24:37.008 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.008 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.008 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:37.008 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.610146+0000", 2024-09-21T12:24:37.008 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0215e-05 2024-09-21T12:24:37.008 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.008 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.008 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:37.008 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.610231+0000", 2024-09-21T12:24:37.008 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4846999999999996e-05 2024-09-21T12:24:37.008 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.008 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.009 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,3", 2024-09-21T12:24:37.009 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.610303+0000", 2024-09-21T12:24:37.009 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1823999999999995e-05 2024-09-21T12:24:37.009 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.009 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.009 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:24:37.009 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.612121+0000", 2024-09-21T12:24:37.009 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00181836 2024-09-21T12:24:37.009 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.009 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.009 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:37.009 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.615984+0000", 2024-09-21T12:24:37.009 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0038623630000000002 2024-09-21T12:24:37.009 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.009 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.010 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:37.010 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.749109+0000", 2024-09-21T12:24:37.010 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.133125308 2024-09-21T12:24:37.010 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.010 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.010 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:37.010 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.749172+0000", 2024-09-21T12:24:37.010 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2738999999999997e-05 2024-09-21T12:24:37.010 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.010 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.010 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:37.010 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.749224+0000", 2024-09-21T12:24:37.010 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2547999999999999e-05 2024-09-21T12:24:37.011 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:37.011 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:37.011 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:37.011 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.011 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.011 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:150333 7.7 7:e8d51399:::benchmark_data_smithi114_78432_object46187:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T12:24:37.011 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:23:42.613878+0000", 2024-09-21T12:24:37.011 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 54.285505354999998, 2024-09-21T12:24:37.011 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13550337900000001, 2024-09-21T12:24:37.011 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:37.011 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:37.011 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:37.011 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:24:37.011 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:24:37.012 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:37.012 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 150333 2024-09-21T12:24:37.012 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.012 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:37.012 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.012 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:37.012 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.613878+0000", 2024-09-21T12:24:37.012 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:37.012 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.012 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.012 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:37.012 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.613878+0000", 2024-09-21T12:24:37.012 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:37.012 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.012 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.013 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:37.013 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.613879+0000", 2024-09-21T12:24:37.013 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.269e-06 2024-09-21T12:24:37.013 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.013 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.013 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:37.013 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.613884+0000", 2024-09-21T12:24:37.013 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8829999999999997e-06 2024-09-21T12:24:37.013 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.013 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.013 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:37.013 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.613895+0000", 2024-09-21T12:24:37.013 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0533000000000001e-05 2024-09-21T12:24:37.014 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.014 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.014 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:37.014 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.613902+0000", 2024-09-21T12:24:37.014 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7490000000000001e-06 2024-09-21T12:24:37.014 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.014 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.014 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:37.014 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.613952+0000", 2024-09-21T12:24:37.014 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0714000000000003e-05 2024-09-21T12:24:37.014 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.014 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.014 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:37.015 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.614062+0000", 2024-09-21T12:24:37.015 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000109759 2024-09-21T12:24:37.015 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.015 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.015 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,3", 2024-09-21T12:24:37.015 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.614186+0000", 2024-09-21T12:24:37.015 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012417799999999999 2024-09-21T12:24:37.015 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.015 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.015 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:24:37.015 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.616077+0000", 2024-09-21T12:24:37.015 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001890547 2024-09-21T12:24:37.015 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.015 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.016 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:37.016 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618473+0000", 2024-09-21T12:24:37.016 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002396329 2024-09-21T12:24:37.016 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.016 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.016 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:37.016 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.749296+0000", 2024-09-21T12:24:37.016 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13082292400000001 2024-09-21T12:24:37.016 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.016 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.016 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:37.016 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.749343+0000", 2024-09-21T12:24:37.016 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7210999999999999e-05 2024-09-21T12:24:37.016 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.017 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.017 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:37.017 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.749382+0000", 2024-09-21T12:24:37.017 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8282999999999999e-05 2024-09-21T12:24:37.017 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:37.017 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:37.017 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:37.017 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.017 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.017 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:150338 7.7 7:ef617dde:::benchmark_data_smithi114_78432_object46192:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T12:24:37.017 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:23:42.618434+0000", 2024-09-21T12:24:37.017 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 54.28094917, 2024-09-21T12:24:37.017 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.131039925, 2024-09-21T12:24:37.017 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:37.018 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:37.018 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:37.018 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:24:37.018 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:24:37.018 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:37.018 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 150338 2024-09-21T12:24:37.018 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.018 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:37.018 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.018 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:37.018 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618434+0000", 2024-09-21T12:24:37.018 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:37.018 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.018 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.019 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:37.019 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618434+0000", 2024-09-21T12:24:37.019 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:37.019 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.019 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.019 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:37.019 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618435+0000", 2024-09-21T12:24:37.019 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5700000000000001e-07 2024-09-21T12:24:37.019 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.019 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.019 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:37.019 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618438+0000", 2024-09-21T12:24:37.019 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.038e-06 2024-09-21T12:24:37.020 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.020 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.020 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:37.020 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618446+0000", 2024-09-21T12:24:37.020 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5920000000000003e-06 2024-09-21T12:24:37.020 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.020 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.020 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:37.020 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618452+0000", 2024-09-21T12:24:37.020 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6529999999999999e-06 2024-09-21T12:24:37.020 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.020 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.020 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:37.021 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618514+0000", 2024-09-21T12:24:37.021 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2787999999999996e-05 2024-09-21T12:24:37.021 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.021 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.021 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:37.021 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618630+0000", 2024-09-21T12:24:37.021 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000115535 2024-09-21T12:24:37.021 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.021 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.021 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,3", 2024-09-21T12:24:37.021 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618744+0000", 2024-09-21T12:24:37.021 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000114658 2024-09-21T12:24:37.021 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.021 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.022 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:24:37.022 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.620521+0000", 2024-09-21T12:24:37.022 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001776945 2024-09-21T12:24:37.022 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.022 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.022 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:37.022 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.749259+0000", 2024-09-21T12:24:37.022 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.12873791300000001 2024-09-21T12:24:37.022 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.022 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.022 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:37.022 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.749410+0000", 2024-09-21T12:24:37.022 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015015499999999999 2024-09-21T12:24:37.022 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.023 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.023 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:37.023 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.749441+0000", 2024-09-21T12:24:37.023 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1415000000000003e-05 2024-09-21T12:24:37.023 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.023 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.023 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:37.023 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.749474+0000", 2024-09-21T12:24:37.023 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3376e-05 2024-09-21T12:24:37.023 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:37.023 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:37.023 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:37.023 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.024 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.024 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:175890 7.d 7:b1ae2c83:::benchmark_data_smithi114_78432_object71744:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T12:24:37.024 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:24:05.016098+0000", 2024-09-21T12:24:37.024 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 31.883285368999999, 2024-09-21T12:24:37.024 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.12709987, 2024-09-21T12:24:37.024 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:24:37.024 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:24:37.024 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:24:37.024 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:24:37.024 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:24:37.024 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:24:37.024 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 175890 2024-09-21T12:24:37.024 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.024 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:24:37.025 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.025 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:24:37.025 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:05.016098+0000", 2024-09-21T12:24:37.025 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:37.025 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.025 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.025 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:24:37.025 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:05.016098+0000", 2024-09-21T12:24:37.025 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:24:37.025 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.025 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.025 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:24:37.025 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:05.016099+0000", 2024-09-21T12:24:37.026 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.195e-06 2024-09-21T12:24:37.026 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.026 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.026 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:24:37.026 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:05.016104+0000", 2024-09-21T12:24:37.026 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9350000000000002e-06 2024-09-21T12:24:37.026 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.026 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.026 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:24:37.026 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:05.016118+0000", 2024-09-21T12:24:37.026 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3788000000000001e-05 2024-09-21T12:24:37.026 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.026 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.026 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:24:37.027 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:05.016126+0000", 2024-09-21T12:24:37.027 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4410000000000004e-06 2024-09-21T12:24:37.027 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.027 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.027 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:24:37.027 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:05.017233+0000", 2024-09-21T12:24:37.027 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0011070469999999999 2024-09-21T12:24:37.027 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.027 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.027 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:24:37.027 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:05.017339+0000", 2024-09-21T12:24:37.027 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000106062 2024-09-21T12:24:37.027 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.028 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.028 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1", 2024-09-21T12:24:37.028 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:05.017467+0000", 2024-09-21T12:24:37.028 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000128508 2024-09-21T12:24:37.028 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.028 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.028 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:24:37.028 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:05.019478+0000", 2024-09-21T12:24:37.028 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002011274 2024-09-21T12:24:37.028 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.028 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.028 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:37.028 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:05.021369+0000", 2024-09-21T12:24:37.028 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001890114 2024-09-21T12:24:37.029 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.029 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.029 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:24:37.029 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:05.143110+0000", 2024-09-21T12:24:37.029 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.12174169 2024-09-21T12:24:37.029 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.029 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.029 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:24:37.029 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:05.143160+0000", 2024-09-21T12:24:37.029 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9917999999999999e-05 2024-09-21T12:24:37.029 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:24:37.029 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:24:37.029 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:24:37.029 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:24:05.143198+0000", 2024-09-21T12:24:37.030 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7898000000000002e-05 2024-09-21T12:24:37.030 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:37.030 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:37.030 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:37.030 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:24:37.030 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:24:37.030 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:24:37.061 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:37.082 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:37.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:37.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:37.111 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:24:37.111 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:24:37.119 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:24:37.119 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:24:37.137 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:24:37.137 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:24:37.155 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:24:37.155 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:24:37.170 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:37.173 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:24:37.174 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:24:37.184 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:37.191 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:24:37.191 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:24:37.205 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:37.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:37.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:37.211 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:24:37.212 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:24:37.228 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:24:37.228 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:24:37.307 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:37.328 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:37.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:37.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:37.417 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:37 smithi114 ceph-mon[40867]: 6.13 scrub starts 2024-09-21T12:24:37.417 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:37 smithi114 ceph-mon[40867]: 6.13 scrub ok 2024-09-21T12:24:37.417 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:37 smithi114 ceph-mon[40867]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:24:37.417 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:37 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:24:37.417 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:37 smithi114 ceph-mon[40867]: osdmap e155: 8 total, 8 up, 7 in 2024-09-21T12:24:37.417 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:37 smithi114 ceph-mon[40867]: 5.12 scrub starts 2024-09-21T12:24:37.417 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:37 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:24:37.417 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:37 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2176782517' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:24:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:37 smithi114 ceph-mon[30324]: 6.13 scrub starts 2024-09-21T12:24:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:37 smithi114 ceph-mon[30324]: 6.13 scrub ok 2024-09-21T12:24:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:37 smithi114 ceph-mon[30324]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:24:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:37 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:24:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:37 smithi114 ceph-mon[30324]: osdmap e155: 8 total, 8 up, 7 in 2024-09-21T12:24:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:37 smithi114 ceph-mon[30324]: 5.12 scrub starts 2024-09-21T12:24:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:37 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:24:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:37 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2176782517' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:24:37.431 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:37.451 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:37.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:37.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:37.553 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:37.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:37.575 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:37.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:37.676 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:37.698 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:37.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:37.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:37.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:37 smithi183 ceph-mon[33277]: 6.13 scrub starts 2024-09-21T12:24:37.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:37 smithi183 ceph-mon[33277]: 6.13 scrub ok 2024-09-21T12:24:37.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:37 smithi183 ceph-mon[33277]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:24:37.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:37 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:24:37.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:37 smithi183 ceph-mon[33277]: osdmap e155: 8 total, 8 up, 7 in 2024-09-21T12:24:37.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:37 smithi183 ceph-mon[33277]: 5.12 scrub starts 2024-09-21T12:24:37.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:37 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:24:37.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:37 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2176782517' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:24:37.800 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:37.821 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:37.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:37.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:37.860 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T12:24:37.923 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:37.964 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:37.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:37.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:38.066 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:38.089 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:38.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:38.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:38.147 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:24:38.191 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:38.212 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:38.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:38.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:38.268 INFO:teuthology.orchestra.run.smithi114.stderr:clear 5.e pg_upmap mapping 2024-09-21T12:24:38.314 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:38.335 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:38.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:38.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:38.437 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:38.458 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:38.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:38.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:38.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:38 smithi114 ceph-mon[30324]: pgmap v199: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 635 KiB data, 896 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 994 op/s 2024-09-21T12:24:38.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:38 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1344601445' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap", "pgid": "5.e"} : dispatch 2024-09-21T12:24:38.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:38 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap", "pgid": "5.e"} : dispatch 2024-09-21T12:24:38.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:38 smithi114 ceph-mon[40867]: pgmap v199: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 635 KiB data, 896 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 994 op/s 2024-09-21T12:24:38.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:38 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1344601445' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap", "pgid": "5.e"} : dispatch 2024-09-21T12:24:38.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:38 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap", "pgid": "5.e"} : dispatch 2024-09-21T12:24:38.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:38 smithi183 ceph-mon[33277]: pgmap v199: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 635 KiB data, 896 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 994 op/s 2024-09-21T12:24:38.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:38 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1344601445' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap", "pgid": "5.e"} : dispatch 2024-09-21T12:24:38.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:38 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap", "pgid": "5.e"} : dispatch 2024-09-21T12:24:38.560 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:38.580 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:38.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:38.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:38.640 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:24:38.640 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:24:38.682 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:38.703 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:38.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:38.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:38.806 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:38.827 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:38.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:38.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:38.929 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:38.950 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:38.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:38.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:38.960 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T12:24:39.052 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:39.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:39.075 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:39.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:39.175 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:39.196 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:39.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:39.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:39.204 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:39.298 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:39.319 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:39.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:39.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:39.370 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T12:24:39.421 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:39.442 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:39.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:39.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:39.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:39 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "5.e"}]': finished 2024-09-21T12:24:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:39 smithi114 ceph-mon[30324]: osdmap e156: 8 total, 8 up, 7 in 2024-09-21T12:24:39.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:39 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "5.e"}]': finished 2024-09-21T12:24:39.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:39 smithi114 ceph-mon[40867]: osdmap e156: 8 total, 8 up, 7 in 2024-09-21T12:24:39.543 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:39.565 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:39.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:39.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:39.617 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:24:39.666 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:39.687 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:39.687 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-21T12:24:39.687 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-21T12:24:39.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:39.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:39.737 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:39 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "5.e"}]': finished 2024-09-21T12:24:39.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:39 smithi183 ceph-mon[33277]: osdmap e156: 8 total, 8 up, 7 in 2024-09-21T12:24:39.789 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:39.811 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:39.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:39.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:39.912 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:39.933 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:39.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:39.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:40.035 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:40.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:40.057 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:40.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:40.101 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:24:40.101 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:24:40.158 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:40.181 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:40.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:40.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:40.283 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:40.287 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:24:40.304 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:40.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:40.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:40.406 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:40.427 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:40.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:40.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:40.504 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-21T12:24:40.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:40 smithi114 ceph-mon[40867]: osdmap e157: 8 total, 8 up, 7 in 2024-09-21T12:24:40.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:40 smithi114 ceph-mon[40867]: pgmap v202: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 635 KiB data, 896 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 351 op/s 2024-09-21T12:24:40.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:40 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2260588077' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:24:40.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:40 smithi114 ceph-mon[30324]: osdmap e157: 8 total, 8 up, 7 in 2024-09-21T12:24:40.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:40 smithi114 ceph-mon[30324]: pgmap v202: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 635 KiB data, 896 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 351 op/s 2024-09-21T12:24:40.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:40 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2260588077' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:24:40.528 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:40.549 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:40.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:40.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:40.650 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:40.672 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:40.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:40.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:40.719 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:40 smithi183 ceph-mon[33277]: osdmap e157: 8 total, 8 up, 7 in 2024-09-21T12:24:40.719 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:40 smithi183 ceph-mon[33277]: pgmap v202: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 635 KiB data, 896 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 351 op/s 2024-09-21T12:24:40.719 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:40 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2260588077' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:24:40.745 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T12:24:40.774 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:40.796 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:40.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:40.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:40.898 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:40.920 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:40.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:40.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:40.987 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:24:41.015 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:41.022 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:41.046 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:41.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:41.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:41.148 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:41.169 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:41.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:41.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:41.271 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:41.293 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:41.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:41.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:41.396 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:41.417 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:41.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:41.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:41.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:41 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2260588077' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:24:41.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:41 smithi114 ceph-mon[30324]: osdmap e158: 8 total, 8 up, 7 in 2024-09-21T12:24:41.419 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:41 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2260588077' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:24:41.419 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:41 smithi114 ceph-mon[40867]: osdmap e158: 8 total, 8 up, 7 in 2024-09-21T12:24:41.431 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:41 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2260588077' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:24:41.431 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:41 smithi183 ceph-mon[33277]: osdmap e158: 8 total, 8 up, 7 in 2024-09-21T12:24:41.438 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:24:41.438 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:24:41.516 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:41.519 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:41.540 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:41.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:41.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:41.642 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:41.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:41.664 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:41.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:41.764 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:41.785 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:41.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:41.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:41.887 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:41.908 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:41.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:41.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:42.010 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:42.032 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:42.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:42.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:42.134 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:42.155 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:42.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:42.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:42.257 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:42.281 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'unique_pool_0' removed 2024-09-21T12:24:42.283 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:42.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:42.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:42.385 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:42.406 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:42.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:42.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:42.467 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:42 smithi114 ceph-mon[40867]: pgmap v204: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 635 KiB data, 897 MiB used, 625 GiB / 626 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 392 op/s 2024-09-21T12:24:42.467 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:42 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/871929195' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T12:24:42.467 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:42 smithi114 ceph-mon[40867]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:24:42.468 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:42 smithi114 ceph-mon[30324]: pgmap v204: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 635 KiB data, 897 MiB used, 625 GiB / 626 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 392 op/s 2024-09-21T12:24:42.468 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:42 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/871929195' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T12:24:42.468 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:42 smithi114 ceph-mon[30324]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:24:42.508 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:42.524 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:42 smithi183 ceph-mon[33277]: pgmap v204: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 635 KiB data, 897 MiB used, 625 GiB / 626 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 392 op/s 2024-09-21T12:24:42.524 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:42 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/871929195' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T12:24:42.524 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:42 smithi183 ceph-mon[33277]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:24:42.531 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:42.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:42.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:42.553 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T12:24:42.634 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:42.656 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:42.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:42.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:42.660 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:24:42.661 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:24:42.661 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:24:42.661 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:24:42.661 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:24:42.661 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:24:42.661 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:24:42.661 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:24:42.661 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:24:42.661 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:24:42.661 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:24:42.661 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:24:42.662 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:24:42.662 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:24:42.662 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:24:42.662 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:24:42.758 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:42.774 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:24:42 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:24:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:24:42.780 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:42.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:42.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:42.882 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:42.903 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:42.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:42.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:43.004 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:43.026 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:43.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:43.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:43.128 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:43.150 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:43.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:43.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:43.252 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:43.273 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:43.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:43.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:43.351 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T12:24:43.353 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T12:24:43.354 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-09-21T12:24:43.354 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_1 16 2024-09-21T12:24:43.374 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:43.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:43.397 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:43.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:43.498 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:43.519 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:43.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:43.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:43.621 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:43.642 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:43.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:43.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:43.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:43 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/871929195' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T12:24:43.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:43 smithi183 ceph-mon[33277]: osdmap e159: 8 total, 8 up, 7 in 2024-09-21T12:24:43.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:43 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/4155170346' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:24:43.744 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:43.765 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:43.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:43.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:43.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:43 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/871929195' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T12:24:43.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:43 smithi114 ceph-mon[40867]: osdmap e159: 8 total, 8 up, 7 in 2024-09-21T12:24:43.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:43 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/4155170346' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:24:43.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:43 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/871929195' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T12:24:43.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:43 smithi114 ceph-mon[30324]: osdmap e159: 8 total, 8 up, 7 in 2024-09-21T12:24:43.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:43 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4155170346' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:24:43.836 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:43.866 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:43.887 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:43.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:43.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:43.968 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T12:24:43.968 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T12:24:43.989 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:44.010 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:44.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:44.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:44.112 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:44.133 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:44.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:44.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:44.235 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:44.241 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T12:24:44.256 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:44.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:44.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:44.355 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:44 smithi114 ceph-mon[30324]: pgmap v206: 142 pgs: 142 active+clean; 635 KiB data, 897 MiB used, 625 GiB / 626 GiB avail; 760 B/s rd, 0 op/s 2024-09-21T12:24:44.355 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:44 smithi114 ceph-mon[40867]: pgmap v206: 142 pgs: 142 active+clean; 635 KiB data, 897 MiB used, 625 GiB / 626 GiB avail; 760 B/s rd, 0 op/s 2024-09-21T12:24:44.358 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:44.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:44.382 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:44.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:44.447 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:44.460 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:44.483 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:44.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:44 smithi183 ceph-mon[33277]: pgmap v206: 142 pgs: 142 active+clean; 635 KiB data, 897 MiB used, 625 GiB / 626 GiB avail; 760 B/s rd, 0 op/s 2024-09-21T12:24:44.507 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:44.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:44.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:44.609 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:44.632 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:44.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:44.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:44.734 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:44.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:44.755 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:44.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:44.856 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:44.877 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:44.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:44.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:44.979 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:45.001 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:45.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:45.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:45.103 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:45.126 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:45.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:45.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:45.228 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:45.248 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:45.251 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:45.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:45.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:45.266 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:24:45.266 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:24:45.266 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 0 2024-09-21T12:24:45.266 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:24:45.308 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'unique_pool_1' created 2024-09-21T12:24:45.352 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:45.374 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:45.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:45.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:45.476 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:45.497 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:45.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:45.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:45.499 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:45 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/12377819' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T12:24:45.499 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:45 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T12:24:45.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:45 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/12377819' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T12:24:45.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:45 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T12:24:45.599 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:45.622 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:45.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:45.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:45.723 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:45.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:45 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/12377819' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T12:24:45.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:45 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T12:24:45.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:45.746 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:45.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:45.826 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:24:45.846 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:45.890 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:45.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:45.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:45.992 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:46.012 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:46.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:46.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:46.114 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:46.140 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:46.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:46.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:46.241 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:46.263 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:46.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:46.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:46.364 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:46.389 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:46.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:46.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:46.490 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:46.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:46 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:24:46.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:46 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:24:46.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:46 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-21T12:24:46.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:46 smithi114 ceph-mon[30324]: osdmap e160: 8 total, 8 up, 7 in 2024-09-21T12:24:46.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:46 smithi114 ceph-mon[30324]: pgmap v208: 158 pgs: 16 unknown, 142 active+clean; 635 KiB data, 897 MiB used, 625 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T12:24:46.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:46 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:24:46.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:24:46.505 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:46 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:24:46.505 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:46 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:24:46.506 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:46 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-21T12:24:46.506 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:46 smithi114 ceph-mon[40867]: osdmap e160: 8 total, 8 up, 7 in 2024-09-21T12:24:46.506 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:46 smithi114 ceph-mon[40867]: pgmap v208: 158 pgs: 16 unknown, 142 active+clean; 635 KiB data, 897 MiB used, 625 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T12:24:46.506 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:46 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:24:46.506 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:24:46.514 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:46.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:46.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:46.616 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:46.637 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:46.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:46.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:46.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:46 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:24:46.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:46 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:24:46.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:46 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-21T12:24:46.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:46 smithi183 ceph-mon[33277]: osdmap e160: 8 total, 8 up, 7 in 2024-09-21T12:24:46.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:46 smithi183 ceph-mon[33277]: pgmap v208: 158 pgs: 16 unknown, 142 active+clean; 635 KiB data, 897 MiB used, 625 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T12:24:46.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:46 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:24:46.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:24:46.739 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:46.760 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:46.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:46.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:46.862 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:46.883 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:46.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:46.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:46.984 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:47.010 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:47.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:47.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:47.112 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:47.133 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:47.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:47.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:47.174 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T12:24:47.234 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:47.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:24:46 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:24:47.260 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:47.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:47.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:47.310 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T12:24:47.317 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-21T12:24:47.362 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:47.388 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:47.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:47.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:47.490 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:47.516 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:47.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:47.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:47.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:47 smithi114 ceph-mon[40867]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T12:24:47.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:47 smithi114 ceph-mon[40867]: osdmap e161: 8 total, 8 up, 7 in 2024-09-21T12:24:47.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:47 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3382624118' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:24:47.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:47 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:24:47.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:47 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/304139518' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:24:47.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:47 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:24:47.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:47 smithi114 ceph-mon[30324]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T12:24:47.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:47 smithi114 ceph-mon[30324]: osdmap e161: 8 total, 8 up, 7 in 2024-09-21T12:24:47.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:47 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3382624118' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:24:47.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:47 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:24:47.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:47 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/304139518' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:24:47.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:47 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:24:47.618 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:47.644 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:47.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:47.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:47.655 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-09-21T12:24:47.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:47 smithi183 ceph-mon[33277]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T12:24:47.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:47 smithi183 ceph-mon[33277]: osdmap e161: 8 total, 8 up, 7 in 2024-09-21T12:24:47.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:47 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3382624118' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:24:47.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:47 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:24:47.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:47 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/304139518' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:24:47.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:47 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:24:47.745 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:47.771 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:47.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:47.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:47.872 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:47.898 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:47.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:47.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:48.000 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:48.025 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:48.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:48.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:48.127 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:48.153 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:48.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:48.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:48.255 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:48.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:48.278 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:48.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:48.286 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T12:24:48.380 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:48.406 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:48.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:48.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:48.507 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:48.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:48 smithi114 ceph-mon[40867]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:24:48.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:48 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:24:48.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:48 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/304139518' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T12:24:48.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:48 smithi114 ceph-mon[40867]: osdmap e162: 8 total, 8 up, 7 in 2024-09-21T12:24:48.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:48 smithi114 ceph-mon[40867]: pgmap v211: 158 pgs: 158 active+clean; 635 KiB data, 919 MiB used, 625 GiB / 626 GiB avail; 955 B/s rd, 0 op/s 2024-09-21T12:24:48.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:48 smithi114 ceph-mon[30324]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:24:48.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:48 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:24:48.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:48 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/304139518' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T12:24:48.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:48 smithi114 ceph-mon[30324]: osdmap e162: 8 total, 8 up, 7 in 2024-09-21T12:24:48.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:48 smithi114 ceph-mon[30324]: pgmap v211: 158 pgs: 158 active+clean; 635 KiB data, 919 MiB used, 625 GiB / 626 GiB avail; 955 B/s rd, 0 op/s 2024-09-21T12:24:48.532 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:48.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:48.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:48.634 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:48.656 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:48.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:48.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:48.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:48 smithi183 ceph-mon[33277]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:24:48.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:48 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:24:48.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:48 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/304139518' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T12:24:48.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:48 smithi183 ceph-mon[33277]: osdmap e162: 8 total, 8 up, 7 in 2024-09-21T12:24:48.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:48 smithi183 ceph-mon[33277]: pgmap v211: 158 pgs: 158 active+clean; 635 KiB data, 919 MiB used, 625 GiB / 626 GiB avail; 955 B/s rd, 0 op/s 2024-09-21T12:24:48.758 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:48.783 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:48.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:48.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:48.885 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:48.910 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:48.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:48.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:48.975 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:48.981 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:49.012 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:49.037 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:49.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:49.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:49.139 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:49.164 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:49.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:49.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:49.253 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:49.265 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:49.290 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:49.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:49.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:49.392 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:49.416 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:49.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:49.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:49.478 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:49 smithi114 ceph-mon[40867]: 3.1 scrub starts 2024-09-21T12:24:49.478 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:49 smithi114 ceph-mon[40867]: 3.1 scrub ok 2024-09-21T12:24:49.479 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:49 smithi114 ceph-mon[30324]: 3.1 scrub starts 2024-09-21T12:24:49.479 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:49 smithi114 ceph-mon[30324]: 3.1 scrub ok 2024-09-21T12:24:49.487 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:49.517 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:49.539 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:49.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:49.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:49.640 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:49.666 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:49.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:49.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:49.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:49 smithi183 ceph-mon[33277]: 3.1 scrub starts 2024-09-21T12:24:49.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:49 smithi183 ceph-mon[33277]: 3.1 scrub ok 2024-09-21T12:24:49.768 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:49.792 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:49.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:49.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:49.894 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:49.916 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:49.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:49.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:50.018 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:50.039 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:50.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:50.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:50.141 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:50.167 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:50.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:50.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:50.268 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:50.290 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:50.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:50.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:50.331 INFO:teuthology.orchestra.run.smithi114.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-09-21T12:24:50.392 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:50.418 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:50.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:50.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:50.520 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:50.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:50.542 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:50.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:50.643 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:50.664 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:50.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:50.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:50.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:50 smithi183 ceph-mon[33277]: pgmap v212: 158 pgs: 158 active+clean; 635 KiB data, 919 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:24:50.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:50 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2073853313' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:24:50.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:50 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:24:50.766 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:50.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:50 smithi114 ceph-mon[30324]: pgmap v212: 158 pgs: 158 active+clean; 635 KiB data, 919 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:24:50.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:50 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2073853313' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:24:50.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:50 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:24:50.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:50 smithi114 ceph-mon[40867]: pgmap v212: 158 pgs: 158 active+clean; 635 KiB data, 919 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:24:50.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:50 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2073853313' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:24:50.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:50 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:24:50.792 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:50.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:50.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:50.894 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:50.918 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:50.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:50.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:50.940 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T12:24:51.020 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:51.041 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:51.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:51.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:51.143 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:51.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:51.165 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:51.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:51.266 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:51.288 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:51.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:51.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:51.316 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:24:51.316 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:24:51.316 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:24:51.316 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:24:51.316 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:24:51.329 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T12:24:51.390 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:51.411 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:51.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:51.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:51.513 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:51.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:51 smithi114 ceph-mon[30324]: 3.1e scrub starts 2024-09-21T12:24:51.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:51 smithi114 ceph-mon[30324]: 3.1e scrub ok 2024-09-21T12:24:51.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:51 smithi114 ceph-mon[30324]: 2.1 scrub starts 2024-09-21T12:24:51.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:51 smithi114 ceph-mon[30324]: 2.1 scrub ok 2024-09-21T12:24:51.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:51 smithi114 ceph-mon[30324]: 3.1f scrub starts 2024-09-21T12:24:51.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:51 smithi114 ceph-mon[30324]: 3.1f scrub ok 2024-09-21T12:24:51.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:51 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T12:24:51.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:51 smithi114 ceph-mon[30324]: osdmap e163: 8 total, 8 up, 7 in 2024-09-21T12:24:51.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:51 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2052207490' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:24:51.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:51 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:24:51.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:51 smithi114 ceph-mon[30324]: 5.1f scrub starts 2024-09-21T12:24:51.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:51 smithi114 ceph-mon[30324]: 5.1f scrub ok 2024-09-21T12:24:51.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:51 smithi114 ceph-mon[40867]: 3.1e scrub starts 2024-09-21T12:24:51.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:51 smithi114 ceph-mon[40867]: 3.1e scrub ok 2024-09-21T12:24:51.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:51 smithi114 ceph-mon[40867]: 2.1 scrub starts 2024-09-21T12:24:51.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:51 smithi114 ceph-mon[40867]: 2.1 scrub ok 2024-09-21T12:24:51.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:51 smithi114 ceph-mon[40867]: 3.1f scrub starts 2024-09-21T12:24:51.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:51 smithi114 ceph-mon[40867]: 3.1f scrub ok 2024-09-21T12:24:51.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:51 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T12:24:51.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:51 smithi114 ceph-mon[40867]: osdmap e163: 8 total, 8 up, 7 in 2024-09-21T12:24:51.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:51 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2052207490' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:24:51.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:51 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:24:51.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:51 smithi114 ceph-mon[40867]: 5.1f scrub starts 2024-09-21T12:24:51.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:51 smithi114 ceph-mon[40867]: 5.1f scrub ok 2024-09-21T12:24:51.534 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:51.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:51.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:51.636 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:51.659 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:51.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:51.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:51.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:51 smithi183 ceph-mon[33277]: 3.1e scrub starts 2024-09-21T12:24:51.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:51 smithi183 ceph-mon[33277]: 3.1e scrub ok 2024-09-21T12:24:51.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:51 smithi183 ceph-mon[33277]: 2.1 scrub starts 2024-09-21T12:24:51.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:51 smithi183 ceph-mon[33277]: 2.1 scrub ok 2024-09-21T12:24:51.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:51 smithi183 ceph-mon[33277]: 3.1f scrub starts 2024-09-21T12:24:51.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:51 smithi183 ceph-mon[33277]: 3.1f scrub ok 2024-09-21T12:24:51.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:51 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T12:24:51.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:51 smithi183 ceph-mon[33277]: osdmap e163: 8 total, 8 up, 7 in 2024-09-21T12:24:51.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:51 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2052207490' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:24:51.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:51 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:24:51.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:51 smithi183 ceph-mon[33277]: 5.1f scrub starts 2024-09-21T12:24:51.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:51 smithi183 ceph-mon[33277]: 5.1f scrub ok 2024-09-21T12:24:51.761 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:51.783 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:51.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:51.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:51.885 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:51.907 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:51.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:51.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:51.995 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:24:51.996 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:24:52.003 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:24:52.004 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:24:52.009 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:52.019 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:24:52.019 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:24:52.031 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:52.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:52.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:52.035 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:24:52.035 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:24:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:24:52.052 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:24:52.068 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:24:52.068 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:24:52.086 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:24:52.087 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:24:52.107 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:24:52.108 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:24:52.133 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:52.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:52.157 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:52.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:52.257 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:52.279 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:52.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:52.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:52.381 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:52.403 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:52.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:52.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:52.505 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:52.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:52 smithi114 ceph-mon[40867]: 4.14 scrub starts 2024-09-21T12:24:52.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:52 smithi114 ceph-mon[40867]: 4.14 scrub ok 2024-09-21T12:24:52.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:52 smithi114 ceph-mon[40867]: 4.1c scrub starts 2024-09-21T12:24:52.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:52 smithi114 ceph-mon[40867]: 4.1c scrub ok 2024-09-21T12:24:52.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:52 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T12:24:52.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:52 smithi114 ceph-mon[40867]: osdmap e164: 8 total, 8 up, 7 in 2024-09-21T12:24:52.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:52 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:24:52.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:52 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:24:52.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:52 smithi114 ceph-mon[40867]: pgmap v215: 158 pgs: 158 active+clean; 635 KiB data, 919 MiB used, 625 GiB / 626 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-09-21T12:24:52.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:52 smithi114 ceph-mon[40867]: 1.7 scrub starts 2024-09-21T12:24:52.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:52 smithi114 ceph-mon[40867]: 1.7 scrub ok 2024-09-21T12:24:52.510 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:52 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3562582425' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:24:52.510 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:52 smithi114 ceph-mon[30324]: 4.14 scrub starts 2024-09-21T12:24:52.510 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:52 smithi114 ceph-mon[30324]: 4.14 scrub ok 2024-09-21T12:24:52.510 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:52 smithi114 ceph-mon[30324]: 4.1c scrub starts 2024-09-21T12:24:52.510 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:52 smithi114 ceph-mon[30324]: 4.1c scrub ok 2024-09-21T12:24:52.510 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:52 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T12:24:52.510 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:52 smithi114 ceph-mon[30324]: osdmap e164: 8 total, 8 up, 7 in 2024-09-21T12:24:52.511 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:52 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:24:52.511 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:52 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:24:52.511 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:52 smithi114 ceph-mon[30324]: pgmap v215: 158 pgs: 158 active+clean; 635 KiB data, 919 MiB used, 625 GiB / 626 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-09-21T12:24:52.511 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:52 smithi114 ceph-mon[30324]: 1.7 scrub starts 2024-09-21T12:24:52.511 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:52 smithi114 ceph-mon[30324]: 1.7 scrub ok 2024-09-21T12:24:52.511 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:52 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3562582425' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:24:52.526 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:52.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:52.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:52.533 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T12:24:52.628 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:52.649 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:52.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:52.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:52.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:52 smithi183 ceph-mon[33277]: 4.14 scrub starts 2024-09-21T12:24:52.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:52 smithi183 ceph-mon[33277]: 4.14 scrub ok 2024-09-21T12:24:52.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:52 smithi183 ceph-mon[33277]: 4.1c scrub starts 2024-09-21T12:24:52.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:52 smithi183 ceph-mon[33277]: 4.1c scrub ok 2024-09-21T12:24:52.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:52 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T12:24:52.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:52 smithi183 ceph-mon[33277]: osdmap e164: 8 total, 8 up, 7 in 2024-09-21T12:24:52.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:52 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:24:52.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:52 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:24:52.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:52 smithi183 ceph-mon[33277]: pgmap v215: 158 pgs: 158 active+clean; 635 KiB data, 919 MiB used, 625 GiB / 626 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-09-21T12:24:52.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:52 smithi183 ceph-mon[33277]: 1.7 scrub starts 2024-09-21T12:24:52.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:52 smithi183 ceph-mon[33277]: 1.7 scrub ok 2024-09-21T12:24:52.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:52 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3562582425' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:24:52.750 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:52.771 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:52.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:52.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:52.774 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:24:52 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:24:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:24:52.873 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:52.894 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:52.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:52.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:52.995 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:53.017 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:53.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:53.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:53.118 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:53.140 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:53.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:53.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:53.241 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:53.262 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:53.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:53.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:53.364 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:53.387 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:53.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:53.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:53.402 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T12:24:53.488 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:53.509 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:53.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:53.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:53.611 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:53.632 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:53.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:53.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:53.734 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:53.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:53 smithi183 ceph-mon[33277]: 6.17 scrub starts 2024-09-21T12:24:53.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:53 smithi183 ceph-mon[33277]: 6.17 scrub ok 2024-09-21T12:24:53.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:53 smithi183 ceph-mon[33277]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T12:24:53.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:53 smithi183 ceph-mon[33277]: osdmap e165: 8 total, 8 up, 7 in 2024-09-21T12:24:53.756 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:53.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:53.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:53.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:53 smithi114 ceph-mon[40867]: 6.17 scrub starts 2024-09-21T12:24:53.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:53 smithi114 ceph-mon[40867]: 6.17 scrub ok 2024-09-21T12:24:53.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:53 smithi114 ceph-mon[40867]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T12:24:53.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:53 smithi114 ceph-mon[40867]: osdmap e165: 8 total, 8 up, 7 in 2024-09-21T12:24:53.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:53 smithi114 ceph-mon[30324]: 6.17 scrub starts 2024-09-21T12:24:53.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:53 smithi114 ceph-mon[30324]: 6.17 scrub ok 2024-09-21T12:24:53.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:53 smithi114 ceph-mon[30324]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T12:24:53.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:53 smithi114 ceph-mon[30324]: osdmap e165: 8 total, 8 up, 7 in 2024-09-21T12:24:53.857 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:53.879 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:53.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:53.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:53.980 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:53.983 DEBUG:teuthology.orchestra.run.smithi114:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-09-21T12:24:53.985 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T12:24:53.985 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T12:24:54.002 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:54.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:54.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:54.035 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:hints = 1 2024-09-21T12:24:54.035 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T12:24:54.035 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Object prefix: benchmark_data_smithi114_104936 2024-09-21T12:24:54.035 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T12:24:54.035 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 0 0 0 0 0 0 - 0 2024-09-21T12:24:54.103 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:54.131 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:54.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:54.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:54.233 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:54.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:54.272 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:54.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:54.342 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:54.373 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:54.423 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:54.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:54.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:54.429 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:54.450 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:54.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:54 smithi114 ceph-mon[30324]: 5.d scrub starts 2024-09-21T12:24:54.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:54 smithi114 ceph-mon[30324]: 5.d scrub ok 2024-09-21T12:24:54.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:54 smithi114 ceph-mon[30324]: 5.5 scrub starts 2024-09-21T12:24:54.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:54 smithi114 ceph-mon[30324]: 5.5 scrub ok 2024-09-21T12:24:54.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:54 smithi114 ceph-mon[30324]: pgmap v217: 158 pgs: 158 active+clean; 635 KiB data, 919 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:24:54.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:54 smithi114 ceph-mon[30324]: 1.6 scrub starts 2024-09-21T12:24:54.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:54 smithi114 ceph-mon[30324]: 1.6 scrub ok 2024-09-21T12:24:54.506 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:54 smithi114 ceph-mon[40867]: 5.d scrub starts 2024-09-21T12:24:54.506 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:54 smithi114 ceph-mon[40867]: 5.d scrub ok 2024-09-21T12:24:54.506 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:54 smithi114 ceph-mon[40867]: 5.5 scrub starts 2024-09-21T12:24:54.506 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:54 smithi114 ceph-mon[40867]: 5.5 scrub ok 2024-09-21T12:24:54.506 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:54 smithi114 ceph-mon[40867]: pgmap v217: 158 pgs: 158 active+clean; 635 KiB data, 919 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:24:54.506 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:54 smithi114 ceph-mon[40867]: 1.6 scrub starts 2024-09-21T12:24:54.507 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:54 smithi114 ceph-mon[40867]: 1.6 scrub ok 2024-09-21T12:24:54.525 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:54.563 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:54.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:54.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:54.664 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:54.691 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:54.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:54.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:54.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:54 smithi183 ceph-mon[33277]: 5.d scrub starts 2024-09-21T12:24:54.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:54 smithi183 ceph-mon[33277]: 5.d scrub ok 2024-09-21T12:24:54.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:54 smithi183 ceph-mon[33277]: 5.5 scrub starts 2024-09-21T12:24:54.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:54 smithi183 ceph-mon[33277]: 5.5 scrub ok 2024-09-21T12:24:54.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:54 smithi183 ceph-mon[33277]: pgmap v217: 158 pgs: 158 active+clean; 635 KiB data, 919 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:24:54.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:54 smithi183 ceph-mon[33277]: 1.6 scrub starts 2024-09-21T12:24:54.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:54 smithi183 ceph-mon[33277]: 1.6 scrub ok 2024-09-21T12:24:54.793 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:54.821 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:54.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:54.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:54.922 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:54.967 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:54.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:54.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:55.035 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 1 15 3334 3319 207.425 207.438 0.00544937 0.00479912 2024-09-21T12:24:55.070 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:55.101 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:55.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:55.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:55.203 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:55.243 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:55.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:55.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:55.346 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:55.384 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:55.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:55.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:55.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:55 smithi114 ceph-mon[40867]: 4.9 scrub starts 2024-09-21T12:24:55.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:55 smithi114 ceph-mon[40867]: 4.9 scrub ok 2024-09-21T12:24:55.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:55 smithi114 ceph-mon[30324]: 4.9 scrub starts 2024-09-21T12:24:55.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:55 smithi114 ceph-mon[30324]: 4.9 scrub ok 2024-09-21T12:24:55.486 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:55.522 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:55.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:55.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:55.624 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:55.659 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:24:55.659 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:24:55.666 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:55.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:55.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:55.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:55 smithi183 ceph-mon[33277]: 4.9 scrub starts 2024-09-21T12:24:55.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:55 smithi183 ceph-mon[33277]: 4.9 scrub ok 2024-09-21T12:24:55.768 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:55.796 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:55.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:55.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:55.898 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:55.937 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:55.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:55.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:56.035 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 2 16 6446 6430 200.923 194.438 0.00249038 0.0049633 2024-09-21T12:24:56.039 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:56.074 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:56.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:56.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:56.176 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:56.203 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:56.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:56.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:56.281 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:24:56.304 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:56.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:56.369 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:56.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:56.376 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-21T12:24:56.470 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:56.520 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:56.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:56.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:56.622 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:56.666 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:56.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:56.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:56.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:56 smithi183 ceph-mon[33277]: 3.1a scrub starts 2024-09-21T12:24:56.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:56 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3418867382' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:24:56.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:56 smithi183 ceph-mon[33277]: 3.1a scrub ok 2024-09-21T12:24:56.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:56 smithi183 ceph-mon[33277]: pgmap v218: 158 pgs: 158 active+clean; 635 KiB data, 919 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:24:56.767 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:56.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:56 smithi114 ceph-mon[40867]: 3.1a scrub starts 2024-09-21T12:24:56.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:56 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3418867382' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:24:56.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:56 smithi114 ceph-mon[40867]: 3.1a scrub ok 2024-09-21T12:24:56.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:56 smithi114 ceph-mon[40867]: pgmap v218: 158 pgs: 158 active+clean; 635 KiB data, 919 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:24:56.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:56 smithi114 ceph-mon[30324]: 3.1a scrub starts 2024-09-21T12:24:56.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:56 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3418867382' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:24:56.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:56 smithi114 ceph-mon[30324]: 3.1a scrub ok 2024-09-21T12:24:56.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:56 smithi114 ceph-mon[30324]: pgmap v218: 158 pgs: 158 active+clean; 635 KiB data, 919 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:24:56.803 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:56.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:56.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:56.843 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T12:24:56.905 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:56.944 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:56.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:56.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:56.976 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T12:24:57.036 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 3 16 9030 9014 187.76 161.5 0.00512054 0.0053176 2024-09-21T12:24:57.046 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:57.074 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:57.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:57.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:57.176 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:57.221 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:57.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:57.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:57.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:24:56 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:24:57.322 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:57.350 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:57.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:57.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:57.452 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:57.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:57.492 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:57.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:57.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:57 smithi114 ceph-mon[40867]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:24:57.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:57 smithi114 ceph-mon[40867]: Cluster is now healthy 2024-09-21T12:24:57.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:57 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3418867382' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:24:57.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:57 smithi114 ceph-mon[40867]: osdmap e166: 8 total, 8 up, 7 in 2024-09-21T12:24:57.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:57 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/547491356' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:24:57.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:57 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:24:57.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:57 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:24:57.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:57 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T12:24:57.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:57 smithi114 ceph-mon[40867]: osdmap e167: 8 total, 8 up, 7 in 2024-09-21T12:24:57.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:57 smithi114 ceph-mon[40867]: 6.11 scrub starts 2024-09-21T12:24:57.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:57 smithi114 ceph-mon[40867]: 6.11 scrub ok 2024-09-21T12:24:57.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:57 smithi114 ceph-mon[30324]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:24:57.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:57 smithi114 ceph-mon[30324]: Cluster is now healthy 2024-09-21T12:24:57.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:57 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3418867382' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:24:57.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:57 smithi114 ceph-mon[30324]: osdmap e166: 8 total, 8 up, 7 in 2024-09-21T12:24:57.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:57 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/547491356' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:24:57.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:57 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:24:57.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:57 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:24:57.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:57 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T12:24:57.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:57 smithi114 ceph-mon[30324]: osdmap e167: 8 total, 8 up, 7 in 2024-09-21T12:24:57.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:57 smithi114 ceph-mon[30324]: 6.11 scrub starts 2024-09-21T12:24:57.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:57 smithi114 ceph-mon[30324]: 6.11 scrub ok 2024-09-21T12:24:57.593 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:57.622 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:57.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:57.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:57.723 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:57.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:57 smithi183 ceph-mon[33277]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:24:57.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:57 smithi183 ceph-mon[33277]: Cluster is now healthy 2024-09-21T12:24:57.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:57 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3418867382' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:24:57.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:57 smithi183 ceph-mon[33277]: osdmap e166: 8 total, 8 up, 7 in 2024-09-21T12:24:57.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:57 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/547491356' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:24:57.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:57 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:24:57.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:57 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:24:57.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:57 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T12:24:57.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:57 smithi183 ceph-mon[33277]: osdmap e167: 8 total, 8 up, 7 in 2024-09-21T12:24:57.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:57 smithi183 ceph-mon[33277]: 6.11 scrub starts 2024-09-21T12:24:57.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:57 smithi183 ceph-mon[33277]: 6.11 scrub ok 2024-09-21T12:24:57.754 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:57.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:57.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:57.856 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:57.884 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:57.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:57.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:57.986 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:58.015 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:58.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:58.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:58.036 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 4 16 11945 11929 186.363 182.188 0.00277764 0.00535949 2024-09-21T12:24:58.117 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:58.147 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:58.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:58.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:58.249 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:58.285 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:58.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:58.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:58.386 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:58.413 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:58.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:58.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:58.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:58 smithi114 ceph-mon[30324]: pgmap v221: 158 pgs: 158 active+clean; 526 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-21T12:24:58.513 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:58 smithi114 ceph-mon[40867]: pgmap v221: 158 pgs: 158 active+clean; 526 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-21T12:24:58.515 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:24:58.543 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:58.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:58.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:24:58.567 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T12:24:58.645 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:58.671 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:58.695 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:58.698 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:58.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:58.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:58.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:58 smithi183 ceph-mon[33277]: pgmap v221: 158 pgs: 158 active+clean; 526 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-21T12:24:58.801 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:24:58.830 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:58.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:58.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:24:58.933 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:24:58.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:58.981 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:58.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:24:59.037 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 5 16 14607 14591 182.32 166.375 0.00348212 0.00547808 2024-09-21T12:24:59.082 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:59.118 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:59.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:59.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:59.220 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:59.247 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:59.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:59.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:59.349 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:24:59.384 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:59.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:59.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:24:59.485 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:24:59.515 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:59.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:59.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:24:59.616 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:24:59.649 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:59.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:59.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:24:59.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:59 smithi183 ceph-mon[33277]: 5.1b scrub starts 2024-09-21T12:24:59.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:59 smithi183 ceph-mon[33277]: 5.1b scrub ok 2024-09-21T12:24:59.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:59 smithi183 ceph-mon[33277]: 4.11 scrub starts 2024-09-21T12:24:59.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:24:59 smithi183 ceph-mon[33277]: 4.11 scrub ok 2024-09-21T12:24:59.751 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:24:59.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:59 smithi114 ceph-mon[40867]: 5.1b scrub starts 2024-09-21T12:24:59.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:59 smithi114 ceph-mon[40867]: 5.1b scrub ok 2024-09-21T12:24:59.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:59 smithi114 ceph-mon[40867]: 4.11 scrub starts 2024-09-21T12:24:59.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:24:59 smithi114 ceph-mon[40867]: 4.11 scrub ok 2024-09-21T12:24:59.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:59 smithi114 ceph-mon[30324]: 5.1b scrub starts 2024-09-21T12:24:59.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:59 smithi114 ceph-mon[30324]: 5.1b scrub ok 2024-09-21T12:24:59.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:59 smithi114 ceph-mon[30324]: 4.11 scrub starts 2024-09-21T12:24:59.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:24:59 smithi114 ceph-mon[30324]: 4.11 scrub ok 2024-09-21T12:24:59.781 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:24:59.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:59.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:24:59.883 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:24:59.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:24:59.907 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:24:59.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:24:59.948 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T12:25:00.008 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:00.037 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 6 16 17043 17027 177.307 152.25 0.00874181 0.00563168 2024-09-21T12:25:00.048 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:00.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:00.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:00.150 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:00.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:00.192 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:00.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:00.293 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:00.320 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:00.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:00.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:00.421 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:00.450 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:00.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:00.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:00.534 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:25:00.534 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:25:00.552 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:00.570 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:25:00.571 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:25:00.606 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:25:00.606 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:25:00.607 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:00 smithi114 ceph-mon[30324]: 6.14 scrub starts 2024-09-21T12:25:00.607 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:00 smithi114 ceph-mon[30324]: 6.14 scrub ok 2024-09-21T12:25:00.607 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:00 smithi114 ceph-mon[30324]: 3.14 scrub starts 2024-09-21T12:25:00.607 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:00 smithi114 ceph-mon[30324]: 3.14 scrub ok 2024-09-21T12:25:00.607 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:00 smithi114 ceph-mon[30324]: pgmap v222: 158 pgs: 158 active+clean; 526 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 700 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-21T12:25:00.607 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:00 smithi114 ceph-mon[40867]: 6.14 scrub starts 2024-09-21T12:25:00.607 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:00 smithi114 ceph-mon[40867]: 6.14 scrub ok 2024-09-21T12:25:00.607 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:00 smithi114 ceph-mon[40867]: 3.14 scrub starts 2024-09-21T12:25:00.608 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:00 smithi114 ceph-mon[40867]: 3.14 scrub ok 2024-09-21T12:25:00.608 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:00 smithi114 ceph-mon[40867]: pgmap v222: 158 pgs: 158 active+clean; 526 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 700 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-21T12:25:00.610 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:00.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:00.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:00.646 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:25:00.646 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:25:00.686 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:25:00.686 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:25:00.712 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:00.720 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:25:00.721 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:25:00.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:00 smithi183 ceph-mon[33277]: 6.14 scrub starts 2024-09-21T12:25:00.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:00 smithi183 ceph-mon[33277]: 6.14 scrub ok 2024-09-21T12:25:00.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:00 smithi183 ceph-mon[33277]: 3.14 scrub starts 2024-09-21T12:25:00.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:00 smithi183 ceph-mon[33277]: 3.14 scrub ok 2024-09-21T12:25:00.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:00 smithi183 ceph-mon[33277]: pgmap v222: 158 pgs: 158 active+clean; 526 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 700 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-21T12:25:00.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:00.755 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:00.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:00.755 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:25:00.755 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:25:00.791 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:25:00.791 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:25:00.856 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:00.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:00.896 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:00.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:00.975 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:25:00.975 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-21T12:25:00.975 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:00.975 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25022.0:19108 8.1 8:8020c9c8:::benchmark_data_smithi114_104936_object19107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-09-21T12:25:00.975 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:25:00.965529+0000", 2024-09-21T12:25:00.975 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0065249449999999999, 2024-09-21T12:25:00.976 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0065676729999999996, 2024-09-21T12:25:00.976 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:25:00.976 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:25:00.976 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:25:00.976 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:25:00.976 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25022", 2024-09-21T12:25:00.976 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:25:00.977 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 19108 2024-09-21T12:25:00.977 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:00.977 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:25:00.977 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:00.977 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:25:00.977 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.965529+0000", 2024-09-21T12:25:00.978 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:00.978 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:00.978 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:00.978 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:25:00.978 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.965529+0000", 2024-09-21T12:25:00.978 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:00.979 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:00.979 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:00.979 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:25:00.979 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.965531+0000", 2024-09-21T12:25:00.979 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7120000000000001e-06 2024-09-21T12:25:00.981 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:00.981 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:00.981 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:25:00.982 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.965557+0000", 2024-09-21T12:25:00.982 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5692999999999999e-05 2024-09-21T12:25:00.982 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:00.982 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:00.982 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:25:00.983 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.965569+0000", 2024-09-21T12:25:00.983 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2079000000000001e-05 2024-09-21T12:25:00.983 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:00.983 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:00.983 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:25:00.984 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.965576+0000", 2024-09-21T12:25:00.984 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0949999999999998e-06 2024-09-21T12:25:00.984 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:00.984 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:00.984 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:25:00.984 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.966756+0000", 2024-09-21T12:25:00.985 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001180232 2024-09-21T12:25:00.985 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:00.985 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:00.985 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:25:00.985 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.966857+0000", 2024-09-21T12:25:00.985 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000100447 2024-09-21T12:25:00.986 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:00.986 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:00.986 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:25:00.986 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.966972+0000", 2024-09-21T12:25:00.986 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000115214 2024-09-21T12:25:00.986 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:00.987 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:00.987 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:00.987 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.970465+0000", 2024-09-21T12:25:00.987 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003492644 2024-09-21T12:25:00.987 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:00.987 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:00.987 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:25:00.988 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.970482+0000", 2024-09-21T12:25:00.988 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7266999999999999e-05 2024-09-21T12:25:00.988 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:00.988 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:25:00.988 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:00.989 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:00.989 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:00.989 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25022.0:19115 8.3 8:cbbd8169:::benchmark_data_smithi114_104936_object19114:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-09-21T12:25:00.989 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:25:00.966487+0000", 2024-09-21T12:25:00.989 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0055673140000000003, 2024-09-21T12:25:00.989 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0057191000000000004, 2024-09-21T12:25:00.989 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:25:00.990 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:25:00.990 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:25:00.990 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:25:00.990 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25022", 2024-09-21T12:25:00.990 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:25:00.990 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 19115 2024-09-21T12:25:00.991 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:00.991 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:25:00.991 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:00.991 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:25:00.991 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.966487+0000", 2024-09-21T12:25:00.991 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:00.991 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:00.992 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:00.992 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:25:00.992 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.966487+0000", 2024-09-21T12:25:00.992 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:00.992 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:00.992 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:00.992 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:25:00.993 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.966488+0000", 2024-09-21T12:25:00.993 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0440000000000001e-06 2024-09-21T12:25:00.993 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:00.993 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:00.993 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:25:00.993 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.966509+0000", 2024-09-21T12:25:00.993 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1197000000000001e-05 2024-09-21T12:25:00.994 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:00.994 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:00.994 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:25:00.994 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.966521+0000", 2024-09-21T12:25:00.994 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1863e-05 2024-09-21T12:25:00.994 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:00.994 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:00.995 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:25:00.995 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.966528+0000", 2024-09-21T12:25:00.995 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0700000000000001e-06 2024-09-21T12:25:00.995 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:00.995 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:00.995 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:25:00.995 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.967106+0000", 2024-09-21T12:25:00.996 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00057768000000000001 2024-09-21T12:25:00.996 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:00.996 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:00.996 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:25:00.996 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.967396+0000", 2024-09-21T12:25:00.996 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00028965099999999998 2024-09-21T12:25:00.996 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:00.996 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:00.997 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 5,6", 2024-09-21T12:25:00.997 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.967528+0000", 2024-09-21T12:25:00.997 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013214499999999999 2024-09-21T12:25:00.997 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:00.997 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:00.997 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:00.997 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.968863+0000", 2024-09-21T12:25:00.998 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001334855 2024-09-21T12:25:00.998 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:00.998 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:00.998 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:00.998 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.969589+0000", 2024-09-21T12:25:00.998 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00072640300000000003 2024-09-21T12:25:00.998 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:00.998 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:25:00.999 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:00.999 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:00.999 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:00.999 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25022.0:19117 8.1 8:83a38b92:::benchmark_data_smithi114_104936_object19116:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-09-21T12:25:00.999 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:25:00.968365+0000", 2024-09-21T12:25:00.999 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0036891570000000002, 2024-09-21T12:25:00.999 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0039159249999999998, 2024-09-21T12:25:01.000 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:25:01.000 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:25:01.000 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:25:01.000 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:25:01.000 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25022", 2024-09-21T12:25:01.000 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:25:01.000 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 19117 2024-09-21T12:25:01.001 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.001 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:25:01.001 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.001 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:25:01.001 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.968365+0000", 2024-09-21T12:25:01.001 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:01.001 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.002 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.002 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:25:01.002 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.968365+0000", 2024-09-21T12:25:01.002 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:01.002 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.002 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.002 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:25:01.003 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.968371+0000", 2024-09-21T12:25:01.003 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9689999999999999e-06 2024-09-21T12:25:01.003 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.003 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.003 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:25:01.003 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.968567+0000", 2024-09-21T12:25:01.003 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00019574499999999999 2024-09-21T12:25:01.004 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.004 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.004 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:25:01.004 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.968577+0000", 2024-09-21T12:25:01.004 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0152999999999999e-05 2024-09-21T12:25:01.004 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.004 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.005 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:01.006 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:25:01.006 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.968584+0000", 2024-09-21T12:25:01.007 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8680000000000001e-06 2024-09-21T12:25:01.007 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.007 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.007 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:25:01.007 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.970736+0000", 2024-09-21T12:25:01.007 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0021517649999999999 2024-09-21T12:25:01.007 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.008 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.008 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:25:01.008 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.970841+0000", 2024-09-21T12:25:01.008 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010527 2024-09-21T12:25:01.008 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.008 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.009 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:25:01.009 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.970962+0000", 2024-09-21T12:25:01.009 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012053300000000001 2024-09-21T12:25:01.009 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:01.009 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:25:01.009 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:01.009 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.009 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.010 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25022.0:19121 8.3 8:c40ba37d:::benchmark_data_smithi114_104936_object19120:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-09-21T12:25:01.010 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:25:00.969272+0000", 2024-09-21T12:25:01.010 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0027825749999999998, 2024-09-21T12:25:01.010 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0030677819999999998, 2024-09-21T12:25:01.010 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:25:01.010 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:25:01.011 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:25:01.011 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:25:01.011 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25022", 2024-09-21T12:25:01.011 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:25:01.011 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 19121 2024-09-21T12:25:01.011 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.011 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:25:01.011 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.012 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:25:01.012 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.969272+0000", 2024-09-21T12:25:01.012 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:01.012 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.012 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.012 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:25:01.012 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.969272+0000", 2024-09-21T12:25:01.013 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:01.013 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.013 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.013 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:25:01.013 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.969273+0000", 2024-09-21T12:25:01.013 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1999999999999998e-07 2024-09-21T12:25:01.013 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.014 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.014 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:25:01.014 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.969320+0000", 2024-09-21T12:25:01.014 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7089000000000003e-05 2024-09-21T12:25:01.014 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.014 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.015 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:25:01.015 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.969332+0000", 2024-09-21T12:25:01.015 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1815e-05 2024-09-21T12:25:01.015 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.015 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.015 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:25:01.015 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.969339+0000", 2024-09-21T12:25:01.015 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6550000000000004e-06 2024-09-21T12:25:01.016 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.016 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.016 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:25:01.016 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.970371+0000", 2024-09-21T12:25:01.016 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0010312570000000001 2024-09-21T12:25:01.016 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.017 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.017 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:25:01.017 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.970480+0000", 2024-09-21T12:25:01.017 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010932 2024-09-21T12:25:01.017 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.017 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.017 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 5,6", 2024-09-21T12:25:01.018 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.970858+0000", 2024-09-21T12:25:01.018 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00037850600000000002 2024-09-21T12:25:01.018 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:01.018 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:25:01.018 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:01.018 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.018 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.019 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25022.0:19122 8.a 8:5df3ccca:::benchmark_data_smithi114_104936_object19121:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-09-21T12:25:01.019 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:25:00.969355+0000", 2024-09-21T12:25:01.019 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0026989789999999998, 2024-09-21T12:25:01.019 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0030399170000000001, 2024-09-21T12:25:01.019 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:25:01.019 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:25:01.019 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:25:01.020 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:25:01.020 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25022", 2024-09-21T12:25:01.020 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:25:01.020 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 19122 2024-09-21T12:25:01.020 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.020 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:25:01.020 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.021 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:25:01.021 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.969355+0000", 2024-09-21T12:25:01.021 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:01.021 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.021 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.021 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:25:01.021 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.969355+0000", 2024-09-21T12:25:01.022 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:01.022 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.022 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.022 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:25:01.022 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.969356+0000", 2024-09-21T12:25:01.022 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4000000000000002e-07 2024-09-21T12:25:01.022 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.023 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.023 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:25:01.023 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.969406+0000", 2024-09-21T12:25:01.023 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9790000000000003e-05 2024-09-21T12:25:01.023 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.024 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.024 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:25:01.024 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.969414+0000", 2024-09-21T12:25:01.024 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5560000000000008e-06 2024-09-21T12:25:01.024 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.024 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.025 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:25:01.025 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.969420+0000", 2024-09-21T12:25:01.025 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9710000000000001e-06 2024-09-21T12:25:01.025 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.025 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.025 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:25:01.025 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.969473+0000", 2024-09-21T12:25:01.025 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2561e-05 2024-09-21T12:25:01.025 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.025 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.025 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:25:01.025 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.971767+0000", 2024-09-21T12:25:01.025 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002294552 2024-09-21T12:25:01.025 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.026 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.026 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,5", 2024-09-21T12:25:01.026 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.971889+0000", 2024-09-21T12:25:01.026 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012172600000000001 2024-09-21T12:25:01.026 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:01.026 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:25:01.026 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:01.026 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.026 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.026 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.25022.0:19119 8.b e167/160 8:d7616eb4:::benchmark_data_smithi114_104936_object19118:head v 167'1174, mlcod=167'1172)", 2024-09-21T12:25:01.026 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:25:00.969424+0000", 2024-09-21T12:25:01.026 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0026303379999999999, 2024-09-21T12:25:01.026 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003025649, 2024-09-21T12:25:01.026 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:25:01.027 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:25:01.027 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:25:01.027 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:25:01.027 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.027 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:25:01.027 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.969424+0000", 2024-09-21T12:25:01.027 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:01.027 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.027 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.027 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:25:01.027 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.969424+0000", 2024-09-21T12:25:01.027 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:01.027 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.027 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.027 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:25:01.028 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.969425+0000", 2024-09-21T12:25:01.028 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6700000000000003e-07 2024-09-21T12:25:01.028 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.028 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.028 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:25:01.028 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.969448+0000", 2024-09-21T12:25:01.028 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3281e-05 2024-09-21T12:25:01.028 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.028 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.028 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:25:01.028 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.969457+0000", 2024-09-21T12:25:01.028 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.4010000000000003e-06 2024-09-21T12:25:01.028 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.028 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.028 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:25:01.029 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.969463+0000", 2024-09-21T12:25:01.029 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8159999999999999e-06 2024-09-21T12:25:01.029 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.029 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.029 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:25:01.029 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.969616+0000", 2024-09-21T12:25:01.029 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015259200000000001 2024-09-21T12:25:01.029 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.029 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.029 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:25:01.029 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.969632+0000", 2024-09-21T12:25:01.029 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5855e-05 2024-09-21T12:25:01.029 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:01.029 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:25:01.030 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:01.030 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.030 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.030 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop_reply(client.25022.0:19121 8.3 e167/160)", 2024-09-21T12:25:01.030 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:25:00.972052+0000", 2024-09-21T12:25:01.030 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 2.7599999999999998e-06, 2024-09-21T12:25:01.030 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00044151699999999999, 2024-09-21T12:25:01.030 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:25:01.030 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:25:01.030 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "queued for pg", 2024-09-21T12:25:01.030 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:25:01.030 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.030 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:25:01.030 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.972052+0000", 2024-09-21T12:25:01.031 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:01.031 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.031 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.031 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:25:01.031 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.972052+0000", 2024-09-21T12:25:01.031 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:01.031 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.031 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.031 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:25:01.031 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.972052+0000", 2024-09-21T12:25:01.031 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1900000000000002e-07 2024-09-21T12:25:01.031 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.031 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.031 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:25:01.031 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.972056+0000", 2024-09-21T12:25:01.032 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8240000000000001e-06 2024-09-21T12:25:01.032 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.032 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.032 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:25:01.032 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.972066+0000", 2024-09-21T12:25:01.032 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7459999999999995e-06 2024-09-21T12:25:01.032 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.032 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.032 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:25:01.032 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.972073+0000", 2024-09-21T12:25:01.032 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2790000000000001e-06 2024-09-21T12:25:01.032 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:01.032 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:25:01.032 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:01.033 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.033 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.033 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.25022.0:19126 8.0 e167/160 8:0bcd5fdb:::benchmark_data_smithi114_104936_object19125:head v 167'1163, mlcod=167'1158)", 2024-09-21T12:25:01.033 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:25:00.972317+0000", 2024-09-21T12:25:01.033 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 4294967295.9997373, 2024-09-21T12:25:01.033 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00021247799999999999, 2024-09-21T12:25:01.033 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:25:01.033 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:25:01.033 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:25:01.033 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:25:01.033 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.033 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:25:01.033 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.972317+0000", 2024-09-21T12:25:01.033 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:01.033 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.034 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.034 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:25:01.034 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.972317+0000", 2024-09-21T12:25:01.034 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:01.034 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.035 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.035 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:25:01.035 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.972318+0000", 2024-09-21T12:25:01.035 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5300000000000004e-07 2024-09-21T12:25:01.035 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.035 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.035 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:25:01.035 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.972340+0000", 2024-09-21T12:25:01.035 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2344000000000002e-05 2024-09-21T12:25:01.036 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.036 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.036 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:25:01.036 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.972350+0000", 2024-09-21T12:25:01.036 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.041e-05 2024-09-21T12:25:01.036 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.036 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.036 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:25:01.036 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.972357+0000", 2024-09-21T12:25:01.036 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4849999999999997e-06 2024-09-21T12:25:01.036 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.036 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.036 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:25:01.036 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.972413+0000", 2024-09-21T12:25:01.036 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6548000000000001e-05 2024-09-21T12:25:01.037 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.037 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.037 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:25:01.037 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.972432+0000", 2024-09-21T12:25:01.037 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9001e-05 2024-09-21T12:25:01.037 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:01.037 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:25:01.037 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:01.037 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.037 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.037 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25022.0:19100 8.7 8:e763a732:::benchmark_data_smithi114_104936_object19099:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-09-21T12:25:01.037 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:25:00.962975+0000", 2024-09-21T12:25:01.037 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0090792300000000006, 2024-09-21T12:25:01.037 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0096015679999999996, 2024-09-21T12:25:01.038 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:25:01.038 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:25:01.038 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:25:01.038 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:25:01.038 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25022", 2024-09-21T12:25:01.038 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:25:01.038 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 19100 2024-09-21T12:25:01.038 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.038 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:25:01.038 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.038 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:25:01.038 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.962975+0000", 2024-09-21T12:25:01.038 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:01.038 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.039 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.039 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:25:01.039 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.962975+0000", 2024-09-21T12:25:01.039 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:01.039 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.039 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.039 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:25:01.039 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.962983+0000", 2024-09-21T12:25:01.039 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8120000000000008e-06 2024-09-21T12:25:01.039 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.039 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.039 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:25:01.039 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.963015+0000", 2024-09-21T12:25:01.039 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.1894999999999999e-05 2024-09-21T12:25:01.039 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.040 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.040 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:25:01.040 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.963031+0000", 2024-09-21T12:25:01.040 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6344000000000001e-05 2024-09-21T12:25:01.040 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.040 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.040 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:25:01.040 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.963038+0000", 2024-09-21T12:25:01.040 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.097e-06 2024-09-21T12:25:01.040 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.040 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.040 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:25:01.040 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.965562+0000", 2024-09-21T12:25:01.040 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0025240409999999999 2024-09-21T12:25:01.040 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.041 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.041 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:25:01.041 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.970403+0000", 2024-09-21T12:25:01.041 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.004840652 2024-09-21T12:25:01.041 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.041 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.041 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,6", 2024-09-21T12:25:01.041 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.970540+0000", 2024-09-21T12:25:01.041 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013653600000000001 2024-09-21T12:25:01.041 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:01.041 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:25:01.041 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:01.041 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.041 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.041 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25022.0:19101 8.7 8:e36a4104:::benchmark_data_smithi114_104936_object19100:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-09-21T12:25:01.042 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:25:00.963061+0000", 2024-09-21T12:25:01.042 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0089934500000000001, 2024-09-21T12:25:01.042 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0095738179999999996, 2024-09-21T12:25:01.042 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:25:01.042 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:25:01.042 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:25:01.042 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:25:01.042 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25022", 2024-09-21T12:25:01.042 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:25:01.042 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 19101 2024-09-21T12:25:01.042 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.043 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:25:01.043 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.043 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:25:01.043 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.963061+0000", 2024-09-21T12:25:01.043 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:01.043 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.043 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.043 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:25:01.043 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.963061+0000", 2024-09-21T12:25:01.043 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:01.043 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.043 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.043 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:25:01.043 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.963062+0000", 2024-09-21T12:25:01.044 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5000000000000003e-07 2024-09-21T12:25:01.044 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.044 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.044 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:25:01.044 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.963081+0000", 2024-09-21T12:25:01.044 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9315000000000001e-05 2024-09-21T12:25:01.044 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.044 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.044 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:25:01.044 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.963088+0000", 2024-09-21T12:25:01.044 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.289e-06 2024-09-21T12:25:01.044 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.044 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.044 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:25:01.044 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.963094+0000", 2024-09-21T12:25:01.045 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8470000000000001e-06 2024-09-21T12:25:01.045 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.045 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.045 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:25:01.045 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.971622+0000", 2024-09-21T12:25:01.045 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0085275349999999993 2024-09-21T12:25:01.045 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.045 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.045 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:25:01.045 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.971738+0000", 2024-09-21T12:25:01.045 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000116894 2024-09-21T12:25:01.045 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:01.045 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:01.045 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,6", 2024-09-21T12:25:01.045 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:00.971865+0000", 2024-09-21T12:25:01.046 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012671000000000001 2024-09-21T12:25:01.046 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:01.046 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:25:01.046 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:01.046 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:01.046 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:25:01.046 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 10 2024-09-21T12:25:01.046 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:25:01.047 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 7 16 19282 19266 171.968 139.938 0.00432908 0.00580922 2024-09-21T12:25:01.074 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:01.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:01.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:01.175 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:01.212 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:01.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:01.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:01.313 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:01.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:01.345 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:01.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:01.446 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:01.474 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:01.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:01.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:01.576 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:01.607 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:01.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:01.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:01.708 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:01.737 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:01.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:01.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:01.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:01 smithi183 ceph-mon[33277]: 4.f scrub starts 2024-09-21T12:25:01.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:01 smithi183 ceph-mon[33277]: 4.f scrub ok 2024-09-21T12:25:01.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:01 smithi183 ceph-mon[33277]: 4.b scrub starts 2024-09-21T12:25:01.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:01 smithi183 ceph-mon[33277]: 4.b scrub ok 2024-09-21T12:25:01.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:01 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1460627370' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:25:01.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:01 smithi183 ceph-mon[33277]: 3.18 scrub starts 2024-09-21T12:25:01.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:01 smithi183 ceph-mon[33277]: 3.18 scrub ok 2024-09-21T12:25:01.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:01 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:25:01.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:01 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:25:01.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:01 smithi114 ceph-mon[40867]: 4.f scrub starts 2024-09-21T12:25:01.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:01 smithi114 ceph-mon[40867]: 4.f scrub ok 2024-09-21T12:25:01.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:01 smithi114 ceph-mon[40867]: 4.b scrub starts 2024-09-21T12:25:01.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:01 smithi114 ceph-mon[40867]: 4.b scrub ok 2024-09-21T12:25:01.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:01 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1460627370' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:25:01.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:01 smithi114 ceph-mon[40867]: 3.18 scrub starts 2024-09-21T12:25:01.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:01 smithi114 ceph-mon[40867]: 3.18 scrub ok 2024-09-21T12:25:01.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:01 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:25:01.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:01 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:25:01.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:01 smithi114 ceph-mon[30324]: 4.f scrub starts 2024-09-21T12:25:01.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:01 smithi114 ceph-mon[30324]: 4.f scrub ok 2024-09-21T12:25:01.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:01 smithi114 ceph-mon[30324]: 4.b scrub starts 2024-09-21T12:25:01.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:01 smithi114 ceph-mon[30324]: 4.b scrub ok 2024-09-21T12:25:01.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:01 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1460627370' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:25:01.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:01 smithi114 ceph-mon[30324]: 3.18 scrub starts 2024-09-21T12:25:01.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:01 smithi114 ceph-mon[30324]: 3.18 scrub ok 2024-09-21T12:25:01.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:01 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:25:01.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:01 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:25:01.838 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:01.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:01.878 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:01.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:01.979 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:02.002 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:02.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:02.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:02.037 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 8 16 21453 21437 167.432 135.688 0.00366976 0.00596615 2024-09-21T12:25:02.104 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:02.130 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:02.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:02.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:02.232 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:02.260 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:02.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:02.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:02.362 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:02.401 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:02.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:02.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:02.503 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:02.533 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:02.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:02.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:02.635 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:02.662 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:02.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:02.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:02.764 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:02.775 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:25:02 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:25:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:25:02.803 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:02.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:02.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:02.905 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:02.942 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:02.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:02.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:03.038 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 9 16 23865 23849 165.577 150.75 0.00313817 0.00603352 2024-09-21T12:25:03.044 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:03.080 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:03.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:03.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:03.133 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:02 smithi114 ceph-mon[30324]: pgmap v223: 158 pgs: 158 active+clean; 773 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-09-21T12:25:03.133 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:02 smithi114 ceph-mon[30324]: osdmap e168: 8 total, 8 up, 7 in 2024-09-21T12:25:03.133 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:02 smithi114 ceph-mon[30324]: 4.19 scrub starts 2024-09-21T12:25:03.133 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:02 smithi114 ceph-mon[30324]: 5.1a scrub starts 2024-09-21T12:25:03.133 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:02 smithi114 ceph-mon[30324]: 5.1a scrub ok 2024-09-21T12:25:03.133 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:02 smithi114 ceph-mon[30324]: 4.19 scrub ok 2024-09-21T12:25:03.133 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:02 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:25:03.133 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:02 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:25:03.134 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:02 smithi114 ceph-mon[40867]: pgmap v223: 158 pgs: 158 active+clean; 773 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-09-21T12:25:03.134 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:02 smithi114 ceph-mon[40867]: osdmap e168: 8 total, 8 up, 7 in 2024-09-21T12:25:03.134 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:02 smithi114 ceph-mon[40867]: 4.19 scrub starts 2024-09-21T12:25:03.134 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:02 smithi114 ceph-mon[40867]: 5.1a scrub starts 2024-09-21T12:25:03.134 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:02 smithi114 ceph-mon[40867]: 5.1a scrub ok 2024-09-21T12:25:03.134 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:02 smithi114 ceph-mon[40867]: 4.19 scrub ok 2024-09-21T12:25:03.134 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:02 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:25:03.134 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:02 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:25:03.147 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:03.147 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:03.181 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:03.203 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T12:25:03.207 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:03.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:03.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:03.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:02 smithi183 ceph-mon[33277]: pgmap v223: 158 pgs: 158 active+clean; 773 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-09-21T12:25:03.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:02 smithi183 ceph-mon[33277]: osdmap e168: 8 total, 8 up, 7 in 2024-09-21T12:25:03.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:02 smithi183 ceph-mon[33277]: 4.19 scrub starts 2024-09-21T12:25:03.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:02 smithi183 ceph-mon[33277]: 5.1a scrub starts 2024-09-21T12:25:03.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:02 smithi183 ceph-mon[33277]: 5.1a scrub ok 2024-09-21T12:25:03.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:02 smithi183 ceph-mon[33277]: 4.19 scrub ok 2024-09-21T12:25:03.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:02 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:25:03.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:02 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:25:03.309 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:03.340 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:03.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:03.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:03.442 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:03.470 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:03.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:03.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:03.572 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:03.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:03.606 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:03.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:03.707 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:03.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:03.781 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:03.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:03.882 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:03.912 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:03.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:03.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:03.920 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:03 smithi183 ceph-mon[33277]: 5.16 scrub starts 2024-09-21T12:25:03.921 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:03 smithi183 ceph-mon[33277]: 5.16 scrub ok 2024-09-21T12:25:03.921 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:03 smithi183 ceph-mon[33277]: osdmap e169: 8 total, 8 up, 7 in 2024-09-21T12:25:03.921 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:03 smithi183 ceph-mon[33277]: 1.4 scrub starts 2024-09-21T12:25:03.921 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:03 smithi183 ceph-mon[33277]: 1.4 scrub ok 2024-09-21T12:25:03.921 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:03 smithi183 ceph-mon[33277]: 5.9 scrub starts 2024-09-21T12:25:03.921 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:03 smithi183 ceph-mon[33277]: 5.9 scrub ok 2024-09-21T12:25:04.014 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:04.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:03 smithi114 ceph-mon[30324]: 5.16 scrub starts 2024-09-21T12:25:04.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:03 smithi114 ceph-mon[30324]: 5.16 scrub ok 2024-09-21T12:25:04.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:03 smithi114 ceph-mon[30324]: osdmap e169: 8 total, 8 up, 7 in 2024-09-21T12:25:04.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:03 smithi114 ceph-mon[30324]: 1.4 scrub starts 2024-09-21T12:25:04.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:03 smithi114 ceph-mon[30324]: 1.4 scrub ok 2024-09-21T12:25:04.027 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:03 smithi114 ceph-mon[30324]: 5.9 scrub starts 2024-09-21T12:25:04.027 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:03 smithi114 ceph-mon[30324]: 5.9 scrub ok 2024-09-21T12:25:04.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:03 smithi114 ceph-mon[40867]: 5.16 scrub starts 2024-09-21T12:25:04.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:03 smithi114 ceph-mon[40867]: 5.16 scrub ok 2024-09-21T12:25:04.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:03 smithi114 ceph-mon[40867]: osdmap e169: 8 total, 8 up, 7 in 2024-09-21T12:25:04.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:03 smithi114 ceph-mon[40867]: 1.4 scrub starts 2024-09-21T12:25:04.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:03 smithi114 ceph-mon[40867]: 1.4 scrub ok 2024-09-21T12:25:04.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:03 smithi114 ceph-mon[40867]: 5.9 scrub starts 2024-09-21T12:25:04.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:03 smithi114 ceph-mon[40867]: 5.9 scrub ok 2024-09-21T12:25:04.037 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 10 16 25980 25964 162.237 132.188 0.00650078 0.00615808 2024-09-21T12:25:04.052 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:04.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:04.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:04.154 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:04.180 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:04.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:04.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:04.282 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:04.302 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:04.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:04.330 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:04.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:04.431 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:04.462 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:04.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:04.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:04.465 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:25:04.564 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:04.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:04.626 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:04.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:04.727 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:04.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:04.773 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:04.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:04.873 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-21T12:25:04.874 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:04.885 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T12:25:04.901 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:04.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:04.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:04.991 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:25:04.991 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:25:04.991 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:25:04.992 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:25:04.992 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:25:05.003 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:05.031 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:05.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:05.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:05.038 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 11 16 27714 27698 157.341 108.375 0.00435752 0.00634949 2024-09-21T12:25:05.132 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:05.137 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:04 smithi114 ceph-mon[30324]: 4.8 scrub starts 2024-09-21T12:25:05.137 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:04 smithi114 ceph-mon[30324]: 4.8 scrub ok 2024-09-21T12:25:05.137 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:04 smithi114 ceph-mon[30324]: pgmap v226: 158 pgs: 158 active+clean; 1.3 GiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 751 B/s rd, 151 MiB/s wr, 2.42k op/s 2024-09-21T12:25:05.137 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:04 smithi114 ceph-mon[30324]: 4.c scrub starts 2024-09-21T12:25:05.138 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:04 smithi114 ceph-mon[30324]: 4.c scrub ok 2024-09-21T12:25:05.138 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:04 smithi114 ceph-mon[30324]: 1.2 scrub starts 2024-09-21T12:25:05.138 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:04 smithi114 ceph-mon[30324]: 1.2 scrub ok 2024-09-21T12:25:05.138 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:04 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1190966336' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:25:05.138 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:04 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:25:05.138 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:04 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1763257485' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:25:05.138 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:04 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:25:05.138 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:04 smithi114 ceph-mon[40867]: 4.8 scrub starts 2024-09-21T12:25:05.138 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:04 smithi114 ceph-mon[40867]: 4.8 scrub ok 2024-09-21T12:25:05.138 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:04 smithi114 ceph-mon[40867]: pgmap v226: 158 pgs: 158 active+clean; 1.3 GiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 751 B/s rd, 151 MiB/s wr, 2.42k op/s 2024-09-21T12:25:05.138 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:04 smithi114 ceph-mon[40867]: 4.c scrub starts 2024-09-21T12:25:05.138 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:04 smithi114 ceph-mon[40867]: 4.c scrub ok 2024-09-21T12:25:05.139 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:04 smithi114 ceph-mon[40867]: 1.2 scrub starts 2024-09-21T12:25:05.139 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:04 smithi114 ceph-mon[40867]: 1.2 scrub ok 2024-09-21T12:25:05.139 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:04 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1190966336' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:25:05.139 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:04 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:25:05.139 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:04 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1763257485' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:25:05.139 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:04 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:25:05.169 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:05.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:05.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:05.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:04 smithi183 ceph-mon[33277]: 4.8 scrub starts 2024-09-21T12:25:05.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:04 smithi183 ceph-mon[33277]: 4.8 scrub ok 2024-09-21T12:25:05.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:04 smithi183 ceph-mon[33277]: pgmap v226: 158 pgs: 158 active+clean; 1.3 GiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 751 B/s rd, 151 MiB/s wr, 2.42k op/s 2024-09-21T12:25:05.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:04 smithi183 ceph-mon[33277]: 4.c scrub starts 2024-09-21T12:25:05.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:04 smithi183 ceph-mon[33277]: 4.c scrub ok 2024-09-21T12:25:05.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:04 smithi183 ceph-mon[33277]: 1.2 scrub starts 2024-09-21T12:25:05.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:04 smithi183 ceph-mon[33277]: 1.2 scrub ok 2024-09-21T12:25:05.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:04 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1190966336' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:25:05.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:04 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:25:05.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:04 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1763257485' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:25:05.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:04 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:25:05.271 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:05.309 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:05.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:05.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:05.410 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:05.438 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:05.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:05.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:05.540 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:05.578 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:05.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:05.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:05.680 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:05.710 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:05.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:05.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:05.812 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:05.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:05.833 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:05.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:05.913 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T12:25:05.935 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:06.002 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:06.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:06.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:06.038 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 12 16 29754 29738 154.853 127.5 0.00975826 0.00645107 2024-09-21T12:25:06.104 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:06.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:06.147 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:06.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:06.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:05 smithi183 ceph-mon[33277]: 5.18 scrub starts 2024-09-21T12:25:06.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:05 smithi183 ceph-mon[33277]: 5.18 scrub ok 2024-09-21T12:25:06.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:05 smithi183 ceph-mon[33277]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:25:06.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:05 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T12:25:06.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:05 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:25:06.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:05 smithi183 ceph-mon[33277]: osdmap e170: 8 total, 8 up, 7 in 2024-09-21T12:25:06.248 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:06.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:05 smithi114 ceph-mon[30324]: 5.18 scrub starts 2024-09-21T12:25:06.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:05 smithi114 ceph-mon[30324]: 5.18 scrub ok 2024-09-21T12:25:06.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:05 smithi114 ceph-mon[30324]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:25:06.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:05 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T12:25:06.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:05 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:25:06.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:05 smithi114 ceph-mon[30324]: osdmap e170: 8 total, 8 up, 7 in 2024-09-21T12:25:06.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:05 smithi114 ceph-mon[40867]: 5.18 scrub starts 2024-09-21T12:25:06.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:05 smithi114 ceph-mon[40867]: 5.18 scrub ok 2024-09-21T12:25:06.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:05 smithi114 ceph-mon[40867]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:25:06.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:05 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T12:25:06.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:05 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:25:06.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:05 smithi114 ceph-mon[40867]: osdmap e170: 8 total, 8 up, 7 in 2024-09-21T12:25:06.278 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:06.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:06.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:06.380 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:06.410 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:06.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:06.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:06.512 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:06.539 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:06.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:06.541 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T12:25:06.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:06.642 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:06.671 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:06.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:06.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:06.773 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:06.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:06.811 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:06.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:06.912 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:06.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:06.934 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:06.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:07.035 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:07.039 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 13 16 31548 31532 151.566 112.125 0.00382686 0.00659211 2024-09-21T12:25:07.073 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:07.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:07.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:07.176 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:07.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:07.199 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:07.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:07.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:06 smithi183 ceph-mon[33277]: pgmap v228: 158 pgs: 158 active+clean; 1.3 GiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 131 MiB/s wr, 2.10k op/s 2024-09-21T12:25:07.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:06 smithi183 ceph-mon[33277]: osdmap e171: 8 total, 8 up, 7 in 2024-09-21T12:25:07.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:25:06 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:25:07.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:06 smithi114 ceph-mon[30324]: pgmap v228: 158 pgs: 158 active+clean; 1.3 GiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 131 MiB/s wr, 2.10k op/s 2024-09-21T12:25:07.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:06 smithi114 ceph-mon[30324]: osdmap e171: 8 total, 8 up, 7 in 2024-09-21T12:25:07.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:06 smithi114 ceph-mon[40867]: pgmap v228: 158 pgs: 158 active+clean; 1.3 GiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 131 MiB/s wr, 2.10k op/s 2024-09-21T12:25:07.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:06 smithi114 ceph-mon[40867]: osdmap e171: 8 total, 8 up, 7 in 2024-09-21T12:25:07.299 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:07.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:07.325 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:07.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:07.426 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:07.455 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:07.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:07.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:07.557 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:07.578 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:07.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:07.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:07.680 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:07.702 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:07.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:07.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:07.803 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:07.834 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:07.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:07.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:07.936 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:07.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:07.958 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:07.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:08.038 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 14 15 32886 32871 146.718 83.6875 0.00501502 0.00680772 2024-09-21T12:25:08.058 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:08.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:08.089 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:08.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:08.190 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:08.227 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:08.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:08.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:08.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:07 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:25:08.245 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:07 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:25:08.245 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:07 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:25:08.329 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:08.353 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:08.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:08.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:08.454 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:08.492 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:08.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:08.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:08.594 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:08.621 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:08.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:08.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:08.723 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:08.760 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:08.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:08.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:08.862 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:08.892 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:08.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:08.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:08.993 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:09.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:09.015 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:09.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:09.038 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 15 16 34521 34505 143.744 102.125 0.00337647 0.00691228 2024-09-21T12:25:09.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:09.151 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:09.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:09.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:09.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:08 smithi183 ceph-mon[33277]: pgmap v230: 158 pgs: 158 active+clean; 1.9 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 881 B/s rd, 169 MiB/s wr, 2.70k op/s 2024-09-21T12:25:09.253 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:09.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:08 smithi114 ceph-mon[40867]: pgmap v230: 158 pgs: 158 active+clean; 1.9 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 881 B/s rd, 169 MiB/s wr, 2.70k op/s 2024-09-21T12:25:09.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:08 smithi114 ceph-mon[30324]: pgmap v230: 158 pgs: 158 active+clean; 1.9 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 881 B/s rd, 169 MiB/s wr, 2.70k op/s 2024-09-21T12:25:09.283 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:09.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:09.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:09.385 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:09.411 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:09.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:09.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:09.513 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:09.542 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:09.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:09.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:09.644 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:09.676 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:09.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:09.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:09.778 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:09.815 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:09.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:09.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:09.917 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:09.952 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:09.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:09.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:10.038 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 16 16 36117 36101 140.994 99.75 0.00185872 0.00705618 2024-09-21T12:25:10.055 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:10.076 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:10.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:10.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:10.178 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:10.215 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:10.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:10.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:10.317 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:10.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:10.342 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:10.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:10.443 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:10.478 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:10.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:10.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:10.579 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:10.614 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:10.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:10.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:10.715 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:10.749 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:10.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:10.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:10.851 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:10.874 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:10.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:10.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:10.975 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:11.009 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:11.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:11.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:11.038 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 17 16 37644 37628 138.314 95.4375 0.00571464 0.00722483 2024-09-21T12:25:11.111 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:11.147 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:11.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:11.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:11.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:10 smithi183 ceph-mon[33277]: pgmap v231: 158 pgs: 158 active+clean; 1.9 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 753 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-09-21T12:25:11.249 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:11.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:10 smithi114 ceph-mon[40867]: pgmap v231: 158 pgs: 158 active+clean; 1.9 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 753 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-09-21T12:25:11.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:10 smithi114 ceph-mon[30324]: pgmap v231: 158 pgs: 158 active+clean; 1.9 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 753 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-09-21T12:25:11.287 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:11.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:11.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:11.389 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:11.410 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:11.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:11.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:11.511 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:11.539 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:11.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:11.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:11.641 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:11.671 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:11.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:11.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:11.773 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:11.795 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:11.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:11.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:11.896 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:11.937 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:11.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:11.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:12.038 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 18 16 39019 39003 135.404 85.9375 0.00730812 0.00738049 2024-09-21T12:25:12.039 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:12.064 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:12.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:12.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:12.165 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:12.211 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:12.222 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:12.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:12.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:12.296 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:12.299 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:12.324 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:12.369 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:12.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:12.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:12.471 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:12.499 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:12.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:12.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:12.601 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:12.650 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:12.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:12.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:12.670 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:25:12 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:25:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:25:12.751 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:12.789 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:12.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:12.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:12.891 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:12.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:12.929 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:12.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:12.950 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:12 smithi114 ceph-mon[30324]: pgmap v232: 158 pgs: 158 active+clean; 2.0 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.56k op/s 2024-09-21T12:25:12.950 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:12 smithi114 ceph-mon[30324]: osdmap e172: 8 total, 8 up, 7 in 2024-09-21T12:25:12.950 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:12 smithi114 ceph-mon[40867]: pgmap v232: 158 pgs: 158 active+clean; 2.0 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.56k op/s 2024-09-21T12:25:12.950 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:12 smithi114 ceph-mon[40867]: osdmap e172: 8 total, 8 up, 7 in 2024-09-21T12:25:13.031 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:13.038 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 19 16 40768 40752 134.031 109.312 0.0152754 0.00745579 2024-09-21T12:25:13.059 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:13.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:13.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:13.139 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:25:13.139 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:25:13.161 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:13.175 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:25:13.176 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:25:13.210 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:25:13.210 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:25:13.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:13.215 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:13.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:13.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:12 smithi183 ceph-mon[33277]: pgmap v232: 158 pgs: 158 active+clean; 2.0 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.56k op/s 2024-09-21T12:25:13.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:12 smithi183 ceph-mon[33277]: osdmap e172: 8 total, 8 up, 7 in 2024-09-21T12:25:13.245 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:25:13.245 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:25:13.282 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:25:13.283 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:25:13.316 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:13.321 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:25:13.321 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:25:13.353 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:25:13.353 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:25:13.356 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:13.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:13.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:13.390 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:25:13.390 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:25:13.458 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:13.485 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:13.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:13.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:13.587 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:13.615 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:13.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:13.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:13.717 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:13.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:13.754 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:13.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:13.855 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:13.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:13.901 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:13.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:13.986 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:25:13.986 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:25:14.002 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:14.038 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-21T12:25:14.038284+0000 min lat: 0.00121483 max lat: 0.158038 avg lat: 0.0075522 2024-09-21T12:25:14.039 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T12:25:14.039 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 20 16 42361 42345 132.307 99.5625 0.00768402 0.0075522 2024-09-21T12:25:14.046 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:14.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:14.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:14.148 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:14.180 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T12:25:14.186 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:14.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:14.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:14.222 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:13 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1627600113' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:25:14.222 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:13 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1627600113' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:25:14.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:13 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1627600113' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:25:14.289 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:14.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:14.310 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:14.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:14.411 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:14.454 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:14.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:14.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:14.555 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:14.584 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:14.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:14.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:14.686 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:14.716 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:14.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:14.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:14.817 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:14.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:14.856 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:14.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:14.893 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T12:25:14.957 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:14.987 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:14.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:14.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:15.039 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 21 16 43932 43916 130.682 98.1875 0.00383618 0.00764737 2024-09-21T12:25:15.089 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:15.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:15.134 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:15.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:15.235 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:15.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:14 smithi183 ceph-mon[33277]: pgmap v234: 158 pgs: 158 active+clean; 2.4 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 138 MiB/s wr, 2.21k op/s 2024-09-21T12:25:15.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:14 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1772507882' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:25:15.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:14 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:25:15.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:14 smithi114 ceph-mon[30324]: pgmap v234: 158 pgs: 158 active+clean; 2.4 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 138 MiB/s wr, 2.21k op/s 2024-09-21T12:25:15.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:14 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1772507882' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:25:15.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:14 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:25:15.243 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:14 smithi114 ceph-mon[40867]: pgmap v234: 158 pgs: 158 active+clean; 2.4 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 138 MiB/s wr, 2.21k op/s 2024-09-21T12:25:15.243 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:14 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1772507882' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:25:15.243 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:14 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:25:15.296 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T12:25:15.301 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:15.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:15.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:15.403 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:15.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:15.472 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:15.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:15.573 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:15.599 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:15.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:15.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:15.700 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:15.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:15.749 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:15.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:15.850 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:15.896 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:15.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:15.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:15.978 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T12:25:15.999 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:16.039 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 22 15 45388 45373 128.881 91.0625 0.0180575 0.00775194 2024-09-21T12:25:16.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:16.077 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:16.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:16.178 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:16.207 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:16.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:16.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:16.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:15 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T12:25:16.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:15 smithi183 ceph-mon[33277]: osdmap e173: 8 total, 8 up, 7 in 2024-09-21T12:25:16.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:15 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:25:16.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:15 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:25:16.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:15 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:25:16.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:15 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T12:25:16.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:15 smithi114 ceph-mon[30324]: osdmap e173: 8 total, 8 up, 7 in 2024-09-21T12:25:16.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:15 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:25:16.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:15 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:25:16.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:15 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:25:16.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:15 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T12:25:16.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:15 smithi114 ceph-mon[40867]: osdmap e173: 8 total, 8 up, 7 in 2024-09-21T12:25:16.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:15 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:25:16.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:15 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:25:16.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:15 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:25:16.308 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:16.335 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:16.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:16.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:16.436 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:16.462 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:16.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:16.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:16.564 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:16.601 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:16.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:16.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:16.703 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:16.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:16.774 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:16.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:16.791 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:16.796 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:16.843 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:16.876 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:16.894 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:16 smithi114 ceph-mon[30324]: pgmap v236: 158 pgs: 158 active+clean; 2.4 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 59 MiB/s wr, 938 op/s 2024-09-21T12:25:16.895 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:16 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:25:16.899 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:16.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:16.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:17.001 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:17.028 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:17.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:17.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:17.039 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 23 16 46778 46762 127.052 86.8125 0.00250527 0.00785937 2024-09-21T12:25:17.130 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:17.159 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:17.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:17.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:17.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:25:16 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:25:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:16 smithi183 ceph-mon[33277]: pgmap v236: 158 pgs: 158 active+clean; 2.4 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 59 MiB/s wr, 938 op/s 2024-09-21T12:25:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:16 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:25:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:16 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:25:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:16 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:25:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:16 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:25:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:16 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:25:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:16 smithi183 ceph-mon[33277]: osdmap e174: 8 total, 8 up, 7 in 2024-09-21T12:25:17.260 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:17.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:16 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:25:17.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:16 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:25:17.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:16 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:25:17.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:16 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:25:17.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:16 smithi114 ceph-mon[30324]: osdmap e174: 8 total, 8 up, 7 in 2024-09-21T12:25:17.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:16 smithi114 ceph-mon[40867]: pgmap v236: 158 pgs: 158 active+clean; 2.4 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 59 MiB/s wr, 938 op/s 2024-09-21T12:25:17.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:16 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:25:17.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:16 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:25:17.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:16 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:25:17.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:16 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:25:17.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:16 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:25:17.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:16 smithi114 ceph-mon[40867]: osdmap e174: 8 total, 8 up, 7 in 2024-09-21T12:25:17.289 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:17.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:17.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:17.391 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:17.412 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:17.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:17.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:17.513 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:17.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:17.542 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:17.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:17.643 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:17.674 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:17.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:17.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:17.776 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:17.806 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:17.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:17.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:17.908 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:17.947 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:17.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:17.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:18.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:17 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:25:18.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:17 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:25:18.039 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 24 16 48372 48356 125.909 99.625 0.00216564 0.0079317 2024-09-21T12:25:18.049 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:18.071 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:18.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:18.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:18.173 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:18.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:18.209 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:18.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:18.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:17 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:25:18.310 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:18.331 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:18.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:18.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:18.433 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:18.470 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:18.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:18.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:18.572 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:18.607 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:18.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:18.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:18.659 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:25:18.709 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:18.735 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:18.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:18.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:18.837 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:18.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:18.873 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:18.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:18.974 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:19.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:19.002 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:19.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:19.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:18 smithi114 ceph-mon[30324]: pgmap v238: 158 pgs: 158 active+clean; 2.8 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 136 MiB/s wr, 2.17k op/s 2024-09-21T12:25:19.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:18 smithi114 ceph-mon[40867]: pgmap v238: 158 pgs: 158 active+clean; 2.8 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 136 MiB/s wr, 2.17k op/s 2024-09-21T12:25:19.039 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 25 16 49716 49700 124.232 84 0.0544072 0.00803868 2024-09-21T12:25:19.103 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:19.129 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:19.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:19.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:19.230 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:19.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:18 smithi183 ceph-mon[33277]: pgmap v238: 158 pgs: 158 active+clean; 2.8 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 136 MiB/s wr, 2.17k op/s 2024-09-21T12:25:19.258 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:19.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:19.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:19.360 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:19.394 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:19.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:19.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:19.495 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:19.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:19.522 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:19.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:19.623 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:19.659 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:19.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:19.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:19.760 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:19.789 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:19.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:19.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:19.891 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:19.916 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:19.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:19.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:20.018 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:20.039 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 26 16 51289 51273 123.235 98.3125 0.0165056 0.00810932 2024-09-21T12:25:20.046 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:20.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:20.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:20.147 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:20.185 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:20.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:20.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:20.287 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:20.324 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:20.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:20.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:20.426 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:20.452 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:20.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:20.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:20.544 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:20.553 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:20.593 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:20.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:20.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:20.695 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:20.722 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:20.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:20.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:20.824 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:20.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:20.866 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:20.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:20.934 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T12:25:20.935 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:20 smithi114 ceph-mon[40867]: pgmap v239: 158 pgs: 158 active+clean; 2.8 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 655 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-21T12:25:20.935 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:20 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1716291248' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:25:20.935 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:20 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:25:20.935 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:20 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2831748617' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:25:20.935 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:20 smithi114 ceph-mon[30324]: pgmap v239: 158 pgs: 158 active+clean; 2.8 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 655 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-21T12:25:20.935 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:20 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1716291248' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:25:20.935 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:20 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:25:20.935 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:20 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2831748617' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:25:20.941 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-21T12:25:20.967 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:20.993 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:20.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:20.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:21.039 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 27 16 52613 52597 121.735 82.75 0.00474475 0.00820905 2024-09-21T12:25:21.095 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:21.136 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:21.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:21.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:21.237 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:21.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:20 smithi183 ceph-mon[33277]: pgmap v239: 158 pgs: 158 active+clean; 2.8 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 655 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-21T12:25:21.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:20 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1716291248' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:25:21.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:20 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:25:21.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:20 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2831748617' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:25:21.264 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:21.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:21.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:21.366 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:21.391 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:21.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:21.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:21.493 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:21.519 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:21.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:21.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:21.553 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:25:21.553 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:25:21.596 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:25:21.596 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:25:21.621 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:21.638 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:25:21.639 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:25:21.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:21.657 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:21.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:21.674 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:25:21.674 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:25:21.705 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:25:21.706 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:25:21.741 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:25:21.742 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:25:21.758 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:21.775 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:25:21.775 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:25:21.784 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:21.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:21.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:21.808 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:25:21.808 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:25:21.885 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:21.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:21.923 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:21.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:21.924 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:21 smithi114 ceph-mon[40867]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:25:21.925 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:21 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:25:21.925 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:21 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2831748617' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T12:25:22.023 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:22.039 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 28 16 54062 54046 120.621 90.5625 0.00806707 0.00828589 2024-09-21T12:25:22.062 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:22.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:22.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:22.155 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:25:22.156 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-21T12:25:22.156 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:22.156 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.25022.0:53988 8.f e176/160 8:f18f8f57:::benchmark_data_smithi114_104936_object53987:head v 176'3447, mlcod=176'3443)", 2024-09-21T12:25:22.156 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:25:21.988836+0000", 2024-09-21T12:25:22.156 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0013365600000000001, 2024-09-21T12:25:22.156 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0013710700000000001, 2024-09-21T12:25:22.156 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:25:22.156 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:25:22.156 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:25:22.156 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:25:22.156 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:22.156 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:25:22.156 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:21.988836+0000", 2024-09-21T12:25:22.157 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:22.157 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:22.157 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:22.157 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:25:22.157 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:21.988836+0000", 2024-09-21T12:25:22.157 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:22.157 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:22.157 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:22.157 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:25:22.157 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:21.988837+0000", 2024-09-21T12:25:22.157 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.9800000000000002e-07 2024-09-21T12:25:22.157 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:22.157 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:22.157 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:25:22.157 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:21.988860+0000", 2024-09-21T12:25:22.158 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3213999999999998e-05 2024-09-21T12:25:22.158 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:22.158 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:22.158 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:25:22.158 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:21.988871+0000", 2024-09-21T12:25:22.158 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0699999999999999e-05 2024-09-21T12:25:22.158 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:22.158 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:22.158 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:25:22.158 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:21.988887+0000", 2024-09-21T12:25:22.158 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5824999999999998e-05 2024-09-21T12:25:22.158 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:22.158 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:22.158 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:25:22.158 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:21.989018+0000", 2024-09-21T12:25:22.159 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013118900000000001 2024-09-21T12:25:22.159 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:22.159 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:22.159 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:25:22.159 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:21.989042+0000", 2024-09-21T12:25:22.159 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.4196999999999999e-05 2024-09-21T12:25:22.159 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:22.159 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:25:22.159 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:22.159 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:22.159 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:25:22.159 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 1 2024-09-21T12:25:22.159 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:25:22.164 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:22.202 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:22.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:22.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:22.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:21 smithi183 ceph-mon[33277]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:25:22.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:21 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:25:22.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:21 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2831748617' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T12:25:22.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:21 smithi183 ceph-mon[33277]: osdmap e175: 8 total, 8 up, 7 in 2024-09-21T12:25:22.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:21 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1156676602' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:25:22.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:21 smithi183 ceph-mon[33277]: osdmap e176: 8 total, 8 up, 7 in 2024-09-21T12:25:22.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:21 smithi114 ceph-mon[30324]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:25:22.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:21 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:25:22.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:21 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2831748617' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T12:25:22.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:21 smithi114 ceph-mon[30324]: osdmap e175: 8 total, 8 up, 7 in 2024-09-21T12:25:22.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:21 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1156676602' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:25:22.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:21 smithi114 ceph-mon[30324]: osdmap e176: 8 total, 8 up, 7 in 2024-09-21T12:25:22.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:21 smithi114 ceph-mon[40867]: osdmap e175: 8 total, 8 up, 7 in 2024-09-21T12:25:22.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:21 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1156676602' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:25:22.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:21 smithi114 ceph-mon[40867]: osdmap e176: 8 total, 8 up, 7 in 2024-09-21T12:25:22.304 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:22.344 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:22.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:22.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:22.446 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:22.486 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:22.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:22.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:22.587 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:22.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:22.624 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:22.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:22.725 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:22.754 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:22.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:22.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:22.774 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:25:22 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:25:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:25:22.845 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T12:25:22.856 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:22.930 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:22.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:22.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:23.032 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:23.039 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 29 16 55602 55586 119.781 96.25 0.00646297 0.00834237 2024-09-21T12:25:23.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:23.077 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:23.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:23.178 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:23.208 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:23.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:23.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:23.225 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:22 smithi114 ceph-mon[30324]: pgmap v241: 158 pgs: 158 active+clean; 3.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.5 KiB/s rd, 93 MiB/s wr, 1.48k op/s 2024-09-21T12:25:23.225 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:22 smithi114 ceph-mon[40867]: pgmap v241: 158 pgs: 158 active+clean; 3.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.5 KiB/s rd, 93 MiB/s wr, 1.48k op/s 2024-09-21T12:25:23.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:22 smithi183 ceph-mon[33277]: pgmap v241: 158 pgs: 158 active+clean; 3.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.5 KiB/s rd, 93 MiB/s wr, 1.48k op/s 2024-09-21T12:25:23.310 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:23.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:23.341 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:23.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:23.441 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:23.479 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:23.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:23.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:23.581 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:23.619 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:23.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:23.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:23.722 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:23.746 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T12:25:23.749 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:23.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:23.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:23.851 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:23.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:23.895 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:23.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:23.996 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:24.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:24.037 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:24.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:24.039 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 30 16 57038 57022 118.779 89.75 0.0134958 0.00841399 2024-09-21T12:25:24.138 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:24.163 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:24.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:24.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:24.227 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:24.265 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:24.295 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:24.325 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:24.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:24.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:24.426 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:24.467 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:24.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:24.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:24.569 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:24.614 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:24.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:24.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:24.715 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:24.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:24.755 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:24.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:24.856 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:24.893 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:24.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:24.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:24.996 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:25.039 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:25.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:25.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:25.040 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 31 16 58494 58478 117.881 91 0.0143016 0.00847774 2024-09-21T12:25:25.141 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:25.188 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:25.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:25.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:25.226 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:24 smithi114 ceph-mon[30324]: pgmap v243: 158 pgs: 158 active+clean; 3.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 752 B/s rd, 98 MiB/s wr, 1.56k op/s 2024-09-21T12:25:25.226 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:24 smithi114 ceph-mon[40867]: pgmap v243: 158 pgs: 158 active+clean; 3.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 752 B/s rd, 98 MiB/s wr, 1.56k op/s 2024-09-21T12:25:25.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:24 smithi183 ceph-mon[33277]: pgmap v243: 158 pgs: 158 active+clean; 3.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 752 B/s rd, 98 MiB/s wr, 1.56k op/s 2024-09-21T12:25:25.267 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:25:25.267 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:25:25.267 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:25:25.267 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:25:25.267 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:25:25.287 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:25:25.289 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:25.314 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:25.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:25.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:25.416 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:25.440 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:25.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:25.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:25.541 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:25.567 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:25.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:25.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:25.669 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:25.696 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:25.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:25.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:25.798 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:25.830 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:25.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:25.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:25.931 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:25.951 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T12:25:25.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:25 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:25:25.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:25 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:25:25.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:25 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4214820820' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:25:25.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:25.974 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:25.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:26.040 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 32 16 59777 59761 116.703 80.1875 0.00485865 0.00856376 2024-09-21T12:25:26.075 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:26.106 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:26.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:26.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:26.174 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T12:25:26.203 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T12:25:26.208 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:26.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:25 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:25:26.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:25 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:25:26.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:25 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/4214820820' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:25:26.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:25 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:25:26.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:25 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:25:26.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:25 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/4214820820' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:25:26.292 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:26.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:26.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:26.394 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:26.419 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:26.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:26.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:26.521 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:26.579 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:26.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:26.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:26.681 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:26.710 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:26.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:26.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:26.811 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:26.849 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:26.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:26.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:26.928 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:26.950 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:26.955 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T12:25:26.955 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:25:26.975 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:26.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:26.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:27.041 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 33 16 61067 61051 115.608 80.625 0.00720613 0.00864404 2024-09-21T12:25:27.042 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:27.077 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:27.101 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:27.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:27.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:27.147 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:26 smithi114 ceph-mon[40867]: pgmap v244: 158 pgs: 158 active+clean; 3.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 57 MiB/s wr, 911 op/s 2024-09-21T12:25:27.148 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:26 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/4214820820' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T12:25:27.148 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:26 smithi114 ceph-mon[40867]: osdmap e177: 8 total, 8 up, 7 in 2024-09-21T12:25:27.148 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:26 smithi114 ceph-mon[40867]: osdmap e178: 8 total, 8 up, 7 in 2024-09-21T12:25:27.148 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:26 smithi114 ceph-mon[30324]: pgmap v244: 158 pgs: 158 active+clean; 3.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 57 MiB/s wr, 911 op/s 2024-09-21T12:25:27.148 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:26 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4214820820' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T12:25:27.148 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:26 smithi114 ceph-mon[30324]: osdmap e177: 8 total, 8 up, 7 in 2024-09-21T12:25:27.148 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:26 smithi114 ceph-mon[30324]: osdmap e178: 8 total, 8 up, 7 in 2024-09-21T12:25:27.203 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:27.224 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:27.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:27.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:27.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:25:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:25:27.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:26 smithi183 ceph-mon[33277]: pgmap v244: 158 pgs: 158 active+clean; 3.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 57 MiB/s wr, 911 op/s 2024-09-21T12:25:27.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:26 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/4214820820' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T12:25:27.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:26 smithi183 ceph-mon[33277]: osdmap e177: 8 total, 8 up, 7 in 2024-09-21T12:25:27.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:26 smithi183 ceph-mon[33277]: osdmap e178: 8 total, 8 up, 7 in 2024-09-21T12:25:27.326 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:27.372 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:27.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:27.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:27.474 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:27.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:27.515 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:27.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:27.616 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:27.650 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:27.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:27.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:27.752 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:27.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:27.805 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:27.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:27.906 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:27.909 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:27.941 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:25:27.942 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:25:27.959 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:27.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:27.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:27.973 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:25:27.973 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:25:28.010 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:25:28.011 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:25:28.041 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:25:28.041 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:25:28.042 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 34 16 62352 62336 114.569 80.3125 0.018048 0.00872281 2024-09-21T12:25:28.061 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:28.072 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:25:28.073 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:25:28.088 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:28.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:28.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:28.103 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:25:28.103 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:25:28.134 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:25:28.134 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:25:28.165 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:27 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:25:28.165 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:27 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3696320477' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:25:28.166 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:25:28.166 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:25:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:27 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:25:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:27 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3696320477' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:25:28.191 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:28.218 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:28.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:28.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:28.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:27 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:25:28.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:27 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3696320477' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:25:28.320 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:28.361 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:28.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:28.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:28.463 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:28.508 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:28.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:28.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:28.610 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:28.637 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:28.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:28.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:28.738 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:28.774 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:28.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:28.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:28.876 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:28.919 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:28.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:28.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:29.021 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:29.043 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 35 16 63750 63734 113.785 87.375 0.00791547 0.00878257 2024-09-21T12:25:29.065 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:29.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:29.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:29.167 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:29.176 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:29.192 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:29.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:29.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:29.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:28 smithi183 ceph-mon[33277]: pgmap v247: 158 pgs: 158 active+clean; 3.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 852 B/s rd, 126 MiB/s wr, 2.02k op/s 2024-09-21T12:25:29.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:28 smithi114 ceph-mon[40867]: pgmap v247: 158 pgs: 158 active+clean; 3.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 852 B/s rd, 126 MiB/s wr, 2.02k op/s 2024-09-21T12:25:29.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:28 smithi114 ceph-mon[30324]: pgmap v247: 158 pgs: 158 active+clean; 3.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 852 B/s rd, 126 MiB/s wr, 2.02k op/s 2024-09-21T12:25:29.294 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:29.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:29.333 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:29.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:29.435 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:29.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:29.486 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:29.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:29.587 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:29.621 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:29.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:29.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:29.722 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:29.769 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:29.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:29.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:29.871 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:29.908 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:29.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:29.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:30.006 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-21T12:25:30.010 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:30.036 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:30.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:30.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:30.043 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 36 16 64961 64945 112.727 75.6875 0.00513396 0.00886574 2024-09-21T12:25:30.138 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:30.178 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:30.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:30.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:30.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:30 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/15128304' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:25:30.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:30 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:25:30.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:30 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/15128304' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:25:30.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:30 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:25:30.280 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:30.281 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:25:30.281 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:25:30.317 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:30.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:30.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:30.419 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:30.446 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:30.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:30.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:30.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:29 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/15128304' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:25:30.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:29 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:25:30.548 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:30.591 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:30.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:30.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:30.693 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:30.723 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:30.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:30.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:30.825 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:30.858 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:30.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:30.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:30.937 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:25:30.959 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:31.028 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:31.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:31.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:31.044 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 37 16 66335 66319 111.999 85.875 0.0135808 0.00892373 2024-09-21T12:25:31.130 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:31.176 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:31.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:31.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:31.230 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:31 smithi114 ceph-mon[40867]: pgmap v248: 158 pgs: 158 active+clean; 3.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 655 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-21T12:25:31.230 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:31 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:25:31.231 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:31 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:25:31.231 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:31 smithi114 ceph-mon[40867]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:25:31.231 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:31 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:25:31.231 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:31 smithi114 ceph-mon[40867]: osdmap e179: 8 total, 8 up, 7 in 2024-09-21T12:25:31.231 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:31 smithi114 ceph-mon[40867]: 3.c scrub starts 2024-09-21T12:25:31.231 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:31 smithi114 ceph-mon[40867]: 3.c scrub ok 2024-09-21T12:25:31.232 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:31 smithi114 ceph-mon[30324]: pgmap v248: 158 pgs: 158 active+clean; 3.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 655 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-21T12:25:31.232 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:31 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:25:31.232 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:31 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:25:31.233 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:31 smithi114 ceph-mon[30324]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:25:31.233 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:31 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:25:31.233 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:31 smithi114 ceph-mon[30324]: osdmap e179: 8 total, 8 up, 7 in 2024-09-21T12:25:31.233 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:31 smithi114 ceph-mon[30324]: 3.c scrub starts 2024-09-21T12:25:31.233 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:31 smithi114 ceph-mon[30324]: 3.c scrub ok 2024-09-21T12:25:31.277 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:31.307 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T12:25:31.330 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:31.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:31.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:31.432 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:31.456 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:31.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:31.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:31.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:31 smithi183 ceph-mon[33277]: pgmap v248: 158 pgs: 158 active+clean; 3.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 655 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-21T12:25:31.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:31 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:25:31.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:31 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:25:31.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:31 smithi183 ceph-mon[33277]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:25:31.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:31 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:25:31.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:31 smithi183 ceph-mon[33277]: osdmap e179: 8 total, 8 up, 7 in 2024-09-21T12:25:31.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:31 smithi183 ceph-mon[33277]: 3.c scrub starts 2024-09-21T12:25:31.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:31 smithi183 ceph-mon[33277]: 3.c scrub ok 2024-09-21T12:25:31.558 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:31.582 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:31.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:31.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:31.684 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:31.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:31.711 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:31.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:31.812 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:31.837 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:31.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:31.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:31.924 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:25:31.926 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:25:31.938 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:31.963 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:31.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:31.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:32.044 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 38 16 67506 67490 110.977 73.1875 0.00809459 0.00900589 2024-09-21T12:25:32.065 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:32.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:32.106 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:32.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:32.207 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:32.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:32.245 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:32.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:32.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:32 smithi114 ceph-mon[30324]: 6.1e scrub starts 2024-09-21T12:25:32.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:32 smithi114 ceph-mon[30324]: 6.1e scrub ok 2024-09-21T12:25:32.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:32 smithi114 ceph-mon[30324]: 4.1d scrub starts 2024-09-21T12:25:32.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:32 smithi114 ceph-mon[30324]: 4.1d scrub ok 2024-09-21T12:25:32.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:32 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:25:32.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:32 smithi114 ceph-mon[30324]: osdmap e180: 8 total, 8 up, 7 in 2024-09-21T12:25:32.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:32 smithi114 ceph-mon[40867]: 6.1e scrub starts 2024-09-21T12:25:32.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:32 smithi114 ceph-mon[40867]: 6.1e scrub ok 2024-09-21T12:25:32.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:32 smithi114 ceph-mon[40867]: 4.1d scrub starts 2024-09-21T12:25:32.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:32 smithi114 ceph-mon[40867]: 4.1d scrub ok 2024-09-21T12:25:32.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:32 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:25:32.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:32 smithi114 ceph-mon[40867]: osdmap e180: 8 total, 8 up, 7 in 2024-09-21T12:25:32.346 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:32.379 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:32.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:32.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:32.481 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:32 smithi183 ceph-mon[33277]: 6.1e scrub starts 2024-09-21T12:25:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:32 smithi183 ceph-mon[33277]: 6.1e scrub ok 2024-09-21T12:25:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:32 smithi183 ceph-mon[33277]: 4.1d scrub starts 2024-09-21T12:25:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:32 smithi183 ceph-mon[33277]: 4.1d scrub ok 2024-09-21T12:25:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:32 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:25:32.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:32 smithi183 ceph-mon[33277]: osdmap e180: 8 total, 8 up, 7 in 2024-09-21T12:25:32.508 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:32.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:32.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:32.610 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:32.641 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:32.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:32.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:32.643 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:25:32 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:25:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:25:32.692 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2024-09-21T12:25:31.858596+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4226280385,"num_objects":64677,"num_object_clones":0,"num_object_copies":194031,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64677,"num_whiteouts":0,"num_read":1036,"num_read_kb":798,"num_write":64924,"num_write_kb":4126785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":223,"num_bytes_recovered":58204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65837,"ondisk_log_size":65837,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":13674288,"kb_used_data":12420332,"kb_used_omap":10,"kb_used_meta":1253813,"kb_avail":642541776,"statfs":{"total":671965249536,"available":657962778624,"internally_reserved":0,"allocated":12718419968,"data_stored":12715552950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":1283904619},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,5,6,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":698679296,"num_objects":10661,"num_object_clones":0,"num_object_copies":31983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10661,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10661,"num_write_kb":682304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.907405"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":268,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.864016+0000","last_change":"2024-09-21T12:24:50.115300+0000","last_active":"2024-09-21T12:25:26.864016+0000","last_peered":"2024-09-21T12:25:26.864016+0000","last_clean":"2024-09-21T12:25:26.864016+0000","last_became_active":"2024-09-21T12:20:15.309431+0000","last_became_peered":"2024-09-21T12:20:15.309431+0000","last_unstale":"2024-09-21T12:25:26.864016+0000","last_undegraded":"2024-09-21T12:25:26.864016+0000","last_fullsized":"2024-09-21T12:25:26.864016+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:50.115253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:50.115253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:01.021124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":232,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.014581+0000","last_change":"2024-09-21T12:22:51.040744+0000","last_active":"2024-09-21T12:25:30.014581+0000","last_peered":"2024-09-21T12:25:30.014581+0000","last_clean":"2024-09-21T12:25:30.014581+0000","last_became_active":"2024-09-21T12:22:50.909699+0000","last_became_peered":"2024-09-21T12:22:50.909699+0000","last_unstale":"2024-09-21T12:25:30.014581+0000","last_undegraded":"2024-09-21T12:25:30.014581+0000","last_fullsized":"2024-09-21T12:25:30.014581+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:08.888235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":262,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.021192+0000","last_change":"2024-09-21T12:24:34.311501+0000","last_active":"2024-09-21T12:25:30.021192+0000","last_peered":"2024-09-21T12:25:30.021192+0000","last_clean":"2024-09-21T12:25:30.021192+0000","last_became_active":"2024-09-21T12:20:19.314908+0000","last_became_peered":"2024-09-21T12:20:19.314908+0000","last_unstale":"2024-09-21T12:25:30.021192+0000","last_undegraded":"2024-09-21T12:25:30.021192+0000","last_fullsized":"2024-09-21T12:25:30.021192+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:34.311451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:34.311451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:44.788773+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":185,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.011431+0000","last_change":"2024-09-21T12:22:50.914064+0000","last_active":"2024-09-21T12:25:30.011431+0000","last_peered":"2024-09-21T12:25:30.011431+0000","last_clean":"2024-09-21T12:25:30.011431+0000","last_became_active":"2024-09-21T12:22:50.913743+0000","last_became_peered":"2024-09-21T12:22:50.913743+0000","last_unstale":"2024-09-21T12:25:30.011431+0000","last_undegraded":"2024-09-21T12:25:30.011431+0000","last_fullsized":"2024-09-21T12:25:30.011431+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:58.189665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:21:58.189665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:06.336105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":238,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.868012+0000","last_change":"2024-09-21T12:20:21.595607+0000","last_active":"2024-09-21T12:25:26.868012+0000","last_peered":"2024-09-21T12:25:26.868012+0000","last_clean":"2024-09-21T12:25:26.868012+0000","last_became_active":"2024-09-21T12:20:21.595301+0000","last_became_peered":"2024-09-21T12:20:21.595301+0000","last_unstale":"2024-09-21T12:25:26.868012+0000","last_undegraded":"2024-09-21T12:25:26.868012+0000","last_fullsized":"2024-09-21T12:25:26.868012+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:41.642996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":266,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.866824+0000","last_change":"2024-09-21T12:24:49.701972+0000","last_active":"2024-09-21T12:25:26.866824+0000","last_peered":"2024-09-21T12:25:26.866824+0000","last_clean":"2024-09-21T12:25:26.866824+0000","last_became_active":"2024-09-21T12:20:15.308655+0000","last_became_peered":"2024-09-21T12:20:15.308655+0000","last_unstale":"2024-09-21T12:25:26.866824+0000","last_undegraded":"2024-09-21T12:25:26.866824+0000","last_fullsized":"2024-09-21T12:25:26.866824+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:49.701915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:49.701915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:07.777041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":287,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.866277+0000","last_change":"2024-09-21T12:25:01.898461+0000","last_active":"2024-09-21T12:25:26.866277+0000","last_peered":"2024-09-21T12:25:26.866277+0000","last_clean":"2024-09-21T12:25:26.866277+0000","last_became_active":"2024-09-21T12:20:17.305220+0000","last_became_peered":"2024-09-21T12:20:17.305220+0000","last_unstale":"2024-09-21T12:25:26.866277+0000","last_undegraded":"2024-09-21T12:25:26.866277+0000","last_fullsized":"2024-09-21T12:25:26.866277+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:25:01.898376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:01.898376+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:05.029581+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":202,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.014739+0000","last_change":"2024-09-21T12:25:04.827596+0000","last_active":"2024-09-21T12:25:30.014739+0000","last_peered":"2024-09-21T12:25:30.014739+0000","last_clean":"2024-09-21T12:25:30.014739+0000","last_became_active":"2024-09-21T12:22:50.926240+0000","last_became_peered":"2024-09-21T12:22:50.926240+0000","last_unstale":"2024-09-21T12:25:30.014739+0000","last_undegraded":"2024-09-21T12:25:30.014739+0000","last_fullsized":"2024-09-21T12:25:30.014739+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:04.827529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:04.827529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:09.166075+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":282,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.010627+0000","last_change":"2024-09-21T12:22:50.912964+0000","last_active":"2024-09-21T12:25:30.010627+0000","last_peered":"2024-09-21T12:25:30.010627+0000","last_clean":"2024-09-21T12:25:30.010627+0000","last_became_active":"2024-09-21T12:22:50.912481+0000","last_became_peered":"2024-09-21T12:22:50.912481+0000","last_unstale":"2024-09-21T12:25:30.010627+0000","last_undegraded":"2024-09-21T12:25:30.010627+0000","last_fullsized":"2024-09-21T12:25:30.010627+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:43.207441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:43.207441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:17.306151+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":247,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.866099+0000","last_change":"2024-09-21T12:24:25.640253+0000","last_active":"2024-09-21T12:25:26.866099+0000","last_peered":"2024-09-21T12:25:26.866099+0000","last_clean":"2024-09-21T12:25:26.866099+0000","last_became_active":"2024-09-21T12:22:50.927804+0000","last_became_peered":"2024-09-21T12:22:50.927804+0000","last_unstale":"2024-09-21T12:25:26.866099+0000","last_undegraded":"2024-09-21T12:25:26.866099+0000","last_fullsized":"2024-09-21T12:25:26.866099+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:24:25.640169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:25.640169+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:45.636860+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":261,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.010082+0000","last_change":"2024-09-21T12:24:58.566964+0000","last_active":"2024-09-21T12:25:30.010082+0000","last_peered":"2024-09-21T12:25:30.010082+0000","last_clean":"2024-09-21T12:25:30.010082+0000","last_became_active":"2024-09-21T12:20:19.308184+0000","last_became_peered":"2024-09-21T12:20:19.308184+0000","last_unstale":"2024-09-21T12:25:30.010082+0000","last_undegraded":"2024-09-21T12:25:30.010082+0000","last_fullsized":"2024-09-21T12:25:30.010082+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:58.566902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:58.566902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:27.540756+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":248,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.862726+0000","last_change":"2024-09-21T12:24:04.091870+0000","last_active":"2024-09-21T12:25:26.862726+0000","last_peered":"2024-09-21T12:25:26.862726+0000","last_clean":"2024-09-21T12:25:26.862726+0000","last_became_active":"2024-09-21T12:20:21.309190+0000","last_became_peered":"2024-09-21T12:20:21.309190+0000","last_unstale":"2024-09-21T12:25:26.862726+0000","last_undegraded":"2024-09-21T12:25:26.862726+0000","last_fullsized":"2024-09-21T12:25:26.862726+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:04.091807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:04.091807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":282,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.010303+0000","last_change":"2024-09-21T12:22:54.174099+0000","last_active":"2024-09-21T12:25:30.010303+0000","last_peered":"2024-09-21T12:25:30.010303+0000","last_clean":"2024-09-21T12:25:30.010303+0000","last_became_active":"2024-09-21T12:22:50.911737+0000","last_became_peered":"2024-09-21T12:22:50.911737+0000","last_unstale":"2024-09-21T12:25:30.010303+0000","last_undegraded":"2024-09-21T12:25:30.010303+0000","last_fullsized":"2024-09-21T12:25:30.010303+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:54.174034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:54.174034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:23.349015+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":233,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.868264+0000","last_change":"2024-09-21T12:24:28.624430+0000","last_active":"2024-09-21T12:25:26.868264+0000","last_peered":"2024-09-21T12:25:26.868264+0000","last_clean":"2024-09-21T12:25:26.868264+0000","last_became_active":"2024-09-21T12:22:50.918794+0000","last_became_peered":"2024-09-21T12:22:50.918794+0000","last_unstale":"2024-09-21T12:25:26.868264+0000","last_undegraded":"2024-09-21T12:25:26.868264+0000","last_fullsized":"2024-09-21T12:25:26.868264+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-21T12:24:28.624364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:28.624364+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:45.839090+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":274,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.012150+0000","last_change":"2024-09-21T12:25:01.897596+0000","last_active":"2024-09-21T12:25:30.012150+0000","last_peered":"2024-09-21T12:25:30.012150+0000","last_clean":"2024-09-21T12:25:30.012150+0000","last_became_active":"2024-09-21T12:22:50.921780+0000","last_became_peered":"2024-09-21T12:22:50.921780+0000","last_unstale":"2024-09-21T12:25:30.012150+0000","last_undegraded":"2024-09-21T12:25:30.012150+0000","last_fullsized":"2024-09-21T12:25:30.012150+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:01.897527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:01.897527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:17.629145+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":265,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.009942+0000","last_change":"2024-09-21T12:24:34.367580+0000","last_active":"2024-09-21T12:25:30.009942+0000","last_peered":"2024-09-21T12:25:30.009942+0000","last_clean":"2024-09-21T12:25:30.009942+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:25:30.009942+0000","last_undegraded":"2024-09-21T12:25:30.009942+0000","last_fullsized":"2024-09-21T12:25:30.009942+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:34.367532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:34.367532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:51.733042+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":201,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.888535+0000","last_change":"2024-09-21T12:25:30.888535+0000","last_active":"2024-09-21T12:25:30.888535+0000","last_peered":"2024-09-21T12:25:30.888535+0000","last_clean":"2024-09-21T12:25:30.888535+0000","last_became_active":"2024-09-21T12:22:50.914792+0000","last_became_peered":"2024-09-21T12:22:50.914792+0000","last_unstale":"2024-09-21T12:25:30.888535+0000","last_undegraded":"2024-09-21T12:25:30.888535+0000","last_fullsized":"2024-09-21T12:25:30.888535+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:30.888491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:30.888491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:32.893945+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":297,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.863668+0000","last_change":"2024-09-21T12:24:30.033946+0000","last_active":"2024-09-21T12:25:26.863668+0000","last_peered":"2024-09-21T12:25:26.863668+0000","last_clean":"2024-09-21T12:25:26.863668+0000","last_became_active":"2024-09-21T12:22:50.921576+0000","last_became_peered":"2024-09-21T12:22:50.921576+0000","last_unstale":"2024-09-21T12:25:26.863668+0000","last_undegraded":"2024-09-21T12:25:26.863668+0000","last_fullsized":"2024-09-21T12:25:26.863668+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:30.033874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:30.033874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:53.842679+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":291,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.026186+0000","last_change":"2024-09-21T12:24:50.834579+0000","last_active":"2024-09-21T12:25:30.026186+0000","last_peered":"2024-09-21T12:25:30.026186+0000","last_clean":"2024-09-21T12:25:30.026186+0000","last_became_active":"2024-09-21T12:20:17.307828+0000","last_became_peered":"2024-09-21T12:20:17.307828+0000","last_unstale":"2024-09-21T12:25:30.026186+0000","last_undegraded":"2024-09-21T12:25:30.026186+0000","last_fullsized":"2024-09-21T12:25:30.026186+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:24:50.834503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:50.834503+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:20.530206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":283,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.020253+0000","last_change":"2024-09-21T12:23:48.435833+0000","last_active":"2024-09-21T12:25:30.020253+0000","last_peered":"2024-09-21T12:25:30.020253+0000","last_clean":"2024-09-21T12:25:30.020253+0000","last_became_active":"2024-09-21T12:22:50.928972+0000","last_became_peered":"2024-09-21T12:22:50.928972+0000","last_unstale":"2024-09-21T12:25:30.020253+0000","last_undegraded":"2024-09-21T12:25:30.020253+0000","last_fullsized":"2024-09-21T12:25:30.020253+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:48.435769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:48.435769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:19.426681+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":246,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.867336+0000","last_change":"2024-09-21T12:23:58.354628+0000","last_active":"2024-09-21T12:25:26.867336+0000","last_peered":"2024-09-21T12:25:26.867336+0000","last_clean":"2024-09-21T12:25:26.867336+0000","last_became_active":"2024-09-21T12:20:21.592796+0000","last_became_peered":"2024-09-21T12:20:21.592796+0000","last_unstale":"2024-09-21T12:25:26.867336+0000","last_undegraded":"2024-09-21T12:25:26.867336+0000","last_fullsized":"2024-09-21T12:25:26.867336+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:58.354570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:58.354570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:21.152937+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":201,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.019479+0000","last_change":"2024-09-21T12:24:55.447829+0000","last_active":"2024-09-21T12:25:30.019479+0000","last_peered":"2024-09-21T12:25:30.019479+0000","last_clean":"2024-09-21T12:25:30.019479+0000","last_became_active":"2024-09-21T12:22:50.925403+0000","last_became_peered":"2024-09-21T12:22:50.925403+0000","last_unstale":"2024-09-21T12:25:30.019479+0000","last_undegraded":"2024-09-21T12:25:30.019479+0000","last_fullsized":"2024-09-21T12:25:30.019479+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:55.447755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:55.447755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:20.748653+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":334,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.866163+0000","last_change":"2024-09-21T12:23:47.285620+0000","last_active":"2024-09-21T12:25:26.866163+0000","last_peered":"2024-09-21T12:25:26.866163+0000","last_clean":"2024-09-21T12:25:26.866163+0000","last_became_active":"2024-09-21T12:22:50.912144+0000","last_became_peered":"2024-09-21T12:22:50.912144+0000","last_unstale":"2024-09-21T12:25:26.866163+0000","last_undegraded":"2024-09-21T12:25:26.866163+0000","last_fullsized":"2024-09-21T12:25:26.866163+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:23:47.285537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:47.285537+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":176,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.868919+0000","last_change":"2024-09-21T12:22:50.920178+0000","last_active":"2024-09-21T12:25:26.868919+0000","last_peered":"2024-09-21T12:25:26.868919+0000","last_clean":"2024-09-21T12:25:26.868919+0000","last_became_active":"2024-09-21T12:22:50.918326+0000","last_became_peered":"2024-09-21T12:22:50.918326+0000","last_unstale":"2024-09-21T12:25:26.868919+0000","last_undegraded":"2024-09-21T12:25:26.868919+0000","last_fullsized":"2024-09-21T12:25:26.868919+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:42.672999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":250,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.012994+0000","last_change":"2024-09-21T12:23:51.065780+0000","last_active":"2024-09-21T12:25:30.012994+0000","last_peered":"2024-09-21T12:25:30.012994+0000","last_clean":"2024-09-21T12:25:30.012994+0000","last_became_active":"2024-09-21T12:20:21.312976+0000","last_became_peered":"2024-09-21T12:20:21.312976+0000","last_unstale":"2024-09-21T12:25:30.012994+0000","last_undegraded":"2024-09-21T12:25:30.012994+0000","last_fullsized":"2024-09-21T12:25:30.012994+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:23:51.065733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:51.065733+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:04.884955+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":296,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.021356+0000","last_change":"2024-09-21T12:22:52.070589+0000","last_active":"2024-09-21T12:25:30.021356+0000","last_peered":"2024-09-21T12:25:30.021356+0000","last_clean":"2024-09-21T12:25:30.021356+0000","last_became_active":"2024-09-21T12:22:50.910933+0000","last_became_peered":"2024-09-21T12:22:50.910933+0000","last_unstale":"2024-09-21T12:25:30.021356+0000","last_undegraded":"2024-09-21T12:25:30.021356+0000","last_fullsized":"2024-09-21T12:25:30.021356+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:22:52.070513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:52.070513+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:16.453567+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":271,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.866681+0000","last_change":"2024-09-21T12:24:03.059782+0000","last_active":"2024-09-21T12:25:26.866681+0000","last_peered":"2024-09-21T12:25:26.866681+0000","last_clean":"2024-09-21T12:25:26.866681+0000","last_became_active":"2024-09-21T12:20:17.695977+0000","last_became_peered":"2024-09-21T12:20:17.695977+0000","last_unstale":"2024-09-21T12:25:26.866681+0000","last_undegraded":"2024-09-21T12:25:26.866681+0000","last_fullsized":"2024-09-21T12:25:26.866681+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:24:03.059549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:03.059549+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:25.640560+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"121'11","reported_seq":369,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.016158+0000","last_change":"2024-09-21T12:24:50.925521+0000","last_active":"2024-09-21T12:25:30.016158+0000","last_peered":"2024-09-21T12:25:30.016158+0000","last_clean":"2024-09-21T12:25:30.016158+0000","last_became_active":"2024-09-21T12:22:50.916683+0000","last_became_peered":"2024-09-21T12:22:50.916683+0000","last_unstale":"2024-09-21T12:25:30.016158+0000","last_undegraded":"2024-09-21T12:25:30.016158+0000","last_fullsized":"2024-09-21T12:25:30.016158+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:24:50.925458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:50.925458+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:07.929632+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"179'4078","reported_seq":4125,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:31.745942+0000","last_change":"2024-09-21T12:24:46.316817+0000","last_active":"2024-09-21T12:25:31.745942+0000","last_peered":"2024-09-21T12:25:31.745942+0000","last_clean":"2024-09-21T12:25:31.745942+0000","last_became_active":"2024-09-21T12:24:46.316132+0000","last_became_peered":"2024-09-21T12:24:46.316132+0000","last_unstale":"2024-09-21T12:25:31.745942+0000","last_undegraded":"2024-09-21T12:25:31.745942+0000","last_fullsized":"2024-09-21T12:25:31.745942+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4078,"log_dups_size":0,"ondisk_log_size":4078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:55.574645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267255808,"num_objects":4078,"num_object_clones":0,"num_object_copies":12234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4078,"num_write_kb":260992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"76'79","reported_seq":162,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.009763+0000","last_change":"2024-09-21T12:22:18.000752+0000","last_active":"2024-09-21T12:25:30.009763+0000","last_peered":"2024-09-21T12:25:30.009763+0000","last_clean":"2024-09-21T12:25:30.009763+0000","last_became_active":"2024-09-21T12:22:17.074721+0000","last_became_peered":"2024-09-21T12:22:17.074721+0000","last_unstale":"2024-09-21T12:25:30.009763+0000","last_undegraded":"2024-09-21T12:25:30.009763+0000","last_fullsized":"2024-09-21T12:25:30.009763+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:22:18.000705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:22:18.000705+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:41.349901+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":120,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.025731+0000","last_change":"2024-09-21T12:24:59.803037+0000","last_active":"2024-09-21T12:25:30.025731+0000","last_peered":"2024-09-21T12:25:30.025731+0000","last_clean":"2024-09-21T12:25:30.025731+0000","last_became_active":"2024-09-21T12:23:46.910112+0000","last_became_peered":"2024-09-21T12:23:46.910112+0000","last_unstale":"2024-09-21T12:25:30.025731+0000","last_undegraded":"2024-09-21T12:25:30.025731+0000","last_fullsized":"2024-09-21T12:25:30.025731+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:24:59.802945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:59.802945+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:22.089988+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":258,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.869491+0000","last_change":"2024-09-21T12:22:14.036290+0000","last_active":"2024-09-21T12:25:26.869491+0000","last_peered":"2024-09-21T12:25:26.869491+0000","last_clean":"2024-09-21T12:25:26.869491+0000","last_became_active":"2024-09-21T12:20:15.295774+0000","last_became_peered":"2024-09-21T12:20:15.295774+0000","last_unstale":"2024-09-21T12:25:26.869491+0000","last_undegraded":"2024-09-21T12:25:26.869491+0000","last_fullsized":"2024-09-21T12:25:26.869491+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:14.036236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.036236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:45.694273+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"158'13","reported_seq":72,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.010397+0000","last_change":"2024-09-21T12:24:39.276556+0000","last_active":"2024-09-21T12:25:30.010397+0000","last_peered":"2024-09-21T12:25:30.010397+0000","last_clean":"2024-09-21T12:25:30.010397+0000","last_became_active":"2024-09-21T12:24:39.272835+0000","last_became_peered":"2024-09-21T12:24:39.272835+0000","last_unstale":"2024-09-21T12:25:30.010397+0000","last_undegraded":"2024-09-21T12:25:30.010397+0000","last_fullsized":"2024-09-21T12:25:30.010397+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'12","last_scrub_stamp":"2024-09-21T12:24:01.051257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:01.051257+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":250,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.010710+0000","last_change":"2024-09-21T12:22:12.145637+0000","last_active":"2024-09-21T12:25:30.010710+0000","last_peered":"2024-09-21T12:25:30.010710+0000","last_clean":"2024-09-21T12:25:30.010710+0000","last_became_active":"2024-09-21T12:20:21.310022+0000","last_became_peered":"2024-09-21T12:20:21.310022+0000","last_unstale":"2024-09-21T12:25:30.010710+0000","last_undegraded":"2024-09-21T12:25:30.010710+0000","last_fullsized":"2024-09-21T12:25:30.010710+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:22:12.145550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:12.145550+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:46.398927+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"178'3786","reported_seq":3833,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:27.088819+0000","last_change":"2024-09-21T12:24:46.326848+0000","last_active":"2024-09-21T12:25:27.088819+0000","last_peered":"2024-09-21T12:25:27.088819+0000","last_clean":"2024-09-21T12:25:27.088819+0000","last_became_active":"2024-09-21T12:24:46.326586+0000","last_became_peered":"2024-09-21T12:24:46.326586+0000","last_unstale":"2024-09-21T12:25:27.088819+0000","last_undegraded":"2024-09-21T12:25:27.088819+0000","last_fullsized":"2024-09-21T12:25:27.088819+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":3786,"log_dups_size":0,"ondisk_log_size":3786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:11.463769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248119296,"num_objects":3786,"num_object_clones":0,"num_object_copies":11358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3786,"num_write_kb":242304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"76'79","reported_seq":151,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.007454+0000","last_change":"2024-09-21T12:22:50.916239+0000","last_active":"2024-09-21T12:25:30.007454+0000","last_peered":"2024-09-21T12:25:30.007454+0000","last_clean":"2024-09-21T12:25:30.007454+0000","last_became_active":"2024-09-21T12:22:50.914885+0000","last_became_peered":"2024-09-21T12:22:50.914885+0000","last_unstale":"2024-09-21T12:25:30.007454+0000","last_undegraded":"2024-09-21T12:25:30.007454+0000","last_fullsized":"2024-09-21T12:25:30.007454+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:22:17.824342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:22:17.824342+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:19.884884+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":286,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.869310+0000","last_change":"2024-09-21T12:23:49.326670+0000","last_active":"2024-09-21T12:25:26.869310+0000","last_peered":"2024-09-21T12:25:26.869310+0000","last_clean":"2024-09-21T12:25:26.869310+0000","last_became_active":"2024-09-21T12:20:17.693179+0000","last_became_peered":"2024-09-21T12:20:17.693179+0000","last_unstale":"2024-09-21T12:25:26.869310+0000","last_undegraded":"2024-09-21T12:25:26.869310+0000","last_fullsized":"2024-09-21T12:25:26.869310+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-21T12:23:49.326428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:49.326428+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":258,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.867841+0000","last_change":"2024-09-21T12:21:18.056305+0000","last_active":"2024-09-21T12:25:26.867841+0000","last_peered":"2024-09-21T12:25:26.867841+0000","last_clean":"2024-09-21T12:25:26.867841+0000","last_became_active":"2024-09-21T12:20:15.297340+0000","last_became_peered":"2024-09-21T12:20:15.297340+0000","last_unstale":"2024-09-21T12:25:26.867841+0000","last_undegraded":"2024-09-21T12:25:26.867841+0000","last_fullsized":"2024-09-21T12:25:26.867841+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:18.056261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:18.056261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:06.946985+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":192,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.865874+0000","last_change":"2024-09-21T12:24:24.661109+0000","last_active":"2024-09-21T12:25:26.865874+0000","last_peered":"2024-09-21T12:25:26.865874+0000","last_clean":"2024-09-21T12:25:26.865874+0000","last_became_active":"2024-09-21T12:22:50.930858+0000","last_became_peered":"2024-09-21T12:22:50.930858+0000","last_unstale":"2024-09-21T12:25:26.865874+0000","last_undegraded":"2024-09-21T12:25:26.865874+0000","last_fullsized":"2024-09-21T12:25:26.865874+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:24.661038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:24.661038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:35.691262+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":191,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.866012+0000","last_change":"2024-09-21T12:24:29.635826+0000","last_active":"2024-09-21T12:25:26.866012+0000","last_peered":"2024-09-21T12:25:26.866012+0000","last_clean":"2024-09-21T12:25:26.866012+0000","last_became_active":"2024-09-21T12:22:50.918940+0000","last_became_peered":"2024-09-21T12:22:50.918940+0000","last_unstale":"2024-09-21T12:25:26.866012+0000","last_undegraded":"2024-09-21T12:25:26.866012+0000","last_fullsized":"2024-09-21T12:25:26.866012+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:29.635514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:29.635514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:47.655178+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"179'4160","reported_seq":4209,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:31.541293+0000","last_change":"2024-09-21T12:24:46.324341+0000","last_active":"2024-09-21T12:25:31.541293+0000","last_peered":"2024-09-21T12:25:31.541293+0000","last_clean":"2024-09-21T12:25:31.541293+0000","last_became_active":"2024-09-21T12:24:46.323986+0000","last_became_peered":"2024-09-21T12:24:46.323986+0000","last_unstale":"2024-09-21T12:25:31.541293+0000","last_undegraded":"2024-09-21T12:25:31.541293+0000","last_fullsized":"2024-09-21T12:25:31.541293+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4160,"log_dups_size":0,"ondisk_log_size":4160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:13.943600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272629760,"num_objects":4160,"num_object_clones":0,"num_object_copies":12480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4160,"num_write_kb":266240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"76'79","reported_seq":173,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.868761+0000","last_change":"2024-09-21T12:25:02.911776+0000","last_active":"2024-09-21T12:25:26.868761+0000","last_peered":"2024-09-21T12:25:26.868761+0000","last_clean":"2024-09-21T12:25:26.868761+0000","last_became_active":"2024-09-21T12:22:17.069042+0000","last_became_peered":"2024-09-21T12:22:17.069042+0000","last_unstale":"2024-09-21T12:25:26.868761+0000","last_undegraded":"2024-09-21T12:25:26.868761+0000","last_fullsized":"2024-09-21T12:25:26.868761+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:25:02.911705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:25:02.911705+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:06.798137+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":258,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.010031+0000","last_change":"2024-09-21T12:22:50.978102+0000","last_active":"2024-09-21T12:25:30.010031+0000","last_peered":"2024-09-21T12:25:30.010031+0000","last_clean":"2024-09-21T12:25:30.010031+0000","last_became_active":"2024-09-21T12:22:50.926008+0000","last_became_peered":"2024-09-21T12:22:50.926008+0000","last_unstale":"2024-09-21T12:25:30.010031+0000","last_undegraded":"2024-09-21T12:25:30.010031+0000","last_fullsized":"2024-09-21T12:25:30.010031+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-21T12:22:14.062464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.062464+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:42.320597+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":294,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.866950+0000","last_change":"2024-09-21T12:22:51.003768+0000","last_active":"2024-09-21T12:25:26.866950+0000","last_peered":"2024-09-21T12:25:26.866950+0000","last_clean":"2024-09-21T12:25:26.866950+0000","last_became_active":"2024-09-21T12:22:50.923284+0000","last_became_peered":"2024-09-21T12:22:50.923284+0000","last_unstale":"2024-09-21T12:25:26.866950+0000","last_undegraded":"2024-09-21T12:25:26.866950+0000","last_fullsized":"2024-09-21T12:25:26.866950+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:22:07.582969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:07.582969+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:47.120857+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"121'11","reported_seq":354,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.869157+0000","last_change":"2024-09-21T12:23:27.244152+0000","last_active":"2024-09-21T12:25:26.869157+0000","last_peered":"2024-09-21T12:25:26.869157+0000","last_clean":"2024-09-21T12:25:26.869157+0000","last_became_active":"2024-09-21T12:22:50.930342+0000","last_became_peered":"2024-09-21T12:22:50.930342+0000","last_unstale":"2024-09-21T12:25:26.869157+0000","last_undegraded":"2024-09-21T12:25:26.869157+0000","last_fullsized":"2024-09-21T12:25:26.869157+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:23:27.243903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:27.243903+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:08.949234+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":249,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.012205+0000","last_change":"2024-09-21T12:21:21.835669+0000","last_active":"2024-09-21T12:25:30.012205+0000","last_peered":"2024-09-21T12:25:30.012205+0000","last_clean":"2024-09-21T12:25:30.012205+0000","last_became_active":"2024-09-21T12:20:21.593367+0000","last_became_peered":"2024-09-21T12:20:21.593367+0000","last_unstale":"2024-09-21T12:25:30.012205+0000","last_undegraded":"2024-09-21T12:25:30.012205+0000","last_fullsized":"2024-09-21T12:25:30.012205+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:21.835623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:21:21.835623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:29.766408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"178'3797","reported_seq":3844,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:27.100018+0000","last_change":"2024-09-21T12:24:46.325776+0000","last_active":"2024-09-21T12:25:27.100018+0000","last_peered":"2024-09-21T12:25:27.100018+0000","last_clean":"2024-09-21T12:25:27.100018+0000","last_became_active":"2024-09-21T12:24:46.325260+0000","last_became_peered":"2024-09-21T12:24:46.325260+0000","last_unstale":"2024-09-21T12:25:27.100018+0000","last_undegraded":"2024-09-21T12:25:27.100018+0000","last_fullsized":"2024-09-21T12:25:27.100018+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":3797,"log_dups_size":0,"ondisk_log_size":3797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:57.283236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248840192,"num_objects":3797,"num_object_clones":0,"num_object_copies":11391,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3797,"num_write_kb":243008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"76'79","reported_seq":260,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.015619+0000","last_change":"2024-09-21T12:24:51.918097+0000","last_active":"2024-09-21T12:25:30.015619+0000","last_peered":"2024-09-21T12:25:30.015619+0000","last_clean":"2024-09-21T12:25:30.015619+0000","last_became_active":"2024-09-21T12:22:17.093298+0000","last_became_peered":"2024-09-21T12:22:17.093298+0000","last_unstale":"2024-09-21T12:25:30.015619+0000","last_undegraded":"2024-09-21T12:25:30.015619+0000","last_fullsized":"2024-09-21T12:25:30.015619+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:24:51.918028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:24:51.918028+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:11.471338+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":330,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.019433+0000","last_change":"2024-09-21T12:24:01.388987+0000","last_active":"2024-09-21T12:25:30.019433+0000","last_peered":"2024-09-21T12:25:30.019433+0000","last_clean":"2024-09-21T12:25:30.019433+0000","last_became_active":"2024-09-21T12:22:50.925840+0000","last_became_peered":"2024-09-21T12:22:50.925840+0000","last_unstale":"2024-09-21T12:25:30.019433+0000","last_undegraded":"2024-09-21T12:25:30.019433+0000","last_fullsized":"2024-09-21T12:25:30.019433+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:24:01.388905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:01.388905+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:20.477756+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":261,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.010708+0000","last_change":"2024-09-21T12:22:37.176483+0000","last_active":"2024-09-21T12:25:30.010708+0000","last_peered":"2024-09-21T12:25:30.010708+0000","last_clean":"2024-09-21T12:25:30.010708+0000","last_became_active":"2024-09-21T12:20:15.308518+0000","last_became_peered":"2024-09-21T12:20:15.308518+0000","last_unstale":"2024-09-21T12:25:30.010708+0000","last_undegraded":"2024-09-21T12:25:30.010708+0000","last_fullsized":"2024-09-21T12:25:30.010708+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:37.176413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:37.176413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:12.301692+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":326,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.863241+0000","last_change":"2024-09-21T12:21:23.139205+0000","last_active":"2024-09-21T12:25:26.863241+0000","last_peered":"2024-09-21T12:25:26.863241+0000","last_clean":"2024-09-21T12:25:26.863241+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:25:26.863241+0000","last_undegraded":"2024-09-21T12:25:26.863241+0000","last_fullsized":"2024-09-21T12:25:26.863241+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:21:23.139124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.139124+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:41.119005+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":248,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.862848+0000","last_change":"2024-09-21T12:22:51.987032+0000","last_active":"2024-09-21T12:25:26.862848+0000","last_peered":"2024-09-21T12:25:26.862848+0000","last_clean":"2024-09-21T12:25:26.862848+0000","last_became_active":"2024-09-21T12:20:21.315801+0000","last_became_peered":"2024-09-21T12:20:21.315801+0000","last_unstale":"2024-09-21T12:25:26.862848+0000","last_undegraded":"2024-09-21T12:25:26.862848+0000","last_fullsized":"2024-09-21T12:25:26.862848+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:51.986953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:51.986953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:21.623214+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"178'3855","reported_seq":3902,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:27.051716+0000","last_change":"2024-09-21T12:24:46.324384+0000","last_active":"2024-09-21T12:25:27.051716+0000","last_peered":"2024-09-21T12:25:27.051716+0000","last_clean":"2024-09-21T12:25:27.051716+0000","last_became_active":"2024-09-21T12:24:46.323283+0000","last_became_peered":"2024-09-21T12:24:46.323283+0000","last_unstale":"2024-09-21T12:25:27.051716+0000","last_undegraded":"2024-09-21T12:25:27.051716+0000","last_fullsized":"2024-09-21T12:25:27.051716+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":3855,"log_dups_size":0,"ondisk_log_size":3855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:58.096500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252641280,"num_objects":3855,"num_object_clones":0,"num_object_copies":11565,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3855,"num_write_kb":246720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"76'79","reported_seq":260,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.016291+0000","last_change":"2024-09-21T12:24:53.927007+0000","last_active":"2024-09-21T12:25:30.016291+0000","last_peered":"2024-09-21T12:25:30.016291+0000","last_clean":"2024-09-21T12:25:30.016291+0000","last_became_active":"2024-09-21T12:22:17.083629+0000","last_became_peered":"2024-09-21T12:22:17.083629+0000","last_unstale":"2024-09-21T12:25:30.016291+0000","last_undegraded":"2024-09-21T12:25:30.016291+0000","last_fullsized":"2024-09-21T12:25:30.016291+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:24:53.926958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:24:53.926958+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:21.193665+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":292,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.863423+0000","last_change":"2024-09-21T12:23:22.124368+0000","last_active":"2024-09-21T12:25:26.863423+0000","last_peered":"2024-09-21T12:25:26.863423+0000","last_clean":"2024-09-21T12:25:26.863423+0000","last_became_active":"2024-09-21T12:20:17.693067+0000","last_became_peered":"2024-09-21T12:20:17.693067+0000","last_unstale":"2024-09-21T12:25:26.863423+0000","last_undegraded":"2024-09-21T12:25:26.863423+0000","last_fullsized":"2024-09-21T12:25:26.863423+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-21T12:23:22.124283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:22.124283+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:07.353016+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":254,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.019595+0000","last_change":"2024-09-21T12:20:15.308741+0000","last_active":"2024-09-21T12:25:30.019595+0000","last_peered":"2024-09-21T12:25:30.019595+0000","last_clean":"2024-09-21T12:25:30.019595+0000","last_became_active":"2024-09-21T12:20:15.308441+0000","last_became_peered":"2024-09-21T12:20:15.308441+0000","last_unstale":"2024-09-21T12:25:30.019595+0000","last_undegraded":"2024-09-21T12:25:30.019595+0000","last_fullsized":"2024-09-21T12:25:30.019595+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:10.468822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":253,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.015669+0000","last_change":"2024-09-21T12:22:34.563365+0000","last_active":"2024-09-21T12:25:30.015669+0000","last_peered":"2024-09-21T12:25:30.015669+0000","last_clean":"2024-09-21T12:25:30.015669+0000","last_became_active":"2024-09-21T12:20:19.309598+0000","last_became_peered":"2024-09-21T12:20:19.309598+0000","last_unstale":"2024-09-21T12:25:30.015669+0000","last_undegraded":"2024-09-21T12:25:30.015669+0000","last_fullsized":"2024-09-21T12:25:30.015669+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:34.563282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:34.563282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:10.919249+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":242,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.020405+0000","last_change":"2024-09-21T12:20:21.593173+0000","last_active":"2024-09-21T12:25:30.020405+0000","last_peered":"2024-09-21T12:25:30.020405+0000","last_clean":"2024-09-21T12:25:30.020405+0000","last_became_active":"2024-09-21T12:20:21.592828+0000","last_became_peered":"2024-09-21T12:20:21.592828+0000","last_unstale":"2024-09-21T12:25:30.020405+0000","last_undegraded":"2024-09-21T12:25:30.020405+0000","last_fullsized":"2024-09-21T12:25:30.020405+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:43.315722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"179'4238","reported_seq":4287,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:31.546939+0000","last_change":"2024-09-21T12:24:46.325092+0000","last_active":"2024-09-21T12:25:31.546939+0000","last_peered":"2024-09-21T12:25:31.546939+0000","last_clean":"2024-09-21T12:25:31.546939+0000","last_became_active":"2024-09-21T12:24:46.324740+0000","last_became_peered":"2024-09-21T12:24:46.324740+0000","last_unstale":"2024-09-21T12:25:31.546939+0000","last_undegraded":"2024-09-21T12:25:31.546939+0000","last_fullsized":"2024-09-21T12:25:31.546939+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4238,"log_dups_size":0,"ondisk_log_size":4238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:14.438068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277741568,"num_objects":4238,"num_object_clones":0,"num_object_copies":12714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4238,"num_write_kb":271232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"76'79","reported_seq":162,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.009729+0000","last_change":"2024-09-21T12:23:21.346117+0000","last_active":"2024-09-21T12:25:30.009729+0000","last_peered":"2024-09-21T12:25:30.009729+0000","last_clean":"2024-09-21T12:25:30.009729+0000","last_became_active":"2024-09-21T12:22:17.084732+0000","last_became_peered":"2024-09-21T12:22:17.084732+0000","last_unstale":"2024-09-21T12:25:30.009729+0000","last_undegraded":"2024-09-21T12:25:30.009729+0000","last_fullsized":"2024-09-21T12:25:30.009729+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:23:21.346046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:23:21.346046+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:44.364115+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":362,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.019484+0000","last_change":"2024-09-21T12:20:17.308049+0000","last_active":"2024-09-21T12:25:30.019484+0000","last_peered":"2024-09-21T12:25:30.019484+0000","last_clean":"2024-09-21T12:25:30.019484+0000","last_became_active":"2024-09-21T12:20:17.307769+0000","last_became_peered":"2024-09-21T12:20:17.307769+0000","last_unstale":"2024-09-21T12:25:30.019484+0000","last_undegraded":"2024-09-21T12:25:30.019484+0000","last_fullsized":"2024-09-21T12:25:30.019484+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:07.537534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":183,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.863595+0000","last_change":"2024-09-21T12:22:50.922961+0000","last_active":"2024-09-21T12:25:26.863595+0000","last_peered":"2024-09-21T12:25:26.863595+0000","last_clean":"2024-09-21T12:25:26.863595+0000","last_became_active":"2024-09-21T12:22:50.922002+0000","last_became_peered":"2024-09-21T12:22:50.922002+0000","last_unstale":"2024-09-21T12:25:26.863595+0000","last_undegraded":"2024-09-21T12:25:26.863595+0000","last_fullsized":"2024-09-21T12:25:26.863595+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:05.507899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:05.507899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:45.117108+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":271,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.011705+0000","last_change":"2024-09-21T12:21:20.045824+0000","last_active":"2024-09-21T12:25:30.011705+0000","last_peered":"2024-09-21T12:25:30.011705+0000","last_clean":"2024-09-21T12:25:30.011705+0000","last_became_active":"2024-09-21T12:20:14.287998+0000","last_became_peered":"2024-09-21T12:20:14.287998+0000","last_unstale":"2024-09-21T12:25:30.011705+0000","last_undegraded":"2024-09-21T12:25:30.011705+0000","last_fullsized":"2024-09-21T12:25:30.011705+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T12:21:20.045755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.045755+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:05.367564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011011149999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":281,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.867399+0000","last_change":"2024-09-21T12:24:53.112915+0000","last_active":"2024-09-21T12:25:26.867399+0000","last_peered":"2024-09-21T12:25:26.867399+0000","last_clean":"2024-09-21T12:25:26.867399+0000","last_became_active":"2024-09-21T12:22:50.921488+0000","last_became_peered":"2024-09-21T12:22:50.921488+0000","last_unstale":"2024-09-21T12:25:26.867399+0000","last_undegraded":"2024-09-21T12:25:26.867399+0000","last_fullsized":"2024-09-21T12:25:26.867399+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:53.112868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:53.112868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:02.884141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":267,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.866575+0000","last_change":"2024-09-21T12:23:20.237788+0000","last_active":"2024-09-21T12:25:26.866575+0000","last_peered":"2024-09-21T12:25:26.866575+0000","last_clean":"2024-09-21T12:25:26.866575+0000","last_became_active":"2024-09-21T12:22:50.917558+0000","last_became_peered":"2024-09-21T12:22:50.917558+0000","last_unstale":"2024-09-21T12:25:26.866575+0000","last_undegraded":"2024-09-21T12:25:26.866575+0000","last_fullsized":"2024-09-21T12:25:26.866575+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:20.237715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:20.237715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:07.980331+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"178'3780","reported_seq":3827,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.943018+0000","last_change":"2024-09-21T12:24:46.323849+0000","last_active":"2024-09-21T12:25:26.943018+0000","last_peered":"2024-09-21T12:25:26.943018+0000","last_clean":"2024-09-21T12:25:26.943018+0000","last_became_active":"2024-09-21T12:24:46.323306+0000","last_became_peered":"2024-09-21T12:24:46.323306+0000","last_unstale":"2024-09-21T12:25:26.943018+0000","last_undegraded":"2024-09-21T12:25:26.943018+0000","last_fullsized":"2024-09-21T12:25:26.943018+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":3780,"log_dups_size":0,"ondisk_log_size":3780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:58.661394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247726080,"num_objects":3780,"num_object_clones":0,"num_object_copies":11340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3780,"num_write_kb":241920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'79","reported_seq":345,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.012592+0000","last_change":"2024-09-21T12:25:03.877681+0000","last_active":"2024-09-21T12:25:30.012592+0000","last_peered":"2024-09-21T12:25:30.012592+0000","last_clean":"2024-09-21T12:25:30.012592+0000","last_became_active":"2024-09-21T12:22:50.921709+0000","last_became_peered":"2024-09-21T12:22:50.921709+0000","last_unstale":"2024-09-21T12:25:30.012592+0000","last_undegraded":"2024-09-21T12:25:30.012592+0000","last_fullsized":"2024-09-21T12:25:30.012592+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:25:03.877610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:25:03.877610+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:26.706650+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":285,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.019454+0000","last_change":"2024-09-21T12:24:32.318918+0000","last_active":"2024-09-21T12:25:30.019454+0000","last_peered":"2024-09-21T12:25:30.019454+0000","last_clean":"2024-09-21T12:25:30.019454+0000","last_became_active":"2024-09-21T12:20:17.297374+0000","last_became_peered":"2024-09-21T12:20:17.297374+0000","last_unstale":"2024-09-21T12:25:30.019454+0000","last_undegraded":"2024-09-21T12:25:30.019454+0000","last_fullsized":"2024-09-21T12:25:30.019454+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-21T12:24:32.318832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:32.318832+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:44.299112+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":270,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.019282+0000","last_change":"2024-09-21T12:23:59.367953+0000","last_active":"2024-09-21T12:25:30.019282+0000","last_peered":"2024-09-21T12:25:30.019282+0000","last_clean":"2024-09-21T12:25:30.019282+0000","last_became_active":"2024-09-21T12:20:15.308356+0000","last_became_peered":"2024-09-21T12:20:15.308356+0000","last_unstale":"2024-09-21T12:25:30.019282+0000","last_undegraded":"2024-09-21T12:25:30.019282+0000","last_fullsized":"2024-09-21T12:25:30.019282+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:59.367778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:23:59.367778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:21.472609+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":279,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.025526+0000","last_change":"2024-09-21T12:24:49.792138+0000","last_active":"2024-09-21T12:25:30.025526+0000","last_peered":"2024-09-21T12:25:30.025526+0000","last_clean":"2024-09-21T12:25:30.025526+0000","last_became_active":"2024-09-21T12:20:14.288358+0000","last_became_peered":"2024-09-21T12:20:14.288358+0000","last_unstale":"2024-09-21T12:25:30.025526+0000","last_undegraded":"2024-09-21T12:25:30.025526+0000","last_fullsized":"2024-09-21T12:25:30.025526+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:24:49.792086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:24:49.792086+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:11.091093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019042449999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":254,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.024919+0000","last_change":"2024-09-21T12:23:29.274584+0000","last_active":"2024-09-21T12:25:30.024919+0000","last_peered":"2024-09-21T12:25:30.024919+0000","last_clean":"2024-09-21T12:25:30.024919+0000","last_became_active":"2024-09-21T12:20:19.309724+0000","last_became_peered":"2024-09-21T12:20:19.309724+0000","last_unstale":"2024-09-21T12:25:30.024919+0000","last_undegraded":"2024-09-21T12:25:30.024919+0000","last_fullsized":"2024-09-21T12:25:30.024919+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:29.274316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:29.274316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:43.938081+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":241,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.012562+0000","last_change":"2024-09-21T12:20:21.593267+0000","last_active":"2024-09-21T12:25:30.012562+0000","last_peered":"2024-09-21T12:25:30.012562+0000","last_clean":"2024-09-21T12:25:30.012562+0000","last_became_active":"2024-09-21T12:20:21.592982+0000","last_became_peered":"2024-09-21T12:20:21.592982+0000","last_unstale":"2024-09-21T12:25:30.012562+0000","last_undegraded":"2024-09-21T12:25:30.012562+0000","last_fullsized":"2024-09-21T12:25:30.012562+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:22.831735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"179'4093","reported_seq":4142,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:31.709285+0000","last_change":"2024-09-21T12:24:46.326200+0000","last_active":"2024-09-21T12:25:31.709285+0000","last_peered":"2024-09-21T12:25:31.709285+0000","last_clean":"2024-09-21T12:25:31.709285+0000","last_became_active":"2024-09-21T12:24:46.325838+0000","last_became_peered":"2024-09-21T12:24:46.325838+0000","last_unstale":"2024-09-21T12:25:31.709285+0000","last_undegraded":"2024-09-21T12:25:31.709285+0000","last_fullsized":"2024-09-21T12:25:31.709285+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4093,"log_dups_size":0,"ondisk_log_size":4093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:15.188003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268238848,"num_objects":4093,"num_object_clones":0,"num_object_copies":12279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4093,"num_write_kb":261952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"76'79","reported_seq":184,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.021377+0000","last_change":"2024-09-21T12:23:57.442430+0000","last_active":"2024-09-21T12:25:30.021377+0000","last_peered":"2024-09-21T12:25:30.021377+0000","last_clean":"2024-09-21T12:25:30.021377+0000","last_became_active":"2024-09-21T12:22:50.912088+0000","last_became_peered":"2024-09-21T12:22:50.912088+0000","last_unstale":"2024-09-21T12:25:30.021377+0000","last_undegraded":"2024-09-21T12:25:30.021377+0000","last_fullsized":"2024-09-21T12:25:30.021377+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:23:57.442284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:23:57.442284+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:25.572064+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":274,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.866773+0000","last_change":"2024-09-21T12:22:49.933621+0000","last_active":"2024-09-21T12:25:26.866773+0000","last_peered":"2024-09-21T12:25:26.866773+0000","last_clean":"2024-09-21T12:25:26.866773+0000","last_became_active":"2024-09-21T12:20:17.299325+0000","last_became_peered":"2024-09-21T12:20:17.299325+0000","last_unstale":"2024-09-21T12:25:26.866773+0000","last_undegraded":"2024-09-21T12:25:26.866773+0000","last_fullsized":"2024-09-21T12:25:26.866773+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:22:49.933382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:49.933382+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:23.687742+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":289,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.864613+0000","last_change":"2024-09-21T12:24:48.073954+0000","last_active":"2024-09-21T12:25:26.864613+0000","last_peered":"2024-09-21T12:25:26.864613+0000","last_clean":"2024-09-21T12:25:26.864613+0000","last_became_active":"2024-09-21T12:22:50.923796+0000","last_became_peered":"2024-09-21T12:22:50.923796+0000","last_unstale":"2024-09-21T12:25:26.864613+0000","last_undegraded":"2024-09-21T12:25:26.864613+0000","last_fullsized":"2024-09-21T12:25:26.864613+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:48.073424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:48.073424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:52.673877+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":583,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.391226+0000","last_change":"2024-09-21T12:21:22.834972+0000","last_active":"2024-09-21T12:25:30.391226+0000","last_peered":"2024-09-21T12:25:30.391226+0000","last_clean":"2024-09-21T12:25:30.391226+0000","last_became_active":"2024-09-21T12:20:14.287543+0000","last_became_peered":"2024-09-21T12:20:14.287543+0000","last_unstale":"2024-09-21T12:25:30.391226+0000","last_undegraded":"2024-09-21T12:25:30.391226+0000","last_fullsized":"2024-09-21T12:25:30.391226+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-21T12:21:22.834917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:21:22.834917+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":311,"num_read_kb":306,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":253,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.010685+0000","last_change":"2024-09-21T12:22:36.135464+0000","last_active":"2024-09-21T12:25:30.010685+0000","last_peered":"2024-09-21T12:25:30.010685+0000","last_clean":"2024-09-21T12:25:30.010685+0000","last_became_active":"2024-09-21T12:20:19.314217+0000","last_became_peered":"2024-09-21T12:20:19.314217+0000","last_unstale":"2024-09-21T12:25:30.010685+0000","last_undegraded":"2024-09-21T12:25:30.010685+0000","last_fullsized":"2024-09-21T12:25:30.010685+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:36.135390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:36.135390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:10.326219+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":258,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.019140+0000","last_change":"2024-09-21T12:24:35.356926+0000","last_active":"2024-09-21T12:25:30.019140+0000","last_peered":"2024-09-21T12:25:30.019140+0000","last_clean":"2024-09-21T12:25:30.019140+0000","last_became_active":"2024-09-21T12:20:21.311188+0000","last_became_peered":"2024-09-21T12:20:21.311188+0000","last_unstale":"2024-09-21T12:25:30.019140+0000","last_undegraded":"2024-09-21T12:25:30.019140+0000","last_fullsized":"2024-09-21T12:25:30.019140+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:35.356865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:35.356865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:38.245203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"179'4242","reported_seq":4291,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:31.600720+0000","last_change":"2024-09-21T12:24:46.318053+0000","last_active":"2024-09-21T12:25:31.600720+0000","last_peered":"2024-09-21T12:25:31.600720+0000","last_clean":"2024-09-21T12:25:31.600720+0000","last_became_active":"2024-09-21T12:24:46.317644+0000","last_became_peered":"2024-09-21T12:24:46.317644+0000","last_unstale":"2024-09-21T12:25:31.600720+0000","last_undegraded":"2024-09-21T12:25:31.600720+0000","last_fullsized":"2024-09-21T12:25:31.600720+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4242,"log_dups_size":0,"ondisk_log_size":4242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:56.464267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278003712,"num_objects":4242,"num_object_clones":0,"num_object_copies":12726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4242,"num_write_kb":271488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"121'19","reported_seq":426,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.016486+0000","last_change":"2024-09-21T12:24:27.932976+0000","last_active":"2024-09-21T12:25:30.016486+0000","last_peered":"2024-09-21T12:25:30.016486+0000","last_clean":"2024-09-21T12:25:30.016486+0000","last_became_active":"2024-09-21T12:22:50.921690+0000","last_became_peered":"2024-09-21T12:22:50.921690+0000","last_unstale":"2024-09-21T12:25:30.016486+0000","last_undegraded":"2024-09-21T12:25:30.016486+0000","last_fullsized":"2024-09-21T12:25:30.016486+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'19","last_scrub_stamp":"2024-09-21T12:24:27.932891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:27.932891+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:56.271337+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":264,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.866227+0000","last_change":"2024-09-21T12:21:17.107378+0000","last_active":"2024-09-21T12:25:26.866227+0000","last_peered":"2024-09-21T12:25:26.866227+0000","last_clean":"2024-09-21T12:25:26.866227+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:25:26.866227+0000","last_undegraded":"2024-09-21T12:25:26.866227+0000","last_fullsized":"2024-09-21T12:25:26.866227+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:21:17.107309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:17.107309+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:12.114826+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":325,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.013384+0000","last_change":"2024-09-21T12:24:31.089221+0000","last_active":"2024-09-21T12:25:30.013384+0000","last_peered":"2024-09-21T12:25:30.013384+0000","last_clean":"2024-09-21T12:25:30.013384+0000","last_became_active":"2024-09-21T12:22:15.049592+0000","last_became_peered":"2024-09-21T12:22:15.049592+0000","last_unstale":"2024-09-21T12:25:30.013384+0000","last_undegraded":"2024-09-21T12:25:30.013384+0000","last_fullsized":"2024-09-21T12:25:30.013384+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:24:31.089155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:24:31.089155+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:00.164503+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":245,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.013239+0000","last_change":"2024-09-21T12:20:19.310280+0000","last_active":"2024-09-21T12:25:30.013239+0000","last_peered":"2024-09-21T12:25:30.013239+0000","last_clean":"2024-09-21T12:25:30.013239+0000","last_became_active":"2024-09-21T12:20:19.309859+0000","last_became_peered":"2024-09-21T12:20:19.309859+0000","last_unstale":"2024-09-21T12:25:30.013239+0000","last_undegraded":"2024-09-21T12:25:30.013239+0000","last_fullsized":"2024-09-21T12:25:30.013239+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:28.753163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":278,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.010677+0000","last_change":"2024-09-21T12:23:56.364599+0000","last_active":"2024-09-21T12:25:30.010677+0000","last_peered":"2024-09-21T12:25:30.010677+0000","last_clean":"2024-09-21T12:25:30.010677+0000","last_became_active":"2024-09-21T12:22:50.913493+0000","last_became_peered":"2024-09-21T12:22:50.913493+0000","last_unstale":"2024-09-21T12:25:30.010677+0000","last_undegraded":"2024-09-21T12:25:30.010677+0000","last_fullsized":"2024-09-21T12:25:30.010677+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:56.364537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:56.364537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"178'3758","reported_seq":3805,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.978711+0000","last_change":"2024-09-21T12:24:46.315286+0000","last_active":"2024-09-21T12:25:26.978711+0000","last_peered":"2024-09-21T12:25:26.978711+0000","last_clean":"2024-09-21T12:25:26.978711+0000","last_became_active":"2024-09-21T12:24:46.314957+0000","last_became_peered":"2024-09-21T12:24:46.314957+0000","last_unstale":"2024-09-21T12:25:26.978711+0000","last_undegraded":"2024-09-21T12:25:26.978711+0000","last_fullsized":"2024-09-21T12:25:26.978711+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":3758,"log_dups_size":0,"ondisk_log_size":3758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:52.237123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246284288,"num_objects":3758,"num_object_clones":0,"num_object_copies":11274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3758,"num_write_kb":240512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":267,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.016514+0000","last_change":"2024-09-21T12:23:51.061783+0000","last_active":"2024-09-21T12:25:30.016514+0000","last_peered":"2024-09-21T12:25:30.016514+0000","last_clean":"2024-09-21T12:25:30.016514+0000","last_became_active":"2024-09-21T12:23:46.885802+0000","last_became_peered":"2024-09-21T12:23:46.885802+0000","last_unstale":"2024-09-21T12:25:30.016514+0000","last_undegraded":"2024-09-21T12:25:30.016514+0000","last_fullsized":"2024-09-21T12:25:30.016514+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:23:51.061706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:51.061706+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:14.764585+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":185,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.025387+0000","last_change":"2024-09-21T12:22:50.908380+0000","last_active":"2024-09-21T12:25:30.025387+0000","last_peered":"2024-09-21T12:25:30.025387+0000","last_clean":"2024-09-21T12:25:30.025387+0000","last_became_active":"2024-09-21T12:22:50.908060+0000","last_became_peered":"2024-09-21T12:22:50.908060+0000","last_unstale":"2024-09-21T12:25:30.025387+0000","last_undegraded":"2024-09-21T12:25:30.025387+0000","last_fullsized":"2024-09-21T12:25:30.025387+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:05.892153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:05.892153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:26.966868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":282,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.010441+0000","last_change":"2024-09-21T12:23:19.372762+0000","last_active":"2024-09-21T12:25:30.010441+0000","last_peered":"2024-09-21T12:25:30.010441+0000","last_clean":"2024-09-21T12:25:30.010441+0000","last_became_active":"2024-09-21T12:22:50.915081+0000","last_became_peered":"2024-09-21T12:22:50.915081+0000","last_unstale":"2024-09-21T12:25:30.010441+0000","last_undegraded":"2024-09-21T12:25:30.010441+0000","last_fullsized":"2024-09-21T12:25:30.010441+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:19.372719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:19.372719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:37.323662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":246,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.868846+0000","last_change":"2024-09-21T12:23:46.282792+0000","last_active":"2024-09-21T12:25:26.868846+0000","last_peered":"2024-09-21T12:25:26.868846+0000","last_clean":"2024-09-21T12:25:26.868846+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:25:26.868846+0000","last_undegraded":"2024-09-21T12:25:26.868846+0000","last_fullsized":"2024-09-21T12:25:26.868846+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:46.282661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:46.282661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:15.069140+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"179'4310","reported_seq":4359,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:31.750769+0000","last_change":"2024-09-21T12:24:46.323690+0000","last_active":"2024-09-21T12:25:31.750769+0000","last_peered":"2024-09-21T12:25:31.750769+0000","last_clean":"2024-09-21T12:25:31.750769+0000","last_became_active":"2024-09-21T12:24:46.323402+0000","last_became_peered":"2024-09-21T12:24:46.323402+0000","last_unstale":"2024-09-21T12:25:31.750769+0000","last_undegraded":"2024-09-21T12:25:31.750769+0000","last_fullsized":"2024-09-21T12:25:31.750769+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4310,"log_dups_size":0,"ondisk_log_size":4310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:55.565560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282460160,"num_objects":4310,"num_object_clones":0,"num_object_copies":12930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4310,"num_write_kb":275840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"76'79","reported_seq":177,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.009726+0000","last_change":"2024-09-21T12:24:28.372308+0000","last_active":"2024-09-21T12:25:30.009726+0000","last_peered":"2024-09-21T12:25:30.009726+0000","last_clean":"2024-09-21T12:25:30.009726+0000","last_became_active":"2024-09-21T12:22:17.087245+0000","last_became_peered":"2024-09-21T12:22:17.087245+0000","last_unstale":"2024-09-21T12:25:30.009726+0000","last_undegraded":"2024-09-21T12:25:30.009726+0000","last_fullsized":"2024-09-21T12:25:30.009726+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:24:28.372230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:24:28.372230+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:46.819824+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":302,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.025090+0000","last_change":"2024-09-21T12:23:50.328004+0000","last_active":"2024-09-21T12:25:30.025090+0000","last_peered":"2024-09-21T12:25:30.025090+0000","last_clean":"2024-09-21T12:25:30.025090+0000","last_became_active":"2024-09-21T12:23:46.897197+0000","last_became_peered":"2024-09-21T12:23:46.897197+0000","last_unstale":"2024-09-21T12:25:30.025090+0000","last_undegraded":"2024-09-21T12:25:30.025090+0000","last_fullsized":"2024-09-21T12:25:30.025090+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-21T12:23:50.327918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:50.327918+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":290,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.016083+0000","last_change":"2024-09-21T12:23:52.067111+0000","last_active":"2024-09-21T12:25:30.016083+0000","last_peered":"2024-09-21T12:25:30.016083+0000","last_clean":"2024-09-21T12:25:30.016083+0000","last_became_active":"2024-09-21T12:22:50.906886+0000","last_became_peered":"2024-09-21T12:22:50.906886+0000","last_unstale":"2024-09-21T12:25:30.016083+0000","last_undegraded":"2024-09-21T12:25:30.016083+0000","last_fullsized":"2024-09-21T12:25:30.016083+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:52.067065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:23:52.067065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:13.795671+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":275,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.019234+0000","last_change":"2024-09-21T12:23:17.434377+0000","last_active":"2024-09-21T12:25:30.019234+0000","last_peered":"2024-09-21T12:25:30.019234+0000","last_clean":"2024-09-21T12:25:30.019234+0000","last_became_active":"2024-09-21T12:22:50.927438+0000","last_became_peered":"2024-09-21T12:22:50.927438+0000","last_unstale":"2024-09-21T12:25:30.019234+0000","last_undegraded":"2024-09-21T12:25:30.019234+0000","last_fullsized":"2024-09-21T12:25:30.019234+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:17.434261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:17.434261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:46.370615+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":263,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.025914+0000","last_change":"2024-09-21T12:22:50.914246+0000","last_active":"2024-09-21T12:25:30.025914+0000","last_peered":"2024-09-21T12:25:30.025914+0000","last_clean":"2024-09-21T12:25:30.025914+0000","last_became_active":"2024-09-21T12:22:50.913898+0000","last_became_peered":"2024-09-21T12:22:50.913898+0000","last_unstale":"2024-09-21T12:25:30.025914+0000","last_undegraded":"2024-09-21T12:25:30.025914+0000","last_fullsized":"2024-09-21T12:25:30.025914+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:37.023546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"178'3855","reported_seq":3902,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:27.040350+0000","last_change":"2024-09-21T12:24:46.315895+0000","last_active":"2024-09-21T12:25:27.040350+0000","last_peered":"2024-09-21T12:25:27.040350+0000","last_clean":"2024-09-21T12:25:27.040350+0000","last_became_active":"2024-09-21T12:24:46.315601+0000","last_became_peered":"2024-09-21T12:24:46.315601+0000","last_unstale":"2024-09-21T12:25:27.040350+0000","last_undegraded":"2024-09-21T12:25:27.040350+0000","last_fullsized":"2024-09-21T12:25:27.040350+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":3855,"log_dups_size":0,"ondisk_log_size":3855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:06.483656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252641280,"num_objects":3855,"num_object_clones":0,"num_object_copies":11565,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3855,"num_write_kb":246720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"76'79","reported_seq":315,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.012092+0000","last_change":"2024-09-21T12:22:17.073301+0000","last_active":"2024-09-21T12:25:30.012092+0000","last_peered":"2024-09-21T12:25:30.012092+0000","last_clean":"2024-09-21T12:25:30.012092+0000","last_became_active":"2024-09-21T12:22:17.072872+0000","last_became_peered":"2024-09-21T12:22:17.072872+0000","last_unstale":"2024-09-21T12:25:30.012092+0000","last_undegraded":"2024-09-21T12:25:30.012092+0000","last_fullsized":"2024-09-21T12:25:30.012092+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T12:20:24.850157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:20:24.850157+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:25:43.066052+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":21,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":116,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.025638+0000","last_change":"2024-09-21T12:25:03.787373+0000","last_active":"2024-09-21T12:25:30.025638+0000","last_peered":"2024-09-21T12:25:30.025638+0000","last_clean":"2024-09-21T12:25:30.025638+0000","last_became_active":"2024-09-21T12:23:46.902260+0000","last_became_peered":"2024-09-21T12:23:46.902260+0000","last_unstale":"2024-09-21T12:25:30.025638+0000","last_undegraded":"2024-09-21T12:25:30.025638+0000","last_fullsized":"2024-09-21T12:25:30.025638+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:25:03.787288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:03.787288+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:04.352295+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":279,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.869893+0000","last_change":"2024-09-21T12:22:50.928240+0000","last_active":"2024-09-21T12:25:26.869893+0000","last_peered":"2024-09-21T12:25:26.869893+0000","last_clean":"2024-09-21T12:25:26.869893+0000","last_became_active":"2024-09-21T12:22:50.927946+0000","last_became_peered":"2024-09-21T12:22:50.927946+0000","last_unstale":"2024-09-21T12:25:26.869893+0000","last_undegraded":"2024-09-21T12:25:26.869893+0000","last_fullsized":"2024-09-21T12:25:26.869893+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:08.999015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:08.999015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:24.102761+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":334,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.007556+0000","last_change":"2024-09-21T12:24:52.846590+0000","last_active":"2024-09-21T12:25:30.007556+0000","last_peered":"2024-09-21T12:25:30.007556+0000","last_clean":"2024-09-21T12:25:30.007556+0000","last_became_active":"2024-09-21T12:20:19.307924+0000","last_became_peered":"2024-09-21T12:20:19.307924+0000","last_unstale":"2024-09-21T12:25:30.007556+0000","last_undegraded":"2024-09-21T12:25:30.007556+0000","last_fullsized":"2024-09-21T12:25:30.007556+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:24:52.846112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:52.846112+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:08.133859+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":193,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.020112+0000","last_change":"2024-09-21T12:23:56.471960+0000","last_active":"2024-09-21T12:25:30.020112+0000","last_peered":"2024-09-21T12:25:30.020112+0000","last_clean":"2024-09-21T12:25:30.020112+0000","last_became_active":"2024-09-21T12:22:50.905470+0000","last_became_peered":"2024-09-21T12:22:50.905470+0000","last_unstale":"2024-09-21T12:25:30.020112+0000","last_undegraded":"2024-09-21T12:25:30.020112+0000","last_fullsized":"2024-09-21T12:25:30.020112+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:56.471116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:56.471116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:22.516139+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.7","version":"179'4218","reported_seq":4267,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:31.714498+0000","last_change":"2024-09-21T12:24:46.317724+0000","last_active":"2024-09-21T12:25:31.714498+0000","last_peered":"2024-09-21T12:25:31.714498+0000","last_clean":"2024-09-21T12:25:31.714498+0000","last_became_active":"2024-09-21T12:24:46.317468+0000","last_became_peered":"2024-09-21T12:24:46.317468+0000","last_unstale":"2024-09-21T12:25:31.714498+0000","last_undegraded":"2024-09-21T12:25:31.714498+0000","last_fullsized":"2024-09-21T12:25:31.714498+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4218,"log_dups_size":0,"ondisk_log_size":4218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:52.389257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276430848,"num_objects":4218,"num_object_clones":0,"num_object_copies":12654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4218,"num_write_kb":269952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":339,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.863334+0000","last_change":"2024-09-21T12:25:00.237083+0000","last_active":"2024-09-21T12:25:26.863334+0000","last_peered":"2024-09-21T12:25:26.863334+0000","last_clean":"2024-09-21T12:25:26.863334+0000","last_became_active":"2024-09-21T12:22:50.922343+0000","last_became_peered":"2024-09-21T12:22:50.922343+0000","last_unstale":"2024-09-21T12:25:26.863334+0000","last_undegraded":"2024-09-21T12:25:26.863334+0000","last_fullsized":"2024-09-21T12:25:26.863334+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:25:00.236990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:00.236990+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:26.107161+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":277,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.451939+0000","last_change":"2024-09-21T12:25:30.451939+0000","last_active":"2024-09-21T12:25:30.451939+0000","last_peered":"2024-09-21T12:25:30.451939+0000","last_clean":"2024-09-21T12:25:30.451939+0000","last_became_active":"2024-09-21T12:20:15.302262+0000","last_became_peered":"2024-09-21T12:20:15.302262+0000","last_unstale":"2024-09-21T12:25:30.451939+0000","last_undegraded":"2024-09-21T12:25:30.451939+0000","last_fullsized":"2024-09-21T12:25:30.451939+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:30.451876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:30.451876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:51.913189+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":283,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.026320+0000","last_change":"2024-09-21T12:23:58.277901+0000","last_active":"2024-09-21T12:25:30.026320+0000","last_peered":"2024-09-21T12:25:30.026320+0000","last_clean":"2024-09-21T12:25:30.026320+0000","last_became_active":"2024-09-21T12:22:50.925974+0000","last_became_peered":"2024-09-21T12:22:50.925974+0000","last_unstale":"2024-09-21T12:25:30.026320+0000","last_undegraded":"2024-09-21T12:25:30.026320+0000","last_fullsized":"2024-09-21T12:25:30.026320+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:58.277829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:58.277829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":248,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.864528+0000","last_change":"2024-09-21T12:23:28.121344+0000","last_active":"2024-09-21T12:25:26.864528+0000","last_peered":"2024-09-21T12:25:26.864528+0000","last_clean":"2024-09-21T12:25:26.864528+0000","last_became_active":"2024-09-21T12:20:21.311493+0000","last_became_peered":"2024-09-21T12:20:21.311493+0000","last_unstale":"2024-09-21T12:25:26.864528+0000","last_undegraded":"2024-09-21T12:25:26.864528+0000","last_fullsized":"2024-09-21T12:25:26.864528+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:28.121286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:28.121286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:24.681035+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"178'3856","reported_seq":3903,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:27.078579+0000","last_change":"2024-09-21T12:24:46.327091+0000","last_active":"2024-09-21T12:25:27.078579+0000","last_peered":"2024-09-21T12:25:27.078579+0000","last_clean":"2024-09-21T12:25:27.078579+0000","last_became_active":"2024-09-21T12:24:46.325838+0000","last_became_peered":"2024-09-21T12:24:46.325838+0000","last_unstale":"2024-09-21T12:25:27.078579+0000","last_undegraded":"2024-09-21T12:25:27.078579+0000","last_fullsized":"2024-09-21T12:25:27.078579+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":3856,"log_dups_size":0,"ondisk_log_size":3856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:07.844175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252706816,"num_objects":3856,"num_object_clones":0,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3856,"num_write_kb":246784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":296,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.015753+0000","last_change":"2024-09-21T12:24:02.960807+0000","last_active":"2024-09-21T12:25:30.015753+0000","last_peered":"2024-09-21T12:25:30.015753+0000","last_clean":"2024-09-21T12:25:30.015753+0000","last_became_active":"2024-09-21T12:20:17.692867+0000","last_became_peered":"2024-09-21T12:20:17.692867+0000","last_unstale":"2024-09-21T12:25:30.015753+0000","last_undegraded":"2024-09-21T12:25:30.015753+0000","last_fullsized":"2024-09-21T12:25:30.015753+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-21T12:24:02.960733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:02.960733+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:18.893627+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":261,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.012906+0000","last_change":"2024-09-21T12:23:59.046000+0000","last_active":"2024-09-21T12:25:30.012906+0000","last_peered":"2024-09-21T12:25:30.012906+0000","last_clean":"2024-09-21T12:25:30.012906+0000","last_became_active":"2024-09-21T12:20:15.299333+0000","last_became_peered":"2024-09-21T12:20:15.299333+0000","last_unstale":"2024-09-21T12:25:30.012906+0000","last_undegraded":"2024-09-21T12:25:30.012906+0000","last_fullsized":"2024-09-21T12:25:30.012906+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:59.045927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:23:59.045927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":254,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.024522+0000","last_change":"2024-09-21T12:24:34.060244+0000","last_active":"2024-09-21T12:25:30.024522+0000","last_peered":"2024-09-21T12:25:30.024522+0000","last_clean":"2024-09-21T12:25:30.024522+0000","last_became_active":"2024-09-21T12:20:19.306952+0000","last_became_peered":"2024-09-21T12:20:19.306952+0000","last_unstale":"2024-09-21T12:25:30.024522+0000","last_undegraded":"2024-09-21T12:25:30.024522+0000","last_fullsized":"2024-09-21T12:25:30.024522+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:34.060188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:34.060188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:46.315441+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":241,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.013283+0000","last_change":"2024-09-21T12:20:21.314891+0000","last_active":"2024-09-21T12:25:30.013283+0000","last_peered":"2024-09-21T12:25:30.013283+0000","last_clean":"2024-09-21T12:25:30.013283+0000","last_became_active":"2024-09-21T12:20:21.314545+0000","last_became_peered":"2024-09-21T12:20:21.314545+0000","last_unstale":"2024-09-21T12:25:30.013283+0000","last_undegraded":"2024-09-21T12:25:30.013283+0000","last_fullsized":"2024-09-21T12:25:30.013283+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:48.052208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"179'4164","reported_seq":4213,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:31.609607+0000","last_change":"2024-09-21T12:24:46.325063+0000","last_active":"2024-09-21T12:25:31.609607+0000","last_peered":"2024-09-21T12:25:31.609607+0000","last_clean":"2024-09-21T12:25:31.609607+0000","last_became_active":"2024-09-21T12:24:46.324554+0000","last_became_peered":"2024-09-21T12:24:46.324554+0000","last_unstale":"2024-09-21T12:25:31.609607+0000","last_undegraded":"2024-09-21T12:25:31.609607+0000","last_fullsized":"2024-09-21T12:25:31.609607+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4164,"log_dups_size":0,"ondisk_log_size":4164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:48.160004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272891904,"num_objects":4164,"num_object_clones":0,"num_object_copies":12492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4164,"num_write_kb":266496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":268,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.021272+0000","last_change":"2024-09-21T12:24:54.445801+0000","last_active":"2024-09-21T12:25:30.021272+0000","last_peered":"2024-09-21T12:25:30.021272+0000","last_clean":"2024-09-21T12:25:30.021272+0000","last_became_active":"2024-09-21T12:22:50.910973+0000","last_became_peered":"2024-09-21T12:22:50.910973+0000","last_unstale":"2024-09-21T12:25:30.021272+0000","last_undegraded":"2024-09-21T12:25:30.021272+0000","last_fullsized":"2024-09-21T12:25:30.021272+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:24:54.445714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:54.445714+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:16.862302+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":282,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.013078+0000","last_change":"2024-09-21T12:22:50.928139+0000","last_active":"2024-09-21T12:25:30.013078+0000","last_peered":"2024-09-21T12:25:30.013078+0000","last_clean":"2024-09-21T12:25:30.013078+0000","last_became_active":"2024-09-21T12:22:50.927859+0000","last_became_peered":"2024-09-21T12:22:50.927859+0000","last_unstale":"2024-09-21T12:25:30.013078+0000","last_undegraded":"2024-09-21T12:25:30.013078+0000","last_fullsized":"2024-09-21T12:25:30.013078+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:20.865749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.865749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":254,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.007348+0000","last_change":"2024-09-21T12:21:23.393320+0000","last_active":"2024-09-21T12:25:30.007348+0000","last_peered":"2024-09-21T12:25:30.007348+0000","last_clean":"2024-09-21T12:25:30.007348+0000","last_became_active":"2024-09-21T12:20:19.312572+0000","last_became_peered":"2024-09-21T12:20:19.312572+0000","last_unstale":"2024-09-21T12:25:30.007348+0000","last_undegraded":"2024-09-21T12:25:30.007348+0000","last_fullsized":"2024-09-21T12:25:30.007348+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.393270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.393270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:39.019942+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":246,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.868110+0000","last_change":"2024-09-21T12:22:36.006177+0000","last_active":"2024-09-21T12:25:26.868110+0000","last_peered":"2024-09-21T12:25:26.868110+0000","last_clean":"2024-09-21T12:25:26.868110+0000","last_became_active":"2024-09-21T12:20:21.317223+0000","last_became_peered":"2024-09-21T12:20:21.317223+0000","last_unstale":"2024-09-21T12:25:26.868110+0000","last_undegraded":"2024-09-21T12:25:26.868110+0000","last_fullsized":"2024-09-21T12:25:26.868110+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:36.006109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:36.006109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:44.698416+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"179'4288","reported_seq":4335,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:31.584132+0000","last_change":"2024-09-21T12:24:46.323500+0000","last_active":"2024-09-21T12:25:31.584132+0000","last_peered":"2024-09-21T12:25:31.584132+0000","last_clean":"2024-09-21T12:25:31.584132+0000","last_became_active":"2024-09-21T12:24:46.323192+0000","last_became_peered":"2024-09-21T12:24:46.323192+0000","last_unstale":"2024-09-21T12:25:31.584132+0000","last_undegraded":"2024-09-21T12:25:31.584132+0000","last_fullsized":"2024-09-21T12:25:31.584132+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4288,"log_dups_size":0,"ondisk_log_size":4288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:57.240434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281018368,"num_objects":4288,"num_object_clones":0,"num_object_copies":12864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4288,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":298,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.009812+0000","last_change":"2024-09-21T12:25:03.416100+0000","last_active":"2024-09-21T12:25:30.009812+0000","last_peered":"2024-09-21T12:25:30.009812+0000","last_clean":"2024-09-21T12:25:30.009812+0000","last_became_active":"2024-09-21T12:20:17.692870+0000","last_became_peered":"2024-09-21T12:20:17.692870+0000","last_unstale":"2024-09-21T12:25:30.009812+0000","last_undegraded":"2024-09-21T12:25:30.009812+0000","last_fullsized":"2024-09-21T12:25:30.009812+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:25:03.416033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:03.416033+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:09.840901+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":320,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.012950+0000","last_change":"2024-09-21T12:24:29.054851+0000","last_active":"2024-09-21T12:25:30.012950+0000","last_peered":"2024-09-21T12:25:30.012950+0000","last_clean":"2024-09-21T12:25:30.012950+0000","last_became_active":"2024-09-21T12:22:50.927141+0000","last_became_peered":"2024-09-21T12:22:50.927141+0000","last_unstale":"2024-09-21T12:25:30.012950+0000","last_undegraded":"2024-09-21T12:25:30.012950+0000","last_fullsized":"2024-09-21T12:25:30.012950+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-21T12:24:29.054772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:29.054772+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:50.900488+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"121'11","reported_seq":358,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.012049+0000","last_change":"2024-09-21T12:25:02.916145+0000","last_active":"2024-09-21T12:25:30.012049+0000","last_peered":"2024-09-21T12:25:30.012049+0000","last_clean":"2024-09-21T12:25:30.012049+0000","last_became_active":"2024-09-21T12:22:50.908371+0000","last_became_peered":"2024-09-21T12:22:50.908371+0000","last_unstale":"2024-09-21T12:25:30.012049+0000","last_undegraded":"2024-09-21T12:25:30.012049+0000","last_fullsized":"2024-09-21T12:25:30.012049+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:25:02.916090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:02.916090+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:31.534554+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":249,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.011500+0000","last_change":"2024-09-21T12:22:45.223570+0000","last_active":"2024-09-21T12:25:30.011500+0000","last_peered":"2024-09-21T12:25:30.011500+0000","last_clean":"2024-09-21T12:25:30.011500+0000","last_became_active":"2024-09-21T12:20:21.592576+0000","last_became_peered":"2024-09-21T12:20:21.592576+0000","last_unstale":"2024-09-21T12:25:30.011500+0000","last_undegraded":"2024-09-21T12:25:30.011500+0000","last_fullsized":"2024-09-21T12:25:30.011500+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:45.223396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:45.223396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:20.386745+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":277,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.014946+0000","last_change":"2024-09-21T12:24:25.927008+0000","last_active":"2024-09-21T12:25:30.014946+0000","last_peered":"2024-09-21T12:25:30.014946+0000","last_clean":"2024-09-21T12:25:30.014946+0000","last_became_active":"2024-09-21T12:20:15.303480+0000","last_became_peered":"2024-09-21T12:20:15.303480+0000","last_unstale":"2024-09-21T12:25:30.014946+0000","last_undegraded":"2024-09-21T12:25:30.014946+0000","last_fullsized":"2024-09-21T12:25:30.014946+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:25.926922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:25.926922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:31.586807+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":263,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.867613+0000","last_change":"2024-09-21T12:22:36.999995+0000","last_active":"2024-09-21T12:25:26.867613+0000","last_peered":"2024-09-21T12:25:26.867613+0000","last_clean":"2024-09-21T12:25:26.867613+0000","last_became_active":"2024-09-21T12:20:17.694628+0000","last_became_peered":"2024-09-21T12:20:17.694628+0000","last_unstale":"2024-09-21T12:25:26.867613+0000","last_undegraded":"2024-09-21T12:25:26.867613+0000","last_fullsized":"2024-09-21T12:25:26.867613+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-21T12:22:36.999705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:36.999705+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:43.707926+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":269,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.010471+0000","last_change":"2024-09-21T12:25:02.414377+0000","last_active":"2024-09-21T12:25:30.010471+0000","last_peered":"2024-09-21T12:25:30.010471+0000","last_clean":"2024-09-21T12:25:30.010471+0000","last_became_active":"2024-09-21T12:20:19.313882+0000","last_became_peered":"2024-09-21T12:20:19.313882+0000","last_unstale":"2024-09-21T12:25:30.010471+0000","last_undegraded":"2024-09-21T12:25:30.010471+0000","last_fullsized":"2024-09-21T12:25:30.010471+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:02.414298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:02.414298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:04.541864+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":270,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.012471+0000","last_change":"2024-09-21T12:24:00.066522+0000","last_active":"2024-09-21T12:25:30.012471+0000","last_peered":"2024-09-21T12:25:30.012471+0000","last_clean":"2024-09-21T12:25:30.012471+0000","last_became_active":"2024-09-21T12:22:50.904404+0000","last_became_peered":"2024-09-21T12:22:50.904404+0000","last_unstale":"2024-09-21T12:25:30.012471+0000","last_undegraded":"2024-09-21T12:25:30.012471+0000","last_fullsized":"2024-09-21T12:25:30.012471+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:00.066451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:00.066451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":272,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.866862+0000","last_change":"2024-09-21T12:21:24.093710+0000","last_active":"2024-09-21T12:25:26.866862+0000","last_peered":"2024-09-21T12:25:26.866862+0000","last_clean":"2024-09-21T12:25:26.866862+0000","last_became_active":"2024-09-21T12:20:17.695898+0000","last_became_peered":"2024-09-21T12:20:17.695898+0000","last_unstale":"2024-09-21T12:25:26.866862+0000","last_undegraded":"2024-09-21T12:25:26.866862+0000","last_fullsized":"2024-09-21T12:25:26.866862+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:21:24.093656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:21:24.093656+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:20.587663+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":290,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.012196+0000","last_change":"2024-09-21T12:24:26.078804+0000","last_active":"2024-09-21T12:25:30.012196+0000","last_peered":"2024-09-21T12:25:30.012196+0000","last_clean":"2024-09-21T12:25:30.012196+0000","last_became_active":"2024-09-21T12:22:50.927649+0000","last_became_peered":"2024-09-21T12:22:50.927649+0000","last_unstale":"2024-09-21T12:25:30.012196+0000","last_undegraded":"2024-09-21T12:25:30.012196+0000","last_fullsized":"2024-09-21T12:25:30.012196+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:26.078737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:26.078737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:51.918796+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":250,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.867425+0000","last_change":"2024-09-21T12:23:53.356954+0000","last_active":"2024-09-21T12:25:26.867425+0000","last_peered":"2024-09-21T12:25:26.867425+0000","last_clean":"2024-09-21T12:25:26.867425+0000","last_became_active":"2024-09-21T12:20:19.311947+0000","last_became_peered":"2024-09-21T12:20:19.311947+0000","last_unstale":"2024-09-21T12:25:26.867425+0000","last_undegraded":"2024-09-21T12:25:26.867425+0000","last_fullsized":"2024-09-21T12:25:26.867425+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:53.356896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:53.356896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":287,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.025777+0000","last_change":"2024-09-21T12:24:58.847710+0000","last_active":"2024-09-21T12:25:30.025777+0000","last_peered":"2024-09-21T12:25:30.025777+0000","last_clean":"2024-09-21T12:25:30.025777+0000","last_became_active":"2024-09-21T12:22:50.924931+0000","last_became_peered":"2024-09-21T12:22:50.924931+0000","last_unstale":"2024-09-21T12:25:30.025777+0000","last_undegraded":"2024-09-21T12:25:30.025777+0000","last_fullsized":"2024-09-21T12:25:30.025777+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:58.847643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:58.847643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:05.629548+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":290,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.010561+0000","last_change":"2024-09-21T12:23:46.925752+0000","last_active":"2024-09-21T12:25:30.010561+0000","last_peered":"2024-09-21T12:25:30.010561+0000","last_clean":"2024-09-21T12:25:30.010561+0000","last_became_active":"2024-09-21T12:23:46.898523+0000","last_became_peered":"2024-09-21T12:23:46.898523+0000","last_unstale":"2024-09-21T12:25:30.010561+0000","last_undegraded":"2024-09-21T12:25:30.010561+0000","last_fullsized":"2024-09-21T12:25:30.010561+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:22:39.201667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:39.201667+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:16.341268+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":268,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.867155+0000","last_change":"2024-09-21T12:24:25.931617+0000","last_active":"2024-09-21T12:25:26.867155+0000","last_peered":"2024-09-21T12:25:26.867155+0000","last_clean":"2024-09-21T12:25:26.867155+0000","last_became_active":"2024-09-21T12:20:15.297519+0000","last_became_peered":"2024-09-21T12:20:15.297519+0000","last_unstale":"2024-09-21T12:25:26.867155+0000","last_undegraded":"2024-09-21T12:25:26.867155+0000","last_fullsized":"2024-09-21T12:25:26.867155+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:25.931561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:25.931561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:44.486853+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":321,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.871106+0000","last_change":"2024-09-21T12:23:57.379033+0000","last_active":"2024-09-21T12:25:26.871106+0000","last_peered":"2024-09-21T12:25:26.871106+0000","last_clean":"2024-09-21T12:25:26.871106+0000","last_became_active":"2024-09-21T12:20:19.305364+0000","last_became_peered":"2024-09-21T12:20:19.305364+0000","last_unstale":"2024-09-21T12:25:26.871106+0000","last_undegraded":"2024-09-21T12:25:26.871106+0000","last_fullsized":"2024-09-21T12:25:26.871106+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:23:57.378959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:57.378959+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:16.020266+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":193,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.025995+0000","last_change":"2024-09-21T12:24:51.831043+0000","last_active":"2024-09-21T12:25:30.025995+0000","last_peered":"2024-09-21T12:25:30.025995+0000","last_clean":"2024-09-21T12:25:30.025995+0000","last_became_active":"2024-09-21T12:22:50.924770+0000","last_became_peered":"2024-09-21T12:22:50.924770+0000","last_unstale":"2024-09-21T12:25:30.025995+0000","last_undegraded":"2024-09-21T12:25:30.025995+0000","last_fullsized":"2024-09-21T12:25:30.025995+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:51.830994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:51.830994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:54.192853+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":277,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.871707+0000","last_change":"2024-09-21T12:24:50.727161+0000","last_active":"2024-09-21T12:25:26.871707+0000","last_peered":"2024-09-21T12:25:26.871707+0000","last_clean":"2024-09-21T12:25:26.871707+0000","last_became_active":"2024-09-21T12:20:17.693209+0000","last_became_peered":"2024-09-21T12:20:17.693209+0000","last_unstale":"2024-09-21T12:25:26.871707+0000","last_undegraded":"2024-09-21T12:25:26.871707+0000","last_fullsized":"2024-09-21T12:25:26.871707+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:24:50.727101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:50.727101+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:11.446387+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":282,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.012490+0000","last_change":"2024-09-21T12:22:50.908560+0000","last_active":"2024-09-21T12:25:30.012490+0000","last_peered":"2024-09-21T12:25:30.012490+0000","last_clean":"2024-09-21T12:25:30.012490+0000","last_became_active":"2024-09-21T12:22:50.908238+0000","last_became_peered":"2024-09-21T12:22:50.908238+0000","last_unstale":"2024-09-21T12:25:30.012490+0000","last_undegraded":"2024-09-21T12:25:30.012490+0000","last_fullsized":"2024-09-21T12:25:30.012490+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:15.938093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:15.938093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":325,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.010334+0000","last_change":"2024-09-21T12:22:46.216242+0000","last_active":"2024-09-21T12:25:30.010334+0000","last_peered":"2024-09-21T12:25:30.010334+0000","last_clean":"2024-09-21T12:25:30.010334+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:25:30.010334+0000","last_undegraded":"2024-09-21T12:25:30.010334+0000","last_fullsized":"2024-09-21T12:25:30.010334+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:22:46.216153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:46.216153+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:22.367827+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":240,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.866435+0000","last_change":"2024-09-21T12:20:21.309936+0000","last_active":"2024-09-21T12:25:26.866435+0000","last_peered":"2024-09-21T12:25:26.866435+0000","last_clean":"2024-09-21T12:25:26.866435+0000","last_became_active":"2024-09-21T12:20:21.309385+0000","last_became_peered":"2024-09-21T12:20:21.309385+0000","last_unstale":"2024-09-21T12:25:26.866435+0000","last_undegraded":"2024-09-21T12:25:26.866435+0000","last_fullsized":"2024-09-21T12:25:26.866435+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:06.355734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":248,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.016048+0000","last_change":"2024-09-21T12:23:59.046758+0000","last_active":"2024-09-21T12:25:30.016048+0000","last_peered":"2024-09-21T12:25:30.016048+0000","last_clean":"2024-09-21T12:25:30.016048+0000","last_became_active":"2024-09-21T12:22:50.906473+0000","last_became_peered":"2024-09-21T12:22:50.906473+0000","last_unstale":"2024-09-21T12:25:30.016048+0000","last_undegraded":"2024-09-21T12:25:30.016048+0000","last_fullsized":"2024-09-21T12:25:30.016048+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:23:59.046704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:59.046704+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:16.851121+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":185,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.013386+0000","last_change":"2024-09-21T12:24:58.948698+0000","last_active":"2024-09-21T12:25:30.013386+0000","last_peered":"2024-09-21T12:25:30.013386+0000","last_clean":"2024-09-21T12:25:30.013386+0000","last_became_active":"2024-09-21T12:22:50.903705+0000","last_became_peered":"2024-09-21T12:22:50.903705+0000","last_unstale":"2024-09-21T12:25:30.013386+0000","last_undegraded":"2024-09-21T12:25:30.013386+0000","last_fullsized":"2024-09-21T12:25:30.013386+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:58.948625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:58.948625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:23.221920+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":249,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.019047+0000","last_change":"2024-09-21T12:24:38.746182+0000","last_active":"2024-09-21T12:25:30.019047+0000","last_peered":"2024-09-21T12:25:30.019047+0000","last_clean":"2024-09-21T12:25:30.019047+0000","last_became_active":"2024-09-21T12:20:19.315418+0000","last_became_peered":"2024-09-21T12:20:19.315418+0000","last_unstale":"2024-09-21T12:25:30.019047+0000","last_undegraded":"2024-09-21T12:25:30.019047+0000","last_fullsized":"2024-09-21T12:25:30.019047+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:15.446578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":254,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.865807+0000","last_change":"2024-09-21T12:24:56.879617+0000","last_active":"2024-09-21T12:25:26.865807+0000","last_peered":"2024-09-21T12:25:26.865807+0000","last_clean":"2024-09-21T12:25:26.865807+0000","last_became_active":"2024-09-21T12:20:21.310053+0000","last_became_peered":"2024-09-21T12:20:21.310053+0000","last_unstale":"2024-09-21T12:25:26.865807+0000","last_undegraded":"2024-09-21T12:25:26.865807+0000","last_fullsized":"2024-09-21T12:25:26.865807+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:56.879558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:56.879558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:26.453298+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":266,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.019341+0000","last_change":"2024-09-21T12:20:17.299367+0000","last_active":"2024-09-21T12:25:30.019341+0000","last_peered":"2024-09-21T12:25:30.019341+0000","last_clean":"2024-09-21T12:25:30.019341+0000","last_became_active":"2024-09-21T12:20:17.299015+0000","last_became_peered":"2024-09-21T12:20:17.299015+0000","last_unstale":"2024-09-21T12:25:30.019341+0000","last_undegraded":"2024-09-21T12:25:30.019341+0000","last_fullsized":"2024-09-21T12:25:30.019341+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:40.326829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":290,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.012771+0000","last_change":"2024-09-21T12:24:30.077339+0000","last_active":"2024-09-21T12:25:30.012771+0000","last_peered":"2024-09-21T12:25:30.012771+0000","last_clean":"2024-09-21T12:25:30.012771+0000","last_became_active":"2024-09-21T12:22:50.922019+0000","last_became_peered":"2024-09-21T12:22:50.922019+0000","last_unstale":"2024-09-21T12:25:30.012771+0000","last_undegraded":"2024-09-21T12:25:30.012771+0000","last_fullsized":"2024-09-21T12:25:30.012771+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:30.077279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:30.077279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:52.329408+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":250,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.868184+0000","last_change":"2024-09-21T12:21:20.931077+0000","last_active":"2024-09-21T12:25:26.868184+0000","last_peered":"2024-09-21T12:25:26.868184+0000","last_clean":"2024-09-21T12:25:26.868184+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:25:26.868184+0000","last_undegraded":"2024-09-21T12:25:26.868184+0000","last_fullsized":"2024-09-21T12:25:26.868184+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:20.931025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.931025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:18.080044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":248,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.862598+0000","last_change":"2024-09-21T12:23:29.134829+0000","last_active":"2024-09-21T12:25:26.862598+0000","last_peered":"2024-09-21T12:25:26.862598+0000","last_clean":"2024-09-21T12:25:26.862598+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:25:26.862598+0000","last_undegraded":"2024-09-21T12:25:26.862598+0000","last_fullsized":"2024-09-21T12:25:26.862598+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:29.134767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:29.134767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:08.422630+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":281,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.866005+0000","last_change":"2024-09-21T12:24:58.824615+0000","last_active":"2024-09-21T12:25:26.866005+0000","last_peered":"2024-09-21T12:25:26.866005+0000","last_clean":"2024-09-21T12:25:26.866005+0000","last_became_active":"2024-09-21T12:20:17.694794+0000","last_became_peered":"2024-09-21T12:20:17.694794+0000","last_unstale":"2024-09-21T12:25:26.866005+0000","last_undegraded":"2024-09-21T12:25:26.866005+0000","last_fullsized":"2024-09-21T12:25:26.866005+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:24:58.824516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:58.824516+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:24.775664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":277,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.010586+0000","last_change":"2024-09-21T12:24:33.364225+0000","last_active":"2024-09-21T12:25:30.010586+0000","last_peered":"2024-09-21T12:25:30.010586+0000","last_clean":"2024-09-21T12:25:30.010586+0000","last_became_active":"2024-09-21T12:20:15.295968+0000","last_became_peered":"2024-09-21T12:20:15.295968+0000","last_unstale":"2024-09-21T12:25:30.010586+0000","last_undegraded":"2024-09-21T12:25:30.010586+0000","last_fullsized":"2024-09-21T12:25:30.010586+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:33.363808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:33.363808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:01.574062+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":274,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.012968+0000","last_change":"2024-09-21T12:23:57.018797+0000","last_active":"2024-09-21T12:25:30.012968+0000","last_peered":"2024-09-21T12:25:30.012968+0000","last_clean":"2024-09-21T12:25:30.012968+0000","last_became_active":"2024-09-21T12:22:50.921604+0000","last_became_peered":"2024-09-21T12:22:50.921604+0000","last_unstale":"2024-09-21T12:25:30.012968+0000","last_undegraded":"2024-09-21T12:25:30.012968+0000","last_fullsized":"2024-09-21T12:25:30.012968+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:57.018749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:57.018749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":246,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.870393+0000","last_change":"2024-09-21T12:24:35.588409+0000","last_active":"2024-09-21T12:25:26.870393+0000","last_peered":"2024-09-21T12:25:26.870393+0000","last_clean":"2024-09-21T12:25:26.870393+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:25:26.870393+0000","last_undegraded":"2024-09-21T12:25:26.870393+0000","last_fullsized":"2024-09-21T12:25:26.870393+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:35.588353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:35.588353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:51.954226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":260,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.869235+0000","last_change":"2024-09-21T12:23:51.343568+0000","last_active":"2024-09-21T12:25:26.869235+0000","last_peered":"2024-09-21T12:25:26.869235+0000","last_clean":"2024-09-21T12:25:26.869235+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:25:26.869235+0000","last_undegraded":"2024-09-21T12:25:26.869235+0000","last_fullsized":"2024-09-21T12:25:26.869235+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-21T12:23:51.343491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:51.343491+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:26.153152+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":266,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.864102+0000","last_change":"2024-09-21T12:22:14.635397+0000","last_active":"2024-09-21T12:25:26.864102+0000","last_peered":"2024-09-21T12:25:26.864102+0000","last_clean":"2024-09-21T12:25:26.864102+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:25:26.864102+0000","last_undegraded":"2024-09-21T12:25:26.864102+0000","last_fullsized":"2024-09-21T12:25:26.864102+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:22:14.635191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.635191+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:43.134430+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":274,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.015334+0000","last_change":"2024-09-21T12:22:50.931202+0000","last_active":"2024-09-21T12:25:30.015334+0000","last_peered":"2024-09-21T12:25:30.015334+0000","last_clean":"2024-09-21T12:25:30.015334+0000","last_became_active":"2024-09-21T12:22:50.930615+0000","last_became_peered":"2024-09-21T12:22:50.930615+0000","last_unstale":"2024-09-21T12:25:30.015334+0000","last_undegraded":"2024-09-21T12:25:30.015334+0000","last_fullsized":"2024-09-21T12:25:30.015334+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:09.649109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:09.649109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:20.855882+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":262,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.012054+0000","last_change":"2024-09-21T12:22:50.917883+0000","last_active":"2024-09-21T12:25:30.012054+0000","last_peered":"2024-09-21T12:25:30.012054+0000","last_clean":"2024-09-21T12:25:30.012054+0000","last_became_active":"2024-09-21T12:22:50.917408+0000","last_became_peered":"2024-09-21T12:22:50.917408+0000","last_unstale":"2024-09-21T12:25:30.012054+0000","last_undegraded":"2024-09-21T12:25:30.012054+0000","last_fullsized":"2024-09-21T12:25:30.012054+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:12.917010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":263,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.019752+0000","last_change":"2024-09-21T12:22:50.914633+0000","last_active":"2024-09-21T12:25:30.019752+0000","last_peered":"2024-09-21T12:25:30.019752+0000","last_clean":"2024-09-21T12:25:30.019752+0000","last_became_active":"2024-09-21T12:22:50.914221+0000","last_became_peered":"2024-09-21T12:22:50.914221+0000","last_unstale":"2024-09-21T12:25:30.019752+0000","last_undegraded":"2024-09-21T12:25:30.019752+0000","last_fullsized":"2024-09-21T12:25:30.019752+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:13.451009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":266,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.869426+0000","last_change":"2024-09-21T12:25:00.871133+0000","last_active":"2024-09-21T12:25:26.869426+0000","last_peered":"2024-09-21T12:25:26.869426+0000","last_clean":"2024-09-21T12:25:26.869426+0000","last_became_active":"2024-09-21T12:20:15.297993+0000","last_became_peered":"2024-09-21T12:20:15.297993+0000","last_unstale":"2024-09-21T12:25:26.869426+0000","last_undegraded":"2024-09-21T12:25:26.869426+0000","last_fullsized":"2024-09-21T12:25:26.869426+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:00.871062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:00.871062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:06.894610+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":268,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-21T12:25:30.016013+0000","last_change":"2024-09-21T12:20:17.299134+0000","last_active":"2024-09-21T12:25:30.016013+0000","last_peered":"2024-09-21T12:25:30.016013+0000","last_clean":"2024-09-21T12:25:30.016013+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:25:30.016013+0000","last_undegraded":"2024-09-21T12:25:30.016013+0000","last_fullsized":"2024-09-21T12:25:30.016013+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:47.878711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":260,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-21T12:25:26.867130+0000","last_change":"2024-09-21T12:24:24.973667+0000","last_active":"2024-09-21T12:25:26.867130+0000","last_peered":"2024-09-21T12:25:26.867130+0000","last_clean":"2024-09-21T12:25:26.867130+0000","last_became_active":"2024-09-21T12:20:19.308247+0000","last_became_peered":"2024-09-21T12:20:19.308247+0000","last_unstale":"2024-09-21T12:25:26.867130+0000","last_undegraded":"2024-09-21T12:25:26.867130+0000","last_fullsized":"2024-09-21T12:25:26.867130+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:24.973604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:24.973604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":4225630208,"num_objects":64478,"num_object_clones":0,"num_object_copies":193434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64478,"num_write_kb":4126592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12845711360,"data_stored":12845711360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64478,"ondisk_log_size":64478,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":690,"num_read_kb":447,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":201,"num_bytes_recovered":3982,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":396,"ondisk_log_size":396,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":5023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":311,"num_read_kb":306,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":644038,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":21,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234051,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1736792,"kb_used_data":1583428,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":92008360,"statfs":{"total":95995035648,"available":94216560640,"internally_reserved":0,"allocated":1621430272,"data_stored":1621027129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":157022666},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462987,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1767960,"kb_used_data":1592388,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":91977192,"statfs":{"total":95995035648,"available":94184644608,"internally_reserved":0,"allocated":1630605312,"data_stored":1630201960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":179763654},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724622,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2260188,"kb_used_data":2134600,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":91484964,"statfs":{"total":95995035648,"available":93680603136,"internally_reserved":0,"allocated":2185830400,"data_stored":2185424595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953558,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149787,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1953948,"kb_used_data":1679816,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":91791204,"statfs":{"total":95995035648,"available":93994192896,"internally_reserved":0,"allocated":1720131584,"data_stored":1719709876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":280689095},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411426,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2028296,"kb_used_data":1858296,"kb_used_omap":1,"kb_used_meta":169982,"kb_avail":91716856,"statfs":{"total":95995035648,"available":93918060544,"internally_reserved":0,"allocated":1902895104,"data_stored":1902504140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":174062021},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574952,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2072376,"kb_used_data":1874600,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":91672776,"statfs":{"total":95995035648,"available":93872922624,"internally_reserved":0,"allocated":1919590400,"data_stored":1919150712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738477,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1854728,"kb_used_data":1697204,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":91890424,"statfs":{"total":95995035648,"available":94095794176,"internally_reserved":0,"allocated":1737936896,"data_stored":1737534538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1754464256,"data_stored":1754464256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1921712128,"data_stored":1921712128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1931673600,"data_stored":1931673600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1750728704,"data_stored":1750728704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2188181504,"data_stored":2188181504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1652228096,"data_stored":1652228096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1646723072,"data_stored":1646723072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:25:32.699 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:25:32.743 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:32.747 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:32.803 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:32.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:32.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:32.905 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:32.944 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:32.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:32.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:33.046 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 39 15 68741 68726 110.107 77.25 0.0186955 0.00907673 2024-09-21T12:25:33.047 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:33.078 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:33.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:33.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:33.180 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:33.210 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:33.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:33.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:33.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:33 smithi114 ceph-mon[40867]: 6.18 scrub starts 2024-09-21T12:25:33.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:33 smithi114 ceph-mon[40867]: 6.18 scrub ok 2024-09-21T12:25:33.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:33 smithi114 ceph-mon[40867]: pgmap v250: 158 pgs: 158 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.7 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-21T12:25:33.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:33 smithi114 ceph-mon[40867]: 3.10 scrub starts 2024-09-21T12:25:33.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:33 smithi114 ceph-mon[40867]: 3.10 scrub ok 2024-09-21T12:25:33.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:33 smithi114 ceph-mon[40867]: from='client.25079 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:25:33.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:33 smithi114 ceph-mon[40867]: 6.7 scrub starts 2024-09-21T12:25:33.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:33 smithi114 ceph-mon[40867]: 6.7 scrub ok 2024-09-21T12:25:33.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:33 smithi114 ceph-mon[30324]: 6.18 scrub starts 2024-09-21T12:25:33.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:33 smithi114 ceph-mon[30324]: 6.18 scrub ok 2024-09-21T12:25:33.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:33 smithi114 ceph-mon[30324]: pgmap v250: 158 pgs: 158 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.7 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-21T12:25:33.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:33 smithi114 ceph-mon[30324]: 3.10 scrub starts 2024-09-21T12:25:33.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:33 smithi114 ceph-mon[30324]: 3.10 scrub ok 2024-09-21T12:25:33.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:33 smithi114 ceph-mon[30324]: from='client.25079 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:25:33.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:33 smithi114 ceph-mon[30324]: 6.7 scrub starts 2024-09-21T12:25:33.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:33 smithi114 ceph-mon[30324]: 6.7 scrub ok 2024-09-21T12:25:33.311 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:33.346 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:33.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:33.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:33.389 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:33 smithi183 ceph-mon[33277]: 6.18 scrub starts 2024-09-21T12:25:33.390 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:33 smithi183 ceph-mon[33277]: 6.18 scrub ok 2024-09-21T12:25:33.390 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:33 smithi183 ceph-mon[33277]: pgmap v250: 158 pgs: 158 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.7 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-21T12:25:33.390 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:33 smithi183 ceph-mon[33277]: 3.10 scrub starts 2024-09-21T12:25:33.390 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:33 smithi183 ceph-mon[33277]: 3.10 scrub ok 2024-09-21T12:25:33.390 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:33 smithi183 ceph-mon[33277]: from='client.25079 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:25:33.390 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:33 smithi183 ceph-mon[33277]: 6.7 scrub starts 2024-09-21T12:25:33.390 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:33 smithi183 ceph-mon[33277]: 6.7 scrub ok 2024-09-21T12:25:33.447 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:33.474 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:33.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:33.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:33.576 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:33.606 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:33.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:33.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:33.707 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:33.742 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:33.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:33.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:33.844 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:33.882 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:33.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:33.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:33.984 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:33.985 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T12:25:34.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:34.023 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:34.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:34.047 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-21T12:25:34.046616+0000 min lat: 0.00121483 max lat: 0.158038 avg lat: 0.00913166 2024-09-21T12:25:34.047 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T12:25:34.047 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 40 15 70080 70065 109.445 83.6875 0.0206308 0.00913166 2024-09-21T12:25:34.124 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:34.152 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:34.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:34.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:34.254 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:34.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:34 smithi114 ceph-mon[30324]: 5.1e scrub starts 2024-09-21T12:25:34.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:34 smithi114 ceph-mon[30324]: 5.1e scrub ok 2024-09-21T12:25:34.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:34 smithi114 ceph-mon[30324]: 5.e scrub starts 2024-09-21T12:25:34.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:34 smithi114 ceph-mon[30324]: 5.e scrub ok 2024-09-21T12:25:34.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:34 smithi114 ceph-mon[40867]: 5.1e scrub starts 2024-09-21T12:25:34.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:34 smithi114 ceph-mon[40867]: 5.1e scrub ok 2024-09-21T12:25:34.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:34 smithi114 ceph-mon[40867]: 5.e scrub starts 2024-09-21T12:25:34.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:34 smithi114 ceph-mon[40867]: 5.e scrub ok 2024-09-21T12:25:34.278 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:34.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:34.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:34.380 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:34.423 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:34.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:34.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:34.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:34 smithi183 ceph-mon[33277]: 5.1e scrub starts 2024-09-21T12:25:34.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:34 smithi183 ceph-mon[33277]: 5.1e scrub ok 2024-09-21T12:25:34.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:34 smithi183 ceph-mon[33277]: 5.e scrub starts 2024-09-21T12:25:34.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:34 smithi183 ceph-mon[33277]: 5.e scrub ok 2024-09-21T12:25:34.525 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:34.552 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:34.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:34.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:34.654 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:34.675 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:34.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:34.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:34.778 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:34.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:34.815 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:34.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:34.916 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:34.942 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:34.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:34.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:35.044 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:35.048 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 41 16 71301 71285 108.635 76.25 0.0229776 0.00919885 2024-09-21T12:25:35.088 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:35.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:35.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:35.190 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:35.218 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:35.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:35.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:35.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:35 smithi114 ceph-mon[40867]: pgmap v252: 158 pgs: 158 active+clean; 4.1 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 752 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-09-21T12:25:35.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:35 smithi114 ceph-mon[30324]: pgmap v252: 158 pgs: 158 active+clean; 4.1 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 752 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-09-21T12:25:35.320 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:35.368 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:35.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:35.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:35.470 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:35.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:35 smithi183 ceph-mon[33277]: pgmap v252: 158 pgs: 158 active+clean; 4.1 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 752 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-09-21T12:25:35.491 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:35.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:35.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:35.592 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:35.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:35.633 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:35.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:35.665 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:25:35.666 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:25:35.666 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:25:35.666 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:25:35.666 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:25:35.666 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:25:35.666 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:25:35.666 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:25:35.684 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:35.693 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:25:35.693 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:25:35.727 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:25:35.727 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:25:35.734 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:35.759 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:35.759 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:25:35.760 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:25:35.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:35.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:35.791 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:25:35.791 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:25:35.848 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:35.862 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:35.899 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:35.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:35.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:36.001 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:36.039 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:36.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:36.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:36.047 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 42 16 72319 72303 107.564 63.625 0.0163722 0.0092921 2024-09-21T12:25:36.061 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:36 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:25:36.061 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:36 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:25:36.061 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:36 smithi114 ceph-mon[40867]: osdmap e181: 8 total, 8 up, 7 in 2024-09-21T12:25:36.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:36 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:25:36.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:36 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:25:36.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:36 smithi114 ceph-mon[30324]: osdmap e181: 8 total, 8 up, 7 in 2024-09-21T12:25:36.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:36 smithi114 ceph-mon[30324]: 4.0 scrub starts 2024-09-21T12:25:36.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:36 smithi114 ceph-mon[30324]: 4.0 scrub ok 2024-09-21T12:25:36.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:36 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3817235651' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:25:36.142 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:36.165 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:36.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:36.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:36.267 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:36.295 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:36.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:36.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:36.397 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:36.423 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:36.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:36.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:36.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:36 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:25:36.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:36 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:25:36.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:36 smithi183 ceph-mon[33277]: osdmap e181: 8 total, 8 up, 7 in 2024-09-21T12:25:36.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:36 smithi183 ceph-mon[33277]: 4.0 scrub starts 2024-09-21T12:25:36.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:36 smithi183 ceph-mon[33277]: 4.0 scrub ok 2024-09-21T12:25:36.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:36 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3817235651' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:25:36.511 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:36 smithi114 ceph-mon[40867]: 4.0 scrub starts 2024-09-21T12:25:36.511 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:36 smithi114 ceph-mon[40867]: 4.0 scrub ok 2024-09-21T12:25:36.511 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:36 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3817235651' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:25:36.525 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:36.550 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:36.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:36.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:36.652 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:36.678 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:36.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:36.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:36.780 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:36.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:36.823 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:36.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:36.924 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:36.953 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:25:36.954 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-21T12:25:36.954 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.954 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25022.0:73477 8.0 8:042b2a16:::benchmark_data_smithi114_104936_object73476:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-21T12:25:36.954 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:25:36.945466+0000", 2024-09-21T12:25:36.954 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0059658150000000002, 2024-09-21T12:25:36.954 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0059950380000000003, 2024-09-21T12:25:36.954 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:25:36.954 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:25:36.954 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:25:36.954 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:25:36.954 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25022", 2024-09-21T12:25:36.954 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:25:36.954 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 73477 2024-09-21T12:25:36.955 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.955 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:25:36.955 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.955 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:25:36.955 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.945466+0000", 2024-09-21T12:25:36.955 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:36.955 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.955 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.955 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:25:36.955 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.945466+0000", 2024-09-21T12:25:36.955 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:36.955 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.955 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.955 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:25:36.956 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.945468+0000", 2024-09-21T12:25:36.956 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3880000000000001e-06 2024-09-21T12:25:36.956 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.956 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.956 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:25:36.956 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.945489+0000", 2024-09-21T12:25:36.956 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1148000000000001e-05 2024-09-21T12:25:36.956 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.956 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.956 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:25:36.956 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.945500+0000", 2024-09-21T12:25:36.956 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1408e-05 2024-09-21T12:25:36.956 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.957 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.957 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:25:36.957 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.945507+0000", 2024-09-21T12:25:36.957 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7750000000000004e-06 2024-09-21T12:25:36.957 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.957 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.957 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:25:36.957 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.947157+0000", 2024-09-21T12:25:36.957 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0016501739999999999 2024-09-21T12:25:36.957 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.957 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.957 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:25:36.957 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.947289+0000", 2024-09-21T12:25:36.957 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013195500000000001 2024-09-21T12:25:36.957 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.958 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.958 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,5", 2024-09-21T12:25:36.958 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.947435+0000", 2024-09-21T12:25:36.958 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000146207 2024-09-21T12:25:36.958 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:36.958 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:25:36.958 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:36.958 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.958 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.958 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.25022.0:73469 8.7 e182/160 8:e125e589:::benchmark_data_smithi114_104936_object73468:head v 182'4596, mlcod=182'4593)", 2024-09-21T12:25:36.958 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:25:36.950029+0000", 2024-09-21T12:25:36.958 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0014035689999999999, 2024-09-21T12:25:36.958 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0014982109999999999, 2024-09-21T12:25:36.958 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:25:36.958 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:25:36.959 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:25:36.959 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:25:36.959 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.959 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:25:36.959 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.950029+0000", 2024-09-21T12:25:36.959 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:36.959 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.959 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.959 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:25:36.959 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.950029+0000", 2024-09-21T12:25:36.959 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:36.959 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.959 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.959 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:25:36.960 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.950030+0000", 2024-09-21T12:25:36.960 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.192e-06 2024-09-21T12:25:36.960 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.960 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.960 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:25:36.960 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.950066+0000", 2024-09-21T12:25:36.960 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.6408000000000001e-05 2024-09-21T12:25:36.960 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.960 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.960 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:25:36.960 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.950082+0000", 2024-09-21T12:25:36.960 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5849e-05 2024-09-21T12:25:36.960 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.960 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.961 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:25:36.961 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.950089+0000", 2024-09-21T12:25:36.961 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1640000000000004e-06 2024-09-21T12:25:36.961 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.961 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.961 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:25:36.961 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.950219+0000", 2024-09-21T12:25:36.961 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012992400000000001 2024-09-21T12:25:36.961 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.961 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.961 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:25:36.961 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.950392+0000", 2024-09-21T12:25:36.961 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017277500000000001 2024-09-21T12:25:36.961 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:36.961 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:25:36.962 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:36.962 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.962 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.962 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25022.0:73479 8.f 8.323fe54f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-21T12:25:36.962 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:25:36.950153+0000", 2024-09-21T12:25:36.962 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.001279278, 2024-09-21T12:25:36.962 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001419039, 2024-09-21T12:25:36.962 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:25:36.962 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:25:36.962 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "queued for pg", 2024-09-21T12:25:36.962 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:25:36.962 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25022", 2024-09-21T12:25:36.962 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:25:36.962 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 73479 2024-09-21T12:25:36.962 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.963 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:25:36.963 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.963 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:25:36.963 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.950153+0000", 2024-09-21T12:25:36.963 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:36.963 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.963 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.963 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:25:36.963 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.950153+0000", 2024-09-21T12:25:36.963 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:36.963 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.963 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.963 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:25:36.963 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.950154+0000", 2024-09-21T12:25:36.964 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3650000000000001e-06 2024-09-21T12:25:36.964 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.964 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.964 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:25:36.964 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.950176+0000", 2024-09-21T12:25:36.964 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2150999999999998e-05 2024-09-21T12:25:36.964 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.964 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.964 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:25:36.964 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.950185+0000", 2024-09-21T12:25:36.964 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6170000000000007e-06 2024-09-21T12:25:36.964 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.964 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.964 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:25:36.964 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.950191+0000", 2024-09-21T12:25:36.965 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.592e-06 2024-09-21T12:25:36.965 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:36.965 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:25:36.965 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:36.965 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.965 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.965 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.25022.0:73474 8.7 e182/160)", 2024-09-21T12:25:36.965 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:25:36.950446+0000", 2024-09-21T12:25:36.965 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.00098637000000000009, 2024-09-21T12:25:36.965 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001168786, 2024-09-21T12:25:36.965 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:25:36.965 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:25:36.965 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "queued for pg", 2024-09-21T12:25:36.965 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:25:36.966 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.966 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:25:36.966 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.950446+0000", 2024-09-21T12:25:36.966 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:36.966 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.966 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.966 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:25:36.966 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.950446+0000", 2024-09-21T12:25:36.966 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:36.966 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.966 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.966 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:25:36.966 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.950446+0000", 2024-09-21T12:25:36.966 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5499999999999998e-07 2024-09-21T12:25:36.967 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.967 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.967 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:25:36.967 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.950466+0000", 2024-09-21T12:25:36.967 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0007999999999999e-05 2024-09-21T12:25:36.967 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.967 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.967 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:25:36.967 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.950475+0000", 2024-09-21T12:25:36.967 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6969999999999999e-06 2024-09-21T12:25:36.967 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.967 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.967 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:25:36.967 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.950480+0000", 2024-09-21T12:25:36.968 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4199999999999998e-06 2024-09-21T12:25:36.968 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:36.968 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:25:36.968 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:36.968 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.968 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.968 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.25022.0:73476 8.2 e182/160 8:4c60d8d4:::benchmark_data_smithi114_104936_object73475:head v 182'4514, mlcod=181'4512)", 2024-09-21T12:25:36.968 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:25:36.950980+0000", 2024-09-21T12:25:36.968 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.000452445, 2024-09-21T12:25:36.968 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00067312000000000003, 2024-09-21T12:25:36.968 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:25:36.968 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:25:36.968 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:25:36.969 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:25:36.969 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.969 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:25:36.969 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.950980+0000", 2024-09-21T12:25:36.969 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:36.969 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.969 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.969 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:25:36.969 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.950980+0000", 2024-09-21T12:25:36.969 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:25:36.969 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.969 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.969 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:25:36.969 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.950980+0000", 2024-09-21T12:25:36.969 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8500000000000001e-07 2024-09-21T12:25:36.970 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.970 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.970 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:25:36.970 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.951004+0000", 2024-09-21T12:25:36.970 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.4077999999999999e-05 2024-09-21T12:25:36.970 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.970 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.970 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:25:36.970 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.951017+0000", 2024-09-21T12:25:36.970 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2694000000000001e-05 2024-09-21T12:25:36.970 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.970 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.970 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:25:36.970 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.951032+0000", 2024-09-21T12:25:36.971 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4582e-05 2024-09-21T12:25:36.971 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.971 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.971 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:25:36.971 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.951106+0000", 2024-09-21T12:25:36.971 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3833999999999998e-05 2024-09-21T12:25:36.971 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:25:36.971 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:25:36.971 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:25:36.971 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:25:36.951123+0000", 2024-09-21T12:25:36.971 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7348999999999999e-05 2024-09-21T12:25:36.971 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:36.971 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:25:36.971 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:36.971 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:25:36.972 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:25:36.972 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 5 2024-09-21T12:25:36.972 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:25:36.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:36.973 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:36.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:37.047 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 43 16 73609 73593 106.937 80.625 0.0174382 0.00934619 2024-09-21T12:25:37.074 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:37.105 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:37.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:37.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:37.207 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:37.235 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:37.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:37.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:37.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:25:36 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:25:37.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:37 smithi183 ceph-mon[33277]: pgmap v254: 158 pgs: 158 active+clean; 4.1 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-21T12:25:37.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:37 smithi183 ceph-mon[33277]: 2.0 scrub starts 2024-09-21T12:25:37.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:37 smithi183 ceph-mon[33277]: 2.0 scrub ok 2024-09-21T12:25:37.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:37 smithi183 ceph-mon[33277]: 5.a scrub starts 2024-09-21T12:25:37.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:37 smithi183 ceph-mon[33277]: 5.a scrub ok 2024-09-21T12:25:37.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:37 smithi183 ceph-mon[33277]: osdmap e182: 8 total, 8 up, 7 in 2024-09-21T12:25:37.337 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:37.371 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:37.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:37.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:37.473 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:37.497 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:37.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:37.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:37.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:37 smithi114 ceph-mon[40867]: pgmap v254: 158 pgs: 158 active+clean; 4.1 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-21T12:25:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:37 smithi114 ceph-mon[40867]: 2.0 scrub starts 2024-09-21T12:25:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:37 smithi114 ceph-mon[40867]: 2.0 scrub ok 2024-09-21T12:25:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:37 smithi114 ceph-mon[40867]: 5.a scrub starts 2024-09-21T12:25:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:37 smithi114 ceph-mon[40867]: 5.a scrub ok 2024-09-21T12:25:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:37 smithi114 ceph-mon[40867]: osdmap e182: 8 total, 8 up, 7 in 2024-09-21T12:25:37.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:37 smithi114 ceph-mon[30324]: pgmap v254: 158 pgs: 158 active+clean; 4.1 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-21T12:25:37.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:37 smithi114 ceph-mon[30324]: 2.0 scrub starts 2024-09-21T12:25:37.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:37 smithi114 ceph-mon[30324]: 2.0 scrub ok 2024-09-21T12:25:37.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:37 smithi114 ceph-mon[30324]: 5.a scrub starts 2024-09-21T12:25:37.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:37 smithi114 ceph-mon[30324]: 5.a scrub ok 2024-09-21T12:25:37.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:37 smithi114 ceph-mon[30324]: osdmap e182: 8 total, 8 up, 7 in 2024-09-21T12:25:37.601 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:37.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:37.659 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:37.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:37.760 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:37.800 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:37.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:37.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:37.901 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:37.929 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:37.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:37.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:38.031 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:38.047 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 44 16 74946 74930 106.406 83.5625 0.00837126 0.00939284 2024-09-21T12:25:38.057 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:38.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:38.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:38.111 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:38 smithi114 ceph-mon[40867]: 3.13 scrub starts 2024-09-21T12:25:38.111 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:38 smithi114 ceph-mon[40867]: 3.13 scrub ok 2024-09-21T12:25:38.111 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:38 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:25:38.111 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:38 smithi114 ceph-mon[30324]: 3.13 scrub starts 2024-09-21T12:25:38.111 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:38 smithi114 ceph-mon[30324]: 3.13 scrub ok 2024-09-21T12:25:38.112 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:38 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:25:38.159 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:38.186 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:38.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:38.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:38.288 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:38.328 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:38.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:38.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:38.374 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:38.429 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:38.485 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:38.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:38.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:38.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:38 smithi183 ceph-mon[33277]: 3.13 scrub starts 2024-09-21T12:25:38.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:38 smithi183 ceph-mon[33277]: 3.13 scrub ok 2024-09-21T12:25:38.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:38 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:25:38.535 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:25:38.542 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:25:38.587 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:38.625 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:38.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:38.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:38.726 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:38.769 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:38.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:38.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:38.871 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:38.898 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:38.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:38.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:39.000 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:39.027 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:39.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:39.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:39.049 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 45 16 76196 76180 105.778 78.125 0.00439536 0.00944847 2024-09-21T12:25:39.084 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T12:25:39.129 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:39.168 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:39.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:39.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:39.270 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:39.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:39 smithi114 ceph-mon[40867]: pgmap v256: 158 pgs: 158 active+clean; 4.5 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 109 MiB/s wr, 1.74k op/s 2024-09-21T12:25:39.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:39 smithi114 ceph-mon[40867]: 5.1 scrub starts 2024-09-21T12:25:39.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:39 smithi114 ceph-mon[40867]: 5.1 scrub ok 2024-09-21T12:25:39.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:39 smithi114 ceph-mon[30324]: pgmap v256: 158 pgs: 158 active+clean; 4.5 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 109 MiB/s wr, 1.74k op/s 2024-09-21T12:25:39.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:39 smithi114 ceph-mon[30324]: 5.1 scrub starts 2024-09-21T12:25:39.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:39 smithi114 ceph-mon[30324]: 5.1 scrub ok 2024-09-21T12:25:39.297 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:39.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:39.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:39.399 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:39.424 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:39.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:39.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:39.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:39 smithi183 ceph-mon[33277]: pgmap v256: 158 pgs: 158 active+clean; 4.5 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 109 MiB/s wr, 1.74k op/s 2024-09-21T12:25:39.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:39 smithi183 ceph-mon[33277]: 5.1 scrub starts 2024-09-21T12:25:39.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:39 smithi183 ceph-mon[33277]: 5.1 scrub ok 2024-09-21T12:25:39.527 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:39.570 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:39.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:39.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:39.643 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:25:39.672 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:39.684 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2024-09-21T12:25:37.663604+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4808829889,"num_objects":73566,"num_object_clones":0,"num_object_copies":220698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73566,"num_whiteouts":0,"num_read":1041,"num_read_kb":803,"num_write":73813,"num_write_kb":4695681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":223,"num_bytes_recovered":58204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70626,"ondisk_log_size":70626,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":15199432,"kb_used_data":13927428,"kb_used_omap":10,"kb_used_meta":1271861,"kb_avail":641016632,"statfs":{"total":671965249536,"available":656401031168,"internally_reserved":0,"allocated":14261686272,"data_stored":14258744878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":1302385771},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":683147264,"num_objects":10424,"num_object_clones":0,"num_object_copies":31272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10424,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":10424,"num_write_kb":667136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002516"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":276,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.875183+0000","last_change":"2024-09-21T12:24:50.115300+0000","last_active":"2024-09-21T12:25:36.875183+0000","last_peered":"2024-09-21T12:25:36.875183+0000","last_clean":"2024-09-21T12:25:36.875183+0000","last_became_active":"2024-09-21T12:20:15.309431+0000","last_became_peered":"2024-09-21T12:20:15.309431+0000","last_unstale":"2024-09-21T12:25:36.875183+0000","last_undegraded":"2024-09-21T12:25:36.875183+0000","last_fullsized":"2024-09-21T12:25:36.875183+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:50.115253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:50.115253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:01.021124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":236,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.040165+0000","last_change":"2024-09-21T12:22:51.040744+0000","last_active":"2024-09-21T12:25:35.040165+0000","last_peered":"2024-09-21T12:25:35.040165+0000","last_clean":"2024-09-21T12:25:35.040165+0000","last_became_active":"2024-09-21T12:22:50.909699+0000","last_became_peered":"2024-09-21T12:22:50.909699+0000","last_unstale":"2024-09-21T12:25:35.040165+0000","last_undegraded":"2024-09-21T12:25:35.040165+0000","last_fullsized":"2024-09-21T12:25:35.040165+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:08.888235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":266,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.039051+0000","last_change":"2024-09-21T12:24:34.311501+0000","last_active":"2024-09-21T12:25:35.039051+0000","last_peered":"2024-09-21T12:25:35.039051+0000","last_clean":"2024-09-21T12:25:35.039051+0000","last_became_active":"2024-09-21T12:20:19.314908+0000","last_became_peered":"2024-09-21T12:20:19.314908+0000","last_unstale":"2024-09-21T12:25:35.039051+0000","last_undegraded":"2024-09-21T12:25:35.039051+0000","last_fullsized":"2024-09-21T12:25:35.039051+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:34.311451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:34.311451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:44.788773+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":189,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.040966+0000","last_change":"2024-09-21T12:22:50.914064+0000","last_active":"2024-09-21T12:25:35.040966+0000","last_peered":"2024-09-21T12:25:35.040966+0000","last_clean":"2024-09-21T12:25:35.040966+0000","last_became_active":"2024-09-21T12:22:50.913743+0000","last_became_peered":"2024-09-21T12:22:50.913743+0000","last_unstale":"2024-09-21T12:25:35.040966+0000","last_undegraded":"2024-09-21T12:25:35.040966+0000","last_fullsized":"2024-09-21T12:25:35.040966+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:58.189665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:21:58.189665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:06.336105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":246,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.886129+0000","last_change":"2024-09-21T12:20:21.595607+0000","last_active":"2024-09-21T12:25:36.886129+0000","last_peered":"2024-09-21T12:25:36.886129+0000","last_clean":"2024-09-21T12:25:36.886129+0000","last_became_active":"2024-09-21T12:20:21.595301+0000","last_became_peered":"2024-09-21T12:20:21.595301+0000","last_unstale":"2024-09-21T12:25:36.886129+0000","last_undegraded":"2024-09-21T12:25:36.886129+0000","last_fullsized":"2024-09-21T12:25:36.886129+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:41.642996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":274,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.886451+0000","last_change":"2024-09-21T12:24:49.701972+0000","last_active":"2024-09-21T12:25:36.886451+0000","last_peered":"2024-09-21T12:25:36.886451+0000","last_clean":"2024-09-21T12:25:36.886451+0000","last_became_active":"2024-09-21T12:20:15.308655+0000","last_became_peered":"2024-09-21T12:20:15.308655+0000","last_unstale":"2024-09-21T12:25:36.886451+0000","last_undegraded":"2024-09-21T12:25:36.886451+0000","last_fullsized":"2024-09-21T12:25:36.886451+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:49.701915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:49.701915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:07.777041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":295,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.885661+0000","last_change":"2024-09-21T12:25:01.898461+0000","last_active":"2024-09-21T12:25:36.885661+0000","last_peered":"2024-09-21T12:25:36.885661+0000","last_clean":"2024-09-21T12:25:36.885661+0000","last_became_active":"2024-09-21T12:20:17.305220+0000","last_became_peered":"2024-09-21T12:20:17.305220+0000","last_unstale":"2024-09-21T12:25:36.885661+0000","last_undegraded":"2024-09-21T12:25:36.885661+0000","last_fullsized":"2024-09-21T12:25:36.885661+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:25:01.898376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:01.898376+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:05.029581+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":206,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.040313+0000","last_change":"2024-09-21T12:25:04.827596+0000","last_active":"2024-09-21T12:25:35.040313+0000","last_peered":"2024-09-21T12:25:35.040313+0000","last_clean":"2024-09-21T12:25:35.040313+0000","last_became_active":"2024-09-21T12:22:50.926240+0000","last_became_peered":"2024-09-21T12:22:50.926240+0000","last_unstale":"2024-09-21T12:25:35.040313+0000","last_undegraded":"2024-09-21T12:25:35.040313+0000","last_fullsized":"2024-09-21T12:25:35.040313+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:04.827529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:04.827529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:09.166075+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":286,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.042381+0000","last_change":"2024-09-21T12:22:50.912964+0000","last_active":"2024-09-21T12:25:35.042381+0000","last_peered":"2024-09-21T12:25:35.042381+0000","last_clean":"2024-09-21T12:25:35.042381+0000","last_became_active":"2024-09-21T12:22:50.912481+0000","last_became_peered":"2024-09-21T12:22:50.912481+0000","last_unstale":"2024-09-21T12:25:35.042381+0000","last_undegraded":"2024-09-21T12:25:35.042381+0000","last_fullsized":"2024-09-21T12:25:35.042381+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:43.207441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:43.207441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:17.306151+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":255,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.885755+0000","last_change":"2024-09-21T12:24:25.640253+0000","last_active":"2024-09-21T12:25:36.885755+0000","last_peered":"2024-09-21T12:25:36.885755+0000","last_clean":"2024-09-21T12:25:36.885755+0000","last_became_active":"2024-09-21T12:22:50.927804+0000","last_became_peered":"2024-09-21T12:22:50.927804+0000","last_unstale":"2024-09-21T12:25:36.885755+0000","last_undegraded":"2024-09-21T12:25:36.885755+0000","last_fullsized":"2024-09-21T12:25:36.885755+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:24:25.640169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:25.640169+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:45.636860+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":265,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.043849+0000","last_change":"2024-09-21T12:24:58.566964+0000","last_active":"2024-09-21T12:25:35.043849+0000","last_peered":"2024-09-21T12:25:35.043849+0000","last_clean":"2024-09-21T12:25:35.043849+0000","last_became_active":"2024-09-21T12:20:19.308184+0000","last_became_peered":"2024-09-21T12:20:19.308184+0000","last_unstale":"2024-09-21T12:25:35.043849+0000","last_undegraded":"2024-09-21T12:25:35.043849+0000","last_fullsized":"2024-09-21T12:25:35.043849+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:58.566902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:58.566902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:27.540756+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":264,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.872389+0000","last_change":"2024-09-21T12:25:31.616293+0000","last_active":"2024-09-21T12:25:36.872389+0000","last_peered":"2024-09-21T12:25:36.872389+0000","last_clean":"2024-09-21T12:25:36.872389+0000","last_became_active":"2024-09-21T12:20:21.309190+0000","last_became_peered":"2024-09-21T12:20:21.309190+0000","last_unstale":"2024-09-21T12:25:36.872389+0000","last_undegraded":"2024-09-21T12:25:36.872389+0000","last_fullsized":"2024-09-21T12:25:36.872389+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:31.616219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:31.616219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:48.224028+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":286,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.041146+0000","last_change":"2024-09-21T12:22:54.174099+0000","last_active":"2024-09-21T12:25:35.041146+0000","last_peered":"2024-09-21T12:25:35.041146+0000","last_clean":"2024-09-21T12:25:35.041146+0000","last_became_active":"2024-09-21T12:22:50.911737+0000","last_became_peered":"2024-09-21T12:22:50.911737+0000","last_unstale":"2024-09-21T12:25:35.041146+0000","last_undegraded":"2024-09-21T12:25:35.041146+0000","last_fullsized":"2024-09-21T12:25:35.041146+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:54.174034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:54.174034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:23.349015+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":241,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.886366+0000","last_change":"2024-09-21T12:24:28.624430+0000","last_active":"2024-09-21T12:25:36.886366+0000","last_peered":"2024-09-21T12:25:36.886366+0000","last_clean":"2024-09-21T12:25:36.886366+0000","last_became_active":"2024-09-21T12:22:50.918794+0000","last_became_peered":"2024-09-21T12:22:50.918794+0000","last_unstale":"2024-09-21T12:25:36.886366+0000","last_undegraded":"2024-09-21T12:25:36.886366+0000","last_fullsized":"2024-09-21T12:25:36.886366+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-21T12:24:28.624364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:28.624364+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:45.839090+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":278,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.042887+0000","last_change":"2024-09-21T12:25:01.897596+0000","last_active":"2024-09-21T12:25:35.042887+0000","last_peered":"2024-09-21T12:25:35.042887+0000","last_clean":"2024-09-21T12:25:35.042887+0000","last_became_active":"2024-09-21T12:22:50.921780+0000","last_became_peered":"2024-09-21T12:22:50.921780+0000","last_unstale":"2024-09-21T12:25:35.042887+0000","last_undegraded":"2024-09-21T12:25:35.042887+0000","last_fullsized":"2024-09-21T12:25:35.042887+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:01.897527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:01.897527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:17.629145+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":269,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.040256+0000","last_change":"2024-09-21T12:24:34.367580+0000","last_active":"2024-09-21T12:25:35.040256+0000","last_peered":"2024-09-21T12:25:35.040256+0000","last_clean":"2024-09-21T12:25:35.040256+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:25:35.040256+0000","last_undegraded":"2024-09-21T12:25:35.040256+0000","last_fullsized":"2024-09-21T12:25:35.040256+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:34.367532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:34.367532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:51.733042+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":205,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.041754+0000","last_change":"2024-09-21T12:25:30.888535+0000","last_active":"2024-09-21T12:25:35.041754+0000","last_peered":"2024-09-21T12:25:35.041754+0000","last_clean":"2024-09-21T12:25:35.041754+0000","last_became_active":"2024-09-21T12:22:50.914792+0000","last_became_peered":"2024-09-21T12:22:50.914792+0000","last_unstale":"2024-09-21T12:25:35.041754+0000","last_undegraded":"2024-09-21T12:25:35.041754+0000","last_fullsized":"2024-09-21T12:25:35.041754+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:30.888491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:30.888491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:32.893945+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":305,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.874702+0000","last_change":"2024-09-21T12:24:30.033946+0000","last_active":"2024-09-21T12:25:36.874702+0000","last_peered":"2024-09-21T12:25:36.874702+0000","last_clean":"2024-09-21T12:25:36.874702+0000","last_became_active":"2024-09-21T12:22:50.921576+0000","last_became_peered":"2024-09-21T12:22:50.921576+0000","last_unstale":"2024-09-21T12:25:36.874702+0000","last_undegraded":"2024-09-21T12:25:36.874702+0000","last_fullsized":"2024-09-21T12:25:36.874702+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:30.033874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:30.033874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:53.842679+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":295,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.056103+0000","last_change":"2024-09-21T12:24:50.834579+0000","last_active":"2024-09-21T12:25:35.056103+0000","last_peered":"2024-09-21T12:25:35.056103+0000","last_clean":"2024-09-21T12:25:35.056103+0000","last_became_active":"2024-09-21T12:20:17.307828+0000","last_became_peered":"2024-09-21T12:20:17.307828+0000","last_unstale":"2024-09-21T12:25:35.056103+0000","last_undegraded":"2024-09-21T12:25:35.056103+0000","last_fullsized":"2024-09-21T12:25:35.056103+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:24:50.834503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:50.834503+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:20.530206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":287,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.037688+0000","last_change":"2024-09-21T12:23:48.435833+0000","last_active":"2024-09-21T12:25:35.037688+0000","last_peered":"2024-09-21T12:25:35.037688+0000","last_clean":"2024-09-21T12:25:35.037688+0000","last_became_active":"2024-09-21T12:22:50.928972+0000","last_became_peered":"2024-09-21T12:22:50.928972+0000","last_unstale":"2024-09-21T12:25:35.037688+0000","last_undegraded":"2024-09-21T12:25:35.037688+0000","last_fullsized":"2024-09-21T12:25:35.037688+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:48.435769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:48.435769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:19.426681+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":254,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.887054+0000","last_change":"2024-09-21T12:23:58.354628+0000","last_active":"2024-09-21T12:25:36.887054+0000","last_peered":"2024-09-21T12:25:36.887054+0000","last_clean":"2024-09-21T12:25:36.887054+0000","last_became_active":"2024-09-21T12:20:21.592796+0000","last_became_peered":"2024-09-21T12:20:21.592796+0000","last_unstale":"2024-09-21T12:25:36.887054+0000","last_undegraded":"2024-09-21T12:25:36.887054+0000","last_fullsized":"2024-09-21T12:25:36.887054+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:58.354570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:58.354570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:21.152937+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":205,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.038926+0000","last_change":"2024-09-21T12:24:55.447829+0000","last_active":"2024-09-21T12:25:35.038926+0000","last_peered":"2024-09-21T12:25:35.038926+0000","last_clean":"2024-09-21T12:25:35.038926+0000","last_became_active":"2024-09-21T12:22:50.925403+0000","last_became_peered":"2024-09-21T12:22:50.925403+0000","last_unstale":"2024-09-21T12:25:35.038926+0000","last_undegraded":"2024-09-21T12:25:35.038926+0000","last_fullsized":"2024-09-21T12:25:35.038926+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:55.447755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:55.447755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:20.748653+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":350,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.885820+0000","last_change":"2024-09-21T12:25:31.270272+0000","last_active":"2024-09-21T12:25:36.885820+0000","last_peered":"2024-09-21T12:25:36.885820+0000","last_clean":"2024-09-21T12:25:36.885820+0000","last_became_active":"2024-09-21T12:22:50.912144+0000","last_became_peered":"2024-09-21T12:22:50.912144+0000","last_unstale":"2024-09-21T12:25:36.885820+0000","last_undegraded":"2024-09-21T12:25:36.885820+0000","last_fullsized":"2024-09-21T12:25:36.885820+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:25:31.270165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:31.270165+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:48.492514+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":184,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.888999+0000","last_change":"2024-09-21T12:22:50.920178+0000","last_active":"2024-09-21T12:25:36.888999+0000","last_peered":"2024-09-21T12:25:36.888999+0000","last_clean":"2024-09-21T12:25:36.888999+0000","last_became_active":"2024-09-21T12:22:50.918326+0000","last_became_peered":"2024-09-21T12:22:50.918326+0000","last_unstale":"2024-09-21T12:25:36.888999+0000","last_undegraded":"2024-09-21T12:25:36.888999+0000","last_fullsized":"2024-09-21T12:25:36.888999+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:42.672999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":254,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.043451+0000","last_change":"2024-09-21T12:23:51.065780+0000","last_active":"2024-09-21T12:25:35.043451+0000","last_peered":"2024-09-21T12:25:35.043451+0000","last_clean":"2024-09-21T12:25:35.043451+0000","last_became_active":"2024-09-21T12:20:21.312976+0000","last_became_peered":"2024-09-21T12:20:21.312976+0000","last_unstale":"2024-09-21T12:25:35.043451+0000","last_undegraded":"2024-09-21T12:25:35.043451+0000","last_fullsized":"2024-09-21T12:25:35.043451+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:23:51.065733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:51.065733+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:04.884955+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":300,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.041781+0000","last_change":"2024-09-21T12:22:52.070589+0000","last_active":"2024-09-21T12:25:35.041781+0000","last_peered":"2024-09-21T12:25:35.041781+0000","last_clean":"2024-09-21T12:25:35.041781+0000","last_became_active":"2024-09-21T12:22:50.910933+0000","last_became_peered":"2024-09-21T12:22:50.910933+0000","last_unstale":"2024-09-21T12:25:35.041781+0000","last_undegraded":"2024-09-21T12:25:35.041781+0000","last_fullsized":"2024-09-21T12:25:35.041781+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:22:52.070513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:52.070513+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:16.453567+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":279,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.875491+0000","last_change":"2024-09-21T12:24:03.059782+0000","last_active":"2024-09-21T12:25:36.875491+0000","last_peered":"2024-09-21T12:25:36.875491+0000","last_clean":"2024-09-21T12:25:36.875491+0000","last_became_active":"2024-09-21T12:20:17.695977+0000","last_became_peered":"2024-09-21T12:20:17.695977+0000","last_unstale":"2024-09-21T12:25:36.875491+0000","last_undegraded":"2024-09-21T12:25:36.875491+0000","last_fullsized":"2024-09-21T12:25:36.875491+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:24:03.059549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:03.059549+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:25.640560+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"121'11","reported_seq":374,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.041935+0000","last_change":"2024-09-21T12:24:50.925521+0000","last_active":"2024-09-21T12:25:35.041935+0000","last_peered":"2024-09-21T12:25:35.041935+0000","last_clean":"2024-09-21T12:25:35.041935+0000","last_became_active":"2024-09-21T12:22:50.916683+0000","last_became_peered":"2024-09-21T12:22:50.916683+0000","last_unstale":"2024-09-21T12:25:35.041935+0000","last_undegraded":"2024-09-21T12:25:35.041935+0000","last_fullsized":"2024-09-21T12:25:35.041935+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:24:50.925458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:50.925458+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:07.929632+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"181'4439","reported_seq":4492,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.754064+0000","last_change":"2024-09-21T12:24:46.316817+0000","last_active":"2024-09-21T12:25:36.754064+0000","last_peered":"2024-09-21T12:25:36.754064+0000","last_clean":"2024-09-21T12:25:36.754064+0000","last_became_active":"2024-09-21T12:24:46.316132+0000","last_became_peered":"2024-09-21T12:24:46.316132+0000","last_unstale":"2024-09-21T12:25:36.754064+0000","last_undegraded":"2024-09-21T12:25:36.754064+0000","last_fullsized":"2024-09-21T12:25:36.754064+0000","mapping_epoch":160,"log_start":"165'200","ondisk_log_start":"165'200","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4239,"log_dups_size":0,"ondisk_log_size":4239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:55.574645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290914304,"num_objects":4439,"num_object_clones":0,"num_object_copies":13317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4439,"num_write_kb":284096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"76'79","reported_seq":166,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.040826+0000","last_change":"2024-09-21T12:22:18.000752+0000","last_active":"2024-09-21T12:25:35.040826+0000","last_peered":"2024-09-21T12:25:35.040826+0000","last_clean":"2024-09-21T12:25:35.040826+0000","last_became_active":"2024-09-21T12:22:17.074721+0000","last_became_peered":"2024-09-21T12:22:17.074721+0000","last_unstale":"2024-09-21T12:25:35.040826+0000","last_undegraded":"2024-09-21T12:25:35.040826+0000","last_fullsized":"2024-09-21T12:25:35.040826+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:22:18.000705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:22:18.000705+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:41.349901+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":124,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.056290+0000","last_change":"2024-09-21T12:24:59.803037+0000","last_active":"2024-09-21T12:25:35.056290+0000","last_peered":"2024-09-21T12:25:35.056290+0000","last_clean":"2024-09-21T12:25:35.056290+0000","last_became_active":"2024-09-21T12:23:46.910112+0000","last_became_peered":"2024-09-21T12:23:46.910112+0000","last_unstale":"2024-09-21T12:25:35.056290+0000","last_undegraded":"2024-09-21T12:25:35.056290+0000","last_fullsized":"2024-09-21T12:25:35.056290+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:24:59.802945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:59.802945+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:22.089988+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":266,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.890625+0000","last_change":"2024-09-21T12:22:14.036290+0000","last_active":"2024-09-21T12:25:36.890625+0000","last_peered":"2024-09-21T12:25:36.890625+0000","last_clean":"2024-09-21T12:25:36.890625+0000","last_became_active":"2024-09-21T12:20:15.295774+0000","last_became_peered":"2024-09-21T12:20:15.295774+0000","last_unstale":"2024-09-21T12:25:36.890625+0000","last_undegraded":"2024-09-21T12:25:36.890625+0000","last_fullsized":"2024-09-21T12:25:36.890625+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:14.036236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.036236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:45.694273+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"158'13","reported_seq":85,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.040499+0000","last_change":"2024-09-21T12:25:33.474458+0000","last_active":"2024-09-21T12:25:35.040499+0000","last_peered":"2024-09-21T12:25:35.040499+0000","last_clean":"2024-09-21T12:25:35.040499+0000","last_became_active":"2024-09-21T12:24:39.272835+0000","last_became_peered":"2024-09-21T12:24:39.272835+0000","last_unstale":"2024-09-21T12:25:35.040499+0000","last_undegraded":"2024-09-21T12:25:35.040499+0000","last_fullsized":"2024-09-21T12:25:35.040499+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'13","last_scrub_stamp":"2024-09-21T12:25:33.474391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:33.474391+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:01.060227+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":254,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.040921+0000","last_change":"2024-09-21T12:22:12.145637+0000","last_active":"2024-09-21T12:25:35.040921+0000","last_peered":"2024-09-21T12:25:35.040921+0000","last_clean":"2024-09-21T12:25:35.040921+0000","last_became_active":"2024-09-21T12:20:21.310022+0000","last_became_peered":"2024-09-21T12:20:21.310022+0000","last_unstale":"2024-09-21T12:25:35.040921+0000","last_undegraded":"2024-09-21T12:25:35.040921+0000","last_fullsized":"2024-09-21T12:25:35.040921+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:22:12.145550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:12.145550+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:46.398927+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"182'4611","reported_seq":4666,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:37.093813+0000","last_change":"2024-09-21T12:24:46.326848+0000","last_active":"2024-09-21T12:25:37.093813+0000","last_peered":"2024-09-21T12:25:37.093813+0000","last_clean":"2024-09-21T12:25:37.093813+0000","last_became_active":"2024-09-21T12:24:46.326586+0000","last_became_peered":"2024-09-21T12:24:46.326586+0000","last_unstale":"2024-09-21T12:25:37.093813+0000","last_undegraded":"2024-09-21T12:25:37.093813+0000","last_fullsized":"2024-09-21T12:25:37.093813+0000","mapping_epoch":160,"log_start":"165'300","ondisk_log_start":"165'300","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4311,"log_dups_size":0,"ondisk_log_size":4311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:11.463769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302186496,"num_objects":4611,"num_object_clones":0,"num_object_copies":13833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4611,"num_write_kb":295104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"76'79","reported_seq":155,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.056055+0000","last_change":"2024-09-21T12:22:50.916239+0000","last_active":"2024-09-21T12:25:35.056055+0000","last_peered":"2024-09-21T12:25:35.056055+0000","last_clean":"2024-09-21T12:25:35.056055+0000","last_became_active":"2024-09-21T12:22:50.914885+0000","last_became_peered":"2024-09-21T12:22:50.914885+0000","last_unstale":"2024-09-21T12:25:35.056055+0000","last_undegraded":"2024-09-21T12:25:35.056055+0000","last_fullsized":"2024-09-21T12:25:35.056055+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:22:17.824342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:22:17.824342+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:19.884884+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":302,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.890443+0000","last_change":"2024-09-21T12:25:35.325593+0000","last_active":"2024-09-21T12:25:36.890443+0000","last_peered":"2024-09-21T12:25:36.890443+0000","last_clean":"2024-09-21T12:25:36.890443+0000","last_became_active":"2024-09-21T12:20:17.693179+0000","last_became_peered":"2024-09-21T12:20:17.693179+0000","last_unstale":"2024-09-21T12:25:36.890443+0000","last_undegraded":"2024-09-21T12:25:36.890443+0000","last_fullsized":"2024-09-21T12:25:36.890443+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-21T12:25:35.325493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:35.325493+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:38.650408+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":266,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.887513+0000","last_change":"2024-09-21T12:21:18.056305+0000","last_active":"2024-09-21T12:25:36.887513+0000","last_peered":"2024-09-21T12:25:36.887513+0000","last_clean":"2024-09-21T12:25:36.887513+0000","last_became_active":"2024-09-21T12:20:15.297340+0000","last_became_peered":"2024-09-21T12:20:15.297340+0000","last_unstale":"2024-09-21T12:25:36.887513+0000","last_undegraded":"2024-09-21T12:25:36.887513+0000","last_fullsized":"2024-09-21T12:25:36.887513+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:18.056261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:18.056261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:06.946985+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":200,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.885279+0000","last_change":"2024-09-21T12:24:24.661109+0000","last_active":"2024-09-21T12:25:36.885279+0000","last_peered":"2024-09-21T12:25:36.885279+0000","last_clean":"2024-09-21T12:25:36.885279+0000","last_became_active":"2024-09-21T12:22:50.930858+0000","last_became_peered":"2024-09-21T12:22:50.930858+0000","last_unstale":"2024-09-21T12:25:36.885279+0000","last_undegraded":"2024-09-21T12:25:36.885279+0000","last_fullsized":"2024-09-21T12:25:36.885279+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:24.661038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:24.661038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":199,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.885661+0000","last_change":"2024-09-21T12:24:29.635826+0000","last_active":"2024-09-21T12:25:36.885661+0000","last_peered":"2024-09-21T12:25:36.885661+0000","last_clean":"2024-09-21T12:25:36.885661+0000","last_became_active":"2024-09-21T12:22:50.918940+0000","last_became_peered":"2024-09-21T12:22:50.918940+0000","last_unstale":"2024-09-21T12:25:36.885661+0000","last_undegraded":"2024-09-21T12:25:36.885661+0000","last_fullsized":"2024-09-21T12:25:36.885661+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:29.635514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:29.635514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:47.655178+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"181'4545","reported_seq":4598,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.603105+0000","last_change":"2024-09-21T12:24:46.324341+0000","last_active":"2024-09-21T12:25:36.603105+0000","last_peered":"2024-09-21T12:25:36.603105+0000","last_clean":"2024-09-21T12:25:36.603105+0000","last_became_active":"2024-09-21T12:24:46.323986+0000","last_became_peered":"2024-09-21T12:24:46.323986+0000","last_unstale":"2024-09-21T12:25:36.603105+0000","last_undegraded":"2024-09-21T12:25:36.603105+0000","last_fullsized":"2024-09-21T12:25:36.603105+0000","mapping_epoch":160,"log_start":"165'100","ondisk_log_start":"165'100","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4445,"log_dups_size":0,"ondisk_log_size":4445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:13.943600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297861120,"num_objects":4545,"num_object_clones":0,"num_object_copies":13635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4545,"num_write_kb":290880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"76'79","reported_seq":181,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.888506+0000","last_change":"2024-09-21T12:25:02.911776+0000","last_active":"2024-09-21T12:25:36.888506+0000","last_peered":"2024-09-21T12:25:36.888506+0000","last_clean":"2024-09-21T12:25:36.888506+0000","last_became_active":"2024-09-21T12:22:17.069042+0000","last_became_peered":"2024-09-21T12:22:17.069042+0000","last_unstale":"2024-09-21T12:25:36.888506+0000","last_undegraded":"2024-09-21T12:25:36.888506+0000","last_fullsized":"2024-09-21T12:25:36.888506+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:25:02.911705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:25:02.911705+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:06.798137+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":262,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.040336+0000","last_change":"2024-09-21T12:22:50.978102+0000","last_active":"2024-09-21T12:25:35.040336+0000","last_peered":"2024-09-21T12:25:35.040336+0000","last_clean":"2024-09-21T12:25:35.040336+0000","last_became_active":"2024-09-21T12:22:50.926008+0000","last_became_peered":"2024-09-21T12:22:50.926008+0000","last_unstale":"2024-09-21T12:25:35.040336+0000","last_undegraded":"2024-09-21T12:25:35.040336+0000","last_fullsized":"2024-09-21T12:25:35.040336+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-21T12:22:14.062464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.062464+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:42.320597+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":302,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.875760+0000","last_change":"2024-09-21T12:22:51.003768+0000","last_active":"2024-09-21T12:25:36.875760+0000","last_peered":"2024-09-21T12:25:36.875760+0000","last_clean":"2024-09-21T12:25:36.875760+0000","last_became_active":"2024-09-21T12:22:50.923284+0000","last_became_peered":"2024-09-21T12:22:50.923284+0000","last_unstale":"2024-09-21T12:25:36.875760+0000","last_undegraded":"2024-09-21T12:25:36.875760+0000","last_fullsized":"2024-09-21T12:25:36.875760+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:22:07.582969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:07.582969+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:47.120857+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"121'11","reported_seq":364,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.890287+0000","last_change":"2024-09-21T12:23:27.244152+0000","last_active":"2024-09-21T12:25:36.890287+0000","last_peered":"2024-09-21T12:25:36.890287+0000","last_clean":"2024-09-21T12:25:36.890287+0000","last_became_active":"2024-09-21T12:22:50.930342+0000","last_became_peered":"2024-09-21T12:22:50.930342+0000","last_unstale":"2024-09-21T12:25:36.890287+0000","last_undegraded":"2024-09-21T12:25:36.890287+0000","last_fullsized":"2024-09-21T12:25:36.890287+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:23:27.243903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:27.243903+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:08.949234+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":253,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.043657+0000","last_change":"2024-09-21T12:21:21.835669+0000","last_active":"2024-09-21T12:25:35.043657+0000","last_peered":"2024-09-21T12:25:35.043657+0000","last_clean":"2024-09-21T12:25:35.043657+0000","last_became_active":"2024-09-21T12:20:21.593367+0000","last_became_peered":"2024-09-21T12:20:21.593367+0000","last_unstale":"2024-09-21T12:25:35.043657+0000","last_undegraded":"2024-09-21T12:25:35.043657+0000","last_fullsized":"2024-09-21T12:25:35.043657+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:21.835623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:21:21.835623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:29.766408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"182'4577","reported_seq":4632,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:37.096219+0000","last_change":"2024-09-21T12:24:46.325776+0000","last_active":"2024-09-21T12:25:37.096219+0000","last_peered":"2024-09-21T12:25:37.096219+0000","last_clean":"2024-09-21T12:25:37.096219+0000","last_became_active":"2024-09-21T12:24:46.325260+0000","last_became_peered":"2024-09-21T12:24:46.325260+0000","last_unstale":"2024-09-21T12:25:37.096219+0000","last_undegraded":"2024-09-21T12:25:37.096219+0000","last_fullsized":"2024-09-21T12:25:37.096219+0000","mapping_epoch":160,"log_start":"165'300","ondisk_log_start":"165'300","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4277,"log_dups_size":0,"ondisk_log_size":4277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:57.283236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299958272,"num_objects":4577,"num_object_clones":0,"num_object_copies":13731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4577,"num_write_kb":292928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"76'79","reported_seq":264,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.041760+0000","last_change":"2024-09-21T12:24:51.918097+0000","last_active":"2024-09-21T12:25:35.041760+0000","last_peered":"2024-09-21T12:25:35.041760+0000","last_clean":"2024-09-21T12:25:35.041760+0000","last_became_active":"2024-09-21T12:22:17.093298+0000","last_became_peered":"2024-09-21T12:22:17.093298+0000","last_unstale":"2024-09-21T12:25:35.041760+0000","last_undegraded":"2024-09-21T12:25:35.041760+0000","last_fullsized":"2024-09-21T12:25:35.041760+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:24:51.918028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:24:51.918028+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:11.471338+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":334,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.037827+0000","last_change":"2024-09-21T12:24:01.388987+0000","last_active":"2024-09-21T12:25:35.037827+0000","last_peered":"2024-09-21T12:25:35.037827+0000","last_clean":"2024-09-21T12:25:35.037827+0000","last_became_active":"2024-09-21T12:22:50.925840+0000","last_became_peered":"2024-09-21T12:22:50.925840+0000","last_unstale":"2024-09-21T12:25:35.037827+0000","last_undegraded":"2024-09-21T12:25:35.037827+0000","last_fullsized":"2024-09-21T12:25:35.037827+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:24:01.388905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:01.388905+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:20.477756+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":265,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.042510+0000","last_change":"2024-09-21T12:22:37.176483+0000","last_active":"2024-09-21T12:25:35.042510+0000","last_peered":"2024-09-21T12:25:35.042510+0000","last_clean":"2024-09-21T12:25:35.042510+0000","last_became_active":"2024-09-21T12:20:15.308518+0000","last_became_peered":"2024-09-21T12:20:15.308518+0000","last_unstale":"2024-09-21T12:25:35.042510+0000","last_undegraded":"2024-09-21T12:25:35.042510+0000","last_fullsized":"2024-09-21T12:25:35.042510+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:37.176413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:37.176413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:12.301692+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":336,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.874280+0000","last_change":"2024-09-21T12:21:23.139205+0000","last_active":"2024-09-21T12:25:36.874280+0000","last_peered":"2024-09-21T12:25:36.874280+0000","last_clean":"2024-09-21T12:25:36.874280+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:25:36.874280+0000","last_undegraded":"2024-09-21T12:25:36.874280+0000","last_fullsized":"2024-09-21T12:25:36.874280+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:21:23.139124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.139124+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:41.119005+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":256,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.872571+0000","last_change":"2024-09-21T12:22:51.987032+0000","last_active":"2024-09-21T12:25:36.872571+0000","last_peered":"2024-09-21T12:25:36.872571+0000","last_clean":"2024-09-21T12:25:36.872571+0000","last_became_active":"2024-09-21T12:20:21.315801+0000","last_became_peered":"2024-09-21T12:20:21.315801+0000","last_unstale":"2024-09-21T12:25:36.872571+0000","last_undegraded":"2024-09-21T12:25:36.872571+0000","last_fullsized":"2024-09-21T12:25:36.872571+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:51.986953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:51.986953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:21.623214+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"182'4635","reported_seq":4688,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:37.050946+0000","last_change":"2024-09-21T12:24:46.324384+0000","last_active":"2024-09-21T12:25:37.050946+0000","last_peered":"2024-09-21T12:25:37.050946+0000","last_clean":"2024-09-21T12:25:37.050946+0000","last_became_active":"2024-09-21T12:24:46.323283+0000","last_became_peered":"2024-09-21T12:24:46.323283+0000","last_unstale":"2024-09-21T12:25:37.050946+0000","last_undegraded":"2024-09-21T12:25:37.050946+0000","last_fullsized":"2024-09-21T12:25:37.050946+0000","mapping_epoch":160,"log_start":"165'400","ondisk_log_start":"165'400","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4235,"log_dups_size":0,"ondisk_log_size":4235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:58.096500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303759360,"num_objects":4635,"num_object_clones":0,"num_object_copies":13905,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4635,"num_write_kb":296640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"76'79","reported_seq":264,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.041612+0000","last_change":"2024-09-21T12:24:53.927007+0000","last_active":"2024-09-21T12:25:35.041612+0000","last_peered":"2024-09-21T12:25:35.041612+0000","last_clean":"2024-09-21T12:25:35.041612+0000","last_became_active":"2024-09-21T12:22:17.083629+0000","last_became_peered":"2024-09-21T12:22:17.083629+0000","last_unstale":"2024-09-21T12:25:35.041612+0000","last_undegraded":"2024-09-21T12:25:35.041612+0000","last_fullsized":"2024-09-21T12:25:35.041612+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:24:53.926958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:24:53.926958+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:21.193665+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":300,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.874457+0000","last_change":"2024-09-21T12:23:22.124368+0000","last_active":"2024-09-21T12:25:36.874457+0000","last_peered":"2024-09-21T12:25:36.874457+0000","last_clean":"2024-09-21T12:25:36.874457+0000","last_became_active":"2024-09-21T12:20:17.693067+0000","last_became_peered":"2024-09-21T12:20:17.693067+0000","last_unstale":"2024-09-21T12:25:36.874457+0000","last_undegraded":"2024-09-21T12:25:36.874457+0000","last_fullsized":"2024-09-21T12:25:36.874457+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-21T12:23:22.124283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:22.124283+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:07.353016+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":258,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.037978+0000","last_change":"2024-09-21T12:20:15.308741+0000","last_active":"2024-09-21T12:25:35.037978+0000","last_peered":"2024-09-21T12:25:35.037978+0000","last_clean":"2024-09-21T12:25:35.037978+0000","last_became_active":"2024-09-21T12:20:15.308441+0000","last_became_peered":"2024-09-21T12:20:15.308441+0000","last_unstale":"2024-09-21T12:25:35.037978+0000","last_undegraded":"2024-09-21T12:25:35.037978+0000","last_fullsized":"2024-09-21T12:25:35.037978+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:20:14.277394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:10.468822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":257,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.041572+0000","last_change":"2024-09-21T12:22:34.563365+0000","last_active":"2024-09-21T12:25:35.041572+0000","last_peered":"2024-09-21T12:25:35.041572+0000","last_clean":"2024-09-21T12:25:35.041572+0000","last_became_active":"2024-09-21T12:20:19.309598+0000","last_became_peered":"2024-09-21T12:20:19.309598+0000","last_unstale":"2024-09-21T12:25:35.041572+0000","last_undegraded":"2024-09-21T12:25:35.041572+0000","last_fullsized":"2024-09-21T12:25:35.041572+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:34.563282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:34.563282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:10.919249+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":246,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.038962+0000","last_change":"2024-09-21T12:20:21.593173+0000","last_active":"2024-09-21T12:25:35.038962+0000","last_peered":"2024-09-21T12:25:35.038962+0000","last_clean":"2024-09-21T12:25:35.038962+0000","last_became_active":"2024-09-21T12:20:21.592828+0000","last_became_peered":"2024-09-21T12:20:21.592828+0000","last_unstale":"2024-09-21T12:25:35.038962+0000","last_undegraded":"2024-09-21T12:25:35.038962+0000","last_fullsized":"2024-09-21T12:25:35.038962+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:43.315722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"181'4579","reported_seq":4632,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.598031+0000","last_change":"2024-09-21T12:24:46.325092+0000","last_active":"2024-09-21T12:25:36.598031+0000","last_peered":"2024-09-21T12:25:36.598031+0000","last_clean":"2024-09-21T12:25:36.598031+0000","last_became_active":"2024-09-21T12:24:46.324740+0000","last_became_peered":"2024-09-21T12:24:46.324740+0000","last_unstale":"2024-09-21T12:25:36.598031+0000","last_undegraded":"2024-09-21T12:25:36.598031+0000","last_fullsized":"2024-09-21T12:25:36.598031+0000","mapping_epoch":160,"log_start":"165'100","ondisk_log_start":"165'100","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4479,"log_dups_size":0,"ondisk_log_size":4479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:14.438068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300089344,"num_objects":4579,"num_object_clones":0,"num_object_copies":13737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4579,"num_write_kb":293056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"76'79","reported_seq":166,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.040035+0000","last_change":"2024-09-21T12:23:21.346117+0000","last_active":"2024-09-21T12:25:35.040035+0000","last_peered":"2024-09-21T12:25:35.040035+0000","last_clean":"2024-09-21T12:25:35.040035+0000","last_became_active":"2024-09-21T12:22:17.084732+0000","last_became_peered":"2024-09-21T12:22:17.084732+0000","last_unstale":"2024-09-21T12:25:35.040035+0000","last_undegraded":"2024-09-21T12:25:35.040035+0000","last_fullsized":"2024-09-21T12:25:35.040035+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:23:21.346046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:23:21.346046+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:44.364115+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":367,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.037894+0000","last_change":"2024-09-21T12:20:17.308049+0000","last_active":"2024-09-21T12:25:35.037894+0000","last_peered":"2024-09-21T12:25:35.037894+0000","last_clean":"2024-09-21T12:25:35.037894+0000","last_became_active":"2024-09-21T12:20:17.307769+0000","last_became_peered":"2024-09-21T12:20:17.307769+0000","last_unstale":"2024-09-21T12:25:35.037894+0000","last_undegraded":"2024-09-21T12:25:35.037894+0000","last_fullsized":"2024-09-21T12:25:35.037894+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:07.537534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":191,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.874634+0000","last_change":"2024-09-21T12:22:50.922961+0000","last_active":"2024-09-21T12:25:36.874634+0000","last_peered":"2024-09-21T12:25:36.874634+0000","last_clean":"2024-09-21T12:25:36.874634+0000","last_became_active":"2024-09-21T12:22:50.922002+0000","last_became_peered":"2024-09-21T12:22:50.922002+0000","last_unstale":"2024-09-21T12:25:36.874634+0000","last_undegraded":"2024-09-21T12:25:36.874634+0000","last_fullsized":"2024-09-21T12:25:36.874634+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:05.507899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:05.507899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:45.117108+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":275,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.043424+0000","last_change":"2024-09-21T12:21:20.045824+0000","last_active":"2024-09-21T12:25:35.043424+0000","last_peered":"2024-09-21T12:25:35.043424+0000","last_clean":"2024-09-21T12:25:35.043424+0000","last_became_active":"2024-09-21T12:20:14.287998+0000","last_became_peered":"2024-09-21T12:20:14.287998+0000","last_unstale":"2024-09-21T12:25:35.043424+0000","last_undegraded":"2024-09-21T12:25:35.043424+0000","last_fullsized":"2024-09-21T12:25:35.043424+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T12:21:20.045755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.045755+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:05.367564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011011149999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":289,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.875836+0000","last_change":"2024-09-21T12:24:53.112915+0000","last_active":"2024-09-21T12:25:36.875836+0000","last_peered":"2024-09-21T12:25:36.875836+0000","last_clean":"2024-09-21T12:25:36.875836+0000","last_became_active":"2024-09-21T12:22:50.921488+0000","last_became_peered":"2024-09-21T12:22:50.921488+0000","last_unstale":"2024-09-21T12:25:36.875836+0000","last_undegraded":"2024-09-21T12:25:36.875836+0000","last_fullsized":"2024-09-21T12:25:36.875836+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:53.112868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:53.112868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:02.884141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":275,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.886222+0000","last_change":"2024-09-21T12:23:20.237788+0000","last_active":"2024-09-21T12:25:36.886222+0000","last_peered":"2024-09-21T12:25:36.886222+0000","last_clean":"2024-09-21T12:25:36.886222+0000","last_became_active":"2024-09-21T12:22:50.917558+0000","last_became_peered":"2024-09-21T12:22:50.917558+0000","last_unstale":"2024-09-21T12:25:36.886222+0000","last_undegraded":"2024-09-21T12:25:36.886222+0000","last_fullsized":"2024-09-21T12:25:36.886222+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:20.237715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:20.237715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:07.980331+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"182'4574","reported_seq":4629,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:37.066626+0000","last_change":"2024-09-21T12:24:46.323849+0000","last_active":"2024-09-21T12:25:37.066626+0000","last_peered":"2024-09-21T12:25:37.066626+0000","last_clean":"2024-09-21T12:25:37.066626+0000","last_became_active":"2024-09-21T12:24:46.323306+0000","last_became_peered":"2024-09-21T12:24:46.323306+0000","last_unstale":"2024-09-21T12:25:37.066626+0000","last_undegraded":"2024-09-21T12:25:37.066626+0000","last_fullsized":"2024-09-21T12:25:37.066626+0000","mapping_epoch":160,"log_start":"165'400","ondisk_log_start":"165'400","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4174,"log_dups_size":0,"ondisk_log_size":4174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:58.661394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299761664,"num_objects":4574,"num_object_clones":0,"num_object_copies":13722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4574,"num_write_kb":292736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'79","reported_seq":349,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.042779+0000","last_change":"2024-09-21T12:25:03.877681+0000","last_active":"2024-09-21T12:25:35.042779+0000","last_peered":"2024-09-21T12:25:35.042779+0000","last_clean":"2024-09-21T12:25:35.042779+0000","last_became_active":"2024-09-21T12:22:50.921709+0000","last_became_peered":"2024-09-21T12:22:50.921709+0000","last_unstale":"2024-09-21T12:25:35.042779+0000","last_undegraded":"2024-09-21T12:25:35.042779+0000","last_fullsized":"2024-09-21T12:25:35.042779+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:25:03.877610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:25:03.877610+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:26.706650+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":289,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.041446+0000","last_change":"2024-09-21T12:24:32.318918+0000","last_active":"2024-09-21T12:25:35.041446+0000","last_peered":"2024-09-21T12:25:35.041446+0000","last_clean":"2024-09-21T12:25:35.041446+0000","last_became_active":"2024-09-21T12:20:17.297374+0000","last_became_peered":"2024-09-21T12:20:17.297374+0000","last_unstale":"2024-09-21T12:25:35.041446+0000","last_undegraded":"2024-09-21T12:25:35.041446+0000","last_fullsized":"2024-09-21T12:25:35.041446+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-21T12:24:32.318832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:32.318832+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:44.299112+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":274,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.038856+0000","last_change":"2024-09-21T12:23:59.367953+0000","last_active":"2024-09-21T12:25:35.038856+0000","last_peered":"2024-09-21T12:25:35.038856+0000","last_clean":"2024-09-21T12:25:35.038856+0000","last_became_active":"2024-09-21T12:20:15.308356+0000","last_became_peered":"2024-09-21T12:20:15.308356+0000","last_unstale":"2024-09-21T12:25:35.038856+0000","last_undegraded":"2024-09-21T12:25:35.038856+0000","last_fullsized":"2024-09-21T12:25:35.038856+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:59.367778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:23:59.367778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:21.472609+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":283,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.056059+0000","last_change":"2024-09-21T12:24:49.792138+0000","last_active":"2024-09-21T12:25:35.056059+0000","last_peered":"2024-09-21T12:25:35.056059+0000","last_clean":"2024-09-21T12:25:35.056059+0000","last_became_active":"2024-09-21T12:20:14.288358+0000","last_became_peered":"2024-09-21T12:20:14.288358+0000","last_unstale":"2024-09-21T12:25:35.056059+0000","last_undegraded":"2024-09-21T12:25:35.056059+0000","last_fullsized":"2024-09-21T12:25:35.056059+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:24:49.792086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:24:49.792086+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:11.091093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019042449999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":258,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.055591+0000","last_change":"2024-09-21T12:23:29.274584+0000","last_active":"2024-09-21T12:25:35.055591+0000","last_peered":"2024-09-21T12:25:35.055591+0000","last_clean":"2024-09-21T12:25:35.055591+0000","last_became_active":"2024-09-21T12:20:19.309724+0000","last_became_peered":"2024-09-21T12:20:19.309724+0000","last_unstale":"2024-09-21T12:25:35.055591+0000","last_undegraded":"2024-09-21T12:25:35.055591+0000","last_fullsized":"2024-09-21T12:25:35.055591+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:29.274316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:29.274316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:43.938081+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":245,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.044106+0000","last_change":"2024-09-21T12:20:21.593267+0000","last_active":"2024-09-21T12:25:35.044106+0000","last_peered":"2024-09-21T12:25:35.044106+0000","last_clean":"2024-09-21T12:25:35.044106+0000","last_became_active":"2024-09-21T12:20:21.592982+0000","last_became_peered":"2024-09-21T12:20:21.592982+0000","last_unstale":"2024-09-21T12:25:35.044106+0000","last_undegraded":"2024-09-21T12:25:35.044106+0000","last_fullsized":"2024-09-21T12:25:35.044106+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:22.831735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"181'4481","reported_seq":4533,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.765042+0000","last_change":"2024-09-21T12:24:46.326200+0000","last_active":"2024-09-21T12:25:36.765042+0000","last_peered":"2024-09-21T12:25:36.765042+0000","last_clean":"2024-09-21T12:25:36.765042+0000","last_became_active":"2024-09-21T12:24:46.325838+0000","last_became_peered":"2024-09-21T12:24:46.325838+0000","last_unstale":"2024-09-21T12:25:36.765042+0000","last_undegraded":"2024-09-21T12:25:36.765042+0000","last_fullsized":"2024-09-21T12:25:36.765042+0000","mapping_epoch":160,"log_start":"165'200","ondisk_log_start":"165'200","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4281,"log_dups_size":0,"ondisk_log_size":4281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:15.188003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293666816,"num_objects":4481,"num_object_clones":0,"num_object_copies":13443,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":286784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"76'79","reported_seq":188,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.038582+0000","last_change":"2024-09-21T12:23:57.442430+0000","last_active":"2024-09-21T12:25:35.038582+0000","last_peered":"2024-09-21T12:25:35.038582+0000","last_clean":"2024-09-21T12:25:35.038582+0000","last_became_active":"2024-09-21T12:22:50.912088+0000","last_became_peered":"2024-09-21T12:22:50.912088+0000","last_unstale":"2024-09-21T12:25:35.038582+0000","last_undegraded":"2024-09-21T12:25:35.038582+0000","last_fullsized":"2024-09-21T12:25:35.038582+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:23:57.442284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:23:57.442284+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:25.572064+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":282,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.875578+0000","last_change":"2024-09-21T12:22:49.933621+0000","last_active":"2024-09-21T12:25:36.875578+0000","last_peered":"2024-09-21T12:25:36.875578+0000","last_clean":"2024-09-21T12:25:36.875578+0000","last_became_active":"2024-09-21T12:20:17.299325+0000","last_became_peered":"2024-09-21T12:20:17.299325+0000","last_unstale":"2024-09-21T12:25:36.875578+0000","last_undegraded":"2024-09-21T12:25:36.875578+0000","last_fullsized":"2024-09-21T12:25:36.875578+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:22:49.933382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:49.933382+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:23.687742+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":297,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.873636+0000","last_change":"2024-09-21T12:24:48.073954+0000","last_active":"2024-09-21T12:25:36.873636+0000","last_peered":"2024-09-21T12:25:36.873636+0000","last_clean":"2024-09-21T12:25:36.873636+0000","last_became_active":"2024-09-21T12:22:50.923796+0000","last_became_peered":"2024-09-21T12:22:50.923796+0000","last_unstale":"2024-09-21T12:25:36.873636+0000","last_undegraded":"2024-09-21T12:25:36.873636+0000","last_fullsized":"2024-09-21T12:25:36.873636+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:48.073424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:48.073424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:52.673877+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":600,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.819878+0000","last_change":"2024-09-21T12:25:35.819787+0000","last_active":"2024-09-21T12:25:35.819878+0000","last_peered":"2024-09-21T12:25:35.819878+0000","last_clean":"2024-09-21T12:25:35.819878+0000","last_became_active":"2024-09-21T12:20:14.287543+0000","last_became_peered":"2024-09-21T12:20:14.287543+0000","last_unstale":"2024-09-21T12:25:35.819878+0000","last_undegraded":"2024-09-21T12:25:35.819878+0000","last_fullsized":"2024-09-21T12:25:35.819878+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-21T12:25:35.819707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:25:35.819707+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:05.641221+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":316,"num_read_kb":311,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":257,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.040909+0000","last_change":"2024-09-21T12:22:36.135464+0000","last_active":"2024-09-21T12:25:35.040909+0000","last_peered":"2024-09-21T12:25:35.040909+0000","last_clean":"2024-09-21T12:25:35.040909+0000","last_became_active":"2024-09-21T12:20:19.314217+0000","last_became_peered":"2024-09-21T12:20:19.314217+0000","last_unstale":"2024-09-21T12:25:35.040909+0000","last_undegraded":"2024-09-21T12:25:35.040909+0000","last_fullsized":"2024-09-21T12:25:35.040909+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:36.135390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:36.135390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:10.326219+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":262,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.037594+0000","last_change":"2024-09-21T12:24:35.356926+0000","last_active":"2024-09-21T12:25:35.037594+0000","last_peered":"2024-09-21T12:25:35.037594+0000","last_clean":"2024-09-21T12:25:35.037594+0000","last_became_active":"2024-09-21T12:20:21.311188+0000","last_became_peered":"2024-09-21T12:20:21.311188+0000","last_unstale":"2024-09-21T12:25:35.037594+0000","last_undegraded":"2024-09-21T12:25:35.037594+0000","last_fullsized":"2024-09-21T12:25:35.037594+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:35.356865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:35.356865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:38.245203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"181'4631","reported_seq":4684,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.618366+0000","last_change":"2024-09-21T12:24:46.318053+0000","last_active":"2024-09-21T12:25:36.618366+0000","last_peered":"2024-09-21T12:25:36.618366+0000","last_clean":"2024-09-21T12:25:36.618366+0000","last_became_active":"2024-09-21T12:24:46.317644+0000","last_became_peered":"2024-09-21T12:24:46.317644+0000","last_unstale":"2024-09-21T12:25:36.618366+0000","last_undegraded":"2024-09-21T12:25:36.618366+0000","last_fullsized":"2024-09-21T12:25:36.618366+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4631,"log_dups_size":0,"ondisk_log_size":4631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:56.464267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303497216,"num_objects":4631,"num_object_clones":0,"num_object_copies":13893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4631,"num_write_kb":296384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"121'19","reported_seq":431,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.043211+0000","last_change":"2024-09-21T12:24:27.932976+0000","last_active":"2024-09-21T12:25:35.043211+0000","last_peered":"2024-09-21T12:25:35.043211+0000","last_clean":"2024-09-21T12:25:35.043211+0000","last_became_active":"2024-09-21T12:22:50.921690+0000","last_became_peered":"2024-09-21T12:22:50.921690+0000","last_unstale":"2024-09-21T12:25:35.043211+0000","last_undegraded":"2024-09-21T12:25:35.043211+0000","last_fullsized":"2024-09-21T12:25:35.043211+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'19","last_scrub_stamp":"2024-09-21T12:24:27.932891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:27.932891+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:56.271337+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":272,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.885886+0000","last_change":"2024-09-21T12:21:17.107378+0000","last_active":"2024-09-21T12:25:36.885886+0000","last_peered":"2024-09-21T12:25:36.885886+0000","last_clean":"2024-09-21T12:25:36.885886+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:25:36.885886+0000","last_undegraded":"2024-09-21T12:25:36.885886+0000","last_fullsized":"2024-09-21T12:25:36.885886+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:21:17.107309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:17.107309+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:12.114826+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":329,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.042875+0000","last_change":"2024-09-21T12:24:31.089221+0000","last_active":"2024-09-21T12:25:35.042875+0000","last_peered":"2024-09-21T12:25:35.042875+0000","last_clean":"2024-09-21T12:25:35.042875+0000","last_became_active":"2024-09-21T12:22:15.049592+0000","last_became_peered":"2024-09-21T12:22:15.049592+0000","last_unstale":"2024-09-21T12:25:35.042875+0000","last_undegraded":"2024-09-21T12:25:35.042875+0000","last_fullsized":"2024-09-21T12:25:35.042875+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:24:31.089155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:24:31.089155+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:00.164503+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":249,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.044212+0000","last_change":"2024-09-21T12:20:19.310280+0000","last_active":"2024-09-21T12:25:35.044212+0000","last_peered":"2024-09-21T12:25:35.044212+0000","last_clean":"2024-09-21T12:25:35.044212+0000","last_became_active":"2024-09-21T12:20:19.309859+0000","last_became_peered":"2024-09-21T12:20:19.309859+0000","last_unstale":"2024-09-21T12:25:35.044212+0000","last_undegraded":"2024-09-21T12:25:35.044212+0000","last_fullsized":"2024-09-21T12:25:35.044212+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:28.753163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":290,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.040697+0000","last_change":"2024-09-21T12:25:32.465748+0000","last_active":"2024-09-21T12:25:35.040697+0000","last_peered":"2024-09-21T12:25:35.040697+0000","last_clean":"2024-09-21T12:25:35.040697+0000","last_became_active":"2024-09-21T12:22:50.913493+0000","last_became_peered":"2024-09-21T12:22:50.913493+0000","last_unstale":"2024-09-21T12:25:35.040697+0000","last_undegraded":"2024-09-21T12:25:35.040697+0000","last_fullsized":"2024-09-21T12:25:35.040697+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:32.465700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:32.465700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:57.893144+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"182'4526","reported_seq":4581,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:37.107412+0000","last_change":"2024-09-21T12:24:46.315286+0000","last_active":"2024-09-21T12:25:37.107412+0000","last_peered":"2024-09-21T12:25:37.107412+0000","last_clean":"2024-09-21T12:25:37.107412+0000","last_became_active":"2024-09-21T12:24:46.314957+0000","last_became_peered":"2024-09-21T12:24:46.314957+0000","last_unstale":"2024-09-21T12:25:37.107412+0000","last_undegraded":"2024-09-21T12:25:37.107412+0000","last_fullsized":"2024-09-21T12:25:37.107412+0000","mapping_epoch":160,"log_start":"165'300","ondisk_log_start":"165'300","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4226,"log_dups_size":0,"ondisk_log_size":4226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:52.237123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296615936,"num_objects":4526,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4526,"num_write_kb":289664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":271,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.041952+0000","last_change":"2024-09-21T12:23:51.061783+0000","last_active":"2024-09-21T12:25:35.041952+0000","last_peered":"2024-09-21T12:25:35.041952+0000","last_clean":"2024-09-21T12:25:35.041952+0000","last_became_active":"2024-09-21T12:23:46.885802+0000","last_became_peered":"2024-09-21T12:23:46.885802+0000","last_unstale":"2024-09-21T12:25:35.041952+0000","last_undegraded":"2024-09-21T12:25:35.041952+0000","last_fullsized":"2024-09-21T12:25:35.041952+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:23:51.061706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:51.061706+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:14.764585+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":189,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.055928+0000","last_change":"2024-09-21T12:22:50.908380+0000","last_active":"2024-09-21T12:25:35.055928+0000","last_peered":"2024-09-21T12:25:35.055928+0000","last_clean":"2024-09-21T12:25:35.055928+0000","last_became_active":"2024-09-21T12:22:50.908060+0000","last_became_peered":"2024-09-21T12:22:50.908060+0000","last_unstale":"2024-09-21T12:25:35.055928+0000","last_undegraded":"2024-09-21T12:25:35.055928+0000","last_fullsized":"2024-09-21T12:25:35.055928+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:05.892153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:05.892153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:26.966868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":286,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.040611+0000","last_change":"2024-09-21T12:23:19.372762+0000","last_active":"2024-09-21T12:25:35.040611+0000","last_peered":"2024-09-21T12:25:35.040611+0000","last_clean":"2024-09-21T12:25:35.040611+0000","last_became_active":"2024-09-21T12:22:50.915081+0000","last_became_peered":"2024-09-21T12:22:50.915081+0000","last_unstale":"2024-09-21T12:25:35.040611+0000","last_undegraded":"2024-09-21T12:25:35.040611+0000","last_fullsized":"2024-09-21T12:25:35.040611+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:19.372719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:19.372719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:37.323662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":254,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.888902+0000","last_change":"2024-09-21T12:23:46.282792+0000","last_active":"2024-09-21T12:25:36.888902+0000","last_peered":"2024-09-21T12:25:36.888902+0000","last_clean":"2024-09-21T12:25:36.888902+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:25:36.888902+0000","last_undegraded":"2024-09-21T12:25:36.888902+0000","last_fullsized":"2024-09-21T12:25:36.888902+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:46.282661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:46.282661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:15.069140+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"181'4671","reported_seq":4724,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.757362+0000","last_change":"2024-09-21T12:24:46.323690+0000","last_active":"2024-09-21T12:25:36.757362+0000","last_peered":"2024-09-21T12:25:36.757362+0000","last_clean":"2024-09-21T12:25:36.757362+0000","last_became_active":"2024-09-21T12:24:46.323402+0000","last_became_peered":"2024-09-21T12:24:46.323402+0000","last_unstale":"2024-09-21T12:25:36.757362+0000","last_undegraded":"2024-09-21T12:25:36.757362+0000","last_fullsized":"2024-09-21T12:25:36.757362+0000","mapping_epoch":160,"log_start":"165'400","ondisk_log_start":"165'400","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4271,"log_dups_size":0,"ondisk_log_size":4271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:55.565560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306118656,"num_objects":4671,"num_object_clones":0,"num_object_copies":14013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4671,"num_write_kb":298944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"76'79","reported_seq":181,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.040962+0000","last_change":"2024-09-21T12:24:28.372308+0000","last_active":"2024-09-21T12:25:35.040962+0000","last_peered":"2024-09-21T12:25:35.040962+0000","last_clean":"2024-09-21T12:25:35.040962+0000","last_became_active":"2024-09-21T12:22:17.087245+0000","last_became_peered":"2024-09-21T12:22:17.087245+0000","last_unstale":"2024-09-21T12:25:35.040962+0000","last_undegraded":"2024-09-21T12:25:35.040962+0000","last_fullsized":"2024-09-21T12:25:35.040962+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:24:28.372230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:24:28.372230+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:46.819824+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":306,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.056199+0000","last_change":"2024-09-21T12:23:50.328004+0000","last_active":"2024-09-21T12:25:35.056199+0000","last_peered":"2024-09-21T12:25:35.056199+0000","last_clean":"2024-09-21T12:25:35.056199+0000","last_became_active":"2024-09-21T12:23:46.897197+0000","last_became_peered":"2024-09-21T12:23:46.897197+0000","last_unstale":"2024-09-21T12:25:35.056199+0000","last_undegraded":"2024-09-21T12:25:35.056199+0000","last_fullsized":"2024-09-21T12:25:35.056199+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-21T12:23:50.327918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:50.327918+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":294,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.041943+0000","last_change":"2024-09-21T12:23:52.067111+0000","last_active":"2024-09-21T12:25:35.041943+0000","last_peered":"2024-09-21T12:25:35.041943+0000","last_clean":"2024-09-21T12:25:35.041943+0000","last_became_active":"2024-09-21T12:22:50.906886+0000","last_became_peered":"2024-09-21T12:22:50.906886+0000","last_unstale":"2024-09-21T12:25:35.041943+0000","last_undegraded":"2024-09-21T12:25:35.041943+0000","last_fullsized":"2024-09-21T12:25:35.041943+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:52.067065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:23:52.067065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:13.795671+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":279,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.037678+0000","last_change":"2024-09-21T12:23:17.434377+0000","last_active":"2024-09-21T12:25:35.037678+0000","last_peered":"2024-09-21T12:25:35.037678+0000","last_clean":"2024-09-21T12:25:35.037678+0000","last_became_active":"2024-09-21T12:22:50.927438+0000","last_became_peered":"2024-09-21T12:22:50.927438+0000","last_unstale":"2024-09-21T12:25:35.037678+0000","last_undegraded":"2024-09-21T12:25:35.037678+0000","last_fullsized":"2024-09-21T12:25:35.037678+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:17.434261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:17.434261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:46.370615+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":267,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.056464+0000","last_change":"2024-09-21T12:22:50.914246+0000","last_active":"2024-09-21T12:25:35.056464+0000","last_peered":"2024-09-21T12:25:35.056464+0000","last_clean":"2024-09-21T12:25:35.056464+0000","last_became_active":"2024-09-21T12:22:50.913898+0000","last_became_peered":"2024-09-21T12:22:50.913898+0000","last_unstale":"2024-09-21T12:25:35.056464+0000","last_undegraded":"2024-09-21T12:25:35.056464+0000","last_fullsized":"2024-09-21T12:25:35.056464+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:37.023546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"182'4670","reported_seq":4725,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:37.027268+0000","last_change":"2024-09-21T12:24:46.315895+0000","last_active":"2024-09-21T12:25:37.027268+0000","last_peered":"2024-09-21T12:25:37.027268+0000","last_clean":"2024-09-21T12:25:37.027268+0000","last_became_active":"2024-09-21T12:24:46.315601+0000","last_became_peered":"2024-09-21T12:24:46.315601+0000","last_unstale":"2024-09-21T12:25:37.027268+0000","last_undegraded":"2024-09-21T12:25:37.027268+0000","last_fullsized":"2024-09-21T12:25:37.027268+0000","mapping_epoch":160,"log_start":"166'500","ondisk_log_start":"166'500","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4170,"log_dups_size":0,"ondisk_log_size":4170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:06.483656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306053120,"num_objects":4670,"num_object_clones":0,"num_object_copies":14010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4670,"num_write_kb":298880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"76'79","reported_seq":319,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.042976+0000","last_change":"2024-09-21T12:22:17.073301+0000","last_active":"2024-09-21T12:25:35.042976+0000","last_peered":"2024-09-21T12:25:35.042976+0000","last_clean":"2024-09-21T12:25:35.042976+0000","last_became_active":"2024-09-21T12:22:17.072872+0000","last_became_peered":"2024-09-21T12:22:17.072872+0000","last_unstale":"2024-09-21T12:25:35.042976+0000","last_undegraded":"2024-09-21T12:25:35.042976+0000","last_fullsized":"2024-09-21T12:25:35.042976+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'72","last_scrub_stamp":"2024-09-21T12:20:24.850157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:20:24.850157+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:25:43.066052+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":21,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":120,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.055604+0000","last_change":"2024-09-21T12:25:03.787373+0000","last_active":"2024-09-21T12:25:35.055604+0000","last_peered":"2024-09-21T12:25:35.055604+0000","last_clean":"2024-09-21T12:25:35.055604+0000","last_became_active":"2024-09-21T12:23:46.902260+0000","last_became_peered":"2024-09-21T12:23:46.902260+0000","last_unstale":"2024-09-21T12:25:35.055604+0000","last_undegraded":"2024-09-21T12:25:35.055604+0000","last_fullsized":"2024-09-21T12:25:35.055604+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:25:03.787288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:03.787288+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:04.352295+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":287,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.886528+0000","last_change":"2024-09-21T12:22:50.928240+0000","last_active":"2024-09-21T12:25:36.886528+0000","last_peered":"2024-09-21T12:25:36.886528+0000","last_clean":"2024-09-21T12:25:36.886528+0000","last_became_active":"2024-09-21T12:22:50.927946+0000","last_became_peered":"2024-09-21T12:22:50.927946+0000","last_unstale":"2024-09-21T12:25:36.886528+0000","last_undegraded":"2024-09-21T12:25:36.886528+0000","last_fullsized":"2024-09-21T12:25:36.886528+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:08.999015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:08.999015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:24.102761+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":339,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.056180+0000","last_change":"2024-09-21T12:24:52.846590+0000","last_active":"2024-09-21T12:25:35.056180+0000","last_peered":"2024-09-21T12:25:35.056180+0000","last_clean":"2024-09-21T12:25:35.056180+0000","last_became_active":"2024-09-21T12:20:19.307924+0000","last_became_peered":"2024-09-21T12:20:19.307924+0000","last_unstale":"2024-09-21T12:25:35.056180+0000","last_undegraded":"2024-09-21T12:25:35.056180+0000","last_fullsized":"2024-09-21T12:25:35.056180+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:24:52.846112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:52.846112+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:08.133859+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":197,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.041981+0000","last_change":"2024-09-21T12:23:56.471960+0000","last_active":"2024-09-21T12:25:35.041981+0000","last_peered":"2024-09-21T12:25:35.041981+0000","last_clean":"2024-09-21T12:25:35.041981+0000","last_became_active":"2024-09-21T12:22:50.905470+0000","last_became_peered":"2024-09-21T12:22:50.905470+0000","last_unstale":"2024-09-21T12:25:35.041981+0000","last_undegraded":"2024-09-21T12:25:35.041981+0000","last_fullsized":"2024-09-21T12:25:35.041981+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:56.471116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:56.471116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:22.516139+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.7","version":"181'4584","reported_seq":4637,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.756719+0000","last_change":"2024-09-21T12:24:46.317724+0000","last_active":"2024-09-21T12:25:36.756719+0000","last_peered":"2024-09-21T12:25:36.756719+0000","last_clean":"2024-09-21T12:25:36.756719+0000","last_became_active":"2024-09-21T12:24:46.317468+0000","last_became_peered":"2024-09-21T12:24:46.317468+0000","last_unstale":"2024-09-21T12:25:36.756719+0000","last_undegraded":"2024-09-21T12:25:36.756719+0000","last_fullsized":"2024-09-21T12:25:36.756719+0000","mapping_epoch":160,"log_start":"165'300","ondisk_log_start":"165'300","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4284,"log_dups_size":0,"ondisk_log_size":4284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:52.389257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300417024,"num_objects":4584,"num_object_clones":0,"num_object_copies":13752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4584,"num_write_kb":293376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":347,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.874374+0000","last_change":"2024-09-21T12:25:00.237083+0000","last_active":"2024-09-21T12:25:36.874374+0000","last_peered":"2024-09-21T12:25:36.874374+0000","last_clean":"2024-09-21T12:25:36.874374+0000","last_became_active":"2024-09-21T12:22:50.922343+0000","last_became_peered":"2024-09-21T12:22:50.922343+0000","last_unstale":"2024-09-21T12:25:36.874374+0000","last_undegraded":"2024-09-21T12:25:36.874374+0000","last_fullsized":"2024-09-21T12:25:36.874374+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:25:00.236990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:00.236990+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:26.107161+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":281,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.044121+0000","last_change":"2024-09-21T12:25:30.451939+0000","last_active":"2024-09-21T12:25:35.044121+0000","last_peered":"2024-09-21T12:25:35.044121+0000","last_clean":"2024-09-21T12:25:35.044121+0000","last_became_active":"2024-09-21T12:20:15.302262+0000","last_became_peered":"2024-09-21T12:20:15.302262+0000","last_unstale":"2024-09-21T12:25:35.044121+0000","last_undegraded":"2024-09-21T12:25:35.044121+0000","last_fullsized":"2024-09-21T12:25:35.044121+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:30.451876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:30.451876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:51.913189+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":295,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.012066+0000","last_change":"2024-09-21T12:25:36.012066+0000","last_active":"2024-09-21T12:25:36.012066+0000","last_peered":"2024-09-21T12:25:36.012066+0000","last_clean":"2024-09-21T12:25:36.012066+0000","last_became_active":"2024-09-21T12:22:50.925974+0000","last_became_peered":"2024-09-21T12:22:50.925974+0000","last_unstale":"2024-09-21T12:25:36.012066+0000","last_undegraded":"2024-09-21T12:25:36.012066+0000","last_fullsized":"2024-09-21T12:25:36.012066+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:36.011984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:36.011984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:40.801014+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":256,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.873553+0000","last_change":"2024-09-21T12:23:28.121344+0000","last_active":"2024-09-21T12:25:36.873553+0000","last_peered":"2024-09-21T12:25:36.873553+0000","last_clean":"2024-09-21T12:25:36.873553+0000","last_became_active":"2024-09-21T12:20:21.311493+0000","last_became_peered":"2024-09-21T12:20:21.311493+0000","last_unstale":"2024-09-21T12:25:36.873553+0000","last_undegraded":"2024-09-21T12:25:36.873553+0000","last_fullsized":"2024-09-21T12:25:36.873553+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:28.121286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:28.121286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:24.681035+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"182'4628","reported_seq":4683,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:37.104152+0000","last_change":"2024-09-21T12:24:46.327091+0000","last_active":"2024-09-21T12:25:37.104152+0000","last_peered":"2024-09-21T12:25:37.104152+0000","last_clean":"2024-09-21T12:25:37.104152+0000","last_became_active":"2024-09-21T12:24:46.325838+0000","last_became_peered":"2024-09-21T12:24:46.325838+0000","last_unstale":"2024-09-21T12:25:37.104152+0000","last_undegraded":"2024-09-21T12:25:37.104152+0000","last_fullsized":"2024-09-21T12:25:37.104152+0000","mapping_epoch":160,"log_start":"165'400","ondisk_log_start":"165'400","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4228,"log_dups_size":0,"ondisk_log_size":4228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:07.844175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303300608,"num_objects":4628,"num_object_clones":0,"num_object_copies":13884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4628,"num_write_kb":296192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":300,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.041665+0000","last_change":"2024-09-21T12:24:02.960807+0000","last_active":"2024-09-21T12:25:35.041665+0000","last_peered":"2024-09-21T12:25:35.041665+0000","last_clean":"2024-09-21T12:25:35.041665+0000","last_became_active":"2024-09-21T12:20:17.692867+0000","last_became_peered":"2024-09-21T12:20:17.692867+0000","last_unstale":"2024-09-21T12:25:35.041665+0000","last_undegraded":"2024-09-21T12:25:35.041665+0000","last_fullsized":"2024-09-21T12:25:35.041665+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-21T12:24:02.960733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:02.960733+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:18.893627+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":265,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.044294+0000","last_change":"2024-09-21T12:23:59.046000+0000","last_active":"2024-09-21T12:25:35.044294+0000","last_peered":"2024-09-21T12:25:35.044294+0000","last_clean":"2024-09-21T12:25:35.044294+0000","last_became_active":"2024-09-21T12:20:15.299333+0000","last_became_peered":"2024-09-21T12:20:15.299333+0000","last_unstale":"2024-09-21T12:25:35.044294+0000","last_undegraded":"2024-09-21T12:25:35.044294+0000","last_fullsized":"2024-09-21T12:25:35.044294+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:59.045927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:23:59.045927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":258,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.055090+0000","last_change":"2024-09-21T12:24:34.060244+0000","last_active":"2024-09-21T12:25:35.055090+0000","last_peered":"2024-09-21T12:25:35.055090+0000","last_clean":"2024-09-21T12:25:35.055090+0000","last_became_active":"2024-09-21T12:20:19.306952+0000","last_became_peered":"2024-09-21T12:20:19.306952+0000","last_unstale":"2024-09-21T12:25:35.055090+0000","last_undegraded":"2024-09-21T12:25:35.055090+0000","last_fullsized":"2024-09-21T12:25:35.055090+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:34.060188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:34.060188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:46.315441+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":245,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.042769+0000","last_change":"2024-09-21T12:20:21.314891+0000","last_active":"2024-09-21T12:25:35.042769+0000","last_peered":"2024-09-21T12:25:35.042769+0000","last_clean":"2024-09-21T12:25:35.042769+0000","last_became_active":"2024-09-21T12:20:21.314545+0000","last_became_peered":"2024-09-21T12:20:21.314545+0000","last_unstale":"2024-09-21T12:25:35.042769+0000","last_undegraded":"2024-09-21T12:25:35.042769+0000","last_fullsized":"2024-09-21T12:25:35.042769+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:48.052208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"181'4562","reported_seq":4615,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.597694+0000","last_change":"2024-09-21T12:24:46.325063+0000","last_active":"2024-09-21T12:25:36.597694+0000","last_peered":"2024-09-21T12:25:36.597694+0000","last_clean":"2024-09-21T12:25:36.597694+0000","last_became_active":"2024-09-21T12:24:46.324554+0000","last_became_peered":"2024-09-21T12:24:46.324554+0000","last_unstale":"2024-09-21T12:25:36.597694+0000","last_undegraded":"2024-09-21T12:25:36.597694+0000","last_fullsized":"2024-09-21T12:25:36.597694+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4562,"log_dups_size":0,"ondisk_log_size":4562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:48.160004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298975232,"num_objects":4562,"num_object_clones":0,"num_object_copies":13686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4562,"num_write_kb":291968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":272,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.039127+0000","last_change":"2024-09-21T12:24:54.445801+0000","last_active":"2024-09-21T12:25:35.039127+0000","last_peered":"2024-09-21T12:25:35.039127+0000","last_clean":"2024-09-21T12:25:35.039127+0000","last_became_active":"2024-09-21T12:22:50.910973+0000","last_became_peered":"2024-09-21T12:22:50.910973+0000","last_unstale":"2024-09-21T12:25:35.039127+0000","last_undegraded":"2024-09-21T12:25:35.039127+0000","last_fullsized":"2024-09-21T12:25:35.039127+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:24:54.445714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:54.445714+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:16.862302+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":286,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.043999+0000","last_change":"2024-09-21T12:22:50.928139+0000","last_active":"2024-09-21T12:25:35.043999+0000","last_peered":"2024-09-21T12:25:35.043999+0000","last_clean":"2024-09-21T12:25:35.043999+0000","last_became_active":"2024-09-21T12:22:50.927859+0000","last_became_peered":"2024-09-21T12:22:50.927859+0000","last_unstale":"2024-09-21T12:25:35.043999+0000","last_undegraded":"2024-09-21T12:25:35.043999+0000","last_fullsized":"2024-09-21T12:25:35.043999+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:20.865749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.865749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":258,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.055067+0000","last_change":"2024-09-21T12:21:23.393320+0000","last_active":"2024-09-21T12:25:35.055067+0000","last_peered":"2024-09-21T12:25:35.055067+0000","last_clean":"2024-09-21T12:25:35.055067+0000","last_became_active":"2024-09-21T12:20:19.312572+0000","last_became_peered":"2024-09-21T12:20:19.312572+0000","last_unstale":"2024-09-21T12:25:35.055067+0000","last_undegraded":"2024-09-21T12:25:35.055067+0000","last_fullsized":"2024-09-21T12:25:35.055067+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:23.393270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:21:23.393270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:39.019942+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":254,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.886207+0000","last_change":"2024-09-21T12:22:36.006177+0000","last_active":"2024-09-21T12:25:36.886207+0000","last_peered":"2024-09-21T12:25:36.886207+0000","last_clean":"2024-09-21T12:25:36.886207+0000","last_became_active":"2024-09-21T12:20:21.317223+0000","last_became_peered":"2024-09-21T12:20:21.317223+0000","last_unstale":"2024-09-21T12:25:36.886207+0000","last_undegraded":"2024-09-21T12:25:36.886207+0000","last_fullsized":"2024-09-21T12:25:36.886207+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:36.006109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:36.006109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:44.698416+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"181'4654","reported_seq":4707,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.573474+0000","last_change":"2024-09-21T12:24:46.323500+0000","last_active":"2024-09-21T12:25:36.573474+0000","last_peered":"2024-09-21T12:25:36.573474+0000","last_clean":"2024-09-21T12:25:36.573474+0000","last_became_active":"2024-09-21T12:24:46.323192+0000","last_became_peered":"2024-09-21T12:24:46.323192+0000","last_unstale":"2024-09-21T12:25:36.573474+0000","last_undegraded":"2024-09-21T12:25:36.573474+0000","last_fullsized":"2024-09-21T12:25:36.573474+0000","mapping_epoch":160,"log_start":"165'200","ondisk_log_start":"165'200","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4454,"log_dups_size":0,"ondisk_log_size":4454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:57.240434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305004544,"num_objects":4654,"num_object_clones":0,"num_object_copies":13962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4654,"num_write_kb":297856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":302,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.041085+0000","last_change":"2024-09-21T12:25:03.416100+0000","last_active":"2024-09-21T12:25:35.041085+0000","last_peered":"2024-09-21T12:25:35.041085+0000","last_clean":"2024-09-21T12:25:35.041085+0000","last_became_active":"2024-09-21T12:20:17.692870+0000","last_became_peered":"2024-09-21T12:20:17.692870+0000","last_unstale":"2024-09-21T12:25:35.041085+0000","last_undegraded":"2024-09-21T12:25:35.041085+0000","last_fullsized":"2024-09-21T12:25:35.041085+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:25:03.416033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:03.416033+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:09.840901+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":324,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.044221+0000","last_change":"2024-09-21T12:24:29.054851+0000","last_active":"2024-09-21T12:25:35.044221+0000","last_peered":"2024-09-21T12:25:35.044221+0000","last_clean":"2024-09-21T12:25:35.044221+0000","last_became_active":"2024-09-21T12:22:50.927141+0000","last_became_peered":"2024-09-21T12:22:50.927141+0000","last_unstale":"2024-09-21T12:25:35.044221+0000","last_undegraded":"2024-09-21T12:25:35.044221+0000","last_fullsized":"2024-09-21T12:25:35.044221+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-21T12:24:29.054772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:29.054772+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:50.900488+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"121'11","reported_seq":363,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.042925+0000","last_change":"2024-09-21T12:25:02.916145+0000","last_active":"2024-09-21T12:25:35.042925+0000","last_peered":"2024-09-21T12:25:35.042925+0000","last_clean":"2024-09-21T12:25:35.042925+0000","last_became_active":"2024-09-21T12:22:50.908371+0000","last_became_peered":"2024-09-21T12:22:50.908371+0000","last_unstale":"2024-09-21T12:25:35.042925+0000","last_undegraded":"2024-09-21T12:25:35.042925+0000","last_fullsized":"2024-09-21T12:25:35.042925+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:25:02.916090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:02.916090+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:31.534554+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":253,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.041045+0000","last_change":"2024-09-21T12:22:45.223570+0000","last_active":"2024-09-21T12:25:35.041045+0000","last_peered":"2024-09-21T12:25:35.041045+0000","last_clean":"2024-09-21T12:25:35.041045+0000","last_became_active":"2024-09-21T12:20:21.592576+0000","last_became_peered":"2024-09-21T12:20:21.592576+0000","last_unstale":"2024-09-21T12:25:35.041045+0000","last_undegraded":"2024-09-21T12:25:35.041045+0000","last_fullsized":"2024-09-21T12:25:35.041045+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:45.223396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:45.223396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:20.386745+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":289,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.040430+0000","last_change":"2024-09-21T12:25:31.845118+0000","last_active":"2024-09-21T12:25:35.040430+0000","last_peered":"2024-09-21T12:25:35.040430+0000","last_clean":"2024-09-21T12:25:35.040430+0000","last_became_active":"2024-09-21T12:20:15.303480+0000","last_became_peered":"2024-09-21T12:20:15.303480+0000","last_unstale":"2024-09-21T12:25:35.040430+0000","last_undegraded":"2024-09-21T12:25:35.040430+0000","last_fullsized":"2024-09-21T12:25:35.040430+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:31.845066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:31.845066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:34.886910+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":271,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.887299+0000","last_change":"2024-09-21T12:22:36.999995+0000","last_active":"2024-09-21T12:25:36.887299+0000","last_peered":"2024-09-21T12:25:36.887299+0000","last_clean":"2024-09-21T12:25:36.887299+0000","last_became_active":"2024-09-21T12:20:17.694628+0000","last_became_peered":"2024-09-21T12:20:17.694628+0000","last_unstale":"2024-09-21T12:25:36.887299+0000","last_undegraded":"2024-09-21T12:25:36.887299+0000","last_fullsized":"2024-09-21T12:25:36.887299+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-21T12:22:36.999705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:36.999705+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:43.707926+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":273,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.040596+0000","last_change":"2024-09-21T12:25:02.414377+0000","last_active":"2024-09-21T12:25:35.040596+0000","last_peered":"2024-09-21T12:25:35.040596+0000","last_clean":"2024-09-21T12:25:35.040596+0000","last_became_active":"2024-09-21T12:20:19.313882+0000","last_became_peered":"2024-09-21T12:20:19.313882+0000","last_unstale":"2024-09-21T12:25:35.040596+0000","last_undegraded":"2024-09-21T12:25:35.040596+0000","last_fullsized":"2024-09-21T12:25:35.040596+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:02.414298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:02.414298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:04.541864+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":274,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.044075+0000","last_change":"2024-09-21T12:24:00.066522+0000","last_active":"2024-09-21T12:25:35.044075+0000","last_peered":"2024-09-21T12:25:35.044075+0000","last_clean":"2024-09-21T12:25:35.044075+0000","last_became_active":"2024-09-21T12:22:50.904404+0000","last_became_peered":"2024-09-21T12:22:50.904404+0000","last_unstale":"2024-09-21T12:25:35.044075+0000","last_undegraded":"2024-09-21T12:25:35.044075+0000","last_fullsized":"2024-09-21T12:25:35.044075+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:00.066451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:00.066451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":280,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.875664+0000","last_change":"2024-09-21T12:21:24.093710+0000","last_active":"2024-09-21T12:25:36.875664+0000","last_peered":"2024-09-21T12:25:36.875664+0000","last_clean":"2024-09-21T12:25:36.875664+0000","last_became_active":"2024-09-21T12:20:17.695898+0000","last_became_peered":"2024-09-21T12:20:17.695898+0000","last_unstale":"2024-09-21T12:25:36.875664+0000","last_undegraded":"2024-09-21T12:25:36.875664+0000","last_fullsized":"2024-09-21T12:25:36.875664+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:21:24.093656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:21:24.093656+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:20.587663+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":294,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.043075+0000","last_change":"2024-09-21T12:24:26.078804+0000","last_active":"2024-09-21T12:25:35.043075+0000","last_peered":"2024-09-21T12:25:35.043075+0000","last_clean":"2024-09-21T12:25:35.043075+0000","last_became_active":"2024-09-21T12:22:50.927649+0000","last_became_peered":"2024-09-21T12:22:50.927649+0000","last_unstale":"2024-09-21T12:25:35.043075+0000","last_undegraded":"2024-09-21T12:25:35.043075+0000","last_fullsized":"2024-09-21T12:25:35.043075+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:26.078737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:26.078737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:51.918796+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":258,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.887135+0000","last_change":"2024-09-21T12:23:53.356954+0000","last_active":"2024-09-21T12:25:36.887135+0000","last_peered":"2024-09-21T12:25:36.887135+0000","last_clean":"2024-09-21T12:25:36.887135+0000","last_became_active":"2024-09-21T12:20:19.311947+0000","last_became_peered":"2024-09-21T12:20:19.311947+0000","last_unstale":"2024-09-21T12:25:36.887135+0000","last_undegraded":"2024-09-21T12:25:36.887135+0000","last_fullsized":"2024-09-21T12:25:36.887135+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:53.356896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:53.356896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:28.202782+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":291,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.055734+0000","last_change":"2024-09-21T12:24:58.847710+0000","last_active":"2024-09-21T12:25:35.055734+0000","last_peered":"2024-09-21T12:25:35.055734+0000","last_clean":"2024-09-21T12:25:35.055734+0000","last_became_active":"2024-09-21T12:22:50.924931+0000","last_became_peered":"2024-09-21T12:22:50.924931+0000","last_unstale":"2024-09-21T12:25:35.055734+0000","last_undegraded":"2024-09-21T12:25:35.055734+0000","last_fullsized":"2024-09-21T12:25:35.055734+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:58.847643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:58.847643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:05.629548+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":294,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.042291+0000","last_change":"2024-09-21T12:23:46.925752+0000","last_active":"2024-09-21T12:25:35.042291+0000","last_peered":"2024-09-21T12:25:35.042291+0000","last_clean":"2024-09-21T12:25:35.042291+0000","last_became_active":"2024-09-21T12:23:46.898523+0000","last_became_peered":"2024-09-21T12:23:46.898523+0000","last_unstale":"2024-09-21T12:25:35.042291+0000","last_undegraded":"2024-09-21T12:25:35.042291+0000","last_fullsized":"2024-09-21T12:25:35.042291+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:22:39.201667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:39.201667+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:16.341268+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":276,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.874135+0000","last_change":"2024-09-21T12:24:25.931617+0000","last_active":"2024-09-21T12:25:36.874135+0000","last_peered":"2024-09-21T12:25:36.874135+0000","last_clean":"2024-09-21T12:25:36.874135+0000","last_became_active":"2024-09-21T12:20:15.297519+0000","last_became_peered":"2024-09-21T12:20:15.297519+0000","last_unstale":"2024-09-21T12:25:36.874135+0000","last_undegraded":"2024-09-21T12:25:36.874135+0000","last_fullsized":"2024-09-21T12:25:36.874135+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:25.931561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:25.931561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:44.486853+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":331,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.889133+0000","last_change":"2024-09-21T12:23:57.379033+0000","last_active":"2024-09-21T12:25:36.889133+0000","last_peered":"2024-09-21T12:25:36.889133+0000","last_clean":"2024-09-21T12:25:36.889133+0000","last_became_active":"2024-09-21T12:20:19.305364+0000","last_became_peered":"2024-09-21T12:20:19.305364+0000","last_unstale":"2024-09-21T12:25:36.889133+0000","last_undegraded":"2024-09-21T12:25:36.889133+0000","last_fullsized":"2024-09-21T12:25:36.889133+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:23:57.378959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:57.378959+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:16.020266+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":197,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.056569+0000","last_change":"2024-09-21T12:24:51.831043+0000","last_active":"2024-09-21T12:25:35.056569+0000","last_peered":"2024-09-21T12:25:35.056569+0000","last_clean":"2024-09-21T12:25:35.056569+0000","last_became_active":"2024-09-21T12:22:50.924770+0000","last_became_peered":"2024-09-21T12:22:50.924770+0000","last_unstale":"2024-09-21T12:25:35.056569+0000","last_undegraded":"2024-09-21T12:25:35.056569+0000","last_fullsized":"2024-09-21T12:25:35.056569+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:51.830994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:51.830994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:54.192853+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":285,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.889299+0000","last_change":"2024-09-21T12:24:50.727161+0000","last_active":"2024-09-21T12:25:36.889299+0000","last_peered":"2024-09-21T12:25:36.889299+0000","last_clean":"2024-09-21T12:25:36.889299+0000","last_became_active":"2024-09-21T12:20:17.693209+0000","last_became_peered":"2024-09-21T12:20:17.693209+0000","last_unstale":"2024-09-21T12:25:36.889299+0000","last_undegraded":"2024-09-21T12:25:36.889299+0000","last_fullsized":"2024-09-21T12:25:36.889299+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:24:50.727101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:50.727101+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:11.446387+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":286,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.043900+0000","last_change":"2024-09-21T12:22:50.908560+0000","last_active":"2024-09-21T12:25:35.043900+0000","last_peered":"2024-09-21T12:25:35.043900+0000","last_clean":"2024-09-21T12:25:35.043900+0000","last_became_active":"2024-09-21T12:22:50.908238+0000","last_became_peered":"2024-09-21T12:22:50.908238+0000","last_unstale":"2024-09-21T12:25:35.043900+0000","last_undegraded":"2024-09-21T12:25:35.043900+0000","last_fullsized":"2024-09-21T12:25:35.043900+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:15.938093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:15.938093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":330,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.041931+0000","last_change":"2024-09-21T12:22:46.216242+0000","last_active":"2024-09-21T12:25:35.041931+0000","last_peered":"2024-09-21T12:25:35.041931+0000","last_clean":"2024-09-21T12:25:35.041931+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:25:35.041931+0000","last_undegraded":"2024-09-21T12:25:35.041931+0000","last_fullsized":"2024-09-21T12:25:35.041931+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:22:46.216153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:46.216153+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:22.367827+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":248,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.875069+0000","last_change":"2024-09-21T12:20:21.309936+0000","last_active":"2024-09-21T12:25:36.875069+0000","last_peered":"2024-09-21T12:25:36.875069+0000","last_clean":"2024-09-21T12:25:36.875069+0000","last_became_active":"2024-09-21T12:20:21.309385+0000","last_became_peered":"2024-09-21T12:20:21.309385+0000","last_unstale":"2024-09-21T12:25:36.875069+0000","last_undegraded":"2024-09-21T12:25:36.875069+0000","last_fullsized":"2024-09-21T12:25:36.875069+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:06.355734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":252,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.041922+0000","last_change":"2024-09-21T12:23:59.046758+0000","last_active":"2024-09-21T12:25:35.041922+0000","last_peered":"2024-09-21T12:25:35.041922+0000","last_clean":"2024-09-21T12:25:35.041922+0000","last_became_active":"2024-09-21T12:22:50.906473+0000","last_became_peered":"2024-09-21T12:22:50.906473+0000","last_unstale":"2024-09-21T12:25:35.041922+0000","last_undegraded":"2024-09-21T12:25:35.041922+0000","last_fullsized":"2024-09-21T12:25:35.041922+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:23:59.046704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:59.046704+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:16.851121+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":189,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.043938+0000","last_change":"2024-09-21T12:24:58.948698+0000","last_active":"2024-09-21T12:25:35.043938+0000","last_peered":"2024-09-21T12:25:35.043938+0000","last_clean":"2024-09-21T12:25:35.043938+0000","last_became_active":"2024-09-21T12:22:50.903705+0000","last_became_peered":"2024-09-21T12:22:50.903705+0000","last_unstale":"2024-09-21T12:25:35.043938+0000","last_undegraded":"2024-09-21T12:25:35.043938+0000","last_fullsized":"2024-09-21T12:25:35.043938+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:58.948625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:58.948625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:23.221920+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":253,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.037680+0000","last_change":"2024-09-21T12:24:38.746182+0000","last_active":"2024-09-21T12:25:35.037680+0000","last_peered":"2024-09-21T12:25:35.037680+0000","last_clean":"2024-09-21T12:25:35.037680+0000","last_became_active":"2024-09-21T12:20:19.315418+0000","last_became_peered":"2024-09-21T12:20:19.315418+0000","last_unstale":"2024-09-21T12:25:35.037680+0000","last_undegraded":"2024-09-21T12:25:35.037680+0000","last_fullsized":"2024-09-21T12:25:35.037680+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:15.446578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":262,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.885188+0000","last_change":"2024-09-21T12:24:56.879617+0000","last_active":"2024-09-21T12:25:36.885188+0000","last_peered":"2024-09-21T12:25:36.885188+0000","last_clean":"2024-09-21T12:25:36.885188+0000","last_became_active":"2024-09-21T12:20:21.310053+0000","last_became_peered":"2024-09-21T12:20:21.310053+0000","last_unstale":"2024-09-21T12:25:36.885188+0000","last_undegraded":"2024-09-21T12:25:36.885188+0000","last_fullsized":"2024-09-21T12:25:36.885188+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:56.879558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:56.879558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:26.453298+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":270,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.037914+0000","last_change":"2024-09-21T12:20:17.299367+0000","last_active":"2024-09-21T12:25:35.037914+0000","last_peered":"2024-09-21T12:25:35.037914+0000","last_clean":"2024-09-21T12:25:35.037914+0000","last_became_active":"2024-09-21T12:20:17.299015+0000","last_became_peered":"2024-09-21T12:20:17.299015+0000","last_unstale":"2024-09-21T12:25:35.037914+0000","last_undegraded":"2024-09-21T12:25:35.037914+0000","last_fullsized":"2024-09-21T12:25:35.037914+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:40.326829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":294,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.044225+0000","last_change":"2024-09-21T12:24:30.077339+0000","last_active":"2024-09-21T12:25:35.044225+0000","last_peered":"2024-09-21T12:25:35.044225+0000","last_clean":"2024-09-21T12:25:35.044225+0000","last_became_active":"2024-09-21T12:22:50.922019+0000","last_became_peered":"2024-09-21T12:22:50.922019+0000","last_unstale":"2024-09-21T12:25:35.044225+0000","last_undegraded":"2024-09-21T12:25:35.044225+0000","last_fullsized":"2024-09-21T12:25:35.044225+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:30.077279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:30.077279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:52.329408+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":258,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.886287+0000","last_change":"2024-09-21T12:21:20.931077+0000","last_active":"2024-09-21T12:25:36.886287+0000","last_peered":"2024-09-21T12:25:36.886287+0000","last_clean":"2024-09-21T12:25:36.886287+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:25:36.886287+0000","last_undegraded":"2024-09-21T12:25:36.886287+0000","last_fullsized":"2024-09-21T12:25:36.886287+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:20.931025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.931025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:18.080044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":256,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.872243+0000","last_change":"2024-09-21T12:23:29.134829+0000","last_active":"2024-09-21T12:25:36.872243+0000","last_peered":"2024-09-21T12:25:36.872243+0000","last_clean":"2024-09-21T12:25:36.872243+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:25:36.872243+0000","last_undegraded":"2024-09-21T12:25:36.872243+0000","last_fullsized":"2024-09-21T12:25:36.872243+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:29.134767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:29.134767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:08.422630+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":289,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.885419+0000","last_change":"2024-09-21T12:24:58.824615+0000","last_active":"2024-09-21T12:25:36.885419+0000","last_peered":"2024-09-21T12:25:36.885419+0000","last_clean":"2024-09-21T12:25:36.885419+0000","last_became_active":"2024-09-21T12:20:17.694794+0000","last_became_peered":"2024-09-21T12:20:17.694794+0000","last_unstale":"2024-09-21T12:25:36.885419+0000","last_undegraded":"2024-09-21T12:25:36.885419+0000","last_fullsized":"2024-09-21T12:25:36.885419+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:24:58.824516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:58.824516+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:24.775664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":281,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.040656+0000","last_change":"2024-09-21T12:24:33.364225+0000","last_active":"2024-09-21T12:25:35.040656+0000","last_peered":"2024-09-21T12:25:35.040656+0000","last_clean":"2024-09-21T12:25:35.040656+0000","last_became_active":"2024-09-21T12:20:15.295968+0000","last_became_peered":"2024-09-21T12:20:15.295968+0000","last_unstale":"2024-09-21T12:25:35.040656+0000","last_undegraded":"2024-09-21T12:25:35.040656+0000","last_fullsized":"2024-09-21T12:25:35.040656+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:33.363808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:33.363808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:01.574062+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":278,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.042488+0000","last_change":"2024-09-21T12:23:57.018797+0000","last_active":"2024-09-21T12:25:35.042488+0000","last_peered":"2024-09-21T12:25:35.042488+0000","last_clean":"2024-09-21T12:25:35.042488+0000","last_became_active":"2024-09-21T12:22:50.921604+0000","last_became_peered":"2024-09-21T12:22:50.921604+0000","last_unstale":"2024-09-21T12:25:35.042488+0000","last_undegraded":"2024-09-21T12:25:35.042488+0000","last_fullsized":"2024-09-21T12:25:35.042488+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:57.018749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:57.018749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":254,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.886452+0000","last_change":"2024-09-21T12:24:35.588409+0000","last_active":"2024-09-21T12:25:36.886452+0000","last_peered":"2024-09-21T12:25:36.886452+0000","last_clean":"2024-09-21T12:25:36.886452+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:25:36.886452+0000","last_undegraded":"2024-09-21T12:25:36.886452+0000","last_fullsized":"2024-09-21T12:25:36.886452+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:35.588353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:35.588353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:51.954226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":268,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.890369+0000","last_change":"2024-09-21T12:23:51.343568+0000","last_active":"2024-09-21T12:25:36.890369+0000","last_peered":"2024-09-21T12:25:36.890369+0000","last_clean":"2024-09-21T12:25:36.890369+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:25:36.890369+0000","last_undegraded":"2024-09-21T12:25:36.890369+0000","last_fullsized":"2024-09-21T12:25:36.890369+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-21T12:23:51.343491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:51.343491+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:26.153152+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":274,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.875256+0000","last_change":"2024-09-21T12:22:14.635397+0000","last_active":"2024-09-21T12:25:36.875256+0000","last_peered":"2024-09-21T12:25:36.875256+0000","last_clean":"2024-09-21T12:25:36.875256+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:25:36.875256+0000","last_undegraded":"2024-09-21T12:25:36.875256+0000","last_fullsized":"2024-09-21T12:25:36.875256+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:22:14.635191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.635191+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:43.134430+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":278,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.041376+0000","last_change":"2024-09-21T12:22:50.931202+0000","last_active":"2024-09-21T12:25:35.041376+0000","last_peered":"2024-09-21T12:25:35.041376+0000","last_clean":"2024-09-21T12:25:35.041376+0000","last_became_active":"2024-09-21T12:22:50.930615+0000","last_became_peered":"2024-09-21T12:22:50.930615+0000","last_unstale":"2024-09-21T12:25:35.041376+0000","last_undegraded":"2024-09-21T12:25:35.041376+0000","last_fullsized":"2024-09-21T12:25:35.041376+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:09.649109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:09.649109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:20.855882+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":266,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.042880+0000","last_change":"2024-09-21T12:22:50.917883+0000","last_active":"2024-09-21T12:25:35.042880+0000","last_peered":"2024-09-21T12:25:35.042880+0000","last_clean":"2024-09-21T12:25:35.042880+0000","last_became_active":"2024-09-21T12:22:50.917408+0000","last_became_peered":"2024-09-21T12:22:50.917408+0000","last_unstale":"2024-09-21T12:25:35.042880+0000","last_undegraded":"2024-09-21T12:25:35.042880+0000","last_fullsized":"2024-09-21T12:25:35.042880+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:12.917010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":267,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.037544+0000","last_change":"2024-09-21T12:22:50.914633+0000","last_active":"2024-09-21T12:25:35.037544+0000","last_peered":"2024-09-21T12:25:35.037544+0000","last_clean":"2024-09-21T12:25:35.037544+0000","last_became_active":"2024-09-21T12:22:50.914221+0000","last_became_peered":"2024-09-21T12:22:50.914221+0000","last_unstale":"2024-09-21T12:25:35.037544+0000","last_undegraded":"2024-09-21T12:25:35.037544+0000","last_fullsized":"2024-09-21T12:25:35.037544+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:13.451009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":274,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.890559+0000","last_change":"2024-09-21T12:25:00.871133+0000","last_active":"2024-09-21T12:25:36.890559+0000","last_peered":"2024-09-21T12:25:36.890559+0000","last_clean":"2024-09-21T12:25:36.890559+0000","last_became_active":"2024-09-21T12:20:15.297993+0000","last_became_peered":"2024-09-21T12:20:15.297993+0000","last_unstale":"2024-09-21T12:25:36.890559+0000","last_undegraded":"2024-09-21T12:25:36.890559+0000","last_fullsized":"2024-09-21T12:25:36.890559+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:00.871062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:00.871062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:06.894610+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":272,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-21T12:25:35.041958+0000","last_change":"2024-09-21T12:20:17.299134+0000","last_active":"2024-09-21T12:25:35.041958+0000","last_peered":"2024-09-21T12:25:35.041958+0000","last_clean":"2024-09-21T12:25:35.041958+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:25:35.041958+0000","last_undegraded":"2024-09-21T12:25:35.041958+0000","last_fullsized":"2024-09-21T12:25:35.041958+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:20:16.284875+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:25:47.878711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":276,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-09-21T12:25:36.875259+0000","last_change":"2024-09-21T12:25:32.660054+0000","last_active":"2024-09-21T12:25:36.875259+0000","last_peered":"2024-09-21T12:25:36.875259+0000","last_clean":"2024-09-21T12:25:36.875259+0000","last_became_active":"2024-09-21T12:20:19.308247+0000","last_became_peered":"2024-09-21T12:20:19.308247+0000","last_unstale":"2024-09-21T12:25:36.875259+0000","last_undegraded":"2024-09-21T12:25:36.875259+0000","last_fullsized":"2024-09-21T12:25:36.875259+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:32.659918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:32.659918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:46.283244+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":4808179712,"num_objects":73367,"num_object_clones":0,"num_object_copies":220101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73367,"num_write_kb":4695488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14414512128,"data_stored":14414512128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69267,"ondisk_log_size":69267,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":690,"num_read_kb":447,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":201,"num_bytes_recovered":3982,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":396,"ondisk_log_size":396,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":5023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":316,"num_read_kb":311,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":644038,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":21,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234052,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1891632,"kb_used_data":1729948,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":91853520,"statfs":{"total":95995035648,"available":94058004480,"internally_reserved":0,"allocated":1771466752,"data_stored":1771055345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":165542346},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462988,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1919728,"kb_used_data":1735836,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":91825424,"statfs":{"total":95995035648,"available":94029234176,"internally_reserved":0,"allocated":1777496064,"data_stored":1777084448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":188283334},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724623,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2451316,"kb_used_data":2329888,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":91293836,"statfs":{"total":95995035648,"available":93484888064,"internally_reserved":0,"allocated":2385805312,"data_stored":2385391243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953559,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149789,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2315272,"kb_used_data":2036984,"kb_used_omap":1,"kb_used_meta":278270,"kb_avail":91429880,"statfs":{"total":95995035648,"available":93624197120,"internally_reserved":0,"allocated":2085871616,"data_stored":2085433380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":284948935},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411427,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2189220,"kb_used_data":2027536,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":91555932,"statfs":{"total":95995035648,"available":93753274368,"internally_reserved":0,"allocated":2076196864,"data_stored":2075797636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574953,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2204244,"kb_used_data":2014784,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":91540908,"statfs":{"total":95995035648,"available":93737889792,"internally_reserved":0,"allocated":2063138816,"data_stored":2062690864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738479,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2228020,"kb_used_data":2052452,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":91517132,"statfs":{"total":95995035648,"available":93713543168,"internally_reserved":0,"allocated":2101710848,"data_stored":2101291962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2117992448,"data_stored":2117992448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2096431104,"data_stored":2096431104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2104754176,"data_stored":2104754176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2113863680,"data_stored":2113863680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2387279872,"data_stored":2387279872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1798504448,"data_stored":1798504448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1795686400,"data_stored":1795686400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:25:39.690 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T12:25:39.691 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:25:39.691 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-21T12:25:39.691 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T12:25:39.691 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T12:25:39.691 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-09-21T12:25:39.763 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:39.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:39.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:39.865 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:39.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:39.910 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:39.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:40.011 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:40.044 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:40.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:40.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:40.046 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:25:40.048 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 46 16 77399 77383 105.109 75.1875 0.0135369 0.00950845 2024-09-21T12:25:40.146 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:40.175 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:40.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:40.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:40.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:40 smithi114 ceph-mon[40867]: from='client.25091 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:25:40.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:40 smithi114 ceph-mon[40867]: 6.4 scrub starts 2024-09-21T12:25:40.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:40 smithi114 ceph-mon[40867]: 6.4 scrub ok 2024-09-21T12:25:40.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:40 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:25:40.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:40 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:25:40.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:40 smithi114 ceph-mon[40867]: 6.f scrub starts 2024-09-21T12:25:40.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:40 smithi114 ceph-mon[40867]: 6.f scrub ok 2024-09-21T12:25:40.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:40 smithi114 ceph-mon[40867]: osdmap e183: 8 total, 8 up, 7 in 2024-09-21T12:25:40.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:40 smithi114 ceph-mon[30324]: from='client.25091 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:25:40.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:40 smithi114 ceph-mon[30324]: 6.4 scrub starts 2024-09-21T12:25:40.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:40 smithi114 ceph-mon[30324]: 6.4 scrub ok 2024-09-21T12:25:40.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:40 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:25:40.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:40 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:25:40.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:40 smithi114 ceph-mon[30324]: 6.f scrub starts 2024-09-21T12:25:40.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:40 smithi114 ceph-mon[30324]: 6.f scrub ok 2024-09-21T12:25:40.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:40 smithi114 ceph-mon[30324]: osdmap e183: 8 total, 8 up, 7 in 2024-09-21T12:25:40.276 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:40.297 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:40.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:40.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:40.399 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:40.429 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:40.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:40.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:40.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:40 smithi183 ceph-mon[33277]: from='client.25091 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:25:40.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:40 smithi183 ceph-mon[33277]: 6.4 scrub starts 2024-09-21T12:25:40.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:40 smithi183 ceph-mon[33277]: 6.4 scrub ok 2024-09-21T12:25:40.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:40 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:25:40.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:40 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:25:40.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:40 smithi183 ceph-mon[33277]: 6.f scrub starts 2024-09-21T12:25:40.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:40 smithi183 ceph-mon[33277]: 6.f scrub ok 2024-09-21T12:25:40.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:40 smithi183 ceph-mon[33277]: osdmap e183: 8 total, 8 up, 7 in 2024-09-21T12:25:40.530 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:40.564 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:40.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:40.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:40.666 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:40.688 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:40.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:40.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:40.790 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:40.827 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:40.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:40.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:40.922 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:25:40.922 INFO:teuthology.orchestra.run.smithi183.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T12:25:40.922 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:25:40.929 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:40.963 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:40.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:40.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:41.049 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 47 16 78544 78528 104.396 71.5625 0.00750435 0.00957444 2024-09-21T12:25:41.065 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:41.100 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:41.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:41.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:41.203 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:41.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:41.225 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:41.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:41.326 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:41.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:41.368 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:41.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:41.468 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:41.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:41 smithi183 ceph-mon[33277]: pgmap v258: 158 pgs: 158 active+clean; 4.5 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-21T12:25:41.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:41 smithi183 ceph-mon[33277]: 5.8 scrub starts 2024-09-21T12:25:41.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:41 smithi183 ceph-mon[33277]: 5.8 scrub ok 2024-09-21T12:25:41.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:41.504 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:41.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:41.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:41 smithi114 ceph-mon[40867]: pgmap v258: 158 pgs: 158 active+clean; 4.5 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-21T12:25:41.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:41 smithi114 ceph-mon[40867]: 5.8 scrub starts 2024-09-21T12:25:41.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:41 smithi114 ceph-mon[40867]: 5.8 scrub ok 2024-09-21T12:25:41.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:41 smithi114 ceph-mon[30324]: pgmap v258: 158 pgs: 158 active+clean; 4.5 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-21T12:25:41.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:41 smithi114 ceph-mon[30324]: 5.8 scrub starts 2024-09-21T12:25:41.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:41 smithi114 ceph-mon[30324]: 5.8 scrub ok 2024-09-21T12:25:41.605 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:41.643 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:41.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:41.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:41.745 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:41.780 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:41.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:41.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:41.881 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:41.922 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:41.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:41.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:42.024 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:42.049 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 48 16 79580 79564 103.569 64.75 0.0255733 0.00965001 2024-09-21T12:25:42.060 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:42.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:42.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:42.162 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:42.192 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:42.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:42.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:42.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:42 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1814325931' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:25:42.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:42 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:25:42.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:42 smithi114 ceph-mon[30324]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:25:42.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:42 smithi114 ceph-mon[30324]: Cluster is now healthy 2024-09-21T12:25:42.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:42 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:25:42.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:42 smithi114 ceph-mon[30324]: osdmap e184: 8 total, 8 up, 7 in 2024-09-21T12:25:42.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:42 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1814325931' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:25:42.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:42 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:25:42.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:42 smithi114 ceph-mon[40867]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:25:42.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:42 smithi114 ceph-mon[40867]: Cluster is now healthy 2024-09-21T12:25:42.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:42 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:25:42.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:42 smithi114 ceph-mon[40867]: osdmap e184: 8 total, 8 up, 7 in 2024-09-21T12:25:42.288 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-21T12:25:42.294 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:42.319 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:42.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:42.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:42.420 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:42.456 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:42.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:42.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:42.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:42 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1814325931' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:25:42.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:42 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:25:42.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:42 smithi183 ceph-mon[33277]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:25:42.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:42 smithi183 ceph-mon[33277]: Cluster is now healthy 2024-09-21T12:25:42.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:42 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:25:42.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:42 smithi183 ceph-mon[33277]: osdmap e184: 8 total, 8 up, 7 in 2024-09-21T12:25:42.557 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:42.588 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:42.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:42.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:42.689 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:42.721 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:42.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:42.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:42.822 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:42.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:42.862 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:42.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:42.963 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:42.991 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:42.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:42.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:43.025 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:25:42 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:25:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:25:43.050 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 49 16 80868 80852 103.098 80.5 0.00527844 0.00969359 2024-09-21T12:25:43.093 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:43.123 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:43.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:43.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:43.225 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:43.254 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:43.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:43.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:43.356 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:43.392 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:43.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:43.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:43.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:43 smithi183 ceph-mon[33277]: pgmap v259: 158 pgs: 158 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.5 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-21T12:25:43.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:43 smithi183 ceph-mon[33277]: 4.12 scrub starts 2024-09-21T12:25:43.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:43 smithi183 ceph-mon[33277]: 4.12 scrub ok 2024-09-21T12:25:43.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:43 smithi183 ceph-mon[33277]: 4.d scrub starts 2024-09-21T12:25:43.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:43 smithi183 ceph-mon[33277]: 4.d scrub ok 2024-09-21T12:25:43.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:43 smithi183 ceph-mon[33277]: 1.a scrub starts 2024-09-21T12:25:43.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:43 smithi183 ceph-mon[33277]: 1.a scrub ok 2024-09-21T12:25:43.493 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:43.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:43 smithi114 ceph-mon[40867]: pgmap v259: 158 pgs: 158 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.5 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-21T12:25:43.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:43 smithi114 ceph-mon[40867]: 4.12 scrub starts 2024-09-21T12:25:43.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:43 smithi114 ceph-mon[40867]: 4.12 scrub ok 2024-09-21T12:25:43.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:43 smithi114 ceph-mon[40867]: 4.d scrub starts 2024-09-21T12:25:43.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:43 smithi114 ceph-mon[40867]: 4.d scrub ok 2024-09-21T12:25:43.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:43 smithi114 ceph-mon[40867]: 1.a scrub starts 2024-09-21T12:25:43.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:43 smithi114 ceph-mon[40867]: 1.a scrub ok 2024-09-21T12:25:43.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:43 smithi114 ceph-mon[30324]: pgmap v259: 158 pgs: 158 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.5 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-21T12:25:43.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:43 smithi114 ceph-mon[30324]: 4.12 scrub starts 2024-09-21T12:25:43.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:43 smithi114 ceph-mon[30324]: 4.12 scrub ok 2024-09-21T12:25:43.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:43 smithi114 ceph-mon[30324]: 4.d scrub starts 2024-09-21T12:25:43.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:43 smithi114 ceph-mon[30324]: 4.d scrub ok 2024-09-21T12:25:43.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:43 smithi114 ceph-mon[30324]: 1.a scrub starts 2024-09-21T12:25:43.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:43 smithi114 ceph-mon[30324]: 1.a scrub ok 2024-09-21T12:25:43.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:43.529 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:43.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:43.630 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:43.676 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:43.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:43.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:43.777 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:43.802 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:43.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:43.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:43.903 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:43.928 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:43.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:43.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:44.030 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:44.049 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 50 16 82178 82162 102.673 81.875 0.0073422 0.00973459 2024-09-21T12:25:44.055 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:44.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:44.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:44.157 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:44.186 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:44.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:44.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:44.254 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:44.288 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:44.324 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:44.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:44.338 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:44.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:44.397 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:44 smithi114 ceph-mon[40867]: 5.3 scrub starts 2024-09-21T12:25:44.397 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:44 smithi114 ceph-mon[40867]: 5.3 scrub ok 2024-09-21T12:25:44.397 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:44 smithi114 ceph-mon[30324]: 5.3 scrub starts 2024-09-21T12:25:44.397 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:44 smithi114 ceph-mon[30324]: 5.3 scrub ok 2024-09-21T12:25:44.439 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:44.469 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:44.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:44.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:44.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:44 smithi183 ceph-mon[33277]: 5.3 scrub starts 2024-09-21T12:25:44.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:44 smithi183 ceph-mon[33277]: 5.3 scrub ok 2024-09-21T12:25:44.571 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:44.607 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:44.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:44.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:44.709 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:44.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:44.738 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:44.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:44.839 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:44.871 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:44.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:44.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:44.972 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:44.996 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:44.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:44.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:45.050 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 51 16 83495 83479 102.274 82.3125 0.00804241 0.0097731 2024-09-21T12:25:45.098 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:45.125 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:45.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:45.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:45.167 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:25:45.167 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:25:45.167 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:25:45.167 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:25:45.167 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:25:45.227 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:45.253 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:45.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:45.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:45.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:45 smithi114 ceph-mon[30324]: pgmap v261: 158 pgs: 158 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 752 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-09-21T12:25:45.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:45 smithi114 ceph-mon[30324]: 1.9 scrub starts 2024-09-21T12:25:45.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:45 smithi114 ceph-mon[30324]: 1.9 scrub ok 2024-09-21T12:25:45.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:45 smithi114 ceph-mon[40867]: pgmap v261: 158 pgs: 158 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 752 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-09-21T12:25:45.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:45 smithi114 ceph-mon[40867]: 1.9 scrub starts 2024-09-21T12:25:45.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:45 smithi114 ceph-mon[40867]: 1.9 scrub ok 2024-09-21T12:25:45.355 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:45.382 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:45.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:45.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:45.391 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:25:45.391 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:25:45.433 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:25:45.433 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:25:45.473 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:25:45.473 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:25:45.484 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:45.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:45 smithi183 ceph-mon[33277]: pgmap v261: 158 pgs: 158 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 752 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-09-21T12:25:45.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:45 smithi183 ceph-mon[33277]: 1.9 scrub starts 2024-09-21T12:25:45.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:45 smithi183 ceph-mon[33277]: 1.9 scrub ok 2024-09-21T12:25:45.505 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:25:45.505 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:25:45.510 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:45.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:45.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:45.543 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:25:45.544 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:25:45.575 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:25:45.575 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:25:45.610 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:25:45.611 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:25:45.612 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:45.638 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:45.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:45.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:45.644 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:25:45.644 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:25:45.740 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:45.780 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:45.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:45.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:45.882 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:45.916 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:45.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:45.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:46.017 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:46.046 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:46.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:46.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:46.050 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 52 16 84761 84745 101.829 79.125 0.0203919 0.0098154 2024-09-21T12:25:46.147 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:46.173 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T12:25:46.199 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:46.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:46.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:46.293 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T12:25:46.301 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:46.326 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:46.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:46.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:46.428 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:46.454 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:46.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:46.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:46.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:46 smithi183 ceph-mon[33277]: 3.e scrub starts 2024-09-21T12:25:46.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:46 smithi183 ceph-mon[33277]: 3.e scrub ok 2024-09-21T12:25:46.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:46 smithi183 ceph-mon[33277]: 6.1 scrub starts 2024-09-21T12:25:46.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:46 smithi183 ceph-mon[33277]: 6.1 scrub ok 2024-09-21T12:25:46.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:46 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3808281879' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:25:46.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1d", "id": [0, 3]} : dispatch 2024-09-21T12:25:46.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:46 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1d", "id": [0, 3]} : dispatch 2024-09-21T12:25:46.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:46 smithi114 ceph-mon[30324]: 3.e scrub starts 2024-09-21T12:25:46.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:46 smithi114 ceph-mon[30324]: 3.e scrub ok 2024-09-21T12:25:46.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:46 smithi114 ceph-mon[30324]: 6.1 scrub starts 2024-09-21T12:25:46.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:46 smithi114 ceph-mon[30324]: 6.1 scrub ok 2024-09-21T12:25:46.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:46 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3808281879' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:25:46.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1d", "id": [0, 3]} : dispatch 2024-09-21T12:25:46.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:46 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1d", "id": [0, 3]} : dispatch 2024-09-21T12:25:46.514 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:46 smithi114 ceph-mon[40867]: 3.e scrub starts 2024-09-21T12:25:46.514 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:46 smithi114 ceph-mon[40867]: 3.e scrub ok 2024-09-21T12:25:46.514 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:46 smithi114 ceph-mon[40867]: 6.1 scrub starts 2024-09-21T12:25:46.514 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:46 smithi114 ceph-mon[40867]: 6.1 scrub ok 2024-09-21T12:25:46.514 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:46 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3808281879' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:25:46.514 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1d", "id": [0, 3]} : dispatch 2024-09-21T12:25:46.514 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:46 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1d", "id": [0, 3]} : dispatch 2024-09-21T12:25:46.556 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:46.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:46.599 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:46.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:46.700 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:46.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:46.728 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:46.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:46.766 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T12:25:46.766 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T12:25:46.828 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:46.869 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:46.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:46.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:46.970 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:47.013 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:47.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:47.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:47.050 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 53 16 85856 85840 101.198 68.4375 0.00782427 0.00987708 2024-09-21T12:25:47.116 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:47.143 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:47.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:47.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:47.159 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:25:46 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:25:47.245 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:47.279 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:47.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:47.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:47.382 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:47.406 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:47.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:47.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:47.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:47 smithi183 ceph-mon[33277]: pgmap v262: 158 pgs: 158 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 47 MiB/s wr, 748 op/s 2024-09-21T12:25:47.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:47 smithi183 ceph-mon[33277]: 4.7 scrub starts 2024-09-21T12:25:47.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:47 smithi183 ceph-mon[33277]: 4.7 scrub ok 2024-09-21T12:25:47.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:47 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1d", "id": [0, 3]}]': finished 2024-09-21T12:25:47.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:47 smithi183 ceph-mon[33277]: osdmap e185: 8 total, 8 up, 7 in 2024-09-21T12:25:47.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:47 smithi183 ceph-mon[33277]: 5.6 scrub starts 2024-09-21T12:25:47.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:47 smithi183 ceph-mon[33277]: 5.6 scrub ok 2024-09-21T12:25:47.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:47 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:25:47.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:47 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:25:47.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:47 smithi183 ceph-mon[33277]: osdmap e186: 8 total, 8 up, 7 in 2024-09-21T12:25:47.509 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:47.521 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:47.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:47 smithi114 ceph-mon[40867]: pgmap v262: 158 pgs: 158 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 47 MiB/s wr, 748 op/s 2024-09-21T12:25:47.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:47 smithi114 ceph-mon[40867]: 4.7 scrub starts 2024-09-21T12:25:47.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:47 smithi114 ceph-mon[40867]: 4.7 scrub ok 2024-09-21T12:25:47.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:47 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1d", "id": [0, 3]}]': finished 2024-09-21T12:25:47.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:47 smithi114 ceph-mon[40867]: osdmap e185: 8 total, 8 up, 7 in 2024-09-21T12:25:47.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:47 smithi114 ceph-mon[40867]: 5.6 scrub starts 2024-09-21T12:25:47.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:47 smithi114 ceph-mon[40867]: 5.6 scrub ok 2024-09-21T12:25:47.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:47 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:25:47.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:47 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:25:47.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:47 smithi114 ceph-mon[40867]: osdmap e186: 8 total, 8 up, 7 in 2024-09-21T12:25:47.528 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:47 smithi114 ceph-mon[30324]: pgmap v262: 158 pgs: 158 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 47 MiB/s wr, 748 op/s 2024-09-21T12:25:47.529 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:47 smithi114 ceph-mon[30324]: 4.7 scrub starts 2024-09-21T12:25:47.529 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:47 smithi114 ceph-mon[30324]: 4.7 scrub ok 2024-09-21T12:25:47.529 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:47 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1d", "id": [0, 3]}]': finished 2024-09-21T12:25:47.529 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:47 smithi114 ceph-mon[30324]: osdmap e185: 8 total, 8 up, 7 in 2024-09-21T12:25:47.529 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:47 smithi114 ceph-mon[30324]: 5.6 scrub starts 2024-09-21T12:25:47.529 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:47 smithi114 ceph-mon[30324]: 5.6 scrub ok 2024-09-21T12:25:47.529 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:47 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:25:47.529 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:47 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:25:47.529 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:47 smithi114 ceph-mon[30324]: osdmap e186: 8 total, 8 up, 7 in 2024-09-21T12:25:47.529 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:47.534 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:47.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:47.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:47.562 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:47.636 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:47.663 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:47.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:47.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:47.765 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:47.806 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:47.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:47.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:47.908 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:47.933 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:47.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:47.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:48.034 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:48.050 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 54 16 87133 87117 100.802 79.8125 0.0151071 0.00991642 2024-09-21T12:25:48.061 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:48.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:48.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:48.163 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:48.188 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:48.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:48.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:48.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:48 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:25:48.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:48 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:25:48.290 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:48.323 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:48.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:48.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:48.424 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:48.461 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:48.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:48.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:48.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:48 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:25:48.562 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:48.580 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:25:48.580 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:25:48.602 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:48.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:48.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:48.704 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:48.729 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:48.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:48.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:48.830 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:48.856 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:48.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:48.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:48.958 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:48.994 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:48.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:48.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:49.050 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 55 16 88361 88345 100.364 76.75 0.0167522 0.00995896 2024-09-21T12:25:49.096 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:49.140 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:49.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:49.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:49.191 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-21T12:25:49.241 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:49.274 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:49.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:49 smithi114 ceph-mon[30324]: pgmap v265: 158 pgs: 1 peering, 157 active+clean; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-09-21T12:25:49.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:49 smithi114 ceph-mon[30324]: 4.1f scrub starts 2024-09-21T12:25:49.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:49 smithi114 ceph-mon[30324]: 4.1f scrub ok 2024-09-21T12:25:49.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:49 smithi114 ceph-mon[30324]: 6.1b scrub starts 2024-09-21T12:25:49.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:49 smithi114 ceph-mon[30324]: 6.1b scrub ok 2024-09-21T12:25:49.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:49 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2418772407' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:25:49.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:49 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:25:49.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:49 smithi114 ceph-mon[30324]: 8.5 scrub starts 2024-09-21T12:25:49.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:49 smithi114 ceph-mon[40867]: pgmap v265: 158 pgs: 1 peering, 157 active+clean; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-09-21T12:25:49.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:49 smithi114 ceph-mon[40867]: 4.1f scrub starts 2024-09-21T12:25:49.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:49 smithi114 ceph-mon[40867]: 4.1f scrub ok 2024-09-21T12:25:49.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:49 smithi114 ceph-mon[40867]: 6.1b scrub starts 2024-09-21T12:25:49.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:49 smithi114 ceph-mon[40867]: 6.1b scrub ok 2024-09-21T12:25:49.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:49 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2418772407' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:25:49.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:49 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:25:49.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:49 smithi114 ceph-mon[40867]: 8.5 scrub starts 2024-09-21T12:25:49.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:49.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:49.378 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:49.391 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:25:49.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:49.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:49.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:49 smithi183 ceph-mon[33277]: pgmap v265: 158 pgs: 1 peering, 157 active+clean; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-09-21T12:25:49.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:49 smithi183 ceph-mon[33277]: 4.1f scrub starts 2024-09-21T12:25:49.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:49 smithi183 ceph-mon[33277]: 4.1f scrub ok 2024-09-21T12:25:49.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:49 smithi183 ceph-mon[33277]: 6.1b scrub starts 2024-09-21T12:25:49.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:49 smithi183 ceph-mon[33277]: 6.1b scrub ok 2024-09-21T12:25:49.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:49 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2418772407' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:25:49.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:49 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:25:49.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:49 smithi183 ceph-mon[33277]: 8.5 scrub starts 2024-09-21T12:25:49.516 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:49.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:49.572 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:49.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:49.673 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:49.701 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:49.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:49.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:49.802 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:49.843 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:49.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:49.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:49.945 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:49.971 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:49.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:49.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:49.980 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T12:25:50.051 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 56 16 89531 89515 99.878 73.125 0.0113339 0.0100075 2024-09-21T12:25:50.073 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:50.114 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:50.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:50.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:50.213 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T12:25:50.215 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:50.266 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:50.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:50.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:50.368 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:50.373 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:25:50.409 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:50.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:50.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:50.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:50 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:25:50.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:50 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:25:50.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:50 smithi183 ceph-mon[33277]: 3.17 scrub starts 2024-09-21T12:25:50.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:50 smithi183 ceph-mon[33277]: 3.17 scrub ok 2024-09-21T12:25:50.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:50 smithi183 ceph-mon[33277]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:25:50.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:50 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:25:50.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:50 smithi183 ceph-mon[33277]: osdmap e187: 8 total, 8 up, 7 in 2024-09-21T12:25:50.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:50 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3272984309' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:25:50.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:50 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:25:50.511 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:50.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:50 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:25:50.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:50 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:25:50.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:50 smithi114 ceph-mon[30324]: 3.17 scrub starts 2024-09-21T12:25:50.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:50 smithi114 ceph-mon[30324]: 3.17 scrub ok 2024-09-21T12:25:50.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:50 smithi114 ceph-mon[30324]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:25:50.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:50 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:25:50.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:50 smithi114 ceph-mon[30324]: osdmap e187: 8 total, 8 up, 7 in 2024-09-21T12:25:50.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:50 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3272984309' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:25:50.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:50 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:25:50.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:50 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:25:50.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:50 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:25:50.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:50 smithi114 ceph-mon[40867]: 3.17 scrub starts 2024-09-21T12:25:50.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:50 smithi114 ceph-mon[40867]: 3.17 scrub ok 2024-09-21T12:25:50.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:50 smithi114 ceph-mon[40867]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:25:50.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:50 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:25:50.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:50 smithi114 ceph-mon[40867]: osdmap e187: 8 total, 8 up, 7 in 2024-09-21T12:25:50.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:50 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3272984309' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:25:50.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:50 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:25:50.547 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:50.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:50.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:50.648 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:50.687 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:50.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:50.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:50.789 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:50.821 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:50.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:50.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:50.923 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:50.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:50.960 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:50.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:50.975 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:25:50.975 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:25:51.051 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 57 16 90874 90858 99.5983 83.9375 0.00454932 0.0100354 2024-09-21T12:25:51.061 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:51.092 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:51.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:51.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:51.194 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:51.228 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:51.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:51.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:51.300 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:51.331 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:51.384 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:51.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:51.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:51.486 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:51.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:51 smithi183 ceph-mon[33277]: pgmap v267: 158 pgs: 1 peering, 157 active+clean; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-21T12:25:51.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:51 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T12:25:51.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:51 smithi183 ceph-mon[33277]: osdmap e188: 8 total, 8 up, 7 in 2024-09-21T12:25:51.519 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:51.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:51.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:51.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:51 smithi114 ceph-mon[30324]: pgmap v267: 158 pgs: 1 peering, 157 active+clean; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-21T12:25:51.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:51 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T12:25:51.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:51 smithi114 ceph-mon[30324]: osdmap e188: 8 total, 8 up, 7 in 2024-09-21T12:25:51.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:51 smithi114 ceph-mon[40867]: pgmap v267: 158 pgs: 1 peering, 157 active+clean; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-21T12:25:51.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:51 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T12:25:51.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:51 smithi114 ceph-mon[40867]: osdmap e188: 8 total, 8 up, 7 in 2024-09-21T12:25:51.621 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:51.655 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:51.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:51.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:51.674 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T12:25:51.757 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:51.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:51.791 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:51.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:51.874 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T12:25:51.891 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:51.921 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:51.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:51.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:52.022 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:52.033 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:25:52.051 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 58 16 92197 92181 99.3064 82.6875 0.0155229 0.0100648 2024-09-21T12:25:52.053 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:52.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:52.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:52.154 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:52.180 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:52.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:52.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:52.282 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:52.312 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:52.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:52.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:52.414 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:52.454 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:52.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:52.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:52.519 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:25:52.519 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:25:52.535 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:52.556 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:52.562 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:25:52.562 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:25:52.590 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:25:52.590 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:25:52.594 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:25:52.594 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:25:52.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:52.606 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:52.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:52.632 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:25:52.632 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:25:52.662 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:25:52.662 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:25:52.698 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:25:52.699 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:25:52.699 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:25:52 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:25:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:25:52.707 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:52.730 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:25:52.731 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:25:52.751 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:52.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:52.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:52.761 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:25:52.761 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:25:52.853 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:52.889 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:52.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:52.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:52.990 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:53.024 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:53.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:53.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:53.051 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 59 16 93375 93359 98.8711 73.625 0.00738701 0.0101089 2024-09-21T12:25:53.125 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:53.162 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:53.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:53.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:53.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:53 smithi183 ceph-mon[33277]: pgmap v269: 158 pgs: 1 peering, 157 active+clean; 5.4 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-09-21T12:25:53.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:53 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2434853443' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:25:53.264 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:53.287 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T12:25:53.298 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:53.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:53.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:53.301 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T12:25:53.400 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:53.426 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:53.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:53.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:53.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:53 smithi114 ceph-mon[30324]: pgmap v269: 158 pgs: 1 peering, 157 active+clean; 5.4 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-09-21T12:25:53.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:53 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2434853443' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:25:53.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:53 smithi114 ceph-mon[40867]: pgmap v269: 158 pgs: 1 peering, 157 active+clean; 5.4 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-09-21T12:25:53.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:53 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2434853443' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:25:53.528 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:53.577 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:53.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:53.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:53.628 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:25:53.678 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:53.723 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:53.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:53.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:53.825 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:53.853 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:53.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:53.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:53.956 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:53.983 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:53.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:53.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:54.053 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-21T12:25:54.052343+0000 min lat: 0.00121483 max lat: 0.158038 avg lat: 0.0101528 2024-09-21T12:25:54.053 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T12:25:54.053 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 60 16 94558 94542 98.453 73.9375 0.0157556 0.0101528 2024-09-21T12:25:54.084 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:54.085 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:54.117 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:54.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:54.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:54.154 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:25:54.154 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-21T12:25:54.155 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-21T12:25:54.155 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:25:54.155 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.155 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:23884 7.1 7:8f148689:::benchmark_data_smithi114_78432_object23883:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T12:25:54.155 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:39.695870+0000", 2024-09-21T12:25:54.155 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 254.453713373, 2024-09-21T12:25:54.155 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.039559285999999999, 2024-09-21T12:25:54.155 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:54.155 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:54.155 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:54.155 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:25:54.156 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:25:54.156 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:25:54.156 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 23884 2024-09-21T12:25:54.156 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.156 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:54.156 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.156 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:54.156 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695870+0000", 2024-09-21T12:25:54.156 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.156 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.156 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.156 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:54.156 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695870+0000", 2024-09-21T12:25:54.156 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.156 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.157 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.157 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:54.157 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695873+0000", 2024-09-21T12:25:54.157 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0090000000000002e-06 2024-09-21T12:25:54.157 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.157 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.157 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:54.157 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695927+0000", 2024-09-21T12:25:54.157 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4941000000000001e-05 2024-09-21T12:25:54.157 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.157 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.158 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:54.158 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695955+0000", 2024-09-21T12:25:54.158 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7120000000000001e-05 2024-09-21T12:25:54.158 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.158 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.158 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:54.158 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695963+0000", 2024-09-21T12:25:54.158 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7760000000000003e-06 2024-09-21T12:25:54.158 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.158 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.158 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:54.158 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696022+0000", 2024-09-21T12:25:54.158 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8941999999999999e-05 2024-09-21T12:25:54.159 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.159 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.159 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:54.159 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696107+0000", 2024-09-21T12:25:54.159 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4853999999999998e-05 2024-09-21T12:25:54.159 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.159 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.159 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,6", 2024-09-21T12:25:54.159 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696202+0000", 2024-09-21T12:25:54.159 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4889999999999994e-05 2024-09-21T12:25:54.159 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.159 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.159 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:25:54.159 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.698132+0000", 2024-09-21T12:25:54.160 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001929998 2024-09-21T12:25:54.160 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.160 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.160 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:54.160 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.698167+0000", 2024-09-21T12:25:54.160 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5247000000000003e-05 2024-09-21T12:25:54.160 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.160 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.160 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:54.160 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.735322+0000", 2024-09-21T12:25:54.160 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.037154514 2024-09-21T12:25:54.160 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.160 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.160 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:54.161 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.735382+0000", 2024-09-21T12:25:54.161 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0371000000000002e-05 2024-09-21T12:25:54.161 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.161 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.161 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:54.161 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.735430+0000", 2024-09-21T12:25:54.161 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7623999999999997e-05 2024-09-21T12:25:54.161 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.161 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:54.161 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.161 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.162 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.162 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:23885 7.1 7:81ccbda1:::benchmark_data_smithi114_78432_object23884:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T12:25:54.162 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:39.696116+0000", 2024-09-21T12:25:54.162 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 254.45346754400001, 2024-09-21T12:25:54.162 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.040219576, 2024-09-21T12:25:54.162 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:54.162 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:54.162 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:54.162 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:25:54.162 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:25:54.162 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:25:54.162 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 23885 2024-09-21T12:25:54.163 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.163 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:54.163 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.163 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:54.163 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696116+0000", 2024-09-21T12:25:54.163 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.163 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.163 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.163 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:54.163 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696116+0000", 2024-09-21T12:25:54.163 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.163 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.163 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.164 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:54.164 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696117+0000", 2024-09-21T12:25:54.164 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0109999999999999e-06 2024-09-21T12:25:54.164 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.164 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.164 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:54.164 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696178+0000", 2024-09-21T12:25:54.164 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1060000000000002e-05 2024-09-21T12:25:54.164 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.164 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.164 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:54.164 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696188+0000", 2024-09-21T12:25:54.164 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0067e-05 2024-09-21T12:25:54.164 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.165 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.165 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:54.165 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696196+0000", 2024-09-21T12:25:54.165 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0740000000000004e-06 2024-09-21T12:25:54.165 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.165 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.165 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:54.165 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.697126+0000", 2024-09-21T12:25:54.165 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00093067599999999996 2024-09-21T12:25:54.165 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.165 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.165 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:54.165 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.697270+0000", 2024-09-21T12:25:54.165 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000143502 2024-09-21T12:25:54.166 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.166 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.166 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,6", 2024-09-21T12:25:54.166 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.697418+0000", 2024-09-21T12:25:54.166 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000147892 2024-09-21T12:25:54.166 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.166 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.166 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:25:54.166 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.698242+0000", 2024-09-21T12:25:54.166 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00082411799999999999 2024-09-21T12:25:54.166 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.166 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.166 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:54.167 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.698485+0000", 2024-09-21T12:25:54.167 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00024341099999999999 2024-09-21T12:25:54.167 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.167 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.167 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:54.167 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.736250+0000", 2024-09-21T12:25:54.167 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.037765265999999999 2024-09-21T12:25:54.167 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.167 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.167 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:54.167 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.736300+0000", 2024-09-21T12:25:54.167 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9886000000000003e-05 2024-09-21T12:25:54.168 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.168 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.168 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:54.168 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.736336+0000", 2024-09-21T12:25:54.168 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5612999999999999e-05 2024-09-21T12:25:54.168 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.168 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:54.168 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.168 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.168 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.168 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:81802 7.8 e115/112 7:110c0c7a:::benchmark_data_smithi114_78432_object81801:head v 115'5198, mlcod=88'242)", 2024-09-21T12:25:54.168 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.137865+0000", 2024-09-21T12:25:54.168 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 199.01171865699999, 2024-09-21T12:25:54.168 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.051160639000000001, 2024-09-21T12:25:54.169 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:54.169 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:54.169 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:54.169 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:54.169 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.169 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:54.169 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137865+0000", 2024-09-21T12:25:54.169 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.169 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.169 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.169 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:54.169 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137865+0000", 2024-09-21T12:25:54.169 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.169 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.170 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.170 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:54.170 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137866+0000", 2024-09-21T12:25:54.170 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5099999999999998e-07 2024-09-21T12:25:54.170 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.170 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.170 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:54.170 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137909+0000", 2024-09-21T12:25:54.170 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2545e-05 2024-09-21T12:25:54.170 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.170 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.170 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:54.171 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137938+0000", 2024-09-21T12:25:54.171 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9737999999999999e-05 2024-09-21T12:25:54.171 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.171 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.171 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:54.171 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137961+0000", 2024-09-21T12:25:54.171 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2320999999999999e-05 2024-09-21T12:25:54.171 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.171 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.171 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:54.171 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.138026+0000", 2024-09-21T12:25:54.171 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5571e-05 2024-09-21T12:25:54.172 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.172 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.172 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:54.172 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.138111+0000", 2024-09-21T12:25:54.172 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4682e-05 2024-09-21T12:25:54.172 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.172 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.172 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:54.172 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.188940+0000", 2024-09-21T12:25:54.172 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.050828867 2024-09-21T12:25:54.172 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.172 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.172 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:54.172 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189026+0000", 2024-09-21T12:25:54.172 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6063999999999995e-05 2024-09-21T12:25:54.173 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.173 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:54.173 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.173 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.173 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.173 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:81809 7.8 e115/112 7:1042b23e:::benchmark_data_smithi114_78432_object81808:head v 115'5199, mlcod=88'242)", 2024-09-21T12:25:54.173 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.140023+0000", 2024-09-21T12:25:54.173 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 199.00956108599999, 2024-09-21T12:25:54.173 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.049109005999999997, 2024-09-21T12:25:54.173 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:54.173 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:54.173 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:54.173 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:54.173 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.173 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:54.173 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140023+0000", 2024-09-21T12:25:54.174 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.174 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.174 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.174 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:54.174 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140023+0000", 2024-09-21T12:25:54.174 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.174 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.174 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.174 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:54.174 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140023+0000", 2024-09-21T12:25:54.174 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6700000000000003e-07 2024-09-21T12:25:54.174 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.174 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.174 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:54.175 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140136+0000", 2024-09-21T12:25:54.175 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000112623 2024-09-21T12:25:54.175 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.175 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.175 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:54.175 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140146+0000", 2024-09-21T12:25:54.175 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0098999999999999e-05 2024-09-21T12:25:54.175 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.175 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.175 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:54.175 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140161+0000", 2024-09-21T12:25:54.175 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5288e-05 2024-09-21T12:25:54.175 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.175 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.176 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:54.176 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146087+0000", 2024-09-21T12:25:54.176 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0059259070000000002 2024-09-21T12:25:54.176 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.176 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.176 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:54.176 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146109+0000", 2024-09-21T12:25:54.176 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2045999999999999e-05 2024-09-21T12:25:54.176 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.176 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.176 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:54.176 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189077+0000", 2024-09-21T12:25:54.176 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.042967639000000002 2024-09-21T12:25:54.176 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.176 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.177 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:54.177 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189132+0000", 2024-09-21T12:25:54.177 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4737000000000001e-05 2024-09-21T12:25:54.177 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.177 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:54.177 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.177 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.177 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.177 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:81811 7.8 e115/112 7:10044f1a:::benchmark_data_smithi114_78432_object81810:head v 115'5200, mlcod=88'242)", 2024-09-21T12:25:54.177 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.146782+0000", 2024-09-21T12:25:54.177 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 199.00280160099999, 2024-09-21T12:25:54.177 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.042406829, 2024-09-21T12:25:54.177 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:54.177 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:54.178 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:54.178 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:54.178 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.178 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:54.178 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146782+0000", 2024-09-21T12:25:54.178 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.178 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.178 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.178 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:54.178 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146782+0000", 2024-09-21T12:25:54.178 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.178 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.178 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.178 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:54.178 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146783+0000", 2024-09-21T12:25:54.179 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2999999999999999e-07 2024-09-21T12:25:54.179 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.179 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.179 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:54.179 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146826+0000", 2024-09-21T12:25:54.179 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2781000000000003e-05 2024-09-21T12:25:54.179 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.179 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.179 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:54.179 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146837+0000", 2024-09-21T12:25:54.179 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1126e-05 2024-09-21T12:25:54.179 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.180 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.180 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:54.180 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146843+0000", 2024-09-21T12:25:54.180 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4710000000000004e-06 2024-09-21T12:25:54.180 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.180 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.180 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:54.180 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.152740+0000", 2024-09-21T12:25:54.180 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0058961209999999998 2024-09-21T12:25:54.180 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.180 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.180 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:54.180 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.152763+0000", 2024-09-21T12:25:54.180 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3101000000000001e-05 2024-09-21T12:25:54.180 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.180 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.181 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:54.181 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189134+0000", 2024-09-21T12:25:54.181 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.036371542999999999 2024-09-21T12:25:54.181 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.181 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.181 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:54.181 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189189+0000", 2024-09-21T12:25:54.181 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4855999999999998e-05 2024-09-21T12:25:54.181 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.181 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:54.181 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.181 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.181 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.181 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:81821 7.8 e115/112 7:1c79dadc:::benchmark_data_smithi114_78432_object81820:head v 115'5201, mlcod=88'242)", 2024-09-21T12:25:54.181 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.148399+0000", 2024-09-21T12:25:54.182 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 199.001184526, 2024-09-21T12:25:54.182 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.040843988999999997, 2024-09-21T12:25:54.182 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:54.182 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:54.182 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:54.182 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:54.182 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.182 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:54.182 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148399+0000", 2024-09-21T12:25:54.182 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.182 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.182 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.182 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:54.182 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148399+0000", 2024-09-21T12:25:54.183 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.183 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.183 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.183 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:54.183 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148400+0000", 2024-09-21T12:25:54.183 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.049e-06 2024-09-21T12:25:54.184 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.184 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.184 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:54.184 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148502+0000", 2024-09-21T12:25:54.184 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000101542 2024-09-21T12:25:54.184 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.184 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.184 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:54.184 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148512+0000", 2024-09-21T12:25:54.184 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0386e-05 2024-09-21T12:25:54.184 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.185 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.185 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:54.185 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148519+0000", 2024-09-21T12:25:54.185 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3589999999999996e-06 2024-09-21T12:25:54.185 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.185 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.185 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:54.185 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.158203+0000", 2024-09-21T12:25:54.185 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0096844710000000001 2024-09-21T12:25:54.185 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.185 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.185 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:54.185 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.158229+0000", 2024-09-21T12:25:54.185 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6078999999999999e-05 2024-09-21T12:25:54.186 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.186 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.186 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:54.186 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189192+0000", 2024-09-21T12:25:54.186 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.030962455 2024-09-21T12:25:54.186 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.186 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.186 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:54.186 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189243+0000", 2024-09-21T12:25:54.186 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1647999999999997e-05 2024-09-21T12:25:54.186 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.186 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:54.186 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.187 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.187 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.187 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82430 7.8 e115/112 7:1762e17c:::benchmark_data_smithi114_78432_object82429:head v 115'5247, mlcod=88'255)", 2024-09-21T12:25:54.187 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.787623+0000", 2024-09-21T12:25:54.187 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 198.36196073799999, 2024-09-21T12:25:54.187 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.048511400000000003, 2024-09-21T12:25:54.187 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:54.187 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:54.187 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:54.187 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:54.187 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.187 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:54.187 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787623+0000", 2024-09-21T12:25:54.188 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.188 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.188 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.188 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:54.188 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787623+0000", 2024-09-21T12:25:54.188 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.188 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.188 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.188 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:54.188 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787625+0000", 2024-09-21T12:25:54.188 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5319999999999999e-06 2024-09-21T12:25:54.189 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.189 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.189 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:54.189 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787667+0000", 2024-09-21T12:25:54.189 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1876999999999999e-05 2024-09-21T12:25:54.189 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.189 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.189 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:54.189 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787688+0000", 2024-09-21T12:25:54.189 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1046000000000001e-05 2024-09-21T12:25:54.189 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.189 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.189 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:54.189 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787697+0000", 2024-09-21T12:25:54.190 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4930000000000004e-06 2024-09-21T12:25:54.190 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.190 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.190 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:54.190 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.788378+0000", 2024-09-21T12:25:54.190 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00068115999999999999 2024-09-21T12:25:54.190 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.190 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.190 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:54.190 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.788427+0000", 2024-09-21T12:25:54.190 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8553000000000001e-05 2024-09-21T12:25:54.190 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.190 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.191 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:54.191 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.836011+0000", 2024-09-21T12:25:54.191 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.047583978999999998 2024-09-21T12:25:54.191 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.191 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.191 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:54.191 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.836135+0000", 2024-09-21T12:25:54.191 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012375999999999999 2024-09-21T12:25:54.191 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.191 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:54.191 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.191 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.192 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.192 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDPGPush(7.8 115/112 [PushOp(7:10df046b:::benchmark_data_smithi114_78432_object3988:head, version: 88'257, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:10df046b:::benchmark_data_smithi114_78432_object3988:head@88'257, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T12:25:54.192 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.787722+0000", 2024-09-21T12:25:54.192 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 198.36186166300001, 2024-09-21T12:25:54.192 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.074165650999999999, 2024-09-21T12:25:54.192 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:54.192 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:54.192 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:25:54.192 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:54.192 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.192 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:54.192 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787722+0000", 2024-09-21T12:25:54.192 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.192 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.193 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.193 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:54.193 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787722+0000", 2024-09-21T12:25:54.193 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.193 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.193 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.193 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:54.193 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787723+0000", 2024-09-21T12:25:54.193 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0499999999999999e-07 2024-09-21T12:25:54.193 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.193 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.193 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:54.193 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787779+0000", 2024-09-21T12:25:54.193 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6053e-05 2024-09-21T12:25:54.194 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.194 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.194 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:54.194 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787803+0000", 2024-09-21T12:25:54.194 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3980000000000001e-05 2024-09-21T12:25:54.194 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.194 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.194 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:54.194 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787832+0000", 2024-09-21T12:25:54.194 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.942e-05 2024-09-21T12:25:54.194 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.195 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.195 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:54.195 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.859697+0000", 2024-09-21T12:25:54.195 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.071864649000000003 2024-09-21T12:25:54.195 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.195 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.195 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:54.195 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.859710+0000", 2024-09-21T12:25:54.195 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3740000000000001e-05 2024-09-21T12:25:54.195 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.195 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.195 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:54.195 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.861888+0000", 2024-09-21T12:25:54.195 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002177404 2024-09-21T12:25:54.196 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.196 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:54.196 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.196 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.196 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.196 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82442 7.8 e115/112 7:1035456a:::benchmark_data_smithi114_78432_object82441:head v 115'5248, mlcod=88'255)", 2024-09-21T12:25:54.196 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.791117+0000", 2024-09-21T12:25:54.196 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 198.35846676, 2024-09-21T12:25:54.196 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.051248759999999997, 2024-09-21T12:25:54.196 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:54.196 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:54.196 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:54.196 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:54.196 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.197 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:54.197 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791117+0000", 2024-09-21T12:25:54.197 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.197 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.197 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.197 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:54.197 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791117+0000", 2024-09-21T12:25:54.197 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.197 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.197 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.197 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:54.197 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791118+0000", 2024-09-21T12:25:54.197 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9999999999999998e-07 2024-09-21T12:25:54.197 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.197 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.198 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:54.198 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791158+0000", 2024-09-21T12:25:54.198 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0025999999999999e-05 2024-09-21T12:25:54.198 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.198 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.198 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:54.198 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791167+0000", 2024-09-21T12:25:54.198 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9910000000000008e-06 2024-09-21T12:25:54.198 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.198 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.198 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:54.198 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791176+0000", 2024-09-21T12:25:54.198 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0329999999999997e-06 2024-09-21T12:25:54.198 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.199 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.199 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:54.199 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.794825+0000", 2024-09-21T12:25:54.199 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0036494700000000001 2024-09-21T12:25:54.199 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.199 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.199 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:54.199 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.794878+0000", 2024-09-21T12:25:54.199 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3292999999999999e-05 2024-09-21T12:25:54.199 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.199 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.199 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:54.199 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842289+0000", 2024-09-21T12:25:54.199 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.047410256999999997 2024-09-21T12:25:54.200 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.200 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.200 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:54.200 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842366+0000", 2024-09-21T12:25:54.200 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7189999999999998e-05 2024-09-21T12:25:54.200 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.200 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:54.200 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.200 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.200 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.200 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82447 7.8 e115/112 7:174332ca:::benchmark_data_smithi114_78432_object82446:head v 115'5249, mlcod=88'255)", 2024-09-21T12:25:54.200 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.793874+0000", 2024-09-21T12:25:54.200 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 198.35571030899999, 2024-09-21T12:25:54.200 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.048541015999999999, 2024-09-21T12:25:54.200 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:54.201 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:54.201 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:54.201 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:54.201 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.201 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:54.201 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793874+0000", 2024-09-21T12:25:54.201 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.201 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.201 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.202 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:54.202 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793874+0000", 2024-09-21T12:25:54.202 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.202 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.202 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.202 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:54.203 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793875+0000", 2024-09-21T12:25:54.203 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0890000000000001e-06 2024-09-21T12:25:54.203 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.203 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.203 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:54.203 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793910+0000", 2024-09-21T12:25:54.203 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5080000000000003e-05 2024-09-21T12:25:54.204 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.204 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.204 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:54.204 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793921+0000", 2024-09-21T12:25:54.204 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1284000000000001e-05 2024-09-21T12:25:54.204 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.204 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.205 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:54.205 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793930+0000", 2024-09-21T12:25:54.205 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5429999999999994e-06 2024-09-21T12:25:54.205 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.205 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.205 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:54.205 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.801012+0000", 2024-09-21T12:25:54.206 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0070825219999999999 2024-09-21T12:25:54.206 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.206 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.206 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:54.206 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.801039+0000", 2024-09-21T12:25:54.206 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7248999999999999e-05 2024-09-21T12:25:54.207 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.207 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.207 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:54.207 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842368+0000", 2024-09-21T12:25:54.207 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.041328831000000003 2024-09-21T12:25:54.207 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.208 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.208 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:54.208 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842415+0000", 2024-09-21T12:25:54.208 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6418000000000002e-05 2024-09-21T12:25:54.208 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.208 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:54.208 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.209 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.209 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.209 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82448 7.8 e115/112 7:11700dea:::benchmark_data_smithi114_78432_object82447:head v 115'5250, mlcod=88'255)", 2024-09-21T12:25:54.209 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.796531+0000", 2024-09-21T12:25:54.209 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 198.35305256500001, 2024-09-21T12:25:54.209 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.045928499999999997, 2024-09-21T12:25:54.210 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:54.210 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:54.210 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:54.210 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:54.210 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.210 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:54.211 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796531+0000", 2024-09-21T12:25:54.211 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.211 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.211 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.211 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:54.211 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796531+0000", 2024-09-21T12:25:54.212 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.212 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.212 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.212 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:54.212 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796532+0000", 2024-09-21T12:25:54.212 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7199999999999999e-07 2024-09-21T12:25:54.213 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.213 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.213 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:54.213 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796647+0000", 2024-09-21T12:25:54.213 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000115097 2024-09-21T12:25:54.213 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.213 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.214 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:54.214 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796660+0000", 2024-09-21T12:25:54.214 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3271999999999999e-05 2024-09-21T12:25:54.214 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.214 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.214 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:54.215 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796666+0000", 2024-09-21T12:25:54.215 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0170000000000002e-06 2024-09-21T12:25:54.215 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.215 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.215 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:54.215 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.809350+0000", 2024-09-21T12:25:54.216 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.012683379 2024-09-21T12:25:54.216 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.216 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.216 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:54.216 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.809374+0000", 2024-09-21T12:25:54.216 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4417e-05 2024-09-21T12:25:54.216 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.217 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.217 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:54.217 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842417+0000", 2024-09-21T12:25:54.217 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.033042646000000002 2024-09-21T12:25:54.217 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.217 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.218 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:54.218 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842460+0000", 2024-09-21T12:25:54.218 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3099999999999997e-05 2024-09-21T12:25:54.218 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.218 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:54.218 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.218 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.219 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.219 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82473 7.8 e115/112 7:15016882:::benchmark_data_smithi114_78432_object82472:head v 115'5254, mlcod=88'255)", 2024-09-21T12:25:54.219 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.818965+0000", 2024-09-21T12:25:54.219 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 198.330618972, 2024-09-21T12:25:54.219 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.045024287000000003, 2024-09-21T12:25:54.219 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:54.220 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:54.220 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:54.220 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:54.220 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.220 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:54.220 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.818965+0000", 2024-09-21T12:25:54.220 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.221 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.221 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.221 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:54.221 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.818965+0000", 2024-09-21T12:25:54.221 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.221 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.222 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.222 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:54.222 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.818966+0000", 2024-09-21T12:25:54.222 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9199999999999999e-07 2024-09-21T12:25:54.222 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.222 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.222 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:54.223 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.819001+0000", 2024-09-21T12:25:54.223 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5117999999999998e-05 2024-09-21T12:25:54.223 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.223 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.223 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:54.224 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.819014+0000", 2024-09-21T12:25:54.224 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2869e-05 2024-09-21T12:25:54.224 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.224 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.224 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:54.224 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.819020+0000", 2024-09-21T12:25:54.224 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6959999999999999e-06 2024-09-21T12:25:54.225 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.225 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.225 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:54.225 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.836164+0000", 2024-09-21T12:25:54.225 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.017143617 2024-09-21T12:25:54.225 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.225 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.226 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:54.226 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.836188+0000", 2024-09-21T12:25:54.226 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4074e-05 2024-09-21T12:25:54.226 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.226 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.226 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:54.227 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.863879+0000", 2024-09-21T12:25:54.227 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.027690392000000001 2024-09-21T12:25:54.227 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.227 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.227 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:54.227 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.863989+0000", 2024-09-21T12:25:54.227 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000110629 2024-09-21T12:25:54.227 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.228 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:54.228 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.228 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.228 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.228 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82539 7.8 e115/112 7:1cd4c60d:::benchmark_data_smithi114_78432_object82538:head v 115'5258, mlcod=88'256)", 2024-09-21T12:25:54.228 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.889845+0000", 2024-09-21T12:25:54.228 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 198.25973856900001, 2024-09-21T12:25:54.228 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.046325071000000002, 2024-09-21T12:25:54.228 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:54.228 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:54.228 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:54.228 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:54.228 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.229 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:54.229 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889845+0000", 2024-09-21T12:25:54.229 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.229 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.229 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.229 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:54.229 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889845+0000", 2024-09-21T12:25:54.229 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.229 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.229 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.229 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:54.229 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889846+0000", 2024-09-21T12:25:54.230 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1800000000000004e-07 2024-09-21T12:25:54.230 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.230 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.230 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:54.230 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889878+0000", 2024-09-21T12:25:54.230 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1619999999999999e-05 2024-09-21T12:25:54.230 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.230 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.230 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:54.230 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889893+0000", 2024-09-21T12:25:54.230 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4819e-05 2024-09-21T12:25:54.230 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.230 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.231 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:54.231 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889900+0000", 2024-09-21T12:25:54.231 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8510000000000001e-06 2024-09-21T12:25:54.231 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.231 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.231 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:54.231 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.898101+0000", 2024-09-21T12:25:54.231 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0082018119999999993 2024-09-21T12:25:54.231 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.231 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.231 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:54.231 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.898164+0000", 2024-09-21T12:25:54.231 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2756999999999996e-05 2024-09-21T12:25:54.231 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.232 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.232 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:54.232 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.936000+0000", 2024-09-21T12:25:54.232 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.037835720000000003 2024-09-21T12:25:54.232 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.232 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.232 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:54.232 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.936170+0000", 2024-09-21T12:25:54.232 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000170574 2024-09-21T12:25:54.232 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.232 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:54.232 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.232 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.233 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.233 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82545 7.8 e115/112 7:1ddea6d1:::benchmark_data_smithi114_78432_object82544:head v 115'5259, mlcod=88'256)", 2024-09-21T12:25:54.233 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.897417+0000", 2024-09-21T12:25:54.233 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 198.25216703199999, 2024-09-21T12:25:54.233 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.038827027, 2024-09-21T12:25:54.233 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:54.233 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:54.233 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:54.233 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:54.233 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.233 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:54.233 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897417+0000", 2024-09-21T12:25:54.233 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.234 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.234 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.234 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:54.234 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897417+0000", 2024-09-21T12:25:54.234 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.234 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.234 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.234 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:54.234 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897418+0000", 2024-09-21T12:25:54.234 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3500000000000005e-07 2024-09-21T12:25:54.234 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.234 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.234 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:54.235 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897462+0000", 2024-09-21T12:25:54.235 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4036999999999999e-05 2024-09-21T12:25:54.235 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.235 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.235 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:54.235 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897472+0000", 2024-09-21T12:25:54.235 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0669999999999999e-05 2024-09-21T12:25:54.235 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.236 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.236 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:54.236 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897479+0000", 2024-09-21T12:25:54.236 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9190000000000002e-06 2024-09-21T12:25:54.236 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.236 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.236 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:54.236 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.905524+0000", 2024-09-21T12:25:54.236 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0080444899999999996 2024-09-21T12:25:54.236 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.236 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.236 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:54.237 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.905597+0000", 2024-09-21T12:25:54.237 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3539000000000002e-05 2024-09-21T12:25:54.237 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.237 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.237 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:54.238 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.936176+0000", 2024-09-21T12:25:54.238 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.030578893999999999 2024-09-21T12:25:54.238 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.238 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.238 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:54.238 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.936244+0000", 2024-09-21T12:25:54.238 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7543e-05 2024-09-21T12:25:54.238 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.238 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:54.238 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.238 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.238 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.238 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:83346 7.8 e115/112 7:1517ab05:::benchmark_data_smithi114_78432_object83345:head v 115'5319, mlcod=88'282)", 2024-09-21T12:25:54.239 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:36.700133+0000", 2024-09-21T12:25:54.239 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 197.44945113099999, 2024-09-21T12:25:54.239 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.044693733999999999, 2024-09-21T12:25:54.239 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:54.239 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:54.239 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:54.239 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:54.239 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.239 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:54.239 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700133+0000", 2024-09-21T12:25:54.239 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.239 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.239 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.240 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:54.240 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700133+0000", 2024-09-21T12:25:54.240 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.240 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.240 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.240 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:54.240 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700133+0000", 2024-09-21T12:25:54.240 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4300000000000003e-07 2024-09-21T12:25:54.240 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.240 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.240 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:54.240 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700182+0000", 2024-09-21T12:25:54.240 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8433000000000002e-05 2024-09-21T12:25:54.240 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.241 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.241 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:54.241 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700192+0000", 2024-09-21T12:25:54.241 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0382e-05 2024-09-21T12:25:54.241 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.241 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.241 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:54.241 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:54.243 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T12:25:54.243 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700201+0000", 2024-09-21T12:25:54.243 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9819999999999997e-06 2024-09-21T12:25:54.243 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.243 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.243 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:54.244 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.703063+0000", 2024-09-21T12:25:54.244 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002862121 2024-09-21T12:25:54.244 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.244 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.244 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:54.244 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.703087+0000", 2024-09-21T12:25:54.244 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3730000000000001e-05 2024-09-21T12:25:54.244 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.244 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.244 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:54.244 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.744677+0000", 2024-09-21T12:25:54.244 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.041589785999999997 2024-09-21T12:25:54.244 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.245 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.245 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:54.245 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.744826+0000", 2024-09-21T12:25:54.245 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014965700000000001 2024-09-21T12:25:54.245 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.245 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:54.245 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.245 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.245 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.245 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:87464 7.8 e117/112 7:17a0a89d:::benchmark_data_smithi114_78432_object87463:head v 117'5593, mlcod=88'417)", 2024-09-21T12:25:54.246 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:41.463661+0000", 2024-09-21T12:25:54.246 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 192.68592319800001, 2024-09-21T12:25:54.246 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.067076070000000002, 2024-09-21T12:25:54.246 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:54.246 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:54.246 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:54.247 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:54.248 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.248 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:54.248 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463661+0000", 2024-09-21T12:25:54.248 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.248 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.248 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.248 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:54.248 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463661+0000", 2024-09-21T12:25:54.248 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.248 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.248 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.248 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:54.248 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463661+0000", 2024-09-21T12:25:54.249 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2799999999999995e-07 2024-09-21T12:25:54.249 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.249 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.249 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:54.249 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463845+0000", 2024-09-21T12:25:54.249 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000183692 2024-09-21T12:25:54.249 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.249 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.249 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:54.249 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463854+0000", 2024-09-21T12:25:54.249 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0750000000000004e-06 2024-09-21T12:25:54.249 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.250 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.250 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:54.250 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463860+0000", 2024-09-21T12:25:54.250 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.015e-06 2024-09-21T12:25:54.250 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.250 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.250 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:54.250 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.480218+0000", 2024-09-21T12:25:54.250 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.016357289000000001 2024-09-21T12:25:54.250 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.251 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.251 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:54.251 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.480239+0000", 2024-09-21T12:25:54.251 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1691999999999998e-05 2024-09-21T12:25:54.251 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.251 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.251 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:54.251 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.530695+0000", 2024-09-21T12:25:54.251 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.050456050000000002 2024-09-21T12:25:54.251 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.251 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.251 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:54.251 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.530737+0000", 2024-09-21T12:25:54.252 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1428999999999999e-05 2024-09-21T12:25:54.252 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.252 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:54.252 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.252 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.252 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.252 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:87470 7.8 e117/112 7:1f3b224f:::benchmark_data_smithi114_78432_object87469:head v 117'5594, mlcod=88'417)", 2024-09-21T12:25:54.252 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:41.472523+0000", 2024-09-21T12:25:54.252 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 192.6770612, 2024-09-21T12:25:54.252 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.06343451, 2024-09-21T12:25:54.252 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:54.252 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:54.252 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:54.252 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:54.253 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.253 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:54.253 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472523+0000", 2024-09-21T12:25:54.253 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.253 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.253 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.253 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:54.253 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472523+0000", 2024-09-21T12:25:54.253 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.253 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.253 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.253 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:54.253 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472524+0000", 2024-09-21T12:25:54.253 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7399999999999991e-07 2024-09-21T12:25:54.254 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.254 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.254 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:54.254 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472596+0000", 2024-09-21T12:25:54.254 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2152999999999998e-05 2024-09-21T12:25:54.254 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.254 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.254 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:54.254 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472606+0000", 2024-09-21T12:25:54.254 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0485e-05 2024-09-21T12:25:54.254 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.254 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.254 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:54.255 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472613+0000", 2024-09-21T12:25:54.255 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5590000000000002e-06 2024-09-21T12:25:54.255 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.255 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.255 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:54.255 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.484029+0000", 2024-09-21T12:25:54.255 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.01141587 2024-09-21T12:25:54.255 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.255 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.255 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:54.255 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.484053+0000", 2024-09-21T12:25:54.255 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4559999999999999e-05 2024-09-21T12:25:54.255 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.255 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.256 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:54.256 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.535915+0000", 2024-09-21T12:25:54.256 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.051861431 2024-09-21T12:25:54.256 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.256 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.256 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:54.256 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.535957+0000", 2024-09-21T12:25:54.256 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2478000000000002e-05 2024-09-21T12:25:54.256 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.256 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:54.256 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.256 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.256 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.257 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:87486 7.8 e117/112 7:14e05f7b:::benchmark_data_smithi114_78432_object87485:head v 117'5595, mlcod=88'417)", 2024-09-21T12:25:54.257 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:41.475017+0000", 2024-09-21T12:25:54.257 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 192.67456686400001, 2024-09-21T12:25:54.257 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.060970067000000003, 2024-09-21T12:25:54.257 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:54.257 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:54.257 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:54.257 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:54.258 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.258 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:54.258 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475017+0000", 2024-09-21T12:25:54.258 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.258 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.258 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.259 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:54.259 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475017+0000", 2024-09-21T12:25:54.259 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.259 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.259 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.259 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:54.260 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475018+0000", 2024-09-21T12:25:54.260 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5799999999999999e-07 2024-09-21T12:25:54.260 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.260 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.260 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:54.260 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475150+0000", 2024-09-21T12:25:54.260 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013201700000000001 2024-09-21T12:25:54.261 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.261 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.261 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:54.261 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475159+0000", 2024-09-21T12:25:54.261 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5840000000000001e-06 2024-09-21T12:25:54.261 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.261 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.262 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:54.262 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475166+0000", 2024-09-21T12:25:54.262 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3589999999999996e-06 2024-09-21T12:25:54.262 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.262 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.262 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:54.262 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.525196+0000", 2024-09-21T12:25:54.263 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.050030507000000002 2024-09-21T12:25:54.263 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.263 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.263 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:54.263 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.525214+0000", 2024-09-21T12:25:54.263 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7884e-05 2024-09-21T12:25:54.263 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.264 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.264 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:54.264 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.535960+0000", 2024-09-21T12:25:54.264 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.010746143999999999 2024-09-21T12:25:54.264 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.264 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.264 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:54.264 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.535987+0000", 2024-09-21T12:25:54.265 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6914e-05 2024-09-21T12:25:54.265 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.265 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:54.265 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.265 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.265 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.265 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDPGPush(7.8 117/112 [PushOp(7:1de1c2d4:::benchmark_data_smithi114_78432_object6336:head, version: 88'418, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:1de1c2d4:::benchmark_data_smithi114_78432_object6336:head@88'418, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T12:25:54.266 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:41.485571+0000", 2024-09-21T12:25:54.266 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 192.664012969, 2024-09-21T12:25:54.266 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.046712901000000001, 2024-09-21T12:25:54.266 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:54.266 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:54.266 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:25:54.266 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:54.267 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.267 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:54.267 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485571+0000", 2024-09-21T12:25:54.267 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.267 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.267 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.268 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:54.268 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485571+0000", 2024-09-21T12:25:54.268 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.268 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.268 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.268 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:54.269 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485572+0000", 2024-09-21T12:25:54.269 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7299999999999995e-07 2024-09-21T12:25:54.269 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.269 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.269 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:54.269 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485643+0000", 2024-09-21T12:25:54.269 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0976999999999995e-05 2024-09-21T12:25:54.269 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.270 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.270 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:54.270 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485655+0000", 2024-09-21T12:25:54.270 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2904e-05 2024-09-21T12:25:54.270 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.270 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.270 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:54.270 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485665+0000", 2024-09-21T12:25:54.270 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6709999999999994e-06 2024-09-21T12:25:54.271 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.271 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.271 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:54.271 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.530794+0000", 2024-09-21T12:25:54.271 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.045129295 2024-09-21T12:25:54.271 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.271 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.271 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:54.271 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.530802+0000", 2024-09-21T12:25:54.272 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5059999999999998e-06 2024-09-21T12:25:54.272 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.272 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.272 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:54.272 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.532284+0000", 2024-09-21T12:25:54.272 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014818749999999999 2024-09-21T12:25:54.272 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.272 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:54.273 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.273 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.273 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.273 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:87497 7.8 e117/112 7:19475f57:::benchmark_data_smithi114_78432_object87496:head v 117'5596, mlcod=88'417)", 2024-09-21T12:25:54.273 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:41.494916+0000", 2024-09-21T12:25:54.273 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 192.65466749000001, 2024-09-21T12:25:54.273 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.041122084000000003, 2024-09-21T12:25:54.274 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:54.274 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:54.274 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:54.274 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:54.274 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.274 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:54.274 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.494916+0000", 2024-09-21T12:25:54.274 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.275 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.275 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.275 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:54.275 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.494916+0000", 2024-09-21T12:25:54.275 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:54.275 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.275 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.275 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:54.275 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.494918+0000", 2024-09-21T12:25:54.275 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.719e-06 2024-09-21T12:25:54.275 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.275 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.275 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:54.275 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.494975+0000", 2024-09-21T12:25:54.275 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6981999999999998e-05 2024-09-21T12:25:54.276 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.276 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.276 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:54.276 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.525059+0000", 2024-09-21T12:25:54.276 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.030083526999999999 2024-09-21T12:25:54.276 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.276 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.276 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:54.276 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.525071+0000", 2024-09-21T12:25:54.276 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2438e-05 2024-09-21T12:25:54.276 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.276 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.276 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:54.276 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.532312+0000", 2024-09-21T12:25:54.277 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0072404909999999999 2024-09-21T12:25:54.277 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.277 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.277 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:54.277 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.532336+0000", 2024-09-21T12:25:54.277 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4090000000000001e-05 2024-09-21T12:25:54.277 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.277 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.277 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:54.277 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.536013+0000", 2024-09-21T12:25:54.277 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003677825 2024-09-21T12:25:54.277 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:54.277 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:54.278 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:54.278 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.536038+0000", 2024-09-21T12:25:54.278 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5012e-05 2024-09-21T12:25:54.278 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.278 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:54.278 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.278 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:54.278 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:54.278 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:25:54.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:54.294 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:54.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:54.395 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:54.423 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:54.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:54.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:54.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:54 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2564260096' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:25:54.524 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:54.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:54 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2564260096' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:25:54.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:54 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2564260096' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:25:54.567 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:54.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:54.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:54.668 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:54.705 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:54.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:54.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:54.807 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:54.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:54.840 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:54.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:54.941 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:54.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:54.969 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:54.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:55.045 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T12:25:55.052 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 61 16 95897 95881 98.2108 83.6875 0.0200758 0.0101779 2024-09-21T12:25:55.070 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:55.114 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:55.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:55.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:55.152 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T12:25:55.216 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:55.243 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:55.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:55.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:55.345 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:55.375 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:55.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:55.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:55.477 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:55.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:55 smithi183 ceph-mon[33277]: pgmap v270: 158 pgs: 158 active+clean; 5.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 752 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-21T12:25:55.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:55 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2564260096' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T12:25:55.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:55 smithi183 ceph-mon[33277]: osdmap e189: 8 total, 8 up, 7 in 2024-09-21T12:25:55.505 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:25:55.506 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:55.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:55.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:55.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:55 smithi114 ceph-mon[40867]: pgmap v270: 158 pgs: 158 active+clean; 5.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 752 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-21T12:25:55.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:55 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2564260096' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T12:25:55.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:55 smithi114 ceph-mon[40867]: osdmap e189: 8 total, 8 up, 7 in 2024-09-21T12:25:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:55 smithi114 ceph-mon[30324]: pgmap v270: 158 pgs: 158 active+clean; 5.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 752 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-21T12:25:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:55 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2564260096' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T12:25:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:55 smithi114 ceph-mon[30324]: osdmap e189: 8 total, 8 up, 7 in 2024-09-21T12:25:55.542 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:55.607 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:25:55.631 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:55.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:55.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:25:55.733 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:55.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:55.773 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:55.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:55.874 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:55.900 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:55.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:55.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:55.917 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:25:56.002 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:56.036 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:56.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:56.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:56.053 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 62 16 97057 97041 97.796 72.5 0.00741274 0.0102205 2024-09-21T12:25:56.119 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:25:56.119 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:25:56.119 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.119 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:97115 8.d 8:bb475619:::benchmark_data_smithi114_104936_object97114:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-21T12:25:56.119 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:56.111449+0000", 2024-09-21T12:25:56.119 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0040730610000000002, 2024-09-21T12:25:56.119 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0041054899999999998, 2024-09-21T12:25:56.119 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:56.119 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:56.119 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:25:56.119 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:25:56.119 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:25:56.120 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:25:56.120 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 97115 2024-09-21T12:25:56.120 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.120 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:56.120 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.120 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:56.120 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.111449+0000", 2024-09-21T12:25:56.120 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:56.120 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.120 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.120 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:56.120 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.111449+0000", 2024-09-21T12:25:56.120 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:56.121 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.121 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.121 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:56.121 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.111450+0000", 2024-09-21T12:25:56.121 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.164e-06 2024-09-21T12:25:56.121 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.121 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.121 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:56.121 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.111511+0000", 2024-09-21T12:25:56.121 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1456000000000002e-05 2024-09-21T12:25:56.121 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.121 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.121 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:56.121 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.111526+0000", 2024-09-21T12:25:56.122 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4602e-05 2024-09-21T12:25:56.122 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.122 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.122 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:56.122 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.111534+0000", 2024-09-21T12:25:56.122 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9379999999999992e-06 2024-09-21T12:25:56.122 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.122 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.122 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:56.122 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.111587+0000", 2024-09-21T12:25:56.122 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3427999999999997e-05 2024-09-21T12:25:56.122 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.122 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.123 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:56.123 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.111710+0000", 2024-09-21T12:25:56.123 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012227000000000001 2024-09-21T12:25:56.123 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.123 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.123 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,7", 2024-09-21T12:25:56.123 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.111869+0000", 2024-09-21T12:25:56.123 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015943199999999999 2024-09-21T12:25:56.123 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:56.123 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:56.123 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:56.123 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.123 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.123 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.25022.0:97113 8.3 e189/160 8:c551a23b:::benchmark_data_smithi114_104936_object97112:head v 189'5920, mlcod=189'5918)", 2024-09-21T12:25:56.124 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:56.112662+0000", 2024-09-21T12:25:56.124 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0028600330000000001, 2024-09-21T12:25:56.124 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002957543, 2024-09-21T12:25:56.124 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:56.124 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:56.124 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:25:56.124 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:56.124 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.124 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:56.124 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.112662+0000", 2024-09-21T12:25:56.124 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:56.124 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.124 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.124 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:56.125 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.112662+0000", 2024-09-21T12:25:56.125 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:56.125 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.125 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.125 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:56.125 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.112662+0000", 2024-09-21T12:25:56.125 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4000000000000001e-07 2024-09-21T12:25:56.125 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.125 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.125 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:56.125 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.112731+0000", 2024-09-21T12:25:56.125 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8805000000000003e-05 2024-09-21T12:25:56.126 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.126 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.126 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:56.126 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.112745+0000", 2024-09-21T12:25:56.126 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4090000000000001e-05 2024-09-21T12:25:56.126 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.126 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.126 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:56.126 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.112752+0000", 2024-09-21T12:25:56.126 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6050000000000003e-06 2024-09-21T12:25:56.126 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.126 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.126 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:56.127 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.112806+0000", 2024-09-21T12:25:56.127 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3869999999999998e-05 2024-09-21T12:25:56.127 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.127 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.127 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:56.127 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.112831+0000", 2024-09-21T12:25:56.127 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5392e-05 2024-09-21T12:25:56.127 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:56.128 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:56.128 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:56.128 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.128 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.128 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:97118 8.d 8:b73f5b58:::benchmark_data_smithi114_104936_object97117:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-21T12:25:56.128 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:56.113529+0000", 2024-09-21T12:25:56.128 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0019926449999999999, 2024-09-21T12:25:56.128 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021397500000000002, 2024-09-21T12:25:56.128 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:56.128 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:56.128 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:25:56.128 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:25:56.128 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:25:56.129 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:25:56.129 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 97118 2024-09-21T12:25:56.129 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.129 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:56.129 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.129 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:56.129 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.113529+0000", 2024-09-21T12:25:56.129 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:56.129 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.129 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.129 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:56.129 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.113529+0000", 2024-09-21T12:25:56.129 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:56.129 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.129 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.130 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:56.130 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.113530+0000", 2024-09-21T12:25:56.130 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.093e-06 2024-09-21T12:25:56.130 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.130 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.130 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:56.130 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.113558+0000", 2024-09-21T12:25:56.130 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7991999999999999e-05 2024-09-21T12:25:56.130 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.130 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.130 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:56.130 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.113573+0000", 2024-09-21T12:25:56.130 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4945e-05 2024-09-21T12:25:56.130 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.130 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.131 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:56.131 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.113580+0000", 2024-09-21T12:25:56.131 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.063e-06 2024-09-21T12:25:56.131 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.131 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.131 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:56.131 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.114084+0000", 2024-09-21T12:25:56.131 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00050382400000000005 2024-09-21T12:25:56.131 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.131 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.131 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:56.131 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.114209+0000", 2024-09-21T12:25:56.131 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012512999999999999 2024-09-21T12:25:56.131 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.132 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.132 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,7", 2024-09-21T12:25:56.132 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.114339+0000", 2024-09-21T12:25:56.132 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000130278 2024-09-21T12:25:56.132 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:56.132 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:56.132 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:56.132 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.132 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.132 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.25022.0:97120 8.9 e189/160 8:98550586:::benchmark_data_smithi114_104936_object97119:head v 189'6196, mlcod=189'6194)", 2024-09-21T12:25:56.132 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:56.114267+0000", 2024-09-21T12:25:56.132 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0012548080000000001, 2024-09-21T12:25:56.132 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001455851, 2024-09-21T12:25:56.132 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:56.132 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:56.133 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:25:56.133 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:56.133 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.133 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:56.133 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.114267+0000", 2024-09-21T12:25:56.133 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:56.133 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.133 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.133 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:56.133 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.114267+0000", 2024-09-21T12:25:56.133 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:56.133 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.133 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.133 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:56.133 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.114268+0000", 2024-09-21T12:25:56.134 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8999999999999995e-07 2024-09-21T12:25:56.134 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.134 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.134 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:56.134 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.114291+0000", 2024-09-21T12:25:56.134 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2844999999999999e-05 2024-09-21T12:25:56.134 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.134 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.134 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:56.134 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.114301+0000", 2024-09-21T12:25:56.134 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0723999999999999e-05 2024-09-21T12:25:56.135 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.135 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.135 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:56.135 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.114308+0000", 2024-09-21T12:25:56.135 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.404e-06 2024-09-21T12:25:56.135 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.135 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.135 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:56.135 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.114355+0000", 2024-09-21T12:25:56.135 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7138999999999997e-05 2024-09-21T12:25:56.135 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.135 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.135 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:56.135 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.114372+0000", 2024-09-21T12:25:56.136 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7435000000000002e-05 2024-09-21T12:25:56.136 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:56.136 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:56.136 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:56.136 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.136 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.136 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.25022.0:97119 8.0 e189/160 8:02a2b0be:::benchmark_data_smithi114_104936_object97118:head v 189'6121, mlcod=189'6119)", 2024-09-21T12:25:56.136 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:56.113971+0000", 2024-09-21T12:25:56.136 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0015505060000000001, 2024-09-21T12:25:56.136 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001796867, 2024-09-21T12:25:56.136 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:56.136 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:56.136 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:25:56.137 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:56.137 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.137 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:56.137 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.113971+0000", 2024-09-21T12:25:56.137 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:56.137 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.137 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.137 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:56.137 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.113971+0000", 2024-09-21T12:25:56.137 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:56.137 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.137 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.137 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:56.137 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.113972+0000", 2024-09-21T12:25:56.138 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9999999999999996e-07 2024-09-21T12:25:56.138 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.138 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.138 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:56.138 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.114305+0000", 2024-09-21T12:25:56.138 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00033319399999999998 2024-09-21T12:25:56.138 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.138 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.138 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:56.138 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.114316+0000", 2024-09-21T12:25:56.138 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0369e-05 2024-09-21T12:25:56.138 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.138 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.138 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:56.139 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.114322+0000", 2024-09-21T12:25:56.139 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5910000000000001e-06 2024-09-21T12:25:56.139 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.139 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.139 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:56.139 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.114388+0000", 2024-09-21T12:25:56.139 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6070999999999998e-05 2024-09-21T12:25:56.139 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:56.139 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:56.139 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:56.139 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:56.114408+0000", 2024-09-21T12:25:56.139 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.929e-05 2024-09-21T12:25:56.139 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:56.139 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:56.140 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:56.140 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:56.140 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-21T12:25:56.140 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 5 2024-09-21T12:25:56.140 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:25:56.140 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:56.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:56.181 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:56.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:56.267 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-21T12:25:56.282 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:56.342 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:56.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:56.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:56.444 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:56.472 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:56.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:56.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:56.497 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:56 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2387717901' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:25:56.497 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:56 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:25:56.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:56 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2387717901' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:25:56.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:56 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:25:56.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:56 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2387717901' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:25:56.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:56 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:25:56.574 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:56.599 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:56.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:56.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:56.701 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:56.728 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:56.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:56.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:56.830 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:56.834 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T12:25:56.872 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:56.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:56.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:56.973 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:57.007 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:57.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:57.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:57.053 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 63 16 98192 98176 97.3697 70.9375 0.0061042 0.0102649 2024-09-21T12:25:57.108 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:57.155 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:57.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:57.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:57.193 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:25:57.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:25:56 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:25:57.257 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:57.269 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T12:25:57.284 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:57.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:57.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:57.386 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:57.422 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:57.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:57.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:57.500 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:57 smithi183 ceph-mon[33277]: pgmap v272: 158 pgs: 158 active+clean; 5.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 788 B/s rd, 60 MiB/s wr, 967 op/s 2024-09-21T12:25:57.500 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:57 smithi183 ceph-mon[33277]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:25:57.500 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:57 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:25:57.500 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:57 smithi183 ceph-mon[33277]: osdmap e190: 8 total, 8 up, 7 in 2024-09-21T12:25:57.500 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:57 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3280888184' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:25:57.500 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:57 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:25:57.523 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:57.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:57 smithi114 ceph-mon[30324]: pgmap v272: 158 pgs: 158 active+clean; 5.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 788 B/s rd, 60 MiB/s wr, 967 op/s 2024-09-21T12:25:57.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:57 smithi114 ceph-mon[30324]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:25:57.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:57 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:25:57.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:57 smithi114 ceph-mon[30324]: osdmap e190: 8 total, 8 up, 7 in 2024-09-21T12:25:57.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:57 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3280888184' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:25:57.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:57 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:25:57.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:57 smithi114 ceph-mon[40867]: pgmap v272: 158 pgs: 158 active+clean; 5.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 788 B/s rd, 60 MiB/s wr, 967 op/s 2024-09-21T12:25:57.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:57 smithi114 ceph-mon[40867]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:25:57.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:57 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:25:57.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:57 smithi114 ceph-mon[40867]: osdmap e190: 8 total, 8 up, 7 in 2024-09-21T12:25:57.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:57 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3280888184' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:25:57.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:57 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:25:57.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:57.562 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:57.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:57.663 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:57.695 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:57.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:57.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:57.766 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:25:57.767 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-21T12:25:57.767 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-21T12:25:57.767 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-21T12:25:57.767 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:25:57.797 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:57.818 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:57.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:57.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:57.920 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:57.959 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:57.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:57.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:58.053 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 64 16 99347 99331 96.9762 72.1875 0.0121012 0.0103072 2024-09-21T12:25:58.060 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:58.098 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:58.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:58.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:58.200 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:58.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:58.238 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:58.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:58.339 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:58.376 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:58.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:58.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:58.399 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:58 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:25:58.399 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:58 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T12:25:58.399 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:58 smithi183 ceph-mon[33277]: osdmap e191: 8 total, 8 up, 7 in 2024-09-21T12:25:58.442 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T12:25:58.478 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:25:58.513 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:58.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:58.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:25:58.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:58 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:25:58.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:58 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T12:25:58.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:58 smithi114 ceph-mon[30324]: osdmap e191: 8 total, 8 up, 7 in 2024-09-21T12:25:58.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:58 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:25:58.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:58 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T12:25:58.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:58 smithi114 ceph-mon[40867]: osdmap e191: 8 total, 8 up, 7 in 2024-09-21T12:25:58.615 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:25:58.646 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:58.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:58.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:25:58.748 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:58.783 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:58.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:58.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:58.790 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:25:58.885 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:25:58.926 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:58.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:58.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:25:59.027 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:59.053 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 65 16 100541 100525 96.6323 74.625 0.00492951 0.0103438 2024-09-21T12:25:59.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:59.059 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:59.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:59.161 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:25:59.190 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:59.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:59.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:25:59.292 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:25:59.329 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:59.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:59.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:25:59.431 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:59.471 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:59.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:59.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:59.488 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:25:59.488 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-21T12:25:59.488 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-21T12:25:59.488 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:25:59.488 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.488 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14478.0:1123 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94620848895616 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:25:59.489 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:40.364823+0000", 2024-09-21T12:25:59.489 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 259.12068617900002, 2024-09-21T12:25:59.489 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.76961642699999999, 2024-09-21T12:25:59.489 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:59.489 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:59.489 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:25:59.489 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:25:59.489 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14478", 2024-09-21T12:25:59.489 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:25:59.489 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 1123 2024-09-21T12:25:59.489 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.489 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:59.489 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.489 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:59.490 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364823+0000", 2024-09-21T12:25:59.490 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.490 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.490 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.490 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:59.490 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364823+0000", 2024-09-21T12:25:59.490 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.490 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.490 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.490 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:59.490 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364824+0000", 2024-09-21T12:25:59.490 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8400000000000003e-07 2024-09-21T12:25:59.490 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.491 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.491 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:59.491 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364828+0000", 2024-09-21T12:25:59.491 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7160000000000001e-06 2024-09-21T12:25:59.491 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.491 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.491 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:59.491 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364839+0000", 2024-09-21T12:25:59.491 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1036000000000001e-05 2024-09-21T12:25:59.491 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.491 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.491 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:59.491 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364844+0000", 2024-09-21T12:25:59.491 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3650000000000003e-06 2024-09-21T12:25:59.492 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.492 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.492 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:59.492 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364904+0000", 2024-09-21T12:25:59.492 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0182000000000001e-05 2024-09-21T12:25:59.492 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.492 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.492 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:25:59.492 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364914+0000", 2024-09-21T12:25:59.492 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5319999999999995e-06 2024-09-21T12:25:59.492 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.492 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.492 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:59.493 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.023847+0000", 2024-09-21T12:25:59.493 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.65893299299999997 2024-09-21T12:25:59.493 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.493 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.493 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for degraded object", 2024-09-21T12:25:59.493 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.024200+0000", 2024-09-21T12:25:59.493 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00035333899999999999 2024-09-21T12:25:59.493 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.493 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.493 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:59.493 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.026259+0000", 2024-09-21T12:25:59.493 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020587169999999998 2024-09-21T12:25:59.493 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.493 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.493 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:59.494 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.026327+0000", 2024-09-21T12:25:59.494 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8362999999999996e-05 2024-09-21T12:25:59.494 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.494 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.494 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:59.494 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.134440+0000", 2024-09-21T12:25:59.494 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10811229999999999 2024-09-21T12:25:59.494 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.494 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:59.494 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.494 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.494 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.494 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:24800 7.0 7:089470e4:::benchmark_data_smithi114_78432_object24799:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T12:25:59.495 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:42.436766+0000", 2024-09-21T12:25:59.495 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 257.04874369999999, 2024-09-21T12:25:59.495 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15453825400000001, 2024-09-21T12:25:59.495 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:59.495 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:59.495 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:59.495 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:25:59.495 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:25:59.495 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:25:59.495 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 24800 2024-09-21T12:25:59.495 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.495 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:59.495 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.495 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:59.496 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.436766+0000", 2024-09-21T12:25:59.496 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.496 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.496 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.496 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:59.496 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.436766+0000", 2024-09-21T12:25:59.496 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.496 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.496 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.496 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:59.496 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.436768+0000", 2024-09-21T12:25:59.496 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7460000000000001e-06 2024-09-21T12:25:59.496 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.496 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.497 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:59.497 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.436800+0000", 2024-09-21T12:25:59.497 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1995000000000001e-05 2024-09-21T12:25:59.497 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.497 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.497 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:59.497 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.436823+0000", 2024-09-21T12:25:59.497 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3107000000000001e-05 2024-09-21T12:25:59.497 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.497 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.497 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:59.497 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.436830+0000", 2024-09-21T12:25:59.497 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2450000000000001e-06 2024-09-21T12:25:59.497 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.498 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.498 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:59.498 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.436907+0000", 2024-09-21T12:25:59.498 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7613000000000004e-05 2024-09-21T12:25:59.498 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.498 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.498 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:59.498 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.437022+0000", 2024-09-21T12:25:59.498 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000114988 2024-09-21T12:25:59.498 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.498 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.498 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:25:59.498 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.437165+0000", 2024-09-21T12:25:59.499 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014216200000000001 2024-09-21T12:25:59.499 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.499 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.499 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:25:59.499 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.438118+0000", 2024-09-21T12:25:59.499 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00095354499999999998 2024-09-21T12:25:59.499 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.499 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.499 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.499 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.442130+0000", 2024-09-21T12:25:59.499 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0040119480000000004 2024-09-21T12:25:59.499 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.499 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.499 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.499 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.591203+0000", 2024-09-21T12:25:59.500 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14907253200000001 2024-09-21T12:25:59.500 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.500 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.500 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:59.500 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.591262+0000", 2024-09-21T12:25:59.500 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9249999999999997e-05 2024-09-21T12:25:59.500 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.500 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.500 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:59.500 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.591304+0000", 2024-09-21T12:25:59.500 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2123000000000003e-05 2024-09-21T12:25:59.500 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.500 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:59.500 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.500 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.501 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.501 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:45500 7.0 7:094d36cc:::benchmark_data_smithi114_78432_object45499:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T12:25:59.501 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:59.293987+0000", 2024-09-21T12:25:59.501 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 240.191522518, 2024-09-21T12:25:59.501 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.16208599700000001, 2024-09-21T12:25:59.501 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:59.501 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:59.501 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:59.501 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:25:59.501 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:25:59.501 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:25:59.501 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 45500 2024-09-21T12:25:59.501 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.501 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:59.502 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.502 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:59.502 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293987+0000", 2024-09-21T12:25:59.502 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.503 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.503 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.503 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:59.503 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293987+0000", 2024-09-21T12:25:59.503 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.503 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.503 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.503 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:59.503 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293988+0000", 2024-09-21T12:25:59.503 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3489999999999999e-06 2024-09-21T12:25:59.503 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.503 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.503 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:59.504 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294038+0000", 2024-09-21T12:25:59.504 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9835e-05 2024-09-21T12:25:59.504 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.504 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.504 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:59.504 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294051+0000", 2024-09-21T12:25:59.504 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2463000000000001e-05 2024-09-21T12:25:59.504 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.504 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.504 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:59.504 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294057+0000", 2024-09-21T12:25:59.504 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7390000000000002e-06 2024-09-21T12:25:59.504 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.504 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.505 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:59.505 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294115+0000", 2024-09-21T12:25:59.505 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7219000000000002e-05 2024-09-21T12:25:59.505 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.505 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.505 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:59.505 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294240+0000", 2024-09-21T12:25:59.505 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012520899999999999 2024-09-21T12:25:59.505 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.505 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.505 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:25:59.505 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294329+0000", 2024-09-21T12:25:59.505 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8871999999999999e-05 2024-09-21T12:25:59.505 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.505 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.506 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:25:59.506 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.296811+0000", 2024-09-21T12:25:59.506 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024828570000000002 2024-09-21T12:25:59.506 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.506 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.506 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.506 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302073+0000", 2024-09-21T12:25:59.506 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0052615539999999999 2024-09-21T12:25:59.506 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.506 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.506 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.506 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.455967+0000", 2024-09-21T12:25:59.506 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15389399200000001 2024-09-21T12:25:59.507 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.507 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.507 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:59.507 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.456027+0000", 2024-09-21T12:25:59.507 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9537999999999998e-05 2024-09-21T12:25:59.507 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.507 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.507 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:59.507 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.456073+0000", 2024-09-21T12:25:59.507 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6369999999999998e-05 2024-09-21T12:25:59.507 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.507 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:59.507 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.508 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.508 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.508 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:45512 7.0 7:00045c2a:::benchmark_data_smithi114_78432_object45511:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T12:25:59.508 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:59.294696+0000", 2024-09-21T12:25:59.508 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 240.19081361799999, 2024-09-21T12:25:59.508 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.16753293999999999, 2024-09-21T12:25:59.508 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:59.508 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:59.508 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:59.508 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:25:59.508 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:25:59.508 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:25:59.508 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 45512 2024-09-21T12:25:59.509 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.509 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:59.509 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.509 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:59.509 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294696+0000", 2024-09-21T12:25:59.509 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.509 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.509 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.509 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:59.509 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294696+0000", 2024-09-21T12:25:59.509 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.509 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.509 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.509 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:59.510 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294697+0000", 2024-09-21T12:25:59.510 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1299999999999999e-07 2024-09-21T12:25:59.510 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.510 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.510 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:59.510 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294777+0000", 2024-09-21T12:25:59.510 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0201e-05 2024-09-21T12:25:59.510 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.510 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.510 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:59.510 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294787+0000", 2024-09-21T12:25:59.510 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0281e-05 2024-09-21T12:25:59.510 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.511 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.511 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:59.511 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294794+0000", 2024-09-21T12:25:59.511 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2960000000000001e-06 2024-09-21T12:25:59.511 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.511 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.511 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:59.511 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.295769+0000", 2024-09-21T12:25:59.511 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00097407400000000001 2024-09-21T12:25:59.511 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.511 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.511 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:59.511 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.295919+0000", 2024-09-21T12:25:59.511 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015044200000000001 2024-09-21T12:25:59.512 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.512 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.512 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:25:59.512 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.296054+0000", 2024-09-21T12:25:59.512 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013465499999999999 2024-09-21T12:25:59.512 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.512 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.512 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:25:59.512 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.296931+0000", 2024-09-21T12:25:59.512 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00087781000000000003 2024-09-21T12:25:59.512 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.512 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.512 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.512 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302444+0000", 2024-09-21T12:25:59.513 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0055121249999999997 2024-09-21T12:25:59.513 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.513 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.513 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.513 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.462132+0000", 2024-09-21T12:25:59.513 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15968875499999999 2024-09-21T12:25:59.513 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.513 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.513 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:59.513 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.462187+0000", 2024-09-21T12:25:59.513 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4363000000000001e-05 2024-09-21T12:25:59.513 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.513 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.513 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:59.513 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.462229+0000", 2024-09-21T12:25:59.514 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2125000000000001e-05 2024-09-21T12:25:59.514 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.514 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:59.514 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.514 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.514 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.514 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:49240 7.0 7:06556a82:::benchmark_data_smithi114_78432_object49239:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:25:59.514 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:04.578874+0000", 2024-09-21T12:25:59.514 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 234.906635596, 2024-09-21T12:25:59.514 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.143013362, 2024-09-21T12:25:59.514 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:59.514 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:59.514 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:59.515 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:25:59.515 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:25:59.515 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:25:59.515 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 49240 2024-09-21T12:25:59.515 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.515 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:59.515 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.515 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:59.515 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.578874+0000", 2024-09-21T12:25:59.515 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.515 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.515 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.515 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:59.515 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.578874+0000", 2024-09-21T12:25:59.516 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.516 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.516 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.516 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:59.516 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.578875+0000", 2024-09-21T12:25:59.516 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4109999999999999e-06 2024-09-21T12:25:59.516 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.516 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.516 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:59.516 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.578925+0000", 2024-09-21T12:25:59.516 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9962e-05 2024-09-21T12:25:59.516 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.516 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.516 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:59.516 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.578936+0000", 2024-09-21T12:25:59.517 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1078e-05 2024-09-21T12:25:59.517 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.517 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.517 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:59.517 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.578941+0000", 2024-09-21T12:25:59.517 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0289999999999996e-06 2024-09-21T12:25:59.517 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.517 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.517 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:59.517 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.578985+0000", 2024-09-21T12:25:59.517 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3659000000000003e-05 2024-09-21T12:25:59.517 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.517 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.517 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:59.518 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.579076+0000", 2024-09-21T12:25:59.518 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1416999999999996e-05 2024-09-21T12:25:59.518 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.518 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.518 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:25:59.518 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.579166+0000", 2024-09-21T12:25:59.518 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9338000000000004e-05 2024-09-21T12:25:59.518 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.518 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.518 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:25:59.518 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.580345+0000", 2024-09-21T12:25:59.518 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0011795899999999999 2024-09-21T12:25:59.518 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.518 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.518 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.519 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.580659+0000", 2024-09-21T12:25:59.519 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00031388 2024-09-21T12:25:59.519 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.519 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.519 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.519 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.721812+0000", 2024-09-21T12:25:59.519 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14115282900000001 2024-09-21T12:25:59.519 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.519 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.519 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:59.519 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.721860+0000", 2024-09-21T12:25:59.519 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8090000000000002e-05 2024-09-21T12:25:59.519 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.520 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.520 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:59.520 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.721887+0000", 2024-09-21T12:25:59.520 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7078999999999999e-05 2024-09-21T12:25:59.520 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.520 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:59.520 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.520 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.520 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.520 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:116819 7.0 7:07411ed9:::benchmark_data_smithi114_78432_object12673:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T12:25:59.520 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:23:12.370997+0000", 2024-09-21T12:25:59.520 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 167.11451279400001, 2024-09-21T12:25:59.520 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15405449900000001, 2024-09-21T12:25:59.520 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:59.521 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:59.521 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:59.521 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:25:59.521 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:25:59.521 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:25:59.521 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 116819 2024-09-21T12:25:59.521 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.521 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:59.521 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.521 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:59.521 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.370997+0000", 2024-09-21T12:25:59.521 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.521 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.521 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.522 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:59.522 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.370997+0000", 2024-09-21T12:25:59.522 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.522 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.522 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.522 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:59.522 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.370998+0000", 2024-09-21T12:25:59.522 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.381e-06 2024-09-21T12:25:59.522 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.522 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.522 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:59.522 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.371002+0000", 2024-09-21T12:25:59.522 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3710000000000002e-06 2024-09-21T12:25:59.522 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.523 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.523 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:59.523 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.371014+0000", 2024-09-21T12:25:59.523 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1865000000000001e-05 2024-09-21T12:25:59.523 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.523 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.523 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:59.523 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.371022+0000", 2024-09-21T12:25:59.523 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4610000000000002e-06 2024-09-21T12:25:59.523 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.523 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.523 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:59.523 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.371071+0000", 2024-09-21T12:25:59.523 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9082000000000002e-05 2024-09-21T12:25:59.524 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.524 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.524 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:59.524 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.371188+0000", 2024-09-21T12:25:59.524 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000117015 2024-09-21T12:25:59.524 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.524 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.524 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:25:59.524 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.371309+0000", 2024-09-21T12:25:59.524 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012115700000000001 2024-09-21T12:25:59.524 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.524 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.524 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:25:59.525 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.373478+0000", 2024-09-21T12:25:59.525 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021686840000000001 2024-09-21T12:25:59.525 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.525 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.525 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.525 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.377634+0000", 2024-09-21T12:25:59.525 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0041562589999999998 2024-09-21T12:25:59.525 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.525 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.525 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.525 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.524881+0000", 2024-09-21T12:25:59.525 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14724746899999999 2024-09-21T12:25:59.525 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.525 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.526 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:59.526 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.525006+0000", 2024-09-21T12:25:59.526 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012413300000000001 2024-09-21T12:25:59.526 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.526 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.526 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:59.526 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.525051+0000", 2024-09-21T12:25:59.526 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5621999999999998e-05 2024-09-21T12:25:59.526 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.526 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:59.526 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.526 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.526 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.526 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:149806 7.0 7:0c9bb691:::benchmark_data_smithi114_78432_object45660:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T12:25:59.527 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:23:42.006140+0000", 2024-09-21T12:25:59.527 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 137.47936969200001, 2024-09-21T12:25:59.527 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13218240000000001, 2024-09-21T12:25:59.527 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:59.527 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:59.527 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:59.527 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:25:59.527 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:25:59.527 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:25:59.527 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 149806 2024-09-21T12:25:59.527 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.527 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:59.527 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.528 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:59.528 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.006140+0000", 2024-09-21T12:25:59.528 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.528 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.528 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.528 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:59.528 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.006140+0000", 2024-09-21T12:25:59.528 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.528 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.528 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.528 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:59.528 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.006141+0000", 2024-09-21T12:25:59.528 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7039999999999999e-06 2024-09-21T12:25:59.528 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.529 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.529 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:59.529 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.006147+0000", 2024-09-21T12:25:59.529 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2499999999999997e-06 2024-09-21T12:25:59.529 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.529 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.529 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:59.529 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.006161+0000", 2024-09-21T12:25:59.529 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3974999999999999e-05 2024-09-21T12:25:59.529 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.529 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.529 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:59.529 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.006169+0000", 2024-09-21T12:25:59.529 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7299999999999994e-06 2024-09-21T12:25:59.530 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.530 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.530 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:59.530 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.006225+0000", 2024-09-21T12:25:59.530 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5232000000000002e-05 2024-09-21T12:25:59.530 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.530 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.530 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:59.530 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.006299+0000", 2024-09-21T12:25:59.530 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4219000000000002e-05 2024-09-21T12:25:59.530 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.530 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.530 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:25:59.531 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.006365+0000", 2024-09-21T12:25:59.531 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6278999999999993e-05 2024-09-21T12:25:59.531 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.531 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.531 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:25:59.531 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.008358+0000", 2024-09-21T12:25:59.531 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0019933220000000001 2024-09-21T12:25:59.531 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.531 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.531 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.531 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.013964+0000", 2024-09-21T12:25:59.531 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.005605751 2024-09-21T12:25:59.531 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.532 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.532 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.532 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.138215+0000", 2024-09-21T12:25:59.532 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.12425032 2024-09-21T12:25:59.532 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.532 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.532 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:59.532 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.138279+0000", 2024-09-21T12:25:59.532 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4129999999999998e-05 2024-09-21T12:25:59.532 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.532 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.532 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:59.532 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.138322+0000", 2024-09-21T12:25:59.532 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3488000000000001e-05 2024-09-21T12:25:59.533 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.533 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:59.533 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.533 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.533 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.533 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:149807 7.0 7:0e04ad06:::benchmark_data_smithi114_78432_object45661:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T12:25:59.533 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:23:42.009615+0000", 2024-09-21T12:25:59.533 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 137.47589493800001, 2024-09-21T12:25:59.533 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.128815979, 2024-09-21T12:25:59.533 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:59.533 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:59.533 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:59.533 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:25:59.534 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:25:59.534 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:25:59.534 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 149807 2024-09-21T12:25:59.534 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.534 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:59.534 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.534 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:59.534 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.009615+0000", 2024-09-21T12:25:59.534 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.534 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.534 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.534 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:59.534 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.009615+0000", 2024-09-21T12:25:59.534 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.534 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.535 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.535 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:59.535 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.009616+0000", 2024-09-21T12:25:59.535 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.311e-06 2024-09-21T12:25:59.535 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.535 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.535 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:59.535 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.009621+0000", 2024-09-21T12:25:59.535 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3850000000000001e-06 2024-09-21T12:25:59.535 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.535 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.535 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:59.535 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.009636+0000", 2024-09-21T12:25:59.535 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4284000000000001e-05 2024-09-21T12:25:59.535 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.536 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.536 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:59.536 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.009644+0000", 2024-09-21T12:25:59.536 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2349999999999999e-06 2024-09-21T12:25:59.536 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.536 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.536 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:59.536 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.009700+0000", 2024-09-21T12:25:59.536 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6236999999999997e-05 2024-09-21T12:25:59.536 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.536 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.536 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:59.536 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.009824+0000", 2024-09-21T12:25:59.536 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012450699999999999 2024-09-21T12:25:59.537 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.537 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.537 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:25:59.537 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.009947+0000", 2024-09-21T12:25:59.537 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012274900000000001 2024-09-21T12:25:59.537 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.537 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.537 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:25:59.537 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.011830+0000", 2024-09-21T12:25:59.537 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00188254 2024-09-21T12:25:59.537 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.537 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.537 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.537 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.013996+0000", 2024-09-21T12:25:59.537 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002166126 2024-09-21T12:25:59.537 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.538 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.538 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.538 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.138364+0000", 2024-09-21T12:25:59.538 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.124368232 2024-09-21T12:25:59.538 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.538 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.538 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:59.538 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.138394+0000", 2024-09-21T12:25:59.538 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0205999999999998e-05 2024-09-21T12:25:59.538 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.538 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.538 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:59.538 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.138431+0000", 2024-09-21T12:25:59.538 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6167000000000001e-05 2024-09-21T12:25:59.539 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.539 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:59.539 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.539 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.539 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.539 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:34506 8.8 8:19b95b45:::benchmark_data_smithi114_104936_object34505:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T12:25:59.539 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:08.945853+0000", 2024-09-21T12:25:59.539 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 50.539656774999997, 2024-09-21T12:25:59.539 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.124861031, 2024-09-21T12:25:59.539 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:59.539 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:59.539 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:59.539 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:25:59.539 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:25:59.540 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:25:59.540 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 34506 2024-09-21T12:25:59.540 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.540 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:59.540 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.540 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:59.540 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:08.945853+0000", 2024-09-21T12:25:59.540 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.540 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.540 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.540 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:59.540 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:08.945853+0000", 2024-09-21T12:25:59.540 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.540 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.540 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.541 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:59.541 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:08.945853+0000", 2024-09-21T12:25:59.541 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5799999999999999e-07 2024-09-21T12:25:59.541 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.541 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.541 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:59.541 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:08.945884+0000", 2024-09-21T12:25:59.541 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1037000000000001e-05 2024-09-21T12:25:59.541 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.541 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.541 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:59.541 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:08.945892+0000", 2024-09-21T12:25:59.541 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8210000000000002e-06 2024-09-21T12:25:59.541 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.541 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.541 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:59.542 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:08.945897+0000", 2024-09-21T12:25:59.542 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1329999999999998e-06 2024-09-21T12:25:59.542 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.542 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.542 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:59.542 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:08.949452+0000", 2024-09-21T12:25:59.542 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0035542030000000001 2024-09-21T12:25:59.542 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.542 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.542 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:59.542 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:08.949563+0000", 2024-09-21T12:25:59.542 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000111208 2024-09-21T12:25:59.542 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.543 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.543 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:25:59.543 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:08.949693+0000", 2024-09-21T12:25:59.543 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013028700000000001 2024-09-21T12:25:59.543 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.543 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.544 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:25:59.544 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:08.952213+0000", 2024-09-21T12:25:59.544 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002519847 2024-09-21T12:25:59.544 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.544 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.544 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.544 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:08.952301+0000", 2024-09-21T12:25:59.544 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8010999999999995e-05 2024-09-21T12:25:59.544 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.544 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.544 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.545 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:09.070606+0000", 2024-09-21T12:25:59.545 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.118305417 2024-09-21T12:25:59.545 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.545 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.545 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:59.545 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:09.070668+0000", 2024-09-21T12:25:59.545 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1915999999999995e-05 2024-09-21T12:25:59.545 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.545 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.545 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:59.545 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:09.070714+0000", 2024-09-21T12:25:59.545 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5493e-05 2024-09-21T12:25:59.545 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.545 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:59.546 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.546 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.546 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.546 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:36116 8.8 8:1e014655:::benchmark_data_smithi114_104936_object36115:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T12:25:59.546 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:09.970630+0000", 2024-09-21T12:25:59.546 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 49.514879002000001, 2024-09-21T12:25:59.546 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.140201205, 2024-09-21T12:25:59.546 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:59.546 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:59.546 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:59.546 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:25:59.546 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:25:59.546 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:25:59.546 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 36116 2024-09-21T12:25:59.547 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.547 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:59.547 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.547 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:59.547 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:09.970630+0000", 2024-09-21T12:25:59.547 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.547 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.547 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.547 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:59.547 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:09.970630+0000", 2024-09-21T12:25:59.547 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.547 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.547 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.547 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:59.548 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:09.970632+0000", 2024-09-21T12:25:59.548 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.305e-06 2024-09-21T12:25:59.548 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.548 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.548 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:59.548 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:09.970678+0000", 2024-09-21T12:25:59.548 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5933999999999998e-05 2024-09-21T12:25:59.548 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.548 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.548 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:59.548 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:09.970689+0000", 2024-09-21T12:25:59.548 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0902e-05 2024-09-21T12:25:59.548 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.548 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.548 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:59.549 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:09.970694+0000", 2024-09-21T12:25:59.549 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5280000000000003e-06 2024-09-21T12:25:59.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.549 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:59.549 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:09.970734+0000", 2024-09-21T12:25:59.549 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0250999999999998e-05 2024-09-21T12:25:59.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.549 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:59.549 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:09.970825+0000", 2024-09-21T12:25:59.549 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0736000000000007e-05 2024-09-21T12:25:59.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.549 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:25:59.550 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:09.970962+0000", 2024-09-21T12:25:59.550 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000136816 2024-09-21T12:25:59.550 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.550 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.550 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:25:59.550 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:09.972086+0000", 2024-09-21T12:25:59.550 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001124105 2024-09-21T12:25:59.550 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.550 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.550 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.550 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:09.972488+0000", 2024-09-21T12:25:59.550 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000401989 2024-09-21T12:25:59.550 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.551 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.551 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.551 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:10.110716+0000", 2024-09-21T12:25:59.551 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.138228301 2024-09-21T12:25:59.551 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.551 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.551 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:59.551 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:10.110781+0000", 2024-09-21T12:25:59.551 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4992000000000004e-05 2024-09-21T12:25:59.551 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.551 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.551 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:59.551 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:10.110832+0000", 2024-09-21T12:25:59.551 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0346000000000002e-05 2024-09-21T12:25:59.552 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.552 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:59.552 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.552 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.552 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.552 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:38136 8.8 8:1fdd00da:::benchmark_data_smithi114_104936_object38135:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T12:25:59.552 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:11.297833+0000", 2024-09-21T12:25:59.552 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 48.187676908, 2024-09-21T12:25:59.552 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.143885176, 2024-09-21T12:25:59.552 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:59.552 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:59.552 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:59.552 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:25:59.552 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:25:59.553 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:25:59.553 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 38136 2024-09-21T12:25:59.553 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.553 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:59.553 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.553 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:59.553 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.297833+0000", 2024-09-21T12:25:59.553 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.553 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.553 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.553 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:59.553 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.297833+0000", 2024-09-21T12:25:59.553 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.553 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.554 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.554 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:59.554 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.297833+0000", 2024-09-21T12:25:59.554 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6000000000000004e-07 2024-09-21T12:25:59.554 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.554 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.554 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:59.554 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.297870+0000", 2024-09-21T12:25:59.554 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6479000000000001e-05 2024-09-21T12:25:59.554 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.554 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.554 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:59.554 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.297877+0000", 2024-09-21T12:25:59.554 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0210000000000002e-06 2024-09-21T12:25:59.555 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.555 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.555 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:59.555 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.297882+0000", 2024-09-21T12:25:59.555 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7780000000000004e-06 2024-09-21T12:25:59.555 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.555 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.555 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:59.555 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.297948+0000", 2024-09-21T12:25:59.555 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5433999999999997e-05 2024-09-21T12:25:59.555 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.555 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.555 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:59.555 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.298088+0000", 2024-09-21T12:25:59.556 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000140495 2024-09-21T12:25:59.556 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.556 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.556 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:25:59.556 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.298271+0000", 2024-09-21T12:25:59.556 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000182261 2024-09-21T12:25:59.556 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.556 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.556 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:25:59.556 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.301068+0000", 2024-09-21T12:25:59.556 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0027971229999999999 2024-09-21T12:25:59.556 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.556 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.556 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.556 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.304972+0000", 2024-09-21T12:25:59.557 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00390398 2024-09-21T12:25:59.557 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.557 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.557 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.557 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.441633+0000", 2024-09-21T12:25:59.557 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13666144199999999 2024-09-21T12:25:59.557 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.557 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.557 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:59.557 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.441694+0000", 2024-09-21T12:25:59.557 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0501999999999998e-05 2024-09-21T12:25:59.557 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.557 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.557 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:59.557 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.441718+0000", 2024-09-21T12:25:59.558 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4100999999999998e-05 2024-09-21T12:25:59.558 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.558 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:59.558 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.558 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.558 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.558 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:38143 8.8 8:133e9789:::benchmark_data_smithi114_104936_object38142:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T12:25:59.558 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:11.302085+0000", 2024-09-21T12:25:59.558 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 48.183424551999998, 2024-09-21T12:25:59.558 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15153207799999999, 2024-09-21T12:25:59.558 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:59.558 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:59.558 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:59.558 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:25:59.559 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:25:59.559 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:25:59.559 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 38143 2024-09-21T12:25:59.559 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.559 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:59.559 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.559 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:59.559 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302085+0000", 2024-09-21T12:25:59.559 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.559 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.559 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.559 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:59.559 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302085+0000", 2024-09-21T12:25:59.559 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.560 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.560 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.560 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:59.560 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302086+0000", 2024-09-21T12:25:59.560 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4339999999999999e-06 2024-09-21T12:25:59.560 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.560 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.560 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:59.560 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302163+0000", 2024-09-21T12:25:59.560 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6200999999999998e-05 2024-09-21T12:25:59.560 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.560 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.560 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:59.560 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302172+0000", 2024-09-21T12:25:59.561 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5759999999999995e-06 2024-09-21T12:25:59.561 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.561 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.561 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:59.561 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302178+0000", 2024-09-21T12:25:59.561 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2260000000000002e-06 2024-09-21T12:25:59.561 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.561 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.561 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:59.561 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.303083+0000", 2024-09-21T12:25:59.561 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00090426600000000003 2024-09-21T12:25:59.561 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.561 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.561 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:59.562 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.303220+0000", 2024-09-21T12:25:59.562 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013702300000000001 2024-09-21T12:25:59.562 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.562 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.562 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:25:59.562 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.303354+0000", 2024-09-21T12:25:59.562 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013437099999999999 2024-09-21T12:25:59.562 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.562 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.562 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:25:59.562 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.305072+0000", 2024-09-21T12:25:59.562 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001718102 2024-09-21T12:25:59.562 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.562 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.563 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.563 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.306241+0000", 2024-09-21T12:25:59.563 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001168716 2024-09-21T12:25:59.563 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.563 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.563 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.563 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.453516+0000", 2024-09-21T12:25:59.563 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14727496200000001 2024-09-21T12:25:59.563 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.563 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.563 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:59.563 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.453572+0000", 2024-09-21T12:25:59.563 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5923e-05 2024-09-21T12:25:59.563 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.564 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.564 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:59.564 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.453617+0000", 2024-09-21T12:25:59.564 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5278000000000003e-05 2024-09-21T12:25:59.564 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.564 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:59.564 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.564 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.564 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.564 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:38148 8.8 8:1c0b8cc6:::benchmark_data_smithi114_104936_object38147:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T12:25:59.564 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:11.305128+0000", 2024-09-21T12:25:59.564 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 48.180381623999999, 2024-09-21T12:25:59.564 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.148605125, 2024-09-21T12:25:59.564 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:59.565 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:59.565 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:59.565 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:25:59.565 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:25:59.565 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:25:59.565 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 38148 2024-09-21T12:25:59.565 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.565 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:59.565 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.565 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:59.565 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.305128+0000", 2024-09-21T12:25:59.565 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.565 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.565 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.565 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:59.566 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.305128+0000", 2024-09-21T12:25:59.566 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.566 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.566 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.566 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:59.566 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.305129+0000", 2024-09-21T12:25:59.566 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.142e-06 2024-09-21T12:25:59.566 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.566 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.567 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:59.567 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.305182+0000", 2024-09-21T12:25:59.567 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3489999999999998e-05 2024-09-21T12:25:59.567 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.567 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.567 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:59.567 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.305192+0000", 2024-09-21T12:25:59.567 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3829999999999998e-06 2024-09-21T12:25:59.567 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.567 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.567 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:59.567 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.305198+0000", 2024-09-21T12:25:59.568 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6860000000000003e-06 2024-09-21T12:25:59.568 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.569 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.569 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:59.569 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.305239+0000", 2024-09-21T12:25:59.569 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1882000000000003e-05 2024-09-21T12:25:59.569 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.570 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.570 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:59.570 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.305351+0000", 2024-09-21T12:25:59.570 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011123799999999999 2024-09-21T12:25:59.570 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.570 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.570 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:25:59.570 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.305474+0000", 2024-09-21T12:25:59.570 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000123363 2024-09-21T12:25:59.570 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.570 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.570 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:25:59.570 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.306408+0000", 2024-09-21T12:25:59.570 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00093416099999999998 2024-09-21T12:25:59.571 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.571 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.571 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.571 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.306612+0000", 2024-09-21T12:25:59.571 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00020365699999999999 2024-09-21T12:25:59.571 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.571 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.571 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.571 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.453666+0000", 2024-09-21T12:25:59.571 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.147053778 2024-09-21T12:25:59.571 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.571 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.571 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:59.571 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.453698+0000", 2024-09-21T12:25:59.572 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2011000000000002e-05 2024-09-21T12:25:59.572 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.572 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.572 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:59.572 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.453733+0000", 2024-09-21T12:25:59.572 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5333999999999997e-05 2024-09-21T12:25:59.572 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.572 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:59.572 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.572 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.572 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.572 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:77674 8.d 8:bb2f14dc:::benchmark_data_smithi114_104936_object77673:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-21T12:25:59.572 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:40.251838+0000", 2024-09-21T12:25:59.572 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 19.233671567999998, 2024-09-21T12:25:59.573 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.126956077, 2024-09-21T12:25:59.573 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:59.573 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:59.573 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:59.573 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:25:59.573 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:25:59.573 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:25:59.573 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 77674 2024-09-21T12:25:59.573 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.573 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:59.573 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.573 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:59.573 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.251838+0000", 2024-09-21T12:25:59.573 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.574 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.574 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.574 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:59.574 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.251838+0000", 2024-09-21T12:25:59.574 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.574 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.574 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.574 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:59.574 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.251839+0000", 2024-09-21T12:25:59.574 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5099999999999999e-06 2024-09-21T12:25:59.574 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.574 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.574 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:59.575 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.251915+0000", 2024-09-21T12:25:59.575 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5210999999999995e-05 2024-09-21T12:25:59.575 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.575 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.575 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:59.575 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.251924+0000", 2024-09-21T12:25:59.575 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2750000000000001e-06 2024-09-21T12:25:59.575 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.575 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.575 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:59.575 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.251930+0000", 2024-09-21T12:25:59.575 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.584e-06 2024-09-21T12:25:59.575 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.575 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.576 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:59.576 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.251977+0000", 2024-09-21T12:25:59.576 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6774999999999999e-05 2024-09-21T12:25:59.576 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.576 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.576 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:59.576 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.252091+0000", 2024-09-21T12:25:59.576 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011381 2024-09-21T12:25:59.576 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.576 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.576 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,7", 2024-09-21T12:25:59.576 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.252238+0000", 2024-09-21T12:25:59.576 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014654499999999999 2024-09-21T12:25:59.576 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.577 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.577 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:25:59.577 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.256039+0000", 2024-09-21T12:25:59.577 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003801305 2024-09-21T12:25:59.577 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.577 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.577 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.577 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.378676+0000", 2024-09-21T12:25:59.577 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.122637309 2024-09-21T12:25:59.577 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.578 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.578 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.578 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.378724+0000", 2024-09-21T12:25:59.578 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8183999999999998e-05 2024-09-21T12:25:59.578 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.578 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.578 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:59.578 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.378762+0000", 2024-09-21T12:25:59.578 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7743000000000001e-05 2024-09-21T12:25:59.578 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.578 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.578 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:59.578 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.378794+0000", 2024-09-21T12:25:59.579 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1826000000000003e-05 2024-09-21T12:25:59.579 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.579 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:59.579 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.579 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.579 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.579 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:77676 8.d 8:b085cec0:::benchmark_data_smithi114_104936_object77675:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-21T12:25:59.579 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:40.252793+0000", 2024-09-21T12:25:59.579 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 19.232716472, 2024-09-21T12:25:59.579 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.126167109, 2024-09-21T12:25:59.579 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:59.579 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:59.579 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:59.579 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:25:59.580 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:25:59.580 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:25:59.580 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 77676 2024-09-21T12:25:59.580 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.580 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:59.580 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.580 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:59.580 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.252793+0000", 2024-09-21T12:25:59.580 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.580 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.580 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.580 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:59.580 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.252793+0000", 2024-09-21T12:25:59.580 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.581 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.581 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.581 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:59.581 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.252794+0000", 2024-09-21T12:25:59.581 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.074e-06 2024-09-21T12:25:59.581 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.581 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.581 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:59.581 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.252828+0000", 2024-09-21T12:25:59.581 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4198e-05 2024-09-21T12:25:59.581 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.581 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.581 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:59.581 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.252840+0000", 2024-09-21T12:25:59.582 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1344e-05 2024-09-21T12:25:59.582 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.582 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.582 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:59.582 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.252847+0000", 2024-09-21T12:25:59.582 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0319999999999998e-06 2024-09-21T12:25:59.582 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.582 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.582 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:59.582 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.254046+0000", 2024-09-21T12:25:59.582 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0011993030000000001 2024-09-21T12:25:59.582 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.582 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.582 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:59.583 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.254155+0000", 2024-09-21T12:25:59.583 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010935800000000001 2024-09-21T12:25:59.583 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.583 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.583 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,7", 2024-09-21T12:25:59.583 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.254263+0000", 2024-09-21T12:25:59.583 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010734 2024-09-21T12:25:59.583 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.583 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.583 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:25:59.583 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.256141+0000", 2024-09-21T12:25:59.583 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001877933 2024-09-21T12:25:59.583 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.583 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.583 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.584 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.378847+0000", 2024-09-21T12:25:59.584 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.12270599 2024-09-21T12:25:59.584 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.584 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.584 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.584 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.378883+0000", 2024-09-21T12:25:59.584 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6026999999999999e-05 2024-09-21T12:25:59.584 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.584 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.584 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:59.584 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.378928+0000", 2024-09-21T12:25:59.584 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5476999999999998e-05 2024-09-21T12:25:59.584 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.584 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.585 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:59.585 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.378960+0000", 2024-09-21T12:25:59.585 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2033000000000003e-05 2024-09-21T12:25:59.585 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.585 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:59.585 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.585 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.585 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.585 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.25022.0:77679 8.a e183/160 8:5f3d1dfc:::benchmark_data_smithi114_104936_object77678:head v 183'4753, mlcod=183'4751)", 2024-09-21T12:25:59.585 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:40.254563+0000", 2024-09-21T12:25:59.585 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 19.230946633999999, 2024-09-21T12:25:59.585 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.12409081, 2024-09-21T12:25:59.585 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:59.585 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:59.586 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:59.586 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:59.586 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.586 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:59.586 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.254563+0000", 2024-09-21T12:25:59.586 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.586 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.586 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.586 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:59.586 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.254563+0000", 2024-09-21T12:25:59.586 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.586 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.587 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:25:59 smithi183 ceph-mon[33277]: pgmap v275: 158 pgs: 158 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 108 MiB/s wr, 1.72k op/s 2024-09-21T12:25:59.587 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.587 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:59.587 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.254564+0000", 2024-09-21T12:25:59.588 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8400000000000003e-07 2024-09-21T12:25:59.588 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.588 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.588 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:59.588 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.254628+0000", 2024-09-21T12:25:59.588 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4419999999999996e-05 2024-09-21T12:25:59.588 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.588 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.588 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:59.588 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.254639+0000", 2024-09-21T12:25:59.588 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0472000000000001e-05 2024-09-21T12:25:59.588 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.588 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.588 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:59.589 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.254645+0000", 2024-09-21T12:25:59.589 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0209999999999996e-06 2024-09-21T12:25:59.589 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.589 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.589 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:59.589 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.254693+0000", 2024-09-21T12:25:59.589 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8195999999999997e-05 2024-09-21T12:25:59.589 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.589 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.589 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:59.589 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.254709+0000", 2024-09-21T12:25:59.589 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6392000000000002e-05 2024-09-21T12:25:59.589 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.589 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.590 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:59.590 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.378614+0000", 2024-09-21T12:25:59.590 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.12390522299999999 2024-09-21T12:25:59.590 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.590 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.590 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:59.590 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.378654+0000", 2024-09-21T12:25:59.590 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9301999999999997e-05 2024-09-21T12:25:59.590 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.590 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:59.590 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.590 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.590 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.590 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:77681 8.4 8:289ac157:::benchmark_data_smithi114_104936_object77680:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-21T12:25:59.591 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:40.255755+0000", 2024-09-21T12:25:59.591 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 19.22975404, 2024-09-21T12:25:59.591 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.12688951700000001, 2024-09-21T12:25:59.591 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:59.591 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:59.591 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:59.591 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:25:59.591 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:25:59.591 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:25:59.591 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 77681 2024-09-21T12:25:59.591 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.591 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:59.591 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.592 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:59.592 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.255755+0000", 2024-09-21T12:25:59.592 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.592 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.592 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.592 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:59.592 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.255755+0000", 2024-09-21T12:25:59.592 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.592 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.592 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.592 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:59.592 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.255757+0000", 2024-09-21T12:25:59.592 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.325e-06 2024-09-21T12:25:59.592 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.593 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.593 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:59.593 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.255837+0000", 2024-09-21T12:25:59.593 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0737999999999998e-05 2024-09-21T12:25:59.593 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.593 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.593 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:59.593 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.255847+0000", 2024-09-21T12:25:59.593 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6700000000000006e-06 2024-09-21T12:25:59.593 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.593 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.593 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:59.593 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.255853+0000", 2024-09-21T12:25:59.593 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0700000000000003e-06 2024-09-21T12:25:59.594 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.594 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.594 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:59.594 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.255908+0000", 2024-09-21T12:25:59.594 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5135e-05 2024-09-21T12:25:59.594 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.594 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.594 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:59.594 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.256027+0000", 2024-09-21T12:25:59.594 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000118538 2024-09-21T12:25:59.594 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.594 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.594 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:25:59.594 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.256175+0000", 2024-09-21T12:25:59.595 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014814400000000001 2024-09-21T12:25:59.595 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.595 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.595 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.595 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.382361+0000", 2024-09-21T12:25:59.595 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.12618574900000001 2024-09-21T12:25:59.595 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.595 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.595 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.595 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.382405+0000", 2024-09-21T12:25:59.595 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4258999999999998e-05 2024-09-21T12:25:59.595 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.595 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.596 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:25:59.596 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.382551+0000", 2024-09-21T12:25:59.596 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000145974 2024-09-21T12:25:59.596 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.596 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.596 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:59.596 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.382602+0000", 2024-09-21T12:25:59.596 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0702999999999999e-05 2024-09-21T12:25:59.596 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.596 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.596 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:59.596 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.382645+0000", 2024-09-21T12:25:59.596 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3211999999999999e-05 2024-09-21T12:25:59.597 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.597 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:59.597 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.597 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.597 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.597 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:99019 8.8 8:16dc5116:::benchmark_data_smithi114_104936_object99018:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-21T12:25:59.597 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:57.695581+0000", 2024-09-21T12:25:59.597 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.789928175, 2024-09-21T12:25:59.597 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14590615700000001, 2024-09-21T12:25:59.597 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:59.597 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:59.597 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:59.597 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:25:59.597 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:25:59.597 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:25:59.598 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 99019 2024-09-21T12:25:59.598 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.598 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:59.598 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.598 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:59.598 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.695581+0000", 2024-09-21T12:25:59.598 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.598 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.598 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.598 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:59.598 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.695581+0000", 2024-09-21T12:25:59.598 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.598 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.599 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.599 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:59.599 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.695583+0000", 2024-09-21T12:25:59.599 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4750000000000001e-06 2024-09-21T12:25:59.599 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.599 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.599 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:59.599 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.695635+0000", 2024-09-21T12:25:59.599 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1758999999999997e-05 2024-09-21T12:25:59.599 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.599 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.599 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:59.599 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.695645+0000", 2024-09-21T12:25:59.599 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0515e-05 2024-09-21T12:25:59.600 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.600 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.600 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:59.600 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.695652+0000", 2024-09-21T12:25:59.600 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1269999999999997e-06 2024-09-21T12:25:59.600 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.600 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.600 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:59.600 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.698722+0000", 2024-09-21T12:25:59.600 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0030701370000000001 2024-09-21T12:25:59.600 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.600 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.600 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:59.600 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.698948+0000", 2024-09-21T12:25:59.600 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000225219 2024-09-21T12:25:59.601 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.601 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.601 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:25:59.601 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.699165+0000", 2024-09-21T12:25:59.601 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00021733500000000001 2024-09-21T12:25:59.601 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.601 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.601 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.601 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.702022+0000", 2024-09-21T12:25:59.601 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0028576259999999998 2024-09-21T12:25:59.601 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.601 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.601 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:25:59.602 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.707874+0000", 2024-09-21T12:25:59.602 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0058513849999999997 2024-09-21T12:25:59.602 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.602 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.602 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.602 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.841381+0000", 2024-09-21T12:25:59.602 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.133507406 2024-09-21T12:25:59.602 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.602 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.602 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:59.602 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.841438+0000", 2024-09-21T12:25:59.602 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6221999999999998e-05 2024-09-21T12:25:59.602 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.603 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.603 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:59.603 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.841487+0000", 2024-09-21T12:25:59.603 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9951000000000003e-05 2024-09-21T12:25:59.603 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.603 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:59.603 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.603 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.603 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.603 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:99025 8.8 8:1b11fe1e:::benchmark_data_smithi114_104936_object99024:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-21T12:25:59.603 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:57.699077+0000", 2024-09-21T12:25:59.603 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.786432807, 2024-09-21T12:25:59.603 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14424990100000001, 2024-09-21T12:25:59.604 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:59.604 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:59.604 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:59.604 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:25:59.604 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:25:59.604 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:25:59.604 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 99025 2024-09-21T12:25:59.604 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.604 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:59.604 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.604 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:59.604 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.699077+0000", 2024-09-21T12:25:59.604 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.604 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.605 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.605 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:59.605 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.699077+0000", 2024-09-21T12:25:59.605 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.605 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.605 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:59.605 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.699078+0000", 2024-09-21T12:25:59.605 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1310000000000001e-06 2024-09-21T12:25:59.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.605 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.605 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:59.605 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.699136+0000", 2024-09-21T12:25:59.605 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8109000000000002e-05 2024-09-21T12:25:59.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.606 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.606 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:59.606 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.699147+0000", 2024-09-21T12:25:59.606 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.15e-05 2024-09-21T12:25:59.606 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.606 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.606 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:59.606 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.699156+0000", 2024-09-21T12:25:59.606 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.884e-06 2024-09-21T12:25:59.606 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.606 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.606 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:59.606 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.702052+0000", 2024-09-21T12:25:59.606 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0028958199999999999 2024-09-21T12:25:59.606 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.607 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.607 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:59.607 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.702173+0000", 2024-09-21T12:25:59.607 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000120518 2024-09-21T12:25:59.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.607 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.607 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:25:59.607 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.702551+0000", 2024-09-21T12:25:59.607 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00037817800000000001 2024-09-21T12:25:59.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.607 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.607 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.607 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.707944+0000", 2024-09-21T12:25:59.607 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0053930769999999996 2024-09-21T12:25:59.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.608 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:25:59.608 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.709806+0000", 2024-09-21T12:25:59.608 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001861824 2024-09-21T12:25:59.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.608 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.608 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.843235+0000", 2024-09-21T12:25:59.608 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13342960200000001 2024-09-21T12:25:59.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.608 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:59.608 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.843288+0000", 2024-09-21T12:25:59.608 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2324000000000003e-05 2024-09-21T12:25:59.609 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.609 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.609 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:59.609 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.843327+0000", 2024-09-21T12:25:59.609 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8933999999999997e-05 2024-09-21T12:25:59.609 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.609 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:59.609 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.609 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.609 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.609 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:99033 8.8 8:1a2a2a5f:::benchmark_data_smithi114_104936_object99032:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-21T12:25:59.609 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:57.702897+0000", 2024-09-21T12:25:59.609 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 1.782612944, 2024-09-21T12:25:59.609 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14051256500000001, 2024-09-21T12:25:59.609 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:25:59.610 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:25:59.610 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:25:59.610 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:25:59.610 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:25:59.610 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:25:59.610 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 99033 2024-09-21T12:25:59.610 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.610 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:25:59.610 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.610 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:25:59.610 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.702897+0000", 2024-09-21T12:25:59.610 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.610 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.610 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.611 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:25:59.611 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.702897+0000", 2024-09-21T12:25:59.611 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:25:59.611 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.611 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.611 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:25:59.611 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.702898+0000", 2024-09-21T12:25:59.611 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.246e-06 2024-09-21T12:25:59.611 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.611 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.611 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:25:59.611 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.702950+0000", 2024-09-21T12:25:59.611 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1962000000000002e-05 2024-09-21T12:25:59.611 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.612 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.612 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:25:59.612 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.702966+0000", 2024-09-21T12:25:59.612 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6586999999999999e-05 2024-09-21T12:25:59.612 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.612 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.612 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:25:59.612 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.702974+0000", 2024-09-21T12:25:59.612 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1079999999999993e-06 2024-09-21T12:25:59.612 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.612 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.612 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:25:59.612 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.705534+0000", 2024-09-21T12:25:59.612 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0025595700000000002 2024-09-21T12:25:59.612 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.613 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.613 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:25:59.613 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.705666+0000", 2024-09-21T12:25:59.613 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013246500000000001 2024-09-21T12:25:59.613 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.613 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.613 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:25:59.613 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.705830+0000", 2024-09-21T12:25:59.613 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016372199999999999 2024-09-21T12:25:59.613 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.613 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.613 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.613 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.709794+0000", 2024-09-21T12:25:59.613 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0039634759999999996 2024-09-21T12:25:59.614 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.614 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.614 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:25:59.614 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.709866+0000", 2024-09-21T12:25:59.614 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2502999999999993e-05 2024-09-21T12:25:59.614 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.614 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.614 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:25:59.614 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.843354+0000", 2024-09-21T12:25:59.614 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13348826499999999 2024-09-21T12:25:59.614 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.614 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.614 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:25:59.615 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.843385+0000", 2024-09-21T12:25:59.615 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0281999999999999e-05 2024-09-21T12:25:59.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:25:59.615 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:25:59.615 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:25:59.615 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.843409+0000", 2024-09-21T12:25:59.615 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4379000000000002e-05 2024-09-21T12:25:59.615 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.615 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:59.615 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.615 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:25:59.615 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:25:59.615 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:25:59.616 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:59.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:59.651 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:59.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:59.752 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:25:59.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:25:59 smithi114 ceph-mon[40867]: pgmap v275: 158 pgs: 158 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 108 MiB/s wr, 1.72k op/s 2024-09-21T12:25:59.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:25:59 smithi114 ceph-mon[30324]: pgmap v275: 158 pgs: 158 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 108 MiB/s wr, 1.72k op/s 2024-09-21T12:25:59.789 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:25:59.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:59.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:25:59.891 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:25:59.920 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:25:59.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:25:59.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:25:59.941 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T12:26:00.022 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:00.054 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:00.117 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 66 16 101862 101846 96.4177 82.5625 0.00752832 0.0103667 2024-09-21T12:26:00.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:00.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:00.218 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:00.266 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:00.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:00.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:00.367 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:00.409 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:00.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:00.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:00.478 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:00.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:00 smithi183 ceph-mon[33277]: pgmap v276: 158 pgs: 158 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-21T12:26:00.511 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:00.573 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:00.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:00.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:00.616 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:00 smithi114 ceph-mon[30324]: pgmap v276: 158 pgs: 158 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-21T12:26:00.616 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:00 smithi114 ceph-mon[40867]: pgmap v276: 158 pgs: 158 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-21T12:26:00.674 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:00.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:00.722 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:00.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:00.823 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:00.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:00.860 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:00.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:00.874 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T12:26:00.961 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:00.996 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:00.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:00.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:01.054 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 67 16 103127 103111 96.1586 79.0625 0.0164501 0.0103943 2024-09-21T12:26:01.099 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:01.118 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:01.128 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:26:01.128 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:26:01.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:01.157 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:01.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:01.166 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:26:01.166 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:26:01.205 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:26:01.205 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:26:01.238 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:26:01.238 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:26:01.257 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:26:01.258 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:01.271 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:26:01.271 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:26:01.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:01.297 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:01.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:01.304 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:26:01.304 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:26:01.335 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:26:01.335 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:26:01.371 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:26:01.371 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:26:01.398 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:01.440 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:01.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:01.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:01.466 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:01 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1135152828' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:26:01.466 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:01 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1135152828' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:26:01.542 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:01.553 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:01 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1135152828' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:26:01.570 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:01.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:01.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:01.672 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:01.711 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:01.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:01.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:01.812 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:01.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:01.857 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:01.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:01.957 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:01.996 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:01.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:01.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:02.054 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 68 16 104319 104303 95.8401 74.5 0.0215393 0.0104297 2024-09-21T12:26:02.098 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:02.130 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:02.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:02.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:02.232 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:26:02.232 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:26:02.232 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:02.268 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:02.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:02.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:02.370 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:02.407 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:02.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:02.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:02.509 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:02.526 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T12:26:02.543 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:02.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:02.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:02.636 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:02 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:26:02.636 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:02 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:26:02.636 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:02 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:26:02.636 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:02 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:26:02.636 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:02 smithi183 ceph-mon[33277]: pgmap v277: 158 pgs: 158 active+clean; 6.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.3 KiB/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-21T12:26:02.636 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:02 smithi183 ceph-mon[33277]: osdmap e192: 8 total, 8 up, 7 in 2024-09-21T12:26:02.636 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:02 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2296829712' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:26:02.636 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:02 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:26:02.645 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:02.670 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:02.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:02.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:02.772 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:02.775 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:26:02 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:26:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:26:02.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:02 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:26:02.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:02 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:26:02.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:02 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:26:02.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:02 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:26:02.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:02 smithi114 ceph-mon[40867]: pgmap v277: 158 pgs: 158 active+clean; 6.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.3 KiB/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-21T12:26:02.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:02 smithi114 ceph-mon[40867]: osdmap e192: 8 total, 8 up, 7 in 2024-09-21T12:26:02.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:02 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2296829712' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:26:02.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:02 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:26:02.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:02 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:26:02.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:02 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:26:02.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:02 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:26:02.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:02 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:26:02.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:02 smithi114 ceph-mon[30324]: pgmap v277: 158 pgs: 158 active+clean; 6.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.3 KiB/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-21T12:26:02.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:02 smithi114 ceph-mon[30324]: osdmap e192: 8 total, 8 up, 7 in 2024-09-21T12:26:02.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:02 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2296829712' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:26:02.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:02 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:26:02.797 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:02.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:02.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:02.899 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:02.908 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T12:26:02.955 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:02.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:02.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:02.989 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T12:26:03.054 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 69 16 105559 105543 95.5742 77.5 0.0110934 0.0104585 2024-09-21T12:26:03.058 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:03.095 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:03.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:03.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:03.197 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:03.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:03.233 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:03.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:03.334 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:03.367 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:26:03.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:03.379 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:03.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:03.479 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:03.497 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:03 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:26:03.497 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:03 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:26:03.498 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:03 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T12:26:03.498 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:03 smithi183 ceph-mon[33277]: osdmap e193: 8 total, 8 up, 7 in 2024-09-21T12:26:03.511 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:03.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:03.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:03.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:03 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:26:03.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:03 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:26:03.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:03 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T12:26:03.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:03 smithi114 ceph-mon[40867]: osdmap e193: 8 total, 8 up, 7 in 2024-09-21T12:26:03.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:03 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:26:03.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:03 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:26:03.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:03 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T12:26:03.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:03 smithi114 ceph-mon[30324]: osdmap e193: 8 total, 8 up, 7 in 2024-09-21T12:26:03.613 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:03.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:03.653 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:03.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:03.700 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T12:26:03.724 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:03.755 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:03.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:03.780 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:03.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:03.881 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:03.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:03.926 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:03.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:03.951 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:26:03.951 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:26:04.027 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:04.055 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 70 16 106815 106799 95.3293 78.5 0.0119963 0.0104854 2024-09-21T12:26:04.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:04.077 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:04.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:04.178 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:04.223 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:04.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:04.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:04.325 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:04.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:04.357 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:04.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:04.458 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:04.469 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:26:04.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:04.495 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:04.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:04.571 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:04 smithi183 ceph-mon[33277]: pgmap v280: 158 pgs: 158 active+clean; 6.4 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 798 B/s rd, 60 MiB/s wr, 953 op/s 2024-09-21T12:26:04.596 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:04.612 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T12:26:04.629 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:04.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:04.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:04.731 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:04.764 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:04.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:04.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:04.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:04 smithi114 ceph-mon[40867]: pgmap v280: 158 pgs: 158 active+clean; 6.4 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 798 B/s rd, 60 MiB/s wr, 953 op/s 2024-09-21T12:26:04.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:04 smithi114 ceph-mon[30324]: pgmap v280: 158 pgs: 158 active+clean; 6.4 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 798 B/s rd, 60 MiB/s wr, 953 op/s 2024-09-21T12:26:04.866 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:04.904 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:04.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:04.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:05.006 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:05.018 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:26:05.042 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:05.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:05.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:05.055 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 71 16 108056 108040 95.079 77.5625 0.022058 0.0105131 2024-09-21T12:26:05.144 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:05.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:05.172 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:05.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:05.273 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:05.304 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:05.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:05.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:05.405 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:05.427 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:05.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:05.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:05.529 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:05.565 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:05.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:05.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:05.650 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:26:05.650 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:26:05.667 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:05.702 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:05.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:05.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:05.803 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:05.836 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:05.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:05.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:05.938 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:05.967 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:05.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:05.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:06.055 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 72 16 109068 109052 94.6369 63.25 0.00438777 0.01055 2024-09-21T12:26:06.068 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:06.102 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:06.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:06.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:06.204 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:06.231 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:06.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:06.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:06.332 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:06.353 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:06.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:06.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:06.455 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:06.483 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:06.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:06.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:06.585 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:06.607 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:06.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:06.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:06.709 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:06.745 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:06.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:06.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:06.847 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:06.884 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:06.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:06.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:06.986 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:06.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:06 smithi183 ceph-mon[33277]: pgmap v281: 158 pgs: 158 active+clean; 6.4 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 48 MiB/s wr, 763 op/s 2024-09-21T12:26:06.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:26:06 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:26:07.023 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:07.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:07.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:07.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:06 smithi114 ceph-mon[30324]: pgmap v281: 158 pgs: 158 active+clean; 6.4 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 48 MiB/s wr, 763 op/s 2024-09-21T12:26:07.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:06 smithi114 ceph-mon[40867]: pgmap v281: 158 pgs: 158 active+clean; 6.4 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 48 MiB/s wr, 763 op/s 2024-09-21T12:26:07.055 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 73 16 110145 110129 94.2625 67.3125 0.00759369 0.0106001 2024-09-21T12:26:07.125 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:07.163 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:07.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:07.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:07.265 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:07.292 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:07.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:07.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:07.394 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:07.427 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:07.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:07.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:07.528 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:07.570 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:07.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:07.574 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:07.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:07.582 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:07.675 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:07.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:07.697 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:07.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:07.798 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:07.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:07.844 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:07.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:07.887 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-21T12:26:07.945 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:07.967 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:07.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:07.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:07.986 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:07 smithi114 ceph-mon[30324]: osdmap e194: 8 total, 8 up, 7 in 2024-09-21T12:26:07.986 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:07 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:26:07.986 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:07 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1459222410' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:26:07.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:07 smithi114 ceph-mon[40867]: osdmap e194: 8 total, 8 up, 7 in 2024-09-21T12:26:07.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:07 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:26:07.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:07 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1459222410' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:26:08.056 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 74 16 111142 111126 93.8308 62.3125 0.00416877 0.010653 2024-09-21T12:26:08.068 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:08.099 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:08.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:08.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:08.200 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:08.225 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:08.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:08.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:08.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:07 smithi183 ceph-mon[33277]: osdmap e194: 8 total, 8 up, 7 in 2024-09-21T12:26:08.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:07 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:26:08.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:07 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1459222410' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:26:08.327 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:08.369 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:08.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:08.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:08.470 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:08.515 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:08.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:08.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:08.521 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:26:08.521 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:26:08.562 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:26:08.562 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:26:08.595 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:26:08.596 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:26:08.617 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:08.630 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:26:08.630 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:26:08.653 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:08.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:08.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:08.665 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:26:08.665 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:26:08.700 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:26:08.700 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:26:08.754 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:08.778 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:26:08.778 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:26:08.782 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:08.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:08.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:08.808 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:26:08.809 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:26:08.884 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:08.925 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:08.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:08.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:09.027 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:09.056 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 75 16 112357 112341 93.5922 75.9375 0.00550273 0.0106807 2024-09-21T12:26:09.064 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:09.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:09.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:09.140 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:08 smithi114 ceph-mon[30324]: pgmap v283: 158 pgs: 158 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 105 MiB/s wr, 1.67k op/s 2024-09-21T12:26:09.140 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:08 smithi114 ceph-mon[30324]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:26:09.140 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:08 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1459222410' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:26:09.140 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:08 smithi114 ceph-mon[30324]: osdmap e195: 8 total, 8 up, 7 in 2024-09-21T12:26:09.140 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:08 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2440435762' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:26:09.140 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:08 smithi114 ceph-mon[40867]: pgmap v283: 158 pgs: 158 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 105 MiB/s wr, 1.67k op/s 2024-09-21T12:26:09.140 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:08 smithi114 ceph-mon[40867]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:26:09.141 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:08 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1459222410' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:26:09.141 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:08 smithi114 ceph-mon[40867]: osdmap e195: 8 total, 8 up, 7 in 2024-09-21T12:26:09.141 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:08 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2440435762' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:26:09.166 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:09.206 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:09.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:09.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:09.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:08 smithi183 ceph-mon[33277]: pgmap v283: 158 pgs: 158 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 105 MiB/s wr, 1.67k op/s 2024-09-21T12:26:09.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:08 smithi183 ceph-mon[33277]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:26:09.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:08 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1459222410' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:26:09.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:08 smithi183 ceph-mon[33277]: osdmap e195: 8 total, 8 up, 7 in 2024-09-21T12:26:09.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:08 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2440435762' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:26:09.308 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:09.336 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:09.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:09.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:09.438 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:09.465 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:09.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:09.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:09.566 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:09.606 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:09.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:09.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:09.708 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:09.748 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:09.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:09.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:09.768 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T12:26:09.849 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:09.879 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:09.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:09.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:09.981 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:09.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:09 smithi183 ceph-mon[33277]: 6.16 scrub starts 2024-09-21T12:26:09.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:09 smithi183 ceph-mon[33277]: 6.16 scrub ok 2024-09-21T12:26:09.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:09 smithi183 ceph-mon[33277]: 4.4 scrub starts 2024-09-21T12:26:09.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:09 smithi183 ceph-mon[33277]: 4.4 scrub ok 2024-09-21T12:26:10.018 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:10.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:10.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:10.056 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 76 15 113609 113594 93.3911 78.3125 0.00830557 0.0107035 2024-09-21T12:26:10.119 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:10.131 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:26:10.154 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:10.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:10.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:10.256 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:10.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:09 smithi114 ceph-mon[40867]: 6.16 scrub starts 2024-09-21T12:26:10.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:09 smithi114 ceph-mon[40867]: 6.16 scrub ok 2024-09-21T12:26:10.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:09 smithi114 ceph-mon[40867]: 4.4 scrub starts 2024-09-21T12:26:10.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:09 smithi114 ceph-mon[40867]: 4.4 scrub ok 2024-09-21T12:26:10.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:09 smithi114 ceph-mon[30324]: 6.16 scrub starts 2024-09-21T12:26:10.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:09 smithi114 ceph-mon[30324]: 6.16 scrub ok 2024-09-21T12:26:10.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:09 smithi114 ceph-mon[30324]: 4.4 scrub starts 2024-09-21T12:26:10.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:09 smithi114 ceph-mon[30324]: 4.4 scrub ok 2024-09-21T12:26:10.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:10.298 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:10.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:10.398 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:10.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:10.438 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:10.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:10.539 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:10.578 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:10.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:10.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:10.680 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:10.682 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:26:10.682 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:26:10.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:10.682 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.25022.0:114418 8.6 e195/160 8:6fde07b5:::benchmark_data_smithi114_104936_object114417:head v 195'7162, mlcod=195'7159)", 2024-09-21T12:26:10.682 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:26:10.680054+0000", 2024-09-21T12:26:10.682 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.00032054499999999999, 2024-09-21T12:26:10.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00033984299999999999, 2024-09-21T12:26:10.683 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:26:10.683 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:26:10.683 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:26:10.683 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:26:10.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:10.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:26:10.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:26:10.680054+0000", 2024-09-21T12:26:10.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:10.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:10.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:10.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:26:10.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:26:10.680054+0000", 2024-09-21T12:26:10.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:10.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:10.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:10.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:26:10.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:26:10.680055+0000", 2024-09-21T12:26:10.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2699999999999998e-07 2024-09-21T12:26:10.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:10.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:10.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:26:10.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:26:10.680104+0000", 2024-09-21T12:26:10.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9268000000000003e-05 2024-09-21T12:26:10.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:10.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:10.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:26:10.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:26:10.680118+0000", 2024-09-21T12:26:10.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4022000000000001e-05 2024-09-21T12:26:10.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:10.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:10.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:26:10.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:26:10.680125+0000", 2024-09-21T12:26:10.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9929999999999998e-06 2024-09-21T12:26:10.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:10.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:10.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:26:10.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:26:10.680181+0000", 2024-09-21T12:26:10.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6458000000000001e-05 2024-09-21T12:26:10.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:10.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:10.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:26:10.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:26:10.680220+0000", 2024-09-21T12:26:10.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8588999999999999e-05 2024-09-21T12:26:10.686 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:10.686 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:26:10.686 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:10.686 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:10.686 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-21T12:26:10.686 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 1 2024-09-21T12:26:10.686 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:26:10.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:10.716 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:10.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:10.817 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:10.842 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:10.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:10.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:10.943 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:10.969 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:10.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:10.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:11.056 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 77 16 114860 114844 93.1928 78.125 0.00905966 0.0107258 2024-09-21T12:26:11.071 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:11.106 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:11.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:11.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:11.191 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T12:26:11.208 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:11.232 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:10 smithi114 ceph-mon[40867]: pgmap v285: 158 pgs: 158 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 754 B/s rd, 54 MiB/s wr, 863 op/s 2024-09-21T12:26:11.232 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:10 smithi114 ceph-mon[40867]: 4.18 scrub starts 2024-09-21T12:26:11.232 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:10 smithi114 ceph-mon[40867]: 4.18 scrub ok 2024-09-21T12:26:11.232 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:10 smithi114 ceph-mon[40867]: 4.c scrub starts 2024-09-21T12:26:11.232 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:10 smithi114 ceph-mon[40867]: 4.c scrub ok 2024-09-21T12:26:11.232 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:10 smithi114 ceph-mon[40867]: 6.6 scrub starts 2024-09-21T12:26:11.232 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:10 smithi114 ceph-mon[40867]: 6.6 scrub ok 2024-09-21T12:26:11.233 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:10 smithi114 ceph-mon[30324]: pgmap v285: 158 pgs: 158 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 754 B/s rd, 54 MiB/s wr, 863 op/s 2024-09-21T12:26:11.233 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:10 smithi114 ceph-mon[30324]: 4.18 scrub starts 2024-09-21T12:26:11.233 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:10 smithi114 ceph-mon[30324]: 4.18 scrub ok 2024-09-21T12:26:11.233 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:10 smithi114 ceph-mon[30324]: 4.c scrub starts 2024-09-21T12:26:11.233 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:10 smithi114 ceph-mon[30324]: 4.c scrub ok 2024-09-21T12:26:11.233 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:10 smithi114 ceph-mon[30324]: 6.6 scrub starts 2024-09-21T12:26:11.233 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:10 smithi114 ceph-mon[30324]: 6.6 scrub ok 2024-09-21T12:26:11.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:10 smithi183 ceph-mon[33277]: pgmap v285: 158 pgs: 158 active+clean; 6.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 754 B/s rd, 54 MiB/s wr, 863 op/s 2024-09-21T12:26:11.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:10 smithi183 ceph-mon[33277]: 4.18 scrub starts 2024-09-21T12:26:11.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:10 smithi183 ceph-mon[33277]: 4.18 scrub ok 2024-09-21T12:26:11.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:10 smithi183 ceph-mon[33277]: 4.c scrub starts 2024-09-21T12:26:11.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:10 smithi183 ceph-mon[33277]: 4.c scrub ok 2024-09-21T12:26:11.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:10 smithi183 ceph-mon[33277]: 6.6 scrub starts 2024-09-21T12:26:11.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:10 smithi183 ceph-mon[33277]: 6.6 scrub ok 2024-09-21T12:26:11.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:11.272 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:11.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:11.374 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:11.397 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T12:26:11.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:11.420 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:11.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:11.521 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:11.553 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:11.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:11.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:11.640 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:11.655 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:11.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:11.684 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:11.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:11.743 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:26:11.785 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:11.817 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:11.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:11.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:11.885 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T12:26:11.919 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:11.956 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:11.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:11.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:12.040 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:11 smithi114 ceph-mon[40867]: 5.1f scrub starts 2024-09-21T12:26:12.040 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:11 smithi114 ceph-mon[40867]: 5.1f scrub ok 2024-09-21T12:26:12.040 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:11 smithi114 ceph-mon[40867]: 6.14 scrub starts 2024-09-21T12:26:12.040 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:11 smithi114 ceph-mon[40867]: 6.14 scrub ok 2024-09-21T12:26:12.040 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:11 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1621280662' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:26:12.041 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:11 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:26:12.041 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:11 smithi114 ceph-mon[40867]: 5.7 scrub starts 2024-09-21T12:26:12.041 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:11 smithi114 ceph-mon[40867]: 5.7 scrub ok 2024-09-21T12:26:12.041 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:11 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T12:26:12.041 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:11 smithi114 ceph-mon[40867]: osdmap e196: 8 total, 8 up, 7 in 2024-09-21T12:26:12.042 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:11 smithi114 ceph-mon[30324]: 5.1f scrub starts 2024-09-21T12:26:12.042 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:11 smithi114 ceph-mon[30324]: 5.1f scrub ok 2024-09-21T12:26:12.042 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:11 smithi114 ceph-mon[30324]: 6.14 scrub starts 2024-09-21T12:26:12.042 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:11 smithi114 ceph-mon[30324]: 6.14 scrub ok 2024-09-21T12:26:12.042 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:11 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1621280662' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:26:12.042 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:11 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:26:12.042 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:11 smithi114 ceph-mon[30324]: 5.7 scrub starts 2024-09-21T12:26:12.042 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:11 smithi114 ceph-mon[30324]: 5.7 scrub ok 2024-09-21T12:26:12.043 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:11 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T12:26:12.043 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:11 smithi114 ceph-mon[30324]: osdmap e196: 8 total, 8 up, 7 in 2024-09-21T12:26:12.056 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 78 16 116068 116052 92.9659 75.5 0.0111821 0.010752 2024-09-21T12:26:12.058 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:12.100 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:12.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:12.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:12.144 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:11 smithi183 ceph-mon[33277]: 5.1f scrub starts 2024-09-21T12:26:12.144 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:11 smithi183 ceph-mon[33277]: 5.1f scrub ok 2024-09-21T12:26:12.145 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:11 smithi183 ceph-mon[33277]: 6.14 scrub starts 2024-09-21T12:26:12.145 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:11 smithi183 ceph-mon[33277]: 6.14 scrub ok 2024-09-21T12:26:12.145 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:11 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1621280662' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:26:12.145 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:11 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:26:12.145 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:11 smithi183 ceph-mon[33277]: 5.7 scrub starts 2024-09-21T12:26:12.145 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:11 smithi183 ceph-mon[33277]: 5.7 scrub ok 2024-09-21T12:26:12.145 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:11 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T12:26:12.145 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:11 smithi183 ceph-mon[33277]: osdmap e196: 8 total, 8 up, 7 in 2024-09-21T12:26:12.201 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:12.228 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:12.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:12.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:12.284 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:26:12.284 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-21T12:26:12.284 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-21T12:26:12.284 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-21T12:26:12.284 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:26:12.330 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:12.369 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:12.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:12.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:12.471 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:12.507 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:12.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:12.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:12.609 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:12.645 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:12.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:12.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:12.746 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:12.757 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:26:12 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:26:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:26:12.775 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:12.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:12.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:12.808 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T12:26:12.877 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:12.905 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:12.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:12.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:12.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:12 smithi183 ceph-mon[33277]: pgmap v286: 158 pgs: 158 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 60 MiB/s wr, 960 op/s 2024-09-21T12:26:12.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:12 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:26:12.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:12 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:26:12.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:12 smithi183 ceph-mon[33277]: 3.4 scrub starts 2024-09-21T12:26:12.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:12 smithi183 ceph-mon[33277]: 3.4 scrub ok 2024-09-21T12:26:12.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:12 smithi183 ceph-mon[33277]: 3.2 scrub starts 2024-09-21T12:26:12.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:12 smithi183 ceph-mon[33277]: 3.2 scrub ok 2024-09-21T12:26:13.007 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:13.030 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:12 smithi114 ceph-mon[30324]: pgmap v286: 158 pgs: 158 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 60 MiB/s wr, 960 op/s 2024-09-21T12:26:13.030 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:12 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:26:13.030 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:12 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:26:13.031 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:12 smithi114 ceph-mon[30324]: 3.4 scrub starts 2024-09-21T12:26:13.031 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:12 smithi114 ceph-mon[30324]: 3.4 scrub ok 2024-09-21T12:26:13.031 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:12 smithi114 ceph-mon[30324]: 3.2 scrub starts 2024-09-21T12:26:13.031 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:12 smithi114 ceph-mon[30324]: 3.2 scrub ok 2024-09-21T12:26:13.031 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:12 smithi114 ceph-mon[40867]: pgmap v286: 158 pgs: 158 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 60 MiB/s wr, 960 op/s 2024-09-21T12:26:13.031 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:12 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:26:13.031 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:12 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:26:13.031 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:12 smithi114 ceph-mon[40867]: 3.4 scrub starts 2024-09-21T12:26:13.031 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:12 smithi114 ceph-mon[40867]: 3.4 scrub ok 2024-09-21T12:26:13.031 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:12 smithi114 ceph-mon[40867]: 3.2 scrub starts 2024-09-21T12:26:13.031 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:12 smithi114 ceph-mon[40867]: 3.2 scrub ok 2024-09-21T12:26:13.033 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:13.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:13.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:13.056 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 79 16 117359 117343 92.8104 80.6875 0.0127424 0.01077 2024-09-21T12:26:13.134 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:13.160 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:13.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:13.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:13.260 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:13.262 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:13.267 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T12:26:13.296 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:26:13.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:13.303 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:13.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:13.404 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:13.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:13.459 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:13.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:13.560 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:13.588 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:13.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:13.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:13.600 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:26:13.691 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:13.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:13.744 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:13.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:13.845 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:13.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:13.874 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:13.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:13.933 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:13.940 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-21T12:26:13.963 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:13 smithi114 ceph-mon[30324]: 5.2 scrub starts 2024-09-21T12:26:13.964 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:13 smithi114 ceph-mon[30324]: 5.2 scrub ok 2024-09-21T12:26:13.964 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:13 smithi114 ceph-mon[30324]: 5.d scrub starts 2024-09-21T12:26:13.964 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:13 smithi114 ceph-mon[30324]: 5.d scrub ok 2024-09-21T12:26:13.964 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:13 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1523207938' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:26:13.964 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:13 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:26:13.964 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:13 smithi114 ceph-mon[40867]: 5.2 scrub starts 2024-09-21T12:26:13.964 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:13 smithi114 ceph-mon[40867]: 5.2 scrub ok 2024-09-21T12:26:13.964 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:13 smithi114 ceph-mon[40867]: 5.d scrub starts 2024-09-21T12:26:13.964 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:13 smithi114 ceph-mon[40867]: 5.d scrub ok 2024-09-21T12:26:13.964 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:13 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1523207938' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:26:13.965 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:13 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:26:13.975 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:13.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:13 smithi183 ceph-mon[33277]: 5.2 scrub starts 2024-09-21T12:26:13.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:13 smithi183 ceph-mon[33277]: 5.2 scrub ok 2024-09-21T12:26:13.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:13 smithi183 ceph-mon[33277]: 5.d scrub starts 2024-09-21T12:26:13.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:13 smithi183 ceph-mon[33277]: 5.d scrub ok 2024-09-21T12:26:13.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:13 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1523207938' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:26:13.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:13 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:26:14.013 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:14.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:14.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:14.056 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-21T12:26:14.056083+0000 min lat: 0.00121483 max lat: 0.158038 avg lat: 0.0108087 2024-09-21T12:26:14.056 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T12:26:14.057 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 80 16 118420 118404 92.4789 66.3125 0.0115269 0.0108087 2024-09-21T12:26:14.115 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:14.152 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:14.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:14.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:14.172 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:26:14.172 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-21T12:26:14.172 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-21T12:26:14.172 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:26:14.172 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.172 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:22578 7.5 7:a6fef64a:::benchmark_data_smithi114_78432_object22577:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T12:26:14.172 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:38.894207+0000", 2024-09-21T12:26:14.172 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 275.27442204599998, 2024-09-21T12:26:14.172 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.076693918, 2024-09-21T12:26:14.173 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:26:14.173 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:26:14.173 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:14.173 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:26:14.173 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:26:14.173 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:14.173 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 22578 2024-09-21T12:26:14.173 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.173 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:26:14.173 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.173 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:26:14.173 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:38.894207+0000", 2024-09-21T12:26:14.173 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.173 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.174 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.174 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:26:14.174 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:38.894207+0000", 2024-09-21T12:26:14.174 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.174 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.174 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.174 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:26:14.174 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:38.894209+0000", 2024-09-21T12:26:14.174 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7010000000000001e-06 2024-09-21T12:26:14.174 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.174 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.174 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:26:14.174 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:38.894251+0000", 2024-09-21T12:26:14.174 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2837999999999999e-05 2024-09-21T12:26:14.175 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.175 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.175 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:26:14.175 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:38.894269+0000", 2024-09-21T12:26:14.175 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7819999999999999e-05 2024-09-21T12:26:14.175 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.175 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.175 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:26:14.175 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:38.894278+0000", 2024-09-21T12:26:14.175 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5380000000000002e-06 2024-09-21T12:26:14.175 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.175 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.175 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:26:14.176 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:38.894344+0000", 2024-09-21T12:26:14.176 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5865999999999996e-05 2024-09-21T12:26:14.176 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.176 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.176 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:26:14.176 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:38.894471+0000", 2024-09-21T12:26:14.176 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012738699999999999 2024-09-21T12:26:14.176 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.176 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.176 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:26:14.176 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:38.894600+0000", 2024-09-21T12:26:14.176 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012902899999999999 2024-09-21T12:26:14.176 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.176 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.176 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:26:14.177 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:38.895641+0000", 2024-09-21T12:26:14.177 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001041143 2024-09-21T12:26:14.177 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.177 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.177 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.177 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:38.897058+0000", 2024-09-21T12:26:14.177 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001417253 2024-09-21T12:26:14.177 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.177 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.177 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.177 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:38.970781+0000", 2024-09-21T12:26:14.177 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.073722756 2024-09-21T12:26:14.177 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.177 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.178 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:26:14.178 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:38.970854+0000", 2024-09-21T12:26:14.178 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2614999999999995e-05 2024-09-21T12:26:14.178 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.178 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.178 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:26:14.178 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:38.970901+0000", 2024-09-21T12:26:14.178 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6971999999999997e-05 2024-09-21T12:26:14.178 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.178 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:26:14.178 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.178 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.178 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.179 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14478.0:1121 5.9 5:93e5b521:::notify.7:head [watch ping cookie 94620848900224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-21T12:26:14.179 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:40.364371+0000", 2024-09-21T12:26:14.179 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 273.80425769099998, 2024-09-21T12:26:14.179 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.77380638599999996, 2024-09-21T12:26:14.179 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:26:14.179 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:26:14.179 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:26:14.179 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:26:14.179 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14478", 2024-09-21T12:26:14.179 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:26:14.179 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 1121 2024-09-21T12:26:14.179 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.179 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:26:14.179 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.180 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:26:14.180 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364371+0000", 2024-09-21T12:26:14.180 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.180 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.180 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.180 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:26:14.180 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364371+0000", 2024-09-21T12:26:14.180 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.180 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.180 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.180 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:26:14.180 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364372+0000", 2024-09-21T12:26:14.180 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7400000000000002e-07 2024-09-21T12:26:14.180 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.180 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.181 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:26:14.181 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364376+0000", 2024-09-21T12:26:14.181 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6789999999999998e-06 2024-09-21T12:26:14.181 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.181 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.181 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:26:14.181 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364384+0000", 2024-09-21T12:26:14.181 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3299999999999999e-06 2024-09-21T12:26:14.181 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.181 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.181 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:26:14.181 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364393+0000", 2024-09-21T12:26:14.181 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6379999999999993e-06 2024-09-21T12:26:14.181 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.181 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.182 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:26:14.182 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364494+0000", 2024-09-21T12:26:14.182 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010131799999999999 2024-09-21T12:26:14.182 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.182 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.182 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:26:14.182 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364510+0000", 2024-09-21T12:26:14.182 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5630000000000001e-05 2024-09-21T12:26:14.182 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.182 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.182 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:26:14.182 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.134462+0000", 2024-09-21T12:26:14.182 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.76995182100000004 2024-09-21T12:26:14.182 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.183 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.183 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for degraded object", 2024-09-21T12:26:14.183 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.134766+0000", 2024-09-21T12:26:14.183 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00030401499999999998 2024-09-21T12:26:14.183 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.183 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.183 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:26:14.183 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.138007+0000", 2024-09-21T12:26:14.183 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0032417420000000001 2024-09-21T12:26:14.183 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.183 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.183 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:26:14.183 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.138098+0000", 2024-09-21T12:26:14.183 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0841000000000007e-05 2024-09-21T12:26:14.183 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.184 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.184 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:26:14.184 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.138178+0000", 2024-09-21T12:26:14.184 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9498000000000004e-05 2024-09-21T12:26:14.184 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.184 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:26:14.184 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.184 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.184 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.184 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14478.0:1126 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94620848900224 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:26:14.184 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:40.364831+0000", 2024-09-21T12:26:14.184 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 273.80379783199999, 2024-09-21T12:26:14.184 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78063636199999997, 2024-09-21T12:26:14.184 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:26:14.184 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:26:14.185 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:26:14.185 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:26:14.185 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14478", 2024-09-21T12:26:14.185 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:26:14.185 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 1126 2024-09-21T12:26:14.185 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.185 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:26:14.185 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.185 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:26:14.185 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364831+0000", 2024-09-21T12:26:14.186 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.186 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.186 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.186 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:26:14.186 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364831+0000", 2024-09-21T12:26:14.186 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.186 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.186 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.186 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:26:14.187 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364832+0000", 2024-09-21T12:26:14.187 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1900000000000001e-07 2024-09-21T12:26:14.187 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.187 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.187 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:26:14.187 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364835+0000", 2024-09-21T12:26:14.187 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5099999999999999e-06 2024-09-21T12:26:14.187 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.187 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.187 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:26:14.187 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364846+0000", 2024-09-21T12:26:14.187 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0139000000000001e-05 2024-09-21T12:26:14.187 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.188 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.188 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:26:14.188 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364853+0000", 2024-09-21T12:26:14.188 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2629999999999997e-06 2024-09-21T12:26:14.188 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.188 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.188 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:26:14.188 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364911+0000", 2024-09-21T12:26:14.188 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8493999999999999e-05 2024-09-21T12:26:14.188 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.188 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.188 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:26:14.188 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364921+0000", 2024-09-21T12:26:14.188 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8400000000000007e-06 2024-09-21T12:26:14.188 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.189 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.189 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:26:14.189 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.134966+0000", 2024-09-21T12:26:14.189 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.77004525300000004 2024-09-21T12:26:14.189 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.189 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.189 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for degraded object", 2024-09-21T12:26:14.189 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.135016+0000", 2024-09-21T12:26:14.189 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9419000000000002e-05 2024-09-21T12:26:14.189 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.189 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.189 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:26:14.189 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.138211+0000", 2024-09-21T12:26:14.189 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0031947360000000001 2024-09-21T12:26:14.190 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.190 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.190 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:26:14.190 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.138283+0000", 2024-09-21T12:26:14.190 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2712000000000004e-05 2024-09-21T12:26:14.190 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.190 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.190 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:26:14.190 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.145467+0000", 2024-09-21T12:26:14.190 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0071841769999999999 2024-09-21T12:26:14.190 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.190 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:26:14.190 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.190 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.191 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.191 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:24816 7.5 7:a47d801e:::benchmark_data_smithi114_78432_object24815:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T12:26:14.191 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:42.446936+0000", 2024-09-21T12:26:14.191 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 271.72169322399998, 2024-09-21T12:26:14.191 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.144229354, 2024-09-21T12:26:14.191 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:26:14.191 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:26:14.191 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:14.191 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:26:14.191 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:26:14.191 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:14.191 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 24816 2024-09-21T12:26:14.191 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.191 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:26:14.192 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.192 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:26:14.192 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.446936+0000", 2024-09-21T12:26:14.192 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.192 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.192 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.192 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:26:14.192 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.446936+0000", 2024-09-21T12:26:14.192 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.192 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.192 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.192 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:26:14.192 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.446938+0000", 2024-09-21T12:26:14.192 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4889999999999998e-06 2024-09-21T12:26:14.192 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.193 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.193 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:26:14.193 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.446993+0000", 2024-09-21T12:26:14.193 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4778999999999998e-05 2024-09-21T12:26:14.193 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.193 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.193 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:26:14.193 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.447007+0000", 2024-09-21T12:26:14.193 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4399000000000001e-05 2024-09-21T12:26:14.193 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.193 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.193 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:26:14.193 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.447014+0000", 2024-09-21T12:26:14.193 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0700000000000001e-06 2024-09-21T12:26:14.194 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.194 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.194 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:26:14.194 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.447082+0000", 2024-09-21T12:26:14.194 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7971000000000004e-05 2024-09-21T12:26:14.194 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.194 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.194 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:26:14.194 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.447211+0000", 2024-09-21T12:26:14.194 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012822300000000001 2024-09-21T12:26:14.194 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.194 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.194 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:26:14.194 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.447358+0000", 2024-09-21T12:26:14.194 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014737999999999999 2024-09-21T12:26:14.195 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.195 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.195 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:26:14.195 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.448450+0000", 2024-09-21T12:26:14.195 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001092148 2024-09-21T12:26:14.195 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.195 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.195 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.195 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.448529+0000", 2024-09-21T12:26:14.195 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9090000000000003e-05 2024-09-21T12:26:14.195 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.195 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.195 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.195 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.591057+0000", 2024-09-21T12:26:14.195 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.142527445 2024-09-21T12:26:14.196 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.196 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.196 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:26:14.196 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.591120+0000", 2024-09-21T12:26:14.196 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3695000000000006e-05 2024-09-21T12:26:14.196 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.196 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.196 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:26:14.196 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.591165+0000", 2024-09-21T12:26:14.196 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4665e-05 2024-09-21T12:26:14.196 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.196 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:26:14.196 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.196 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.197 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.197 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:41241 7.5 7:a7b2dcc6:::benchmark_data_smithi114_78432_object41240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-21T12:26:14.197 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:55.817543+0000", 2024-09-21T12:26:14.197 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 258.35108618499999, 2024-09-21T12:26:14.197 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.094563281999999999, 2024-09-21T12:26:14.197 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:26:14.197 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:26:14.197 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:14.197 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:26:14.197 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:26:14.197 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:14.197 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 41241 2024-09-21T12:26:14.197 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.197 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:26:14.198 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.198 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:26:14.198 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:55.817543+0000", 2024-09-21T12:26:14.198 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.198 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.198 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.198 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:26:14.198 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:55.817543+0000", 2024-09-21T12:26:14.198 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.198 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.198 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.198 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:26:14.198 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:55.817544+0000", 2024-09-21T12:26:14.198 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.488e-06 2024-09-21T12:26:14.199 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.199 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.199 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:26:14.199 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:55.817593+0000", 2024-09-21T12:26:14.199 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8977000000000002e-05 2024-09-21T12:26:14.199 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.199 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.199 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:26:14.199 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:55.817607+0000", 2024-09-21T12:26:14.199 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3623e-05 2024-09-21T12:26:14.199 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.199 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.199 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:26:14.199 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:55.817614+0000", 2024-09-21T12:26:14.200 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3610000000000003e-06 2024-09-21T12:26:14.200 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.200 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.200 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:26:14.200 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:55.817669+0000", 2024-09-21T12:26:14.200 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5089000000000001e-05 2024-09-21T12:26:14.200 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.200 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.200 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:26:14.200 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:55.817767+0000", 2024-09-21T12:26:14.200 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7943999999999996e-05 2024-09-21T12:26:14.200 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.200 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.200 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:26:14.201 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:55.817863+0000", 2024-09-21T12:26:14.201 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6253000000000002e-05 2024-09-21T12:26:14.201 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.201 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.201 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:26:14.201 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:55.818953+0000", 2024-09-21T12:26:14.201 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010892180000000001 2024-09-21T12:26:14.201 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.201 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.201 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.201 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:55.819836+0000", 2024-09-21T12:26:14.201 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00088346699999999998 2024-09-21T12:26:14.201 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.202 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.202 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.202 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:55.912001+0000", 2024-09-21T12:26:14.202 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.092164974999999996 2024-09-21T12:26:14.202 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.202 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.202 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:26:14.202 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:55.912065+0000", 2024-09-21T12:26:14.202 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3942000000000005e-05 2024-09-21T12:26:14.202 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.202 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.202 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:26:14.202 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:55.912106+0000", 2024-09-21T12:26:14.202 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0945000000000002e-05 2024-09-21T12:26:14.202 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.203 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:26:14.203 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.203 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.203 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.203 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:45176 7.5 7:ae38956b:::benchmark_data_smithi114_78432_object45175:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T12:26:14.203 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:58.998297+0000", 2024-09-21T12:26:14.203 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 255.17033178099999, 2024-09-21T12:26:14.203 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.082483624000000005, 2024-09-21T12:26:14.203 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:26:14.203 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:26:14.203 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:14.203 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:26:14.203 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:26:14.203 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:14.204 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 45176 2024-09-21T12:26:14.204 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.204 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:26:14.204 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.204 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:26:14.204 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:58.998297+0000", 2024-09-21T12:26:14.204 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.204 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.204 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.204 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:26:14.204 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:58.998297+0000", 2024-09-21T12:26:14.204 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.204 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.204 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.204 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:26:14.205 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:58.998299+0000", 2024-09-21T12:26:14.205 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5829999999999999e-06 2024-09-21T12:26:14.205 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.205 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.205 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:26:14.205 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:58.998352+0000", 2024-09-21T12:26:14.205 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3068000000000001e-05 2024-09-21T12:26:14.205 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.205 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.205 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:26:14.205 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:58.998364+0000", 2024-09-21T12:26:14.205 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2105e-05 2024-09-21T12:26:14.205 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.205 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.205 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:26:14.206 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:58.998372+0000", 2024-09-21T12:26:14.206 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7340000000000008e-06 2024-09-21T12:26:14.206 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.206 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.206 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:26:14.206 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:58.998424+0000", 2024-09-21T12:26:14.206 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2556999999999998e-05 2024-09-21T12:26:14.206 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.206 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.206 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:26:14.206 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:58.998550+0000", 2024-09-21T12:26:14.206 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012533999999999999 2024-09-21T12:26:14.206 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.206 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.207 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:26:14.207 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:58.998683+0000", 2024-09-21T12:26:14.207 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000133521 2024-09-21T12:26:14.207 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.207 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.207 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:26:14.207 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.000234+0000", 2024-09-21T12:26:14.207 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0015507120000000001 2024-09-21T12:26:14.207 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.207 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.207 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.207 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.001510+0000", 2024-09-21T12:26:14.207 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012758050000000001 2024-09-21T12:26:14.207 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.207 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.208 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.208 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.080665+0000", 2024-09-21T12:26:14.208 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.079155602000000005 2024-09-21T12:26:14.208 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.208 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.208 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:26:14.208 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.080733+0000", 2024-09-21T12:26:14.208 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7799000000000006e-05 2024-09-21T12:26:14.208 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.208 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.208 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:26:14.208 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.080781+0000", 2024-09-21T12:26:14.208 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7797999999999999e-05 2024-09-21T12:26:14.208 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.209 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:26:14.209 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.209 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.209 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.209 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:45184 7.5 7:a309e6d9:::benchmark_data_smithi114_78432_object45183:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T12:26:14.209 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:59.002715+0000", 2024-09-21T12:26:14.209 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 255.165913698, 2024-09-21T12:26:14.209 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.085275684000000004, 2024-09-21T12:26:14.209 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:26:14.209 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:26:14.209 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:14.209 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:26:14.209 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:26:14.209 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:14.209 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 45184 2024-09-21T12:26:14.210 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.210 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:26:14.210 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.210 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:26:14.210 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.002715+0000", 2024-09-21T12:26:14.210 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.210 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.210 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.210 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:26:14.210 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.002715+0000", 2024-09-21T12:26:14.210 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.210 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.210 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.210 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:26:14.210 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.002716+0000", 2024-09-21T12:26:14.211 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2529999999999999e-06 2024-09-21T12:26:14.211 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.211 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.211 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:26:14.211 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.002783+0000", 2024-09-21T12:26:14.211 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6637999999999995e-05 2024-09-21T12:26:14.211 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.211 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.211 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:26:14.211 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.002795+0000", 2024-09-21T12:26:14.211 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1576999999999999e-05 2024-09-21T12:26:14.211 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.211 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.211 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:26:14.211 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.002802+0000", 2024-09-21T12:26:14.212 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4340000000000003e-06 2024-09-21T12:26:14.212 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.212 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.212 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:26:14.212 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.002863+0000", 2024-09-21T12:26:14.212 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1058999999999999e-05 2024-09-21T12:26:14.212 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.212 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.212 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:26:14.212 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.002952+0000", 2024-09-21T12:26:14.212 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8561999999999997e-05 2024-09-21T12:26:14.212 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.212 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.212 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:26:14.213 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.003037+0000", 2024-09-21T12:26:14.213 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5730999999999996e-05 2024-09-21T12:26:14.213 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.213 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.213 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:26:14.213 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.004105+0000", 2024-09-21T12:26:14.213 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010670650000000001 2024-09-21T12:26:14.213 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.213 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.213 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.213 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.049583+0000", 2024-09-21T12:26:14.213 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.045478867999999999 2024-09-21T12:26:14.213 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.213 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.213 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.214 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.087882+0000", 2024-09-21T12:26:14.214 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.038298454000000003 2024-09-21T12:26:14.214 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.214 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.214 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:26:14.214 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.087950+0000", 2024-09-21T12:26:14.214 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7972000000000006e-05 2024-09-21T12:26:14.214 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.214 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.214 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:26:14.214 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.087991+0000", 2024-09-21T12:26:14.214 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1071e-05 2024-09-21T12:26:14.214 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.214 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:26:14.215 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.215 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.215 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.215 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:48503 7.5 7:aed652f8:::benchmark_data_smithi114_78432_object48502:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:26:14.215 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:04.030415+0000", 2024-09-21T12:26:14.215 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 250.13821424, 2024-09-21T12:26:14.215 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.086380786000000001, 2024-09-21T12:26:14.215 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:26:14.215 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:26:14.215 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:14.215 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:26:14.215 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:26:14.215 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:14.215 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 48503 2024-09-21T12:26:14.216 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.216 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:26:14.216 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.216 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:26:14.216 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.030415+0000", 2024-09-21T12:26:14.216 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.216 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.216 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.216 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:26:14.216 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.030415+0000", 2024-09-21T12:26:14.216 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.216 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.216 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.216 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:26:14.217 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.030416+0000", 2024-09-21T12:26:14.217 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.437e-06 2024-09-21T12:26:14.217 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.217 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.217 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:26:14.217 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.030461+0000", 2024-09-21T12:26:14.217 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5272000000000003e-05 2024-09-21T12:26:14.217 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.217 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.217 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:26:14.217 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.030471+0000", 2024-09-21T12:26:14.217 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7480000000000005e-06 2024-09-21T12:26:14.217 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.217 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.217 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:26:14.218 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.030477+0000", 2024-09-21T12:26:14.218 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8119999999999996e-06 2024-09-21T12:26:14.218 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.218 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.218 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:26:14.218 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.030530+0000", 2024-09-21T12:26:14.218 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2809000000000002e-05 2024-09-21T12:26:14.218 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.218 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.218 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:26:14.218 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.030663+0000", 2024-09-21T12:26:14.218 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000133647 2024-09-21T12:26:14.218 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.218 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.218 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:26:14.219 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.030798+0000", 2024-09-21T12:26:14.219 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013433100000000001 2024-09-21T12:26:14.219 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.219 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.219 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:26:14.219 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.032271+0000", 2024-09-21T12:26:14.219 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001472945 2024-09-21T12:26:14.219 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.219 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.219 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.219 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.032510+0000", 2024-09-21T12:26:14.219 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00023884399999999999 2024-09-21T12:26:14.219 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.219 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.219 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.220 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.116713+0000", 2024-09-21T12:26:14.220 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.084203518000000005 2024-09-21T12:26:14.220 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.220 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.220 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:26:14.220 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.116756+0000", 2024-09-21T12:26:14.220 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2917999999999998e-05 2024-09-21T12:26:14.220 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.220 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.220 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:26:14.220 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.116795+0000", 2024-09-21T12:26:14.220 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9505000000000002e-05 2024-09-21T12:26:14.220 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.220 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:26:14.221 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.221 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.221 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.221 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:49232 7.5 7:a28409c5:::benchmark_data_smithi114_78432_object49231:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:26:14.221 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:04.569750+0000", 2024-09-21T12:26:14.221 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 249.59887872499999, 2024-09-21T12:26:14.221 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.147577404, 2024-09-21T12:26:14.221 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:26:14.221 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:26:14.221 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:14.221 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:26:14.221 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:26:14.221 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:14.221 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 49232 2024-09-21T12:26:14.221 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.222 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:26:14.222 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.222 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:26:14.222 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569750+0000", 2024-09-21T12:26:14.222 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.222 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.222 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.222 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:26:14.222 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569750+0000", 2024-09-21T12:26:14.222 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.222 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.222 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.222 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:26:14.222 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569752+0000", 2024-09-21T12:26:14.222 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6050000000000001e-06 2024-09-21T12:26:14.223 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.223 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.223 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:26:14.223 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569802+0000", 2024-09-21T12:26:14.223 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0664999999999997e-05 2024-09-21T12:26:14.223 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.223 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.223 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:26:14.223 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569819+0000", 2024-09-21T12:26:14.223 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6348e-05 2024-09-21T12:26:14.223 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.223 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.223 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:26:14.223 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569826+0000", 2024-09-21T12:26:14.223 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3830000000000003e-06 2024-09-21T12:26:14.224 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.224 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.224 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:26:14.224 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569882+0000", 2024-09-21T12:26:14.224 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5519000000000002e-05 2024-09-21T12:26:14.224 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.224 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.224 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:26:14.224 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.570014+0000", 2024-09-21T12:26:14.224 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013235499999999999 2024-09-21T12:26:14.224 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.224 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.224 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:26:14.224 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.570150+0000", 2024-09-21T12:26:14.224 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013635700000000001 2024-09-21T12:26:14.225 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.225 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.225 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:26:14.225 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.572290+0000", 2024-09-21T12:26:14.226 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021393050000000002 2024-09-21T12:26:14.226 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.226 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.226 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.226 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.575659+0000", 2024-09-21T12:26:14.226 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0033696720000000002 2024-09-21T12:26:14.226 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.226 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.226 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.226 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.717213+0000", 2024-09-21T12:26:14.227 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14155382899999999 2024-09-21T12:26:14.227 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.227 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.227 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:26:14.227 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.717282+0000", 2024-09-21T12:26:14.227 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8554999999999997e-05 2024-09-21T12:26:14.227 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.227 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.227 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:26:14.227 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.717328+0000", 2024-09-21T12:26:14.227 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5810999999999999e-05 2024-09-21T12:26:14.227 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.227 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:26:14.227 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.228 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.228 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.228 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:49947 7.5 7:a57824f9:::benchmark_data_smithi114_78432_object49946:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:26:14.228 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:05.215753+0000", 2024-09-21T12:26:14.228 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 248.95287542400001, 2024-09-21T12:26:14.228 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.173951617, 2024-09-21T12:26:14.228 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:26:14.228 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:26:14.228 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:14.228 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:26:14.228 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:26:14.228 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:14.228 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 49947 2024-09-21T12:26:14.228 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.228 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:26:14.229 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.229 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:26:14.229 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215753+0000", 2024-09-21T12:26:14.229 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.229 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.229 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.229 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:26:14.229 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215753+0000", 2024-09-21T12:26:14.229 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.229 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.229 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.229 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:26:14.229 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215755+0000", 2024-09-21T12:26:14.229 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.578e-06 2024-09-21T12:26:14.229 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.230 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.230 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:26:14.230 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215824+0000", 2024-09-21T12:26:14.230 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9362000000000004e-05 2024-09-21T12:26:14.230 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.230 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.230 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:26:14.230 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215835+0000", 2024-09-21T12:26:14.230 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0791999999999999e-05 2024-09-21T12:26:14.230 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.230 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.230 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:26:14.230 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215841+0000", 2024-09-21T12:26:14.230 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2430000000000002e-06 2024-09-21T12:26:14.230 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.231 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.231 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:26:14.231 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215884+0000", 2024-09-21T12:26:14.231 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2599000000000003e-05 2024-09-21T12:26:14.231 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.231 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.231 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:26:14.231 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215975+0000", 2024-09-21T12:26:14.231 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1390000000000004e-05 2024-09-21T12:26:14.231 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.231 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.231 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:26:14.231 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.216072+0000", 2024-09-21T12:26:14.231 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6669000000000006e-05 2024-09-21T12:26:14.232 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.232 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.232 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:26:14.232 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217412+0000", 2024-09-21T12:26:14.232 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001340171 2024-09-21T12:26:14.232 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.232 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.232 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.232 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.219412+0000", 2024-09-21T12:26:14.232 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001999883 2024-09-21T12:26:14.232 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.232 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.232 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.232 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.389620+0000", 2024-09-21T12:26:14.233 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.17020745300000001 2024-09-21T12:26:14.233 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.233 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.233 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:26:14.233 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.389671+0000", 2024-09-21T12:26:14.233 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1823999999999997e-05 2024-09-21T12:26:14.233 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.233 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.233 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:26:14.233 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.389705+0000", 2024-09-21T12:26:14.233 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3652999999999998e-05 2024-09-21T12:26:14.233 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.233 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:26:14.233 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.234 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.234 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.234 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:51201 7.5 7:a9552cc9:::benchmark_data_smithi114_78432_object51200:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:26:14.234 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:06.204175+0000", 2024-09-21T12:26:14.234 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 247.96445374999999, 2024-09-21T12:26:14.234 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.080261853999999994, 2024-09-21T12:26:14.234 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:26:14.234 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:26:14.234 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:14.234 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:26:14.234 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:26:14.234 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:14.234 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 51201 2024-09-21T12:26:14.234 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.235 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:26:14.235 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.235 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:26:14.235 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.204175+0000", 2024-09-21T12:26:14.235 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.235 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.235 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.235 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:26:14.235 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.204175+0000", 2024-09-21T12:26:14.235 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.235 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.235 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.235 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:26:14.235 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.204177+0000", 2024-09-21T12:26:14.236 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6759999999999999e-06 2024-09-21T12:26:14.236 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.236 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.236 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:26:14.236 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.204224+0000", 2024-09-21T12:26:14.236 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7623000000000001e-05 2024-09-21T12:26:14.236 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.236 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.236 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:26:14.236 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.204236+0000", 2024-09-21T12:26:14.236 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1637000000000001e-05 2024-09-21T12:26:14.236 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.236 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.236 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:26:14.237 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.204242+0000", 2024-09-21T12:26:14.237 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5820000000000001e-06 2024-09-21T12:26:14.237 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.237 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.237 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:26:14.237 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.204301+0000", 2024-09-21T12:26:14.237 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8986000000000001e-05 2024-09-21T12:26:14.237 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.237 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.237 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:26:14.237 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.204412+0000", 2024-09-21T12:26:14.237 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011177 2024-09-21T12:26:14.237 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.237 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.238 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:26:14.238 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.204521+0000", 2024-09-21T12:26:14.238 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010887699999999999 2024-09-21T12:26:14.238 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.238 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.238 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:26:14.238 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.206333+0000", 2024-09-21T12:26:14.238 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001811638 2024-09-21T12:26:14.238 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.238 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.238 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.238 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.207745+0000", 2024-09-21T12:26:14.238 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014116529999999999 2024-09-21T12:26:14.239 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.239 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.239 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.239 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.284357+0000", 2024-09-21T12:26:14.239 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.076612069000000005 2024-09-21T12:26:14.239 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.239 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.239 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:26:14.239 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.284400+0000", 2024-09-21T12:26:14.239 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3668000000000003e-05 2024-09-21T12:26:14.239 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.239 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.239 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:26:14.240 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.284437+0000", 2024-09-21T12:26:14.240 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6674999999999997e-05 2024-09-21T12:26:14.240 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.240 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:26:14.240 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.240 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.240 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.240 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:51710 7.5 7:a2195e36:::benchmark_data_smithi114_78432_object51709:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:26:14.240 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:06.588918+0000", 2024-09-21T12:26:14.240 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 247.57971073100001, 2024-09-21T12:26:14.240 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.083421032000000006, 2024-09-21T12:26:14.240 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:26:14.240 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:26:14.241 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:14.241 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:26:14.241 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:26:14.241 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:14.241 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 51710 2024-09-21T12:26:14.241 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.241 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:26:14.241 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.241 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:26:14.241 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.588918+0000", 2024-09-21T12:26:14.241 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.241 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.241 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.241 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:26:14.242 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.588918+0000", 2024-09-21T12:26:14.242 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.242 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.242 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.242 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:26:14.242 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.588920+0000", 2024-09-21T12:26:14.242 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.751e-06 2024-09-21T12:26:14.242 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.242 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.242 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:26:14.242 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.588976+0000", 2024-09-21T12:26:14.242 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5767000000000003e-05 2024-09-21T12:26:14.242 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.243 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.243 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:26:14.243 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.588995+0000", 2024-09-21T12:26:14.243 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8833000000000001e-05 2024-09-21T12:26:14.243 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.243 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.243 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:26:14.243 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.589004+0000", 2024-09-21T12:26:14.243 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2320000000000007e-06 2024-09-21T12:26:14.243 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.243 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.243 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:26:14.243 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.589075+0000", 2024-09-21T12:26:14.244 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1706999999999996e-05 2024-09-21T12:26:14.244 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.244 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.244 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:26:14.244 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.589205+0000", 2024-09-21T12:26:14.244 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012908799999999999 2024-09-21T12:26:14.244 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.244 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.244 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:26:14.244 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.589363+0000", 2024-09-21T12:26:14.244 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015867999999999999 2024-09-21T12:26:14.244 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.244 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.244 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:26:14.244 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.590869+0000", 2024-09-21T12:26:14.245 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0015061230000000001 2024-09-21T12:26:14.245 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.245 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.245 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.245 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.592836+0000", 2024-09-21T12:26:14.245 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0019670069999999998 2024-09-21T12:26:14.245 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.245 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.245 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.245 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.672241+0000", 2024-09-21T12:26:14.245 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.079404634000000002 2024-09-21T12:26:14.245 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.245 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.245 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:26:14.246 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.672297+0000", 2024-09-21T12:26:14.246 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6416000000000003e-05 2024-09-21T12:26:14.246 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.246 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.246 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:26:14.246 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.672339+0000", 2024-09-21T12:26:14.246 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1794e-05 2024-09-21T12:26:14.246 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.246 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:26:14.246 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.246 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.246 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.246 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:55405 7.5 7:a62aaadb:::benchmark_data_smithi114_78432_object55404:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-21T12:26:14.247 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:09.503352+0000", 2024-09-21T12:26:14.247 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 244.66527659299999, 2024-09-21T12:26:14.247 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.073823544000000005, 2024-09-21T12:26:14.247 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:26:14.247 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:26:14.247 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:14.247 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:26:14.247 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:26:14.247 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:14.247 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 55405 2024-09-21T12:26:14.247 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.247 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:26:14.247 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.247 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:26:14.248 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:09.503352+0000", 2024-09-21T12:26:14.248 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.248 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.248 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.248 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:26:14.248 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:09.503352+0000", 2024-09-21T12:26:14.248 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.248 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.248 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.248 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:26:14.248 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:09.503353+0000", 2024-09-21T12:26:14.248 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.144e-06 2024-09-21T12:26:14.248 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.248 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.248 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:26:14.249 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:09.503412+0000", 2024-09-21T12:26:14.249 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8941999999999999e-05 2024-09-21T12:26:14.249 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.249 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.249 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:26:14.249 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:09.503423+0000", 2024-09-21T12:26:14.249 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1036000000000001e-05 2024-09-21T12:26:14.249 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.249 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.249 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:26:14.249 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:09.503431+0000", 2024-09-21T12:26:14.249 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3860000000000001e-06 2024-09-21T12:26:14.249 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.249 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.249 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:26:14.250 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:09.503484+0000", 2024-09-21T12:26:14.250 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3188e-05 2024-09-21T12:26:14.250 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.250 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.250 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:26:14.250 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:09.503567+0000", 2024-09-21T12:26:14.250 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3178999999999998e-05 2024-09-21T12:26:14.250 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.250 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.250 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:26:14.250 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:09.503660+0000", 2024-09-21T12:26:14.250 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2363999999999999e-05 2024-09-21T12:26:14.250 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.250 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.251 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:26:14.251 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:09.504991+0000", 2024-09-21T12:26:14.251 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001331334 2024-09-21T12:26:14.251 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.251 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.251 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.251 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:09.506471+0000", 2024-09-21T12:26:14.251 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001480213 2024-09-21T12:26:14.251 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.251 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.251 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.251 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:09.577110+0000", 2024-09-21T12:26:14.251 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.070639088000000003 2024-09-21T12:26:14.251 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.251 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.252 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:26:14.252 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:09.577146+0000", 2024-09-21T12:26:14.252 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6041000000000003e-05 2024-09-21T12:26:14.252 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.252 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.252 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:26:14.252 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:09.577176+0000", 2024-09-21T12:26:14.252 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9629e-05 2024-09-21T12:26:14.252 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.252 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:26:14.252 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.252 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.252 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.252 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:76910 7.5 7:a2a1b392:::benchmark_data_smithi114_78432_object76909:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-21T12:26:14.252 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:29.843487+0000", 2024-09-21T12:26:14.253 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 224.32514232400001, 2024-09-21T12:26:14.253 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.080587959000000001, 2024-09-21T12:26:14.253 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:26:14.253 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:26:14.253 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:14.253 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:26:14.253 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:26:14.253 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:14.253 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 76910 2024-09-21T12:26:14.253 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.253 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:26:14.253 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.253 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:26:14.253 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.843487+0000", 2024-09-21T12:26:14.253 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.254 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.254 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.254 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:26:14.254 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.843487+0000", 2024-09-21T12:26:14.254 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.254 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.254 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.254 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:26:14.254 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.843488+0000", 2024-09-21T12:26:14.254 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.314e-06 2024-09-21T12:26:14.254 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.254 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.254 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:26:14.254 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.843533+0000", 2024-09-21T12:26:14.255 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4628e-05 2024-09-21T12:26:14.255 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.255 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.255 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:26:14.255 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.843542+0000", 2024-09-21T12:26:14.255 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9480000000000003e-06 2024-09-21T12:26:14.255 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.255 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.255 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:26:14.255 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.843548+0000", 2024-09-21T12:26:14.255 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6289999999999998e-06 2024-09-21T12:26:14.255 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.255 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.255 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:26:14.256 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.843590+0000", 2024-09-21T12:26:14.256 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2070000000000002e-05 2024-09-21T12:26:14.256 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.256 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.256 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:26:14.256 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.843681+0000", 2024-09-21T12:26:14.256 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0459999999999998e-05 2024-09-21T12:26:14.256 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.256 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.256 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:26:14.256 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.843768+0000", 2024-09-21T12:26:14.256 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7200999999999994e-05 2024-09-21T12:26:14.256 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.256 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.256 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:26:14.257 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.847962+0000", 2024-09-21T12:26:14.257 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0041937830000000004 2024-09-21T12:26:14.257 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.257 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.257 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.257 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.850111+0000", 2024-09-21T12:26:14.257 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021488990000000001 2024-09-21T12:26:14.257 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.257 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.257 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.257 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.923993+0000", 2024-09-21T12:26:14.257 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.073882858999999995 2024-09-21T12:26:14.257 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.257 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.258 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:26:14.258 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.924032+0000", 2024-09-21T12:26:14.258 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8432000000000001e-05 2024-09-21T12:26:14.258 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.258 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.258 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:26:14.258 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.924075+0000", 2024-09-21T12:26:14.258 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2735999999999999e-05 2024-09-21T12:26:14.258 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.258 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:26:14.258 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.258 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.258 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.258 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:76915 7.5 7:af00b811:::benchmark_data_smithi114_78432_object76914:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-21T12:26:14.259 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:29.845405+0000", 2024-09-21T12:26:14.259 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 224.32322406, 2024-09-21T12:26:14.259 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.092908165000000001, 2024-09-21T12:26:14.259 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:26:14.259 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:26:14.259 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:14.259 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:26:14.259 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:26:14.259 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:14.259 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 76915 2024-09-21T12:26:14.259 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.259 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:26:14.259 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.259 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:26:14.260 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.845405+0000", 2024-09-21T12:26:14.260 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.260 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.260 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.260 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:26:14.260 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.845405+0000", 2024-09-21T12:26:14.260 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.260 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.260 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.260 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:26:14.260 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.845406+0000", 2024-09-21T12:26:14.260 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.626e-06 2024-09-21T12:26:14.260 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.260 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.260 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:26:14.260 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.845465+0000", 2024-09-21T12:26:14.261 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8341000000000003e-05 2024-09-21T12:26:14.261 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.261 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.261 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:26:14.261 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.845482+0000", 2024-09-21T12:26:14.261 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7076999999999999e-05 2024-09-21T12:26:14.261 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.261 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.261 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:26:14.261 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.845491+0000", 2024-09-21T12:26:14.261 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7919999999999998e-06 2024-09-21T12:26:14.261 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.261 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.262 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:26:14.262 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.845687+0000", 2024-09-21T12:26:14.262 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019604900000000001 2024-09-21T12:26:14.262 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.262 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.262 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:26:14.262 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.845805+0000", 2024-09-21T12:26:14.262 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000117836 2024-09-21T12:26:14.262 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.262 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.262 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:26:14.262 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.845934+0000", 2024-09-21T12:26:14.262 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012931600000000001 2024-09-21T12:26:14.263 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.263 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.263 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:26:14.263 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.848128+0000", 2024-09-21T12:26:14.263 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021943330000000001 2024-09-21T12:26:14.263 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.263 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.263 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.263 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.850159+0000", 2024-09-21T12:26:14.263 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020305369999999998 2024-09-21T12:26:14.263 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.263 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.263 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.263 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.938214+0000", 2024-09-21T12:26:14.263 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.088055156999999995 2024-09-21T12:26:14.264 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.264 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.264 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:26:14.264 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.938271+0000", 2024-09-21T12:26:14.264 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6588000000000001e-05 2024-09-21T12:26:14.264 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.264 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.264 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:26:14.264 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.938313+0000", 2024-09-21T12:26:14.264 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2512999999999998e-05 2024-09-21T12:26:14.264 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.264 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:26:14.264 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.264 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.264 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.265 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:83603 7.5 7:a477e0d2:::benchmark_data_smithi114_78432_object83602:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-21T12:26:14.265 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:36.922642+0000", 2024-09-21T12:26:14.265 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 217.245986644, 2024-09-21T12:26:14.265 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.071816557000000003, 2024-09-21T12:26:14.265 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:26:14.265 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:26:14.265 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:14.265 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:26:14.265 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:26:14.265 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:14.265 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 83603 2024-09-21T12:26:14.265 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.265 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:26:14.265 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.266 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:26:14.266 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.922642+0000", 2024-09-21T12:26:14.266 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.266 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.266 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.266 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:26:14.266 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.922642+0000", 2024-09-21T12:26:14.266 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.266 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.266 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.266 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:26:14.266 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.922644+0000", 2024-09-21T12:26:14.266 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9589999999999998e-06 2024-09-21T12:26:14.266 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.267 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.267 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:26:14.267 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.922764+0000", 2024-09-21T12:26:14.267 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012024200000000001 2024-09-21T12:26:14.267 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.267 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.267 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:26:14.267 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.922784+0000", 2024-09-21T12:26:14.267 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9925999999999999e-05 2024-09-21T12:26:14.267 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.267 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.267 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:26:14.267 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.922793+0000", 2024-09-21T12:26:14.267 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2530000000000004e-06 2024-09-21T12:26:14.268 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.268 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.268 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:26:14.268 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.923531+0000", 2024-09-21T12:26:14.268 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00073868499999999999 2024-09-21T12:26:14.268 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.268 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.268 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:26:14.268 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.923853+0000", 2024-09-21T12:26:14.268 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00032182000000000001 2024-09-21T12:26:14.268 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.268 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.268 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:26:14.269 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.924101+0000", 2024-09-21T12:26:14.269 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000248216 2024-09-21T12:26:14.269 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.269 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.269 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:26:14.269 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.932031+0000", 2024-09-21T12:26:14.269 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0079295170000000005 2024-09-21T12:26:14.269 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.269 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.269 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.269 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.990474+0000", 2024-09-21T12:26:14.269 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.058443551000000003 2024-09-21T12:26:14.269 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.269 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.269 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.270 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.994327+0000", 2024-09-21T12:26:14.270 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0038525600000000001 2024-09-21T12:26:14.270 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.270 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.270 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:26:14.270 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.994408+0000", 2024-09-21T12:26:14.270 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.106e-05 2024-09-21T12:26:14.270 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.270 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.270 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:26:14.270 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.994459+0000", 2024-09-21T12:26:14.270 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0767999999999999e-05 2024-09-21T12:26:14.270 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.270 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:26:14.271 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.271 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.271 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.271 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:84081 7.5 7:a05cd650:::benchmark_data_smithi114_78432_object84080:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-21T12:26:14.271 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:37.466269+0000", 2024-09-21T12:26:14.271 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 216.70235977300001, 2024-09-21T12:26:14.271 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.072265354000000004, 2024-09-21T12:26:14.271 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:26:14.271 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:26:14.271 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:14.271 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:26:14.271 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:26:14.271 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:14.271 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 84081 2024-09-21T12:26:14.271 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.272 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:26:14.272 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.272 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:26:14.272 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:37.466269+0000", 2024-09-21T12:26:14.272 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.272 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.272 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.272 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:26:14.272 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:37.466269+0000", 2024-09-21T12:26:14.272 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.272 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.272 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.272 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:26:14.272 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:37.466270+0000", 2024-09-21T12:26:14.273 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2200000000000002e-07 2024-09-21T12:26:14.273 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.273 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.273 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:26:14.273 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:37.466330+0000", 2024-09-21T12:26:14.273 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9583000000000002e-05 2024-09-21T12:26:14.273 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.273 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.273 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:26:14.273 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:37.466342+0000", 2024-09-21T12:26:14.273 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1929999999999999e-05 2024-09-21T12:26:14.273 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.273 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.273 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:26:14.274 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:37.466349+0000", 2024-09-21T12:26:14.274 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9430000000000003e-06 2024-09-21T12:26:14.274 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.274 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.274 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:26:14.274 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:37.468107+0000", 2024-09-21T12:26:14.274 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001758608 2024-09-21T12:26:14.274 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.274 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.274 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:26:14.274 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:37.468229+0000", 2024-09-21T12:26:14.274 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012149700000000001 2024-09-21T12:26:14.274 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.275 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.275 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:26:14.275 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:37.468375+0000", 2024-09-21T12:26:14.275 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000146837 2024-09-21T12:26:14.275 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.275 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.275 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:26:14.275 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:37.470232+0000", 2024-09-21T12:26:14.275 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00185626 2024-09-21T12:26:14.275 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.275 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.275 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.275 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:37.476327+0000", 2024-09-21T12:26:14.275 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.006095591 2024-09-21T12:26:14.275 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.276 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.276 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.276 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:37.538422+0000", 2024-09-21T12:26:14.276 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.062094804000000003 2024-09-21T12:26:14.276 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.276 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.276 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:26:14.276 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:37.538486+0000", 2024-09-21T12:26:14.276 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3501000000000001e-05 2024-09-21T12:26:14.276 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.276 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.276 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:26:14.276 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:37.538535+0000", 2024-09-21T12:26:14.276 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8878000000000002e-05 2024-09-21T12:26:14.277 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.277 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:26:14.277 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.277 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.277 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.277 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:150335 7.5 7:aef4e123:::benchmark_data_smithi114_78432_object46189:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T12:26:14.277 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:23:42.614761+0000", 2024-09-21T12:26:14.277 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 151.55386789600001, 2024-09-21T12:26:14.277 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13488366900000001, 2024-09-21T12:26:14.277 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:26:14.277 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:26:14.277 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:14.277 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:26:14.277 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:26:14.278 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:14.278 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 150335 2024-09-21T12:26:14.278 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.278 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:26:14.278 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.278 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:26:14.278 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.614761+0000", 2024-09-21T12:26:14.278 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.278 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.278 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.278 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:26:14.278 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.614761+0000", 2024-09-21T12:26:14.278 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.278 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.279 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.279 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:26:14.279 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.614763+0000", 2024-09-21T12:26:14.279 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6500000000000001e-06 2024-09-21T12:26:14.279 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.279 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.279 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:26:14.279 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.614768+0000", 2024-09-21T12:26:14.279 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.101e-06 2024-09-21T12:26:14.279 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.279 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.279 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:26:14.279 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.614781+0000", 2024-09-21T12:26:14.280 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3572e-05 2024-09-21T12:26:14.280 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.280 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.280 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:26:14.280 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.614788+0000", 2024-09-21T12:26:14.280 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7279999999999998e-06 2024-09-21T12:26:14.280 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.280 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.280 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:26:14.280 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.614853+0000", 2024-09-21T12:26:14.280 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4647999999999995e-05 2024-09-21T12:26:14.280 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.280 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.280 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:26:14.281 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.614963+0000", 2024-09-21T12:26:14.281 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011069399999999999 2024-09-21T12:26:14.281 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.281 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.281 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:26:14.281 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.615070+0000", 2024-09-21T12:26:14.281 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010666999999999999 2024-09-21T12:26:14.281 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.281 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.281 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:26:14.281 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.617101+0000", 2024-09-21T12:26:14.281 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020314040000000001 2024-09-21T12:26:14.281 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.281 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.281 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.282 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.617869+0000", 2024-09-21T12:26:14.282 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00076795400000000005 2024-09-21T12:26:14.282 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.282 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.282 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.282 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.749524+0000", 2024-09-21T12:26:14.282 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.131655037 2024-09-21T12:26:14.282 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.282 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.282 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:26:14.282 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.749590+0000", 2024-09-21T12:26:14.282 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5864999999999994e-05 2024-09-21T12:26:14.282 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.282 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.283 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:26:14.283 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.749645+0000", 2024-09-21T12:26:14.283 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4345999999999998e-05 2024-09-21T12:26:14.283 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.283 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:26:14.283 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.283 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.283 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.283 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:150337 7.5 7:aae8cbea:::benchmark_data_smithi114_78432_object46191:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T12:26:14.283 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:23:42.617985+0000", 2024-09-21T12:26:14.283 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 151.55064366799999, 2024-09-21T12:26:14.283 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13997900399999999, 2024-09-21T12:26:14.283 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:26:14.283 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:26:14.283 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:14.284 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:26:14.284 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:26:14.284 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:14.284 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 150337 2024-09-21T12:26:14.284 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.284 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:26:14.284 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.284 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:26:14.284 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.617985+0000", 2024-09-21T12:26:14.284 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.284 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.284 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.284 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:26:14.284 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.617985+0000", 2024-09-21T12:26:14.285 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.285 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.285 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.285 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:26:14.285 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.617986+0000", 2024-09-21T12:26:14.285 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1289999999999999e-06 2024-09-21T12:26:14.285 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.285 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.285 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:26:14.285 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.617991+0000", 2024-09-21T12:26:14.285 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5870000000000003e-06 2024-09-21T12:26:14.285 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.285 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.285 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:26:14.286 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618001+0000", 2024-09-21T12:26:14.286 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0370000000000001e-05 2024-09-21T12:26:14.286 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.286 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.286 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:26:14.286 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618007+0000", 2024-09-21T12:26:14.286 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.215e-06 2024-09-21T12:26:14.286 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.286 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.286 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:26:14.286 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618052+0000", 2024-09-21T12:26:14.286 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5525999999999997e-05 2024-09-21T12:26:14.286 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.286 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.287 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:26:14.287 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618122+0000", 2024-09-21T12:26:14.287 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9733000000000005e-05 2024-09-21T12:26:14.287 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.287 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.287 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:26:14.287 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618190+0000", 2024-09-21T12:26:14.287 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8684999999999997e-05 2024-09-21T12:26:14.287 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.287 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.287 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:26:14.287 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.620946+0000", 2024-09-21T12:26:14.287 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002755965 2024-09-21T12:26:14.287 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.288 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.288 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.288 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.621448+0000", 2024-09-21T12:26:14.288 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00050168700000000001 2024-09-21T12:26:14.288 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.288 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.288 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.288 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.757862+0000", 2024-09-21T12:26:14.288 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13641423799999999 2024-09-21T12:26:14.288 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.288 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.288 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:26:14.288 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.757920+0000", 2024-09-21T12:26:14.288 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7630999999999998e-05 2024-09-21T12:26:14.288 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.289 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.289 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:26:14.289 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.757964+0000", 2024-09-21T12:26:14.289 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4237999999999999e-05 2024-09-21T12:26:14.289 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.289 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:26:14.289 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.289 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.289 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.289 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:150339 7.5 7:ac9cca71:::benchmark_data_smithi114_78432_object46193:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T12:26:14.289 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:23:42.618551+0000", 2024-09-21T12:26:14.289 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 151.550077582, 2024-09-21T12:26:14.289 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.139580281, 2024-09-21T12:26:14.289 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:26:14.290 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:26:14.290 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:14.290 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:26:14.290 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:26:14.290 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:14.290 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 150339 2024-09-21T12:26:14.290 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.290 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:26:14.290 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.290 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:26:14.290 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618551+0000", 2024-09-21T12:26:14.290 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.290 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.290 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.291 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:26:14.291 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618551+0000", 2024-09-21T12:26:14.291 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:26:14.291 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.291 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.291 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:26:14.291 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618552+0000", 2024-09-21T12:26:14.291 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8200000000000008e-07 2024-09-21T12:26:14.291 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.291 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.291 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:26:14.291 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618557+0000", 2024-09-21T12:26:14.291 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3560000000000003e-06 2024-09-21T12:26:14.291 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.291 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.292 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:26:14.292 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618570+0000", 2024-09-21T12:26:14.292 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3118e-05 2024-09-21T12:26:14.292 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.292 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.292 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:26:14.292 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618577+0000", 2024-09-21T12:26:14.292 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9149999999999999e-06 2024-09-21T12:26:14.292 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.292 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.292 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:26:14.292 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.619781+0000", 2024-09-21T12:26:14.292 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012041790000000001 2024-09-21T12:26:14.293 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.293 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.293 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:26:14.293 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.619857+0000", 2024-09-21T12:26:14.293 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6151999999999999e-05 2024-09-21T12:26:14.293 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.293 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.293 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:26:14.293 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.619919+0000", 2024-09-21T12:26:14.293 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.19e-05 2024-09-21T12:26:14.293 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.293 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.293 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:26:14.293 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.621027+0000", 2024-09-21T12:26:14.293 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001108211 2024-09-21T12:26:14.294 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.294 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.294 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.294 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.621491+0000", 2024-09-21T12:26:14.294 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00046415200000000001 2024-09-21T12:26:14.294 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.294 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.294 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:14.294 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.758040+0000", 2024-09-21T12:26:14.294 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13654828999999999 2024-09-21T12:26:14.294 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.294 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.294 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:26:14.294 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.758095+0000", 2024-09-21T12:26:14.295 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5386999999999997e-05 2024-09-21T12:26:14.295 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:26:14.295 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:26:14.295 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:26:14.295 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.758132+0000", 2024-09-21T12:26:14.295 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6638999999999999e-05 2024-09-21T12:26:14.295 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.295 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:26:14.295 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.295 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:26:14.295 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:26:14.295 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:26:14.296 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:14.324 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:14.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:14.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:14.427 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:14.463 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:14.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:14.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:14.564 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:14.601 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:14.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:14.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:14.702 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:14.735 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:14.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:14.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:14.837 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:14.866 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:14.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:14.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:14.939 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T12:26:14.968 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:14.996 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:14.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:14.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:15.012 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:14 smithi114 ceph-mon[40867]: pgmap v288: 158 pgs: 158 active+clean; 7.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 752 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-21T12:26:15.012 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:14 smithi114 ceph-mon[40867]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:26:15.012 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:14 smithi114 ceph-mon[40867]: Cluster is now healthy 2024-09-21T12:26:15.012 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:14 smithi114 ceph-mon[40867]: 3.a scrub starts 2024-09-21T12:26:15.012 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:14 smithi114 ceph-mon[40867]: 3.a scrub ok 2024-09-21T12:26:15.012 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:14 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:26:15.012 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:14 smithi114 ceph-mon[40867]: osdmap e197: 8 total, 8 up, 7 in 2024-09-21T12:26:15.012 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:14 smithi114 ceph-mon[40867]: 2.1 scrub starts 2024-09-21T12:26:15.012 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:14 smithi114 ceph-mon[40867]: 2.1 scrub ok 2024-09-21T12:26:15.012 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:14 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/179971319' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:26:15.012 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:14 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:26:15.012 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:14 smithi114 ceph-mon[40867]: 5.0 scrub starts 2024-09-21T12:26:15.013 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:14 smithi114 ceph-mon[40867]: 5.0 scrub ok 2024-09-21T12:26:15.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:14 smithi114 ceph-mon[30324]: pgmap v288: 158 pgs: 158 active+clean; 7.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 752 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-21T12:26:15.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:14 smithi114 ceph-mon[30324]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:26:15.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:14 smithi114 ceph-mon[30324]: Cluster is now healthy 2024-09-21T12:26:15.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:14 smithi114 ceph-mon[30324]: 3.a scrub starts 2024-09-21T12:26:15.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:14 smithi114 ceph-mon[30324]: 3.a scrub ok 2024-09-21T12:26:15.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:14 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:26:15.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:14 smithi114 ceph-mon[30324]: osdmap e197: 8 total, 8 up, 7 in 2024-09-21T12:26:15.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:14 smithi114 ceph-mon[30324]: 2.1 scrub starts 2024-09-21T12:26:15.014 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:14 smithi114 ceph-mon[30324]: 2.1 scrub ok 2024-09-21T12:26:15.014 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:14 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/179971319' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:26:15.014 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:14 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:26:15.014 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:14 smithi114 ceph-mon[30324]: 5.0 scrub starts 2024-09-21T12:26:15.014 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:14 smithi114 ceph-mon[30324]: 5.0 scrub ok 2024-09-21T12:26:15.056 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 81 16 119711 119695 92.3332 80.6875 0.0138452 0.0108258 2024-09-21T12:26:15.098 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:15.119 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T12:26:15.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:15.136 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:15.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:15.237 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:15.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:14 smithi183 ceph-mon[33277]: pgmap v288: 158 pgs: 158 active+clean; 7.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 752 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-21T12:26:15.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:14 smithi183 ceph-mon[33277]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:26:15.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:14 smithi183 ceph-mon[33277]: Cluster is now healthy 2024-09-21T12:26:15.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:14 smithi183 ceph-mon[33277]: 3.a scrub starts 2024-09-21T12:26:15.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:14 smithi183 ceph-mon[33277]: 3.a scrub ok 2024-09-21T12:26:15.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:14 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:26:15.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:14 smithi183 ceph-mon[33277]: osdmap e197: 8 total, 8 up, 7 in 2024-09-21T12:26:15.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:14 smithi183 ceph-mon[33277]: 2.1 scrub starts 2024-09-21T12:26:15.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:14 smithi183 ceph-mon[33277]: 2.1 scrub ok 2024-09-21T12:26:15.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:14 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/179971319' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:26:15.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:14 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:26:15.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:14 smithi183 ceph-mon[33277]: 5.0 scrub starts 2024-09-21T12:26:15.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:14 smithi183 ceph-mon[33277]: 5.0 scrub ok 2024-09-21T12:26:15.285 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:15.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:15.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:15.387 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:15.435 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:15.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:15.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:15.536 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:15.567 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:15.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:15.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:15.668 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:15.695 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:15.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:15.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:15.791 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T12:26:15.797 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:15.873 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:15.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:15.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:15.975 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:16.014 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:16.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:16.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:16.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:15 smithi114 ceph-mon[40867]: 4.3 scrub starts 2024-09-21T12:26:16.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:15 smithi114 ceph-mon[40867]: 4.3 scrub ok 2024-09-21T12:26:16.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:15 smithi114 ceph-mon[40867]: 6.12 scrub starts 2024-09-21T12:26:16.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:15 smithi114 ceph-mon[40867]: 6.12 scrub ok 2024-09-21T12:26:16.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:15 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T12:26:16.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:15 smithi114 ceph-mon[40867]: osdmap e198: 8 total, 8 up, 7 in 2024-09-21T12:26:16.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:15 smithi114 ceph-mon[40867]: 6.1d scrub starts 2024-09-21T12:26:16.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:15 smithi114 ceph-mon[40867]: 6.1d scrub ok 2024-09-21T12:26:16.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:15 smithi114 ceph-mon[30324]: 4.3 scrub starts 2024-09-21T12:26:16.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:15 smithi114 ceph-mon[30324]: 4.3 scrub ok 2024-09-21T12:26:16.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:15 smithi114 ceph-mon[30324]: 6.12 scrub starts 2024-09-21T12:26:16.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:15 smithi114 ceph-mon[30324]: 6.12 scrub ok 2024-09-21T12:26:16.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:15 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T12:26:16.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:15 smithi114 ceph-mon[30324]: osdmap e198: 8 total, 8 up, 7 in 2024-09-21T12:26:16.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:15 smithi114 ceph-mon[30324]: 6.1d scrub starts 2024-09-21T12:26:16.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:15 smithi114 ceph-mon[30324]: 6.1d scrub ok 2024-09-21T12:26:16.056 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 82 15 120931 120916 92.1378 76.3125 0.00704757 0.0108483 2024-09-21T12:26:16.116 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:16.143 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:16.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:16.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:16.152 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:26:16.152 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:26:16.208 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:26:16.208 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:26:16.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:15 smithi183 ceph-mon[33277]: 4.3 scrub starts 2024-09-21T12:26:16.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:15 smithi183 ceph-mon[33277]: 4.3 scrub ok 2024-09-21T12:26:16.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:15 smithi183 ceph-mon[33277]: 6.12 scrub starts 2024-09-21T12:26:16.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:15 smithi183 ceph-mon[33277]: 6.12 scrub ok 2024-09-21T12:26:16.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:15 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T12:26:16.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:15 smithi183 ceph-mon[33277]: osdmap e198: 8 total, 8 up, 7 in 2024-09-21T12:26:16.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:15 smithi183 ceph-mon[33277]: 6.1d scrub starts 2024-09-21T12:26:16.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:15 smithi183 ceph-mon[33277]: 6.1d scrub ok 2024-09-21T12:26:16.240 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:26:16.240 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:26:16.245 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:16.268 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:16.272 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:16.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:16.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:16.275 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:26:16.276 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:26:16.303 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:16.308 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:26:16.308 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:26:16.347 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:26:16.347 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:26:16.375 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:16.380 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:26:16.380 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:26:16.402 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:16.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:16.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:16.412 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:26:16.413 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:26:16.504 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:16.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:16.541 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:16.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:16.642 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:16.680 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:16.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:16.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:16.774 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T12:26:16.782 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:16.810 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:16.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:16.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:16.912 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:16.937 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:16.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:16.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:17.038 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:17.057 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 83 16 122039 122023 91.8611 69.1875 0.0100038 0.0108816 2024-09-21T12:26:17.068 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:17.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:17.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:17.170 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:17.203 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:17.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:17.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:17.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:26:16 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:26:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:16 smithi183 ceph-mon[33277]: pgmap v291: 158 pgs: 158 active+clean; 7.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 852 B/s rd, 61 MiB/s wr, 983 op/s 2024-09-21T12:26:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:16 smithi183 ceph-mon[33277]: 4.e scrub starts 2024-09-21T12:26:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:16 smithi183 ceph-mon[33277]: 4.e scrub ok 2024-09-21T12:26:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:16 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2584432667' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:26:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:16 smithi183 ceph-mon[33277]: 5.12 scrub starts 2024-09-21T12:26:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:16 smithi183 ceph-mon[33277]: 5.12 scrub ok 2024-09-21T12:26:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:16 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:26:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:16 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:26:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:16 smithi183 ceph-mon[33277]: 3.5 scrub starts 2024-09-21T12:26:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:16 smithi183 ceph-mon[33277]: 3.5 scrub ok 2024-09-21T12:26:17.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:16 smithi114 ceph-mon[30324]: pgmap v291: 158 pgs: 158 active+clean; 7.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 852 B/s rd, 61 MiB/s wr, 983 op/s 2024-09-21T12:26:17.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:16 smithi114 ceph-mon[30324]: 4.e scrub starts 2024-09-21T12:26:17.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:16 smithi114 ceph-mon[30324]: 4.e scrub ok 2024-09-21T12:26:17.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:16 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2584432667' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:26:17.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:16 smithi114 ceph-mon[30324]: 5.12 scrub starts 2024-09-21T12:26:17.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:16 smithi114 ceph-mon[30324]: 5.12 scrub ok 2024-09-21T12:26:17.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:16 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:26:17.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:16 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:26:17.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:16 smithi114 ceph-mon[30324]: 3.5 scrub starts 2024-09-21T12:26:17.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:16 smithi114 ceph-mon[30324]: 3.5 scrub ok 2024-09-21T12:26:17.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:16 smithi114 ceph-mon[40867]: pgmap v291: 158 pgs: 158 active+clean; 7.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 852 B/s rd, 61 MiB/s wr, 983 op/s 2024-09-21T12:26:17.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:16 smithi114 ceph-mon[40867]: 4.e scrub starts 2024-09-21T12:26:17.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:16 smithi114 ceph-mon[40867]: 4.e scrub ok 2024-09-21T12:26:17.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:16 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2584432667' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:26:17.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:16 smithi114 ceph-mon[40867]: 5.12 scrub starts 2024-09-21T12:26:17.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:16 smithi114 ceph-mon[40867]: 5.12 scrub ok 2024-09-21T12:26:17.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:16 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:26:17.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:16 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:26:17.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:16 smithi114 ceph-mon[40867]: 3.5 scrub starts 2024-09-21T12:26:17.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:16 smithi114 ceph-mon[40867]: 3.5 scrub ok 2024-09-21T12:26:17.305 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:17.346 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:17.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:17.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:17.448 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:17.477 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:17.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:17.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:17.578 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:17.608 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:17.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:17.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:17.710 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:17.748 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:17.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:17.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:17.851 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:17.877 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:17.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:17.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:17.979 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:18.002 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:18.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:18.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:18.057 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 84 16 123141 123125 91.5875 68.875 0.0274021 0.0109129 2024-09-21T12:26:18.104 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:18.142 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:18.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:18.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:18.189 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:26:18.189 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:26:18.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:17 smithi183 ceph-mon[33277]: 3.11 scrub starts 2024-09-21T12:26:18.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:17 smithi183 ceph-mon[33277]: 3.11 scrub ok 2024-09-21T12:26:18.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:17 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:26:18.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:17 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:26:18.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:17 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:26:18.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:17 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:26:18.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:17 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:26:18.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:17 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:26:18.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:17 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:26:18.244 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:18.270 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:18.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:18.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:18.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:17 smithi114 ceph-mon[40867]: 3.11 scrub starts 2024-09-21T12:26:18.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:17 smithi114 ceph-mon[40867]: 3.11 scrub ok 2024-09-21T12:26:18.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:17 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:26:18.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:17 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:26:18.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:17 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:26:18.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:17 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:26:18.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:17 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:26:18.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:17 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:26:18.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:17 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:26:18.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:17 smithi114 ceph-mon[30324]: 3.11 scrub starts 2024-09-21T12:26:18.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:17 smithi114 ceph-mon[30324]: 3.11 scrub ok 2024-09-21T12:26:18.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:17 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:26:18.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:17 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:26:18.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:17 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:26:18.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:17 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:26:18.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:17 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:26:18.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:17 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:26:18.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:17 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:26:18.372 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:18.411 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:18.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:18.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:18.514 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:18.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:18.551 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:18.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:18.652 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:18.679 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:18.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:18.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:18.781 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:18.811 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:18.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:18.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:18.913 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:18.944 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T12:26:18.965 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:18.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:18.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:18.983 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T12:26:19.057 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 85 15 124337 124322 91.3897 74.8125 0.0073493 0.0109377 2024-09-21T12:26:19.067 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:19.094 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:19.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:19.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:19.196 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:19.216 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:19.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:19.274 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:19.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:19.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:18 smithi114 ceph-mon[30324]: pgmap v292: 158 pgs: 158 active+clean; 7.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-21T12:26:19.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:18 smithi114 ceph-mon[30324]: 3.19 scrub starts 2024-09-21T12:26:19.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:18 smithi114 ceph-mon[30324]: 3.19 scrub ok 2024-09-21T12:26:19.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:18 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2759021822' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:26:19.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:18 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:26:19.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:18 smithi114 ceph-mon[30324]: 4.15 scrub starts 2024-09-21T12:26:19.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:18 smithi114 ceph-mon[30324]: 4.15 scrub ok 2024-09-21T12:26:19.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:18 smithi114 ceph-mon[40867]: pgmap v292: 158 pgs: 158 active+clean; 7.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-21T12:26:19.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:18 smithi114 ceph-mon[40867]: 3.19 scrub starts 2024-09-21T12:26:19.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:18 smithi114 ceph-mon[40867]: 3.19 scrub ok 2024-09-21T12:26:19.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:18 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2759021822' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:26:19.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:18 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:26:19.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:18 smithi114 ceph-mon[40867]: 4.15 scrub starts 2024-09-21T12:26:19.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:18 smithi114 ceph-mon[40867]: 4.15 scrub ok 2024-09-21T12:26:19.375 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:19.419 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:26:19.420 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:19.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:19.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:18 smithi183 ceph-mon[33277]: pgmap v292: 158 pgs: 158 active+clean; 7.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-21T12:26:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:18 smithi183 ceph-mon[33277]: 3.19 scrub starts 2024-09-21T12:26:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:18 smithi183 ceph-mon[33277]: 3.19 scrub ok 2024-09-21T12:26:19.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:18 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2759021822' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:26:19.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:18 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:26:19.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:18 smithi183 ceph-mon[33277]: 4.15 scrub starts 2024-09-21T12:26:19.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:18 smithi183 ceph-mon[33277]: 4.15 scrub ok 2024-09-21T12:26:19.522 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:19.548 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:19.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:19.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:19.649 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:19.678 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:19.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:19.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:19.780 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:19.810 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:19.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:19.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:19.912 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:19.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:19.944 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:19.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:20.045 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:20.060 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 86 16 125602 125586 91.2432 79 0.0078162 0.0109548 2024-09-21T12:26:20.087 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:20.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:20.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:20.189 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:20.212 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:20.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:20.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:20.234 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T12:26:20.234 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:26:20.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:20 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T12:26:20.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:20 smithi114 ceph-mon[40867]: osdmap e199: 8 total, 8 up, 7 in 2024-09-21T12:26:20.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:20 smithi114 ceph-mon[40867]: 3.1d scrub starts 2024-09-21T12:26:20.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:20 smithi114 ceph-mon[40867]: 3.1d scrub ok 2024-09-21T12:26:20.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:20 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T12:26:20.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:20 smithi114 ceph-mon[30324]: osdmap e199: 8 total, 8 up, 7 in 2024-09-21T12:26:20.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:20 smithi114 ceph-mon[30324]: 3.1d scrub starts 2024-09-21T12:26:20.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:20 smithi114 ceph-mon[30324]: 3.1d scrub ok 2024-09-21T12:26:20.314 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:20.368 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:20.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:20.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:20.470 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:20 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T12:26:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:20 smithi183 ceph-mon[33277]: osdmap e199: 8 total, 8 up, 7 in 2024-09-21T12:26:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:20 smithi183 ceph-mon[33277]: 3.1d scrub starts 2024-09-21T12:26:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:20 smithi183 ceph-mon[33277]: 3.1d scrub ok 2024-09-21T12:26:20.494 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:20.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:20.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:20.596 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:20.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:20.627 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:20.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:20.728 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:20.754 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:20.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:20.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:20.850 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:20.856 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:20.879 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:20.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:20.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:20.884 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:20.884 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:20.981 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:21.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:21.031 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:21.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:21.060 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 87 16 126717 126701 90.9948 69.6875 0.00679246 0.0109849 2024-09-21T12:26:21.132 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:21.168 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:21.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:21.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:21.263 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:21 smithi114 ceph-mon[40867]: pgmap v294: 158 pgs: 158 active+clean; 7.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 62 MiB/s wr, 997 op/s 2024-09-21T12:26:21.263 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:21 smithi114 ceph-mon[40867]: 3.15 scrub starts 2024-09-21T12:26:21.263 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:21 smithi114 ceph-mon[40867]: 3.15 scrub ok 2024-09-21T12:26:21.263 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:21 smithi114 ceph-mon[30324]: pgmap v294: 158 pgs: 158 active+clean; 7.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 62 MiB/s wr, 997 op/s 2024-09-21T12:26:21.263 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:21 smithi114 ceph-mon[30324]: 3.15 scrub starts 2024-09-21T12:26:21.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:21 smithi114 ceph-mon[30324]: 3.15 scrub ok 2024-09-21T12:26:21.270 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:21.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:21.310 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:21.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:21.411 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:21.437 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:21.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:21.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:21.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:21 smithi183 ceph-mon[33277]: pgmap v294: 158 pgs: 158 active+clean; 7.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 853 B/s rd, 62 MiB/s wr, 997 op/s 2024-09-21T12:26:21.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:21 smithi183 ceph-mon[33277]: 3.15 scrub starts 2024-09-21T12:26:21.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:21 smithi183 ceph-mon[33277]: 3.15 scrub ok 2024-09-21T12:26:21.538 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:21.576 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:21.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:21.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:21.677 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:21.703 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:21.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:21.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:21.805 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:21.834 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:21.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:21.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:21.881 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:26:21.881 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:26:21.936 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:21.963 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:21.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:21.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:22.030 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-21T12:26:22.060 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 88 16 127935 127919 90.8257 76.125 0.00709436 0.0110053 2024-09-21T12:26:22.065 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:22.086 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:22 smithi114 ceph-mon[40867]: 4.16 scrub starts 2024-09-21T12:26:22.086 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:22 smithi114 ceph-mon[40867]: 4.16 scrub ok 2024-09-21T12:26:22.086 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:22 smithi114 ceph-mon[40867]: 1.5 scrub starts 2024-09-21T12:26:22.087 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:22 smithi114 ceph-mon[40867]: 1.5 scrub ok 2024-09-21T12:26:22.087 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:22 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/4178358509' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:26:22.087 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:22 smithi114 ceph-mon[30324]: 4.16 scrub starts 2024-09-21T12:26:22.087 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:22 smithi114 ceph-mon[30324]: 4.16 scrub ok 2024-09-21T12:26:22.087 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:22 smithi114 ceph-mon[30324]: 1.5 scrub starts 2024-09-21T12:26:22.087 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:22 smithi114 ceph-mon[30324]: 1.5 scrub ok 2024-09-21T12:26:22.087 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:22 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4178358509' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:26:22.090 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:22.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:22.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:22.192 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:22.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:22.229 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:22.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:22.330 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:22.358 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:22.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:22.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:22.460 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:22.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:22 smithi183 ceph-mon[33277]: 4.16 scrub starts 2024-09-21T12:26:22.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:22 smithi183 ceph-mon[33277]: 4.16 scrub ok 2024-09-21T12:26:22.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:22 smithi183 ceph-mon[33277]: 1.5 scrub starts 2024-09-21T12:26:22.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:22 smithi183 ceph-mon[33277]: 1.5 scrub ok 2024-09-21T12:26:22.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:22 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/4178358509' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:26:22.498 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:22.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:22.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:22.600 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:22.622 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:26:22.622 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:26:22 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:26:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:26:22.628 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:26:22.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:22.630 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:22.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:22.731 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:22.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:22.776 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:22.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:22.877 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:22.915 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:22.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:22.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:23.016 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:23.055 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:23.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:23.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:23.060 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 89 16 129120 129104 90.6374 74.0625 0.0116 0.0110283 2024-09-21T12:26:23.157 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:23.181 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:23.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:23.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:23.199 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:23 smithi114 ceph-mon[30324]: 5.13 scrub starts 2024-09-21T12:26:23.199 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:23 smithi114 ceph-mon[30324]: 5.13 scrub ok 2024-09-21T12:26:23.199 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:23 smithi114 ceph-mon[30324]: pgmap v295: 158 pgs: 158 active+clean; 7.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.3 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-21T12:26:23.199 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:23 smithi114 ceph-mon[30324]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:26:23.199 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:23 smithi114 ceph-mon[30324]: 5.11 scrub starts 2024-09-21T12:26:23.199 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:23 smithi114 ceph-mon[30324]: 5.11 scrub ok 2024-09-21T12:26:23.199 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:23 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4178358509' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:26:23.199 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:23 smithi114 ceph-mon[30324]: osdmap e200: 8 total, 8 up, 7 in 2024-09-21T12:26:23.199 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:23 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1391615218' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:26:23.200 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:23 smithi114 ceph-mon[40867]: 5.13 scrub starts 2024-09-21T12:26:23.200 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:23 smithi114 ceph-mon[40867]: 5.13 scrub ok 2024-09-21T12:26:23.200 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:23 smithi114 ceph-mon[40867]: pgmap v295: 158 pgs: 158 active+clean; 7.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.3 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-21T12:26:23.200 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:23 smithi114 ceph-mon[40867]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:26:23.200 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:23 smithi114 ceph-mon[40867]: 5.11 scrub starts 2024-09-21T12:26:23.200 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:23 smithi114 ceph-mon[40867]: 5.11 scrub ok 2024-09-21T12:26:23.200 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:23 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/4178358509' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:26:23.200 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:23 smithi114 ceph-mon[40867]: osdmap e200: 8 total, 8 up, 7 in 2024-09-21T12:26:23.200 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:23 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1391615218' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:26:23.284 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:23.311 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:23.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:23.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:23.390 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:26:23.390 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:26:23.391 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:26:23.391 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:26:23.391 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:26:23.391 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:26:23.391 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:26:23.391 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:26:23.391 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:26:23.391 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:26:23.391 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:26:23.391 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:26:23.391 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:26:23.392 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:26:23.392 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:26:23.392 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:26:23.413 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:23.440 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:23.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:23.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:23.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:23 smithi183 ceph-mon[33277]: 5.13 scrub starts 2024-09-21T12:26:23.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:23 smithi183 ceph-mon[33277]: 5.13 scrub ok 2024-09-21T12:26:23.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:23 smithi183 ceph-mon[33277]: pgmap v295: 158 pgs: 158 active+clean; 7.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.3 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-21T12:26:23.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:23 smithi183 ceph-mon[33277]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:26:23.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:23 smithi183 ceph-mon[33277]: 5.11 scrub starts 2024-09-21T12:26:23.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:23 smithi183 ceph-mon[33277]: 5.11 scrub ok 2024-09-21T12:26:23.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:23 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/4178358509' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:26:23.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:23 smithi183 ceph-mon[33277]: osdmap e200: 8 total, 8 up, 7 in 2024-09-21T12:26:23.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:23 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1391615218' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:26:23.542 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:23.572 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:23.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:23.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:23.674 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:23.711 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:23.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:23.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:23.813 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:23.849 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:23.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:23.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:23.951 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:23.982 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:23.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:23.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:24.061 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Total time run: 90.0058 2024-09-21T12:26:24.061 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Total writes made: 130413 2024-09-21T12:26:24.061 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Write size: 65536 2024-09-21T12:26:24.061 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Object size: 65536 2024-09-21T12:26:24.061 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Bandwidth (MB/sec): 90.5588 2024-09-21T12:26:24.061 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev Bandwidth: 29.1087 2024-09-21T12:26:24.061 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max bandwidth (MB/sec): 207.438 2024-09-21T12:26:24.061 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min bandwidth (MB/sec): 62.3125 2024-09-21T12:26:24.061 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Average IOPS: 1448 2024-09-21T12:26:24.061 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev IOPS: 465.74 2024-09-21T12:26:24.061 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max IOPS: 3319 2024-09-21T12:26:24.062 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min IOPS: 997 2024-09-21T12:26:24.062 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Average Latency(s): 0.0110387 2024-09-21T12:26:24.062 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev Latency(s): 0.00846448 2024-09-21T12:26:24.062 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max latency(s): 0.158038 2024-09-21T12:26:24.062 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min latency(s): 0.00121483 2024-09-21T12:26:24.066 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T12:26:24.085 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:24.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:24.122 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:24.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:24.223 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:24.256 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:24.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:24.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:24.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:24 smithi114 ceph-mon[40867]: from='client.25178 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:26:24.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:24 smithi114 ceph-mon[30324]: from='client.25178 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:26:24.358 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:24.384 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:24.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:24.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:24.486 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:24.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:24 smithi183 ceph-mon[33277]: from='client.25178 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:26:24.515 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:24.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:24.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:24.617 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:24.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:24.659 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:24.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:24.760 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:24.785 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:24.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:24.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:24.853 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T12:26:24.887 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:24.936 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:24.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:24.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:25.038 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:25.065 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:25.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:25.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:25.166 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:25.192 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:25.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:25.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:25.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:25 smithi114 ceph-mon[40867]: pgmap v297: 158 pgs: 158 active+clean; 7.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-21T12:26:25.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:25 smithi114 ceph-mon[30324]: pgmap v297: 158 pgs: 158 active+clean; 7.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-21T12:26:25.294 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:25.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:25.330 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:25.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:25.431 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:25.469 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:25.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:25.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:25.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:25 smithi183 ceph-mon[33277]: pgmap v297: 158 pgs: 158 active+clean; 7.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-21T12:26:25.571 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:25.595 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:25.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:25.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:25.697 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:25.725 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:25.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:25.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:25.826 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:25.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:25.854 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:25.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:25.955 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:25.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:25.989 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:25.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:26.074 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-09-21T12:26:22.023558+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8205572635,"num_objects":125397,"num_object_clones":0,"num_object_copies":376191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":125397,"num_whiteouts":0,"num_read":1075,"num_read_kb":827,"num_write":125634,"num_write_kb":8012717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":223,"num_bytes_recovered":4534,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71206,"ondisk_log_size":71206,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":25794736,"kb_used_data":24067024,"kb_used_omap":10,"kb_used_meta":1727221,"kb_avail":630421328,"statfs":{"total":671965249536,"available":645551439872,"internally_reserved":0,"allocated":24644632576,"data_stored":24641485481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":1768674411},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":610336768,"num_objects":9313,"num_object_clones":0,"num_object_copies":27939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9313,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9313,"num_write_kb":596032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.097318"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":308,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.937340+0000","last_change":"2024-09-21T12:24:50.115300+0000","last_active":"2024-09-21T12:26:14.937340+0000","last_peered":"2024-09-21T12:26:14.937340+0000","last_clean":"2024-09-21T12:26:14.937340+0000","last_became_active":"2024-09-21T12:20:15.309431+0000","last_became_peered":"2024-09-21T12:20:15.309431+0000","last_unstale":"2024-09-21T12:26:14.937340+0000","last_undegraded":"2024-09-21T12:26:14.937340+0000","last_fullsized":"2024-09-21T12:26:14.937340+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:50.115253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:50.115253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:04.907671+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":280,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.986977+0000","last_change":"2024-09-21T12:26:09.823435+0000","last_active":"2024-09-21T12:26:18.986977+0000","last_peered":"2024-09-21T12:26:18.986977+0000","last_clean":"2024-09-21T12:26:18.986977+0000","last_became_active":"2024-09-21T12:22:50.909699+0000","last_became_peered":"2024-09-21T12:22:50.909699+0000","last_unstale":"2024-09-21T12:26:18.986977+0000","last_undegraded":"2024-09-21T12:26:18.986977+0000","last_fullsized":"2024-09-21T12:26:18.986977+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:26:09.823363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:09.823363+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:17.313787+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":302,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:19.002512+0000","last_change":"2024-09-21T12:24:34.311501+0000","last_active":"2024-09-21T12:26:19.002512+0000","last_peered":"2024-09-21T12:26:19.002512+0000","last_clean":"2024-09-21T12:26:19.002512+0000","last_became_active":"2024-09-21T12:20:19.314908+0000","last_became_peered":"2024-09-21T12:20:19.314908+0000","last_unstale":"2024-09-21T12:26:19.002512+0000","last_undegraded":"2024-09-21T12:26:19.002512+0000","last_fullsized":"2024-09-21T12:26:19.002512+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:34.311451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:34.311451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:00.014883+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":225,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.981423+0000","last_change":"2024-09-21T12:22:50.914064+0000","last_active":"2024-09-21T12:26:18.981423+0000","last_peered":"2024-09-21T12:26:18.981423+0000","last_clean":"2024-09-21T12:26:18.981423+0000","last_became_active":"2024-09-21T12:22:50.913743+0000","last_became_peered":"2024-09-21T12:22:50.913743+0000","last_unstale":"2024-09-21T12:26:18.981423+0000","last_undegraded":"2024-09-21T12:26:18.981423+0000","last_fullsized":"2024-09-21T12:26:18.981423+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:58.189665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:21:58.189665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:07.338098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":286,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.945811+0000","last_change":"2024-09-21T12:25:48.316122+0000","last_active":"2024-09-21T12:26:14.945811+0000","last_peered":"2024-09-21T12:26:14.945811+0000","last_clean":"2024-09-21T12:26:14.945811+0000","last_became_active":"2024-09-21T12:20:21.595301+0000","last_became_peered":"2024-09-21T12:20:21.595301+0000","last_unstale":"2024-09-21T12:26:14.945811+0000","last_undegraded":"2024-09-21T12:26:14.945811+0000","last_fullsized":"2024-09-21T12:26:14.945811+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:48.316039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:48.316039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:08.665134+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":306,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.944102+0000","last_change":"2024-09-21T12:24:49.701972+0000","last_active":"2024-09-21T12:26:14.944102+0000","last_peered":"2024-09-21T12:26:14.944102+0000","last_clean":"2024-09-21T12:26:14.944102+0000","last_became_active":"2024-09-21T12:20:15.308655+0000","last_became_peered":"2024-09-21T12:20:15.308655+0000","last_unstale":"2024-09-21T12:26:14.944102+0000","last_undegraded":"2024-09-21T12:26:14.944102+0000","last_fullsized":"2024-09-21T12:26:14.944102+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:49.701915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:49.701915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":327,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.943649+0000","last_change":"2024-09-21T12:25:01.898461+0000","last_active":"2024-09-21T12:26:14.943649+0000","last_peered":"2024-09-21T12:26:14.943649+0000","last_clean":"2024-09-21T12:26:14.943649+0000","last_became_active":"2024-09-21T12:20:17.305220+0000","last_became_peered":"2024-09-21T12:20:17.305220+0000","last_unstale":"2024-09-21T12:26:14.943649+0000","last_undegraded":"2024-09-21T12:26:14.943649+0000","last_fullsized":"2024-09-21T12:26:14.943649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:25:01.898376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:01.898376+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":242,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.987062+0000","last_change":"2024-09-21T12:25:04.827596+0000","last_active":"2024-09-21T12:26:18.987062+0000","last_peered":"2024-09-21T12:26:18.987062+0000","last_clean":"2024-09-21T12:26:18.987062+0000","last_became_active":"2024-09-21T12:22:50.926240+0000","last_became_peered":"2024-09-21T12:22:50.926240+0000","last_unstale":"2024-09-21T12:26:18.987062+0000","last_undegraded":"2024-09-21T12:26:18.987062+0000","last_fullsized":"2024-09-21T12:26:18.987062+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:04.827529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:04.827529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":330,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:19.446300+0000","last_change":"2024-09-21T12:26:19.446300+0000","last_active":"2024-09-21T12:26:19.446300+0000","last_peered":"2024-09-21T12:26:19.446300+0000","last_clean":"2024-09-21T12:26:19.446300+0000","last_became_active":"2024-09-21T12:22:50.912481+0000","last_became_peered":"2024-09-21T12:22:50.912481+0000","last_unstale":"2024-09-21T12:26:19.446300+0000","last_undegraded":"2024-09-21T12:26:19.446300+0000","last_fullsized":"2024-09-21T12:26:19.446300+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:19.446251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:19.446251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:36.052328+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":287,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.945433+0000","last_change":"2024-09-21T12:24:25.640253+0000","last_active":"2024-09-21T12:26:14.945433+0000","last_peered":"2024-09-21T12:26:14.945433+0000","last_clean":"2024-09-21T12:26:14.945433+0000","last_became_active":"2024-09-21T12:22:50.927804+0000","last_became_peered":"2024-09-21T12:22:50.927804+0000","last_unstale":"2024-09-21T12:26:14.945433+0000","last_undegraded":"2024-09-21T12:26:14.945433+0000","last_fullsized":"2024-09-21T12:26:14.945433+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:24:25.640169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:25.640169+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:58.377864+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":301,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.982061+0000","last_change":"2024-09-21T12:24:58.566964+0000","last_active":"2024-09-21T12:26:18.982061+0000","last_peered":"2024-09-21T12:26:18.982061+0000","last_clean":"2024-09-21T12:26:18.982061+0000","last_became_active":"2024-09-21T12:20:19.308184+0000","last_became_peered":"2024-09-21T12:20:19.308184+0000","last_unstale":"2024-09-21T12:26:18.982061+0000","last_undegraded":"2024-09-21T12:26:18.982061+0000","last_fullsized":"2024-09-21T12:26:18.982061+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:58.566902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:58.566902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:27.540756+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":296,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.933167+0000","last_change":"2024-09-21T12:25:31.616293+0000","last_active":"2024-09-21T12:26:14.933167+0000","last_peered":"2024-09-21T12:26:14.933167+0000","last_clean":"2024-09-21T12:26:14.933167+0000","last_became_active":"2024-09-21T12:20:21.309190+0000","last_became_peered":"2024-09-21T12:20:21.309190+0000","last_unstale":"2024-09-21T12:26:14.933167+0000","last_undegraded":"2024-09-21T12:26:14.933167+0000","last_fullsized":"2024-09-21T12:26:14.933167+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:31.616219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:31.616219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:48.224028+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":322,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.981980+0000","last_change":"2024-09-21T12:22:54.174099+0000","last_active":"2024-09-21T12:26:18.981980+0000","last_peered":"2024-09-21T12:26:18.981980+0000","last_clean":"2024-09-21T12:26:18.981980+0000","last_became_active":"2024-09-21T12:22:50.911737+0000","last_became_peered":"2024-09-21T12:22:50.911737+0000","last_unstale":"2024-09-21T12:26:18.981980+0000","last_undegraded":"2024-09-21T12:26:18.981980+0000","last_fullsized":"2024-09-21T12:26:18.981980+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:54.174034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:54.174034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:23.349015+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":273,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.946058+0000","last_change":"2024-09-21T12:24:28.624430+0000","last_active":"2024-09-21T12:26:14.946058+0000","last_peered":"2024-09-21T12:26:14.946058+0000","last_clean":"2024-09-21T12:26:14.946058+0000","last_became_active":"2024-09-21T12:22:50.918794+0000","last_became_peered":"2024-09-21T12:22:50.918794+0000","last_unstale":"2024-09-21T12:26:14.946058+0000","last_undegraded":"2024-09-21T12:26:14.946058+0000","last_fullsized":"2024-09-21T12:26:14.946058+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-21T12:24:28.624364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:28.624364+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":314,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.980208+0000","last_change":"2024-09-21T12:25:01.897596+0000","last_active":"2024-09-21T12:26:18.980208+0000","last_peered":"2024-09-21T12:26:18.980208+0000","last_clean":"2024-09-21T12:26:18.980208+0000","last_became_active":"2024-09-21T12:22:50.921780+0000","last_became_peered":"2024-09-21T12:22:50.921780+0000","last_unstale":"2024-09-21T12:26:18.980208+0000","last_undegraded":"2024-09-21T12:26:18.980208+0000","last_fullsized":"2024-09-21T12:26:18.980208+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:01.897527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:01.897527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":305,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.981417+0000","last_change":"2024-09-21T12:24:34.367580+0000","last_active":"2024-09-21T12:26:18.981417+0000","last_peered":"2024-09-21T12:26:18.981417+0000","last_clean":"2024-09-21T12:26:18.981417+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:26:18.981417+0000","last_undegraded":"2024-09-21T12:26:18.981417+0000","last_fullsized":"2024-09-21T12:26:18.981417+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:34.367532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:34.367532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":242,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.989968+0000","last_change":"2024-09-21T12:25:30.888535+0000","last_active":"2024-09-21T12:26:18.989968+0000","last_peered":"2024-09-21T12:26:18.989968+0000","last_clean":"2024-09-21T12:26:18.989968+0000","last_became_active":"2024-09-21T12:22:50.914792+0000","last_became_peered":"2024-09-21T12:22:50.914792+0000","last_unstale":"2024-09-21T12:26:18.989968+0000","last_undegraded":"2024-09-21T12:26:18.989968+0000","last_fullsized":"2024-09-21T12:26:18.989968+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:30.888491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:30.888491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:32.893945+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":337,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.933685+0000","last_change":"2024-09-21T12:24:30.033946+0000","last_active":"2024-09-21T12:26:14.933685+0000","last_peered":"2024-09-21T12:26:14.933685+0000","last_clean":"2024-09-21T12:26:14.933685+0000","last_became_active":"2024-09-21T12:22:50.921576+0000","last_became_peered":"2024-09-21T12:22:50.921576+0000","last_unstale":"2024-09-21T12:26:14.933685+0000","last_undegraded":"2024-09-21T12:26:14.933685+0000","last_fullsized":"2024-09-21T12:26:14.933685+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:30.033874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:30.033874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:00.878523+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":331,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.980363+0000","last_change":"2024-09-21T12:24:50.834579+0000","last_active":"2024-09-21T12:26:18.980363+0000","last_peered":"2024-09-21T12:26:18.980363+0000","last_clean":"2024-09-21T12:26:18.980363+0000","last_became_active":"2024-09-21T12:20:17.307828+0000","last_became_peered":"2024-09-21T12:20:17.307828+0000","last_unstale":"2024-09-21T12:26:18.980363+0000","last_undegraded":"2024-09-21T12:26:18.980363+0000","last_fullsized":"2024-09-21T12:26:18.980363+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:24:50.834503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:50.834503+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:20.530206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":330,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:19.000816+0000","last_change":"2024-09-21T12:25:46.906275+0000","last_active":"2024-09-21T12:26:19.000816+0000","last_peered":"2024-09-21T12:26:19.000816+0000","last_clean":"2024-09-21T12:26:19.000816+0000","last_became_active":"2024-09-21T12:25:46.905878+0000","last_became_peered":"2024-09-21T12:25:46.905878+0000","last_unstale":"2024-09-21T12:26:19.000816+0000","last_undegraded":"2024-09-21T12:26:19.000816+0000","last_fullsized":"2024-09-21T12:26:19.000816+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:48.435769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:48.435769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:57.929606+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":286,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.950560+0000","last_change":"2024-09-21T12:23:58.354628+0000","last_active":"2024-09-21T12:26:14.950560+0000","last_peered":"2024-09-21T12:26:14.950560+0000","last_clean":"2024-09-21T12:26:14.950560+0000","last_became_active":"2024-09-21T12:20:21.592796+0000","last_became_peered":"2024-09-21T12:20:21.592796+0000","last_unstale":"2024-09-21T12:26:14.950560+0000","last_undegraded":"2024-09-21T12:26:14.950560+0000","last_fullsized":"2024-09-21T12:26:14.950560+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:58.354570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:58.354570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:21.152937+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":241,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:19.001617+0000","last_change":"2024-09-21T12:24:55.447829+0000","last_active":"2024-09-21T12:26:19.001617+0000","last_peered":"2024-09-21T12:26:19.001617+0000","last_clean":"2024-09-21T12:26:19.001617+0000","last_became_active":"2024-09-21T12:22:50.925403+0000","last_became_peered":"2024-09-21T12:22:50.925403+0000","last_unstale":"2024-09-21T12:26:19.001617+0000","last_undegraded":"2024-09-21T12:26:19.001617+0000","last_fullsized":"2024-09-21T12:26:19.001617+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:55.447755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:55.447755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:20.748653+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":382,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.943594+0000","last_change":"2024-09-21T12:25:31.270272+0000","last_active":"2024-09-21T12:26:14.943594+0000","last_peered":"2024-09-21T12:26:14.943594+0000","last_clean":"2024-09-21T12:26:14.943594+0000","last_became_active":"2024-09-21T12:22:50.912144+0000","last_became_peered":"2024-09-21T12:22:50.912144+0000","last_unstale":"2024-09-21T12:26:14.943594+0000","last_undegraded":"2024-09-21T12:26:14.943594+0000","last_fullsized":"2024-09-21T12:26:14.943594+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:25:31.270165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:31.270165+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:48.492514+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":216,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.944220+0000","last_change":"2024-09-21T12:22:50.920178+0000","last_active":"2024-09-21T12:26:14.944220+0000","last_peered":"2024-09-21T12:26:14.944220+0000","last_clean":"2024-09-21T12:26:14.944220+0000","last_became_active":"2024-09-21T12:22:50.918326+0000","last_became_peered":"2024-09-21T12:22:50.918326+0000","last_unstale":"2024-09-21T12:26:14.944220+0000","last_undegraded":"2024-09-21T12:26:14.944220+0000","last_fullsized":"2024-09-21T12:26:14.944220+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:50.318827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":290,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.980849+0000","last_change":"2024-09-21T12:23:51.065780+0000","last_active":"2024-09-21T12:26:18.980849+0000","last_peered":"2024-09-21T12:26:18.980849+0000","last_clean":"2024-09-21T12:26:18.980849+0000","last_became_active":"2024-09-21T12:20:21.312976+0000","last_became_peered":"2024-09-21T12:20:21.312976+0000","last_unstale":"2024-09-21T12:26:18.980849+0000","last_undegraded":"2024-09-21T12:26:18.980849+0000","last_fullsized":"2024-09-21T12:26:18.980849+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:23:51.065733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:51.065733+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:06.800726+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":344,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:19.009283+0000","last_change":"2024-09-21T12:26:18.112687+0000","last_active":"2024-09-21T12:26:19.009283+0000","last_peered":"2024-09-21T12:26:19.009283+0000","last_clean":"2024-09-21T12:26:19.009283+0000","last_became_active":"2024-09-21T12:22:50.910933+0000","last_became_peered":"2024-09-21T12:22:50.910933+0000","last_unstale":"2024-09-21T12:26:19.009283+0000","last_undegraded":"2024-09-21T12:26:19.009283+0000","last_fullsized":"2024-09-21T12:26:19.009283+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:26:18.112614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:18.112614+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:47.795526+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":311,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.936873+0000","last_change":"2024-09-21T12:24:03.059782+0000","last_active":"2024-09-21T12:26:14.936873+0000","last_peered":"2024-09-21T12:26:14.936873+0000","last_clean":"2024-09-21T12:26:14.936873+0000","last_became_active":"2024-09-21T12:20:17.695977+0000","last_became_peered":"2024-09-21T12:20:17.695977+0000","last_unstale":"2024-09-21T12:26:14.936873+0000","last_undegraded":"2024-09-21T12:26:14.936873+0000","last_fullsized":"2024-09-21T12:26:14.936873+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:24:03.059549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:03.059549+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:25.640560+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"121'11","reported_seq":427,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.988561+0000","last_change":"2024-09-21T12:26:10.846256+0000","last_active":"2024-09-21T12:26:18.988561+0000","last_peered":"2024-09-21T12:26:18.988561+0000","last_clean":"2024-09-21T12:26:18.988561+0000","last_became_active":"2024-09-21T12:22:50.916683+0000","last_became_peered":"2024-09-21T12:22:50.916683+0000","last_unstale":"2024-09-21T12:26:18.988561+0000","last_undegraded":"2024-09-21T12:26:18.988561+0000","last_fullsized":"2024-09-21T12:26:18.988561+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:26:10.846183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:10.846183+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:32.395882+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"199'7814","reported_seq":7903,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:21.812738+0000","last_change":"2024-09-21T12:24:46.316817+0000","last_active":"2024-09-21T12:26:21.812738+0000","last_peered":"2024-09-21T12:26:21.812738+0000","last_clean":"2024-09-21T12:26:21.812738+0000","last_became_active":"2024-09-21T12:24:46.316132+0000","last_became_peered":"2024-09-21T12:24:46.316132+0000","last_unstale":"2024-09-21T12:26:21.812738+0000","last_undegraded":"2024-09-21T12:26:21.812738+0000","last_fullsized":"2024-09-21T12:26:21.812738+0000","mapping_epoch":160,"log_start":"176'3500","ondisk_log_start":"176'3500","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4314,"log_dups_size":0,"ondisk_log_size":4314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:05.983258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":512098304,"num_objects":7814,"num_object_clones":0,"num_object_copies":23442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7814,"num_write_kb":500096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"76'79","reported_seq":210,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.981231+0000","last_change":"2024-09-21T12:25:42.586460+0000","last_active":"2024-09-21T12:26:18.981231+0000","last_peered":"2024-09-21T12:26:18.981231+0000","last_clean":"2024-09-21T12:26:18.981231+0000","last_became_active":"2024-09-21T12:22:17.074721+0000","last_became_peered":"2024-09-21T12:22:17.074721+0000","last_unstale":"2024-09-21T12:26:18.981231+0000","last_undegraded":"2024-09-21T12:26:18.981231+0000","last_fullsized":"2024-09-21T12:26:18.981231+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:25:42.586392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:25:42.586392+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:57.694710+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":161,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.979264+0000","last_change":"2024-09-21T12:24:59.803037+0000","last_active":"2024-09-21T12:26:18.979264+0000","last_peered":"2024-09-21T12:26:18.979264+0000","last_clean":"2024-09-21T12:26:18.979264+0000","last_became_active":"2024-09-21T12:23:46.910112+0000","last_became_peered":"2024-09-21T12:23:46.910112+0000","last_unstale":"2024-09-21T12:26:18.979264+0000","last_undegraded":"2024-09-21T12:26:18.979264+0000","last_fullsized":"2024-09-21T12:26:18.979264+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:24:59.802945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:59.802945+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:22.089988+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":298,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.949426+0000","last_change":"2024-09-21T12:22:14.036290+0000","last_active":"2024-09-21T12:26:14.949426+0000","last_peered":"2024-09-21T12:26:14.949426+0000","last_clean":"2024-09-21T12:26:14.949426+0000","last_became_active":"2024-09-21T12:20:15.295774+0000","last_became_peered":"2024-09-21T12:20:15.295774+0000","last_unstale":"2024-09-21T12:26:14.949426+0000","last_undegraded":"2024-09-21T12:26:14.949426+0000","last_fullsized":"2024-09-21T12:26:14.949426+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:14.036236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.036236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:55.467472+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"158'13","reported_seq":130,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.982049+0000","last_change":"2024-09-21T12:25:33.474458+0000","last_active":"2024-09-21T12:26:18.982049+0000","last_peered":"2024-09-21T12:26:18.982049+0000","last_clean":"2024-09-21T12:26:18.982049+0000","last_became_active":"2024-09-21T12:24:39.272835+0000","last_became_peered":"2024-09-21T12:24:39.272835+0000","last_unstale":"2024-09-21T12:26:18.982049+0000","last_undegraded":"2024-09-21T12:26:18.982049+0000","last_fullsized":"2024-09-21T12:26:18.982049+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'13","last_scrub_stamp":"2024-09-21T12:25:33.474391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:33.474391+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:01.060227+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":290,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.981534+0000","last_change":"2024-09-21T12:22:12.145637+0000","last_active":"2024-09-21T12:26:18.981534+0000","last_peered":"2024-09-21T12:26:18.981534+0000","last_clean":"2024-09-21T12:26:18.981534+0000","last_became_active":"2024-09-21T12:20:21.310022+0000","last_became_peered":"2024-09-21T12:20:21.310022+0000","last_unstale":"2024-09-21T12:26:18.981534+0000","last_undegraded":"2024-09-21T12:26:18.981534+0000","last_fullsized":"2024-09-21T12:26:18.981534+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:22:12.145550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:12.145550+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:51.460208+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"198'7677","reported_seq":7762,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:17.198212+0000","last_change":"2024-09-21T12:24:46.326848+0000","last_active":"2024-09-21T12:26:17.198212+0000","last_peered":"2024-09-21T12:26:17.198212+0000","last_clean":"2024-09-21T12:26:17.198212+0000","last_became_active":"2024-09-21T12:24:46.326586+0000","last_became_peered":"2024-09-21T12:24:46.326586+0000","last_unstale":"2024-09-21T12:26:17.198212+0000","last_undegraded":"2024-09-21T12:26:17.198212+0000","last_fullsized":"2024-09-21T12:26:17.198212+0000","mapping_epoch":160,"log_start":"175'3300","ondisk_log_start":"175'3300","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4377,"log_dups_size":0,"ondisk_log_size":4377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503119872,"num_objects":7677,"num_object_clones":0,"num_object_copies":23031,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7677,"num_write_kb":491328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"76'79","reported_seq":199,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:21.091972+0000","last_change":"2024-09-21T12:26:21.091972+0000","last_active":"2024-09-21T12:26:21.091972+0000","last_peered":"2024-09-21T12:26:21.091972+0000","last_clean":"2024-09-21T12:26:21.091972+0000","last_became_active":"2024-09-21T12:22:50.914885+0000","last_became_peered":"2024-09-21T12:22:50.914885+0000","last_unstale":"2024-09-21T12:26:21.091972+0000","last_undegraded":"2024-09-21T12:26:21.091972+0000","last_fullsized":"2024-09-21T12:26:21.091972+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:21.091797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:21.091797+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:25.919035+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":334,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.949205+0000","last_change":"2024-09-21T12:25:35.325593+0000","last_active":"2024-09-21T12:26:14.949205+0000","last_peered":"2024-09-21T12:26:14.949205+0000","last_clean":"2024-09-21T12:26:14.949205+0000","last_became_active":"2024-09-21T12:20:17.693179+0000","last_became_peered":"2024-09-21T12:20:17.693179+0000","last_unstale":"2024-09-21T12:26:14.949205+0000","last_undegraded":"2024-09-21T12:26:14.949205+0000","last_fullsized":"2024-09-21T12:26:14.949205+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-21T12:25:35.325493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:35.325493+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:38.650408+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":298,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.951353+0000","last_change":"2024-09-21T12:21:18.056305+0000","last_active":"2024-09-21T12:26:14.951353+0000","last_peered":"2024-09-21T12:26:14.951353+0000","last_clean":"2024-09-21T12:26:14.951353+0000","last_became_active":"2024-09-21T12:20:15.297340+0000","last_became_peered":"2024-09-21T12:20:15.297340+0000","last_unstale":"2024-09-21T12:26:14.951353+0000","last_undegraded":"2024-09-21T12:26:14.951353+0000","last_fullsized":"2024-09-21T12:26:14.951353+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:18.056261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:18.056261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:07.391958+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":241,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.943290+0000","last_change":"2024-09-21T12:25:38.374222+0000","last_active":"2024-09-21T12:26:14.943290+0000","last_peered":"2024-09-21T12:26:14.943290+0000","last_clean":"2024-09-21T12:26:14.943290+0000","last_became_active":"2024-09-21T12:22:50.930858+0000","last_became_peered":"2024-09-21T12:22:50.930858+0000","last_unstale":"2024-09-21T12:26:14.943290+0000","last_undegraded":"2024-09-21T12:26:14.943290+0000","last_fullsized":"2024-09-21T12:26:14.943290+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:38.374146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:38.374146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:58.524870+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":232,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.945344+0000","last_change":"2024-09-21T12:24:29.635826+0000","last_active":"2024-09-21T12:26:14.945344+0000","last_peered":"2024-09-21T12:26:14.945344+0000","last_clean":"2024-09-21T12:26:14.945344+0000","last_became_active":"2024-09-21T12:22:50.918940+0000","last_became_peered":"2024-09-21T12:22:50.918940+0000","last_unstale":"2024-09-21T12:26:14.945344+0000","last_undegraded":"2024-09-21T12:26:14.945344+0000","last_fullsized":"2024-09-21T12:26:14.945344+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:29.635514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:29.635514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"199'7957","reported_seq":8046,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:21.597861+0000","last_change":"2024-09-21T12:24:46.324341+0000","last_active":"2024-09-21T12:26:21.597861+0000","last_peered":"2024-09-21T12:26:21.597861+0000","last_clean":"2024-09-21T12:26:21.597861+0000","last_became_active":"2024-09-21T12:24:46.323986+0000","last_became_peered":"2024-09-21T12:24:46.323986+0000","last_unstale":"2024-09-21T12:26:21.597861+0000","last_undegraded":"2024-09-21T12:26:21.597861+0000","last_fullsized":"2024-09-21T12:26:21.597861+0000","mapping_epoch":160,"log_start":"176'3500","ondisk_log_start":"176'3500","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4457,"log_dups_size":0,"ondisk_log_size":4457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":521469952,"num_objects":7957,"num_object_clones":0,"num_object_copies":23871,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7957,"num_write_kb":509248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"76'79","reported_seq":213,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.944045+0000","last_change":"2024-09-21T12:25:02.911776+0000","last_active":"2024-09-21T12:26:14.944045+0000","last_peered":"2024-09-21T12:26:14.944045+0000","last_clean":"2024-09-21T12:26:14.944045+0000","last_became_active":"2024-09-21T12:22:17.069042+0000","last_became_peered":"2024-09-21T12:22:17.069042+0000","last_unstale":"2024-09-21T12:26:14.944045+0000","last_undegraded":"2024-09-21T12:26:14.944045+0000","last_fullsized":"2024-09-21T12:26:14.944045+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:25:02.911705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:25:02.911705+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":299,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.981518+0000","last_change":"2024-09-21T12:22:50.978102+0000","last_active":"2024-09-21T12:26:18.981518+0000","last_peered":"2024-09-21T12:26:18.981518+0000","last_clean":"2024-09-21T12:26:18.981518+0000","last_became_active":"2024-09-21T12:22:50.926008+0000","last_became_peered":"2024-09-21T12:22:50.926008+0000","last_unstale":"2024-09-21T12:26:18.981518+0000","last_undegraded":"2024-09-21T12:26:18.981518+0000","last_fullsized":"2024-09-21T12:26:18.981518+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-21T12:22:14.062464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.062464+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:49.540547+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":334,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.937164+0000","last_change":"2024-09-21T12:22:51.003768+0000","last_active":"2024-09-21T12:26:14.937164+0000","last_peered":"2024-09-21T12:26:14.937164+0000","last_clean":"2024-09-21T12:26:14.937164+0000","last_became_active":"2024-09-21T12:22:50.923284+0000","last_became_peered":"2024-09-21T12:22:50.923284+0000","last_unstale":"2024-09-21T12:26:14.937164+0000","last_undegraded":"2024-09-21T12:26:14.937164+0000","last_fullsized":"2024-09-21T12:26:14.937164+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:22:07.582969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:07.582969+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:53.794213+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"121'11","reported_seq":412,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.949064+0000","last_change":"2024-09-21T12:26:14.490045+0000","last_active":"2024-09-21T12:26:14.949064+0000","last_peered":"2024-09-21T12:26:14.949064+0000","last_clean":"2024-09-21T12:26:14.949064+0000","last_became_active":"2024-09-21T12:22:50.930342+0000","last_became_peered":"2024-09-21T12:22:50.930342+0000","last_unstale":"2024-09-21T12:26:14.949064+0000","last_undegraded":"2024-09-21T12:26:14.949064+0000","last_fullsized":"2024-09-21T12:26:14.949064+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:26:14.489972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:14.489972+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:31.388160+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":289,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.980293+0000","last_change":"2024-09-21T12:21:21.835669+0000","last_active":"2024-09-21T12:26:18.980293+0000","last_peered":"2024-09-21T12:26:18.980293+0000","last_clean":"2024-09-21T12:26:18.980293+0000","last_became_active":"2024-09-21T12:20:21.593367+0000","last_became_peered":"2024-09-21T12:20:21.593367+0000","last_unstale":"2024-09-21T12:26:18.980293+0000","last_undegraded":"2024-09-21T12:26:18.980293+0000","last_fullsized":"2024-09-21T12:26:18.980293+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:21.835623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:21:21.835623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:29.766408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"198'7575","reported_seq":7662,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:17.206619+0000","last_change":"2024-09-21T12:24:46.325776+0000","last_active":"2024-09-21T12:26:17.206619+0000","last_peered":"2024-09-21T12:26:17.206619+0000","last_clean":"2024-09-21T12:26:17.206619+0000","last_became_active":"2024-09-21T12:24:46.325260+0000","last_became_peered":"2024-09-21T12:24:46.325260+0000","last_unstale":"2024-09-21T12:26:17.206619+0000","last_undegraded":"2024-09-21T12:26:17.206619+0000","last_fullsized":"2024-09-21T12:26:17.206619+0000","mapping_epoch":160,"log_start":"174'3200","ondisk_log_start":"174'3200","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4375,"log_dups_size":0,"ondisk_log_size":4375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:01.850898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":496435200,"num_objects":7575,"num_object_clones":0,"num_object_copies":22725,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7575,"num_write_kb":484800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"76'79","reported_seq":300,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.988353+0000","last_change":"2024-09-21T12:24:51.918097+0000","last_active":"2024-09-21T12:26:18.988353+0000","last_peered":"2024-09-21T12:26:18.988353+0000","last_clean":"2024-09-21T12:26:18.988353+0000","last_became_active":"2024-09-21T12:22:17.093298+0000","last_became_peered":"2024-09-21T12:22:17.093298+0000","last_unstale":"2024-09-21T12:26:18.988353+0000","last_undegraded":"2024-09-21T12:26:18.988353+0000","last_fullsized":"2024-09-21T12:26:18.988353+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:24:51.918028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:24:51.918028+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":370,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:19.001404+0000","last_change":"2024-09-21T12:24:01.388987+0000","last_active":"2024-09-21T12:26:19.001404+0000","last_peered":"2024-09-21T12:26:19.001404+0000","last_clean":"2024-09-21T12:26:19.001404+0000","last_became_active":"2024-09-21T12:22:50.925840+0000","last_became_peered":"2024-09-21T12:22:50.925840+0000","last_unstale":"2024-09-21T12:26:19.001404+0000","last_undegraded":"2024-09-21T12:26:19.001404+0000","last_fullsized":"2024-09-21T12:26:19.001404+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:24:01.388905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:01.388905+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:20.477756+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":309,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.982185+0000","last_change":"2024-09-21T12:26:16.410118+0000","last_active":"2024-09-21T12:26:18.982185+0000","last_peered":"2024-09-21T12:26:18.982185+0000","last_clean":"2024-09-21T12:26:18.982185+0000","last_became_active":"2024-09-21T12:20:15.308518+0000","last_became_peered":"2024-09-21T12:20:15.308518+0000","last_unstale":"2024-09-21T12:26:18.982185+0000","last_undegraded":"2024-09-21T12:26:18.982185+0000","last_fullsized":"2024-09-21T12:26:18.982185+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:16.410067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:16.410067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:24.383217+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":384,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.933300+0000","last_change":"2024-09-21T12:25:42.615523+0000","last_active":"2024-09-21T12:26:14.933300+0000","last_peered":"2024-09-21T12:26:14.933300+0000","last_clean":"2024-09-21T12:26:14.933300+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:26:14.933300+0000","last_undegraded":"2024-09-21T12:26:14.933300+0000","last_fullsized":"2024-09-21T12:26:14.933300+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:25:42.615446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:42.615446+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:06.344897+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":288,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.933301+0000","last_change":"2024-09-21T12:22:51.987032+0000","last_active":"2024-09-21T12:26:14.933301+0000","last_peered":"2024-09-21T12:26:14.933301+0000","last_clean":"2024-09-21T12:26:14.933301+0000","last_became_active":"2024-09-21T12:20:21.315801+0000","last_became_peered":"2024-09-21T12:20:21.315801+0000","last_unstale":"2024-09-21T12:26:14.933301+0000","last_undegraded":"2024-09-21T12:26:14.933301+0000","last_fullsized":"2024-09-21T12:26:14.933301+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:51.986953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:51.986953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:21.623214+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"198'7604","reported_seq":7691,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:17.060539+0000","last_change":"2024-09-21T12:24:46.324384+0000","last_active":"2024-09-21T12:26:17.060539+0000","last_peered":"2024-09-21T12:26:17.060539+0000","last_clean":"2024-09-21T12:26:17.060539+0000","last_became_active":"2024-09-21T12:24:46.323283+0000","last_became_peered":"2024-09-21T12:24:46.323283+0000","last_unstale":"2024-09-21T12:26:17.060539+0000","last_undegraded":"2024-09-21T12:26:17.060539+0000","last_fullsized":"2024-09-21T12:26:17.060539+0000","mapping_epoch":160,"log_start":"176'3451","ondisk_log_start":"176'3451","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4153,"log_dups_size":0,"ondisk_log_size":4153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":498335744,"num_objects":7604,"num_object_clones":0,"num_object_copies":22812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7604,"num_write_kb":486656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"76'79","reported_seq":300,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.989968+0000","last_change":"2024-09-21T12:24:53.927007+0000","last_active":"2024-09-21T12:26:18.989968+0000","last_peered":"2024-09-21T12:26:18.989968+0000","last_clean":"2024-09-21T12:26:18.989968+0000","last_became_active":"2024-09-21T12:22:17.083629+0000","last_became_peered":"2024-09-21T12:22:17.083629+0000","last_unstale":"2024-09-21T12:26:18.989968+0000","last_undegraded":"2024-09-21T12:26:18.989968+0000","last_fullsized":"2024-09-21T12:26:18.989968+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:24:53.926958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:24:53.926958+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:21.193665+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":340,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.933438+0000","last_change":"2024-09-21T12:26:14.834328+0000","last_active":"2024-09-21T12:26:14.933438+0000","last_peered":"2024-09-21T12:26:14.933438+0000","last_clean":"2024-09-21T12:26:14.933438+0000","last_became_active":"2024-09-21T12:20:17.693067+0000","last_became_peered":"2024-09-21T12:20:17.693067+0000","last_unstale":"2024-09-21T12:26:14.933438+0000","last_undegraded":"2024-09-21T12:26:14.933438+0000","last_fullsized":"2024-09-21T12:26:14.933438+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-21T12:26:14.834236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:14.834236+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:34.377077+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":302,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:19.005469+0000","last_change":"2024-09-21T12:26:12.131784+0000","last_active":"2024-09-21T12:26:19.005469+0000","last_peered":"2024-09-21T12:26:19.005469+0000","last_clean":"2024-09-21T12:26:19.005469+0000","last_became_active":"2024-09-21T12:20:15.308441+0000","last_became_peered":"2024-09-21T12:20:15.308441+0000","last_unstale":"2024-09-21T12:26:19.005469+0000","last_undegraded":"2024-09-21T12:26:19.005469+0000","last_fullsized":"2024-09-21T12:26:19.005469+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:12.131721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:12.131721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:19.426420+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":301,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.987239+0000","last_change":"2024-09-21T12:26:12.887076+0000","last_active":"2024-09-21T12:26:18.987239+0000","last_peered":"2024-09-21T12:26:18.987239+0000","last_clean":"2024-09-21T12:26:18.987239+0000","last_became_active":"2024-09-21T12:20:19.309598+0000","last_became_peered":"2024-09-21T12:20:19.309598+0000","last_unstale":"2024-09-21T12:26:18.987239+0000","last_undegraded":"2024-09-21T12:26:18.987239+0000","last_fullsized":"2024-09-21T12:26:18.987239+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:12.887007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:12.887007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:22.153550+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":290,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:19.002429+0000","last_change":"2024-09-21T12:25:44.918356+0000","last_active":"2024-09-21T12:26:19.002429+0000","last_peered":"2024-09-21T12:26:19.002429+0000","last_clean":"2024-09-21T12:26:19.002429+0000","last_became_active":"2024-09-21T12:20:21.592828+0000","last_became_peered":"2024-09-21T12:20:21.592828+0000","last_unstale":"2024-09-21T12:26:19.002429+0000","last_undegraded":"2024-09-21T12:26:19.002429+0000","last_fullsized":"2024-09-21T12:26:19.002429+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:44.918276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:44.918276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:07.775801+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"199'8008","reported_seq":8097,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:21.605386+0000","last_change":"2024-09-21T12:24:46.325092+0000","last_active":"2024-09-21T12:26:21.605386+0000","last_peered":"2024-09-21T12:26:21.605386+0000","last_clean":"2024-09-21T12:26:21.605386+0000","last_became_active":"2024-09-21T12:24:46.324740+0000","last_became_peered":"2024-09-21T12:24:46.324740+0000","last_unstale":"2024-09-21T12:26:21.605386+0000","last_undegraded":"2024-09-21T12:26:21.605386+0000","last_fullsized":"2024-09-21T12:26:21.605386+0000","mapping_epoch":160,"log_start":"176'3500","ondisk_log_start":"176'3500","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4508,"log_dups_size":0,"ondisk_log_size":4508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":524812288,"num_objects":8008,"num_object_clones":0,"num_object_copies":24024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8008,"num_write_kb":512512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"76'79","reported_seq":202,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.981081+0000","last_change":"2024-09-21T12:23:21.346117+0000","last_active":"2024-09-21T12:26:18.981081+0000","last_peered":"2024-09-21T12:26:18.981081+0000","last_clean":"2024-09-21T12:26:18.981081+0000","last_became_active":"2024-09-21T12:22:17.084732+0000","last_became_peered":"2024-09-21T12:22:17.084732+0000","last_unstale":"2024-09-21T12:26:18.981081+0000","last_undegraded":"2024-09-21T12:26:18.981081+0000","last_fullsized":"2024-09-21T12:26:18.981081+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:23:21.346046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:23:21.346046+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:04.385057+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":420,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:19.005377+0000","last_change":"2024-09-21T12:26:09.082395+0000","last_active":"2024-09-21T12:26:19.005377+0000","last_peered":"2024-09-21T12:26:19.005377+0000","last_clean":"2024-09-21T12:26:19.005377+0000","last_became_active":"2024-09-21T12:20:17.307769+0000","last_became_peered":"2024-09-21T12:20:17.307769+0000","last_unstale":"2024-09-21T12:26:19.005377+0000","last_undegraded":"2024-09-21T12:26:19.005377+0000","last_fullsized":"2024-09-21T12:26:19.005377+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-21T12:26:09.082314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:09.082314+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:17.369834+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":224,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.933615+0000","last_change":"2024-09-21T12:22:50.922961+0000","last_active":"2024-09-21T12:26:14.933615+0000","last_peered":"2024-09-21T12:26:14.933615+0000","last_clean":"2024-09-21T12:26:14.933615+0000","last_became_active":"2024-09-21T12:22:50.922002+0000","last_became_peered":"2024-09-21T12:22:50.922002+0000","last_unstale":"2024-09-21T12:26:14.933615+0000","last_undegraded":"2024-09-21T12:26:14.933615+0000","last_fullsized":"2024-09-21T12:26:14.933615+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:05.507899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:05.507899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:50.702651+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":311,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.982206+0000","last_change":"2024-09-21T12:21:20.045824+0000","last_active":"2024-09-21T12:26:18.982206+0000","last_peered":"2024-09-21T12:26:18.982206+0000","last_clean":"2024-09-21T12:26:18.982206+0000","last_became_active":"2024-09-21T12:20:14.287998+0000","last_became_peered":"2024-09-21T12:20:14.287998+0000","last_unstale":"2024-09-21T12:26:18.982206+0000","last_undegraded":"2024-09-21T12:26:18.982206+0000","last_fullsized":"2024-09-21T12:26:18.982206+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T12:21:20.045755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.045755+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:05.385454+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011011149999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":321,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.937199+0000","last_change":"2024-09-21T12:24:53.112915+0000","last_active":"2024-09-21T12:26:14.937199+0000","last_peered":"2024-09-21T12:26:14.937199+0000","last_clean":"2024-09-21T12:26:14.937199+0000","last_became_active":"2024-09-21T12:22:50.921488+0000","last_became_peered":"2024-09-21T12:22:50.921488+0000","last_unstale":"2024-09-21T12:26:14.937199+0000","last_undegraded":"2024-09-21T12:26:14.937199+0000","last_fullsized":"2024-09-21T12:26:14.937199+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:53.112868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:53.112868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:05.952435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":315,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.943948+0000","last_change":"2024-09-21T12:26:10.435410+0000","last_active":"2024-09-21T12:26:14.943948+0000","last_peered":"2024-09-21T12:26:14.943948+0000","last_clean":"2024-09-21T12:26:14.943948+0000","last_became_active":"2024-09-21T12:22:50.917558+0000","last_became_peered":"2024-09-21T12:22:50.917558+0000","last_unstale":"2024-09-21T12:26:14.943948+0000","last_undegraded":"2024-09-21T12:26:14.943948+0000","last_fullsized":"2024-09-21T12:26:14.943948+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:10.435349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:10.435349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:19.570759+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"198'7641","reported_seq":7728,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:17.105863+0000","last_change":"2024-09-21T12:24:46.323849+0000","last_active":"2024-09-21T12:26:17.105863+0000","last_peered":"2024-09-21T12:26:17.105863+0000","last_clean":"2024-09-21T12:26:17.105863+0000","last_became_active":"2024-09-21T12:24:46.323306+0000","last_became_peered":"2024-09-21T12:24:46.323306+0000","last_unstale":"2024-09-21T12:26:17.105863+0000","last_undegraded":"2024-09-21T12:26:17.105863+0000","last_fullsized":"2024-09-21T12:26:17.105863+0000","mapping_epoch":160,"log_start":"176'3500","ondisk_log_start":"176'3500","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4141,"log_dups_size":0,"ondisk_log_size":4141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":500760576,"num_objects":7641,"num_object_clones":0,"num_object_copies":22923,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7641,"num_write_kb":489024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'79","reported_seq":385,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.980084+0000","last_change":"2024-09-21T12:25:03.877681+0000","last_active":"2024-09-21T12:26:18.980084+0000","last_peered":"2024-09-21T12:26:18.980084+0000","last_clean":"2024-09-21T12:26:18.980084+0000","last_became_active":"2024-09-21T12:22:50.921709+0000","last_became_peered":"2024-09-21T12:22:50.921709+0000","last_unstale":"2024-09-21T12:26:18.980084+0000","last_undegraded":"2024-09-21T12:26:18.980084+0000","last_fullsized":"2024-09-21T12:26:18.980084+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:25:03.877610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:25:03.877610+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:26.706650+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":333,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:19.011248+0000","last_change":"2024-09-21T12:25:45.962538+0000","last_active":"2024-09-21T12:26:19.011248+0000","last_peered":"2024-09-21T12:26:19.011248+0000","last_clean":"2024-09-21T12:26:19.011248+0000","last_became_active":"2024-09-21T12:20:17.297374+0000","last_became_peered":"2024-09-21T12:20:17.297374+0000","last_unstale":"2024-09-21T12:26:19.011248+0000","last_undegraded":"2024-09-21T12:26:19.011248+0000","last_fullsized":"2024-09-21T12:26:19.011248+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-21T12:25:45.962433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:45.962433+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:08.718722+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":311,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:19.002242+0000","last_change":"2024-09-21T12:23:59.367953+0000","last_active":"2024-09-21T12:26:19.002242+0000","last_peered":"2024-09-21T12:26:19.002242+0000","last_clean":"2024-09-21T12:26:19.002242+0000","last_became_active":"2024-09-21T12:20:15.308356+0000","last_became_peered":"2024-09-21T12:20:15.308356+0000","last_unstale":"2024-09-21T12:26:19.002242+0000","last_undegraded":"2024-09-21T12:26:19.002242+0000","last_fullsized":"2024-09-21T12:26:19.002242+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:59.367778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:23:59.367778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":327,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.979213+0000","last_change":"2024-09-21T12:26:14.109964+0000","last_active":"2024-09-21T12:26:18.979213+0000","last_peered":"2024-09-21T12:26:18.979213+0000","last_clean":"2024-09-21T12:26:18.979213+0000","last_became_active":"2024-09-21T12:20:14.288358+0000","last_became_peered":"2024-09-21T12:20:14.288358+0000","last_unstale":"2024-09-21T12:26:18.979213+0000","last_undegraded":"2024-09-21T12:26:18.979213+0000","last_fullsized":"2024-09-21T12:26:18.979213+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:26:14.109886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:26:14.109886+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:31.748840+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.0019042449999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":302,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.985178+0000","last_change":"2024-09-21T12:25:46.181836+0000","last_active":"2024-09-21T12:26:18.985178+0000","last_peered":"2024-09-21T12:26:18.985178+0000","last_clean":"2024-09-21T12:26:18.985178+0000","last_became_active":"2024-09-21T12:20:19.309724+0000","last_became_peered":"2024-09-21T12:20:19.309724+0000","last_unstale":"2024-09-21T12:26:18.985178+0000","last_undegraded":"2024-09-21T12:26:18.985178+0000","last_fullsized":"2024-09-21T12:26:18.985178+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:46.181531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:46.181531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:11.368016+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":281,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.980490+0000","last_change":"2024-09-21T12:20:21.593267+0000","last_active":"2024-09-21T12:26:18.980490+0000","last_peered":"2024-09-21T12:26:18.980490+0000","last_clean":"2024-09-21T12:26:18.980490+0000","last_became_active":"2024-09-21T12:20:21.592982+0000","last_became_peered":"2024-09-21T12:20:21.592982+0000","last_unstale":"2024-09-21T12:26:18.980490+0000","last_undegraded":"2024-09-21T12:26:18.980490+0000","last_fullsized":"2024-09-21T12:26:18.980490+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:22.831735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"199'7903","reported_seq":7992,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:21.800918+0000","last_change":"2024-09-21T12:24:46.326200+0000","last_active":"2024-09-21T12:26:21.800918+0000","last_peered":"2024-09-21T12:26:21.800918+0000","last_clean":"2024-09-21T12:26:21.800918+0000","last_became_active":"2024-09-21T12:24:46.325838+0000","last_became_peered":"2024-09-21T12:24:46.325838+0000","last_unstale":"2024-09-21T12:26:21.800918+0000","last_undegraded":"2024-09-21T12:26:21.800918+0000","last_fullsized":"2024-09-21T12:26:21.800918+0000","mapping_epoch":160,"log_start":"176'3600","ondisk_log_start":"176'3600","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4303,"log_dups_size":0,"ondisk_log_size":4303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":517931008,"num_objects":7903,"num_object_clones":0,"num_object_copies":23709,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7903,"num_write_kb":505792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"76'79","reported_seq":224,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:19.007656+0000","last_change":"2024-09-21T12:23:57.442430+0000","last_active":"2024-09-21T12:26:19.007656+0000","last_peered":"2024-09-21T12:26:19.007656+0000","last_clean":"2024-09-21T12:26:19.007656+0000","last_became_active":"2024-09-21T12:22:50.912088+0000","last_became_peered":"2024-09-21T12:22:50.912088+0000","last_unstale":"2024-09-21T12:26:19.007656+0000","last_undegraded":"2024-09-21T12:26:19.007656+0000","last_fullsized":"2024-09-21T12:26:19.007656+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:23:57.442284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:23:57.442284+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:25.572064+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":314,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.936969+0000","last_change":"2024-09-21T12:22:49.933621+0000","last_active":"2024-09-21T12:26:14.936969+0000","last_peered":"2024-09-21T12:26:14.936969+0000","last_clean":"2024-09-21T12:26:14.936969+0000","last_became_active":"2024-09-21T12:20:17.299325+0000","last_became_peered":"2024-09-21T12:20:17.299325+0000","last_unstale":"2024-09-21T12:26:14.936969+0000","last_undegraded":"2024-09-21T12:26:14.936969+0000","last_fullsized":"2024-09-21T12:26:14.936969+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:22:49.933382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:49.933382+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:23.687742+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":329,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.934570+0000","last_change":"2024-09-21T12:24:48.073954+0000","last_active":"2024-09-21T12:26:14.934570+0000","last_peered":"2024-09-21T12:26:14.934570+0000","last_clean":"2024-09-21T12:26:14.934570+0000","last_became_active":"2024-09-21T12:22:50.923796+0000","last_became_peered":"2024-09-21T12:22:50.923796+0000","last_unstale":"2024-09-21T12:26:14.934570+0000","last_undegraded":"2024-09-21T12:26:14.934570+0000","last_fullsized":"2024-09-21T12:26:14.934570+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:48.073424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:48.073424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:58.857724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":682,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:20.443168+0000","last_change":"2024-09-21T12:25:35.819787+0000","last_active":"2024-09-21T12:26:20.443168+0000","last_peered":"2024-09-21T12:26:20.443168+0000","last_clean":"2024-09-21T12:26:20.443168+0000","last_became_active":"2024-09-21T12:20:14.287543+0000","last_became_peered":"2024-09-21T12:20:14.287543+0000","last_unstale":"2024-09-21T12:26:20.443168+0000","last_undegraded":"2024-09-21T12:26:20.443168+0000","last_fullsized":"2024-09-21T12:26:20.443168+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-21T12:25:35.819707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:25:35.819707+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:05.641221+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":361,"num_read_kb":356,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":301,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.981943+0000","last_change":"2024-09-21T12:26:11.422340+0000","last_active":"2024-09-21T12:26:18.981943+0000","last_peered":"2024-09-21T12:26:18.981943+0000","last_clean":"2024-09-21T12:26:18.981943+0000","last_became_active":"2024-09-21T12:20:19.314217+0000","last_became_peered":"2024-09-21T12:20:19.314217+0000","last_unstale":"2024-09-21T12:26:18.981943+0000","last_undegraded":"2024-09-21T12:26:18.981943+0000","last_fullsized":"2024-09-21T12:26:18.981943+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:11.422293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:11.422293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:13.710870+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":306,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:19.005045+0000","last_change":"2024-09-21T12:25:38.882984+0000","last_active":"2024-09-21T12:26:19.005045+0000","last_peered":"2024-09-21T12:26:19.005045+0000","last_clean":"2024-09-21T12:26:19.005045+0000","last_became_active":"2024-09-21T12:20:21.311188+0000","last_became_peered":"2024-09-21T12:20:21.311188+0000","last_unstale":"2024-09-21T12:26:19.005045+0000","last_undegraded":"2024-09-21T12:26:19.005045+0000","last_fullsized":"2024-09-21T12:26:19.005045+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:38.882915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:38.882915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:00.415003+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"199'8067","reported_seq":8155,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:21.641980+0000","last_change":"2024-09-21T12:24:46.318053+0000","last_active":"2024-09-21T12:26:21.641980+0000","last_peered":"2024-09-21T12:26:21.641980+0000","last_clean":"2024-09-21T12:26:21.641980+0000","last_became_active":"2024-09-21T12:24:46.317644+0000","last_became_peered":"2024-09-21T12:24:46.317644+0000","last_unstale":"2024-09-21T12:26:21.641980+0000","last_undegraded":"2024-09-21T12:26:21.641980+0000","last_fullsized":"2024-09-21T12:26:21.641980+0000","mapping_epoch":160,"log_start":"175'3400","ondisk_log_start":"175'3400","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4667,"log_dups_size":0,"ondisk_log_size":4667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:57.982352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528678912,"num_objects":8067,"num_object_clones":0,"num_object_copies":24201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8067,"num_write_kb":516288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"121'19","reported_seq":476,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.988333+0000","last_change":"2024-09-21T12:24:27.932976+0000","last_active":"2024-09-21T12:26:18.988333+0000","last_peered":"2024-09-21T12:26:18.988333+0000","last_clean":"2024-09-21T12:26:18.988333+0000","last_became_active":"2024-09-21T12:22:50.921690+0000","last_became_peered":"2024-09-21T12:22:50.921690+0000","last_unstale":"2024-09-21T12:26:18.988333+0000","last_undegraded":"2024-09-21T12:26:18.988333+0000","last_fullsized":"2024-09-21T12:26:18.988333+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'19","last_scrub_stamp":"2024-09-21T12:24:27.932891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:27.932891+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:57.001434+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":312,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.945563+0000","last_change":"2024-09-21T12:26:12.394327+0000","last_active":"2024-09-21T12:26:14.945563+0000","last_peered":"2024-09-21T12:26:14.945563+0000","last_clean":"2024-09-21T12:26:14.945563+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:26:14.945563+0000","last_undegraded":"2024-09-21T12:26:14.945563+0000","last_fullsized":"2024-09-21T12:26:14.945563+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:26:12.394247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:12.394247+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:12.954462+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":366,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.980326+0000","last_change":"2024-09-21T12:24:31.089221+0000","last_active":"2024-09-21T12:26:18.980326+0000","last_peered":"2024-09-21T12:26:18.980326+0000","last_clean":"2024-09-21T12:26:18.980326+0000","last_became_active":"2024-09-21T12:22:15.049592+0000","last_became_peered":"2024-09-21T12:22:15.049592+0000","last_unstale":"2024-09-21T12:26:18.980326+0000","last_undegraded":"2024-09-21T12:26:18.980326+0000","last_fullsized":"2024-09-21T12:26:18.980326+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:24:31.089155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:24:31.089155+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":285,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.981102+0000","last_change":"2024-09-21T12:20:19.310280+0000","last_active":"2024-09-21T12:26:18.981102+0000","last_peered":"2024-09-21T12:26:18.981102+0000","last_clean":"2024-09-21T12:26:18.981102+0000","last_became_active":"2024-09-21T12:20:19.309859+0000","last_became_peered":"2024-09-21T12:20:19.309859+0000","last_unstale":"2024-09-21T12:26:18.981102+0000","last_undegraded":"2024-09-21T12:26:18.981102+0000","last_fullsized":"2024-09-21T12:26:18.981102+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:28.753163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":326,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.981813+0000","last_change":"2024-09-21T12:25:32.465748+0000","last_active":"2024-09-21T12:26:18.981813+0000","last_peered":"2024-09-21T12:26:18.981813+0000","last_clean":"2024-09-21T12:26:18.981813+0000","last_became_active":"2024-09-21T12:22:50.913493+0000","last_became_peered":"2024-09-21T12:22:50.913493+0000","last_unstale":"2024-09-21T12:26:18.981813+0000","last_undegraded":"2024-09-21T12:26:18.981813+0000","last_fullsized":"2024-09-21T12:26:18.981813+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:32.465700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:32.465700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:57.893144+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"198'7569","reported_seq":7656,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:17.198433+0000","last_change":"2024-09-21T12:24:46.315286+0000","last_active":"2024-09-21T12:26:17.198433+0000","last_peered":"2024-09-21T12:26:17.198433+0000","last_clean":"2024-09-21T12:26:17.198433+0000","last_became_active":"2024-09-21T12:24:46.314957+0000","last_became_peered":"2024-09-21T12:24:46.314957+0000","last_unstale":"2024-09-21T12:26:17.198433+0000","last_undegraded":"2024-09-21T12:26:17.198433+0000","last_fullsized":"2024-09-21T12:26:17.198433+0000","mapping_epoch":160,"log_start":"174'3200","ondisk_log_start":"174'3200","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4369,"log_dups_size":0,"ondisk_log_size":4369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:57.838388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":496041984,"num_objects":7569,"num_object_clones":0,"num_object_copies":22707,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7569,"num_write_kb":484416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":316,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.990099+0000","last_change":"2024-09-21T12:26:15.952570+0000","last_active":"2024-09-21T12:26:18.990099+0000","last_peered":"2024-09-21T12:26:18.990099+0000","last_clean":"2024-09-21T12:26:18.990099+0000","last_became_active":"2024-09-21T12:23:46.885802+0000","last_became_peered":"2024-09-21T12:23:46.885802+0000","last_unstale":"2024-09-21T12:26:18.990099+0000","last_undegraded":"2024-09-21T12:26:18.990099+0000","last_fullsized":"2024-09-21T12:26:18.990099+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:26:15.952496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:15.952496+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:37.337083+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":226,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.979065+0000","last_change":"2024-09-21T12:22:50.908380+0000","last_active":"2024-09-21T12:26:18.979065+0000","last_peered":"2024-09-21T12:26:18.979065+0000","last_clean":"2024-09-21T12:26:18.979065+0000","last_became_active":"2024-09-21T12:22:50.908060+0000","last_became_peered":"2024-09-21T12:22:50.908060+0000","last_unstale":"2024-09-21T12:26:18.979065+0000","last_undegraded":"2024-09-21T12:26:18.979065+0000","last_fullsized":"2024-09-21T12:26:18.979065+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:05.892153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:05.892153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:26.966868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":322,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.981785+0000","last_change":"2024-09-21T12:23:19.372762+0000","last_active":"2024-09-21T12:26:18.981785+0000","last_peered":"2024-09-21T12:26:18.981785+0000","last_clean":"2024-09-21T12:26:18.981785+0000","last_became_active":"2024-09-21T12:22:50.915081+0000","last_became_peered":"2024-09-21T12:22:50.915081+0000","last_unstale":"2024-09-21T12:26:18.981785+0000","last_undegraded":"2024-09-21T12:26:18.981785+0000","last_fullsized":"2024-09-21T12:26:18.981785+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:19.372719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:19.372719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:03.351727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":286,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.944143+0000","last_change":"2024-09-21T12:23:46.282792+0000","last_active":"2024-09-21T12:26:14.944143+0000","last_peered":"2024-09-21T12:26:14.944143+0000","last_clean":"2024-09-21T12:26:14.944143+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:26:14.944143+0000","last_undegraded":"2024-09-21T12:26:14.944143+0000","last_fullsized":"2024-09-21T12:26:14.944143+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:46.282661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:46.282661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:15.069140+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"199'8059","reported_seq":8148,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:21.780949+0000","last_change":"2024-09-21T12:24:46.323690+0000","last_active":"2024-09-21T12:26:21.780949+0000","last_peered":"2024-09-21T12:26:21.780949+0000","last_clean":"2024-09-21T12:26:21.780949+0000","last_became_active":"2024-09-21T12:24:46.323402+0000","last_became_peered":"2024-09-21T12:24:46.323402+0000","last_unstale":"2024-09-21T12:26:21.780949+0000","last_undegraded":"2024-09-21T12:26:21.780949+0000","last_fullsized":"2024-09-21T12:26:21.780949+0000","mapping_epoch":160,"log_start":"176'3800","ondisk_log_start":"176'3800","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4259,"log_dups_size":0,"ondisk_log_size":4259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:01.998231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528154624,"num_objects":8059,"num_object_clones":0,"num_object_copies":24177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8059,"num_write_kb":515776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"76'79","reported_seq":217,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.981214+0000","last_change":"2024-09-21T12:24:28.372308+0000","last_active":"2024-09-21T12:26:18.981214+0000","last_peered":"2024-09-21T12:26:18.981214+0000","last_clean":"2024-09-21T12:26:18.981214+0000","last_became_active":"2024-09-21T12:22:17.087245+0000","last_became_peered":"2024-09-21T12:22:17.087245+0000","last_unstale":"2024-09-21T12:26:18.981214+0000","last_undegraded":"2024-09-21T12:26:18.981214+0000","last_fullsized":"2024-09-21T12:26:18.981214+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:24:28.372230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:24:28.372230+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":351,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.983997+0000","last_change":"2024-09-21T12:25:42.183260+0000","last_active":"2024-09-21T12:26:18.983997+0000","last_peered":"2024-09-21T12:26:18.983997+0000","last_clean":"2024-09-21T12:26:18.983997+0000","last_became_active":"2024-09-21T12:23:46.897197+0000","last_became_peered":"2024-09-21T12:23:46.897197+0000","last_unstale":"2024-09-21T12:26:18.983997+0000","last_undegraded":"2024-09-21T12:26:18.983997+0000","last_fullsized":"2024-09-21T12:26:18.983997+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-21T12:25:42.183169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:42.183169+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:03.947737+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":338,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.987651+0000","last_change":"2024-09-21T12:26:13.918157+0000","last_active":"2024-09-21T12:26:18.987651+0000","last_peered":"2024-09-21T12:26:18.987651+0000","last_clean":"2024-09-21T12:26:18.987651+0000","last_became_active":"2024-09-21T12:22:50.906886+0000","last_became_peered":"2024-09-21T12:22:50.906886+0000","last_unstale":"2024-09-21T12:26:18.987651+0000","last_undegraded":"2024-09-21T12:26:18.987651+0000","last_fullsized":"2024-09-21T12:26:18.987651+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:13.918088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:13.918088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:43.184195+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":315,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:19.005145+0000","last_change":"2024-09-21T12:23:17.434377+0000","last_active":"2024-09-21T12:26:19.005145+0000","last_peered":"2024-09-21T12:26:19.005145+0000","last_clean":"2024-09-21T12:26:19.005145+0000","last_became_active":"2024-09-21T12:22:50.927438+0000","last_became_peered":"2024-09-21T12:22:50.927438+0000","last_unstale":"2024-09-21T12:26:19.005145+0000","last_undegraded":"2024-09-21T12:26:19.005145+0000","last_fullsized":"2024-09-21T12:26:19.005145+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:17.434261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:17.434261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:54.989691+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":311,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.979399+0000","last_change":"2024-09-21T12:25:39.079146+0000","last_active":"2024-09-21T12:26:18.979399+0000","last_peered":"2024-09-21T12:26:18.979399+0000","last_clean":"2024-09-21T12:26:18.979399+0000","last_became_active":"2024-09-21T12:22:50.913898+0000","last_became_peered":"2024-09-21T12:22:50.913898+0000","last_unstale":"2024-09-21T12:26:18.979399+0000","last_undegraded":"2024-09-21T12:26:18.979399+0000","last_fullsized":"2024-09-21T12:26:18.979399+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:39.079015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:39.079015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:50.881520+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"198'7666","reported_seq":7753,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:17.022714+0000","last_change":"2024-09-21T12:24:46.315895+0000","last_active":"2024-09-21T12:26:17.022714+0000","last_peered":"2024-09-21T12:26:17.022714+0000","last_clean":"2024-09-21T12:26:17.022714+0000","last_became_active":"2024-09-21T12:24:46.315601+0000","last_became_peered":"2024-09-21T12:24:46.315601+0000","last_unstale":"2024-09-21T12:26:17.022714+0000","last_undegraded":"2024-09-21T12:26:17.022714+0000","last_fullsized":"2024-09-21T12:26:17.022714+0000","mapping_epoch":160,"log_start":"176'3500","ondisk_log_start":"176'3500","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4166,"log_dups_size":0,"ondisk_log_size":4166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502398976,"num_objects":7666,"num_object_clones":0,"num_object_copies":22998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7666,"num_write_kb":490624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"76'79","reported_seq":363,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.980295+0000","last_change":"2024-09-21T12:25:43.825573+0000","last_active":"2024-09-21T12:26:18.980295+0000","last_peered":"2024-09-21T12:26:18.980295+0000","last_clean":"2024-09-21T12:26:18.980295+0000","last_became_active":"2024-09-21T12:22:17.072872+0000","last_became_peered":"2024-09-21T12:22:17.072872+0000","last_unstale":"2024-09-21T12:26:18.980295+0000","last_undegraded":"2024-09-21T12:26:18.980295+0000","last_fullsized":"2024-09-21T12:26:18.980295+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:25:43.825511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:25:43.825511+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:08.940139+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":165,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.979809+0000","last_change":"2024-09-21T12:26:10.141012+0000","last_active":"2024-09-21T12:26:18.979809+0000","last_peered":"2024-09-21T12:26:18.979809+0000","last_clean":"2024-09-21T12:26:18.979809+0000","last_became_active":"2024-09-21T12:23:46.902260+0000","last_became_peered":"2024-09-21T12:23:46.902260+0000","last_unstale":"2024-09-21T12:26:18.979809+0000","last_undegraded":"2024-09-21T12:26:18.979809+0000","last_fullsized":"2024-09-21T12:26:18.979809+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:26:10.137029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:10.137029+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:19.970928+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":319,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.946234+0000","last_change":"2024-09-21T12:22:50.928240+0000","last_active":"2024-09-21T12:26:14.946234+0000","last_peered":"2024-09-21T12:26:14.946234+0000","last_clean":"2024-09-21T12:26:14.946234+0000","last_became_active":"2024-09-21T12:22:50.927946+0000","last_became_peered":"2024-09-21T12:22:50.927946+0000","last_unstale":"2024-09-21T12:26:14.946234+0000","last_undegraded":"2024-09-21T12:26:14.946234+0000","last_fullsized":"2024-09-21T12:26:14.946234+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:08.999015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:08.999015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:24.102761+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":392,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.994440+0000","last_change":"2024-09-21T12:26:13.114174+0000","last_active":"2024-09-21T12:26:18.994440+0000","last_peered":"2024-09-21T12:26:18.994440+0000","last_clean":"2024-09-21T12:26:18.994440+0000","last_became_active":"2024-09-21T12:20:19.307924+0000","last_became_peered":"2024-09-21T12:20:19.307924+0000","last_unstale":"2024-09-21T12:26:18.994440+0000","last_undegraded":"2024-09-21T12:26:18.994440+0000","last_fullsized":"2024-09-21T12:26:18.994440+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:26:13.113863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:13.113863+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:23.394893+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":234,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:19.001414+0000","last_change":"2024-09-21T12:23:56.471960+0000","last_active":"2024-09-21T12:26:19.001414+0000","last_peered":"2024-09-21T12:26:19.001414+0000","last_clean":"2024-09-21T12:26:19.001414+0000","last_became_active":"2024-09-21T12:22:50.905470+0000","last_became_peered":"2024-09-21T12:22:50.905470+0000","last_unstale":"2024-09-21T12:26:19.001414+0000","last_undegraded":"2024-09-21T12:26:19.001414+0000","last_fullsized":"2024-09-21T12:26:19.001414+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:56.471116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:56.471116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:22.516139+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.7","version":"199'7955","reported_seq":8043,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:21.820908+0000","last_change":"2024-09-21T12:24:46.317724+0000","last_active":"2024-09-21T12:26:21.820908+0000","last_peered":"2024-09-21T12:26:21.820908+0000","last_clean":"2024-09-21T12:26:21.820908+0000","last_became_active":"2024-09-21T12:24:46.317468+0000","last_became_peered":"2024-09-21T12:24:46.317468+0000","last_unstale":"2024-09-21T12:26:21.820908+0000","last_undegraded":"2024-09-21T12:26:21.820908+0000","last_fullsized":"2024-09-21T12:26:21.820908+0000","mapping_epoch":160,"log_start":"176'3700","ondisk_log_start":"176'3700","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4255,"log_dups_size":0,"ondisk_log_size":4255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:01.017517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":521338880,"num_objects":7955,"num_object_clones":0,"num_object_copies":23865,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7955,"num_write_kb":509120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":379,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.933354+0000","last_change":"2024-09-21T12:25:00.237083+0000","last_active":"2024-09-21T12:26:14.933354+0000","last_peered":"2024-09-21T12:26:14.933354+0000","last_clean":"2024-09-21T12:26:14.933354+0000","last_became_active":"2024-09-21T12:22:50.922343+0000","last_became_peered":"2024-09-21T12:22:50.922343+0000","last_unstale":"2024-09-21T12:26:14.933354+0000","last_undegraded":"2024-09-21T12:26:14.933354+0000","last_fullsized":"2024-09-21T12:26:14.933354+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:25:00.236990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:00.236990+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:26.107161+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":317,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.982119+0000","last_change":"2024-09-21T12:25:30.451939+0000","last_active":"2024-09-21T12:26:18.982119+0000","last_peered":"2024-09-21T12:26:18.982119+0000","last_clean":"2024-09-21T12:26:18.982119+0000","last_became_active":"2024-09-21T12:20:15.302262+0000","last_became_peered":"2024-09-21T12:20:15.302262+0000","last_unstale":"2024-09-21T12:26:18.982119+0000","last_undegraded":"2024-09-21T12:26:18.982119+0000","last_fullsized":"2024-09-21T12:26:18.982119+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:30.451876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:30.451876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:51.913189+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":331,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.981225+0000","last_change":"2024-09-21T12:25:36.012066+0000","last_active":"2024-09-21T12:26:18.981225+0000","last_peered":"2024-09-21T12:26:18.981225+0000","last_clean":"2024-09-21T12:26:18.981225+0000","last_became_active":"2024-09-21T12:22:50.925974+0000","last_became_peered":"2024-09-21T12:22:50.925974+0000","last_unstale":"2024-09-21T12:26:18.981225+0000","last_undegraded":"2024-09-21T12:26:18.981225+0000","last_fullsized":"2024-09-21T12:26:18.981225+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:36.011984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:36.011984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:40.801014+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":288,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.934478+0000","last_change":"2024-09-21T12:23:28.121344+0000","last_active":"2024-09-21T12:26:14.934478+0000","last_peered":"2024-09-21T12:26:14.934478+0000","last_clean":"2024-09-21T12:26:14.934478+0000","last_became_active":"2024-09-21T12:20:21.311493+0000","last_became_peered":"2024-09-21T12:20:21.311493+0000","last_unstale":"2024-09-21T12:26:14.934478+0000","last_undegraded":"2024-09-21T12:26:14.934478+0000","last_fullsized":"2024-09-21T12:26:14.934478+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:28.121286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:28.121286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:24.681035+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"198'7667","reported_seq":7754,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:17.148267+0000","last_change":"2024-09-21T12:24:46.327091+0000","last_active":"2024-09-21T12:26:17.148267+0000","last_peered":"2024-09-21T12:26:17.148267+0000","last_clean":"2024-09-21T12:26:17.148267+0000","last_became_active":"2024-09-21T12:24:46.325838+0000","last_became_peered":"2024-09-21T12:24:46.325838+0000","last_unstale":"2024-09-21T12:26:17.148267+0000","last_undegraded":"2024-09-21T12:26:17.148267+0000","last_fullsized":"2024-09-21T12:26:17.148267+0000","mapping_epoch":160,"log_start":"174'3300","ondisk_log_start":"174'3300","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4367,"log_dups_size":0,"ondisk_log_size":4367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502464512,"num_objects":7667,"num_object_clones":0,"num_object_copies":23001,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7667,"num_write_kb":490688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":336,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.987270+0000","last_change":"2024-09-21T12:24:02.960807+0000","last_active":"2024-09-21T12:26:18.987270+0000","last_peered":"2024-09-21T12:26:18.987270+0000","last_clean":"2024-09-21T12:26:18.987270+0000","last_became_active":"2024-09-21T12:20:17.692867+0000","last_became_peered":"2024-09-21T12:20:17.692867+0000","last_unstale":"2024-09-21T12:26:18.987270+0000","last_undegraded":"2024-09-21T12:26:18.987270+0000","last_fullsized":"2024-09-21T12:26:18.987270+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-21T12:24:02.960733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:02.960733+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":301,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.980715+0000","last_change":"2024-09-21T12:23:59.046000+0000","last_active":"2024-09-21T12:26:18.980715+0000","last_peered":"2024-09-21T12:26:18.980715+0000","last_clean":"2024-09-21T12:26:18.980715+0000","last_became_active":"2024-09-21T12:20:15.299333+0000","last_became_peered":"2024-09-21T12:20:15.299333+0000","last_unstale":"2024-09-21T12:26:18.980715+0000","last_undegraded":"2024-09-21T12:26:18.980715+0000","last_fullsized":"2024-09-21T12:26:18.980715+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:59.045927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:23:59.045927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:55.760821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":294,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.979215+0000","last_change":"2024-09-21T12:24:34.060244+0000","last_active":"2024-09-21T12:26:18.979215+0000","last_peered":"2024-09-21T12:26:18.979215+0000","last_clean":"2024-09-21T12:26:18.979215+0000","last_became_active":"2024-09-21T12:20:19.306952+0000","last_became_peered":"2024-09-21T12:20:19.306952+0000","last_unstale":"2024-09-21T12:26:18.979215+0000","last_undegraded":"2024-09-21T12:26:18.979215+0000","last_fullsized":"2024-09-21T12:26:18.979215+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:34.060188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:34.060188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:50.161659+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":281,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.980214+0000","last_change":"2024-09-21T12:20:21.314891+0000","last_active":"2024-09-21T12:26:18.980214+0000","last_peered":"2024-09-21T12:26:18.980214+0000","last_clean":"2024-09-21T12:26:18.980214+0000","last_became_active":"2024-09-21T12:20:21.314545+0000","last_became_peered":"2024-09-21T12:20:21.314545+0000","last_unstale":"2024-09-21T12:26:18.980214+0000","last_undegraded":"2024-09-21T12:26:18.980214+0000","last_fullsized":"2024-09-21T12:26:18.980214+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:49.780292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"199'7974","reported_seq":8075,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:21.632505+0000","last_change":"2024-09-21T12:25:49.210492+0000","last_active":"2024-09-21T12:26:21.632505+0000","last_peered":"2024-09-21T12:26:21.632505+0000","last_clean":"2024-09-21T12:26:21.632505+0000","last_became_active":"2024-09-21T12:24:46.324554+0000","last_became_peered":"2024-09-21T12:24:46.324554+0000","last_unstale":"2024-09-21T12:26:21.632505+0000","last_undegraded":"2024-09-21T12:26:21.632505+0000","last_fullsized":"2024-09-21T12:26:21.632505+0000","mapping_epoch":160,"log_start":"175'3300","ondisk_log_start":"175'3300","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":899,"log_size":4674,"log_dups_size":0,"ondisk_log_size":4674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":522584064,"num_objects":7974,"num_object_clones":0,"num_object_copies":23922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7974,"num_write_kb":510336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":309,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:19.009171+0000","last_change":"2024-09-21T12:24:54.445801+0000","last_active":"2024-09-21T12:26:19.009171+0000","last_peered":"2024-09-21T12:26:19.009171+0000","last_clean":"2024-09-21T12:26:19.009171+0000","last_became_active":"2024-09-21T12:22:50.910973+0000","last_became_peered":"2024-09-21T12:22:50.910973+0000","last_unstale":"2024-09-21T12:26:19.009171+0000","last_undegraded":"2024-09-21T12:26:19.009171+0000","last_fullsized":"2024-09-21T12:26:19.009171+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:24:54.445714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:54.445714+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":330,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.981124+0000","last_change":"2024-09-21T12:25:44.800799+0000","last_active":"2024-09-21T12:26:18.981124+0000","last_peered":"2024-09-21T12:26:18.981124+0000","last_clean":"2024-09-21T12:26:18.981124+0000","last_became_active":"2024-09-21T12:22:50.927859+0000","last_became_peered":"2024-09-21T12:22:50.927859+0000","last_unstale":"2024-09-21T12:26:18.981124+0000","last_undegraded":"2024-09-21T12:26:18.981124+0000","last_fullsized":"2024-09-21T12:26:18.981124+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:44.800657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:44.800657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:49.068224+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":302,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.982510+0000","last_change":"2024-09-21T12:25:40.148844+0000","last_active":"2024-09-21T12:26:18.982510+0000","last_peered":"2024-09-21T12:26:18.982510+0000","last_clean":"2024-09-21T12:26:18.982510+0000","last_became_active":"2024-09-21T12:20:19.312572+0000","last_became_peered":"2024-09-21T12:20:19.312572+0000","last_unstale":"2024-09-21T12:26:18.982510+0000","last_undegraded":"2024-09-21T12:26:18.982510+0000","last_fullsized":"2024-09-21T12:26:18.982510+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:40.148766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:40.148766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:07.648989+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":286,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.945891+0000","last_change":"2024-09-21T12:22:36.006177+0000","last_active":"2024-09-21T12:26:14.945891+0000","last_peered":"2024-09-21T12:26:14.945891+0000","last_clean":"2024-09-21T12:26:14.945891+0000","last_became_active":"2024-09-21T12:20:21.317223+0000","last_became_peered":"2024-09-21T12:20:21.317223+0000","last_unstale":"2024-09-21T12:26:14.945891+0000","last_undegraded":"2024-09-21T12:26:14.945891+0000","last_fullsized":"2024-09-21T12:26:14.945891+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:36.006109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:36.006109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:54.431499+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"199'8062","reported_seq":8151,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:21.575794+0000","last_change":"2024-09-21T12:24:46.323500+0000","last_active":"2024-09-21T12:26:21.575794+0000","last_peered":"2024-09-21T12:26:21.575794+0000","last_clean":"2024-09-21T12:26:21.575794+0000","last_became_active":"2024-09-21T12:24:46.323192+0000","last_became_peered":"2024-09-21T12:24:46.323192+0000","last_unstale":"2024-09-21T12:26:21.575794+0000","last_undegraded":"2024-09-21T12:26:21.575794+0000","last_fullsized":"2024-09-21T12:26:21.575794+0000","mapping_epoch":160,"log_start":"176'3600","ondisk_log_start":"176'3600","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4462,"log_dups_size":0,"ondisk_log_size":4462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528351232,"num_objects":8062,"num_object_clones":0,"num_object_copies":24186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8062,"num_write_kb":515968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":338,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.981722+0000","last_change":"2024-09-21T12:25:03.416100+0000","last_active":"2024-09-21T12:26:18.981722+0000","last_peered":"2024-09-21T12:26:18.981722+0000","last_clean":"2024-09-21T12:26:18.981722+0000","last_became_active":"2024-09-21T12:20:17.692870+0000","last_became_peered":"2024-09-21T12:20:17.692870+0000","last_unstale":"2024-09-21T12:26:18.981722+0000","last_undegraded":"2024-09-21T12:26:18.981722+0000","last_fullsized":"2024-09-21T12:26:18.981722+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:25:03.416033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:03.416033+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":360,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.981116+0000","last_change":"2024-09-21T12:24:29.054851+0000","last_active":"2024-09-21T12:26:18.981116+0000","last_peered":"2024-09-21T12:26:18.981116+0000","last_clean":"2024-09-21T12:26:18.981116+0000","last_became_active":"2024-09-21T12:22:50.927141+0000","last_became_peered":"2024-09-21T12:22:50.927141+0000","last_unstale":"2024-09-21T12:26:18.981116+0000","last_undegraded":"2024-09-21T12:26:18.981116+0000","last_fullsized":"2024-09-21T12:26:18.981116+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-21T12:24:29.054772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:29.054772+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:07.827845+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"121'11","reported_seq":408,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.980225+0000","last_change":"2024-09-21T12:25:02.916145+0000","last_active":"2024-09-21T12:26:18.980225+0000","last_peered":"2024-09-21T12:26:18.980225+0000","last_clean":"2024-09-21T12:26:18.980225+0000","last_became_active":"2024-09-21T12:22:50.908371+0000","last_became_peered":"2024-09-21T12:22:50.908371+0000","last_unstale":"2024-09-21T12:26:18.980225+0000","last_undegraded":"2024-09-21T12:26:18.980225+0000","last_fullsized":"2024-09-21T12:26:18.980225+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:25:02.916090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:02.916090+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:31.534554+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":289,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.981691+0000","last_change":"2024-09-21T12:22:45.223570+0000","last_active":"2024-09-21T12:26:18.981691+0000","last_peered":"2024-09-21T12:26:18.981691+0000","last_clean":"2024-09-21T12:26:18.981691+0000","last_became_active":"2024-09-21T12:20:21.592576+0000","last_became_peered":"2024-09-21T12:20:21.592576+0000","last_unstale":"2024-09-21T12:26:18.981691+0000","last_undegraded":"2024-09-21T12:26:18.981691+0000","last_fullsized":"2024-09-21T12:26:18.981691+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:45.223396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:45.223396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:20.386745+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":325,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.987151+0000","last_change":"2024-09-21T12:25:31.845118+0000","last_active":"2024-09-21T12:26:18.987151+0000","last_peered":"2024-09-21T12:26:18.987151+0000","last_clean":"2024-09-21T12:26:18.987151+0000","last_became_active":"2024-09-21T12:20:15.303480+0000","last_became_peered":"2024-09-21T12:20:15.303480+0000","last_unstale":"2024-09-21T12:26:18.987151+0000","last_undegraded":"2024-09-21T12:26:18.987151+0000","last_fullsized":"2024-09-21T12:26:18.987151+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:31.845066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:31.845066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:34.886910+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":303,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.951008+0000","last_change":"2024-09-21T12:22:36.999995+0000","last_active":"2024-09-21T12:26:14.951008+0000","last_peered":"2024-09-21T12:26:14.951008+0000","last_clean":"2024-09-21T12:26:14.951008+0000","last_became_active":"2024-09-21T12:20:17.694628+0000","last_became_peered":"2024-09-21T12:20:17.694628+0000","last_unstale":"2024-09-21T12:26:14.951008+0000","last_undegraded":"2024-09-21T12:26:14.951008+0000","last_fullsized":"2024-09-21T12:26:14.951008+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-21T12:22:36.999705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:36.999705+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:51.345667+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":309,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.982193+0000","last_change":"2024-09-21T12:25:02.414377+0000","last_active":"2024-09-21T12:26:18.982193+0000","last_peered":"2024-09-21T12:26:18.982193+0000","last_clean":"2024-09-21T12:26:18.982193+0000","last_became_active":"2024-09-21T12:20:19.313882+0000","last_became_peered":"2024-09-21T12:20:19.313882+0000","last_unstale":"2024-09-21T12:26:18.982193+0000","last_undegraded":"2024-09-21T12:26:18.982193+0000","last_fullsized":"2024-09-21T12:26:18.982193+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:02.414298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:02.414298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":310,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.980468+0000","last_change":"2024-09-21T12:24:00.066522+0000","last_active":"2024-09-21T12:26:18.980468+0000","last_peered":"2024-09-21T12:26:18.980468+0000","last_clean":"2024-09-21T12:26:18.980468+0000","last_became_active":"2024-09-21T12:22:50.904404+0000","last_became_peered":"2024-09-21T12:22:50.904404+0000","last_unstale":"2024-09-21T12:26:18.980468+0000","last_undegraded":"2024-09-21T12:26:18.980468+0000","last_fullsized":"2024-09-21T12:26:18.980468+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:00.066451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:00.066451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:01.771403+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":312,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.937059+0000","last_change":"2024-09-21T12:21:24.093710+0000","last_active":"2024-09-21T12:26:14.937059+0000","last_peered":"2024-09-21T12:26:14.937059+0000","last_clean":"2024-09-21T12:26:14.937059+0000","last_became_active":"2024-09-21T12:20:17.695898+0000","last_became_peered":"2024-09-21T12:20:17.695898+0000","last_unstale":"2024-09-21T12:26:14.937059+0000","last_undegraded":"2024-09-21T12:26:14.937059+0000","last_fullsized":"2024-09-21T12:26:14.937059+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:21:24.093656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:21:24.093656+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:20.587663+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":338,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.980508+0000","last_change":"2024-09-21T12:26:16.907351+0000","last_active":"2024-09-21T12:26:18.980508+0000","last_peered":"2024-09-21T12:26:18.980508+0000","last_clean":"2024-09-21T12:26:18.980508+0000","last_became_active":"2024-09-21T12:22:50.927649+0000","last_became_peered":"2024-09-21T12:22:50.927649+0000","last_unstale":"2024-09-21T12:26:18.980508+0000","last_undegraded":"2024-09-21T12:26:18.980508+0000","last_fullsized":"2024-09-21T12:26:18.980508+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:16.907284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:16.907284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:30.534090+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":290,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.950777+0000","last_change":"2024-09-21T12:23:53.356954+0000","last_active":"2024-09-21T12:26:14.950777+0000","last_peered":"2024-09-21T12:26:14.950777+0000","last_clean":"2024-09-21T12:26:14.950777+0000","last_became_active":"2024-09-21T12:20:19.311947+0000","last_became_peered":"2024-09-21T12:20:19.311947+0000","last_unstale":"2024-09-21T12:26:14.950777+0000","last_undegraded":"2024-09-21T12:26:14.950777+0000","last_fullsized":"2024-09-21T12:26:14.950777+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:53.356896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:53.356896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:28.202782+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":335,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.979955+0000","last_change":"2024-09-21T12:26:11.105716+0000","last_active":"2024-09-21T12:26:18.979955+0000","last_peered":"2024-09-21T12:26:18.979955+0000","last_clean":"2024-09-21T12:26:18.979955+0000","last_became_active":"2024-09-21T12:22:50.924931+0000","last_became_peered":"2024-09-21T12:22:50.924931+0000","last_unstale":"2024-09-21T12:26:18.979955+0000","last_undegraded":"2024-09-21T12:26:18.979955+0000","last_fullsized":"2024-09-21T12:26:18.979955+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:11.105583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:11.105583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:36.449727+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":338,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.981967+0000","last_change":"2024-09-21T12:26:18.481098+0000","last_active":"2024-09-21T12:26:18.981967+0000","last_peered":"2024-09-21T12:26:18.981967+0000","last_clean":"2024-09-21T12:26:18.981967+0000","last_became_active":"2024-09-21T12:23:46.898523+0000","last_became_peered":"2024-09-21T12:23:46.898523+0000","last_unstale":"2024-09-21T12:26:18.981967+0000","last_undegraded":"2024-09-21T12:26:18.981967+0000","last_fullsized":"2024-09-21T12:26:18.981967+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:26:18.480630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:18.480630+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:39.872336+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":308,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.935383+0000","last_change":"2024-09-21T12:24:25.931617+0000","last_active":"2024-09-21T12:26:14.935383+0000","last_peered":"2024-09-21T12:26:14.935383+0000","last_clean":"2024-09-21T12:26:14.935383+0000","last_became_active":"2024-09-21T12:20:15.297519+0000","last_became_peered":"2024-09-21T12:20:15.297519+0000","last_unstale":"2024-09-21T12:26:14.935383+0000","last_undegraded":"2024-09-21T12:26:14.935383+0000","last_fullsized":"2024-09-21T12:26:14.935383+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:25.931561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:25.931561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:54.835832+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":371,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.945027+0000","last_change":"2024-09-21T12:23:57.379033+0000","last_active":"2024-09-21T12:26:14.945027+0000","last_peered":"2024-09-21T12:26:14.945027+0000","last_clean":"2024-09-21T12:26:14.945027+0000","last_became_active":"2024-09-21T12:20:19.305364+0000","last_became_peered":"2024-09-21T12:20:19.305364+0000","last_unstale":"2024-09-21T12:26:14.945027+0000","last_undegraded":"2024-09-21T12:26:14.945027+0000","last_fullsized":"2024-09-21T12:26:14.945027+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:23:57.378959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:57.378959+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:16.020266+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":233,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.979485+0000","last_change":"2024-09-21T12:24:51.831043+0000","last_active":"2024-09-21T12:26:18.979485+0000","last_peered":"2024-09-21T12:26:18.979485+0000","last_clean":"2024-09-21T12:26:18.979485+0000","last_became_active":"2024-09-21T12:22:50.924770+0000","last_became_peered":"2024-09-21T12:22:50.924770+0000","last_unstale":"2024-09-21T12:26:18.979485+0000","last_undegraded":"2024-09-21T12:26:18.979485+0000","last_fullsized":"2024-09-21T12:26:18.979485+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:51.830994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:51.830994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:58.084812+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":317,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.944813+0000","last_change":"2024-09-21T12:24:50.727161+0000","last_active":"2024-09-21T12:26:14.944813+0000","last_peered":"2024-09-21T12:26:14.944813+0000","last_clean":"2024-09-21T12:26:14.944813+0000","last_became_active":"2024-09-21T12:20:17.693209+0000","last_became_peered":"2024-09-21T12:20:17.693209+0000","last_unstale":"2024-09-21T12:26:14.944813+0000","last_undegraded":"2024-09-21T12:26:14.944813+0000","last_fullsized":"2024-09-21T12:26:14.944813+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:24:50.727101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:50.727101+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":330,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.980579+0000","last_change":"2024-09-21T12:25:36.808211+0000","last_active":"2024-09-21T12:26:18.980579+0000","last_peered":"2024-09-21T12:26:18.980579+0000","last_clean":"2024-09-21T12:26:18.980579+0000","last_became_active":"2024-09-21T12:22:50.908238+0000","last_became_peered":"2024-09-21T12:22:50.908238+0000","last_unstale":"2024-09-21T12:26:18.980579+0000","last_undegraded":"2024-09-21T12:26:18.980579+0000","last_fullsized":"2024-09-21T12:26:18.980579+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:36.808147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:36.808147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:50.893952+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":375,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.981740+0000","last_change":"2024-09-21T12:22:46.216242+0000","last_active":"2024-09-21T12:26:18.981740+0000","last_peered":"2024-09-21T12:26:18.981740+0000","last_clean":"2024-09-21T12:26:18.981740+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:26:18.981740+0000","last_undegraded":"2024-09-21T12:26:18.981740+0000","last_fullsized":"2024-09-21T12:26:18.981740+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:22:46.216153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:46.216153+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:22.367827+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":288,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.936592+0000","last_change":"2024-09-21T12:26:08.926904+0000","last_active":"2024-09-21T12:26:14.936592+0000","last_peered":"2024-09-21T12:26:14.936592+0000","last_clean":"2024-09-21T12:26:14.936592+0000","last_became_active":"2024-09-21T12:20:21.309385+0000","last_became_peered":"2024-09-21T12:20:21.309385+0000","last_unstale":"2024-09-21T12:26:14.936592+0000","last_undegraded":"2024-09-21T12:26:14.936592+0000","last_fullsized":"2024-09-21T12:26:14.936592+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:08.923458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:08.923458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:38.391592+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":289,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.987485+0000","last_change":"2024-09-21T12:23:59.046758+0000","last_active":"2024-09-21T12:26:18.987485+0000","last_peered":"2024-09-21T12:26:18.987485+0000","last_clean":"2024-09-21T12:26:18.987485+0000","last_became_active":"2024-09-21T12:22:50.906473+0000","last_became_peered":"2024-09-21T12:22:50.906473+0000","last_unstale":"2024-09-21T12:26:18.987485+0000","last_undegraded":"2024-09-21T12:26:18.987485+0000","last_fullsized":"2024-09-21T12:26:18.987485+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:23:59.046704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:59.046704+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":226,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.981250+0000","last_change":"2024-09-21T12:24:58.948698+0000","last_active":"2024-09-21T12:26:18.981250+0000","last_peered":"2024-09-21T12:26:18.981250+0000","last_clean":"2024-09-21T12:26:18.981250+0000","last_became_active":"2024-09-21T12:22:50.903705+0000","last_became_peered":"2024-09-21T12:22:50.903705+0000","last_unstale":"2024-09-21T12:26:18.981250+0000","last_undegraded":"2024-09-21T12:26:18.981250+0000","last_fullsized":"2024-09-21T12:26:18.981250+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:58.948625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:58.948625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:23.221920+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":297,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:19.001264+0000","last_change":"2024-09-21T12:26:16.149823+0000","last_active":"2024-09-21T12:26:19.001264+0000","last_peered":"2024-09-21T12:26:19.001264+0000","last_clean":"2024-09-21T12:26:19.001264+0000","last_became_active":"2024-09-21T12:20:19.315418+0000","last_became_peered":"2024-09-21T12:20:19.315418+0000","last_unstale":"2024-09-21T12:26:19.001264+0000","last_undegraded":"2024-09-21T12:26:19.001264+0000","last_fullsized":"2024-09-21T12:26:19.001264+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:16.149550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:16.149550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:27.681950+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":294,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.943201+0000","last_change":"2024-09-21T12:24:56.879617+0000","last_active":"2024-09-21T12:26:14.943201+0000","last_peered":"2024-09-21T12:26:14.943201+0000","last_clean":"2024-09-21T12:26:14.943201+0000","last_became_active":"2024-09-21T12:20:21.310053+0000","last_became_peered":"2024-09-21T12:20:21.310053+0000","last_unstale":"2024-09-21T12:26:14.943201+0000","last_undegraded":"2024-09-21T12:26:14.943201+0000","last_fullsized":"2024-09-21T12:26:14.943201+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:56.879558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:56.879558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:26.453298+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":314,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:19.001489+0000","last_change":"2024-09-21T12:25:41.888249+0000","last_active":"2024-09-21T12:26:19.001489+0000","last_peered":"2024-09-21T12:26:19.001489+0000","last_clean":"2024-09-21T12:26:19.001489+0000","last_became_active":"2024-09-21T12:20:17.299015+0000","last_became_peered":"2024-09-21T12:20:17.299015+0000","last_unstale":"2024-09-21T12:26:19.001489+0000","last_undegraded":"2024-09-21T12:26:19.001489+0000","last_fullsized":"2024-09-21T12:26:19.001489+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:25:41.888037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:41.888037+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:10.841319+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":338,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:19.973528+0000","last_change":"2024-09-21T12:26:19.973528+0000","last_active":"2024-09-21T12:26:19.973528+0000","last_peered":"2024-09-21T12:26:19.973528+0000","last_clean":"2024-09-21T12:26:19.973528+0000","last_became_active":"2024-09-21T12:22:50.922019+0000","last_became_peered":"2024-09-21T12:22:50.922019+0000","last_unstale":"2024-09-21T12:26:19.973528+0000","last_undegraded":"2024-09-21T12:26:19.973528+0000","last_fullsized":"2024-09-21T12:26:19.973528+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:19.973461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:19.973461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:36.925984+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":290,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.945967+0000","last_change":"2024-09-21T12:21:20.931077+0000","last_active":"2024-09-21T12:26:14.945967+0000","last_peered":"2024-09-21T12:26:14.945967+0000","last_clean":"2024-09-21T12:26:14.945967+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:26:14.945967+0000","last_undegraded":"2024-09-21T12:26:14.945967+0000","last_fullsized":"2024-09-21T12:26:14.945967+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:20.931025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.931025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:18.080044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":288,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.933024+0000","last_change":"2024-09-21T12:23:29.134829+0000","last_active":"2024-09-21T12:26:14.933024+0000","last_peered":"2024-09-21T12:26:14.933024+0000","last_clean":"2024-09-21T12:26:14.933024+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:26:14.933024+0000","last_undegraded":"2024-09-21T12:26:14.933024+0000","last_fullsized":"2024-09-21T12:26:14.933024+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:29.134767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:29.134767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":321,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.943422+0000","last_change":"2024-09-21T12:24:58.824615+0000","last_active":"2024-09-21T12:26:14.943422+0000","last_peered":"2024-09-21T12:26:14.943422+0000","last_clean":"2024-09-21T12:26:14.943422+0000","last_became_active":"2024-09-21T12:20:17.694794+0000","last_became_peered":"2024-09-21T12:20:17.694794+0000","last_unstale":"2024-09-21T12:26:14.943422+0000","last_undegraded":"2024-09-21T12:26:14.943422+0000","last_fullsized":"2024-09-21T12:26:14.943422+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:24:58.824516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:58.824516+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:24.775664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":317,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.982258+0000","last_change":"2024-09-21T12:24:33.364225+0000","last_active":"2024-09-21T12:26:18.982258+0000","last_peered":"2024-09-21T12:26:18.982258+0000","last_clean":"2024-09-21T12:26:18.982258+0000","last_became_active":"2024-09-21T12:20:15.295968+0000","last_became_peered":"2024-09-21T12:20:15.295968+0000","last_unstale":"2024-09-21T12:26:18.982258+0000","last_undegraded":"2024-09-21T12:26:18.982258+0000","last_fullsized":"2024-09-21T12:26:18.982258+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:33.363808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:33.363808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":314,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.979929+0000","last_change":"2024-09-21T12:23:57.018797+0000","last_active":"2024-09-21T12:26:18.979929+0000","last_peered":"2024-09-21T12:26:18.979929+0000","last_clean":"2024-09-21T12:26:18.979929+0000","last_became_active":"2024-09-21T12:22:50.921604+0000","last_became_peered":"2024-09-21T12:22:50.921604+0000","last_unstale":"2024-09-21T12:26:18.979929+0000","last_undegraded":"2024-09-21T12:26:18.979929+0000","last_fullsized":"2024-09-21T12:26:18.979929+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:57.018749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:57.018749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:53.766292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":286,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.946148+0000","last_change":"2024-09-21T12:24:35.588409+0000","last_active":"2024-09-21T12:26:14.946148+0000","last_peered":"2024-09-21T12:26:14.946148+0000","last_clean":"2024-09-21T12:26:14.946148+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:26:14.946148+0000","last_undegraded":"2024-09-21T12:26:14.946148+0000","last_fullsized":"2024-09-21T12:26:14.946148+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:35.588353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:35.588353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":300,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.949114+0000","last_change":"2024-09-21T12:23:51.343568+0000","last_active":"2024-09-21T12:26:14.949114+0000","last_peered":"2024-09-21T12:26:14.949114+0000","last_clean":"2024-09-21T12:26:14.949114+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:26:14.949114+0000","last_undegraded":"2024-09-21T12:26:14.949114+0000","last_fullsized":"2024-09-21T12:26:14.949114+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-21T12:23:51.343491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:51.343491+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:26.153152+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":314,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.937415+0000","last_change":"2024-09-21T12:25:48.713463+0000","last_active":"2024-09-21T12:26:14.937415+0000","last_peered":"2024-09-21T12:26:14.937415+0000","last_clean":"2024-09-21T12:26:14.937415+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:26:14.937415+0000","last_undegraded":"2024-09-21T12:26:14.937415+0000","last_fullsized":"2024-09-21T12:26:14.937415+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:25:48.713392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:48.713392+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:06.809432+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":314,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.987614+0000","last_change":"2024-09-21T12:22:50.931202+0000","last_active":"2024-09-21T12:26:18.987614+0000","last_peered":"2024-09-21T12:26:18.987614+0000","last_clean":"2024-09-21T12:26:18.987614+0000","last_became_active":"2024-09-21T12:22:50.930615+0000","last_became_peered":"2024-09-21T12:22:50.930615+0000","last_unstale":"2024-09-21T12:26:18.987614+0000","last_undegraded":"2024-09-21T12:26:18.987614+0000","last_fullsized":"2024-09-21T12:26:18.987614+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:09.649109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:09.649109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:20.855882+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":310,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.980201+0000","last_change":"2024-09-21T12:26:14.855364+0000","last_active":"2024-09-21T12:26:18.980201+0000","last_peered":"2024-09-21T12:26:18.980201+0000","last_clean":"2024-09-21T12:26:18.980201+0000","last_became_active":"2024-09-21T12:22:50.917408+0000","last_became_peered":"2024-09-21T12:22:50.917408+0000","last_unstale":"2024-09-21T12:26:18.980201+0000","last_undegraded":"2024-09-21T12:26:18.980201+0000","last_fullsized":"2024-09-21T12:26:18.980201+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:14.855248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:14.855248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:15.644031+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":311,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:19.000650+0000","last_change":"2024-09-21T12:26:15.152613+0000","last_active":"2024-09-21T12:26:19.000650+0000","last_peered":"2024-09-21T12:26:19.000650+0000","last_clean":"2024-09-21T12:26:19.000650+0000","last_became_active":"2024-09-21T12:22:50.914221+0000","last_became_peered":"2024-09-21T12:22:50.914221+0000","last_unstale":"2024-09-21T12:26:19.000650+0000","last_undegraded":"2024-09-21T12:26:19.000650+0000","last_fullsized":"2024-09-21T12:26:19.000650+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:15.152540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:15.152540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:17.180451+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":306,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.949347+0000","last_change":"2024-09-21T12:25:00.871133+0000","last_active":"2024-09-21T12:26:14.949347+0000","last_peered":"2024-09-21T12:26:14.949347+0000","last_clean":"2024-09-21T12:26:14.949347+0000","last_became_active":"2024-09-21T12:20:15.297993+0000","last_became_peered":"2024-09-21T12:20:15.297993+0000","last_unstale":"2024-09-21T12:26:14.949347+0000","last_undegraded":"2024-09-21T12:26:14.949347+0000","last_fullsized":"2024-09-21T12:26:14.949347+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:00.871062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:00.871062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":316,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T12:26:18.988604+0000","last_change":"2024-09-21T12:25:47.974973+0000","last_active":"2024-09-21T12:26:18.988604+0000","last_peered":"2024-09-21T12:26:18.988604+0000","last_clean":"2024-09-21T12:26:18.988604+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:26:18.988604+0000","last_undegraded":"2024-09-21T12:26:18.988604+0000","last_fullsized":"2024-09-21T12:26:18.988604+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:25:47.974897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:47.974897+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:04.942357+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":308,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-09-21T12:26:14.936759+0000","last_change":"2024-09-21T12:25:32.660054+0000","last_active":"2024-09-21T12:26:14.936759+0000","last_peered":"2024-09-21T12:26:14.936759+0000","last_clean":"2024-09-21T12:26:14.936759+0000","last_became_active":"2024-09-21T12:20:19.308247+0000","last_became_peered":"2024-09-21T12:20:19.308247+0000","last_unstale":"2024-09-21T12:26:14.936759+0000","last_undegraded":"2024-09-21T12:26:14.936759+0000","last_fullsized":"2024-09-21T12:26:14.936759+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:32.659918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:32.659918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:46.283244+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":8204976128,"num_objects":125198,"num_object_clones":0,"num_object_copies":375594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":125198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":125198,"num_write_kb":8012672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24766906368,"data_stored":24766906368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69847,"ondisk_log_size":69847,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":690,"num_read_kb":447,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":201,"num_bytes_recovered":3982,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":396,"ondisk_log_size":396,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":5023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":361,"num_read_kb":356,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234061,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3273616,"kb_used_data":3039744,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":90471536,"statfs":{"total":95995035648,"available":92642852864,"internally_reserved":0,"allocated":3112697856,"data_stored":3112273280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":239466954},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462997,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3297224,"kb_used_data":3045120,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":90447928,"statfs":{"total":95995035648,"available":92618678272,"internally_reserved":0,"allocated":3118202880,"data_stored":3117778095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":257948102},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724632,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4228052,"kb_used_data":4066372,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":89517100,"statfs":{"total":95995035648,"available":91665510400,"internally_reserved":0,"allocated":4163964928,"data_stored":4163537690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953568,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149797,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3703636,"kb_used_data":3401732,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":90041516,"statfs":{"total":95995035648,"available":92202512384,"internally_reserved":0,"allocated":3483373568,"data_stored":3482865404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":309131719},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411436,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3785596,"kb_used_data":3551540,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":89959556,"statfs":{"total":95995035648,"available":92118585344,"internally_reserved":0,"allocated":3636776960,"data_stored":3636364563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":239466949},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574962,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3836276,"kb_used_data":3544100,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":89908876,"statfs":{"total":95995035648,"available":92066689024,"internally_reserved":0,"allocated":3629158400,"data_stored":3628697279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738487,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3670336,"kb_used_data":3418416,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":90074816,"statfs":{"total":95995035648,"available":92236611584,"internally_reserved":0,"allocated":3500457984,"data_stored":3499969170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3512860672,"data_stored":3512860672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3651469312,"data_stored":3651469312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3662741504,"data_stored":3662741504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3500998656,"data_stored":3500998656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4171169792,"data_stored":4171169792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3135242240,"data_stored":3135242240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3132424192,"data_stored":3132424192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:26:26.080 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:26:26.090 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:26.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:26.177 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:26.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:26.277 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:26.330 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:26.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:26.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:26.406 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T12:26:26.432 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:26.497 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:26.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:26.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:26.599 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:26.626 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:26.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:26.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:26.728 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:26.737 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:26.799 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:26.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:26.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:26.901 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:26.927 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:26.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:26.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:27.028 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:27.082 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:27.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:27.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:27.183 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:27.214 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:27.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:27.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:27.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:26:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:26:27.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:27 smithi183 ceph-mon[33277]: pgmap v298: 158 pgs: 158 active+clean; 7.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 46 MiB/s wr, 739 op/s 2024-09-21T12:26:27.243 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:27 smithi114 ceph-mon[40867]: pgmap v298: 158 pgs: 158 active+clean; 7.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 46 MiB/s wr, 739 op/s 2024-09-21T12:26:27.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:27 smithi114 ceph-mon[30324]: pgmap v298: 158 pgs: 158 active+clean; 7.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 46 MiB/s wr, 739 op/s 2024-09-21T12:26:27.316 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:27.340 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:27.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:27.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:27.372 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:27.442 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:27.454 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:27.498 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:27.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:27.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:27.600 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:27.625 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:27.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:27.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:27.727 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:27.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:27.765 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:27.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:27.771 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:26:27.775 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:26:27.866 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:27.904 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:27.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:27.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:28.006 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:28.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:28.045 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:28.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:28.146 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:28.177 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:28.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:28.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:28.279 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:28.314 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:28.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:28.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:28.416 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:28.452 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:28.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:28.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:28.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:28 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:26:28.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:28 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:26:28.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:28 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:26:28.554 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:28.581 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:28.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:28.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:28.675 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:26:28.683 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:28.711 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:28.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:28.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:28.813 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:28.839 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:28.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:28.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:28.940 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:28.980 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:28.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:28.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:29.081 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:29.116 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:29.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:29.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:29.219 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:29.248 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:29.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:29.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:29.350 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:29.386 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:29.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:29.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:29.488 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:29.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:29 smithi183 ceph-mon[33277]: pgmap v299: 158 pgs: 158 active+clean; 7.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.3 KiB/s rd, 61 MiB/s wr, 1.42k op/s 2024-09-21T12:26:29.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:29 smithi183 ceph-mon[33277]: from='client.25187 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:26:29.523 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:29.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:29 smithi114 ceph-mon[40867]: pgmap v299: 158 pgs: 158 active+clean; 7.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.3 KiB/s rd, 61 MiB/s wr, 1.42k op/s 2024-09-21T12:26:29.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:29 smithi114 ceph-mon[40867]: from='client.25187 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:26:29.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:29.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:29.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:29 smithi114 ceph-mon[30324]: pgmap v299: 158 pgs: 158 active+clean; 7.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.3 KiB/s rd, 61 MiB/s wr, 1.42k op/s 2024-09-21T12:26:29.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:29 smithi114 ceph-mon[30324]: from='client.25187 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:26:29.627 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:29.647 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:29.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:29.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:29.749 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:29.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:29.779 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:29.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:29.880 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:29.909 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:29.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:29.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:30.011 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:30.043 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:30.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:30.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:30.145 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:30.173 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:30.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:30.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:30.274 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:30.300 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:30.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:30.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:30.402 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:30.438 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:30.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:30.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:30.540 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:30.581 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:30.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:30.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:30.683 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:30.704 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:30.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:30.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:30.805 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:30.834 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:30.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:30.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:30.936 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:30.963 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:30.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:30.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:31.065 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:31.092 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:31.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:31.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:31.193 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:31.219 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:31.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:31.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:31.320 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:31.343 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:31.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:31.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:31.344 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:31 smithi114 ceph-mon[30324]: pgmap v300: 158 pgs: 158 active+clean; 7.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 53 MiB/s wr, 1.24k op/s 2024-09-21T12:26:31.344 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:31 smithi114 ceph-mon[40867]: pgmap v300: 158 pgs: 158 active+clean; 7.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 53 MiB/s wr, 1.24k op/s 2024-09-21T12:26:31.445 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:31.472 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:31.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:31.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:31.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:31 smithi183 ceph-mon[33277]: pgmap v300: 158 pgs: 158 active+clean; 7.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 53 MiB/s wr, 1.24k op/s 2024-09-21T12:26:31.574 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:31.610 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:31.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:31.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:31.712 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:31.739 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:31.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:31.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:31.840 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:31.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:31.877 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:31.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:31.978 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:32.007 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:32.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:32.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:32.109 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:32.145 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:32.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:32.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:32.224 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:26:32.224 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 20, 2024-09-21T12:26:32.224 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 600, 2024-09-21T12:26:32.224 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-21T12:26:32.224 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.224 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:24386 7.4 7:2a04c23a:::benchmark_data_smithi114_78432_object24385:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T12:26:32.224 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:39.999693+0000", 2024-09-21T12:26:32.224 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 291.34230439499999, 2024-09-21T12:26:32.225 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0730027350000002, 2024-09-21T12:26:32.225 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:26:32.225 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:26:32.225 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:26:32.225 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:26:32.225 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:26:32.225 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:32.225 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 24386 2024-09-21T12:26:32.225 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.225 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:26:32.225 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.225 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:26:32.225 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.999693+0000", 2024-09-21T12:26:32.225 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.225 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.226 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.226 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:26:32.226 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.999693+0000", 2024-09-21T12:26:32.226 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.226 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.226 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.226 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:26:32.226 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.999694+0000", 2024-09-21T12:26:32.226 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.418e-06 2024-09-21T12:26:32.226 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.226 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.226 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:26:32.226 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.999715+0000", 2024-09-21T12:26:32.227 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0661000000000001e-05 2024-09-21T12:26:32.227 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.227 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.227 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:26:32.227 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.999728+0000", 2024-09-21T12:26:32.227 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3458000000000001e-05 2024-09-21T12:26:32.227 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.227 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.227 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:26:32.227 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.999735+0000", 2024-09-21T12:26:32.227 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0639999999999996e-06 2024-09-21T12:26:32.227 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.227 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.227 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.228 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.999779+0000", 2024-09-21T12:26:32.228 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3435e-05 2024-09-21T12:26:32.228 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.228 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.228 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:26:32.228 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.999898+0000", 2024-09-21T12:26:32.228 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000119195 2024-09-21T12:26:32.228 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.228 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.228 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,4", 2024-09-21T12:26:32.228 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.000344+0000", 2024-09-21T12:26:32.228 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00044621400000000001 2024-09-21T12:26:32.228 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.228 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.228 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:26:32.229 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.002914+0000", 2024-09-21T12:26:32.229 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0025697770000000001 2024-09-21T12:26:32.229 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.229 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.229 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.229 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.015475+0000", 2024-09-21T12:26:32.229 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.01256142 2024-09-21T12:26:32.229 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.229 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.229 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:26:32.229 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.015487+0000", 2024-09-21T12:26:32.229 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1788e-05 2024-09-21T12:26:32.229 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.229 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.229 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.230 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.015007+0000", 2024-09-21T12:26:32.230 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99952012000000001 2024-09-21T12:26:32.230 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.230 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.230 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:26:32.230 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.015027+0000", 2024-09-21T12:26:32.230 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9877e-05 2024-09-21T12:26:32.230 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.230 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.230 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.230 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.071991+0000", 2024-09-21T12:26:32.230 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0569637000000001 2024-09-21T12:26:32.230 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.231 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.231 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:26:32.231 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.072695+0000", 2024-09-21T12:26:32.231 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00070460800000000001 2024-09-21T12:26:32.231 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.231 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:26:32.231 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.231 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.231 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.231 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:24394 7.9 7:941770ad:::benchmark_data_smithi114_78432_object24393:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T12:26:32.231 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:40.011414+0000", 2024-09-21T12:26:32.231 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 291.33058323300003, 2024-09-21T12:26:32.231 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0723253210000001, 2024-09-21T12:26:32.232 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:26:32.232 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:26:32.232 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:32.232 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:26:32.232 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:26:32.232 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:32.232 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 24394 2024-09-21T12:26:32.232 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.232 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:26:32.232 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.232 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:26:32.232 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.011414+0000", 2024-09-21T12:26:32.232 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.232 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.232 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.233 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:26:32.233 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.011414+0000", 2024-09-21T12:26:32.233 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.233 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.233 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.233 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:26:32.233 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.011415+0000", 2024-09-21T12:26:32.233 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.643e-06 2024-09-21T12:26:32.233 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.233 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.233 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:26:32.233 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.011440+0000", 2024-09-21T12:26:32.233 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.4264e-05 2024-09-21T12:26:32.233 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.234 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.234 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:26:32.234 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.011819+0000", 2024-09-21T12:26:32.234 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00037957300000000002 2024-09-21T12:26:32.234 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.234 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.234 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:26:32.234 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.011828+0000", 2024-09-21T12:26:32.234 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.2140000000000002e-06 2024-09-21T12:26:32.234 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.234 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.234 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.234 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.018835+0000", 2024-09-21T12:26:32.234 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0070068450000000003 2024-09-21T12:26:32.235 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.235 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.235 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:26:32.235 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.018846+0000", 2024-09-21T12:26:32.235 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0254e-05 2024-09-21T12:26:32.235 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.235 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.235 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.235 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.010053+0000", 2024-09-21T12:26:32.235 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99120757699999995 2024-09-21T12:26:32.235 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.235 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.236 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:26:32.236 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.010069+0000", 2024-09-21T12:26:32.236 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5917e-05 2024-09-21T12:26:32.236 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.236 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.236 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.236 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.075233+0000", 2024-09-21T12:26:32.236 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0651640630000001 2024-09-21T12:26:32.236 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.236 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.236 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:26:32.236 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.075347+0000", 2024-09-21T12:26:32.236 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000114372 2024-09-21T12:26:32.236 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.236 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.237 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:26:32.237 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.075505+0000", 2024-09-21T12:26:32.237 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015764199999999999 2024-09-21T12:26:32.237 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.237 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.237 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.237 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.079440+0000", 2024-09-21T12:26:32.237 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0039353110000000004 2024-09-21T12:26:32.237 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.237 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.237 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.237 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.079579+0000", 2024-09-21T12:26:32.237 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000138889 2024-09-21T12:26:32.237 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.238 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.238 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:26:32.238 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.082598+0000", 2024-09-21T12:26:32.238 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0030184309999999998 2024-09-21T12:26:32.238 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.238 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.238 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:26:32.238 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.082650+0000", 2024-09-21T12:26:32.238 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2404000000000002e-05 2024-09-21T12:26:32.238 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.238 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.238 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:26:32.239 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.083739+0000", 2024-09-21T12:26:32.239 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001088922 2024-09-21T12:26:32.239 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.239 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:26:32.239 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.239 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.239 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.239 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:24396 7.9 7:9f86fa7e:::benchmark_data_smithi114_78432_object24395:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:26:32.239 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:40.030139+0000", 2024-09-21T12:26:32.239 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 291.31185835999997, 2024-09-21T12:26:32.239 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0595277680000001, 2024-09-21T12:26:32.239 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:26:32.239 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:26:32.239 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:32.240 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:26:32.240 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:26:32.240 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:32.240 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 24396 2024-09-21T12:26:32.240 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.240 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:26:32.240 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.240 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:26:32.240 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.030139+0000", 2024-09-21T12:26:32.240 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.240 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.240 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.240 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:26:32.240 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.030139+0000", 2024-09-21T12:26:32.241 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.241 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.241 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.241 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:26:32.241 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.030140+0000", 2024-09-21T12:26:32.241 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.561e-06 2024-09-21T12:26:32.241 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.241 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.241 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:26:32.241 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.030167+0000", 2024-09-21T12:26:32.241 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.6979e-05 2024-09-21T12:26:32.241 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.241 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.241 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:26:32.242 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.030178+0000", 2024-09-21T12:26:32.242 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.082e-05 2024-09-21T12:26:32.242 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.242 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.242 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:26:32.242 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.030186+0000", 2024-09-21T12:26:32.242 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.0250000000000002e-06 2024-09-21T12:26:32.242 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.242 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.242 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.242 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.030262+0000", 2024-09-21T12:26:32.242 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5919000000000003e-05 2024-09-21T12:26:32.242 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.243 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.243 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:26:32.243 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.030272+0000", 2024-09-21T12:26:32.243 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0584000000000001e-05 2024-09-21T12:26:32.243 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.243 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.243 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.243 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.011135+0000", 2024-09-21T12:26:32.243 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98086274600000001 2024-09-21T12:26:32.243 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.243 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.243 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:26:32.243 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.011145+0000", 2024-09-21T12:26:32.243 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0057e-05 2024-09-21T12:26:32.244 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.244 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.244 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.244 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.079615+0000", 2024-09-21T12:26:32.244 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0684699259999999 2024-09-21T12:26:32.244 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.244 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.244 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:26:32.244 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.079896+0000", 2024-09-21T12:26:32.244 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000280566 2024-09-21T12:26:32.244 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.244 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.244 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:26:32.244 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.080059+0000", 2024-09-21T12:26:32.245 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016277500000000001 2024-09-21T12:26:32.245 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.245 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.245 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:26:32.245 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.084001+0000", 2024-09-21T12:26:32.245 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0039429360000000002 2024-09-21T12:26:32.245 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.245 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.245 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.245 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.084050+0000", 2024-09-21T12:26:32.245 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8019999999999998e-05 2024-09-21T12:26:32.245 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.245 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.246 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.246 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.088208+0000", 2024-09-21T12:26:32.246 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.004158134 2024-09-21T12:26:32.246 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.246 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.246 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:26:32.246 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.088262+0000", 2024-09-21T12:26:32.246 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4845000000000001e-05 2024-09-21T12:26:32.246 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.246 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.246 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:26:32.246 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.089666+0000", 2024-09-21T12:26:32.246 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001403875 2024-09-21T12:26:32.247 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.247 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:26:32.247 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.247 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.247 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.247 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:24398 7.4 7:2e16e6f6:::benchmark_data_smithi114_78432_object24397:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:26:32.247 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:40.031799+0000", 2024-09-21T12:26:32.247 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 291.31019762800003, 2024-09-21T12:26:32.247 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0455989959999998, 2024-09-21T12:26:32.247 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:26:32.247 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:26:32.247 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:32.247 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:26:32.247 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:26:32.248 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:32.248 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 24398 2024-09-21T12:26:32.248 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.248 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:26:32.248 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.248 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:26:32.248 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.031799+0000", 2024-09-21T12:26:32.248 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.248 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.248 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.248 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:26:32.248 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.031799+0000", 2024-09-21T12:26:32.248 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.249 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.249 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.249 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:26:32.249 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.031801+0000", 2024-09-21T12:26:32.249 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.294e-06 2024-09-21T12:26:32.249 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.249 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.249 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:26:32.249 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.031832+0000", 2024-09-21T12:26:32.249 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.1411000000000001e-05 2024-09-21T12:26:32.249 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.249 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.249 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:26:32.249 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.031844+0000", 2024-09-21T12:26:32.249 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2228999999999999e-05 2024-09-21T12:26:32.250 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.250 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.250 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:26:32.250 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.031852+0000", 2024-09-21T12:26:32.250 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8900000000000007e-06 2024-09-21T12:26:32.250 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.250 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.250 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.250 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.032037+0000", 2024-09-21T12:26:32.250 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00018500799999999999 2024-09-21T12:26:32.250 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.250 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.251 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:26:32.251 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.032045+0000", 2024-09-21T12:26:32.251 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4089999999999997e-06 2024-09-21T12:26:32.251 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.251 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.251 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.251 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.015144+0000", 2024-09-21T12:26:32.251 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98309912399999999 2024-09-21T12:26:32.251 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.251 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.251 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:26:32.251 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.015154+0000", 2024-09-21T12:26:32.251 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0530000000000001e-05 2024-09-21T12:26:32.251 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.251 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.252 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.252 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.073610+0000", 2024-09-21T12:26:32.252 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.058456026 2024-09-21T12:26:32.252 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.252 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.252 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:26:32.252 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.073735+0000", 2024-09-21T12:26:32.252 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012485299999999999 2024-09-21T12:26:32.252 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.252 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.252 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:26:32.252 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.073884+0000", 2024-09-21T12:26:32.252 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014936800000000001 2024-09-21T12:26:32.252 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.253 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.253 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:26:32.253 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.075415+0000", 2024-09-21T12:26:32.253 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0015308089999999999 2024-09-21T12:26:32.253 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.253 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.253 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.253 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.075983+0000", 2024-09-21T12:26:32.253 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00056768900000000002 2024-09-21T12:26:32.253 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.253 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.253 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.253 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.076436+0000", 2024-09-21T12:26:32.253 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00045266300000000001 2024-09-21T12:26:32.253 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.254 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.254 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:26:32.254 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.076508+0000", 2024-09-21T12:26:32.254 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2834000000000001e-05 2024-09-21T12:26:32.254 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.254 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.254 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:26:32.254 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.077398+0000", 2024-09-21T12:26:32.254 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00088985900000000001 2024-09-21T12:26:32.254 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.254 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:26:32.254 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.254 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.254 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.255 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:24402 7.9 7:9cd72310:::benchmark_data_smithi114_78432_object24401:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:26:32.255 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:40.035210+0000", 2024-09-21T12:26:32.255 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 291.30678697299999, 2024-09-21T12:26:32.255 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0693564000000002, 2024-09-21T12:26:32.255 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:26:32.255 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:26:32.255 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:32.255 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:26:32.255 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:26:32.255 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:32.255 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 24402 2024-09-21T12:26:32.255 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.255 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:26:32.255 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.255 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:26:32.256 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.035210+0000", 2024-09-21T12:26:32.256 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.256 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.256 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.256 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:26:32.256 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.035210+0000", 2024-09-21T12:26:32.256 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.256 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.256 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.256 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:26:32.256 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.035211+0000", 2024-09-21T12:26:32.256 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5899999999999996e-07 2024-09-21T12:26:32.257 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.257 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.257 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:26:32.257 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.035234+0000", 2024-09-21T12:26:32.257 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3084000000000001e-05 2024-09-21T12:26:32.257 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.257 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.257 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:26:32.257 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.035241+0000", 2024-09-21T12:26:32.257 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7170000000000008e-06 2024-09-21T12:26:32.257 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.257 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.257 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:26:32.257 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.035247+0000", 2024-09-21T12:26:32.257 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8479999999999997e-06 2024-09-21T12:26:32.258 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.258 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.258 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.258 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.035965+0000", 2024-09-21T12:26:32.258 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00071792500000000001 2024-09-21T12:26:32.258 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.258 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.258 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:26:32.258 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.035976+0000", 2024-09-21T12:26:32.258 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1241e-05 2024-09-21T12:26:32.258 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.258 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.258 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.258 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.013557+0000", 2024-09-21T12:26:32.259 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.977580169 2024-09-21T12:26:32.259 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.259 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.259 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:26:32.259 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.013568+0000", 2024-09-21T12:26:32.259 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1241e-05 2024-09-21T12:26:32.259 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.259 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.259 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.259 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.084082+0000", 2024-09-21T12:26:32.259 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.070514008 2024-09-21T12:26:32.259 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.259 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.259 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:26:32.259 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.084215+0000", 2024-09-21T12:26:32.260 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013308999999999999 2024-09-21T12:26:32.260 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.260 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.260 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:26:32.260 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.084367+0000", 2024-09-21T12:26:32.260 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000152118 2024-09-21T12:26:32.260 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.260 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.260 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.260 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.089784+0000", 2024-09-21T12:26:32.260 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0054174610000000002 2024-09-21T12:26:32.260 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.260 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.260 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.260 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.089831+0000", 2024-09-21T12:26:32.260 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6094000000000003e-05 2024-09-21T12:26:32.261 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.261 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.261 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:26:32.261 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.102915+0000", 2024-09-21T12:26:32.261 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.013084181 2024-09-21T12:26:32.261 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.261 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.261 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:26:32.261 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.103000+0000", 2024-09-21T12:26:32.261 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5103000000000002e-05 2024-09-21T12:26:32.261 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.261 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.261 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:26:32.261 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.104566+0000", 2024-09-21T12:26:32.261 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001566561 2024-09-21T12:26:32.262 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.262 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:26:32.262 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.262 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.262 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.262 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:24414 7.9 7:91740cc4:::benchmark_data_smithi114_78432_object24413:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:26:32.262 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:40.045494+0000", 2024-09-21T12:26:32.262 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 291.29650311900002, 2024-09-21T12:26:32.262 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0603511019999998, 2024-09-21T12:26:32.262 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:26:32.262 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:26:32.262 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:32.262 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:26:32.263 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:26:32.263 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:32.263 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 24414 2024-09-21T12:26:32.263 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.263 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:26:32.263 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.263 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:26:32.263 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.045494+0000", 2024-09-21T12:26:32.263 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.263 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.263 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.264 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:26:32.264 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.045494+0000", 2024-09-21T12:26:32.264 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.264 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.264 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.264 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:26:32.264 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.045496+0000", 2024-09-21T12:26:32.264 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.181e-06 2024-09-21T12:26:32.264 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.264 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.264 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:26:32.264 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.045521+0000", 2024-09-21T12:26:32.264 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5400000000000001e-05 2024-09-21T12:26:32.264 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.265 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.265 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:26:32.265 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.045540+0000", 2024-09-21T12:26:32.265 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.838e-05 2024-09-21T12:26:32.265 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.265 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.265 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:26:32.265 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.045549+0000", 2024-09-21T12:26:32.265 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1889999999999996e-06 2024-09-21T12:26:32.265 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.265 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.265 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.266 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.045619+0000", 2024-09-21T12:26:32.266 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0378999999999998e-05 2024-09-21T12:26:32.266 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.266 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.266 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:26:32.266 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.045630+0000", 2024-09-21T12:26:32.266 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0474e-05 2024-09-21T12:26:32.267 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.267 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.267 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.267 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.013596+0000", 2024-09-21T12:26:32.267 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.96796572599999997 2024-09-21T12:26:32.267 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.267 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.267 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:26:32.267 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.013604+0000", 2024-09-21T12:26:32.267 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9549999999999992e-06 2024-09-21T12:26:32.267 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.267 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.268 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.268 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.085579+0000", 2024-09-21T12:26:32.268 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.071975613 2024-09-21T12:26:32.268 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.268 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.268 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:26:32.268 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.085734+0000", 2024-09-21T12:26:32.268 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015513199999999999 2024-09-21T12:26:32.268 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.268 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.268 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:26:32.268 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.085912+0000", 2024-09-21T12:26:32.269 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017730999999999999 2024-09-21T12:26:32.269 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.269 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.269 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.269 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.094582+0000", 2024-09-21T12:26:32.269 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0086700010000000001 2024-09-21T12:26:32.269 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.269 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.269 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.269 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.094647+0000", 2024-09-21T12:26:32.269 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5315999999999996e-05 2024-09-21T12:26:32.269 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.269 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.270 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:26:32.270 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.104634+0000", 2024-09-21T12:26:32.270 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0099874270000000001 2024-09-21T12:26:32.270 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.270 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.270 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:26:32.270 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.104728+0000", 2024-09-21T12:26:32.270 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.3770000000000005e-05 2024-09-21T12:26:32.270 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.270 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.270 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:26:32.270 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.105845+0000", 2024-09-21T12:26:32.270 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0011168490000000001 2024-09-21T12:26:32.271 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.271 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:26:32.271 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.271 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.271 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.271 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46116 7.4 7:24b34938:::benchmark_data_smithi114_78432_object46115:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-21T12:26:32.271 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:59.859491+0000", 2024-09-21T12:26:32.271 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 271.48250581299999, 2024-09-21T12:26:32.271 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0102598979999999, 2024-09-21T12:26:32.272 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:26:32.272 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:26:32.272 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:32.272 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:26:32.272 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:26:32.272 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:32.272 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46116 2024-09-21T12:26:32.272 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.272 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:26:32.272 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.272 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:26:32.273 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.859491+0000", 2024-09-21T12:26:32.273 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.273 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.273 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.273 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:26:32.273 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.859491+0000", 2024-09-21T12:26:32.273 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.273 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.273 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.273 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:26:32.273 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.859493+0000", 2024-09-21T12:26:32.273 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.514e-06 2024-09-21T12:26:32.273 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.273 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.273 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:26:32.274 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.859515+0000", 2024-09-21T12:26:32.274 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1984000000000002e-05 2024-09-21T12:26:32.274 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.274 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.274 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:26:32.274 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.859527+0000", 2024-09-21T12:26:32.274 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2765000000000001e-05 2024-09-21T12:26:32.274 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.274 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.274 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:26:32.274 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.859538+0000", 2024-09-21T12:26:32.274 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0112000000000001e-05 2024-09-21T12:26:32.274 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.275 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.275 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.275 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.860240+0000", 2024-09-21T12:26:32.275 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00070220700000000005 2024-09-21T12:26:32.275 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.275 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.275 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:26:32.275 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.860248+0000", 2024-09-21T12:26:32.275 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8399999999999995e-06 2024-09-21T12:26:32.275 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.275 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.275 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.275 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.864080+0000", 2024-09-21T12:26:32.275 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.003832177 2024-09-21T12:26:32.276 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.276 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.276 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:26:32.276 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.864211+0000", 2024-09-21T12:26:32.276 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013077499999999999 2024-09-21T12:26:32.276 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.276 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.276 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:26:32.276 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.864382+0000", 2024-09-21T12:26:32.276 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000171736 2024-09-21T12:26:32.276 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.276 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.277 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:26:32.277 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.869406+0000", 2024-09-21T12:26:32.277 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0050239079999999997 2024-09-21T12:26:32.277 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.277 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.277 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.277 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.869604+0000", 2024-09-21T12:26:32.277 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00019803600000000001 2024-09-21T12:26:32.277 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.277 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.277 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.277 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.869659+0000", 2024-09-21T12:26:32.277 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5278999999999997e-05 2024-09-21T12:26:32.277 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.278 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.278 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:26:32.278 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.869716+0000", 2024-09-21T12:26:32.278 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6634e-05 2024-09-21T12:26:32.278 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.278 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.278 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:26:32.278 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.869751+0000", 2024-09-21T12:26:32.278 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4931000000000001e-05 2024-09-21T12:26:32.278 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.278 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:26:32.278 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.278 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.278 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.279 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46122 7.4 7:2258f125:::benchmark_data_smithi114_78432_object46121:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-21T12:26:32.279 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:59.865240+0000", 2024-09-21T12:26:32.279 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 271.47675704599999, 2024-09-21T12:26:32.279 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0066945679999999, 2024-09-21T12:26:32.279 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:26:32.279 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:26:32.279 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:32.279 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:26:32.279 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:26:32.279 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:32.279 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46122 2024-09-21T12:26:32.279 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.279 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:26:32.279 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.279 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:26:32.280 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.865240+0000", 2024-09-21T12:26:32.280 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.280 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.280 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.280 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:26:32.280 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.865240+0000", 2024-09-21T12:26:32.280 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.280 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.280 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.280 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:26:32.280 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.865241+0000", 2024-09-21T12:26:32.280 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.02e-07 2024-09-21T12:26:32.280 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.281 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.281 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:26:32.281 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.865262+0000", 2024-09-21T12:26:32.281 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1659e-05 2024-09-21T12:26:32.281 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.281 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.281 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:26:32.281 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.865273+0000", 2024-09-21T12:26:32.281 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0169e-05 2024-09-21T12:26:32.281 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.281 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.282 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:26:32.282 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.865279+0000", 2024-09-21T12:26:32.282 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4420000000000003e-06 2024-09-21T12:26:32.282 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.282 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.282 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.282 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.865336+0000", 2024-09-21T12:26:32.282 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6609e-05 2024-09-21T12:26:32.282 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.282 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.282 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:26:32.282 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.865345+0000", 2024-09-21T12:26:32.282 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.8729999999999996e-06 2024-09-21T12:26:32.283 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.283 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.283 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.283 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.866291+0000", 2024-09-21T12:26:32.283 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.000946307 2024-09-21T12:26:32.283 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.283 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.283 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:26:32.283 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.866418+0000", 2024-09-21T12:26:32.283 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012749500000000001 2024-09-21T12:26:32.283 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.283 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.283 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:26:32.283 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.866527+0000", 2024-09-21T12:26:32.284 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010840699999999999 2024-09-21T12:26:32.284 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.284 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.284 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.284 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.869790+0000", 2024-09-21T12:26:32.284 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0032636050000000002 2024-09-21T12:26:32.284 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.284 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.284 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.284 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.869825+0000", 2024-09-21T12:26:32.284 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4363999999999998e-05 2024-09-21T12:26:32.284 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.284 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.284 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:26:32.284 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.871758+0000", 2024-09-21T12:26:32.285 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0019335019999999999 2024-09-21T12:26:32.285 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.285 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.285 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:26:32.285 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.871815+0000", 2024-09-21T12:26:32.285 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6474000000000002e-05 2024-09-21T12:26:32.285 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.285 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.285 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:26:32.285 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.871934+0000", 2024-09-21T12:26:32.285 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011976000000000001 2024-09-21T12:26:32.285 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.285 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:26:32.285 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.285 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.286 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.286 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46139 7.4 7:280fcf10:::benchmark_data_smithi114_78432_object46138:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-21T12:26:32.286 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:59.880524+0000", 2024-09-21T12:26:32.286 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 271.46147324399999, 2024-09-21T12:26:32.286 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99498322900000002, 2024-09-21T12:26:32.286 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:26:32.286 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:26:32.286 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:32.286 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:26:32.286 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:26:32.286 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:32.286 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46139 2024-09-21T12:26:32.287 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.287 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:26:32.287 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.287 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:26:32.287 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.880524+0000", 2024-09-21T12:26:32.287 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.287 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.287 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.287 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:26:32.287 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.880524+0000", 2024-09-21T12:26:32.287 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.288 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.288 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.288 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:26:32.288 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.880525+0000", 2024-09-21T12:26:32.288 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.279e-06 2024-09-21T12:26:32.288 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.288 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.288 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:26:32.288 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.880544+0000", 2024-09-21T12:26:32.288 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8630999999999998e-05 2024-09-21T12:26:32.288 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.289 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.289 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:26:32.289 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.880555+0000", 2024-09-21T12:26:32.289 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1569999999999999e-05 2024-09-21T12:26:32.289 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.289 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.289 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:26:32.289 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.880561+0000", 2024-09-21T12:26:32.289 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4029999999999998e-06 2024-09-21T12:26:32.289 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.289 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.289 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.290 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.880612+0000", 2024-09-21T12:26:32.290 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1508999999999998e-05 2024-09-21T12:26:32.290 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.290 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.290 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:26:32.290 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.880619+0000", 2024-09-21T12:26:32.290 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9410000000000001e-06 2024-09-21T12:26:32.290 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.290 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.290 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.290 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.867972+0000", 2024-09-21T12:26:32.291 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98735288099999996 2024-09-21T12:26:32.291 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.291 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.291 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:26:32.291 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.868095+0000", 2024-09-21T12:26:32.291 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012281099999999999 2024-09-21T12:26:32.291 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.291 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.291 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:26:32.291 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.868195+0000", 2024-09-21T12:26:32.292 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000100257 2024-09-21T12:26:32.292 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.292 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.292 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.292 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.869898+0000", 2024-09-21T12:26:32.292 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001702641 2024-09-21T12:26:32.292 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.292 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.292 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.292 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.871992+0000", 2024-09-21T12:26:32.292 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002094392 2024-09-21T12:26:32.293 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.293 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.293 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:26:32.293 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.875422+0000", 2024-09-21T12:26:32.293 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00342983 2024-09-21T12:26:32.293 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.293 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.293 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:26:32.293 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.875468+0000", 2024-09-21T12:26:32.293 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5661999999999998e-05 2024-09-21T12:26:32.293 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.293 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.293 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:26:32.294 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.875507+0000", 2024-09-21T12:26:32.294 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.9422000000000003e-05 2024-09-21T12:26:32.294 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.294 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:26:32.294 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.294 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.294 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.294 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46144 7.4 7:25363ab6:::benchmark_data_smithi114_78432_object46143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-21T12:26:32.294 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:59.884835+0000", 2024-09-21T12:26:32.294 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 271.45716192100002, 2024-09-21T12:26:32.294 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99791898099999998, 2024-09-21T12:26:32.294 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:26:32.294 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:26:32.295 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:32.295 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:26:32.295 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:26:32.295 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:32.295 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46144 2024-09-21T12:26:32.295 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.295 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:26:32.295 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.295 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:26:32.295 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.884835+0000", 2024-09-21T12:26:32.295 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.295 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.295 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.295 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:26:32.296 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.884835+0000", 2024-09-21T12:26:32.296 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.296 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.296 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.296 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:26:32.296 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.884837+0000", 2024-09-21T12:26:32.296 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.812e-06 2024-09-21T12:26:32.296 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.296 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.296 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:26:32.296 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.884861+0000", 2024-09-21T12:26:32.296 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3887e-05 2024-09-21T12:26:32.297 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.297 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.297 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:26:32.297 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.884874+0000", 2024-09-21T12:26:32.297 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3655000000000001e-05 2024-09-21T12:26:32.297 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.297 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.297 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:26:32.297 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.884884+0000", 2024-09-21T12:26:32.297 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0064999999999999e-05 2024-09-21T12:26:32.297 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.297 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.297 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.297 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.884942+0000", 2024-09-21T12:26:32.297 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7698000000000002e-05 2024-09-21T12:26:32.298 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.298 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.298 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:26:32.298 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.884954+0000", 2024-09-21T12:26:32.298 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1501999999999999e-05 2024-09-21T12:26:32.298 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.298 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.298 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.298 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.869934+0000", 2024-09-21T12:26:32.298 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98498016499999996 2024-09-21T12:26:32.298 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.298 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.299 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:26:32.299 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.870058+0000", 2024-09-21T12:26:32.299 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000124163 2024-09-21T12:26:32.299 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.299 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.299 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:26:32.299 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.870188+0000", 2024-09-21T12:26:32.299 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013018700000000001 2024-09-21T12:26:32.299 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.299 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.299 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.299 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.877343+0000", 2024-09-21T12:26:32.299 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0071551590000000003 2024-09-21T12:26:32.299 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.300 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.300 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.300 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.877377+0000", 2024-09-21T12:26:32.300 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.3600999999999999e-05 2024-09-21T12:26:32.300 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.300 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.300 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:26:32.300 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.882624+0000", 2024-09-21T12:26:32.300 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0052467089999999996 2024-09-21T12:26:32.300 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.300 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.300 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:26:32.300 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.882701+0000", 2024-09-21T12:26:32.300 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7439000000000002e-05 2024-09-21T12:26:32.301 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.301 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.301 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:26:32.301 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.882754+0000", 2024-09-21T12:26:32.301 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2939000000000003e-05 2024-09-21T12:26:32.301 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.301 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:26:32.301 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.301 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.301 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.301 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46156 7.4 7:2867ce09:::benchmark_data_smithi114_78432_object46155:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-21T12:26:32.301 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:59.892209+0000", 2024-09-21T12:26:32.301 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 271.44978768599998, 2024-09-21T12:26:32.301 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99488361400000003, 2024-09-21T12:26:32.302 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:26:32.302 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:26:32.302 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:32.302 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:26:32.302 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:26:32.302 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:32.302 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46156 2024-09-21T12:26:32.302 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.302 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:26:32.302 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.303 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:26:32.303 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.892209+0000", 2024-09-21T12:26:32.303 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.303 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.303 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.303 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:26:32.303 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.892209+0000", 2024-09-21T12:26:32.303 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.303 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.303 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.303 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:26:32.303 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.892211+0000", 2024-09-21T12:26:32.303 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.536e-06 2024-09-21T12:26:32.303 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.304 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.304 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:26:32.304 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.892237+0000", 2024-09-21T12:26:32.304 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.6188999999999999e-05 2024-09-21T12:26:32.304 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.304 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.304 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:26:32.304 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.892249+0000", 2024-09-21T12:26:32.304 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2086000000000001e-05 2024-09-21T12:26:32.304 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.304 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.304 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:26:32.304 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.892255+0000", 2024-09-21T12:26:32.304 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9739999999999999e-06 2024-09-21T12:26:32.305 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.305 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.305 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.305 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.892320+0000", 2024-09-21T12:26:32.305 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4566999999999994e-05 2024-09-21T12:26:32.305 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.305 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.305 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:26:32.305 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.892333+0000", 2024-09-21T12:26:32.305 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2872e-05 2024-09-21T12:26:32.305 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.305 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.305 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.305 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.872042+0000", 2024-09-21T12:26:32.305 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97970955100000001 2024-09-21T12:26:32.306 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.306 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.306 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:26:32.306 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.872150+0000", 2024-09-21T12:26:32.306 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010774 2024-09-21T12:26:32.306 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.306 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.306 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:26:32.306 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.872281+0000", 2024-09-21T12:26:32.306 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000131054 2024-09-21T12:26:32.306 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.306 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.306 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.306 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.877407+0000", 2024-09-21T12:26:32.307 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0051259410000000002 2024-09-21T12:26:32.307 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.307 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.307 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.307 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.877436+0000", 2024-09-21T12:26:32.307 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.9337999999999999e-05 2024-09-21T12:26:32.307 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.307 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.307 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:26:32.307 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.886994+0000", 2024-09-21T12:26:32.307 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0095576859999999993 2024-09-21T12:26:32.307 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.307 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.307 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:26:32.308 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.887050+0000", 2024-09-21T12:26:32.308 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6509999999999999e-05 2024-09-21T12:26:32.308 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.308 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.308 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:26:32.308 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.887093+0000", 2024-09-21T12:26:32.308 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.2570000000000001e-05 2024-09-21T12:26:32.308 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.308 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:26:32.308 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.308 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.308 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.309 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46177 7.4 7:263d7178:::benchmark_data_smithi114_78432_object46176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-21T12:26:32.309 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:59.905947+0000", 2024-09-21T12:26:32.309 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 271.436049948, 2024-09-21T12:26:32.309 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98122774999999995, 2024-09-21T12:26:32.309 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:26:32.309 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:26:32.309 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:32.309 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:26:32.309 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:26:32.309 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:32.309 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46177 2024-09-21T12:26:32.309 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.309 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:26:32.309 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.310 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:26:32.310 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.905947+0000", 2024-09-21T12:26:32.310 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.310 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.310 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.310 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:26:32.310 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.905947+0000", 2024-09-21T12:26:32.310 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.310 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.310 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.310 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:26:32.310 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.905948+0000", 2024-09-21T12:26:32.310 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.13e-06 2024-09-21T12:26:32.310 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.311 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.311 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:26:32.311 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.905969+0000", 2024-09-21T12:26:32.311 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1195e-05 2024-09-21T12:26:32.311 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.311 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.311 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:26:32.311 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.906067+0000", 2024-09-21T12:26:32.311 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7285000000000001e-05 2024-09-21T12:26:32.311 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.311 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.311 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:26:32.311 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.906101+0000", 2024-09-21T12:26:32.311 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4461000000000001e-05 2024-09-21T12:26:32.311 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.312 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.312 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.312 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.906199+0000", 2024-09-21T12:26:32.312 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7796999999999999e-05 2024-09-21T12:26:32.312 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.312 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.312 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:26:32.312 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.906209+0000", 2024-09-21T12:26:32.312 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.9990000000000003e-06 2024-09-21T12:26:32.312 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.312 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.312 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.313 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.873752+0000", 2024-09-21T12:26:32.313 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.96754281399999997 2024-09-21T12:26:32.313 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.313 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.313 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:26:32.313 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.873868+0000", 2024-09-21T12:26:32.313 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011646699999999999 2024-09-21T12:26:32.313 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.313 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.313 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:26:32.313 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.873997+0000", 2024-09-21T12:26:32.313 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000129009 2024-09-21T12:26:32.314 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.314 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.314 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.314 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.877465+0000", 2024-09-21T12:26:32.314 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0034678439999999999 2024-09-21T12:26:32.314 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.314 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.314 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.314 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.879197+0000", 2024-09-21T12:26:32.314 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0017320199999999999 2024-09-21T12:26:32.314 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.314 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.314 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:26:32.315 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.887100+0000", 2024-09-21T12:26:32.315 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0079033609999999994 2024-09-21T12:26:32.315 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.315 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.315 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:26:32.315 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.887142+0000", 2024-09-21T12:26:32.315 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1428999999999999e-05 2024-09-21T12:26:32.315 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.315 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.315 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:26:32.315 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.887175+0000", 2024-09-21T12:26:32.315 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2938999999999998e-05 2024-09-21T12:26:32.315 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.315 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:26:32.315 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.316 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.316 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.316 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46178 7.4 7:2a0dc45a:::benchmark_data_smithi114_78432_object46177:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-21T12:26:32.316 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:59.906137+0000", 2024-09-21T12:26:32.316 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 271.43586026700001, 2024-09-21T12:26:32.316 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98111571099999995, 2024-09-21T12:26:32.316 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:26:32.316 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:26:32.316 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:32.316 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:26:32.316 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:26:32.316 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:32.316 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46178 2024-09-21T12:26:32.317 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.317 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:26:32.317 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.317 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:26:32.317 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.906137+0000", 2024-09-21T12:26:32.317 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.317 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.317 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.317 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:26:32.317 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.906137+0000", 2024-09-21T12:26:32.317 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.317 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.317 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.318 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:26:32.318 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.906137+0000", 2024-09-21T12:26:32.318 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7599999999999996e-07 2024-09-21T12:26:32.318 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.318 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.318 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:26:32.318 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.906170+0000", 2024-09-21T12:26:32.318 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2963999999999998e-05 2024-09-21T12:26:32.318 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.318 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.319 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:26:32.319 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.906179+0000", 2024-09-21T12:26:32.319 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.4700000000000002e-06 2024-09-21T12:26:32.319 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.319 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.319 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:26:32.319 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.906185+0000", 2024-09-21T12:26:32.319 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9689999999999999e-06 2024-09-21T12:26:32.319 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.319 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.319 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.319 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.906232+0000", 2024-09-21T12:26:32.319 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7567e-05 2024-09-21T12:26:32.319 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.320 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.320 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:26:32.320 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.906239+0000", 2024-09-21T12:26:32.320 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5699999999999998e-06 2024-09-21T12:26:32.320 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.320 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.320 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.320 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.875536+0000", 2024-09-21T12:26:32.321 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.96929714899999997 2024-09-21T12:26:32.321 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.321 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.321 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:26:32.321 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.875639+0000", 2024-09-21T12:26:32.321 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000103107 2024-09-21T12:26:32.321 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.321 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.322 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:26:32.322 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.875758+0000", 2024-09-21T12:26:32.322 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000119191 2024-09-21T12:26:32.322 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.322 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.322 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.322 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.880946+0000", 2024-09-21T12:26:32.322 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0051874709999999999 2024-09-21T12:26:32.322 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.322 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.322 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.323 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.882821+0000", 2024-09-21T12:26:32.323 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001875137 2024-09-21T12:26:32.323 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.323 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.323 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:26:32.323 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.887181+0000", 2024-09-21T12:26:32.323 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0043603660000000001 2024-09-21T12:26:32.323 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.323 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.323 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:26:32.324 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.887220+0000", 2024-09-21T12:26:32.324 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8894999999999999e-05 2024-09-21T12:26:32.324 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.324 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.324 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:26:32.324 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.887252+0000", 2024-09-21T12:26:32.324 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2079000000000002e-05 2024-09-21T12:26:32.324 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.324 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:26:32.324 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.324 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.324 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.324 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46199 7.4 7:23cabb67:::benchmark_data_smithi114_78432_object46198:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-21T12:26:32.325 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:59.924374+0000", 2024-09-21T12:26:32.325 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 271.41762255899999, 2024-09-21T12:26:32.325 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.96295298900000004, 2024-09-21T12:26:32.325 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:26:32.325 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:26:32.325 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:32.325 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:26:32.325 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:26:32.325 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:32.325 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46199 2024-09-21T12:26:32.325 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.325 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:26:32.325 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.325 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:26:32.325 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.924374+0000", 2024-09-21T12:26:32.326 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.326 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.326 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.326 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:26:32.326 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.924374+0000", 2024-09-21T12:26:32.326 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.326 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.326 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.326 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:26:32.326 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.924376+0000", 2024-09-21T12:26:32.326 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5969999999999999e-06 2024-09-21T12:26:32.326 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.326 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.326 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:26:32.327 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.924488+0000", 2024-09-21T12:26:32.327 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000112175 2024-09-21T12:26:32.327 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.327 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.327 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:26:32.327 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.924501+0000", 2024-09-21T12:26:32.327 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2916e-05 2024-09-21T12:26:32.327 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.327 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.327 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:26:32.327 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.924509+0000", 2024-09-21T12:26:32.327 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8199999999999997e-06 2024-09-21T12:26:32.327 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.327 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.328 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.328 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.924587+0000", 2024-09-21T12:26:32.328 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7861000000000006e-05 2024-09-21T12:26:32.328 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.328 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.328 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:26:32.328 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.924597+0000", 2024-09-21T12:26:32.328 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0448999999999999e-05 2024-09-21T12:26:32.328 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.328 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.328 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.328 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.877490+0000", 2024-09-21T12:26:32.329 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.95289245 2024-09-21T12:26:32.329 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.329 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.329 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:26:32.329 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.877595+0000", 2024-09-21T12:26:32.329 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000105764 2024-09-21T12:26:32.329 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.329 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.329 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:26:32.329 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.877724+0000", 2024-09-21T12:26:32.329 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012821100000000001 2024-09-21T12:26:32.329 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.329 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.330 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.330 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.880984+0000", 2024-09-21T12:26:32.330 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0032604610000000001 2024-09-21T12:26:32.330 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.330 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.330 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.330 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.882869+0000", 2024-09-21T12:26:32.330 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001884978 2024-09-21T12:26:32.330 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.330 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.330 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:26:32.330 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.887259+0000", 2024-09-21T12:26:32.330 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0043894160000000002 2024-09-21T12:26:32.330 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.331 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.331 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:26:32.331 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.887298+0000", 2024-09-21T12:26:32.331 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.9962999999999997e-05 2024-09-21T12:26:32.331 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.331 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.331 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:26:32.331 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.887327+0000", 2024-09-21T12:26:32.331 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.8928000000000002e-05 2024-09-21T12:26:32.331 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.331 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:26:32.331 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.332 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.332 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.332 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46208 7.4 7:28d25463:::benchmark_data_smithi114_78432_object46207:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-21T12:26:32.332 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:59.929005+0000", 2024-09-21T12:26:32.332 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 271.41299185100002, 2024-09-21T12:26:32.332 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.95853571400000004, 2024-09-21T12:26:32.332 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:26:32.332 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:26:32.332 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:32.332 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:26:32.332 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:26:32.332 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:32.332 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46208 2024-09-21T12:26:32.332 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.333 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:26:32.333 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.333 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:26:32.333 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.929005+0000", 2024-09-21T12:26:32.333 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.333 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.333 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.333 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:26:32.333 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.929005+0000", 2024-09-21T12:26:32.333 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.333 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.334 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.334 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:26:32.334 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.929007+0000", 2024-09-21T12:26:32.334 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9889999999999999e-06 2024-09-21T12:26:32.334 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.334 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.334 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:26:32.335 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.929039+0000", 2024-09-21T12:26:32.335 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.1786999999999999e-05 2024-09-21T12:26:32.335 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.335 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.335 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:26:32.335 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.929053+0000", 2024-09-21T12:26:32.335 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4139e-05 2024-09-21T12:26:32.335 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.335 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.336 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:26:32.336 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.929061+0000", 2024-09-21T12:26:32.336 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6290000000000001e-06 2024-09-21T12:26:32.336 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.336 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.336 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.336 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.929124+0000", 2024-09-21T12:26:32.336 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3482e-05 2024-09-21T12:26:32.336 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.336 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.336 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:26:32.336 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.929133+0000", 2024-09-21T12:26:32.337 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5690000000000005e-06 2024-09-21T12:26:32.337 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.337 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.337 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.337 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.879232+0000", 2024-09-21T12:26:32.337 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.950098989 2024-09-21T12:26:32.337 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.337 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.337 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:26:32.337 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.879347+0000", 2024-09-21T12:26:32.337 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011483 2024-09-21T12:26:32.337 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.337 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.337 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:26:32.338 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.879497+0000", 2024-09-21T12:26:32.338 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015045299999999999 2024-09-21T12:26:32.338 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.338 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.338 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.338 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.882907+0000", 2024-09-21T12:26:32.338 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0034099249999999998 2024-09-21T12:26:32.338 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.338 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.338 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:26:32.338 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.887333+0000", 2024-09-21T12:26:32.338 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0044262470000000003 2024-09-21T12:26:32.338 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.338 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.339 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.339 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.887369+0000", 2024-09-21T12:26:32.339 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.6232000000000001e-05 2024-09-21T12:26:32.339 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.339 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.339 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:26:32.339 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.887415+0000", 2024-09-21T12:26:32.339 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5636000000000001e-05 2024-09-21T12:26:32.339 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.339 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.339 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:26:32.339 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.887541+0000", 2024-09-21T12:26:32.339 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000125807 2024-09-21T12:26:32.339 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.339 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:26:32.340 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.340 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.340 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.340 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46225 7.4 7:21215515:::benchmark_data_smithi114_78432_object46224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-21T12:26:32.340 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:59.947393+0000", 2024-09-21T12:26:32.340 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 271.39460431700002, 2024-09-21T12:26:32.340 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.94032970199999999, 2024-09-21T12:26:32.340 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:26:32.340 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:26:32.340 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:32.340 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:26:32.340 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:26:32.340 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:32.341 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46225 2024-09-21T12:26:32.341 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.341 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:26:32.341 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.341 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:26:32.341 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.947393+0000", 2024-09-21T12:26:32.341 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.341 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.341 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.341 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:26:32.341 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.947393+0000", 2024-09-21T12:26:32.341 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.341 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.341 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.341 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:26:32.342 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.947395+0000", 2024-09-21T12:26:32.342 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.886e-06 2024-09-21T12:26:32.342 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.342 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.342 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:26:32.342 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.947415+0000", 2024-09-21T12:26:32.342 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0562999999999999e-05 2024-09-21T12:26:32.342 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.342 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.342 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:26:32.342 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.947428+0000", 2024-09-21T12:26:32.342 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3327e-05 2024-09-21T12:26:32.342 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.343 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.343 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:26:32.343 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.947436+0000", 2024-09-21T12:26:32.343 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1250000000000004e-06 2024-09-21T12:26:32.343 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.343 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.343 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.343 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.947510+0000", 2024-09-21T12:26:32.343 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4422e-05 2024-09-21T12:26:32.343 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.343 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.343 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:26:32.343 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.947520+0000", 2024-09-21T12:26:32.343 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.9040000000000004e-06 2024-09-21T12:26:32.344 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.344 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.344 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.344 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.881016+0000", 2024-09-21T12:26:32.344 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.93349622499999996 2024-09-21T12:26:32.344 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.344 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.344 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:26:32.344 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.881126+0000", 2024-09-21T12:26:32.344 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000109832 2024-09-21T12:26:32.344 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.344 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.344 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:26:32.344 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.881258+0000", 2024-09-21T12:26:32.344 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013171299999999999 2024-09-21T12:26:32.345 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.345 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.345 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:26:32.345 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.887550+0000", 2024-09-21T12:26:32.345 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0062928369999999999 2024-09-21T12:26:32.345 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.345 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.345 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.345 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.887597+0000", 2024-09-21T12:26:32.345 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6626999999999999e-05 2024-09-21T12:26:32.345 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.345 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.346 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.346 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.887635+0000", 2024-09-21T12:26:32.346 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.7540999999999999e-05 2024-09-21T12:26:32.346 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.346 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.346 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:26:32.346 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.887687+0000", 2024-09-21T12:26:32.347 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2564e-05 2024-09-21T12:26:32.347 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.347 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.347 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:26:32.347 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.887722+0000", 2024-09-21T12:26:32.347 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5135999999999997e-05 2024-09-21T12:26:32.347 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.348 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:26:32.348 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.348 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.348 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.348 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46232 7.4 7:28020f30:::benchmark_data_smithi114_78432_object46231:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-21T12:26:32.348 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:59.953499+0000", 2024-09-21T12:26:32.349 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 271.38849846099998, 2024-09-21T12:26:32.349 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.93437983300000005, 2024-09-21T12:26:32.349 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:26:32.349 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:26:32.349 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:32.349 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:26:32.350 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:26:32.350 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:32.350 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46232 2024-09-21T12:26:32.350 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.350 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:26:32.350 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.350 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:26:32.351 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.953499+0000", 2024-09-21T12:26:32.351 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.351 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.351 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.351 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:26:32.351 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.953499+0000", 2024-09-21T12:26:32.351 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.352 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.352 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.352 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:26:32.352 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.953500+0000", 2024-09-21T12:26:32.352 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.875e-06 2024-09-21T12:26:32.352 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.352 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.353 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:26:32.353 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.953523+0000", 2024-09-21T12:26:32.353 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2158999999999999e-05 2024-09-21T12:26:32.353 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.353 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.353 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:26:32.353 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.953536+0000", 2024-09-21T12:26:32.354 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2969000000000001e-05 2024-09-21T12:26:32.354 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.354 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.354 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:26:32.354 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.953542+0000", 2024-09-21T12:26:32.354 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9989999999999997e-06 2024-09-21T12:26:32.354 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.355 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.355 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.355 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.953605+0000", 2024-09-21T12:26:32.355 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3720000000000007e-05 2024-09-21T12:26:32.355 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.355 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.355 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:26:32.355 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.953616+0000", 2024-09-21T12:26:32.356 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0626e-05 2024-09-21T12:26:32.356 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.356 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.356 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.356 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.882945+0000", 2024-09-21T12:26:32.356 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.929328712 2024-09-21T12:26:32.356 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.356 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.357 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:26:32.357 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.883080+0000", 2024-09-21T12:26:32.357 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013578 2024-09-21T12:26:32.357 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.357 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.357 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:26:32.357 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.883209+0000", 2024-09-21T12:26:32.358 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000129097 2024-09-21T12:26:32.358 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.358 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.358 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:26:32.358 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.887555+0000", 2024-09-21T12:26:32.358 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0043456780000000004 2024-09-21T12:26:32.359 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.359 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.359 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.359 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.887763+0000", 2024-09-21T12:26:32.359 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00020833399999999999 2024-09-21T12:26:32.359 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.360 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.360 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.360 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.887800+0000", 2024-09-21T12:26:32.360 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.6291000000000002e-05 2024-09-21T12:26:32.360 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.360 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.361 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:26:32.361 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.887845+0000", 2024-09-21T12:26:32.361 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5621999999999998e-05 2024-09-21T12:26:32.361 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.361 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.361 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:26:32.361 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.887878+0000", 2024-09-21T12:26:32.362 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2971e-05 2024-09-21T12:26:32.362 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.362 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:26:32.362 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.362 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.362 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.362 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46238 7.4 7:2e538fbd:::benchmark_data_smithi114_78432_object46237:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-21T12:26:32.362 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:59.964558+0000", 2024-09-21T12:26:32.362 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 271.37743887900001, 2024-09-21T12:26:32.363 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.92920913699999996, 2024-09-21T12:26:32.363 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:26:32.363 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:26:32.363 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:32.363 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:26:32.363 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:26:32.363 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:32.363 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46238 2024-09-21T12:26:32.363 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.364 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:26:32.364 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.364 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:26:32.364 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.964558+0000", 2024-09-21T12:26:32.364 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.364 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.364 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.364 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:26:32.364 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.964558+0000", 2024-09-21T12:26:32.365 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.365 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.365 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.365 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:26:32.365 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.964560+0000", 2024-09-21T12:26:32.365 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9700000000000002e-06 2024-09-21T12:26:32.365 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.365 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.366 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:26:32.366 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.964585+0000", 2024-09-21T12:26:32.366 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5029999999999999e-05 2024-09-21T12:26:32.366 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.366 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.366 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:26:32.367 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.964602+0000", 2024-09-21T12:26:32.367 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6756999999999999e-05 2024-09-21T12:26:32.367 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.367 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.367 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:26:32.368 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.964610+0000", 2024-09-21T12:26:32.368 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.96e-06 2024-09-21T12:26:32.368 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.368 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.368 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.368 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.964683+0000", 2024-09-21T12:26:32.368 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3359e-05 2024-09-21T12:26:32.368 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.368 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.368 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:26:32.368 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.964696+0000", 2024-09-21T12:26:32.368 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2831e-05 2024-09-21T12:26:32.368 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.369 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.369 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.369 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.885181+0000", 2024-09-21T12:26:32.369 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.92048453500000005 2024-09-21T12:26:32.369 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.369 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.369 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:26:32.369 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.885305+0000", 2024-09-21T12:26:32.370 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012489700000000001 2024-09-21T12:26:32.370 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.370 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.370 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-21T12:26:32.370 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.885447+0000", 2024-09-21T12:26:32.370 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014179700000000001 2024-09-21T12:26:32.370 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.370 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.371 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:26:32.371 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.887559+0000", 2024-09-21T12:26:32.371 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0021120710000000001 2024-09-21T12:26:32.371 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.371 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.371 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.371 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.887914+0000", 2024-09-21T12:26:32.371 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00035519700000000002 2024-09-21T12:26:32.371 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.371 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.371 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.371 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.893643+0000", 2024-09-21T12:26:32.371 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0057281770000000001 2024-09-21T12:26:32.372 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.372 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.372 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:26:32.372 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.893697+0000", 2024-09-21T12:26:32.372 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4561999999999997e-05 2024-09-21T12:26:32.372 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.372 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.372 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:26:32.372 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.893767+0000", 2024-09-21T12:26:32.372 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9993999999999994e-05 2024-09-21T12:26:32.373 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.373 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:26:32.373 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.373 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.373 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.373 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46270 7.9 7:94425725:::benchmark_data_smithi114_78432_object46269:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-21T12:26:32.373 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:00.884498+0000", 2024-09-21T12:26:32.373 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 270.45749893099998, 2024-09-21T12:26:32.373 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.90768878200000003, 2024-09-21T12:26:32.373 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:26:32.373 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:26:32.373 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:32.373 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:26:32.373 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:26:32.374 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:32.374 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46270 2024-09-21T12:26:32.374 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.374 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:26:32.374 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.374 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:26:32.374 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.884498+0000", 2024-09-21T12:26:32.374 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.374 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.374 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.374 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:26:32.374 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.884498+0000", 2024-09-21T12:26:32.374 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.375 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.375 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.375 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:26:32.375 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.884500+0000", 2024-09-21T12:26:32.375 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9489999999999999e-06 2024-09-21T12:26:32.375 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.375 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.375 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:26:32.375 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.884523+0000", 2024-09-21T12:26:32.375 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3091e-05 2024-09-21T12:26:32.375 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.375 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.375 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:26:32.375 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.884539+0000", 2024-09-21T12:26:32.376 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6368999999999999e-05 2024-09-21T12:26:32.376 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.376 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.376 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:26:32.376 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.884548+0000", 2024-09-21T12:26:32.376 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.0649999999999999e-06 2024-09-21T12:26:32.376 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.376 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.376 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.376 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.884632+0000", 2024-09-21T12:26:32.376 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.4129999999999996e-05 2024-09-21T12:26:32.376 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.376 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.376 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:26:32.377 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.884645+0000", 2024-09-21T12:26:32.377 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.383e-05 2024-09-21T12:26:32.377 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.377 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.377 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.377 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.789278+0000", 2024-09-21T12:26:32.377 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.90463295600000004 2024-09-21T12:26:32.377 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.377 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.377 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:26:32.377 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.789393+0000", 2024-09-21T12:26:32.377 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000114523 2024-09-21T12:26:32.377 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.378 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.378 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:26:32.378 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.789490+0000", 2024-09-21T12:26:32.378 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7528000000000005e-05 2024-09-21T12:26:32.378 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.378 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.378 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.379 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.792083+0000", 2024-09-21T12:26:32.379 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0025928840000000002 2024-09-21T12:26:32.379 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.379 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.379 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:26:32.379 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.792091+0000", 2024-09-21T12:26:32.379 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9189999999999999e-06 2024-09-21T12:26:32.379 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.379 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.379 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.380 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.792124+0000", 2024-09-21T12:26:32.380 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.3118999999999999e-05 2024-09-21T12:26:32.380 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.380 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.380 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:26:32.380 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.792155+0000", 2024-09-21T12:26:32.380 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.0239999999999998e-05 2024-09-21T12:26:32.380 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.380 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.380 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:26:32.380 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.792187+0000", 2024-09-21T12:26:32.381 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2178999999999998e-05 2024-09-21T12:26:32.381 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.381 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:26:32.381 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.381 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.381 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.381 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46316 7.9 7:91032405:::benchmark_data_smithi114_78432_object46315:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-21T12:26:32.381 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:00.914256+0000", 2024-09-21T12:26:32.381 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 270.42774142600001, 2024-09-21T12:26:32.381 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.87904488400000003, 2024-09-21T12:26:32.381 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:26:32.381 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:26:32.381 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:26:32.382 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:26:32.382 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:26:32.382 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:26:32.382 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46316 2024-09-21T12:26:32.382 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.382 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:26:32.382 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.382 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:26:32.382 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.914256+0000", 2024-09-21T12:26:32.382 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.382 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.382 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.382 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:26:32.383 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.914256+0000", 2024-09-21T12:26:32.383 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:26:32.383 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.383 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.383 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:26:32.383 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.914257+0000", 2024-09-21T12:26:32.383 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.33e-06 2024-09-21T12:26:32.383 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.383 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.383 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:26:32.383 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.914280+0000", 2024-09-21T12:26:32.383 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3119999999999999e-05 2024-09-21T12:26:32.383 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.383 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.383 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:26:32.384 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.914290+0000", 2024-09-21T12:26:32.384 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0178e-05 2024-09-21T12:26:32.384 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.384 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.384 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:26:32.384 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.914298+0000", 2024-09-21T12:26:32.384 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5830000000000001e-06 2024-09-21T12:26:32.384 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.384 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.384 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.384 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.914474+0000", 2024-09-21T12:26:32.385 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017619700000000001 2024-09-21T12:26:32.385 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.385 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.385 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:26:32.385 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:00.914483+0000", 2024-09-21T12:26:32.385 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.2399999999999996e-06 2024-09-21T12:26:32.385 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.385 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.385 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:26:32.385 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.790990+0000", 2024-09-21T12:26:32.386 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.87650638599999997 2024-09-21T12:26:32.386 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.386 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.386 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:26:32.386 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.791118+0000", 2024-09-21T12:26:32.386 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012878299999999999 2024-09-21T12:26:32.386 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.386 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.386 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:26:32.386 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.791207+0000", 2024-09-21T12:26:32.386 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.8964999999999999e-05 2024-09-21T12:26:32.387 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.387 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.387 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.387 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.793207+0000", 2024-09-21T12:26:32.387 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0020000180000000001 2024-09-21T12:26:32.387 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.387 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.387 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:26:32.387 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.793242+0000", 2024-09-21T12:26:32.387 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4687000000000002e-05 2024-09-21T12:26:32.387 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.387 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.387 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:26:32.388 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.793251+0000", 2024-09-21T12:26:32.388 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.901e-06 2024-09-21T12:26:32.388 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.388 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.388 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:26:32.388 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.793278+0000", 2024-09-21T12:26:32.388 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.7328999999999999e-05 2024-09-21T12:26:32.388 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:26:32.388 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:26:32.388 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:26:32.388 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.793300+0000", 2024-09-21T12:26:32.388 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2167e-05 2024-09-21T12:26:32.388 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.388 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:26:32.388 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.389 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:26:32.389 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:26:32.389 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:26:32.389 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:32.417 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:32.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:32.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:32.519 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:32.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:32 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:26:32.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:32 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:26:32.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:32 smithi114 ceph-mon[40867]: pgmap v301: 158 pgs: 158 active+clean; 7.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 42 MiB/s wr, 1.26k op/s 2024-09-21T12:26:32.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:32 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:26:32.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:32 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:26:32.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:32 smithi114 ceph-mon[30324]: pgmap v301: 158 pgs: 158 active+clean; 7.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 42 MiB/s wr, 1.26k op/s 2024-09-21T12:26:32.546 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:32.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:32.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:32.647 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:32.674 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:32.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:32.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:32.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:32 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:26:32.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:32 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:26:32.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:32 smithi183 ceph-mon[33277]: pgmap v301: 158 pgs: 158 active+clean; 7.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 42 MiB/s wr, 1.26k op/s 2024-09-21T12:26:32.776 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:32.802 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:32.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:32.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:32.904 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:32.944 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:32.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:32.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:33.024 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:26:32 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:26:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:26:33.046 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:33.077 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:33.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:33.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:33.178 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:33.225 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:33.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:33.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:33.327 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:33.356 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:33.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:33.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:33.458 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:33.497 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:33.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:33.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:33.599 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:33.626 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:33.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:33.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:33.728 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:33.754 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:33.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:33.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:33.857 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:33.895 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:33.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:33.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:33.997 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:34.028 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:34.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:34.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:34.124 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-09-21T12:26:27.686083+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8296405555,"num_objects":126784,"num_object_clones":0,"num_object_copies":380352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":126784,"num_whiteouts":0,"num_read":1081,"num_read_kb":833,"num_write":134679,"num_write_kb":8346478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":223,"num_bytes_recovered":4534,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70951,"ondisk_log_size":70951,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":26233956,"kb_used_data":24458676,"kb_used_omap":10,"kb_used_meta":1774453,"kb_avail":629982108,"statfs":{"total":671965249536,"available":645101678592,"internally_reserved":0,"allocated":25045684224,"data_stored":25043041015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":1817039979},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":308936728,"num_objects":4715,"num_object_clones":0,"num_object_copies":14145,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4715,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":12373,"num_write_kb":546753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.713932"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":312,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.028509+0000","last_change":"2024-09-21T12:24:50.115300+0000","last_active":"2024-09-21T12:26:22.028509+0000","last_peered":"2024-09-21T12:26:22.028509+0000","last_clean":"2024-09-21T12:26:22.028509+0000","last_became_active":"2024-09-21T12:20:15.309431+0000","last_became_peered":"2024-09-21T12:20:15.309431+0000","last_unstale":"2024-09-21T12:26:22.028509+0000","last_undegraded":"2024-09-21T12:26:22.028509+0000","last_fullsized":"2024-09-21T12:26:22.028509+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:50.115253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:50.115253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:04.907671+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":282,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036610+0000","last_change":"2024-09-21T12:26:09.823435+0000","last_active":"2024-09-21T12:26:22.036610+0000","last_peered":"2024-09-21T12:26:22.036610+0000","last_clean":"2024-09-21T12:26:22.036610+0000","last_became_active":"2024-09-21T12:22:50.909699+0000","last_became_peered":"2024-09-21T12:22:50.909699+0000","last_unstale":"2024-09-21T12:26:22.036610+0000","last_undegraded":"2024-09-21T12:26:22.036610+0000","last_fullsized":"2024-09-21T12:26:22.036610+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:26:09.823363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:09.823363+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:17.313787+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":304,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.035875+0000","last_change":"2024-09-21T12:24:34.311501+0000","last_active":"2024-09-21T12:26:22.035875+0000","last_peered":"2024-09-21T12:26:22.035875+0000","last_clean":"2024-09-21T12:26:22.035875+0000","last_became_active":"2024-09-21T12:20:19.314908+0000","last_became_peered":"2024-09-21T12:20:19.314908+0000","last_unstale":"2024-09-21T12:26:22.035875+0000","last_undegraded":"2024-09-21T12:26:22.035875+0000","last_fullsized":"2024-09-21T12:26:22.035875+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:34.311451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:34.311451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:00.014883+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":227,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039419+0000","last_change":"2024-09-21T12:22:50.914064+0000","last_active":"2024-09-21T12:26:22.039419+0000","last_peered":"2024-09-21T12:26:22.039419+0000","last_clean":"2024-09-21T12:26:22.039419+0000","last_became_active":"2024-09-21T12:22:50.913743+0000","last_became_peered":"2024-09-21T12:22:50.913743+0000","last_unstale":"2024-09-21T12:26:22.039419+0000","last_undegraded":"2024-09-21T12:26:22.039419+0000","last_fullsized":"2024-09-21T12:26:22.039419+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:58.189665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:21:58.189665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:07.338098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":290,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.032565+0000","last_change":"2024-09-21T12:25:48.316122+0000","last_active":"2024-09-21T12:26:22.032565+0000","last_peered":"2024-09-21T12:26:22.032565+0000","last_clean":"2024-09-21T12:26:22.032565+0000","last_became_active":"2024-09-21T12:20:21.595301+0000","last_became_peered":"2024-09-21T12:20:21.595301+0000","last_unstale":"2024-09-21T12:26:22.032565+0000","last_undegraded":"2024-09-21T12:26:22.032565+0000","last_fullsized":"2024-09-21T12:26:22.032565+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:48.316039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:48.316039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:08.665134+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":311,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.031696+0000","last_change":"2024-09-21T12:24:49.701972+0000","last_active":"2024-09-21T12:26:22.031696+0000","last_peered":"2024-09-21T12:26:22.031696+0000","last_clean":"2024-09-21T12:26:22.031696+0000","last_became_active":"2024-09-21T12:20:15.308655+0000","last_became_peered":"2024-09-21T12:20:15.308655+0000","last_unstale":"2024-09-21T12:26:22.031696+0000","last_undegraded":"2024-09-21T12:26:22.031696+0000","last_fullsized":"2024-09-21T12:26:22.031696+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:49.701915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:49.701915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":331,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.032104+0000","last_change":"2024-09-21T12:25:01.898461+0000","last_active":"2024-09-21T12:26:22.032104+0000","last_peered":"2024-09-21T12:26:22.032104+0000","last_clean":"2024-09-21T12:26:22.032104+0000","last_became_active":"2024-09-21T12:20:17.305220+0000","last_became_peered":"2024-09-21T12:20:17.305220+0000","last_unstale":"2024-09-21T12:26:22.032104+0000","last_undegraded":"2024-09-21T12:26:22.032104+0000","last_fullsized":"2024-09-21T12:26:22.032104+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:25:01.898376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:01.898376+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":244,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036754+0000","last_change":"2024-09-21T12:25:04.827596+0000","last_active":"2024-09-21T12:26:22.036754+0000","last_peered":"2024-09-21T12:26:22.036754+0000","last_clean":"2024-09-21T12:26:22.036754+0000","last_became_active":"2024-09-21T12:22:50.926240+0000","last_became_peered":"2024-09-21T12:22:50.926240+0000","last_unstale":"2024-09-21T12:26:22.036754+0000","last_undegraded":"2024-09-21T12:26:22.036754+0000","last_fullsized":"2024-09-21T12:26:22.036754+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:04.827529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:04.827529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":332,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.040061+0000","last_change":"2024-09-21T12:26:19.446300+0000","last_active":"2024-09-21T12:26:22.040061+0000","last_peered":"2024-09-21T12:26:22.040061+0000","last_clean":"2024-09-21T12:26:22.040061+0000","last_became_active":"2024-09-21T12:22:50.912481+0000","last_became_peered":"2024-09-21T12:22:50.912481+0000","last_unstale":"2024-09-21T12:26:22.040061+0000","last_undegraded":"2024-09-21T12:26:22.040061+0000","last_fullsized":"2024-09-21T12:26:22.040061+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:19.446251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:19.446251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:36.052328+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":291,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.030071+0000","last_change":"2024-09-21T12:24:25.640253+0000","last_active":"2024-09-21T12:26:22.030071+0000","last_peered":"2024-09-21T12:26:22.030071+0000","last_clean":"2024-09-21T12:26:22.030071+0000","last_became_active":"2024-09-21T12:22:50.927804+0000","last_became_peered":"2024-09-21T12:22:50.927804+0000","last_unstale":"2024-09-21T12:26:22.030071+0000","last_undegraded":"2024-09-21T12:26:22.030071+0000","last_fullsized":"2024-09-21T12:26:22.030071+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:24:25.640169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:25.640169+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:58.377864+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":303,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039580+0000","last_change":"2024-09-21T12:24:58.566964+0000","last_active":"2024-09-21T12:26:22.039580+0000","last_peered":"2024-09-21T12:26:22.039580+0000","last_clean":"2024-09-21T12:26:22.039580+0000","last_became_active":"2024-09-21T12:20:19.308184+0000","last_became_peered":"2024-09-21T12:20:19.308184+0000","last_unstale":"2024-09-21T12:26:22.039580+0000","last_undegraded":"2024-09-21T12:26:22.039580+0000","last_fullsized":"2024-09-21T12:26:22.039580+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:58.566902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:58.566902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:27.540756+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":300,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.026521+0000","last_change":"2024-09-21T12:25:31.616293+0000","last_active":"2024-09-21T12:26:22.026521+0000","last_peered":"2024-09-21T12:26:22.026521+0000","last_clean":"2024-09-21T12:26:22.026521+0000","last_became_active":"2024-09-21T12:20:21.309190+0000","last_became_peered":"2024-09-21T12:20:21.309190+0000","last_unstale":"2024-09-21T12:26:22.026521+0000","last_undegraded":"2024-09-21T12:26:22.026521+0000","last_fullsized":"2024-09-21T12:26:22.026521+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:31.616219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:31.616219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:48.224028+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":324,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039857+0000","last_change":"2024-09-21T12:22:54.174099+0000","last_active":"2024-09-21T12:26:22.039857+0000","last_peered":"2024-09-21T12:26:22.039857+0000","last_clean":"2024-09-21T12:26:22.039857+0000","last_became_active":"2024-09-21T12:22:50.911737+0000","last_became_peered":"2024-09-21T12:22:50.911737+0000","last_unstale":"2024-09-21T12:26:22.039857+0000","last_undegraded":"2024-09-21T12:26:22.039857+0000","last_fullsized":"2024-09-21T12:26:22.039857+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:54.174034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:54.174034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:23.349015+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":277,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.032818+0000","last_change":"2024-09-21T12:24:28.624430+0000","last_active":"2024-09-21T12:26:22.032818+0000","last_peered":"2024-09-21T12:26:22.032818+0000","last_clean":"2024-09-21T12:26:22.032818+0000","last_became_active":"2024-09-21T12:22:50.918794+0000","last_became_peered":"2024-09-21T12:22:50.918794+0000","last_unstale":"2024-09-21T12:26:22.032818+0000","last_undegraded":"2024-09-21T12:26:22.032818+0000","last_fullsized":"2024-09-21T12:26:22.032818+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-21T12:24:28.624364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:28.624364+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":316,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036516+0000","last_change":"2024-09-21T12:25:01.897596+0000","last_active":"2024-09-21T12:26:22.036516+0000","last_peered":"2024-09-21T12:26:22.036516+0000","last_clean":"2024-09-21T12:26:22.036516+0000","last_became_active":"2024-09-21T12:22:50.921780+0000","last_became_peered":"2024-09-21T12:22:50.921780+0000","last_unstale":"2024-09-21T12:26:22.036516+0000","last_undegraded":"2024-09-21T12:26:22.036516+0000","last_fullsized":"2024-09-21T12:26:22.036516+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:01.897527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:01.897527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":307,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039403+0000","last_change":"2024-09-21T12:24:34.367580+0000","last_active":"2024-09-21T12:26:22.039403+0000","last_peered":"2024-09-21T12:26:22.039403+0000","last_clean":"2024-09-21T12:26:22.039403+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:26:22.039403+0000","last_undegraded":"2024-09-21T12:26:22.039403+0000","last_fullsized":"2024-09-21T12:26:22.039403+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:34.367532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:34.367532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":244,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037928+0000","last_change":"2024-09-21T12:25:30.888535+0000","last_active":"2024-09-21T12:26:22.037928+0000","last_peered":"2024-09-21T12:26:22.037928+0000","last_clean":"2024-09-21T12:26:22.037928+0000","last_became_active":"2024-09-21T12:22:50.914792+0000","last_became_peered":"2024-09-21T12:22:50.914792+0000","last_unstale":"2024-09-21T12:26:22.037928+0000","last_undegraded":"2024-09-21T12:26:22.037928+0000","last_fullsized":"2024-09-21T12:26:22.037928+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:30.888491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:30.888491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:32.893945+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":341,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.029197+0000","last_change":"2024-09-21T12:24:30.033946+0000","last_active":"2024-09-21T12:26:22.029197+0000","last_peered":"2024-09-21T12:26:22.029197+0000","last_clean":"2024-09-21T12:26:22.029197+0000","last_became_active":"2024-09-21T12:22:50.921576+0000","last_became_peered":"2024-09-21T12:22:50.921576+0000","last_unstale":"2024-09-21T12:26:22.029197+0000","last_undegraded":"2024-09-21T12:26:22.029197+0000","last_fullsized":"2024-09-21T12:26:22.029197+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:30.033874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:30.033874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:00.878523+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":333,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038044+0000","last_change":"2024-09-21T12:24:50.834579+0000","last_active":"2024-09-21T12:26:22.038044+0000","last_peered":"2024-09-21T12:26:22.038044+0000","last_clean":"2024-09-21T12:26:22.038044+0000","last_became_active":"2024-09-21T12:20:17.307828+0000","last_became_peered":"2024-09-21T12:20:17.307828+0000","last_unstale":"2024-09-21T12:26:22.038044+0000","last_undegraded":"2024-09-21T12:26:22.038044+0000","last_fullsized":"2024-09-21T12:26:22.038044+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:24:50.834503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:50.834503+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":332,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037949+0000","last_change":"2024-09-21T12:25:46.906275+0000","last_active":"2024-09-21T12:26:22.037949+0000","last_peered":"2024-09-21T12:26:22.037949+0000","last_clean":"2024-09-21T12:26:22.037949+0000","last_became_active":"2024-09-21T12:25:46.905878+0000","last_became_peered":"2024-09-21T12:25:46.905878+0000","last_unstale":"2024-09-21T12:26:22.037949+0000","last_undegraded":"2024-09-21T12:26:22.037949+0000","last_fullsized":"2024-09-21T12:26:22.037949+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:48.435769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:48.435769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:57.929606+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":290,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.029802+0000","last_change":"2024-09-21T12:23:58.354628+0000","last_active":"2024-09-21T12:26:22.029802+0000","last_peered":"2024-09-21T12:26:22.029802+0000","last_clean":"2024-09-21T12:26:22.029802+0000","last_became_active":"2024-09-21T12:20:21.592796+0000","last_became_peered":"2024-09-21T12:20:21.592796+0000","last_unstale":"2024-09-21T12:26:22.029802+0000","last_undegraded":"2024-09-21T12:26:22.029802+0000","last_fullsized":"2024-09-21T12:26:22.029802+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:58.354570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:58.354570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":243,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038676+0000","last_change":"2024-09-21T12:24:55.447829+0000","last_active":"2024-09-21T12:26:22.038676+0000","last_peered":"2024-09-21T12:26:22.038676+0000","last_clean":"2024-09-21T12:26:22.038676+0000","last_became_active":"2024-09-21T12:22:50.925403+0000","last_became_peered":"2024-09-21T12:22:50.925403+0000","last_unstale":"2024-09-21T12:26:22.038676+0000","last_undegraded":"2024-09-21T12:26:22.038676+0000","last_fullsized":"2024-09-21T12:26:22.038676+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:55.447755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:55.447755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":386,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.029742+0000","last_change":"2024-09-21T12:25:31.270272+0000","last_active":"2024-09-21T12:26:22.029742+0000","last_peered":"2024-09-21T12:26:22.029742+0000","last_clean":"2024-09-21T12:26:22.029742+0000","last_became_active":"2024-09-21T12:22:50.912144+0000","last_became_peered":"2024-09-21T12:22:50.912144+0000","last_unstale":"2024-09-21T12:26:22.029742+0000","last_undegraded":"2024-09-21T12:26:22.029742+0000","last_fullsized":"2024-09-21T12:26:22.029742+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:25:31.270165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:31.270165+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:48.492514+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":220,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.030756+0000","last_change":"2024-09-21T12:22:50.920178+0000","last_active":"2024-09-21T12:26:22.030756+0000","last_peered":"2024-09-21T12:26:22.030756+0000","last_clean":"2024-09-21T12:26:22.030756+0000","last_became_active":"2024-09-21T12:22:50.918326+0000","last_became_peered":"2024-09-21T12:22:50.918326+0000","last_unstale":"2024-09-21T12:26:22.030756+0000","last_undegraded":"2024-09-21T12:26:22.030756+0000","last_fullsized":"2024-09-21T12:26:22.030756+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:50.318827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":292,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037320+0000","last_change":"2024-09-21T12:23:51.065780+0000","last_active":"2024-09-21T12:26:22.037320+0000","last_peered":"2024-09-21T12:26:22.037320+0000","last_clean":"2024-09-21T12:26:22.037320+0000","last_became_active":"2024-09-21T12:20:21.312976+0000","last_became_peered":"2024-09-21T12:20:21.312976+0000","last_unstale":"2024-09-21T12:26:22.037320+0000","last_undegraded":"2024-09-21T12:26:22.037320+0000","last_fullsized":"2024-09-21T12:26:22.037320+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:23:51.065733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:51.065733+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:06.800726+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":346,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036066+0000","last_change":"2024-09-21T12:26:18.112687+0000","last_active":"2024-09-21T12:26:22.036066+0000","last_peered":"2024-09-21T12:26:22.036066+0000","last_clean":"2024-09-21T12:26:22.036066+0000","last_became_active":"2024-09-21T12:22:50.910933+0000","last_became_peered":"2024-09-21T12:22:50.910933+0000","last_unstale":"2024-09-21T12:26:22.036066+0000","last_undegraded":"2024-09-21T12:26:22.036066+0000","last_fullsized":"2024-09-21T12:26:22.036066+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:26:18.112614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:18.112614+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:47.795526+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":316,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.028890+0000","last_change":"2024-09-21T12:24:03.059782+0000","last_active":"2024-09-21T12:26:22.028890+0000","last_peered":"2024-09-21T12:26:22.028890+0000","last_clean":"2024-09-21T12:26:22.028890+0000","last_became_active":"2024-09-21T12:20:17.695977+0000","last_became_peered":"2024-09-21T12:20:17.695977+0000","last_unstale":"2024-09-21T12:26:22.028890+0000","last_undegraded":"2024-09-21T12:26:22.028890+0000","last_fullsized":"2024-09-21T12:26:22.028890+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:24:03.059549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:03.059549+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:25.640560+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"121'11","reported_seq":430,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038348+0000","last_change":"2024-09-21T12:26:10.846256+0000","last_active":"2024-09-21T12:26:22.038348+0000","last_peered":"2024-09-21T12:26:22.038348+0000","last_clean":"2024-09-21T12:26:22.038348+0000","last_became_active":"2024-09-21T12:22:50.916683+0000","last_became_peered":"2024-09-21T12:22:50.916683+0000","last_unstale":"2024-09-21T12:26:22.038348+0000","last_undegraded":"2024-09-21T12:26:22.038348+0000","last_fullsized":"2024-09-21T12:26:22.038348+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:26:10.846183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:10.846183+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:32.395882+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"200'8203","reported_seq":8294,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:26.831423+0000","last_change":"2024-09-21T12:24:46.316817+0000","last_active":"2024-09-21T12:26:26.831423+0000","last_peered":"2024-09-21T12:26:26.831423+0000","last_clean":"2024-09-21T12:26:26.831423+0000","last_became_active":"2024-09-21T12:24:46.316132+0000","last_became_peered":"2024-09-21T12:24:46.316132+0000","last_unstale":"2024-09-21T12:26:26.831423+0000","last_undegraded":"2024-09-21T12:26:26.831423+0000","last_fullsized":"2024-09-21T12:26:26.831423+0000","mapping_epoch":160,"log_start":"178'3900","ondisk_log_start":"178'3900","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4303,"log_dups_size":0,"ondisk_log_size":4303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:05.983258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":507576320,"num_objects":7745,"num_object_clones":0,"num_object_copies":23235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8203,"num_write_kb":510336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"76'79","reported_seq":212,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039238+0000","last_change":"2024-09-21T12:25:42.586460+0000","last_active":"2024-09-21T12:26:22.039238+0000","last_peered":"2024-09-21T12:26:22.039238+0000","last_clean":"2024-09-21T12:26:22.039238+0000","last_became_active":"2024-09-21T12:22:17.074721+0000","last_became_peered":"2024-09-21T12:22:17.074721+0000","last_unstale":"2024-09-21T12:26:22.039238+0000","last_undegraded":"2024-09-21T12:26:22.039238+0000","last_fullsized":"2024-09-21T12:26:22.039238+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:25:42.586392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:25:42.586392+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:57.694710+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":163,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037681+0000","last_change":"2024-09-21T12:24:59.803037+0000","last_active":"2024-09-21T12:26:22.037681+0000","last_peered":"2024-09-21T12:26:22.037681+0000","last_clean":"2024-09-21T12:26:22.037681+0000","last_became_active":"2024-09-21T12:23:46.910112+0000","last_became_peered":"2024-09-21T12:23:46.910112+0000","last_unstale":"2024-09-21T12:26:22.037681+0000","last_undegraded":"2024-09-21T12:26:22.037681+0000","last_fullsized":"2024-09-21T12:26:22.037681+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:24:59.802945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:59.802945+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:22.089988+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":302,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.031584+0000","last_change":"2024-09-21T12:22:14.036290+0000","last_active":"2024-09-21T12:26:22.031584+0000","last_peered":"2024-09-21T12:26:22.031584+0000","last_clean":"2024-09-21T12:26:22.031584+0000","last_became_active":"2024-09-21T12:20:15.295774+0000","last_became_peered":"2024-09-21T12:20:15.295774+0000","last_unstale":"2024-09-21T12:26:22.031584+0000","last_undegraded":"2024-09-21T12:26:22.031584+0000","last_fullsized":"2024-09-21T12:26:22.031584+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:14.036236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.036236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:55.467472+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"158'13","reported_seq":133,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039636+0000","last_change":"2024-09-21T12:25:33.474458+0000","last_active":"2024-09-21T12:26:22.039636+0000","last_peered":"2024-09-21T12:26:22.039636+0000","last_clean":"2024-09-21T12:26:22.039636+0000","last_became_active":"2024-09-21T12:24:39.272835+0000","last_became_peered":"2024-09-21T12:24:39.272835+0000","last_unstale":"2024-09-21T12:26:22.039636+0000","last_undegraded":"2024-09-21T12:26:22.039636+0000","last_fullsized":"2024-09-21T12:26:22.039636+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'13","last_scrub_stamp":"2024-09-21T12:25:33.474391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:33.474391+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:01.060227+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":292,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039652+0000","last_change":"2024-09-21T12:22:12.145637+0000","last_active":"2024-09-21T12:26:22.039652+0000","last_peered":"2024-09-21T12:26:22.039652+0000","last_clean":"2024-09-21T12:26:22.039652+0000","last_became_active":"2024-09-21T12:20:21.310022+0000","last_became_peered":"2024-09-21T12:20:21.310022+0000","last_unstale":"2024-09-21T12:26:22.039652+0000","last_undegraded":"2024-09-21T12:26:22.039652+0000","last_fullsized":"2024-09-21T12:26:22.039652+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:22:12.145550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:12.145550+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:51.460208+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"200'8438","reported_seq":8529,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:27.185944+0000","last_change":"2024-09-21T12:24:46.326848+0000","last_active":"2024-09-21T12:26:27.185944+0000","last_peered":"2024-09-21T12:26:27.185944+0000","last_clean":"2024-09-21T12:26:27.185944+0000","last_became_active":"2024-09-21T12:24:46.326586+0000","last_became_peered":"2024-09-21T12:24:46.326586+0000","last_unstale":"2024-09-21T12:26:27.185944+0000","last_undegraded":"2024-09-21T12:26:27.185944+0000","last_fullsized":"2024-09-21T12:26:27.185944+0000","mapping_epoch":160,"log_start":"179'4100","ondisk_log_start":"179'4100","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4338,"log_dups_size":0,"ondisk_log_size":4338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":519045120,"num_objects":7920,"num_object_clones":0,"num_object_copies":23760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8438,"num_write_kb":523456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"76'79","reported_seq":202,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037642+0000","last_change":"2024-09-21T12:26:21.091972+0000","last_active":"2024-09-21T12:26:22.037642+0000","last_peered":"2024-09-21T12:26:22.037642+0000","last_clean":"2024-09-21T12:26:22.037642+0000","last_became_active":"2024-09-21T12:22:50.914885+0000","last_became_peered":"2024-09-21T12:22:50.914885+0000","last_unstale":"2024-09-21T12:26:22.037642+0000","last_undegraded":"2024-09-21T12:26:22.037642+0000","last_fullsized":"2024-09-21T12:26:22.037642+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:21.091797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:21.091797+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:25.919035+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":339,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.031414+0000","last_change":"2024-09-21T12:25:35.325593+0000","last_active":"2024-09-21T12:26:22.031414+0000","last_peered":"2024-09-21T12:26:22.031414+0000","last_clean":"2024-09-21T12:26:22.031414+0000","last_became_active":"2024-09-21T12:20:17.693179+0000","last_became_peered":"2024-09-21T12:20:17.693179+0000","last_unstale":"2024-09-21T12:26:22.031414+0000","last_undegraded":"2024-09-21T12:26:22.031414+0000","last_fullsized":"2024-09-21T12:26:22.031414+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-21T12:25:35.325493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:35.325493+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:38.650408+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":302,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.031170+0000","last_change":"2024-09-21T12:21:18.056305+0000","last_active":"2024-09-21T12:26:22.031170+0000","last_peered":"2024-09-21T12:26:22.031170+0000","last_clean":"2024-09-21T12:26:22.031170+0000","last_became_active":"2024-09-21T12:20:15.297340+0000","last_became_peered":"2024-09-21T12:20:15.297340+0000","last_unstale":"2024-09-21T12:26:22.031170+0000","last_undegraded":"2024-09-21T12:26:22.031170+0000","last_fullsized":"2024-09-21T12:26:22.031170+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:18.056261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:18.056261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:07.391958+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":245,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.031748+0000","last_change":"2024-09-21T12:25:38.374222+0000","last_active":"2024-09-21T12:26:22.031748+0000","last_peered":"2024-09-21T12:26:22.031748+0000","last_clean":"2024-09-21T12:26:22.031748+0000","last_became_active":"2024-09-21T12:22:50.930858+0000","last_became_peered":"2024-09-21T12:22:50.930858+0000","last_unstale":"2024-09-21T12:26:22.031748+0000","last_undegraded":"2024-09-21T12:26:22.031748+0000","last_fullsized":"2024-09-21T12:26:22.031748+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:38.374146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:38.374146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:58.524870+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":236,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.029963+0000","last_change":"2024-09-21T12:24:29.635826+0000","last_active":"2024-09-21T12:26:22.029963+0000","last_peered":"2024-09-21T12:26:22.029963+0000","last_clean":"2024-09-21T12:26:22.029963+0000","last_became_active":"2024-09-21T12:22:50.918940+0000","last_became_peered":"2024-09-21T12:22:50.918940+0000","last_unstale":"2024-09-21T12:26:22.029963+0000","last_undegraded":"2024-09-21T12:26:22.029963+0000","last_fullsized":"2024-09-21T12:26:22.029963+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:29.635514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:29.635514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"200'8364","reported_seq":8456,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:26.591070+0000","last_change":"2024-09-21T12:24:46.324341+0000","last_active":"2024-09-21T12:26:26.591070+0000","last_peered":"2024-09-21T12:26:26.591070+0000","last_clean":"2024-09-21T12:26:26.591070+0000","last_became_active":"2024-09-21T12:24:46.323986+0000","last_became_peered":"2024-09-21T12:24:46.323986+0000","last_unstale":"2024-09-21T12:26:26.591070+0000","last_undegraded":"2024-09-21T12:26:26.591070+0000","last_fullsized":"2024-09-21T12:26:26.591070+0000","mapping_epoch":160,"log_start":"178'3900","ondisk_log_start":"178'3900","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4464,"log_dups_size":0,"ondisk_log_size":4464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":518848536,"num_objects":7918,"num_object_clones":0,"num_object_copies":23754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7918,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8364,"num_write_kb":520961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"76'79","reported_seq":218,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.030460+0000","last_change":"2024-09-21T12:25:02.911776+0000","last_active":"2024-09-21T12:26:22.030460+0000","last_peered":"2024-09-21T12:26:22.030460+0000","last_clean":"2024-09-21T12:26:22.030460+0000","last_became_active":"2024-09-21T12:22:17.069042+0000","last_became_peered":"2024-09-21T12:22:17.069042+0000","last_unstale":"2024-09-21T12:26:22.030460+0000","last_undegraded":"2024-09-21T12:26:22.030460+0000","last_fullsized":"2024-09-21T12:26:22.030460+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:25:02.911705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:25:02.911705+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":301,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039447+0000","last_change":"2024-09-21T12:22:50.978102+0000","last_active":"2024-09-21T12:26:22.039447+0000","last_peered":"2024-09-21T12:26:22.039447+0000","last_clean":"2024-09-21T12:26:22.039447+0000","last_became_active":"2024-09-21T12:22:50.926008+0000","last_became_peered":"2024-09-21T12:22:50.926008+0000","last_unstale":"2024-09-21T12:26:22.039447+0000","last_undegraded":"2024-09-21T12:26:22.039447+0000","last_fullsized":"2024-09-21T12:26:22.039447+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-21T12:22:14.062464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.062464+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:49.540547+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":338,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.029193+0000","last_change":"2024-09-21T12:22:51.003768+0000","last_active":"2024-09-21T12:26:22.029193+0000","last_peered":"2024-09-21T12:26:22.029193+0000","last_clean":"2024-09-21T12:26:22.029193+0000","last_became_active":"2024-09-21T12:22:50.923284+0000","last_became_peered":"2024-09-21T12:22:50.923284+0000","last_unstale":"2024-09-21T12:26:22.029193+0000","last_undegraded":"2024-09-21T12:26:22.029193+0000","last_fullsized":"2024-09-21T12:26:22.029193+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:22:07.582969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:07.582969+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:53.794213+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"121'11","reported_seq":419,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:25.436656+0000","last_change":"2024-09-21T12:26:14.490045+0000","last_active":"2024-09-21T12:26:25.436656+0000","last_peered":"2024-09-21T12:26:25.436656+0000","last_clean":"2024-09-21T12:26:25.436656+0000","last_became_active":"2024-09-21T12:22:50.930342+0000","last_became_peered":"2024-09-21T12:22:50.930342+0000","last_unstale":"2024-09-21T12:26:25.436656+0000","last_undegraded":"2024-09-21T12:26:25.436656+0000","last_fullsized":"2024-09-21T12:26:25.436656+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:26:14.489972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:14.489972+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:31.388160+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":291,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036752+0000","last_change":"2024-09-21T12:21:21.835669+0000","last_active":"2024-09-21T12:26:22.036752+0000","last_peered":"2024-09-21T12:26:22.036752+0000","last_clean":"2024-09-21T12:26:22.036752+0000","last_became_active":"2024-09-21T12:20:21.593367+0000","last_became_peered":"2024-09-21T12:20:21.593367+0000","last_unstale":"2024-09-21T12:26:22.036752+0000","last_undegraded":"2024-09-21T12:26:22.036752+0000","last_fullsized":"2024-09-21T12:26:22.036752+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:21.835623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:21:21.835623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:29.766408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"200'8329","reported_seq":8420,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:27.225016+0000","last_change":"2024-09-21T12:24:46.325776+0000","last_active":"2024-09-21T12:26:27.225016+0000","last_peered":"2024-09-21T12:26:27.225016+0000","last_clean":"2024-09-21T12:26:27.225016+0000","last_became_active":"2024-09-21T12:24:46.325260+0000","last_became_peered":"2024-09-21T12:24:46.325260+0000","last_unstale":"2024-09-21T12:26:27.225016+0000","last_undegraded":"2024-09-21T12:26:27.225016+0000","last_fullsized":"2024-09-21T12:26:27.225016+0000","mapping_epoch":160,"log_start":"178'4000","ondisk_log_start":"178'4000","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4329,"log_dups_size":0,"ondisk_log_size":4329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:01.850898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":511115264,"num_objects":7799,"num_object_clones":0,"num_object_copies":23397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8329,"num_write_kb":516096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"76'79","reported_seq":302,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038101+0000","last_change":"2024-09-21T12:24:51.918097+0000","last_active":"2024-09-21T12:26:22.038101+0000","last_peered":"2024-09-21T12:26:22.038101+0000","last_clean":"2024-09-21T12:26:22.038101+0000","last_became_active":"2024-09-21T12:22:17.093298+0000","last_became_peered":"2024-09-21T12:22:17.093298+0000","last_unstale":"2024-09-21T12:26:22.038101+0000","last_undegraded":"2024-09-21T12:26:22.038101+0000","last_fullsized":"2024-09-21T12:26:22.038101+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:24:51.918028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:24:51.918028+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":372,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038158+0000","last_change":"2024-09-21T12:24:01.388987+0000","last_active":"2024-09-21T12:26:22.038158+0000","last_peered":"2024-09-21T12:26:22.038158+0000","last_clean":"2024-09-21T12:26:22.038158+0000","last_became_active":"2024-09-21T12:22:50.925840+0000","last_became_peered":"2024-09-21T12:22:50.925840+0000","last_unstale":"2024-09-21T12:26:22.038158+0000","last_undegraded":"2024-09-21T12:26:22.038158+0000","last_fullsized":"2024-09-21T12:26:22.038158+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:24:01.388905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:01.388905+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":311,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.040202+0000","last_change":"2024-09-21T12:26:16.410118+0000","last_active":"2024-09-21T12:26:22.040202+0000","last_peered":"2024-09-21T12:26:22.040202+0000","last_clean":"2024-09-21T12:26:22.040202+0000","last_became_active":"2024-09-21T12:20:15.308518+0000","last_became_peered":"2024-09-21T12:20:15.308518+0000","last_unstale":"2024-09-21T12:26:22.040202+0000","last_undegraded":"2024-09-21T12:26:22.040202+0000","last_fullsized":"2024-09-21T12:26:22.040202+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:16.410067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:16.410067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:24.383217+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":390,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.027609+0000","last_change":"2024-09-21T12:25:42.615523+0000","last_active":"2024-09-21T12:26:22.027609+0000","last_peered":"2024-09-21T12:26:22.027609+0000","last_clean":"2024-09-21T12:26:22.027609+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:26:22.027609+0000","last_undegraded":"2024-09-21T12:26:22.027609+0000","last_fullsized":"2024-09-21T12:26:22.027609+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:25:42.615446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:42.615446+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:06.344897+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":293,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.026676+0000","last_change":"2024-09-21T12:22:51.987032+0000","last_active":"2024-09-21T12:26:22.026676+0000","last_peered":"2024-09-21T12:26:22.026676+0000","last_clean":"2024-09-21T12:26:22.026676+0000","last_became_active":"2024-09-21T12:20:21.315801+0000","last_became_peered":"2024-09-21T12:20:21.315801+0000","last_unstale":"2024-09-21T12:26:22.026676+0000","last_undegraded":"2024-09-21T12:26:22.026676+0000","last_fullsized":"2024-09-21T12:26:22.026676+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:51.986953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:51.986953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:22.782321+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"200'8445","reported_seq":8536,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:27.068486+0000","last_change":"2024-09-21T12:24:46.324384+0000","last_active":"2024-09-21T12:26:27.068486+0000","last_peered":"2024-09-21T12:26:27.068486+0000","last_clean":"2024-09-21T12:26:27.068486+0000","last_became_active":"2024-09-21T12:24:46.323283+0000","last_became_peered":"2024-09-21T12:24:46.323283+0000","last_unstale":"2024-09-21T12:26:27.068486+0000","last_undegraded":"2024-09-21T12:26:27.068486+0000","last_fullsized":"2024-09-21T12:26:27.068486+0000","mapping_epoch":160,"log_start":"180'4251","ondisk_log_start":"180'4251","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4194,"log_dups_size":0,"ondisk_log_size":4194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":519110656,"num_objects":7921,"num_object_clones":0,"num_object_copies":23763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8445,"num_write_kb":523712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"76'79","reported_seq":302,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037831+0000","last_change":"2024-09-21T12:24:53.927007+0000","last_active":"2024-09-21T12:26:22.037831+0000","last_peered":"2024-09-21T12:26:22.037831+0000","last_clean":"2024-09-21T12:26:22.037831+0000","last_became_active":"2024-09-21T12:22:17.083629+0000","last_became_peered":"2024-09-21T12:22:17.083629+0000","last_unstale":"2024-09-21T12:26:22.037831+0000","last_undegraded":"2024-09-21T12:26:22.037831+0000","last_fullsized":"2024-09-21T12:26:22.037831+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:24:53.926958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:24:53.926958+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":344,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.028781+0000","last_change":"2024-09-21T12:26:14.834328+0000","last_active":"2024-09-21T12:26:22.028781+0000","last_peered":"2024-09-21T12:26:22.028781+0000","last_clean":"2024-09-21T12:26:22.028781+0000","last_became_active":"2024-09-21T12:20:17.693067+0000","last_became_peered":"2024-09-21T12:20:17.693067+0000","last_unstale":"2024-09-21T12:26:22.028781+0000","last_undegraded":"2024-09-21T12:26:22.028781+0000","last_fullsized":"2024-09-21T12:26:22.028781+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-21T12:26:14.834236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:14.834236+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:34.377077+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":304,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038265+0000","last_change":"2024-09-21T12:26:12.131784+0000","last_active":"2024-09-21T12:26:22.038265+0000","last_peered":"2024-09-21T12:26:22.038265+0000","last_clean":"2024-09-21T12:26:22.038265+0000","last_became_active":"2024-09-21T12:20:15.308441+0000","last_became_peered":"2024-09-21T12:20:15.308441+0000","last_unstale":"2024-09-21T12:26:22.038265+0000","last_undegraded":"2024-09-21T12:26:22.038265+0000","last_fullsized":"2024-09-21T12:26:22.038265+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:12.131721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:12.131721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:19.426420+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":303,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036125+0000","last_change":"2024-09-21T12:26:12.887076+0000","last_active":"2024-09-21T12:26:22.036125+0000","last_peered":"2024-09-21T12:26:22.036125+0000","last_clean":"2024-09-21T12:26:22.036125+0000","last_became_active":"2024-09-21T12:20:19.309598+0000","last_became_peered":"2024-09-21T12:20:19.309598+0000","last_unstale":"2024-09-21T12:26:22.036125+0000","last_undegraded":"2024-09-21T12:26:22.036125+0000","last_fullsized":"2024-09-21T12:26:22.036125+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:12.887007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:12.887007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:22.153550+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":292,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.035790+0000","last_change":"2024-09-21T12:25:44.918356+0000","last_active":"2024-09-21T12:26:22.035790+0000","last_peered":"2024-09-21T12:26:22.035790+0000","last_clean":"2024-09-21T12:26:22.035790+0000","last_became_active":"2024-09-21T12:20:21.592828+0000","last_became_peered":"2024-09-21T12:20:21.592828+0000","last_unstale":"2024-09-21T12:26:22.035790+0000","last_undegraded":"2024-09-21T12:26:22.035790+0000","last_fullsized":"2024-09-21T12:26:22.035790+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:44.918276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:44.918276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:07.775801+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"200'8440","reported_seq":8531,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:26.590684+0000","last_change":"2024-09-21T12:24:46.325092+0000","last_active":"2024-09-21T12:26:26.590684+0000","last_peered":"2024-09-21T12:26:26.590684+0000","last_clean":"2024-09-21T12:26:26.590684+0000","last_became_active":"2024-09-21T12:24:46.324740+0000","last_became_peered":"2024-09-21T12:24:46.324740+0000","last_unstale":"2024-09-21T12:26:26.590684+0000","last_undegraded":"2024-09-21T12:26:26.590684+0000","last_fullsized":"2024-09-21T12:26:26.590684+0000","mapping_epoch":160,"log_start":"178'4000","ondisk_log_start":"178'4000","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4440,"log_dups_size":0,"ondisk_log_size":4440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":522059776,"num_objects":7966,"num_object_clones":0,"num_object_copies":23898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8440,"num_write_kb":524992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"76'79","reported_seq":205,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039260+0000","last_change":"2024-09-21T12:23:21.346117+0000","last_active":"2024-09-21T12:26:22.039260+0000","last_peered":"2024-09-21T12:26:22.039260+0000","last_clean":"2024-09-21T12:26:22.039260+0000","last_became_active":"2024-09-21T12:22:17.084732+0000","last_became_peered":"2024-09-21T12:22:17.084732+0000","last_unstale":"2024-09-21T12:26:22.039260+0000","last_undegraded":"2024-09-21T12:26:22.039260+0000","last_fullsized":"2024-09-21T12:26:22.039260+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:23:21.346046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:23:21.346046+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:04.385057+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":423,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036051+0000","last_change":"2024-09-21T12:26:09.082395+0000","last_active":"2024-09-21T12:26:22.036051+0000","last_peered":"2024-09-21T12:26:22.036051+0000","last_clean":"2024-09-21T12:26:22.036051+0000","last_became_active":"2024-09-21T12:20:17.307769+0000","last_became_peered":"2024-09-21T12:20:17.307769+0000","last_unstale":"2024-09-21T12:26:22.036051+0000","last_undegraded":"2024-09-21T12:26:22.036051+0000","last_fullsized":"2024-09-21T12:26:22.036051+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-21T12:26:09.082314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:09.082314+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:17.369834+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":228,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.029125+0000","last_change":"2024-09-21T12:22:50.922961+0000","last_active":"2024-09-21T12:26:22.029125+0000","last_peered":"2024-09-21T12:26:22.029125+0000","last_clean":"2024-09-21T12:26:22.029125+0000","last_became_active":"2024-09-21T12:22:50.922002+0000","last_became_peered":"2024-09-21T12:22:50.922002+0000","last_unstale":"2024-09-21T12:26:22.029125+0000","last_undegraded":"2024-09-21T12:26:22.029125+0000","last_fullsized":"2024-09-21T12:26:22.029125+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:05.507899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:05.507899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:50.702651+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":313,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.040204+0000","last_change":"2024-09-21T12:21:20.045824+0000","last_active":"2024-09-21T12:26:22.040204+0000","last_peered":"2024-09-21T12:26:22.040204+0000","last_clean":"2024-09-21T12:26:22.040204+0000","last_became_active":"2024-09-21T12:20:14.287998+0000","last_became_peered":"2024-09-21T12:20:14.287998+0000","last_unstale":"2024-09-21T12:26:22.040204+0000","last_undegraded":"2024-09-21T12:26:22.040204+0000","last_fullsized":"2024-09-21T12:26:22.040204+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T12:21:20.045755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.045755+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:05.385454+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011011149999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":325,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.028157+0000","last_change":"2024-09-21T12:24:53.112915+0000","last_active":"2024-09-21T12:26:22.028157+0000","last_peered":"2024-09-21T12:26:22.028157+0000","last_clean":"2024-09-21T12:26:22.028157+0000","last_became_active":"2024-09-21T12:22:50.921488+0000","last_became_peered":"2024-09-21T12:22:50.921488+0000","last_unstale":"2024-09-21T12:26:22.028157+0000","last_undegraded":"2024-09-21T12:26:22.028157+0000","last_fullsized":"2024-09-21T12:26:22.028157+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:53.112868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:53.112868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:05.952435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":319,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.031431+0000","last_change":"2024-09-21T12:26:10.435410+0000","last_active":"2024-09-21T12:26:22.031431+0000","last_peered":"2024-09-21T12:26:22.031431+0000","last_clean":"2024-09-21T12:26:22.031431+0000","last_became_active":"2024-09-21T12:22:50.917558+0000","last_became_peered":"2024-09-21T12:22:50.917558+0000","last_unstale":"2024-09-21T12:26:22.031431+0000","last_undegraded":"2024-09-21T12:26:22.031431+0000","last_fullsized":"2024-09-21T12:26:22.031431+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:10.435349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:10.435349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:19.570759+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"200'8444","reported_seq":8535,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:27.125266+0000","last_change":"2024-09-21T12:24:46.323849+0000","last_active":"2024-09-21T12:26:27.125266+0000","last_peered":"2024-09-21T12:26:27.125266+0000","last_clean":"2024-09-21T12:26:27.125266+0000","last_became_active":"2024-09-21T12:24:46.323306+0000","last_became_peered":"2024-09-21T12:24:46.323306+0000","last_unstale":"2024-09-21T12:26:27.125266+0000","last_undegraded":"2024-09-21T12:26:27.125266+0000","last_fullsized":"2024-09-21T12:26:27.125266+0000","mapping_epoch":160,"log_start":"180'4300","ondisk_log_start":"180'4300","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4144,"log_dups_size":0,"ondisk_log_size":4144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":517996544,"num_objects":7904,"num_object_clones":0,"num_object_copies":23712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8444,"num_write_kb":523136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'79","reported_seq":388,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036405+0000","last_change":"2024-09-21T12:25:03.877681+0000","last_active":"2024-09-21T12:26:22.036405+0000","last_peered":"2024-09-21T12:26:22.036405+0000","last_clean":"2024-09-21T12:26:22.036405+0000","last_became_active":"2024-09-21T12:22:50.921709+0000","last_became_peered":"2024-09-21T12:22:50.921709+0000","last_unstale":"2024-09-21T12:26:22.036405+0000","last_undegraded":"2024-09-21T12:26:22.036405+0000","last_fullsized":"2024-09-21T12:26:22.036405+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:25:03.877610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:25:03.877610+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:26.706650+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":335,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.043559+0000","last_change":"2024-09-21T12:25:45.962538+0000","last_active":"2024-09-21T12:26:22.043559+0000","last_peered":"2024-09-21T12:26:22.043559+0000","last_clean":"2024-09-21T12:26:22.043559+0000","last_became_active":"2024-09-21T12:20:17.297374+0000","last_became_peered":"2024-09-21T12:20:17.297374+0000","last_unstale":"2024-09-21T12:26:22.043559+0000","last_undegraded":"2024-09-21T12:26:22.043559+0000","last_fullsized":"2024-09-21T12:26:22.043559+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-21T12:25:45.962433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:45.962433+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:08.718722+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":313,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036059+0000","last_change":"2024-09-21T12:23:59.367953+0000","last_active":"2024-09-21T12:26:22.036059+0000","last_peered":"2024-09-21T12:26:22.036059+0000","last_clean":"2024-09-21T12:26:22.036059+0000","last_became_active":"2024-09-21T12:20:15.308356+0000","last_became_peered":"2024-09-21T12:20:15.308356+0000","last_unstale":"2024-09-21T12:26:22.036059+0000","last_undegraded":"2024-09-21T12:26:22.036059+0000","last_fullsized":"2024-09-21T12:26:22.036059+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:59.367778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:23:59.367778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":329,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037122+0000","last_change":"2024-09-21T12:26:14.109964+0000","last_active":"2024-09-21T12:26:22.037122+0000","last_peered":"2024-09-21T12:26:22.037122+0000","last_clean":"2024-09-21T12:26:22.037122+0000","last_became_active":"2024-09-21T12:20:14.288358+0000","last_became_peered":"2024-09-21T12:20:14.288358+0000","last_unstale":"2024-09-21T12:26:22.037122+0000","last_undegraded":"2024-09-21T12:26:22.037122+0000","last_fullsized":"2024-09-21T12:26:22.037122+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:26:14.109886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:26:14.109886+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:31.748840+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.0019042449999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":304,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038917+0000","last_change":"2024-09-21T12:25:46.181836+0000","last_active":"2024-09-21T12:26:22.038917+0000","last_peered":"2024-09-21T12:26:22.038917+0000","last_clean":"2024-09-21T12:26:22.038917+0000","last_became_active":"2024-09-21T12:20:19.309724+0000","last_became_peered":"2024-09-21T12:20:19.309724+0000","last_unstale":"2024-09-21T12:26:22.038917+0000","last_undegraded":"2024-09-21T12:26:22.038917+0000","last_fullsized":"2024-09-21T12:26:22.038917+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:46.181531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:46.181531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:11.368016+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":283,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037727+0000","last_change":"2024-09-21T12:20:21.593267+0000","last_active":"2024-09-21T12:26:22.037727+0000","last_peered":"2024-09-21T12:26:22.037727+0000","last_clean":"2024-09-21T12:26:22.037727+0000","last_became_active":"2024-09-21T12:20:21.592982+0000","last_became_peered":"2024-09-21T12:20:21.592982+0000","last_unstale":"2024-09-21T12:26:22.037727+0000","last_undegraded":"2024-09-21T12:26:22.037727+0000","last_fullsized":"2024-09-21T12:26:22.037727+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:22.831735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"200'8300","reported_seq":8391,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:26.840469+0000","last_change":"2024-09-21T12:24:46.326200+0000","last_active":"2024-09-21T12:26:26.840469+0000","last_peered":"2024-09-21T12:26:26.840469+0000","last_clean":"2024-09-21T12:26:26.840469+0000","last_became_active":"2024-09-21T12:24:46.325838+0000","last_became_peered":"2024-09-21T12:24:46.325838+0000","last_unstale":"2024-09-21T12:26:26.840469+0000","last_undegraded":"2024-09-21T12:26:26.840469+0000","last_fullsized":"2024-09-21T12:26:26.840469+0000","mapping_epoch":160,"log_start":"179'4000","ondisk_log_start":"179'4000","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4300,"log_dups_size":0,"ondisk_log_size":4300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":514588672,"num_objects":7852,"num_object_clones":0,"num_object_copies":23556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8300,"num_write_kb":516864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"76'79","reported_seq":227,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.047286+0000","last_change":"2024-09-21T12:23:57.442430+0000","last_active":"2024-09-21T12:26:22.047286+0000","last_peered":"2024-09-21T12:26:22.047286+0000","last_clean":"2024-09-21T12:26:22.047286+0000","last_became_active":"2024-09-21T12:22:50.912088+0000","last_became_peered":"2024-09-21T12:22:50.912088+0000","last_unstale":"2024-09-21T12:26:22.047286+0000","last_undegraded":"2024-09-21T12:26:22.047286+0000","last_fullsized":"2024-09-21T12:26:22.047286+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:23:57.442284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:23:57.442284+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":318,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.028932+0000","last_change":"2024-09-21T12:22:49.933621+0000","last_active":"2024-09-21T12:26:22.028932+0000","last_peered":"2024-09-21T12:26:22.028932+0000","last_clean":"2024-09-21T12:26:22.028932+0000","last_became_active":"2024-09-21T12:20:17.299325+0000","last_became_peered":"2024-09-21T12:20:17.299325+0000","last_unstale":"2024-09-21T12:26:22.028932+0000","last_undegraded":"2024-09-21T12:26:22.028932+0000","last_fullsized":"2024-09-21T12:26:22.028932+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:22:49.933382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:49.933382+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:23.687742+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":333,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.029835+0000","last_change":"2024-09-21T12:24:48.073954+0000","last_active":"2024-09-21T12:26:22.029835+0000","last_peered":"2024-09-21T12:26:22.029835+0000","last_clean":"2024-09-21T12:26:22.029835+0000","last_became_active":"2024-09-21T12:22:50.923796+0000","last_became_peered":"2024-09-21T12:22:50.923796+0000","last_unstale":"2024-09-21T12:26:22.029835+0000","last_undegraded":"2024-09-21T12:26:22.029835+0000","last_fullsized":"2024-09-21T12:26:22.029835+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:48.073424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:48.073424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:58.857724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":689,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:25.449157+0000","last_change":"2024-09-21T12:25:35.819787+0000","last_active":"2024-09-21T12:26:25.449157+0000","last_peered":"2024-09-21T12:26:25.449157+0000","last_clean":"2024-09-21T12:26:25.449157+0000","last_became_active":"2024-09-21T12:20:14.287543+0000","last_became_peered":"2024-09-21T12:20:14.287543+0000","last_unstale":"2024-09-21T12:26:25.449157+0000","last_undegraded":"2024-09-21T12:26:25.449157+0000","last_fullsized":"2024-09-21T12:26:25.449157+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-21T12:25:35.819707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:25:35.819707+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:05.641221+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":366,"num_read_kb":361,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":303,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.040111+0000","last_change":"2024-09-21T12:26:11.422340+0000","last_active":"2024-09-21T12:26:22.040111+0000","last_peered":"2024-09-21T12:26:22.040111+0000","last_clean":"2024-09-21T12:26:22.040111+0000","last_became_active":"2024-09-21T12:20:19.314217+0000","last_became_peered":"2024-09-21T12:20:19.314217+0000","last_unstale":"2024-09-21T12:26:22.040111+0000","last_undegraded":"2024-09-21T12:26:22.040111+0000","last_fullsized":"2024-09-21T12:26:22.040111+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:11.422293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:11.422293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:13.710870+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":308,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.035799+0000","last_change":"2024-09-21T12:25:38.882984+0000","last_active":"2024-09-21T12:26:22.035799+0000","last_peered":"2024-09-21T12:26:22.035799+0000","last_clean":"2024-09-21T12:26:22.035799+0000","last_became_active":"2024-09-21T12:20:21.311188+0000","last_became_peered":"2024-09-21T12:20:21.311188+0000","last_unstale":"2024-09-21T12:26:22.035799+0000","last_undegraded":"2024-09-21T12:26:22.035799+0000","last_fullsized":"2024-09-21T12:26:22.035799+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:38.882915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:38.882915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:00.415003+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"200'8489","reported_seq":8580,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:26.612912+0000","last_change":"2024-09-21T12:24:46.318053+0000","last_active":"2024-09-21T12:26:26.612912+0000","last_peered":"2024-09-21T12:26:26.612912+0000","last_clean":"2024-09-21T12:26:26.612912+0000","last_became_active":"2024-09-21T12:24:46.317644+0000","last_became_peered":"2024-09-21T12:24:46.317644+0000","last_unstale":"2024-09-21T12:26:26.612912+0000","last_undegraded":"2024-09-21T12:26:26.612912+0000","last_fullsized":"2024-09-21T12:26:26.612912+0000","mapping_epoch":160,"log_start":"177'3800","ondisk_log_start":"177'3800","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4689,"log_dups_size":0,"ondisk_log_size":4689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:57.982352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":527499264,"num_objects":8049,"num_object_clones":0,"num_object_copies":24147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8489,"num_write_kb":529216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"121'19","reported_seq":479,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038405+0000","last_change":"2024-09-21T12:24:27.932976+0000","last_active":"2024-09-21T12:26:22.038405+0000","last_peered":"2024-09-21T12:26:22.038405+0000","last_clean":"2024-09-21T12:26:22.038405+0000","last_became_active":"2024-09-21T12:22:50.921690+0000","last_became_peered":"2024-09-21T12:22:50.921690+0000","last_unstale":"2024-09-21T12:26:22.038405+0000","last_undegraded":"2024-09-21T12:26:22.038405+0000","last_fullsized":"2024-09-21T12:26:22.038405+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'19","last_scrub_stamp":"2024-09-21T12:24:27.932891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:27.932891+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:57.001434+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":316,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.035497+0000","last_change":"2024-09-21T12:26:12.394327+0000","last_active":"2024-09-21T12:26:22.035497+0000","last_peered":"2024-09-21T12:26:22.035497+0000","last_clean":"2024-09-21T12:26:22.035497+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:26:22.035497+0000","last_undegraded":"2024-09-21T12:26:22.035497+0000","last_fullsized":"2024-09-21T12:26:22.035497+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:26:12.394247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:12.394247+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:12.954462+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":368,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036124+0000","last_change":"2024-09-21T12:24:31.089221+0000","last_active":"2024-09-21T12:26:22.036124+0000","last_peered":"2024-09-21T12:26:22.036124+0000","last_clean":"2024-09-21T12:26:22.036124+0000","last_became_active":"2024-09-21T12:22:15.049592+0000","last_became_peered":"2024-09-21T12:22:15.049592+0000","last_unstale":"2024-09-21T12:26:22.036124+0000","last_undegraded":"2024-09-21T12:26:22.036124+0000","last_fullsized":"2024-09-21T12:26:22.036124+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:24:31.089155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:24:31.089155+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":287,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037524+0000","last_change":"2024-09-21T12:20:19.310280+0000","last_active":"2024-09-21T12:26:22.037524+0000","last_peered":"2024-09-21T12:26:22.037524+0000","last_clean":"2024-09-21T12:26:22.037524+0000","last_became_active":"2024-09-21T12:20:19.309859+0000","last_became_peered":"2024-09-21T12:20:19.309859+0000","last_unstale":"2024-09-21T12:26:22.037524+0000","last_undegraded":"2024-09-21T12:26:22.037524+0000","last_fullsized":"2024-09-21T12:26:22.037524+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:28.753163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":328,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.040007+0000","last_change":"2024-09-21T12:25:32.465748+0000","last_active":"2024-09-21T12:26:22.040007+0000","last_peered":"2024-09-21T12:26:22.040007+0000","last_clean":"2024-09-21T12:26:22.040007+0000","last_became_active":"2024-09-21T12:22:50.913493+0000","last_became_peered":"2024-09-21T12:22:50.913493+0000","last_unstale":"2024-09-21T12:26:22.040007+0000","last_undegraded":"2024-09-21T12:26:22.040007+0000","last_fullsized":"2024-09-21T12:26:22.040007+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:32.465700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:32.465700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:57.893144+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"200'8305","reported_seq":8396,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:27.233796+0000","last_change":"2024-09-21T12:24:46.315286+0000","last_active":"2024-09-21T12:26:27.233796+0000","last_peered":"2024-09-21T12:26:27.233796+0000","last_clean":"2024-09-21T12:26:27.233796+0000","last_became_active":"2024-09-21T12:24:46.314957+0000","last_became_peered":"2024-09-21T12:24:46.314957+0000","last_unstale":"2024-09-21T12:26:27.233796+0000","last_undegraded":"2024-09-21T12:26:27.233796+0000","last_fullsized":"2024-09-21T12:26:27.233796+0000","mapping_epoch":160,"log_start":"179'4000","ondisk_log_start":"179'4000","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4305,"log_dups_size":0,"ondisk_log_size":4305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:57.838388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":514654208,"num_objects":7853,"num_object_clones":0,"num_object_copies":23559,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8305,"num_write_kb":517056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":318,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038025+0000","last_change":"2024-09-21T12:26:15.952570+0000","last_active":"2024-09-21T12:26:22.038025+0000","last_peered":"2024-09-21T12:26:22.038025+0000","last_clean":"2024-09-21T12:26:22.038025+0000","last_became_active":"2024-09-21T12:23:46.885802+0000","last_became_peered":"2024-09-21T12:23:46.885802+0000","last_unstale":"2024-09-21T12:26:22.038025+0000","last_undegraded":"2024-09-21T12:26:22.038025+0000","last_fullsized":"2024-09-21T12:26:22.038025+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:26:15.952496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:15.952496+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:37.337083+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":228,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036981+0000","last_change":"2024-09-21T12:22:50.908380+0000","last_active":"2024-09-21T12:26:22.036981+0000","last_peered":"2024-09-21T12:26:22.036981+0000","last_clean":"2024-09-21T12:26:22.036981+0000","last_became_active":"2024-09-21T12:22:50.908060+0000","last_became_peered":"2024-09-21T12:22:50.908060+0000","last_unstale":"2024-09-21T12:26:22.036981+0000","last_undegraded":"2024-09-21T12:26:22.036981+0000","last_fullsized":"2024-09-21T12:26:22.036981+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:05.892153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:05.892153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:26.966868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":324,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039897+0000","last_change":"2024-09-21T12:23:19.372762+0000","last_active":"2024-09-21T12:26:22.039897+0000","last_peered":"2024-09-21T12:26:22.039897+0000","last_clean":"2024-09-21T12:26:22.039897+0000","last_became_active":"2024-09-21T12:22:50.915081+0000","last_became_peered":"2024-09-21T12:22:50.915081+0000","last_unstale":"2024-09-21T12:26:22.039897+0000","last_undegraded":"2024-09-21T12:26:22.039897+0000","last_fullsized":"2024-09-21T12:26:22.039897+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:19.372719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:19.372719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:03.351727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":290,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.030674+0000","last_change":"2024-09-21T12:23:46.282792+0000","last_active":"2024-09-21T12:26:22.030674+0000","last_peered":"2024-09-21T12:26:22.030674+0000","last_clean":"2024-09-21T12:26:22.030674+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:26:22.030674+0000","last_undegraded":"2024-09-21T12:26:22.030674+0000","last_fullsized":"2024-09-21T12:26:22.030674+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:46.282661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:46.282661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"200'8476","reported_seq":8567,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:26.820214+0000","last_change":"2024-09-21T12:24:46.323690+0000","last_active":"2024-09-21T12:26:26.820214+0000","last_peered":"2024-09-21T12:26:26.820214+0000","last_clean":"2024-09-21T12:26:26.820214+0000","last_became_active":"2024-09-21T12:24:46.323402+0000","last_became_peered":"2024-09-21T12:24:46.323402+0000","last_unstale":"2024-09-21T12:26:26.820214+0000","last_undegraded":"2024-09-21T12:26:26.820214+0000","last_fullsized":"2024-09-21T12:26:26.820214+0000","mapping_epoch":160,"log_start":"179'4200","ondisk_log_start":"179'4200","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4276,"log_dups_size":0,"ondisk_log_size":4276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:01.998231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":522452992,"num_objects":7972,"num_object_clones":0,"num_object_copies":23916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8476,"num_write_kb":526336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"76'79","reported_seq":219,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039307+0000","last_change":"2024-09-21T12:24:28.372308+0000","last_active":"2024-09-21T12:26:22.039307+0000","last_peered":"2024-09-21T12:26:22.039307+0000","last_clean":"2024-09-21T12:26:22.039307+0000","last_became_active":"2024-09-21T12:22:17.087245+0000","last_became_peered":"2024-09-21T12:22:17.087245+0000","last_unstale":"2024-09-21T12:26:22.039307+0000","last_undegraded":"2024-09-21T12:26:22.039307+0000","last_fullsized":"2024-09-21T12:26:22.039307+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:24:28.372230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:24:28.372230+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":353,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037769+0000","last_change":"2024-09-21T12:25:42.183260+0000","last_active":"2024-09-21T12:26:22.037769+0000","last_peered":"2024-09-21T12:26:22.037769+0000","last_clean":"2024-09-21T12:26:22.037769+0000","last_became_active":"2024-09-21T12:23:46.897197+0000","last_became_peered":"2024-09-21T12:23:46.897197+0000","last_unstale":"2024-09-21T12:26:22.037769+0000","last_undegraded":"2024-09-21T12:26:22.037769+0000","last_fullsized":"2024-09-21T12:26:22.037769+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-21T12:25:42.183169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:42.183169+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:03.947737+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":340,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036459+0000","last_change":"2024-09-21T12:26:13.918157+0000","last_active":"2024-09-21T12:26:22.036459+0000","last_peered":"2024-09-21T12:26:22.036459+0000","last_clean":"2024-09-21T12:26:22.036459+0000","last_became_active":"2024-09-21T12:22:50.906886+0000","last_became_peered":"2024-09-21T12:22:50.906886+0000","last_unstale":"2024-09-21T12:26:22.036459+0000","last_undegraded":"2024-09-21T12:26:22.036459+0000","last_fullsized":"2024-09-21T12:26:22.036459+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:13.918088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:13.918088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:43.184195+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":317,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.035862+0000","last_change":"2024-09-21T12:23:17.434377+0000","last_active":"2024-09-21T12:26:22.035862+0000","last_peered":"2024-09-21T12:26:22.035862+0000","last_clean":"2024-09-21T12:26:22.035862+0000","last_became_active":"2024-09-21T12:22:50.927438+0000","last_became_peered":"2024-09-21T12:22:50.927438+0000","last_unstale":"2024-09-21T12:26:22.035862+0000","last_undegraded":"2024-09-21T12:26:22.035862+0000","last_fullsized":"2024-09-21T12:26:22.035862+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:17.434261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:17.434261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:54.989691+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":313,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037872+0000","last_change":"2024-09-21T12:25:39.079146+0000","last_active":"2024-09-21T12:26:22.037872+0000","last_peered":"2024-09-21T12:26:22.037872+0000","last_clean":"2024-09-21T12:26:22.037872+0000","last_became_active":"2024-09-21T12:22:50.913898+0000","last_became_peered":"2024-09-21T12:22:50.913898+0000","last_unstale":"2024-09-21T12:26:22.037872+0000","last_undegraded":"2024-09-21T12:26:22.037872+0000","last_fullsized":"2024-09-21T12:26:22.037872+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:39.079015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:39.079015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:50.881520+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"200'8465","reported_seq":8556,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:27.126582+0000","last_change":"2024-09-21T12:24:46.315895+0000","last_active":"2024-09-21T12:26:27.126582+0000","last_peered":"2024-09-21T12:26:27.126582+0000","last_clean":"2024-09-21T12:26:27.126582+0000","last_became_active":"2024-09-21T12:24:46.315601+0000","last_became_peered":"2024-09-21T12:24:46.315601+0000","last_unstale":"2024-09-21T12:26:27.126582+0000","last_undegraded":"2024-09-21T12:26:27.126582+0000","last_fullsized":"2024-09-21T12:26:27.126582+0000","mapping_epoch":160,"log_start":"180'4300","ondisk_log_start":"180'4300","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4165,"log_dups_size":0,"ondisk_log_size":4165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520290304,"num_objects":7939,"num_object_clones":0,"num_object_copies":23817,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8465,"num_write_kb":524928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"76'79","reported_seq":365,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036059+0000","last_change":"2024-09-21T12:25:43.825573+0000","last_active":"2024-09-21T12:26:22.036059+0000","last_peered":"2024-09-21T12:26:22.036059+0000","last_clean":"2024-09-21T12:26:22.036059+0000","last_became_active":"2024-09-21T12:22:17.072872+0000","last_became_peered":"2024-09-21T12:22:17.072872+0000","last_unstale":"2024-09-21T12:26:22.036059+0000","last_undegraded":"2024-09-21T12:26:22.036059+0000","last_fullsized":"2024-09-21T12:26:22.036059+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:25:43.825511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:25:43.825511+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:08.940139+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":167,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036952+0000","last_change":"2024-09-21T12:26:10.141012+0000","last_active":"2024-09-21T12:26:22.036952+0000","last_peered":"2024-09-21T12:26:22.036952+0000","last_clean":"2024-09-21T12:26:22.036952+0000","last_became_active":"2024-09-21T12:23:46.902260+0000","last_became_peered":"2024-09-21T12:23:46.902260+0000","last_unstale":"2024-09-21T12:26:22.036952+0000","last_undegraded":"2024-09-21T12:26:22.036952+0000","last_fullsized":"2024-09-21T12:26:22.036952+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:26:10.137029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:10.137029+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:19.970928+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":323,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.032982+0000","last_change":"2024-09-21T12:22:50.928240+0000","last_active":"2024-09-21T12:26:22.032982+0000","last_peered":"2024-09-21T12:26:22.032982+0000","last_clean":"2024-09-21T12:26:22.032982+0000","last_became_active":"2024-09-21T12:22:50.927946+0000","last_became_peered":"2024-09-21T12:22:50.927946+0000","last_unstale":"2024-09-21T12:26:22.032982+0000","last_undegraded":"2024-09-21T12:26:22.032982+0000","last_fullsized":"2024-09-21T12:26:22.032982+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:08.999015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:08.999015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:24.102761+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":395,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037721+0000","last_change":"2024-09-21T12:26:13.114174+0000","last_active":"2024-09-21T12:26:22.037721+0000","last_peered":"2024-09-21T12:26:22.037721+0000","last_clean":"2024-09-21T12:26:22.037721+0000","last_became_active":"2024-09-21T12:20:19.307924+0000","last_became_peered":"2024-09-21T12:20:19.307924+0000","last_unstale":"2024-09-21T12:26:22.037721+0000","last_undegraded":"2024-09-21T12:26:22.037721+0000","last_fullsized":"2024-09-21T12:26:22.037721+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:26:13.113863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:13.113863+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:23.394893+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":236,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038472+0000","last_change":"2024-09-21T12:23:56.471960+0000","last_active":"2024-09-21T12:26:22.038472+0000","last_peered":"2024-09-21T12:26:22.038472+0000","last_clean":"2024-09-21T12:26:22.038472+0000","last_became_active":"2024-09-21T12:22:50.905470+0000","last_became_peered":"2024-09-21T12:22:50.905470+0000","last_unstale":"2024-09-21T12:26:22.038472+0000","last_undegraded":"2024-09-21T12:26:22.038472+0000","last_fullsized":"2024-09-21T12:26:22.038472+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:56.471116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:56.471116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:22.516139+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.7","version":"200'8333","reported_seq":8424,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:26.799911+0000","last_change":"2024-09-21T12:24:46.317724+0000","last_active":"2024-09-21T12:26:26.799911+0000","last_peered":"2024-09-21T12:26:26.799911+0000","last_clean":"2024-09-21T12:26:26.799911+0000","last_became_active":"2024-09-21T12:24:46.317468+0000","last_became_peered":"2024-09-21T12:24:46.317468+0000","last_unstale":"2024-09-21T12:26:26.799911+0000","last_undegraded":"2024-09-21T12:26:26.799911+0000","last_fullsized":"2024-09-21T12:26:26.799911+0000","mapping_epoch":160,"log_start":"179'4100","ondisk_log_start":"179'4100","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4233,"log_dups_size":0,"ondisk_log_size":4233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:01.017517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":517931008,"num_objects":7903,"num_object_clones":0,"num_object_copies":23709,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8333,"num_write_kb":519552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":383,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.028694+0000","last_change":"2024-09-21T12:25:00.237083+0000","last_active":"2024-09-21T12:26:22.028694+0000","last_peered":"2024-09-21T12:26:22.028694+0000","last_clean":"2024-09-21T12:26:22.028694+0000","last_became_active":"2024-09-21T12:22:50.922343+0000","last_became_peered":"2024-09-21T12:22:50.922343+0000","last_unstale":"2024-09-21T12:26:22.028694+0000","last_undegraded":"2024-09-21T12:26:22.028694+0000","last_fullsized":"2024-09-21T12:26:22.028694+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:25:00.236990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:00.236990+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:26.107161+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":319,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039938+0000","last_change":"2024-09-21T12:25:30.451939+0000","last_active":"2024-09-21T12:26:22.039938+0000","last_peered":"2024-09-21T12:26:22.039938+0000","last_clean":"2024-09-21T12:26:22.039938+0000","last_became_active":"2024-09-21T12:20:15.302262+0000","last_became_peered":"2024-09-21T12:20:15.302262+0000","last_unstale":"2024-09-21T12:26:22.039938+0000","last_undegraded":"2024-09-21T12:26:22.039938+0000","last_fullsized":"2024-09-21T12:26:22.039938+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:30.451876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:30.451876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:51.913189+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":333,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037402+0000","last_change":"2024-09-21T12:25:36.012066+0000","last_active":"2024-09-21T12:26:22.037402+0000","last_peered":"2024-09-21T12:26:22.037402+0000","last_clean":"2024-09-21T12:26:22.037402+0000","last_became_active":"2024-09-21T12:22:50.925974+0000","last_became_peered":"2024-09-21T12:22:50.925974+0000","last_unstale":"2024-09-21T12:26:22.037402+0000","last_undegraded":"2024-09-21T12:26:22.037402+0000","last_fullsized":"2024-09-21T12:26:22.037402+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:36.011984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:36.011984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:40.801014+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":292,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.029754+0000","last_change":"2024-09-21T12:23:28.121344+0000","last_active":"2024-09-21T12:26:22.029754+0000","last_peered":"2024-09-21T12:26:22.029754+0000","last_clean":"2024-09-21T12:26:22.029754+0000","last_became_active":"2024-09-21T12:20:21.311493+0000","last_became_peered":"2024-09-21T12:20:21.311493+0000","last_unstale":"2024-09-21T12:26:22.029754+0000","last_undegraded":"2024-09-21T12:26:22.029754+0000","last_fullsized":"2024-09-21T12:26:22.029754+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:28.121286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:28.121286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:24.681035+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"200'8435","reported_seq":8526,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:27.217779+0000","last_change":"2024-09-21T12:24:46.327091+0000","last_active":"2024-09-21T12:26:27.217779+0000","last_peered":"2024-09-21T12:26:27.217779+0000","last_clean":"2024-09-21T12:26:27.217779+0000","last_became_active":"2024-09-21T12:24:46.325838+0000","last_became_peered":"2024-09-21T12:24:46.325838+0000","last_unstale":"2024-09-21T12:26:27.217779+0000","last_undegraded":"2024-09-21T12:26:27.217779+0000","last_fullsized":"2024-09-21T12:26:27.217779+0000","mapping_epoch":160,"log_start":"179'4100","ondisk_log_start":"179'4100","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4335,"log_dups_size":0,"ondisk_log_size":4335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":517144576,"num_objects":7891,"num_object_clones":0,"num_object_copies":23673,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8435,"num_write_kb":522432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":338,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036212+0000","last_change":"2024-09-21T12:24:02.960807+0000","last_active":"2024-09-21T12:26:22.036212+0000","last_peered":"2024-09-21T12:26:22.036212+0000","last_clean":"2024-09-21T12:26:22.036212+0000","last_became_active":"2024-09-21T12:20:17.692867+0000","last_became_peered":"2024-09-21T12:20:17.692867+0000","last_unstale":"2024-09-21T12:26:22.036212+0000","last_undegraded":"2024-09-21T12:26:22.036212+0000","last_fullsized":"2024-09-21T12:26:22.036212+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-21T12:24:02.960733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:02.960733+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":303,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038040+0000","last_change":"2024-09-21T12:23:59.046000+0000","last_active":"2024-09-21T12:26:22.038040+0000","last_peered":"2024-09-21T12:26:22.038040+0000","last_clean":"2024-09-21T12:26:22.038040+0000","last_became_active":"2024-09-21T12:20:15.299333+0000","last_became_peered":"2024-09-21T12:20:15.299333+0000","last_unstale":"2024-09-21T12:26:22.038040+0000","last_undegraded":"2024-09-21T12:26:22.038040+0000","last_fullsized":"2024-09-21T12:26:22.038040+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:59.045927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:23:59.045927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:55.760821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":296,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037229+0000","last_change":"2024-09-21T12:24:34.060244+0000","last_active":"2024-09-21T12:26:22.037229+0000","last_peered":"2024-09-21T12:26:22.037229+0000","last_clean":"2024-09-21T12:26:22.037229+0000","last_became_active":"2024-09-21T12:20:19.306952+0000","last_became_peered":"2024-09-21T12:20:19.306952+0000","last_unstale":"2024-09-21T12:26:22.037229+0000","last_undegraded":"2024-09-21T12:26:22.037229+0000","last_fullsized":"2024-09-21T12:26:22.037229+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:34.060188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:34.060188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:50.161659+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":283,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036044+0000","last_change":"2024-09-21T12:20:21.314891+0000","last_active":"2024-09-21T12:26:22.036044+0000","last_peered":"2024-09-21T12:26:22.036044+0000","last_clean":"2024-09-21T12:26:22.036044+0000","last_became_active":"2024-09-21T12:20:21.314545+0000","last_became_peered":"2024-09-21T12:20:21.314545+0000","last_unstale":"2024-09-21T12:26:22.036044+0000","last_undegraded":"2024-09-21T12:26:22.036044+0000","last_fullsized":"2024-09-21T12:26:22.036044+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:49.780292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"200'8352","reported_seq":8454,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:26.641305+0000","last_change":"2024-09-21T12:25:49.210492+0000","last_active":"2024-09-21T12:26:26.641305+0000","last_peered":"2024-09-21T12:26:26.641305+0000","last_clean":"2024-09-21T12:26:26.641305+0000","last_became_active":"2024-09-21T12:24:46.324554+0000","last_became_peered":"2024-09-21T12:24:46.324554+0000","last_unstale":"2024-09-21T12:26:26.641305+0000","last_undegraded":"2024-09-21T12:26:26.641305+0000","last_fullsized":"2024-09-21T12:26:26.641305+0000","mapping_epoch":160,"log_start":"176'3700","ondisk_log_start":"176'3700","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":899,"log_size":4652,"log_dups_size":0,"ondisk_log_size":4652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":519962624,"num_objects":7934,"num_object_clones":0,"num_object_copies":23802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8352,"num_write_kb":521152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":311,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.035955+0000","last_change":"2024-09-21T12:24:54.445801+0000","last_active":"2024-09-21T12:26:22.035955+0000","last_peered":"2024-09-21T12:26:22.035955+0000","last_clean":"2024-09-21T12:26:22.035955+0000","last_became_active":"2024-09-21T12:22:50.910973+0000","last_became_peered":"2024-09-21T12:22:50.910973+0000","last_unstale":"2024-09-21T12:26:22.035955+0000","last_undegraded":"2024-09-21T12:26:22.035955+0000","last_fullsized":"2024-09-21T12:26:22.035955+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:24:54.445714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:54.445714+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":332,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038093+0000","last_change":"2024-09-21T12:25:44.800799+0000","last_active":"2024-09-21T12:26:22.038093+0000","last_peered":"2024-09-21T12:26:22.038093+0000","last_clean":"2024-09-21T12:26:22.038093+0000","last_became_active":"2024-09-21T12:22:50.927859+0000","last_became_peered":"2024-09-21T12:22:50.927859+0000","last_unstale":"2024-09-21T12:26:22.038093+0000","last_undegraded":"2024-09-21T12:26:22.038093+0000","last_fullsized":"2024-09-21T12:26:22.038093+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:44.800657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:44.800657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:49.068224+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":304,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036702+0000","last_change":"2024-09-21T12:25:40.148844+0000","last_active":"2024-09-21T12:26:22.036702+0000","last_peered":"2024-09-21T12:26:22.036702+0000","last_clean":"2024-09-21T12:26:22.036702+0000","last_became_active":"2024-09-21T12:20:19.312572+0000","last_became_peered":"2024-09-21T12:20:19.312572+0000","last_unstale":"2024-09-21T12:26:22.036702+0000","last_undegraded":"2024-09-21T12:26:22.036702+0000","last_fullsized":"2024-09-21T12:26:22.036702+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:40.148766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:40.148766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:07.648989+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":290,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.032652+0000","last_change":"2024-09-21T12:22:36.006177+0000","last_active":"2024-09-21T12:26:22.032652+0000","last_peered":"2024-09-21T12:26:22.032652+0000","last_clean":"2024-09-21T12:26:22.032652+0000","last_became_active":"2024-09-21T12:20:21.317223+0000","last_became_peered":"2024-09-21T12:20:21.317223+0000","last_unstale":"2024-09-21T12:26:22.032652+0000","last_undegraded":"2024-09-21T12:26:22.032652+0000","last_fullsized":"2024-09-21T12:26:22.032652+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:36.006109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:36.006109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:54.431499+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"200'8425","reported_seq":8516,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:26.610965+0000","last_change":"2024-09-21T12:24:46.323500+0000","last_active":"2024-09-21T12:26:26.610965+0000","last_peered":"2024-09-21T12:26:26.610965+0000","last_clean":"2024-09-21T12:26:26.610965+0000","last_became_active":"2024-09-21T12:24:46.323192+0000","last_became_peered":"2024-09-21T12:24:46.323192+0000","last_unstale":"2024-09-21T12:26:26.610965+0000","last_undegraded":"2024-09-21T12:26:26.610965+0000","last_fullsized":"2024-09-21T12:26:26.610965+0000","mapping_epoch":160,"log_start":"178'4000","ondisk_log_start":"178'4000","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4425,"log_dups_size":0,"ondisk_log_size":4425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":525533184,"num_objects":8019,"num_object_clones":0,"num_object_copies":24057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8425,"num_write_kb":526208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":340,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039658+0000","last_change":"2024-09-21T12:25:03.416100+0000","last_active":"2024-09-21T12:26:22.039658+0000","last_peered":"2024-09-21T12:26:22.039658+0000","last_clean":"2024-09-21T12:26:22.039658+0000","last_became_active":"2024-09-21T12:20:17.692870+0000","last_became_peered":"2024-09-21T12:20:17.692870+0000","last_unstale":"2024-09-21T12:26:22.039658+0000","last_undegraded":"2024-09-21T12:26:22.039658+0000","last_fullsized":"2024-09-21T12:26:22.039658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:25:03.416033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:03.416033+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":362,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038114+0000","last_change":"2024-09-21T12:24:29.054851+0000","last_active":"2024-09-21T12:26:22.038114+0000","last_peered":"2024-09-21T12:26:22.038114+0000","last_clean":"2024-09-21T12:26:22.038114+0000","last_became_active":"2024-09-21T12:22:50.927141+0000","last_became_peered":"2024-09-21T12:22:50.927141+0000","last_unstale":"2024-09-21T12:26:22.038114+0000","last_undegraded":"2024-09-21T12:26:22.038114+0000","last_fullsized":"2024-09-21T12:26:22.038114+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-21T12:24:29.054772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:29.054772+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:07.827845+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"121'11","reported_seq":411,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036013+0000","last_change":"2024-09-21T12:25:02.916145+0000","last_active":"2024-09-21T12:26:22.036013+0000","last_peered":"2024-09-21T12:26:22.036013+0000","last_clean":"2024-09-21T12:26:22.036013+0000","last_became_active":"2024-09-21T12:22:50.908371+0000","last_became_peered":"2024-09-21T12:22:50.908371+0000","last_unstale":"2024-09-21T12:26:22.036013+0000","last_undegraded":"2024-09-21T12:26:22.036013+0000","last_fullsized":"2024-09-21T12:26:22.036013+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:25:02.916090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:02.916090+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:31.534554+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":291,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039512+0000","last_change":"2024-09-21T12:22:45.223570+0000","last_active":"2024-09-21T12:26:22.039512+0000","last_peered":"2024-09-21T12:26:22.039512+0000","last_clean":"2024-09-21T12:26:22.039512+0000","last_became_active":"2024-09-21T12:20:21.592576+0000","last_became_peered":"2024-09-21T12:20:21.592576+0000","last_unstale":"2024-09-21T12:26:22.039512+0000","last_undegraded":"2024-09-21T12:26:22.039512+0000","last_fullsized":"2024-09-21T12:26:22.039512+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:45.223396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:45.223396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":327,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036891+0000","last_change":"2024-09-21T12:25:31.845118+0000","last_active":"2024-09-21T12:26:22.036891+0000","last_peered":"2024-09-21T12:26:22.036891+0000","last_clean":"2024-09-21T12:26:22.036891+0000","last_became_active":"2024-09-21T12:20:15.303480+0000","last_became_peered":"2024-09-21T12:20:15.303480+0000","last_unstale":"2024-09-21T12:26:22.036891+0000","last_undegraded":"2024-09-21T12:26:22.036891+0000","last_fullsized":"2024-09-21T12:26:22.036891+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:31.845066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:31.845066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:34.886910+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":307,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.030096+0000","last_change":"2024-09-21T12:22:36.999995+0000","last_active":"2024-09-21T12:26:22.030096+0000","last_peered":"2024-09-21T12:26:22.030096+0000","last_clean":"2024-09-21T12:26:22.030096+0000","last_became_active":"2024-09-21T12:20:17.694628+0000","last_became_peered":"2024-09-21T12:20:17.694628+0000","last_unstale":"2024-09-21T12:26:22.030096+0000","last_undegraded":"2024-09-21T12:26:22.030096+0000","last_fullsized":"2024-09-21T12:26:22.030096+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-21T12:22:36.999705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:36.999705+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:51.345667+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":311,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039758+0000","last_change":"2024-09-21T12:25:02.414377+0000","last_active":"2024-09-21T12:26:22.039758+0000","last_peered":"2024-09-21T12:26:22.039758+0000","last_clean":"2024-09-21T12:26:22.039758+0000","last_became_active":"2024-09-21T12:20:19.313882+0000","last_became_peered":"2024-09-21T12:20:19.313882+0000","last_unstale":"2024-09-21T12:26:22.039758+0000","last_undegraded":"2024-09-21T12:26:22.039758+0000","last_fullsized":"2024-09-21T12:26:22.039758+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:02.414298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:02.414298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":312,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037625+0000","last_change":"2024-09-21T12:24:00.066522+0000","last_active":"2024-09-21T12:26:22.037625+0000","last_peered":"2024-09-21T12:26:22.037625+0000","last_clean":"2024-09-21T12:26:22.037625+0000","last_became_active":"2024-09-21T12:22:50.904404+0000","last_became_peered":"2024-09-21T12:22:50.904404+0000","last_unstale":"2024-09-21T12:26:22.037625+0000","last_undegraded":"2024-09-21T12:26:22.037625+0000","last_fullsized":"2024-09-21T12:26:22.037625+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:00.066451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:00.066451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:01.771403+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":324,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.029014+0000","last_change":"2024-09-21T12:26:20.767526+0000","last_active":"2024-09-21T12:26:22.029014+0000","last_peered":"2024-09-21T12:26:22.029014+0000","last_clean":"2024-09-21T12:26:22.029014+0000","last_became_active":"2024-09-21T12:20:17.695898+0000","last_became_peered":"2024-09-21T12:20:17.695898+0000","last_unstale":"2024-09-21T12:26:22.029014+0000","last_undegraded":"2024-09-21T12:26:22.029014+0000","last_fullsized":"2024-09-21T12:26:22.029014+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:26:20.767293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:20.767293+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:26.917942+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":340,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036233+0000","last_change":"2024-09-21T12:26:16.907351+0000","last_active":"2024-09-21T12:26:22.036233+0000","last_peered":"2024-09-21T12:26:22.036233+0000","last_clean":"2024-09-21T12:26:22.036233+0000","last_became_active":"2024-09-21T12:22:50.927649+0000","last_became_peered":"2024-09-21T12:22:50.927649+0000","last_unstale":"2024-09-21T12:26:22.036233+0000","last_undegraded":"2024-09-21T12:26:22.036233+0000","last_fullsized":"2024-09-21T12:26:22.036233+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:16.907284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:16.907284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:30.534090+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":294,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.029889+0000","last_change":"2024-09-21T12:23:53.356954+0000","last_active":"2024-09-21T12:26:22.029889+0000","last_peered":"2024-09-21T12:26:22.029889+0000","last_clean":"2024-09-21T12:26:22.029889+0000","last_became_active":"2024-09-21T12:20:19.311947+0000","last_became_peered":"2024-09-21T12:20:19.311947+0000","last_unstale":"2024-09-21T12:26:22.029889+0000","last_undegraded":"2024-09-21T12:26:22.029889+0000","last_fullsized":"2024-09-21T12:26:22.029889+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:53.356896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:53.356896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:28.202782+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":337,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037217+0000","last_change":"2024-09-21T12:26:11.105716+0000","last_active":"2024-09-21T12:26:22.037217+0000","last_peered":"2024-09-21T12:26:22.037217+0000","last_clean":"2024-09-21T12:26:22.037217+0000","last_became_active":"2024-09-21T12:22:50.924931+0000","last_became_peered":"2024-09-21T12:22:50.924931+0000","last_unstale":"2024-09-21T12:26:22.037217+0000","last_undegraded":"2024-09-21T12:26:22.037217+0000","last_fullsized":"2024-09-21T12:26:22.037217+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:11.105583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:11.105583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:36.449727+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":340,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.040057+0000","last_change":"2024-09-21T12:26:18.481098+0000","last_active":"2024-09-21T12:26:22.040057+0000","last_peered":"2024-09-21T12:26:22.040057+0000","last_clean":"2024-09-21T12:26:22.040057+0000","last_became_active":"2024-09-21T12:23:46.898523+0000","last_became_peered":"2024-09-21T12:23:46.898523+0000","last_unstale":"2024-09-21T12:26:22.040057+0000","last_undegraded":"2024-09-21T12:26:22.040057+0000","last_fullsized":"2024-09-21T12:26:22.040057+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:26:18.480630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:18.480630+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:39.872336+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":312,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.027247+0000","last_change":"2024-09-21T12:24:25.931617+0000","last_active":"2024-09-21T12:26:22.027247+0000","last_peered":"2024-09-21T12:26:22.027247+0000","last_clean":"2024-09-21T12:26:22.027247+0000","last_became_active":"2024-09-21T12:20:15.297519+0000","last_became_peered":"2024-09-21T12:20:15.297519+0000","last_unstale":"2024-09-21T12:26:22.027247+0000","last_undegraded":"2024-09-21T12:26:22.027247+0000","last_fullsized":"2024-09-21T12:26:22.027247+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:25.931561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:25.931561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:54.835832+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":377,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.030916+0000","last_change":"2024-09-21T12:23:57.379033+0000","last_active":"2024-09-21T12:26:22.030916+0000","last_peered":"2024-09-21T12:26:22.030916+0000","last_clean":"2024-09-21T12:26:22.030916+0000","last_became_active":"2024-09-21T12:20:19.305364+0000","last_became_peered":"2024-09-21T12:20:19.305364+0000","last_unstale":"2024-09-21T12:26:22.030916+0000","last_undegraded":"2024-09-21T12:26:22.030916+0000","last_fullsized":"2024-09-21T12:26:22.030916+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:23:57.378959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:57.378959+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":235,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037961+0000","last_change":"2024-09-21T12:24:51.831043+0000","last_active":"2024-09-21T12:26:22.037961+0000","last_peered":"2024-09-21T12:26:22.037961+0000","last_clean":"2024-09-21T12:26:22.037961+0000","last_became_active":"2024-09-21T12:22:50.924770+0000","last_became_peered":"2024-09-21T12:22:50.924770+0000","last_unstale":"2024-09-21T12:26:22.037961+0000","last_undegraded":"2024-09-21T12:26:22.037961+0000","last_fullsized":"2024-09-21T12:26:22.037961+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:51.830994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:51.830994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:58.084812+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":321,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.031070+0000","last_change":"2024-09-21T12:24:50.727161+0000","last_active":"2024-09-21T12:26:22.031070+0000","last_peered":"2024-09-21T12:26:22.031070+0000","last_clean":"2024-09-21T12:26:22.031070+0000","last_became_active":"2024-09-21T12:20:17.693209+0000","last_became_peered":"2024-09-21T12:20:17.693209+0000","last_unstale":"2024-09-21T12:26:22.031070+0000","last_undegraded":"2024-09-21T12:26:22.031070+0000","last_fullsized":"2024-09-21T12:26:22.031070+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:24:50.727101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:50.727101+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":332,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037049+0000","last_change":"2024-09-21T12:25:36.808211+0000","last_active":"2024-09-21T12:26:22.037049+0000","last_peered":"2024-09-21T12:26:22.037049+0000","last_clean":"2024-09-21T12:26:22.037049+0000","last_became_active":"2024-09-21T12:22:50.908238+0000","last_became_peered":"2024-09-21T12:22:50.908238+0000","last_unstale":"2024-09-21T12:26:22.037049+0000","last_undegraded":"2024-09-21T12:26:22.037049+0000","last_fullsized":"2024-09-21T12:26:22.037049+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:36.808147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:36.808147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:50.893952+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":378,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039918+0000","last_change":"2024-09-21T12:22:46.216242+0000","last_active":"2024-09-21T12:26:22.039918+0000","last_peered":"2024-09-21T12:26:22.039918+0000","last_clean":"2024-09-21T12:26:22.039918+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:26:22.039918+0000","last_undegraded":"2024-09-21T12:26:22.039918+0000","last_fullsized":"2024-09-21T12:26:22.039918+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:22:46.216153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:46.216153+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":292,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.028564+0000","last_change":"2024-09-21T12:26:08.926904+0000","last_active":"2024-09-21T12:26:22.028564+0000","last_peered":"2024-09-21T12:26:22.028564+0000","last_clean":"2024-09-21T12:26:22.028564+0000","last_became_active":"2024-09-21T12:20:21.309385+0000","last_became_peered":"2024-09-21T12:20:21.309385+0000","last_unstale":"2024-09-21T12:26:22.028564+0000","last_undegraded":"2024-09-21T12:26:22.028564+0000","last_fullsized":"2024-09-21T12:26:22.028564+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:08.923458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:08.923458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:38.391592+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":291,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037349+0000","last_change":"2024-09-21T12:23:59.046758+0000","last_active":"2024-09-21T12:26:22.037349+0000","last_peered":"2024-09-21T12:26:22.037349+0000","last_clean":"2024-09-21T12:26:22.037349+0000","last_became_active":"2024-09-21T12:22:50.906473+0000","last_became_peered":"2024-09-21T12:22:50.906473+0000","last_unstale":"2024-09-21T12:26:22.037349+0000","last_undegraded":"2024-09-21T12:26:22.037349+0000","last_fullsized":"2024-09-21T12:26:22.037349+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:23:59.046704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:59.046704+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":228,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037671+0000","last_change":"2024-09-21T12:24:58.948698+0000","last_active":"2024-09-21T12:26:22.037671+0000","last_peered":"2024-09-21T12:26:22.037671+0000","last_clean":"2024-09-21T12:26:22.037671+0000","last_became_active":"2024-09-21T12:22:50.903705+0000","last_became_peered":"2024-09-21T12:22:50.903705+0000","last_unstale":"2024-09-21T12:26:22.037671+0000","last_undegraded":"2024-09-21T12:26:22.037671+0000","last_fullsized":"2024-09-21T12:26:22.037671+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:58.948625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:58.948625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:23.221920+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":299,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038013+0000","last_change":"2024-09-21T12:26:16.149823+0000","last_active":"2024-09-21T12:26:22.038013+0000","last_peered":"2024-09-21T12:26:22.038013+0000","last_clean":"2024-09-21T12:26:22.038013+0000","last_became_active":"2024-09-21T12:20:19.315418+0000","last_became_peered":"2024-09-21T12:20:19.315418+0000","last_unstale":"2024-09-21T12:26:22.038013+0000","last_undegraded":"2024-09-21T12:26:22.038013+0000","last_fullsized":"2024-09-21T12:26:22.038013+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:16.149550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:16.149550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:27.681950+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":298,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.029203+0000","last_change":"2024-09-21T12:24:56.879617+0000","last_active":"2024-09-21T12:26:22.029203+0000","last_peered":"2024-09-21T12:26:22.029203+0000","last_clean":"2024-09-21T12:26:22.029203+0000","last_became_active":"2024-09-21T12:20:21.310053+0000","last_became_peered":"2024-09-21T12:20:21.310053+0000","last_unstale":"2024-09-21T12:26:22.029203+0000","last_undegraded":"2024-09-21T12:26:22.029203+0000","last_fullsized":"2024-09-21T12:26:22.029203+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:56.879558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:56.879558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:26.453298+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":316,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038245+0000","last_change":"2024-09-21T12:25:41.888249+0000","last_active":"2024-09-21T12:26:22.038245+0000","last_peered":"2024-09-21T12:26:22.038245+0000","last_clean":"2024-09-21T12:26:22.038245+0000","last_became_active":"2024-09-21T12:20:17.299015+0000","last_became_peered":"2024-09-21T12:20:17.299015+0000","last_unstale":"2024-09-21T12:26:22.038245+0000","last_undegraded":"2024-09-21T12:26:22.038245+0000","last_fullsized":"2024-09-21T12:26:22.038245+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:25:41.888037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:41.888037+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:10.841319+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":340,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037922+0000","last_change":"2024-09-21T12:26:19.973528+0000","last_active":"2024-09-21T12:26:22.037922+0000","last_peered":"2024-09-21T12:26:22.037922+0000","last_clean":"2024-09-21T12:26:22.037922+0000","last_became_active":"2024-09-21T12:22:50.922019+0000","last_became_peered":"2024-09-21T12:22:50.922019+0000","last_unstale":"2024-09-21T12:26:22.037922+0000","last_undegraded":"2024-09-21T12:26:22.037922+0000","last_fullsized":"2024-09-21T12:26:22.037922+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:19.973461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:19.973461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:36.925984+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":302,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.032732+0000","last_change":"2024-09-21T12:26:21.572474+0000","last_active":"2024-09-21T12:26:22.032732+0000","last_peered":"2024-09-21T12:26:22.032732+0000","last_clean":"2024-09-21T12:26:22.032732+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:26:22.032732+0000","last_undegraded":"2024-09-21T12:26:22.032732+0000","last_fullsized":"2024-09-21T12:26:22.032732+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:21.571726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:21.571726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:50.797908+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":292,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.026388+0000","last_change":"2024-09-21T12:23:29.134829+0000","last_active":"2024-09-21T12:26:22.026388+0000","last_peered":"2024-09-21T12:26:22.026388+0000","last_clean":"2024-09-21T12:26:22.026388+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:26:22.026388+0000","last_undegraded":"2024-09-21T12:26:22.026388+0000","last_fullsized":"2024-09-21T12:26:22.026388+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:29.134767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:29.134767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":325,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.031870+0000","last_change":"2024-09-21T12:24:58.824615+0000","last_active":"2024-09-21T12:26:22.031870+0000","last_peered":"2024-09-21T12:26:22.031870+0000","last_clean":"2024-09-21T12:26:22.031870+0000","last_became_active":"2024-09-21T12:20:17.694794+0000","last_became_peered":"2024-09-21T12:20:17.694794+0000","last_unstale":"2024-09-21T12:26:22.031870+0000","last_undegraded":"2024-09-21T12:26:22.031870+0000","last_fullsized":"2024-09-21T12:26:22.031870+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:24:58.824516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:58.824516+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:24.775664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":319,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039911+0000","last_change":"2024-09-21T12:24:33.364225+0000","last_active":"2024-09-21T12:26:22.039911+0000","last_peered":"2024-09-21T12:26:22.039911+0000","last_clean":"2024-09-21T12:26:22.039911+0000","last_became_active":"2024-09-21T12:20:15.295968+0000","last_became_peered":"2024-09-21T12:20:15.295968+0000","last_unstale":"2024-09-21T12:26:22.039911+0000","last_undegraded":"2024-09-21T12:26:22.039911+0000","last_fullsized":"2024-09-21T12:26:22.039911+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:33.363808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:33.363808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":316,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.035666+0000","last_change":"2024-09-21T12:23:57.018797+0000","last_active":"2024-09-21T12:26:22.035666+0000","last_peered":"2024-09-21T12:26:22.035666+0000","last_clean":"2024-09-21T12:26:22.035666+0000","last_became_active":"2024-09-21T12:22:50.921604+0000","last_became_peered":"2024-09-21T12:22:50.921604+0000","last_unstale":"2024-09-21T12:26:22.035666+0000","last_undegraded":"2024-09-21T12:26:22.035666+0000","last_fullsized":"2024-09-21T12:26:22.035666+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:57.018749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:57.018749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:53.766292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":290,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.032902+0000","last_change":"2024-09-21T12:24:35.588409+0000","last_active":"2024-09-21T12:26:22.032902+0000","last_peered":"2024-09-21T12:26:22.032902+0000","last_clean":"2024-09-21T12:26:22.032902+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:26:22.032902+0000","last_undegraded":"2024-09-21T12:26:22.032902+0000","last_fullsized":"2024-09-21T12:26:22.032902+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:35.588353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:35.588353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":304,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.031637+0000","last_change":"2024-09-21T12:23:51.343568+0000","last_active":"2024-09-21T12:26:22.031637+0000","last_peered":"2024-09-21T12:26:22.031637+0000","last_clean":"2024-09-21T12:26:22.031637+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:26:22.031637+0000","last_undegraded":"2024-09-21T12:26:22.031637+0000","last_fullsized":"2024-09-21T12:26:22.031637+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-21T12:23:51.343491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:51.343491+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:26.153152+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":318,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.028587+0000","last_change":"2024-09-21T12:25:48.713463+0000","last_active":"2024-09-21T12:26:22.028587+0000","last_peered":"2024-09-21T12:26:22.028587+0000","last_clean":"2024-09-21T12:26:22.028587+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:26:22.028587+0000","last_undegraded":"2024-09-21T12:26:22.028587+0000","last_fullsized":"2024-09-21T12:26:22.028587+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:25:48.713392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:48.713392+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:06.809432+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":324,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037159+0000","last_change":"2024-09-21T12:26:22.014941+0000","last_active":"2024-09-21T12:26:22.037159+0000","last_peered":"2024-09-21T12:26:22.037159+0000","last_clean":"2024-09-21T12:26:22.037159+0000","last_became_active":"2024-09-21T12:22:50.930615+0000","last_became_peered":"2024-09-21T12:22:50.930615+0000","last_unstale":"2024-09-21T12:26:22.037159+0000","last_undegraded":"2024-09-21T12:26:22.037159+0000","last_fullsized":"2024-09-21T12:26:22.037159+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:22.014896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:22.014896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:34.583908+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":312,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036445+0000","last_change":"2024-09-21T12:26:14.855364+0000","last_active":"2024-09-21T12:26:22.036445+0000","last_peered":"2024-09-21T12:26:22.036445+0000","last_clean":"2024-09-21T12:26:22.036445+0000","last_became_active":"2024-09-21T12:22:50.917408+0000","last_became_peered":"2024-09-21T12:22:50.917408+0000","last_unstale":"2024-09-21T12:26:22.036445+0000","last_undegraded":"2024-09-21T12:26:22.036445+0000","last_fullsized":"2024-09-21T12:26:22.036445+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:14.855248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:14.855248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:15.644031+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":313,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036030+0000","last_change":"2024-09-21T12:26:15.152613+0000","last_active":"2024-09-21T12:26:22.036030+0000","last_peered":"2024-09-21T12:26:22.036030+0000","last_clean":"2024-09-21T12:26:22.036030+0000","last_became_active":"2024-09-21T12:22:50.914221+0000","last_became_peered":"2024-09-21T12:22:50.914221+0000","last_unstale":"2024-09-21T12:26:22.036030+0000","last_undegraded":"2024-09-21T12:26:22.036030+0000","last_fullsized":"2024-09-21T12:26:22.036030+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:15.152540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:15.152540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:17.180451+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":310,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.031512+0000","last_change":"2024-09-21T12:25:00.871133+0000","last_active":"2024-09-21T12:26:22.031512+0000","last_peered":"2024-09-21T12:26:22.031512+0000","last_clean":"2024-09-21T12:26:22.031512+0000","last_became_active":"2024-09-21T12:20:15.297993+0000","last_became_peered":"2024-09-21T12:20:15.297993+0000","last_unstale":"2024-09-21T12:26:22.031512+0000","last_undegraded":"2024-09-21T12:26:22.031512+0000","last_fullsized":"2024-09-21T12:26:22.031512+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:00.871062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:00.871062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":318,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038400+0000","last_change":"2024-09-21T12:25:47.974973+0000","last_active":"2024-09-21T12:26:22.038400+0000","last_peered":"2024-09-21T12:26:22.038400+0000","last_clean":"2024-09-21T12:26:22.038400+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:26:22.038400+0000","last_undegraded":"2024-09-21T12:26:22.038400+0000","last_fullsized":"2024-09-21T12:26:22.038400+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:25:47.974897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:47.974897+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:04.942357+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":313,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.028768+0000","last_change":"2024-09-21T12:25:32.660054+0000","last_active":"2024-09-21T12:26:22.028768+0000","last_peered":"2024-09-21T12:26:22.028768+0000","last_clean":"2024-09-21T12:26:22.028768+0000","last_became_active":"2024-09-21T12:20:19.308247+0000","last_became_peered":"2024-09-21T12:20:19.308247+0000","last_unstale":"2024-09-21T12:26:22.028768+0000","last_undegraded":"2024-09-21T12:26:22.028768+0000","last_fullsized":"2024-09-21T12:26:22.028768+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:32.659918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:32.659918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:46.283244+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":8295809048,"num_objects":126585,"num_object_clones":0,"num_object_copies":379755,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":126585,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134243,"num_write_kb":8346433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24902512640,"data_stored":24902500424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69592,"ondisk_log_size":69592,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":690,"num_read_kb":447,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":201,"num_bytes_recovered":3982,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":396,"ondisk_log_size":396,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":5023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":366,"num_read_kb":361,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234062,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3311980,"kb_used_data":3068380,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":90433172,"statfs":{"total":95995035648,"available":92603568128,"internally_reserved":0,"allocated":3142021120,"data_stored":3141649984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":249428426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462998,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3315752,"kb_used_data":3067992,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":90429400,"statfs":{"total":95995035648,"available":92599705600,"internally_reserved":0,"allocated":3141623808,"data_stored":3141256535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":253688262},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724633,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4261424,"kb_used_data":4067744,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":89483728,"statfs":{"total":95995035648,"available":91631337472,"internally_reserved":0,"allocated":4165369856,"data_stored":4164996058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":198244805},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953569,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149799,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3867332,"kb_used_data":3565172,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":89877820,"statfs":{"total":95995035648,"available":92034887680,"internally_reserved":0,"allocated":3650736128,"data_stored":3650342235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":309131719},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411437,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3796768,"kb_used_data":3571024,"kb_used_omap":1,"kb_used_meta":225534,"kb_avail":89948384,"statfs":{"total":95995035648,"available":92107145216,"internally_reserved":0,"allocated":3656728576,"data_stored":3656369619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":230947269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574963,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3850316,"kb_used_data":3562300,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":89894836,"statfs":{"total":95995035648,"available":92052312064,"internally_reserved":0,"allocated":3647795200,"data_stored":3647391591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738489,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3830384,"kb_used_data":3556064,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":89914768,"statfs":{"total":95995035648,"available":92072722432,"internally_reserved":0,"allocated":3641409536,"data_stored":3641034993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":280689093},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3621978112,"data_stored":3621978112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3624468480,"data_stored":3624468480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3633319936,"data_stored":3633315864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3627155456,"data_stored":3627155456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4157083648,"data_stored":4157079576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3119906816,"data_stored":3119906816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3118600192,"data_stored":3118596120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:26:34.130 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T12:26:34.131 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:26:34.131 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-21T12:26:34.131 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:26:34.131 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:34.174 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:34.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:34.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:34.276 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:34.321 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:34.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:34.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:34.423 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:34.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:34.450 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:34.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:34.551 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:34.576 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:34.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:34.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:34.678 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:34.719 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:34.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:34.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:34.821 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:34.854 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:34.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:34.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:34.903 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T12:26:34.956 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:34.982 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:34.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:34.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:35.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:34 smithi114 ceph-mon[30324]: pgmap v302: 158 pgs: 158 active+clean; 7.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 965 B/s rd, 28 MiB/s wr, 1.34k op/s 2024-09-21T12:26:35.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:34 smithi114 ceph-mon[40867]: pgmap v302: 158 pgs: 158 active+clean; 7.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 965 B/s rd, 28 MiB/s wr, 1.34k op/s 2024-09-21T12:26:35.083 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:35.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:35.137 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:35.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:35.238 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:35.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:34 smithi183 ceph-mon[33277]: pgmap v302: 158 pgs: 158 active+clean; 7.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 965 B/s rd, 28 MiB/s wr, 1.34k op/s 2024-09-21T12:26:35.272 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:35.280 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:35.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:35.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:35.357 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:35.382 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:35.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:35.447 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:35.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:35.548 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:35.585 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:35.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:35.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:35.687 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:35.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:35.730 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:35.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:35.760 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-21T12:26:35.832 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:35.841 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:35 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/856535741' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:26:35.875 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:35.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:35.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:35.958 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:35 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/856535741' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:26:35.958 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:35 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/856535741' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:26:35.977 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:36.003 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:36.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:36.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:36.105 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:36.134 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:36.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:36.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:36.236 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:36.277 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:36.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:36.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:36.379 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:36.422 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:36.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:36.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:36.479 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:26:36.479 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:26:36.479 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:26:36.479 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:26:36.479 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:26:36.479 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:26:36.479 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:26:36.480 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:26:36.480 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:26:36.480 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:26:36.480 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:26:36.480 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:26:36.480 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:26:36.480 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:26:36.480 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:26:36.480 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:26:36.524 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:36.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:36.554 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:36.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:36.655 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:36.684 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:36.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:36.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:36.762 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:36 smithi114 ceph-mon[30324]: pgmap v303: 158 pgs: 158 active+clean; 7.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 938 B/s rd, 14 MiB/s wr, 1.08k op/s 2024-09-21T12:26:36.762 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:36 smithi114 ceph-mon[30324]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:26:36.762 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:36 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/856535741' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:26:36.762 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:36 smithi114 ceph-mon[30324]: osdmap e201: 8 total, 8 up, 7 in 2024-09-21T12:26:36.762 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:36 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3028306110' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:26:36.762 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:36 smithi114 ceph-mon[40867]: pgmap v303: 158 pgs: 158 active+clean; 7.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 938 B/s rd, 14 MiB/s wr, 1.08k op/s 2024-09-21T12:26:36.786 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:36.814 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:36.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:36.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:36.916 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:36.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:36.951 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:36.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:37.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:36 smithi114 ceph-mon[40867]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:26:37.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:36 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/856535741' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:26:37.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:36 smithi114 ceph-mon[40867]: osdmap e201: 8 total, 8 up, 7 in 2024-09-21T12:26:37.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:36 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3028306110' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:26:37.050 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:37.053 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:37.109 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:37.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:37.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:37.211 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:37.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:26:36 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:26:37.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:36 smithi183 ceph-mon[33277]: pgmap v303: 158 pgs: 158 active+clean; 7.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 938 B/s rd, 14 MiB/s wr, 1.08k op/s 2024-09-21T12:26:37.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:36 smithi183 ceph-mon[33277]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:26:37.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:36 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/856535741' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:26:37.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:36 smithi183 ceph-mon[33277]: osdmap e201: 8 total, 8 up, 7 in 2024-09-21T12:26:37.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:36 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3028306110' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:26:37.260 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:37.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:37.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:37.362 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:37.422 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:37.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:37.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:37.524 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:37.554 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:37.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:37.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:37.656 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:37.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:37.691 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:37.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:37.695 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:26:37.696 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:26:37.791 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:37.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:37.828 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:37.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:37.929 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:37.968 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:37.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:37.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:38.031 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:26:38.034 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:26:38.070 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:38.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:38.107 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:38.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:38.208 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:38.233 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:38.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:38.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:38.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:37 smithi183 ceph-mon[33277]: from='client.25003 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:26:38.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:37 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:26:38.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:37 smithi114 ceph-mon[40867]: from='client.25003 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:26:38.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:37 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:26:38.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:37 smithi114 ceph-mon[30324]: from='client.25003 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:26:38.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:37 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:26:38.336 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:38.370 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:38.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:38.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:38.472 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:38.510 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:38.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:38.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:38.612 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:38.649 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:38.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:38.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:38.751 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:38.786 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:38.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:38.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:38.888 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:38.924 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:38.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:38.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:39.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:38 smithi114 ceph-mon[40867]: pgmap v305: 158 pgs: 158 active+clean; 6.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-21T12:26:39.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:38 smithi114 ceph-mon[30324]: pgmap v305: 158 pgs: 158 active+clean; 6.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-21T12:26:39.026 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:39.063 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:39.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:39.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:39.103 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T12:26:39.165 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:39.208 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:39.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:39.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:39.234 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2024-09-21T12:26:35.761090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7867210291,"num_objects":120235,"num_object_clones":0,"num_object_copies":360705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":120235,"num_whiteouts":0,"num_read":1086,"num_read_kb":838,"num_write":141228,"num_write_kb":8346478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":223,"num_bytes_recovered":4534,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70900,"ondisk_log_size":70900,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":25019364,"kb_used_data":23201652,"kb_used_omap":10,"kb_used_meta":1817461,"kb_avail":631196700,"statfs":{"total":671965249536,"available":646345420800,"internally_reserved":0,"allocated":23758491648,"data_stored":23755848439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":1861080171},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,3,8,4,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-507707368,"num_objects":-7746,"num_object_clones":0,"num_object_copies":-23238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7746,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":13010,"num_write_kb":168385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.075745"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":313,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.028545+0000","last_change":"2024-09-21T12:24:50.115300+0000","last_active":"2024-09-21T12:26:22.028545+0000","last_peered":"2024-09-21T12:26:22.028545+0000","last_clean":"2024-09-21T12:26:22.028545+0000","last_became_active":"2024-09-21T12:20:15.309431+0000","last_became_peered":"2024-09-21T12:20:15.309431+0000","last_unstale":"2024-09-21T12:26:22.028545+0000","last_undegraded":"2024-09-21T12:26:22.028545+0000","last_fullsized":"2024-09-21T12:26:22.028545+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:50.115253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:50.115253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:04.907671+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":283,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036652+0000","last_change":"2024-09-21T12:26:09.823435+0000","last_active":"2024-09-21T12:26:22.036652+0000","last_peered":"2024-09-21T12:26:22.036652+0000","last_clean":"2024-09-21T12:26:22.036652+0000","last_became_active":"2024-09-21T12:22:50.909699+0000","last_became_peered":"2024-09-21T12:22:50.909699+0000","last_unstale":"2024-09-21T12:26:22.036652+0000","last_undegraded":"2024-09-21T12:26:22.036652+0000","last_fullsized":"2024-09-21T12:26:22.036652+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:26:09.823363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:09.823363+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:17.313787+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":304,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.035875+0000","last_change":"2024-09-21T12:24:34.311501+0000","last_active":"2024-09-21T12:26:22.035875+0000","last_peered":"2024-09-21T12:26:22.035875+0000","last_clean":"2024-09-21T12:26:22.035875+0000","last_became_active":"2024-09-21T12:20:19.314908+0000","last_became_peered":"2024-09-21T12:20:19.314908+0000","last_unstale":"2024-09-21T12:26:22.035875+0000","last_undegraded":"2024-09-21T12:26:22.035875+0000","last_fullsized":"2024-09-21T12:26:22.035875+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:34.311451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:34.311451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:00.014883+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":227,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039419+0000","last_change":"2024-09-21T12:22:50.914064+0000","last_active":"2024-09-21T12:26:22.039419+0000","last_peered":"2024-09-21T12:26:22.039419+0000","last_clean":"2024-09-21T12:26:22.039419+0000","last_became_active":"2024-09-21T12:22:50.913743+0000","last_became_peered":"2024-09-21T12:22:50.913743+0000","last_unstale":"2024-09-21T12:26:22.039419+0000","last_undegraded":"2024-09-21T12:26:22.039419+0000","last_fullsized":"2024-09-21T12:26:22.039419+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:58.189665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:21:58.189665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:07.338098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":290,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.032565+0000","last_change":"2024-09-21T12:25:48.316122+0000","last_active":"2024-09-21T12:26:22.032565+0000","last_peered":"2024-09-21T12:26:22.032565+0000","last_clean":"2024-09-21T12:26:22.032565+0000","last_became_active":"2024-09-21T12:20:21.595301+0000","last_became_peered":"2024-09-21T12:20:21.595301+0000","last_unstale":"2024-09-21T12:26:22.032565+0000","last_undegraded":"2024-09-21T12:26:22.032565+0000","last_fullsized":"2024-09-21T12:26:22.032565+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:48.316039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:48.316039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:08.665134+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":311,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.031696+0000","last_change":"2024-09-21T12:24:49.701972+0000","last_active":"2024-09-21T12:26:22.031696+0000","last_peered":"2024-09-21T12:26:22.031696+0000","last_clean":"2024-09-21T12:26:22.031696+0000","last_became_active":"2024-09-21T12:20:15.308655+0000","last_became_peered":"2024-09-21T12:20:15.308655+0000","last_unstale":"2024-09-21T12:26:22.031696+0000","last_undegraded":"2024-09-21T12:26:22.031696+0000","last_fullsized":"2024-09-21T12:26:22.031696+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:49.701915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:49.701915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":331,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.032104+0000","last_change":"2024-09-21T12:25:01.898461+0000","last_active":"2024-09-21T12:26:22.032104+0000","last_peered":"2024-09-21T12:26:22.032104+0000","last_clean":"2024-09-21T12:26:22.032104+0000","last_became_active":"2024-09-21T12:20:17.305220+0000","last_became_peered":"2024-09-21T12:20:17.305220+0000","last_unstale":"2024-09-21T12:26:22.032104+0000","last_undegraded":"2024-09-21T12:26:22.032104+0000","last_fullsized":"2024-09-21T12:26:22.032104+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:25:01.898376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:01.898376+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":245,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036791+0000","last_change":"2024-09-21T12:25:04.827596+0000","last_active":"2024-09-21T12:26:22.036791+0000","last_peered":"2024-09-21T12:26:22.036791+0000","last_clean":"2024-09-21T12:26:22.036791+0000","last_became_active":"2024-09-21T12:22:50.926240+0000","last_became_peered":"2024-09-21T12:22:50.926240+0000","last_unstale":"2024-09-21T12:26:22.036791+0000","last_undegraded":"2024-09-21T12:26:22.036791+0000","last_fullsized":"2024-09-21T12:26:22.036791+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:04.827529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:04.827529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":332,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.040061+0000","last_change":"2024-09-21T12:26:19.446300+0000","last_active":"2024-09-21T12:26:22.040061+0000","last_peered":"2024-09-21T12:26:22.040061+0000","last_clean":"2024-09-21T12:26:22.040061+0000","last_became_active":"2024-09-21T12:22:50.912481+0000","last_became_peered":"2024-09-21T12:22:50.912481+0000","last_unstale":"2024-09-21T12:26:22.040061+0000","last_undegraded":"2024-09-21T12:26:22.040061+0000","last_fullsized":"2024-09-21T12:26:22.040061+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:19.446251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:19.446251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:36.052328+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":291,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.030071+0000","last_change":"2024-09-21T12:24:25.640253+0000","last_active":"2024-09-21T12:26:22.030071+0000","last_peered":"2024-09-21T12:26:22.030071+0000","last_clean":"2024-09-21T12:26:22.030071+0000","last_became_active":"2024-09-21T12:22:50.927804+0000","last_became_peered":"2024-09-21T12:22:50.927804+0000","last_unstale":"2024-09-21T12:26:22.030071+0000","last_undegraded":"2024-09-21T12:26:22.030071+0000","last_fullsized":"2024-09-21T12:26:22.030071+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:24:25.640169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:25.640169+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:58.377864+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":303,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039580+0000","last_change":"2024-09-21T12:24:58.566964+0000","last_active":"2024-09-21T12:26:22.039580+0000","last_peered":"2024-09-21T12:26:22.039580+0000","last_clean":"2024-09-21T12:26:22.039580+0000","last_became_active":"2024-09-21T12:20:19.308184+0000","last_became_peered":"2024-09-21T12:20:19.308184+0000","last_unstale":"2024-09-21T12:26:22.039580+0000","last_undegraded":"2024-09-21T12:26:22.039580+0000","last_fullsized":"2024-09-21T12:26:22.039580+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:58.566902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:58.566902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:27.540756+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":300,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.026521+0000","last_change":"2024-09-21T12:25:31.616293+0000","last_active":"2024-09-21T12:26:22.026521+0000","last_peered":"2024-09-21T12:26:22.026521+0000","last_clean":"2024-09-21T12:26:22.026521+0000","last_became_active":"2024-09-21T12:20:21.309190+0000","last_became_peered":"2024-09-21T12:20:21.309190+0000","last_unstale":"2024-09-21T12:26:22.026521+0000","last_undegraded":"2024-09-21T12:26:22.026521+0000","last_fullsized":"2024-09-21T12:26:22.026521+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:31.616219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:31.616219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:48.224028+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":324,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039857+0000","last_change":"2024-09-21T12:22:54.174099+0000","last_active":"2024-09-21T12:26:22.039857+0000","last_peered":"2024-09-21T12:26:22.039857+0000","last_clean":"2024-09-21T12:26:22.039857+0000","last_became_active":"2024-09-21T12:22:50.911737+0000","last_became_peered":"2024-09-21T12:22:50.911737+0000","last_unstale":"2024-09-21T12:26:22.039857+0000","last_undegraded":"2024-09-21T12:26:22.039857+0000","last_fullsized":"2024-09-21T12:26:22.039857+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:54.174034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:54.174034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:23.349015+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":277,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.032818+0000","last_change":"2024-09-21T12:24:28.624430+0000","last_active":"2024-09-21T12:26:22.032818+0000","last_peered":"2024-09-21T12:26:22.032818+0000","last_clean":"2024-09-21T12:26:22.032818+0000","last_became_active":"2024-09-21T12:22:50.918794+0000","last_became_peered":"2024-09-21T12:22:50.918794+0000","last_unstale":"2024-09-21T12:26:22.032818+0000","last_undegraded":"2024-09-21T12:26:22.032818+0000","last_fullsized":"2024-09-21T12:26:22.032818+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-21T12:24:28.624364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:28.624364+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":316,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036516+0000","last_change":"2024-09-21T12:25:01.897596+0000","last_active":"2024-09-21T12:26:22.036516+0000","last_peered":"2024-09-21T12:26:22.036516+0000","last_clean":"2024-09-21T12:26:22.036516+0000","last_became_active":"2024-09-21T12:22:50.921780+0000","last_became_peered":"2024-09-21T12:22:50.921780+0000","last_unstale":"2024-09-21T12:26:22.036516+0000","last_undegraded":"2024-09-21T12:26:22.036516+0000","last_fullsized":"2024-09-21T12:26:22.036516+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:01.897527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:01.897527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":307,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039403+0000","last_change":"2024-09-21T12:24:34.367580+0000","last_active":"2024-09-21T12:26:22.039403+0000","last_peered":"2024-09-21T12:26:22.039403+0000","last_clean":"2024-09-21T12:26:22.039403+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:26:22.039403+0000","last_undegraded":"2024-09-21T12:26:22.039403+0000","last_fullsized":"2024-09-21T12:26:22.039403+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:34.367532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:34.367532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":244,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037928+0000","last_change":"2024-09-21T12:25:30.888535+0000","last_active":"2024-09-21T12:26:22.037928+0000","last_peered":"2024-09-21T12:26:22.037928+0000","last_clean":"2024-09-21T12:26:22.037928+0000","last_became_active":"2024-09-21T12:22:50.914792+0000","last_became_peered":"2024-09-21T12:22:50.914792+0000","last_unstale":"2024-09-21T12:26:22.037928+0000","last_undegraded":"2024-09-21T12:26:22.037928+0000","last_fullsized":"2024-09-21T12:26:22.037928+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:30.888491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:30.888491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:32.893945+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":341,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.029197+0000","last_change":"2024-09-21T12:24:30.033946+0000","last_active":"2024-09-21T12:26:22.029197+0000","last_peered":"2024-09-21T12:26:22.029197+0000","last_clean":"2024-09-21T12:26:22.029197+0000","last_became_active":"2024-09-21T12:22:50.921576+0000","last_became_peered":"2024-09-21T12:22:50.921576+0000","last_unstale":"2024-09-21T12:26:22.029197+0000","last_undegraded":"2024-09-21T12:26:22.029197+0000","last_fullsized":"2024-09-21T12:26:22.029197+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:30.033874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:30.033874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:00.878523+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":333,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038044+0000","last_change":"2024-09-21T12:24:50.834579+0000","last_active":"2024-09-21T12:26:22.038044+0000","last_peered":"2024-09-21T12:26:22.038044+0000","last_clean":"2024-09-21T12:26:22.038044+0000","last_became_active":"2024-09-21T12:20:17.307828+0000","last_became_peered":"2024-09-21T12:20:17.307828+0000","last_unstale":"2024-09-21T12:26:22.038044+0000","last_undegraded":"2024-09-21T12:26:22.038044+0000","last_fullsized":"2024-09-21T12:26:22.038044+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:24:50.834503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:50.834503+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":332,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037949+0000","last_change":"2024-09-21T12:25:46.906275+0000","last_active":"2024-09-21T12:26:22.037949+0000","last_peered":"2024-09-21T12:26:22.037949+0000","last_clean":"2024-09-21T12:26:22.037949+0000","last_became_active":"2024-09-21T12:25:46.905878+0000","last_became_peered":"2024-09-21T12:25:46.905878+0000","last_unstale":"2024-09-21T12:26:22.037949+0000","last_undegraded":"2024-09-21T12:26:22.037949+0000","last_fullsized":"2024-09-21T12:26:22.037949+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:48.435769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:48.435769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:57.929606+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":290,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.029802+0000","last_change":"2024-09-21T12:23:58.354628+0000","last_active":"2024-09-21T12:26:22.029802+0000","last_peered":"2024-09-21T12:26:22.029802+0000","last_clean":"2024-09-21T12:26:22.029802+0000","last_became_active":"2024-09-21T12:20:21.592796+0000","last_became_peered":"2024-09-21T12:20:21.592796+0000","last_unstale":"2024-09-21T12:26:22.029802+0000","last_undegraded":"2024-09-21T12:26:22.029802+0000","last_fullsized":"2024-09-21T12:26:22.029802+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:58.354570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:58.354570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":243,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038676+0000","last_change":"2024-09-21T12:24:55.447829+0000","last_active":"2024-09-21T12:26:22.038676+0000","last_peered":"2024-09-21T12:26:22.038676+0000","last_clean":"2024-09-21T12:26:22.038676+0000","last_became_active":"2024-09-21T12:22:50.925403+0000","last_became_peered":"2024-09-21T12:22:50.925403+0000","last_unstale":"2024-09-21T12:26:22.038676+0000","last_undegraded":"2024-09-21T12:26:22.038676+0000","last_fullsized":"2024-09-21T12:26:22.038676+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:55.447755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:55.447755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":386,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.029742+0000","last_change":"2024-09-21T12:25:31.270272+0000","last_active":"2024-09-21T12:26:22.029742+0000","last_peered":"2024-09-21T12:26:22.029742+0000","last_clean":"2024-09-21T12:26:22.029742+0000","last_became_active":"2024-09-21T12:22:50.912144+0000","last_became_peered":"2024-09-21T12:22:50.912144+0000","last_unstale":"2024-09-21T12:26:22.029742+0000","last_undegraded":"2024-09-21T12:26:22.029742+0000","last_fullsized":"2024-09-21T12:26:22.029742+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:25:31.270165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:31.270165+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:48.492514+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":220,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.030756+0000","last_change":"2024-09-21T12:22:50.920178+0000","last_active":"2024-09-21T12:26:22.030756+0000","last_peered":"2024-09-21T12:26:22.030756+0000","last_clean":"2024-09-21T12:26:22.030756+0000","last_became_active":"2024-09-21T12:22:50.918326+0000","last_became_peered":"2024-09-21T12:22:50.918326+0000","last_unstale":"2024-09-21T12:26:22.030756+0000","last_undegraded":"2024-09-21T12:26:22.030756+0000","last_fullsized":"2024-09-21T12:26:22.030756+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:50.318827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":292,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037320+0000","last_change":"2024-09-21T12:23:51.065780+0000","last_active":"2024-09-21T12:26:22.037320+0000","last_peered":"2024-09-21T12:26:22.037320+0000","last_clean":"2024-09-21T12:26:22.037320+0000","last_became_active":"2024-09-21T12:20:21.312976+0000","last_became_peered":"2024-09-21T12:20:21.312976+0000","last_unstale":"2024-09-21T12:26:22.037320+0000","last_undegraded":"2024-09-21T12:26:22.037320+0000","last_fullsized":"2024-09-21T12:26:22.037320+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:23:51.065733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:51.065733+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:06.800726+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":346,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036066+0000","last_change":"2024-09-21T12:26:18.112687+0000","last_active":"2024-09-21T12:26:22.036066+0000","last_peered":"2024-09-21T12:26:22.036066+0000","last_clean":"2024-09-21T12:26:22.036066+0000","last_became_active":"2024-09-21T12:22:50.910933+0000","last_became_peered":"2024-09-21T12:22:50.910933+0000","last_unstale":"2024-09-21T12:26:22.036066+0000","last_undegraded":"2024-09-21T12:26:22.036066+0000","last_fullsized":"2024-09-21T12:26:22.036066+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:26:18.112614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:18.112614+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:47.795526+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":316,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.028890+0000","last_change":"2024-09-21T12:24:03.059782+0000","last_active":"2024-09-21T12:26:22.028890+0000","last_peered":"2024-09-21T12:26:22.028890+0000","last_clean":"2024-09-21T12:26:22.028890+0000","last_became_active":"2024-09-21T12:20:17.695977+0000","last_became_peered":"2024-09-21T12:20:17.695977+0000","last_unstale":"2024-09-21T12:26:22.028890+0000","last_undegraded":"2024-09-21T12:26:22.028890+0000","last_fullsized":"2024-09-21T12:26:22.028890+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:24:03.059549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:03.059549+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:25.640560+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"121'11","reported_seq":432,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:30.421601+0000","last_change":"2024-09-21T12:26:10.846256+0000","last_active":"2024-09-21T12:26:30.421601+0000","last_peered":"2024-09-21T12:26:30.421601+0000","last_clean":"2024-09-21T12:26:30.421601+0000","last_became_active":"2024-09-21T12:22:50.916683+0000","last_became_peered":"2024-09-21T12:22:50.916683+0000","last_unstale":"2024-09-21T12:26:30.421601+0000","last_undegraded":"2024-09-21T12:26:30.421601+0000","last_fullsized":"2024-09-21T12:26:30.421601+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:26:10.846183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:10.846183+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:32.395882+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"200'8591","reported_seq":8682,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:31.783511+0000","last_change":"2024-09-21T12:24:46.316817+0000","last_active":"2024-09-21T12:26:31.783511+0000","last_peered":"2024-09-21T12:26:31.783511+0000","last_clean":"2024-09-21T12:26:31.783511+0000","last_became_active":"2024-09-21T12:24:46.316132+0000","last_became_peered":"2024-09-21T12:24:46.316132+0000","last_unstale":"2024-09-21T12:26:31.783511+0000","last_undegraded":"2024-09-21T12:26:31.783511+0000","last_fullsized":"2024-09-21T12:26:31.783511+0000","mapping_epoch":160,"log_start":"180'4300","ondisk_log_start":"180'4300","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4291,"log_dups_size":0,"ondisk_log_size":4291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:05.983258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482148352,"num_objects":7357,"num_object_clones":0,"num_object_copies":22071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8591,"num_write_kb":510336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"76'79","reported_seq":212,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039238+0000","last_change":"2024-09-21T12:25:42.586460+0000","last_active":"2024-09-21T12:26:22.039238+0000","last_peered":"2024-09-21T12:26:22.039238+0000","last_clean":"2024-09-21T12:26:22.039238+0000","last_became_active":"2024-09-21T12:22:17.074721+0000","last_became_peered":"2024-09-21T12:22:17.074721+0000","last_unstale":"2024-09-21T12:26:22.039238+0000","last_undegraded":"2024-09-21T12:26:22.039238+0000","last_fullsized":"2024-09-21T12:26:22.039238+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:25:42.586392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:25:42.586392+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:57.694710+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":163,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037681+0000","last_change":"2024-09-21T12:24:59.803037+0000","last_active":"2024-09-21T12:26:22.037681+0000","last_peered":"2024-09-21T12:26:22.037681+0000","last_clean":"2024-09-21T12:26:22.037681+0000","last_became_active":"2024-09-21T12:23:46.910112+0000","last_became_peered":"2024-09-21T12:23:46.910112+0000","last_unstale":"2024-09-21T12:26:22.037681+0000","last_undegraded":"2024-09-21T12:26:22.037681+0000","last_fullsized":"2024-09-21T12:26:22.037681+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:24:59.802945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:59.802945+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:22.089988+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":302,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.031584+0000","last_change":"2024-09-21T12:22:14.036290+0000","last_active":"2024-09-21T12:26:22.031584+0000","last_peered":"2024-09-21T12:26:22.031584+0000","last_clean":"2024-09-21T12:26:22.031584+0000","last_became_active":"2024-09-21T12:20:15.295774+0000","last_became_peered":"2024-09-21T12:20:15.295774+0000","last_unstale":"2024-09-21T12:26:22.031584+0000","last_undegraded":"2024-09-21T12:26:22.031584+0000","last_fullsized":"2024-09-21T12:26:22.031584+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:14.036236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.036236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:55.467472+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"158'13","reported_seq":134,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:25.420756+0000","last_change":"2024-09-21T12:25:33.474458+0000","last_active":"2024-09-21T12:26:25.420756+0000","last_peered":"2024-09-21T12:26:25.420756+0000","last_clean":"2024-09-21T12:26:25.420756+0000","last_became_active":"2024-09-21T12:24:39.272835+0000","last_became_peered":"2024-09-21T12:24:39.272835+0000","last_unstale":"2024-09-21T12:26:25.420756+0000","last_undegraded":"2024-09-21T12:26:25.420756+0000","last_fullsized":"2024-09-21T12:26:25.420756+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'13","last_scrub_stamp":"2024-09-21T12:25:33.474391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:33.474391+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:01.060227+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":292,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039652+0000","last_change":"2024-09-21T12:22:12.145637+0000","last_active":"2024-09-21T12:26:22.039652+0000","last_peered":"2024-09-21T12:26:22.039652+0000","last_clean":"2024-09-21T12:26:22.039652+0000","last_became_active":"2024-09-21T12:20:21.310022+0000","last_became_peered":"2024-09-21T12:20:21.310022+0000","last_unstale":"2024-09-21T12:26:22.039652+0000","last_undegraded":"2024-09-21T12:26:22.039652+0000","last_fullsized":"2024-09-21T12:26:22.039652+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:22:12.145550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:12.145550+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:51.460208+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"200'8858","reported_seq":8949,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:32.244813+0000","last_change":"2024-09-21T12:24:46.326848+0000","last_active":"2024-09-21T12:26:32.244813+0000","last_peered":"2024-09-21T12:26:32.244813+0000","last_clean":"2024-09-21T12:26:32.244813+0000","last_became_active":"2024-09-21T12:24:46.326586+0000","last_became_peered":"2024-09-21T12:24:46.326586+0000","last_unstale":"2024-09-21T12:26:32.244813+0000","last_undegraded":"2024-09-21T12:26:32.244813+0000","last_fullsized":"2024-09-21T12:26:32.244813+0000","mapping_epoch":160,"log_start":"181'4500","ondisk_log_start":"181'4500","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4358,"log_dups_size":0,"ondisk_log_size":4358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":491520000,"num_objects":7500,"num_object_clones":0,"num_object_copies":22500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8858,"num_write_kb":523456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"76'79","reported_seq":202,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037642+0000","last_change":"2024-09-21T12:26:21.091972+0000","last_active":"2024-09-21T12:26:22.037642+0000","last_peered":"2024-09-21T12:26:22.037642+0000","last_clean":"2024-09-21T12:26:22.037642+0000","last_became_active":"2024-09-21T12:22:50.914885+0000","last_became_peered":"2024-09-21T12:22:50.914885+0000","last_unstale":"2024-09-21T12:26:22.037642+0000","last_undegraded":"2024-09-21T12:26:22.037642+0000","last_fullsized":"2024-09-21T12:26:22.037642+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:21.091797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:21.091797+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:25.919035+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":339,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.031414+0000","last_change":"2024-09-21T12:25:35.325593+0000","last_active":"2024-09-21T12:26:22.031414+0000","last_peered":"2024-09-21T12:26:22.031414+0000","last_clean":"2024-09-21T12:26:22.031414+0000","last_became_active":"2024-09-21T12:20:17.693179+0000","last_became_peered":"2024-09-21T12:20:17.693179+0000","last_unstale":"2024-09-21T12:26:22.031414+0000","last_undegraded":"2024-09-21T12:26:22.031414+0000","last_fullsized":"2024-09-21T12:26:22.031414+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-21T12:25:35.325493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:35.325493+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:38.650408+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":302,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.031170+0000","last_change":"2024-09-21T12:21:18.056305+0000","last_active":"2024-09-21T12:26:22.031170+0000","last_peered":"2024-09-21T12:26:22.031170+0000","last_clean":"2024-09-21T12:26:22.031170+0000","last_became_active":"2024-09-21T12:20:15.297340+0000","last_became_peered":"2024-09-21T12:20:15.297340+0000","last_unstale":"2024-09-21T12:26:22.031170+0000","last_undegraded":"2024-09-21T12:26:22.031170+0000","last_fullsized":"2024-09-21T12:26:22.031170+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:18.056261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:18.056261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:07.391958+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":245,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.031748+0000","last_change":"2024-09-21T12:25:38.374222+0000","last_active":"2024-09-21T12:26:22.031748+0000","last_peered":"2024-09-21T12:26:22.031748+0000","last_clean":"2024-09-21T12:26:22.031748+0000","last_became_active":"2024-09-21T12:22:50.930858+0000","last_became_peered":"2024-09-21T12:22:50.930858+0000","last_unstale":"2024-09-21T12:26:22.031748+0000","last_undegraded":"2024-09-21T12:26:22.031748+0000","last_fullsized":"2024-09-21T12:26:22.031748+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:38.374146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:38.374146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:58.524870+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":236,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.029963+0000","last_change":"2024-09-21T12:24:29.635826+0000","last_active":"2024-09-21T12:26:22.029963+0000","last_peered":"2024-09-21T12:26:22.029963+0000","last_clean":"2024-09-21T12:26:22.029963+0000","last_became_active":"2024-09-21T12:22:50.918940+0000","last_became_peered":"2024-09-21T12:22:50.918940+0000","last_unstale":"2024-09-21T12:26:22.029963+0000","last_undegraded":"2024-09-21T12:26:22.029963+0000","last_fullsized":"2024-09-21T12:26:22.029963+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:29.635514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:29.635514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"200'8805","reported_seq":8897,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:31.618843+0000","last_change":"2024-09-21T12:24:46.324341+0000","last_active":"2024-09-21T12:26:31.618843+0000","last_peered":"2024-09-21T12:26:31.618843+0000","last_clean":"2024-09-21T12:26:31.618843+0000","last_became_active":"2024-09-21T12:24:46.323986+0000","last_became_peered":"2024-09-21T12:24:46.323986+0000","last_unstale":"2024-09-21T12:26:31.618843+0000","last_undegraded":"2024-09-21T12:26:31.618843+0000","last_fullsized":"2024-09-21T12:26:31.618843+0000","mapping_epoch":160,"log_start":"180'4300","ondisk_log_start":"180'4300","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4505,"log_dups_size":0,"ondisk_log_size":4505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":489947160,"num_objects":7477,"num_object_clones":0,"num_object_copies":22431,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7477,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8805,"num_write_kb":520961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"76'79","reported_seq":218,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.030460+0000","last_change":"2024-09-21T12:25:02.911776+0000","last_active":"2024-09-21T12:26:22.030460+0000","last_peered":"2024-09-21T12:26:22.030460+0000","last_clean":"2024-09-21T12:26:22.030460+0000","last_became_active":"2024-09-21T12:22:17.069042+0000","last_became_peered":"2024-09-21T12:22:17.069042+0000","last_unstale":"2024-09-21T12:26:22.030460+0000","last_undegraded":"2024-09-21T12:26:22.030460+0000","last_fullsized":"2024-09-21T12:26:22.030460+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:25:02.911705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:25:02.911705+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":301,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039447+0000","last_change":"2024-09-21T12:22:50.978102+0000","last_active":"2024-09-21T12:26:22.039447+0000","last_peered":"2024-09-21T12:26:22.039447+0000","last_clean":"2024-09-21T12:26:22.039447+0000","last_became_active":"2024-09-21T12:22:50.926008+0000","last_became_peered":"2024-09-21T12:22:50.926008+0000","last_unstale":"2024-09-21T12:26:22.039447+0000","last_undegraded":"2024-09-21T12:26:22.039447+0000","last_fullsized":"2024-09-21T12:26:22.039447+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-21T12:22:14.062464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.062464+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:49.540547+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":338,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.029193+0000","last_change":"2024-09-21T12:22:51.003768+0000","last_active":"2024-09-21T12:26:22.029193+0000","last_peered":"2024-09-21T12:26:22.029193+0000","last_clean":"2024-09-21T12:26:22.029193+0000","last_became_active":"2024-09-21T12:22:50.923284+0000","last_became_peered":"2024-09-21T12:22:50.923284+0000","last_unstale":"2024-09-21T12:26:22.029193+0000","last_undegraded":"2024-09-21T12:26:22.029193+0000","last_fullsized":"2024-09-21T12:26:22.029193+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:22:07.582969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:07.582969+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:53.794213+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"121'11","reported_seq":420,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:27.130623+0000","last_change":"2024-09-21T12:26:14.490045+0000","last_active":"2024-09-21T12:26:27.130623+0000","last_peered":"2024-09-21T12:26:27.130623+0000","last_clean":"2024-09-21T12:26:27.130623+0000","last_became_active":"2024-09-21T12:22:50.930342+0000","last_became_peered":"2024-09-21T12:22:50.930342+0000","last_unstale":"2024-09-21T12:26:27.130623+0000","last_undegraded":"2024-09-21T12:26:27.130623+0000","last_fullsized":"2024-09-21T12:26:27.130623+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:26:14.489972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:14.489972+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:31.388160+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":291,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036752+0000","last_change":"2024-09-21T12:21:21.835669+0000","last_active":"2024-09-21T12:26:22.036752+0000","last_peered":"2024-09-21T12:26:22.036752+0000","last_clean":"2024-09-21T12:26:22.036752+0000","last_became_active":"2024-09-21T12:20:21.593367+0000","last_became_peered":"2024-09-21T12:20:21.593367+0000","last_unstale":"2024-09-21T12:26:22.036752+0000","last_undegraded":"2024-09-21T12:26:22.036752+0000","last_fullsized":"2024-09-21T12:26:22.036752+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:21.835623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:21:21.835623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:29.766408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"200'8731","reported_seq":8822,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:32.208187+0000","last_change":"2024-09-21T12:24:46.325776+0000","last_active":"2024-09-21T12:26:32.208187+0000","last_peered":"2024-09-21T12:26:32.208187+0000","last_clean":"2024-09-21T12:26:32.208187+0000","last_became_active":"2024-09-21T12:24:46.325260+0000","last_became_peered":"2024-09-21T12:24:46.325260+0000","last_unstale":"2024-09-21T12:26:32.208187+0000","last_undegraded":"2024-09-21T12:26:32.208187+0000","last_fullsized":"2024-09-21T12:26:32.208187+0000","mapping_epoch":160,"log_start":"180'4400","ondisk_log_start":"180'4400","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4331,"log_dups_size":0,"ondisk_log_size":4331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:01.850898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484769792,"num_objects":7397,"num_object_clones":0,"num_object_copies":22191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8731,"num_write_kb":516096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"76'79","reported_seq":302,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038101+0000","last_change":"2024-09-21T12:24:51.918097+0000","last_active":"2024-09-21T12:26:22.038101+0000","last_peered":"2024-09-21T12:26:22.038101+0000","last_clean":"2024-09-21T12:26:22.038101+0000","last_became_active":"2024-09-21T12:22:17.093298+0000","last_became_peered":"2024-09-21T12:22:17.093298+0000","last_unstale":"2024-09-21T12:26:22.038101+0000","last_undegraded":"2024-09-21T12:26:22.038101+0000","last_fullsized":"2024-09-21T12:26:22.038101+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:24:51.918028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:24:51.918028+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":373,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038198+0000","last_change":"2024-09-21T12:24:01.388987+0000","last_active":"2024-09-21T12:26:22.038198+0000","last_peered":"2024-09-21T12:26:22.038198+0000","last_clean":"2024-09-21T12:26:22.038198+0000","last_became_active":"2024-09-21T12:22:50.925840+0000","last_became_peered":"2024-09-21T12:22:50.925840+0000","last_unstale":"2024-09-21T12:26:22.038198+0000","last_undegraded":"2024-09-21T12:26:22.038198+0000","last_fullsized":"2024-09-21T12:26:22.038198+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:24:01.388905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:01.388905+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:24.187759+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":311,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.040202+0000","last_change":"2024-09-21T12:26:16.410118+0000","last_active":"2024-09-21T12:26:22.040202+0000","last_peered":"2024-09-21T12:26:22.040202+0000","last_clean":"2024-09-21T12:26:22.040202+0000","last_became_active":"2024-09-21T12:20:15.308518+0000","last_became_peered":"2024-09-21T12:20:15.308518+0000","last_unstale":"2024-09-21T12:26:22.040202+0000","last_undegraded":"2024-09-21T12:26:22.040202+0000","last_fullsized":"2024-09-21T12:26:22.040202+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:16.410067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:16.410067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:24.383217+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":391,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:25.420536+0000","last_change":"2024-09-21T12:25:42.615523+0000","last_active":"2024-09-21T12:26:25.420536+0000","last_peered":"2024-09-21T12:26:25.420536+0000","last_clean":"2024-09-21T12:26:25.420536+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:26:25.420536+0000","last_undegraded":"2024-09-21T12:26:25.420536+0000","last_fullsized":"2024-09-21T12:26:25.420536+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:25:42.615446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:42.615446+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:06.344897+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":293,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.026676+0000","last_change":"2024-09-21T12:22:51.987032+0000","last_active":"2024-09-21T12:26:22.026676+0000","last_peered":"2024-09-21T12:26:22.026676+0000","last_clean":"2024-09-21T12:26:22.026676+0000","last_became_active":"2024-09-21T12:20:21.315801+0000","last_became_peered":"2024-09-21T12:20:21.315801+0000","last_unstale":"2024-09-21T12:26:22.026676+0000","last_undegraded":"2024-09-21T12:26:22.026676+0000","last_fullsized":"2024-09-21T12:26:22.026676+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:51.986953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:51.986953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:22.782321+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"200'8877","reported_seq":8968,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:32.134813+0000","last_change":"2024-09-21T12:24:46.324384+0000","last_active":"2024-09-21T12:26:32.134813+0000","last_peered":"2024-09-21T12:26:32.134813+0000","last_clean":"2024-09-21T12:26:32.134813+0000","last_became_active":"2024-09-21T12:24:46.323283+0000","last_became_peered":"2024-09-21T12:24:46.323283+0000","last_unstale":"2024-09-21T12:26:32.134813+0000","last_undegraded":"2024-09-21T12:26:32.134813+0000","last_fullsized":"2024-09-21T12:26:32.134813+0000","mapping_epoch":160,"log_start":"182'4751","ondisk_log_start":"182'4751","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4126,"log_dups_size":0,"ondisk_log_size":4126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":490799104,"num_objects":7489,"num_object_clones":0,"num_object_copies":22467,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8877,"num_write_kb":523712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"76'79","reported_seq":302,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037831+0000","last_change":"2024-09-21T12:24:53.927007+0000","last_active":"2024-09-21T12:26:22.037831+0000","last_peered":"2024-09-21T12:26:22.037831+0000","last_clean":"2024-09-21T12:26:22.037831+0000","last_became_active":"2024-09-21T12:22:17.083629+0000","last_became_peered":"2024-09-21T12:22:17.083629+0000","last_unstale":"2024-09-21T12:26:22.037831+0000","last_undegraded":"2024-09-21T12:26:22.037831+0000","last_fullsized":"2024-09-21T12:26:22.037831+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:24:53.926958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:24:53.926958+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":345,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.028819+0000","last_change":"2024-09-21T12:26:14.834328+0000","last_active":"2024-09-21T12:26:22.028819+0000","last_peered":"2024-09-21T12:26:22.028819+0000","last_clean":"2024-09-21T12:26:22.028819+0000","last_became_active":"2024-09-21T12:20:17.693067+0000","last_became_peered":"2024-09-21T12:20:17.693067+0000","last_unstale":"2024-09-21T12:26:22.028819+0000","last_undegraded":"2024-09-21T12:26:22.028819+0000","last_fullsized":"2024-09-21T12:26:22.028819+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-21T12:26:14.834236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:14.834236+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:34.377077+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":304,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038265+0000","last_change":"2024-09-21T12:26:12.131784+0000","last_active":"2024-09-21T12:26:22.038265+0000","last_peered":"2024-09-21T12:26:22.038265+0000","last_clean":"2024-09-21T12:26:22.038265+0000","last_became_active":"2024-09-21T12:20:15.308441+0000","last_became_peered":"2024-09-21T12:20:15.308441+0000","last_unstale":"2024-09-21T12:26:22.038265+0000","last_undegraded":"2024-09-21T12:26:22.038265+0000","last_fullsized":"2024-09-21T12:26:22.038265+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:12.131721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:12.131721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:19.426420+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":304,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036165+0000","last_change":"2024-09-21T12:26:12.887076+0000","last_active":"2024-09-21T12:26:22.036165+0000","last_peered":"2024-09-21T12:26:22.036165+0000","last_clean":"2024-09-21T12:26:22.036165+0000","last_became_active":"2024-09-21T12:20:19.309598+0000","last_became_peered":"2024-09-21T12:20:19.309598+0000","last_unstale":"2024-09-21T12:26:22.036165+0000","last_undegraded":"2024-09-21T12:26:22.036165+0000","last_fullsized":"2024-09-21T12:26:22.036165+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:12.887007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:12.887007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:22.153550+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":292,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.035790+0000","last_change":"2024-09-21T12:25:44.918356+0000","last_active":"2024-09-21T12:26:22.035790+0000","last_peered":"2024-09-21T12:26:22.035790+0000","last_clean":"2024-09-21T12:26:22.035790+0000","last_became_active":"2024-09-21T12:20:21.592828+0000","last_became_peered":"2024-09-21T12:20:21.592828+0000","last_unstale":"2024-09-21T12:26:22.035790+0000","last_undegraded":"2024-09-21T12:26:22.035790+0000","last_fullsized":"2024-09-21T12:26:22.035790+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:44.918276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:44.918276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:07.775801+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"200'8853","reported_seq":8944,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:31.500635+0000","last_change":"2024-09-21T12:24:46.325092+0000","last_active":"2024-09-21T12:26:31.500635+0000","last_peered":"2024-09-21T12:26:31.500635+0000","last_clean":"2024-09-21T12:26:31.500635+0000","last_became_active":"2024-09-21T12:24:46.324740+0000","last_became_peered":"2024-09-21T12:24:46.324740+0000","last_unstale":"2024-09-21T12:26:31.500635+0000","last_undegraded":"2024-09-21T12:26:31.500635+0000","last_fullsized":"2024-09-21T12:26:31.500635+0000","mapping_epoch":160,"log_start":"180'4400","ondisk_log_start":"180'4400","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":494993408,"num_objects":7553,"num_object_clones":0,"num_object_copies":22659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8853,"num_write_kb":524992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"76'79","reported_seq":205,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039260+0000","last_change":"2024-09-21T12:23:21.346117+0000","last_active":"2024-09-21T12:26:22.039260+0000","last_peered":"2024-09-21T12:26:22.039260+0000","last_clean":"2024-09-21T12:26:22.039260+0000","last_became_active":"2024-09-21T12:22:17.084732+0000","last_became_peered":"2024-09-21T12:22:17.084732+0000","last_unstale":"2024-09-21T12:26:22.039260+0000","last_undegraded":"2024-09-21T12:26:22.039260+0000","last_fullsized":"2024-09-21T12:26:22.039260+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:23:21.346046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:23:21.346046+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:04.385057+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":424,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:25.436701+0000","last_change":"2024-09-21T12:26:09.082395+0000","last_active":"2024-09-21T12:26:25.436701+0000","last_peered":"2024-09-21T12:26:25.436701+0000","last_clean":"2024-09-21T12:26:25.436701+0000","last_became_active":"2024-09-21T12:20:17.307769+0000","last_became_peered":"2024-09-21T12:20:17.307769+0000","last_unstale":"2024-09-21T12:26:25.436701+0000","last_undegraded":"2024-09-21T12:26:25.436701+0000","last_fullsized":"2024-09-21T12:26:25.436701+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-21T12:26:09.082314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:09.082314+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:17.369834+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":228,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.029125+0000","last_change":"2024-09-21T12:22:50.922961+0000","last_active":"2024-09-21T12:26:22.029125+0000","last_peered":"2024-09-21T12:26:22.029125+0000","last_clean":"2024-09-21T12:26:22.029125+0000","last_became_active":"2024-09-21T12:22:50.922002+0000","last_became_peered":"2024-09-21T12:22:50.922002+0000","last_unstale":"2024-09-21T12:26:22.029125+0000","last_undegraded":"2024-09-21T12:26:22.029125+0000","last_fullsized":"2024-09-21T12:26:22.029125+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:05.507899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:05.507899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:50.702651+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":314,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.040250+0000","last_change":"2024-09-21T12:21:20.045824+0000","last_active":"2024-09-21T12:26:22.040250+0000","last_peered":"2024-09-21T12:26:22.040250+0000","last_clean":"2024-09-21T12:26:22.040250+0000","last_became_active":"2024-09-21T12:20:14.287998+0000","last_became_peered":"2024-09-21T12:20:14.287998+0000","last_unstale":"2024-09-21T12:26:22.040250+0000","last_undegraded":"2024-09-21T12:26:22.040250+0000","last_fullsized":"2024-09-21T12:26:22.040250+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T12:21:20.045755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:21:20.045755+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:05.385454+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011011149999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":325,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.028157+0000","last_change":"2024-09-21T12:24:53.112915+0000","last_active":"2024-09-21T12:26:22.028157+0000","last_peered":"2024-09-21T12:26:22.028157+0000","last_clean":"2024-09-21T12:26:22.028157+0000","last_became_active":"2024-09-21T12:22:50.921488+0000","last_became_peered":"2024-09-21T12:22:50.921488+0000","last_unstale":"2024-09-21T12:26:22.028157+0000","last_undegraded":"2024-09-21T12:26:22.028157+0000","last_fullsized":"2024-09-21T12:26:22.028157+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:53.112868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:53.112868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:05.952435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":319,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.031431+0000","last_change":"2024-09-21T12:26:10.435410+0000","last_active":"2024-09-21T12:26:22.031431+0000","last_peered":"2024-09-21T12:26:22.031431+0000","last_clean":"2024-09-21T12:26:22.031431+0000","last_became_active":"2024-09-21T12:22:50.917558+0000","last_became_peered":"2024-09-21T12:22:50.917558+0000","last_unstale":"2024-09-21T12:26:22.031431+0000","last_undegraded":"2024-09-21T12:26:22.031431+0000","last_fullsized":"2024-09-21T12:26:22.031431+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:10.435349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:10.435349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:19.570759+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"200'8818","reported_seq":8908,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:32.121305+0000","last_change":"2024-09-21T12:24:46.323849+0000","last_active":"2024-09-21T12:26:32.121305+0000","last_peered":"2024-09-21T12:26:32.121305+0000","last_clean":"2024-09-21T12:26:32.121305+0000","last_became_active":"2024-09-21T12:24:46.323306+0000","last_became_peered":"2024-09-21T12:24:46.323306+0000","last_unstale":"2024-09-21T12:26:32.121305+0000","last_undegraded":"2024-09-21T12:26:32.121305+0000","last_fullsized":"2024-09-21T12:26:32.121305+0000","mapping_epoch":160,"log_start":"182'4700","ondisk_log_start":"182'4700","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4118,"log_dups_size":0,"ondisk_log_size":4118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":493486080,"num_objects":7530,"num_object_clones":0,"num_object_copies":22590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8818,"num_write_kb":523136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'79","reported_seq":388,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036405+0000","last_change":"2024-09-21T12:25:03.877681+0000","last_active":"2024-09-21T12:26:22.036405+0000","last_peered":"2024-09-21T12:26:22.036405+0000","last_clean":"2024-09-21T12:26:22.036405+0000","last_became_active":"2024-09-21T12:22:50.921709+0000","last_became_peered":"2024-09-21T12:22:50.921709+0000","last_unstale":"2024-09-21T12:26:22.036405+0000","last_undegraded":"2024-09-21T12:26:22.036405+0000","last_fullsized":"2024-09-21T12:26:22.036405+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:25:03.877610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:25:03.877610+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:26.706650+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":335,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.043559+0000","last_change":"2024-09-21T12:25:45.962538+0000","last_active":"2024-09-21T12:26:22.043559+0000","last_peered":"2024-09-21T12:26:22.043559+0000","last_clean":"2024-09-21T12:26:22.043559+0000","last_became_active":"2024-09-21T12:20:17.297374+0000","last_became_peered":"2024-09-21T12:20:17.297374+0000","last_unstale":"2024-09-21T12:26:22.043559+0000","last_undegraded":"2024-09-21T12:26:22.043559+0000","last_fullsized":"2024-09-21T12:26:22.043559+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-21T12:25:45.962433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:45.962433+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:08.718722+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":313,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036059+0000","last_change":"2024-09-21T12:23:59.367953+0000","last_active":"2024-09-21T12:26:22.036059+0000","last_peered":"2024-09-21T12:26:22.036059+0000","last_clean":"2024-09-21T12:26:22.036059+0000","last_became_active":"2024-09-21T12:20:15.308356+0000","last_became_peered":"2024-09-21T12:20:15.308356+0000","last_unstale":"2024-09-21T12:26:22.036059+0000","last_undegraded":"2024-09-21T12:26:22.036059+0000","last_fullsized":"2024-09-21T12:26:22.036059+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:59.367778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:23:59.367778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":330,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037159+0000","last_change":"2024-09-21T12:26:14.109964+0000","last_active":"2024-09-21T12:26:22.037159+0000","last_peered":"2024-09-21T12:26:22.037159+0000","last_clean":"2024-09-21T12:26:22.037159+0000","last_became_active":"2024-09-21T12:20:14.288358+0000","last_became_peered":"2024-09-21T12:20:14.288358+0000","last_unstale":"2024-09-21T12:26:22.037159+0000","last_undegraded":"2024-09-21T12:26:22.037159+0000","last_fullsized":"2024-09-21T12:26:22.037159+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:26:14.109886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:26:14.109886+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:31.748840+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.0019042449999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":304,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038917+0000","last_change":"2024-09-21T12:25:46.181836+0000","last_active":"2024-09-21T12:26:22.038917+0000","last_peered":"2024-09-21T12:26:22.038917+0000","last_clean":"2024-09-21T12:26:22.038917+0000","last_became_active":"2024-09-21T12:20:19.309724+0000","last_became_peered":"2024-09-21T12:20:19.309724+0000","last_unstale":"2024-09-21T12:26:22.038917+0000","last_undegraded":"2024-09-21T12:26:22.038917+0000","last_fullsized":"2024-09-21T12:26:22.038917+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:46.181531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:46.181531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:11.368016+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":283,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037727+0000","last_change":"2024-09-21T12:20:21.593267+0000","last_active":"2024-09-21T12:26:22.037727+0000","last_peered":"2024-09-21T12:26:22.037727+0000","last_clean":"2024-09-21T12:26:22.037727+0000","last_became_active":"2024-09-21T12:20:21.592982+0000","last_became_peered":"2024-09-21T12:20:21.592982+0000","last_unstale":"2024-09-21T12:26:22.037727+0000","last_undegraded":"2024-09-21T12:26:22.037727+0000","last_fullsized":"2024-09-21T12:26:22.037727+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:22.831735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"200'8676","reported_seq":8767,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:31.841125+0000","last_change":"2024-09-21T12:24:46.326200+0000","last_active":"2024-09-21T12:26:31.841125+0000","last_peered":"2024-09-21T12:26:31.841125+0000","last_clean":"2024-09-21T12:26:31.841125+0000","last_became_active":"2024-09-21T12:24:46.325838+0000","last_became_peered":"2024-09-21T12:24:46.325838+0000","last_unstale":"2024-09-21T12:26:31.841125+0000","last_undegraded":"2024-09-21T12:26:31.841125+0000","last_fullsized":"2024-09-21T12:26:31.841125+0000","mapping_epoch":160,"log_start":"181'4400","ondisk_log_start":"181'4400","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4276,"log_dups_size":0,"ondisk_log_size":4276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":489947136,"num_objects":7476,"num_object_clones":0,"num_object_copies":22428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8676,"num_write_kb":516864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"76'79","reported_seq":227,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.047286+0000","last_change":"2024-09-21T12:23:57.442430+0000","last_active":"2024-09-21T12:26:22.047286+0000","last_peered":"2024-09-21T12:26:22.047286+0000","last_clean":"2024-09-21T12:26:22.047286+0000","last_became_active":"2024-09-21T12:22:50.912088+0000","last_became_peered":"2024-09-21T12:22:50.912088+0000","last_unstale":"2024-09-21T12:26:22.047286+0000","last_undegraded":"2024-09-21T12:26:22.047286+0000","last_fullsized":"2024-09-21T12:26:22.047286+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:23:57.442284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:23:57.442284+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":318,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.028932+0000","last_change":"2024-09-21T12:22:49.933621+0000","last_active":"2024-09-21T12:26:22.028932+0000","last_peered":"2024-09-21T12:26:22.028932+0000","last_clean":"2024-09-21T12:26:22.028932+0000","last_became_active":"2024-09-21T12:20:17.299325+0000","last_became_peered":"2024-09-21T12:20:17.299325+0000","last_unstale":"2024-09-21T12:26:22.028932+0000","last_undegraded":"2024-09-21T12:26:22.028932+0000","last_fullsized":"2024-09-21T12:26:22.028932+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:22:49.933382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:49.933382+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:23.687742+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":334,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.029868+0000","last_change":"2024-09-21T12:24:48.073954+0000","last_active":"2024-09-21T12:26:22.029868+0000","last_peered":"2024-09-21T12:26:22.029868+0000","last_clean":"2024-09-21T12:26:22.029868+0000","last_became_active":"2024-09-21T12:22:50.923796+0000","last_became_peered":"2024-09-21T12:22:50.923796+0000","last_unstale":"2024-09-21T12:26:22.029868+0000","last_undegraded":"2024-09-21T12:26:22.029868+0000","last_fullsized":"2024-09-21T12:26:22.029868+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:48.073424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:48.073424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:58.857724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":694,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:30.454246+0000","last_change":"2024-09-21T12:25:35.819787+0000","last_active":"2024-09-21T12:26:30.454246+0000","last_peered":"2024-09-21T12:26:30.454246+0000","last_clean":"2024-09-21T12:26:30.454246+0000","last_became_active":"2024-09-21T12:20:14.287543+0000","last_became_peered":"2024-09-21T12:20:14.287543+0000","last_unstale":"2024-09-21T12:26:30.454246+0000","last_undegraded":"2024-09-21T12:26:30.454246+0000","last_fullsized":"2024-09-21T12:26:30.454246+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-21T12:25:35.819707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:25:35.819707+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:05.641221+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":371,"num_read_kb":366,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":303,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.040111+0000","last_change":"2024-09-21T12:26:11.422340+0000","last_active":"2024-09-21T12:26:22.040111+0000","last_peered":"2024-09-21T12:26:22.040111+0000","last_clean":"2024-09-21T12:26:22.040111+0000","last_became_active":"2024-09-21T12:20:19.314217+0000","last_became_peered":"2024-09-21T12:20:19.314217+0000","last_unstale":"2024-09-21T12:26:22.040111+0000","last_undegraded":"2024-09-21T12:26:22.040111+0000","last_fullsized":"2024-09-21T12:26:22.040111+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:11.422293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:11.422293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:13.710870+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":308,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.035799+0000","last_change":"2024-09-21T12:25:38.882984+0000","last_active":"2024-09-21T12:26:22.035799+0000","last_peered":"2024-09-21T12:26:22.035799+0000","last_clean":"2024-09-21T12:26:22.035799+0000","last_became_active":"2024-09-21T12:20:21.311188+0000","last_became_peered":"2024-09-21T12:20:21.311188+0000","last_unstale":"2024-09-21T12:26:22.035799+0000","last_undegraded":"2024-09-21T12:26:22.035799+0000","last_fullsized":"2024-09-21T12:26:22.035799+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:38.882915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:38.882915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:00.415003+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"200'8922","reported_seq":9013,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:31.622651+0000","last_change":"2024-09-21T12:24:46.318053+0000","last_active":"2024-09-21T12:26:31.622651+0000","last_peered":"2024-09-21T12:26:31.622651+0000","last_clean":"2024-09-21T12:26:31.622651+0000","last_became_active":"2024-09-21T12:24:46.317644+0000","last_became_peered":"2024-09-21T12:24:46.317644+0000","last_unstale":"2024-09-21T12:26:31.622651+0000","last_undegraded":"2024-09-21T12:26:31.622651+0000","last_fullsized":"2024-09-21T12:26:31.622651+0000","mapping_epoch":160,"log_start":"180'4300","ondisk_log_start":"180'4300","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4622,"log_dups_size":0,"ondisk_log_size":4622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:57.982352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":499122176,"num_objects":7616,"num_object_clones":0,"num_object_copies":22848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8922,"num_write_kb":529216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"121'19","reported_seq":480,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:25.420588+0000","last_change":"2024-09-21T12:24:27.932976+0000","last_active":"2024-09-21T12:26:25.420588+0000","last_peered":"2024-09-21T12:26:25.420588+0000","last_clean":"2024-09-21T12:26:25.420588+0000","last_became_active":"2024-09-21T12:22:50.921690+0000","last_became_peered":"2024-09-21T12:22:50.921690+0000","last_unstale":"2024-09-21T12:26:25.420588+0000","last_undegraded":"2024-09-21T12:26:25.420588+0000","last_fullsized":"2024-09-21T12:26:25.420588+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'19","last_scrub_stamp":"2024-09-21T12:24:27.932891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:27.932891+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:57.001434+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":317,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.035542+0000","last_change":"2024-09-21T12:26:12.394327+0000","last_active":"2024-09-21T12:26:22.035542+0000","last_peered":"2024-09-21T12:26:22.035542+0000","last_clean":"2024-09-21T12:26:22.035542+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:26:22.035542+0000","last_undegraded":"2024-09-21T12:26:22.035542+0000","last_fullsized":"2024-09-21T12:26:22.035542+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:26:12.394247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:12.394247+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:12.954462+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":368,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036124+0000","last_change":"2024-09-21T12:24:31.089221+0000","last_active":"2024-09-21T12:26:22.036124+0000","last_peered":"2024-09-21T12:26:22.036124+0000","last_clean":"2024-09-21T12:26:22.036124+0000","last_became_active":"2024-09-21T12:22:15.049592+0000","last_became_peered":"2024-09-21T12:22:15.049592+0000","last_unstale":"2024-09-21T12:26:22.036124+0000","last_undegraded":"2024-09-21T12:26:22.036124+0000","last_fullsized":"2024-09-21T12:26:22.036124+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:24:31.089155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:24:31.089155+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":287,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037524+0000","last_change":"2024-09-21T12:20:19.310280+0000","last_active":"2024-09-21T12:26:22.037524+0000","last_peered":"2024-09-21T12:26:22.037524+0000","last_clean":"2024-09-21T12:26:22.037524+0000","last_became_active":"2024-09-21T12:20:19.309859+0000","last_became_peered":"2024-09-21T12:20:19.309859+0000","last_unstale":"2024-09-21T12:26:22.037524+0000","last_undegraded":"2024-09-21T12:26:22.037524+0000","last_fullsized":"2024-09-21T12:26:22.037524+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:28.753163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":328,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.040007+0000","last_change":"2024-09-21T12:25:32.465748+0000","last_active":"2024-09-21T12:26:22.040007+0000","last_peered":"2024-09-21T12:26:22.040007+0000","last_clean":"2024-09-21T12:26:22.040007+0000","last_became_active":"2024-09-21T12:22:50.913493+0000","last_became_peered":"2024-09-21T12:22:50.913493+0000","last_unstale":"2024-09-21T12:26:22.040007+0000","last_undegraded":"2024-09-21T12:26:22.040007+0000","last_fullsized":"2024-09-21T12:26:22.040007+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:32.465700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:32.465700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:57.893144+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"200'8740","reported_seq":8831,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:32.235087+0000","last_change":"2024-09-21T12:24:46.315286+0000","last_active":"2024-09-21T12:26:32.235087+0000","last_peered":"2024-09-21T12:26:32.235087+0000","last_clean":"2024-09-21T12:26:32.235087+0000","last_became_active":"2024-09-21T12:24:46.314957+0000","last_became_peered":"2024-09-21T12:24:46.314957+0000","last_unstale":"2024-09-21T12:26:32.235087+0000","last_undegraded":"2024-09-21T12:26:32.235087+0000","last_fullsized":"2024-09-21T12:26:32.235087+0000","mapping_epoch":160,"log_start":"181'4400","ondisk_log_start":"181'4400","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4340,"log_dups_size":0,"ondisk_log_size":4340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:57.838388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":486146048,"num_objects":7418,"num_object_clones":0,"num_object_copies":22254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8740,"num_write_kb":517056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":318,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038025+0000","last_change":"2024-09-21T12:26:15.952570+0000","last_active":"2024-09-21T12:26:22.038025+0000","last_peered":"2024-09-21T12:26:22.038025+0000","last_clean":"2024-09-21T12:26:22.038025+0000","last_became_active":"2024-09-21T12:23:46.885802+0000","last_became_peered":"2024-09-21T12:23:46.885802+0000","last_unstale":"2024-09-21T12:26:22.038025+0000","last_undegraded":"2024-09-21T12:26:22.038025+0000","last_fullsized":"2024-09-21T12:26:22.038025+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:26:15.952496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:15.952496+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:37.337083+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":228,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036981+0000","last_change":"2024-09-21T12:22:50.908380+0000","last_active":"2024-09-21T12:26:22.036981+0000","last_peered":"2024-09-21T12:26:22.036981+0000","last_clean":"2024-09-21T12:26:22.036981+0000","last_became_active":"2024-09-21T12:22:50.908060+0000","last_became_peered":"2024-09-21T12:22:50.908060+0000","last_unstale":"2024-09-21T12:26:22.036981+0000","last_undegraded":"2024-09-21T12:26:22.036981+0000","last_fullsized":"2024-09-21T12:26:22.036981+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:05.892153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:05.892153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:26.966868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":324,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039897+0000","last_change":"2024-09-21T12:23:19.372762+0000","last_active":"2024-09-21T12:26:22.039897+0000","last_peered":"2024-09-21T12:26:22.039897+0000","last_clean":"2024-09-21T12:26:22.039897+0000","last_became_active":"2024-09-21T12:22:50.915081+0000","last_became_peered":"2024-09-21T12:22:50.915081+0000","last_unstale":"2024-09-21T12:26:22.039897+0000","last_undegraded":"2024-09-21T12:26:22.039897+0000","last_fullsized":"2024-09-21T12:26:22.039897+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:19.372719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:19.372719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:03.351727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":290,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.030674+0000","last_change":"2024-09-21T12:23:46.282792+0000","last_active":"2024-09-21T12:26:22.030674+0000","last_peered":"2024-09-21T12:26:22.030674+0000","last_clean":"2024-09-21T12:26:22.030674+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:26:22.030674+0000","last_undegraded":"2024-09-21T12:26:22.030674+0000","last_fullsized":"2024-09-21T12:26:22.030674+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:46.282661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:46.282661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"200'8889","reported_seq":8980,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:31.818541+0000","last_change":"2024-09-21T12:24:46.323690+0000","last_active":"2024-09-21T12:26:31.818541+0000","last_peered":"2024-09-21T12:26:31.818541+0000","last_clean":"2024-09-21T12:26:31.818541+0000","last_became_active":"2024-09-21T12:24:46.323402+0000","last_became_peered":"2024-09-21T12:24:46.323402+0000","last_unstale":"2024-09-21T12:26:31.818541+0000","last_undegraded":"2024-09-21T12:26:31.818541+0000","last_fullsized":"2024-09-21T12:26:31.818541+0000","mapping_epoch":160,"log_start":"181'4600","ondisk_log_start":"181'4600","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4289,"log_dups_size":0,"ondisk_log_size":4289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:01.998231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":495386624,"num_objects":7559,"num_object_clones":0,"num_object_copies":22677,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8889,"num_write_kb":526336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"76'79","reported_seq":219,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039307+0000","last_change":"2024-09-21T12:24:28.372308+0000","last_active":"2024-09-21T12:26:22.039307+0000","last_peered":"2024-09-21T12:26:22.039307+0000","last_clean":"2024-09-21T12:26:22.039307+0000","last_became_active":"2024-09-21T12:22:17.087245+0000","last_became_peered":"2024-09-21T12:22:17.087245+0000","last_unstale":"2024-09-21T12:26:22.039307+0000","last_undegraded":"2024-09-21T12:26:22.039307+0000","last_fullsized":"2024-09-21T12:26:22.039307+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:24:28.372230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:24:28.372230+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":353,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037769+0000","last_change":"2024-09-21T12:25:42.183260+0000","last_active":"2024-09-21T12:26:22.037769+0000","last_peered":"2024-09-21T12:26:22.037769+0000","last_clean":"2024-09-21T12:26:22.037769+0000","last_became_active":"2024-09-21T12:23:46.897197+0000","last_became_peered":"2024-09-21T12:23:46.897197+0000","last_unstale":"2024-09-21T12:26:22.037769+0000","last_undegraded":"2024-09-21T12:26:22.037769+0000","last_fullsized":"2024-09-21T12:26:22.037769+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-21T12:25:42.183169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:42.183169+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:03.947737+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":340,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036459+0000","last_change":"2024-09-21T12:26:13.918157+0000","last_active":"2024-09-21T12:26:22.036459+0000","last_peered":"2024-09-21T12:26:22.036459+0000","last_clean":"2024-09-21T12:26:22.036459+0000","last_became_active":"2024-09-21T12:22:50.906886+0000","last_became_peered":"2024-09-21T12:22:50.906886+0000","last_unstale":"2024-09-21T12:26:22.036459+0000","last_undegraded":"2024-09-21T12:26:22.036459+0000","last_fullsized":"2024-09-21T12:26:22.036459+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:13.918088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:13.918088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:43.184195+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":317,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.035862+0000","last_change":"2024-09-21T12:23:17.434377+0000","last_active":"2024-09-21T12:26:22.035862+0000","last_peered":"2024-09-21T12:26:22.035862+0000","last_clean":"2024-09-21T12:26:22.035862+0000","last_became_active":"2024-09-21T12:22:50.927438+0000","last_became_peered":"2024-09-21T12:22:50.927438+0000","last_unstale":"2024-09-21T12:26:22.035862+0000","last_undegraded":"2024-09-21T12:26:22.035862+0000","last_fullsized":"2024-09-21T12:26:22.035862+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:17.434261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:17.434261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:54.989691+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":313,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037872+0000","last_change":"2024-09-21T12:25:39.079146+0000","last_active":"2024-09-21T12:26:22.037872+0000","last_peered":"2024-09-21T12:26:22.037872+0000","last_clean":"2024-09-21T12:26:22.037872+0000","last_became_active":"2024-09-21T12:22:50.913898+0000","last_became_peered":"2024-09-21T12:22:50.913898+0000","last_unstale":"2024-09-21T12:26:22.037872+0000","last_undegraded":"2024-09-21T12:26:22.037872+0000","last_fullsized":"2024-09-21T12:26:22.037872+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:39.079015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:39.079015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:50.881520+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"200'8852","reported_seq":8942,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:32.134559+0000","last_change":"2024-09-21T12:24:46.315895+0000","last_active":"2024-09-21T12:26:32.134559+0000","last_peered":"2024-09-21T12:26:32.134559+0000","last_clean":"2024-09-21T12:26:32.134559+0000","last_became_active":"2024-09-21T12:24:46.315601+0000","last_became_peered":"2024-09-21T12:24:46.315601+0000","last_unstale":"2024-09-21T12:26:32.134559+0000","last_undegraded":"2024-09-21T12:26:32.134559+0000","last_fullsized":"2024-09-21T12:26:32.134559+0000","mapping_epoch":160,"log_start":"182'4700","ondisk_log_start":"182'4700","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4152,"log_dups_size":0,"ondisk_log_size":4152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":494927872,"num_objects":7552,"num_object_clones":0,"num_object_copies":22656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8852,"num_write_kb":524928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"76'79","reported_seq":365,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036059+0000","last_change":"2024-09-21T12:25:43.825573+0000","last_active":"2024-09-21T12:26:22.036059+0000","last_peered":"2024-09-21T12:26:22.036059+0000","last_clean":"2024-09-21T12:26:22.036059+0000","last_became_active":"2024-09-21T12:22:17.072872+0000","last_became_peered":"2024-09-21T12:22:17.072872+0000","last_unstale":"2024-09-21T12:26:22.036059+0000","last_undegraded":"2024-09-21T12:26:22.036059+0000","last_fullsized":"2024-09-21T12:26:22.036059+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:25:43.825511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:25:43.825511+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:08.940139+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":167,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036952+0000","last_change":"2024-09-21T12:26:10.141012+0000","last_active":"2024-09-21T12:26:22.036952+0000","last_peered":"2024-09-21T12:26:22.036952+0000","last_clean":"2024-09-21T12:26:22.036952+0000","last_became_active":"2024-09-21T12:23:46.902260+0000","last_became_peered":"2024-09-21T12:23:46.902260+0000","last_unstale":"2024-09-21T12:26:22.036952+0000","last_undegraded":"2024-09-21T12:26:22.036952+0000","last_fullsized":"2024-09-21T12:26:22.036952+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:26:10.137029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:10.137029+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:19.970928+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":323,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.032982+0000","last_change":"2024-09-21T12:22:50.928240+0000","last_active":"2024-09-21T12:26:22.032982+0000","last_peered":"2024-09-21T12:26:22.032982+0000","last_clean":"2024-09-21T12:26:22.032982+0000","last_became_active":"2024-09-21T12:22:50.927946+0000","last_became_peered":"2024-09-21T12:22:50.927946+0000","last_unstale":"2024-09-21T12:26:22.032982+0000","last_undegraded":"2024-09-21T12:26:22.032982+0000","last_fullsized":"2024-09-21T12:26:22.032982+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:08.999015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:08.999015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:24.102761+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":396,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:25.420661+0000","last_change":"2024-09-21T12:26:13.114174+0000","last_active":"2024-09-21T12:26:25.420661+0000","last_peered":"2024-09-21T12:26:25.420661+0000","last_clean":"2024-09-21T12:26:25.420661+0000","last_became_active":"2024-09-21T12:20:19.307924+0000","last_became_peered":"2024-09-21T12:20:19.307924+0000","last_unstale":"2024-09-21T12:26:25.420661+0000","last_undegraded":"2024-09-21T12:26:25.420661+0000","last_fullsized":"2024-09-21T12:26:25.420661+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:26:13.113863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:13.113863+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:23.394893+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":236,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038472+0000","last_change":"2024-09-21T12:23:56.471960+0000","last_active":"2024-09-21T12:26:22.038472+0000","last_peered":"2024-09-21T12:26:22.038472+0000","last_clean":"2024-09-21T12:26:22.038472+0000","last_became_active":"2024-09-21T12:22:50.905470+0000","last_became_peered":"2024-09-21T12:22:50.905470+0000","last_unstale":"2024-09-21T12:26:22.038472+0000","last_undegraded":"2024-09-21T12:26:22.038472+0000","last_fullsized":"2024-09-21T12:26:22.038472+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:56.471116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:56.471116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:22.516139+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.7","version":"200'8766","reported_seq":8857,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:31.845552+0000","last_change":"2024-09-21T12:24:46.317724+0000","last_active":"2024-09-21T12:26:31.845552+0000","last_peered":"2024-09-21T12:26:31.845552+0000","last_clean":"2024-09-21T12:26:31.845552+0000","last_became_active":"2024-09-21T12:24:46.317468+0000","last_became_peered":"2024-09-21T12:24:46.317468+0000","last_unstale":"2024-09-21T12:26:31.845552+0000","last_undegraded":"2024-09-21T12:26:31.845552+0000","last_fullsized":"2024-09-21T12:26:31.845552+0000","mapping_epoch":160,"log_start":"181'4500","ondisk_log_start":"181'4500","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4266,"log_dups_size":0,"ondisk_log_size":4266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:01.017517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":489553920,"num_objects":7470,"num_object_clones":0,"num_object_copies":22410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8766,"num_write_kb":519552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":383,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.028694+0000","last_change":"2024-09-21T12:25:00.237083+0000","last_active":"2024-09-21T12:26:22.028694+0000","last_peered":"2024-09-21T12:26:22.028694+0000","last_clean":"2024-09-21T12:26:22.028694+0000","last_became_active":"2024-09-21T12:22:50.922343+0000","last_became_peered":"2024-09-21T12:22:50.922343+0000","last_unstale":"2024-09-21T12:26:22.028694+0000","last_undegraded":"2024-09-21T12:26:22.028694+0000","last_fullsized":"2024-09-21T12:26:22.028694+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:25:00.236990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:00.236990+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:26.107161+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":319,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039938+0000","last_change":"2024-09-21T12:25:30.451939+0000","last_active":"2024-09-21T12:26:22.039938+0000","last_peered":"2024-09-21T12:26:22.039938+0000","last_clean":"2024-09-21T12:26:22.039938+0000","last_became_active":"2024-09-21T12:20:15.302262+0000","last_became_peered":"2024-09-21T12:20:15.302262+0000","last_unstale":"2024-09-21T12:26:22.039938+0000","last_undegraded":"2024-09-21T12:26:22.039938+0000","last_fullsized":"2024-09-21T12:26:22.039938+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:30.451876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:30.451876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:51.913189+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":333,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037402+0000","last_change":"2024-09-21T12:25:36.012066+0000","last_active":"2024-09-21T12:26:22.037402+0000","last_peered":"2024-09-21T12:26:22.037402+0000","last_clean":"2024-09-21T12:26:22.037402+0000","last_became_active":"2024-09-21T12:22:50.925974+0000","last_became_peered":"2024-09-21T12:22:50.925974+0000","last_unstale":"2024-09-21T12:26:22.037402+0000","last_undegraded":"2024-09-21T12:26:22.037402+0000","last_fullsized":"2024-09-21T12:26:22.037402+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:36.011984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:36.011984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:40.801014+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":292,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.029754+0000","last_change":"2024-09-21T12:23:28.121344+0000","last_active":"2024-09-21T12:26:22.029754+0000","last_peered":"2024-09-21T12:26:22.029754+0000","last_clean":"2024-09-21T12:26:22.029754+0000","last_became_active":"2024-09-21T12:20:21.311493+0000","last_became_peered":"2024-09-21T12:20:21.311493+0000","last_unstale":"2024-09-21T12:26:22.029754+0000","last_undegraded":"2024-09-21T12:26:22.029754+0000","last_fullsized":"2024-09-21T12:26:22.029754+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:28.121286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:28.121286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:24.681035+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"200'8840","reported_seq":8930,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:32.227344+0000","last_change":"2024-09-21T12:24:46.327091+0000","last_active":"2024-09-21T12:26:32.227344+0000","last_peered":"2024-09-21T12:26:32.227344+0000","last_clean":"2024-09-21T12:26:32.227344+0000","last_became_active":"2024-09-21T12:24:46.325838+0000","last_became_peered":"2024-09-21T12:24:46.325838+0000","last_unstale":"2024-09-21T12:26:32.227344+0000","last_undegraded":"2024-09-21T12:26:32.227344+0000","last_fullsized":"2024-09-21T12:26:32.227344+0000","mapping_epoch":160,"log_start":"181'4500","ondisk_log_start":"181'4500","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4340,"log_dups_size":0,"ondisk_log_size":4340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":490602496,"num_objects":7486,"num_object_clones":0,"num_object_copies":22458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8840,"num_write_kb":522432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":338,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036212+0000","last_change":"2024-09-21T12:24:02.960807+0000","last_active":"2024-09-21T12:26:22.036212+0000","last_peered":"2024-09-21T12:26:22.036212+0000","last_clean":"2024-09-21T12:26:22.036212+0000","last_became_active":"2024-09-21T12:20:17.692867+0000","last_became_peered":"2024-09-21T12:20:17.692867+0000","last_unstale":"2024-09-21T12:26:22.036212+0000","last_undegraded":"2024-09-21T12:26:22.036212+0000","last_fullsized":"2024-09-21T12:26:22.036212+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-21T12:24:02.960733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:02.960733+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":303,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038040+0000","last_change":"2024-09-21T12:23:59.046000+0000","last_active":"2024-09-21T12:26:22.038040+0000","last_peered":"2024-09-21T12:26:22.038040+0000","last_clean":"2024-09-21T12:26:22.038040+0000","last_became_active":"2024-09-21T12:20:15.299333+0000","last_became_peered":"2024-09-21T12:20:15.299333+0000","last_unstale":"2024-09-21T12:26:22.038040+0000","last_undegraded":"2024-09-21T12:26:22.038040+0000","last_fullsized":"2024-09-21T12:26:22.038040+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:59.045927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:23:59.045927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:55.760821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":296,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037229+0000","last_change":"2024-09-21T12:24:34.060244+0000","last_active":"2024-09-21T12:26:22.037229+0000","last_peered":"2024-09-21T12:26:22.037229+0000","last_clean":"2024-09-21T12:26:22.037229+0000","last_became_active":"2024-09-21T12:20:19.306952+0000","last_became_peered":"2024-09-21T12:20:19.306952+0000","last_unstale":"2024-09-21T12:26:22.037229+0000","last_undegraded":"2024-09-21T12:26:22.037229+0000","last_fullsized":"2024-09-21T12:26:22.037229+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:34.060188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:34.060188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:50.161659+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":283,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036044+0000","last_change":"2024-09-21T12:20:21.314891+0000","last_active":"2024-09-21T12:26:22.036044+0000","last_peered":"2024-09-21T12:26:22.036044+0000","last_clean":"2024-09-21T12:26:22.036044+0000","last_became_active":"2024-09-21T12:20:21.314545+0000","last_became_peered":"2024-09-21T12:20:21.314545+0000","last_unstale":"2024-09-21T12:26:22.036044+0000","last_undegraded":"2024-09-21T12:26:22.036044+0000","last_fullsized":"2024-09-21T12:26:22.036044+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:49.780292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"200'8739","reported_seq":8842,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:31.637283+0000","last_change":"2024-09-21T12:25:49.210492+0000","last_active":"2024-09-21T12:26:31.637283+0000","last_peered":"2024-09-21T12:26:31.637283+0000","last_clean":"2024-09-21T12:26:31.637283+0000","last_became_active":"2024-09-21T12:24:46.324554+0000","last_became_peered":"2024-09-21T12:24:46.324554+0000","last_unstale":"2024-09-21T12:26:31.637283+0000","last_undegraded":"2024-09-21T12:26:31.637283+0000","last_fullsized":"2024-09-21T12:26:31.637283+0000","mapping_epoch":160,"log_start":"179'4100","ondisk_log_start":"179'4100","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":899,"log_size":4639,"log_dups_size":0,"ondisk_log_size":4639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":494600192,"num_objects":7547,"num_object_clones":0,"num_object_copies":22641,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8739,"num_write_kb":521152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":311,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.035955+0000","last_change":"2024-09-21T12:24:54.445801+0000","last_active":"2024-09-21T12:26:22.035955+0000","last_peered":"2024-09-21T12:26:22.035955+0000","last_clean":"2024-09-21T12:26:22.035955+0000","last_became_active":"2024-09-21T12:22:50.910973+0000","last_became_peered":"2024-09-21T12:22:50.910973+0000","last_unstale":"2024-09-21T12:26:22.035955+0000","last_undegraded":"2024-09-21T12:26:22.035955+0000","last_fullsized":"2024-09-21T12:26:22.035955+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:24:54.445714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:54.445714+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":332,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038093+0000","last_change":"2024-09-21T12:25:44.800799+0000","last_active":"2024-09-21T12:26:22.038093+0000","last_peered":"2024-09-21T12:26:22.038093+0000","last_clean":"2024-09-21T12:26:22.038093+0000","last_became_active":"2024-09-21T12:22:50.927859+0000","last_became_peered":"2024-09-21T12:22:50.927859+0000","last_unstale":"2024-09-21T12:26:22.038093+0000","last_undegraded":"2024-09-21T12:26:22.038093+0000","last_fullsized":"2024-09-21T12:26:22.038093+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:44.800657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:44.800657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:49.068224+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":304,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036702+0000","last_change":"2024-09-21T12:25:40.148844+0000","last_active":"2024-09-21T12:26:22.036702+0000","last_peered":"2024-09-21T12:26:22.036702+0000","last_clean":"2024-09-21T12:26:22.036702+0000","last_became_active":"2024-09-21T12:20:19.312572+0000","last_became_peered":"2024-09-21T12:20:19.312572+0000","last_unstale":"2024-09-21T12:26:22.036702+0000","last_undegraded":"2024-09-21T12:26:22.036702+0000","last_fullsized":"2024-09-21T12:26:22.036702+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:40.148766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:40.148766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:07.648989+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":290,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.032652+0000","last_change":"2024-09-21T12:22:36.006177+0000","last_active":"2024-09-21T12:26:22.032652+0000","last_peered":"2024-09-21T12:26:22.032652+0000","last_clean":"2024-09-21T12:26:22.032652+0000","last_became_active":"2024-09-21T12:20:21.317223+0000","last_became_peered":"2024-09-21T12:20:21.317223+0000","last_unstale":"2024-09-21T12:26:22.032652+0000","last_undegraded":"2024-09-21T12:26:22.032652+0000","last_fullsized":"2024-09-21T12:26:22.032652+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:36.006109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:36.006109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:54.431499+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"200'8835","reported_seq":8926,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:31.626353+0000","last_change":"2024-09-21T12:24:46.323500+0000","last_active":"2024-09-21T12:26:31.626353+0000","last_peered":"2024-09-21T12:26:31.626353+0000","last_clean":"2024-09-21T12:26:31.626353+0000","last_became_active":"2024-09-21T12:24:46.323192+0000","last_became_peered":"2024-09-21T12:24:46.323192+0000","last_unstale":"2024-09-21T12:26:31.626353+0000","last_undegraded":"2024-09-21T12:26:31.626353+0000","last_fullsized":"2024-09-21T12:26:31.626353+0000","mapping_epoch":160,"log_start":"180'4400","ondisk_log_start":"180'4400","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4435,"log_dups_size":0,"ondisk_log_size":4435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":498663424,"num_objects":7609,"num_object_clones":0,"num_object_copies":22827,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8835,"num_write_kb":526208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":340,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039658+0000","last_change":"2024-09-21T12:25:03.416100+0000","last_active":"2024-09-21T12:26:22.039658+0000","last_peered":"2024-09-21T12:26:22.039658+0000","last_clean":"2024-09-21T12:26:22.039658+0000","last_became_active":"2024-09-21T12:20:17.692870+0000","last_became_peered":"2024-09-21T12:20:17.692870+0000","last_unstale":"2024-09-21T12:26:22.039658+0000","last_undegraded":"2024-09-21T12:26:22.039658+0000","last_fullsized":"2024-09-21T12:26:22.039658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:25:03.416033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:03.416033+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":362,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038114+0000","last_change":"2024-09-21T12:24:29.054851+0000","last_active":"2024-09-21T12:26:22.038114+0000","last_peered":"2024-09-21T12:26:22.038114+0000","last_clean":"2024-09-21T12:26:22.038114+0000","last_became_active":"2024-09-21T12:22:50.927141+0000","last_became_peered":"2024-09-21T12:22:50.927141+0000","last_unstale":"2024-09-21T12:26:22.038114+0000","last_undegraded":"2024-09-21T12:26:22.038114+0000","last_fullsized":"2024-09-21T12:26:22.038114+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-21T12:24:29.054772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:29.054772+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:07.827845+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"121'11","reported_seq":412,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:25.420580+0000","last_change":"2024-09-21T12:25:02.916145+0000","last_active":"2024-09-21T12:26:25.420580+0000","last_peered":"2024-09-21T12:26:25.420580+0000","last_clean":"2024-09-21T12:26:25.420580+0000","last_became_active":"2024-09-21T12:22:50.908371+0000","last_became_peered":"2024-09-21T12:22:50.908371+0000","last_unstale":"2024-09-21T12:26:25.420580+0000","last_undegraded":"2024-09-21T12:26:25.420580+0000","last_fullsized":"2024-09-21T12:26:25.420580+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:25:02.916090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:02.916090+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:31.534554+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":291,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039512+0000","last_change":"2024-09-21T12:22:45.223570+0000","last_active":"2024-09-21T12:26:22.039512+0000","last_peered":"2024-09-21T12:26:22.039512+0000","last_clean":"2024-09-21T12:26:22.039512+0000","last_became_active":"2024-09-21T12:20:21.592576+0000","last_became_peered":"2024-09-21T12:20:21.592576+0000","last_unstale":"2024-09-21T12:26:22.039512+0000","last_undegraded":"2024-09-21T12:26:22.039512+0000","last_fullsized":"2024-09-21T12:26:22.039512+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:45.223396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:45.223396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":327,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036891+0000","last_change":"2024-09-21T12:25:31.845118+0000","last_active":"2024-09-21T12:26:22.036891+0000","last_peered":"2024-09-21T12:26:22.036891+0000","last_clean":"2024-09-21T12:26:22.036891+0000","last_became_active":"2024-09-21T12:20:15.303480+0000","last_became_peered":"2024-09-21T12:20:15.303480+0000","last_unstale":"2024-09-21T12:26:22.036891+0000","last_undegraded":"2024-09-21T12:26:22.036891+0000","last_fullsized":"2024-09-21T12:26:22.036891+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:31.845066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:31.845066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:34.886910+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":307,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.030096+0000","last_change":"2024-09-21T12:22:36.999995+0000","last_active":"2024-09-21T12:26:22.030096+0000","last_peered":"2024-09-21T12:26:22.030096+0000","last_clean":"2024-09-21T12:26:22.030096+0000","last_became_active":"2024-09-21T12:20:17.694628+0000","last_became_peered":"2024-09-21T12:20:17.694628+0000","last_unstale":"2024-09-21T12:26:22.030096+0000","last_undegraded":"2024-09-21T12:26:22.030096+0000","last_fullsized":"2024-09-21T12:26:22.030096+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-21T12:22:36.999705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:36.999705+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:51.345667+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":311,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039758+0000","last_change":"2024-09-21T12:25:02.414377+0000","last_active":"2024-09-21T12:26:22.039758+0000","last_peered":"2024-09-21T12:26:22.039758+0000","last_clean":"2024-09-21T12:26:22.039758+0000","last_became_active":"2024-09-21T12:20:19.313882+0000","last_became_peered":"2024-09-21T12:20:19.313882+0000","last_unstale":"2024-09-21T12:26:22.039758+0000","last_undegraded":"2024-09-21T12:26:22.039758+0000","last_fullsized":"2024-09-21T12:26:22.039758+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:02.414298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:02.414298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":312,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037625+0000","last_change":"2024-09-21T12:24:00.066522+0000","last_active":"2024-09-21T12:26:22.037625+0000","last_peered":"2024-09-21T12:26:22.037625+0000","last_clean":"2024-09-21T12:26:22.037625+0000","last_became_active":"2024-09-21T12:22:50.904404+0000","last_became_peered":"2024-09-21T12:22:50.904404+0000","last_unstale":"2024-09-21T12:26:22.037625+0000","last_undegraded":"2024-09-21T12:26:22.037625+0000","last_fullsized":"2024-09-21T12:26:22.037625+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:00.066451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:00.066451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:01.771403+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":324,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.029014+0000","last_change":"2024-09-21T12:26:20.767526+0000","last_active":"2024-09-21T12:26:22.029014+0000","last_peered":"2024-09-21T12:26:22.029014+0000","last_clean":"2024-09-21T12:26:22.029014+0000","last_became_active":"2024-09-21T12:20:17.695898+0000","last_became_peered":"2024-09-21T12:20:17.695898+0000","last_unstale":"2024-09-21T12:26:22.029014+0000","last_undegraded":"2024-09-21T12:26:22.029014+0000","last_fullsized":"2024-09-21T12:26:22.029014+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:26:20.767293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:20.767293+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:26.917942+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":340,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036233+0000","last_change":"2024-09-21T12:26:16.907351+0000","last_active":"2024-09-21T12:26:22.036233+0000","last_peered":"2024-09-21T12:26:22.036233+0000","last_clean":"2024-09-21T12:26:22.036233+0000","last_became_active":"2024-09-21T12:22:50.927649+0000","last_became_peered":"2024-09-21T12:22:50.927649+0000","last_unstale":"2024-09-21T12:26:22.036233+0000","last_undegraded":"2024-09-21T12:26:22.036233+0000","last_fullsized":"2024-09-21T12:26:22.036233+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:16.907284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:16.907284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:30.534090+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":294,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.029889+0000","last_change":"2024-09-21T12:23:53.356954+0000","last_active":"2024-09-21T12:26:22.029889+0000","last_peered":"2024-09-21T12:26:22.029889+0000","last_clean":"2024-09-21T12:26:22.029889+0000","last_became_active":"2024-09-21T12:20:19.311947+0000","last_became_peered":"2024-09-21T12:20:19.311947+0000","last_unstale":"2024-09-21T12:26:22.029889+0000","last_undegraded":"2024-09-21T12:26:22.029889+0000","last_fullsized":"2024-09-21T12:26:22.029889+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:53.356896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:53.356896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:28.202782+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":337,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037217+0000","last_change":"2024-09-21T12:26:11.105716+0000","last_active":"2024-09-21T12:26:22.037217+0000","last_peered":"2024-09-21T12:26:22.037217+0000","last_clean":"2024-09-21T12:26:22.037217+0000","last_became_active":"2024-09-21T12:22:50.924931+0000","last_became_peered":"2024-09-21T12:22:50.924931+0000","last_unstale":"2024-09-21T12:26:22.037217+0000","last_undegraded":"2024-09-21T12:26:22.037217+0000","last_fullsized":"2024-09-21T12:26:22.037217+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:11.105583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:11.105583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:36.449727+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":340,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.040057+0000","last_change":"2024-09-21T12:26:18.481098+0000","last_active":"2024-09-21T12:26:22.040057+0000","last_peered":"2024-09-21T12:26:22.040057+0000","last_clean":"2024-09-21T12:26:22.040057+0000","last_became_active":"2024-09-21T12:23:46.898523+0000","last_became_peered":"2024-09-21T12:23:46.898523+0000","last_unstale":"2024-09-21T12:26:22.040057+0000","last_undegraded":"2024-09-21T12:26:22.040057+0000","last_fullsized":"2024-09-21T12:26:22.040057+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:26:18.480630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:18.480630+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:39.872336+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":312,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.027247+0000","last_change":"2024-09-21T12:24:25.931617+0000","last_active":"2024-09-21T12:26:22.027247+0000","last_peered":"2024-09-21T12:26:22.027247+0000","last_clean":"2024-09-21T12:26:22.027247+0000","last_became_active":"2024-09-21T12:20:15.297519+0000","last_became_peered":"2024-09-21T12:20:15.297519+0000","last_unstale":"2024-09-21T12:26:22.027247+0000","last_undegraded":"2024-09-21T12:26:22.027247+0000","last_fullsized":"2024-09-21T12:26:22.027247+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:25.931561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:25.931561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:54.835832+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":378,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:25.436710+0000","last_change":"2024-09-21T12:23:57.379033+0000","last_active":"2024-09-21T12:26:25.436710+0000","last_peered":"2024-09-21T12:26:25.436710+0000","last_clean":"2024-09-21T12:26:25.436710+0000","last_became_active":"2024-09-21T12:20:19.305364+0000","last_became_peered":"2024-09-21T12:20:19.305364+0000","last_unstale":"2024-09-21T12:26:25.436710+0000","last_undegraded":"2024-09-21T12:26:25.436710+0000","last_fullsized":"2024-09-21T12:26:25.436710+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:23:57.378959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:57.378959+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:29.503205+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":236,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038000+0000","last_change":"2024-09-21T12:24:51.831043+0000","last_active":"2024-09-21T12:26:22.038000+0000","last_peered":"2024-09-21T12:26:22.038000+0000","last_clean":"2024-09-21T12:26:22.038000+0000","last_became_active":"2024-09-21T12:22:50.924770+0000","last_became_peered":"2024-09-21T12:22:50.924770+0000","last_unstale":"2024-09-21T12:26:22.038000+0000","last_undegraded":"2024-09-21T12:26:22.038000+0000","last_fullsized":"2024-09-21T12:26:22.038000+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:51.830994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:51.830994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:58.084812+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":321,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.031070+0000","last_change":"2024-09-21T12:24:50.727161+0000","last_active":"2024-09-21T12:26:22.031070+0000","last_peered":"2024-09-21T12:26:22.031070+0000","last_clean":"2024-09-21T12:26:22.031070+0000","last_became_active":"2024-09-21T12:20:17.693209+0000","last_became_peered":"2024-09-21T12:20:17.693209+0000","last_unstale":"2024-09-21T12:26:22.031070+0000","last_undegraded":"2024-09-21T12:26:22.031070+0000","last_fullsized":"2024-09-21T12:26:22.031070+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:24:50.727101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:50.727101+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":332,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037049+0000","last_change":"2024-09-21T12:25:36.808211+0000","last_active":"2024-09-21T12:26:22.037049+0000","last_peered":"2024-09-21T12:26:22.037049+0000","last_clean":"2024-09-21T12:26:22.037049+0000","last_became_active":"2024-09-21T12:22:50.908238+0000","last_became_peered":"2024-09-21T12:22:50.908238+0000","last_unstale":"2024-09-21T12:26:22.037049+0000","last_undegraded":"2024-09-21T12:26:22.037049+0000","last_fullsized":"2024-09-21T12:26:22.037049+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:36.808147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:36.808147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:50.893952+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":379,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:25.420496+0000","last_change":"2024-09-21T12:22:46.216242+0000","last_active":"2024-09-21T12:26:25.420496+0000","last_peered":"2024-09-21T12:26:25.420496+0000","last_clean":"2024-09-21T12:26:25.420496+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:26:25.420496+0000","last_undegraded":"2024-09-21T12:26:25.420496+0000","last_fullsized":"2024-09-21T12:26:25.420496+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:22:46.216153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:46.216153+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:27.470181+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":292,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.028564+0000","last_change":"2024-09-21T12:26:08.926904+0000","last_active":"2024-09-21T12:26:22.028564+0000","last_peered":"2024-09-21T12:26:22.028564+0000","last_clean":"2024-09-21T12:26:22.028564+0000","last_became_active":"2024-09-21T12:20:21.309385+0000","last_became_peered":"2024-09-21T12:20:21.309385+0000","last_unstale":"2024-09-21T12:26:22.028564+0000","last_undegraded":"2024-09-21T12:26:22.028564+0000","last_fullsized":"2024-09-21T12:26:22.028564+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:08.923458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:08.923458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:38.391592+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":291,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037349+0000","last_change":"2024-09-21T12:23:59.046758+0000","last_active":"2024-09-21T12:26:22.037349+0000","last_peered":"2024-09-21T12:26:22.037349+0000","last_clean":"2024-09-21T12:26:22.037349+0000","last_became_active":"2024-09-21T12:22:50.906473+0000","last_became_peered":"2024-09-21T12:22:50.906473+0000","last_unstale":"2024-09-21T12:26:22.037349+0000","last_undegraded":"2024-09-21T12:26:22.037349+0000","last_fullsized":"2024-09-21T12:26:22.037349+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:23:59.046704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:59.046704+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":228,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037671+0000","last_change":"2024-09-21T12:24:58.948698+0000","last_active":"2024-09-21T12:26:22.037671+0000","last_peered":"2024-09-21T12:26:22.037671+0000","last_clean":"2024-09-21T12:26:22.037671+0000","last_became_active":"2024-09-21T12:22:50.903705+0000","last_became_peered":"2024-09-21T12:22:50.903705+0000","last_unstale":"2024-09-21T12:26:22.037671+0000","last_undegraded":"2024-09-21T12:26:22.037671+0000","last_fullsized":"2024-09-21T12:26:22.037671+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:58.948625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:58.948625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:23.221920+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":299,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038013+0000","last_change":"2024-09-21T12:26:16.149823+0000","last_active":"2024-09-21T12:26:22.038013+0000","last_peered":"2024-09-21T12:26:22.038013+0000","last_clean":"2024-09-21T12:26:22.038013+0000","last_became_active":"2024-09-21T12:20:19.315418+0000","last_became_peered":"2024-09-21T12:20:19.315418+0000","last_unstale":"2024-09-21T12:26:22.038013+0000","last_undegraded":"2024-09-21T12:26:22.038013+0000","last_fullsized":"2024-09-21T12:26:22.038013+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:16.149550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:16.149550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:27.681950+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":298,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.029203+0000","last_change":"2024-09-21T12:24:56.879617+0000","last_active":"2024-09-21T12:26:22.029203+0000","last_peered":"2024-09-21T12:26:22.029203+0000","last_clean":"2024-09-21T12:26:22.029203+0000","last_became_active":"2024-09-21T12:20:21.310053+0000","last_became_peered":"2024-09-21T12:20:21.310053+0000","last_unstale":"2024-09-21T12:26:22.029203+0000","last_undegraded":"2024-09-21T12:26:22.029203+0000","last_fullsized":"2024-09-21T12:26:22.029203+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:56.879558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:56.879558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:26.453298+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":316,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038245+0000","last_change":"2024-09-21T12:25:41.888249+0000","last_active":"2024-09-21T12:26:22.038245+0000","last_peered":"2024-09-21T12:26:22.038245+0000","last_clean":"2024-09-21T12:26:22.038245+0000","last_became_active":"2024-09-21T12:20:17.299015+0000","last_became_peered":"2024-09-21T12:20:17.299015+0000","last_unstale":"2024-09-21T12:26:22.038245+0000","last_undegraded":"2024-09-21T12:26:22.038245+0000","last_fullsized":"2024-09-21T12:26:22.038245+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:25:41.888037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:41.888037+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:10.841319+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":340,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037922+0000","last_change":"2024-09-21T12:26:19.973528+0000","last_active":"2024-09-21T12:26:22.037922+0000","last_peered":"2024-09-21T12:26:22.037922+0000","last_clean":"2024-09-21T12:26:22.037922+0000","last_became_active":"2024-09-21T12:22:50.922019+0000","last_became_peered":"2024-09-21T12:22:50.922019+0000","last_unstale":"2024-09-21T12:26:22.037922+0000","last_undegraded":"2024-09-21T12:26:22.037922+0000","last_fullsized":"2024-09-21T12:26:22.037922+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:19.973461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:19.973461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:36.925984+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":302,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.032732+0000","last_change":"2024-09-21T12:26:21.572474+0000","last_active":"2024-09-21T12:26:22.032732+0000","last_peered":"2024-09-21T12:26:22.032732+0000","last_clean":"2024-09-21T12:26:22.032732+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:26:22.032732+0000","last_undegraded":"2024-09-21T12:26:22.032732+0000","last_fullsized":"2024-09-21T12:26:22.032732+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:21.571726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:21.571726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:50.797908+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":292,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.026388+0000","last_change":"2024-09-21T12:23:29.134829+0000","last_active":"2024-09-21T12:26:22.026388+0000","last_peered":"2024-09-21T12:26:22.026388+0000","last_clean":"2024-09-21T12:26:22.026388+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:26:22.026388+0000","last_undegraded":"2024-09-21T12:26:22.026388+0000","last_fullsized":"2024-09-21T12:26:22.026388+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:29.134767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:29.134767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":325,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.031870+0000","last_change":"2024-09-21T12:24:58.824615+0000","last_active":"2024-09-21T12:26:22.031870+0000","last_peered":"2024-09-21T12:26:22.031870+0000","last_clean":"2024-09-21T12:26:22.031870+0000","last_became_active":"2024-09-21T12:20:17.694794+0000","last_became_peered":"2024-09-21T12:20:17.694794+0000","last_unstale":"2024-09-21T12:26:22.031870+0000","last_undegraded":"2024-09-21T12:26:22.031870+0000","last_fullsized":"2024-09-21T12:26:22.031870+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:24:58.824516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:58.824516+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:24.775664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":319,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.039911+0000","last_change":"2024-09-21T12:24:33.364225+0000","last_active":"2024-09-21T12:26:22.039911+0000","last_peered":"2024-09-21T12:26:22.039911+0000","last_clean":"2024-09-21T12:26:22.039911+0000","last_became_active":"2024-09-21T12:20:15.295968+0000","last_became_peered":"2024-09-21T12:20:15.295968+0000","last_unstale":"2024-09-21T12:26:22.039911+0000","last_undegraded":"2024-09-21T12:26:22.039911+0000","last_fullsized":"2024-09-21T12:26:22.039911+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:33.363808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:33.363808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":316,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.035666+0000","last_change":"2024-09-21T12:23:57.018797+0000","last_active":"2024-09-21T12:26:22.035666+0000","last_peered":"2024-09-21T12:26:22.035666+0000","last_clean":"2024-09-21T12:26:22.035666+0000","last_became_active":"2024-09-21T12:22:50.921604+0000","last_became_peered":"2024-09-21T12:22:50.921604+0000","last_unstale":"2024-09-21T12:26:22.035666+0000","last_undegraded":"2024-09-21T12:26:22.035666+0000","last_fullsized":"2024-09-21T12:26:22.035666+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:57.018749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:57.018749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:53.766292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":290,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.032902+0000","last_change":"2024-09-21T12:24:35.588409+0000","last_active":"2024-09-21T12:26:22.032902+0000","last_peered":"2024-09-21T12:26:22.032902+0000","last_clean":"2024-09-21T12:26:22.032902+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:26:22.032902+0000","last_undegraded":"2024-09-21T12:26:22.032902+0000","last_fullsized":"2024-09-21T12:26:22.032902+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:35.588353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:35.588353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":304,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.031637+0000","last_change":"2024-09-21T12:23:51.343568+0000","last_active":"2024-09-21T12:26:22.031637+0000","last_peered":"2024-09-21T12:26:22.031637+0000","last_clean":"2024-09-21T12:26:22.031637+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:26:22.031637+0000","last_undegraded":"2024-09-21T12:26:22.031637+0000","last_fullsized":"2024-09-21T12:26:22.031637+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-21T12:23:51.343491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:51.343491+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:26.153152+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":318,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.028587+0000","last_change":"2024-09-21T12:25:48.713463+0000","last_active":"2024-09-21T12:26:22.028587+0000","last_peered":"2024-09-21T12:26:22.028587+0000","last_clean":"2024-09-21T12:26:22.028587+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:26:22.028587+0000","last_undegraded":"2024-09-21T12:26:22.028587+0000","last_fullsized":"2024-09-21T12:26:22.028587+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:25:48.713392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:48.713392+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:06.809432+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":324,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.037159+0000","last_change":"2024-09-21T12:26:22.014941+0000","last_active":"2024-09-21T12:26:22.037159+0000","last_peered":"2024-09-21T12:26:22.037159+0000","last_clean":"2024-09-21T12:26:22.037159+0000","last_became_active":"2024-09-21T12:22:50.930615+0000","last_became_peered":"2024-09-21T12:22:50.930615+0000","last_unstale":"2024-09-21T12:26:22.037159+0000","last_undegraded":"2024-09-21T12:26:22.037159+0000","last_fullsized":"2024-09-21T12:26:22.037159+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:22.014896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:22.014896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:34.583908+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":312,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036445+0000","last_change":"2024-09-21T12:26:14.855364+0000","last_active":"2024-09-21T12:26:22.036445+0000","last_peered":"2024-09-21T12:26:22.036445+0000","last_clean":"2024-09-21T12:26:22.036445+0000","last_became_active":"2024-09-21T12:22:50.917408+0000","last_became_peered":"2024-09-21T12:22:50.917408+0000","last_unstale":"2024-09-21T12:26:22.036445+0000","last_undegraded":"2024-09-21T12:26:22.036445+0000","last_fullsized":"2024-09-21T12:26:22.036445+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:14.855248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:14.855248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:15.644031+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":313,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.036030+0000","last_change":"2024-09-21T12:26:15.152613+0000","last_active":"2024-09-21T12:26:22.036030+0000","last_peered":"2024-09-21T12:26:22.036030+0000","last_clean":"2024-09-21T12:26:22.036030+0000","last_became_active":"2024-09-21T12:22:50.914221+0000","last_became_peered":"2024-09-21T12:22:50.914221+0000","last_unstale":"2024-09-21T12:26:22.036030+0000","last_undegraded":"2024-09-21T12:26:22.036030+0000","last_fullsized":"2024-09-21T12:26:22.036030+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:15.152540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:15.152540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:17.180451+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":310,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.031512+0000","last_change":"2024-09-21T12:25:00.871133+0000","last_active":"2024-09-21T12:26:22.031512+0000","last_peered":"2024-09-21T12:26:22.031512+0000","last_clean":"2024-09-21T12:26:22.031512+0000","last_became_active":"2024-09-21T12:20:15.297993+0000","last_became_peered":"2024-09-21T12:20:15.297993+0000","last_unstale":"2024-09-21T12:26:22.031512+0000","last_undegraded":"2024-09-21T12:26:22.031512+0000","last_fullsized":"2024-09-21T12:26:22.031512+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:00.871062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:00.871062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":319,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.038440+0000","last_change":"2024-09-21T12:25:47.974973+0000","last_active":"2024-09-21T12:26:22.038440+0000","last_peered":"2024-09-21T12:26:22.038440+0000","last_clean":"2024-09-21T12:26:22.038440+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:26:22.038440+0000","last_undegraded":"2024-09-21T12:26:22.038440+0000","last_fullsized":"2024-09-21T12:26:22.038440+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:25:47.974897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:47.974897+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:04.942357+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":313,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T12:26:22.028768+0000","last_change":"2024-09-21T12:25:32.660054+0000","last_active":"2024-09-21T12:26:22.028768+0000","last_peered":"2024-09-21T12:26:22.028768+0000","last_clean":"2024-09-21T12:26:22.028768+0000","last_became_active":"2024-09-21T12:20:19.308247+0000","last_became_peered":"2024-09-21T12:20:19.308247+0000","last_unstale":"2024-09-21T12:26:22.028768+0000","last_undegraded":"2024-09-21T12:26:22.028768+0000","last_fullsized":"2024-09-21T12:26:22.028768+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:32.659918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:32.659918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:26:46.283244+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":7866613784,"num_objects":120036,"num_object_clones":0,"num_object_copies":360108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":120036,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140792,"num_write_kb":8346433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23619645440,"data_stored":23619633224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69541,"ondisk_log_size":69541,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":690,"num_read_kb":447,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":201,"num_bytes_recovered":3982,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":396,"ondisk_log_size":396,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":5023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":371,"num_read_kb":366,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234063,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3157676,"kb_used_data":2905756,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":90587476,"statfs":{"total":95995035648,"available":92761575424,"internally_reserved":0,"allocated":2975494144,"data_stored":2975123008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":257948106},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462999,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3170472,"kb_used_data":2914392,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":90574680,"statfs":{"total":95995035648,"available":92748472320,"internally_reserved":0,"allocated":2984337408,"data_stored":2983970135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":262207942},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724634,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4070448,"kb_used_data":3858784,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":89674704,"statfs":{"total":95995035648,"available":91826896896,"internally_reserved":0,"allocated":3951394816,"data_stored":3951021018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953570,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149800,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3699012,"kb_used_data":3384628,"kb_used_omap":1,"kb_used_meta":314366,"kb_avail":90046140,"statfs":{"total":95995035648,"available":92207247360,"internally_reserved":0,"allocated":3465859072,"data_stored":3465465179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":321911239},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411438,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3624160,"kb_used_data":3386128,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":90120992,"statfs":{"total":95995035648,"available":92283895808,"internally_reserved":0,"allocated":3467395072,"data_stored":3467036115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243726789},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574964,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3660812,"kb_used_data":3381116,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":90084340,"statfs":{"total":95995035648,"available":92246364160,"internally_reserved":0,"allocated":3462262784,"data_stored":3461859175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":286390725},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738490,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3636784,"kb_used_data":3370848,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":90108368,"statfs":{"total":95995035648,"available":92270968832,"internally_reserved":0,"allocated":3451748352,"data_stored":3451373809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272169413},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3433431040,"data_stored":3433431040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3441164288,"data_stored":3441164288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3445362688,"data_stored":3445358616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3438149632,"data_stored":3438149632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3944681472,"data_stored":3944677400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2960195584,"data_stored":2960195584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2956660736,"data_stored":2956656664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:26:39.239 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T12:26:39.240 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pg_num 21 2024-09-21T12:26:39.240 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:38 smithi183 ceph-mon[33277]: pgmap v305: 158 pgs: 158 active+clean; 6.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-21T12:26:39.311 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:39.358 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:39.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:39.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:39.459 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:39.488 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:39.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:39.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:39.590 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:39.616 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:39.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:39.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:39.717 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:39.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:39.768 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:39.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:39.827 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T12:26:39.868 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:39.869 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:39.891 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:39.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:39.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:39.917 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:39.992 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:40.022 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:40.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:40.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:40.124 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:40.152 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:40.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:40.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:40.254 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:40.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:40.280 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:40.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:40.381 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:40.415 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:40.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:40.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:40.516 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:40.547 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:40.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:40.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:40.648 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:40.676 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:40.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:40.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:40.757 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:26:40.777 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:40.804 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:40.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:40.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:40.906 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:40.932 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:40.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:40.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:41.033 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:41.060 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:41.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:41.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:41.162 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:41.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:41.199 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:41.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:41.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:40 smithi183 ceph-mon[33277]: pgmap v306: 158 pgs: 158 active+clean; 6.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-21T12:26:41.253 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:40 smithi114 ceph-mon[30324]: pgmap v306: 158 pgs: 158 active+clean; 6.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-21T12:26:41.253 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:40 smithi114 ceph-mon[40867]: pgmap v306: 158 pgs: 158 active+clean; 6.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-21T12:26:41.301 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:41.339 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:41.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:41.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:41.441 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:41.467 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:41.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:41.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:41.569 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:41.594 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:41.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:41.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:41.696 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:41.721 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:41.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:41.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:41.822 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:41.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:41.863 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:41.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:41.964 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:41.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:41.999 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:41.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:42.100 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:42.124 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:42.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:42.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:42.168 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:42.178 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:42.226 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:42.251 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:42.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:42.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:42.352 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:42.378 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:42.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:42.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:42.479 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:42.517 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:42.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:42.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:42.587 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:26:42 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:26:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:26:42.620 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:42.647 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:42.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:42.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:42.749 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:42.788 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:42.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:42.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:42.890 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:42.918 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:42.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:42.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:42.920 INFO:teuthology.orchestra.run.smithi114.stderr:set pool 1 pg_num to 21 2024-09-21T12:26:42.920 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-21T12:26:43.020 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:43.055 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:43.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:43.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:43.074 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:26:43.075 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:26:43.118 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:26:43.118 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:26:43.118 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:42 smithi114 ceph-mon[40867]: pgmap v307: 158 pgs: 158 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-21T12:26:43.119 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:42 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2775649337' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T12:26:43.119 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:42 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T12:26:43.119 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:42 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1792665030' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:26:43.119 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:42 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:26:43.119 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:42 smithi114 ceph-mon[30324]: pgmap v307: 158 pgs: 158 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-21T12:26:43.119 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:42 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2775649337' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T12:26:43.119 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:42 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T12:26:43.119 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:42 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1792665030' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:26:43.119 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:42 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:26:43.146 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:26:43.146 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:26:43.157 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:43.183 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:26:43.183 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:26:43.197 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:43.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:43.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:43.212 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:26:43.213 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:26:43.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:42 smithi183 ceph-mon[33277]: pgmap v307: 158 pgs: 158 active+clean; 6.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-21T12:26:43.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:42 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2775649337' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T12:26:43.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:42 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T12:26:43.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:42 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1792665030' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:26:43.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:42 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:26:43.244 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:26:43.244 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:26:43.273 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:26:43.273 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:26:43.298 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:43.305 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:26:43.305 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:26:43.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:43.323 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:43.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:43.425 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:43.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:43.455 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:43.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:43.556 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:43.590 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:43.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:43.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:43.692 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:43.721 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:43.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:43.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:43.823 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:43.854 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:43.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:43.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:43.956 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:43.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:43.983 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:43.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:44.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:43 smithi114 ceph-mon[40867]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:26:44.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:43 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-21T12:26:44.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:43 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:26:44.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:43 smithi114 ceph-mon[40867]: osdmap e202: 8 total, 8 up, 7 in 2024-09-21T12:26:44.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:43 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1421933225' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:26:44.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:43 smithi114 ceph-mon[40867]: 1.0 scrub starts 2024-09-21T12:26:44.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:43 smithi114 ceph-mon[40867]: 1.0 scrub ok 2024-09-21T12:26:44.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:43 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-21T12:26:44.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:43 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-21T12:26:44.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:43 smithi114 ceph-mon[30324]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:26:44.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:43 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-21T12:26:44.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:43 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:26:44.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:43 smithi114 ceph-mon[30324]: osdmap e202: 8 total, 8 up, 7 in 2024-09-21T12:26:44.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:43 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1421933225' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:26:44.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:43 smithi114 ceph-mon[30324]: 1.0 scrub starts 2024-09-21T12:26:44.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:43 smithi114 ceph-mon[30324]: 1.0 scrub ok 2024-09-21T12:26:44.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:43 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-21T12:26:44.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:43 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-21T12:26:44.084 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:44.110 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:44.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:44.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:44.212 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:44.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:43 smithi183 ceph-mon[33277]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:26:44.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:43 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-21T12:26:44.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:43 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:26:44.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:43 smithi183 ceph-mon[33277]: osdmap e202: 8 total, 8 up, 7 in 2024-09-21T12:26:44.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:43 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1421933225' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:26:44.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:43 smithi183 ceph-mon[33277]: 1.0 scrub starts 2024-09-21T12:26:44.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:43 smithi183 ceph-mon[33277]: 1.0 scrub ok 2024-09-21T12:26:44.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:43 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-21T12:26:44.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:43 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-21T12:26:44.250 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:44.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:44.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:44.351 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:44.388 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:44.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:44.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:44.476 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:26:44.476 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:26:44.477 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:26:44.477 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:26:44.477 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:26:44.490 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:44.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:44.528 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:44.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:44.629 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:44.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:44.664 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:44.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:44.764 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:44.789 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:44.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:44.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:44.891 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:44.946 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:44.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:44.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:45.047 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:45.086 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:45.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:45.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:45.188 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:45.221 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:45.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:45.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:45.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:44 smithi183 ceph-mon[33277]: pgmap v309: 158 pgs: 158 active+clean; 6.5 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.65k op/s 2024-09-21T12:26:45.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:44 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-21T12:26:45.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:44 smithi183 ceph-mon[33277]: osdmap e203: 8 total, 8 up, 7 in 2024-09-21T12:26:45.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:44 smithi183 ceph-mon[33277]: 6.2 scrub starts 2024-09-21T12:26:45.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:44 smithi183 ceph-mon[33277]: 6.2 scrub ok 2024-09-21T12:26:45.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:44 smithi183 ceph-mon[33277]: 1.8 scrub starts 2024-09-21T12:26:45.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:44 smithi183 ceph-mon[33277]: 1.8 scrub ok 2024-09-21T12:26:45.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:44 smithi114 ceph-mon[30324]: pgmap v309: 158 pgs: 158 active+clean; 6.5 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.65k op/s 2024-09-21T12:26:45.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:44 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-21T12:26:45.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:44 smithi114 ceph-mon[30324]: osdmap e203: 8 total, 8 up, 7 in 2024-09-21T12:26:45.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:44 smithi114 ceph-mon[30324]: 6.2 scrub starts 2024-09-21T12:26:45.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:44 smithi114 ceph-mon[30324]: 6.2 scrub ok 2024-09-21T12:26:45.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:44 smithi114 ceph-mon[30324]: 1.8 scrub starts 2024-09-21T12:26:45.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:44 smithi114 ceph-mon[30324]: 1.8 scrub ok 2024-09-21T12:26:45.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:44 smithi114 ceph-mon[40867]: pgmap v309: 158 pgs: 158 active+clean; 6.5 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.65k op/s 2024-09-21T12:26:45.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:44 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-21T12:26:45.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:44 smithi114 ceph-mon[40867]: osdmap e203: 8 total, 8 up, 7 in 2024-09-21T12:26:45.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:44 smithi114 ceph-mon[40867]: 6.2 scrub starts 2024-09-21T12:26:45.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:44 smithi114 ceph-mon[40867]: 6.2 scrub ok 2024-09-21T12:26:45.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:44 smithi114 ceph-mon[40867]: 1.8 scrub starts 2024-09-21T12:26:45.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:44 smithi114 ceph-mon[40867]: 1.8 scrub ok 2024-09-21T12:26:45.323 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:45.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:45.359 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:45.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:45.460 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:45.498 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:45.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:45.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:45.600 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:45.626 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:45.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:45.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:45.728 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:45.761 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:45.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:45.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:45.863 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:45.891 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:45.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:45.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:45.993 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:46.031 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:46.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:46.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:46.134 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:46.166 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:46.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:46.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:46.206 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:45 smithi114 ceph-mon[40867]: 6.1e scrub starts 2024-09-21T12:26:46.206 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:45 smithi114 ceph-mon[40867]: 6.1e scrub ok 2024-09-21T12:26:46.206 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:45 smithi114 ceph-mon[40867]: osdmap e204: 8 total, 8 up, 7 in 2024-09-21T12:26:46.207 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:45 smithi114 ceph-mon[40867]: 1.3 scrub starts 2024-09-21T12:26:46.207 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:45 smithi114 ceph-mon[40867]: 1.3 scrub ok 2024-09-21T12:26:46.207 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:45 smithi114 ceph-mon[40867]: 1.4 scrub starts 2024-09-21T12:26:46.207 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:45 smithi114 ceph-mon[40867]: 1.4 scrub ok 2024-09-21T12:26:46.207 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:45 smithi114 ceph-mon[40867]: 1.1 scrub starts 2024-09-21T12:26:46.207 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:45 smithi114 ceph-mon[40867]: 1.1 scrub ok 2024-09-21T12:26:46.207 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:45 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T12:26:46.207 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:45 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T12:26:46.207 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:45 smithi114 ceph-mon[30324]: 6.1e scrub starts 2024-09-21T12:26:46.207 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:45 smithi114 ceph-mon[30324]: 6.1e scrub ok 2024-09-21T12:26:46.208 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:45 smithi114 ceph-mon[30324]: osdmap e204: 8 total, 8 up, 7 in 2024-09-21T12:26:46.208 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:45 smithi114 ceph-mon[30324]: 1.3 scrub starts 2024-09-21T12:26:46.208 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:45 smithi114 ceph-mon[30324]: 1.3 scrub ok 2024-09-21T12:26:46.208 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:45 smithi114 ceph-mon[30324]: 1.4 scrub starts 2024-09-21T12:26:46.208 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:45 smithi114 ceph-mon[30324]: 1.4 scrub ok 2024-09-21T12:26:46.208 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:45 smithi114 ceph-mon[30324]: 1.1 scrub starts 2024-09-21T12:26:46.208 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:45 smithi114 ceph-mon[30324]: 1.1 scrub ok 2024-09-21T12:26:46.208 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:45 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T12:26:46.208 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:45 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T12:26:46.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:45 smithi183 ceph-mon[33277]: 6.1e scrub starts 2024-09-21T12:26:46.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:45 smithi183 ceph-mon[33277]: 6.1e scrub ok 2024-09-21T12:26:46.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:45 smithi183 ceph-mon[33277]: osdmap e204: 8 total, 8 up, 7 in 2024-09-21T12:26:46.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:45 smithi183 ceph-mon[33277]: 1.3 scrub starts 2024-09-21T12:26:46.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:45 smithi183 ceph-mon[33277]: 1.3 scrub ok 2024-09-21T12:26:46.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:45 smithi183 ceph-mon[33277]: 1.4 scrub starts 2024-09-21T12:26:46.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:45 smithi183 ceph-mon[33277]: 1.4 scrub ok 2024-09-21T12:26:46.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:45 smithi183 ceph-mon[33277]: 1.1 scrub starts 2024-09-21T12:26:46.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:45 smithi183 ceph-mon[33277]: 1.1 scrub ok 2024-09-21T12:26:46.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:45 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T12:26:46.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:45 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T12:26:46.239 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:26:46.252 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T12:26:46.269 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:46.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:46.351 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:46.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:46.452 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:46.475 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:46.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:46.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:46.577 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:46.632 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:46.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:46.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:46.714 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:46.734 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:46.782 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:46.797 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:46.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:46.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:46.900 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:46.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:46.942 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:46.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:47.043 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:47.064 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:47.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:47.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:47.166 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:47.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:47.202 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:47.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:47.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:26:46 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:26:47.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:46 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:26:47.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:46 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:26:47.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:46 smithi183 ceph-mon[33277]: pgmap v312: 168 pgs: 10 unknown, 158 active+clean; 6.5 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.10k op/s 2024-09-21T12:26:47.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:46 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-21T12:26:47.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:46 smithi183 ceph-mon[33277]: osdmap e205: 8 total, 8 up, 7 in 2024-09-21T12:26:47.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:46 smithi183 ceph-mon[33277]: 1.2 scrub starts 2024-09-21T12:26:47.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:46 smithi183 ceph-mon[33277]: 1.2 scrub ok 2024-09-21T12:26:47.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:26:47.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:46 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:26:47.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:46 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:26:47.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:46 smithi114 ceph-mon[40867]: pgmap v312: 168 pgs: 10 unknown, 158 active+clean; 6.5 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.10k op/s 2024-09-21T12:26:47.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:46 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-21T12:26:47.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:46 smithi114 ceph-mon[40867]: osdmap e205: 8 total, 8 up, 7 in 2024-09-21T12:26:47.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:46 smithi114 ceph-mon[40867]: 1.2 scrub starts 2024-09-21T12:26:47.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:46 smithi114 ceph-mon[40867]: 1.2 scrub ok 2024-09-21T12:26:47.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:26:47.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:46 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:26:47.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:46 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:26:47.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:46 smithi114 ceph-mon[30324]: pgmap v312: 168 pgs: 10 unknown, 158 active+clean; 6.5 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.10k op/s 2024-09-21T12:26:47.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:46 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-21T12:26:47.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:46 smithi114 ceph-mon[30324]: osdmap e205: 8 total, 8 up, 7 in 2024-09-21T12:26:47.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:46 smithi114 ceph-mon[30324]: 1.2 scrub starts 2024-09-21T12:26:47.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:46 smithi114 ceph-mon[30324]: 1.2 scrub ok 2024-09-21T12:26:47.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:26:47.303 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:47.342 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:47.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:47.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:47.445 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:47.474 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:47.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:47.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:47.532 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T12:26:47.576 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:47.601 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:47.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:47.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:47.703 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:47.729 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:47.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:47.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:47.831 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:47.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:47.875 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:47.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:47.976 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:47.996 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:47 smithi114 ceph-mon[30324]: 5.1e scrub starts 2024-09-21T12:26:47.996 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:47 smithi114 ceph-mon[30324]: 5.1e scrub ok 2024-09-21T12:26:47.996 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:47 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:26:47.996 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:47 smithi114 ceph-mon[30324]: osdmap e206: 8 total, 8 up, 7 in 2024-09-21T12:26:47.996 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:47 smithi114 ceph-mon[30324]: 1.12 scrub starts 2024-09-21T12:26:47.996 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:47 smithi114 ceph-mon[30324]: 1.12 scrub ok 2024-09-21T12:26:47.997 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:47 smithi114 ceph-mon[40867]: 5.1e scrub starts 2024-09-21T12:26:47.997 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:47 smithi114 ceph-mon[40867]: 5.1e scrub ok 2024-09-21T12:26:47.997 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:47 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:26:47.997 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:47 smithi114 ceph-mon[40867]: osdmap e206: 8 total, 8 up, 7 in 2024-09-21T12:26:47.997 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:47 smithi114 ceph-mon[40867]: 1.12 scrub starts 2024-09-21T12:26:47.997 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:47 smithi114 ceph-mon[40867]: 1.12 scrub ok 2024-09-21T12:26:48.005 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:48.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:48.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:48.106 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:48.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:48.135 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:48.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:48.236 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:48.263 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:48.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:48.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:48.356 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:26:48.356 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:26:48.357 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:26:48.357 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:26:48.357 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:26:48.357 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:26:48.357 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:26:48.357 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:26:48.357 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:26:48.357 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:26:48.357 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:26:48.358 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:26:48.364 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:48.389 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:48.389 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:26:48.389 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:26:48.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:48.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:48.423 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:26:48.423 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:26:48.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:47 smithi183 ceph-mon[33277]: 5.1e scrub starts 2024-09-21T12:26:48.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:47 smithi183 ceph-mon[33277]: 5.1e scrub ok 2024-09-21T12:26:48.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:47 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:26:48.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:47 smithi183 ceph-mon[33277]: osdmap e206: 8 total, 8 up, 7 in 2024-09-21T12:26:48.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:47 smithi183 ceph-mon[33277]: 1.12 scrub starts 2024-09-21T12:26:48.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:47 smithi183 ceph-mon[33277]: 1.12 scrub ok 2024-09-21T12:26:48.491 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:48.520 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:48.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:48.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:48.622 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:48.665 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:48.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:48.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:48.691 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:26:48.691 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:26:48.767 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:48.800 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:48.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:48.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:48.902 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:48.931 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:48.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:48.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:49.032 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:49.059 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:49.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:49.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:49.160 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:49.187 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:49.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:49.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:49.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:48 smithi114 ceph-mon[30324]: 1.5 scrub starts 2024-09-21T12:26:49.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:48 smithi114 ceph-mon[30324]: 1.5 scrub ok 2024-09-21T12:26:49.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:48 smithi114 ceph-mon[30324]: pgmap v315: 168 pgs: 8 peering, 160 active+clean; 6.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 11 KiB/s rd, 74 KiB/s wr, 1.68k op/s; 106 KiB/s, 0 objects/s recovering 2024-09-21T12:26:49.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:48 smithi114 ceph-mon[30324]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-09-21T12:26:49.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:48 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4186100923' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:26:49.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:48 smithi114 ceph-mon[30324]: 1.b scrub starts 2024-09-21T12:26:49.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:48 smithi114 ceph-mon[30324]: 1.b scrub ok 2024-09-21T12:26:49.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:48 smithi114 ceph-mon[40867]: 1.5 scrub starts 2024-09-21T12:26:49.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:48 smithi114 ceph-mon[40867]: 1.5 scrub ok 2024-09-21T12:26:49.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:48 smithi114 ceph-mon[40867]: pgmap v315: 168 pgs: 8 peering, 160 active+clean; 6.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 11 KiB/s rd, 74 KiB/s wr, 1.68k op/s; 106 KiB/s, 0 objects/s recovering 2024-09-21T12:26:49.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:48 smithi114 ceph-mon[40867]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-09-21T12:26:49.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:48 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/4186100923' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:26:49.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:48 smithi114 ceph-mon[40867]: 1.b scrub starts 2024-09-21T12:26:49.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:48 smithi114 ceph-mon[40867]: 1.b scrub ok 2024-09-21T12:26:49.288 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:49.324 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:49.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:49.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:49.426 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:49.470 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:49.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:49.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:49.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:48 smithi183 ceph-mon[33277]: 1.5 scrub starts 2024-09-21T12:26:49.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:48 smithi183 ceph-mon[33277]: 1.5 scrub ok 2024-09-21T12:26:49.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:48 smithi183 ceph-mon[33277]: pgmap v315: 168 pgs: 8 peering, 160 active+clean; 6.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 11 KiB/s rd, 74 KiB/s wr, 1.68k op/s; 106 KiB/s, 0 objects/s recovering 2024-09-21T12:26:49.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:48 smithi183 ceph-mon[33277]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-09-21T12:26:49.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:48 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/4186100923' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:26:49.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:48 smithi183 ceph-mon[33277]: 1.b scrub starts 2024-09-21T12:26:49.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:48 smithi183 ceph-mon[33277]: 1.b scrub ok 2024-09-21T12:26:49.572 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:49.603 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:49.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:49.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:49.705 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:49.739 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:49.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:49.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:49.841 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:49.870 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:49.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:49.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:49.935 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T12:26:49.935 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:26:49.935 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T12:26:49.935 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.476907 2024-09-21T12:26:49.935 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 5 0.47690672218227226 2024-09-21T12:26:49.972 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:49.999 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:50.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:50.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:50.101 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:50.112 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T12:26:50.124 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:50.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:50.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:50.226 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:50.245 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:50.249 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:50.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:50.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:50.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:50 smithi114 ceph-mon[40867]: 1.11 scrub starts 2024-09-21T12:26:50.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:50 smithi114 ceph-mon[40867]: 1.11 scrub ok 2024-09-21T12:26:50.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:50 smithi114 ceph-mon[40867]: 1.0 scrub starts 2024-09-21T12:26:50.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:50 smithi114 ceph-mon[40867]: 1.0 scrub ok 2024-09-21T12:26:50.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:50 smithi114 ceph-mon[40867]: 1.c scrub starts 2024-09-21T12:26:50.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:50 smithi114 ceph-mon[40867]: 1.c scrub ok 2024-09-21T12:26:50.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:50 smithi114 ceph-mon[30324]: 1.11 scrub starts 2024-09-21T12:26:50.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:50 smithi114 ceph-mon[30324]: 1.11 scrub ok 2024-09-21T12:26:50.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:50 smithi114 ceph-mon[30324]: 1.0 scrub starts 2024-09-21T12:26:50.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:50 smithi114 ceph-mon[30324]: 1.0 scrub ok 2024-09-21T12:26:50.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:50 smithi114 ceph-mon[30324]: 1.c scrub starts 2024-09-21T12:26:50.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:50 smithi114 ceph-mon[30324]: 1.c scrub ok 2024-09-21T12:26:50.350 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:50.375 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:50.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:50.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:50.477 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:50.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:50 smithi183 ceph-mon[33277]: 1.11 scrub starts 2024-09-21T12:26:50.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:50 smithi183 ceph-mon[33277]: 1.11 scrub ok 2024-09-21T12:26:50.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:50 smithi183 ceph-mon[33277]: 1.0 scrub starts 2024-09-21T12:26:50.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:50 smithi183 ceph-mon[33277]: 1.0 scrub ok 2024-09-21T12:26:50.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:50 smithi183 ceph-mon[33277]: 1.c scrub starts 2024-09-21T12:26:50.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:50 smithi183 ceph-mon[33277]: 1.c scrub ok 2024-09-21T12:26:50.502 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:50.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:50.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:50.603 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:50.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:50.654 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:50.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:50.755 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:50.792 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:50.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:50.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:50.894 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:50.900 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:50.919 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:50.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:50.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:50.963 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:51.021 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:51.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:51 smithi114 ceph-mon[30324]: 1.14 scrub starts 2024-09-21T12:26:51.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:51 smithi114 ceph-mon[30324]: 1.14 scrub ok 2024-09-21T12:26:51.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:51 smithi114 ceph-mon[30324]: pgmap v316: 168 pgs: 8 peering, 160 active+clean; 6.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 7.8 KiB/s rd, 51 KiB/s wr, 1.17k op/s; 74 KiB/s, 0 objects/s recovering 2024-09-21T12:26:51.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:51 smithi114 ceph-mon[30324]: 1.10 scrub starts 2024-09-21T12:26:51.027 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:51 smithi114 ceph-mon[30324]: 1.10 scrub ok 2024-09-21T12:26:51.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:51 smithi114 ceph-mon[40867]: 1.14 scrub starts 2024-09-21T12:26:51.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:51 smithi114 ceph-mon[40867]: 1.14 scrub ok 2024-09-21T12:26:51.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:51 smithi114 ceph-mon[40867]: pgmap v316: 168 pgs: 8 peering, 160 active+clean; 6.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 7.8 KiB/s rd, 51 KiB/s wr, 1.17k op/s; 74 KiB/s, 0 objects/s recovering 2024-09-21T12:26:51.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:51 smithi114 ceph-mon[40867]: 1.10 scrub starts 2024-09-21T12:26:51.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:51 smithi114 ceph-mon[40867]: 1.10 scrub ok 2024-09-21T12:26:51.046 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:51.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:51.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:51.147 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:51.174 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:51.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:51.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:51.276 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:51.333 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:51.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:51.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:51.435 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:51.462 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:26:51.469 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:51.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:51.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:51.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:51 smithi183 ceph-mon[33277]: 1.14 scrub starts 2024-09-21T12:26:51.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:51 smithi183 ceph-mon[33277]: 1.14 scrub ok 2024-09-21T12:26:51.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:51 smithi183 ceph-mon[33277]: pgmap v316: 168 pgs: 8 peering, 160 active+clean; 6.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 7.8 KiB/s rd, 51 KiB/s wr, 1.17k op/s; 74 KiB/s, 0 objects/s recovering 2024-09-21T12:26:51.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:51 smithi183 ceph-mon[33277]: 1.10 scrub starts 2024-09-21T12:26:51.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:51 smithi183 ceph-mon[33277]: 1.10 scrub ok 2024-09-21T12:26:51.573 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:51.597 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:51.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:51.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:51.700 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:51.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:51.751 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:51.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:51.852 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:51.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:51.900 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:51.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:51.994 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:26:51.994 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:26:52.000 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:52.002 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:52.027 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-21T12:26:52.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:52.061 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:52.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:52.162 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:52.196 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:52.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:52.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:52.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:52 smithi114 ceph-mon[40867]: 5.b scrub starts 2024-09-21T12:26:52.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:52 smithi114 ceph-mon[40867]: 5.b scrub ok 2024-09-21T12:26:52.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:52 smithi114 ceph-mon[40867]: 3.3 scrub starts 2024-09-21T12:26:52.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:52 smithi114 ceph-mon[40867]: 3.3 scrub ok 2024-09-21T12:26:52.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:52 smithi114 ceph-mon[40867]: 1.7 scrub starts 2024-09-21T12:26:52.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:52 smithi114 ceph-mon[40867]: 1.7 scrub ok 2024-09-21T12:26:52.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:52 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2567903717' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:26:52.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:52 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:26:52.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:52 smithi114 ceph-mon[40867]: 4.1 scrub starts 2024-09-21T12:26:52.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:52 smithi114 ceph-mon[40867]: 4.1 scrub ok 2024-09-21T12:26:52.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:52 smithi114 ceph-mon[30324]: 5.b scrub starts 2024-09-21T12:26:52.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:52 smithi114 ceph-mon[30324]: 5.b scrub ok 2024-09-21T12:26:52.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:52 smithi114 ceph-mon[30324]: 3.3 scrub starts 2024-09-21T12:26:52.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:52 smithi114 ceph-mon[30324]: 3.3 scrub ok 2024-09-21T12:26:52.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:52 smithi114 ceph-mon[30324]: 1.7 scrub starts 2024-09-21T12:26:52.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:52 smithi114 ceph-mon[30324]: 1.7 scrub ok 2024-09-21T12:26:52.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:52 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2567903717' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:26:52.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:52 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:26:52.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:52 smithi114 ceph-mon[30324]: 4.1 scrub starts 2024-09-21T12:26:52.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:52 smithi114 ceph-mon[30324]: 4.1 scrub ok 2024-09-21T12:26:52.298 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:52.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:52.336 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:52.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:52.437 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:52.465 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:52.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:52.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:52.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:52 smithi183 ceph-mon[33277]: 5.b scrub starts 2024-09-21T12:26:52.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:52 smithi183 ceph-mon[33277]: 5.b scrub ok 2024-09-21T12:26:52.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:52 smithi183 ceph-mon[33277]: 3.3 scrub starts 2024-09-21T12:26:52.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:52 smithi183 ceph-mon[33277]: 3.3 scrub ok 2024-09-21T12:26:52.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:52 smithi183 ceph-mon[33277]: 1.7 scrub starts 2024-09-21T12:26:52.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:52 smithi183 ceph-mon[33277]: 1.7 scrub ok 2024-09-21T12:26:52.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:52 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2567903717' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:26:52.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:52 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:26:52.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:52 smithi183 ceph-mon[33277]: 4.1 scrub starts 2024-09-21T12:26:52.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:52 smithi183 ceph-mon[33277]: 4.1 scrub ok 2024-09-21T12:26:52.566 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:52.587 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:52.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:52.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:52.689 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:52.722 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:52.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:52.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:52.824 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:52.860 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:52.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:52.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:52.962 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:52.993 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:52.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:52.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:53.024 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:26:52 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:26:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:26:53.025 INFO:teuthology.orchestra.run.smithi114.stderr:set osd.5 primary-affinity to 0.476907 (8312542) 2024-09-21T12:26:53.095 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:53.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:53.117 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:53.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:53.218 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:53.239 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:53.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:53.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:53.341 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:53.364 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:53.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:53.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:53.466 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:53.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:53 smithi183 ceph-mon[33277]: pgmap v317: 168 pgs: 3 peering, 165 active+clean; 6.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 4.4 KiB/s rd, 22 KiB/s wr, 1.30k op/s; 55 KiB/s, 0 objects/s recovering 2024-09-21T12:26:53.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:53 smithi183 ceph-mon[33277]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-09-21T12:26:53.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:53 smithi183 ceph-mon[33277]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:26:53.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:53 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:26:53.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:53 smithi183 ceph-mon[33277]: osdmap e207: 8 total, 8 up, 7 in 2024-09-21T12:26:53.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:53 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/635323541' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.47690672218227226} : dispatch 2024-09-21T12:26:53.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:53 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.47690672218227226} : dispatch 2024-09-21T12:26:53.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:53 smithi183 ceph-mon[33277]: 5.1c scrub starts 2024-09-21T12:26:53.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:53 smithi183 ceph-mon[33277]: 5.1c scrub ok 2024-09-21T12:26:53.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:53.490 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:53.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:53.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:53 smithi114 ceph-mon[30324]: pgmap v317: 168 pgs: 3 peering, 165 active+clean; 6.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 4.4 KiB/s rd, 22 KiB/s wr, 1.30k op/s; 55 KiB/s, 0 objects/s recovering 2024-09-21T12:26:53.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:53 smithi114 ceph-mon[30324]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-09-21T12:26:53.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:53 smithi114 ceph-mon[30324]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:26:53.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:53 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:26:53.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:53 smithi114 ceph-mon[30324]: osdmap e207: 8 total, 8 up, 7 in 2024-09-21T12:26:53.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:53 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/635323541' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.47690672218227226} : dispatch 2024-09-21T12:26:53.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:53 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.47690672218227226} : dispatch 2024-09-21T12:26:53.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:53 smithi114 ceph-mon[30324]: 5.1c scrub starts 2024-09-21T12:26:53.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:53 smithi114 ceph-mon[30324]: 5.1c scrub ok 2024-09-21T12:26:53.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:53 smithi114 ceph-mon[40867]: pgmap v317: 168 pgs: 3 peering, 165 active+clean; 6.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 4.4 KiB/s rd, 22 KiB/s wr, 1.30k op/s; 55 KiB/s, 0 objects/s recovering 2024-09-21T12:26:53.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:53 smithi114 ceph-mon[40867]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-09-21T12:26:53.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:53 smithi114 ceph-mon[40867]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:26:53.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:53 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:26:53.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:53 smithi114 ceph-mon[40867]: osdmap e207: 8 total, 8 up, 7 in 2024-09-21T12:26:53.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:53 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/635323541' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.47690672218227226} : dispatch 2024-09-21T12:26:53.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:53 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.47690672218227226} : dispatch 2024-09-21T12:26:53.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:53 smithi114 ceph-mon[40867]: 5.1c scrub starts 2024-09-21T12:26:53.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:53 smithi114 ceph-mon[40867]: 5.1c scrub ok 2024-09-21T12:26:53.591 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:53.613 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:53.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:53.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:53.715 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:53.736 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:53.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:53.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:53.838 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:53.864 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:53.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:53.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:53.966 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:53.987 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:53.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:53.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:54.088 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:54.113 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:54.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:54.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:54.215 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:54.243 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:54.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:54.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:54.345 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:54.366 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:54.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:54.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:54.468 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:54.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:54 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:26:54.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:54 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:26:54.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:54 smithi183 ceph-mon[33277]: 1.6 scrub starts 2024-09-21T12:26:54.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:54 smithi183 ceph-mon[33277]: 1.6 scrub ok 2024-09-21T12:26:54.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:54 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.47690672218227226}]': finished 2024-09-21T12:26:54.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:54 smithi183 ceph-mon[33277]: osdmap e208: 8 total, 8 up, 7 in 2024-09-21T12:26:54.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:54 smithi183 ceph-mon[33277]: 1.e scrub starts 2024-09-21T12:26:54.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:54 smithi183 ceph-mon[33277]: 1.e scrub ok 2024-09-21T12:26:54.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:54.497 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:54.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:54.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:54 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:26:54.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:54 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:26:54.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:54 smithi114 ceph-mon[30324]: 1.6 scrub starts 2024-09-21T12:26:54.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:54 smithi114 ceph-mon[30324]: 1.6 scrub ok 2024-09-21T12:26:54.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:54 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.47690672218227226}]': finished 2024-09-21T12:26:54.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:54 smithi114 ceph-mon[30324]: osdmap e208: 8 total, 8 up, 7 in 2024-09-21T12:26:54.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:54 smithi114 ceph-mon[30324]: 1.e scrub starts 2024-09-21T12:26:54.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:54 smithi114 ceph-mon[30324]: 1.e scrub ok 2024-09-21T12:26:54.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:54 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:26:54.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:54 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:26:54.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:54 smithi114 ceph-mon[40867]: 1.6 scrub starts 2024-09-21T12:26:54.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:54 smithi114 ceph-mon[40867]: 1.6 scrub ok 2024-09-21T12:26:54.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:54 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.47690672218227226}]': finished 2024-09-21T12:26:54.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:54 smithi114 ceph-mon[40867]: osdmap e208: 8 total, 8 up, 7 in 2024-09-21T12:26:54.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:54 smithi114 ceph-mon[40867]: 1.e scrub starts 2024-09-21T12:26:54.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:54 smithi114 ceph-mon[40867]: 1.e scrub ok 2024-09-21T12:26:54.598 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:54.629 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:54.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:54.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:54.731 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:54.757 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:54.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:54.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:54.859 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:54.885 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:54.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:54.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:54.986 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:55.011 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:55.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:55.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:55.112 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:55.141 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:55.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:55.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:55.243 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:55.275 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:55.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:55.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:55.376 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:55.405 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:55.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:55.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:55.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:55 smithi183 ceph-mon[33277]: pgmap v320: 168 pgs: 168 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 1.44k op/s 2024-09-21T12:26:55.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:55 smithi183 ceph-mon[33277]: osdmap e209: 8 total, 8 up, 7 in 2024-09-21T12:26:55.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:55 smithi183 ceph-mon[33277]: 5.1b scrub starts 2024-09-21T12:26:55.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:55 smithi183 ceph-mon[33277]: 5.1b scrub ok 2024-09-21T12:26:55.507 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:55.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:55 smithi114 ceph-mon[30324]: pgmap v320: 168 pgs: 168 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 1.44k op/s 2024-09-21T12:26:55.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:55 smithi114 ceph-mon[30324]: osdmap e209: 8 total, 8 up, 7 in 2024-09-21T12:26:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:55 smithi114 ceph-mon[30324]: 5.1b scrub starts 2024-09-21T12:26:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:55 smithi114 ceph-mon[30324]: 5.1b scrub ok 2024-09-21T12:26:55.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:55 smithi114 ceph-mon[40867]: pgmap v320: 168 pgs: 168 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 1.44k op/s 2024-09-21T12:26:55.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:55 smithi114 ceph-mon[40867]: osdmap e209: 8 total, 8 up, 7 in 2024-09-21T12:26:55.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:55 smithi114 ceph-mon[40867]: 5.1b scrub starts 2024-09-21T12:26:55.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:55 smithi114 ceph-mon[40867]: 5.1b scrub ok 2024-09-21T12:26:55.537 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:55.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:55.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:55.638 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:55.663 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:55.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:55.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:55.765 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:55.801 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:55.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:55.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:55.903 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:55.938 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:55.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:55.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:56.039 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:56.080 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:56.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:56.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:56.182 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:56.210 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:56.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:56.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:56.311 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:56.340 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:56.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:56.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:56.442 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:56.463 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:56.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:56.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:56.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:56 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:26:56.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:56 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:26:56.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:56 smithi183 ceph-mon[33277]: osdmap e210: 8 total, 8 up, 7 in 2024-09-21T12:26:56.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:56 smithi183 ceph-mon[33277]: 1.13 scrub starts 2024-09-21T12:26:56.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:56 smithi183 ceph-mon[33277]: 1.13 scrub ok 2024-09-21T12:26:56.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:56 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:26:56.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:56 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:26:56.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:56 smithi114 ceph-mon[40867]: osdmap e210: 8 total, 8 up, 7 in 2024-09-21T12:26:56.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:56 smithi114 ceph-mon[40867]: 1.13 scrub starts 2024-09-21T12:26:56.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:56 smithi114 ceph-mon[40867]: 1.13 scrub ok 2024-09-21T12:26:56.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:56 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:26:56.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:56 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:26:56.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:56 smithi114 ceph-mon[30324]: osdmap e210: 8 total, 8 up, 7 in 2024-09-21T12:26:56.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:56 smithi114 ceph-mon[30324]: 1.13 scrub starts 2024-09-21T12:26:56.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:56 smithi114 ceph-mon[30324]: 1.13 scrub ok 2024-09-21T12:26:56.565 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:56.600 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:56.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:56.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:56.701 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:56.737 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:56.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:56.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:56.839 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:56.865 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:56.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:56.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:56.967 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:56.994 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:56.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:56.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:57.096 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:57.125 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:57.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:57.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:57.227 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:57.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:26:56 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:26:57.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:57 smithi183 ceph-mon[33277]: 1.a scrub starts 2024-09-21T12:26:57.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:57 smithi183 ceph-mon[33277]: 1.a scrub ok 2024-09-21T12:26:57.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:57 smithi183 ceph-mon[33277]: pgmap v323: 168 pgs: 168 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 1.17k op/s 2024-09-21T12:26:57.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:57 smithi183 ceph-mon[33277]: 1.f scrub starts 2024-09-21T12:26:57.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:57 smithi183 ceph-mon[33277]: 1.f scrub ok 2024-09-21T12:26:57.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:57 smithi183 ceph-mon[33277]: 1.d scrub starts 2024-09-21T12:26:57.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:57 smithi183 ceph-mon[33277]: 1.d scrub ok 2024-09-21T12:26:57.254 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:57.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:57.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:57.356 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:26:57.369 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:57 smithi114 ceph-mon[40867]: 1.a scrub starts 2024-09-21T12:26:57.369 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:57 smithi114 ceph-mon[40867]: 1.a scrub ok 2024-09-21T12:26:57.369 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:57 smithi114 ceph-mon[40867]: pgmap v323: 168 pgs: 168 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 1.17k op/s 2024-09-21T12:26:57.369 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:57 smithi114 ceph-mon[40867]: 1.f scrub starts 2024-09-21T12:26:57.369 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:57 smithi114 ceph-mon[40867]: 1.f scrub ok 2024-09-21T12:26:57.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:57 smithi114 ceph-mon[40867]: 1.d scrub starts 2024-09-21T12:26:57.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:57 smithi114 ceph-mon[40867]: 1.d scrub ok 2024-09-21T12:26:57.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:57 smithi114 ceph-mon[30324]: 1.a scrub starts 2024-09-21T12:26:57.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:57 smithi114 ceph-mon[30324]: 1.a scrub ok 2024-09-21T12:26:57.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:57 smithi114 ceph-mon[30324]: pgmap v323: 168 pgs: 168 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 1.17k op/s 2024-09-21T12:26:57.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:57 smithi114 ceph-mon[30324]: 1.f scrub starts 2024-09-21T12:26:57.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:57 smithi114 ceph-mon[30324]: 1.f scrub ok 2024-09-21T12:26:57.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:57 smithi114 ceph-mon[30324]: 1.d scrub starts 2024-09-21T12:26:57.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:57 smithi114 ceph-mon[30324]: 1.d scrub ok 2024-09-21T12:26:57.406 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:57.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:57.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:26:57.508 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:57.532 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:57.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:57.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:57.634 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:57.658 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:57.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:57.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:57.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:57.797 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:57.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:57.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:57.899 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:57.932 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:57.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:57.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:58.033 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:58.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:58.068 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:58.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:58.169 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:58.174 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T12:26:58.196 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:58.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:58.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:58.298 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:58.323 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:58.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:58.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:58.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:58 smithi114 ceph-mon[40867]: 6.a scrub starts 2024-09-21T12:26:58.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:58 smithi114 ceph-mon[40867]: 6.a scrub ok 2024-09-21T12:26:58.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:58 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:26:58.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:58 smithi114 ceph-mon[40867]: 2.2 scrub starts 2024-09-21T12:26:58.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:58 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:26:58.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:58 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:26:58.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:58 smithi114 ceph-mon[40867]: 2.2 scrub ok 2024-09-21T12:26:58.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:58 smithi114 ceph-mon[40867]: 4.17 scrub starts 2024-09-21T12:26:58.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:58 smithi114 ceph-mon[40867]: 4.17 scrub ok 2024-09-21T12:26:58.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:58 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:26:58.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:58 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:26:58.380 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:58 smithi114 ceph-mon[30324]: 6.a scrub starts 2024-09-21T12:26:58.380 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:58 smithi114 ceph-mon[30324]: 6.a scrub ok 2024-09-21T12:26:58.380 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:58 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:26:58.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:58 smithi114 ceph-mon[30324]: 2.2 scrub starts 2024-09-21T12:26:58.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:58 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:26:58.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:58 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:26:58.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:58 smithi114 ceph-mon[30324]: 2.2 scrub ok 2024-09-21T12:26:58.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:58 smithi114 ceph-mon[30324]: 4.17 scrub starts 2024-09-21T12:26:58.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:58 smithi114 ceph-mon[30324]: 4.17 scrub ok 2024-09-21T12:26:58.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:58 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:26:58.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:58 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:26:58.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T12:26:58.425 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:26:58.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:58 smithi183 ceph-mon[33277]: 6.a scrub starts 2024-09-21T12:26:58.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:58 smithi183 ceph-mon[33277]: 6.a scrub ok 2024-09-21T12:26:58.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:58 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:26:58.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:58 smithi183 ceph-mon[33277]: 2.2 scrub starts 2024-09-21T12:26:58.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:58 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:26:58.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:58 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:26:58.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:58 smithi183 ceph-mon[33277]: 2.2 scrub ok 2024-09-21T12:26:58.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:58 smithi183 ceph-mon[33277]: 4.17 scrub starts 2024-09-21T12:26:58.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:58 smithi183 ceph-mon[33277]: 4.17 scrub ok 2024-09-21T12:26:58.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:58 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:26:58.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:58 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:26:58.552 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:58.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:58.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:26:58.653 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:58.699 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:58.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:58.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:58.742 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:26:58.743 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:26:58.782 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:26:58.783 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:26:58.791 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:58.801 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:26:58.823 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:58.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:58.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:26:58.825 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:26:58.825 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:26:58.859 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:26:58.859 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:26:58.897 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:26:58.897 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:26:58.925 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:58.928 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:26:58.928 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:26:58.951 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:58.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:58.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:58.960 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:26:58.960 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:26:58.990 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:26:58.990 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:26:59.053 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:59.079 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:59.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:59.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:59.181 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:26:59.223 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:59.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:59.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:26:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:59 smithi114 ceph-mon[40867]: 1.8 scrub starts 2024-09-21T12:26:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:59 smithi114 ceph-mon[40867]: 1.8 scrub ok 2024-09-21T12:26:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:59 smithi114 ceph-mon[40867]: pgmap v324: 168 pgs: 168 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 2.5 KiB/s rd, 13 KiB/s wr, 1.49k op/s 2024-09-21T12:26:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:59 smithi114 ceph-mon[40867]: 1.1 scrub starts 2024-09-21T12:26:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:59 smithi114 ceph-mon[40867]: 1.1 scrub ok 2024-09-21T12:26:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:59 smithi114 ceph-mon[40867]: osdmap e211: 8 total, 8 up, 7 in 2024-09-21T12:26:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:59 smithi114 ceph-mon[40867]: 6.d scrub starts 2024-09-21T12:26:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:59 smithi114 ceph-mon[40867]: 6.d scrub ok 2024-09-21T12:26:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:59 smithi114 ceph-mon[40867]: 6.b scrub starts 2024-09-21T12:26:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:59 smithi114 ceph-mon[40867]: 6.b scrub ok 2024-09-21T12:26:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:26:59 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1200294735' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:26:59.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:59 smithi114 ceph-mon[30324]: 1.8 scrub starts 2024-09-21T12:26:59.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:59 smithi114 ceph-mon[30324]: 1.8 scrub ok 2024-09-21T12:26:59.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:59 smithi114 ceph-mon[30324]: pgmap v324: 168 pgs: 168 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 2.5 KiB/s rd, 13 KiB/s wr, 1.49k op/s 2024-09-21T12:26:59.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:59 smithi114 ceph-mon[30324]: 1.1 scrub starts 2024-09-21T12:26:59.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:59 smithi114 ceph-mon[30324]: 1.1 scrub ok 2024-09-21T12:26:59.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:59 smithi114 ceph-mon[30324]: osdmap e211: 8 total, 8 up, 7 in 2024-09-21T12:26:59.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:59 smithi114 ceph-mon[30324]: 6.d scrub starts 2024-09-21T12:26:59.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:59 smithi114 ceph-mon[30324]: 6.d scrub ok 2024-09-21T12:26:59.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:59 smithi114 ceph-mon[30324]: 6.b scrub starts 2024-09-21T12:26:59.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:59 smithi114 ceph-mon[30324]: 6.b scrub ok 2024-09-21T12:26:59.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:26:59 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1200294735' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:26:59.325 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:59.359 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:59.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:59.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:59.461 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:26:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:59 smithi183 ceph-mon[33277]: 1.8 scrub starts 2024-09-21T12:26:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:59 smithi183 ceph-mon[33277]: 1.8 scrub ok 2024-09-21T12:26:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:59 smithi183 ceph-mon[33277]: pgmap v324: 168 pgs: 168 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 2.5 KiB/s rd, 13 KiB/s wr, 1.49k op/s 2024-09-21T12:26:59.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:59 smithi183 ceph-mon[33277]: 1.1 scrub starts 2024-09-21T12:26:59.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:59 smithi183 ceph-mon[33277]: 1.1 scrub ok 2024-09-21T12:26:59.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:59 smithi183 ceph-mon[33277]: osdmap e211: 8 total, 8 up, 7 in 2024-09-21T12:26:59.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:59 smithi183 ceph-mon[33277]: 6.d scrub starts 2024-09-21T12:26:59.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:59 smithi183 ceph-mon[33277]: 6.d scrub ok 2024-09-21T12:26:59.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:59 smithi183 ceph-mon[33277]: 6.b scrub starts 2024-09-21T12:26:59.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:59 smithi183 ceph-mon[33277]: 6.b scrub ok 2024-09-21T12:26:59.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:26:59 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1200294735' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:26:59.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:59.490 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:59.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:26:59.591 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:26:59.624 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:59.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:59.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:26:59.726 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:26:59.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:59.766 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:26:59.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:26:59.867 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:26:59.891 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:26:59.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:26:59.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:26:59.992 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:00.020 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:00.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:00.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:00.099 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-21T12:27:00.122 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:00.161 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:00.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:00.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:00.263 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:00.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:00 smithi114 ceph-mon[40867]: 3.1c scrub starts 2024-09-21T12:27:00.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:00 smithi114 ceph-mon[40867]: 3.1c scrub ok 2024-09-21T12:27:00.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:00 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1969301282' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:27:00.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:00 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:27:00.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:00 smithi114 ceph-mon[30324]: 3.1c scrub starts 2024-09-21T12:27:00.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:00 smithi114 ceph-mon[30324]: 3.1c scrub ok 2024-09-21T12:27:00.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:00 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1969301282' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:27:00.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:00 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:27:00.300 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:00.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:00.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:00.402 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:00.426 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:00.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:00.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:00.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:00 smithi183 ceph-mon[33277]: 3.1c scrub starts 2024-09-21T12:27:00.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:00 smithi183 ceph-mon[33277]: 3.1c scrub ok 2024-09-21T12:27:00.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:00 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1969301282' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:27:00.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:00 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:27:00.528 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:00.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:00.571 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:00.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:00.671 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:00.699 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:00.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:00.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:00.795 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:27:00.801 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:00.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:00.867 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:00.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:00.968 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:01.004 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:01.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:01.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:01.105 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:01.131 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:01.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:01.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:01.233 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:01.258 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:01.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:01.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:01.360 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:01.396 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:01.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:01.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:01.445 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:01 smithi114 ceph-mon[40867]: pgmap v326: 168 pgs: 168 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 4.0 KiB/s rd, 25 KiB/s wr, 918 op/s; 8.7 KiB/s, 0 objects/s recovering 2024-09-21T12:27:01.445 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:01 smithi114 ceph-mon[40867]: 3.6 scrub starts 2024-09-21T12:27:01.445 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:01 smithi114 ceph-mon[40867]: 3.6 scrub ok 2024-09-21T12:27:01.445 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:01 smithi114 ceph-mon[40867]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:27:01.445 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:01 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:27:01.445 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:01 smithi114 ceph-mon[40867]: osdmap e212: 8 total, 8 up, 7 in 2024-09-21T12:27:01.445 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:01 smithi114 ceph-mon[30324]: pgmap v326: 168 pgs: 168 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 4.0 KiB/s rd, 25 KiB/s wr, 918 op/s; 8.7 KiB/s, 0 objects/s recovering 2024-09-21T12:27:01.445 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:01 smithi114 ceph-mon[30324]: 3.6 scrub starts 2024-09-21T12:27:01.446 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:01 smithi114 ceph-mon[30324]: 3.6 scrub ok 2024-09-21T12:27:01.446 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:01 smithi114 ceph-mon[30324]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:27:01.446 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:01 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:27:01.446 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:01 smithi114 ceph-mon[30324]: osdmap e212: 8 total, 8 up, 7 in 2024-09-21T12:27:01.461 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:27:01.462 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:27:01.462 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:27:01.462 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:27:01.462 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:27:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:01 smithi183 ceph-mon[33277]: pgmap v326: 168 pgs: 168 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 4.0 KiB/s rd, 25 KiB/s wr, 918 op/s; 8.7 KiB/s, 0 objects/s recovering 2024-09-21T12:27:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:01 smithi183 ceph-mon[33277]: 3.6 scrub starts 2024-09-21T12:27:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:01 smithi183 ceph-mon[33277]: 3.6 scrub ok 2024-09-21T12:27:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:01 smithi183 ceph-mon[33277]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:27:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:01 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:27:01.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:01 smithi183 ceph-mon[33277]: osdmap e212: 8 total, 8 up, 7 in 2024-09-21T12:27:01.498 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:01.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:01.536 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:01.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:01.636 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:01.672 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:01.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:01.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:01.774 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:01.801 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:01.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:01.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:01.903 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:01.930 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:01.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:01.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:02.032 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:02.058 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:02.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:02.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:02.063 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:27:02.160 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:02.185 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:02.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:02.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:02.287 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:02.311 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:02.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:02.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:02.413 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:02.421 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T12:27:02.421 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:27:02.421 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-21T12:27:02.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:27:02.455 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:02.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:02.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:02.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:02 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:27:02.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:02 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:27:02.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:02 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:27:02.558 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:02.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:02.602 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:02.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:02.702 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:02.728 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:02.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:02.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:02.775 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:27:02 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:27:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:27:02.807 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T12:27:02.829 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:02.894 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:02.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:02.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:02.996 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:03.042 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:03.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:03.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:03.108 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:03 smithi114 ceph-mon[30324]: pgmap v328: 168 pgs: 168 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 4.4 KiB/s rd, 22 KiB/s wr, 1.03k op/s; 7.9 KiB/s, 0 objects/s recovering 2024-09-21T12:27:03.143 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:03.193 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:03.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:03.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:03.295 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:03.322 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:03.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:03.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:03.424 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:03.445 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:03.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:03.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:03.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:03 smithi183 ceph-mon[33277]: pgmap v328: 168 pgs: 168 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 4.4 KiB/s rd, 22 KiB/s wr, 1.03k op/s; 7.9 KiB/s, 0 objects/s recovering 2024-09-21T12:27:03.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:03 smithi114 ceph-mon[40867]: pgmap v328: 168 pgs: 168 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 4.4 KiB/s rd, 22 KiB/s wr, 1.03k op/s; 7.9 KiB/s, 0 objects/s recovering 2024-09-21T12:27:03.547 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:03.586 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:03.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:03.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:03.687 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:03.715 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:03.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:03.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:03.816 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:03.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:03.851 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:03.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:03.952 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:03.973 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:03.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:03.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:04.075 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:04.113 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:04.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:04.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:04.215 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:04.244 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:04.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:04.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:04.345 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:04.350 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T12:27:04.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:04.384 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:04.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:04.485 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:04.506 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:27:04.518 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:27:04.525 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:27:04.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:04.573 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:04.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:04.674 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:04.702 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:04.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:04.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:04.804 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:04.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:04.845 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:04.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:04.946 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:04.972 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:04.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:04.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:05.074 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:05.099 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:05.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:05.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:05.201 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:05.227 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:05.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:05.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:05.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:05 smithi114 ceph-mon[40867]: pgmap v329: 168 pgs: 168 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s rd, 0 B/s wr, 1.54k op/s 2024-09-21T12:27:05.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:05 smithi114 ceph-mon[30324]: pgmap v329: 168 pgs: 168 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s rd, 0 B/s wr, 1.54k op/s 2024-09-21T12:27:05.328 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:05.368 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:05.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:05.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:05.470 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:05.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:05 smithi183 ceph-mon[33277]: pgmap v329: 168 pgs: 168 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s rd, 0 B/s wr, 1.54k op/s 2024-09-21T12:27:05.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:05.498 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:05.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:05.599 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:05.624 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:05.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:05.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:05.726 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:05.752 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:05.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:05.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:05.854 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:05.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:05.899 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:05.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:05.951 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:27:05.957 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:27:06.000 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:06.027 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:06.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:06.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:06.128 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:06.156 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:06.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:06.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:06.258 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:06.288 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:06.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:06.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:06.389 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:06.418 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:06.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:06.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:06.520 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:06.553 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:06.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:06.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:06.655 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:06.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:06.680 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:06.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:06.781 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:06.808 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:06.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:06.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:06.910 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:06.940 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:06.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:06.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:07.042 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:07.080 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:07.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:07.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:07.154 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-21T12:27:07.182 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:07.209 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:07.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:07.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:07.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:27:06 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:27:07.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:07 smithi183 ceph-mon[33277]: pgmap v330: 168 pgs: 168 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 854 op/s 2024-09-21T12:27:07.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:07 smithi183 ceph-mon[33277]: from='client.25048 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:27:07.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:07 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3366562626' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:27:07.310 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:07.347 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:07.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:07.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:07.449 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:07.475 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:07.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:07.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:07.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:07 smithi114 ceph-mon[40867]: pgmap v330: 168 pgs: 168 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 854 op/s 2024-09-21T12:27:07.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:07 smithi114 ceph-mon[40867]: from='client.25048 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:27:07.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:07 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3366562626' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:27:07.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:07 smithi114 ceph-mon[30324]: pgmap v330: 168 pgs: 168 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 854 op/s 2024-09-21T12:27:07.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:07 smithi114 ceph-mon[30324]: from='client.25048 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:27:07.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:07 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3366562626' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:27:07.577 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:07.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:07.606 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:07.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:07.707 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:07.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:07.743 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:07.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:07.845 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:07.868 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:07.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:07.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:07.970 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:08.003 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:08.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:08.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:08.105 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:08.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:08.135 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:08.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:08.236 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:08.263 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:08.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:08.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:08.273 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:08 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:27:08.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:08 smithi114 ceph-mon[30324]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:27:08.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:08 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3366562626' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:27:08.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:08 smithi114 ceph-mon[30324]: osdmap e213: 8 total, 8 up, 7 in 2024-09-21T12:27:08.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:08 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:27:08.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:08 smithi114 ceph-mon[40867]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:27:08.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:08 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3366562626' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:27:08.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:08 smithi114 ceph-mon[40867]: osdmap e213: 8 total, 8 up, 7 in 2024-09-21T12:27:08.275 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:27:08.275 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:27:08.308 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:27:08.308 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:27:08.345 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:27:08.346 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:27:08.365 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:08.381 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:27:08.381 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:27:08.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:08.406 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:08.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:08.411 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:27:08.412 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:27:08.446 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:27:08.446 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:27:08.482 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:27:08.482 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:27:08.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:08 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:27:08.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:08 smithi183 ceph-mon[33277]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:27:08.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:08 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3366562626' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:27:08.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:08 smithi183 ceph-mon[33277]: osdmap e213: 8 total, 8 up, 7 in 2024-09-21T12:27:08.507 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:08.517 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:27:08.517 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:27:08.540 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:08.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:08.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:08.572 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":330,"stamp":"2024-09-21T12:27:05.698943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5313075763,"num_objects":81262,"num_object_clones":0,"num_object_copies":243786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81262,"num_whiteouts":0,"num_read":1115,"num_read_kb":867,"num_write":180201,"num_write_kb":8346478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":218,"num_bytes_recovered":4424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68013,"ondisk_log_size":68013,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":17667792,"kb_used_data":15737680,"kb_used_omap":10,"kb_used_meta":1929909,"kb_avail":638548272,"statfs":{"total":671965249536,"available":653873430528,"internally_reserved":0,"allocated":16115384320,"data_stored":16112900410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":1976226923},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,6,8,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-449198169,"num_objects":-6856,"num_object_clones":0,"num_object_copies":-20568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6856,"num_whiteouts":0,"num_read":-10,"num_read_kb":-25,"num_write":6842,"num_write_kb":-222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":-80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002609"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":337,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100762+0000","last_change":"2024-09-21T12:24:50.115300+0000","last_active":"2024-09-21T12:27:00.100762+0000","last_peered":"2024-09-21T12:27:00.100762+0000","last_clean":"2024-09-21T12:27:00.100762+0000","last_became_active":"2024-09-21T12:20:15.309431+0000","last_became_peered":"2024-09-21T12:20:15.309431+0000","last_unstale":"2024-09-21T12:27:00.100762+0000","last_undegraded":"2024-09-21T12:27:00.100762+0000","last_fullsized":"2024-09-21T12:27:00.100762+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:50.115253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:50.115253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:04.907671+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":307,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.098857+0000","last_change":"2024-09-21T12:26:09.823435+0000","last_active":"2024-09-21T12:27:00.098857+0000","last_peered":"2024-09-21T12:27:00.098857+0000","last_clean":"2024-09-21T12:27:00.098857+0000","last_became_active":"2024-09-21T12:22:50.909699+0000","last_became_peered":"2024-09-21T12:22:50.909699+0000","last_unstale":"2024-09-21T12:27:00.098857+0000","last_undegraded":"2024-09-21T12:27:00.098857+0000","last_fullsized":"2024-09-21T12:27:00.098857+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:26:09.823363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:09.823363+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:17.313787+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":328,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.102340+0000","last_change":"2024-09-21T12:24:34.311501+0000","last_active":"2024-09-21T12:27:00.102340+0000","last_peered":"2024-09-21T12:27:00.102340+0000","last_clean":"2024-09-21T12:27:00.102340+0000","last_became_active":"2024-09-21T12:20:19.314908+0000","last_became_peered":"2024-09-21T12:20:19.314908+0000","last_unstale":"2024-09-21T12:27:00.102340+0000","last_undegraded":"2024-09-21T12:27:00.102340+0000","last_fullsized":"2024-09-21T12:27:00.102340+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:34.311451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:34.311451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":252,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.101725+0000","last_change":"2024-09-21T12:22:50.914064+0000","last_active":"2024-09-21T12:27:00.101725+0000","last_peered":"2024-09-21T12:27:00.101725+0000","last_clean":"2024-09-21T12:27:00.101725+0000","last_became_active":"2024-09-21T12:22:50.913743+0000","last_became_peered":"2024-09-21T12:22:50.913743+0000","last_unstale":"2024-09-21T12:27:00.101725+0000","last_undegraded":"2024-09-21T12:27:00.101725+0000","last_fullsized":"2024-09-21T12:27:00.101725+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:58.189665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:21:58.189665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:07.338098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":314,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.103625+0000","last_change":"2024-09-21T12:25:48.316122+0000","last_active":"2024-09-21T12:27:00.103625+0000","last_peered":"2024-09-21T12:27:00.103625+0000","last_clean":"2024-09-21T12:27:00.103625+0000","last_became_active":"2024-09-21T12:20:21.595301+0000","last_became_peered":"2024-09-21T12:20:21.595301+0000","last_unstale":"2024-09-21T12:27:00.103625+0000","last_undegraded":"2024-09-21T12:27:00.103625+0000","last_fullsized":"2024-09-21T12:27:00.103625+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:48.316039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:48.316039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:08.665134+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":335,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.107589+0000","last_change":"2024-09-21T12:24:49.701972+0000","last_active":"2024-09-21T12:27:00.107589+0000","last_peered":"2024-09-21T12:27:00.107589+0000","last_clean":"2024-09-21T12:27:00.107589+0000","last_became_active":"2024-09-21T12:20:15.308655+0000","last_became_peered":"2024-09-21T12:20:15.308655+0000","last_unstale":"2024-09-21T12:27:00.107589+0000","last_undegraded":"2024-09-21T12:27:00.107589+0000","last_fullsized":"2024-09-21T12:27:00.107589+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:49.701915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:49.701915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":355,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.103086+0000","last_change":"2024-09-21T12:25:01.898461+0000","last_active":"2024-09-21T12:27:00.103086+0000","last_peered":"2024-09-21T12:27:00.103086+0000","last_clean":"2024-09-21T12:27:00.103086+0000","last_became_active":"2024-09-21T12:20:17.305220+0000","last_became_peered":"2024-09-21T12:20:17.305220+0000","last_unstale":"2024-09-21T12:27:00.103086+0000","last_undegraded":"2024-09-21T12:27:00.103086+0000","last_fullsized":"2024-09-21T12:27:00.103086+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:25:01.898376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:01.898376+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":269,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.098973+0000","last_change":"2024-09-21T12:25:04.827596+0000","last_active":"2024-09-21T12:27:00.098973+0000","last_peered":"2024-09-21T12:27:00.098973+0000","last_clean":"2024-09-21T12:27:00.098973+0000","last_became_active":"2024-09-21T12:22:50.926240+0000","last_became_peered":"2024-09-21T12:22:50.926240+0000","last_unstale":"2024-09-21T12:27:00.098973+0000","last_undegraded":"2024-09-21T12:27:00.098973+0000","last_fullsized":"2024-09-21T12:27:00.098973+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:04.827529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:04.827529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:35.929060+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":173,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100538+0000","last_change":"2024-09-21T12:26:54.084511+0000","last_active":"2024-09-21T12:27:00.100538+0000","last_peered":"2024-09-21T12:27:00.100538+0000","last_clean":"2024-09-21T12:27:00.100538+0000","last_became_active":"2024-09-21T12:26:54.083981+0000","last_became_peered":"2024-09-21T12:26:54.083981+0000","last_unstale":"2024-09-21T12:27:00.100538+0000","last_undegraded":"2024-09-21T12:27:00.100538+0000","last_fullsized":"2024-09-21T12:27:00.100538+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:19.446251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:19.446251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:35.995013+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":316,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.109395+0000","last_change":"2024-09-21T12:24:25.640253+0000","last_active":"2024-09-21T12:27:00.109395+0000","last_peered":"2024-09-21T12:27:00.109395+0000","last_clean":"2024-09-21T12:27:00.109395+0000","last_became_active":"2024-09-21T12:22:50.927804+0000","last_became_peered":"2024-09-21T12:22:50.927804+0000","last_unstale":"2024-09-21T12:27:00.109395+0000","last_undegraded":"2024-09-21T12:27:00.109395+0000","last_fullsized":"2024-09-21T12:27:00.109395+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:24:25.640169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:25.640169+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":335,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.102045+0000","last_change":"2024-09-21T12:26:54.497247+0000","last_active":"2024-09-21T12:27:00.102045+0000","last_peered":"2024-09-21T12:27:00.102045+0000","last_clean":"2024-09-21T12:27:00.102045+0000","last_became_active":"2024-09-21T12:20:19.308184+0000","last_became_peered":"2024-09-21T12:20:19.308184+0000","last_unstale":"2024-09-21T12:27:00.102045+0000","last_undegraded":"2024-09-21T12:27:00.102045+0000","last_fullsized":"2024-09-21T12:27:00.102045+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:54.497175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:54.497175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:19.740097+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":324,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.095200+0000","last_change":"2024-09-21T12:25:31.616293+0000","last_active":"2024-09-21T12:27:00.095200+0000","last_peered":"2024-09-21T12:27:00.095200+0000","last_clean":"2024-09-21T12:27:00.095200+0000","last_became_active":"2024-09-21T12:20:21.309190+0000","last_became_peered":"2024-09-21T12:20:21.309190+0000","last_unstale":"2024-09-21T12:27:00.095200+0000","last_undegraded":"2024-09-21T12:27:00.095200+0000","last_fullsized":"2024-09-21T12:27:00.095200+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:31.616219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:31.616219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":174,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.101987+0000","last_change":"2024-09-21T12:26:59.165789+0000","last_active":"2024-09-21T12:27:00.101987+0000","last_peered":"2024-09-21T12:27:00.101987+0000","last_clean":"2024-09-21T12:27:00.101987+0000","last_became_active":"2024-09-21T12:26:54.085055+0000","last_became_peered":"2024-09-21T12:26:54.085055+0000","last_unstale":"2024-09-21T12:27:00.101987+0000","last_undegraded":"2024-09-21T12:27:00.101987+0000","last_fullsized":"2024-09-21T12:27:00.101987+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:59.165721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:59.165721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:26.609580+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":302,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.106036+0000","last_change":"2024-09-21T12:24:28.624430+0000","last_active":"2024-09-21T12:27:00.106036+0000","last_peered":"2024-09-21T12:27:00.106036+0000","last_clean":"2024-09-21T12:27:00.106036+0000","last_became_active":"2024-09-21T12:22:50.918794+0000","last_became_peered":"2024-09-21T12:22:50.918794+0000","last_unstale":"2024-09-21T12:27:00.106036+0000","last_undegraded":"2024-09-21T12:27:00.106036+0000","last_fullsized":"2024-09-21T12:27:00.106036+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-21T12:24:28.624364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:28.624364+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:42.310054+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":340,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.099423+0000","last_change":"2024-09-21T12:25:01.897596+0000","last_active":"2024-09-21T12:27:00.099423+0000","last_peered":"2024-09-21T12:27:00.099423+0000","last_clean":"2024-09-21T12:27:00.099423+0000","last_became_active":"2024-09-21T12:22:50.921780+0000","last_became_peered":"2024-09-21T12:22:50.921780+0000","last_unstale":"2024-09-21T12:27:00.099423+0000","last_undegraded":"2024-09-21T12:27:00.099423+0000","last_fullsized":"2024-09-21T12:27:00.099423+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:01.897527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:01.897527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:35.032952+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":331,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.101697+0000","last_change":"2024-09-21T12:24:34.367580+0000","last_active":"2024-09-21T12:27:00.101697+0000","last_peered":"2024-09-21T12:27:00.101697+0000","last_clean":"2024-09-21T12:27:00.101697+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:27:00.101697+0000","last_undegraded":"2024-09-21T12:27:00.101697+0000","last_fullsized":"2024-09-21T12:27:00.101697+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:34.367532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:34.367532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:35.460132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":276,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100306+0000","last_change":"2024-09-21T12:26:44.845249+0000","last_active":"2024-09-21T12:27:00.100306+0000","last_peered":"2024-09-21T12:27:00.100306+0000","last_clean":"2024-09-21T12:27:00.100306+0000","last_became_active":"2024-09-21T12:22:50.914792+0000","last_became_peered":"2024-09-21T12:22:50.914792+0000","last_unstale":"2024-09-21T12:27:00.100306+0000","last_undegraded":"2024-09-21T12:27:00.100306+0000","last_fullsized":"2024-09-21T12:27:00.100306+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:44.845176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:44.845176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:47.131661+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":365,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100371+0000","last_change":"2024-09-21T12:24:30.033946+0000","last_active":"2024-09-21T12:27:00.100371+0000","last_peered":"2024-09-21T12:27:00.100371+0000","last_clean":"2024-09-21T12:27:00.100371+0000","last_became_active":"2024-09-21T12:22:50.921576+0000","last_became_peered":"2024-09-21T12:22:50.921576+0000","last_unstale":"2024-09-21T12:27:00.100371+0000","last_undegraded":"2024-09-21T12:27:00.100371+0000","last_fullsized":"2024-09-21T12:27:00.100371+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:30.033874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:30.033874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:00.878523+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":357,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.107802+0000","last_change":"2024-09-21T12:24:50.834579+0000","last_active":"2024-09-21T12:27:00.107802+0000","last_peered":"2024-09-21T12:27:00.107802+0000","last_clean":"2024-09-21T12:27:00.107802+0000","last_became_active":"2024-09-21T12:20:17.307828+0000","last_became_peered":"2024-09-21T12:20:17.307828+0000","last_unstale":"2024-09-21T12:27:00.107802+0000","last_undegraded":"2024-09-21T12:27:00.107802+0000","last_fullsized":"2024-09-21T12:27:00.107802+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:24:50.834503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:50.834503+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:22.112484+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":356,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.101703+0000","last_change":"2024-09-21T12:25:46.906275+0000","last_active":"2024-09-21T12:27:00.101703+0000","last_peered":"2024-09-21T12:27:00.101703+0000","last_clean":"2024-09-21T12:27:00.101703+0000","last_became_active":"2024-09-21T12:25:46.905878+0000","last_became_peered":"2024-09-21T12:25:46.905878+0000","last_unstale":"2024-09-21T12:27:00.101703+0000","last_undegraded":"2024-09-21T12:27:00.101703+0000","last_fullsized":"2024-09-21T12:27:00.101703+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:48.435769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:48.435769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":315,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.106877+0000","last_change":"2024-09-21T12:23:58.354628+0000","last_active":"2024-09-21T12:27:00.106877+0000","last_peered":"2024-09-21T12:27:00.106877+0000","last_clean":"2024-09-21T12:27:00.106877+0000","last_became_active":"2024-09-21T12:20:21.592796+0000","last_became_peered":"2024-09-21T12:20:21.592796+0000","last_unstale":"2024-09-21T12:27:00.106877+0000","last_undegraded":"2024-09-21T12:27:00.106877+0000","last_fullsized":"2024-09-21T12:27:00.106877+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:58.354570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:58.354570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:33.466611+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":268,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.107104+0000","last_change":"2024-09-21T12:24:55.447829+0000","last_active":"2024-09-21T12:27:00.107104+0000","last_peered":"2024-09-21T12:27:00.107104+0000","last_clean":"2024-09-21T12:27:00.107104+0000","last_became_active":"2024-09-21T12:22:50.925403+0000","last_became_peered":"2024-09-21T12:22:50.925403+0000","last_unstale":"2024-09-21T12:27:00.107104+0000","last_undegraded":"2024-09-21T12:27:00.107104+0000","last_fullsized":"2024-09-21T12:27:00.107104+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:55.447755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:55.447755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:40.161143+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":410,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.106341+0000","last_change":"2024-09-21T12:25:31.270272+0000","last_active":"2024-09-21T12:27:00.106341+0000","last_peered":"2024-09-21T12:27:00.106341+0000","last_clean":"2024-09-21T12:27:00.106341+0000","last_became_active":"2024-09-21T12:22:50.912144+0000","last_became_peered":"2024-09-21T12:22:50.912144+0000","last_unstale":"2024-09-21T12:27:00.106341+0000","last_undegraded":"2024-09-21T12:27:00.106341+0000","last_fullsized":"2024-09-21T12:27:00.106341+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:25:31.270165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:31.270165+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":253,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.103634+0000","last_change":"2024-09-21T12:26:52.295004+0000","last_active":"2024-09-21T12:27:00.103634+0000","last_peered":"2024-09-21T12:27:00.103634+0000","last_clean":"2024-09-21T12:27:00.103634+0000","last_became_active":"2024-09-21T12:22:50.918326+0000","last_became_peered":"2024-09-21T12:22:50.918326+0000","last_unstale":"2024-09-21T12:27:00.103634+0000","last_undegraded":"2024-09-21T12:27:00.103634+0000","last_fullsized":"2024-09-21T12:27:00.103634+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:52.294931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:52.294931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:18.460650+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":316,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100306+0000","last_change":"2024-09-21T12:23:51.065780+0000","last_active":"2024-09-21T12:27:00.100306+0000","last_peered":"2024-09-21T12:27:00.100306+0000","last_clean":"2024-09-21T12:27:00.100306+0000","last_became_active":"2024-09-21T12:20:21.312976+0000","last_became_peered":"2024-09-21T12:20:21.312976+0000","last_unstale":"2024-09-21T12:27:00.100306+0000","last_undegraded":"2024-09-21T12:27:00.100306+0000","last_fullsized":"2024-09-21T12:27:00.100306+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:23:51.065733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:51.065733+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:06.800726+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":370,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100321+0000","last_change":"2024-09-21T12:26:18.112687+0000","last_active":"2024-09-21T12:27:00.100321+0000","last_peered":"2024-09-21T12:27:00.100321+0000","last_clean":"2024-09-21T12:27:00.100321+0000","last_became_active":"2024-09-21T12:22:50.910933+0000","last_became_peered":"2024-09-21T12:22:50.910933+0000","last_unstale":"2024-09-21T12:27:00.100321+0000","last_undegraded":"2024-09-21T12:27:00.100321+0000","last_fullsized":"2024-09-21T12:27:00.100321+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:26:18.112614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:18.112614+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:47.795526+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":340,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.101710+0000","last_change":"2024-09-21T12:24:03.059782+0000","last_active":"2024-09-21T12:27:00.101710+0000","last_peered":"2024-09-21T12:27:00.101710+0000","last_clean":"2024-09-21T12:27:00.101710+0000","last_became_active":"2024-09-21T12:20:17.695977+0000","last_became_peered":"2024-09-21T12:20:17.695977+0000","last_unstale":"2024-09-21T12:27:00.101710+0000","last_undegraded":"2024-09-21T12:27:00.101710+0000","last_fullsized":"2024-09-21T12:27:00.101710+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:24:03.059549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:03.059549+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:31.790959+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"121'11","reported_seq":462,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100909+0000","last_change":"2024-09-21T12:26:10.846256+0000","last_active":"2024-09-21T12:27:00.100909+0000","last_peered":"2024-09-21T12:27:00.100909+0000","last_clean":"2024-09-21T12:27:00.100909+0000","last_became_active":"2024-09-21T12:22:50.916683+0000","last_became_peered":"2024-09-21T12:22:50.916683+0000","last_unstale":"2024-09-21T12:27:00.100909+0000","last_undegraded":"2024-09-21T12:27:00.100909+0000","last_fullsized":"2024-09-21T12:27:00.100909+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:26:10.846183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:10.846183+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:32.395882+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"212'10940","reported_seq":11056,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:01.890740+0000","last_change":"2024-09-21T12:24:46.316817+0000","last_active":"2024-09-21T12:27:01.890740+0000","last_peered":"2024-09-21T12:27:01.890740+0000","last_clean":"2024-09-21T12:27:01.890740+0000","last_became_active":"2024-09-21T12:24:46.316132+0000","last_became_peered":"2024-09-21T12:24:46.316132+0000","last_unstale":"2024-09-21T12:27:01.890740+0000","last_undegraded":"2024-09-21T12:27:01.890740+0000","last_fullsized":"2024-09-21T12:27:01.890740+0000","mapping_epoch":160,"log_start":"195'6943","ondisk_log_start":"195'6943","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":3997,"log_dups_size":0,"ondisk_log_size":3997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:05.983258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328204288,"num_objects":5008,"num_object_clones":0,"num_object_copies":15024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10940,"num_write_kb":510336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"76'79","reported_seq":29,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.105473+0000","last_change":"2024-09-21T12:26:55.640843+0000","last_active":"2024-09-21T12:27:00.105473+0000","last_peered":"2024-09-21T12:27:00.105473+0000","last_clean":"2024-09-21T12:27:00.105473+0000","last_became_active":"2024-09-21T12:26:54.091451+0000","last_became_peered":"2024-09-21T12:26:54.091451+0000","last_unstale":"2024-09-21T12:27:00.105473+0000","last_undegraded":"2024-09-21T12:27:00.105473+0000","last_fullsized":"2024-09-21T12:27:00.105473+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:55.640420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:55.640420+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:56.693423+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":187,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.105966+0000","last_change":"2024-09-21T12:24:59.803037+0000","last_active":"2024-09-21T12:27:00.105966+0000","last_peered":"2024-09-21T12:27:00.105966+0000","last_clean":"2024-09-21T12:27:00.105966+0000","last_became_active":"2024-09-21T12:23:46.910112+0000","last_became_peered":"2024-09-21T12:23:46.910112+0000","last_unstale":"2024-09-21T12:27:00.105966+0000","last_undegraded":"2024-09-21T12:27:00.105966+0000","last_fullsized":"2024-09-21T12:27:00.105966+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:24:59.802945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:59.802945+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:26.207932+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":327,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.103404+0000","last_change":"2024-09-21T12:22:14.036290+0000","last_active":"2024-09-21T12:27:00.103404+0000","last_peered":"2024-09-21T12:27:00.103404+0000","last_clean":"2024-09-21T12:27:00.103404+0000","last_became_active":"2024-09-21T12:20:15.295774+0000","last_became_peered":"2024-09-21T12:20:15.295774+0000","last_unstale":"2024-09-21T12:27:00.103404+0000","last_undegraded":"2024-09-21T12:27:00.103404+0000","last_fullsized":"2024-09-21T12:27:00.103404+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:14.036236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:14.036236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"210'14","reported_seq":22,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.101303+0000","last_change":"2024-09-21T12:26:54.091833+0000","last_active":"2024-09-21T12:27:00.101303+0000","last_peered":"2024-09-21T12:27:00.101303+0000","last_clean":"2024-09-21T12:27:00.101303+0000","last_became_active":"2024-09-21T12:26:54.091451+0000","last_became_peered":"2024-09-21T12:26:54.091451+0000","last_unstale":"2024-09-21T12:27:00.101303+0000","last_undegraded":"2024-09-21T12:27:00.101303+0000","last_fullsized":"2024-09-21T12:27:00.101303+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'13","last_scrub_stamp":"2024-09-21T12:25:33.474391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:33.474391+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":39,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.101611+0000","last_change":"2024-09-21T12:26:58.114572+0000","last_active":"2024-09-21T12:27:00.101611+0000","last_peered":"2024-09-21T12:27:00.101611+0000","last_clean":"2024-09-21T12:27:00.101611+0000","last_became_active":"2024-09-21T12:26:54.089699+0000","last_became_peered":"2024-09-21T12:26:54.089699+0000","last_unstale":"2024-09-21T12:27:00.101611+0000","last_undegraded":"2024-09-21T12:27:00.101611+0000","last_fullsized":"2024-09-21T12:27:00.101611+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:26:58.114502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:58.114502+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:20.979235+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.c","version":"212'11243","reported_seq":11356,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:02.247691+0000","last_change":"2024-09-21T12:24:46.326848+0000","last_active":"2024-09-21T12:27:02.247691+0000","last_peered":"2024-09-21T12:27:02.247691+0000","last_clean":"2024-09-21T12:27:02.247691+0000","last_became_active":"2024-09-21T12:24:46.326586+0000","last_became_peered":"2024-09-21T12:24:46.326586+0000","last_unstale":"2024-09-21T12:27:02.247691+0000","last_undegraded":"2024-09-21T12:27:02.247691+0000","last_fullsized":"2024-09-21T12:27:02.247691+0000","mapping_epoch":160,"log_start":"195'7136","ondisk_log_start":"195'7136","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4107,"log_dups_size":0,"ondisk_log_size":4107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:36.738404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335216640,"num_objects":5115,"num_object_clones":0,"num_object_copies":15345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11243,"num_write_kb":523456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"76'79","reported_seq":243,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.108427+0000","last_change":"2024-09-21T12:26:47.578416+0000","last_active":"2024-09-21T12:27:00.108427+0000","last_peered":"2024-09-21T12:27:00.108427+0000","last_clean":"2024-09-21T12:27:00.108427+0000","last_became_active":"2024-09-21T12:26:44.955678+0000","last_became_peered":"2024-09-21T12:26:44.955678+0000","last_unstale":"2024-09-21T12:27:00.108427+0000","last_undegraded":"2024-09-21T12:27:00.108427+0000","last_fullsized":"2024-09-21T12:27:00.108427+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:47.578226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:47.578226+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:02.771684+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":363,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.103196+0000","last_change":"2024-09-21T12:25:35.325593+0000","last_active":"2024-09-21T12:27:00.103196+0000","last_peered":"2024-09-21T12:27:00.103196+0000","last_clean":"2024-09-21T12:27:00.103196+0000","last_became_active":"2024-09-21T12:20:17.693179+0000","last_became_peered":"2024-09-21T12:20:17.693179+0000","last_unstale":"2024-09-21T12:27:00.103196+0000","last_undegraded":"2024-09-21T12:27:00.103196+0000","last_fullsized":"2024-09-21T12:27:00.103196+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-21T12:25:35.325493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:35.325493+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":327,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.106947+0000","last_change":"2024-09-21T12:21:18.056305+0000","last_active":"2024-09-21T12:27:00.106947+0000","last_peered":"2024-09-21T12:27:00.106947+0000","last_clean":"2024-09-21T12:27:00.106947+0000","last_became_active":"2024-09-21T12:20:15.297340+0000","last_became_peered":"2024-09-21T12:20:15.297340+0000","last_unstale":"2024-09-21T12:27:00.106947+0000","last_undegraded":"2024-09-21T12:27:00.106947+0000","last_fullsized":"2024-09-21T12:27:00.106947+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:18.056261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:18.056261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:07.391958+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":269,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.102611+0000","last_change":"2024-09-21T12:25:38.374222+0000","last_active":"2024-09-21T12:27:00.102611+0000","last_peered":"2024-09-21T12:27:00.102611+0000","last_clean":"2024-09-21T12:27:00.102611+0000","last_became_active":"2024-09-21T12:22:50.930858+0000","last_became_peered":"2024-09-21T12:22:50.930858+0000","last_unstale":"2024-09-21T12:27:00.102611+0000","last_undegraded":"2024-09-21T12:27:00.102611+0000","last_fullsized":"2024-09-21T12:27:00.102611+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:38.374146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:38.374146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":268,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.109310+0000","last_change":"2024-09-21T12:26:44.325219+0000","last_active":"2024-09-21T12:27:00.109310+0000","last_peered":"2024-09-21T12:27:00.109310+0000","last_clean":"2024-09-21T12:27:00.109310+0000","last_became_active":"2024-09-21T12:22:50.918940+0000","last_became_peered":"2024-09-21T12:22:50.918940+0000","last_unstale":"2024-09-21T12:27:00.109310+0000","last_undegraded":"2024-09-21T12:27:00.109310+0000","last_fullsized":"2024-09-21T12:27:00.109310+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:44.325124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:44.325124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:06.833401+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"212'11174","reported_seq":11297,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:01.601782+0000","last_change":"2024-09-21T12:26:54.101934+0000","last_active":"2024-09-21T12:27:01.601782+0000","last_peered":"2024-09-21T12:27:01.601782+0000","last_clean":"2024-09-21T12:27:01.601782+0000","last_became_active":"2024-09-21T12:26:54.088525+0000","last_became_peered":"2024-09-21T12:26:54.088525+0000","last_unstale":"2024-09-21T12:27:01.601782+0000","last_undegraded":"2024-09-21T12:27:01.601782+0000","last_fullsized":"2024-09-21T12:27:01.601782+0000","mapping_epoch":208,"log_start":"194'6849","ondisk_log_start":"194'6849","created":160,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4325,"log_dups_size":0,"ondisk_log_size":4325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334692376,"num_objects":5108,"num_object_clones":0,"num_object_copies":15324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5108,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11174,"num_write_kb":520961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"76'79","reported_seq":259,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.103383+0000","last_change":"2024-09-21T12:26:45.354836+0000","last_active":"2024-09-21T12:27:00.103383+0000","last_peered":"2024-09-21T12:27:00.103383+0000","last_clean":"2024-09-21T12:27:00.103383+0000","last_became_active":"2024-09-21T12:26:44.960093+0000","last_became_peered":"2024-09-21T12:26:44.960093+0000","last_unstale":"2024-09-21T12:27:00.103383+0000","last_undegraded":"2024-09-21T12:27:00.103383+0000","last_fullsized":"2024-09-21T12:27:00.103383+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:45.354774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:45.354774+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:14.195205+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":333,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.101721+0000","last_change":"2024-09-21T12:26:51.463759+0000","last_active":"2024-09-21T12:27:00.101721+0000","last_peered":"2024-09-21T12:27:00.101721+0000","last_clean":"2024-09-21T12:27:00.101721+0000","last_became_active":"2024-09-21T12:22:50.926008+0000","last_became_peered":"2024-09-21T12:22:50.926008+0000","last_unstale":"2024-09-21T12:27:00.101721+0000","last_undegraded":"2024-09-21T12:27:00.101721+0000","last_fullsized":"2024-09-21T12:27:00.101721+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-21T12:26:51.463698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:51.463698+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:20.413839+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":371,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.102103+0000","last_change":"2024-09-21T12:26:59.887632+0000","last_active":"2024-09-21T12:27:00.102103+0000","last_peered":"2024-09-21T12:27:00.102103+0000","last_clean":"2024-09-21T12:27:00.102103+0000","last_became_active":"2024-09-21T12:22:50.923284+0000","last_became_peered":"2024-09-21T12:22:50.923284+0000","last_unstale":"2024-09-21T12:27:00.102103+0000","last_undegraded":"2024-09-21T12:27:00.102103+0000","last_fullsized":"2024-09-21T12:27:00.102103+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:26:59.887565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:59.887565+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:22.402122+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"121'11","reported_seq":450,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.103005+0000","last_change":"2024-09-21T12:26:14.490045+0000","last_active":"2024-09-21T12:27:00.103005+0000","last_peered":"2024-09-21T12:27:00.103005+0000","last_clean":"2024-09-21T12:27:00.103005+0000","last_became_active":"2024-09-21T12:22:50.930342+0000","last_became_peered":"2024-09-21T12:22:50.930342+0000","last_unstale":"2024-09-21T12:27:00.103005+0000","last_undegraded":"2024-09-21T12:27:00.103005+0000","last_fullsized":"2024-09-21T12:27:00.103005+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:26:14.489972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:14.489972+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:31.388160+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":316,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.099316+0000","last_change":"2024-09-21T12:21:21.835669+0000","last_active":"2024-09-21T12:27:00.099316+0000","last_peered":"2024-09-21T12:27:00.099316+0000","last_clean":"2024-09-21T12:27:00.099316+0000","last_became_active":"2024-09-21T12:20:21.593367+0000","last_became_peered":"2024-09-21T12:20:21.593367+0000","last_unstale":"2024-09-21T12:27:00.099316+0000","last_undegraded":"2024-09-21T12:27:00.099316+0000","last_fullsized":"2024-09-21T12:27:00.099316+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:21.835623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:21:21.835623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:31.973693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"212'11140","reported_seq":11255,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:02.252516+0000","last_change":"2024-09-21T12:24:46.325776+0000","last_active":"2024-09-21T12:27:02.252516+0000","last_peered":"2024-09-21T12:27:02.252516+0000","last_clean":"2024-09-21T12:27:02.252516+0000","last_became_active":"2024-09-21T12:24:46.325260+0000","last_became_peered":"2024-09-21T12:24:46.325260+0000","last_unstale":"2024-09-21T12:27:02.252516+0000","last_undegraded":"2024-09-21T12:27:02.252516+0000","last_fullsized":"2024-09-21T12:27:02.252516+0000","mapping_epoch":160,"log_start":"195'7055","ondisk_log_start":"195'7055","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4085,"log_dups_size":0,"ondisk_log_size":4085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326893568,"num_objects":4988,"num_object_clones":0,"num_object_copies":14964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11140,"num_write_kb":516096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"76'79","reported_seq":344,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100702+0000","last_change":"2024-09-21T12:26:50.869921+0000","last_active":"2024-09-21T12:27:00.100702+0000","last_peered":"2024-09-21T12:27:00.100702+0000","last_clean":"2024-09-21T12:27:00.100702+0000","last_became_active":"2024-09-21T12:26:44.965536+0000","last_became_peered":"2024-09-21T12:26:44.965536+0000","last_unstale":"2024-09-21T12:27:00.100702+0000","last_undegraded":"2024-09-21T12:27:00.100702+0000","last_fullsized":"2024-09-21T12:27:00.100702+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:50.869852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:50.869852+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:18.254060+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":397,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.103991+0000","last_change":"2024-09-21T12:24:01.388987+0000","last_active":"2024-09-21T12:27:00.103991+0000","last_peered":"2024-09-21T12:27:00.103991+0000","last_clean":"2024-09-21T12:27:00.103991+0000","last_became_active":"2024-09-21T12:22:50.925840+0000","last_became_peered":"2024-09-21T12:22:50.925840+0000","last_unstale":"2024-09-21T12:27:00.103991+0000","last_undegraded":"2024-09-21T12:27:00.103991+0000","last_fullsized":"2024-09-21T12:27:00.103991+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:24:01.388905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:01.388905+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:24.187759+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":336,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.101964+0000","last_change":"2024-09-21T12:26:16.410118+0000","last_active":"2024-09-21T12:27:00.101964+0000","last_peered":"2024-09-21T12:27:00.101964+0000","last_clean":"2024-09-21T12:27:00.101964+0000","last_became_active":"2024-09-21T12:20:15.308518+0000","last_became_peered":"2024-09-21T12:20:15.308518+0000","last_unstale":"2024-09-21T12:27:00.101964+0000","last_undegraded":"2024-09-21T12:27:00.101964+0000","last_fullsized":"2024-09-21T12:27:00.101964+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:16.410067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:16.410067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:24.383217+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":422,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.099771+0000","last_change":"2024-09-21T12:25:42.615523+0000","last_active":"2024-09-21T12:27:00.099771+0000","last_peered":"2024-09-21T12:27:00.099771+0000","last_clean":"2024-09-21T12:27:00.099771+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:27:00.099771+0000","last_undegraded":"2024-09-21T12:27:00.099771+0000","last_fullsized":"2024-09-21T12:27:00.099771+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:25:42.615446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:42.615446+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:06.344897+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":317,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.098107+0000","last_change":"2024-09-21T12:22:51.987032+0000","last_active":"2024-09-21T12:27:00.098107+0000","last_peered":"2024-09-21T12:27:00.098107+0000","last_clean":"2024-09-21T12:27:00.098107+0000","last_became_active":"2024-09-21T12:20:21.315801+0000","last_became_peered":"2024-09-21T12:20:21.315801+0000","last_unstale":"2024-09-21T12:27:00.098107+0000","last_undegraded":"2024-09-21T12:27:00.098107+0000","last_fullsized":"2024-09-21T12:27:00.098107+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:51.986953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:22:51.986953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:22.782321+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"212'11357","reported_seq":11470,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:02.198067+0000","last_change":"2024-09-21T12:24:46.324384+0000","last_active":"2024-09-21T12:27:02.198067+0000","last_peered":"2024-09-21T12:27:02.198067+0000","last_clean":"2024-09-21T12:27:02.198067+0000","last_became_active":"2024-09-21T12:24:46.323283+0000","last_became_peered":"2024-09-21T12:24:46.323283+0000","last_unstale":"2024-09-21T12:27:02.198067+0000","last_undegraded":"2024-09-21T12:27:02.198067+0000","last_fullsized":"2024-09-21T12:27:02.198067+0000","mapping_epoch":160,"log_start":"198'7503","ondisk_log_start":"198'7503","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":3854,"log_dups_size":0,"ondisk_log_size":3854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328269824,"num_objects":5009,"num_object_clones":0,"num_object_copies":15027,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11357,"num_write_kb":523712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"76'79","reported_seq":344,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100226+0000","last_change":"2024-09-21T12:26:52.857078+0000","last_active":"2024-09-21T12:27:00.100226+0000","last_peered":"2024-09-21T12:27:00.100226+0000","last_clean":"2024-09-21T12:27:00.100226+0000","last_became_active":"2024-09-21T12:26:44.963927+0000","last_became_peered":"2024-09-21T12:26:44.963927+0000","last_unstale":"2024-09-21T12:27:00.100226+0000","last_undegraded":"2024-09-21T12:27:00.100226+0000","last_fullsized":"2024-09-21T12:27:00.100226+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:52.857011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:52.857011+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:16.015733+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":369,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.099911+0000","last_change":"2024-09-21T12:26:14.834328+0000","last_active":"2024-09-21T12:27:00.099911+0000","last_peered":"2024-09-21T12:27:00.099911+0000","last_clean":"2024-09-21T12:27:00.099911+0000","last_became_active":"2024-09-21T12:20:17.693067+0000","last_became_peered":"2024-09-21T12:20:17.693067+0000","last_unstale":"2024-09-21T12:27:00.099911+0000","last_undegraded":"2024-09-21T12:27:00.099911+0000","last_fullsized":"2024-09-21T12:27:00.099911+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-21T12:26:14.834236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:14.834236+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:34.377077+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":329,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.102198+0000","last_change":"2024-09-21T12:26:12.131784+0000","last_active":"2024-09-21T12:27:00.102198+0000","last_peered":"2024-09-21T12:27:00.102198+0000","last_clean":"2024-09-21T12:27:00.102198+0000","last_became_active":"2024-09-21T12:20:15.308441+0000","last_became_peered":"2024-09-21T12:20:15.308441+0000","last_unstale":"2024-09-21T12:27:00.102198+0000","last_undegraded":"2024-09-21T12:27:00.102198+0000","last_fullsized":"2024-09-21T12:27:00.102198+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:12.131721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:12.131721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:19.426420+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":328,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.099290+0000","last_change":"2024-09-21T12:26:12.887076+0000","last_active":"2024-09-21T12:27:00.099290+0000","last_peered":"2024-09-21T12:27:00.099290+0000","last_clean":"2024-09-21T12:27:00.099290+0000","last_became_active":"2024-09-21T12:20:19.309598+0000","last_became_peered":"2024-09-21T12:20:19.309598+0000","last_unstale":"2024-09-21T12:27:00.099290+0000","last_undegraded":"2024-09-21T12:27:00.099290+0000","last_fullsized":"2024-09-21T12:27:00.099290+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:12.887007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:12.887007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:22.153550+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":317,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100089+0000","last_change":"2024-09-21T12:25:44.918356+0000","last_active":"2024-09-21T12:27:00.100089+0000","last_peered":"2024-09-21T12:27:00.100089+0000","last_clean":"2024-09-21T12:27:00.100089+0000","last_became_active":"2024-09-21T12:20:21.592828+0000","last_became_peered":"2024-09-21T12:20:21.592828+0000","last_unstale":"2024-09-21T12:27:00.100089+0000","last_undegraded":"2024-09-21T12:27:00.100089+0000","last_fullsized":"2024-09-21T12:27:00.100089+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:44.918276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:44.918276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:07.775801+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"212'11313","reported_seq":11433,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:02.205220+0000","last_change":"2024-09-21T12:26:54.107550+0000","last_active":"2024-09-21T12:27:02.205220+0000","last_peered":"2024-09-21T12:27:02.205220+0000","last_clean":"2024-09-21T12:27:02.205220+0000","last_became_active":"2024-09-21T12:26:54.084202+0000","last_became_peered":"2024-09-21T12:26:54.084202+0000","last_unstale":"2024-09-21T12:27:02.205220+0000","last_undegraded":"2024-09-21T12:27:02.205220+0000","last_fullsized":"2024-09-21T12:27:02.205220+0000","mapping_epoch":208,"log_start":"196'7396","ondisk_log_start":"196'7396","created":160,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":3917,"log_dups_size":0,"ondisk_log_size":3917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333774848,"num_objects":5093,"num_object_clones":0,"num_object_copies":15279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11313,"num_write_kb":524992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"76'79","reported_seq":116,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.098742+0000","last_change":"2024-09-21T12:26:57.868747+0000","last_active":"2024-09-21T12:27:00.098742+0000","last_peered":"2024-09-21T12:27:00.098742+0000","last_clean":"2024-09-21T12:27:00.098742+0000","last_became_active":"2024-09-21T12:26:54.092726+0000","last_became_peered":"2024-09-21T12:26:54.092726+0000","last_unstale":"2024-09-21T12:27:00.098742+0000","last_undegraded":"2024-09-21T12:27:00.098742+0000","last_fullsized":"2024-09-21T12:27:00.098742+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:57.868634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:57.868634+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:13.507332+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":455,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.101940+0000","last_change":"2024-09-21T12:26:09.082395+0000","last_active":"2024-09-21T12:27:00.101940+0000","last_peered":"2024-09-21T12:27:00.101940+0000","last_clean":"2024-09-21T12:27:00.101940+0000","last_became_active":"2024-09-21T12:20:17.307769+0000","last_became_peered":"2024-09-21T12:20:17.307769+0000","last_unstale":"2024-09-21T12:27:00.101940+0000","last_undegraded":"2024-09-21T12:27:00.101940+0000","last_fullsized":"2024-09-21T12:27:00.101940+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-21T12:26:09.082314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:09.082314+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:17.369834+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":260,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100288+0000","last_change":"2024-09-21T12:26:50.829669+0000","last_active":"2024-09-21T12:27:00.100288+0000","last_peered":"2024-09-21T12:27:00.100288+0000","last_clean":"2024-09-21T12:27:00.100288+0000","last_became_active":"2024-09-21T12:22:50.922002+0000","last_became_peered":"2024-09-21T12:22:50.922002+0000","last_unstale":"2024-09-21T12:27:00.100288+0000","last_undegraded":"2024-09-21T12:27:00.100288+0000","last_fullsized":"2024-09-21T12:27:00.100288+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:50.829599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:50.829599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:17.519325+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":47,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.107199+0000","last_change":"2024-09-21T12:26:57.166297+0000","last_active":"2024-09-21T12:27:00.107199+0000","last_peered":"2024-09-21T12:27:00.107199+0000","last_clean":"2024-09-21T12:27:00.107199+0000","last_became_active":"2024-09-21T12:26:54.083584+0000","last_became_peered":"2024-09-21T12:26:54.083584+0000","last_unstale":"2024-09-21T12:27:00.107199+0000","last_undegraded":"2024-09-21T12:27:00.107199+0000","last_fullsized":"2024-09-21T12:27:00.107199+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T12:26:57.166234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:26:57.166234+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:26.819412+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":350,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.101243+0000","last_change":"2024-09-21T12:24:53.112915+0000","last_active":"2024-09-21T12:27:00.101243+0000","last_peered":"2024-09-21T12:27:00.101243+0000","last_clean":"2024-09-21T12:27:00.101243+0000","last_became_active":"2024-09-21T12:22:50.921488+0000","last_became_peered":"2024-09-21T12:22:50.921488+0000","last_unstale":"2024-09-21T12:27:00.101243+0000","last_undegraded":"2024-09-21T12:27:00.101243+0000","last_fullsized":"2024-09-21T12:27:00.101243+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:53.112868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:53.112868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:05.952435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":344,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.106911+0000","last_change":"2024-09-21T12:26:10.435410+0000","last_active":"2024-09-21T12:27:00.106911+0000","last_peered":"2024-09-21T12:27:00.106911+0000","last_clean":"2024-09-21T12:27:00.106911+0000","last_became_active":"2024-09-21T12:22:50.917558+0000","last_became_peered":"2024-09-21T12:22:50.917558+0000","last_unstale":"2024-09-21T12:27:00.106911+0000","last_undegraded":"2024-09-21T12:27:00.106911+0000","last_fullsized":"2024-09-21T12:27:00.106911+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:10.435349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:10.435349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:19.570759+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"212'11260","reported_seq":11376,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:02.207361+0000","last_change":"2024-09-21T12:24:46.323849+0000","last_active":"2024-09-21T12:27:02.207361+0000","last_peered":"2024-09-21T12:27:02.207361+0000","last_clean":"2024-09-21T12:27:02.207361+0000","last_became_active":"2024-09-21T12:24:46.323306+0000","last_became_peered":"2024-09-21T12:24:46.323306+0000","last_unstale":"2024-09-21T12:27:02.207361+0000","last_undegraded":"2024-09-21T12:27:02.207361+0000","last_fullsized":"2024-09-21T12:27:02.207361+0000","mapping_epoch":160,"log_start":"196'7399","ondisk_log_start":"196'7399","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":3861,"log_dups_size":0,"ondisk_log_size":3861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333447168,"num_objects":5088,"num_object_clones":0,"num_object_copies":15264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11260,"num_write_kb":523136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'79","reported_seq":429,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.099150+0000","last_change":"2024-09-21T12:26:46.188733+0000","last_active":"2024-09-21T12:27:00.099150+0000","last_peered":"2024-09-21T12:27:00.099150+0000","last_clean":"2024-09-21T12:27:00.099150+0000","last_became_active":"2024-09-21T12:26:44.956003+0000","last_became_peered":"2024-09-21T12:26:44.956003+0000","last_unstale":"2024-09-21T12:27:00.099150+0000","last_undegraded":"2024-09-21T12:27:00.099150+0000","last_fullsized":"2024-09-21T12:27:00.099150+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:46.188649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:46.188649+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:04.754238+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":360,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.103765+0000","last_change":"2024-09-21T12:25:45.962538+0000","last_active":"2024-09-21T12:27:00.103765+0000","last_peered":"2024-09-21T12:27:00.103765+0000","last_clean":"2024-09-21T12:27:00.103765+0000","last_became_active":"2024-09-21T12:20:17.297374+0000","last_became_peered":"2024-09-21T12:20:17.297374+0000","last_unstale":"2024-09-21T12:27:00.103765+0000","last_undegraded":"2024-09-21T12:27:00.103765+0000","last_fullsized":"2024-09-21T12:27:00.103765+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-21T12:25:45.962433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:45.962433+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:08.718722+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":337,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100574+0000","last_change":"2024-09-21T12:23:59.367953+0000","last_active":"2024-09-21T12:27:00.100574+0000","last_peered":"2024-09-21T12:27:00.100574+0000","last_clean":"2024-09-21T12:27:00.100574+0000","last_became_active":"2024-09-21T12:20:15.308356+0000","last_became_peered":"2024-09-21T12:20:15.308356+0000","last_unstale":"2024-09-21T12:27:00.100574+0000","last_undegraded":"2024-09-21T12:27:00.100574+0000","last_fullsized":"2024-09-21T12:27:00.100574+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:59.367778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:23:59.367778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:28.219853+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":354,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.105248+0000","last_change":"2024-09-21T12:26:14.109964+0000","last_active":"2024-09-21T12:27:00.105248+0000","last_peered":"2024-09-21T12:27:00.105248+0000","last_clean":"2024-09-21T12:27:00.105248+0000","last_became_active":"2024-09-21T12:20:14.288358+0000","last_became_peered":"2024-09-21T12:20:14.288358+0000","last_unstale":"2024-09-21T12:27:00.105248+0000","last_undegraded":"2024-09-21T12:27:00.105248+0000","last_fullsized":"2024-09-21T12:27:00.105248+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:26:14.109886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:26:14.109886+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:31.748840+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.0019042449999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":329,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.110354+0000","last_change":"2024-09-21T12:25:46.181836+0000","last_active":"2024-09-21T12:27:00.110354+0000","last_peered":"2024-09-21T12:27:00.110354+0000","last_clean":"2024-09-21T12:27:00.110354+0000","last_became_active":"2024-09-21T12:20:19.309724+0000","last_became_peered":"2024-09-21T12:20:19.309724+0000","last_unstale":"2024-09-21T12:27:00.110354+0000","last_undegraded":"2024-09-21T12:27:00.110354+0000","last_fullsized":"2024-09-21T12:27:00.110354+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:46.181531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:46.181531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:11.368016+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":308,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100666+0000","last_change":"2024-09-21T12:20:21.593267+0000","last_active":"2024-09-21T12:27:00.100666+0000","last_peered":"2024-09-21T12:27:00.100666+0000","last_clean":"2024-09-21T12:27:00.100666+0000","last_became_active":"2024-09-21T12:20:21.592982+0000","last_became_peered":"2024-09-21T12:20:21.592982+0000","last_unstale":"2024-09-21T12:27:00.100666+0000","last_undegraded":"2024-09-21T12:27:00.100666+0000","last_fullsized":"2024-09-21T12:27:00.100666+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:24.954831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"212'11053","reported_seq":11169,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:01.868935+0000","last_change":"2024-09-21T12:24:46.326200+0000","last_active":"2024-09-21T12:27:01.868935+0000","last_peered":"2024-09-21T12:27:01.868935+0000","last_clean":"2024-09-21T12:27:01.868935+0000","last_became_active":"2024-09-21T12:24:46.325838+0000","last_became_peered":"2024-09-21T12:24:46.325838+0000","last_unstale":"2024-09-21T12:27:01.868935+0000","last_undegraded":"2024-09-21T12:27:01.868935+0000","last_fullsized":"2024-09-21T12:27:01.868935+0000","mapping_epoch":160,"log_start":"195'7059","ondisk_log_start":"195'7059","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":3994,"log_dups_size":0,"ondisk_log_size":3994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:37.170761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334168064,"num_objects":5099,"num_object_clones":0,"num_object_copies":15297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11053,"num_write_kb":516864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"76'79","reported_seq":268,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.103126+0000","last_change":"2024-09-21T12:26:45.193196+0000","last_active":"2024-09-21T12:27:00.103126+0000","last_peered":"2024-09-21T12:27:00.103126+0000","last_clean":"2024-09-21T12:27:00.103126+0000","last_became_active":"2024-09-21T12:26:44.949867+0000","last_became_peered":"2024-09-21T12:26:44.949867+0000","last_unstale":"2024-09-21T12:27:00.103126+0000","last_undegraded":"2024-09-21T12:27:00.103126+0000","last_fullsized":"2024-09-21T12:27:00.103126+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:45.193133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:45.193133+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:50.015706+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":343,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.101918+0000","last_change":"2024-09-21T12:22:49.933621+0000","last_active":"2024-09-21T12:27:00.101918+0000","last_peered":"2024-09-21T12:27:00.101918+0000","last_clean":"2024-09-21T12:27:00.101918+0000","last_became_active":"2024-09-21T12:20:17.299325+0000","last_became_peered":"2024-09-21T12:20:17.299325+0000","last_unstale":"2024-09-21T12:27:00.101918+0000","last_undegraded":"2024-09-21T12:27:00.101918+0000","last_fullsized":"2024-09-21T12:27:00.101918+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:22:49.933382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:22:49.933382+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:23.739550+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":358,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.099160+0000","last_change":"2024-09-21T12:24:48.073954+0000","last_active":"2024-09-21T12:27:00.099160+0000","last_peered":"2024-09-21T12:27:00.099160+0000","last_clean":"2024-09-21T12:27:00.099160+0000","last_became_active":"2024-09-21T12:22:50.923796+0000","last_became_peered":"2024-09-21T12:22:50.923796+0000","last_unstale":"2024-09-21T12:27:00.099160+0000","last_undegraded":"2024-09-21T12:27:00.099160+0000","last_fullsized":"2024-09-21T12:27:00.099160+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:48.073424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:48.073424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":748,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.486178+0000","last_change":"2024-09-21T12:25:35.819787+0000","last_active":"2024-09-21T12:27:00.486178+0000","last_peered":"2024-09-21T12:27:00.486178+0000","last_clean":"2024-09-21T12:27:00.486178+0000","last_became_active":"2024-09-21T12:20:14.287543+0000","last_became_peered":"2024-09-21T12:20:14.287543+0000","last_unstale":"2024-09-21T12:27:00.486178+0000","last_undegraded":"2024-09-21T12:27:00.486178+0000","last_fullsized":"2024-09-21T12:27:00.486178+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-21T12:25:35.819707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:25:35.819707+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:05.641221+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":401,"num_read_kb":396,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":328,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.102574+0000","last_change":"2024-09-21T12:26:11.422340+0000","last_active":"2024-09-21T12:27:00.102574+0000","last_peered":"2024-09-21T12:27:00.102574+0000","last_clean":"2024-09-21T12:27:00.102574+0000","last_became_active":"2024-09-21T12:20:19.314217+0000","last_became_peered":"2024-09-21T12:20:19.314217+0000","last_unstale":"2024-09-21T12:27:00.102574+0000","last_undegraded":"2024-09-21T12:27:00.102574+0000","last_fullsized":"2024-09-21T12:27:00.102574+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:11.422293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:11.422293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:13.710870+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":333,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100775+0000","last_change":"2024-09-21T12:25:38.882984+0000","last_active":"2024-09-21T12:27:00.100775+0000","last_peered":"2024-09-21T12:27:00.100775+0000","last_clean":"2024-09-21T12:27:00.100775+0000","last_became_active":"2024-09-21T12:20:21.311188+0000","last_became_peered":"2024-09-21T12:20:21.311188+0000","last_unstale":"2024-09-21T12:27:00.100775+0000","last_undegraded":"2024-09-21T12:27:00.100775+0000","last_fullsized":"2024-09-21T12:27:00.100775+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:38.882915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:38.882915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:00.415003+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"212'11392","reported_seq":11508,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:01.647486+0000","last_change":"2024-09-21T12:24:46.318053+0000","last_active":"2024-09-21T12:27:01.647486+0000","last_peered":"2024-09-21T12:27:01.647486+0000","last_clean":"2024-09-21T12:27:01.647486+0000","last_became_active":"2024-09-21T12:24:46.317644+0000","last_became_peered":"2024-09-21T12:24:46.317644+0000","last_unstale":"2024-09-21T12:27:01.647486+0000","last_undegraded":"2024-09-21T12:27:01.647486+0000","last_fullsized":"2024-09-21T12:27:01.647486+0000","mapping_epoch":160,"log_start":"194'7020","ondisk_log_start":"194'7020","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4372,"log_dups_size":0,"ondisk_log_size":4372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337248256,"num_objects":5146,"num_object_clones":0,"num_object_copies":15438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11392,"num_write_kb":529216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"121'19","reported_seq":511,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100662+0000","last_change":"2024-09-21T12:24:27.932976+0000","last_active":"2024-09-21T12:27:00.100662+0000","last_peered":"2024-09-21T12:27:00.100662+0000","last_clean":"2024-09-21T12:27:00.100662+0000","last_became_active":"2024-09-21T12:22:50.921690+0000","last_became_peered":"2024-09-21T12:22:50.921690+0000","last_unstale":"2024-09-21T12:27:00.100662+0000","last_undegraded":"2024-09-21T12:27:00.100662+0000","last_fullsized":"2024-09-21T12:27:00.100662+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'19","last_scrub_stamp":"2024-09-21T12:24:27.932891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:27.932891+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":341,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.109529+0000","last_change":"2024-09-21T12:26:12.394327+0000","last_active":"2024-09-21T12:27:00.109529+0000","last_peered":"2024-09-21T12:27:00.109529+0000","last_clean":"2024-09-21T12:27:00.109529+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:27:00.109529+0000","last_undegraded":"2024-09-21T12:27:00.109529+0000","last_fullsized":"2024-09-21T12:27:00.109529+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:26:12.394247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:12.394247+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:12.954462+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":417,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.099225+0000","last_change":"2024-09-21T12:26:49.248015+0000","last_active":"2024-09-21T12:27:00.099225+0000","last_peered":"2024-09-21T12:27:00.099225+0000","last_clean":"2024-09-21T12:27:00.099225+0000","last_became_active":"2024-09-21T12:26:44.950132+0000","last_became_peered":"2024-09-21T12:26:44.950132+0000","last_unstale":"2024-09-21T12:27:00.099225+0000","last_undegraded":"2024-09-21T12:27:00.099225+0000","last_fullsized":"2024-09-21T12:27:00.099225+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:49.247952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:49.247952+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:08.566085+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":312,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100526+0000","last_change":"2024-09-21T12:20:19.310280+0000","last_active":"2024-09-21T12:27:00.100526+0000","last_peered":"2024-09-21T12:27:00.100526+0000","last_clean":"2024-09-21T12:27:00.100526+0000","last_became_active":"2024-09-21T12:20:19.309859+0000","last_became_peered":"2024-09-21T12:20:19.309859+0000","last_unstale":"2024-09-21T12:27:00.100526+0000","last_undegraded":"2024-09-21T12:27:00.100526+0000","last_fullsized":"2024-09-21T12:27:00.100526+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:20:18.288890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:31.022254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":162,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.106356+0000","last_change":"2024-09-21T12:26:54.085280+0000","last_active":"2024-09-21T12:27:00.106356+0000","last_peered":"2024-09-21T12:27:00.106356+0000","last_clean":"2024-09-21T12:27:00.106356+0000","last_became_active":"2024-09-21T12:26:54.084914+0000","last_became_peered":"2024-09-21T12:26:54.084914+0000","last_unstale":"2024-09-21T12:27:00.106356+0000","last_undegraded":"2024-09-21T12:27:00.106356+0000","last_fullsized":"2024-09-21T12:27:00.106356+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:32.465700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:32.465700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"76'79","reported_seq":274,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.102814+0000","last_change":"2024-09-21T12:26:48.188393+0000","last_active":"2024-09-21T12:27:00.102814+0000","last_peered":"2024-09-21T12:27:00.102814+0000","last_clean":"2024-09-21T12:27:00.102814+0000","last_became_active":"2024-09-21T12:26:46.987001+0000","last_became_peered":"2024-09-21T12:26:46.987001+0000","last_unstale":"2024-09-21T12:27:00.102814+0000","last_undegraded":"2024-09-21T12:27:00.102814+0000","last_fullsized":"2024-09-21T12:27:00.102814+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:48.188322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:48.188322+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:16.613843+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"212'11153","reported_seq":11269,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:02.219575+0000","last_change":"2024-09-21T12:24:46.315286+0000","last_active":"2024-09-21T12:27:02.219575+0000","last_peered":"2024-09-21T12:27:02.219575+0000","last_clean":"2024-09-21T12:27:02.219575+0000","last_became_active":"2024-09-21T12:24:46.314957+0000","last_became_peered":"2024-09-21T12:24:46.314957+0000","last_unstale":"2024-09-21T12:27:02.219575+0000","last_undegraded":"2024-09-21T12:27:02.219575+0000","last_fullsized":"2024-09-21T12:27:02.219575+0000","mapping_epoch":160,"log_start":"195'7075","ondisk_log_start":"195'7075","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4078,"log_dups_size":0,"ondisk_log_size":4078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328007680,"num_objects":5005,"num_object_clones":0,"num_object_copies":15015,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11153,"num_write_kb":517056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":342,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100327+0000","last_change":"2024-09-21T12:26:15.952570+0000","last_active":"2024-09-21T12:27:00.100327+0000","last_peered":"2024-09-21T12:27:00.100327+0000","last_clean":"2024-09-21T12:27:00.100327+0000","last_became_active":"2024-09-21T12:23:46.885802+0000","last_became_peered":"2024-09-21T12:23:46.885802+0000","last_unstale":"2024-09-21T12:27:00.100327+0000","last_undegraded":"2024-09-21T12:27:00.100327+0000","last_fullsized":"2024-09-21T12:27:00.100327+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:26:15.952496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:15.952496+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:37.337083+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":252,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.105105+0000","last_change":"2024-09-21T12:22:50.908380+0000","last_active":"2024-09-21T12:27:00.105105+0000","last_peered":"2024-09-21T12:27:00.105105+0000","last_clean":"2024-09-21T12:27:00.105105+0000","last_became_active":"2024-09-21T12:22:50.908060+0000","last_became_peered":"2024-09-21T12:22:50.908060+0000","last_unstale":"2024-09-21T12:27:00.105105+0000","last_undegraded":"2024-09-21T12:27:00.105105+0000","last_fullsized":"2024-09-21T12:27:00.105105+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:05.892153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:05.892153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:30.212019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":166,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100864+0000","last_change":"2024-09-21T12:26:54.090144+0000","last_active":"2024-09-21T12:27:00.100864+0000","last_peered":"2024-09-21T12:27:00.100864+0000","last_clean":"2024-09-21T12:27:00.100864+0000","last_became_active":"2024-09-21T12:26:54.089839+0000","last_became_peered":"2024-09-21T12:26:54.089839+0000","last_unstale":"2024-09-21T12:27:00.100864+0000","last_undegraded":"2024-09-21T12:27:00.100864+0000","last_fullsized":"2024-09-21T12:27:00.100864+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:19.372719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:19.372719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":314,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.105634+0000","last_change":"2024-09-21T12:23:46.282792+0000","last_active":"2024-09-21T12:27:00.105634+0000","last_peered":"2024-09-21T12:27:00.105634+0000","last_clean":"2024-09-21T12:27:00.105634+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:27:00.105634+0000","last_undegraded":"2024-09-21T12:27:00.105634+0000","last_fullsized":"2024-09-21T12:27:00.105634+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:46.282661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:46.282661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:22.559196+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"212'11391","reported_seq":11507,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:01.906850+0000","last_change":"2024-09-21T12:24:46.323690+0000","last_active":"2024-09-21T12:27:01.906850+0000","last_peered":"2024-09-21T12:27:01.906850+0000","last_clean":"2024-09-21T12:27:01.906850+0000","last_became_active":"2024-09-21T12:24:46.323402+0000","last_became_peered":"2024-09-21T12:24:46.323402+0000","last_unstale":"2024-09-21T12:27:01.906850+0000","last_undegraded":"2024-09-21T12:27:01.906850+0000","last_fullsized":"2024-09-21T12:27:01.906850+0000","mapping_epoch":160,"log_start":"196'7394","ondisk_log_start":"196'7394","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":3997,"log_dups_size":0,"ondisk_log_size":3997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:01.998231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331415552,"num_objects":5057,"num_object_clones":0,"num_object_copies":15171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11391,"num_write_kb":526336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"76'79","reported_seq":29,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.104407+0000","last_change":"2024-09-21T12:26:57.680254+0000","last_active":"2024-09-21T12:27:00.104407+0000","last_peered":"2024-09-21T12:27:00.104407+0000","last_clean":"2024-09-21T12:27:00.104407+0000","last_became_active":"2024-09-21T12:26:54.107910+0000","last_became_peered":"2024-09-21T12:26:54.107910+0000","last_unstale":"2024-09-21T12:27:00.104407+0000","last_undegraded":"2024-09-21T12:27:00.104407+0000","last_fullsized":"2024-09-21T12:27:00.104407+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:57.680181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:57.680181+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:10.225908+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":377,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.108593+0000","last_change":"2024-09-21T12:25:42.183260+0000","last_active":"2024-09-21T12:27:00.108593+0000","last_peered":"2024-09-21T12:27:00.108593+0000","last_clean":"2024-09-21T12:27:00.108593+0000","last_became_active":"2024-09-21T12:23:46.897197+0000","last_became_peered":"2024-09-21T12:23:46.897197+0000","last_unstale":"2024-09-21T12:27:00.108593+0000","last_undegraded":"2024-09-21T12:27:00.108593+0000","last_fullsized":"2024-09-21T12:27:00.108593+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-21T12:25:42.183169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:42.183169+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:03.947737+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":365,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.099475+0000","last_change":"2024-09-21T12:26:13.918157+0000","last_active":"2024-09-21T12:27:00.099475+0000","last_peered":"2024-09-21T12:27:00.099475+0000","last_clean":"2024-09-21T12:27:00.099475+0000","last_became_active":"2024-09-21T12:22:50.906886+0000","last_became_peered":"2024-09-21T12:22:50.906886+0000","last_unstale":"2024-09-21T12:27:00.099475+0000","last_undegraded":"2024-09-21T12:27:00.099475+0000","last_fullsized":"2024-09-21T12:27:00.099475+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:13.918088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:13.918088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:43.184195+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":341,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.102060+0000","last_change":"2024-09-21T12:23:17.434377+0000","last_active":"2024-09-21T12:27:00.102060+0000","last_peered":"2024-09-21T12:27:00.102060+0000","last_clean":"2024-09-21T12:27:00.102060+0000","last_became_active":"2024-09-21T12:22:50.927438+0000","last_became_peered":"2024-09-21T12:22:50.927438+0000","last_unstale":"2024-09-21T12:27:00.102060+0000","last_undegraded":"2024-09-21T12:27:00.102060+0000","last_fullsized":"2024-09-21T12:27:00.102060+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:17.434261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:17.434261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":337,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.106162+0000","last_change":"2024-09-21T12:25:39.079146+0000","last_active":"2024-09-21T12:27:00.106162+0000","last_peered":"2024-09-21T12:27:00.106162+0000","last_clean":"2024-09-21T12:27:00.106162+0000","last_became_active":"2024-09-21T12:22:50.913898+0000","last_became_peered":"2024-09-21T12:22:50.913898+0000","last_unstale":"2024-09-21T12:27:00.106162+0000","last_undegraded":"2024-09-21T12:27:00.106162+0000","last_fullsized":"2024-09-21T12:27:00.106162+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:39.079015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:39.079015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"212'11316","reported_seq":11432,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:02.181278+0000","last_change":"2024-09-21T12:24:46.315895+0000","last_active":"2024-09-21T12:27:02.181278+0000","last_peered":"2024-09-21T12:27:02.181278+0000","last_clean":"2024-09-21T12:27:02.181278+0000","last_became_active":"2024-09-21T12:24:46.315601+0000","last_became_peered":"2024-09-21T12:24:46.315601+0000","last_unstale":"2024-09-21T12:27:02.181278+0000","last_undegraded":"2024-09-21T12:27:02.181278+0000","last_fullsized":"2024-09-21T12:27:02.181278+0000","mapping_epoch":160,"log_start":"196'7396","ondisk_log_start":"196'7396","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":3920,"log_dups_size":0,"ondisk_log_size":3920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333447168,"num_objects":5088,"num_object_clones":0,"num_object_copies":15264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11316,"num_write_kb":524928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"76'79","reported_seq":393,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.098999+0000","last_change":"2024-09-21T12:25:43.825573+0000","last_active":"2024-09-21T12:27:00.098999+0000","last_peered":"2024-09-21T12:27:00.098999+0000","last_clean":"2024-09-21T12:27:00.098999+0000","last_became_active":"2024-09-21T12:22:17.072872+0000","last_became_peered":"2024-09-21T12:22:17.072872+0000","last_unstale":"2024-09-21T12:27:00.098999+0000","last_undegraded":"2024-09-21T12:27:00.098999+0000","last_fullsized":"2024-09-21T12:27:00.098999+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:25:43.825511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:25:43.825511+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:05.500022+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":191,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.107199+0000","last_change":"2024-09-21T12:26:10.141012+0000","last_active":"2024-09-21T12:27:00.107199+0000","last_peered":"2024-09-21T12:27:00.107199+0000","last_clean":"2024-09-21T12:27:00.107199+0000","last_became_active":"2024-09-21T12:23:46.902260+0000","last_became_peered":"2024-09-21T12:23:46.902260+0000","last_unstale":"2024-09-21T12:27:00.107199+0000","last_undegraded":"2024-09-21T12:27:00.107199+0000","last_fullsized":"2024-09-21T12:27:00.107199+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:26:10.137029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:10.137029+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:19.970928+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":348,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.106279+0000","last_change":"2024-09-21T12:22:50.928240+0000","last_active":"2024-09-21T12:27:00.106279+0000","last_peered":"2024-09-21T12:27:00.106279+0000","last_clean":"2024-09-21T12:27:00.106279+0000","last_became_active":"2024-09-21T12:22:50.927946+0000","last_became_peered":"2024-09-21T12:22:50.927946+0000","last_unstale":"2024-09-21T12:27:00.106279+0000","last_undegraded":"2024-09-21T12:27:00.106279+0000","last_fullsized":"2024-09-21T12:27:00.106279+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:22:08.999015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:22:08.999015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:24.553076+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":426,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.108547+0000","last_change":"2024-09-21T12:26:13.114174+0000","last_active":"2024-09-21T12:27:00.108547+0000","last_peered":"2024-09-21T12:27:00.108547+0000","last_clean":"2024-09-21T12:27:00.108547+0000","last_became_active":"2024-09-21T12:20:19.307924+0000","last_became_peered":"2024-09-21T12:20:19.307924+0000","last_unstale":"2024-09-21T12:27:00.108547+0000","last_undegraded":"2024-09-21T12:27:00.108547+0000","last_fullsized":"2024-09-21T12:27:00.108547+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:26:13.113863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:13.113863+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:23.394893+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":260,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.101625+0000","last_change":"2024-09-21T12:23:56.471960+0000","last_active":"2024-09-21T12:27:00.101625+0000","last_peered":"2024-09-21T12:27:00.101625+0000","last_clean":"2024-09-21T12:27:00.101625+0000","last_became_active":"2024-09-21T12:22:50.905470+0000","last_became_peered":"2024-09-21T12:22:50.905470+0000","last_unstale":"2024-09-21T12:27:00.101625+0000","last_undegraded":"2024-09-21T12:27:00.101625+0000","last_fullsized":"2024-09-21T12:27:00.101625+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:56.471116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:56.471116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:29.258665+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"76'79","reported_seq":32,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100407+0000","last_change":"2024-09-21T12:26:53.217824+0000","last_active":"2024-09-21T12:27:00.100407+0000","last_peered":"2024-09-21T12:27:00.100407+0000","last_clean":"2024-09-21T12:27:00.100407+0000","last_became_active":"2024-09-21T12:26:46.997281+0000","last_became_peered":"2024-09-21T12:26:46.997281+0000","last_unstale":"2024-09-21T12:27:00.100407+0000","last_undegraded":"2024-09-21T12:27:00.100407+0000","last_fullsized":"2024-09-21T12:27:00.100407+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:53.217767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:53.217767+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:11.859435+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"212'11220","reported_seq":11336,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:01.878673+0000","last_change":"2024-09-21T12:24:46.317724+0000","last_active":"2024-09-21T12:27:01.878673+0000","last_peered":"2024-09-21T12:27:01.878673+0000","last_clean":"2024-09-21T12:27:01.878673+0000","last_became_active":"2024-09-21T12:24:46.317468+0000","last_became_peered":"2024-09-21T12:24:46.317468+0000","last_unstale":"2024-09-21T12:27:01.878673+0000","last_undegraded":"2024-09-21T12:27:01.878673+0000","last_fullsized":"2024-09-21T12:27:01.878673+0000","mapping_epoch":160,"log_start":"195'7189","ondisk_log_start":"195'7189","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4031,"log_dups_size":0,"ondisk_log_size":4031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328728576,"num_objects":5016,"num_object_clones":0,"num_object_copies":15048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11220,"num_write_kb":519552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":408,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.099861+0000","last_change":"2024-09-21T12:25:00.237083+0000","last_active":"2024-09-21T12:27:00.099861+0000","last_peered":"2024-09-21T12:27:00.099861+0000","last_clean":"2024-09-21T12:27:00.099861+0000","last_became_active":"2024-09-21T12:22:50.922343+0000","last_became_peered":"2024-09-21T12:22:50.922343+0000","last_unstale":"2024-09-21T12:27:00.099861+0000","last_undegraded":"2024-09-21T12:27:00.099861+0000","last_fullsized":"2024-09-21T12:27:00.099861+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:25:00.236990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:00.236990+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:38.757954+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":19,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.105904+0000","last_change":"2024-09-21T12:26:54.084563+0000","last_active":"2024-09-21T12:27:00.105904+0000","last_peered":"2024-09-21T12:27:00.105904+0000","last_clean":"2024-09-21T12:27:00.105904+0000","last_became_active":"2024-09-21T12:26:54.084228+0000","last_became_peered":"2024-09-21T12:26:54.084228+0000","last_unstale":"2024-09-21T12:27:00.105904+0000","last_undegraded":"2024-09-21T12:27:00.105904+0000","last_fullsized":"2024-09-21T12:27:00.105904+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:30.451876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:30.451876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":358,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.105771+0000","last_change":"2024-09-21T12:25:36.012066+0000","last_active":"2024-09-21T12:27:00.105771+0000","last_peered":"2024-09-21T12:27:00.105771+0000","last_clean":"2024-09-21T12:27:00.105771+0000","last_became_active":"2024-09-21T12:22:50.925974+0000","last_became_peered":"2024-09-21T12:22:50.925974+0000","last_unstale":"2024-09-21T12:27:00.105771+0000","last_undegraded":"2024-09-21T12:27:00.105771+0000","last_fullsized":"2024-09-21T12:27:00.105771+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:36.011984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:36.011984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:42.463265+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":317,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.099070+0000","last_change":"2024-09-21T12:23:28.121344+0000","last_active":"2024-09-21T12:27:00.099070+0000","last_peered":"2024-09-21T12:27:00.099070+0000","last_clean":"2024-09-21T12:27:00.099070+0000","last_became_active":"2024-09-21T12:20:21.311493+0000","last_became_peered":"2024-09-21T12:20:21.311493+0000","last_unstale":"2024-09-21T12:27:00.099070+0000","last_undegraded":"2024-09-21T12:27:00.099070+0000","last_fullsized":"2024-09-21T12:27:00.099070+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:28.121286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:28.121286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:29.750995+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"76'79","reported_seq":350,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100977+0000","last_change":"2024-09-21T12:26:55.832199+0000","last_active":"2024-09-21T12:27:00.100977+0000","last_peered":"2024-09-21T12:27:00.100977+0000","last_clean":"2024-09-21T12:27:00.100977+0000","last_became_active":"2024-09-21T12:26:46.986203+0000","last_became_peered":"2024-09-21T12:26:46.986203+0000","last_unstale":"2024-09-21T12:27:00.100977+0000","last_undegraded":"2024-09-21T12:27:00.100977+0000","last_fullsized":"2024-09-21T12:27:00.100977+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:55.832132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:55.832132+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:05.661004+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.6","version":"212'11285","reported_seq":11401,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:02.248944+0000","last_change":"2024-09-21T12:24:46.327091+0000","last_active":"2024-09-21T12:27:02.248944+0000","last_peered":"2024-09-21T12:27:02.248944+0000","last_clean":"2024-09-21T12:27:02.248944+0000","last_became_active":"2024-09-21T12:24:46.325838+0000","last_became_peered":"2024-09-21T12:24:46.325838+0000","last_unstale":"2024-09-21T12:27:02.248944+0000","last_undegraded":"2024-09-21T12:27:02.248944+0000","last_fullsized":"2024-09-21T12:27:02.248944+0000","mapping_epoch":160,"log_start":"195'7192","ondisk_log_start":"195'7192","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4093,"log_dups_size":0,"ondisk_log_size":4093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:34.761851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330366976,"num_objects":5041,"num_object_clones":0,"num_object_copies":15123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11285,"num_write_kb":522432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":363,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.099348+0000","last_change":"2024-09-21T12:24:02.960807+0000","last_active":"2024-09-21T12:27:00.099348+0000","last_peered":"2024-09-21T12:27:00.099348+0000","last_clean":"2024-09-21T12:27:00.099348+0000","last_became_active":"2024-09-21T12:20:17.692867+0000","last_became_peered":"2024-09-21T12:20:17.692867+0000","last_unstale":"2024-09-21T12:27:00.099348+0000","last_undegraded":"2024-09-21T12:27:00.099348+0000","last_fullsized":"2024-09-21T12:27:00.099348+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-21T12:24:02.960733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:02.960733+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:29.944995+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":327,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100985+0000","last_change":"2024-09-21T12:23:59.046000+0000","last_active":"2024-09-21T12:27:00.100985+0000","last_peered":"2024-09-21T12:27:00.100985+0000","last_clean":"2024-09-21T12:27:00.100985+0000","last_became_active":"2024-09-21T12:20:15.299333+0000","last_became_peered":"2024-09-21T12:20:15.299333+0000","last_unstale":"2024-09-21T12:27:00.100985+0000","last_undegraded":"2024-09-21T12:27:00.100985+0000","last_fullsized":"2024-09-21T12:27:00.100985+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:59.045927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:23:59.045927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":328,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.105664+0000","last_change":"2024-09-21T12:26:50.626932+0000","last_active":"2024-09-21T12:27:00.105664+0000","last_peered":"2024-09-21T12:27:00.105664+0000","last_clean":"2024-09-21T12:27:00.105664+0000","last_became_active":"2024-09-21T12:20:19.306952+0000","last_became_peered":"2024-09-21T12:20:19.306952+0000","last_unstale":"2024-09-21T12:27:00.105664+0000","last_undegraded":"2024-09-21T12:27:00.105664+0000","last_fullsized":"2024-09-21T12:27:00.105664+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:50.626780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:50.626780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:07.790008+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":308,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.099142+0000","last_change":"2024-09-21T12:20:21.314891+0000","last_active":"2024-09-21T12:27:00.099142+0000","last_peered":"2024-09-21T12:27:00.099142+0000","last_clean":"2024-09-21T12:27:00.099142+0000","last_became_active":"2024-09-21T12:20:21.314545+0000","last_became_peered":"2024-09-21T12:20:21.314545+0000","last_unstale":"2024-09-21T12:27:00.099142+0000","last_undegraded":"2024-09-21T12:27:00.099142+0000","last_fullsized":"2024-09-21T12:27:00.099142+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"76'79","reported_seq":271,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.104460+0000","last_change":"2024-09-21T12:26:49.368792+0000","last_active":"2024-09-21T12:27:00.104460+0000","last_peered":"2024-09-21T12:27:00.104460+0000","last_clean":"2024-09-21T12:27:00.104460+0000","last_became_active":"2024-09-21T12:26:46.987201+0000","last_became_peered":"2024-09-21T12:26:46.987201+0000","last_unstale":"2024-09-21T12:27:00.104460+0000","last_undegraded":"2024-09-21T12:27:00.104460+0000","last_fullsized":"2024-09-21T12:27:00.104460+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:49.368725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:49.368725+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:09.993614+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.5","version":"212'11176","reported_seq":11304,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:01.655631+0000","last_change":"2024-09-21T12:25:49.210492+0000","last_active":"2024-09-21T12:27:01.655631+0000","last_peered":"2024-09-21T12:27:01.655631+0000","last_clean":"2024-09-21T12:27:01.655631+0000","last_became_active":"2024-09-21T12:24:46.324554+0000","last_became_peered":"2024-09-21T12:24:46.324554+0000","last_unstale":"2024-09-21T12:27:01.655631+0000","last_undegraded":"2024-09-21T12:27:01.655631+0000","last_fullsized":"2024-09-21T12:27:01.655631+0000","mapping_epoch":160,"log_start":"193'6766","ondisk_log_start":"193'6766","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":899,"log_size":4410,"log_dups_size":0,"ondisk_log_size":4410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:34.919444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334888960,"num_objects":5110,"num_object_clones":0,"num_object_copies":15330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11176,"num_write_kb":521152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":335,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100221+0000","last_change":"2024-09-21T12:24:54.445801+0000","last_active":"2024-09-21T12:27:00.100221+0000","last_peered":"2024-09-21T12:27:00.100221+0000","last_clean":"2024-09-21T12:27:00.100221+0000","last_became_active":"2024-09-21T12:22:50.910973+0000","last_became_peered":"2024-09-21T12:22:50.910973+0000","last_unstale":"2024-09-21T12:27:00.100221+0000","last_undegraded":"2024-09-21T12:27:00.100221+0000","last_fullsized":"2024-09-21T12:27:00.100221+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:24:54.445714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:54.445714+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:38.168573+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":356,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100923+0000","last_change":"2024-09-21T12:25:44.800799+0000","last_active":"2024-09-21T12:27:00.100923+0000","last_peered":"2024-09-21T12:27:00.100923+0000","last_clean":"2024-09-21T12:27:00.100923+0000","last_became_active":"2024-09-21T12:22:50.927859+0000","last_became_peered":"2024-09-21T12:22:50.927859+0000","last_unstale":"2024-09-21T12:27:00.100923+0000","last_undegraded":"2024-09-21T12:27:00.100923+0000","last_fullsized":"2024-09-21T12:27:00.100923+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:44.800657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:44.800657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":329,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.105060+0000","last_change":"2024-09-21T12:25:40.148844+0000","last_active":"2024-09-21T12:27:00.105060+0000","last_peered":"2024-09-21T12:27:00.105060+0000","last_clean":"2024-09-21T12:27:00.105060+0000","last_became_active":"2024-09-21T12:20:19.312572+0000","last_became_peered":"2024-09-21T12:20:19.312572+0000","last_unstale":"2024-09-21T12:27:00.105060+0000","last_undegraded":"2024-09-21T12:27:00.105060+0000","last_fullsized":"2024-09-21T12:27:00.105060+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:40.148766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:40.148766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:07.648989+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":322,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.103716+0000","last_change":"2024-09-21T12:26:58.321950+0000","last_active":"2024-09-21T12:27:00.103716+0000","last_peered":"2024-09-21T12:27:00.103716+0000","last_clean":"2024-09-21T12:27:00.103716+0000","last_became_active":"2024-09-21T12:20:21.317223+0000","last_became_peered":"2024-09-21T12:20:21.317223+0000","last_unstale":"2024-09-21T12:27:00.103716+0000","last_undegraded":"2024-09-21T12:27:00.103716+0000","last_fullsized":"2024-09-21T12:27:00.103716+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:58.321890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:58.321890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:13.356328+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"76'79","reported_seq":32,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100313+0000","last_change":"2024-09-21T12:26:56.264813+0000","last_active":"2024-09-21T12:27:00.100313+0000","last_peered":"2024-09-21T12:27:00.100313+0000","last_clean":"2024-09-21T12:27:00.100313+0000","last_became_active":"2024-09-21T12:26:46.997561+0000","last_became_peered":"2024-09-21T12:26:46.997561+0000","last_unstale":"2024-09-21T12:27:00.100313+0000","last_undegraded":"2024-09-21T12:27:00.100313+0000","last_fullsized":"2024-09-21T12:27:00.100313+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:56.264746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:56.264746+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:12.555807+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.4","version":"212'11352","reported_seq":11472,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:01.892081+0000","last_change":"2024-09-21T12:26:54.109615+0000","last_active":"2024-09-21T12:27:01.892081+0000","last_peered":"2024-09-21T12:27:01.892081+0000","last_clean":"2024-09-21T12:27:01.892081+0000","last_became_active":"2024-09-21T12:26:54.090950+0000","last_became_peered":"2024-09-21T12:26:54.090950+0000","last_unstale":"2024-09-21T12:27:01.892081+0000","last_undegraded":"2024-09-21T12:27:01.892081+0000","last_fullsized":"2024-09-21T12:27:01.892081+0000","mapping_epoch":208,"log_start":"191'6530","ondisk_log_start":"191'6530","created":160,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:24:45.289239+0000","last_clean_scrub_stamp":"2024-09-21T12:24:45.289239+0000","objects_scrubbed":0,"log_size":4822,"log_dups_size":0,"ondisk_log_size":4822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333709312,"num_objects":5092,"num_object_clones":0,"num_object_copies":15276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11352,"num_write_kb":526208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":365,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.103095+0000","last_change":"2024-09-21T12:25:03.416100+0000","last_active":"2024-09-21T12:27:00.103095+0000","last_peered":"2024-09-21T12:27:00.103095+0000","last_clean":"2024-09-21T12:27:00.103095+0000","last_became_active":"2024-09-21T12:20:17.692870+0000","last_became_peered":"2024-09-21T12:20:17.692870+0000","last_unstale":"2024-09-21T12:27:00.103095+0000","last_undegraded":"2024-09-21T12:27:00.103095+0000","last_fullsized":"2024-09-21T12:27:00.103095+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:25:03.416033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:03.416033+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:42.448728+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":386,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100793+0000","last_change":"2024-09-21T12:24:29.054851+0000","last_active":"2024-09-21T12:27:00.100793+0000","last_peered":"2024-09-21T12:27:00.100793+0000","last_clean":"2024-09-21T12:27:00.100793+0000","last_became_active":"2024-09-21T12:22:50.927141+0000","last_became_peered":"2024-09-21T12:22:50.927141+0000","last_unstale":"2024-09-21T12:27:00.100793+0000","last_undegraded":"2024-09-21T12:27:00.100793+0000","last_fullsized":"2024-09-21T12:27:00.100793+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-21T12:24:29.054772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:29.054772+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:07.827845+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"121'11","reported_seq":443,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.099028+0000","last_change":"2024-09-21T12:25:02.916145+0000","last_active":"2024-09-21T12:27:00.099028+0000","last_peered":"2024-09-21T12:27:00.099028+0000","last_clean":"2024-09-21T12:27:00.099028+0000","last_became_active":"2024-09-21T12:22:50.908371+0000","last_became_peered":"2024-09-21T12:22:50.908371+0000","last_unstale":"2024-09-21T12:27:00.099028+0000","last_undegraded":"2024-09-21T12:27:00.099028+0000","last_fullsized":"2024-09-21T12:27:00.099028+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:25:02.916090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:02.916090+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":27,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.099257+0000","last_change":"2024-09-21T12:26:56.862895+0000","last_active":"2024-09-21T12:27:00.099257+0000","last_peered":"2024-09-21T12:27:00.099257+0000","last_clean":"2024-09-21T12:27:00.099257+0000","last_became_active":"2024-09-21T12:26:54.089155+0000","last_became_peered":"2024-09-21T12:26:54.089155+0000","last_unstale":"2024-09-21T12:27:00.099257+0000","last_undegraded":"2024-09-21T12:27:00.099257+0000","last_fullsized":"2024-09-21T12:27:00.099257+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:56.862823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:56.862823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:14.918908+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"76'79","reported_seq":32,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.101971+0000","last_change":"2024-09-21T12:26:47.453710+0000","last_active":"2024-09-21T12:27:00.101971+0000","last_peered":"2024-09-21T12:27:00.101971+0000","last_clean":"2024-09-21T12:27:00.101971+0000","last_became_active":"2024-09-21T12:26:46.971481+0000","last_became_peered":"2024-09-21T12:26:46.971481+0000","last_unstale":"2024-09-21T12:27:00.101971+0000","last_undegraded":"2024-09-21T12:27:00.101971+0000","last_fullsized":"2024-09-21T12:27:00.101971+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:47.453650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:47.453650+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:09.381691+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":351,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.099073+0000","last_change":"2024-09-21T12:25:31.845118+0000","last_active":"2024-09-21T12:27:00.099073+0000","last_peered":"2024-09-21T12:27:00.099073+0000","last_clean":"2024-09-21T12:27:00.099073+0000","last_became_active":"2024-09-21T12:20:15.303480+0000","last_became_peered":"2024-09-21T12:20:15.303480+0000","last_unstale":"2024-09-21T12:27:00.099073+0000","last_undegraded":"2024-09-21T12:27:00.099073+0000","last_fullsized":"2024-09-21T12:27:00.099073+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:31.845066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:31.845066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":339,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.106674+0000","last_change":"2024-09-21T12:26:57.369721+0000","last_active":"2024-09-21T12:27:00.106674+0000","last_peered":"2024-09-21T12:27:00.106674+0000","last_clean":"2024-09-21T12:27:00.106674+0000","last_became_active":"2024-09-21T12:20:17.694628+0000","last_became_peered":"2024-09-21T12:20:17.694628+0000","last_unstale":"2024-09-21T12:27:00.106674+0000","last_undegraded":"2024-09-21T12:27:00.106674+0000","last_fullsized":"2024-09-21T12:27:00.106674+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-21T12:26:57.369648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:57.369648+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:14.175550+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":20,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.106822+0000","last_change":"2024-09-21T12:26:54.084503+0000","last_active":"2024-09-21T12:27:00.106822+0000","last_peered":"2024-09-21T12:27:00.106822+0000","last_clean":"2024-09-21T12:27:00.106822+0000","last_became_active":"2024-09-21T12:26:54.084109+0000","last_became_peered":"2024-09-21T12:26:54.084109+0000","last_unstale":"2024-09-21T12:27:00.106822+0000","last_undegraded":"2024-09-21T12:27:00.106822+0000","last_fullsized":"2024-09-21T12:27:00.106822+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:02.414298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:02.414298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":336,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100574+0000","last_change":"2024-09-21T12:24:00.066522+0000","last_active":"2024-09-21T12:27:00.100574+0000","last_peered":"2024-09-21T12:27:00.100574+0000","last_clean":"2024-09-21T12:27:00.100574+0000","last_became_active":"2024-09-21T12:22:50.904404+0000","last_became_peered":"2024-09-21T12:22:50.904404+0000","last_unstale":"2024-09-21T12:27:00.100574+0000","last_undegraded":"2024-09-21T12:27:00.100574+0000","last_fullsized":"2024-09-21T12:27:00.100574+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:00.066451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:00.066451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:01.771403+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'79","reported_seq":274,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.102965+0000","last_change":"2024-09-21T12:26:55.162518+0000","last_active":"2024-09-21T12:27:00.102965+0000","last_peered":"2024-09-21T12:27:00.102965+0000","last_clean":"2024-09-21T12:27:00.102965+0000","last_became_active":"2024-09-21T12:26:46.994966+0000","last_became_peered":"2024-09-21T12:26:46.994966+0000","last_unstale":"2024-09-21T12:27:00.102965+0000","last_undegraded":"2024-09-21T12:27:00.102965+0000","last_fullsized":"2024-09-21T12:27:00.102965+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:55.162456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:55.162456+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:06.697021+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":348,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.102011+0000","last_change":"2024-09-21T12:26:20.767526+0000","last_active":"2024-09-21T12:27:00.102011+0000","last_peered":"2024-09-21T12:27:00.102011+0000","last_clean":"2024-09-21T12:27:00.102011+0000","last_became_active":"2024-09-21T12:20:17.695898+0000","last_became_peered":"2024-09-21T12:20:17.695898+0000","last_unstale":"2024-09-21T12:27:00.102011+0000","last_undegraded":"2024-09-21T12:27:00.102011+0000","last_fullsized":"2024-09-21T12:27:00.102011+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:26:20.767293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:20.767293+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:26.917942+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":364,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.099141+0000","last_change":"2024-09-21T12:26:16.907351+0000","last_active":"2024-09-21T12:27:00.099141+0000","last_peered":"2024-09-21T12:27:00.099141+0000","last_clean":"2024-09-21T12:27:00.099141+0000","last_became_active":"2024-09-21T12:22:50.927649+0000","last_became_peered":"2024-09-21T12:22:50.927649+0000","last_unstale":"2024-09-21T12:27:00.099141+0000","last_undegraded":"2024-09-21T12:27:00.099141+0000","last_fullsized":"2024-09-21T12:27:00.099141+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:16.907284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:16.907284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:30.534090+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":318,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.106477+0000","last_change":"2024-09-21T12:23:53.356954+0000","last_active":"2024-09-21T12:27:00.106477+0000","last_peered":"2024-09-21T12:27:00.106477+0000","last_clean":"2024-09-21T12:27:00.106477+0000","last_became_active":"2024-09-21T12:20:19.311947+0000","last_became_peered":"2024-09-21T12:20:19.311947+0000","last_unstale":"2024-09-21T12:27:00.106477+0000","last_undegraded":"2024-09-21T12:27:00.106477+0000","last_fullsized":"2024-09-21T12:27:00.106477+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:53.356896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:53.356896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:40.344669+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":361,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.107344+0000","last_change":"2024-09-21T12:26:11.105716+0000","last_active":"2024-09-21T12:27:00.107344+0000","last_peered":"2024-09-21T12:27:00.107344+0000","last_clean":"2024-09-21T12:27:00.107344+0000","last_became_active":"2024-09-21T12:22:50.924931+0000","last_became_peered":"2024-09-21T12:22:50.924931+0000","last_unstale":"2024-09-21T12:27:00.107344+0000","last_undegraded":"2024-09-21T12:27:00.107344+0000","last_fullsized":"2024-09-21T12:27:00.107344+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:11.105583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:11.105583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:36.449727+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"76'79","reported_seq":409,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.098720+0000","last_change":"2024-09-21T12:26:54.085274+0000","last_active":"2024-09-21T12:27:00.098720+0000","last_peered":"2024-09-21T12:27:00.098720+0000","last_clean":"2024-09-21T12:27:00.098720+0000","last_became_active":"2024-09-21T12:26:54.084939+0000","last_became_peered":"2024-09-21T12:26:54.084939+0000","last_unstale":"2024-09-21T12:27:00.098720+0000","last_undegraded":"2024-09-21T12:27:00.098720+0000","last_fullsized":"2024-09-21T12:27:00.098720+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:50.443344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:50.443344+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":55,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100822+0000","last_change":"2024-09-21T12:26:54.102326+0000","last_active":"2024-09-21T12:27:00.100822+0000","last_peered":"2024-09-21T12:27:00.100822+0000","last_clean":"2024-09-21T12:27:00.100822+0000","last_became_active":"2024-09-21T12:26:54.101937+0000","last_became_peered":"2024-09-21T12:26:54.101937+0000","last_unstale":"2024-09-21T12:27:00.100822+0000","last_undegraded":"2024-09-21T12:27:00.100822+0000","last_fullsized":"2024-09-21T12:27:00.100822+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:26:18.480630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:18.480630+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:23.128415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":336,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.099172+0000","last_change":"2024-09-21T12:24:25.931617+0000","last_active":"2024-09-21T12:27:00.099172+0000","last_peered":"2024-09-21T12:27:00.099172+0000","last_clean":"2024-09-21T12:27:00.099172+0000","last_became_active":"2024-09-21T12:20:15.297519+0000","last_became_peered":"2024-09-21T12:20:15.297519+0000","last_unstale":"2024-09-21T12:27:00.099172+0000","last_undegraded":"2024-09-21T12:27:00.099172+0000","last_fullsized":"2024-09-21T12:27:00.099172+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:25.931561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:25.931561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":408,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.104263+0000","last_change":"2024-09-21T12:23:57.379033+0000","last_active":"2024-09-21T12:27:00.104263+0000","last_peered":"2024-09-21T12:27:00.104263+0000","last_clean":"2024-09-21T12:27:00.104263+0000","last_became_active":"2024-09-21T12:20:19.305364+0000","last_became_peered":"2024-09-21T12:20:19.305364+0000","last_unstale":"2024-09-21T12:27:00.104263+0000","last_undegraded":"2024-09-21T12:27:00.104263+0000","last_fullsized":"2024-09-21T12:27:00.104263+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:23:57.378959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:57.378959+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:29.503205+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":260,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.106245+0000","last_change":"2024-09-21T12:24:51.831043+0000","last_active":"2024-09-21T12:27:00.106245+0000","last_peered":"2024-09-21T12:27:00.106245+0000","last_clean":"2024-09-21T12:27:00.106245+0000","last_became_active":"2024-09-21T12:22:50.924770+0000","last_became_peered":"2024-09-21T12:22:50.924770+0000","last_unstale":"2024-09-21T12:27:00.106245+0000","last_undegraded":"2024-09-21T12:27:00.106245+0000","last_fullsized":"2024-09-21T12:27:00.106245+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:51.830994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:51.830994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"76'79","reported_seq":122,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.098892+0000","last_change":"2024-09-21T12:26:48.804298+0000","last_active":"2024-09-21T12:27:00.098892+0000","last_peered":"2024-09-21T12:27:00.098892+0000","last_clean":"2024-09-21T12:27:00.098892+0000","last_became_active":"2024-09-21T12:26:46.993017+0000","last_became_peered":"2024-09-21T12:26:46.993017+0000","last_unstale":"2024-09-21T12:27:00.098892+0000","last_undegraded":"2024-09-21T12:27:00.098892+0000","last_fullsized":"2024-09-21T12:27:00.098892+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:48.804235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:48.804235+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:02.149320+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":345,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.105777+0000","last_change":"2024-09-21T12:24:50.727161+0000","last_active":"2024-09-21T12:27:00.105777+0000","last_peered":"2024-09-21T12:27:00.105777+0000","last_clean":"2024-09-21T12:27:00.105777+0000","last_became_active":"2024-09-21T12:20:17.693209+0000","last_became_peered":"2024-09-21T12:20:17.693209+0000","last_unstale":"2024-09-21T12:27:00.105777+0000","last_undegraded":"2024-09-21T12:27:00.105777+0000","last_fullsized":"2024-09-21T12:27:00.105777+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:24:50.727101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:50.727101+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":356,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.099493+0000","last_change":"2024-09-21T12:25:36.808211+0000","last_active":"2024-09-21T12:27:00.099493+0000","last_peered":"2024-09-21T12:27:00.099493+0000","last_clean":"2024-09-21T12:27:00.099493+0000","last_became_active":"2024-09-21T12:22:50.908238+0000","last_became_peered":"2024-09-21T12:22:50.908238+0000","last_unstale":"2024-09-21T12:27:00.099493+0000","last_undegraded":"2024-09-21T12:27:00.099493+0000","last_fullsized":"2024-09-21T12:27:00.099493+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:36.808147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:36.808147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":409,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.102698+0000","last_change":"2024-09-21T12:22:46.216242+0000","last_active":"2024-09-21T12:27:00.102698+0000","last_peered":"2024-09-21T12:27:00.102698+0000","last_clean":"2024-09-21T12:27:00.102698+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:27:00.102698+0000","last_undegraded":"2024-09-21T12:27:00.102698+0000","last_fullsized":"2024-09-21T12:27:00.102698+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:22:46.216153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:22:46.216153+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:27.470181+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":316,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.101413+0000","last_change":"2024-09-21T12:26:08.926904+0000","last_active":"2024-09-21T12:27:00.101413+0000","last_peered":"2024-09-21T12:27:00.101413+0000","last_clean":"2024-09-21T12:27:00.101413+0000","last_became_active":"2024-09-21T12:20:21.309385+0000","last_became_peered":"2024-09-21T12:20:21.309385+0000","last_unstale":"2024-09-21T12:27:00.101413+0000","last_undegraded":"2024-09-21T12:27:00.101413+0000","last_fullsized":"2024-09-21T12:27:00.101413+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:08.923458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:08.923458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:38.391592+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":315,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.099948+0000","last_change":"2024-09-21T12:23:59.046758+0000","last_active":"2024-09-21T12:27:00.099948+0000","last_peered":"2024-09-21T12:27:00.099948+0000","last_clean":"2024-09-21T12:27:00.099948+0000","last_became_active":"2024-09-21T12:22:50.906473+0000","last_became_peered":"2024-09-21T12:22:50.906473+0000","last_unstale":"2024-09-21T12:27:00.099948+0000","last_undegraded":"2024-09-21T12:27:00.099948+0000","last_fullsized":"2024-09-21T12:27:00.099948+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:23:59.046704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:59.046704+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:25.009336+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":252,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100647+0000","last_change":"2024-09-21T12:24:58.948698+0000","last_active":"2024-09-21T12:27:00.100647+0000","last_peered":"2024-09-21T12:27:00.100647+0000","last_clean":"2024-09-21T12:27:00.100647+0000","last_became_active":"2024-09-21T12:22:50.903705+0000","last_became_peered":"2024-09-21T12:22:50.903705+0000","last_unstale":"2024-09-21T12:27:00.100647+0000","last_undegraded":"2024-09-21T12:27:00.100647+0000","last_fullsized":"2024-09-21T12:27:00.100647+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:58.948625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:58.948625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:37.071940+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":323,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.103793+0000","last_change":"2024-09-21T12:26:16.149823+0000","last_active":"2024-09-21T12:27:00.103793+0000","last_peered":"2024-09-21T12:27:00.103793+0000","last_clean":"2024-09-21T12:27:00.103793+0000","last_became_active":"2024-09-21T12:20:19.315418+0000","last_became_peered":"2024-09-21T12:20:19.315418+0000","last_unstale":"2024-09-21T12:27:00.103793+0000","last_undegraded":"2024-09-21T12:27:00.103793+0000","last_fullsized":"2024-09-21T12:27:00.103793+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:16.149550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:16.149550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:27.681950+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":322,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.102520+0000","last_change":"2024-09-21T12:24:56.879617+0000","last_active":"2024-09-21T12:27:00.102520+0000","last_peered":"2024-09-21T12:27:00.102520+0000","last_clean":"2024-09-21T12:27:00.102520+0000","last_became_active":"2024-09-21T12:20:21.310053+0000","last_became_peered":"2024-09-21T12:20:21.310053+0000","last_unstale":"2024-09-21T12:27:00.102520+0000","last_undegraded":"2024-09-21T12:27:00.102520+0000","last_fullsized":"2024-09-21T12:27:00.102520+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:56.879558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:56.879558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":340,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.104087+0000","last_change":"2024-09-21T12:25:41.888249+0000","last_active":"2024-09-21T12:27:00.104087+0000","last_peered":"2024-09-21T12:27:00.104087+0000","last_clean":"2024-09-21T12:27:00.104087+0000","last_became_active":"2024-09-21T12:20:17.299015+0000","last_became_peered":"2024-09-21T12:20:17.299015+0000","last_unstale":"2024-09-21T12:27:00.104087+0000","last_undegraded":"2024-09-21T12:27:00.104087+0000","last_fullsized":"2024-09-21T12:27:00.104087+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:25:41.888037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:41.888037+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:10.841319+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":364,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100919+0000","last_change":"2024-09-21T12:26:19.973528+0000","last_active":"2024-09-21T12:27:00.100919+0000","last_peered":"2024-09-21T12:27:00.100919+0000","last_clean":"2024-09-21T12:27:00.100919+0000","last_became_active":"2024-09-21T12:22:50.922019+0000","last_became_peered":"2024-09-21T12:22:50.922019+0000","last_unstale":"2024-09-21T12:27:00.100919+0000","last_undegraded":"2024-09-21T12:27:00.100919+0000","last_fullsized":"2024-09-21T12:27:00.100919+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:19.973461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:19.973461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:36.925984+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":326,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.103798+0000","last_change":"2024-09-21T12:26:21.572474+0000","last_active":"2024-09-21T12:27:00.103798+0000","last_peered":"2024-09-21T12:27:00.103798+0000","last_clean":"2024-09-21T12:27:00.103798+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:27:00.103798+0000","last_undegraded":"2024-09-21T12:27:00.103798+0000","last_fullsized":"2024-09-21T12:27:00.103798+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:21.571726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:21.571726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:50.797908+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":316,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.095069+0000","last_change":"2024-09-21T12:23:29.134829+0000","last_active":"2024-09-21T12:27:00.095069+0000","last_peered":"2024-09-21T12:27:00.095069+0000","last_clean":"2024-09-21T12:27:00.095069+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:27:00.095069+0000","last_undegraded":"2024-09-21T12:27:00.095069+0000","last_fullsized":"2024-09-21T12:27:00.095069+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:29.134767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:29.134767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:26.780715+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"76'79","reported_seq":32,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.107092+0000","last_change":"2024-09-21T12:26:49.624804+0000","last_active":"2024-09-21T12:27:00.107092+0000","last_peered":"2024-09-21T12:27:00.107092+0000","last_clean":"2024-09-21T12:27:00.107092+0000","last_became_active":"2024-09-21T12:26:46.989742+0000","last_became_peered":"2024-09-21T12:26:46.989742+0000","last_unstale":"2024-09-21T12:27:00.107092+0000","last_undegraded":"2024-09-21T12:27:00.107092+0000","last_fullsized":"2024-09-21T12:27:00.107092+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:49.623735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:49.623735+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:55.667557+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":349,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.102750+0000","last_change":"2024-09-21T12:24:58.824615+0000","last_active":"2024-09-21T12:27:00.102750+0000","last_peered":"2024-09-21T12:27:00.102750+0000","last_clean":"2024-09-21T12:27:00.102750+0000","last_became_active":"2024-09-21T12:20:17.694794+0000","last_became_peered":"2024-09-21T12:20:17.694794+0000","last_unstale":"2024-09-21T12:27:00.102750+0000","last_undegraded":"2024-09-21T12:27:00.102750+0000","last_fullsized":"2024-09-21T12:27:00.102750+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:24:58.824516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:58.824516+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":343,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.102519+0000","last_change":"2024-09-21T12:24:33.364225+0000","last_active":"2024-09-21T12:27:00.102519+0000","last_peered":"2024-09-21T12:27:00.102519+0000","last_clean":"2024-09-21T12:27:00.102519+0000","last_became_active":"2024-09-21T12:20:15.295968+0000","last_became_peered":"2024-09-21T12:20:15.295968+0000","last_unstale":"2024-09-21T12:27:00.102519+0000","last_undegraded":"2024-09-21T12:27:00.102519+0000","last_fullsized":"2024-09-21T12:27:00.102519+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:33.363808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:33.363808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:40.461614+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":340,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.098611+0000","last_change":"2024-09-21T12:23:57.018797+0000","last_active":"2024-09-21T12:27:00.098611+0000","last_peered":"2024-09-21T12:27:00.098611+0000","last_clean":"2024-09-21T12:27:00.098611+0000","last_became_active":"2024-09-21T12:22:50.921604+0000","last_became_peered":"2024-09-21T12:22:50.921604+0000","last_unstale":"2024-09-21T12:27:00.098611+0000","last_undegraded":"2024-09-21T12:27:00.098611+0000","last_fullsized":"2024-09-21T12:27:00.098611+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:57.018749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:57.018749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":314,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.106141+0000","last_change":"2024-09-21T12:24:35.588409+0000","last_active":"2024-09-21T12:27:00.106141+0000","last_peered":"2024-09-21T12:27:00.106141+0000","last_clean":"2024-09-21T12:27:00.106141+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:27:00.106141+0000","last_undegraded":"2024-09-21T12:27:00.106141+0000","last_fullsized":"2024-09-21T12:27:00.106141+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:35.588353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:35.588353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":328,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.103063+0000","last_change":"2024-09-21T12:23:51.343568+0000","last_active":"2024-09-21T12:27:00.103063+0000","last_peered":"2024-09-21T12:27:00.103063+0000","last_clean":"2024-09-21T12:27:00.103063+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:27:00.103063+0000","last_undegraded":"2024-09-21T12:27:00.103063+0000","last_fullsized":"2024-09-21T12:27:00.103063+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-21T12:23:51.343491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:23:51.343491+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:34.429508+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":342,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100843+0000","last_change":"2024-09-21T12:25:48.713463+0000","last_active":"2024-09-21T12:27:00.100843+0000","last_peered":"2024-09-21T12:27:00.100843+0000","last_clean":"2024-09-21T12:27:00.100843+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:27:00.100843+0000","last_undegraded":"2024-09-21T12:27:00.100843+0000","last_fullsized":"2024-09-21T12:27:00.100843+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:25:48.713392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:48.713392+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:06.809432+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":348,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.099860+0000","last_change":"2024-09-21T12:26:22.014941+0000","last_active":"2024-09-21T12:27:00.099860+0000","last_peered":"2024-09-21T12:27:00.099860+0000","last_clean":"2024-09-21T12:27:00.099860+0000","last_became_active":"2024-09-21T12:22:50.930615+0000","last_became_peered":"2024-09-21T12:22:50.930615+0000","last_unstale":"2024-09-21T12:27:00.099860+0000","last_undegraded":"2024-09-21T12:27:00.099860+0000","last_fullsized":"2024-09-21T12:27:00.099860+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:22.014896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:22.014896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:34.583908+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":336,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.099248+0000","last_change":"2024-09-21T12:26:14.855364+0000","last_active":"2024-09-21T12:27:00.099248+0000","last_peered":"2024-09-21T12:27:00.099248+0000","last_clean":"2024-09-21T12:27:00.099248+0000","last_became_active":"2024-09-21T12:22:50.917408+0000","last_became_peered":"2024-09-21T12:22:50.917408+0000","last_unstale":"2024-09-21T12:27:00.099248+0000","last_undegraded":"2024-09-21T12:27:00.099248+0000","last_fullsized":"2024-09-21T12:27:00.099248+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:14.855248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:14.855248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:15.644031+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":337,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.101521+0000","last_change":"2024-09-21T12:26:15.152613+0000","last_active":"2024-09-21T12:27:00.101521+0000","last_peered":"2024-09-21T12:27:00.101521+0000","last_clean":"2024-09-21T12:27:00.101521+0000","last_became_active":"2024-09-21T12:22:50.914221+0000","last_became_peered":"2024-09-21T12:22:50.914221+0000","last_unstale":"2024-09-21T12:27:00.101521+0000","last_undegraded":"2024-09-21T12:27:00.101521+0000","last_fullsized":"2024-09-21T12:27:00.101521+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:15.152540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:15.152540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:17.180451+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":334,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.103332+0000","last_change":"2024-09-21T12:25:00.871133+0000","last_active":"2024-09-21T12:27:00.103332+0000","last_peered":"2024-09-21T12:27:00.103332+0000","last_clean":"2024-09-21T12:27:00.103332+0000","last_became_active":"2024-09-21T12:20:15.297993+0000","last_became_peered":"2024-09-21T12:20:15.297993+0000","last_unstale":"2024-09-21T12:27:00.103332+0000","last_undegraded":"2024-09-21T12:27:00.103332+0000","last_fullsized":"2024-09-21T12:27:00.103332+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:00.871062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:00.871062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":343,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.100984+0000","last_change":"2024-09-21T12:25:47.974973+0000","last_active":"2024-09-21T12:27:00.100984+0000","last_peered":"2024-09-21T12:27:00.100984+0000","last_clean":"2024-09-21T12:27:00.100984+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:27:00.100984+0000","last_undegraded":"2024-09-21T12:27:00.100984+0000","last_fullsized":"2024-09-21T12:27:00.100984+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:25:47.974897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:47.974897+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:27:04.942357+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":345,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-21T12:27:00.101586+0000","last_change":"2024-09-21T12:26:46.824947+0000","last_active":"2024-09-21T12:27:00.101586+0000","last_peered":"2024-09-21T12:27:00.101586+0000","last_clean":"2024-09-21T12:27:00.101586+0000","last_became_active":"2024-09-21T12:20:19.308247+0000","last_became_peered":"2024-09-21T12:20:19.308247+0000","last_unstale":"2024-09-21T12:27:00.101586+0000","last_undegraded":"2024-09-21T12:27:00.101586+0000","last_fullsized":"2024-09-21T12:27:00.101586+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:46.824891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:46.824891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:05.202051+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5312479256,"num_objects":81063,"num_object_clones":0,"num_object_copies":243189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81063,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179765,"num_write_kb":8346433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15973429248,"data_stored":15973417032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65863,"ondisk_log_size":65863,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":689,"num_read_kb":446,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":3872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":396,"ondisk_log_size":396,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":5023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":401,"num_read_kb":396,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1659,"ondisk_log_size":1659,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234069,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2222708,"kb_used_data":1974944,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":91522444,"statfs":{"total":95995035648,"available":93718982656,"internally_reserved":0,"allocated":2022342656,"data_stored":2021994277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":253688266},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463005,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2267180,"kb_used_data":1983260,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":91477972,"statfs":{"total":95995035648,"available":93673443328,"internally_reserved":0,"allocated":2030858240,"data_stored":2030513724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":290650566},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724640,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2859192,"kb_used_data":2621156,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":90885960,"statfs":{"total":95995035648,"available":93067223040,"internally_reserved":0,"allocated":2684063744,"data_stored":2683712703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243726789},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953576,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149806,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2643080,"kb_used_data":2300920,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":91102072,"statfs":{"total":95995035648,"available":93288521728,"internally_reserved":0,"allocated":2356142080,"data_stored":2355770944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":350353863},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411444,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2519140,"kb_used_data":2285268,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":91226012,"statfs":{"total":95995035648,"available":93415436288,"internally_reserved":0,"allocated":2340114432,"data_stored":2339778232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":239466949},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574970,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2603156,"kb_used_data":2297088,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":91141996,"statfs":{"total":95995035648,"available":93329403904,"internally_reserved":0,"allocated":2352218112,"data_stored":2351837260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738496,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2553336,"kb_used_data":2275044,"kb_used_omap":1,"kb_used_meta":278270,"kb_avail":91191816,"statfs":{"total":95995035648,"available":93380419584,"internally_reserved":0,"allocated":2329645056,"data_stored":2329293270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":284948933},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2314797056,"data_stored":2314797056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2322530304,"data_stored":2322530304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2327121920,"data_stored":2327117848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2320105472,"data_stored":2320105472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2677608448,"data_stored":2677604376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2000158720,"data_stored":2000158720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2011107328,"data_stored":2011103256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:27:08.584 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T12:27:08.584 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pg_num 31 2024-09-21T12:27:08.642 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:08.668 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:08.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:08.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:08.769 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:08.796 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:08.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:08.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:08.898 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:08.924 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:08.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:08.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:09.026 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:09.055 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:09.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:09.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:09.156 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:09.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:09.182 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:09.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:09.282 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:09.329 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:27:09.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:09.337 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:09.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:09.395 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:27:09.438 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:09.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:09.467 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:09.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:09.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:09 smithi114 ceph-mon[30324]: pgmap v332: 168 pgs: 168 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.66k op/s 2024-09-21T12:27:09.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:09 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3721596484' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:27:09.470 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:09 smithi114 ceph-mon[40867]: pgmap v332: 168 pgs: 168 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.66k op/s 2024-09-21T12:27:09.470 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:09 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3721596484' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:27:09.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:09 smithi183 ceph-mon[33277]: pgmap v332: 168 pgs: 168 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.66k op/s 2024-09-21T12:27:09.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:09 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3721596484' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:27:09.568 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:09.615 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:09.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:09.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:09.717 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:09.757 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:09.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:09.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:09.859 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:09.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:09.893 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:09.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:09.994 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:10.029 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T12:27:10.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:10.033 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:10.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:10.133 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:10.176 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:10.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:10.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:10.179 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:27:10.179 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:27:10.278 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:10.301 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:10.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:10.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:10.402 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:10.440 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:10.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:10.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:10.541 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:10.575 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:10.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:10.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:10.677 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:10.703 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:10.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:10.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:10.805 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:10.832 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:10.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:10.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:10.934 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:10.960 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:10.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:10.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:11.062 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:11.089 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:11.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:11.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:11.191 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:11.227 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:11.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:11.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:11.269 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:27:11.329 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:11.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:11.380 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:11.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:11.481 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:11.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:11 smithi183 ceph-mon[33277]: pgmap v333: 168 pgs: 168 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.38k op/s 2024-09-21T12:27:11.506 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:11.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:11.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:11.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:11 smithi114 ceph-mon[40867]: pgmap v333: 168 pgs: 168 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.38k op/s 2024-09-21T12:27:11.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:11 smithi114 ceph-mon[30324]: pgmap v333: 168 pgs: 168 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.38k op/s 2024-09-21T12:27:11.608 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:11.648 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:11.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:11.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:11.750 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:11.798 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:11.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:11.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:11.900 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:11.926 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:11.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:11.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:12.028 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:12.068 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:12.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:12.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:12.169 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:12.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:12.196 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:12.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:12.297 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:12.326 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:12.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:12.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:12.428 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:12.467 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:12.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:12.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:12.569 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:12.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:12.591 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:12.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:12.692 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:12.717 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:12.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:12.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:12.811 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:27:12.819 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:12.841 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:12.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:12.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:12.850 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:27:12 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:27:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:27:12.852 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T12:27:12.920 INFO:teuthology.orchestra.run.smithi114.stderr:set pool 1 pg_num to 31 2024-09-21T12:27:12.943 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:12.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:12.990 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:12.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:13.090 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:13.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:13.116 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:13.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:13.127 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:12 smithi114 ceph-mon[30324]: pgmap v334: 168 pgs: 168 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.32k op/s 2024-09-21T12:27:13.127 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:12 smithi114 ceph-mon[30324]: osdmap e214: 8 total, 8 up, 7 in 2024-09-21T12:27:13.127 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:12 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1428966789' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-09-21T12:27:13.128 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:12 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-09-21T12:27:13.128 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:12 smithi114 ceph-mon[40867]: pgmap v334: 168 pgs: 168 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.32k op/s 2024-09-21T12:27:13.128 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:12 smithi114 ceph-mon[40867]: osdmap e214: 8 total, 8 up, 7 in 2024-09-21T12:27:13.128 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:12 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1428966789' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-09-21T12:27:13.128 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:12 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-09-21T12:27:13.217 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:13.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:12 smithi183 ceph-mon[33277]: pgmap v334: 168 pgs: 168 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.32k op/s 2024-09-21T12:27:13.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:12 smithi183 ceph-mon[33277]: osdmap e214: 8 total, 8 up, 7 in 2024-09-21T12:27:13.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:12 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1428966789' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-09-21T12:27:13.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:12 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-09-21T12:27:13.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:13.249 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:13.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:13.350 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:13.398 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:13.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:13.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:13.499 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:13.527 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:13.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:13.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:13.629 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:13.655 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:13.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:13.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:13.757 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:13.799 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:13.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:13.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:13.901 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:13.964 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:13.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:13.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:14.066 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:14.093 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:14.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:14.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:14.195 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:14.210 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:27:14.216 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:27:14.216 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:27:14.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:13 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:27:14.219 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:13 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:27:14.219 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:13 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-09-21T12:27:14.219 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:13 smithi114 ceph-mon[30324]: osdmap e215: 8 total, 8 up, 7 in 2024-09-21T12:27:14.219 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:13 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-09-21T12:27:14.219 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:13 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-09-21T12:27:14.219 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:13 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:27:14.219 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:13 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:27:14.219 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:13 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-09-21T12:27:14.219 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:13 smithi114 ceph-mon[40867]: osdmap e215: 8 total, 8 up, 7 in 2024-09-21T12:27:14.219 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:13 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-09-21T12:27:14.220 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:13 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-09-21T12:27:14.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:14.220 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:14.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:14.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:13 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:27:14.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:13 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:27:14.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:13 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-09-21T12:27:14.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:13 smithi183 ceph-mon[33277]: osdmap e215: 8 total, 8 up, 7 in 2024-09-21T12:27:14.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:13 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-09-21T12:27:14.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:13 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-09-21T12:27:14.252 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:27:14.258 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:27:14.258 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:27:14.297 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:27:14.297 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:27:14.322 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:14.335 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:27:14.335 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:27:14.364 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:27:14.365 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:27:14.370 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:14.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:14.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:14.396 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:27:14.397 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:27:14.430 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:27:14.430 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:27:14.463 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:27:14.463 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:27:14.472 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:14.497 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:14.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:14.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:14.598 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:14.634 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:14.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:14.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:14.736 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:14.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:14.766 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:14.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:14.867 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:14.894 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:14.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:14.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:14.996 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:15.029 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:15.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:15.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:15.131 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:15.166 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:15.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:15.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:15.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:14 smithi183 ceph-mon[33277]: pgmap v337: 168 pgs: 168 active+clean; 4.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 782 B/s rd, 0 B/s wr, 1.49k op/s 2024-09-21T12:27:15.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:14 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-21T12:27:15.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:14 smithi183 ceph-mon[33277]: osdmap e216: 8 total, 8 up, 7 in 2024-09-21T12:27:15.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:14 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:27:15.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:14 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:27:15.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:14 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1234375414' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:27:15.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:14 smithi183 ceph-mon[33277]: osdmap e217: 8 total, 8 up, 7 in 2024-09-21T12:27:15.268 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:15.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:14 smithi114 ceph-mon[40867]: pgmap v337: 168 pgs: 168 active+clean; 4.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 782 B/s rd, 0 B/s wr, 1.49k op/s 2024-09-21T12:27:15.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:14 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-21T12:27:15.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:14 smithi114 ceph-mon[40867]: osdmap e216: 8 total, 8 up, 7 in 2024-09-21T12:27:15.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:14 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:27:15.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:14 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:27:15.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:14 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1234375414' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:27:15.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:14 smithi114 ceph-mon[40867]: osdmap e217: 8 total, 8 up, 7 in 2024-09-21T12:27:15.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:14 smithi114 ceph-mon[30324]: pgmap v337: 168 pgs: 168 active+clean; 4.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 782 B/s rd, 0 B/s wr, 1.49k op/s 2024-09-21T12:27:15.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:14 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-21T12:27:15.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:14 smithi114 ceph-mon[30324]: osdmap e216: 8 total, 8 up, 7 in 2024-09-21T12:27:15.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:14 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:27:15.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:14 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:27:15.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:14 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1234375414' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:27:15.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:14 smithi114 ceph-mon[30324]: osdmap e217: 8 total, 8 up, 7 in 2024-09-21T12:27:15.295 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:15.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:15.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:15.396 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:15.433 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:15.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:15.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:15.536 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:15.567 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:15.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:15.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:15.669 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:15.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:15.705 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:15.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:15.805 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:15.831 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:15.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:15.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:15.932 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:15.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:15.983 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:15.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:16.084 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:16.115 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:15 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-09-21T12:27:16.115 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:15 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-09-21T12:27:16.116 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:15 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-09-21T12:27:16.116 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:15 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-09-21T12:27:16.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:16.124 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:16.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:16.225 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:16.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:15 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-09-21T12:27:16.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:15 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-09-21T12:27:16.258 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:16.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:16.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:16.361 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:16.385 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:16.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:16.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:16.487 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:16.531 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:16.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:16.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:16.633 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:16.659 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:16.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:16.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:16.761 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:16.788 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:16.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:16.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:16.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:16.911 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-21T12:27:16.939 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:16.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:16.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:17.041 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:17.079 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:17.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:17.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:17.181 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:17.217 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:17.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:17.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:17.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:27:16 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:27:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:16 smithi183 ceph-mon[33277]: pgmap v340: 178 pgs: 10 unknown, 168 active+clean; 4.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.34k op/s 2024-09-21T12:27:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:16 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-21T12:27:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:16 smithi183 ceph-mon[33277]: osdmap e218: 8 total, 8 up, 7 in 2024-09-21T12:27:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:16 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:27:17.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:16 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:27:17.240 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:16 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:27:17.240 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:16 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/86574919' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:27:17.240 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:16 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:27:17.240 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:16 smithi183 ceph-mon[33277]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:27:17.240 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:16 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:27:17.241 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:16 smithi183 ceph-mon[33277]: osdmap e219: 8 total, 8 up, 7 in 2024-09-21T12:27:17.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:16 smithi114 ceph-mon[40867]: pgmap v340: 178 pgs: 10 unknown, 168 active+clean; 4.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.34k op/s 2024-09-21T12:27:17.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:16 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-21T12:27:17.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:16 smithi114 ceph-mon[40867]: osdmap e218: 8 total, 8 up, 7 in 2024-09-21T12:27:17.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:16 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:27:17.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:16 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:27:17.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:16 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:27:17.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:16 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/86574919' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:27:17.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:16 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:27:17.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:16 smithi114 ceph-mon[40867]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:27:17.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:16 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:27:17.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:16 smithi114 ceph-mon[40867]: osdmap e219: 8 total, 8 up, 7 in 2024-09-21T12:27:17.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:16 smithi114 ceph-mon[30324]: pgmap v340: 178 pgs: 10 unknown, 168 active+clean; 4.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.34k op/s 2024-09-21T12:27:17.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:16 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-21T12:27:17.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:16 smithi114 ceph-mon[30324]: osdmap e218: 8 total, 8 up, 7 in 2024-09-21T12:27:17.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:16 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:27:17.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:16 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:27:17.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:16 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:27:17.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:16 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/86574919' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:27:17.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:16 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:27:17.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:16 smithi114 ceph-mon[30324]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:27:17.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:16 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:27:17.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:16 smithi114 ceph-mon[30324]: osdmap e219: 8 total, 8 up, 7 in 2024-09-21T12:27:17.319 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:17.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:17.357 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:17.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:17.458 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:17.486 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:17.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:17.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:17.562 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:27:17.563 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-21T12:27:17.563 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.563 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25022.0:200191 8.0 8:0fefa142:::benchmark_data_smithi114_104936_object69775:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-21T12:27:17.563 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:27:17.546665+0000", 2024-09-21T12:27:17.563 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.013558373, 2024-09-21T12:27:17.563 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.013588258000000001, 2024-09-21T12:27:17.563 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:27:17.563 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:27:17.563 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:27:17.563 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:27:17.563 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25022", 2024-09-21T12:27:17.563 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:27:17.564 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 200191 2024-09-21T12:27:17.564 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.564 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:27:17.564 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.564 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:27:17.564 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.546665+0000", 2024-09-21T12:27:17.564 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:27:17.564 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.564 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.564 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:27:17.564 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.546665+0000", 2024-09-21T12:27:17.564 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:27:17.564 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.564 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.564 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:27:17.565 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.546666+0000", 2024-09-21T12:27:17.565 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9400000000000005e-07 2024-09-21T12:27:17.565 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.565 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.565 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:27:17.565 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.546668+0000", 2024-09-21T12:27:17.565 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.7209999999999999e-06 2024-09-21T12:27:17.565 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.565 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.565 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:27:17.565 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.546675+0000", 2024-09-21T12:27:17.565 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.058e-06 2024-09-21T12:27:17.565 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.565 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.565 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:27:17.566 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.546681+0000", 2024-09-21T12:27:17.566 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0980000000000001e-06 2024-09-21T12:27:17.566 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.566 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.566 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:27:17.566 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.547650+0000", 2024-09-21T12:27:17.566 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00096919900000000002 2024-09-21T12:27:17.566 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.566 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.566 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:27:17.566 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.547745+0000", 2024-09-21T12:27:17.566 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.4998000000000001e-05 2024-09-21T12:27:17.566 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.567 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.567 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,5", 2024-09-21T12:27:17.567 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.547845+0000", 2024-09-21T12:27:17.567 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.9792999999999998e-05 2024-09-21T12:27:17.567 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.567 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.567 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:27:17.567 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.552170+0000", 2024-09-21T12:27:17.567 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0043257030000000002 2024-09-21T12:27:17.567 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.567 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.567 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:17.567 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.552439+0000", 2024-09-21T12:27:17.567 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00026880699999999998 2024-09-21T12:27:17.567 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:27:17.568 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:27:17.568 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:27:17.568 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.568 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.568 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25022.0:200192 8.f 8:f11f5294:::benchmark_data_smithi114_104936_object69776:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-21T12:27:17.568 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:27:17.546691+0000", 2024-09-21T12:27:17.568 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.01353263, 2024-09-21T12:27:17.568 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.013637668, 2024-09-21T12:27:17.568 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:27:17.568 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:27:17.568 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:27:17.568 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:27:17.569 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25022", 2024-09-21T12:27:17.569 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:27:17.569 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 200192 2024-09-21T12:27:17.569 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.569 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:27:17.569 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.569 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:27:17.569 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.546691+0000", 2024-09-21T12:27:17.569 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:27:17.569 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.569 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.569 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:27:17.569 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.546691+0000", 2024-09-21T12:27:17.569 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:27:17.569 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.570 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.570 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:27:17.570 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.546691+0000", 2024-09-21T12:27:17.570 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.01e-07 2024-09-21T12:27:17.570 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.570 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.570 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:27:17.570 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.546693+0000", 2024-09-21T12:27:17.570 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1009999999999999e-06 2024-09-21T12:27:17.570 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.570 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.570 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:27:17.571 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.546698+0000", 2024-09-21T12:27:17.571 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0509999999999996e-06 2024-09-21T12:27:17.571 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.571 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.571 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:27:17.571 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.546703+0000", 2024-09-21T12:27:17.571 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.498e-06 2024-09-21T12:27:17.571 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.571 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.571 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:27:17.571 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.552621+0000", 2024-09-21T12:27:17.572 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0059178310000000001 2024-09-21T12:27:17.572 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.572 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.572 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:27:17.572 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.552742+0000", 2024-09-21T12:27:17.572 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012108399999999999 2024-09-21T12:27:17.572 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.572 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.572 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,2", 2024-09-21T12:27:17.572 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.552859+0000", 2024-09-21T12:27:17.572 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000117581 2024-09-21T12:27:17.572 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.572 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.572 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:27:17.572 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.558165+0000", 2024-09-21T12:27:17.573 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.005306046 2024-09-21T12:27:17.573 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:27:17.573 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:27:17.573 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:27:17.573 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.573 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.573 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25022.0:200195 8.b 8:d6300d39:::benchmark_data_smithi114_104936_object69779:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-21T12:27:17.573 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:27:17.546708+0000", 2024-09-21T12:27:17.573 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.013515009999999999, 2024-09-21T12:27:17.573 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.013684254999999999, 2024-09-21T12:27:17.573 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:27:17.573 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:27:17.573 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:27:17.573 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:27:17.574 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25022", 2024-09-21T12:27:17.574 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:27:17.574 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 200195 2024-09-21T12:27:17.574 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.574 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:27:17.574 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.574 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:27:17.574 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.546708+0000", 2024-09-21T12:27:17.574 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:27:17.574 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.575 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.575 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:27:17.575 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.546708+0000", 2024-09-21T12:27:17.575 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:27:17.575 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.575 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.575 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:27:17.575 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.546709+0000", 2024-09-21T12:27:17.575 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.9200000000000002e-07 2024-09-21T12:27:17.575 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.575 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.575 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:27:17.575 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.546710+0000", 2024-09-21T12:27:17.575 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6899999999999999e-06 2024-09-21T12:27:17.575 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.576 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.576 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:27:17.576 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.546715+0000", 2024-09-21T12:27:17.576 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5190000000000003e-06 2024-09-21T12:27:17.576 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.576 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.576 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:27:17.576 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.546719+0000", 2024-09-21T12:27:17.576 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.7479999999999999e-06 2024-09-21T12:27:17.576 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.576 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.576 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:27:17.576 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.546768+0000", 2024-09-21T12:27:17.576 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9529e-05 2024-09-21T12:27:17.576 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:27:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.546873+0000", 2024-09-21T12:27:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000105241 2024-09-21T12:27:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,2", 2024-09-21T12:27:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.546984+0000", 2024-09-21T12:27:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011109599999999999 2024-09-21T12:27:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:27:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.550242+0000", 2024-09-21T12:27:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0032576549999999999 2024-09-21T12:27:17.577 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.578 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.578 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:17.578 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.556742+0000", 2024-09-21T12:27:17.578 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0064999999999999997 2024-09-21T12:27:17.578 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:27:17.578 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:27:17.578 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:27:17.578 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.578 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.578 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25022.0:200208 8.f 8:ffe54a1c:::benchmark_data_smithi114_104936_object69792:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-21T12:27:17.578 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:27:17.556576+0000", 2024-09-21T12:27:17.578 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0036468709999999999, 2024-09-21T12:27:17.578 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003883386, 2024-09-21T12:27:17.578 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:27:17.579 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:27:17.579 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:27:17.579 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:27:17.579 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25022", 2024-09-21T12:27:17.579 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:27:17.579 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 200208 2024-09-21T12:27:17.579 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.579 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:27:17.579 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.579 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:27:17.579 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.556576+0000", 2024-09-21T12:27:17.579 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:27:17.579 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.579 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.579 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:27:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.556576+0000", 2024-09-21T12:27:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:27:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:27:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.556578+0000", 2024-09-21T12:27:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7099999999999999e-06 2024-09-21T12:27:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:27:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.556584+0000", 2024-09-21T12:27:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.28e-06 2024-09-21T12:27:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:27:17.580 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.556599+0000", 2024-09-21T12:27:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4996e-05 2024-09-21T12:27:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:27:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.556608+0000", 2024-09-21T12:27:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.935e-06 2024-09-21T12:27:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:27:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.558512+0000", 2024-09-21T12:27:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001903323 2024-09-21T12:27:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:27:17.581 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.558623+0000", 2024-09-21T12:27:17.582 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000111399 2024-09-21T12:27:17.582 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:17.582 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:17.582 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,2", 2024-09-21T12:27:17.582 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:17.558733+0000", 2024-09-21T12:27:17.582 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000109598 2024-09-21T12:27:17.582 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:27:17.582 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:27:17.582 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:27:17.582 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:27:17.582 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:27:17.582 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 4 2024-09-21T12:27:17.582 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:27:17.588 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:17.629 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:17.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:17.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:17.730 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:17.764 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:17.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:17.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:17.866 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:17.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:17.905 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:17.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:18.006 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:18.034 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:18.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:18.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:18.136 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:18.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:18.173 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:18.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:18.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:17 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:27:18.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:17 smithi183 ceph-mon[33277]: 1.c scrub starts 2024-09-21T12:27:18.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:17 smithi183 ceph-mon[33277]: 1.c scrub ok 2024-09-21T12:27:18.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:17 smithi183 ceph-mon[33277]: 1.9 scrub starts 2024-09-21T12:27:18.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:17 smithi183 ceph-mon[33277]: 1.9 scrub ok 2024-09-21T12:27:18.274 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:18.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:17 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:27:18.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:17 smithi114 ceph-mon[40867]: 1.c scrub starts 2024-09-21T12:27:18.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:17 smithi114 ceph-mon[40867]: 1.c scrub ok 2024-09-21T12:27:18.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:17 smithi114 ceph-mon[40867]: 1.9 scrub starts 2024-09-21T12:27:18.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:17 smithi114 ceph-mon[40867]: 1.9 scrub ok 2024-09-21T12:27:18.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:17 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:27:18.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:17 smithi114 ceph-mon[30324]: 1.c scrub starts 2024-09-21T12:27:18.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:17 smithi114 ceph-mon[30324]: 1.c scrub ok 2024-09-21T12:27:18.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:17 smithi114 ceph-mon[30324]: 1.9 scrub starts 2024-09-21T12:27:18.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:17 smithi114 ceph-mon[30324]: 1.9 scrub ok 2024-09-21T12:27:18.301 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:18.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:18.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:18.403 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:18.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:18.451 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:18.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:18.552 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:18.582 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:18.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:18.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:18.684 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:18.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:18.732 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:18.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:18.822 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T12:27:18.833 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:18.856 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:18.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:18.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:18.958 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T12:27:18.958 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:27:18.958 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-21T12:27:18.959 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.0 'trim stale osdmaps' 2024-09-21T12:27:18.959 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:19.030 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:19.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:19.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:19.132 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:19.193 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:19.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:19.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:19.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:18 smithi183 ceph-mon[33277]: pgmap v343: 178 pgs: 1 activating, 6 peering, 171 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 7.7 KiB/s rd, 46 KiB/s wr, 1.68k op/s; 16 KiB/s, 0 objects/s recovering 2024-09-21T12:27:19.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:18 smithi183 ceph-mon[33277]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T12:27:19.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:18 smithi183 ceph-mon[33277]: 1.16 scrub starts 2024-09-21T12:27:19.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:18 smithi183 ceph-mon[33277]: 1.16 scrub ok 2024-09-21T12:27:19.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:18 smithi183 ceph-mon[33277]: 1.d scrub starts 2024-09-21T12:27:19.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:18 smithi183 ceph-mon[33277]: 1.d scrub ok 2024-09-21T12:27:19.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:18 smithi183 ceph-mon[33277]: 1.17 scrub starts 2024-09-21T12:27:19.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:18 smithi183 ceph-mon[33277]: 1.17 scrub ok 2024-09-21T12:27:19.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:18 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:27:19.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:18 smithi183 ceph-mon[33277]: 1.a scrub starts 2024-09-21T12:27:19.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:18 smithi183 ceph-mon[33277]: 1.a scrub ok 2024-09-21T12:27:19.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:18 smithi114 ceph-mon[30324]: pgmap v343: 178 pgs: 1 activating, 6 peering, 171 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 7.7 KiB/s rd, 46 KiB/s wr, 1.68k op/s; 16 KiB/s, 0 objects/s recovering 2024-09-21T12:27:19.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:18 smithi114 ceph-mon[30324]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T12:27:19.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:18 smithi114 ceph-mon[30324]: 1.16 scrub starts 2024-09-21T12:27:19.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:18 smithi114 ceph-mon[30324]: 1.16 scrub ok 2024-09-21T12:27:19.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:18 smithi114 ceph-mon[30324]: 1.d scrub starts 2024-09-21T12:27:19.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:18 smithi114 ceph-mon[30324]: 1.d scrub ok 2024-09-21T12:27:19.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:18 smithi114 ceph-mon[30324]: 1.17 scrub starts 2024-09-21T12:27:19.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:18 smithi114 ceph-mon[30324]: 1.17 scrub ok 2024-09-21T12:27:19.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:18 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:27:19.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:18 smithi114 ceph-mon[30324]: 1.a scrub starts 2024-09-21T12:27:19.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:18 smithi114 ceph-mon[30324]: 1.a scrub ok 2024-09-21T12:27:19.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:18 smithi114 ceph-mon[40867]: pgmap v343: 178 pgs: 1 activating, 6 peering, 171 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 7.7 KiB/s rd, 46 KiB/s wr, 1.68k op/s; 16 KiB/s, 0 objects/s recovering 2024-09-21T12:27:19.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:18 smithi114 ceph-mon[40867]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T12:27:19.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:18 smithi114 ceph-mon[40867]: 1.16 scrub starts 2024-09-21T12:27:19.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:18 smithi114 ceph-mon[40867]: 1.16 scrub ok 2024-09-21T12:27:19.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:18 smithi114 ceph-mon[40867]: 1.d scrub starts 2024-09-21T12:27:19.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:18 smithi114 ceph-mon[40867]: 1.d scrub ok 2024-09-21T12:27:19.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:18 smithi114 ceph-mon[40867]: 1.17 scrub starts 2024-09-21T12:27:19.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:18 smithi114 ceph-mon[40867]: 1.17 scrub ok 2024-09-21T12:27:19.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:18 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:27:19.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:18 smithi114 ceph-mon[40867]: 1.a scrub starts 2024-09-21T12:27:19.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:18 smithi114 ceph-mon[40867]: 1.a scrub ok 2024-09-21T12:27:19.295 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:19.307 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:27:19.319 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:19.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:19.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:19.328 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:27:19.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:19.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:19.500 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:19.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:19.601 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:19.623 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:19.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:19.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:19.724 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:19.751 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:19.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:19.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:19.852 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:19.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:19.904 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:19.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:19.967 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:27:19.993 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:27:19.993 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:27:19.994 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:27:19.994 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:27:19.994 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:27:20.005 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:20.061 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:20.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:20.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:20.163 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:20.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:20.200 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:20.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:20.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:19 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:27:20.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:19 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:27:20.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:19 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:27:20.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:19 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:27:20.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:19 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:27:20.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:19 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:27:20.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:19 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:27:20.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:19 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:27:20.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:19 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:27:20.301 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:20.336 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:20.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:20.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:20.382 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T12:27:20.438 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:20.474 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:20.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:20.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:20.576 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:20.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:20.617 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:20.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:20.717 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:20.752 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:20.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:20.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:20.854 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:20.875 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:20.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:20.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:20.977 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:21.003 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:21.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:21.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:21.105 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:21.135 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:21.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:21.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:21.237 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:21.262 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:21.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:21.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:21.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:21 smithi114 ceph-mon[30324]: pgmap v344: 178 pgs: 1 activating, 6 peering, 171 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 5.4 KiB/s rd, 32 KiB/s wr, 1.16k op/s; 11 KiB/s, 0 objects/s recovering 2024-09-21T12:27:21.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:21 smithi114 ceph-mon[30324]: 1.6 scrub starts 2024-09-21T12:27:21.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:21 smithi114 ceph-mon[30324]: 1.6 scrub ok 2024-09-21T12:27:21.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:21 smithi114 ceph-mon[30324]: 1.1e scrub starts 2024-09-21T12:27:21.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:21 smithi114 ceph-mon[30324]: 1.1e scrub ok 2024-09-21T12:27:21.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:21 smithi114 ceph-mon[30324]: 1.5 scrub starts 2024-09-21T12:27:21.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:21 smithi114 ceph-mon[30324]: 1.5 scrub ok 2024-09-21T12:27:21.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:21 smithi114 ceph-mon[40867]: pgmap v344: 178 pgs: 1 activating, 6 peering, 171 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 5.4 KiB/s rd, 32 KiB/s wr, 1.16k op/s; 11 KiB/s, 0 objects/s recovering 2024-09-21T12:27:21.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:21 smithi114 ceph-mon[40867]: 1.6 scrub starts 2024-09-21T12:27:21.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:21 smithi114 ceph-mon[40867]: 1.6 scrub ok 2024-09-21T12:27:21.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:21 smithi114 ceph-mon[40867]: 1.1e scrub starts 2024-09-21T12:27:21.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:21 smithi114 ceph-mon[40867]: 1.1e scrub ok 2024-09-21T12:27:21.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:21 smithi114 ceph-mon[40867]: 1.5 scrub starts 2024-09-21T12:27:21.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:21 smithi114 ceph-mon[40867]: 1.5 scrub ok 2024-09-21T12:27:21.360 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:27:21.363 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:21.365 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T12:27:21.414 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:21.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:21.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:21.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:21 smithi183 ceph-mon[33277]: pgmap v344: 178 pgs: 1 activating, 6 peering, 171 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 5.4 KiB/s rd, 32 KiB/s wr, 1.16k op/s; 11 KiB/s, 0 objects/s recovering 2024-09-21T12:27:21.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:21 smithi183 ceph-mon[33277]: 1.6 scrub starts 2024-09-21T12:27:21.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:21 smithi183 ceph-mon[33277]: 1.6 scrub ok 2024-09-21T12:27:21.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:21 smithi183 ceph-mon[33277]: 1.1e scrub starts 2024-09-21T12:27:21.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:21 smithi183 ceph-mon[33277]: 1.1e scrub ok 2024-09-21T12:27:21.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:21 smithi183 ceph-mon[33277]: 1.5 scrub starts 2024-09-21T12:27:21.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:21 smithi183 ceph-mon[33277]: 1.5 scrub ok 2024-09-21T12:27:21.516 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:21.542 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:21.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:21.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:21.643 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:21.669 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:21.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:21.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:21.770 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:21.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:21.817 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:21.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:21.917 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:21.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:21.945 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:21.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:22.045 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:22.071 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:22.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:22.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:22.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:22 smithi114 ceph-mon[30324]: 1.e scrub starts 2024-09-21T12:27:22.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:22 smithi114 ceph-mon[30324]: 1.e scrub ok 2024-09-21T12:27:22.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:22 smithi114 ceph-mon[30324]: 1.18 scrub starts 2024-09-21T12:27:22.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:22 smithi114 ceph-mon[30324]: 1.18 scrub ok 2024-09-21T12:27:22.171 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:22 smithi114 ceph-mon[40867]: 1.e scrub starts 2024-09-21T12:27:22.171 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:22 smithi114 ceph-mon[40867]: 1.e scrub ok 2024-09-21T12:27:22.171 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:22 smithi114 ceph-mon[40867]: 1.18 scrub starts 2024-09-21T12:27:22.171 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:22 smithi114 ceph-mon[40867]: 1.18 scrub ok 2024-09-21T12:27:22.173 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:22.209 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:22.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:22.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:22.273 INFO:teuthology.orchestra.run.smithi114.stderr:no valid command found; 10 closest matches: 2024-09-21T12:27:22.274 INFO:teuthology.orchestra.run.smithi114.stderr:0 2024-09-21T12:27:22.274 INFO:teuthology.orchestra.run.smithi114.stderr:1 2024-09-21T12:27:22.274 INFO:teuthology.orchestra.run.smithi114.stderr:2 2024-09-21T12:27:22.274 INFO:teuthology.orchestra.run.smithi114.stderr:abort 2024-09-21T12:27:22.274 INFO:teuthology.orchestra.run.smithi114.stderr:assert 2024-09-21T12:27:22.274 INFO:teuthology.orchestra.run.smithi114.stderr:bench [] [] [] [] 2024-09-21T12:27:22.274 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs debug_inject_read_zeros 2024-09-21T12:27:22.274 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs files list 2024-09-21T12:27:22.274 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs stats 2024-09-21T12:27:22.274 INFO:teuthology.orchestra.run.smithi114.stderr:bluestore allocator dump block 2024-09-21T12:27:22.275 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: invalid command 2024-09-21T12:27:22.311 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:22.354 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:22.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:22.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:22.456 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:22.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:22 smithi183 ceph-mon[33277]: 1.e scrub starts 2024-09-21T12:27:22.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:22 smithi183 ceph-mon[33277]: 1.e scrub ok 2024-09-21T12:27:22.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:22 smithi183 ceph-mon[33277]: 1.18 scrub starts 2024-09-21T12:27:22.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:22 smithi183 ceph-mon[33277]: 1.18 scrub ok 2024-09-21T12:27:22.491 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:22.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:22.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:22.593 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:22.621 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:22.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:22.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:22.682 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:27:22.682 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:27:22.682 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:27:22.682 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:27:22.682 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:27:22.683 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:27:22.683 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:27:22.683 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:27:22.683 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:27:22.683 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:27:22.699 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:27:22.699 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:27:22.723 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:22.731 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:27:22.732 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:27:22.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:22.749 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:22.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:22.765 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:27:22.765 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:27:22.784 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:27:22 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:27:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:27:22.850 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:22.890 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:22.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:22.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:22.992 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:23.021 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:23.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:23.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:23.123 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:23.150 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:23.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:23.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:23.252 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:23.288 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:23.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:23.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:23.390 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:23.416 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:23.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:23.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:23.481 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:23 smithi114 ceph-mon[30324]: pgmap v345: 178 pgs: 1 activating, 177 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 4.6 KiB/s rd, 22 KiB/s wr, 1.17k op/s; 7.7 KiB/s, 0 objects/s recovering 2024-09-21T12:27:23.481 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:23 smithi114 ceph-mon[30324]: 5.5 scrub starts 2024-09-21T12:27:23.481 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:23 smithi114 ceph-mon[30324]: 5.5 scrub ok 2024-09-21T12:27:23.482 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:23 smithi114 ceph-mon[30324]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-21T12:27:23.482 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:23 smithi114 ceph-mon[30324]: 1.19 scrub starts 2024-09-21T12:27:23.482 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:23 smithi114 ceph-mon[30324]: 1.19 scrub ok 2024-09-21T12:27:23.482 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:23 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/849064950' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:27:23.482 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:23 smithi114 ceph-mon[30324]: 4.1c scrub starts 2024-09-21T12:27:23.482 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:23 smithi114 ceph-mon[30324]: 4.1c scrub ok 2024-09-21T12:27:23.482 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:23 smithi114 ceph-mon[40867]: pgmap v345: 178 pgs: 1 activating, 177 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 4.6 KiB/s rd, 22 KiB/s wr, 1.17k op/s; 7.7 KiB/s, 0 objects/s recovering 2024-09-21T12:27:23.482 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:23 smithi114 ceph-mon[40867]: 5.5 scrub starts 2024-09-21T12:27:23.482 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:23 smithi114 ceph-mon[40867]: 5.5 scrub ok 2024-09-21T12:27:23.483 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:23 smithi114 ceph-mon[40867]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-21T12:27:23.483 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:23 smithi114 ceph-mon[40867]: 1.19 scrub starts 2024-09-21T12:27:23.483 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:23 smithi114 ceph-mon[40867]: 1.19 scrub ok 2024-09-21T12:27:23.483 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:23 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/849064950' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:27:23.483 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:23 smithi114 ceph-mon[40867]: 4.1c scrub starts 2024-09-21T12:27:23.483 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:23 smithi114 ceph-mon[40867]: 4.1c scrub ok 2024-09-21T12:27:23.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:23 smithi183 ceph-mon[33277]: pgmap v345: 178 pgs: 1 activating, 177 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 4.6 KiB/s rd, 22 KiB/s wr, 1.17k op/s; 7.7 KiB/s, 0 objects/s recovering 2024-09-21T12:27:23.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:23 smithi183 ceph-mon[33277]: 5.5 scrub starts 2024-09-21T12:27:23.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:23 smithi183 ceph-mon[33277]: 5.5 scrub ok 2024-09-21T12:27:23.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:23 smithi183 ceph-mon[33277]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-21T12:27:23.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:23 smithi183 ceph-mon[33277]: 1.19 scrub starts 2024-09-21T12:27:23.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:23 smithi183 ceph-mon[33277]: 1.19 scrub ok 2024-09-21T12:27:23.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:23 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/849064950' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:27:23.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:23 smithi183 ceph-mon[33277]: 4.1c scrub starts 2024-09-21T12:27:23.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:23 smithi183 ceph-mon[33277]: 4.1c scrub ok 2024-09-21T12:27:23.518 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:23.529 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T12:27:23.529 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T12:27:23.530 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.1 'trim stale osdmaps' 2024-09-21T12:27:23.545 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:23.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:23.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:23.647 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:23.674 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:23.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:23.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:23.776 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:23.803 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:23.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:23.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:23.904 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:23.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:23.947 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:23.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:24.018 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:27:24.023 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:27:24.048 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:24.071 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:24.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:24.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:24.082 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:27:24.172 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:24.214 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:24.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:24.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:24.316 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:24.343 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:24.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:24.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:24.445 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:24.470 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:24.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:24.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:24.572 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:24.607 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:24.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:24.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:24.709 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:24.749 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:24.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:24.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:24.850 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:24.890 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:24.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:24.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:24.992 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:25.016 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:25.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:25.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:25.117 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:25.156 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:27:25.156 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:27:25.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:25.159 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:25.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:25.259 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:25.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:25 smithi114 ceph-mon[40867]: pgmap v346: 178 pgs: 178 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 2.6 KiB/s rd, 9.2 KiB/s wr, 1.62k op/s; 3.3 KiB/s, 0 objects/s recovering 2024-09-21T12:27:25.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:25 smithi114 ceph-mon[40867]: 1.1d scrub starts 2024-09-21T12:27:25.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:25 smithi114 ceph-mon[40867]: 1.1d scrub ok 2024-09-21T12:27:25.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:25 smithi114 ceph-mon[40867]: 1.15 scrub starts 2024-09-21T12:27:25.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:25 smithi114 ceph-mon[40867]: 1.15 scrub ok 2024-09-21T12:27:25.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:25 smithi114 ceph-mon[40867]: 1.1b scrub starts 2024-09-21T12:27:25.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:25 smithi114 ceph-mon[40867]: 1.1b scrub ok 2024-09-21T12:27:25.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:25 smithi114 ceph-mon[40867]: 4.d scrub starts 2024-09-21T12:27:25.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:25 smithi114 ceph-mon[40867]: 4.d scrub ok 2024-09-21T12:27:25.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:25 smithi114 ceph-mon[30324]: pgmap v346: 178 pgs: 178 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 2.6 KiB/s rd, 9.2 KiB/s wr, 1.62k op/s; 3.3 KiB/s, 0 objects/s recovering 2024-09-21T12:27:25.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:25 smithi114 ceph-mon[30324]: 1.1d scrub starts 2024-09-21T12:27:25.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:25 smithi114 ceph-mon[30324]: 1.1d scrub ok 2024-09-21T12:27:25.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:25 smithi114 ceph-mon[30324]: 1.15 scrub starts 2024-09-21T12:27:25.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:25 smithi114 ceph-mon[30324]: 1.15 scrub ok 2024-09-21T12:27:25.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:25 smithi114 ceph-mon[30324]: 1.1b scrub starts 2024-09-21T12:27:25.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:25 smithi114 ceph-mon[30324]: 1.1b scrub ok 2024-09-21T12:27:25.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:25 smithi114 ceph-mon[30324]: 4.d scrub starts 2024-09-21T12:27:25.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:25 smithi114 ceph-mon[30324]: 4.d scrub ok 2024-09-21T12:27:25.285 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:25.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:25.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:25.386 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:25.410 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:25.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:25.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:25.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:25 smithi183 ceph-mon[33277]: pgmap v346: 178 pgs: 178 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 2.6 KiB/s rd, 9.2 KiB/s wr, 1.62k op/s; 3.3 KiB/s, 0 objects/s recovering 2024-09-21T12:27:25.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:25 smithi183 ceph-mon[33277]: 1.1d scrub starts 2024-09-21T12:27:25.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:25 smithi183 ceph-mon[33277]: 1.1d scrub ok 2024-09-21T12:27:25.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:25 smithi183 ceph-mon[33277]: 1.15 scrub starts 2024-09-21T12:27:25.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:25 smithi183 ceph-mon[33277]: 1.15 scrub ok 2024-09-21T12:27:25.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:25 smithi183 ceph-mon[33277]: 1.1b scrub starts 2024-09-21T12:27:25.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:25 smithi183 ceph-mon[33277]: 1.1b scrub ok 2024-09-21T12:27:25.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:25 smithi183 ceph-mon[33277]: 4.d scrub starts 2024-09-21T12:27:25.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:25 smithi183 ceph-mon[33277]: 4.d scrub ok 2024-09-21T12:27:25.512 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:25.539 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:25.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:25.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:25.641 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:25.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:25.677 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:25.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:25.777 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:25.815 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:25.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:25.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:25.916 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:25.939 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:25.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:25.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:25.944 INFO:teuthology.orchestra.run.smithi114.stderr:no valid command found; 10 closest matches: 2024-09-21T12:27:25.944 INFO:teuthology.orchestra.run.smithi114.stderr:0 2024-09-21T12:27:25.944 INFO:teuthology.orchestra.run.smithi114.stderr:1 2024-09-21T12:27:25.945 INFO:teuthology.orchestra.run.smithi114.stderr:2 2024-09-21T12:27:25.945 INFO:teuthology.orchestra.run.smithi114.stderr:abort 2024-09-21T12:27:25.945 INFO:teuthology.orchestra.run.smithi114.stderr:assert 2024-09-21T12:27:25.945 INFO:teuthology.orchestra.run.smithi114.stderr:bench [] [] [] [] 2024-09-21T12:27:25.945 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs debug_inject_read_zeros 2024-09-21T12:27:25.945 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs files list 2024-09-21T12:27:25.945 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs stats 2024-09-21T12:27:25.945 INFO:teuthology.orchestra.run.smithi114.stderr:bluestore allocator dump block 2024-09-21T12:27:25.945 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: invalid command 2024-09-21T12:27:25.992 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:27:26.041 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:26.067 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:26.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:26.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:26.137 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-21T12:27:26.138 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:26 smithi114 ceph-mon[30324]: 1.7 scrub starts 2024-09-21T12:27:26.138 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:26 smithi114 ceph-mon[30324]: 1.7 scrub ok 2024-09-21T12:27:26.138 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:26 smithi114 ceph-mon[30324]: 3.5 scrub starts 2024-09-21T12:27:26.138 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:26 smithi114 ceph-mon[30324]: 3.5 scrub ok 2024-09-21T12:27:26.138 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:26 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3809348850' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:27:26.138 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:26 smithi114 ceph-mon[40867]: 1.7 scrub starts 2024-09-21T12:27:26.138 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:26 smithi114 ceph-mon[40867]: 1.7 scrub ok 2024-09-21T12:27:26.138 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:26 smithi114 ceph-mon[40867]: 3.5 scrub starts 2024-09-21T12:27:26.138 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:26 smithi114 ceph-mon[40867]: 3.5 scrub ok 2024-09-21T12:27:26.138 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:26 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3809348850' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:27:26.169 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:26.194 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:26.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:26.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:26.296 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:26.339 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:26.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:26.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:26.441 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:26.474 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:26.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:26.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:26.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:26 smithi183 ceph-mon[33277]: 1.7 scrub starts 2024-09-21T12:27:26.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:26 smithi183 ceph-mon[33277]: 1.7 scrub ok 2024-09-21T12:27:26.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:26 smithi183 ceph-mon[33277]: 3.5 scrub starts 2024-09-21T12:27:26.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:26 smithi183 ceph-mon[33277]: 3.5 scrub ok 2024-09-21T12:27:26.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:26 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3809348850' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:27:26.576 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:26.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:26.614 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:26.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:26.715 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:26.755 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T12:27:26.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:26.773 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:26.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:26.874 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:26.901 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:26.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:26.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:27.003 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:27.036 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:27.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:27.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:27.091 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:27:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:27:27.102 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:27:27.138 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:27.178 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:27.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:27.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:27.280 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:27.308 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:27.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:27.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:27.395 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:27 smithi183 ceph-mon[33277]: pgmap v347: 178 pgs: 178 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 2.2 KiB/s rd, 7.6 KiB/s wr, 1.33k op/s; 2.7 KiB/s, 0 objects/s recovering 2024-09-21T12:27:27.395 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:27 smithi183 ceph-mon[33277]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:27:27.395 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:27 smithi183 ceph-mon[33277]: Cluster is now healthy 2024-09-21T12:27:27.395 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:27 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3809348850' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:27:27.395 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:27 smithi183 ceph-mon[33277]: osdmap e220: 8 total, 8 up, 7 in 2024-09-21T12:27:27.395 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:27 smithi183 ceph-mon[33277]: osdmap e221: 8 total, 8 up, 7 in 2024-09-21T12:27:27.410 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:27.415 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:27 smithi114 ceph-mon[40867]: pgmap v347: 178 pgs: 178 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 2.2 KiB/s rd, 7.6 KiB/s wr, 1.33k op/s; 2.7 KiB/s, 0 objects/s recovering 2024-09-21T12:27:27.415 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:27 smithi114 ceph-mon[40867]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:27:27.415 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:27 smithi114 ceph-mon[40867]: Cluster is now healthy 2024-09-21T12:27:27.415 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:27 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3809348850' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:27:27.415 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:27 smithi114 ceph-mon[40867]: osdmap e220: 8 total, 8 up, 7 in 2024-09-21T12:27:27.415 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:27 smithi114 ceph-mon[40867]: osdmap e221: 8 total, 8 up, 7 in 2024-09-21T12:27:27.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:27 smithi114 ceph-mon[30324]: pgmap v347: 178 pgs: 178 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 2.2 KiB/s rd, 7.6 KiB/s wr, 1.33k op/s; 2.7 KiB/s, 0 objects/s recovering 2024-09-21T12:27:27.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:27 smithi114 ceph-mon[30324]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:27:27.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:27 smithi114 ceph-mon[30324]: Cluster is now healthy 2024-09-21T12:27:27.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:27 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3809348850' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:27:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:27 smithi114 ceph-mon[30324]: osdmap e220: 8 total, 8 up, 7 in 2024-09-21T12:27:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:27 smithi114 ceph-mon[30324]: osdmap e221: 8 total, 8 up, 7 in 2024-09-21T12:27:27.440 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:27.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:27.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:27.541 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:27.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:27.572 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:27.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:27.673 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:27.710 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:27.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:27.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:27.715 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:27:27.715 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:27:27.812 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:27.845 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:27.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:27.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:27.916 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T12:27:27.916 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T12:27:27.916 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.2 'trim stale osdmaps' 2024-09-21T12:27:27.947 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:27.972 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:27.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:27.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:28.074 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:28.100 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:28.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:28.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:28.202 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:28.240 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:28.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:28.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:28.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:28 smithi114 ceph-mon[30324]: 4.13 scrub starts 2024-09-21T12:27:28.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:28 smithi114 ceph-mon[30324]: 4.13 scrub ok 2024-09-21T12:27:28.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:28 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:27:28.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:28 smithi114 ceph-mon[30324]: 4.2 scrub starts 2024-09-21T12:27:28.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:28 smithi114 ceph-mon[30324]: 4.2 scrub ok 2024-09-21T12:27:28.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:28 smithi114 ceph-mon[40867]: 4.13 scrub starts 2024-09-21T12:27:28.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:28 smithi114 ceph-mon[40867]: 4.13 scrub ok 2024-09-21T12:27:28.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:28 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:27:28.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:28 smithi114 ceph-mon[40867]: 4.2 scrub starts 2024-09-21T12:27:28.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:28 smithi114 ceph-mon[40867]: 4.2 scrub ok 2024-09-21T12:27:28.342 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:28.370 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:28.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:28.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:28.472 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:28.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:28 smithi183 ceph-mon[33277]: 4.13 scrub starts 2024-09-21T12:27:28.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:28 smithi183 ceph-mon[33277]: 4.13 scrub ok 2024-09-21T12:27:28.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:28 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:27:28.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:28 smithi183 ceph-mon[33277]: 4.2 scrub starts 2024-09-21T12:27:28.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:28 smithi183 ceph-mon[33277]: 4.2 scrub ok 2024-09-21T12:27:28.498 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:28.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:28.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:28.559 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T12:27:28.599 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:28.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:28.636 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:28.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:28.737 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:28.767 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:28.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:28.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:28.868 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:28.912 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:27:28.948 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:28.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:28.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:28.961 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:27:28.990 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:27:29.050 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:29.090 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:29.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:29.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:29.173 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:29 smithi114 ceph-mon[30324]: pgmap v350: 178 pgs: 178 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.65k op/s 2024-09-21T12:27:29.173 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:29 smithi114 ceph-mon[30324]: 3.0 scrub starts 2024-09-21T12:27:29.173 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:29 smithi114 ceph-mon[30324]: 3.0 scrub ok 2024-09-21T12:27:29.173 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:29 smithi114 ceph-mon[30324]: 1.1c scrub starts 2024-09-21T12:27:29.174 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:29 smithi114 ceph-mon[30324]: 1.1c scrub ok 2024-09-21T12:27:29.174 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:29 smithi114 ceph-mon[40867]: pgmap v350: 178 pgs: 178 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.65k op/s 2024-09-21T12:27:29.174 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:29 smithi114 ceph-mon[40867]: 3.0 scrub starts 2024-09-21T12:27:29.174 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:29 smithi114 ceph-mon[40867]: 3.0 scrub ok 2024-09-21T12:27:29.174 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:29 smithi114 ceph-mon[40867]: 1.1c scrub starts 2024-09-21T12:27:29.174 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:29 smithi114 ceph-mon[40867]: 1.1c scrub ok 2024-09-21T12:27:29.192 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:29.225 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:29.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:29.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:29.327 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:29.368 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:29.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:29.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:29.470 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:29.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:29 smithi183 ceph-mon[33277]: pgmap v350: 178 pgs: 178 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.65k op/s 2024-09-21T12:27:29.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:29 smithi183 ceph-mon[33277]: 3.0 scrub starts 2024-09-21T12:27:29.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:29 smithi183 ceph-mon[33277]: 3.0 scrub ok 2024-09-21T12:27:29.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:29 smithi183 ceph-mon[33277]: 1.1c scrub starts 2024-09-21T12:27:29.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:29 smithi183 ceph-mon[33277]: 1.1c scrub ok 2024-09-21T12:27:29.495 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:29.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:29.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:29.597 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:29.638 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:29.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:29.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:29.658 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:27:29.658 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:27:29.695 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:27:29.696 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:27:29.728 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:27:29.728 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:27:29.740 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:29.752 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:27:29.752 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:27:29.760 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:29.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:29.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:29.768 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:27:29.769 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:27:29.822 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:27:29.822 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:27:29.856 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:27:29.856 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:27:29.862 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:29.889 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:27:29.889 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:27:29.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:29.897 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:29.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:29.998 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:30.025 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:30.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:30.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:30.126 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:30.154 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:30.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:30.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:30.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:30 smithi183 ceph-mon[33277]: 1.1a scrub starts 2024-09-21T12:27:30.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:30 smithi183 ceph-mon[33277]: 1.1a scrub ok 2024-09-21T12:27:30.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:30 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/624903671' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:27:30.256 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:30.292 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:30.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:30.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:30.394 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:30.400 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:30 smithi114 ceph-mon[30324]: 1.1a scrub starts 2024-09-21T12:27:30.400 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:30 smithi114 ceph-mon[30324]: 1.1a scrub ok 2024-09-21T12:27:30.400 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:30 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/624903671' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:27:30.401 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:30 smithi114 ceph-mon[40867]: 1.1a scrub starts 2024-09-21T12:27:30.401 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:30 smithi114 ceph-mon[40867]: 1.1a scrub ok 2024-09-21T12:27:30.401 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:30 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/624903671' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:27:30.434 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:30.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:30.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:30.496 INFO:teuthology.orchestra.run.smithi114.stderr:no valid command found; 10 closest matches: 2024-09-21T12:27:30.496 INFO:teuthology.orchestra.run.smithi114.stderr:0 2024-09-21T12:27:30.496 INFO:teuthology.orchestra.run.smithi114.stderr:1 2024-09-21T12:27:30.496 INFO:teuthology.orchestra.run.smithi114.stderr:2 2024-09-21T12:27:30.496 INFO:teuthology.orchestra.run.smithi114.stderr:abort 2024-09-21T12:27:30.496 INFO:teuthology.orchestra.run.smithi114.stderr:assert 2024-09-21T12:27:30.496 INFO:teuthology.orchestra.run.smithi114.stderr:bench [] [] [] [] 2024-09-21T12:27:30.496 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs debug_inject_read_zeros 2024-09-21T12:27:30.496 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs files list 2024-09-21T12:27:30.496 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs stats 2024-09-21T12:27:30.497 INFO:teuthology.orchestra.run.smithi114.stderr:bluestore allocator dump block 2024-09-21T12:27:30.497 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: invalid command 2024-09-21T12:27:30.536 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:30.563 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:30.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:30.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:30.664 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:30.699 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:30.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:30.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:30.801 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:30.834 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:30.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:30.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:30.925 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T12:27:30.936 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:30.968 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:30.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:30.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:31.071 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:31.100 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:31.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:31.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:31.144 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:31 smithi183 ceph-mon[33277]: pgmap v351: 178 pgs: 178 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.50k op/s 2024-09-21T12:27:31.144 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:31 smithi183 ceph-mon[33277]: 3.10 scrub starts 2024-09-21T12:27:31.144 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:31 smithi183 ceph-mon[33277]: 3.10 scrub ok 2024-09-21T12:27:31.145 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:31 smithi183 ceph-mon[33277]: 3.9 scrub starts 2024-09-21T12:27:31.145 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:31 smithi183 ceph-mon[33277]: 3.9 scrub ok 2024-09-21T12:27:31.180 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:27:31.181 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-21T12:27:31.181 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-21T12:27:31.181 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-21T12:27:31.181 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:27:31.202 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:31.245 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:31.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:31.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:31.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:31 smithi114 ceph-mon[30324]: pgmap v351: 178 pgs: 178 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.50k op/s 2024-09-21T12:27:31.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:31 smithi114 ceph-mon[30324]: 3.10 scrub starts 2024-09-21T12:27:31.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:31 smithi114 ceph-mon[30324]: 3.10 scrub ok 2024-09-21T12:27:31.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:31 smithi114 ceph-mon[30324]: 3.9 scrub starts 2024-09-21T12:27:31.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:31 smithi114 ceph-mon[30324]: 3.9 scrub ok 2024-09-21T12:27:31.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:31 smithi114 ceph-mon[40867]: pgmap v351: 178 pgs: 178 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.50k op/s 2024-09-21T12:27:31.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:31 smithi114 ceph-mon[40867]: 3.10 scrub starts 2024-09-21T12:27:31.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:31 smithi114 ceph-mon[40867]: 3.10 scrub ok 2024-09-21T12:27:31.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:31 smithi114 ceph-mon[40867]: 3.9 scrub starts 2024-09-21T12:27:31.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:31 smithi114 ceph-mon[40867]: 3.9 scrub ok 2024-09-21T12:27:31.347 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:31.375 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:31.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:31.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:31.477 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:31.518 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:31.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:31.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:31.620 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:31.659 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:31.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:31.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:31.761 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:31.788 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:31.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:31.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:31.890 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:31.921 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:31.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:31.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:32.022 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:32.061 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:32.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:32.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:32.146 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T12:27:32.146 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T12:27:32.147 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.3 'trim stale osdmaps' 2024-09-21T12:27:32.162 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:32.249 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:32.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:32.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:32.307 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:27:32.351 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:32.376 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:32.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:32.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:32.443 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:32 smithi114 ceph-mon[30324]: 4.a scrub starts 2024-09-21T12:27:32.443 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:32 smithi114 ceph-mon[30324]: 4.a scrub ok 2024-09-21T12:27:32.443 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:32 smithi114 ceph-mon[30324]: 6.0 scrub starts 2024-09-21T12:27:32.443 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:32 smithi114 ceph-mon[30324]: 6.0 scrub ok 2024-09-21T12:27:32.443 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:32 smithi114 ceph-mon[30324]: 4.f scrub starts 2024-09-21T12:27:32.443 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:32 smithi114 ceph-mon[30324]: 4.f scrub ok 2024-09-21T12:27:32.443 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:32 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:27:32.443 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:32 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:27:32.443 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:32 smithi114 ceph-mon[40867]: 4.a scrub starts 2024-09-21T12:27:32.444 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:32 smithi114 ceph-mon[40867]: 4.a scrub ok 2024-09-21T12:27:32.444 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:32 smithi114 ceph-mon[40867]: 6.0 scrub starts 2024-09-21T12:27:32.444 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:32 smithi114 ceph-mon[40867]: 6.0 scrub ok 2024-09-21T12:27:32.444 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:32 smithi114 ceph-mon[40867]: 4.f scrub starts 2024-09-21T12:27:32.444 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:32 smithi114 ceph-mon[40867]: 4.f scrub ok 2024-09-21T12:27:32.444 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:32 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:27:32.444 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:32 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:27:32.477 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:32 smithi183 ceph-mon[33277]: 4.a scrub starts 2024-09-21T12:27:32.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:32 smithi183 ceph-mon[33277]: 4.a scrub ok 2024-09-21T12:27:32.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:32 smithi183 ceph-mon[33277]: 6.0 scrub starts 2024-09-21T12:27:32.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:32 smithi183 ceph-mon[33277]: 6.0 scrub ok 2024-09-21T12:27:32.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:32 smithi183 ceph-mon[33277]: 4.f scrub starts 2024-09-21T12:27:32.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:32 smithi183 ceph-mon[33277]: 4.f scrub ok 2024-09-21T12:27:32.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:32 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:27:32.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:32 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:27:32.504 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:32.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:32.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:32.606 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:32.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:32.650 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:32.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:32.751 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:32.764 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T12:27:32.777 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:27:32 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:27:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:27:32.789 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:32.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:32.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:32.891 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:32.921 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:32.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:32.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:33.016 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:27:33.022 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:33.060 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:33.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:33.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:33.123 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:27:33.162 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:33.187 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:33.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:33.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:33.289 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:33.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:33.332 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:33.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:33.400 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:33 smithi183 ceph-mon[33277]: pgmap v352: 178 pgs: 178 active+clean; 2.9 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-21T12:27:33.400 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:33 smithi183 ceph-mon[33277]: 4.1f scrub starts 2024-09-21T12:27:33.400 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:33 smithi183 ceph-mon[33277]: 4.1f scrub ok 2024-09-21T12:27:33.400 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:33 smithi183 ceph-mon[33277]: 3.b scrub starts 2024-09-21T12:27:33.400 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:33 smithi183 ceph-mon[33277]: 3.b scrub ok 2024-09-21T12:27:33.400 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:33 smithi183 ceph-mon[33277]: 5.8 scrub starts 2024-09-21T12:27:33.400 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:33 smithi183 ceph-mon[33277]: 5.8 scrub ok 2024-09-21T12:27:33.433 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:33.469 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:33.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:33.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:33.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:33 smithi114 ceph-mon[30324]: pgmap v352: 178 pgs: 178 active+clean; 2.9 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-21T12:27:33.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:33 smithi114 ceph-mon[30324]: 4.1f scrub starts 2024-09-21T12:27:33.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:33 smithi114 ceph-mon[30324]: 4.1f scrub ok 2024-09-21T12:27:33.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:33 smithi114 ceph-mon[30324]: 3.b scrub starts 2024-09-21T12:27:33.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:33 smithi114 ceph-mon[30324]: 3.b scrub ok 2024-09-21T12:27:33.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:33 smithi114 ceph-mon[30324]: 5.8 scrub starts 2024-09-21T12:27:33.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:33 smithi114 ceph-mon[30324]: 5.8 scrub ok 2024-09-21T12:27:33.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:33 smithi114 ceph-mon[40867]: pgmap v352: 178 pgs: 178 active+clean; 2.9 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s rd, 0 B/s wr, 1.03k op/s 2024-09-21T12:27:33.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:33 smithi114 ceph-mon[40867]: 4.1f scrub starts 2024-09-21T12:27:33.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:33 smithi114 ceph-mon[40867]: 4.1f scrub ok 2024-09-21T12:27:33.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:33 smithi114 ceph-mon[40867]: 3.b scrub starts 2024-09-21T12:27:33.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:33 smithi114 ceph-mon[40867]: 3.b scrub ok 2024-09-21T12:27:33.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:33 smithi114 ceph-mon[40867]: 5.8 scrub starts 2024-09-21T12:27:33.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:33 smithi114 ceph-mon[40867]: 5.8 scrub ok 2024-09-21T12:27:33.570 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:33.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:33.608 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:33.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:33.663 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:27:33.663 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-21T12:27:33.664 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-21T12:27:33.664 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:27:33.664 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.664 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:23884 7.1 7:8f148689:::benchmark_data_smithi114_78432_object23883:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T12:27:33.664 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:39.695870+0000", 2024-09-21T12:27:33.664 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 353.96257194700001, 2024-09-21T12:27:33.664 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.039559285999999999, 2024-09-21T12:27:33.664 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:33.664 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:33.664 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:33.664 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:27:33.664 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:27:33.664 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:27:33.664 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 23884 2024-09-21T12:27:33.665 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.665 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:33.665 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.665 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:33.665 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695870+0000", 2024-09-21T12:27:33.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.665 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.665 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.665 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:33.665 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695870+0000", 2024-09-21T12:27:33.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.665 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:33.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695873+0000", 2024-09-21T12:27:33.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0090000000000002e-06 2024-09-21T12:27:33.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:33.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695927+0000", 2024-09-21T12:27:33.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4941000000000001e-05 2024-09-21T12:27:33.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:33.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695955+0000", 2024-09-21T12:27:33.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7120000000000001e-05 2024-09-21T12:27:33.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:33.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695963+0000", 2024-09-21T12:27:33.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7760000000000003e-06 2024-09-21T12:27:33.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:33.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696022+0000", 2024-09-21T12:27:33.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8941999999999999e-05 2024-09-21T12:27:33.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:33.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696107+0000", 2024-09-21T12:27:33.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4853999999999998e-05 2024-09-21T12:27:33.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,6", 2024-09-21T12:27:33.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696202+0000", 2024-09-21T12:27:33.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4889999999999994e-05 2024-09-21T12:27:33.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:27:33.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.698132+0000", 2024-09-21T12:27:33.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001929998 2024-09-21T12:27:33.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:33.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.698167+0000", 2024-09-21T12:27:33.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5247000000000003e-05 2024-09-21T12:27:33.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:33.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.735322+0000", 2024-09-21T12:27:33.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.037154514 2024-09-21T12:27:33.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:33.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.735382+0000", 2024-09-21T12:27:33.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0371000000000002e-05 2024-09-21T12:27:33.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:33.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.735430+0000", 2024-09-21T12:27:33.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7623999999999997e-05 2024-09-21T12:27:33.670 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.670 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:33.670 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.670 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:23885 7.1 7:81ccbda1:::benchmark_data_smithi114_78432_object23884:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T12:27:33.670 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:39.696116+0000", 2024-09-21T12:27:33.670 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 353.96232611800002, 2024-09-21T12:27:33.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.040219576, 2024-09-21T12:27:33.670 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:33.670 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:33.671 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:33.671 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:27:33.671 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:27:33.671 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:27:33.671 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 23885 2024-09-21T12:27:33.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.671 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:33.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:33.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696116+0000", 2024-09-21T12:27:33.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:33.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696116+0000", 2024-09-21T12:27:33.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:33.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696117+0000", 2024-09-21T12:27:33.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0109999999999999e-06 2024-09-21T12:27:33.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:33.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696178+0000", 2024-09-21T12:27:33.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1060000000000002e-05 2024-09-21T12:27:33.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:33.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696188+0000", 2024-09-21T12:27:33.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0067e-05 2024-09-21T12:27:33.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:33.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696196+0000", 2024-09-21T12:27:33.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0740000000000004e-06 2024-09-21T12:27:33.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:33.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.697126+0000", 2024-09-21T12:27:33.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00093067599999999996 2024-09-21T12:27:33.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:33.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.697270+0000", 2024-09-21T12:27:33.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000143502 2024-09-21T12:27:33.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,6", 2024-09-21T12:27:33.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.697418+0000", 2024-09-21T12:27:33.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000147892 2024-09-21T12:27:33.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:27:33.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.698242+0000", 2024-09-21T12:27:33.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00082411799999999999 2024-09-21T12:27:33.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:33.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.698485+0000", 2024-09-21T12:27:33.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00024341099999999999 2024-09-21T12:27:33.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:33.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.736250+0000", 2024-09-21T12:27:33.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.037765265999999999 2024-09-21T12:27:33.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:33.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.736300+0000", 2024-09-21T12:27:33.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9886000000000003e-05 2024-09-21T12:27:33.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:33.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.736336+0000", 2024-09-21T12:27:33.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5612999999999999e-05 2024-09-21T12:27:33.676 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.676 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:33.676 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.676 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:81802 7.8 e115/112 7:110c0c7a:::benchmark_data_smithi114_78432_object81801:head v 115'5198, mlcod=88'242)", 2024-09-21T12:27:33.676 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.137865+0000", 2024-09-21T12:27:33.676 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 298.520577231, 2024-09-21T12:27:33.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.051160639000000001, 2024-09-21T12:27:33.676 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:33.676 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:33.676 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:33.677 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:33.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:33.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137865+0000", 2024-09-21T12:27:33.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:33.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137865+0000", 2024-09-21T12:27:33.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:33.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137866+0000", 2024-09-21T12:27:33.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5099999999999998e-07 2024-09-21T12:27:33.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:33.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137909+0000", 2024-09-21T12:27:33.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2545e-05 2024-09-21T12:27:33.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:33.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137938+0000", 2024-09-21T12:27:33.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9737999999999999e-05 2024-09-21T12:27:33.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:33.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137961+0000", 2024-09-21T12:27:33.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2320999999999999e-05 2024-09-21T12:27:33.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:33.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.138026+0000", 2024-09-21T12:27:33.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5571e-05 2024-09-21T12:27:33.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:33.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.138111+0000", 2024-09-21T12:27:33.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4682e-05 2024-09-21T12:27:33.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:33.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.188940+0000", 2024-09-21T12:27:33.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.050828867 2024-09-21T12:27:33.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:33.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189026+0000", 2024-09-21T12:27:33.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6063999999999995e-05 2024-09-21T12:27:33.680 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.680 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:33.681 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.681 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:81809 7.8 e115/112 7:1042b23e:::benchmark_data_smithi114_78432_object81808:head v 115'5199, mlcod=88'242)", 2024-09-21T12:27:33.681 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.140023+0000", 2024-09-21T12:27:33.681 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 298.51841966000001, 2024-09-21T12:27:33.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.049109005999999997, 2024-09-21T12:27:33.681 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:33.681 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:33.681 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:33.681 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:33.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:33.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140023+0000", 2024-09-21T12:27:33.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:33.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140023+0000", 2024-09-21T12:27:33.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:33.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140023+0000", 2024-09-21T12:27:33.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6700000000000003e-07 2024-09-21T12:27:33.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:33.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140136+0000", 2024-09-21T12:27:33.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000112623 2024-09-21T12:27:33.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:33.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140146+0000", 2024-09-21T12:27:33.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0098999999999999e-05 2024-09-21T12:27:33.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:33.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140161+0000", 2024-09-21T12:27:33.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5288e-05 2024-09-21T12:27:33.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:33.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146087+0000", 2024-09-21T12:27:33.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0059259070000000002 2024-09-21T12:27:33.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:33.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146109+0000", 2024-09-21T12:27:33.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2045999999999999e-05 2024-09-21T12:27:33.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:33.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189077+0000", 2024-09-21T12:27:33.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.042967639000000002 2024-09-21T12:27:33.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:33.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189132+0000", 2024-09-21T12:27:33.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4737000000000001e-05 2024-09-21T12:27:33.685 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.685 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:33.685 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.685 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:81811 7.8 e115/112 7:10044f1a:::benchmark_data_smithi114_78432_object81810:head v 115'5200, mlcod=88'242)", 2024-09-21T12:27:33.685 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.146782+0000", 2024-09-21T12:27:33.685 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 298.51166017499997, 2024-09-21T12:27:33.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.042406829, 2024-09-21T12:27:33.686 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:33.686 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:33.686 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:33.686 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:33.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:33.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146782+0000", 2024-09-21T12:27:33.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:33.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146782+0000", 2024-09-21T12:27:33.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:33.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146783+0000", 2024-09-21T12:27:33.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2999999999999999e-07 2024-09-21T12:27:33.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:33.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146826+0000", 2024-09-21T12:27:33.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2781000000000003e-05 2024-09-21T12:27:33.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:33.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146837+0000", 2024-09-21T12:27:33.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1126e-05 2024-09-21T12:27:33.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:33.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146843+0000", 2024-09-21T12:27:33.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4710000000000004e-06 2024-09-21T12:27:33.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:33.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.152740+0000", 2024-09-21T12:27:33.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0058961209999999998 2024-09-21T12:27:33.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:33.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.152763+0000", 2024-09-21T12:27:33.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3101000000000001e-05 2024-09-21T12:27:33.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:33.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189134+0000", 2024-09-21T12:27:33.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.036371542999999999 2024-09-21T12:27:33.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:33.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189189+0000", 2024-09-21T12:27:33.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4855999999999998e-05 2024-09-21T12:27:33.689 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.689 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:33.689 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.690 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:81821 7.8 e115/112 7:1c79dadc:::benchmark_data_smithi114_78432_object81820:head v 115'5201, mlcod=88'242)", 2024-09-21T12:27:33.690 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.148399+0000", 2024-09-21T12:27:33.690 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 298.51004310000002, 2024-09-21T12:27:33.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.040843988999999997, 2024-09-21T12:27:33.690 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:33.690 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:33.690 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:33.690 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:33.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:33.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148399+0000", 2024-09-21T12:27:33.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.691 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:33.691 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148399+0000", 2024-09-21T12:27:33.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:33.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148400+0000", 2024-09-21T12:27:33.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.049e-06 2024-09-21T12:27:33.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:33.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148502+0000", 2024-09-21T12:27:33.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000101542 2024-09-21T12:27:33.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:33.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148512+0000", 2024-09-21T12:27:33.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0386e-05 2024-09-21T12:27:33.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:33.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148519+0000", 2024-09-21T12:27:33.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3589999999999996e-06 2024-09-21T12:27:33.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:33.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.158203+0000", 2024-09-21T12:27:33.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0096844710000000001 2024-09-21T12:27:33.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:33.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.158229+0000", 2024-09-21T12:27:33.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6078999999999999e-05 2024-09-21T12:27:33.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:33.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189192+0000", 2024-09-21T12:27:33.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.030962455 2024-09-21T12:27:33.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:33.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189243+0000", 2024-09-21T12:27:33.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1647999999999997e-05 2024-09-21T12:27:33.694 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.694 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:33.695 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.695 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82430 7.8 e115/112 7:1762e17c:::benchmark_data_smithi114_78432_object82429:head v 115'5247, mlcod=88'255)", 2024-09-21T12:27:33.695 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.787623+0000", 2024-09-21T12:27:33.695 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 297.87081931199998, 2024-09-21T12:27:33.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.048511400000000003, 2024-09-21T12:27:33.695 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:33.695 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:33.695 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:33.695 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:33.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:33.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787623+0000", 2024-09-21T12:27:33.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:33.696 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787623+0000", 2024-09-21T12:27:33.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:33.696 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787625+0000", 2024-09-21T12:27:33.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5319999999999999e-06 2024-09-21T12:27:33.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:33.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787667+0000", 2024-09-21T12:27:33.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1876999999999999e-05 2024-09-21T12:27:33.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:33.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787688+0000", 2024-09-21T12:27:33.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1046000000000001e-05 2024-09-21T12:27:33.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:33.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787697+0000", 2024-09-21T12:27:33.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4930000000000004e-06 2024-09-21T12:27:33.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:33.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.788378+0000", 2024-09-21T12:27:33.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00068115999999999999 2024-09-21T12:27:33.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:33.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.788427+0000", 2024-09-21T12:27:33.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8553000000000001e-05 2024-09-21T12:27:33.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:33.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.836011+0000", 2024-09-21T12:27:33.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.047583978999999998 2024-09-21T12:27:33.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:33.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.836135+0000", 2024-09-21T12:27:33.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012375999999999999 2024-09-21T12:27:33.699 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.699 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:33.699 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.699 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDPGPush(7.8 115/112 [PushOp(7:10df046b:::benchmark_data_smithi114_78432_object3988:head, version: 88'257, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:10df046b:::benchmark_data_smithi114_78432_object3988:head@88'257, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T12:27:33.699 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.787722+0000", 2024-09-21T12:27:33.699 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 297.870720237, 2024-09-21T12:27:33.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.074165650999999999, 2024-09-21T12:27:33.699 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:33.699 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:33.699 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:27:33.699 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:33.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:33.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787722+0000", 2024-09-21T12:27:33.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:33.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787722+0000", 2024-09-21T12:27:33.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:33.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787723+0000", 2024-09-21T12:27:33.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0499999999999999e-07 2024-09-21T12:27:33.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:33.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787779+0000", 2024-09-21T12:27:33.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6053e-05 2024-09-21T12:27:33.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:33.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787803+0000", 2024-09-21T12:27:33.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3980000000000001e-05 2024-09-21T12:27:33.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:33.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787832+0000", 2024-09-21T12:27:33.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.942e-05 2024-09-21T12:27:33.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:33.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.859697+0000", 2024-09-21T12:27:33.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.071864649000000003 2024-09-21T12:27:33.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:33.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.859710+0000", 2024-09-21T12:27:33.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3740000000000001e-05 2024-09-21T12:27:33.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:33.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.861888+0000", 2024-09-21T12:27:33.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002177404 2024-09-21T12:27:33.703 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.703 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:33.703 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.703 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82442 7.8 e115/112 7:1035456a:::benchmark_data_smithi114_78432_object82441:head v 115'5248, mlcod=88'255)", 2024-09-21T12:27:33.703 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.791117+0000", 2024-09-21T12:27:33.703 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 297.86732533399999, 2024-09-21T12:27:33.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.051248759999999997, 2024-09-21T12:27:33.703 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:33.703 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:33.703 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:33.703 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:33.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:33.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791117+0000", 2024-09-21T12:27:33.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:33.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791117+0000", 2024-09-21T12:27:33.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:33.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791118+0000", 2024-09-21T12:27:33.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9999999999999998e-07 2024-09-21T12:27:33.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:33.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791158+0000", 2024-09-21T12:27:33.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0025999999999999e-05 2024-09-21T12:27:33.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:33.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791167+0000", 2024-09-21T12:27:33.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9910000000000008e-06 2024-09-21T12:27:33.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:33.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791176+0000", 2024-09-21T12:27:33.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0329999999999997e-06 2024-09-21T12:27:33.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:33.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.794825+0000", 2024-09-21T12:27:33.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0036494700000000001 2024-09-21T12:27:33.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:33.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.794878+0000", 2024-09-21T12:27:33.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3292999999999999e-05 2024-09-21T12:27:33.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:33.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842289+0000", 2024-09-21T12:27:33.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.047410256999999997 2024-09-21T12:27:33.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:33.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842366+0000", 2024-09-21T12:27:33.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7189999999999998e-05 2024-09-21T12:27:33.707 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.707 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:33.707 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.708 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82447 7.8 e115/112 7:174332ca:::benchmark_data_smithi114_78432_object82446:head v 115'5249, mlcod=88'255)", 2024-09-21T12:27:33.708 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.793874+0000", 2024-09-21T12:27:33.708 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 297.864568883, 2024-09-21T12:27:33.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.048541015999999999, 2024-09-21T12:27:33.708 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:33.708 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:33.708 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:33.708 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:33.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:33.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793874+0000", 2024-09-21T12:27:33.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:33.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793874+0000", 2024-09-21T12:27:33.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:33.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793875+0000", 2024-09-21T12:27:33.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0890000000000001e-06 2024-09-21T12:27:33.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:33.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793910+0000", 2024-09-21T12:27:33.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5080000000000003e-05 2024-09-21T12:27:33.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:33.710 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793921+0000", 2024-09-21T12:27:33.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1284000000000001e-05 2024-09-21T12:27:33.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.710 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:33.710 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793930+0000", 2024-09-21T12:27:33.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5429999999999994e-06 2024-09-21T12:27:33.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.710 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:33.710 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.801012+0000", 2024-09-21T12:27:33.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0070825219999999999 2024-09-21T12:27:33.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:33.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.801039+0000", 2024-09-21T12:27:33.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7248999999999999e-05 2024-09-21T12:27:33.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:33.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842368+0000", 2024-09-21T12:27:33.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.041328831000000003 2024-09-21T12:27:33.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:33.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842415+0000", 2024-09-21T12:27:33.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6418000000000002e-05 2024-09-21T12:27:33.712 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.712 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:33.712 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.712 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82448 7.8 e115/112 7:11700dea:::benchmark_data_smithi114_78432_object82447:head v 115'5250, mlcod=88'255)", 2024-09-21T12:27:33.712 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.796531+0000", 2024-09-21T12:27:33.712 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 297.86191113899997, 2024-09-21T12:27:33.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.045928499999999997, 2024-09-21T12:27:33.712 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:33.712 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:33.712 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:33.712 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:33.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:33.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796531+0000", 2024-09-21T12:27:33.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:33.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796531+0000", 2024-09-21T12:27:33.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:33.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796532+0000", 2024-09-21T12:27:33.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7199999999999999e-07 2024-09-21T12:27:33.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:33.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796647+0000", 2024-09-21T12:27:33.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000115097 2024-09-21T12:27:33.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:33.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796660+0000", 2024-09-21T12:27:33.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3271999999999999e-05 2024-09-21T12:27:33.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:33.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796666+0000", 2024-09-21T12:27:33.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0170000000000002e-06 2024-09-21T12:27:33.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:33.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.809350+0000", 2024-09-21T12:27:33.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.012683379 2024-09-21T12:27:33.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:33.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.809374+0000", 2024-09-21T12:27:33.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4417e-05 2024-09-21T12:27:33.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:33.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842417+0000", 2024-09-21T12:27:33.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.033042646000000002 2024-09-21T12:27:33.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.716 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.716 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:33.716 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842460+0000", 2024-09-21T12:27:33.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3099999999999997e-05 2024-09-21T12:27:33.716 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.716 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:33.716 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.716 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.716 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82473 7.8 e115/112 7:15016882:::benchmark_data_smithi114_78432_object82472:head v 115'5254, mlcod=88'255)", 2024-09-21T12:27:33.716 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.818965+0000", 2024-09-21T12:27:33.716 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 297.83947754600001, 2024-09-21T12:27:33.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.045024287000000003, 2024-09-21T12:27:33.716 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:33.717 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:33.717 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:33.717 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:33.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:33.717 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.818965+0000", 2024-09-21T12:27:33.717 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.717 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:33.717 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.818965+0000", 2024-09-21T12:27:33.717 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.717 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:33.717 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.818966+0000", 2024-09-21T12:27:33.718 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9199999999999999e-07 2024-09-21T12:27:33.718 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.718 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.718 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:33.718 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.819001+0000", 2024-09-21T12:27:33.718 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5117999999999998e-05 2024-09-21T12:27:33.718 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.718 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.718 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:33.718 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.819014+0000", 2024-09-21T12:27:33.718 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2869e-05 2024-09-21T12:27:33.718 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.718 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.718 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:33.718 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.819020+0000", 2024-09-21T12:27:33.719 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6959999999999999e-06 2024-09-21T12:27:33.719 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.719 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.719 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:33.719 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.836164+0000", 2024-09-21T12:27:33.719 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.017143617 2024-09-21T12:27:33.719 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.719 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.719 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:33.719 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.836188+0000", 2024-09-21T12:27:33.719 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4074e-05 2024-09-21T12:27:33.719 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.719 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.719 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:33.719 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.863879+0000", 2024-09-21T12:27:33.720 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.027690392000000001 2024-09-21T12:27:33.720 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.720 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.720 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:33.720 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.863989+0000", 2024-09-21T12:27:33.720 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000110629 2024-09-21T12:27:33.720 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.720 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:33.720 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.720 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.720 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.720 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82539 7.8 e115/112 7:1cd4c60d:::benchmark_data_smithi114_78432_object82538:head v 115'5258, mlcod=88'256)", 2024-09-21T12:27:33.720 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.889845+0000", 2024-09-21T12:27:33.720 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 297.76859714300002, 2024-09-21T12:27:33.720 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.046325071000000002, 2024-09-21T12:27:33.721 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:33.721 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:33.721 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:33.721 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:33.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.721 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:33.721 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889845+0000", 2024-09-21T12:27:33.721 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.721 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:33.721 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889845+0000", 2024-09-21T12:27:33.721 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.721 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:33.722 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889846+0000", 2024-09-21T12:27:33.722 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1800000000000004e-07 2024-09-21T12:27:33.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.722 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.722 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:33.722 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889878+0000", 2024-09-21T12:27:33.722 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1619999999999999e-05 2024-09-21T12:27:33.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.722 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.722 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:33.722 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889893+0000", 2024-09-21T12:27:33.722 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4819e-05 2024-09-21T12:27:33.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.722 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.723 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:33.723 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889900+0000", 2024-09-21T12:27:33.723 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8510000000000001e-06 2024-09-21T12:27:33.723 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.723 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.723 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:33.723 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.898101+0000", 2024-09-21T12:27:33.723 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0082018119999999993 2024-09-21T12:27:33.723 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.723 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.723 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:33.723 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.898164+0000", 2024-09-21T12:27:33.723 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2756999999999996e-05 2024-09-21T12:27:33.724 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.724 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.724 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:33.724 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.936000+0000", 2024-09-21T12:27:33.724 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.037835720000000003 2024-09-21T12:27:33.724 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.724 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.724 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:33.724 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.936170+0000", 2024-09-21T12:27:33.724 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000170574 2024-09-21T12:27:33.724 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.724 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:33.724 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.724 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.724 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.725 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82545 7.8 e115/112 7:1ddea6d1:::benchmark_data_smithi114_78432_object82544:head v 115'5259, mlcod=88'256)", 2024-09-21T12:27:33.725 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.897417+0000", 2024-09-21T12:27:33.725 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 297.76102560599998, 2024-09-21T12:27:33.725 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.038827027, 2024-09-21T12:27:33.725 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:33.725 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:33.725 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:33.725 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:33.725 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.725 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:33.725 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897417+0000", 2024-09-21T12:27:33.725 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.725 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.725 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.725 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:33.726 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897417+0000", 2024-09-21T12:27:33.726 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.726 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.726 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.726 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:33.726 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897418+0000", 2024-09-21T12:27:33.726 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3500000000000005e-07 2024-09-21T12:27:33.726 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.726 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.726 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:33.726 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897462+0000", 2024-09-21T12:27:33.726 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4036999999999999e-05 2024-09-21T12:27:33.726 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.726 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.727 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:33.727 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897472+0000", 2024-09-21T12:27:33.727 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0669999999999999e-05 2024-09-21T12:27:33.727 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.727 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.727 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:33.727 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897479+0000", 2024-09-21T12:27:33.727 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9190000000000002e-06 2024-09-21T12:27:33.727 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.727 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.727 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:33.727 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.905524+0000", 2024-09-21T12:27:33.727 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0080444899999999996 2024-09-21T12:27:33.727 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.727 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.728 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:33.728 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.905597+0000", 2024-09-21T12:27:33.728 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3539000000000002e-05 2024-09-21T12:27:33.728 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.728 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.728 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:33.728 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.936176+0000", 2024-09-21T12:27:33.728 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.030578893999999999 2024-09-21T12:27:33.728 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.728 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.728 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:33.728 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.936244+0000", 2024-09-21T12:27:33.728 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7543e-05 2024-09-21T12:27:33.728 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.729 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:33.729 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.729 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.729 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.729 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:83346 7.8 e115/112 7:1517ab05:::benchmark_data_smithi114_78432_object83345:head v 115'5319, mlcod=88'282)", 2024-09-21T12:27:33.729 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:36.700133+0000", 2024-09-21T12:27:33.729 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 296.95830970499998, 2024-09-21T12:27:33.729 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.044693733999999999, 2024-09-21T12:27:33.729 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:33.729 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:33.729 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:33.729 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:33.729 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.729 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:33.729 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700133+0000", 2024-09-21T12:27:33.730 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.730 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:33.730 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700133+0000", 2024-09-21T12:27:33.730 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.730 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:33.730 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700133+0000", 2024-09-21T12:27:33.730 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4300000000000003e-07 2024-09-21T12:27:33.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.730 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:33.731 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700182+0000", 2024-09-21T12:27:33.731 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8433000000000002e-05 2024-09-21T12:27:33.731 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.731 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.731 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:33.731 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700192+0000", 2024-09-21T12:27:33.731 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0382e-05 2024-09-21T12:27:33.731 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.731 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.731 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:33.731 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:33.732 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700201+0000", 2024-09-21T12:27:33.732 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9819999999999997e-06 2024-09-21T12:27:33.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.733 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.733 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:33.733 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.703063+0000", 2024-09-21T12:27:33.733 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002862121 2024-09-21T12:27:33.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.733 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.733 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:33.733 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.703087+0000", 2024-09-21T12:27:33.733 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3730000000000001e-05 2024-09-21T12:27:33.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.733 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.733 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:33.734 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.744677+0000", 2024-09-21T12:27:33.734 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.041589785999999997 2024-09-21T12:27:33.734 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.734 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.734 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:33.734 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.744826+0000", 2024-09-21T12:27:33.734 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014965700000000001 2024-09-21T12:27:33.734 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.734 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:33.734 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.734 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.734 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.734 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:87464 7.8 e117/112 7:17a0a89d:::benchmark_data_smithi114_78432_object87463:head v 117'5593, mlcod=88'417)", 2024-09-21T12:27:33.734 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:41.463661+0000", 2024-09-21T12:27:33.734 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 292.194781772, 2024-09-21T12:27:33.734 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.067076070000000002, 2024-09-21T12:27:33.735 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:33.735 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:33.735 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:33.735 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:33.735 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.735 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:33.735 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463661+0000", 2024-09-21T12:27:33.735 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.735 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.735 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.735 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:33.735 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463661+0000", 2024-09-21T12:27:33.735 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.735 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.736 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.736 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:33.736 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463661+0000", 2024-09-21T12:27:33.736 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2799999999999995e-07 2024-09-21T12:27:33.736 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.736 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.736 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:33.736 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463845+0000", 2024-09-21T12:27:33.736 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000183692 2024-09-21T12:27:33.736 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.736 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.736 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:33.736 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463854+0000", 2024-09-21T12:27:33.736 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0750000000000004e-06 2024-09-21T12:27:33.736 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.737 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.737 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:33.737 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463860+0000", 2024-09-21T12:27:33.737 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.015e-06 2024-09-21T12:27:33.737 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.737 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.737 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:33.737 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.480218+0000", 2024-09-21T12:27:33.737 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.016357289000000001 2024-09-21T12:27:33.737 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.737 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.737 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:33.737 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.480239+0000", 2024-09-21T12:27:33.737 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1691999999999998e-05 2024-09-21T12:27:33.738 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.738 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:33.738 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.530695+0000", 2024-09-21T12:27:33.738 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.050456050000000002 2024-09-21T12:27:33.738 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.738 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:33.738 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.530737+0000", 2024-09-21T12:27:33.738 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1428999999999999e-05 2024-09-21T12:27:33.738 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.738 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:33.738 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.738 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.738 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:87470 7.8 e117/112 7:1f3b224f:::benchmark_data_smithi114_78432_object87469:head v 117'5594, mlcod=88'417)", 2024-09-21T12:27:33.739 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:41.472523+0000", 2024-09-21T12:27:33.739 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 292.18591977400001, 2024-09-21T12:27:33.739 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.06343451, 2024-09-21T12:27:33.739 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:33.739 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:33.739 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:33.739 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:33.739 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.739 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:33.739 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472523+0000", 2024-09-21T12:27:33.739 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.739 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.739 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.739 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:33.739 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472523+0000", 2024-09-21T12:27:33.740 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.740 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.740 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.740 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:33.740 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472524+0000", 2024-09-21T12:27:33.740 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7399999999999991e-07 2024-09-21T12:27:33.740 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.740 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.740 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:33.740 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472596+0000", 2024-09-21T12:27:33.740 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2152999999999998e-05 2024-09-21T12:27:33.740 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.740 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.740 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:33.741 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472606+0000", 2024-09-21T12:27:33.741 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0485e-05 2024-09-21T12:27:33.741 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.741 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.741 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:33.741 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472613+0000", 2024-09-21T12:27:33.741 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5590000000000002e-06 2024-09-21T12:27:33.741 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.741 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.741 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:33.741 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.484029+0000", 2024-09-21T12:27:33.741 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.01141587 2024-09-21T12:27:33.741 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.742 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.742 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:33.742 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.484053+0000", 2024-09-21T12:27:33.742 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4559999999999999e-05 2024-09-21T12:27:33.742 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.742 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.742 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:33.742 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.535915+0000", 2024-09-21T12:27:33.742 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.051861431 2024-09-21T12:27:33.742 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.742 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.742 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:33.742 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.535957+0000", 2024-09-21T12:27:33.742 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2478000000000002e-05 2024-09-21T12:27:33.743 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.743 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:33.743 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.743 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.743 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.743 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:87486 7.8 e117/112 7:14e05f7b:::benchmark_data_smithi114_78432_object87485:head v 117'5595, mlcod=88'417)", 2024-09-21T12:27:33.743 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:41.475017+0000", 2024-09-21T12:27:33.743 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 292.18342543799997, 2024-09-21T12:27:33.743 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.060970067000000003, 2024-09-21T12:27:33.743 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:33.743 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:33.743 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:33.743 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:33.743 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.744 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:33.744 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475017+0000", 2024-09-21T12:27:33.744 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.744 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.744 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.744 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:33.744 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475017+0000", 2024-09-21T12:27:33.744 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.744 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.744 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.744 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:33.744 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475018+0000", 2024-09-21T12:27:33.744 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5799999999999999e-07 2024-09-21T12:27:33.744 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.744 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.745 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:33.745 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475150+0000", 2024-09-21T12:27:33.745 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013201700000000001 2024-09-21T12:27:33.745 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.745 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.745 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:33.745 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475159+0000", 2024-09-21T12:27:33.745 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5840000000000001e-06 2024-09-21T12:27:33.745 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.745 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.745 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:33.745 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475166+0000", 2024-09-21T12:27:33.745 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3589999999999996e-06 2024-09-21T12:27:33.745 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.745 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.746 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:33.746 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.525196+0000", 2024-09-21T12:27:33.746 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.050030507000000002 2024-09-21T12:27:33.746 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.746 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.746 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:33.746 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.525214+0000", 2024-09-21T12:27:33.746 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7884e-05 2024-09-21T12:27:33.746 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.746 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.746 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:33.746 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.535960+0000", 2024-09-21T12:27:33.746 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.010746143999999999 2024-09-21T12:27:33.746 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.747 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.747 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:33.747 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.535987+0000", 2024-09-21T12:27:33.747 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6914e-05 2024-09-21T12:27:33.747 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.747 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:33.747 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.747 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.747 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.747 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDPGPush(7.8 117/112 [PushOp(7:1de1c2d4:::benchmark_data_smithi114_78432_object6336:head, version: 88'418, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:1de1c2d4:::benchmark_data_smithi114_78432_object6336:head@88'418, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T12:27:33.747 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:41.485571+0000", 2024-09-21T12:27:33.747 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 292.17287154299999, 2024-09-21T12:27:33.747 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.046712901000000001, 2024-09-21T12:27:33.748 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:33.748 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:33.748 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:27:33.748 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:33.748 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.748 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:33.748 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485571+0000", 2024-09-21T12:27:33.748 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.748 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.748 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.748 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:33.748 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485571+0000", 2024-09-21T12:27:33.748 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.749 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.749 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.749 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:33.749 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485572+0000", 2024-09-21T12:27:33.749 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7299999999999995e-07 2024-09-21T12:27:33.749 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.749 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.749 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:33.749 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485643+0000", 2024-09-21T12:27:33.749 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0976999999999995e-05 2024-09-21T12:27:33.749 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.749 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.749 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:33.749 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485655+0000", 2024-09-21T12:27:33.749 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2904e-05 2024-09-21T12:27:33.750 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.750 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.750 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:33.750 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485665+0000", 2024-09-21T12:27:33.750 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6709999999999994e-06 2024-09-21T12:27:33.750 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.750 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.750 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:33.750 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.530794+0000", 2024-09-21T12:27:33.750 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.045129295 2024-09-21T12:27:33.750 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.750 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.750 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:33.750 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.530802+0000", 2024-09-21T12:27:33.751 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5059999999999998e-06 2024-09-21T12:27:33.751 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.751 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.751 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:33.751 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.532284+0000", 2024-09-21T12:27:33.751 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014818749999999999 2024-09-21T12:27:33.751 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.751 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:33.751 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.751 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.751 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.751 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:87497 7.8 e117/112 7:19475f57:::benchmark_data_smithi114_78432_object87496:head v 117'5596, mlcod=88'417)", 2024-09-21T12:27:33.751 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:41.494916+0000", 2024-09-21T12:27:33.751 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 292.163526064, 2024-09-21T12:27:33.752 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.041122084000000003, 2024-09-21T12:27:33.752 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:33.752 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:33.752 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:33.752 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:33.752 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.752 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:33.752 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.494916+0000", 2024-09-21T12:27:33.752 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.752 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.752 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.752 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:33.752 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.494916+0000", 2024-09-21T12:27:33.752 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:33.752 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.753 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.753 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:33.753 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.494918+0000", 2024-09-21T12:27:33.753 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.719e-06 2024-09-21T12:27:33.753 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.753 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.753 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:33.753 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.494975+0000", 2024-09-21T12:27:33.753 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6981999999999998e-05 2024-09-21T12:27:33.753 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.753 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.753 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:33.753 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.525059+0000", 2024-09-21T12:27:33.753 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.030083526999999999 2024-09-21T12:27:33.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.754 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.754 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:33.754 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.525071+0000", 2024-09-21T12:27:33.754 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2438e-05 2024-09-21T12:27:33.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.754 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.754 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:33.754 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.532312+0000", 2024-09-21T12:27:33.754 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0072404909999999999 2024-09-21T12:27:33.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.754 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.754 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:33.754 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.532336+0000", 2024-09-21T12:27:33.755 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4090000000000001e-05 2024-09-21T12:27:33.755 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.755 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.755 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:33.755 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.536013+0000", 2024-09-21T12:27:33.755 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003677825 2024-09-21T12:27:33.755 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:33.755 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:33.755 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:33.755 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.536038+0000", 2024-09-21T12:27:33.755 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5012e-05 2024-09-21T12:27:33.755 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.755 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:33.755 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.756 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:33.756 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:33.756 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:27:33.787 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:33.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:33.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:33.889 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:33.917 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:33.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:33.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:34.019 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:34.052 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:34.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:34.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:34.149 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:34 smithi114 ceph-mon[40867]: 4.18 scrub starts 2024-09-21T12:27:34.149 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:34 smithi114 ceph-mon[30324]: 4.18 scrub starts 2024-09-21T12:27:34.153 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:34.186 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:34.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:34.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:34.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:34 smithi114 ceph-mon[40867]: 4.18 scrub ok 2024-09-21T12:27:34.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:34 smithi114 ceph-mon[40867]: 6.5 scrub starts 2024-09-21T12:27:34.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:34 smithi114 ceph-mon[40867]: 6.5 scrub ok 2024-09-21T12:27:34.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:34 smithi114 ceph-mon[40867]: 5.6 scrub starts 2024-09-21T12:27:34.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:34 smithi114 ceph-mon[40867]: 5.6 scrub ok 2024-09-21T12:27:34.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:34 smithi114 ceph-mon[30324]: 4.18 scrub ok 2024-09-21T12:27:34.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:34 smithi114 ceph-mon[30324]: 6.5 scrub starts 2024-09-21T12:27:34.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:34 smithi114 ceph-mon[30324]: 6.5 scrub ok 2024-09-21T12:27:34.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:34 smithi114 ceph-mon[30324]: 5.6 scrub starts 2024-09-21T12:27:34.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:34 smithi114 ceph-mon[30324]: 5.6 scrub ok 2024-09-21T12:27:34.288 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:34.330 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:34.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:34.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:34.432 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:34.453 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:34.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:34.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:34.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:34 smithi183 ceph-mon[33277]: 4.18 scrub starts 2024-09-21T12:27:34.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:34 smithi183 ceph-mon[33277]: 4.18 scrub ok 2024-09-21T12:27:34.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:34 smithi183 ceph-mon[33277]: 6.5 scrub starts 2024-09-21T12:27:34.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:34 smithi183 ceph-mon[33277]: 6.5 scrub ok 2024-09-21T12:27:34.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:34 smithi183 ceph-mon[33277]: 5.6 scrub starts 2024-09-21T12:27:34.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:34 smithi183 ceph-mon[33277]: 5.6 scrub ok 2024-09-21T12:27:34.549 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T12:27:34.554 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:34.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:34.588 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:34.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:34.688 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:34.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:34.718 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:34.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:34.818 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:34.851 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:34.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:34.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:34.902 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:27:34.953 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:34.984 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:34.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:34.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:35.086 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:35.125 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:35.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:35.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:35.226 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:35.238 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-21T12:27:35.264 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:35.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:35.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:35.341 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:35 smithi183 ceph-mon[33277]: pgmap v353: 178 pgs: 178 active+clean; 2.5 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s rd, 0 B/s wr, 1.74k op/s 2024-09-21T12:27:35.341 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:35 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3917116054' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:27:35.341 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:35 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:27:35.342 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:35 smithi183 ceph-mon[33277]: 4.c scrub starts 2024-09-21T12:27:35.342 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:35 smithi183 ceph-mon[33277]: 4.c scrub ok 2024-09-21T12:27:35.366 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:35.389 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:35.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:35.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:35.490 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:35.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:35 smithi114 ceph-mon[40867]: pgmap v353: 178 pgs: 178 active+clean; 2.5 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s rd, 0 B/s wr, 1.74k op/s 2024-09-21T12:27:35.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:35 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3917116054' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:27:35.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:35 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:27:35.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:35 smithi114 ceph-mon[40867]: 4.c scrub starts 2024-09-21T12:27:35.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:35 smithi114 ceph-mon[40867]: 4.c scrub ok 2024-09-21T12:27:35.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:35 smithi114 ceph-mon[30324]: pgmap v353: 178 pgs: 178 active+clean; 2.5 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s rd, 0 B/s wr, 1.74k op/s 2024-09-21T12:27:35.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:35 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3917116054' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:27:35.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:35 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:27:35.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:35 smithi114 ceph-mon[30324]: 4.c scrub starts 2024-09-21T12:27:35.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:35 smithi114 ceph-mon[30324]: 4.c scrub ok 2024-09-21T12:27:35.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:35.526 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:35.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:35.627 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:35.648 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:35.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:35.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:35.676 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:27:35.676 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:27:35.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.676 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.25022.0:223840 8.a e222/160 8:5e142a4d:::benchmark_data_smithi114_104936_object93424:head v 222'13854, mlcod=222'13852)", 2024-09-21T12:27:35.676 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:27:35.441909+0000", 2024-09-21T12:27:35.676 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.004915666, 2024-09-21T12:27:35.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0049435820000000002, 2024-09-21T12:27:35.676 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:35.676 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:35.676 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:27:35.677 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:35.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:35.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.441909+0000", 2024-09-21T12:27:35.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:35.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:35.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.441909+0000", 2024-09-21T12:27:35.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:35.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:35.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.441910+0000", 2024-09-21T12:27:35.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2699999999999999e-06 2024-09-21T12:27:35.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:35.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.441916+0000", 2024-09-21T12:27:35.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4020000000000001e-06 2024-09-21T12:27:35.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:35.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.441932+0000", 2024-09-21T12:27:35.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6217000000000001e-05 2024-09-21T12:27:35.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:35.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.441940+0000", 2024-09-21T12:27:35.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1170000000000004e-06 2024-09-21T12:27:35.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:35.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.442011+0000", 2024-09-21T12:27:35.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0618999999999996e-05 2024-09-21T12:27:35.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:35.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.442035+0000", 2024-09-21T12:27:35.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.444e-05 2024-09-21T12:27:35.679 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:35.679 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:35.680 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:35.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.680 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.25022.0:223842 8.a e222/160 8:5edbad97:::benchmark_data_smithi114_104936_object93426:head v 222'13855, mlcod=222'13852)", 2024-09-21T12:27:35.680 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:27:35.442790+0000", 2024-09-21T12:27:35.680 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0040347610000000004, 2024-09-21T12:27:35.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0041174580000000001, 2024-09-21T12:27:35.680 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:35.680 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:35.680 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:27:35.680 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:35.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:35.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.442790+0000", 2024-09-21T12:27:35.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:35.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:35.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.442790+0000", 2024-09-21T12:27:35.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:35.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:35.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.442790+0000", 2024-09-21T12:27:35.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0200000000000002e-07 2024-09-21T12:27:35.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:35.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.442795+0000", 2024-09-21T12:27:35.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4020000000000004e-06 2024-09-21T12:27:35.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:35.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.442804+0000", 2024-09-21T12:27:35.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0089999999999996e-06 2024-09-21T12:27:35.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:35.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.442808+0000", 2024-09-21T12:27:35.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6489999999999999e-06 2024-09-21T12:27:35.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:35.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.444021+0000", 2024-09-21T12:27:35.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012128919999999999 2024-09-21T12:27:35.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:35.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.444040+0000", 2024-09-21T12:27:35.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8797999999999999e-05 2024-09-21T12:27:35.683 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:35.683 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:35.683 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:35.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.684 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.25022.0:223844 8.2 e222/160 8:49617762:::benchmark_data_smithi114_104936_object93428:head v 222'13847, mlcod=222'13845)", 2024-09-21T12:27:35.684 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:27:35.444294+0000", 2024-09-21T12:27:35.684 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0025310179999999999, 2024-09-21T12:27:35.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002646322, 2024-09-21T12:27:35.684 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:35.684 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:35.684 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:27:35.684 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:35.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:35.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.444294+0000", 2024-09-21T12:27:35.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:35.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:35.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.444294+0000", 2024-09-21T12:27:35.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:35.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:35.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.444294+0000", 2024-09-21T12:27:35.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4799999999999998e-07 2024-09-21T12:27:35.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:35.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.444299+0000", 2024-09-21T12:27:35.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2310000000000004e-06 2024-09-21T12:27:35.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:35.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.444309+0000", 2024-09-21T12:27:35.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9539999999999999e-06 2024-09-21T12:27:35.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:35.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.444315+0000", 2024-09-21T12:27:35.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5010000000000004e-06 2024-09-21T12:27:35.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:35.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.444365+0000", 2024-09-21T12:27:35.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0095000000000001e-05 2024-09-21T12:27:35.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:35.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.444381+0000", 2024-09-21T12:27:35.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5818999999999999e-05 2024-09-21T12:27:35.687 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:35.687 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:35.687 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:35.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.687 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.25022.0:223847 8.0 e222/160 8:071348df:::benchmark_data_smithi114_104936_object93431:head v 222'14102, mlcod=222'14100)", 2024-09-21T12:27:35.687 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:27:35.445101+0000", 2024-09-21T12:27:35.688 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.001723327, 2024-09-21T12:27:35.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001863272, 2024-09-21T12:27:35.688 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:35.688 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:35.688 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:27:35.688 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:35.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:35.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.445101+0000", 2024-09-21T12:27:35.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:35.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:35.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.445101+0000", 2024-09-21T12:27:35.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:35.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:35.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.445102+0000", 2024-09-21T12:27:35.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4900000000000005e-07 2024-09-21T12:27:35.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:35.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.445107+0000", 2024-09-21T12:27:35.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1229999999999999e-06 2024-09-21T12:27:35.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:35.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.445117+0000", 2024-09-21T12:27:35.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.825e-06 2024-09-21T12:27:35.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:35.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.445123+0000", 2024-09-21T12:27:35.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6999999999999996e-06 2024-09-21T12:27:35.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:35.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.445187+0000", 2024-09-21T12:27:35.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4068999999999999e-05 2024-09-21T12:27:35.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.691 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:35.691 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.445205+0000", 2024-09-21T12:27:35.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8168e-05 2024-09-21T12:27:35.691 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:35.691 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:35.691 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:35.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.691 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.25022.0:223846 8.a e222/160 8:56d9d43d:::benchmark_data_smithi114_104936_object93430:head v 222'13856, mlcod=222'13852)", 2024-09-21T12:27:35.691 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:27:35.445627+0000", 2024-09-21T12:27:35.691 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.001197112, 2024-09-21T12:27:35.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001364299, 2024-09-21T12:27:35.691 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:35.692 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:35.692 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:27:35.692 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:35.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:35.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.445627+0000", 2024-09-21T12:27:35.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:35.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:35.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.445627+0000", 2024-09-21T12:27:35.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:35.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:35.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.445628+0000", 2024-09-21T12:27:35.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6499999999999998e-07 2024-09-21T12:27:35.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:35.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.445633+0000", 2024-09-21T12:27:35.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2349999999999999e-06 2024-09-21T12:27:35.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:35.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.445643+0000", 2024-09-21T12:27:35.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6830000000000003e-06 2024-09-21T12:27:35.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:35.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.445649+0000", 2024-09-21T12:27:35.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6790000000000002e-06 2024-09-21T12:27:35.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:35.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.445908+0000", 2024-09-21T12:27:35.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000258903 2024-09-21T12:27:35.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:35.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:35.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:35.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.445923+0000", 2024-09-21T12:27:35.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5048999999999999e-05 2024-09-21T12:27:35.694 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:35.695 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:35.695 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:35.695 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:35.695 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-21T12:27:35.695 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 5 2024-09-21T12:27:35.695 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:27:35.750 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:35.782 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:35.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:35.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:35.884 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:35.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:35.915 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:35.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:36.016 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:36.048 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:36.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:36.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:36.150 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:36.183 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:36.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:36.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:36.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:36 smithi183 ceph-mon[33277]: 4.6 scrub starts 2024-09-21T12:27:36.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:36 smithi183 ceph-mon[33277]: 4.6 scrub ok 2024-09-21T12:27:36.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:36 smithi183 ceph-mon[33277]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:27:36.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:36 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:27:36.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:36 smithi183 ceph-mon[33277]: osdmap e222: 8 total, 8 up, 7 in 2024-09-21T12:27:36.285 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:36.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:36.318 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:36.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:36.419 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:36.454 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:36.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:36.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:36.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:36 smithi114 ceph-mon[40867]: 4.6 scrub starts 2024-09-21T12:27:36.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:36 smithi114 ceph-mon[40867]: 4.6 scrub ok 2024-09-21T12:27:36.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:36 smithi114 ceph-mon[40867]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:27:36.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:36 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:27:36.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:36 smithi114 ceph-mon[40867]: osdmap e222: 8 total, 8 up, 7 in 2024-09-21T12:27:36.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:36 smithi114 ceph-mon[30324]: 4.6 scrub starts 2024-09-21T12:27:36.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:36 smithi114 ceph-mon[30324]: 4.6 scrub ok 2024-09-21T12:27:36.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:36 smithi114 ceph-mon[30324]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:27:36.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:36 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:27:36.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:36 smithi114 ceph-mon[30324]: osdmap e222: 8 total, 8 up, 7 in 2024-09-21T12:27:36.556 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:36.576 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:36.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:36.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:36.639 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T12:27:36.678 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:36.698 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:36.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:36.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:36.800 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:36.822 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:36.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:36.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:36.923 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:36.945 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:27:36.964 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:36.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:36.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:37.066 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:37.098 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:37.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:37.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:37.200 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:37.237 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:37.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:37.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:37.251 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:37 smithi183 ceph-mon[33277]: pgmap v355: 178 pgs: 178 active+clean; 2.5 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.44k op/s 2024-09-21T12:27:37.252 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:37 smithi183 ceph-mon[33277]: osdmap e223: 8 total, 8 up, 7 in 2024-09-21T12:27:37.252 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:27:36 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:27:37.338 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:37.375 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:37.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:37.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:37.476 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:37.511 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:37.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:37.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:37.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:37 smithi114 ceph-mon[30324]: pgmap v355: 178 pgs: 178 active+clean; 2.5 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.44k op/s 2024-09-21T12:27:37.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:37 smithi114 ceph-mon[30324]: osdmap e223: 8 total, 8 up, 7 in 2024-09-21T12:27:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:37 smithi114 ceph-mon[40867]: pgmap v355: 178 pgs: 178 active+clean; 2.5 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.44k op/s 2024-09-21T12:27:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:37 smithi114 ceph-mon[40867]: osdmap e223: 8 total, 8 up, 7 in 2024-09-21T12:27:37.613 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:37.617 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:27:37.635 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:37.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:37.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:37.697 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:27:37.697 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-21T12:27:37.697 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-21T12:27:37.697 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-21T12:27:37.697 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:27:37.737 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:37.778 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:37.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:37.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:37.880 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:37.909 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:37.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:37.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:38.011 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:38.039 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:38.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:38.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:38.141 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:38.168 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:38.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:38.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:38.270 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:38.304 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:38.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:38.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:38.394 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:38 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:27:38.406 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:38.438 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T12:27:38.441 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:38.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:38.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:38.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:38 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:27:38.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:38 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:27:38.543 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:38.579 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:38.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:38.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:38.681 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:38.718 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:38.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:38.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:38.820 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:38.849 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:27:38.851 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:38.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:38.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:38.953 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:38.975 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:38.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:38.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:39.077 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:39.108 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:39.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:39.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:39.209 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:39.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:39 smithi183 ceph-mon[33277]: pgmap v357: 178 pgs: 178 active+clean; 2.1 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.63k op/s 2024-09-21T12:27:39.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:39.246 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:39.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:39.347 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:39.377 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:39.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:39.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:39.421 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:27:39.421 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-21T12:27:39.421 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-21T12:27:39.421 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:27:39.421 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.421 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14478.0:1123 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94620848895616 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:27:39.421 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:40.364823+0000", 2024-09-21T12:27:39.421 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 359.05315851300003, 2024-09-21T12:27:39.421 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.76961642699999999, 2024-09-21T12:27:39.421 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:39.421 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:39.422 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:27:39.422 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:27:39.422 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14478", 2024-09-21T12:27:39.422 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:27:39.422 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 1123 2024-09-21T12:27:39.422 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.422 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:39.422 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.422 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:39.422 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364823+0000", 2024-09-21T12:27:39.422 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.422 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.422 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.422 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:39.422 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364823+0000", 2024-09-21T12:27:39.423 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.423 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.423 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.423 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:39.423 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364824+0000", 2024-09-21T12:27:39.423 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8400000000000003e-07 2024-09-21T12:27:39.423 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.423 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.423 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:39.423 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364828+0000", 2024-09-21T12:27:39.423 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7160000000000001e-06 2024-09-21T12:27:39.423 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.423 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.424 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:39.424 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364839+0000", 2024-09-21T12:27:39.424 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1036000000000001e-05 2024-09-21T12:27:39.424 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.424 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.424 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:39.424 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364844+0000", 2024-09-21T12:27:39.424 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3650000000000003e-06 2024-09-21T12:27:39.424 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.424 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.424 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:39.424 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364904+0000", 2024-09-21T12:27:39.425 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0182000000000001e-05 2024-09-21T12:27:39.425 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.425 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.425 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:27:39.425 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364914+0000", 2024-09-21T12:27:39.425 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5319999999999995e-06 2024-09-21T12:27:39.425 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.425 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.425 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:39.425 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.023847+0000", 2024-09-21T12:27:39.425 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.65893299299999997 2024-09-21T12:27:39.425 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.425 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.426 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for degraded object", 2024-09-21T12:27:39.426 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.024200+0000", 2024-09-21T12:27:39.426 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00035333899999999999 2024-09-21T12:27:39.426 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.426 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.426 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:39.426 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.026259+0000", 2024-09-21T12:27:39.426 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020587169999999998 2024-09-21T12:27:39.426 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.426 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.426 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:39.426 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.026327+0000", 2024-09-21T12:27:39.426 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8362999999999996e-05 2024-09-21T12:27:39.427 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.427 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.427 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:39.427 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.134440+0000", 2024-09-21T12:27:39.427 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10811229999999999 2024-09-21T12:27:39.427 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.427 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:39.427 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.428 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.428 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.428 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:24800 7.0 7:089470e4:::benchmark_data_smithi114_78432_object24799:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T12:27:39.428 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:42.436766+0000", 2024-09-21T12:27:39.428 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 356.981216034, 2024-09-21T12:27:39.428 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15453825400000001, 2024-09-21T12:27:39.428 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:39.428 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:39.428 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:39.428 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:27:39.428 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:27:39.428 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:27:39.428 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 24800 2024-09-21T12:27:39.429 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.429 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:39.429 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.429 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:39.429 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.436766+0000", 2024-09-21T12:27:39.429 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.429 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.429 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.429 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:39.429 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.436766+0000", 2024-09-21T12:27:39.429 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.429 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.429 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.429 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:39.429 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.436768+0000", 2024-09-21T12:27:39.430 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7460000000000001e-06 2024-09-21T12:27:39.430 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.430 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.430 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:39.430 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.436800+0000", 2024-09-21T12:27:39.430 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1995000000000001e-05 2024-09-21T12:27:39.430 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.430 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.430 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:39.430 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.436823+0000", 2024-09-21T12:27:39.430 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3107000000000001e-05 2024-09-21T12:27:39.430 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.431 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.431 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:39.431 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.436830+0000", 2024-09-21T12:27:39.431 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2450000000000001e-06 2024-09-21T12:27:39.431 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.431 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.431 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:39.431 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.436907+0000", 2024-09-21T12:27:39.431 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7613000000000004e-05 2024-09-21T12:27:39.431 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.431 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.431 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:39.431 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.437022+0000", 2024-09-21T12:27:39.432 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000114988 2024-09-21T12:27:39.432 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.432 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.432 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:27:39.432 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.437165+0000", 2024-09-21T12:27:39.432 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014216200000000001 2024-09-21T12:27:39.432 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.432 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.432 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:27:39.432 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.438118+0000", 2024-09-21T12:27:39.432 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00095354499999999998 2024-09-21T12:27:39.432 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.432 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.433 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.433 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.442130+0000", 2024-09-21T12:27:39.433 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0040119480000000004 2024-09-21T12:27:39.433 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.433 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.433 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.433 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.591203+0000", 2024-09-21T12:27:39.433 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14907253200000001 2024-09-21T12:27:39.433 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.433 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.433 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:39.433 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.591262+0000", 2024-09-21T12:27:39.433 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9249999999999997e-05 2024-09-21T12:27:39.433 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.433 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.433 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:39.434 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.591304+0000", 2024-09-21T12:27:39.434 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2123000000000003e-05 2024-09-21T12:27:39.434 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.434 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:39.434 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.434 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.434 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.434 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:45500 7.0 7:094d36cc:::benchmark_data_smithi114_78432_object45499:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T12:27:39.434 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:59.293987+0000", 2024-09-21T12:27:39.434 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 340.12399485200001, 2024-09-21T12:27:39.434 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.16208599700000001, 2024-09-21T12:27:39.434 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:39.434 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:39.434 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:39.435 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:27:39.435 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:27:39.435 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:27:39.435 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 45500 2024-09-21T12:27:39.435 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.435 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:39.435 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.436 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:39.436 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293987+0000", 2024-09-21T12:27:39.436 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.436 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.436 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.436 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:39.436 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293987+0000", 2024-09-21T12:27:39.436 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.436 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.436 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.436 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:39.436 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293988+0000", 2024-09-21T12:27:39.436 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3489999999999999e-06 2024-09-21T12:27:39.437 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.437 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.437 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:39.437 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294038+0000", 2024-09-21T12:27:39.437 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9835e-05 2024-09-21T12:27:39.437 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.437 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.437 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:39.437 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294051+0000", 2024-09-21T12:27:39.437 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2463000000000001e-05 2024-09-21T12:27:39.437 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.437 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.437 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:39.437 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294057+0000", 2024-09-21T12:27:39.438 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7390000000000002e-06 2024-09-21T12:27:39.438 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.438 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.438 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:39.438 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294115+0000", 2024-09-21T12:27:39.438 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7219000000000002e-05 2024-09-21T12:27:39.438 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.438 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.438 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:39.438 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294240+0000", 2024-09-21T12:27:39.438 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012520899999999999 2024-09-21T12:27:39.438 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.438 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.438 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:27:39.438 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294329+0000", 2024-09-21T12:27:39.439 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8871999999999999e-05 2024-09-21T12:27:39.439 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.439 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.439 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:27:39.439 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.296811+0000", 2024-09-21T12:27:39.439 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024828570000000002 2024-09-21T12:27:39.439 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.439 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.439 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.439 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302073+0000", 2024-09-21T12:27:39.439 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0052615539999999999 2024-09-21T12:27:39.439 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.439 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.439 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.440 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.455967+0000", 2024-09-21T12:27:39.440 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15389399200000001 2024-09-21T12:27:39.440 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.440 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.440 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:39.440 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.456027+0000", 2024-09-21T12:27:39.440 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9537999999999998e-05 2024-09-21T12:27:39.440 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.440 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.440 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:39.440 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.456073+0000", 2024-09-21T12:27:39.440 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6369999999999998e-05 2024-09-21T12:27:39.440 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.440 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:39.440 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.440 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.441 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.441 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:45512 7.0 7:00045c2a:::benchmark_data_smithi114_78432_object45511:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T12:27:39.441 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:59.294696+0000", 2024-09-21T12:27:39.441 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 340.123285952, 2024-09-21T12:27:39.441 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.16753293999999999, 2024-09-21T12:27:39.441 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:39.441 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:39.441 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:39.441 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:27:39.441 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:27:39.441 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:27:39.441 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 45512 2024-09-21T12:27:39.441 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.441 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:39.441 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.442 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:39.442 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294696+0000", 2024-09-21T12:27:39.442 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.442 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.442 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.442 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:39.442 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294696+0000", 2024-09-21T12:27:39.442 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.442 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.442 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.442 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:39.442 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294697+0000", 2024-09-21T12:27:39.442 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1299999999999999e-07 2024-09-21T12:27:39.442 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.443 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.443 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:39.443 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294777+0000", 2024-09-21T12:27:39.443 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0201e-05 2024-09-21T12:27:39.443 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.443 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.443 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:39.443 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294787+0000", 2024-09-21T12:27:39.443 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0281e-05 2024-09-21T12:27:39.443 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.443 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.443 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:39.443 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294794+0000", 2024-09-21T12:27:39.443 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2960000000000001e-06 2024-09-21T12:27:39.443 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.444 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.444 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:39.444 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.295769+0000", 2024-09-21T12:27:39.444 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00097407400000000001 2024-09-21T12:27:39.444 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.444 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.444 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:39.444 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.295919+0000", 2024-09-21T12:27:39.444 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015044200000000001 2024-09-21T12:27:39.444 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.444 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.444 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:27:39.445 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.296054+0000", 2024-09-21T12:27:39.445 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013465499999999999 2024-09-21T12:27:39.445 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.445 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.445 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:27:39.445 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.296931+0000", 2024-09-21T12:27:39.445 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00087781000000000003 2024-09-21T12:27:39.445 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.445 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.445 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.445 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302444+0000", 2024-09-21T12:27:39.445 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0055121249999999997 2024-09-21T12:27:39.445 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.445 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.445 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.445 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.462132+0000", 2024-09-21T12:27:39.446 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15968875499999999 2024-09-21T12:27:39.446 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.446 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.446 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:39.446 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.462187+0000", 2024-09-21T12:27:39.446 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4363000000000001e-05 2024-09-21T12:27:39.446 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.446 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.446 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:39.446 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.462229+0000", 2024-09-21T12:27:39.446 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2125000000000001e-05 2024-09-21T12:27:39.446 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.446 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:39.446 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.446 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.447 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.447 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:49240 7.0 7:06556a82:::benchmark_data_smithi114_78432_object49239:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:27:39.447 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:04.578874+0000", 2024-09-21T12:27:39.447 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 334.83910793000001, 2024-09-21T12:27:39.447 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.143013362, 2024-09-21T12:27:39.447 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:39.447 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:39.447 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:39.447 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:27:39.447 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:27:39.447 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:27:39.447 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 49240 2024-09-21T12:27:39.447 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.447 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:39.447 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.448 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:39.448 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.578874+0000", 2024-09-21T12:27:39.448 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.448 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.448 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.448 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:39.448 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.578874+0000", 2024-09-21T12:27:39.448 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.448 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.448 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.448 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:39.448 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.578875+0000", 2024-09-21T12:27:39.448 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4109999999999999e-06 2024-09-21T12:27:39.448 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.448 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.448 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:39.449 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.578925+0000", 2024-09-21T12:27:39.449 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9962e-05 2024-09-21T12:27:39.449 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.449 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.449 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:39.449 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.578936+0000", 2024-09-21T12:27:39.449 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1078e-05 2024-09-21T12:27:39.449 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.449 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.449 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:39.449 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.578941+0000", 2024-09-21T12:27:39.449 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0289999999999996e-06 2024-09-21T12:27:39.449 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.449 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.450 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:39.450 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.578985+0000", 2024-09-21T12:27:39.450 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3659000000000003e-05 2024-09-21T12:27:39.450 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.450 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.450 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:39.450 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.579076+0000", 2024-09-21T12:27:39.450 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1416999999999996e-05 2024-09-21T12:27:39.450 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.450 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.450 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:27:39.450 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.579166+0000", 2024-09-21T12:27:39.450 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9338000000000004e-05 2024-09-21T12:27:39.450 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.450 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.451 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:27:39.451 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.580345+0000", 2024-09-21T12:27:39.451 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0011795899999999999 2024-09-21T12:27:39.451 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.451 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.451 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.451 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.580659+0000", 2024-09-21T12:27:39.451 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00031388 2024-09-21T12:27:39.451 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.451 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.451 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.451 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.721812+0000", 2024-09-21T12:27:39.451 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14115282900000001 2024-09-21T12:27:39.451 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.451 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.452 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:39.452 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.721860+0000", 2024-09-21T12:27:39.452 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8090000000000002e-05 2024-09-21T12:27:39.452 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.452 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.452 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:39.452 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.721887+0000", 2024-09-21T12:27:39.452 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7078999999999999e-05 2024-09-21T12:27:39.452 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.452 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:39.452 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.452 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.452 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.452 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:116819 7.0 7:07411ed9:::benchmark_data_smithi114_78432_object12673:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T12:27:39.453 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:23:12.370997+0000", 2024-09-21T12:27:39.453 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 267.04698512800002, 2024-09-21T12:27:39.453 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15405449900000001, 2024-09-21T12:27:39.453 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:39.453 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:39.453 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:39.453 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:27:39.453 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:27:39.453 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:27:39.453 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 116819 2024-09-21T12:27:39.453 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.453 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:39.453 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.453 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:39.453 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.370997+0000", 2024-09-21T12:27:39.454 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.454 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.454 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.454 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:39.454 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.370997+0000", 2024-09-21T12:27:39.454 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.454 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.454 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.454 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:39.454 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.370998+0000", 2024-09-21T12:27:39.454 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.381e-06 2024-09-21T12:27:39.454 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.454 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.454 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:39.455 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.371002+0000", 2024-09-21T12:27:39.455 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3710000000000002e-06 2024-09-21T12:27:39.455 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.455 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.455 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:39.455 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.371014+0000", 2024-09-21T12:27:39.455 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1865000000000001e-05 2024-09-21T12:27:39.455 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.455 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.455 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:39.455 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.371022+0000", 2024-09-21T12:27:39.455 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4610000000000002e-06 2024-09-21T12:27:39.455 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.455 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.455 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:39.456 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.371071+0000", 2024-09-21T12:27:39.456 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9082000000000002e-05 2024-09-21T12:27:39.456 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.456 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.456 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:39.456 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.371188+0000", 2024-09-21T12:27:39.456 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000117015 2024-09-21T12:27:39.456 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.456 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.456 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:27:39.456 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.371309+0000", 2024-09-21T12:27:39.456 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012115700000000001 2024-09-21T12:27:39.456 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.456 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.456 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:27:39.457 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.373478+0000", 2024-09-21T12:27:39.457 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021686840000000001 2024-09-21T12:27:39.457 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.457 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.457 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.457 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.377634+0000", 2024-09-21T12:27:39.457 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0041562589999999998 2024-09-21T12:27:39.457 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.457 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.457 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.457 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.524881+0000", 2024-09-21T12:27:39.457 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14724746899999999 2024-09-21T12:27:39.457 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.457 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.457 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:39.458 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.525006+0000", 2024-09-21T12:27:39.458 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012413300000000001 2024-09-21T12:27:39.458 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.458 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.458 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:39.458 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.525051+0000", 2024-09-21T12:27:39.458 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5621999999999998e-05 2024-09-21T12:27:39.458 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.458 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:39.458 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.458 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.458 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.458 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:149806 7.0 7:0c9bb691:::benchmark_data_smithi114_78432_object45660:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T12:27:39.458 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:23:42.006140+0000", 2024-09-21T12:27:39.458 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 237.41184202599999, 2024-09-21T12:27:39.459 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13218240000000001, 2024-09-21T12:27:39.459 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:39.459 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:39.459 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:39.459 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:27:39.459 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:27:39.459 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:27:39.459 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 149806 2024-09-21T12:27:39.459 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.459 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:39.459 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.459 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:39.459 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.006140+0000", 2024-09-21T12:27:39.459 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.459 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.460 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.460 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:39.460 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.006140+0000", 2024-09-21T12:27:39.460 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.460 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.460 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.460 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:39.460 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.006141+0000", 2024-09-21T12:27:39.460 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7039999999999999e-06 2024-09-21T12:27:39.460 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.460 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.460 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:39.460 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.006147+0000", 2024-09-21T12:27:39.460 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2499999999999997e-06 2024-09-21T12:27:39.460 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.460 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.461 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:39.461 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.006161+0000", 2024-09-21T12:27:39.461 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3974999999999999e-05 2024-09-21T12:27:39.461 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.461 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.461 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:39.461 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.006169+0000", 2024-09-21T12:27:39.461 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7299999999999994e-06 2024-09-21T12:27:39.461 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.461 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.461 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:39.461 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.006225+0000", 2024-09-21T12:27:39.461 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5232000000000002e-05 2024-09-21T12:27:39.461 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.461 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.462 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:39.462 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.006299+0000", 2024-09-21T12:27:39.462 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4219000000000002e-05 2024-09-21T12:27:39.462 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.462 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.462 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:27:39.462 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.006365+0000", 2024-09-21T12:27:39.462 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6278999999999993e-05 2024-09-21T12:27:39.462 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.462 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.462 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:27:39.462 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.008358+0000", 2024-09-21T12:27:39.462 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0019933220000000001 2024-09-21T12:27:39.462 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.463 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.463 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.463 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.013964+0000", 2024-09-21T12:27:39.463 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.005605751 2024-09-21T12:27:39.463 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.463 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.463 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.463 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.138215+0000", 2024-09-21T12:27:39.463 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.12425032 2024-09-21T12:27:39.463 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.463 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.463 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:39.463 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.138279+0000", 2024-09-21T12:27:39.463 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4129999999999998e-05 2024-09-21T12:27:39.463 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.463 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.464 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:39.464 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.138322+0000", 2024-09-21T12:27:39.464 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3488000000000001e-05 2024-09-21T12:27:39.464 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.464 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:39.464 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.464 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.464 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.464 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:149807 7.0 7:0e04ad06:::benchmark_data_smithi114_78432_object45661:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T12:27:39.464 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:23:42.009615+0000", 2024-09-21T12:27:39.464 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 237.40836727199999, 2024-09-21T12:27:39.464 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.128815979, 2024-09-21T12:27:39.464 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:39.464 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:39.464 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:39.465 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:27:39.465 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:27:39.465 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:27:39.465 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 149807 2024-09-21T12:27:39.465 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.465 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:39.465 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.465 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:39.465 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.009615+0000", 2024-09-21T12:27:39.465 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.465 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.465 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.465 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:39.465 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.009615+0000", 2024-09-21T12:27:39.465 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.466 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.466 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.466 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:39.466 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.009616+0000", 2024-09-21T12:27:39.466 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.311e-06 2024-09-21T12:27:39.466 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.466 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.466 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:39.466 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.009621+0000", 2024-09-21T12:27:39.466 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3850000000000001e-06 2024-09-21T12:27:39.466 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.466 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.466 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:39.466 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.009636+0000", 2024-09-21T12:27:39.467 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4284000000000001e-05 2024-09-21T12:27:39.467 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.467 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.467 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:39.467 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.009644+0000", 2024-09-21T12:27:39.467 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2349999999999999e-06 2024-09-21T12:27:39.467 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.467 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.467 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:39.467 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.009700+0000", 2024-09-21T12:27:39.467 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6236999999999997e-05 2024-09-21T12:27:39.467 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.467 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.467 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:39.467 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.009824+0000", 2024-09-21T12:27:39.467 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012450699999999999 2024-09-21T12:27:39.468 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.468 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.468 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:27:39.468 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.009947+0000", 2024-09-21T12:27:39.468 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012274900000000001 2024-09-21T12:27:39.468 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.468 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.468 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:27:39.468 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.011830+0000", 2024-09-21T12:27:39.468 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00188254 2024-09-21T12:27:39.468 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.468 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.468 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.468 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.013996+0000", 2024-09-21T12:27:39.469 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002166126 2024-09-21T12:27:39.469 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.469 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.469 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.469 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.138364+0000", 2024-09-21T12:27:39.469 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.124368232 2024-09-21T12:27:39.469 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.469 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.469 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:39.469 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.138394+0000", 2024-09-21T12:27:39.469 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0205999999999998e-05 2024-09-21T12:27:39.469 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.469 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.469 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:39.469 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.138431+0000", 2024-09-21T12:27:39.470 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6167000000000001e-05 2024-09-21T12:27:39.470 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.470 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:39.470 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.470 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.470 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.470 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:34506 8.8 8:19b95b45:::benchmark_data_smithi114_104936_object34505:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T12:27:39.470 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:08.945853+0000", 2024-09-21T12:27:39.470 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 150.47212910900001, 2024-09-21T12:27:39.470 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.124861031, 2024-09-21T12:27:39.470 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:39.470 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:39.470 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:39.470 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:27:39.471 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:27:39.471 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:27:39.471 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 34506 2024-09-21T12:27:39.471 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.471 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:39.471 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.471 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:39.471 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:08.945853+0000", 2024-09-21T12:27:39.471 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.471 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.471 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.471 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:39.471 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:08.945853+0000", 2024-09-21T12:27:39.472 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.472 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.472 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.472 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:39.472 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:08.945853+0000", 2024-09-21T12:27:39.472 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5799999999999999e-07 2024-09-21T12:27:39.472 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.472 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.472 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:39.472 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:08.945884+0000", 2024-09-21T12:27:39.472 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1037000000000001e-05 2024-09-21T12:27:39.472 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.472 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.472 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:39.472 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:08.945892+0000", 2024-09-21T12:27:39.473 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8210000000000002e-06 2024-09-21T12:27:39.473 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.473 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.473 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:39.473 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:08.945897+0000", 2024-09-21T12:27:39.473 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1329999999999998e-06 2024-09-21T12:27:39.473 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.473 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.473 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:39.473 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:08.949452+0000", 2024-09-21T12:27:39.473 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0035542030000000001 2024-09-21T12:27:39.473 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.473 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.473 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:39.473 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:08.949563+0000", 2024-09-21T12:27:39.474 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000111208 2024-09-21T12:27:39.474 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.474 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.474 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:27:39.474 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:08.949693+0000", 2024-09-21T12:27:39.474 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013028700000000001 2024-09-21T12:27:39.474 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.474 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.475 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:39 smithi114 ceph-mon[30324]: pgmap v357: 178 pgs: 178 active+clean; 2.1 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.63k op/s 2024-09-21T12:27:39.475 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:39 smithi114 ceph-mon[40867]: pgmap v357: 178 pgs: 178 active+clean; 2.1 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.63k op/s 2024-09-21T12:27:39.476 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:27:39.476 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:08.952213+0000", 2024-09-21T12:27:39.476 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002519847 2024-09-21T12:27:39.476 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.476 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.476 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.476 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:08.952301+0000", 2024-09-21T12:27:39.476 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8010999999999995e-05 2024-09-21T12:27:39.476 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.476 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.477 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.477 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:09.070606+0000", 2024-09-21T12:27:39.477 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.118305417 2024-09-21T12:27:39.477 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.477 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.477 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:39.477 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:09.070668+0000", 2024-09-21T12:27:39.477 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1915999999999995e-05 2024-09-21T12:27:39.477 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.477 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.477 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:39.477 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:09.070714+0000", 2024-09-21T12:27:39.477 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5493e-05 2024-09-21T12:27:39.477 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.477 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:39.478 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.478 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.478 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.478 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:36116 8.8 8:1e014655:::benchmark_data_smithi114_104936_object36115:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T12:27:39.478 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:09.970630+0000", 2024-09-21T12:27:39.478 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 149.447351336, 2024-09-21T12:27:39.478 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.140201205, 2024-09-21T12:27:39.478 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:39.478 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:39.478 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:39.478 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:27:39.478 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:27:39.478 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:27:39.479 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 36116 2024-09-21T12:27:39.479 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.479 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:39.479 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.479 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:39.479 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:09.970630+0000", 2024-09-21T12:27:39.479 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.479 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.479 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.479 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:39.479 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:09.970630+0000", 2024-09-21T12:27:39.479 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.479 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.479 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.479 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:39.480 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:09.970632+0000", 2024-09-21T12:27:39.480 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.305e-06 2024-09-21T12:27:39.480 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.480 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.480 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:39.480 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:09.970678+0000", 2024-09-21T12:27:39.480 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5933999999999998e-05 2024-09-21T12:27:39.480 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.480 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.480 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:39.480 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:09.970689+0000", 2024-09-21T12:27:39.480 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0902e-05 2024-09-21T12:27:39.480 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.480 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.481 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:39.481 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:09.970694+0000", 2024-09-21T12:27:39.481 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5280000000000003e-06 2024-09-21T12:27:39.481 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.481 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.481 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:39.481 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:09.970734+0000", 2024-09-21T12:27:39.481 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0250999999999998e-05 2024-09-21T12:27:39.481 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.481 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.481 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:39.481 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:09.970825+0000", 2024-09-21T12:27:39.481 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0736000000000007e-05 2024-09-21T12:27:39.481 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.482 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.482 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:27:39.482 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:09.970962+0000", 2024-09-21T12:27:39.482 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000136816 2024-09-21T12:27:39.482 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.482 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.482 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:27:39.482 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:09.972086+0000", 2024-09-21T12:27:39.482 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001124105 2024-09-21T12:27:39.482 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.482 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.482 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.482 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:09.972488+0000", 2024-09-21T12:27:39.482 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000401989 2024-09-21T12:27:39.482 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.483 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.483 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.483 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:10.110716+0000", 2024-09-21T12:27:39.483 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.138228301 2024-09-21T12:27:39.483 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.483 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.483 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:39.483 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:10.110781+0000", 2024-09-21T12:27:39.483 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4992000000000004e-05 2024-09-21T12:27:39.483 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.483 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.483 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:39.483 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:10.110832+0000", 2024-09-21T12:27:39.483 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0346000000000002e-05 2024-09-21T12:27:39.484 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.484 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:39.484 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.484 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.484 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.484 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:38136 8.8 8:1fdd00da:::benchmark_data_smithi114_104936_object38135:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T12:27:39.484 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:11.297833+0000", 2024-09-21T12:27:39.484 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 148.120149242, 2024-09-21T12:27:39.484 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.143885176, 2024-09-21T12:27:39.484 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:39.484 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:39.484 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:39.484 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:27:39.484 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:27:39.485 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:27:39.485 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 38136 2024-09-21T12:27:39.485 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.485 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:39.485 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.485 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:39.485 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.297833+0000", 2024-09-21T12:27:39.485 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.485 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.485 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.485 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:39.485 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.297833+0000", 2024-09-21T12:27:39.485 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.485 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.485 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.486 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:39.486 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.297833+0000", 2024-09-21T12:27:39.486 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6000000000000004e-07 2024-09-21T12:27:39.486 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.486 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.486 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:39.486 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.297870+0000", 2024-09-21T12:27:39.486 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6479000000000001e-05 2024-09-21T12:27:39.486 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.486 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.486 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:39.486 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.297877+0000", 2024-09-21T12:27:39.486 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0210000000000002e-06 2024-09-21T12:27:39.486 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.487 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.487 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:39.487 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.297882+0000", 2024-09-21T12:27:39.487 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7780000000000004e-06 2024-09-21T12:27:39.487 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.487 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.487 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:39.487 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.297948+0000", 2024-09-21T12:27:39.487 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5433999999999997e-05 2024-09-21T12:27:39.487 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.487 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.487 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:39.487 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.298088+0000", 2024-09-21T12:27:39.487 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000140495 2024-09-21T12:27:39.488 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.488 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.488 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:27:39.488 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.298271+0000", 2024-09-21T12:27:39.488 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000182261 2024-09-21T12:27:39.488 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.488 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.488 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:27:39.488 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.301068+0000", 2024-09-21T12:27:39.488 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0027971229999999999 2024-09-21T12:27:39.488 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.488 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.488 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.488 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.304972+0000", 2024-09-21T12:27:39.489 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00390398 2024-09-21T12:27:39.489 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.489 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.489 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.489 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.441633+0000", 2024-09-21T12:27:39.489 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13666144199999999 2024-09-21T12:27:39.489 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.489 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.489 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:39.489 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.441694+0000", 2024-09-21T12:27:39.489 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0501999999999998e-05 2024-09-21T12:27:39.489 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.489 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.489 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:39.489 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.441718+0000", 2024-09-21T12:27:39.490 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4100999999999998e-05 2024-09-21T12:27:39.490 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.490 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:39.490 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.490 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.490 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.490 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:38143 8.8 8:133e9789:::benchmark_data_smithi114_104936_object38142:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T12:27:39.490 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:11.302085+0000", 2024-09-21T12:27:39.490 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 148.115896886, 2024-09-21T12:27:39.490 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15153207799999999, 2024-09-21T12:27:39.490 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:39.490 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:39.490 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:39.491 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:27:39.491 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:27:39.491 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:27:39.491 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 38143 2024-09-21T12:27:39.491 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.491 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:39.491 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.491 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:39.491 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302085+0000", 2024-09-21T12:27:39.491 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.491 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.491 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.491 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:39.491 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302085+0000", 2024-09-21T12:27:39.491 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.492 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.492 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.492 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:39.492 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302086+0000", 2024-09-21T12:27:39.492 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4339999999999999e-06 2024-09-21T12:27:39.492 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.492 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.492 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:39.492 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302163+0000", 2024-09-21T12:27:39.492 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6200999999999998e-05 2024-09-21T12:27:39.493 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.493 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.493 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:39.493 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302172+0000", 2024-09-21T12:27:39.493 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5759999999999995e-06 2024-09-21T12:27:39.493 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.493 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.493 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:39.493 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302178+0000", 2024-09-21T12:27:39.493 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2260000000000002e-06 2024-09-21T12:27:39.493 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.493 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.493 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:39.493 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.303083+0000", 2024-09-21T12:27:39.494 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00090426600000000003 2024-09-21T12:27:39.494 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.494 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.494 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:39.494 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.303220+0000", 2024-09-21T12:27:39.494 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013702300000000001 2024-09-21T12:27:39.494 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.494 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.494 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:27:39.494 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.303354+0000", 2024-09-21T12:27:39.494 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013437099999999999 2024-09-21T12:27:39.494 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.494 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.494 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:27:39.495 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.305072+0000", 2024-09-21T12:27:39.495 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001718102 2024-09-21T12:27:39.495 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.495 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.495 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.495 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.306241+0000", 2024-09-21T12:27:39.495 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001168716 2024-09-21T12:27:39.495 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.495 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.495 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.495 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.453516+0000", 2024-09-21T12:27:39.495 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14727496200000001 2024-09-21T12:27:39.495 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.495 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.496 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:39.496 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.453572+0000", 2024-09-21T12:27:39.496 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5923e-05 2024-09-21T12:27:39.496 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.496 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.496 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:39.496 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.453617+0000", 2024-09-21T12:27:39.496 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5278000000000003e-05 2024-09-21T12:27:39.496 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.496 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:39.496 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.496 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.496 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.496 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:38148 8.8 8:1c0b8cc6:::benchmark_data_smithi114_104936_object38147:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T12:27:39.497 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:11.305128+0000", 2024-09-21T12:27:39.497 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 148.11285395799999, 2024-09-21T12:27:39.497 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.148605125, 2024-09-21T12:27:39.497 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:39.497 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:39.497 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:39.497 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:27:39.497 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:27:39.497 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:27:39.497 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 38148 2024-09-21T12:27:39.497 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.497 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:39.497 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.497 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:39.497 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.305128+0000", 2024-09-21T12:27:39.498 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.498 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.498 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.498 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:39.498 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.305128+0000", 2024-09-21T12:27:39.498 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.498 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.498 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.498 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:39.498 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.305129+0000", 2024-09-21T12:27:39.498 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.142e-06 2024-09-21T12:27:39.498 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.498 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.499 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:39.499 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.305182+0000", 2024-09-21T12:27:39.499 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3489999999999998e-05 2024-09-21T12:27:39.499 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.499 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.499 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:39.499 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.305192+0000", 2024-09-21T12:27:39.499 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3829999999999998e-06 2024-09-21T12:27:39.499 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.499 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.499 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:39.499 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.305198+0000", 2024-09-21T12:27:39.499 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6860000000000003e-06 2024-09-21T12:27:39.499 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.500 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.500 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:39.500 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.305239+0000", 2024-09-21T12:27:39.500 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1882000000000003e-05 2024-09-21T12:27:39.500 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.500 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.500 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:39.500 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.305351+0000", 2024-09-21T12:27:39.500 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011123799999999999 2024-09-21T12:27:39.500 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.500 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.500 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:27:39.500 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.305474+0000", 2024-09-21T12:27:39.501 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000123363 2024-09-21T12:27:39.501 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.501 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.501 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:27:39.501 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.306408+0000", 2024-09-21T12:27:39.501 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00093416099999999998 2024-09-21T12:27:39.501 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.501 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.501 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.501 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.306612+0000", 2024-09-21T12:27:39.501 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00020365699999999999 2024-09-21T12:27:39.501 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.501 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.501 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.501 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.453666+0000", 2024-09-21T12:27:39.502 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.147053778 2024-09-21T12:27:39.502 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.502 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.502 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:39.502 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.453698+0000", 2024-09-21T12:27:39.502 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2011000000000002e-05 2024-09-21T12:27:39.502 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.502 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.502 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:39.502 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.453733+0000", 2024-09-21T12:27:39.502 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5333999999999997e-05 2024-09-21T12:27:39.502 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.502 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:39.503 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.503 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.503 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.503 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:77674 8.d 8:bb2f14dc:::benchmark_data_smithi114_104936_object77673:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-21T12:27:39.503 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:40.251838+0000", 2024-09-21T12:27:39.503 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 119.166143902, 2024-09-21T12:27:39.503 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.126956077, 2024-09-21T12:27:39.503 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:39.503 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:39.503 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:39.503 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:27:39.503 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:27:39.503 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:27:39.503 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 77674 2024-09-21T12:27:39.504 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.504 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:39.504 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.504 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:39.504 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.251838+0000", 2024-09-21T12:27:39.504 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.504 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.504 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.504 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:39.504 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.251838+0000", 2024-09-21T12:27:39.504 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.504 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.504 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.505 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:39.505 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.251839+0000", 2024-09-21T12:27:39.505 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5099999999999999e-06 2024-09-21T12:27:39.505 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.505 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.505 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:39.505 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.251915+0000", 2024-09-21T12:27:39.505 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5210999999999995e-05 2024-09-21T12:27:39.505 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.505 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.505 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:39.505 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.251924+0000", 2024-09-21T12:27:39.505 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2750000000000001e-06 2024-09-21T12:27:39.505 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.506 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.506 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:39.506 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.251930+0000", 2024-09-21T12:27:39.506 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.584e-06 2024-09-21T12:27:39.506 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.506 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.506 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:39.506 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.251977+0000", 2024-09-21T12:27:39.506 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6774999999999999e-05 2024-09-21T12:27:39.506 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.506 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.506 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:39.506 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.252091+0000", 2024-09-21T12:27:39.506 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011381 2024-09-21T12:27:39.507 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.507 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.507 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,7", 2024-09-21T12:27:39.507 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.252238+0000", 2024-09-21T12:27:39.507 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014654499999999999 2024-09-21T12:27:39.507 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.507 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.507 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:27:39.507 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.256039+0000", 2024-09-21T12:27:39.507 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003801305 2024-09-21T12:27:39.507 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.507 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.507 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.507 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.378676+0000", 2024-09-21T12:27:39.507 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.122637309 2024-09-21T12:27:39.508 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.508 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.508 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.508 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.378724+0000", 2024-09-21T12:27:39.508 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8183999999999998e-05 2024-09-21T12:27:39.508 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.508 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.508 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:39.508 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.378762+0000", 2024-09-21T12:27:39.508 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7743000000000001e-05 2024-09-21T12:27:39.508 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.508 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.508 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:39.508 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.378794+0000", 2024-09-21T12:27:39.509 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1826000000000003e-05 2024-09-21T12:27:39.509 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.509 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:39.509 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.509 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.509 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.509 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:77676 8.d 8:b085cec0:::benchmark_data_smithi114_104936_object77675:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-21T12:27:39.509 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:40.252793+0000", 2024-09-21T12:27:39.509 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 119.165188806, 2024-09-21T12:27:39.509 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.126167109, 2024-09-21T12:27:39.509 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:39.509 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:39.509 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:39.509 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:27:39.510 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:27:39.510 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:27:39.510 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 77676 2024-09-21T12:27:39.510 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.510 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:39.510 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.510 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:39.510 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.252793+0000", 2024-09-21T12:27:39.510 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.510 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.510 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.510 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:39.510 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.252793+0000", 2024-09-21T12:27:39.511 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.511 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.511 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.511 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:39.511 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.252794+0000", 2024-09-21T12:27:39.511 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.074e-06 2024-09-21T12:27:39.511 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.511 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.511 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:39.511 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.252828+0000", 2024-09-21T12:27:39.511 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4198e-05 2024-09-21T12:27:39.511 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.511 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.511 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:39.512 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.252840+0000", 2024-09-21T12:27:39.512 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1344e-05 2024-09-21T12:27:39.512 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.512 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.512 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:39.512 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.252847+0000", 2024-09-21T12:27:39.512 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0319999999999998e-06 2024-09-21T12:27:39.512 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.512 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.512 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:39.512 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.254046+0000", 2024-09-21T12:27:39.512 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0011993030000000001 2024-09-21T12:27:39.512 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.512 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.513 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:39.513 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.254155+0000", 2024-09-21T12:27:39.513 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010935800000000001 2024-09-21T12:27:39.513 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.513 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.513 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,7", 2024-09-21T12:27:39.513 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.254263+0000", 2024-09-21T12:27:39.513 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010734 2024-09-21T12:27:39.513 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.513 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.513 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:27:39.513 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.256141+0000", 2024-09-21T12:27:39.513 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001877933 2024-09-21T12:27:39.513 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.513 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.514 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.514 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.378847+0000", 2024-09-21T12:27:39.514 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.12270599 2024-09-21T12:27:39.514 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.514 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.514 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.514 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.378883+0000", 2024-09-21T12:27:39.514 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6026999999999999e-05 2024-09-21T12:27:39.514 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.514 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.514 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:39.514 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.378928+0000", 2024-09-21T12:27:39.514 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5476999999999998e-05 2024-09-21T12:27:39.514 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.515 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.515 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:39.515 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.378960+0000", 2024-09-21T12:27:39.515 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2033000000000003e-05 2024-09-21T12:27:39.515 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.515 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:39.515 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.515 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.515 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.515 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.25022.0:77679 8.a e183/160 8:5f3d1dfc:::benchmark_data_smithi114_104936_object77678:head v 183'4753, mlcod=183'4751)", 2024-09-21T12:27:39.515 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:40.254563+0000", 2024-09-21T12:27:39.515 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 119.163418968, 2024-09-21T12:27:39.515 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.12409081, 2024-09-21T12:27:39.515 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:39.516 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:39.516 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:39.516 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:39.516 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.516 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:39.516 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.254563+0000", 2024-09-21T12:27:39.516 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.516 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.516 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.516 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:39.516 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.254563+0000", 2024-09-21T12:27:39.516 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.516 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.516 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.516 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:39.517 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.254564+0000", 2024-09-21T12:27:39.517 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8400000000000003e-07 2024-09-21T12:27:39.517 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.517 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.517 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:39.517 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.254628+0000", 2024-09-21T12:27:39.517 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4419999999999996e-05 2024-09-21T12:27:39.517 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.517 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.517 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:39.517 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.254639+0000", 2024-09-21T12:27:39.517 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0472000000000001e-05 2024-09-21T12:27:39.517 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.517 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.518 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:39.518 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.254645+0000", 2024-09-21T12:27:39.518 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0209999999999996e-06 2024-09-21T12:27:39.518 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.518 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.518 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:39.518 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.254693+0000", 2024-09-21T12:27:39.518 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8195999999999997e-05 2024-09-21T12:27:39.518 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.518 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.518 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:39.518 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.254709+0000", 2024-09-21T12:27:39.518 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6392000000000002e-05 2024-09-21T12:27:39.518 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.518 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.519 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:39.519 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.378614+0000", 2024-09-21T12:27:39.519 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.12390522299999999 2024-09-21T12:27:39.519 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.519 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.519 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:39.519 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.378654+0000", 2024-09-21T12:27:39.519 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9301999999999997e-05 2024-09-21T12:27:39.519 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.519 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:39.519 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.519 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.519 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.519 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:77681 8.4 8:289ac157:::benchmark_data_smithi114_104936_object77680:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-21T12:27:39.520 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:40.255755+0000", 2024-09-21T12:27:39.520 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 119.162226374, 2024-09-21T12:27:39.520 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.12688951700000001, 2024-09-21T12:27:39.520 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:39.520 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:39.520 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:39.520 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:27:39.520 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:27:39.520 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:27:39.520 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 77681 2024-09-21T12:27:39.520 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.520 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:39.520 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.520 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:39.520 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.255755+0000", 2024-09-21T12:27:39.521 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.521 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.521 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.521 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:39.521 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.255755+0000", 2024-09-21T12:27:39.521 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.521 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.521 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.521 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:39.521 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.255757+0000", 2024-09-21T12:27:39.521 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.325e-06 2024-09-21T12:27:39.521 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.521 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.522 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:39.522 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.255837+0000", 2024-09-21T12:27:39.522 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0737999999999998e-05 2024-09-21T12:27:39.522 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.522 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.522 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:39.522 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.255847+0000", 2024-09-21T12:27:39.522 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6700000000000006e-06 2024-09-21T12:27:39.522 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.522 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.522 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:39.522 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.255853+0000", 2024-09-21T12:27:39.522 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0700000000000003e-06 2024-09-21T12:27:39.522 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.523 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.523 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:39.523 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.255908+0000", 2024-09-21T12:27:39.523 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5135e-05 2024-09-21T12:27:39.523 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.523 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.523 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:39.523 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.256027+0000", 2024-09-21T12:27:39.523 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000118538 2024-09-21T12:27:39.523 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.523 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.523 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:27:39.523 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.256175+0000", 2024-09-21T12:27:39.523 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014814400000000001 2024-09-21T12:27:39.524 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.524 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.524 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.524 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.382361+0000", 2024-09-21T12:27:39.524 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.12618574900000001 2024-09-21T12:27:39.524 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.524 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.524 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.524 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.382405+0000", 2024-09-21T12:27:39.524 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4258999999999998e-05 2024-09-21T12:27:39.524 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.524 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.524 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:27:39.524 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.382551+0000", 2024-09-21T12:27:39.525 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000145974 2024-09-21T12:27:39.525 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.525 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.525 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:39.525 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.382602+0000", 2024-09-21T12:27:39.525 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0702999999999999e-05 2024-09-21T12:27:39.525 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.525 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.525 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:39.525 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:40.382645+0000", 2024-09-21T12:27:39.525 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3211999999999999e-05 2024-09-21T12:27:39.525 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.525 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:39.525 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.526 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.526 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.526 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:99019 8.8 8:16dc5116:::benchmark_data_smithi114_104936_object99018:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-21T12:27:39.526 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:57.695581+0000", 2024-09-21T12:27:39.526 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 101.722400509, 2024-09-21T12:27:39.526 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14590615700000001, 2024-09-21T12:27:39.526 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:39.526 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:39.526 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:39.526 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:27:39.526 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:27:39.526 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:27:39.526 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 99019 2024-09-21T12:27:39.527 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.527 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:39.527 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.527 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:39.527 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.695581+0000", 2024-09-21T12:27:39.527 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.527 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.527 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.527 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:39.527 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.695581+0000", 2024-09-21T12:27:39.527 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.527 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.527 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.527 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:39.528 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.695583+0000", 2024-09-21T12:27:39.528 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4750000000000001e-06 2024-09-21T12:27:39.528 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.528 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.528 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:39.528 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.695635+0000", 2024-09-21T12:27:39.528 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1758999999999997e-05 2024-09-21T12:27:39.528 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.528 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.528 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:39.528 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.695645+0000", 2024-09-21T12:27:39.528 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0515e-05 2024-09-21T12:27:39.528 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.528 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.528 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:39.528 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.695652+0000", 2024-09-21T12:27:39.529 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1269999999999997e-06 2024-09-21T12:27:39.529 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.529 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.529 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:39.529 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.698722+0000", 2024-09-21T12:27:39.529 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0030701370000000001 2024-09-21T12:27:39.529 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.529 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.529 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:39.529 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.698948+0000", 2024-09-21T12:27:39.529 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000225219 2024-09-21T12:27:39.529 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.529 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.529 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:27:39.530 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.699165+0000", 2024-09-21T12:27:39.530 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00021733500000000001 2024-09-21T12:27:39.530 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.530 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.530 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.530 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.702022+0000", 2024-09-21T12:27:39.530 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0028576259999999998 2024-09-21T12:27:39.530 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.530 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.530 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:27:39.530 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.707874+0000", 2024-09-21T12:27:39.530 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0058513849999999997 2024-09-21T12:27:39.530 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.530 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.531 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.531 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.841381+0000", 2024-09-21T12:27:39.531 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.133507406 2024-09-21T12:27:39.531 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.531 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.531 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:39.531 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.841438+0000", 2024-09-21T12:27:39.531 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6221999999999998e-05 2024-09-21T12:27:39.531 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.531 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.531 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:39.531 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.841487+0000", 2024-09-21T12:27:39.531 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9951000000000003e-05 2024-09-21T12:27:39.531 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.532 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:39.532 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.532 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.532 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.532 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:99025 8.8 8:1b11fe1e:::benchmark_data_smithi114_104936_object99024:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-21T12:27:39.532 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:57.699077+0000", 2024-09-21T12:27:39.532 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 101.71890514099999, 2024-09-21T12:27:39.532 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14424990100000001, 2024-09-21T12:27:39.532 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:39.532 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:39.532 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:39.532 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:27:39.532 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:27:39.532 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:27:39.533 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 99025 2024-09-21T12:27:39.533 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.533 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:39.533 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.533 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:39.533 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.699077+0000", 2024-09-21T12:27:39.533 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.533 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.533 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.533 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:39.533 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.699077+0000", 2024-09-21T12:27:39.533 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.533 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.533 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.534 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:39.534 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.699078+0000", 2024-09-21T12:27:39.534 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1310000000000001e-06 2024-09-21T12:27:39.534 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.534 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.534 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:39.534 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.699136+0000", 2024-09-21T12:27:39.534 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8109000000000002e-05 2024-09-21T12:27:39.534 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.534 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.534 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:39.534 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.699147+0000", 2024-09-21T12:27:39.534 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.15e-05 2024-09-21T12:27:39.534 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.535 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.535 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:39.535 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.699156+0000", 2024-09-21T12:27:39.535 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.884e-06 2024-09-21T12:27:39.535 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.535 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.535 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:39.535 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.702052+0000", 2024-09-21T12:27:39.535 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0028958199999999999 2024-09-21T12:27:39.535 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.535 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.535 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:39.535 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.702173+0000", 2024-09-21T12:27:39.535 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000120518 2024-09-21T12:27:39.536 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.536 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.536 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:27:39.536 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.702551+0000", 2024-09-21T12:27:39.536 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00037817800000000001 2024-09-21T12:27:39.536 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.536 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.536 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.536 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.707944+0000", 2024-09-21T12:27:39.536 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0053930769999999996 2024-09-21T12:27:39.536 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.536 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.536 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:27:39.536 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.709806+0000", 2024-09-21T12:27:39.537 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001861824 2024-09-21T12:27:39.537 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.537 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.537 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.537 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.843235+0000", 2024-09-21T12:27:39.537 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13342960200000001 2024-09-21T12:27:39.537 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.537 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.537 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:39.537 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.843288+0000", 2024-09-21T12:27:39.537 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2324000000000003e-05 2024-09-21T12:27:39.537 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.537 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.537 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:39.537 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.843327+0000", 2024-09-21T12:27:39.538 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8933999999999997e-05 2024-09-21T12:27:39.538 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.538 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:39.538 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.538 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.538 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.538 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:99033 8.8 8:1a2a2a5f:::benchmark_data_smithi114_104936_object99032:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-21T12:27:39.538 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:57.702897+0000", 2024-09-21T12:27:39.538 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 101.715085278, 2024-09-21T12:27:39.538 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14051256500000001, 2024-09-21T12:27:39.538 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:39.538 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:39.538 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:27:39.538 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:27:39.539 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:27:39.539 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:27:39.539 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 99033 2024-09-21T12:27:39.539 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.539 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:39.539 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.539 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:39.539 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.702897+0000", 2024-09-21T12:27:39.539 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.539 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.539 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.539 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:39.539 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.702897+0000", 2024-09-21T12:27:39.539 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:39.540 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.540 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.540 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:39.540 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.702898+0000", 2024-09-21T12:27:39.540 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.246e-06 2024-09-21T12:27:39.540 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.540 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.540 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:39.540 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.702950+0000", 2024-09-21T12:27:39.540 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1962000000000002e-05 2024-09-21T12:27:39.540 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.540 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.540 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:39.540 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.702966+0000", 2024-09-21T12:27:39.541 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6586999999999999e-05 2024-09-21T12:27:39.541 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.541 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.541 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:39.541 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.702974+0000", 2024-09-21T12:27:39.541 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1079999999999993e-06 2024-09-21T12:27:39.541 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.541 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.541 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:39.541 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.705534+0000", 2024-09-21T12:27:39.541 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0025595700000000002 2024-09-21T12:27:39.541 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.541 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.541 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:39.541 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.705666+0000", 2024-09-21T12:27:39.542 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013246500000000001 2024-09-21T12:27:39.542 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.542 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.542 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:27:39.542 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.705830+0000", 2024-09-21T12:27:39.542 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016372199999999999 2024-09-21T12:27:39.542 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.542 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.542 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.542 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.709794+0000", 2024-09-21T12:27:39.542 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0039634759999999996 2024-09-21T12:27:39.542 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.542 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.542 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:27:39.543 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.709866+0000", 2024-09-21T12:27:39.543 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2502999999999993e-05 2024-09-21T12:27:39.543 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.543 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.543 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:39.543 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.843354+0000", 2024-09-21T12:27:39.543 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13348826499999999 2024-09-21T12:27:39.543 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.543 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.543 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:27:39.543 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.843385+0000", 2024-09-21T12:27:39.543 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0281999999999999e-05 2024-09-21T12:27:39.543 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:39.543 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:39.544 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:27:39.544 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:57.843409+0000", 2024-09-21T12:27:39.544 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4379000000000002e-05 2024-09-21T12:27:39.544 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.544 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:39.544 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.544 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:39.544 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:39.544 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:27:39.544 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:39.573 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:39.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:39.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:39.674 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:39.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:39.717 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:39.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:39.818 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:39.848 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:39.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:39.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:39.895 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:27:39.950 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:39.993 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:39.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:39.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:40.094 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:40.124 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:40.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:40.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:40.226 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:40.251 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:40.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:40.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:40.276 INFO:teuthology.orchestra.run.smithi114.stderr:no valid command found; 10 closest matches: 2024-09-21T12:27:40.276 INFO:teuthology.orchestra.run.smithi114.stderr:0 2024-09-21T12:27:40.276 INFO:teuthology.orchestra.run.smithi114.stderr:1 2024-09-21T12:27:40.276 INFO:teuthology.orchestra.run.smithi114.stderr:2 2024-09-21T12:27:40.276 INFO:teuthology.orchestra.run.smithi114.stderr:abort 2024-09-21T12:27:40.276 INFO:teuthology.orchestra.run.smithi114.stderr:assert 2024-09-21T12:27:40.276 INFO:teuthology.orchestra.run.smithi114.stderr:bench [] [] [] [] 2024-09-21T12:27:40.276 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs debug_inject_read_zeros 2024-09-21T12:27:40.276 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs files list 2024-09-21T12:27:40.276 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs stats 2024-09-21T12:27:40.277 INFO:teuthology.orchestra.run.smithi114.stderr:bluestore allocator dump block 2024-09-21T12:27:40.277 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: invalid command 2024-09-21T12:27:40.353 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:40.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:40.382 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:40.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:40.463 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T12:27:40.484 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:40.533 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:40.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:40.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:40.635 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:40.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:40.676 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:40.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:40.776 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:40.816 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:27:40.818 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:40.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:40.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:40.920 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:40.960 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:40.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:40.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:40.987 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T12:27:40.987 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T12:27:40.987 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.5 'trim stale osdmaps' 2024-09-21T12:27:41.062 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:41.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:41.116 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:41.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:41.189 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:41 smithi114 ceph-mon[30324]: pgmap v358: 178 pgs: 178 active+clean; 2.1 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.47k op/s 2024-09-21T12:27:41.217 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:41.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:41 smithi183 ceph-mon[33277]: pgmap v358: 178 pgs: 178 active+clean; 2.1 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.47k op/s 2024-09-21T12:27:41.255 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:41.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:41.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:41.357 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:41.375 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:27:41.375 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:27:41.375 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.376 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:231771 8.5 8:abd6ca3f:::benchmark_data_smithi114_104936_object101355:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-09-21T12:27:41.376 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:27:41.369006+0000", 2024-09-21T12:27:41.376 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0043280920000000004, 2024-09-21T12:27:41.376 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0043575879999999999, 2024-09-21T12:27:41.376 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:41.376 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:41.376 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:27:41.376 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:27:41.376 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:27:41.376 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:27:41.376 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 231771 2024-09-21T12:27:41.376 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.376 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:41.376 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.377 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:41.377 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.369006+0000", 2024-09-21T12:27:41.377 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:41.377 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.377 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.377 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:41.377 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.369006+0000", 2024-09-21T12:27:41.377 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:41.377 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.377 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.377 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:41.377 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.369007+0000", 2024-09-21T12:27:41.377 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2500000000000001e-06 2024-09-21T12:27:41.378 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.378 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.378 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:41.378 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.369012+0000", 2024-09-21T12:27:41.378 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8930000000000004e-06 2024-09-21T12:27:41.378 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.378 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.378 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:41.378 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.369023+0000", 2024-09-21T12:27:41.378 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0424e-05 2024-09-21T12:27:41.378 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.378 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.378 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:41.378 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.369030+0000", 2024-09-21T12:27:41.379 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5769999999999996e-06 2024-09-21T12:27:41.379 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.379 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.379 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:41.379 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.369081+0000", 2024-09-21T12:27:41.379 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0590000000000002e-05 2024-09-21T12:27:41.379 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.379 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.379 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:41.379 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.369182+0000", 2024-09-21T12:27:41.379 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100869 2024-09-21T12:27:41.379 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.379 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.379 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,7", 2024-09-21T12:27:41.379 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.369263+0000", 2024-09-21T12:27:41.380 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1074000000000003e-05 2024-09-21T12:27:41.380 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:41.380 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:41.380 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:41.380 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.380 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.380 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:231780 8.5 8:a3434b13:::benchmark_data_smithi114_104936_object101364:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-09-21T12:27:41.380 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:27:41.370486+0000", 2024-09-21T12:27:41.380 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0028478349999999999, 2024-09-21T12:27:41.380 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0029429220000000002, 2024-09-21T12:27:41.380 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:41.380 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:41.380 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:27:41.380 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:27:41.380 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:27:41.380 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:27:41.381 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 231780 2024-09-21T12:27:41.381 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.381 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:41.381 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.381 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:41.381 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.370486+0000", 2024-09-21T12:27:41.381 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:41.381 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.381 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.381 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:41.381 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.370486+0000", 2024-09-21T12:27:41.381 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:41.381 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.381 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.381 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:41.382 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.370487+0000", 2024-09-21T12:27:41.382 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4600000000000003e-07 2024-09-21T12:27:41.382 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.382 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.382 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:41.382 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.370492+0000", 2024-09-21T12:27:41.382 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1960000000000001e-06 2024-09-21T12:27:41.382 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.382 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.382 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:41.382 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.370505+0000", 2024-09-21T12:27:41.382 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3524e-05 2024-09-21T12:27:41.382 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.383 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.383 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:41.383 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.370512+0000", 2024-09-21T12:27:41.383 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1350000000000003e-06 2024-09-21T12:27:41.383 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.383 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.383 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:41.383 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.371683+0000", 2024-09-21T12:27:41.383 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001170393 2024-09-21T12:27:41.383 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.383 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.383 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:41.383 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.371781+0000", 2024-09-21T12:27:41.383 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8245000000000006e-05 2024-09-21T12:27:41.384 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.384 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.384 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,7", 2024-09-21T12:27:41.384 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.371914+0000", 2024-09-21T12:27:41.384 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013279000000000001 2024-09-21T12:27:41.384 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:41.384 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:41.384 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:41.384 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.384 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.384 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop_reply(client.25022.0:231771 8.5 e223/160)", 2024-09-21T12:27:41.384 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:27:41.371683+0000", 2024-09-21T12:27:41.384 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0016509599999999999, 2024-09-21T12:27:41.384 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0017995019999999999, 2024-09-21T12:27:41.384 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:41.385 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:41.385 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-09-21T12:27:41.385 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:41.385 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.385 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:41.385 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.371683+0000", 2024-09-21T12:27:41.385 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:41.385 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.385 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.385 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:41.385 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.371683+0000", 2024-09-21T12:27:41.385 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:41.385 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.385 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.385 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:41.386 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.371684+0000", 2024-09-21T12:27:41.386 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9100000000000003e-07 2024-09-21T12:27:41.386 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.386 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.386 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:41.386 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.371689+0000", 2024-09-21T12:27:41.386 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4560000000000002e-06 2024-09-21T12:27:41.386 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.386 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.386 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:41.386 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.371698+0000", 2024-09-21T12:27:41.386 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8169999999999994e-06 2024-09-21T12:27:41.386 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.386 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.386 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:41.387 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.371705+0000", 2024-09-21T12:27:41.387 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6919999999999996e-06 2024-09-21T12:27:41.387 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:41.387 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:41.387 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:41.387 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.387 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.387 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.25022.0:231777 8.4 e223/208 8:29a4914f:::benchmark_data_smithi114_104936_object101361:head v 223'14662, mlcod=223'14660)", 2024-09-21T12:27:41.387 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:27:41.371926+0000", 2024-09-21T12:27:41.387 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0014081370000000001, 2024-09-21T12:27:41.387 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0015951889999999999, 2024-09-21T12:27:41.387 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:41.387 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:41.387 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:27:41.387 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:41.388 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.388 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:41.388 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.371926+0000", 2024-09-21T12:27:41.388 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:41.388 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.388 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.388 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:41.388 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.371926+0000", 2024-09-21T12:27:41.388 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:41.388 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.388 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.388 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:41.388 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.371927+0000", 2024-09-21T12:27:41.388 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3200000000000004e-07 2024-09-21T12:27:41.388 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.389 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.389 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:41.389 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.371932+0000", 2024-09-21T12:27:41.389 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0640000000000001e-06 2024-09-21T12:27:41.389 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.389 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.389 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:41.389 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.371941+0000", 2024-09-21T12:27:41.389 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3819999999999993e-06 2024-09-21T12:27:41.389 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.389 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.389 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:41.389 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.371947+0000", 2024-09-21T12:27:41.389 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9819999999999996e-06 2024-09-21T12:27:41.390 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.390 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.390 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:41.390 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.372020+0000", 2024-09-21T12:27:41.390 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2627999999999996e-05 2024-09-21T12:27:41.390 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.390 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.390 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:41.390 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.372037+0000", 2024-09-21T12:27:41.390 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7510999999999999e-05 2024-09-21T12:27:41.390 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:41.390 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:41.390 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:41.390 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.391 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.391 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.25022.0:231776 8.3 e223/160 8:cfeeeb68:::benchmark_data_smithi114_104936_object101360:head v 223'14149, mlcod=223'14145)", 2024-09-21T12:27:41.391 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:27:41.372672+0000", 2024-09-21T12:27:41.391 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.00066266999999999997, 2024-09-21T12:27:41.391 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00089447099999999998, 2024-09-21T12:27:41.391 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:27:41.391 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:27:41.391 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:27:41.391 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:27:41.391 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.391 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:27:41.391 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.372672+0000", 2024-09-21T12:27:41.391 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:41.391 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.392 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.392 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:27:41.392 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.372672+0000", 2024-09-21T12:27:41.392 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:27:41.392 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.392 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.392 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:27:41.392 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.372672+0000", 2024-09-21T12:27:41.392 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3e-07 2024-09-21T12:27:41.392 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.392 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.392 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:27:41.392 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.372677+0000", 2024-09-21T12:27:41.392 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6450000000000004e-06 2024-09-21T12:27:41.392 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.393 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.393 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:27:41.393 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.372690+0000", 2024-09-21T12:27:41.393 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2937e-05 2024-09-21T12:27:41.393 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.393 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.393 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:27:41.393 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.372696+0000", 2024-09-21T12:27:41.393 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8819999999999998e-06 2024-09-21T12:27:41.393 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.393 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.393 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:27:41.393 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.372733+0000", 2024-09-21T12:27:41.393 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6696999999999998e-05 2024-09-21T12:27:41.394 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:27:41.394 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:27:41.394 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:27:41.394 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:41.372749+0000", 2024-09-21T12:27:41.394 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6053e-05 2024-09-21T12:27:41.394 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:41.394 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:27:41.394 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:41.394 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:27:41.394 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-21T12:27:41.394 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 5 2024-09-21T12:27:41.394 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:27:41.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:41.395 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:41.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:41.496 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:41.521 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:41.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:41.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:41.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:41 smithi114 ceph-mon[40867]: pgmap v358: 178 pgs: 178 active+clean; 2.1 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.47k op/s 2024-09-21T12:27:41.546 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T12:27:41.622 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:41.648 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:41.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:41.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:41.750 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:41.790 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:41.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:41.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:41.892 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:41.921 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:41.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:41.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:42.023 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:42.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:42.066 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:42.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:42.090 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T12:27:42.143 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:27:42.144 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:27:42.167 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:42.183 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:27:42.183 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:27:42.200 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:42 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1099696625' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:27:42.223 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:27:42.224 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:27:42.229 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:42.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:42.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:42.254 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:27:42.264 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:27:42.264 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:27:42.268 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:27:42.290 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:27:42.291 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:27:42.325 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:27:42.325 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:27:42.331 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:42.360 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:27:42.361 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:27:42.380 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:42.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:42.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:42.394 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:27:42.394 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:27:42.407 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:27:42.482 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:42.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:42 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1099696625' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:27:42.508 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:42.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:42.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:42.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:42 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1099696625' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:27:42.609 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:42.643 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:42.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:42.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:42.745 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:42.767 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:42.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:42.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:42.869 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:42.903 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:42.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:42.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:42.983 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:27:42.983 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:27:43.004 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:43.024 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:27:42 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:27:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:27:43.025 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:43.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:43.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:43.126 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:43.153 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:43.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:43.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:43.255 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:43.295 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:43.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:43.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:43.396 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:43.423 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:43.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:43.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:43.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:43 smithi183 ceph-mon[33277]: pgmap v359: 178 pgs: 178 active+clean; 2.1 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 927 op/s 2024-09-21T12:27:43.524 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:43.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:43 smithi114 ceph-mon[30324]: pgmap v359: 178 pgs: 178 active+clean; 2.1 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 927 op/s 2024-09-21T12:27:43.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:43 smithi114 ceph-mon[40867]: pgmap v359: 178 pgs: 178 active+clean; 2.1 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 927 op/s 2024-09-21T12:27:43.551 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:43.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:43.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:43.653 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:43.699 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:43.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:43.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:43.801 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:43.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:43.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:43.942 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:43.978 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:43.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:43.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:43.995 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T12:27:44.079 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:44.106 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:44.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:44.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:44.208 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:44.220 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-21T12:27:44.234 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:44.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:44.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:44.324 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:27:44.336 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:44.373 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:44.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:44.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:44.456 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:44 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3815063523' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:27:44.456 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:44 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:27:44.456 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:44 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3815063523' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:27:44.457 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:44 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:27:44.475 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:44.485 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:44 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3815063523' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:27:44.485 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:44 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:27:44.503 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:44.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:44.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:44.605 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:44.636 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:44.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:44.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:44.738 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:44.769 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:44.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:44.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:44.871 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:44.907 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:44.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:44.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:44.924 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:27:44.924 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:27:45.009 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:45.047 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:45.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:45.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:45.149 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:45.187 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:45.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:45.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:45.288 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:45.329 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:45.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:45.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:45.424 INFO:teuthology.orchestra.run.smithi114.stderr:no valid command found; 10 closest matches: 2024-09-21T12:27:45.425 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:45 smithi114 ceph-mon[40867]: pgmap v360: 178 pgs: 178 active+clean; 1.7 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s 2024-09-21T12:27:45.425 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:45 smithi114 ceph-mon[40867]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:27:45.425 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:45 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:27:45.425 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:45 smithi114 ceph-mon[40867]: osdmap e224: 8 total, 8 up, 7 in 2024-09-21T12:27:45.425 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:45 smithi114 ceph-mon[30324]: pgmap v360: 178 pgs: 178 active+clean; 1.7 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s 2024-09-21T12:27:45.425 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:45 smithi114 ceph-mon[30324]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:27:45.425 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:45 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:27:45.425 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:45 smithi114 ceph-mon[30324]: osdmap e224: 8 total, 8 up, 7 in 2024-09-21T12:27:45.426 INFO:teuthology.orchestra.run.smithi114.stderr:0 2024-09-21T12:27:45.426 INFO:teuthology.orchestra.run.smithi114.stderr:1 2024-09-21T12:27:45.426 INFO:teuthology.orchestra.run.smithi114.stderr:2 2024-09-21T12:27:45.426 INFO:teuthology.orchestra.run.smithi114.stderr:abort 2024-09-21T12:27:45.427 INFO:teuthology.orchestra.run.smithi114.stderr:assert 2024-09-21T12:27:45.427 INFO:teuthology.orchestra.run.smithi114.stderr:bench [] [] [] [] 2024-09-21T12:27:45.427 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs debug_inject_read_zeros 2024-09-21T12:27:45.427 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs files list 2024-09-21T12:27:45.427 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs stats 2024-09-21T12:27:45.427 INFO:teuthology.orchestra.run.smithi114.stderr:bluestore allocator dump block 2024-09-21T12:27:45.427 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: invalid command 2024-09-21T12:27:45.431 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:45.459 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:45.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:45.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:45.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:45 smithi183 ceph-mon[33277]: pgmap v360: 178 pgs: 178 active+clean; 1.7 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s 2024-09-21T12:27:45.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:45 smithi183 ceph-mon[33277]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:27:45.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:45 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:27:45.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:45 smithi183 ceph-mon[33277]: osdmap e224: 8 total, 8 up, 7 in 2024-09-21T12:27:45.560 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:45.589 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:45.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:45.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:45.633 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T12:27:45.691 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:45.735 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:45.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:45.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:45.837 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:45.868 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:45.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:45.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:45.970 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:45.974 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:27:46.007 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:46.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:46.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:46.109 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:46.138 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:46.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:46.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:46.214 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T12:27:46.215 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T12:27:46.215 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.6 'trim stale osdmaps' 2024-09-21T12:27:46.240 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:46.292 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:46.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:46.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:46.394 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:46.421 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:46.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:46.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:46.506 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:27:46.523 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:46.525 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:27:46.525 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:27:46.550 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:46.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:46.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:46.651 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:46.661 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:27:46.677 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:46.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:46.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:46.779 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:46.805 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:46.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:46.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:46.906 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:46.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:46.961 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:46.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:47.061 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:47.104 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:47.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:47.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:47.154 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:27:47 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:27:47.168 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T12:27:47.191 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T12:27:47.206 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:47.256 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:47.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:47.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:47.358 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:47.395 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:47.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:47.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:47.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:47 smithi183 ceph-mon[33277]: pgmap v362: 178 pgs: 178 active+clean; 1.7 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 581 B/s rd, 0 B/s wr, 1.34k op/s 2024-09-21T12:27:47.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:47 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:27:47.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:47 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:27:47.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:47 smithi183 ceph-mon[33277]: osdmap e225: 8 total, 8 up, 7 in 2024-09-21T12:27:47.497 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:47.500 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:27:47.519 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:47.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:47.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:47.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:47 smithi114 ceph-mon[40867]: pgmap v362: 178 pgs: 178 active+clean; 1.7 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 581 B/s rd, 0 B/s wr, 1.34k op/s 2024-09-21T12:27:47.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:47 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:27:47.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:47 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:27:47.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:47 smithi114 ceph-mon[40867]: osdmap e225: 8 total, 8 up, 7 in 2024-09-21T12:27:47.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:47 smithi114 ceph-mon[30324]: pgmap v362: 178 pgs: 178 active+clean; 1.7 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 581 B/s rd, 0 B/s wr, 1.34k op/s 2024-09-21T12:27:47.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:47 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:27:47.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:47 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:27:47.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:47 smithi114 ceph-mon[30324]: osdmap e225: 8 total, 8 up, 7 in 2024-09-21T12:27:47.621 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:47.647 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:47.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:47.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:47.749 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:47.783 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:47.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:47.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:47.885 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:47.914 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:47.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:47.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:48.016 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:48.038 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:27:48.038 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:27:48.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:48.056 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:48.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:48.157 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:48.201 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:48.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:48.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:48.236 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:48 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:27:48.306 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:48.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:48.341 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:48.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:48.442 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:48.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:48.487 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:48.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:48.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:48 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:27:48.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:48 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:27:48.588 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:48.615 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:48.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:48.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:48.718 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:48.751 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:48.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:48.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:48.847 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T12:27:48.852 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:48.892 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:48.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:48.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:48.994 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:49.018 INFO:teuthology.orchestra.run.smithi114.stderr:no valid command found; 10 closest matches: 2024-09-21T12:27:49.018 INFO:teuthology.orchestra.run.smithi114.stderr:0 2024-09-21T12:27:49.018 INFO:teuthology.orchestra.run.smithi114.stderr:1 2024-09-21T12:27:49.018 INFO:teuthology.orchestra.run.smithi114.stderr:2 2024-09-21T12:27:49.018 INFO:teuthology.orchestra.run.smithi114.stderr:abort 2024-09-21T12:27:49.018 INFO:teuthology.orchestra.run.smithi114.stderr:assert 2024-09-21T12:27:49.018 INFO:teuthology.orchestra.run.smithi114.stderr:bench [] [] [] [] 2024-09-21T12:27:49.018 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs debug_inject_read_zeros 2024-09-21T12:27:49.018 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs files list 2024-09-21T12:27:49.018 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs stats 2024-09-21T12:27:49.018 INFO:teuthology.orchestra.run.smithi114.stderr:bluestore allocator dump block 2024-09-21T12:27:49.018 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: invalid command 2024-09-21T12:27:49.031 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:49.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:49.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:49.133 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:49.169 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:49.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:49.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:49.190 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:27:49.271 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:49.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:49 smithi114 ceph-mon[40867]: pgmap v364: 178 pgs: 178 active+clean; 1.3 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.71k op/s 2024-09-21T12:27:49.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:49 smithi114 ceph-mon[30324]: pgmap v364: 178 pgs: 178 active+clean; 1.3 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.71k op/s 2024-09-21T12:27:49.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:49.298 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:49.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:49.398 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:49.426 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:49.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:49.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:49.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:49 smithi183 ceph-mon[33277]: pgmap v364: 178 pgs: 178 active+clean; 1.3 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.71k op/s 2024-09-21T12:27:49.528 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:49.565 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:49.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:49.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:49.667 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:49.695 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:49.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:49.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:49.797 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:49.800 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:27:49.800 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:27:49.830 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:49.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:49.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:49.932 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:49.958 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:49.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:49.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:50.060 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:50.090 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:50.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:50.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:50.192 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:50.213 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:50.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:50.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:50.314 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:50.352 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:50.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:50.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:50.453 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:50.474 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:50.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:50.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:50.576 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:50.577 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T12:27:50.619 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:50.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:50.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:50.723 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:50.747 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:27:50.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:50.779 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:50.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:50.785 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T12:27:50.786 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T12:27:50.786 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.7 'trim stale osdmaps' 2024-09-21T12:27:50.883 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:50.927 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:50.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:50.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:51.030 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:51.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:51.068 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:51.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:51.169 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:51.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:51.194 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:51.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:51.295 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:51.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:51.339 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:51.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:51.426 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:27:51.439 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:27:51.441 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:51.454 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:27:51.455 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:27:51.464 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:51.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:51.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:51.474 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:27:51.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:51 smithi183 ceph-mon[33277]: pgmap v365: 178 pgs: 178 active+clean; 1.3 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.54k op/s 2024-09-21T12:27:51.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:51 smithi114 ceph-mon[40867]: pgmap v365: 178 pgs: 178 active+clean; 1.3 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.54k op/s 2024-09-21T12:27:51.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:51 smithi114 ceph-mon[30324]: pgmap v365: 178 pgs: 178 active+clean; 1.3 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.54k op/s 2024-09-21T12:27:51.491 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:27:51.491 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:27:51.528 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:27:51.529 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:27:51.560 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:27:51.560 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:27:51.565 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:51.593 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:27:51.593 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:27:51.594 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:51.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:51.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:51.627 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:27:51.627 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:27:51.660 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:27:51.660 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:27:51.691 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:27:51.691 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:27:51.696 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:51.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:51.723 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:51.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:51.824 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:51.852 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:51.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:51.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:51.954 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:51.998 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:51.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:51.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:52.100 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:52.125 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:52.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:52.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:52.228 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:52.267 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:52.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:52.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:52.364 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:27:52.364 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-21T12:27:52.364 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.364 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.25022.0:245853 8.5 e225/160 8:a9568923:::benchmark_data_smithi114_104936_object115437:head v 225'15366, mlcod=225'15364)", 2024-09-21T12:27:52.364 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:27:52.354301+0000", 2024-09-21T12:27:52.364 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0072560130000000004, 2024-09-21T12:27:52.364 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0072828959999999996, 2024-09-21T12:27:52.364 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:27:52.364 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:27:52.364 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:27:52.365 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:27:52.365 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.365 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:27:52.365 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.354301+0000", 2024-09-21T12:27:52.365 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:27:52.365 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.365 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.365 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:27:52.365 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.354301+0000", 2024-09-21T12:27:52.365 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:27:52.365 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.365 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.365 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:27:52.366 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.354301+0000", 2024-09-21T12:27:52.366 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4799999999999998e-07 2024-09-21T12:27:52.366 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.366 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.366 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:27:52.366 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.354308+0000", 2024-09-21T12:27:52.366 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3319999999999997e-06 2024-09-21T12:27:52.366 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.366 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.366 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:27:52.366 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.354316+0000", 2024-09-21T12:27:52.366 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9729999999999997e-06 2024-09-21T12:27:52.366 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.367 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.367 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:27:52.367 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.354322+0000", 2024-09-21T12:27:52.367 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9839999999999998e-06 2024-09-21T12:27:52.367 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.367 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.367 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:27:52.367 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.354643+0000", 2024-09-21T12:27:52.367 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00032118000000000001 2024-09-21T12:27:52.367 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.367 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.367 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:27:52.368 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.354664+0000", 2024-09-21T12:27:52.368 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1226e-05 2024-09-21T12:27:52.368 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:27:52.368 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:27:52.368 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:27:52.368 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.368 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.368 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25022.0:245854 8.6 8.bdbbba96 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-09-21T12:27:52.368 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:27:52.354465+0000", 2024-09-21T12:27:52.368 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.007091895, 2024-09-21T12:27:52.368 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0071707630000000001, 2024-09-21T12:27:52.368 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:27:52.368 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:27:52.369 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "queued for pg", 2024-09-21T12:27:52.369 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:27:52.369 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25022", 2024-09-21T12:27:52.369 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:27:52.369 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 245854 2024-09-21T12:27:52.369 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.369 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:27:52.369 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.369 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:27:52.369 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.354465+0000", 2024-09-21T12:27:52.369 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:27:52.369 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.370 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.370 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:27:52.370 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.354465+0000", 2024-09-21T12:27:52.370 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:27:52.370 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.370 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.370 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:27:52.370 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.354466+0000", 2024-09-21T12:27:52.370 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3480000000000001e-06 2024-09-21T12:27:52.370 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.370 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.370 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:27:52.371 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.354470+0000", 2024-09-21T12:27:52.371 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3420000000000001e-06 2024-09-21T12:27:52.371 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.371 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.371 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:27:52.371 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.354481+0000", 2024-09-21T12:27:52.371 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0764999999999999e-05 2024-09-21T12:27:52.371 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.371 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.371 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:27:52.371 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.354488+0000", 2024-09-21T12:27:52.372 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7129999999999999e-06 2024-09-21T12:27:52.372 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:27:52.372 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:27:52.372 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:27:52.372 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.372 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.372 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25022.0:245857 8.2 8:453089c8:::benchmark_data_smithi114_104936_object115441:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-09-21T12:27:52.372 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:27:52.355894+0000", 2024-09-21T12:27:52.372 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0056624739999999998, 2024-09-21T12:27:52.372 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0057893750000000003, 2024-09-21T12:27:52.372 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:27:52.372 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:27:52.373 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:27:52.373 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:27:52.373 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25022", 2024-09-21T12:27:52.373 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:27:52.373 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 245857 2024-09-21T12:27:52.373 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.373 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:27:52.373 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.373 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:27:52.373 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.355894+0000", 2024-09-21T12:27:52.373 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:27:52.373 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.373 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.374 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:27:52.374 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.355894+0000", 2024-09-21T12:27:52.374 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:27:52.374 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.374 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.374 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:27:52.374 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.355895+0000", 2024-09-21T12:27:52.374 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5099999999999998e-07 2024-09-21T12:27:52.374 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.374 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.374 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:27:52.374 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.355899+0000", 2024-09-21T12:27:52.374 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0790000000000002e-06 2024-09-21T12:27:52.374 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.375 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.375 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:27:52.375 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.355912+0000", 2024-09-21T12:27:52.375 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2656999999999999e-05 2024-09-21T12:27:52.375 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.375 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.375 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:27:52.375 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.355919+0000", 2024-09-21T12:27:52.375 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7429999999999996e-06 2024-09-21T12:27:52.375 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.375 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.375 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:27:52.375 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.355971+0000", 2024-09-21T12:27:52.375 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2315999999999999e-05 2024-09-21T12:27:52.376 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.376 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.376 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:27:52.376 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.356074+0000", 2024-09-21T12:27:52.376 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010275199999999999 2024-09-21T12:27:52.376 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.376 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.376 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,5", 2024-09-21T12:27:52.376 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.356170+0000", 2024-09-21T12:27:52.376 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6265000000000001e-05 2024-09-21T12:27:52.376 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.376 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.376 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:27:52.376 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.359305+0000", 2024-09-21T12:27:52.377 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0031351899999999999 2024-09-21T12:27:52.377 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.377 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.377 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:27:52.377 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.360610+0000", 2024-09-21T12:27:52.377 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001305304 2024-09-21T12:27:52.377 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:27:52.377 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:27:52.377 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:27:52.377 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.377 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.377 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25022.0:245859 8.e 8.8b975d3e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-09-21T12:27:52.377 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:27:52.357817+0000", 2024-09-21T12:27:52.377 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0037400739999999999, 2024-09-21T12:27:52.377 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0039279520000000002, 2024-09-21T12:27:52.378 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:27:52.378 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:27:52.378 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "queued for pg", 2024-09-21T12:27:52.378 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:27:52.378 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25022", 2024-09-21T12:27:52.378 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:27:52.378 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 245859 2024-09-21T12:27:52.378 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.378 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:27:52.378 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.378 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:27:52.378 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.357817+0000", 2024-09-21T12:27:52.378 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:27:52.378 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.379 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.379 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:27:52.379 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.357817+0000", 2024-09-21T12:27:52.379 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:27:52.379 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.379 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.379 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:27:52.379 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.357818+0000", 2024-09-21T12:27:52.379 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0529999999999999e-06 2024-09-21T12:27:52.379 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.379 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.379 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:27:52.379 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.357829+0000", 2024-09-21T12:27:52.380 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1671000000000001e-05 2024-09-21T12:27:52.380 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.380 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.380 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:27:52.380 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.357865+0000", 2024-09-21T12:27:52.380 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5979999999999998e-05 2024-09-21T12:27:52.380 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.380 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.380 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:27:52.380 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.357879+0000", 2024-09-21T12:27:52.380 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3198e-05 2024-09-21T12:27:52.380 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:27:52.380 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:27:52.380 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:27:52.381 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.381 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.381 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.25022.0:245855 8.1 e225/160 8:8225aa93:::benchmark_data_smithi114_104936_object115439:head v 225'15533, mlcod=225'15531)", 2024-09-21T12:27:52.381 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:27:52.358576+0000", 2024-09-21T12:27:52.381 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0029810050000000001, 2024-09-21T12:27:52.381 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0032121490000000001, 2024-09-21T12:27:52.381 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:27:52.381 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:27:52.381 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:27:52.381 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:27:52.381 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.381 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:27:52.381 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.358576+0000", 2024-09-21T12:27:52.381 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:27:52.382 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.382 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.382 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:27:52.382 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.358576+0000", 2024-09-21T12:27:52.382 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:27:52.382 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.382 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.382 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:27:52.382 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.358577+0000", 2024-09-21T12:27:52.382 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.04e-06 2024-09-21T12:27:52.382 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.382 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.382 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:27:52.383 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.358584+0000", 2024-09-21T12:27:52.383 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9650000000000002e-06 2024-09-21T12:27:52.383 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.383 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.383 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:27:52.383 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.358595+0000", 2024-09-21T12:27:52.383 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1653e-05 2024-09-21T12:27:52.383 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.383 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.383 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:27:52.383 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.358603+0000", 2024-09-21T12:27:52.383 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2080000000000003e-06 2024-09-21T12:27:52.383 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.384 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.384 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:27:52.384 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.358758+0000", 2024-09-21T12:27:52.384 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000155475 2024-09-21T12:27:52.384 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.384 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.384 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:27:52.384 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.358778+0000", 2024-09-21T12:27:52.384 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0134000000000001e-05 2024-09-21T12:27:52.384 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:27:52.384 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:27:52.384 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:27:52.384 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.384 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.385 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.25022.0:245864 8.5 e225/160 8:ae6662fa:::benchmark_data_smithi114_104936_object115448:head v 225'15367, mlcod=225'15364)", 2024-09-21T12:27:52.385 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:27:52.361091+0000", 2024-09-21T12:27:52.385 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.00046571200000000002, 2024-09-21T12:27:52.385 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000740882, 2024-09-21T12:27:52.385 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:27:52.385 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:27:52.385 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:27:52.385 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:27:52.385 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.385 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:27:52.385 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.361091+0000", 2024-09-21T12:27:52.385 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:27:52.385 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.386 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.386 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:27:52.386 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.361091+0000", 2024-09-21T12:27:52.386 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:27:52.386 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.386 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.386 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:27:52.386 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.361092+0000", 2024-09-21T12:27:52.386 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5199999999999996e-07 2024-09-21T12:27:52.386 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.386 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.386 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:27:52.386 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.361097+0000", 2024-09-21T12:27:52.386 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2599999999999996e-06 2024-09-21T12:27:52.387 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.387 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.387 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:27:52.387 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.361105+0000", 2024-09-21T12:27:52.387 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.2900000000000002e-06 2024-09-21T12:27:52.387 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.387 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.387 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:27:52.387 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.361112+0000", 2024-09-21T12:27:52.387 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8310000000000003e-06 2024-09-21T12:27:52.387 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.387 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.387 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:27:52.387 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.361225+0000", 2024-09-21T12:27:52.387 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000113008 2024-09-21T12:27:52.388 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.388 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.388 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:27:52.388 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.361286+0000", 2024-09-21T12:27:52.388 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1252000000000002e-05 2024-09-21T12:27:52.388 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:27:52.388 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:27:52.388 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:27:52.388 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.388 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.388 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25022.0:245841 8.6 8:6f92a1c5:::benchmark_data_smithi114_104936_object115425:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-09-21T12:27:52.388 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:27:52.345835+0000", 2024-09-21T12:27:52.389 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.015721254, 2024-09-21T12:27:52.389 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.016042008999999999, 2024-09-21T12:27:52.389 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:27:52.389 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:27:52.389 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:27:52.389 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:27:52.389 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25022", 2024-09-21T12:27:52.389 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:27:52.389 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 245841 2024-09-21T12:27:52.389 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.389 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:27:52.389 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.389 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:27:52.390 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.345835+0000", 2024-09-21T12:27:52.390 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:27:52.390 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.390 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.390 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:27:52.390 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.345835+0000", 2024-09-21T12:27:52.390 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:27:52.390 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.390 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.390 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:27:52.390 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.345837+0000", 2024-09-21T12:27:52.390 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3009999999999999e-06 2024-09-21T12:27:52.391 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.391 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.391 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:27:52.391 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.345841+0000", 2024-09-21T12:27:52.391 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5090000000000004e-06 2024-09-21T12:27:52.391 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.391 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.391 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:27:52.391 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.345856+0000", 2024-09-21T12:27:52.391 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5058999999999999e-05 2024-09-21T12:27:52.391 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.391 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.391 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:27:52.391 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.345863+0000", 2024-09-21T12:27:52.392 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1949999999999997e-06 2024-09-21T12:27:52.392 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.392 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.392 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:27:52.392 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.351407+0000", 2024-09-21T12:27:52.392 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0055439749999999996 2024-09-21T12:27:52.392 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.392 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.392 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:27:52.392 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.360769+0000", 2024-09-21T12:27:52.392 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0093615539999999994 2024-09-21T12:27:52.392 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.392 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.392 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 6,7", 2024-09-21T12:27:52.393 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.360872+0000", 2024-09-21T12:27:52.393 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000102914 2024-09-21T12:27:52.393 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:27:52.393 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:27:52.393 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:27:52.393 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.393 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.393 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:52.394 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25022.0:245845 8.e 8:75967b94:::benchmark_data_smithi114_104936_object115429:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-09-21T12:27:52.394 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:27:52.348589+0000", 2024-09-21T12:27:52.394 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.012967809, 2024-09-21T12:27:52.394 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.013345991999999999, 2024-09-21T12:27:52.394 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:27:52.394 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:27:52.394 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:27:52.395 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:27:52.395 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25022", 2024-09-21T12:27:52.395 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:27:52.395 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 245845 2024-09-21T12:27:52.395 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.395 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:27:52.395 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.395 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:27:52.395 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.348589+0000", 2024-09-21T12:27:52.395 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:27:52.395 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.395 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.396 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:27:52.396 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.348589+0000", 2024-09-21T12:27:52.396 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:27:52.396 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.396 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.396 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:27:52.396 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.348590+0000", 2024-09-21T12:27:52.396 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5099999999999999e-07 2024-09-21T12:27:52.396 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.396 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.396 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:27:52.396 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.348593+0000", 2024-09-21T12:27:52.396 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.6689999999999999e-06 2024-09-21T12:27:52.397 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.397 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.397 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:27:52.397 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.348601+0000", 2024-09-21T12:27:52.397 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6809999999999998e-06 2024-09-21T12:27:52.397 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.397 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.397 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:27:52.397 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.348606+0000", 2024-09-21T12:27:52.397 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6399999999999996e-06 2024-09-21T12:27:52.397 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.397 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.397 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:27:52.398 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.359801+0000", 2024-09-21T12:27:52.398 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.011195464 2024-09-21T12:27:52.398 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.398 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.398 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:27:52.398 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.359897+0000", 2024-09-21T12:27:52.398 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6243000000000007e-05 2024-09-21T12:27:52.398 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.398 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.398 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,7", 2024-09-21T12:27:52.398 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.359994+0000", 2024-09-21T12:27:52.398 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6191000000000001e-05 2024-09-21T12:27:52.398 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:27:52.399 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:27:52.399 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:27:52.399 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.399 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.399 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25022.0:245846 8.e 8.3ab9980e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-09-21T12:27:52.399 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:27:52.349674+0000", 2024-09-21T12:27:52.399 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.011882917999999999, 2024-09-21T12:27:52.399 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.012314789, 2024-09-21T12:27:52.399 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:27:52.399 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:27:52.399 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "queued for pg", 2024-09-21T12:27:52.399 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:27:52.399 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25022", 2024-09-21T12:27:52.399 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:27:52.400 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 245846 2024-09-21T12:27:52.400 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.400 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:27:52.400 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.400 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:27:52.400 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.349674+0000", 2024-09-21T12:27:52.400 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:27:52.400 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.400 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.400 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:27:52.400 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.349674+0000", 2024-09-21T12:27:52.400 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:27:52.400 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.400 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.400 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:27:52.401 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.349675+0000", 2024-09-21T12:27:52.401 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9100000000000002e-07 2024-09-21T12:27:52.401 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.401 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.401 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:27:52.401 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.349679+0000", 2024-09-21T12:27:52.401 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.33e-06 2024-09-21T12:27:52.401 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.401 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.401 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:27:52.401 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.349693+0000", 2024-09-21T12:27:52.401 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4388e-05 2024-09-21T12:27:52.401 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:27:52.401 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:27:52.402 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:27:52.402 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:27:52.349701+0000", 2024-09-21T12:27:52.402 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.216e-06 2024-09-21T12:27:52.402 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:27:52.402 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:27:52.402 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:27:52.402 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:27:52.402 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:27:52.402 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 9 2024-09-21T12:27:52.402 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:27:52.442 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:52.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:52.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:52.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:52 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3231393596' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:27:52.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:52 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3231393596' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:27:52.545 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:52.575 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:52.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:52.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:52.677 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:52.705 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:52.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:52.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:52.770 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:52 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3231393596' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:27:52.871 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:52.895 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:52.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:52.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:52.997 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:53.024 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:27:52 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:27:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:27:53.035 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:53.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:53.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:53.138 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:53.163 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:53.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:53.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:53.265 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:53.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:53 smithi114 ceph-mon[30324]: pgmap v366: 178 pgs: 178 active+clean; 1.2 GiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 985 op/s 2024-09-21T12:27:53.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:53 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:27:53.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:53 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:27:53.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:53 smithi114 ceph-mon[30324]: osdmap e226: 8 total, 8 up, 7 in 2024-09-21T12:27:53.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:53 smithi114 ceph-mon[40867]: pgmap v366: 178 pgs: 178 active+clean; 1.2 GiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 985 op/s 2024-09-21T12:27:53.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:53 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:27:53.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:53 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:27:53.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:53 smithi114 ceph-mon[40867]: osdmap e226: 8 total, 8 up, 7 in 2024-09-21T12:27:53.304 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:53.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:53.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:53.406 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:53.436 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:53.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:53.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:53.538 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:53.564 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:53.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:53.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:53.666 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:53.710 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:53.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:53.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:53.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:53 smithi183 ceph-mon[33277]: pgmap v366: 178 pgs: 178 active+clean; 1.2 GiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 985 op/s 2024-09-21T12:27:53.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:53 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:27:53.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:53 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:27:53.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:53 smithi183 ceph-mon[33277]: osdmap e226: 8 total, 8 up, 7 in 2024-09-21T12:27:53.812 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:53.836 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:53.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:53.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:53.880 INFO:teuthology.orchestra.run.smithi114.stderr:no valid command found; 10 closest matches: 2024-09-21T12:27:53.880 INFO:teuthology.orchestra.run.smithi114.stderr:0 2024-09-21T12:27:53.881 INFO:teuthology.orchestra.run.smithi114.stderr:1 2024-09-21T12:27:53.881 INFO:teuthology.orchestra.run.smithi114.stderr:2 2024-09-21T12:27:53.881 INFO:teuthology.orchestra.run.smithi114.stderr:abort 2024-09-21T12:27:53.881 INFO:teuthology.orchestra.run.smithi114.stderr:assert 2024-09-21T12:27:53.881 INFO:teuthology.orchestra.run.smithi114.stderr:bench [] [] [] [] 2024-09-21T12:27:53.881 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs debug_inject_read_zeros 2024-09-21T12:27:53.881 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs files list 2024-09-21T12:27:53.881 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs stats 2024-09-21T12:27:53.881 INFO:teuthology.orchestra.run.smithi114.stderr:bluestore allocator dump block 2024-09-21T12:27:53.881 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: invalid command 2024-09-21T12:27:53.938 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:53.964 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:53.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:53.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:54.065 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:54.091 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:54.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:54.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:54.193 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:54.228 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:54.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:54.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:54.274 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-21T12:27:54.330 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:54.366 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:54.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:54.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:54.468 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:54.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:54.503 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:54.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:54.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:54 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1042415202' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:27:54.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:54 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:27:54.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:54 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1042415202' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:27:54.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:54 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:27:54.604 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:54.644 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:54.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:54.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:54.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:54 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1042415202' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:27:54.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:54 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:27:54.747 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:54.774 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:54.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:54.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:54.876 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:54.913 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:54.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:54.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:55.015 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:55.042 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:55.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:55.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:55.076 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:27:55.143 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:55.191 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:55.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:55.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:55.293 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:55.344 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:55.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:55.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:55.347 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T12:27:55.446 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:55.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:55.509 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:55.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:55 smithi114 ceph-mon[30324]: pgmap v368: 178 pgs: 178 active+clean; 961 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s 2024-09-21T12:27:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:55 smithi114 ceph-mon[30324]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:27:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:55 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:27:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:55 smithi114 ceph-mon[30324]: osdmap e227: 8 total, 8 up, 7 in 2024-09-21T12:27:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:55 smithi114 ceph-mon[30324]: 4.19 scrub starts 2024-09-21T12:27:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:55 smithi114 ceph-mon[30324]: 4.19 scrub ok 2024-09-21T12:27:55.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:55 smithi114 ceph-mon[40867]: pgmap v368: 178 pgs: 178 active+clean; 961 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s 2024-09-21T12:27:55.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:55 smithi114 ceph-mon[40867]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:27:55.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:55 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:27:55.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:55 smithi114 ceph-mon[40867]: osdmap e227: 8 total, 8 up, 7 in 2024-09-21T12:27:55.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:55 smithi114 ceph-mon[40867]: 4.19 scrub starts 2024-09-21T12:27:55.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:55 smithi114 ceph-mon[40867]: 4.19 scrub ok 2024-09-21T12:27:55.610 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:55.655 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:55.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:55.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:55.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:55 smithi183 ceph-mon[33277]: pgmap v368: 178 pgs: 178 active+clean; 961 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s 2024-09-21T12:27:55.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:55 smithi183 ceph-mon[33277]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:27:55.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:55 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:27:55.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:55 smithi183 ceph-mon[33277]: osdmap e227: 8 total, 8 up, 7 in 2024-09-21T12:27:55.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:55 smithi183 ceph-mon[33277]: 4.19 scrub starts 2024-09-21T12:27:55.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:55 smithi183 ceph-mon[33277]: 4.19 scrub ok 2024-09-21T12:27:55.757 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:55.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:55.786 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:55.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:55.887 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:55.916 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:55.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:55.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:56.016 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T12:27:56.017 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T12:27:56.017 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:27:56.044 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:56.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:56.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:27:56.146 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:56.178 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:56.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:56.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:56.280 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:56.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:56.310 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:56.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:56.411 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:56.452 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:56.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:56.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:56.554 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:56.630 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:27:56.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:56.635 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:56.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:56.681 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:27:56.737 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:56.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:56 smithi183 ceph-mon[33277]: 6.18 scrub starts 2024-09-21T12:27:56.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:56 smithi183 ceph-mon[33277]: 6.18 scrub ok 2024-09-21T12:27:56.762 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:56.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:56.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:56.772 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:56 smithi114 ceph-mon[30324]: 6.18 scrub starts 2024-09-21T12:27:56.772 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:56 smithi114 ceph-mon[30324]: 6.18 scrub ok 2024-09-21T12:27:56.772 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:56 smithi114 ceph-mon[40867]: 6.18 scrub starts 2024-09-21T12:27:56.772 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:56 smithi114 ceph-mon[40867]: 6.18 scrub ok 2024-09-21T12:27:56.864 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:56.903 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:56.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:56.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:57.004 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:57.033 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:57.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:57.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:57.135 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:57.169 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:57.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:57.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:57.271 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:57.276 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:27:57 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:27:57.307 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:57 smithi114 ceph-mon[40867]: pgmap v370: 178 pgs: 178 active+clean; 961 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 756 op/s 2024-09-21T12:27:57.307 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:57 smithi114 ceph-mon[40867]: 3.19 scrub starts 2024-09-21T12:27:57.308 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:57 smithi114 ceph-mon[40867]: 3.19 scrub ok 2024-09-21T12:27:57.308 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:57 smithi114 ceph-mon[40867]: 4.e scrub starts 2024-09-21T12:27:57.308 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:57 smithi114 ceph-mon[40867]: 4.e scrub ok 2024-09-21T12:27:57.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:57 smithi114 ceph-mon[30324]: pgmap v370: 178 pgs: 178 active+clean; 961 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 756 op/s 2024-09-21T12:27:57.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:57 smithi114 ceph-mon[30324]: 3.19 scrub starts 2024-09-21T12:27:57.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:57 smithi114 ceph-mon[30324]: 3.19 scrub ok 2024-09-21T12:27:57.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:57 smithi114 ceph-mon[30324]: 4.e scrub starts 2024-09-21T12:27:57.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:57 smithi114 ceph-mon[30324]: 4.e scrub ok 2024-09-21T12:27:57.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:57.309 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:57.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:57.410 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:57.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:57.439 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:57.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:57.540 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:57.577 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:57.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:57.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:57.621 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:27:57.621 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:27:57.679 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:57.710 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:57.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:57.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:57.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:57 smithi183 ceph-mon[33277]: pgmap v370: 178 pgs: 178 active+clean; 961 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 756 op/s 2024-09-21T12:27:57.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:57 smithi183 ceph-mon[33277]: 3.19 scrub starts 2024-09-21T12:27:57.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:57 smithi183 ceph-mon[33277]: 3.19 scrub ok 2024-09-21T12:27:57.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:57 smithi183 ceph-mon[33277]: 4.e scrub starts 2024-09-21T12:27:57.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:57 smithi183 ceph-mon[33277]: 4.e scrub ok 2024-09-21T12:27:57.750 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:27:57.750 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:27:57.750 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:27:57.750 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:27:57.750 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:27:57.750 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:27:57.750 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:27:57.751 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:27:57.751 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:27:57.751 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:27:57.751 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:27:57.751 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:27:57.751 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:27:57.751 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:27:57.754 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:27:57.755 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:27:57.812 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:57.843 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:57.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:57.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:57.945 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:57.990 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:57.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:57.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:58.092 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:58.129 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:58.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:58.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:58.230 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:58.261 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:58.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:58.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:58.363 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:27:58.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:58.395 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:58.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:27:58.496 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:27:58.504 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T12:27:58.531 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:58.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:58.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:27:58.572 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:58 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:27:58.572 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:58 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3624094230' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:27:58.572 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:58 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:27:58.572 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:58 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3624094230' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:27:58.633 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:58.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:58.660 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:58.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:58.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:58 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:27:58.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:58 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3624094230' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:27:58.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:58.806 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:58.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:58.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:58.807 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T12:27:58.908 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:58.934 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:58.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:58.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:59.035 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:59.059 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:59.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:59.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:59.160 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:27:59.188 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:59.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:59.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:27:59.290 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:59.320 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:59.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:59.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:59.422 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:59.455 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:59.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:59.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:27:59.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:59 smithi114 ceph-mon[40867]: pgmap v371: 178 pgs: 178 active+clean; 524 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.63k op/s 2024-09-21T12:27:59.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:59 smithi114 ceph-mon[40867]: 5.3 scrub starts 2024-09-21T12:27:59.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:27:59 smithi114 ceph-mon[40867]: 5.3 scrub ok 2024-09-21T12:27:59.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:59 smithi114 ceph-mon[30324]: pgmap v371: 178 pgs: 178 active+clean; 524 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.63k op/s 2024-09-21T12:27:59.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:59 smithi114 ceph-mon[30324]: 5.3 scrub starts 2024-09-21T12:27:59.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:27:59 smithi114 ceph-mon[30324]: 5.3 scrub ok 2024-09-21T12:27:59.557 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:27:59.581 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:59.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:59.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:27:59.683 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:27:59.710 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:27:59.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:59.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:27:59.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:59 smithi183 ceph-mon[33277]: pgmap v371: 178 pgs: 178 active+clean; 524 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.63k op/s 2024-09-21T12:27:59.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:59 smithi183 ceph-mon[33277]: 5.3 scrub starts 2024-09-21T12:27:59.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:27:59 smithi183 ceph-mon[33277]: 5.3 scrub ok 2024-09-21T12:27:59.812 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:27:59.850 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:59.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:59.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:27:59.952 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:27:59.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:27:59.988 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:27:59.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:00.089 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:00.113 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:00.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:00.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:00.135 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:00.141 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:00.215 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:00.242 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:00.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:00.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:00.344 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:00.390 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:00.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:00.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:00.492 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:00.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:00.519 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:00.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:00.620 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:00.655 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:00.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:00.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:00.757 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:00.799 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:00.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:00.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:00.901 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:00.925 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:00.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:00.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:01.018 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T12:28:01.018 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T12:28:01.027 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:01.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:01.127 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:01.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:01.184 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:28:01.184 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:28:01.229 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:01.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:01.252 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:01.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:01.353 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:01.379 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:01.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:01.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:01.480 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:01.507 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:01.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:01.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:01.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:01 smithi114 ceph-mon[40867]: pgmap v372: 178 pgs: 178 active+clean; 524 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.49k op/s 2024-09-21T12:28:01.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:01 smithi114 ceph-mon[40867]: 3.17 scrub starts 2024-09-21T12:28:01.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:01 smithi114 ceph-mon[40867]: 3.17 scrub ok 2024-09-21T12:28:01.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:01 smithi114 ceph-mon[30324]: pgmap v372: 178 pgs: 178 active+clean; 524 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.49k op/s 2024-09-21T12:28:01.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:01 smithi114 ceph-mon[30324]: 3.17 scrub starts 2024-09-21T12:28:01.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:01 smithi114 ceph-mon[30324]: 3.17 scrub ok 2024-09-21T12:28:01.609 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:01.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:01.638 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:01.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:01.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:01 smithi183 ceph-mon[33277]: pgmap v372: 178 pgs: 178 active+clean; 524 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.49k op/s 2024-09-21T12:28:01.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:01 smithi183 ceph-mon[33277]: 3.17 scrub starts 2024-09-21T12:28:01.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:01 smithi183 ceph-mon[33277]: 3.17 scrub ok 2024-09-21T12:28:01.739 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:01.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:01.778 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:01.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:01.878 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:01.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:01.903 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:01.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:01.926 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T12:28:01.990 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:28:02.004 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:02.021 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:02.028 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:02.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:02.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:02.129 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:02.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:02.186 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:02.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:02.287 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:02.314 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-21T12:28:02.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:02.314 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:02.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:02.415 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:02.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:02.455 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:02.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:02.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:02 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1962582982' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:28:02.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:02 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:28:02.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:02 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:28:02.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:02 smithi114 ceph-mon[30324]: 8.0 scrub starts 2024-09-21T12:28:02.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:02 smithi114 ceph-mon[30324]: 1.3 scrub starts 2024-09-21T12:28:02.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:02 smithi114 ceph-mon[30324]: 1.3 scrub ok 2024-09-21T12:28:02.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:02 smithi114 ceph-mon[30324]: 4.16 scrub starts 2024-09-21T12:28:02.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:02 smithi114 ceph-mon[30324]: 4.16 scrub ok 2024-09-21T12:28:02.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:02 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1962582982' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:28:02.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:02 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:28:02.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:02 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:28:02.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:02 smithi114 ceph-mon[40867]: 8.0 scrub starts 2024-09-21T12:28:02.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:02 smithi114 ceph-mon[40867]: 1.3 scrub starts 2024-09-21T12:28:02.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:02 smithi114 ceph-mon[40867]: 1.3 scrub ok 2024-09-21T12:28:02.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:02 smithi114 ceph-mon[40867]: 4.16 scrub starts 2024-09-21T12:28:02.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:02 smithi114 ceph-mon[40867]: 4.16 scrub ok 2024-09-21T12:28:02.556 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:02.607 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:02.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:02.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:02.708 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:02.736 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:02.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:02.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:02.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:02 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1962582982' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:28:02.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:02 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:28:02.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:02 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:28:02.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:02 smithi183 ceph-mon[33277]: 8.0 scrub starts 2024-09-21T12:28:02.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:02 smithi183 ceph-mon[33277]: 1.3 scrub starts 2024-09-21T12:28:02.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:02 smithi183 ceph-mon[33277]: 1.3 scrub ok 2024-09-21T12:28:02.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:02 smithi183 ceph-mon[33277]: 4.16 scrub starts 2024-09-21T12:28:02.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:02 smithi183 ceph-mon[33277]: 4.16 scrub ok 2024-09-21T12:28:02.838 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:02.863 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:02.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:02.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:02.964 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:03.002 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:03.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:03.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:03.025 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:28:02 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:28:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:28:03.103 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:03.132 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:03.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:03.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:03.234 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:03.254 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Removed 130413 objects 2024-09-21T12:28:03.254 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Clean up completed and total clean up time :99.1884 2024-09-21T12:28:03.258 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:03.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:03.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:03.353 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:03.360 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:03.361 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:03.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:03.387 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:03.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:03.477 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:03 smithi114 ceph-mon[40867]: pgmap v373: 178 pgs: 178 active+clean; 443 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 941 op/s 2024-09-21T12:28:03.477 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:03 smithi114 ceph-mon[40867]: 8.0 scrub ok 2024-09-21T12:28:03.477 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:03 smithi114 ceph-mon[40867]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:28:03.477 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:03 smithi114 ceph-mon[40867]: Cluster is now healthy 2024-09-21T12:28:03.477 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:03 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:28:03.477 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:03 smithi114 ceph-mon[40867]: osdmap e228: 8 total, 8 up, 7 in 2024-09-21T12:28:03.477 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:03 smithi114 ceph-mon[30324]: pgmap v373: 178 pgs: 178 active+clean; 443 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 941 op/s 2024-09-21T12:28:03.477 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:03 smithi114 ceph-mon[30324]: 8.0 scrub ok 2024-09-21T12:28:03.477 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:03 smithi114 ceph-mon[30324]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:28:03.477 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:03 smithi114 ceph-mon[30324]: Cluster is now healthy 2024-09-21T12:28:03.477 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:03 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:28:03.478 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:03 smithi114 ceph-mon[30324]: osdmap e228: 8 total, 8 up, 7 in 2024-09-21T12:28:03.488 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:03.508 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:03.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:03.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:03.610 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:03.631 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:03.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:03.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:03.732 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:03.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:03 smithi183 ceph-mon[33277]: pgmap v373: 178 pgs: 178 active+clean; 443 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 941 op/s 2024-09-21T12:28:03.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:03 smithi183 ceph-mon[33277]: 8.0 scrub ok 2024-09-21T12:28:03.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:03 smithi183 ceph-mon[33277]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:28:03.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:03 smithi183 ceph-mon[33277]: Cluster is now healthy 2024-09-21T12:28:03.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:03 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:28:03.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:03 smithi183 ceph-mon[33277]: osdmap e228: 8 total, 8 up, 7 in 2024-09-21T12:28:03.753 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:03.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:03.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:03.855 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:03.876 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:03.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:03.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:03.978 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:03.999 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:04.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:04.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:04.101 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:04.122 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:04.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:04.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:04.208 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:28:04.208 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:28:04.225 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:04.246 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:04.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:04.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:04.327 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T12:28:04.348 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:04.371 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:04.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:04.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:04.473 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:04.494 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:04.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:04.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:04.595 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:04.616 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:04.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:04.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:04.717 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:04.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:04 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1569003369' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:28:04.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:04 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:28:04.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:04 smithi183 ceph-mon[33277]: 5.c scrub starts 2024-09-21T12:28:04.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:04 smithi183 ceph-mon[33277]: 5.c scrub ok 2024-09-21T12:28:04.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:04 smithi183 ceph-mon[33277]: pgmap v375: 178 pgs: 178 active+clean; 100 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.46k op/s 2024-09-21T12:28:04.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:04.739 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:04.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:04.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:04 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1569003369' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:28:04.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:04 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:28:04.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:04 smithi114 ceph-mon[30324]: 5.c scrub starts 2024-09-21T12:28:04.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:04 smithi114 ceph-mon[30324]: 5.c scrub ok 2024-09-21T12:28:04.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:04 smithi114 ceph-mon[30324]: pgmap v375: 178 pgs: 178 active+clean; 100 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.46k op/s 2024-09-21T12:28:04.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:04 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1569003369' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:28:04.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:04 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:28:04.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:04 smithi114 ceph-mon[40867]: 5.c scrub starts 2024-09-21T12:28:04.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:04 smithi114 ceph-mon[40867]: 5.c scrub ok 2024-09-21T12:28:04.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:04 smithi114 ceph-mon[40867]: pgmap v375: 178 pgs: 178 active+clean; 100 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.46k op/s 2024-09-21T12:28:04.841 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:04.866 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:04.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:04.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:04.935 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:28:04.936 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:28:04.936 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:28:04.936 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:28:04.936 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:28:04.936 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:28:04.936 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:28:04.936 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:28:04.936 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:28:04.936 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:28:04.936 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:28:04.936 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:28:04.936 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:28:04.937 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:28:04.937 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:28:04.937 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:28:04.968 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:04.989 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:04.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:04.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:05.091 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:05.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:05.113 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:05.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:05.173 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T12:28:05.214 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:05.240 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:05.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:05.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:05.341 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:05.367 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:05.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:05.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:05.469 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:05.474 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:05 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T12:28:05.474 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:05 smithi114 ceph-mon[40867]: osdmap e229: 8 total, 8 up, 7 in 2024-09-21T12:28:05.474 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:05 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1024592810' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:28:05.474 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:05 smithi114 ceph-mon[40867]: 5.1a scrub starts 2024-09-21T12:28:05.474 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:05 smithi114 ceph-mon[40867]: 5.1a scrub ok 2024-09-21T12:28:05.475 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:05 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T12:28:05.475 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:05 smithi114 ceph-mon[30324]: osdmap e229: 8 total, 8 up, 7 in 2024-09-21T12:28:05.475 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:05 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1024592810' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:28:05.475 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:05 smithi114 ceph-mon[30324]: 5.1a scrub starts 2024-09-21T12:28:05.475 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:05 smithi114 ceph-mon[30324]: 5.1a scrub ok 2024-09-21T12:28:05.491 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:05.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:05.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:05.593 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:05.614 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:05.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:05.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:05.640 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:05 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T12:28:05.640 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:05 smithi183 ceph-mon[33277]: osdmap e229: 8 total, 8 up, 7 in 2024-09-21T12:28:05.640 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:05 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1024592810' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:28:05.640 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:05 smithi183 ceph-mon[33277]: 5.1a scrub starts 2024-09-21T12:28:05.640 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:05 smithi183 ceph-mon[33277]: 5.1a scrub ok 2024-09-21T12:28:05.715 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:05.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:05.744 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:05.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:05.845 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:05.867 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:05.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:05.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:05.939 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T12:28:05.968 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:05.990 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:05.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:05.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:06.092 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:06.113 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:06.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:06.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:06.214 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:06.235 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:06.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:06.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:06.337 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:06.358 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:06.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:06.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:06.460 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:06.480 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:06.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:06.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:06.582 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:06.604 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:06.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:06.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:06.705 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:06.726 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:06.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:06.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:06.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:06 smithi183 ceph-mon[33277]: 5.f scrub starts 2024-09-21T12:28:06.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:06 smithi183 ceph-mon[33277]: 5.f scrub ok 2024-09-21T12:28:06.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:06 smithi183 ceph-mon[33277]: 1.12 scrub starts 2024-09-21T12:28:06.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:06 smithi183 ceph-mon[33277]: 1.12 scrub ok 2024-09-21T12:28:06.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:06 smithi183 ceph-mon[33277]: 6.17 scrub starts 2024-09-21T12:28:06.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:06 smithi183 ceph-mon[33277]: 6.17 scrub ok 2024-09-21T12:28:06.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:06 smithi183 ceph-mon[33277]: 3.8 scrub starts 2024-09-21T12:28:06.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:06 smithi183 ceph-mon[33277]: 3.8 scrub ok 2024-09-21T12:28:06.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:06 smithi183 ceph-mon[33277]: pgmap v377: 178 pgs: 178 active+clean; 100 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 847 op/s 2024-09-21T12:28:06.772 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:06 smithi114 ceph-mon[40867]: 5.f scrub starts 2024-09-21T12:28:06.772 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:06 smithi114 ceph-mon[40867]: 5.f scrub ok 2024-09-21T12:28:06.772 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:06 smithi114 ceph-mon[40867]: 1.12 scrub starts 2024-09-21T12:28:06.772 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:06 smithi114 ceph-mon[40867]: 1.12 scrub ok 2024-09-21T12:28:06.772 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:06 smithi114 ceph-mon[40867]: 6.17 scrub starts 2024-09-21T12:28:06.772 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:06 smithi114 ceph-mon[40867]: 6.17 scrub ok 2024-09-21T12:28:06.772 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:06 smithi114 ceph-mon[40867]: 3.8 scrub starts 2024-09-21T12:28:06.773 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:06 smithi114 ceph-mon[40867]: 3.8 scrub ok 2024-09-21T12:28:06.773 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:06 smithi114 ceph-mon[40867]: pgmap v377: 178 pgs: 178 active+clean; 100 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 847 op/s 2024-09-21T12:28:06.773 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:06 smithi114 ceph-mon[30324]: 5.f scrub starts 2024-09-21T12:28:06.773 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:06 smithi114 ceph-mon[30324]: 5.f scrub ok 2024-09-21T12:28:06.773 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:06 smithi114 ceph-mon[30324]: 1.12 scrub starts 2024-09-21T12:28:06.773 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:06 smithi114 ceph-mon[30324]: 1.12 scrub ok 2024-09-21T12:28:06.773 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:06 smithi114 ceph-mon[30324]: 6.17 scrub starts 2024-09-21T12:28:06.773 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:06 smithi114 ceph-mon[30324]: 6.17 scrub ok 2024-09-21T12:28:06.773 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:06 smithi114 ceph-mon[30324]: 3.8 scrub starts 2024-09-21T12:28:06.773 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:06 smithi114 ceph-mon[30324]: 3.8 scrub ok 2024-09-21T12:28:06.773 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:06 smithi114 ceph-mon[30324]: pgmap v377: 178 pgs: 178 active+clean; 100 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 847 op/s 2024-09-21T12:28:06.797 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T12:28:06.827 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:06.848 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:06.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:06.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:06.949 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:06.971 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:06.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:06.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:07.073 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:07.094 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:07.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:07.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:07.196 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:07.218 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:07.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:07.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:07.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:28:07 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:28:07.295 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:07.320 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:07.341 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:07.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:07.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:07.442 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:07.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:07.464 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:07.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:07.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:07 smithi114 ceph-mon[40867]: 3.a scrub starts 2024-09-21T12:28:07.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:07 smithi114 ceph-mon[40867]: 3.a scrub ok 2024-09-21T12:28:07.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:07 smithi114 ceph-mon[40867]: 4.1a scrub starts 2024-09-21T12:28:07.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:07 smithi114 ceph-mon[40867]: 4.1a scrub ok 2024-09-21T12:28:07.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:07 smithi114 ceph-mon[40867]: 5.1d scrub starts 2024-09-21T12:28:07.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:07 smithi114 ceph-mon[40867]: 5.1d scrub ok 2024-09-21T12:28:07.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:07 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:28:07.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:07 smithi114 ceph-mon[30324]: 3.a scrub starts 2024-09-21T12:28:07.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:07 smithi114 ceph-mon[30324]: 3.a scrub ok 2024-09-21T12:28:07.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:07 smithi114 ceph-mon[30324]: 4.1a scrub starts 2024-09-21T12:28:07.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:07 smithi114 ceph-mon[30324]: 4.1a scrub ok 2024-09-21T12:28:07.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:07 smithi114 ceph-mon[30324]: 5.1d scrub starts 2024-09-21T12:28:07.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:07 smithi114 ceph-mon[30324]: 5.1d scrub ok 2024-09-21T12:28:07.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:07 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:28:07.565 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:07.588 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:07.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:07.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:07.690 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:07.712 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:07.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:07.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:07.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:07 smithi183 ceph-mon[33277]: 3.a scrub starts 2024-09-21T12:28:07.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:07 smithi183 ceph-mon[33277]: 3.a scrub ok 2024-09-21T12:28:07.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:07 smithi183 ceph-mon[33277]: 4.1a scrub starts 2024-09-21T12:28:07.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:07 smithi183 ceph-mon[33277]: 4.1a scrub ok 2024-09-21T12:28:07.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:07 smithi183 ceph-mon[33277]: 5.1d scrub starts 2024-09-21T12:28:07.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:07 smithi183 ceph-mon[33277]: 5.1d scrub ok 2024-09-21T12:28:07.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:07 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:28:07.782 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:07.800 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:07.814 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:07.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:07.836 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:07.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:07.937 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:07.957 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:07.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:07.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:08.059 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:08.080 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:08.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:08.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:08.182 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:08.204 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:08.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:08.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:08.305 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:08.315 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:28:08.315 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:28:08.315 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:28:08.315 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:28:08.315 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:28:08.327 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:08.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:08.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:08.335 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-21T12:28:08.429 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:08.450 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:08.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:08.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:08.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:08 smithi114 ceph-mon[40867]: pgmap v378: 178 pgs: 178 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-09-21T12:28:08.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:08 smithi114 ceph-mon[40867]: 3.12 scrub starts 2024-09-21T12:28:08.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:08 smithi114 ceph-mon[40867]: 3.12 scrub ok 2024-09-21T12:28:08.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:08 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/4184289054' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:28:08.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:08 smithi114 ceph-mon[30324]: pgmap v378: 178 pgs: 178 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-09-21T12:28:08.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:08 smithi114 ceph-mon[30324]: 3.12 scrub starts 2024-09-21T12:28:08.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:08 smithi114 ceph-mon[30324]: 3.12 scrub ok 2024-09-21T12:28:08.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:08 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4184289054' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:28:08.551 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:08.576 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:08.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:08.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:08.677 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:08.698 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:08.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:08.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:08.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:08 smithi183 ceph-mon[33277]: pgmap v378: 178 pgs: 178 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-09-21T12:28:08.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:08 smithi183 ceph-mon[33277]: 3.12 scrub starts 2024-09-21T12:28:08.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:08 smithi183 ceph-mon[33277]: 3.12 scrub ok 2024-09-21T12:28:08.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:08 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/4184289054' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:28:08.800 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:08.821 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:08.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:08.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:08.922 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:08.948 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:08.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:08.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:09.050 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:09.071 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:09.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:09.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:09.173 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:09.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:09.200 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:09.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:09.221 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-09-21T12:28:09.221 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-09-21T12:28:09.270 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:28:09.301 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:09.335 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:09.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:09.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:09.418 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:09 smithi114 ceph-mon[40867]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:28:09.418 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:09 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/4184289054' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:28:09.418 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:09 smithi114 ceph-mon[40867]: osdmap e230: 8 total, 8 up, 7 in 2024-09-21T12:28:09.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:09 smithi114 ceph-mon[30324]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:28:09.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:09 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4184289054' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:28:09.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:09 smithi114 ceph-mon[30324]: osdmap e230: 8 total, 8 up, 7 in 2024-09-21T12:28:09.437 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:09.463 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:09.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:09.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:09.565 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:09.587 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:09.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:09.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:09.689 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:09.715 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:09.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:09.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:09.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:09 smithi183 ceph-mon[33277]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:28:09.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:09 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/4184289054' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:28:09.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:09 smithi183 ceph-mon[33277]: osdmap e230: 8 total, 8 up, 7 in 2024-09-21T12:28:09.817 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:09.842 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:09.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:09.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:09.944 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:09.966 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:09.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:09.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:10.067 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:10.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:10.090 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:10.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:10.191 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:10.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:10.218 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:10.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:10.319 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:10.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:10.347 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:10.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:10.348 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T12:28:10.448 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:10.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:10.471 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:10.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:10.572 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:10.595 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:10.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:10.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:10.605 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:10 smithi114 ceph-mon[40867]: pgmap v380: 178 pgs: 178 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 690 B/s rd, 0 B/s wr, 670 op/s 2024-09-21T12:28:10.605 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:10 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2881363237' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:28:10.605 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:10 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:28:10.605 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:10 smithi114 ceph-mon[30324]: pgmap v380: 178 pgs: 178 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 690 B/s rd, 0 B/s wr, 670 op/s 2024-09-21T12:28:10.605 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:10 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2881363237' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:28:10.605 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:10 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:28:10.635 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T12:28:10.697 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:10.723 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:10.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:10.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:10.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:10 smithi183 ceph-mon[33277]: pgmap v380: 178 pgs: 178 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 690 B/s rd, 0 B/s wr, 670 op/s 2024-09-21T12:28:10.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:10 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2881363237' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:28:10.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:10 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:28:10.825 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:10.845 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:10.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:10.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:10.947 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:10.972 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:10.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:10.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:11.073 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:11.097 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:11.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:11.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:11.124 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T12:28:11.199 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:11.223 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:11.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:11.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:11.325 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:11.351 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:11.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:11.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:11.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:11 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T12:28:11.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:11 smithi114 ceph-mon[30324]: osdmap e231: 8 total, 8 up, 7 in 2024-09-21T12:28:11.427 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:11 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T12:28:11.427 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:11 smithi114 ceph-mon[40867]: osdmap e231: 8 total, 8 up, 7 in 2024-09-21T12:28:11.453 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:11.475 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:11.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:11.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:11.577 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:11.601 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:11.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:11.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:11.703 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:11.727 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:11.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:11.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:11.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:11 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T12:28:11.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:11 smithi183 ceph-mon[33277]: osdmap e231: 8 total, 8 up, 7 in 2024-09-21T12:28:11.829 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:11.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:11.853 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:11.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:11.954 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:11.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:11.981 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:11.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:12.082 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:12.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:12.108 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:12.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:12.209 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:12.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:12.234 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:12.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:12.335 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:12.360 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:12.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:12.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:12.462 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:12.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:12.487 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:12.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:12.576 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T12:28:12.587 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:12.600 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:12.603 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:28:12 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:28:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:28:12.603 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:12 smithi114 ceph-mon[30324]: pgmap v382: 178 pgs: 178 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 216 op/s 2024-09-21T12:28:12.603 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:12 smithi114 ceph-mon[40867]: pgmap v382: 178 pgs: 178 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 216 op/s 2024-09-21T12:28:12.603 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:12.611 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:12.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:12.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:12.649 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:12.713 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:12.734 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:12.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:12.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:12.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:12 smithi183 ceph-mon[33277]: pgmap v382: 178 pgs: 178 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 216 op/s 2024-09-21T12:28:12.836 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:12.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:12.863 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:12.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:12.963 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:12.988 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:12.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:12.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:13.089 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:13.111 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:13.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:13.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:13.213 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:13.239 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:13.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:13.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:13.341 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:13.367 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:13.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:13.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:13.376 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:28:13.377 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:28:13.393 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:28:13.393 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:28:13.410 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:28:13.410 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:28:13.428 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:28:13.428 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:28:13.429 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:13 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1831161407' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:28:13.429 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:13 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1831161407' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:28:13.446 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:28:13.446 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:28:13.462 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:28:13.462 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:28:13.469 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:13.479 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:28:13.479 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:28:13.493 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:13.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:13.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:13.495 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:28:13.496 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:28:13.595 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:13.619 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:13.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:13.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:13.720 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:13.724 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:13.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:13 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1831161407' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:28:13.744 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:13.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:13.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:13.845 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:13.868 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:13.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:13.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:13.970 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:13.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:13.991 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:13.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:14.092 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:14.113 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:14.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:14.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:14.215 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:14.239 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:14.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:14.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:14.341 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:14.364 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:14.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:14.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:14.418 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'unique_pool_1' removed 2024-09-21T12:28:14.444 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:14 smithi114 ceph-mon[30324]: pgmap v383: 178 pgs: 178 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 200 op/s 2024-09-21T12:28:14.444 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:14 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2157854055' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T12:28:14.447 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:14 smithi114 ceph-mon[40867]: pgmap v383: 178 pgs: 178 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 200 op/s 2024-09-21T12:28:14.447 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:14 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2157854055' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T12:28:14.465 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:14.488 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:14.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:14.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:14.590 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:14.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:14.612 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:14.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:14.713 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:14.735 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:14.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:14.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:14.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:14 smithi183 ceph-mon[33277]: pgmap v383: 178 pgs: 178 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 200 op/s 2024-09-21T12:28:14.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:14 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2157854055' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T12:28:14.752 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:28:14.752 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:28:14.837 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:14.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:14.861 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:14.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:14.962 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:14.984 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:14.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:14.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:15.086 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:15.107 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:15.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:15.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:15.209 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:15.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:15.230 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:15.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:15.331 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:15.353 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:15.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:15.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:15.423 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-21T12:28:15.455 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:15.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:15.478 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:15.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:15.579 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:15.600 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:15.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:15.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:15.702 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:15.723 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:15.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:15.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:15.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:15 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:28:15.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:15 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:28:15.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:15 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2157854055' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T12:28:15.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:15 smithi183 ceph-mon[33277]: osdmap e232: 8 total, 8 up, 7 in 2024-09-21T12:28:15.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:15 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2629506363' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:28:15.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:15 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:28:15.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:15 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:28:15.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:15 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2157854055' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T12:28:15.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:15 smithi114 ceph-mon[40867]: osdmap e232: 8 total, 8 up, 7 in 2024-09-21T12:28:15.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:15 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2629506363' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:28:15.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:15 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:28:15.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:15 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:28:15.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:15 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2157854055' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T12:28:15.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:15 smithi114 ceph-mon[30324]: osdmap e232: 8 total, 8 up, 7 in 2024-09-21T12:28:15.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:15 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2629506363' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:28:15.825 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:15.846 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:15.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:15.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:15.948 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:15.970 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:15.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:15.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:16.072 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:16.093 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:16.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:16.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:16.195 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:16.216 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:16.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:16.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:16.280 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T12:28:16.319 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:16.344 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:16.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:16.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:16.446 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:16.468 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:16.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:16.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:16.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:16 smithi114 ceph-mon[40867]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:28:16.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:16 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2629506363' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:28:16.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:16 smithi114 ceph-mon[40867]: osdmap e233: 8 total, 8 up, 7 in 2024-09-21T12:28:16.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:16 smithi114 ceph-mon[40867]: pgmap v386: 162 pgs: 162 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T12:28:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:16 smithi114 ceph-mon[30324]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:28:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:16 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2629506363' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:28:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:16 smithi114 ceph-mon[30324]: osdmap e233: 8 total, 8 up, 7 in 2024-09-21T12:28:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:16 smithi114 ceph-mon[30324]: pgmap v386: 162 pgs: 162 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T12:28:16.571 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:16.592 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:16.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:16.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:16.694 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:16.715 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:16.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:16.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:16.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:16 smithi183 ceph-mon[33277]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:28:16.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:16 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2629506363' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:28:16.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:16 smithi183 ceph-mon[33277]: osdmap e233: 8 total, 8 up, 7 in 2024-09-21T12:28:16.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:16 smithi183 ceph-mon[33277]: pgmap v386: 162 pgs: 162 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T12:28:16.817 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:16.838 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:16.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:16.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:16.940 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:16.961 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:16.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:16.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:17.063 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:17.084 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:17.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:17.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:17.186 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:17.207 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:17.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:17.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:17.308 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:17.330 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:17.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:17.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:17.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:28:17 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:28:17.432 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:17.453 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:17.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:17.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:17.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:17 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:28:17.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:17 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:28:17.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:17 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:28:17.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:17 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:28:17.554 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:17.559 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T12:28:17.559 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T12:28:17.560 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-09-21T12:28:17.560 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_2 16 2024-09-21T12:28:17.578 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:17.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:17.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:17.598 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:17.598 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:17.602 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:28:17.684 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:17.706 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:17.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:17.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:17.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:17 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:28:17.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:17 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:28:17.808 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:17.829 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:17.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:17.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:17.931 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:17.953 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:17.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:17.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:18.054 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:18.076 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:18.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:18.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:18.170 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:18.170 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:18.177 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:18.203 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:18.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:18.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:18.305 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:18.397 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:18.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:18.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:18.499 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:18.520 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:18.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:18.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:18.622 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:18.632 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:18 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:28:18.632 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:18 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:28:18.632 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:18 smithi114 ceph-mon[40867]: pgmap v387: 162 pgs: 162 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:28:18.632 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:18 smithi114 ceph-mon[40867]: osdmap e234: 8 total, 8 up, 7 in 2024-09-21T12:28:18.632 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:18 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2461596395' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:28:18.632 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:18 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:28:18.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:18 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:28:18.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:18 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:28:18.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:18 smithi114 ceph-mon[30324]: pgmap v387: 162 pgs: 162 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:28:18.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:18 smithi114 ceph-mon[30324]: osdmap e234: 8 total, 8 up, 7 in 2024-09-21T12:28:18.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:18 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2461596395' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:28:18.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:18 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:28:18.643 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:18.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:18.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:18.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:18 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:28:18.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:18 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:28:18.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:18 smithi183 ceph-mon[33277]: pgmap v387: 162 pgs: 162 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:28:18.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:18 smithi183 ceph-mon[33277]: osdmap e234: 8 total, 8 up, 7 in 2024-09-21T12:28:18.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:18 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2461596395' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:28:18.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:18 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:28:18.745 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:18.767 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:18.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:18.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:18.868 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:18.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:18.890 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:18.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:18.892 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T12:28:18.937 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T12:28:18.991 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:19.013 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:19.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:19.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:19.114 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:19.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:19.136 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:19.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:19.237 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:19.259 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:19.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:19.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:19.318 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:28:19.318 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:28:19.329 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:28:19.329 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:28:19.352 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:28:19.352 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:28:19.360 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:19.369 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:28:19.369 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:28:19.381 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:19.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:19.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:19.386 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:28:19.386 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:28:19.403 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:28:19.403 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:28:19.419 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:28:19.420 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:28:19.436 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:28:19.436 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:28:19.483 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:19.504 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:19.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:19.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:19.606 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:19.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:19.628 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:19.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:19.729 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:19.749 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:19.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:19.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:19.851 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:19.871 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:19.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:19.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:19.936 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'unique_pool_2' created 2024-09-21T12:28:19.973 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:19.994 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:19.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:19.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:20.095 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:20.116 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:20.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:20.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:20.217 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:20.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:19 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T12:28:20.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:19 smithi183 ceph-mon[33277]: osdmap e235: 8 total, 8 up, 7 in 2024-09-21T12:28:20.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:19 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1103837642' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:28:20.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:19 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/715354378' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-21T12:28:20.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:19 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:28:20.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:20.240 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:20.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:20.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:19 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T12:28:20.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:19 smithi114 ceph-mon[30324]: osdmap e235: 8 total, 8 up, 7 in 2024-09-21T12:28:20.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:19 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1103837642' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:28:20.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:19 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/715354378' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-21T12:28:20.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:19 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:28:20.243 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:19 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T12:28:20.243 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:19 smithi114 ceph-mon[40867]: osdmap e235: 8 total, 8 up, 7 in 2024-09-21T12:28:20.243 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:19 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1103837642' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:28:20.243 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:19 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/715354378' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-21T12:28:20.243 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:19 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:28:20.341 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:20.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:20.363 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:20.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:20.464 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:20.485 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:20.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:20.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:20.587 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:20.608 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:20.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:20.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:20.709 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:20.731 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:20.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:20.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:20.833 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:20.855 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:20.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:20.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:20.956 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:20.978 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:20.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:20.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:21.080 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:21.102 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:21.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:21.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:21.158 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-09-21T12:28:21.203 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:21.225 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:21.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:21.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:21.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:20 smithi183 ceph-mon[33277]: pgmap v390: 162 pgs: 162 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 965 B/s rd, 0 op/s 2024-09-21T12:28:21.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:20 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/715354378' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-21T12:28:21.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:20 smithi183 ceph-mon[33277]: osdmap e236: 8 total, 8 up, 7 in 2024-09-21T12:28:21.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:20 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:28:21.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:20 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:28:21.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:20 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:28:21.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:20 smithi114 ceph-mon[30324]: pgmap v390: 162 pgs: 162 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 965 B/s rd, 0 op/s 2024-09-21T12:28:21.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:20 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/715354378' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-21T12:28:21.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:20 smithi114 ceph-mon[30324]: osdmap e236: 8 total, 8 up, 7 in 2024-09-21T12:28:21.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:20 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:28:21.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:20 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:28:21.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:20 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:28:21.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:20 smithi114 ceph-mon[40867]: pgmap v390: 162 pgs: 162 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 965 B/s rd, 0 op/s 2024-09-21T12:28:21.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:20 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/715354378' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-21T12:28:21.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:20 smithi114 ceph-mon[40867]: osdmap e236: 8 total, 8 up, 7 in 2024-09-21T12:28:21.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:20 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:28:21.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:20 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:28:21.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:20 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:28:21.327 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:21.348 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:21.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:21.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:21.450 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:21.461 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:28:21.461 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:28:21.462 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 0 2024-09-21T12:28:21.462 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:28:21.472 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:21.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:21.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:21.573 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:21.595 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:21.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:21.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:21.697 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:21.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:21.721 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:21.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:21.822 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:21.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:21.845 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:21.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:21.946 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:21.969 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:21.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:21.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:22.071 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:22.093 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:22.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:22.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:22.108 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:21 smithi114 ceph-mon[40867]: osdmap e237: 8 total, 8 up, 7 in 2024-09-21T12:28:22.108 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:21 smithi114 ceph-mon[40867]: osdmap e238: 8 total, 8 up, 7 in 2024-09-21T12:28:22.108 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:21 smithi114 ceph-mon[30324]: osdmap e237: 8 total, 8 up, 7 in 2024-09-21T12:28:22.108 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:21 smithi114 ceph-mon[30324]: osdmap e238: 8 total, 8 up, 7 in 2024-09-21T12:28:22.137 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T12:28:22.195 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:22.217 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:22.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:22.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:22.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:21 smithi183 ceph-mon[33277]: osdmap e237: 8 total, 8 up, 7 in 2024-09-21T12:28:22.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:21 smithi183 ceph-mon[33277]: osdmap e238: 8 total, 8 up, 7 in 2024-09-21T12:28:22.319 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:22.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:22.341 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:22.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:22.442 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:22.464 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:22.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:22.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:22.565 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:22.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:22.587 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:22.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:22.689 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:22.704 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T12:28:22.712 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:22.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:22.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:22.813 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:22.834 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:22.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:22.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:22.936 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:22.960 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:22.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:22.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:22.962 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:28:22 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:28:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:28:23.063 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:23.085 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:23.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:23.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:23.145 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:23.148 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:23.188 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:23.189 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:23.210 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:23.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:23.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:23.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:22 smithi183 ceph-mon[33277]: pgmap v393: 178 pgs: 10 unknown, 168 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:28:23.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:22 smithi183 ceph-mon[33277]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T12:28:23.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:22 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:28:23.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:22 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:28:23.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:22 smithi114 ceph-mon[40867]: pgmap v393: 178 pgs: 10 unknown, 168 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:28:23.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:22 smithi114 ceph-mon[40867]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T12:28:23.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:22 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:28:23.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:22 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:28:23.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:22 smithi114 ceph-mon[30324]: pgmap v393: 178 pgs: 10 unknown, 168 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:28:23.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:22 smithi114 ceph-mon[30324]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T12:28:23.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:22 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:28:23.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:22 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:28:23.313 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:23.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:23.336 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:23.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:23.437 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:23.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:23.461 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:23.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:23.562 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:23.583 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:23.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:23.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:23.685 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:23.709 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:23.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:23.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:23.811 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:23.834 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:23.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:23.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:23.935 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:23.958 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:23.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:23.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:24.060 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:24.081 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:24.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:24.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:24.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:23 smithi114 ceph-mon[30324]: osdmap e239: 8 total, 8 up, 7 in 2024-09-21T12:28:24.168 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:23 smithi114 ceph-mon[40867]: osdmap e239: 8 total, 8 up, 7 in 2024-09-21T12:28:24.174 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:24.183 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:24.206 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:24.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:24.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:24.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:23 smithi183 ceph-mon[33277]: osdmap e239: 8 total, 8 up, 7 in 2024-09-21T12:28:24.308 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:24.331 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:24.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:24.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:24.432 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:24.455 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:24.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:24.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:24.557 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:24.579 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:24.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:24.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:24.681 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:24.702 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:24.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:24.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:24.803 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:24.829 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:24.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:24.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:24.931 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:24.954 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:24.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:24.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:24.981 INFO:teuthology.orchestra.run.smithi114.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-09-21T12:28:24.982 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T12:28:24.987 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-21T12:28:25.056 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:25.082 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:25.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:25.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:25.091 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:24 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:28:25.091 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:24 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:28:25.092 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:24 smithi114 ceph-mon[30324]: pgmap v396: 178 pgs: 178 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:28:25.092 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:24 smithi114 ceph-mon[30324]: osdmap e240: 8 total, 8 up, 7 in 2024-09-21T12:28:25.092 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:24 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2940626645' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:28:25.092 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:24 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1914015314' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:28:25.092 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:24 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:28:25.092 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:24 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1822405759' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:28:25.092 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:24 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:28:25.092 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:24 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:28:25.092 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:24 smithi114 ceph-mon[40867]: pgmap v396: 178 pgs: 178 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:28:25.092 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:24 smithi114 ceph-mon[40867]: osdmap e240: 8 total, 8 up, 7 in 2024-09-21T12:28:25.092 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:24 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2940626645' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:28:25.093 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:24 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1914015314' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:28:25.093 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:24 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:28:25.093 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:24 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1822405759' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:28:25.140 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:28:25.183 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:25.208 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:25.209 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:28:25.209 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:28:25.209 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:28:25.209 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:28:25.209 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:28:25.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:25.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:25.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:24 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:28:25.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:24 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:28:25.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:24 smithi183 ceph-mon[33277]: pgmap v396: 178 pgs: 178 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:28:25.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:24 smithi183 ceph-mon[33277]: osdmap e240: 8 total, 8 up, 7 in 2024-09-21T12:28:25.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:24 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2940626645' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:28:25.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:24 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1914015314' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:28:25.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:24 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:28:25.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:24 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1822405759' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:28:25.311 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:25.335 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:25.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:25.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:25.437 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:25.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:25.462 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:25.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:25.563 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:25.590 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:25.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:25.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:25.692 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:25.713 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:25.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:25.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:25.815 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:25.839 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:25.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:25.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:25.941 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:25.966 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:25.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:25.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:26.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:25 smithi114 ceph-mon[30324]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:28:26.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:25 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2940626645' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T12:28:26.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:25 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:28:26.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:25 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1822405759' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T12:28:26.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:25 smithi114 ceph-mon[30324]: osdmap e241: 8 total, 8 up, 7 in 2024-09-21T12:28:26.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:25 smithi114 ceph-mon[30324]: 4.7 scrub starts 2024-09-21T12:28:26.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:25 smithi114 ceph-mon[30324]: 4.7 scrub ok 2024-09-21T12:28:26.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:25 smithi114 ceph-mon[30324]: 1.1 scrub starts 2024-09-21T12:28:26.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:25 smithi114 ceph-mon[30324]: 1.1 scrub ok 2024-09-21T12:28:26.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:25 smithi114 ceph-mon[40867]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:28:26.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:25 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2940626645' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T12:28:26.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:25 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:28:26.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:25 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1822405759' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T12:28:26.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:25 smithi114 ceph-mon[40867]: osdmap e241: 8 total, 8 up, 7 in 2024-09-21T12:28:26.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:25 smithi114 ceph-mon[40867]: 4.7 scrub starts 2024-09-21T12:28:26.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:25 smithi114 ceph-mon[40867]: 4.7 scrub ok 2024-09-21T12:28:26.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:25 smithi114 ceph-mon[40867]: 1.1 scrub starts 2024-09-21T12:28:26.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:25 smithi114 ceph-mon[40867]: 1.1 scrub ok 2024-09-21T12:28:26.067 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:26.089 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:26.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:26.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:26.167 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T12:28:26.190 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:26.233 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:26.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:26.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:26.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:25 smithi183 ceph-mon[33277]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:28:26.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:25 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2940626645' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T12:28:26.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:25 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:28:26.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:25 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1822405759' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T12:28:26.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:25 smithi183 ceph-mon[33277]: osdmap e241: 8 total, 8 up, 7 in 2024-09-21T12:28:26.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:25 smithi183 ceph-mon[33277]: 4.7 scrub starts 2024-09-21T12:28:26.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:25 smithi183 ceph-mon[33277]: 4.7 scrub ok 2024-09-21T12:28:26.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:25 smithi183 ceph-mon[33277]: 1.1 scrub starts 2024-09-21T12:28:26.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:25 smithi183 ceph-mon[33277]: 1.1 scrub ok 2024-09-21T12:28:26.334 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:26.356 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:26.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:26.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:26.458 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:26.479 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:26.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:26.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:26.580 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:26.602 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:26.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:26.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:26.704 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:26.727 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:26.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:26.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:26.829 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:26.851 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:26.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:26.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:26.952 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:26.976 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:26.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:26.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:27.078 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:27.099 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:27.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:27.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:27.201 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:27.225 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:27.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:27.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:27.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:28:27 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:28:27.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:26 smithi183 ceph-mon[33277]: pgmap v399: 178 pgs: 178 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T12:28:27.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:26 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:28:27.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:26 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:28:27.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:26 smithi183 ceph-mon[33277]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T12:28:27.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:26 smithi183 ceph-mon[33277]: osdmap e242: 8 total, 8 up, 7 in 2024-09-21T12:28:27.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:26 smithi183 ceph-mon[33277]: 4.0 scrub starts 2024-09-21T12:28:27.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:26 smithi183 ceph-mon[33277]: 4.0 scrub ok 2024-09-21T12:28:27.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:26 smithi183 ceph-mon[33277]: osdmap e243: 8 total, 8 up, 7 in 2024-09-21T12:28:27.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:26 smithi114 ceph-mon[30324]: pgmap v399: 178 pgs: 178 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T12:28:27.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:26 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:28:27.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:26 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:28:27.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:26 smithi114 ceph-mon[30324]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T12:28:27.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:26 smithi114 ceph-mon[30324]: osdmap e242: 8 total, 8 up, 7 in 2024-09-21T12:28:27.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:26 smithi114 ceph-mon[30324]: 4.0 scrub starts 2024-09-21T12:28:27.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:26 smithi114 ceph-mon[30324]: 4.0 scrub ok 2024-09-21T12:28:27.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:26 smithi114 ceph-mon[30324]: osdmap e243: 8 total, 8 up, 7 in 2024-09-21T12:28:27.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:26 smithi114 ceph-mon[40867]: pgmap v399: 178 pgs: 178 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T12:28:27.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:26 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:28:27.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:26 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:28:27.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:26 smithi114 ceph-mon[40867]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T12:28:27.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:26 smithi114 ceph-mon[40867]: osdmap e242: 8 total, 8 up, 7 in 2024-09-21T12:28:27.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:26 smithi114 ceph-mon[40867]: 4.0 scrub starts 2024-09-21T12:28:27.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:26 smithi114 ceph-mon[40867]: 4.0 scrub ok 2024-09-21T12:28:27.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:26 smithi114 ceph-mon[40867]: osdmap e243: 8 total, 8 up, 7 in 2024-09-21T12:28:27.327 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:27.349 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:27.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:27.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:27.376 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T12:28:27.451 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:27.479 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:27.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:27.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:27.580 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:27.607 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:27.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:27.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:27.708 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:27.731 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:27.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:27.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:27.741 DEBUG:teuthology.orchestra.run.smithi114:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-09-21T12:28:27.774 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T12:28:27.774 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T12:28:27.824 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:hints = 1 2024-09-21T12:28:27.824 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T12:28:27.824 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Object prefix: benchmark_data_smithi114_133214 2024-09-21T12:28:27.825 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T12:28:27.825 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 0 0 0 0 0 0 - 0 2024-09-21T12:28:27.832 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:27.872 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:27.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:27.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:27.974 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:28.016 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:28.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:28.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:28.109 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:28.109 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:28.119 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:28.169 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:28.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:28.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:28.250 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:27 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:28:28.250 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:27 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:28:28.271 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:28.301 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:28.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:28.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:28.403 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:28.432 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:28.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:28.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:28.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:27 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:28:28.533 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:28.564 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:28.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:28.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:28.665 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:28.708 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:28.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:28.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:28.803 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:28:28.803 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 20, 2024-09-21T12:28:28.803 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 600, 2024-09-21T12:28:28.803 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-21T12:28:28.803 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.803 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:24383 7.8 7:1f298c15:::benchmark_data_smithi114_78432_object24382:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T12:28:28.803 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:39.998832+0000", 2024-09-21T12:28:28.804 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 408.79898342400003, 2024-09-21T12:28:28.804 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0658084639999998, 2024-09-21T12:28:28.804 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:28:28.804 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:28:28.804 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:28:28.804 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:28:28.804 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:28:28.804 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:28:28.804 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 24383 2024-09-21T12:28:28.804 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.804 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:28:28.804 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.804 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:28:28.805 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.998832+0000", 2024-09-21T12:28:28.805 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.805 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.805 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.805 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:28:28.805 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.998832+0000", 2024-09-21T12:28:28.805 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.805 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.805 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.805 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:28:28.805 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.998833+0000", 2024-09-21T12:28:28.806 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.37e-06 2024-09-21T12:28:28.806 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.806 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.806 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:28:28.806 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.998855+0000", 2024-09-21T12:28:28.806 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1846999999999999e-05 2024-09-21T12:28:28.806 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.806 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.806 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:28:28.806 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.998866+0000", 2024-09-21T12:28:28.806 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0363999999999999e-05 2024-09-21T12:28:28.806 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.807 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.807 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:28:28.807 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.998873+0000", 2024-09-21T12:28:28.807 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2110000000000001e-06 2024-09-21T12:28:28.807 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.807 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.807 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.807 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.998970+0000", 2024-09-21T12:28:28.807 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6693000000000004e-05 2024-09-21T12:28:28.807 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.807 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.808 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:28:28.808 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.999115+0000", 2024-09-21T12:28:28.808 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000145048 2024-09-21T12:28:28.808 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.808 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.808 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,7", 2024-09-21T12:28:28.808 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.999246+0000", 2024-09-21T12:28:28.808 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000131068 2024-09-21T12:28:28.808 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.808 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.808 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:28:28.808 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.002347+0000", 2024-09-21T12:28:28.809 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0031012959999999999 2024-09-21T12:28:28.809 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.809 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.809 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.809 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.002577+0000", 2024-09-21T12:28:28.809 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000229769 2024-09-21T12:28:28.809 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.809 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.809 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.809 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.010579+0000", 2024-09-21T12:28:28.809 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0080019189999999997 2024-09-21T12:28:28.809 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.809 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.810 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:28:28.810 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.010595+0000", 2024-09-21T12:28:28.810 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6161e-05 2024-09-21T12:28:28.810 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.810 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.810 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.810 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.020743+0000", 2024-09-21T12:28:28.810 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0101476890000001 2024-09-21T12:28:28.810 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.810 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.810 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:28:28.811 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.020791+0000", 2024-09-21T12:28:28.811 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.795e-05 2024-09-21T12:28:28.811 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.811 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.811 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.811 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.064509+0000", 2024-09-21T12:28:28.811 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0437187219999999 2024-09-21T12:28:28.811 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.811 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.811 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:28:28.811 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.064641+0000", 2024-09-21T12:28:28.811 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000131357 2024-09-21T12:28:28.812 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.812 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:28:28.812 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.812 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.812 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.812 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:24385 7.8 7:16dcbae5:::benchmark_data_smithi114_78432_object24384:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T12:28:28.812 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:39.998892+0000", 2024-09-21T12:28:28.812 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 408.79892310399998, 2024-09-21T12:28:28.812 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0668990809999999, 2024-09-21T12:28:28.812 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:28:28.812 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:28:28.812 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:28:28.813 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:28:28.813 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:28:28.813 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:28:28.813 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 24385 2024-09-21T12:28:28.813 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.813 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:28:28.813 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.813 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:28:28.813 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.998892+0000", 2024-09-21T12:28:28.813 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.813 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.813 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.813 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:28:28.814 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.998892+0000", 2024-09-21T12:28:28.814 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.814 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.814 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.814 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:28:28.814 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.998893+0000", 2024-09-21T12:28:28.814 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.06e-07 2024-09-21T12:28:28.814 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.814 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.814 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:28:28.814 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.998918+0000", 2024-09-21T12:28:28.814 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5378999999999999e-05 2024-09-21T12:28:28.814 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.815 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.815 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:28:28.815 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.998927+0000", 2024-09-21T12:28:28.815 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.3909999999999998e-06 2024-09-21T12:28:28.815 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.815 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.815 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:28:28.815 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:39.998932+0000", 2024-09-21T12:28:28.815 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6860000000000003e-06 2024-09-21T12:28:28.815 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.815 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.816 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.816 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.000469+0000", 2024-09-21T12:28:28.816 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001536716 2024-09-21T12:28:28.816 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.816 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.816 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:28:28.816 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.000583+0000", 2024-09-21T12:28:28.816 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000113354 2024-09-21T12:28:28.816 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.816 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.816 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,7", 2024-09-21T12:28:28.816 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.000713+0000", 2024-09-21T12:28:28.816 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000130932 2024-09-21T12:28:28.817 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.817 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.817 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:28:28.817 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.002357+0000", 2024-09-21T12:28:28.817 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001643207 2024-09-21T12:28:28.817 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.817 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.817 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.817 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.006431+0000", 2024-09-21T12:28:28.817 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0040742169999999998 2024-09-21T12:28:28.817 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.817 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.818 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.818 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.010634+0000", 2024-09-21T12:28:28.818 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0042035079999999999 2024-09-21T12:28:28.818 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.818 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.818 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:28:28.818 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.010644+0000", 2024-09-21T12:28:28.818 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1249999999999999e-06 2024-09-21T12:28:28.818 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.818 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.818 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.818 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.020852+0000", 2024-09-21T12:28:28.818 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0102084520000001 2024-09-21T12:28:28.819 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.819 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.819 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:28.820 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:28:28.820 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.020866+0000", 2024-09-21T12:28:28.820 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4199000000000001e-05 2024-09-21T12:28:28.820 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.820 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.820 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.820 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.065675+0000", 2024-09-21T12:28:28.820 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0448089519999999 2024-09-21T12:28:28.820 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.821 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.821 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:28:28.821 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.065792+0000", 2024-09-21T12:28:28.821 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000116357 2024-09-21T12:28:28.821 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.821 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:28:28.821 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.821 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.821 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.822 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:24410 7.8 7:1e9f2e1e:::benchmark_data_smithi114_78432_object24409:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:28:28.822 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:40.041873+0000", 2024-09-21T12:28:28.822 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 408.75594231299999, 2024-09-21T12:28:28.822 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0306425940000001, 2024-09-21T12:28:28.822 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:28:28.822 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:28:28.822 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:28.822 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:28:28.822 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:28:28.822 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:28:28.823 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 24410 2024-09-21T12:28:28.823 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.823 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:28:28.823 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.823 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:28:28.823 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.041873+0000", 2024-09-21T12:28:28.823 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.823 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.823 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.823 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:28:28.823 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.041873+0000", 2024-09-21T12:28:28.823 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.823 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.824 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.824 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:28:28.824 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.041875+0000", 2024-09-21T12:28:28.824 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.42e-06 2024-09-21T12:28:28.824 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.824 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.824 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:28:28.824 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.041896+0000", 2024-09-21T12:28:28.824 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1631e-05 2024-09-21T12:28:28.824 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.824 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.825 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:28:28.825 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.041928+0000", 2024-09-21T12:28:28.825 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.1977000000000002e-05 2024-09-21T12:28:28.825 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.825 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.825 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:28:28.825 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.041937+0000", 2024-09-21T12:28:28.825 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1649999999999995e-06 2024-09-21T12:28:28.825 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.825 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.825 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.825 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.042006+0000", 2024-09-21T12:28:28.825 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8824000000000004e-05 2024-09-21T12:28:28.826 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.826 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.826 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:28:28.826 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.042015+0000", 2024-09-21T12:28:28.826 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0909999999999999e-06 2024-09-21T12:28:28.826 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.826 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.826 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.826 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.020897+0000", 2024-09-21T12:28:28.826 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97888188200000004 2024-09-21T12:28:28.826 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.826 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.826 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:28:28.826 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.020908+0000", 2024-09-21T12:28:28.827 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1007e-05 2024-09-21T12:28:28.827 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.827 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.827 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.827 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.066247+0000", 2024-09-21T12:28:28.827 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.045338791 2024-09-21T12:28:28.827 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.827 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.827 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:28:28.827 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.066399+0000", 2024-09-21T12:28:28.827 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000151681 2024-09-21T12:28:28.827 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.827 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.828 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:28:28.828 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.066635+0000", 2024-09-21T12:28:28.828 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00023616899999999999 2024-09-21T12:28:28.828 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.828 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.828 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.828 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.070382+0000", 2024-09-21T12:28:28.828 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003747205 2024-09-21T12:28:28.828 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.828 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.829 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:28:28.829 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.070401+0000", 2024-09-21T12:28:28.829 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8635e-05 2024-09-21T12:28:28.829 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.829 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.829 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.829 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.071452+0000", 2024-09-21T12:28:28.829 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0010513250000000001 2024-09-21T12:28:28.829 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.829 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.829 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:28:28.829 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.071575+0000", 2024-09-21T12:28:28.829 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000122526 2024-09-21T12:28:28.829 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.830 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.830 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:28:28.830 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.072516+0000", 2024-09-21T12:28:28.830 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00094126500000000005 2024-09-21T12:28:28.830 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.830 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:28:28.830 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.830 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.830 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.830 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:24415 7.8 7:16bdf497:::benchmark_data_smithi114_78432_object24414:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:28:28.830 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:40.045617+0000", 2024-09-21T12:28:28.830 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 408.75219897400001, 2024-09-21T12:28:28.831 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0278312839999999, 2024-09-21T12:28:28.831 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:28:28.831 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:28:28.831 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:28.831 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:28:28.831 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:28:28.831 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:28:28.831 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 24415 2024-09-21T12:28:28.831 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.831 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:28:28.831 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.831 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:28:28.831 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.045617+0000", 2024-09-21T12:28:28.832 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.832 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.832 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.832 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:28:28.832 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.045617+0000", 2024-09-21T12:28:28.832 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.832 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.832 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.832 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:28:28.832 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.045618+0000", 2024-09-21T12:28:28.832 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.578e-06 2024-09-21T12:28:28.832 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.832 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.833 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:28:28.833 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.045639+0000", 2024-09-21T12:28:28.833 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1047999999999999e-05 2024-09-21T12:28:28.833 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.833 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.833 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:28:28.833 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.045653+0000", 2024-09-21T12:28:28.833 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3849000000000001e-05 2024-09-21T12:28:28.833 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.833 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.833 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:28:28.833 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.045660+0000", 2024-09-21T12:28:28.834 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8839999999999997e-06 2024-09-21T12:28:28.834 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.834 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.834 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.834 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.045723+0000", 2024-09-21T12:28:28.834 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2618000000000003e-05 2024-09-21T12:28:28.834 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.834 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.834 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:28:28.834 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:40.045732+0000", 2024-09-21T12:28:28.834 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7669999999999998e-06 2024-09-21T12:28:28.835 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.835 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.835 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.835 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.022590+0000", 2024-09-21T12:28:28.835 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97685747999999994 2024-09-21T12:28:28.835 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.835 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.835 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:28:28.835 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:41.022599+0000", 2024-09-21T12:28:28.835 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.4669999999999993e-06 2024-09-21T12:28:28.835 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.835 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.835 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.836 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.068684+0000", 2024-09-21T12:28:28.836 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0460851920000001 2024-09-21T12:28:28.836 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.836 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.836 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:28:28.836 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.068838+0000", 2024-09-21T12:28:28.836 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015381600000000001 2024-09-21T12:28:28.836 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.836 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.836 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:28:28.836 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.069029+0000", 2024-09-21T12:28:28.836 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000191227 2024-09-21T12:28:28.837 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.837 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.837 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:28:28.837 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.070406+0000", 2024-09-21T12:28:28.837 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0013769710000000001 2024-09-21T12:28:28.837 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.837 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.837 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.837 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.072571+0000", 2024-09-21T12:28:28.837 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0021641730000000001 2024-09-21T12:28:28.837 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.838 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.838 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.838 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.072668+0000", 2024-09-21T12:28:28.838 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7110999999999999e-05 2024-09-21T12:28:28.838 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.838 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.838 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:28:28.838 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.072724+0000", 2024-09-21T12:28:28.838 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6276000000000001e-05 2024-09-21T12:28:28.838 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.838 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.838 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:28:28.839 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.073448+0000", 2024-09-21T12:28:28.839 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00072382700000000002 2024-09-21T12:28:28.839 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.839 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:28:28.839 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.839 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.839 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.839 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:24791 7.8 7:153a621d:::benchmark_data_smithi114_78432_object24790:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T12:28:28.839 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:42.432155+0000", 2024-09-21T12:28:28.839 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 406.36566035200002, 2024-09-21T12:28:28.839 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.159980077, 2024-09-21T12:28:28.839 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:28:28.839 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:28:28.840 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:28.840 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:28:28.840 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:28:28.840 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:28:28.840 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 24791 2024-09-21T12:28:28.840 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.840 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:28:28.840 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.840 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:28:28.840 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.432155+0000", 2024-09-21T12:28:28.840 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.840 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.840 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.840 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:28:28.841 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.432155+0000", 2024-09-21T12:28:28.841 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.841 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.841 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.841 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:28:28.841 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.432157+0000", 2024-09-21T12:28:28.841 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8959999999999999e-06 2024-09-21T12:28:28.841 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.841 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.841 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:28:28.841 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.432179+0000", 2024-09-21T12:28:28.841 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1662999999999999e-05 2024-09-21T12:28:28.841 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.841 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.842 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:28:28.842 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.432189+0000", 2024-09-21T12:28:28.842 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.061e-05 2024-09-21T12:28:28.842 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.842 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.842 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:28:28.842 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.432196+0000", 2024-09-21T12:28:28.842 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3679999999999998e-06 2024-09-21T12:28:28.842 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.842 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.842 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.842 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.442074+0000", 2024-09-21T12:28:28.843 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0098782109999999996 2024-09-21T12:28:28.843 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.843 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.843 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:28:28.843 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.442218+0000", 2024-09-21T12:28:28.843 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014386499999999999 2024-09-21T12:28:28.843 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.843 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.843 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:28:28.843 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.442384+0000", 2024-09-21T12:28:28.843 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016588099999999999 2024-09-21T12:28:28.843 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.844 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.844 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.844 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.443960+0000", 2024-09-21T12:28:28.844 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0015763730000000001 2024-09-21T12:28:28.844 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.844 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.844 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:28:28.844 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.444343+0000", 2024-09-21T12:28:28.844 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00038294799999999998 2024-09-21T12:28:28.844 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.844 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.844 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.844 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.591068+0000", 2024-09-21T12:28:28.845 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.146724875 2024-09-21T12:28:28.845 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.845 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.845 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:28:28.845 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.591137+0000", 2024-09-21T12:28:28.845 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9438999999999997e-05 2024-09-21T12:28:28.845 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.845 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.845 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:28:28.845 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:42.592135+0000", 2024-09-21T12:28:28.845 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00099794800000000002 2024-09-21T12:28:28.845 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.846 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:28:28.846 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.846 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.846 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.846 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:45502 7.e 7:71c90ed0:::benchmark_data_smithi114_78432_object45501:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T12:28:28.846 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:59.294060+0000", 2024-09-21T12:28:28.846 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 389.50375558000002, 2024-09-21T12:28:28.846 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.16001768, 2024-09-21T12:28:28.846 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:28:28.846 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:28:28.846 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:28.846 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:28:28.847 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:28:28.847 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:28:28.847 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 45502 2024-09-21T12:28:28.847 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.849 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:28:28.849 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.849 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:28:28.849 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.294060+0000", 2024-09-21T12:28:28.849 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.850 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.850 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.850 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:28:28.850 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.294060+0000", 2024-09-21T12:28:28.850 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.850 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.850 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.850 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:28:28.850 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.294061+0000", 2024-09-21T12:28:28.850 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4440000000000001e-06 2024-09-21T12:28:28.850 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.850 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.851 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:28:28.851 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.294082+0000", 2024-09-21T12:28:28.851 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0598000000000002e-05 2024-09-21T12:28:28.851 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.851 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.851 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:28:28.851 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.294093+0000", 2024-09-21T12:28:28.851 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1158000000000001e-05 2024-09-21T12:28:28.851 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.851 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.851 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:28:28.851 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.294101+0000", 2024-09-21T12:28:28.851 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3499999999999999e-06 2024-09-21T12:28:28.852 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.852 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.852 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.852 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.294153+0000", 2024-09-21T12:28:28.852 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2098000000000002e-05 2024-09-21T12:28:28.852 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.852 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.852 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:28:28.852 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.294271+0000", 2024-09-21T12:28:28.852 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000118664 2024-09-21T12:28:28.852 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.852 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.853 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,5", 2024-09-21T12:28:28.853 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.294397+0000", 2024-09-21T12:28:28.853 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:28.853 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000125542 2024-09-21T12:28:28.853 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.853 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.854 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.854 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.296815+0000", 2024-09-21T12:28:28.854 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002417874 2024-09-21T12:28:28.854 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.854 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.854 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:28:28.854 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.296826+0000", 2024-09-21T12:28:28.854 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1321999999999999e-05 2024-09-21T12:28:28.854 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.854 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.854 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.854 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.453830+0000", 2024-09-21T12:28:28.854 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.15700447300000001 2024-09-21T12:28:28.855 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.855 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.855 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:28:28.855 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.454033+0000", 2024-09-21T12:28:28.855 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000202664 2024-09-21T12:28:28.855 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.855 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.855 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:28:28.855 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.454078+0000", 2024-09-21T12:28:28.855 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4493000000000002e-05 2024-09-21T12:28:28.855 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.855 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:28:28.855 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.856 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.856 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.856 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:45506 7.8 7:1157a2d2:::benchmark_data_smithi114_78432_object45505:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T12:28:28.856 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:59.297399+0000", 2024-09-21T12:28:28.856 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 389.50041604400002, 2024-09-21T12:28:28.856 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.160065606, 2024-09-21T12:28:28.856 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:28:28.856 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:28:28.856 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:28.856 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:28:28.856 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:28:28.856 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:28:28.857 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 45506 2024-09-21T12:28:28.857 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.857 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:28:28.857 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.857 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:28:28.857 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.297399+0000", 2024-09-21T12:28:28.857 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.857 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.857 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.857 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:28:28.857 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.297399+0000", 2024-09-21T12:28:28.857 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.857 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.857 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.858 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:28:28.858 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.297401+0000", 2024-09-21T12:28:28.858 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7969999999999999e-06 2024-09-21T12:28:28.858 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.858 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.858 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:28:28.858 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.297427+0000", 2024-09-21T12:28:28.858 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.6075e-05 2024-09-21T12:28:28.858 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.858 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.858 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:28:28.858 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.297440+0000", 2024-09-21T12:28:28.859 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.252e-05 2024-09-21T12:28:28.859 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.859 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.859 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:28:28.859 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.297447+0000", 2024-09-21T12:28:28.859 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2980000000000003e-06 2024-09-21T12:28:28.859 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.859 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.859 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.859 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.297599+0000", 2024-09-21T12:28:28.859 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015153599999999999 2024-09-21T12:28:28.859 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.860 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.860 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:28:28.860 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.297724+0000", 2024-09-21T12:28:28.860 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000125148 2024-09-21T12:28:28.860 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.860 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.860 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:28:28.860 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.297871+0000", 2024-09-21T12:28:28.860 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000147505 2024-09-21T12:28:28.860 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.860 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.860 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.860 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.301568+0000", 2024-09-21T12:28:28.861 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0036965230000000002 2024-09-21T12:28:28.861 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.861 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.861 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:28:28.861 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.301639+0000", 2024-09-21T12:28:28.861 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0963999999999993e-05 2024-09-21T12:28:28.861 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.861 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.861 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.861 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.456770+0000", 2024-09-21T12:28:28.861 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.15513142099999999 2024-09-21T12:28:28.861 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.862 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.862 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:28:28.862 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.456833+0000", 2024-09-21T12:28:28.862 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2525000000000002e-05 2024-09-21T12:28:28.862 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.862 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.862 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:28:28.862 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.457465+0000", 2024-09-21T12:28:28.862 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00063229400000000004 2024-09-21T12:28:28.862 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.862 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:28:28.862 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.863 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.863 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.863 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:45509 7.8 7:1989e657:::benchmark_data_smithi114_78432_object45508:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T12:28:28.863 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:59.297470+0000", 2024-09-21T12:28:28.863 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 389.50034584600002, 2024-09-21T12:28:28.863 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.164477763, 2024-09-21T12:28:28.863 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:28:28.863 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:28:28.863 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:28.863 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:28:28.863 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:28:28.863 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:28:28.863 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 45509 2024-09-21T12:28:28.864 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.864 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:28:28.864 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.864 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:28:28.864 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.297470+0000", 2024-09-21T12:28:28.864 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.864 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.864 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.864 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:28:28.864 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.297470+0000", 2024-09-21T12:28:28.864 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.864 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.865 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.865 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:28:28.865 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.297470+0000", 2024-09-21T12:28:28.865 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3700000000000001e-07 2024-09-21T12:28:28.865 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.865 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.865 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:28:28.865 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.297493+0000", 2024-09-21T12:28:28.865 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2374e-05 2024-09-21T12:28:28.865 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.865 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.865 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:28:28.865 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.297499+0000", 2024-09-21T12:28:28.866 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9870000000000002e-06 2024-09-21T12:28:28.866 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.866 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.866 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:28:28.866 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.297505+0000", 2024-09-21T12:28:28.866 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.074e-06 2024-09-21T12:28:28.866 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.866 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.866 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.866 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.299566+0000", 2024-09-21T12:28:28.866 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0020616940000000002 2024-09-21T12:28:28.866 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.866 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.867 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:28:28.867 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.299708+0000", 2024-09-21T12:28:28.867 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014215500000000001 2024-09-21T12:28:28.867 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.867 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.867 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:28:28.867 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.299865+0000", 2024-09-21T12:28:28.867 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000156704 2024-09-21T12:28:28.867 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.867 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.867 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.867 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.302117+0000", 2024-09-21T12:28:28.867 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002251739 2024-09-21T12:28:28.868 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.868 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.868 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:28:28.868 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.302132+0000", 2024-09-21T12:28:28.868 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5095e-05 2024-09-21T12:28:28.868 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.868 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.868 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.868 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.461162+0000", 2024-09-21T12:28:28.868 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.159030112 2024-09-21T12:28:28.869 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 1 16 3319 3303 206.426 206.438 0.00705499 0.00482734 2024-09-21T12:28:28.869 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.869 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.869 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:28:28.869 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.461226+0000", 2024-09-21T12:28:28.870 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4228999999999998e-05 2024-09-21T12:28:28.870 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.870 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.870 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:28:28.870 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.461947+0000", 2024-09-21T12:28:28.870 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00072116300000000003 2024-09-21T12:28:28.870 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.870 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:28:28.870 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.870 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.870 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.871 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:45516 7.e 7:78799872:::benchmark_data_smithi114_78432_object45515:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T12:28:28.871 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:21:59.304654+0000", 2024-09-21T12:28:28.871 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 389.49316119500003, 2024-09-21T12:28:28.871 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.15534616300000001, 2024-09-21T12:28:28.871 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:28:28.871 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:28:28.871 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:28.871 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:28:28.871 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:28:28.871 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:28:28.871 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 45516 2024-09-21T12:28:28.871 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.872 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:28:28.872 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.872 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:28:28.872 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.304654+0000", 2024-09-21T12:28:28.872 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.872 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.872 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.872 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:28:28.872 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.304654+0000", 2024-09-21T12:28:28.872 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.872 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.873 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.873 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:28:28.873 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.304656+0000", 2024-09-21T12:28:28.873 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9139999999999998e-06 2024-09-21T12:28:28.873 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.873 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.873 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:28:28.873 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.304674+0000", 2024-09-21T12:28:28.873 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7822999999999999e-05 2024-09-21T12:28:28.873 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.873 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.873 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:28:28.873 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.304684+0000", 2024-09-21T12:28:28.874 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.8830000000000001e-06 2024-09-21T12:28:28.874 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.874 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.874 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:28:28.874 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.304689+0000", 2024-09-21T12:28:28.874 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.446e-06 2024-09-21T12:28:28.874 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.874 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.874 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.874 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.304743+0000", 2024-09-21T12:28:28.874 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3708000000000001e-05 2024-09-21T12:28:28.875 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.875 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.875 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:28:28.875 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.304821+0000", 2024-09-21T12:28:28.875 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7923000000000006e-05 2024-09-21T12:28:28.875 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.875 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.875 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,5", 2024-09-21T12:28:28.875 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.304900+0000", 2024-09-21T12:28:28.875 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8699e-05 2024-09-21T12:28:28.875 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.875 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.876 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:28:28.876 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.306078+0000", 2024-09-21T12:28:28.876 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00117793 2024-09-21T12:28:28.876 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.876 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.876 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.876 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.306379+0000", 2024-09-21T12:28:28.876 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00030152999999999999 2024-09-21T12:28:28.876 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.876 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.876 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.876 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.459763+0000", 2024-09-21T12:28:28.877 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.153383995 2024-09-21T12:28:28.877 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.877 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.877 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:28:28.877 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.459954+0000", 2024-09-21T12:28:28.877 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00019092100000000001 2024-09-21T12:28:28.877 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.877 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.877 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:28:28.877 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:21:59.460001+0000", 2024-09-21T12:28:28.877 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6390999999999997e-05 2024-09-21T12:28:28.877 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.877 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:28:28.878 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.878 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.878 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.878 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46523 7.8 7:11c4ba2e:::benchmark_data_smithi114_78432_object46522:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:28:28.878 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.793498+0000", 2024-09-21T12:28:28.878 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 387.004317238, 2024-09-21T12:28:28.878 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0224531560000001, 2024-09-21T12:28:28.878 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:28:28.878 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:28:28.878 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:28.878 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:28:28.878 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:28:28.878 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:28:28.879 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46523 2024-09-21T12:28:28.879 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.879 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:28:28.879 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.879 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:28:28.879 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.793498+0000", 2024-09-21T12:28:28.879 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.879 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.879 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.879 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:28:28.879 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.793498+0000", 2024-09-21T12:28:28.879 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.879 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.880 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.880 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:28:28.880 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.793500+0000", 2024-09-21T12:28:28.880 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9379999999999999e-06 2024-09-21T12:28:28.880 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.880 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.880 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:28:28.880 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.793526+0000", 2024-09-21T12:28:28.880 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5816000000000002e-05 2024-09-21T12:28:28.880 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.880 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.880 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:28:28.880 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.793544+0000", 2024-09-21T12:28:28.881 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7889000000000001e-05 2024-09-21T12:28:28.881 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.881 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.881 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:28:28.881 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.793552+0000", 2024-09-21T12:28:28.881 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.3159999999999997e-06 2024-09-21T12:28:28.881 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.881 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.881 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.881 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.793610+0000", 2024-09-21T12:28:28.881 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8180999999999997e-05 2024-09-21T12:28:28.881 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.881 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.882 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:28:28.882 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.793617+0000", 2024-09-21T12:28:28.882 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.691e-06 2024-09-21T12:28:28.882 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.882 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.882 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.882 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.811363+0000", 2024-09-21T12:28:28.882 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.01774579 2024-09-21T12:28:28.882 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.882 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.882 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:28:28.883 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.811475+0000", 2024-09-21T12:28:28.883 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000111846 2024-09-21T12:28:28.883 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.883 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.883 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:28:28.883 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.811637+0000", 2024-09-21T12:28:28.883 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000162268 2024-09-21T12:28:28.883 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.883 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.883 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.883 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.815751+0000", 2024-09-21T12:28:28.883 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0041139089999999998 2024-09-21T12:28:28.883 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.884 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.884 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.884 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.815826+0000", 2024-09-21T12:28:28.884 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4887000000000003e-05 2024-09-21T12:28:28.884 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.884 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.884 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:28:28.884 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.815842+0000", 2024-09-21T12:28:28.884 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6518999999999999e-05 2024-09-21T12:28:28.884 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.884 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.884 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:28:28.884 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.815905+0000", 2024-09-21T12:28:28.885 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2515999999999996e-05 2024-09-21T12:28:28.885 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.885 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.885 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:28:28.885 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.815951+0000", 2024-09-21T12:28:28.885 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6589999999999999e-05 2024-09-21T12:28:28.885 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.885 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:28:28.885 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.885 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.885 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.885 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46531 7.8 7:1bc837da:::benchmark_data_smithi114_78432_object46530:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:28:28.885 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.804524+0000", 2024-09-21T12:28:28.885 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 386.99329159299998, 2024-09-21T12:28:28.886 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0155765649999999, 2024-09-21T12:28:28.886 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:28:28.886 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:28:28.886 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:28.886 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:28:28.886 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:28:28.886 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:28:28.886 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46531 2024-09-21T12:28:28.886 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.886 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:28:28.886 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.886 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:28:28.886 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.804524+0000", 2024-09-21T12:28:28.887 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.887 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.887 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.887 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:28:28.887 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.804524+0000", 2024-09-21T12:28:28.887 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.887 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.887 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.887 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:28:28.887 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.804526+0000", 2024-09-21T12:28:28.887 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6139999999999999e-06 2024-09-21T12:28:28.888 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.888 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.888 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:28:28.888 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.804558+0000", 2024-09-21T12:28:28.888 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2119000000000002e-05 2024-09-21T12:28:28.888 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.888 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.888 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:28:28.888 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.804578+0000", 2024-09-21T12:28:28.888 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.001e-05 2024-09-21T12:28:28.888 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.888 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.888 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:28:28.888 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.804586+0000", 2024-09-21T12:28:28.889 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1210000000000007e-06 2024-09-21T12:28:28.889 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.889 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.889 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.889 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.804675+0000", 2024-09-21T12:28:28.889 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9492000000000003e-05 2024-09-21T12:28:28.889 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.889 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.889 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:28:28.889 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.804685+0000", 2024-09-21T12:28:28.889 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.9529999999999994e-06 2024-09-21T12:28:28.889 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.889 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.890 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.890 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.813729+0000", 2024-09-21T12:28:28.890 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0090438020000001 2024-09-21T12:28:28.890 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.890 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.890 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:28:28.890 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.813870+0000", 2024-09-21T12:28:28.890 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014118099999999999 2024-09-21T12:28:28.890 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.890 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.890 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:28:28.890 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.814037+0000", 2024-09-21T12:28:28.891 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016691199999999999 2024-09-21T12:28:28.891 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.891 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.891 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.891 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.816120+0000", 2024-09-21T12:28:28.891 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0020824039999999999 2024-09-21T12:28:28.891 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.891 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.891 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:28:28.891 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.817843+0000", 2024-09-21T12:28:28.891 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0017235390000000001 2024-09-21T12:28:28.891 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.891 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.892 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.892 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.819999+0000", 2024-09-21T12:28:28.892 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0021563659999999998 2024-09-21T12:28:28.892 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.892 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.892 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:28:28.892 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.820050+0000", 2024-09-21T12:28:28.892 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0439999999999998e-05 2024-09-21T12:28:28.892 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.892 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.892 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:28:28.892 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.820101+0000", 2024-09-21T12:28:28.892 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0612000000000003e-05 2024-09-21T12:28:28.892 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.893 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:28:28.893 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.893 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.893 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.893 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46554 7.8 7:1ef36561:::benchmark_data_smithi114_78432_object46553:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:28:28.893 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.823773+0000", 2024-09-21T12:28:28.893 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 386.97404294699999, 2024-09-21T12:28:28.893 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99858891699999996, 2024-09-21T12:28:28.893 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:28:28.893 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:28:28.893 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:28.893 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:28:28.893 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:28:28.893 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:28:28.894 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46554 2024-09-21T12:28:28.894 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.894 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:28:28.894 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.894 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:28:28.894 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.823773+0000", 2024-09-21T12:28:28.894 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.894 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.894 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.894 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:28:28.894 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.823773+0000", 2024-09-21T12:28:28.894 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.894 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.895 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.895 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:28:28.895 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.823774+0000", 2024-09-21T12:28:28.895 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.077e-06 2024-09-21T12:28:28.895 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.895 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.895 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:28:28.895 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.823797+0000", 2024-09-21T12:28:28.895 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3478000000000001e-05 2024-09-21T12:28:28.895 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.895 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.895 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:28:28.895 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.823811+0000", 2024-09-21T12:28:28.896 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3878e-05 2024-09-21T12:28:28.896 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.896 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.896 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:28:28.896 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.823819+0000", 2024-09-21T12:28:28.896 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9810000000000003e-06 2024-09-21T12:28:28.896 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.896 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.896 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.896 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.823933+0000", 2024-09-21T12:28:28.896 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000113807 2024-09-21T12:28:28.896 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.897 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.897 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:28:28.897 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.823942+0000", 2024-09-21T12:28:28.897 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.5459999999999997e-06 2024-09-21T12:28:28.897 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.897 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.897 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.897 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.816164+0000", 2024-09-21T12:28:28.897 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99222121799999996 2024-09-21T12:28:28.897 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.897 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.897 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:28:28.897 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.816287+0000", 2024-09-21T12:28:28.897 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012309699999999999 2024-09-21T12:28:28.897 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.898 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.898 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:28:28.898 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.816419+0000", 2024-09-21T12:28:28.898 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013215199999999999 2024-09-21T12:28:28.898 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.898 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.898 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.898 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.820187+0000", 2024-09-21T12:28:28.898 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003768633 2024-09-21T12:28:28.898 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.898 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.898 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.898 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.820256+0000", 2024-09-21T12:28:28.898 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8388999999999999e-05 2024-09-21T12:28:28.899 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.899 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.899 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:28:28.899 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.822263+0000", 2024-09-21T12:28:28.899 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0020074030000000001 2024-09-21T12:28:28.899 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.899 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.899 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:28:28.899 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.822317+0000", 2024-09-21T12:28:28.899 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4095999999999998e-05 2024-09-21T12:28:28.899 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.899 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.899 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:28:28.900 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.822362+0000", 2024-09-21T12:28:28.900 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4162000000000002e-05 2024-09-21T12:28:28.900 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.900 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:28:28.900 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.900 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.900 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.900 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46568 7.8 7:1389cba8:::benchmark_data_smithi114_78432_object46567:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:28:28.900 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.834184+0000", 2024-09-21T12:28:28.900 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 386.96363122100001, 2024-09-21T12:28:28.900 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99022050800000005, 2024-09-21T12:28:28.900 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:28:28.900 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:28:28.900 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:28.901 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:28:28.901 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:28:28.901 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:28:28.901 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46568 2024-09-21T12:28:28.901 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.901 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:28:28.901 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.901 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:28:28.901 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.834184+0000", 2024-09-21T12:28:28.901 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.902 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.902 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.902 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:28:28.902 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.834184+0000", 2024-09-21T12:28:28.902 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.902 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.902 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.902 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:28:28.902 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.834186+0000", 2024-09-21T12:28:28.902 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9410000000000001e-06 2024-09-21T12:28:28.902 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.903 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.903 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:28:28.903 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.834208+0000", 2024-09-21T12:28:28.903 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2039999999999999e-05 2024-09-21T12:28:28.903 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.903 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.903 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:28:28.903 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.834225+0000", 2024-09-21T12:28:28.903 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6875999999999999e-05 2024-09-21T12:28:28.903 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.903 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.904 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:28:28.904 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.834233+0000", 2024-09-21T12:28:28.904 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8199999999999997e-06 2024-09-21T12:28:28.904 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.904 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.904 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.904 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.834320+0000", 2024-09-21T12:28:28.904 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.7083000000000006e-05 2024-09-21T12:28:28.904 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.904 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.904 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:28:28.904 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.834331+0000", 2024-09-21T12:28:28.905 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1090000000000001e-05 2024-09-21T12:28:28.905 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.905 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.905 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.905 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.817892+0000", 2024-09-21T12:28:28.905 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98356091400000001 2024-09-21T12:28:28.905 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.905 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.905 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:28:28.905 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.818008+0000", 2024-09-21T12:28:28.905 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000115606 2024-09-21T12:28:28.906 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.906 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.906 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:28:28.906 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.818251+0000", 2024-09-21T12:28:28.906 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00024378099999999999 2024-09-21T12:28:28.906 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.906 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.906 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:28:28.906 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.822369+0000", 2024-09-21T12:28:28.906 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0041175159999999999 2024-09-21T12:28:28.906 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.907 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.907 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.907 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.824256+0000", 2024-09-21T12:28:28.907 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0018866620000000001 2024-09-21T12:28:28.907 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.907 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.907 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.907 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.824304+0000", 2024-09-21T12:28:28.907 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8281000000000001e-05 2024-09-21T12:28:28.907 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.907 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.907 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:28:28.908 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.824361+0000", 2024-09-21T12:28:28.908 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6783000000000002e-05 2024-09-21T12:28:28.908 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.908 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.908 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:28:28.908 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.824405+0000", 2024-09-21T12:28:28.908 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4115e-05 2024-09-21T12:28:28.908 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.908 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:28:28.908 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.908 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.908 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.909 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46574 7.8 7:11204ae1:::benchmark_data_smithi114_78432_object46573:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:28:28.909 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.838311+0000", 2024-09-21T12:28:28.909 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 386.95950410099999, 2024-09-21T12:28:28.909 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99371615499999999, 2024-09-21T12:28:28.909 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:28:28.909 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:28:28.909 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:28.909 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:28:28.909 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:28:28.909 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:28:28.909 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46574 2024-09-21T12:28:28.909 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.909 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:28:28.910 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.910 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:28:28.910 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.838311+0000", 2024-09-21T12:28:28.910 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.910 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.910 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.910 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:28:28.910 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.838311+0000", 2024-09-21T12:28:28.910 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.910 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.910 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.911 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:28:28.911 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.838313+0000", 2024-09-21T12:28:28.911 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.854e-06 2024-09-21T12:28:28.911 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.911 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.911 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:28:28.911 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.838337+0000", 2024-09-21T12:28:28.911 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3574000000000002e-05 2024-09-21T12:28:28.911 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.911 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.911 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:28:28.911 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.838354+0000", 2024-09-21T12:28:28.912 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.702e-05 2024-09-21T12:28:28.912 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.912 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.912 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:28:28.912 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.838363+0000", 2024-09-21T12:28:28.912 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9150000000000002e-06 2024-09-21T12:28:28.912 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.912 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.912 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.912 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.838438+0000", 2024-09-21T12:28:28.912 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5680999999999996e-05 2024-09-21T12:28:28.912 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.913 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.913 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:28:28.913 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.838448+0000", 2024-09-21T12:28:28.913 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1409999999999994e-06 2024-09-21T12:28:28.913 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.913 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.913 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.913 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.820308+0000", 2024-09-21T12:28:28.913 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98186004500000001 2024-09-21T12:28:28.913 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.913 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.913 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:28:28.913 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.820415+0000", 2024-09-21T12:28:28.914 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010713 2024-09-21T12:28:28.914 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.914 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.914 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:28:28.914 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.820568+0000", 2024-09-21T12:28:28.914 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000152907 2024-09-21T12:28:28.914 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.914 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.914 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:28:28.914 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.824267+0000", 2024-09-21T12:28:28.914 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0036996249999999998 2024-09-21T12:28:28.914 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.915 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.916 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.916 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.828377+0000", 2024-09-21T12:28:28.916 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0041097740000000001 2024-09-21T12:28:28.916 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.916 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.916 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.917 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.831764+0000", 2024-09-21T12:28:28.917 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0033872970000000001 2024-09-21T12:28:28.917 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.917 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.917 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:28:28.917 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.831972+0000", 2024-09-21T12:28:28.917 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000207512 2024-09-21T12:28:28.917 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.917 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.917 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:28:28.917 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.832028+0000", 2024-09-21T12:28:28.917 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5680000000000002e-05 2024-09-21T12:28:28.917 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.918 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:28:28.918 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.918 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.918 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.918 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46597 7.8 7:113cfed4:::benchmark_data_smithi114_78432_object46596:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:28:28.918 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.859457+0000", 2024-09-21T12:28:28.918 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 386.93835873900002, 2024-09-21T12:28:28.918 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97467767900000002, 2024-09-21T12:28:28.918 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:28:28.918 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:28:28.918 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:28.918 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:28:28.919 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:28:28.919 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:28:28.919 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46597 2024-09-21T12:28:28.919 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.919 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:28:28.919 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.919 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:28:28.919 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.859457+0000", 2024-09-21T12:28:28.919 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.919 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.919 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.919 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:28:28.919 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.859457+0000", 2024-09-21T12:28:28.919 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.920 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.920 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.920 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:28:28.920 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.859458+0000", 2024-09-21T12:28:28.920 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.618e-06 2024-09-21T12:28:28.920 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.920 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.920 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:28:28.920 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.859479+0000", 2024-09-21T12:28:28.920 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0678000000000001e-05 2024-09-21T12:28:28.920 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.920 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.920 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:28:28.921 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.859501+0000", 2024-09-21T12:28:28.921 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1908000000000001e-05 2024-09-21T12:28:28.921 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.921 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.921 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:28:28.921 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.859508+0000", 2024-09-21T12:28:28.921 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4610000000000002e-06 2024-09-21T12:28:28.921 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.921 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.921 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.921 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.859569+0000", 2024-09-21T12:28:28.921 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0092e-05 2024-09-21T12:28:28.922 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.922 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.922 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:28:28.922 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.859575+0000", 2024-09-21T12:28:28.922 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8390000000000001e-06 2024-09-21T12:28:28.922 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.922 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.922 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.922 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.822397+0000", 2024-09-21T12:28:28.922 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.96282112500000006 2024-09-21T12:28:28.922 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.922 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.923 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:28:28.923 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.822504+0000", 2024-09-21T12:28:28.923 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010725700000000001 2024-09-21T12:28:28.923 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.923 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.923 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:28:28.923 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.822635+0000", 2024-09-21T12:28:28.923 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013144299999999999 2024-09-21T12:28:28.923 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.923 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.923 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:28:28.923 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.826386+0000", 2024-09-21T12:28:28.923 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0037504970000000002 2024-09-21T12:28:28.924 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.924 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.924 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.924 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.833905+0000", 2024-09-21T12:28:28.924 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0075191850000000003 2024-09-21T12:28:28.924 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.924 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.924 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.924 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.834031+0000", 2024-09-21T12:28:28.924 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000126091 2024-09-21T12:28:28.924 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.924 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.925 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:28:28.925 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.834088+0000", 2024-09-21T12:28:28.925 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6740999999999998e-05 2024-09-21T12:28:28.925 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.925 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.925 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:28:28.925 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.834134+0000", 2024-09-21T12:28:28.925 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6743999999999998e-05 2024-09-21T12:28:28.925 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.925 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:28:28.925 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.925 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.925 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.925 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46610 7.8 7:1a279a8a:::benchmark_data_smithi114_78432_object46609:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:28:28.926 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.873647+0000", 2024-09-21T12:28:28.926 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 386.92416858600001, 2024-09-21T12:28:28.926 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.96062798800000004, 2024-09-21T12:28:28.926 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:28:28.926 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:28:28.926 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:28.926 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:28:28.926 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:28:28.926 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:28:28.926 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46610 2024-09-21T12:28:28.926 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.926 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:28:28.927 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.927 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:28:28.927 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.873647+0000", 2024-09-21T12:28:28.927 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.927 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.927 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.927 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:28:28.927 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.873647+0000", 2024-09-21T12:28:28.927 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.927 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.927 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.927 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:28:28.927 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.873649+0000", 2024-09-21T12:28:28.927 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5999999999999999e-06 2024-09-21T12:28:28.928 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.928 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.928 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:28:28.928 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.873670+0000", 2024-09-21T12:28:28.928 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1336e-05 2024-09-21T12:28:28.928 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.928 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.928 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:28:28.928 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.873684+0000", 2024-09-21T12:28:28.928 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4185000000000001e-05 2024-09-21T12:28:28.928 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.928 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.928 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:28:28.929 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.873691+0000", 2024-09-21T12:28:28.929 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7429999999999996e-06 2024-09-21T12:28:28.929 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.929 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.929 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.929 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.873748+0000", 2024-09-21T12:28:28.929 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6804e-05 2024-09-21T12:28:28.929 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.929 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.929 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:28:28.929 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.873755+0000", 2024-09-21T12:28:28.929 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0700000000000001e-06 2024-09-21T12:28:28.929 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.930 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.930 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.930 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.824447+0000", 2024-09-21T12:28:28.930 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.95069197299999997 2024-09-21T12:28:28.930 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.930 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.930 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:28:28.930 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.824554+0000", 2024-09-21T12:28:28.930 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000107299 2024-09-21T12:28:28.930 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.930 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.930 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:28:28.930 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.824691+0000", 2024-09-21T12:28:28.930 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013733500000000001 2024-09-21T12:28:28.931 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.931 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.931 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:28:28.931 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.833924+0000", 2024-09-21T12:28:28.931 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0092329390000000008 2024-09-21T12:28:28.931 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.931 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.931 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.931 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.833971+0000", 2024-09-21T12:28:28.931 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6675000000000003e-05 2024-09-21T12:28:28.931 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.931 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.932 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.932 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.834188+0000", 2024-09-21T12:28:28.932 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000216987 2024-09-21T12:28:28.932 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.932 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.932 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:28:28.932 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.834240+0000", 2024-09-21T12:28:28.932 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2129999999999997e-05 2024-09-21T12:28:28.932 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.932 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.932 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:28:28.932 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.834275+0000", 2024-09-21T12:28:28.932 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4912e-05 2024-09-21T12:28:28.933 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.933 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:28:28.933 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.933 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.933 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.933 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:46657 7.8 7:1a3a6588:::benchmark_data_smithi114_78432_object46656:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T12:28:28.933 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:01.970902+0000", 2024-09-21T12:28:28.933 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 386.82691389500002, 2024-09-21T12:28:28.933 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.86373514500000004, 2024-09-21T12:28:28.933 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:28:28.933 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:28:28.933 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:28.933 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:28:28.934 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:28:28.934 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:28:28.934 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46657 2024-09-21T12:28:28.934 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.934 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:28:28.934 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.934 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:28:28.934 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.970902+0000", 2024-09-21T12:28:28.934 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.934 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.934 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.934 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:28:28.935 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.970902+0000", 2024-09-21T12:28:28.935 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.935 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.935 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.935 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:28:28.935 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.970904+0000", 2024-09-21T12:28:28.935 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8700000000000001e-06 2024-09-21T12:28:28.935 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.935 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.935 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:28:28.935 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.970925+0000", 2024-09-21T12:28:28.935 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1316999999999999e-05 2024-09-21T12:28:28.935 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.936 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.936 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:28:28.936 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.970942+0000", 2024-09-21T12:28:28.936 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7574000000000001e-05 2024-09-21T12:28:28.936 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.936 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.936 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:28:28.936 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.970950+0000", 2024-09-21T12:28:28.936 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2080000000000003e-06 2024-09-21T12:28:28.936 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.936 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.936 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.936 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.971012+0000", 2024-09-21T12:28:28.937 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2797000000000002e-05 2024-09-21T12:28:28.937 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.937 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.937 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:28:28.937 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:01.971026+0000", 2024-09-21T12:28:28.937 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.314e-05 2024-09-21T12:28:28.937 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.937 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.937 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.937 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.826416+0000", 2024-09-21T12:28:28.937 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.85538999699999996 2024-09-21T12:28:28.937 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.937 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.938 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:28:28.938 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.826517+0000", 2024-09-21T12:28:28.938 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000101659 2024-09-21T12:28:28.938 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.938 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.938 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:28:28.938 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.826643+0000", 2024-09-21T12:28:28.938 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000126198 2024-09-21T12:28:28.938 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.938 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.938 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:28:28.938 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.833933+0000", 2024-09-21T12:28:28.939 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0072896000000000002 2024-09-21T12:28:28.939 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.939 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.939 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.939 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.834316+0000", 2024-09-21T12:28:28.939 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00038313900000000001 2024-09-21T12:28:28.939 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.939 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.939 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.939 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.834533+0000", 2024-09-21T12:28:28.939 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00021702000000000001 2024-09-21T12:28:28.939 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.940 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.940 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:28:28.940 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.834591+0000", 2024-09-21T12:28:28.940 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7979999999999997e-05 2024-09-21T12:28:28.940 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.940 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.940 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:28:28.940 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:02.834637+0000", 2024-09-21T12:28:28.940 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5646000000000003e-05 2024-09-21T12:28:28.940 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.940 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:28:28.940 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.941 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.941 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.941 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:49225 7.e 7:78039099:::benchmark_data_smithi114_78432_object49224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:28:28.941 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:04.569229+0000", 2024-09-21T12:28:28.941 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 384.22858614699999, 2024-09-21T12:28:28.941 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.14412950999999999, 2024-09-21T12:28:28.941 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:28:28.941 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:28:28.941 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:28.941 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:28:28.941 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:28:28.942 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:28:28.942 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 49225 2024-09-21T12:28:28.942 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.942 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:28:28.942 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.942 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:28:28.942 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.569229+0000", 2024-09-21T12:28:28.942 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.942 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.942 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.942 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:28:28.942 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.569229+0000", 2024-09-21T12:28:28.943 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.943 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.943 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.943 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:28:28.943 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.569235+0000", 2024-09-21T12:28:28.943 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1109999999999999e-06 2024-09-21T12:28:28.943 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.943 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.943 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:28:28.943 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.569276+0000", 2024-09-21T12:28:28.943 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1179e-05 2024-09-21T12:28:28.944 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.944 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.944 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:28:28.944 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.569292+0000", 2024-09-21T12:28:28.944 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6422999999999998e-05 2024-09-21T12:28:28.944 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.944 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.944 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:28:28.944 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.569309+0000", 2024-09-21T12:28:28.944 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6631000000000001e-05 2024-09-21T12:28:28.944 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.944 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.945 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.945 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.569383+0000", 2024-09-21T12:28:28.945 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4114000000000002e-05 2024-09-21T12:28:28.945 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.945 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.945 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:28:28.945 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.569522+0000", 2024-09-21T12:28:28.945 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013892 2024-09-21T12:28:28.945 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.945 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.945 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,5", 2024-09-21T12:28:28.945 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.569657+0000", 2024-09-21T12:28:28.946 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013499900000000001 2024-09-21T12:28:28.946 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.946 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.946 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.946 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.572520+0000", 2024-09-21T12:28:28.946 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0028635930000000002 2024-09-21T12:28:28.946 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.946 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.946 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:28:28.946 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.572542+0000", 2024-09-21T12:28:28.946 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1373999999999999e-05 2024-09-21T12:28:28.946 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.947 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.947 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.947 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.713243+0000", 2024-09-21T12:28:28.947 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.140700941 2024-09-21T12:28:28.947 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.947 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.947 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:28:28.947 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.713309+0000", 2024-09-21T12:28:28.947 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6767999999999995e-05 2024-09-21T12:28:28.947 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.947 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.947 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:28:28.947 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.713359+0000", 2024-09-21T12:28:28.948 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9456999999999998e-05 2024-09-21T12:28:28.948 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.948 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:28:28.948 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.948 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.948 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.948 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:49235 7.8 7:1f0dee80:::benchmark_data_smithi114_78432_object49234:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:28:28.948 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:22:04.574390+0000", 2024-09-21T12:28:28.948 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 384.22342596999999, 2024-09-21T12:28:28.948 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.141800871, 2024-09-21T12:28:28.948 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:28:28.948 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:28:28.949 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:28.949 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:28:28.949 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:28:28.949 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:28:28.949 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 49235 2024-09-21T12:28:28.949 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.949 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:28:28.949 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.949 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:28:28.949 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.574390+0000", 2024-09-21T12:28:28.949 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.949 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.949 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.950 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:28:28.950 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.574390+0000", 2024-09-21T12:28:28.950 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.950 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.950 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.950 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:28:28.950 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.574391+0000", 2024-09-21T12:28:28.950 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.753e-06 2024-09-21T12:28:28.950 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.950 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.950 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:28:28.950 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.574414+0000", 2024-09-21T12:28:28.951 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3071e-05 2024-09-21T12:28:28.951 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.951 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.951 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:28:28.951 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.574428+0000", 2024-09-21T12:28:28.951 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3309000000000001e-05 2024-09-21T12:28:28.951 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.951 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.951 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:28:28.951 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.574435+0000", 2024-09-21T12:28:28.951 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4050000000000003e-06 2024-09-21T12:28:28.951 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.952 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.952 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.952 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.574494+0000", 2024-09-21T12:28:28.952 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8749000000000003e-05 2024-09-21T12:28:28.952 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.952 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.952 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:28:28.952 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.574623+0000", 2024-09-21T12:28:28.952 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012902700000000001 2024-09-21T12:28:28.952 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.952 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.952 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:28:28.952 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.574749+0000", 2024-09-21T12:28:28.953 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000125897 2024-09-21T12:28:28.953 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.953 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.953 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.953 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.576774+0000", 2024-09-21T12:28:28.953 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002025542 2024-09-21T12:28:28.953 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.953 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.953 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:28:28.953 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.576879+0000", 2024-09-21T12:28:28.953 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000105079 2024-09-21T12:28:28.953 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.953 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.954 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.954 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.716089+0000", 2024-09-21T12:28:28.954 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.139209902 2024-09-21T12:28:28.954 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.954 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.954 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:28:28.954 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.716149+0000", 2024-09-21T12:28:28.954 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9413000000000002e-05 2024-09-21T12:28:28.954 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.954 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.954 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:28:28.954 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:22:04.716190+0000", 2024-09-21T12:28:28.954 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1724000000000002e-05 2024-09-21T12:28:28.955 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.955 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:28:28.955 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.955 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.955 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.955 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.24695.0:116825 7.8 7:1dbf5839:::benchmark_data_smithi114_78432_object12679:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T12:28:28.955 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:23:12.374864+0000", 2024-09-21T12:28:28.955 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 316.422951861, 2024-09-21T12:28:28.955 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.16173959600000001, 2024-09-21T12:28:28.955 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:28:28.955 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:28:28.955 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:28.955 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:28:28.956 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.24695", 2024-09-21T12:28:28.956 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:28:28.956 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 116825 2024-09-21T12:28:28.956 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.956 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:28:28.956 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.956 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:28:28.956 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:12.374864+0000", 2024-09-21T12:28:28.956 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.956 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.956 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.956 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:28:28.956 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:12.374864+0000", 2024-09-21T12:28:28.957 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:28:28.957 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.957 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.957 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:28:28.957 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:12.374865+0000", 2024-09-21T12:28:28.957 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.373e-06 2024-09-21T12:28:28.957 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.957 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.957 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:28:28.957 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:12.374870+0000", 2024-09-21T12:28:28.957 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4040000000000003e-06 2024-09-21T12:28:28.957 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.957 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.957 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:28:28.958 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:12.374883+0000", 2024-09-21T12:28:28.958 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2486999999999999e-05 2024-09-21T12:28:28.958 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.958 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.958 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:28:28.958 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:12.374890+0000", 2024-09-21T12:28:28.958 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1049999999999997e-06 2024-09-21T12:28:28.958 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.958 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.958 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:28:28.958 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:12.374951+0000", 2024-09-21T12:28:28.958 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0841999999999998e-05 2024-09-21T12:28:28.958 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.959 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.959 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:28:28.959 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:12.375093+0000", 2024-09-21T12:28:28.959 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014165599999999999 2024-09-21T12:28:28.959 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.959 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.959 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:28:28.959 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:12.375225+0000", 2024-09-21T12:28:28.959 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013289500000000001 2024-09-21T12:28:28.959 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.959 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.959 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:28:28.959 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:12.377645+0000", 2024-09-21T12:28:28.959 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0024194889999999999 2024-09-21T12:28:28.960 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.960 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.960 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.960 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:12.378041+0000", 2024-09-21T12:28:28.960 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00039582700000000002 2024-09-21T12:28:28.960 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.960 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.960 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:28.960 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:12.536116+0000", 2024-09-21T12:28:28.960 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.158075304 2024-09-21T12:28:28.960 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.960 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.960 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:28:28.961 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:12.536183+0000", 2024-09-21T12:28:28.961 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6668999999999995e-05 2024-09-21T12:28:28.961 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:28:28.961 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:28:28.961 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:28:28.961 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:23:12.536603+0000", 2024-09-21T12:28:28.961 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00042054499999999998 2024-09-21T12:28:28.961 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.961 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:28:28.961 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.961 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:28:28.961 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:28:28.961 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:28:28.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:28.962 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:28.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:29.063 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:29.101 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:29.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:29.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:29.203 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:29.204 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:28 smithi114 ceph-mon[40867]: pgmap v402: 178 pgs: 178 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:28:29.204 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:28 smithi114 ceph-mon[40867]: 1.18 scrub starts 2024-09-21T12:28:29.204 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:28 smithi114 ceph-mon[40867]: 1.18 scrub ok 2024-09-21T12:28:29.205 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:28 smithi114 ceph-mon[40867]: 6.7 scrub starts 2024-09-21T12:28:29.205 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:28 smithi114 ceph-mon[40867]: 6.7 scrub ok 2024-09-21T12:28:29.205 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:28 smithi114 ceph-mon[40867]: 5.1e scrub starts 2024-09-21T12:28:29.205 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:28 smithi114 ceph-mon[40867]: 5.1e scrub ok 2024-09-21T12:28:29.205 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:28 smithi114 ceph-mon[30324]: pgmap v402: 178 pgs: 178 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:28:29.205 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:28 smithi114 ceph-mon[30324]: 1.18 scrub starts 2024-09-21T12:28:29.205 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:28 smithi114 ceph-mon[30324]: 1.18 scrub ok 2024-09-21T12:28:29.205 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:28 smithi114 ceph-mon[30324]: 6.7 scrub starts 2024-09-21T12:28:29.205 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:28 smithi114 ceph-mon[30324]: 6.7 scrub ok 2024-09-21T12:28:29.205 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:28 smithi114 ceph-mon[30324]: 5.1e scrub starts 2024-09-21T12:28:29.205 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:28 smithi114 ceph-mon[30324]: 5.1e scrub ok 2024-09-21T12:28:29.232 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:29.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:29.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:29.334 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:29.364 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:29.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:29.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:29.466 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:29.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:28 smithi183 ceph-mon[33277]: pgmap v402: 178 pgs: 178 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:28:29.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:28 smithi183 ceph-mon[33277]: 1.18 scrub starts 2024-09-21T12:28:29.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:28 smithi183 ceph-mon[33277]: 1.18 scrub ok 2024-09-21T12:28:29.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:28 smithi183 ceph-mon[33277]: 6.7 scrub starts 2024-09-21T12:28:29.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:28 smithi183 ceph-mon[33277]: 6.7 scrub ok 2024-09-21T12:28:29.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:28 smithi183 ceph-mon[33277]: 5.1e scrub starts 2024-09-21T12:28:29.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:28 smithi183 ceph-mon[33277]: 5.1e scrub ok 2024-09-21T12:28:29.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:29.504 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:29.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:29.605 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:29.641 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:29.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:29.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:29.707 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:28:29.707 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:28:29.743 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:29.744 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:28:29.744 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:28:29.779 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:29.780 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:28:29.780 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:28:29.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:29.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:29.823 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:28:29.823 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:28:29.824 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 2 16 6413 6397 199.892 193.375 0.00437675 0.00499083 2024-09-21T12:28:29.858 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:28:29.858 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:28:29.881 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:29.895 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:28:29.895 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:28:29.908 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:29.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:29.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:29.941 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:28:29.942 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:28:29.976 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:28:29.976 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:28:30.009 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:30.039 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:30.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:30.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:30.040 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:30 smithi114 ceph-mon[30324]: 1.a scrub starts 2024-09-21T12:28:30.041 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:30 smithi114 ceph-mon[30324]: 1.a scrub ok 2024-09-21T12:28:30.041 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:30 smithi114 ceph-mon[30324]: 1.13 scrub starts 2024-09-21T12:28:30.041 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:30 smithi114 ceph-mon[30324]: 1.13 scrub ok 2024-09-21T12:28:30.041 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:30 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2769773142' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:28:30.041 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:30 smithi114 ceph-mon[40867]: 1.a scrub starts 2024-09-21T12:28:30.041 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:30 smithi114 ceph-mon[40867]: 1.a scrub ok 2024-09-21T12:28:30.041 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:30 smithi114 ceph-mon[40867]: 1.13 scrub starts 2024-09-21T12:28:30.041 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:30 smithi114 ceph-mon[40867]: 1.13 scrub ok 2024-09-21T12:28:30.041 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:30 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2769773142' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:28:30.058 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T12:28:30.141 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:30.170 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:30.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:30.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:30.272 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:30.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:30.313 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:30.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:30.414 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:30.456 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:30.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:30.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:30.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:30 smithi183 ceph-mon[33277]: 1.a scrub starts 2024-09-21T12:28:30.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:30 smithi183 ceph-mon[33277]: 1.a scrub ok 2024-09-21T12:28:30.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:30 smithi183 ceph-mon[33277]: 1.13 scrub starts 2024-09-21T12:28:30.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:30 smithi183 ceph-mon[33277]: 1.13 scrub ok 2024-09-21T12:28:30.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:30 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2769773142' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:28:30.558 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:30.597 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:30.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:30.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:30.700 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:30.729 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:30.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:30.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:30.824 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 3 16 9227 9211 191.881 175.875 0.00337088 0.00520337 2024-09-21T12:28:30.831 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:30.868 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:30.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:30.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:30.969 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:30.999 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:30.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:31.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:31.008 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T12:28:31.101 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:31.131 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:31.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:31.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:31.233 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:31.263 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:31.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:31.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:31.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:31 smithi114 ceph-mon[40867]: pgmap v403: 178 pgs: 178 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 889 B/s rd, 0 op/s 2024-09-21T12:28:31.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:31 smithi114 ceph-mon[40867]: 3.e scrub starts 2024-09-21T12:28:31.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:31 smithi114 ceph-mon[40867]: 3.e scrub ok 2024-09-21T12:28:31.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:31 smithi114 ceph-mon[40867]: 4.1d scrub starts 2024-09-21T12:28:31.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:31 smithi114 ceph-mon[40867]: 4.1d scrub ok 2024-09-21T12:28:31.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:31 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3930229544' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:28:31.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:31 smithi114 ceph-mon[30324]: pgmap v403: 178 pgs: 178 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 889 B/s rd, 0 op/s 2024-09-21T12:28:31.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:31 smithi114 ceph-mon[30324]: 3.e scrub starts 2024-09-21T12:28:31.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:31 smithi114 ceph-mon[30324]: 3.e scrub ok 2024-09-21T12:28:31.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:31 smithi114 ceph-mon[30324]: 4.1d scrub starts 2024-09-21T12:28:31.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:31 smithi114 ceph-mon[30324]: 4.1d scrub ok 2024-09-21T12:28:31.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:31 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3930229544' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:28:31.365 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:31.400 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:31.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:31.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:31.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:31 smithi183 ceph-mon[33277]: pgmap v403: 178 pgs: 178 active+clean; 711 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 889 B/s rd, 0 op/s 2024-09-21T12:28:31.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:31 smithi183 ceph-mon[33277]: 3.e scrub starts 2024-09-21T12:28:31.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:31 smithi183 ceph-mon[33277]: 3.e scrub ok 2024-09-21T12:28:31.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:31 smithi183 ceph-mon[33277]: 4.1d scrub starts 2024-09-21T12:28:31.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:31 smithi183 ceph-mon[33277]: 4.1d scrub ok 2024-09-21T12:28:31.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:31 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3930229544' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:28:31.502 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:31.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:31.537 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:31.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:31.638 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:31.673 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:31.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:31.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:31.775 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:31.804 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:31.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:31.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:31.824 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 4 16 12169 12153 189.875 183.875 0.00660576 0.00526016 2024-09-21T12:28:31.906 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:31.936 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:31.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:31.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:32.038 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:32.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:32.075 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:32.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:32.176 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:32.207 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:32.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:32.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:32.292 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:32 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3930229544' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T12:28:32.292 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:32 smithi114 ceph-mon[30324]: osdmap e244: 8 total, 8 up, 7 in 2024-09-21T12:28:32.293 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:32 smithi114 ceph-mon[30324]: 1.17 scrub starts 2024-09-21T12:28:32.293 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:32 smithi114 ceph-mon[30324]: 1.17 scrub ok 2024-09-21T12:28:32.293 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:32 smithi114 ceph-mon[30324]: 6.d scrub starts 2024-09-21T12:28:32.293 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:32 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:28:32.293 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:32 smithi114 ceph-mon[30324]: 6.d scrub ok 2024-09-21T12:28:32.293 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:32 smithi114 ceph-mon[30324]: 3.3 scrub starts 2024-09-21T12:28:32.293 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:32 smithi114 ceph-mon[30324]: 3.3 scrub ok 2024-09-21T12:28:32.293 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:32 smithi114 ceph-mon[30324]: osdmap e245: 8 total, 8 up, 7 in 2024-09-21T12:28:32.293 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:32 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3930229544' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T12:28:32.293 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:32 smithi114 ceph-mon[40867]: osdmap e244: 8 total, 8 up, 7 in 2024-09-21T12:28:32.293 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:32 smithi114 ceph-mon[40867]: 1.17 scrub starts 2024-09-21T12:28:32.293 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:32 smithi114 ceph-mon[40867]: 1.17 scrub ok 2024-09-21T12:28:32.293 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:32 smithi114 ceph-mon[40867]: 6.d scrub starts 2024-09-21T12:28:32.294 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:32 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:28:32.294 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:32 smithi114 ceph-mon[40867]: 6.d scrub ok 2024-09-21T12:28:32.294 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:32 smithi114 ceph-mon[40867]: 3.3 scrub starts 2024-09-21T12:28:32.294 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:32 smithi114 ceph-mon[40867]: 3.3 scrub ok 2024-09-21T12:28:32.294 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:32 smithi114 ceph-mon[40867]: osdmap e245: 8 total, 8 up, 7 in 2024-09-21T12:28:32.295 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T12:28:32.309 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:32.383 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:32.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:32.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:32.485 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:32 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3930229544' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T12:28:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:32 smithi183 ceph-mon[33277]: osdmap e244: 8 total, 8 up, 7 in 2024-09-21T12:28:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:32 smithi183 ceph-mon[33277]: 1.17 scrub starts 2024-09-21T12:28:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:32 smithi183 ceph-mon[33277]: 1.17 scrub ok 2024-09-21T12:28:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:32 smithi183 ceph-mon[33277]: 6.d scrub starts 2024-09-21T12:28:32.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:32 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:28:32.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:32 smithi183 ceph-mon[33277]: 6.d scrub ok 2024-09-21T12:28:32.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:32 smithi183 ceph-mon[33277]: 3.3 scrub starts 2024-09-21T12:28:32.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:32 smithi183 ceph-mon[33277]: 3.3 scrub ok 2024-09-21T12:28:32.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:32 smithi183 ceph-mon[33277]: osdmap e245: 8 total, 8 up, 7 in 2024-09-21T12:28:32.514 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:32.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:32.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:32.616 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:32.645 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:32.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:32.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:32.747 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:32.788 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T12:28:32.804 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:32.825 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 5 16 14787 14771 184.61 163.625 0.00667565 0.00540614 2024-09-21T12:28:32.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:32.827 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:32.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:32.859 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:28:32 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:28:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:28:32.882 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:32.928 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:32.959 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:32.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:32.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:32.978 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:28:33.060 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:33.087 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:33.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:33.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:33.189 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:33.218 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:33.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:33.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:33.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:33 smithi114 ceph-mon[30324]: pgmap v405: 178 pgs: 178 active+clean; 273 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 1.7 KiB/s rd, 45 MiB/s wr, 727 op/s 2024-09-21T12:28:33.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:33 smithi114 ceph-mon[30324]: 3.13 scrub starts 2024-09-21T12:28:33.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:33 smithi114 ceph-mon[30324]: 3.13 scrub ok 2024-09-21T12:28:33.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:33 smithi114 ceph-mon[30324]: 3.6 scrub starts 2024-09-21T12:28:33.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:33 smithi114 ceph-mon[30324]: 3.6 scrub ok 2024-09-21T12:28:33.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:33 smithi114 ceph-mon[40867]: pgmap v405: 178 pgs: 178 active+clean; 273 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 1.7 KiB/s rd, 45 MiB/s wr, 727 op/s 2024-09-21T12:28:33.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:33 smithi114 ceph-mon[40867]: 3.13 scrub starts 2024-09-21T12:28:33.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:33 smithi114 ceph-mon[40867]: 3.13 scrub ok 2024-09-21T12:28:33.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:33 smithi114 ceph-mon[40867]: 3.6 scrub starts 2024-09-21T12:28:33.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:33 smithi114 ceph-mon[40867]: 3.6 scrub ok 2024-09-21T12:28:33.320 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:33.352 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:33.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:33.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:33.453 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:33.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:33 smithi183 ceph-mon[33277]: pgmap v405: 178 pgs: 178 active+clean; 273 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 1.7 KiB/s rd, 45 MiB/s wr, 727 op/s 2024-09-21T12:28:33.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:33 smithi183 ceph-mon[33277]: 3.13 scrub starts 2024-09-21T12:28:33.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:33 smithi183 ceph-mon[33277]: 3.13 scrub ok 2024-09-21T12:28:33.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:33 smithi183 ceph-mon[33277]: 3.6 scrub starts 2024-09-21T12:28:33.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:33 smithi183 ceph-mon[33277]: 3.6 scrub ok 2024-09-21T12:28:33.496 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:33.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:33.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:33.598 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:33.630 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:33.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:33.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:33.732 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:33.762 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:33.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:33.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:33.825 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 6 16 17260 17244 179.6 154.562 0.00412553 0.00556166 2024-09-21T12:28:33.864 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:33.894 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:33.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:33.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:33.996 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:34.027 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:34.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:34.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:34.128 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:34.166 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:34.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:34.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:34.261 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:34 smithi114 ceph-mon[30324]: 3.1c scrub starts 2024-09-21T12:28:34.261 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:34 smithi114 ceph-mon[30324]: 3.1c scrub ok 2024-09-21T12:28:34.262 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:34 smithi114 ceph-mon[30324]: 5.1 scrub starts 2024-09-21T12:28:34.262 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:34 smithi114 ceph-mon[40867]: 3.1c scrub starts 2024-09-21T12:28:34.262 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:34 smithi114 ceph-mon[40867]: 3.1c scrub ok 2024-09-21T12:28:34.262 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:34 smithi114 ceph-mon[40867]: 5.1 scrub starts 2024-09-21T12:28:34.268 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:34.297 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:34.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:34.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:34.325 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:28:34.325 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:28:34.399 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:34.436 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:34.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:34.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:34.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:34 smithi183 ceph-mon[33277]: 3.1c scrub starts 2024-09-21T12:28:34.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:34 smithi183 ceph-mon[33277]: 3.1c scrub ok 2024-09-21T12:28:34.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:34 smithi183 ceph-mon[33277]: 5.1 scrub starts 2024-09-21T12:28:34.539 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:34.581 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:34.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:34.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:34.682 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:34.712 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:34.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:34.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:34.814 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:34.825 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 7 16 19831 19815 176.897 160.688 0.00733303 0.00564729 2024-09-21T12:28:34.844 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:34.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:34.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:34.946 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:34.982 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:34.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:34.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:35.084 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:35.087 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T12:28:35.111 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:35.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:35.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:35.213 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:35.260 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:35.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:35.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:35.314 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:35 smithi114 ceph-mon[30324]: 5.1 scrub ok 2024-09-21T12:28:35.314 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:35 smithi114 ceph-mon[30324]: pgmap v407: 178 pgs: 178 active+clean; 793 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 752 B/s rd, 117 MiB/s wr, 1.86k op/s 2024-09-21T12:28:35.314 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:35 smithi114 ceph-mon[30324]: 2.0 scrub starts 2024-09-21T12:28:35.314 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:35 smithi114 ceph-mon[30324]: 2.0 scrub ok 2024-09-21T12:28:35.314 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:35 smithi114 ceph-mon[40867]: 5.1 scrub ok 2024-09-21T12:28:35.314 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:35 smithi114 ceph-mon[40867]: pgmap v407: 178 pgs: 178 active+clean; 793 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 752 B/s rd, 117 MiB/s wr, 1.86k op/s 2024-09-21T12:28:35.314 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:35 smithi114 ceph-mon[40867]: 2.0 scrub starts 2024-09-21T12:28:35.314 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:35 smithi114 ceph-mon[40867]: 2.0 scrub ok 2024-09-21T12:28:35.362 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:35.387 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:35.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:35.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:35.389 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:35 smithi183 ceph-mon[33277]: 5.1 scrub ok 2024-09-21T12:28:35.389 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:35 smithi183 ceph-mon[33277]: pgmap v407: 178 pgs: 178 active+clean; 793 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 752 B/s rd, 117 MiB/s wr, 1.86k op/s 2024-09-21T12:28:35.389 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:35 smithi183 ceph-mon[33277]: 2.0 scrub starts 2024-09-21T12:28:35.389 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:35 smithi183 ceph-mon[33277]: 2.0 scrub ok 2024-09-21T12:28:35.402 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:35.414 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:35.489 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:35.518 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:35.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:35.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:35.620 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:35.646 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:35.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:35.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:35.748 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:35.776 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:35.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:35.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:35.825 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 8 16 21987 21971 171.627 134.75 0.00416719 0.00581939 2024-09-21T12:28:35.879 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:35.908 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:35.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:35.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:36.010 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:36.041 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:36.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:36.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:36.072 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-21T12:28:36.096 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:36 smithi114 ceph-mon[30324]: 6.e scrub starts 2024-09-21T12:28:36.096 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:36 smithi114 ceph-mon[30324]: 6.e scrub ok 2024-09-21T12:28:36.097 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:36 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3668996036' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:28:36.097 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:36 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:28:36.097 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:36 smithi114 ceph-mon[40867]: 6.e scrub starts 2024-09-21T12:28:36.097 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:36 smithi114 ceph-mon[40867]: 6.e scrub ok 2024-09-21T12:28:36.097 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:36 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3668996036' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:28:36.097 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:36 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:28:36.142 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:36.172 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:36.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:36.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:36.274 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:36.297 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:36.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:36.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:36.398 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:36.433 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:36.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:36.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:36.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:36 smithi183 ceph-mon[33277]: 6.e scrub starts 2024-09-21T12:28:36.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:36 smithi183 ceph-mon[33277]: 6.e scrub ok 2024-09-21T12:28:36.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:36 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3668996036' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:28:36.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:36 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:28:36.535 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:36.562 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:36.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:36.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:36.664 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:36.700 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:36.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:36.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:36.802 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:36.825 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 9 16 24158 24142 167.628 135.688 0.00614593 0.00595998 2024-09-21T12:28:36.832 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:36.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:36.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:36.927 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T12:28:36.934 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:36.960 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:36.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:36.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:37.062 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:37.098 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:37.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:37.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:37.195 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:37 smithi114 ceph-mon[40867]: pgmap v408: 178 pgs: 178 active+clean; 793 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-09-21T12:28:37.195 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:37 smithi114 ceph-mon[40867]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:28:37.195 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:37 smithi114 ceph-mon[40867]: Cluster is now healthy 2024-09-21T12:28:37.195 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:37 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:28:37.195 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:37 smithi114 ceph-mon[40867]: osdmap e246: 8 total, 8 up, 7 in 2024-09-21T12:28:37.195 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:37 smithi114 ceph-mon[40867]: 5.15 scrub starts 2024-09-21T12:28:37.195 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:37 smithi114 ceph-mon[40867]: 5.15 scrub ok 2024-09-21T12:28:37.195 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:37 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/930256372' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:28:37.195 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:37 smithi114 ceph-mon[40867]: 4.12 scrub starts 2024-09-21T12:28:37.195 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:37 smithi114 ceph-mon[40867]: 4.12 scrub ok 2024-09-21T12:28:37.195 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:37 smithi114 ceph-mon[40867]: 5.4 scrub starts 2024-09-21T12:28:37.195 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:37 smithi114 ceph-mon[40867]: 5.4 scrub ok 2024-09-21T12:28:37.196 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:37 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/930256372' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T12:28:37.196 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:37 smithi114 ceph-mon[40867]: osdmap e247: 8 total, 8 up, 7 in 2024-09-21T12:28:37.196 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:37 smithi114 ceph-mon[30324]: pgmap v408: 178 pgs: 178 active+clean; 793 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-09-21T12:28:37.196 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:37 smithi114 ceph-mon[30324]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:28:37.196 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:37 smithi114 ceph-mon[30324]: Cluster is now healthy 2024-09-21T12:28:37.196 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:37 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:28:37.196 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:37 smithi114 ceph-mon[30324]: osdmap e246: 8 total, 8 up, 7 in 2024-09-21T12:28:37.196 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:37 smithi114 ceph-mon[30324]: 5.15 scrub starts 2024-09-21T12:28:37.196 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:37 smithi114 ceph-mon[30324]: 5.15 scrub ok 2024-09-21T12:28:37.196 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:37 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/930256372' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:28:37.196 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:37 smithi114 ceph-mon[30324]: 4.12 scrub starts 2024-09-21T12:28:37.196 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:37 smithi114 ceph-mon[30324]: 4.12 scrub ok 2024-09-21T12:28:37.197 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:37 smithi114 ceph-mon[30324]: 5.4 scrub starts 2024-09-21T12:28:37.197 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:37 smithi114 ceph-mon[30324]: 5.4 scrub ok 2024-09-21T12:28:37.197 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:37 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/930256372' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T12:28:37.197 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:37 smithi114 ceph-mon[30324]: osdmap e247: 8 total, 8 up, 7 in 2024-09-21T12:28:37.200 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:37.231 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:37.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:37.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:37.333 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:37.361 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:37.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:37.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:37.463 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:37.488 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:28:37 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:28:37.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:37 smithi183 ceph-mon[33277]: pgmap v408: 178 pgs: 178 active+clean; 793 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-09-21T12:28:37.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:37 smithi183 ceph-mon[33277]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:28:37.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:37 smithi183 ceph-mon[33277]: Cluster is now healthy 2024-09-21T12:28:37.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:37 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:28:37.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:37 smithi183 ceph-mon[33277]: osdmap e246: 8 total, 8 up, 7 in 2024-09-21T12:28:37.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:37 smithi183 ceph-mon[33277]: 5.15 scrub starts 2024-09-21T12:28:37.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:37 smithi183 ceph-mon[33277]: 5.15 scrub ok 2024-09-21T12:28:37.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:37 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/930256372' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:28:37.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:37 smithi183 ceph-mon[33277]: 4.12 scrub starts 2024-09-21T12:28:37.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:37 smithi183 ceph-mon[33277]: 4.12 scrub ok 2024-09-21T12:28:37.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:37 smithi183 ceph-mon[33277]: 5.4 scrub starts 2024-09-21T12:28:37.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:37 smithi183 ceph-mon[33277]: 5.4 scrub ok 2024-09-21T12:28:37.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:37 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/930256372' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T12:28:37.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:37 smithi183 ceph-mon[33277]: osdmap e247: 8 total, 8 up, 7 in 2024-09-21T12:28:37.493 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:37.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:37.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:37.595 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:37.627 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:37.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:37.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:37.729 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:37.752 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:37.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:37.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:37.825 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 10 16 26151 26135 163.32 124.562 0.00544752 0.0060647 2024-09-21T12:28:37.854 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:37.883 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:37.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:37.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:37.985 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:38.013 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:38.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:38.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:38.114 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:38.155 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:38.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:38.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:38.257 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:38.295 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:38.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:38.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:38.340 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:28:38.340 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:28:38.341 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:28:38.341 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:28:38.341 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:28:38.341 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:28:38.341 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:28:38.341 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:28:38.341 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:28:38.341 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:28:38.341 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:28:38.342 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:28:38.342 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:28:38.342 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:28:38.342 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:28:38.342 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:28:38.342 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:38 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:28:38.342 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:38 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2746438115' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:28:38.342 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:38 smithi114 ceph-mon[40867]: 6.3 scrub starts 2024-09-21T12:28:38.342 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:38 smithi114 ceph-mon[40867]: 6.3 scrub ok 2024-09-21T12:28:38.343 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:38 smithi114 ceph-mon[40867]: 2.2 scrub starts 2024-09-21T12:28:38.343 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:38 smithi114 ceph-mon[40867]: 2.2 scrub ok 2024-09-21T12:28:38.343 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:38 smithi114 ceph-mon[40867]: 6.0 scrub starts 2024-09-21T12:28:38.343 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:38 smithi114 ceph-mon[40867]: 6.0 scrub ok 2024-09-21T12:28:38.343 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:38 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:28:38.343 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:38 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2746438115' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:28:38.343 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:38 smithi114 ceph-mon[30324]: 6.3 scrub starts 2024-09-21T12:28:38.343 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:38 smithi114 ceph-mon[30324]: 6.3 scrub ok 2024-09-21T12:28:38.343 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:38 smithi114 ceph-mon[30324]: 2.2 scrub starts 2024-09-21T12:28:38.343 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:38 smithi114 ceph-mon[30324]: 2.2 scrub ok 2024-09-21T12:28:38.343 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:38 smithi114 ceph-mon[30324]: 6.0 scrub starts 2024-09-21T12:28:38.343 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:38 smithi114 ceph-mon[30324]: 6.0 scrub ok 2024-09-21T12:28:38.397 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:38.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:38.429 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:38.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:38.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:38 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:28:38.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:38 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2746438115' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:28:38.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:38 smithi183 ceph-mon[33277]: 6.3 scrub starts 2024-09-21T12:28:38.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:38 smithi183 ceph-mon[33277]: 6.3 scrub ok 2024-09-21T12:28:38.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:38 smithi183 ceph-mon[33277]: 2.2 scrub starts 2024-09-21T12:28:38.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:38 smithi183 ceph-mon[33277]: 2.2 scrub ok 2024-09-21T12:28:38.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:38 smithi183 ceph-mon[33277]: 6.0 scrub starts 2024-09-21T12:28:38.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:38 smithi183 ceph-mon[33277]: 6.0 scrub ok 2024-09-21T12:28:38.531 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:38.566 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:38.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:38.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:38.667 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:38.688 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:38.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:38.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:38.789 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:38.824 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:38.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:38.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:38.826 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 11 16 28269 28253 160.506 132.375 0.0131208 0.00622396 2024-09-21T12:28:38.926 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:38.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:38.948 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:38.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:39.049 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:39.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:39.087 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:39.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:39.188 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:39.217 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:39.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:39.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:39.319 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:39.345 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:39.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:39.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:39.447 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:39.486 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:39.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:39.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:39.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:39 smithi183 ceph-mon[33277]: pgmap v411: 178 pgs: 178 active+clean; 1.5 GiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 210 MiB/s wr, 3.37k op/s 2024-09-21T12:28:39.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:39 smithi183 ceph-mon[33277]: 6.19 scrub starts 2024-09-21T12:28:39.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:39 smithi183 ceph-mon[33277]: 6.19 scrub ok 2024-09-21T12:28:39.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:39 smithi114 ceph-mon[40867]: pgmap v411: 178 pgs: 178 active+clean; 1.5 GiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 210 MiB/s wr, 3.37k op/s 2024-09-21T12:28:39.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:39 smithi114 ceph-mon[40867]: 6.19 scrub starts 2024-09-21T12:28:39.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:39 smithi114 ceph-mon[40867]: 6.19 scrub ok 2024-09-21T12:28:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:39 smithi114 ceph-mon[30324]: pgmap v411: 178 pgs: 178 active+clean; 1.5 GiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 210 MiB/s wr, 3.37k op/s 2024-09-21T12:28:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:39 smithi114 ceph-mon[30324]: 6.19 scrub starts 2024-09-21T12:28:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:39 smithi114 ceph-mon[30324]: 6.19 scrub ok 2024-09-21T12:28:39.588 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:39.615 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:39.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:39.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:39.717 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:39.747 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:39.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:39.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:39.826 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 12 16 30159 30143 156.974 118.125 0.001919 0.00632869 2024-09-21T12:28:39.849 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:39.885 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:39.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:39.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:39.987 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:40.019 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:40.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:40.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:40.121 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:40.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:40.157 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:40.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:40.258 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:40.287 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:40.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:40.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:40.389 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:40.425 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:40.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:40.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:40.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:40 smithi183 ceph-mon[33277]: 5.18 scrub starts 2024-09-21T12:28:40.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:40 smithi183 ceph-mon[33277]: 5.18 scrub ok 2024-09-21T12:28:40.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:40 smithi183 ceph-mon[33277]: 5.d scrub starts 2024-09-21T12:28:40.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:40 smithi183 ceph-mon[33277]: 5.d scrub ok 2024-09-21T12:28:40.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:40 smithi183 ceph-mon[33277]: 3.0 scrub starts 2024-09-21T12:28:40.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:40 smithi183 ceph-mon[33277]: 3.0 scrub ok 2024-09-21T12:28:40.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:40 smithi114 ceph-mon[40867]: 5.18 scrub starts 2024-09-21T12:28:40.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:40 smithi114 ceph-mon[40867]: 5.18 scrub ok 2024-09-21T12:28:40.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:40 smithi114 ceph-mon[40867]: 5.d scrub starts 2024-09-21T12:28:40.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:40 smithi114 ceph-mon[40867]: 5.d scrub ok 2024-09-21T12:28:40.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:40 smithi114 ceph-mon[40867]: 3.0 scrub starts 2024-09-21T12:28:40.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:40 smithi114 ceph-mon[40867]: 3.0 scrub ok 2024-09-21T12:28:40.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:40 smithi114 ceph-mon[30324]: 5.18 scrub starts 2024-09-21T12:28:40.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:40 smithi114 ceph-mon[30324]: 5.18 scrub ok 2024-09-21T12:28:40.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:40 smithi114 ceph-mon[30324]: 5.d scrub starts 2024-09-21T12:28:40.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:40 smithi114 ceph-mon[30324]: 5.d scrub ok 2024-09-21T12:28:40.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:40 smithi114 ceph-mon[30324]: 3.0 scrub starts 2024-09-21T12:28:40.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:40 smithi114 ceph-mon[30324]: 3.0 scrub ok 2024-09-21T12:28:40.527 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:40.562 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:40.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:40.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:40.664 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:40.694 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:40.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:40.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:40.796 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:40.823 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:40.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:40.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:40.826 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 13 16 32131 32115 154.379 123.25 0.00545878 0.00647277 2024-09-21T12:28:40.924 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:40.961 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:40.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:40.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:41.063 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:41.083 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:41.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:41.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:41.185 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:41.222 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:41.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:41.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:41.324 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:41.357 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:41.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:41.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:41.459 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:41.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:41 smithi183 ceph-mon[33277]: pgmap v412: 178 pgs: 178 active+clean; 1.5 GiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 655 B/s rd, 162 MiB/s wr, 2.59k op/s 2024-09-21T12:28:41.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:41 smithi183 ceph-mon[33277]: 6.c scrub starts 2024-09-21T12:28:41.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:41 smithi183 ceph-mon[33277]: 6.c scrub ok 2024-09-21T12:28:41.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:41 smithi183 ceph-mon[33277]: 4.13 scrub starts 2024-09-21T12:28:41.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:41 smithi183 ceph-mon[33277]: 4.13 scrub ok 2024-09-21T12:28:41.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:41 smithi183 ceph-mon[33277]: 4.9 scrub starts 2024-09-21T12:28:41.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:41 smithi183 ceph-mon[33277]: 4.9 scrub ok 2024-09-21T12:28:41.493 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:41.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:41.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:41.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:41 smithi114 ceph-mon[40867]: pgmap v412: 178 pgs: 178 active+clean; 1.5 GiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 655 B/s rd, 162 MiB/s wr, 2.59k op/s 2024-09-21T12:28:41.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:41 smithi114 ceph-mon[40867]: 6.c scrub starts 2024-09-21T12:28:41.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:41 smithi114 ceph-mon[40867]: 6.c scrub ok 2024-09-21T12:28:41.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:41 smithi114 ceph-mon[40867]: 4.13 scrub starts 2024-09-21T12:28:41.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:41 smithi114 ceph-mon[40867]: 4.13 scrub ok 2024-09-21T12:28:41.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:41 smithi114 ceph-mon[40867]: 4.9 scrub starts 2024-09-21T12:28:41.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:41 smithi114 ceph-mon[40867]: 4.9 scrub ok 2024-09-21T12:28:41.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:41 smithi114 ceph-mon[30324]: pgmap v412: 178 pgs: 178 active+clean; 1.5 GiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 655 B/s rd, 162 MiB/s wr, 2.59k op/s 2024-09-21T12:28:41.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:41 smithi114 ceph-mon[30324]: 6.c scrub starts 2024-09-21T12:28:41.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:41 smithi114 ceph-mon[30324]: 6.c scrub ok 2024-09-21T12:28:41.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:41 smithi114 ceph-mon[30324]: 4.13 scrub starts 2024-09-21T12:28:41.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:41 smithi114 ceph-mon[30324]: 4.13 scrub ok 2024-09-21T12:28:41.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:41 smithi114 ceph-mon[30324]: 4.9 scrub starts 2024-09-21T12:28:41.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:41 smithi114 ceph-mon[30324]: 4.9 scrub ok 2024-09-21T12:28:41.596 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:41.629 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:41.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:41.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:41.732 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:41.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:41.764 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:41.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:41.826 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 14 16 33911 33895 151.298 111.25 0.00265197 0.0065931 2024-09-21T12:28:41.865 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:41.904 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:41.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:41.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:42.006 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:42.037 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:42.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:42.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:42.139 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:42.176 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:42.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:42.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:42.277 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:42.308 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:42.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:42.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:42.410 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:42.439 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:42.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:42.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:42.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:42 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:28:42.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:42 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:28:42.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:42 smithi183 ceph-mon[33277]: osdmap e248: 8 total, 8 up, 7 in 2024-09-21T12:28:42.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:42 smithi183 ceph-mon[33277]: 6.12 scrub starts 2024-09-21T12:28:42.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:42 smithi183 ceph-mon[33277]: 6.12 scrub ok 2024-09-21T12:28:42.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:42 smithi183 ceph-mon[33277]: 6.10 scrub starts 2024-09-21T12:28:42.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:42 smithi183 ceph-mon[33277]: 6.10 scrub ok 2024-09-21T12:28:42.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:42 smithi183 ceph-mon[33277]: osdmap e249: 8 total, 8 up, 7 in 2024-09-21T12:28:42.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:42 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:28:42.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:42 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:28:42.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:42 smithi114 ceph-mon[30324]: osdmap e248: 8 total, 8 up, 7 in 2024-09-21T12:28:42.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:42 smithi114 ceph-mon[30324]: 6.12 scrub starts 2024-09-21T12:28:42.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:42 smithi114 ceph-mon[30324]: 6.12 scrub ok 2024-09-21T12:28:42.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:42 smithi114 ceph-mon[30324]: 6.10 scrub starts 2024-09-21T12:28:42.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:42 smithi114 ceph-mon[30324]: 6.10 scrub ok 2024-09-21T12:28:42.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:42 smithi114 ceph-mon[30324]: osdmap e249: 8 total, 8 up, 7 in 2024-09-21T12:28:42.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:42 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:28:42.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:42 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:28:42.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:42 smithi114 ceph-mon[40867]: osdmap e248: 8 total, 8 up, 7 in 2024-09-21T12:28:42.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:42 smithi114 ceph-mon[40867]: 6.12 scrub starts 2024-09-21T12:28:42.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:42 smithi114 ceph-mon[40867]: 6.12 scrub ok 2024-09-21T12:28:42.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:42 smithi114 ceph-mon[40867]: 6.10 scrub starts 2024-09-21T12:28:42.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:42 smithi114 ceph-mon[40867]: 6.10 scrub ok 2024-09-21T12:28:42.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:42 smithi114 ceph-mon[40867]: osdmap e249: 8 total, 8 up, 7 in 2024-09-21T12:28:42.541 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:42.580 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:42.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:42.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:42.649 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T12:28:42.659 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:42.682 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:42.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:42.726 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:42.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:42.826 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 15 15 35831 35816 149.215 120.062 0.0142393 0.00669674 2024-09-21T12:28:42.826 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:42.853 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:42.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:42.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:42.955 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:42.984 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:42.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:42.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:43.025 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:28:42 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:28:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:28:43.086 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:43.118 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:43.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:43.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:43.220 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:43.251 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:43.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:43.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:43.353 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:43.384 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:43.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:43.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:43.486 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:43.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:43 smithi183 ceph-mon[33277]: pgmap v414: 178 pgs: 178 active+clean; 1.7 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1.7 KiB/s rd, 163 MiB/s wr, 2.60k op/s 2024-09-21T12:28:43.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:43 smithi183 ceph-mon[33277]: 4.15 scrub starts 2024-09-21T12:28:43.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:43 smithi183 ceph-mon[33277]: 4.15 scrub ok 2024-09-21T12:28:43.521 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:43.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:43.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:43.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:43 smithi114 ceph-mon[40867]: pgmap v414: 178 pgs: 178 active+clean; 1.7 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1.7 KiB/s rd, 163 MiB/s wr, 2.60k op/s 2024-09-21T12:28:43.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:43 smithi114 ceph-mon[40867]: 4.15 scrub starts 2024-09-21T12:28:43.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:43 smithi114 ceph-mon[40867]: 4.15 scrub ok 2024-09-21T12:28:43.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:43 smithi114 ceph-mon[30324]: pgmap v414: 178 pgs: 178 active+clean; 1.7 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1.7 KiB/s rd, 163 MiB/s wr, 2.60k op/s 2024-09-21T12:28:43.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:43 smithi114 ceph-mon[30324]: 4.15 scrub starts 2024-09-21T12:28:43.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:43 smithi114 ceph-mon[30324]: 4.15 scrub ok 2024-09-21T12:28:43.623 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:43.652 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:43.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:43.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:43.721 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T12:28:43.753 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:43.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:43.815 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:43.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:43.826 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 16 16 37520 37504 146.481 105.5 0.00589169 0.00681976 2024-09-21T12:28:43.916 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:43.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:43.964 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:43.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:44.065 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:44.095 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:44.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:44.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:44.196 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:44.259 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:44.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:44.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:44.361 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:44.381 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:44.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:44.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:44.483 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:44.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:44 smithi183 ceph-mon[33277]: 2.1 scrub starts 2024-09-21T12:28:44.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:44 smithi183 ceph-mon[33277]: 2.1 scrub ok 2024-09-21T12:28:44.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:44 smithi183 ceph-mon[33277]: 3.11 scrub starts 2024-09-21T12:28:44.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:44 smithi183 ceph-mon[33277]: 3.11 scrub ok 2024-09-21T12:28:44.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:44 smithi183 ceph-mon[33277]: 6.9 scrub starts 2024-09-21T12:28:44.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:44 smithi183 ceph-mon[33277]: 6.9 scrub ok 2024-09-21T12:28:44.520 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:44.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:44.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:44.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:44 smithi114 ceph-mon[30324]: 2.1 scrub starts 2024-09-21T12:28:44.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:44 smithi114 ceph-mon[30324]: 2.1 scrub ok 2024-09-21T12:28:44.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:44 smithi114 ceph-mon[30324]: 3.11 scrub starts 2024-09-21T12:28:44.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:44 smithi114 ceph-mon[30324]: 3.11 scrub ok 2024-09-21T12:28:44.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:44 smithi114 ceph-mon[30324]: 6.9 scrub starts 2024-09-21T12:28:44.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:44 smithi114 ceph-mon[30324]: 6.9 scrub ok 2024-09-21T12:28:44.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:44 smithi114 ceph-mon[40867]: 2.1 scrub starts 2024-09-21T12:28:44.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:44 smithi114 ceph-mon[40867]: 2.1 scrub ok 2024-09-21T12:28:44.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:44 smithi114 ceph-mon[40867]: 3.11 scrub starts 2024-09-21T12:28:44.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:44 smithi114 ceph-mon[40867]: 3.11 scrub ok 2024-09-21T12:28:44.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:44 smithi114 ceph-mon[40867]: 6.9 scrub starts 2024-09-21T12:28:44.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:44 smithi114 ceph-mon[40867]: 6.9 scrub ok 2024-09-21T12:28:44.622 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:44.652 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:44.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:44.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:44.753 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:44.787 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:44.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:44.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:44.826 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 17 16 39084 39068 143.614 97.75 0.00373879 0.00695731 2024-09-21T12:28:44.889 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:44.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:44.926 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:44.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:45.027 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:45.060 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:45.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:45.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:45.161 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:45.190 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:28:45.192 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:45.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:45.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:45.294 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:45.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:45.320 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:45.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:45.376 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:45 smithi114 ceph-mon[40867]: pgmap v416: 178 pgs: 178 active+clean; 2.1 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 752 B/s rd, 157 MiB/s wr, 2.52k op/s 2024-09-21T12:28:45.376 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:45 smithi114 ceph-mon[40867]: 5.14 scrub starts 2024-09-21T12:28:45.376 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:45 smithi114 ceph-mon[40867]: 5.14 scrub ok 2024-09-21T12:28:45.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:45 smithi114 ceph-mon[30324]: pgmap v416: 178 pgs: 178 active+clean; 2.1 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 752 B/s rd, 157 MiB/s wr, 2.52k op/s 2024-09-21T12:28:45.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:45 smithi114 ceph-mon[30324]: 5.14 scrub starts 2024-09-21T12:28:45.377 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:45 smithi114 ceph-mon[30324]: 5.14 scrub ok 2024-09-21T12:28:45.377 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:28:45.377 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:28:45.377 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:28:45.377 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:28:45.377 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:28:45.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:45.475 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:45.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:45.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:45.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:45 smithi183 ceph-mon[33277]: pgmap v416: 178 pgs: 178 active+clean; 2.1 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 752 B/s rd, 157 MiB/s wr, 2.52k op/s 2024-09-21T12:28:45.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:45 smithi183 ceph-mon[33277]: 5.14 scrub starts 2024-09-21T12:28:45.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:45 smithi183 ceph-mon[33277]: 5.14 scrub ok 2024-09-21T12:28:45.578 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:45.623 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:45.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:45.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:45.725 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:45.768 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:45.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:45.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:45.826 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 18 16 40843 40827 141.743 109.938 0.00469014 0.00704962 2024-09-21T12:28:45.871 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:45.905 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:45.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:45.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:46.006 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:46.037 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:46.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:46.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:46.138 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:46.168 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:46.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:46.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:46.223 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T12:28:46.260 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:46.270 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:46.292 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:46.306 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:46.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:46.359 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:46.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:46.399 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:46 smithi114 ceph-mon[30324]: 3.16 scrub starts 2024-09-21T12:28:46.399 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:46 smithi114 ceph-mon[30324]: 3.16 scrub ok 2024-09-21T12:28:46.399 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:46 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:28:46.399 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:46 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:28:46.399 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:46 smithi114 ceph-mon[30324]: 3.1a scrub starts 2024-09-21T12:28:46.399 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:46 smithi114 ceph-mon[30324]: 3.1a scrub ok 2024-09-21T12:28:46.400 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:46 smithi114 ceph-mon[30324]: 4.6 scrub starts 2024-09-21T12:28:46.400 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:46 smithi114 ceph-mon[30324]: 4.6 scrub ok 2024-09-21T12:28:46.400 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:46 smithi114 ceph-mon[40867]: 3.16 scrub starts 2024-09-21T12:28:46.400 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:46 smithi114 ceph-mon[40867]: 3.16 scrub ok 2024-09-21T12:28:46.400 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:46 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:28:46.400 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:46 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:28:46.400 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:46 smithi114 ceph-mon[40867]: 3.1a scrub starts 2024-09-21T12:28:46.400 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:46 smithi114 ceph-mon[40867]: 3.1a scrub ok 2024-09-21T12:28:46.400 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:46 smithi114 ceph-mon[40867]: 4.6 scrub starts 2024-09-21T12:28:46.400 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:46 smithi114 ceph-mon[40867]: 4.6 scrub ok 2024-09-21T12:28:46.462 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:46.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:46.487 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:46.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:46.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:46 smithi183 ceph-mon[33277]: 3.16 scrub starts 2024-09-21T12:28:46.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:46 smithi183 ceph-mon[33277]: 3.16 scrub ok 2024-09-21T12:28:46.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:46 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:28:46.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:46 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:28:46.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:46 smithi183 ceph-mon[33277]: 3.1a scrub starts 2024-09-21T12:28:46.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:46 smithi183 ceph-mon[33277]: 3.1a scrub ok 2024-09-21T12:28:46.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:46 smithi183 ceph-mon[33277]: 4.6 scrub starts 2024-09-21T12:28:46.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:46 smithi183 ceph-mon[33277]: 4.6 scrub ok 2024-09-21T12:28:46.588 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:46.618 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:46.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:46.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:46.720 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:46.751 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:46.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:46.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:46.826 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 19 16 42431 42415 139.506 99.25 0.00673491 0.00716326 2024-09-21T12:28:46.853 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:46.881 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:46.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:46.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:46.983 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:47.010 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:47.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:47.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:47.111 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:47.147 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:47.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:47.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:47.249 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:47.274 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:47.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:47.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:47.376 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:47.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:47.421 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:47.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:47.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:47 smithi114 ceph-mon[30324]: pgmap v417: 178 pgs: 178 active+clean; 2.1 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-09-21T12:28:47.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:47 smithi114 ceph-mon[30324]: 6.1f scrub starts 2024-09-21T12:28:47.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:47 smithi114 ceph-mon[30324]: 6.1f scrub ok 2024-09-21T12:28:47.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:47 smithi114 ceph-mon[30324]: osdmap e250: 8 total, 8 up, 7 in 2024-09-21T12:28:47.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:47 smithi114 ceph-mon[30324]: 1.1b scrub starts 2024-09-21T12:28:47.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:47 smithi114 ceph-mon[30324]: 1.1b scrub ok 2024-09-21T12:28:47.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:47 smithi114 ceph-mon[30324]: 4.d scrub starts 2024-09-21T12:28:47.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:47 smithi114 ceph-mon[30324]: 4.d scrub ok 2024-09-21T12:28:47.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:47 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:28:47.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:47 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:28:47.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:47 smithi114 ceph-mon[30324]: 6.1d scrub starts 2024-09-21T12:28:47.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:47 smithi114 ceph-mon[30324]: 6.1d scrub ok 2024-09-21T12:28:47.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:47 smithi114 ceph-mon[30324]: osdmap e251: 8 total, 8 up, 7 in 2024-09-21T12:28:47.438 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:47 smithi114 ceph-mon[40867]: pgmap v417: 178 pgs: 178 active+clean; 2.1 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-09-21T12:28:47.438 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:47 smithi114 ceph-mon[40867]: 6.1f scrub starts 2024-09-21T12:28:47.438 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:47 smithi114 ceph-mon[40867]: 6.1f scrub ok 2024-09-21T12:28:47.438 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:47 smithi114 ceph-mon[40867]: osdmap e250: 8 total, 8 up, 7 in 2024-09-21T12:28:47.438 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:47 smithi114 ceph-mon[40867]: 1.1b scrub starts 2024-09-21T12:28:47.438 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:47 smithi114 ceph-mon[40867]: 1.1b scrub ok 2024-09-21T12:28:47.438 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:47 smithi114 ceph-mon[40867]: 4.d scrub starts 2024-09-21T12:28:47.438 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:47 smithi114 ceph-mon[40867]: 4.d scrub ok 2024-09-21T12:28:47.438 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:47 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:28:47.438 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:47 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:28:47.438 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:47 smithi114 ceph-mon[40867]: 6.1d scrub starts 2024-09-21T12:28:47.438 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:47 smithi114 ceph-mon[40867]: 6.1d scrub ok 2024-09-21T12:28:47.439 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:47 smithi114 ceph-mon[40867]: osdmap e251: 8 total, 8 up, 7 in 2024-09-21T12:28:47.488 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:28:47 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:28:47.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:47 smithi183 ceph-mon[33277]: pgmap v417: 178 pgs: 178 active+clean; 2.1 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-09-21T12:28:47.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:47 smithi183 ceph-mon[33277]: 6.1f scrub starts 2024-09-21T12:28:47.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:47 smithi183 ceph-mon[33277]: 6.1f scrub ok 2024-09-21T12:28:47.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:47 smithi183 ceph-mon[33277]: osdmap e250: 8 total, 8 up, 7 in 2024-09-21T12:28:47.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:47 smithi183 ceph-mon[33277]: 1.1b scrub starts 2024-09-21T12:28:47.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:47 smithi183 ceph-mon[33277]: 1.1b scrub ok 2024-09-21T12:28:47.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:47 smithi183 ceph-mon[33277]: 4.d scrub starts 2024-09-21T12:28:47.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:47 smithi183 ceph-mon[33277]: 4.d scrub ok 2024-09-21T12:28:47.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:47 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:28:47.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:47 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:28:47.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:47 smithi183 ceph-mon[33277]: 6.1d scrub starts 2024-09-21T12:28:47.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:47 smithi183 ceph-mon[33277]: 6.1d scrub ok 2024-09-21T12:28:47.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:47 smithi183 ceph-mon[33277]: osdmap e251: 8 total, 8 up, 7 in 2024-09-21T12:28:47.521 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:47.558 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:47.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:47.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:47.660 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:47.703 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:47.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:47.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:47.800 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:28:47.800 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:28:47.805 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:47.826 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-21T12:28:47.826297+0000 min lat: 0.00136588 max lat: 0.130086 avg lat: 0.00726596 2024-09-21T12:28:47.827 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T12:28:47.827 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 20 16 44028 44012 137.521 99.8125 0.0121182 0.00726596 2024-09-21T12:28:47.832 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:47.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:47.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:47.833 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:28:47.833 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:28:47.869 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:28:47.869 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:28:47.918 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:28:47.918 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:28:47.933 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:47.956 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:28:47.956 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:28:47.985 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:47.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:47.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:47.993 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:28:47.994 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:28:48.026 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:28:48.026 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:28:48.061 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:28:48.061 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:28:48.087 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:48.112 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:48.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:48.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:48.213 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:48.215 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:48 smithi114 ceph-mon[40867]: 4.10 scrub starts 2024-09-21T12:28:48.215 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:48 smithi114 ceph-mon[40867]: 4.10 scrub ok 2024-09-21T12:28:48.215 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:48 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:28:48.215 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:48 smithi114 ceph-mon[40867]: 5.a scrub starts 2024-09-21T12:28:48.215 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:48 smithi114 ceph-mon[40867]: 5.a scrub ok 2024-09-21T12:28:48.215 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:48 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1194357341' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:28:48.215 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:48 smithi114 ceph-mon[40867]: 1.1d scrub starts 2024-09-21T12:28:48.215 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:48 smithi114 ceph-mon[40867]: 1.1d scrub ok 2024-09-21T12:28:48.215 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:48 smithi114 ceph-mon[30324]: 4.10 scrub starts 2024-09-21T12:28:48.216 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:48 smithi114 ceph-mon[30324]: 4.10 scrub ok 2024-09-21T12:28:48.216 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:48 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:28:48.216 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:48 smithi114 ceph-mon[30324]: 5.a scrub starts 2024-09-21T12:28:48.216 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:48 smithi114 ceph-mon[30324]: 5.a scrub ok 2024-09-21T12:28:48.216 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:48 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1194357341' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:28:48.216 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:48 smithi114 ceph-mon[30324]: 1.1d scrub starts 2024-09-21T12:28:48.216 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:48 smithi114 ceph-mon[30324]: 1.1d scrub ok 2024-09-21T12:28:48.240 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:48.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:48.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:48.342 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:48.382 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:48.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:48.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:48.484 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:48.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:48 smithi183 ceph-mon[33277]: 4.10 scrub starts 2024-09-21T12:28:48.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:48 smithi183 ceph-mon[33277]: 4.10 scrub ok 2024-09-21T12:28:48.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:48 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:28:48.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:48 smithi183 ceph-mon[33277]: 5.a scrub starts 2024-09-21T12:28:48.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:48 smithi183 ceph-mon[33277]: 5.a scrub ok 2024-09-21T12:28:48.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:48 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1194357341' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:28:48.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:48 smithi183 ceph-mon[33277]: 1.1d scrub starts 2024-09-21T12:28:48.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:48 smithi183 ceph-mon[33277]: 1.1d scrub ok 2024-09-21T12:28:48.511 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:48.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:48.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:48.615 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:48.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:48.658 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:48.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:48.758 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:48.783 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:48.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:48.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:48.827 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 21 16 45272 45256 134.674 77.75 0.00955886 0.00741573 2024-09-21T12:28:48.885 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:48.913 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:48.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:48.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:49.015 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:49.118 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:49.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:49.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:49.150 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T12:28:49.157 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-21T12:28:49.220 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:49.248 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:49.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:49.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:49.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:49 smithi114 ceph-mon[30324]: pgmap v420: 178 pgs: 178 active+clean; 2.6 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 150 MiB/s wr, 2.40k op/s 2024-09-21T12:28:49.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:49 smithi114 ceph-mon[30324]: 5.17 scrub starts 2024-09-21T12:28:49.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:49 smithi114 ceph-mon[30324]: 5.17 scrub ok 2024-09-21T12:28:49.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:49 smithi114 ceph-mon[30324]: 3.d scrub starts 2024-09-21T12:28:49.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:49 smithi114 ceph-mon[30324]: 3.d scrub ok 2024-09-21T12:28:49.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:49 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3900982972' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:28:49.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:49 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3186323581' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:28:49.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:49 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:28:49.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:49 smithi114 ceph-mon[40867]: pgmap v420: 178 pgs: 178 active+clean; 2.6 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 150 MiB/s wr, 2.40k op/s 2024-09-21T12:28:49.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:49 smithi114 ceph-mon[40867]: 5.17 scrub starts 2024-09-21T12:28:49.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:49 smithi114 ceph-mon[40867]: 5.17 scrub ok 2024-09-21T12:28:49.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:49 smithi114 ceph-mon[40867]: 3.d scrub starts 2024-09-21T12:28:49.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:49 smithi114 ceph-mon[40867]: 3.d scrub ok 2024-09-21T12:28:49.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:49 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3900982972' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:28:49.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:49 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3186323581' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:28:49.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:49 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:28:49.350 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:49.376 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:49.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:49.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:49.478 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:49.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:49 smithi183 ceph-mon[33277]: pgmap v420: 178 pgs: 178 active+clean; 2.6 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 150 MiB/s wr, 2.40k op/s 2024-09-21T12:28:49.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:49 smithi183 ceph-mon[33277]: 5.17 scrub starts 2024-09-21T12:28:49.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:49 smithi183 ceph-mon[33277]: 5.17 scrub ok 2024-09-21T12:28:49.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:49 smithi183 ceph-mon[33277]: 3.d scrub starts 2024-09-21T12:28:49.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:49 smithi183 ceph-mon[33277]: 3.d scrub ok 2024-09-21T12:28:49.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:49 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3900982972' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:28:49.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:49 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3186323581' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:28:49.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:49 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:28:49.514 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:49.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:49.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:49.616 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:49.645 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:49.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:49.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:49.746 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:49.784 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:49.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:49.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:49.827 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 22 16 46825 46809 132.964 97.0625 0.0155254 0.0075147 2024-09-21T12:28:49.886 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:49.922 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:49.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:49.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:50.023 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:50.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:50.056 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:50.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:50.158 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:50.201 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:50.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:50.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:50.302 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:50.345 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:50.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:50.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:50.407 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T12:28:50.407 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:28:50.447 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:50.447 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:50 smithi114 ceph-mon[30324]: 5.2 scrub starts 2024-09-21T12:28:50.447 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:50 smithi114 ceph-mon[30324]: 5.2 scrub ok 2024-09-21T12:28:50.448 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:50 smithi114 ceph-mon[30324]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:28:50.448 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:50 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3900982972' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T12:28:50.448 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:50 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:28:50.448 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:50 smithi114 ceph-mon[30324]: osdmap e252: 8 total, 8 up, 7 in 2024-09-21T12:28:50.448 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:50 smithi114 ceph-mon[40867]: 5.2 scrub starts 2024-09-21T12:28:50.448 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:50 smithi114 ceph-mon[40867]: 5.2 scrub ok 2024-09-21T12:28:50.448 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:50 smithi114 ceph-mon[40867]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:28:50.448 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:50 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3900982972' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T12:28:50.448 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:50 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:28:50.448 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:50 smithi114 ceph-mon[40867]: osdmap e252: 8 total, 8 up, 7 in 2024-09-21T12:28:50.456 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:50.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:50 smithi183 ceph-mon[33277]: 5.2 scrub starts 2024-09-21T12:28:50.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:50 smithi183 ceph-mon[33277]: 5.2 scrub ok 2024-09-21T12:28:50.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:50 smithi183 ceph-mon[33277]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:28:50.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:50 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3900982972' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T12:28:50.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:50 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:28:50.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:50 smithi183 ceph-mon[33277]: osdmap e252: 8 total, 8 up, 7 in 2024-09-21T12:28:50.495 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:50.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:50.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:50.597 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:50.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:50.648 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:50.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:50.749 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:50.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:50.791 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:50.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:50.827 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 23 16 48468 48452 131.647 102.688 0.00605119 0.00759111 2024-09-21T12:28:50.892 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:50.919 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:50.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:50.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:51.020 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:51.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:51.069 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:51.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:51.170 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:51.200 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:51.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:51.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:51.301 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:51.330 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:51.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:51.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:51.431 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:51.478 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:51.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:51.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:51.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:51 smithi183 ceph-mon[33277]: pgmap v422: 178 pgs: 178 active+clean; 2.6 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-21T12:28:51.523 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:51 smithi114 ceph-mon[30324]: pgmap v422: 178 pgs: 178 active+clean; 2.6 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-21T12:28:51.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:51 smithi114 ceph-mon[40867]: pgmap v422: 178 pgs: 178 active+clean; 2.6 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-21T12:28:51.570 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:28:51.570 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:28:51.580 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:51.609 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:51.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:51.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:51.711 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:51.739 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:51.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:51.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:51.827 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 24 16 50091 50075 130.388 101.438 0.0039494 0.00766463 2024-09-21T12:28:51.841 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:51.865 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:28:51.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:51.871 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:51.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:51.972 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:52.000 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:52.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:52.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:52.101 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:52.161 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:52.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:52.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:52.263 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:52.292 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:52.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:52.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:52.394 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:52.425 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:52.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:52.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:52.444 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:52.486 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T12:28:52.526 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:52.561 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:52.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:52.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:52.662 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:52.699 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:52.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:52.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:52.774 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:28:52 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:28:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:28:52.801 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:52.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:52.825 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:52.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:52.827 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 25 16 51608 51592 128.965 94.8125 0.00425366 0.00773534 2024-09-21T12:28:52.845 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:28:52.926 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:52.955 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:52.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:52.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:52.968 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T12:28:53.056 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:53.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:53.094 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:53.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:53.195 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:53.198 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:52 smithi114 ceph-mon[30324]: pgmap v423: 178 pgs: 178 active+clean; 2.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.7 KiB/s rd, 117 MiB/s wr, 1.88k op/s 2024-09-21T12:28:53.198 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:52 smithi114 ceph-mon[30324]: osdmap e253: 8 total, 8 up, 7 in 2024-09-21T12:28:53.198 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:52 smithi114 ceph-mon[40867]: pgmap v423: 178 pgs: 178 active+clean; 2.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.7 KiB/s rd, 117 MiB/s wr, 1.88k op/s 2024-09-21T12:28:53.198 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:52 smithi114 ceph-mon[40867]: osdmap e253: 8 total, 8 up, 7 in 2024-09-21T12:28:53.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:53.232 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:53.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:53.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:52 smithi183 ceph-mon[33277]: pgmap v423: 178 pgs: 178 active+clean; 2.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.7 KiB/s rd, 117 MiB/s wr, 1.88k op/s 2024-09-21T12:28:53.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:52 smithi183 ceph-mon[33277]: osdmap e253: 8 total, 8 up, 7 in 2024-09-21T12:28:53.333 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:53.362 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:53.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:53.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:53.391 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:28:53.391 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:28:53.464 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:53.497 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:53.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:53.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:53.599 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:53.639 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:53.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:53.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:53.706 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:53.713 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:53.741 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:53.767 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:53.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:53.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:53.827 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 26 16 52944 52928 127.216 83.5 0.0102029 0.00785509 2024-09-21T12:28:53.869 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:53.918 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:53.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:53.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:54.020 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:54.033 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:28:54.037 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:28:54.045 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:54.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:54.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:54.147 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:54.180 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T12:28:54.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:54.181 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:54.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:54.282 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:54.319 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:54.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:54.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:54.420 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:54.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:54.457 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:54.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:54.558 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:54.571 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:28:54.584 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:54.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:54.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:54.619 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:28:54.619 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:28:54.634 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:28:54.634 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:28:54.671 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:28:54.672 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:28:54.686 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:54.713 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:28:54.713 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:28:54.720 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:54.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:54.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:54.755 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:28:54.756 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:28:54.801 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:28:54.801 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:28:54.822 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:54.827 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 27 16 54337 54321 125.729 87.0625 0.0081079 0.00794822 2024-09-21T12:28:54.843 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:28:54.843 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:28:54.861 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:54.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:54.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:54.884 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:28:54.884 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:28:54.963 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:54.991 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:54.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:54.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:55.092 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:55.125 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:55.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:55.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:55.139 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:28:55.139 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-21T12:28:55.139 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-21T12:28:55.139 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-21T12:28:55.139 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:28:55.227 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:55.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:54 smithi183 ceph-mon[33277]: pgmap v425: 178 pgs: 178 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 752 B/s rd, 118 MiB/s wr, 1.89k op/s 2024-09-21T12:28:55.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:54 smithi183 ceph-mon[33277]: from='client.15582 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:28:55.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:54 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:28:55.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:54 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:28:55.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:54 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/181645101' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:28:55.265 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:55.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:55.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:55.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:54 smithi114 ceph-mon[40867]: pgmap v425: 178 pgs: 178 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 752 B/s rd, 118 MiB/s wr, 1.89k op/s 2024-09-21T12:28:55.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:54 smithi114 ceph-mon[40867]: from='client.15582 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:28:55.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:54 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:28:55.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:54 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:28:55.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:54 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/181645101' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:28:55.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:54 smithi114 ceph-mon[30324]: pgmap v425: 178 pgs: 178 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 752 B/s rd, 118 MiB/s wr, 1.89k op/s 2024-09-21T12:28:55.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:54 smithi114 ceph-mon[30324]: from='client.15582 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:28:55.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:54 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:28:55.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:54 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:28:55.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:54 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/181645101' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:28:55.368 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:55.397 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:55.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:55.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:55.499 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:55.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:55.522 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:55.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:55.622 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:28:55.659 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:55.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:55.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:28:55.760 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:55.796 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:55.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:55.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:55.827 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 28 16 55806 55790 124.517 91.8125 0.00874293 0.00802662 2024-09-21T12:28:55.898 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:55.929 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:55.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:55.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:55.977 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-21T12:28:56.031 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:56.056 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:56.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:56.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:56.120 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:55 smithi183 ceph-mon[33277]: osdmap e254: 8 total, 8 up, 7 in 2024-09-21T12:28:56.120 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:55 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/399085011' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:28:56.120 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:55 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:28:56.158 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:56.167 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T12:28:56.190 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:56.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:56.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:56.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:55 smithi114 ceph-mon[40867]: osdmap e254: 8 total, 8 up, 7 in 2024-09-21T12:28:56.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:55 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/399085011' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:28:56.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:55 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:28:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:55 smithi114 ceph-mon[30324]: osdmap e254: 8 total, 8 up, 7 in 2024-09-21T12:28:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:55 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/399085011' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:28:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:55 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:28:56.292 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:56.327 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:56.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:56.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:56.429 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:56.461 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:56.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:56.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:56.562 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:56.588 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:28:56.601 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:56.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:56.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:56.655 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":425,"stamp":"2024-09-21T12:28:53.737600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3301841707,"num_objects":50570,"num_object_clones":0,"num_object_copies":151710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50570,"num_whiteouts":0,"num_read":1224,"num_read_kb":976,"num_write":50807,"num_write_kb":3223789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":218,"num_bytes_recovered":4424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53311,"ondisk_log_size":53311,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":534,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":11857604,"kb_used_data":9548556,"kb_used_omap":10,"kb_used_meta":2308917,"kb_avail":644358460,"statfs":{"total":671965249536,"available":659823063040,"internally_reserved":0,"allocated":9777721344,"data_stored":9774082119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":2364331115},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":843972608,"num_objects":12878,"num_object_clones":0,"num_object_copies":38634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12878,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":12878,"num_write_kb":824192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.801629"},"pg_stats":[{"pgid":"1.1d","version":"76'79","reported_seq":97,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.968910+0000","last_change":"2024-09-21T12:28:47.816204+0000","last_active":"2024-09-21T12:28:51.968910+0000","last_peered":"2024-09-21T12:28:51.968910+0000","last_clean":"2024-09-21T12:28:51.968910+0000","last_became_active":"2024-09-21T12:27:16.947651+0000","last_became_peered":"2024-09-21T12:27:16.947651+0000","last_unstale":"2024-09-21T12:28:51.968910+0000","last_undegraded":"2024-09-21T12:28:51.968910+0000","last_fullsized":"2024-09-21T12:28:51.968910+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:47.816137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:47.816137+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:55.307153+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":420,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.949439+0000","last_change":"2024-09-21T12:24:50.115300+0000","last_active":"2024-09-21T12:28:51.949439+0000","last_peered":"2024-09-21T12:28:51.949439+0000","last_clean":"2024-09-21T12:28:51.949439+0000","last_became_active":"2024-09-21T12:20:15.309431+0000","last_became_peered":"2024-09-21T12:20:15.309431+0000","last_unstale":"2024-09-21T12:28:51.949439+0000","last_undegraded":"2024-09-21T12:28:51.949439+0000","last_fullsized":"2024-09-21T12:28:51.949439+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:50.115253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:50.115253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:18.674292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":394,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.166194+0000","last_change":"2024-09-21T12:27:32.886429+0000","last_active":"2024-09-21T12:28:49.166194+0000","last_peered":"2024-09-21T12:28:49.166194+0000","last_clean":"2024-09-21T12:28:49.166194+0000","last_became_active":"2024-09-21T12:22:50.909699+0000","last_became_peered":"2024-09-21T12:22:50.909699+0000","last_unstale":"2024-09-21T12:28:49.166194+0000","last_undegraded":"2024-09-21T12:28:49.166194+0000","last_fullsized":"2024-09-21T12:28:49.166194+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:27:32.886351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:32.886351+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":409,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.968009+0000","last_change":"2024-09-21T12:24:34.311501+0000","last_active":"2024-09-21T12:28:51.968009+0000","last_peered":"2024-09-21T12:28:51.968009+0000","last_clean":"2024-09-21T12:28:51.968009+0000","last_became_active":"2024-09-21T12:20:19.314908+0000","last_became_peered":"2024-09-21T12:20:19.314908+0000","last_unstale":"2024-09-21T12:28:51.968009+0000","last_undegraded":"2024-09-21T12:28:51.968009+0000","last_fullsized":"2024-09-21T12:28:51.968009+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:34.311451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:34.311451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:08.657850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":331,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.151853+0000","last_change":"2024-09-21T12:22:50.914064+0000","last_active":"2024-09-21T12:28:49.151853+0000","last_peered":"2024-09-21T12:28:49.151853+0000","last_clean":"2024-09-21T12:28:49.151853+0000","last_became_active":"2024-09-21T12:22:50.913743+0000","last_became_peered":"2024-09-21T12:22:50.913743+0000","last_unstale":"2024-09-21T12:28:49.151853+0000","last_undegraded":"2024-09-21T12:28:49.151853+0000","last_fullsized":"2024-09-21T12:28:49.151853+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:58.189665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:21:58.189665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:13.364470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1c","version":"76'79","reported_seq":85,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.155230+0000","last_change":"2024-09-21T12:27:28.374689+0000","last_active":"2024-09-21T12:28:49.155230+0000","last_peered":"2024-09-21T12:28:49.155230+0000","last_clean":"2024-09-21T12:28:49.155230+0000","last_became_active":"2024-09-21T12:27:16.920545+0000","last_became_peered":"2024-09-21T12:27:16.920545+0000","last_unstale":"2024-09-21T12:28:49.155230+0000","last_undegraded":"2024-09-21T12:28:49.155230+0000","last_fullsized":"2024-09-21T12:28:49.155230+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:28.374616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:28.374616+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":396,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.939917+0000","last_change":"2024-09-21T12:25:48.316122+0000","last_active":"2024-09-21T12:28:51.939917+0000","last_peered":"2024-09-21T12:28:51.939917+0000","last_clean":"2024-09-21T12:28:51.939917+0000","last_became_active":"2024-09-21T12:20:21.595301+0000","last_became_peered":"2024-09-21T12:20:21.595301+0000","last_unstale":"2024-09-21T12:28:51.939917+0000","last_undegraded":"2024-09-21T12:28:51.939917+0000","last_fullsized":"2024-09-21T12:28:51.939917+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:48.316039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:48.316039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":417,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.940532+0000","last_change":"2024-09-21T12:24:49.701972+0000","last_active":"2024-09-21T12:28:51.940532+0000","last_peered":"2024-09-21T12:28:51.940532+0000","last_clean":"2024-09-21T12:28:51.940532+0000","last_became_active":"2024-09-21T12:20:15.308655+0000","last_became_peered":"2024-09-21T12:20:15.308655+0000","last_unstale":"2024-09-21T12:28:51.940532+0000","last_undegraded":"2024-09-21T12:28:51.940532+0000","last_fullsized":"2024-09-21T12:28:51.940532+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:49.701915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:49.701915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:12.598978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":445,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.941095+0000","last_change":"2024-09-21T12:27:54.371491+0000","last_active":"2024-09-21T12:28:51.941095+0000","last_peered":"2024-09-21T12:28:51.941095+0000","last_clean":"2024-09-21T12:28:51.941095+0000","last_became_active":"2024-09-21T12:20:17.305220+0000","last_became_peered":"2024-09-21T12:20:17.305220+0000","last_unstale":"2024-09-21T12:28:51.941095+0000","last_undegraded":"2024-09-21T12:28:51.941095+0000","last_fullsized":"2024-09-21T12:28:51.941095+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:27:54.369084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:54.369084+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:02.669546+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":356,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.166266+0000","last_change":"2024-09-21T12:28:39.179737+0000","last_active":"2024-09-21T12:28:49.166266+0000","last_peered":"2024-09-21T12:28:49.166266+0000","last_clean":"2024-09-21T12:28:49.166266+0000","last_became_active":"2024-09-21T12:22:50.926240+0000","last_became_peered":"2024-09-21T12:22:50.926240+0000","last_unstale":"2024-09-21T12:28:49.166266+0000","last_undegraded":"2024-09-21T12:28:49.166266+0000","last_fullsized":"2024-09-21T12:28:49.166266+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:39.179665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:39.179665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:55.735084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":252,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.166922+0000","last_change":"2024-09-21T12:26:54.084511+0000","last_active":"2024-09-21T12:28:49.166922+0000","last_peered":"2024-09-21T12:28:49.166922+0000","last_clean":"2024-09-21T12:28:49.166922+0000","last_became_active":"2024-09-21T12:26:54.083981+0000","last_became_peered":"2024-09-21T12:26:54.083981+0000","last_unstale":"2024-09-21T12:28:49.166922+0000","last_undegraded":"2024-09-21T12:28:49.166922+0000","last_fullsized":"2024-09-21T12:28:49.166922+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:19.446251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:19.446251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:49.993731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":405,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.939553+0000","last_change":"2024-09-21T12:28:06.589984+0000","last_active":"2024-09-21T12:28:51.939553+0000","last_peered":"2024-09-21T12:28:51.939553+0000","last_clean":"2024-09-21T12:28:51.939553+0000","last_became_active":"2024-09-21T12:22:50.927804+0000","last_became_peered":"2024-09-21T12:22:50.927804+0000","last_unstale":"2024-09-21T12:28:51.939553+0000","last_undegraded":"2024-09-21T12:28:51.939553+0000","last_fullsized":"2024-09-21T12:28:51.939553+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:28:06.589926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:06.589926+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:33.744896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":415,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.152187+0000","last_change":"2024-09-21T12:26:54.497247+0000","last_active":"2024-09-21T12:28:49.152187+0000","last_peered":"2024-09-21T12:28:49.152187+0000","last_clean":"2024-09-21T12:28:49.152187+0000","last_became_active":"2024-09-21T12:20:19.308184+0000","last_became_peered":"2024-09-21T12:20:19.308184+0000","last_unstale":"2024-09-21T12:28:49.152187+0000","last_undegraded":"2024-09-21T12:28:49.152187+0000","last_fullsized":"2024-09-21T12:28:49.152187+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:54.497175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:54.497175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:20.364274+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":415,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.945431+0000","last_change":"2024-09-21T12:27:54.959865+0000","last_active":"2024-09-21T12:28:51.945431+0000","last_peered":"2024-09-21T12:28:51.945431+0000","last_clean":"2024-09-21T12:28:51.945431+0000","last_became_active":"2024-09-21T12:20:21.309190+0000","last_became_peered":"2024-09-21T12:20:21.309190+0000","last_unstale":"2024-09-21T12:28:51.945431+0000","last_undegraded":"2024-09-21T12:28:51.945431+0000","last_fullsized":"2024-09-21T12:28:51.945431+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:54.959806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:27:54.959806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:15.995385+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"76'79","reported_seq":180,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.940348+0000","last_change":"2024-09-21T12:27:20.216167+0000","last_active":"2024-09-21T12:28:51.940348+0000","last_peered":"2024-09-21T12:28:51.940348+0000","last_clean":"2024-09-21T12:28:51.940348+0000","last_became_active":"2024-09-21T12:27:16.937711+0000","last_became_peered":"2024-09-21T12:27:16.937711+0000","last_unstale":"2024-09-21T12:28:51.940348+0000","last_undegraded":"2024-09-21T12:28:51.940348+0000","last_fullsized":"2024-09-21T12:28:51.940348+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:20.216102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:20.216102+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":262,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.968952+0000","last_change":"2024-09-21T12:28:33.644630+0000","last_active":"2024-09-21T12:28:51.968952+0000","last_peered":"2024-09-21T12:28:51.968952+0000","last_clean":"2024-09-21T12:28:51.968952+0000","last_became_active":"2024-09-21T12:26:54.085055+0000","last_became_peered":"2024-09-21T12:26:54.085055+0000","last_unstale":"2024-09-21T12:28:51.968952+0000","last_undegraded":"2024-09-21T12:28:51.968952+0000","last_fullsized":"2024-09-21T12:28:51.968952+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:33.644570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:33.644570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:00.099267+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":383,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.940257+0000","last_change":"2024-09-21T12:24:28.624430+0000","last_active":"2024-09-21T12:28:51.940257+0000","last_peered":"2024-09-21T12:28:51.940257+0000","last_clean":"2024-09-21T12:28:51.940257+0000","last_became_active":"2024-09-21T12:22:50.918794+0000","last_became_peered":"2024-09-21T12:22:50.918794+0000","last_unstale":"2024-09-21T12:28:51.940257+0000","last_undegraded":"2024-09-21T12:28:51.940257+0000","last_fullsized":"2024-09-21T12:28:51.940257+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-21T12:24:28.624364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:28.624364+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:50.980764+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":428,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.154791+0000","last_change":"2024-09-21T12:28:04.623499+0000","last_active":"2024-09-21T12:28:49.154791+0000","last_peered":"2024-09-21T12:28:49.154791+0000","last_clean":"2024-09-21T12:28:49.154791+0000","last_became_active":"2024-09-21T12:22:50.921780+0000","last_became_peered":"2024-09-21T12:22:50.921780+0000","last_unstale":"2024-09-21T12:28:49.154791+0000","last_undegraded":"2024-09-21T12:28:49.154791+0000","last_fullsized":"2024-09-21T12:28:49.154791+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:04.623389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:04.623389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:13.184696+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":419,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.152087+0000","last_change":"2024-09-21T12:28:38.237336+0000","last_active":"2024-09-21T12:28:49.152087+0000","last_peered":"2024-09-21T12:28:49.152087+0000","last_clean":"2024-09-21T12:28:49.152087+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:28:49.152087+0000","last_undegraded":"2024-09-21T12:28:49.152087+0000","last_fullsized":"2024-09-21T12:28:49.152087+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:38.237289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:38.237289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:50.035648+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"76'79","reported_seq":251,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.940176+0000","last_change":"2024-09-21T12:27:22.289065+0000","last_active":"2024-09-21T12:28:51.940176+0000","last_peered":"2024-09-21T12:28:51.940176+0000","last_clean":"2024-09-21T12:28:51.940176+0000","last_became_active":"2024-09-21T12:27:16.957569+0000","last_became_peered":"2024-09-21T12:27:16.957569+0000","last_unstale":"2024-09-21T12:28:51.940176+0000","last_undegraded":"2024-09-21T12:28:51.940176+0000","last_fullsized":"2024-09-21T12:28:51.940176+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:22.288989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:22.288989+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":356,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.167204+0000","last_change":"2024-09-21T12:26:44.845249+0000","last_active":"2024-09-21T12:28:49.167204+0000","last_peered":"2024-09-21T12:28:49.167204+0000","last_clean":"2024-09-21T12:28:49.167204+0000","last_became_active":"2024-09-21T12:22:50.914792+0000","last_became_peered":"2024-09-21T12:22:50.914792+0000","last_unstale":"2024-09-21T12:28:49.167204+0000","last_undegraded":"2024-09-21T12:28:49.167204+0000","last_fullsized":"2024-09-21T12:28:49.167204+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:44.845176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:44.845176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:12.195899+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":448,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.946742+0000","last_change":"2024-09-21T12:24:30.033946+0000","last_active":"2024-09-21T12:28:51.946742+0000","last_peered":"2024-09-21T12:28:51.946742+0000","last_clean":"2024-09-21T12:28:51.946742+0000","last_became_active":"2024-09-21T12:22:50.921576+0000","last_became_peered":"2024-09-21T12:22:50.921576+0000","last_unstale":"2024-09-21T12:28:51.946742+0000","last_undegraded":"2024-09-21T12:28:51.946742+0000","last_fullsized":"2024-09-21T12:28:51.946742+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:30.033874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:30.033874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:14.654620+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":447,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.964994+0000","last_change":"2024-09-21T12:27:22.516532+0000","last_active":"2024-09-21T12:28:51.964994+0000","last_peered":"2024-09-21T12:28:51.964994+0000","last_clean":"2024-09-21T12:28:51.964994+0000","last_became_active":"2024-09-21T12:20:17.307828+0000","last_became_peered":"2024-09-21T12:20:17.307828+0000","last_unstale":"2024-09-21T12:28:51.964994+0000","last_undegraded":"2024-09-21T12:28:51.964994+0000","last_fullsized":"2024-09-21T12:28:51.964994+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:27:22.516442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:22.516442+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":445,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.969631+0000","last_change":"2024-09-21T12:28:06.636769+0000","last_active":"2024-09-21T12:28:51.969631+0000","last_peered":"2024-09-21T12:28:51.969631+0000","last_clean":"2024-09-21T12:28:51.969631+0000","last_became_active":"2024-09-21T12:25:46.905878+0000","last_became_peered":"2024-09-21T12:25:46.905878+0000","last_unstale":"2024-09-21T12:28:51.969631+0000","last_undegraded":"2024-09-21T12:28:51.969631+0000","last_fullsized":"2024-09-21T12:28:51.969631+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:06.636729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:06.636729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:23.255350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"76'79","reported_seq":342,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.151804+0000","last_change":"2024-09-21T12:28:28.293163+0000","last_active":"2024-09-21T12:28:49.151804+0000","last_peered":"2024-09-21T12:28:49.151804+0000","last_clean":"2024-09-21T12:28:49.151804+0000","last_became_active":"2024-09-21T12:27:16.927791+0000","last_became_peered":"2024-09-21T12:27:16.927791+0000","last_unstale":"2024-09-21T12:28:49.151804+0000","last_undegraded":"2024-09-21T12:28:49.151804+0000","last_fullsized":"2024-09-21T12:28:49.151804+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:28.293087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:28.293087+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:44.932604+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":405,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.941727+0000","last_change":"2024-09-21T12:28:45.880599+0000","last_active":"2024-09-21T12:28:51.941727+0000","last_peered":"2024-09-21T12:28:51.941727+0000","last_clean":"2024-09-21T12:28:51.941727+0000","last_became_active":"2024-09-21T12:20:21.592796+0000","last_became_peered":"2024-09-21T12:20:21.592796+0000","last_unstale":"2024-09-21T12:28:51.941727+0000","last_undegraded":"2024-09-21T12:28:51.941727+0000","last_fullsized":"2024-09-21T12:28:51.941727+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:45.880383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.880383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:01.380533+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":356,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.967890+0000","last_change":"2024-09-21T12:28:45.726096+0000","last_active":"2024-09-21T12:28:51.967890+0000","last_peered":"2024-09-21T12:28:51.967890+0000","last_clean":"2024-09-21T12:28:51.967890+0000","last_became_active":"2024-09-21T12:22:50.925403+0000","last_became_peered":"2024-09-21T12:22:50.925403+0000","last_unstale":"2024-09-21T12:28:51.967890+0000","last_undegraded":"2024-09-21T12:28:51.967890+0000","last_fullsized":"2024-09-21T12:28:51.967890+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:45.725968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.725968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:08.835537+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":500,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.939629+0000","last_change":"2024-09-21T12:28:30.620035+0000","last_active":"2024-09-21T12:28:51.939629+0000","last_peered":"2024-09-21T12:28:51.939629+0000","last_clean":"2024-09-21T12:28:51.939629+0000","last_became_active":"2024-09-21T12:22:50.912144+0000","last_became_peered":"2024-09-21T12:22:50.912144+0000","last_unstale":"2024-09-21T12:28:51.939629+0000","last_undegraded":"2024-09-21T12:28:51.939629+0000","last_fullsized":"2024-09-21T12:28:51.939629+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:28:30.619954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:30.619954+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:52.523678+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":334,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.943324+0000","last_change":"2024-09-21T12:26:52.295004+0000","last_active":"2024-09-21T12:28:51.943324+0000","last_peered":"2024-09-21T12:28:51.943324+0000","last_clean":"2024-09-21T12:28:51.943324+0000","last_became_active":"2024-09-21T12:22:50.918326+0000","last_became_peered":"2024-09-21T12:22:50.918326+0000","last_unstale":"2024-09-21T12:28:51.943324+0000","last_undegraded":"2024-09-21T12:28:51.943324+0000","last_fullsized":"2024-09-21T12:28:51.943324+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:52.294931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:52.294931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"76'79","reported_seq":142,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.151617+0000","last_change":"2024-09-21T12:28:46.294913+0000","last_active":"2024-09-21T12:28:49.151617+0000","last_peered":"2024-09-21T12:28:49.151617+0000","last_clean":"2024-09-21T12:28:49.151617+0000","last_became_active":"2024-09-21T12:27:16.933358+0000","last_became_peered":"2024-09-21T12:27:16.933358+0000","last_unstale":"2024-09-21T12:28:49.151617+0000","last_undegraded":"2024-09-21T12:28:49.151617+0000","last_fullsized":"2024-09-21T12:28:49.151617+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:46.294855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:46.294855+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:07.298788+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":396,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.154789+0000","last_change":"2024-09-21T12:23:51.065780+0000","last_active":"2024-09-21T12:28:49.154789+0000","last_peered":"2024-09-21T12:28:49.154789+0000","last_clean":"2024-09-21T12:28:49.154789+0000","last_became_active":"2024-09-21T12:20:21.312976+0000","last_became_peered":"2024-09-21T12:20:21.312976+0000","last_unstale":"2024-09-21T12:28:49.154789+0000","last_undegraded":"2024-09-21T12:28:49.154789+0000","last_fullsized":"2024-09-21T12:28:49.154789+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:23:51.065733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:51.065733+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:15.680094+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":459,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.968109+0000","last_change":"2024-09-21T12:27:56.597623+0000","last_active":"2024-09-21T12:28:51.968109+0000","last_peered":"2024-09-21T12:28:51.968109+0000","last_clean":"2024-09-21T12:28:51.968109+0000","last_became_active":"2024-09-21T12:22:50.910933+0000","last_became_peered":"2024-09-21T12:22:50.910933+0000","last_unstale":"2024-09-21T12:28:51.968109+0000","last_undegraded":"2024-09-21T12:28:51.968109+0000","last_fullsized":"2024-09-21T12:28:51.968109+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:27:56.597473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:27:56.597473+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:25.388812+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":423,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.950911+0000","last_change":"2024-09-21T12:24:03.059782+0000","last_active":"2024-09-21T12:28:51.950911+0000","last_peered":"2024-09-21T12:28:51.950911+0000","last_clean":"2024-09-21T12:28:51.950911+0000","last_became_active":"2024-09-21T12:20:17.695977+0000","last_became_peered":"2024-09-21T12:20:17.695977+0000","last_unstale":"2024-09-21T12:28:51.950911+0000","last_undegraded":"2024-09-21T12:28:51.950911+0000","last_fullsized":"2024-09-21T12:28:51.950911+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:24:03.059549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:03.059549+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:50.851518+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"121'11","reported_seq":564,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.167517+0000","last_change":"2024-09-21T12:26:10.846256+0000","last_active":"2024-09-21T12:28:49.167517+0000","last_peered":"2024-09-21T12:28:49.167517+0000","last_clean":"2024-09-21T12:28:49.167517+0000","last_became_active":"2024-09-21T12:22:50.916683+0000","last_became_peered":"2024-09-21T12:22:50.916683+0000","last_unstale":"2024-09-21T12:28:49.167517+0000","last_undegraded":"2024-09-21T12:28:49.167517+0000","last_fullsized":"2024-09-21T12:28:49.167517+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:26:10.846183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:10.846183+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:50.024651+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"253'3142","reported_seq":3185,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:52.327656+0000","last_change":"2024-09-21T12:28:20.956387+0000","last_active":"2024-09-21T12:28:52.327656+0000","last_peered":"2024-09-21T12:28:52.327656+0000","last_clean":"2024-09-21T12:28:52.327656+0000","last_became_active":"2024-09-21T12:28:20.955475+0000","last_became_peered":"2024-09-21T12:28:20.955475+0000","last_unstale":"2024-09-21T12:28:52.327656+0000","last_undegraded":"2024-09-21T12:28:52.327656+0000","last_fullsized":"2024-09-21T12:28:52.327656+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3142,"log_dups_size":0,"ondisk_log_size":3142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:48.118898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205914112,"num_objects":3142,"num_object_clones":0,"num_object_copies":9426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3142,"num_write_kb":201088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"76'79","reported_seq":135,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.963954+0000","last_change":"2024-09-21T12:28:29.446170+0000","last_active":"2024-09-21T12:28:51.963954+0000","last_peered":"2024-09-21T12:28:51.963954+0000","last_clean":"2024-09-21T12:28:51.963954+0000","last_became_active":"2024-09-21T12:27:14.943286+0000","last_became_peered":"2024-09-21T12:27:14.943286+0000","last_unstale":"2024-09-21T12:28:51.963954+0000","last_undegraded":"2024-09-21T12:28:51.963954+0000","last_fullsized":"2024-09-21T12:28:51.963954+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:29.446103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:29.446103+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:34.602581+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":277,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.963290+0000","last_change":"2024-09-21T12:27:31.491370+0000","last_active":"2024-09-21T12:28:51.963290+0000","last_peered":"2024-09-21T12:28:51.963290+0000","last_clean":"2024-09-21T12:28:51.963290+0000","last_became_active":"2024-09-21T12:23:46.910112+0000","last_became_peered":"2024-09-21T12:23:46.910112+0000","last_unstale":"2024-09-21T12:28:51.963290+0000","last_undegraded":"2024-09-21T12:28:51.963290+0000","last_fullsized":"2024-09-21T12:28:51.963290+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:27:31.490985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:31.490985+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":416,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.940833+0000","last_change":"2024-09-21T12:28:05.580894+0000","last_active":"2024-09-21T12:28:51.940833+0000","last_peered":"2024-09-21T12:28:51.940833+0000","last_clean":"2024-09-21T12:28:51.940833+0000","last_became_active":"2024-09-21T12:20:15.295774+0000","last_became_peered":"2024-09-21T12:20:15.295774+0000","last_unstale":"2024-09-21T12:28:51.940833+0000","last_undegraded":"2024-09-21T12:28:51.940833+0000","last_fullsized":"2024-09-21T12:28:51.940833+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:05.580847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:05.580847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:25.392108+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"210'14","reported_seq":126,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.946360+0000","last_change":"2024-09-21T12:26:54.091833+0000","last_active":"2024-09-21T12:28:51.946360+0000","last_peered":"2024-09-21T12:28:51.946360+0000","last_clean":"2024-09-21T12:28:51.946360+0000","last_became_active":"2024-09-21T12:26:54.091451+0000","last_became_peered":"2024-09-21T12:26:54.091451+0000","last_unstale":"2024-09-21T12:28:51.946360+0000","last_undegraded":"2024-09-21T12:28:51.946360+0000","last_fullsized":"2024-09-21T12:28:51.946360+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'13","last_scrub_stamp":"2024-09-21T12:25:33.474391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:33.474391+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:53.972439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":127,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.969533+0000","last_change":"2024-09-21T12:28:31.580349+0000","last_active":"2024-09-21T12:28:51.969533+0000","last_peered":"2024-09-21T12:28:51.969533+0000","last_clean":"2024-09-21T12:28:51.969533+0000","last_became_active":"2024-09-21T12:26:54.089699+0000","last_became_peered":"2024-09-21T12:26:54.089699+0000","last_unstale":"2024-09-21T12:28:51.969533+0000","last_undegraded":"2024-09-21T12:28:51.969533+0000","last_fullsized":"2024-09-21T12:28:51.969533+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:28:31.580245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:31.580245+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:42.630759+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.d","version":"252'3070","reported_seq":3111,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.653032+0000","last_change":"2024-09-21T12:28:20.958630+0000","last_active":"2024-09-21T12:28:51.653032+0000","last_peered":"2024-09-21T12:28:51.653032+0000","last_clean":"2024-09-21T12:28:51.653032+0000","last_became_active":"2024-09-21T12:28:20.958322+0000","last_became_peered":"2024-09-21T12:28:20.958322+0000","last_unstale":"2024-09-21T12:28:51.653032+0000","last_undegraded":"2024-09-21T12:28:51.653032+0000","last_fullsized":"2024-09-21T12:28:51.653032+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3070,"log_dups_size":0,"ondisk_log_size":3070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:24.412238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201195520,"num_objects":3070,"num_object_clones":0,"num_object_copies":9210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3070,"num_write_kb":196480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"76'79","reported_seq":342,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.963776+0000","last_change":"2024-09-21T12:27:20.527090+0000","last_active":"2024-09-21T12:28:51.963776+0000","last_peered":"2024-09-21T12:28:51.963776+0000","last_clean":"2024-09-21T12:28:51.963776+0000","last_became_active":"2024-09-21T12:27:14.941905+0000","last_became_peered":"2024-09-21T12:27:14.941905+0000","last_unstale":"2024-09-21T12:28:51.963776+0000","last_undegraded":"2024-09-21T12:28:51.963776+0000","last_fullsized":"2024-09-21T12:28:51.963776+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:20.526951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:20.526951+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":453,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.940675+0000","last_change":"2024-09-21T12:28:26.632364+0000","last_active":"2024-09-21T12:28:51.940675+0000","last_peered":"2024-09-21T12:28:51.940675+0000","last_clean":"2024-09-21T12:28:51.940675+0000","last_became_active":"2024-09-21T12:20:17.693179+0000","last_became_peered":"2024-09-21T12:20:17.693179+0000","last_unstale":"2024-09-21T12:28:51.940675+0000","last_undegraded":"2024-09-21T12:28:51.940675+0000","last_fullsized":"2024-09-21T12:28:51.940675+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-21T12:28:26.632272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:26.632272+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:48.718928+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":408,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.942227+0000","last_change":"2024-09-21T12:21:18.056305+0000","last_active":"2024-09-21T12:28:51.942227+0000","last_peered":"2024-09-21T12:28:51.942227+0000","last_clean":"2024-09-21T12:28:51.942227+0000","last_became_active":"2024-09-21T12:20:15.297340+0000","last_became_peered":"2024-09-21T12:20:15.297340+0000","last_unstale":"2024-09-21T12:28:51.942227+0000","last_undegraded":"2024-09-21T12:28:51.942227+0000","last_fullsized":"2024-09-21T12:28:51.942227+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:18.056261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:18.056261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:09.624424+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":359,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.940429+0000","last_change":"2024-09-21T12:28:33.697269+0000","last_active":"2024-09-21T12:28:51.940429+0000","last_peered":"2024-09-21T12:28:51.940429+0000","last_clean":"2024-09-21T12:28:51.940429+0000","last_became_active":"2024-09-21T12:22:50.930858+0000","last_became_peered":"2024-09-21T12:22:50.930858+0000","last_unstale":"2024-09-21T12:28:51.940429+0000","last_undegraded":"2024-09-21T12:28:51.940429+0000","last_fullsized":"2024-09-21T12:28:51.940429+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:33.697208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:33.697208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:03.607813+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":350,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.939467+0000","last_change":"2024-09-21T12:26:44.325219+0000","last_active":"2024-09-21T12:28:51.939467+0000","last_peered":"2024-09-21T12:28:51.939467+0000","last_clean":"2024-09-21T12:28:51.939467+0000","last_became_active":"2024-09-21T12:22:50.918940+0000","last_became_peered":"2024-09-21T12:22:50.918940+0000","last_unstale":"2024-09-21T12:28:51.939467+0000","last_undegraded":"2024-09-21T12:28:51.939467+0000","last_fullsized":"2024-09-21T12:28:51.939467+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:44.325124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:44.325124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"253'3213","reported_seq":3257,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:52.325305+0000","last_change":"2024-09-21T12:28:20.956665+0000","last_active":"2024-09-21T12:28:52.325305+0000","last_peered":"2024-09-21T12:28:52.325305+0000","last_clean":"2024-09-21T12:28:52.325305+0000","last_became_active":"2024-09-21T12:28:20.955557+0000","last_became_peered":"2024-09-21T12:28:20.955557+0000","last_unstale":"2024-09-21T12:28:52.325305+0000","last_undegraded":"2024-09-21T12:28:52.325305+0000","last_fullsized":"2024-09-21T12:28:52.325305+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3213,"log_dups_size":0,"ondisk_log_size":3213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:38.621432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210567168,"num_objects":3213,"num_object_clones":0,"num_object_copies":9639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3213,"num_write_kb":205632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"76'79","reported_seq":344,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.943150+0000","last_change":"2024-09-21T12:26:45.354836+0000","last_active":"2024-09-21T12:28:51.943150+0000","last_peered":"2024-09-21T12:28:51.943150+0000","last_clean":"2024-09-21T12:28:51.943150+0000","last_became_active":"2024-09-21T12:26:44.960093+0000","last_became_peered":"2024-09-21T12:26:44.960093+0000","last_unstale":"2024-09-21T12:28:51.943150+0000","last_undegraded":"2024-09-21T12:28:51.943150+0000","last_fullsized":"2024-09-21T12:28:51.943150+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:45.354774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:45.354774+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":413,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.152249+0000","last_change":"2024-09-21T12:26:51.463759+0000","last_active":"2024-09-21T12:28:49.152249+0000","last_peered":"2024-09-21T12:28:49.152249+0000","last_clean":"2024-09-21T12:28:49.152249+0000","last_became_active":"2024-09-21T12:22:50.926008+0000","last_became_peered":"2024-09-21T12:22:50.926008+0000","last_unstale":"2024-09-21T12:28:49.152249+0000","last_undegraded":"2024-09-21T12:28:49.152249+0000","last_fullsized":"2024-09-21T12:28:49.152249+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-21T12:26:51.463698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:51.463698+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:23.362051+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":461,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.951187+0000","last_change":"2024-09-21T12:28:32.727794+0000","last_active":"2024-09-21T12:28:51.951187+0000","last_peered":"2024-09-21T12:28:51.951187+0000","last_clean":"2024-09-21T12:28:51.951187+0000","last_became_active":"2024-09-21T12:22:50.923284+0000","last_became_peered":"2024-09-21T12:22:50.923284+0000","last_unstale":"2024-09-21T12:28:51.951187+0000","last_undegraded":"2024-09-21T12:28:51.951187+0000","last_fullsized":"2024-09-21T12:28:51.951187+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:28:32.727727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:32.727727+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:52.197596+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"121'11","reported_seq":554,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.940527+0000","last_change":"2024-09-21T12:26:14.490045+0000","last_active":"2024-09-21T12:28:51.940527+0000","last_peered":"2024-09-21T12:28:51.940527+0000","last_clean":"2024-09-21T12:28:51.940527+0000","last_became_active":"2024-09-21T12:22:50.930342+0000","last_became_peered":"2024-09-21T12:22:50.930342+0000","last_unstale":"2024-09-21T12:28:51.940527+0000","last_undegraded":"2024-09-21T12:28:51.940527+0000","last_fullsized":"2024-09-21T12:28:51.940527+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:26:14.489972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:14.489972+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":404,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.154725+0000","last_change":"2024-09-21T12:28:37.686403+0000","last_active":"2024-09-21T12:28:49.154725+0000","last_peered":"2024-09-21T12:28:49.154725+0000","last_clean":"2024-09-21T12:28:49.154725+0000","last_became_active":"2024-09-21T12:20:21.593367+0000","last_became_peered":"2024-09-21T12:20:21.593367+0000","last_unstale":"2024-09-21T12:28:49.154725+0000","last_undegraded":"2024-09-21T12:28:49.154725+0000","last_fullsized":"2024-09-21T12:28:49.154725+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:37.686350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:37.686350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:53.013407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"252'3107","reported_seq":3149,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.644205+0000","last_change":"2024-09-21T12:28:20.956941+0000","last_active":"2024-09-21T12:28:51.644205+0000","last_peered":"2024-09-21T12:28:51.644205+0000","last_clean":"2024-09-21T12:28:51.644205+0000","last_became_active":"2024-09-21T12:28:20.956619+0000","last_became_peered":"2024-09-21T12:28:20.956619+0000","last_unstale":"2024-09-21T12:28:51.644205+0000","last_undegraded":"2024-09-21T12:28:51.644205+0000","last_fullsized":"2024-09-21T12:28:51.644205+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3107,"log_dups_size":0,"ondisk_log_size":3107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:45.844072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203620352,"num_objects":3107,"num_object_clones":0,"num_object_copies":9321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3107,"num_write_kb":198848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"76'79","reported_seq":441,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.167386+0000","last_change":"2024-09-21T12:27:24.874811+0000","last_active":"2024-09-21T12:28:49.167386+0000","last_peered":"2024-09-21T12:28:49.167386+0000","last_clean":"2024-09-21T12:28:49.167386+0000","last_became_active":"2024-09-21T12:27:14.937216+0000","last_became_peered":"2024-09-21T12:27:14.937216+0000","last_unstale":"2024-09-21T12:28:49.167386+0000","last_undegraded":"2024-09-21T12:28:49.167386+0000","last_fullsized":"2024-09-21T12:28:49.167386+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:24.874768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:24.874768+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":486,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.968928+0000","last_change":"2024-09-21T12:27:27.325248+0000","last_active":"2024-09-21T12:28:51.968928+0000","last_peered":"2024-09-21T12:28:51.968928+0000","last_clean":"2024-09-21T12:28:51.968928+0000","last_became_active":"2024-09-21T12:22:50.925840+0000","last_became_peered":"2024-09-21T12:22:50.925840+0000","last_unstale":"2024-09-21T12:28:51.968928+0000","last_undegraded":"2024-09-21T12:28:51.968928+0000","last_fullsized":"2024-09-21T12:28:51.968928+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:27:27.325105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:27.325105+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":424,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.152435+0000","last_change":"2024-09-21T12:27:25.427260+0000","last_active":"2024-09-21T12:28:49.152435+0000","last_peered":"2024-09-21T12:28:49.152435+0000","last_clean":"2024-09-21T12:28:49.152435+0000","last_became_active":"2024-09-21T12:20:15.308518+0000","last_became_peered":"2024-09-21T12:20:15.308518+0000","last_unstale":"2024-09-21T12:28:49.152435+0000","last_undegraded":"2024-09-21T12:28:49.152435+0000","last_fullsized":"2024-09-21T12:28:49.152435+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:25.427219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:27:25.427219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":535,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.947235+0000","last_change":"2024-09-21T12:27:58.871800+0000","last_active":"2024-09-21T12:28:51.947235+0000","last_peered":"2024-09-21T12:28:51.947235+0000","last_clean":"2024-09-21T12:28:51.947235+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:28:51.947235+0000","last_undegraded":"2024-09-21T12:28:51.947235+0000","last_fullsized":"2024-09-21T12:28:51.947235+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:27:58.871710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:27:58.871710+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:27.524400+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":416,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.945603+0000","last_change":"2024-09-21T12:28:37.850166+0000","last_active":"2024-09-21T12:28:51.945603+0000","last_peered":"2024-09-21T12:28:51.945603+0000","last_clean":"2024-09-21T12:28:51.945603+0000","last_became_active":"2024-09-21T12:20:21.315801+0000","last_became_peered":"2024-09-21T12:20:21.315801+0000","last_unstale":"2024-09-21T12:28:51.945603+0000","last_undegraded":"2024-09-21T12:28:51.945603+0000","last_fullsized":"2024-09-21T12:28:51.945603+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:37.850100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:37.850100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:52.359950+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"252'3155","reported_seq":3197,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.659172+0000","last_change":"2024-09-21T12:28:20.958462+0000","last_active":"2024-09-21T12:28:51.659172+0000","last_peered":"2024-09-21T12:28:51.659172+0000","last_clean":"2024-09-21T12:28:51.659172+0000","last_became_active":"2024-09-21T12:28:20.958143+0000","last_became_peered":"2024-09-21T12:28:20.958143+0000","last_unstale":"2024-09-21T12:28:51.659172+0000","last_undegraded":"2024-09-21T12:28:51.659172+0000","last_fullsized":"2024-09-21T12:28:51.659172+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3155,"log_dups_size":0,"ondisk_log_size":3155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:47.169876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206766080,"num_objects":3155,"num_object_clones":0,"num_object_copies":9465,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3155,"num_write_kb":201920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"76'79","reported_seq":441,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.167116+0000","last_change":"2024-09-21T12:27:19.888983+0000","last_active":"2024-09-21T12:28:49.167116+0000","last_peered":"2024-09-21T12:28:49.167116+0000","last_clean":"2024-09-21T12:28:49.167116+0000","last_became_active":"2024-09-21T12:27:14.954642+0000","last_became_peered":"2024-09-21T12:27:14.954642+0000","last_unstale":"2024-09-21T12:28:49.167116+0000","last_undegraded":"2024-09-21T12:28:49.167116+0000","last_fullsized":"2024-09-21T12:28:49.167116+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:19.888914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:19.888914+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":452,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.947062+0000","last_change":"2024-09-21T12:26:14.834328+0000","last_active":"2024-09-21T12:28:51.947062+0000","last_peered":"2024-09-21T12:28:51.947062+0000","last_clean":"2024-09-21T12:28:51.947062+0000","last_became_active":"2024-09-21T12:20:17.693067+0000","last_became_peered":"2024-09-21T12:20:17.693067+0000","last_unstale":"2024-09-21T12:28:51.947062+0000","last_undegraded":"2024-09-21T12:28:51.947062+0000","last_fullsized":"2024-09-21T12:28:51.947062+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-21T12:26:14.834236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:14.834236+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:15.646442+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":410,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.969046+0000","last_change":"2024-09-21T12:26:12.131784+0000","last_active":"2024-09-21T12:28:51.969046+0000","last_peered":"2024-09-21T12:28:51.969046+0000","last_clean":"2024-09-21T12:28:51.969046+0000","last_became_active":"2024-09-21T12:20:15.308441+0000","last_became_peered":"2024-09-21T12:20:15.308441+0000","last_unstale":"2024-09-21T12:28:51.969046+0000","last_undegraded":"2024-09-21T12:28:51.969046+0000","last_fullsized":"2024-09-21T12:28:51.969046+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:12.131721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:12.131721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:52.566222+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":416,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.166131+0000","last_change":"2024-09-21T12:28:48.999950+0000","last_active":"2024-09-21T12:28:49.166131+0000","last_peered":"2024-09-21T12:28:49.166131+0000","last_clean":"2024-09-21T12:28:49.166131+0000","last_became_active":"2024-09-21T12:20:19.309598+0000","last_became_peered":"2024-09-21T12:20:19.309598+0000","last_unstale":"2024-09-21T12:28:49.166131+0000","last_undegraded":"2024-09-21T12:28:49.166131+0000","last_fullsized":"2024-09-21T12:28:49.166131+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:48.999878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:48.999878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:15.295060+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":398,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.967988+0000","last_change":"2024-09-21T12:25:44.918356+0000","last_active":"2024-09-21T12:28:51.967988+0000","last_peered":"2024-09-21T12:28:51.967988+0000","last_clean":"2024-09-21T12:28:51.967988+0000","last_became_active":"2024-09-21T12:20:21.592828+0000","last_became_peered":"2024-09-21T12:20:21.592828+0000","last_unstale":"2024-09-21T12:28:51.967988+0000","last_undegraded":"2024-09-21T12:28:51.967988+0000","last_fullsized":"2024-09-21T12:28:51.967988+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:44.918276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:44.918276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:23.575759+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"253'3173","reported_seq":3218,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:52.379190+0000","last_change":"2024-09-21T12:28:20.952252+0000","last_active":"2024-09-21T12:28:52.379190+0000","last_peered":"2024-09-21T12:28:52.379190+0000","last_clean":"2024-09-21T12:28:52.379190+0000","last_became_active":"2024-09-21T12:28:20.951980+0000","last_became_peered":"2024-09-21T12:28:20.951980+0000","last_unstale":"2024-09-21T12:28:52.379190+0000","last_undegraded":"2024-09-21T12:28:52.379190+0000","last_fullsized":"2024-09-21T12:28:52.379190+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3173,"log_dups_size":0,"ondisk_log_size":3173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:48.108618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207945728,"num_objects":3173,"num_object_clones":0,"num_object_copies":9519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3173,"num_write_kb":203072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"76'79","reported_seq":209,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.952038+0000","last_change":"2024-09-21T12:28:25.714131+0000","last_active":"2024-09-21T12:28:51.952038+0000","last_peered":"2024-09-21T12:28:51.952038+0000","last_clean":"2024-09-21T12:28:51.952038+0000","last_became_active":"2024-09-21T12:26:54.092726+0000","last_became_peered":"2024-09-21T12:26:54.092726+0000","last_unstale":"2024-09-21T12:28:51.952038+0000","last_undegraded":"2024-09-21T12:28:51.952038+0000","last_fullsized":"2024-09-21T12:28:51.952038+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:25.714084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:25.714084+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:37.309436+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":558,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.968814+0000","last_change":"2024-09-21T12:26:09.082395+0000","last_active":"2024-09-21T12:28:51.968814+0000","last_peered":"2024-09-21T12:28:51.968814+0000","last_clean":"2024-09-21T12:28:51.968814+0000","last_became_active":"2024-09-21T12:20:17.307769+0000","last_became_peered":"2024-09-21T12:20:17.307769+0000","last_unstale":"2024-09-21T12:28:51.968814+0000","last_undegraded":"2024-09-21T12:28:51.968814+0000","last_fullsized":"2024-09-21T12:28:51.968814+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-21T12:26:09.082314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:09.082314+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":351,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.946822+0000","last_change":"2024-09-21T12:28:31.738463+0000","last_active":"2024-09-21T12:28:51.946822+0000","last_peered":"2024-09-21T12:28:51.946822+0000","last_clean":"2024-09-21T12:28:51.946822+0000","last_became_active":"2024-09-21T12:22:50.922002+0000","last_became_peered":"2024-09-21T12:22:50.922002+0000","last_unstale":"2024-09-21T12:28:51.946822+0000","last_undegraded":"2024-09-21T12:28:51.946822+0000","last_fullsized":"2024-09-21T12:28:51.946822+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:31.738415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:31.738415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:33.595522+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":135,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.967789+0000","last_change":"2024-09-21T12:28:37.691938+0000","last_active":"2024-09-21T12:28:51.967789+0000","last_peered":"2024-09-21T12:28:51.967789+0000","last_clean":"2024-09-21T12:28:51.967789+0000","last_became_active":"2024-09-21T12:26:54.083584+0000","last_became_peered":"2024-09-21T12:26:54.083584+0000","last_unstale":"2024-09-21T12:28:51.967789+0000","last_undegraded":"2024-09-21T12:28:51.967789+0000","last_fullsized":"2024-09-21T12:28:51.967789+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T12:28:37.691868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:28:37.691868+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:45.015722+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":441,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.946366+0000","last_change":"2024-09-21T12:27:21.956252+0000","last_active":"2024-09-21T12:28:51.946366+0000","last_peered":"2024-09-21T12:28:51.946366+0000","last_clean":"2024-09-21T12:28:51.946366+0000","last_became_active":"2024-09-21T12:22:50.921488+0000","last_became_peered":"2024-09-21T12:22:50.921488+0000","last_unstale":"2024-09-21T12:28:51.946366+0000","last_undegraded":"2024-09-21T12:28:51.946366+0000","last_fullsized":"2024-09-21T12:28:51.946366+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:21.956185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:27:21.956185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":425,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.940205+0000","last_change":"2024-09-21T12:26:10.435410+0000","last_active":"2024-09-21T12:28:51.940205+0000","last_peered":"2024-09-21T12:28:51.940205+0000","last_clean":"2024-09-21T12:28:51.940205+0000","last_became_active":"2024-09-21T12:22:50.917558+0000","last_became_peered":"2024-09-21T12:22:50.917558+0000","last_unstale":"2024-09-21T12:28:51.940205+0000","last_undegraded":"2024-09-21T12:28:51.940205+0000","last_fullsized":"2024-09-21T12:28:51.940205+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:10.435349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:10.435349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"252'3094","reported_seq":3136,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.678900+0000","last_change":"2024-09-21T12:28:20.953805+0000","last_active":"2024-09-21T12:28:51.678900+0000","last_peered":"2024-09-21T12:28:51.678900+0000","last_clean":"2024-09-21T12:28:51.678900+0000","last_became_active":"2024-09-21T12:28:20.952781+0000","last_became_peered":"2024-09-21T12:28:20.952781+0000","last_unstale":"2024-09-21T12:28:51.678900+0000","last_undegraded":"2024-09-21T12:28:51.678900+0000","last_fullsized":"2024-09-21T12:28:51.678900+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3094,"log_dups_size":0,"ondisk_log_size":3094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:45.666222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202768384,"num_objects":3094,"num_object_clones":0,"num_object_copies":9282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3094,"num_write_kb":198016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"76'79","reported_seq":512,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.154958+0000","last_change":"2024-09-21T12:26:46.188733+0000","last_active":"2024-09-21T12:28:49.154958+0000","last_peered":"2024-09-21T12:28:49.154958+0000","last_clean":"2024-09-21T12:28:49.154958+0000","last_became_active":"2024-09-21T12:26:44.956003+0000","last_became_peered":"2024-09-21T12:26:44.956003+0000","last_unstale":"2024-09-21T12:28:49.154958+0000","last_undegraded":"2024-09-21T12:28:49.154958+0000","last_fullsized":"2024-09-21T12:28:49.154958+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:46.188649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:46.188649+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":448,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.969113+0000","last_change":"2024-09-21T12:28:25.549378+0000","last_active":"2024-09-21T12:28:51.969113+0000","last_peered":"2024-09-21T12:28:51.969113+0000","last_clean":"2024-09-21T12:28:51.969113+0000","last_became_active":"2024-09-21T12:20:17.297374+0000","last_became_peered":"2024-09-21T12:20:17.297374+0000","last_unstale":"2024-09-21T12:28:51.969113+0000","last_undegraded":"2024-09-21T12:28:51.969113+0000","last_fullsized":"2024-09-21T12:28:51.969113+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-21T12:28:25.549320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:25.549320+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:32.588927+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":434,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.968607+0000","last_change":"2024-09-21T12:28:39.705480+0000","last_active":"2024-09-21T12:28:51.968607+0000","last_peered":"2024-09-21T12:28:51.968607+0000","last_clean":"2024-09-21T12:28:51.968607+0000","last_became_active":"2024-09-21T12:20:15.308356+0000","last_became_peered":"2024-09-21T12:20:15.308356+0000","last_unstale":"2024-09-21T12:28:51.968607+0000","last_undegraded":"2024-09-21T12:28:51.968607+0000","last_fullsized":"2024-09-21T12:28:51.968607+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:39.705175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:39.705175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:54.221915+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":444,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.963945+0000","last_change":"2024-09-21T12:28:43.390160+0000","last_active":"2024-09-21T12:28:51.963945+0000","last_peered":"2024-09-21T12:28:51.963945+0000","last_clean":"2024-09-21T12:28:51.963945+0000","last_became_active":"2024-09-21T12:20:14.288358+0000","last_became_peered":"2024-09-21T12:20:14.288358+0000","last_unstale":"2024-09-21T12:28:51.963945+0000","last_undegraded":"2024-09-21T12:28:51.963945+0000","last_fullsized":"2024-09-21T12:28:51.963945+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:28:43.390087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:28:43.390087+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:12.445374+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0019042449999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":418,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.964506+0000","last_change":"2024-09-21T12:27:33.447579+0000","last_active":"2024-09-21T12:28:51.964506+0000","last_peered":"2024-09-21T12:28:51.964506+0000","last_clean":"2024-09-21T12:28:51.964506+0000","last_became_active":"2024-09-21T12:20:19.309724+0000","last_became_peered":"2024-09-21T12:20:19.309724+0000","last_unstale":"2024-09-21T12:28:51.964506+0000","last_undegraded":"2024-09-21T12:28:51.964506+0000","last_fullsized":"2024-09-21T12:28:51.964506+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:33.447236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:27:33.447236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:02.627972+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":396,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.155665+0000","last_change":"2024-09-21T12:27:33.395611+0000","last_active":"2024-09-21T12:28:49.155665+0000","last_peered":"2024-09-21T12:28:49.155665+0000","last_clean":"2024-09-21T12:28:49.155665+0000","last_became_active":"2024-09-21T12:20:21.592982+0000","last_became_peered":"2024-09-21T12:20:21.592982+0000","last_unstale":"2024-09-21T12:28:49.155665+0000","last_undegraded":"2024-09-21T12:28:49.155665+0000","last_fullsized":"2024-09-21T12:28:49.155665+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:33.395568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:27:33.395568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"253'3292","reported_seq":3337,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:52.375295+0000","last_change":"2024-09-21T12:28:21.114897+0000","last_active":"2024-09-21T12:28:52.375295+0000","last_peered":"2024-09-21T12:28:52.375295+0000","last_clean":"2024-09-21T12:28:52.375295+0000","last_became_active":"2024-09-21T12:28:21.114609+0000","last_became_peered":"2024-09-21T12:28:21.114609+0000","last_unstale":"2024-09-21T12:28:52.375295+0000","last_undegraded":"2024-09-21T12:28:52.375295+0000","last_fullsized":"2024-09-21T12:28:52.375295+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3292,"log_dups_size":0,"ondisk_log_size":3292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:31.696957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215744512,"num_objects":3292,"num_object_clones":0,"num_object_copies":9876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3292,"num_write_kb":210688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"76'79","reported_seq":360,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.968597+0000","last_change":"2024-09-21T12:28:01.638945+0000","last_active":"2024-09-21T12:28:51.968597+0000","last_peered":"2024-09-21T12:28:51.968597+0000","last_clean":"2024-09-21T12:28:51.968597+0000","last_became_active":"2024-09-21T12:26:44.949867+0000","last_became_peered":"2024-09-21T12:26:44.949867+0000","last_unstale":"2024-09-21T12:28:51.968597+0000","last_undegraded":"2024-09-21T12:28:51.968597+0000","last_fullsized":"2024-09-21T12:28:51.968597+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:01.638886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:01.638886+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:04.265728+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":441,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.951001+0000","last_change":"2024-09-21T12:28:45.783033+0000","last_active":"2024-09-21T12:28:51.951001+0000","last_peered":"2024-09-21T12:28:51.951001+0000","last_clean":"2024-09-21T12:28:51.951001+0000","last_became_active":"2024-09-21T12:20:17.299325+0000","last_became_peered":"2024-09-21T12:20:17.299325+0000","last_unstale":"2024-09-21T12:28:51.951001+0000","last_undegraded":"2024-09-21T12:28:51.951001+0000","last_fullsized":"2024-09-21T12:28:51.951001+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:28:45.782955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.782955+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:03.191455+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":441,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.945309+0000","last_change":"2024-09-21T12:24:48.073954+0000","last_active":"2024-09-21T12:28:51.945309+0000","last_peered":"2024-09-21T12:28:51.945309+0000","last_clean":"2024-09-21T12:28:51.945309+0000","last_became_active":"2024-09-21T12:22:50.923796+0000","last_became_peered":"2024-09-21T12:22:50.923796+0000","last_unstale":"2024-09-21T12:28:51.945309+0000","last_undegraded":"2024-09-21T12:28:51.945309+0000","last_fullsized":"2024-09-21T12:28:51.945309+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:48.073424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:48.073424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:12.742306+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":946,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:50.617075+0000","last_change":"2024-09-21T12:28:34.639438+0000","last_active":"2024-09-21T12:28:50.617075+0000","last_peered":"2024-09-21T12:28:50.617075+0000","last_clean":"2024-09-21T12:28:50.617075+0000","last_became_active":"2024-09-21T12:20:14.287543+0000","last_became_peered":"2024-09-21T12:20:14.287543+0000","last_unstale":"2024-09-21T12:28:50.617075+0000","last_undegraded":"2024-09-21T12:28:50.617075+0000","last_fullsized":"2024-09-21T12:28:50.617075+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-21T12:28:34.639361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:28:34.639361+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:48.713397+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":511,"num_read_kb":506,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":407,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.153023+0000","last_change":"2024-09-21T12:26:11.422340+0000","last_active":"2024-09-21T12:28:49.153023+0000","last_peered":"2024-09-21T12:28:49.153023+0000","last_clean":"2024-09-21T12:28:49.153023+0000","last_became_active":"2024-09-21T12:20:19.314217+0000","last_became_peered":"2024-09-21T12:20:19.314217+0000","last_unstale":"2024-09-21T12:28:49.153023+0000","last_undegraded":"2024-09-21T12:28:49.153023+0000","last_fullsized":"2024-09-21T12:28:49.153023+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:11.422293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:11.422293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:18.396029+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":414,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.968382+0000","last_change":"2024-09-21T12:25:38.882984+0000","last_active":"2024-09-21T12:28:51.968382+0000","last_peered":"2024-09-21T12:28:51.968382+0000","last_clean":"2024-09-21T12:28:51.968382+0000","last_became_active":"2024-09-21T12:20:21.311188+0000","last_became_peered":"2024-09-21T12:20:21.311188+0000","last_unstale":"2024-09-21T12:28:51.968382+0000","last_undegraded":"2024-09-21T12:28:51.968382+0000","last_fullsized":"2024-09-21T12:28:51.968382+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:38.882915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:38.882915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:21.632466+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"253'3161","reported_seq":3205,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:52.315820+0000","last_change":"2024-09-21T12:28:20.957134+0000","last_active":"2024-09-21T12:28:52.315820+0000","last_peered":"2024-09-21T12:28:52.315820+0000","last_clean":"2024-09-21T12:28:52.315820+0000","last_became_active":"2024-09-21T12:28:20.956557+0000","last_became_peered":"2024-09-21T12:28:20.956557+0000","last_unstale":"2024-09-21T12:28:52.315820+0000","last_undegraded":"2024-09-21T12:28:52.315820+0000","last_fullsized":"2024-09-21T12:28:52.315820+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3161,"log_dups_size":0,"ondisk_log_size":3161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:31.714770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207159296,"num_objects":3161,"num_object_clones":0,"num_object_copies":9483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3161,"num_write_kb":202304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"121'19","reported_seq":613,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.166972+0000","last_change":"2024-09-21T12:24:27.932976+0000","last_active":"2024-09-21T12:28:49.166972+0000","last_peered":"2024-09-21T12:28:49.166972+0000","last_clean":"2024-09-21T12:28:49.166972+0000","last_became_active":"2024-09-21T12:22:50.921690+0000","last_became_peered":"2024-09-21T12:22:50.921690+0000","last_unstale":"2024-09-21T12:28:49.166972+0000","last_undegraded":"2024-09-21T12:28:49.166972+0000","last_fullsized":"2024-09-21T12:28:49.166972+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'19","last_scrub_stamp":"2024-09-21T12:24:27.932891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:27.932891+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:11.225971+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":423,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.939696+0000","last_change":"2024-09-21T12:26:12.394327+0000","last_active":"2024-09-21T12:28:51.939696+0000","last_peered":"2024-09-21T12:28:51.939696+0000","last_clean":"2024-09-21T12:28:51.939696+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:28:51.939696+0000","last_undegraded":"2024-09-21T12:28:51.939696+0000","last_fullsized":"2024-09-21T12:28:51.939696+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:26:12.394247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:12.394247+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":500,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.154727+0000","last_change":"2024-09-21T12:26:49.248015+0000","last_active":"2024-09-21T12:28:49.154727+0000","last_peered":"2024-09-21T12:28:49.154727+0000","last_clean":"2024-09-21T12:28:49.154727+0000","last_became_active":"2024-09-21T12:26:44.950132+0000","last_became_peered":"2024-09-21T12:26:44.950132+0000","last_unstale":"2024-09-21T12:28:49.154727+0000","last_undegraded":"2024-09-21T12:28:49.154727+0000","last_fullsized":"2024-09-21T12:28:49.154727+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:49.247952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:49.247952+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":400,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.154903+0000","last_change":"2024-09-21T12:28:36.684525+0000","last_active":"2024-09-21T12:28:49.154903+0000","last_peered":"2024-09-21T12:28:49.154903+0000","last_clean":"2024-09-21T12:28:49.154903+0000","last_became_active":"2024-09-21T12:20:19.309859+0000","last_became_peered":"2024-09-21T12:20:19.309859+0000","last_unstale":"2024-09-21T12:28:49.154903+0000","last_undegraded":"2024-09-21T12:28:49.154903+0000","last_fullsized":"2024-09-21T12:28:49.154903+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:36.684459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:36.684459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:43.653752+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":251,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.941639+0000","last_change":"2024-09-21T12:28:28.647344+0000","last_active":"2024-09-21T12:28:51.941639+0000","last_peered":"2024-09-21T12:28:51.941639+0000","last_clean":"2024-09-21T12:28:51.941639+0000","last_became_active":"2024-09-21T12:26:54.084914+0000","last_became_peered":"2024-09-21T12:26:54.084914+0000","last_unstale":"2024-09-21T12:28:51.941639+0000","last_undegraded":"2024-09-21T12:28:51.941639+0000","last_fullsized":"2024-09-21T12:28:51.941639+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:28.647287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:28.647287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:44.468892+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3","version":"252'3125","reported_seq":3167,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.644545+0000","last_change":"2024-09-21T12:28:20.958439+0000","last_active":"2024-09-21T12:28:51.644545+0000","last_peered":"2024-09-21T12:28:51.644545+0000","last_clean":"2024-09-21T12:28:51.644545+0000","last_became_active":"2024-09-21T12:28:20.958025+0000","last_became_peered":"2024-09-21T12:28:20.958025+0000","last_unstale":"2024-09-21T12:28:51.644545+0000","last_undegraded":"2024-09-21T12:28:51.644545+0000","last_fullsized":"2024-09-21T12:28:51.644545+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3125,"log_dups_size":0,"ondisk_log_size":3125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:33.887095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204800000,"num_objects":3125,"num_object_clones":0,"num_object_copies":9375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3125,"num_write_kb":200000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"76'79","reported_seq":363,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.969237+0000","last_change":"2024-09-21T12:27:14.954671+0000","last_active":"2024-09-21T12:28:51.969237+0000","last_peered":"2024-09-21T12:28:51.969237+0000","last_clean":"2024-09-21T12:28:51.969237+0000","last_became_active":"2024-09-21T12:27:14.954351+0000","last_became_peered":"2024-09-21T12:27:14.954351+0000","last_unstale":"2024-09-21T12:28:51.969237+0000","last_undegraded":"2024-09-21T12:28:51.969237+0000","last_fullsized":"2024-09-21T12:28:51.969237+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:48.188322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:48.188322+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:29:17.673456+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":430,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.167263+0000","last_change":"2024-09-21T12:27:57.114150+0000","last_active":"2024-09-21T12:28:49.167263+0000","last_peered":"2024-09-21T12:28:49.167263+0000","last_clean":"2024-09-21T12:28:49.167263+0000","last_became_active":"2024-09-21T12:23:46.885802+0000","last_became_peered":"2024-09-21T12:23:46.885802+0000","last_unstale":"2024-09-21T12:28:49.167263+0000","last_undegraded":"2024-09-21T12:28:49.167263+0000","last_fullsized":"2024-09-21T12:28:49.167263+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:27:57.114076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:57.114076+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:19.583144+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":342,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.963784+0000","last_change":"2024-09-21T12:27:30.501480+0000","last_active":"2024-09-21T12:28:51.963784+0000","last_peered":"2024-09-21T12:28:51.963784+0000","last_clean":"2024-09-21T12:28:51.963784+0000","last_became_active":"2024-09-21T12:22:50.908060+0000","last_became_peered":"2024-09-21T12:22:50.908060+0000","last_unstale":"2024-09-21T12:28:51.963784+0000","last_undegraded":"2024-09-21T12:28:51.963784+0000","last_fullsized":"2024-09-21T12:28:51.963784+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:30.498831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:27:30.498831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":253,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.167405+0000","last_change":"2024-09-21T12:28:05.266905+0000","last_active":"2024-09-21T12:28:49.167405+0000","last_peered":"2024-09-21T12:28:49.167405+0000","last_clean":"2024-09-21T12:28:49.167405+0000","last_became_active":"2024-09-21T12:26:54.089839+0000","last_became_peered":"2024-09-21T12:26:54.089839+0000","last_unstale":"2024-09-21T12:28:49.167405+0000","last_undegraded":"2024-09-21T12:28:49.167405+0000","last_fullsized":"2024-09-21T12:28:49.167405+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:05.266842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:05.266842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:08.140709+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":404,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.943238+0000","last_change":"2024-09-21T12:28:39.853931+0000","last_active":"2024-09-21T12:28:51.943238+0000","last_peered":"2024-09-21T12:28:51.943238+0000","last_clean":"2024-09-21T12:28:51.943238+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:28:51.943238+0000","last_undegraded":"2024-09-21T12:28:51.943238+0000","last_fullsized":"2024-09-21T12:28:51.943238+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:39.853791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:39.853791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:04.491080+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"253'3119","reported_seq":3163,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.992491+0000","last_change":"2024-09-21T12:28:21.114546+0000","last_active":"2024-09-21T12:28:51.992491+0000","last_peered":"2024-09-21T12:28:51.992491+0000","last_clean":"2024-09-21T12:28:51.992491+0000","last_became_active":"2024-09-21T12:28:21.114218+0000","last_became_peered":"2024-09-21T12:28:21.114218+0000","last_unstale":"2024-09-21T12:28:51.992491+0000","last_undegraded":"2024-09-21T12:28:51.992491+0000","last_fullsized":"2024-09-21T12:28:51.992491+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3119,"log_dups_size":0,"ondisk_log_size":3119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:31.328174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204406784,"num_objects":3119,"num_object_clones":0,"num_object_copies":9357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3119,"num_write_kb":199616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"76'79","reported_seq":119,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.963729+0000","last_change":"2024-09-21T12:27:14.942259+0000","last_active":"2024-09-21T12:28:51.963729+0000","last_peered":"2024-09-21T12:28:51.963729+0000","last_clean":"2024-09-21T12:28:51.963729+0000","last_became_active":"2024-09-21T12:27:14.941888+0000","last_became_peered":"2024-09-21T12:27:14.941888+0000","last_unstale":"2024-09-21T12:28:51.963729+0000","last_undegraded":"2024-09-21T12:28:51.963729+0000","last_fullsized":"2024-09-21T12:28:51.963729+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:57.680181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:57.680181+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:29:17.469205+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131344,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":475,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.963982+0000","last_change":"2024-09-21T12:28:46.355910+0000","last_active":"2024-09-21T12:28:51.963982+0000","last_peered":"2024-09-21T12:28:51.963982+0000","last_clean":"2024-09-21T12:28:51.963982+0000","last_became_active":"2024-09-21T12:23:46.897197+0000","last_became_peered":"2024-09-21T12:23:46.897197+0000","last_unstale":"2024-09-21T12:28:51.963982+0000","last_undegraded":"2024-09-21T12:28:51.963982+0000","last_fullsized":"2024-09-21T12:28:51.963982+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-21T12:28:46.355822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:46.355822+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:54.101822+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":452,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.166380+0000","last_change":"2024-09-21T12:28:06.236794+0000","last_active":"2024-09-21T12:28:49.166380+0000","last_peered":"2024-09-21T12:28:49.166380+0000","last_clean":"2024-09-21T12:28:49.166380+0000","last_became_active":"2024-09-21T12:22:50.906886+0000","last_became_peered":"2024-09-21T12:22:50.906886+0000","last_unstale":"2024-09-21T12:28:49.166380+0000","last_undegraded":"2024-09-21T12:28:49.166380+0000","last_fullsized":"2024-09-21T12:28:49.166380+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:06.236749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:06.236749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:29.504064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":430,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.968471+0000","last_change":"2024-09-21T12:28:03.571042+0000","last_active":"2024-09-21T12:28:51.968471+0000","last_peered":"2024-09-21T12:28:51.968471+0000","last_clean":"2024-09-21T12:28:51.968471+0000","last_became_active":"2024-09-21T12:22:50.927438+0000","last_became_peered":"2024-09-21T12:22:50.927438+0000","last_unstale":"2024-09-21T12:28:51.968471+0000","last_undegraded":"2024-09-21T12:28:51.968471+0000","last_fullsized":"2024-09-21T12:28:51.968471+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:03.570984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:03.570984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:18.975167+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":419,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.963483+0000","last_change":"2024-09-21T12:25:39.079146+0000","last_active":"2024-09-21T12:28:51.963483+0000","last_peered":"2024-09-21T12:28:51.963483+0000","last_clean":"2024-09-21T12:28:51.963483+0000","last_became_active":"2024-09-21T12:22:50.913898+0000","last_became_peered":"2024-09-21T12:22:50.913898+0000","last_unstale":"2024-09-21T12:28:51.963483+0000","last_undegraded":"2024-09-21T12:28:51.963483+0000","last_fullsized":"2024-09-21T12:28:51.963483+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:39.079015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:39.079015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:15.448755+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"253'3121","reported_seq":3164,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:52.319463+0000","last_change":"2024-09-21T12:28:21.114763+0000","last_active":"2024-09-21T12:28:52.319463+0000","last_peered":"2024-09-21T12:28:52.319463+0000","last_clean":"2024-09-21T12:28:52.319463+0000","last_became_active":"2024-09-21T12:28:21.114432+0000","last_became_peered":"2024-09-21T12:28:21.114432+0000","last_unstale":"2024-09-21T12:28:52.319463+0000","last_undegraded":"2024-09-21T12:28:52.319463+0000","last_fullsized":"2024-09-21T12:28:52.319463+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3121,"log_dups_size":0,"ondisk_log_size":3121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:43.773638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204537856,"num_objects":3121,"num_object_clones":0,"num_object_copies":9363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3121,"num_write_kb":199744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"76'79","reported_seq":490,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.155081+0000","last_change":"2024-09-21T12:27:17.370605+0000","last_active":"2024-09-21T12:28:49.155081+0000","last_peered":"2024-09-21T12:28:49.155081+0000","last_clean":"2024-09-21T12:28:49.155081+0000","last_became_active":"2024-09-21T12:27:14.943705+0000","last_became_peered":"2024-09-21T12:27:14.943705+0000","last_unstale":"2024-09-21T12:28:49.155081+0000","last_undegraded":"2024-09-21T12:28:49.155081+0000","last_fullsized":"2024-09-21T12:28:49.155081+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:17.370530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:17.370530+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":281,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.964453+0000","last_change":"2024-09-21T12:27:34.477082+0000","last_active":"2024-09-21T12:28:51.964453+0000","last_peered":"2024-09-21T12:28:51.964453+0000","last_clean":"2024-09-21T12:28:51.964453+0000","last_became_active":"2024-09-21T12:23:46.902260+0000","last_became_peered":"2024-09-21T12:23:46.902260+0000","last_unstale":"2024-09-21T12:28:51.964453+0000","last_undegraded":"2024-09-21T12:28:51.964453+0000","last_fullsized":"2024-09-21T12:28:51.964453+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:27:34.476998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:34.476998+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:04.288014+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":437,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.940542+0000","last_change":"2024-09-21T12:27:32.433938+0000","last_active":"2024-09-21T12:28:51.940542+0000","last_peered":"2024-09-21T12:28:51.940542+0000","last_clean":"2024-09-21T12:28:51.940542+0000","last_became_active":"2024-09-21T12:22:50.927946+0000","last_became_peered":"2024-09-21T12:22:50.927946+0000","last_unstale":"2024-09-21T12:28:51.940542+0000","last_undegraded":"2024-09-21T12:28:51.940542+0000","last_fullsized":"2024-09-21T12:28:51.940542+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:32.433867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:27:32.433867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":538,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.963871+0000","last_change":"2024-09-21T12:28:39.418167+0000","last_active":"2024-09-21T12:28:51.963871+0000","last_peered":"2024-09-21T12:28:51.963871+0000","last_clean":"2024-09-21T12:28:51.963871+0000","last_became_active":"2024-09-21T12:20:19.307924+0000","last_became_peered":"2024-09-21T12:20:19.307924+0000","last_unstale":"2024-09-21T12:28:51.963871+0000","last_undegraded":"2024-09-21T12:28:51.963871+0000","last_fullsized":"2024-09-21T12:28:51.963871+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:28:39.418085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:39.418085+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:08.013783+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":349,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.972230+0000","last_change":"2024-09-21T12:28:35.670078+0000","last_active":"2024-09-21T12:28:51.972230+0000","last_peered":"2024-09-21T12:28:51.972230+0000","last_clean":"2024-09-21T12:28:51.972230+0000","last_became_active":"2024-09-21T12:22:50.905470+0000","last_became_peered":"2024-09-21T12:22:50.905470+0000","last_unstale":"2024-09-21T12:28:51.972230+0000","last_undegraded":"2024-09-21T12:28:51.972230+0000","last_fullsized":"2024-09-21T12:28:51.972230+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:35.669942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:35.669942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:01.370279+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"252'3174","reported_seq":3219,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.964295+0000","last_change":"2024-09-21T12:28:20.952349+0000","last_active":"2024-09-21T12:28:51.964295+0000","last_peered":"2024-09-21T12:28:51.964295+0000","last_clean":"2024-09-21T12:28:51.964295+0000","last_became_active":"2024-09-21T12:28:20.952083+0000","last_became_peered":"2024-09-21T12:28:20.952083+0000","last_unstale":"2024-09-21T12:28:51.964295+0000","last_undegraded":"2024-09-21T12:28:51.964295+0000","last_fullsized":"2024-09-21T12:28:51.964295+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3174,"log_dups_size":0,"ondisk_log_size":3174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:45.753435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208011264,"num_objects":3174,"num_object_clones":0,"num_object_copies":9522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3174,"num_write_kb":203136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"76'79","reported_seq":128,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.154983+0000","last_change":"2024-09-21T12:27:21.345414+0000","last_active":"2024-09-21T12:28:49.154983+0000","last_peered":"2024-09-21T12:28:49.154983+0000","last_clean":"2024-09-21T12:28:49.154983+0000","last_became_active":"2024-09-21T12:27:14.943330+0000","last_became_peered":"2024-09-21T12:27:14.943330+0000","last_unstale":"2024-09-21T12:28:49.154983+0000","last_undegraded":"2024-09-21T12:28:49.154983+0000","last_fullsized":"2024-09-21T12:28:49.154983+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:21.345364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:21.345364+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":490,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.947149+0000","last_change":"2024-09-21T12:25:00.237083+0000","last_active":"2024-09-21T12:28:51.947149+0000","last_peered":"2024-09-21T12:28:51.947149+0000","last_clean":"2024-09-21T12:28:51.947149+0000","last_became_active":"2024-09-21T12:22:50.922343+0000","last_became_peered":"2024-09-21T12:22:50.922343+0000","last_unstale":"2024-09-21T12:28:51.947149+0000","last_undegraded":"2024-09-21T12:28:51.947149+0000","last_fullsized":"2024-09-21T12:28:51.947149+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:25:00.236990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:00.236990+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:52.946058+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":100,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.943864+0000","last_change":"2024-09-21T12:26:54.084563+0000","last_active":"2024-09-21T12:28:51.943864+0000","last_peered":"2024-09-21T12:28:51.943864+0000","last_clean":"2024-09-21T12:28:51.943864+0000","last_became_active":"2024-09-21T12:26:54.084228+0000","last_became_peered":"2024-09-21T12:26:54.084228+0000","last_unstale":"2024-09-21T12:28:51.943864+0000","last_undegraded":"2024-09-21T12:28:51.943864+0000","last_fullsized":"2024-09-21T12:28:51.943864+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:30.451876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:30.451876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:25.674786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":447,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.967091+0000","last_change":"2024-09-21T12:28:47.361181+0000","last_active":"2024-09-21T12:28:51.967091+0000","last_peered":"2024-09-21T12:28:51.967091+0000","last_clean":"2024-09-21T12:28:51.967091+0000","last_became_active":"2024-09-21T12:22:50.925974+0000","last_became_peered":"2024-09-21T12:22:50.925974+0000","last_unstale":"2024-09-21T12:28:51.967091+0000","last_undegraded":"2024-09-21T12:28:51.967091+0000","last_fullsized":"2024-09-21T12:28:51.967091+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:47.361115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:47.361115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:57.938678+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":407,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.945395+0000","last_change":"2024-09-21T12:28:43.821102+0000","last_active":"2024-09-21T12:28:51.945395+0000","last_peered":"2024-09-21T12:28:51.945395+0000","last_clean":"2024-09-21T12:28:51.945395+0000","last_became_active":"2024-09-21T12:20:21.311493+0000","last_became_peered":"2024-09-21T12:20:21.311493+0000","last_unstale":"2024-09-21T12:28:51.945395+0000","last_undegraded":"2024-09-21T12:28:51.945395+0000","last_fullsized":"2024-09-21T12:28:51.945395+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:43.820979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:43.820979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:48.567558+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"252'3078","reported_seq":3120,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.627220+0000","last_change":"2024-09-21T12:28:20.958504+0000","last_active":"2024-09-21T12:28:51.627220+0000","last_peered":"2024-09-21T12:28:51.627220+0000","last_clean":"2024-09-21T12:28:51.627220+0000","last_became_active":"2024-09-21T12:28:20.958112+0000","last_became_peered":"2024-09-21T12:28:20.958112+0000","last_unstale":"2024-09-21T12:28:51.627220+0000","last_undegraded":"2024-09-21T12:28:51.627220+0000","last_fullsized":"2024-09-21T12:28:51.627220+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3078,"log_dups_size":0,"ondisk_log_size":3078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:27.450929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201719808,"num_objects":3078,"num_object_clones":0,"num_object_copies":9234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3078,"num_write_kb":196992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"76'79","reported_seq":432,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.167482+0000","last_change":"2024-09-21T12:26:55.832199+0000","last_active":"2024-09-21T12:28:49.167482+0000","last_peered":"2024-09-21T12:28:49.167482+0000","last_clean":"2024-09-21T12:28:49.167482+0000","last_became_active":"2024-09-21T12:26:46.986203+0000","last_became_peered":"2024-09-21T12:26:46.986203+0000","last_unstale":"2024-09-21T12:28:49.167482+0000","last_undegraded":"2024-09-21T12:28:49.167482+0000","last_fullsized":"2024-09-21T12:28:49.167482+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:55.832132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:55.832132+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:16.157831+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":450,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.166206+0000","last_change":"2024-09-21T12:27:30.898787+0000","last_active":"2024-09-21T12:28:49.166206+0000","last_peered":"2024-09-21T12:28:49.166206+0000","last_clean":"2024-09-21T12:28:49.166206+0000","last_became_active":"2024-09-21T12:20:17.692867+0000","last_became_peered":"2024-09-21T12:20:17.692867+0000","last_unstale":"2024-09-21T12:28:49.166206+0000","last_undegraded":"2024-09-21T12:28:49.166206+0000","last_fullsized":"2024-09-21T12:28:49.166206+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-21T12:27:30.898706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:30.898706+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":415,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.155847+0000","last_change":"2024-09-21T12:28:48.438834+0000","last_active":"2024-09-21T12:28:49.155847+0000","last_peered":"2024-09-21T12:28:49.155847+0000","last_clean":"2024-09-21T12:28:49.155847+0000","last_became_active":"2024-09-21T12:20:15.299333+0000","last_became_peered":"2024-09-21T12:20:15.299333+0000","last_unstale":"2024-09-21T12:28:49.155847+0000","last_undegraded":"2024-09-21T12:28:49.155847+0000","last_fullsized":"2024-09-21T12:28:49.155847+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:48.438755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:48.438755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:17.800439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":410,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.964233+0000","last_change":"2024-09-21T12:26:50.626932+0000","last_active":"2024-09-21T12:28:51.964233+0000","last_peered":"2024-09-21T12:28:51.964233+0000","last_clean":"2024-09-21T12:28:51.964233+0000","last_became_active":"2024-09-21T12:20:19.306952+0000","last_became_peered":"2024-09-21T12:20:19.306952+0000","last_unstale":"2024-09-21T12:28:51.964233+0000","last_undegraded":"2024-09-21T12:28:51.964233+0000","last_fullsized":"2024-09-21T12:28:51.964233+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:50.626780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:50.626780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:18.506880+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":387,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.154644+0000","last_change":"2024-09-21T12:20:21.314891+0000","last_active":"2024-09-21T12:28:49.154644+0000","last_peered":"2024-09-21T12:28:49.154644+0000","last_clean":"2024-09-21T12:28:49.154644+0000","last_became_active":"2024-09-21T12:20:21.314545+0000","last_became_peered":"2024-09-21T12:20:21.314545+0000","last_unstale":"2024-09-21T12:28:49.154644+0000","last_undegraded":"2024-09-21T12:28:49.154644+0000","last_fullsized":"2024-09-21T12:28:49.154644+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:09.634903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"253'3183","reported_seq":3227,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.999776+0000","last_change":"2024-09-21T12:28:20.955882+0000","last_active":"2024-09-21T12:28:51.999776+0000","last_peered":"2024-09-21T12:28:51.999776+0000","last_clean":"2024-09-21T12:28:51.999776+0000","last_became_active":"2024-09-21T12:28:20.955605+0000","last_became_peered":"2024-09-21T12:28:20.955605+0000","last_unstale":"2024-09-21T12:28:51.999776+0000","last_undegraded":"2024-09-21T12:28:51.999776+0000","last_fullsized":"2024-09-21T12:28:51.999776+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3183,"log_dups_size":0,"ondisk_log_size":3183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:37.225243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208601088,"num_objects":3183,"num_object_clones":0,"num_object_copies":9549,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3183,"num_write_kb":203712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"76'79","reported_seq":370,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.943599+0000","last_change":"2024-09-21T12:27:17.158443+0000","last_active":"2024-09-21T12:28:51.943599+0000","last_peered":"2024-09-21T12:28:51.943599+0000","last_clean":"2024-09-21T12:28:51.943599+0000","last_became_active":"2024-09-21T12:27:14.951375+0000","last_became_peered":"2024-09-21T12:27:14.951375+0000","last_unstale":"2024-09-21T12:28:51.943599+0000","last_undegraded":"2024-09-21T12:28:51.943599+0000","last_fullsized":"2024-09-21T12:28:51.943599+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:17.158380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:17.158380+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":424,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.968243+0000","last_change":"2024-09-21T12:28:40.694225+0000","last_active":"2024-09-21T12:28:51.968243+0000","last_peered":"2024-09-21T12:28:51.968243+0000","last_clean":"2024-09-21T12:28:51.968243+0000","last_became_active":"2024-09-21T12:22:50.910973+0000","last_became_peered":"2024-09-21T12:22:50.910973+0000","last_unstale":"2024-09-21T12:28:51.968243+0000","last_undegraded":"2024-09-21T12:28:51.968243+0000","last_fullsized":"2024-09-21T12:28:51.968243+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:28:40.694138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:40.694138+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:56.646482+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":444,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.155496+0000","last_change":"2024-09-21T12:28:30.612213+0000","last_active":"2024-09-21T12:28:49.155496+0000","last_peered":"2024-09-21T12:28:49.155496+0000","last_clean":"2024-09-21T12:28:49.155496+0000","last_became_active":"2024-09-21T12:22:50.927859+0000","last_became_peered":"2024-09-21T12:22:50.927859+0000","last_unstale":"2024-09-21T12:28:49.155496+0000","last_undegraded":"2024-09-21T12:28:49.155496+0000","last_fullsized":"2024-09-21T12:28:49.155496+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:30.612156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:30.612156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:55.060481+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":418,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.963447+0000","last_change":"2024-09-21T12:27:32.457547+0000","last_active":"2024-09-21T12:28:51.963447+0000","last_peered":"2024-09-21T12:28:51.963447+0000","last_clean":"2024-09-21T12:28:51.963447+0000","last_became_active":"2024-09-21T12:20:19.312572+0000","last_became_peered":"2024-09-21T12:20:19.312572+0000","last_unstale":"2024-09-21T12:28:51.963447+0000","last_undegraded":"2024-09-21T12:28:51.963447+0000","last_fullsized":"2024-09-21T12:28:51.963447+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:32.457484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:27:32.457484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:01.778622+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":404,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.940003+0000","last_change":"2024-09-21T12:26:58.321950+0000","last_active":"2024-09-21T12:28:51.940003+0000","last_peered":"2024-09-21T12:28:51.940003+0000","last_clean":"2024-09-21T12:28:51.940003+0000","last_became_active":"2024-09-21T12:20:21.317223+0000","last_became_peered":"2024-09-21T12:20:21.317223+0000","last_unstale":"2024-09-21T12:28:51.940003+0000","last_undegraded":"2024-09-21T12:28:51.940003+0000","last_fullsized":"2024-09-21T12:28:51.940003+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:58.321890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:58.321890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"253'3164","reported_seq":3209,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:52.380877+0000","last_change":"2024-09-21T12:28:20.951697+0000","last_active":"2024-09-21T12:28:52.380877+0000","last_peered":"2024-09-21T12:28:52.380877+0000","last_clean":"2024-09-21T12:28:52.380877+0000","last_became_active":"2024-09-21T12:28:20.951382+0000","last_became_peered":"2024-09-21T12:28:20.951382+0000","last_unstale":"2024-09-21T12:28:52.380877+0000","last_undegraded":"2024-09-21T12:28:52.380877+0000","last_fullsized":"2024-09-21T12:28:52.380877+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3164,"log_dups_size":0,"ondisk_log_size":3164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:21.160519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207355904,"num_objects":3164,"num_object_clones":0,"num_object_copies":9492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3164,"num_write_kb":202496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"76'79","reported_seq":128,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.155587+0000","last_change":"2024-09-21T12:27:18.331811+0000","last_active":"2024-09-21T12:28:49.155587+0000","last_peered":"2024-09-21T12:28:49.155587+0000","last_clean":"2024-09-21T12:28:49.155587+0000","last_became_active":"2024-09-21T12:27:14.944162+0000","last_became_peered":"2024-09-21T12:27:14.944162+0000","last_unstale":"2024-09-21T12:28:49.155587+0000","last_undegraded":"2024-09-21T12:28:49.155587+0000","last_fullsized":"2024-09-21T12:28:49.155587+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:18.331743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:18.331743+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":444,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.151970+0000","last_change":"2024-09-21T12:25:03.416100+0000","last_active":"2024-09-21T12:28:49.151970+0000","last_peered":"2024-09-21T12:28:49.151970+0000","last_clean":"2024-09-21T12:28:49.151970+0000","last_became_active":"2024-09-21T12:20:17.692870+0000","last_became_peered":"2024-09-21T12:20:17.692870+0000","last_unstale":"2024-09-21T12:28:49.151970+0000","last_undegraded":"2024-09-21T12:28:49.151970+0000","last_fullsized":"2024-09-21T12:28:49.151970+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:25:03.416033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:03.416033+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":466,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.155433+0000","last_change":"2024-09-21T12:24:29.054851+0000","last_active":"2024-09-21T12:28:49.155433+0000","last_peered":"2024-09-21T12:28:49.155433+0000","last_clean":"2024-09-21T12:28:49.155433+0000","last_became_active":"2024-09-21T12:22:50.927141+0000","last_became_peered":"2024-09-21T12:22:50.927141+0000","last_unstale":"2024-09-21T12:28:49.155433+0000","last_undegraded":"2024-09-21T12:28:49.155433+0000","last_fullsized":"2024-09-21T12:28:49.155433+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-21T12:24:29.054772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:29.054772+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:51.583706+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"121'11","reported_seq":544,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.154902+0000","last_change":"2024-09-21T12:25:02.916145+0000","last_active":"2024-09-21T12:28:49.154902+0000","last_peered":"2024-09-21T12:28:49.154902+0000","last_clean":"2024-09-21T12:28:49.154902+0000","last_became_active":"2024-09-21T12:22:50.908371+0000","last_became_peered":"2024-09-21T12:22:50.908371+0000","last_unstale":"2024-09-21T12:28:49.154902+0000","last_undegraded":"2024-09-21T12:28:49.154902+0000","last_fullsized":"2024-09-21T12:28:49.154902+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:25:02.916090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:02.916090+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:22.744691+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":106,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.166054+0000","last_change":"2024-09-21T12:26:56.862895+0000","last_active":"2024-09-21T12:28:49.166054+0000","last_peered":"2024-09-21T12:28:49.166054+0000","last_clean":"2024-09-21T12:28:49.166054+0000","last_became_active":"2024-09-21T12:26:54.089155+0000","last_became_peered":"2024-09-21T12:26:54.089155+0000","last_unstale":"2024-09-21T12:28:49.166054+0000","last_undegraded":"2024-09-21T12:28:49.166054+0000","last_fullsized":"2024-09-21T12:28:49.166054+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:56.862823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:56.862823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:17.174105+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"76'79","reported_seq":122,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.152090+0000","last_change":"2024-09-21T12:28:05.321813+0000","last_active":"2024-09-21T12:28:49.152090+0000","last_peered":"2024-09-21T12:28:49.152090+0000","last_clean":"2024-09-21T12:28:49.152090+0000","last_became_active":"2024-09-21T12:26:46.971481+0000","last_became_peered":"2024-09-21T12:26:46.971481+0000","last_unstale":"2024-09-21T12:28:49.152090+0000","last_undegraded":"2024-09-21T12:28:49.152090+0000","last_fullsized":"2024-09-21T12:28:49.152090+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:05.321753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:05.321753+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:10.468782+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":439,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.166345+0000","last_change":"2024-09-21T12:27:29.901101+0000","last_active":"2024-09-21T12:28:49.166345+0000","last_peered":"2024-09-21T12:28:49.166345+0000","last_clean":"2024-09-21T12:28:49.166345+0000","last_became_active":"2024-09-21T12:20:15.303480+0000","last_became_peered":"2024-09-21T12:20:15.303480+0000","last_unstale":"2024-09-21T12:28:49.166345+0000","last_undegraded":"2024-09-21T12:28:49.166345+0000","last_fullsized":"2024-09-21T12:28:49.166345+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:29.901043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:27:29.901043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":421,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.942061+0000","last_change":"2024-09-21T12:26:57.369721+0000","last_active":"2024-09-21T12:28:51.942061+0000","last_peered":"2024-09-21T12:28:51.942061+0000","last_clean":"2024-09-21T12:28:51.942061+0000","last_became_active":"2024-09-21T12:20:17.694628+0000","last_became_peered":"2024-09-21T12:20:17.694628+0000","last_unstale":"2024-09-21T12:28:51.942061+0000","last_undegraded":"2024-09-21T12:28:51.942061+0000","last_fullsized":"2024-09-21T12:28:51.942061+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-21T12:26:57.369648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:57.369648+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":101,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.940117+0000","last_change":"2024-09-21T12:26:54.084503+0000","last_active":"2024-09-21T12:28:51.940117+0000","last_peered":"2024-09-21T12:28:51.940117+0000","last_clean":"2024-09-21T12:28:51.940117+0000","last_became_active":"2024-09-21T12:26:54.084109+0000","last_became_peered":"2024-09-21T12:26:54.084109+0000","last_unstale":"2024-09-21T12:28:51.940117+0000","last_undegraded":"2024-09-21T12:28:51.940117+0000","last_fullsized":"2024-09-21T12:28:51.940117+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:02.414298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:02.414298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:20.592593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":416,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.155629+0000","last_change":"2024-09-21T12:24:00.066522+0000","last_active":"2024-09-21T12:28:49.155629+0000","last_peered":"2024-09-21T12:28:49.155629+0000","last_clean":"2024-09-21T12:28:49.155629+0000","last_became_active":"2024-09-21T12:22:50.904404+0000","last_became_peered":"2024-09-21T12:22:50.904404+0000","last_unstale":"2024-09-21T12:28:49.155629+0000","last_undegraded":"2024-09-21T12:28:49.155629+0000","last_fullsized":"2024-09-21T12:28:49.155629+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:00.066451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:00.066451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:49.621051+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'79","reported_seq":365,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.968713+0000","last_change":"2024-09-21T12:28:29.546727+0000","last_active":"2024-09-21T12:28:51.968713+0000","last_peered":"2024-09-21T12:28:51.968713+0000","last_clean":"2024-09-21T12:28:51.968713+0000","last_became_active":"2024-09-21T12:26:46.994966+0000","last_became_peered":"2024-09-21T12:26:46.994966+0000","last_unstale":"2024-09-21T12:28:51.968713+0000","last_undegraded":"2024-09-21T12:28:51.968713+0000","last_fullsized":"2024-09-21T12:28:51.968713+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:29.546657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:29.546657+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:42.115154+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":439,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.951099+0000","last_change":"2024-09-21T12:28:01.881249+0000","last_active":"2024-09-21T12:28:51.951099+0000","last_peered":"2024-09-21T12:28:51.951099+0000","last_clean":"2024-09-21T12:28:51.951099+0000","last_became_active":"2024-09-21T12:20:17.695898+0000","last_became_peered":"2024-09-21T12:20:17.695898+0000","last_unstale":"2024-09-21T12:28:51.951099+0000","last_undegraded":"2024-09-21T12:28:51.951099+0000","last_fullsized":"2024-09-21T12:28:51.951099+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:28:01.881164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:01.881164+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:03.000373+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":452,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.155188+0000","last_change":"2024-09-21T12:28:43.545259+0000","last_active":"2024-09-21T12:28:49.155188+0000","last_peered":"2024-09-21T12:28:49.155188+0000","last_clean":"2024-09-21T12:28:49.155188+0000","last_became_active":"2024-09-21T12:22:50.927649+0000","last_became_peered":"2024-09-21T12:22:50.927649+0000","last_unstale":"2024-09-21T12:28:49.155188+0000","last_undegraded":"2024-09-21T12:28:49.155188+0000","last_fullsized":"2024-09-21T12:28:49.155188+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:43.545197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:43.545197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:51.627798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":408,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.941806+0000","last_change":"2024-09-21T12:28:47.964092+0000","last_active":"2024-09-21T12:28:51.941806+0000","last_peered":"2024-09-21T12:28:51.941806+0000","last_clean":"2024-09-21T12:28:51.941806+0000","last_became_active":"2024-09-21T12:20:19.311947+0000","last_became_peered":"2024-09-21T12:20:19.311947+0000","last_unstale":"2024-09-21T12:28:51.941806+0000","last_undegraded":"2024-09-21T12:28:51.941806+0000","last_fullsized":"2024-09-21T12:28:51.941806+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:47.963747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:47.963747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:12.338200+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":443,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.964610+0000","last_change":"2024-09-21T12:26:11.105716+0000","last_active":"2024-09-21T12:28:51.964610+0000","last_peered":"2024-09-21T12:28:51.964610+0000","last_clean":"2024-09-21T12:28:51.964610+0000","last_became_active":"2024-09-21T12:22:50.924931+0000","last_became_peered":"2024-09-21T12:22:50.924931+0000","last_unstale":"2024-09-21T12:28:51.964610+0000","last_undegraded":"2024-09-21T12:28:51.964610+0000","last_fullsized":"2024-09-21T12:28:51.964610+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:11.105583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:11.105583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:51.335162+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"76'79","reported_seq":491,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.154289+0000","last_change":"2024-09-21T12:26:54.085274+0000","last_active":"2024-09-21T12:28:49.154289+0000","last_peered":"2024-09-21T12:28:49.154289+0000","last_clean":"2024-09-21T12:28:49.154289+0000","last_became_active":"2024-09-21T12:26:54.084939+0000","last_became_peered":"2024-09-21T12:26:54.084939+0000","last_unstale":"2024-09-21T12:28:49.154289+0000","last_undegraded":"2024-09-21T12:28:49.154289+0000","last_fullsized":"2024-09-21T12:28:49.154289+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:50.443344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:50.443344+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:29:16.709698+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":142,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.155736+0000","last_change":"2024-09-21T12:28:42.511948+0000","last_active":"2024-09-21T12:28:49.155736+0000","last_peered":"2024-09-21T12:28:49.155736+0000","last_clean":"2024-09-21T12:28:49.155736+0000","last_became_active":"2024-09-21T12:26:54.101937+0000","last_became_peered":"2024-09-21T12:26:54.101937+0000","last_unstale":"2024-09-21T12:28:49.155736+0000","last_undegraded":"2024-09-21T12:28:49.155736+0000","last_fullsized":"2024-09-21T12:28:49.155736+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:28:42.511874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:42.511874+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:46.711686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":427,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.948833+0000","last_change":"2024-09-21T12:28:07.875348+0000","last_active":"2024-09-21T12:28:51.948833+0000","last_peered":"2024-09-21T12:28:51.948833+0000","last_clean":"2024-09-21T12:28:51.948833+0000","last_became_active":"2024-09-21T12:20:15.297519+0000","last_became_peered":"2024-09-21T12:20:15.297519+0000","last_unstale":"2024-09-21T12:28:51.948833+0000","last_undegraded":"2024-09-21T12:28:51.948833+0000","last_fullsized":"2024-09-21T12:28:51.948833+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:07.875304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:07.875304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:36.761971+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":520,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.943396+0000","last_change":"2024-09-21T12:28:43.854094+0000","last_active":"2024-09-21T12:28:51.943396+0000","last_peered":"2024-09-21T12:28:51.943396+0000","last_clean":"2024-09-21T12:28:51.943396+0000","last_became_active":"2024-09-21T12:20:19.305364+0000","last_became_peered":"2024-09-21T12:20:19.305364+0000","last_unstale":"2024-09-21T12:28:51.943396+0000","last_undegraded":"2024-09-21T12:28:51.943396+0000","last_fullsized":"2024-09-21T12:28:51.943396+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:28:43.853712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:43.853712+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:03.508462+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":349,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.963526+0000","last_change":"2024-09-21T12:28:05.323501+0000","last_active":"2024-09-21T12:28:51.963526+0000","last_peered":"2024-09-21T12:28:51.963526+0000","last_clean":"2024-09-21T12:28:51.963526+0000","last_became_active":"2024-09-21T12:22:50.924770+0000","last_became_peered":"2024-09-21T12:22:50.924770+0000","last_unstale":"2024-09-21T12:28:51.963526+0000","last_undegraded":"2024-09-21T12:28:51.963526+0000","last_fullsized":"2024-09-21T12:28:51.963526+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:05.323448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:05.323448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:19.896140+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"76'79","reported_seq":207,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.945534+0000","last_change":"2024-09-21T12:26:48.804298+0000","last_active":"2024-09-21T12:28:51.945534+0000","last_peered":"2024-09-21T12:28:51.945534+0000","last_clean":"2024-09-21T12:28:51.945534+0000","last_became_active":"2024-09-21T12:26:46.993017+0000","last_became_peered":"2024-09-21T12:26:46.993017+0000","last_unstale":"2024-09-21T12:28:51.945534+0000","last_undegraded":"2024-09-21T12:28:51.945534+0000","last_fullsized":"2024-09-21T12:28:51.945534+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:48.804235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:48.804235+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:19.648260+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":427,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.943741+0000","last_change":"2024-09-21T12:24:50.727161+0000","last_active":"2024-09-21T12:28:51.943741+0000","last_peered":"2024-09-21T12:28:51.943741+0000","last_clean":"2024-09-21T12:28:51.943741+0000","last_became_active":"2024-09-21T12:20:17.693209+0000","last_became_peered":"2024-09-21T12:20:17.693209+0000","last_unstale":"2024-09-21T12:28:51.943741+0000","last_undegraded":"2024-09-21T12:28:51.943741+0000","last_fullsized":"2024-09-21T12:28:51.943741+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:24:50.727101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:50.727101+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:17.579086+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":444,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.155030+0000","last_change":"2024-09-21T12:28:32.630320+0000","last_active":"2024-09-21T12:28:49.155030+0000","last_peered":"2024-09-21T12:28:49.155030+0000","last_clean":"2024-09-21T12:28:49.155030+0000","last_became_active":"2024-09-21T12:22:50.908238+0000","last_became_peered":"2024-09-21T12:22:50.908238+0000","last_unstale":"2024-09-21T12:28:49.155030+0000","last_undegraded":"2024-09-21T12:28:49.155030+0000","last_fullsized":"2024-09-21T12:28:49.155030+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:32.630261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:32.630261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:53.164228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":519,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.152931+0000","last_change":"2024-09-21T12:28:36.293443+0000","last_active":"2024-09-21T12:28:49.152931+0000","last_peered":"2024-09-21T12:28:49.152931+0000","last_clean":"2024-09-21T12:28:49.152931+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:28:49.152931+0000","last_undegraded":"2024-09-21T12:28:49.152931+0000","last_fullsized":"2024-09-21T12:28:49.152931+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:28:36.293377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:36.293377+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:01.258654+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":399,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.950670+0000","last_change":"2024-09-21T12:26:08.926904+0000","last_active":"2024-09-21T12:28:51.950670+0000","last_peered":"2024-09-21T12:28:51.950670+0000","last_clean":"2024-09-21T12:28:51.950670+0000","last_became_active":"2024-09-21T12:20:21.309385+0000","last_became_peered":"2024-09-21T12:20:21.309385+0000","last_unstale":"2024-09-21T12:28:51.950670+0000","last_undegraded":"2024-09-21T12:28:51.950670+0000","last_fullsized":"2024-09-21T12:28:51.950670+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:08.923458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:08.923458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:16.660312+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"76'79","reported_seq":86,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.971706+0000","last_change":"2024-09-21T12:27:18.292942+0000","last_active":"2024-09-21T12:28:51.971706+0000","last_peered":"2024-09-21T12:28:51.971706+0000","last_clean":"2024-09-21T12:28:51.971706+0000","last_became_active":"2024-09-21T12:27:16.940347+0000","last_became_peered":"2024-09-21T12:27:16.940347+0000","last_unstale":"2024-09-21T12:28:51.971706+0000","last_undegraded":"2024-09-21T12:28:51.971706+0000","last_fullsized":"2024-09-21T12:28:51.971706+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:18.292862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:18.292862+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":411,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.166935+0000","last_change":"2024-09-21T12:28:40.150322+0000","last_active":"2024-09-21T12:28:49.166935+0000","last_peered":"2024-09-21T12:28:49.166935+0000","last_clean":"2024-09-21T12:28:49.166935+0000","last_became_active":"2024-09-21T12:22:50.906473+0000","last_became_peered":"2024-09-21T12:22:50.906473+0000","last_unstale":"2024-09-21T12:28:49.166935+0000","last_undegraded":"2024-09-21T12:28:49.166935+0000","last_fullsized":"2024-09-21T12:28:49.166935+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:28:40.150250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:40.150250+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:42.939288+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":332,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.155337+0000","last_change":"2024-09-21T12:24:58.948698+0000","last_active":"2024-09-21T12:28:49.155337+0000","last_peered":"2024-09-21T12:28:49.155337+0000","last_clean":"2024-09-21T12:28:49.155337+0000","last_became_active":"2024-09-21T12:22:50.903705+0000","last_became_peered":"2024-09-21T12:22:50.903705+0000","last_unstale":"2024-09-21T12:28:49.155337+0000","last_undegraded":"2024-09-21T12:28:49.155337+0000","last_fullsized":"2024-09-21T12:28:49.155337+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:58.948625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:58.948625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:21.759502+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":404,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.969223+0000","last_change":"2024-09-21T12:26:16.149823+0000","last_active":"2024-09-21T12:28:51.969223+0000","last_peered":"2024-09-21T12:28:51.969223+0000","last_clean":"2024-09-21T12:28:51.969223+0000","last_became_active":"2024-09-21T12:20:19.315418+0000","last_became_peered":"2024-09-21T12:20:19.315418+0000","last_unstale":"2024-09-21T12:28:51.969223+0000","last_undegraded":"2024-09-21T12:28:51.969223+0000","last_fullsized":"2024-09-21T12:28:51.969223+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:16.149550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:16.149550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:28:53.566203+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":404,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.940342+0000","last_change":"2024-09-21T12:24:56.879617+0000","last_active":"2024-09-21T12:28:51.940342+0000","last_peered":"2024-09-21T12:28:51.940342+0000","last_clean":"2024-09-21T12:28:51.940342+0000","last_became_active":"2024-09-21T12:20:21.310053+0000","last_became_peered":"2024-09-21T12:20:21.310053+0000","last_unstale":"2024-09-21T12:28:51.940342+0000","last_undegraded":"2024-09-21T12:28:51.940342+0000","last_fullsized":"2024-09-21T12:28:51.940342+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:56.879558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:56.879558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:23.616386+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"76'79","reported_seq":93,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.151990+0000","last_change":"2024-09-21T12:28:31.187538+0000","last_active":"2024-09-21T12:28:49.151990+0000","last_peered":"2024-09-21T12:28:49.151990+0000","last_clean":"2024-09-21T12:28:49.151990+0000","last_became_active":"2024-09-21T12:27:16.921880+0000","last_became_peered":"2024-09-21T12:27:16.921880+0000","last_unstale":"2024-09-21T12:28:49.151990+0000","last_undegraded":"2024-09-21T12:28:49.151990+0000","last_fullsized":"2024-09-21T12:28:49.151990+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:31.187467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:31.187467+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:53.371638+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":429,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.968828+0000","last_change":"2024-09-21T12:28:36.646045+0000","last_active":"2024-09-21T12:28:51.968828+0000","last_peered":"2024-09-21T12:28:51.968828+0000","last_clean":"2024-09-21T12:28:51.968828+0000","last_became_active":"2024-09-21T12:20:17.299015+0000","last_became_peered":"2024-09-21T12:20:17.299015+0000","last_unstale":"2024-09-21T12:28:51.968828+0000","last_undegraded":"2024-09-21T12:28:51.968828+0000","last_fullsized":"2024-09-21T12:28:51.968828+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:28:36.645961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:36.645961+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:01.394775+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":444,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.155770+0000","last_change":"2024-09-21T12:26:19.973528+0000","last_active":"2024-09-21T12:28:49.155770+0000","last_peered":"2024-09-21T12:28:49.155770+0000","last_clean":"2024-09-21T12:28:49.155770+0000","last_became_active":"2024-09-21T12:22:50.922019+0000","last_became_peered":"2024-09-21T12:22:50.922019+0000","last_unstale":"2024-09-21T12:28:49.155770+0000","last_undegraded":"2024-09-21T12:28:49.155770+0000","last_fullsized":"2024-09-21T12:28:49.155770+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:19.973461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:19.973461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":408,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.940106+0000","last_change":"2024-09-21T12:26:21.572474+0000","last_active":"2024-09-21T12:28:51.940106+0000","last_peered":"2024-09-21T12:28:51.940106+0000","last_clean":"2024-09-21T12:28:51.940106+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:28:51.940106+0000","last_undegraded":"2024-09-21T12:28:51.940106+0000","last_fullsized":"2024-09-21T12:28:51.940106+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:21.571726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:21.571726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":407,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.945282+0000","last_change":"2024-09-21T12:28:41.864711+0000","last_active":"2024-09-21T12:28:51.945282+0000","last_peered":"2024-09-21T12:28:51.945282+0000","last_clean":"2024-09-21T12:28:51.945282+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:28:51.945282+0000","last_undegraded":"2024-09-21T12:28:51.945282+0000","last_fullsized":"2024-09-21T12:28:51.945282+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:41.864630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:41.864630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:09.118573+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"76'79","reported_seq":116,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.965157+0000","last_change":"2024-09-21T12:26:49.624804+0000","last_active":"2024-09-21T12:28:51.965157+0000","last_peered":"2024-09-21T12:28:51.965157+0000","last_clean":"2024-09-21T12:28:51.965157+0000","last_became_active":"2024-09-21T12:26:46.989742+0000","last_became_peered":"2024-09-21T12:26:46.989742+0000","last_unstale":"2024-09-21T12:28:51.965157+0000","last_undegraded":"2024-09-21T12:28:51.965157+0000","last_fullsized":"2024-09-21T12:28:51.965157+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:49.623735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:49.623735+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":431,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.940681+0000","last_change":"2024-09-21T12:24:58.824615+0000","last_active":"2024-09-21T12:28:51.940681+0000","last_peered":"2024-09-21T12:28:51.940681+0000","last_clean":"2024-09-21T12:28:51.940681+0000","last_became_active":"2024-09-21T12:20:17.694794+0000","last_became_peered":"2024-09-21T12:20:17.694794+0000","last_unstale":"2024-09-21T12:28:51.940681+0000","last_undegraded":"2024-09-21T12:28:51.940681+0000","last_fullsized":"2024-09-21T12:28:51.940681+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:24:58.824516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:58.824516+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:19.560401+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":431,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.152351+0000","last_change":"2024-09-21T12:28:45.330667+0000","last_active":"2024-09-21T12:28:49.152351+0000","last_peered":"2024-09-21T12:28:49.152351+0000","last_clean":"2024-09-21T12:28:49.152351+0000","last_became_active":"2024-09-21T12:20:15.295968+0000","last_became_peered":"2024-09-21T12:20:15.295968+0000","last_unstale":"2024-09-21T12:28:49.152351+0000","last_undegraded":"2024-09-21T12:28:49.152351+0000","last_fullsized":"2024-09-21T12:28:49.152351+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:45.330623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.330623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:52.780526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":420,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.154281+0000","last_change":"2024-09-21T12:23:57.018797+0000","last_active":"2024-09-21T12:28:49.154281+0000","last_peered":"2024-09-21T12:28:49.154281+0000","last_clean":"2024-09-21T12:28:49.154281+0000","last_became_active":"2024-09-21T12:22:50.921604+0000","last_became_peered":"2024-09-21T12:22:50.921604+0000","last_unstale":"2024-09-21T12:28:49.154281+0000","last_undegraded":"2024-09-21T12:28:49.154281+0000","last_fullsized":"2024-09-21T12:28:49.154281+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:57.018749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:57.018749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:20.736741+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":396,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.940353+0000","last_change":"2024-09-21T12:24:35.588409+0000","last_active":"2024-09-21T12:28:51.940353+0000","last_peered":"2024-09-21T12:28:51.940353+0000","last_clean":"2024-09-21T12:28:51.940353+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:28:51.940353+0000","last_undegraded":"2024-09-21T12:28:51.940353+0000","last_fullsized":"2024-09-21T12:28:51.940353+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:35.588353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:35.588353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:08.583895+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"76'79","reported_seq":85,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.154913+0000","last_change":"2024-09-21T12:27:24.373557+0000","last_active":"2024-09-21T12:28:49.154913+0000","last_peered":"2024-09-21T12:28:49.154913+0000","last_clean":"2024-09-21T12:28:49.154913+0000","last_became_active":"2024-09-21T12:27:16.914713+0000","last_became_peered":"2024-09-21T12:27:16.914713+0000","last_unstale":"2024-09-21T12:28:49.154913+0000","last_undegraded":"2024-09-21T12:28:49.154913+0000","last_fullsized":"2024-09-21T12:28:49.154913+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:24.373501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:24.373501+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":418,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.940618+0000","last_change":"2024-09-21T12:28:46.932522+0000","last_active":"2024-09-21T12:28:51.940618+0000","last_peered":"2024-09-21T12:28:51.940618+0000","last_clean":"2024-09-21T12:28:51.940618+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:28:51.940618+0000","last_undegraded":"2024-09-21T12:28:51.940618+0000","last_fullsized":"2024-09-21T12:28:51.940618+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-21T12:28:46.931901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:46.931901+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:14.037686+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":433,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.949517+0000","last_change":"2024-09-21T12:28:00.838008+0000","last_active":"2024-09-21T12:28:51.949517+0000","last_peered":"2024-09-21T12:28:51.949517+0000","last_clean":"2024-09-21T12:28:51.949517+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:28:51.949517+0000","last_undegraded":"2024-09-21T12:28:51.949517+0000","last_fullsized":"2024-09-21T12:28:51.949517+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:28:00.837937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:00.837937+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:12.552751+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":428,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.165908+0000","last_change":"2024-09-21T12:26:22.014941+0000","last_active":"2024-09-21T12:28:49.165908+0000","last_peered":"2024-09-21T12:28:49.165908+0000","last_clean":"2024-09-21T12:28:49.165908+0000","last_became_active":"2024-09-21T12:22:50.930615+0000","last_became_peered":"2024-09-21T12:22:50.930615+0000","last_unstale":"2024-09-21T12:28:49.165908+0000","last_undegraded":"2024-09-21T12:28:49.165908+0000","last_fullsized":"2024-09-21T12:28:49.165908+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:22.014896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:22.014896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":424,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.154652+0000","last_change":"2024-09-21T12:28:41.537845+0000","last_active":"2024-09-21T12:28:49.154652+0000","last_peered":"2024-09-21T12:28:49.154652+0000","last_clean":"2024-09-21T12:28:49.154652+0000","last_became_active":"2024-09-21T12:22:50.917408+0000","last_became_peered":"2024-09-21T12:22:50.917408+0000","last_unstale":"2024-09-21T12:28:49.154652+0000","last_undegraded":"2024-09-21T12:28:49.154652+0000","last_fullsized":"2024-09-21T12:28:49.154652+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:41.537786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:41.537786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:52.332880+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"76'79","reported_seq":85,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.154930+0000","last_change":"2024-09-21T12:27:29.341314+0000","last_active":"2024-09-21T12:28:49.154930+0000","last_peered":"2024-09-21T12:28:49.154930+0000","last_clean":"2024-09-21T12:28:49.154930+0000","last_became_active":"2024-09-21T12:27:16.948697+0000","last_became_peered":"2024-09-21T12:27:16.948697+0000","last_unstale":"2024-09-21T12:28:49.154930+0000","last_undegraded":"2024-09-21T12:28:49.154930+0000","last_fullsized":"2024-09-21T12:28:49.154930+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:29.341268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:29.341268+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":426,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.969442+0000","last_change":"2024-09-21T12:28:46.749904+0000","last_active":"2024-09-21T12:28:51.969442+0000","last_peered":"2024-09-21T12:28:51.969442+0000","last_clean":"2024-09-21T12:28:51.969442+0000","last_became_active":"2024-09-21T12:22:50.914221+0000","last_became_peered":"2024-09-21T12:22:50.914221+0000","last_unstale":"2024-09-21T12:28:51.969442+0000","last_undegraded":"2024-09-21T12:28:51.969442+0000","last_fullsized":"2024-09-21T12:28:51.969442+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:46.749839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:46.749839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:07.174169+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":416,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.941240+0000","last_change":"2024-09-21T12:25:00.871133+0000","last_active":"2024-09-21T12:28:51.941240+0000","last_peered":"2024-09-21T12:28:51.941240+0000","last_clean":"2024-09-21T12:28:51.941240+0000","last_became_active":"2024-09-21T12:20:15.297993+0000","last_became_peered":"2024-09-21T12:20:15.297993+0000","last_unstale":"2024-09-21T12:28:51.941240+0000","last_undegraded":"2024-09-21T12:28:51.941240+0000","last_fullsized":"2024-09-21T12:28:51.941240+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:00.871062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:00.871062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:11.582883+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":431,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T12:28:49.167575+0000","last_change":"2024-09-21T12:27:31.860075+0000","last_active":"2024-09-21T12:28:49.167575+0000","last_peered":"2024-09-21T12:28:49.167575+0000","last_clean":"2024-09-21T12:28:49.167575+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:28:49.167575+0000","last_undegraded":"2024-09-21T12:28:49.167575+0000","last_fullsized":"2024-09-21T12:28:49.167575+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:27:31.860002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:31.860002+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":436,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-21T12:28:51.950771+0000","last_change":"2024-09-21T12:28:28.741483+0000","last_active":"2024-09-21T12:28:51.950771+0000","last_peered":"2024-09-21T12:28:51.950771+0000","last_clean":"2024-09-21T12:28:51.950771+0000","last_became_active":"2024-09-21T12:20:19.308247+0000","last_became_peered":"2024-09-21T12:20:19.308247+0000","last_unstale":"2024-09-21T12:28:51.950771+0000","last_undegraded":"2024-09-21T12:28:51.950771+0000","last_fullsized":"2024-09-21T12:28:51.950771+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:28.741418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:28.741418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:39.129562+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":3301113856,"num_objects":50371,"num_object_clones":0,"num_object_copies":151113,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50371,"num_write_kb":3223744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9884794880,"data_stored":9884794880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50371,"ondisk_log_size":50371,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":689,"num_read_kb":446,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":3872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":396,"ondisk_log_size":396,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":5023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":511,"num_read_kb":506,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":721712,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2449,"ondisk_log_size":2449,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234091,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2087380,"kb_used_data":1777156,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":91657772,"statfs":{"total":95995035648,"available":93857558528,"internally_reserved":0,"allocated":1819807744,"data_stored":1819293680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":317651402},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463027,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":901780,"kb_used_data":585984,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":92843372,"statfs":{"total":95995035648,"available":95071612928,"internally_reserved":0,"allocated":600047616,"data_stored":599536895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":323353030},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724662,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2078356,"kb_used_data":1763972,"kb_used_omap":1,"kb_used_meta":314366,"kb_avail":91666796,"statfs":{"total":95995035648,"available":93866799104,"internally_reserved":0,"allocated":1806307328,"data_stored":1805794154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":321911237},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953598,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149828,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2008568,"kb_used_data":1608100,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":91736584,"statfs":{"total":95995035648,"available":93938262016,"internally_reserved":0,"allocated":1646694400,"data_stored":1646156515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":410057159},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411466,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":879748,"kb_used_data":591732,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":92865404,"statfs":{"total":95995035648,"available":95094173696,"internally_reserved":0,"allocated":605933568,"data_stored":605435235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574992,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2155572,"kb_used_data":1803684,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":91589580,"statfs":{"total":95995035648,"available":93787729920,"internally_reserved":0,"allocated":1846972416,"data_stored":1846425359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738518,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1746200,"kb_used_data":1417928,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":91998952,"statfs":{"total":95995035648,"available":94206926848,"internally_reserved":0,"allocated":1451958272,"data_stored":1451440281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1473380352,"data_stored":1473380352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1856176128,"data_stored":1856176128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":616431616,"data_stored":616431616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1661730816,"data_stored":1661730816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1835597824,"data_stored":1835597824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":612564992,"data_stored":612564992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1828913152,"data_stored":1828913152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:28:56.662 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:28:56.703 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:56.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:56.744 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:56.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:56.827 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 29 16 57334 57318 123.516 95.5 0.0154267 0.00809108 2024-09-21T12:28:56.845 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:28:56.876 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:56.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:56.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:28:56.978 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:28:57.023 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:57.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:57.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:28:57.125 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:57.143 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:28:57.144 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-21T12:28:57.144 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-21T12:28:57.144 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:28:57.144 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.144 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:23884 7.1 7:8f148689:::benchmark_data_smithi114_78432_object23883:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T12:28:57.144 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:39.695870+0000", 2024-09-21T12:28:57.144 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 437.44454369499999, 2024-09-21T12:28:57.144 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.039559285999999999, 2024-09-21T12:28:57.144 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:28:57.144 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:28:57.144 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:57.144 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:28:57.144 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:28:57.144 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:28:57.145 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 23884 2024-09-21T12:28:57.145 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.145 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:28:57.145 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.145 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:28:57.145 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695870+0000", 2024-09-21T12:28:57.145 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.145 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.145 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.145 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:28:57.145 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695870+0000", 2024-09-21T12:28:57.145 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.145 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.145 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.146 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:28:57.146 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695873+0000", 2024-09-21T12:28:57.146 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0090000000000002e-06 2024-09-21T12:28:57.146 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.146 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.146 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:28:57.146 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695927+0000", 2024-09-21T12:28:57.146 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4941000000000001e-05 2024-09-21T12:28:57.146 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.146 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.146 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:28:57.146 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695955+0000", 2024-09-21T12:28:57.146 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7120000000000001e-05 2024-09-21T12:28:57.147 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.147 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.147 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:28:57.147 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695963+0000", 2024-09-21T12:28:57.147 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7760000000000003e-06 2024-09-21T12:28:57.147 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.147 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.147 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:28:57.147 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696022+0000", 2024-09-21T12:28:57.147 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8941999999999999e-05 2024-09-21T12:28:57.147 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.147 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.147 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:28:57.147 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696107+0000", 2024-09-21T12:28:57.147 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4853999999999998e-05 2024-09-21T12:28:57.148 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.148 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.148 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,6", 2024-09-21T12:28:57.148 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696202+0000", 2024-09-21T12:28:57.148 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4889999999999994e-05 2024-09-21T12:28:57.148 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.148 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.148 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:28:57.148 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.698132+0000", 2024-09-21T12:28:57.148 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001929998 2024-09-21T12:28:57.148 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.148 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.148 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:57.148 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.698167+0000", 2024-09-21T12:28:57.149 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5247000000000003e-05 2024-09-21T12:28:57.149 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.149 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.149 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:57.149 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.735322+0000", 2024-09-21T12:28:57.149 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.037154514 2024-09-21T12:28:57.149 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.149 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.149 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:28:57.149 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.735382+0000", 2024-09-21T12:28:57.149 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0371000000000002e-05 2024-09-21T12:28:57.149 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.149 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.149 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:28:57.149 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.735430+0000", 2024-09-21T12:28:57.150 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7623999999999997e-05 2024-09-21T12:28:57.150 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.150 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:28:57.150 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.150 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.150 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.150 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:23885 7.1 7:81ccbda1:::benchmark_data_smithi114_78432_object23884:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T12:28:57.150 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:39.696116+0000", 2024-09-21T12:28:57.150 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 437.444297866, 2024-09-21T12:28:57.150 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.040219576, 2024-09-21T12:28:57.150 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:28:57.150 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:28:57.150 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:57.150 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:28:57.151 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:28:57.151 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:28:57.151 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 23885 2024-09-21T12:28:57.151 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.151 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:28:57.151 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.151 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:28:57.151 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696116+0000", 2024-09-21T12:28:57.151 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.151 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.151 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.151 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:28:57.151 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696116+0000", 2024-09-21T12:28:57.151 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.152 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.152 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.152 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:28:57.152 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696117+0000", 2024-09-21T12:28:57.152 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0109999999999999e-06 2024-09-21T12:28:57.152 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.152 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.152 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:28:57.152 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696178+0000", 2024-09-21T12:28:57.152 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1060000000000002e-05 2024-09-21T12:28:57.152 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.152 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.152 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:28:57.152 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696188+0000", 2024-09-21T12:28:57.152 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0067e-05 2024-09-21T12:28:57.153 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.153 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.153 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:28:57.153 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696196+0000", 2024-09-21T12:28:57.153 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0740000000000004e-06 2024-09-21T12:28:57.153 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.153 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.153 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:28:57.153 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.697126+0000", 2024-09-21T12:28:57.153 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00093067599999999996 2024-09-21T12:28:57.153 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.153 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.153 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:28:57.153 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.697270+0000", 2024-09-21T12:28:57.154 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000143502 2024-09-21T12:28:57.154 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.154 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.154 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,6", 2024-09-21T12:28:57.154 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.697418+0000", 2024-09-21T12:28:57.154 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000147892 2024-09-21T12:28:57.154 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.154 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.154 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:28:57.154 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.698242+0000", 2024-09-21T12:28:57.154 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00082411799999999999 2024-09-21T12:28:57.154 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.154 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.154 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:57.154 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.698485+0000", 2024-09-21T12:28:57.155 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00024341099999999999 2024-09-21T12:28:57.155 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.155 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.155 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:57.155 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.736250+0000", 2024-09-21T12:28:57.155 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.037765265999999999 2024-09-21T12:28:57.155 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.155 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.155 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:28:57.155 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.736300+0000", 2024-09-21T12:28:57.155 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9886000000000003e-05 2024-09-21T12:28:57.155 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.155 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.156 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:28:57.156 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.736336+0000", 2024-09-21T12:28:57.156 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5612999999999999e-05 2024-09-21T12:28:57.156 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.156 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:28:57.156 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.156 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.156 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.156 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:81802 7.8 e115/112 7:110c0c7a:::benchmark_data_smithi114_78432_object81801:head v 115'5198, mlcod=88'242)", 2024-09-21T12:28:57.156 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.137865+0000", 2024-09-21T12:28:57.156 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 382.00254897899998, 2024-09-21T12:28:57.156 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.051160639000000001, 2024-09-21T12:28:57.156 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:28:57.156 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:28:57.157 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:57.157 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:28:57.157 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.157 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:28:57.158 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137865+0000", 2024-09-21T12:28:57.158 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.158 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.158 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.158 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:28:57.158 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137865+0000", 2024-09-21T12:28:57.159 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.159 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.159 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.159 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:28:57.159 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137866+0000", 2024-09-21T12:28:57.159 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5099999999999998e-07 2024-09-21T12:28:57.159 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.159 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.159 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:28:57.159 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137909+0000", 2024-09-21T12:28:57.159 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2545e-05 2024-09-21T12:28:57.159 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.159 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.159 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:28:57.159 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137938+0000", 2024-09-21T12:28:57.160 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9737999999999999e-05 2024-09-21T12:28:57.160 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.160 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.160 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:28:57.160 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137961+0000", 2024-09-21T12:28:57.160 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2320999999999999e-05 2024-09-21T12:28:57.160 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.160 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.160 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:28:57.160 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.138026+0000", 2024-09-21T12:28:57.160 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5571e-05 2024-09-21T12:28:57.160 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.160 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.160 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:28:57.161 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.138111+0000", 2024-09-21T12:28:57.161 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4682e-05 2024-09-21T12:28:57.161 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.161 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.161 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:28:57.161 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.188940+0000", 2024-09-21T12:28:57.161 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.050828867 2024-09-21T12:28:57.161 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.161 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.161 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:28:57.161 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189026+0000", 2024-09-21T12:28:57.161 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6063999999999995e-05 2024-09-21T12:28:57.161 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.161 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:28:57.162 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.162 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.162 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.162 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:81809 7.8 e115/112 7:1042b23e:::benchmark_data_smithi114_78432_object81808:head v 115'5199, mlcod=88'242)", 2024-09-21T12:28:57.162 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.140023+0000", 2024-09-21T12:28:57.162 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 382.00039140799998, 2024-09-21T12:28:57.162 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.049109005999999997, 2024-09-21T12:28:57.162 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:28:57.162 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:28:57.162 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:57.162 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:28:57.162 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.162 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:28:57.162 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140023+0000", 2024-09-21T12:28:57.163 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.163 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.163 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.163 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:28:57.163 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140023+0000", 2024-09-21T12:28:57.163 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.163 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.163 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.163 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:28:57.163 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140023+0000", 2024-09-21T12:28:57.163 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6700000000000003e-07 2024-09-21T12:28:57.163 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.163 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.163 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:28:57.164 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140136+0000", 2024-09-21T12:28:57.164 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000112623 2024-09-21T12:28:57.164 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.164 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.164 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:28:57.164 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140146+0000", 2024-09-21T12:28:57.164 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0098999999999999e-05 2024-09-21T12:28:57.164 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.164 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.164 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:28:57.164 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140161+0000", 2024-09-21T12:28:57.164 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5288e-05 2024-09-21T12:28:57.164 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.164 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.164 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:28:57.165 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146087+0000", 2024-09-21T12:28:57.165 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0059259070000000002 2024-09-21T12:28:57.165 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.165 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.165 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:28:57.165 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146109+0000", 2024-09-21T12:28:57.165 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2045999999999999e-05 2024-09-21T12:28:57.165 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.165 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.165 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:28:57.165 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189077+0000", 2024-09-21T12:28:57.165 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.042967639000000002 2024-09-21T12:28:57.165 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.165 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.165 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:28:57.166 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189132+0000", 2024-09-21T12:28:57.166 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4737000000000001e-05 2024-09-21T12:28:57.166 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.166 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:28:57.166 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.166 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.166 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.166 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:81811 7.8 e115/112 7:10044f1a:::benchmark_data_smithi114_78432_object81810:head v 115'5200, mlcod=88'242)", 2024-09-21T12:28:57.166 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.146782+0000", 2024-09-21T12:28:57.166 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 381.99363192300001, 2024-09-21T12:28:57.166 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.042406829, 2024-09-21T12:28:57.166 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:28:57.166 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:28:57.166 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:57.167 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:28:57.167 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.167 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:28:57.167 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146782+0000", 2024-09-21T12:28:57.167 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.167 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.167 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.167 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:28:57.167 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146782+0000", 2024-09-21T12:28:57.167 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.167 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.167 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.167 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:28:57.167 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146783+0000", 2024-09-21T12:28:57.167 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2999999999999999e-07 2024-09-21T12:28:57.168 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.168 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.168 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:28:57.168 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146826+0000", 2024-09-21T12:28:57.168 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2781000000000003e-05 2024-09-21T12:28:57.168 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.168 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.168 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:28:57.168 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146837+0000", 2024-09-21T12:28:57.168 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1126e-05 2024-09-21T12:28:57.168 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.168 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.168 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:28:57.168 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146843+0000", 2024-09-21T12:28:57.168 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4710000000000004e-06 2024-09-21T12:28:57.169 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.169 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.169 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:28:57.169 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.152740+0000", 2024-09-21T12:28:57.169 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0058961209999999998 2024-09-21T12:28:57.169 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.169 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.169 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:28:57.169 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.152763+0000", 2024-09-21T12:28:57.169 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3101000000000001e-05 2024-09-21T12:28:57.169 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.169 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.169 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:28:57.169 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189134+0000", 2024-09-21T12:28:57.169 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.036371542999999999 2024-09-21T12:28:57.170 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.170 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.170 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:28:57.170 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189189+0000", 2024-09-21T12:28:57.170 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4855999999999998e-05 2024-09-21T12:28:57.170 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.170 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:28:57.170 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.170 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.170 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.170 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:81821 7.8 e115/112 7:1c79dadc:::benchmark_data_smithi114_78432_object81820:head v 115'5201, mlcod=88'242)", 2024-09-21T12:28:57.170 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.148399+0000", 2024-09-21T12:28:57.170 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 381.992014848, 2024-09-21T12:28:57.170 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.040843988999999997, 2024-09-21T12:28:57.171 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:28:57.171 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:28:57.171 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:57.171 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:28:57.171 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.171 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:28:57.171 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148399+0000", 2024-09-21T12:28:57.171 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.171 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.171 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.171 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:28:57.171 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148399+0000", 2024-09-21T12:28:57.171 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.171 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.172 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.172 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:28:57.172 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148400+0000", 2024-09-21T12:28:57.172 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.049e-06 2024-09-21T12:28:57.172 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.172 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.172 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:28:57.172 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148502+0000", 2024-09-21T12:28:57.172 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000101542 2024-09-21T12:28:57.172 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.172 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.172 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:28:57.172 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148512+0000", 2024-09-21T12:28:57.172 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0386e-05 2024-09-21T12:28:57.173 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.173 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.173 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:28:57.173 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148519+0000", 2024-09-21T12:28:57.173 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3589999999999996e-06 2024-09-21T12:28:57.173 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.173 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.173 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:28:57.173 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.158203+0000", 2024-09-21T12:28:57.173 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0096844710000000001 2024-09-21T12:28:57.173 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.173 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.173 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:28:57.173 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.158229+0000", 2024-09-21T12:28:57.173 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6078999999999999e-05 2024-09-21T12:28:57.174 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.174 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.174 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:28:57.174 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189192+0000", 2024-09-21T12:28:57.174 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.030962455 2024-09-21T12:28:57.174 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.174 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.174 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:28:57.174 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189243+0000", 2024-09-21T12:28:57.174 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1647999999999997e-05 2024-09-21T12:28:57.174 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.174 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:28:57.174 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.174 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.175 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.175 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82430 7.8 e115/112 7:1762e17c:::benchmark_data_smithi114_78432_object82429:head v 115'5247, mlcod=88'255)", 2024-09-21T12:28:57.175 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.787623+0000", 2024-09-21T12:28:57.175 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 381.35279106000002, 2024-09-21T12:28:57.175 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.048511400000000003, 2024-09-21T12:28:57.175 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:28:57.175 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:28:57.175 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:57.175 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:28:57.175 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.175 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:28:57.175 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787623+0000", 2024-09-21T12:28:57.175 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.175 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.175 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.176 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:28:57.176 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787623+0000", 2024-09-21T12:28:57.176 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.176 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.176 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.176 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:28:57.176 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787625+0000", 2024-09-21T12:28:57.176 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5319999999999999e-06 2024-09-21T12:28:57.176 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.176 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.176 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:28:57.176 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787667+0000", 2024-09-21T12:28:57.176 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1876999999999999e-05 2024-09-21T12:28:57.176 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.177 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.177 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:28:57.177 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787688+0000", 2024-09-21T12:28:57.177 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1046000000000001e-05 2024-09-21T12:28:57.177 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.177 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.177 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:28:57.177 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787697+0000", 2024-09-21T12:28:57.177 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4930000000000004e-06 2024-09-21T12:28:57.177 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.177 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.177 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:28:57.177 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.788378+0000", 2024-09-21T12:28:57.177 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00068115999999999999 2024-09-21T12:28:57.178 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.178 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.178 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:28:57.178 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.788427+0000", 2024-09-21T12:28:57.178 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8553000000000001e-05 2024-09-21T12:28:57.178 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.178 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.178 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:28:57.178 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.836011+0000", 2024-09-21T12:28:57.178 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.047583978999999998 2024-09-21T12:28:57.178 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.178 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.178 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:28:57.178 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.836135+0000", 2024-09-21T12:28:57.179 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012375999999999999 2024-09-21T12:28:57.179 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.179 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:28:57.179 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.179 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.179 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.179 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDPGPush(7.8 115/112 [PushOp(7:10df046b:::benchmark_data_smithi114_78432_object3988:head, version: 88'257, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:10df046b:::benchmark_data_smithi114_78432_object3988:head@88'257, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T12:28:57.179 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.787722+0000", 2024-09-21T12:28:57.179 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 381.35269198499998, 2024-09-21T12:28:57.179 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.074165650999999999, 2024-09-21T12:28:57.179 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:28:57.179 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:28:57.179 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:28:57.179 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:28:57.180 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.180 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:28:57.180 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787722+0000", 2024-09-21T12:28:57.180 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.180 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.180 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.180 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:28:57.180 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787722+0000", 2024-09-21T12:28:57.180 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.180 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.180 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.180 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:28:57.180 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787723+0000", 2024-09-21T12:28:57.180 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0499999999999999e-07 2024-09-21T12:28:57.180 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.181 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.181 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:28:57.181 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787779+0000", 2024-09-21T12:28:57.181 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6053e-05 2024-09-21T12:28:57.181 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.181 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.181 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:28:57.181 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787803+0000", 2024-09-21T12:28:57.181 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3980000000000001e-05 2024-09-21T12:28:57.181 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.181 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.181 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:28:57.181 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787832+0000", 2024-09-21T12:28:57.181 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.942e-05 2024-09-21T12:28:57.181 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.182 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.182 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:28:57.182 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.859697+0000", 2024-09-21T12:28:57.182 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.071864649000000003 2024-09-21T12:28:57.182 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.182 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.182 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:28:57.182 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.859710+0000", 2024-09-21T12:28:57.182 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3740000000000001e-05 2024-09-21T12:28:57.182 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.182 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.182 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:28:57.182 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.861888+0000", 2024-09-21T12:28:57.182 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002177404 2024-09-21T12:28:57.183 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.183 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:28:57.183 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.183 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.183 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.183 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82442 7.8 e115/112 7:1035456a:::benchmark_data_smithi114_78432_object82441:head v 115'5248, mlcod=88'255)", 2024-09-21T12:28:57.183 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.791117+0000", 2024-09-21T12:28:57.183 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 381.34929708200002, 2024-09-21T12:28:57.183 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.051248759999999997, 2024-09-21T12:28:57.183 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:28:57.183 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:28:57.183 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:57.183 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:28:57.183 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.184 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:28:57.184 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791117+0000", 2024-09-21T12:28:57.184 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.184 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.184 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.184 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:28:57.184 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791117+0000", 2024-09-21T12:28:57.184 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.184 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.184 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.184 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:28:57.184 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791118+0000", 2024-09-21T12:28:57.184 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9999999999999998e-07 2024-09-21T12:28:57.184 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.184 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.185 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:28:57.185 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791158+0000", 2024-09-21T12:28:57.185 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0025999999999999e-05 2024-09-21T12:28:57.185 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.185 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.185 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:28:57.185 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791167+0000", 2024-09-21T12:28:57.185 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9910000000000008e-06 2024-09-21T12:28:57.185 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.185 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.185 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:28:57.185 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791176+0000", 2024-09-21T12:28:57.185 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0329999999999997e-06 2024-09-21T12:28:57.185 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.186 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.186 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:28:57.186 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.794825+0000", 2024-09-21T12:28:57.186 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0036494700000000001 2024-09-21T12:28:57.186 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.186 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.186 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:28:57.186 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.794878+0000", 2024-09-21T12:28:57.186 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3292999999999999e-05 2024-09-21T12:28:57.186 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.186 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.186 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:28:57.186 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842289+0000", 2024-09-21T12:28:57.186 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.047410256999999997 2024-09-21T12:28:57.187 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.187 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.187 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:28:57.187 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842366+0000", 2024-09-21T12:28:57.187 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7189999999999998e-05 2024-09-21T12:28:57.187 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.187 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:28:57.187 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.187 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.187 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.187 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82447 7.8 e115/112 7:174332ca:::benchmark_data_smithi114_78432_object82446:head v 115'5249, mlcod=88'255)", 2024-09-21T12:28:57.187 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.793874+0000", 2024-09-21T12:28:57.187 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 381.34654063099998, 2024-09-21T12:28:57.187 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.048541015999999999, 2024-09-21T12:28:57.188 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:28:57.188 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:28:57.188 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:57.188 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:28:57.188 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.188 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:28:57.188 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793874+0000", 2024-09-21T12:28:57.188 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.188 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.188 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.188 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:28:57.188 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793874+0000", 2024-09-21T12:28:57.188 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.188 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.189 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.189 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:28:57.189 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793875+0000", 2024-09-21T12:28:57.189 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0890000000000001e-06 2024-09-21T12:28:57.189 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.189 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.189 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:28:57.189 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793910+0000", 2024-09-21T12:28:57.189 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5080000000000003e-05 2024-09-21T12:28:57.189 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.189 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.189 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:28:57.189 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793921+0000", 2024-09-21T12:28:57.189 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1284000000000001e-05 2024-09-21T12:28:57.189 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.190 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.190 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:28:57.190 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793930+0000", 2024-09-21T12:28:57.190 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5429999999999994e-06 2024-09-21T12:28:57.190 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.190 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.190 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:28:57.190 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.801012+0000", 2024-09-21T12:28:57.190 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0070825219999999999 2024-09-21T12:28:57.190 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.190 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.190 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:28:57.190 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.801039+0000", 2024-09-21T12:28:57.191 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7248999999999999e-05 2024-09-21T12:28:57.191 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.191 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.191 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:28:57.191 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842368+0000", 2024-09-21T12:28:57.191 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.041328831000000003 2024-09-21T12:28:57.191 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.191 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.191 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:28:57.191 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842415+0000", 2024-09-21T12:28:57.191 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6418000000000002e-05 2024-09-21T12:28:57.191 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.191 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:28:57.191 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.192 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.192 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.192 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82448 7.8 e115/112 7:11700dea:::benchmark_data_smithi114_78432_object82447:head v 115'5250, mlcod=88'255)", 2024-09-21T12:28:57.192 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.796531+0000", 2024-09-21T12:28:57.192 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 381.34388288700001, 2024-09-21T12:28:57.192 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.045928499999999997, 2024-09-21T12:28:57.192 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:28:57.192 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:28:57.192 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:57.192 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:28:57.192 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.192 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:28:57.192 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796531+0000", 2024-09-21T12:28:57.192 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.192 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.193 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.193 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:28:57.193 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796531+0000", 2024-09-21T12:28:57.193 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.193 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.193 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.193 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:28:57.193 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796532+0000", 2024-09-21T12:28:57.193 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7199999999999999e-07 2024-09-21T12:28:57.193 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.193 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.193 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:28:57.193 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796647+0000", 2024-09-21T12:28:57.193 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000115097 2024-09-21T12:28:57.193 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.194 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.194 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:28:57.194 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796660+0000", 2024-09-21T12:28:57.194 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3271999999999999e-05 2024-09-21T12:28:57.194 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.194 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.194 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:28:57.194 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796666+0000", 2024-09-21T12:28:57.194 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0170000000000002e-06 2024-09-21T12:28:57.194 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.194 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.194 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:28:57.194 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.809350+0000", 2024-09-21T12:28:57.194 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.012683379 2024-09-21T12:28:57.194 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.195 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.195 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:28:57.195 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.809374+0000", 2024-09-21T12:28:57.195 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4417e-05 2024-09-21T12:28:57.195 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.195 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.195 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:28:57.195 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842417+0000", 2024-09-21T12:28:57.195 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.033042646000000002 2024-09-21T12:28:57.195 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.195 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.195 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:28:57.195 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842460+0000", 2024-09-21T12:28:57.195 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3099999999999997e-05 2024-09-21T12:28:57.196 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.196 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:28:57.196 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.196 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.196 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.196 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82473 7.8 e115/112 7:15016882:::benchmark_data_smithi114_78432_object82472:head v 115'5254, mlcod=88'255)", 2024-09-21T12:28:57.196 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.818965+0000", 2024-09-21T12:28:57.196 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 381.32144929399999, 2024-09-21T12:28:57.196 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.045024287000000003, 2024-09-21T12:28:57.196 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:28:57.196 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:28:57.196 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:57.196 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:28:57.196 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.197 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:28:57.197 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.818965+0000", 2024-09-21T12:28:57.197 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.197 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.197 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.197 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:28:57.197 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.818965+0000", 2024-09-21T12:28:57.197 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.197 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.197 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.197 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:28:57.197 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.818966+0000", 2024-09-21T12:28:57.197 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9199999999999999e-07 2024-09-21T12:28:57.197 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.197 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.198 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:28:57.198 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.819001+0000", 2024-09-21T12:28:57.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:57.199 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:57.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:57.200 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5117999999999998e-05 2024-09-21T12:28:57.200 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.200 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.200 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:28:57.200 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.819014+0000", 2024-09-21T12:28:57.200 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2869e-05 2024-09-21T12:28:57.200 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.200 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.200 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:28:57.200 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.819020+0000", 2024-09-21T12:28:57.200 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6959999999999999e-06 2024-09-21T12:28:57.201 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.201 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.201 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:28:57.201 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.836164+0000", 2024-09-21T12:28:57.201 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.017143617 2024-09-21T12:28:57.201 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.201 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.201 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:28:57.201 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.836188+0000", 2024-09-21T12:28:57.201 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4074e-05 2024-09-21T12:28:57.201 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.201 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.201 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:28:57.201 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.863879+0000", 2024-09-21T12:28:57.202 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.027690392000000001 2024-09-21T12:28:57.202 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.202 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.202 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:28:57.202 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.863989+0000", 2024-09-21T12:28:57.202 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000110629 2024-09-21T12:28:57.202 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.202 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:28:57.202 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.202 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.202 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.202 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82539 7.8 e115/112 7:1cd4c60d:::benchmark_data_smithi114_78432_object82538:head v 115'5258, mlcod=88'256)", 2024-09-21T12:28:57.202 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.889845+0000", 2024-09-21T12:28:57.202 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 381.250568891, 2024-09-21T12:28:57.203 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.046325071000000002, 2024-09-21T12:28:57.203 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:28:57.203 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:28:57.203 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:57.203 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:28:57.203 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.203 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:28:57.203 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889845+0000", 2024-09-21T12:28:57.203 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.203 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.203 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.203 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:28:57.203 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889845+0000", 2024-09-21T12:28:57.203 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.204 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.204 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.204 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:28:57.204 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889846+0000", 2024-09-21T12:28:57.204 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1800000000000004e-07 2024-09-21T12:28:57.204 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.204 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.204 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:28:57.204 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889878+0000", 2024-09-21T12:28:57.204 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1619999999999999e-05 2024-09-21T12:28:57.204 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.204 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.204 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:28:57.204 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889893+0000", 2024-09-21T12:28:57.204 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4819e-05 2024-09-21T12:28:57.205 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.205 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.205 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:28:57.205 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889900+0000", 2024-09-21T12:28:57.205 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8510000000000001e-06 2024-09-21T12:28:57.205 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.205 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.205 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:28:57.205 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.898101+0000", 2024-09-21T12:28:57.205 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0082018119999999993 2024-09-21T12:28:57.205 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.205 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.205 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:28:57.205 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.898164+0000", 2024-09-21T12:28:57.206 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2756999999999996e-05 2024-09-21T12:28:57.206 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.206 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.206 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:28:57.206 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.936000+0000", 2024-09-21T12:28:57.206 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.037835720000000003 2024-09-21T12:28:57.206 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.206 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.206 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:28:57.206 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.936170+0000", 2024-09-21T12:28:57.206 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000170574 2024-09-21T12:28:57.206 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.206 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:28:57.206 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.206 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.207 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.207 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82545 7.8 e115/112 7:1ddea6d1:::benchmark_data_smithi114_78432_object82544:head v 115'5259, mlcod=88'256)", 2024-09-21T12:28:57.207 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.897417+0000", 2024-09-21T12:28:57.207 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 381.24299735400001, 2024-09-21T12:28:57.207 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.038827027, 2024-09-21T12:28:57.207 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:28:57.207 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:28:57.207 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:57.207 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:28:57.207 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.207 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:28:57.207 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897417+0000", 2024-09-21T12:28:57.207 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.207 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.208 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.208 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:28:57.208 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897417+0000", 2024-09-21T12:28:57.208 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.208 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.208 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.208 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:28:57.208 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897418+0000", 2024-09-21T12:28:57.208 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3500000000000005e-07 2024-09-21T12:28:57.208 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.208 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.208 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:28:57.208 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897462+0000", 2024-09-21T12:28:57.208 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4036999999999999e-05 2024-09-21T12:28:57.209 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.209 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.209 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:28:57.209 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897472+0000", 2024-09-21T12:28:57.209 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0669999999999999e-05 2024-09-21T12:28:57.209 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.209 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.209 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:28:57.209 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897479+0000", 2024-09-21T12:28:57.209 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9190000000000002e-06 2024-09-21T12:28:57.209 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.209 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.209 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:28:57.209 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.905524+0000", 2024-09-21T12:28:57.210 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0080444899999999996 2024-09-21T12:28:57.210 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.210 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.210 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:28:57.210 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.905597+0000", 2024-09-21T12:28:57.210 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3539000000000002e-05 2024-09-21T12:28:57.210 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.210 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.210 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:28:57.210 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.936176+0000", 2024-09-21T12:28:57.210 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.030578893999999999 2024-09-21T12:28:57.210 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.210 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.210 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:28:57.211 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.936244+0000", 2024-09-21T12:28:57.211 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7543e-05 2024-09-21T12:28:57.211 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.211 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:28:57.211 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.211 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.211 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.211 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:83346 7.8 e115/112 7:1517ab05:::benchmark_data_smithi114_78432_object83345:head v 115'5319, mlcod=88'282)", 2024-09-21T12:28:57.211 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:36.700133+0000", 2024-09-21T12:28:57.211 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 380.44028145300001, 2024-09-21T12:28:57.211 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.044693733999999999, 2024-09-21T12:28:57.211 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:28:57.211 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:28:57.211 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:57.212 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:28:57.212 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.212 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:28:57.212 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700133+0000", 2024-09-21T12:28:57.212 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.212 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.212 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.212 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:28:57.212 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700133+0000", 2024-09-21T12:28:57.212 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.212 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.212 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.212 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:28:57.212 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700133+0000", 2024-09-21T12:28:57.213 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4300000000000003e-07 2024-09-21T12:28:57.213 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.213 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.213 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:28:57.213 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700182+0000", 2024-09-21T12:28:57.213 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8433000000000002e-05 2024-09-21T12:28:57.213 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.213 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.213 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:28:57.213 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700192+0000", 2024-09-21T12:28:57.213 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0382e-05 2024-09-21T12:28:57.213 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.213 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.213 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:28:57.214 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700201+0000", 2024-09-21T12:28:57.214 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9819999999999997e-06 2024-09-21T12:28:57.214 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.214 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.214 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:28:57.214 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.703063+0000", 2024-09-21T12:28:57.214 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002862121 2024-09-21T12:28:57.214 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.214 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.214 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:28:57.214 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.703087+0000", 2024-09-21T12:28:57.214 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3730000000000001e-05 2024-09-21T12:28:57.214 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.214 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.215 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:28:57.215 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.744677+0000", 2024-09-21T12:28:57.215 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.041589785999999997 2024-09-21T12:28:57.215 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.215 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.215 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:28:57.215 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.744826+0000", 2024-09-21T12:28:57.215 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014965700000000001 2024-09-21T12:28:57.215 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.215 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:28:57.215 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.215 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.215 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.215 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:87464 7.8 e117/112 7:17a0a89d:::benchmark_data_smithi114_78432_object87463:head v 117'5593, mlcod=88'417)", 2024-09-21T12:28:57.216 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:41.463661+0000", 2024-09-21T12:28:57.216 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 375.67675351999998, 2024-09-21T12:28:57.216 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.067076070000000002, 2024-09-21T12:28:57.216 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:28:57.216 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:28:57.216 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:57.216 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:28:57.216 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.216 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:28:57.216 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463661+0000", 2024-09-21T12:28:57.216 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.216 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.216 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.216 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:28:57.217 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463661+0000", 2024-09-21T12:28:57.217 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.217 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.217 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.217 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:28:57.217 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463661+0000", 2024-09-21T12:28:57.217 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2799999999999995e-07 2024-09-21T12:28:57.217 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.217 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.217 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:28:57.217 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463845+0000", 2024-09-21T12:28:57.217 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000183692 2024-09-21T12:28:57.217 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.217 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.218 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:28:57.218 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463854+0000", 2024-09-21T12:28:57.218 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0750000000000004e-06 2024-09-21T12:28:57.218 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.218 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.218 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:28:57.218 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463860+0000", 2024-09-21T12:28:57.218 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.015e-06 2024-09-21T12:28:57.218 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.218 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.218 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:28:57.218 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.480218+0000", 2024-09-21T12:28:57.218 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.016357289000000001 2024-09-21T12:28:57.218 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.219 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.219 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:28:57.219 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.480239+0000", 2024-09-21T12:28:57.219 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1691999999999998e-05 2024-09-21T12:28:57.219 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.219 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.219 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:28:57.219 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.530695+0000", 2024-09-21T12:28:57.219 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.050456050000000002 2024-09-21T12:28:57.219 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.219 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.219 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:28:57.219 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.530737+0000", 2024-09-21T12:28:57.219 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1428999999999999e-05 2024-09-21T12:28:57.220 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.220 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:28:57.220 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.220 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.220 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.220 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:87470 7.8 e117/112 7:1f3b224f:::benchmark_data_smithi114_78432_object87469:head v 117'5594, mlcod=88'417)", 2024-09-21T12:28:57.220 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:41.472523+0000", 2024-09-21T12:28:57.220 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 375.66789152199999, 2024-09-21T12:28:57.220 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.06343451, 2024-09-21T12:28:57.220 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:28:57.220 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:28:57.220 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:57.220 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:28:57.220 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.221 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:28:57.221 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472523+0000", 2024-09-21T12:28:57.221 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.221 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.221 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.221 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:28:57.221 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472523+0000", 2024-09-21T12:28:57.221 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.221 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.221 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.221 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:28:57.221 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472524+0000", 2024-09-21T12:28:57.221 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7399999999999991e-07 2024-09-21T12:28:57.221 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.222 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.222 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:28:57.222 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472596+0000", 2024-09-21T12:28:57.222 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2152999999999998e-05 2024-09-21T12:28:57.222 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.222 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.222 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:28:57.222 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472606+0000", 2024-09-21T12:28:57.222 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0485e-05 2024-09-21T12:28:57.222 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.222 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.222 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:28:57.222 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472613+0000", 2024-09-21T12:28:57.222 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5590000000000002e-06 2024-09-21T12:28:57.223 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.223 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.223 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:28:57.223 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.484029+0000", 2024-09-21T12:28:57.223 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.01141587 2024-09-21T12:28:57.223 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.223 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.223 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:28:57.223 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.484053+0000", 2024-09-21T12:28:57.223 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4559999999999999e-05 2024-09-21T12:28:57.223 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.223 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.223 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:28:57.223 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.535915+0000", 2024-09-21T12:28:57.224 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.051861431 2024-09-21T12:28:57.224 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.224 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.224 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:28:57.224 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.535957+0000", 2024-09-21T12:28:57.224 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2478000000000002e-05 2024-09-21T12:28:57.224 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.224 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:28:57.224 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.224 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.224 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.224 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:87486 7.8 e117/112 7:14e05f7b:::benchmark_data_smithi114_78432_object87485:head v 117'5595, mlcod=88'417)", 2024-09-21T12:28:57.224 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:41.475017+0000", 2024-09-21T12:28:57.224 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 375.66539718600001, 2024-09-21T12:28:57.225 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.060970067000000003, 2024-09-21T12:28:57.225 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:28:57.225 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:28:57.225 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:57.225 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:28:57.225 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.225 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:28:57.225 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475017+0000", 2024-09-21T12:28:57.225 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.225 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.225 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.225 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:28:57.225 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475017+0000", 2024-09-21T12:28:57.225 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.226 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.226 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.226 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:28:57.226 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475018+0000", 2024-09-21T12:28:57.226 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5799999999999999e-07 2024-09-21T12:28:57.226 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.226 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.226 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:28:57.226 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475150+0000", 2024-09-21T12:28:57.226 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013201700000000001 2024-09-21T12:28:57.226 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.226 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.226 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:28:57.226 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475159+0000", 2024-09-21T12:28:57.227 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5840000000000001e-06 2024-09-21T12:28:57.227 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.227 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.227 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:28:57.227 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475166+0000", 2024-09-21T12:28:57.227 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3589999999999996e-06 2024-09-21T12:28:57.227 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.227 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.227 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:28:57.227 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.525196+0000", 2024-09-21T12:28:57.227 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.050030507000000002 2024-09-21T12:28:57.227 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.227 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.227 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:28:57.227 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.525214+0000", 2024-09-21T12:28:57.228 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7884e-05 2024-09-21T12:28:57.228 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.228 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.228 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:28:57.228 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.535960+0000", 2024-09-21T12:28:57.228 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.010746143999999999 2024-09-21T12:28:57.228 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.228 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.228 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:28:57.228 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.535987+0000", 2024-09-21T12:28:57.228 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6914e-05 2024-09-21T12:28:57.228 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.228 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:28:57.228 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.229 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.229 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.229 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDPGPush(7.8 117/112 [PushOp(7:1de1c2d4:::benchmark_data_smithi114_78432_object6336:head, version: 88'418, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:1de1c2d4:::benchmark_data_smithi114_78432_object6336:head@88'418, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T12:28:57.229 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:41.485571+0000", 2024-09-21T12:28:57.229 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 375.65484329100002, 2024-09-21T12:28:57.229 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.046712901000000001, 2024-09-21T12:28:57.229 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:28:57.229 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:28:57.229 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:28:57.229 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:28:57.229 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.229 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:28:57.229 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485571+0000", 2024-09-21T12:28:57.229 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.230 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.230 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.230 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:28:57.230 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485571+0000", 2024-09-21T12:28:57.230 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.230 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.230 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.230 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:28:57.230 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485572+0000", 2024-09-21T12:28:57.230 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7299999999999995e-07 2024-09-21T12:28:57.230 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.230 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.230 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:28:57.230 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485643+0000", 2024-09-21T12:28:57.230 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0976999999999995e-05 2024-09-21T12:28:57.231 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.231 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.231 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:28:57.231 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485655+0000", 2024-09-21T12:28:57.231 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2904e-05 2024-09-21T12:28:57.231 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.231 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.231 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:28:57.231 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485665+0000", 2024-09-21T12:28:57.231 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6709999999999994e-06 2024-09-21T12:28:57.231 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.231 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.231 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:28:57.231 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.530794+0000", 2024-09-21T12:28:57.232 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.045129295 2024-09-21T12:28:57.232 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.232 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.232 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:28:57.232 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.530802+0000", 2024-09-21T12:28:57.232 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5059999999999998e-06 2024-09-21T12:28:57.232 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.232 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.232 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:28:57.232 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.532284+0000", 2024-09-21T12:28:57.232 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014818749999999999 2024-09-21T12:28:57.232 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.232 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:28:57.232 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.233 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.233 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.233 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:87497 7.8 e117/112 7:19475f57:::benchmark_data_smithi114_78432_object87496:head v 117'5596, mlcod=88'417)", 2024-09-21T12:28:57.233 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:41.494916+0000", 2024-09-21T12:28:57.233 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 375.64549781199997, 2024-09-21T12:28:57.233 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.041122084000000003, 2024-09-21T12:28:57.233 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:28:57.233 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:28:57.233 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:28:57.233 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:28:57.233 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.233 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:28:57.233 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.494916+0000", 2024-09-21T12:28:57.233 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.233 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.234 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.234 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:28:57.234 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.494916+0000", 2024-09-21T12:28:57.234 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:57.234 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.234 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.234 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:28:57.234 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.494918+0000", 2024-09-21T12:28:57.234 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.719e-06 2024-09-21T12:28:57.234 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.234 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.234 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:28:57.234 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.494975+0000", 2024-09-21T12:28:57.235 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6981999999999998e-05 2024-09-21T12:28:57.235 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.235 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.235 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:28:57.235 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.525059+0000", 2024-09-21T12:28:57.235 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.030083526999999999 2024-09-21T12:28:57.235 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.235 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.235 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:28:57.235 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.525071+0000", 2024-09-21T12:28:57.235 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2438e-05 2024-09-21T12:28:57.235 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.235 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.235 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:28:57.236 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.532312+0000", 2024-09-21T12:28:57.236 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0072404909999999999 2024-09-21T12:28:57.236 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.236 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.236 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:28:57.236 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.532336+0000", 2024-09-21T12:28:57.236 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4090000000000001e-05 2024-09-21T12:28:57.236 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.236 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.236 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:28:57.236 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.536013+0000", 2024-09-21T12:28:57.236 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003677825 2024-09-21T12:28:57.236 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:57.236 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:57.236 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:28:57.237 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.536038+0000", 2024-09-21T12:28:57.237 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5012e-05 2024-09-21T12:28:57.237 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.237 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:28:57.237 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.237 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:57.237 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:28:57.237 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:28:57.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:56 smithi183 ceph-mon[33277]: pgmap v427: 178 pgs: 178 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 777 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-09-21T12:28:57.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:56 smithi183 ceph-mon[33277]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:28:57.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:56 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:28:57.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:56 smithi183 ceph-mon[33277]: osdmap e255: 8 total, 8 up, 7 in 2024-09-21T12:28:57.239 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:28:57 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:28:57.243 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:57.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:56 smithi114 ceph-mon[30324]: pgmap v427: 178 pgs: 178 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 777 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-09-21T12:28:57.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:56 smithi114 ceph-mon[30324]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:28:57.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:56 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:28:57.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:56 smithi114 ceph-mon[30324]: osdmap e255: 8 total, 8 up, 7 in 2024-09-21T12:28:57.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:56 smithi114 ceph-mon[40867]: pgmap v427: 178 pgs: 178 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 777 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-09-21T12:28:57.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:56 smithi114 ceph-mon[40867]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:28:57.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:56 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:28:57.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:56 smithi114 ceph-mon[40867]: osdmap e255: 8 total, 8 up, 7 in 2024-09-21T12:28:57.300 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:57.347 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:57.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:57.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:57.448 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:57.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:57.483 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:57.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:57.584 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:57.613 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:57.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:57.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:57.714 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:57.744 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:57.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:57.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:57.828 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 30 16 58769 58753 122.388 89.6875 0.00236532 0.00816162 2024-09-21T12:28:57.845 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:57.871 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:57.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:57.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:57.973 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:58.003 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:58.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:58.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:58.058 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:28:58.105 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:58.127 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T12:28:58.133 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:58.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:58.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:58.235 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:58.268 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:58.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:58.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:58.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:57 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:28:58.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:57 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:28:58.370 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:28:58.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:58.418 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:58.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:28:58.481 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:28:58.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:57 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:28:58.518 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:58.543 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:58.548 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:58.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:58.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:58.630 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:28:58.633 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:28:58.649 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:58.675 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:58.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:58.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:58.777 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:58.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:58.811 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:58.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:58.828 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 31 16 60055 60039 121.033 80.375 0.0102622 0.00825614 2024-09-21T12:28:58.912 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:58.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:58.950 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:58.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:59.051 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:28:59.069 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:28:59.069 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:28:59.069 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:59.069 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:60399 9.a 9:5b5d39d9:::benchmark_data_smithi114_133214_object60398:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-21T12:28:59.069 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:28:59.064516+0000", 2024-09-21T12:28:59.070 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.003389881, 2024-09-21T12:28:59.070 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003420827, 2024-09-21T12:28:59.070 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:28:59.070 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:28:59.070 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:28:59.070 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:28:59.070 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:28:59.070 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:28:59.070 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 60399 2024-09-21T12:28:59.070 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:59.070 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:28:59.070 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:59.070 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:28:59.070 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:59.064516+0000", 2024-09-21T12:28:59.070 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:59.071 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:59.071 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:59.071 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:28:59.071 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:59.064516+0000", 2024-09-21T12:28:59.071 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:59.071 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:59.071 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:59.071 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:28:59.071 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:59.064517+0000", 2024-09-21T12:28:59.071 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1429999999999999e-06 2024-09-21T12:28:59.071 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:59.071 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:59.071 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:28:59.071 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:59.064546+0000", 2024-09-21T12:28:59.071 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9128999999999998e-05 2024-09-21T12:28:59.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:59.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:59.072 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:28:59.072 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:59.064556+0000", 2024-09-21T12:28:59.072 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9739999999999997e-06 2024-09-21T12:28:59.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:59.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:59.072 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:28:59.072 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:59.064561+0000", 2024-09-21T12:28:59.072 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0730000000000004e-06 2024-09-21T12:28:59.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:59.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:59.072 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:28:59.072 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:59.064610+0000", 2024-09-21T12:28:59.072 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8245999999999999e-05 2024-09-21T12:28:59.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:59.073 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:59.073 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:28:59.073 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:59.064745+0000", 2024-09-21T12:28:59.073 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013560300000000001 2024-09-21T12:28:59.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:59.073 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:59.073 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:28:59.073 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:59.064892+0000", 2024-09-21T12:28:59.073 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014635200000000001 2024-09-21T12:28:59.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:59.073 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:59.073 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:28:59.073 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:59.067163+0000", 2024-09-21T12:28:59.073 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0022715980000000001 2024-09-21T12:28:59.074 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:59.074 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:59.074 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:28:59.074 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:59.067396+0000", 2024-09-21T12:28:59.074 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00023285299999999999 2024-09-21T12:28:59.074 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:59.074 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:28:59.074 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:59.074 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:59.074 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:59.074 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:60403 9.f 9:f98bf3d5:::benchmark_data_smithi114_133214_object60402:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-21T12:28:59.074 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:28:59.066597+0000", 2024-09-21T12:28:59.074 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0013084189999999999, 2024-09-21T12:28:59.074 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001387101, 2024-09-21T12:28:59.075 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:28:59.075 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:28:59.075 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:28:59.075 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:28:59.075 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:28:59.075 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:28:59.075 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 60403 2024-09-21T12:28:59.075 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:59.075 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:28:59.075 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:59.075 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:28:59.075 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:59.066597+0000", 2024-09-21T12:28:59.075 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:59.075 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:59.075 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:59.075 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:28:59.076 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:59.066597+0000", 2024-09-21T12:28:59.076 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:28:59.076 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:59.076 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:59.076 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:28:59.076 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:59.066599+0000", 2024-09-21T12:28:59.076 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3790000000000001e-06 2024-09-21T12:28:59.076 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:59.076 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:59.076 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:28:59.076 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:59.066654+0000", 2024-09-21T12:28:59.076 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5210999999999997e-05 2024-09-21T12:28:59.076 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:59.076 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:59.077 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:28:59.077 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:59.066669+0000", 2024-09-21T12:28:59.077 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4477999999999999e-05 2024-09-21T12:28:59.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:59.077 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:59.077 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:28:59.077 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:59.066678+0000", 2024-09-21T12:28:59.077 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4380000000000001e-06 2024-09-21T12:28:59.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:59.077 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:59.077 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:28:59.077 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:59.066732+0000", 2024-09-21T12:28:59.077 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4305000000000003e-05 2024-09-21T12:28:59.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:59.077 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:59.078 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:28:59.078 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:59.066898+0000", 2024-09-21T12:28:59.078 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016611999999999999 2024-09-21T12:28:59.078 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:28:59.078 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:28:59.078 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:28:59.078 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:59.067040+0000", 2024-09-21T12:28:59.078 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014162699999999999 2024-09-21T12:28:59.078 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:59.078 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:28:59.078 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:59.078 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:28:59.078 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-21T12:28:59.078 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 2 2024-09-21T12:28:59.078 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:28:59.088 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:28:59.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:59.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:28:59.176 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:28:58 smithi114 ceph-mon[30324]: pgmap v429: 178 pgs: 178 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 125 MiB/s wr, 2.01k op/s 2024-09-21T12:28:59.177 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:28:58 smithi114 ceph-mon[40867]: pgmap v429: 178 pgs: 178 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 125 MiB/s wr, 2.01k op/s 2024-09-21T12:28:59.190 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:59.199 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T12:28:59.215 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:59.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:59.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:59.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:28:58 smithi183 ceph-mon[33277]: pgmap v429: 178 pgs: 178 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 125 MiB/s wr, 2.01k op/s 2024-09-21T12:28:59.317 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:59.345 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:59.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:59.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:59.370 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:28:59.371 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:28:59.419 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:28:59.420 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:28:59.447 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:59.453 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:28:59.453 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:28:59.473 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:59.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:59.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:59.492 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:28:59.492 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:28:59.525 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:28:59.525 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:28:59.557 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:28:59.557 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:28:59.575 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:28:59.588 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:28:59.588 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:28:59.603 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:59.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:59.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:28:59.622 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:28:59.622 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:28:59.705 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:28:59.735 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:59.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:59.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:28:59.795 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T12:28:59.827 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 32 16 61414 61398 119.905 84.9375 0.00514761 0.00833527 2024-09-21T12:28:59.837 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:28:59.872 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:28:59.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:28:59.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:28:59.974 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:00.021 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2024-09-21T12:28:57.738854+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3776912171,"num_objects":57819,"num_object_clones":0,"num_object_copies":173457,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57819,"num_whiteouts":0,"num_read":1229,"num_read_kb":981,"num_write":58056,"num_write_kb":3687725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":218,"num_bytes_recovered":4424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60560,"ondisk_log_size":60560,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":534,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":13295624,"kb_used_data":10937968,"kb_used_omap":10,"kb_used_meta":2357493,"kb_avail":642920440,"statfs":{"total":671965249536,"available":658350530560,"internally_reserved":0,"allocated":11200479232,"data_stored":11196762533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":2414072939},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":788922368,"num_objects":12038,"num_object_clones":0,"num_object_copies":36114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12038,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":12038,"num_write_kb":770432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001940"},"pg_stats":[{"pgid":"1.1d","version":"76'79","reported_seq":101,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986226+0000","last_change":"2024-09-21T12:28:47.816204+0000","last_active":"2024-09-21T12:28:55.986226+0000","last_peered":"2024-09-21T12:28:55.986226+0000","last_clean":"2024-09-21T12:28:55.986226+0000","last_became_active":"2024-09-21T12:27:16.947651+0000","last_became_peered":"2024-09-21T12:27:16.947651+0000","last_unstale":"2024-09-21T12:28:55.986226+0000","last_undegraded":"2024-09-21T12:28:55.986226+0000","last_fullsized":"2024-09-21T12:28:55.986226+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:47.816137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:47.816137+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:55.307153+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":424,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.995515+0000","last_change":"2024-09-21T12:24:50.115300+0000","last_active":"2024-09-21T12:28:55.995515+0000","last_peered":"2024-09-21T12:28:55.995515+0000","last_clean":"2024-09-21T12:28:55.995515+0000","last_became_active":"2024-09-21T12:20:15.309431+0000","last_became_peered":"2024-09-21T12:20:15.309431+0000","last_unstale":"2024-09-21T12:28:55.995515+0000","last_undegraded":"2024-09-21T12:28:55.995515+0000","last_fullsized":"2024-09-21T12:28:55.995515+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:50.115253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:50.115253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:18.674292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":401,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.993178+0000","last_change":"2024-09-21T12:27:32.886429+0000","last_active":"2024-09-21T12:28:55.993178+0000","last_peered":"2024-09-21T12:28:55.993178+0000","last_clean":"2024-09-21T12:28:55.993178+0000","last_became_active":"2024-09-21T12:22:50.909699+0000","last_became_peered":"2024-09-21T12:22:50.909699+0000","last_unstale":"2024-09-21T12:28:55.993178+0000","last_undegraded":"2024-09-21T12:28:55.993178+0000","last_fullsized":"2024-09-21T12:28:55.993178+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:27:32.886351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:32.886351+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":413,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986003+0000","last_change":"2024-09-21T12:24:34.311501+0000","last_active":"2024-09-21T12:28:55.986003+0000","last_peered":"2024-09-21T12:28:55.986003+0000","last_clean":"2024-09-21T12:28:55.986003+0000","last_became_active":"2024-09-21T12:20:19.314908+0000","last_became_peered":"2024-09-21T12:20:19.314908+0000","last_unstale":"2024-09-21T12:28:55.986003+0000","last_undegraded":"2024-09-21T12:28:55.986003+0000","last_fullsized":"2024-09-21T12:28:55.986003+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:34.311451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:24:34.311451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:08.657850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":337,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.987766+0000","last_change":"2024-09-21T12:22:50.914064+0000","last_active":"2024-09-21T12:28:55.987766+0000","last_peered":"2024-09-21T12:28:55.987766+0000","last_clean":"2024-09-21T12:28:55.987766+0000","last_became_active":"2024-09-21T12:22:50.913743+0000","last_became_peered":"2024-09-21T12:22:50.913743+0000","last_unstale":"2024-09-21T12:28:55.987766+0000","last_undegraded":"2024-09-21T12:28:55.987766+0000","last_fullsized":"2024-09-21T12:28:55.987766+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:58.189665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:21:58.189665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:13.364470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1c","version":"76'79","reported_seq":91,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986276+0000","last_change":"2024-09-21T12:27:28.374689+0000","last_active":"2024-09-21T12:28:55.986276+0000","last_peered":"2024-09-21T12:28:55.986276+0000","last_clean":"2024-09-21T12:28:55.986276+0000","last_became_active":"2024-09-21T12:27:16.920545+0000","last_became_peered":"2024-09-21T12:27:16.920545+0000","last_unstale":"2024-09-21T12:28:55.986276+0000","last_undegraded":"2024-09-21T12:28:55.986276+0000","last_fullsized":"2024-09-21T12:28:55.986276+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:28.374616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:28.374616+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":400,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.984416+0000","last_change":"2024-09-21T12:25:48.316122+0000","last_active":"2024-09-21T12:28:55.984416+0000","last_peered":"2024-09-21T12:28:55.984416+0000","last_clean":"2024-09-21T12:28:55.984416+0000","last_became_active":"2024-09-21T12:20:21.595301+0000","last_became_peered":"2024-09-21T12:20:21.595301+0000","last_unstale":"2024-09-21T12:28:55.984416+0000","last_undegraded":"2024-09-21T12:28:55.984416+0000","last_fullsized":"2024-09-21T12:28:55.984416+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:48.316039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:48.316039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:55.122483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":421,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.985536+0000","last_change":"2024-09-21T12:24:49.701972+0000","last_active":"2024-09-21T12:28:55.985536+0000","last_peered":"2024-09-21T12:28:55.985536+0000","last_clean":"2024-09-21T12:28:55.985536+0000","last_became_active":"2024-09-21T12:20:15.308655+0000","last_became_peered":"2024-09-21T12:20:15.308655+0000","last_unstale":"2024-09-21T12:28:55.985536+0000","last_undegraded":"2024-09-21T12:28:55.985536+0000","last_fullsized":"2024-09-21T12:28:55.985536+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:49.701915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:49.701915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:12.598978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":449,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.987202+0000","last_change":"2024-09-21T12:27:54.371491+0000","last_active":"2024-09-21T12:28:55.987202+0000","last_peered":"2024-09-21T12:28:55.987202+0000","last_clean":"2024-09-21T12:28:55.987202+0000","last_became_active":"2024-09-21T12:20:17.305220+0000","last_became_peered":"2024-09-21T12:20:17.305220+0000","last_unstale":"2024-09-21T12:28:55.987202+0000","last_undegraded":"2024-09-21T12:28:55.987202+0000","last_fullsized":"2024-09-21T12:28:55.987202+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:27:54.369084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:54.369084+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:02.669546+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":363,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.993277+0000","last_change":"2024-09-21T12:28:39.179737+0000","last_active":"2024-09-21T12:28:55.993277+0000","last_peered":"2024-09-21T12:28:55.993277+0000","last_clean":"2024-09-21T12:28:55.993277+0000","last_became_active":"2024-09-21T12:22:50.926240+0000","last_became_peered":"2024-09-21T12:22:50.926240+0000","last_unstale":"2024-09-21T12:28:55.993277+0000","last_undegraded":"2024-09-21T12:28:55.993277+0000","last_fullsized":"2024-09-21T12:28:55.993277+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:39.179665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:39.179665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:55.735084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":258,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.993765+0000","last_change":"2024-09-21T12:26:54.084511+0000","last_active":"2024-09-21T12:28:55.993765+0000","last_peered":"2024-09-21T12:28:55.993765+0000","last_clean":"2024-09-21T12:28:55.993765+0000","last_became_active":"2024-09-21T12:26:54.083981+0000","last_became_peered":"2024-09-21T12:26:54.083981+0000","last_unstale":"2024-09-21T12:28:55.993765+0000","last_undegraded":"2024-09-21T12:28:55.993765+0000","last_fullsized":"2024-09-21T12:28:55.993765+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:19.446251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:19.446251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:51.995762+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":409,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.993374+0000","last_change":"2024-09-21T12:28:06.589984+0000","last_active":"2024-09-21T12:28:55.993374+0000","last_peered":"2024-09-21T12:28:55.993374+0000","last_clean":"2024-09-21T12:28:55.993374+0000","last_became_active":"2024-09-21T12:22:50.927804+0000","last_became_peered":"2024-09-21T12:22:50.927804+0000","last_unstale":"2024-09-21T12:28:55.993374+0000","last_undegraded":"2024-09-21T12:28:55.993374+0000","last_fullsized":"2024-09-21T12:28:55.993374+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:28:06.589926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:06.589926+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:33.744896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":421,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.988042+0000","last_change":"2024-09-21T12:26:54.497247+0000","last_active":"2024-09-21T12:28:55.988042+0000","last_peered":"2024-09-21T12:28:55.988042+0000","last_clean":"2024-09-21T12:28:55.988042+0000","last_became_active":"2024-09-21T12:20:19.308184+0000","last_became_peered":"2024-09-21T12:20:19.308184+0000","last_unstale":"2024-09-21T12:28:55.988042+0000","last_undegraded":"2024-09-21T12:28:55.988042+0000","last_fullsized":"2024-09-21T12:28:55.988042+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:54.497175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:54.497175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:20.364274+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":419,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.988142+0000","last_change":"2024-09-21T12:27:54.959865+0000","last_active":"2024-09-21T12:28:55.988142+0000","last_peered":"2024-09-21T12:28:55.988142+0000","last_clean":"2024-09-21T12:28:55.988142+0000","last_became_active":"2024-09-21T12:20:21.309190+0000","last_became_peered":"2024-09-21T12:20:21.309190+0000","last_unstale":"2024-09-21T12:28:55.988142+0000","last_undegraded":"2024-09-21T12:28:55.988142+0000","last_fullsized":"2024-09-21T12:28:55.988142+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:54.959806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:27:54.959806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:15.995385+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"76'79","reported_seq":184,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.985353+0000","last_change":"2024-09-21T12:27:20.216167+0000","last_active":"2024-09-21T12:28:55.985353+0000","last_peered":"2024-09-21T12:28:55.985353+0000","last_clean":"2024-09-21T12:28:55.985353+0000","last_became_active":"2024-09-21T12:27:16.937711+0000","last_became_peered":"2024-09-21T12:27:16.937711+0000","last_unstale":"2024-09-21T12:28:55.985353+0000","last_undegraded":"2024-09-21T12:28:55.985353+0000","last_fullsized":"2024-09-21T12:28:55.985353+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:20.216102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:20.216102+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":266,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986989+0000","last_change":"2024-09-21T12:28:33.644630+0000","last_active":"2024-09-21T12:28:55.986989+0000","last_peered":"2024-09-21T12:28:55.986989+0000","last_clean":"2024-09-21T12:28:55.986989+0000","last_became_active":"2024-09-21T12:26:54.085055+0000","last_became_peered":"2024-09-21T12:26:54.085055+0000","last_unstale":"2024-09-21T12:28:55.986989+0000","last_undegraded":"2024-09-21T12:28:55.986989+0000","last_fullsized":"2024-09-21T12:28:55.986989+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:33.644570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:33.644570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:00.099267+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":387,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.984727+0000","last_change":"2024-09-21T12:24:28.624430+0000","last_active":"2024-09-21T12:28:55.984727+0000","last_peered":"2024-09-21T12:28:55.984727+0000","last_clean":"2024-09-21T12:28:55.984727+0000","last_became_active":"2024-09-21T12:22:50.918794+0000","last_became_peered":"2024-09-21T12:22:50.918794+0000","last_unstale":"2024-09-21T12:28:55.984727+0000","last_undegraded":"2024-09-21T12:28:55.984727+0000","last_fullsized":"2024-09-21T12:28:55.984727+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-21T12:24:28.624364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:28.624364+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:50.980764+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":434,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.985685+0000","last_change":"2024-09-21T12:28:04.623499+0000","last_active":"2024-09-21T12:28:55.985685+0000","last_peered":"2024-09-21T12:28:55.985685+0000","last_clean":"2024-09-21T12:28:55.985685+0000","last_became_active":"2024-09-21T12:22:50.921780+0000","last_became_peered":"2024-09-21T12:22:50.921780+0000","last_unstale":"2024-09-21T12:28:55.985685+0000","last_undegraded":"2024-09-21T12:28:55.985685+0000","last_fullsized":"2024-09-21T12:28:55.985685+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:04.623389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:04.623389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:13.184696+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":425,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.987492+0000","last_change":"2024-09-21T12:28:38.237336+0000","last_active":"2024-09-21T12:28:55.987492+0000","last_peered":"2024-09-21T12:28:55.987492+0000","last_clean":"2024-09-21T12:28:55.987492+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:28:55.987492+0000","last_undegraded":"2024-09-21T12:28:55.987492+0000","last_fullsized":"2024-09-21T12:28:55.987492+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:38.237289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:38.237289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:50.035648+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"76'79","reported_seq":255,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.984149+0000","last_change":"2024-09-21T12:27:22.289065+0000","last_active":"2024-09-21T12:28:55.984149+0000","last_peered":"2024-09-21T12:28:55.984149+0000","last_clean":"2024-09-21T12:28:55.984149+0000","last_became_active":"2024-09-21T12:27:16.957569+0000","last_became_peered":"2024-09-21T12:27:16.957569+0000","last_unstale":"2024-09-21T12:28:55.984149+0000","last_undegraded":"2024-09-21T12:28:55.984149+0000","last_fullsized":"2024-09-21T12:28:55.984149+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:22.288989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:22.288989+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":362,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.993862+0000","last_change":"2024-09-21T12:26:44.845249+0000","last_active":"2024-09-21T12:28:55.993862+0000","last_peered":"2024-09-21T12:28:55.993862+0000","last_clean":"2024-09-21T12:28:55.993862+0000","last_became_active":"2024-09-21T12:22:50.914792+0000","last_became_peered":"2024-09-21T12:22:50.914792+0000","last_unstale":"2024-09-21T12:28:55.993862+0000","last_undegraded":"2024-09-21T12:28:55.993862+0000","last_fullsized":"2024-09-21T12:28:55.993862+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:44.845176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:44.845176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:12.195899+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":452,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.997207+0000","last_change":"2024-09-21T12:24:30.033946+0000","last_active":"2024-09-21T12:28:55.997207+0000","last_peered":"2024-09-21T12:28:55.997207+0000","last_clean":"2024-09-21T12:28:55.997207+0000","last_became_active":"2024-09-21T12:22:50.921576+0000","last_became_peered":"2024-09-21T12:22:50.921576+0000","last_unstale":"2024-09-21T12:28:55.997207+0000","last_undegraded":"2024-09-21T12:28:55.997207+0000","last_fullsized":"2024-09-21T12:28:55.997207+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:30.033874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:30.033874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:14.654620+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":451,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.981223+0000","last_change":"2024-09-21T12:27:22.516532+0000","last_active":"2024-09-21T12:28:55.981223+0000","last_peered":"2024-09-21T12:28:55.981223+0000","last_clean":"2024-09-21T12:28:55.981223+0000","last_became_active":"2024-09-21T12:20:17.307828+0000","last_became_peered":"2024-09-21T12:20:17.307828+0000","last_unstale":"2024-09-21T12:28:55.981223+0000","last_undegraded":"2024-09-21T12:28:55.981223+0000","last_fullsized":"2024-09-21T12:28:55.981223+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:27:22.516442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:22.516442+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:55.268380+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":449,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986603+0000","last_change":"2024-09-21T12:28:06.636769+0000","last_active":"2024-09-21T12:28:55.986603+0000","last_peered":"2024-09-21T12:28:55.986603+0000","last_clean":"2024-09-21T12:28:55.986603+0000","last_became_active":"2024-09-21T12:25:46.905878+0000","last_became_peered":"2024-09-21T12:25:46.905878+0000","last_unstale":"2024-09-21T12:28:55.986603+0000","last_undegraded":"2024-09-21T12:28:55.986603+0000","last_fullsized":"2024-09-21T12:28:55.986603+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:06.636729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:06.636729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:23.255350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"76'79","reported_seq":348,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.987355+0000","last_change":"2024-09-21T12:28:28.293163+0000","last_active":"2024-09-21T12:28:55.987355+0000","last_peered":"2024-09-21T12:28:55.987355+0000","last_clean":"2024-09-21T12:28:55.987355+0000","last_became_active":"2024-09-21T12:27:16.927791+0000","last_became_peered":"2024-09-21T12:27:16.927791+0000","last_unstale":"2024-09-21T12:28:55.987355+0000","last_undegraded":"2024-09-21T12:28:55.987355+0000","last_fullsized":"2024-09-21T12:28:55.987355+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:28.293087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:28.293087+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:44.932604+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":409,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.985475+0000","last_change":"2024-09-21T12:28:45.880599+0000","last_active":"2024-09-21T12:28:55.985475+0000","last_peered":"2024-09-21T12:28:55.985475+0000","last_clean":"2024-09-21T12:28:55.985475+0000","last_became_active":"2024-09-21T12:20:21.592796+0000","last_became_peered":"2024-09-21T12:20:21.592796+0000","last_unstale":"2024-09-21T12:28:55.985475+0000","last_undegraded":"2024-09-21T12:28:55.985475+0000","last_fullsized":"2024-09-21T12:28:55.985475+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:45.880383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.880383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:01.380533+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":360,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.985915+0000","last_change":"2024-09-21T12:28:45.726096+0000","last_active":"2024-09-21T12:28:55.985915+0000","last_peered":"2024-09-21T12:28:55.985915+0000","last_clean":"2024-09-21T12:28:55.985915+0000","last_became_active":"2024-09-21T12:22:50.925403+0000","last_became_peered":"2024-09-21T12:22:50.925403+0000","last_unstale":"2024-09-21T12:28:55.985915+0000","last_undegraded":"2024-09-21T12:28:55.985915+0000","last_fullsized":"2024-09-21T12:28:55.985915+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:45.725968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.725968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:08.835537+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":504,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.984639+0000","last_change":"2024-09-21T12:28:30.620035+0000","last_active":"2024-09-21T12:28:55.984639+0000","last_peered":"2024-09-21T12:28:55.984639+0000","last_clean":"2024-09-21T12:28:55.984639+0000","last_became_active":"2024-09-21T12:22:50.912144+0000","last_became_peered":"2024-09-21T12:22:50.912144+0000","last_unstale":"2024-09-21T12:28:55.984639+0000","last_undegraded":"2024-09-21T12:28:55.984639+0000","last_fullsized":"2024-09-21T12:28:55.984639+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:28:30.619954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:30.619954+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:52.523678+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":338,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986061+0000","last_change":"2024-09-21T12:26:52.295004+0000","last_active":"2024-09-21T12:28:55.986061+0000","last_peered":"2024-09-21T12:28:55.986061+0000","last_clean":"2024-09-21T12:28:55.986061+0000","last_became_active":"2024-09-21T12:22:50.918326+0000","last_became_peered":"2024-09-21T12:22:50.918326+0000","last_unstale":"2024-09-21T12:28:55.986061+0000","last_undegraded":"2024-09-21T12:28:55.986061+0000","last_fullsized":"2024-09-21T12:28:55.986061+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:52.294931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:52.294931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"76'79","reported_seq":148,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.987532+0000","last_change":"2024-09-21T12:28:46.294913+0000","last_active":"2024-09-21T12:28:55.987532+0000","last_peered":"2024-09-21T12:28:55.987532+0000","last_clean":"2024-09-21T12:28:55.987532+0000","last_became_active":"2024-09-21T12:27:16.933358+0000","last_became_peered":"2024-09-21T12:27:16.933358+0000","last_unstale":"2024-09-21T12:28:55.987532+0000","last_undegraded":"2024-09-21T12:28:55.987532+0000","last_fullsized":"2024-09-21T12:28:55.987532+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:46.294855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:46.294855+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:07.298788+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":402,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.985943+0000","last_change":"2024-09-21T12:23:51.065780+0000","last_active":"2024-09-21T12:28:55.985943+0000","last_peered":"2024-09-21T12:28:55.985943+0000","last_clean":"2024-09-21T12:28:55.985943+0000","last_became_active":"2024-09-21T12:20:21.312976+0000","last_became_peered":"2024-09-21T12:20:21.312976+0000","last_unstale":"2024-09-21T12:28:55.985943+0000","last_undegraded":"2024-09-21T12:28:55.985943+0000","last_fullsized":"2024-09-21T12:28:55.985943+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:23:51.065733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:51.065733+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:15.680094+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":463,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986187+0000","last_change":"2024-09-21T12:27:56.597623+0000","last_active":"2024-09-21T12:28:55.986187+0000","last_peered":"2024-09-21T12:28:55.986187+0000","last_clean":"2024-09-21T12:28:55.986187+0000","last_became_active":"2024-09-21T12:22:50.910933+0000","last_became_peered":"2024-09-21T12:22:50.910933+0000","last_unstale":"2024-09-21T12:28:55.986187+0000","last_undegraded":"2024-09-21T12:28:55.986187+0000","last_fullsized":"2024-09-21T12:28:55.986187+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:27:56.597473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:27:56.597473+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:25.388812+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":427,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.994358+0000","last_change":"2024-09-21T12:24:03.059782+0000","last_active":"2024-09-21T12:28:55.994358+0000","last_peered":"2024-09-21T12:28:55.994358+0000","last_clean":"2024-09-21T12:28:55.994358+0000","last_became_active":"2024-09-21T12:20:17.695977+0000","last_became_peered":"2024-09-21T12:20:17.695977+0000","last_unstale":"2024-09-21T12:28:55.994358+0000","last_undegraded":"2024-09-21T12:28:55.994358+0000","last_fullsized":"2024-09-21T12:28:55.994358+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:24:03.059549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:03.059549+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:50.851518+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"121'11","reported_seq":571,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.994159+0000","last_change":"2024-09-21T12:26:10.846256+0000","last_active":"2024-09-21T12:28:55.994159+0000","last_peered":"2024-09-21T12:28:55.994159+0000","last_clean":"2024-09-21T12:28:55.994159+0000","last_became_active":"2024-09-21T12:22:50.916683+0000","last_became_peered":"2024-09-21T12:22:50.916683+0000","last_unstale":"2024-09-21T12:28:55.994159+0000","last_undegraded":"2024-09-21T12:28:55.994159+0000","last_fullsized":"2024-09-21T12:28:55.994159+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:26:10.846183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:10.846183+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:50.024651+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"255'3571","reported_seq":3619,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:57.317661+0000","last_change":"2024-09-21T12:28:20.956387+0000","last_active":"2024-09-21T12:28:57.317661+0000","last_peered":"2024-09-21T12:28:57.317661+0000","last_clean":"2024-09-21T12:28:57.317661+0000","last_became_active":"2024-09-21T12:28:20.955475+0000","last_became_peered":"2024-09-21T12:28:20.955475+0000","last_unstale":"2024-09-21T12:28:57.317661+0000","last_undegraded":"2024-09-21T12:28:57.317661+0000","last_fullsized":"2024-09-21T12:28:57.317661+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3571,"log_dups_size":0,"ondisk_log_size":3571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:48.118898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234029056,"num_objects":3571,"num_object_clones":0,"num_object_copies":10713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3571,"num_write_kb":228544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"76'79","reported_seq":139,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.980822+0000","last_change":"2024-09-21T12:28:29.446170+0000","last_active":"2024-09-21T12:28:55.980822+0000","last_peered":"2024-09-21T12:28:55.980822+0000","last_clean":"2024-09-21T12:28:55.980822+0000","last_became_active":"2024-09-21T12:27:14.943286+0000","last_became_peered":"2024-09-21T12:27:14.943286+0000","last_unstale":"2024-09-21T12:28:55.980822+0000","last_undegraded":"2024-09-21T12:28:55.980822+0000","last_fullsized":"2024-09-21T12:28:55.980822+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:29.446103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:29.446103+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:34.602581+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":281,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.981329+0000","last_change":"2024-09-21T12:27:31.491370+0000","last_active":"2024-09-21T12:28:55.981329+0000","last_peered":"2024-09-21T12:28:55.981329+0000","last_clean":"2024-09-21T12:28:55.981329+0000","last_became_active":"2024-09-21T12:23:46.910112+0000","last_became_peered":"2024-09-21T12:23:46.910112+0000","last_unstale":"2024-09-21T12:28:55.981329+0000","last_undegraded":"2024-09-21T12:28:55.981329+0000","last_fullsized":"2024-09-21T12:28:55.981329+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:27:31.490985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:31.490985+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":421,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986908+0000","last_change":"2024-09-21T12:28:05.580894+0000","last_active":"2024-09-21T12:28:55.986908+0000","last_peered":"2024-09-21T12:28:55.986908+0000","last_clean":"2024-09-21T12:28:55.986908+0000","last_became_active":"2024-09-21T12:20:15.295774+0000","last_became_peered":"2024-09-21T12:20:15.295774+0000","last_unstale":"2024-09-21T12:28:55.986908+0000","last_undegraded":"2024-09-21T12:28:55.986908+0000","last_fullsized":"2024-09-21T12:28:55.986908+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:05.580847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:05.580847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:25.392108+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"210'14","reported_seq":131,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.994048+0000","last_change":"2024-09-21T12:26:54.091833+0000","last_active":"2024-09-21T12:28:55.994048+0000","last_peered":"2024-09-21T12:28:55.994048+0000","last_clean":"2024-09-21T12:28:55.994048+0000","last_became_active":"2024-09-21T12:26:54.091451+0000","last_became_peered":"2024-09-21T12:26:54.091451+0000","last_unstale":"2024-09-21T12:28:55.994048+0000","last_undegraded":"2024-09-21T12:28:55.994048+0000","last_fullsized":"2024-09-21T12:28:55.994048+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'13","last_scrub_stamp":"2024-09-21T12:25:33.474391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:33.474391+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:55.951883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":131,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986515+0000","last_change":"2024-09-21T12:28:31.580349+0000","last_active":"2024-09-21T12:28:55.986515+0000","last_peered":"2024-09-21T12:28:55.986515+0000","last_clean":"2024-09-21T12:28:55.986515+0000","last_became_active":"2024-09-21T12:26:54.089699+0000","last_became_peered":"2024-09-21T12:26:54.089699+0000","last_unstale":"2024-09-21T12:28:55.986515+0000","last_undegraded":"2024-09-21T12:28:55.986515+0000","last_fullsized":"2024-09-21T12:28:55.986515+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:28:31.580245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:31.580245+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:42.630759+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.d","version":"255'3534","reported_seq":3582,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:56.651387+0000","last_change":"2024-09-21T12:28:20.958630+0000","last_active":"2024-09-21T12:28:56.651387+0000","last_peered":"2024-09-21T12:28:56.651387+0000","last_clean":"2024-09-21T12:28:56.651387+0000","last_became_active":"2024-09-21T12:28:20.958322+0000","last_became_peered":"2024-09-21T12:28:20.958322+0000","last_unstale":"2024-09-21T12:28:56.651387+0000","last_undegraded":"2024-09-21T12:28:56.651387+0000","last_fullsized":"2024-09-21T12:28:56.651387+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3534,"log_dups_size":0,"ondisk_log_size":3534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:24.412238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231604224,"num_objects":3534,"num_object_clones":0,"num_object_copies":10602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3534,"num_write_kb":226176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"76'79","reported_seq":346,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.980889+0000","last_change":"2024-09-21T12:27:20.527090+0000","last_active":"2024-09-21T12:28:55.980889+0000","last_peered":"2024-09-21T12:28:55.980889+0000","last_clean":"2024-09-21T12:28:55.980889+0000","last_became_active":"2024-09-21T12:27:14.941905+0000","last_became_peered":"2024-09-21T12:27:14.941905+0000","last_unstale":"2024-09-21T12:28:55.980889+0000","last_undegraded":"2024-09-21T12:28:55.980889+0000","last_fullsized":"2024-09-21T12:28:55.980889+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:20.526951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:20.526951+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":457,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.984440+0000","last_change":"2024-09-21T12:28:26.632364+0000","last_active":"2024-09-21T12:28:55.984440+0000","last_peered":"2024-09-21T12:28:55.984440+0000","last_clean":"2024-09-21T12:28:55.984440+0000","last_became_active":"2024-09-21T12:20:17.693179+0000","last_became_peered":"2024-09-21T12:20:17.693179+0000","last_unstale":"2024-09-21T12:28:55.984440+0000","last_undegraded":"2024-09-21T12:28:55.984440+0000","last_fullsized":"2024-09-21T12:28:55.984440+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-21T12:28:26.632272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:26.632272+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:48.718928+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":413,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.985998+0000","last_change":"2024-09-21T12:21:18.056305+0000","last_active":"2024-09-21T12:28:55.985998+0000","last_peered":"2024-09-21T12:28:55.985998+0000","last_clean":"2024-09-21T12:28:55.985998+0000","last_became_active":"2024-09-21T12:20:15.297340+0000","last_became_peered":"2024-09-21T12:20:15.297340+0000","last_unstale":"2024-09-21T12:28:55.985998+0000","last_undegraded":"2024-09-21T12:28:55.985998+0000","last_fullsized":"2024-09-21T12:28:55.985998+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:21:18.056261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:21:18.056261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:09.624424+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":363,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.984389+0000","last_change":"2024-09-21T12:28:33.697269+0000","last_active":"2024-09-21T12:28:55.984389+0000","last_peered":"2024-09-21T12:28:55.984389+0000","last_clean":"2024-09-21T12:28:55.984389+0000","last_became_active":"2024-09-21T12:22:50.930858+0000","last_became_peered":"2024-09-21T12:22:50.930858+0000","last_unstale":"2024-09-21T12:28:55.984389+0000","last_undegraded":"2024-09-21T12:28:55.984389+0000","last_fullsized":"2024-09-21T12:28:55.984389+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:33.697208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:33.697208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:03.607813+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":354,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.993338+0000","last_change":"2024-09-21T12:26:44.325219+0000","last_active":"2024-09-21T12:28:55.993338+0000","last_peered":"2024-09-21T12:28:55.993338+0000","last_clean":"2024-09-21T12:28:55.993338+0000","last_became_active":"2024-09-21T12:22:50.918940+0000","last_became_peered":"2024-09-21T12:22:50.918940+0000","last_unstale":"2024-09-21T12:28:55.993338+0000","last_undegraded":"2024-09-21T12:28:55.993338+0000","last_fullsized":"2024-09-21T12:28:55.993338+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:44.325124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:44.325124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"255'3612","reported_seq":3660,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:57.308915+0000","last_change":"2024-09-21T12:28:20.956665+0000","last_active":"2024-09-21T12:28:57.308915+0000","last_peered":"2024-09-21T12:28:57.308915+0000","last_clean":"2024-09-21T12:28:57.308915+0000","last_became_active":"2024-09-21T12:28:20.955557+0000","last_became_peered":"2024-09-21T12:28:20.955557+0000","last_unstale":"2024-09-21T12:28:57.308915+0000","last_undegraded":"2024-09-21T12:28:57.308915+0000","last_fullsized":"2024-09-21T12:28:57.308915+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3612,"log_dups_size":0,"ondisk_log_size":3612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:38.621432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236716032,"num_objects":3612,"num_object_clones":0,"num_object_copies":10836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3612,"num_write_kb":231168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"76'79","reported_seq":348,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.985866+0000","last_change":"2024-09-21T12:26:45.354836+0000","last_active":"2024-09-21T12:28:55.985866+0000","last_peered":"2024-09-21T12:28:55.985866+0000","last_clean":"2024-09-21T12:28:55.985866+0000","last_became_active":"2024-09-21T12:26:44.960093+0000","last_became_peered":"2024-09-21T12:26:44.960093+0000","last_unstale":"2024-09-21T12:28:55.985866+0000","last_undegraded":"2024-09-21T12:28:55.985866+0000","last_fullsized":"2024-09-21T12:28:55.985866+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:45.354774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:45.354774+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":419,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.987545+0000","last_change":"2024-09-21T12:26:51.463759+0000","last_active":"2024-09-21T12:28:55.987545+0000","last_peered":"2024-09-21T12:28:55.987545+0000","last_clean":"2024-09-21T12:28:55.987545+0000","last_became_active":"2024-09-21T12:22:50.926008+0000","last_became_peered":"2024-09-21T12:22:50.926008+0000","last_unstale":"2024-09-21T12:28:55.987545+0000","last_undegraded":"2024-09-21T12:28:55.987545+0000","last_fullsized":"2024-09-21T12:28:55.987545+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-21T12:26:51.463698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:51.463698+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:23.362051+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":466,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.994485+0000","last_change":"2024-09-21T12:28:32.727794+0000","last_active":"2024-09-21T12:28:55.994485+0000","last_peered":"2024-09-21T12:28:55.994485+0000","last_clean":"2024-09-21T12:28:55.994485+0000","last_became_active":"2024-09-21T12:22:50.923284+0000","last_became_peered":"2024-09-21T12:22:50.923284+0000","last_unstale":"2024-09-21T12:28:55.994485+0000","last_undegraded":"2024-09-21T12:28:55.994485+0000","last_fullsized":"2024-09-21T12:28:55.994485+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:28:32.727727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:32.727727+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:52.197596+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"121'11","reported_seq":559,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.984266+0000","last_change":"2024-09-21T12:26:14.490045+0000","last_active":"2024-09-21T12:28:55.984266+0000","last_peered":"2024-09-21T12:28:55.984266+0000","last_clean":"2024-09-21T12:28:55.984266+0000","last_became_active":"2024-09-21T12:22:50.930342+0000","last_became_peered":"2024-09-21T12:22:50.930342+0000","last_unstale":"2024-09-21T12:28:55.984266+0000","last_undegraded":"2024-09-21T12:28:55.984266+0000","last_fullsized":"2024-09-21T12:28:55.984266+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:26:14.489972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:14.489972+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":410,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.985363+0000","last_change":"2024-09-21T12:28:37.686403+0000","last_active":"2024-09-21T12:28:55.985363+0000","last_peered":"2024-09-21T12:28:55.985363+0000","last_clean":"2024-09-21T12:28:55.985363+0000","last_became_active":"2024-09-21T12:20:21.593367+0000","last_became_peered":"2024-09-21T12:20:21.593367+0000","last_unstale":"2024-09-21T12:28:55.985363+0000","last_undegraded":"2024-09-21T12:28:55.985363+0000","last_fullsized":"2024-09-21T12:28:55.985363+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:37.686350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:37.686350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:53.013407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"255'3534","reported_seq":3582,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:56.675466+0000","last_change":"2024-09-21T12:28:20.956941+0000","last_active":"2024-09-21T12:28:56.675466+0000","last_peered":"2024-09-21T12:28:56.675466+0000","last_clean":"2024-09-21T12:28:56.675466+0000","last_became_active":"2024-09-21T12:28:20.956619+0000","last_became_peered":"2024-09-21T12:28:20.956619+0000","last_unstale":"2024-09-21T12:28:56.675466+0000","last_undegraded":"2024-09-21T12:28:56.675466+0000","last_fullsized":"2024-09-21T12:28:56.675466+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3534,"log_dups_size":0,"ondisk_log_size":3534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:45.844072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231604224,"num_objects":3534,"num_object_clones":0,"num_object_copies":10602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3534,"num_write_kb":226176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"76'79","reported_seq":447,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.993993+0000","last_change":"2024-09-21T12:27:24.874811+0000","last_active":"2024-09-21T12:28:55.993993+0000","last_peered":"2024-09-21T12:28:55.993993+0000","last_clean":"2024-09-21T12:28:55.993993+0000","last_became_active":"2024-09-21T12:27:14.937216+0000","last_became_peered":"2024-09-21T12:27:14.937216+0000","last_unstale":"2024-09-21T12:28:55.993993+0000","last_undegraded":"2024-09-21T12:28:55.993993+0000","last_fullsized":"2024-09-21T12:28:55.993993+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:24.874768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:24.874768+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":490,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986398+0000","last_change":"2024-09-21T12:27:27.325248+0000","last_active":"2024-09-21T12:28:55.986398+0000","last_peered":"2024-09-21T12:28:55.986398+0000","last_clean":"2024-09-21T12:28:55.986398+0000","last_became_active":"2024-09-21T12:22:50.925840+0000","last_became_peered":"2024-09-21T12:22:50.925840+0000","last_unstale":"2024-09-21T12:28:55.986398+0000","last_undegraded":"2024-09-21T12:28:55.986398+0000","last_fullsized":"2024-09-21T12:28:55.986398+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:27:27.325105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:27.325105+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":430,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.987846+0000","last_change":"2024-09-21T12:27:25.427260+0000","last_active":"2024-09-21T12:28:55.987846+0000","last_peered":"2024-09-21T12:28:55.987846+0000","last_clean":"2024-09-21T12:28:55.987846+0000","last_became_active":"2024-09-21T12:20:15.308518+0000","last_became_peered":"2024-09-21T12:20:15.308518+0000","last_unstale":"2024-09-21T12:28:55.987846+0000","last_undegraded":"2024-09-21T12:28:55.987846+0000","last_fullsized":"2024-09-21T12:28:55.987846+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:25.427219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:27:25.427219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":540,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.997505+0000","last_change":"2024-09-21T12:27:58.871800+0000","last_active":"2024-09-21T12:28:55.997505+0000","last_peered":"2024-09-21T12:28:55.997505+0000","last_clean":"2024-09-21T12:28:55.997505+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:28:55.997505+0000","last_undegraded":"2024-09-21T12:28:55.997505+0000","last_fullsized":"2024-09-21T12:28:55.997505+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:27:58.871710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:27:58.871710+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:27.524400+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":420,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.988273+0000","last_change":"2024-09-21T12:28:37.850166+0000","last_active":"2024-09-21T12:28:55.988273+0000","last_peered":"2024-09-21T12:28:55.988273+0000","last_clean":"2024-09-21T12:28:55.988273+0000","last_became_active":"2024-09-21T12:20:21.315801+0000","last_became_peered":"2024-09-21T12:20:21.315801+0000","last_unstale":"2024-09-21T12:28:55.988273+0000","last_undegraded":"2024-09-21T12:28:55.988273+0000","last_fullsized":"2024-09-21T12:28:55.988273+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:37.850100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:37.850100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:52.359950+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"255'3630","reported_seq":3678,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:56.652565+0000","last_change":"2024-09-21T12:28:20.958462+0000","last_active":"2024-09-21T12:28:56.652565+0000","last_peered":"2024-09-21T12:28:56.652565+0000","last_clean":"2024-09-21T12:28:56.652565+0000","last_became_active":"2024-09-21T12:28:20.958143+0000","last_became_peered":"2024-09-21T12:28:20.958143+0000","last_unstale":"2024-09-21T12:28:56.652565+0000","last_undegraded":"2024-09-21T12:28:56.652565+0000","last_fullsized":"2024-09-21T12:28:56.652565+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3630,"log_dups_size":0,"ondisk_log_size":3630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:47.169876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237895680,"num_objects":3630,"num_object_clones":0,"num_object_copies":10890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3630,"num_write_kb":232320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"76'79","reported_seq":447,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.993781+0000","last_change":"2024-09-21T12:27:19.888983+0000","last_active":"2024-09-21T12:28:55.993781+0000","last_peered":"2024-09-21T12:28:55.993781+0000","last_clean":"2024-09-21T12:28:55.993781+0000","last_became_active":"2024-09-21T12:27:14.954642+0000","last_became_peered":"2024-09-21T12:27:14.954642+0000","last_unstale":"2024-09-21T12:28:55.993781+0000","last_undegraded":"2024-09-21T12:28:55.993781+0000","last_fullsized":"2024-09-21T12:28:55.993781+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:19.888914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:19.888914+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:54.960286+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":456,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.997411+0000","last_change":"2024-09-21T12:26:14.834328+0000","last_active":"2024-09-21T12:28:55.997411+0000","last_peered":"2024-09-21T12:28:55.997411+0000","last_clean":"2024-09-21T12:28:55.997411+0000","last_became_active":"2024-09-21T12:20:17.693067+0000","last_became_peered":"2024-09-21T12:20:17.693067+0000","last_unstale":"2024-09-21T12:28:55.997411+0000","last_undegraded":"2024-09-21T12:28:55.997411+0000","last_fullsized":"2024-09-21T12:28:55.997411+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-21T12:26:14.834236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:14.834236+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:15.646442+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":414,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986995+0000","last_change":"2024-09-21T12:26:12.131784+0000","last_active":"2024-09-21T12:28:55.986995+0000","last_peered":"2024-09-21T12:28:55.986995+0000","last_clean":"2024-09-21T12:28:55.986995+0000","last_became_active":"2024-09-21T12:20:15.308441+0000","last_became_peered":"2024-09-21T12:20:15.308441+0000","last_unstale":"2024-09-21T12:28:55.986995+0000","last_undegraded":"2024-09-21T12:28:55.986995+0000","last_fullsized":"2024-09-21T12:28:55.986995+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:12.131721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:12.131721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:54.870193+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":422,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.993453+0000","last_change":"2024-09-21T12:28:48.999950+0000","last_active":"2024-09-21T12:28:55.993453+0000","last_peered":"2024-09-21T12:28:55.993453+0000","last_clean":"2024-09-21T12:28:55.993453+0000","last_became_active":"2024-09-21T12:20:19.309598+0000","last_became_peered":"2024-09-21T12:20:19.309598+0000","last_unstale":"2024-09-21T12:28:55.993453+0000","last_undegraded":"2024-09-21T12:28:55.993453+0000","last_fullsized":"2024-09-21T12:28:55.993453+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:48.999878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:48.999878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:15.295060+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":402,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.985922+0000","last_change":"2024-09-21T12:25:44.918356+0000","last_active":"2024-09-21T12:28:55.985922+0000","last_peered":"2024-09-21T12:28:55.985922+0000","last_clean":"2024-09-21T12:28:55.985922+0000","last_became_active":"2024-09-21T12:20:21.592828+0000","last_became_peered":"2024-09-21T12:20:21.592828+0000","last_unstale":"2024-09-21T12:28:55.985922+0000","last_undegraded":"2024-09-21T12:28:55.985922+0000","last_fullsized":"2024-09-21T12:28:55.985922+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:44.918276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:44.918276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:23.575759+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"255'3620","reported_seq":3669,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:57.398650+0000","last_change":"2024-09-21T12:28:20.952252+0000","last_active":"2024-09-21T12:28:57.398650+0000","last_peered":"2024-09-21T12:28:57.398650+0000","last_clean":"2024-09-21T12:28:57.398650+0000","last_became_active":"2024-09-21T12:28:20.951980+0000","last_became_peered":"2024-09-21T12:28:20.951980+0000","last_unstale":"2024-09-21T12:28:57.398650+0000","last_undegraded":"2024-09-21T12:28:57.398650+0000","last_fullsized":"2024-09-21T12:28:57.398650+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3620,"log_dups_size":0,"ondisk_log_size":3620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:48.108618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237240320,"num_objects":3620,"num_object_clones":0,"num_object_copies":10860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3620,"num_write_kb":231680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"76'79","reported_seq":213,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.994380+0000","last_change":"2024-09-21T12:28:25.714131+0000","last_active":"2024-09-21T12:28:55.994380+0000","last_peered":"2024-09-21T12:28:55.994380+0000","last_clean":"2024-09-21T12:28:55.994380+0000","last_became_active":"2024-09-21T12:26:54.092726+0000","last_became_peered":"2024-09-21T12:26:54.092726+0000","last_unstale":"2024-09-21T12:28:55.994380+0000","last_undegraded":"2024-09-21T12:28:55.994380+0000","last_fullsized":"2024-09-21T12:28:55.994380+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:25.714084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:25.714084+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:37.309436+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":563,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986904+0000","last_change":"2024-09-21T12:26:09.082395+0000","last_active":"2024-09-21T12:28:55.986904+0000","last_peered":"2024-09-21T12:28:55.986904+0000","last_clean":"2024-09-21T12:28:55.986904+0000","last_became_active":"2024-09-21T12:20:17.307769+0000","last_became_peered":"2024-09-21T12:20:17.307769+0000","last_unstale":"2024-09-21T12:28:55.986904+0000","last_undegraded":"2024-09-21T12:28:55.986904+0000","last_fullsized":"2024-09-21T12:28:55.986904+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-21T12:26:09.082314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:09.082314+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:52.842072+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":355,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.997270+0000","last_change":"2024-09-21T12:28:31.738463+0000","last_active":"2024-09-21T12:28:55.997270+0000","last_peered":"2024-09-21T12:28:55.997270+0000","last_clean":"2024-09-21T12:28:55.997270+0000","last_became_active":"2024-09-21T12:22:50.922002+0000","last_became_peered":"2024-09-21T12:22:50.922002+0000","last_unstale":"2024-09-21T12:28:55.997270+0000","last_undegraded":"2024-09-21T12:28:55.997270+0000","last_fullsized":"2024-09-21T12:28:55.997270+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:31.738415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:31.738415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:33.595522+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":139,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.985815+0000","last_change":"2024-09-21T12:28:37.691938+0000","last_active":"2024-09-21T12:28:55.985815+0000","last_peered":"2024-09-21T12:28:55.985815+0000","last_clean":"2024-09-21T12:28:55.985815+0000","last_became_active":"2024-09-21T12:26:54.083584+0000","last_became_peered":"2024-09-21T12:26:54.083584+0000","last_unstale":"2024-09-21T12:28:55.985815+0000","last_undegraded":"2024-09-21T12:28:55.985815+0000","last_fullsized":"2024-09-21T12:28:55.985815+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T12:28:37.691868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:28:37.691868+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:45.015722+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":445,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.993945+0000","last_change":"2024-09-21T12:27:21.956252+0000","last_active":"2024-09-21T12:28:55.993945+0000","last_peered":"2024-09-21T12:28:55.993945+0000","last_clean":"2024-09-21T12:28:55.993945+0000","last_became_active":"2024-09-21T12:22:50.921488+0000","last_became_peered":"2024-09-21T12:22:50.921488+0000","last_unstale":"2024-09-21T12:28:55.993945+0000","last_undegraded":"2024-09-21T12:28:55.993945+0000","last_fullsized":"2024-09-21T12:28:55.993945+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:21.956185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:27:21.956185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:52.937668+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":429,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.985217+0000","last_change":"2024-09-21T12:26:10.435410+0000","last_active":"2024-09-21T12:28:55.985217+0000","last_peered":"2024-09-21T12:28:55.985217+0000","last_clean":"2024-09-21T12:28:55.985217+0000","last_became_active":"2024-09-21T12:22:50.917558+0000","last_became_peered":"2024-09-21T12:22:50.917558+0000","last_unstale":"2024-09-21T12:28:55.985217+0000","last_undegraded":"2024-09-21T12:28:55.985217+0000","last_fullsized":"2024-09-21T12:28:55.985217+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:10.435349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:10.435349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"255'3553","reported_seq":3601,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:56.645340+0000","last_change":"2024-09-21T12:28:20.953805+0000","last_active":"2024-09-21T12:28:56.645340+0000","last_peered":"2024-09-21T12:28:56.645340+0000","last_clean":"2024-09-21T12:28:56.645340+0000","last_became_active":"2024-09-21T12:28:20.952781+0000","last_became_peered":"2024-09-21T12:28:20.952781+0000","last_unstale":"2024-09-21T12:28:56.645340+0000","last_undegraded":"2024-09-21T12:28:56.645340+0000","last_fullsized":"2024-09-21T12:28:56.645340+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3553,"log_dups_size":0,"ondisk_log_size":3553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:45.666222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232849408,"num_objects":3553,"num_object_clones":0,"num_object_copies":10659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3553,"num_write_kb":227392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"76'79","reported_seq":518,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.985590+0000","last_change":"2024-09-21T12:26:46.188733+0000","last_active":"2024-09-21T12:28:55.985590+0000","last_peered":"2024-09-21T12:28:55.985590+0000","last_clean":"2024-09-21T12:28:55.985590+0000","last_became_active":"2024-09-21T12:26:44.956003+0000","last_became_peered":"2024-09-21T12:26:44.956003+0000","last_unstale":"2024-09-21T12:28:55.985590+0000","last_undegraded":"2024-09-21T12:28:55.985590+0000","last_fullsized":"2024-09-21T12:28:55.985590+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:46.188649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:46.188649+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":453,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.987389+0000","last_change":"2024-09-21T12:28:25.549378+0000","last_active":"2024-09-21T12:28:55.987389+0000","last_peered":"2024-09-21T12:28:55.987389+0000","last_clean":"2024-09-21T12:28:55.987389+0000","last_became_active":"2024-09-21T12:20:17.297374+0000","last_became_peered":"2024-09-21T12:20:17.297374+0000","last_unstale":"2024-09-21T12:28:55.987389+0000","last_undegraded":"2024-09-21T12:28:55.987389+0000","last_fullsized":"2024-09-21T12:28:55.987389+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-21T12:28:25.549320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:25.549320+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:32.588927+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":438,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.988536+0000","last_change":"2024-09-21T12:28:39.705480+0000","last_active":"2024-09-21T12:28:55.988536+0000","last_peered":"2024-09-21T12:28:55.988536+0000","last_clean":"2024-09-21T12:28:55.988536+0000","last_became_active":"2024-09-21T12:20:15.308356+0000","last_became_peered":"2024-09-21T12:20:15.308356+0000","last_unstale":"2024-09-21T12:28:55.988536+0000","last_undegraded":"2024-09-21T12:28:55.988536+0000","last_fullsized":"2024-09-21T12:28:55.988536+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:39.705175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:39.705175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:54.221915+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":448,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.980018+0000","last_change":"2024-09-21T12:28:43.390160+0000","last_active":"2024-09-21T12:28:55.980018+0000","last_peered":"2024-09-21T12:28:55.980018+0000","last_clean":"2024-09-21T12:28:55.980018+0000","last_became_active":"2024-09-21T12:20:14.288358+0000","last_became_peered":"2024-09-21T12:20:14.288358+0000","last_unstale":"2024-09-21T12:28:55.980018+0000","last_undegraded":"2024-09-21T12:28:55.980018+0000","last_fullsized":"2024-09-21T12:28:55.980018+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:28:43.390087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:28:43.390087+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:12.445374+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0019042449999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":422,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.981718+0000","last_change":"2024-09-21T12:27:33.447579+0000","last_active":"2024-09-21T12:28:55.981718+0000","last_peered":"2024-09-21T12:28:55.981718+0000","last_clean":"2024-09-21T12:28:55.981718+0000","last_became_active":"2024-09-21T12:20:19.309724+0000","last_became_peered":"2024-09-21T12:20:19.309724+0000","last_unstale":"2024-09-21T12:28:55.981718+0000","last_undegraded":"2024-09-21T12:28:55.981718+0000","last_fullsized":"2024-09-21T12:28:55.981718+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:33.447236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:27:33.447236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:02.627972+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":402,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986187+0000","last_change":"2024-09-21T12:27:33.395611+0000","last_active":"2024-09-21T12:28:55.986187+0000","last_peered":"2024-09-21T12:28:55.986187+0000","last_clean":"2024-09-21T12:28:55.986187+0000","last_became_active":"2024-09-21T12:20:21.592982+0000","last_became_peered":"2024-09-21T12:20:21.592982+0000","last_unstale":"2024-09-21T12:28:55.986187+0000","last_undegraded":"2024-09-21T12:28:55.986187+0000","last_fullsized":"2024-09-21T12:28:55.986187+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:33.395568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:27:33.395568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"255'3774","reported_seq":3823,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:57.393369+0000","last_change":"2024-09-21T12:28:21.114897+0000","last_active":"2024-09-21T12:28:57.393369+0000","last_peered":"2024-09-21T12:28:57.393369+0000","last_clean":"2024-09-21T12:28:57.393369+0000","last_became_active":"2024-09-21T12:28:21.114609+0000","last_became_peered":"2024-09-21T12:28:21.114609+0000","last_unstale":"2024-09-21T12:28:57.393369+0000","last_undegraded":"2024-09-21T12:28:57.393369+0000","last_fullsized":"2024-09-21T12:28:57.393369+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3774,"log_dups_size":0,"ondisk_log_size":3774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:31.696957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247332864,"num_objects":3774,"num_object_clones":0,"num_object_copies":11322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3774,"num_write_kb":241536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"76'79","reported_seq":364,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986865+0000","last_change":"2024-09-21T12:28:01.638945+0000","last_active":"2024-09-21T12:28:55.986865+0000","last_peered":"2024-09-21T12:28:55.986865+0000","last_clean":"2024-09-21T12:28:55.986865+0000","last_became_active":"2024-09-21T12:26:44.949867+0000","last_became_peered":"2024-09-21T12:26:44.949867+0000","last_unstale":"2024-09-21T12:28:55.986865+0000","last_undegraded":"2024-09-21T12:28:55.986865+0000","last_fullsized":"2024-09-21T12:28:55.986865+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:01.638886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:01.638886+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:04.265728+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":446,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.994374+0000","last_change":"2024-09-21T12:28:45.783033+0000","last_active":"2024-09-21T12:28:55.994374+0000","last_peered":"2024-09-21T12:28:55.994374+0000","last_clean":"2024-09-21T12:28:55.994374+0000","last_became_active":"2024-09-21T12:20:17.299325+0000","last_became_peered":"2024-09-21T12:20:17.299325+0000","last_unstale":"2024-09-21T12:28:55.994374+0000","last_undegraded":"2024-09-21T12:28:55.994374+0000","last_fullsized":"2024-09-21T12:28:55.994374+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:28:45.782955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.782955+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:03.191455+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":445,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.993516+0000","last_change":"2024-09-21T12:24:48.073954+0000","last_active":"2024-09-21T12:28:55.993516+0000","last_peered":"2024-09-21T12:28:55.993516+0000","last_clean":"2024-09-21T12:28:55.993516+0000","last_became_active":"2024-09-21T12:22:50.923796+0000","last_became_peered":"2024-09-21T12:22:50.923796+0000","last_unstale":"2024-09-21T12:28:55.993516+0000","last_undegraded":"2024-09-21T12:28:55.993516+0000","last_fullsized":"2024-09-21T12:28:55.993516+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:48.073424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:48.073424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:12.742306+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":957,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.985261+0000","last_change":"2024-09-21T12:28:34.639438+0000","last_active":"2024-09-21T12:28:55.985261+0000","last_peered":"2024-09-21T12:28:55.985261+0000","last_clean":"2024-09-21T12:28:55.985261+0000","last_became_active":"2024-09-21T12:20:14.287543+0000","last_became_peered":"2024-09-21T12:20:14.287543+0000","last_unstale":"2024-09-21T12:28:55.985261+0000","last_undegraded":"2024-09-21T12:28:55.985261+0000","last_fullsized":"2024-09-21T12:28:55.985261+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-21T12:28:34.639361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:28:34.639361+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:48.713397+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":516,"num_read_kb":511,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":414,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.988417+0000","last_change":"2024-09-21T12:26:11.422340+0000","last_active":"2024-09-21T12:28:55.988417+0000","last_peered":"2024-09-21T12:28:55.988417+0000","last_clean":"2024-09-21T12:28:55.988417+0000","last_became_active":"2024-09-21T12:20:19.314217+0000","last_became_peered":"2024-09-21T12:20:19.314217+0000","last_unstale":"2024-09-21T12:28:55.988417+0000","last_undegraded":"2024-09-21T12:28:55.988417+0000","last_fullsized":"2024-09-21T12:28:55.988417+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:11.422293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:11.422293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:18.396029+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":418,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986450+0000","last_change":"2024-09-21T12:25:38.882984+0000","last_active":"2024-09-21T12:28:55.986450+0000","last_peered":"2024-09-21T12:28:55.986450+0000","last_clean":"2024-09-21T12:28:55.986450+0000","last_became_active":"2024-09-21T12:20:21.311188+0000","last_became_peered":"2024-09-21T12:20:21.311188+0000","last_unstale":"2024-09-21T12:28:55.986450+0000","last_undegraded":"2024-09-21T12:28:55.986450+0000","last_fullsized":"2024-09-21T12:28:55.986450+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:38.882915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:38.882915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:21.632466+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"255'3598","reported_seq":3646,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:57.317125+0000","last_change":"2024-09-21T12:28:20.957134+0000","last_active":"2024-09-21T12:28:57.317125+0000","last_peered":"2024-09-21T12:28:57.317125+0000","last_clean":"2024-09-21T12:28:57.317125+0000","last_became_active":"2024-09-21T12:28:20.956557+0000","last_became_peered":"2024-09-21T12:28:20.956557+0000","last_unstale":"2024-09-21T12:28:57.317125+0000","last_undegraded":"2024-09-21T12:28:57.317125+0000","last_fullsized":"2024-09-21T12:28:57.317125+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3598,"log_dups_size":0,"ondisk_log_size":3598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:31.714770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235798528,"num_objects":3598,"num_object_clones":0,"num_object_copies":10794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3598,"num_write_kb":230272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"121'19","reported_seq":620,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.993772+0000","last_change":"2024-09-21T12:24:27.932976+0000","last_active":"2024-09-21T12:28:55.993772+0000","last_peered":"2024-09-21T12:28:55.993772+0000","last_clean":"2024-09-21T12:28:55.993772+0000","last_became_active":"2024-09-21T12:22:50.921690+0000","last_became_peered":"2024-09-21T12:22:50.921690+0000","last_unstale":"2024-09-21T12:28:55.993772+0000","last_undegraded":"2024-09-21T12:28:55.993772+0000","last_fullsized":"2024-09-21T12:28:55.993772+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'19","last_scrub_stamp":"2024-09-21T12:24:27.932891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:27.932891+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:11.225971+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":427,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.993474+0000","last_change":"2024-09-21T12:26:12.394327+0000","last_active":"2024-09-21T12:28:55.993474+0000","last_peered":"2024-09-21T12:28:55.993474+0000","last_clean":"2024-09-21T12:28:55.993474+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:28:55.993474+0000","last_undegraded":"2024-09-21T12:28:55.993474+0000","last_fullsized":"2024-09-21T12:28:55.993474+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:26:12.394247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:12.394247+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":506,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.985327+0000","last_change":"2024-09-21T12:26:49.248015+0000","last_active":"2024-09-21T12:28:55.985327+0000","last_peered":"2024-09-21T12:28:55.985327+0000","last_clean":"2024-09-21T12:28:55.985327+0000","last_became_active":"2024-09-21T12:26:44.950132+0000","last_became_peered":"2024-09-21T12:26:44.950132+0000","last_unstale":"2024-09-21T12:28:55.985327+0000","last_undegraded":"2024-09-21T12:28:55.985327+0000","last_fullsized":"2024-09-21T12:28:55.985327+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:49.247952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:49.247952+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":406,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986109+0000","last_change":"2024-09-21T12:28:36.684525+0000","last_active":"2024-09-21T12:28:55.986109+0000","last_peered":"2024-09-21T12:28:55.986109+0000","last_clean":"2024-09-21T12:28:55.986109+0000","last_became_active":"2024-09-21T12:20:19.309859+0000","last_became_peered":"2024-09-21T12:20:19.309859+0000","last_unstale":"2024-09-21T12:28:55.986109+0000","last_undegraded":"2024-09-21T12:28:55.986109+0000","last_fullsized":"2024-09-21T12:28:55.986109+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:36.684459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:36.684459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:43.653752+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":255,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.985389+0000","last_change":"2024-09-21T12:28:28.647344+0000","last_active":"2024-09-21T12:28:55.985389+0000","last_peered":"2024-09-21T12:28:55.985389+0000","last_clean":"2024-09-21T12:28:55.985389+0000","last_became_active":"2024-09-21T12:26:54.084914+0000","last_became_peered":"2024-09-21T12:26:54.084914+0000","last_unstale":"2024-09-21T12:28:55.985389+0000","last_undegraded":"2024-09-21T12:28:55.985389+0000","last_fullsized":"2024-09-21T12:28:55.985389+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:28.647287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:28.647287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:44.468892+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3","version":"255'3575","reported_seq":3623,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:56.654859+0000","last_change":"2024-09-21T12:28:20.958439+0000","last_active":"2024-09-21T12:28:56.654859+0000","last_peered":"2024-09-21T12:28:56.654859+0000","last_clean":"2024-09-21T12:28:56.654859+0000","last_became_active":"2024-09-21T12:28:20.958025+0000","last_became_peered":"2024-09-21T12:28:20.958025+0000","last_unstale":"2024-09-21T12:28:56.654859+0000","last_undegraded":"2024-09-21T12:28:56.654859+0000","last_fullsized":"2024-09-21T12:28:56.654859+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3575,"log_dups_size":0,"ondisk_log_size":3575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:33.887095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234291200,"num_objects":3575,"num_object_clones":0,"num_object_copies":10725,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3575,"num_write_kb":228800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"76'79","reported_seq":367,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.987538+0000","last_change":"2024-09-21T12:27:14.954671+0000","last_active":"2024-09-21T12:28:55.987538+0000","last_peered":"2024-09-21T12:28:55.987538+0000","last_clean":"2024-09-21T12:28:55.987538+0000","last_became_active":"2024-09-21T12:27:14.954351+0000","last_became_peered":"2024-09-21T12:27:14.954351+0000","last_unstale":"2024-09-21T12:28:55.987538+0000","last_undegraded":"2024-09-21T12:28:55.987538+0000","last_fullsized":"2024-09-21T12:28:55.987538+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:48.188322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:48.188322+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:29:17.673456+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":436,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.993921+0000","last_change":"2024-09-21T12:27:57.114150+0000","last_active":"2024-09-21T12:28:55.993921+0000","last_peered":"2024-09-21T12:28:55.993921+0000","last_clean":"2024-09-21T12:28:55.993921+0000","last_became_active":"2024-09-21T12:23:46.885802+0000","last_became_peered":"2024-09-21T12:23:46.885802+0000","last_unstale":"2024-09-21T12:28:55.993921+0000","last_undegraded":"2024-09-21T12:28:55.993921+0000","last_fullsized":"2024-09-21T12:28:55.993921+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:27:57.114076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:57.114076+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:19.583144+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":346,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.979885+0000","last_change":"2024-09-21T12:27:30.501480+0000","last_active":"2024-09-21T12:28:55.979885+0000","last_peered":"2024-09-21T12:28:55.979885+0000","last_clean":"2024-09-21T12:28:55.979885+0000","last_became_active":"2024-09-21T12:22:50.908060+0000","last_became_peered":"2024-09-21T12:22:50.908060+0000","last_unstale":"2024-09-21T12:28:55.979885+0000","last_undegraded":"2024-09-21T12:28:55.979885+0000","last_fullsized":"2024-09-21T12:28:55.979885+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:30.498831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:27:30.498831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":259,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.994076+0000","last_change":"2024-09-21T12:28:05.266905+0000","last_active":"2024-09-21T12:28:55.994076+0000","last_peered":"2024-09-21T12:28:55.994076+0000","last_clean":"2024-09-21T12:28:55.994076+0000","last_became_active":"2024-09-21T12:26:54.089839+0000","last_became_peered":"2024-09-21T12:26:54.089839+0000","last_unstale":"2024-09-21T12:28:55.994076+0000","last_undegraded":"2024-09-21T12:28:55.994076+0000","last_fullsized":"2024-09-21T12:28:55.994076+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:05.266842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:05.266842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:08.140709+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":408,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.985957+0000","last_change":"2024-09-21T12:28:39.853931+0000","last_active":"2024-09-21T12:28:55.985957+0000","last_peered":"2024-09-21T12:28:55.985957+0000","last_clean":"2024-09-21T12:28:55.985957+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:28:55.985957+0000","last_undegraded":"2024-09-21T12:28:55.985957+0000","last_fullsized":"2024-09-21T12:28:55.985957+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:39.853791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:39.853791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:04.491080+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"255'3574","reported_seq":3622,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:56.998307+0000","last_change":"2024-09-21T12:28:21.114546+0000","last_active":"2024-09-21T12:28:56.998307+0000","last_peered":"2024-09-21T12:28:56.998307+0000","last_clean":"2024-09-21T12:28:56.998307+0000","last_became_active":"2024-09-21T12:28:21.114218+0000","last_became_peered":"2024-09-21T12:28:21.114218+0000","last_unstale":"2024-09-21T12:28:56.998307+0000","last_undegraded":"2024-09-21T12:28:56.998307+0000","last_fullsized":"2024-09-21T12:28:56.998307+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3574,"log_dups_size":0,"ondisk_log_size":3574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:31.328174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234225664,"num_objects":3574,"num_object_clones":0,"num_object_copies":10722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3574,"num_write_kb":228736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"76'79","reported_seq":123,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.979666+0000","last_change":"2024-09-21T12:27:14.942259+0000","last_active":"2024-09-21T12:28:55.979666+0000","last_peered":"2024-09-21T12:28:55.979666+0000","last_clean":"2024-09-21T12:28:55.979666+0000","last_became_active":"2024-09-21T12:27:14.941888+0000","last_became_peered":"2024-09-21T12:27:14.941888+0000","last_unstale":"2024-09-21T12:28:55.979666+0000","last_undegraded":"2024-09-21T12:28:55.979666+0000","last_fullsized":"2024-09-21T12:28:55.979666+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:57.680181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:57.680181+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:29:17.469205+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131344,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":479,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.981606+0000","last_change":"2024-09-21T12:28:46.355910+0000","last_active":"2024-09-21T12:28:55.981606+0000","last_peered":"2024-09-21T12:28:55.981606+0000","last_clean":"2024-09-21T12:28:55.981606+0000","last_became_active":"2024-09-21T12:23:46.897197+0000","last_became_peered":"2024-09-21T12:23:46.897197+0000","last_unstale":"2024-09-21T12:28:55.981606+0000","last_undegraded":"2024-09-21T12:28:55.981606+0000","last_fullsized":"2024-09-21T12:28:55.981606+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-21T12:28:46.355822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:46.355822+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:54.101822+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":458,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.993740+0000","last_change":"2024-09-21T12:28:06.236794+0000","last_active":"2024-09-21T12:28:55.993740+0000","last_peered":"2024-09-21T12:28:55.993740+0000","last_clean":"2024-09-21T12:28:55.993740+0000","last_became_active":"2024-09-21T12:22:50.906886+0000","last_became_peered":"2024-09-21T12:22:50.906886+0000","last_unstale":"2024-09-21T12:28:55.993740+0000","last_undegraded":"2024-09-21T12:28:55.993740+0000","last_fullsized":"2024-09-21T12:28:55.993740+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:06.236749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:06.236749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:29.504064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":434,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986539+0000","last_change":"2024-09-21T12:28:03.571042+0000","last_active":"2024-09-21T12:28:55.986539+0000","last_peered":"2024-09-21T12:28:55.986539+0000","last_clean":"2024-09-21T12:28:55.986539+0000","last_became_active":"2024-09-21T12:22:50.927438+0000","last_became_peered":"2024-09-21T12:22:50.927438+0000","last_unstale":"2024-09-21T12:28:55.986539+0000","last_undegraded":"2024-09-21T12:28:55.986539+0000","last_fullsized":"2024-09-21T12:28:55.986539+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:03.570984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:03.570984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:18.975167+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":423,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.981534+0000","last_change":"2024-09-21T12:25:39.079146+0000","last_active":"2024-09-21T12:28:55.981534+0000","last_peered":"2024-09-21T12:28:55.981534+0000","last_clean":"2024-09-21T12:28:55.981534+0000","last_became_active":"2024-09-21T12:22:50.913898+0000","last_became_peered":"2024-09-21T12:22:50.913898+0000","last_unstale":"2024-09-21T12:28:55.981534+0000","last_undegraded":"2024-09-21T12:28:55.981534+0000","last_fullsized":"2024-09-21T12:28:55.981534+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:39.079015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:39.079015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:15.448755+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"255'3573","reported_seq":3621,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:57.322578+0000","last_change":"2024-09-21T12:28:21.114763+0000","last_active":"2024-09-21T12:28:57.322578+0000","last_peered":"2024-09-21T12:28:57.322578+0000","last_clean":"2024-09-21T12:28:57.322578+0000","last_became_active":"2024-09-21T12:28:21.114432+0000","last_became_peered":"2024-09-21T12:28:21.114432+0000","last_unstale":"2024-09-21T12:28:57.322578+0000","last_undegraded":"2024-09-21T12:28:57.322578+0000","last_fullsized":"2024-09-21T12:28:57.322578+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3573,"log_dups_size":0,"ondisk_log_size":3573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:43.773638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234160128,"num_objects":3573,"num_object_clones":0,"num_object_copies":10719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3573,"num_write_kb":228672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"76'79","reported_seq":496,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.985910+0000","last_change":"2024-09-21T12:27:17.370605+0000","last_active":"2024-09-21T12:28:55.985910+0000","last_peered":"2024-09-21T12:28:55.985910+0000","last_clean":"2024-09-21T12:28:55.985910+0000","last_became_active":"2024-09-21T12:27:14.943705+0000","last_became_peered":"2024-09-21T12:27:14.943705+0000","last_unstale":"2024-09-21T12:28:55.985910+0000","last_undegraded":"2024-09-21T12:28:55.985910+0000","last_fullsized":"2024-09-21T12:28:55.985910+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:17.370530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:17.370530+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":285,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.980492+0000","last_change":"2024-09-21T12:27:34.477082+0000","last_active":"2024-09-21T12:28:55.980492+0000","last_peered":"2024-09-21T12:28:55.980492+0000","last_clean":"2024-09-21T12:28:55.980492+0000","last_became_active":"2024-09-21T12:23:46.902260+0000","last_became_peered":"2024-09-21T12:23:46.902260+0000","last_unstale":"2024-09-21T12:28:55.980492+0000","last_undegraded":"2024-09-21T12:28:55.980492+0000","last_fullsized":"2024-09-21T12:28:55.980492+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:27:34.476998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:34.476998+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:04.288014+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":441,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.985127+0000","last_change":"2024-09-21T12:27:32.433938+0000","last_active":"2024-09-21T12:28:55.985127+0000","last_peered":"2024-09-21T12:28:55.985127+0000","last_clean":"2024-09-21T12:28:55.985127+0000","last_became_active":"2024-09-21T12:22:50.927946+0000","last_became_peered":"2024-09-21T12:22:50.927946+0000","last_unstale":"2024-09-21T12:28:55.985127+0000","last_undegraded":"2024-09-21T12:28:55.985127+0000","last_fullsized":"2024-09-21T12:28:55.985127+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:32.433867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:27:32.433867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":543,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.980983+0000","last_change":"2024-09-21T12:28:39.418167+0000","last_active":"2024-09-21T12:28:55.980983+0000","last_peered":"2024-09-21T12:28:55.980983+0000","last_clean":"2024-09-21T12:28:55.980983+0000","last_became_active":"2024-09-21T12:20:19.307924+0000","last_became_peered":"2024-09-21T12:20:19.307924+0000","last_unstale":"2024-09-21T12:28:55.980983+0000","last_undegraded":"2024-09-21T12:28:55.980983+0000","last_fullsized":"2024-09-21T12:28:55.980983+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:28:39.418085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:39.418085+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:08.013783+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":353,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986836+0000","last_change":"2024-09-21T12:28:35.670078+0000","last_active":"2024-09-21T12:28:55.986836+0000","last_peered":"2024-09-21T12:28:55.986836+0000","last_clean":"2024-09-21T12:28:55.986836+0000","last_became_active":"2024-09-21T12:22:50.905470+0000","last_became_peered":"2024-09-21T12:22:50.905470+0000","last_unstale":"2024-09-21T12:28:55.986836+0000","last_undegraded":"2024-09-21T12:28:55.986836+0000","last_fullsized":"2024-09-21T12:28:55.986836+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:35.669942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:35.669942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:01.370279+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"255'3689","reported_seq":3738,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:56.953778+0000","last_change":"2024-09-21T12:28:20.952349+0000","last_active":"2024-09-21T12:28:56.953778+0000","last_peered":"2024-09-21T12:28:56.953778+0000","last_clean":"2024-09-21T12:28:56.953778+0000","last_became_active":"2024-09-21T12:28:20.952083+0000","last_became_peered":"2024-09-21T12:28:20.952083+0000","last_unstale":"2024-09-21T12:28:56.953778+0000","last_undegraded":"2024-09-21T12:28:56.953778+0000","last_fullsized":"2024-09-21T12:28:56.953778+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3689,"log_dups_size":0,"ondisk_log_size":3689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:45.753435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241762304,"num_objects":3689,"num_object_clones":0,"num_object_copies":11067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3689,"num_write_kb":236096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"76'79","reported_seq":134,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.985444+0000","last_change":"2024-09-21T12:27:21.345414+0000","last_active":"2024-09-21T12:28:55.985444+0000","last_peered":"2024-09-21T12:28:55.985444+0000","last_clean":"2024-09-21T12:28:55.985444+0000","last_became_active":"2024-09-21T12:27:14.943330+0000","last_became_peered":"2024-09-21T12:27:14.943330+0000","last_unstale":"2024-09-21T12:28:55.985444+0000","last_undegraded":"2024-09-21T12:28:55.985444+0000","last_fullsized":"2024-09-21T12:28:55.985444+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:21.345364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:21.345364+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":494,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.997459+0000","last_change":"2024-09-21T12:25:00.237083+0000","last_active":"2024-09-21T12:28:55.997459+0000","last_peered":"2024-09-21T12:28:55.997459+0000","last_clean":"2024-09-21T12:28:55.997459+0000","last_became_active":"2024-09-21T12:22:50.922343+0000","last_became_peered":"2024-09-21T12:22:50.922343+0000","last_unstale":"2024-09-21T12:28:55.997459+0000","last_undegraded":"2024-09-21T12:28:55.997459+0000","last_fullsized":"2024-09-21T12:28:55.997459+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:25:00.236990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:00.236990+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:54.947680+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":104,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986651+0000","last_change":"2024-09-21T12:26:54.084563+0000","last_active":"2024-09-21T12:28:55.986651+0000","last_peered":"2024-09-21T12:28:55.986651+0000","last_clean":"2024-09-21T12:28:55.986651+0000","last_became_active":"2024-09-21T12:26:54.084228+0000","last_became_peered":"2024-09-21T12:26:54.084228+0000","last_unstale":"2024-09-21T12:28:55.986651+0000","last_undegraded":"2024-09-21T12:28:55.986651+0000","last_fullsized":"2024-09-21T12:28:55.986651+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:30.451876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:30.451876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:25.674786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":451,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.981224+0000","last_change":"2024-09-21T12:28:47.361181+0000","last_active":"2024-09-21T12:28:55.981224+0000","last_peered":"2024-09-21T12:28:55.981224+0000","last_clean":"2024-09-21T12:28:55.981224+0000","last_became_active":"2024-09-21T12:22:50.925974+0000","last_became_peered":"2024-09-21T12:22:50.925974+0000","last_unstale":"2024-09-21T12:28:55.981224+0000","last_undegraded":"2024-09-21T12:28:55.981224+0000","last_fullsized":"2024-09-21T12:28:55.981224+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:47.361115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:47.361115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:57.938678+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":411,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.993622+0000","last_change":"2024-09-21T12:28:43.821102+0000","last_active":"2024-09-21T12:28:55.993622+0000","last_peered":"2024-09-21T12:28:55.993622+0000","last_clean":"2024-09-21T12:28:55.993622+0000","last_became_active":"2024-09-21T12:20:21.311493+0000","last_became_peered":"2024-09-21T12:20:21.311493+0000","last_unstale":"2024-09-21T12:28:55.993622+0000","last_undegraded":"2024-09-21T12:28:55.993622+0000","last_fullsized":"2024-09-21T12:28:55.993622+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:43.820979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:43.820979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:48.567558+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"255'3536","reported_seq":3584,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:56.659270+0000","last_change":"2024-09-21T12:28:20.958504+0000","last_active":"2024-09-21T12:28:56.659270+0000","last_peered":"2024-09-21T12:28:56.659270+0000","last_clean":"2024-09-21T12:28:56.659270+0000","last_became_active":"2024-09-21T12:28:20.958112+0000","last_became_peered":"2024-09-21T12:28:20.958112+0000","last_unstale":"2024-09-21T12:28:56.659270+0000","last_undegraded":"2024-09-21T12:28:56.659270+0000","last_fullsized":"2024-09-21T12:28:56.659270+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3536,"log_dups_size":0,"ondisk_log_size":3536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:27.450929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231735296,"num_objects":3536,"num_object_clones":0,"num_object_copies":10608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3536,"num_write_kb":226304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"76'79","reported_seq":438,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.994248+0000","last_change":"2024-09-21T12:26:55.832199+0000","last_active":"2024-09-21T12:28:55.994248+0000","last_peered":"2024-09-21T12:28:55.994248+0000","last_clean":"2024-09-21T12:28:55.994248+0000","last_became_active":"2024-09-21T12:26:46.986203+0000","last_became_peered":"2024-09-21T12:26:46.986203+0000","last_unstale":"2024-09-21T12:28:55.994248+0000","last_undegraded":"2024-09-21T12:28:55.994248+0000","last_fullsized":"2024-09-21T12:28:55.994248+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:55.832132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:55.832132+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:16.157831+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":456,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.993538+0000","last_change":"2024-09-21T12:27:30.898787+0000","last_active":"2024-09-21T12:28:55.993538+0000","last_peered":"2024-09-21T12:28:55.993538+0000","last_clean":"2024-09-21T12:28:55.993538+0000","last_became_active":"2024-09-21T12:20:17.692867+0000","last_became_peered":"2024-09-21T12:20:17.692867+0000","last_unstale":"2024-09-21T12:28:55.993538+0000","last_undegraded":"2024-09-21T12:28:55.993538+0000","last_fullsized":"2024-09-21T12:28:55.993538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-21T12:27:30.898706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:30.898706+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":421,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986505+0000","last_change":"2024-09-21T12:28:48.438834+0000","last_active":"2024-09-21T12:28:55.986505+0000","last_peered":"2024-09-21T12:28:55.986505+0000","last_clean":"2024-09-21T12:28:55.986505+0000","last_became_active":"2024-09-21T12:20:15.299333+0000","last_became_peered":"2024-09-21T12:20:15.299333+0000","last_unstale":"2024-09-21T12:28:55.986505+0000","last_undegraded":"2024-09-21T12:28:55.986505+0000","last_fullsized":"2024-09-21T12:28:55.986505+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:48.438755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:48.438755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:17.800439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":414,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.980294+0000","last_change":"2024-09-21T12:26:50.626932+0000","last_active":"2024-09-21T12:28:55.980294+0000","last_peered":"2024-09-21T12:28:55.980294+0000","last_clean":"2024-09-21T12:28:55.980294+0000","last_became_active":"2024-09-21T12:20:19.306952+0000","last_became_peered":"2024-09-21T12:20:19.306952+0000","last_unstale":"2024-09-21T12:28:55.980294+0000","last_undegraded":"2024-09-21T12:28:55.980294+0000","last_fullsized":"2024-09-21T12:28:55.980294+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:50.626780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:50.626780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:18.506880+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":393,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.985221+0000","last_change":"2024-09-21T12:20:21.314891+0000","last_active":"2024-09-21T12:28:55.985221+0000","last_peered":"2024-09-21T12:28:55.985221+0000","last_clean":"2024-09-21T12:28:55.985221+0000","last_became_active":"2024-09-21T12:20:21.314545+0000","last_became_peered":"2024-09-21T12:20:21.314545+0000","last_unstale":"2024-09-21T12:28:55.985221+0000","last_undegraded":"2024-09-21T12:28:55.985221+0000","last_fullsized":"2024-09-21T12:28:55.985221+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:20:20.292557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:09.634903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"255'3651","reported_seq":3699,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:56.980229+0000","last_change":"2024-09-21T12:28:20.955882+0000","last_active":"2024-09-21T12:28:56.980229+0000","last_peered":"2024-09-21T12:28:56.980229+0000","last_clean":"2024-09-21T12:28:56.980229+0000","last_became_active":"2024-09-21T12:28:20.955605+0000","last_became_peered":"2024-09-21T12:28:20.955605+0000","last_unstale":"2024-09-21T12:28:56.980229+0000","last_undegraded":"2024-09-21T12:28:56.980229+0000","last_fullsized":"2024-09-21T12:28:56.980229+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3651,"log_dups_size":0,"ondisk_log_size":3651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:37.225243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239271936,"num_objects":3651,"num_object_clones":0,"num_object_copies":10953,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3651,"num_write_kb":233664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"76'79","reported_seq":374,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986367+0000","last_change":"2024-09-21T12:27:17.158443+0000","last_active":"2024-09-21T12:28:55.986367+0000","last_peered":"2024-09-21T12:28:55.986367+0000","last_clean":"2024-09-21T12:28:55.986367+0000","last_became_active":"2024-09-21T12:27:14.951375+0000","last_became_peered":"2024-09-21T12:27:14.951375+0000","last_unstale":"2024-09-21T12:28:55.986367+0000","last_undegraded":"2024-09-21T12:28:55.986367+0000","last_fullsized":"2024-09-21T12:28:55.986367+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:17.158380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:17.158380+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":428,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986090+0000","last_change":"2024-09-21T12:28:40.694225+0000","last_active":"2024-09-21T12:28:55.986090+0000","last_peered":"2024-09-21T12:28:55.986090+0000","last_clean":"2024-09-21T12:28:55.986090+0000","last_became_active":"2024-09-21T12:22:50.910973+0000","last_became_peered":"2024-09-21T12:22:50.910973+0000","last_unstale":"2024-09-21T12:28:55.986090+0000","last_undegraded":"2024-09-21T12:28:55.986090+0000","last_fullsized":"2024-09-21T12:28:55.986090+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:28:40.694138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:40.694138+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:56.646482+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":450,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986450+0000","last_change":"2024-09-21T12:28:30.612213+0000","last_active":"2024-09-21T12:28:55.986450+0000","last_peered":"2024-09-21T12:28:55.986450+0000","last_clean":"2024-09-21T12:28:55.986450+0000","last_became_active":"2024-09-21T12:22:50.927859+0000","last_became_peered":"2024-09-21T12:22:50.927859+0000","last_unstale":"2024-09-21T12:28:55.986450+0000","last_undegraded":"2024-09-21T12:28:55.986450+0000","last_fullsized":"2024-09-21T12:28:55.986450+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:30.612156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:30.612156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:55.060481+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":422,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.979859+0000","last_change":"2024-09-21T12:27:32.457547+0000","last_active":"2024-09-21T12:28:55.979859+0000","last_peered":"2024-09-21T12:28:55.979859+0000","last_clean":"2024-09-21T12:28:55.979859+0000","last_became_active":"2024-09-21T12:20:19.312572+0000","last_became_peered":"2024-09-21T12:20:19.312572+0000","last_unstale":"2024-09-21T12:28:55.979859+0000","last_undegraded":"2024-09-21T12:28:55.979859+0000","last_fullsized":"2024-09-21T12:28:55.979859+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:32.457484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:27:32.457484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:01.778622+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":408,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.984502+0000","last_change":"2024-09-21T12:26:58.321950+0000","last_active":"2024-09-21T12:28:55.984502+0000","last_peered":"2024-09-21T12:28:55.984502+0000","last_clean":"2024-09-21T12:28:55.984502+0000","last_became_active":"2024-09-21T12:20:21.317223+0000","last_became_peered":"2024-09-21T12:20:21.317223+0000","last_unstale":"2024-09-21T12:28:55.984502+0000","last_undegraded":"2024-09-21T12:28:55.984502+0000","last_fullsized":"2024-09-21T12:28:55.984502+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:58.321890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:58.321890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"255'3596","reported_seq":3645,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:57.363625+0000","last_change":"2024-09-21T12:28:20.951697+0000","last_active":"2024-09-21T12:28:57.363625+0000","last_peered":"2024-09-21T12:28:57.363625+0000","last_clean":"2024-09-21T12:28:57.363625+0000","last_became_active":"2024-09-21T12:28:20.951382+0000","last_became_peered":"2024-09-21T12:28:20.951382+0000","last_unstale":"2024-09-21T12:28:57.363625+0000","last_undegraded":"2024-09-21T12:28:57.363625+0000","last_fullsized":"2024-09-21T12:28:57.363625+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3596,"log_dups_size":0,"ondisk_log_size":3596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:21.160519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235667456,"num_objects":3596,"num_object_clones":0,"num_object_copies":10788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3596,"num_write_kb":230144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"76'79","reported_seq":134,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986023+0000","last_change":"2024-09-21T12:27:18.331811+0000","last_active":"2024-09-21T12:28:55.986023+0000","last_peered":"2024-09-21T12:28:55.986023+0000","last_clean":"2024-09-21T12:28:55.986023+0000","last_became_active":"2024-09-21T12:27:14.944162+0000","last_became_peered":"2024-09-21T12:27:14.944162+0000","last_unstale":"2024-09-21T12:28:55.986023+0000","last_undegraded":"2024-09-21T12:28:55.986023+0000","last_fullsized":"2024-09-21T12:28:55.986023+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:18.331743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:18.331743+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":451,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.987415+0000","last_change":"2024-09-21T12:25:03.416100+0000","last_active":"2024-09-21T12:28:55.987415+0000","last_peered":"2024-09-21T12:28:55.987415+0000","last_clean":"2024-09-21T12:28:55.987415+0000","last_became_active":"2024-09-21T12:20:17.692870+0000","last_became_peered":"2024-09-21T12:20:17.692870+0000","last_unstale":"2024-09-21T12:28:55.987415+0000","last_undegraded":"2024-09-21T12:28:55.987415+0000","last_fullsized":"2024-09-21T12:28:55.987415+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:25:03.416033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:03.416033+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:53.209881+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":472,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986366+0000","last_change":"2024-09-21T12:24:29.054851+0000","last_active":"2024-09-21T12:28:55.986366+0000","last_peered":"2024-09-21T12:28:55.986366+0000","last_clean":"2024-09-21T12:28:55.986366+0000","last_became_active":"2024-09-21T12:22:50.927141+0000","last_became_peered":"2024-09-21T12:22:50.927141+0000","last_unstale":"2024-09-21T12:28:55.986366+0000","last_undegraded":"2024-09-21T12:28:55.986366+0000","last_fullsized":"2024-09-21T12:28:55.986366+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-21T12:24:29.054772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:29.054772+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:55.390786+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"121'11","reported_seq":551,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986136+0000","last_change":"2024-09-21T12:25:02.916145+0000","last_active":"2024-09-21T12:28:55.986136+0000","last_peered":"2024-09-21T12:28:55.986136+0000","last_clean":"2024-09-21T12:28:55.986136+0000","last_became_active":"2024-09-21T12:22:50.908371+0000","last_became_peered":"2024-09-21T12:22:50.908371+0000","last_unstale":"2024-09-21T12:28:55.986136+0000","last_undegraded":"2024-09-21T12:28:55.986136+0000","last_fullsized":"2024-09-21T12:28:55.986136+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:25:02.916090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:02.916090+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:22.744691+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":112,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.993474+0000","last_change":"2024-09-21T12:26:56.862895+0000","last_active":"2024-09-21T12:28:55.993474+0000","last_peered":"2024-09-21T12:28:55.993474+0000","last_clean":"2024-09-21T12:28:55.993474+0000","last_became_active":"2024-09-21T12:26:54.089155+0000","last_became_peered":"2024-09-21T12:26:54.089155+0000","last_unstale":"2024-09-21T12:28:55.993474+0000","last_undegraded":"2024-09-21T12:28:55.993474+0000","last_fullsized":"2024-09-21T12:28:55.993474+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:56.862823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:56.862823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:17.174105+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"76'79","reported_seq":128,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.988120+0000","last_change":"2024-09-21T12:28:05.321813+0000","last_active":"2024-09-21T12:28:55.988120+0000","last_peered":"2024-09-21T12:28:55.988120+0000","last_clean":"2024-09-21T12:28:55.988120+0000","last_became_active":"2024-09-21T12:26:46.971481+0000","last_became_peered":"2024-09-21T12:26:46.971481+0000","last_unstale":"2024-09-21T12:28:55.988120+0000","last_undegraded":"2024-09-21T12:28:55.988120+0000","last_fullsized":"2024-09-21T12:28:55.988120+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:05.321753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:05.321753+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:10.468782+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":445,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.993299+0000","last_change":"2024-09-21T12:27:29.901101+0000","last_active":"2024-09-21T12:28:55.993299+0000","last_peered":"2024-09-21T12:28:55.993299+0000","last_clean":"2024-09-21T12:28:55.993299+0000","last_became_active":"2024-09-21T12:20:15.303480+0000","last_became_peered":"2024-09-21T12:20:15.303480+0000","last_unstale":"2024-09-21T12:28:55.993299+0000","last_undegraded":"2024-09-21T12:28:55.993299+0000","last_fullsized":"2024-09-21T12:28:55.993299+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:29.901043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:27:29.901043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:55.925004+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":425,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.985822+0000","last_change":"2024-09-21T12:26:57.369721+0000","last_active":"2024-09-21T12:28:55.985822+0000","last_peered":"2024-09-21T12:28:55.985822+0000","last_clean":"2024-09-21T12:28:55.985822+0000","last_became_active":"2024-09-21T12:20:17.694628+0000","last_became_peered":"2024-09-21T12:20:17.694628+0000","last_unstale":"2024-09-21T12:28:55.985822+0000","last_undegraded":"2024-09-21T12:28:55.985822+0000","last_fullsized":"2024-09-21T12:28:55.985822+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-21T12:26:57.369648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:57.369648+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":105,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.985133+0000","last_change":"2024-09-21T12:26:54.084503+0000","last_active":"2024-09-21T12:28:55.985133+0000","last_peered":"2024-09-21T12:28:55.985133+0000","last_clean":"2024-09-21T12:28:55.985133+0000","last_became_active":"2024-09-21T12:26:54.084109+0000","last_became_peered":"2024-09-21T12:26:54.084109+0000","last_unstale":"2024-09-21T12:28:55.985133+0000","last_undegraded":"2024-09-21T12:28:55.985133+0000","last_fullsized":"2024-09-21T12:28:55.985133+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:02.414298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:02.414298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:20.592593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":422,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986039+0000","last_change":"2024-09-21T12:24:00.066522+0000","last_active":"2024-09-21T12:28:55.986039+0000","last_peered":"2024-09-21T12:28:55.986039+0000","last_clean":"2024-09-21T12:28:55.986039+0000","last_became_active":"2024-09-21T12:22:50.904404+0000","last_became_peered":"2024-09-21T12:22:50.904404+0000","last_unstale":"2024-09-21T12:28:55.986039+0000","last_undegraded":"2024-09-21T12:28:55.986039+0000","last_fullsized":"2024-09-21T12:28:55.986039+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:00.066451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:00.066451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:50.498528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'79","reported_seq":369,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.987034+0000","last_change":"2024-09-21T12:28:29.546727+0000","last_active":"2024-09-21T12:28:55.987034+0000","last_peered":"2024-09-21T12:28:55.987034+0000","last_clean":"2024-09-21T12:28:55.987034+0000","last_became_active":"2024-09-21T12:26:46.994966+0000","last_became_peered":"2024-09-21T12:26:46.994966+0000","last_unstale":"2024-09-21T12:28:55.987034+0000","last_undegraded":"2024-09-21T12:28:55.987034+0000","last_fullsized":"2024-09-21T12:28:55.987034+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:29.546657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:29.546657+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:42.115154+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":443,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.994436+0000","last_change":"2024-09-21T12:28:01.881249+0000","last_active":"2024-09-21T12:28:55.994436+0000","last_peered":"2024-09-21T12:28:55.994436+0000","last_clean":"2024-09-21T12:28:55.994436+0000","last_became_active":"2024-09-21T12:20:17.695898+0000","last_became_peered":"2024-09-21T12:20:17.695898+0000","last_unstale":"2024-09-21T12:28:55.994436+0000","last_undegraded":"2024-09-21T12:28:55.994436+0000","last_fullsized":"2024-09-21T12:28:55.994436+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:28:01.881164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:01.881164+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:03.000373+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":458,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986119+0000","last_change":"2024-09-21T12:28:43.545259+0000","last_active":"2024-09-21T12:28:55.986119+0000","last_peered":"2024-09-21T12:28:55.986119+0000","last_clean":"2024-09-21T12:28:55.986119+0000","last_became_active":"2024-09-21T12:22:50.927649+0000","last_became_peered":"2024-09-21T12:22:50.927649+0000","last_unstale":"2024-09-21T12:28:55.986119+0000","last_undegraded":"2024-09-21T12:28:55.986119+0000","last_fullsized":"2024-09-21T12:28:55.986119+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:43.545197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:43.545197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:51.627798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":412,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.985557+0000","last_change":"2024-09-21T12:28:47.964092+0000","last_active":"2024-09-21T12:28:55.985557+0000","last_peered":"2024-09-21T12:28:55.985557+0000","last_clean":"2024-09-21T12:28:55.985557+0000","last_became_active":"2024-09-21T12:20:19.311947+0000","last_became_peered":"2024-09-21T12:20:19.311947+0000","last_unstale":"2024-09-21T12:28:55.985557+0000","last_undegraded":"2024-09-21T12:28:55.985557+0000","last_fullsized":"2024-09-21T12:28:55.985557+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:47.963747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:47.963747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:12.338200+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":447,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.980582+0000","last_change":"2024-09-21T12:26:11.105716+0000","last_active":"2024-09-21T12:28:55.980582+0000","last_peered":"2024-09-21T12:28:55.980582+0000","last_clean":"2024-09-21T12:28:55.980582+0000","last_became_active":"2024-09-21T12:22:50.924931+0000","last_became_peered":"2024-09-21T12:22:50.924931+0000","last_unstale":"2024-09-21T12:28:55.980582+0000","last_undegraded":"2024-09-21T12:28:55.980582+0000","last_fullsized":"2024-09-21T12:28:55.980582+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:11.105583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:11.105583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:51.335162+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"76'79","reported_seq":497,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.984946+0000","last_change":"2024-09-21T12:26:54.085274+0000","last_active":"2024-09-21T12:28:55.984946+0000","last_peered":"2024-09-21T12:28:55.984946+0000","last_clean":"2024-09-21T12:28:55.984946+0000","last_became_active":"2024-09-21T12:26:54.084939+0000","last_became_peered":"2024-09-21T12:26:54.084939+0000","last_unstale":"2024-09-21T12:28:55.984946+0000","last_undegraded":"2024-09-21T12:28:55.984946+0000","last_fullsized":"2024-09-21T12:28:55.984946+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:50.443344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:50.443344+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:29:16.709698+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":148,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986337+0000","last_change":"2024-09-21T12:28:42.511948+0000","last_active":"2024-09-21T12:28:55.986337+0000","last_peered":"2024-09-21T12:28:55.986337+0000","last_clean":"2024-09-21T12:28:55.986337+0000","last_became_active":"2024-09-21T12:26:54.101937+0000","last_became_peered":"2024-09-21T12:26:54.101937+0000","last_unstale":"2024-09-21T12:28:55.986337+0000","last_undegraded":"2024-09-21T12:28:55.986337+0000","last_fullsized":"2024-09-21T12:28:55.986337+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:28:42.511874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:42.511874+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:46.711686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":431,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.993639+0000","last_change":"2024-09-21T12:28:07.875348+0000","last_active":"2024-09-21T12:28:55.993639+0000","last_peered":"2024-09-21T12:28:55.993639+0000","last_clean":"2024-09-21T12:28:55.993639+0000","last_became_active":"2024-09-21T12:20:15.297519+0000","last_became_peered":"2024-09-21T12:20:15.297519+0000","last_unstale":"2024-09-21T12:28:55.993639+0000","last_undegraded":"2024-09-21T12:28:55.993639+0000","last_fullsized":"2024-09-21T12:28:55.993639+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:07.875304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:07.875304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:36.761971+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":525,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986147+0000","last_change":"2024-09-21T12:28:43.854094+0000","last_active":"2024-09-21T12:28:55.986147+0000","last_peered":"2024-09-21T12:28:55.986147+0000","last_clean":"2024-09-21T12:28:55.986147+0000","last_became_active":"2024-09-21T12:20:19.305364+0000","last_became_peered":"2024-09-21T12:20:19.305364+0000","last_unstale":"2024-09-21T12:28:55.986147+0000","last_undegraded":"2024-09-21T12:28:55.986147+0000","last_fullsized":"2024-09-21T12:28:55.986147+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:28:43.853712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:43.853712+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:03.508462+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":354,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.981661+0000","last_change":"2024-09-21T12:28:05.323501+0000","last_active":"2024-09-21T12:28:55.981661+0000","last_peered":"2024-09-21T12:28:55.981661+0000","last_clean":"2024-09-21T12:28:55.981661+0000","last_became_active":"2024-09-21T12:22:50.924770+0000","last_became_peered":"2024-09-21T12:22:50.924770+0000","last_unstale":"2024-09-21T12:28:55.981661+0000","last_undegraded":"2024-09-21T12:28:55.981661+0000","last_fullsized":"2024-09-21T12:28:55.981661+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:05.323448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:05.323448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:19.896140+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"76'79","reported_seq":211,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.993712+0000","last_change":"2024-09-21T12:26:48.804298+0000","last_active":"2024-09-21T12:28:55.993712+0000","last_peered":"2024-09-21T12:28:55.993712+0000","last_clean":"2024-09-21T12:28:55.993712+0000","last_became_active":"2024-09-21T12:26:46.993017+0000","last_became_peered":"2024-09-21T12:26:46.993017+0000","last_unstale":"2024-09-21T12:28:55.993712+0000","last_undegraded":"2024-09-21T12:28:55.993712+0000","last_fullsized":"2024-09-21T12:28:55.993712+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:48.804235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:48.804235+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:19.648260+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":431,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986512+0000","last_change":"2024-09-21T12:24:50.727161+0000","last_active":"2024-09-21T12:28:55.986512+0000","last_peered":"2024-09-21T12:28:55.986512+0000","last_clean":"2024-09-21T12:28:55.986512+0000","last_became_active":"2024-09-21T12:20:17.693209+0000","last_became_peered":"2024-09-21T12:20:17.693209+0000","last_unstale":"2024-09-21T12:28:55.986512+0000","last_undegraded":"2024-09-21T12:28:55.986512+0000","last_fullsized":"2024-09-21T12:28:55.986512+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:24:50.727101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:50.727101+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:17.579086+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":450,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.985895+0000","last_change":"2024-09-21T12:28:32.630320+0000","last_active":"2024-09-21T12:28:55.985895+0000","last_peered":"2024-09-21T12:28:55.985895+0000","last_clean":"2024-09-21T12:28:55.985895+0000","last_became_active":"2024-09-21T12:22:50.908238+0000","last_became_peered":"2024-09-21T12:22:50.908238+0000","last_unstale":"2024-09-21T12:28:55.985895+0000","last_undegraded":"2024-09-21T12:28:55.985895+0000","last_fullsized":"2024-09-21T12:28:55.985895+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:32.630261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:32.630261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:53.164228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":526,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.988233+0000","last_change":"2024-09-21T12:28:36.293443+0000","last_active":"2024-09-21T12:28:55.988233+0000","last_peered":"2024-09-21T12:28:55.988233+0000","last_clean":"2024-09-21T12:28:55.988233+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:28:55.988233+0000","last_undegraded":"2024-09-21T12:28:55.988233+0000","last_fullsized":"2024-09-21T12:28:55.988233+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:28:36.293377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:36.293377+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:01.258654+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":403,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.994190+0000","last_change":"2024-09-21T12:26:08.926904+0000","last_active":"2024-09-21T12:28:55.994190+0000","last_peered":"2024-09-21T12:28:55.994190+0000","last_clean":"2024-09-21T12:28:55.994190+0000","last_became_active":"2024-09-21T12:20:21.309385+0000","last_became_peered":"2024-09-21T12:20:21.309385+0000","last_unstale":"2024-09-21T12:28:55.994190+0000","last_undegraded":"2024-09-21T12:28:55.994190+0000","last_fullsized":"2024-09-21T12:28:55.994190+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:08.923458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:08.923458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:16.660312+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"76'79","reported_seq":90,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.985784+0000","last_change":"2024-09-21T12:27:18.292942+0000","last_active":"2024-09-21T12:28:55.985784+0000","last_peered":"2024-09-21T12:28:55.985784+0000","last_clean":"2024-09-21T12:28:55.985784+0000","last_became_active":"2024-09-21T12:27:16.940347+0000","last_became_peered":"2024-09-21T12:27:16.940347+0000","last_unstale":"2024-09-21T12:28:55.985784+0000","last_undegraded":"2024-09-21T12:28:55.985784+0000","last_fullsized":"2024-09-21T12:28:55.985784+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:18.292862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:18.292862+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":417,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.994254+0000","last_change":"2024-09-21T12:28:40.150322+0000","last_active":"2024-09-21T12:28:55.994254+0000","last_peered":"2024-09-21T12:28:55.994254+0000","last_clean":"2024-09-21T12:28:55.994254+0000","last_became_active":"2024-09-21T12:22:50.906473+0000","last_became_peered":"2024-09-21T12:22:50.906473+0000","last_unstale":"2024-09-21T12:28:55.994254+0000","last_undegraded":"2024-09-21T12:28:55.994254+0000","last_fullsized":"2024-09-21T12:28:55.994254+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:28:40.150250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:40.150250+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:42.939288+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":338,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986401+0000","last_change":"2024-09-21T12:24:58.948698+0000","last_active":"2024-09-21T12:28:55.986401+0000","last_peered":"2024-09-21T12:28:55.986401+0000","last_clean":"2024-09-21T12:28:55.986401+0000","last_became_active":"2024-09-21T12:22:50.903705+0000","last_became_peered":"2024-09-21T12:22:50.903705+0000","last_unstale":"2024-09-21T12:28:55.986401+0000","last_undegraded":"2024-09-21T12:28:55.986401+0000","last_fullsized":"2024-09-21T12:28:55.986401+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:58.948625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:58.948625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:21.759502+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":408,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986655+0000","last_change":"2024-09-21T12:26:16.149823+0000","last_active":"2024-09-21T12:28:55.986655+0000","last_peered":"2024-09-21T12:28:55.986655+0000","last_clean":"2024-09-21T12:28:55.986655+0000","last_became_active":"2024-09-21T12:20:19.315418+0000","last_became_peered":"2024-09-21T12:20:19.315418+0000","last_unstale":"2024-09-21T12:28:55.986655+0000","last_undegraded":"2024-09-21T12:28:55.986655+0000","last_fullsized":"2024-09-21T12:28:55.986655+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:16.149550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:16.149550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:55.852478+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":408,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.984304+0000","last_change":"2024-09-21T12:24:56.879617+0000","last_active":"2024-09-21T12:28:55.984304+0000","last_peered":"2024-09-21T12:28:55.984304+0000","last_clean":"2024-09-21T12:28:55.984304+0000","last_became_active":"2024-09-21T12:20:21.310053+0000","last_became_peered":"2024-09-21T12:20:21.310053+0000","last_unstale":"2024-09-21T12:28:55.984304+0000","last_undegraded":"2024-09-21T12:28:55.984304+0000","last_fullsized":"2024-09-21T12:28:55.984304+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:56.879558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:56.879558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:23.616386+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"76'79","reported_seq":99,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.988000+0000","last_change":"2024-09-21T12:28:31.187538+0000","last_active":"2024-09-21T12:28:55.988000+0000","last_peered":"2024-09-21T12:28:55.988000+0000","last_clean":"2024-09-21T12:28:55.988000+0000","last_became_active":"2024-09-21T12:27:16.921880+0000","last_became_peered":"2024-09-21T12:27:16.921880+0000","last_unstale":"2024-09-21T12:28:55.988000+0000","last_undegraded":"2024-09-21T12:28:55.988000+0000","last_fullsized":"2024-09-21T12:28:55.988000+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:31.187467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:31.187467+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:53.371638+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":433,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986226+0000","last_change":"2024-09-21T12:28:36.646045+0000","last_active":"2024-09-21T12:28:55.986226+0000","last_peered":"2024-09-21T12:28:55.986226+0000","last_clean":"2024-09-21T12:28:55.986226+0000","last_became_active":"2024-09-21T12:20:17.299015+0000","last_became_peered":"2024-09-21T12:20:17.299015+0000","last_unstale":"2024-09-21T12:28:55.986226+0000","last_undegraded":"2024-09-21T12:28:55.986226+0000","last_fullsized":"2024-09-21T12:28:55.986226+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:28:36.645961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:36.645961+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:01.394775+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":450,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986472+0000","last_change":"2024-09-21T12:26:19.973528+0000","last_active":"2024-09-21T12:28:55.986472+0000","last_peered":"2024-09-21T12:28:55.986472+0000","last_clean":"2024-09-21T12:28:55.986472+0000","last_became_active":"2024-09-21T12:22:50.922019+0000","last_became_peered":"2024-09-21T12:22:50.922019+0000","last_unstale":"2024-09-21T12:28:55.986472+0000","last_undegraded":"2024-09-21T12:28:55.986472+0000","last_fullsized":"2024-09-21T12:28:55.986472+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:19.973461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:19.973461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:51.477477+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":412,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.984590+0000","last_change":"2024-09-21T12:26:21.572474+0000","last_active":"2024-09-21T12:28:55.984590+0000","last_peered":"2024-09-21T12:28:55.984590+0000","last_clean":"2024-09-21T12:28:55.984590+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:28:55.984590+0000","last_undegraded":"2024-09-21T12:28:55.984590+0000","last_fullsized":"2024-09-21T12:28:55.984590+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:21.571726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:21.571726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":411,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.987994+0000","last_change":"2024-09-21T12:28:41.864711+0000","last_active":"2024-09-21T12:28:55.987994+0000","last_peered":"2024-09-21T12:28:55.987994+0000","last_clean":"2024-09-21T12:28:55.987994+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:28:55.987994+0000","last_undegraded":"2024-09-21T12:28:55.987994+0000","last_fullsized":"2024-09-21T12:28:55.987994+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:41.864630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:41.864630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:09.118573+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"76'79","reported_seq":120,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.980440+0000","last_change":"2024-09-21T12:26:49.624804+0000","last_active":"2024-09-21T12:28:55.980440+0000","last_peered":"2024-09-21T12:28:55.980440+0000","last_clean":"2024-09-21T12:28:55.980440+0000","last_became_active":"2024-09-21T12:26:46.989742+0000","last_became_peered":"2024-09-21T12:26:46.989742+0000","last_unstale":"2024-09-21T12:28:55.980440+0000","last_undegraded":"2024-09-21T12:28:55.980440+0000","last_fullsized":"2024-09-21T12:28:55.980440+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:49.623735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:49.623735+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:54.270969+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":435,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.984623+0000","last_change":"2024-09-21T12:24:58.824615+0000","last_active":"2024-09-21T12:28:55.984623+0000","last_peered":"2024-09-21T12:28:55.984623+0000","last_clean":"2024-09-21T12:28:55.984623+0000","last_became_active":"2024-09-21T12:20:17.694794+0000","last_became_peered":"2024-09-21T12:20:17.694794+0000","last_unstale":"2024-09-21T12:28:55.984623+0000","last_undegraded":"2024-09-21T12:28:55.984623+0000","last_fullsized":"2024-09-21T12:28:55.984623+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:24:58.824516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:58.824516+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:19.560401+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":437,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.988007+0000","last_change":"2024-09-21T12:28:45.330667+0000","last_active":"2024-09-21T12:28:55.988007+0000","last_peered":"2024-09-21T12:28:55.988007+0000","last_clean":"2024-09-21T12:28:55.988007+0000","last_became_active":"2024-09-21T12:20:15.295968+0000","last_became_peered":"2024-09-21T12:20:15.295968+0000","last_unstale":"2024-09-21T12:28:55.988007+0000","last_undegraded":"2024-09-21T12:28:55.988007+0000","last_fullsized":"2024-09-21T12:28:55.988007+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:45.330623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.330623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:52.780526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":426,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.984939+0000","last_change":"2024-09-21T12:23:57.018797+0000","last_active":"2024-09-21T12:28:55.984939+0000","last_peered":"2024-09-21T12:28:55.984939+0000","last_clean":"2024-09-21T12:28:55.984939+0000","last_became_active":"2024-09-21T12:22:50.921604+0000","last_became_peered":"2024-09-21T12:22:50.921604+0000","last_unstale":"2024-09-21T12:28:55.984939+0000","last_undegraded":"2024-09-21T12:28:55.984939+0000","last_fullsized":"2024-09-21T12:28:55.984939+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:57.018749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:57.018749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:20.736741+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":400,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.984811+0000","last_change":"2024-09-21T12:24:35.588409+0000","last_active":"2024-09-21T12:28:55.984811+0000","last_peered":"2024-09-21T12:28:55.984811+0000","last_clean":"2024-09-21T12:28:55.984811+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:28:55.984811+0000","last_undegraded":"2024-09-21T12:28:55.984811+0000","last_fullsized":"2024-09-21T12:28:55.984811+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:35.588353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:35.588353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:08.583895+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"76'79","reported_seq":91,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.985701+0000","last_change":"2024-09-21T12:27:24.373557+0000","last_active":"2024-09-21T12:28:55.985701+0000","last_peered":"2024-09-21T12:28:55.985701+0000","last_clean":"2024-09-21T12:28:55.985701+0000","last_became_active":"2024-09-21T12:27:16.914713+0000","last_became_peered":"2024-09-21T12:27:16.914713+0000","last_unstale":"2024-09-21T12:28:55.985701+0000","last_undegraded":"2024-09-21T12:28:55.985701+0000","last_fullsized":"2024-09-21T12:28:55.985701+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:24.373501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:24.373501+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":422,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.984357+0000","last_change":"2024-09-21T12:28:46.932522+0000","last_active":"2024-09-21T12:28:55.984357+0000","last_peered":"2024-09-21T12:28:55.984357+0000","last_clean":"2024-09-21T12:28:55.984357+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:28:55.984357+0000","last_undegraded":"2024-09-21T12:28:55.984357+0000","last_fullsized":"2024-09-21T12:28:55.984357+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-21T12:28:46.931901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:46.931901+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:14.037686+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":437,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.995601+0000","last_change":"2024-09-21T12:28:00.838008+0000","last_active":"2024-09-21T12:28:55.995601+0000","last_peered":"2024-09-21T12:28:55.995601+0000","last_clean":"2024-09-21T12:28:55.995601+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:28:55.995601+0000","last_undegraded":"2024-09-21T12:28:55.995601+0000","last_fullsized":"2024-09-21T12:28:55.995601+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:28:00.837937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:00.837937+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:12.552751+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":434,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.993324+0000","last_change":"2024-09-21T12:26:22.014941+0000","last_active":"2024-09-21T12:28:55.993324+0000","last_peered":"2024-09-21T12:28:55.993324+0000","last_clean":"2024-09-21T12:28:55.993324+0000","last_became_active":"2024-09-21T12:22:50.930615+0000","last_became_peered":"2024-09-21T12:22:50.930615+0000","last_unstale":"2024-09-21T12:28:55.993324+0000","last_undegraded":"2024-09-21T12:28:55.993324+0000","last_fullsized":"2024-09-21T12:28:55.993324+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:22.014896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:22.014896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:50.976154+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":430,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.985614+0000","last_change":"2024-09-21T12:28:41.537845+0000","last_active":"2024-09-21T12:28:55.985614+0000","last_peered":"2024-09-21T12:28:55.985614+0000","last_clean":"2024-09-21T12:28:55.985614+0000","last_became_active":"2024-09-21T12:22:50.917408+0000","last_became_peered":"2024-09-21T12:22:50.917408+0000","last_unstale":"2024-09-21T12:28:55.985614+0000","last_undegraded":"2024-09-21T12:28:55.985614+0000","last_fullsized":"2024-09-21T12:28:55.985614+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:41.537786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:41.537786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:52.332880+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"76'79","reported_seq":91,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.985763+0000","last_change":"2024-09-21T12:27:29.341314+0000","last_active":"2024-09-21T12:28:55.985763+0000","last_peered":"2024-09-21T12:28:55.985763+0000","last_clean":"2024-09-21T12:28:55.985763+0000","last_became_active":"2024-09-21T12:27:16.948697+0000","last_became_peered":"2024-09-21T12:27:16.948697+0000","last_unstale":"2024-09-21T12:28:55.985763+0000","last_undegraded":"2024-09-21T12:28:55.985763+0000","last_fullsized":"2024-09-21T12:28:55.985763+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:29.341268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:29.341268+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":430,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986430+0000","last_change":"2024-09-21T12:28:46.749904+0000","last_active":"2024-09-21T12:28:55.986430+0000","last_peered":"2024-09-21T12:28:55.986430+0000","last_clean":"2024-09-21T12:28:55.986430+0000","last_became_active":"2024-09-21T12:22:50.914221+0000","last_became_peered":"2024-09-21T12:22:50.914221+0000","last_unstale":"2024-09-21T12:28:55.986430+0000","last_undegraded":"2024-09-21T12:28:55.986430+0000","last_fullsized":"2024-09-21T12:28:55.986430+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:46.749839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:46.749839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:07.174169+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":420,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.986796+0000","last_change":"2024-09-21T12:25:00.871133+0000","last_active":"2024-09-21T12:28:55.986796+0000","last_peered":"2024-09-21T12:28:55.986796+0000","last_clean":"2024-09-21T12:28:55.986796+0000","last_became_active":"2024-09-21T12:20:15.297993+0000","last_became_peered":"2024-09-21T12:20:15.297993+0000","last_unstale":"2024-09-21T12:28:55.986796+0000","last_undegraded":"2024-09-21T12:28:55.986796+0000","last_fullsized":"2024-09-21T12:28:55.986796+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:00.871062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:00.871062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:11.582883+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":437,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.994391+0000","last_change":"2024-09-21T12:27:31.860075+0000","last_active":"2024-09-21T12:28:55.994391+0000","last_peered":"2024-09-21T12:28:55.994391+0000","last_clean":"2024-09-21T12:28:55.994391+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:28:55.994391+0000","last_undegraded":"2024-09-21T12:28:55.994391+0000","last_fullsized":"2024-09-21T12:28:55.994391+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:27:31.860002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:31.860002+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:53.990527+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":440,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-21T12:28:55.994236+0000","last_change":"2024-09-21T12:28:28.741483+0000","last_active":"2024-09-21T12:28:55.994236+0000","last_peered":"2024-09-21T12:28:55.994236+0000","last_clean":"2024-09-21T12:28:55.994236+0000","last_became_active":"2024-09-21T12:20:19.308247+0000","last_became_peered":"2024-09-21T12:20:19.308247+0000","last_unstale":"2024-09-21T12:28:55.994236+0000","last_undegraded":"2024-09-21T12:28:55.994236+0000","last_fullsized":"2024-09-21T12:28:55.994236+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:28.741418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:28.741418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:29:39.129562+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":3776184320,"num_objects":57620,"num_object_clones":0,"num_object_copies":172860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57620,"num_write_kb":3687680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11311251456,"data_stored":11311251456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57620,"ondisk_log_size":57620,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":689,"num_read_kb":446,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":3872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":396,"ondisk_log_size":396,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":5023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":516,"num_read_kb":511,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":721712,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2449,"ondisk_log_size":2449,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234092,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2344636,"kb_used_data":2034408,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":91400516,"statfs":{"total":95995035648,"available":93594128384,"internally_reserved":0,"allocated":2083233792,"data_stored":2082707490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":317651402},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463028,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":994412,"kb_used_data":674456,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92750740,"statfs":{"total":95995035648,"available":94976757760,"internally_reserved":0,"allocated":690642944,"data_stored":690124081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":327612870},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724663,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2336704,"kb_used_data":2026472,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":91408448,"statfs":{"total":95995035648,"available":93602250752,"internally_reserved":0,"allocated":2075107328,"data_stored":2074581916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953599,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149829,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2232204,"kb_used_data":1835900,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":91512948,"statfs":{"total":95995035648,"available":93709258752,"internally_reserved":0,"allocated":1879961600,"data_stored":1879415573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":405797319},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411467,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":972080,"kb_used_data":679896,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":92773072,"statfs":{"total":95995035648,"available":94999625728,"internally_reserved":0,"allocated":696213504,"data_stored":695702933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574993,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2453280,"kb_used_data":2066696,"kb_used_omap":1,"kb_used_meta":386558,"kb_avail":91291872,"statfs":{"total":95995035648,"available":93482876928,"internally_reserved":0,"allocated":2116296704,"data_stored":2115737409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":395835845},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738519,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1962308,"kb_used_data":1620140,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":91782844,"statfs":{"total":95995035648,"available":93985632256,"internally_reserved":0,"allocated":1659023360,"data_stored":1658493131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1676804096,"data_stored":1676804096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2125135872,"data_stored":2125135872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":707330048,"data_stored":707330048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1891827712,"data_stored":1891827712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2107703296,"data_stored":2107703296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":705560576,"data_stored":705560576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2096889856,"data_stored":2096889856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:29:00.060 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T12:29:00.061 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:29:00.061 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T12:29:00.061 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 0.409714 2024-09-21T12:29:00.061 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 6 0.4097139799106059 2024-09-21T12:29:00.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:00.062 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:00.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:00.163 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:00.173 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:29:00.208 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:00.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:00.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:00.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:00 smithi114 ceph-mon[30324]: from='client.25487 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:29:00.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:00 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1231930604' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:29:00.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:00 smithi114 ceph-mon[40867]: from='client.25487 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:29:00.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:00 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1231930604' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:29:00.301 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:00 smithi183 ceph-mon[33277]: from='client.25487 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:29:00.301 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:00 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1231930604' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:29:00.309 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:00.340 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:00.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:00.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:00.442 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:00.486 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:00.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:00.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:00.588 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:00.626 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:00.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:00.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:00.729 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:00.757 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:00.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:00.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:00.789 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:29:00.789 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:29:00.816 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:00.828 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 33 16 62813 62797 118.921 87.4375 0.0187505 0.00840323 2024-09-21T12:29:00.859 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:00.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:00.904 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:00.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:01.002 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:01.005 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:01.034 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:01.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:01.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:01.135 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:01.165 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:01.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:01.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:01.267 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:01.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:01 smithi114 ceph-mon[30324]: pgmap v430: 178 pgs: 178 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 656 B/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-21T12:29:01.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:01 smithi114 ceph-mon[40867]: pgmap v430: 178 pgs: 178 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 656 B/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-21T12:29:01.296 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:01.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:01.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:01.398 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:01.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:01.434 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:01.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:01.446 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:01 smithi183 ceph-mon[33277]: pgmap v430: 178 pgs: 178 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 656 B/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-21T12:29:01.487 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T12:29:01.535 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:01.555 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:01.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:01.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:01.656 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:01.691 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:01.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:01.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:01.793 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:01.824 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:01.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:01.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:01.828 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 34 16 64079 64063 117.75 79.125 0.0240537 0.00848682 2024-09-21T12:29:01.832 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:29:01.926 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:01.966 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:01.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:01.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:02.067 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:02.103 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:02.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:02.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:02.204 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:02.235 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:02.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:02.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:02.337 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:02.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:02.371 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:02.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:02.472 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:02.504 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:02.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:02.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:02.605 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:02.643 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:02.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:02.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:02.745 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:02.750 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:29:02.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:02.780 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:02.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:02.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:02 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:29:02.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:02 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:29:02.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:02 smithi114 ceph-mon[30324]: pgmap v431: 178 pgs: 178 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-09-21T12:29:02.801 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:29:02 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:29:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:29:02.801 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:02 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:29:02.801 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:02 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:29:02.801 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:02 smithi114 ceph-mon[40867]: pgmap v431: 178 pgs: 178 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-09-21T12:29:02.828 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 35 16 65457 65441 116.846 86.125 0.0487741 0.00854965 2024-09-21T12:29:02.881 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:02.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:02.920 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:02.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:02.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:02 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:29:02.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:02 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:29:02.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:02 smithi183 ceph-mon[33277]: pgmap v431: 178 pgs: 178 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-09-21T12:29:03.021 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:03.047 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:03.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:03.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:03.149 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:03.189 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:03.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:03.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:03.291 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:03.332 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:03.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:03.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:03.433 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:03.462 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:03.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:03.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:03.563 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:03.591 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:03.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:03.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:03.693 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:03.722 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:03.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:03.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:03.824 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:03.828 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 36 16 66565 66549 115.524 69.25 0.0113553 0.00865128 2024-09-21T12:29:03.855 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:03.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:03.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:03.957 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:03.990 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:03.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:03.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:04.092 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:04.123 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:04.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:04.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:04.225 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:04.261 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:04.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:04.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:04.363 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:04.392 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:04.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:04.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:04.494 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:04.523 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:04.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:04.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:04.585 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:04.625 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:04.648 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:04.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:04.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:04.749 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:04.794 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:04.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:04.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:04.800 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-21T12:29:04.801 INFO:teuthology.orchestra.run.smithi114.stderr:set osd.6 primary-affinity to 0.409714 (8268512) 2024-09-21T12:29:04.828 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 37 16 67656 67640 114.244 68.1875 0.0270819 0.00874642 2024-09-21T12:29:04.866 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:29:04.866 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:29:04.896 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:04.929 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:04.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:04.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:04.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:04 smithi183 ceph-mon[33277]: pgmap v432: 178 pgs: 178 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 100 MiB/s wr, 1.59k op/s 2024-09-21T12:29:04.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:04 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1419188300' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:29:04.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:04 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2392829313' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 0.4097139799106059} : dispatch 2024-09-21T12:29:05.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:04 smithi114 ceph-mon[40867]: pgmap v432: 178 pgs: 178 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 100 MiB/s wr, 1.59k op/s 2024-09-21T12:29:05.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:04 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1419188300' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:29:05.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:04 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2392829313' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 0.4097139799106059} : dispatch 2024-09-21T12:29:05.027 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:04 smithi114 ceph-mon[30324]: pgmap v432: 178 pgs: 178 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 100 MiB/s wr, 1.59k op/s 2024-09-21T12:29:05.027 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:04 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1419188300' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:29:05.027 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:04 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2392829313' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 0.4097139799106059} : dispatch 2024-09-21T12:29:05.031 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:05.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:05.058 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:05.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:05.159 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:05.188 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:05.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:05.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:05.290 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:05.324 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:05.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:05.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:05.425 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:05.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:05.468 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:05.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:05.569 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:05.598 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T12:29:05.604 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:05.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:05.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:05.706 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:05.744 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:05.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:05.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:05.828 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 38 16 69147 69131 113.69 93.1875 0.0133432 0.00878923 2024-09-21T12:29:05.830 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:05 smithi114 ceph-mon[30324]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:29:05.830 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:05 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1419188300' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:29:05.830 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:05 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2392829313' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.4097139799106059}]': finished 2024-09-21T12:29:05.830 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:05 smithi114 ceph-mon[30324]: osdmap e256: 8 total, 8 up, 7 in 2024-09-21T12:29:05.830 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:05 smithi114 ceph-mon[30324]: 6.2 scrub starts 2024-09-21T12:29:05.831 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:05 smithi114 ceph-mon[30324]: 6.2 scrub ok 2024-09-21T12:29:05.831 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:05 smithi114 ceph-mon[30324]: 1.9 scrub starts 2024-09-21T12:29:05.831 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:05 smithi114 ceph-mon[30324]: 1.9 scrub ok 2024-09-21T12:29:05.846 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:05.888 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:05.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:05.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:05.979 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:29:05.989 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:06.022 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:06.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:06.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:06.108 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:05 smithi183 ceph-mon[33277]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:29:06.108 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:05 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1419188300' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:29:06.108 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:05 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2392829313' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.4097139799106059}]': finished 2024-09-21T12:29:06.108 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:05 smithi183 ceph-mon[33277]: osdmap e256: 8 total, 8 up, 7 in 2024-09-21T12:29:06.109 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:05 smithi183 ceph-mon[33277]: 6.2 scrub starts 2024-09-21T12:29:06.109 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:05 smithi183 ceph-mon[33277]: 6.2 scrub ok 2024-09-21T12:29:06.109 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:05 smithi183 ceph-mon[33277]: 1.9 scrub starts 2024-09-21T12:29:06.109 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:05 smithi183 ceph-mon[33277]: 1.9 scrub ok 2024-09-21T12:29:06.123 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:05 smithi114 ceph-mon[40867]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:29:06.123 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:05 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1419188300' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:29:06.123 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:05 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2392829313' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.4097139799106059}]': finished 2024-09-21T12:29:06.123 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:05 smithi114 ceph-mon[40867]: osdmap e256: 8 total, 8 up, 7 in 2024-09-21T12:29:06.123 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:05 smithi114 ceph-mon[40867]: 6.2 scrub starts 2024-09-21T12:29:06.123 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:05 smithi114 ceph-mon[40867]: 6.2 scrub ok 2024-09-21T12:29:06.123 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:05 smithi114 ceph-mon[40867]: 1.9 scrub starts 2024-09-21T12:29:06.123 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:05 smithi114 ceph-mon[40867]: 1.9 scrub ok 2024-09-21T12:29:06.124 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:06.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:06.167 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:06.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:06.268 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:06.305 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:29:06.306 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:29:06.306 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:06.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:06.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:06.343 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:29:06.343 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:29:06.378 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:29:06.378 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:29:06.408 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:06.412 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:29:06.412 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:29:06.444 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:29:06.444 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:29:06.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:06.451 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:06.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:06.474 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:29:06.475 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:29:06.509 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:29:06.509 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:29:06.541 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:29:06.541 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:29:06.551 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:06.569 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:29:06.570 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-21T12:29:06.570 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 0 2024-09-21T12:29:06.570 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:29:06.589 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:06.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:06.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:06.690 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:06.719 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:06.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:06.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:06.821 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:06.828 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 39 16 70527 70511 112.986 86.25 0.00996347 0.00884583 2024-09-21T12:29:06.858 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:06.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:06.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:06.960 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:06.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:06 smithi183 ceph-mon[33277]: pgmap v434: 178 pgs: 178 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.0 KiB/s rd, 90 MiB/s wr, 1.44k op/s 2024-09-21T12:29:06.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:06 smithi183 ceph-mon[33277]: osdmap e257: 8 total, 8 up, 7 in 2024-09-21T12:29:06.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:06 smithi183 ceph-mon[33277]: 1.7 scrub starts 2024-09-21T12:29:06.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:06 smithi183 ceph-mon[33277]: 1.7 scrub ok 2024-09-21T12:29:06.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:06 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3903805581' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:29:06.993 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:06.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:06.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:07.095 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:07.128 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:07.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:07.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:07.230 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:07.251 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:29:07 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:29:07.253 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T12:29:07.274 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:07.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:06 smithi114 ceph-mon[40867]: pgmap v434: 178 pgs: 178 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.0 KiB/s rd, 90 MiB/s wr, 1.44k op/s 2024-09-21T12:29:07.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:06 smithi114 ceph-mon[40867]: osdmap e257: 8 total, 8 up, 7 in 2024-09-21T12:29:07.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:06 smithi114 ceph-mon[40867]: 1.7 scrub starts 2024-09-21T12:29:07.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:06 smithi114 ceph-mon[40867]: 1.7 scrub ok 2024-09-21T12:29:07.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:06 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3903805581' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:29:07.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:06 smithi114 ceph-mon[30324]: pgmap v434: 178 pgs: 178 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.0 KiB/s rd, 90 MiB/s wr, 1.44k op/s 2024-09-21T12:29:07.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:06 smithi114 ceph-mon[30324]: osdmap e257: 8 total, 8 up, 7 in 2024-09-21T12:29:07.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:06 smithi114 ceph-mon[30324]: 1.7 scrub starts 2024-09-21T12:29:07.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:06 smithi114 ceph-mon[30324]: 1.7 scrub ok 2024-09-21T12:29:07.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:06 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3903805581' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:29:07.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:07.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:07.377 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:07.415 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:07.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:07.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:07.517 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:07.550 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:07.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:07.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:07.651 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:07.675 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:29:07.687 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:07.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:07.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:07.789 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:07.817 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:07.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:07.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:07.828 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-21T12:29:07.828114+0000 min lat: 0.00136588 max lat: 0.130748 avg lat: 0.00890859 2024-09-21T12:29:07.828 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T12:29:07.828 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 40 15 71827 71812 112.194 81.3125 0.0140169 0.00890859 2024-09-21T12:29:07.921 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:07.960 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:07.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:07.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:08.061 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:08.095 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:07 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:29:08.097 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:08.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:08.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:08.199 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:08.237 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:08.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:08.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:08.259 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:29:08.259 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-21T12:29:08.259 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-21T12:29:08.259 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-21T12:29:08.259 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:29:08.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:07 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:29:08.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:07 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:29:08.339 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:08.376 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:08.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:08.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:08.478 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:08.512 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:08.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:08.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:08.614 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:08.652 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:08.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:08.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:08.753 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:08.791 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:08.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:08.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:08.828 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 41 15 73306 73291 111.712 92.4375 0.0125723 0.00894709 2024-09-21T12:29:08.893 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:08.926 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:08.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:08.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:08.945 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T12:29:09.028 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:09.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:09.070 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:09.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:09.170 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:09.208 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:09.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:09.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:09.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:08 smithi183 ceph-mon[33277]: 1.f scrub starts 2024-09-21T12:29:09.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:08 smithi183 ceph-mon[33277]: 1.f scrub ok 2024-09-21T12:29:09.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:08 smithi183 ceph-mon[33277]: pgmap v436: 178 pgs: 178 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.6 KiB/s rd, 103 MiB/s wr, 1.65k op/s; 1.5 KiB/s, 0 objects/s recovering 2024-09-21T12:29:09.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:08 smithi183 ceph-mon[33277]: 4.18 scrub starts 2024-09-21T12:29:09.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:08 smithi183 ceph-mon[33277]: 4.18 scrub ok 2024-09-21T12:29:09.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:08 smithi114 ceph-mon[40867]: 1.f scrub starts 2024-09-21T12:29:09.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:08 smithi114 ceph-mon[40867]: 1.f scrub ok 2024-09-21T12:29:09.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:08 smithi114 ceph-mon[40867]: pgmap v436: 178 pgs: 178 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.6 KiB/s rd, 103 MiB/s wr, 1.65k op/s; 1.5 KiB/s, 0 objects/s recovering 2024-09-21T12:29:09.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:08 smithi114 ceph-mon[40867]: 4.18 scrub starts 2024-09-21T12:29:09.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:08 smithi114 ceph-mon[40867]: 4.18 scrub ok 2024-09-21T12:29:09.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:08 smithi114 ceph-mon[30324]: 1.f scrub starts 2024-09-21T12:29:09.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:08 smithi114 ceph-mon[30324]: 1.f scrub ok 2024-09-21T12:29:09.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:08 smithi114 ceph-mon[30324]: pgmap v436: 178 pgs: 178 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.6 KiB/s rd, 103 MiB/s wr, 1.65k op/s; 1.5 KiB/s, 0 objects/s recovering 2024-09-21T12:29:09.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:08 smithi114 ceph-mon[30324]: 4.18 scrub starts 2024-09-21T12:29:09.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:08 smithi114 ceph-mon[30324]: 4.18 scrub ok 2024-09-21T12:29:09.310 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:09.343 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:09.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:09.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:09.361 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:29:09.444 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:09.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:09.483 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:09.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:09.584 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:09.816 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:09.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:09.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:09.828 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 42 16 74802 74786 111.277 93.4375 0.00609479 0.00898216 2024-09-21T12:29:09.918 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:09.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:09.956 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:09.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:10.057 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:10.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:10.093 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:10.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:10.097 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:09 smithi183 ceph-mon[33277]: 1.6 scrub starts 2024-09-21T12:29:10.097 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:09 smithi183 ceph-mon[33277]: 1.6 scrub ok 2024-09-21T12:29:10.097 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:09 smithi183 ceph-mon[33277]: 4.5 scrub starts 2024-09-21T12:29:10.097 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:09 smithi183 ceph-mon[33277]: 4.5 scrub ok 2024-09-21T12:29:10.098 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:29:10.098 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-21T12:29:10.098 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-21T12:29:10.098 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:29:10.098 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.098 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14478.0:1119 5.1f 5:f95f44c2:::notify.0:head [watch ping cookie 94620848892160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-21T12:29:10.098 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:40.364371+0000", 2024-09-21T12:29:10.098 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 449.72706612500002, 2024-09-21T12:29:10.098 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78371777300000001, 2024-09-21T12:29:10.098 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:10.098 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:10.098 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:29:10.098 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:10.099 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14478", 2024-09-21T12:29:10.099 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:29:10.099 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 1119 2024-09-21T12:29:10.099 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.099 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:10.099 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.099 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:10.099 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364371+0000", 2024-09-21T12:29:10.099 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.099 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.099 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.099 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:10.099 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364371+0000", 2024-09-21T12:29:10.099 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.100 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.100 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.100 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:10.100 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364372+0000", 2024-09-21T12:29:10.100 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6199999999999997e-07 2024-09-21T12:29:10.100 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.100 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.100 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:10.100 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364376+0000", 2024-09-21T12:29:10.100 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8099999999999999e-06 2024-09-21T12:29:10.100 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.100 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.100 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:10.100 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364384+0000", 2024-09-21T12:29:10.101 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2870000000000004e-06 2024-09-21T12:29:10.101 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.101 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.101 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:10.101 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364393+0000", 2024-09-21T12:29:10.101 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5820000000000002e-06 2024-09-21T12:29:10.101 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.101 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.101 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:10.101 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364557+0000", 2024-09-21T12:29:10.101 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016425399999999999 2024-09-21T12:29:10.101 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.101 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.101 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:29:10.101 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364573+0000", 2024-09-21T12:29:10.102 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6286999999999998e-05 2024-09-21T12:29:10.102 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.102 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.102 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:10.102 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.144878+0000", 2024-09-21T12:29:10.102 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78030514600000001 2024-09-21T12:29:10.102 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.102 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.102 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for degraded object", 2024-09-21T12:29:10.102 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.145417+0000", 2024-09-21T12:29:10.102 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00053866500000000002 2024-09-21T12:29:10.102 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.102 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.103 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:10.103 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.147869+0000", 2024-09-21T12:29:10.103 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024518779999999998 2024-09-21T12:29:10.103 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.103 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.103 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:10.103 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.147939+0000", 2024-09-21T12:29:10.103 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0574000000000006e-05 2024-09-21T12:29:10.103 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.103 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.103 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:10.103 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.148089+0000", 2024-09-21T12:29:10.103 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014952800000000001 2024-09-21T12:29:10.103 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.103 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:10.104 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.104 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.104 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.104 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14478.0:1120 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch ping cookie 94620852177152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-21T12:29:10.104 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:40.364419+0000", 2024-09-21T12:29:10.104 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 449.72701844900001, 2024-09-21T12:29:10.104 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78472206099999997, 2024-09-21T12:29:10.104 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:10.104 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:10.104 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:29:10.104 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:10.104 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14478", 2024-09-21T12:29:10.104 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:29:10.104 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 1120 2024-09-21T12:29:10.104 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.105 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:10.105 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.105 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:10.105 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364419+0000", 2024-09-21T12:29:10.105 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.105 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.105 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.105 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:10.105 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364419+0000", 2024-09-21T12:29:10.105 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.105 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.105 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.105 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:10.105 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364419+0000", 2024-09-21T12:29:10.106 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5400000000000002e-07 2024-09-21T12:29:10.106 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.106 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.106 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:10.106 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364426+0000", 2024-09-21T12:29:10.106 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8650000000000003e-06 2024-09-21T12:29:10.106 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.106 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.106 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:10.106 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364434+0000", 2024-09-21T12:29:10.106 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2309999999999996e-06 2024-09-21T12:29:10.106 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.106 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.106 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:10.107 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364441+0000", 2024-09-21T12:29:10.107 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0929999999999999e-06 2024-09-21T12:29:10.107 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.107 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.107 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:10.107 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364520+0000", 2024-09-21T12:29:10.107 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9487999999999995e-05 2024-09-21T12:29:10.107 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.107 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.107 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:29:10.107 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364543+0000", 2024-09-21T12:29:10.107 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.296e-05 2024-09-21T12:29:10.107 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.108 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.108 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:10.108 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.147074+0000", 2024-09-21T12:29:10.108 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78253067600000004 2024-09-21T12:29:10.108 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.108 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.108 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for degraded object", 2024-09-21T12:29:10.108 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.147381+0000", 2024-09-21T12:29:10.108 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00030698200000000002 2024-09-21T12:29:10.108 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.108 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.108 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:10.108 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.148992+0000", 2024-09-21T12:29:10.108 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001611267 2024-09-21T12:29:10.109 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.109 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.109 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:10.109 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.149064+0000", 2024-09-21T12:29:10.109 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2058999999999996e-05 2024-09-21T12:29:10.109 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.109 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.109 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:10.109 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.149141+0000", 2024-09-21T12:29:10.109 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6985999999999998e-05 2024-09-21T12:29:10.109 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.109 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:10.109 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.110 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.110 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.110 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14478.0:1124 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94620848892160 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:29:10.110 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:40.364832+0000", 2024-09-21T12:29:10.110 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 449.726605398, 2024-09-21T12:29:10.110 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78486121600000003, 2024-09-21T12:29:10.110 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:10.110 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:10.110 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:29:10.110 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:10.110 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14478", 2024-09-21T12:29:10.110 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:29:10.110 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 1124 2024-09-21T12:29:10.110 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.111 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:10.111 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.111 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:10.111 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364832+0000", 2024-09-21T12:29:10.111 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.111 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.111 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.111 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:10.111 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364832+0000", 2024-09-21T12:29:10.111 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.111 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.111 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.112 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:10.112 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364833+0000", 2024-09-21T12:29:10.112 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3900000000000004e-07 2024-09-21T12:29:10.112 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.112 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.112 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:10.112 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364838+0000", 2024-09-21T12:29:10.112 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1909999999999999e-06 2024-09-21T12:29:10.112 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.113 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.113 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:10.113 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364846+0000", 2024-09-21T12:29:10.113 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7000000000000008e-06 2024-09-21T12:29:10.113 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.113 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.113 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:10.113 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364850+0000", 2024-09-21T12:29:10.113 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.921e-06 2024-09-21T12:29:10.113 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.113 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.113 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:10.113 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364911+0000", 2024-09-21T12:29:10.114 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0636e-05 2024-09-21T12:29:10.114 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.114 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.114 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:29:10.114 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364922+0000", 2024-09-21T12:29:10.114 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0964e-05 2024-09-21T12:29:10.114 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.114 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.114 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:10.114 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.145447+0000", 2024-09-21T12:29:10.114 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78052487599999998 2024-09-21T12:29:10.114 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.114 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.115 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for degraded object", 2024-09-21T12:29:10.115 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.145493+0000", 2024-09-21T12:29:10.115 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5745000000000003e-05 2024-09-21T12:29:10.115 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.115 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.115 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:10.115 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.148124+0000", 2024-09-21T12:29:10.115 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0026310719999999999 2024-09-21T12:29:10.115 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.115 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.115 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:10.115 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.148191+0000", 2024-09-21T12:29:10.115 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7447999999999995e-05 2024-09-21T12:29:10.115 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.116 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.116 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:10.116 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.149693+0000", 2024-09-21T12:29:10.116 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001501924 2024-09-21T12:29:10.116 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.116 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:10.116 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.116 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.116 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.116 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14478.0:1125 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94620852177152 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:29:10.116 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:40.364865+0000", 2024-09-21T12:29:10.116 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 449.726571885, 2024-09-21T12:29:10.116 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78551680400000001, 2024-09-21T12:29:10.116 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:10.117 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:10.117 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:29:10.117 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:10.117 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14478", 2024-09-21T12:29:10.117 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:29:10.117 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 1125 2024-09-21T12:29:10.117 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.117 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:10.117 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.117 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:10.117 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364865+0000", 2024-09-21T12:29:10.117 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.117 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.117 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.117 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:10.117 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364865+0000", 2024-09-21T12:29:10.118 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.118 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.118 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.118 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:10.118 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364866+0000", 2024-09-21T12:29:10.118 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6499999999999999e-07 2024-09-21T12:29:10.118 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.118 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.118 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:10.118 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364868+0000", 2024-09-21T12:29:10.118 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.39e-06 2024-09-21T12:29:10.118 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.118 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.118 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:10.119 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364882+0000", 2024-09-21T12:29:10.119 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3890000000000001e-05 2024-09-21T12:29:10.119 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.119 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.119 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:10.119 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364888+0000", 2024-09-21T12:29:10.119 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3940000000000004e-06 2024-09-21T12:29:10.119 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.119 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.119 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:10.119 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364953+0000", 2024-09-21T12:29:10.119 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5877000000000006e-05 2024-09-21T12:29:10.119 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.119 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.119 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:29:10.120 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364963+0000", 2024-09-21T12:29:10.120 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5349999999999993e-06 2024-09-21T12:29:10.120 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.120 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.120 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:10.120 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.147420+0000", 2024-09-21T12:29:10.120 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78245673400000004 2024-09-21T12:29:10.120 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.120 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.120 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for degraded object", 2024-09-21T12:29:10.120 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.147469+0000", 2024-09-21T12:29:10.120 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9699e-05 2024-09-21T12:29:10.120 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.120 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.120 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.149190+0000", 2024-09-21T12:29:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001720606 2024-09-21T12:29:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.149267+0000", 2024-09-21T12:29:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7194000000000007e-05 2024-09-21T12:29:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.150382+0000", 2024-09-21T12:29:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00111502 2024-09-21T12:29:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:10.121 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.122 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.122 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.122 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:24811 7.7 7:ec8774ff:::benchmark_data_smithi114_78432_object24810:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T12:29:10.122 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:42.445492+0000", 2024-09-21T12:29:10.122 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 447.64594495400002, 2024-09-21T12:29:10.122 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.145669727, 2024-09-21T12:29:10.122 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:10.122 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:10.122 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:10.122 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:10.122 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:29:10.122 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:29:10.122 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 24811 2024-09-21T12:29:10.122 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.122 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:10.123 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.123 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:10.123 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445492+0000", 2024-09-21T12:29:10.123 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.123 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.123 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.123 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:10.123 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445492+0000", 2024-09-21T12:29:10.123 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.123 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.123 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.123 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:10.123 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445494+0000", 2024-09-21T12:29:10.123 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5460000000000001e-06 2024-09-21T12:29:10.124 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.124 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.124 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:10.124 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445541+0000", 2024-09-21T12:29:10.124 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6687999999999998e-05 2024-09-21T12:29:10.124 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.124 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.124 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:10.124 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445553+0000", 2024-09-21T12:29:10.124 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2517999999999999e-05 2024-09-21T12:29:10.124 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.124 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.124 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:10.124 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445560+0000", 2024-09-21T12:29:10.124 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6579999999999996e-06 2024-09-21T12:29:10.124 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.125 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.125 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:10.125 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445625+0000", 2024-09-21T12:29:10.125 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5092000000000006e-05 2024-09-21T12:29:10.125 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.125 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.125 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:10.125 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445776+0000", 2024-09-21T12:29:10.125 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0001507 2024-09-21T12:29:10.125 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.125 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.125 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,3", 2024-09-21T12:29:10.125 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445923+0000", 2024-09-21T12:29:10.125 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014760199999999999 2024-09-21T12:29:10.125 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.126 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.126 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:10.126 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.446851+0000", 2024-09-21T12:29:10.126 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00092788099999999997 2024-09-21T12:29:10.126 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.126 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.126 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.126 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.590740+0000", 2024-09-21T12:29:10.126 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14388880500000001 2024-09-21T12:29:10.126 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.126 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.126 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.126 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.591050+0000", 2024-09-21T12:29:10.126 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000310217 2024-09-21T12:29:10.127 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.127 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.127 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:10.127 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.591109+0000", 2024-09-21T12:29:10.127 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9188000000000003e-05 2024-09-21T12:29:10.127 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.127 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.127 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:10.127 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.591162+0000", 2024-09-21T12:29:10.127 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2831999999999998e-05 2024-09-21T12:29:10.127 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.127 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:10.127 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.128 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.128 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.128 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:45498 7.7 7:e1336a67:::benchmark_data_smithi114_78432_object45497:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T12:29:10.128 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:59.293850+0000", 2024-09-21T12:29:10.128 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 430.79758703099998, 2024-09-21T12:29:10.128 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15681044, 2024-09-21T12:29:10.128 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:10.128 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:10.128 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:10.128 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:10.128 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:29:10.128 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:29:10.128 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 45498 2024-09-21T12:29:10.128 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.129 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:10.129 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.129 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:10.129 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293850+0000", 2024-09-21T12:29:10.129 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.129 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.129 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.129 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:10.129 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293850+0000", 2024-09-21T12:29:10.129 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.129 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.129 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.129 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:10.129 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293851+0000", 2024-09-21T12:29:10.130 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.096e-06 2024-09-21T12:29:10.130 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.130 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.130 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:10.130 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293907+0000", 2024-09-21T12:29:10.130 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5126e-05 2024-09-21T12:29:10.130 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.130 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.130 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:10.130 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293915+0000", 2024-09-21T12:29:10.130 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8419999999999994e-06 2024-09-21T12:29:10.130 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.130 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.130 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:10.131 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293920+0000", 2024-09-21T12:29:10.131 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5900000000000001e-06 2024-09-21T12:29:10.131 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.131 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.131 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:10.131 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293959+0000", 2024-09-21T12:29:10.131 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9447999999999999e-05 2024-09-21T12:29:10.131 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.131 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.131 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:10.131 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294060+0000", 2024-09-21T12:29:10.131 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100247 2024-09-21T12:29:10.131 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.131 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.132 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,3", 2024-09-21T12:29:10.132 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294145+0000", 2024-09-21T12:29:10.132 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5099999999999995e-05 2024-09-21T12:29:10.132 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.132 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.132 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:10.132 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.295566+0000", 2024-09-21T12:29:10.132 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014215429999999999 2024-09-21T12:29:10.132 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.132 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.132 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.132 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.449406+0000", 2024-09-21T12:29:10.132 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15383943999999999 2024-09-21T12:29:10.133 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.133 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.133 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.133 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.450563+0000", 2024-09-21T12:29:10.133 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00115749 2024-09-21T12:29:10.133 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.133 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.133 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:10.133 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.450621+0000", 2024-09-21T12:29:10.133 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8161000000000001e-05 2024-09-21T12:29:10.133 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.133 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.133 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:10.133 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.450661+0000", 2024-09-21T12:29:10.134 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9357000000000003e-05 2024-09-21T12:29:10.134 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.134 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:10.134 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.134 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.134 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.134 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:45514 7.c 7:3b027556:::benchmark_data_smithi114_78432_object45513:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T12:29:10.134 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:59.302539+0000", 2024-09-21T12:29:10.134 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 430.78889813799998, 2024-09-21T12:29:10.134 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.150159925, 2024-09-21T12:29:10.134 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:10.134 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:10.134 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:10.134 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:10.135 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:29:10.135 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:29:10.135 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 45514 2024-09-21T12:29:10.135 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.135 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:10.135 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.135 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:10.135 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302539+0000", 2024-09-21T12:29:10.135 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.135 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.135 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.135 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:10.135 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302539+0000", 2024-09-21T12:29:10.135 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.135 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.136 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.136 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:10.136 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302541+0000", 2024-09-21T12:29:10.136 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.376e-06 2024-09-21T12:29:10.136 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.136 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.136 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:10.136 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302592+0000", 2024-09-21T12:29:10.136 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1054999999999998e-05 2024-09-21T12:29:10.136 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.136 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.136 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:10.136 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302603+0000", 2024-09-21T12:29:10.136 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0883000000000001e-05 2024-09-21T12:29:10.137 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.137 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.137 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:10.137 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302608+0000", 2024-09-21T12:29:10.137 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5339999999999999e-06 2024-09-21T12:29:10.137 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.137 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.137 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:10.137 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302655+0000", 2024-09-21T12:29:10.137 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7125e-05 2024-09-21T12:29:10.137 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.137 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.137 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:10.137 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302784+0000", 2024-09-21T12:29:10.137 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012886700000000001 2024-09-21T12:29:10.137 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.138 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.138 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:29:10.138 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.303003+0000", 2024-09-21T12:29:10.138 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00021902099999999999 2024-09-21T12:29:10.138 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.138 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.138 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:10.138 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.304304+0000", 2024-09-21T12:29:10.138 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0013008060000000001 2024-09-21T12:29:10.138 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.138 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.138 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.138 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.304689+0000", 2024-09-21T12:29:10.138 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000385208 2024-09-21T12:29:10.139 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.139 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.139 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.139 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.452633+0000", 2024-09-21T12:29:10.139 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14794360000000001 2024-09-21T12:29:10.139 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.139 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.139 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:10.139 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.452669+0000", 2024-09-21T12:29:10.139 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6202000000000003e-05 2024-09-21T12:29:10.139 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.139 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.139 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:10.139 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.452699+0000", 2024-09-21T12:29:10.139 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0247999999999999e-05 2024-09-21T12:29:10.140 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.140 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:10.140 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.140 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.140 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.140 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:49226 7.d 7:bcebfbbf:::benchmark_data_smithi114_78432_object49225:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:29:10.140 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:04.569255+0000", 2024-09-21T12:29:10.140 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 425.52218215099998, 2024-09-21T12:29:10.140 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14786647, 2024-09-21T12:29:10.140 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:10.140 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:10.140 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:10.140 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:10.140 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:29:10.140 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:29:10.141 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 49226 2024-09-21T12:29:10.141 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.141 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:10.141 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.141 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:10.141 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569255+0000", 2024-09-21T12:29:10.141 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.141 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.141 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.141 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:10.141 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569255+0000", 2024-09-21T12:29:10.141 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.141 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.141 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.142 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:10.142 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569257+0000", 2024-09-21T12:29:10.142 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3480000000000001e-06 2024-09-21T12:29:10.142 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.142 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.142 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:10.142 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569332+0000", 2024-09-21T12:29:10.142 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4969000000000007e-05 2024-09-21T12:29:10.142 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.142 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.142 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:10.142 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569344+0000", 2024-09-21T12:29:10.142 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2700999999999999e-05 2024-09-21T12:29:10.142 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.143 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.143 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:10.143 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569351+0000", 2024-09-21T12:29:10.143 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0369999999999997e-06 2024-09-21T12:29:10.143 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.143 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.143 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:10.143 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569427+0000", 2024-09-21T12:29:10.143 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6206999999999997e-05 2024-09-21T12:29:10.143 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.143 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.143 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:10.143 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569517+0000", 2024-09-21T12:29:10.143 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9875000000000003e-05 2024-09-21T12:29:10.143 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.144 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.144 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1", 2024-09-21T12:29:10.144 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569601+0000", 2024-09-21T12:29:10.144 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3849000000000003e-05 2024-09-21T12:29:10.144 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.144 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.144 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:10.144 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.573956+0000", 2024-09-21T12:29:10.144 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0043544689999999997 2024-09-21T12:29:10.144 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.144 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.144 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.144 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.575786+0000", 2024-09-21T12:29:10.144 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0018307480000000001 2024-09-21T12:29:10.144 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.145 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.145 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.145 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.717016+0000", 2024-09-21T12:29:10.145 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14122955000000001 2024-09-21T12:29:10.145 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.145 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.145 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:10.145 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.717072+0000", 2024-09-21T12:29:10.145 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6063000000000002e-05 2024-09-21T12:29:10.145 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.145 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.145 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:10.145 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.717122+0000", 2024-09-21T12:29:10.145 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9654000000000003e-05 2024-09-21T12:29:10.145 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.146 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:10.146 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.146 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.146 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.146 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:49229 7.d 7:b095dac2:::benchmark_data_smithi114_78432_object49228:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:29:10.146 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:04.569388+0000", 2024-09-21T12:29:10.146 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 425.52204961699999, 2024-09-21T12:29:10.146 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.147855664, 2024-09-21T12:29:10.146 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:10.146 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:10.146 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:10.146 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:10.146 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:29:10.146 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:29:10.147 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 49229 2024-09-21T12:29:10.147 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.147 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:10.147 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.147 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:10.147 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569388+0000", 2024-09-21T12:29:10.147 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.147 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.147 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.147 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:10.147 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569388+0000", 2024-09-21T12:29:10.147 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.147 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.147 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.148 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:10.148 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569388+0000", 2024-09-21T12:29:10.148 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.37e-07 2024-09-21T12:29:10.148 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.148 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.148 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:10.148 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569462+0000", 2024-09-21T12:29:10.148 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3244000000000005e-05 2024-09-21T12:29:10.148 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.148 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.148 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:10.148 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569471+0000", 2024-09-21T12:29:10.148 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2499999999999995e-06 2024-09-21T12:29:10.148 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.148 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.149 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:10.149 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569477+0000", 2024-09-21T12:29:10.149 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.494e-06 2024-09-21T12:29:10.149 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.149 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.149 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:10.149 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.572570+0000", 2024-09-21T12:29:10.149 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0030923930000000001 2024-09-21T12:29:10.149 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.149 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.149 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:10.149 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.572664+0000", 2024-09-21T12:29:10.149 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4609000000000002e-05 2024-09-21T12:29:10.149 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.149 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.150 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1", 2024-09-21T12:29:10.150 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.572751+0000", 2024-09-21T12:29:10.150 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6725999999999996e-05 2024-09-21T12:29:10.150 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.150 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.150 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:10.150 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.574094+0000", 2024-09-21T12:29:10.150 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00134326 2024-09-21T12:29:10.150 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.150 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.150 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.150 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.576231+0000", 2024-09-21T12:29:10.150 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021369510000000002 2024-09-21T12:29:10.150 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.151 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.151 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.151 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.717166+0000", 2024-09-21T12:29:10.151 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14093446000000001 2024-09-21T12:29:10.151 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.151 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.151 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:10.151 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.717210+0000", 2024-09-21T12:29:10.151 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4239000000000001e-05 2024-09-21T12:29:10.151 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.152 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.152 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:10.152 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.717243+0000", 2024-09-21T12:29:10.152 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3401000000000001e-05 2024-09-21T12:29:10.152 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.152 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:10.152 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.152 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.152 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.152 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:49936 7.7 7:ee2fb7d7:::benchmark_data_smithi114_78432_object49935:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:29:10.153 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:05.207697+0000", 2024-09-21T12:29:10.153 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 424.883740746, 2024-09-21T12:29:10.153 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.180234797, 2024-09-21T12:29:10.153 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:10.153 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:10.153 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:10.153 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:10.153 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:29:10.153 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:29:10.153 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 49936 2024-09-21T12:29:10.153 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.153 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:10.153 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.153 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:10.154 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207697+0000", 2024-09-21T12:29:10.154 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.154 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.154 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.154 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:10.154 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207697+0000", 2024-09-21T12:29:10.154 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.154 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.154 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.154 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:10.154 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207698+0000", 2024-09-21T12:29:10.154 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6449999999999999e-06 2024-09-21T12:29:10.154 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.154 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.155 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:10.155 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207744+0000", 2024-09-21T12:29:10.155 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6171999999999998e-05 2024-09-21T12:29:10.155 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.155 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.155 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:10.155 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207755+0000", 2024-09-21T12:29:10.155 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0998999999999999e-05 2024-09-21T12:29:10.155 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.155 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.155 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:10.155 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207761+0000", 2024-09-21T12:29:10.155 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9789999999999998e-06 2024-09-21T12:29:10.155 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.155 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.156 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:10.156 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207822+0000", 2024-09-21T12:29:10.156 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0504000000000002e-05 2024-09-21T12:29:10.156 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.156 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.156 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:10.156 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207921+0000", 2024-09-21T12:29:10.156 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9524000000000005e-05 2024-09-21T12:29:10.156 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.156 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.156 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,3", 2024-09-21T12:29:10.156 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.208068+0000", 2024-09-21T12:29:10.156 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014648299999999999 2024-09-21T12:29:10.156 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.156 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.156 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:10.157 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.209637+0000", 2024-09-21T12:29:10.157 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001569555 2024-09-21T12:29:10.157 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.157 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.157 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.157 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.214355+0000", 2024-09-21T12:29:10.157 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0047178810000000002 2024-09-21T12:29:10.157 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.157 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.157 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.157 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.387868+0000", 2024-09-21T12:29:10.157 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.173512203 2024-09-21T12:29:10.157 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.157 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.157 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:10.158 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.387905+0000", 2024-09-21T12:29:10.158 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7678000000000001e-05 2024-09-21T12:29:10.158 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.158 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.158 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:10.158 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.387931+0000", 2024-09-21T12:29:10.158 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6174e-05 2024-09-21T12:29:10.158 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.158 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:10.158 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.158 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.158 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.158 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:49949 7.d 7:b8e0b394:::benchmark_data_smithi114_78432_object49948:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:29:10.158 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:05.215908+0000", 2024-09-21T12:29:10.159 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 424.87552902900001, 2024-09-21T12:29:10.159 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.17212050400000001, 2024-09-21T12:29:10.159 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:10.159 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:10.159 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:10.159 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:10.159 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:29:10.159 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:29:10.159 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 49949 2024-09-21T12:29:10.159 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.159 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:10.159 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.159 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:10.159 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215908+0000", 2024-09-21T12:29:10.159 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.160 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.160 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.160 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:10.160 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215908+0000", 2024-09-21T12:29:10.160 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.160 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.160 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.160 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:10.160 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215910+0000", 2024-09-21T12:29:10.160 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.389e-06 2024-09-21T12:29:10.160 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.160 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.160 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:10.160 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215966+0000", 2024-09-21T12:29:10.160 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6189e-05 2024-09-21T12:29:10.161 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.161 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.161 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:10.161 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215976+0000", 2024-09-21T12:29:10.161 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0115999999999999e-05 2024-09-21T12:29:10.161 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.161 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.161 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:10.161 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215983+0000", 2024-09-21T12:29:10.161 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9240000000000001e-06 2024-09-21T12:29:10.161 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.161 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.161 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:10.161 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.216032+0000", 2024-09-21T12:29:10.161 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9237000000000003e-05 2024-09-21T12:29:10.162 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.162 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.162 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:10.162 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.216109+0000", 2024-09-21T12:29:10.162 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6538999999999993e-05 2024-09-21T12:29:10.162 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.162 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.162 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1", 2024-09-21T12:29:10.162 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.216208+0000", 2024-09-21T12:29:10.162 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8979000000000002e-05 2024-09-21T12:29:10.162 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.162 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.162 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:10.162 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217649+0000", 2024-09-21T12:29:10.162 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014417500000000001 2024-09-21T12:29:10.163 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.163 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.163 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.163 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.219275+0000", 2024-09-21T12:29:10.163 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0016254749999999999 2024-09-21T12:29:10.163 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.163 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.163 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.163 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.387917+0000", 2024-09-21T12:29:10.163 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.16864237100000001 2024-09-21T12:29:10.163 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.163 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.163 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:10.163 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.387990+0000", 2024-09-21T12:29:10.163 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2249999999999994e-05 2024-09-21T12:29:10.164 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.164 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.164 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:10.164 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.388029+0000", 2024-09-21T12:29:10.164 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9285000000000001e-05 2024-09-21T12:29:10.164 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.164 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:10.164 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.164 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.164 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.164 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:49953 7.7 7:e1a3875f:::benchmark_data_smithi114_78432_object49952:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:29:10.164 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:05.217615+0000", 2024-09-21T12:29:10.164 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 424.87382210800001, 2024-09-21T12:29:10.164 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.17616359000000001, 2024-09-21T12:29:10.164 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:10.165 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:10.165 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:10.165 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:10.165 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:29:10.165 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:29:10.165 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 49953 2024-09-21T12:29:10.165 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.165 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:10.165 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.165 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:10.165 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217615+0000", 2024-09-21T12:29:10.165 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.165 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.165 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.166 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:10.166 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217615+0000", 2024-09-21T12:29:10.166 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.166 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.166 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.166 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:10.166 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217616+0000", 2024-09-21T12:29:10.166 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.209e-06 2024-09-21T12:29:10.166 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.166 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.166 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:10.166 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217687+0000", 2024-09-21T12:29:10.166 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0570999999999999e-05 2024-09-21T12:29:10.166 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.166 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.167 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:10.167 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217701+0000", 2024-09-21T12:29:10.167 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3828e-05 2024-09-21T12:29:10.167 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.167 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.167 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:10.167 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217709+0000", 2024-09-21T12:29:10.167 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3240000000000003e-06 2024-09-21T12:29:10.167 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.167 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.167 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:10.167 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217763+0000", 2024-09-21T12:29:10.167 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3804999999999997e-05 2024-09-21T12:29:10.167 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.168 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.168 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:10.168 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217854+0000", 2024-09-21T12:29:10.168 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0771000000000002e-05 2024-09-21T12:29:10.168 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.168 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.168 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,3", 2024-09-21T12:29:10.168 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217935+0000", 2024-09-21T12:29:10.168 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0951000000000005e-05 2024-09-21T12:29:10.168 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.168 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.168 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:10.168 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.219237+0000", 2024-09-21T12:29:10.168 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0013019679999999999 2024-09-21T12:29:10.168 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.168 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.169 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.169 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.220275+0000", 2024-09-21T12:29:10.169 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010378609999999999 2024-09-21T12:29:10.169 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.169 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.169 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.169 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.393564+0000", 2024-09-21T12:29:10.169 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.17328943899999999 2024-09-21T12:29:10.169 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.169 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.169 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:10.169 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.393734+0000", 2024-09-21T12:29:10.169 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017039999999999999 2024-09-21T12:29:10.169 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.169 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.170 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:10.170 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.393779+0000", 2024-09-21T12:29:10.170 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4462999999999998e-05 2024-09-21T12:29:10.170 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.170 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:10.170 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.170 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.170 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.170 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:49961 7.7 7:e82f0c80:::benchmark_data_smithi114_78432_object49960:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:29:10.170 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:05.226839+0000", 2024-09-21T12:29:10.170 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 424.86459811200001, 2024-09-21T12:29:10.170 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.170311145, 2024-09-21T12:29:10.170 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:10.170 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:10.171 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:10.171 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:10.171 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:29:10.171 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:29:10.171 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 49961 2024-09-21T12:29:10.171 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.171 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:10.171 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.171 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:10.171 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.226839+0000", 2024-09-21T12:29:10.171 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.171 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.171 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.171 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:10.172 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.226839+0000", 2024-09-21T12:29:10.172 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.172 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.172 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.172 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:10.172 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.226840+0000", 2024-09-21T12:29:10.172 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8400000000000003e-07 2024-09-21T12:29:10.172 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.172 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.172 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:10.172 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.226896+0000", 2024-09-21T12:29:10.172 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5517999999999999e-05 2024-09-21T12:29:10.172 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.172 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.172 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:10.173 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.226906+0000", 2024-09-21T12:29:10.173 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0589e-05 2024-09-21T12:29:10.173 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.173 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.173 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:10.173 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.226914+0000", 2024-09-21T12:29:10.173 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2779999999999996e-06 2024-09-21T12:29:10.173 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.173 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.173 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:10.173 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.226963+0000", 2024-09-21T12:29:10.173 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9021999999999999e-05 2024-09-21T12:29:10.173 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.173 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.173 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:10.174 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.227045+0000", 2024-09-21T12:29:10.174 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2800999999999995e-05 2024-09-21T12:29:10.174 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.174 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.174 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,3", 2024-09-21T12:29:10.174 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.227126+0000", 2024-09-21T12:29:10.174 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0467000000000001e-05 2024-09-21T12:29:10.174 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.174 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.174 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:10.174 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.228347+0000", 2024-09-21T12:29:10.174 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001220907 2024-09-21T12:29:10.174 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.174 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.175 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.175 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.393832+0000", 2024-09-21T12:29:10.175 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.165485038 2024-09-21T12:29:10.175 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.175 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.175 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.175 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.397086+0000", 2024-09-21T12:29:10.175 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0032546799999999998 2024-09-21T12:29:10.175 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.175 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.175 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:10.175 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.397121+0000", 2024-09-21T12:29:10.175 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4584e-05 2024-09-21T12:29:10.175 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.176 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.176 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:10.176 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.397150+0000", 2024-09-21T12:29:10.176 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9377e-05 2024-09-21T12:29:10.176 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.176 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:10.176 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.176 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.176 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.176 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:116822 7.c 7:3b47e54f:::benchmark_data_smithi114_78432_object12676:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T12:29:10.176 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:23:12.373717+0000", 2024-09-21T12:29:10.176 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 357.717720439, 2024-09-21T12:29:10.176 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15758128199999999, 2024-09-21T12:29:10.176 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:10.176 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:10.177 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:10.177 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:10.177 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:29:10.177 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:29:10.177 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 116822 2024-09-21T12:29:10.177 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.177 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:10.177 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.177 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:10.177 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.373717+0000", 2024-09-21T12:29:10.177 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.177 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.177 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.177 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:10.178 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.373717+0000", 2024-09-21T12:29:10.178 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.178 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.178 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.178 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:10.178 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.373718+0000", 2024-09-21T12:29:10.178 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2190000000000001e-06 2024-09-21T12:29:10.178 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.178 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.178 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:10.178 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.373723+0000", 2024-09-21T12:29:10.178 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7400000000000004e-06 2024-09-21T12:29:10.178 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.178 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.179 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:10.179 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.373732+0000", 2024-09-21T12:29:10.179 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8319999999999995e-06 2024-09-21T12:29:10.179 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.179 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.179 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:10.179 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.373737+0000", 2024-09-21T12:29:10.179 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2789999999999998e-06 2024-09-21T12:29:10.179 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.179 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.179 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:10.179 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.373777+0000", 2024-09-21T12:29:10.179 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0123999999999997e-05 2024-09-21T12:29:10.179 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.179 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.180 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:10.180 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.373854+0000", 2024-09-21T12:29:10.180 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6743999999999996e-05 2024-09-21T12:29:10.180 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.180 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.180 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:29:10.180 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.373916+0000", 2024-09-21T12:29:10.180 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2451000000000002e-05 2024-09-21T12:29:10.180 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.180 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.180 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:10.180 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.375743+0000", 2024-09-21T12:29:10.180 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001826338 2024-09-21T12:29:10.180 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.180 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.181 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.181 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.375885+0000", 2024-09-21T12:29:10.181 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014278699999999999 2024-09-21T12:29:10.181 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.181 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.181 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.181 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.531196+0000", 2024-09-21T12:29:10.181 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.155311 2024-09-21T12:29:10.181 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.181 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.181 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:10.181 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.531247+0000", 2024-09-21T12:29:10.181 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0476000000000003e-05 2024-09-21T12:29:10.181 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.181 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.182 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:10.182 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.531298+0000", 2024-09-21T12:29:10.182 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1292000000000003e-05 2024-09-21T12:29:10.182 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.182 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:10.182 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.182 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.182 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.182 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:116828 7.c 7:31e7a7e6:::benchmark_data_smithi114_78432_object12682:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T12:29:10.182 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:23:12.377844+0000", 2024-09-21T12:29:10.182 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 357.71359353999998, 2024-09-21T12:29:10.182 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15356805100000001, 2024-09-21T12:29:10.182 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:10.182 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:10.183 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:10.183 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:10.183 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:29:10.183 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:29:10.183 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 116828 2024-09-21T12:29:10.183 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.183 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:10.183 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.183 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:10.183 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.377844+0000", 2024-09-21T12:29:10.183 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.183 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.183 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.183 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:10.183 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.377844+0000", 2024-09-21T12:29:10.184 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.184 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.184 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.184 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:10.184 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.377845+0000", 2024-09-21T12:29:10.184 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3600000000000002e-07 2024-09-21T12:29:10.184 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.184 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.184 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:10.184 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.377849+0000", 2024-09-21T12:29:10.184 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.622e-06 2024-09-21T12:29:10.184 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.184 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.184 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:10.184 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.377862+0000", 2024-09-21T12:29:10.184 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2989999999999999e-05 2024-09-21T12:29:10.185 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.185 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.185 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:10.185 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.377870+0000", 2024-09-21T12:29:10.185 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5780000000000001e-06 2024-09-21T12:29:10.185 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.185 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.185 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:10.185 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.377917+0000", 2024-09-21T12:29:10.185 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7178999999999997e-05 2024-09-21T12:29:10.185 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.185 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.185 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:10.185 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.377988+0000", 2024-09-21T12:29:10.186 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0890999999999996e-05 2024-09-21T12:29:10.186 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.186 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.186 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:29:10.186 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.378052+0000", 2024-09-21T12:29:10.186 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4194000000000002e-05 2024-09-21T12:29:10.186 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.186 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.186 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:10.186 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.379417+0000", 2024-09-21T12:29:10.186 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001365184 2024-09-21T12:29:10.186 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.186 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.186 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.186 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.379801+0000", 2024-09-21T12:29:10.187 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000383483 2024-09-21T12:29:10.187 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.187 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.187 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.187 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.531334+0000", 2024-09-21T12:29:10.187 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15153333499999999 2024-09-21T12:29:10.187 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.187 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.187 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:10.187 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.531378+0000", 2024-09-21T12:29:10.187 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3557999999999999e-05 2024-09-21T12:29:10.187 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.187 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.188 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:10.188 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.531412+0000", 2024-09-21T12:29:10.188 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4100999999999998e-05 2024-09-21T12:29:10.188 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.188 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:10.188 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.188 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.188 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.188 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:150329 7.7 7:ebd08636:::benchmark_data_smithi114_78432_object46183:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T12:29:10.188 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:23:42.610086+0000", 2024-09-21T12:29:10.188 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 327.481351287, 2024-09-21T12:29:10.188 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13913827400000001, 2024-09-21T12:29:10.188 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:10.188 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:10.188 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:10.189 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:10.189 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:29:10.189 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:29:10.189 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 150329 2024-09-21T12:29:10.189 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.189 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:10.189 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.189 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:10.189 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.610086+0000", 2024-09-21T12:29:10.189 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.189 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.189 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.189 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:10.189 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.610086+0000", 2024-09-21T12:29:10.189 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.190 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.190 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.190 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:10.190 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.610087+0000", 2024-09-21T12:29:10.190 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9699999999999994e-07 2024-09-21T12:29:10.190 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.190 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.190 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:10.190 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.610091+0000", 2024-09-21T12:29:10.190 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8139999999999998e-06 2024-09-21T12:29:10.190 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.190 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.190 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:10.190 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.610100+0000", 2024-09-21T12:29:10.191 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T12:29:10.192 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6239999999999997e-06 2024-09-21T12:29:10.192 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.192 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.192 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:10.192 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.610106+0000", 2024-09-21T12:29:10.192 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6350000000000003e-06 2024-09-21T12:29:10.192 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.192 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.192 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:10.192 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.610146+0000", 2024-09-21T12:29:10.192 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0215e-05 2024-09-21T12:29:10.193 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.193 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.193 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:10.193 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.610231+0000", 2024-09-21T12:29:10.193 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4846999999999996e-05 2024-09-21T12:29:10.193 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.193 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.193 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,3", 2024-09-21T12:29:10.193 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.610303+0000", 2024-09-21T12:29:10.193 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1823999999999995e-05 2024-09-21T12:29:10.193 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.193 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.193 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:10.193 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.612121+0000", 2024-09-21T12:29:10.194 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00181836 2024-09-21T12:29:10.194 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.194 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.194 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.194 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.615984+0000", 2024-09-21T12:29:10.194 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0038623630000000002 2024-09-21T12:29:10.194 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.194 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.194 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.194 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.749109+0000", 2024-09-21T12:29:10.194 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.133125308 2024-09-21T12:29:10.194 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.194 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.194 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:10.195 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.749172+0000", 2024-09-21T12:29:10.195 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2738999999999997e-05 2024-09-21T12:29:10.195 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.195 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.195 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:10.195 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.749224+0000", 2024-09-21T12:29:10.195 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2547999999999999e-05 2024-09-21T12:29:10.195 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.195 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:10.195 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.195 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.195 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.195 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:32454 8.5 8:afb3ecb0:::benchmark_data_smithi114_104936_object32453:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T12:29:10.195 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:07.550337+0000", 2024-09-21T12:29:10.195 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 242.541100395, 2024-09-21T12:29:10.196 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15084788399999999, 2024-09-21T12:29:10.196 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:10.196 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:10.196 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:10.196 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:10.196 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:29:10.196 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:29:10.196 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 32454 2024-09-21T12:29:10.196 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.196 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:10.196 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.196 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:10.196 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.550337+0000", 2024-09-21T12:29:10.196 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.197 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.197 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.197 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:10.197 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.550337+0000", 2024-09-21T12:29:10.197 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.197 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.197 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.197 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:10.197 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.550338+0000", 2024-09-21T12:29:10.197 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.525e-06 2024-09-21T12:29:10.197 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.197 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.197 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:10.197 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.550385+0000", 2024-09-21T12:29:10.197 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6434999999999998e-05 2024-09-21T12:29:10.198 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.198 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.198 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:10.198 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.550398+0000", 2024-09-21T12:29:10.198 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3502999999999999e-05 2024-09-21T12:29:10.198 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.198 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.198 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:10.198 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.550406+0000", 2024-09-21T12:29:10.198 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.187e-06 2024-09-21T12:29:10.198 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.198 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.198 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:10.198 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.550466+0000", 2024-09-21T12:29:10.198 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0708000000000002e-05 2024-09-21T12:29:10.199 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.199 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.199 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:10.199 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.550611+0000", 2024-09-21T12:29:10.199 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000144864 2024-09-21T12:29:10.199 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.199 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.199 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,7", 2024-09-21T12:29:10.199 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.550704+0000", 2024-09-21T12:29:10.199 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2888999999999998e-05 2024-09-21T12:29:10.199 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.199 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.199 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:10.199 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.552093+0000", 2024-09-21T12:29:10.199 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0013884590000000001 2024-09-21T12:29:10.200 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.200 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.200 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.200 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.553102+0000", 2024-09-21T12:29:10.200 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010093299999999999 2024-09-21T12:29:10.200 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.200 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.200 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.200 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.701083+0000", 2024-09-21T12:29:10.200 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14798086099999999 2024-09-21T12:29:10.200 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.200 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.200 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:10.200 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.701142+0000", 2024-09-21T12:29:10.200 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8788e-05 2024-09-21T12:29:10.201 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.201 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.201 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:10.201 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.701185+0000", 2024-09-21T12:29:10.201 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3334999999999998e-05 2024-09-21T12:29:10.201 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.201 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:10.201 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.201 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.201 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.201 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:38141 8.9 8:961cc3e2:::benchmark_data_smithi114_104936_object38140:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T12:29:10.201 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:11.301060+0000", 2024-09-21T12:29:10.201 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 238.790377562, 2024-09-21T12:29:10.201 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14064707500000001, 2024-09-21T12:29:10.201 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:10.202 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:10.202 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:10.202 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:10.202 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:29:10.202 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:29:10.202 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 38141 2024-09-21T12:29:10.202 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.202 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:10.202 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.202 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:10.202 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.301060+0000", 2024-09-21T12:29:10.202 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.202 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.202 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.203 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:10.203 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.301060+0000", 2024-09-21T12:29:10.203 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.203 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.203 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.203 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:10.203 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.301061+0000", 2024-09-21T12:29:10.203 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3209999999999999e-06 2024-09-21T12:29:10.203 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.203 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.203 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:10.203 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.301107+0000", 2024-09-21T12:29:10.203 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5577000000000001e-05 2024-09-21T12:29:10.203 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.203 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.204 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:10.204 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.301121+0000", 2024-09-21T12:29:10.204 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4518999999999999e-05 2024-09-21T12:29:10.204 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.204 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.204 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:10.204 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.301129+0000", 2024-09-21T12:29:10.204 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6739999999999997e-06 2024-09-21T12:29:10.204 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.204 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.204 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:10.204 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.301183+0000", 2024-09-21T12:29:10.204 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4363000000000001e-05 2024-09-21T12:29:10.204 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.204 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.205 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:10.205 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.301317+0000", 2024-09-21T12:29:10.205 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013395099999999999 2024-09-21T12:29:10.205 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.205 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.205 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-21T12:29:10.205 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.301459+0000", 2024-09-21T12:29:10.205 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014159599999999999 2024-09-21T12:29:10.205 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.205 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.205 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.205 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.304358+0000", 2024-09-21T12:29:10.205 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0028990130000000002 2024-09-21T12:29:10.205 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.205 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.206 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:10.206 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.304497+0000", 2024-09-21T12:29:10.206 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013938500000000001 2024-09-21T12:29:10.206 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.206 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.206 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.206 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.441590+0000", 2024-09-21T12:29:10.206 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13709248099999999 2024-09-21T12:29:10.206 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.206 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.206 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:10.206 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.441667+0000", 2024-09-21T12:29:10.206 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7088999999999993e-05 2024-09-21T12:29:10.206 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.206 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.207 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:10.207 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.441707+0000", 2024-09-21T12:29:10.207 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0105999999999999e-05 2024-09-21T12:29:10.207 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.207 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:10.207 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.207 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.207 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.207 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:38144 8.5 8:a00fd695:::benchmark_data_smithi114_104936_object38143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T12:29:10.207 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:11.302034+0000", 2024-09-21T12:29:10.207 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 238.789403389, 2024-09-21T12:29:10.207 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13963275, 2024-09-21T12:29:10.207 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:10.207 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:10.207 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:10.208 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:10.208 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:29:10.208 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:29:10.208 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 38144 2024-09-21T12:29:10.208 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.208 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:10.208 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.208 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:10.208 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302034+0000", 2024-09-21T12:29:10.208 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.208 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.208 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.208 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:10.208 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302034+0000", 2024-09-21T12:29:10.208 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.208 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.209 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.209 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:10.209 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302035+0000", 2024-09-21T12:29:10.209 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1349999999999999e-06 2024-09-21T12:29:10.209 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.209 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.209 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:10.209 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302074+0000", 2024-09-21T12:29:10.209 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9057999999999998e-05 2024-09-21T12:29:10.209 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.209 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.209 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:10.209 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302090+0000", 2024-09-21T12:29:10.209 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6007000000000001e-05 2024-09-21T12:29:10.209 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.209 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.210 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:10.210 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302098+0000", 2024-09-21T12:29:10.210 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9070000000000007e-06 2024-09-21T12:29:10.210 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.210 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.210 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:10.210 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302165+0000", 2024-09-21T12:29:10.210 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7008999999999995e-05 2024-09-21T12:29:10.210 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.210 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.210 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:10.210 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302300+0000", 2024-09-21T12:29:10.210 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013537900000000001 2024-09-21T12:29:10.210 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.211 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.211 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,7", 2024-09-21T12:29:10.211 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302451+0000", 2024-09-21T12:29:10.211 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015088600000000001 2024-09-21T12:29:10.211 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.211 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.211 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:10.211 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.304486+0000", 2024-09-21T12:29:10.211 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020348950000000001 2024-09-21T12:29:10.211 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.211 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.211 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.211 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.304554+0000", 2024-09-21T12:29:10.211 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8201999999999995e-05 2024-09-21T12:29:10.212 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.212 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.212 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.212 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.441565+0000", 2024-09-21T12:29:10.212 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.137010256 2024-09-21T12:29:10.212 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.212 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.212 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:10.212 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.441624+0000", 2024-09-21T12:29:10.212 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9367999999999998e-05 2024-09-21T12:29:10.212 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.212 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.212 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:10.212 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.441667+0000", 2024-09-21T12:29:10.213 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2648000000000002e-05 2024-09-21T12:29:10.213 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.213 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:10.213 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.213 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.213 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.213 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:223832 8.9 8:95f724a3:::benchmark_data_smithi114_104936_object93416:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-09-21T12:29:10.213 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:27:35.314953+0000", 2024-09-21T12:29:10.213 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 94.776484831999994, 2024-09-21T12:29:10.213 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13631865800000001, 2024-09-21T12:29:10.213 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:10.213 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:10.213 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:10.213 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:10.213 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:29:10.214 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:29:10.214 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 223832 2024-09-21T12:29:10.214 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.214 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:10.214 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.214 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:10.214 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.314953+0000", 2024-09-21T12:29:10.214 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.214 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.214 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.214 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:10.214 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.314953+0000", 2024-09-21T12:29:10.214 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:10.214 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.215 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.215 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:10.215 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.314954+0000", 2024-09-21T12:29:10.215 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4249999999999999e-06 2024-09-21T12:29:10.215 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.215 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.215 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:10.215 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.314959+0000", 2024-09-21T12:29:10.215 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6149999999999999e-06 2024-09-21T12:29:10.215 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.215 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.215 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:10.215 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.314973+0000", 2024-09-21T12:29:10.215 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4134999999999999e-05 2024-09-21T12:29:10.216 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.216 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.216 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:10.216 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.314979+0000", 2024-09-21T12:29:10.216 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3489999999999997e-06 2024-09-21T12:29:10.216 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.216 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.216 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:10.216 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.315029+0000", 2024-09-21T12:29:10.216 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0272999999999998e-05 2024-09-21T12:29:10.216 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.216 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.216 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:10.216 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.315137+0000", 2024-09-21T12:29:10.216 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000107878 2024-09-21T12:29:10.216 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.217 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.217 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-21T12:29:10.217 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.315251+0000", 2024-09-21T12:29:10.217 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000113453 2024-09-21T12:29:10.217 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.217 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.217 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:10.217 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.317452+0000", 2024-09-21T12:29:10.217 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0022017080000000001 2024-09-21T12:29:10.217 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.217 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.217 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.217 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.317539+0000", 2024-09-21T12:29:10.217 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6935999999999996e-05 2024-09-21T12:29:10.218 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.218 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.218 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:10.218 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.451201+0000", 2024-09-21T12:29:10.218 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13366187199999999 2024-09-21T12:29:10.218 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.218 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.218 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:10.218 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.451237+0000", 2024-09-21T12:29:10.218 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6273999999999999e-05 2024-09-21T12:29:10.218 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:10.218 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:10.218 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:10.218 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.451271+0000", 2024-09-21T12:29:10.218 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3739999999999999e-05 2024-09-21T12:29:10.219 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.219 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:10.219 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.219 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:10.219 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:10.219 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:29:10.219 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:10.227 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:09 smithi114 ceph-mon[40867]: 1.6 scrub starts 2024-09-21T12:29:10.227 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:09 smithi114 ceph-mon[40867]: 1.6 scrub ok 2024-09-21T12:29:10.227 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:09 smithi114 ceph-mon[40867]: 4.5 scrub starts 2024-09-21T12:29:10.227 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:09 smithi114 ceph-mon[40867]: 4.5 scrub ok 2024-09-21T12:29:10.227 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:09 smithi114 ceph-mon[30324]: 1.6 scrub starts 2024-09-21T12:29:10.227 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:09 smithi114 ceph-mon[30324]: 1.6 scrub ok 2024-09-21T12:29:10.228 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:09 smithi114 ceph-mon[30324]: 4.5 scrub starts 2024-09-21T12:29:10.228 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:09 smithi114 ceph-mon[30324]: 4.5 scrub ok 2024-09-21T12:29:10.266 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:10.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:10.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:10.367 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:10.394 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:10.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:10.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:10.495 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:10.527 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:10.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:10.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:10.616 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:10.628 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:10.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:10.683 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:10.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:10.784 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:10.823 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:10.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:10.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:10.828 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 43 16 76216 76200 110.744 88.375 0.0163893 0.00902527 2024-09-21T12:29:10.925 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:10.953 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:10.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:10.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:11.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:10 smithi114 ceph-mon[30324]: pgmap v437: 178 pgs: 178 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.33k op/s; 1.5 KiB/s, 0 objects/s recovering 2024-09-21T12:29:11.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:10 smithi114 ceph-mon[30324]: 3.7 scrub starts 2024-09-21T12:29:11.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:10 smithi114 ceph-mon[30324]: 3.7 scrub ok 2024-09-21T12:29:11.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:10 smithi114 ceph-mon[30324]: 5.19 scrub starts 2024-09-21T12:29:11.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:10 smithi114 ceph-mon[30324]: 5.19 scrub ok 2024-09-21T12:29:11.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:10 smithi114 ceph-mon[40867]: pgmap v437: 178 pgs: 178 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.33k op/s; 1.5 KiB/s, 0 objects/s recovering 2024-09-21T12:29:11.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:10 smithi114 ceph-mon[40867]: 3.7 scrub starts 2024-09-21T12:29:11.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:10 smithi114 ceph-mon[40867]: 3.7 scrub ok 2024-09-21T12:29:11.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:10 smithi114 ceph-mon[40867]: 5.19 scrub starts 2024-09-21T12:29:11.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:10 smithi114 ceph-mon[40867]: 5.19 scrub ok 2024-09-21T12:29:11.052 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:29:11.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:11.076 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T12:29:11.076 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:29:11.076 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-21T12:29:11.076 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 1000 1.0 2024-09-21T12:29:11.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:11.152 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:11.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:11.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:10 smithi183 ceph-mon[33277]: pgmap v437: 178 pgs: 178 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.33k op/s; 1.5 KiB/s, 0 objects/s recovering 2024-09-21T12:29:11.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:10 smithi183 ceph-mon[33277]: 3.7 scrub starts 2024-09-21T12:29:11.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:10 smithi183 ceph-mon[33277]: 3.7 scrub ok 2024-09-21T12:29:11.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:10 smithi183 ceph-mon[33277]: 5.19 scrub starts 2024-09-21T12:29:11.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:10 smithi183 ceph-mon[33277]: 5.19 scrub ok 2024-09-21T12:29:11.253 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:11.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:11.323 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:11.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:11.424 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:11.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:11.487 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:11.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:11.532 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:11.587 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:11.614 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:11.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:11.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:11.715 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:11.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:11.747 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:11.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:11.829 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 44 16 77408 77392 109.92 74.5 0.00801946 0.00909203 2024-09-21T12:29:11.848 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:11.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:11.891 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:11.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:11.992 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:11.995 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-21T12:29:12.020 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:12.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:12.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:12.122 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:12.147 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:12.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:12.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:12.248 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:11 smithi114 ceph-mon[40867]: 5.8 scrub starts 2024-09-21T12:29:12.248 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:11 smithi114 ceph-mon[40867]: 5.8 scrub ok 2024-09-21T12:29:12.248 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:11 smithi114 ceph-mon[40867]: 1.3 scrub starts 2024-09-21T12:29:12.248 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:11 smithi114 ceph-mon[40867]: 1.3 scrub ok 2024-09-21T12:29:12.248 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:11 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1731477164' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:29:12.248 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:12.250 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:11 smithi114 ceph-mon[30324]: 5.8 scrub starts 2024-09-21T12:29:12.250 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:11 smithi114 ceph-mon[30324]: 5.8 scrub ok 2024-09-21T12:29:12.250 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:11 smithi114 ceph-mon[30324]: 1.3 scrub starts 2024-09-21T12:29:12.250 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:11 smithi114 ceph-mon[30324]: 1.3 scrub ok 2024-09-21T12:29:12.250 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:11 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1731477164' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:29:12.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:12.278 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:12.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:12.321 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:11 smithi183 ceph-mon[33277]: 5.8 scrub starts 2024-09-21T12:29:12.321 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:11 smithi183 ceph-mon[33277]: 5.8 scrub ok 2024-09-21T12:29:12.321 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:11 smithi183 ceph-mon[33277]: 1.3 scrub starts 2024-09-21T12:29:12.321 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:11 smithi183 ceph-mon[33277]: 1.3 scrub ok 2024-09-21T12:29:12.321 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:11 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1731477164' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:29:12.333 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:29:12.333 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:29:12.334 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:29:12.334 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:29:12.334 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:29:12.334 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:29:12.334 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:29:12.334 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:29:12.334 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:29:12.334 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:29:12.334 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:29:12.334 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:29:12.359 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:29:12.359 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:29:12.379 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:12.396 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:29:12.396 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:29:12.421 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:12.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:12.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:12.522 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:12.556 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:12.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:12.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:12.658 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:12.695 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:12.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:12.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:12.796 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:12.823 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T12:29:12.828 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:12.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:12.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:12.829 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 45 16 78737 78721 109.323 83.0625 0.00450922 0.00914267 2024-09-21T12:29:12.930 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:12.964 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:12.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:12.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:12.994 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:29:12 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:29:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:29:12.994 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:12 smithi114 ceph-mon[40867]: pgmap v438: 178 pgs: 178 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.6 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 1.5 KiB/s, 0 objects/s recovering 2024-09-21T12:29:12.994 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:12 smithi114 ceph-mon[30324]: pgmap v438: 178 pgs: 178 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.6 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 1.5 KiB/s, 0 objects/s recovering 2024-09-21T12:29:12.994 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:12 smithi114 ceph-mon[30324]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:29:12.994 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:12 smithi114 ceph-mon[30324]: Cluster is now healthy 2024-09-21T12:29:12.995 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:12 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1731477164' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:29:12.995 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:12 smithi114 ceph-mon[30324]: osdmap e258: 8 total, 8 up, 7 in 2024-09-21T12:29:13.065 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:13.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:13.101 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:13.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:13.202 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:13.207 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:29:13.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:12 smithi183 ceph-mon[33277]: pgmap v438: 178 pgs: 178 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.6 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 1.5 KiB/s, 0 objects/s recovering 2024-09-21T12:29:13.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:12 smithi183 ceph-mon[33277]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:29:13.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:12 smithi183 ceph-mon[33277]: Cluster is now healthy 2024-09-21T12:29:13.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:12 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1731477164' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:29:13.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:12 smithi183 ceph-mon[33277]: osdmap e258: 8 total, 8 up, 7 in 2024-09-21T12:29:13.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:12 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2407196434' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:29:13.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:12 smithi183 ceph-mon[33277]: 1.12 scrub starts 2024-09-21T12:29:13.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:12 smithi183 ceph-mon[33277]: 1.12 scrub ok 2024-09-21T12:29:13.243 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:13.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:13.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:13.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:12 smithi114 ceph-mon[40867]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:29:13.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:12 smithi114 ceph-mon[40867]: Cluster is now healthy 2024-09-21T12:29:13.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:12 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1731477164' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:29:13.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:12 smithi114 ceph-mon[40867]: osdmap e258: 8 total, 8 up, 7 in 2024-09-21T12:29:13.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:12 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2407196434' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:29:13.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:12 smithi114 ceph-mon[40867]: 1.12 scrub starts 2024-09-21T12:29:13.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:12 smithi114 ceph-mon[40867]: 1.12 scrub ok 2024-09-21T12:29:13.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:12 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2407196434' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:29:13.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:12 smithi114 ceph-mon[30324]: 1.12 scrub starts 2024-09-21T12:29:13.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:12 smithi114 ceph-mon[30324]: 1.12 scrub ok 2024-09-21T12:29:13.345 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:13.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:13.377 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:13.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:13.478 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:13.513 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:13.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:13.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:13.615 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:13.630 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:13.642 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:13.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:13.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:13.744 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:13.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:13.782 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:13.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:13.799 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:29:13.799 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:29:13.829 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 46 16 80223 80207 108.966 92.875 0.00626014 0.00917191 2024-09-21T12:29:13.883 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:13.914 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:13.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:13.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:14.015 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:14.048 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:14.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:14.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:14.150 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:14.176 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:14.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:14.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:14.207 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:14 smithi114 ceph-mon[40867]: 5.6 scrub starts 2024-09-21T12:29:14.207 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:14 smithi114 ceph-mon[40867]: 5.6 scrub ok 2024-09-21T12:29:14.207 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:14 smithi114 ceph-mon[30324]: 5.6 scrub starts 2024-09-21T12:29:14.207 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:14 smithi114 ceph-mon[30324]: 5.6 scrub ok 2024-09-21T12:29:14.278 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:14.317 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:14.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:14.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:14.419 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:14.446 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:14.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:14.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:14.474 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:14 smithi183 ceph-mon[33277]: 5.6 scrub starts 2024-09-21T12:29:14.474 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:14 smithi183 ceph-mon[33277]: 5.6 scrub ok 2024-09-21T12:29:14.519 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T12:29:14.548 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:14.594 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:14.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:14.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:14.621 INFO:teuthology.orchestra.run.smithi114.stdout:moved 0 / 534 (0%) 2024-09-21T12:29:14.621 INFO:teuthology.orchestra.run.smithi114.stdout:avg 76.2857 2024-09-21T12:29:14.622 INFO:teuthology.orchestra.run.smithi114.stdout:stddev 7.08548 -> 7.08548 (expected baseline 8.08627) 2024-09-21T12:29:14.622 INFO:teuthology.orchestra.run.smithi114.stdout:min osd.2 with 62 -> 62 pgs (0.812734 -> 0.812734 * mean) 2024-09-21T12:29:14.622 INFO:teuthology.orchestra.run.smithi114.stdout:max osd.1 with 83 -> 83 pgs (1.08801 -> 1.08801 * mean) 2024-09-21T12:29:14.622 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:29:14.622 INFO:teuthology.orchestra.run.smithi114.stdout:oload 1000 2024-09-21T12:29:14.622 INFO:teuthology.orchestra.run.smithi114.stdout:max_change 1 2024-09-21T12:29:14.622 INFO:teuthology.orchestra.run.smithi114.stdout:max_change_osds 4 2024-09-21T12:29:14.622 INFO:teuthology.orchestra.run.smithi114.stdout:average_utilization 0.0263 2024-09-21T12:29:14.622 INFO:teuthology.orchestra.run.smithi114.stdout:overload_utilization 0.2627 2024-09-21T12:29:14.622 INFO:teuthology.orchestra.run.smithi114.stderr:no change 2024-09-21T12:29:14.696 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:14.743 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:14.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:14.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:14.829 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 47 16 81624 81608 108.51 87.5625 0.00764949 0.00921138 2024-09-21T12:29:14.846 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:14.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:14.888 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:14.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:14.938 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:29:14.989 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:15.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:15.025 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:15.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:15.126 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:15.161 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:15.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:15.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:15.177 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T12:29:15.263 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:15.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:15 smithi114 ceph-mon[40867]: pgmap v440: 178 pgs: 178 active+clean; 4.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-09-21T12:29:15.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:15 smithi114 ceph-mon[40867]: 6.13 scrub starts 2024-09-21T12:29:15.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:15 smithi114 ceph-mon[40867]: 6.13 scrub ok 2024-09-21T12:29:15.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:15 smithi114 ceph-mon[40867]: 6.1a scrub starts 2024-09-21T12:29:15.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:15 smithi114 ceph-mon[40867]: 6.1a scrub ok 2024-09-21T12:29:15.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:15 smithi114 ceph-mon[40867]: 4.c scrub starts 2024-09-21T12:29:15.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:15 smithi114 ceph-mon[40867]: 4.c scrub ok 2024-09-21T12:29:15.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:15 smithi114 ceph-mon[30324]: pgmap v440: 178 pgs: 178 active+clean; 4.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-09-21T12:29:15.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:15 smithi114 ceph-mon[30324]: 6.13 scrub starts 2024-09-21T12:29:15.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:15 smithi114 ceph-mon[30324]: 6.13 scrub ok 2024-09-21T12:29:15.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:15 smithi114 ceph-mon[30324]: 6.1a scrub starts 2024-09-21T12:29:15.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:15 smithi114 ceph-mon[30324]: 6.1a scrub ok 2024-09-21T12:29:15.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:15 smithi114 ceph-mon[30324]: 4.c scrub starts 2024-09-21T12:29:15.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:15 smithi114 ceph-mon[30324]: 4.c scrub ok 2024-09-21T12:29:15.291 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:15.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:15.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:15.392 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:15.437 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:15.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:15.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:15.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:15 smithi183 ceph-mon[33277]: pgmap v440: 178 pgs: 178 active+clean; 4.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-09-21T12:29:15.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:15 smithi183 ceph-mon[33277]: 6.13 scrub starts 2024-09-21T12:29:15.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:15 smithi183 ceph-mon[33277]: 6.13 scrub ok 2024-09-21T12:29:15.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:15 smithi183 ceph-mon[33277]: 6.1a scrub starts 2024-09-21T12:29:15.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:15 smithi183 ceph-mon[33277]: 6.1a scrub ok 2024-09-21T12:29:15.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:15 smithi183 ceph-mon[33277]: 4.c scrub starts 2024-09-21T12:29:15.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:15 smithi183 ceph-mon[33277]: 4.c scrub ok 2024-09-21T12:29:15.538 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:15.582 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:15.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:15.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:15.677 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:29:15.684 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:15.714 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:15.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:15.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:15.736 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-21T12:29:15.736 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd test-reweight-by-utilization 110 1.0 2024-09-21T12:29:15.816 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:15.832 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 48 16 82934 82918 107.948 81.875 0.010399 0.00925831 2024-09-21T12:29:15.841 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:15.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:15.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:15.878 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:15.943 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:16.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:16.032 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:16.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:16.133 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:16.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:16.201 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:16.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:16.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:16 smithi114 ceph-mon[40867]: from='client.25270 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:29:16.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:16 smithi114 ceph-mon[40867]: 3.18 scrub starts 2024-09-21T12:29:16.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:16 smithi114 ceph-mon[40867]: 3.18 scrub ok 2024-09-21T12:29:16.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:16 smithi114 ceph-mon[40867]: 3.1 scrub starts 2024-09-21T12:29:16.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:16 smithi114 ceph-mon[40867]: 3.1 scrub ok 2024-09-21T12:29:16.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:16 smithi114 ceph-mon[40867]: 5.f scrub starts 2024-09-21T12:29:16.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:16 smithi114 ceph-mon[40867]: 5.f scrub ok 2024-09-21T12:29:16.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:16 smithi114 ceph-mon[40867]: 6.8 scrub starts 2024-09-21T12:29:16.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:16 smithi114 ceph-mon[40867]: 6.8 scrub ok 2024-09-21T12:29:16.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:16 smithi114 ceph-mon[30324]: from='client.25270 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:29:16.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:16 smithi114 ceph-mon[30324]: 3.18 scrub starts 2024-09-21T12:29:16.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:16 smithi114 ceph-mon[30324]: 3.18 scrub ok 2024-09-21T12:29:16.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:16 smithi114 ceph-mon[30324]: 3.1 scrub starts 2024-09-21T12:29:16.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:16 smithi114 ceph-mon[30324]: 3.1 scrub ok 2024-09-21T12:29:16.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:16 smithi114 ceph-mon[30324]: 5.f scrub starts 2024-09-21T12:29:16.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:16 smithi114 ceph-mon[30324]: 5.f scrub ok 2024-09-21T12:29:16.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:16 smithi114 ceph-mon[30324]: 6.8 scrub starts 2024-09-21T12:29:16.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:16 smithi114 ceph-mon[30324]: 6.8 scrub ok 2024-09-21T12:29:16.302 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:16.334 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:16.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:16.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:16.366 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:16 smithi183 ceph-mon[33277]: from='client.25270 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:29:16.366 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:16 smithi183 ceph-mon[33277]: 3.18 scrub starts 2024-09-21T12:29:16.366 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:16 smithi183 ceph-mon[33277]: 3.18 scrub ok 2024-09-21T12:29:16.366 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:16 smithi183 ceph-mon[33277]: 3.1 scrub starts 2024-09-21T12:29:16.366 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:16 smithi183 ceph-mon[33277]: 3.1 scrub ok 2024-09-21T12:29:16.367 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:16 smithi183 ceph-mon[33277]: 5.f scrub starts 2024-09-21T12:29:16.367 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:16 smithi183 ceph-mon[33277]: 5.f scrub ok 2024-09-21T12:29:16.367 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:16 smithi183 ceph-mon[33277]: 6.8 scrub starts 2024-09-21T12:29:16.367 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:16 smithi183 ceph-mon[33277]: 6.8 scrub ok 2024-09-21T12:29:16.435 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:16.475 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:16.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:16.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:16.576 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:16.601 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:16.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:16.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:16.622 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:16.638 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:16.703 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:16.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:16.752 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:16.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:16.783 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:29:16.783 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:29:16.833 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 49 16 84078 84062 107.202 71.5 0.00631188 0.00932332 2024-09-21T12:29:16.853 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:16.890 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:16.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:16.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:16.955 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-21T12:29:16.992 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:17.034 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:17.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:17.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:17.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:17 smithi183 ceph-mon[33277]: pgmap v441: 178 pgs: 178 active+clean; 4.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 824 B/s rd, 83 MiB/s wr, 1.34k op/s 2024-09-21T12:29:17.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:17 smithi183 ceph-mon[33277]: 3.1e scrub starts 2024-09-21T12:29:17.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:17 smithi183 ceph-mon[33277]: 3.1e scrub ok 2024-09-21T12:29:17.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:17 smithi183 ceph-mon[33277]: 3.1b scrub starts 2024-09-21T12:29:17.136 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:17.168 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:17.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:17.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:17.269 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:17.296 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:17.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:17.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:17.298 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:17 smithi114 ceph-mon[30324]: pgmap v441: 178 pgs: 178 active+clean; 4.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 824 B/s rd, 83 MiB/s wr, 1.34k op/s 2024-09-21T12:29:17.298 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:17 smithi114 ceph-mon[30324]: 3.1e scrub starts 2024-09-21T12:29:17.298 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:17 smithi114 ceph-mon[30324]: 3.1e scrub ok 2024-09-21T12:29:17.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:17 smithi114 ceph-mon[30324]: 3.1b scrub starts 2024-09-21T12:29:17.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:17 smithi114 ceph-mon[30324]: 3.1b scrub ok 2024-09-21T12:29:17.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:17 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:29:17.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:17 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:29:17.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:17 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3290282313' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:29:17.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:17 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:29:17.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:17 smithi114 ceph-mon[30324]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:29:17.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:17 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:29:17.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:17 smithi114 ceph-mon[30324]: osdmap e259: 8 total, 8 up, 7 in 2024-09-21T12:29:17.299 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:17 smithi114 ceph-mon[40867]: pgmap v441: 178 pgs: 178 active+clean; 4.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 824 B/s rd, 83 MiB/s wr, 1.34k op/s 2024-09-21T12:29:17.299 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:17 smithi114 ceph-mon[40867]: 3.1e scrub starts 2024-09-21T12:29:17.299 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:17 smithi114 ceph-mon[40867]: 3.1e scrub ok 2024-09-21T12:29:17.299 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:17 smithi114 ceph-mon[40867]: 3.1b scrub starts 2024-09-21T12:29:17.300 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:17 smithi114 ceph-mon[40867]: 3.1b scrub ok 2024-09-21T12:29:17.300 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:17 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:29:17.300 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:17 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:29:17.300 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:17 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3290282313' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:29:17.300 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:17 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:29:17.300 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:17 smithi114 ceph-mon[40867]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:29:17.300 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:17 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:29:17.300 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:17 smithi114 ceph-mon[40867]: osdmap e259: 8 total, 8 up, 7 in 2024-09-21T12:29:17.387 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:29:17 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:29:17.388 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:17 smithi183 ceph-mon[33277]: 3.1b scrub ok 2024-09-21T12:29:17.388 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:17 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:29:17.388 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:17 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:29:17.388 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:17 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3290282313' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:29:17.388 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:17 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:29:17.388 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:17 smithi183 ceph-mon[33277]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:29:17.388 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:17 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:29:17.388 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:17 smithi183 ceph-mon[33277]: osdmap e259: 8 total, 8 up, 7 in 2024-09-21T12:29:17.398 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:17.424 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:17.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:17.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:17.500 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:29:17.500 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:29:17.525 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:17.539 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:29:17.539 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:29:17.554 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:17.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:17.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:17.582 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:29:17.582 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:29:17.621 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:29:17.622 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:29:17.628 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T12:29:17.655 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:17.662 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:29:17.662 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:29:17.699 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:17.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:17.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:17.705 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:29:17.705 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:29:17.740 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:29:17.741 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:29:17.779 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:29:17.779 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:29:17.800 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:17.827 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:17.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:17.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:17.833 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 50 16 85375 85359 106.679 81.0625 0.024345 0.00936916 2024-09-21T12:29:17.929 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:17.963 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:29:17.973 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:17.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:17.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:18.075 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:18.089 INFO:teuthology.orchestra.run.smithi114.stdout:moved 64 / 534 (11.985%) 2024-09-21T12:29:18.089 INFO:teuthology.orchestra.run.smithi114.stdout:avg 76.2857 2024-09-21T12:29:18.089 INFO:teuthology.orchestra.run.smithi114.stdout:stddev 7.08548 -> 3.91752 (expected baseline 8.08627) 2024-09-21T12:29:18.089 INFO:teuthology.orchestra.run.smithi114.stdout:min osd.2 with 62 -> 73 pgs (0.812734 -> 0.956929 * mean) 2024-09-21T12:29:18.089 INFO:teuthology.orchestra.run.smithi114.stdout:max osd.1 with 83 -> 74 pgs (1.08801 -> 0.970037 * mean) 2024-09-21T12:29:18.089 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:29:18.089 INFO:teuthology.orchestra.run.smithi114.stdout:oload 110 2024-09-21T12:29:18.089 INFO:teuthology.orchestra.run.smithi114.stdout:max_change 1 2024-09-21T12:29:18.089 INFO:teuthology.orchestra.run.smithi114.stdout:max_change_osds 4 2024-09-21T12:29:18.089 INFO:teuthology.orchestra.run.smithi114.stdout:average_utilization 0.0283 2024-09-21T12:29:18.089 INFO:teuthology.orchestra.run.smithi114.stdout:overload_utilization 0.0311 2024-09-21T12:29:18.089 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7 weight 1.0000 -> 0.7946 2024-09-21T12:29:18.090 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 weight 1.0000 -> 0.7976 2024-09-21T12:29:18.090 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 weight 1.0000 -> 0.7744 2024-09-21T12:29:18.090 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 weight 1.0000 -> 0.8509 2024-09-21T12:29:18.090 INFO:teuthology.orchestra.run.smithi114.stderr:no change 2024-09-21T12:29:18.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:18.111 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:18.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:18.212 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:18.240 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:18.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:18.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:18.341 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:18.371 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:18 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:29:18.371 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:18 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3965873083' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:29:18.378 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:18.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:18.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:18.479 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:18.518 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:18.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:18.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:18.524 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:29:18 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: 2024-09-21T12:29:18.085+0000 7f0da3311640 -1 mgr.server reply reply (4) Interrupted system call no change 2024-09-21T12:29:18.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:18 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:29:18.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:18 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3965873083' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:29:18.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:18 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:29:18.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:18 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3965873083' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:29:18.529 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:29:18.529 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-21T12:29:18.529 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-21T12:29:18.529 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:29:18.529 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.529 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14478.0:1121 5.9 5:93e5b521:::notify.7:head [watch ping cookie 94620848900224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-21T12:29:18.529 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:40.364371+0000", 2024-09-21T12:29:18.529 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 458.16075319999999, 2024-09-21T12:29:18.529 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.77380638599999996, 2024-09-21T12:29:18.529 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:18.529 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:18.530 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:29:18.530 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:18.530 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14478", 2024-09-21T12:29:18.530 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:29:18.530 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 1121 2024-09-21T12:29:18.530 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.530 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:18.530 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.530 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:18.530 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364371+0000", 2024-09-21T12:29:18.530 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.530 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.530 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.530 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:18.530 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364371+0000", 2024-09-21T12:29:18.531 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.531 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.531 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.531 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:18.531 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364372+0000", 2024-09-21T12:29:18.531 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7400000000000002e-07 2024-09-21T12:29:18.531 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.531 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.531 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:18.531 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364376+0000", 2024-09-21T12:29:18.531 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6789999999999998e-06 2024-09-21T12:29:18.531 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.531 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.531 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:18.532 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364384+0000", 2024-09-21T12:29:18.532 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3299999999999999e-06 2024-09-21T12:29:18.532 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.532 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.532 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:18.532 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364393+0000", 2024-09-21T12:29:18.532 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6379999999999993e-06 2024-09-21T12:29:18.532 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.532 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.532 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:18.532 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364494+0000", 2024-09-21T12:29:18.532 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010131799999999999 2024-09-21T12:29:18.532 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.532 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.532 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:29:18.533 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364510+0000", 2024-09-21T12:29:18.533 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5630000000000001e-05 2024-09-21T12:29:18.533 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.533 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.533 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:18.533 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.134462+0000", 2024-09-21T12:29:18.533 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.76995182100000004 2024-09-21T12:29:18.533 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.533 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.533 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for degraded object", 2024-09-21T12:29:18.533 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.134766+0000", 2024-09-21T12:29:18.533 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00030401499999999998 2024-09-21T12:29:18.533 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.533 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.534 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:18.534 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.138007+0000", 2024-09-21T12:29:18.534 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0032417420000000001 2024-09-21T12:29:18.534 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.534 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.534 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:18.534 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.138098+0000", 2024-09-21T12:29:18.534 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0841000000000007e-05 2024-09-21T12:29:18.534 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.534 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.534 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:18.534 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.138178+0000", 2024-09-21T12:29:18.534 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9498000000000004e-05 2024-09-21T12:29:18.534 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.535 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:18.535 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.535 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.535 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.535 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14478.0:1126 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94620848900224 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:29:18.535 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:40.364831+0000", 2024-09-21T12:29:18.535 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 458.160293341, 2024-09-21T12:29:18.535 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78063636199999997, 2024-09-21T12:29:18.535 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:18.535 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:18.535 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:29:18.535 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:18.535 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14478", 2024-09-21T12:29:18.536 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:29:18.536 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 1126 2024-09-21T12:29:18.536 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.536 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:18.536 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.536 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:18.536 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364831+0000", 2024-09-21T12:29:18.536 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.536 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.536 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.536 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:18.536 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364831+0000", 2024-09-21T12:29:18.536 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.537 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.537 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.537 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:18.537 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364832+0000", 2024-09-21T12:29:18.537 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1900000000000001e-07 2024-09-21T12:29:18.537 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.537 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.537 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:18.537 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364835+0000", 2024-09-21T12:29:18.537 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5099999999999999e-06 2024-09-21T12:29:18.537 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.537 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.537 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:18.537 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364846+0000", 2024-09-21T12:29:18.538 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0139000000000001e-05 2024-09-21T12:29:18.538 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.538 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.538 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:18.538 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364853+0000", 2024-09-21T12:29:18.538 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2629999999999997e-06 2024-09-21T12:29:18.538 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.538 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.538 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:18.538 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364911+0000", 2024-09-21T12:29:18.538 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8493999999999999e-05 2024-09-21T12:29:18.538 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.538 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.538 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:29:18.539 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364921+0000", 2024-09-21T12:29:18.539 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8400000000000007e-06 2024-09-21T12:29:18.539 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.539 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.539 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:18.539 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.134966+0000", 2024-09-21T12:29:18.539 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.77004525300000004 2024-09-21T12:29:18.539 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.539 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.539 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for degraded object", 2024-09-21T12:29:18.539 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.135016+0000", 2024-09-21T12:29:18.539 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9419000000000002e-05 2024-09-21T12:29:18.539 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.539 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.539 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:18.540 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.138211+0000", 2024-09-21T12:29:18.540 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0031947360000000001 2024-09-21T12:29:18.540 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.540 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.540 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:18.540 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.138283+0000", 2024-09-21T12:29:18.540 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2712000000000004e-05 2024-09-21T12:29:18.540 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.540 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.540 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:18.540 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.145467+0000", 2024-09-21T12:29:18.540 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0071841769999999999 2024-09-21T12:29:18.540 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.540 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:18.541 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.541 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.541 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.541 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:24816 7.5 7:a47d801e:::benchmark_data_smithi114_78432_object24815:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T12:29:18.541 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:42.446936+0000", 2024-09-21T12:29:18.541 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 456.07818873299999, 2024-09-21T12:29:18.541 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.144229354, 2024-09-21T12:29:18.541 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:18.541 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:18.541 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:18.541 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:18.541 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:29:18.541 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:29:18.541 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 24816 2024-09-21T12:29:18.541 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.541 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:18.542 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.542 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:18.542 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.446936+0000", 2024-09-21T12:29:18.542 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.542 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.542 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.542 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:18.542 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.446936+0000", 2024-09-21T12:29:18.543 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.543 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.543 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.543 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:18.543 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.446938+0000", 2024-09-21T12:29:18.543 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4889999999999998e-06 2024-09-21T12:29:18.543 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.543 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.543 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:18.543 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.446993+0000", 2024-09-21T12:29:18.543 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4778999999999998e-05 2024-09-21T12:29:18.543 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.543 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.544 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:18.544 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.447007+0000", 2024-09-21T12:29:18.544 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4399000000000001e-05 2024-09-21T12:29:18.544 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.544 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.544 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:18.544 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.447014+0000", 2024-09-21T12:29:18.544 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0700000000000001e-06 2024-09-21T12:29:18.544 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.544 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.544 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:18.544 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.447082+0000", 2024-09-21T12:29:18.544 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7971000000000004e-05 2024-09-21T12:29:18.544 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.544 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.544 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:18.545 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.447211+0000", 2024-09-21T12:29:18.545 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012822300000000001 2024-09-21T12:29:18.545 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.545 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.545 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:29:18.545 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.447358+0000", 2024-09-21T12:29:18.545 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014737999999999999 2024-09-21T12:29:18.545 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.545 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.545 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:18.545 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.448450+0000", 2024-09-21T12:29:18.545 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001092148 2024-09-21T12:29:18.545 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.545 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.545 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.546 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.448529+0000", 2024-09-21T12:29:18.546 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9090000000000003e-05 2024-09-21T12:29:18.546 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.546 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.546 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.546 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.591057+0000", 2024-09-21T12:29:18.546 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.142527445 2024-09-21T12:29:18.546 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.546 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.546 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:18.546 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.591120+0000", 2024-09-21T12:29:18.546 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3695000000000006e-05 2024-09-21T12:29:18.546 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.546 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.547 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:18.547 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.591165+0000", 2024-09-21T12:29:18.547 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4665e-05 2024-09-21T12:29:18.547 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.547 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:18.547 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.547 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.547 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.547 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:41241 7.5 7:a7b2dcc6:::benchmark_data_smithi114_78432_object41240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-21T12:29:18.547 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:55.817543+0000", 2024-09-21T12:29:18.547 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 442.707581694, 2024-09-21T12:29:18.547 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.094563281999999999, 2024-09-21T12:29:18.547 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:18.547 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:18.547 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:18.548 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:18.548 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:29:18.548 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:29:18.548 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 41241 2024-09-21T12:29:18.548 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.548 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:18.548 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.548 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:18.548 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:55.817543+0000", 2024-09-21T12:29:18.548 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.548 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.548 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.548 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:18.548 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:55.817543+0000", 2024-09-21T12:29:18.549 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.549 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:18.549 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:55.817544+0000", 2024-09-21T12:29:18.549 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.488e-06 2024-09-21T12:29:18.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.549 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:18.549 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:55.817593+0000", 2024-09-21T12:29:18.549 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8977000000000002e-05 2024-09-21T12:29:18.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.549 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:18.549 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:55.817607+0000", 2024-09-21T12:29:18.550 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3623e-05 2024-09-21T12:29:18.550 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.550 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.550 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:18.550 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:55.817614+0000", 2024-09-21T12:29:18.550 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3610000000000003e-06 2024-09-21T12:29:18.550 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.550 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.550 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:18.550 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:55.817669+0000", 2024-09-21T12:29:18.550 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5089000000000001e-05 2024-09-21T12:29:18.550 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.550 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.550 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:18.550 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:55.817767+0000", 2024-09-21T12:29:18.550 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7943999999999996e-05 2024-09-21T12:29:18.551 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.551 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.551 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:29:18.551 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:55.817863+0000", 2024-09-21T12:29:18.551 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6253000000000002e-05 2024-09-21T12:29:18.551 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.551 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.551 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:18.551 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:55.818953+0000", 2024-09-21T12:29:18.551 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010892180000000001 2024-09-21T12:29:18.551 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.551 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.551 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.551 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:55.819836+0000", 2024-09-21T12:29:18.552 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00088346699999999998 2024-09-21T12:29:18.552 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.552 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.552 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.552 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:55.912001+0000", 2024-09-21T12:29:18.552 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.092164974999999996 2024-09-21T12:29:18.552 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.552 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.552 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:18.552 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:55.912065+0000", 2024-09-21T12:29:18.552 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3942000000000005e-05 2024-09-21T12:29:18.552 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.552 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.552 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:18.552 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:55.912106+0000", 2024-09-21T12:29:18.552 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0945000000000002e-05 2024-09-21T12:29:18.553 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.553 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:18.553 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.553 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.553 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.553 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:45176 7.5 7:ae38956b:::benchmark_data_smithi114_78432_object45175:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T12:29:18.553 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:58.998297+0000", 2024-09-21T12:29:18.553 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 439.52682729000003, 2024-09-21T12:29:18.553 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.082483624000000005, 2024-09-21T12:29:18.553 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:18.553 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:18.553 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:18.553 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:18.553 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:29:18.553 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:29:18.554 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 45176 2024-09-21T12:29:18.554 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.554 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:18.554 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.554 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:18.554 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:58.998297+0000", 2024-09-21T12:29:18.554 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.554 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.554 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.554 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:18.554 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:58.998297+0000", 2024-09-21T12:29:18.554 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.554 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.554 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.554 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:18.554 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:58.998299+0000", 2024-09-21T12:29:18.555 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5829999999999999e-06 2024-09-21T12:29:18.555 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.555 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.555 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:18.555 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:58.998352+0000", 2024-09-21T12:29:18.555 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3068000000000001e-05 2024-09-21T12:29:18.555 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.555 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.555 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:18.555 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:58.998364+0000", 2024-09-21T12:29:18.555 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2105e-05 2024-09-21T12:29:18.555 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.555 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.555 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:18.555 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:58.998372+0000", 2024-09-21T12:29:18.556 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7340000000000008e-06 2024-09-21T12:29:18.556 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.556 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.556 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:18.556 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:58.998424+0000", 2024-09-21T12:29:18.556 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2556999999999998e-05 2024-09-21T12:29:18.556 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.556 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.556 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:18.556 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:58.998550+0000", 2024-09-21T12:29:18.556 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012533999999999999 2024-09-21T12:29:18.556 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.556 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.556 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:29:18.556 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:58.998683+0000", 2024-09-21T12:29:18.556 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000133521 2024-09-21T12:29:18.557 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.557 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.557 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:18.557 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.000234+0000", 2024-09-21T12:29:18.557 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0015507120000000001 2024-09-21T12:29:18.557 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.557 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.557 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.557 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.001510+0000", 2024-09-21T12:29:18.557 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012758050000000001 2024-09-21T12:29:18.557 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.557 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.557 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.557 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.080665+0000", 2024-09-21T12:29:18.557 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.079155602000000005 2024-09-21T12:29:18.558 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.558 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.558 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:18.558 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.080733+0000", 2024-09-21T12:29:18.558 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7799000000000006e-05 2024-09-21T12:29:18.558 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.558 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.558 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:18.558 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.080781+0000", 2024-09-21T12:29:18.558 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7797999999999999e-05 2024-09-21T12:29:18.558 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.558 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:18.558 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.558 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.558 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.559 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:45184 7.5 7:a309e6d9:::benchmark_data_smithi114_78432_object45183:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T12:29:18.559 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:59.002715+0000", 2024-09-21T12:29:18.559 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 439.52240920700001, 2024-09-21T12:29:18.559 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.085275684000000004, 2024-09-21T12:29:18.559 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:18.559 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:18.559 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:18.559 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:18.559 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:29:18.559 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:29:18.559 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 45184 2024-09-21T12:29:18.559 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.559 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:18.559 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.559 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:18.559 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.002715+0000", 2024-09-21T12:29:18.560 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.560 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.560 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.560 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:18.560 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.002715+0000", 2024-09-21T12:29:18.560 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.560 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.560 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.560 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:18.560 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.002716+0000", 2024-09-21T12:29:18.560 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2529999999999999e-06 2024-09-21T12:29:18.560 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.560 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.560 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:18.560 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.002783+0000", 2024-09-21T12:29:18.560 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6637999999999995e-05 2024-09-21T12:29:18.561 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.561 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.561 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:18.561 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.002795+0000", 2024-09-21T12:29:18.561 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1576999999999999e-05 2024-09-21T12:29:18.561 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.561 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.561 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:18.561 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.002802+0000", 2024-09-21T12:29:18.561 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4340000000000003e-06 2024-09-21T12:29:18.561 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.561 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.561 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:18.561 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.002863+0000", 2024-09-21T12:29:18.562 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1058999999999999e-05 2024-09-21T12:29:18.562 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.562 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.562 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:18.562 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.002952+0000", 2024-09-21T12:29:18.562 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8561999999999997e-05 2024-09-21T12:29:18.562 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.562 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.562 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:29:18.562 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.003037+0000", 2024-09-21T12:29:18.562 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5730999999999996e-05 2024-09-21T12:29:18.562 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.562 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.562 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:18.563 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.004105+0000", 2024-09-21T12:29:18.563 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010670650000000001 2024-09-21T12:29:18.563 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.563 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.563 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.563 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.049583+0000", 2024-09-21T12:29:18.563 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.045478867999999999 2024-09-21T12:29:18.563 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.563 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.563 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.563 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.087882+0000", 2024-09-21T12:29:18.563 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.038298454000000003 2024-09-21T12:29:18.563 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.564 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.564 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:18.564 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.087950+0000", 2024-09-21T12:29:18.564 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7972000000000006e-05 2024-09-21T12:29:18.564 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.564 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.564 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:18.564 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.087991+0000", 2024-09-21T12:29:18.564 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1071e-05 2024-09-21T12:29:18.564 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.564 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:18.564 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.564 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.564 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.565 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:48503 7.5 7:aed652f8:::benchmark_data_smithi114_78432_object48502:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:29:18.565 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:04.030415+0000", 2024-09-21T12:29:18.565 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 434.49470974899998, 2024-09-21T12:29:18.565 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.086380786000000001, 2024-09-21T12:29:18.565 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:18.565 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:18.565 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:18.565 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:18.565 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:29:18.565 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:29:18.565 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 48503 2024-09-21T12:29:18.565 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.565 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:18.565 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.565 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:18.566 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.030415+0000", 2024-09-21T12:29:18.566 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.566 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.566 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.566 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:18.566 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.030415+0000", 2024-09-21T12:29:18.566 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.566 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.566 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.566 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:18.566 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.030416+0000", 2024-09-21T12:29:18.566 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.437e-06 2024-09-21T12:29:18.566 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.566 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.566 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:18.567 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.030461+0000", 2024-09-21T12:29:18.567 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5272000000000003e-05 2024-09-21T12:29:18.567 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.567 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.567 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:18.567 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.030471+0000", 2024-09-21T12:29:18.567 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7480000000000005e-06 2024-09-21T12:29:18.567 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.567 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.567 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:18.567 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.030477+0000", 2024-09-21T12:29:18.567 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8119999999999996e-06 2024-09-21T12:29:18.567 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.567 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.568 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:18.568 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.030530+0000", 2024-09-21T12:29:18.568 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2809000000000002e-05 2024-09-21T12:29:18.568 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.568 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.568 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:18.568 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.030663+0000", 2024-09-21T12:29:18.568 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000133647 2024-09-21T12:29:18.568 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.568 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.568 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:29:18.568 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.030798+0000", 2024-09-21T12:29:18.568 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013433100000000001 2024-09-21T12:29:18.568 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.568 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.568 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:18.569 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.032271+0000", 2024-09-21T12:29:18.569 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001472945 2024-09-21T12:29:18.569 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.569 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.569 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.569 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.032510+0000", 2024-09-21T12:29:18.569 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00023884399999999999 2024-09-21T12:29:18.569 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.569 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.569 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.569 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.116713+0000", 2024-09-21T12:29:18.569 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.084203518000000005 2024-09-21T12:29:18.569 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.569 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.570 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:18.570 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.116756+0000", 2024-09-21T12:29:18.570 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2917999999999998e-05 2024-09-21T12:29:18.570 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.570 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.570 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:18.570 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.116795+0000", 2024-09-21T12:29:18.570 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9505000000000002e-05 2024-09-21T12:29:18.570 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.570 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:18.570 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.570 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.570 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.570 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:49232 7.5 7:a28409c5:::benchmark_data_smithi114_78432_object49231:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:29:18.570 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:04.569750+0000", 2024-09-21T12:29:18.571 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 433.95537423399998, 2024-09-21T12:29:18.571 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.147577404, 2024-09-21T12:29:18.571 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:18.571 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:18.571 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:18.571 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:18.571 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:29:18.571 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:29:18.571 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 49232 2024-09-21T12:29:18.571 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.571 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:18.571 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.571 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:18.571 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569750+0000", 2024-09-21T12:29:18.571 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.572 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.572 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.572 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:18.572 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569750+0000", 2024-09-21T12:29:18.572 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.572 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.572 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.572 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:18.572 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569752+0000", 2024-09-21T12:29:18.572 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6050000000000001e-06 2024-09-21T12:29:18.572 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.572 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.572 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:18.572 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569802+0000", 2024-09-21T12:29:18.572 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0664999999999997e-05 2024-09-21T12:29:18.572 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.572 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.573 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:18.573 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569819+0000", 2024-09-21T12:29:18.573 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6348e-05 2024-09-21T12:29:18.573 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.573 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.573 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:18.573 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569826+0000", 2024-09-21T12:29:18.573 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3830000000000003e-06 2024-09-21T12:29:18.573 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.573 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.573 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:18.573 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569882+0000", 2024-09-21T12:29:18.573 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5519000000000002e-05 2024-09-21T12:29:18.573 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.574 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.574 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:18.574 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.570014+0000", 2024-09-21T12:29:18.574 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013235499999999999 2024-09-21T12:29:18.574 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.574 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.574 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:29:18.574 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.570150+0000", 2024-09-21T12:29:18.574 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013635700000000001 2024-09-21T12:29:18.574 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.574 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.574 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:18.574 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.572290+0000", 2024-09-21T12:29:18.574 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021393050000000002 2024-09-21T12:29:18.575 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.575 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.575 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.575 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.575659+0000", 2024-09-21T12:29:18.575 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0033696720000000002 2024-09-21T12:29:18.575 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.575 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.575 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.575 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.717213+0000", 2024-09-21T12:29:18.575 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14155382899999999 2024-09-21T12:29:18.575 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.575 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.575 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:18.575 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.717282+0000", 2024-09-21T12:29:18.575 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8554999999999997e-05 2024-09-21T12:29:18.576 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.576 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.576 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:18.576 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.717328+0000", 2024-09-21T12:29:18.576 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5810999999999999e-05 2024-09-21T12:29:18.576 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.576 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:18.576 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.576 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.576 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.576 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:49947 7.5 7:a57824f9:::benchmark_data_smithi114_78432_object49946:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:29:18.576 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:05.215753+0000", 2024-09-21T12:29:18.576 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 433.30937093300003, 2024-09-21T12:29:18.576 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.173951617, 2024-09-21T12:29:18.576 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:18.577 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:18.577 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:18.577 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:18.577 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:29:18.577 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:29:18.577 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 49947 2024-09-21T12:29:18.577 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.577 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:18.577 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.577 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:18.577 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215753+0000", 2024-09-21T12:29:18.577 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.577 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.577 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.577 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:18.577 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215753+0000", 2024-09-21T12:29:18.578 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.578 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.578 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.578 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:18.578 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215755+0000", 2024-09-21T12:29:18.578 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.578e-06 2024-09-21T12:29:18.578 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.578 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.578 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:18.578 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215824+0000", 2024-09-21T12:29:18.578 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9362000000000004e-05 2024-09-21T12:29:18.578 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.578 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.578 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:18.578 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215835+0000", 2024-09-21T12:29:18.579 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0791999999999999e-05 2024-09-21T12:29:18.579 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.579 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.579 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:18.579 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215841+0000", 2024-09-21T12:29:18.579 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2430000000000002e-06 2024-09-21T12:29:18.579 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.579 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.579 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:18.579 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215884+0000", 2024-09-21T12:29:18.579 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2599000000000003e-05 2024-09-21T12:29:18.579 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.579 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.579 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:18.579 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215975+0000", 2024-09-21T12:29:18.580 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1390000000000004e-05 2024-09-21T12:29:18.580 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.580 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.580 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:29:18.580 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.216072+0000", 2024-09-21T12:29:18.580 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6669000000000006e-05 2024-09-21T12:29:18.580 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.580 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.580 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:18.580 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217412+0000", 2024-09-21T12:29:18.581 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001340171 2024-09-21T12:29:18.581 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.581 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.581 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.581 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.219412+0000", 2024-09-21T12:29:18.581 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001999883 2024-09-21T12:29:18.581 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.581 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.581 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.581 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.389620+0000", 2024-09-21T12:29:18.581 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.17020745300000001 2024-09-21T12:29:18.581 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.582 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.582 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:18.582 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.389671+0000", 2024-09-21T12:29:18.582 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1823999999999997e-05 2024-09-21T12:29:18.582 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.582 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.582 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:18.582 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.389705+0000", 2024-09-21T12:29:18.582 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3652999999999998e-05 2024-09-21T12:29:18.582 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.582 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:18.582 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.582 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.582 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.583 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:51710 7.5 7:a2195e36:::benchmark_data_smithi114_78432_object51709:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:29:18.583 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:06.588918+0000", 2024-09-21T12:29:18.583 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 431.93620623999999, 2024-09-21T12:29:18.583 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.083421032000000006, 2024-09-21T12:29:18.583 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:18.583 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:18.583 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:18.583 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:18.583 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:29:18.583 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:29:18.583 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 51710 2024-09-21T12:29:18.583 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.583 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:18.583 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.583 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:18.584 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.588918+0000", 2024-09-21T12:29:18.584 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.584 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.584 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.584 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:18.584 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.588918+0000", 2024-09-21T12:29:18.584 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.584 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.584 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.584 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:18.584 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.588920+0000", 2024-09-21T12:29:18.584 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.751e-06 2024-09-21T12:29:18.584 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.584 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.585 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:18.585 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.588976+0000", 2024-09-21T12:29:18.585 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5767000000000003e-05 2024-09-21T12:29:18.585 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.585 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.585 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:18.585 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.588995+0000", 2024-09-21T12:29:18.585 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8833000000000001e-05 2024-09-21T12:29:18.585 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.585 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.585 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:18.585 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.589004+0000", 2024-09-21T12:29:18.585 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2320000000000007e-06 2024-09-21T12:29:18.585 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.585 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.585 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:18.585 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.589075+0000", 2024-09-21T12:29:18.586 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1706999999999996e-05 2024-09-21T12:29:18.586 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.586 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.586 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:18.586 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.589205+0000", 2024-09-21T12:29:18.586 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012908799999999999 2024-09-21T12:29:18.586 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.586 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.586 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:29:18.586 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.589363+0000", 2024-09-21T12:29:18.586 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015867999999999999 2024-09-21T12:29:18.586 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.586 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.586 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:18.587 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.590869+0000", 2024-09-21T12:29:18.587 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0015061230000000001 2024-09-21T12:29:18.587 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.587 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.587 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.587 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.592836+0000", 2024-09-21T12:29:18.587 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0019670069999999998 2024-09-21T12:29:18.587 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.587 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.587 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.587 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.672241+0000", 2024-09-21T12:29:18.587 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.079404634000000002 2024-09-21T12:29:18.587 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.588 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.588 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:18.588 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.672297+0000", 2024-09-21T12:29:18.588 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6416000000000003e-05 2024-09-21T12:29:18.588 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.588 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.588 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:18.588 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:06.672339+0000", 2024-09-21T12:29:18.588 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1794e-05 2024-09-21T12:29:18.588 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.588 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:18.588 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.588 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.588 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.588 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:76910 7.5 7:a2a1b392:::benchmark_data_smithi114_78432_object76909:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-21T12:29:18.589 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:29.843487+0000", 2024-09-21T12:29:18.589 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 408.68163783300002, 2024-09-21T12:29:18.589 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.080587959000000001, 2024-09-21T12:29:18.589 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:18.589 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:18.589 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:18.589 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:18.589 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:29:18.589 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:29:18.589 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 76910 2024-09-21T12:29:18.589 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.589 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:18.589 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.589 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:18.589 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.843487+0000", 2024-09-21T12:29:18.590 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.590 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.590 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.590 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:18.590 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.843487+0000", 2024-09-21T12:29:18.590 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.590 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.590 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.590 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:18.590 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.843488+0000", 2024-09-21T12:29:18.590 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.314e-06 2024-09-21T12:29:18.590 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.590 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.590 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:18.591 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.843533+0000", 2024-09-21T12:29:18.591 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4628e-05 2024-09-21T12:29:18.591 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.591 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.591 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:18.591 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.843542+0000", 2024-09-21T12:29:18.591 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9480000000000003e-06 2024-09-21T12:29:18.591 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.591 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.591 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:18.591 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.843548+0000", 2024-09-21T12:29:18.591 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6289999999999998e-06 2024-09-21T12:29:18.591 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.591 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.591 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:18.591 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.843590+0000", 2024-09-21T12:29:18.592 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2070000000000002e-05 2024-09-21T12:29:18.592 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.592 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.592 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:18.592 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.843681+0000", 2024-09-21T12:29:18.592 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0459999999999998e-05 2024-09-21T12:29:18.592 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.592 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.592 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:29:18.592 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.843768+0000", 2024-09-21T12:29:18.592 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7200999999999994e-05 2024-09-21T12:29:18.592 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.592 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.592 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:18.592 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.847962+0000", 2024-09-21T12:29:18.593 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0041937830000000004 2024-09-21T12:29:18.593 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.593 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.593 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.593 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.850111+0000", 2024-09-21T12:29:18.593 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021488990000000001 2024-09-21T12:29:18.593 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.593 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.593 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.593 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.923993+0000", 2024-09-21T12:29:18.593 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.073882858999999995 2024-09-21T12:29:18.593 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.593 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.593 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:18.593 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.924032+0000", 2024-09-21T12:29:18.593 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8432000000000001e-05 2024-09-21T12:29:18.594 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.594 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.594 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:18.594 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.924075+0000", 2024-09-21T12:29:18.594 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2735999999999999e-05 2024-09-21T12:29:18.594 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.594 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:18.594 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.594 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.594 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.594 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:76915 7.5 7:af00b811:::benchmark_data_smithi114_78432_object76914:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-21T12:29:18.594 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:29.845405+0000", 2024-09-21T12:29:18.594 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 408.67971956899999, 2024-09-21T12:29:18.594 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.092908165000000001, 2024-09-21T12:29:18.595 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:18.595 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:18.595 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:18.595 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:18.595 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:29:18.595 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:29:18.595 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 76915 2024-09-21T12:29:18.595 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.595 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:18.595 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.595 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:18.595 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.845405+0000", 2024-09-21T12:29:18.595 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.595 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.595 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.596 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:18.596 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.845405+0000", 2024-09-21T12:29:18.596 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.596 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.596 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.596 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:18.596 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.845406+0000", 2024-09-21T12:29:18.596 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.626e-06 2024-09-21T12:29:18.596 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.596 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.596 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:18.596 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.845465+0000", 2024-09-21T12:29:18.596 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8341000000000003e-05 2024-09-21T12:29:18.596 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.596 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.597 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:18.597 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.845482+0000", 2024-09-21T12:29:18.597 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7076999999999999e-05 2024-09-21T12:29:18.597 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.597 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.597 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:18.597 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.845491+0000", 2024-09-21T12:29:18.597 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7919999999999998e-06 2024-09-21T12:29:18.597 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.597 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.597 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:18.597 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.845687+0000", 2024-09-21T12:29:18.597 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019604900000000001 2024-09-21T12:29:18.597 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.597 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.597 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:18.597 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.845805+0000", 2024-09-21T12:29:18.598 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000117836 2024-09-21T12:29:18.598 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.598 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.598 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:29:18.598 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.845934+0000", 2024-09-21T12:29:18.598 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012931600000000001 2024-09-21T12:29:18.598 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.598 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.598 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:18.598 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.848128+0000", 2024-09-21T12:29:18.598 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021943330000000001 2024-09-21T12:29:18.598 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.598 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.599 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.599 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.850159+0000", 2024-09-21T12:29:18.599 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020305369999999998 2024-09-21T12:29:18.599 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.599 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.599 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.599 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.938214+0000", 2024-09-21T12:29:18.599 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.088055156999999995 2024-09-21T12:29:18.599 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.599 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.599 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:18.599 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.938271+0000", 2024-09-21T12:29:18.599 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6588000000000001e-05 2024-09-21T12:29:18.599 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.599 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.599 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:18.600 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:29.938313+0000", 2024-09-21T12:29:18.600 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2512999999999998e-05 2024-09-21T12:29:18.600 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.600 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:18.600 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.600 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.600 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.600 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:150335 7.5 7:aef4e123:::benchmark_data_smithi114_78432_object46189:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T12:29:18.600 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:23:42.614761+0000", 2024-09-21T12:29:18.600 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 335.910363405, 2024-09-21T12:29:18.600 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13488366900000001, 2024-09-21T12:29:18.600 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:18.600 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:18.600 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:18.600 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:18.600 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:29:18.601 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:29:18.601 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 150335 2024-09-21T12:29:18.601 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.601 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:18.601 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.601 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:18.601 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.614761+0000", 2024-09-21T12:29:18.601 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.601 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.601 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.601 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:18.601 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.614761+0000", 2024-09-21T12:29:18.601 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.601 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.601 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.601 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:18.602 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.614763+0000", 2024-09-21T12:29:18.602 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6500000000000001e-06 2024-09-21T12:29:18.602 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.602 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.602 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:18.602 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.614768+0000", 2024-09-21T12:29:18.602 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.101e-06 2024-09-21T12:29:18.602 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.602 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.602 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:18.602 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.614781+0000", 2024-09-21T12:29:18.602 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3572e-05 2024-09-21T12:29:18.602 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.602 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.602 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:18.603 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.614788+0000", 2024-09-21T12:29:18.603 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7279999999999998e-06 2024-09-21T12:29:18.603 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.603 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.603 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:18.603 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.614853+0000", 2024-09-21T12:29:18.603 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4647999999999995e-05 2024-09-21T12:29:18.603 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.603 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.603 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:18.603 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.614963+0000", 2024-09-21T12:29:18.603 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011069399999999999 2024-09-21T12:29:18.603 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.603 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.603 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:29:18.603 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.615070+0000", 2024-09-21T12:29:18.604 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010666999999999999 2024-09-21T12:29:18.604 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.604 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.604 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:18.604 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.617101+0000", 2024-09-21T12:29:18.604 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020314040000000001 2024-09-21T12:29:18.604 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.604 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.604 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.604 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.617869+0000", 2024-09-21T12:29:18.604 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00076795400000000005 2024-09-21T12:29:18.604 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.604 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.604 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.604 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.749524+0000", 2024-09-21T12:29:18.605 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.131655037 2024-09-21T12:29:18.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.605 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.605 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:18.605 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.749590+0000", 2024-09-21T12:29:18.605 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5864999999999994e-05 2024-09-21T12:29:18.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.605 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.605 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:18.605 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.749645+0000", 2024-09-21T12:29:18.605 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4345999999999998e-05 2024-09-21T12:29:18.605 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.605 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:18.605 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.606 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.606 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:150337 7.5 7:aae8cbea:::benchmark_data_smithi114_78432_object46191:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T12:29:18.606 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:23:42.617985+0000", 2024-09-21T12:29:18.606 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 335.90713917699998, 2024-09-21T12:29:18.606 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13997900399999999, 2024-09-21T12:29:18.606 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:18.606 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:18.606 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:18.606 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:18.606 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:29:18.606 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:29:18.606 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 150337 2024-09-21T12:29:18.606 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.606 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:18.606 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.607 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:18.607 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.617985+0000", 2024-09-21T12:29:18.607 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.607 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.607 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:18.607 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.617985+0000", 2024-09-21T12:29:18.607 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.607 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.607 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:18.607 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.617986+0000", 2024-09-21T12:29:18.607 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1289999999999999e-06 2024-09-21T12:29:18.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.607 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.607 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:18.608 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.617991+0000", 2024-09-21T12:29:18.608 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5870000000000003e-06 2024-09-21T12:29:18.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.608 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:18.608 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618001+0000", 2024-09-21T12:29:18.608 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0370000000000001e-05 2024-09-21T12:29:18.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.608 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:18.608 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618007+0000", 2024-09-21T12:29:18.608 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.215e-06 2024-09-21T12:29:18.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.609 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:18.609 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618052+0000", 2024-09-21T12:29:18.609 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5525999999999997e-05 2024-09-21T12:29:18.609 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.609 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.609 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:18.609 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618122+0000", 2024-09-21T12:29:18.609 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9733000000000005e-05 2024-09-21T12:29:18.609 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.609 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.609 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:29:18.609 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618190+0000", 2024-09-21T12:29:18.609 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8684999999999997e-05 2024-09-21T12:29:18.609 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.609 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.610 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:18.610 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.620946+0000", 2024-09-21T12:29:18.610 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002755965 2024-09-21T12:29:18.610 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.610 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.610 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.610 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.621448+0000", 2024-09-21T12:29:18.610 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00050168700000000001 2024-09-21T12:29:18.610 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.610 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.610 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.610 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.757862+0000", 2024-09-21T12:29:18.610 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13641423799999999 2024-09-21T12:29:18.611 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.611 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.611 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:18.611 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.757920+0000", 2024-09-21T12:29:18.611 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7630999999999998e-05 2024-09-21T12:29:18.611 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.611 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.611 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:18.611 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.757964+0000", 2024-09-21T12:29:18.611 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4237999999999999e-05 2024-09-21T12:29:18.611 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.611 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:18.611 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.611 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.612 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.612 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:150339 7.5 7:ac9cca71:::benchmark_data_smithi114_78432_object46193:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T12:29:18.612 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:23:42.618551+0000", 2024-09-21T12:29:18.612 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 335.90657309099998, 2024-09-21T12:29:18.612 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.139580281, 2024-09-21T12:29:18.612 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:18.612 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:18.612 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:18.612 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:18.612 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:29:18.612 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:29:18.612 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 150339 2024-09-21T12:29:18.612 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.612 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:18.612 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.612 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:18.613 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618551+0000", 2024-09-21T12:29:18.613 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.613 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.613 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.613 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:18.613 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618551+0000", 2024-09-21T12:29:18.613 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.613 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.613 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.613 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:18.613 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618552+0000", 2024-09-21T12:29:18.613 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8200000000000008e-07 2024-09-21T12:29:18.613 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.613 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.613 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:18.614 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618557+0000", 2024-09-21T12:29:18.614 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3560000000000003e-06 2024-09-21T12:29:18.614 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.614 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.614 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:18.614 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618570+0000", 2024-09-21T12:29:18.614 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3118e-05 2024-09-21T12:29:18.614 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.614 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.614 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:18.614 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.618577+0000", 2024-09-21T12:29:18.614 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9149999999999999e-06 2024-09-21T12:29:18.614 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.614 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.615 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:18.615 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.619781+0000", 2024-09-21T12:29:18.615 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012041790000000001 2024-09-21T12:29:18.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.615 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.615 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:18.615 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.619857+0000", 2024-09-21T12:29:18.615 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6151999999999999e-05 2024-09-21T12:29:18.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.615 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.615 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,2", 2024-09-21T12:29:18.615 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.619919+0000", 2024-09-21T12:29:18.615 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.19e-05 2024-09-21T12:29:18.616 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.616 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.616 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:18.616 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.621027+0000", 2024-09-21T12:29:18.616 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001108211 2024-09-21T12:29:18.616 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.616 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.616 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.616 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.621491+0000", 2024-09-21T12:29:18.616 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00046415200000000001 2024-09-21T12:29:18.616 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.616 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.616 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.616 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.758040+0000", 2024-09-21T12:29:18.616 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13654828999999999 2024-09-21T12:29:18.616 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.617 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.617 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:18.617 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.758095+0000", 2024-09-21T12:29:18.617 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5386999999999997e-05 2024-09-21T12:29:18.617 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.617 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.617 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:18.617 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.758132+0000", 2024-09-21T12:29:18.617 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6638999999999999e-05 2024-09-21T12:29:18.617 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.617 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:18.617 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.617 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.617 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.617 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.25022.0:226447 8.c e223/160 8:3c15209f:::benchmark_data_smithi114_104936_object96031:head v 223'14226, mlcod=223'14221)", 2024-09-21T12:29:18.618 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:27:37.566833+0000", 2024-09-21T12:29:18.618 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 100.95829146200001, 2024-09-21T12:29:18.618 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.085894057999999995, 2024-09-21T12:29:18.618 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:18.618 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:18.618 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:18.618 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:18.618 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.618 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:18.618 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:37.566833+0000", 2024-09-21T12:29:18.618 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.618 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.618 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.618 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:18.618 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:37.566833+0000", 2024-09-21T12:29:18.619 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.619 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.619 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.619 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:18.619 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:18.620 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:37.566834+0000", 2024-09-21T12:29:18.620 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.09e-07 2024-09-21T12:29:18.620 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.620 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.620 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:18.620 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:37.566839+0000", 2024-09-21T12:29:18.620 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3349999999999997e-06 2024-09-21T12:29:18.620 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.620 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.620 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:18.620 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:37.566847+0000", 2024-09-21T12:29:18.620 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3389999999999993e-06 2024-09-21T12:29:18.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.621 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:18.621 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:37.566852+0000", 2024-09-21T12:29:18.621 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4240000000000003e-06 2024-09-21T12:29:18.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.621 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:18.621 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:37.566894+0000", 2024-09-21T12:29:18.621 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2265000000000003e-05 2024-09-21T12:29:18.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.621 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:18.621 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:37.566911+0000", 2024-09-21T12:29:18.621 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6711e-05 2024-09-21T12:29:18.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.622 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.622 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:18.622 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:37.652693+0000", 2024-09-21T12:29:18.622 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.085781868999999997 2024-09-21T12:29:18.622 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.622 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.622 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:18.622 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:37.652727+0000", 2024-09-21T12:29:18.622 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4306e-05 2024-09-21T12:29:18.622 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.622 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:18.622 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.622 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.622 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.623 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.25022.0:226453 8.e e223/160 8:79605493:::benchmark_data_smithi114_104936_object96037:head v 223'14003, mlcod=223'14001)", 2024-09-21T12:29:18.623 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:27:37.567459+0000", 2024-09-21T12:29:18.623 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 100.957665507, 2024-09-21T12:29:18.623 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.081204598000000003, 2024-09-21T12:29:18.623 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:18.623 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:18.623 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:18.623 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:18.623 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.623 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:18.623 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:37.567459+0000", 2024-09-21T12:29:18.623 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.623 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.623 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.623 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:18.623 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:37.567459+0000", 2024-09-21T12:29:18.624 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.624 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:18.624 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:37.567460+0000", 2024-09-21T12:29:18.624 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7800000000000001e-07 2024-09-21T12:29:18.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.624 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:18.624 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:37.567464+0000", 2024-09-21T12:29:18.624 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4850000000000003e-06 2024-09-21T12:29:18.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.624 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:18.624 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:37.644866+0000", 2024-09-21T12:29:18.625 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.077402084999999995 2024-09-21T12:29:18.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:18.625 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:37.644872+0000", 2024-09-21T12:29:18.625 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1940000000000003e-06 2024-09-21T12:29:18.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:18.625 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:37.644925+0000", 2024-09-21T12:29:18.625 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3156999999999999e-05 2024-09-21T12:29:18.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:18.625 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:37.644947+0000", 2024-09-21T12:29:18.626 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1826000000000001e-05 2024-09-21T12:29:18.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.626 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:18.626 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:37.648627+0000", 2024-09-21T12:29:18.626 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0036798439999999998 2024-09-21T12:29:18.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.626 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:18.626 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:37.648664+0000", 2024-09-21T12:29:18.626 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6429e-05 2024-09-21T12:29:18.626 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.626 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:18.626 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.627 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:30136 9.7 9:e103d6f8:::benchmark_data_smithi114_133214_object30135:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-09-21T12:29:18.627 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:28:39.740283+0000", 2024-09-21T12:29:18.627 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 38.784841292000003, 2024-09-21T12:29:18.627 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.098402304999999995, 2024-09-21T12:29:18.627 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:18.627 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:18.627 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:18.627 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:18.627 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:29:18.627 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:29:18.627 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 30136 2024-09-21T12:29:18.627 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.627 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:18.627 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.627 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:18.628 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:39.740283+0000", 2024-09-21T12:29:18.628 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.628 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.628 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.628 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:18.628 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:39.740283+0000", 2024-09-21T12:29:18.628 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.628 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.628 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.628 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:18.628 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:39.740284+0000", 2024-09-21T12:29:18.628 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7400000000000002e-07 2024-09-21T12:29:18.628 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.628 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.628 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:18.628 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:39.740340+0000", 2024-09-21T12:29:18.629 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5843e-05 2024-09-21T12:29:18.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.629 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.629 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:18.629 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:39.740350+0000", 2024-09-21T12:29:18.629 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0550000000000001e-05 2024-09-21T12:29:18.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.629 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.629 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:18.629 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:39.740356+0000", 2024-09-21T12:29:18.629 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.259e-06 2024-09-21T12:29:18.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.629 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.629 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:18.629 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:39.740405+0000", 2024-09-21T12:29:18.630 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9388000000000002e-05 2024-09-21T12:29:18.630 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.630 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.630 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:18.630 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:39.740534+0000", 2024-09-21T12:29:18.630 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012858800000000001 2024-09-21T12:29:18.630 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.630 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.630 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,5", 2024-09-21T12:29:18.630 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:39.740678+0000", 2024-09-21T12:29:18.630 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014396600000000001 2024-09-21T12:29:18.630 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.630 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.630 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:18.630 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:39.742004+0000", 2024-09-21T12:29:18.630 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001326299 2024-09-21T12:29:18.631 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.631 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.631 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.631 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:39.742273+0000", 2024-09-21T12:29:18.631 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000269425 2024-09-21T12:29:18.631 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.631 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.631 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.631 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:39.838620+0000", 2024-09-21T12:29:18.631 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.096346984999999996 2024-09-21T12:29:18.631 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.631 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.631 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:18.631 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:39.838657+0000", 2024-09-21T12:29:18.632 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6715000000000003e-05 2024-09-21T12:29:18.632 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.632 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.632 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:18.632 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:39.838685+0000", 2024-09-21T12:29:18.632 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8413000000000001e-05 2024-09-21T12:29:18.632 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.632 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:18.632 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.632 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.632 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.632 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:30146 9.3 9:cc7b2e5c:::benchmark_data_smithi114_133214_object30145:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-09-21T12:29:18.632 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:28:39.745028+0000", 2024-09-21T12:29:18.632 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 38.780096329999999, 2024-09-21T12:29:18.632 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.095269453000000004, 2024-09-21T12:29:18.633 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:18.633 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:18.633 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:18.633 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:18.633 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:29:18.633 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:29:18.633 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 30146 2024-09-21T12:29:18.633 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.633 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:18.633 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.633 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:18.633 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:39.745028+0000", 2024-09-21T12:29:18.633 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.633 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.633 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.634 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:18.634 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:39.745028+0000", 2024-09-21T12:29:18.634 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.634 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.634 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:18.634 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:39.745030+0000", 2024-09-21T12:29:18.634 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.539e-06 2024-09-21T12:29:18.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.634 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.634 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:18.634 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:39.745136+0000", 2024-09-21T12:29:18.634 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010596599999999999 2024-09-21T12:29:18.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.634 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.635 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:18.635 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:39.745147+0000", 2024-09-21T12:29:18.635 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1429e-05 2024-09-21T12:29:18.635 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.635 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.635 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:18.635 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:39.745154+0000", 2024-09-21T12:29:18.635 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2660000000000004e-06 2024-09-21T12:29:18.635 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.635 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.635 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:18.635 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:39.745215+0000", 2024-09-21T12:29:18.635 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0291999999999998e-05 2024-09-21T12:29:18.635 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.636 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:18.636 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:39.745493+0000", 2024-09-21T12:29:18.636 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000278421 2024-09-21T12:29:18.636 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.636 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:29:18.636 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:39.745617+0000", 2024-09-21T12:29:18.636 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000124077 2024-09-21T12:29:18.636 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.636 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:18.636 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:39.747041+0000", 2024-09-21T12:29:18.636 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001423899 2024-09-21T12:29:18.636 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.637 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.637 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:39.747290+0000", 2024-09-21T12:29:18.637 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00024919500000000002 2024-09-21T12:29:18.637 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.637 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.637 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.637 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:39.840239+0000", 2024-09-21T12:29:18.637 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.092948723999999996 2024-09-21T12:29:18.637 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.637 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.637 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:18.637 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:39.840272+0000", 2024-09-21T12:29:18.637 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3068999999999998e-05 2024-09-21T12:29:18.637 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.637 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.637 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:18.638 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:39.840298+0000", 2024-09-21T12:29:18.638 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5576e-05 2024-09-21T12:29:18.638 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.638 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:18.638 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.638 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.638 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.638 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:32590 9.7 9:e5f70adc:::benchmark_data_smithi114_133214_object32589:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-09-21T12:29:18.638 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:28:41.038324+0000", 2024-09-21T12:29:18.638 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 37.486800551999998, 2024-09-21T12:29:18.638 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.090660773, 2024-09-21T12:29:18.638 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:29:18.638 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:29:18.638 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:29:18.638 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:29:18.638 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:29:18.639 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:29:18.639 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 32590 2024-09-21T12:29:18.639 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.639 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:29:18.639 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.639 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:29:18.639 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:41.038324+0000", 2024-09-21T12:29:18.639 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.639 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.639 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.639 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:29:18.639 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:41.038324+0000", 2024-09-21T12:29:18.639 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:29:18.639 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.639 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.640 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:29:18.640 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:41.038325+0000", 2024-09-21T12:29:18.640 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.206e-06 2024-09-21T12:29:18.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.640 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:29:18.640 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:41.038372+0000", 2024-09-21T12:29:18.640 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6560000000000001e-05 2024-09-21T12:29:18.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.640 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:29:18.640 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:41.038382+0000", 2024-09-21T12:29:18.640 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0695e-05 2024-09-21T12:29:18.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.641 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:29:18.641 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:41.038388+0000", 2024-09-21T12:29:18.641 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.587e-06 2024-09-21T12:29:18.641 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.641 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.641 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:29:18.641 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:41.038441+0000", 2024-09-21T12:29:18.641 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2812000000000002e-05 2024-09-21T12:29:18.641 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.641 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.641 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:29:18.641 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:41.038566+0000", 2024-09-21T12:29:18.641 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012509700000000001 2024-09-21T12:29:18.641 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.641 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.641 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,5", 2024-09-21T12:29:18.642 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:41.038704+0000", 2024-09-21T12:29:18.642 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000138424 2024-09-21T12:29:18.642 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.642 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.642 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:29:18.642 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:41.039766+0000", 2024-09-21T12:29:18.642 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010619710000000001 2024-09-21T12:29:18.642 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.642 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.642 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.642 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:41.040147+0000", 2024-09-21T12:29:18.642 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000381197 2024-09-21T12:29:18.642 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.642 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.642 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:18.643 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:41.128878+0000", 2024-09-21T12:29:18.643 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.088730299999999998 2024-09-21T12:29:18.643 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.643 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.643 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:29:18.643 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:41.128955+0000", 2024-09-21T12:29:18.643 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7405999999999997e-05 2024-09-21T12:29:18.643 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:29:18.643 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:29:18.643 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:29:18.643 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:28:41.128985+0000", 2024-09-21T12:29:18.643 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9518000000000001e-05 2024-09-21T12:29:18.643 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.643 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:18.643 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.643 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:29:18.644 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:29:18.644 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:29:18.672 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:18.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:18.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:18.774 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:18.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:18.797 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:18.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:18.833 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 51 16 86672 86656 106.177 81.0625 0.00301528 0.00940396 2024-09-21T12:29:18.898 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:18.936 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:18.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:18.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:19.038 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:19.065 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:19.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:19.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:19.166 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:19.204 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:19.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:19.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:19.306 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:19.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:19.346 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:19.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:19.447 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:19.453 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:19 smithi183 ceph-mon[33277]: pgmap v443: 178 pgs: 178 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 106 MiB/s wr, 1.70k op/s 2024-09-21T12:29:19.453 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:19 smithi183 ceph-mon[33277]: from='client.15645 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:29:19.484 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:19.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:19.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:19.505 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T12:29:19.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:19 smithi114 ceph-mon[40867]: pgmap v443: 178 pgs: 178 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 106 MiB/s wr, 1.70k op/s 2024-09-21T12:29:19.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:19 smithi114 ceph-mon[40867]: from='client.15645 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:29:19.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:19 smithi114 ceph-mon[30324]: pgmap v443: 178 pgs: 178 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 106 MiB/s wr, 1.70k op/s 2024-09-21T12:29:19.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:19 smithi114 ceph-mon[30324]: from='client.15645 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:29:19.587 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:19.612 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:19.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:19.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:19.714 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:19.741 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:19.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:19.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:19.835 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 52 16 87858 87842 105.559 74.125 0.0093694 0.00946872 2024-09-21T12:29:19.843 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:19.870 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:19.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:19.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:19.972 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:20.015 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:20.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:20.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:20.121 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:20.159 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:20.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:20.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:20.261 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:20.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:20.295 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:20.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:20.396 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:20.440 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:20.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:20.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:20.542 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:20.569 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:20.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:20.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:20.671 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:20.703 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:20.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:20.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:20.805 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:20.832 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:20.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:20.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:20.834 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 53 16 89046 89030 104.968 74.25 0.0124407 0.0095218 2024-09-21T12:29:20.934 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:20.959 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:20.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:20.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:21.061 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:21.087 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:21.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:21.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:21.188 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:21.191 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:21 smithi114 ceph-mon[30324]: pgmap v444: 178 pgs: 178 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-21T12:29:21.192 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:21 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:29:21.192 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:21 smithi114 ceph-mon[40867]: pgmap v444: 178 pgs: 178 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-21T12:29:21.192 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:21 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:29:21.217 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:21.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:21.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:21.319 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:21.359 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:21.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:21.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:21.461 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:21.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:21 smithi183 ceph-mon[33277]: pgmap v444: 178 pgs: 178 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-21T12:29:21.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:21 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:29:21.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:21.496 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:21.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:21.583 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-21T12:29:21.583 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 1000 0.05 2024-09-21T12:29:21.598 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:21.610 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:21.625 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:21.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:21.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:21.673 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T12:29:21.727 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:21.752 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:21.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:21.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:21.835 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 54 16 90212 90196 104.374 72.875 0.0120899 0.00957601 2024-09-21T12:29:21.853 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:21.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:21.913 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:21.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:22.014 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:22.063 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:22 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:29:22.063 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:22 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:29:22.063 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:22 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:29:22.063 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:22 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:29:22.063 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:22 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:29:22.064 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:22 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:29:22.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:22.064 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:22.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:22.165 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:22.193 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:22.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:22.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:22.294 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:22.303 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:29:22.303 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-21T12:29:22.304 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.304 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:90736 9.b 9:d19b4b17:::benchmark_data_smithi114_133214_object90735:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-21T12:29:22.304 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:29:22.290402+0000", 2024-09-21T12:29:22.304 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0063616999999999996, 2024-09-21T12:29:22.304 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0063968510000000003, 2024-09-21T12:29:22.304 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:29:22.304 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:29:22.304 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:29:22.304 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:29:22.304 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:29:22.304 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:29:22.304 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 90736 2024-09-21T12:29:22.304 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.305 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:29:22.305 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.305 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:29:22.305 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.290402+0000", 2024-09-21T12:29:22.305 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:29:22.305 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.305 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.305 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:29:22.305 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.290402+0000", 2024-09-21T12:29:22.305 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:29:22.305 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.305 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.305 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:29:22.305 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.290403+0000", 2024-09-21T12:29:22.306 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.573e-06 2024-09-21T12:29:22.306 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.306 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.306 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:29:22.306 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.290426+0000", 2024-09-21T12:29:22.306 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2449000000000001e-05 2024-09-21T12:29:22.306 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.306 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.306 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:29:22.306 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.290440+0000", 2024-09-21T12:29:22.306 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4572e-05 2024-09-21T12:29:22.306 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.306 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.306 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:29:22.307 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.290448+0000", 2024-09-21T12:29:22.307 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7309999999999993e-06 2024-09-21T12:29:22.307 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.307 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.307 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:29:22.307 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.290501+0000", 2024-09-21T12:29:22.307 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2855000000000001e-05 2024-09-21T12:29:22.307 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.307 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.307 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:29:22.307 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.290642+0000", 2024-09-21T12:29:22.307 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000140641 2024-09-21T12:29:22.307 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.307 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.308 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 5,6", 2024-09-21T12:29:22.308 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.290795+0000", 2024-09-21T12:29:22.308 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015336700000000001 2024-09-21T12:29:22.308 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.308 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.308 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:22.308 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.293281+0000", 2024-09-21T12:29:22.308 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002485892 2024-09-21T12:29:22.308 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.308 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.308 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:29:22.308 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.293366+0000", 2024-09-21T12:29:22.308 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5445000000000006e-05 2024-09-21T12:29:22.308 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.308 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.309 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:29:22.309 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.296792+0000", 2024-09-21T12:29:22.309 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0034258719999999999 2024-09-21T12:29:22.309 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:29:22.309 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:29:22.309 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:29:22.309 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.309 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.309 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:90742 9.9 9:9bfb89cc:::benchmark_data_smithi114_133214_object90741:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-21T12:29:22.309 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:29:22.293954+0000", 2024-09-21T12:29:22.309 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0028094470000000001, 2024-09-21T12:29:22.309 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0029399220000000002, 2024-09-21T12:29:22.309 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:29:22.309 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:29:22.309 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:29:22.310 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:29:22.310 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:29:22.310 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:29:22.310 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 90742 2024-09-21T12:29:22.310 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.310 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:29:22.310 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.310 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:29:22.310 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.293954+0000", 2024-09-21T12:29:22.310 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:29:22.310 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.310 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.310 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:29:22.310 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.293954+0000", 2024-09-21T12:29:22.311 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:29:22.311 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.311 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.311 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:29:22.311 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.293956+0000", 2024-09-21T12:29:22.311 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.559e-06 2024-09-21T12:29:22.311 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.311 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.311 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:29:22.311 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.293989+0000", 2024-09-21T12:29:22.311 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.3105000000000002e-05 2024-09-21T12:29:22.311 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.311 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.311 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:29:22.311 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.294003+0000", 2024-09-21T12:29:22.312 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4270000000000001e-05 2024-09-21T12:29:22.312 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.312 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.312 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:29:22.312 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.294011+0000", 2024-09-21T12:29:22.312 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6469999999999998e-06 2024-09-21T12:29:22.312 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.312 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.312 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:29:22.312 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.296503+0000", 2024-09-21T12:29:22.312 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0024928149999999998 2024-09-21T12:29:22.312 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.312 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.312 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:29:22.312 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.296655+0000", 2024-09-21T12:29:22.313 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015186099999999999 2024-09-21T12:29:22.313 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.313 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.313 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 5,7", 2024-09-21T12:29:22.313 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.296804+0000", 2024-09-21T12:29:22.313 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014880199999999999 2024-09-21T12:29:22.313 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:29:22.313 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:29:22.313 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:29:22.313 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.313 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.313 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.25436.0:90733 9.8 e259/236)", 2024-09-21T12:29:22.313 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:29:22.296534+0000", 2024-09-21T12:29:22.313 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.00022910500000000001, 2024-09-21T12:29:22.313 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00042082300000000002, 2024-09-21T12:29:22.314 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:29:22.314 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:29:22.314 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "queued for pg", 2024-09-21T12:29:22.314 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:29:22.314 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.314 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:29:22.314 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.296534+0000", 2024-09-21T12:29:22.314 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:29:22.314 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.314 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.314 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:29:22.314 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.296534+0000", 2024-09-21T12:29:22.314 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:29:22.314 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.314 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.315 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:29:22.315 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.296536+0000", 2024-09-21T12:29:22.315 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.257e-06 2024-09-21T12:29:22.315 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.315 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.315 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:29:22.315 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.296566+0000", 2024-09-21T12:29:22.315 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.9955999999999999e-05 2024-09-21T12:29:22.315 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.315 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.315 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:29:22.315 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.296577+0000", 2024-09-21T12:29:22.315 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.19e-05 2024-09-21T12:29:22.315 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.315 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.316 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:29:22.316 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.296584+0000", 2024-09-21T12:29:22.316 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9330000000000004e-06 2024-09-21T12:29:22.316 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:29:22.316 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:29:22.316 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:29:22.316 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.316 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.316 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.25436.0:90738 9.c e259/236)", 2024-09-21T12:29:22.316 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:29:22.296607+0000", 2024-09-21T12:29:22.316 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.00015626299999999999, 2024-09-21T12:29:22.316 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000385666, 2024-09-21T12:29:22.316 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:29:22.317 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:29:22.317 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "queued for pg", 2024-09-21T12:29:22.317 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:29:22.317 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.317 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:29:22.317 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.296607+0000", 2024-09-21T12:29:22.317 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:29:22.317 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.317 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.317 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:29:22.317 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.296607+0000", 2024-09-21T12:29:22.317 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:29:22.317 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.317 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.317 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:29:22.318 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.296608+0000", 2024-09-21T12:29:22.318 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4499999999999998e-07 2024-09-21T12:29:22.318 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.318 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.318 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:29:22.318 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.296633+0000", 2024-09-21T12:29:22.318 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5020000000000001e-05 2024-09-21T12:29:22.318 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.318 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.318 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:29:22.318 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.296639+0000", 2024-09-21T12:29:22.318 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5989999999999998e-06 2024-09-21T12:29:22.318 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.318 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.319 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:29:22.319 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.296644+0000", 2024-09-21T12:29:22.319 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8239999999999999e-06 2024-09-21T12:29:22.319 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:29:22.319 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:29:22.319 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:29:22.319 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.319 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.319 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop_reply(client.25436.0:90736 9.b e259/236 ondisk, result = 0)", 2024-09-21T12:29:22.319 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:29:22.296669+0000", 2024-09-21T12:29:22.319 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 9.4714000000000002e-05, 2024-09-21T12:29:22.319 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00038659799999999999, 2024-09-21T12:29:22.319 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:29:22.320 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:29:22.320 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:29:22.320 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:29:22.320 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.320 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:29:22.320 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.296669+0000", 2024-09-21T12:29:22.320 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:29:22.320 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.320 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.320 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:29:22.320 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.296669+0000", 2024-09-21T12:29:22.320 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:29:22.320 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.320 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.321 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:29:22.321 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.296669+0000", 2024-09-21T12:29:22.321 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5600000000000001e-07 2024-09-21T12:29:22.321 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.321 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.321 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:29:22.321 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.296672+0000", 2024-09-21T12:29:22.321 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.123e-06 2024-09-21T12:29:22.321 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.321 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.321 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:29:22.321 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.296684+0000", 2024-09-21T12:29:22.321 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1559000000000001e-05 2024-09-21T12:29:22.321 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.321 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.322 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:29:22.322 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.296689+0000", 2024-09-21T12:29:22.322 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9969999999999998e-06 2024-09-21T12:29:22.322 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.322 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.322 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:29:22.322 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.296777+0000", 2024-09-21T12:29:22.322 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.8480000000000007e-05 2024-09-21T12:29:22.322 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:29:22.322 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:29:22.322 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:29:22.322 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:29:22.296786+0000", 2024-09-21T12:29:22.322 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5129999999999996e-06 2024-09-21T12:29:22.322 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:29:22.323 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:29:22.323 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:29:22.323 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:29:22.323 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:29:22.323 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 5 2024-09-21T12:29:22.323 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:29:22.323 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:22.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:22.324 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:22.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:22.326 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:22.425 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:22.456 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:22.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:22.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:22.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:22 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:29:22.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:22 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:29:22.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:22 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:29:22.558 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:22.595 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:22.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:22.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:22.698 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:22.742 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:22.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:22.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:22.775 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:29:22 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:29:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:29:22.834 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 55 16 91509 91493 103.95 81.0625 0.0166503 0.00961533 2024-09-21T12:29:22.844 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:22.879 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:29:22.881 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:22.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:22.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:22.984 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:23.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:23.013 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:23.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:23.114 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:23.140 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:23.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:23.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:23.244 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:23.273 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:23.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:23.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:23.354 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:23 smithi114 ceph-mon[30324]: pgmap v445: 178 pgs: 178 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 524 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-21T12:29:23.354 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:23 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:29:23.354 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:23 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:29:23.354 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:23 smithi114 ceph-mon[40867]: pgmap v445: 178 pgs: 178 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 524 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-21T12:29:23.354 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:23 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:29:23.354 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:23 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:29:23.356 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T12:29:23.375 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:23.400 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:23.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:23.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:23.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:23 smithi183 ceph-mon[33277]: pgmap v445: 178 pgs: 178 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 524 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-21T12:29:23.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:23 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:29:23.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:23 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:29:23.503 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:23.531 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:23.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:23.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:23.633 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:23.658 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:23.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:23.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:23.760 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:23.808 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:23.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:23.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:23.835 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 56 16 92834 92818 103.572 82.8125 0.00502148 0.00965078 2024-09-21T12:29:23.909 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:23.938 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:23.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:23.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:24.040 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:24.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:24.098 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:24.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:24.198 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:24.233 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:24 smithi114 ceph-mon[30324]: osdmap e260: 8 total, 8 up, 7 in 2024-09-21T12:29:24.233 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:24 smithi114 ceph-mon[40867]: osdmap e260: 8 total, 8 up, 7 in 2024-09-21T12:29:24.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:24.245 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:24.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:24.346 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:24.380 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:24.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:24.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:24.482 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:24.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:24 smithi183 ceph-mon[33277]: osdmap e260: 8 total, 8 up, 7 in 2024-09-21T12:29:24.510 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:24.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:24.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:24.611 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:24.642 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:24.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:24.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:24.743 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:24.749 INFO:teuthology.orchestra.run.smithi114.stdout:moved 0 / 534 (0%) 2024-09-21T12:29:24.749 INFO:teuthology.orchestra.run.smithi114.stdout:avg 76.2857 2024-09-21T12:29:24.749 INFO:teuthology.orchestra.run.smithi114.stdout:stddev 7.08548 -> 7.08548 (expected baseline 8.08627) 2024-09-21T12:29:24.749 INFO:teuthology.orchestra.run.smithi114.stdout:min osd.2 with 62 -> 62 pgs (0.812734 -> 0.812734 * mean) 2024-09-21T12:29:24.749 INFO:teuthology.orchestra.run.smithi114.stdout:max osd.1 with 83 -> 83 pgs (1.08801 -> 1.08801 * mean) 2024-09-21T12:29:24.750 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:29:24.750 INFO:teuthology.orchestra.run.smithi114.stdout:oload 1000 2024-09-21T12:29:24.750 INFO:teuthology.orchestra.run.smithi114.stdout:max_change 0.05 2024-09-21T12:29:24.750 INFO:teuthology.orchestra.run.smithi114.stdout:max_change_osds 4 2024-09-21T12:29:24.750 INFO:teuthology.orchestra.run.smithi114.stdout:average_utilization 0.0301 2024-09-21T12:29:24.750 INFO:teuthology.orchestra.run.smithi114.stdout:overload_utilization 0.3007 2024-09-21T12:29:24.750 INFO:teuthology.orchestra.run.smithi114.stderr:no change 2024-09-21T12:29:24.770 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:24.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:24.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:24.835 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 57 15 94047 94032 103.086 75.875 0.0081804 0.00969616 2024-09-21T12:29:24.871 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:24.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:24.898 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:24.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:24.999 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:25.029 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:25.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:25.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:25.110 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-21T12:29:25.131 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:25.152 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:25.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:25.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:25.254 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:25.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:25.283 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:25.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:25.383 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:25.423 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:25.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:25.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:25.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:25 smithi183 ceph-mon[33277]: pgmap v447: 178 pgs: 178 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-21T12:29:25.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:25 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2952321808' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:29:25.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:25 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:29:25.525 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:25.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:25 smithi114 ceph-mon[30324]: pgmap v447: 178 pgs: 178 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-21T12:29:25.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:25 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2952321808' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:29:25.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:25 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:29:25.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:25 smithi114 ceph-mon[40867]: pgmap v447: 178 pgs: 178 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-21T12:29:25.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:25 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2952321808' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:29:25.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:25 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:29:25.558 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:25.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:25.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:25.660 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:25.695 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:25.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:25.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:25.797 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:25.827 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:25.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:25.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:25.835 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 58 16 95253 95237 102.607 75.3125 0.00834879 0.00974135 2024-09-21T12:29:25.929 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:25.956 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:25.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:25.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:26.057 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:26.078 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:26.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:26.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:26.180 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:26.208 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:26.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:26.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:26.310 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:26.341 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:26.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:26.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:26.442 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:26.468 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:26.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:26.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:26.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:26 smithi183 ceph-mon[33277]: from='client.25541 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:29:26.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:26 smithi183 ceph-mon[33277]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:29:26.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:26 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:29:26.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:26 smithi183 ceph-mon[33277]: osdmap e261: 8 total, 8 up, 7 in 2024-09-21T12:29:26.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:26 smithi114 ceph-mon[40867]: from='client.25541 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:29:26.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:26 smithi114 ceph-mon[40867]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:29:26.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:26 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:29:26.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:26 smithi114 ceph-mon[40867]: osdmap e261: 8 total, 8 up, 7 in 2024-09-21T12:29:26.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:26 smithi114 ceph-mon[30324]: from='client.25541 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:29:26.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:26 smithi114 ceph-mon[30324]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:29:26.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:26 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:29:26.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:26 smithi114 ceph-mon[30324]: osdmap e261: 8 total, 8 up, 7 in 2024-09-21T12:29:26.569 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:26.599 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:26.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:26.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:26.700 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:26.721 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:26.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:26.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:26.822 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:26.835 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 59 16 96299 96283 101.976 65.375 0.00295403 0.00979786 2024-09-21T12:29:26.850 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:26.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:26.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:26.952 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:26.987 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:26.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:26.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:27.089 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:27.118 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:27.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:27.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:27.220 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:27.257 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:27.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:27.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:27.358 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:27.383 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:27.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:27.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:27.485 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:27.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:27 smithi183 ceph-mon[33277]: pgmap v449: 178 pgs: 178 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 48 MiB/s wr, 763 op/s 2024-09-21T12:29:27.488 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:29:27 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:29:27.515 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:27.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:27.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:27.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:27 smithi114 ceph-mon[30324]: pgmap v449: 178 pgs: 178 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 48 MiB/s wr, 763 op/s 2024-09-21T12:29:27.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:27 smithi114 ceph-mon[40867]: pgmap v449: 178 pgs: 178 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 48 MiB/s wr, 763 op/s 2024-09-21T12:29:27.616 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:27.662 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:27.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:27.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:27.764 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:27.791 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:27.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:27.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:27.835 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-21T12:29:27.834942+0000 min lat: 0.00136588 max lat: 0.192682 avg lat: 0.00982853 2024-09-21T12:29:27.835 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T12:29:27.835 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 60 16 97581 97565 101.611 80.125 0.00356277 0.00982853 2024-09-21T12:29:27.893 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:27.922 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:27.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:27.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:28.024 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:28.059 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:28.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:28.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:28.161 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:28.191 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:28.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:28.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:28.292 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:28.328 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:28.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:28.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:28.430 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:28.466 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:28.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:28.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:28.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:28 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:29:28.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:28 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:29:28.499 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:28 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:29:28.569 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:28.598 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:28.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:28.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:28.699 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:28.736 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:28.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:28.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:28.835 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 61 16 99012 98996 101.412 89.4375 0.00322371 0.00985603 2024-09-21T12:29:28.839 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:28.859 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:28.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:28.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:28.960 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:28.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:28.997 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:28.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:29.098 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:29.126 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:29.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:29.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:29.227 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:29.259 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:29.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:29.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:29.361 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:29.399 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-21T12:29:29.399 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 110 3.0 2024-09-21T12:29:29.412 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:29.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:29.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:29.457 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:29 smithi114 ceph-mon[40867]: pgmap v450: 178 pgs: 178 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-09-21T12:29:29.457 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:29 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:29:29.457 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:29 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:29:29.457 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:29 smithi114 ceph-mon[30324]: pgmap v450: 178 pgs: 178 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-09-21T12:29:29.458 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:29 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:29:29.458 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:29 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:29:29.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:29 smithi183 ceph-mon[33277]: pgmap v450: 178 pgs: 178 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-09-21T12:29:29.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:29 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:29:29.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:29 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:29:29.513 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:29.557 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:29.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:29.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:29.592 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:29.660 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:29.687 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:29.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:29.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:29.789 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:29.828 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:29.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:29.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:29.836 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 62 16 100343 100327 101.116 83.1875 0.0205547 0.00988528 2024-09-21T12:29:29.930 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:29.970 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:29.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:29.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:30.072 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:30.108 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:30.135 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:30.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:30.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:30.236 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:30.266 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:30.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:30.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:30.364 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:30 smithi114 ceph-mon[30324]: osdmap e262: 8 total, 8 up, 7 in 2024-09-21T12:29:30.364 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:30 smithi114 ceph-mon[40867]: osdmap e262: 8 total, 8 up, 7 in 2024-09-21T12:29:30.367 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:30.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:30.405 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:30.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:30.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:30 smithi183 ceph-mon[33277]: osdmap e262: 8 total, 8 up, 7 in 2024-09-21T12:29:30.506 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:30.547 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:30.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:30.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:30.650 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:30.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:30.706 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:30.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:30.806 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:30.833 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:30.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:30.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:30.836 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 63 16 101747 101731 100.904 87.75 0.00479988 0.0099058 2024-09-21T12:29:30.935 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:30.962 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:30.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:30.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:30.972 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:29:30.972 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:29:31.014 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:29:31.014 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:29:31.060 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:29:31.060 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:29:31.064 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:31.090 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:31.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:31.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:31.116 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:29:31.117 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:29:31.159 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:29:31.159 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:29:31.168 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:29:31.168 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:29:31.192 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:31.197 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:29:31.197 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:29:31.216 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:31.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:31.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:31.231 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:31 smithi114 ceph-mon[30324]: pgmap v452: 178 pgs: 178 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 768 B/s rd, 58 MiB/s wr, 924 op/s 2024-09-21T12:29:31.232 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:31 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2305262438' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:29:31.232 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:31 smithi114 ceph-mon[40867]: pgmap v452: 178 pgs: 178 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 768 B/s rd, 58 MiB/s wr, 924 op/s 2024-09-21T12:29:31.232 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:31 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2305262438' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:29:31.232 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:29:31.232 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:29:31.264 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:29:31.264 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:29:31.318 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:31.347 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:31.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:31.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:31.449 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:31.487 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:31.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:31 smithi183 ceph-mon[33277]: pgmap v452: 178 pgs: 178 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 768 B/s rd, 58 MiB/s wr, 924 op/s 2024-09-21T12:29:31.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:31 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2305262438' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:29:31.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:31.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:31.589 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:31.628 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:31.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:31.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:31.729 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:31.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:31.761 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:31.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:31.836 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 64 16 103093 103077 100.642 84.125 0.0235569 0.00993123 2024-09-21T12:29:31.862 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:31.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:31.900 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:31.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:32.001 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:32.037 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:32.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:32.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:32.090 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T12:29:32.140 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:32.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:32.187 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:32.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:32.288 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:32.292 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T12:29:32.314 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:32.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:32.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:32.416 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:32.442 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:32.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:32.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:32 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:29:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:32 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:29:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:32 smithi183 ceph-mon[33277]: osdmap e263: 8 total, 8 up, 7 in 2024-09-21T12:29:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:32 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:29:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:32 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:29:32.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:32 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:29:32.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:32 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:29:32.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:32 smithi114 ceph-mon[30324]: osdmap e263: 8 total, 8 up, 7 in 2024-09-21T12:29:32.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:32 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:29:32.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:32 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:29:32.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:32 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:29:32.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:32 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:29:32.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:32 smithi114 ceph-mon[40867]: osdmap e263: 8 total, 8 up, 7 in 2024-09-21T12:29:32.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:32 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:29:32.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:32 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:29:32.544 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:32.596 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:32.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:32.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:32.698 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:32.723 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:32.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:32.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:32.754 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:32.806 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:29:32 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:29:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:29:32.824 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:32.837 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 65 15 104383 104368 100.334 80.6875 0.0101241 0.00996233 2024-09-21T12:29:32.859 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:32.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:32.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:32.961 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:33.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:33.013 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:33.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:33.114 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:33.153 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:33.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:33.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:33.254 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:33.302 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:33.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:33.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:33.377 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:33 smithi114 ceph-mon[40867]: pgmap v454: 178 pgs: 178 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.5 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-21T12:29:33.377 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:33 smithi114 ceph-mon[30324]: pgmap v454: 178 pgs: 178 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.5 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-21T12:29:33.404 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:33.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:33.432 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:33.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:33.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:33 smithi183 ceph-mon[33277]: pgmap v454: 178 pgs: 178 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.5 KiB/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-21T12:29:33.533 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:33.559 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:33.565 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:33.598 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:33.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:33.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:33.700 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:33.750 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:33.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:33.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:33.837 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 66 16 105676 105660 100.037 80.75 0.00341039 0.00999158 2024-09-21T12:29:33.853 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:33.893 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:33.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:33.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:33.995 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:34.026 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:34.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:34.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:34.128 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:34.167 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:34.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:34.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:34.268 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:34.297 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:34.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:34.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:34.398 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:34.424 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:34.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:34.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:34.525 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:34.555 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:34.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:34.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:34.657 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:34.704 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:34.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:34.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:34.751 INFO:teuthology.orchestra.run.smithi114.stdout:moved 64 / 534 (11.985%) 2024-09-21T12:29:34.751 INFO:teuthology.orchestra.run.smithi114.stdout:avg 76.2857 2024-09-21T12:29:34.751 INFO:teuthology.orchestra.run.smithi114.stdout:stddev 7.08548 -> 3.91752 (expected baseline 8.08627) 2024-09-21T12:29:34.751 INFO:teuthology.orchestra.run.smithi114.stdout:min osd.2 with 62 -> 73 pgs (0.812734 -> 0.956929 * mean) 2024-09-21T12:29:34.751 INFO:teuthology.orchestra.run.smithi114.stdout:max osd.1 with 83 -> 74 pgs (1.08801 -> 0.970037 * mean) 2024-09-21T12:29:34.751 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:29:34.752 INFO:teuthology.orchestra.run.smithi114.stdout:oload 110 2024-09-21T12:29:34.752 INFO:teuthology.orchestra.run.smithi114.stdout:max_change 3 2024-09-21T12:29:34.752 INFO:teuthology.orchestra.run.smithi114.stdout:max_change_osds 4 2024-09-21T12:29:34.752 INFO:teuthology.orchestra.run.smithi114.stdout:average_utilization 0.0341 2024-09-21T12:29:34.752 INFO:teuthology.orchestra.run.smithi114.stdout:overload_utilization 0.0375 2024-09-21T12:29:34.752 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7 weight 1.0000 -> 0.7920 2024-09-21T12:29:34.752 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 weight 1.0000 -> 0.7920 2024-09-21T12:29:34.752 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 weight 1.0000 -> 0.7760 2024-09-21T12:29:34.752 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 weight 1.0000 -> 0.8459 2024-09-21T12:29:34.808 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:29:34.810 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:34.837 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 67 16 106981 106965 99.7614 81.5625 0.00545552 0.010012 2024-09-21T12:29:34.857 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:34.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:34.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:34.959 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:34.980 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:34.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:34.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:35.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:34 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:29:35.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:34 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:29:35.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:34 smithi114 ceph-mon[30324]: osdmap e264: 8 total, 8 up, 7 in 2024-09-21T12:29:35.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:34 smithi114 ceph-mon[30324]: pgmap v456: 178 pgs: 178 active+clean; 6.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-21T12:29:35.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:34 smithi114 ceph-mon[30324]: from='client.25300 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:29:35.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:34 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'1':'50857','3':'55438','5':'51905','7':'51905'}"} : dispatch 2024-09-21T12:29:35.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:34 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'1':'50857','3':'55438','5':'51905','7':'51905'}"} : dispatch 2024-09-21T12:29:35.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:34 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:29:35.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:34 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:29:35.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:34 smithi114 ceph-mon[40867]: osdmap e264: 8 total, 8 up, 7 in 2024-09-21T12:29:35.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:34 smithi114 ceph-mon[40867]: pgmap v456: 178 pgs: 178 active+clean; 6.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-21T12:29:35.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:34 smithi114 ceph-mon[40867]: from='client.25300 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:29:35.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:34 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'1':'50857','3':'55438','5':'51905','7':'51905'}"} : dispatch 2024-09-21T12:29:35.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:34 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'1':'50857','3':'55438','5':'51905','7':'51905'}"} : dispatch 2024-09-21T12:29:35.081 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:35.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:35.102 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:35.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:35.170 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:29:35.170 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:29:35.203 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:35.224 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:35.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:35.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:35.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:34 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:29:35.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:34 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:29:35.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:34 smithi183 ceph-mon[33277]: osdmap e264: 8 total, 8 up, 7 in 2024-09-21T12:29:35.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:34 smithi183 ceph-mon[33277]: pgmap v456: 178 pgs: 178 active+clean; 6.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-21T12:29:35.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:34 smithi183 ceph-mon[33277]: from='client.25300 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:29:35.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:34 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'1':'50857','3':'55438','5':'51905','7':'51905'}"} : dispatch 2024-09-21T12:29:35.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:34 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'1':'50857','3':'55438','5':'51905','7':'51905'}"} : dispatch 2024-09-21T12:29:35.326 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:35.348 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:35.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:35.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:35.449 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:35.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:35.471 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:35.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:35.572 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:35.594 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:35.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:35.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:35.695 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:35.716 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:35.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:35.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:35.746 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-21T12:29:35.818 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:35.837 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 68 16 106981 106965 98.2945 0 - 0.010012 2024-09-21T12:29:35.839 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:35.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:35.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:35.893 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T12:29:35.941 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:35.961 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:35.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:35.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:35.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:35 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'50857','3':'55438','5':'51905','7':'51905'}"}]': finished 2024-09-21T12:29:35.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:35 smithi183 ceph-mon[33277]: osdmap e265: 8 total, 8 up, 7 in 2024-09-21T12:29:35.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:35 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1757834275' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:29:36.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:35 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'50857','3':'55438','5':'51905','7':'51905'}"}]': finished 2024-09-21T12:29:36.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:35 smithi114 ceph-mon[40867]: osdmap e265: 8 total, 8 up, 7 in 2024-09-21T12:29:36.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:35 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1757834275' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:29:36.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:35 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'50857','3':'55438','5':'51905','7':'51905'}"}]': finished 2024-09-21T12:29:36.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:35 smithi114 ceph-mon[30324]: osdmap e265: 8 total, 8 up, 7 in 2024-09-21T12:29:36.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:35 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1757834275' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:29:36.063 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:36.084 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:36.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:36.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:36.186 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:36.207 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:36.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:36.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:36.309 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:36.330 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:36.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:36.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:36.431 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:36.452 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:36.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:36.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:36.553 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:36.575 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:36.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:36.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:36.676 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:36.698 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:36.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:36.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:36.800 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:36.802 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-21T12:29:36.802 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd test-reweight-by-utilization 110 0.05 2024-09-21T12:29:36.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:36.840 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:36.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:36.840 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 69 16 106981 106965 96.8673 0 - 0.010012 2024-09-21T12:29:36.941 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:36.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:36.972 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:36.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:36.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:36 smithi183 ceph-mon[33277]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:29:36.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:36 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1757834275' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:29:36.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:36 smithi183 ceph-mon[33277]: osdmap e266: 8 total, 8 up, 7 in 2024-09-21T12:29:36.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:36 smithi183 ceph-mon[33277]: pgmap v459: 178 pgs: 178 active+clean; 6.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.1 KiB/s rd, 92 MiB/s wr, 1.48k op/s 2024-09-21T12:29:37.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:36 smithi114 ceph-mon[30324]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:29:37.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:36 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1757834275' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:29:37.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:36 smithi114 ceph-mon[30324]: osdmap e266: 8 total, 8 up, 7 in 2024-09-21T12:29:37.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:36 smithi114 ceph-mon[30324]: pgmap v459: 178 pgs: 178 active+clean; 6.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.1 KiB/s rd, 92 MiB/s wr, 1.48k op/s 2024-09-21T12:29:37.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:36 smithi114 ceph-mon[40867]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:29:37.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:36 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1757834275' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:29:37.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:36 smithi114 ceph-mon[40867]: osdmap e266: 8 total, 8 up, 7 in 2024-09-21T12:29:37.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:36 smithi114 ceph-mon[40867]: pgmap v459: 178 pgs: 178 active+clean; 6.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.1 KiB/s rd, 92 MiB/s wr, 1.48k op/s 2024-09-21T12:29:37.072 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:37.099 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:37.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:37.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:37.200 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:37.231 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:37.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:37.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:37.333 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:37.362 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:37.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:37.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:37.464 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:37.488 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:29:37 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:29:37.491 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:37.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:37.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:37.567 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:37.593 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:37.631 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:37.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:37.643 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:37.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:37.743 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:37.751 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[40867]: 4.1b scrub starts 2024-09-21T12:29:37.751 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[30324]: 4.1b scrub starts 2024-09-21T12:29:37.752 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[30324]: 4.1b scrub ok 2024-09-21T12:29:37.752 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[30324]: 6.4 scrub starts 2024-09-21T12:29:37.752 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[30324]: 6.4 scrub ok 2024-09-21T12:29:37.752 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[30324]: 1.d scrub starts 2024-09-21T12:29:37.752 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[30324]: 1.d scrub ok 2024-09-21T12:29:37.752 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[30324]: 9.5 starting backfill to osd.1 from (0'0,0'0] MAX to 264'6689 2024-09-21T12:29:37.752 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[30324]: 9.8 starting backfill to osd.1 from (0'0,0'0] MAX to 264'6657 2024-09-21T12:29:37.752 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[30324]: 9.a starting backfill to osd.3 from (0'0,0'0] MAX to 264'6662 2024-09-21T12:29:37.752 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[30324]: osdmap e267: 8 total, 8 up, 7 in 2024-09-21T12:29:37.752 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[30324]: 9.2 starting backfill to osd.7 from (0'0,0'0] MAX to 264'6642 2024-09-21T12:29:37.752 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[30324]: 9.0 starting backfill to osd.0 from (0'0,0'0] MAX to 264'6627 2024-09-21T12:29:37.752 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[30324]: 9.1 starting backfill to osd.2 from (0'0,0'0] MAX to 264'6694 2024-09-21T12:29:37.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:37.798 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:37.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:37.839 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 70 16 107825 107809 96.237 17.5833 0.00581299 0.0103858 2024-09-21T12:29:37.899 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:37.948 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:37.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:37.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:38.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[30324]: 9.7 starting backfill to osd.1 from (0'0,0'0] MAX to 264'6632 2024-09-21T12:29:38.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[30324]: 9.d starting backfill to osd.6 from (0'0,0'0] MAX to 264'6666 2024-09-21T12:29:38.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[30324]: 9.8 starting backfill to osd.7 from (0'0,0'0] MAX to 264'6657 2024-09-21T12:29:38.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[30324]: 9.7 starting backfill to osd.2 from (0'0,0'0] MAX to 264'6632 2024-09-21T12:29:38.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:29:38.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[40867]: 4.1b scrub ok 2024-09-21T12:29:38.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[40867]: 6.4 scrub starts 2024-09-21T12:29:38.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[40867]: 6.4 scrub ok 2024-09-21T12:29:38.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[40867]: 1.d scrub starts 2024-09-21T12:29:38.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[40867]: 1.d scrub ok 2024-09-21T12:29:38.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[40867]: 9.5 starting backfill to osd.1 from (0'0,0'0] MAX to 264'6689 2024-09-21T12:29:38.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[40867]: 9.8 starting backfill to osd.1 from (0'0,0'0] MAX to 264'6657 2024-09-21T12:29:38.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[40867]: 9.a starting backfill to osd.3 from (0'0,0'0] MAX to 264'6662 2024-09-21T12:29:38.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[40867]: osdmap e267: 8 total, 8 up, 7 in 2024-09-21T12:29:38.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[40867]: 9.2 starting backfill to osd.7 from (0'0,0'0] MAX to 264'6642 2024-09-21T12:29:38.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[40867]: 9.0 starting backfill to osd.0 from (0'0,0'0] MAX to 264'6627 2024-09-21T12:29:38.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[40867]: 9.1 starting backfill to osd.2 from (0'0,0'0] MAX to 264'6694 2024-09-21T12:29:38.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[40867]: 9.7 starting backfill to osd.1 from (0'0,0'0] MAX to 264'6632 2024-09-21T12:29:38.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[40867]: 9.d starting backfill to osd.6 from (0'0,0'0] MAX to 264'6666 2024-09-21T12:29:38.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[40867]: 9.8 starting backfill to osd.7 from (0'0,0'0] MAX to 264'6657 2024-09-21T12:29:38.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[40867]: 9.7 starting backfill to osd.2 from (0'0,0'0] MAX to 264'6632 2024-09-21T12:29:38.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:37 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:29:38.050 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:38.080 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:38.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:38.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:38.181 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:38.210 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:38.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:38.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:38.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:37 smithi183 ceph-mon[33277]: 4.1b scrub starts 2024-09-21T12:29:38.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:37 smithi183 ceph-mon[33277]: 4.1b scrub ok 2024-09-21T12:29:38.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:37 smithi183 ceph-mon[33277]: 6.4 scrub starts 2024-09-21T12:29:38.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:37 smithi183 ceph-mon[33277]: 6.4 scrub ok 2024-09-21T12:29:38.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:37 smithi183 ceph-mon[33277]: 1.d scrub starts 2024-09-21T12:29:38.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:37 smithi183 ceph-mon[33277]: 1.d scrub ok 2024-09-21T12:29:38.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:37 smithi183 ceph-mon[33277]: 9.5 starting backfill to osd.1 from (0'0,0'0] MAX to 264'6689 2024-09-21T12:29:38.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:37 smithi183 ceph-mon[33277]: 9.8 starting backfill to osd.1 from (0'0,0'0] MAX to 264'6657 2024-09-21T12:29:38.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:37 smithi183 ceph-mon[33277]: 9.a starting backfill to osd.3 from (0'0,0'0] MAX to 264'6662 2024-09-21T12:29:38.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:37 smithi183 ceph-mon[33277]: osdmap e267: 8 total, 8 up, 7 in 2024-09-21T12:29:38.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:37 smithi183 ceph-mon[33277]: 9.2 starting backfill to osd.7 from (0'0,0'0] MAX to 264'6642 2024-09-21T12:29:38.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:37 smithi183 ceph-mon[33277]: 9.0 starting backfill to osd.0 from (0'0,0'0] MAX to 264'6627 2024-09-21T12:29:38.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:37 smithi183 ceph-mon[33277]: 9.1 starting backfill to osd.2 from (0'0,0'0] MAX to 264'6694 2024-09-21T12:29:38.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:37 smithi183 ceph-mon[33277]: 9.7 starting backfill to osd.1 from (0'0,0'0] MAX to 264'6632 2024-09-21T12:29:38.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:37 smithi183 ceph-mon[33277]: 9.d starting backfill to osd.6 from (0'0,0'0] MAX to 264'6666 2024-09-21T12:29:38.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:37 smithi183 ceph-mon[33277]: 9.8 starting backfill to osd.7 from (0'0,0'0] MAX to 264'6657 2024-09-21T12:29:38.240 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:37 smithi183 ceph-mon[33277]: 9.7 starting backfill to osd.2 from (0'0,0'0] MAX to 264'6632 2024-09-21T12:29:38.240 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:37 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:29:38.312 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:38.341 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:38.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:38.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:38.442 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:38.471 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:38.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:38.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:38.503 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:29:38.503 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:29:38.573 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:38.607 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:38.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:38.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:38.709 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:38.723 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:38.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:38.762 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:38.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:38.839 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 71 16 108801 108785 95.7406 61 0.0140818 0.0104404 2024-09-21T12:29:38.851 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:29:38.851 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:29:38.851 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:29:38.851 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:29:38.851 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:29:38.851 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:29:38.851 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:29:38.851 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:29:38.852 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:29:38.852 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:29:38.852 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:29:38.852 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:29:38.853 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:29:38.853 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:29:38.864 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:38.888 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:29:38.888 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:29:38.903 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:38.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:38.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:38.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:38 smithi114 ceph-mon[30324]: 5.1b scrub starts 2024-09-21T12:29:38.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:38 smithi114 ceph-mon[30324]: 5.1b scrub ok 2024-09-21T12:29:38.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:38 smithi114 ceph-mon[30324]: pgmap v461: 178 pgs: 5 active+remapped+backfilling, 1 remapped+peering, 2 active+remapped+backfill_toofull, 170 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 4.0 KiB/s rd, 59 MiB/s wr, 955 op/s; 59956/322257 objects misplaced (18.605%); 2.0 MiB/s, 49 objects/s recovering 2024-09-21T12:29:38.973 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:38 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/688060297' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:29:38.973 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:38 smithi114 ceph-mon[40867]: 5.1b scrub starts 2024-09-21T12:29:38.973 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:38 smithi114 ceph-mon[40867]: 5.1b scrub ok 2024-09-21T12:29:38.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:38 smithi114 ceph-mon[40867]: pgmap v461: 178 pgs: 5 active+remapped+backfilling, 1 remapped+peering, 2 active+remapped+backfill_toofull, 170 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 4.0 KiB/s rd, 59 MiB/s wr, 955 op/s; 59956/322257 objects misplaced (18.605%); 2.0 MiB/s, 49 objects/s recovering 2024-09-21T12:29:38.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:38 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/688060297' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:29:39.004 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:39.042 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:39.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:39.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:39.143 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:39.183 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:39.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:39.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:39.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:38 smithi183 ceph-mon[33277]: 5.1b scrub starts 2024-09-21T12:29:39.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:38 smithi183 ceph-mon[33277]: 5.1b scrub ok 2024-09-21T12:29:39.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:38 smithi183 ceph-mon[33277]: pgmap v461: 178 pgs: 5 active+remapped+backfilling, 1 remapped+peering, 2 active+remapped+backfill_toofull, 170 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 4.0 KiB/s rd, 59 MiB/s wr, 955 op/s; 59956/322257 objects misplaced (18.605%); 2.0 MiB/s, 49 objects/s recovering 2024-09-21T12:29:39.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:38 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/688060297' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:29:39.287 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:39.326 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:39.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:39.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:39.428 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:39.456 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:39.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:39.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:39.551 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T12:29:39.558 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:39.648 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:39.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:39.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:39.699 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T12:29:39.750 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:39.778 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:39.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:39.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:39.839 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 72 16 109781 109765 95.2616 61.25 0.0177497 0.0104922 2024-09-21T12:29:39.880 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:39.940 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:39.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:39.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:40.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:39 smithi114 ceph-mon[40867]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:29:40.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:39 smithi114 ceph-mon[30324]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:29:40.041 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:40.071 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:40.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:40.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:40.173 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:40.202 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:40.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:40.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:40.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:39 smithi183 ceph-mon[33277]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:29:40.304 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:40.347 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:40.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:40.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:40.449 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:40.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:40.540 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:40.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:40.640 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:40.683 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:40.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:40.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:40.785 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:40.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:40.834 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:40.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:40.840 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 73 15 110778 110763 94.811 62.375 0.0211056 0.0105428 2024-09-21T12:29:40.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:40 smithi114 ceph-mon[30324]: pgmap v462: 178 pgs: 5 active+remapped+backfilling, 1 remapped+peering, 2 active+remapped+backfill_toofull, 170 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 2.7 KiB/s rd, 40 MiB/s wr, 640 op/s; 59956/322257 objects misplaced (18.605%); 1.4 MiB/s, 33 objects/s recovering 2024-09-21T12:29:40.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:40 smithi114 ceph-mon[40867]: pgmap v462: 178 pgs: 5 active+remapped+backfilling, 1 remapped+peering, 2 active+remapped+backfill_toofull, 170 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 2.7 KiB/s rd, 40 MiB/s wr, 640 op/s; 59956/322257 objects misplaced (18.605%); 1.4 MiB/s, 33 objects/s recovering 2024-09-21T12:29:40.935 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:40.962 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:40.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:40.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:41.063 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:41.065 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:41.090 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:41.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:41.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:41.104 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:41.191 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:41.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:41.223 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:41.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:41.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:40 smithi183 ceph-mon[33277]: pgmap v462: 178 pgs: 5 active+remapped+backfilling, 1 remapped+peering, 2 active+remapped+backfill_toofull, 170 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 2.7 KiB/s rd, 40 MiB/s wr, 640 op/s; 59956/322257 objects misplaced (18.605%); 1.4 MiB/s, 33 objects/s recovering 2024-09-21T12:29:41.325 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:41.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:41.374 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:41.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:41.416 INFO:teuthology.orchestra.run.smithi114.stdout:moved 14 / 534 (2.62172%) 2024-09-21T12:29:41.416 INFO:teuthology.orchestra.run.smithi114.stdout:avg 76.2857 2024-09-21T12:29:41.416 INFO:teuthology.orchestra.run.smithi114.stdout:stddev 3.91752 -> 5.84843 (expected baseline 8.08627) 2024-09-21T12:29:41.416 INFO:teuthology.orchestra.run.smithi114.stdout:min osd.7 with 72 -> 67 pgs (0.94382 -> 0.878277 * mean) 2024-09-21T12:29:41.416 INFO:teuthology.orchestra.run.smithi114.stdout:max osd.0 with 83 -> 85 pgs (1.08801 -> 1.11423 * mean) 2024-09-21T12:29:41.416 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:29:41.416 INFO:teuthology.orchestra.run.smithi114.stdout:oload 110 2024-09-21T12:29:41.416 INFO:teuthology.orchestra.run.smithi114.stdout:max_change 0.05 2024-09-21T12:29:41.416 INFO:teuthology.orchestra.run.smithi114.stdout:max_change_osds 4 2024-09-21T12:29:41.416 INFO:teuthology.orchestra.run.smithi114.stdout:average_utilization 0.0353 2024-09-21T12:29:41.416 INFO:teuthology.orchestra.run.smithi114.stdout:overload_utilization 0.0388 2024-09-21T12:29:41.417 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 weight 0.7920 -> 0.7420 2024-09-21T12:29:41.417 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7 weight 0.7920 -> 0.7420 2024-09-21T12:29:41.417 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 weight 0.7760 -> 0.7260 2024-09-21T12:29:41.417 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 weight 0.8459 -> 0.7959 2024-09-21T12:29:41.417 INFO:teuthology.orchestra.run.smithi114.stderr:no change 2024-09-21T12:29:41.475 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:41.486 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:29:41 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: 2024-09-21T12:29:41.411+0000 7f0da3311640 -1 mgr.server reply reply (4) Interrupted system call no change 2024-09-21T12:29:41.520 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:41.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:41.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:41.622 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:41.650 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:41.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:41.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:41.751 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:41.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:41.796 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:41.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:41.840 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 74 16 111696 111680 94.3043 57.3125 0.0248822 0.0105998 2024-09-21T12:29:41.896 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:41.940 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:41.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:41.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:42.042 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:42.087 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:42.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:42.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:42.189 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:42.216 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:42.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:42.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:42.318 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:42.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:42.358 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:42.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:42.407 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:29:42.426 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:29:42.426 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:29:42.426 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:29:42.427 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:29:42.427 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:29:42.459 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:42.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:42.512 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:42.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:42.613 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:42.641 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:42.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:42.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:42.742 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:42.771 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:42.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:42.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:42.774 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:29:42 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:29:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:29:42.785 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-21T12:29:42.840 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 75 15 112584 112569 93.7878 55.5625 0.0183827 0.0106573 2024-09-21T12:29:42.872 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:42.909 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:42.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:42.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:43.010 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:43.043 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:43.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:43.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:43.144 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:43.191 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:43.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:43.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:43.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:42 smithi114 ceph-mon[30324]: from='client.25315 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:29:43.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:42 smithi114 ceph-mon[30324]: pgmap v463: 178 pgs: 6 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 3.0 KiB/s rd, 44 MiB/s wr, 710 op/s; 64967/325659 objects misplaced (19.949%); 18 MiB/s, 299 objects/s recovering 2024-09-21T12:29:43.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:42 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3892703158' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:29:43.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:42 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:29:43.237 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:42 smithi114 ceph-mon[40867]: from='client.25315 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:29:43.237 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:42 smithi114 ceph-mon[40867]: pgmap v463: 178 pgs: 6 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 3.0 KiB/s rd, 44 MiB/s wr, 710 op/s; 64967/325659 objects misplaced (19.949%); 18 MiB/s, 299 objects/s recovering 2024-09-21T12:29:43.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:42 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3892703158' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:29:43.238 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:42 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:29:43.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:42 smithi183 ceph-mon[33277]: from='client.25315 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:29:43.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:42 smithi183 ceph-mon[33277]: pgmap v463: 178 pgs: 6 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 3.0 KiB/s rd, 44 MiB/s wr, 710 op/s; 64967/325659 objects misplaced (19.949%); 18 MiB/s, 299 objects/s recovering 2024-09-21T12:29:43.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:42 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3892703158' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:29:43.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:42 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:29:43.293 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:43.323 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:43.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:43.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:43.424 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:43.462 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:43.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:43.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:43.563 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:43.592 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:43.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:43.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:43.694 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:43.725 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:43.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:43.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:43.827 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:43.840 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 76 16 113628 113612 93.4114 65.1875 0.00636134 0.0107006 2024-09-21T12:29:43.855 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:43.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:43.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:43.957 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:43.994 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:43.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:43.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:44.096 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:44.139 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:44.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:44.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:44.182 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:43 smithi114 ceph-mon[40867]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:29:44.182 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:43 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:29:44.182 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:43 smithi114 ceph-mon[40867]: osdmap e268: 8 total, 8 up, 7 in 2024-09-21T12:29:44.182 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:43 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:29:44.183 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:43 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:29:44.184 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:43 smithi114 ceph-mon[30324]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:29:44.184 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:43 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:29:44.184 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:43 smithi114 ceph-mon[30324]: osdmap e268: 8 total, 8 up, 7 in 2024-09-21T12:29:44.184 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:43 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:29:44.184 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:43 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:29:44.235 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T12:29:44.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:43 smithi183 ceph-mon[33277]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:29:44.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:43 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:29:44.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:43 smithi183 ceph-mon[33277]: osdmap e268: 8 total, 8 up, 7 in 2024-09-21T12:29:44.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:43 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:29:44.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:43 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:29:44.241 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:44.270 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:44.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:44.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:44.372 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:44.416 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:44.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:44.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:44.517 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:44.584 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:44.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:44.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:44.686 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:44.760 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:44.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:44.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:44.780 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:44.838 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:44.840 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 77 16 114556 114540 92.9515 58 0.0163593 0.0107538 2024-09-21T12:29:44.862 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:44.918 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:44.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:44.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:44.924 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:44 smithi114 ceph-mon[40867]: pgmap v465: 178 pgs: 6 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 2.6 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 65488/336471 objects misplaced (19.463%); 30 MiB/s, 496 objects/s recovering 2024-09-21T12:29:44.924 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:44 smithi114 ceph-mon[40867]: osdmap e269: 8 total, 8 up, 7 in 2024-09-21T12:29:44.924 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:44 smithi114 ceph-mon[30324]: pgmap v465: 178 pgs: 6 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 2.6 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 65488/336471 objects misplaced (19.463%); 30 MiB/s, 496 objects/s recovering 2024-09-21T12:29:44.924 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:44 smithi114 ceph-mon[30324]: osdmap e269: 8 total, 8 up, 7 in 2024-09-21T12:29:45.020 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:45.072 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:45.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:45.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:45.174 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:45.214 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:45.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:45.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:45.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:44 smithi183 ceph-mon[33277]: pgmap v465: 178 pgs: 6 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 2.6 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 65488/336471 objects misplaced (19.463%); 30 MiB/s, 496 objects/s recovering 2024-09-21T12:29:45.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:44 smithi183 ceph-mon[33277]: osdmap e269: 8 total, 8 up, 7 in 2024-09-21T12:29:45.316 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:45.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:45.356 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:45.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:45.457 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:45.496 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:45.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:45.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:45.598 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:45.628 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:45.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:45.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:45.686 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:29:45.686 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:29:45.729 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:29:45.730 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:29:45.730 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:45.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:45.759 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:45.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:45.778 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:29:45.778 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:29:45.826 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:29:45.826 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:29:45.840 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 78 16 115518 115502 92.5306 60.125 0.0190459 0.0108018 2024-09-21T12:29:45.860 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:45.865 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:29:45.866 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:29:45.890 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:45.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:45.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:45.904 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:29:45.904 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:29:45.939 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:29:45.940 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:29:45.957 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:29:45.957 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:29:45.991 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:46.034 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:46.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:46.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:46.136 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:46.164 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:46.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:46.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:46.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:45 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/91603042' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:29:46.266 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:46.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:45 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/91603042' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:29:46.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:45 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/91603042' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:29:46.294 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:46.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:46.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:46.350 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:29:46.351 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:29:46.396 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:46.433 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:46.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:46.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:46.534 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:46.570 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:46.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:46.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:46.671 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:46.699 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:46.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:46.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:46.730 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T12:29:46.801 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:46.828 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:46.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:46.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:46.840 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 79 16 116466 116450 92.1094 59.25 0.00723726 0.0108519 2024-09-21T12:29:46.930 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:46.954 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:46.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:46.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:46.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:46 smithi183 ceph-mon[33277]: pgmap v467: 178 pgs: 6 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 37 MiB/s wr, 592 op/s; 65488/336471 objects misplaced (19.463%); 29 MiB/s, 471 objects/s recovering 2024-09-21T12:29:46.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:46 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:29:46.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:29:47.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:46 smithi114 ceph-mon[30324]: pgmap v467: 178 pgs: 6 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 37 MiB/s wr, 592 op/s; 65488/336471 objects misplaced (19.463%); 29 MiB/s, 471 objects/s recovering 2024-09-21T12:29:47.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:46 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:29:47.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:29:47.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:46 smithi114 ceph-mon[40867]: pgmap v467: 178 pgs: 6 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 37 MiB/s wr, 592 op/s; 65488/336471 objects misplaced (19.463%); 29 MiB/s, 471 objects/s recovering 2024-09-21T12:29:47.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:46 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:29:47.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:29:47.055 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:47.102 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:47.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:47.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:47.204 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:47.245 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:47.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:47.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:47.346 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:47.374 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:47.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:47.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:47.476 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:47.488 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:29:47 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:29:47.503 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:47.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:47.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:47.604 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:47.626 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:47.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:47.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:47.728 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:47.756 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:47.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:47.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:47.840 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-21T12:29:47.840083+0000 min lat: 0.00136588 max lat: 2.12297 avg lat: 0.0109188 2024-09-21T12:29:47.840 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T12:29:47.840 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 80 16 117199 117183 91.5307 45.8125 0.0100606 0.0109188 2024-09-21T12:29:47.858 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:47.880 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:47.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:47.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:47.982 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:48.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:48.022 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:48.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:48.022 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T12:29:48.023 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:29:48.023 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_2 2024-09-21T12:29:48.023 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_2 by 10 min 8 2024-09-21T12:29:48.023 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:29:48.108 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:48 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:29:48.108 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:48 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:29:48.123 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:48.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:48.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:48.278 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:48.322 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:48.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:48.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:48.425 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:48.470 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:48.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:48.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:48.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:48 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:29:48.572 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:48.594 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:48.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:48.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:48.696 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:48.724 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:48.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:48.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:48.826 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:48.840 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 81 15 117844 117829 90.8991 40.375 0.00886079 0.0109964 2024-09-21T12:29:48.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:48.858 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:48.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:48.959 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:48.981 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:48.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:48.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:49.082 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:49.117 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:49.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:49.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:49.219 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:49.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:49.266 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:49.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:49.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:49 smithi114 ceph-mon[40867]: pgmap v468: 178 pgs: 6 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 7.1 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.18k op/s; 64014/350574 objects misplaced (18.260%); 60 MiB/s, 955 objects/s recovering 2024-09-21T12:29:49.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:49 smithi114 ceph-mon[30324]: pgmap v468: 178 pgs: 6 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 7.1 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.18k op/s; 64014/350574 objects misplaced (18.260%); 60 MiB/s, 955 objects/s recovering 2024-09-21T12:29:49.366 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:49.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:49.396 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:49.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:49.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:49 smithi183 ceph-mon[33277]: pgmap v468: 178 pgs: 6 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 7.1 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.18k op/s; 64014/350574 objects misplaced (18.260%); 60 MiB/s, 955 objects/s recovering 2024-09-21T12:29:49.497 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:49.529 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:49.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:49.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:49.630 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:49.654 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:49.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:49.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:49.756 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:49.777 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:49.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:49.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:49.840 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 82 16 118496 118480 90.2868 40.6875 0.0139985 0.0110711 2024-09-21T12:29:49.879 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:49.919 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:49.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:49.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:50.021 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:50.048 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:50.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:50.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:50.150 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:50.172 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:50.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:50.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:50.273 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:50.301 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:50.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:50.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:50.403 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:50.425 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:50.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:50.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:50.527 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:50.556 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:50.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:50.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:50.657 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:50.695 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:50.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:50.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:50.797 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:50.821 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:50.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:50.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:50.841 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 83 16 119287 119271 89.7947 49.4375 0.0120686 0.0111322 2024-09-21T12:29:50.923 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:50.953 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:50.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:50.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:51.055 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:51.082 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:51.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:51.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:51.184 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:51.226 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:51.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:51.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:51.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:51 smithi114 ceph-mon[30324]: pgmap v469: 178 pgs: 6 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 7.1 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 65 MiB/s wr, 1.04k op/s; 64014/350574 objects misplaced (18.260%); 45 MiB/s, 717 objects/s recovering 2024-09-21T12:29:51.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:51 smithi114 ceph-mon[40867]: pgmap v469: 178 pgs: 6 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 7.1 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 65 MiB/s wr, 1.04k op/s; 64014/350574 objects misplaced (18.260%); 45 MiB/s, 717 objects/s recovering 2024-09-21T12:29:51.328 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:51.355 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:51.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:51.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:51.457 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:51.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:51 smithi183 ceph-mon[33277]: pgmap v469: 178 pgs: 6 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 7.1 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 65 MiB/s wr, 1.04k op/s; 64014/350574 objects misplaced (18.260%); 45 MiB/s, 717 objects/s recovering 2024-09-21T12:29:51.492 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:51.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:51.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:51.594 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:51.625 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:51.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:51.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:51.726 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:51.754 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:51.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:51.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:51.840 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 84 16 120291 120275 89.4727 62.75 0.0102559 0.0111718 2024-09-21T12:29:51.856 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:51.895 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:51.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:51.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:51.997 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:52.018 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:52.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:52.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:52.120 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:52.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:52.151 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:52.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:52.252 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:52.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:52 smithi114 ceph-mon[40867]: 1.18 scrub starts 2024-09-21T12:29:52.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:52 smithi114 ceph-mon[40867]: 1.18 scrub ok 2024-09-21T12:29:52.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:52 smithi114 ceph-mon[30324]: 1.18 scrub starts 2024-09-21T12:29:52.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:52 smithi114 ceph-mon[30324]: 1.18 scrub ok 2024-09-21T12:29:52.287 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:52.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:52.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:52.391 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:52.414 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:52.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:52.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:52.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:52 smithi183 ceph-mon[33277]: 1.18 scrub starts 2024-09-21T12:29:52.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:52 smithi183 ceph-mon[33277]: 1.18 scrub ok 2024-09-21T12:29:52.516 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:52.550 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:52.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:52.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:52.652 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:52.675 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:52.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:52.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:52.776 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:52.813 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:52.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:52.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:52.841 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 85 16 120989 120973 88.9333 43.625 0.00737867 0.0112398 2024-09-21T12:29:52.915 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:52.947 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:52.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:52.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:53.017 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:29:52 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:29:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:29:53.048 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:53.075 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:53.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:53.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:53.176 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:53.199 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:53.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:53.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:53.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:53 smithi114 ceph-mon[40867]: pgmap v470: 178 pgs: 6 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 7.2 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 39 MiB/s wr, 632 op/s; 61872/353469 objects misplaced (17.504%); 42 MiB/s, 677 objects/s recovering 2024-09-21T12:29:53.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:53 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:29:53.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:53 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:29:53.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:53 smithi114 ceph-mon[40867]: 9.f scrub starts 2024-09-21T12:29:53.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:53 smithi114 ceph-mon[30324]: pgmap v470: 178 pgs: 6 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 7.2 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 39 MiB/s wr, 632 op/s; 61872/353469 objects misplaced (17.504%); 42 MiB/s, 677 objects/s recovering 2024-09-21T12:29:53.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:53 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:29:53.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:53 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:29:53.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:53 smithi114 ceph-mon[30324]: 9.f scrub starts 2024-09-21T12:29:53.300 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:53.332 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:53.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:53.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:53.434 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:53.461 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:53.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:53.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:53.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:53 smithi183 ceph-mon[33277]: pgmap v470: 178 pgs: 6 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 7.2 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 39 MiB/s wr, 632 op/s; 61872/353469 objects misplaced (17.504%); 42 MiB/s, 677 objects/s recovering 2024-09-21T12:29:53.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:53 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:29:53.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:53 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:29:53.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:53 smithi183 ceph-mon[33277]: 9.f scrub starts 2024-09-21T12:29:53.564 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:53.604 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:53.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:53.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:53.706 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:53.739 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:53.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:53.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:53.841 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 86 16 121666 121650 88.3912 42.3125 0.0238448 0.0113044 2024-09-21T12:29:53.841 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:53.880 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:53.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:53.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:53.982 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:54.015 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:54.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:54.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:54.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:54.161 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:54.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:54.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:54.262 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:54.283 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:54.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:54.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:54.384 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:54.429 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:54.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:54.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:54.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:54 smithi183 ceph-mon[33277]: osdmap e270: 8 total, 8 up, 7 in 2024-09-21T12:29:54.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:54 smithi183 ceph-mon[33277]: 6.19 scrub starts 2024-09-21T12:29:54.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:54 smithi183 ceph-mon[33277]: 6.19 scrub ok 2024-09-21T12:29:54.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:54 smithi114 ceph-mon[40867]: osdmap e270: 8 total, 8 up, 7 in 2024-09-21T12:29:54.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:54 smithi114 ceph-mon[40867]: 6.19 scrub starts 2024-09-21T12:29:54.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:54 smithi114 ceph-mon[40867]: 6.19 scrub ok 2024-09-21T12:29:54.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:54 smithi114 ceph-mon[30324]: osdmap e270: 8 total, 8 up, 7 in 2024-09-21T12:29:54.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:54 smithi114 ceph-mon[30324]: 6.19 scrub starts 2024-09-21T12:29:54.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:54 smithi114 ceph-mon[30324]: 6.19 scrub ok 2024-09-21T12:29:54.531 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:54.555 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:54.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:54.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:54.656 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:54.684 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:54.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:54.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:54.786 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:54.823 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:54.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:54.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:54.841 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 87 16 122450 122434 87.9385 49 0.0166994 0.0113664 2024-09-21T12:29:54.925 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:54.956 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:54.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:54.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:55.057 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:55.080 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:55.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:55.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:55.182 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:55.207 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:55.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:55.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:55.309 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:55.333 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:55.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:55.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:55.435 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:55.455 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:55.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:55.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:55.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:55 smithi183 ceph-mon[33277]: pgmap v472: 178 pgs: 6 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 7.4 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 1.0 KiB/s rd, 54 MiB/s wr, 859 op/s; 61968/362085 objects misplaced (17.114%); 48 MiB/s, 770 objects/s recovering 2024-09-21T12:29:55.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:55 smithi183 ceph-mon[33277]: 3.16 scrub starts 2024-09-21T12:29:55.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:55 smithi183 ceph-mon[33277]: 3.16 scrub ok 2024-09-21T12:29:55.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:55 smithi114 ceph-mon[30324]: pgmap v472: 178 pgs: 6 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 7.4 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 1.0 KiB/s rd, 54 MiB/s wr, 859 op/s; 61968/362085 objects misplaced (17.114%); 48 MiB/s, 770 objects/s recovering 2024-09-21T12:29:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:55 smithi114 ceph-mon[30324]: 3.16 scrub starts 2024-09-21T12:29:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:55 smithi114 ceph-mon[30324]: 3.16 scrub ok 2024-09-21T12:29:55.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:55 smithi114 ceph-mon[40867]: pgmap v472: 178 pgs: 6 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 7.4 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 1.0 KiB/s rd, 54 MiB/s wr, 859 op/s; 61968/362085 objects misplaced (17.114%); 48 MiB/s, 770 objects/s recovering 2024-09-21T12:29:55.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:55 smithi114 ceph-mon[40867]: 3.16 scrub starts 2024-09-21T12:29:55.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:55 smithi114 ceph-mon[40867]: 3.16 scrub ok 2024-09-21T12:29:55.557 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:55.579 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:55.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:55.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:55.681 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:55.702 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:55.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:55.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:55.803 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:55.825 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:55.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:55.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:55.841 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 88 16 122826 122810 87.2063 23.5 0.0274259 0.0113959 2024-09-21T12:29:55.926 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:55.947 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:55.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:55.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:56.049 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:56.079 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:56.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:56.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:56.181 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:56.204 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:56.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:56.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:56.306 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:56.327 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:56.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:56.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:56.429 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:56.450 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:56.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:56.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:56.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:56 smithi183 ceph-mon[33277]: 9.f scrub ok 2024-09-21T12:29:56.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:56 smithi183 ceph-mon[33277]: osdmap e271: 8 total, 8 up, 7 in 2024-09-21T12:29:56.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:56 smithi183 ceph-mon[33277]: 1.17 scrub starts 2024-09-21T12:29:56.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:56 smithi183 ceph-mon[33277]: 1.17 scrub ok 2024-09-21T12:29:56.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:56 smithi114 ceph-mon[40867]: 9.f scrub ok 2024-09-21T12:29:56.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:56 smithi114 ceph-mon[40867]: osdmap e271: 8 total, 8 up, 7 in 2024-09-21T12:29:56.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:56 smithi114 ceph-mon[40867]: 1.17 scrub starts 2024-09-21T12:29:56.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:56 smithi114 ceph-mon[40867]: 1.17 scrub ok 2024-09-21T12:29:56.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:56 smithi114 ceph-mon[30324]: 9.f scrub ok 2024-09-21T12:29:56.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:56 smithi114 ceph-mon[30324]: osdmap e271: 8 total, 8 up, 7 in 2024-09-21T12:29:56.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:56 smithi114 ceph-mon[30324]: 1.17 scrub starts 2024-09-21T12:29:56.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:56 smithi114 ceph-mon[30324]: 1.17 scrub ok 2024-09-21T12:29:56.551 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:56.581 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:56.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:56.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:56.682 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:56.710 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:56.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:56.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:56.811 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:29:56.839 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:56.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:56.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:29:56.841 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 89 16 123189 123173 86.4814 22.6875 0.0173722 0.0115585 2024-09-21T12:29:56.940 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:56.970 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:56.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:56.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:57.072 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:57.097 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:57.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:57.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:57.199 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:57.224 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:57.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:57.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:57.326 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:57.347 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:57.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:57.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:57.449 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:57.476 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:57.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:57.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:57.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:57 smithi183 ceph-mon[33277]: pgmap v474: 178 pgs: 6 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 7.4 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 639 B/s rd, 30 MiB/s wr, 480 op/s; 61968/362085 objects misplaced (17.114%); 30 MiB/s, 474 objects/s recovering 2024-09-21T12:29:57.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:57 smithi183 ceph-mon[33277]: osdmap e272: 8 total, 8 up, 7 in 2024-09-21T12:29:57.488 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:29:57 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:29:57.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:57 smithi114 ceph-mon[40867]: pgmap v474: 178 pgs: 6 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 7.4 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 639 B/s rd, 30 MiB/s wr, 480 op/s; 61968/362085 objects misplaced (17.114%); 30 MiB/s, 474 objects/s recovering 2024-09-21T12:29:57.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:57 smithi114 ceph-mon[40867]: osdmap e272: 8 total, 8 up, 7 in 2024-09-21T12:29:57.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:57 smithi114 ceph-mon[30324]: pgmap v474: 178 pgs: 6 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 170 active+clean; 7.4 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 639 B/s rd, 30 MiB/s wr, 480 op/s; 61968/362085 objects misplaced (17.114%); 30 MiB/s, 474 objects/s recovering 2024-09-21T12:29:57.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:57 smithi114 ceph-mon[30324]: osdmap e272: 8 total, 8 up, 7 in 2024-09-21T12:29:57.578 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:57.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:57.600 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:57.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:57.701 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:57.738 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:57.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:57.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:57.839 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:57.842 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Total time run: 90.0103 2024-09-21T12:29:57.842 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Total writes made: 124031 2024-09-21T12:29:57.843 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Write size: 65536 2024-09-21T12:29:57.843 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Object size: 65536 2024-09-21T12:29:57.843 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Bandwidth (MB/sec): 86.1227 2024-09-21T12:29:57.843 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev Bandwidth: 37.5236 2024-09-21T12:29:57.843 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max bandwidth (MB/sec): 206.438 2024-09-21T12:29:57.843 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min bandwidth (MB/sec): 0 2024-09-21T12:29:57.843 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Average IOPS: 1377 2024-09-21T12:29:57.843 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev IOPS: 600.385 2024-09-21T12:29:57.843 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max IOPS: 3303 2024-09-21T12:29:57.843 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min IOPS: 0 2024-09-21T12:29:57.843 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Average Latency(s): 0.0116074 2024-09-21T12:29:57.843 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev Latency(s): 0.0278597 2024-09-21T12:29:57.843 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max latency(s): 2.12297 2024-09-21T12:29:57.844 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min latency(s): 0.00136588 2024-09-21T12:29:57.848 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T12:29:57.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:57.874 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:57.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:57.975 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:57.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:57.997 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:57.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:58.098 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:58.140 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:58.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:58.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:58.241 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:58.276 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:58.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:58.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:58.377 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:58.399 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:58.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:58.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:58.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:58 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:29:58.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:58 smithi183 ceph-mon[33277]: 3.5 scrub starts 2024-09-21T12:29:58.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:58 smithi183 ceph-mon[33277]: 3.5 scrub ok 2024-09-21T12:29:58.500 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:29:58.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:58 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:29:58.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:58 smithi114 ceph-mon[30324]: 3.5 scrub starts 2024-09-21T12:29:58.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:58 smithi114 ceph-mon[30324]: 3.5 scrub ok 2024-09-21T12:29:58.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:58 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:29:58.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:58 smithi114 ceph-mon[40867]: 3.5 scrub starts 2024-09-21T12:29:58.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:58 smithi114 ceph-mon[40867]: 3.5 scrub ok 2024-09-21T12:29:58.543 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:58.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:58.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:29:58.644 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:58.669 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:58.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:58.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:58.771 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:58.792 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:58.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:58.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:58.893 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:29:58.931 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:58.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:58.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:29:59.033 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:29:59.054 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:59.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:59.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:29:59.156 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:29:59.181 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:59.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:59.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:29:59.282 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:59.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:59.309 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:59.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:29:59.410 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:59.432 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:59.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:59.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:59 smithi183 ceph-mon[33277]: pgmap v476: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 7.5 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 853 B/s rd, 61 MiB/s wr, 971 op/s; 58783/370947 objects misplaced (15.847%); 59 MiB/s, 944 objects/s recovering 2024-09-21T12:29:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:59 smithi183 ceph-mon[33277]: 4.8 scrub starts 2024-09-21T12:29:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:59 smithi183 ceph-mon[33277]: 4.8 scrub ok 2024-09-21T12:29:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:59 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:29:59.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:29:59 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:29:59.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:59 smithi114 ceph-mon[40867]: pgmap v476: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 7.5 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 853 B/s rd, 61 MiB/s wr, 971 op/s; 58783/370947 objects misplaced (15.847%); 59 MiB/s, 944 objects/s recovering 2024-09-21T12:29:59.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:59 smithi114 ceph-mon[40867]: 4.8 scrub starts 2024-09-21T12:29:59.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:59 smithi114 ceph-mon[40867]: 4.8 scrub ok 2024-09-21T12:29:59.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:59 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:29:59.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:29:59 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:29:59.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:59 smithi114 ceph-mon[30324]: pgmap v476: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 7.5 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 853 B/s rd, 61 MiB/s wr, 971 op/s; 58783/370947 objects misplaced (15.847%); 59 MiB/s, 944 objects/s recovering 2024-09-21T12:29:59.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:59 smithi114 ceph-mon[30324]: 4.8 scrub starts 2024-09-21T12:29:59.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:59 smithi114 ceph-mon[30324]: 4.8 scrub ok 2024-09-21T12:29:59.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:59 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:29:59.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:29:59 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:29:59.534 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:59.570 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:59.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:59.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:59.672 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:29:59.713 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:59.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:59.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:29:59.815 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:29:59.848 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:29:59.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:59.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:29:59.950 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:29:59.976 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:29:59.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:29:59.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:00.078 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:00.099 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:00.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:00.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:00.200 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:00.227 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:00.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:00.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:00.329 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:00.354 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:00.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:00.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:00.456 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:00.477 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:00.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:00.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:00.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:00 smithi183 ceph-mon[33277]: osdmap e273: 8 total, 8 up, 7 in 2024-09-21T12:30:00.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:00 smithi183 ceph-mon[33277]: Health detail: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull 2024-09-21T12:30:00.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:00 smithi183 ceph-mon[33277]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull 2024-09-21T12:30:00.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:00 smithi183 ceph-mon[33277]: pg 9.0 is active+remapped+backfill_toofull, acting [2,6,1] 2024-09-21T12:30:00.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:00 smithi183 ceph-mon[33277]: pg 9.7 is active+remapped+backfill_toofull, acting [3,5,7] 2024-09-21T12:30:00.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:00 smithi114 ceph-mon[40867]: osdmap e273: 8 total, 8 up, 7 in 2024-09-21T12:30:00.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:00 smithi114 ceph-mon[40867]: Health detail: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull 2024-09-21T12:30:00.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:00 smithi114 ceph-mon[40867]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull 2024-09-21T12:30:00.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:00 smithi114 ceph-mon[40867]: pg 9.0 is active+remapped+backfill_toofull, acting [2,6,1] 2024-09-21T12:30:00.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:00 smithi114 ceph-mon[40867]: pg 9.7 is active+remapped+backfill_toofull, acting [3,5,7] 2024-09-21T12:30:00.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:00 smithi114 ceph-mon[30324]: osdmap e273: 8 total, 8 up, 7 in 2024-09-21T12:30:00.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:00 smithi114 ceph-mon[30324]: Health detail: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull 2024-09-21T12:30:00.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:00 smithi114 ceph-mon[30324]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull 2024-09-21T12:30:00.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:00 smithi114 ceph-mon[30324]: pg 9.0 is active+remapped+backfill_toofull, acting [2,6,1] 2024-09-21T12:30:00.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:00 smithi114 ceph-mon[30324]: pg 9.7 is active+remapped+backfill_toofull, acting [3,5,7] 2024-09-21T12:30:00.579 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:00.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:00.611 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:00.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:00.712 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:00.734 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:00.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:00.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:00.835 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:00.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:00.873 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:00.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:00.974 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:01.001 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:01.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:01.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:01.102 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:01.127 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:01.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:01.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:01.228 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:01.256 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:01.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:01.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:01.357 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:01.397 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:01 smithi114 ceph-mon[30324]: 5.11 scrub starts 2024-09-21T12:30:01.397 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:01 smithi114 ceph-mon[30324]: 5.11 scrub ok 2024-09-21T12:30:01.397 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:01 smithi114 ceph-mon[30324]: pgmap v478: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 7.5 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 853 B/s rd, 31 MiB/s wr, 492 op/s; 58783/370947 objects misplaced (15.847%); 42 MiB/s, 679 objects/s recovering 2024-09-21T12:30:01.398 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:01 smithi114 ceph-mon[40867]: 5.11 scrub starts 2024-09-21T12:30:01.398 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:01 smithi114 ceph-mon[40867]: 5.11 scrub ok 2024-09-21T12:30:01.398 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:01 smithi114 ceph-mon[40867]: pgmap v478: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 7.5 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 853 B/s rd, 31 MiB/s wr, 492 op/s; 58783/370947 objects misplaced (15.847%); 42 MiB/s, 679 objects/s recovering 2024-09-21T12:30:01.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:01.400 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:01.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:01 smithi183 ceph-mon[33277]: 5.11 scrub starts 2024-09-21T12:30:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:01 smithi183 ceph-mon[33277]: 5.11 scrub ok 2024-09-21T12:30:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:01 smithi183 ceph-mon[33277]: pgmap v478: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 7.5 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 853 B/s rd, 31 MiB/s wr, 492 op/s; 58783/370947 objects misplaced (15.847%); 42 MiB/s, 679 objects/s recovering 2024-09-21T12:30:01.501 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:01.526 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:01.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:01.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:01.574 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T12:30:01.628 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:01.653 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:01.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:01.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:01.686 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:30:01.703 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:30:01.755 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:01.778 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:01.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:01.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:01.879 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:01.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:01.928 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:01.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:02.028 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:02.068 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:02.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:02.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:02.170 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:02.191 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:02.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:02.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:02.292 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:02.319 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:02.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:02.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:02.420 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:02.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:02.447 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:02.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:02.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:02 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:30:02.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:02 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:30:02.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:02 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:30:02.548 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:02.584 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:02.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:02.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:02.686 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:02.712 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:02.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:02.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:02.814 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:02.842 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:02.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:02.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:02.944 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:02.979 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:30:02 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:30:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:30:02.988 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:02.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:02.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:03.090 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:03.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:03.134 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:03.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:03.235 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:03.265 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:03.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:03.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:03.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:03 smithi114 ceph-mon[40867]: 3.1d scrub starts 2024-09-21T12:30:03.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:03 smithi114 ceph-mon[40867]: 3.1d scrub ok 2024-09-21T12:30:03.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:03 smithi114 ceph-mon[40867]: pgmap v479: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 7.5 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.7 KiB/s rd, 30 MiB/s wr, 598 op/s; 58783/369351 objects misplaced (15.915%); 38 MiB/s, 612 objects/s recovering 2024-09-21T12:30:03.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:03 smithi114 ceph-mon[30324]: 3.1d scrub starts 2024-09-21T12:30:03.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:03 smithi114 ceph-mon[30324]: 3.1d scrub ok 2024-09-21T12:30:03.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:03 smithi114 ceph-mon[30324]: pgmap v479: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 7.5 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.7 KiB/s rd, 30 MiB/s wr, 598 op/s; 58783/369351 objects misplaced (15.915%); 38 MiB/s, 612 objects/s recovering 2024-09-21T12:30:03.366 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:03.416 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:03.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:03.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:03.440 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:30:03.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:03 smithi183 ceph-mon[33277]: 3.1d scrub starts 2024-09-21T12:30:03.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:03 smithi183 ceph-mon[33277]: 3.1d scrub ok 2024-09-21T12:30:03.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:03 smithi183 ceph-mon[33277]: pgmap v479: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 7.5 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.7 KiB/s rd, 30 MiB/s wr, 598 op/s; 58783/369351 objects misplaced (15.915%); 38 MiB/s, 612 objects/s recovering 2024-09-21T12:30:03.518 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:03.540 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:03.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:03.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:03.641 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:03.662 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:03.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:03.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:03.764 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:03.789 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:03.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:03.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:03.835 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:30:03.838 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:30:03.890 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:03.917 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:03.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:03.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:04.019 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:04.044 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:04.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:04.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:04.146 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:04.192 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:04.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:04.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:04.294 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:04.338 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:04.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:04.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:04.440 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:04.469 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:04.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:04.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:04.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:04 smithi183 ceph-mon[33277]: 4.1f scrub starts 2024-09-21T12:30:04.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:04 smithi183 ceph-mon[33277]: 4.1f scrub ok 2024-09-21T12:30:04.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:04 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1963084394' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:30:04.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:04 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:30:04.518 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:04 smithi114 ceph-mon[40867]: 4.1f scrub starts 2024-09-21T12:30:04.518 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:04 smithi114 ceph-mon[40867]: 4.1f scrub ok 2024-09-21T12:30:04.518 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:04 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1963084394' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:30:04.518 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:04 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:30:04.518 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:04 smithi114 ceph-mon[30324]: 4.1f scrub starts 2024-09-21T12:30:04.519 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:04 smithi114 ceph-mon[30324]: 4.1f scrub ok 2024-09-21T12:30:04.519 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:04 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1963084394' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:30:04.519 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:04 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:30:04.536 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-21T12:30:04.571 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:04.611 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:04.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:04.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:04.712 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:04.751 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:04.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:04.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:04.853 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:04.880 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:04.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:04.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:04.982 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:05.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:05.022 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:05.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:05.123 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:05.159 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:05.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:05.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:05.261 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:05.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:05 smithi114 ceph-mon[40867]: pgmap v480: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 7.3 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.4 KiB/s rd, 28 MiB/s wr, 898 op/s; 55056/361746 objects misplaced (15.220%); 46 MiB/s, 743 objects/s recovering 2024-09-21T12:30:05.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:05 smithi114 ceph-mon[40867]: from='client.25327 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:30:05.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:05 smithi114 ceph-mon[40867]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:30:05.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:05 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:30:05.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:05 smithi114 ceph-mon[40867]: osdmap e274: 8 total, 8 up, 7 in 2024-09-21T12:30:05.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:05 smithi114 ceph-mon[30324]: pgmap v480: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 7.3 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.4 KiB/s rd, 28 MiB/s wr, 898 op/s; 55056/361746 objects misplaced (15.220%); 46 MiB/s, 743 objects/s recovering 2024-09-21T12:30:05.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:05 smithi114 ceph-mon[30324]: from='client.25327 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:30:05.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:05 smithi114 ceph-mon[30324]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:30:05.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:05 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:30:05.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:05 smithi114 ceph-mon[30324]: osdmap e274: 8 total, 8 up, 7 in 2024-09-21T12:30:05.281 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:05.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:05.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:05.383 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:05.412 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:05.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:05.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:05.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:05 smithi183 ceph-mon[33277]: pgmap v480: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 7.3 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.4 KiB/s rd, 28 MiB/s wr, 898 op/s; 55056/361746 objects misplaced (15.220%); 46 MiB/s, 743 objects/s recovering 2024-09-21T12:30:05.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:05 smithi183 ceph-mon[33277]: from='client.25327 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:30:05.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:05 smithi183 ceph-mon[33277]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:30:05.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:05 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:30:05.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:05 smithi183 ceph-mon[33277]: osdmap e274: 8 total, 8 up, 7 in 2024-09-21T12:30:05.514 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:05.553 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:05.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:05.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:05.654 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:05.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:05.689 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:05.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:05.790 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:05.832 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:05.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:05.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:05.934 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:05.980 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:05.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:05.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:06.082 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:06.150 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:06.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:06.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:06.337 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:06.378 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:06.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:06.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:06.480 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:06.509 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:06.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:06.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:06.611 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:06.654 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:06.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:06.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:06.755 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:06.797 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:06.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:06.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:06.832 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:30:06.871 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":480,"stamp":"2024-09-21T12:30:03.763924+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7890122879,"num_objects":120582,"num_object_clones":0,"num_object_copies":361746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":55056,"num_objects_unfound":0,"num_objects_dirty":120582,"num_whiteouts":0,"num_read":1296,"num_read_kb":1056,"num_write":128124,"num_write_kb":7938141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17767,"num_bytes_recovered":1148498219,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65935,"ondisk_log_size":65935,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":543,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":26823912,"kb_used_data":23984192,"kb_used_omap":10,"kb_used_meta":2839285,"kb_avail":629392152,"statfs":{"total":671965249536,"available":644497563648,"internally_reserved":0,"allocated":24559812608,"data_stored":24557309126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":2907427947},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7471080,"num_objects":-113,"num_object_clones":0,"num_object_copies":-339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-6912,"num_objects_unfound":0,"num_objects_dirty":-113,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":7185,"num_write_kb":226241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5954,"num_bytes_recovered":390201344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004672"},"pg_stats":[{"pgid":"1.1d","version":"76'79","reported_seq":145,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.152009+0000","last_change":"2024-09-21T12:29:35.768052+0000","last_active":"2024-09-21T12:29:59.152009+0000","last_peered":"2024-09-21T12:29:59.152009+0000","last_clean":"2024-09-21T12:29:59.152009+0000","last_became_active":"2024-09-21T12:29:35.765079+0000","last_became_peered":"2024-09-21T12:29:35.765079+0000","last_unstale":"2024-09-21T12:29:59.152009+0000","last_undegraded":"2024-09-21T12:29:59.152009+0000","last_fullsized":"2024-09-21T12:29:59.152009+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:47.816137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:47.816137+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:08.137559+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":460,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.152433+0000","last_change":"2024-09-21T12:24:50.115300+0000","last_active":"2024-09-21T12:29:59.152433+0000","last_peered":"2024-09-21T12:29:59.152433+0000","last_clean":"2024-09-21T12:29:59.152433+0000","last_became_active":"2024-09-21T12:20:15.309431+0000","last_became_peered":"2024-09-21T12:20:15.309431+0000","last_unstale":"2024-09-21T12:29:59.152433+0000","last_undegraded":"2024-09-21T12:29:59.152433+0000","last_fullsized":"2024-09-21T12:29:59.152433+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:50.115253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:50.115253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:24.094870+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":88,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.148428+0000","last_change":"2024-09-21T12:29:08.671789+0000","last_active":"2024-09-21T12:29:59.148428+0000","last_peered":"2024-09-21T12:29:59.148428+0000","last_clean":"2024-09-21T12:29:59.148428+0000","last_became_active":"2024-09-21T12:29:05.833372+0000","last_became_peered":"2024-09-21T12:29:05.833372+0000","last_unstale":"2024-09-21T12:29:59.148428+0000","last_undegraded":"2024-09-21T12:29:59.148428+0000","last_fullsized":"2024-09-21T12:29:59.148428+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:29:08.671723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:29:08.671723+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:20.962766+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":457,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.148470+0000","last_change":"2024-09-21T12:29:10.672659+0000","last_active":"2024-09-21T12:29:59.148470+0000","last_peered":"2024-09-21T12:29:59.148470+0000","last_clean":"2024-09-21T12:29:59.148470+0000","last_became_active":"2024-09-21T12:20:19.314908+0000","last_became_peered":"2024-09-21T12:20:19.314908+0000","last_unstale":"2024-09-21T12:29:59.148470+0000","last_undegraded":"2024-09-21T12:29:59.148470+0000","last_fullsized":"2024-09-21T12:29:59.148470+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:10.672599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:29:10.672599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:36.036960+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":389,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.150652+0000","last_change":"2024-09-21T12:29:35.761515+0000","last_active":"2024-09-21T12:29:59.150652+0000","last_peered":"2024-09-21T12:29:59.150652+0000","last_clean":"2024-09-21T12:29:59.150652+0000","last_became_active":"2024-09-21T12:29:35.760508+0000","last_became_peered":"2024-09-21T12:29:35.760508+0000","last_unstale":"2024-09-21T12:29:59.150652+0000","last_undegraded":"2024-09-21T12:29:59.150652+0000","last_fullsized":"2024-09-21T12:29:59.150652+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:14.158965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:29:14.158965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:41.582027+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1c","version":"76'79","reported_seq":128,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.157668+0000","last_change":"2024-09-21T12:27:28.374689+0000","last_active":"2024-09-21T12:29:59.157668+0000","last_peered":"2024-09-21T12:29:59.157668+0000","last_clean":"2024-09-21T12:29:59.157668+0000","last_became_active":"2024-09-21T12:27:16.920545+0000","last_became_peered":"2024-09-21T12:27:16.920545+0000","last_unstale":"2024-09-21T12:29:59.157668+0000","last_undegraded":"2024-09-21T12:29:59.157668+0000","last_fullsized":"2024-09-21T12:29:59.157668+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:28.374616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:28.374616+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:32.518277+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":436,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.157763+0000","last_change":"2024-09-21T12:25:48.316122+0000","last_active":"2024-09-21T12:29:59.157763+0000","last_peered":"2024-09-21T12:29:59.157763+0000","last_clean":"2024-09-21T12:29:59.157763+0000","last_became_active":"2024-09-21T12:20:21.595301+0000","last_became_peered":"2024-09-21T12:20:21.595301+0000","last_unstale":"2024-09-21T12:29:59.157763+0000","last_undegraded":"2024-09-21T12:29:59.157763+0000","last_fullsized":"2024-09-21T12:29:59.157763+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:48.316039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:48.316039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":465,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.157419+0000","last_change":"2024-09-21T12:29:16.079291+0000","last_active":"2024-09-21T12:29:59.157419+0000","last_peered":"2024-09-21T12:29:59.157419+0000","last_clean":"2024-09-21T12:29:59.157419+0000","last_became_active":"2024-09-21T12:20:15.308655+0000","last_became_peered":"2024-09-21T12:20:15.308655+0000","last_unstale":"2024-09-21T12:29:59.157419+0000","last_undegraded":"2024-09-21T12:29:59.157419+0000","last_fullsized":"2024-09-21T12:29:59.157419+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:16.079221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:16.079221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:18.495891+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":486,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.159638+0000","last_change":"2024-09-21T12:27:54.371491+0000","last_active":"2024-09-21T12:29:59.159638+0000","last_peered":"2024-09-21T12:29:59.159638+0000","last_clean":"2024-09-21T12:29:59.159638+0000","last_became_active":"2024-09-21T12:20:17.305220+0000","last_became_peered":"2024-09-21T12:20:17.305220+0000","last_unstale":"2024-09-21T12:29:59.159638+0000","last_undegraded":"2024-09-21T12:29:59.159638+0000","last_fullsized":"2024-09-21T12:29:59.159638+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:27:54.369084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:54.369084+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":399,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.156277+0000","last_change":"2024-09-21T12:28:39.179737+0000","last_active":"2024-09-21T12:29:59.156277+0000","last_peered":"2024-09-21T12:29:59.156277+0000","last_clean":"2024-09-21T12:29:59.156277+0000","last_became_active":"2024-09-21T12:22:50.926240+0000","last_became_peered":"2024-09-21T12:22:50.926240+0000","last_unstale":"2024-09-21T12:29:59.156277+0000","last_undegraded":"2024-09-21T12:29:59.156277+0000","last_fullsized":"2024-09-21T12:29:59.156277+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:39.179665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:39.179665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":303,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:30:01.256907+0000","last_change":"2024-09-21T12:30:01.256907+0000","last_active":"2024-09-21T12:30:01.256907+0000","last_peered":"2024-09-21T12:30:01.256907+0000","last_clean":"2024-09-21T12:30:01.256907+0000","last_became_active":"2024-09-21T12:26:54.083981+0000","last_became_peered":"2024-09-21T12:26:54.083981+0000","last_unstale":"2024-09-21T12:30:01.256907+0000","last_undegraded":"2024-09-21T12:30:01.256907+0000","last_fullsized":"2024-09-21T12:30:01.256907+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:30:01.256865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:30:01.256865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:31:17.456222+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":446,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.158684+0000","last_change":"2024-09-21T12:28:06.589984+0000","last_active":"2024-09-21T12:29:59.158684+0000","last_peered":"2024-09-21T12:29:59.158684+0000","last_clean":"2024-09-21T12:29:59.158684+0000","last_became_active":"2024-09-21T12:22:50.927804+0000","last_became_peered":"2024-09-21T12:22:50.927804+0000","last_unstale":"2024-09-21T12:29:59.158684+0000","last_undegraded":"2024-09-21T12:29:59.158684+0000","last_fullsized":"2024-09-21T12:29:59.158684+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:28:06.589926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:06.589926+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":472,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.149736+0000","last_change":"2024-09-21T12:29:37.148220+0000","last_active":"2024-09-21T12:29:59.149736+0000","last_peered":"2024-09-21T12:29:59.149736+0000","last_clean":"2024-09-21T12:29:59.149736+0000","last_became_active":"2024-09-21T12:29:35.778190+0000","last_became_peered":"2024-09-21T12:29:35.778190+0000","last_unstale":"2024-09-21T12:29:59.149736+0000","last_undegraded":"2024-09-21T12:29:59.149736+0000","last_fullsized":"2024-09-21T12:29:59.149736+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:37.148174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:29:37.148174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:37.365028+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":456,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.151563+0000","last_change":"2024-09-21T12:27:54.959865+0000","last_active":"2024-09-21T12:29:59.151563+0000","last_peered":"2024-09-21T12:29:59.151563+0000","last_clean":"2024-09-21T12:29:59.151563+0000","last_became_active":"2024-09-21T12:20:21.309190+0000","last_became_peered":"2024-09-21T12:20:21.309190+0000","last_unstale":"2024-09-21T12:29:59.151563+0000","last_undegraded":"2024-09-21T12:29:59.151563+0000","last_fullsized":"2024-09-21T12:29:59.151563+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:54.959806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:27:54.959806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"76'79","reported_seq":221,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.157286+0000","last_change":"2024-09-21T12:27:20.216167+0000","last_active":"2024-09-21T12:29:59.157286+0000","last_peered":"2024-09-21T12:29:59.157286+0000","last_clean":"2024-09-21T12:29:59.157286+0000","last_became_active":"2024-09-21T12:27:16.937711+0000","last_became_peered":"2024-09-21T12:27:16.937711+0000","last_unstale":"2024-09-21T12:29:59.157286+0000","last_undegraded":"2024-09-21T12:29:59.157286+0000","last_fullsized":"2024-09-21T12:29:59.157286+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:20.216102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:20.216102+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":303,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.154054+0000","last_change":"2024-09-21T12:28:33.644630+0000","last_active":"2024-09-21T12:29:59.154054+0000","last_peered":"2024-09-21T12:29:59.154054+0000","last_clean":"2024-09-21T12:29:59.154054+0000","last_became_active":"2024-09-21T12:26:54.085055+0000","last_became_peered":"2024-09-21T12:26:54.085055+0000","last_unstale":"2024-09-21T12:29:59.154054+0000","last_undegraded":"2024-09-21T12:29:59.154054+0000","last_fullsized":"2024-09-21T12:29:59.154054+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:33.644570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:33.644570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:00.099267+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":445,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.158002+0000","last_change":"2024-09-21T12:29:35.982520+0000","last_active":"2024-09-21T12:29:59.158002+0000","last_peered":"2024-09-21T12:29:59.158002+0000","last_clean":"2024-09-21T12:29:59.158002+0000","last_became_active":"2024-09-21T12:29:35.767433+0000","last_became_peered":"2024-09-21T12:29:35.767433+0000","last_unstale":"2024-09-21T12:29:59.158002+0000","last_undegraded":"2024-09-21T12:29:59.158002+0000","last_fullsized":"2024-09-21T12:29:59.158002+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-21T12:29:35.982468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:29:35.982468+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:43.856648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":470,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.160235+0000","last_change":"2024-09-21T12:28:04.623499+0000","last_active":"2024-09-21T12:29:59.160235+0000","last_peered":"2024-09-21T12:29:59.160235+0000","last_clean":"2024-09-21T12:29:59.160235+0000","last_became_active":"2024-09-21T12:22:50.921780+0000","last_became_peered":"2024-09-21T12:22:50.921780+0000","last_unstale":"2024-09-21T12:29:59.160235+0000","last_undegraded":"2024-09-21T12:29:59.160235+0000","last_fullsized":"2024-09-21T12:29:59.160235+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:04.623389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:04.623389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":469,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.149471+0000","last_change":"2024-09-21T12:29:53.301655+0000","last_active":"2024-09-21T12:29:59.149471+0000","last_peered":"2024-09-21T12:29:59.149471+0000","last_clean":"2024-09-21T12:29:59.149471+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:29:59.149471+0000","last_undegraded":"2024-09-21T12:29:59.149471+0000","last_fullsized":"2024-09-21T12:29:59.149471+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:53.301583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:29:53.301583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:31:13.015049+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"76'79","reported_seq":292,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.160355+0000","last_change":"2024-09-21T12:27:22.289065+0000","last_active":"2024-09-21T12:29:59.160355+0000","last_peered":"2024-09-21T12:29:59.160355+0000","last_clean":"2024-09-21T12:29:59.160355+0000","last_became_active":"2024-09-21T12:27:16.957569+0000","last_became_peered":"2024-09-21T12:27:16.957569+0000","last_unstale":"2024-09-21T12:29:59.160355+0000","last_undegraded":"2024-09-21T12:29:59.160355+0000","last_fullsized":"2024-09-21T12:29:59.160355+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:22.288989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:22.288989+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":103,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.156867+0000","last_change":"2024-09-21T12:29:05.826853+0000","last_active":"2024-09-21T12:29:59.156867+0000","last_peered":"2024-09-21T12:29:59.156867+0000","last_clean":"2024-09-21T12:29:59.156867+0000","last_became_active":"2024-09-21T12:29:05.826251+0000","last_became_peered":"2024-09-21T12:29:05.826251+0000","last_unstale":"2024-09-21T12:29:59.156867+0000","last_undegraded":"2024-09-21T12:29:59.156867+0000","last_fullsized":"2024-09-21T12:29:59.156867+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:44.845176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:44.845176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:33.080052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":503,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.152687+0000","last_change":"2024-09-21T12:29:35.773690+0000","last_active":"2024-09-21T12:29:59.152687+0000","last_peered":"2024-09-21T12:29:59.152687+0000","last_clean":"2024-09-21T12:29:59.152687+0000","last_became_active":"2024-09-21T12:29:35.772180+0000","last_became_peered":"2024-09-21T12:29:35.772180+0000","last_unstale":"2024-09-21T12:29:59.152687+0000","last_undegraded":"2024-09-21T12:29:59.152687+0000","last_fullsized":"2024-09-21T12:29:59.152687+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:16.148785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:16.148785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:17.347765+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":487,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.155198+0000","last_change":"2024-09-21T12:27:22.516532+0000","last_active":"2024-09-21T12:29:59.155198+0000","last_peered":"2024-09-21T12:29:59.155198+0000","last_clean":"2024-09-21T12:29:59.155198+0000","last_became_active":"2024-09-21T12:20:17.307828+0000","last_became_peered":"2024-09-21T12:20:17.307828+0000","last_unstale":"2024-09-21T12:29:59.155198+0000","last_undegraded":"2024-09-21T12:29:59.155198+0000","last_fullsized":"2024-09-21T12:29:59.155198+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:27:22.516442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:22.516442+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":485,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.152268+0000","last_change":"2024-09-21T12:28:06.636769+0000","last_active":"2024-09-21T12:29:59.152268+0000","last_peered":"2024-09-21T12:29:59.152268+0000","last_clean":"2024-09-21T12:29:59.152268+0000","last_became_active":"2024-09-21T12:25:46.905878+0000","last_became_peered":"2024-09-21T12:25:46.905878+0000","last_unstale":"2024-09-21T12:29:59.152268+0000","last_undegraded":"2024-09-21T12:29:59.152268+0000","last_fullsized":"2024-09-21T12:29:59.152268+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:06.636729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:06.636729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:33.605451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"76'79","reported_seq":393,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.149158+0000","last_change":"2024-09-21T12:29:51.267936+0000","last_active":"2024-09-21T12:29:59.149158+0000","last_peered":"2024-09-21T12:29:59.149158+0000","last_clean":"2024-09-21T12:29:59.149158+0000","last_became_active":"2024-09-21T12:27:16.927791+0000","last_became_peered":"2024-09-21T12:27:16.927791+0000","last_unstale":"2024-09-21T12:29:59.149158+0000","last_undegraded":"2024-09-21T12:29:59.149158+0000","last_fullsized":"2024-09-21T12:29:59.149158+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:51.267884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:51.267884+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:31:04.534329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":452,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.157714+0000","last_change":"2024-09-21T12:29:35.784513+0000","last_active":"2024-09-21T12:29:59.157714+0000","last_peered":"2024-09-21T12:29:59.157714+0000","last_clean":"2024-09-21T12:29:59.157714+0000","last_became_active":"2024-09-21T12:29:35.781172+0000","last_became_peered":"2024-09-21T12:29:35.781172+0000","last_unstale":"2024-09-21T12:29:59.157714+0000","last_undegraded":"2024-09-21T12:29:59.157714+0000","last_fullsized":"2024-09-21T12:29:59.157714+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:45.880383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.880383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:03.712343+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":397,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.150703+0000","last_change":"2024-09-21T12:28:45.726096+0000","last_active":"2024-09-21T12:29:59.150703+0000","last_peered":"2024-09-21T12:29:59.150703+0000","last_clean":"2024-09-21T12:29:59.150703+0000","last_became_active":"2024-09-21T12:22:50.925403+0000","last_became_peered":"2024-09-21T12:22:50.925403+0000","last_unstale":"2024-09-21T12:29:59.150703+0000","last_undegraded":"2024-09-21T12:29:59.150703+0000","last_fullsized":"2024-09-21T12:29:59.150703+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:45.725968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.725968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:08.835537+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":540,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.156613+0000","last_change":"2024-09-21T12:28:30.620035+0000","last_active":"2024-09-21T12:29:59.156613+0000","last_peered":"2024-09-21T12:29:59.156613+0000","last_clean":"2024-09-21T12:29:59.156613+0000","last_became_active":"2024-09-21T12:22:50.912144+0000","last_became_peered":"2024-09-21T12:22:50.912144+0000","last_unstale":"2024-09-21T12:29:59.156613+0000","last_undegraded":"2024-09-21T12:29:59.156613+0000","last_fullsized":"2024-09-21T12:29:59.156613+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:28:30.619954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:30.619954+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":375,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.160666+0000","last_change":"2024-09-21T12:26:52.295004+0000","last_active":"2024-09-21T12:29:59.160666+0000","last_peered":"2024-09-21T12:29:59.160666+0000","last_clean":"2024-09-21T12:29:59.160666+0000","last_became_active":"2024-09-21T12:22:50.918326+0000","last_became_peered":"2024-09-21T12:22:50.918326+0000","last_unstale":"2024-09-21T12:29:59.160666+0000","last_undegraded":"2024-09-21T12:29:59.160666+0000","last_fullsized":"2024-09-21T12:29:59.160666+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:52.294931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:52.294931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"76'79","reported_seq":185,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.149353+0000","last_change":"2024-09-21T12:28:46.294913+0000","last_active":"2024-09-21T12:29:59.149353+0000","last_peered":"2024-09-21T12:29:59.149353+0000","last_clean":"2024-09-21T12:29:59.149353+0000","last_became_active":"2024-09-21T12:27:16.933358+0000","last_became_peered":"2024-09-21T12:27:16.933358+0000","last_unstale":"2024-09-21T12:29:59.149353+0000","last_undegraded":"2024-09-21T12:29:59.149353+0000","last_fullsized":"2024-09-21T12:29:59.149353+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:46.294855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:46.294855+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:07.298788+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":438,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.158156+0000","last_change":"2024-09-21T12:23:51.065780+0000","last_active":"2024-09-21T12:29:59.158156+0000","last_peered":"2024-09-21T12:29:59.158156+0000","last_clean":"2024-09-21T12:29:59.158156+0000","last_became_active":"2024-09-21T12:20:21.312976+0000","last_became_peered":"2024-09-21T12:20:21.312976+0000","last_unstale":"2024-09-21T12:29:59.158156+0000","last_undegraded":"2024-09-21T12:29:59.158156+0000","last_fullsized":"2024-09-21T12:29:59.158156+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:23:51.065733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:51.065733+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:20.808745+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":500,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.148700+0000","last_change":"2024-09-21T12:27:56.597623+0000","last_active":"2024-09-21T12:29:59.148700+0000","last_peered":"2024-09-21T12:29:59.148700+0000","last_clean":"2024-09-21T12:29:59.148700+0000","last_became_active":"2024-09-21T12:22:50.910933+0000","last_became_peered":"2024-09-21T12:22:50.910933+0000","last_unstale":"2024-09-21T12:29:59.148700+0000","last_undegraded":"2024-09-21T12:29:59.148700+0000","last_fullsized":"2024-09-21T12:29:59.148700+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:27:56.597473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:27:56.597473+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:34.612211+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":487,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.152257+0000","last_change":"2024-09-21T12:29:35.808571+0000","last_active":"2024-09-21T12:29:59.152257+0000","last_peered":"2024-09-21T12:29:59.152257+0000","last_clean":"2024-09-21T12:29:59.152257+0000","last_became_active":"2024-09-21T12:29:35.782223+0000","last_became_peered":"2024-09-21T12:29:35.782223+0000","last_unstale":"2024-09-21T12:29:59.152257+0000","last_undegraded":"2024-09-21T12:29:59.152257+0000","last_fullsized":"2024-09-21T12:29:59.152257+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:24:03.059549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:03.059549+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"266'13","reported_seq":234,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.155738+0000","last_change":"2024-09-21T12:29:35.786269+0000","last_active":"2024-09-21T12:29:59.155738+0000","last_peered":"2024-09-21T12:29:59.155738+0000","last_clean":"2024-09-21T12:29:59.155738+0000","last_became_active":"2024-09-21T12:29:35.769891+0000","last_became_peered":"2024-09-21T12:29:35.769891+0000","last_unstale":"2024-09-21T12:29:59.155738+0000","last_undegraded":"2024-09-21T12:29:59.155738+0000","last_fullsized":"2024-09-21T12:29:59.155738+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:26:10.846183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:10.846183+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2","version":"273'7968","reported_seq":12263,"reported_epoch":273,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T12:30:02.222291+0000","last_change":"2024-09-21T12:29:36.859459+0000","last_active":"2024-09-21T12:30:02.222291+0000","last_peered":"2024-09-21T12:30:02.222291+0000","last_clean":"2024-09-21T12:29:34.730529+0000","last_became_active":"2024-09-21T12:29:36.844864+0000","last_became_peered":"2024-09-21T12:29:36.844864+0000","last_unstale":"2024-09-21T12:30:02.222291+0000","last_undegraded":"2024-09-21T12:30:02.222291+0000","last_fullsized":"2024-09-21T12:30:02.222291+0000","mapping_epoch":266,"log_start":"255'3900","ondisk_log_start":"255'3900","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":4068,"log_dups_size":0,"ondisk_log_size":4068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":490209280,"num_objects":7480,"num_object_clones":0,"num_object_copies":22440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5333,"num_objects_unfound":0,"num_objects_dirty":7480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7968,"num_write_kb":494336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2103,"num_bytes_recovered":137822208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,3],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":7480}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"76'79","reported_seq":176,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.156163+0000","last_change":"2024-09-21T12:28:29.446170+0000","last_active":"2024-09-21T12:29:59.156163+0000","last_peered":"2024-09-21T12:29:59.156163+0000","last_clean":"2024-09-21T12:29:59.156163+0000","last_became_active":"2024-09-21T12:27:14.943286+0000","last_became_peered":"2024-09-21T12:27:14.943286+0000","last_unstale":"2024-09-21T12:29:59.156163+0000","last_undegraded":"2024-09-21T12:29:59.156163+0000","last_fullsized":"2024-09-21T12:29:59.156163+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:29.446103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:29.446103+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":317,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.155523+0000","last_change":"2024-09-21T12:27:31.491370+0000","last_active":"2024-09-21T12:29:59.155523+0000","last_peered":"2024-09-21T12:29:59.155523+0000","last_clean":"2024-09-21T12:29:59.155523+0000","last_became_active":"2024-09-21T12:23:46.910112+0000","last_became_peered":"2024-09-21T12:23:46.910112+0000","last_unstale":"2024-09-21T12:29:59.155523+0000","last_undegraded":"2024-09-21T12:29:59.155523+0000","last_fullsized":"2024-09-21T12:29:59.155523+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:27:31.490985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:31.490985+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":27,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.148527+0000","last_change":"2024-09-21T12:29:35.772883+0000","last_active":"2024-09-21T12:29:59.148527+0000","last_peered":"2024-09-21T12:29:59.148527+0000","last_clean":"2024-09-21T12:29:59.148527+0000","last_became_active":"2024-09-21T12:29:35.772590+0000","last_became_peered":"2024-09-21T12:29:35.772590+0000","last_unstale":"2024-09-21T12:29:59.148527+0000","last_undegraded":"2024-09-21T12:29:59.148527+0000","last_fullsized":"2024-09-21T12:29:59.148527+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:05.580847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:05.580847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"210'14","reported_seq":181,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.151828+0000","last_change":"2024-09-21T12:26:54.091833+0000","last_active":"2024-09-21T12:29:59.151828+0000","last_peered":"2024-09-21T12:29:59.151828+0000","last_clean":"2024-09-21T12:29:59.151828+0000","last_became_active":"2024-09-21T12:26:54.091451+0000","last_became_peered":"2024-09-21T12:26:54.091451+0000","last_unstale":"2024-09-21T12:29:59.151828+0000","last_undegraded":"2024-09-21T12:29:59.151828+0000","last_fullsized":"2024-09-21T12:29:59.151828+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'13","last_scrub_stamp":"2024-09-21T12:25:33.474391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:33.474391+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":43,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.154033+0000","last_change":"2024-09-21T12:29:35.802015+0000","last_active":"2024-09-21T12:29:59.154033+0000","last_peered":"2024-09-21T12:29:59.154033+0000","last_clean":"2024-09-21T12:29:59.154033+0000","last_became_active":"2024-09-21T12:29:35.776594+0000","last_became_peered":"2024-09-21T12:29:35.776594+0000","last_unstale":"2024-09-21T12:29:59.154033+0000","last_undegraded":"2024-09-21T12:29:59.154033+0000","last_fullsized":"2024-09-21T12:29:59.154033+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:28:31.580245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:31.580245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"273'7904","reported_seq":22302,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:30:01.671932+0000","last_change":"2024-09-21T12:29:56.127804+0000","last_active":"2024-09-21T12:30:01.671932+0000","last_peered":"2024-09-21T12:30:01.671932+0000","last_clean":"2024-09-21T12:30:01.671932+0000","last_became_active":"2024-09-21T12:29:56.122000+0000","last_became_peered":"2024-09-21T12:29:56.122000+0000","last_unstale":"2024-09-21T12:30:01.671932+0000","last_undegraded":"2024-09-21T12:30:01.671932+0000","last_fullsized":"2024-09-21T12:30:01.671932+0000","mapping_epoch":271,"log_start":"255'3900","ondisk_log_start":"255'3900","created":236,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":4004,"log_dups_size":0,"ondisk_log_size":4004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":493682712,"num_objects":7534,"num_object_clones":0,"num_object_copies":22602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7534,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7904,"num_write_kb":493953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7146,"num_bytes_recovered":468320256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"76'79","reported_seq":382,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.154157+0000","last_change":"2024-09-21T12:27:20.527090+0000","last_active":"2024-09-21T12:29:59.154157+0000","last_peered":"2024-09-21T12:29:59.154157+0000","last_clean":"2024-09-21T12:29:59.154157+0000","last_became_active":"2024-09-21T12:27:14.941905+0000","last_became_peered":"2024-09-21T12:27:14.941905+0000","last_unstale":"2024-09-21T12:29:59.154157+0000","last_undegraded":"2024-09-21T12:29:59.154157+0000","last_fullsized":"2024-09-21T12:29:59.154157+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:20.526951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:20.526951+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:04.224114+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":518,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.156701+0000","last_change":"2024-09-21T12:29:35.809258+0000","last_active":"2024-09-21T12:29:59.156701+0000","last_peered":"2024-09-21T12:29:59.156701+0000","last_clean":"2024-09-21T12:29:59.156701+0000","last_became_active":"2024-09-21T12:29:35.772160+0000","last_became_peered":"2024-09-21T12:29:35.772160+0000","last_unstale":"2024-09-21T12:29:59.156701+0000","last_undegraded":"2024-09-21T12:29:59.156701+0000","last_fullsized":"2024-09-21T12:29:59.156701+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-21T12:28:26.632272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:26.632272+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":457,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.158048+0000","last_change":"2024-09-21T12:29:10.089053+0000","last_active":"2024-09-21T12:29:59.158048+0000","last_peered":"2024-09-21T12:29:59.158048+0000","last_clean":"2024-09-21T12:29:59.158048+0000","last_became_active":"2024-09-21T12:20:15.297340+0000","last_became_peered":"2024-09-21T12:20:15.297340+0000","last_unstale":"2024-09-21T12:29:59.158048+0000","last_undegraded":"2024-09-21T12:29:59.158048+0000","last_fullsized":"2024-09-21T12:29:59.158048+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:10.088986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:10.088986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:35.634033+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":407,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.160244+0000","last_change":"2024-09-21T12:29:35.784165+0000","last_active":"2024-09-21T12:29:59.160244+0000","last_peered":"2024-09-21T12:29:59.160244+0000","last_clean":"2024-09-21T12:29:59.160244+0000","last_became_active":"2024-09-21T12:29:35.783649+0000","last_became_peered":"2024-09-21T12:29:35.783649+0000","last_unstale":"2024-09-21T12:29:59.160244+0000","last_undegraded":"2024-09-21T12:29:59.160244+0000","last_fullsized":"2024-09-21T12:29:59.160244+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:33.697208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:33.697208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":399,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.158642+0000","last_change":"2024-09-21T12:29:05.143941+0000","last_active":"2024-09-21T12:29:59.158642+0000","last_peered":"2024-09-21T12:29:59.158642+0000","last_clean":"2024-09-21T12:29:59.158642+0000","last_became_active":"2024-09-21T12:22:50.918940+0000","last_became_peered":"2024-09-21T12:22:50.918940+0000","last_unstale":"2024-09-21T12:29:59.158642+0000","last_undegraded":"2024-09-21T12:29:59.158642+0000","last_fullsized":"2024-09-21T12:29:59.158642+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:05.143736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:29:05.143736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:27.414365+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"273'7942","reported_seq":8026,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:30:02.446741+0000","last_change":"2024-09-21T12:28:20.956665+0000","last_active":"2024-09-21T12:30:02.446741+0000","last_peered":"2024-09-21T12:30:02.446741+0000","last_clean":"2024-09-21T12:30:02.446741+0000","last_became_active":"2024-09-21T12:28:20.955557+0000","last_became_peered":"2024-09-21T12:28:20.955557+0000","last_unstale":"2024-09-21T12:30:02.446741+0000","last_undegraded":"2024-09-21T12:30:02.446741+0000","last_fullsized":"2024-09-21T12:30:02.446741+0000","mapping_epoch":236,"log_start":"255'4000","ondisk_log_start":"255'4000","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3942,"log_dups_size":0,"ondisk_log_size":3942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":487587840,"num_objects":7440,"num_object_clones":0,"num_object_copies":22320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7942,"num_write_kb":492224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"76'79","reported_seq":384,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.160476+0000","last_change":"2024-09-21T12:26:45.354836+0000","last_active":"2024-09-21T12:29:59.160476+0000","last_peered":"2024-09-21T12:29:59.160476+0000","last_clean":"2024-09-21T12:29:59.160476+0000","last_became_active":"2024-09-21T12:26:44.960093+0000","last_became_peered":"2024-09-21T12:26:44.960093+0000","last_unstale":"2024-09-21T12:29:59.160476+0000","last_undegraded":"2024-09-21T12:29:59.160476+0000","last_fullsized":"2024-09-21T12:29:59.160476+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:45.354774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:45.354774+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:35.018407+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":456,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.149595+0000","last_change":"2024-09-21T12:26:51.463759+0000","last_active":"2024-09-21T12:29:59.149595+0000","last_peered":"2024-09-21T12:29:59.149595+0000","last_clean":"2024-09-21T12:29:59.149595+0000","last_became_active":"2024-09-21T12:22:50.926008+0000","last_became_peered":"2024-09-21T12:22:50.926008+0000","last_unstale":"2024-09-21T12:29:59.149595+0000","last_undegraded":"2024-09-21T12:29:59.149595+0000","last_fullsized":"2024-09-21T12:29:59.149595+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-21T12:26:51.463698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:51.463698+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:27.121901+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":513,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.152522+0000","last_change":"2024-09-21T12:29:35.806072+0000","last_active":"2024-09-21T12:29:59.152522+0000","last_peered":"2024-09-21T12:29:59.152522+0000","last_clean":"2024-09-21T12:29:59.152522+0000","last_became_active":"2024-09-21T12:29:35.782812+0000","last_became_peered":"2024-09-21T12:29:35.782812+0000","last_unstale":"2024-09-21T12:29:59.152522+0000","last_undegraded":"2024-09-21T12:29:59.152522+0000","last_fullsized":"2024-09-21T12:29:59.152522+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:28:32.727727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:32.727727+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"121'11","reported_seq":608,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.156556+0000","last_change":"2024-09-21T12:26:14.490045+0000","last_active":"2024-09-21T12:29:59.156556+0000","last_peered":"2024-09-21T12:29:59.156556+0000","last_clean":"2024-09-21T12:29:59.156556+0000","last_became_active":"2024-09-21T12:22:50.930342+0000","last_became_peered":"2024-09-21T12:22:50.930342+0000","last_unstale":"2024-09-21T12:29:59.156556+0000","last_undegraded":"2024-09-21T12:29:59.156556+0000","last_fullsized":"2024-09-21T12:29:59.156556+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:26:14.489972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:14.489972+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:01.026395+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":446,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.156797+0000","last_change":"2024-09-21T12:28:37.686403+0000","last_active":"2024-09-21T12:29:59.156797+0000","last_peered":"2024-09-21T12:29:59.156797+0000","last_clean":"2024-09-21T12:29:59.156797+0000","last_became_active":"2024-09-21T12:20:21.593367+0000","last_became_peered":"2024-09-21T12:20:21.593367+0000","last_unstale":"2024-09-21T12:29:59.156797+0000","last_undegraded":"2024-09-21T12:29:59.156797+0000","last_fullsized":"2024-09-21T12:29:59.156797+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:37.686350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:37.686350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"273'7987","reported_seq":8155,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:30:01.694851+0000","last_change":"2024-09-21T12:29:54.985804+0000","last_active":"2024-09-21T12:30:01.694851+0000","last_peered":"2024-09-21T12:30:01.694851+0000","last_clean":"2024-09-21T12:30:01.694851+0000","last_became_active":"2024-09-21T12:28:20.956619+0000","last_became_peered":"2024-09-21T12:28:20.956619+0000","last_unstale":"2024-09-21T12:30:01.694851+0000","last_undegraded":"2024-09-21T12:30:01.694851+0000","last_fullsized":"2024-09-21T12:30:01.694851+0000","mapping_epoch":236,"log_start":"255'4100","ondisk_log_start":"255'4100","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'7729","last_scrub_stamp":"2024-09-21T12:29:54.985720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:29:54.985720+0000","objects_scrubbed":7675,"log_size":3887,"log_dups_size":0,"ondisk_log_size":3887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:31:00.198248+0000","scrub_duration":2694,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":501153792,"num_objects":7647,"num_object_clones":0,"num_object_copies":22941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7987,"num_write_kb":500288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"76'79","reported_seq":187,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.157306+0000","last_change":"2024-09-21T12:29:35.768692+0000","last_active":"2024-09-21T12:29:59.157306+0000","last_peered":"2024-09-21T12:29:59.157306+0000","last_clean":"2024-09-21T12:29:59.157306+0000","last_became_active":"2024-09-21T12:29:35.768234+0000","last_became_peered":"2024-09-21T12:29:35.768234+0000","last_unstale":"2024-09-21T12:29:59.157306+0000","last_undegraded":"2024-09-21T12:29:59.157306+0000","last_fullsized":"2024-09-21T12:29:59.157306+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:06.154040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:06.154040+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13417,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":5,"num_write":2,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":13417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":227,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.151847+0000","last_change":"2024-09-21T12:29:35.795971+0000","last_active":"2024-09-21T12:29:59.151847+0000","last_peered":"2024-09-21T12:29:59.151847+0000","last_clean":"2024-09-21T12:29:59.151847+0000","last_became_active":"2024-09-21T12:29:35.765303+0000","last_became_peered":"2024-09-21T12:29:35.765303+0000","last_unstale":"2024-09-21T12:29:59.151847+0000","last_undegraded":"2024-09-21T12:29:59.151847+0000","last_fullsized":"2024-09-21T12:29:59.151847+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:27:27.325105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:27.325105+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":474,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.149901+0000","last_change":"2024-09-21T12:29:57.398083+0000","last_active":"2024-09-21T12:29:59.149901+0000","last_peered":"2024-09-21T12:29:59.149901+0000","last_clean":"2024-09-21T12:29:59.149901+0000","last_became_active":"2024-09-21T12:20:15.308518+0000","last_became_peered":"2024-09-21T12:20:15.308518+0000","last_unstale":"2024-09-21T12:29:59.149901+0000","last_undegraded":"2024-09-21T12:29:59.149901+0000","last_fullsized":"2024-09-21T12:29:59.149901+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:57.398039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:57.398039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:31:00.876322+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":589,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.152988+0000","last_change":"2024-09-21T12:27:58.871800+0000","last_active":"2024-09-21T12:29:59.152988+0000","last_peered":"2024-09-21T12:29:59.152988+0000","last_clean":"2024-09-21T12:29:59.152988+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:29:59.152988+0000","last_undegraded":"2024-09-21T12:29:59.152988+0000","last_fullsized":"2024-09-21T12:29:59.152988+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:27:58.871710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:27:58.871710+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":456,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.151681+0000","last_change":"2024-09-21T12:28:37.850166+0000","last_active":"2024-09-21T12:29:59.151681+0000","last_peered":"2024-09-21T12:29:59.151681+0000","last_clean":"2024-09-21T12:29:59.151681+0000","last_became_active":"2024-09-21T12:20:21.315801+0000","last_became_peered":"2024-09-21T12:20:21.315801+0000","last_unstale":"2024-09-21T12:29:59.151681+0000","last_undegraded":"2024-09-21T12:29:59.151681+0000","last_fullsized":"2024-09-21T12:29:59.151681+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:37.850100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:37.850100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"273'8067","reported_seq":8151,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:30:01.677301+0000","last_change":"2024-09-21T12:28:20.958462+0000","last_active":"2024-09-21T12:30:01.677301+0000","last_peered":"2024-09-21T12:30:01.677301+0000","last_clean":"2024-09-21T12:30:01.677301+0000","last_became_active":"2024-09-21T12:28:20.958143+0000","last_became_peered":"2024-09-21T12:28:20.958143+0000","last_unstale":"2024-09-21T12:30:01.677301+0000","last_undegraded":"2024-09-21T12:30:01.677301+0000","last_fullsized":"2024-09-21T12:30:01.677301+0000","mapping_epoch":236,"log_start":"255'4000","ondisk_log_start":"255'4000","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":4067,"log_dups_size":0,"ondisk_log_size":4067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":499580928,"num_objects":7623,"num_object_clones":0,"num_object_copies":22869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8067,"num_write_kb":502080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"76'79","reported_seq":142,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.151932+0000","last_change":"2024-09-21T12:29:08.081975+0000","last_active":"2024-09-21T12:29:59.151932+0000","last_peered":"2024-09-21T12:29:59.151932+0000","last_clean":"2024-09-21T12:29:59.151932+0000","last_became_active":"2024-09-21T12:29:05.835884+0000","last_became_peered":"2024-09-21T12:29:05.835884+0000","last_unstale":"2024-09-21T12:29:59.151932+0000","last_undegraded":"2024-09-21T12:29:59.151932+0000","last_fullsized":"2024-09-21T12:29:59.151932+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:08.081901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:08.081901+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:20.926140+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":492,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.152872+0000","last_change":"2024-09-21T12:26:14.834328+0000","last_active":"2024-09-21T12:29:59.152872+0000","last_peered":"2024-09-21T12:29:59.152872+0000","last_clean":"2024-09-21T12:29:59.152872+0000","last_became_active":"2024-09-21T12:20:17.693067+0000","last_became_peered":"2024-09-21T12:20:17.693067+0000","last_unstale":"2024-09-21T12:29:59.152872+0000","last_undegraded":"2024-09-21T12:29:59.152872+0000","last_fullsized":"2024-09-21T12:29:59.152872+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-21T12:26:14.834236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:14.834236+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:18.108327+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":450,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.154135+0000","last_change":"2024-09-21T12:26:12.131784+0000","last_active":"2024-09-21T12:29:59.154135+0000","last_peered":"2024-09-21T12:29:59.154135+0000","last_clean":"2024-09-21T12:29:59.154135+0000","last_became_active":"2024-09-21T12:20:15.308441+0000","last_became_peered":"2024-09-21T12:20:15.308441+0000","last_unstale":"2024-09-21T12:29:59.154135+0000","last_undegraded":"2024-09-21T12:29:59.154135+0000","last_fullsized":"2024-09-21T12:29:59.154135+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:12.131721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:12.131721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":46,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.152182+0000","last_change":"2024-09-21T12:29:05.826167+0000","last_active":"2024-09-21T12:29:59.152182+0000","last_peered":"2024-09-21T12:29:59.152182+0000","last_clean":"2024-09-21T12:29:59.152182+0000","last_became_active":"2024-09-21T12:29:05.825869+0000","last_became_peered":"2024-09-21T12:29:05.825869+0000","last_unstale":"2024-09-21T12:29:59.152182+0000","last_undegraded":"2024-09-21T12:29:59.152182+0000","last_fullsized":"2024-09-21T12:29:59.152182+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:48.999878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:48.999878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:10.986945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":438,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.148381+0000","last_change":"2024-09-21T12:25:44.918356+0000","last_active":"2024-09-21T12:29:59.148381+0000","last_peered":"2024-09-21T12:29:59.148381+0000","last_clean":"2024-09-21T12:29:59.148381+0000","last_became_active":"2024-09-21T12:20:21.592828+0000","last_became_peered":"2024-09-21T12:20:21.592828+0000","last_unstale":"2024-09-21T12:29:59.148381+0000","last_undegraded":"2024-09-21T12:29:59.148381+0000","last_fullsized":"2024-09-21T12:29:59.148381+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:44.918276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:44.918276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:27.631123+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"273'7927","reported_seq":8012,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:30:02.499903+0000","last_change":"2024-09-21T12:28:20.952252+0000","last_active":"2024-09-21T12:30:02.499903+0000","last_peered":"2024-09-21T12:30:02.499903+0000","last_clean":"2024-09-21T12:30:02.499903+0000","last_became_active":"2024-09-21T12:28:20.951980+0000","last_became_peered":"2024-09-21T12:28:20.951980+0000","last_unstale":"2024-09-21T12:30:02.499903+0000","last_undegraded":"2024-09-21T12:30:02.499903+0000","last_fullsized":"2024-09-21T12:30:02.499903+0000","mapping_epoch":236,"log_start":"256'4268","ondisk_log_start":"256'4268","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3659,"log_dups_size":0,"ondisk_log_size":3659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":488701952,"num_objects":7457,"num_object_clones":0,"num_object_copies":22371,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7927,"num_write_kb":492288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"76'79","reported_seq":250,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.154040+0000","last_change":"2024-09-21T12:28:25.714131+0000","last_active":"2024-09-21T12:29:59.154040+0000","last_peered":"2024-09-21T12:29:59.154040+0000","last_clean":"2024-09-21T12:29:59.154040+0000","last_became_active":"2024-09-21T12:26:54.092726+0000","last_became_peered":"2024-09-21T12:26:54.092726+0000","last_unstale":"2024-09-21T12:29:59.154040+0000","last_undegraded":"2024-09-21T12:29:59.154040+0000","last_fullsized":"2024-09-21T12:29:59.154040+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:25.714084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:25.714084+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":612,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.153978+0000","last_change":"2024-09-21T12:26:09.082395+0000","last_active":"2024-09-21T12:29:59.153978+0000","last_peered":"2024-09-21T12:29:59.153978+0000","last_clean":"2024-09-21T12:29:59.153978+0000","last_became_active":"2024-09-21T12:20:17.307769+0000","last_became_peered":"2024-09-21T12:20:17.307769+0000","last_unstale":"2024-09-21T12:29:59.153978+0000","last_undegraded":"2024-09-21T12:29:59.153978+0000","last_fullsized":"2024-09-21T12:29:59.153978+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-21T12:26:09.082314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:09.082314+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":392,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.152789+0000","last_change":"2024-09-21T12:28:31.738463+0000","last_active":"2024-09-21T12:29:59.152789+0000","last_peered":"2024-09-21T12:29:59.152789+0000","last_clean":"2024-09-21T12:29:59.152789+0000","last_became_active":"2024-09-21T12:22:50.922002+0000","last_became_peered":"2024-09-21T12:22:50.922002+0000","last_unstale":"2024-09-21T12:29:59.152789+0000","last_undegraded":"2024-09-21T12:29:59.152789+0000","last_fullsized":"2024-09-21T12:29:59.152789+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:31.738415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:31.738415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":176,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.150607+0000","last_change":"2024-09-21T12:28:37.691938+0000","last_active":"2024-09-21T12:29:59.150607+0000","last_peered":"2024-09-21T12:29:59.150607+0000","last_clean":"2024-09-21T12:29:59.150607+0000","last_became_active":"2024-09-21T12:26:54.083584+0000","last_became_peered":"2024-09-21T12:26:54.083584+0000","last_unstale":"2024-09-21T12:29:59.150607+0000","last_undegraded":"2024-09-21T12:29:59.150607+0000","last_fullsized":"2024-09-21T12:29:59.150607+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T12:28:37.691868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:28:37.691868+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":481,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.154188+0000","last_change":"2024-09-21T12:27:21.956252+0000","last_active":"2024-09-21T12:29:59.154188+0000","last_peered":"2024-09-21T12:29:59.154188+0000","last_clean":"2024-09-21T12:29:59.154188+0000","last_became_active":"2024-09-21T12:22:50.921488+0000","last_became_peered":"2024-09-21T12:22:50.921488+0000","last_unstale":"2024-09-21T12:29:59.154188+0000","last_undegraded":"2024-09-21T12:29:59.154188+0000","last_fullsized":"2024-09-21T12:29:59.154188+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:21.956185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:27:21.956185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":466,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.157210+0000","last_change":"2024-09-21T12:26:10.435410+0000","last_active":"2024-09-21T12:29:59.157210+0000","last_peered":"2024-09-21T12:29:59.157210+0000","last_clean":"2024-09-21T12:29:59.157210+0000","last_became_active":"2024-09-21T12:22:50.917558+0000","last_became_peered":"2024-09-21T12:22:50.917558+0000","last_unstale":"2024-09-21T12:29:59.157210+0000","last_undegraded":"2024-09-21T12:29:59.157210+0000","last_fullsized":"2024-09-21T12:29:59.157210+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:10.435349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:10.435349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"273'8022","reported_seq":11397,"reported_epoch":273,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T12:30:02.142436+0000","last_change":"2024-09-21T12:29:36.874226+0000","last_active":"2024-09-21T12:30:02.142436+0000","last_peered":"2024-09-21T12:30:02.142436+0000","last_clean":"2024-09-21T12:29:34.745065+0000","last_became_active":"2024-09-21T12:29:36.855579+0000","last_became_peered":"2024-09-21T12:29:36.855579+0000","last_unstale":"2024-09-21T12:30:02.142436+0000","last_undegraded":"2024-09-21T12:30:02.142436+0000","last_fullsized":"2024-09-21T12:30:02.142436+0000","mapping_epoch":266,"log_start":"255'4000","ondisk_log_start":"255'4000","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":4022,"log_dups_size":0,"ondisk_log_size":4022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":493223936,"num_objects":7526,"num_object_clones":0,"num_object_copies":22578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5862,"num_objects_unfound":0,"num_objects_dirty":7526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8022,"num_write_kb":497536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1643,"num_bytes_recovered":107675648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,5],"avail_no_missing":["1","5","7"],"object_location_counts":[{"shards":"1,5,7","objects":7526}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"76'79","reported_seq":116,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.151986+0000","last_change":"2024-09-21T12:29:35.771874+0000","last_active":"2024-09-21T12:29:59.151986+0000","last_peered":"2024-09-21T12:29:59.151986+0000","last_clean":"2024-09-21T12:29:59.151986+0000","last_became_active":"2024-09-21T12:29:35.768315+0000","last_became_peered":"2024-09-21T12:29:35.768315+0000","last_unstale":"2024-09-21T12:29:59.151986+0000","last_undegraded":"2024-09-21T12:29:59.151986+0000","last_fullsized":"2024-09-21T12:29:59.151986+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:46.188649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:46.188649+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":86,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.151467+0000","last_change":"2024-09-21T12:29:35.794153+0000","last_active":"2024-09-21T12:29:59.151467+0000","last_peered":"2024-09-21T12:29:59.151467+0000","last_clean":"2024-09-21T12:29:59.151467+0000","last_became_active":"2024-09-21T12:29:35.768686+0000","last_became_peered":"2024-09-21T12:29:35.768686+0000","last_unstale":"2024-09-21T12:29:59.151467+0000","last_undegraded":"2024-09-21T12:29:59.151467+0000","last_fullsized":"2024-09-21T12:29:59.151467+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-21T12:28:25.549320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:25.549320+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":474,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.148140+0000","last_change":"2024-09-21T12:28:39.705480+0000","last_active":"2024-09-21T12:29:59.148140+0000","last_peered":"2024-09-21T12:29:59.148140+0000","last_clean":"2024-09-21T12:29:59.148140+0000","last_became_active":"2024-09-21T12:20:15.308356+0000","last_became_peered":"2024-09-21T12:20:15.308356+0000","last_unstale":"2024-09-21T12:29:59.148140+0000","last_undegraded":"2024-09-21T12:29:59.148140+0000","last_fullsized":"2024-09-21T12:29:59.148140+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:39.705175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:39.705175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":484,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.156826+0000","last_change":"2024-09-21T12:28:43.390160+0000","last_active":"2024-09-21T12:29:59.156826+0000","last_peered":"2024-09-21T12:29:59.156826+0000","last_clean":"2024-09-21T12:29:59.156826+0000","last_became_active":"2024-09-21T12:20:14.288358+0000","last_became_peered":"2024-09-21T12:20:14.288358+0000","last_unstale":"2024-09-21T12:29:59.156826+0000","last_undegraded":"2024-09-21T12:29:59.156826+0000","last_fullsized":"2024-09-21T12:29:59.156826+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:28:43.390087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:28:43.390087+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:12.445374+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0019042449999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":467,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.155114+0000","last_change":"2024-09-21T12:29:13.200503+0000","last_active":"2024-09-21T12:29:59.155114+0000","last_peered":"2024-09-21T12:29:59.155114+0000","last_clean":"2024-09-21T12:29:59.155114+0000","last_became_active":"2024-09-21T12:20:19.309724+0000","last_became_peered":"2024-09-21T12:20:19.309724+0000","last_unstale":"2024-09-21T12:29:59.155114+0000","last_undegraded":"2024-09-21T12:29:59.155114+0000","last_fullsized":"2024-09-21T12:29:59.155114+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:13.200321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:29:13.200321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:21.207169+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":438,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.158448+0000","last_change":"2024-09-21T12:27:33.395611+0000","last_active":"2024-09-21T12:29:59.158448+0000","last_peered":"2024-09-21T12:29:59.158448+0000","last_clean":"2024-09-21T12:29:59.158448+0000","last_became_active":"2024-09-21T12:20:21.592982+0000","last_became_peered":"2024-09-21T12:20:21.592982+0000","last_unstale":"2024-09-21T12:29:59.158448+0000","last_undegraded":"2024-09-21T12:29:59.158448+0000","last_fullsized":"2024-09-21T12:29:59.158448+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:33.395568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:27:33.395568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"273'8118","reported_seq":8203,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:30:02.514754+0000","last_change":"2024-09-21T12:28:21.114897+0000","last_active":"2024-09-21T12:30:02.514754+0000","last_peered":"2024-09-21T12:30:02.514754+0000","last_clean":"2024-09-21T12:30:02.514754+0000","last_became_active":"2024-09-21T12:28:21.114609+0000","last_became_peered":"2024-09-21T12:28:21.114609+0000","last_unstale":"2024-09-21T12:30:02.514754+0000","last_undegraded":"2024-09-21T12:30:02.514754+0000","last_fullsized":"2024-09-21T12:30:02.514754+0000","mapping_epoch":236,"log_start":"256'4411","ondisk_log_start":"256'4411","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3707,"log_dups_size":0,"ondisk_log_size":3707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":498860032,"num_objects":7612,"num_object_clones":0,"num_object_copies":22836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8118,"num_write_kb":503360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"76'79","reported_seq":408,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.150894+0000","last_change":"2024-09-21T12:29:11.678268+0000","last_active":"2024-09-21T12:29:59.150894+0000","last_peered":"2024-09-21T12:29:59.150894+0000","last_clean":"2024-09-21T12:29:59.150894+0000","last_became_active":"2024-09-21T12:26:44.949867+0000","last_became_peered":"2024-09-21T12:26:44.949867+0000","last_unstale":"2024-09-21T12:29:59.150894+0000","last_undegraded":"2024-09-21T12:29:59.150894+0000","last_fullsized":"2024-09-21T12:29:59.150894+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:11.677923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:11.677923+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:27.641123+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":482,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.152341+0000","last_change":"2024-09-21T12:28:45.783033+0000","last_active":"2024-09-21T12:29:59.152341+0000","last_peered":"2024-09-21T12:29:59.152341+0000","last_clean":"2024-09-21T12:29:59.152341+0000","last_became_active":"2024-09-21T12:20:17.299325+0000","last_became_peered":"2024-09-21T12:20:17.299325+0000","last_unstale":"2024-09-21T12:29:59.152341+0000","last_undegraded":"2024-09-21T12:29:59.152341+0000","last_fullsized":"2024-09-21T12:29:59.152341+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:28:45.782955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.782955+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:03.191455+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":496,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.153520+0000","last_change":"2024-09-21T12:29:35.764002+0000","last_active":"2024-09-21T12:29:59.153520+0000","last_peered":"2024-09-21T12:29:59.153520+0000","last_clean":"2024-09-21T12:29:59.153520+0000","last_became_active":"2024-09-21T12:29:35.763587+0000","last_became_peered":"2024-09-21T12:29:35.763587+0000","last_unstale":"2024-09-21T12:29:59.153520+0000","last_undegraded":"2024-09-21T12:29:59.153520+0000","last_fullsized":"2024-09-21T12:29:59.153520+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:15.115449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:15.115449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:30.532202+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":1057,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.787271+0000","last_change":"2024-09-21T12:28:34.639438+0000","last_active":"2024-09-21T12:29:59.787271+0000","last_peered":"2024-09-21T12:29:59.787271+0000","last_clean":"2024-09-21T12:29:59.787271+0000","last_became_active":"2024-09-21T12:20:14.287543+0000","last_became_peered":"2024-09-21T12:20:14.287543+0000","last_unstale":"2024-09-21T12:29:59.787271+0000","last_undegraded":"2024-09-21T12:29:59.787271+0000","last_fullsized":"2024-09-21T12:29:59.787271+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-21T12:28:34.639361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:28:34.639361+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":580,"num_read_kb":575,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":28,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.152630+0000","last_change":"2024-09-21T12:29:35.767793+0000","last_active":"2024-09-21T12:29:59.152630+0000","last_peered":"2024-09-21T12:29:59.152630+0000","last_clean":"2024-09-21T12:29:59.152630+0000","last_became_active":"2024-09-21T12:29:35.767237+0000","last_became_peered":"2024-09-21T12:29:35.767237+0000","last_unstale":"2024-09-21T12:29:59.152630+0000","last_undegraded":"2024-09-21T12:29:59.152630+0000","last_fullsized":"2024-09-21T12:29:59.152630+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:11.422293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:11.422293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":35,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.149734+0000","last_change":"2024-09-21T12:29:36.145777+0000","last_active":"2024-09-21T12:29:59.149734+0000","last_peered":"2024-09-21T12:29:59.149734+0000","last_clean":"2024-09-21T12:29:59.149734+0000","last_became_active":"2024-09-21T12:29:35.780184+0000","last_became_peered":"2024-09-21T12:29:35.780184+0000","last_unstale":"2024-09-21T12:29:59.149734+0000","last_undegraded":"2024-09-21T12:29:59.149734+0000","last_fullsized":"2024-09-21T12:29:59.149734+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:36.145708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:29:36.145708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:39.520222+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"273'7974","reported_seq":13516,"reported_epoch":273,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T12:30:02.522230+0000","last_change":"2024-09-21T12:29:36.915959+0000","last_active":"2024-09-21T12:30:02.522230+0000","last_peered":"2024-09-21T12:30:02.522230+0000","last_clean":"2024-09-21T12:29:34.727203+0000","last_became_active":"2024-09-21T12:29:36.901917+0000","last_became_peered":"2024-09-21T12:29:36.901917+0000","last_unstale":"2024-09-21T12:30:02.522230+0000","last_undegraded":"2024-09-21T12:30:02.522230+0000","last_fullsized":"2024-09-21T12:30:02.522230+0000","mapping_epoch":266,"log_start":"257'4300","ondisk_log_start":"257'4300","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3674,"log_dups_size":0,"ondisk_log_size":3674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":489160704,"num_objects":7464,"num_object_clones":0,"num_object_copies":22392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11236,"num_objects_unfound":0,"num_objects_dirty":7464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7974,"num_write_kb":494016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1818,"num_bytes_recovered":119144448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":7464}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"257'20","reported_seq":110,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.158966+0000","last_change":"2024-09-21T12:29:09.625844+0000","last_active":"2024-09-21T12:29:59.158966+0000","last_peered":"2024-09-21T12:29:59.158966+0000","last_clean":"2024-09-21T12:29:59.158966+0000","last_became_active":"2024-09-21T12:29:05.827560+0000","last_became_peered":"2024-09-21T12:29:05.827560+0000","last_unstale":"2024-09-21T12:29:59.158966+0000","last_undegraded":"2024-09-21T12:29:59.158966+0000","last_fullsized":"2024-09-21T12:29:59.158966+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'20","last_scrub_stamp":"2024-09-21T12:29:09.625763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:29:09.625763+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:36.533631+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":463,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.158815+0000","last_change":"2024-09-21T12:26:12.394327+0000","last_active":"2024-09-21T12:29:59.158815+0000","last_peered":"2024-09-21T12:29:59.158815+0000","last_clean":"2024-09-21T12:29:59.158815+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:29:59.158815+0000","last_undegraded":"2024-09-21T12:29:59.158815+0000","last_fullsized":"2024-09-21T12:29:59.158815+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:26:12.394247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:12.394247+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":542,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.157113+0000","last_change":"2024-09-21T12:26:49.248015+0000","last_active":"2024-09-21T12:29:59.157113+0000","last_peered":"2024-09-21T12:29:59.157113+0000","last_clean":"2024-09-21T12:29:59.157113+0000","last_became_active":"2024-09-21T12:26:44.950132+0000","last_became_peered":"2024-09-21T12:26:44.950132+0000","last_unstale":"2024-09-21T12:29:59.157113+0000","last_undegraded":"2024-09-21T12:29:59.157113+0000","last_fullsized":"2024-09-21T12:29:59.157113+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:49.247952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:49.247952+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:02.353972+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":28,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.154867+0000","last_change":"2024-09-21T12:29:35.765255+0000","last_active":"2024-09-21T12:29:59.154867+0000","last_peered":"2024-09-21T12:29:59.154867+0000","last_clean":"2024-09-21T12:29:59.154867+0000","last_became_active":"2024-09-21T12:29:35.764934+0000","last_became_peered":"2024-09-21T12:29:35.764934+0000","last_unstale":"2024-09-21T12:29:59.154867+0000","last_undegraded":"2024-09-21T12:29:59.154867+0000","last_fullsized":"2024-09-21T12:29:59.154867+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:36.684459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:36.684459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:00.249365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":299,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.157492+0000","last_change":"2024-09-21T12:29:35.779845+0000","last_active":"2024-09-21T12:29:59.157492+0000","last_peered":"2024-09-21T12:29:59.157492+0000","last_clean":"2024-09-21T12:29:59.157492+0000","last_became_active":"2024-09-21T12:29:35.779133+0000","last_became_peered":"2024-09-21T12:29:35.779133+0000","last_unstale":"2024-09-21T12:29:59.157492+0000","last_undegraded":"2024-09-21T12:29:59.157492+0000","last_fullsized":"2024-09-21T12:29:59.157492+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:28.647287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:28.647287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3","version":"273'7904","reported_seq":7988,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:30:01.658983+0000","last_change":"2024-09-21T12:28:20.958439+0000","last_active":"2024-09-21T12:30:01.658983+0000","last_peered":"2024-09-21T12:30:01.658983+0000","last_clean":"2024-09-21T12:30:01.658983+0000","last_became_active":"2024-09-21T12:28:20.958025+0000","last_became_peered":"2024-09-21T12:28:20.958025+0000","last_unstale":"2024-09-21T12:30:01.658983+0000","last_undegraded":"2024-09-21T12:30:01.658983+0000","last_fullsized":"2024-09-21T12:30:01.658983+0000","mapping_epoch":236,"log_start":"255'3900","ondisk_log_start":"255'3900","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":4004,"log_dups_size":0,"ondisk_log_size":4004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":491520000,"num_objects":7500,"num_object_clones":0,"num_object_copies":22500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7904,"num_write_kb":492928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"76'79","reported_seq":404,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.151193+0000","last_change":"2024-09-21T12:27:14.954671+0000","last_active":"2024-09-21T12:29:59.151193+0000","last_peered":"2024-09-21T12:29:59.151193+0000","last_clean":"2024-09-21T12:29:59.151193+0000","last_became_active":"2024-09-21T12:27:14.954351+0000","last_became_peered":"2024-09-21T12:27:14.954351+0000","last_unstale":"2024-09-21T12:29:59.151193+0000","last_undegraded":"2024-09-21T12:29:59.151193+0000","last_fullsized":"2024-09-21T12:29:59.151193+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:48.188322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:48.188322+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:30:24.569371+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":146,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.156999+0000","last_change":"2024-09-21T12:29:05.826584+0000","last_active":"2024-09-21T12:29:59.156999+0000","last_peered":"2024-09-21T12:29:59.156999+0000","last_clean":"2024-09-21T12:29:59.156999+0000","last_became_active":"2024-09-21T12:29:05.826075+0000","last_became_peered":"2024-09-21T12:29:05.826075+0000","last_unstale":"2024-09-21T12:29:59.156999+0000","last_undegraded":"2024-09-21T12:29:59.156999+0000","last_fullsized":"2024-09-21T12:29:59.156999+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:27:57.114076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:57.114076+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":382,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.155282+0000","last_change":"2024-09-21T12:27:30.501480+0000","last_active":"2024-09-21T12:29:59.155282+0000","last_peered":"2024-09-21T12:29:59.155282+0000","last_clean":"2024-09-21T12:29:59.155282+0000","last_became_active":"2024-09-21T12:22:50.908060+0000","last_became_peered":"2024-09-21T12:22:50.908060+0000","last_unstale":"2024-09-21T12:29:59.155282+0000","last_undegraded":"2024-09-21T12:29:59.155282+0000","last_fullsized":"2024-09-21T12:29:59.155282+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:30.498831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:27:30.498831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:02.178308+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":202,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.155611+0000","last_change":"2024-09-21T12:29:15.275262+0000","last_active":"2024-09-21T12:29:59.155611+0000","last_peered":"2024-09-21T12:29:59.155611+0000","last_clean":"2024-09-21T12:29:59.155611+0000","last_became_active":"2024-09-21T12:29:05.825426+0000","last_became_peered":"2024-09-21T12:29:05.825426+0000","last_unstale":"2024-09-21T12:29:59.155611+0000","last_undegraded":"2024-09-21T12:29:59.155611+0000","last_fullsized":"2024-09-21T12:29:59.155611+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:15.275189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:29:15.275189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:19.963943+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":445,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.160582+0000","last_change":"2024-09-21T12:28:39.853931+0000","last_active":"2024-09-21T12:29:59.160582+0000","last_peered":"2024-09-21T12:29:59.160582+0000","last_clean":"2024-09-21T12:29:59.160582+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:29:59.160582+0000","last_undegraded":"2024-09-21T12:29:59.160582+0000","last_fullsized":"2024-09-21T12:29:59.160582+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:39.853791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:39.853791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:04.491080+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"273'7924","reported_seq":8015,"reported_epoch":273,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-21T12:30:01.976536+0000","last_change":"2024-09-21T12:29:36.861734+0000","last_active":"2024-09-21T12:30:01.976536+0000","last_peered":"2024-09-21T12:30:01.976536+0000","last_clean":"2024-09-21T12:29:34.704985+0000","last_became_active":"2024-09-21T12:29:36.848982+0000","last_became_peered":"2024-09-21T12:29:36.848982+0000","last_unstale":"2024-09-21T12:30:01.976536+0000","last_undegraded":"2024-09-21T12:30:01.976536+0000","last_fullsized":"2024-09-21T12:30:01.976536+0000","mapping_epoch":266,"log_start":"255'3800","ondisk_log_start":"255'3800","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":4124,"log_dups_size":0,"ondisk_log_size":4124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":489553920,"num_objects":7470,"num_object_clones":0,"num_object_copies":22410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7470,"num_objects_unfound":0,"num_objects_dirty":7470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7924,"num_write_kb":492608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,1],"avail_no_missing":["2","1","6"],"object_location_counts":[{"shards":"1,2,6","objects":7470}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"76'79","reported_seq":167,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.154286+0000","last_change":"2024-09-21T12:29:35.778553+0000","last_active":"2024-09-21T12:29:59.154286+0000","last_peered":"2024-09-21T12:29:59.154286+0000","last_clean":"2024-09-21T12:29:59.154286+0000","last_became_active":"2024-09-21T12:29:35.777794+0000","last_became_peered":"2024-09-21T12:29:35.777794+0000","last_unstale":"2024-09-21T12:29:59.154286+0000","last_undegraded":"2024-09-21T12:29:59.154286+0000","last_fullsized":"2024-09-21T12:29:59.154286+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:57.680181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:57.680181+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131344,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":515,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.154261+0000","last_change":"2024-09-21T12:28:46.355910+0000","last_active":"2024-09-21T12:29:59.154261+0000","last_peered":"2024-09-21T12:29:59.154261+0000","last_clean":"2024-09-21T12:29:59.154261+0000","last_became_active":"2024-09-21T12:23:46.897197+0000","last_became_peered":"2024-09-21T12:23:46.897197+0000","last_unstale":"2024-09-21T12:29:59.154261+0000","last_undegraded":"2024-09-21T12:29:59.154261+0000","last_fullsized":"2024-09-21T12:29:59.154261+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-21T12:28:46.355822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:46.355822+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":184,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.158453+0000","last_change":"2024-09-21T12:29:35.784391+0000","last_active":"2024-09-21T12:29:59.158453+0000","last_peered":"2024-09-21T12:29:59.158453+0000","last_clean":"2024-09-21T12:29:59.158453+0000","last_became_active":"2024-09-21T12:29:35.783874+0000","last_became_peered":"2024-09-21T12:29:35.783874+0000","last_unstale":"2024-09-21T12:29:59.158453+0000","last_undegraded":"2024-09-21T12:29:59.158453+0000","last_fullsized":"2024-09-21T12:29:59.158453+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:06.236749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:06.236749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":471,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.152836+0000","last_change":"2024-09-21T12:28:03.571042+0000","last_active":"2024-09-21T12:29:59.152836+0000","last_peered":"2024-09-21T12:29:59.152836+0000","last_clean":"2024-09-21T12:29:59.152836+0000","last_became_active":"2024-09-21T12:22:50.927438+0000","last_became_peered":"2024-09-21T12:22:50.927438+0000","last_unstale":"2024-09-21T12:29:59.152836+0000","last_undegraded":"2024-09-21T12:29:59.152836+0000","last_fullsized":"2024-09-21T12:29:59.152836+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:03.570984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:03.570984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:29.578920+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":460,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.155788+0000","last_change":"2024-09-21T12:25:39.079146+0000","last_active":"2024-09-21T12:29:59.155788+0000","last_peered":"2024-09-21T12:29:59.155788+0000","last_clean":"2024-09-21T12:29:59.155788+0000","last_became_active":"2024-09-21T12:22:50.913898+0000","last_became_peered":"2024-09-21T12:22:50.913898+0000","last_unstale":"2024-09-21T12:29:59.155788+0000","last_undegraded":"2024-09-21T12:29:59.155788+0000","last_fullsized":"2024-09-21T12:29:59.155788+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:39.079015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:39.079015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:26.100425+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"273'8032","reported_seq":13487,"reported_epoch":273,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T12:30:02.467306+0000","last_change":"2024-09-21T12:29:36.877320+0000","last_active":"2024-09-21T12:30:02.467306+0000","last_peered":"2024-09-21T12:30:02.467306+0000","last_clean":"2024-09-21T12:29:34.747646+0000","last_became_active":"2024-09-21T12:29:36.851101+0000","last_became_peered":"2024-09-21T12:29:36.851101+0000","last_unstale":"2024-09-21T12:30:02.467306+0000","last_undegraded":"2024-09-21T12:30:02.467306+0000","last_fullsized":"2024-09-21T12:30:02.467306+0000","mapping_epoch":266,"log_start":"255'4100","ondisk_log_start":"255'4100","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3932,"log_dups_size":0,"ondisk_log_size":3932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":495321088,"num_objects":7558,"num_object_clones":0,"num_object_copies":22674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4817,"num_objects_unfound":0,"num_objects_dirty":7558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8032,"num_write_kb":498880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2683,"num_bytes_recovered":175833088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,7],"avail_no_missing":["3","6","7"],"object_location_counts":[{"shards":"3,6,7","objects":7558}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"76'79","reported_seq":540,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.159615+0000","last_change":"2024-09-21T12:29:05.501052+0000","last_active":"2024-09-21T12:29:59.159615+0000","last_peered":"2024-09-21T12:29:59.159615+0000","last_clean":"2024-09-21T12:29:59.159615+0000","last_became_active":"2024-09-21T12:27:14.943705+0000","last_became_peered":"2024-09-21T12:27:14.943705+0000","last_unstale":"2024-09-21T12:29:59.159615+0000","last_undegraded":"2024-09-21T12:29:59.159615+0000","last_fullsized":"2024-09-21T12:29:59.159615+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:05.500994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:05.500994+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:19.934083+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":329,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.154536+0000","last_change":"2024-09-21T12:29:14.222764+0000","last_active":"2024-09-21T12:29:59.154536+0000","last_peered":"2024-09-21T12:29:59.154536+0000","last_clean":"2024-09-21T12:29:59.154536+0000","last_became_active":"2024-09-21T12:23:46.902260+0000","last_became_peered":"2024-09-21T12:23:46.902260+0000","last_unstale":"2024-09-21T12:29:59.154536+0000","last_undegraded":"2024-09-21T12:29:59.154536+0000","last_fullsized":"2024-09-21T12:29:59.154536+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:29:14.222681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:29:14.222681+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:38.308175+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":478,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.158174+0000","last_change":"2024-09-21T12:27:32.433938+0000","last_active":"2024-09-21T12:29:59.158174+0000","last_peered":"2024-09-21T12:29:59.158174+0000","last_clean":"2024-09-21T12:29:59.158174+0000","last_became_active":"2024-09-21T12:22:50.927946+0000","last_became_peered":"2024-09-21T12:22:50.927946+0000","last_unstale":"2024-09-21T12:29:59.158174+0000","last_undegraded":"2024-09-21T12:29:59.158174+0000","last_fullsized":"2024-09-21T12:29:59.158174+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:32.433867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:27:32.433867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":593,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.154258+0000","last_change":"2024-09-21T12:28:39.418167+0000","last_active":"2024-09-21T12:29:59.154258+0000","last_peered":"2024-09-21T12:29:59.154258+0000","last_clean":"2024-09-21T12:29:59.154258+0000","last_became_active":"2024-09-21T12:20:19.307924+0000","last_became_peered":"2024-09-21T12:20:19.307924+0000","last_unstale":"2024-09-21T12:29:59.154258+0000","last_undegraded":"2024-09-21T12:29:59.154258+0000","last_fullsized":"2024-09-21T12:29:59.154258+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:28:39.418085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:39.418085+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:08.013783+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":85,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.154445+0000","last_change":"2024-09-21T12:29:35.771598+0000","last_active":"2024-09-21T12:29:59.154445+0000","last_peered":"2024-09-21T12:29:59.154445+0000","last_clean":"2024-09-21T12:29:59.154445+0000","last_became_active":"2024-09-21T12:29:35.771280+0000","last_became_peered":"2024-09-21T12:29:35.771280+0000","last_unstale":"2024-09-21T12:29:59.154445+0000","last_undegraded":"2024-09-21T12:29:59.154445+0000","last_fullsized":"2024-09-21T12:29:59.154445+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:35.669942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:35.669942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"273'7989","reported_seq":8074,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:30:01.977364+0000","last_change":"2024-09-21T12:28:20.952349+0000","last_active":"2024-09-21T12:30:01.977364+0000","last_peered":"2024-09-21T12:30:01.977364+0000","last_clean":"2024-09-21T12:30:01.977364+0000","last_became_active":"2024-09-21T12:28:20.952083+0000","last_became_peered":"2024-09-21T12:28:20.952083+0000","last_unstale":"2024-09-21T12:30:01.977364+0000","last_undegraded":"2024-09-21T12:30:01.977364+0000","last_fullsized":"2024-09-21T12:30:01.977364+0000","mapping_epoch":236,"log_start":"255'3837","ondisk_log_start":"255'3837","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":4152,"log_dups_size":0,"ondisk_log_size":4152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":493551616,"num_objects":7531,"num_object_clones":0,"num_object_copies":22593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7989,"num_write_kb":496640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"76'79","reported_seq":171,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.157215+0000","last_change":"2024-09-21T12:27:21.345414+0000","last_active":"2024-09-21T12:29:59.157215+0000","last_peered":"2024-09-21T12:29:59.157215+0000","last_clean":"2024-09-21T12:29:59.157215+0000","last_became_active":"2024-09-21T12:27:14.943330+0000","last_became_peered":"2024-09-21T12:27:14.943330+0000","last_unstale":"2024-09-21T12:29:59.157215+0000","last_undegraded":"2024-09-21T12:29:59.157215+0000","last_fullsized":"2024-09-21T12:29:59.157215+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:21.345364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:21.345364+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:03.403271+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":531,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.152911+0000","last_change":"2024-09-21T12:25:00.237083+0000","last_active":"2024-09-21T12:29:59.152911+0000","last_peered":"2024-09-21T12:29:59.152911+0000","last_clean":"2024-09-21T12:29:59.152911+0000","last_became_active":"2024-09-21T12:22:50.922343+0000","last_became_peered":"2024-09-21T12:22:50.922343+0000","last_unstale":"2024-09-21T12:29:59.152911+0000","last_undegraded":"2024-09-21T12:29:59.152911+0000","last_fullsized":"2024-09-21T12:29:59.152911+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:25:00.236990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:00.236990+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":141,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.165003+0000","last_change":"2024-09-21T12:26:54.084563+0000","last_active":"2024-09-21T12:29:59.165003+0000","last_peered":"2024-09-21T12:29:59.165003+0000","last_clean":"2024-09-21T12:29:59.165003+0000","last_became_active":"2024-09-21T12:26:54.084228+0000","last_became_peered":"2024-09-21T12:26:54.084228+0000","last_unstale":"2024-09-21T12:29:59.165003+0000","last_undegraded":"2024-09-21T12:29:59.165003+0000","last_fullsized":"2024-09-21T12:29:59.165003+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:30.451876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:30.451876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:28.176176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":488,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.156549+0000","last_change":"2024-09-21T12:28:47.361181+0000","last_active":"2024-09-21T12:29:59.156549+0000","last_peered":"2024-09-21T12:29:59.156549+0000","last_clean":"2024-09-21T12:29:59.156549+0000","last_became_active":"2024-09-21T12:22:50.925974+0000","last_became_peered":"2024-09-21T12:22:50.925974+0000","last_unstale":"2024-09-21T12:29:59.156549+0000","last_undegraded":"2024-09-21T12:29:59.156549+0000","last_fullsized":"2024-09-21T12:29:59.156549+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:47.361115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:47.361115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":448,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.153593+0000","last_change":"2024-09-21T12:28:43.821102+0000","last_active":"2024-09-21T12:29:59.153593+0000","last_peered":"2024-09-21T12:29:59.153593+0000","last_clean":"2024-09-21T12:29:59.153593+0000","last_became_active":"2024-09-21T12:20:21.311493+0000","last_became_peered":"2024-09-21T12:20:21.311493+0000","last_unstale":"2024-09-21T12:29:59.153593+0000","last_undegraded":"2024-09-21T12:29:59.153593+0000","last_fullsized":"2024-09-21T12:29:59.153593+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:43.820979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:43.820979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"273'7968","reported_seq":8058,"reported_epoch":273,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-21T12:30:02.454004+0000","last_change":"2024-09-21T12:29:36.939486+0000","last_active":"2024-09-21T12:30:02.454004+0000","last_peered":"2024-09-21T12:30:02.454004+0000","last_clean":"2024-09-21T12:29:34.734937+0000","last_became_active":"2024-09-21T12:29:36.898598+0000","last_became_peered":"2024-09-21T12:29:36.898598+0000","last_unstale":"2024-09-21T12:30:02.454004+0000","last_undegraded":"2024-09-21T12:30:02.454004+0000","last_fullsized":"2024-09-21T12:30:02.454004+0000","mapping_epoch":266,"log_start":"255'4000","ondisk_log_start":"255'4000","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3968,"log_dups_size":0,"ondisk_log_size":3968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":492044288,"num_objects":7508,"num_object_clones":0,"num_object_copies":22524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":15016,"num_objects_unfound":0,"num_objects_dirty":7508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7968,"num_write_kb":495232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[3,5,7],"avail_no_missing":["3","5","7"],"object_location_counts":[{"shards":"3,5,7","objects":7508}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.f","version":"76'79","reported_seq":373,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.150422+0000","last_change":"2024-09-21T12:29:07.210453+0000","last_active":"2024-09-21T12:29:59.150422+0000","last_peered":"2024-09-21T12:29:59.150422+0000","last_clean":"2024-09-21T12:29:59.150422+0000","last_became_active":"2024-09-21T12:29:05.842070+0000","last_became_peered":"2024-09-21T12:29:05.842070+0000","last_unstale":"2024-09-21T12:29:59.150422+0000","last_undegraded":"2024-09-21T12:29:59.150422+0000","last_fullsized":"2024-09-21T12:29:59.150422+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:07.210381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:07.210381+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:07.733148+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":520,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.157044+0000","last_change":"2024-09-21T12:29:35.792626+0000","last_active":"2024-09-21T12:29:59.157044+0000","last_peered":"2024-09-21T12:29:59.157044+0000","last_clean":"2024-09-21T12:29:59.157044+0000","last_became_active":"2024-09-21T12:29:35.765582+0000","last_became_peered":"2024-09-21T12:29:35.765582+0000","last_unstale":"2024-09-21T12:29:59.157044+0000","last_undegraded":"2024-09-21T12:29:59.157044+0000","last_fullsized":"2024-09-21T12:29:59.157044+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-21T12:27:30.898706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:30.898706+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":465,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.159009+0000","last_change":"2024-09-21T12:29:35.770561+0000","last_active":"2024-09-21T12:29:59.159009+0000","last_peered":"2024-09-21T12:29:59.159009+0000","last_clean":"2024-09-21T12:29:59.159009+0000","last_became_active":"2024-09-21T12:29:35.770095+0000","last_became_peered":"2024-09-21T12:29:35.770095+0000","last_unstale":"2024-09-21T12:29:59.159009+0000","last_undegraded":"2024-09-21T12:29:59.159009+0000","last_fullsized":"2024-09-21T12:29:59.159009+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:48.438755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:48.438755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:03.501368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":458,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.154005+0000","last_change":"2024-09-21T12:29:35.769084+0000","last_active":"2024-09-21T12:29:59.154005+0000","last_peered":"2024-09-21T12:29:59.154005+0000","last_clean":"2024-09-21T12:29:59.154005+0000","last_became_active":"2024-09-21T12:29:35.767996+0000","last_became_peered":"2024-09-21T12:29:35.767996+0000","last_unstale":"2024-09-21T12:29:59.154005+0000","last_undegraded":"2024-09-21T12:29:59.154005+0000","last_fullsized":"2024-09-21T12:29:59.154005+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:50.626780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:50.626780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":438,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.157029+0000","last_change":"2024-09-21T12:29:15.641929+0000","last_active":"2024-09-21T12:29:59.157029+0000","last_peered":"2024-09-21T12:29:59.157029+0000","last_clean":"2024-09-21T12:29:59.157029+0000","last_became_active":"2024-09-21T12:20:21.314545+0000","last_became_peered":"2024-09-21T12:20:21.314545+0000","last_unstale":"2024-09-21T12:29:59.157029+0000","last_undegraded":"2024-09-21T12:29:59.157029+0000","last_fullsized":"2024-09-21T12:29:59.157029+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:15.641881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:29:15.641881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:27.876807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"273'7972","reported_seq":8056,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:30:02.220129+0000","last_change":"2024-09-21T12:28:20.955882+0000","last_active":"2024-09-21T12:30:02.220129+0000","last_peered":"2024-09-21T12:30:02.220129+0000","last_clean":"2024-09-21T12:30:02.220129+0000","last_became_active":"2024-09-21T12:28:20.955605+0000","last_became_peered":"2024-09-21T12:28:20.955605+0000","last_unstale":"2024-09-21T12:30:02.220129+0000","last_undegraded":"2024-09-21T12:30:02.220129+0000","last_fullsized":"2024-09-21T12:30:02.220129+0000","mapping_epoch":236,"log_start":"255'3900","ondisk_log_start":"255'3900","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":4072,"log_dups_size":0,"ondisk_log_size":4072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":493879296,"num_objects":7536,"num_object_clones":0,"num_object_copies":22608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7972,"num_write_kb":496256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"76'79","reported_seq":411,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.164816+0000","last_change":"2024-09-21T12:27:17.158443+0000","last_active":"2024-09-21T12:29:59.164816+0000","last_peered":"2024-09-21T12:29:59.164816+0000","last_clean":"2024-09-21T12:29:59.164816+0000","last_became_active":"2024-09-21T12:27:14.951375+0000","last_became_peered":"2024-09-21T12:27:14.951375+0000","last_unstale":"2024-09-21T12:29:59.164816+0000","last_undegraded":"2024-09-21T12:29:59.164816+0000","last_fullsized":"2024-09-21T12:29:59.164816+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:17.158380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:17.158380+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":483,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.148558+0000","last_change":"2024-09-21T12:29:35.797253+0000","last_active":"2024-09-21T12:29:59.148558+0000","last_peered":"2024-09-21T12:29:59.148558+0000","last_clean":"2024-09-21T12:29:59.148558+0000","last_became_active":"2024-09-21T12:29:35.765273+0000","last_became_peered":"2024-09-21T12:29:35.765273+0000","last_unstale":"2024-09-21T12:29:59.148558+0000","last_undegraded":"2024-09-21T12:29:59.148558+0000","last_fullsized":"2024-09-21T12:29:59.148558+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:28:40.694138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:40.694138+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":487,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.157747+0000","last_change":"2024-09-21T12:28:30.612213+0000","last_active":"2024-09-21T12:29:59.157747+0000","last_peered":"2024-09-21T12:29:59.157747+0000","last_clean":"2024-09-21T12:29:59.157747+0000","last_became_active":"2024-09-21T12:22:50.927859+0000","last_became_peered":"2024-09-21T12:22:50.927859+0000","last_unstale":"2024-09-21T12:29:59.157747+0000","last_undegraded":"2024-09-21T12:29:59.157747+0000","last_fullsized":"2024-09-21T12:29:59.157747+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:30.612156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:30.612156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":467,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.155082+0000","last_change":"2024-09-21T12:29:11.218769+0000","last_active":"2024-09-21T12:29:59.155082+0000","last_peered":"2024-09-21T12:29:59.155082+0000","last_clean":"2024-09-21T12:29:59.155082+0000","last_became_active":"2024-09-21T12:20:19.312572+0000","last_became_peered":"2024-09-21T12:20:19.312572+0000","last_unstale":"2024-09-21T12:29:59.155082+0000","last_undegraded":"2024-09-21T12:29:59.155082+0000","last_fullsized":"2024-09-21T12:29:59.155082+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:11.218334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:29:11.218334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:35.978478+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":452,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.157845+0000","last_change":"2024-09-21T12:29:35.777977+0000","last_active":"2024-09-21T12:29:59.157845+0000","last_peered":"2024-09-21T12:29:59.157845+0000","last_clean":"2024-09-21T12:29:59.157845+0000","last_became_active":"2024-09-21T12:29:35.776977+0000","last_became_peered":"2024-09-21T12:29:35.776977+0000","last_unstale":"2024-09-21T12:29:59.157845+0000","last_undegraded":"2024-09-21T12:29:59.157845+0000","last_fullsized":"2024-09-21T12:29:59.157845+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:58.321890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:58.321890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"273'7983","reported_seq":12335,"reported_epoch":273,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T12:30:02.517999+0000","last_change":"2024-09-21T12:29:36.894379+0000","last_active":"2024-09-21T12:30:02.517999+0000","last_peered":"2024-09-21T12:30:02.517999+0000","last_clean":"2024-09-21T12:29:34.748613+0000","last_became_active":"2024-09-21T12:29:36.865103+0000","last_became_peered":"2024-09-21T12:29:36.865103+0000","last_unstale":"2024-09-21T12:30:02.517999+0000","last_undegraded":"2024-09-21T12:30:02.517999+0000","last_fullsized":"2024-09-21T12:30:02.517999+0000","mapping_epoch":266,"log_start":"256'4275","ondisk_log_start":"256'4275","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3708,"log_dups_size":0,"ondisk_log_size":3708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":491323392,"num_objects":7497,"num_object_clones":0,"num_object_copies":22491,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5322,"num_objects_unfound":0,"num_objects_dirty":7497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7983,"num_write_kb":495360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2131,"num_bytes_recovered":139657216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,7],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":7497}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"76'79","reported_seq":37,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.152374+0000","last_change":"2024-09-21T12:29:36.701016+0000","last_active":"2024-09-21T12:29:59.152374+0000","last_peered":"2024-09-21T12:29:59.152374+0000","last_clean":"2024-09-21T12:29:59.152374+0000","last_became_active":"2024-09-21T12:29:35.767716+0000","last_became_peered":"2024-09-21T12:29:35.767716+0000","last_unstale":"2024-09-21T12:29:59.152374+0000","last_undegraded":"2024-09-21T12:29:59.152374+0000","last_fullsized":"2024-09-21T12:29:59.152374+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:36.700968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:36.700968+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:31:00.892429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":103,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.156166+0000","last_change":"2024-09-21T12:29:58.184503+0000","last_active":"2024-09-21T12:29:59.156166+0000","last_peered":"2024-09-21T12:29:59.156166+0000","last_clean":"2024-09-21T12:29:59.156166+0000","last_became_active":"2024-09-21T12:29:35.765891+0000","last_became_peered":"2024-09-21T12:29:35.765891+0000","last_unstale":"2024-09-21T12:29:59.156166+0000","last_undegraded":"2024-09-21T12:29:59.156166+0000","last_fullsized":"2024-09-21T12:29:59.156166+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:29:58.184426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:29:58.184426+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:31:18.574790+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":522,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.157453+0000","last_change":"2024-09-21T12:29:35.785869+0000","last_active":"2024-09-21T12:29:59.157453+0000","last_peered":"2024-09-21T12:29:59.157453+0000","last_clean":"2024-09-21T12:29:59.157453+0000","last_became_active":"2024-09-21T12:29:35.769415+0000","last_became_peered":"2024-09-21T12:29:35.769415+0000","last_unstale":"2024-09-21T12:29:59.157453+0000","last_undegraded":"2024-09-21T12:29:59.157453+0000","last_fullsized":"2024-09-21T12:29:59.157453+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-21T12:24:29.054772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:29.054772+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"266'12","reported_seq":612,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.158360+0000","last_change":"2024-09-21T12:29:35.788485+0000","last_active":"2024-09-21T12:29:59.158360+0000","last_peered":"2024-09-21T12:29:59.158360+0000","last_clean":"2024-09-21T12:29:59.158360+0000","last_became_active":"2024-09-21T12:29:35.770167+0000","last_became_peered":"2024-09-21T12:29:35.770167+0000","last_unstale":"2024-09-21T12:29:59.158360+0000","last_undegraded":"2024-09-21T12:29:59.158360+0000","last_fullsized":"2024-09-21T12:29:59.158360+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:25:02.916090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:02.916090+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":149,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.156925+0000","last_change":"2024-09-21T12:26:56.862895+0000","last_active":"2024-09-21T12:29:59.156925+0000","last_peered":"2024-09-21T12:29:59.156925+0000","last_clean":"2024-09-21T12:29:59.156925+0000","last_became_active":"2024-09-21T12:26:54.089155+0000","last_became_peered":"2024-09-21T12:26:54.089155+0000","last_unstale":"2024-09-21T12:29:59.156925+0000","last_undegraded":"2024-09-21T12:29:59.156925+0000","last_fullsized":"2024-09-21T12:29:59.156925+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:56.862823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:56.862823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:17.786563+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"76'79","reported_seq":180,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.150082+0000","last_change":"2024-09-21T12:29:35.778736+0000","last_active":"2024-09-21T12:29:59.150082+0000","last_peered":"2024-09-21T12:29:59.150082+0000","last_clean":"2024-09-21T12:29:59.150082+0000","last_became_active":"2024-09-21T12:29:35.778366+0000","last_became_peered":"2024-09-21T12:29:35.778366+0000","last_unstale":"2024-09-21T12:29:59.150082+0000","last_undegraded":"2024-09-21T12:29:59.150082+0000","last_fullsized":"2024-09-21T12:29:59.150082+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:12.141269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:12.141269+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:40.502382+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":489,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.156406+0000","last_change":"2024-09-21T12:29:35.776307+0000","last_active":"2024-09-21T12:29:59.156406+0000","last_peered":"2024-09-21T12:29:59.156406+0000","last_clean":"2024-09-21T12:29:59.156406+0000","last_became_active":"2024-09-21T12:29:35.775852+0000","last_became_peered":"2024-09-21T12:29:35.775852+0000","last_unstale":"2024-09-21T12:29:59.156406+0000","last_undegraded":"2024-09-21T12:29:59.156406+0000","last_fullsized":"2024-09-21T12:29:59.156406+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:29.901043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:27:29.901043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":67,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.152766+0000","last_change":"2024-09-21T12:29:35.800413+0000","last_active":"2024-09-21T12:29:59.152766+0000","last_peered":"2024-09-21T12:29:59.152766+0000","last_clean":"2024-09-21T12:29:59.152766+0000","last_became_active":"2024-09-21T12:29:35.773526+0000","last_became_peered":"2024-09-21T12:29:35.773526+0000","last_unstale":"2024-09-21T12:29:59.152766+0000","last_undegraded":"2024-09-21T12:29:59.152766+0000","last_fullsized":"2024-09-21T12:29:59.152766+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-21T12:26:57.369648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:57.369648+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":142,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.157136+0000","last_change":"2024-09-21T12:26:54.084503+0000","last_active":"2024-09-21T12:29:59.157136+0000","last_peered":"2024-09-21T12:29:59.157136+0000","last_clean":"2024-09-21T12:29:59.157136+0000","last_became_active":"2024-09-21T12:26:54.084109+0000","last_became_peered":"2024-09-21T12:26:54.084109+0000","last_unstale":"2024-09-21T12:29:59.157136+0000","last_undegraded":"2024-09-21T12:29:59.157136+0000","last_fullsized":"2024-09-21T12:29:59.157136+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:02.414298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:02.414298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:21.193891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":459,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.158405+0000","last_change":"2024-09-21T12:24:00.066522+0000","last_active":"2024-09-21T12:29:59.158405+0000","last_peered":"2024-09-21T12:29:59.158405+0000","last_clean":"2024-09-21T12:29:59.158405+0000","last_became_active":"2024-09-21T12:22:50.904404+0000","last_became_peered":"2024-09-21T12:22:50.904404+0000","last_unstale":"2024-09-21T12:29:59.158405+0000","last_undegraded":"2024-09-21T12:29:59.158405+0000","last_fullsized":"2024-09-21T12:29:59.158405+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:00.066451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:00.066451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'79","reported_seq":413,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.150993+0000","last_change":"2024-09-21T12:29:35.767301+0000","last_active":"2024-09-21T12:29:59.150993+0000","last_peered":"2024-09-21T12:29:59.150993+0000","last_clean":"2024-09-21T12:29:59.150993+0000","last_became_active":"2024-09-21T12:29:35.766922+0000","last_became_peered":"2024-09-21T12:29:35.766922+0000","last_unstale":"2024-09-21T12:29:59.150993+0000","last_undegraded":"2024-09-21T12:29:59.150993+0000","last_fullsized":"2024-09-21T12:29:59.150993+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:29.546657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:29.546657+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":497,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.152434+0000","last_change":"2024-09-21T12:29:35.797003+0000","last_active":"2024-09-21T12:29:59.152434+0000","last_peered":"2024-09-21T12:29:59.152434+0000","last_clean":"2024-09-21T12:29:59.152434+0000","last_became_active":"2024-09-21T12:29:35.774676+0000","last_became_peered":"2024-09-21T12:29:35.774676+0000","last_unstale":"2024-09-21T12:29:59.152434+0000","last_undegraded":"2024-09-21T12:29:59.152434+0000","last_fullsized":"2024-09-21T12:29:59.152434+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:28:01.881164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:01.881164+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":495,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.159778+0000","last_change":"2024-09-21T12:28:43.545259+0000","last_active":"2024-09-21T12:29:59.159778+0000","last_peered":"2024-09-21T12:29:59.159778+0000","last_clean":"2024-09-21T12:29:59.159778+0000","last_became_active":"2024-09-21T12:22:50.927649+0000","last_became_peered":"2024-09-21T12:22:50.927649+0000","last_unstale":"2024-09-21T12:29:59.159778+0000","last_undegraded":"2024-09-21T12:29:59.159778+0000","last_fullsized":"2024-09-21T12:29:59.159778+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:43.545197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:43.545197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":449,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.157836+0000","last_change":"2024-09-21T12:28:47.964092+0000","last_active":"2024-09-21T12:29:59.157836+0000","last_peered":"2024-09-21T12:29:59.157836+0000","last_clean":"2024-09-21T12:29:59.157836+0000","last_became_active":"2024-09-21T12:20:19.311947+0000","last_became_peered":"2024-09-21T12:20:19.311947+0000","last_unstale":"2024-09-21T12:29:59.157836+0000","last_undegraded":"2024-09-21T12:29:59.157836+0000","last_fullsized":"2024-09-21T12:29:59.157836+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:47.963747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:47.963747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:12.338200+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":484,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.154688+0000","last_change":"2024-09-21T12:26:11.105716+0000","last_active":"2024-09-21T12:29:59.154688+0000","last_peered":"2024-09-21T12:29:59.154688+0000","last_clean":"2024-09-21T12:29:59.154688+0000","last_became_active":"2024-09-21T12:22:50.924931+0000","last_became_peered":"2024-09-21T12:22:50.924931+0000","last_unstale":"2024-09-21T12:29:59.154688+0000","last_undegraded":"2024-09-21T12:29:59.154688+0000","last_fullsized":"2024-09-21T12:29:59.154688+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:11.105583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:11.105583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"76'79","reported_seq":534,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.156611+0000","last_change":"2024-09-21T12:26:54.085274+0000","last_active":"2024-09-21T12:29:59.156611+0000","last_peered":"2024-09-21T12:29:59.156611+0000","last_clean":"2024-09-21T12:29:59.156611+0000","last_became_active":"2024-09-21T12:26:54.084939+0000","last_became_peered":"2024-09-21T12:26:54.084939+0000","last_unstale":"2024-09-21T12:29:59.156611+0000","last_undegraded":"2024-09-21T12:29:59.156611+0000","last_fullsized":"2024-09-21T12:29:59.156611+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:50.443344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:50.443344+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:30:19.773272+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":202,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.158687+0000","last_change":"2024-09-21T12:29:35.783028+0000","last_active":"2024-09-21T12:29:59.158687+0000","last_peered":"2024-09-21T12:29:59.158687+0000","last_clean":"2024-09-21T12:29:59.158687+0000","last_became_active":"2024-09-21T12:29:35.763537+0000","last_became_peered":"2024-09-21T12:29:35.763537+0000","last_unstale":"2024-09-21T12:29:59.158687+0000","last_undegraded":"2024-09-21T12:29:59.158687+0000","last_fullsized":"2024-09-21T12:29:59.158687+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:28:42.511874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:42.511874+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":475,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.152308+0000","last_change":"2024-09-21T12:29:35.770103+0000","last_active":"2024-09-21T12:29:59.152308+0000","last_peered":"2024-09-21T12:29:59.152308+0000","last_clean":"2024-09-21T12:29:59.152308+0000","last_became_active":"2024-09-21T12:29:35.769652+0000","last_became_peered":"2024-09-21T12:29:35.769652+0000","last_unstale":"2024-09-21T12:29:59.152308+0000","last_undegraded":"2024-09-21T12:29:59.152308+0000","last_fullsized":"2024-09-21T12:29:59.152308+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:07.875304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:07.875304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"266'9","reported_seq":62,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.157614+0000","last_change":"2024-09-21T12:29:35.798385+0000","last_active":"2024-09-21T12:29:59.157614+0000","last_peered":"2024-09-21T12:29:59.157614+0000","last_clean":"2024-09-21T12:29:59.157614+0000","last_became_active":"2024-09-21T12:29:35.768270+0000","last_became_peered":"2024-09-21T12:29:35.768270+0000","last_unstale":"2024-09-21T12:29:59.157614+0000","last_undegraded":"2024-09-21T12:29:59.157614+0000","last_fullsized":"2024-09-21T12:29:59.157614+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:28:43.853712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:43.853712+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":390,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.155877+0000","last_change":"2024-09-21T12:28:05.323501+0000","last_active":"2024-09-21T12:29:59.155877+0000","last_peered":"2024-09-21T12:29:59.155877+0000","last_clean":"2024-09-21T12:29:59.155877+0000","last_became_active":"2024-09-21T12:22:50.924770+0000","last_became_peered":"2024-09-21T12:22:50.924770+0000","last_unstale":"2024-09-21T12:29:59.155877+0000","last_undegraded":"2024-09-21T12:29:59.155877+0000","last_fullsized":"2024-09-21T12:29:59.155877+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:05.323448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:05.323448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"76'79","reported_seq":255,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.153722+0000","last_change":"2024-09-21T12:29:35.774892+0000","last_active":"2024-09-21T12:29:59.153722+0000","last_peered":"2024-09-21T12:29:59.153722+0000","last_clean":"2024-09-21T12:29:59.153722+0000","last_became_active":"2024-09-21T12:29:35.773989+0000","last_became_peered":"2024-09-21T12:29:35.773989+0000","last_unstale":"2024-09-21T12:29:59.153722+0000","last_undegraded":"2024-09-21T12:29:59.153722+0000","last_fullsized":"2024-09-21T12:29:59.153722+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:48.804235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:48.804235+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":487,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.164911+0000","last_change":"2024-09-21T12:29:35.805229+0000","last_active":"2024-09-21T12:29:59.164911+0000","last_peered":"2024-09-21T12:29:59.164911+0000","last_clean":"2024-09-21T12:29:59.164911+0000","last_became_active":"2024-09-21T12:29:35.780460+0000","last_became_peered":"2024-09-21T12:29:35.780460+0000","last_unstale":"2024-09-21T12:29:59.164911+0000","last_undegraded":"2024-09-21T12:29:59.164911+0000","last_fullsized":"2024-09-21T12:29:59.164911+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:24:50.727101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:50.727101+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":487,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.157035+0000","last_change":"2024-09-21T12:28:32.630320+0000","last_active":"2024-09-21T12:29:59.157035+0000","last_peered":"2024-09-21T12:29:59.157035+0000","last_clean":"2024-09-21T12:29:59.157035+0000","last_became_active":"2024-09-21T12:22:50.908238+0000","last_became_peered":"2024-09-21T12:22:50.908238+0000","last_unstale":"2024-09-21T12:29:59.157035+0000","last_undegraded":"2024-09-21T12:29:59.157035+0000","last_fullsized":"2024-09-21T12:29:59.157035+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:32.630261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:32.630261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":576,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.150454+0000","last_change":"2024-09-21T12:28:36.293443+0000","last_active":"2024-09-21T12:29:59.150454+0000","last_peered":"2024-09-21T12:29:59.150454+0000","last_clean":"2024-09-21T12:29:59.150454+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:29:59.150454+0000","last_undegraded":"2024-09-21T12:29:59.150454+0000","last_fullsized":"2024-09-21T12:29:59.150454+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:28:36.293377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:36.293377+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:01.258654+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":447,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.152086+0000","last_change":"2024-09-21T12:29:35.771966+0000","last_active":"2024-09-21T12:29:59.152086+0000","last_peered":"2024-09-21T12:29:59.152086+0000","last_clean":"2024-09-21T12:29:59.152086+0000","last_became_active":"2024-09-21T12:29:35.768276+0000","last_became_peered":"2024-09-21T12:29:35.768276+0000","last_unstale":"2024-09-21T12:29:59.152086+0000","last_undegraded":"2024-09-21T12:29:59.152086+0000","last_fullsized":"2024-09-21T12:29:59.152086+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:08.923458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:08.923458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"76'79","reported_seq":127,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.160202+0000","last_change":"2024-09-21T12:27:18.292942+0000","last_active":"2024-09-21T12:29:59.160202+0000","last_peered":"2024-09-21T12:29:59.160202+0000","last_clean":"2024-09-21T12:29:59.160202+0000","last_became_active":"2024-09-21T12:27:16.940347+0000","last_became_peered":"2024-09-21T12:27:16.940347+0000","last_unstale":"2024-09-21T12:29:59.160202+0000","last_undegraded":"2024-09-21T12:29:59.160202+0000","last_fullsized":"2024-09-21T12:29:59.160202+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:18.292862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:18.292862+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:01.616924+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":453,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.158931+0000","last_change":"2024-09-21T12:28:40.150322+0000","last_active":"2024-09-21T12:29:59.158931+0000","last_peered":"2024-09-21T12:29:59.158931+0000","last_clean":"2024-09-21T12:29:59.158931+0000","last_became_active":"2024-09-21T12:22:50.906473+0000","last_became_peered":"2024-09-21T12:22:50.906473+0000","last_unstale":"2024-09-21T12:29:59.158931+0000","last_undegraded":"2024-09-21T12:29:59.158931+0000","last_fullsized":"2024-09-21T12:29:59.158931+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:28:40.150250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:40.150250+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":374,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.157889+0000","last_change":"2024-09-21T12:24:58.948698+0000","last_active":"2024-09-21T12:29:59.157889+0000","last_peered":"2024-09-21T12:29:59.157889+0000","last_clean":"2024-09-21T12:29:59.157889+0000","last_became_active":"2024-09-21T12:22:50.903705+0000","last_became_peered":"2024-09-21T12:22:50.903705+0000","last_unstale":"2024-09-21T12:29:59.157889+0000","last_undegraded":"2024-09-21T12:29:59.157889+0000","last_fullsized":"2024-09-21T12:29:59.157889+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:58.948625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:58.948625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:23.707170+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":452,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.150885+0000","last_change":"2024-09-21T12:29:35.769972+0000","last_active":"2024-09-21T12:29:59.150885+0000","last_peered":"2024-09-21T12:29:59.150885+0000","last_clean":"2024-09-21T12:29:59.150885+0000","last_became_active":"2024-09-21T12:29:35.769510+0000","last_became_peered":"2024-09-21T12:29:35.769510+0000","last_unstale":"2024-09-21T12:29:59.150885+0000","last_undegraded":"2024-09-21T12:29:59.150885+0000","last_fullsized":"2024-09-21T12:29:59.150885+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:16.149550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:16.149550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":27,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.153802+0000","last_change":"2024-09-21T12:29:35.774512+0000","last_active":"2024-09-21T12:29:59.153802+0000","last_peered":"2024-09-21T12:29:59.153802+0000","last_clean":"2024-09-21T12:29:59.153802+0000","last_became_active":"2024-09-21T12:29:35.768083+0000","last_became_peered":"2024-09-21T12:29:35.768083+0000","last_unstale":"2024-09-21T12:29:59.153802+0000","last_undegraded":"2024-09-21T12:29:59.153802+0000","last_fullsized":"2024-09-21T12:29:59.153802+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:56.879558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:56.879558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.17","version":"76'79","reported_seq":144,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.150566+0000","last_change":"2024-09-21T12:29:55.291563+0000","last_active":"2024-09-21T12:29:59.150566+0000","last_peered":"2024-09-21T12:29:59.150566+0000","last_clean":"2024-09-21T12:29:59.150566+0000","last_became_active":"2024-09-21T12:27:16.921880+0000","last_became_peered":"2024-09-21T12:27:16.921880+0000","last_unstale":"2024-09-21T12:29:59.150566+0000","last_undegraded":"2024-09-21T12:29:59.150566+0000","last_fullsized":"2024-09-21T12:29:59.150566+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:55.291509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:55.291509+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:57.999281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":73,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.157218+0000","last_change":"2024-09-21T12:29:35.787713+0000","last_active":"2024-09-21T12:29:59.157218+0000","last_peered":"2024-09-21T12:29:59.157218+0000","last_clean":"2024-09-21T12:29:59.157218+0000","last_became_active":"2024-09-21T12:29:35.765649+0000","last_became_peered":"2024-09-21T12:29:35.765649+0000","last_unstale":"2024-09-21T12:29:59.157218+0000","last_undegraded":"2024-09-21T12:29:59.157218+0000","last_fullsized":"2024-09-21T12:29:59.157218+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:28:36.645961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:36.645961+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":487,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.158850+0000","last_change":"2024-09-21T12:26:19.973528+0000","last_active":"2024-09-21T12:29:59.158850+0000","last_peered":"2024-09-21T12:29:59.158850+0000","last_clean":"2024-09-21T12:29:59.158850+0000","last_became_active":"2024-09-21T12:22:50.922019+0000","last_became_peered":"2024-09-21T12:22:50.922019+0000","last_unstale":"2024-09-21T12:29:59.158850+0000","last_undegraded":"2024-09-21T12:29:59.158850+0000","last_fullsized":"2024-09-21T12:29:59.158850+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:19.973461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:19.973461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":449,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.157920+0000","last_change":"2024-09-21T12:26:21.572474+0000","last_active":"2024-09-21T12:29:59.157920+0000","last_peered":"2024-09-21T12:29:59.157920+0000","last_clean":"2024-09-21T12:29:59.157920+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:29:59.157920+0000","last_undegraded":"2024-09-21T12:29:59.157920+0000","last_fullsized":"2024-09-21T12:29:59.157920+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:21.571726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:21.571726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:02.068091+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":448,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.151441+0000","last_change":"2024-09-21T12:28:41.864711+0000","last_active":"2024-09-21T12:29:59.151441+0000","last_peered":"2024-09-21T12:29:59.151441+0000","last_clean":"2024-09-21T12:29:59.151441+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:29:59.151441+0000","last_undegraded":"2024-09-21T12:29:59.151441+0000","last_fullsized":"2024-09-21T12:29:59.151441+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:41.864630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:41.864630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:09.118573+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"76'79","reported_seq":157,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.154326+0000","last_change":"2024-09-21T12:26:49.624804+0000","last_active":"2024-09-21T12:29:59.154326+0000","last_peered":"2024-09-21T12:29:59.154326+0000","last_clean":"2024-09-21T12:29:59.154326+0000","last_became_active":"2024-09-21T12:26:46.989742+0000","last_became_peered":"2024-09-21T12:26:46.989742+0000","last_unstale":"2024-09-21T12:29:59.154326+0000","last_undegraded":"2024-09-21T12:29:59.154326+0000","last_fullsized":"2024-09-21T12:29:59.154326+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:49.623735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:49.623735+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":472,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.159798+0000","last_change":"2024-09-21T12:24:58.824615+0000","last_active":"2024-09-21T12:29:59.159798+0000","last_peered":"2024-09-21T12:29:59.159798+0000","last_clean":"2024-09-21T12:29:59.159798+0000","last_became_active":"2024-09-21T12:20:17.694794+0000","last_became_peered":"2024-09-21T12:20:17.694794+0000","last_unstale":"2024-09-21T12:29:59.159798+0000","last_undegraded":"2024-09-21T12:29:59.159798+0000","last_fullsized":"2024-09-21T12:29:59.159798+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:24:58.824516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:58.824516+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:20.197865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":482,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.150134+0000","last_change":"2024-09-21T12:29:54.303655+0000","last_active":"2024-09-21T12:29:59.150134+0000","last_peered":"2024-09-21T12:29:59.150134+0000","last_clean":"2024-09-21T12:29:59.150134+0000","last_became_active":"2024-09-21T12:20:15.295968+0000","last_became_peered":"2024-09-21T12:20:15.295968+0000","last_unstale":"2024-09-21T12:29:59.150134+0000","last_undegraded":"2024-09-21T12:29:59.150134+0000","last_fullsized":"2024-09-21T12:29:59.150134+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:54.303590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:54.303590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:31:16.447834+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":463,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.156517+0000","last_change":"2024-09-21T12:23:57.018797+0000","last_active":"2024-09-21T12:29:59.156517+0000","last_peered":"2024-09-21T12:29:59.156517+0000","last_clean":"2024-09-21T12:29:59.156517+0000","last_became_active":"2024-09-21T12:22:50.921604+0000","last_became_peered":"2024-09-21T12:22:50.921604+0000","last_unstale":"2024-09-21T12:29:59.156517+0000","last_undegraded":"2024-09-21T12:29:59.156517+0000","last_fullsized":"2024-09-21T12:29:59.156517+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:57.018749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:57.018749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:22.755530+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":445,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.158099+0000","last_change":"2024-09-21T12:29:14.051153+0000","last_active":"2024-09-21T12:29:59.158099+0000","last_peered":"2024-09-21T12:29:59.158099+0000","last_clean":"2024-09-21T12:29:59.158099+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:29:59.158099+0000","last_undegraded":"2024-09-21T12:29:59.158099+0000","last_fullsized":"2024-09-21T12:29:59.158099+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:14.051043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:29:14.051043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:23.917212+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"76'79","reported_seq":128,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.158211+0000","last_change":"2024-09-21T12:27:24.373557+0000","last_active":"2024-09-21T12:29:59.158211+0000","last_peered":"2024-09-21T12:29:59.158211+0000","last_clean":"2024-09-21T12:29:59.158211+0000","last_became_active":"2024-09-21T12:27:16.914713+0000","last_became_peered":"2024-09-21T12:27:16.914713+0000","last_unstale":"2024-09-21T12:29:59.158211+0000","last_undegraded":"2024-09-21T12:29:59.158211+0000","last_fullsized":"2024-09-21T12:29:59.158211+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:24.373501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:24.373501+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:35.520215+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":459,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.156608+0000","last_change":"2024-09-21T12:28:46.932522+0000","last_active":"2024-09-21T12:29:59.156608+0000","last_peered":"2024-09-21T12:29:59.156608+0000","last_clean":"2024-09-21T12:29:59.156608+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:29:59.156608+0000","last_undegraded":"2024-09-21T12:29:59.156608+0000","last_fullsized":"2024-09-21T12:29:59.156608+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-21T12:28:46.931901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:46.931901+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:14.037686+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":474,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.152512+0000","last_change":"2024-09-21T12:28:00.838008+0000","last_active":"2024-09-21T12:29:59.152512+0000","last_peered":"2024-09-21T12:29:59.152512+0000","last_clean":"2024-09-21T12:29:59.152512+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:29:59.152512+0000","last_undegraded":"2024-09-21T12:29:59.152512+0000","last_fullsized":"2024-09-21T12:29:59.152512+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:28:00.837937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:00.837937+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":479,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.216485+0000","last_change":"2024-09-21T12:29:59.216485+0000","last_active":"2024-09-21T12:29:59.216485+0000","last_peered":"2024-09-21T12:29:59.216485+0000","last_clean":"2024-09-21T12:29:59.216485+0000","last_became_active":"2024-09-21T12:22:50.930615+0000","last_became_peered":"2024-09-21T12:22:50.930615+0000","last_unstale":"2024-09-21T12:29:59.216485+0000","last_undegraded":"2024-09-21T12:29:59.216485+0000","last_fullsized":"2024-09-21T12:29:59.216485+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:59.216442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:29:59.216442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:31:06.036173+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":467,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.158645+0000","last_change":"2024-09-21T12:28:41.537845+0000","last_active":"2024-09-21T12:29:59.158645+0000","last_peered":"2024-09-21T12:29:59.158645+0000","last_clean":"2024-09-21T12:29:59.158645+0000","last_became_active":"2024-09-21T12:22:50.917408+0000","last_became_peered":"2024-09-21T12:22:50.917408+0000","last_unstale":"2024-09-21T12:29:59.158645+0000","last_undegraded":"2024-09-21T12:29:59.158645+0000","last_fullsized":"2024-09-21T12:29:59.158645+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:41.537786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:41.537786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"76'79","reported_seq":128,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.158891+0000","last_change":"2024-09-21T12:27:29.341314+0000","last_active":"2024-09-21T12:29:59.158891+0000","last_peered":"2024-09-21T12:29:59.158891+0000","last_clean":"2024-09-21T12:29:59.158891+0000","last_became_active":"2024-09-21T12:27:16.948697+0000","last_became_peered":"2024-09-21T12:27:16.948697+0000","last_unstale":"2024-09-21T12:29:59.158891+0000","last_undegraded":"2024-09-21T12:29:59.158891+0000","last_fullsized":"2024-09-21T12:29:59.158891+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:29.341268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:29.341268+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:31.477463+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":466,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.152169+0000","last_change":"2024-09-21T12:28:46.749904+0000","last_active":"2024-09-21T12:29:59.152169+0000","last_peered":"2024-09-21T12:29:59.152169+0000","last_clean":"2024-09-21T12:29:59.152169+0000","last_became_active":"2024-09-21T12:22:50.914221+0000","last_became_peered":"2024-09-21T12:22:50.914221+0000","last_unstale":"2024-09-21T12:29:59.152169+0000","last_undegraded":"2024-09-21T12:29:59.152169+0000","last_fullsized":"2024-09-21T12:29:59.152169+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:46.749839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:46.749839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:07.174169+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":472,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.156921+0000","last_change":"2024-09-21T12:29:35.772635+0000","last_active":"2024-09-21T12:29:59.156921+0000","last_peered":"2024-09-21T12:29:59.156921+0000","last_clean":"2024-09-21T12:29:59.156921+0000","last_became_active":"2024-09-21T12:29:35.771591+0000","last_became_peered":"2024-09-21T12:29:35.771591+0000","last_unstale":"2024-09-21T12:29:59.156921+0000","last_undegraded":"2024-09-21T12:29:59.156921+0000","last_fullsized":"2024-09-21T12:29:59.156921+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:15.071234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:15.071234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:35.981843+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":473,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.158378+0000","last_change":"2024-09-21T12:27:31.860075+0000","last_active":"2024-09-21T12:29:59.158378+0000","last_peered":"2024-09-21T12:29:59.158378+0000","last_clean":"2024-09-21T12:29:59.158378+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:29:59.158378+0000","last_undegraded":"2024-09-21T12:29:59.158378+0000","last_fullsized":"2024-09-21T12:29:59.158378+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:27:31.860002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:31.860002+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":476,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T12:29:59.152131+0000","last_change":"2024-09-21T12:28:28.741483+0000","last_active":"2024-09-21T12:29:59.152131+0000","last_peered":"2024-09-21T12:29:59.152131+0000","last_clean":"2024-09-21T12:29:59.152131+0000","last_became_active":"2024-09-21T12:20:19.308247+0000","last_became_peered":"2024-09-21T12:20:19.308247+0000","last_unstale":"2024-09-21T12:29:59.152131+0000","last_undegraded":"2024-09-21T12:29:59.152131+0000","last_fullsized":"2024-09-21T12:29:59.152131+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:28.741418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:28.741418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":7889354776,"num_objects":120383,"num_object_clones":0,"num_object_copies":361149,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":55056,"num_objects_unfound":0,"num_objects_dirty":120383,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127681,"num_write_kb":7937985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17524,"num_bytes_recovered":1148452864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24494419968,"data_stored":24494407752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62990,"ondisk_log_size":62990,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":684,"num_read_kb":441,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":216,"num_bytes_recovered":4400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":397,"ondisk_log_size":397,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":690,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":5023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":580,"num_read_kb":575,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":761964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":15,"num_write":7,"num_write_kb":111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":40252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2449,"ondisk_log_size":2449,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234105,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4990900,"kb_used_data":4598572,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":88754252,"statfs":{"total":95995035648,"available":90884354048,"internally_reserved":0,"allocated":4708937728,"data_stored":4708591037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":401537482},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463041,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2320520,"kb_used_data":1946424,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":91424632,"statfs":{"total":95995035648,"available":93618823168,"internally_reserved":0,"allocated":1993138176,"data_stored":1992778756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":383056326},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724676,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4720072,"kb_used_data":4368184,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":89025080,"statfs":{"total":95995035648,"available":91161681920,"internally_reserved":0,"allocated":4473020416,"data_stored":4472662465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953612,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149842,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4489872,"kb_used_data":3971264,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":89255280,"statfs":{"total":95995035648,"available":91397406720,"internally_reserved":0,"allocated":4066574336,"data_stored":4066207942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":530905543},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411480,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1958008,"kb_used_data":1615848,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":91787144,"statfs":{"total":95995035648,"available":93990035456,"internally_reserved":0,"allocated":1654628352,"data_stored":1654285359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575006,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4596876,"kb_used_data":4110332,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":89148276,"statfs":{"total":95995035648,"available":91287834624,"internally_reserved":0,"allocated":4208979968,"data_stored":4208616475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":498203077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738532,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3747664,"kb_used_data":3373568,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":89997488,"statfs":{"total":95995035648,"available":92157427712,"internally_reserved":0,"allocated":3454533632,"data_stored":3454167092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3443654656,"data_stored":3443654656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4207738880,"data_stored":4207738880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1650720768,"data_stored":1650720768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4042719232,"data_stored":4042719232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4460122112,"data_stored":4460118040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1985024000,"data_stored":1985019928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4704440320,"data_stored":4704436248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:30:06.878 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-09-21T12:30:06.899 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:06.905 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:30:06.926 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:06.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:06.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:07.028 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:07.075 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:07.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:07.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:07.177 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:07.202 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:07.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:07.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:07.304 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:07.331 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:07.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:07.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:07.405 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:07 smithi114 ceph-mon[30324]: pgmap v482: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 7.3 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 767 B/s rd, 4.5 MiB/s wr, 529 op/s; 55056/361746 objects misplaced (15.220%); 15 MiB/s, 234 objects/s recovering 2024-09-21T12:30:07.405 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:07 smithi114 ceph-mon[40867]: pgmap v482: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 7.3 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 767 B/s rd, 4.5 MiB/s wr, 529 op/s; 55056/361746 objects misplaced (15.220%); 15 MiB/s, 234 objects/s recovering 2024-09-21T12:30:07.432 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:07.458 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:07.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:07.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:07.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:07 smithi183 ceph-mon[33277]: pgmap v482: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 7.3 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 767 B/s rd, 4.5 MiB/s wr, 529 op/s; 55056/361746 objects misplaced (15.220%); 15 MiB/s, 234 objects/s recovering 2024-09-21T12:30:07.488 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:30:07 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:30:07.560 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:07.611 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:07.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:07.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:07.712 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:07.725 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:30:07.725 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:30:07.739 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:07.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:07.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:07.841 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:07.870 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:07.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:07.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:07.971 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:08.012 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:08.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:08.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:08.113 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:08.157 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:08.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:08.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:08.258 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:08.266 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:08 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:30:08.266 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:08 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4210494097' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:30:08.266 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:08 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:30:08.266 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:08 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/4210494097' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:30:08.295 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:08.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:08.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:08.389 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:30:08.389 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:30:08.389 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:30:08.389 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:30:08.389 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:30:08.389 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:30:08.389 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:30:08.389 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:30:08.389 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:30:08.389 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:30:08.389 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:30:08.389 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:30:08.390 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:30:08.390 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:30:08.390 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:30:08.390 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:30:08.397 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:08.424 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:08.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:08.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:08.428 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T12:30:08.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:08 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:30:08.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:08 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/4210494097' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:30:08.526 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:08.547 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:08.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:08.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:08.649 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:08.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:08.689 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:08.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:08.790 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:08.812 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:08.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:08.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:08.914 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:08.953 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:08.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:08.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:09.054 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:09.075 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:09.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:09.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:09.178 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:09.207 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:09.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:09.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:09.308 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:09.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:09.347 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:09.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:09.448 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:09.473 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:09.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:09.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:09.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:09 smithi183 ceph-mon[33277]: pgmap v483: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.3 KiB/s rd, 4.2 MiB/s wr, 1.09k op/s; 50396/346272 objects misplaced (14.554%); 28 MiB/s, 445 objects/s recovering 2024-09-21T12:30:09.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:09 smithi114 ceph-mon[40867]: pgmap v483: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.3 KiB/s rd, 4.2 MiB/s wr, 1.09k op/s; 50396/346272 objects misplaced (14.554%); 28 MiB/s, 445 objects/s recovering 2024-09-21T12:30:09.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:09 smithi114 ceph-mon[30324]: pgmap v483: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.3 KiB/s rd, 4.2 MiB/s wr, 1.09k op/s; 50396/346272 objects misplaced (14.554%); 28 MiB/s, 445 objects/s recovering 2024-09-21T12:30:09.575 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:09.602 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:09.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:09.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:09.704 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:09.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:09.726 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:09.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:09.826 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:09.855 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:09.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:09.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:09.957 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:09.982 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:09.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:09.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:10.084 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:10.105 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:10.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:10.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:10.207 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:10.244 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:10.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:10.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:10.346 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:10.376 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:10.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:10.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:10.478 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:10.520 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:10.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:10.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:10.622 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:10.649 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:10.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:10.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:10.751 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:10.773 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:10.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:10.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:10.874 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:10.898 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:10.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:10.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:11.000 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:11.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:11.027 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:11.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:11.128 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:11.149 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:11.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:11.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:11.250 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:11.287 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:11.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:11.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:11.389 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:11.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:11.411 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:11.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:11.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:11 smithi183 ceph-mon[33277]: pgmap v484: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 3.6 MiB/s wr, 939 op/s; 50396/346272 objects misplaced (14.554%); 24 MiB/s, 384 objects/s recovering 2024-09-21T12:30:11.512 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:11.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:11 smithi114 ceph-mon[40867]: pgmap v484: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 3.6 MiB/s wr, 939 op/s; 50396/346272 objects misplaced (14.554%); 24 MiB/s, 384 objects/s recovering 2024-09-21T12:30:11.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:11 smithi114 ceph-mon[30324]: pgmap v484: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 3.6 MiB/s wr, 939 op/s; 50396/346272 objects misplaced (14.554%); 24 MiB/s, 384 objects/s recovering 2024-09-21T12:30:11.535 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:11.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:11.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:11.637 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:11.661 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:11.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:11.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:11.763 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:11.784 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:11.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:11.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:11.878 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T12:30:11.878 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:30:11.879 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_2 2024-09-21T12:30:11.879 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:30:11.886 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:11.909 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:11.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:11.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:12.011 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:12.037 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:12.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:12.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:12.138 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:12.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:12.196 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:12.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:12.297 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:12.324 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:12.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:12.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:12.426 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:12.460 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:12.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:12.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:12.561 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:12.583 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:12.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:12.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:12.684 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:12.721 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:12.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:12.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:12.823 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:12.863 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:12.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:12.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:12.965 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:13.002 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:13.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:13.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:13.024 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:30:12 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:30:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:30:13.105 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:13.131 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:13.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:13.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:13.232 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:13.259 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:13.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:13.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:13.360 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:13.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:13.406 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:13.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:13.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:13 smithi183 ceph-mon[33277]: pgmap v485: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 2.1 MiB/s wr, 940 op/s; 50396/343149 objects misplaced (14.686%); 24 MiB/s, 384 objects/s recovering 2024-09-21T12:30:13.506 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:13.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:13 smithi114 ceph-mon[40867]: pgmap v485: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 2.1 MiB/s wr, 940 op/s; 50396/343149 objects misplaced (14.686%); 24 MiB/s, 384 objects/s recovering 2024-09-21T12:30:13.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:13 smithi114 ceph-mon[30324]: pgmap v485: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 2.1 MiB/s wr, 940 op/s; 50396/343149 objects misplaced (14.686%); 24 MiB/s, 384 objects/s recovering 2024-09-21T12:30:13.534 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:13.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:13.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:13.635 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:13.668 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:13.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:13.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:13.770 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:13.794 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:13.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:13.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:13.895 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:13.934 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:13.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:13.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:14.036 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:14.057 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:14.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:14.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:14.159 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:14.184 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:14.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:14.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:14.286 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:14.323 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:14.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:14.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:14.424 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:14.459 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:14.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:14.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:14.561 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:14.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:14.584 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:14.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:14.685 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:14.725 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:14.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:14.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:14.827 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:14.866 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:14.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:14.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:14.968 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:14.989 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:14.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:14.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:15.091 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:15.114 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:15.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:15.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:15.215 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:15.241 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:15.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:15.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:15.342 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:15.363 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:15.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:15.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:15.465 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:15.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:15 smithi183 ceph-mon[33277]: pgmap v486: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 920 op/s; 46560/334149 objects misplaced (13.934%); 24 MiB/s, 377 objects/s recovering 2024-09-21T12:30:15.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:15 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:30:15.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:15 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:30:15.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:15 smithi183 ceph-mon[33277]: osdmap e275: 8 total, 8 up, 7 in 2024-09-21T12:30:15.498 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:15.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:15.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:15.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:15 smithi114 ceph-mon[40867]: pgmap v486: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 920 op/s; 46560/334149 objects misplaced (13.934%); 24 MiB/s, 377 objects/s recovering 2024-09-21T12:30:15.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:15 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:30:15.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:15 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:30:15.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:15 smithi114 ceph-mon[40867]: osdmap e275: 8 total, 8 up, 7 in 2024-09-21T12:30:15.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:15 smithi114 ceph-mon[30324]: pgmap v486: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 920 op/s; 46560/334149 objects misplaced (13.934%); 24 MiB/s, 377 objects/s recovering 2024-09-21T12:30:15.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:15 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:30:15.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:15 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:30:15.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:15 smithi114 ceph-mon[30324]: osdmap e275: 8 total, 8 up, 7 in 2024-09-21T12:30:15.600 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:15.641 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:15.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:15.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:15.743 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:15.768 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:15.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:15.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:15.869 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:15.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:15.895 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:15.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:15.996 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:16.022 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:16.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:16.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:16.123 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:16.150 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:16.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:16.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:16.252 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:16.279 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:16.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:16.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:16.380 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:16.407 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:16.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:16.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:16.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T12:30:16.509 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:16.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:16.559 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:16.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:16.660 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:16.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:16.700 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:16.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:16.801 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:16.827 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:16.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:16.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:16.929 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:16.952 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:16.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:16.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:16.971 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:30:16.983 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:30:17.053 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:17.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:17.097 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:17.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:17.198 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:17.223 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:17.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:17.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:17.325 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:17.350 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:17.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:17.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:17.391 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:17 smithi114 ceph-mon[40867]: pgmap v488: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 920 op/s; 46560/334149 objects misplaced (13.934%); 24 MiB/s, 377 objects/s recovering 2024-09-21T12:30:17.392 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:17 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:30:17.392 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:17 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:30:17.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:17 smithi114 ceph-mon[30324]: pgmap v488: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 920 op/s; 46560/334149 objects misplaced (13.934%); 24 MiB/s, 377 objects/s recovering 2024-09-21T12:30:17.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:17 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:30:17.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:17 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:30:17.451 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:17.475 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:17.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:17.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:17.490 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:30:17 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:30:17.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:17 smithi183 ceph-mon[33277]: pgmap v488: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 920 op/s; 46560/334149 objects misplaced (13.934%); 24 MiB/s, 377 objects/s recovering 2024-09-21T12:30:17.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:17 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:30:17.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:17 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:30:17.577 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:17.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:17.616 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:17.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:17.717 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:17.759 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:17.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:17.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:17.861 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:17.886 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:17.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:17.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:17.988 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:18.011 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:18.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:18.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:18.112 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:18.137 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:18.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:18.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:18.239 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:18.281 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:18.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:18.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:18.383 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:18.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:18.405 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:18.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:18.433 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:30:18.434 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:18 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:30:18.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:18 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:30:18.439 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:30:18.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:18 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:30:18.506 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:18.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:18.541 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:18.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:18.642 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:18.682 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:18.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:18.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:18.783 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:18.821 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:18.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:18.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:18.922 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:18.942 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:18.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:18.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:18.988 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:30:19.045 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:19.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:19.081 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:19.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:19.182 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:19.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:19.220 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:19.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:19.253 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-21T12:30:19.272 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:30:19.321 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:19.347 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:19.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:19.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:19.449 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:19 smithi183 ceph-mon[33277]: pgmap v489: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 850 op/s; 42070/320787 objects misplaced (13.115%); 27 MiB/s, 424 objects/s recovering 2024-09-21T12:30:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:19 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2304803973' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:30:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:19 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:30:19.491 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:19.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:19.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:19.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:19 smithi114 ceph-mon[40867]: pgmap v489: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 850 op/s; 42070/320787 objects misplaced (13.115%); 27 MiB/s, 424 objects/s recovering 2024-09-21T12:30:19.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:19 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2304803973' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:30:19.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:19 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:30:19.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:19 smithi114 ceph-mon[30324]: pgmap v489: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 850 op/s; 42070/320787 objects misplaced (13.115%); 27 MiB/s, 424 objects/s recovering 2024-09-21T12:30:19.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:19 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2304803973' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:30:19.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:19 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:30:19.593 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:19.620 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:19.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:19.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:19.721 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:19.764 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:19.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:19.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:19.866 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:19.893 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:19.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:19.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:19.994 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:20.033 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:20.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:20.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:20.135 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:20.156 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:20.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:20.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:20.257 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:20.278 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:20.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:20.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:20.353 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:20 smithi114 ceph-mon[40867]: from='client.15711 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:30:20.353 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:20 smithi114 ceph-mon[30324]: from='client.15711 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:30:20.353 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:20 smithi114 ceph-mon[30324]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:30:20.380 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:20.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:20.412 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:20.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:20 smithi183 ceph-mon[33277]: from='client.15711 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:30:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:20 smithi183 ceph-mon[33277]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:30:20.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:20 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:30:20.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:20 smithi183 ceph-mon[33277]: osdmap e276: 8 total, 8 up, 7 in 2024-09-21T12:30:20.513 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:20.534 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:20.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:20.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:20.635 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:20.656 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:20.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:20.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:20.758 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:20.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:20 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:30:20.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:20 smithi114 ceph-mon[30324]: osdmap e276: 8 total, 8 up, 7 in 2024-09-21T12:30:20.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:20 smithi114 ceph-mon[40867]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:30:20.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:20 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:30:20.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:20 smithi114 ceph-mon[40867]: osdmap e276: 8 total, 8 up, 7 in 2024-09-21T12:30:20.786 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:20.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:20.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:20.887 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:20.913 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:20.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:20.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:21.014 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:21.039 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:21.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:21.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:21.141 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:21.182 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:21.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:21.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:21.284 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:21.309 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:21.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:21.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:21.411 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:21.439 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:21.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:21.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:21.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:21 smithi183 ceph-mon[33277]: pgmap v491: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 932 op/s; 42070/320787 objects misplaced (13.115%); 33 MiB/s, 530 objects/s recovering 2024-09-21T12:30:21.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:21 smithi114 ceph-mon[40867]: pgmap v491: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 932 op/s; 42070/320787 objects misplaced (13.115%); 33 MiB/s, 530 objects/s recovering 2024-09-21T12:30:21.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:21 smithi114 ceph-mon[30324]: pgmap v491: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 932 op/s; 42070/320787 objects misplaced (13.115%); 33 MiB/s, 530 objects/s recovering 2024-09-21T12:30:21.541 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:21.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:21.566 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:21.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:21.666 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:21.707 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:21.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:21.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:21.809 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:21.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:21.845 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:21.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:21.946 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:21.975 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:21.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:21.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:22.077 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:22.101 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:22.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:22.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:22.202 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:22.239 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:22.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:22.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:22.341 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:22.367 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:22.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:22.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:22.469 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:22.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:22 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:30:22.499 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:22.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:22.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:22.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:22 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:30:22.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:22 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:30:22.601 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:22.642 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:22.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:22.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:22.744 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:22.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:22.784 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:22.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:22.885 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:22.909 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:22.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:22.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:23.011 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:23.025 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:30:22 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:30:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:30:23.054 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:23.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:23.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:23.156 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:23.176 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:23.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:23.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:23.277 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:23.314 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:23.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:23.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:23.416 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:23.457 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:23.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:23.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:23.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:23 smithi183 ceph-mon[33277]: pgmap v492: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 6.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 685 op/s; 42070/317724 objects misplaced (13.241%); 19 MiB/s, 304 objects/s recovering 2024-09-21T12:30:23.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:23 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:30:23.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:23 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:30:23.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:23 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:30:23.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:23 smithi114 ceph-mon[40867]: pgmap v492: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 6.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 685 op/s; 42070/317724 objects misplaced (13.241%); 19 MiB/s, 304 objects/s recovering 2024-09-21T12:30:23.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:23 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:30:23.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:23 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:30:23.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:23 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:30:23.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:23 smithi114 ceph-mon[30324]: pgmap v492: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 6.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 685 op/s; 42070/317724 objects misplaced (13.241%); 19 MiB/s, 304 objects/s recovering 2024-09-21T12:30:23.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:23 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:30:23.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:23 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:30:23.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:23 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:30:23.558 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:23.583 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:23.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:23.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:23.685 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:23.713 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:23.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:23.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:23.815 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:23.837 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:23.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:23.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:23.938 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:23.963 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:23.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:23.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:24.065 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:24.106 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:24.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:24.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:24.208 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:24.247 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:24.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:24.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:24.349 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:24.374 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:24.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:24.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:24.476 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:24.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:24.517 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:24.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:24.618 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:24.644 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:24.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:24.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:24.745 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:24.790 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:24.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:24.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:24.892 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:24.917 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:24.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:24.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:25.019 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:25.047 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:25.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:25.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:25.149 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:25.174 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:25.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:25.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:25.276 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:25.301 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:25.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:25.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:25.403 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:25.423 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:25.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:25.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:25.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:25 smithi114 ceph-mon[40867]: pgmap v493: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 7.7 KiB/s rd, 0 B/s wr, 912 op/s; 38277/308406 objects misplaced (12.411%); 29 MiB/s, 459 objects/s recovering 2024-09-21T12:30:25.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:25 smithi114 ceph-mon[30324]: pgmap v493: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 7.7 KiB/s rd, 0 B/s wr, 912 op/s; 38277/308406 objects misplaced (12.411%); 29 MiB/s, 459 objects/s recovering 2024-09-21T12:30:25.525 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:25.547 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:25.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:25.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:25.649 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:25.670 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:25.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:25.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:25.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:25 smithi183 ceph-mon[33277]: pgmap v493: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 7.7 KiB/s rd, 0 B/s wr, 912 op/s; 38277/308406 objects misplaced (12.411%); 29 MiB/s, 459 objects/s recovering 2024-09-21T12:30:25.771 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:25.810 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:25.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:25.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:25.912 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:25.938 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:25.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:25.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:26.039 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:26.072 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:26.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:26.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:26.174 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:26.213 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:26.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:26.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:26.291 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":489,"stamp":"2024-09-21T12:30:17.770238+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6995359871,"num_objects":106929,"num_object_clones":0,"num_object_copies":320787,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":42070,"num_objects_unfound":0,"num_objects_dirty":106929,"num_whiteouts":0,"num_read":1311,"num_read_kb":1071,"num_write":141777,"num_write_kb":7938141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23980,"num_bytes_recovered":1555673387,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65888,"ondisk_log_size":65888,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":543,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":24663424,"kb_used_data":21716748,"kb_used_omap":10,"kb_used_meta":2946165,"kb_avail":631552640,"statfs":{"total":671965249536,"available":646709903360,"internally_reserved":0,"allocated":22237949952,"data_stored":22235697098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":3016873067},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,5,0,9,9,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-556728320,"num_objects":-8495,"num_object_clones":0,"num_object_copies":-25485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-8326,"num_objects_unfound":0,"num_objects_dirty":-8495,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8495,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4243,"num_bytes_recovered":278069248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003484"},"pg_stats":[{"pgid":"1.1d","version":"76'79","reported_seq":149,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.265553+0000","last_change":"2024-09-21T12:29:35.768052+0000","last_active":"2024-09-21T12:30:14.265553+0000","last_peered":"2024-09-21T12:30:14.265553+0000","last_clean":"2024-09-21T12:30:14.265553+0000","last_became_active":"2024-09-21T12:29:35.765079+0000","last_became_peered":"2024-09-21T12:29:35.765079+0000","last_unstale":"2024-09-21T12:30:14.265553+0000","last_undegraded":"2024-09-21T12:30:14.265553+0000","last_fullsized":"2024-09-21T12:30:14.265553+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:47.816137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:47.816137+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":464,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.257955+0000","last_change":"2024-09-21T12:24:50.115300+0000","last_active":"2024-09-21T12:30:14.257955+0000","last_peered":"2024-09-21T12:30:14.257955+0000","last_clean":"2024-09-21T12:30:14.257955+0000","last_became_active":"2024-09-21T12:20:15.309431+0000","last_became_peered":"2024-09-21T12:20:15.309431+0000","last_unstale":"2024-09-21T12:30:14.257955+0000","last_undegraded":"2024-09-21T12:30:14.257955+0000","last_fullsized":"2024-09-21T12:30:14.257955+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:50.115253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:50.115253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:24.094870+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":92,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.264742+0000","last_change":"2024-09-21T12:29:08.671789+0000","last_active":"2024-09-21T12:30:14.264742+0000","last_peered":"2024-09-21T12:30:14.264742+0000","last_clean":"2024-09-21T12:30:14.264742+0000","last_became_active":"2024-09-21T12:29:05.833372+0000","last_became_peered":"2024-09-21T12:29:05.833372+0000","last_unstale":"2024-09-21T12:30:14.264742+0000","last_undegraded":"2024-09-21T12:30:14.264742+0000","last_fullsized":"2024-09-21T12:30:14.264742+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:29:08.671723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:29:08.671723+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:20.962766+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":462,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.265229+0000","last_change":"2024-09-21T12:29:10.672659+0000","last_active":"2024-09-21T12:30:14.265229+0000","last_peered":"2024-09-21T12:30:14.265229+0000","last_clean":"2024-09-21T12:30:14.265229+0000","last_became_active":"2024-09-21T12:20:19.314908+0000","last_became_peered":"2024-09-21T12:20:19.314908+0000","last_unstale":"2024-09-21T12:30:14.265229+0000","last_undegraded":"2024-09-21T12:30:14.265229+0000","last_fullsized":"2024-09-21T12:30:14.265229+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:10.672599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:29:10.672599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:36.036960+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":393,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.258181+0000","last_change":"2024-09-21T12:29:35.761515+0000","last_active":"2024-09-21T12:30:14.258181+0000","last_peered":"2024-09-21T12:30:14.258181+0000","last_clean":"2024-09-21T12:30:14.258181+0000","last_became_active":"2024-09-21T12:29:35.760508+0000","last_became_peered":"2024-09-21T12:29:35.760508+0000","last_unstale":"2024-09-21T12:30:14.258181+0000","last_undegraded":"2024-09-21T12:30:14.258181+0000","last_fullsized":"2024-09-21T12:30:14.258181+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:14.158965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:29:14.158965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:41.582027+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1c","version":"76'79","reported_seq":132,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.257428+0000","last_change":"2024-09-21T12:27:28.374689+0000","last_active":"2024-09-21T12:30:14.257428+0000","last_peered":"2024-09-21T12:30:14.257428+0000","last_clean":"2024-09-21T12:30:14.257428+0000","last_became_active":"2024-09-21T12:27:16.920545+0000","last_became_peered":"2024-09-21T12:27:16.920545+0000","last_unstale":"2024-09-21T12:30:14.257428+0000","last_undegraded":"2024-09-21T12:30:14.257428+0000","last_fullsized":"2024-09-21T12:30:14.257428+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:28.374616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:28.374616+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:32.518277+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":440,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.258714+0000","last_change":"2024-09-21T12:25:48.316122+0000","last_active":"2024-09-21T12:30:14.258714+0000","last_peered":"2024-09-21T12:30:14.258714+0000","last_clean":"2024-09-21T12:30:14.258714+0000","last_became_active":"2024-09-21T12:20:21.595301+0000","last_became_peered":"2024-09-21T12:20:21.595301+0000","last_unstale":"2024-09-21T12:30:14.258714+0000","last_undegraded":"2024-09-21T12:30:14.258714+0000","last_fullsized":"2024-09-21T12:30:14.258714+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:48.316039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:48.316039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":469,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.259350+0000","last_change":"2024-09-21T12:29:16.079291+0000","last_active":"2024-09-21T12:30:14.259350+0000","last_peered":"2024-09-21T12:30:14.259350+0000","last_clean":"2024-09-21T12:30:14.259350+0000","last_became_active":"2024-09-21T12:20:15.308655+0000","last_became_peered":"2024-09-21T12:20:15.308655+0000","last_unstale":"2024-09-21T12:30:14.259350+0000","last_undegraded":"2024-09-21T12:30:14.259350+0000","last_fullsized":"2024-09-21T12:30:14.259350+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:16.079221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:16.079221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:18.495891+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":490,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.264959+0000","last_change":"2024-09-21T12:27:54.371491+0000","last_active":"2024-09-21T12:30:14.264959+0000","last_peered":"2024-09-21T12:30:14.264959+0000","last_clean":"2024-09-21T12:30:14.264959+0000","last_became_active":"2024-09-21T12:20:17.305220+0000","last_became_peered":"2024-09-21T12:20:17.305220+0000","last_unstale":"2024-09-21T12:30:14.264959+0000","last_undegraded":"2024-09-21T12:30:14.264959+0000","last_fullsized":"2024-09-21T12:30:14.264959+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:27:54.369084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:54.369084+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":403,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.261911+0000","last_change":"2024-09-21T12:28:39.179737+0000","last_active":"2024-09-21T12:30:14.261911+0000","last_peered":"2024-09-21T12:30:14.261911+0000","last_clean":"2024-09-21T12:30:14.261911+0000","last_became_active":"2024-09-21T12:22:50.926240+0000","last_became_peered":"2024-09-21T12:22:50.926240+0000","last_unstale":"2024-09-21T12:30:14.261911+0000","last_undegraded":"2024-09-21T12:30:14.261911+0000","last_fullsized":"2024-09-21T12:30:14.261911+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:39.179665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:39.179665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:31:12.110934+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":307,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.261256+0000","last_change":"2024-09-21T12:30:01.256907+0000","last_active":"2024-09-21T12:30:14.261256+0000","last_peered":"2024-09-21T12:30:14.261256+0000","last_clean":"2024-09-21T12:30:14.261256+0000","last_became_active":"2024-09-21T12:26:54.083981+0000","last_became_peered":"2024-09-21T12:26:54.083981+0000","last_unstale":"2024-09-21T12:30:14.261256+0000","last_undegraded":"2024-09-21T12:30:14.261256+0000","last_fullsized":"2024-09-21T12:30:14.261256+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:30:01.256865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:30:01.256865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:31:17.456222+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":450,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.260165+0000","last_change":"2024-09-21T12:28:06.589984+0000","last_active":"2024-09-21T12:30:14.260165+0000","last_peered":"2024-09-21T12:30:14.260165+0000","last_clean":"2024-09-21T12:30:14.260165+0000","last_became_active":"2024-09-21T12:22:50.927804+0000","last_became_peered":"2024-09-21T12:22:50.927804+0000","last_unstale":"2024-09-21T12:30:14.260165+0000","last_undegraded":"2024-09-21T12:30:14.260165+0000","last_fullsized":"2024-09-21T12:30:14.260165+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:28:06.589926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:06.589926+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":477,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.258314+0000","last_change":"2024-09-21T12:29:37.148220+0000","last_active":"2024-09-21T12:30:14.258314+0000","last_peered":"2024-09-21T12:30:14.258314+0000","last_clean":"2024-09-21T12:30:14.258314+0000","last_became_active":"2024-09-21T12:29:35.778190+0000","last_became_peered":"2024-09-21T12:29:35.778190+0000","last_unstale":"2024-09-21T12:30:14.258314+0000","last_undegraded":"2024-09-21T12:30:14.258314+0000","last_fullsized":"2024-09-21T12:30:14.258314+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:37.148174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:29:37.148174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:37.365028+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":460,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.258498+0000","last_change":"2024-09-21T12:27:54.959865+0000","last_active":"2024-09-21T12:30:14.258498+0000","last_peered":"2024-09-21T12:30:14.258498+0000","last_clean":"2024-09-21T12:30:14.258498+0000","last_became_active":"2024-09-21T12:20:21.309190+0000","last_became_peered":"2024-09-21T12:20:21.309190+0000","last_unstale":"2024-09-21T12:30:14.258498+0000","last_undegraded":"2024-09-21T12:30:14.258498+0000","last_fullsized":"2024-09-21T12:30:14.258498+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:54.959806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:27:54.959806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"76'79","reported_seq":225,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.259204+0000","last_change":"2024-09-21T12:27:20.216167+0000","last_active":"2024-09-21T12:30:14.259204+0000","last_peered":"2024-09-21T12:30:14.259204+0000","last_clean":"2024-09-21T12:30:14.259204+0000","last_became_active":"2024-09-21T12:27:16.937711+0000","last_became_peered":"2024-09-21T12:27:16.937711+0000","last_unstale":"2024-09-21T12:30:14.259204+0000","last_undegraded":"2024-09-21T12:30:14.259204+0000","last_fullsized":"2024-09-21T12:30:14.259204+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:20.216102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:20.216102+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":307,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.266355+0000","last_change":"2024-09-21T12:28:33.644630+0000","last_active":"2024-09-21T12:30:14.266355+0000","last_peered":"2024-09-21T12:30:14.266355+0000","last_clean":"2024-09-21T12:30:14.266355+0000","last_became_active":"2024-09-21T12:26:54.085055+0000","last_became_peered":"2024-09-21T12:26:54.085055+0000","last_unstale":"2024-09-21T12:30:14.266355+0000","last_undegraded":"2024-09-21T12:30:14.266355+0000","last_fullsized":"2024-09-21T12:30:14.266355+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:33.644570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:33.644570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":449,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.258941+0000","last_change":"2024-09-21T12:29:35.982520+0000","last_active":"2024-09-21T12:30:14.258941+0000","last_peered":"2024-09-21T12:30:14.258941+0000","last_clean":"2024-09-21T12:30:14.258941+0000","last_became_active":"2024-09-21T12:29:35.767433+0000","last_became_peered":"2024-09-21T12:29:35.767433+0000","last_unstale":"2024-09-21T12:30:14.258941+0000","last_undegraded":"2024-09-21T12:30:14.258941+0000","last_fullsized":"2024-09-21T12:30:14.258941+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-21T12:29:35.982468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:29:35.982468+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:43.856648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":475,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.258673+0000","last_change":"2024-09-21T12:28:04.623499+0000","last_active":"2024-09-21T12:30:14.258673+0000","last_peered":"2024-09-21T12:30:14.258673+0000","last_clean":"2024-09-21T12:30:14.258673+0000","last_became_active":"2024-09-21T12:22:50.921780+0000","last_became_peered":"2024-09-21T12:22:50.921780+0000","last_unstale":"2024-09-21T12:30:14.258673+0000","last_undegraded":"2024-09-21T12:30:14.258673+0000","last_fullsized":"2024-09-21T12:30:14.258673+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:04.623389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:04.623389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":474,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.257722+0000","last_change":"2024-09-21T12:29:53.301655+0000","last_active":"2024-09-21T12:30:14.257722+0000","last_peered":"2024-09-21T12:30:14.257722+0000","last_clean":"2024-09-21T12:30:14.257722+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:30:14.257722+0000","last_undegraded":"2024-09-21T12:30:14.257722+0000","last_fullsized":"2024-09-21T12:30:14.257722+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:53.301583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:29:53.301583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:31:13.015049+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"76'79","reported_seq":296,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.268308+0000","last_change":"2024-09-21T12:27:22.289065+0000","last_active":"2024-09-21T12:30:14.268308+0000","last_peered":"2024-09-21T12:30:14.268308+0000","last_clean":"2024-09-21T12:30:14.268308+0000","last_became_active":"2024-09-21T12:27:16.957569+0000","last_became_peered":"2024-09-21T12:27:16.957569+0000","last_unstale":"2024-09-21T12:30:14.268308+0000","last_undegraded":"2024-09-21T12:30:14.268308+0000","last_fullsized":"2024-09-21T12:30:14.268308+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:22.288989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:22.288989+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":107,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.256774+0000","last_change":"2024-09-21T12:29:05.826853+0000","last_active":"2024-09-21T12:30:14.256774+0000","last_peered":"2024-09-21T12:30:14.256774+0000","last_clean":"2024-09-21T12:30:14.256774+0000","last_became_active":"2024-09-21T12:29:05.826251+0000","last_became_peered":"2024-09-21T12:29:05.826251+0000","last_unstale":"2024-09-21T12:30:14.256774+0000","last_undegraded":"2024-09-21T12:30:14.256774+0000","last_fullsized":"2024-09-21T12:30:14.256774+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:44.845176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:44.845176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:33.080052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":508,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.262310+0000","last_change":"2024-09-21T12:29:35.773690+0000","last_active":"2024-09-21T12:30:14.262310+0000","last_peered":"2024-09-21T12:30:14.262310+0000","last_clean":"2024-09-21T12:30:14.262310+0000","last_became_active":"2024-09-21T12:29:35.772180+0000","last_became_peered":"2024-09-21T12:29:35.772180+0000","last_unstale":"2024-09-21T12:30:14.262310+0000","last_undegraded":"2024-09-21T12:30:14.262310+0000","last_fullsized":"2024-09-21T12:30:14.262310+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:16.148785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:16.148785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"66'15","reported_seq":492,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.251984+0000","last_change":"2024-09-21T12:27:22.516532+0000","last_active":"2024-09-21T12:30:14.251984+0000","last_peered":"2024-09-21T12:30:14.251984+0000","last_clean":"2024-09-21T12:30:14.251984+0000","last_became_active":"2024-09-21T12:20:17.307828+0000","last_became_peered":"2024-09-21T12:20:17.307828+0000","last_unstale":"2024-09-21T12:30:14.251984+0000","last_undegraded":"2024-09-21T12:30:14.251984+0000","last_fullsized":"2024-09-21T12:30:14.251984+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:27:22.516442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:22.516442+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":489,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.265788+0000","last_change":"2024-09-21T12:28:06.636769+0000","last_active":"2024-09-21T12:30:14.265788+0000","last_peered":"2024-09-21T12:30:14.265788+0000","last_clean":"2024-09-21T12:30:14.265788+0000","last_became_active":"2024-09-21T12:25:46.905878+0000","last_became_peered":"2024-09-21T12:25:46.905878+0000","last_unstale":"2024-09-21T12:30:14.265788+0000","last_undegraded":"2024-09-21T12:30:14.265788+0000","last_fullsized":"2024-09-21T12:30:14.265788+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:06.636729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:06.636729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:33.605451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"76'79","reported_seq":397,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.257568+0000","last_change":"2024-09-21T12:29:51.267936+0000","last_active":"2024-09-21T12:30:14.257568+0000","last_peered":"2024-09-21T12:30:14.257568+0000","last_clean":"2024-09-21T12:30:14.257568+0000","last_became_active":"2024-09-21T12:27:16.927791+0000","last_became_peered":"2024-09-21T12:27:16.927791+0000","last_unstale":"2024-09-21T12:30:14.257568+0000","last_undegraded":"2024-09-21T12:30:14.257568+0000","last_fullsized":"2024-09-21T12:30:14.257568+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:51.267884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:51.267884+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:31:04.534329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":456,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.268810+0000","last_change":"2024-09-21T12:29:35.784513+0000","last_active":"2024-09-21T12:30:14.268810+0000","last_peered":"2024-09-21T12:30:14.268810+0000","last_clean":"2024-09-21T12:30:14.268810+0000","last_became_active":"2024-09-21T12:29:35.781172+0000","last_became_peered":"2024-09-21T12:29:35.781172+0000","last_unstale":"2024-09-21T12:30:14.268810+0000","last_undegraded":"2024-09-21T12:30:14.268810+0000","last_fullsized":"2024-09-21T12:30:14.268810+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:45.880383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.880383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":401,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.265046+0000","last_change":"2024-09-21T12:28:45.726096+0000","last_active":"2024-09-21T12:30:14.265046+0000","last_peered":"2024-09-21T12:30:14.265046+0000","last_clean":"2024-09-21T12:30:14.265046+0000","last_became_active":"2024-09-21T12:22:50.925403+0000","last_became_peered":"2024-09-21T12:22:50.925403+0000","last_unstale":"2024-09-21T12:30:14.265046+0000","last_undegraded":"2024-09-21T12:30:14.265046+0000","last_fullsized":"2024-09-21T12:30:14.265046+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:45.725968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.725968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":545,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.259437+0000","last_change":"2024-09-21T12:28:30.620035+0000","last_active":"2024-09-21T12:30:14.259437+0000","last_peered":"2024-09-21T12:30:14.259437+0000","last_clean":"2024-09-21T12:30:14.259437+0000","last_became_active":"2024-09-21T12:22:50.912144+0000","last_became_peered":"2024-09-21T12:22:50.912144+0000","last_unstale":"2024-09-21T12:30:14.259437+0000","last_undegraded":"2024-09-21T12:30:14.259437+0000","last_fullsized":"2024-09-21T12:30:14.259437+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:28:30.619954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:30.619954+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":379,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.256980+0000","last_change":"2024-09-21T12:26:52.295004+0000","last_active":"2024-09-21T12:30:14.256980+0000","last_peered":"2024-09-21T12:30:14.256980+0000","last_clean":"2024-09-21T12:30:14.256980+0000","last_became_active":"2024-09-21T12:22:50.918326+0000","last_became_peered":"2024-09-21T12:22:50.918326+0000","last_unstale":"2024-09-21T12:30:14.256980+0000","last_undegraded":"2024-09-21T12:30:14.256980+0000","last_fullsized":"2024-09-21T12:30:14.256980+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:52.294931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:52.294931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"76'79","reported_seq":189,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.258025+0000","last_change":"2024-09-21T12:28:46.294913+0000","last_active":"2024-09-21T12:30:14.258025+0000","last_peered":"2024-09-21T12:30:14.258025+0000","last_clean":"2024-09-21T12:30:14.258025+0000","last_became_active":"2024-09-21T12:27:16.933358+0000","last_became_peered":"2024-09-21T12:27:16.933358+0000","last_unstale":"2024-09-21T12:30:14.258025+0000","last_undegraded":"2024-09-21T12:30:14.258025+0000","last_fullsized":"2024-09-21T12:30:14.258025+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:46.294855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:46.294855+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:31:08.362790+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":442,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.256688+0000","last_change":"2024-09-21T12:23:51.065780+0000","last_active":"2024-09-21T12:30:14.256688+0000","last_peered":"2024-09-21T12:30:14.256688+0000","last_clean":"2024-09-21T12:30:14.256688+0000","last_became_active":"2024-09-21T12:20:21.312976+0000","last_became_peered":"2024-09-21T12:20:21.312976+0000","last_unstale":"2024-09-21T12:30:14.256688+0000","last_undegraded":"2024-09-21T12:30:14.256688+0000","last_fullsized":"2024-09-21T12:30:14.256688+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:23:51.065733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:51.065733+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:20.808745+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":504,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.265419+0000","last_change":"2024-09-21T12:27:56.597623+0000","last_active":"2024-09-21T12:30:14.265419+0000","last_peered":"2024-09-21T12:30:14.265419+0000","last_clean":"2024-09-21T12:30:14.265419+0000","last_became_active":"2024-09-21T12:22:50.910933+0000","last_became_peered":"2024-09-21T12:22:50.910933+0000","last_unstale":"2024-09-21T12:30:14.265419+0000","last_undegraded":"2024-09-21T12:30:14.265419+0000","last_fullsized":"2024-09-21T12:30:14.265419+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:27:56.597473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:27:56.597473+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:34.612211+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"66'10","reported_seq":491,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.258216+0000","last_change":"2024-09-21T12:29:35.808571+0000","last_active":"2024-09-21T12:30:14.258216+0000","last_peered":"2024-09-21T12:30:14.258216+0000","last_clean":"2024-09-21T12:30:14.258216+0000","last_became_active":"2024-09-21T12:29:35.782223+0000","last_became_peered":"2024-09-21T12:29:35.782223+0000","last_unstale":"2024-09-21T12:30:14.258216+0000","last_undegraded":"2024-09-21T12:30:14.258216+0000","last_fullsized":"2024-09-21T12:30:14.258216+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:24:03.059549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:03.059549+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"266'13","reported_seq":241,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.252391+0000","last_change":"2024-09-21T12:29:35.786269+0000","last_active":"2024-09-21T12:30:14.252391+0000","last_peered":"2024-09-21T12:30:14.252391+0000","last_clean":"2024-09-21T12:30:14.252391+0000","last_became_active":"2024-09-21T12:29:35.769891+0000","last_became_peered":"2024-09-21T12:29:35.769891+0000","last_unstale":"2024-09-21T12:30:14.252391+0000","last_undegraded":"2024-09-21T12:30:14.252391+0000","last_fullsized":"2024-09-21T12:30:14.252391+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:26:10.846183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:10.846183+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2","version":"275'8801","reported_seq":15868,"reported_epoch":275,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T12:30:17.263689+0000","last_change":"2024-09-21T12:29:36.859459+0000","last_active":"2024-09-21T12:30:17.263689+0000","last_peered":"2024-09-21T12:30:17.263689+0000","last_clean":"2024-09-21T12:29:34.730529+0000","last_became_active":"2024-09-21T12:29:36.844864+0000","last_became_peered":"2024-09-21T12:29:36.844864+0000","last_unstale":"2024-09-21T12:30:17.263689+0000","last_undegraded":"2024-09-21T12:30:17.263689+0000","last_fullsized":"2024-09-21T12:30:17.263689+0000","mapping_epoch":266,"log_start":"257'4800","ondisk_log_start":"257'4800","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":4001,"log_dups_size":0,"ondisk_log_size":4001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435617792,"num_objects":6647,"num_object_clones":0,"num_object_copies":19941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3435,"num_objects_unfound":0,"num_objects_dirty":6647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8801,"num_write_kb":494336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3487,"num_bytes_recovered":228524032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,3],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":6647}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"76'79","reported_seq":180,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.251567+0000","last_change":"2024-09-21T12:28:29.446170+0000","last_active":"2024-09-21T12:30:14.251567+0000","last_peered":"2024-09-21T12:30:14.251567+0000","last_clean":"2024-09-21T12:30:14.251567+0000","last_became_active":"2024-09-21T12:27:14.943286+0000","last_became_peered":"2024-09-21T12:27:14.943286+0000","last_unstale":"2024-09-21T12:30:14.251567+0000","last_undegraded":"2024-09-21T12:30:14.251567+0000","last_fullsized":"2024-09-21T12:30:14.251567+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:29.446103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:29.446103+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":321,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.252181+0000","last_change":"2024-09-21T12:27:31.491370+0000","last_active":"2024-09-21T12:30:14.252181+0000","last_peered":"2024-09-21T12:30:14.252181+0000","last_clean":"2024-09-21T12:30:14.252181+0000","last_became_active":"2024-09-21T12:23:46.910112+0000","last_became_peered":"2024-09-21T12:23:46.910112+0000","last_unstale":"2024-09-21T12:30:14.252181+0000","last_undegraded":"2024-09-21T12:30:14.252181+0000","last_fullsized":"2024-09-21T12:30:14.252181+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:27:31.490985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:31.490985+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":31,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.264763+0000","last_change":"2024-09-21T12:29:35.772883+0000","last_active":"2024-09-21T12:30:14.264763+0000","last_peered":"2024-09-21T12:30:14.264763+0000","last_clean":"2024-09-21T12:30:14.264763+0000","last_became_active":"2024-09-21T12:29:35.772590+0000","last_became_peered":"2024-09-21T12:29:35.772590+0000","last_unstale":"2024-09-21T12:30:14.264763+0000","last_undegraded":"2024-09-21T12:30:14.264763+0000","last_fullsized":"2024-09-21T12:30:14.264763+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:05.580847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:05.580847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"210'14","reported_seq":188,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.258087+0000","last_change":"2024-09-21T12:26:54.091833+0000","last_active":"2024-09-21T12:30:14.258087+0000","last_peered":"2024-09-21T12:30:14.258087+0000","last_clean":"2024-09-21T12:30:14.258087+0000","last_became_active":"2024-09-21T12:26:54.091451+0000","last_became_peered":"2024-09-21T12:26:54.091451+0000","last_unstale":"2024-09-21T12:30:14.258087+0000","last_undegraded":"2024-09-21T12:30:14.258087+0000","last_fullsized":"2024-09-21T12:30:14.258087+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'13","last_scrub_stamp":"2024-09-21T12:25:33.474391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:33.474391+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":47,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.259459+0000","last_change":"2024-09-21T12:29:35.802015+0000","last_active":"2024-09-21T12:30:14.259459+0000","last_peered":"2024-09-21T12:30:14.259459+0000","last_clean":"2024-09-21T12:30:14.259459+0000","last_became_active":"2024-09-21T12:29:35.776594+0000","last_became_peered":"2024-09-21T12:29:35.776594+0000","last_unstale":"2024-09-21T12:30:14.259459+0000","last_undegraded":"2024-09-21T12:30:14.259459+0000","last_fullsized":"2024-09-21T12:30:14.259459+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:28:31.580245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:31.580245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"275'8720","reported_seq":23122,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:16.706829+0000","last_change":"2024-09-21T12:29:56.127804+0000","last_active":"2024-09-21T12:30:16.706829+0000","last_peered":"2024-09-21T12:30:16.706829+0000","last_clean":"2024-09-21T12:30:16.706829+0000","last_became_active":"2024-09-21T12:29:56.122000+0000","last_became_peered":"2024-09-21T12:29:56.122000+0000","last_unstale":"2024-09-21T12:30:16.706829+0000","last_undegraded":"2024-09-21T12:30:16.706829+0000","last_fullsized":"2024-09-21T12:30:16.706829+0000","mapping_epoch":271,"log_start":"257'4700","ondisk_log_start":"257'4700","created":236,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":4020,"log_dups_size":0,"ondisk_log_size":4020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440205336,"num_objects":6718,"num_object_clones":0,"num_object_copies":20154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6718,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8720,"num_write_kb":493953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7146,"num_bytes_recovered":468320256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"76'79","reported_seq":386,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.251492+0000","last_change":"2024-09-21T12:27:20.527090+0000","last_active":"2024-09-21T12:30:14.251492+0000","last_peered":"2024-09-21T12:30:14.251492+0000","last_clean":"2024-09-21T12:30:14.251492+0000","last_became_active":"2024-09-21T12:27:14.941905+0000","last_became_peered":"2024-09-21T12:27:14.941905+0000","last_unstale":"2024-09-21T12:30:14.251492+0000","last_undegraded":"2024-09-21T12:30:14.251492+0000","last_fullsized":"2024-09-21T12:30:14.251492+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:20.526951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:20.526951+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":522,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.257209+0000","last_change":"2024-09-21T12:29:35.809258+0000","last_active":"2024-09-21T12:30:14.257209+0000","last_peered":"2024-09-21T12:30:14.257209+0000","last_clean":"2024-09-21T12:30:14.257209+0000","last_became_active":"2024-09-21T12:29:35.772160+0000","last_became_peered":"2024-09-21T12:29:35.772160+0000","last_unstale":"2024-09-21T12:30:14.257209+0000","last_undegraded":"2024-09-21T12:30:14.257209+0000","last_fullsized":"2024-09-21T12:30:14.257209+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-21T12:28:26.632272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:26.632272+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":461,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.269166+0000","last_change":"2024-09-21T12:29:10.089053+0000","last_active":"2024-09-21T12:30:14.269166+0000","last_peered":"2024-09-21T12:30:14.269166+0000","last_clean":"2024-09-21T12:30:14.269166+0000","last_became_active":"2024-09-21T12:20:15.297340+0000","last_became_peered":"2024-09-21T12:20:15.297340+0000","last_unstale":"2024-09-21T12:30:14.269166+0000","last_undegraded":"2024-09-21T12:30:14.269166+0000","last_fullsized":"2024-09-21T12:30:14.269166+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:10.088986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:10.088986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:35.634033+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":411,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.265598+0000","last_change":"2024-09-21T12:29:35.784165+0000","last_active":"2024-09-21T12:30:14.265598+0000","last_peered":"2024-09-21T12:30:14.265598+0000","last_clean":"2024-09-21T12:30:14.265598+0000","last_became_active":"2024-09-21T12:29:35.783649+0000","last_became_peered":"2024-09-21T12:29:35.783649+0000","last_unstale":"2024-09-21T12:30:14.265598+0000","last_undegraded":"2024-09-21T12:30:14.265598+0000","last_fullsized":"2024-09-21T12:30:14.265598+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:33.697208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:33.697208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":403,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.260087+0000","last_change":"2024-09-21T12:29:05.143941+0000","last_active":"2024-09-21T12:30:14.260087+0000","last_peered":"2024-09-21T12:30:14.260087+0000","last_clean":"2024-09-21T12:30:14.260087+0000","last_became_active":"2024-09-21T12:22:50.918940+0000","last_became_peered":"2024-09-21T12:22:50.918940+0000","last_unstale":"2024-09-21T12:30:14.260087+0000","last_undegraded":"2024-09-21T12:30:14.260087+0000","last_fullsized":"2024-09-21T12:30:14.260087+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:05.143736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:29:05.143736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:27.414365+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"275'8768","reported_seq":8856,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:17.488215+0000","last_change":"2024-09-21T12:28:20.956665+0000","last_active":"2024-09-21T12:30:17.488215+0000","last_peered":"2024-09-21T12:30:17.488215+0000","last_clean":"2024-09-21T12:30:17.488215+0000","last_became_active":"2024-09-21T12:28:20.955557+0000","last_became_peered":"2024-09-21T12:28:20.955557+0000","last_unstale":"2024-09-21T12:30:17.488215+0000","last_undegraded":"2024-09-21T12:30:17.488215+0000","last_fullsized":"2024-09-21T12:30:17.488215+0000","mapping_epoch":236,"log_start":"257'4800","ondisk_log_start":"257'4800","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3968,"log_dups_size":0,"ondisk_log_size":3968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433455104,"num_objects":6614,"num_object_clones":0,"num_object_copies":19842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8768,"num_write_kb":492224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"76'79","reported_seq":388,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.256789+0000","last_change":"2024-09-21T12:26:45.354836+0000","last_active":"2024-09-21T12:30:14.256789+0000","last_peered":"2024-09-21T12:30:14.256789+0000","last_clean":"2024-09-21T12:30:14.256789+0000","last_became_active":"2024-09-21T12:26:44.960093+0000","last_became_peered":"2024-09-21T12:26:44.960093+0000","last_unstale":"2024-09-21T12:30:14.256789+0000","last_undegraded":"2024-09-21T12:30:14.256789+0000","last_fullsized":"2024-09-21T12:30:14.256789+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:45.354774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:45.354774+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:35.018407+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":460,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.257828+0000","last_change":"2024-09-21T12:26:51.463759+0000","last_active":"2024-09-21T12:30:14.257828+0000","last_peered":"2024-09-21T12:30:14.257828+0000","last_clean":"2024-09-21T12:30:14.257828+0000","last_became_active":"2024-09-21T12:22:50.926008+0000","last_became_peered":"2024-09-21T12:22:50.926008+0000","last_unstale":"2024-09-21T12:30:14.257828+0000","last_undegraded":"2024-09-21T12:30:14.257828+0000","last_fullsized":"2024-09-21T12:30:14.257828+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-21T12:26:51.463698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:51.463698+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:27.121901+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":517,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.258397+0000","last_change":"2024-09-21T12:29:35.806072+0000","last_active":"2024-09-21T12:30:14.258397+0000","last_peered":"2024-09-21T12:30:14.258397+0000","last_clean":"2024-09-21T12:30:14.258397+0000","last_became_active":"2024-09-21T12:29:35.782812+0000","last_became_peered":"2024-09-21T12:29:35.782812+0000","last_unstale":"2024-09-21T12:30:14.258397+0000","last_undegraded":"2024-09-21T12:30:14.258397+0000","last_fullsized":"2024-09-21T12:30:14.258397+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:28:32.727727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:32.727727+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"121'11","reported_seq":615,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.257064+0000","last_change":"2024-09-21T12:26:14.490045+0000","last_active":"2024-09-21T12:30:14.257064+0000","last_peered":"2024-09-21T12:30:14.257064+0000","last_clean":"2024-09-21T12:30:14.257064+0000","last_became_active":"2024-09-21T12:22:50.930342+0000","last_became_peered":"2024-09-21T12:22:50.930342+0000","last_unstale":"2024-09-21T12:30:14.257064+0000","last_undegraded":"2024-09-21T12:30:14.257064+0000","last_fullsized":"2024-09-21T12:30:14.257064+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:26:14.489972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:14.489972+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":450,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.256371+0000","last_change":"2024-09-21T12:28:37.686403+0000","last_active":"2024-09-21T12:30:14.256371+0000","last_peered":"2024-09-21T12:30:14.256371+0000","last_clean":"2024-09-21T12:30:14.256371+0000","last_became_active":"2024-09-21T12:20:21.593367+0000","last_became_peered":"2024-09-21T12:20:21.593367+0000","last_unstale":"2024-09-21T12:30:14.256371+0000","last_undegraded":"2024-09-21T12:30:14.256371+0000","last_fullsized":"2024-09-21T12:30:14.256371+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:37.686350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:37.686350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"275'8816","reported_seq":8988,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:16.691480+0000","last_change":"2024-09-21T12:29:54.985804+0000","last_active":"2024-09-21T12:30:16.691480+0000","last_peered":"2024-09-21T12:30:16.691480+0000","last_clean":"2024-09-21T12:30:16.691480+0000","last_became_active":"2024-09-21T12:28:20.956619+0000","last_became_peered":"2024-09-21T12:28:20.956619+0000","last_unstale":"2024-09-21T12:30:16.691480+0000","last_undegraded":"2024-09-21T12:30:16.691480+0000","last_fullsized":"2024-09-21T12:30:16.691480+0000","mapping_epoch":236,"log_start":"258'5000","ondisk_log_start":"258'5000","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'7729","last_scrub_stamp":"2024-09-21T12:29:54.985720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:29:54.985720+0000","objects_scrubbed":7675,"log_size":3816,"log_dups_size":0,"ondisk_log_size":3816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:31:00.198248+0000","scrub_duration":2694,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446824448,"num_objects":6818,"num_object_clones":0,"num_object_copies":20454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8816,"num_write_kb":500288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"76'79","reported_seq":191,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.256941+0000","last_change":"2024-09-21T12:29:35.768692+0000","last_active":"2024-09-21T12:30:14.256941+0000","last_peered":"2024-09-21T12:30:14.256941+0000","last_clean":"2024-09-21T12:30:14.256941+0000","last_became_active":"2024-09-21T12:29:35.768234+0000","last_became_peered":"2024-09-21T12:29:35.768234+0000","last_unstale":"2024-09-21T12:30:14.256941+0000","last_undegraded":"2024-09-21T12:30:14.256941+0000","last_fullsized":"2024-09-21T12:30:14.256941+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:06.154040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:06.154040+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13417,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":5,"num_write":2,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":13417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":231,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.255404+0000","last_change":"2024-09-21T12:29:35.795971+0000","last_active":"2024-09-21T12:30:14.255404+0000","last_peered":"2024-09-21T12:30:14.255404+0000","last_clean":"2024-09-21T12:30:14.255404+0000","last_became_active":"2024-09-21T12:29:35.765303+0000","last_became_peered":"2024-09-21T12:29:35.765303+0000","last_unstale":"2024-09-21T12:30:14.255404+0000","last_undegraded":"2024-09-21T12:30:14.255404+0000","last_fullsized":"2024-09-21T12:30:14.255404+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:27:27.325105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:27.325105+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":478,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.258546+0000","last_change":"2024-09-21T12:29:57.398083+0000","last_active":"2024-09-21T12:30:14.258546+0000","last_peered":"2024-09-21T12:30:14.258546+0000","last_clean":"2024-09-21T12:30:14.258546+0000","last_became_active":"2024-09-21T12:20:15.308518+0000","last_became_peered":"2024-09-21T12:20:15.308518+0000","last_unstale":"2024-09-21T12:30:14.258546+0000","last_undegraded":"2024-09-21T12:30:14.258546+0000","last_fullsized":"2024-09-21T12:30:14.258546+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:57.398039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:57.398039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:31:00.876322+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":596,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.262766+0000","last_change":"2024-09-21T12:27:58.871800+0000","last_active":"2024-09-21T12:30:14.262766+0000","last_peered":"2024-09-21T12:30:14.262766+0000","last_clean":"2024-09-21T12:30:14.262766+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:30:14.262766+0000","last_undegraded":"2024-09-21T12:30:14.262766+0000","last_fullsized":"2024-09-21T12:30:14.262766+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:27:58.871710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:27:58.871710+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":460,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.258668+0000","last_change":"2024-09-21T12:28:37.850166+0000","last_active":"2024-09-21T12:30:14.258668+0000","last_peered":"2024-09-21T12:30:14.258668+0000","last_clean":"2024-09-21T12:30:14.258668+0000","last_became_active":"2024-09-21T12:20:21.315801+0000","last_became_peered":"2024-09-21T12:20:21.315801+0000","last_unstale":"2024-09-21T12:30:14.258668+0000","last_undegraded":"2024-09-21T12:30:14.258668+0000","last_fullsized":"2024-09-21T12:30:14.258668+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:37.850100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:37.850100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"275'8959","reported_seq":9047,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:16.660700+0000","last_change":"2024-09-21T12:28:20.958462+0000","last_active":"2024-09-21T12:30:16.660700+0000","last_peered":"2024-09-21T12:30:16.660700+0000","last_clean":"2024-09-21T12:30:16.660700+0000","last_became_active":"2024-09-21T12:28:20.958143+0000","last_became_peered":"2024-09-21T12:28:20.958143+0000","last_unstale":"2024-09-21T12:30:16.660700+0000","last_undegraded":"2024-09-21T12:30:16.660700+0000","last_fullsized":"2024-09-21T12:30:16.660700+0000","mapping_epoch":236,"log_start":"257'4900","ondisk_log_start":"257'4900","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":4059,"log_dups_size":0,"ondisk_log_size":4059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":441122816,"num_objects":6731,"num_object_clones":0,"num_object_copies":20193,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8959,"num_write_kb":502080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"76'79","reported_seq":146,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.257789+0000","last_change":"2024-09-21T12:29:08.081975+0000","last_active":"2024-09-21T12:30:14.257789+0000","last_peered":"2024-09-21T12:30:14.257789+0000","last_clean":"2024-09-21T12:30:14.257789+0000","last_became_active":"2024-09-21T12:29:05.835884+0000","last_became_peered":"2024-09-21T12:29:05.835884+0000","last_unstale":"2024-09-21T12:30:14.257789+0000","last_undegraded":"2024-09-21T12:30:14.257789+0000","last_fullsized":"2024-09-21T12:30:14.257789+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:08.081901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:08.081901+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:20.926140+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":496,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.262572+0000","last_change":"2024-09-21T12:26:14.834328+0000","last_active":"2024-09-21T12:30:14.262572+0000","last_peered":"2024-09-21T12:30:14.262572+0000","last_clean":"2024-09-21T12:30:14.262572+0000","last_became_active":"2024-09-21T12:20:17.693067+0000","last_became_peered":"2024-09-21T12:20:17.693067+0000","last_unstale":"2024-09-21T12:30:14.262572+0000","last_undegraded":"2024-09-21T12:30:14.262572+0000","last_fullsized":"2024-09-21T12:30:14.262572+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-21T12:26:14.834236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:14.834236+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:18.108327+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":454,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.266423+0000","last_change":"2024-09-21T12:26:12.131784+0000","last_active":"2024-09-21T12:30:14.266423+0000","last_peered":"2024-09-21T12:30:14.266423+0000","last_clean":"2024-09-21T12:30:14.266423+0000","last_became_active":"2024-09-21T12:20:15.308441+0000","last_became_peered":"2024-09-21T12:20:15.308441+0000","last_unstale":"2024-09-21T12:30:14.266423+0000","last_undegraded":"2024-09-21T12:30:14.266423+0000","last_fullsized":"2024-09-21T12:30:14.266423+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:12.131721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:12.131721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":50,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.256368+0000","last_change":"2024-09-21T12:29:05.826167+0000","last_active":"2024-09-21T12:30:14.256368+0000","last_peered":"2024-09-21T12:30:14.256368+0000","last_clean":"2024-09-21T12:30:14.256368+0000","last_became_active":"2024-09-21T12:29:05.825869+0000","last_became_peered":"2024-09-21T12:29:05.825869+0000","last_unstale":"2024-09-21T12:30:14.256368+0000","last_undegraded":"2024-09-21T12:30:14.256368+0000","last_fullsized":"2024-09-21T12:30:14.256368+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:48.999878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:48.999878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":442,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.265132+0000","last_change":"2024-09-21T12:25:44.918356+0000","last_active":"2024-09-21T12:30:14.265132+0000","last_peered":"2024-09-21T12:30:14.265132+0000","last_clean":"2024-09-21T12:30:14.265132+0000","last_became_active":"2024-09-21T12:20:21.592828+0000","last_became_peered":"2024-09-21T12:20:21.592828+0000","last_unstale":"2024-09-21T12:30:14.265132+0000","last_undegraded":"2024-09-21T12:30:14.265132+0000","last_fullsized":"2024-09-21T12:30:14.265132+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:44.918276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:44.918276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:27.631123+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"275'8777","reported_seq":8865,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:17.615064+0000","last_change":"2024-09-21T12:28:20.952252+0000","last_active":"2024-09-21T12:30:17.615064+0000","last_peered":"2024-09-21T12:30:17.615064+0000","last_clean":"2024-09-21T12:30:17.615064+0000","last_became_active":"2024-09-21T12:28:20.951980+0000","last_became_peered":"2024-09-21T12:28:20.951980+0000","last_unstale":"2024-09-21T12:30:17.615064+0000","last_undegraded":"2024-09-21T12:30:17.615064+0000","last_fullsized":"2024-09-21T12:30:17.615064+0000","mapping_epoch":236,"log_start":"258'5068","ondisk_log_start":"258'5068","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3709,"log_dups_size":0,"ondisk_log_size":3709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432996352,"num_objects":6607,"num_object_clones":0,"num_object_copies":19821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8777,"num_write_kb":492288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"76'79","reported_seq":254,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.256350+0000","last_change":"2024-09-21T12:28:25.714131+0000","last_active":"2024-09-21T12:30:14.256350+0000","last_peered":"2024-09-21T12:30:14.256350+0000","last_clean":"2024-09-21T12:30:14.256350+0000","last_became_active":"2024-09-21T12:26:54.092726+0000","last_became_peered":"2024-09-21T12:26:54.092726+0000","last_unstale":"2024-09-21T12:30:14.256350+0000","last_undegraded":"2024-09-21T12:30:14.256350+0000","last_fullsized":"2024-09-21T12:30:14.256350+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:25.714084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:25.714084+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":619,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.266312+0000","last_change":"2024-09-21T12:26:09.082395+0000","last_active":"2024-09-21T12:30:14.266312+0000","last_peered":"2024-09-21T12:30:14.266312+0000","last_clean":"2024-09-21T12:30:14.266312+0000","last_became_active":"2024-09-21T12:20:17.307769+0000","last_became_peered":"2024-09-21T12:20:17.307769+0000","last_unstale":"2024-09-21T12:30:14.266312+0000","last_undegraded":"2024-09-21T12:30:14.266312+0000","last_fullsized":"2024-09-21T12:30:14.266312+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-21T12:26:09.082314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:09.082314+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":396,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.262357+0000","last_change":"2024-09-21T12:28:31.738463+0000","last_active":"2024-09-21T12:30:14.262357+0000","last_peered":"2024-09-21T12:30:14.262357+0000","last_clean":"2024-09-21T12:30:14.262357+0000","last_became_active":"2024-09-21T12:22:50.922002+0000","last_became_peered":"2024-09-21T12:22:50.922002+0000","last_unstale":"2024-09-21T12:30:14.262357+0000","last_undegraded":"2024-09-21T12:30:14.262357+0000","last_fullsized":"2024-09-21T12:30:14.262357+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:31.738415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:31.738415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":180,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.264753+0000","last_change":"2024-09-21T12:28:37.691938+0000","last_active":"2024-09-21T12:30:14.264753+0000","last_peered":"2024-09-21T12:30:14.264753+0000","last_clean":"2024-09-21T12:30:14.264753+0000","last_became_active":"2024-09-21T12:26:54.083584+0000","last_became_peered":"2024-09-21T12:26:54.083584+0000","last_unstale":"2024-09-21T12:30:14.264753+0000","last_undegraded":"2024-09-21T12:30:14.264753+0000","last_fullsized":"2024-09-21T12:30:14.264753+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T12:28:37.691868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:28:37.691868+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":485,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.259616+0000","last_change":"2024-09-21T12:27:21.956252+0000","last_active":"2024-09-21T12:30:14.259616+0000","last_peered":"2024-09-21T12:30:14.259616+0000","last_clean":"2024-09-21T12:30:14.259616+0000","last_became_active":"2024-09-21T12:22:50.921488+0000","last_became_peered":"2024-09-21T12:22:50.921488+0000","last_unstale":"2024-09-21T12:30:14.259616+0000","last_undegraded":"2024-09-21T12:30:14.259616+0000","last_fullsized":"2024-09-21T12:30:14.259616+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:21.956185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:27:21.956185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":470,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.257256+0000","last_change":"2024-09-21T12:26:10.435410+0000","last_active":"2024-09-21T12:30:14.257256+0000","last_peered":"2024-09-21T12:30:14.257256+0000","last_clean":"2024-09-21T12:30:14.257256+0000","last_became_active":"2024-09-21T12:22:50.917558+0000","last_became_peered":"2024-09-21T12:22:50.917558+0000","last_unstale":"2024-09-21T12:30:14.257256+0000","last_undegraded":"2024-09-21T12:30:14.257256+0000","last_fullsized":"2024-09-21T12:30:14.257256+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:10.435349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:10.435349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"275'8877","reported_seq":14306,"reported_epoch":275,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T12:30:17.251856+0000","last_change":"2024-09-21T12:29:36.874226+0000","last_active":"2024-09-21T12:30:17.251856+0000","last_peered":"2024-09-21T12:30:17.251856+0000","last_clean":"2024-09-21T12:29:34.745065+0000","last_became_active":"2024-09-21T12:29:36.855579+0000","last_became_peered":"2024-09-21T12:29:36.855579+0000","last_unstale":"2024-09-21T12:30:17.251856+0000","last_undegraded":"2024-09-21T12:30:17.251856+0000","last_fullsized":"2024-09-21T12:30:17.251856+0000","mapping_epoch":266,"log_start":"258'4800","ondisk_log_start":"258'4800","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":4077,"log_dups_size":0,"ondisk_log_size":4077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437190656,"num_objects":6671,"num_object_clones":0,"num_object_copies":20013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4223,"num_objects_unfound":0,"num_objects_dirty":6671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8877,"num_write_kb":497536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2668,"num_bytes_recovered":174850048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,5],"avail_no_missing":["1","5","7"],"object_location_counts":[{"shards":"1,5,7","objects":6671}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"76'79","reported_seq":120,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.256588+0000","last_change":"2024-09-21T12:29:35.771874+0000","last_active":"2024-09-21T12:30:14.256588+0000","last_peered":"2024-09-21T12:30:14.256588+0000","last_clean":"2024-09-21T12:30:14.256588+0000","last_became_active":"2024-09-21T12:29:35.768315+0000","last_became_peered":"2024-09-21T12:29:35.768315+0000","last_unstale":"2024-09-21T12:30:14.256588+0000","last_undegraded":"2024-09-21T12:30:14.256588+0000","last_fullsized":"2024-09-21T12:30:14.256588+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:46.188649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:46.188649+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":90,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.257659+0000","last_change":"2024-09-21T12:29:35.794153+0000","last_active":"2024-09-21T12:30:14.257659+0000","last_peered":"2024-09-21T12:30:14.257659+0000","last_clean":"2024-09-21T12:30:14.257659+0000","last_became_active":"2024-09-21T12:29:35.768686+0000","last_became_peered":"2024-09-21T12:29:35.768686+0000","last_unstale":"2024-09-21T12:30:14.257659+0000","last_undegraded":"2024-09-21T12:30:14.257659+0000","last_fullsized":"2024-09-21T12:30:14.257659+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-21T12:28:25.549320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:25.549320+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":478,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.264648+0000","last_change":"2024-09-21T12:28:39.705480+0000","last_active":"2024-09-21T12:30:14.264648+0000","last_peered":"2024-09-21T12:30:14.264648+0000","last_clean":"2024-09-21T12:30:14.264648+0000","last_became_active":"2024-09-21T12:20:15.308356+0000","last_became_peered":"2024-09-21T12:20:15.308356+0000","last_unstale":"2024-09-21T12:30:14.264648+0000","last_undegraded":"2024-09-21T12:30:14.264648+0000","last_fullsized":"2024-09-21T12:30:14.264648+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:39.705175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:39.705175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":488,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.251491+0000","last_change":"2024-09-21T12:28:43.390160+0000","last_active":"2024-09-21T12:30:14.251491+0000","last_peered":"2024-09-21T12:30:14.251491+0000","last_clean":"2024-09-21T12:30:14.251491+0000","last_became_active":"2024-09-21T12:20:14.288358+0000","last_became_peered":"2024-09-21T12:20:14.288358+0000","last_unstale":"2024-09-21T12:30:14.251491+0000","last_undegraded":"2024-09-21T12:30:14.251491+0000","last_fullsized":"2024-09-21T12:30:14.251491+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:28:43.390087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:28:43.390087+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0019042449999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":471,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.252019+0000","last_change":"2024-09-21T12:29:13.200503+0000","last_active":"2024-09-21T12:30:14.252019+0000","last_peered":"2024-09-21T12:30:14.252019+0000","last_clean":"2024-09-21T12:30:14.252019+0000","last_became_active":"2024-09-21T12:20:19.309724+0000","last_became_peered":"2024-09-21T12:20:19.309724+0000","last_unstale":"2024-09-21T12:30:14.252019+0000","last_undegraded":"2024-09-21T12:30:14.252019+0000","last_fullsized":"2024-09-21T12:30:14.252019+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:13.200321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:29:13.200321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:21.207169+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":442,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.257180+0000","last_change":"2024-09-21T12:27:33.395611+0000","last_active":"2024-09-21T12:30:14.257180+0000","last_peered":"2024-09-21T12:30:14.257180+0000","last_clean":"2024-09-21T12:30:14.257180+0000","last_became_active":"2024-09-21T12:20:21.592982+0000","last_became_peered":"2024-09-21T12:20:21.592982+0000","last_unstale":"2024-09-21T12:30:14.257180+0000","last_undegraded":"2024-09-21T12:30:14.257180+0000","last_fullsized":"2024-09-21T12:30:14.257180+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:33.395568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:27:33.395568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"275'8963","reported_seq":9051,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:17.603967+0000","last_change":"2024-09-21T12:28:21.114897+0000","last_active":"2024-09-21T12:30:17.603967+0000","last_peered":"2024-09-21T12:30:17.603967+0000","last_clean":"2024-09-21T12:30:17.603967+0000","last_became_active":"2024-09-21T12:28:21.114609+0000","last_became_peered":"2024-09-21T12:28:21.114609+0000","last_unstale":"2024-09-21T12:30:17.603967+0000","last_undegraded":"2024-09-21T12:30:17.603967+0000","last_fullsized":"2024-09-21T12:30:17.603967+0000","mapping_epoch":236,"log_start":"258'5311","ondisk_log_start":"258'5311","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3652,"log_dups_size":0,"ondisk_log_size":3652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":443482112,"num_objects":6767,"num_object_clones":0,"num_object_copies":20301,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8963,"num_write_kb":503360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"76'79","reported_seq":412,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.264971+0000","last_change":"2024-09-21T12:29:11.678268+0000","last_active":"2024-09-21T12:30:14.264971+0000","last_peered":"2024-09-21T12:30:14.264971+0000","last_clean":"2024-09-21T12:30:14.264971+0000","last_became_active":"2024-09-21T12:26:44.949867+0000","last_became_peered":"2024-09-21T12:26:44.949867+0000","last_unstale":"2024-09-21T12:30:14.264971+0000","last_undegraded":"2024-09-21T12:30:14.264971+0000","last_fullsized":"2024-09-21T12:30:14.264971+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:11.677923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:11.677923+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:27.641123+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":486,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.258217+0000","last_change":"2024-09-21T12:28:45.783033+0000","last_active":"2024-09-21T12:30:14.258217+0000","last_peered":"2024-09-21T12:30:14.258217+0000","last_clean":"2024-09-21T12:30:14.258217+0000","last_became_active":"2024-09-21T12:20:17.299325+0000","last_became_peered":"2024-09-21T12:20:17.299325+0000","last_unstale":"2024-09-21T12:30:14.258217+0000","last_undegraded":"2024-09-21T12:30:14.258217+0000","last_fullsized":"2024-09-21T12:30:14.258217+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:28:45.782955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.782955+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":500,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.255755+0000","last_change":"2024-09-21T12:29:35.764002+0000","last_active":"2024-09-21T12:30:14.255755+0000","last_peered":"2024-09-21T12:30:14.255755+0000","last_clean":"2024-09-21T12:30:14.255755+0000","last_became_active":"2024-09-21T12:29:35.763587+0000","last_became_peered":"2024-09-21T12:29:35.763587+0000","last_unstale":"2024-09-21T12:30:14.255755+0000","last_undegraded":"2024-09-21T12:30:14.255755+0000","last_fullsized":"2024-09-21T12:30:14.255755+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:15.115449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:15.115449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:30.532202+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":1077,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:15.804178+0000","last_change":"2024-09-21T12:28:34.639438+0000","last_active":"2024-09-21T12:30:15.804178+0000","last_peered":"2024-09-21T12:30:15.804178+0000","last_clean":"2024-09-21T12:30:15.804178+0000","last_became_active":"2024-09-21T12:20:14.287543+0000","last_became_peered":"2024-09-21T12:20:14.287543+0000","last_unstale":"2024-09-21T12:30:15.804178+0000","last_undegraded":"2024-09-21T12:30:15.804178+0000","last_fullsized":"2024-09-21T12:30:15.804178+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-21T12:28:34.639361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:28:34.639361+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":595,"num_read_kb":590,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":32,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.268009+0000","last_change":"2024-09-21T12:29:35.767793+0000","last_active":"2024-09-21T12:30:14.268009+0000","last_peered":"2024-09-21T12:30:14.268009+0000","last_clean":"2024-09-21T12:30:14.268009+0000","last_became_active":"2024-09-21T12:29:35.767237+0000","last_became_peered":"2024-09-21T12:29:35.767237+0000","last_unstale":"2024-09-21T12:30:14.268009+0000","last_undegraded":"2024-09-21T12:30:14.268009+0000","last_fullsized":"2024-09-21T12:30:14.268009+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:11.422293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:11.422293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":39,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.258196+0000","last_change":"2024-09-21T12:29:36.145777+0000","last_active":"2024-09-21T12:30:14.258196+0000","last_peered":"2024-09-21T12:30:14.258196+0000","last_clean":"2024-09-21T12:30:14.258196+0000","last_became_active":"2024-09-21T12:29:35.780184+0000","last_became_peered":"2024-09-21T12:29:35.780184+0000","last_unstale":"2024-09-21T12:30:14.258196+0000","last_undegraded":"2024-09-21T12:30:14.258196+0000","last_fullsized":"2024-09-21T12:30:14.258196+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:36.145708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:29:36.145708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:39.520222+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"275'8836","reported_seq":17242,"reported_epoch":275,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T12:30:17.614366+0000","last_change":"2024-09-21T12:29:36.915959+0000","last_active":"2024-09-21T12:30:17.614366+0000","last_peered":"2024-09-21T12:30:17.614366+0000","last_clean":"2024-09-21T12:29:34.727203+0000","last_became_active":"2024-09-21T12:29:36.901917+0000","last_became_peered":"2024-09-21T12:29:36.901917+0000","last_unstale":"2024-09-21T12:30:17.614366+0000","last_undegraded":"2024-09-21T12:30:17.614366+0000","last_fullsized":"2024-09-21T12:30:17.614366+0000","mapping_epoch":266,"log_start":"259'5200","ondisk_log_start":"259'5200","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3636,"log_dups_size":0,"ondisk_log_size":3636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432668672,"num_objects":6602,"num_object_clones":0,"num_object_copies":19806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8198,"num_objects_unfound":0,"num_objects_dirty":6602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8836,"num_write_kb":494016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2771,"num_bytes_recovered":181600256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":6602}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"257'20","reported_seq":117,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.257843+0000","last_change":"2024-09-21T12:29:09.625844+0000","last_active":"2024-09-21T12:30:14.257843+0000","last_peered":"2024-09-21T12:30:14.257843+0000","last_clean":"2024-09-21T12:30:14.257843+0000","last_became_active":"2024-09-21T12:29:05.827560+0000","last_became_peered":"2024-09-21T12:29:05.827560+0000","last_unstale":"2024-09-21T12:30:14.257843+0000","last_undegraded":"2024-09-21T12:30:14.257843+0000","last_fullsized":"2024-09-21T12:30:14.257843+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'20","last_scrub_stamp":"2024-09-21T12:29:09.625763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:29:09.625763+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:36.533631+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":467,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.260290+0000","last_change":"2024-09-21T12:26:12.394327+0000","last_active":"2024-09-21T12:30:14.260290+0000","last_peered":"2024-09-21T12:30:14.260290+0000","last_clean":"2024-09-21T12:30:14.260290+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:30:14.260290+0000","last_undegraded":"2024-09-21T12:30:14.260290+0000","last_fullsized":"2024-09-21T12:30:14.260290+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:26:12.394247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:12.394247+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":547,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.256361+0000","last_change":"2024-09-21T12:26:49.248015+0000","last_active":"2024-09-21T12:30:14.256361+0000","last_peered":"2024-09-21T12:30:14.256361+0000","last_clean":"2024-09-21T12:30:14.256361+0000","last_became_active":"2024-09-21T12:26:44.950132+0000","last_became_peered":"2024-09-21T12:26:44.950132+0000","last_unstale":"2024-09-21T12:30:14.256361+0000","last_undegraded":"2024-09-21T12:30:14.256361+0000","last_fullsized":"2024-09-21T12:30:14.256361+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:49.247952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:49.247952+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":32,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.251673+0000","last_change":"2024-09-21T12:29:35.765255+0000","last_active":"2024-09-21T12:30:14.251673+0000","last_peered":"2024-09-21T12:30:14.251673+0000","last_clean":"2024-09-21T12:30:14.251673+0000","last_became_active":"2024-09-21T12:29:35.764934+0000","last_became_peered":"2024-09-21T12:29:35.764934+0000","last_unstale":"2024-09-21T12:30:14.251673+0000","last_undegraded":"2024-09-21T12:30:14.251673+0000","last_fullsized":"2024-09-21T12:30:14.251673+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:36.684459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:36.684459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":303,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.268711+0000","last_change":"2024-09-21T12:29:35.779845+0000","last_active":"2024-09-21T12:30:14.268711+0000","last_peered":"2024-09-21T12:30:14.268711+0000","last_clean":"2024-09-21T12:30:14.268711+0000","last_became_active":"2024-09-21T12:29:35.779133+0000","last_became_peered":"2024-09-21T12:29:35.779133+0000","last_unstale":"2024-09-21T12:30:14.268711+0000","last_undegraded":"2024-09-21T12:30:14.268711+0000","last_fullsized":"2024-09-21T12:30:14.268711+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:28.647287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:28.647287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3","version":"275'8723","reported_seq":8809,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:16.671681+0000","last_change":"2024-09-21T12:28:20.958439+0000","last_active":"2024-09-21T12:30:16.671681+0000","last_peered":"2024-09-21T12:30:16.671681+0000","last_clean":"2024-09-21T12:30:16.671681+0000","last_became_active":"2024-09-21T12:28:20.958025+0000","last_became_peered":"2024-09-21T12:28:20.958025+0000","last_unstale":"2024-09-21T12:30:16.671681+0000","last_undegraded":"2024-09-21T12:30:16.671681+0000","last_fullsized":"2024-09-21T12:30:16.671681+0000","mapping_epoch":236,"log_start":"257'4700","ondisk_log_start":"257'4700","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":4023,"log_dups_size":0,"ondisk_log_size":4023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437846016,"num_objects":6681,"num_object_clones":0,"num_object_copies":20043,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8723,"num_write_kb":492928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"76'79","reported_seq":408,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.265285+0000","last_change":"2024-09-21T12:27:14.954671+0000","last_active":"2024-09-21T12:30:14.265285+0000","last_peered":"2024-09-21T12:30:14.265285+0000","last_clean":"2024-09-21T12:30:14.265285+0000","last_became_active":"2024-09-21T12:27:14.954351+0000","last_became_peered":"2024-09-21T12:27:14.954351+0000","last_unstale":"2024-09-21T12:30:14.265285+0000","last_undegraded":"2024-09-21T12:30:14.265285+0000","last_fullsized":"2024-09-21T12:30:14.265285+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:48.188322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:48.188322+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:30:24.569371+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":150,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.257054+0000","last_change":"2024-09-21T12:29:05.826584+0000","last_active":"2024-09-21T12:30:14.257054+0000","last_peered":"2024-09-21T12:30:14.257054+0000","last_clean":"2024-09-21T12:30:14.257054+0000","last_became_active":"2024-09-21T12:29:05.826075+0000","last_became_peered":"2024-09-21T12:29:05.826075+0000","last_unstale":"2024-09-21T12:30:14.257054+0000","last_undegraded":"2024-09-21T12:30:14.257054+0000","last_fullsized":"2024-09-21T12:30:14.257054+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:27:57.114076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:57.114076+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":386,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.251338+0000","last_change":"2024-09-21T12:27:30.501480+0000","last_active":"2024-09-21T12:30:14.251338+0000","last_peered":"2024-09-21T12:30:14.251338+0000","last_clean":"2024-09-21T12:30:14.251338+0000","last_became_active":"2024-09-21T12:22:50.908060+0000","last_became_peered":"2024-09-21T12:22:50.908060+0000","last_unstale":"2024-09-21T12:30:14.251338+0000","last_undegraded":"2024-09-21T12:30:14.251338+0000","last_fullsized":"2024-09-21T12:30:14.251338+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:30.498831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:27:30.498831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":206,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.252268+0000","last_change":"2024-09-21T12:29:15.275262+0000","last_active":"2024-09-21T12:30:14.252268+0000","last_peered":"2024-09-21T12:30:14.252268+0000","last_clean":"2024-09-21T12:30:14.252268+0000","last_became_active":"2024-09-21T12:29:05.825426+0000","last_became_peered":"2024-09-21T12:29:05.825426+0000","last_unstale":"2024-09-21T12:30:14.252268+0000","last_undegraded":"2024-09-21T12:30:14.252268+0000","last_fullsized":"2024-09-21T12:30:14.252268+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:15.275189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:29:15.275189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:19.963943+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":449,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.256885+0000","last_change":"2024-09-21T12:28:39.853931+0000","last_active":"2024-09-21T12:30:14.256885+0000","last_peered":"2024-09-21T12:30:14.256885+0000","last_clean":"2024-09-21T12:30:14.256885+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:30:14.256885+0000","last_undegraded":"2024-09-21T12:30:14.256885+0000","last_fullsized":"2024-09-21T12:30:14.256885+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:39.853791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:39.853791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"275'8745","reported_seq":8844,"reported_epoch":275,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-21T12:30:16.965298+0000","last_change":"2024-09-21T12:30:06.869001+0000","last_active":"2024-09-21T12:30:16.965298+0000","last_peered":"2024-09-21T12:30:16.965298+0000","last_clean":"2024-09-21T12:29:34.704985+0000","last_became_active":"2024-09-21T12:29:36.848982+0000","last_became_peered":"2024-09-21T12:29:36.848982+0000","last_unstale":"2024-09-21T12:30:16.965298+0000","last_undegraded":"2024-09-21T12:30:16.965298+0000","last_fullsized":"2024-09-21T12:30:16.965298+0000","mapping_epoch":266,"log_start":"257'4600","ondisk_log_start":"257'4600","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":4145,"log_dups_size":0,"ondisk_log_size":4145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435748864,"num_objects":6649,"num_object_clones":0,"num_object_copies":19947,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6649,"num_objects_unfound":0,"num_objects_dirty":6649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8745,"num_write_kb":492608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,1],"avail_no_missing":["2","1","6"],"object_location_counts":[{"shards":"1,2,6","objects":6649}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"76'79","reported_seq":171,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.251078+0000","last_change":"2024-09-21T12:29:35.778553+0000","last_active":"2024-09-21T12:30:14.251078+0000","last_peered":"2024-09-21T12:30:14.251078+0000","last_clean":"2024-09-21T12:30:14.251078+0000","last_became_active":"2024-09-21T12:29:35.777794+0000","last_became_peered":"2024-09-21T12:29:35.777794+0000","last_unstale":"2024-09-21T12:30:14.251078+0000","last_undegraded":"2024-09-21T12:30:14.251078+0000","last_fullsized":"2024-09-21T12:30:14.251078+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:57.680181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:57.680181+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131344,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":519,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.251681+0000","last_change":"2024-09-21T12:28:46.355910+0000","last_active":"2024-09-21T12:30:14.251681+0000","last_peered":"2024-09-21T12:30:14.251681+0000","last_clean":"2024-09-21T12:30:14.251681+0000","last_became_active":"2024-09-21T12:23:46.897197+0000","last_became_peered":"2024-09-21T12:23:46.897197+0000","last_unstale":"2024-09-21T12:30:14.251681+0000","last_undegraded":"2024-09-21T12:30:14.251681+0000","last_fullsized":"2024-09-21T12:30:14.251681+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-21T12:28:46.355822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:46.355822+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":188,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.259764+0000","last_change":"2024-09-21T12:29:35.784391+0000","last_active":"2024-09-21T12:30:14.259764+0000","last_peered":"2024-09-21T12:30:14.259764+0000","last_clean":"2024-09-21T12:30:14.259764+0000","last_became_active":"2024-09-21T12:29:35.783874+0000","last_became_peered":"2024-09-21T12:29:35.783874+0000","last_unstale":"2024-09-21T12:30:14.259764+0000","last_undegraded":"2024-09-21T12:30:14.259764+0000","last_fullsized":"2024-09-21T12:30:14.259764+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:06.236749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:06.236749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":475,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.266000+0000","last_change":"2024-09-21T12:28:03.571042+0000","last_active":"2024-09-21T12:30:14.266000+0000","last_peered":"2024-09-21T12:30:14.266000+0000","last_clean":"2024-09-21T12:30:14.266000+0000","last_became_active":"2024-09-21T12:22:50.927438+0000","last_became_peered":"2024-09-21T12:22:50.927438+0000","last_unstale":"2024-09-21T12:30:14.266000+0000","last_undegraded":"2024-09-21T12:30:14.266000+0000","last_fullsized":"2024-09-21T12:30:14.266000+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:03.570984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:03.570984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:29.578920+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":464,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.252440+0000","last_change":"2024-09-21T12:25:39.079146+0000","last_active":"2024-09-21T12:30:14.252440+0000","last_peered":"2024-09-21T12:30:14.252440+0000","last_clean":"2024-09-21T12:30:14.252440+0000","last_became_active":"2024-09-21T12:22:50.913898+0000","last_became_peered":"2024-09-21T12:22:50.913898+0000","last_unstale":"2024-09-21T12:30:14.252440+0000","last_undegraded":"2024-09-21T12:30:14.252440+0000","last_fullsized":"2024-09-21T12:30:14.252440+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:39.079015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:39.079015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:26.100425+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"275'8889","reported_seq":17497,"reported_epoch":275,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T12:30:17.509136+0000","last_change":"2024-09-21T12:29:36.877320+0000","last_active":"2024-09-21T12:30:17.509136+0000","last_peered":"2024-09-21T12:30:17.509136+0000","last_clean":"2024-09-21T12:29:34.747646+0000","last_became_active":"2024-09-21T12:29:36.851101+0000","last_became_peered":"2024-09-21T12:29:36.851101+0000","last_unstale":"2024-09-21T12:30:17.509136+0000","last_undegraded":"2024-09-21T12:30:17.509136+0000","last_fullsized":"2024-09-21T12:30:17.509136+0000","mapping_epoch":266,"log_start":"258'4900","ondisk_log_start":"258'4900","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3989,"log_dups_size":0,"ondisk_log_size":3989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439156736,"num_objects":6701,"num_object_clones":0,"num_object_copies":20103,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2795,"num_objects_unfound":0,"num_objects_dirty":6701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8889,"num_write_kb":498880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4257,"num_bytes_recovered":278986752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,7],"avail_no_missing":["3","6","7"],"object_location_counts":[{"shards":"3,6,7","objects":6701}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"76'79","reported_seq":544,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.256372+0000","last_change":"2024-09-21T12:29:05.501052+0000","last_active":"2024-09-21T12:30:14.256372+0000","last_peered":"2024-09-21T12:30:14.256372+0000","last_clean":"2024-09-21T12:30:14.256372+0000","last_became_active":"2024-09-21T12:27:14.943705+0000","last_became_peered":"2024-09-21T12:27:14.943705+0000","last_unstale":"2024-09-21T12:30:14.256372+0000","last_undegraded":"2024-09-21T12:30:14.256372+0000","last_fullsized":"2024-09-21T12:30:14.256372+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:05.500994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:05.500994+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:19.934083+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":333,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.251385+0000","last_change":"2024-09-21T12:29:14.222764+0000","last_active":"2024-09-21T12:30:14.251385+0000","last_peered":"2024-09-21T12:30:14.251385+0000","last_clean":"2024-09-21T12:30:14.251385+0000","last_became_active":"2024-09-21T12:23:46.902260+0000","last_became_peered":"2024-09-21T12:23:46.902260+0000","last_unstale":"2024-09-21T12:30:14.251385+0000","last_undegraded":"2024-09-21T12:30:14.251385+0000","last_fullsized":"2024-09-21T12:30:14.251385+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:29:14.222681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:29:14.222681+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:38.308175+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":482,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.259104+0000","last_change":"2024-09-21T12:27:32.433938+0000","last_active":"2024-09-21T12:30:14.259104+0000","last_peered":"2024-09-21T12:30:14.259104+0000","last_clean":"2024-09-21T12:30:14.259104+0000","last_became_active":"2024-09-21T12:22:50.927946+0000","last_became_peered":"2024-09-21T12:22:50.927946+0000","last_unstale":"2024-09-21T12:30:14.259104+0000","last_undegraded":"2024-09-21T12:30:14.259104+0000","last_fullsized":"2024-09-21T12:30:14.259104+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:32.433867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:27:32.433867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":600,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.251631+0000","last_change":"2024-09-21T12:28:39.418167+0000","last_active":"2024-09-21T12:30:14.251631+0000","last_peered":"2024-09-21T12:30:14.251631+0000","last_clean":"2024-09-21T12:30:14.251631+0000","last_became_active":"2024-09-21T12:20:19.307924+0000","last_became_peered":"2024-09-21T12:20:19.307924+0000","last_unstale":"2024-09-21T12:30:14.251631+0000","last_undegraded":"2024-09-21T12:30:14.251631+0000","last_fullsized":"2024-09-21T12:30:14.251631+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:28:39.418085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:39.418085+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":89,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.251660+0000","last_change":"2024-09-21T12:29:35.771598+0000","last_active":"2024-09-21T12:30:14.251660+0000","last_peered":"2024-09-21T12:30:14.251660+0000","last_clean":"2024-09-21T12:30:14.251660+0000","last_became_active":"2024-09-21T12:29:35.771280+0000","last_became_peered":"2024-09-21T12:29:35.771280+0000","last_unstale":"2024-09-21T12:30:14.251660+0000","last_undegraded":"2024-09-21T12:30:14.251660+0000","last_fullsized":"2024-09-21T12:30:14.251660+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:35.669942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:35.669942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"275'8877","reported_seq":8966,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:16.957350+0000","last_change":"2024-09-21T12:28:20.952349+0000","last_active":"2024-09-21T12:30:16.957350+0000","last_peered":"2024-09-21T12:30:16.957350+0000","last_clean":"2024-09-21T12:30:16.957350+0000","last_became_active":"2024-09-21T12:28:20.952083+0000","last_became_peered":"2024-09-21T12:28:20.952083+0000","last_unstale":"2024-09-21T12:30:16.957350+0000","last_undegraded":"2024-09-21T12:30:16.957350+0000","last_fullsized":"2024-09-21T12:30:16.957350+0000","mapping_epoch":236,"log_start":"257'4737","ondisk_log_start":"257'4737","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":4140,"log_dups_size":0,"ondisk_log_size":4140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435355648,"num_objects":6643,"num_object_clones":0,"num_object_copies":19929,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8877,"num_write_kb":496640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"76'79","reported_seq":175,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.256606+0000","last_change":"2024-09-21T12:27:21.345414+0000","last_active":"2024-09-21T12:30:14.256606+0000","last_peered":"2024-09-21T12:30:14.256606+0000","last_clean":"2024-09-21T12:30:14.256606+0000","last_became_active":"2024-09-21T12:27:14.943330+0000","last_became_peered":"2024-09-21T12:27:14.943330+0000","last_unstale":"2024-09-21T12:30:14.256606+0000","last_undegraded":"2024-09-21T12:30:14.256606+0000","last_fullsized":"2024-09-21T12:30:14.256606+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:21.345364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:21.345364+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":535,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.262652+0000","last_change":"2024-09-21T12:25:00.237083+0000","last_active":"2024-09-21T12:30:14.262652+0000","last_peered":"2024-09-21T12:30:14.262652+0000","last_clean":"2024-09-21T12:30:14.262652+0000","last_became_active":"2024-09-21T12:22:50.922343+0000","last_became_peered":"2024-09-21T12:22:50.922343+0000","last_unstale":"2024-09-21T12:30:14.262652+0000","last_undegraded":"2024-09-21T12:30:14.262652+0000","last_fullsized":"2024-09-21T12:30:14.262652+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:25:00.236990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:00.236990+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":145,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.259615+0000","last_change":"2024-09-21T12:26:54.084563+0000","last_active":"2024-09-21T12:30:14.259615+0000","last_peered":"2024-09-21T12:30:14.259615+0000","last_clean":"2024-09-21T12:30:14.259615+0000","last_became_active":"2024-09-21T12:26:54.084228+0000","last_became_peered":"2024-09-21T12:26:54.084228+0000","last_unstale":"2024-09-21T12:30:14.259615+0000","last_undegraded":"2024-09-21T12:30:14.259615+0000","last_fullsized":"2024-09-21T12:30:14.259615+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:30.451876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:25:30.451876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:28.176176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":492,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.252859+0000","last_change":"2024-09-21T12:28:47.361181+0000","last_active":"2024-09-21T12:30:14.252859+0000","last_peered":"2024-09-21T12:30:14.252859+0000","last_clean":"2024-09-21T12:30:14.252859+0000","last_became_active":"2024-09-21T12:22:50.925974+0000","last_became_peered":"2024-09-21T12:22:50.925974+0000","last_unstale":"2024-09-21T12:30:14.252859+0000","last_undegraded":"2024-09-21T12:30:14.252859+0000","last_fullsized":"2024-09-21T12:30:14.252859+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:47.361115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:47.361115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":452,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.255839+0000","last_change":"2024-09-21T12:28:43.821102+0000","last_active":"2024-09-21T12:30:14.255839+0000","last_peered":"2024-09-21T12:30:14.255839+0000","last_clean":"2024-09-21T12:30:14.255839+0000","last_became_active":"2024-09-21T12:20:21.311493+0000","last_became_peered":"2024-09-21T12:20:21.311493+0000","last_unstale":"2024-09-21T12:30:14.255839+0000","last_undegraded":"2024-09-21T12:30:14.255839+0000","last_fullsized":"2024-09-21T12:30:14.255839+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:43.820979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:43.820979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"275'8853","reported_seq":8951,"reported_epoch":275,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-21T12:30:17.487562+0000","last_change":"2024-09-21T12:30:06.945306+0000","last_active":"2024-09-21T12:30:17.487562+0000","last_peered":"2024-09-21T12:30:17.487562+0000","last_clean":"2024-09-21T12:29:34.734937+0000","last_became_active":"2024-09-21T12:29:36.898598+0000","last_became_peered":"2024-09-21T12:29:36.898598+0000","last_unstale":"2024-09-21T12:30:17.487562+0000","last_undegraded":"2024-09-21T12:30:17.487562+0000","last_fullsized":"2024-09-21T12:30:17.487562+0000","mapping_epoch":266,"log_start":"258'4900","ondisk_log_start":"258'4900","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3953,"log_dups_size":0,"ondisk_log_size":3953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434044928,"num_objects":6623,"num_object_clones":0,"num_object_copies":19869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13246,"num_objects_unfound":0,"num_objects_dirty":6623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8853,"num_write_kb":495232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[3,5,7],"avail_no_missing":["3","5","7"],"object_location_counts":[{"shards":"3,5,7","objects":6623}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.f","version":"76'79","reported_seq":377,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.258703+0000","last_change":"2024-09-21T12:29:07.210453+0000","last_active":"2024-09-21T12:30:14.258703+0000","last_peered":"2024-09-21T12:30:14.258703+0000","last_clean":"2024-09-21T12:30:14.258703+0000","last_became_active":"2024-09-21T12:29:05.842070+0000","last_became_peered":"2024-09-21T12:29:05.842070+0000","last_unstale":"2024-09-21T12:30:14.258703+0000","last_undegraded":"2024-09-21T12:30:14.258703+0000","last_fullsized":"2024-09-21T12:30:14.258703+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:07.210381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:07.210381+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:31:10.350877+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":524,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.260563+0000","last_change":"2024-09-21T12:29:35.792626+0000","last_active":"2024-09-21T12:30:14.260563+0000","last_peered":"2024-09-21T12:30:14.260563+0000","last_clean":"2024-09-21T12:30:14.260563+0000","last_became_active":"2024-09-21T12:29:35.765582+0000","last_became_peered":"2024-09-21T12:29:35.765582+0000","last_unstale":"2024-09-21T12:30:14.260563+0000","last_undegraded":"2024-09-21T12:30:14.260563+0000","last_fullsized":"2024-09-21T12:30:14.260563+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-21T12:27:30.898706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:27:30.898706+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:31:06.220734+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":469,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.257892+0000","last_change":"2024-09-21T12:29:35.770561+0000","last_active":"2024-09-21T12:30:14.257892+0000","last_peered":"2024-09-21T12:30:14.257892+0000","last_clean":"2024-09-21T12:30:14.257892+0000","last_became_active":"2024-09-21T12:29:35.770095+0000","last_became_peered":"2024-09-21T12:29:35.770095+0000","last_unstale":"2024-09-21T12:30:14.257892+0000","last_undegraded":"2024-09-21T12:30:14.257892+0000","last_fullsized":"2024-09-21T12:30:14.257892+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:48.438755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:48.438755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":462,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.251860+0000","last_change":"2024-09-21T12:29:35.769084+0000","last_active":"2024-09-21T12:30:14.251860+0000","last_peered":"2024-09-21T12:30:14.251860+0000","last_clean":"2024-09-21T12:30:14.251860+0000","last_became_active":"2024-09-21T12:29:35.767996+0000","last_became_peered":"2024-09-21T12:29:35.767996+0000","last_unstale":"2024-09-21T12:30:14.251860+0000","last_undegraded":"2024-09-21T12:30:14.251860+0000","last_fullsized":"2024-09-21T12:30:14.251860+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:50.626780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:50.626780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":442,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.256282+0000","last_change":"2024-09-21T12:29:15.641929+0000","last_active":"2024-09-21T12:30:14.256282+0000","last_peered":"2024-09-21T12:30:14.256282+0000","last_clean":"2024-09-21T12:30:14.256282+0000","last_became_active":"2024-09-21T12:20:21.314545+0000","last_became_peered":"2024-09-21T12:20:21.314545+0000","last_unstale":"2024-09-21T12:30:14.256282+0000","last_undegraded":"2024-09-21T12:30:14.256282+0000","last_fullsized":"2024-09-21T12:30:14.256282+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:15.641881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:29:15.641881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:27.876807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"275'8890","reported_seq":8978,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:17.246665+0000","last_change":"2024-09-21T12:28:20.955882+0000","last_active":"2024-09-21T12:30:17.246665+0000","last_peered":"2024-09-21T12:30:17.246665+0000","last_clean":"2024-09-21T12:30:17.246665+0000","last_became_active":"2024-09-21T12:28:20.955605+0000","last_became_peered":"2024-09-21T12:28:20.955605+0000","last_unstale":"2024-09-21T12:30:17.246665+0000","last_undegraded":"2024-09-21T12:30:17.246665+0000","last_fullsized":"2024-09-21T12:30:17.246665+0000","mapping_epoch":236,"log_start":"257'4800","ondisk_log_start":"257'4800","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":4090,"log_dups_size":0,"ondisk_log_size":4090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433717248,"num_objects":6618,"num_object_clones":0,"num_object_copies":19854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8890,"num_write_kb":496256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"76'79","reported_seq":415,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.259699+0000","last_change":"2024-09-21T12:27:17.158443+0000","last_active":"2024-09-21T12:30:14.259699+0000","last_peered":"2024-09-21T12:30:14.259699+0000","last_clean":"2024-09-21T12:30:14.259699+0000","last_became_active":"2024-09-21T12:27:14.951375+0000","last_became_peered":"2024-09-21T12:27:14.951375+0000","last_unstale":"2024-09-21T12:30:14.259699+0000","last_undegraded":"2024-09-21T12:30:14.259699+0000","last_fullsized":"2024-09-21T12:30:14.259699+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:17.158380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:17.158380+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":487,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.265322+0000","last_change":"2024-09-21T12:29:35.797253+0000","last_active":"2024-09-21T12:30:14.265322+0000","last_peered":"2024-09-21T12:30:14.265322+0000","last_clean":"2024-09-21T12:30:14.265322+0000","last_became_active":"2024-09-21T12:29:35.765273+0000","last_became_peered":"2024-09-21T12:29:35.765273+0000","last_unstale":"2024-09-21T12:30:14.265322+0000","last_undegraded":"2024-09-21T12:30:14.265322+0000","last_fullsized":"2024-09-21T12:30:14.265322+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:28:40.694138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:40.694138+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":491,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.257436+0000","last_change":"2024-09-21T12:28:30.612213+0000","last_active":"2024-09-21T12:30:14.257436+0000","last_peered":"2024-09-21T12:30:14.257436+0000","last_clean":"2024-09-21T12:30:14.257436+0000","last_became_active":"2024-09-21T12:22:50.927859+0000","last_became_peered":"2024-09-21T12:22:50.927859+0000","last_unstale":"2024-09-21T12:30:14.257436+0000","last_undegraded":"2024-09-21T12:30:14.257436+0000","last_fullsized":"2024-09-21T12:30:14.257436+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:30.612156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:30.612156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":471,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.251302+0000","last_change":"2024-09-21T12:29:11.218769+0000","last_active":"2024-09-21T12:30:14.251302+0000","last_peered":"2024-09-21T12:30:14.251302+0000","last_clean":"2024-09-21T12:30:14.251302+0000","last_became_active":"2024-09-21T12:20:19.312572+0000","last_became_peered":"2024-09-21T12:20:19.312572+0000","last_unstale":"2024-09-21T12:30:14.251302+0000","last_undegraded":"2024-09-21T12:30:14.251302+0000","last_fullsized":"2024-09-21T12:30:14.251302+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:11.218334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:29:11.218334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:35.978478+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":456,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.258794+0000","last_change":"2024-09-21T12:29:35.777977+0000","last_active":"2024-09-21T12:30:14.258794+0000","last_peered":"2024-09-21T12:30:14.258794+0000","last_clean":"2024-09-21T12:30:14.258794+0000","last_became_active":"2024-09-21T12:29:35.776977+0000","last_became_peered":"2024-09-21T12:29:35.776977+0000","last_unstale":"2024-09-21T12:30:14.258794+0000","last_undegraded":"2024-09-21T12:30:14.258794+0000","last_fullsized":"2024-09-21T12:30:14.258794+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:58.321890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:58.321890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"275'8840","reported_seq":15750,"reported_epoch":275,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T12:30:17.623005+0000","last_change":"2024-09-21T12:29:36.894379+0000","last_active":"2024-09-21T12:30:17.623005+0000","last_peered":"2024-09-21T12:30:17.623005+0000","last_clean":"2024-09-21T12:29:34.748613+0000","last_became_active":"2024-09-21T12:29:36.865103+0000","last_became_peered":"2024-09-21T12:29:36.865103+0000","last_unstale":"2024-09-21T12:30:17.623005+0000","last_undegraded":"2024-09-21T12:30:17.623005+0000","last_fullsized":"2024-09-21T12:30:17.623005+0000","mapping_epoch":266,"log_start":"258'5175","ondisk_log_start":"258'5175","created":236,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:28:19.930988+0000","last_clean_scrub_stamp":"2024-09-21T12:28:19.930988+0000","objects_scrubbed":0,"log_size":3665,"log_dups_size":0,"ondisk_log_size":3665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435159040,"num_objects":6640,"num_object_clones":0,"num_object_copies":19920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3524,"num_objects_unfound":0,"num_objects_dirty":6640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8840,"num_write_kb":495360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3408,"num_bytes_recovered":223346688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,7],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":6640}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"76'79","reported_seq":41,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.266046+0000","last_change":"2024-09-21T12:29:36.701016+0000","last_active":"2024-09-21T12:30:14.266046+0000","last_peered":"2024-09-21T12:30:14.266046+0000","last_clean":"2024-09-21T12:30:14.266046+0000","last_became_active":"2024-09-21T12:29:35.767716+0000","last_became_peered":"2024-09-21T12:29:35.767716+0000","last_unstale":"2024-09-21T12:30:14.266046+0000","last_undegraded":"2024-09-21T12:30:14.266046+0000","last_fullsized":"2024-09-21T12:30:14.266046+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:36.700968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:36.700968+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:31:00.892429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":107,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.261860+0000","last_change":"2024-09-21T12:29:58.184503+0000","last_active":"2024-09-21T12:30:14.261860+0000","last_peered":"2024-09-21T12:30:14.261860+0000","last_clean":"2024-09-21T12:30:14.261860+0000","last_became_active":"2024-09-21T12:29:35.765891+0000","last_became_peered":"2024-09-21T12:29:35.765891+0000","last_unstale":"2024-09-21T12:30:14.261860+0000","last_undegraded":"2024-09-21T12:30:14.261860+0000","last_fullsized":"2024-09-21T12:30:14.261860+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-21T12:29:58.184426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:29:58.184426+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:31:18.574790+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":526,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.256977+0000","last_change":"2024-09-21T12:29:35.785869+0000","last_active":"2024-09-21T12:30:14.256977+0000","last_peered":"2024-09-21T12:30:14.256977+0000","last_clean":"2024-09-21T12:30:14.256977+0000","last_became_active":"2024-09-21T12:29:35.769415+0000","last_became_peered":"2024-09-21T12:29:35.769415+0000","last_unstale":"2024-09-21T12:30:14.256977+0000","last_undegraded":"2024-09-21T12:30:14.256977+0000","last_fullsized":"2024-09-21T12:30:14.256977+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-21T12:24:29.054772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:29.054772+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"266'12","reported_seq":619,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.256300+0000","last_change":"2024-09-21T12:29:35.788485+0000","last_active":"2024-09-21T12:30:14.256300+0000","last_peered":"2024-09-21T12:30:14.256300+0000","last_clean":"2024-09-21T12:30:14.256300+0000","last_became_active":"2024-09-21T12:29:35.770167+0000","last_became_peered":"2024-09-21T12:29:35.770167+0000","last_unstale":"2024-09-21T12:30:14.256300+0000","last_undegraded":"2024-09-21T12:30:14.256300+0000","last_fullsized":"2024-09-21T12:30:14.256300+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:25:02.916090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:02.916090+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":153,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.260492+0000","last_change":"2024-09-21T12:26:56.862895+0000","last_active":"2024-09-21T12:30:14.260492+0000","last_peered":"2024-09-21T12:30:14.260492+0000","last_clean":"2024-09-21T12:30:14.260492+0000","last_became_active":"2024-09-21T12:26:54.089155+0000","last_became_peered":"2024-09-21T12:26:54.089155+0000","last_unstale":"2024-09-21T12:30:14.260492+0000","last_undegraded":"2024-09-21T12:30:14.260492+0000","last_fullsized":"2024-09-21T12:30:14.260492+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:56.862823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:56.862823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:17.786563+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"76'79","reported_seq":184,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.257944+0000","last_change":"2024-09-21T12:29:35.778736+0000","last_active":"2024-09-21T12:30:14.257944+0000","last_peered":"2024-09-21T12:30:14.257944+0000","last_clean":"2024-09-21T12:30:14.257944+0000","last_became_active":"2024-09-21T12:29:35.778366+0000","last_became_peered":"2024-09-21T12:29:35.778366+0000","last_unstale":"2024-09-21T12:30:14.257944+0000","last_undegraded":"2024-09-21T12:30:14.257944+0000","last_fullsized":"2024-09-21T12:30:14.257944+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:12.141269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:12.141269+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:40.502382+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":493,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.262022+0000","last_change":"2024-09-21T12:29:35.776307+0000","last_active":"2024-09-21T12:30:14.262022+0000","last_peered":"2024-09-21T12:30:14.262022+0000","last_clean":"2024-09-21T12:30:14.262022+0000","last_became_active":"2024-09-21T12:29:35.775852+0000","last_became_peered":"2024-09-21T12:29:35.775852+0000","last_unstale":"2024-09-21T12:30:14.262022+0000","last_undegraded":"2024-09-21T12:30:14.262022+0000","last_fullsized":"2024-09-21T12:30:14.262022+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:29.901043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:27:29.901043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:31:04.258876+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":71,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.268154+0000","last_change":"2024-09-21T12:29:35.800413+0000","last_active":"2024-09-21T12:30:14.268154+0000","last_peered":"2024-09-21T12:30:14.268154+0000","last_clean":"2024-09-21T12:30:14.268154+0000","last_became_active":"2024-09-21T12:29:35.773526+0000","last_became_peered":"2024-09-21T12:29:35.773526+0000","last_unstale":"2024-09-21T12:30:14.268154+0000","last_undegraded":"2024-09-21T12:30:14.268154+0000","last_fullsized":"2024-09-21T12:30:14.268154+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-21T12:26:57.369648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:57.369648+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":146,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.257178+0000","last_change":"2024-09-21T12:26:54.084503+0000","last_active":"2024-09-21T12:30:14.257178+0000","last_peered":"2024-09-21T12:30:14.257178+0000","last_clean":"2024-09-21T12:30:14.257178+0000","last_became_active":"2024-09-21T12:26:54.084109+0000","last_became_peered":"2024-09-21T12:26:54.084109+0000","last_unstale":"2024-09-21T12:30:14.257178+0000","last_undegraded":"2024-09-21T12:30:14.257178+0000","last_fullsized":"2024-09-21T12:30:14.257178+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:02.414298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:02.414298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:21.193891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":463,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.256684+0000","last_change":"2024-09-21T12:24:00.066522+0000","last_active":"2024-09-21T12:30:14.256684+0000","last_peered":"2024-09-21T12:30:14.256684+0000","last_clean":"2024-09-21T12:30:14.256684+0000","last_became_active":"2024-09-21T12:22:50.904404+0000","last_became_peered":"2024-09-21T12:22:50.904404+0000","last_unstale":"2024-09-21T12:30:14.256684+0000","last_undegraded":"2024-09-21T12:30:14.256684+0000","last_fullsized":"2024-09-21T12:30:14.256684+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:00.066451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:00.066451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'79","reported_seq":417,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.265081+0000","last_change":"2024-09-21T12:29:35.767301+0000","last_active":"2024-09-21T12:30:14.265081+0000","last_peered":"2024-09-21T12:30:14.265081+0000","last_clean":"2024-09-21T12:30:14.265081+0000","last_became_active":"2024-09-21T12:29:35.766922+0000","last_became_peered":"2024-09-21T12:29:35.766922+0000","last_unstale":"2024-09-21T12:30:14.265081+0000","last_undegraded":"2024-09-21T12:30:14.265081+0000","last_fullsized":"2024-09-21T12:30:14.265081+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:29.546657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:29.546657+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":501,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.258309+0000","last_change":"2024-09-21T12:29:35.797003+0000","last_active":"2024-09-21T12:30:14.258309+0000","last_peered":"2024-09-21T12:30:14.258309+0000","last_clean":"2024-09-21T12:30:14.258309+0000","last_became_active":"2024-09-21T12:29:35.774676+0000","last_became_peered":"2024-09-21T12:29:35.774676+0000","last_unstale":"2024-09-21T12:30:14.258309+0000","last_undegraded":"2024-09-21T12:30:14.258309+0000","last_fullsized":"2024-09-21T12:30:14.258309+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:28:01.881164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:01.881164+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":499,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.256488+0000","last_change":"2024-09-21T12:28:43.545259+0000","last_active":"2024-09-21T12:30:14.256488+0000","last_peered":"2024-09-21T12:30:14.256488+0000","last_clean":"2024-09-21T12:30:14.256488+0000","last_became_active":"2024-09-21T12:22:50.927649+0000","last_became_peered":"2024-09-21T12:22:50.927649+0000","last_unstale":"2024-09-21T12:30:14.256488+0000","last_undegraded":"2024-09-21T12:30:14.256488+0000","last_fullsized":"2024-09-21T12:30:14.256488+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:43.545197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:43.545197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":453,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.268896+0000","last_change":"2024-09-21T12:28:47.964092+0000","last_active":"2024-09-21T12:30:14.268896+0000","last_peered":"2024-09-21T12:30:14.268896+0000","last_clean":"2024-09-21T12:30:14.268896+0000","last_became_active":"2024-09-21T12:20:19.311947+0000","last_became_peered":"2024-09-21T12:20:19.311947+0000","last_unstale":"2024-09-21T12:30:14.268896+0000","last_undegraded":"2024-09-21T12:30:14.268896+0000","last_fullsized":"2024-09-21T12:30:14.268896+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:47.963747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:47.963747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":488,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.251534+0000","last_change":"2024-09-21T12:26:11.105716+0000","last_active":"2024-09-21T12:30:14.251534+0000","last_peered":"2024-09-21T12:30:14.251534+0000","last_clean":"2024-09-21T12:30:14.251534+0000","last_became_active":"2024-09-21T12:22:50.924931+0000","last_became_peered":"2024-09-21T12:22:50.924931+0000","last_unstale":"2024-09-21T12:30:14.251534+0000","last_undegraded":"2024-09-21T12:30:14.251534+0000","last_fullsized":"2024-09-21T12:30:14.251534+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:11.105583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:11.105583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"76'79","reported_seq":538,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.256024+0000","last_change":"2024-09-21T12:26:54.085274+0000","last_active":"2024-09-21T12:30:14.256024+0000","last_peered":"2024-09-21T12:30:14.256024+0000","last_clean":"2024-09-21T12:30:14.256024+0000","last_became_active":"2024-09-21T12:26:54.084939+0000","last_became_peered":"2024-09-21T12:26:54.084939+0000","last_unstale":"2024-09-21T12:30:14.256024+0000","last_undegraded":"2024-09-21T12:30:14.256024+0000","last_fullsized":"2024-09-21T12:30:14.256024+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:50.443344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:50.443344+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:30:19.773272+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":206,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.257686+0000","last_change":"2024-09-21T12:29:35.783028+0000","last_active":"2024-09-21T12:30:14.257686+0000","last_peered":"2024-09-21T12:30:14.257686+0000","last_clean":"2024-09-21T12:30:14.257686+0000","last_became_active":"2024-09-21T12:29:35.763537+0000","last_became_peered":"2024-09-21T12:29:35.763537+0000","last_unstale":"2024-09-21T12:30:14.257686+0000","last_undegraded":"2024-09-21T12:30:14.257686+0000","last_fullsized":"2024-09-21T12:30:14.257686+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:28:42.511874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:42.511874+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":479,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.256525+0000","last_change":"2024-09-21T12:29:35.770103+0000","last_active":"2024-09-21T12:30:14.256525+0000","last_peered":"2024-09-21T12:30:14.256525+0000","last_clean":"2024-09-21T12:30:14.256525+0000","last_became_active":"2024-09-21T12:29:35.769652+0000","last_became_peered":"2024-09-21T12:29:35.769652+0000","last_unstale":"2024-09-21T12:30:14.256525+0000","last_undegraded":"2024-09-21T12:30:14.256525+0000","last_fullsized":"2024-09-21T12:30:14.256525+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:07.875304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:07.875304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"266'9","reported_seq":69,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.257674+0000","last_change":"2024-09-21T12:29:35.798385+0000","last_active":"2024-09-21T12:30:14.257674+0000","last_peered":"2024-09-21T12:30:14.257674+0000","last_clean":"2024-09-21T12:30:14.257674+0000","last_became_active":"2024-09-21T12:29:35.768270+0000","last_became_peered":"2024-09-21T12:29:35.768270+0000","last_unstale":"2024-09-21T12:30:14.257674+0000","last_undegraded":"2024-09-21T12:30:14.257674+0000","last_fullsized":"2024-09-21T12:30:14.257674+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:28:43.853712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:43.853712+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":394,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.252526+0000","last_change":"2024-09-21T12:28:05.323501+0000","last_active":"2024-09-21T12:30:14.252526+0000","last_peered":"2024-09-21T12:30:14.252526+0000","last_clean":"2024-09-21T12:30:14.252526+0000","last_became_active":"2024-09-21T12:22:50.924770+0000","last_became_peered":"2024-09-21T12:22:50.924770+0000","last_unstale":"2024-09-21T12:30:14.252526+0000","last_undegraded":"2024-09-21T12:30:14.252526+0000","last_fullsized":"2024-09-21T12:30:14.252526+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:05.323448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:05.323448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"76'79","reported_seq":259,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.256012+0000","last_change":"2024-09-21T12:29:35.774892+0000","last_active":"2024-09-21T12:30:14.256012+0000","last_peered":"2024-09-21T12:30:14.256012+0000","last_clean":"2024-09-21T12:30:14.256012+0000","last_became_active":"2024-09-21T12:29:35.773989+0000","last_became_peered":"2024-09-21T12:29:35.773989+0000","last_unstale":"2024-09-21T12:30:14.256012+0000","last_undegraded":"2024-09-21T12:30:14.256012+0000","last_fullsized":"2024-09-21T12:30:14.256012+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:48.804235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:48.804235+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":491,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.259436+0000","last_change":"2024-09-21T12:29:35.805229+0000","last_active":"2024-09-21T12:30:14.259436+0000","last_peered":"2024-09-21T12:30:14.259436+0000","last_clean":"2024-09-21T12:30:14.259436+0000","last_became_active":"2024-09-21T12:29:35.780460+0000","last_became_peered":"2024-09-21T12:29:35.780460+0000","last_unstale":"2024-09-21T12:30:14.259436+0000","last_undegraded":"2024-09-21T12:30:14.259436+0000","last_fullsized":"2024-09-21T12:30:14.259436+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:24:50.727101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:50.727101+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":491,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.256579+0000","last_change":"2024-09-21T12:28:32.630320+0000","last_active":"2024-09-21T12:30:14.256579+0000","last_peered":"2024-09-21T12:30:14.256579+0000","last_clean":"2024-09-21T12:30:14.256579+0000","last_became_active":"2024-09-21T12:22:50.908238+0000","last_became_peered":"2024-09-21T12:22:50.908238+0000","last_unstale":"2024-09-21T12:30:14.256579+0000","last_undegraded":"2024-09-21T12:30:14.256579+0000","last_fullsized":"2024-09-21T12:30:14.256579+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:32.630261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:32.630261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":583,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.258965+0000","last_change":"2024-09-21T12:28:36.293443+0000","last_active":"2024-09-21T12:30:14.258965+0000","last_peered":"2024-09-21T12:30:14.258965+0000","last_clean":"2024-09-21T12:30:14.258965+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:30:14.258965+0000","last_undegraded":"2024-09-21T12:30:14.258965+0000","last_fullsized":"2024-09-21T12:30:14.258965+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:28:36.293377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:36.293377+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:31:04.270506+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":451,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.257875+0000","last_change":"2024-09-21T12:29:35.771966+0000","last_active":"2024-09-21T12:30:14.257875+0000","last_peered":"2024-09-21T12:30:14.257875+0000","last_clean":"2024-09-21T12:30:14.257875+0000","last_became_active":"2024-09-21T12:29:35.768276+0000","last_became_peered":"2024-09-21T12:29:35.768276+0000","last_unstale":"2024-09-21T12:30:14.257875+0000","last_undegraded":"2024-09-21T12:30:14.257875+0000","last_fullsized":"2024-09-21T12:30:14.257875+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:08.923458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:08.923458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"76'79","reported_seq":131,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.267719+0000","last_change":"2024-09-21T12:27:18.292942+0000","last_active":"2024-09-21T12:30:14.267719+0000","last_peered":"2024-09-21T12:30:14.267719+0000","last_clean":"2024-09-21T12:30:14.267719+0000","last_became_active":"2024-09-21T12:27:16.940347+0000","last_became_peered":"2024-09-21T12:27:16.940347+0000","last_unstale":"2024-09-21T12:30:14.267719+0000","last_undegraded":"2024-09-21T12:30:14.267719+0000","last_fullsized":"2024-09-21T12:30:14.267719+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:18.292862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:18.292862+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":457,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.260863+0000","last_change":"2024-09-21T12:28:40.150322+0000","last_active":"2024-09-21T12:30:14.260863+0000","last_peered":"2024-09-21T12:30:14.260863+0000","last_clean":"2024-09-21T12:30:14.260863+0000","last_became_active":"2024-09-21T12:22:50.906473+0000","last_became_peered":"2024-09-21T12:22:50.906473+0000","last_unstale":"2024-09-21T12:30:14.260863+0000","last_undegraded":"2024-09-21T12:30:14.260863+0000","last_fullsized":"2024-09-21T12:30:14.260863+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:28:40.150250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:40.150250+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:31:09.150710+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":378,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.257559+0000","last_change":"2024-09-21T12:24:58.948698+0000","last_active":"2024-09-21T12:30:14.257559+0000","last_peered":"2024-09-21T12:30:14.257559+0000","last_clean":"2024-09-21T12:30:14.257559+0000","last_became_active":"2024-09-21T12:22:50.903705+0000","last_became_peered":"2024-09-21T12:22:50.903705+0000","last_unstale":"2024-09-21T12:30:14.257559+0000","last_undegraded":"2024-09-21T12:30:14.257559+0000","last_fullsized":"2024-09-21T12:30:14.257559+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:58.948625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:58.948625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:23.707170+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":456,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.265256+0000","last_change":"2024-09-21T12:29:35.769972+0000","last_active":"2024-09-21T12:30:14.265256+0000","last_peered":"2024-09-21T12:30:14.265256+0000","last_clean":"2024-09-21T12:30:14.265256+0000","last_became_active":"2024-09-21T12:29:35.769510+0000","last_became_peered":"2024-09-21T12:29:35.769510+0000","last_unstale":"2024-09-21T12:30:14.265256+0000","last_undegraded":"2024-09-21T12:30:14.265256+0000","last_fullsized":"2024-09-21T12:30:14.265256+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:16.149550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:16.149550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":31,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.256112+0000","last_change":"2024-09-21T12:29:35.774512+0000","last_active":"2024-09-21T12:30:14.256112+0000","last_peered":"2024-09-21T12:30:14.256112+0000","last_clean":"2024-09-21T12:30:14.256112+0000","last_became_active":"2024-09-21T12:29:35.768083+0000","last_became_peered":"2024-09-21T12:29:35.768083+0000","last_unstale":"2024-09-21T12:30:14.256112+0000","last_undegraded":"2024-09-21T12:30:14.256112+0000","last_fullsized":"2024-09-21T12:30:14.256112+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:56.879558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:56.879558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.17","version":"76'79","reported_seq":148,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.258802+0000","last_change":"2024-09-21T12:29:55.291563+0000","last_active":"2024-09-21T12:30:14.258802+0000","last_peered":"2024-09-21T12:30:14.258802+0000","last_clean":"2024-09-21T12:30:14.258802+0000","last_became_active":"2024-09-21T12:27:16.921880+0000","last_became_peered":"2024-09-21T12:27:16.921880+0000","last_unstale":"2024-09-21T12:30:14.258802+0000","last_undegraded":"2024-09-21T12:30:14.258802+0000","last_fullsized":"2024-09-21T12:30:14.258802+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:55.291509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:55.291509+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:57.999281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"66'9","reported_seq":77,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.260684+0000","last_change":"2024-09-21T12:29:35.787713+0000","last_active":"2024-09-21T12:30:14.260684+0000","last_peered":"2024-09-21T12:30:14.260684+0000","last_clean":"2024-09-21T12:30:14.260684+0000","last_became_active":"2024-09-21T12:29:35.765649+0000","last_became_peered":"2024-09-21T12:29:35.765649+0000","last_unstale":"2024-09-21T12:30:14.260684+0000","last_undegraded":"2024-09-21T12:30:14.260684+0000","last_fullsized":"2024-09-21T12:30:14.260684+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:28:36.645961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:36.645961+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:31:07.210144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":491,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.257765+0000","last_change":"2024-09-21T12:26:19.973528+0000","last_active":"2024-09-21T12:30:14.257765+0000","last_peered":"2024-09-21T12:30:14.257765+0000","last_clean":"2024-09-21T12:30:14.257765+0000","last_became_active":"2024-09-21T12:22:50.922019+0000","last_became_peered":"2024-09-21T12:22:50.922019+0000","last_unstale":"2024-09-21T12:30:14.257765+0000","last_undegraded":"2024-09-21T12:30:14.257765+0000","last_fullsized":"2024-09-21T12:30:14.257765+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:19.973461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:19.973461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":453,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.258865+0000","last_change":"2024-09-21T12:26:21.572474+0000","last_active":"2024-09-21T12:30:14.258865+0000","last_peered":"2024-09-21T12:30:14.258865+0000","last_clean":"2024-09-21T12:30:14.258865+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:30:14.258865+0000","last_undegraded":"2024-09-21T12:30:14.258865+0000","last_fullsized":"2024-09-21T12:30:14.258865+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:21.571726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:21.571726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":452,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.258364+0000","last_change":"2024-09-21T12:28:41.864711+0000","last_active":"2024-09-21T12:30:14.258364+0000","last_peered":"2024-09-21T12:30:14.258364+0000","last_clean":"2024-09-21T12:30:14.258364+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:30:14.258364+0000","last_undegraded":"2024-09-21T12:30:14.258364+0000","last_fullsized":"2024-09-21T12:30:14.258364+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:41.864630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:41.864630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"76'79","reported_seq":161,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.251279+0000","last_change":"2024-09-21T12:26:49.624804+0000","last_active":"2024-09-21T12:30:14.251279+0000","last_peered":"2024-09-21T12:30:14.251279+0000","last_clean":"2024-09-21T12:30:14.251279+0000","last_became_active":"2024-09-21T12:26:46.989742+0000","last_became_peered":"2024-09-21T12:26:46.989742+0000","last_unstale":"2024-09-21T12:30:14.251279+0000","last_undegraded":"2024-09-21T12:30:14.251279+0000","last_fullsized":"2024-09-21T12:30:14.251279+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:49.623735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:49.623735+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":476,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.265181+0000","last_change":"2024-09-21T12:24:58.824615+0000","last_active":"2024-09-21T12:30:14.265181+0000","last_peered":"2024-09-21T12:30:14.265181+0000","last_clean":"2024-09-21T12:30:14.265181+0000","last_became_active":"2024-09-21T12:20:17.694794+0000","last_became_peered":"2024-09-21T12:20:17.694794+0000","last_unstale":"2024-09-21T12:30:14.265181+0000","last_undegraded":"2024-09-21T12:30:14.265181+0000","last_fullsized":"2024-09-21T12:30:14.265181+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:24:58.824516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:24:58.824516+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:20.197865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":486,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.258581+0000","last_change":"2024-09-21T12:29:54.303655+0000","last_active":"2024-09-21T12:30:14.258581+0000","last_peered":"2024-09-21T12:30:14.258581+0000","last_clean":"2024-09-21T12:30:14.258581+0000","last_became_active":"2024-09-21T12:20:15.295968+0000","last_became_peered":"2024-09-21T12:20:15.295968+0000","last_unstale":"2024-09-21T12:30:14.258581+0000","last_undegraded":"2024-09-21T12:30:14.258581+0000","last_fullsized":"2024-09-21T12:30:14.258581+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:54.303590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:54.303590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:31:16.447834+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":467,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.256015+0000","last_change":"2024-09-21T12:23:57.018797+0000","last_active":"2024-09-21T12:30:14.256015+0000","last_peered":"2024-09-21T12:30:14.256015+0000","last_clean":"2024-09-21T12:30:14.256015+0000","last_became_active":"2024-09-21T12:22:50.921604+0000","last_became_peered":"2024-09-21T12:22:50.921604+0000","last_unstale":"2024-09-21T12:30:14.256015+0000","last_undegraded":"2024-09-21T12:30:14.256015+0000","last_fullsized":"2024-09-21T12:30:14.256015+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:57.018749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:57.018749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:22.755530+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":449,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.259017+0000","last_change":"2024-09-21T12:29:14.051153+0000","last_active":"2024-09-21T12:30:14.259017+0000","last_peered":"2024-09-21T12:30:14.259017+0000","last_clean":"2024-09-21T12:30:14.259017+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:30:14.259017+0000","last_undegraded":"2024-09-21T12:30:14.259017+0000","last_fullsized":"2024-09-21T12:30:14.259017+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:14.051043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:29:14.051043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:23.917212+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"76'79","reported_seq":132,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.256566+0000","last_change":"2024-09-21T12:27:24.373557+0000","last_active":"2024-09-21T12:30:14.256566+0000","last_peered":"2024-09-21T12:30:14.256566+0000","last_clean":"2024-09-21T12:30:14.256566+0000","last_became_active":"2024-09-21T12:27:16.914713+0000","last_became_peered":"2024-09-21T12:27:16.914713+0000","last_unstale":"2024-09-21T12:30:14.256566+0000","last_undegraded":"2024-09-21T12:30:14.256566+0000","last_fullsized":"2024-09-21T12:30:14.256566+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:24.373501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:24.373501+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:35.520215+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":463,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.257116+0000","last_change":"2024-09-21T12:28:46.932522+0000","last_active":"2024-09-21T12:30:14.257116+0000","last_peered":"2024-09-21T12:30:14.257116+0000","last_clean":"2024-09-21T12:30:14.257116+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:30:14.257116+0000","last_undegraded":"2024-09-21T12:30:14.257116+0000","last_fullsized":"2024-09-21T12:30:14.257116+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-21T12:28:46.931901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:46.931901+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":478,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.258049+0000","last_change":"2024-09-21T12:28:00.838008+0000","last_active":"2024-09-21T12:30:14.258049+0000","last_peered":"2024-09-21T12:30:14.258049+0000","last_clean":"2024-09-21T12:30:14.258049+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:30:14.258049+0000","last_undegraded":"2024-09-21T12:30:14.258049+0000","last_fullsized":"2024-09-21T12:30:14.258049+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:28:00.837937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:00.837937+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":483,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.260402+0000","last_change":"2024-09-21T12:29:59.216485+0000","last_active":"2024-09-21T12:30:14.260402+0000","last_peered":"2024-09-21T12:30:14.260402+0000","last_clean":"2024-09-21T12:30:14.260402+0000","last_became_active":"2024-09-21T12:22:50.930615+0000","last_became_peered":"2024-09-21T12:22:50.930615+0000","last_unstale":"2024-09-21T12:30:14.260402+0000","last_undegraded":"2024-09-21T12:30:14.260402+0000","last_fullsized":"2024-09-21T12:30:14.260402+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:59.216442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:29:59.216442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:31:06.036173+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":471,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.258531+0000","last_change":"2024-09-21T12:28:41.537845+0000","last_active":"2024-09-21T12:30:14.258531+0000","last_peered":"2024-09-21T12:30:14.258531+0000","last_clean":"2024-09-21T12:30:14.258531+0000","last_became_active":"2024-09-21T12:22:50.917408+0000","last_became_peered":"2024-09-21T12:22:50.917408+0000","last_unstale":"2024-09-21T12:30:14.258531+0000","last_undegraded":"2024-09-21T12:30:14.258531+0000","last_fullsized":"2024-09-21T12:30:14.258531+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:41.537786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:41.537786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"76'79","reported_seq":132,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.258740+0000","last_change":"2024-09-21T12:27:29.341314+0000","last_active":"2024-09-21T12:30:14.258740+0000","last_peered":"2024-09-21T12:30:14.258740+0000","last_clean":"2024-09-21T12:30:14.258740+0000","last_became_active":"2024-09-21T12:27:16.948697+0000","last_became_peered":"2024-09-21T12:27:16.948697+0000","last_unstale":"2024-09-21T12:30:14.258740+0000","last_undegraded":"2024-09-21T12:30:14.258740+0000","last_fullsized":"2024-09-21T12:30:14.258740+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:29.341268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:29.341268+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:31.477463+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":470,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.265701+0000","last_change":"2024-09-21T12:28:46.749904+0000","last_active":"2024-09-21T12:30:14.265701+0000","last_peered":"2024-09-21T12:30:14.265701+0000","last_clean":"2024-09-21T12:30:14.265701+0000","last_became_active":"2024-09-21T12:22:50.914221+0000","last_became_peered":"2024-09-21T12:22:50.914221+0000","last_unstale":"2024-09-21T12:30:14.265701+0000","last_undegraded":"2024-09-21T12:30:14.265701+0000","last_fullsized":"2024-09-21T12:30:14.265701+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:46.749839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:46.749839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":476,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.258435+0000","last_change":"2024-09-21T12:29:35.772635+0000","last_active":"2024-09-21T12:30:14.258435+0000","last_peered":"2024-09-21T12:30:14.258435+0000","last_clean":"2024-09-21T12:30:14.258435+0000","last_became_active":"2024-09-21T12:29:35.771591+0000","last_became_peered":"2024-09-21T12:29:35.771591+0000","last_unstale":"2024-09-21T12:30:14.258435+0000","last_undegraded":"2024-09-21T12:30:14.258435+0000","last_fullsized":"2024-09-21T12:30:14.258435+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:15.071234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:15.071234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:30:35.981843+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":485,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.261368+0000","last_change":"2024-09-21T12:30:02.290006+0000","last_active":"2024-09-21T12:30:14.261368+0000","last_peered":"2024-09-21T12:30:14.261368+0000","last_clean":"2024-09-21T12:30:14.261368+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:30:14.261368+0000","last_undegraded":"2024-09-21T12:30:14.261368+0000","last_fullsized":"2024-09-21T12:30:14.261368+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:30:02.289930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:30:02.289930+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:31:13.918080+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":480,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T12:30:14.257955+0000","last_change":"2024-09-21T12:28:28.741483+0000","last_active":"2024-09-21T12:30:14.257955+0000","last_peered":"2024-09-21T12:30:14.257955+0000","last_clean":"2024-09-21T12:30:14.257955+0000","last_became_active":"2024-09-21T12:20:19.308247+0000","last_became_peered":"2024-09-21T12:20:19.308247+0000","last_unstale":"2024-09-21T12:30:14.257955+0000","last_undegraded":"2024-09-21T12:30:14.257955+0000","last_fullsized":"2024-09-21T12:30:14.257955+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:28.741418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:28.741418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":6994591768,"num_objects":106730,"num_object_clones":0,"num_object_copies":320190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":42070,"num_objects_unfound":0,"num_objects_dirty":106730,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141334,"num_write_kb":7937985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23737,"num_bytes_recovered":1555628032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22158782464,"data_stored":22158770248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62943,"ondisk_log_size":62943,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":684,"num_read_kb":441,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":216,"num_bytes_recovered":4400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":397,"ondisk_log_size":397,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":690,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":5023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":595,"num_read_kb":590,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":761964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":15,"num_write":7,"num_write_kb":111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":40252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2449,"ondisk_log_size":2449,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234108,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4661972,"kb_used_data":4229568,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":89083180,"statfs":{"total":95995035648,"available":91221176320,"internally_reserved":0,"allocated":4331077632,"data_stored":4330766745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":442759626},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463044,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2105500,"kb_used_data":1735564,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":91639652,"statfs":{"total":95995035648,"available":93839003648,"internally_reserved":0,"allocated":1777217536,"data_stored":1776893920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":378796486},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724679,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4244892,"kb_used_data":3880524,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":89500260,"statfs":{"total":95995035648,"available":91648266240,"internally_reserved":0,"allocated":3973656576,"data_stored":3973334429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953615,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149845,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4102500,"kb_used_data":3587988,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":89642652,"statfs":{"total":95995035648,"available":91794075648,"internally_reserved":0,"allocated":3674099712,"data_stored":3673769122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":526645703},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411483,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1883980,"kb_used_data":1527932,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":91861172,"statfs":{"total":95995035648,"available":94065840128,"internally_reserved":0,"allocated":1564602368,"data_stored":1564295179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575009,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4303328,"kb_used_data":3772240,"kb_used_omap":1,"kb_used_meta":530942,"kb_avail":89441824,"statfs":{"total":95995035648,"available":91588427776,"internally_reserved":0,"allocated":3862773760,"data_stored":3862446071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":543685061},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,4,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738535,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3361252,"kb_used_data":2982932,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":90383900,"statfs":{"total":95995035648,"available":92553113600,"internally_reserved":0,"allocated":3054522368,"data_stored":3054191632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3046768640,"data_stored":3046768640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3861512192,"data_stored":3861512192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1559887872,"data_stored":1559887872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3646488576,"data_stored":3646488576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3958902784,"data_stored":3958898712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1765806080,"data_stored":1765802008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4319416320,"data_stored":4319412248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:30:26.298 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T12:30:26.298 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pool set unique_pool_2 pg_num 26 2024-09-21T12:30:26.315 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:26.338 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:26.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:26.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:26.440 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:26.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:26.484 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:26.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:26.585 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:26.611 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:26.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:26.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:26.713 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:26.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:26.740 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:26.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:26.841 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:26.865 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:26.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:26.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:26.950 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:30:26.967 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:27.019 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:27.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:27.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:27.121 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:27.153 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:27.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:27.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:27.255 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:27.281 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:27.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:27.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:27.381 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:27 smithi114 ceph-mon[40867]: pgmap v494: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 7.3 KiB/s rd, 0 B/s wr, 869 op/s; 38277/308406 objects misplaced (12.411%); 27 MiB/s, 437 objects/s recovering 2024-09-21T12:30:27.383 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:27.383 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:27 smithi114 ceph-mon[30324]: pgmap v494: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 7.3 KiB/s rd, 0 B/s wr, 869 op/s; 38277/308406 objects misplaced (12.411%); 27 MiB/s, 437 objects/s recovering 2024-09-21T12:30:27.409 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:27.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:27.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:27.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:27 smithi183 ceph-mon[33277]: pgmap v494: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 7.3 KiB/s rd, 0 B/s wr, 869 op/s; 38277/308406 objects misplaced (12.411%); 27 MiB/s, 437 objects/s recovering 2024-09-21T12:30:27.488 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:30:27 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:30:27.511 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:27.535 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:27.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:27.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:27.637 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:27.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:27.680 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:27.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:27.781 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:27.829 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:27.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:27.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:27.930 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:27.956 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:27.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:27.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:27.996 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:30:27.996 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:30:28.039 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:30:28.039 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:30:28.044 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:30:28.044 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:30:28.045 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:30:28.045 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:30:28.045 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:30:28.058 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:28.073 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:30:28.073 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:30:28.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:28.096 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:28.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:28.109 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:30:28.109 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:30:28.140 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:30:28.141 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:30:28.171 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:30:28.172 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:30:28.197 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:28.204 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:30:28.204 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:30:28.222 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:28.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:28.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:28.238 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:30:28.238 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:30:28.323 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:28.351 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:28.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:28.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:28.453 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:28.488 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:28.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:28.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:28.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:28 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:30:28.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:28 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2805694731' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:30:28.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:28 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:30:28.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:28 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2805694731' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:30:28.590 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:28.617 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:28.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:28.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:28.718 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:28.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:28 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:30:28.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:28 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2805694731' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:30:28.746 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:28.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:28.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:28.848 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:28.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:28.888 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:28.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:28.917 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T12:30:28.988 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:29.013 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:29.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:29.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:29.115 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:29.146 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:29.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:29.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:29.247 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:29.296 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:29.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:29.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:29.397 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:29.434 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:29.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:29.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:29.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:29 smithi114 ceph-mon[30324]: pgmap v495: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 44 KiB/s rd, 0 B/s wr, 926 op/s; 34347/295140 objects misplaced (11.638%); 25 MiB/s, 398 objects/s recovering 2024-09-21T12:30:29.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:29 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:30:29.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:29 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:30:29.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:29 smithi114 ceph-mon[30324]: osdmap e277: 8 total, 8 up, 7 in 2024-09-21T12:30:29.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:29 smithi114 ceph-mon[40867]: pgmap v495: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 44 KiB/s rd, 0 B/s wr, 926 op/s; 34347/295140 objects misplaced (11.638%); 25 MiB/s, 398 objects/s recovering 2024-09-21T12:30:29.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:29 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:30:29.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:29 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:30:29.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:29 smithi114 ceph-mon[40867]: osdmap e277: 8 total, 8 up, 7 in 2024-09-21T12:30:29.536 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:29.562 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:29.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:29.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:29.664 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:29.700 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:29.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:29.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:29 smithi183 ceph-mon[33277]: pgmap v495: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 44 KiB/s rd, 0 B/s wr, 926 op/s; 34347/295140 objects misplaced (11.638%); 25 MiB/s, 398 objects/s recovering 2024-09-21T12:30:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:29 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:30:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:29 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:30:29.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:29 smithi183 ceph-mon[33277]: osdmap e277: 8 total, 8 up, 7 in 2024-09-21T12:30:29.802 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:29.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:29.840 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:29.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:29.941 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:29.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:29.963 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:29.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:30.064 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:30.091 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:30.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:30.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:30.193 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:30.218 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:30.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:30.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:30.320 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:30.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:30.357 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:30.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:30.458 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:30.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:30.498 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:30.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:30.599 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:30.625 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:30.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:30.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:30.727 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:30.752 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:30.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:30.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:30.854 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:30.880 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:30.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:30.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:30.981 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:31.008 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:31.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:31.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:31.109 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:31.135 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:31.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:31.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:31.237 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:31.275 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:31.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:31.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:31.378 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:31.414 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:31.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:31.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:31.516 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:31.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:31 smithi114 ceph-mon[30324]: pgmap v497: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 44 KiB/s rd, 0 B/s wr, 926 op/s; 34347/295140 objects misplaced (11.638%); 25 MiB/s, 398 objects/s recovering 2024-09-21T12:30:31.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:31 smithi114 ceph-mon[40867]: pgmap v497: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 44 KiB/s rd, 0 B/s wr, 926 op/s; 34347/295140 objects misplaced (11.638%); 25 MiB/s, 398 objects/s recovering 2024-09-21T12:30:31.556 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:31.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:31.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:31.658 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:31.700 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:31.784 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:31 smithi183 ceph-mon[33277]: pgmap v497: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 44 KiB/s rd, 0 B/s wr, 926 op/s; 34347/295140 objects misplaced (11.638%); 25 MiB/s, 398 objects/s recovering 2024-09-21T12:30:31.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:31.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:31.864 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:30:31.885 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:31.911 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:31.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:31.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:32.013 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:32.041 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:32.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:32.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:32.143 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:32.182 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:32.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:32.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:32.284 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:32.321 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:32.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:32.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:32.422 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:32.451 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:32.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:32.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:32.508 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:32 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:30:32.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:32 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:30:32.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:32 smithi114 ceph-mon[40867]: osdmap e278: 8 total, 8 up, 7 in 2024-09-21T12:30:32.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:32 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:30:32.509 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:32 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:30:32.509 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:32 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:30:32.509 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:32 smithi114 ceph-mon[30324]: osdmap e278: 8 total, 8 up, 7 in 2024-09-21T12:30:32.509 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:32 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:30:32.510 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T12:30:32.553 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:32.597 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:32.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:32.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:32.699 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:32.723 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:32.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:32.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:32.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:32 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:30:32.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:32 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:30:32.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:32 smithi183 ceph-mon[33277]: osdmap e278: 8 total, 8 up, 7 in 2024-09-21T12:30:32.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:32 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:30:32.774 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:30:32 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:30:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:30:32.825 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:32.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:32.851 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:32.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:32.952 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:32.999 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:33.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:33.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:33.101 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:33.122 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:33.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:33.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:33.224 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:33.245 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:33.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:33.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:33.347 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:33.375 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:33.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:33.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:33.476 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:33.502 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:33.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:33.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:33.604 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:33.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:33.631 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:33.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:33.682 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:33 smithi114 ceph-mon[40867]: pgmap v499: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 47 KiB/s rd, 0 B/s wr, 775 op/s; 34347/291636 objects misplaced (11.777%); 16 MiB/s, 256 objects/s recovering 2024-09-21T12:30:33.683 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:33 smithi114 ceph-mon[30324]: pgmap v499: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 47 KiB/s rd, 0 B/s wr, 775 op/s; 34347/291636 objects misplaced (11.777%); 16 MiB/s, 256 objects/s recovering 2024-09-21T12:30:33.732 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:33.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:33 smithi183 ceph-mon[33277]: pgmap v499: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 47 KiB/s rd, 0 B/s wr, 775 op/s; 34347/291636 objects misplaced (11.777%); 16 MiB/s, 256 objects/s recovering 2024-09-21T12:30:33.777 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:33.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:33.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:33.850 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:30:33.879 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:33.902 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:33.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:33.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:34.003 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:34.045 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:34.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:34.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:34.146 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:34.168 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:34.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:34.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:34.269 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:34.308 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:34.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:34.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:34.410 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:34.448 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:34.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:34.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:34.549 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:34.576 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:34.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:34.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:34.677 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:34.703 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:34.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:34.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:34.805 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:34.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:34.851 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:34.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:34.952 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:34.987 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:34.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:34.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:35.024 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:30:35.089 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:35.115 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:35.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:35.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:35.217 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:35.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:35.278 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:35.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:35.379 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:35.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:35.415 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:35.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:35.516 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:35.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:35 smithi114 ceph-mon[40867]: pgmap v500: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 47 KiB/s rd, 0 B/s wr, 1.21k op/s; 30557/281265 objects misplaced (10.864%); 31 MiB/s, 492 objects/s recovering 2024-09-21T12:30:35.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:35 smithi114 ceph-mon[30324]: pgmap v500: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 47 KiB/s rd, 0 B/s wr, 1.21k op/s; 30557/281265 objects misplaced (10.864%); 31 MiB/s, 492 objects/s recovering 2024-09-21T12:30:35.540 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:35.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:35.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:35.642 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:35.666 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:35.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:35.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:35.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:35 smithi183 ceph-mon[33277]: pgmap v500: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 47 KiB/s rd, 0 B/s wr, 1.21k op/s; 30557/281265 objects misplaced (10.864%); 31 MiB/s, 492 objects/s recovering 2024-09-21T12:30:35.738 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:30:35.738 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:30:35.768 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:35.791 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:35.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:35.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:35.893 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:35.918 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:35.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:35.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:36.020 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:36.064 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:36.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:36.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:36.166 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:36.191 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:36.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:36.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:36.293 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:36.312 INFO:teuthology.orchestra.run.smithi114.stderr:set pool 9 pg_num to 26 2024-09-21T12:30:36.317 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:36.318 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-21T12:30:36.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:36.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:36.420 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:36.430 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:36 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/969309167' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T12:30:36.430 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:36 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T12:30:36.430 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:36 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2226704574' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:30:36.430 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:36 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:30:36.431 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:36 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/969309167' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T12:30:36.431 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:36 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T12:30:36.431 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:36 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2226704574' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:30:36.431 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:36 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:30:36.443 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:36.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:36.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:36.545 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:36.577 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:36.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:36.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:36.678 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:36.699 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:36.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:36.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:36.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:36 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/969309167' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T12:30:36.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:36 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T12:30:36.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:36 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2226704574' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:30:36.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:36 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:30:36.801 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:36.836 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:36.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:36.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:36.937 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:36.964 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:36.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:36.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:37.066 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:37.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:37.094 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:37.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:37.195 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:37.234 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:37.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:37.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:37.335 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:37.360 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:37.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:37.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:37.461 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:37.488 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:30:37 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:30:37.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:37 smithi183 ceph-mon[33277]: pgmap v501: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 578 op/s; 30557/281265 objects misplaced (10.864%); 15 MiB/s, 236 objects/s recovering 2024-09-21T12:30:37.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:37 smithi183 ceph-mon[33277]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:30:37.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:37 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]': finished 2024-09-21T12:30:37.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:37 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:30:37.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:37 smithi183 ceph-mon[33277]: osdmap e279: 8 total, 8 up, 7 in 2024-09-21T12:30:37.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:37 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:30:37.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:37 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:30:37.497 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:37.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:37.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:37.599 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:37.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:37.626 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:37.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:37.727 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:37.755 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:37.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:37.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:37.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:37 smithi114 ceph-mon[40867]: pgmap v501: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 578 op/s; 30557/281265 objects misplaced (10.864%); 15 MiB/s, 236 objects/s recovering 2024-09-21T12:30:37.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:37 smithi114 ceph-mon[40867]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:30:37.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:37 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]': finished 2024-09-21T12:30:37.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:37 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:30:37.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:37 smithi114 ceph-mon[40867]: osdmap e279: 8 total, 8 up, 7 in 2024-09-21T12:30:37.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:37 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:30:37.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:37 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:30:37.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:37 smithi114 ceph-mon[30324]: pgmap v501: 178 pgs: 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 171 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 578 op/s; 30557/281265 objects misplaced (10.864%); 15 MiB/s, 236 objects/s recovering 2024-09-21T12:30:37.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:37 smithi114 ceph-mon[30324]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:30:37.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:37 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]': finished 2024-09-21T12:30:37.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:37 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:30:37.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:37 smithi114 ceph-mon[30324]: osdmap e279: 8 total, 8 up, 7 in 2024-09-21T12:30:37.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:37 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:30:37.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:37 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:30:37.857 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:37.882 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:37.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:37.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:37.983 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:38.011 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:38.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:38.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:38.113 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:38.150 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:38.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:38.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:38.252 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:38.276 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:38.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:38.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:38.378 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:38.440 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:38.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:38.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:38.541 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:38.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:38.563 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:38.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:38.664 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:38.687 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:38.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:38.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:38.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:38 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:30:38.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:38 smithi183 ceph-mon[33277]: osdmap e280: 8 total, 8 up, 7 in 2024-09-21T12:30:38.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:38 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T12:30:38.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:38 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T12:30:38.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:38 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:30:38.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:38 smithi114 ceph-mon[40867]: osdmap e280: 8 total, 8 up, 7 in 2024-09-21T12:30:38.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:38 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T12:30:38.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:38 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T12:30:38.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:38 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:30:38.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:38 smithi114 ceph-mon[30324]: osdmap e280: 8 total, 8 up, 7 in 2024-09-21T12:30:38.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:38 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T12:30:38.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:38 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T12:30:38.789 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:38.811 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:38.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:38.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:38.912 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:38.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:38.935 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:38.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:39.036 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:39.058 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:39.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:39.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:39.160 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:39.182 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:39.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:39.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:39.284 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:39.306 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:39.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:39.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:39.408 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:39.445 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:39.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:39.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:39.547 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:39.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:39.579 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:39.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:39.680 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:39.702 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:39.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:39.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:39.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:39 smithi183 ceph-mon[33277]: pgmap v504: 178 pgs: 7 active+remapped+backfilling, 171 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.44k op/s; 26700/266997 objects misplaced (10.000%); 38 MiB/s, 609 objects/s recovering 2024-09-21T12:30:39.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:39 smithi183 ceph-mon[33277]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-21T12:30:39.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:39 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T12:30:39.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:39 smithi183 ceph-mon[33277]: osdmap e281: 8 total, 8 up, 7 in 2024-09-21T12:30:39.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:39 smithi183 ceph-mon[33277]: 5.1b scrub starts 2024-09-21T12:30:39.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:39 smithi183 ceph-mon[33277]: 5.1b scrub ok 2024-09-21T12:30:39.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:39 smithi183 ceph-mon[33277]: 1.8 scrub starts 2024-09-21T12:30:39.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:39 smithi183 ceph-mon[33277]: 1.8 scrub ok 2024-09-21T12:30:39.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:39 smithi114 ceph-mon[40867]: pgmap v504: 178 pgs: 7 active+remapped+backfilling, 171 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.44k op/s; 26700/266997 objects misplaced (10.000%); 38 MiB/s, 609 objects/s recovering 2024-09-21T12:30:39.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:39 smithi114 ceph-mon[40867]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-21T12:30:39.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:39 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T12:30:39.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:39 smithi114 ceph-mon[40867]: osdmap e281: 8 total, 8 up, 7 in 2024-09-21T12:30:39.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:39 smithi114 ceph-mon[40867]: 5.1b scrub starts 2024-09-21T12:30:39.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:39 smithi114 ceph-mon[40867]: 5.1b scrub ok 2024-09-21T12:30:39.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:39 smithi114 ceph-mon[40867]: 1.8 scrub starts 2024-09-21T12:30:39.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:39 smithi114 ceph-mon[40867]: 1.8 scrub ok 2024-09-21T12:30:39.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:39 smithi114 ceph-mon[30324]: pgmap v504: 178 pgs: 7 active+remapped+backfilling, 171 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.44k op/s; 26700/266997 objects misplaced (10.000%); 38 MiB/s, 609 objects/s recovering 2024-09-21T12:30:39.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:39 smithi114 ceph-mon[30324]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-21T12:30:39.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:39 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T12:30:39.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:39 smithi114 ceph-mon[30324]: osdmap e281: 8 total, 8 up, 7 in 2024-09-21T12:30:39.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:39 smithi114 ceph-mon[30324]: 5.1b scrub starts 2024-09-21T12:30:39.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:39 smithi114 ceph-mon[30324]: 5.1b scrub ok 2024-09-21T12:30:39.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:39 smithi114 ceph-mon[30324]: 1.8 scrub starts 2024-09-21T12:30:39.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:39 smithi114 ceph-mon[30324]: 1.8 scrub ok 2024-09-21T12:30:39.803 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:39.825 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:39.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:39.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:39.926 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:39.947 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:39.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:39.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:40.048 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:40.078 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:40.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:40.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:40.179 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:40.201 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:40.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:40.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:40.302 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:40.323 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:40.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:40.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:40.425 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:40.445 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:40.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:40.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:40.547 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:40.577 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:40.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:40.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:40.678 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:40.699 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:40.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:40.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:40.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:40 smithi183 ceph-mon[33277]: 1.2 scrub starts 2024-09-21T12:30:40.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:40 smithi183 ceph-mon[33277]: 1.2 scrub ok 2024-09-21T12:30:40.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:40 smithi183 ceph-mon[33277]: osdmap e282: 8 total, 8 up, 7 in 2024-09-21T12:30:40.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:40 smithi183 ceph-mon[33277]: 9.0 continuing backfill to osd.0 from (260'5800,280'9956] MIN to 280'9956 2024-09-21T12:30:40.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:40 smithi183 ceph-mon[33277]: 9.1 continuing backfill to osd.2 from (261'6100,280'10054] MIN to 280'10054 2024-09-21T12:30:40.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:40 smithi183 ceph-mon[33277]: 9.8 continuing backfill to osd.1 from (262'6300,280'9982] MIN to 280'9982 2024-09-21T12:30:40.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:40 smithi183 ceph-mon[33277]: 9.8 continuing backfill to osd.7 from (262'6300,280'9982] MIN to 280'9982 2024-09-21T12:30:40.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:40 smithi183 ceph-mon[33277]: 9.5 continuing backfill to osd.1 from (262'6275,280'9963] MIN to 280'9963 2024-09-21T12:30:40.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:40 smithi183 ceph-mon[33277]: 9.7 continuing backfill to osd.1 from (261'6100,280'10002] MIN to 280'10002 2024-09-21T12:30:40.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:40 smithi183 ceph-mon[33277]: 9.7 continuing backfill to osd.2 from (261'6100,280'10002] MIN to 280'10002 2024-09-21T12:30:40.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:40 smithi183 ceph-mon[33277]: 9.2 continuing backfill to osd.7 from (260'5900,280'9992] MIN to 280'9992 2024-09-21T12:30:40.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:40 smithi183 ceph-mon[33277]: pgmap v507: 188 pgs: 10 unknown, 7 active+remapped+backfilling, 171 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 26700/266997 objects misplaced (10.000%); 32 MiB/s, 516 objects/s recovering 2024-09-21T12:30:40.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:40 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-21T12:30:40.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:40 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-21T12:30:40.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:40 smithi114 ceph-mon[30324]: 1.2 scrub starts 2024-09-21T12:30:40.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:40 smithi114 ceph-mon[30324]: 1.2 scrub ok 2024-09-21T12:30:40.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:40 smithi114 ceph-mon[30324]: osdmap e282: 8 total, 8 up, 7 in 2024-09-21T12:30:40.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:40 smithi114 ceph-mon[30324]: 9.0 continuing backfill to osd.0 from (260'5800,280'9956] MIN to 280'9956 2024-09-21T12:30:40.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:40 smithi114 ceph-mon[30324]: 9.1 continuing backfill to osd.2 from (261'6100,280'10054] MIN to 280'10054 2024-09-21T12:30:40.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:40 smithi114 ceph-mon[30324]: 9.8 continuing backfill to osd.1 from (262'6300,280'9982] MIN to 280'9982 2024-09-21T12:30:40.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:40 smithi114 ceph-mon[30324]: 9.8 continuing backfill to osd.7 from (262'6300,280'9982] MIN to 280'9982 2024-09-21T12:30:40.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:40 smithi114 ceph-mon[30324]: 9.5 continuing backfill to osd.1 from (262'6275,280'9963] MIN to 280'9963 2024-09-21T12:30:40.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:40 smithi114 ceph-mon[30324]: 9.7 continuing backfill to osd.1 from (261'6100,280'10002] MIN to 280'10002 2024-09-21T12:30:40.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:40 smithi114 ceph-mon[30324]: 9.7 continuing backfill to osd.2 from (261'6100,280'10002] MIN to 280'10002 2024-09-21T12:30:40.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:40 smithi114 ceph-mon[30324]: 9.2 continuing backfill to osd.7 from (260'5900,280'9992] MIN to 280'9992 2024-09-21T12:30:40.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:40 smithi114 ceph-mon[30324]: pgmap v507: 188 pgs: 10 unknown, 7 active+remapped+backfilling, 171 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 26700/266997 objects misplaced (10.000%); 32 MiB/s, 516 objects/s recovering 2024-09-21T12:30:40.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:40 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-21T12:30:40.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:40 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-21T12:30:40.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:40 smithi114 ceph-mon[40867]: 1.2 scrub starts 2024-09-21T12:30:40.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:40 smithi114 ceph-mon[40867]: 1.2 scrub ok 2024-09-21T12:30:40.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:40 smithi114 ceph-mon[40867]: osdmap e282: 8 total, 8 up, 7 in 2024-09-21T12:30:40.777 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:40 smithi114 ceph-mon[40867]: 9.0 continuing backfill to osd.0 from (260'5800,280'9956] MIN to 280'9956 2024-09-21T12:30:40.777 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:40 smithi114 ceph-mon[40867]: 9.1 continuing backfill to osd.2 from (261'6100,280'10054] MIN to 280'10054 2024-09-21T12:30:40.777 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:40 smithi114 ceph-mon[40867]: 9.8 continuing backfill to osd.1 from (262'6300,280'9982] MIN to 280'9982 2024-09-21T12:30:40.777 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:40 smithi114 ceph-mon[40867]: 9.8 continuing backfill to osd.7 from (262'6300,280'9982] MIN to 280'9982 2024-09-21T12:30:40.777 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:40 smithi114 ceph-mon[40867]: 9.5 continuing backfill to osd.1 from (262'6275,280'9963] MIN to 280'9963 2024-09-21T12:30:40.777 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:40 smithi114 ceph-mon[40867]: 9.7 continuing backfill to osd.1 from (261'6100,280'10002] MIN to 280'10002 2024-09-21T12:30:40.777 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:40 smithi114 ceph-mon[40867]: 9.7 continuing backfill to osd.2 from (261'6100,280'10002] MIN to 280'10002 2024-09-21T12:30:40.777 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:40 smithi114 ceph-mon[40867]: 9.2 continuing backfill to osd.7 from (260'5900,280'9992] MIN to 280'9992 2024-09-21T12:30:40.777 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:40 smithi114 ceph-mon[40867]: pgmap v507: 188 pgs: 10 unknown, 7 active+remapped+backfilling, 171 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 26700/266997 objects misplaced (10.000%); 32 MiB/s, 516 objects/s recovering 2024-09-21T12:30:40.777 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:40 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-21T12:30:40.777 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:40 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-21T12:30:40.802 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:40.828 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:40.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:40.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:40.930 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:40.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:40.951 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:40.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:41.052 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:41.074 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:41.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:41.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:41.175 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:41.197 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:41.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:41.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:41.298 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:41.325 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:41.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:41.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:41.427 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:41.448 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:41.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:41.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:41.550 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:41.578 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:41.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:41.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:41.679 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:41.700 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:41.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:41.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:41.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:41 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-21T12:30:41.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:41 smithi183 ceph-mon[33277]: osdmap e283: 8 total, 8 up, 7 in 2024-09-21T12:30:41.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:41 smithi183 ceph-mon[33277]: 6.4 scrub starts 2024-09-21T12:30:41.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:41 smithi183 ceph-mon[33277]: 6.4 scrub ok 2024-09-21T12:30:41.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:41 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-21T12:30:41.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:41 smithi114 ceph-mon[40867]: osdmap e283: 8 total, 8 up, 7 in 2024-09-21T12:30:41.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:41 smithi114 ceph-mon[40867]: 6.4 scrub starts 2024-09-21T12:30:41.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:41 smithi114 ceph-mon[40867]: 6.4 scrub ok 2024-09-21T12:30:41.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:41 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-21T12:30:41.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:41 smithi114 ceph-mon[30324]: osdmap e283: 8 total, 8 up, 7 in 2024-09-21T12:30:41.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:41 smithi114 ceph-mon[30324]: 6.4 scrub starts 2024-09-21T12:30:41.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:41 smithi114 ceph-mon[30324]: 6.4 scrub ok 2024-09-21T12:30:41.801 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:41.831 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:41.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:41.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:41.933 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:41.964 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:41.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:41.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:42.066 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:42.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:42.102 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:42.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:42.203 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:42.227 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:42.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:42.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:42.329 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:42.359 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:42.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:42.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:42.460 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:42.492 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:42.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:42.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:42.594 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:42.623 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:42.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:42.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:42.641 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:30:42 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:30:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:30:42.641 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:42 smithi114 ceph-mon[40867]: 9.18 starting backfill to osd.6 from (0'0,0'0] MAX to 280'9982 2024-09-21T12:30:42.641 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:42 smithi114 ceph-mon[40867]: 9.10 continuing backfill to osd.0 from (260'5800,280'9956] MIN to 280'9956 2024-09-21T12:30:42.641 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:42 smithi114 ceph-mon[40867]: 9.10 starting backfill to osd.7 from (0'0,0'0] MAX to 280'9956 2024-09-21T12:30:42.642 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:42 smithi114 ceph-mon[40867]: osdmap e284: 8 total, 8 up, 7 in 2024-09-21T12:30:42.642 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:42 smithi114 ceph-mon[40867]: 9.12 starting backfill to osd.0 from (0'0,0'0] MAX to 280'9992 2024-09-21T12:30:42.642 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:42 smithi114 ceph-mon[40867]: 9.18 continuing backfill to osd.7 from (262'6300,280'9982] MIN to 280'9982 2024-09-21T12:30:42.642 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:42 smithi114 ceph-mon[40867]: 9.12 starting backfill to osd.5 from (0'0,0'0] MAX to 280'9992 2024-09-21T12:30:42.642 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:42 smithi114 ceph-mon[40867]: pgmap v510: 188 pgs: 1 peering, 4 remapped+peering, 5 unknown, 7 active+remapped+backfilling, 171 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 242 MiB/s wr, 5.11k op/s; 26700/298611 objects misplaced (8.941%); 92 MiB/s, 1.47k objects/s recovering 2024-09-21T12:30:42.642 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:42 smithi114 ceph-mon[40867]: osdmap e285: 8 total, 8 up, 7 in 2024-09-21T12:30:42.642 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:42 smithi114 ceph-mon[30324]: 9.18 starting backfill to osd.6 from (0'0,0'0] MAX to 280'9982 2024-09-21T12:30:42.642 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:42 smithi114 ceph-mon[30324]: 9.10 continuing backfill to osd.0 from (260'5800,280'9956] MIN to 280'9956 2024-09-21T12:30:42.642 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:42 smithi114 ceph-mon[30324]: 9.10 starting backfill to osd.7 from (0'0,0'0] MAX to 280'9956 2024-09-21T12:30:42.642 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:42 smithi114 ceph-mon[30324]: osdmap e284: 8 total, 8 up, 7 in 2024-09-21T12:30:42.642 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:42 smithi114 ceph-mon[30324]: 9.12 starting backfill to osd.0 from (0'0,0'0] MAX to 280'9992 2024-09-21T12:30:42.642 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:42 smithi114 ceph-mon[30324]: 9.18 continuing backfill to osd.7 from (262'6300,280'9982] MIN to 280'9982 2024-09-21T12:30:42.642 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:42 smithi114 ceph-mon[30324]: 9.12 starting backfill to osd.5 from (0'0,0'0] MAX to 280'9992 2024-09-21T12:30:42.643 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:42 smithi114 ceph-mon[30324]: pgmap v510: 188 pgs: 1 peering, 4 remapped+peering, 5 unknown, 7 active+remapped+backfilling, 171 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 242 MiB/s wr, 5.11k op/s; 26700/298611 objects misplaced (8.941%); 92 MiB/s, 1.47k objects/s recovering 2024-09-21T12:30:42.643 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:42 smithi114 ceph-mon[30324]: osdmap e285: 8 total, 8 up, 7 in 2024-09-21T12:30:42.724 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:42.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:42 smithi183 ceph-mon[33277]: 9.18 starting backfill to osd.6 from (0'0,0'0] MAX to 280'9982 2024-09-21T12:30:42.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:42 smithi183 ceph-mon[33277]: 9.10 continuing backfill to osd.0 from (260'5800,280'9956] MIN to 280'9956 2024-09-21T12:30:42.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:42 smithi183 ceph-mon[33277]: 9.10 starting backfill to osd.7 from (0'0,0'0] MAX to 280'9956 2024-09-21T12:30:42.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:42 smithi183 ceph-mon[33277]: osdmap e284: 8 total, 8 up, 7 in 2024-09-21T12:30:42.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:42 smithi183 ceph-mon[33277]: 9.12 starting backfill to osd.0 from (0'0,0'0] MAX to 280'9992 2024-09-21T12:30:42.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:42 smithi183 ceph-mon[33277]: 9.18 continuing backfill to osd.7 from (262'6300,280'9982] MIN to 280'9982 2024-09-21T12:30:42.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:42 smithi183 ceph-mon[33277]: 9.12 starting backfill to osd.5 from (0'0,0'0] MAX to 280'9992 2024-09-21T12:30:42.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:42 smithi183 ceph-mon[33277]: pgmap v510: 188 pgs: 1 peering, 4 remapped+peering, 5 unknown, 7 active+remapped+backfilling, 171 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 242 MiB/s wr, 5.11k op/s; 26700/298611 objects misplaced (8.941%); 92 MiB/s, 1.47k objects/s recovering 2024-09-21T12:30:42.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:42 smithi183 ceph-mon[33277]: osdmap e285: 8 total, 8 up, 7 in 2024-09-21T12:30:42.759 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:42.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:42.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:42.861 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:42.893 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:42.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:42.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:42.994 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:43.025 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:43.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:43.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:43.126 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:43.156 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:43.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:43.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:43.258 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:43.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:43.299 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:43.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:43.399 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:43.430 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:43.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:43.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:43.531 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:43.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:43.565 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:43.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:43.575 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:30:43.575 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:30:43.666 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:43.704 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:43.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:43.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:43.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:43 smithi183 ceph-mon[33277]: 9.13 starting backfill to osd.5 from (0'0,0'0] MAX to 282'9963 2024-09-21T12:30:43.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:43 smithi183 ceph-mon[33277]: 9.19 starting backfill to osd.6 from (0'0,0'0] MAX to 280'9985 2024-09-21T12:30:43.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:43 smithi183 ceph-mon[33277]: 9.17 starting backfill to osd.0 from (0'0,0'0] MAX to 280'10002 2024-09-21T12:30:43.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:43 smithi183 ceph-mon[33277]: 9.15 starting backfill to osd.2 from (0'0,0'0] MAX to 280'9963 2024-09-21T12:30:43.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:43 smithi183 ceph-mon[33277]: 9.14 starting backfill to osd.2 from (0'0,0'0] MAX to 280'10022 2024-09-21T12:30:43.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:43 smithi183 ceph-mon[33277]: 9.16 starting backfill to osd.6 from (0'0,0'0] MAX to 282'10024 2024-09-21T12:30:43.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:43 smithi183 ceph-mon[33277]: 9.13 starting backfill to osd.6 from (0'0,0'0] MAX to 282'9963 2024-09-21T12:30:43.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:43 smithi183 ceph-mon[33277]: 9.17 continuing backfill to osd.1 from (261'6100,280'10002] MIN to 280'10002 2024-09-21T12:30:43.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:43 smithi183 ceph-mon[33277]: 9.15 starting backfill to osd.3 from (0'0,0'0] MAX to 280'9963 2024-09-21T12:30:43.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:43 smithi183 ceph-mon[33277]: 9.14 starting backfill to osd.5 from (0'0,0'0] MAX to 280'10022 2024-09-21T12:30:43.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:43 smithi183 ceph-mon[33277]: osdmap e286: 8 total, 8 up, 7 in 2024-09-21T12:30:43.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:43 smithi114 ceph-mon[30324]: 9.13 starting backfill to osd.5 from (0'0,0'0] MAX to 282'9963 2024-09-21T12:30:43.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:43 smithi114 ceph-mon[30324]: 9.19 starting backfill to osd.6 from (0'0,0'0] MAX to 280'9985 2024-09-21T12:30:43.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:43 smithi114 ceph-mon[30324]: 9.17 starting backfill to osd.0 from (0'0,0'0] MAX to 280'10002 2024-09-21T12:30:43.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:43 smithi114 ceph-mon[30324]: 9.15 starting backfill to osd.2 from (0'0,0'0] MAX to 280'9963 2024-09-21T12:30:43.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:43 smithi114 ceph-mon[30324]: 9.14 starting backfill to osd.2 from (0'0,0'0] MAX to 280'10022 2024-09-21T12:30:43.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:43 smithi114 ceph-mon[30324]: 9.16 starting backfill to osd.6 from (0'0,0'0] MAX to 282'10024 2024-09-21T12:30:43.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:43 smithi114 ceph-mon[30324]: 9.13 starting backfill to osd.6 from (0'0,0'0] MAX to 282'9963 2024-09-21T12:30:43.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:43 smithi114 ceph-mon[30324]: 9.17 continuing backfill to osd.1 from (261'6100,280'10002] MIN to 280'10002 2024-09-21T12:30:43.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:43 smithi114 ceph-mon[30324]: 9.15 starting backfill to osd.3 from (0'0,0'0] MAX to 280'9963 2024-09-21T12:30:43.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:43 smithi114 ceph-mon[30324]: 9.14 starting backfill to osd.5 from (0'0,0'0] MAX to 280'10022 2024-09-21T12:30:43.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:43 smithi114 ceph-mon[30324]: osdmap e286: 8 total, 8 up, 7 in 2024-09-21T12:30:43.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:43 smithi114 ceph-mon[40867]: 9.13 starting backfill to osd.5 from (0'0,0'0] MAX to 282'9963 2024-09-21T12:30:43.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:43 smithi114 ceph-mon[40867]: 9.19 starting backfill to osd.6 from (0'0,0'0] MAX to 280'9985 2024-09-21T12:30:43.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:43 smithi114 ceph-mon[40867]: 9.17 starting backfill to osd.0 from (0'0,0'0] MAX to 280'10002 2024-09-21T12:30:43.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:43 smithi114 ceph-mon[40867]: 9.15 starting backfill to osd.2 from (0'0,0'0] MAX to 280'9963 2024-09-21T12:30:43.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:43 smithi114 ceph-mon[40867]: 9.14 starting backfill to osd.2 from (0'0,0'0] MAX to 280'10022 2024-09-21T12:30:43.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:43 smithi114 ceph-mon[40867]: 9.16 starting backfill to osd.6 from (0'0,0'0] MAX to 282'10024 2024-09-21T12:30:43.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:43 smithi114 ceph-mon[40867]: 9.13 starting backfill to osd.6 from (0'0,0'0] MAX to 282'9963 2024-09-21T12:30:43.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:43 smithi114 ceph-mon[40867]: 9.17 continuing backfill to osd.1 from (261'6100,280'10002] MIN to 280'10002 2024-09-21T12:30:43.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:43 smithi114 ceph-mon[40867]: 9.15 starting backfill to osd.3 from (0'0,0'0] MAX to 280'9963 2024-09-21T12:30:43.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:43 smithi114 ceph-mon[40867]: 9.14 starting backfill to osd.5 from (0'0,0'0] MAX to 280'10022 2024-09-21T12:30:43.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:43 smithi114 ceph-mon[40867]: osdmap e286: 8 total, 8 up, 7 in 2024-09-21T12:30:43.806 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:43.843 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:43.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:43.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:43.945 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:43.966 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:43.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:43.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:44.068 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:44.100 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:44.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:44.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:44.202 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:44.241 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:44.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:44.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:44.342 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:44.382 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:44.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:44.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:44.484 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:44.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:44.508 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:44.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:44.609 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:44.647 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:44.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:44.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:44.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:44 smithi183 ceph-mon[33277]: pgmap v513: 188 pgs: 2 peering, 4 remapped+peering, 5 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 173 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 344 op/s; 30474/262845 objects misplaced (11.594%); 42 MiB/s, 678 objects/s recovering 2024-09-21T12:30:44.749 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:44.773 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:44.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:44.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:44.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:44 smithi114 ceph-mon[40867]: pgmap v513: 188 pgs: 2 peering, 4 remapped+peering, 5 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 173 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 344 op/s; 30474/262845 objects misplaced (11.594%); 42 MiB/s, 678 objects/s recovering 2024-09-21T12:30:44.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:44 smithi114 ceph-mon[30324]: pgmap v513: 188 pgs: 2 peering, 4 remapped+peering, 5 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 173 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 344 op/s; 30474/262845 objects misplaced (11.594%); 42 MiB/s, 678 objects/s recovering 2024-09-21T12:30:44.874 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:44.911 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:44.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:44.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:45.013 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:45.049 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:45.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:45.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:45.151 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:45.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:45.173 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:45.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:45.274 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:45.299 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:45.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:45.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:45.401 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:45.433 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:45.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:45.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:45.534 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:45.557 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:45.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:45.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:45.659 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:45.688 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:45.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:45.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:45.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:45 smithi183 ceph-mon[33277]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:30:45.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:45 smithi114 ceph-mon[40867]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:30:45.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:45 smithi114 ceph-mon[30324]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:30:45.791 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:45.822 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:45.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:45.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:45.924 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:45.945 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:45.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:45.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:46.046 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:46.076 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:46.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:46.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:46.178 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:46.216 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:46.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:46.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:46.318 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:46.355 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:46.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:46.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:46.457 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:46.478 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:46.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:46.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:46.580 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:46.609 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:46.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:46.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:46.711 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:46.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:46 smithi183 ceph-mon[33277]: pgmap v514: 188 pgs: 2 peering, 4 remapped+peering, 5 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 173 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 254 op/s; 30474/262845 objects misplaced (11.594%); 31 MiB/s, 500 objects/s recovering 2024-09-21T12:30:46.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:46.739 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:46.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:46.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:46 smithi114 ceph-mon[40867]: pgmap v514: 188 pgs: 2 peering, 4 remapped+peering, 5 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 173 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 254 op/s; 30474/262845 objects misplaced (11.594%); 31 MiB/s, 500 objects/s recovering 2024-09-21T12:30:46.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:46 smithi114 ceph-mon[30324]: pgmap v514: 188 pgs: 2 peering, 4 remapped+peering, 5 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 173 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 254 op/s; 30474/262845 objects misplaced (11.594%); 31 MiB/s, 500 objects/s recovering 2024-09-21T12:30:46.840 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:46.863 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:46.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:46.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:46.964 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:46.995 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:46.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:46.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:47.096 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:47.117 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:47.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:47.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:47.218 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:47.252 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:47.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:47.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:47.353 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:47.384 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:47.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:47.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:47.486 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:47.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:47 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:30:47.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:47 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:30:47.489 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:30:47 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:30:47.506 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:47.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:47.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:47.608 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:47.641 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:47.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:47.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:47.676 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:47 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:30:47.676 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:47 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:30:47.676 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:47 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:30:47.676 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:47 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:30:47.742 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:47.781 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:47.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:47.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:47.882 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:47.913 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:47.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:47.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:48.014 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:48.043 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:48.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:48.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:48.145 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:48.174 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:48.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:48.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:48.276 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:48.308 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:48.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:48.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:48.410 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:48.441 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:48.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:48.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:48.543 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:48.568 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:48.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:48.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:48.670 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:48.699 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:48.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:48.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:48.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:48 smithi183 ceph-mon[33277]: pgmap v515: 188 pgs: 7 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 175 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s rd, 0 B/s wr, 920 op/s; 39935/249252 objects misplaced (16.022%); 68 MiB/s, 1.08k objects/s recovering 2024-09-21T12:30:48.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:48 smithi114 ceph-mon[40867]: pgmap v515: 188 pgs: 7 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 175 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s rd, 0 B/s wr, 920 op/s; 39935/249252 objects misplaced (16.022%); 68 MiB/s, 1.08k objects/s recovering 2024-09-21T12:30:48.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:48 smithi114 ceph-mon[30324]: pgmap v515: 188 pgs: 7 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 175 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s rd, 0 B/s wr, 920 op/s; 39935/249252 objects misplaced (16.022%); 68 MiB/s, 1.08k objects/s recovering 2024-09-21T12:30:48.801 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:48.840 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:48.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:48.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:48.942 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:48.983 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:48.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:48.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:49.084 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:49.108 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:49.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:49.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:49.209 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:49.242 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:49.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:49.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:49.344 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:49.379 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:49.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:49.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:49.481 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:49.530 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:49.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:49.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:49.632 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:49.657 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:49.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:49.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:49.759 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:49.780 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:49.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:49.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:49.882 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:49.914 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:49.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:49.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:50.016 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:50.043 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:50.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:50.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:50.145 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:50.179 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:50.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:50.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:50.280 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:50.303 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:50.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:50.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:50.404 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:50.428 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:50.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:50.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:50.530 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:50.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:50.563 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:50.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:50.664 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:50.689 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:50.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:50.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:50.791 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:50.817 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:50.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:50.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:50.919 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:50.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:50.960 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:50.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:50.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:50 smithi183 ceph-mon[33277]: osdmap e287: 8 total, 8 up, 7 in 2024-09-21T12:30:50.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:50 smithi183 ceph-mon[33277]: pgmap v517: 188 pgs: 7 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 175 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 39935/249252 objects misplaced (16.022%); 8.6 MiB/s, 136 objects/s recovering 2024-09-21T12:30:51.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:50 smithi114 ceph-mon[40867]: osdmap e287: 8 total, 8 up, 7 in 2024-09-21T12:30:51.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:50 smithi114 ceph-mon[40867]: pgmap v517: 188 pgs: 7 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 175 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 39935/249252 objects misplaced (16.022%); 8.6 MiB/s, 136 objects/s recovering 2024-09-21T12:30:51.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:50 smithi114 ceph-mon[30324]: osdmap e287: 8 total, 8 up, 7 in 2024-09-21T12:30:51.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:50 smithi114 ceph-mon[30324]: pgmap v517: 188 pgs: 7 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 175 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 39935/249252 objects misplaced (16.022%); 8.6 MiB/s, 136 objects/s recovering 2024-09-21T12:30:51.061 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:51.090 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:51.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:51.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:51.192 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:51.221 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:51.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:51.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:51.322 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:51.349 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:51.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:51.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:51.451 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:51.484 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:51.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:51.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:51.586 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:51.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:51.608 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:51.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:51.709 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:51.747 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:51.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:51.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:51.849 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:51.873 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:51.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:51.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:51.974 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:51.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:51 smithi183 ceph-mon[33277]: osdmap e288: 8 total, 8 up, 7 in 2024-09-21T12:30:52.013 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:52.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:52.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:52.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:51 smithi114 ceph-mon[40867]: osdmap e288: 8 total, 8 up, 7 in 2024-09-21T12:30:52.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:51 smithi114 ceph-mon[30324]: osdmap e288: 8 total, 8 up, 7 in 2024-09-21T12:30:52.115 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:52.136 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:52.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:52.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:52.238 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:52.282 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:52.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:52.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:52.384 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:52.405 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:52.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:52.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:52.506 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:52.546 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:52.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:52.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:52.648 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:52.676 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:52.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:52.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:52.777 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:52.806 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:52.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:52.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:52.908 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:52.934 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:52.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:52.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:52.946 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:30:52 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:30:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:30:52.947 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:52 smithi114 ceph-mon[40867]: pgmap v519: 188 pgs: 6 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 176 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 773 op/s; 36174/244293 objects misplaced (14.808%); 45 MiB/s, 724 objects/s recovering 2024-09-21T12:30:52.947 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:52 smithi114 ceph-mon[40867]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:30:52.947 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:52 smithi114 ceph-mon[30324]: pgmap v519: 188 pgs: 6 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 176 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 773 op/s; 36174/244293 objects misplaced (14.808%); 45 MiB/s, 724 objects/s recovering 2024-09-21T12:30:52.947 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:52 smithi114 ceph-mon[30324]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:30:52.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:52 smithi183 ceph-mon[33277]: pgmap v519: 188 pgs: 6 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 176 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 773 op/s; 36174/244293 objects misplaced (14.808%); 45 MiB/s, 724 objects/s recovering 2024-09-21T12:30:52.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:52 smithi183 ceph-mon[33277]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:30:53.036 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:53.082 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:53.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:53.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:53.184 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:53.225 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:53.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:53.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:53.327 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:53.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:53.363 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:53.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:53.381 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:30:53.381 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:30:53.381 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:30:53.381 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:30:53.381 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:30:53.381 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:30:53.381 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:30:53.382 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:30:53.382 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:30:53.382 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:30:53.382 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:30:53.382 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:30:53.382 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:30:53.382 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:30:53.382 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:30:53.383 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:30:53.464 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:53.491 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:53.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:53.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:53.592 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:53.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:53.622 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:53.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:53.723 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:53.753 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:53.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:53.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:53.854 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:53.895 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:53.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:53.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:53.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:53 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3748684468' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:30:53.997 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:54.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:53 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3748684468' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:30:54.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:53 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3748684468' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:30:54.025 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:54.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:54.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:54.127 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:54.166 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:54.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:54.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:54.268 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:54.299 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:54.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:54.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:54.401 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:54.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:54.438 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:54.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:54.539 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:54.569 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:54.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:54.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:54.671 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:54.704 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:54.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:54.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:54.805 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:54.835 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:54.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:54.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:54.937 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:54.949 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:54 smithi114 ceph-mon[30324]: pgmap v520: 188 pgs: 5 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 176 active+clean; 4.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 31308/235527 objects misplaced (13.293%); 64 MiB/s, 1.03k objects/s recovering 2024-09-21T12:30:54.949 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:54 smithi114 ceph-mon[40867]: pgmap v520: 188 pgs: 5 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 176 active+clean; 4.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 31308/235527 objects misplaced (13.293%); 64 MiB/s, 1.03k objects/s recovering 2024-09-21T12:30:54.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:54 smithi183 ceph-mon[33277]: pgmap v520: 188 pgs: 5 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 176 active+clean; 4.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 31308/235527 objects misplaced (13.293%); 64 MiB/s, 1.03k objects/s recovering 2024-09-21T12:30:54.999 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:54.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:55.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:55.101 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:55.138 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:55.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:55.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:55.239 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:55.244 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T12:30:55.278 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:55.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:55.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:55.380 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:55.410 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:55.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:55.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:55.512 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:55.543 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:55.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:55.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:55.644 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:55.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:55.691 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:55.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:55.794 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:55.829 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:55.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:55.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:55.931 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:55.959 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:55.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:55.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:56.061 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:56.089 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:56.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:56.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:56.190 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:56.226 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:56.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:56.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:56.327 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:56.354 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:56.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:56.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:56.456 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:56.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:56.494 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:56.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:56.595 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:56.634 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:56.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:56.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:56.735 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:56.773 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:56.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:56.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:56.875 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:56.896 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:56.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:56.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:56.997 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:57.002 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T12:30:57.037 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:57.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:57.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:57.138 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:57.162 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:57.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:57.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:57.214 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:56 smithi183 ceph-mon[33277]: pgmap v521: 188 pgs: 5 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 176 active+clean; 4.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 572 op/s; 31308/235527 objects misplaced (13.293%); 31 MiB/s, 498 objects/s recovering 2024-09-21T12:30:57.264 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:57.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:56 smithi114 ceph-mon[30324]: pgmap v521: 188 pgs: 5 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 176 active+clean; 4.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 572 op/s; 31308/235527 objects misplaced (13.293%); 31 MiB/s, 498 objects/s recovering 2024-09-21T12:30:57.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:56 smithi114 ceph-mon[40867]: pgmap v521: 188 pgs: 5 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 176 active+clean; 4.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 572 op/s; 31308/235527 objects misplaced (13.293%); 31 MiB/s, 498 objects/s recovering 2024-09-21T12:30:57.285 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:57.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:57.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:57.386 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:57.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:57.434 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:57.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:30:57.488 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:30:57 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:30:57.534 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:57.582 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:57.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:57.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:57.684 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:57.722 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:57.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:57.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:57.824 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:57.853 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:57.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:57.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:57.955 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:57.984 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:57.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:57.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:58.086 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:58.111 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:58.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:58.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:58.213 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:58.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:57 smithi183 ceph-mon[33277]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:30:58.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:57 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:30:58.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:57 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:30:58.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:57 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:30:58.251 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:58.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:58.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:58.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:57 smithi114 ceph-mon[40867]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:30:58.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:57 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:30:58.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:57 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:30:58.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:57 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:30:58.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:57 smithi114 ceph-mon[30324]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:30:58.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:57 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:30:58.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:57 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:30:58.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:57 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:30:58.354 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:58.384 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:58.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:58.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:58.486 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:58.516 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:58.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:58.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:58.618 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:58.643 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:58.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:58.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:58.745 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:58.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:58.779 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:58.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:58.880 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:58.907 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:58.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:58.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:59.009 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:30:59.047 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:59.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:59.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:30:59.149 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:30:59.179 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:59.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:59.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:30:59.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:59 smithi114 ceph-mon[40867]: pgmap v522: 188 pgs: 5 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 176 active+clean; 4.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 25648/220779 objects misplaced (11.617%); 48 MiB/s, 767 objects/s recovering 2024-09-21T12:30:59.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:30:59 smithi114 ceph-mon[40867]: osdmap e289: 8 total, 8 up, 7 in 2024-09-21T12:30:59.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:59 smithi114 ceph-mon[30324]: pgmap v522: 188 pgs: 5 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 176 active+clean; 4.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 25648/220779 objects misplaced (11.617%); 48 MiB/s, 767 objects/s recovering 2024-09-21T12:30:59.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:30:59 smithi114 ceph-mon[30324]: osdmap e289: 8 total, 8 up, 7 in 2024-09-21T12:30:59.281 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:30:59.310 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:59.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:59.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:30:59.412 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:30:59.438 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:59.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:59.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:30:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:59 smithi183 ceph-mon[33277]: pgmap v522: 188 pgs: 5 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 176 active+clean; 4.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 25648/220779 objects misplaced (11.617%); 48 MiB/s, 767 objects/s recovering 2024-09-21T12:30:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:30:59 smithi183 ceph-mon[33277]: osdmap e289: 8 total, 8 up, 7 in 2024-09-21T12:30:59.540 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:30:59.567 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:59.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:59.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:30:59.668 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:30:59.697 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:30:59.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:59.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:30:59.799 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:30:59.835 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:59.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:59.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:30:59.937 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:30:59.972 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:30:59.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:30:59.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:00.074 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:00.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:00.098 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:00.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:00.199 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:00.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:00.220 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:00.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:00.321 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:00.345 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:00.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:00.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:00.447 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:00.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:00.483 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:00.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:00.584 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:00.612 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:00.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:00.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:00.713 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:00.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:00.765 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:00.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:00.866 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:00.891 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:00.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:00.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:00.993 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:01.036 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:01.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:01.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:01.138 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:01.174 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:01.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:01.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:01.266 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:01 smithi114 ceph-mon[30324]: pgmap v524: 188 pgs: 5 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 176 active+clean; 4.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.16k op/s; 25298/217401 objects misplaced (11.637%); 43 MiB/s, 685 objects/s recovering 2024-09-21T12:31:01.267 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:01 smithi114 ceph-mon[40867]: pgmap v524: 188 pgs: 5 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 176 active+clean; 4.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.16k op/s; 25298/217401 objects misplaced (11.637%); 43 MiB/s, 685 objects/s recovering 2024-09-21T12:31:01.276 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:01.311 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:01.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:01.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:01.413 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:01.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:01.446 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:01.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:01 smithi183 ceph-mon[33277]: pgmap v524: 188 pgs: 5 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 176 active+clean; 4.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.16k op/s; 25298/217401 objects misplaced (11.637%); 43 MiB/s, 685 objects/s recovering 2024-09-21T12:31:01.548 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:01.578 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:01.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:01.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:01.680 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:01.707 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:01.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:01.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:01.809 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:01.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:01.832 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:01.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:01.933 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:01.956 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:01.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:01.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:01.974 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:31:01.977 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:31:02.057 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:02.086 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:02.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:02.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:02.188 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:02.209 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:02.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:02.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:02.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:02 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:31:02.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:02 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:31:02.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:02 smithi114 ceph-mon[30324]: osdmap e290: 8 total, 8 up, 7 in 2024-09-21T12:31:02.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:02 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:31:02.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:02 smithi114 ceph-mon[30324]: osdmap e291: 8 total, 8 up, 7 in 2024-09-21T12:31:02.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:02 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:31:02.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:02 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:31:02.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:02 smithi114 ceph-mon[40867]: osdmap e290: 8 total, 8 up, 7 in 2024-09-21T12:31:02.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:02 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:31:02.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:02 smithi114 ceph-mon[40867]: osdmap e291: 8 total, 8 up, 7 in 2024-09-21T12:31:02.311 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:02.336 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:02.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:02.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:02.438 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:02.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:02 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:31:02.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:02 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:31:02.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:02 smithi183 ceph-mon[33277]: osdmap e290: 8 total, 8 up, 7 in 2024-09-21T12:31:02.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:02 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:31:02.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:02 smithi183 ceph-mon[33277]: osdmap e291: 8 total, 8 up, 7 in 2024-09-21T12:31:02.490 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:02.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:02.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:02.591 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:02.614 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:02.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:02.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:02.716 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:02.744 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:02.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:02.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:02.774 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:31:02 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:31:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:31:02.846 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:02.874 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:02.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:02.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:02.926 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:31:02.926 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:31:02.926 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 0 2024-09-21T12:31:02.926 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:31:02.976 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:02.997 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:02.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:02.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:03.098 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:03.125 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:03.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:03.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:03.227 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:03.254 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:03.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:03.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:03.299 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:03 smithi114 ceph-mon[40867]: pgmap v526: 188 pgs: 5 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 176 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 976 op/s; 23611/212112 objects misplaced (11.131%); 22 MiB/s, 359 objects/s recovering 2024-09-21T12:31:03.299 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:03 smithi114 ceph-mon[40867]: osdmap e292: 8 total, 8 up, 7 in 2024-09-21T12:31:03.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:03 smithi114 ceph-mon[30324]: pgmap v526: 188 pgs: 5 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 176 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 976 op/s; 23611/212112 objects misplaced (11.131%); 22 MiB/s, 359 objects/s recovering 2024-09-21T12:31:03.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:03 smithi114 ceph-mon[30324]: osdmap e292: 8 total, 8 up, 7 in 2024-09-21T12:31:03.356 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:03.379 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:03.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:03.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:03.481 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:03.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:03 smithi183 ceph-mon[33277]: pgmap v526: 188 pgs: 5 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 176 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 976 op/s; 23611/212112 objects misplaced (11.131%); 22 MiB/s, 359 objects/s recovering 2024-09-21T12:31:03.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:03 smithi183 ceph-mon[33277]: osdmap e292: 8 total, 8 up, 7 in 2024-09-21T12:31:03.503 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:03.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:03.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:03.605 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:03.625 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:03.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:03.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:03.727 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:03.749 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:03.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:03.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:03.755 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T12:31:03.851 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:03.872 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:03.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:03.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:03.926 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:31:03.974 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:03.994 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:03.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:03.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:04.096 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:04.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:04.149 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:04.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:04.250 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:04.285 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:04.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:04.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:04.387 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:04.411 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:04.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:04.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:04.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:04 smithi183 ceph-mon[33277]: 9.1 scrub starts 2024-09-21T12:31:04.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:04 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:31:04.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:04 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:31:04.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:04 smithi183 ceph-mon[33277]: 1.10 scrub starts 2024-09-21T12:31:04.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:04 smithi183 ceph-mon[33277]: 1.10 scrub ok 2024-09-21T12:31:04.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:04 smithi183 ceph-mon[33277]: osdmap e293: 8 total, 8 up, 7 in 2024-09-21T12:31:04.513 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:04.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:04 smithi114 ceph-mon[40867]: 9.1 scrub starts 2024-09-21T12:31:04.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:04 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:31:04.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:04 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:31:04.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:04 smithi114 ceph-mon[40867]: 1.10 scrub starts 2024-09-21T12:31:04.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:04 smithi114 ceph-mon[40867]: 1.10 scrub ok 2024-09-21T12:31:04.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:04 smithi114 ceph-mon[40867]: osdmap e293: 8 total, 8 up, 7 in 2024-09-21T12:31:04.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:04 smithi114 ceph-mon[30324]: 9.1 scrub starts 2024-09-21T12:31:04.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:04 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:31:04.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:04 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:31:04.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:04 smithi114 ceph-mon[30324]: 1.10 scrub starts 2024-09-21T12:31:04.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:04 smithi114 ceph-mon[30324]: 1.10 scrub ok 2024-09-21T12:31:04.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:04 smithi114 ceph-mon[30324]: osdmap e293: 8 total, 8 up, 7 in 2024-09-21T12:31:04.534 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:04.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:04.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:04.636 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:04.658 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:04.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:04.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:04.760 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:04.780 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:04.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:04.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:04.882 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:04.902 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:04.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:04.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:05.004 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:05.032 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:05.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:05.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:05.133 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:05.175 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:05.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:05.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:05.276 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:05.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:05.315 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:05.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:05.390 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:05 smithi183 ceph-mon[33277]: 9.1 scrub ok 2024-09-21T12:31:05.390 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:05 smithi183 ceph-mon[33277]: pgmap v529: 188 pgs: 1 peering, 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 177 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 885 B/s rd, 0 B/s wr, 798 op/s; 19553/203559 objects misplaced (9.606%); 28 MiB/s, 446 objects/s recovering 2024-09-21T12:31:05.390 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:05 smithi183 ceph-mon[33277]: 1.b scrub starts 2024-09-21T12:31:05.390 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:05 smithi183 ceph-mon[33277]: 1.b scrub ok 2024-09-21T12:31:05.390 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:05 smithi183 ceph-mon[33277]: 4.14 scrub starts 2024-09-21T12:31:05.390 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:05 smithi183 ceph-mon[33277]: 4.14 scrub ok 2024-09-21T12:31:05.390 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:05 smithi183 ceph-mon[33277]: 1.7 scrub starts 2024-09-21T12:31:05.390 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:05 smithi183 ceph-mon[33277]: 1.7 scrub ok 2024-09-21T12:31:05.390 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:05 smithi183 ceph-mon[33277]: 9.9 scrub starts 2024-09-21T12:31:05.390 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:05 smithi183 ceph-mon[33277]: osdmap e294: 8 total, 8 up, 7 in 2024-09-21T12:31:05.417 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:05.445 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:05.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:05.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:05.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:05 smithi114 ceph-mon[30324]: 9.1 scrub ok 2024-09-21T12:31:05.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:05 smithi114 ceph-mon[30324]: pgmap v529: 188 pgs: 1 peering, 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 177 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 885 B/s rd, 0 B/s wr, 798 op/s; 19553/203559 objects misplaced (9.606%); 28 MiB/s, 446 objects/s recovering 2024-09-21T12:31:05.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:05 smithi114 ceph-mon[30324]: 1.b scrub starts 2024-09-21T12:31:05.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:05 smithi114 ceph-mon[30324]: 1.b scrub ok 2024-09-21T12:31:05.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:05 smithi114 ceph-mon[30324]: 4.14 scrub starts 2024-09-21T12:31:05.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:05 smithi114 ceph-mon[30324]: 4.14 scrub ok 2024-09-21T12:31:05.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:05 smithi114 ceph-mon[30324]: 1.7 scrub starts 2024-09-21T12:31:05.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:05 smithi114 ceph-mon[30324]: 1.7 scrub ok 2024-09-21T12:31:05.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:05 smithi114 ceph-mon[30324]: 9.9 scrub starts 2024-09-21T12:31:05.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:05 smithi114 ceph-mon[30324]: osdmap e294: 8 total, 8 up, 7 in 2024-09-21T12:31:05.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:05 smithi114 ceph-mon[40867]: 9.1 scrub ok 2024-09-21T12:31:05.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:05 smithi114 ceph-mon[40867]: pgmap v529: 188 pgs: 1 peering, 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 177 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 885 B/s rd, 0 B/s wr, 798 op/s; 19553/203559 objects misplaced (9.606%); 28 MiB/s, 446 objects/s recovering 2024-09-21T12:31:05.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:05 smithi114 ceph-mon[40867]: 1.b scrub starts 2024-09-21T12:31:05.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:05 smithi114 ceph-mon[40867]: 1.b scrub ok 2024-09-21T12:31:05.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:05 smithi114 ceph-mon[40867]: 4.14 scrub starts 2024-09-21T12:31:05.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:05 smithi114 ceph-mon[40867]: 4.14 scrub ok 2024-09-21T12:31:05.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:05 smithi114 ceph-mon[40867]: 1.7 scrub starts 2024-09-21T12:31:05.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:05 smithi114 ceph-mon[40867]: 1.7 scrub ok 2024-09-21T12:31:05.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:05 smithi114 ceph-mon[40867]: 9.9 scrub starts 2024-09-21T12:31:05.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:05 smithi114 ceph-mon[40867]: osdmap e294: 8 total, 8 up, 7 in 2024-09-21T12:31:05.547 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:05.578 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:05.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:05.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:05.680 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:05.709 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:05.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:05.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:05.811 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:05.857 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:05.859 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:31:05.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:05.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:05.961 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:05.990 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:05.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:05.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:06.076 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-21T12:31:06.092 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:06.117 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:06.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:06.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:06.219 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:06.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:06.252 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:06.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:06.267 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T12:31:06.267 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:31:06.267 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-21T12:31:06.267 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T12:31:06.267 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T12:31:06.268 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-09-21T12:31:06.328 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:06 smithi183 ceph-mon[33277]: 9.9 scrub ok 2024-09-21T12:31:06.328 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:06 smithi183 ceph-mon[33277]: 9.4 scrub starts 2024-09-21T12:31:06.328 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:06 smithi183 ceph-mon[33277]: 9.3 scrub starts 2024-09-21T12:31:06.328 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:06 smithi183 ceph-mon[33277]: 9.4 scrub ok 2024-09-21T12:31:06.328 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:06 smithi183 ceph-mon[33277]: 9.3 scrub ok 2024-09-21T12:31:06.328 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:06 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2843063439' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:31:06.353 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:06.396 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:06.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:06.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:06.498 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:06.519 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:06.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:06.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:06.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:06 smithi114 ceph-mon[40867]: 9.9 scrub ok 2024-09-21T12:31:06.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:06 smithi114 ceph-mon[40867]: 9.4 scrub starts 2024-09-21T12:31:06.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:06 smithi114 ceph-mon[40867]: 9.3 scrub starts 2024-09-21T12:31:06.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:06 smithi114 ceph-mon[40867]: 9.4 scrub ok 2024-09-21T12:31:06.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:06 smithi114 ceph-mon[40867]: 9.3 scrub ok 2024-09-21T12:31:06.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:06 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2843063439' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:31:06.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:06 smithi114 ceph-mon[30324]: 9.9 scrub ok 2024-09-21T12:31:06.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:06 smithi114 ceph-mon[30324]: 9.4 scrub starts 2024-09-21T12:31:06.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:06 smithi114 ceph-mon[30324]: 9.3 scrub starts 2024-09-21T12:31:06.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:06 smithi114 ceph-mon[30324]: 9.4 scrub ok 2024-09-21T12:31:06.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:06 smithi114 ceph-mon[30324]: 9.3 scrub ok 2024-09-21T12:31:06.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:06 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2843063439' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:31:06.621 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:06.629 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:31:06.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:06.664 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:06.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:06.765 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:06.809 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:06.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:06.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:06.911 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:06.947 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:06.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:06.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:07.049 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:07.071 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:07.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:07.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:07.173 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:07.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:07.212 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:07.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:07.313 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:07.328 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:07 smithi114 ceph-mon[40867]: pgmap v532: 188 pgs: 1 peering, 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 177 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 712 op/s; 19553/203559 objects misplaced (9.606%); 31 MiB/s, 501 objects/s recovering 2024-09-21T12:31:07.328 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:07 smithi114 ceph-mon[40867]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:31:07.328 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:07 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2843063439' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:31:07.329 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:07 smithi114 ceph-mon[40867]: osdmap e295: 8 total, 8 up, 7 in 2024-09-21T12:31:07.329 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:07 smithi114 ceph-mon[40867]: 9.11 scrub starts 2024-09-21T12:31:07.329 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:07 smithi114 ceph-mon[40867]: osdmap e296: 8 total, 8 up, 7 in 2024-09-21T12:31:07.329 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:07 smithi114 ceph-mon[30324]: pgmap v532: 188 pgs: 1 peering, 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 177 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 712 op/s; 19553/203559 objects misplaced (9.606%); 31 MiB/s, 501 objects/s recovering 2024-09-21T12:31:07.329 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:07 smithi114 ceph-mon[30324]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:31:07.329 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:07 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2843063439' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:31:07.329 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:07 smithi114 ceph-mon[30324]: osdmap e295: 8 total, 8 up, 7 in 2024-09-21T12:31:07.329 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:07 smithi114 ceph-mon[30324]: 9.11 scrub starts 2024-09-21T12:31:07.329 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:07 smithi114 ceph-mon[30324]: osdmap e296: 8 total, 8 up, 7 in 2024-09-21T12:31:07.335 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:07.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:07.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:07.366 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:31:07 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:31:07.366 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:07 smithi183 ceph-mon[33277]: pgmap v532: 188 pgs: 1 peering, 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 177 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 712 op/s; 19553/203559 objects misplaced (9.606%); 31 MiB/s, 501 objects/s recovering 2024-09-21T12:31:07.366 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:07 smithi183 ceph-mon[33277]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:31:07.366 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:07 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2843063439' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:31:07.366 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:07 smithi183 ceph-mon[33277]: osdmap e295: 8 total, 8 up, 7 in 2024-09-21T12:31:07.366 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:07 smithi183 ceph-mon[33277]: 9.11 scrub starts 2024-09-21T12:31:07.366 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:07 smithi183 ceph-mon[33277]: osdmap e296: 8 total, 8 up, 7 in 2024-09-21T12:31:07.436 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:07.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:07.473 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:07.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:07.526 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:31:07.526 INFO:teuthology.orchestra.run.smithi183.stdout: "success": "" 2024-09-21T12:31:07.526 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:31:07.574 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:07.599 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:07.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:07.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:07.701 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:07.739 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:07.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:07.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:07.841 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:07.864 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:07.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:07.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:07.966 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:08.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:08.009 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:08.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:08.110 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:08.131 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:08.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:08.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:08.233 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:08.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:08 smithi183 ceph-mon[33277]: 9.16 scrub starts 2024-09-21T12:31:08.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:08 smithi183 ceph-mon[33277]: 9.11 scrub ok 2024-09-21T12:31:08.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:08 smithi183 ceph-mon[33277]: 4.11 scrub starts 2024-09-21T12:31:08.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:08 smithi183 ceph-mon[33277]: 4.11 scrub ok 2024-09-21T12:31:08.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:08 smithi183 ceph-mon[33277]: 9.13 scrub starts 2024-09-21T12:31:08.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:08 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:08.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:08 smithi183 ceph-mon[33277]: 5.15 scrub starts 2024-09-21T12:31:08.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:08 smithi183 ceph-mon[33277]: 5.15 scrub ok 2024-09-21T12:31:08.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:08 smithi183 ceph-mon[33277]: 9.13 scrub ok 2024-09-21T12:31:08.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:08 smithi183 ceph-mon[33277]: 9.16 scrub ok 2024-09-21T12:31:08.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:08.269 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:08.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:08.371 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:08.408 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:08.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:08.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:08.510 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:08.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:08 smithi114 ceph-mon[30324]: 9.16 scrub starts 2024-09-21T12:31:08.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:08 smithi114 ceph-mon[30324]: 9.11 scrub ok 2024-09-21T12:31:08.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:08 smithi114 ceph-mon[30324]: 4.11 scrub starts 2024-09-21T12:31:08.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:08 smithi114 ceph-mon[30324]: 4.11 scrub ok 2024-09-21T12:31:08.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:08 smithi114 ceph-mon[30324]: 9.13 scrub starts 2024-09-21T12:31:08.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:08 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:08.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:08 smithi114 ceph-mon[30324]: 5.15 scrub starts 2024-09-21T12:31:08.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:08 smithi114 ceph-mon[30324]: 5.15 scrub ok 2024-09-21T12:31:08.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:08 smithi114 ceph-mon[30324]: 9.13 scrub ok 2024-09-21T12:31:08.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:08 smithi114 ceph-mon[30324]: 9.16 scrub ok 2024-09-21T12:31:08.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:08 smithi114 ceph-mon[40867]: 9.16 scrub starts 2024-09-21T12:31:08.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:08 smithi114 ceph-mon[40867]: 9.11 scrub ok 2024-09-21T12:31:08.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:08 smithi114 ceph-mon[40867]: 4.11 scrub starts 2024-09-21T12:31:08.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:08 smithi114 ceph-mon[40867]: 4.11 scrub ok 2024-09-21T12:31:08.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:08 smithi114 ceph-mon[40867]: 9.13 scrub starts 2024-09-21T12:31:08.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:08 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:08.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:08 smithi114 ceph-mon[40867]: 5.15 scrub starts 2024-09-21T12:31:08.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:08 smithi114 ceph-mon[40867]: 5.15 scrub ok 2024-09-21T12:31:08.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:08 smithi114 ceph-mon[40867]: 9.13 scrub ok 2024-09-21T12:31:08.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:08 smithi114 ceph-mon[40867]: 9.16 scrub ok 2024-09-21T12:31:08.546 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:08.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:08.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:08.648 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:08.675 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:08.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:08.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:08.777 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:08.804 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:08.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:08.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:08.906 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:08.927 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:08.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:08.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:08.947 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:31:08.947 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:31:08.947 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:31:08.947 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:31:08.948 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:31:09.028 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:09.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:09.066 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:09.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:09.167 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:09.194 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:09.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:09.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:09.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:09 smithi114 ceph-mon[40867]: 3.10 scrub starts 2024-09-21T12:31:09.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:09 smithi114 ceph-mon[40867]: 3.10 scrub ok 2024-09-21T12:31:09.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:09 smithi114 ceph-mon[40867]: pgmap v535: 188 pgs: 2 active+clean+scrubbing, 7 active+remapped+backfill_toofull, 179 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s rd, 0 op/s; 16889/194658 objects misplaced (8.676%) 2024-09-21T12:31:09.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:09 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:31:09.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:09 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:31:09.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:09 smithi114 ceph-mon[40867]: osdmap e297: 8 total, 8 up, 7 in 2024-09-21T12:31:09.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:09 smithi114 ceph-mon[40867]: 9.17 scrub starts 2024-09-21T12:31:09.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:09 smithi114 ceph-mon[40867]: 1.1b scrub starts 2024-09-21T12:31:09.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:09 smithi114 ceph-mon[40867]: 1.1b scrub ok 2024-09-21T12:31:09.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:09 smithi114 ceph-mon[40867]: 9.5 scrub starts 2024-09-21T12:31:09.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:09 smithi114 ceph-mon[40867]: 9.6 scrub starts 2024-09-21T12:31:09.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:09 smithi114 ceph-mon[40867]: 9.17 scrub ok 2024-09-21T12:31:09.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:09 smithi114 ceph-mon[30324]: 3.10 scrub starts 2024-09-21T12:31:09.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:09 smithi114 ceph-mon[30324]: 3.10 scrub ok 2024-09-21T12:31:09.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:09 smithi114 ceph-mon[30324]: pgmap v535: 188 pgs: 2 active+clean+scrubbing, 7 active+remapped+backfill_toofull, 179 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s rd, 0 op/s; 16889/194658 objects misplaced (8.676%) 2024-09-21T12:31:09.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:09 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:31:09.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:09 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:31:09.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:09 smithi114 ceph-mon[30324]: osdmap e297: 8 total, 8 up, 7 in 2024-09-21T12:31:09.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:09 smithi114 ceph-mon[30324]: 9.17 scrub starts 2024-09-21T12:31:09.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:09 smithi114 ceph-mon[30324]: 1.1b scrub starts 2024-09-21T12:31:09.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:09 smithi114 ceph-mon[30324]: 1.1b scrub ok 2024-09-21T12:31:09.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:09 smithi114 ceph-mon[30324]: 9.5 scrub starts 2024-09-21T12:31:09.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:09 smithi114 ceph-mon[30324]: 9.6 scrub starts 2024-09-21T12:31:09.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:09 smithi114 ceph-mon[30324]: 9.17 scrub ok 2024-09-21T12:31:09.296 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:09.324 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:09.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:09.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:09.427 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:09.456 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:09.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:09.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:09.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:09 smithi183 ceph-mon[33277]: 3.10 scrub starts 2024-09-21T12:31:09.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:09 smithi183 ceph-mon[33277]: 3.10 scrub ok 2024-09-21T12:31:09.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:09 smithi183 ceph-mon[33277]: pgmap v535: 188 pgs: 2 active+clean+scrubbing, 7 active+remapped+backfill_toofull, 179 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s rd, 0 op/s; 16889/194658 objects misplaced (8.676%) 2024-09-21T12:31:09.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:09 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:31:09.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:09 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:31:09.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:09 smithi183 ceph-mon[33277]: osdmap e297: 8 total, 8 up, 7 in 2024-09-21T12:31:09.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:09 smithi183 ceph-mon[33277]: 9.17 scrub starts 2024-09-21T12:31:09.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:09 smithi183 ceph-mon[33277]: 1.1b scrub starts 2024-09-21T12:31:09.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:09 smithi183 ceph-mon[33277]: 1.1b scrub ok 2024-09-21T12:31:09.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:09 smithi183 ceph-mon[33277]: 9.5 scrub starts 2024-09-21T12:31:09.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:09 smithi183 ceph-mon[33277]: 9.6 scrub starts 2024-09-21T12:31:09.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:09 smithi183 ceph-mon[33277]: 9.17 scrub ok 2024-09-21T12:31:09.559 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:09.580 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:09.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:09.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:09.682 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:09.708 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:09.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:09.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:09.810 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:09.831 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:09.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:09.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:09.933 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:09.965 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:09.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:09.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:10.066 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:10.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:10.093 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:10.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:10.154 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:31:10.194 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:10.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:10.219 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:10.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:10.306 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:10 smithi114 ceph-mon[40867]: 4.a scrub starts 2024-09-21T12:31:10.306 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:10 smithi114 ceph-mon[40867]: 4.a scrub ok 2024-09-21T12:31:10.306 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:10 smithi114 ceph-mon[40867]: 9.5 scrub ok 2024-09-21T12:31:10.306 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:10 smithi114 ceph-mon[40867]: 9.6 scrub ok 2024-09-21T12:31:10.306 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:10 smithi114 ceph-mon[30324]: 4.a scrub starts 2024-09-21T12:31:10.306 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:10 smithi114 ceph-mon[30324]: 4.a scrub ok 2024-09-21T12:31:10.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:10 smithi114 ceph-mon[30324]: 9.5 scrub ok 2024-09-21T12:31:10.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:10 smithi114 ceph-mon[30324]: 9.6 scrub ok 2024-09-21T12:31:10.320 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:10.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:10.353 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:10.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:10.454 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:10.475 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:10.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:10.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:10.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:10 smithi183 ceph-mon[33277]: 4.a scrub starts 2024-09-21T12:31:10.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:10 smithi183 ceph-mon[33277]: 4.a scrub ok 2024-09-21T12:31:10.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:10 smithi183 ceph-mon[33277]: 9.5 scrub ok 2024-09-21T12:31:10.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:10 smithi183 ceph-mon[33277]: 9.6 scrub ok 2024-09-21T12:31:10.577 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:10.605 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:10.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:10.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:10.681 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T12:31:10.706 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:10.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:10.774 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:10.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:10.875 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:10.922 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:10.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:10.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:11.024 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:11.050 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:11.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:11.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:11.152 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:11.181 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:11.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:11.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:11.283 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:11.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:11 smithi114 ceph-mon[40867]: 4.12 scrub starts 2024-09-21T12:31:11.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:11 smithi114 ceph-mon[40867]: 4.12 scrub ok 2024-09-21T12:31:11.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:11 smithi114 ceph-mon[40867]: pgmap v537: 188 pgs: 2 active+clean+scrubbing, 7 active+remapped+backfill_toofull, 179 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s rd, 0 op/s; 16603/192153 objects misplaced (8.641%) 2024-09-21T12:31:11.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:11 smithi114 ceph-mon[40867]: 9.8 scrub starts 2024-09-21T12:31:11.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:11 smithi114 ceph-mon[40867]: 1.f scrub starts 2024-09-21T12:31:11.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:11 smithi114 ceph-mon[40867]: 9.0 scrub starts 2024-09-21T12:31:11.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:11 smithi114 ceph-mon[40867]: 1.f scrub ok 2024-09-21T12:31:11.288 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:11 smithi114 ceph-mon[30324]: 4.12 scrub starts 2024-09-21T12:31:11.288 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:11 smithi114 ceph-mon[30324]: 4.12 scrub ok 2024-09-21T12:31:11.288 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:11 smithi114 ceph-mon[30324]: pgmap v537: 188 pgs: 2 active+clean+scrubbing, 7 active+remapped+backfill_toofull, 179 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s rd, 0 op/s; 16603/192153 objects misplaced (8.641%) 2024-09-21T12:31:11.288 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:11 smithi114 ceph-mon[30324]: 9.8 scrub starts 2024-09-21T12:31:11.288 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:11 smithi114 ceph-mon[30324]: 1.f scrub starts 2024-09-21T12:31:11.288 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:11 smithi114 ceph-mon[30324]: 9.0 scrub starts 2024-09-21T12:31:11.288 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:11 smithi114 ceph-mon[30324]: 1.f scrub ok 2024-09-21T12:31:11.319 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:11.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:11.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:11.343 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:31:11.362 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:31:11.362 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:31:11.389 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:31:11.389 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:31:11.415 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:31:11.415 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:31:11.422 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:11.440 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:31:11.440 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:31:11.446 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:11.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:11.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:11.463 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:31:11.464 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:31:11.485 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:31:11.485 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:31:11.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:11 smithi183 ceph-mon[33277]: 4.12 scrub starts 2024-09-21T12:31:11.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:11 smithi183 ceph-mon[33277]: 4.12 scrub ok 2024-09-21T12:31:11.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:11 smithi183 ceph-mon[33277]: pgmap v537: 188 pgs: 2 active+clean+scrubbing, 7 active+remapped+backfill_toofull, 179 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s rd, 0 op/s; 16603/192153 objects misplaced (8.641%) 2024-09-21T12:31:11.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:11 smithi183 ceph-mon[33277]: 9.8 scrub starts 2024-09-21T12:31:11.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:11 smithi183 ceph-mon[33277]: 1.f scrub starts 2024-09-21T12:31:11.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:11 smithi183 ceph-mon[33277]: 9.0 scrub starts 2024-09-21T12:31:11.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:11 smithi183 ceph-mon[33277]: 1.f scrub ok 2024-09-21T12:31:11.503 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:31:11.504 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:31:11.521 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:31:11.521 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:31:11.548 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:11.576 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:11.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:11.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:11.678 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:11.701 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:11.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:11.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:11.803 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:11.827 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:11.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:11.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:11.928 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:11.950 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:11.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:11.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:12.051 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:12.083 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T12:31:12.087 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:12.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:12.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:12.189 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:12.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:12.237 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:12.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:12.338 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:12.368 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:12.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:12.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:12.469 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:12 smithi183 ceph-mon[33277]: 9.12 scrub starts 2024-09-21T12:31:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:12 smithi183 ceph-mon[33277]: 5.16 scrub starts 2024-09-21T12:31:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:12 smithi183 ceph-mon[33277]: 5.16 scrub ok 2024-09-21T12:31:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:12 smithi183 ceph-mon[33277]: osdmap e298: 8 total, 8 up, 7 in 2024-09-21T12:31:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:12 smithi183 ceph-mon[33277]: 9.8 scrub ok 2024-09-21T12:31:12.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:12 smithi183 ceph-mon[33277]: 9.0 scrub ok 2024-09-21T12:31:12.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:12 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1822371425' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:31:12.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:12 smithi183 ceph-mon[33277]: 9.12 scrub ok 2024-09-21T12:31:12.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:12 smithi183 ceph-mon[33277]: 4.1e scrub starts 2024-09-21T12:31:12.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:12 smithi183 ceph-mon[33277]: 4.1e scrub ok 2024-09-21T12:31:12.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:12 smithi183 ceph-mon[33277]: 1.12 scrub starts 2024-09-21T12:31:12.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:12 smithi183 ceph-mon[33277]: 1.12 scrub ok 2024-09-21T12:31:12.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:12 smithi183 ceph-mon[33277]: osdmap e299: 8 total, 8 up, 7 in 2024-09-21T12:31:12.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:12 smithi114 ceph-mon[40867]: 9.12 scrub starts 2024-09-21T12:31:12.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:12 smithi114 ceph-mon[40867]: 5.16 scrub starts 2024-09-21T12:31:12.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:12 smithi114 ceph-mon[40867]: 5.16 scrub ok 2024-09-21T12:31:12.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:12 smithi114 ceph-mon[40867]: osdmap e298: 8 total, 8 up, 7 in 2024-09-21T12:31:12.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:12 smithi114 ceph-mon[40867]: 9.8 scrub ok 2024-09-21T12:31:12.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:12 smithi114 ceph-mon[40867]: 9.0 scrub ok 2024-09-21T12:31:12.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:12 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1822371425' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:31:12.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:12 smithi114 ceph-mon[40867]: 9.12 scrub ok 2024-09-21T12:31:12.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:12 smithi114 ceph-mon[40867]: 4.1e scrub starts 2024-09-21T12:31:12.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:12 smithi114 ceph-mon[40867]: 4.1e scrub ok 2024-09-21T12:31:12.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:12 smithi114 ceph-mon[40867]: 1.12 scrub starts 2024-09-21T12:31:12.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:12 smithi114 ceph-mon[40867]: 1.12 scrub ok 2024-09-21T12:31:12.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:12 smithi114 ceph-mon[40867]: osdmap e299: 8 total, 8 up, 7 in 2024-09-21T12:31:12.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:12 smithi114 ceph-mon[30324]: 9.12 scrub starts 2024-09-21T12:31:12.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:12 smithi114 ceph-mon[30324]: 5.16 scrub starts 2024-09-21T12:31:12.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:12 smithi114 ceph-mon[30324]: 5.16 scrub ok 2024-09-21T12:31:12.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:12 smithi114 ceph-mon[30324]: osdmap e298: 8 total, 8 up, 7 in 2024-09-21T12:31:12.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:12 smithi114 ceph-mon[30324]: 9.8 scrub ok 2024-09-21T12:31:12.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:12 smithi114 ceph-mon[30324]: 9.0 scrub ok 2024-09-21T12:31:12.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:12 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1822371425' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:31:12.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:12 smithi114 ceph-mon[30324]: 9.12 scrub ok 2024-09-21T12:31:12.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:12 smithi114 ceph-mon[30324]: 4.1e scrub starts 2024-09-21T12:31:12.501 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:12 smithi114 ceph-mon[30324]: 4.1e scrub ok 2024-09-21T12:31:12.501 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:12 smithi114 ceph-mon[30324]: 1.12 scrub starts 2024-09-21T12:31:12.501 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:12 smithi114 ceph-mon[30324]: 1.12 scrub ok 2024-09-21T12:31:12.501 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:12 smithi114 ceph-mon[30324]: osdmap e299: 8 total, 8 up, 7 in 2024-09-21T12:31:12.513 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:12.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:12.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:12.616 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:12.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:12.647 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:12.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:12.668 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:31:12.668 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:31:12.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:12.775 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:31:12 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:31:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:31:12.790 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:12.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:12.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:12.892 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:12.928 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:12.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:12.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:13.029 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:13.063 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:13.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:13.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:13.164 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:13.194 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:13.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:13.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:13.296 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:13.326 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T12:31:13.326 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:31:13.326 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-21T12:31:13.326 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T12:31:13.326 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T12:31:13.326 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-21T12:31:13.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:13.328 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:13.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:13.397 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:13 smithi183 ceph-mon[33277]: pgmap v539: 188 pgs: 1 active+clean+scrubbing, 7 active+remapped+backfill_toofull, 180 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 895 B/s rd, 1 op/s; 16057/185865 objects misplaced (8.639%) 2024-09-21T12:31:13.398 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:13 smithi183 ceph-mon[33277]: 6.1 scrub starts 2024-09-21T12:31:13.398 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:13 smithi183 ceph-mon[33277]: 6.1 scrub ok 2024-09-21T12:31:13.398 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:13 smithi183 ceph-mon[33277]: 6.1a scrub starts 2024-09-21T12:31:13.398 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:13 smithi183 ceph-mon[33277]: 6.1a scrub ok 2024-09-21T12:31:13.421 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:13 smithi114 ceph-mon[40867]: pgmap v539: 188 pgs: 1 active+clean+scrubbing, 7 active+remapped+backfill_toofull, 180 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 895 B/s rd, 1 op/s; 16057/185865 objects misplaced (8.639%) 2024-09-21T12:31:13.421 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:13 smithi114 ceph-mon[40867]: 6.1 scrub starts 2024-09-21T12:31:13.422 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:13 smithi114 ceph-mon[40867]: 6.1 scrub ok 2024-09-21T12:31:13.422 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:13 smithi114 ceph-mon[40867]: 6.1a scrub starts 2024-09-21T12:31:13.422 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:13 smithi114 ceph-mon[40867]: 6.1a scrub ok 2024-09-21T12:31:13.423 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:13 smithi114 ceph-mon[30324]: pgmap v539: 188 pgs: 1 active+clean+scrubbing, 7 active+remapped+backfill_toofull, 180 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 895 B/s rd, 1 op/s; 16057/185865 objects misplaced (8.639%) 2024-09-21T12:31:13.423 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:13 smithi114 ceph-mon[30324]: 6.1 scrub starts 2024-09-21T12:31:13.423 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:13 smithi114 ceph-mon[30324]: 6.1 scrub ok 2024-09-21T12:31:13.423 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:13 smithi114 ceph-mon[30324]: 6.1a scrub starts 2024-09-21T12:31:13.423 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:13 smithi114 ceph-mon[30324]: 6.1a scrub ok 2024-09-21T12:31:13.429 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:13.477 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T12:31:13.480 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:13.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:13.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:13.539 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:31:13.583 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:13.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:13.627 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:13.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:13.728 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:13.773 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:13.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:13.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:13.807 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:31:13.875 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:13.917 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:13.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:13.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:14.019 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:14.052 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:14.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:14.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:14.153 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:14.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:14.193 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:14.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:14.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:14 smithi183 ceph-mon[33277]: 3.c scrub starts 2024-09-21T12:31:14.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:14 smithi183 ceph-mon[33277]: 3.c scrub ok 2024-09-21T12:31:14.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:14 smithi183 ceph-mon[33277]: 1.17 scrub starts 2024-09-21T12:31:14.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:14 smithi183 ceph-mon[33277]: 1.17 scrub ok 2024-09-21T12:31:14.295 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:14.337 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:14.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:14.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:14.346 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:31:14.375 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:31:14.414 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:14 smithi114 ceph-mon[40867]: 3.c scrub starts 2024-09-21T12:31:14.414 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:14 smithi114 ceph-mon[40867]: 3.c scrub ok 2024-09-21T12:31:14.414 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:14 smithi114 ceph-mon[40867]: 1.17 scrub starts 2024-09-21T12:31:14.414 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:14 smithi114 ceph-mon[40867]: 1.17 scrub ok 2024-09-21T12:31:14.414 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:14 smithi114 ceph-mon[30324]: 3.c scrub starts 2024-09-21T12:31:14.414 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:14 smithi114 ceph-mon[30324]: 3.c scrub ok 2024-09-21T12:31:14.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:14 smithi114 ceph-mon[30324]: 1.17 scrub starts 2024-09-21T12:31:14.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:14 smithi114 ceph-mon[30324]: 1.17 scrub ok 2024-09-21T12:31:14.415 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:31:14.415 INFO:teuthology.orchestra.run.smithi183.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T12:31:14.415 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:31:14.440 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:14.501 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:14.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:14.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:14.603 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:14.646 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:14.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:14.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:14.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:14.795 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:14.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:14.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:14.849 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:31:14.897 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:14.936 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:14.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:14.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:15.037 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:15.072 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:15.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:15.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:15.115 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:31:15.115 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:31:15.141 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-21T12:31:15.174 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:15.206 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:15.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:15.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:15.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:15 smithi183 ceph-mon[33277]: pgmap v541: 188 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 182 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 14669/176298 objects misplaced (8.321%) 2024-09-21T12:31:15.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:15 smithi183 ceph-mon[33277]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:31:15.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:15 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3315454848' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:31:15.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:15 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:31:15.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:15 smithi114 ceph-mon[30324]: pgmap v541: 188 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 182 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 14669/176298 objects misplaced (8.321%) 2024-09-21T12:31:15.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:15 smithi114 ceph-mon[30324]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:31:15.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:15 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3315454848' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:31:15.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:15 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:31:15.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:15 smithi114 ceph-mon[40867]: pgmap v541: 188 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 182 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 14669/176298 objects misplaced (8.321%) 2024-09-21T12:31:15.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:15 smithi114 ceph-mon[40867]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:31:15.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:15 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3315454848' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:31:15.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:15 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:31:15.308 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:15.339 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:15.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:15.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:15.441 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:15.474 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:15.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:15.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:15.576 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:15.617 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:15.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:15.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:15.719 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:15.740 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:15.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:15.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:15.842 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:15.873 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:15.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:15.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:15.975 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:16.001 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:16.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:16.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:16.103 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:16.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:16.135 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:16.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:16.165 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T12:31:16.237 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:16.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:16.271 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:16.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:16.372 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:16.404 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:16.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:16.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:16.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:16 smithi183 ceph-mon[33277]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:31:16.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:16 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:31:16.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:16 smithi183 ceph-mon[33277]: osdmap e300: 8 total, 8 up, 7 in 2024-09-21T12:31:16.506 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:16.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:16 smithi114 ceph-mon[40867]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:31:16.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:16 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:31:16.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:16 smithi114 ceph-mon[40867]: osdmap e300: 8 total, 8 up, 7 in 2024-09-21T12:31:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:16 smithi114 ceph-mon[30324]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:31:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:16 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:31:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:16 smithi114 ceph-mon[30324]: osdmap e300: 8 total, 8 up, 7 in 2024-09-21T12:31:16.545 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:16.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:16.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:16.647 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:16.683 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:16.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:16.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:16.724 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:31:16.785 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:16.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:16.812 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:16.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:16.913 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:16.934 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:16.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:16.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:17.036 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:17.065 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:17.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:17.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:17.167 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:17.205 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:17.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:17.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:17.285 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:17 smithi114 ceph-mon[40867]: pgmap v543: 188 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 182 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 14669/176298 objects misplaced (8.321%) 2024-09-21T12:31:17.285 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:17 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:31:17.285 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:17 smithi114 ceph-mon[30324]: pgmap v543: 188 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 182 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 14669/176298 objects misplaced (8.321%) 2024-09-21T12:31:17.285 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:17 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:31:17.308 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:17.342 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:17.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:17.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:17.354 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:31:17.354 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:31:17.445 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:17.484 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:17.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:17.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:17.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:17 smithi183 ceph-mon[33277]: pgmap v543: 188 pgs: 1 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 182 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 14669/176298 objects misplaced (8.321%) 2024-09-21T12:31:17.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:17 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:31:17.488 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:31:17 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:31:17.586 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:17.620 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:17.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:17.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:17.722 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:17.758 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:17.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:17.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:17.860 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:17.899 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:17.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:17.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:18.001 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:18.028 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:18.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:18.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:18.114 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T12:31:18.130 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:18.192 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:18.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:18.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:18.294 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:18.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:18.334 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:18.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:18.356 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:18 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:18.357 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:18 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:18.416 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:31:18.416 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:31:18.435 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:18.443 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:31:18.444 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:31:18.467 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:31:18.470 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:18.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:18.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:18.474 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:31:18.474 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:31:18.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:18 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:18.504 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:31:18.505 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:31:18.533 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:31:18.533 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:31:18.560 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:31:18.560 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:31:18.572 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:18.585 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:31:18.585 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:31:18.596 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:18.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:18.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:18.612 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:31:18.612 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:31:18.697 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:18.728 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:18.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:18.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:18.830 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:18.856 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:18.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:18.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:18.957 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:18.981 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:18.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:18.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:18.985 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:31:18.985 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-21T12:31:18.985 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-21T12:31:18.985 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:31:18.985 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.986 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:23884 7.1 7:8f148689:::benchmark_data_smithi114_78432_object23883:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T12:31:18.986 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:39.695870+0000", 2024-09-21T12:31:18.986 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 579.28721385699998, 2024-09-21T12:31:18.986 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.039559285999999999, 2024-09-21T12:31:18.986 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:18.986 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:18.986 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:18.986 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:31:18.986 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:31:18.986 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:31:18.986 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 23884 2024-09-21T12:31:18.986 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:18.986 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:18.986 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.987 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:18.987 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695870+0000", 2024-09-21T12:31:18.987 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:18.987 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:18.987 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.987 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:18.987 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695870+0000", 2024-09-21T12:31:18.987 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:18.987 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:18.987 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.987 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:18.987 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695873+0000", 2024-09-21T12:31:18.987 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0090000000000002e-06 2024-09-21T12:31:18.987 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:18.988 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.988 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:18.988 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695927+0000", 2024-09-21T12:31:18.988 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4941000000000001e-05 2024-09-21T12:31:18.988 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:18.988 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.988 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:18.988 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695955+0000", 2024-09-21T12:31:18.988 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7120000000000001e-05 2024-09-21T12:31:18.988 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:18.988 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.988 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:18.988 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.695963+0000", 2024-09-21T12:31:18.988 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7760000000000003e-06 2024-09-21T12:31:18.989 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:18.989 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.989 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:18.989 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696022+0000", 2024-09-21T12:31:18.989 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8941999999999999e-05 2024-09-21T12:31:18.989 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:18.989 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.989 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:18.989 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696107+0000", 2024-09-21T12:31:18.989 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4853999999999998e-05 2024-09-21T12:31:18.989 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:18.989 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.989 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,6", 2024-09-21T12:31:18.989 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696202+0000", 2024-09-21T12:31:18.990 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4889999999999994e-05 2024-09-21T12:31:18.990 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:18.990 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.990 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:31:18.990 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.698132+0000", 2024-09-21T12:31:18.990 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001929998 2024-09-21T12:31:18.990 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:18.990 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.990 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:18.990 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.698167+0000", 2024-09-21T12:31:18.990 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5247000000000003e-05 2024-09-21T12:31:18.990 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:18.990 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.990 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:18.991 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.735322+0000", 2024-09-21T12:31:18.991 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.037154514 2024-09-21T12:31:18.991 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:18.991 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.991 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:18.991 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.735382+0000", 2024-09-21T12:31:18.991 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0371000000000002e-05 2024-09-21T12:31:18.991 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:18.991 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.991 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:18.991 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.735430+0000", 2024-09-21T12:31:18.991 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7623999999999997e-05 2024-09-21T12:31:18.991 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:18.991 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:18.992 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:18.992 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:18.992 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.992 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:23885 7.1 7:81ccbda1:::benchmark_data_smithi114_78432_object23884:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T12:31:18.992 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:39.696116+0000", 2024-09-21T12:31:18.992 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 579.28696802800005, 2024-09-21T12:31:18.992 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.040219576, 2024-09-21T12:31:18.992 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:18.992 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:18.992 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:18.992 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:31:18.992 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:31:18.992 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:31:18.992 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 23885 2024-09-21T12:31:18.992 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:18.993 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:18.993 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.993 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:18.993 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696116+0000", 2024-09-21T12:31:18.993 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:18.993 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:18.993 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.993 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:18.993 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696116+0000", 2024-09-21T12:31:18.993 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:18.993 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:18.993 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.993 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:18.993 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696117+0000", 2024-09-21T12:31:18.994 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0109999999999999e-06 2024-09-21T12:31:18.994 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:18.994 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.994 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:18.994 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696178+0000", 2024-09-21T12:31:18.994 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1060000000000002e-05 2024-09-21T12:31:18.994 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:18.994 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.994 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:18.994 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696188+0000", 2024-09-21T12:31:18.994 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0067e-05 2024-09-21T12:31:18.994 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:18.994 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.994 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:18.995 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.696196+0000", 2024-09-21T12:31:18.995 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0740000000000004e-06 2024-09-21T12:31:18.995 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:18.995 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.995 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:18.995 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.697126+0000", 2024-09-21T12:31:18.995 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00093067599999999996 2024-09-21T12:31:18.995 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:18.995 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.995 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:18.995 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.697270+0000", 2024-09-21T12:31:18.995 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000143502 2024-09-21T12:31:18.995 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:18.996 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.996 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,6", 2024-09-21T12:31:18.996 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.697418+0000", 2024-09-21T12:31:18.996 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000147892 2024-09-21T12:31:18.996 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:18.996 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.996 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:31:18.996 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.698242+0000", 2024-09-21T12:31:18.996 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00082411799999999999 2024-09-21T12:31:18.996 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:18.996 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.996 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:18.996 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.698485+0000", 2024-09-21T12:31:18.997 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00024341099999999999 2024-09-21T12:31:18.997 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:18.997 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.997 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:18.997 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.736250+0000", 2024-09-21T12:31:18.997 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.037765265999999999 2024-09-21T12:31:18.997 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:18.997 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.997 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:18.997 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.736300+0000", 2024-09-21T12:31:18.997 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9886000000000003e-05 2024-09-21T12:31:18.997 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:18.997 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.997 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:18.998 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:39.736336+0000", 2024-09-21T12:31:18.998 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5612999999999999e-05 2024-09-21T12:31:18.998 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:18.998 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:18.998 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:18.998 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:18.998 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.998 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:81802 7.8 e115/112 7:110c0c7a:::benchmark_data_smithi114_78432_object81801:head v 115'5198, mlcod=88'242)", 2024-09-21T12:31:18.998 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.137865+0000", 2024-09-21T12:31:18.998 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 523.84521914100003, 2024-09-21T12:31:18.998 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.051160639000000001, 2024-09-21T12:31:18.998 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:18.998 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:18.998 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:18.998 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:18.999 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:18.999 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:18.999 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137865+0000", 2024-09-21T12:31:19.000 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.000 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.000 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.000 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:19.000 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137865+0000", 2024-09-21T12:31:19.000 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.000 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.000 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.000 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:19.000 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137866+0000", 2024-09-21T12:31:19.000 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5099999999999998e-07 2024-09-21T12:31:19.000 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.000 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.000 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:19.001 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137909+0000", 2024-09-21T12:31:19.001 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2545e-05 2024-09-21T12:31:19.001 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.001 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.001 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:19.001 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137938+0000", 2024-09-21T12:31:19.001 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9737999999999999e-05 2024-09-21T12:31:19.001 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.001 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.001 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:19.001 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.137961+0000", 2024-09-21T12:31:19.001 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2320999999999999e-05 2024-09-21T12:31:19.001 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.001 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.002 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:19.002 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.138026+0000", 2024-09-21T12:31:19.002 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5571e-05 2024-09-21T12:31:19.002 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.002 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.002 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:19.002 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.138111+0000", 2024-09-21T12:31:19.002 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4682e-05 2024-09-21T12:31:19.002 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.002 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.002 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:19.002 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.188940+0000", 2024-09-21T12:31:19.002 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.050828867 2024-09-21T12:31:19.002 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.003 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.003 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:19.003 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189026+0000", 2024-09-21T12:31:19.003 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6063999999999995e-05 2024-09-21T12:31:19.003 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.003 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:19.003 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.003 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.003 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.003 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:81809 7.8 e115/112 7:1042b23e:::benchmark_data_smithi114_78432_object81808:head v 115'5199, mlcod=88'242)", 2024-09-21T12:31:19.003 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.140023+0000", 2024-09-21T12:31:19.003 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 523.84306157000003, 2024-09-21T12:31:19.003 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.049109005999999997, 2024-09-21T12:31:19.003 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:19.004 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:19.004 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:19.004 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:19.004 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.004 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:19.004 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140023+0000", 2024-09-21T12:31:19.004 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.004 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.004 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.004 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:19.004 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140023+0000", 2024-09-21T12:31:19.004 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.004 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.005 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.005 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:19.005 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140023+0000", 2024-09-21T12:31:19.005 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6700000000000003e-07 2024-09-21T12:31:19.005 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.005 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.005 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:19.005 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140136+0000", 2024-09-21T12:31:19.005 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000112623 2024-09-21T12:31:19.005 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.005 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.005 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:19.005 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140146+0000", 2024-09-21T12:31:19.005 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0098999999999999e-05 2024-09-21T12:31:19.005 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.006 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.006 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:19.006 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.140161+0000", 2024-09-21T12:31:19.006 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5288e-05 2024-09-21T12:31:19.006 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.006 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.006 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:19.006 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146087+0000", 2024-09-21T12:31:19.006 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0059259070000000002 2024-09-21T12:31:19.006 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.006 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.006 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:19.006 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146109+0000", 2024-09-21T12:31:19.007 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2045999999999999e-05 2024-09-21T12:31:19.007 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.007 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.007 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:19.007 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189077+0000", 2024-09-21T12:31:19.007 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.042967639000000002 2024-09-21T12:31:19.007 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.007 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.007 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:19.007 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189132+0000", 2024-09-21T12:31:19.007 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4737000000000001e-05 2024-09-21T12:31:19.007 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.007 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:19.007 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.008 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.008 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.008 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:81811 7.8 e115/112 7:10044f1a:::benchmark_data_smithi114_78432_object81810:head v 115'5200, mlcod=88'242)", 2024-09-21T12:31:19.008 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.146782+0000", 2024-09-21T12:31:19.008 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 523.83630208500006, 2024-09-21T12:31:19.008 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.042406829, 2024-09-21T12:31:19.008 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:19.008 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:19.008 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:19.008 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:19.008 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.008 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:19.008 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146782+0000", 2024-09-21T12:31:19.009 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.009 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.009 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.009 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:19.009 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146782+0000", 2024-09-21T12:31:19.009 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.009 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.009 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.009 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:19.009 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146783+0000", 2024-09-21T12:31:19.009 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2999999999999999e-07 2024-09-21T12:31:19.009 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.009 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.009 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:19.010 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146826+0000", 2024-09-21T12:31:19.010 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2781000000000003e-05 2024-09-21T12:31:19.010 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.010 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.010 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:19.010 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146837+0000", 2024-09-21T12:31:19.010 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1126e-05 2024-09-21T12:31:19.010 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.010 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.010 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:19.010 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.146843+0000", 2024-09-21T12:31:19.010 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4710000000000004e-06 2024-09-21T12:31:19.010 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.010 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.011 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:19.011 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.152740+0000", 2024-09-21T12:31:19.011 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0058961209999999998 2024-09-21T12:31:19.011 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.011 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.011 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:19.011 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.152763+0000", 2024-09-21T12:31:19.011 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3101000000000001e-05 2024-09-21T12:31:19.011 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.011 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.011 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:19.011 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189134+0000", 2024-09-21T12:31:19.011 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.036371542999999999 2024-09-21T12:31:19.011 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.012 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.012 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:19.012 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189189+0000", 2024-09-21T12:31:19.012 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4855999999999998e-05 2024-09-21T12:31:19.012 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.012 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:19.012 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.012 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.012 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.012 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:81821 7.8 e115/112 7:1c79dadc:::benchmark_data_smithi114_78432_object81820:head v 115'5201, mlcod=88'242)", 2024-09-21T12:31:19.012 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.148399+0000", 2024-09-21T12:31:19.012 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 523.83468501000004, 2024-09-21T12:31:19.012 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.040843988999999997, 2024-09-21T12:31:19.012 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:19.013 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:19.013 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:19.013 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:19.013 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.013 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:19.013 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148399+0000", 2024-09-21T12:31:19.014 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.014 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.014 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.014 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:19.014 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148399+0000", 2024-09-21T12:31:19.014 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.014 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.014 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.014 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:19.014 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148400+0000", 2024-09-21T12:31:19.014 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.049e-06 2024-09-21T12:31:19.014 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.014 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.015 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:19.015 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148502+0000", 2024-09-21T12:31:19.015 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000101542 2024-09-21T12:31:19.015 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.015 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.015 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:19.015 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148512+0000", 2024-09-21T12:31:19.015 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0386e-05 2024-09-21T12:31:19.015 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.015 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.015 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:19.015 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.148519+0000", 2024-09-21T12:31:19.015 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3589999999999996e-06 2024-09-21T12:31:19.015 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.016 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.016 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:19.016 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.158203+0000", 2024-09-21T12:31:19.016 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0096844710000000001 2024-09-21T12:31:19.016 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.016 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.016 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:19.016 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.158229+0000", 2024-09-21T12:31:19.016 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6078999999999999e-05 2024-09-21T12:31:19.016 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.016 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.016 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:19.016 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189192+0000", 2024-09-21T12:31:19.016 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.030962455 2024-09-21T12:31:19.017 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.017 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.017 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:19.017 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.189243+0000", 2024-09-21T12:31:19.017 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1647999999999997e-05 2024-09-21T12:31:19.017 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.017 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:19.017 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.017 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.017 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.017 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82430 7.8 e115/112 7:1762e17c:::benchmark_data_smithi114_78432_object82429:head v 115'5247, mlcod=88'255)", 2024-09-21T12:31:19.017 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.787623+0000", 2024-09-21T12:31:19.017 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 523.19546122199995, 2024-09-21T12:31:19.017 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.048511400000000003, 2024-09-21T12:31:19.017 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:19.018 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:19.018 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:19.018 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:19.018 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.018 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:19.018 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787623+0000", 2024-09-21T12:31:19.018 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.018 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.018 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.018 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:19.018 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787623+0000", 2024-09-21T12:31:19.018 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.018 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.018 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.019 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:19.019 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787625+0000", 2024-09-21T12:31:19.019 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5319999999999999e-06 2024-09-21T12:31:19.019 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.019 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.019 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:19.019 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787667+0000", 2024-09-21T12:31:19.019 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1876999999999999e-05 2024-09-21T12:31:19.019 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.019 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.019 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:19.019 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787688+0000", 2024-09-21T12:31:19.019 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1046000000000001e-05 2024-09-21T12:31:19.019 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.020 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.020 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:19.020 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787697+0000", 2024-09-21T12:31:19.020 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4930000000000004e-06 2024-09-21T12:31:19.020 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.020 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.020 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:19.020 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.788378+0000", 2024-09-21T12:31:19.020 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00068115999999999999 2024-09-21T12:31:19.020 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.020 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.020 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:19.020 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.788427+0000", 2024-09-21T12:31:19.021 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8553000000000001e-05 2024-09-21T12:31:19.021 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.021 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.021 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:19.021 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.836011+0000", 2024-09-21T12:31:19.021 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.047583978999999998 2024-09-21T12:31:19.021 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.021 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.021 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:19.021 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.836135+0000", 2024-09-21T12:31:19.021 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012375999999999999 2024-09-21T12:31:19.021 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.021 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:19.021 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.022 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.022 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.022 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDPGPush(7.8 115/112 [PushOp(7:10df046b:::benchmark_data_smithi114_78432_object3988:head, version: 88'257, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:10df046b:::benchmark_data_smithi114_78432_object3988:head@88'257, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T12:31:19.022 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.787722+0000", 2024-09-21T12:31:19.022 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 523.19536214699997, 2024-09-21T12:31:19.022 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.074165650999999999, 2024-09-21T12:31:19.022 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:19.022 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:19.022 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:31:19.022 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:19.022 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.022 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:19.022 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787722+0000", 2024-09-21T12:31:19.023 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.023 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.023 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.023 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:19.023 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787722+0000", 2024-09-21T12:31:19.023 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.023 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.023 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.023 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:19.023 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787723+0000", 2024-09-21T12:31:19.023 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0499999999999999e-07 2024-09-21T12:31:19.023 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.023 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.023 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:19.024 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787779+0000", 2024-09-21T12:31:19.024 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6053e-05 2024-09-21T12:31:19.024 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.024 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.024 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:19.024 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787803+0000", 2024-09-21T12:31:19.024 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3980000000000001e-05 2024-09-21T12:31:19.024 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.024 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.024 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:19.024 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.787832+0000", 2024-09-21T12:31:19.024 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.942e-05 2024-09-21T12:31:19.024 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.025 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.025 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:19.025 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.859697+0000", 2024-09-21T12:31:19.025 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.071864649000000003 2024-09-21T12:31:19.025 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.025 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.025 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:19.025 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.859710+0000", 2024-09-21T12:31:19.025 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3740000000000001e-05 2024-09-21T12:31:19.025 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.025 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.025 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:19.025 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.861888+0000", 2024-09-21T12:31:19.026 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002177404 2024-09-21T12:31:19.026 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.026 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:19.026 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.026 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.026 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.026 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82442 7.8 e115/112 7:1035456a:::benchmark_data_smithi114_78432_object82441:head v 115'5248, mlcod=88'255)", 2024-09-21T12:31:19.026 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.791117+0000", 2024-09-21T12:31:19.026 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 523.19196724400001, 2024-09-21T12:31:19.026 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.051248759999999997, 2024-09-21T12:31:19.026 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:19.026 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:19.026 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:19.026 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:19.027 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.027 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:19.027 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791117+0000", 2024-09-21T12:31:19.027 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.027 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.027 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.027 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:19.027 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791117+0000", 2024-09-21T12:31:19.027 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.027 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.027 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.027 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:19.027 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791118+0000", 2024-09-21T12:31:19.028 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9999999999999998e-07 2024-09-21T12:31:19.028 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.028 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.028 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:19.028 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791158+0000", 2024-09-21T12:31:19.028 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0025999999999999e-05 2024-09-21T12:31:19.028 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.028 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.028 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:19.028 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791167+0000", 2024-09-21T12:31:19.028 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9910000000000008e-06 2024-09-21T12:31:19.028 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.028 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.029 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:19.029 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.791176+0000", 2024-09-21T12:31:19.029 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0329999999999997e-06 2024-09-21T12:31:19.029 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.029 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.029 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:19.029 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.794825+0000", 2024-09-21T12:31:19.029 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0036494700000000001 2024-09-21T12:31:19.029 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.029 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.029 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:19.029 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.794878+0000", 2024-09-21T12:31:19.029 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3292999999999999e-05 2024-09-21T12:31:19.029 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.030 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.030 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:19.030 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842289+0000", 2024-09-21T12:31:19.030 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.047410256999999997 2024-09-21T12:31:19.030 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.030 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.030 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:19.030 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842366+0000", 2024-09-21T12:31:19.030 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7189999999999998e-05 2024-09-21T12:31:19.030 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.030 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:19.030 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.030 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.031 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.031 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82447 7.8 e115/112 7:174332ca:::benchmark_data_smithi114_78432_object82446:head v 115'5249, mlcod=88'255)", 2024-09-21T12:31:19.031 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.793874+0000", 2024-09-21T12:31:19.031 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 523.18921079300003, 2024-09-21T12:31:19.031 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.048541015999999999, 2024-09-21T12:31:19.031 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:19.031 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:19.031 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:19.031 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:19.031 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.031 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:19.031 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793874+0000", 2024-09-21T12:31:19.031 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.031 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.031 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.032 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:19.032 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793874+0000", 2024-09-21T12:31:19.032 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.032 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.032 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.032 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:19.032 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793875+0000", 2024-09-21T12:31:19.032 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0890000000000001e-06 2024-09-21T12:31:19.032 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.032 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.032 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:19.032 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793910+0000", 2024-09-21T12:31:19.032 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5080000000000003e-05 2024-09-21T12:31:19.032 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.032 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.033 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:19.033 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793921+0000", 2024-09-21T12:31:19.033 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1284000000000001e-05 2024-09-21T12:31:19.033 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.033 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.033 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:19.033 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.793930+0000", 2024-09-21T12:31:19.033 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5429999999999994e-06 2024-09-21T12:31:19.033 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.033 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.033 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:19.033 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.801012+0000", 2024-09-21T12:31:19.033 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0070825219999999999 2024-09-21T12:31:19.033 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.033 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.034 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:19.034 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.801039+0000", 2024-09-21T12:31:19.034 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7248999999999999e-05 2024-09-21T12:31:19.034 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.034 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.034 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:19.034 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842368+0000", 2024-09-21T12:31:19.034 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.041328831000000003 2024-09-21T12:31:19.034 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.034 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.034 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:19.034 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842415+0000", 2024-09-21T12:31:19.034 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6418000000000002e-05 2024-09-21T12:31:19.034 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.034 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:19.035 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.035 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.035 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.035 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82448 7.8 e115/112 7:11700dea:::benchmark_data_smithi114_78432_object82447:head v 115'5250, mlcod=88'255)", 2024-09-21T12:31:19.035 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.796531+0000", 2024-09-21T12:31:19.035 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 523.18655304900005, 2024-09-21T12:31:19.035 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.045928499999999997, 2024-09-21T12:31:19.035 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:19.035 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:19.035 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:19.035 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:19.035 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.035 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:19.035 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796531+0000", 2024-09-21T12:31:19.036 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.036 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.036 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.036 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:19.036 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796531+0000", 2024-09-21T12:31:19.036 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.036 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.036 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.036 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:19.036 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796532+0000", 2024-09-21T12:31:19.036 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7199999999999999e-07 2024-09-21T12:31:19.036 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.036 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.036 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:19.037 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796647+0000", 2024-09-21T12:31:19.037 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000115097 2024-09-21T12:31:19.037 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.037 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.037 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:19.037 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796660+0000", 2024-09-21T12:31:19.037 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3271999999999999e-05 2024-09-21T12:31:19.037 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.037 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.037 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:19.037 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.796666+0000", 2024-09-21T12:31:19.037 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0170000000000002e-06 2024-09-21T12:31:19.037 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.037 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.037 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:19.038 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.809350+0000", 2024-09-21T12:31:19.038 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.012683379 2024-09-21T12:31:19.038 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.038 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.038 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:19.038 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.809374+0000", 2024-09-21T12:31:19.038 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4417e-05 2024-09-21T12:31:19.038 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.038 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.038 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:19.038 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842417+0000", 2024-09-21T12:31:19.038 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.033042646000000002 2024-09-21T12:31:19.038 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.038 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.039 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:19.039 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.842460+0000", 2024-09-21T12:31:19.039 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3099999999999997e-05 2024-09-21T12:31:19.039 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.039 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:19.039 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.039 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.039 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.039 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82473 7.8 e115/112 7:15016882:::benchmark_data_smithi114_78432_object82472:head v 115'5254, mlcod=88'255)", 2024-09-21T12:31:19.039 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.818965+0000", 2024-09-21T12:31:19.039 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 523.16411945599998, 2024-09-21T12:31:19.039 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.045024287000000003, 2024-09-21T12:31:19.039 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:19.039 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:19.039 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:19.040 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:19.040 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.040 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:19.040 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.818965+0000", 2024-09-21T12:31:19.040 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.040 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.040 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.040 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:19.040 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.818965+0000", 2024-09-21T12:31:19.040 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.040 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.040 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.040 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:19.041 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.818966+0000", 2024-09-21T12:31:19.041 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9199999999999999e-07 2024-09-21T12:31:19.041 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.041 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.041 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:19.041 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.819001+0000", 2024-09-21T12:31:19.042 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5117999999999998e-05 2024-09-21T12:31:19.042 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.042 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.042 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:19.042 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.819014+0000", 2024-09-21T12:31:19.042 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2869e-05 2024-09-21T12:31:19.042 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.042 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.042 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:19.042 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.819020+0000", 2024-09-21T12:31:19.042 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6959999999999999e-06 2024-09-21T12:31:19.042 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.043 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.043 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:19.043 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.836164+0000", 2024-09-21T12:31:19.043 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.017143617 2024-09-21T12:31:19.043 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.043 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.043 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:19.043 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.836188+0000", 2024-09-21T12:31:19.043 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4074e-05 2024-09-21T12:31:19.043 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.043 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.043 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:19.043 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.863879+0000", 2024-09-21T12:31:19.043 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.027690392000000001 2024-09-21T12:31:19.044 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.044 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.044 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:19.044 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.863989+0000", 2024-09-21T12:31:19.044 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000110629 2024-09-21T12:31:19.044 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.044 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:19.044 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.044 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.044 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.044 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82539 7.8 e115/112 7:1cd4c60d:::benchmark_data_smithi114_78432_object82538:head v 115'5258, mlcod=88'256)", 2024-09-21T12:31:19.044 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.889845+0000", 2024-09-21T12:31:19.044 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 523.09323905300005, 2024-09-21T12:31:19.044 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.046325071000000002, 2024-09-21T12:31:19.045 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:19.045 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:19.045 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:19.045 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:19.045 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.045 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:19.045 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889845+0000", 2024-09-21T12:31:19.045 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.045 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.045 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.045 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:19.045 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889845+0000", 2024-09-21T12:31:19.045 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.045 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.046 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.046 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:19.046 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889846+0000", 2024-09-21T12:31:19.046 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1800000000000004e-07 2024-09-21T12:31:19.046 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.046 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.046 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:19.046 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889878+0000", 2024-09-21T12:31:19.046 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1619999999999999e-05 2024-09-21T12:31:19.046 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.046 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.046 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:19.046 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889893+0000", 2024-09-21T12:31:19.047 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4819e-05 2024-09-21T12:31:19.047 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.047 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.047 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:19.047 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.889900+0000", 2024-09-21T12:31:19.047 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8510000000000001e-06 2024-09-21T12:31:19.047 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.047 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.047 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:19.047 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.898101+0000", 2024-09-21T12:31:19.047 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0082018119999999993 2024-09-21T12:31:19.047 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.047 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.047 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:19.048 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.898164+0000", 2024-09-21T12:31:19.048 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2756999999999996e-05 2024-09-21T12:31:19.048 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.048 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.048 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:19.048 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.936000+0000", 2024-09-21T12:31:19.048 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.037835720000000003 2024-09-21T12:31:19.048 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.048 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.048 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:19.048 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.936170+0000", 2024-09-21T12:31:19.048 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000170574 2024-09-21T12:31:19.048 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.048 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:19.049 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.049 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.049 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.049 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:82545 7.8 e115/112 7:1ddea6d1:::benchmark_data_smithi114_78432_object82544:head v 115'5259, mlcod=88'256)", 2024-09-21T12:31:19.049 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:35.897417+0000", 2024-09-21T12:31:19.049 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 523.08566751599994, 2024-09-21T12:31:19.049 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.038827027, 2024-09-21T12:31:19.049 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:19.049 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:19.049 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:19.049 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:19.049 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.049 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:19.050 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897417+0000", 2024-09-21T12:31:19.050 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.050 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.050 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.050 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:19.050 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897417+0000", 2024-09-21T12:31:19.050 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.050 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.050 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.050 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:19.050 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897418+0000", 2024-09-21T12:31:19.050 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3500000000000005e-07 2024-09-21T12:31:19.050 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.050 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.051 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:19.051 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897462+0000", 2024-09-21T12:31:19.051 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4036999999999999e-05 2024-09-21T12:31:19.051 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.051 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.051 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:19.051 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897472+0000", 2024-09-21T12:31:19.051 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0669999999999999e-05 2024-09-21T12:31:19.051 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.051 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.051 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:19.051 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.897479+0000", 2024-09-21T12:31:19.051 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9190000000000002e-06 2024-09-21T12:31:19.051 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.052 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.052 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:19.052 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.905524+0000", 2024-09-21T12:31:19.052 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0080444899999999996 2024-09-21T12:31:19.052 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.052 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.052 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:19.052 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.905597+0000", 2024-09-21T12:31:19.052 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3539000000000002e-05 2024-09-21T12:31:19.052 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.052 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.052 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:19.052 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.936176+0000", 2024-09-21T12:31:19.052 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.030578893999999999 2024-09-21T12:31:19.053 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.053 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.053 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:19.053 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:35.936244+0000", 2024-09-21T12:31:19.053 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7543e-05 2024-09-21T12:31:19.053 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.053 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:19.053 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.053 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.053 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.053 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:83346 7.8 e115/112 7:1517ab05:::benchmark_data_smithi114_78432_object83345:head v 115'5319, mlcod=88'282)", 2024-09-21T12:31:19.053 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:36.700133+0000", 2024-09-21T12:31:19.053 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 522.282951615, 2024-09-21T12:31:19.053 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.044693733999999999, 2024-09-21T12:31:19.054 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:19.054 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:19.054 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:19.054 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:19.054 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.054 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:19.054 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700133+0000", 2024-09-21T12:31:19.054 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.054 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.054 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.054 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:19.054 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700133+0000", 2024-09-21T12:31:19.054 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.054 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.055 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.055 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:19.055 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700133+0000", 2024-09-21T12:31:19.055 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4300000000000003e-07 2024-09-21T12:31:19.055 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.055 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.055 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:19.055 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700182+0000", 2024-09-21T12:31:19.055 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8433000000000002e-05 2024-09-21T12:31:19.055 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.055 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.055 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:19.055 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700192+0000", 2024-09-21T12:31:19.055 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0382e-05 2024-09-21T12:31:19.056 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.056 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.056 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:19.056 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.700201+0000", 2024-09-21T12:31:19.056 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9819999999999997e-06 2024-09-21T12:31:19.056 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.056 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.056 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:19.056 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.703063+0000", 2024-09-21T12:31:19.056 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002862121 2024-09-21T12:31:19.056 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.056 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.056 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:19.057 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.703087+0000", 2024-09-21T12:31:19.057 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3730000000000001e-05 2024-09-21T12:31:19.057 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.057 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.057 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:19.057 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.744677+0000", 2024-09-21T12:31:19.057 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.041589785999999997 2024-09-21T12:31:19.057 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.057 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.057 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:19.057 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:36.744826+0000", 2024-09-21T12:31:19.057 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014965700000000001 2024-09-21T12:31:19.057 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.057 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:19.058 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.058 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.058 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.058 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:87464 7.8 e117/112 7:17a0a89d:::benchmark_data_smithi114_78432_object87463:head v 117'5593, mlcod=88'417)", 2024-09-21T12:31:19.058 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:41.463661+0000", 2024-09-21T12:31:19.058 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 517.51942368200002, 2024-09-21T12:31:19.058 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.067076070000000002, 2024-09-21T12:31:19.058 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:19.058 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:19.058 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:19.058 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:19.058 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.058 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:19.058 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463661+0000", 2024-09-21T12:31:19.059 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.059 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.059 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.059 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:19.059 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463661+0000", 2024-09-21T12:31:19.059 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.059 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.059 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.059 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:19.059 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463661+0000", 2024-09-21T12:31:19.059 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2799999999999995e-07 2024-09-21T12:31:19.059 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.059 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.059 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:19.060 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463845+0000", 2024-09-21T12:31:19.060 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000183692 2024-09-21T12:31:19.060 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.060 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.060 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:19.060 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463854+0000", 2024-09-21T12:31:19.060 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0750000000000004e-06 2024-09-21T12:31:19.060 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.060 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.060 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:19.060 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.463860+0000", 2024-09-21T12:31:19.060 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.015e-06 2024-09-21T12:31:19.060 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.060 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.061 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:19.061 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.480218+0000", 2024-09-21T12:31:19.061 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.016357289000000001 2024-09-21T12:31:19.061 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.061 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.061 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:19.061 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.480239+0000", 2024-09-21T12:31:19.061 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1691999999999998e-05 2024-09-21T12:31:19.061 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.061 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.061 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:19.061 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.530695+0000", 2024-09-21T12:31:19.061 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.050456050000000002 2024-09-21T12:31:19.061 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.062 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.062 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:19.062 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.530737+0000", 2024-09-21T12:31:19.062 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1428999999999999e-05 2024-09-21T12:31:19.062 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.062 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:19.062 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.062 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.062 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.062 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:87470 7.8 e117/112 7:1f3b224f:::benchmark_data_smithi114_78432_object87469:head v 117'5594, mlcod=88'417)", 2024-09-21T12:31:19.062 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:41.472523+0000", 2024-09-21T12:31:19.062 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 517.51056168399998, 2024-09-21T12:31:19.062 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.06343451, 2024-09-21T12:31:19.062 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:19.063 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:19.063 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:19.063 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:19.063 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.063 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:19.063 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472523+0000", 2024-09-21T12:31:19.063 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.063 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.063 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.063 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:19.063 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472523+0000", 2024-09-21T12:31:19.063 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.063 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.064 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.064 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:19.064 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472524+0000", 2024-09-21T12:31:19.064 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7399999999999991e-07 2024-09-21T12:31:19.064 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.064 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.064 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:19.064 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472596+0000", 2024-09-21T12:31:19.064 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2152999999999998e-05 2024-09-21T12:31:19.064 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.064 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.064 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:19.064 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472606+0000", 2024-09-21T12:31:19.064 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0485e-05 2024-09-21T12:31:19.064 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.064 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.065 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:19.065 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.472613+0000", 2024-09-21T12:31:19.065 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5590000000000002e-06 2024-09-21T12:31:19.065 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.065 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.065 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:19.065 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.484029+0000", 2024-09-21T12:31:19.065 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.01141587 2024-09-21T12:31:19.065 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.065 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.065 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:19.065 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.484053+0000", 2024-09-21T12:31:19.065 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4559999999999999e-05 2024-09-21T12:31:19.065 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.065 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.066 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:19.066 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.535915+0000", 2024-09-21T12:31:19.066 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.051861431 2024-09-21T12:31:19.066 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.066 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.066 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:19.066 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.535957+0000", 2024-09-21T12:31:19.066 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2478000000000002e-05 2024-09-21T12:31:19.066 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.066 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:19.066 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.066 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.066 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.066 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:87486 7.8 e117/112 7:14e05f7b:::benchmark_data_smithi114_78432_object87485:head v 117'5595, mlcod=88'417)", 2024-09-21T12:31:19.067 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:41.475017+0000", 2024-09-21T12:31:19.067 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 517.50806734800005, 2024-09-21T12:31:19.067 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.060970067000000003, 2024-09-21T12:31:19.067 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:19.067 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:19.067 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:19.067 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:19.067 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.067 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:19.067 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475017+0000", 2024-09-21T12:31:19.067 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.067 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.067 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.067 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:19.068 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475017+0000", 2024-09-21T12:31:19.068 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.068 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.068 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.068 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:19.068 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475018+0000", 2024-09-21T12:31:19.068 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5799999999999999e-07 2024-09-21T12:31:19.068 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.068 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.068 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:19.068 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475150+0000", 2024-09-21T12:31:19.068 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013201700000000001 2024-09-21T12:31:19.068 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.068 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.068 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:19.069 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475159+0000", 2024-09-21T12:31:19.069 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5840000000000001e-06 2024-09-21T12:31:19.069 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.069 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.069 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:19.069 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.475166+0000", 2024-09-21T12:31:19.069 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3589999999999996e-06 2024-09-21T12:31:19.069 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.069 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.069 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:19.069 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.525196+0000", 2024-09-21T12:31:19.069 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.050030507000000002 2024-09-21T12:31:19.069 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.069 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.069 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:19.070 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.525214+0000", 2024-09-21T12:31:19.070 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7884e-05 2024-09-21T12:31:19.070 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.070 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.070 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:19.070 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.535960+0000", 2024-09-21T12:31:19.070 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.010746143999999999 2024-09-21T12:31:19.070 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.070 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.070 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:19.070 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.535987+0000", 2024-09-21T12:31:19.070 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6914e-05 2024-09-21T12:31:19.070 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.070 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:19.071 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.071 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.071 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.071 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDPGPush(7.8 117/112 [PushOp(7:1de1c2d4:::benchmark_data_smithi114_78432_object6336:head, version: 88'418, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:1de1c2d4:::benchmark_data_smithi114_78432_object6336:head@88'418, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T12:31:19.071 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:41.485571+0000", 2024-09-21T12:31:19.071 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 517.49751345300001, 2024-09-21T12:31:19.071 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.046712901000000001, 2024-09-21T12:31:19.071 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:19.071 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:19.071 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:31:19.071 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:19.071 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.071 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:19.071 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485571+0000", 2024-09-21T12:31:19.072 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.072 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:19.072 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485571+0000", 2024-09-21T12:31:19.072 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.072 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:19.072 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485572+0000", 2024-09-21T12:31:19.072 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7299999999999995e-07 2024-09-21T12:31:19.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.072 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:19.073 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485643+0000", 2024-09-21T12:31:19.073 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0976999999999995e-05 2024-09-21T12:31:19.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.073 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.073 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:19.073 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485655+0000", 2024-09-21T12:31:19.073 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2904e-05 2024-09-21T12:31:19.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.073 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.073 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:19.073 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.485665+0000", 2024-09-21T12:31:19.073 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6709999999999994e-06 2024-09-21T12:31:19.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.074 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.074 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:19.074 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.530794+0000", 2024-09-21T12:31:19.074 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.045129295 2024-09-21T12:31:19.074 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.074 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.074 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:19.074 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.530802+0000", 2024-09-21T12:31:19.074 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5059999999999998e-06 2024-09-21T12:31:19.074 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.074 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.074 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:19.074 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.532284+0000", 2024-09-21T12:31:19.074 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014818749999999999 2024-09-21T12:31:19.075 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.075 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:19.075 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.075 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.075 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.075 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.24695.0:87497 7.8 e117/112 7:19475f57:::benchmark_data_smithi114_78432_object87496:head v 117'5596, mlcod=88'417)", 2024-09-21T12:31:19.075 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:41.494916+0000", 2024-09-21T12:31:19.075 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 517.48816797400002, 2024-09-21T12:31:19.075 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.041122084000000003, 2024-09-21T12:31:19.075 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:19.075 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:19.075 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:19.075 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:19.075 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.076 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:19.076 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.494916+0000", 2024-09-21T12:31:19.076 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.076 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.076 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.076 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:19.076 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.494916+0000", 2024-09-21T12:31:19.076 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:19.076 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.076 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.076 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:19.076 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.494918+0000", 2024-09-21T12:31:19.076 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.719e-06 2024-09-21T12:31:19.076 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.076 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.076 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:19.077 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.494975+0000", 2024-09-21T12:31:19.077 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6981999999999998e-05 2024-09-21T12:31:19.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.077 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.077 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:19.077 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.525059+0000", 2024-09-21T12:31:19.077 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.030083526999999999 2024-09-21T12:31:19.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.077 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.077 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:19.077 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.525071+0000", 2024-09-21T12:31:19.077 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2438e-05 2024-09-21T12:31:19.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.077 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.078 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:19.078 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.532312+0000", 2024-09-21T12:31:19.078 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0072404909999999999 2024-09-21T12:31:19.078 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.078 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.078 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:19.078 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.532336+0000", 2024-09-21T12:31:19.078 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4090000000000001e-05 2024-09-21T12:31:19.078 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.078 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.078 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:19.078 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.536013+0000", 2024-09-21T12:31:19.078 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003677825 2024-09-21T12:31:19.078 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:19.078 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:19.079 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:19.079 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:41.536038+0000", 2024-09-21T12:31:19.079 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5012e-05 2024-09-21T12:31:19.079 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.079 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:19.079 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.079 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:19.079 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:19.079 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:31:19.082 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:19.107 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:19.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:19.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:19.208 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:19.233 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:19.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:19.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:19.336 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:19.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:19.372 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:19.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:19.432 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:19 smithi114 ceph-mon[40867]: pgmap v544: 188 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 182 active+clean; 3.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 11045/159825 objects misplaced (6.911%) 2024-09-21T12:31:19.432 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:19 smithi114 ceph-mon[40867]: osdmap e301: 8 total, 8 up, 7 in 2024-09-21T12:31:19.432 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:19 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1482581998' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:31:19.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:19 smithi114 ceph-mon[30324]: pgmap v544: 188 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 182 active+clean; 3.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 11045/159825 objects misplaced (6.911%) 2024-09-21T12:31:19.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:19 smithi114 ceph-mon[30324]: osdmap e301: 8 total, 8 up, 7 in 2024-09-21T12:31:19.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:19 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1482581998' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:31:19.474 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:19 smithi183 ceph-mon[33277]: pgmap v544: 188 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 182 active+clean; 3.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 11045/159825 objects misplaced (6.911%) 2024-09-21T12:31:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:19 smithi183 ceph-mon[33277]: osdmap e301: 8 total, 8 up, 7 in 2024-09-21T12:31:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:19 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1482581998' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:31:19.499 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:19.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:19.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:19.601 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:19.638 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:19.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:19.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:19.740 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:19.774 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:19.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:19.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:19.876 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:19.905 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:19.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:19.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:20.006 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:20.041 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:20.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:20.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:20.143 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:20.147 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T12:31:20.171 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:20.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:20.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:20.264 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T12:31:20.273 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:20.335 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:20.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:20.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:20.437 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:20.467 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:20.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:20.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:20 smithi183 ceph-mon[33277]: osdmap e302: 8 total, 8 up, 7 in 2024-09-21T12:31:20.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:20 smithi114 ceph-mon[30324]: osdmap e302: 8 total, 8 up, 7 in 2024-09-21T12:31:20.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:20 smithi114 ceph-mon[40867]: osdmap e302: 8 total, 8 up, 7 in 2024-09-21T12:31:20.568 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:20.588 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:31:20.616 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:20.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:20.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:20.623 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:31:20.718 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:20.751 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:20.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:20.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:20.852 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:20.881 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:20.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:20.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:20.983 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:21.004 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:21.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:21.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:21.106 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:21.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:21.140 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:21.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:21.241 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:21.274 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:21.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:21.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:21.376 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:21.406 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:21.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:21.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:21.469 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:21 smithi183 ceph-mon[33277]: pgmap v547: 188 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 182 active+clean; 3.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.13k op/s; 10609/155910 objects misplaced (6.805%); 19 MiB/s, 305 objects/s recovering 2024-09-21T12:31:21.478 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:31:21.508 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:21.520 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:21 smithi114 ceph-mon[40867]: pgmap v547: 188 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 182 active+clean; 3.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.13k op/s; 10609/155910 objects misplaced (6.805%); 19 MiB/s, 305 objects/s recovering 2024-09-21T12:31:21.520 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:21 smithi114 ceph-mon[30324]: pgmap v547: 188 pgs: 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 182 active+clean; 3.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.13k op/s; 10609/155910 objects misplaced (6.805%); 19 MiB/s, 305 objects/s recovering 2024-09-21T12:31:21.534 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:21.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:21.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:21.636 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:21.639 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T12:31:21.639 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T12:31:21.666 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:21.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:21.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:21.759 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:31:21.759 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:31:21.759 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:21.759 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:201550 9.f 9:f14e5ae4:::benchmark_data_smithi114_133214_object77516:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-21T12:31:21.759 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:21.747602+0000", 2024-09-21T12:31:21.759 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0097629299999999995, 2024-09-21T12:31:21.759 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0097842190000000002, 2024-09-21T12:31:21.759 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:21.759 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:21.760 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:31:21.760 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:31:21.760 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:31:21.760 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:31:21.760 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 201550 2024-09-21T12:31:21.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:21.760 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:21.760 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:21.760 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:21.760 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:21.747602+0000", 2024-09-21T12:31:21.760 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:21.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:21.760 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:21.760 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:21.761 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:21.747602+0000", 2024-09-21T12:31:21.761 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:21.761 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:21.761 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:21.761 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:21.761 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:21.747604+0000", 2024-09-21T12:31:21.761 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7689999999999999e-06 2024-09-21T12:31:21.761 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:21.761 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:21.761 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:21.761 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:21.747608+0000", 2024-09-21T12:31:21.761 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7169999999999999e-06 2024-09-21T12:31:21.762 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:21.762 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:21.762 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:21.762 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:21.747620+0000", 2024-09-21T12:31:21.762 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1659e-05 2024-09-21T12:31:21.762 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:21.762 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:21.762 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:21.762 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:21.747627+0000", 2024-09-21T12:31:21.762 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7939999999999997e-06 2024-09-21T12:31:21.762 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:21.762 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:21.762 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:21.763 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:21.747823+0000", 2024-09-21T12:31:21.763 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000196627 2024-09-21T12:31:21.763 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:21.763 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:21.763 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:21.763 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:21.747921+0000", 2024-09-21T12:31:21.763 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7372000000000002e-05 2024-09-21T12:31:21.763 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:21.763 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:21.763 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:31:21.763 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:21.748020+0000", 2024-09-21T12:31:21.763 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9344000000000003e-05 2024-09-21T12:31:21.763 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:21.763 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:21.763 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:31:21.764 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:21.754434+0000", 2024-09-21T12:31:21.764 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0064134300000000003 2024-09-21T12:31:21.764 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:21.764 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:21.764 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:21.764 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:21.764 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:21.764 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:201555 9.f 9:f505fb56:::benchmark_data_smithi114_133214_object77521:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-09-21T12:31:21.764 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:21.749800+0000", 2024-09-21T12:31:21.764 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0075645440000000003, 2024-09-21T12:31:21.764 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0076319120000000002, 2024-09-21T12:31:21.765 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:21.765 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:21.765 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:31:21.765 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:31:21.765 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:31:21.765 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:31:21.765 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 201555 2024-09-21T12:31:21.765 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:21.765 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:21.765 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:21.765 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:21.765 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:21.749800+0000", 2024-09-21T12:31:21.765 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:21.765 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:21.765 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:21.766 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:21.766 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:21.749800+0000", 2024-09-21T12:31:21.766 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:21.766 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:21.766 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:21.766 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:21.766 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:21.749802+0000", 2024-09-21T12:31:21.766 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3230000000000001e-06 2024-09-21T12:31:21.766 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:21.766 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:21.766 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:21.766 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:21.749806+0000", 2024-09-21T12:31:21.766 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5260000000000004e-06 2024-09-21T12:31:21.766 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:21.766 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:21.767 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:21.767 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:21.749823+0000", 2024-09-21T12:31:21.767 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7193999999999998e-05 2024-09-21T12:31:21.767 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:21.767 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:21.767 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:21.767 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:21.749830+0000", 2024-09-21T12:31:21.767 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5259999999999999e-06 2024-09-21T12:31:21.767 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:21.767 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:21.767 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:21.767 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:21.752349+0000", 2024-09-21T12:31:21.767 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002518705 2024-09-21T12:31:21.767 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:21.767 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:21.752463+0000", 2024-09-21T12:31:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000114982 2024-09-21T12:31:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:31:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:21.752547+0000", 2024-09-21T12:31:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3605999999999999e-05 2024-09-21T12:31:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:31:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:21.754578+0000", 2024-09-21T12:31:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020305309999999999 2024-09-21T12:31:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:21.769 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:21.769 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-21T12:31:21.769 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 2 2024-09-21T12:31:21.769 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:31:21.769 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:21.847 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:21.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:21.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:21.949 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:21.960 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:31:21.977 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:21.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:21.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:22.078 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:22.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:22.121 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:22.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:22.219 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-21T12:31:22.222 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:22.272 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:22.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:22.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:22.374 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:22.402 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:22.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:22.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:22.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:22 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3150824729' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:31:22.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:22 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:31:22.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:22 smithi183 ceph-mon[33277]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:31:22.503 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:22.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:22 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3150824729' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:31:22.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:22 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:31:22.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:22 smithi114 ceph-mon[40867]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:31:22.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:22 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3150824729' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:31:22.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:22 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:31:22.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:22 smithi114 ceph-mon[30324]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:31:22.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:22.535 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:22.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:22.636 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:22.659 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:22.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:22.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:22.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:22.797 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:22.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:22.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:22.851 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T12:31:22.899 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:22.939 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:22.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:22.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:23.024 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:31:22 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:31:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:31:23.041 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:23.084 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:23.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:23.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:23.185 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:23.222 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:23.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:23.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:23.232 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:31:23.294 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:23 smithi114 ceph-mon[40867]: pgmap v548: 188 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 183 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.36k op/s; 9977/149160 objects misplaced (6.689%); 20 MiB/s, 316 objects/s recovering 2024-09-21T12:31:23.294 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:23 smithi114 ceph-mon[40867]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:31:23.295 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:23 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:31:23.295 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:23 smithi114 ceph-mon[40867]: osdmap e303: 8 total, 8 up, 7 in 2024-09-21T12:31:23.295 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:23 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2108898750' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:31:23.295 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:23 smithi114 ceph-mon[30324]: pgmap v548: 188 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 183 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.36k op/s; 9977/149160 objects misplaced (6.689%); 20 MiB/s, 316 objects/s recovering 2024-09-21T12:31:23.295 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:23 smithi114 ceph-mon[30324]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:31:23.295 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:23 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:31:23.295 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:23 smithi114 ceph-mon[30324]: osdmap e303: 8 total, 8 up, 7 in 2024-09-21T12:31:23.295 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:23 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2108898750' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:31:23.324 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:23.347 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:23.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:23.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:23.448 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:23.484 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:23.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:23.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:23.514 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:23 smithi183 ceph-mon[33277]: pgmap v548: 188 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 183 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.36k op/s; 9977/149160 objects misplaced (6.689%); 20 MiB/s, 316 objects/s recovering 2024-09-21T12:31:23.514 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:23 smithi183 ceph-mon[33277]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:31:23.514 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:23 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:31:23.514 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:23 smithi183 ceph-mon[33277]: osdmap e303: 8 total, 8 up, 7 in 2024-09-21T12:31:23.515 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:23 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2108898750' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:31:23.585 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:23.622 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:23.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:23.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:23.724 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:23.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:23.767 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:23.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:23.859 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:31:23.859 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:31:23.868 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:23.895 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:23.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:23.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:23.996 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:24.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:24.041 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:24.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:24.065 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:31:24.065 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:31:24.065 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:31:24.065 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:31:24.065 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:31:24.065 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:31:24.066 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:31:24.066 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:31:24.066 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:31:24.066 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:31:24.066 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:31:24.067 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:31:24.067 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:31:24.067 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:31:24.067 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:31:24.067 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:31:24.142 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:24.177 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:24.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:24.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:24.279 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:24.317 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:24.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:24.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:24.418 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:24.452 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:24.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:24.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:24.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:24 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:31:24.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:24 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:31:24.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:24 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:31:24.553 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:24.592 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:24.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:24.593 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T12:31:24.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:24.694 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:24.721 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:24.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:24.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:24.824 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:24.854 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:24.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:24.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:24.956 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:24.998 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:24.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:24.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:25.010 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:31:25.099 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:25.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:25.136 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:25.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:25.237 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:25.279 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:25.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:25.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:25.381 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:25.409 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:25.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:25.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:25.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:25 smithi183 ceph-mon[33277]: pgmap v550: 188 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 183 active+clean; 2.8 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.21k op/s; 7863/138003 objects misplaced (5.698%); 10 MiB/s, 163 objects/s recovering 2024-09-21T12:31:25.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:25 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:25 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:25 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:25 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:25 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:25 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:25 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:25 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:25 smithi183 ceph-mon[33277]: Adjusting osd_memory_target on smithi183 to 3248M 2024-09-21T12:31:25.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:25 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:31:25.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:25 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:31:25.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:25 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:31:25.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:25 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:25 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:25 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:25 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:25 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:25 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:25 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:25 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:25 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:31:25.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:25 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:31:25.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:25 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:31:25.491 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:25 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:31:25.505 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:31:25.510 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:25.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[30324]: pgmap v550: 188 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 183 active+clean; 2.8 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.21k op/s; 7863/138003 objects misplaced (5.698%); 10 MiB/s, 163 objects/s recovering 2024-09-21T12:31:25.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[30324]: Adjusting osd_memory_target on smithi183 to 3248M 2024-09-21T12:31:25.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:31:25.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:31:25.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:31:25.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:31:25.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:31:25.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:31:25.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:31:25.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[40867]: pgmap v550: 188 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 183 active+clean; 2.8 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.21k op/s; 7863/138003 objects misplaced (5.698%); 10 MiB/s, 163 objects/s recovering 2024-09-21T12:31:25.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[40867]: Adjusting osd_memory_target on smithi183 to 3248M 2024-09-21T12:31:25.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:31:25.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:31:25.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:31:25.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T12:31:25.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:31:25.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:31:25.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:31:25.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:25 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:31:25.534 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:25.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:25.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:25.636 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:25.658 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:25.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:25.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:25.759 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:25.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:25.782 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:25.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:25.883 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:25.904 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:25.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:25.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:26.005 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:26.027 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:26.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:26.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:26.128 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:26.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:26.150 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:26.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:26.251 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:26.277 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:26.278 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T12:31:26.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:26.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:26.379 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:26.416 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:26.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:26.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:26.517 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:26.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:26.555 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:26.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:26.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:26 smithi114 ceph-mon[30324]: Adjusting osd_memory_target on smithi114 to 3248M 2024-09-21T12:31:26.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:26 smithi114 ceph-mon[30324]: osdmap e304: 8 total, 8 up, 7 in 2024-09-21T12:31:26.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:26 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1141588004' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:31:26.633 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:26 smithi114 ceph-mon[40867]: Adjusting osd_memory_target on smithi114 to 3248M 2024-09-21T12:31:26.633 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:26 smithi114 ceph-mon[40867]: osdmap e304: 8 total, 8 up, 7 in 2024-09-21T12:31:26.633 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:26 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1141588004' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:31:26.657 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:26.687 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:26.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:26.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:26.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:26 smithi183 ceph-mon[33277]: Adjusting osd_memory_target on smithi114 to 3248M 2024-09-21T12:31:26.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:26 smithi183 ceph-mon[33277]: osdmap e304: 8 total, 8 up, 7 in 2024-09-21T12:31:26.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:26 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1141588004' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:31:26.791 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:26.828 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:26.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:26.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:26.930 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:26.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:26.972 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:26.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:27.041 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:31:27.060 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T12:31:27.072 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:27.101 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:27.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:27.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:27.203 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:27.238 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:31:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=cleanup t=2024-09-21T12:31:26.989130839Z level=info msg="Completed cleanup jobs" duration=29.785889ms 2024-09-21T12:31:27.238 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:31:27 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=plugins.update.checker t=2024-09-21T12:31:27.098204092Z level=info msg="Update check succeeded" duration=60.942135ms 2024-09-21T12:31:27.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:27.270 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:27.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:27.371 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:27.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:27.425 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:27.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:27.489 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T12:31:27.504 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:31:27.527 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:27.544 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:31:27.552 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:27.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:27.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:27.630 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:27 smithi114 ceph-mon[40867]: pgmap v552: 188 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 183 active+clean; 2.8 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 775 B/s rd, 0 B/s wr, 904 op/s; 7863/138003 objects misplaced (5.698%); 9.3 MiB/s, 148 objects/s recovering 2024-09-21T12:31:27.630 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:27 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1141588004' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T12:31:27.630 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:27 smithi114 ceph-mon[40867]: osdmap e305: 8 total, 8 up, 7 in 2024-09-21T12:31:27.630 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:27 smithi114 ceph-mon[40867]: osdmap e306: 8 total, 8 up, 7 in 2024-09-21T12:31:27.631 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:27 smithi114 ceph-mon[30324]: pgmap v552: 188 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 183 active+clean; 2.8 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 775 B/s rd, 0 B/s wr, 904 op/s; 7863/138003 objects misplaced (5.698%); 9.3 MiB/s, 148 objects/s recovering 2024-09-21T12:31:27.631 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:27 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1141588004' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T12:31:27.631 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:27 smithi114 ceph-mon[30324]: osdmap e305: 8 total, 8 up, 7 in 2024-09-21T12:31:27.631 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:27 smithi114 ceph-mon[30324]: osdmap e306: 8 total, 8 up, 7 in 2024-09-21T12:31:27.653 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:27.726 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:27.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:27.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:27.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:31:27 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:31:27.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:27 smithi183 ceph-mon[33277]: pgmap v552: 188 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 183 active+clean; 2.8 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 775 B/s rd, 0 B/s wr, 904 op/s; 7863/138003 objects misplaced (5.698%); 9.3 MiB/s, 148 objects/s recovering 2024-09-21T12:31:27.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:27 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1141588004' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T12:31:27.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:27 smithi183 ceph-mon[33277]: osdmap e305: 8 total, 8 up, 7 in 2024-09-21T12:31:27.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:27 smithi183 ceph-mon[33277]: osdmap e306: 8 total, 8 up, 7 in 2024-09-21T12:31:27.827 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:27.858 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:27.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:27.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:27.959 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:27.984 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:27.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:27.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:28.086 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:28.124 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:28.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:28.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:28.183 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:31:28.183 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:31:28.215 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:31:28.215 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:31:28.225 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:28.232 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:31:28.232 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:31:28.233 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:31:28.247 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:28.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:28.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:28.251 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:31:28.251 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:31:28.269 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:31:28.269 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:31:28.295 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:31:28.295 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:31:28.328 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:31:28.328 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:31:28.349 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:28.353 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:31:28.353 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:31:28.369 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:28.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:28.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:28.470 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:28.503 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:28.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:28.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:28.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:28 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:28.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:28 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2999407105' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:31:28.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:28 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:28.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:28 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2999407105' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:31:28.604 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:28.640 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:28.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:28.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:28.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:28 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:28.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:28 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2999407105' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:31:28.742 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:28.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:28.767 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:28.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:28.868 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:28.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:28.900 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:28.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:29.001 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:29.024 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:29.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:29.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:29.126 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:29.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:29.158 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:29.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:29.184 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:31:29.184 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:31:29.260 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:29.282 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:29.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:29.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:29.383 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:29.405 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:29.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:29.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:29.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:29 smithi183 ceph-mon[33277]: pgmap v555: 188 pgs: 4 active+remapped+backfill_toofull, 184 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 916 B/s rd, 0 B/s wr, 1.39k op/s; 6755/123594 objects misplaced (5.465%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T12:31:29.507 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:29.539 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:29.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:29.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:29.641 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:29.662 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:29 smithi114 ceph-mon[40867]: pgmap v555: 188 pgs: 4 active+remapped+backfill_toofull, 184 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 916 B/s rd, 0 B/s wr, 1.39k op/s; 6755/123594 objects misplaced (5.465%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T12:31:29.662 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:29 smithi114 ceph-mon[30324]: pgmap v555: 188 pgs: 4 active+remapped+backfill_toofull, 184 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 916 B/s rd, 0 B/s wr, 1.39k op/s; 6755/123594 objects misplaced (5.465%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T12:31:29.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:29.663 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:29.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:29.764 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:29.784 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:29.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:29.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:29.886 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:29.924 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:29.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:29.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:30.025 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:30.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:30.057 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:30.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:30.158 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:30.192 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:30.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:30.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:30.294 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:30.317 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:30.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:30.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:30.419 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:30.454 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:30.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:30.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:30.556 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:30.558 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T12:31:30.589 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:30.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:30.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:30.691 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:30.727 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:30.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:30.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:30.829 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:30.854 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:30.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:30.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:30.924 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:31:30.955 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:30.987 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:30.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:30.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:31.088 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:31.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:31.130 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:31.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:31.231 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:31.266 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:31.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:31.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:31.367 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:31.375 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:31 smithi183 ceph-mon[33277]: pgmap v556: 188 pgs: 4 active+remapped+backfill_toofull, 184 active+clean; 2.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 975 op/s; 6582/120450 objects misplaced (5.465%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T12:31:31.406 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:31.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:31.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:31.510 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:31.513 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:31:31.513 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-21T12:31:31.513 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 0 2024-09-21T12:31:31.513 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:31:31.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:31 smithi114 ceph-mon[40867]: pgmap v556: 188 pgs: 4 active+remapped+backfill_toofull, 184 active+clean; 2.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 975 op/s; 6582/120450 objects misplaced (5.465%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T12:31:31.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:31 smithi114 ceph-mon[30324]: pgmap v556: 188 pgs: 4 active+remapped+backfill_toofull, 184 active+clean; 2.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 975 op/s; 6582/120450 objects misplaced (5.465%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T12:31:31.546 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:31.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:31.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:31.648 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:31.685 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:31.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:31.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:31.787 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:31.807 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:31.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:31.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:31.909 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:31.948 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:31.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:31.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:32.050 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:32.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:32.083 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:32.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:32.184 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:32.215 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T12:31:32.216 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:32.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:32.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:32.317 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:32.350 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:32.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:32.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:32.452 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:32 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:31:32.490 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:32.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:32.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:32.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:32 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:31:32.551 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:32 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:31:32.591 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:32.609 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:31:32.628 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:32.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:32.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:32.729 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:32.754 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:32.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:32.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:32.856 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:32.893 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:32.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:32.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:32.994 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:33.024 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:31:32 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:31:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:31:33.027 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:33.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:33.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:33.129 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:33.195 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:33.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:33.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:33.297 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:33.329 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:31:33.329 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-21T12:31:33.329 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-21T12:31:33.329 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-21T12:31:33.330 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:31:33.336 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:33.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:33.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:33.438 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:33.469 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:33.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:33.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:33.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:33 smithi183 ceph-mon[33277]: pgmap v557: 188 pgs: 4 active+remapped+backfill_toofull, 184 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.27k op/s; 6192/113220 objects misplaced (5.469%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T12:31:33.571 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:33.608 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:33.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:33.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:33.710 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:33.750 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:33.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:33.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:33.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:33 smithi114 ceph-mon[30324]: pgmap v557: 188 pgs: 4 active+remapped+backfill_toofull, 184 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.27k op/s; 6192/113220 objects misplaced (5.469%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T12:31:33.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:33 smithi114 ceph-mon[40867]: pgmap v557: 188 pgs: 4 active+remapped+backfill_toofull, 184 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.27k op/s; 6192/113220 objects misplaced (5.469%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T12:31:33.852 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:33.885 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:33.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:33.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:33.986 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:34.009 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:34.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:34.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:34.010 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T12:31:34.110 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:34.131 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:34.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:34.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:34.233 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:34.256 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:34.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:34.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:34.304 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:31:34.357 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:34.396 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:34.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:34.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:34.497 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:34.522 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:34.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:34.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:34.624 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:34.656 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:34.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:34.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:34.758 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:34.788 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:34.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:34.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:34.889 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:34.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:34.931 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:34.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:35.032 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:35.068 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:35.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:35.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:35.170 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:35.201 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:35.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:35.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:35.303 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:35.339 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:35.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:35.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:35.436 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:35 smithi183 ceph-mon[33277]: pgmap v558: 188 pgs: 4 active+remapped+backfill_toofull, 184 active+clean; 2.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s; 5556/102126 objects misplaced (5.440%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T12:31:35.441 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:35.473 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:35.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:35.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:35.483 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:31:35.483 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-21T12:31:35.483 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-21T12:31:35.484 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:31:35.484 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.484 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14478.0:1119 5.1f 5:f95f44c2:::notify.0:head [watch ping cookie 94620848892160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-21T12:31:35.484 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:40.364371+0000", 2024-09-21T12:31:35.484 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 595.11566085300001, 2024-09-21T12:31:35.484 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78371777300000001, 2024-09-21T12:31:35.484 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:35.484 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:35.484 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:31:35.484 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:31:35.484 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14478", 2024-09-21T12:31:35.484 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:31:35.484 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 1119 2024-09-21T12:31:35.484 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.485 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:35.485 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.485 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:35.485 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364371+0000", 2024-09-21T12:31:35.485 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.485 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.485 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.485 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:35.485 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364371+0000", 2024-09-21T12:31:35.485 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.486 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.486 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.486 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:35.486 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364372+0000", 2024-09-21T12:31:35.486 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6199999999999997e-07 2024-09-21T12:31:35.486 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.486 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.486 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:35.486 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364376+0000", 2024-09-21T12:31:35.486 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8099999999999999e-06 2024-09-21T12:31:35.486 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.486 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.486 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:35.487 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364384+0000", 2024-09-21T12:31:35.487 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2870000000000004e-06 2024-09-21T12:31:35.487 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.487 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.487 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:35.487 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364393+0000", 2024-09-21T12:31:35.487 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5820000000000002e-06 2024-09-21T12:31:35.487 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.487 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.487 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:35.487 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364557+0000", 2024-09-21T12:31:35.487 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016425399999999999 2024-09-21T12:31:35.487 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.487 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.487 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:31:35.488 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364573+0000", 2024-09-21T12:31:35.488 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6286999999999998e-05 2024-09-21T12:31:35.488 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.488 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.488 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:35.488 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.144878+0000", 2024-09-21T12:31:35.488 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78030514600000001 2024-09-21T12:31:35.488 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.488 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.488 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for degraded object", 2024-09-21T12:31:35.488 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.145417+0000", 2024-09-21T12:31:35.488 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00053866500000000002 2024-09-21T12:31:35.488 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.488 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.488 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:35.489 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.147869+0000", 2024-09-21T12:31:35.489 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024518779999999998 2024-09-21T12:31:35.489 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.489 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.489 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:35.489 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.147939+0000", 2024-09-21T12:31:35.489 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0574000000000006e-05 2024-09-21T12:31:35.489 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.489 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.489 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:35.489 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.148089+0000", 2024-09-21T12:31:35.489 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014952800000000001 2024-09-21T12:31:35.489 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.489 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:35.489 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.489 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.490 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.490 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14478.0:1120 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch ping cookie 94620852177152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-21T12:31:35.490 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:40.364419+0000", 2024-09-21T12:31:35.490 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 595.115613177, 2024-09-21T12:31:35.490 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78472206099999997, 2024-09-21T12:31:35.490 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:35.490 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:35.490 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:31:35.490 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:31:35.490 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14478", 2024-09-21T12:31:35.490 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:31:35.490 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 1120 2024-09-21T12:31:35.490 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.490 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:35.490 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.491 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:35.491 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364419+0000", 2024-09-21T12:31:35.491 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.491 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.491 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.491 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:35.491 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364419+0000", 2024-09-21T12:31:35.491 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.491 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.491 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.491 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:35.491 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364419+0000", 2024-09-21T12:31:35.491 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5400000000000002e-07 2024-09-21T12:31:35.491 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.492 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.492 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:35.492 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364426+0000", 2024-09-21T12:31:35.492 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8650000000000003e-06 2024-09-21T12:31:35.492 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.492 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.492 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:35.492 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364434+0000", 2024-09-21T12:31:35.492 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2309999999999996e-06 2024-09-21T12:31:35.492 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.492 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.492 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:35.492 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364441+0000", 2024-09-21T12:31:35.493 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0929999999999999e-06 2024-09-21T12:31:35.493 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.493 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.493 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:35.493 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364520+0000", 2024-09-21T12:31:35.493 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9487999999999995e-05 2024-09-21T12:31:35.493 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.493 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.493 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:31:35.493 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364543+0000", 2024-09-21T12:31:35.493 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.296e-05 2024-09-21T12:31:35.493 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.493 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.493 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:35.494 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.147074+0000", 2024-09-21T12:31:35.494 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78253067600000004 2024-09-21T12:31:35.494 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.494 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.494 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for degraded object", 2024-09-21T12:31:35.494 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.147381+0000", 2024-09-21T12:31:35.494 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00030698200000000002 2024-09-21T12:31:35.494 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.494 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.494 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:35.494 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.148992+0000", 2024-09-21T12:31:35.494 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001611267 2024-09-21T12:31:35.494 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.494 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.495 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:35.495 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.149064+0000", 2024-09-21T12:31:35.495 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2058999999999996e-05 2024-09-21T12:31:35.495 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.495 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.495 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:35.495 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.149141+0000", 2024-09-21T12:31:35.495 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6985999999999998e-05 2024-09-21T12:31:35.495 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.495 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:35.495 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.495 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.495 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.495 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14478.0:1124 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94620848892160 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:31:35.496 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:40.364832+0000", 2024-09-21T12:31:35.496 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 595.11520012599999, 2024-09-21T12:31:35.496 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78486121600000003, 2024-09-21T12:31:35.496 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:35.496 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:35.496 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:31:35.496 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:31:35.496 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14478", 2024-09-21T12:31:35.496 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:31:35.496 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 1124 2024-09-21T12:31:35.496 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.496 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:35.496 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.496 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:35.496 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364832+0000", 2024-09-21T12:31:35.497 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.497 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.497 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.497 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:35.497 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364832+0000", 2024-09-21T12:31:35.497 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.497 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.497 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.498 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:35.498 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364833+0000", 2024-09-21T12:31:35.498 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3900000000000004e-07 2024-09-21T12:31:35.498 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.498 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.498 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:35.498 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364838+0000", 2024-09-21T12:31:35.498 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1909999999999999e-06 2024-09-21T12:31:35.498 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.498 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.498 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:35.498 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364846+0000", 2024-09-21T12:31:35.498 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7000000000000008e-06 2024-09-21T12:31:35.499 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.499 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.499 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:35.499 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364850+0000", 2024-09-21T12:31:35.499 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.921e-06 2024-09-21T12:31:35.499 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.499 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.499 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:35.499 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364911+0000", 2024-09-21T12:31:35.499 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0636e-05 2024-09-21T12:31:35.499 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.499 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.499 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:31:35.500 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364922+0000", 2024-09-21T12:31:35.500 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0964e-05 2024-09-21T12:31:35.500 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.500 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.500 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:35.500 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.145447+0000", 2024-09-21T12:31:35.500 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78052487599999998 2024-09-21T12:31:35.500 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.500 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.500 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for degraded object", 2024-09-21T12:31:35.500 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.145493+0000", 2024-09-21T12:31:35.500 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5745000000000003e-05 2024-09-21T12:31:35.500 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.500 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.501 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:35.501 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.148124+0000", 2024-09-21T12:31:35.501 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0026310719999999999 2024-09-21T12:31:35.501 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.501 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.501 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:35.501 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.148191+0000", 2024-09-21T12:31:35.501 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7447999999999995e-05 2024-09-21T12:31:35.501 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.501 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.501 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:35.501 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.149693+0000", 2024-09-21T12:31:35.501 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001501924 2024-09-21T12:31:35.501 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.502 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:35.502 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.502 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.502 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.502 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14478.0:1125 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94620852177152 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T12:31:35.502 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:40.364865+0000", 2024-09-21T12:31:35.502 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 595.11516661300004, 2024-09-21T12:31:35.502 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78551680400000001, 2024-09-21T12:31:35.502 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:35.502 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:35.502 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:31:35.502 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:31:35.502 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14478", 2024-09-21T12:31:35.502 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:31:35.503 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 1125 2024-09-21T12:31:35.503 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.503 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:35.503 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.503 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:35.503 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364865+0000", 2024-09-21T12:31:35.503 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.503 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.503 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.503 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:35.503 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364865+0000", 2024-09-21T12:31:35.503 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.503 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.503 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.504 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:35.504 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364866+0000", 2024-09-21T12:31:35.504 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6499999999999999e-07 2024-09-21T12:31:35.504 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.504 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.504 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:35.504 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364868+0000", 2024-09-21T12:31:35.504 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.39e-06 2024-09-21T12:31:35.504 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.504 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.504 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:35.504 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364882+0000", 2024-09-21T12:31:35.504 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3890000000000001e-05 2024-09-21T12:31:35.504 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.505 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.505 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:35.505 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364888+0000", 2024-09-21T12:31:35.505 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3940000000000004e-06 2024-09-21T12:31:35.505 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.505 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.505 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:35.505 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364953+0000", 2024-09-21T12:31:35.505 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5877000000000006e-05 2024-09-21T12:31:35.505 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.505 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.505 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:31:35.505 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:40.364963+0000", 2024-09-21T12:31:35.505 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5349999999999993e-06 2024-09-21T12:31:35.505 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.506 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.506 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:35.506 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.147420+0000", 2024-09-21T12:31:35.506 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.78245673400000004 2024-09-21T12:31:35.506 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.506 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.506 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for degraded object", 2024-09-21T12:31:35.506 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.147469+0000", 2024-09-21T12:31:35.506 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9699e-05 2024-09-21T12:31:35.506 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.506 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.506 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:35.506 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.149190+0000", 2024-09-21T12:31:35.506 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001720606 2024-09-21T12:31:35.506 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.507 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.507 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:35.507 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.149267+0000", 2024-09-21T12:31:35.507 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7194000000000007e-05 2024-09-21T12:31:35.507 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.507 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.507 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:35.507 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:41.150382+0000", 2024-09-21T12:31:35.507 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00111502 2024-09-21T12:31:35.507 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.507 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:35.507 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.507 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.507 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.508 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:24811 7.7 7:ec8774ff:::benchmark_data_smithi114_78432_object24810:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T12:31:35.508 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:42.445492+0000", 2024-09-21T12:31:35.508 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 593.03453968199995, 2024-09-21T12:31:35.508 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.145669727, 2024-09-21T12:31:35.508 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:35.508 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:35.508 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:35.508 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:31:35.508 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:31:35.508 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:31:35.508 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 24811 2024-09-21T12:31:35.508 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.508 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:35.508 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.508 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:35.509 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445492+0000", 2024-09-21T12:31:35.509 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.509 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.509 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.509 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:35.509 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445492+0000", 2024-09-21T12:31:35.509 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.509 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.509 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.509 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:35.509 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445494+0000", 2024-09-21T12:31:35.509 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5460000000000001e-06 2024-09-21T12:31:35.509 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.509 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.509 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:35.510 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445541+0000", 2024-09-21T12:31:35.510 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6687999999999998e-05 2024-09-21T12:31:35.510 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.510 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.510 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:35.510 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445553+0000", 2024-09-21T12:31:35.510 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2517999999999999e-05 2024-09-21T12:31:35.510 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.510 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.510 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:35.510 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445560+0000", 2024-09-21T12:31:35.510 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6579999999999996e-06 2024-09-21T12:31:35.510 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.511 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.511 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:35.511 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445625+0000", 2024-09-21T12:31:35.511 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5092000000000006e-05 2024-09-21T12:31:35.511 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.511 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.511 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:35.511 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445776+0000", 2024-09-21T12:31:35.511 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0001507 2024-09-21T12:31:35.511 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.511 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.511 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,3", 2024-09-21T12:31:35.511 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.445923+0000", 2024-09-21T12:31:35.511 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014760199999999999 2024-09-21T12:31:35.512 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.512 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.512 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:31:35.512 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.446851+0000", 2024-09-21T12:31:35.512 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00092788099999999997 2024-09-21T12:31:35.512 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.512 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.512 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.512 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.590740+0000", 2024-09-21T12:31:35.512 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14388880500000001 2024-09-21T12:31:35.512 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.512 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.512 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.512 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.591050+0000", 2024-09-21T12:31:35.513 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000310217 2024-09-21T12:31:35.513 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.513 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.513 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:35.513 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.591109+0000", 2024-09-21T12:31:35.513 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9188000000000003e-05 2024-09-21T12:31:35.513 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.513 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.513 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:35.513 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:42.591162+0000", 2024-09-21T12:31:35.513 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2831999999999998e-05 2024-09-21T12:31:35.513 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.513 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:35.514 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.514 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.514 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.514 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:45498 7.7 7:e1336a67:::benchmark_data_smithi114_78432_object45497:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T12:31:35.514 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:59.293850+0000", 2024-09-21T12:31:35.514 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 576.18618175899996, 2024-09-21T12:31:35.514 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15681044, 2024-09-21T12:31:35.514 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:35.514 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:35.514 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:35.514 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:31:35.514 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:31:35.514 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:31:35.514 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 45498 2024-09-21T12:31:35.515 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.515 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:35.515 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.515 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:35.515 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293850+0000", 2024-09-21T12:31:35.515 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.515 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.515 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.515 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:35.515 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293850+0000", 2024-09-21T12:31:35.515 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.515 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.515 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.515 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:35.516 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293851+0000", 2024-09-21T12:31:35.516 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.096e-06 2024-09-21T12:31:35.516 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.516 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.516 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:35.516 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293907+0000", 2024-09-21T12:31:35.516 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5126e-05 2024-09-21T12:31:35.516 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.516 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.516 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:35.516 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293915+0000", 2024-09-21T12:31:35.516 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8419999999999994e-06 2024-09-21T12:31:35.516 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.516 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.516 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:35.517 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293920+0000", 2024-09-21T12:31:35.517 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5900000000000001e-06 2024-09-21T12:31:35.517 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.517 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.517 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:35.517 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.293959+0000", 2024-09-21T12:31:35.517 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9447999999999999e-05 2024-09-21T12:31:35.517 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.517 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.517 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:35.517 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294060+0000", 2024-09-21T12:31:35.517 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100247 2024-09-21T12:31:35.517 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.517 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.518 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,3", 2024-09-21T12:31:35.518 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.294145+0000", 2024-09-21T12:31:35.518 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5099999999999995e-05 2024-09-21T12:31:35.518 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.518 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.518 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:31:35.518 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.295566+0000", 2024-09-21T12:31:35.518 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014215429999999999 2024-09-21T12:31:35.518 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.518 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.518 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.518 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.449406+0000", 2024-09-21T12:31:35.518 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15383943999999999 2024-09-21T12:31:35.519 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.519 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.519 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.519 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.450563+0000", 2024-09-21T12:31:35.519 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00115749 2024-09-21T12:31:35.519 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.519 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.519 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:35.519 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.450621+0000", 2024-09-21T12:31:35.519 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8161000000000001e-05 2024-09-21T12:31:35.519 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.519 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.519 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:35.519 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.450661+0000", 2024-09-21T12:31:35.520 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9357000000000003e-05 2024-09-21T12:31:35.520 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.520 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:35.520 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.520 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.520 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.520 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:45514 7.c 7:3b027556:::benchmark_data_smithi114_78432_object45513:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T12:31:35.520 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:21:59.302539+0000", 2024-09-21T12:31:35.520 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 576.17749286599997, 2024-09-21T12:31:35.520 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.150159925, 2024-09-21T12:31:35.520 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:35.520 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:35.520 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:35.520 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:31:35.520 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:31:35.521 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:31:35.521 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 45514 2024-09-21T12:31:35.521 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.521 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:35.521 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.521 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:35.521 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302539+0000", 2024-09-21T12:31:35.521 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.521 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.521 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.521 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:35.521 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302539+0000", 2024-09-21T12:31:35.521 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.522 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.522 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.522 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:35.522 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302541+0000", 2024-09-21T12:31:35.522 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.376e-06 2024-09-21T12:31:35.522 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.522 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.522 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:35.522 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302592+0000", 2024-09-21T12:31:35.522 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1054999999999998e-05 2024-09-21T12:31:35.522 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.522 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.522 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:35.522 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302603+0000", 2024-09-21T12:31:35.523 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0883000000000001e-05 2024-09-21T12:31:35.523 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.523 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.523 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:35.523 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302608+0000", 2024-09-21T12:31:35.523 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5339999999999999e-06 2024-09-21T12:31:35.523 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.523 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.523 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:35.523 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302655+0000", 2024-09-21T12:31:35.523 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7125e-05 2024-09-21T12:31:35.523 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.523 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.523 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:35.524 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.302784+0000", 2024-09-21T12:31:35.524 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012886700000000001 2024-09-21T12:31:35.524 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.524 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.524 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:31:35.524 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.303003+0000", 2024-09-21T12:31:35.524 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00021902099999999999 2024-09-21T12:31:35.524 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.524 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.524 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:31:35.524 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.304304+0000", 2024-09-21T12:31:35.524 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0013008060000000001 2024-09-21T12:31:35.524 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.524 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.525 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.525 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.304689+0000", 2024-09-21T12:31:35.525 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000385208 2024-09-21T12:31:35.525 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.526 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.526 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.526 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.452633+0000", 2024-09-21T12:31:35.526 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14794360000000001 2024-09-21T12:31:35.526 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.526 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.526 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:35.526 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.452669+0000", 2024-09-21T12:31:35.526 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6202000000000003e-05 2024-09-21T12:31:35.526 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.526 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.526 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:35.526 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:21:59.452699+0000", 2024-09-21T12:31:35.527 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0247999999999999e-05 2024-09-21T12:31:35.527 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.527 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:35.527 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.527 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.527 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.527 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:49226 7.d 7:bcebfbbf:::benchmark_data_smithi114_78432_object49225:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:31:35.527 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:04.569255+0000", 2024-09-21T12:31:35.527 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 570.91077687899997, 2024-09-21T12:31:35.527 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14786647, 2024-09-21T12:31:35.527 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:35.527 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:35.527 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:35.527 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:31:35.528 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:31:35.528 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:31:35.528 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 49226 2024-09-21T12:31:35.528 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.528 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:35.528 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.528 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:35.528 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569255+0000", 2024-09-21T12:31:35.528 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.528 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.528 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.528 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:35.528 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569255+0000", 2024-09-21T12:31:35.528 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.529 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.529 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.529 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:35.529 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569257+0000", 2024-09-21T12:31:35.529 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3480000000000001e-06 2024-09-21T12:31:35.529 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.529 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.529 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:35.529 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569332+0000", 2024-09-21T12:31:35.529 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4969000000000007e-05 2024-09-21T12:31:35.529 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.529 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.529 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:35.530 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569344+0000", 2024-09-21T12:31:35.530 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2700999999999999e-05 2024-09-21T12:31:35.530 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.530 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.530 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:35.530 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569351+0000", 2024-09-21T12:31:35.530 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0369999999999997e-06 2024-09-21T12:31:35.530 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.530 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.530 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:35.530 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569427+0000", 2024-09-21T12:31:35.530 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6206999999999997e-05 2024-09-21T12:31:35.530 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.531 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.531 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:35.531 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569517+0000", 2024-09-21T12:31:35.531 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9875000000000003e-05 2024-09-21T12:31:35.531 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.531 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.531 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1", 2024-09-21T12:31:35.531 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569601+0000", 2024-09-21T12:31:35.531 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3849000000000003e-05 2024-09-21T12:31:35.531 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.531 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.531 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:31:35.531 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.573956+0000", 2024-09-21T12:31:35.532 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0043544689999999997 2024-09-21T12:31:35.532 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.532 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.532 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.532 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.575786+0000", 2024-09-21T12:31:35.532 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0018307480000000001 2024-09-21T12:31:35.532 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.532 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.532 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.532 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.717016+0000", 2024-09-21T12:31:35.532 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14122955000000001 2024-09-21T12:31:35.532 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.532 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.532 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:35.533 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.717072+0000", 2024-09-21T12:31:35.533 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6063000000000002e-05 2024-09-21T12:31:35.533 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.533 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.533 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:35.533 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.717122+0000", 2024-09-21T12:31:35.533 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9654000000000003e-05 2024-09-21T12:31:35.533 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.533 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:35.533 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.533 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.533 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.533 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:49229 7.d 7:b095dac2:::benchmark_data_smithi114_78432_object49228:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:31:35.534 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:04.569388+0000", 2024-09-21T12:31:35.534 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 570.91064434500004, 2024-09-21T12:31:35.534 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.147855664, 2024-09-21T12:31:35.534 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:35.534 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:35.534 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:35.534 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:31:35.534 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:31:35.534 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:31:35.534 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 49229 2024-09-21T12:31:35.534 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.534 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:35.534 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.535 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:35.535 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569388+0000", 2024-09-21T12:31:35.535 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.535 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.535 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.535 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:35.535 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569388+0000", 2024-09-21T12:31:35.535 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.535 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.535 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.535 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:35.535 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569388+0000", 2024-09-21T12:31:35.535 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.37e-07 2024-09-21T12:31:35.535 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.535 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.536 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:35.536 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569462+0000", 2024-09-21T12:31:35.536 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3244000000000005e-05 2024-09-21T12:31:35.536 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.536 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.536 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:35.536 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569471+0000", 2024-09-21T12:31:35.536 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2499999999999995e-06 2024-09-21T12:31:35.536 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.536 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.536 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:35.536 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.569477+0000", 2024-09-21T12:31:35.536 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.494e-06 2024-09-21T12:31:35.537 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.537 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.537 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:35.537 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.572570+0000", 2024-09-21T12:31:35.537 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0030923930000000001 2024-09-21T12:31:35.537 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.537 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.537 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:35.537 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.572664+0000", 2024-09-21T12:31:35.537 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4609000000000002e-05 2024-09-21T12:31:35.537 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.537 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.537 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1", 2024-09-21T12:31:35.537 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.572751+0000", 2024-09-21T12:31:35.537 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6725999999999996e-05 2024-09-21T12:31:35.537 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.538 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.538 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:31:35.538 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.574094+0000", 2024-09-21T12:31:35.538 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00134326 2024-09-21T12:31:35.538 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.538 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.538 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.538 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.576231+0000", 2024-09-21T12:31:35.538 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021369510000000002 2024-09-21T12:31:35.538 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.538 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.538 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.538 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.717166+0000", 2024-09-21T12:31:35.539 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14093446000000001 2024-09-21T12:31:35.539 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.539 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.539 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:35.539 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.717210+0000", 2024-09-21T12:31:35.539 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4239000000000001e-05 2024-09-21T12:31:35.539 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.539 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.539 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:35.539 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:04.717243+0000", 2024-09-21T12:31:35.539 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3401000000000001e-05 2024-09-21T12:31:35.539 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.539 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:35.540 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.540 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.540 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.540 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:49936 7.7 7:ee2fb7d7:::benchmark_data_smithi114_78432_object49935:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:31:35.540 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:05.207697+0000", 2024-09-21T12:31:35.540 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 570.27233547399999, 2024-09-21T12:31:35.540 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.180234797, 2024-09-21T12:31:35.540 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:35.540 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:35.540 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:35.540 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:31:35.540 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:31:35.540 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:31:35.541 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 49936 2024-09-21T12:31:35.541 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.541 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:35.541 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.541 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:35.541 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207697+0000", 2024-09-21T12:31:35.541 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.541 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.541 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.541 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:35.541 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207697+0000", 2024-09-21T12:31:35.541 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.541 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.541 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.542 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:35.542 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207698+0000", 2024-09-21T12:31:35.542 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6449999999999999e-06 2024-09-21T12:31:35.542 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.542 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.542 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:35.542 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207744+0000", 2024-09-21T12:31:35.542 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6171999999999998e-05 2024-09-21T12:31:35.542 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.542 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.542 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:35.542 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207755+0000", 2024-09-21T12:31:35.542 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0998999999999999e-05 2024-09-21T12:31:35.543 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.543 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.543 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:35.543 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207761+0000", 2024-09-21T12:31:35.543 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9789999999999998e-06 2024-09-21T12:31:35.543 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.543 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.543 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:35.543 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207822+0000", 2024-09-21T12:31:35.543 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0504000000000002e-05 2024-09-21T12:31:35.543 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.543 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.543 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:35.544 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.207921+0000", 2024-09-21T12:31:35.544 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9524000000000005e-05 2024-09-21T12:31:35.544 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.544 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.544 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,3", 2024-09-21T12:31:35.544 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.208068+0000", 2024-09-21T12:31:35.544 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014648299999999999 2024-09-21T12:31:35.544 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.544 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.544 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:31:35.544 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.209637+0000", 2024-09-21T12:31:35.544 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001569555 2024-09-21T12:31:35.544 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.545 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.545 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.545 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.214355+0000", 2024-09-21T12:31:35.545 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0047178810000000002 2024-09-21T12:31:35.545 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.545 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.545 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.545 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.387868+0000", 2024-09-21T12:31:35.545 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.173512203 2024-09-21T12:31:35.545 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.545 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.545 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:35.545 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.387905+0000", 2024-09-21T12:31:35.546 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7678000000000001e-05 2024-09-21T12:31:35.546 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.546 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.546 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:35.546 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.387931+0000", 2024-09-21T12:31:35.546 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6174e-05 2024-09-21T12:31:35.546 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.546 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:35.546 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.546 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.546 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.546 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:49949 7.d 7:b8e0b394:::benchmark_data_smithi114_78432_object49948:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:31:35.546 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:05.215908+0000", 2024-09-21T12:31:35.547 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 570.26412375699999, 2024-09-21T12:31:35.547 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.17212050400000001, 2024-09-21T12:31:35.547 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:35.547 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:35.547 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:35.547 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:31:35.547 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:31:35.547 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:31:35.547 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 49949 2024-09-21T12:31:35.547 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.547 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:35.547 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.547 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:35.547 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215908+0000", 2024-09-21T12:31:35.548 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.548 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.548 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.548 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:35.548 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215908+0000", 2024-09-21T12:31:35.548 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.548 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.548 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.548 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:35.548 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215910+0000", 2024-09-21T12:31:35.548 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.389e-06 2024-09-21T12:31:35.548 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.548 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.548 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:35.549 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215966+0000", 2024-09-21T12:31:35.549 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6189e-05 2024-09-21T12:31:35.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.549 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:35.549 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215976+0000", 2024-09-21T12:31:35.549 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0115999999999999e-05 2024-09-21T12:31:35.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.549 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:35.549 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.215983+0000", 2024-09-21T12:31:35.549 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9240000000000001e-06 2024-09-21T12:31:35.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.550 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:35.550 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.216032+0000", 2024-09-21T12:31:35.550 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9237000000000003e-05 2024-09-21T12:31:35.550 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.550 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.550 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:35.550 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.216109+0000", 2024-09-21T12:31:35.550 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6538999999999993e-05 2024-09-21T12:31:35.550 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.550 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.550 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1", 2024-09-21T12:31:35.550 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.216208+0000", 2024-09-21T12:31:35.550 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8979000000000002e-05 2024-09-21T12:31:35.551 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.551 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.551 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:31:35.551 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217649+0000", 2024-09-21T12:31:35.551 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014417500000000001 2024-09-21T12:31:35.551 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.551 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.551 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.551 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.219275+0000", 2024-09-21T12:31:35.551 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0016254749999999999 2024-09-21T12:31:35.551 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.551 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.551 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.551 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.387917+0000", 2024-09-21T12:31:35.552 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.16864237100000001 2024-09-21T12:31:35.552 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.552 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.552 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:35.552 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.387990+0000", 2024-09-21T12:31:35.552 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2249999999999994e-05 2024-09-21T12:31:35.552 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.552 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.552 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:35.552 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.388029+0000", 2024-09-21T12:31:35.552 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9285000000000001e-05 2024-09-21T12:31:35.552 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.552 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:35.552 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.553 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.553 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.553 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:49953 7.7 7:e1a3875f:::benchmark_data_smithi114_78432_object49952:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:31:35.553 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:05.217615+0000", 2024-09-21T12:31:35.553 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 570.26241683600006, 2024-09-21T12:31:35.553 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.17616359000000001, 2024-09-21T12:31:35.553 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:35.553 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:35.553 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:35.553 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:31:35.553 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:31:35.553 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:31:35.553 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 49953 2024-09-21T12:31:35.554 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.554 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:35.554 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.554 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:35.554 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217615+0000", 2024-09-21T12:31:35.554 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.554 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.554 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.554 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:35.554 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217615+0000", 2024-09-21T12:31:35.554 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.554 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.554 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.555 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:35.555 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217616+0000", 2024-09-21T12:31:35.555 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.209e-06 2024-09-21T12:31:35.555 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.555 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.555 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:35.555 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217687+0000", 2024-09-21T12:31:35.555 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0570999999999999e-05 2024-09-21T12:31:35.555 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.555 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.555 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:35.555 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217701+0000", 2024-09-21T12:31:35.556 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3828e-05 2024-09-21T12:31:35.556 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.556 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.556 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:35.556 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217709+0000", 2024-09-21T12:31:35.556 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3240000000000003e-06 2024-09-21T12:31:35.556 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.556 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.556 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:35.556 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217763+0000", 2024-09-21T12:31:35.556 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3804999999999997e-05 2024-09-21T12:31:35.556 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.556 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.557 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:35.557 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217854+0000", 2024-09-21T12:31:35.557 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0771000000000002e-05 2024-09-21T12:31:35.557 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.557 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.557 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,3", 2024-09-21T12:31:35.557 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.217935+0000", 2024-09-21T12:31:35.557 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0951000000000005e-05 2024-09-21T12:31:35.557 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.557 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.557 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:31:35.557 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.219237+0000", 2024-09-21T12:31:35.558 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0013019679999999999 2024-09-21T12:31:35.558 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.558 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.558 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.558 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.220275+0000", 2024-09-21T12:31:35.558 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010378609999999999 2024-09-21T12:31:35.558 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.558 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.558 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.558 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.393564+0000", 2024-09-21T12:31:35.558 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.17328943899999999 2024-09-21T12:31:35.558 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.558 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.559 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:35.559 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.393734+0000", 2024-09-21T12:31:35.559 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017039999999999999 2024-09-21T12:31:35.559 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.559 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.559 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:35.559 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.393779+0000", 2024-09-21T12:31:35.559 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4462999999999998e-05 2024-09-21T12:31:35.559 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.559 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:35.559 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.559 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.559 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.560 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:49961 7.7 7:e82f0c80:::benchmark_data_smithi114_78432_object49960:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-21T12:31:35.560 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:22:05.226839+0000", 2024-09-21T12:31:35.560 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 570.25319284, 2024-09-21T12:31:35.560 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.170311145, 2024-09-21T12:31:35.560 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:35.560 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:35.560 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:35.560 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:31:35.560 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:31:35.560 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:31:35.560 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 49961 2024-09-21T12:31:35.560 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.560 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:35.560 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.561 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:35.561 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.226839+0000", 2024-09-21T12:31:35.561 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.561 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.561 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.561 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:35.561 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.226839+0000", 2024-09-21T12:31:35.561 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.561 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.561 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.561 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:35.561 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.226840+0000", 2024-09-21T12:31:35.561 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8400000000000003e-07 2024-09-21T12:31:35.562 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.562 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.562 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:35.562 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.226896+0000", 2024-09-21T12:31:35.562 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5517999999999999e-05 2024-09-21T12:31:35.562 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.562 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.562 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:35.562 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.226906+0000", 2024-09-21T12:31:35.562 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0589e-05 2024-09-21T12:31:35.562 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.562 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.562 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:35.562 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.226914+0000", 2024-09-21T12:31:35.563 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2779999999999996e-06 2024-09-21T12:31:35.563 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.563 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.563 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:35.563 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.226963+0000", 2024-09-21T12:31:35.563 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9021999999999999e-05 2024-09-21T12:31:35.563 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.563 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.563 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:35.563 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.227045+0000", 2024-09-21T12:31:35.563 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2800999999999995e-05 2024-09-21T12:31:35.563 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.563 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.564 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,3", 2024-09-21T12:31:35.564 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.227126+0000", 2024-09-21T12:31:35.564 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0467000000000001e-05 2024-09-21T12:31:35.564 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.564 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.564 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:31:35.564 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.228347+0000", 2024-09-21T12:31:35.564 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001220907 2024-09-21T12:31:35.564 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.564 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.564 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.564 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.393832+0000", 2024-09-21T12:31:35.564 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.165485038 2024-09-21T12:31:35.564 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.565 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.565 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.565 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.397086+0000", 2024-09-21T12:31:35.565 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0032546799999999998 2024-09-21T12:31:35.565 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.565 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.565 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:35.565 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.397121+0000", 2024-09-21T12:31:35.565 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4584e-05 2024-09-21T12:31:35.565 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.565 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.565 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:35.565 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:22:05.397150+0000", 2024-09-21T12:31:35.566 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9377e-05 2024-09-21T12:31:35.566 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.566 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:35.566 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.566 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.566 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.566 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:116822 7.c 7:3b47e54f:::benchmark_data_smithi114_78432_object12676:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T12:31:35.566 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:23:12.373717+0000", 2024-09-21T12:31:35.566 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 503.10631516699999, 2024-09-21T12:31:35.566 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15758128199999999, 2024-09-21T12:31:35.566 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:35.566 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:35.566 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:35.566 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:31:35.566 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:31:35.567 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:31:35.567 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 116822 2024-09-21T12:31:35.567 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.567 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:35.567 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.567 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:35.567 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.373717+0000", 2024-09-21T12:31:35.567 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.567 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.567 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.567 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:35.568 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.373717+0000", 2024-09-21T12:31:35.568 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.568 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.568 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.568 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:35.568 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.373718+0000", 2024-09-21T12:31:35.569 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2190000000000001e-06 2024-09-21T12:31:35.569 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.569 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.569 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:35.569 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.373723+0000", 2024-09-21T12:31:35.569 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7400000000000004e-06 2024-09-21T12:31:35.569 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.569 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.569 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:35.569 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.373732+0000", 2024-09-21T12:31:35.569 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8319999999999995e-06 2024-09-21T12:31:35.569 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.569 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.569 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:35.570 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.373737+0000", 2024-09-21T12:31:35.570 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2789999999999998e-06 2024-09-21T12:31:35.570 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.570 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.570 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:35.570 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.373777+0000", 2024-09-21T12:31:35.570 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0123999999999997e-05 2024-09-21T12:31:35.570 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.570 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.570 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:35.570 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.373854+0000", 2024-09-21T12:31:35.570 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6743999999999996e-05 2024-09-21T12:31:35.570 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.571 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.571 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:31:35.571 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.373916+0000", 2024-09-21T12:31:35.571 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2451000000000002e-05 2024-09-21T12:31:35.571 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.571 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.571 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:31:35.571 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.375743+0000", 2024-09-21T12:31:35.571 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001826338 2024-09-21T12:31:35.571 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.571 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.571 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.571 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.375885+0000", 2024-09-21T12:31:35.572 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014278699999999999 2024-09-21T12:31:35.572 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.572 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.572 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.572 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.531196+0000", 2024-09-21T12:31:35.572 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.155311 2024-09-21T12:31:35.572 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.572 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.572 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:35.572 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.531247+0000", 2024-09-21T12:31:35.572 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0476000000000003e-05 2024-09-21T12:31:35.572 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.572 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.572 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:35.573 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.531298+0000", 2024-09-21T12:31:35.573 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1292000000000003e-05 2024-09-21T12:31:35.573 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.573 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:35.573 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.573 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.573 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.573 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:116828 7.c 7:31e7a7e6:::benchmark_data_smithi114_78432_object12682:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T12:31:35.573 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:23:12.377844+0000", 2024-09-21T12:31:35.573 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 503.10218826800002, 2024-09-21T12:31:35.573 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15356805100000001, 2024-09-21T12:31:35.573 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:35.573 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:35.573 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:35.573 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:31:35.573 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:31:35.574 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:31:35.574 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 116828 2024-09-21T12:31:35.574 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.574 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:35.574 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.574 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:35.574 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.377844+0000", 2024-09-21T12:31:35.574 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.574 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.574 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.574 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:35.574 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.377844+0000", 2024-09-21T12:31:35.574 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.574 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.575 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.575 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:35.575 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.377845+0000", 2024-09-21T12:31:35.575 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3600000000000002e-07 2024-09-21T12:31:35.575 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.575 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.575 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:35.575 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.377849+0000", 2024-09-21T12:31:35.575 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.622e-06 2024-09-21T12:31:35.575 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.575 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.575 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:35.575 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.377862+0000", 2024-09-21T12:31:35.576 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2989999999999999e-05 2024-09-21T12:31:35.576 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.576 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.576 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:35.576 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.377870+0000", 2024-09-21T12:31:35.576 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5780000000000001e-06 2024-09-21T12:31:35.576 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.576 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.576 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:35.576 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.377917+0000", 2024-09-21T12:31:35.576 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7178999999999997e-05 2024-09-21T12:31:35.576 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.576 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.576 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:35.576 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.377988+0000", 2024-09-21T12:31:35.576 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0890999999999996e-05 2024-09-21T12:31:35.577 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.577 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.577 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:31:35.577 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.378052+0000", 2024-09-21T12:31:35.577 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4194000000000002e-05 2024-09-21T12:31:35.577 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.577 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.577 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:31:35.577 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.379417+0000", 2024-09-21T12:31:35.577 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001365184 2024-09-21T12:31:35.577 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.577 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.577 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.577 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.379801+0000", 2024-09-21T12:31:35.577 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000383483 2024-09-21T12:31:35.578 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.578 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.578 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.578 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.531334+0000", 2024-09-21T12:31:35.578 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15153333499999999 2024-09-21T12:31:35.578 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.578 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.578 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:35.578 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.531378+0000", 2024-09-21T12:31:35.578 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3557999999999999e-05 2024-09-21T12:31:35.578 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.578 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.578 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:35.578 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.531412+0000", 2024-09-21T12:31:35.579 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4100999999999998e-05 2024-09-21T12:31:35.579 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.579 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:35.579 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.579 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.579 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.579 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:150329 7.7 7:ebd08636:::benchmark_data_smithi114_78432_object46183:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T12:31:35.579 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:23:42.610086+0000", 2024-09-21T12:31:35.579 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 472.86994601499998, 2024-09-21T12:31:35.579 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13913827400000001, 2024-09-21T12:31:35.579 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:35.579 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:35.579 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:35.579 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:31:35.579 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:31:35.580 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:31:35.580 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 150329 2024-09-21T12:31:35.580 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.580 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:35.580 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.580 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:35.580 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.610086+0000", 2024-09-21T12:31:35.580 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.580 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.580 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.580 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:35.580 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.610086+0000", 2024-09-21T12:31:35.580 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.581 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.581 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.581 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:35.581 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.610087+0000", 2024-09-21T12:31:35.581 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9699999999999994e-07 2024-09-21T12:31:35.581 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.581 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.581 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:35.581 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.610091+0000", 2024-09-21T12:31:35.581 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8139999999999998e-06 2024-09-21T12:31:35.581 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.581 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.581 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:35.581 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.610100+0000", 2024-09-21T12:31:35.582 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6239999999999997e-06 2024-09-21T12:31:35.582 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.582 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.582 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:35.582 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.610106+0000", 2024-09-21T12:31:35.582 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6350000000000003e-06 2024-09-21T12:31:35.582 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.582 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.582 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:35.582 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.610146+0000", 2024-09-21T12:31:35.582 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0215e-05 2024-09-21T12:31:35.582 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.582 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.583 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:35.583 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.610231+0000", 2024-09-21T12:31:35.583 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4846999999999996e-05 2024-09-21T12:31:35.583 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.583 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.583 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,3", 2024-09-21T12:31:35.583 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.610303+0000", 2024-09-21T12:31:35.583 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1823999999999995e-05 2024-09-21T12:31:35.583 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.583 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.583 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:31:35.583 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.612121+0000", 2024-09-21T12:31:35.583 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00181836 2024-09-21T12:31:35.584 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.584 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.584 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.584 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.615984+0000", 2024-09-21T12:31:35.584 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0038623630000000002 2024-09-21T12:31:35.584 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.584 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.584 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.584 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.749109+0000", 2024-09-21T12:31:35.584 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.133125308 2024-09-21T12:31:35.584 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.584 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.584 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:35.585 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.749172+0000", 2024-09-21T12:31:35.585 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2738999999999997e-05 2024-09-21T12:31:35.585 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.585 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.585 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:35.585 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:42.749224+0000", 2024-09-21T12:31:35.585 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2547999999999999e-05 2024-09-21T12:31:35.585 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.585 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:35.585 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.585 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.585 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.585 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:32454 8.5 8:afb3ecb0:::benchmark_data_smithi114_104936_object32453:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T12:31:35.585 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:07.550337+0000", 2024-09-21T12:31:35.586 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 387.92969512299999, 2024-09-21T12:31:35.586 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15084788399999999, 2024-09-21T12:31:35.586 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:35.586 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:35.586 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:35.586 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:31:35.586 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:31:35.586 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:31:35.586 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 32454 2024-09-21T12:31:35.586 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.586 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:35.586 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.586 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:35.586 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.550337+0000", 2024-09-21T12:31:35.587 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.587 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.587 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.587 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:35.587 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.550337+0000", 2024-09-21T12:31:35.587 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.587 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.587 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.587 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:35.587 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.550338+0000", 2024-09-21T12:31:35.587 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.525e-06 2024-09-21T12:31:35.587 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.587 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.588 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:35.588 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.550385+0000", 2024-09-21T12:31:35.588 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6434999999999998e-05 2024-09-21T12:31:35.588 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.588 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.588 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:35.588 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.550398+0000", 2024-09-21T12:31:35.588 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3502999999999999e-05 2024-09-21T12:31:35.588 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.588 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.588 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:35.588 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.550406+0000", 2024-09-21T12:31:35.588 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.187e-06 2024-09-21T12:31:35.589 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.589 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.589 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:35.589 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.550466+0000", 2024-09-21T12:31:35.589 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0708000000000002e-05 2024-09-21T12:31:35.589 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.589 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.589 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:35.589 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.550611+0000", 2024-09-21T12:31:35.589 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000144864 2024-09-21T12:31:35.589 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.589 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.589 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,7", 2024-09-21T12:31:35.590 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.550704+0000", 2024-09-21T12:31:35.590 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2888999999999998e-05 2024-09-21T12:31:35.590 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.590 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.590 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:31:35.590 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.552093+0000", 2024-09-21T12:31:35.590 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0013884590000000001 2024-09-21T12:31:35.590 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.590 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.590 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.590 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.553102+0000", 2024-09-21T12:31:35.590 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010093299999999999 2024-09-21T12:31:35.590 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.590 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.591 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.591 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.701083+0000", 2024-09-21T12:31:35.591 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14798086099999999 2024-09-21T12:31:35.591 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.591 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.591 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:35.591 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.701142+0000", 2024-09-21T12:31:35.591 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8788e-05 2024-09-21T12:31:35.591 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.591 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.591 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:35.591 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.701185+0000", 2024-09-21T12:31:35.591 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3334999999999998e-05 2024-09-21T12:31:35.591 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.592 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:35.592 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.592 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.592 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.592 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:38141 8.9 8:961cc3e2:::benchmark_data_smithi114_104936_object38140:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T12:31:35.592 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:11.301060+0000", 2024-09-21T12:31:35.592 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 384.17897228999999, 2024-09-21T12:31:35.592 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14064707500000001, 2024-09-21T12:31:35.592 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:35.592 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:35.592 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:35.592 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:31:35.592 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:31:35.592 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:31:35.593 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 38141 2024-09-21T12:31:35.593 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.593 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:35.593 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.593 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:35.593 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.301060+0000", 2024-09-21T12:31:35.593 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.593 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.593 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.593 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:35.593 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.301060+0000", 2024-09-21T12:31:35.593 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.593 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.593 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.593 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:35.594 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.301061+0000", 2024-09-21T12:31:35.594 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3209999999999999e-06 2024-09-21T12:31:35.594 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.594 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.594 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:35.594 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.301107+0000", 2024-09-21T12:31:35.594 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5577000000000001e-05 2024-09-21T12:31:35.594 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.594 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.594 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:35.594 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.301121+0000", 2024-09-21T12:31:35.594 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4518999999999999e-05 2024-09-21T12:31:35.594 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.595 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.595 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:35.595 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.301129+0000", 2024-09-21T12:31:35.595 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6739999999999997e-06 2024-09-21T12:31:35.595 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.595 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.595 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:35.595 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.301183+0000", 2024-09-21T12:31:35.595 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4363000000000001e-05 2024-09-21T12:31:35.595 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.595 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.595 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:35.595 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.301317+0000", 2024-09-21T12:31:35.595 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013395099999999999 2024-09-21T12:31:35.596 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.596 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.596 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-21T12:31:35.596 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.301459+0000", 2024-09-21T12:31:35.596 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014159599999999999 2024-09-21T12:31:35.596 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.596 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.596 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.596 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.304358+0000", 2024-09-21T12:31:35.596 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0028990130000000002 2024-09-21T12:31:35.596 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.596 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.596 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:31:35.596 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.304497+0000", 2024-09-21T12:31:35.597 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013938500000000001 2024-09-21T12:31:35.597 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.597 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.597 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.597 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.441590+0000", 2024-09-21T12:31:35.597 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13709248099999999 2024-09-21T12:31:35.597 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.597 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.597 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:35.597 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.441667+0000", 2024-09-21T12:31:35.597 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7088999999999993e-05 2024-09-21T12:31:35.597 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.597 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.597 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:35.598 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.441707+0000", 2024-09-21T12:31:35.598 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0105999999999999e-05 2024-09-21T12:31:35.598 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.598 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:35.598 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.598 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.598 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.598 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:38144 8.5 8:a00fd695:::benchmark_data_smithi114_104936_object38143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T12:31:35.598 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:11.302034+0000", 2024-09-21T12:31:35.598 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 384.17799811700002, 2024-09-21T12:31:35.598 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13963275, 2024-09-21T12:31:35.598 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:35.598 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:35.598 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:35.599 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:31:35.599 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:31:35.599 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:31:35.599 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 38144 2024-09-21T12:31:35.599 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.599 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:35.599 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.599 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:35.599 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302034+0000", 2024-09-21T12:31:35.599 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.599 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.599 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.599 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:35.600 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302034+0000", 2024-09-21T12:31:35.600 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.600 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.600 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.600 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:35.600 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302035+0000", 2024-09-21T12:31:35.600 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1349999999999999e-06 2024-09-21T12:31:35.600 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.600 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.600 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:35.600 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302074+0000", 2024-09-21T12:31:35.600 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9057999999999998e-05 2024-09-21T12:31:35.600 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.600 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.601 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:35.601 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302090+0000", 2024-09-21T12:31:35.601 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6007000000000001e-05 2024-09-21T12:31:35.601 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.601 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.601 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:35.601 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302098+0000", 2024-09-21T12:31:35.601 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9070000000000007e-06 2024-09-21T12:31:35.601 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.601 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.601 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:35.601 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302165+0000", 2024-09-21T12:31:35.601 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7008999999999995e-05 2024-09-21T12:31:35.601 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.602 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.602 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:35.602 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302300+0000", 2024-09-21T12:31:35.602 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013537900000000001 2024-09-21T12:31:35.602 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.602 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.602 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,7", 2024-09-21T12:31:35.602 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302451+0000", 2024-09-21T12:31:35.602 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015088600000000001 2024-09-21T12:31:35.602 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.602 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.602 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:31:35.602 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.304486+0000", 2024-09-21T12:31:35.602 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020348950000000001 2024-09-21T12:31:35.603 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.603 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.603 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.603 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.304554+0000", 2024-09-21T12:31:35.603 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8201999999999995e-05 2024-09-21T12:31:35.603 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.603 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.603 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.603 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.441565+0000", 2024-09-21T12:31:35.603 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.137010256 2024-09-21T12:31:35.603 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.603 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.603 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:35.604 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.441624+0000", 2024-09-21T12:31:35.604 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9367999999999998e-05 2024-09-21T12:31:35.604 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.604 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.604 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:35.604 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.441667+0000", 2024-09-21T12:31:35.604 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2648000000000002e-05 2024-09-21T12:31:35.604 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.604 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:35.604 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.604 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.604 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.604 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:223832 8.9 8:95f724a3:::benchmark_data_smithi114_104936_object93416:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-09-21T12:31:35.605 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:27:35.314953+0000", 2024-09-21T12:31:35.605 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 240.16507956000001, 2024-09-21T12:31:35.605 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13631865800000001, 2024-09-21T12:31:35.605 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:35.605 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:35.605 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:31:35.605 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:31:35.605 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:31:35.605 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:31:35.605 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 223832 2024-09-21T12:31:35.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.605 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:35.605 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.605 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:35.605 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.314953+0000", 2024-09-21T12:31:35.605 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.606 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.606 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.606 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:35.606 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.314953+0000", 2024-09-21T12:31:35.606 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:35.606 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.606 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.606 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:35.606 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.314954+0000", 2024-09-21T12:31:35.606 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4249999999999999e-06 2024-09-21T12:31:35.606 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.606 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.606 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:35.606 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.314959+0000", 2024-09-21T12:31:35.607 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6149999999999999e-06 2024-09-21T12:31:35.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.607 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.607 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:35.607 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.314973+0000", 2024-09-21T12:31:35.607 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4134999999999999e-05 2024-09-21T12:31:35.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.607 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.607 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:35.607 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.314979+0000", 2024-09-21T12:31:35.607 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3489999999999997e-06 2024-09-21T12:31:35.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.607 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.607 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:35.608 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.315029+0000", 2024-09-21T12:31:35.608 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0272999999999998e-05 2024-09-21T12:31:35.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.608 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:35.608 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.315137+0000", 2024-09-21T12:31:35.608 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000107878 2024-09-21T12:31:35.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.608 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-21T12:31:35.608 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.315251+0000", 2024-09-21T12:31:35.608 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000113453 2024-09-21T12:31:35.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.608 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:31:35.608 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.317452+0000", 2024-09-21T12:31:35.609 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0022017080000000001 2024-09-21T12:31:35.609 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.609 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.609 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.609 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.317539+0000", 2024-09-21T12:31:35.609 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6935999999999996e-05 2024-09-21T12:31:35.609 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.609 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.609 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:31:35.609 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.451201+0000", 2024-09-21T12:31:35.609 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13366187199999999 2024-09-21T12:31:35.609 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.609 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.609 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:31:35.609 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.451237+0000", 2024-09-21T12:31:35.610 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6273999999999999e-05 2024-09-21T12:31:35.610 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:35.610 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:35.610 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:31:35.610 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.451271+0000", 2024-09-21T12:31:35.610 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3739999999999999e-05 2024-09-21T12:31:35.610 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.610 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:35.610 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.610 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:35.610 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:35.610 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:31:35.610 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:35.648 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:35.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:35.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:35.750 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:35.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:35 smithi114 ceph-mon[30324]: pgmap v558: 188 pgs: 4 active+remapped+backfill_toofull, 184 active+clean; 2.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s; 5556/102126 objects misplaced (5.440%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T12:31:35.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:35 smithi114 ceph-mon[40867]: pgmap v558: 188 pgs: 4 active+remapped+backfill_toofull, 184 active+clean; 2.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s; 5556/102126 objects misplaced (5.440%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T12:31:35.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:35.799 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:35.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:35.900 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:35.929 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:35.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:35.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:36.031 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:36.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:36.072 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:36.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:36.172 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:36.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:36.207 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:36.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:36.308 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:36.345 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:36.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:36.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:36.446 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:36.489 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:36.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:36.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:36.509 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T12:31:36.591 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:36.633 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:36.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:36.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:36.734 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:36.772 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:36.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:36.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:36.874 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:36.939 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:31:36.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:36.941 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:36.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:37.042 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:37.074 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:37.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:37.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:37.175 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:37.208 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:37.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:37.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:37.309 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:37.341 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:37.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:37.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:37.387 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:31:37 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:31:37.388 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:37 smithi183 ceph-mon[33277]: pgmap v559: 188 pgs: 4 active+remapped+backfill_toofull, 184 active+clean; 2.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.26k op/s; 5556/102126 objects misplaced (5.440%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T12:31:37.388 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:37 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/44135713' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:31:37.388 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:37 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:31:37.443 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:37.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:37.487 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:37.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:37 smithi114 ceph-mon[40867]: pgmap v559: 188 pgs: 4 active+remapped+backfill_toofull, 184 active+clean; 2.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.26k op/s; 5556/102126 objects misplaced (5.440%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T12:31:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:37 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/44135713' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:31:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:37 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:31:37.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:37 smithi114 ceph-mon[30324]: pgmap v559: 188 pgs: 4 active+remapped+backfill_toofull, 184 active+clean; 2.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.26k op/s; 5556/102126 objects misplaced (5.440%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T12:31:37.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:37 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/44135713' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:31:37.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:37 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:31:37.588 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:37.602 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:31:37.602 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:31:37.602 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.602 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.25436.0:221231 9.a e306/266 9:52e135e8:::benchmark_data_smithi114_133214_object97197:head v 306'13824, mlcod=306'13822)", 2024-09-21T12:31:37.602 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:37.594188+0000", 2024-09-21T12:31:37.602 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0043637509999999999, 2024-09-21T12:31:37.602 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0043885540000000002, 2024-09-21T12:31:37.602 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:37.603 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:37.603 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:31:37.603 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:37.603 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.603 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:37.603 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.594188+0000", 2024-09-21T12:31:37.603 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:37.603 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.603 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.603 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:37.603 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.594188+0000", 2024-09-21T12:31:37.603 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:37.603 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.603 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.604 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:37.604 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.594189+0000", 2024-09-21T12:31:37.604 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2099999999999995e-07 2024-09-21T12:31:37.604 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.604 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.604 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:37.604 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.594195+0000", 2024-09-21T12:31:37.604 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1940000000000003e-06 2024-09-21T12:31:37.604 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.604 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.604 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:37.604 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.594209+0000", 2024-09-21T12:31:37.604 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3716e-05 2024-09-21T12:31:37.604 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.605 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.605 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:37.605 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.594216+0000", 2024-09-21T12:31:37.605 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0450000000000003e-06 2024-09-21T12:31:37.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.605 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.605 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:37.605 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.594295+0000", 2024-09-21T12:31:37.605 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9240999999999996e-05 2024-09-21T12:31:37.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.605 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.605 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:37.605 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.594318+0000", 2024-09-21T12:31:37.605 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.323e-05 2024-09-21T12:31:37.606 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:37.606 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:37.606 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:37.606 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.606 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.606 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.25436.0:221236 9.c e306/236 9:37ae10c0:::benchmark_data_smithi114_133214_object97202:head v 306'13679, mlcod=306'13677)", 2024-09-21T12:31:37.606 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:37.595325+0000", 2024-09-21T12:31:37.606 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0032260790000000002, 2024-09-21T12:31:37.606 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003308685, 2024-09-21T12:31:37.606 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:37.606 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:37.606 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:31:37.606 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:37.606 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.607 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:37.607 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.595325+0000", 2024-09-21T12:31:37.607 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:37.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.607 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.607 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:37.607 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.595325+0000", 2024-09-21T12:31:37.607 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:37.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.607 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.607 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:37.607 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.595326+0000", 2024-09-21T12:31:37.607 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5600000000000001e-07 2024-09-21T12:31:37.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.607 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.608 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:37.608 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.595330+0000", 2024-09-21T12:31:37.608 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6140000000000002e-06 2024-09-21T12:31:37.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.608 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:37.608 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.595336+0000", 2024-09-21T12:31:37.608 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9649999999999996e-06 2024-09-21T12:31:37.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.608 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:37.608 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.595341+0000", 2024-09-21T12:31:37.608 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1849999999999997e-06 2024-09-21T12:31:37.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.609 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:37.609 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.595407+0000", 2024-09-21T12:31:37.609 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6820000000000001e-05 2024-09-21T12:31:37.609 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.609 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.609 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:37.609 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.595425+0000", 2024-09-21T12:31:37.609 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7096e-05 2024-09-21T12:31:37.609 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:37.609 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:37.609 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:37.609 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.609 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.609 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:221240 9.d 9:b8195781:::benchmark_data_smithi114_133214_object97206:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e306)", 2024-09-21T12:31:37.609 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:37.596469+0000", 2024-09-21T12:31:37.610 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0020823249999999999, 2024-09-21T12:31:37.610 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0022178499999999999, 2024-09-21T12:31:37.610 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:37.610 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:37.610 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:31:37.610 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:31:37.610 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:31:37.610 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:31:37.610 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 221240 2024-09-21T12:31:37.610 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.610 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:37.610 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.610 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:37.610 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.596469+0000", 2024-09-21T12:31:37.611 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:37.611 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.611 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.611 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:37.611 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.596469+0000", 2024-09-21T12:31:37.611 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:37.611 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.611 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.611 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:37.611 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.596471+0000", 2024-09-21T12:31:37.611 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5370000000000001e-06 2024-09-21T12:31:37.611 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.611 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.611 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:37.612 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.596476+0000", 2024-09-21T12:31:37.612 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5770000000000001e-06 2024-09-21T12:31:37.612 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.612 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.612 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:37.612 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.596490+0000", 2024-09-21T12:31:37.612 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.414e-05 2024-09-21T12:31:37.612 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.612 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.612 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:37.612 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.596498+0000", 2024-09-21T12:31:37.612 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5299999999999999e-06 2024-09-21T12:31:37.612 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.612 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.613 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:37.613 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.596563+0000", 2024-09-21T12:31:37.613 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4688000000000001e-05 2024-09-21T12:31:37.613 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.613 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.613 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:37.613 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.596695+0000", 2024-09-21T12:31:37.613 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013193600000000001 2024-09-21T12:31:37.613 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.613 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.613 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,6", 2024-09-21T12:31:37.613 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.596829+0000", 2024-09-21T12:31:37.613 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013419199999999999 2024-09-21T12:31:37.613 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:37.613 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:37.614 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:37.614 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.614 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.614 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.25436.0:221234 9.a e306/266 9:50ab23ae:::benchmark_data_smithi114_133214_object97200:head v 306'13825, mlcod=306'13822)", 2024-09-21T12:31:37.614 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:37.596883+0000", 2024-09-21T12:31:37.614 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.001668707, 2024-09-21T12:31:37.614 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0018690320000000001, 2024-09-21T12:31:37.614 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:31:37.614 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:31:37.614 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:31:37.614 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:31:37.614 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.614 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:31:37.615 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.596883+0000", 2024-09-21T12:31:37.615 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:37.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.615 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.615 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:31:37.615 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.596883+0000", 2024-09-21T12:31:37.615 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:31:37.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.615 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.615 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:31:37.615 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.596884+0000", 2024-09-21T12:31:37.615 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.75e-07 2024-09-21T12:31:37.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.615 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.615 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:31:37.615 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.596889+0000", 2024-09-21T12:31:37.616 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8119999999999996e-06 2024-09-21T12:31:37.616 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.616 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.616 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:31:37.616 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.596898+0000", 2024-09-21T12:31:37.616 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4780000000000008e-06 2024-09-21T12:31:37.616 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.616 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.616 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:31:37.616 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.596904+0000", 2024-09-21T12:31:37.616 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4849999999999997e-06 2024-09-21T12:31:37.616 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.616 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.616 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:31:37.617 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.598642+0000", 2024-09-21T12:31:37.617 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001738103 2024-09-21T12:31:37.617 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:31:37.617 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:31:37.617 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:31:37.617 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:37.598666+0000", 2024-09-21T12:31:37.617 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3774e-05 2024-09-21T12:31:37.617 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:37.617 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:31:37.617 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:37.617 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:31:37.617 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-21T12:31:37.617 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 4 2024-09-21T12:31:37.617 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:31:37.626 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:37.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:37.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:37.728 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:37.767 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:37.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:37.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:37.869 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:37.909 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:37.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:37.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:38.010 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:38.041 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-21T12:31:38.047 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T12:31:38.053 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:38.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:38.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:38.155 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:38.177 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:38.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:38.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:38.279 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:38.301 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:38.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:38.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:38.403 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:38.425 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:38.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:38.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:38.440 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:31:38.442 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T12:31:38.527 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:38.548 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:38.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:38.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:38.650 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:38.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:38.673 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:38.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:38.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:38 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:38.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:38 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3454619048' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:31:38.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:38 smithi183 ceph-mon[33277]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:31:38.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:38 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T12:31:38.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:38 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3454619048' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:31:38.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:38 smithi183 ceph-mon[33277]: osdmap e307: 8 total, 8 up, 7 in 2024-09-21T12:31:38.744 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:31:38.774 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:38.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:38 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:38.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:38 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3454619048' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:31:38.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:38 smithi114 ceph-mon[40867]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:31:38.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:38 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T12:31:38.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:38 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3454619048' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:31:38.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:38 smithi114 ceph-mon[40867]: osdmap e307: 8 total, 8 up, 7 in 2024-09-21T12:31:38.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:38 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:38.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:38 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3454619048' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:31:38.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:38 smithi114 ceph-mon[30324]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:31:38.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:38 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T12:31:38.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:38 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3454619048' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:31:38.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:38 smithi114 ceph-mon[30324]: osdmap e307: 8 total, 8 up, 7 in 2024-09-21T12:31:38.797 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:38.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:38.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:38.899 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:38.920 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:38.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:38.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:39.021 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:39.046 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:39.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:39.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:39.148 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:39.175 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:39.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:39.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:39.233 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:31:39.233 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-21T12:31:39.233 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-21T12:31:39.233 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-21T12:31:39.233 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:31:39.278 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:39.300 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:39.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:39.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:39.402 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:39.423 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:39.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:39.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:39.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:39 smithi183 ceph-mon[33277]: pgmap v560: 188 pgs: 4 active+remapped+backfill_toofull, 184 active+clean; 1.7 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 948 B/s rd, 0 B/s wr, 1.39k op/s; 4631/85854 objects misplaced (5.394%) 2024-09-21T12:31:39.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:39 smithi183 ceph-mon[33277]: 1.16 scrub starts 2024-09-21T12:31:39.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:39 smithi183 ceph-mon[33277]: 1.16 scrub ok 2024-09-21T12:31:39.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:39 smithi183 ceph-mon[33277]: 3.14 scrub starts 2024-09-21T12:31:39.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:39 smithi183 ceph-mon[33277]: 3.14 scrub ok 2024-09-21T12:31:39.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:39 smithi183 ceph-mon[33277]: 1.1e scrub starts 2024-09-21T12:31:39.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:39 smithi183 ceph-mon[33277]: 1.1e scrub ok 2024-09-21T12:31:39.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:39 smithi183 ceph-mon[33277]: 3.1d scrub starts 2024-09-21T12:31:39.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:39 smithi183 ceph-mon[33277]: 3.1d scrub ok 2024-09-21T12:31:39.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:39 smithi183 ceph-mon[33277]: 9.f scrub starts 2024-09-21T12:31:39.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:39 smithi183 ceph-mon[33277]: 1.14 scrub starts 2024-09-21T12:31:39.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:39 smithi183 ceph-mon[33277]: 1.14 scrub ok 2024-09-21T12:31:39.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:39 smithi183 ceph-mon[33277]: 9.f scrub ok 2024-09-21T12:31:39.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:39 smithi183 ceph-mon[33277]: osdmap e308: 8 total, 8 up, 7 in 2024-09-21T12:31:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:39 smithi114 ceph-mon[30324]: pgmap v560: 188 pgs: 4 active+remapped+backfill_toofull, 184 active+clean; 1.7 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 948 B/s rd, 0 B/s wr, 1.39k op/s; 4631/85854 objects misplaced (5.394%) 2024-09-21T12:31:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:39 smithi114 ceph-mon[30324]: 1.16 scrub starts 2024-09-21T12:31:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:39 smithi114 ceph-mon[30324]: 1.16 scrub ok 2024-09-21T12:31:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:39 smithi114 ceph-mon[30324]: 3.14 scrub starts 2024-09-21T12:31:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:39 smithi114 ceph-mon[30324]: 3.14 scrub ok 2024-09-21T12:31:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:39 smithi114 ceph-mon[30324]: 1.1e scrub starts 2024-09-21T12:31:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:39 smithi114 ceph-mon[30324]: 1.1e scrub ok 2024-09-21T12:31:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:39 smithi114 ceph-mon[30324]: 3.1d scrub starts 2024-09-21T12:31:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:39 smithi114 ceph-mon[30324]: 3.1d scrub ok 2024-09-21T12:31:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:39 smithi114 ceph-mon[30324]: 9.f scrub starts 2024-09-21T12:31:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:39 smithi114 ceph-mon[30324]: 1.14 scrub starts 2024-09-21T12:31:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:39 smithi114 ceph-mon[30324]: 1.14 scrub ok 2024-09-21T12:31:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:39 smithi114 ceph-mon[30324]: 9.f scrub ok 2024-09-21T12:31:39.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:39 smithi114 ceph-mon[30324]: osdmap e308: 8 total, 8 up, 7 in 2024-09-21T12:31:39.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:39 smithi114 ceph-mon[40867]: pgmap v560: 188 pgs: 4 active+remapped+backfill_toofull, 184 active+clean; 1.7 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 948 B/s rd, 0 B/s wr, 1.39k op/s; 4631/85854 objects misplaced (5.394%) 2024-09-21T12:31:39.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:39 smithi114 ceph-mon[40867]: 1.16 scrub starts 2024-09-21T12:31:39.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:39 smithi114 ceph-mon[40867]: 1.16 scrub ok 2024-09-21T12:31:39.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:39 smithi114 ceph-mon[40867]: 3.14 scrub starts 2024-09-21T12:31:39.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:39 smithi114 ceph-mon[40867]: 3.14 scrub ok 2024-09-21T12:31:39.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:39 smithi114 ceph-mon[40867]: 1.1e scrub starts 2024-09-21T12:31:39.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:39 smithi114 ceph-mon[40867]: 1.1e scrub ok 2024-09-21T12:31:39.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:39 smithi114 ceph-mon[40867]: 3.1d scrub starts 2024-09-21T12:31:39.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:39 smithi114 ceph-mon[40867]: 3.1d scrub ok 2024-09-21T12:31:39.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:39 smithi114 ceph-mon[40867]: 9.f scrub starts 2024-09-21T12:31:39.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:39 smithi114 ceph-mon[40867]: 1.14 scrub starts 2024-09-21T12:31:39.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:39 smithi114 ceph-mon[40867]: 1.14 scrub ok 2024-09-21T12:31:39.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:39 smithi114 ceph-mon[40867]: 9.f scrub ok 2024-09-21T12:31:39.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:39 smithi114 ceph-mon[40867]: osdmap e308: 8 total, 8 up, 7 in 2024-09-21T12:31:39.527 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:39.549 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:39.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:39.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:39.651 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:39.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:39.673 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:39.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:39.753 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T12:31:39.774 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:39.796 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:39.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:39.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:39.815 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:31:39.898 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:39.920 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:39.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:39.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:39.947 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T12:31:40.022 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:40.047 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:40.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:40.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:40.149 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:40.186 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:40.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:40.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:40.288 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:40.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:40.325 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:40.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:40.327 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:31:40.426 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:40.462 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:40.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:40.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:40.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:40 smithi114 ceph-mon[40867]: 9.15 starting backfill to osd.1 from (262'6275,280'9963] MIN to 306'11898 2024-09-21T12:31:40.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:40 smithi114 ceph-mon[40867]: 9.15 continuing backfill to osd.3 from (262'6275,306'11898] MIN to 306'11898 2024-09-21T12:31:40.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:40 smithi114 ceph-mon[40867]: 5.b scrub starts 2024-09-21T12:31:40.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:40 smithi114 ceph-mon[40867]: 5.b scrub ok 2024-09-21T12:31:40.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:40 smithi114 ceph-mon[40867]: osdmap e309: 8 total, 8 up, 7 in 2024-09-21T12:31:40.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:40 smithi114 ceph-mon[30324]: 9.15 starting backfill to osd.1 from (262'6275,280'9963] MIN to 306'11898 2024-09-21T12:31:40.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:40 smithi114 ceph-mon[30324]: 9.15 continuing backfill to osd.3 from (262'6275,306'11898] MIN to 306'11898 2024-09-21T12:31:40.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:40 smithi114 ceph-mon[30324]: 5.b scrub starts 2024-09-21T12:31:40.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:40 smithi114 ceph-mon[30324]: 5.b scrub ok 2024-09-21T12:31:40.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:40 smithi114 ceph-mon[30324]: osdmap e309: 8 total, 8 up, 7 in 2024-09-21T12:31:40.564 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:40.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:40.608 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:40.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:40.680 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:31:40.680 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:31:40.707 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:31:40.709 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:40.724 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:31:40.724 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:31:40.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:40 smithi183 ceph-mon[33277]: 9.15 starting backfill to osd.1 from (262'6275,280'9963] MIN to 306'11898 2024-09-21T12:31:40.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:40 smithi183 ceph-mon[33277]: 9.15 continuing backfill to osd.3 from (262'6275,306'11898] MIN to 306'11898 2024-09-21T12:31:40.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:40 smithi183 ceph-mon[33277]: 5.b scrub starts 2024-09-21T12:31:40.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:40 smithi183 ceph-mon[33277]: 5.b scrub ok 2024-09-21T12:31:40.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:40 smithi183 ceph-mon[33277]: osdmap e309: 8 total, 8 up, 7 in 2024-09-21T12:31:40.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:40.755 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:40.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:40.761 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:31:40.761 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:31:40.796 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:31:40.797 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:31:40.828 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:31:40.828 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:31:40.835 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T12:31:40.856 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:40.861 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:31:40.861 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:31:40.896 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:31:40.896 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:31:40.923 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:40.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:40.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:40.929 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:31:40.929 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:31:41.025 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:41.052 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:41.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:41.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:41.154 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:41.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:41.190 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:41.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:41.246 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:31:41.246 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:31:41.290 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:41.321 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:41.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:41.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:41.423 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:41.453 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:41.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:41.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:41.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:41 smithi183 ceph-mon[33277]: pgmap v563: 188 pgs: 4 active+remapped+backfill_toofull, 184 active+clean; 1.7 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.29k op/s; 4433/82122 objects misplaced (5.398%) 2024-09-21T12:31:41.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:41 smithi183 ceph-mon[33277]: 9.0 starting backfill to osd.1 from (0'0,0'0] MAX to 306'11860 2024-09-21T12:31:41.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:41 smithi183 ceph-mon[33277]: 9.d starting backfill to osd.1 from (0'0,0'0] MAX to 306'13798 2024-09-21T12:31:41.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:41 smithi183 ceph-mon[33277]: 4.8 scrub starts 2024-09-21T12:31:41.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:41 smithi183 ceph-mon[33277]: 4.8 scrub ok 2024-09-21T12:31:41.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:41 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2843018609' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:31:41.554 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:41.580 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:41.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:41.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:41.682 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:41.712 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:41.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:41.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:41.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:41 smithi114 ceph-mon[40867]: pgmap v563: 188 pgs: 4 active+remapped+backfill_toofull, 184 active+clean; 1.7 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.29k op/s; 4433/82122 objects misplaced (5.398%) 2024-09-21T12:31:41.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:41 smithi114 ceph-mon[40867]: 9.0 starting backfill to osd.1 from (0'0,0'0] MAX to 306'11860 2024-09-21T12:31:41.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:41 smithi114 ceph-mon[40867]: 9.d starting backfill to osd.1 from (0'0,0'0] MAX to 306'13798 2024-09-21T12:31:41.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:41 smithi114 ceph-mon[40867]: 4.8 scrub starts 2024-09-21T12:31:41.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:41 smithi114 ceph-mon[40867]: 4.8 scrub ok 2024-09-21T12:31:41.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:41 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2843018609' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:31:41.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:41 smithi114 ceph-mon[30324]: pgmap v563: 188 pgs: 4 active+remapped+backfill_toofull, 184 active+clean; 1.7 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.29k op/s; 4433/82122 objects misplaced (5.398%) 2024-09-21T12:31:41.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:41 smithi114 ceph-mon[30324]: 9.0 starting backfill to osd.1 from (0'0,0'0] MAX to 306'11860 2024-09-21T12:31:41.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:41 smithi114 ceph-mon[30324]: 9.d starting backfill to osd.1 from (0'0,0'0] MAX to 306'13798 2024-09-21T12:31:41.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:41 smithi114 ceph-mon[30324]: 4.8 scrub starts 2024-09-21T12:31:41.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:41 smithi114 ceph-mon[30324]: 4.8 scrub ok 2024-09-21T12:31:41.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:41 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2843018609' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:31:41.814 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:41.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:41.851 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:41.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:41.893 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T12:31:41.951 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:41.978 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:41.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:41.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:42.010 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T12:31:42.081 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:42.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:42.163 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:42.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:42.264 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:42.307 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:42.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:42.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:42.409 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:42.443 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:42.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:42.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:42.545 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:42.574 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:42.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:42.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:42.610 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:31:42 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:31:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:31:42.610 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:42 smithi114 ceph-mon[40867]: 4.1c scrub starts 2024-09-21T12:31:42.610 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:42 smithi114 ceph-mon[40867]: 4.1c scrub ok 2024-09-21T12:31:42.610 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:42 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3640643591' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:31:42.610 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:42 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3640643591' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T12:31:42.610 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:42 smithi114 ceph-mon[40867]: osdmap e310: 8 total, 8 up, 7 in 2024-09-21T12:31:42.610 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:42 smithi114 ceph-mon[30324]: 4.1c scrub starts 2024-09-21T12:31:42.611 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:42 smithi114 ceph-mon[30324]: 4.1c scrub ok 2024-09-21T12:31:42.611 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:42 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3640643591' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:31:42.611 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:42 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3640643591' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T12:31:42.611 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:42 smithi114 ceph-mon[30324]: osdmap e310: 8 total, 8 up, 7 in 2024-09-21T12:31:42.676 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:42.733 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:42.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:42.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:42.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:42 smithi183 ceph-mon[33277]: 4.1c scrub starts 2024-09-21T12:31:42.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:42 smithi183 ceph-mon[33277]: 4.1c scrub ok 2024-09-21T12:31:42.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:42 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3640643591' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:31:42.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:42 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3640643591' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T12:31:42.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:42 smithi183 ceph-mon[33277]: osdmap e310: 8 total, 8 up, 7 in 2024-09-21T12:31:42.836 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:42.866 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:42.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:42.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:42.881 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:31:42.967 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:42.998 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:42.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:42.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:43.100 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:43.132 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:43.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:43.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:43.234 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:43.268 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:43.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:43.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:43.370 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:43.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:43.405 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:43.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:43.482 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T12:31:43.486 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:31:43.506 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:43.534 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:43.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:43.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:43.617 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:43 smithi114 ceph-mon[30324]: pgmap v565: 188 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 183 active+clean; 1.6 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.38k op/s; 4649/77238 objects misplaced (6.019%); 5.8 MiB/s, 92 objects/s recovering 2024-09-21T12:31:43.617 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:43 smithi114 ceph-mon[40867]: pgmap v565: 188 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 183 active+clean; 1.6 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.38k op/s; 4649/77238 objects misplaced (6.019%); 5.8 MiB/s, 92 objects/s recovering 2024-09-21T12:31:43.637 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:43.663 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:43.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:43.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:43.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:43 smithi183 ceph-mon[33277]: pgmap v565: 188 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 183 active+clean; 1.6 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.38k op/s; 4649/77238 objects misplaced (6.019%); 5.8 MiB/s, 92 objects/s recovering 2024-09-21T12:31:43.765 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:43.790 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:43.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:43.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:43.892 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:43.950 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:43.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:43.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:44.051 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:44.083 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:44.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:44.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:44.185 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:44.226 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:44.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:44.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:44.328 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:44.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:44.356 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:44.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:44.457 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:44.485 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:44.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:44.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:44.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:44 smithi114 ceph-mon[30324]: pgmap v567: 188 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 182 active+clean; 1.4 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 710 B/s rd, 0 B/s wr, 700 op/s; 4612/70008 objects misplaced (6.588%); 9.3 MiB/s, 154 objects/s recovering 2024-09-21T12:31:44.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:44 smithi114 ceph-mon[30324]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:31:44.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:44 smithi114 ceph-mon[30324]: osdmap e311: 8 total, 8 up, 7 in 2024-09-21T12:31:44.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:44 smithi114 ceph-mon[40867]: pgmap v567: 188 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 182 active+clean; 1.4 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 710 B/s rd, 0 B/s wr, 700 op/s; 4612/70008 objects misplaced (6.588%); 9.3 MiB/s, 154 objects/s recovering 2024-09-21T12:31:44.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:44 smithi114 ceph-mon[40867]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:31:44.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:44 smithi114 ceph-mon[40867]: osdmap e311: 8 total, 8 up, 7 in 2024-09-21T12:31:44.588 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:44.615 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:44.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:44.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:44.658 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:31:44.718 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:44.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:44 smithi183 ceph-mon[33277]: pgmap v567: 188 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 182 active+clean; 1.4 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 710 B/s rd, 0 B/s wr, 700 op/s; 4612/70008 objects misplaced (6.588%); 9.3 MiB/s, 154 objects/s recovering 2024-09-21T12:31:44.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:44 smithi183 ceph-mon[33277]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:31:44.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:44 smithi183 ceph-mon[33277]: osdmap e311: 8 total, 8 up, 7 in 2024-09-21T12:31:44.741 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:44.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:44.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:44.843 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:44.864 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:44.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:44.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:44.966 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:44.987 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:44.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:44.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:45.089 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:45.111 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:45.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:45.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:45.200 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:31:45.200 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:31:45.212 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:45.233 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:45.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:45.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:45.334 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:45.357 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:45.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:45.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:45.368 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-21T12:31:45.459 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:45.490 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:45.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:45.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:45.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:45 smithi114 ceph-mon[30324]: 9.15 scrub starts 2024-09-21T12:31:45.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:45 smithi114 ceph-mon[30324]: 9.15 scrub ok 2024-09-21T12:31:45.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:45 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/899787966' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:31:45.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:45 smithi114 ceph-mon[30324]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:31:45.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:45 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/899787966' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:31:45.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:45 smithi114 ceph-mon[30324]: osdmap e312: 8 total, 8 up, 7 in 2024-09-21T12:31:45.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:45 smithi114 ceph-mon[40867]: 9.15 scrub starts 2024-09-21T12:31:45.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:45 smithi114 ceph-mon[40867]: 9.15 scrub ok 2024-09-21T12:31:45.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:45 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/899787966' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:31:45.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:45 smithi114 ceph-mon[40867]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:31:45.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:45 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/899787966' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:31:45.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:45 smithi114 ceph-mon[40867]: osdmap e312: 8 total, 8 up, 7 in 2024-09-21T12:31:45.591 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:45.621 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:45.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:45.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:45.723 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:45.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:45 smithi183 ceph-mon[33277]: 9.15 scrub starts 2024-09-21T12:31:45.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:45 smithi183 ceph-mon[33277]: 9.15 scrub ok 2024-09-21T12:31:45.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:45 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/899787966' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:31:45.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:45 smithi183 ceph-mon[33277]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:31:45.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:45 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/899787966' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:31:45.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:45 smithi183 ceph-mon[33277]: osdmap e312: 8 total, 8 up, 7 in 2024-09-21T12:31:45.744 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:45.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:45.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:45.846 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:45.867 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:45.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:45.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:45.969 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:45.991 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:45.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:45.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:46.093 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:46.115 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:46.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:46.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:46.218 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:46.241 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:31:46.242 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:31:46.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:46.243 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:46.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:46.344 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:46.365 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:46.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:46.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:46.467 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:46.499 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:46.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:46.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:46.601 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:46.637 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:46.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:46.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:46.701 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:46 smithi114 ceph-mon[40867]: pgmap v570: 188 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 182 active+clean; 1.4 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 710 B/s rd, 0 B/s wr, 701 op/s; 4612/70008 objects misplaced (6.588%); 9.3 MiB/s, 154 objects/s recovering 2024-09-21T12:31:46.701 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:46 smithi114 ceph-mon[40867]: osdmap e313: 8 total, 8 up, 7 in 2024-09-21T12:31:46.701 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:46 smithi114 ceph-mon[30324]: pgmap v570: 188 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 182 active+clean; 1.4 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 710 B/s rd, 0 B/s wr, 701 op/s; 4612/70008 objects misplaced (6.588%); 9.3 MiB/s, 154 objects/s recovering 2024-09-21T12:31:46.701 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:46 smithi114 ceph-mon[30324]: osdmap e313: 8 total, 8 up, 7 in 2024-09-21T12:31:46.702 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T12:31:46.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:46 smithi183 ceph-mon[33277]: pgmap v570: 188 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 182 active+clean; 1.4 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 710 B/s rd, 0 B/s wr, 701 op/s; 4612/70008 objects misplaced (6.588%); 9.3 MiB/s, 154 objects/s recovering 2024-09-21T12:31:46.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:46 smithi183 ceph-mon[33277]: osdmap e313: 8 total, 8 up, 7 in 2024-09-21T12:31:46.738 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:46.786 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:46.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:46.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:46.888 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:46.912 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:46.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:46.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:47.013 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:47.036 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:47.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:47.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:47.137 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:47.171 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:47.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:47.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:47.273 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:47.304 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:47.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:47.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:47.406 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:47.451 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:47.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:47.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:47.553 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:47.578 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:47.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:47.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:47.680 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:47.706 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:47.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:47.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:47.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:31:47 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:31:47.808 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:47.833 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:47.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:47.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:47.885 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:31:47.885 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:31:47.886 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:31:47.886 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:31:47.886 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:31:47.886 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:31:47.886 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:31:47.886 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:31:47.886 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:31:47.886 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:31:47.887 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:31:47.887 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:31:47.887 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:31:47.887 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:31:47.887 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:31:47.887 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:31:47.934 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:47.964 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:47.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:47.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:48.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:47 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:31:48.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:47 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:31:48.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:47 smithi114 ceph-mon[40867]: osdmap e314: 8 total, 8 up, 7 in 2024-09-21T12:31:48.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:47 smithi114 ceph-mon[40867]: 3.b scrub starts 2024-09-21T12:31:48.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:47 smithi114 ceph-mon[40867]: 3.b scrub ok 2024-09-21T12:31:48.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:47 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:48.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:47 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3596443315' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:31:48.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:47 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:31:48.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:47 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:31:48.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:47 smithi114 ceph-mon[30324]: osdmap e314: 8 total, 8 up, 7 in 2024-09-21T12:31:48.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:47 smithi114 ceph-mon[30324]: 3.b scrub starts 2024-09-21T12:31:48.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:47 smithi114 ceph-mon[30324]: 3.b scrub ok 2024-09-21T12:31:48.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:47 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:48.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:47 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3596443315' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:31:48.066 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:48.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:48.106 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:48.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:48.179 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T12:31:48.207 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:48.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:47 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:31:48.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:47 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:31:48.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:47 smithi183 ceph-mon[33277]: osdmap e314: 8 total, 8 up, 7 in 2024-09-21T12:31:48.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:47 smithi183 ceph-mon[33277]: 3.b scrub starts 2024-09-21T12:31:48.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:47 smithi183 ceph-mon[33277]: 3.b scrub ok 2024-09-21T12:31:48.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:47 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:48.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:47 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3596443315' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:31:48.271 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:48.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:48.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:48.373 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:48.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:48.410 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:48.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:48.511 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:48.554 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:48.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:48.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:48.656 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:48.686 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:48.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:48.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:48.787 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:48.787 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:31:48.838 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:48.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:48.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:48.941 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:48.972 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:48.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:48.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:49.035 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T12:31:49.075 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:49.098 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:49.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:49.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:49.200 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:49.221 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:49.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:49.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:49.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:49 smithi114 ceph-mon[30324]: pgmap v573: 188 pgs: 3 active+remapped+backfill_toofull, 185 active+clean; 1.2 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2006/58671 objects misplaced (3.419%) 2024-09-21T12:31:49.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:49 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2174164400' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:31:49.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:49 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:31:49.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:49 smithi114 ceph-mon[30324]: 4.19 scrub starts 2024-09-21T12:31:49.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:49 smithi114 ceph-mon[30324]: 4.19 scrub ok 2024-09-21T12:31:49.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:49 smithi114 ceph-mon[30324]: 1.11 scrub starts 2024-09-21T12:31:49.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:49 smithi114 ceph-mon[30324]: 1.11 scrub ok 2024-09-21T12:31:49.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:49 smithi114 ceph-mon[40867]: pgmap v573: 188 pgs: 3 active+remapped+backfill_toofull, 185 active+clean; 1.2 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2006/58671 objects misplaced (3.419%) 2024-09-21T12:31:49.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:49 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2174164400' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:31:49.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:49 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:31:49.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:49 smithi114 ceph-mon[40867]: 4.19 scrub starts 2024-09-21T12:31:49.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:49 smithi114 ceph-mon[40867]: 4.19 scrub ok 2024-09-21T12:31:49.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:49 smithi114 ceph-mon[40867]: 1.11 scrub starts 2024-09-21T12:31:49.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:49 smithi114 ceph-mon[40867]: 1.11 scrub ok 2024-09-21T12:31:49.322 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:49.345 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:49.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:49.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:49.446 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:49.467 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:49.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:49.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:49.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:49 smithi183 ceph-mon[33277]: pgmap v573: 188 pgs: 3 active+remapped+backfill_toofull, 185 active+clean; 1.2 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 2006/58671 objects misplaced (3.419%) 2024-09-21T12:31:49.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:49 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2174164400' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:31:49.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:49 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:31:49.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:49 smithi183 ceph-mon[33277]: 4.19 scrub starts 2024-09-21T12:31:49.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:49 smithi183 ceph-mon[33277]: 4.19 scrub ok 2024-09-21T12:31:49.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:49 smithi183 ceph-mon[33277]: 1.11 scrub starts 2024-09-21T12:31:49.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:49 smithi183 ceph-mon[33277]: 1.11 scrub ok 2024-09-21T12:31:49.569 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:49.591 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:49.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:49.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:49.693 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:49.714 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:49.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:49.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:49.816 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:49.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:49.839 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:49.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:49.939 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:49.961 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:49.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:49.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:50.063 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:50.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:50.084 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:50.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:50.185 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:50.207 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:50.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:50.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:50.308 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:50.331 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:50.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:50.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:50.432 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:50.453 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:50.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:50.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:50.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:50 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T12:31:50.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:50 smithi183 ceph-mon[33277]: osdmap e315: 8 total, 8 up, 7 in 2024-09-21T12:31:50.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:50 smithi183 ceph-mon[33277]: 5.5 scrub starts 2024-09-21T12:31:50.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:50 smithi183 ceph-mon[33277]: 5.5 scrub ok 2024-09-21T12:31:50.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:50 smithi183 ceph-mon[33277]: osdmap e316: 8 total, 8 up, 7 in 2024-09-21T12:31:50.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:50 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T12:31:50.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:50 smithi114 ceph-mon[30324]: osdmap e315: 8 total, 8 up, 7 in 2024-09-21T12:31:50.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:50 smithi114 ceph-mon[30324]: 5.5 scrub starts 2024-09-21T12:31:50.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:50 smithi114 ceph-mon[30324]: 5.5 scrub ok 2024-09-21T12:31:50.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:50 smithi114 ceph-mon[30324]: osdmap e316: 8 total, 8 up, 7 in 2024-09-21T12:31:50.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:50 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T12:31:50.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:50 smithi114 ceph-mon[40867]: osdmap e315: 8 total, 8 up, 7 in 2024-09-21T12:31:50.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:50 smithi114 ceph-mon[40867]: 5.5 scrub starts 2024-09-21T12:31:50.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:50 smithi114 ceph-mon[40867]: 5.5 scrub ok 2024-09-21T12:31:50.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:50 smithi114 ceph-mon[40867]: osdmap e316: 8 total, 8 up, 7 in 2024-09-21T12:31:50.555 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:50.575 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:50.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:50.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:50.677 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:50.699 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:50.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:50.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:50.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:31:50.800 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:50.826 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:50.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:50.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:50.927 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:50.949 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:50.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:50.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:51.050 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:51.060 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:51 smithi114 ceph-mon[30324]: pgmap v575: 188 pgs: 3 active+remapped+backfill_toofull, 185 active+clean; 1.1 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 1861/56196 objects misplaced (3.312%) 2024-09-21T12:31:51.060 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:51 smithi114 ceph-mon[30324]: 1.7 scrub starts 2024-09-21T12:31:51.061 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:51 smithi114 ceph-mon[30324]: 1.7 scrub ok 2024-09-21T12:31:51.061 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:51 smithi114 ceph-mon[30324]: osdmap e317: 8 total, 8 up, 7 in 2024-09-21T12:31:51.096 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:51.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:51.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:51.198 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:51.225 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:51.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:51.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:51.327 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:51.362 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:51.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:51.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:51.464 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:51.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:51.486 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:51.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:51.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:51 smithi183 ceph-mon[33277]: pgmap v575: 188 pgs: 3 active+remapped+backfill_toofull, 185 active+clean; 1.1 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 1861/56196 objects misplaced (3.312%) 2024-09-21T12:31:51.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:51 smithi183 ceph-mon[33277]: 1.7 scrub starts 2024-09-21T12:31:51.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:51 smithi183 ceph-mon[33277]: 1.7 scrub ok 2024-09-21T12:31:51.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:51 smithi183 ceph-mon[33277]: osdmap e317: 8 total, 8 up, 7 in 2024-09-21T12:31:51.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:51 smithi114 ceph-mon[40867]: pgmap v575: 188 pgs: 3 active+remapped+backfill_toofull, 185 active+clean; 1.1 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 1861/56196 objects misplaced (3.312%) 2024-09-21T12:31:51.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:51 smithi114 ceph-mon[40867]: 1.7 scrub starts 2024-09-21T12:31:51.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:51 smithi114 ceph-mon[40867]: 1.7 scrub ok 2024-09-21T12:31:51.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:51 smithi114 ceph-mon[40867]: osdmap e317: 8 total, 8 up, 7 in 2024-09-21T12:31:51.587 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:51.614 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:51.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:51.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:51.715 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:51.737 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:51.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:51.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:51.838 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:51.860 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:51.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:51.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:51.962 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:51.987 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:51.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:51.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:52.089 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:52.111 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:52.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:52.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:52.213 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:52.246 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:52.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:52.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:52.347 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:52.372 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:52.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:52.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:52.473 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:52.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:52 smithi183 ceph-mon[33277]: 5.12 scrub starts 2024-09-21T12:31:52.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:52 smithi183 ceph-mon[33277]: 5.12 scrub ok 2024-09-21T12:31:52.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:52 smithi183 ceph-mon[33277]: 9.2 starting backfill to osd.3 from (260'5900,297'10890] MIN to 314'12214 2024-09-21T12:31:52.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:52 smithi183 ceph-mon[33277]: 9.8 starting backfill to osd.3 from (262'6300,282'9984] MIN to 314'12158 2024-09-21T12:31:52.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:52 smithi183 ceph-mon[33277]: 9.19 starting backfill to osd.3 from (0'0,0'0] MAX to 314'12207 2024-09-21T12:31:52.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:52 smithi183 ceph-mon[33277]: 9.14 starting backfill to osd.3 from (274'8389,312'12155] MIN to 314'12244 2024-09-21T12:31:52.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:52 smithi183 ceph-mon[33277]: 9.7 starting backfill to osd.3 from (261'6100,291'10672] MIN to 314'12285 2024-09-21T12:31:52.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:52 smithi183 ceph-mon[33277]: osdmap e318: 8 total, 8 up, 7 in 2024-09-21T12:31:52.501 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:52.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:52.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:52.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:52 smithi114 ceph-mon[40867]: 5.12 scrub starts 2024-09-21T12:31:52.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:52 smithi114 ceph-mon[40867]: 5.12 scrub ok 2024-09-21T12:31:52.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:52 smithi114 ceph-mon[40867]: 9.2 starting backfill to osd.3 from (260'5900,297'10890] MIN to 314'12214 2024-09-21T12:31:52.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:52 smithi114 ceph-mon[40867]: 9.8 starting backfill to osd.3 from (262'6300,282'9984] MIN to 314'12158 2024-09-21T12:31:52.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:52 smithi114 ceph-mon[40867]: 9.19 starting backfill to osd.3 from (0'0,0'0] MAX to 314'12207 2024-09-21T12:31:52.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:52 smithi114 ceph-mon[40867]: 9.14 starting backfill to osd.3 from (274'8389,312'12155] MIN to 314'12244 2024-09-21T12:31:52.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:52 smithi114 ceph-mon[40867]: 9.7 starting backfill to osd.3 from (261'6100,291'10672] MIN to 314'12285 2024-09-21T12:31:52.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:52 smithi114 ceph-mon[40867]: osdmap e318: 8 total, 8 up, 7 in 2024-09-21T12:31:52.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:52 smithi114 ceph-mon[30324]: 5.12 scrub starts 2024-09-21T12:31:52.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:52 smithi114 ceph-mon[30324]: 5.12 scrub ok 2024-09-21T12:31:52.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:52 smithi114 ceph-mon[30324]: 9.2 starting backfill to osd.3 from (260'5900,297'10890] MIN to 314'12214 2024-09-21T12:31:52.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:52 smithi114 ceph-mon[30324]: 9.8 starting backfill to osd.3 from (262'6300,282'9984] MIN to 314'12158 2024-09-21T12:31:52.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:52 smithi114 ceph-mon[30324]: 9.19 starting backfill to osd.3 from (0'0,0'0] MAX to 314'12207 2024-09-21T12:31:52.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:52 smithi114 ceph-mon[30324]: 9.14 starting backfill to osd.3 from (274'8389,312'12155] MIN to 314'12244 2024-09-21T12:31:52.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:52 smithi114 ceph-mon[30324]: 9.7 starting backfill to osd.3 from (261'6100,291'10672] MIN to 314'12285 2024-09-21T12:31:52.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:52 smithi114 ceph-mon[30324]: osdmap e318: 8 total, 8 up, 7 in 2024-09-21T12:31:52.603 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:52.624 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:52.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:52.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:52.726 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:52.747 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:52.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:52.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:52.848 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:52.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:52.870 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:52.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:52.971 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:52.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:52.994 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:52.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:53.024 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:31:52 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:31:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:31:53.095 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:53.133 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:53.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:53.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:53.234 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:53.256 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:53.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:53.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:53.357 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:53.379 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:53.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:53.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:53.480 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:53.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:53 smithi183 ceph-mon[33277]: pgmap v578: 188 pgs: 4 active+remapped+backfill_toofull, 184 active+clean; 1.1 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 814 op/s; 2135/52398 objects misplaced (4.075%); 13 MiB/s, 210 objects/s recovering 2024-09-21T12:31:53.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:53 smithi183 ceph-mon[33277]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:31:53.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:53 smithi183 ceph-mon[33277]: 6.16 scrub starts 2024-09-21T12:31:53.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:53 smithi183 ceph-mon[33277]: 6.16 scrub ok 2024-09-21T12:31:53.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:53 smithi183 ceph-mon[33277]: osdmap e319: 8 total, 8 up, 7 in 2024-09-21T12:31:53.502 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:53.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:53.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:53.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:53 smithi114 ceph-mon[40867]: pgmap v578: 188 pgs: 4 active+remapped+backfill_toofull, 184 active+clean; 1.1 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 814 op/s; 2135/52398 objects misplaced (4.075%); 13 MiB/s, 210 objects/s recovering 2024-09-21T12:31:53.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:53 smithi114 ceph-mon[40867]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:31:53.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:53 smithi114 ceph-mon[40867]: 6.16 scrub starts 2024-09-21T12:31:53.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:53 smithi114 ceph-mon[40867]: 6.16 scrub ok 2024-09-21T12:31:53.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:53 smithi114 ceph-mon[40867]: osdmap e319: 8 total, 8 up, 7 in 2024-09-21T12:31:53.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:53 smithi114 ceph-mon[30324]: pgmap v578: 188 pgs: 4 active+remapped+backfill_toofull, 184 active+clean; 1.1 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 814 op/s; 2135/52398 objects misplaced (4.075%); 13 MiB/s, 210 objects/s recovering 2024-09-21T12:31:53.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:53 smithi114 ceph-mon[30324]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:31:53.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:53 smithi114 ceph-mon[30324]: 6.16 scrub starts 2024-09-21T12:31:53.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:53 smithi114 ceph-mon[30324]: 6.16 scrub ok 2024-09-21T12:31:53.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:53 smithi114 ceph-mon[30324]: osdmap e319: 8 total, 8 up, 7 in 2024-09-21T12:31:53.606 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:53.704 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:53.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:53.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:53.806 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:53.828 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:53.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:53.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:53.930 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:53.951 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:53.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:53.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:53.966 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:31:53.966 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:31:53.966 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:31:53.966 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:31:53.966 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:31:54.054 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:54.076 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:54.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:54.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:54.178 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:54.207 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:54.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:54.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:54.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:54 smithi114 ceph-mon[30324]: 3.f scrub starts 2024-09-21T12:31:54.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:54 smithi114 ceph-mon[30324]: 3.f scrub ok 2024-09-21T12:31:54.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:54 smithi114 ceph-mon[30324]: osdmap e320: 8 total, 8 up, 7 in 2024-09-21T12:31:54.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:54 smithi114 ceph-mon[40867]: 3.f scrub starts 2024-09-21T12:31:54.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:54 smithi114 ceph-mon[40867]: 3.f scrub ok 2024-09-21T12:31:54.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:54 smithi114 ceph-mon[40867]: osdmap e320: 8 total, 8 up, 7 in 2024-09-21T12:31:54.308 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:54.339 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:54.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:54.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:54.440 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:54.465 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:54.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:54.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:54.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:54 smithi183 ceph-mon[33277]: 3.f scrub starts 2024-09-21T12:31:54.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:54 smithi183 ceph-mon[33277]: 3.f scrub ok 2024-09-21T12:31:54.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:54 smithi183 ceph-mon[33277]: osdmap e320: 8 total, 8 up, 7 in 2024-09-21T12:31:54.568 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:54.596 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:54.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:54.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:54.698 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:54.733 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:54.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:54.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:54.835 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:54.871 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:54.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:54.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:54.973 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:55.014 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:55.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:55.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:55.115 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:55.149 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:55.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:55.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:55.251 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:55.283 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:55.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:55.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:55.385 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:55.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:55.423 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:55.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:55.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:55 smithi183 ceph-mon[33277]: 6.11 scrub starts 2024-09-21T12:31:55.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:55 smithi183 ceph-mon[33277]: 6.11 scrub ok 2024-09-21T12:31:55.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:55 smithi183 ceph-mon[33277]: pgmap v581: 188 pgs: 1 peering, 2 active+clean+remapped, 4 active+remapped+backfill_toofull, 181 active+clean; 985 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 695 op/s; 2343/47868 objects misplaced (4.895%); 13 MiB/s, 209 objects/s recovering 2024-09-21T12:31:55.524 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:55.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:55 smithi114 ceph-mon[30324]: 6.11 scrub starts 2024-09-21T12:31:55.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:55 smithi114 ceph-mon[30324]: 6.11 scrub ok 2024-09-21T12:31:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:55 smithi114 ceph-mon[30324]: pgmap v581: 188 pgs: 1 peering, 2 active+clean+remapped, 4 active+remapped+backfill_toofull, 181 active+clean; 985 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 695 op/s; 2343/47868 objects misplaced (4.895%); 13 MiB/s, 209 objects/s recovering 2024-09-21T12:31:55.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:55 smithi114 ceph-mon[40867]: 6.11 scrub starts 2024-09-21T12:31:55.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:55 smithi114 ceph-mon[40867]: 6.11 scrub ok 2024-09-21T12:31:55.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:55 smithi114 ceph-mon[40867]: pgmap v581: 188 pgs: 1 peering, 2 active+clean+remapped, 4 active+remapped+backfill_toofull, 181 active+clean; 985 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 695 op/s; 2343/47868 objects misplaced (4.895%); 13 MiB/s, 209 objects/s recovering 2024-09-21T12:31:55.553 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:55.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:55.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:55.655 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:55.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:55.691 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:55.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:55.792 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:55.822 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:55.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:55.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:55.924 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:55.944 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:55.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:55.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:56.046 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:56.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:56.087 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:56.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:56.187 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:56.222 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:56.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:56.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:56.324 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:56.345 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:56.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:56.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:56.447 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:56.482 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:56.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:56.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:56.584 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:56.620 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:56.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:56.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:56.722 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:56.752 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:56.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:56.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:56.854 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:56.879 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:56.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:56.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:56.981 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:57.010 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:57.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:57.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:57.111 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:57.144 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:57.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:57.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:57.215 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:31:57.245 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T12:31:57.245 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:57.289 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:57.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:57.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:57.349 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:57 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:31:57.349 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:57 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:31:57.349 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:57 smithi114 ceph-mon[40867]: pgmap v583: 188 pgs: 1 peering, 2 active+clean+remapped, 4 active+remapped+backfill_toofull, 181 active+clean; 985 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 581 op/s; 2343/47868 objects misplaced (4.895%); 11 MiB/s, 175 objects/s recovering 2024-09-21T12:31:57.349 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:57 smithi114 ceph-mon[40867]: osdmap e321: 8 total, 8 up, 7 in 2024-09-21T12:31:57.350 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:57 smithi114 ceph-mon[40867]: 5.1b scrub starts 2024-09-21T12:31:57.350 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:57 smithi114 ceph-mon[40867]: 5.1b scrub ok 2024-09-21T12:31:57.350 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:57 smithi114 ceph-mon[40867]: 4.f scrub starts 2024-09-21T12:31:57.350 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:57 smithi114 ceph-mon[40867]: 4.f scrub ok 2024-09-21T12:31:57.350 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:57 smithi114 ceph-mon[40867]: osdmap e322: 8 total, 8 up, 7 in 2024-09-21T12:31:57.350 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:57 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:31:57.350 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:57 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:31:57.350 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:57 smithi114 ceph-mon[30324]: pgmap v583: 188 pgs: 1 peering, 2 active+clean+remapped, 4 active+remapped+backfill_toofull, 181 active+clean; 985 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 581 op/s; 2343/47868 objects misplaced (4.895%); 11 MiB/s, 175 objects/s recovering 2024-09-21T12:31:57.350 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:57 smithi114 ceph-mon[30324]: osdmap e321: 8 total, 8 up, 7 in 2024-09-21T12:31:57.350 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:57 smithi114 ceph-mon[30324]: 5.1b scrub starts 2024-09-21T12:31:57.350 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:57 smithi114 ceph-mon[30324]: 5.1b scrub ok 2024-09-21T12:31:57.350 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:57 smithi114 ceph-mon[30324]: 4.f scrub starts 2024-09-21T12:31:57.350 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:57 smithi114 ceph-mon[30324]: 4.f scrub ok 2024-09-21T12:31:57.351 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:57 smithi114 ceph-mon[30324]: osdmap e322: 8 total, 8 up, 7 in 2024-09-21T12:31:57.393 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:57.423 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:57.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:57.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:57.488 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:31:57 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:31:57.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:57 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:31:57.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:57 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:31:57.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:57 smithi183 ceph-mon[33277]: pgmap v583: 188 pgs: 1 peering, 2 active+clean+remapped, 4 active+remapped+backfill_toofull, 181 active+clean; 985 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 581 op/s; 2343/47868 objects misplaced (4.895%); 11 MiB/s, 175 objects/s recovering 2024-09-21T12:31:57.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:57 smithi183 ceph-mon[33277]: osdmap e321: 8 total, 8 up, 7 in 2024-09-21T12:31:57.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:57 smithi183 ceph-mon[33277]: 5.1b scrub starts 2024-09-21T12:31:57.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:57 smithi183 ceph-mon[33277]: 5.1b scrub ok 2024-09-21T12:31:57.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:57 smithi183 ceph-mon[33277]: 4.f scrub starts 2024-09-21T12:31:57.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:57 smithi183 ceph-mon[33277]: 4.f scrub ok 2024-09-21T12:31:57.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:57 smithi183 ceph-mon[33277]: osdmap e322: 8 total, 8 up, 7 in 2024-09-21T12:31:57.525 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:57.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:57.546 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:57.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:57.647 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:57.675 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:57.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:57.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:57.777 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:57.816 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:57.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:57.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:57.918 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:57.945 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:57.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:57.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:58.047 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:58.075 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:58.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:58.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:58.177 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:58.208 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:58.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:58.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:58.309 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:58.340 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:58.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:58.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:58.362 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:58 smithi114 ceph-mon[40867]: 5.9 scrub starts 2024-09-21T12:31:58.362 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:58 smithi114 ceph-mon[40867]: 5.9 scrub ok 2024-09-21T12:31:58.362 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:58 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:58.362 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:58 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:31:58.362 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:58 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:31:58.362 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:58 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:31:58.362 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:58 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:31:58.362 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:58 smithi114 ceph-mon[40867]: 3.17 scrub starts 2024-09-21T12:31:58.362 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:58 smithi114 ceph-mon[40867]: 3.17 scrub ok 2024-09-21T12:31:58.362 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:58 smithi114 ceph-mon[30324]: 5.9 scrub starts 2024-09-21T12:31:58.363 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:58 smithi114 ceph-mon[30324]: 5.9 scrub ok 2024-09-21T12:31:58.363 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:58 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:58.363 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:58 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:31:58.363 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:58 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:31:58.363 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:58 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:31:58.363 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:58 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:31:58.363 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:58 smithi114 ceph-mon[30324]: 3.17 scrub starts 2024-09-21T12:31:58.363 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:58 smithi114 ceph-mon[30324]: 3.17 scrub ok 2024-09-21T12:31:58.442 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:58.470 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:58.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:58.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:58.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:58 smithi183 ceph-mon[33277]: 5.9 scrub starts 2024-09-21T12:31:58.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:58 smithi183 ceph-mon[33277]: 5.9 scrub ok 2024-09-21T12:31:58.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:58 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:58.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:58 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:31:58.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:58 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:31:58.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:58 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:31:58.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:58 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:31:58.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:58 smithi183 ceph-mon[33277]: 3.17 scrub starts 2024-09-21T12:31:58.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:58 smithi183 ceph-mon[33277]: 3.17 scrub ok 2024-09-21T12:31:58.545 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T12:31:58.572 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:58.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:58.641 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:58.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:58.742 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:58.769 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:58.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:58.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:58.840 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:31:58.871 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:31:58.893 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:58.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:58.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:31:58.994 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:31:59.019 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:59.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:59.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:31:59.120 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:31:59.140 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:31:59.173 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:59.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:59.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:31:59.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:59 smithi114 ceph-mon[30324]: pgmap v586: 188 pgs: 4 active+remapped+backfill_toofull, 184 active+clean; 755 MiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 769 op/s; 1844/36807 objects misplaced (5.010%) 2024-09-21T12:31:59.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:59 smithi114 ceph-mon[30324]: 9.14 scrub starts 2024-09-21T12:31:59.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:59 smithi114 ceph-mon[30324]: osdmap e323: 8 total, 8 up, 7 in 2024-09-21T12:31:59.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:31:59 smithi114 ceph-mon[30324]: 9.14 scrub ok 2024-09-21T12:31:59.275 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:31:59.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:59 smithi114 ceph-mon[40867]: pgmap v586: 188 pgs: 4 active+remapped+backfill_toofull, 184 active+clean; 755 MiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 769 op/s; 1844/36807 objects misplaced (5.010%) 2024-09-21T12:31:59.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:59 smithi114 ceph-mon[40867]: 9.14 scrub starts 2024-09-21T12:31:59.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:59 smithi114 ceph-mon[40867]: osdmap e323: 8 total, 8 up, 7 in 2024-09-21T12:31:59.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:31:59 smithi114 ceph-mon[40867]: 9.14 scrub ok 2024-09-21T12:31:59.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:59.314 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:59.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:31:59.415 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:31:59.448 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:59.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:59.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:31:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:59 smithi183 ceph-mon[33277]: pgmap v586: 188 pgs: 4 active+remapped+backfill_toofull, 184 active+clean; 755 MiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 769 op/s; 1844/36807 objects misplaced (5.010%) 2024-09-21T12:31:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:59 smithi183 ceph-mon[33277]: 9.14 scrub starts 2024-09-21T12:31:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:59 smithi183 ceph-mon[33277]: osdmap e323: 8 total, 8 up, 7 in 2024-09-21T12:31:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:31:59 smithi183 ceph-mon[33277]: 9.14 scrub ok 2024-09-21T12:31:59.550 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:31:59.598 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:59.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:59.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:31:59.700 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:31:59.745 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:31:59.745 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:31:59.754 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:31:59.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:59.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:31:59.780 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:31:59.780 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:31:59.816 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:31:59.816 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:31:59.856 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:31:59.863 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:31:59.863 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:31:59.883 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:31:59.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:31:59.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:31:59.898 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:31:59.899 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:31:59.934 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:31:59.944 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:31:59.944 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:31:59.980 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:31:59.981 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:31:59.985 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:00.013 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:00.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:00.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:00.024 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:32:00.025 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:32:00.115 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:00.126 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-21T12:32:00.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:00.162 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:00.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:00.262 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:00.287 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:00.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:00.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:00.389 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:00.418 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:00.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:00.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:00.475 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:00 smithi114 ceph-mon[40867]: 9.2 scrub starts 2024-09-21T12:32:00.475 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:00 smithi114 ceph-mon[40867]: 9.2 scrub ok 2024-09-21T12:32:00.475 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:00 smithi114 ceph-mon[40867]: 4.2 scrub starts 2024-09-21T12:32:00.475 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:00 smithi114 ceph-mon[40867]: 4.2 scrub ok 2024-09-21T12:32:00.475 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:00 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1476704787' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:32:00.475 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:00 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/377415094' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:32:00.475 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:00 smithi114 ceph-mon[30324]: 9.2 scrub starts 2024-09-21T12:32:00.476 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:00 smithi114 ceph-mon[30324]: 9.2 scrub ok 2024-09-21T12:32:00.476 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:00 smithi114 ceph-mon[30324]: 4.2 scrub starts 2024-09-21T12:32:00.476 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:00 smithi114 ceph-mon[30324]: 4.2 scrub ok 2024-09-21T12:32:00.476 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:00 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1476704787' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:32:00.476 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:00 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/377415094' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:32:00.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:00 smithi183 ceph-mon[33277]: 9.2 scrub starts 2024-09-21T12:32:00.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:00 smithi183 ceph-mon[33277]: 9.2 scrub ok 2024-09-21T12:32:00.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:00 smithi183 ceph-mon[33277]: 4.2 scrub starts 2024-09-21T12:32:00.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:00 smithi183 ceph-mon[33277]: 4.2 scrub ok 2024-09-21T12:32:00.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:00 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1476704787' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:32:00.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:00 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/377415094' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:32:00.520 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:00.547 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:00.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:00.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:00.649 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:00.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:00.679 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:00.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:00.780 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:00.810 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:00.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:00.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:00.912 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:00.944 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:00.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:00.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:01.046 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:01.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:01.085 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:01.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:01.133 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T12:32:01.186 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:01.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:01.209 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:01.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:01.310 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:01.331 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:01.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:01.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:01.432 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:01.453 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:01.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:01.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:01 smithi183 ceph-mon[33277]: pgmap v588: 188 pgs: 4 active+remapped+backfill_toofull, 184 active+clean; 708 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 883 B/s rd, 0 B/s wr, 765 op/s; 1688/34569 objects misplaced (4.883%) 2024-09-21T12:32:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:01 smithi183 ceph-mon[33277]: 4.e scrub starts 2024-09-21T12:32:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:01 smithi183 ceph-mon[33277]: 4.e scrub ok 2024-09-21T12:32:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:01 smithi183 ceph-mon[33277]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:32:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:01 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1476704787' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:32:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:01 smithi183 ceph-mon[33277]: osdmap e324: 8 total, 8 up, 7 in 2024-09-21T12:32:01.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:01 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3780694936' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:32:01.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:01 smithi114 ceph-mon[40867]: pgmap v588: 188 pgs: 4 active+remapped+backfill_toofull, 184 active+clean; 708 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 883 B/s rd, 0 B/s wr, 765 op/s; 1688/34569 objects misplaced (4.883%) 2024-09-21T12:32:01.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:01 smithi114 ceph-mon[40867]: 4.e scrub starts 2024-09-21T12:32:01.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:01 smithi114 ceph-mon[40867]: 4.e scrub ok 2024-09-21T12:32:01.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:01 smithi114 ceph-mon[40867]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:32:01.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:01 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1476704787' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:32:01.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:01 smithi114 ceph-mon[40867]: osdmap e324: 8 total, 8 up, 7 in 2024-09-21T12:32:01.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:01 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3780694936' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:32:01.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:01 smithi114 ceph-mon[30324]: pgmap v588: 188 pgs: 4 active+remapped+backfill_toofull, 184 active+clean; 708 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 883 B/s rd, 0 B/s wr, 765 op/s; 1688/34569 objects misplaced (4.883%) 2024-09-21T12:32:01.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:01 smithi114 ceph-mon[30324]: 4.e scrub starts 2024-09-21T12:32:01.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:01 smithi114 ceph-mon[30324]: 4.e scrub ok 2024-09-21T12:32:01.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:01 smithi114 ceph-mon[30324]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:32:01.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:01 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1476704787' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:32:01.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:01 smithi114 ceph-mon[30324]: osdmap e324: 8 total, 8 up, 7 in 2024-09-21T12:32:01.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:01 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3780694936' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:32:01.556 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:01.577 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:01.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:01.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:01.678 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:01.702 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:01.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:01.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:01.752 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:32:01.752 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:32:01.803 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:01.827 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:01.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:01.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:01.929 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:01.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:01.950 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:01.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:02.051 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:02.072 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:02.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:02.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:02.174 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:02.195 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:02.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:02.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:02.296 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:02.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:02.319 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:02.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:02.420 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:02.442 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:02.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:02.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:02.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:02 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3780694936' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T12:32:02.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:02 smithi183 ceph-mon[33277]: osdmap e325: 8 total, 8 up, 7 in 2024-09-21T12:32:02.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:02 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:32:02.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:02 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:32:02.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:02 smithi183 ceph-mon[33277]: osdmap e326: 8 total, 8 up, 7 in 2024-09-21T12:32:02.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:02 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3780694936' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T12:32:02.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:02 smithi114 ceph-mon[40867]: osdmap e325: 8 total, 8 up, 7 in 2024-09-21T12:32:02.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:02 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:32:02.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:02 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:32:02.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:02 smithi114 ceph-mon[40867]: osdmap e326: 8 total, 8 up, 7 in 2024-09-21T12:32:02.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:02 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3780694936' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T12:32:02.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:02 smithi114 ceph-mon[30324]: osdmap e325: 8 total, 8 up, 7 in 2024-09-21T12:32:02.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:02 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:32:02.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:02 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:32:02.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:02 smithi114 ceph-mon[30324]: osdmap e326: 8 total, 8 up, 7 in 2024-09-21T12:32:02.543 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:02.565 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:02.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:02.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:02.567 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T12:32:02.667 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:02.688 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:02.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:02.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:02.789 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:02.811 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:02.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:02.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:02.912 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:02.935 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:02.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:02.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:03.019 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:32:02 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:32:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:32:03.036 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:03.060 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:03.061 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T12:32:03.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:03.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:03.163 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:03.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:03.224 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:03.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:03.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:03 smithi114 ceph-mon[40867]: pgmap v591: 188 pgs: 1 unknown, 5 peering, 4 active+remapped+backfill_toofull, 178 active+clean; 580 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s; 1136/28392 objects misplaced (4.001%) 2024-09-21T12:32:03.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:03 smithi114 ceph-mon[40867]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T12:32:03.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:03 smithi114 ceph-mon[40867]: osdmap e327: 8 total, 8 up, 7 in 2024-09-21T12:32:03.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:03 smithi114 ceph-mon[30324]: pgmap v591: 188 pgs: 1 unknown, 5 peering, 4 active+remapped+backfill_toofull, 178 active+clean; 580 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s; 1136/28392 objects misplaced (4.001%) 2024-09-21T12:32:03.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:03 smithi114 ceph-mon[30324]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T12:32:03.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:03 smithi114 ceph-mon[30324]: osdmap e327: 8 total, 8 up, 7 in 2024-09-21T12:32:03.325 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:03.335 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T12:32:03.353 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:03.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:03.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:03.455 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:03.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:03 smithi183 ceph-mon[33277]: pgmap v591: 188 pgs: 1 unknown, 5 peering, 4 active+remapped+backfill_toofull, 178 active+clean; 580 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s; 1136/28392 objects misplaced (4.001%) 2024-09-21T12:32:03.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:03 smithi183 ceph-mon[33277]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T12:32:03.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:03 smithi183 ceph-mon[33277]: osdmap e327: 8 total, 8 up, 7 in 2024-09-21T12:32:03.528 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:03.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:03.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:03.631 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:03.658 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:03.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:03.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:03.663 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:03.685 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:03.760 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:03.791 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:03.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:03.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:03.892 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:03.924 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:03.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:03.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:04.026 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:04.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:04.076 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:04.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:04.177 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:04.202 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:04.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:04.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:04.304 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:04.331 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:04.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:04.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:04.434 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:04.457 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:04 smithi114 ceph-mon[40867]: 9.18 continuing backfill to osd.6 from (274'8258,324'12571] MIN to 324'12571 2024-09-21T12:32:04.457 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:04 smithi114 ceph-mon[40867]: 9.18 continuing backfill to osd.7 from (274'8258,324'12571] MIN to 324'12571 2024-09-21T12:32:04.457 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:04 smithi114 ceph-mon[40867]: 9.a starting backfill to osd.5 from (295'11486,315'14489] MIN to 324'15143 2024-09-21T12:32:04.457 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:04 smithi114 ceph-mon[40867]: 9.8 starting backfill to osd.5 from (262'6300,282'9984] MIN to 324'12472 2024-09-21T12:32:04.457 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:04 smithi114 ceph-mon[30324]: 9.18 continuing backfill to osd.6 from (274'8258,324'12571] MIN to 324'12571 2024-09-21T12:32:04.457 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:04 smithi114 ceph-mon[30324]: 9.18 continuing backfill to osd.7 from (274'8258,324'12571] MIN to 324'12571 2024-09-21T12:32:04.457 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:04 smithi114 ceph-mon[30324]: 9.a starting backfill to osd.5 from (295'11486,315'14489] MIN to 324'15143 2024-09-21T12:32:04.458 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:04 smithi114 ceph-mon[30324]: 9.8 starting backfill to osd.5 from (262'6300,282'9984] MIN to 324'12472 2024-09-21T12:32:04.458 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:32:04.458 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:32:04.460 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:04.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:04.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:04.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:04 smithi183 ceph-mon[33277]: 9.18 continuing backfill to osd.6 from (274'8258,324'12571] MIN to 324'12571 2024-09-21T12:32:04.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:04 smithi183 ceph-mon[33277]: 9.18 continuing backfill to osd.7 from (274'8258,324'12571] MIN to 324'12571 2024-09-21T12:32:04.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:04 smithi183 ceph-mon[33277]: 9.a starting backfill to osd.5 from (295'11486,315'14489] MIN to 324'15143 2024-09-21T12:32:04.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:04 smithi183 ceph-mon[33277]: 9.8 starting backfill to osd.5 from (262'6300,282'9984] MIN to 324'12472 2024-09-21T12:32:04.562 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:04.604 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:04.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:04.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:04.705 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:04.740 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:04.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:04.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:04.842 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:04.878 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:04.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:04.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:04.982 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:05.010 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:05.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:05.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:05.066 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:05.112 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:05.159 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:05.160 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:05 smithi114 ceph-mon[30324]: pgmap v594: 188 pgs: 3 remapped+peering, 5 peering, 3 active+remapped+backfill_toofull, 177 active+clean; 378 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 691/18705 objects misplaced (3.694%) 2024-09-21T12:32:05.160 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:05 smithi114 ceph-mon[30324]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:32:05.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:05.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:05.261 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:05.291 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:05.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:05.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:05.393 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:05.408 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:05 smithi183 ceph-mon[33277]: pgmap v594: 188 pgs: 3 remapped+peering, 5 peering, 3 active+remapped+backfill_toofull, 177 active+clean; 378 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 691/18705 objects misplaced (3.694%) 2024-09-21T12:32:05.408 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:05 smithi183 ceph-mon[33277]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:32:05.415 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:05.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:05.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:05.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:05 smithi114 ceph-mon[40867]: pgmap v594: 188 pgs: 3 remapped+peering, 5 peering, 3 active+remapped+backfill_toofull, 177 active+clean; 378 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 691/18705 objects misplaced (3.694%) 2024-09-21T12:32:05.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:05 smithi114 ceph-mon[40867]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:32:05.516 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T12:32:05.516 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:05.538 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:05.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:05.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:05.588 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:32:05.639 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:05.660 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:05.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:05.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:05.762 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:05.789 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:05.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:05.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:05.891 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:05.912 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:05.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:05.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:06.014 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:06.036 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:06.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:06.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:06.065 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:06.068 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:06.138 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:06.159 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:06.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:06.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:06.177 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T12:32:06.261 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:06.304 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:06.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:06.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:06.328 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:06 smithi114 ceph-mon[30324]: osdmap e328: 8 total, 8 up, 7 in 2024-09-21T12:32:06.328 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:06 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4124588655' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:32:06.329 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:06 smithi114 ceph-mon[40867]: osdmap e328: 8 total, 8 up, 7 in 2024-09-21T12:32:06.329 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:06 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/4124588655' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:32:06.406 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:06.440 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:06.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:06.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:06.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:06 smithi183 ceph-mon[33277]: osdmap e328: 8 total, 8 up, 7 in 2024-09-21T12:32:06.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:06 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/4124588655' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:32:06.542 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:06.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:06.581 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:06.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:06.682 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:06.718 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:06.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:06.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:06.819 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:06.855 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:06.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:06.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:06.957 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:06.998 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:06.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:06.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:07.034 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-21T12:32:07.100 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:07.131 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:07.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:07.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:07.233 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:07.264 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:07.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:07.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:07.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:07 smithi114 ceph-mon[30324]: pgmap v596: 188 pgs: 3 remapped+peering, 5 peering, 3 active+remapped+backfill_toofull, 177 active+clean; 378 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 875 B/s rd, 0 op/s; 691/18705 objects misplaced (3.694%) 2024-09-21T12:32:07.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:07 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4124588655' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T12:32:07.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:07 smithi114 ceph-mon[30324]: osdmap e329: 8 total, 8 up, 7 in 2024-09-21T12:32:07.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:07 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4123197322' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:32:07.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:07 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:32:07.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:07 smithi114 ceph-mon[30324]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:32:07.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:07 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:32:07.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:07 smithi114 ceph-mon[30324]: osdmap e330: 8 total, 8 up, 7 in 2024-09-21T12:32:07.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:07 smithi114 ceph-mon[40867]: pgmap v596: 188 pgs: 3 remapped+peering, 5 peering, 3 active+remapped+backfill_toofull, 177 active+clean; 378 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 875 B/s rd, 0 op/s; 691/18705 objects misplaced (3.694%) 2024-09-21T12:32:07.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:07 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/4124588655' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T12:32:07.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:07 smithi114 ceph-mon[40867]: osdmap e329: 8 total, 8 up, 7 in 2024-09-21T12:32:07.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:07 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/4123197322' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:32:07.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:07 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:32:07.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:07 smithi114 ceph-mon[40867]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:32:07.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:07 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:32:07.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:07 smithi114 ceph-mon[40867]: osdmap e330: 8 total, 8 up, 7 in 2024-09-21T12:32:07.366 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:07.403 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:07.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:07.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:07.488 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:32:07 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:32:07.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:07 smithi183 ceph-mon[33277]: pgmap v596: 188 pgs: 3 remapped+peering, 5 peering, 3 active+remapped+backfill_toofull, 177 active+clean; 378 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 875 B/s rd, 0 op/s; 691/18705 objects misplaced (3.694%) 2024-09-21T12:32:07.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:07 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/4124588655' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T12:32:07.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:07 smithi183 ceph-mon[33277]: osdmap e329: 8 total, 8 up, 7 in 2024-09-21T12:32:07.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:07 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/4123197322' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:32:07.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:07 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:32:07.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:07 smithi183 ceph-mon[33277]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:32:07.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:07 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:32:07.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:07 smithi183 ceph-mon[33277]: osdmap e330: 8 total, 8 up, 7 in 2024-09-21T12:32:07.504 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T12:32:07.504 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:07.543 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:07.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:07.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:07.645 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:07.698 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:07.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:07.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:07.799 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:07.806 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:32:07.806 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:32:07.826 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:07.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:07.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:07.928 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:07.955 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:07.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:07.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:08.056 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:08.109 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:08.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:08.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:08.211 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:08.250 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:08.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:08.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:08.351 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:08.389 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:08.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:08.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:08.457 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Removed 124031 objects 2024-09-21T12:32:08.457 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Clean up completed and total clean up time :130.609 2024-09-21T12:32:08.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:08 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:08.492 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:08.512 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:08.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:08.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:08.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:08 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:08.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:08 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:08.613 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:08.636 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:08.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:08.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:08.738 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:08.759 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:08.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:08.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:08.765 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T12:32:08.861 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:08.882 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:08.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:08.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:08.959 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:32:08.959 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:32:08.977 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:32:08.978 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:32:08.984 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:08.998 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:32:08.999 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:32:09.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:09.006 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:09.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:09.015 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:32:09.015 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:32:09.032 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:32:09.033 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:32:09.050 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:32:09.050 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:32:09.067 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:32:09.067 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:32:09.083 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:32:09.084 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:32:09.107 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:09.129 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:09.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:09.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:09.230 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:09.256 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:09.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:09.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:09.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:09 smithi114 ceph-mon[30324]: pgmap v599: 188 pgs: 3 active+remapped+backfill_toofull, 185 active+clean; 156 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 742 op/s; 218/8055 objects misplaced (2.706%); 5.2 MiB/s, 86 objects/s recovering 2024-09-21T12:32:09.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:09 smithi114 ceph-mon[30324]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-21T12:32:09.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:09 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1104973605' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:32:09.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:09 smithi114 ceph-mon[40867]: pgmap v599: 188 pgs: 3 active+remapped+backfill_toofull, 185 active+clean; 156 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 742 op/s; 218/8055 objects misplaced (2.706%); 5.2 MiB/s, 86 objects/s recovering 2024-09-21T12:32:09.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:09 smithi114 ceph-mon[40867]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-21T12:32:09.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:09 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1104973605' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:32:09.357 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:09.384 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:09.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:09.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:09.485 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:09.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:09 smithi183 ceph-mon[33277]: pgmap v599: 188 pgs: 3 active+remapped+backfill_toofull, 185 active+clean; 156 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 742 op/s; 218/8055 objects misplaced (2.706%); 5.2 MiB/s, 86 objects/s recovering 2024-09-21T12:32:09.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:09 smithi183 ceph-mon[33277]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-21T12:32:09.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:09 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1104973605' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:32:09.507 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:09.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:09.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:09.608 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:09.634 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:09.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:09.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:09.736 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:09.757 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:09.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:09.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:09.859 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:09.880 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:09.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:09.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:09.982 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:10.008 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:10.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:10.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:10.110 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:10.135 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:10.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:10.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:10.236 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:10.257 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:10.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:10.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:10.359 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:10.383 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:10.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:10.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:10.449 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:10.487 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:10.510 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:10.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:10.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:10.612 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:10.633 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:10.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:10.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:10.735 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:10.756 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:10.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:10.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:10.858 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:10.878 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:10.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:10.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:10.962 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:32:10.962 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 20, 2024-09-21T12:32:10.962 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 600, 2024-09-21T12:32:10.962 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-21T12:32:10.962 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.962 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:179611 9.16 9:6ab28fd3:::benchmark_data_smithi114_133214_object55577:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-21T12:32:10.962 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:01.026416+0000", 2024-09-21T12:32:10.962 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 69.928608511999997, 2024-09-21T12:32:10.962 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0040347110000001, 2024-09-21T12:32:10.963 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:10.963 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:10.963 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:32:10.963 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:10.963 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:10.963 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:10.963 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 179611 2024-09-21T12:32:10.963 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.963 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:10.963 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.963 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:10.963 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.026416+0000", 2024-09-21T12:32:10.963 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:10.963 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.963 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.964 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:10.964 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.026416+0000", 2024-09-21T12:32:10.964 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:10.964 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.964 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.964 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:10.964 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.026417+0000", 2024-09-21T12:32:10.964 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2240000000000001e-06 2024-09-21T12:32:10.964 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.964 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.964 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:10.964 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.026423+0000", 2024-09-21T12:32:10.964 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0679999999999996e-06 2024-09-21T12:32:10.964 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.965 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.965 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:10.965 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.026436+0000", 2024-09-21T12:32:10.965 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.34e-05 2024-09-21T12:32:10.965 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.965 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.965 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:10.965 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.026443+0000", 2024-09-21T12:32:10.965 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5079999999999999e-06 2024-09-21T12:32:10.965 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.965 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.965 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:10.965 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.026505+0000", 2024-09-21T12:32:10.966 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1450000000000003e-05 2024-09-21T12:32:10.966 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.966 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.966 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:10.966 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.026632+0000", 2024-09-21T12:32:10.966 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012670800000000001 2024-09-21T12:32:10.966 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.966 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.966 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,5,6", 2024-09-21T12:32:10.966 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.026929+0000", 2024-09-21T12:32:10.966 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00029765500000000002 2024-09-21T12:32:10.966 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.967 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.967 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:10.967 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.034732+0000", 2024-09-21T12:32:10.967 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.007802425 2024-09-21T12:32:10.967 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.967 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.967 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:32:10.967 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.034741+0000", 2024-09-21T12:32:10.967 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1020000000000003e-06 2024-09-21T12:32:10.967 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.967 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.967 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:10.968 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.030372+0000", 2024-09-21T12:32:10.968 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99563096699999998 2024-09-21T12:32:10.968 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.968 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.968 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:10.968 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.030451+0000", 2024-09-21T12:32:10.968 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9203999999999996e-05 2024-09-21T12:32:10.968 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:10.968 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:10.968 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:10.968 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.969 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.969 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:179657 9.16 9:6bdfd1a2:::benchmark_data_smithi114_133214_object55623:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-21T12:32:10.969 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:01.064476+0000", 2024-09-21T12:32:10.969 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 69.890548288000005, 2024-09-21T12:32:10.969 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98292075899999998, 2024-09-21T12:32:10.969 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:10.969 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:10.969 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:32:10.969 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:10.969 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:10.969 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:10.969 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 179657 2024-09-21T12:32:10.969 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.969 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:10.970 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.970 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:10.970 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.064476+0000", 2024-09-21T12:32:10.970 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:10.970 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.970 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.970 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:10.970 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.064476+0000", 2024-09-21T12:32:10.970 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:10.970 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.970 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.970 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:10.971 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.064478+0000", 2024-09-21T12:32:10.971 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4079999999999999e-06 2024-09-21T12:32:10.971 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.971 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.971 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:10.971 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.064482+0000", 2024-09-21T12:32:10.971 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.2089999999999999e-06 2024-09-21T12:32:10.971 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.971 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.971 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:10.971 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.064491+0000", 2024-09-21T12:32:10.971 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1740000000000006e-06 2024-09-21T12:32:10.971 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.971 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.972 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:10.972 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.064497+0000", 2024-09-21T12:32:10.972 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7540000000000003e-06 2024-09-21T12:32:10.972 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.972 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.972 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:10.972 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.067484+0000", 2024-09-21T12:32:10.972 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0029873439999999999 2024-09-21T12:32:10.972 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.972 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.972 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:32:10.972 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.067493+0000", 2024-09-21T12:32:10.972 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.3890000000000005e-06 2024-09-21T12:32:10.972 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.973 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.973 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:10.973 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.031059+0000", 2024-09-21T12:32:10.973 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.96356651000000004 2024-09-21T12:32:10.973 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.973 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.973 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:10.973 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.031150+0000", 2024-09-21T12:32:10.973 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0994000000000003e-05 2024-09-21T12:32:10.973 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.974 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.974 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:32:10.974 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.031439+0000", 2024-09-21T12:32:10.974 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00028878699999999998 2024-09-21T12:32:10.974 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.974 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.974 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:10.974 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.034468+0000", 2024-09-21T12:32:10.974 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003029482 2024-09-21T12:32:10.974 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.974 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.974 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:32:10.974 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.041480+0000", 2024-09-21T12:32:10.975 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0070113199999999997 2024-09-21T12:32:10.975 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.975 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.975 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:10.975 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.047273+0000", 2024-09-21T12:32:10.975 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0057935139999999996 2024-09-21T12:32:10.975 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.975 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.975 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:32:10.975 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.047353+0000", 2024-09-21T12:32:10.975 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9925999999999993e-05 2024-09-21T12:32:10.975 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.975 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.975 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:10.975 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.047397+0000", 2024-09-21T12:32:10.976 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3948e-05 2024-09-21T12:32:10.976 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:10.976 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:10.976 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:10.976 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.976 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.976 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:179691 9.16 9:693ea84b:::benchmark_data_smithi114_133214_object55657:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-21T12:32:10.976 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:01.086828+0000", 2024-09-21T12:32:10.976 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 69.868196311000005, 2024-09-21T12:32:10.976 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.96072276199999995, 2024-09-21T12:32:10.976 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:10.976 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:10.976 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:32:10.976 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:10.977 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:10.977 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:10.977 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 179691 2024-09-21T12:32:10.977 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.977 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:10.977 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.977 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:10.977 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.086828+0000", 2024-09-21T12:32:10.977 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:10.977 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.977 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.977 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:10.977 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.086828+0000", 2024-09-21T12:32:10.977 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:10.977 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.978 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.978 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:10.978 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.086830+0000", 2024-09-21T12:32:10.978 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2270000000000001e-06 2024-09-21T12:32:10.978 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.978 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.978 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:10.978 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.086834+0000", 2024-09-21T12:32:10.978 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.9990000000000002e-06 2024-09-21T12:32:10.978 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.978 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.979 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:10.979 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.086843+0000", 2024-09-21T12:32:10.979 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9090000000000006e-06 2024-09-21T12:32:10.979 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.979 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.979 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:10.979 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.086849+0000", 2024-09-21T12:32:10.979 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7440000000000001e-06 2024-09-21T12:32:10.979 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.979 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.979 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:10.979 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.101158+0000", 2024-09-21T12:32:10.979 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.014308922999999999 2024-09-21T12:32:10.979 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.979 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.980 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:32:10.980 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.101176+0000", 2024-09-21T12:32:10.980 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7980999999999999e-05 2024-09-21T12:32:10.980 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.980 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.980 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:10.980 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.032538+0000", 2024-09-21T12:32:10.980 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.93136199600000003 2024-09-21T12:32:10.980 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.980 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.980 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:10.980 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.032634+0000", 2024-09-21T12:32:10.980 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.5873999999999997e-05 2024-09-21T12:32:10.980 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.981 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.981 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:32:10.981 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.032790+0000", 2024-09-21T12:32:10.981 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015569999999999999 2024-09-21T12:32:10.981 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.981 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.981 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:10.981 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.034503+0000", 2024-09-21T12:32:10.981 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0017132219999999999 2024-09-21T12:32:10.981 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.981 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.981 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:32:10.981 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.041484+0000", 2024-09-21T12:32:10.981 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0069813119999999999 2024-09-21T12:32:10.982 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.982 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.982 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:10.982 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.047470+0000", 2024-09-21T12:32:10.982 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0059858710000000002 2024-09-21T12:32:10.982 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.982 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.982 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:32:10.982 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.047517+0000", 2024-09-21T12:32:10.982 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6820000000000002e-05 2024-09-21T12:32:10.982 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.982 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.982 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:10.982 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.047551+0000", 2024-09-21T12:32:10.983 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4183999999999997e-05 2024-09-21T12:32:10.983 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:10.983 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:10.983 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:10.983 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.983 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.983 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:180848 9.12 9:4ecd9eaa:::benchmark_data_smithi114_133214_object56814:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-09-21T12:32:10.983 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:04.045860+0000", 2024-09-21T12:32:10.983 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 66.909164349999998, 2024-09-21T12:32:10.983 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99015821699999995, 2024-09-21T12:32:10.983 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:10.983 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:10.983 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:32:10.984 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:10.984 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:10.984 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:10.984 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 180848 2024-09-21T12:32:10.984 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.984 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:10.984 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.984 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:10.984 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.045860+0000", 2024-09-21T12:32:10.984 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:10.984 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.984 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.984 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:10.984 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.045860+0000", 2024-09-21T12:32:10.984 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:10.985 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.985 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.985 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:10.985 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.045862+0000", 2024-09-21T12:32:10.985 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.466e-06 2024-09-21T12:32:10.985 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.985 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.985 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:10.985 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.045866+0000", 2024-09-21T12:32:10.986 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.89e-06 2024-09-21T12:32:10.986 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.986 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.986 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:10.986 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.045887+0000", 2024-09-21T12:32:10.986 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0801e-05 2024-09-21T12:32:10.987 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.987 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.987 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:10.987 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.045894+0000", 2024-09-21T12:32:10.987 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2830000000000004e-06 2024-09-21T12:32:10.987 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.987 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.987 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:10.988 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.045948+0000", 2024-09-21T12:32:10.988 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3677999999999997e-05 2024-09-21T12:32:10.988 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.988 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.988 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:32:10.988 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.045956+0000", 2024-09-21T12:32:10.988 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1079999999999993e-06 2024-09-21T12:32:10.988 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.988 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.988 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:10.988 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:05.032414+0000", 2024-09-21T12:32:10.988 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98645813900000001 2024-09-21T12:32:10.988 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.988 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.989 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:10.989 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:05.032550+0000", 2024-09-21T12:32:10.989 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013655699999999999 2024-09-21T12:32:10.989 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.989 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.989 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,5", 2024-09-21T12:32:10.989 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:05.032837+0000", 2024-09-21T12:32:10.989 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00028715899999999999 2024-09-21T12:32:10.989 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.989 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.989 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:10.989 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:05.035846+0000", 2024-09-21T12:32:10.989 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0030088020000000001 2024-09-21T12:32:10.990 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.990 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.990 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:32:10.990 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:05.035860+0000", 2024-09-21T12:32:10.990 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3366e-05 2024-09-21T12:32:10.990 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.990 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.990 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:10.990 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:05.035895+0000", 2024-09-21T12:32:10.990 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5751999999999999e-05 2024-09-21T12:32:10.990 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.990 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.990 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:32:10.991 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:05.035973+0000", 2024-09-21T12:32:10.991 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7999999999999999e-05 2024-09-21T12:32:10.991 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.991 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.991 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:10.991 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:05.036019+0000", 2024-09-21T12:32:10.991 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5216000000000002e-05 2024-09-21T12:32:10.991 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:10.991 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:10.991 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:10.991 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.991 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.992 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:221947 9.0 9.c6972ae0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:32:10.992 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:38.070413+0000", 2024-09-21T12:32:10.992 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 32.884611251000003, 2024-09-21T12:32:10.992 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97957693599999995, 2024-09-21T12:32:10.992 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:10.992 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:10.992 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:32:10.992 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:10.992 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:10.992 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:10.992 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 221947 2024-09-21T12:32:10.992 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.993 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:10.993 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.993 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:10.993 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.070413+0000", 2024-09-21T12:32:10.993 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:10.993 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.993 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.993 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:10.993 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.070413+0000", 2024-09-21T12:32:10.993 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:10.993 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.993 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.993 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:10.993 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.070414+0000", 2024-09-21T12:32:10.994 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4000000000000002e-07 2024-09-21T12:32:10.994 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.994 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.994 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:10.994 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.070416+0000", 2024-09-21T12:32:10.994 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1730000000000002e-06 2024-09-21T12:32:10.994 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.994 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.994 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:10.994 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.070423+0000", 2024-09-21T12:32:10.994 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7320000000000001e-06 2024-09-21T12:32:10.995 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.995 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.995 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:10.995 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.070428+0000", 2024-09-21T12:32:10.995 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0100000000000003e-06 2024-09-21T12:32:10.995 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.995 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.995 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:10.995 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.071108+0000", 2024-09-21T12:32:10.995 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00067975200000000005 2024-09-21T12:32:10.995 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.995 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.995 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:32:10.995 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.071117+0000", 2024-09-21T12:32:10.996 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0880000000000001e-06 2024-09-21T12:32:10.996 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.996 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.996 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:10.996 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.049979+0000", 2024-09-21T12:32:10.996 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97886278999999998 2024-09-21T12:32:10.996 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.996 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.996 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:10.996 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.049990+0000", 2024-09-21T12:32:10.996 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0951000000000001e-05 2024-09-21T12:32:10.996 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:10.997 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:10.997 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:10.997 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.997 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.997 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:221951 9.0 9.29003820 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:32:10.997 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:38.078421+0000", 2024-09-21T12:32:10.997 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 32.876603836000001, 2024-09-21T12:32:10.997 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97160824800000001, 2024-09-21T12:32:10.997 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:10.997 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:10.997 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:32:10.997 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:10.997 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:10.997 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:10.998 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 221951 2024-09-21T12:32:10.998 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.998 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:10.998 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.998 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:10.998 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.078421+0000", 2024-09-21T12:32:10.998 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:10.998 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.998 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.998 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:10.998 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.078421+0000", 2024-09-21T12:32:10.998 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:10.999 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.999 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.999 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:10.999 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.078422+0000", 2024-09-21T12:32:10.999 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4300000000000001e-06 2024-09-21T12:32:10.999 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:10.999 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:10.999 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:10.999 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.078426+0000", 2024-09-21T12:32:10.999 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5140000000000002e-06 2024-09-21T12:32:10.999 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.000 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.000 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:11.000 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.078435+0000", 2024-09-21T12:32:11.000 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6570000000000003e-06 2024-09-21T12:32:11.000 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.000 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.000 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:11.000 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.078440+0000", 2024-09-21T12:32:11.000 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2660000000000001e-06 2024-09-21T12:32:11.000 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.000 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.000 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.000 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.078657+0000", 2024-09-21T12:32:11.000 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00021765200000000001 2024-09-21T12:32:11.001 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.001 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.001 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:32:11.001 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.078666+0000", 2024-09-21T12:32:11.001 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.2109999999999998e-06 2024-09-21T12:32:11.001 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.001 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.001 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.001 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.050016+0000", 2024-09-21T12:32:11.001 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97134992799999997 2024-09-21T12:32:11.001 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.001 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.001 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:11.001 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.050029+0000", 2024-09-21T12:32:11.001 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.359e-05 2024-09-21T12:32:11.002 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.002 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:11.002 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.002 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.002 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.002 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231723 9.14 9:2de8fd01:::benchmark_data_smithi114_133214_object107689:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T12:32:11.002 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:49.066280+0000", 2024-09-21T12:32:11.002 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 21.888745179000001, 2024-09-21T12:32:11.002 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1384219230000001, 2024-09-21T12:32:11.002 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:11.002 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:11.002 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:32:11.002 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:11.002 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:11.003 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:11.003 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231723 2024-09-21T12:32:11.003 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.003 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:11.003 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.003 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:11.003 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.066280+0000", 2024-09-21T12:32:11.003 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:11.003 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.003 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.003 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:11.003 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.066280+0000", 2024-09-21T12:32:11.003 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:11.003 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.004 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.004 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:11.004 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.066281+0000", 2024-09-21T12:32:11.004 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5939999999999999e-06 2024-09-21T12:32:11.004 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.004 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.004 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:11.004 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.066286+0000", 2024-09-21T12:32:11.004 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0189999999999997e-06 2024-09-21T12:32:11.004 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.004 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.004 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:11.004 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.066298+0000", 2024-09-21T12:32:11.004 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2024999999999999e-05 2024-09-21T12:32:11.005 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.005 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.005 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:11.005 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.066305+0000", 2024-09-21T12:32:11.005 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9870000000000002e-06 2024-09-21T12:32:11.005 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.005 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.005 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.005 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.066371+0000", 2024-09-21T12:32:11.005 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6025999999999994e-05 2024-09-21T12:32:11.005 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.005 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.005 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:32:11.005 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.066382+0000", 2024-09-21T12:32:11.006 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0311e-05 2024-09-21T12:32:11.006 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.006 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.006 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.006 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.038211+0000", 2024-09-21T12:32:11.006 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97182923300000001 2024-09-21T12:32:11.006 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.006 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.006 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:32:11.006 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.038222+0000", 2024-09-21T12:32:11.006 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1032e-05 2024-09-21T12:32:11.007 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:11.008 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.008 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.008 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.008 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.201658+0000", 2024-09-21T12:32:11.009 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1634362899999999 2024-09-21T12:32:11.009 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.009 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.009 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:11.009 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.201745+0000", 2024-09-21T12:32:11.009 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6829000000000005e-05 2024-09-21T12:32:11.009 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.009 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.009 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,3,5", 2024-09-21T12:32:11.009 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.201998+0000", 2024-09-21T12:32:11.009 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000253212 2024-09-21T12:32:11.009 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.009 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.009 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:32:11.010 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.203777+0000", 2024-09-21T12:32:11.010 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001778565 2024-09-21T12:32:11.010 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.010 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.010 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:11.010 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.204187+0000", 2024-09-21T12:32:11.010 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000409873 2024-09-21T12:32:11.010 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.010 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.010 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:11.010 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.204239+0000", 2024-09-21T12:32:11.010 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2166999999999997e-05 2024-09-21T12:32:11.010 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.011 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.011 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:11.011 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.204276+0000", 2024-09-21T12:32:11.011 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.7181999999999997e-05 2024-09-21T12:32:11.011 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.011 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.011 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:32:11.011 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.204485+0000", 2024-09-21T12:32:11.011 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00020932399999999999 2024-09-21T12:32:11.011 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.011 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.011 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:11.011 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.204701+0000", 2024-09-21T12:32:11.012 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00021625400000000001 2024-09-21T12:32:11.012 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.012 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:11.012 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.012 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.012 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.012 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231668 9.7 9:e40b995a:::benchmark_data_smithi114_133214_object107634:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:32:11.012 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029737+0000", 2024-09-21T12:32:11.012 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 20.925287829999998, 2024-09-21T12:32:11.012 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1844050580000001, 2024-09-21T12:32:11.012 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:11.012 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:11.012 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:32:11.012 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:11.013 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:11.013 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:11.013 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231668 2024-09-21T12:32:11.013 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.013 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:11.013 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.013 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:11.013 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029737+0000", 2024-09-21T12:32:11.013 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:11.013 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.013 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.013 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:11.013 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029737+0000", 2024-09-21T12:32:11.013 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:11.013 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.013 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.014 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:11.014 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029739+0000", 2024-09-21T12:32:11.014 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9300000000000002e-06 2024-09-21T12:32:11.014 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.014 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.014 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:11.014 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029744+0000", 2024-09-21T12:32:11.014 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6970000000000001e-06 2024-09-21T12:32:11.014 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.014 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.014 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:11.014 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029764+0000", 2024-09-21T12:32:11.014 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0959e-05 2024-09-21T12:32:11.014 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.015 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.015 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:11.015 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029773+0000", 2024-09-21T12:32:11.015 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9390000000000003e-06 2024-09-21T12:32:11.015 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.015 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.015 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.015 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.030032+0000", 2024-09-21T12:32:11.015 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00025845499999999999 2024-09-21T12:32:11.015 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.015 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.015 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op must wait for map", 2024-09-21T12:32:11.015 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.030039+0000", 2024-09-21T12:32:11.015 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5310000000000004e-06 2024-09-21T12:32:11.015 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.016 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.016 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.016 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033252+0000", 2024-09-21T12:32:11.016 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0032126519999999999 2024-09-21T12:32:11.016 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.016 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.016 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:32:11.016 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033262+0000", 2024-09-21T12:32:11.016 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6039999999999999e-06 2024-09-21T12:32:11.016 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.016 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.016 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.016 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.214079+0000", 2024-09-21T12:32:11.016 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1808168320000001 2024-09-21T12:32:11.016 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.017 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.017 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:11.017 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.214142+0000", 2024-09-21T12:32:11.017 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3459000000000003e-05 2024-09-21T12:32:11.017 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.017 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:11.017 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.017 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.017 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.017 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231684 9.2 9:431c20d3:::benchmark_data_smithi114_133214_object107650:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:32:11.017 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029798+0000", 2024-09-21T12:32:11.017 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 20.925226967, 2024-09-21T12:32:11.017 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.174456956, 2024-09-21T12:32:11.017 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:11.018 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:11.018 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:32:11.018 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:11.018 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:11.018 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:11.018 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231684 2024-09-21T12:32:11.018 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.018 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:11.018 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.018 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:11.018 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029798+0000", 2024-09-21T12:32:11.018 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:11.018 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.018 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.019 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:11.019 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029798+0000", 2024-09-21T12:32:11.019 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:11.019 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.019 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.019 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:11.019 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029798+0000", 2024-09-21T12:32:11.019 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4600000000000005e-07 2024-09-21T12:32:11.019 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.019 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.019 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:11.019 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029802+0000", 2024-09-21T12:32:11.019 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.399e-06 2024-09-21T12:32:11.019 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.020 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.020 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:11.020 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029813+0000", 2024-09-21T12:32:11.020 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0927999999999999e-05 2024-09-21T12:32:11.020 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.020 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.020 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:11.020 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029819+0000", 2024-09-21T12:32:11.020 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3840000000000002e-06 2024-09-21T12:32:11.020 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.020 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.020 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.020 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029883+0000", 2024-09-21T12:32:11.020 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3903000000000001e-05 2024-09-21T12:32:11.021 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.021 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.021 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op must wait for map", 2024-09-21T12:32:11.021 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029891+0000", 2024-09-21T12:32:11.021 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1929999999999993e-06 2024-09-21T12:32:11.021 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.021 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.021 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.021 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033472+0000", 2024-09-21T12:32:11.021 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003581009 2024-09-21T12:32:11.021 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.021 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.021 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:32:11.021 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033481+0000", 2024-09-21T12:32:11.022 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5320000000000006e-06 2024-09-21T12:32:11.022 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.022 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.022 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.022 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.204173+0000", 2024-09-21T12:32:11.022 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.170692377 2024-09-21T12:32:11.022 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.022 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.022 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:11.022 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.204255+0000", 2024-09-21T12:32:11.022 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1685000000000001e-05 2024-09-21T12:32:11.022 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.022 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:11.023 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.023 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.023 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.023 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231688 9.7 9:e423c016:::benchmark_data_smithi114_133214_object107654:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:32:11.023 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029833+0000", 2024-09-21T12:32:11.023 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 20.925191577, 2024-09-21T12:32:11.023 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.184627359, 2024-09-21T12:32:11.023 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:11.023 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:11.023 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:32:11.023 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:11.023 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:11.023 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:11.023 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231688 2024-09-21T12:32:11.024 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.024 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:11.024 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.024 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:11.024 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029833+0000", 2024-09-21T12:32:11.024 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:11.024 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.024 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.024 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:11.024 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029833+0000", 2024-09-21T12:32:11.024 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:11.024 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.024 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.024 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:11.024 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029834+0000", 2024-09-21T12:32:11.025 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6600000000000002e-07 2024-09-21T12:32:11.025 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.025 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.025 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:11.025 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029836+0000", 2024-09-21T12:32:11.025 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.6450000000000001e-06 2024-09-21T12:32:11.025 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.025 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.025 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:11.025 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029843+0000", 2024-09-21T12:32:11.025 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6440000000000003e-06 2024-09-21T12:32:11.025 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.025 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.025 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:11.026 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029848+0000", 2024-09-21T12:32:11.026 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0549999999999999e-06 2024-09-21T12:32:11.026 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.026 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.026 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.026 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033286+0000", 2024-09-21T12:32:11.026 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0034382089999999998 2024-09-21T12:32:11.026 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.026 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.026 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:32:11.026 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033294+0000", 2024-09-21T12:32:11.026 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9279999999999993e-06 2024-09-21T12:32:11.026 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.214400+0000", 2024-09-21T12:32:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.181105869 2024-09-21T12:32:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.214461+0000", 2024-09-21T12:32:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0543e-05 2024-09-21T12:32:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.028 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231694 9.7 9:e4532815:::benchmark_data_smithi114_133214_object107660:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:32:11.028 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029855+0000", 2024-09-21T12:32:11.028 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 20.925169652000001, 2024-09-21T12:32:11.028 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.212233884, 2024-09-21T12:32:11.028 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:11.028 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:11.028 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:32:11.028 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:11.028 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:11.028 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:11.028 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231694 2024-09-21T12:32:11.028 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.028 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:11.028 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.029 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:11.029 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029855+0000", 2024-09-21T12:32:11.029 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:11.029 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.029 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.029 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:11.029 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029855+0000", 2024-09-21T12:32:11.029 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:11.029 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.029 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.029 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:11.029 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029856+0000", 2024-09-21T12:32:11.029 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4499999999999997e-07 2024-09-21T12:32:11.030 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.030 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.030 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:11.030 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029858+0000", 2024-09-21T12:32:11.030 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.131e-06 2024-09-21T12:32:11.030 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.030 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.030 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:11.030 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029864+0000", 2024-09-21T12:32:11.031 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1410000000000002e-06 2024-09-21T12:32:11.031 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.031 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.031 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:11.031 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029869+0000", 2024-09-21T12:32:11.031 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7829999999999998e-06 2024-09-21T12:32:11.031 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.031 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.031 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.031 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033319+0000", 2024-09-21T12:32:11.031 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003450304 2024-09-21T12:32:11.031 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.031 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.031 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:32:11.032 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033326+0000", 2024-09-21T12:32:11.032 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3590000000000002e-06 2024-09-21T12:32:11.032 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.032 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.032 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.032 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.214613+0000", 2024-09-21T12:32:11.032 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1812871789999999 2024-09-21T12:32:11.032 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.032 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.032 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:11.032 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.214822+0000", 2024-09-21T12:32:11.032 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00020822599999999999 2024-09-21T12:32:11.032 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.032 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.032 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,3,5", 2024-09-21T12:32:11.033 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.215218+0000", 2024-09-21T12:32:11.033 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000396728 2024-09-21T12:32:11.033 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.033 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.033 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:32:11.033 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.229559+0000", 2024-09-21T12:32:11.033 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.014340258 2024-09-21T12:32:11.033 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.033 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.033 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:11.033 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.232468+0000", 2024-09-21T12:32:11.033 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0029092469999999998 2024-09-21T12:32:11.033 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.033 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.033 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:11.034 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.232552+0000", 2024-09-21T12:32:11.034 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.3855000000000003e-05 2024-09-21T12:32:11.034 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.034 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.034 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:11.034 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.241712+0000", 2024-09-21T12:32:11.034 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0091604219999999997 2024-09-21T12:32:11.034 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.034 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.034 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:32:11.034 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.241968+0000", 2024-09-21T12:32:11.034 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00025556599999999999 2024-09-21T12:32:11.034 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.034 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.035 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:11.035 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.242089+0000", 2024-09-21T12:32:11.035 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012124 2024-09-21T12:32:11.035 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.035 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:11.035 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.035 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.035 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.035 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231696 9.2 9:4615e034:::benchmark_data_smithi114_133214_object107662:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:32:11.035 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029874+0000", 2024-09-21T12:32:11.035 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 20.925150673000001, 2024-09-21T12:32:11.035 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.181378748, 2024-09-21T12:32:11.035 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:11.035 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:11.036 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:32:11.036 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:11.036 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:11.036 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:11.036 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231696 2024-09-21T12:32:11.036 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.036 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:11.036 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.036 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:11.036 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029874+0000", 2024-09-21T12:32:11.036 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:11.036 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.036 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.036 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:11.037 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029874+0000", 2024-09-21T12:32:11.037 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:11.037 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.037 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.037 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:11.037 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029874+0000", 2024-09-21T12:32:11.037 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5600000000000001e-07 2024-09-21T12:32:11.037 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.037 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.037 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:11.037 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029876+0000", 2024-09-21T12:32:11.037 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.99e-06 2024-09-21T12:32:11.037 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.037 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.038 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:11.038 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029882+0000", 2024-09-21T12:32:11.038 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4750000000000001e-06 2024-09-21T12:32:11.038 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.038 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.038 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:11.038 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029887+0000", 2024-09-21T12:32:11.038 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2839999999999997e-06 2024-09-21T12:32:11.038 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.038 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.038 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.038 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033506+0000", 2024-09-21T12:32:11.039 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00361875 2024-09-21T12:32:11.039 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.039 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.039 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:32:11.039 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033514+0000", 2024-09-21T12:32:11.039 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6850000000000001e-06 2024-09-21T12:32:11.039 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.039 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.039 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.039 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.204539+0000", 2024-09-21T12:32:11.039 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1710255940000001 2024-09-21T12:32:11.039 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.039 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.039 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:11.039 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.204647+0000", 2024-09-21T12:32:11.040 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000107538 2024-09-21T12:32:11.040 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.040 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.040 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,3,7", 2024-09-21T12:32:11.040 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.204970+0000", 2024-09-21T12:32:11.040 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00032291999999999998 2024-09-21T12:32:11.040 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.040 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.040 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:32:11.040 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.207582+0000", 2024-09-21T12:32:11.040 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0026120230000000002 2024-09-21T12:32:11.040 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.040 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.040 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:11.040 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.210625+0000", 2024-09-21T12:32:11.041 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0030428959999999998 2024-09-21T12:32:11.041 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.041 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.041 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:11.041 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.210912+0000", 2024-09-21T12:32:11.041 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00028736700000000002 2024-09-21T12:32:11.041 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.041 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.041 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:11.041 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.211091+0000", 2024-09-21T12:32:11.041 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017910199999999999 2024-09-21T12:32:11.041 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.041 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.041 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:32:11.041 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.211150+0000", 2024-09-21T12:32:11.041 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8934999999999997e-05 2024-09-21T12:32:11.042 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.042 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.042 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:11.042 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.211253+0000", 2024-09-21T12:32:11.042 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000102833 2024-09-21T12:32:11.042 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.042 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:11.042 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.042 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.042 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.042 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231712 9.2 9:41d44c3f:::benchmark_data_smithi114_133214_object107678:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:32:11.042 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029893+0000", 2024-09-21T12:32:11.042 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 20.925131689000001, 2024-09-21T12:32:11.042 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.194669177, 2024-09-21T12:32:11.042 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:11.043 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:11.043 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:32:11.043 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:11.043 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:11.043 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:11.043 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231712 2024-09-21T12:32:11.043 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.043 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:11.043 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.043 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:11.043 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029893+0000", 2024-09-21T12:32:11.043 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:11.043 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.043 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.044 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:11.044 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029893+0000", 2024-09-21T12:32:11.044 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:11.044 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.044 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.044 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:11.044 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029893+0000", 2024-09-21T12:32:11.044 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5499999999999999e-07 2024-09-21T12:32:11.044 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.044 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.044 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:11.044 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029895+0000", 2024-09-21T12:32:11.044 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7859999999999999e-06 2024-09-21T12:32:11.044 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.044 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.044 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:11.045 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029901+0000", 2024-09-21T12:32:11.045 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7239999999999997e-06 2024-09-21T12:32:11.045 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.045 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.045 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:11.045 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029905+0000", 2024-09-21T12:32:11.045 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5210000000000004e-06 2024-09-21T12:32:11.045 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.045 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.045 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.045 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033538+0000", 2024-09-21T12:32:11.045 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0036328900000000002 2024-09-21T12:32:11.045 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.045 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.045 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:32:11.046 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033546+0000", 2024-09-21T12:32:11.046 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9170000000000006e-06 2024-09-21T12:32:11.046 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.046 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.046 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.046 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.207834+0000", 2024-09-21T12:32:11.046 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.17428818 2024-09-21T12:32:11.046 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.046 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.046 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:11.046 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.207946+0000", 2024-09-21T12:32:11.046 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000111501 2024-09-21T12:32:11.047 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.047 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.047 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,3,7", 2024-09-21T12:32:11.047 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.209056+0000", 2024-09-21T12:32:11.047 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0011098670000000001 2024-09-21T12:32:11.047 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.047 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.047 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:11.047 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.212301+0000", 2024-09-21T12:32:11.047 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0032456199999999998 2024-09-21T12:32:11.047 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.047 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.047 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:32:11.048 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.212318+0000", 2024-09-21T12:32:11.048 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7031999999999999e-05 2024-09-21T12:32:11.048 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.048 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.048 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:11.048 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.214089+0000", 2024-09-21T12:32:11.048 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001770726 2024-09-21T12:32:11.048 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.048 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.048 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:11.048 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.224405+0000", 2024-09-21T12:32:11.048 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010315658 2024-09-21T12:32:11.048 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.049 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.049 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:32:11.049 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.224461+0000", 2024-09-21T12:32:11.049 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6341999999999997e-05 2024-09-21T12:32:11.049 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.049 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.049 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:11.049 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.224562+0000", 2024-09-21T12:32:11.049 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010105799999999999 2024-09-21T12:32:11.049 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.049 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:11.049 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.049 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.049 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.050 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231746 9.7 9:e6f0ab86:::benchmark_data_smithi114_133214_object107712:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:32:11.050 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029911+0000", 2024-09-21T12:32:11.050 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 20.925113694, 2024-09-21T12:32:11.050 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2150179190000001, 2024-09-21T12:32:11.050 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:11.050 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:11.050 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:32:11.050 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:11.050 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:11.050 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:11.050 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231746 2024-09-21T12:32:11.050 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.050 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:11.051 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.051 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:11.051 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029911+0000", 2024-09-21T12:32:11.051 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:11.051 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.051 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.051 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:11.051 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029911+0000", 2024-09-21T12:32:11.051 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:11.051 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.051 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.051 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:11.051 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029911+0000", 2024-09-21T12:32:11.052 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4700000000000002e-07 2024-09-21T12:32:11.052 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.052 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.052 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:11.052 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029913+0000", 2024-09-21T12:32:11.052 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.889e-06 2024-09-21T12:32:11.052 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.052 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.052 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:11.052 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029919+0000", 2024-09-21T12:32:11.052 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3630000000000002e-06 2024-09-21T12:32:11.052 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.052 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.053 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:11.053 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029923+0000", 2024-09-21T12:32:11.053 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4379999999999997e-06 2024-09-21T12:32:11.053 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.053 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.053 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.053 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033349+0000", 2024-09-21T12:32:11.053 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0034264059999999999 2024-09-21T12:32:11.053 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.053 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.053 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:32:11.053 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033357+0000", 2024-09-21T12:32:11.053 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4590000000000001e-06 2024-09-21T12:32:11.053 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.054 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.054 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.054 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.225869+0000", 2024-09-21T12:32:11.054 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.192512056 2024-09-21T12:32:11.054 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.054 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.054 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:11.054 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.225992+0000", 2024-09-21T12:32:11.054 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012313800000000001 2024-09-21T12:32:11.054 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.054 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.054 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,3,5", 2024-09-21T12:32:11.054 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.226276+0000", 2024-09-21T12:32:11.055 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00028393899999999998 2024-09-21T12:32:11.055 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.055 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.055 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:11.055 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.232512+0000", 2024-09-21T12:32:11.055 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0062360330000000002 2024-09-21T12:32:11.055 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.055 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.055 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:11.055 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.232592+0000", 2024-09-21T12:32:11.055 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9430000000000004e-05 2024-09-21T12:32:11.055 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.055 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.056 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:32:11.056 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.241522+0000", 2024-09-21T12:32:11.056 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0089301380000000007 2024-09-21T12:32:11.056 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.056 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.056 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:11.056 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.244727+0000", 2024-09-21T12:32:11.056 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0032050389999999998 2024-09-21T12:32:11.056 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.056 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.056 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:32:11.056 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.244780+0000", 2024-09-21T12:32:11.056 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3777999999999999e-05 2024-09-21T12:32:11.057 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.057 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.057 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:11.057 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.244929+0000", 2024-09-21T12:32:11.057 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014846600000000001 2024-09-21T12:32:11.057 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.057 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:11.057 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.057 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.057 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.057 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231751 9.7 9:e24ce5a0:::benchmark_data_smithi114_133214_object107717:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:32:11.057 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029929+0000", 2024-09-21T12:32:11.057 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 20.925096078999999, 2024-09-21T12:32:11.057 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2152077539999999, 2024-09-21T12:32:11.057 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:11.058 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:11.058 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:32:11.058 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:11.058 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:11.058 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:11.058 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231751 2024-09-21T12:32:11.058 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.058 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:11.058 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.058 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:11.058 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029929+0000", 2024-09-21T12:32:11.058 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:11.058 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.058 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.059 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:11.059 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029929+0000", 2024-09-21T12:32:11.059 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:11.059 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.059 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.059 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:11.059 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029929+0000", 2024-09-21T12:32:11.059 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5199999999999998e-07 2024-09-21T12:32:11.059 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.059 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.059 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:11.059 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029931+0000", 2024-09-21T12:32:11.059 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0090000000000002e-06 2024-09-21T12:32:11.060 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.060 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.060 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:11.060 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029936+0000", 2024-09-21T12:32:11.060 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.215e-06 2024-09-21T12:32:11.060 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.060 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.060 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:11.060 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029940+0000", 2024-09-21T12:32:11.060 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.2570000000000001e-06 2024-09-21T12:32:11.060 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.060 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.060 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.060 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033382+0000", 2024-09-21T12:32:11.061 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0034412370000000002 2024-09-21T12:32:11.061 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.061 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.061 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:32:11.061 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033388+0000", 2024-09-21T12:32:11.061 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6780000000000003e-06 2024-09-21T12:32:11.061 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.061 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.061 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.061 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.233447+0000", 2024-09-21T12:32:11.061 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.200058297 2024-09-21T12:32:11.061 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.061 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.061 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:11.061 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.233585+0000", 2024-09-21T12:32:11.062 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000138429 2024-09-21T12:32:11.062 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.062 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.062 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,3,5", 2024-09-21T12:32:11.062 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.233890+0000", 2024-09-21T12:32:11.062 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00030514399999999998 2024-09-21T12:32:11.062 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.062 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.062 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:11.062 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.237017+0000", 2024-09-21T12:32:11.062 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003126789 2024-09-21T12:32:11.062 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.062 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.062 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:11.063 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.237064+0000", 2024-09-21T12:32:11.063 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7287000000000003e-05 2024-09-21T12:32:11.063 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.063 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.063 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:32:11.063 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.242303+0000", 2024-09-21T12:32:11.063 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.005238599 2024-09-21T12:32:11.063 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.063 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.063 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:11.063 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.244979+0000", 2024-09-21T12:32:11.063 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0026761549999999999 2024-09-21T12:32:11.063 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.063 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.064 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:32:11.064 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.245035+0000", 2024-09-21T12:32:11.064 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6159000000000002e-05 2024-09-21T12:32:11.064 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.064 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.064 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:11.064 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.245136+0000", 2024-09-21T12:32:11.064 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000101147 2024-09-21T12:32:11.064 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.064 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:11.064 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.064 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.064 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.065 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231758 9.7 9:e06abb50:::benchmark_data_smithi114_133214_object107724:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:32:11.065 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029948+0000", 2024-09-21T12:32:11.065 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 20.925077013999999, 2024-09-21T12:32:11.065 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2159567739999999, 2024-09-21T12:32:11.065 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:11.065 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:11.065 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:32:11.065 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:11.065 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:11.065 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:11.065 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231758 2024-09-21T12:32:11.065 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.065 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:11.066 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.066 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:11.066 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029948+0000", 2024-09-21T12:32:11.066 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:11.066 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.066 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.066 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:11.066 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029948+0000", 2024-09-21T12:32:11.066 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:11.066 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.066 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.066 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:11.066 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029948+0000", 2024-09-21T12:32:11.066 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.41e-07 2024-09-21T12:32:11.067 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.067 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.067 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:11.067 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029950+0000", 2024-09-21T12:32:11.067 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0470000000000001e-06 2024-09-21T12:32:11.067 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.067 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.067 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:11.067 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029956+0000", 2024-09-21T12:32:11.067 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7069999999999997e-06 2024-09-21T12:32:11.067 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.067 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.067 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:11.068 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029961+0000", 2024-09-21T12:32:11.068 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1320000000000002e-06 2024-09-21T12:32:11.068 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.068 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.068 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.068 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033413+0000", 2024-09-21T12:32:11.068 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0034517609999999998 2024-09-21T12:32:11.068 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.068 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.068 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:32:11.068 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033420+0000", 2024-09-21T12:32:11.068 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6240000000000002e-06 2024-09-21T12:32:11.068 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.069 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.069 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.069 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.235474+0000", 2024-09-21T12:32:11.069 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.202053832 2024-09-21T12:32:11.069 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.069 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.069 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:11.069 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.235573+0000", 2024-09-21T12:32:11.069 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.9216999999999996e-05 2024-09-21T12:32:11.069 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.069 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.069 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,3,5", 2024-09-21T12:32:11.069 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.235777+0000", 2024-09-21T12:32:11.069 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000203687 2024-09-21T12:32:11.070 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.070 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.070 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:11.070 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.239048+0000", 2024-09-21T12:32:11.070 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003271173 2024-09-21T12:32:11.070 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.070 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.070 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:11.070 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.241655+0000", 2024-09-21T12:32:11.070 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002607023 2024-09-21T12:32:11.070 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.070 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.071 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:32:11.071 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.242308+0000", 2024-09-21T12:32:11.071 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00065309099999999996 2024-09-21T12:32:11.071 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.071 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.071 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:11.071 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.245732+0000", 2024-09-21T12:32:11.071 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003423461 2024-09-21T12:32:11.071 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.071 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.071 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:32:11.071 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.245786+0000", 2024-09-21T12:32:11.071 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4289999999999997e-05 2024-09-21T12:32:11.072 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.072 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.072 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:11.072 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.245904+0000", 2024-09-21T12:32:11.072 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011838800000000001 2024-09-21T12:32:11.072 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.072 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:11.072 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.072 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.072 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.072 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231761 9.7 9:e362f0a9:::benchmark_data_smithi114_133214_object107727:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:32:11.072 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029967+0000", 2024-09-21T12:32:11.072 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 20.925058216, 2024-09-21T12:32:11.072 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.217986754, 2024-09-21T12:32:11.073 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:11.073 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:11.073 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:32:11.073 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:11.073 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:11.073 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:11.073 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231761 2024-09-21T12:32:11.073 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.073 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:11.073 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.073 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:11.073 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029967+0000", 2024-09-21T12:32:11.073 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:11.074 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.074 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.074 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:11.074 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029967+0000", 2024-09-21T12:32:11.074 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:11.074 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.074 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.074 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:11.074 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029967+0000", 2024-09-21T12:32:11.074 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4900000000000001e-07 2024-09-21T12:32:11.074 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.074 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.074 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:11.074 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029969+0000", 2024-09-21T12:32:11.075 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.68e-06 2024-09-21T12:32:11.075 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.075 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.075 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:11.075 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029974+0000", 2024-09-21T12:32:11.075 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5899999999999998e-06 2024-09-21T12:32:11.075 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.075 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.075 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:11.075 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029979+0000", 2024-09-21T12:32:11.075 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6530000000000002e-06 2024-09-21T12:32:11.075 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.075 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.075 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.076 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033442+0000", 2024-09-21T12:32:11.076 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0034634930000000002 2024-09-21T12:32:11.076 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.076 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.076 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:32:11.076 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033449+0000", 2024-09-21T12:32:11.076 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9290000000000001e-06 2024-09-21T12:32:11.076 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.076 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.076 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.076 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.237181+0000", 2024-09-21T12:32:11.076 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2037321190000001 2024-09-21T12:32:11.076 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.077 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.077 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:11.077 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.237313+0000", 2024-09-21T12:32:11.077 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013190700000000001 2024-09-21T12:32:11.077 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.077 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.077 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,3,5", 2024-09-21T12:32:11.077 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.237544+0000", 2024-09-21T12:32:11.077 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00023036199999999999 2024-09-21T12:32:11.077 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.077 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.077 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:11.077 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.241579+0000", 2024-09-21T12:32:11.078 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.004035294 2024-09-21T12:32:11.078 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.078 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.078 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:32:11.078 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.242313+0000", 2024-09-21T12:32:11.078 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00073439600000000001 2024-09-21T12:32:11.078 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.078 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.078 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:11.078 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.245175+0000", 2024-09-21T12:32:11.078 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0028617299999999998 2024-09-21T12:32:11.078 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.078 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.079 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:11.079 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.247800+0000", 2024-09-21T12:32:11.079 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002624548 2024-09-21T12:32:11.079 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.079 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.079 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:32:11.079 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.247850+0000", 2024-09-21T12:32:11.079 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0158999999999999e-05 2024-09-21T12:32:11.079 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.079 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.079 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:11.079 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.247953+0000", 2024-09-21T12:32:11.079 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000103545 2024-09-21T12:32:11.080 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.080 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:11.080 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.080 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.080 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.080 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231982 9.14 9:2d540493:::benchmark_data_smithi114_133214_object107948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-09-21T12:32:11.080 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:52.048929+0000", 2024-09-21T12:32:11.080 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 18.906096087999998, 2024-09-21T12:32:11.080 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99998082899999996, 2024-09-21T12:32:11.080 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:11.080 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:11.080 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:32:11.080 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:11.081 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:11.081 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:11.081 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231982 2024-09-21T12:32:11.081 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.081 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:11.081 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.081 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:11.081 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.048929+0000", 2024-09-21T12:32:11.081 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:11.081 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.081 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.081 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:11.081 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.048929+0000", 2024-09-21T12:32:11.082 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:11.082 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.082 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.082 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:11.082 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.048931+0000", 2024-09-21T12:32:11.082 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.6589999999999999e-06 2024-09-21T12:32:11.082 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.082 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.082 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:11.082 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.048936+0000", 2024-09-21T12:32:11.082 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0869999999999997e-06 2024-09-21T12:32:11.082 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.082 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.082 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:11.083 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.048950+0000", 2024-09-21T12:32:11.083 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3976e-05 2024-09-21T12:32:11.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.083 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.083 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:11.083 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.048962+0000", 2024-09-21T12:32:11.083 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1225e-05 2024-09-21T12:32:11.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.083 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.083 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.083 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.049061+0000", 2024-09-21T12:32:11.083 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.9086999999999995e-05 2024-09-21T12:32:11.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.084 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.084 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:32:11.084 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.049068+0000", 2024-09-21T12:32:11.084 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6539999999999999e-06 2024-09-21T12:32:11.084 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.084 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.084 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.084 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.045781+0000", 2024-09-21T12:32:11.084 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99671266999999997 2024-09-21T12:32:11.084 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.084 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.084 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:11.084 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.045859+0000", 2024-09-21T12:32:11.084 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7891000000000004e-05 2024-09-21T12:32:11.085 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.085 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.085 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,5", 2024-09-21T12:32:11.085 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.046118+0000", 2024-09-21T12:32:11.085 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00025944599999999998 2024-09-21T12:32:11.085 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.085 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.085 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:11.085 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.048757+0000", 2024-09-21T12:32:11.085 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0026386069999999998 2024-09-21T12:32:11.085 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.085 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.085 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:32:11.086 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.048771+0000", 2024-09-21T12:32:11.086 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4440999999999999e-05 2024-09-21T12:32:11.086 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.086 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.086 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:11.086 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.048824+0000", 2024-09-21T12:32:11.086 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2460999999999998e-05 2024-09-21T12:32:11.086 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.086 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.086 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:32:11.086 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.048880+0000", 2024-09-21T12:32:11.086 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6067999999999999e-05 2024-09-21T12:32:11.086 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.086 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.086 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:11.087 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.048909+0000", 2024-09-21T12:32:11.087 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.9556999999999998e-05 2024-09-21T12:32:11.087 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.087 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:11.087 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.087 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.087 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.087 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231987 9.14 9:2dd07ba6:::benchmark_data_smithi114_133214_object107953:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-09-21T12:32:11.087 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:52.052061+0000", 2024-09-21T12:32:11.087 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 18.902964105999999, 2024-09-21T12:32:11.087 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99820888500000005, 2024-09-21T12:32:11.087 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:11.087 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:11.087 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:32:11.088 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:11.088 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:11.088 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:11.088 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231987 2024-09-21T12:32:11.088 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.088 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:11.088 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.088 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:11.088 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.052061+0000", 2024-09-21T12:32:11.088 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:11.088 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.088 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.088 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:11.088 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.052061+0000", 2024-09-21T12:32:11.088 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:11.089 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.089 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.089 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:11.089 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.052062+0000", 2024-09-21T12:32:11.089 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.026e-06 2024-09-21T12:32:11.089 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.089 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.089 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:11.089 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.052479+0000", 2024-09-21T12:32:11.089 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00041693600000000002 2024-09-21T12:32:11.089 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.089 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.089 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:11.089 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.052490+0000", 2024-09-21T12:32:11.089 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1270000000000001e-05 2024-09-21T12:32:11.089 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.090 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.090 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:11.090 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.052496+0000", 2024-09-21T12:32:11.090 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4330000000000001e-06 2024-09-21T12:32:11.090 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.090 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.090 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.090 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.052654+0000", 2024-09-21T12:32:11.090 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015731400000000001 2024-09-21T12:32:11.090 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.090 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.090 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:32:11.090 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.052663+0000", 2024-09-21T12:32:11.091 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9760000000000001e-06 2024-09-21T12:32:11.091 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.091 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.091 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.091 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.047676+0000", 2024-09-21T12:32:11.091 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99501326700000003 2024-09-21T12:32:11.091 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.091 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.091 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:11.091 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.047793+0000", 2024-09-21T12:32:11.091 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000117099 2024-09-21T12:32:11.091 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.091 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.091 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,5", 2024-09-21T12:32:11.092 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.047970+0000", 2024-09-21T12:32:11.092 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017714599999999999 2024-09-21T12:32:11.092 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.092 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.092 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:11.092 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.050154+0000", 2024-09-21T12:32:11.092 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0021837110000000001 2024-09-21T12:32:11.092 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.092 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.092 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:32:11.092 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.050164+0000", 2024-09-21T12:32:11.092 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0156000000000001e-05 2024-09-21T12:32:11.092 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.092 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.092 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:11.092 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.050195+0000", 2024-09-21T12:32:11.093 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.0966999999999997e-05 2024-09-21T12:32:11.093 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.093 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.093 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:32:11.093 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.050237+0000", 2024-09-21T12:32:11.093 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1909999999999997e-05 2024-09-21T12:32:11.093 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.093 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.093 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:11.093 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.050270+0000", 2024-09-21T12:32:11.093 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2673999999999999e-05 2024-09-21T12:32:11.093 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.093 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:11.093 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.093 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.093 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.094 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:232023 9.14 9:2a2d427f:::benchmark_data_smithi114_133214_object107989:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-09-21T12:32:11.094 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:52.084360+0000", 2024-09-21T12:32:11.094 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 18.870664747999999, 2024-09-21T12:32:11.094 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.96746416199999996, 2024-09-21T12:32:11.094 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:11.094 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:11.094 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:32:11.094 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:11.094 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:11.094 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:11.094 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 232023 2024-09-21T12:32:11.094 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.094 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:11.094 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.094 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:11.095 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.084360+0000", 2024-09-21T12:32:11.095 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:11.095 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.095 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.095 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:11.095 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.084360+0000", 2024-09-21T12:32:11.095 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:11.095 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.095 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.095 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:11.095 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.084361+0000", 2024-09-21T12:32:11.095 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0589999999999999e-06 2024-09-21T12:32:11.095 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.095 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.095 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:11.096 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.084364+0000", 2024-09-21T12:32:11.096 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.3050000000000001e-06 2024-09-21T12:32:11.096 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.096 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.096 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:11.096 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.084373+0000", 2024-09-21T12:32:11.096 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.2269999999999993e-06 2024-09-21T12:32:11.096 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.096 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.096 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:11.096 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.084378+0000", 2024-09-21T12:32:11.096 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7400000000000001e-06 2024-09-21T12:32:11.096 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.097 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.097 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.097 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.084461+0000", 2024-09-21T12:32:11.097 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.2650000000000003e-05 2024-09-21T12:32:11.097 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.097 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.097 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:32:11.097 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.084469+0000", 2024-09-21T12:32:11.097 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.695e-06 2024-09-21T12:32:11.097 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.097 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.097 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:11.097 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.048948+0000", 2024-09-21T12:32:11.097 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.96447884399999995 2024-09-21T12:32:11.097 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.098 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.098 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:11.098 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.049009+0000", 2024-09-21T12:32:11.098 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1017999999999998e-05 2024-09-21T12:32:11.098 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.098 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.098 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,5", 2024-09-21T12:32:11.098 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.049163+0000", 2024-09-21T12:32:11.098 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000154375 2024-09-21T12:32:11.098 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.098 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.098 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:11.098 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.051692+0000", 2024-09-21T12:32:11.098 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002528831 2024-09-21T12:32:11.098 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.099 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.099 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:32:11.099 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.051702+0000", 2024-09-21T12:32:11.099 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0161e-05 2024-09-21T12:32:11.099 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.099 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.099 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:11.099 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.051739+0000", 2024-09-21T12:32:11.099 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.6704e-05 2024-09-21T12:32:11.099 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.099 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.099 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:32:11.099 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.051790+0000", 2024-09-21T12:32:11.099 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0979000000000001e-05 2024-09-21T12:32:11.100 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:11.100 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:11.100 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:11.100 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.051824+0000", 2024-09-21T12:32:11.100 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4573999999999998e-05 2024-09-21T12:32:11.100 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.100 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:11.100 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.100 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:11.100 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:11.100 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:32:11.101 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:11.121 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:11.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:11.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:11.223 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:11.244 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:11.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:11.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:11.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:11 smithi114 ceph-mon[40867]: pgmap v600: 188 pgs: 3 active+remapped+backfill_toofull, 185 active+clean; 115 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 702 op/s; 218/6102 objects misplaced (3.573%); 4.2 MiB/s, 69 objects/s recovering 2024-09-21T12:32:11.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:11 smithi114 ceph-mon[40867]: osdmap e331: 8 total, 8 up, 7 in 2024-09-21T12:32:11.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:11 smithi114 ceph-mon[30324]: pgmap v600: 188 pgs: 3 active+remapped+backfill_toofull, 185 active+clean; 115 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 702 op/s; 218/6102 objects misplaced (3.573%); 4.2 MiB/s, 69 objects/s recovering 2024-09-21T12:32:11.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:11 smithi114 ceph-mon[30324]: osdmap e331: 8 total, 8 up, 7 in 2024-09-21T12:32:11.345 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:11.367 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:11.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:11.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:11.469 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:11.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:11 smithi183 ceph-mon[33277]: pgmap v600: 188 pgs: 3 active+remapped+backfill_toofull, 185 active+clean; 115 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 702 op/s; 218/6102 objects misplaced (3.573%); 4.2 MiB/s, 69 objects/s recovering 2024-09-21T12:32:11.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:11 smithi183 ceph-mon[33277]: osdmap e331: 8 total, 8 up, 7 in 2024-09-21T12:32:11.490 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:11.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:11.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:11.591 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:11.617 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:11.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:11.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:11.719 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:11.744 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:11.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:11.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:11.846 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:11.866 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T12:32:11.871 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:11.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:11.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:11.973 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:11.994 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:11.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:11.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:12.096 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:12.121 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:12.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:12.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:12.223 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:12.247 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:12.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:12.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:12.348 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:12.353 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T12:32:12.372 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:12.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:12.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:12.435 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:32:12.458 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:12 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:32:12.458 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:12 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:32:12.458 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:12 smithi114 ceph-mon[30324]: osdmap e332: 8 total, 8 up, 7 in 2024-09-21T12:32:12.458 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:12 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:32:12.458 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:12 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:32:12.458 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:12 smithi114 ceph-mon[40867]: osdmap e332: 8 total, 8 up, 7 in 2024-09-21T12:32:12.474 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:12 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:32:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:12 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:32:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:12 smithi183 ceph-mon[33277]: osdmap e332: 8 total, 8 up, 7 in 2024-09-21T12:32:12.497 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:12.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:12.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:12.599 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:12.623 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:12.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:12.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:12.724 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:12.735 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:32:12 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:32:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:32:12.750 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:12.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:12.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:12.852 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:12.877 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:12.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:12.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:12.979 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:13.003 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:13.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:13.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:13.012 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:13.052 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:13.058 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:13.105 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:13.126 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:13.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:13.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:13.223 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T12:32:13.227 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:13.251 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:13.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:13.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:13.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:13 smithi114 ceph-mon[30324]: pgmap v603: 188 pgs: 2 active+remapped+backfill_toofull, 186 active+clean; 38 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 979 op/s; 7/2373 objects misplaced (0.295%); 4.5 MiB/s, 73 objects/s recovering 2024-09-21T12:32:13.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:13 smithi114 ceph-mon[30324]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:32:13.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:13 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1230087805' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:32:13.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:13 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:32:13.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:13 smithi114 ceph-mon[40867]: pgmap v603: 188 pgs: 2 active+remapped+backfill_toofull, 186 active+clean; 38 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 979 op/s; 7/2373 objects misplaced (0.295%); 4.5 MiB/s, 73 objects/s recovering 2024-09-21T12:32:13.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:13 smithi114 ceph-mon[40867]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:32:13.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:13 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1230087805' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:32:13.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:13 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:32:13.352 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:13.375 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:13.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:13.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:13.476 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:13.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:13 smithi183 ceph-mon[33277]: pgmap v603: 188 pgs: 2 active+remapped+backfill_toofull, 186 active+clean; 38 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 979 op/s; 7/2373 objects misplaced (0.295%); 4.5 MiB/s, 73 objects/s recovering 2024-09-21T12:32:13.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:13 smithi183 ceph-mon[33277]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:32:13.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:13 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1230087805' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:32:13.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:13 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:32:13.498 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:13.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:13.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:13.523 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-09-21T12:32:13.523 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-09-21T12:32:13.599 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:13.621 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:13.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:13.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:13.723 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:13.744 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:13.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:13.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:13.846 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:13.867 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:13.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:13.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:13.969 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:13.990 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:13.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:13.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:14.069 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T12:32:14.069 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:32:14.092 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:14.117 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:14.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:14.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:14.219 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:14.263 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:14.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:14.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:14.364 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:14.387 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:14.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:14.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:14.404 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:14 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T12:32:14.404 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:14 smithi114 ceph-mon[40867]: osdmap e333: 8 total, 8 up, 7 in 2024-09-21T12:32:14.404 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:14 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T12:32:14.405 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:14 smithi114 ceph-mon[30324]: osdmap e333: 8 total, 8 up, 7 in 2024-09-21T12:32:14.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:14 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T12:32:14.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:14 smithi183 ceph-mon[33277]: osdmap e333: 8 total, 8 up, 7 in 2024-09-21T12:32:14.488 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:14.509 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:14.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:14.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:14.611 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:14.634 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:14.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:14.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:14.735 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:14.758 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:14.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:14.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:14.859 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:14.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:14.882 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:14.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:14.964 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:32:14.964 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:32:14.964 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 0 2024-09-21T12:32:14.964 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:32:14.983 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:15.005 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:15.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:15.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:15.106 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:15.133 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:15.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:15.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:15.220 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-21T12:32:15.235 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:15.260 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:15.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:15.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:15.362 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:15.383 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:15.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:15.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:15.485 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:15.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:15 smithi183 ceph-mon[33277]: pgmap v605: 188 pgs: 2 active+remapped+backfill_toofull, 186 active+clean; 999 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 430 op/s; 7/618 objects misplaced (1.133%) 2024-09-21T12:32:15.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:15 smithi183 ceph-mon[33277]: osdmap e334: 8 total, 8 up, 7 in 2024-09-21T12:32:15.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:15 smithi183 ceph-mon[33277]: 9.7 continuing backfill to osd.3 from (276'9323,330'12788] MIN to 330'12788 2024-09-21T12:32:15.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:15 smithi183 ceph-mon[33277]: 9.7 starting backfill to osd.7 from (0'0,0'0] MAX to 330'12788 2024-09-21T12:32:15.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:15 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/4148851128' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:32:15.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:15 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1620834445' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:32:15.506 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:15.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:15.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:15.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:15 smithi114 ceph-mon[30324]: pgmap v605: 188 pgs: 2 active+remapped+backfill_toofull, 186 active+clean; 999 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 430 op/s; 7/618 objects misplaced (1.133%) 2024-09-21T12:32:15.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:15 smithi114 ceph-mon[30324]: osdmap e334: 8 total, 8 up, 7 in 2024-09-21T12:32:15.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:15 smithi114 ceph-mon[30324]: 9.7 continuing backfill to osd.3 from (276'9323,330'12788] MIN to 330'12788 2024-09-21T12:32:15.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:15 smithi114 ceph-mon[30324]: 9.7 starting backfill to osd.7 from (0'0,0'0] MAX to 330'12788 2024-09-21T12:32:15.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:15 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4148851128' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:32:15.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:15 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1620834445' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:32:15.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:15 smithi114 ceph-mon[40867]: pgmap v605: 188 pgs: 2 active+remapped+backfill_toofull, 186 active+clean; 999 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 430 op/s; 7/618 objects misplaced (1.133%) 2024-09-21T12:32:15.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:15 smithi114 ceph-mon[40867]: osdmap e334: 8 total, 8 up, 7 in 2024-09-21T12:32:15.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:15 smithi114 ceph-mon[40867]: 9.7 continuing backfill to osd.3 from (276'9323,330'12788] MIN to 330'12788 2024-09-21T12:32:15.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:15 smithi114 ceph-mon[40867]: 9.7 starting backfill to osd.7 from (0'0,0'0] MAX to 330'12788 2024-09-21T12:32:15.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:15 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/4148851128' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:32:15.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:15 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1620834445' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:32:15.608 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:15.629 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:15.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:15.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:15.731 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:15.752 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:15.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:15.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:15.853 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:15.875 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:15.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:15.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:15.976 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:16.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:16.003 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:16.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:16.104 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:16.130 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:16.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:16.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:16.232 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:16.254 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:16.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:16.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:16.356 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:16.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:16.377 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:16.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:16.478 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:16 smithi183 ceph-mon[33277]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:32:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:16 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/4148851128' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:32:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:16 smithi183 ceph-mon[33277]: 9.5 starting backfill to osd.7 from (0'0,0'0] MAX to 330'12726 2024-09-21T12:32:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:16 smithi183 ceph-mon[33277]: osdmap e335: 8 total, 8 up, 7 in 2024-09-21T12:32:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:16 smithi183 ceph-mon[33277]: 9.1 starting backfill to osd.7 from (0'0,0'0] MAX to 330'12847 2024-09-21T12:32:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:16 smithi183 ceph-mon[33277]: osdmap e336: 8 total, 8 up, 7 in 2024-09-21T12:32:16.501 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:16.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:16.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:16.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:16 smithi114 ceph-mon[40867]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:32:16.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:16 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/4148851128' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:32:16.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:16 smithi114 ceph-mon[40867]: 9.5 starting backfill to osd.7 from (0'0,0'0] MAX to 330'12726 2024-09-21T12:32:16.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:16 smithi114 ceph-mon[40867]: osdmap e335: 8 total, 8 up, 7 in 2024-09-21T12:32:16.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:16 smithi114 ceph-mon[40867]: 9.1 starting backfill to osd.7 from (0'0,0'0] MAX to 330'12847 2024-09-21T12:32:16.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:16 smithi114 ceph-mon[40867]: osdmap e336: 8 total, 8 up, 7 in 2024-09-21T12:32:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:16 smithi114 ceph-mon[30324]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:32:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:16 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4148851128' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:32:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:16 smithi114 ceph-mon[30324]: 9.5 starting backfill to osd.7 from (0'0,0'0] MAX to 330'12726 2024-09-21T12:32:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:16 smithi114 ceph-mon[30324]: osdmap e335: 8 total, 8 up, 7 in 2024-09-21T12:32:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:16 smithi114 ceph-mon[30324]: 9.1 starting backfill to osd.7 from (0'0,0'0] MAX to 330'12847 2024-09-21T12:32:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:16 smithi114 ceph-mon[30324]: osdmap e336: 8 total, 8 up, 7 in 2024-09-21T12:32:16.603 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:16.624 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:16.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:16.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:16.726 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:16.751 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:16.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:16.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:16.852 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:16.876 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:16.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:16.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:16.978 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:16.999 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:17.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:17.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:17.101 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:17.122 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:17.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:17.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:17.223 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:17.247 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:17.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:17.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:17.349 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:17.373 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:17.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:17.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:17.474 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:17.496 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:17.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:17.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:17.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:17 smithi114 ceph-mon[40867]: 6.b scrub starts 2024-09-21T12:32:17.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:17 smithi114 ceph-mon[40867]: 6.b scrub ok 2024-09-21T12:32:17.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:17 smithi114 ceph-mon[40867]: pgmap v608: 188 pgs: 2 active+remapped+backfill_toofull, 186 active+clean; 999 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 419 op/s; 7/618 objects misplaced (1.133%) 2024-09-21T12:32:17.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:17 smithi114 ceph-mon[40867]: 9.7 scrub starts 2024-09-21T12:32:17.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:17 smithi114 ceph-mon[40867]: 9.7 scrub ok 2024-09-21T12:32:17.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:17 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:32:17.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:17 smithi114 ceph-mon[40867]: osdmap e337: 8 total, 8 up, 7 in 2024-09-21T12:32:17.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:17 smithi114 ceph-mon[30324]: 6.b scrub starts 2024-09-21T12:32:17.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:17 smithi114 ceph-mon[30324]: 6.b scrub ok 2024-09-21T12:32:17.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:17 smithi114 ceph-mon[30324]: pgmap v608: 188 pgs: 2 active+remapped+backfill_toofull, 186 active+clean; 999 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 419 op/s; 7/618 objects misplaced (1.133%) 2024-09-21T12:32:17.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:17 smithi114 ceph-mon[30324]: 9.7 scrub starts 2024-09-21T12:32:17.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:17 smithi114 ceph-mon[30324]: 9.7 scrub ok 2024-09-21T12:32:17.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:17 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:32:17.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:17 smithi114 ceph-mon[30324]: osdmap e337: 8 total, 8 up, 7 in 2024-09-21T12:32:17.599 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:17.620 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:17.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:17.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:17.672 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T12:32:17.721 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:17.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:32:17 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:32:17.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:17 smithi183 ceph-mon[33277]: 6.b scrub starts 2024-09-21T12:32:17.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:17 smithi183 ceph-mon[33277]: 6.b scrub ok 2024-09-21T12:32:17.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:17 smithi183 ceph-mon[33277]: pgmap v608: 188 pgs: 2 active+remapped+backfill_toofull, 186 active+clean; 999 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 419 op/s; 7/618 objects misplaced (1.133%) 2024-09-21T12:32:17.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:17 smithi183 ceph-mon[33277]: 9.7 scrub starts 2024-09-21T12:32:17.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:17 smithi183 ceph-mon[33277]: 9.7 scrub ok 2024-09-21T12:32:17.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:17 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:32:17.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:17 smithi183 ceph-mon[33277]: osdmap e337: 8 total, 8 up, 7 in 2024-09-21T12:32:17.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:17.743 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:17.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:17.808 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:32:17.808 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:32:17.808 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:32:17.808 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:32:17.808 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:32:17.808 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:32:17.808 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:32:17.808 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:32:17.809 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:32:17.809 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:32:17.809 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:32:17.809 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:32:17.809 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:32:17.809 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:32:17.809 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:32:17.809 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:32:17.844 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:17.866 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:17.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:17.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:17.967 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:17.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:17.989 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:17.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:18.073 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:18.089 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:18.112 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:18.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:18.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:18.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T12:32:18.214 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:18.237 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:18.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:18.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:18.338 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:18.361 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:18.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:18.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:18.398 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:18 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:18.398 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:18 smithi114 ceph-mon[30324]: 5.1f scrub starts 2024-09-21T12:32:18.398 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:18 smithi114 ceph-mon[30324]: 5.1f scrub ok 2024-09-21T12:32:18.399 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:18 smithi114 ceph-mon[30324]: 9.a scrub starts 2024-09-21T12:32:18.399 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:18 smithi114 ceph-mon[30324]: 9.a scrub ok 2024-09-21T12:32:18.399 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:18 smithi114 ceph-mon[30324]: 3.12 scrub starts 2024-09-21T12:32:18.399 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:18 smithi114 ceph-mon[30324]: 3.12 scrub ok 2024-09-21T12:32:18.399 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:18 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:18.399 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:18 smithi114 ceph-mon[40867]: 5.1f scrub starts 2024-09-21T12:32:18.399 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:18 smithi114 ceph-mon[40867]: 5.1f scrub ok 2024-09-21T12:32:18.399 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:18 smithi114 ceph-mon[40867]: 9.a scrub starts 2024-09-21T12:32:18.400 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:18 smithi114 ceph-mon[40867]: 9.a scrub ok 2024-09-21T12:32:18.400 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:18 smithi114 ceph-mon[40867]: 3.12 scrub starts 2024-09-21T12:32:18.400 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:18 smithi114 ceph-mon[40867]: 3.12 scrub ok 2024-09-21T12:32:18.462 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:18.484 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:18.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:18.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:18.586 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:18.607 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:18.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:18.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:18.709 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:18.730 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:18.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:18.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:18.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:18 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:18.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:18 smithi183 ceph-mon[33277]: 5.1f scrub starts 2024-09-21T12:32:18.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:18 smithi183 ceph-mon[33277]: 5.1f scrub ok 2024-09-21T12:32:18.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:18 smithi183 ceph-mon[33277]: 9.a scrub starts 2024-09-21T12:32:18.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:18 smithi183 ceph-mon[33277]: 9.a scrub ok 2024-09-21T12:32:18.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:18 smithi183 ceph-mon[33277]: 3.12 scrub starts 2024-09-21T12:32:18.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:18 smithi183 ceph-mon[33277]: 3.12 scrub ok 2024-09-21T12:32:18.831 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:18.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:18.853 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:18.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:18.954 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:18.975 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:18.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:18.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:19.077 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:19.099 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:19.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:19.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:19.200 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:19.222 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:19.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:19.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:19.325 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:19.348 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:19.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:19.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:19.450 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:19.473 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:19.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:19.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:19.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:19 smithi114 ceph-mon[40867]: 6.1b scrub starts 2024-09-21T12:32:19.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:19 smithi114 ceph-mon[40867]: 6.1b scrub ok 2024-09-21T12:32:19.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:19 smithi114 ceph-mon[40867]: pgmap v611: 188 pgs: 2 active+remapped+backfill_toofull, 186 active+clean; 999 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 7/618 objects misplaced (1.133%) 2024-09-21T12:32:19.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:19 smithi114 ceph-mon[30324]: 6.1b scrub starts 2024-09-21T12:32:19.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:19 smithi114 ceph-mon[30324]: 6.1b scrub ok 2024-09-21T12:32:19.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:19 smithi114 ceph-mon[30324]: pgmap v611: 188 pgs: 2 active+remapped+backfill_toofull, 186 active+clean; 999 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 7/618 objects misplaced (1.133%) 2024-09-21T12:32:19.575 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:19.598 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:19.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:19.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:19.700 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:19.721 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:19.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:19.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:19.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:19 smithi183 ceph-mon[33277]: 6.1b scrub starts 2024-09-21T12:32:19.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:19 smithi183 ceph-mon[33277]: 6.1b scrub ok 2024-09-21T12:32:19.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:19 smithi183 ceph-mon[33277]: pgmap v611: 188 pgs: 2 active+remapped+backfill_toofull, 186 active+clean; 999 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 7/618 objects misplaced (1.133%) 2024-09-21T12:32:19.823 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:19.844 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:19.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:19.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:19.945 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:19.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:19.967 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:19.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:20.068 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:20.091 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:20.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:20.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:20.093 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:20.094 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:20.134 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:20.193 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:20.220 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:20.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:20.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:20.265 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'unique_pool_2' removed 2024-09-21T12:32:20.322 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:20.365 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:20.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:20.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:20.375 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:20 smithi114 ceph-mon[40867]: 2.2 scrub starts 2024-09-21T12:32:20.375 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:20 smithi114 ceph-mon[40867]: 2.2 scrub ok 2024-09-21T12:32:20.375 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:20 smithi114 ceph-mon[40867]: 3.5 scrub starts 2024-09-21T12:32:20.375 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:20 smithi114 ceph-mon[40867]: 3.5 scrub ok 2024-09-21T12:32:20.375 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:20 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2747829298' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T12:32:20.375 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:20 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T12:32:20.376 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:20 smithi114 ceph-mon[40867]: 4.16 scrub starts 2024-09-21T12:32:20.376 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:20 smithi114 ceph-mon[40867]: 4.16 scrub ok 2024-09-21T12:32:20.376 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:20 smithi114 ceph-mon[40867]: 5.d scrub starts 2024-09-21T12:32:20.376 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:20 smithi114 ceph-mon[40867]: 5.d scrub ok 2024-09-21T12:32:20.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:20 smithi114 ceph-mon[30324]: 2.2 scrub starts 2024-09-21T12:32:20.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:20 smithi114 ceph-mon[30324]: 2.2 scrub ok 2024-09-21T12:32:20.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:20 smithi114 ceph-mon[30324]: 3.5 scrub starts 2024-09-21T12:32:20.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:20 smithi114 ceph-mon[30324]: 3.5 scrub ok 2024-09-21T12:32:20.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:20 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2747829298' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T12:32:20.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:20 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T12:32:20.377 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:20 smithi114 ceph-mon[30324]: 4.16 scrub starts 2024-09-21T12:32:20.377 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:20 smithi114 ceph-mon[30324]: 4.16 scrub ok 2024-09-21T12:32:20.377 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:20 smithi114 ceph-mon[30324]: 5.d scrub starts 2024-09-21T12:32:20.377 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:20 smithi114 ceph-mon[30324]: 5.d scrub ok 2024-09-21T12:32:20.467 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:20.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:20.491 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:20.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:20.592 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:20.613 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:20.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:20.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:20.714 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:20.736 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:20.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:20.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:20.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:20 smithi183 ceph-mon[33277]: 2.2 scrub starts 2024-09-21T12:32:20.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:20 smithi183 ceph-mon[33277]: 2.2 scrub ok 2024-09-21T12:32:20.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:20 smithi183 ceph-mon[33277]: 3.5 scrub starts 2024-09-21T12:32:20.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:20 smithi183 ceph-mon[33277]: 3.5 scrub ok 2024-09-21T12:32:20.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:20 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2747829298' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T12:32:20.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:20 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T12:32:20.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:20 smithi183 ceph-mon[33277]: 4.16 scrub starts 2024-09-21T12:32:20.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:20 smithi183 ceph-mon[33277]: 4.16 scrub ok 2024-09-21T12:32:20.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:20 smithi183 ceph-mon[33277]: 5.d scrub starts 2024-09-21T12:32:20.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:20 smithi183 ceph-mon[33277]: 5.d scrub ok 2024-09-21T12:32:20.838 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:20.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:20.862 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:20.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:20.963 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:20.985 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:20.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:20.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:21.087 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:21.108 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:21.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:21.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:21.210 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:21.232 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:21.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:21.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:21.271 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-21T12:32:21.334 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:21.356 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:21.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:21.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:21.457 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:21.478 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:21.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:21.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:21.499 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:32:21.501 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:32:21.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:21 smithi114 ceph-mon[40867]: 3.2 scrub starts 2024-09-21T12:32:21.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:21 smithi114 ceph-mon[40867]: 3.2 scrub ok 2024-09-21T12:32:21.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:21 smithi114 ceph-mon[40867]: pgmap v612: 188 pgs: 2 active+remapped+backfill_toofull, 186 active+clean; 999 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 915 B/s rd, 0 op/s; 7/618 objects misplaced (1.133%) 2024-09-21T12:32:21.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:21 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T12:32:21.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:21 smithi114 ceph-mon[40867]: osdmap e338: 8 total, 8 up, 7 in 2024-09-21T12:32:21.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:21 smithi114 ceph-mon[40867]: 1.15 scrub starts 2024-09-21T12:32:21.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:21 smithi114 ceph-mon[40867]: 1.15 scrub ok 2024-09-21T12:32:21.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:21 smithi114 ceph-mon[40867]: 2.1 scrub starts 2024-09-21T12:32:21.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:21 smithi114 ceph-mon[40867]: 2.1 scrub ok 2024-09-21T12:32:21.510 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:21 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3516841600' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:32:21.510 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:21 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:32:21.510 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:21 smithi114 ceph-mon[30324]: 3.2 scrub starts 2024-09-21T12:32:21.510 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:21 smithi114 ceph-mon[30324]: 3.2 scrub ok 2024-09-21T12:32:21.510 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:21 smithi114 ceph-mon[30324]: pgmap v612: 188 pgs: 2 active+remapped+backfill_toofull, 186 active+clean; 999 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 915 B/s rd, 0 op/s; 7/618 objects misplaced (1.133%) 2024-09-21T12:32:21.510 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:21 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T12:32:21.510 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:21 smithi114 ceph-mon[30324]: osdmap e338: 8 total, 8 up, 7 in 2024-09-21T12:32:21.510 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:21 smithi114 ceph-mon[30324]: 1.15 scrub starts 2024-09-21T12:32:21.510 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:21 smithi114 ceph-mon[30324]: 1.15 scrub ok 2024-09-21T12:32:21.510 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:21 smithi114 ceph-mon[30324]: 2.1 scrub starts 2024-09-21T12:32:21.510 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:21 smithi114 ceph-mon[30324]: 2.1 scrub ok 2024-09-21T12:32:21.511 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:21 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3516841600' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:32:21.511 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:21 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:32:21.580 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:21.602 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:21.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:21.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:21.703 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:21.725 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:21.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:21.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:21.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:21 smithi183 ceph-mon[33277]: 3.2 scrub starts 2024-09-21T12:32:21.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:21 smithi183 ceph-mon[33277]: 3.2 scrub ok 2024-09-21T12:32:21.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:21 smithi183 ceph-mon[33277]: pgmap v612: 188 pgs: 2 active+remapped+backfill_toofull, 186 active+clean; 999 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 915 B/s rd, 0 op/s; 7/618 objects misplaced (1.133%) 2024-09-21T12:32:21.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:21 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T12:32:21.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:21 smithi183 ceph-mon[33277]: osdmap e338: 8 total, 8 up, 7 in 2024-09-21T12:32:21.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:21 smithi183 ceph-mon[33277]: 1.15 scrub starts 2024-09-21T12:32:21.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:21 smithi183 ceph-mon[33277]: 1.15 scrub ok 2024-09-21T12:32:21.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:21 smithi183 ceph-mon[33277]: 2.1 scrub starts 2024-09-21T12:32:21.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:21 smithi183 ceph-mon[33277]: 2.1 scrub ok 2024-09-21T12:32:21.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:21 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3516841600' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:32:21.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:21 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:32:21.827 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:21.848 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:21.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:21.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:21.949 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:21.971 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:21.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:21.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:22.072 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:22.093 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:22.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:22.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:22.138 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:32:22.195 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:22.216 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:22.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:22.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:22.285 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":614,"stamp":"2024-09-21T12:32:21.261613+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":1850,"num_read_kb":1602,"num_write":722,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":340,"num_bytes_recovered":6950,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3234,"ondisk_log_size":3234,"up":486,"acting":486,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":575,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":3892268,"kb_used_data":49080,"kb_used_omap":10,"kb_used_meta":3842997,"kb_avail":652323796,"statfs":{"total":671965249536,"available":667979567104,"internally_reserved":0,"allocated":50257920,"data_stored":46405754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":3935229035},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-425984,"num_objects":-7,"num_object_clones":0,"num_object_copies":-21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-7,"num_objects_unfound":0,"num_objects_dirty":-7,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":-151296,"num_write_kb":-3976513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-22083,"num_bytes_recovered":-1449360812,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.445677"},"pg_stats":[{"pgid":"1.1d","version":"76'79","reported_seq":281,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.514160+0000","last_change":"2024-09-21T12:31:50.043467+0000","last_active":"2024-09-21T12:32:17.514160+0000","last_peered":"2024-09-21T12:32:17.514160+0000","last_clean":"2024-09-21T12:32:17.514160+0000","last_became_active":"2024-09-21T12:31:50.043059+0000","last_became_peered":"2024-09-21T12:31:50.043059+0000","last_unstale":"2024-09-21T12:32:17.514160+0000","last_undegraded":"2024-09-21T12:32:17.514160+0000","last_fullsized":"2024-09-21T12:32:17.514160+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:47.816137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:47.816137+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":589,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.227895+0000","last_change":"2024-09-21T12:24:50.115300+0000","last_active":"2024-09-21T12:32:17.227895+0000","last_peered":"2024-09-21T12:32:17.227895+0000","last_clean":"2024-09-21T12:32:17.227895+0000","last_became_active":"2024-09-21T12:20:15.309431+0000","last_became_peered":"2024-09-21T12:20:15.309431+0000","last_unstale":"2024-09-21T12:32:17.227895+0000","last_undegraded":"2024-09-21T12:32:17.227895+0000","last_fullsized":"2024-09-21T12:32:17.227895+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:50.115253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:50.115253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:24.563784+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":226,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.513107+0000","last_change":"2024-09-21T12:29:08.671789+0000","last_active":"2024-09-21T12:32:17.513107+0000","last_peered":"2024-09-21T12:32:17.513107+0000","last_clean":"2024-09-21T12:32:17.513107+0000","last_became_active":"2024-09-21T12:29:05.833372+0000","last_became_peered":"2024-09-21T12:29:05.833372+0000","last_unstale":"2024-09-21T12:32:17.513107+0000","last_undegraded":"2024-09-21T12:32:17.513107+0000","last_fullsized":"2024-09-21T12:32:17.513107+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:29:08.671723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:29:08.671723+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":586,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.513175+0000","last_change":"2024-09-21T12:29:10.672659+0000","last_active":"2024-09-21T12:32:17.513175+0000","last_peered":"2024-09-21T12:32:17.513175+0000","last_clean":"2024-09-21T12:32:17.513175+0000","last_became_active":"2024-09-21T12:20:19.314908+0000","last_became_peered":"2024-09-21T12:20:19.314908+0000","last_unstale":"2024-09-21T12:32:17.513175+0000","last_undegraded":"2024-09-21T12:32:17.513175+0000","last_fullsized":"2024-09-21T12:32:17.513175+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:10.672599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:29:10.672599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":531,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.231218+0000","last_change":"2024-09-21T12:31:39.061111+0000","last_active":"2024-09-21T12:32:16.231218+0000","last_peered":"2024-09-21T12:32:16.231218+0000","last_clean":"2024-09-21T12:32:16.231218+0000","last_became_active":"2024-09-21T12:31:39.060551+0000","last_became_peered":"2024-09-21T12:31:39.060551+0000","last_unstale":"2024-09-21T12:32:16.231218+0000","last_undegraded":"2024-09-21T12:32:16.231218+0000","last_fullsized":"2024-09-21T12:32:16.231218+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:12.581400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:31:12.581400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:31.372481+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1c","version":"76'79","reported_seq":255,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.228792+0000","last_change":"2024-09-21T12:27:28.374689+0000","last_active":"2024-09-21T12:32:16.228792+0000","last_peered":"2024-09-21T12:32:16.228792+0000","last_clean":"2024-09-21T12:32:16.228792+0000","last_became_active":"2024-09-21T12:27:16.920545+0000","last_became_peered":"2024-09-21T12:27:16.920545+0000","last_unstale":"2024-09-21T12:32:16.228792+0000","last_undegraded":"2024-09-21T12:32:16.228792+0000","last_fullsized":"2024-09-21T12:32:16.228792+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:28.374616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:28.374616+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:11.168406+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":573,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.528064+0000","last_change":"2024-09-21T12:32:17.528064+0000","last_active":"2024-09-21T12:32:17.528064+0000","last_peered":"2024-09-21T12:32:17.528064+0000","last_clean":"2024-09-21T12:32:17.528064+0000","last_became_active":"2024-09-21T12:20:21.595301+0000","last_became_peered":"2024-09-21T12:20:21.595301+0000","last_unstale":"2024-09-21T12:32:17.528064+0000","last_undegraded":"2024-09-21T12:32:17.528064+0000","last_fullsized":"2024-09-21T12:32:17.528064+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:17.527992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:32:17.527992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:44.229295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":594,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.228563+0000","last_change":"2024-09-21T12:29:16.079291+0000","last_active":"2024-09-21T12:32:17.228563+0000","last_peered":"2024-09-21T12:32:17.228563+0000","last_clean":"2024-09-21T12:32:17.228563+0000","last_became_active":"2024-09-21T12:20:15.308655+0000","last_became_peered":"2024-09-21T12:20:15.308655+0000","last_unstale":"2024-09-21T12:32:17.228563+0000","last_undegraded":"2024-09-21T12:32:17.228563+0000","last_fullsized":"2024-09-21T12:32:17.228563+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:16.079221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:16.079221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":640,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.225598+0000","last_change":"2024-09-21T12:31:48.294386+0000","last_active":"2024-09-21T12:32:17.225598+0000","last_peered":"2024-09-21T12:32:17.225598+0000","last_clean":"2024-09-21T12:32:17.225598+0000","last_became_active":"2024-09-21T12:20:17.305220+0000","last_became_peered":"2024-09-21T12:20:17.305220+0000","last_unstale":"2024-09-21T12:32:17.225598+0000","last_undegraded":"2024-09-21T12:32:17.225598+0000","last_fullsized":"2024-09-21T12:32:17.225598+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:31:48.294300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:48.294300+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:53.699472+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":526,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.685236+0000","last_change":"2024-09-21T12:28:39.179737+0000","last_active":"2024-09-21T12:32:16.685236+0000","last_peered":"2024-09-21T12:32:16.685236+0000","last_clean":"2024-09-21T12:32:16.685236+0000","last_became_active":"2024-09-21T12:22:50.926240+0000","last_became_peered":"2024-09-21T12:22:50.926240+0000","last_unstale":"2024-09-21T12:32:16.685236+0000","last_undegraded":"2024-09-21T12:32:16.685236+0000","last_fullsized":"2024-09-21T12:32:16.685236+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:39.179665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:39.179665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:30.468144+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":438,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.685881+0000","last_change":"2024-09-21T12:31:38.380246+0000","last_active":"2024-09-21T12:32:16.685881+0000","last_peered":"2024-09-21T12:32:16.685881+0000","last_clean":"2024-09-21T12:32:16.685881+0000","last_became_active":"2024-09-21T12:26:54.083981+0000","last_became_peered":"2024-09-21T12:26:54.083981+0000","last_unstale":"2024-09-21T12:32:16.685881+0000","last_undegraded":"2024-09-21T12:32:16.685881+0000","last_fullsized":"2024-09-21T12:32:16.685881+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:38.380155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:31:38.380155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:04.416228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":584,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.226039+0000","last_change":"2024-09-21T12:28:06.589984+0000","last_active":"2024-09-21T12:32:17.226039+0000","last_peered":"2024-09-21T12:32:17.226039+0000","last_clean":"2024-09-21T12:32:17.226039+0000","last_became_active":"2024-09-21T12:22:50.927804+0000","last_became_peered":"2024-09-21T12:22:50.927804+0000","last_unstale":"2024-09-21T12:32:17.226039+0000","last_undegraded":"2024-09-21T12:32:17.226039+0000","last_fullsized":"2024-09-21T12:32:17.226039+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:28:06.589926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:06.589926+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:05.555607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":622,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.231752+0000","last_change":"2024-09-21T12:32:14.238277+0000","last_active":"2024-09-21T12:32:16.231752+0000","last_peered":"2024-09-21T12:32:16.231752+0000","last_clean":"2024-09-21T12:32:16.231752+0000","last_became_active":"2024-09-21T12:32:14.237803+0000","last_became_peered":"2024-09-21T12:32:14.237803+0000","last_unstale":"2024-09-21T12:32:16.231752+0000","last_undegraded":"2024-09-21T12:32:16.231752+0000","last_fullsized":"2024-09-21T12:32:16.231752+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:56.620241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:31:56.620241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:01.685704+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":584,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.225589+0000","last_change":"2024-09-21T12:27:54.959865+0000","last_active":"2024-09-21T12:32:17.225589+0000","last_peered":"2024-09-21T12:32:17.225589+0000","last_clean":"2024-09-21T12:32:17.225589+0000","last_became_active":"2024-09-21T12:20:21.309190+0000","last_became_peered":"2024-09-21T12:20:21.309190+0000","last_unstale":"2024-09-21T12:32:17.225589+0000","last_undegraded":"2024-09-21T12:32:17.225589+0000","last_fullsized":"2024-09-21T12:32:17.225589+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:54.959806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:27:54.959806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:06.710394+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"76'79","reported_seq":358,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.228446+0000","last_change":"2024-09-21T12:31:38.367318+0000","last_active":"2024-09-21T12:32:17.228446+0000","last_peered":"2024-09-21T12:32:17.228446+0000","last_clean":"2024-09-21T12:32:17.228446+0000","last_became_active":"2024-09-21T12:27:16.937711+0000","last_became_peered":"2024-09-21T12:27:16.937711+0000","last_unstale":"2024-09-21T12:32:17.228446+0000","last_undegraded":"2024-09-21T12:32:17.228446+0000","last_fullsized":"2024-09-21T12:32:17.228446+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:31:38.367273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:31:38.367273+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:41.463777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":432,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.513958+0000","last_change":"2024-09-21T12:28:33.644630+0000","last_active":"2024-09-21T12:32:17.513958+0000","last_peered":"2024-09-21T12:32:17.513958+0000","last_clean":"2024-09-21T12:32:17.513958+0000","last_became_active":"2024-09-21T12:26:54.085055+0000","last_became_peered":"2024-09-21T12:26:54.085055+0000","last_unstale":"2024-09-21T12:32:17.513958+0000","last_undegraded":"2024-09-21T12:32:17.513958+0000","last_fullsized":"2024-09-21T12:32:17.513958+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:33.644570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:33.644570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":590,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.226713+0000","last_change":"2024-09-21T12:31:39.064759+0000","last_active":"2024-09-21T12:32:17.226713+0000","last_peered":"2024-09-21T12:32:17.226713+0000","last_clean":"2024-09-21T12:32:17.226713+0000","last_became_active":"2024-09-21T12:31:39.055199+0000","last_became_peered":"2024-09-21T12:31:39.055199+0000","last_unstale":"2024-09-21T12:32:17.226713+0000","last_undegraded":"2024-09-21T12:32:17.226713+0000","last_fullsized":"2024-09-21T12:32:17.226713+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-21T12:29:35.982468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:29:35.982468+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":597,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.228042+0000","last_change":"2024-09-21T12:28:04.623499+0000","last_active":"2024-09-21T12:32:16.228042+0000","last_peered":"2024-09-21T12:32:16.228042+0000","last_clean":"2024-09-21T12:32:16.228042+0000","last_became_active":"2024-09-21T12:22:50.921780+0000","last_became_peered":"2024-09-21T12:22:50.921780+0000","last_unstale":"2024-09-21T12:32:16.228042+0000","last_undegraded":"2024-09-21T12:32:16.228042+0000","last_fullsized":"2024-09-21T12:32:16.228042+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:04.623389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:04.623389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":596,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.231586+0000","last_change":"2024-09-21T12:29:53.301655+0000","last_active":"2024-09-21T12:32:16.231586+0000","last_peered":"2024-09-21T12:32:16.231586+0000","last_clean":"2024-09-21T12:32:16.231586+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:32:16.231586+0000","last_undegraded":"2024-09-21T12:32:16.231586+0000","last_fullsized":"2024-09-21T12:32:16.231586+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:53.301583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:29:53.301583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:32.635151+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"76'79","reported_seq":421,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.226483+0000","last_change":"2024-09-21T12:27:22.289065+0000","last_active":"2024-09-21T12:32:17.226483+0000","last_peered":"2024-09-21T12:32:17.226483+0000","last_clean":"2024-09-21T12:32:17.226483+0000","last_became_active":"2024-09-21T12:27:16.957569+0000","last_became_peered":"2024-09-21T12:27:16.957569+0000","last_unstale":"2024-09-21T12:32:17.226483+0000","last_undegraded":"2024-09-21T12:32:17.226483+0000","last_fullsized":"2024-09-21T12:32:17.226483+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:22.288989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:22.288989+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:37.384920+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":232,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.227158+0000","last_change":"2024-09-21T12:29:05.826853+0000","last_active":"2024-09-21T12:32:17.227158+0000","last_peered":"2024-09-21T12:32:17.227158+0000","last_clean":"2024-09-21T12:32:17.227158+0000","last_became_active":"2024-09-21T12:29:05.826251+0000","last_became_peered":"2024-09-21T12:29:05.826251+0000","last_unstale":"2024-09-21T12:32:17.227158+0000","last_undegraded":"2024-09-21T12:32:17.227158+0000","last_fullsized":"2024-09-21T12:32:17.227158+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:44.845176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:44.845176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:25.947407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":639,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.225743+0000","last_change":"2024-09-21T12:31:39.056292+0000","last_active":"2024-09-21T12:32:17.225743+0000","last_peered":"2024-09-21T12:32:17.225743+0000","last_clean":"2024-09-21T12:32:17.225743+0000","last_became_active":"2024-09-21T12:31:39.055972+0000","last_became_peered":"2024-09-21T12:31:39.055972+0000","last_unstale":"2024-09-21T12:32:17.225743+0000","last_undegraded":"2024-09-21T12:32:17.225743+0000","last_fullsized":"2024-09-21T12:32:17.225743+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:16.148785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:16.148785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":640,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.861808+0000","last_change":"2024-09-21T12:31:41.717598+0000","last_active":"2024-09-21T12:32:16.861808+0000","last_peered":"2024-09-21T12:32:16.861808+0000","last_clean":"2024-09-21T12:32:16.861808+0000","last_became_active":"2024-09-21T12:20:17.307828+0000","last_became_peered":"2024-09-21T12:20:17.307828+0000","last_unstale":"2024-09-21T12:32:16.861808+0000","last_undegraded":"2024-09-21T12:32:16.861808+0000","last_fullsized":"2024-09-21T12:32:16.861808+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:31:41.717520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:41.717520+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:43.527174+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":614,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.514533+0000","last_change":"2024-09-21T12:28:06.636769+0000","last_active":"2024-09-21T12:32:17.514533+0000","last_peered":"2024-09-21T12:32:17.514533+0000","last_clean":"2024-09-21T12:32:17.514533+0000","last_became_active":"2024-09-21T12:25:46.905878+0000","last_became_peered":"2024-09-21T12:25:46.905878+0000","last_unstale":"2024-09-21T12:32:17.514533+0000","last_undegraded":"2024-09-21T12:32:17.514533+0000","last_fullsized":"2024-09-21T12:32:17.514533+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:06.636729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:06.636729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:01.978797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"76'79","reported_seq":520,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.231038+0000","last_change":"2024-09-21T12:29:51.267936+0000","last_active":"2024-09-21T12:32:16.231038+0000","last_peered":"2024-09-21T12:32:16.231038+0000","last_clean":"2024-09-21T12:32:16.231038+0000","last_became_active":"2024-09-21T12:27:16.927791+0000","last_became_peered":"2024-09-21T12:27:16.927791+0000","last_unstale":"2024-09-21T12:32:16.231038+0000","last_undegraded":"2024-09-21T12:32:16.231038+0000","last_fullsized":"2024-09-21T12:32:16.231038+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:51.267884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:51.267884+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:22.554630+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":595,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.228645+0000","last_change":"2024-09-21T12:32:02.033734+0000","last_active":"2024-09-21T12:32:17.228645+0000","last_peered":"2024-09-21T12:32:17.228645+0000","last_clean":"2024-09-21T12:32:17.228645+0000","last_became_active":"2024-09-21T12:32:02.033276+0000","last_became_peered":"2024-09-21T12:32:02.033276+0000","last_unstale":"2024-09-21T12:32:17.228645+0000","last_undegraded":"2024-09-21T12:32:17.228645+0000","last_fullsized":"2024-09-21T12:32:17.228645+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:45.880383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.880383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":526,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.513549+0000","last_change":"2024-09-21T12:28:45.726096+0000","last_active":"2024-09-21T12:32:17.513549+0000","last_peered":"2024-09-21T12:32:17.513549+0000","last_clean":"2024-09-21T12:32:17.513549+0000","last_became_active":"2024-09-21T12:22:50.925403+0000","last_became_peered":"2024-09-21T12:22:50.925403+0000","last_unstale":"2024-09-21T12:32:17.513549+0000","last_undegraded":"2024-09-21T12:32:17.513549+0000","last_fullsized":"2024-09-21T12:32:17.513549+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:45.725968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.725968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":678,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.227945+0000","last_change":"2024-09-21T12:28:30.620035+0000","last_active":"2024-09-21T12:32:17.227945+0000","last_peered":"2024-09-21T12:32:17.227945+0000","last_clean":"2024-09-21T12:32:17.227945+0000","last_became_active":"2024-09-21T12:22:50.912144+0000","last_became_peered":"2024-09-21T12:22:50.912144+0000","last_unstale":"2024-09-21T12:32:17.227945+0000","last_undegraded":"2024-09-21T12:32:17.227945+0000","last_fullsized":"2024-09-21T12:32:17.227945+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:28:30.619954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:30.619954+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":504,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.227158+0000","last_change":"2024-09-21T12:26:52.295004+0000","last_active":"2024-09-21T12:32:17.227158+0000","last_peered":"2024-09-21T12:32:17.227158+0000","last_clean":"2024-09-21T12:32:17.227158+0000","last_became_active":"2024-09-21T12:22:50.918326+0000","last_became_peered":"2024-09-21T12:22:50.918326+0000","last_unstale":"2024-09-21T12:32:17.227158+0000","last_undegraded":"2024-09-21T12:32:17.227158+0000","last_fullsized":"2024-09-21T12:32:17.227158+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:52.294931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:52.294931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:31.026551+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"76'79","reported_seq":320,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.231118+0000","last_change":"2024-09-21T12:31:08.533342+0000","last_active":"2024-09-21T12:32:16.231118+0000","last_peered":"2024-09-21T12:32:16.231118+0000","last_clean":"2024-09-21T12:32:16.231118+0000","last_became_active":"2024-09-21T12:27:16.933358+0000","last_became_peered":"2024-09-21T12:27:16.933358+0000","last_unstale":"2024-09-21T12:32:16.231118+0000","last_undegraded":"2024-09-21T12:32:16.231118+0000","last_fullsized":"2024-09-21T12:32:16.231118+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:31:08.533297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:31:08.533297+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:28.058118+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":565,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.228568+0000","last_change":"2024-09-21T12:23:51.065780+0000","last_active":"2024-09-21T12:32:16.228568+0000","last_peered":"2024-09-21T12:32:16.228568+0000","last_clean":"2024-09-21T12:32:16.228568+0000","last_became_active":"2024-09-21T12:20:21.312976+0000","last_became_peered":"2024-09-21T12:20:21.312976+0000","last_unstale":"2024-09-21T12:32:16.228568+0000","last_undegraded":"2024-09-21T12:32:16.228568+0000","last_fullsized":"2024-09-21T12:32:16.228568+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:23:51.065733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:51.065733+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:32.158902+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":628,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.513308+0000","last_change":"2024-09-21T12:27:56.597623+0000","last_active":"2024-09-21T12:32:17.513308+0000","last_peered":"2024-09-21T12:32:17.513308+0000","last_clean":"2024-09-21T12:32:17.513308+0000","last_became_active":"2024-09-21T12:22:50.910933+0000","last_became_peered":"2024-09-21T12:22:50.910933+0000","last_unstale":"2024-09-21T12:32:17.513308+0000","last_undegraded":"2024-09-21T12:32:17.513308+0000","last_fullsized":"2024-09-21T12:32:17.513308+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:27:56.597473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:27:56.597473+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:06.028462+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":677,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.228583+0000","last_change":"2024-09-21T12:32:14.255243+0000","last_active":"2024-09-21T12:32:17.228583+0000","last_peered":"2024-09-21T12:32:17.228583+0000","last_clean":"2024-09-21T12:32:17.228583+0000","last_became_active":"2024-09-21T12:32:14.238429+0000","last_became_peered":"2024-09-21T12:32:14.238429+0000","last_unstale":"2024-09-21T12:32:17.228583+0000","last_undegraded":"2024-09-21T12:32:17.228583+0000","last_fullsized":"2024-09-21T12:32:17.228583+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:31:11.543034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:11.543034+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:39.571277+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"335'14","reported_seq":21,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.228933+0000","last_change":"2024-09-21T12:32:14.240726+0000","last_active":"2024-09-21T12:32:16.228933+0000","last_peered":"2024-09-21T12:32:16.228933+0000","last_clean":"2024-09-21T12:32:16.228933+0000","last_became_active":"2024-09-21T12:32:14.236917+0000","last_became_peered":"2024-09-21T12:32:14.236917+0000","last_unstale":"2024-09-21T12:32:16.228933+0000","last_undegraded":"2024-09-21T12:32:16.228933+0000","last_fullsized":"2024-09-21T12:32:16.228933+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:26:10.846183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:10.846183+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"76'79","reported_seq":303,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.832494+0000","last_change":"2024-09-21T12:28:29.446170+0000","last_active":"2024-09-21T12:32:16.832494+0000","last_peered":"2024-09-21T12:32:16.832494+0000","last_clean":"2024-09-21T12:32:16.832494+0000","last_became_active":"2024-09-21T12:27:14.943286+0000","last_became_peered":"2024-09-21T12:27:14.943286+0000","last_unstale":"2024-09-21T12:32:16.832494+0000","last_undegraded":"2024-09-21T12:32:16.832494+0000","last_fullsized":"2024-09-21T12:32:16.832494+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:29.446103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:29.446103+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:10.842450+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":470,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.869491+0000","last_change":"2024-09-21T12:31:56.722328+0000","last_active":"2024-09-21T12:32:16.869491+0000","last_peered":"2024-09-21T12:32:16.869491+0000","last_clean":"2024-09-21T12:32:16.869491+0000","last_became_active":"2024-09-21T12:23:46.910112+0000","last_became_peered":"2024-09-21T12:23:46.910112+0000","last_unstale":"2024-09-21T12:32:16.869491+0000","last_undegraded":"2024-09-21T12:32:16.869491+0000","last_fullsized":"2024-09-21T12:32:16.869491+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:31:56.722241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:56.722241+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:09.661988+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":54,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.224935+0000","last_change":"2024-09-21T12:31:50.039964+0000","last_active":"2024-09-21T12:32:17.224935+0000","last_peered":"2024-09-21T12:32:17.224935+0000","last_clean":"2024-09-21T12:32:17.224935+0000","last_became_active":"2024-09-21T12:31:50.039390+0000","last_became_peered":"2024-09-21T12:31:50.039390+0000","last_unstale":"2024-09-21T12:32:17.224935+0000","last_undegraded":"2024-09-21T12:32:17.224935+0000","last_fullsized":"2024-09-21T12:32:17.224935+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:05.580847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:05.580847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:04.521335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"210'14","reported_seq":337,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.228309+0000","last_change":"2024-09-21T12:26:54.091833+0000","last_active":"2024-09-21T12:32:17.228309+0000","last_peered":"2024-09-21T12:32:17.228309+0000","last_clean":"2024-09-21T12:32:17.228309+0000","last_became_active":"2024-09-21T12:26:54.091451+0000","last_became_peered":"2024-09-21T12:26:54.091451+0000","last_unstale":"2024-09-21T12:32:17.228309+0000","last_undegraded":"2024-09-21T12:32:17.228309+0000","last_fullsized":"2024-09-21T12:32:17.228309+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'13","last_scrub_stamp":"2024-09-21T12:25:33.474391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:33.474391+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:29.516680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":74,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.514425+0000","last_change":"2024-09-21T12:31:39.062941+0000","last_active":"2024-09-21T12:32:17.514425+0000","last_peered":"2024-09-21T12:32:17.514425+0000","last_clean":"2024-09-21T12:32:17.514425+0000","last_became_active":"2024-09-21T12:31:39.054600+0000","last_became_peered":"2024-09-21T12:31:39.054600+0000","last_unstale":"2024-09-21T12:32:17.514425+0000","last_undegraded":"2024-09-21T12:32:17.514425+0000","last_fullsized":"2024-09-21T12:32:17.514425+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:28:31.580245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:31.580245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:09.036420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"76'79","reported_seq":509,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.861799+0000","last_change":"2024-09-21T12:27:20.527090+0000","last_active":"2024-09-21T12:32:16.861799+0000","last_peered":"2024-09-21T12:32:16.861799+0000","last_clean":"2024-09-21T12:32:16.861799+0000","last_became_active":"2024-09-21T12:27:14.941905+0000","last_became_peered":"2024-09-21T12:27:14.941905+0000","last_unstale":"2024-09-21T12:32:16.861799+0000","last_undegraded":"2024-09-21T12:32:16.861799+0000","last_fullsized":"2024-09-21T12:32:16.861799+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:20.526951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:20.526951+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:07.843874+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":690,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.225217+0000","last_change":"2024-09-21T12:32:14.262629+0000","last_active":"2024-09-21T12:32:17.225217+0000","last_peered":"2024-09-21T12:32:17.225217+0000","last_clean":"2024-09-21T12:32:17.225217+0000","last_became_active":"2024-09-21T12:32:14.242826+0000","last_became_peered":"2024-09-21T12:32:14.242826+0000","last_unstale":"2024-09-21T12:32:17.225217+0000","last_undegraded":"2024-09-21T12:32:17.225217+0000","last_fullsized":"2024-09-21T12:32:17.225217+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-21T12:28:26.632272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:26.632272+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":586,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.229014+0000","last_change":"2024-09-21T12:29:10.089053+0000","last_active":"2024-09-21T12:32:17.229014+0000","last_peered":"2024-09-21T12:32:17.229014+0000","last_clean":"2024-09-21T12:32:17.229014+0000","last_became_active":"2024-09-21T12:20:15.297340+0000","last_became_peered":"2024-09-21T12:20:15.297340+0000","last_unstale":"2024-09-21T12:32:17.229014+0000","last_undegraded":"2024-09-21T12:32:17.229014+0000","last_fullsized":"2024-09-21T12:32:17.229014+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:10.088986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:10.088986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":543,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.226314+0000","last_change":"2024-09-21T12:32:14.239263+0000","last_active":"2024-09-21T12:32:17.226314+0000","last_peered":"2024-09-21T12:32:17.226314+0000","last_clean":"2024-09-21T12:32:17.226314+0000","last_became_active":"2024-09-21T12:32:14.238820+0000","last_became_peered":"2024-09-21T12:32:14.238820+0000","last_unstale":"2024-09-21T12:32:17.226314+0000","last_undegraded":"2024-09-21T12:32:17.226314+0000","last_fullsized":"2024-09-21T12:32:17.226314+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:33.697208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:33.697208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":528,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.225930+0000","last_change":"2024-09-21T12:29:05.143941+0000","last_active":"2024-09-21T12:32:17.225930+0000","last_peered":"2024-09-21T12:32:17.225930+0000","last_clean":"2024-09-21T12:32:17.225930+0000","last_became_active":"2024-09-21T12:22:50.918940+0000","last_became_peered":"2024-09-21T12:22:50.918940+0000","last_unstale":"2024-09-21T12:32:17.225930+0000","last_undegraded":"2024-09-21T12:32:17.225930+0000","last_fullsized":"2024-09-21T12:32:17.225930+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:05.143736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:29:05.143736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"76'79","reported_seq":513,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.226971+0000","last_change":"2024-09-21T12:26:45.354836+0000","last_active":"2024-09-21T12:32:17.226971+0000","last_peered":"2024-09-21T12:32:17.226971+0000","last_clean":"2024-09-21T12:32:17.226971+0000","last_became_active":"2024-09-21T12:26:44.960093+0000","last_became_peered":"2024-09-21T12:26:44.960093+0000","last_unstale":"2024-09-21T12:32:17.226971+0000","last_undegraded":"2024-09-21T12:32:17.226971+0000","last_fullsized":"2024-09-21T12:32:17.226971+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:45.354774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:45.354774+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:30.011873+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":604,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.231704+0000","last_change":"2024-09-21T12:26:51.463759+0000","last_active":"2024-09-21T12:32:16.231704+0000","last_peered":"2024-09-21T12:32:16.231704+0000","last_clean":"2024-09-21T12:32:16.231704+0000","last_became_active":"2024-09-21T12:22:50.926008+0000","last_became_peered":"2024-09-21T12:22:50.926008+0000","last_unstale":"2024-09-21T12:32:16.231704+0000","last_undegraded":"2024-09-21T12:32:16.231704+0000","last_fullsized":"2024-09-21T12:32:16.231704+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-21T12:26:51.463698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:51.463698+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:35.628660+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":664,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.228748+0000","last_change":"2024-09-21T12:32:14.247441+0000","last_active":"2024-09-21T12:32:17.228748+0000","last_peered":"2024-09-21T12:32:17.228748+0000","last_clean":"2024-09-21T12:32:17.228748+0000","last_became_active":"2024-09-21T12:32:14.240850+0000","last_became_peered":"2024-09-21T12:32:14.240850+0000","last_unstale":"2024-09-21T12:32:17.228748+0000","last_undegraded":"2024-09-21T12:32:17.228748+0000","last_fullsized":"2024-09-21T12:32:17.228748+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:28:32.727727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:32.727727+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"121'11","reported_seq":764,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.225113+0000","last_change":"2024-09-21T12:26:14.490045+0000","last_active":"2024-09-21T12:32:17.225113+0000","last_peered":"2024-09-21T12:32:17.225113+0000","last_clean":"2024-09-21T12:32:17.225113+0000","last_became_active":"2024-09-21T12:22:50.930342+0000","last_became_peered":"2024-09-21T12:22:50.930342+0000","last_unstale":"2024-09-21T12:32:17.225113+0000","last_undegraded":"2024-09-21T12:32:17.225113+0000","last_fullsized":"2024-09-21T12:32:17.225113+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:26:14.489972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:14.489972+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:21.811951+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":573,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.228379+0000","last_change":"2024-09-21T12:28:37.686403+0000","last_active":"2024-09-21T12:32:16.228379+0000","last_peered":"2024-09-21T12:32:16.228379+0000","last_clean":"2024-09-21T12:32:16.228379+0000","last_became_active":"2024-09-21T12:20:21.593367+0000","last_became_peered":"2024-09-21T12:20:21.593367+0000","last_unstale":"2024-09-21T12:32:16.228379+0000","last_undegraded":"2024-09-21T12:32:16.228379+0000","last_fullsized":"2024-09-21T12:32:16.228379+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:37.686350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:37.686350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"76'79","reported_seq":62,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.228918+0000","last_change":"2024-09-21T12:31:50.320881+0000","last_active":"2024-09-21T12:32:17.228918+0000","last_peered":"2024-09-21T12:32:17.228918+0000","last_clean":"2024-09-21T12:32:17.228918+0000","last_became_active":"2024-09-21T12:31:50.040087+0000","last_became_peered":"2024-09-21T12:31:50.040087+0000","last_unstale":"2024-09-21T12:32:17.228918+0000","last_undegraded":"2024-09-21T12:32:17.228918+0000","last_fullsized":"2024-09-21T12:32:17.228918+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:31:50.320834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:31:50.320834+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:17.807274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":109,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.513315+0000","last_change":"2024-09-21T12:32:02.054999+0000","last_active":"2024-09-21T12:32:17.513315+0000","last_peered":"2024-09-21T12:32:17.513315+0000","last_clean":"2024-09-21T12:32:17.513315+0000","last_became_active":"2024-09-21T12:32:02.037278+0000","last_became_peered":"2024-09-21T12:32:02.037278+0000","last_unstale":"2024-09-21T12:32:17.513315+0000","last_undegraded":"2024-09-21T12:32:17.513315+0000","last_fullsized":"2024-09-21T12:32:17.513315+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:31:59.022007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:59.022007+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:11.299729+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":601,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.232319+0000","last_change":"2024-09-21T12:29:57.398083+0000","last_active":"2024-09-21T12:32:16.232319+0000","last_peered":"2024-09-21T12:32:16.232319+0000","last_clean":"2024-09-21T12:32:16.232319+0000","last_became_active":"2024-09-21T12:20:15.308518+0000","last_became_peered":"2024-09-21T12:20:15.308518+0000","last_unstale":"2024-09-21T12:32:16.232319+0000","last_undegraded":"2024-09-21T12:32:16.232319+0000","last_fullsized":"2024-09-21T12:32:16.232319+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:57.398039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:57.398039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":745,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.227319+0000","last_change":"2024-09-21T12:27:58.871800+0000","last_active":"2024-09-21T12:32:17.227319+0000","last_peered":"2024-09-21T12:32:17.227319+0000","last_clean":"2024-09-21T12:32:17.227319+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:32:17.227319+0000","last_undegraded":"2024-09-21T12:32:17.227319+0000","last_fullsized":"2024-09-21T12:32:17.227319+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:27:58.871710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:27:58.871710+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:08.726110+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":585,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.225315+0000","last_change":"2024-09-21T12:28:37.850166+0000","last_active":"2024-09-21T12:32:17.225315+0000","last_peered":"2024-09-21T12:32:17.225315+0000","last_clean":"2024-09-21T12:32:17.225315+0000","last_became_active":"2024-09-21T12:20:21.315801+0000","last_became_peered":"2024-09-21T12:20:21.315801+0000","last_unstale":"2024-09-21T12:32:17.225315+0000","last_undegraded":"2024-09-21T12:32:17.225315+0000","last_fullsized":"2024-09-21T12:32:17.225315+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:37.850100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:37.850100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"76'79","reported_seq":271,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.228404+0000","last_change":"2024-09-21T12:29:08.081975+0000","last_active":"2024-09-21T12:32:17.228404+0000","last_peered":"2024-09-21T12:32:17.228404+0000","last_clean":"2024-09-21T12:32:17.228404+0000","last_became_active":"2024-09-21T12:29:05.835884+0000","last_became_peered":"2024-09-21T12:29:05.835884+0000","last_unstale":"2024-09-21T12:32:17.228404+0000","last_undegraded":"2024-09-21T12:32:17.228404+0000","last_fullsized":"2024-09-21T12:32:17.228404+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:08.081901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:08.081901+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":636,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.228111+0000","last_change":"2024-09-21T12:26:14.834328+0000","last_active":"2024-09-21T12:32:17.228111+0000","last_peered":"2024-09-21T12:32:17.228111+0000","last_clean":"2024-09-21T12:32:17.228111+0000","last_became_active":"2024-09-21T12:20:17.693067+0000","last_became_peered":"2024-09-21T12:20:17.693067+0000","last_unstale":"2024-09-21T12:32:17.228111+0000","last_undegraded":"2024-09-21T12:32:17.228111+0000","last_fullsized":"2024-09-21T12:32:17.228111+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-21T12:26:14.834236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:14.834236+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:37.533965+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":579,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.513963+0000","last_change":"2024-09-21T12:26:12.131784+0000","last_active":"2024-09-21T12:32:17.513963+0000","last_peered":"2024-09-21T12:32:17.513963+0000","last_clean":"2024-09-21T12:32:17.513963+0000","last_became_active":"2024-09-21T12:20:15.308441+0000","last_became_peered":"2024-09-21T12:20:15.308441+0000","last_unstale":"2024-09-21T12:32:17.513963+0000","last_undegraded":"2024-09-21T12:32:17.513963+0000","last_fullsized":"2024-09-21T12:32:17.513963+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:12.131721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:12.131721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:34.057507+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":175,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.226012+0000","last_change":"2024-09-21T12:29:05.826167+0000","last_active":"2024-09-21T12:32:17.226012+0000","last_peered":"2024-09-21T12:32:17.226012+0000","last_clean":"2024-09-21T12:32:17.226012+0000","last_became_active":"2024-09-21T12:29:05.825869+0000","last_became_peered":"2024-09-21T12:29:05.825869+0000","last_unstale":"2024-09-21T12:32:17.226012+0000","last_undegraded":"2024-09-21T12:32:17.226012+0000","last_fullsized":"2024-09-21T12:32:17.226012+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:48.999878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:48.999878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":575,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.513165+0000","last_change":"2024-09-21T12:31:11.827090+0000","last_active":"2024-09-21T12:32:17.513165+0000","last_peered":"2024-09-21T12:32:17.513165+0000","last_clean":"2024-09-21T12:32:17.513165+0000","last_became_active":"2024-09-21T12:20:21.592828+0000","last_became_peered":"2024-09-21T12:20:21.592828+0000","last_unstale":"2024-09-21T12:32:17.513165+0000","last_undegraded":"2024-09-21T12:32:17.513165+0000","last_fullsized":"2024-09-21T12:32:17.513165+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:11.827011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:31:11.827011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:33.068131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"76'79","reported_seq":379,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.226657+0000","last_change":"2024-09-21T12:28:25.714131+0000","last_active":"2024-09-21T12:32:17.226657+0000","last_peered":"2024-09-21T12:32:17.226657+0000","last_clean":"2024-09-21T12:32:17.226657+0000","last_became_active":"2024-09-21T12:26:54.092726+0000","last_became_peered":"2024-09-21T12:26:54.092726+0000","last_unstale":"2024-09-21T12:32:17.226657+0000","last_undegraded":"2024-09-21T12:32:17.226657+0000","last_fullsized":"2024-09-21T12:32:17.226657+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:25.714084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:25.714084+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"276'40","reported_seq":786,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.513878+0000","last_change":"2024-09-21T12:26:09.082395+0000","last_active":"2024-09-21T12:32:17.513878+0000","last_peered":"2024-09-21T12:32:17.513878+0000","last_clean":"2024-09-21T12:32:17.513878+0000","last_became_active":"2024-09-21T12:20:17.307769+0000","last_became_peered":"2024-09-21T12:20:17.307769+0000","last_unstale":"2024-09-21T12:32:17.513878+0000","last_undegraded":"2024-09-21T12:32:17.513878+0000","last_fullsized":"2024-09-21T12:32:17.513878+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-21T12:26:09.082314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:09.082314+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:25.018714+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":521,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.226827+0000","last_change":"2024-09-21T12:28:31.738463+0000","last_active":"2024-09-21T12:32:17.226827+0000","last_peered":"2024-09-21T12:32:17.226827+0000","last_clean":"2024-09-21T12:32:17.226827+0000","last_became_active":"2024-09-21T12:22:50.922002+0000","last_became_peered":"2024-09-21T12:22:50.922002+0000","last_unstale":"2024-09-21T12:32:17.226827+0000","last_undegraded":"2024-09-21T12:32:17.226827+0000","last_fullsized":"2024-09-21T12:32:17.226827+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:31.738415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:31.738415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":305,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.513451+0000","last_change":"2024-09-21T12:28:37.691938+0000","last_active":"2024-09-21T12:32:17.513451+0000","last_peered":"2024-09-21T12:32:17.513451+0000","last_clean":"2024-09-21T12:32:17.513451+0000","last_became_active":"2024-09-21T12:26:54.083584+0000","last_became_peered":"2024-09-21T12:26:54.083584+0000","last_unstale":"2024-09-21T12:32:17.513451+0000","last_undegraded":"2024-09-21T12:32:17.513451+0000","last_fullsized":"2024-09-21T12:32:17.513451+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T12:28:37.691868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:28:37.691868+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":618,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.227924+0000","last_change":"2024-09-21T12:31:49.662004+0000","last_active":"2024-09-21T12:32:17.227924+0000","last_peered":"2024-09-21T12:32:17.227924+0000","last_clean":"2024-09-21T12:32:17.227924+0000","last_became_active":"2024-09-21T12:22:50.921488+0000","last_became_peered":"2024-09-21T12:22:50.921488+0000","last_unstale":"2024-09-21T12:32:17.227924+0000","last_undegraded":"2024-09-21T12:32:17.227924+0000","last_fullsized":"2024-09-21T12:32:17.227924+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:49.661957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:31:49.661957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:07.929316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":595,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.228306+0000","last_change":"2024-09-21T12:26:10.435410+0000","last_active":"2024-09-21T12:32:17.228306+0000","last_peered":"2024-09-21T12:32:17.228306+0000","last_clean":"2024-09-21T12:32:17.228306+0000","last_became_active":"2024-09-21T12:22:50.917558+0000","last_became_peered":"2024-09-21T12:22:50.917558+0000","last_unstale":"2024-09-21T12:32:17.228306+0000","last_undegraded":"2024-09-21T12:32:17.228306+0000","last_fullsized":"2024-09-21T12:32:17.228306+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:10.435349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:10.435349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:20.826209+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'79","reported_seq":16,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.227891+0000","last_change":"2024-09-21T12:32:14.238048+0000","last_active":"2024-09-21T12:32:16.227891+0000","last_peered":"2024-09-21T12:32:16.227891+0000","last_clean":"2024-09-21T12:32:16.227891+0000","last_became_active":"2024-09-21T12:32:14.237652+0000","last_became_peered":"2024-09-21T12:32:14.237652+0000","last_unstale":"2024-09-21T12:32:16.227891+0000","last_undegraded":"2024-09-21T12:32:16.227891+0000","last_fullsized":"2024-09-21T12:32:16.227891+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:30:39.179014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:30:39.179014+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":82,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.514341+0000","last_change":"2024-09-21T12:31:39.072454+0000","last_active":"2024-09-21T12:32:17.514341+0000","last_peered":"2024-09-21T12:32:17.514341+0000","last_clean":"2024-09-21T12:32:17.514341+0000","last_became_active":"2024-09-21T12:31:39.057316+0000","last_became_peered":"2024-09-21T12:31:39.057316+0000","last_unstale":"2024-09-21T12:32:17.514341+0000","last_undegraded":"2024-09-21T12:32:17.514341+0000","last_fullsized":"2024-09-21T12:32:17.514341+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-21T12:28:25.549320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:25.549320+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:10.042213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":603,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.512957+0000","last_change":"2024-09-21T12:28:39.705480+0000","last_active":"2024-09-21T12:32:17.512957+0000","last_peered":"2024-09-21T12:32:17.512957+0000","last_clean":"2024-09-21T12:32:17.512957+0000","last_became_active":"2024-09-21T12:20:15.308356+0000","last_became_peered":"2024-09-21T12:20:15.308356+0000","last_unstale":"2024-09-21T12:32:17.512957+0000","last_undegraded":"2024-09-21T12:32:17.512957+0000","last_fullsized":"2024-09-21T12:32:17.512957+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:39.705175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:39.705175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":611,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.832590+0000","last_change":"2024-09-21T12:28:43.390160+0000","last_active":"2024-09-21T12:32:16.832590+0000","last_peered":"2024-09-21T12:32:16.832590+0000","last_clean":"2024-09-21T12:32:16.832590+0000","last_became_active":"2024-09-21T12:20:14.288358+0000","last_became_peered":"2024-09-21T12:20:14.288358+0000","last_unstale":"2024-09-21T12:32:16.832590+0000","last_undegraded":"2024-09-21T12:32:16.832590+0000","last_fullsized":"2024-09-21T12:32:16.832590+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:28:43.390087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:28:43.390087+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0019042449999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":594,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.862099+0000","last_change":"2024-09-21T12:29:13.200503+0000","last_active":"2024-09-21T12:32:16.862099+0000","last_peered":"2024-09-21T12:32:16.862099+0000","last_clean":"2024-09-21T12:32:16.862099+0000","last_became_active":"2024-09-21T12:20:19.309724+0000","last_became_peered":"2024-09-21T12:20:19.309724+0000","last_unstale":"2024-09-21T12:32:16.862099+0000","last_undegraded":"2024-09-21T12:32:16.862099+0000","last_fullsized":"2024-09-21T12:32:16.862099+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:13.200321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:29:13.200321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":565,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.228788+0000","last_change":"2024-09-21T12:27:33.395611+0000","last_active":"2024-09-21T12:32:16.228788+0000","last_peered":"2024-09-21T12:32:16.228788+0000","last_clean":"2024-09-21T12:32:16.228788+0000","last_became_active":"2024-09-21T12:20:21.592982+0000","last_became_peered":"2024-09-21T12:20:21.592982+0000","last_unstale":"2024-09-21T12:32:16.228788+0000","last_undegraded":"2024-09-21T12:32:16.228788+0000","last_fullsized":"2024-09-21T12:32:16.228788+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:33.395568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:27:33.395568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"76'79","reported_seq":537,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.513586+0000","last_change":"2024-09-21T12:29:11.678268+0000","last_active":"2024-09-21T12:32:17.513586+0000","last_peered":"2024-09-21T12:32:17.513586+0000","last_clean":"2024-09-21T12:32:17.513586+0000","last_became_active":"2024-09-21T12:26:44.949867+0000","last_became_peered":"2024-09-21T12:26:44.949867+0000","last_unstale":"2024-09-21T12:32:17.513586+0000","last_undegraded":"2024-09-21T12:32:17.513586+0000","last_fullsized":"2024-09-21T12:32:17.513586+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:11.677923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:11.677923+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":629,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.228634+0000","last_change":"2024-09-21T12:28:45.783033+0000","last_active":"2024-09-21T12:32:17.228634+0000","last_peered":"2024-09-21T12:32:17.228634+0000","last_clean":"2024-09-21T12:32:17.228634+0000","last_became_active":"2024-09-21T12:20:17.299325+0000","last_became_peered":"2024-09-21T12:20:17.299325+0000","last_unstale":"2024-09-21T12:32:17.228634+0000","last_undegraded":"2024-09-21T12:32:17.228634+0000","last_fullsized":"2024-09-21T12:32:17.228634+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:28:45.782955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.782955+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":632,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.226013+0000","last_change":"2024-09-21T12:31:50.041678+0000","last_active":"2024-09-21T12:32:17.226013+0000","last_peered":"2024-09-21T12:32:17.226013+0000","last_clean":"2024-09-21T12:32:17.226013+0000","last_became_active":"2024-09-21T12:31:50.041367+0000","last_became_peered":"2024-09-21T12:31:50.041367+0000","last_unstale":"2024-09-21T12:32:17.226013+0000","last_undegraded":"2024-09-21T12:32:17.226013+0000","last_fullsized":"2024-09-21T12:32:17.226013+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:15.115449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:15.115449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":1320,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.228024+0000","last_change":"2024-09-21T12:28:34.639438+0000","last_active":"2024-09-21T12:32:16.228024+0000","last_peered":"2024-09-21T12:32:16.228024+0000","last_clean":"2024-09-21T12:32:16.228024+0000","last_became_active":"2024-09-21T12:20:14.287543+0000","last_became_peered":"2024-09-21T12:20:14.287543+0000","last_unstale":"2024-09-21T12:32:16.228024+0000","last_undegraded":"2024-09-21T12:32:16.228024+0000","last_fullsized":"2024-09-21T12:32:16.228024+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-21T12:28:34.639361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:28:34.639361+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":715,"num_read_kb":710,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":32,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.231918+0000","last_change":"2024-09-21T12:32:02.030536+0000","last_active":"2024-09-21T12:32:16.231918+0000","last_peered":"2024-09-21T12:32:16.231918+0000","last_clean":"2024-09-21T12:32:16.231918+0000","last_became_active":"2024-09-21T12:32:02.030240+0000","last_became_peered":"2024-09-21T12:32:02.030240+0000","last_unstale":"2024-09-21T12:32:16.231918+0000","last_undegraded":"2024-09-21T12:32:16.231918+0000","last_fullsized":"2024-09-21T12:32:16.231918+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:11.422293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:11.422293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:12.691461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":70,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.513468+0000","last_change":"2024-09-21T12:31:39.060916+0000","last_active":"2024-09-21T12:32:17.513468+0000","last_peered":"2024-09-21T12:32:17.513468+0000","last_clean":"2024-09-21T12:32:17.513468+0000","last_became_active":"2024-09-21T12:31:39.060160+0000","last_became_peered":"2024-09-21T12:31:39.060160+0000","last_unstale":"2024-09-21T12:32:17.513468+0000","last_undegraded":"2024-09-21T12:32:17.513468+0000","last_fullsized":"2024-09-21T12:32:17.513468+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:30:40.556857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:30:40.556857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"276'32","reported_seq":282,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.229175+0000","last_change":"2024-09-21T12:29:09.625844+0000","last_active":"2024-09-21T12:32:16.229175+0000","last_peered":"2024-09-21T12:32:16.229175+0000","last_clean":"2024-09-21T12:32:16.229175+0000","last_became_active":"2024-09-21T12:29:05.827560+0000","last_became_peered":"2024-09-21T12:29:05.827560+0000","last_unstale":"2024-09-21T12:32:16.229175+0000","last_undegraded":"2024-09-21T12:32:16.229175+0000","last_fullsized":"2024-09-21T12:32:16.229175+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'20","last_scrub_stamp":"2024-09-21T12:29:09.625763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:29:09.625763+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":592,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.226190+0000","last_change":"2024-09-21T12:26:12.394327+0000","last_active":"2024-09-21T12:32:17.226190+0000","last_peered":"2024-09-21T12:32:17.226190+0000","last_clean":"2024-09-21T12:32:17.226190+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:32:17.226190+0000","last_undegraded":"2024-09-21T12:32:17.226190+0000","last_fullsized":"2024-09-21T12:32:17.226190+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:26:12.394247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:12.394247+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:18.778920+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":670,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.228115+0000","last_change":"2024-09-21T12:26:49.248015+0000","last_active":"2024-09-21T12:32:16.228115+0000","last_peered":"2024-09-21T12:32:16.228115+0000","last_clean":"2024-09-21T12:32:16.228115+0000","last_became_active":"2024-09-21T12:26:44.950132+0000","last_became_peered":"2024-09-21T12:26:44.950132+0000","last_unstale":"2024-09-21T12:32:16.228115+0000","last_undegraded":"2024-09-21T12:32:16.228115+0000","last_fullsized":"2024-09-21T12:32:16.228115+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:49.247952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:49.247952+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:02.161443+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":16,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.228292+0000","last_change":"2024-09-21T12:32:14.233428+0000","last_active":"2024-09-21T12:32:16.228292+0000","last_peered":"2024-09-21T12:32:16.228292+0000","last_clean":"2024-09-21T12:32:16.228292+0000","last_became_active":"2024-09-21T12:32:14.233034+0000","last_became_peered":"2024-09-21T12:32:14.233034+0000","last_unstale":"2024-09-21T12:32:16.228292+0000","last_undegraded":"2024-09-21T12:32:16.228292+0000","last_fullsized":"2024-09-21T12:32:16.228292+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:36.684459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:36.684459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":435,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.228515+0000","last_change":"2024-09-21T12:31:39.057057+0000","last_active":"2024-09-21T12:32:17.228515+0000","last_peered":"2024-09-21T12:32:17.228515+0000","last_clean":"2024-09-21T12:32:17.228515+0000","last_became_active":"2024-09-21T12:31:39.055947+0000","last_became_peered":"2024-09-21T12:31:39.055947+0000","last_unstale":"2024-09-21T12:32:17.228515+0000","last_undegraded":"2024-09-21T12:32:17.228515+0000","last_fullsized":"2024-09-21T12:32:17.228515+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:28.647287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:28.647287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"76'79","reported_seq":541,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.513768+0000","last_change":"2024-09-21T12:31:03.884522+0000","last_active":"2024-09-21T12:32:17.513768+0000","last_peered":"2024-09-21T12:32:17.513768+0000","last_clean":"2024-09-21T12:32:17.513768+0000","last_became_active":"2024-09-21T12:27:14.954351+0000","last_became_peered":"2024-09-21T12:27:14.954351+0000","last_unstale":"2024-09-21T12:32:17.513768+0000","last_undegraded":"2024-09-21T12:32:17.513768+0000","last_fullsized":"2024-09-21T12:32:17.513768+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:31:03.884464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:31:03.884464+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:24.047755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":90,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.514240+0000","last_change":"2024-09-21T12:32:00.043176+0000","last_active":"2024-09-21T12:32:17.514240+0000","last_peered":"2024-09-21T12:32:17.514240+0000","last_clean":"2024-09-21T12:32:17.514240+0000","last_became_active":"2024-09-21T12:31:39.052759+0000","last_became_peered":"2024-09-21T12:31:39.052759+0000","last_unstale":"2024-09-21T12:32:17.514240+0000","last_undegraded":"2024-09-21T12:32:17.514240+0000","last_fullsized":"2024-09-21T12:32:17.514240+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:32:00.043108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:00.043108+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:01.713101+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":509,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.832514+0000","last_change":"2024-09-21T12:27:30.501480+0000","last_active":"2024-09-21T12:32:16.832514+0000","last_peered":"2024-09-21T12:32:16.832514+0000","last_clean":"2024-09-21T12:32:16.832514+0000","last_became_active":"2024-09-21T12:22:50.908060+0000","last_became_peered":"2024-09-21T12:22:50.908060+0000","last_unstale":"2024-09-21T12:32:16.832514+0000","last_undegraded":"2024-09-21T12:32:16.832514+0000","last_fullsized":"2024-09-21T12:32:16.832514+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:30.498831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:27:30.498831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:06.870075+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":329,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.869487+0000","last_change":"2024-09-21T12:29:15.275262+0000","last_active":"2024-09-21T12:32:16.869487+0000","last_peered":"2024-09-21T12:32:16.869487+0000","last_clean":"2024-09-21T12:32:16.869487+0000","last_became_active":"2024-09-21T12:29:05.825426+0000","last_became_peered":"2024-09-21T12:29:05.825426+0000","last_unstale":"2024-09-21T12:32:16.869487+0000","last_undegraded":"2024-09-21T12:32:16.869487+0000","last_fullsized":"2024-09-21T12:32:16.869487+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:15.275189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:29:15.275189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":574,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.227071+0000","last_change":"2024-09-21T12:28:39.853931+0000","last_active":"2024-09-21T12:32:17.227071+0000","last_peered":"2024-09-21T12:32:17.227071+0000","last_clean":"2024-09-21T12:32:17.227071+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:32:17.227071+0000","last_undegraded":"2024-09-21T12:32:17.227071+0000","last_fullsized":"2024-09-21T12:32:17.227071+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:39.853791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:39.853791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"76'79","reported_seq":309,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.832318+0000","last_change":"2024-09-21T12:32:02.027640+0000","last_active":"2024-09-21T12:32:16.832318+0000","last_peered":"2024-09-21T12:32:16.832318+0000","last_clean":"2024-09-21T12:32:16.832318+0000","last_became_active":"2024-09-21T12:32:02.026943+0000","last_became_peered":"2024-09-21T12:32:02.026943+0000","last_unstale":"2024-09-21T12:32:16.832318+0000","last_undegraded":"2024-09-21T12:32:16.832318+0000","last_fullsized":"2024-09-21T12:32:16.832318+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:30:38.611133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:30:38.611133+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":663,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.861910+0000","last_change":"2024-09-21T12:28:46.355910+0000","last_active":"2024-09-21T12:32:16.861910+0000","last_peered":"2024-09-21T12:32:16.861910+0000","last_clean":"2024-09-21T12:32:16.861910+0000","last_became_active":"2024-09-21T12:23:46.897197+0000","last_became_peered":"2024-09-21T12:23:46.897197+0000","last_unstale":"2024-09-21T12:32:16.861910+0000","last_undegraded":"2024-09-21T12:32:16.861910+0000","last_fullsized":"2024-09-21T12:32:16.861910+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-21T12:28:46.355822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:46.355822+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:11.864750+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":70,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.513362+0000","last_change":"2024-09-21T12:31:39.060456+0000","last_active":"2024-09-21T12:32:17.513362+0000","last_peered":"2024-09-21T12:32:17.513362+0000","last_clean":"2024-09-21T12:32:17.513362+0000","last_became_active":"2024-09-21T12:31:39.059854+0000","last_became_peered":"2024-09-21T12:31:39.059854+0000","last_unstale":"2024-09-21T12:32:17.513362+0000","last_undegraded":"2024-09-21T12:32:17.513362+0000","last_fullsized":"2024-09-21T12:32:17.513362+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:06.236749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:06.236749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:08.037262+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":600,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.513655+0000","last_change":"2024-09-21T12:28:03.571042+0000","last_active":"2024-09-21T12:32:17.513655+0000","last_peered":"2024-09-21T12:32:17.513655+0000","last_clean":"2024-09-21T12:32:17.513655+0000","last_became_active":"2024-09-21T12:22:50.927438+0000","last_became_peered":"2024-09-21T12:22:50.927438+0000","last_unstale":"2024-09-21T12:32:17.513655+0000","last_undegraded":"2024-09-21T12:32:17.513655+0000","last_fullsized":"2024-09-21T12:32:17.513655+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:03.570984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:03.570984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:00.985882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":587,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.869626+0000","last_change":"2024-09-21T12:25:39.079146+0000","last_active":"2024-09-21T12:32:16.869626+0000","last_peered":"2024-09-21T12:32:16.869626+0000","last_clean":"2024-09-21T12:32:16.869626+0000","last_became_active":"2024-09-21T12:22:50.913898+0000","last_became_peered":"2024-09-21T12:22:50.913898+0000","last_unstale":"2024-09-21T12:32:16.869626+0000","last_undegraded":"2024-09-21T12:32:16.869626+0000","last_fullsized":"2024-09-21T12:32:16.869626+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:39.079015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:39.079015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:29.619606+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"76'79","reported_seq":667,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.228103+0000","last_change":"2024-09-21T12:29:05.501052+0000","last_active":"2024-09-21T12:32:16.228103+0000","last_peered":"2024-09-21T12:32:16.228103+0000","last_clean":"2024-09-21T12:32:16.228103+0000","last_became_active":"2024-09-21T12:27:14.943705+0000","last_became_peered":"2024-09-21T12:27:14.943705+0000","last_unstale":"2024-09-21T12:32:16.228103+0000","last_undegraded":"2024-09-21T12:32:16.228103+0000","last_fullsized":"2024-09-21T12:32:16.228103+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:05.500994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:05.500994+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":471,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.832716+0000","last_change":"2024-09-21T12:29:14.222764+0000","last_active":"2024-09-21T12:32:16.832716+0000","last_peered":"2024-09-21T12:32:16.832716+0000","last_clean":"2024-09-21T12:32:16.832716+0000","last_became_active":"2024-09-21T12:23:46.902260+0000","last_became_peered":"2024-09-21T12:23:46.902260+0000","last_unstale":"2024-09-21T12:32:16.832716+0000","last_undegraded":"2024-09-21T12:32:16.832716+0000","last_fullsized":"2024-09-21T12:32:16.832716+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:29:14.222681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:29:14.222681+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":615,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.226825+0000","last_change":"2024-09-21T12:31:47.248454+0000","last_active":"2024-09-21T12:32:17.226825+0000","last_peered":"2024-09-21T12:32:17.226825+0000","last_clean":"2024-09-21T12:32:17.226825+0000","last_became_active":"2024-09-21T12:22:50.927946+0000","last_became_peered":"2024-09-21T12:22:50.927946+0000","last_unstale":"2024-09-21T12:32:17.226825+0000","last_undegraded":"2024-09-21T12:32:17.226825+0000","last_fullsized":"2024-09-21T12:32:17.226825+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:47.248389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:31:47.248389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:03.989135+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":747,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.861881+0000","last_change":"2024-09-21T12:28:39.418167+0000","last_active":"2024-09-21T12:32:16.861881+0000","last_peered":"2024-09-21T12:32:16.861881+0000","last_clean":"2024-09-21T12:32:16.861881+0000","last_became_active":"2024-09-21T12:20:19.307924+0000","last_became_peered":"2024-09-21T12:20:19.307924+0000","last_unstale":"2024-09-21T12:32:16.861881+0000","last_undegraded":"2024-09-21T12:32:16.861881+0000","last_fullsized":"2024-09-21T12:32:16.861881+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:28:39.418085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:39.418085+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":77,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.514540+0000","last_change":"2024-09-21T12:32:02.030976+0000","last_active":"2024-09-21T12:32:17.514540+0000","last_peered":"2024-09-21T12:32:17.514540+0000","last_clean":"2024-09-21T12:32:17.514540+0000","last_became_active":"2024-09-21T12:32:02.030553+0000","last_became_peered":"2024-09-21T12:32:02.030553+0000","last_unstale":"2024-09-21T12:32:17.514540+0000","last_undegraded":"2024-09-21T12:32:17.514540+0000","last_fullsized":"2024-09-21T12:32:17.514540+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:35.669942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:35.669942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:11.048104+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"76'79","reported_seq":298,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.228495+0000","last_change":"2024-09-21T12:27:21.345414+0000","last_active":"2024-09-21T12:32:16.228495+0000","last_peered":"2024-09-21T12:32:16.228495+0000","last_clean":"2024-09-21T12:32:16.228495+0000","last_became_active":"2024-09-21T12:27:14.943330+0000","last_became_peered":"2024-09-21T12:27:14.943330+0000","last_unstale":"2024-09-21T12:32:16.228495+0000","last_undegraded":"2024-09-21T12:32:16.228495+0000","last_fullsized":"2024-09-21T12:32:16.228495+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:21.345364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:21.345364+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:08.135730+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":669,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.227211+0000","last_change":"2024-09-21T12:25:00.237083+0000","last_active":"2024-09-21T12:32:17.227211+0000","last_peered":"2024-09-21T12:32:17.227211+0000","last_clean":"2024-09-21T12:32:17.227211+0000","last_became_active":"2024-09-21T12:22:50.922343+0000","last_became_peered":"2024-09-21T12:22:50.922343+0000","last_unstale":"2024-09-21T12:32:17.227211+0000","last_undegraded":"2024-09-21T12:32:17.227211+0000","last_fullsized":"2024-09-21T12:32:17.227211+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:25:00.236990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:00.236990+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:28.520541+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":278,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.227756+0000","last_change":"2024-09-21T12:31:12.884054+0000","last_active":"2024-09-21T12:32:17.227756+0000","last_peered":"2024-09-21T12:32:17.227756+0000","last_clean":"2024-09-21T12:32:17.227756+0000","last_became_active":"2024-09-21T12:26:54.084228+0000","last_became_peered":"2024-09-21T12:26:54.084228+0000","last_unstale":"2024-09-21T12:32:17.227756+0000","last_undegraded":"2024-09-21T12:32:17.227756+0000","last_fullsized":"2024-09-21T12:32:17.227756+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:12.883983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:31:12.883983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":615,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.832692+0000","last_change":"2024-09-21T12:28:47.361181+0000","last_active":"2024-09-21T12:32:16.832692+0000","last_peered":"2024-09-21T12:32:16.832692+0000","last_clean":"2024-09-21T12:32:16.832692+0000","last_became_active":"2024-09-21T12:22:50.925974+0000","last_became_peered":"2024-09-21T12:22:50.925974+0000","last_unstale":"2024-09-21T12:32:16.832692+0000","last_undegraded":"2024-09-21T12:32:16.832692+0000","last_fullsized":"2024-09-21T12:32:16.832692+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:47.361115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:47.361115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:13.824334+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":577,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.226121+0000","last_change":"2024-09-21T12:28:43.821102+0000","last_active":"2024-09-21T12:32:17.226121+0000","last_peered":"2024-09-21T12:32:17.226121+0000","last_clean":"2024-09-21T12:32:17.226121+0000","last_became_active":"2024-09-21T12:20:21.311493+0000","last_became_peered":"2024-09-21T12:20:21.311493+0000","last_unstale":"2024-09-21T12:32:17.226121+0000","last_undegraded":"2024-09-21T12:32:17.226121+0000","last_fullsized":"2024-09-21T12:32:17.226121+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:43.820979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:43.820979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"76'79","reported_seq":508,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.232241+0000","last_change":"2024-09-21T12:31:10.616302+0000","last_active":"2024-09-21T12:32:16.232241+0000","last_peered":"2024-09-21T12:32:16.232241+0000","last_clean":"2024-09-21T12:32:16.232241+0000","last_became_active":"2024-09-21T12:29:05.842070+0000","last_became_peered":"2024-09-21T12:29:05.842070+0000","last_unstale":"2024-09-21T12:32:16.232241+0000","last_undegraded":"2024-09-21T12:32:16.232241+0000","last_fullsized":"2024-09-21T12:32:16.232241+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:31:10.616253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:31:10.616253+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:24.519004+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":706,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.685406+0000","last_change":"2024-09-21T12:32:14.251878+0000","last_active":"2024-09-21T12:32:16.685406+0000","last_peered":"2024-09-21T12:32:16.685406+0000","last_clean":"2024-09-21T12:32:16.685406+0000","last_became_active":"2024-09-21T12:32:14.236905+0000","last_became_peered":"2024-09-21T12:32:14.236905+0000","last_unstale":"2024-09-21T12:32:16.685406+0000","last_undegraded":"2024-09-21T12:32:16.685406+0000","last_fullsized":"2024-09-21T12:32:16.685406+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'35","last_scrub_stamp":"2024-09-21T12:31:08.327617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:08.327617+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:26.744405+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":599,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.229250+0000","last_change":"2024-09-21T12:32:02.026673+0000","last_active":"2024-09-21T12:32:16.229250+0000","last_peered":"2024-09-21T12:32:16.229250+0000","last_clean":"2024-09-21T12:32:16.229250+0000","last_became_active":"2024-09-21T12:32:02.025668+0000","last_became_peered":"2024-09-21T12:32:02.025668+0000","last_unstale":"2024-09-21T12:32:16.229250+0000","last_undegraded":"2024-09-21T12:32:16.229250+0000","last_fullsized":"2024-09-21T12:32:16.229250+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:48.438755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:48.438755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":600,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.832743+0000","last_change":"2024-09-21T12:32:02.043989+0000","last_active":"2024-09-21T12:32:16.832743+0000","last_peered":"2024-09-21T12:32:16.832743+0000","last_clean":"2024-09-21T12:32:16.832743+0000","last_became_active":"2024-09-21T12:32:02.043597+0000","last_became_peered":"2024-09-21T12:32:02.043597+0000","last_unstale":"2024-09-21T12:32:16.832743+0000","last_undegraded":"2024-09-21T12:32:16.832743+0000","last_fullsized":"2024-09-21T12:32:16.832743+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:39.732474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:31:39.732474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:53.917152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":565,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.227977+0000","last_change":"2024-09-21T12:29:15.641929+0000","last_active":"2024-09-21T12:32:16.227977+0000","last_peered":"2024-09-21T12:32:16.227977+0000","last_clean":"2024-09-21T12:32:16.227977+0000","last_became_active":"2024-09-21T12:20:21.314545+0000","last_became_peered":"2024-09-21T12:20:21.314545+0000","last_unstale":"2024-09-21T12:32:16.227977+0000","last_undegraded":"2024-09-21T12:32:16.227977+0000","last_fullsized":"2024-09-21T12:32:16.227977+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:15.641881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:29:15.641881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"76'79","reported_seq":540,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.227376+0000","last_change":"2024-09-21T12:27:17.158443+0000","last_active":"2024-09-21T12:32:17.227376+0000","last_peered":"2024-09-21T12:32:17.227376+0000","last_clean":"2024-09-21T12:32:17.227376+0000","last_became_active":"2024-09-21T12:27:14.951375+0000","last_became_peered":"2024-09-21T12:27:14.951375+0000","last_unstale":"2024-09-21T12:32:17.227376+0000","last_undegraded":"2024-09-21T12:32:17.227376+0000","last_fullsized":"2024-09-21T12:32:17.227376+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:17.158380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:17.158380+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:34.281598+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":640,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.513305+0000","last_change":"2024-09-21T12:31:50.054697+0000","last_active":"2024-09-21T12:32:17.513305+0000","last_peered":"2024-09-21T12:32:17.513305+0000","last_clean":"2024-09-21T12:32:17.513305+0000","last_became_active":"2024-09-21T12:31:50.040103+0000","last_became_peered":"2024-09-21T12:31:50.040103+0000","last_unstale":"2024-09-21T12:32:17.513305+0000","last_undegraded":"2024-09-21T12:32:17.513305+0000","last_fullsized":"2024-09-21T12:32:17.513305+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:28:40.694138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:40.694138+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":614,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.229166+0000","last_change":"2024-09-21T12:28:30.612213+0000","last_active":"2024-09-21T12:32:16.229166+0000","last_peered":"2024-09-21T12:32:16.229166+0000","last_clean":"2024-09-21T12:32:16.229166+0000","last_became_active":"2024-09-21T12:22:50.927859+0000","last_became_peered":"2024-09-21T12:22:50.927859+0000","last_unstale":"2024-09-21T12:32:16.229166+0000","last_undegraded":"2024-09-21T12:32:16.229166+0000","last_fullsized":"2024-09-21T12:32:16.229166+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:30.612156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:30.612156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":594,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.832492+0000","last_change":"2024-09-21T12:29:11.218769+0000","last_active":"2024-09-21T12:32:16.832492+0000","last_peered":"2024-09-21T12:32:16.832492+0000","last_clean":"2024-09-21T12:32:16.832492+0000","last_became_active":"2024-09-21T12:20:19.312572+0000","last_became_peered":"2024-09-21T12:20:19.312572+0000","last_unstale":"2024-09-21T12:32:16.832492+0000","last_undegraded":"2024-09-21T12:32:16.832492+0000","last_fullsized":"2024-09-21T12:32:16.832492+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:11.218334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:29:11.218334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":596,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.226529+0000","last_change":"2024-09-21T12:32:15.555334+0000","last_active":"2024-09-21T12:32:17.226529+0000","last_peered":"2024-09-21T12:32:17.226529+0000","last_clean":"2024-09-21T12:32:17.226529+0000","last_became_active":"2024-09-21T12:32:14.242180+0000","last_became_peered":"2024-09-21T12:32:14.242180+0000","last_unstale":"2024-09-21T12:32:17.226529+0000","last_undegraded":"2024-09-21T12:32:17.226529+0000","last_fullsized":"2024-09-21T12:32:17.226529+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:15.555279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:32:15.555279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:42.589957+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"76'79","reported_seq":16,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.228338+0000","last_change":"2024-09-21T12:32:14.234357+0000","last_active":"2024-09-21T12:32:16.228338+0000","last_peered":"2024-09-21T12:32:16.228338+0000","last_clean":"2024-09-21T12:32:16.228338+0000","last_became_active":"2024-09-21T12:32:14.233937+0000","last_became_peered":"2024-09-21T12:32:14.233937+0000","last_unstale":"2024-09-21T12:32:16.228338+0000","last_undegraded":"2024-09-21T12:32:16.228338+0000","last_fullsized":"2024-09-21T12:32:16.228338+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:36.700968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:36.700968+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":71,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.230945+0000","last_change":"2024-09-21T12:32:14.250281+0000","last_active":"2024-09-21T12:32:16.230945+0000","last_peered":"2024-09-21T12:32:16.230945+0000","last_clean":"2024-09-21T12:32:16.230945+0000","last_became_active":"2024-09-21T12:32:14.234897+0000","last_became_peered":"2024-09-21T12:32:14.234897+0000","last_unstale":"2024-09-21T12:32:16.230945+0000","last_undegraded":"2024-09-21T12:32:16.230945+0000","last_fullsized":"2024-09-21T12:32:16.230945+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:31:40.395713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:40.395713+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:00.527233+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":670,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.229412+0000","last_change":"2024-09-21T12:31:53.103654+0000","last_active":"2024-09-21T12:32:16.229412+0000","last_peered":"2024-09-21T12:32:16.229412+0000","last_clean":"2024-09-21T12:32:16.229412+0000","last_became_active":"2024-09-21T12:31:39.054205+0000","last_became_peered":"2024-09-21T12:31:39.054205+0000","last_unstale":"2024-09-21T12:32:16.229412+0000","last_undegraded":"2024-09-21T12:32:16.229412+0000","last_fullsized":"2024-09-21T12:32:16.229412+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-21T12:31:53.103575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:31:53.103575+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:08.028708+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"308'13","reported_seq":786,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.227933+0000","last_change":"2024-09-21T12:31:57.223219+0000","last_active":"2024-09-21T12:32:16.227933+0000","last_peered":"2024-09-21T12:32:16.227933+0000","last_clean":"2024-09-21T12:32:16.227933+0000","last_became_active":"2024-09-21T12:31:39.054099+0000","last_became_peered":"2024-09-21T12:31:39.054099+0000","last_unstale":"2024-09-21T12:32:16.227933+0000","last_undegraded":"2024-09-21T12:32:16.227933+0000","last_fullsized":"2024-09-21T12:32:16.227933+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'13","last_scrub_stamp":"2024-09-21T12:31:57.223134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:31:57.223134+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:26.569138+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":276,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.685264+0000","last_change":"2024-09-21T12:26:56.862895+0000","last_active":"2024-09-21T12:32:16.685264+0000","last_peered":"2024-09-21T12:32:16.685264+0000","last_clean":"2024-09-21T12:32:16.685264+0000","last_became_active":"2024-09-21T12:26:54.089155+0000","last_became_peered":"2024-09-21T12:26:54.089155+0000","last_unstale":"2024-09-21T12:32:16.685264+0000","last_undegraded":"2024-09-21T12:32:16.685264+0000","last_fullsized":"2024-09-21T12:32:16.685264+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:56.862823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:56.862823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:19.430608+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"76'79","reported_seq":322,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.230986+0000","last_change":"2024-09-21T12:31:39.060779+0000","last_active":"2024-09-21T12:32:16.230986+0000","last_peered":"2024-09-21T12:32:16.230986+0000","last_clean":"2024-09-21T12:32:16.230986+0000","last_became_active":"2024-09-21T12:31:39.060422+0000","last_became_peered":"2024-09-21T12:31:39.060422+0000","last_unstale":"2024-09-21T12:32:16.230986+0000","last_undegraded":"2024-09-21T12:32:16.230986+0000","last_fullsized":"2024-09-21T12:32:16.230986+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:31:11.587537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:31:11.587537+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:25.911669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":631,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.685326+0000","last_change":"2024-09-21T12:32:02.027084+0000","last_active":"2024-09-21T12:32:16.685326+0000","last_peered":"2024-09-21T12:32:16.685326+0000","last_clean":"2024-09-21T12:32:16.685326+0000","last_became_active":"2024-09-21T12:32:02.026693+0000","last_became_peered":"2024-09-21T12:32:02.026693+0000","last_unstale":"2024-09-21T12:32:16.685326+0000","last_undegraded":"2024-09-21T12:32:16.685326+0000","last_fullsized":"2024-09-21T12:32:16.685326+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:07.375741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:31:07.375741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:21.851449+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":77,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.228850+0000","last_change":"2024-09-21T12:32:14.287400+0000","last_active":"2024-09-21T12:32:17.228850+0000","last_peered":"2024-09-21T12:32:17.228850+0000","last_clean":"2024-09-21T12:32:17.228850+0000","last_became_active":"2024-09-21T12:32:14.244680+0000","last_became_peered":"2024-09-21T12:32:14.244680+0000","last_unstale":"2024-09-21T12:32:17.228850+0000","last_undegraded":"2024-09-21T12:32:17.228850+0000","last_fullsized":"2024-09-21T12:32:17.228850+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-21T12:26:57.369648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:57.369648+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":279,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.228327+0000","last_change":"2024-09-21T12:31:10.873131+0000","last_active":"2024-09-21T12:32:17.228327+0000","last_peered":"2024-09-21T12:32:17.228327+0000","last_clean":"2024-09-21T12:32:17.228327+0000","last_became_active":"2024-09-21T12:26:54.084109+0000","last_became_peered":"2024-09-21T12:26:54.084109+0000","last_unstale":"2024-09-21T12:32:17.228327+0000","last_undegraded":"2024-09-21T12:32:17.228327+0000","last_fullsized":"2024-09-21T12:32:17.228327+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:10.873039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:31:10.873039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:29.776805+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":585,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.228635+0000","last_change":"2024-09-21T12:24:00.066522+0000","last_active":"2024-09-21T12:32:16.228635+0000","last_peered":"2024-09-21T12:32:16.228635+0000","last_clean":"2024-09-21T12:32:16.228635+0000","last_became_active":"2024-09-21T12:22:50.904404+0000","last_became_peered":"2024-09-21T12:22:50.904404+0000","last_unstale":"2024-09-21T12:32:16.228635+0000","last_undegraded":"2024-09-21T12:32:16.228635+0000","last_fullsized":"2024-09-21T12:32:16.228635+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:00.066451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:00.066451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:35.132786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'79","reported_seq":549,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.513542+0000","last_change":"2024-09-21T12:31:50.040921+0000","last_active":"2024-09-21T12:32:17.513542+0000","last_peered":"2024-09-21T12:32:17.513542+0000","last_clean":"2024-09-21T12:32:17.513542+0000","last_became_active":"2024-09-21T12:31:50.040502+0000","last_became_peered":"2024-09-21T12:31:50.040502+0000","last_unstale":"2024-09-21T12:32:17.513542+0000","last_undegraded":"2024-09-21T12:32:17.513542+0000","last_fullsized":"2024-09-21T12:32:17.513542+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:29.546657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:29.546657+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":651,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.228722+0000","last_change":"2024-09-21T12:32:14.256591+0000","last_active":"2024-09-21T12:32:17.228722+0000","last_peered":"2024-09-21T12:32:17.228722+0000","last_clean":"2024-09-21T12:32:17.228722+0000","last_became_active":"2024-09-21T12:32:14.242339+0000","last_became_peered":"2024-09-21T12:32:14.242339+0000","last_unstale":"2024-09-21T12:32:17.228722+0000","last_undegraded":"2024-09-21T12:32:17.228722+0000","last_fullsized":"2024-09-21T12:32:17.228722+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:28:01.881164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:01.881164+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":622,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.228410+0000","last_change":"2024-09-21T12:28:43.545259+0000","last_active":"2024-09-21T12:32:16.228410+0000","last_peered":"2024-09-21T12:32:16.228410+0000","last_clean":"2024-09-21T12:32:16.228410+0000","last_became_active":"2024-09-21T12:22:50.927649+0000","last_became_peered":"2024-09-21T12:22:50.927649+0000","last_unstale":"2024-09-21T12:32:16.228410+0000","last_undegraded":"2024-09-21T12:32:16.228410+0000","last_fullsized":"2024-09-21T12:32:16.228410+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:43.545197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:43.545197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":577,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.228735+0000","last_change":"2024-09-21T12:28:47.964092+0000","last_active":"2024-09-21T12:32:17.228735+0000","last_peered":"2024-09-21T12:32:17.228735+0000","last_clean":"2024-09-21T12:32:17.228735+0000","last_became_active":"2024-09-21T12:20:19.311947+0000","last_became_peered":"2024-09-21T12:20:19.311947+0000","last_unstale":"2024-09-21T12:32:17.228735+0000","last_undegraded":"2024-09-21T12:32:17.228735+0000","last_fullsized":"2024-09-21T12:32:17.228735+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:47.963747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:47.963747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":611,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.861525+0000","last_change":"2024-09-21T12:26:11.105716+0000","last_active":"2024-09-21T12:32:16.861525+0000","last_peered":"2024-09-21T12:32:16.861525+0000","last_clean":"2024-09-21T12:32:16.861525+0000","last_became_active":"2024-09-21T12:22:50.924931+0000","last_became_peered":"2024-09-21T12:22:50.924931+0000","last_unstale":"2024-09-21T12:32:16.861525+0000","last_undegraded":"2024-09-21T12:32:16.861525+0000","last_fullsized":"2024-09-21T12:32:16.861525+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:11.105583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:11.105583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:36.717202+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"76'79","reported_seq":669,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.227616+0000","last_change":"2024-09-21T12:31:03.293976+0000","last_active":"2024-09-21T12:32:16.227616+0000","last_peered":"2024-09-21T12:32:16.227616+0000","last_clean":"2024-09-21T12:32:16.227616+0000","last_became_active":"2024-09-21T12:26:54.084939+0000","last_became_peered":"2024-09-21T12:26:54.084939+0000","last_unstale":"2024-09-21T12:32:16.227616+0000","last_undegraded":"2024-09-21T12:32:16.227616+0000","last_fullsized":"2024-09-21T12:32:16.227616+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:31:03.293909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:31:03.293909+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:18.516266+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":355,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.229030+0000","last_change":"2024-09-21T12:32:02.034771+0000","last_active":"2024-09-21T12:32:16.229030+0000","last_peered":"2024-09-21T12:32:16.229030+0000","last_clean":"2024-09-21T12:32:16.229030+0000","last_became_active":"2024-09-21T12:32:02.027460+0000","last_became_peered":"2024-09-21T12:32:02.027460+0000","last_unstale":"2024-09-21T12:32:16.229030+0000","last_undegraded":"2024-09-21T12:32:16.229030+0000","last_fullsized":"2024-09-21T12:32:16.229030+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:28:42.511874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:42.511874+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":626,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.813348+0000","last_change":"2024-09-21T12:32:17.813348+0000","last_active":"2024-09-21T12:32:17.813348+0000","last_peered":"2024-09-21T12:32:17.813348+0000","last_clean":"2024-09-21T12:32:17.813348+0000","last_became_active":"2024-09-21T12:32:14.236220+0000","last_became_peered":"2024-09-21T12:32:14.236220+0000","last_unstale":"2024-09-21T12:32:17.813348+0000","last_undegraded":"2024-09-21T12:32:17.813348+0000","last_fullsized":"2024-09-21T12:32:17.813348+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:17.813293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:17.813293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:39.028980+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"316'10","reported_seq":64,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.226746+0000","last_change":"2024-09-21T12:31:50.050850+0000","last_active":"2024-09-21T12:32:17.226746+0000","last_peered":"2024-09-21T12:32:17.226746+0000","last_clean":"2024-09-21T12:32:17.226746+0000","last_became_active":"2024-09-21T12:31:50.040167+0000","last_became_peered":"2024-09-21T12:31:50.040167+0000","last_unstale":"2024-09-21T12:32:17.226746+0000","last_undegraded":"2024-09-21T12:32:17.226746+0000","last_fullsized":"2024-09-21T12:32:17.226746+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:28:43.853712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:43.853712+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:13.575849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":517,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.869621+0000","last_change":"2024-09-21T12:28:05.323501+0000","last_active":"2024-09-21T12:32:16.869621+0000","last_peered":"2024-09-21T12:32:16.869621+0000","last_clean":"2024-09-21T12:32:16.869621+0000","last_became_active":"2024-09-21T12:22:50.924770+0000","last_became_peered":"2024-09-21T12:22:50.924770+0000","last_unstale":"2024-09-21T12:32:16.869621+0000","last_undegraded":"2024-09-21T12:32:16.869621+0000","last_fullsized":"2024-09-21T12:32:16.869621+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:05.323448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:05.323448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:08.883462+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"76'79","reported_seq":399,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.226480+0000","last_change":"2024-09-21T12:32:02.039507+0000","last_active":"2024-09-21T12:32:17.226480+0000","last_peered":"2024-09-21T12:32:17.226480+0000","last_clean":"2024-09-21T12:32:17.226480+0000","last_became_active":"2024-09-21T12:32:02.034387+0000","last_became_peered":"2024-09-21T12:32:02.034387+0000","last_unstale":"2024-09-21T12:32:17.226480+0000","last_undegraded":"2024-09-21T12:32:17.226480+0000","last_fullsized":"2024-09-21T12:32:17.226480+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:31:48.635956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:31:48.635956+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:58.208578+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":658,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.227606+0000","last_change":"2024-09-21T12:31:39.068884+0000","last_active":"2024-09-21T12:32:17.227606+0000","last_peered":"2024-09-21T12:32:17.227606+0000","last_clean":"2024-09-21T12:32:17.227606+0000","last_became_active":"2024-09-21T12:31:39.058798+0000","last_became_peered":"2024-09-21T12:31:39.058798+0000","last_unstale":"2024-09-21T12:32:17.227606+0000","last_undegraded":"2024-09-21T12:32:17.227606+0000","last_fullsized":"2024-09-21T12:32:17.227606+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:31:03.996594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:03.996594+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:26.397125+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":614,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.228789+0000","last_change":"2024-09-21T12:28:32.630320+0000","last_active":"2024-09-21T12:32:16.228789+0000","last_peered":"2024-09-21T12:32:16.228789+0000","last_clean":"2024-09-21T12:32:16.228789+0000","last_became_active":"2024-09-21T12:22:50.908238+0000","last_became_peered":"2024-09-21T12:22:50.908238+0000","last_unstale":"2024-09-21T12:32:16.228789+0000","last_undegraded":"2024-09-21T12:32:16.228789+0000","last_fullsized":"2024-09-21T12:32:16.228789+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:32.630261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:32.630261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":737,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.232535+0000","last_change":"2024-09-21T12:31:07.510064+0000","last_active":"2024-09-21T12:32:16.232535+0000","last_peered":"2024-09-21T12:32:16.232535+0000","last_clean":"2024-09-21T12:32:16.232535+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:32:16.232535+0000","last_undegraded":"2024-09-21T12:32:16.232535+0000","last_fullsized":"2024-09-21T12:32:16.232535+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:31:07.509980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:31:07.509980+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:28.145931+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":590,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.228468+0000","last_change":"2024-09-21T12:31:52.608520+0000","last_active":"2024-09-21T12:32:17.228468+0000","last_peered":"2024-09-21T12:32:17.228468+0000","last_clean":"2024-09-21T12:32:17.228468+0000","last_became_active":"2024-09-21T12:31:50.037934+0000","last_became_peered":"2024-09-21T12:31:50.037934+0000","last_unstale":"2024-09-21T12:32:17.228468+0000","last_undegraded":"2024-09-21T12:32:17.228468+0000","last_fullsized":"2024-09-21T12:32:17.228468+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:52.608467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:31:52.608467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:11.560519+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"76'79","reported_seq":264,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.514261+0000","last_change":"2024-09-21T12:31:38.104199+0000","last_active":"2024-09-21T12:32:17.514261+0000","last_peered":"2024-09-21T12:32:17.514261+0000","last_clean":"2024-09-21T12:32:17.514261+0000","last_became_active":"2024-09-21T12:27:16.940347+0000","last_became_peered":"2024-09-21T12:27:16.940347+0000","last_unstale":"2024-09-21T12:32:17.514261+0000","last_undegraded":"2024-09-21T12:32:17.514261+0000","last_fullsized":"2024-09-21T12:32:17.514261+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:31:38.104129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:31:38.104129+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:51.210438+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":592,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.685972+0000","last_change":"2024-09-21T12:28:40.150322+0000","last_active":"2024-09-21T12:32:16.685972+0000","last_peered":"2024-09-21T12:32:16.685972+0000","last_clean":"2024-09-21T12:32:16.685972+0000","last_became_active":"2024-09-21T12:22:50.906473+0000","last_became_peered":"2024-09-21T12:22:50.906473+0000","last_unstale":"2024-09-21T12:32:16.685972+0000","last_undegraded":"2024-09-21T12:32:16.685972+0000","last_fullsized":"2024-09-21T12:32:16.685972+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:28:40.150250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:40.150250+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:20.432362+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":509,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.228971+0000","last_change":"2024-09-21T12:31:38.106169+0000","last_active":"2024-09-21T12:32:16.228971+0000","last_peered":"2024-09-21T12:32:16.228971+0000","last_clean":"2024-09-21T12:32:16.228971+0000","last_became_active":"2024-09-21T12:22:50.903705+0000","last_became_peered":"2024-09-21T12:22:50.903705+0000","last_unstale":"2024-09-21T12:32:16.228971+0000","last_undegraded":"2024-09-21T12:32:16.228971+0000","last_fullsized":"2024-09-21T12:32:16.228971+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:38.106115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:31:38.106115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:03.057233+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":596,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.513746+0000","last_change":"2024-09-21T12:31:51.006655+0000","last_active":"2024-09-21T12:32:17.513746+0000","last_peered":"2024-09-21T12:32:17.513746+0000","last_clean":"2024-09-21T12:32:17.513746+0000","last_became_active":"2024-09-21T12:31:50.040754+0000","last_became_peered":"2024-09-21T12:31:50.040754+0000","last_unstale":"2024-09-21T12:32:17.513746+0000","last_undegraded":"2024-09-21T12:32:17.513746+0000","last_fullsized":"2024-09-21T12:32:17.513746+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:51.006611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:31:51.006611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:16.792439+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":62,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.224949+0000","last_change":"2024-09-21T12:31:53.359206+0000","last_active":"2024-09-21T12:32:17.224949+0000","last_peered":"2024-09-21T12:32:17.224949+0000","last_clean":"2024-09-21T12:32:17.224949+0000","last_became_active":"2024-09-21T12:31:50.036150+0000","last_became_peered":"2024-09-21T12:31:50.036150+0000","last_unstale":"2024-09-21T12:32:17.224949+0000","last_undegraded":"2024-09-21T12:32:17.224949+0000","last_fullsized":"2024-09-21T12:32:17.224949+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:53.359159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:31:53.359159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:09.450093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"76'79","reported_seq":279,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.232362+0000","last_change":"2024-09-21T12:31:13.631259+0000","last_active":"2024-09-21T12:32:16.232362+0000","last_peered":"2024-09-21T12:32:16.232362+0000","last_clean":"2024-09-21T12:32:16.232362+0000","last_became_active":"2024-09-21T12:27:16.921880+0000","last_became_peered":"2024-09-21T12:27:16.921880+0000","last_unstale":"2024-09-21T12:32:16.232362+0000","last_undegraded":"2024-09-21T12:32:16.232362+0000","last_fullsized":"2024-09-21T12:32:16.232362+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:31:13.629309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:31:13.629309+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:21.381624+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":80,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.513446+0000","last_change":"2024-09-21T12:31:39.071315+0000","last_active":"2024-09-21T12:32:17.513446+0000","last_peered":"2024-09-21T12:32:17.513446+0000","last_clean":"2024-09-21T12:32:17.513446+0000","last_became_active":"2024-09-21T12:31:39.058221+0000","last_became_peered":"2024-09-21T12:31:39.058221+0000","last_unstale":"2024-09-21T12:32:17.513446+0000","last_undegraded":"2024-09-21T12:32:17.513446+0000","last_fullsized":"2024-09-21T12:32:17.513446+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:31:09.319883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:09.319883+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:21.213987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":613,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.229165+0000","last_change":"2024-09-21T12:26:19.973528+0000","last_active":"2024-09-21T12:32:16.229165+0000","last_peered":"2024-09-21T12:32:16.229165+0000","last_clean":"2024-09-21T12:32:16.229165+0000","last_became_active":"2024-09-21T12:22:50.922019+0000","last_became_peered":"2024-09-21T12:22:50.922019+0000","last_unstale":"2024-09-21T12:32:16.229165+0000","last_undegraded":"2024-09-21T12:32:16.229165+0000","last_fullsized":"2024-09-21T12:32:16.229165+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:19.973461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:19.973461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:00.210381+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":578,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.226594+0000","last_change":"2024-09-21T12:26:21.572474+0000","last_active":"2024-09-21T12:32:17.226594+0000","last_peered":"2024-09-21T12:32:17.226594+0000","last_clean":"2024-09-21T12:32:17.226594+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:32:17.226594+0000","last_undegraded":"2024-09-21T12:32:17.226594+0000","last_fullsized":"2024-09-21T12:32:17.226594+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:21.571726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:21.571726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:22.821247+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":577,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.225754+0000","last_change":"2024-09-21T12:28:41.864711+0000","last_active":"2024-09-21T12:32:17.225754+0000","last_peered":"2024-09-21T12:32:17.225754+0000","last_clean":"2024-09-21T12:32:17.225754+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:32:17.225754+0000","last_undegraded":"2024-09-21T12:32:17.225754+0000","last_fullsized":"2024-09-21T12:32:17.225754+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:41.864630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:41.864630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"76'79","reported_seq":292,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.832673+0000","last_change":"2024-09-21T12:31:38.745628+0000","last_active":"2024-09-21T12:32:16.832673+0000","last_peered":"2024-09-21T12:32:16.832673+0000","last_clean":"2024-09-21T12:32:16.832673+0000","last_became_active":"2024-09-21T12:26:46.989742+0000","last_became_peered":"2024-09-21T12:26:46.989742+0000","last_unstale":"2024-09-21T12:32:16.832673+0000","last_undegraded":"2024-09-21T12:32:16.832673+0000","last_fullsized":"2024-09-21T12:32:16.832673+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:31:38.745581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:31:38.745581+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:43.839127+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":621,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.225902+0000","last_change":"2024-09-21T12:31:06.945554+0000","last_active":"2024-09-21T12:32:17.225902+0000","last_peered":"2024-09-21T12:32:17.225902+0000","last_clean":"2024-09-21T12:32:17.225902+0000","last_became_active":"2024-09-21T12:20:17.694794+0000","last_became_peered":"2024-09-21T12:20:17.694794+0000","last_unstale":"2024-09-21T12:32:17.225902+0000","last_undegraded":"2024-09-21T12:32:17.225902+0000","last_fullsized":"2024-09-21T12:32:17.225902+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:31:06.945472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:06.945472+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:24.736973+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":608,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.232086+0000","last_change":"2024-09-21T12:29:54.303655+0000","last_active":"2024-09-21T12:32:16.232086+0000","last_peered":"2024-09-21T12:32:16.232086+0000","last_clean":"2024-09-21T12:32:16.232086+0000","last_became_active":"2024-09-21T12:20:15.295968+0000","last_became_peered":"2024-09-21T12:20:15.295968+0000","last_unstale":"2024-09-21T12:32:16.232086+0000","last_undegraded":"2024-09-21T12:32:16.232086+0000","last_fullsized":"2024-09-21T12:32:16.232086+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:54.303590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:54.303590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:33.627493+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":590,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.227516+0000","last_change":"2024-09-21T12:23:57.018797+0000","last_active":"2024-09-21T12:32:16.227516+0000","last_peered":"2024-09-21T12:32:16.227516+0000","last_clean":"2024-09-21T12:32:16.227516+0000","last_became_active":"2024-09-21T12:22:50.921604+0000","last_became_peered":"2024-09-21T12:22:50.921604+0000","last_unstale":"2024-09-21T12:32:16.227516+0000","last_undegraded":"2024-09-21T12:32:16.227516+0000","last_fullsized":"2024-09-21T12:32:16.227516+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:57.018749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:57.018749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:33.121402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":574,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.226742+0000","last_change":"2024-09-21T12:29:14.051153+0000","last_active":"2024-09-21T12:32:17.226742+0000","last_peered":"2024-09-21T12:32:17.226742+0000","last_clean":"2024-09-21T12:32:17.226742+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:32:17.226742+0000","last_undegraded":"2024-09-21T12:32:17.226742+0000","last_fullsized":"2024-09-21T12:32:17.226742+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:14.051043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:29:14.051043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"76'79","reported_seq":255,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.228525+0000","last_change":"2024-09-21T12:27:24.373557+0000","last_active":"2024-09-21T12:32:16.228525+0000","last_peered":"2024-09-21T12:32:16.228525+0000","last_clean":"2024-09-21T12:32:16.228525+0000","last_became_active":"2024-09-21T12:27:16.914713+0000","last_became_peered":"2024-09-21T12:27:16.914713+0000","last_unstale":"2024-09-21T12:32:16.228525+0000","last_undegraded":"2024-09-21T12:32:16.228525+0000","last_fullsized":"2024-09-21T12:32:16.228525+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:24.373501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:24.373501+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":594,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.225188+0000","last_change":"2024-09-21T12:28:46.932522+0000","last_active":"2024-09-21T12:32:17.225188+0000","last_peered":"2024-09-21T12:32:17.225188+0000","last_clean":"2024-09-21T12:32:17.225188+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:32:17.225188+0000","last_undegraded":"2024-09-21T12:32:17.225188+0000","last_fullsized":"2024-09-21T12:32:17.225188+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-21T12:28:46.931901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:46.931901+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":610,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.227945+0000","last_change":"2024-09-21T12:31:57.681487+0000","last_active":"2024-09-21T12:32:17.227945+0000","last_peered":"2024-09-21T12:32:17.227945+0000","last_clean":"2024-09-21T12:32:17.227945+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:32:17.227945+0000","last_undegraded":"2024-09-21T12:32:17.227945+0000","last_fullsized":"2024-09-21T12:32:17.227945+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:31:57.681417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:31:57.681417+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:17.342617+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":606,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.685446+0000","last_change":"2024-09-21T12:29:59.216485+0000","last_active":"2024-09-21T12:32:16.685446+0000","last_peered":"2024-09-21T12:32:16.685446+0000","last_clean":"2024-09-21T12:32:16.685446+0000","last_became_active":"2024-09-21T12:22:50.930615+0000","last_became_peered":"2024-09-21T12:22:50.930615+0000","last_unstale":"2024-09-21T12:32:16.685446+0000","last_undegraded":"2024-09-21T12:32:16.685446+0000","last_fullsized":"2024-09-21T12:32:16.685446+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:59.216442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:29:59.216442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:34.428945+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":594,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.227991+0000","last_change":"2024-09-21T12:28:41.537845+0000","last_active":"2024-09-21T12:32:16.227991+0000","last_peered":"2024-09-21T12:32:16.227991+0000","last_clean":"2024-09-21T12:32:16.227991+0000","last_became_active":"2024-09-21T12:22:50.917408+0000","last_became_peered":"2024-09-21T12:22:50.917408+0000","last_unstale":"2024-09-21T12:32:16.227991+0000","last_undegraded":"2024-09-21T12:32:16.227991+0000","last_fullsized":"2024-09-21T12:32:16.227991+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:41.537786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:41.537786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"76'79","reported_seq":255,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.228225+0000","last_change":"2024-09-21T12:27:29.341314+0000","last_active":"2024-09-21T12:32:16.228225+0000","last_peered":"2024-09-21T12:32:16.228225+0000","last_clean":"2024-09-21T12:32:16.228225+0000","last_became_active":"2024-09-21T12:27:16.948697+0000","last_became_peered":"2024-09-21T12:27:16.948697+0000","last_unstale":"2024-09-21T12:32:16.228225+0000","last_undegraded":"2024-09-21T12:32:16.228225+0000","last_fullsized":"2024-09-21T12:32:16.228225+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:29.341268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:29.341268+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:09.132577+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":595,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.514319+0000","last_change":"2024-09-21T12:28:46.749904+0000","last_active":"2024-09-21T12:32:17.514319+0000","last_peered":"2024-09-21T12:32:17.514319+0000","last_clean":"2024-09-21T12:32:17.514319+0000","last_became_active":"2024-09-21T12:22:50.914221+0000","last_became_peered":"2024-09-21T12:22:50.914221+0000","last_unstale":"2024-09-21T12:32:17.514319+0000","last_undegraded":"2024-09-21T12:32:17.514319+0000","last_fullsized":"2024-09-21T12:32:17.514319+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:46.749839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:46.749839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":607,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.225415+0000","last_change":"2024-09-21T12:31:39.052864+0000","last_active":"2024-09-21T12:32:17.225415+0000","last_peered":"2024-09-21T12:32:17.225415+0000","last_clean":"2024-09-21T12:32:17.225415+0000","last_became_active":"2024-09-21T12:31:39.052486+0000","last_became_peered":"2024-09-21T12:31:39.052486+0000","last_unstale":"2024-09-21T12:32:17.225415+0000","last_undegraded":"2024-09-21T12:32:17.225415+0000","last_fullsized":"2024-09-21T12:32:17.225415+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:15.071234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:15.071234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":620,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-09-21T12:32:16.686248+0000","last_change":"2024-09-21T12:30:02.290006+0000","last_active":"2024-09-21T12:32:16.686248+0000","last_peered":"2024-09-21T12:32:16.686248+0000","last_clean":"2024-09-21T12:32:16.686248+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:32:16.686248+0000","last_undegraded":"2024-09-21T12:32:16.686248+0000","last_fullsized":"2024-09-21T12:32:16.686248+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:30:02.289930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:30:02.289930+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:36.380982+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":605,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-09-21T12:32:17.228494+0000","last_change":"2024-09-21T12:28:28.741483+0000","last_active":"2024-09-21T12:32:17.228494+0000","last_peered":"2024-09-21T12:32:17.228494+0000","last_clean":"2024-09-21T12:32:17.228494+0000","last_became_active":"2024-09-21T12:20:19.308247+0000","last_became_peered":"2024-09-21T12:20:19.308247+0000","last_unstale":"2024-09-21T12:32:17.228494+0000","last_undegraded":"2024-09-21T12:32:17.228494+0000","last_fullsized":"2024-09-21T12:32:17.228494+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:28.741418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:28.741418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1111,"num_read_kb":868,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":307,"num_bytes_recovered":6050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":683,"ondisk_log_size":683,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":5023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":715,"num_read_kb":710,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2449,"ondisk_log_size":2449,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234132,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":582628,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93162524,"statfs":{"total":95995035648,"available":95398424576,"internally_reserved":0,"allocated":1716224,"data_stored":1171163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":594868682},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463068,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":461912,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":93283240,"statfs":{"total":95995035648,"available":95522037760,"internally_reserved":0,"allocated":1765376,"data_stored":1227680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":471202246},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724703,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":542368,"kb_used_data":1672,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":93202784,"statfs":{"total":95995035648,"available":95439650816,"internally_reserved":0,"allocated":1712128,"data_stored":1171943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953639,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149869,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":636872,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":93108280,"statfs":{"total":95995035648,"available":95342878720,"internally_reserved":0,"allocated":1814528,"data_stored":1245691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":650312135},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411507,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":416008,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":93329144,"statfs":{"total":95995035648,"available":95569043456,"internally_reserved":0,"allocated":1687552,"data_stored":1162244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575033,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":654604,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93090548,"statfs":{"total":95995035648,"available":95324721152,"internally_reserved":0,"allocated":1490944,"data_stored":908904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738559,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":597876,"kb_used_data":39132,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":93147276,"statfs":{"total":95995035648,"available":95382810624,"internally_reserved":0,"allocated":40071168,"data_stored":39518129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:32:22.293 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:32:22.317 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:22.339 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:22.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:22.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:22.441 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:22.462 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:22.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:22.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:22.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:22 smithi114 ceph-mon[30324]: 4.17 scrub starts 2024-09-21T12:32:22.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:22 smithi114 ceph-mon[30324]: 4.17 scrub ok 2024-09-21T12:32:22.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:22 smithi114 ceph-mon[30324]: 3.0 scrub starts 2024-09-21T12:32:22.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:22 smithi114 ceph-mon[30324]: 3.0 scrub ok 2024-09-21T12:32:22.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:22 smithi114 ceph-mon[30324]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:32:22.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:22 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:32:22.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:22 smithi114 ceph-mon[30324]: osdmap e339: 8 total, 8 up, 7 in 2024-09-21T12:32:22.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:22 smithi114 ceph-mon[30324]: 5.f scrub starts 2024-09-21T12:32:22.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:22 smithi114 ceph-mon[30324]: 5.f scrub ok 2024-09-21T12:32:22.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:22 smithi114 ceph-mon[30324]: osdmap e340: 8 total, 8 up, 7 in 2024-09-21T12:32:22.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:22 smithi114 ceph-mon[40867]: 4.17 scrub starts 2024-09-21T12:32:22.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:22 smithi114 ceph-mon[40867]: 4.17 scrub ok 2024-09-21T12:32:22.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:22 smithi114 ceph-mon[40867]: 3.0 scrub starts 2024-09-21T12:32:22.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:22 smithi114 ceph-mon[40867]: 3.0 scrub ok 2024-09-21T12:32:22.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:22 smithi114 ceph-mon[40867]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:32:22.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:22 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:32:22.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:22 smithi114 ceph-mon[40867]: osdmap e339: 8 total, 8 up, 7 in 2024-09-21T12:32:22.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:22 smithi114 ceph-mon[40867]: 5.f scrub starts 2024-09-21T12:32:22.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:22 smithi114 ceph-mon[40867]: 5.f scrub ok 2024-09-21T12:32:22.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:22 smithi114 ceph-mon[40867]: osdmap e340: 8 total, 8 up, 7 in 2024-09-21T12:32:22.564 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:22.584 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:22.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:22.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:22.686 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:22.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:22.709 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:22.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:22.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:22 smithi183 ceph-mon[33277]: 4.17 scrub starts 2024-09-21T12:32:22.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:22 smithi183 ceph-mon[33277]: 4.17 scrub ok 2024-09-21T12:32:22.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:22 smithi183 ceph-mon[33277]: 3.0 scrub starts 2024-09-21T12:32:22.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:22 smithi183 ceph-mon[33277]: 3.0 scrub ok 2024-09-21T12:32:22.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:22 smithi183 ceph-mon[33277]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:32:22.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:22 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:32:22.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:22 smithi183 ceph-mon[33277]: osdmap e339: 8 total, 8 up, 7 in 2024-09-21T12:32:22.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:22 smithi183 ceph-mon[33277]: 5.f scrub starts 2024-09-21T12:32:22.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:22 smithi183 ceph-mon[33277]: 5.f scrub ok 2024-09-21T12:32:22.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:22 smithi183 ceph-mon[33277]: osdmap e340: 8 total, 8 up, 7 in 2024-09-21T12:32:22.810 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:22.819 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:32:22 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:32:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:32:22.832 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:22.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:22.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:22.933 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:22.955 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:22.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:22.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:23.057 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:23.078 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:23.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:23.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:23.180 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:23.201 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:23.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:23.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:23.291 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T12:32:23.292 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T12:32:23.292 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-09-21T12:32:23.292 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_3 16 2024-09-21T12:32:23.303 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:23.329 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:23.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:23.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:23.430 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:23.456 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:23.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:23.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:23.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:23 smithi114 ceph-mon[40867]: from='client.25739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:32:23.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:23 smithi114 ceph-mon[40867]: 6.6 scrub starts 2024-09-21T12:32:23.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:23 smithi114 ceph-mon[40867]: 6.6 scrub ok 2024-09-21T12:32:23.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:23 smithi114 ceph-mon[40867]: pgmap v615: 162 pgs: 162 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T12:32:23.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:23 smithi114 ceph-mon[40867]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-21T12:32:23.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:23 smithi114 ceph-mon[40867]: Cluster is now healthy 2024-09-21T12:32:23.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:23 smithi114 ceph-mon[40867]: 1.18 scrub starts 2024-09-21T12:32:23.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:23 smithi114 ceph-mon[40867]: 1.18 scrub ok 2024-09-21T12:32:23.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:23 smithi114 ceph-mon[40867]: 5.6 scrub starts 2024-09-21T12:32:23.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:23 smithi114 ceph-mon[40867]: 5.6 scrub ok 2024-09-21T12:32:23.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:23 smithi114 ceph-mon[30324]: from='client.25739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:32:23.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:23 smithi114 ceph-mon[30324]: 6.6 scrub starts 2024-09-21T12:32:23.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:23 smithi114 ceph-mon[30324]: 6.6 scrub ok 2024-09-21T12:32:23.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:23 smithi114 ceph-mon[30324]: pgmap v615: 162 pgs: 162 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T12:32:23.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:23 smithi114 ceph-mon[30324]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-21T12:32:23.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:23 smithi114 ceph-mon[30324]: Cluster is now healthy 2024-09-21T12:32:23.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:23 smithi114 ceph-mon[30324]: 1.18 scrub starts 2024-09-21T12:32:23.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:23 smithi114 ceph-mon[30324]: 1.18 scrub ok 2024-09-21T12:32:23.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:23 smithi114 ceph-mon[30324]: 5.6 scrub starts 2024-09-21T12:32:23.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:23 smithi114 ceph-mon[30324]: 5.6 scrub ok 2024-09-21T12:32:23.558 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:23.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:23.583 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:23.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:23.684 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:23.710 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:23.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:23.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:23.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:23 smithi183 ceph-mon[33277]: from='client.25739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:32:23.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:23 smithi183 ceph-mon[33277]: 6.6 scrub starts 2024-09-21T12:32:23.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:23 smithi183 ceph-mon[33277]: 6.6 scrub ok 2024-09-21T12:32:23.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:23 smithi183 ceph-mon[33277]: pgmap v615: 162 pgs: 162 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-21T12:32:23.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:23 smithi183 ceph-mon[33277]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-21T12:32:23.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:23 smithi183 ceph-mon[33277]: Cluster is now healthy 2024-09-21T12:32:23.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:23 smithi183 ceph-mon[33277]: 1.18 scrub starts 2024-09-21T12:32:23.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:23 smithi183 ceph-mon[33277]: 1.18 scrub ok 2024-09-21T12:32:23.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:23 smithi183 ceph-mon[33277]: 5.6 scrub starts 2024-09-21T12:32:23.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:23 smithi183 ceph-mon[33277]: 5.6 scrub ok 2024-09-21T12:32:23.803 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:32:23.804 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:32:23.812 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:23.834 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:23.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:23.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:23.935 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:23.957 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:23.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:23.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:24.059 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:24.081 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:24.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:24.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:24.182 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:24.209 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:24.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:24.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:24.311 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:24.399 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:24.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:24.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:24.501 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:24.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:24 smithi114 ceph-mon[40867]: 5.8 scrub starts 2024-09-21T12:32:24.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:24 smithi114 ceph-mon[40867]: 5.8 scrub ok 2024-09-21T12:32:24.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:24 smithi114 ceph-mon[30324]: 5.8 scrub starts 2024-09-21T12:32:24.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:24 smithi114 ceph-mon[30324]: 5.8 scrub ok 2024-09-21T12:32:24.527 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:24.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:24.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:24.605 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T12:32:24.629 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:24.641 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:24.668 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:24.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:24.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:24.688 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:24.688 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:24.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:24 smithi183 ceph-mon[33277]: 5.8 scrub starts 2024-09-21T12:32:24.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:24 smithi183 ceph-mon[33277]: 5.8 scrub ok 2024-09-21T12:32:24.770 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:24.792 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:24.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:24.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:24.865 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T12:32:24.894 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:24.920 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:24.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:24.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:25.022 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:25.047 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:25.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:25.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:25.149 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:25.170 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:25.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:25.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:25.272 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:25.293 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:25.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:25.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:25.395 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:25.415 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:25.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:25.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:25.517 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:25.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:25 smithi114 ceph-mon[30324]: pgmap v617: 162 pgs: 162 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T12:32:25.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:25 smithi114 ceph-mon[30324]: 6.5 scrub starts 2024-09-21T12:32:25.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:25 smithi114 ceph-mon[30324]: 6.5 scrub ok 2024-09-21T12:32:25.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:25 smithi114 ceph-mon[30324]: 1.17 scrub starts 2024-09-21T12:32:25.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:25 smithi114 ceph-mon[30324]: 1.17 scrub ok 2024-09-21T12:32:25.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:25 smithi114 ceph-mon[30324]: 3.1f scrub starts 2024-09-21T12:32:25.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:25 smithi114 ceph-mon[30324]: 3.1f scrub ok 2024-09-21T12:32:25.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:25 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:32:25.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:25 smithi114 ceph-mon[40867]: pgmap v617: 162 pgs: 162 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T12:32:25.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:25 smithi114 ceph-mon[40867]: 6.5 scrub starts 2024-09-21T12:32:25.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:25 smithi114 ceph-mon[40867]: 6.5 scrub ok 2024-09-21T12:32:25.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:25 smithi114 ceph-mon[40867]: 1.17 scrub starts 2024-09-21T12:32:25.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:25 smithi114 ceph-mon[40867]: 1.17 scrub ok 2024-09-21T12:32:25.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:25 smithi114 ceph-mon[40867]: 3.1f scrub starts 2024-09-21T12:32:25.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:25 smithi114 ceph-mon[40867]: 3.1f scrub ok 2024-09-21T12:32:25.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:25 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:32:25.537 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:25.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:25.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:25.639 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:25.660 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:25.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:25.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:25.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:25 smithi183 ceph-mon[33277]: pgmap v617: 162 pgs: 162 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T12:32:25.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:25 smithi183 ceph-mon[33277]: 6.5 scrub starts 2024-09-21T12:32:25.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:25 smithi183 ceph-mon[33277]: 6.5 scrub ok 2024-09-21T12:32:25.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:25 smithi183 ceph-mon[33277]: 1.17 scrub starts 2024-09-21T12:32:25.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:25 smithi183 ceph-mon[33277]: 1.17 scrub ok 2024-09-21T12:32:25.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:25 smithi183 ceph-mon[33277]: 3.1f scrub starts 2024-09-21T12:32:25.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:25 smithi183 ceph-mon[33277]: 3.1f scrub ok 2024-09-21T12:32:25.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:25 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:32:25.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:25.782 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:25.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:25.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:25.883 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:25.904 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:25.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:25.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:26.005 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:26.027 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:26.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:26.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:26.129 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:26.149 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:26.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:26.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:26.251 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:26.272 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:26.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:26.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:26.374 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:26.396 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:26.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:26.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:26.498 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:26.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:26.521 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:26.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:26.622 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:26.644 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:26.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:26.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:26.688 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:26 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:32:26.688 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:26 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:32:26.688 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:26 smithi114 ceph-mon[30324]: 4.4 scrub starts 2024-09-21T12:32:26.688 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:26 smithi114 ceph-mon[30324]: 4.4 scrub ok 2024-09-21T12:32:26.688 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:26 smithi114 ceph-mon[30324]: osdmap e341: 8 total, 8 up, 7 in 2024-09-21T12:32:26.688 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:26 smithi114 ceph-mon[30324]: 1.f scrub starts 2024-09-21T12:32:26.688 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:26 smithi114 ceph-mon[30324]: 1.f scrub ok 2024-09-21T12:32:26.689 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:26 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:32:26.689 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:26 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:32:26.689 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:26 smithi114 ceph-mon[40867]: 4.4 scrub starts 2024-09-21T12:32:26.689 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:26 smithi114 ceph-mon[40867]: 4.4 scrub ok 2024-09-21T12:32:26.689 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:26 smithi114 ceph-mon[40867]: osdmap e341: 8 total, 8 up, 7 in 2024-09-21T12:32:26.689 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:26 smithi114 ceph-mon[40867]: 1.f scrub starts 2024-09-21T12:32:26.689 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:26 smithi114 ceph-mon[40867]: 1.f scrub ok 2024-09-21T12:32:26.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:26 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:32:26.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:26 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:32:26.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:26 smithi183 ceph-mon[33277]: 4.4 scrub starts 2024-09-21T12:32:26.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:26 smithi183 ceph-mon[33277]: 4.4 scrub ok 2024-09-21T12:32:26.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:26 smithi183 ceph-mon[33277]: osdmap e341: 8 total, 8 up, 7 in 2024-09-21T12:32:26.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:26 smithi183 ceph-mon[33277]: 1.f scrub starts 2024-09-21T12:32:26.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:26 smithi183 ceph-mon[33277]: 1.f scrub ok 2024-09-21T12:32:26.746 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:26.767 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:26.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:26.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:26.869 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:26.891 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:26.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:26.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:26.992 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:27.014 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:27.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:27.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:27.115 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:27.137 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:27.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:27.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:27.238 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:27.259 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:27.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:27.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:27.275 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:32:27.275 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:32:27.287 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:32:27.287 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:32:27.304 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:32:27.304 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:32:27.321 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:32:27.322 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:32:27.338 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:32:27.338 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:32:27.355 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:32:27.355 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:32:27.360 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:27.371 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:32:27.372 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:32:27.381 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:27.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:27.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:27.388 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:32:27.388 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:32:27.483 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:27.505 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:27.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:27.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:27.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:27 smithi114 ceph-mon[40867]: pgmap v619: 162 pgs: 162 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 920 B/s rd, 0 op/s 2024-09-21T12:32:27.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:27 smithi114 ceph-mon[40867]: 4.c scrub starts 2024-09-21T12:32:27.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:27 smithi114 ceph-mon[40867]: 4.c scrub ok 2024-09-21T12:32:27.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:27 smithi114 ceph-mon[40867]: 1.12 scrub starts 2024-09-21T12:32:27.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:27 smithi114 ceph-mon[40867]: 1.12 scrub ok 2024-09-21T12:32:27.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:27 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:32:27.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:27 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:32:27.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:27 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:32:27.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:27 smithi114 ceph-mon[40867]: osdmap e342: 8 total, 8 up, 7 in 2024-09-21T12:32:27.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:27 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1049050466' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:32:27.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:27 smithi114 ceph-mon[30324]: pgmap v619: 162 pgs: 162 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 920 B/s rd, 0 op/s 2024-09-21T12:32:27.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:27 smithi114 ceph-mon[30324]: 4.c scrub starts 2024-09-21T12:32:27.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:27 smithi114 ceph-mon[30324]: 4.c scrub ok 2024-09-21T12:32:27.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:27 smithi114 ceph-mon[30324]: 1.12 scrub starts 2024-09-21T12:32:27.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:27 smithi114 ceph-mon[30324]: 1.12 scrub ok 2024-09-21T12:32:27.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:27 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:32:27.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:27 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:32:27.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:27 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:32:27.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:27 smithi114 ceph-mon[30324]: osdmap e342: 8 total, 8 up, 7 in 2024-09-21T12:32:27.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:27 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1049050466' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:32:27.607 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:27.630 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:27.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:27.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:27.732 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:27.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:32:27 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:32:27.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:27 smithi183 ceph-mon[33277]: pgmap v619: 162 pgs: 162 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 920 B/s rd, 0 op/s 2024-09-21T12:32:27.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:27 smithi183 ceph-mon[33277]: 4.c scrub starts 2024-09-21T12:32:27.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:27 smithi183 ceph-mon[33277]: 4.c scrub ok 2024-09-21T12:32:27.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:27 smithi183 ceph-mon[33277]: 1.12 scrub starts 2024-09-21T12:32:27.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:27 smithi183 ceph-mon[33277]: 1.12 scrub ok 2024-09-21T12:32:27.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:27 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:32:27.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:27 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:32:27.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:27 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:32:27.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:27 smithi183 ceph-mon[33277]: osdmap e342: 8 total, 8 up, 7 in 2024-09-21T12:32:27.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:27 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1049050466' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:32:27.753 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:27.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:27.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:27.854 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:27.875 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:27.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:27.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:27.977 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:28.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:28.000 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:28.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:28.094 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:32:28.097 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:32:28.101 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:28.123 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:28.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:28.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:28.224 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:28.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:28.246 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:28.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:28.347 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:28.368 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:28.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:28.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:28.469 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:28.491 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:28.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:28.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:28.594 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:28.615 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:28.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:28.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:28.672 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:28 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:32:28.672 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:28 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:32:28.672 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:28 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:28.672 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:28 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:32:28.672 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:28 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:32:28.672 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:28 smithi114 ceph-mon[40867]: 6.a scrub starts 2024-09-21T12:32:28.672 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:28 smithi114 ceph-mon[40867]: 6.a scrub ok 2024-09-21T12:32:28.672 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:28 smithi114 ceph-mon[40867]: 3.3 scrub starts 2024-09-21T12:32:28.672 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:28 smithi114 ceph-mon[40867]: 3.3 scrub ok 2024-09-21T12:32:28.672 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:28 smithi114 ceph-mon[40867]: osdmap e343: 8 total, 8 up, 7 in 2024-09-21T12:32:28.673 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:28 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:32:28.673 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:28 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:32:28.673 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:28 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:28.673 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:28 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:32:28.673 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:28 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:32:28.673 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:28 smithi114 ceph-mon[30324]: 6.a scrub starts 2024-09-21T12:32:28.673 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:28 smithi114 ceph-mon[30324]: 6.a scrub ok 2024-09-21T12:32:28.673 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:28 smithi114 ceph-mon[30324]: 3.3 scrub starts 2024-09-21T12:32:28.673 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:28 smithi114 ceph-mon[30324]: 3.3 scrub ok 2024-09-21T12:32:28.673 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:28 smithi114 ceph-mon[30324]: osdmap e343: 8 total, 8 up, 7 in 2024-09-21T12:32:28.717 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:28.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:28 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:32:28.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:28 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:32:28.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:28 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:28.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:28 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:32:28.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:28 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:32:28.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:28 smithi183 ceph-mon[33277]: 6.a scrub starts 2024-09-21T12:32:28.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:28 smithi183 ceph-mon[33277]: 6.a scrub ok 2024-09-21T12:32:28.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:28 smithi183 ceph-mon[33277]: 3.3 scrub starts 2024-09-21T12:32:28.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:28 smithi183 ceph-mon[33277]: 3.3 scrub ok 2024-09-21T12:32:28.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:28 smithi183 ceph-mon[33277]: osdmap e343: 8 total, 8 up, 7 in 2024-09-21T12:32:28.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:28.740 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:28.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:28.841 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:28.863 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:28.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:28.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:28.964 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:28.986 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:28.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:28.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:29.050 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:29.051 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:29.088 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:29.110 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:29.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:29.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:29.212 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:29.233 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:29.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:29.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:29.292 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'unique_pool_3' created 2024-09-21T12:32:29.335 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:29.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:29.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:29.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:29 smithi114 ceph-mon[30324]: 5.0 scrub starts 2024-09-21T12:32:29.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:29 smithi114 ceph-mon[30324]: 5.0 scrub ok 2024-09-21T12:32:29.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:29 smithi114 ceph-mon[30324]: from='client.25751 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:32:29.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:29 smithi114 ceph-mon[30324]: pgmap v621: 162 pgs: 162 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:29.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:29 smithi114 ceph-mon[30324]: 1.13 scrub starts 2024-09-21T12:32:29.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:29 smithi114 ceph-mon[30324]: 1.13 scrub ok 2024-09-21T12:32:29.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:29 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1841669477' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T12:32:29.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:29 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T12:32:29.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:29 smithi114 ceph-mon[30324]: 1.8 scrub starts 2024-09-21T12:32:29.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:29 smithi114 ceph-mon[30324]: 1.8 scrub ok 2024-09-21T12:32:29.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:29 smithi114 ceph-mon[30324]: 4.b scrub starts 2024-09-21T12:32:29.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:29 smithi114 ceph-mon[30324]: 4.b scrub ok 2024-09-21T12:32:29.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:29 smithi114 ceph-mon[40867]: 5.0 scrub starts 2024-09-21T12:32:29.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:29 smithi114 ceph-mon[40867]: 5.0 scrub ok 2024-09-21T12:32:29.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:29 smithi114 ceph-mon[40867]: from='client.25751 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:32:29.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:29 smithi114 ceph-mon[40867]: pgmap v621: 162 pgs: 162 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:29.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:29 smithi114 ceph-mon[40867]: 1.13 scrub starts 2024-09-21T12:32:29.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:29 smithi114 ceph-mon[40867]: 1.13 scrub ok 2024-09-21T12:32:29.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:29 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1841669477' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T12:32:29.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:29 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T12:32:29.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:29 smithi114 ceph-mon[40867]: 1.8 scrub starts 2024-09-21T12:32:29.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:29 smithi114 ceph-mon[40867]: 1.8 scrub ok 2024-09-21T12:32:29.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:29 smithi114 ceph-mon[40867]: 4.b scrub starts 2024-09-21T12:32:29.437 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:29 smithi114 ceph-mon[40867]: 4.b scrub ok 2024-09-21T12:32:29.459 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:29.481 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:29.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:29.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:29.583 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:29.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:29.606 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:29.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:29.706 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:29.728 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:29.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:29.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:29 smithi183 ceph-mon[33277]: 5.0 scrub starts 2024-09-21T12:32:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:29 smithi183 ceph-mon[33277]: 5.0 scrub ok 2024-09-21T12:32:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:29 smithi183 ceph-mon[33277]: from='client.25751 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:32:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:29 smithi183 ceph-mon[33277]: pgmap v621: 162 pgs: 162 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:29 smithi183 ceph-mon[33277]: 1.13 scrub starts 2024-09-21T12:32:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:29 smithi183 ceph-mon[33277]: 1.13 scrub ok 2024-09-21T12:32:29.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:29 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1841669477' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T12:32:29.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:29 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T12:32:29.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:29 smithi183 ceph-mon[33277]: 1.8 scrub starts 2024-09-21T12:32:29.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:29 smithi183 ceph-mon[33277]: 1.8 scrub ok 2024-09-21T12:32:29.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:29 smithi183 ceph-mon[33277]: 4.b scrub starts 2024-09-21T12:32:29.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:29 smithi183 ceph-mon[33277]: 4.b scrub ok 2024-09-21T12:32:29.774 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":620,"stamp":"2024-09-21T12:32:27.023940+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":1860,"num_read_kb":1612,"num_write":722,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":340,"num_bytes_recovered":6950,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3234,"ondisk_log_size":3234,"up":486,"acting":486,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":515,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":3906880,"kb_used_data":45668,"kb_used_omap":10,"kb_used_meta":3861045,"kb_avail":652309184,"statfs":{"total":671965249536,"available":667964604416,"internally_reserved":0,"allocated":46764032,"data_stored":42763660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":3953710187},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":49,"apply_latency_ms":49,"commit_latency_ns":49000000,"apply_latency_ns":49000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.762327"},"pg_stats":[{"pgid":"1.1d","version":"76'79","reported_seq":287,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.025331+0000","last_change":"2024-09-21T12:31:50.043467+0000","last_active":"2024-09-21T12:32:22.025331+0000","last_peered":"2024-09-21T12:32:22.025331+0000","last_clean":"2024-09-21T12:32:22.025331+0000","last_became_active":"2024-09-21T12:31:50.043059+0000","last_became_peered":"2024-09-21T12:31:50.043059+0000","last_unstale":"2024-09-21T12:32:22.025331+0000","last_undegraded":"2024-09-21T12:32:22.025331+0000","last_fullsized":"2024-09-21T12:32:22.025331+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:47.816137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:47.816137+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":595,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.025540+0000","last_change":"2024-09-21T12:24:50.115300+0000","last_active":"2024-09-21T12:32:22.025540+0000","last_peered":"2024-09-21T12:32:22.025540+0000","last_clean":"2024-09-21T12:32:22.025540+0000","last_became_active":"2024-09-21T12:20:15.309431+0000","last_became_peered":"2024-09-21T12:20:15.309431+0000","last_unstale":"2024-09-21T12:32:22.025540+0000","last_undegraded":"2024-09-21T12:32:22.025540+0000","last_fullsized":"2024-09-21T12:32:22.025540+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:50.115253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:24:50.115253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:24.563784+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":232,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.023921+0000","last_change":"2024-09-21T12:29:08.671789+0000","last_active":"2024-09-21T12:32:22.023921+0000","last_peered":"2024-09-21T12:32:22.023921+0000","last_clean":"2024-09-21T12:32:22.023921+0000","last_became_active":"2024-09-21T12:29:05.833372+0000","last_became_peered":"2024-09-21T12:29:05.833372+0000","last_unstale":"2024-09-21T12:32:22.023921+0000","last_undegraded":"2024-09-21T12:32:22.023921+0000","last_fullsized":"2024-09-21T12:32:22.023921+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:29:08.671723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:29:08.671723+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":592,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.024985+0000","last_change":"2024-09-21T12:29:10.672659+0000","last_active":"2024-09-21T12:32:22.024985+0000","last_peered":"2024-09-21T12:32:22.024985+0000","last_clean":"2024-09-21T12:32:22.024985+0000","last_became_active":"2024-09-21T12:20:19.314908+0000","last_became_peered":"2024-09-21T12:20:19.314908+0000","last_unstale":"2024-09-21T12:32:22.024985+0000","last_undegraded":"2024-09-21T12:32:22.024985+0000","last_fullsized":"2024-09-21T12:32:22.024985+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:10.672599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:29:10.672599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":541,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.729300+0000","last_change":"2024-09-21T12:31:39.061111+0000","last_active":"2024-09-21T12:32:25.729300+0000","last_peered":"2024-09-21T12:32:25.729300+0000","last_clean":"2024-09-21T12:32:25.729300+0000","last_became_active":"2024-09-21T12:31:39.060551+0000","last_became_peered":"2024-09-21T12:31:39.060551+0000","last_unstale":"2024-09-21T12:32:25.729300+0000","last_undegraded":"2024-09-21T12:32:25.729300+0000","last_fullsized":"2024-09-21T12:32:25.729300+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:12.581400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:31:12.581400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:31.372481+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1c","version":"76'79","reported_seq":265,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.837264+0000","last_change":"2024-09-21T12:27:28.374689+0000","last_active":"2024-09-21T12:32:25.837264+0000","last_peered":"2024-09-21T12:32:25.837264+0000","last_clean":"2024-09-21T12:32:25.837264+0000","last_became_active":"2024-09-21T12:27:16.920545+0000","last_became_peered":"2024-09-21T12:27:16.920545+0000","last_unstale":"2024-09-21T12:32:25.837264+0000","last_undegraded":"2024-09-21T12:32:25.837264+0000","last_fullsized":"2024-09-21T12:32:25.837264+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:28.374616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:28.374616+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:11.168406+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":577,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.614480+0000","last_change":"2024-09-21T12:32:17.528064+0000","last_active":"2024-09-21T12:32:21.614480+0000","last_peered":"2024-09-21T12:32:21.614480+0000","last_clean":"2024-09-21T12:32:21.614480+0000","last_became_active":"2024-09-21T12:20:21.595301+0000","last_became_peered":"2024-09-21T12:20:21.595301+0000","last_unstale":"2024-09-21T12:32:21.614480+0000","last_undegraded":"2024-09-21T12:32:21.614480+0000","last_fullsized":"2024-09-21T12:32:21.614480+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:17.527992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:32:17.527992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:44.229295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":599,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.615206+0000","last_change":"2024-09-21T12:29:16.079291+0000","last_active":"2024-09-21T12:32:21.615206+0000","last_peered":"2024-09-21T12:32:21.615206+0000","last_clean":"2024-09-21T12:32:21.615206+0000","last_became_active":"2024-09-21T12:20:15.308655+0000","last_became_peered":"2024-09-21T12:20:15.308655+0000","last_unstale":"2024-09-21T12:32:21.615206+0000","last_undegraded":"2024-09-21T12:32:21.615206+0000","last_fullsized":"2024-09-21T12:32:21.615206+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:16.079221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:16.079221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":644,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.613881+0000","last_change":"2024-09-21T12:31:48.294386+0000","last_active":"2024-09-21T12:32:21.613881+0000","last_peered":"2024-09-21T12:32:21.613881+0000","last_clean":"2024-09-21T12:32:21.613881+0000","last_became_active":"2024-09-21T12:20:17.305220+0000","last_became_peered":"2024-09-21T12:20:17.305220+0000","last_unstale":"2024-09-21T12:32:21.613881+0000","last_undegraded":"2024-09-21T12:32:21.613881+0000","last_fullsized":"2024-09-21T12:32:21.613881+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:31:48.294300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:48.294300+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:53.699472+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":536,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.677768+0000","last_change":"2024-09-21T12:28:39.179737+0000","last_active":"2024-09-21T12:32:25.677768+0000","last_peered":"2024-09-21T12:32:25.677768+0000","last_clean":"2024-09-21T12:32:25.677768+0000","last_became_active":"2024-09-21T12:22:50.926240+0000","last_became_peered":"2024-09-21T12:22:50.926240+0000","last_unstale":"2024-09-21T12:32:25.677768+0000","last_undegraded":"2024-09-21T12:32:25.677768+0000","last_fullsized":"2024-09-21T12:32:25.677768+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:39.179665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:39.179665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:30.468144+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":448,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.678273+0000","last_change":"2024-09-21T12:31:38.380246+0000","last_active":"2024-09-21T12:32:25.678273+0000","last_peered":"2024-09-21T12:32:25.678273+0000","last_clean":"2024-09-21T12:32:25.678273+0000","last_became_active":"2024-09-21T12:26:54.083981+0000","last_became_peered":"2024-09-21T12:26:54.083981+0000","last_unstale":"2024-09-21T12:32:25.678273+0000","last_undegraded":"2024-09-21T12:32:25.678273+0000","last_fullsized":"2024-09-21T12:32:25.678273+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:38.380155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:31:38.380155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:04.416228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":588,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.614520+0000","last_change":"2024-09-21T12:28:06.589984+0000","last_active":"2024-09-21T12:32:21.614520+0000","last_peered":"2024-09-21T12:32:21.614520+0000","last_clean":"2024-09-21T12:32:21.614520+0000","last_became_active":"2024-09-21T12:22:50.927804+0000","last_became_peered":"2024-09-21T12:22:50.927804+0000","last_unstale":"2024-09-21T12:32:21.614520+0000","last_undegraded":"2024-09-21T12:32:21.614520+0000","last_fullsized":"2024-09-21T12:32:21.614520+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:28:06.589926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:06.589926+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:05.555607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":632,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.729257+0000","last_change":"2024-09-21T12:32:14.238277+0000","last_active":"2024-09-21T12:32:25.729257+0000","last_peered":"2024-09-21T12:32:25.729257+0000","last_clean":"2024-09-21T12:32:25.729257+0000","last_became_active":"2024-09-21T12:32:14.237803+0000","last_became_peered":"2024-09-21T12:32:14.237803+0000","last_unstale":"2024-09-21T12:32:25.729257+0000","last_undegraded":"2024-09-21T12:32:25.729257+0000","last_fullsized":"2024-09-21T12:32:25.729257+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:56.620241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:31:56.620241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:01.685704+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":590,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.025010+0000","last_change":"2024-09-21T12:27:54.959865+0000","last_active":"2024-09-21T12:32:22.025010+0000","last_peered":"2024-09-21T12:32:22.025010+0000","last_clean":"2024-09-21T12:32:22.025010+0000","last_became_active":"2024-09-21T12:20:21.309190+0000","last_became_peered":"2024-09-21T12:20:21.309190+0000","last_unstale":"2024-09-21T12:32:22.025010+0000","last_undegraded":"2024-09-21T12:32:22.025010+0000","last_fullsized":"2024-09-21T12:32:22.025010+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:54.959806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:27:54.959806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:06.710394+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"76'79","reported_seq":362,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.615000+0000","last_change":"2024-09-21T12:31:38.367318+0000","last_active":"2024-09-21T12:32:21.615000+0000","last_peered":"2024-09-21T12:32:21.615000+0000","last_clean":"2024-09-21T12:32:21.615000+0000","last_became_active":"2024-09-21T12:27:16.937711+0000","last_became_peered":"2024-09-21T12:27:16.937711+0000","last_unstale":"2024-09-21T12:32:21.615000+0000","last_undegraded":"2024-09-21T12:32:21.615000+0000","last_fullsized":"2024-09-21T12:32:21.615000+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:31:38.367273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:31:38.367273+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:41.463777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":438,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.026259+0000","last_change":"2024-09-21T12:28:33.644630+0000","last_active":"2024-09-21T12:32:22.026259+0000","last_peered":"2024-09-21T12:32:22.026259+0000","last_clean":"2024-09-21T12:32:22.026259+0000","last_became_active":"2024-09-21T12:26:54.085055+0000","last_became_peered":"2024-09-21T12:26:54.085055+0000","last_unstale":"2024-09-21T12:32:22.026259+0000","last_undegraded":"2024-09-21T12:32:22.026259+0000","last_fullsized":"2024-09-21T12:32:22.026259+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:33.644570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:33.644570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":594,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.614799+0000","last_change":"2024-09-21T12:31:39.064759+0000","last_active":"2024-09-21T12:32:21.614799+0000","last_peered":"2024-09-21T12:32:21.614799+0000","last_clean":"2024-09-21T12:32:21.614799+0000","last_became_active":"2024-09-21T12:31:39.055199+0000","last_became_peered":"2024-09-21T12:31:39.055199+0000","last_unstale":"2024-09-21T12:32:21.614799+0000","last_undegraded":"2024-09-21T12:32:21.614799+0000","last_fullsized":"2024-09-21T12:32:21.614799+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-21T12:29:35.982468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:29:35.982468+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":607,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.837396+0000","last_change":"2024-09-21T12:28:04.623499+0000","last_active":"2024-09-21T12:32:25.837396+0000","last_peered":"2024-09-21T12:32:25.837396+0000","last_clean":"2024-09-21T12:32:25.837396+0000","last_became_active":"2024-09-21T12:22:50.921780+0000","last_became_peered":"2024-09-21T12:22:50.921780+0000","last_unstale":"2024-09-21T12:32:25.837396+0000","last_undegraded":"2024-09-21T12:32:25.837396+0000","last_fullsized":"2024-09-21T12:32:25.837396+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:04.623389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:04.623389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":606,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.728697+0000","last_change":"2024-09-21T12:29:53.301655+0000","last_active":"2024-09-21T12:32:25.728697+0000","last_peered":"2024-09-21T12:32:25.728697+0000","last_clean":"2024-09-21T12:32:25.728697+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:32:25.728697+0000","last_undegraded":"2024-09-21T12:32:25.728697+0000","last_fullsized":"2024-09-21T12:32:25.728697+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:53.301583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:29:53.301583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:32.635151+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"76'79","reported_seq":425,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.614403+0000","last_change":"2024-09-21T12:27:22.289065+0000","last_active":"2024-09-21T12:32:21.614403+0000","last_peered":"2024-09-21T12:32:21.614403+0000","last_clean":"2024-09-21T12:32:21.614403+0000","last_became_active":"2024-09-21T12:27:16.957569+0000","last_became_peered":"2024-09-21T12:27:16.957569+0000","last_unstale":"2024-09-21T12:32:21.614403+0000","last_undegraded":"2024-09-21T12:32:21.614403+0000","last_fullsized":"2024-09-21T12:32:21.614403+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:22.288989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:22.288989+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:37.384920+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":236,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.614668+0000","last_change":"2024-09-21T12:29:05.826853+0000","last_active":"2024-09-21T12:32:21.614668+0000","last_peered":"2024-09-21T12:32:21.614668+0000","last_clean":"2024-09-21T12:32:21.614668+0000","last_became_active":"2024-09-21T12:29:05.826251+0000","last_became_peered":"2024-09-21T12:29:05.826251+0000","last_unstale":"2024-09-21T12:32:21.614668+0000","last_undegraded":"2024-09-21T12:32:21.614668+0000","last_fullsized":"2024-09-21T12:32:21.614668+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:44.845176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:44.845176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:25.947407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":645,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.025599+0000","last_change":"2024-09-21T12:31:39.056292+0000","last_active":"2024-09-21T12:32:22.025599+0000","last_peered":"2024-09-21T12:32:22.025599+0000","last_clean":"2024-09-21T12:32:22.025599+0000","last_became_active":"2024-09-21T12:31:39.055972+0000","last_became_peered":"2024-09-21T12:31:39.055972+0000","last_unstale":"2024-09-21T12:32:22.025599+0000","last_undegraded":"2024-09-21T12:32:22.025599+0000","last_fullsized":"2024-09-21T12:32:22.025599+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:16.148785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:16.148785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":646,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.614587+0000","last_change":"2024-09-21T12:31:41.717598+0000","last_active":"2024-09-21T12:32:21.614587+0000","last_peered":"2024-09-21T12:32:21.614587+0000","last_clean":"2024-09-21T12:32:21.614587+0000","last_became_active":"2024-09-21T12:20:17.307828+0000","last_became_peered":"2024-09-21T12:20:17.307828+0000","last_unstale":"2024-09-21T12:32:21.614587+0000","last_undegraded":"2024-09-21T12:32:21.614587+0000","last_fullsized":"2024-09-21T12:32:21.614587+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:31:41.717520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:41.717520+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:43.527174+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":620,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.025753+0000","last_change":"2024-09-21T12:28:06.636769+0000","last_active":"2024-09-21T12:32:22.025753+0000","last_peered":"2024-09-21T12:32:22.025753+0000","last_clean":"2024-09-21T12:32:22.025753+0000","last_became_active":"2024-09-21T12:25:46.905878+0000","last_became_peered":"2024-09-21T12:25:46.905878+0000","last_unstale":"2024-09-21T12:32:22.025753+0000","last_undegraded":"2024-09-21T12:32:22.025753+0000","last_fullsized":"2024-09-21T12:32:22.025753+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:06.636729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:06.636729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:01.978797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"76'79","reported_seq":538,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.728630+0000","last_change":"2024-09-21T12:32:22.709108+0000","last_active":"2024-09-21T12:32:25.728630+0000","last_peered":"2024-09-21T12:32:25.728630+0000","last_clean":"2024-09-21T12:32:25.728630+0000","last_became_active":"2024-09-21T12:27:16.927791+0000","last_became_peered":"2024-09-21T12:27:16.927791+0000","last_unstale":"2024-09-21T12:32:25.728630+0000","last_undegraded":"2024-09-21T12:32:25.728630+0000","last_fullsized":"2024-09-21T12:32:25.728630+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:22.709049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:22.709049+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:25.781647+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":599,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.615280+0000","last_change":"2024-09-21T12:32:02.033734+0000","last_active":"2024-09-21T12:32:21.615280+0000","last_peered":"2024-09-21T12:32:21.615280+0000","last_clean":"2024-09-21T12:32:21.615280+0000","last_became_active":"2024-09-21T12:32:02.033276+0000","last_became_peered":"2024-09-21T12:32:02.033276+0000","last_unstale":"2024-09-21T12:32:21.615280+0000","last_undegraded":"2024-09-21T12:32:21.615280+0000","last_fullsized":"2024-09-21T12:32:21.615280+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:45.880383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.880383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":532,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.024352+0000","last_change":"2024-09-21T12:28:45.726096+0000","last_active":"2024-09-21T12:32:22.024352+0000","last_peered":"2024-09-21T12:32:22.024352+0000","last_clean":"2024-09-21T12:32:22.024352+0000","last_became_active":"2024-09-21T12:22:50.925403+0000","last_became_peered":"2024-09-21T12:22:50.925403+0000","last_unstale":"2024-09-21T12:32:22.024352+0000","last_undegraded":"2024-09-21T12:32:22.024352+0000","last_fullsized":"2024-09-21T12:32:22.024352+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:45.725968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.725968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":682,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.614565+0000","last_change":"2024-09-21T12:28:30.620035+0000","last_active":"2024-09-21T12:32:21.614565+0000","last_peered":"2024-09-21T12:32:21.614565+0000","last_clean":"2024-09-21T12:32:21.614565+0000","last_became_active":"2024-09-21T12:22:50.912144+0000","last_became_peered":"2024-09-21T12:22:50.912144+0000","last_unstale":"2024-09-21T12:32:21.614565+0000","last_undegraded":"2024-09-21T12:32:21.614565+0000","last_fullsized":"2024-09-21T12:32:21.614565+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:28:30.619954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:30.619954+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":508,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.615222+0000","last_change":"2024-09-21T12:26:52.295004+0000","last_active":"2024-09-21T12:32:21.615222+0000","last_peered":"2024-09-21T12:32:21.615222+0000","last_clean":"2024-09-21T12:32:21.615222+0000","last_became_active":"2024-09-21T12:22:50.918326+0000","last_became_peered":"2024-09-21T12:22:50.918326+0000","last_unstale":"2024-09-21T12:32:21.615222+0000","last_undegraded":"2024-09-21T12:32:21.615222+0000","last_fullsized":"2024-09-21T12:32:21.615222+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:52.294931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:52.294931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:31.026551+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"76'79","reported_seq":330,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.729543+0000","last_change":"2024-09-21T12:31:08.533342+0000","last_active":"2024-09-21T12:32:25.729543+0000","last_peered":"2024-09-21T12:32:25.729543+0000","last_clean":"2024-09-21T12:32:25.729543+0000","last_became_active":"2024-09-21T12:27:16.933358+0000","last_became_peered":"2024-09-21T12:27:16.933358+0000","last_unstale":"2024-09-21T12:32:25.729543+0000","last_undegraded":"2024-09-21T12:32:25.729543+0000","last_fullsized":"2024-09-21T12:32:25.729543+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:31:08.533297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:31:08.533297+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:28.058118+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":575,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.837288+0000","last_change":"2024-09-21T12:23:51.065780+0000","last_active":"2024-09-21T12:32:25.837288+0000","last_peered":"2024-09-21T12:32:25.837288+0000","last_clean":"2024-09-21T12:32:25.837288+0000","last_became_active":"2024-09-21T12:20:21.312976+0000","last_became_peered":"2024-09-21T12:20:21.312976+0000","last_unstale":"2024-09-21T12:32:25.837288+0000","last_undegraded":"2024-09-21T12:32:25.837288+0000","last_fullsized":"2024-09-21T12:32:25.837288+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:23:51.065733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:23:51.065733+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:32.158902+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":634,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.025170+0000","last_change":"2024-09-21T12:27:56.597623+0000","last_active":"2024-09-21T12:32:22.025170+0000","last_peered":"2024-09-21T12:32:22.025170+0000","last_clean":"2024-09-21T12:32:22.025170+0000","last_became_active":"2024-09-21T12:22:50.910933+0000","last_became_peered":"2024-09-21T12:22:50.910933+0000","last_unstale":"2024-09-21T12:32:22.025170+0000","last_undegraded":"2024-09-21T12:32:22.025170+0000","last_fullsized":"2024-09-21T12:32:22.025170+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:27:56.597473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:27:56.597473+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:06.028462+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":684,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.026412+0000","last_change":"2024-09-21T12:32:14.255243+0000","last_active":"2024-09-21T12:32:22.026412+0000","last_peered":"2024-09-21T12:32:22.026412+0000","last_clean":"2024-09-21T12:32:22.026412+0000","last_became_active":"2024-09-21T12:32:14.238429+0000","last_became_peered":"2024-09-21T12:32:14.238429+0000","last_unstale":"2024-09-21T12:32:22.026412+0000","last_undegraded":"2024-09-21T12:32:22.026412+0000","last_fullsized":"2024-09-21T12:32:22.026412+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:31:11.543034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:11.543034+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:39.571277+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"335'14","reported_seq":41,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.837288+0000","last_change":"2024-09-21T12:32:17.332497+0000","last_active":"2024-09-21T12:32:25.837288+0000","last_peered":"2024-09-21T12:32:25.837288+0000","last_clean":"2024-09-21T12:32:25.837288+0000","last_became_active":"2024-09-21T12:32:14.236917+0000","last_became_peered":"2024-09-21T12:32:14.236917+0000","last_unstale":"2024-09-21T12:32:25.837288+0000","last_undegraded":"2024-09-21T12:32:25.837288+0000","last_fullsized":"2024-09-21T12:32:25.837288+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"335'14","last_scrub_stamp":"2024-09-21T12:32:17.332448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:32:17.332448+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:25.665397+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"76'79","reported_seq":309,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.613837+0000","last_change":"2024-09-21T12:28:29.446170+0000","last_active":"2024-09-21T12:32:21.613837+0000","last_peered":"2024-09-21T12:32:21.613837+0000","last_clean":"2024-09-21T12:32:21.613837+0000","last_became_active":"2024-09-21T12:27:14.943286+0000","last_became_peered":"2024-09-21T12:27:14.943286+0000","last_unstale":"2024-09-21T12:32:21.613837+0000","last_undegraded":"2024-09-21T12:32:21.613837+0000","last_fullsized":"2024-09-21T12:32:21.613837+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:29.446103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:29.446103+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:10.842450+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":476,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.614191+0000","last_change":"2024-09-21T12:31:56.722328+0000","last_active":"2024-09-21T12:32:21.614191+0000","last_peered":"2024-09-21T12:32:21.614191+0000","last_clean":"2024-09-21T12:32:21.614191+0000","last_became_active":"2024-09-21T12:23:46.910112+0000","last_became_peered":"2024-09-21T12:23:46.910112+0000","last_unstale":"2024-09-21T12:32:21.614191+0000","last_undegraded":"2024-09-21T12:32:21.614191+0000","last_fullsized":"2024-09-21T12:32:21.614191+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:31:56.722241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:56.722241+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:09.661988+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":58,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.613528+0000","last_change":"2024-09-21T12:31:50.039964+0000","last_active":"2024-09-21T12:32:21.613528+0000","last_peered":"2024-09-21T12:32:21.613528+0000","last_clean":"2024-09-21T12:32:21.613528+0000","last_became_active":"2024-09-21T12:31:50.039390+0000","last_became_peered":"2024-09-21T12:31:50.039390+0000","last_unstale":"2024-09-21T12:32:21.613528+0000","last_undegraded":"2024-09-21T12:32:21.613528+0000","last_fullsized":"2024-09-21T12:32:21.613528+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:05.580847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:05.580847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:04.521335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"210'14","reported_seq":344,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.025918+0000","last_change":"2024-09-21T12:26:54.091833+0000","last_active":"2024-09-21T12:32:22.025918+0000","last_peered":"2024-09-21T12:32:22.025918+0000","last_clean":"2024-09-21T12:32:22.025918+0000","last_became_active":"2024-09-21T12:26:54.091451+0000","last_became_peered":"2024-09-21T12:26:54.091451+0000","last_unstale":"2024-09-21T12:32:22.025918+0000","last_undegraded":"2024-09-21T12:32:22.025918+0000","last_fullsized":"2024-09-21T12:32:22.025918+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'13","last_scrub_stamp":"2024-09-21T12:25:33.474391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:33.474391+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:29.516680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":80,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.025656+0000","last_change":"2024-09-21T12:31:39.062941+0000","last_active":"2024-09-21T12:32:22.025656+0000","last_peered":"2024-09-21T12:32:22.025656+0000","last_clean":"2024-09-21T12:32:22.025656+0000","last_became_active":"2024-09-21T12:31:39.054600+0000","last_became_peered":"2024-09-21T12:31:39.054600+0000","last_unstale":"2024-09-21T12:32:22.025656+0000","last_undegraded":"2024-09-21T12:32:22.025656+0000","last_fullsized":"2024-09-21T12:32:22.025656+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:28:31.580245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:31.580245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:09.036420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"76'79","reported_seq":515,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.614317+0000","last_change":"2024-09-21T12:27:20.527090+0000","last_active":"2024-09-21T12:32:21.614317+0000","last_peered":"2024-09-21T12:32:21.614317+0000","last_clean":"2024-09-21T12:32:21.614317+0000","last_became_active":"2024-09-21T12:27:14.941905+0000","last_became_peered":"2024-09-21T12:27:14.941905+0000","last_unstale":"2024-09-21T12:32:21.614317+0000","last_undegraded":"2024-09-21T12:32:21.614317+0000","last_fullsized":"2024-09-21T12:32:21.614317+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:20.526951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:20.526951+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:07.843874+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":695,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.613831+0000","last_change":"2024-09-21T12:32:14.262629+0000","last_active":"2024-09-21T12:32:21.613831+0000","last_peered":"2024-09-21T12:32:21.613831+0000","last_clean":"2024-09-21T12:32:21.613831+0000","last_became_active":"2024-09-21T12:32:14.242826+0000","last_became_peered":"2024-09-21T12:32:14.242826+0000","last_unstale":"2024-09-21T12:32:21.613831+0000","last_undegraded":"2024-09-21T12:32:21.613831+0000","last_fullsized":"2024-09-21T12:32:21.613831+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-21T12:28:26.632272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:26.632272+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":590,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.615512+0000","last_change":"2024-09-21T12:29:10.089053+0000","last_active":"2024-09-21T12:32:21.615512+0000","last_peered":"2024-09-21T12:32:21.615512+0000","last_clean":"2024-09-21T12:32:21.615512+0000","last_became_active":"2024-09-21T12:20:15.297340+0000","last_became_peered":"2024-09-21T12:20:15.297340+0000","last_unstale":"2024-09-21T12:32:21.615512+0000","last_undegraded":"2024-09-21T12:32:21.615512+0000","last_fullsized":"2024-09-21T12:32:21.615512+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:10.088986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:10.088986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":547,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.614247+0000","last_change":"2024-09-21T12:32:14.239263+0000","last_active":"2024-09-21T12:32:21.614247+0000","last_peered":"2024-09-21T12:32:21.614247+0000","last_clean":"2024-09-21T12:32:21.614247+0000","last_became_active":"2024-09-21T12:32:14.238820+0000","last_became_peered":"2024-09-21T12:32:14.238820+0000","last_unstale":"2024-09-21T12:32:21.614247+0000","last_undegraded":"2024-09-21T12:32:21.614247+0000","last_fullsized":"2024-09-21T12:32:21.614247+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:33.697208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:33.697208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":532,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.613872+0000","last_change":"2024-09-21T12:29:05.143941+0000","last_active":"2024-09-21T12:32:21.613872+0000","last_peered":"2024-09-21T12:32:21.613872+0000","last_clean":"2024-09-21T12:32:21.613872+0000","last_became_active":"2024-09-21T12:22:50.918940+0000","last_became_peered":"2024-09-21T12:22:50.918940+0000","last_unstale":"2024-09-21T12:32:21.613872+0000","last_undegraded":"2024-09-21T12:32:21.613872+0000","last_fullsized":"2024-09-21T12:32:21.613872+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:05.143736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:29:05.143736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"76'79","reported_seq":517,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.615108+0000","last_change":"2024-09-21T12:26:45.354836+0000","last_active":"2024-09-21T12:32:21.615108+0000","last_peered":"2024-09-21T12:32:21.615108+0000","last_clean":"2024-09-21T12:32:21.615108+0000","last_became_active":"2024-09-21T12:26:44.960093+0000","last_became_peered":"2024-09-21T12:26:44.960093+0000","last_unstale":"2024-09-21T12:32:21.615108+0000","last_undegraded":"2024-09-21T12:32:21.615108+0000","last_fullsized":"2024-09-21T12:32:21.615108+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:45.354774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:45.354774+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:30.011873+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":614,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.729151+0000","last_change":"2024-09-21T12:26:51.463759+0000","last_active":"2024-09-21T12:32:25.729151+0000","last_peered":"2024-09-21T12:32:25.729151+0000","last_clean":"2024-09-21T12:32:25.729151+0000","last_became_active":"2024-09-21T12:22:50.926008+0000","last_became_peered":"2024-09-21T12:22:50.926008+0000","last_unstale":"2024-09-21T12:32:25.729151+0000","last_undegraded":"2024-09-21T12:32:25.729151+0000","last_fullsized":"2024-09-21T12:32:25.729151+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-21T12:26:51.463698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:51.463698+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:35.628660+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":670,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.026585+0000","last_change":"2024-09-21T12:32:14.247441+0000","last_active":"2024-09-21T12:32:22.026585+0000","last_peered":"2024-09-21T12:32:22.026585+0000","last_clean":"2024-09-21T12:32:22.026585+0000","last_became_active":"2024-09-21T12:32:14.240850+0000","last_became_peered":"2024-09-21T12:32:14.240850+0000","last_unstale":"2024-09-21T12:32:22.026585+0000","last_undegraded":"2024-09-21T12:32:22.026585+0000","last_fullsized":"2024-09-21T12:32:22.026585+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:28:32.727727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:32.727727+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"121'11","reported_seq":769,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.613656+0000","last_change":"2024-09-21T12:26:14.490045+0000","last_active":"2024-09-21T12:32:21.613656+0000","last_peered":"2024-09-21T12:32:21.613656+0000","last_clean":"2024-09-21T12:32:21.613656+0000","last_became_active":"2024-09-21T12:22:50.930342+0000","last_became_peered":"2024-09-21T12:22:50.930342+0000","last_unstale":"2024-09-21T12:32:21.613656+0000","last_undegraded":"2024-09-21T12:32:21.613656+0000","last_fullsized":"2024-09-21T12:32:21.613656+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:26:14.489972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:14.489972+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:21.811951+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":583,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.836972+0000","last_change":"2024-09-21T12:28:37.686403+0000","last_active":"2024-09-21T12:32:25.836972+0000","last_peered":"2024-09-21T12:32:25.836972+0000","last_clean":"2024-09-21T12:32:25.836972+0000","last_became_active":"2024-09-21T12:20:21.593367+0000","last_became_peered":"2024-09-21T12:20:21.593367+0000","last_unstale":"2024-09-21T12:32:25.836972+0000","last_undegraded":"2024-09-21T12:32:25.836972+0000","last_fullsized":"2024-09-21T12:32:25.836972+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:37.686350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:37.686350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"76'79","reported_seq":66,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.615439+0000","last_change":"2024-09-21T12:31:50.320881+0000","last_active":"2024-09-21T12:32:21.615439+0000","last_peered":"2024-09-21T12:32:21.615439+0000","last_clean":"2024-09-21T12:32:21.615439+0000","last_became_active":"2024-09-21T12:31:50.040087+0000","last_became_peered":"2024-09-21T12:31:50.040087+0000","last_unstale":"2024-09-21T12:32:21.615439+0000","last_undegraded":"2024-09-21T12:32:21.615439+0000","last_fullsized":"2024-09-21T12:32:21.615439+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:31:50.320834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:31:50.320834+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:17.807274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":115,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.024027+0000","last_change":"2024-09-21T12:32:02.054999+0000","last_active":"2024-09-21T12:32:22.024027+0000","last_peered":"2024-09-21T12:32:22.024027+0000","last_clean":"2024-09-21T12:32:22.024027+0000","last_became_active":"2024-09-21T12:32:02.037278+0000","last_became_peered":"2024-09-21T12:32:02.037278+0000","last_unstale":"2024-09-21T12:32:22.024027+0000","last_undegraded":"2024-09-21T12:32:22.024027+0000","last_fullsized":"2024-09-21T12:32:22.024027+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:31:59.022007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:59.022007+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:11.299729+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":619,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.729021+0000","last_change":"2024-09-21T12:32:19.629135+0000","last_active":"2024-09-21T12:32:25.729021+0000","last_peered":"2024-09-21T12:32:25.729021+0000","last_clean":"2024-09-21T12:32:25.729021+0000","last_became_active":"2024-09-21T12:20:15.308518+0000","last_became_peered":"2024-09-21T12:20:15.308518+0000","last_unstale":"2024-09-21T12:32:25.729021+0000","last_undegraded":"2024-09-21T12:32:25.729021+0000","last_fullsized":"2024-09-21T12:32:25.729021+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:19.629083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:19.629083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:40.110418+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":752,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.026733+0000","last_change":"2024-09-21T12:27:58.871800+0000","last_active":"2024-09-21T12:32:22.026733+0000","last_peered":"2024-09-21T12:32:22.026733+0000","last_clean":"2024-09-21T12:32:22.026733+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:32:22.026733+0000","last_undegraded":"2024-09-21T12:32:22.026733+0000","last_fullsized":"2024-09-21T12:32:22.026733+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:27:58.871710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:27:58.871710+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:08.726110+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":591,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.024793+0000","last_change":"2024-09-21T12:28:37.850166+0000","last_active":"2024-09-21T12:32:22.024793+0000","last_peered":"2024-09-21T12:32:22.024793+0000","last_clean":"2024-09-21T12:32:22.024793+0000","last_became_active":"2024-09-21T12:20:21.315801+0000","last_became_peered":"2024-09-21T12:20:21.315801+0000","last_unstale":"2024-09-21T12:32:22.024793+0000","last_undegraded":"2024-09-21T12:32:22.024793+0000","last_fullsized":"2024-09-21T12:32:22.024793+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:37.850100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:37.850100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"76'79","reported_seq":277,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.026031+0000","last_change":"2024-09-21T12:29:08.081975+0000","last_active":"2024-09-21T12:32:22.026031+0000","last_peered":"2024-09-21T12:32:22.026031+0000","last_clean":"2024-09-21T12:32:22.026031+0000","last_became_active":"2024-09-21T12:29:05.835884+0000","last_became_peered":"2024-09-21T12:29:05.835884+0000","last_unstale":"2024-09-21T12:32:22.026031+0000","last_undegraded":"2024-09-21T12:32:22.026031+0000","last_fullsized":"2024-09-21T12:32:22.026031+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:08.081901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:08.081901+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":642,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.026633+0000","last_change":"2024-09-21T12:26:14.834328+0000","last_active":"2024-09-21T12:32:22.026633+0000","last_peered":"2024-09-21T12:32:22.026633+0000","last_clean":"2024-09-21T12:32:22.026633+0000","last_became_active":"2024-09-21T12:20:17.693067+0000","last_became_peered":"2024-09-21T12:20:17.693067+0000","last_unstale":"2024-09-21T12:32:22.026633+0000","last_undegraded":"2024-09-21T12:32:22.026633+0000","last_fullsized":"2024-09-21T12:32:22.026633+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-21T12:26:14.834236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:14.834236+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:37.533965+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":585,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.026180+0000","last_change":"2024-09-21T12:26:12.131784+0000","last_active":"2024-09-21T12:32:22.026180+0000","last_peered":"2024-09-21T12:32:22.026180+0000","last_clean":"2024-09-21T12:32:22.026180+0000","last_became_active":"2024-09-21T12:20:15.308441+0000","last_became_peered":"2024-09-21T12:20:15.308441+0000","last_unstale":"2024-09-21T12:32:22.026180+0000","last_undegraded":"2024-09-21T12:32:22.026180+0000","last_fullsized":"2024-09-21T12:32:22.026180+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:12.131721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:12.131721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:34.057507+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":181,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.024088+0000","last_change":"2024-09-21T12:29:05.826167+0000","last_active":"2024-09-21T12:32:22.024088+0000","last_peered":"2024-09-21T12:32:22.024088+0000","last_clean":"2024-09-21T12:32:22.024088+0000","last_became_active":"2024-09-21T12:29:05.825869+0000","last_became_peered":"2024-09-21T12:29:05.825869+0000","last_unstale":"2024-09-21T12:32:22.024088+0000","last_undegraded":"2024-09-21T12:32:22.024088+0000","last_fullsized":"2024-09-21T12:32:22.024088+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:48.999878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:48.999878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":581,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.024808+0000","last_change":"2024-09-21T12:31:11.827090+0000","last_active":"2024-09-21T12:32:22.024808+0000","last_peered":"2024-09-21T12:32:22.024808+0000","last_clean":"2024-09-21T12:32:22.024808+0000","last_became_active":"2024-09-21T12:20:21.592828+0000","last_became_peered":"2024-09-21T12:20:21.592828+0000","last_unstale":"2024-09-21T12:32:22.024808+0000","last_undegraded":"2024-09-21T12:32:22.024808+0000","last_fullsized":"2024-09-21T12:32:22.024808+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:11.827011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:31:11.827011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:33.068131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"76'79","reported_seq":385,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.024267+0000","last_change":"2024-09-21T12:28:25.714131+0000","last_active":"2024-09-21T12:32:22.024267+0000","last_peered":"2024-09-21T12:32:22.024267+0000","last_clean":"2024-09-21T12:32:22.024267+0000","last_became_active":"2024-09-21T12:26:54.092726+0000","last_became_peered":"2024-09-21T12:26:54.092726+0000","last_unstale":"2024-09-21T12:32:22.024267+0000","last_undegraded":"2024-09-21T12:32:22.024267+0000","last_fullsized":"2024-09-21T12:32:22.024267+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:25.714084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:25.714084+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"276'40","reported_seq":793,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.026079+0000","last_change":"2024-09-21T12:26:09.082395+0000","last_active":"2024-09-21T12:32:22.026079+0000","last_peered":"2024-09-21T12:32:22.026079+0000","last_clean":"2024-09-21T12:32:22.026079+0000","last_became_active":"2024-09-21T12:20:17.307769+0000","last_became_peered":"2024-09-21T12:20:17.307769+0000","last_unstale":"2024-09-21T12:32:22.026079+0000","last_undegraded":"2024-09-21T12:32:22.026079+0000","last_fullsized":"2024-09-21T12:32:22.026079+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-21T12:26:09.082314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:09.082314+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:25.018714+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":527,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.026483+0000","last_change":"2024-09-21T12:28:31.738463+0000","last_active":"2024-09-21T12:32:22.026483+0000","last_peered":"2024-09-21T12:32:22.026483+0000","last_clean":"2024-09-21T12:32:22.026483+0000","last_became_active":"2024-09-21T12:22:50.922002+0000","last_became_peered":"2024-09-21T12:22:50.922002+0000","last_unstale":"2024-09-21T12:32:22.026483+0000","last_undegraded":"2024-09-21T12:32:22.026483+0000","last_fullsized":"2024-09-21T12:32:22.026483+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:31.738415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:31.738415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":319,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.024278+0000","last_change":"2024-09-21T12:32:18.988488+0000","last_active":"2024-09-21T12:32:22.024278+0000","last_peered":"2024-09-21T12:32:22.024278+0000","last_clean":"2024-09-21T12:32:22.024278+0000","last_became_active":"2024-09-21T12:26:54.083584+0000","last_became_peered":"2024-09-21T12:26:54.083584+0000","last_unstale":"2024-09-21T12:32:22.024278+0000","last_undegraded":"2024-09-21T12:32:22.024278+0000","last_fullsized":"2024-09-21T12:32:22.024278+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T12:32:18.988439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:32:18.988439+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:22.692523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":624,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.024833+0000","last_change":"2024-09-21T12:31:49.662004+0000","last_active":"2024-09-21T12:32:22.024833+0000","last_peered":"2024-09-21T12:32:22.024833+0000","last_clean":"2024-09-21T12:32:22.024833+0000","last_became_active":"2024-09-21T12:22:50.921488+0000","last_became_peered":"2024-09-21T12:22:50.921488+0000","last_unstale":"2024-09-21T12:32:22.024833+0000","last_undegraded":"2024-09-21T12:32:22.024833+0000","last_fullsized":"2024-09-21T12:32:22.024833+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:49.661957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:31:49.661957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:07.929316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":607,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.620812+0000","last_change":"2024-09-21T12:32:21.620812+0000","last_active":"2024-09-21T12:32:21.620812+0000","last_peered":"2024-09-21T12:32:21.620812+0000","last_clean":"2024-09-21T12:32:21.620812+0000","last_became_active":"2024-09-21T12:22:50.917558+0000","last_became_peered":"2024-09-21T12:22:50.917558+0000","last_unstale":"2024-09-21T12:32:21.620812+0000","last_undegraded":"2024-09-21T12:32:21.620812+0000","last_fullsized":"2024-09-21T12:32:21.620812+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:21.620769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:32:21.620769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:41.202331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'79","reported_seq":26,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.836836+0000","last_change":"2024-09-21T12:32:14.238048+0000","last_active":"2024-09-21T12:32:25.836836+0000","last_peered":"2024-09-21T12:32:25.836836+0000","last_clean":"2024-09-21T12:32:25.836836+0000","last_became_active":"2024-09-21T12:32:14.237652+0000","last_became_peered":"2024-09-21T12:32:14.237652+0000","last_unstale":"2024-09-21T12:32:25.836836+0000","last_undegraded":"2024-09-21T12:32:25.836836+0000","last_fullsized":"2024-09-21T12:32:25.836836+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:30:39.179014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:30:39.179014+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":88,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.026252+0000","last_change":"2024-09-21T12:31:39.072454+0000","last_active":"2024-09-21T12:32:22.026252+0000","last_peered":"2024-09-21T12:32:22.026252+0000","last_clean":"2024-09-21T12:32:22.026252+0000","last_became_active":"2024-09-21T12:31:39.057316+0000","last_became_peered":"2024-09-21T12:31:39.057316+0000","last_unstale":"2024-09-21T12:32:22.026252+0000","last_undegraded":"2024-09-21T12:32:22.026252+0000","last_fullsized":"2024-09-21T12:32:22.026252+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-21T12:28:25.549320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:25.549320+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:10.042213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":618,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.023758+0000","last_change":"2024-09-21T12:32:21.077501+0000","last_active":"2024-09-21T12:32:22.023758+0000","last_peered":"2024-09-21T12:32:22.023758+0000","last_clean":"2024-09-21T12:32:22.023758+0000","last_became_active":"2024-09-21T12:20:15.308356+0000","last_became_peered":"2024-09-21T12:20:15.308356+0000","last_unstale":"2024-09-21T12:32:22.023758+0000","last_undegraded":"2024-09-21T12:32:22.023758+0000","last_fullsized":"2024-09-21T12:32:22.023758+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:21.077447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:21.077447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:47.493927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":625,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.613899+0000","last_change":"2024-09-21T12:32:20.845517+0000","last_active":"2024-09-21T12:32:21.613899+0000","last_peered":"2024-09-21T12:32:21.613899+0000","last_clean":"2024-09-21T12:32:21.613899+0000","last_became_active":"2024-09-21T12:20:14.288358+0000","last_became_peered":"2024-09-21T12:20:14.288358+0000","last_unstale":"2024-09-21T12:32:21.613899+0000","last_undegraded":"2024-09-21T12:32:21.613899+0000","last_fullsized":"2024-09-21T12:32:21.613899+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:32:20.845428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:32:20.845428+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:48.913124+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0019042449999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":600,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.614400+0000","last_change":"2024-09-21T12:29:13.200503+0000","last_active":"2024-09-21T12:32:21.614400+0000","last_peered":"2024-09-21T12:32:21.614400+0000","last_clean":"2024-09-21T12:32:21.614400+0000","last_became_active":"2024-09-21T12:20:19.309724+0000","last_became_peered":"2024-09-21T12:20:19.309724+0000","last_unstale":"2024-09-21T12:32:21.614400+0000","last_undegraded":"2024-09-21T12:32:21.614400+0000","last_fullsized":"2024-09-21T12:32:21.614400+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:13.200321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:29:13.200321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":583,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.837557+0000","last_change":"2024-09-21T12:32:24.398751+0000","last_active":"2024-09-21T12:32:25.837557+0000","last_peered":"2024-09-21T12:32:25.837557+0000","last_clean":"2024-09-21T12:32:25.837557+0000","last_became_active":"2024-09-21T12:20:21.592982+0000","last_became_peered":"2024-09-21T12:20:21.592982+0000","last_unstale":"2024-09-21T12:32:25.837557+0000","last_undegraded":"2024-09-21T12:32:25.837557+0000","last_fullsized":"2024-09-21T12:32:25.837557+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:24.398686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:32:24.398686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:27.211671+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"76'79","reported_seq":543,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.025298+0000","last_change":"2024-09-21T12:29:11.678268+0000","last_active":"2024-09-21T12:32:22.025298+0000","last_peered":"2024-09-21T12:32:22.025298+0000","last_clean":"2024-09-21T12:32:22.025298+0000","last_became_active":"2024-09-21T12:26:44.949867+0000","last_became_peered":"2024-09-21T12:26:44.949867+0000","last_unstale":"2024-09-21T12:32:22.025298+0000","last_undegraded":"2024-09-21T12:32:22.025298+0000","last_fullsized":"2024-09-21T12:32:22.025298+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:11.677923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:11.677923+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":635,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.026470+0000","last_change":"2024-09-21T12:28:45.783033+0000","last_active":"2024-09-21T12:32:22.026470+0000","last_peered":"2024-09-21T12:32:22.026470+0000","last_clean":"2024-09-21T12:32:22.026470+0000","last_became_active":"2024-09-21T12:20:17.299325+0000","last_became_peered":"2024-09-21T12:20:17.299325+0000","last_unstale":"2024-09-21T12:32:22.026470+0000","last_undegraded":"2024-09-21T12:32:22.026470+0000","last_fullsized":"2024-09-21T12:32:22.026470+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:28:45.782955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.782955+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":638,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.023762+0000","last_change":"2024-09-21T12:31:50.041678+0000","last_active":"2024-09-21T12:32:22.023762+0000","last_peered":"2024-09-21T12:32:22.023762+0000","last_clean":"2024-09-21T12:32:22.023762+0000","last_became_active":"2024-09-21T12:31:50.041367+0000","last_became_peered":"2024-09-21T12:31:50.041367+0000","last_unstale":"2024-09-21T12:32:22.023762+0000","last_undegraded":"2024-09-21T12:32:22.023762+0000","last_fullsized":"2024-09-21T12:32:22.023762+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:15.115449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:15.115449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":1340,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.836920+0000","last_change":"2024-09-21T12:28:34.639438+0000","last_active":"2024-09-21T12:32:25.836920+0000","last_peered":"2024-09-21T12:32:25.836920+0000","last_clean":"2024-09-21T12:32:25.836920+0000","last_became_active":"2024-09-21T12:20:14.287543+0000","last_became_peered":"2024-09-21T12:20:14.287543+0000","last_unstale":"2024-09-21T12:32:25.836920+0000","last_undegraded":"2024-09-21T12:32:25.836920+0000","last_fullsized":"2024-09-21T12:32:25.836920+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-21T12:28:34.639361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:28:34.639361+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":725,"num_read_kb":720,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":42,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.729335+0000","last_change":"2024-09-21T12:32:02.030536+0000","last_active":"2024-09-21T12:32:25.729335+0000","last_peered":"2024-09-21T12:32:25.729335+0000","last_clean":"2024-09-21T12:32:25.729335+0000","last_became_active":"2024-09-21T12:32:02.030240+0000","last_became_peered":"2024-09-21T12:32:02.030240+0000","last_unstale":"2024-09-21T12:32:25.729335+0000","last_undegraded":"2024-09-21T12:32:25.729335+0000","last_fullsized":"2024-09-21T12:32:25.729335+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:11.422293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:11.422293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:12.691461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":76,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.025831+0000","last_change":"2024-09-21T12:31:39.060916+0000","last_active":"2024-09-21T12:32:22.025831+0000","last_peered":"2024-09-21T12:32:22.025831+0000","last_clean":"2024-09-21T12:32:22.025831+0000","last_became_active":"2024-09-21T12:31:39.060160+0000","last_became_peered":"2024-09-21T12:31:39.060160+0000","last_unstale":"2024-09-21T12:32:22.025831+0000","last_undegraded":"2024-09-21T12:32:22.025831+0000","last_fullsized":"2024-09-21T12:32:22.025831+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:30:40.556857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:30:40.556857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"276'32","reported_seq":294,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.837722+0000","last_change":"2024-09-21T12:29:09.625844+0000","last_active":"2024-09-21T12:32:25.837722+0000","last_peered":"2024-09-21T12:32:25.837722+0000","last_clean":"2024-09-21T12:32:25.837722+0000","last_became_active":"2024-09-21T12:29:05.827560+0000","last_became_peered":"2024-09-21T12:29:05.827560+0000","last_unstale":"2024-09-21T12:32:25.837722+0000","last_undegraded":"2024-09-21T12:32:25.837722+0000","last_fullsized":"2024-09-21T12:32:25.837722+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'20","last_scrub_stamp":"2024-09-21T12:29:09.625763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:29:09.625763+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":604,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.614675+0000","last_change":"2024-09-21T12:32:19.534660+0000","last_active":"2024-09-21T12:32:21.614675+0000","last_peered":"2024-09-21T12:32:21.614675+0000","last_clean":"2024-09-21T12:32:21.614675+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:32:21.614675+0000","last_undegraded":"2024-09-21T12:32:21.614675+0000","last_fullsized":"2024-09-21T12:32:21.614675+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:32:19.534573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:19.534573+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:34.017643+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":680,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.836952+0000","last_change":"2024-09-21T12:26:49.248015+0000","last_active":"2024-09-21T12:32:25.836952+0000","last_peered":"2024-09-21T12:32:25.836952+0000","last_clean":"2024-09-21T12:32:25.836952+0000","last_became_active":"2024-09-21T12:26:44.950132+0000","last_became_peered":"2024-09-21T12:26:44.950132+0000","last_unstale":"2024-09-21T12:32:25.836952+0000","last_undegraded":"2024-09-21T12:32:25.836952+0000","last_fullsized":"2024-09-21T12:32:25.836952+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:26:49.247952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:26:49.247952+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:02.161443+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":26,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.836949+0000","last_change":"2024-09-21T12:32:14.233428+0000","last_active":"2024-09-21T12:32:25.836949+0000","last_peered":"2024-09-21T12:32:25.836949+0000","last_clean":"2024-09-21T12:32:25.836949+0000","last_became_active":"2024-09-21T12:32:14.233034+0000","last_became_peered":"2024-09-21T12:32:14.233034+0000","last_unstale":"2024-09-21T12:32:25.836949+0000","last_undegraded":"2024-09-21T12:32:25.836949+0000","last_fullsized":"2024-09-21T12:32:25.836949+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:36.684459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:36.684459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":439,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.615259+0000","last_change":"2024-09-21T12:31:39.057057+0000","last_active":"2024-09-21T12:32:21.615259+0000","last_peered":"2024-09-21T12:32:21.615259+0000","last_clean":"2024-09-21T12:32:21.615259+0000","last_became_active":"2024-09-21T12:31:39.055947+0000","last_became_peered":"2024-09-21T12:31:39.055947+0000","last_unstale":"2024-09-21T12:32:21.615259+0000","last_undegraded":"2024-09-21T12:32:21.615259+0000","last_fullsized":"2024-09-21T12:32:21.615259+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:28.647287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:28.647287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"76'79","reported_seq":547,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.025426+0000","last_change":"2024-09-21T12:31:03.884522+0000","last_active":"2024-09-21T12:32:22.025426+0000","last_peered":"2024-09-21T12:32:22.025426+0000","last_clean":"2024-09-21T12:32:22.025426+0000","last_became_active":"2024-09-21T12:27:14.954351+0000","last_became_peered":"2024-09-21T12:27:14.954351+0000","last_unstale":"2024-09-21T12:32:22.025426+0000","last_undegraded":"2024-09-21T12:32:22.025426+0000","last_fullsized":"2024-09-21T12:32:22.025426+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:31:03.884464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:31:03.884464+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:24.047755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":96,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.025578+0000","last_change":"2024-09-21T12:32:00.043176+0000","last_active":"2024-09-21T12:32:22.025578+0000","last_peered":"2024-09-21T12:32:22.025578+0000","last_clean":"2024-09-21T12:32:22.025578+0000","last_became_active":"2024-09-21T12:31:39.052759+0000","last_became_peered":"2024-09-21T12:31:39.052759+0000","last_unstale":"2024-09-21T12:32:22.025578+0000","last_undegraded":"2024-09-21T12:32:22.025578+0000","last_fullsized":"2024-09-21T12:32:22.025578+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:32:00.043108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:00.043108+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:01.713101+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":515,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.613762+0000","last_change":"2024-09-21T12:27:30.501480+0000","last_active":"2024-09-21T12:32:21.613762+0000","last_peered":"2024-09-21T12:32:21.613762+0000","last_clean":"2024-09-21T12:32:21.613762+0000","last_became_active":"2024-09-21T12:22:50.908060+0000","last_became_peered":"2024-09-21T12:22:50.908060+0000","last_unstale":"2024-09-21T12:32:21.613762+0000","last_undegraded":"2024-09-21T12:32:21.613762+0000","last_fullsized":"2024-09-21T12:32:21.613762+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:30.498831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:27:30.498831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:06.870075+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":343,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.847392+0000","last_change":"2024-09-21T12:32:21.847392+0000","last_active":"2024-09-21T12:32:21.847392+0000","last_peered":"2024-09-21T12:32:21.847392+0000","last_clean":"2024-09-21T12:32:21.847392+0000","last_became_active":"2024-09-21T12:29:05.825426+0000","last_became_peered":"2024-09-21T12:29:05.825426+0000","last_unstale":"2024-09-21T12:32:21.847392+0000","last_undegraded":"2024-09-21T12:32:21.847392+0000","last_fullsized":"2024-09-21T12:32:21.847392+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:21.847336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:32:21.847336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:22.367090+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":578,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.615116+0000","last_change":"2024-09-21T12:28:39.853931+0000","last_active":"2024-09-21T12:32:21.615116+0000","last_peered":"2024-09-21T12:32:21.615116+0000","last_clean":"2024-09-21T12:32:21.615116+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:32:21.615116+0000","last_undegraded":"2024-09-21T12:32:21.615116+0000","last_fullsized":"2024-09-21T12:32:21.615116+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:39.853791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:39.853791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"76'79","reported_seq":315,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.613531+0000","last_change":"2024-09-21T12:32:02.027640+0000","last_active":"2024-09-21T12:32:21.613531+0000","last_peered":"2024-09-21T12:32:21.613531+0000","last_clean":"2024-09-21T12:32:21.613531+0000","last_became_active":"2024-09-21T12:32:02.026943+0000","last_became_peered":"2024-09-21T12:32:02.026943+0000","last_unstale":"2024-09-21T12:32:21.613531+0000","last_undegraded":"2024-09-21T12:32:21.613531+0000","last_fullsized":"2024-09-21T12:32:21.613531+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:30:38.611133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:30:38.611133+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":669,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.614505+0000","last_change":"2024-09-21T12:28:46.355910+0000","last_active":"2024-09-21T12:32:21.614505+0000","last_peered":"2024-09-21T12:32:21.614505+0000","last_clean":"2024-09-21T12:32:21.614505+0000","last_became_active":"2024-09-21T12:23:46.897197+0000","last_became_peered":"2024-09-21T12:23:46.897197+0000","last_unstale":"2024-09-21T12:32:21.614505+0000","last_undegraded":"2024-09-21T12:32:21.614505+0000","last_fullsized":"2024-09-21T12:32:21.614505+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-21T12:28:46.355822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:46.355822+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:11.864750+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":76,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.024117+0000","last_change":"2024-09-21T12:31:39.060456+0000","last_active":"2024-09-21T12:32:22.024117+0000","last_peered":"2024-09-21T12:32:22.024117+0000","last_clean":"2024-09-21T12:32:22.024117+0000","last_became_active":"2024-09-21T12:31:39.059854+0000","last_became_peered":"2024-09-21T12:31:39.059854+0000","last_unstale":"2024-09-21T12:32:22.024117+0000","last_undegraded":"2024-09-21T12:32:22.024117+0000","last_fullsized":"2024-09-21T12:32:22.024117+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:06.236749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:06.236749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:08.037262+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":606,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.025922+0000","last_change":"2024-09-21T12:28:03.571042+0000","last_active":"2024-09-21T12:32:22.025922+0000","last_peered":"2024-09-21T12:32:22.025922+0000","last_clean":"2024-09-21T12:32:22.025922+0000","last_became_active":"2024-09-21T12:22:50.927438+0000","last_became_peered":"2024-09-21T12:22:50.927438+0000","last_unstale":"2024-09-21T12:32:22.025922+0000","last_undegraded":"2024-09-21T12:32:22.025922+0000","last_fullsized":"2024-09-21T12:32:22.025922+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:03.570984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:03.570984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:00.985882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":593,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.614836+0000","last_change":"2024-09-21T12:25:39.079146+0000","last_active":"2024-09-21T12:32:21.614836+0000","last_peered":"2024-09-21T12:32:21.614836+0000","last_clean":"2024-09-21T12:32:21.614836+0000","last_became_active":"2024-09-21T12:22:50.913898+0000","last_became_peered":"2024-09-21T12:22:50.913898+0000","last_unstale":"2024-09-21T12:32:21.614836+0000","last_undegraded":"2024-09-21T12:32:21.614836+0000","last_fullsized":"2024-09-21T12:32:21.614836+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:39.079015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:39.079015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:29.619606+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"76'79","reported_seq":677,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.836800+0000","last_change":"2024-09-21T12:29:05.501052+0000","last_active":"2024-09-21T12:32:25.836800+0000","last_peered":"2024-09-21T12:32:25.836800+0000","last_clean":"2024-09-21T12:32:25.836800+0000","last_became_active":"2024-09-21T12:27:14.943705+0000","last_became_peered":"2024-09-21T12:27:14.943705+0000","last_unstale":"2024-09-21T12:32:25.836800+0000","last_undegraded":"2024-09-21T12:32:25.836800+0000","last_fullsized":"2024-09-21T12:32:25.836800+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:05.500994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:05.500994+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":477,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.614192+0000","last_change":"2024-09-21T12:29:14.222764+0000","last_active":"2024-09-21T12:32:21.614192+0000","last_peered":"2024-09-21T12:32:21.614192+0000","last_clean":"2024-09-21T12:32:21.614192+0000","last_became_active":"2024-09-21T12:23:46.902260+0000","last_became_peered":"2024-09-21T12:23:46.902260+0000","last_unstale":"2024-09-21T12:32:21.614192+0000","last_undegraded":"2024-09-21T12:32:21.614192+0000","last_fullsized":"2024-09-21T12:32:21.614192+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-21T12:29:14.222681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:29:14.222681+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":619,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.614905+0000","last_change":"2024-09-21T12:31:47.248454+0000","last_active":"2024-09-21T12:32:21.614905+0000","last_peered":"2024-09-21T12:32:21.614905+0000","last_clean":"2024-09-21T12:32:21.614905+0000","last_became_active":"2024-09-21T12:22:50.927946+0000","last_became_peered":"2024-09-21T12:22:50.927946+0000","last_unstale":"2024-09-21T12:32:21.614905+0000","last_undegraded":"2024-09-21T12:32:21.614905+0000","last_fullsized":"2024-09-21T12:32:21.614905+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:47.248389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:31:47.248389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:03.989135+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":762,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.614413+0000","last_change":"2024-09-21T12:32:19.812229+0000","last_active":"2024-09-21T12:32:21.614413+0000","last_peered":"2024-09-21T12:32:21.614413+0000","last_clean":"2024-09-21T12:32:21.614413+0000","last_became_active":"2024-09-21T12:20:19.307924+0000","last_became_peered":"2024-09-21T12:20:19.307924+0000","last_unstale":"2024-09-21T12:32:21.614413+0000","last_undegraded":"2024-09-21T12:32:21.614413+0000","last_fullsized":"2024-09-21T12:32:21.614413+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:32:19.812174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:32:19.812174+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:33.338743+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":83,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.026305+0000","last_change":"2024-09-21T12:32:02.030976+0000","last_active":"2024-09-21T12:32:22.026305+0000","last_peered":"2024-09-21T12:32:22.026305+0000","last_clean":"2024-09-21T12:32:22.026305+0000","last_became_active":"2024-09-21T12:32:02.030553+0000","last_became_peered":"2024-09-21T12:32:02.030553+0000","last_unstale":"2024-09-21T12:32:22.026305+0000","last_undegraded":"2024-09-21T12:32:22.026305+0000","last_fullsized":"2024-09-21T12:32:22.026305+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:35.669942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:35.669942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:11.048104+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"76'79","reported_seq":308,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.837145+0000","last_change":"2024-09-21T12:27:21.345414+0000","last_active":"2024-09-21T12:32:25.837145+0000","last_peered":"2024-09-21T12:32:25.837145+0000","last_clean":"2024-09-21T12:32:25.837145+0000","last_became_active":"2024-09-21T12:27:14.943330+0000","last_became_peered":"2024-09-21T12:27:14.943330+0000","last_unstale":"2024-09-21T12:32:25.837145+0000","last_undegraded":"2024-09-21T12:32:25.837145+0000","last_fullsized":"2024-09-21T12:32:25.837145+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:21.345364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:21.345364+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:08.135730+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":675,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.026684+0000","last_change":"2024-09-21T12:25:00.237083+0000","last_active":"2024-09-21T12:32:22.026684+0000","last_peered":"2024-09-21T12:32:22.026684+0000","last_clean":"2024-09-21T12:32:22.026684+0000","last_became_active":"2024-09-21T12:22:50.922343+0000","last_became_peered":"2024-09-21T12:22:50.922343+0000","last_unstale":"2024-09-21T12:32:22.026684+0000","last_undegraded":"2024-09-21T12:32:22.026684+0000","last_fullsized":"2024-09-21T12:32:22.026684+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:25:00.236990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:25:00.236990+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:28.520541+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":282,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.615425+0000","last_change":"2024-09-21T12:31:12.884054+0000","last_active":"2024-09-21T12:32:21.615425+0000","last_peered":"2024-09-21T12:32:21.615425+0000","last_clean":"2024-09-21T12:32:21.615425+0000","last_became_active":"2024-09-21T12:26:54.084228+0000","last_became_peered":"2024-09-21T12:26:54.084228+0000","last_unstale":"2024-09-21T12:32:21.615425+0000","last_undegraded":"2024-09-21T12:32:21.615425+0000","last_fullsized":"2024-09-21T12:32:21.615425+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:12.883983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:31:12.883983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":621,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.614076+0000","last_change":"2024-09-21T12:28:47.361181+0000","last_active":"2024-09-21T12:32:21.614076+0000","last_peered":"2024-09-21T12:32:21.614076+0000","last_clean":"2024-09-21T12:32:21.614076+0000","last_became_active":"2024-09-21T12:22:50.925974+0000","last_became_peered":"2024-09-21T12:22:50.925974+0000","last_unstale":"2024-09-21T12:32:21.614076+0000","last_undegraded":"2024-09-21T12:32:21.614076+0000","last_fullsized":"2024-09-21T12:32:21.614076+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:47.361115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:47.361115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:13.824334+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":583,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.023854+0000","last_change":"2024-09-21T12:28:43.821102+0000","last_active":"2024-09-21T12:32:22.023854+0000","last_peered":"2024-09-21T12:32:22.023854+0000","last_clean":"2024-09-21T12:32:22.023854+0000","last_became_active":"2024-09-21T12:20:21.311493+0000","last_became_peered":"2024-09-21T12:20:21.311493+0000","last_unstale":"2024-09-21T12:32:22.023854+0000","last_undegraded":"2024-09-21T12:32:22.023854+0000","last_fullsized":"2024-09-21T12:32:22.023854+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:43.820979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:43.820979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"76'79","reported_seq":526,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.729848+0000","last_change":"2024-09-21T12:32:25.729848+0000","last_active":"2024-09-21T12:32:25.729848+0000","last_peered":"2024-09-21T12:32:25.729848+0000","last_clean":"2024-09-21T12:32:25.729848+0000","last_became_active":"2024-09-21T12:29:05.842070+0000","last_became_peered":"2024-09-21T12:29:05.842070+0000","last_unstale":"2024-09-21T12:32:25.729848+0000","last_undegraded":"2024-09-21T12:32:25.729848+0000","last_fullsized":"2024-09-21T12:32:25.729848+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:25.729795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:25.729795+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:46.658660+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":716,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.677923+0000","last_change":"2024-09-21T12:32:14.251878+0000","last_active":"2024-09-21T12:32:25.677923+0000","last_peered":"2024-09-21T12:32:25.677923+0000","last_clean":"2024-09-21T12:32:25.677923+0000","last_became_active":"2024-09-21T12:32:14.236905+0000","last_became_peered":"2024-09-21T12:32:14.236905+0000","last_unstale":"2024-09-21T12:32:25.677923+0000","last_undegraded":"2024-09-21T12:32:25.677923+0000","last_fullsized":"2024-09-21T12:32:25.677923+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'35","last_scrub_stamp":"2024-09-21T12:31:08.327617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:08.327617+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:26.744405+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":609,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.837732+0000","last_change":"2024-09-21T12:32:02.026673+0000","last_active":"2024-09-21T12:32:25.837732+0000","last_peered":"2024-09-21T12:32:25.837732+0000","last_clean":"2024-09-21T12:32:25.837732+0000","last_became_active":"2024-09-21T12:32:02.025668+0000","last_became_peered":"2024-09-21T12:32:02.025668+0000","last_unstale":"2024-09-21T12:32:25.837732+0000","last_undegraded":"2024-09-21T12:32:25.837732+0000","last_fullsized":"2024-09-21T12:32:25.837732+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:48.438755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:48.438755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":606,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.613707+0000","last_change":"2024-09-21T12:32:02.043989+0000","last_active":"2024-09-21T12:32:21.613707+0000","last_peered":"2024-09-21T12:32:21.613707+0000","last_clean":"2024-09-21T12:32:21.613707+0000","last_became_active":"2024-09-21T12:32:02.043597+0000","last_became_peered":"2024-09-21T12:32:02.043597+0000","last_unstale":"2024-09-21T12:32:21.613707+0000","last_undegraded":"2024-09-21T12:32:21.613707+0000","last_fullsized":"2024-09-21T12:32:21.613707+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:39.732474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:31:39.732474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:53.917152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":575,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.836861+0000","last_change":"2024-09-21T12:29:15.641929+0000","last_active":"2024-09-21T12:32:25.836861+0000","last_peered":"2024-09-21T12:32:25.836861+0000","last_clean":"2024-09-21T12:32:25.836861+0000","last_became_active":"2024-09-21T12:20:21.314545+0000","last_became_peered":"2024-09-21T12:20:21.314545+0000","last_unstale":"2024-09-21T12:32:25.836861+0000","last_undegraded":"2024-09-21T12:32:25.836861+0000","last_fullsized":"2024-09-21T12:32:25.836861+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:15.641881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:29:15.641881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"76'79","reported_seq":544,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.615287+0000","last_change":"2024-09-21T12:27:17.158443+0000","last_active":"2024-09-21T12:32:21.615287+0000","last_peered":"2024-09-21T12:32:21.615287+0000","last_clean":"2024-09-21T12:32:21.615287+0000","last_became_active":"2024-09-21T12:27:14.951375+0000","last_became_peered":"2024-09-21T12:27:14.951375+0000","last_unstale":"2024-09-21T12:32:21.615287+0000","last_undegraded":"2024-09-21T12:32:21.615287+0000","last_fullsized":"2024-09-21T12:32:21.615287+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:17.158380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:17.158380+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:34.281598+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":646,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.025076+0000","last_change":"2024-09-21T12:31:50.054697+0000","last_active":"2024-09-21T12:32:22.025076+0000","last_peered":"2024-09-21T12:32:22.025076+0000","last_clean":"2024-09-21T12:32:22.025076+0000","last_became_active":"2024-09-21T12:31:50.040103+0000","last_became_peered":"2024-09-21T12:31:50.040103+0000","last_unstale":"2024-09-21T12:32:22.025076+0000","last_undegraded":"2024-09-21T12:32:22.025076+0000","last_fullsized":"2024-09-21T12:32:22.025076+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:28:40.694138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:40.694138+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":624,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.837474+0000","last_change":"2024-09-21T12:28:30.612213+0000","last_active":"2024-09-21T12:32:25.837474+0000","last_peered":"2024-09-21T12:32:25.837474+0000","last_clean":"2024-09-21T12:32:25.837474+0000","last_became_active":"2024-09-21T12:22:50.927859+0000","last_became_peered":"2024-09-21T12:22:50.927859+0000","last_unstale":"2024-09-21T12:32:25.837474+0000","last_undegraded":"2024-09-21T12:32:25.837474+0000","last_fullsized":"2024-09-21T12:32:25.837474+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:30.612156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:30.612156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":600,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.613894+0000","last_change":"2024-09-21T12:29:11.218769+0000","last_active":"2024-09-21T12:32:21.613894+0000","last_peered":"2024-09-21T12:32:21.613894+0000","last_clean":"2024-09-21T12:32:21.613894+0000","last_became_active":"2024-09-21T12:20:19.312572+0000","last_became_peered":"2024-09-21T12:20:19.312572+0000","last_unstale":"2024-09-21T12:32:21.613894+0000","last_undegraded":"2024-09-21T12:32:21.613894+0000","last_fullsized":"2024-09-21T12:32:21.613894+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:11.218334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:29:11.218334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":600,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.614576+0000","last_change":"2024-09-21T12:32:15.555334+0000","last_active":"2024-09-21T12:32:21.614576+0000","last_peered":"2024-09-21T12:32:21.614576+0000","last_clean":"2024-09-21T12:32:21.614576+0000","last_became_active":"2024-09-21T12:32:14.242180+0000","last_became_peered":"2024-09-21T12:32:14.242180+0000","last_unstale":"2024-09-21T12:32:21.614576+0000","last_undegraded":"2024-09-21T12:32:21.614576+0000","last_fullsized":"2024-09-21T12:32:21.614576+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:15.555279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:32:15.555279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:42.589957+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"76'79","reported_seq":26,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.837081+0000","last_change":"2024-09-21T12:32:14.234357+0000","last_active":"2024-09-21T12:32:25.837081+0000","last_peered":"2024-09-21T12:32:25.837081+0000","last_clean":"2024-09-21T12:32:25.837081+0000","last_became_active":"2024-09-21T12:32:14.233937+0000","last_became_peered":"2024-09-21T12:32:14.233937+0000","last_unstale":"2024-09-21T12:32:25.837081+0000","last_undegraded":"2024-09-21T12:32:25.837081+0000","last_fullsized":"2024-09-21T12:32:25.837081+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:36.700968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:36.700968+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":81,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.728597+0000","last_change":"2024-09-21T12:32:14.250281+0000","last_active":"2024-09-21T12:32:25.728597+0000","last_peered":"2024-09-21T12:32:25.728597+0000","last_clean":"2024-09-21T12:32:25.728597+0000","last_became_active":"2024-09-21T12:32:14.234897+0000","last_became_peered":"2024-09-21T12:32:14.234897+0000","last_unstale":"2024-09-21T12:32:25.728597+0000","last_undegraded":"2024-09-21T12:32:25.728597+0000","last_fullsized":"2024-09-21T12:32:25.728597+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:31:40.395713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:40.395713+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:00.527233+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":680,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.837566+0000","last_change":"2024-09-21T12:31:53.103654+0000","last_active":"2024-09-21T12:32:25.837566+0000","last_peered":"2024-09-21T12:32:25.837566+0000","last_clean":"2024-09-21T12:32:25.837566+0000","last_became_active":"2024-09-21T12:31:39.054205+0000","last_became_peered":"2024-09-21T12:31:39.054205+0000","last_unstale":"2024-09-21T12:32:25.837566+0000","last_undegraded":"2024-09-21T12:32:25.837566+0000","last_fullsized":"2024-09-21T12:32:25.837566+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-21T12:31:53.103575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:31:53.103575+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:08.028708+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"308'13","reported_seq":798,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.836793+0000","last_change":"2024-09-21T12:31:57.223219+0000","last_active":"2024-09-21T12:32:25.836793+0000","last_peered":"2024-09-21T12:32:25.836793+0000","last_clean":"2024-09-21T12:32:25.836793+0000","last_became_active":"2024-09-21T12:31:39.054099+0000","last_became_peered":"2024-09-21T12:31:39.054099+0000","last_unstale":"2024-09-21T12:32:25.836793+0000","last_undegraded":"2024-09-21T12:32:25.836793+0000","last_fullsized":"2024-09-21T12:32:25.836793+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'13","last_scrub_stamp":"2024-09-21T12:31:57.223134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:31:57.223134+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:26.569138+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":286,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.677805+0000","last_change":"2024-09-21T12:26:56.862895+0000","last_active":"2024-09-21T12:32:25.677805+0000","last_peered":"2024-09-21T12:32:25.677805+0000","last_clean":"2024-09-21T12:32:25.677805+0000","last_became_active":"2024-09-21T12:26:54.089155+0000","last_became_peered":"2024-09-21T12:26:54.089155+0000","last_unstale":"2024-09-21T12:32:25.677805+0000","last_undegraded":"2024-09-21T12:32:25.677805+0000","last_fullsized":"2024-09-21T12:32:25.677805+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:56.862823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:56.862823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"76'79","reported_seq":340,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:26.698667+0000","last_change":"2024-09-21T12:32:26.698667+0000","last_active":"2024-09-21T12:32:26.698667+0000","last_peered":"2024-09-21T12:32:26.698667+0000","last_clean":"2024-09-21T12:32:26.698667+0000","last_became_active":"2024-09-21T12:31:39.060422+0000","last_became_peered":"2024-09-21T12:31:39.060422+0000","last_unstale":"2024-09-21T12:32:26.698667+0000","last_undegraded":"2024-09-21T12:32:26.698667+0000","last_fullsized":"2024-09-21T12:32:26.698667+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:26.698622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:26.698622+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:53.950707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":641,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.677830+0000","last_change":"2024-09-21T12:32:02.027084+0000","last_active":"2024-09-21T12:32:25.677830+0000","last_peered":"2024-09-21T12:32:25.677830+0000","last_clean":"2024-09-21T12:32:25.677830+0000","last_became_active":"2024-09-21T12:32:02.026693+0000","last_became_peered":"2024-09-21T12:32:02.026693+0000","last_unstale":"2024-09-21T12:32:25.677830+0000","last_undegraded":"2024-09-21T12:32:25.677830+0000","last_fullsized":"2024-09-21T12:32:25.677830+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:07.375741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:31:07.375741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":89,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.615431+0000","last_change":"2024-09-21T12:32:20.584288+0000","last_active":"2024-09-21T12:32:21.615431+0000","last_peered":"2024-09-21T12:32:21.615431+0000","last_clean":"2024-09-21T12:32:21.615431+0000","last_became_active":"2024-09-21T12:32:14.244680+0000","last_became_peered":"2024-09-21T12:32:14.244680+0000","last_unstale":"2024-09-21T12:32:21.615431+0000","last_undegraded":"2024-09-21T12:32:21.615431+0000","last_fullsized":"2024-09-21T12:32:21.615431+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T12:32:20.584227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:20.584227+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:45.670992+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":283,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.614917+0000","last_change":"2024-09-21T12:31:10.873131+0000","last_active":"2024-09-21T12:32:21.614917+0000","last_peered":"2024-09-21T12:32:21.614917+0000","last_clean":"2024-09-21T12:32:21.614917+0000","last_became_active":"2024-09-21T12:26:54.084109+0000","last_became_peered":"2024-09-21T12:26:54.084109+0000","last_unstale":"2024-09-21T12:32:21.614917+0000","last_undegraded":"2024-09-21T12:32:21.614917+0000","last_fullsized":"2024-09-21T12:32:21.614917+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:10.873039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:31:10.873039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:29.776805+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":596,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.837497+0000","last_change":"2024-09-21T12:24:00.066522+0000","last_active":"2024-09-21T12:32:25.837497+0000","last_peered":"2024-09-21T12:32:25.837497+0000","last_clean":"2024-09-21T12:32:25.837497+0000","last_became_active":"2024-09-21T12:22:50.904404+0000","last_became_peered":"2024-09-21T12:22:50.904404+0000","last_unstale":"2024-09-21T12:32:25.837497+0000","last_undegraded":"2024-09-21T12:32:25.837497+0000","last_fullsized":"2024-09-21T12:32:25.837497+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:24:00.066451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:24:00.066451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:35.132786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'79","reported_seq":555,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.025327+0000","last_change":"2024-09-21T12:31:50.040921+0000","last_active":"2024-09-21T12:32:22.025327+0000","last_peered":"2024-09-21T12:32:22.025327+0000","last_clean":"2024-09-21T12:32:22.025327+0000","last_became_active":"2024-09-21T12:31:50.040502+0000","last_became_peered":"2024-09-21T12:31:50.040502+0000","last_unstale":"2024-09-21T12:32:22.025327+0000","last_undegraded":"2024-09-21T12:32:22.025327+0000","last_fullsized":"2024-09-21T12:32:22.025327+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:29.546657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:29.546657+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":666,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.026626+0000","last_change":"2024-09-21T12:32:19.786992+0000","last_active":"2024-09-21T12:32:22.026626+0000","last_peered":"2024-09-21T12:32:22.026626+0000","last_clean":"2024-09-21T12:32:22.026626+0000","last_became_active":"2024-09-21T12:32:14.242339+0000","last_became_peered":"2024-09-21T12:32:14.242339+0000","last_unstale":"2024-09-21T12:32:22.026626+0000","last_undegraded":"2024-09-21T12:32:22.026626+0000","last_fullsized":"2024-09-21T12:32:22.026626+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:32:19.786928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:19.786928+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:20.683939+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":632,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.837010+0000","last_change":"2024-09-21T12:28:43.545259+0000","last_active":"2024-09-21T12:32:25.837010+0000","last_peered":"2024-09-21T12:32:25.837010+0000","last_clean":"2024-09-21T12:32:25.837010+0000","last_became_active":"2024-09-21T12:22:50.927649+0000","last_became_peered":"2024-09-21T12:22:50.927649+0000","last_unstale":"2024-09-21T12:32:25.837010+0000","last_undegraded":"2024-09-21T12:32:25.837010+0000","last_fullsized":"2024-09-21T12:32:25.837010+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:43.545197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:43.545197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":581,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.615342+0000","last_change":"2024-09-21T12:28:47.964092+0000","last_active":"2024-09-21T12:32:21.615342+0000","last_peered":"2024-09-21T12:32:21.615342+0000","last_clean":"2024-09-21T12:32:21.615342+0000","last_became_active":"2024-09-21T12:20:19.311947+0000","last_became_peered":"2024-09-21T12:20:19.311947+0000","last_unstale":"2024-09-21T12:32:21.615342+0000","last_undegraded":"2024-09-21T12:32:21.615342+0000","last_fullsized":"2024-09-21T12:32:21.615342+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:47.963747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:47.963747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":617,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.614231+0000","last_change":"2024-09-21T12:26:11.105716+0000","last_active":"2024-09-21T12:32:21.614231+0000","last_peered":"2024-09-21T12:32:21.614231+0000","last_clean":"2024-09-21T12:32:21.614231+0000","last_became_active":"2024-09-21T12:22:50.924931+0000","last_became_peered":"2024-09-21T12:22:50.924931+0000","last_unstale":"2024-09-21T12:32:21.614231+0000","last_undegraded":"2024-09-21T12:32:21.614231+0000","last_fullsized":"2024-09-21T12:32:21.614231+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:11.105583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:11.105583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:36.717202+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"76'79","reported_seq":679,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.836645+0000","last_change":"2024-09-21T12:31:03.293976+0000","last_active":"2024-09-21T12:32:25.836645+0000","last_peered":"2024-09-21T12:32:25.836645+0000","last_clean":"2024-09-21T12:32:25.836645+0000","last_became_active":"2024-09-21T12:26:54.084939+0000","last_became_peered":"2024-09-21T12:26:54.084939+0000","last_unstale":"2024-09-21T12:32:25.836645+0000","last_undegraded":"2024-09-21T12:32:25.836645+0000","last_fullsized":"2024-09-21T12:32:25.836645+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:31:03.293909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:31:03.293909+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":365,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.837661+0000","last_change":"2024-09-21T12:32:02.034771+0000","last_active":"2024-09-21T12:32:25.837661+0000","last_peered":"2024-09-21T12:32:25.837661+0000","last_clean":"2024-09-21T12:32:25.837661+0000","last_became_active":"2024-09-21T12:32:02.027460+0000","last_became_peered":"2024-09-21T12:32:02.027460+0000","last_unstale":"2024-09-21T12:32:25.837661+0000","last_undegraded":"2024-09-21T12:32:25.837661+0000","last_fullsized":"2024-09-21T12:32:25.837661+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:28:42.511874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:42.511874+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":632,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.025592+0000","last_change":"2024-09-21T12:32:17.813348+0000","last_active":"2024-09-21T12:32:22.025592+0000","last_peered":"2024-09-21T12:32:22.025592+0000","last_clean":"2024-09-21T12:32:22.025592+0000","last_became_active":"2024-09-21T12:32:14.236220+0000","last_became_peered":"2024-09-21T12:32:14.236220+0000","last_unstale":"2024-09-21T12:32:22.025592+0000","last_undegraded":"2024-09-21T12:32:22.025592+0000","last_fullsized":"2024-09-21T12:32:22.025592+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:17.813293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:17.813293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:39.028980+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"316'10","reported_seq":69,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.614484+0000","last_change":"2024-09-21T12:31:50.050850+0000","last_active":"2024-09-21T12:32:21.614484+0000","last_peered":"2024-09-21T12:32:21.614484+0000","last_clean":"2024-09-21T12:32:21.614484+0000","last_became_active":"2024-09-21T12:31:50.040167+0000","last_became_peered":"2024-09-21T12:31:50.040167+0000","last_unstale":"2024-09-21T12:32:21.614484+0000","last_undegraded":"2024-09-21T12:32:21.614484+0000","last_fullsized":"2024-09-21T12:32:21.614484+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:28:43.853712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:43.853712+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:13.575849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":523,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.614849+0000","last_change":"2024-09-21T12:28:05.323501+0000","last_active":"2024-09-21T12:32:21.614849+0000","last_peered":"2024-09-21T12:32:21.614849+0000","last_clean":"2024-09-21T12:32:21.614849+0000","last_became_active":"2024-09-21T12:22:50.924770+0000","last_became_peered":"2024-09-21T12:22:50.924770+0000","last_unstale":"2024-09-21T12:32:21.614849+0000","last_undegraded":"2024-09-21T12:32:21.614849+0000","last_fullsized":"2024-09-21T12:32:21.614849+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:05.323448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:05.323448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:08.883462+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"76'79","reported_seq":405,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.024064+0000","last_change":"2024-09-21T12:32:02.039507+0000","last_active":"2024-09-21T12:32:22.024064+0000","last_peered":"2024-09-21T12:32:22.024064+0000","last_clean":"2024-09-21T12:32:22.024064+0000","last_became_active":"2024-09-21T12:32:02.034387+0000","last_became_peered":"2024-09-21T12:32:02.034387+0000","last_unstale":"2024-09-21T12:32:22.024064+0000","last_undegraded":"2024-09-21T12:32:22.024064+0000","last_fullsized":"2024-09-21T12:32:22.024064+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:31:48.635956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:31:48.635956+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:58.208578+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":662,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.615368+0000","last_change":"2024-09-21T12:31:39.068884+0000","last_active":"2024-09-21T12:32:21.615368+0000","last_peered":"2024-09-21T12:32:21.615368+0000","last_clean":"2024-09-21T12:32:21.615368+0000","last_became_active":"2024-09-21T12:31:39.058798+0000","last_became_peered":"2024-09-21T12:31:39.058798+0000","last_unstale":"2024-09-21T12:32:21.615368+0000","last_undegraded":"2024-09-21T12:32:21.615368+0000","last_fullsized":"2024-09-21T12:32:21.615368+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:31:03.996594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:03.996594+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:26.397125+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":624,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.837206+0000","last_change":"2024-09-21T12:28:32.630320+0000","last_active":"2024-09-21T12:32:25.837206+0000","last_peered":"2024-09-21T12:32:25.837206+0000","last_clean":"2024-09-21T12:32:25.837206+0000","last_became_active":"2024-09-21T12:22:50.908238+0000","last_became_peered":"2024-09-21T12:22:50.908238+0000","last_unstale":"2024-09-21T12:32:25.837206+0000","last_undegraded":"2024-09-21T12:32:25.837206+0000","last_fullsized":"2024-09-21T12:32:25.837206+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:32.630261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:32.630261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":750,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.729220+0000","last_change":"2024-09-21T12:31:07.510064+0000","last_active":"2024-09-21T12:32:25.729220+0000","last_peered":"2024-09-21T12:32:25.729220+0000","last_clean":"2024-09-21T12:32:25.729220+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:32:25.729220+0000","last_undegraded":"2024-09-21T12:32:25.729220+0000","last_fullsized":"2024-09-21T12:32:25.729220+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:31:07.509980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:31:07.509980+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:28.145931+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":596,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.026134+0000","last_change":"2024-09-21T12:31:52.608520+0000","last_active":"2024-09-21T12:32:22.026134+0000","last_peered":"2024-09-21T12:32:22.026134+0000","last_clean":"2024-09-21T12:32:22.026134+0000","last_became_active":"2024-09-21T12:31:50.037934+0000","last_became_peered":"2024-09-21T12:31:50.037934+0000","last_unstale":"2024-09-21T12:32:22.026134+0000","last_undegraded":"2024-09-21T12:32:22.026134+0000","last_fullsized":"2024-09-21T12:32:22.026134+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:52.608467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:31:52.608467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:11.560519+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"76'79","reported_seq":270,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.026077+0000","last_change":"2024-09-21T12:31:38.104199+0000","last_active":"2024-09-21T12:32:22.026077+0000","last_peered":"2024-09-21T12:32:22.026077+0000","last_clean":"2024-09-21T12:32:22.026077+0000","last_became_active":"2024-09-21T12:27:16.940347+0000","last_became_peered":"2024-09-21T12:27:16.940347+0000","last_unstale":"2024-09-21T12:32:22.026077+0000","last_undegraded":"2024-09-21T12:32:22.026077+0000","last_fullsized":"2024-09-21T12:32:22.026077+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:31:38.104129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:31:38.104129+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:51.210438+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":602,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.678096+0000","last_change":"2024-09-21T12:28:40.150322+0000","last_active":"2024-09-21T12:32:25.678096+0000","last_peered":"2024-09-21T12:32:25.678096+0000","last_clean":"2024-09-21T12:32:25.678096+0000","last_became_active":"2024-09-21T12:22:50.906473+0000","last_became_peered":"2024-09-21T12:22:50.906473+0000","last_unstale":"2024-09-21T12:32:25.678096+0000","last_undegraded":"2024-09-21T12:32:25.678096+0000","last_fullsized":"2024-09-21T12:32:25.678096+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:28:40.150250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:40.150250+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":519,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.837398+0000","last_change":"2024-09-21T12:31:38.106169+0000","last_active":"2024-09-21T12:32:25.837398+0000","last_peered":"2024-09-21T12:32:25.837398+0000","last_clean":"2024-09-21T12:32:25.837398+0000","last_became_active":"2024-09-21T12:22:50.903705+0000","last_became_peered":"2024-09-21T12:22:50.903705+0000","last_unstale":"2024-09-21T12:32:25.837398+0000","last_undegraded":"2024-09-21T12:32:25.837398+0000","last_fullsized":"2024-09-21T12:32:25.837398+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:38.106115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:31:38.106115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:03.057233+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":602,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.024514+0000","last_change":"2024-09-21T12:31:51.006655+0000","last_active":"2024-09-21T12:32:22.024514+0000","last_peered":"2024-09-21T12:32:22.024514+0000","last_clean":"2024-09-21T12:32:22.024514+0000","last_became_active":"2024-09-21T12:31:50.040754+0000","last_became_peered":"2024-09-21T12:31:50.040754+0000","last_unstale":"2024-09-21T12:32:22.024514+0000","last_undegraded":"2024-09-21T12:32:22.024514+0000","last_fullsized":"2024-09-21T12:32:22.024514+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:51.006611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:31:51.006611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:16.792439+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":66,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.613728+0000","last_change":"2024-09-21T12:31:53.359206+0000","last_active":"2024-09-21T12:32:21.613728+0000","last_peered":"2024-09-21T12:32:21.613728+0000","last_clean":"2024-09-21T12:32:21.613728+0000","last_became_active":"2024-09-21T12:31:50.036150+0000","last_became_peered":"2024-09-21T12:31:50.036150+0000","last_unstale":"2024-09-21T12:32:21.613728+0000","last_undegraded":"2024-09-21T12:32:21.613728+0000","last_fullsized":"2024-09-21T12:32:21.613728+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:53.359159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:31:53.359159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:09.450093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"76'79","reported_seq":297,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.729628+0000","last_change":"2024-09-21T12:32:24.750378+0000","last_active":"2024-09-21T12:32:25.729628+0000","last_peered":"2024-09-21T12:32:25.729628+0000","last_clean":"2024-09-21T12:32:25.729628+0000","last_became_active":"2024-09-21T12:27:16.921880+0000","last_became_peered":"2024-09-21T12:27:16.921880+0000","last_unstale":"2024-09-21T12:32:25.729628+0000","last_undegraded":"2024-09-21T12:32:25.729628+0000","last_fullsized":"2024-09-21T12:32:25.729628+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:24.750320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:24.750320+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:54.665474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":86,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.024198+0000","last_change":"2024-09-21T12:31:39.071315+0000","last_active":"2024-09-21T12:32:22.024198+0000","last_peered":"2024-09-21T12:32:22.024198+0000","last_clean":"2024-09-21T12:32:22.024198+0000","last_became_active":"2024-09-21T12:31:39.058221+0000","last_became_peered":"2024-09-21T12:31:39.058221+0000","last_unstale":"2024-09-21T12:32:22.024198+0000","last_undegraded":"2024-09-21T12:32:22.024198+0000","last_fullsized":"2024-09-21T12:32:22.024198+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:31:09.319883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:09.319883+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":624,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.837664+0000","last_change":"2024-09-21T12:26:19.973528+0000","last_active":"2024-09-21T12:32:25.837664+0000","last_peered":"2024-09-21T12:32:25.837664+0000","last_clean":"2024-09-21T12:32:25.837664+0000","last_became_active":"2024-09-21T12:22:50.922019+0000","last_became_peered":"2024-09-21T12:22:50.922019+0000","last_unstale":"2024-09-21T12:32:25.837664+0000","last_undegraded":"2024-09-21T12:32:25.837664+0000","last_fullsized":"2024-09-21T12:32:25.837664+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:19.973461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:26:19.973461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:00.210381+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":582,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.614661+0000","last_change":"2024-09-21T12:26:21.572474+0000","last_active":"2024-09-21T12:32:21.614661+0000","last_peered":"2024-09-21T12:32:21.614661+0000","last_clean":"2024-09-21T12:32:21.614661+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:32:21.614661+0000","last_undegraded":"2024-09-21T12:32:21.614661+0000","last_fullsized":"2024-09-21T12:32:21.614661+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:21.571726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:21.571726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:22.821247+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":583,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.025109+0000","last_change":"2024-09-21T12:28:41.864711+0000","last_active":"2024-09-21T12:32:22.025109+0000","last_peered":"2024-09-21T12:32:22.025109+0000","last_clean":"2024-09-21T12:32:22.025109+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:32:22.025109+0000","last_undegraded":"2024-09-21T12:32:22.025109+0000","last_fullsized":"2024-09-21T12:32:22.025109+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:41.864630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:41.864630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"76'79","reported_seq":298,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.613843+0000","last_change":"2024-09-21T12:31:38.745628+0000","last_active":"2024-09-21T12:32:21.613843+0000","last_peered":"2024-09-21T12:32:21.613843+0000","last_clean":"2024-09-21T12:32:21.613843+0000","last_became_active":"2024-09-21T12:26:46.989742+0000","last_became_peered":"2024-09-21T12:26:46.989742+0000","last_unstale":"2024-09-21T12:32:21.613843+0000","last_undegraded":"2024-09-21T12:32:21.613843+0000","last_fullsized":"2024-09-21T12:32:21.613843+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:31:38.745581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:31:38.745581+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:43.839127+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":625,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.614095+0000","last_change":"2024-09-21T12:31:06.945554+0000","last_active":"2024-09-21T12:32:21.614095+0000","last_peered":"2024-09-21T12:32:21.614095+0000","last_clean":"2024-09-21T12:32:21.614095+0000","last_became_active":"2024-09-21T12:20:17.694794+0000","last_became_peered":"2024-09-21T12:20:17.694794+0000","last_unstale":"2024-09-21T12:32:21.614095+0000","last_undegraded":"2024-09-21T12:32:21.614095+0000","last_fullsized":"2024-09-21T12:32:21.614095+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:31:06.945472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:06.945472+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:24.736973+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":619,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.729200+0000","last_change":"2024-09-21T12:29:54.303655+0000","last_active":"2024-09-21T12:32:25.729200+0000","last_peered":"2024-09-21T12:32:25.729200+0000","last_clean":"2024-09-21T12:32:25.729200+0000","last_became_active":"2024-09-21T12:20:15.295968+0000","last_became_peered":"2024-09-21T12:20:15.295968+0000","last_unstale":"2024-09-21T12:32:25.729200+0000","last_undegraded":"2024-09-21T12:32:25.729200+0000","last_fullsized":"2024-09-21T12:32:25.729200+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:54.303590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:54.303590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:33.627493+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":600,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.836575+0000","last_change":"2024-09-21T12:23:57.018797+0000","last_active":"2024-09-21T12:32:25.836575+0000","last_peered":"2024-09-21T12:32:25.836575+0000","last_clean":"2024-09-21T12:32:25.836575+0000","last_became_active":"2024-09-21T12:22:50.921604+0000","last_became_peered":"2024-09-21T12:22:50.921604+0000","last_unstale":"2024-09-21T12:32:25.836575+0000","last_undegraded":"2024-09-21T12:32:25.836575+0000","last_fullsized":"2024-09-21T12:32:25.836575+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:57.018749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:57.018749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:33.121402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":578,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.614798+0000","last_change":"2024-09-21T12:29:14.051153+0000","last_active":"2024-09-21T12:32:21.614798+0000","last_peered":"2024-09-21T12:32:21.614798+0000","last_clean":"2024-09-21T12:32:21.614798+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:32:21.614798+0000","last_undegraded":"2024-09-21T12:32:21.614798+0000","last_fullsized":"2024-09-21T12:32:21.614798+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:14.051043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:29:14.051043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"76'79","reported_seq":273,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.837518+0000","last_change":"2024-09-21T12:32:20.425343+0000","last_active":"2024-09-21T12:32:25.837518+0000","last_peered":"2024-09-21T12:32:25.837518+0000","last_clean":"2024-09-21T12:32:25.837518+0000","last_became_active":"2024-09-21T12:27:16.914713+0000","last_became_peered":"2024-09-21T12:27:16.914713+0000","last_unstale":"2024-09-21T12:32:25.837518+0000","last_undegraded":"2024-09-21T12:32:25.837518+0000","last_fullsized":"2024-09-21T12:32:25.837518+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:20.417296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:20.417296+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:35.912073+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":598,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.613694+0000","last_change":"2024-09-21T12:28:46.932522+0000","last_active":"2024-09-21T12:32:21.613694+0000","last_peered":"2024-09-21T12:32:21.613694+0000","last_clean":"2024-09-21T12:32:21.613694+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:32:21.613694+0000","last_undegraded":"2024-09-21T12:32:21.613694+0000","last_fullsized":"2024-09-21T12:32:21.613694+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-21T12:28:46.931901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:46.931901+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":617,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.025731+0000","last_change":"2024-09-21T12:31:57.681487+0000","last_active":"2024-09-21T12:32:22.025731+0000","last_peered":"2024-09-21T12:32:22.025731+0000","last_clean":"2024-09-21T12:32:22.025731+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:32:22.025731+0000","last_undegraded":"2024-09-21T12:32:22.025731+0000","last_fullsized":"2024-09-21T12:32:22.025731+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:31:57.681417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:31:57.681417+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:17.342617+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":616,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.677877+0000","last_change":"2024-09-21T12:29:59.216485+0000","last_active":"2024-09-21T12:32:25.677877+0000","last_peered":"2024-09-21T12:32:25.677877+0000","last_clean":"2024-09-21T12:32:25.677877+0000","last_became_active":"2024-09-21T12:22:50.930615+0000","last_became_peered":"2024-09-21T12:22:50.930615+0000","last_unstale":"2024-09-21T12:32:25.677877+0000","last_undegraded":"2024-09-21T12:32:25.677877+0000","last_fullsized":"2024-09-21T12:32:25.677877+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:59.216442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:29:59.216442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:34.428945+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":604,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.836844+0000","last_change":"2024-09-21T12:28:41.537845+0000","last_active":"2024-09-21T12:32:25.836844+0000","last_peered":"2024-09-21T12:32:25.836844+0000","last_clean":"2024-09-21T12:32:25.836844+0000","last_became_active":"2024-09-21T12:22:50.917408+0000","last_became_peered":"2024-09-21T12:22:50.917408+0000","last_unstale":"2024-09-21T12:32:25.836844+0000","last_undegraded":"2024-09-21T12:32:25.836844+0000","last_fullsized":"2024-09-21T12:32:25.836844+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:41.537786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:41.537786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"76'79","reported_seq":265,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.837083+0000","last_change":"2024-09-21T12:27:29.341314+0000","last_active":"2024-09-21T12:32:25.837083+0000","last_peered":"2024-09-21T12:32:25.837083+0000","last_clean":"2024-09-21T12:32:25.837083+0000","last_became_active":"2024-09-21T12:27:16.948697+0000","last_became_peered":"2024-09-21T12:27:16.948697+0000","last_unstale":"2024-09-21T12:32:25.837083+0000","last_undegraded":"2024-09-21T12:32:25.837083+0000","last_fullsized":"2024-09-21T12:32:25.837083+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:29.341268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:29.341268+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:09.132577+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":601,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.025494+0000","last_change":"2024-09-21T12:28:46.749904+0000","last_active":"2024-09-21T12:32:22.025494+0000","last_peered":"2024-09-21T12:32:22.025494+0000","last_clean":"2024-09-21T12:32:22.025494+0000","last_became_active":"2024-09-21T12:22:50.914221+0000","last_became_peered":"2024-09-21T12:22:50.914221+0000","last_unstale":"2024-09-21T12:32:22.025494+0000","last_undegraded":"2024-09-21T12:32:22.025494+0000","last_fullsized":"2024-09-21T12:32:22.025494+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:46.749839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:46.749839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":612,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-21T12:32:21.613911+0000","last_change":"2024-09-21T12:31:39.052864+0000","last_active":"2024-09-21T12:32:21.613911+0000","last_peered":"2024-09-21T12:32:21.613911+0000","last_clean":"2024-09-21T12:32:21.613911+0000","last_became_active":"2024-09-21T12:31:39.052486+0000","last_became_peered":"2024-09-21T12:31:39.052486+0000","last_unstale":"2024-09-21T12:32:21.613911+0000","last_undegraded":"2024-09-21T12:32:21.613911+0000","last_fullsized":"2024-09-21T12:32:21.613911+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:15.071234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:15.071234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":630,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-21T12:32:25.678523+0000","last_change":"2024-09-21T12:30:02.290006+0000","last_active":"2024-09-21T12:32:25.678523+0000","last_peered":"2024-09-21T12:32:25.678523+0000","last_clean":"2024-09-21T12:32:25.678523+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:32:25.678523+0000","last_undegraded":"2024-09-21T12:32:25.678523+0000","last_fullsized":"2024-09-21T12:32:25.678523+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:30:02.289930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:30:02.289930+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:32:36.380982+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":611,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-21T12:32:22.026224+0000","last_change":"2024-09-21T12:28:28.741483+0000","last_active":"2024-09-21T12:32:22.026224+0000","last_peered":"2024-09-21T12:32:22.026224+0000","last_clean":"2024-09-21T12:32:22.026224+0000","last_became_active":"2024-09-21T12:20:19.308247+0000","last_became_peered":"2024-09-21T12:20:19.308247+0000","last_unstale":"2024-09-21T12:32:22.026224+0000","last_undegraded":"2024-09-21T12:32:22.026224+0000","last_fullsized":"2024-09-21T12:32:22.026224+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:28.741418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:28.741418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1111,"num_read_kb":868,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":307,"num_bytes_recovered":6050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":683,"ondisk_log_size":683,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":5023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":725,"num_read_kb":720,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2449,"ondisk_log_size":2449,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234134,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":582688,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93162464,"statfs":{"total":95995035648,"available":95398363136,"internally_reserved":0,"allocated":1781760,"data_stored":1211057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":594868682},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463070,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":480028,"kb_used_data":1800,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":93265124,"statfs":{"total":95995035648,"available":95503486976,"internally_reserved":0,"allocated":1843200,"data_stored":1275840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":489683398},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724705,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":542428,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":93202724,"statfs":{"total":95995035648,"available":95439589376,"internally_reserved":0,"allocated":1777664,"data_stored":1211837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953641,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149870,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":636892,"kb_used_data":1800,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":93108260,"statfs":{"total":95995035648,"available":95342858240,"internally_reserved":0,"allocated":1843200,"data_stored":1261702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":650312135},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411508,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":416060,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":93329092,"statfs":{"total":95995035648,"available":95568990208,"internally_reserved":0,"allocated":1740800,"data_stored":1194758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575034,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":654656,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93090496,"statfs":{"total":95995035648,"available":95324667904,"internally_reserved":0,"allocated":1544192,"data_stored":940950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738560,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":594128,"kb_used_data":35384,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":93151024,"statfs":{"total":95995035648,"available":95386648576,"internally_reserved":0,"allocated":36233216,"data_stored":35667516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:32:29.781 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T12:32:29.781 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:32:29.781 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-09-21T12:32:29.781 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd in 4 2024-09-21T12:32:29.830 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:29.851 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:29.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:29.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:29.955 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:29.976 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:29.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:29.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:30.078 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:30.100 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:30.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:30.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:30.202 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:30.223 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:30.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:30.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:30.251 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-09-21T12:32:30.325 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:30.347 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:30.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:30.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:30.448 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:30.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:30.471 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:30.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:30.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:30 smithi114 ceph-mon[40867]: 3.1c scrub starts 2024-09-21T12:32:30.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:30 smithi114 ceph-mon[40867]: 3.1c scrub ok 2024-09-21T12:32:30.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:30 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-21T12:32:30.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:30 smithi114 ceph-mon[40867]: osdmap e344: 8 total, 8 up, 7 in 2024-09-21T12:32:30.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:30 smithi114 ceph-mon[30324]: 3.1c scrub starts 2024-09-21T12:32:30.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:30 smithi114 ceph-mon[30324]: 3.1c scrub ok 2024-09-21T12:32:30.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:30 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-21T12:32:30.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:30 smithi114 ceph-mon[30324]: osdmap e344: 8 total, 8 up, 7 in 2024-09-21T12:32:30.572 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:30.594 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:30.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:30.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:30.695 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:30.700 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:32:30.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:30.717 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:30.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:30.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:30 smithi183 ceph-mon[33277]: 3.1c scrub starts 2024-09-21T12:32:30.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:30 smithi183 ceph-mon[33277]: 3.1c scrub ok 2024-09-21T12:32:30.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:30 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-21T12:32:30.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:30 smithi183 ceph-mon[33277]: osdmap e344: 8 total, 8 up, 7 in 2024-09-21T12:32:30.819 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:30.840 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:30.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:30.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:30.941 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:30.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:30.963 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:30.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:31.064 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:31.090 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:31.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:31.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:31.130 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:32:31.130 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 20, 2024-09-21T12:32:31.130 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 600, 2024-09-21T12:32:31.130 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-21T12:32:31.130 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.131 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:180252 9.0 9:031fbf63:::benchmark_data_smithi114_133214_object56218:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-21T12:32:31.131 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:02.091460+0000", 2024-09-21T12:32:31.131 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 89.031902944999999, 2024-09-21T12:32:31.131 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.94274876900000004, 2024-09-21T12:32:31.131 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:31.131 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:31.131 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:32:31.131 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:31.131 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:31.131 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:31.131 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 180252 2024-09-21T12:32:31.131 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.131 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:31.131 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.132 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:31.132 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.091460+0000", 2024-09-21T12:32:31.132 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.132 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.132 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.132 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:31.132 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.091460+0000", 2024-09-21T12:32:31.132 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.132 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.132 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.132 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:31.132 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.091461+0000", 2024-09-21T12:32:31.133 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1500000000000005e-07 2024-09-21T12:32:31.133 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.133 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.133 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:31.133 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.091464+0000", 2024-09-21T12:32:31.133 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.937e-06 2024-09-21T12:32:31.133 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.133 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.133 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:31.133 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.091471+0000", 2024-09-21T12:32:31.133 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1160000000000001e-06 2024-09-21T12:32:31.133 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.133 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.133 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:31.133 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.091477+0000", 2024-09-21T12:32:31.133 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6799999999999998e-06 2024-09-21T12:32:31.134 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.134 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.134 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:31.134 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.091897+0000", 2024-09-21T12:32:31.134 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00042072499999999999 2024-09-21T12:32:31.134 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.134 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.134 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:32:31.134 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.091908+0000", 2024-09-21T12:32:31.134 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.064e-05 2024-09-21T12:32:31.134 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.134 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.134 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:31.134 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.029652+0000", 2024-09-21T12:32:31.135 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.93774429699999995 2024-09-21T12:32:31.135 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.135 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.135 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:31.135 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.029776+0000", 2024-09-21T12:32:31.135 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000123845 2024-09-21T12:32:31.135 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.135 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.135 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,6", 2024-09-21T12:32:31.135 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.030080+0000", 2024-09-21T12:32:31.135 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00030401199999999999 2024-09-21T12:32:31.135 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.135 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.135 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:31.136 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.033578+0000", 2024-09-21T12:32:31.136 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003497778 2024-09-21T12:32:31.136 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.136 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.136 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:31.136 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.033683+0000", 2024-09-21T12:32:31.136 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000105123 2024-09-21T12:32:31.136 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.136 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.136 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:32:31.136 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.034144+0000", 2024-09-21T12:32:31.136 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00046067200000000002 2024-09-21T12:32:31.136 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.136 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.136 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:32:31.136 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.034178+0000", 2024-09-21T12:32:31.137 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4462999999999999e-05 2024-09-21T12:32:31.137 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.137 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.137 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:31.137 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.034209+0000", 2024-09-21T12:32:31.137 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.0966000000000001e-05 2024-09-21T12:32:31.137 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.137 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:31.137 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.137 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.137 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.137 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:180283 9.0 9:06b0c872:::benchmark_data_smithi114_133214_object56249:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-21T12:32:31.137 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:02.111347+0000", 2024-09-21T12:32:31.137 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 89.012016532000004, 2024-09-21T12:32:31.137 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.92469939300000004, 2024-09-21T12:32:31.138 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:31.138 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:31.138 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:32:31.138 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:31.138 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:31.138 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:31.138 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 180283 2024-09-21T12:32:31.138 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.138 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:31.138 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.138 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:31.138 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.111347+0000", 2024-09-21T12:32:31.138 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.138 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.139 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.139 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:31.139 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.111347+0000", 2024-09-21T12:32:31.139 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.139 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.139 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.139 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:31.139 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.111348+0000", 2024-09-21T12:32:31.139 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3060000000000001e-06 2024-09-21T12:32:31.139 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.139 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.139 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:31.139 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.111353+0000", 2024-09-21T12:32:31.139 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0470000000000001e-06 2024-09-21T12:32:31.139 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.139 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.140 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:31.140 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.111365+0000", 2024-09-21T12:32:31.140 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1464000000000001e-05 2024-09-21T12:32:31.140 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.140 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.140 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:31.140 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.111372+0000", 2024-09-21T12:32:31.140 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0119999999999999e-06 2024-09-21T12:32:31.140 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.140 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.140 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:31.140 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.111442+0000", 2024-09-21T12:32:31.140 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0350000000000002e-05 2024-09-21T12:32:31.140 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.141 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.141 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:32:31.141 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.111451+0000", 2024-09-21T12:32:31.141 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9949999999999994e-06 2024-09-21T12:32:31.141 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.141 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.141 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:31.141 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.031889+0000", 2024-09-21T12:32:31.141 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.92043774499999997 2024-09-21T12:32:31.141 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.142 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.142 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:31.142 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.031996+0000", 2024-09-21T12:32:31.142 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000106772 2024-09-21T12:32:31.142 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.142 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.142 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,6", 2024-09-21T12:32:31.142 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.032192+0000", 2024-09-21T12:32:31.142 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00019618399999999999 2024-09-21T12:32:31.142 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.142 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.142 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:31.142 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.034123+0000", 2024-09-21T12:32:31.142 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001930764 2024-09-21T12:32:31.143 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.143 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.143 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:31.143 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.034244+0000", 2024-09-21T12:32:31.143 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000121495 2024-09-21T12:32:31.143 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.143 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.143 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:32:31.143 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.035876+0000", 2024-09-21T12:32:31.143 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0016319279999999999 2024-09-21T12:32:31.143 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.143 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.143 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:32:31.143 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.036000+0000", 2024-09-21T12:32:31.144 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000124447 2024-09-21T12:32:31.144 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.144 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.144 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:31.144 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.036046+0000", 2024-09-21T12:32:31.144 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5884000000000003e-05 2024-09-21T12:32:31.144 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.144 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:31.144 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.144 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.144 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.144 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:180336 9.0 9:0411b9cd:::benchmark_data_smithi114_133214_object56302:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-21T12:32:31.144 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:02.145721+0000", 2024-09-21T12:32:31.144 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 88.977642510999999, 2024-09-21T12:32:31.144 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.89603516699999997, 2024-09-21T12:32:31.145 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:31.145 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:31.145 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:32:31.145 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:31.145 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:31.145 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:31.145 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 180336 2024-09-21T12:32:31.145 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.145 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:31.145 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.145 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:31.145 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.145721+0000", 2024-09-21T12:32:31.145 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.145 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.146 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.146 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:31.146 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.145721+0000", 2024-09-21T12:32:31.146 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.146 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.146 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.146 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:31.146 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.145723+0000", 2024-09-21T12:32:31.146 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.663e-06 2024-09-21T12:32:31.146 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.146 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.146 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:31.146 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.145728+0000", 2024-09-21T12:32:31.146 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6899999999999997e-06 2024-09-21T12:32:31.147 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.147 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.147 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:31.147 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.145744+0000", 2024-09-21T12:32:31.147 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5580999999999999e-05 2024-09-21T12:32:31.147 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.147 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.147 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:31.147 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.145751+0000", 2024-09-21T12:32:31.147 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3069999999999997e-06 2024-09-21T12:32:31.147 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.147 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.147 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:31.147 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.145817+0000", 2024-09-21T12:32:31.148 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6107999999999998e-05 2024-09-21T12:32:31.148 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.148 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.148 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:32:31.148 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.145826+0000", 2024-09-21T12:32:31.148 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.7499999999999992e-06 2024-09-21T12:32:31.148 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.148 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.148 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:31.148 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.034408+0000", 2024-09-21T12:32:31.148 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.88858221100000001 2024-09-21T12:32:31.148 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.148 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.148 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:31.149 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.034528+0000", 2024-09-21T12:32:31.149 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000120147 2024-09-21T12:32:31.149 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.149 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.149 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,6", 2024-09-21T12:32:31.149 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.034719+0000", 2024-09-21T12:32:31.149 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000190717 2024-09-21T12:32:31.149 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.149 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.149 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:31.149 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.037717+0000", 2024-09-21T12:32:31.149 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002998166 2024-09-21T12:32:31.149 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.149 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.149 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:32:31.150 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.039018+0000", 2024-09-21T12:32:31.150 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0013011819999999999 2024-09-21T12:32:31.150 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.150 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.150 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:31.150 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.041124+0000", 2024-09-21T12:32:31.150 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002105516 2024-09-21T12:32:31.150 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.150 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.150 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:32:31.150 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.041468+0000", 2024-09-21T12:32:31.150 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00034375600000000002 2024-09-21T12:32:31.150 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.150 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.150 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:31.151 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.041756+0000", 2024-09-21T12:32:31.151 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00028837300000000003 2024-09-21T12:32:31.151 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.151 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:31.151 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.151 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.151 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.151 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:180349 9.0 9:07c14967:::benchmark_data_smithi114_133214_object56315:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-21T12:32:31.151 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:02.157937+0000", 2024-09-21T12:32:31.151 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 88.965426424, 2024-09-21T12:32:31.151 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.88401848800000005, 2024-09-21T12:32:31.151 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:31.151 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:31.151 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:32:31.152 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:31.152 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:31.152 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:31.152 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 180349 2024-09-21T12:32:31.152 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.152 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:31.152 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.152 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:31.152 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.157937+0000", 2024-09-21T12:32:31.152 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.152 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.152 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.152 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:31.152 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.157937+0000", 2024-09-21T12:32:31.152 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.153 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.153 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.153 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:31.153 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.157938+0000", 2024-09-21T12:32:31.153 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.356e-06 2024-09-21T12:32:31.153 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.153 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.153 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:31.153 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.157942+0000", 2024-09-21T12:32:31.153 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8519999999999997e-06 2024-09-21T12:32:31.153 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.153 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.153 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:31.153 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.157951+0000", 2024-09-21T12:32:31.153 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0189999999999995e-06 2024-09-21T12:32:31.154 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.154 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.154 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:31.154 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.157957+0000", 2024-09-21T12:32:31.154 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7570000000000001e-06 2024-09-21T12:32:31.154 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.154 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.154 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:31.154 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.158094+0000", 2024-09-21T12:32:31.154 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000137379 2024-09-21T12:32:31.154 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.154 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.154 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:32:31.154 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.158103+0000", 2024-09-21T12:32:31.155 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9099999999999994e-06 2024-09-21T12:32:31.155 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.155 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.155 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:31.155 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.036077+0000", 2024-09-21T12:32:31.155 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.87797348799999997 2024-09-21T12:32:31.155 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.155 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.155 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:31.155 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.036172+0000", 2024-09-21T12:32:31.155 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.4819000000000001e-05 2024-09-21T12:32:31.155 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.155 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.155 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,6", 2024-09-21T12:32:31.156 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.036360+0000", 2024-09-21T12:32:31.156 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00018840799999999999 2024-09-21T12:32:31.156 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.156 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.156 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:31.156 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.041085+0000", 2024-09-21T12:32:31.156 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0047250180000000001 2024-09-21T12:32:31.156 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.156 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.156 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:31.156 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.041836+0000", 2024-09-21T12:32:31.156 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00075079699999999999 2024-09-21T12:32:31.156 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.156 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.156 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:32:31.157 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.041860+0000", 2024-09-21T12:32:31.157 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.4672999999999999e-05 2024-09-21T12:32:31.157 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.157 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.157 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:32:31.157 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.041913+0000", 2024-09-21T12:32:31.157 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2494000000000003e-05 2024-09-21T12:32:31.157 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.157 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.157 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:31.157 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.041955+0000", 2024-09-21T12:32:31.157 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.2518000000000002e-05 2024-09-21T12:32:31.157 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.157 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:31.158 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.158 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.158 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.158 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:180350 9.0 9:0395bf83:::benchmark_data_smithi114_133214_object56316:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-21T12:32:31.158 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:02.157968+0000", 2024-09-21T12:32:31.158 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 88.965395709999996, 2024-09-21T12:32:31.158 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.88417305000000002, 2024-09-21T12:32:31.158 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:31.158 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:31.158 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:32:31.158 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:31.158 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:31.158 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:31.158 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 180350 2024-09-21T12:32:31.159 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.159 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:31.159 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.159 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:31.159 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.157968+0000", 2024-09-21T12:32:31.159 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.159 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.159 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.159 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:31.159 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.157968+0000", 2024-09-21T12:32:31.159 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.159 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.159 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.160 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:31.160 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.157968+0000", 2024-09-21T12:32:31.160 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3300000000000003e-07 2024-09-21T12:32:31.160 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.160 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.160 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:31.160 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.157970+0000", 2024-09-21T12:32:31.160 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9319999999999999e-06 2024-09-21T12:32:31.160 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.160 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.160 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:31.160 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.157976+0000", 2024-09-21T12:32:31.160 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5369999999999997e-06 2024-09-21T12:32:31.160 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.160 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.160 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:31.161 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.157980+0000", 2024-09-21T12:32:31.161 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5419999999999999e-06 2024-09-21T12:32:31.161 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.161 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.161 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:31.161 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.158127+0000", 2024-09-21T12:32:31.161 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000146933 2024-09-21T12:32:31.161 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.161 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.161 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:32:31.161 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.158133+0000", 2024-09-21T12:32:31.161 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1940000000000003e-06 2024-09-21T12:32:31.161 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.161 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.162 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:31.162 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.037787+0000", 2024-09-21T12:32:31.162 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.87965400000000005 2024-09-21T12:32:31.162 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.162 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.162 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:31.162 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.037891+0000", 2024-09-21T12:32:31.162 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010336399999999999 2024-09-21T12:32:31.162 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.162 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.162 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,6", 2024-09-21T12:32:31.162 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.038075+0000", 2024-09-21T12:32:31.162 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00018392699999999999 2024-09-21T12:32:31.162 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.163 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.163 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:31.163 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.042000+0000", 2024-09-21T12:32:31.163 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003925174 2024-09-21T12:32:31.163 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.163 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.163 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:32:31.163 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.042010+0000", 2024-09-21T12:32:31.163 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0037000000000001e-05 2024-09-21T12:32:31.163 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.163 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.163 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:31.163 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.042054+0000", 2024-09-21T12:32:31.163 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4100999999999997e-05 2024-09-21T12:32:31.163 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.164 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.164 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:32:31.164 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.042097+0000", 2024-09-21T12:32:31.164 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3510000000000002e-05 2024-09-21T12:32:31.164 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.164 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.164 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:31.164 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.042141+0000", 2024-09-21T12:32:31.164 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3365999999999998e-05 2024-09-21T12:32:31.164 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.164 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:31.164 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.164 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.164 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.164 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:180357 9.0 9:06cc3e12:::benchmark_data_smithi114_133214_object56323:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-21T12:32:31.164 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:02.163130+0000", 2024-09-21T12:32:31.165 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 88.960233635999998, 2024-09-21T12:32:31.165 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.87918328199999995, 2024-09-21T12:32:31.165 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:31.165 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:31.165 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:32:31.165 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:31.165 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:31.165 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:31.165 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 180357 2024-09-21T12:32:31.165 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.165 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:31.165 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.165 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:31.165 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.163130+0000", 2024-09-21T12:32:31.165 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.166 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.166 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.166 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:31.166 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.163130+0000", 2024-09-21T12:32:31.166 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.166 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.166 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.166 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:31.166 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.163131+0000", 2024-09-21T12:32:31.166 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3969999999999999e-06 2024-09-21T12:32:31.166 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.166 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.166 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:31.166 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.163136+0000", 2024-09-21T12:32:31.166 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9710000000000003e-06 2024-09-21T12:32:31.167 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.167 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.167 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:31.167 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.163151+0000", 2024-09-21T12:32:31.167 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4558e-05 2024-09-21T12:32:31.167 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.167 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.167 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:31.167 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.163158+0000", 2024-09-21T12:32:31.167 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5399999999999998e-06 2024-09-21T12:32:31.167 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.167 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.167 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:31.167 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.163224+0000", 2024-09-21T12:32:31.167 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5479000000000001e-05 2024-09-21T12:32:31.168 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.168 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.168 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:32:31.168 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.163233+0000", 2024-09-21T12:32:31.168 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.5200000000000003e-06 2024-09-21T12:32:31.168 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.168 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.168 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:31.168 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.039053+0000", 2024-09-21T12:32:31.168 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.87581987299999997 2024-09-21T12:32:31.168 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.168 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.168 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:31.168 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.039151+0000", 2024-09-21T12:32:31.169 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7744999999999993e-05 2024-09-21T12:32:31.169 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.169 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.169 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,6", 2024-09-21T12:32:31.169 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.039330+0000", 2024-09-21T12:32:31.169 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017912099999999999 2024-09-21T12:32:31.169 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.169 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.169 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:32:31.169 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.042014+0000", 2024-09-21T12:32:31.169 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002683843 2024-09-21T12:32:31.169 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.169 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.169 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:31.169 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.042193+0000", 2024-09-21T12:32:31.169 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017954700000000001 2024-09-21T12:32:31.170 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.170 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.170 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:31.171 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.042235+0000", 2024-09-21T12:32:31.171 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.2110000000000002e-05 2024-09-21T12:32:31.171 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.171 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.171 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:32:31.171 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.042280+0000", 2024-09-21T12:32:31.171 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4397000000000002e-05 2024-09-21T12:32:31.171 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.171 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.172 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:31.172 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.042313+0000", 2024-09-21T12:32:31.172 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.3181e-05 2024-09-21T12:32:31.172 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.172 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:31.172 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.172 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.172 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.172 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:180773 9.7 9:e34209ad:::benchmark_data_smithi114_133214_object56739:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-21T12:32:31.172 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:03.118125+0000", 2024-09-21T12:32:31.172 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 88.005238367999993, 2024-09-21T12:32:31.172 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.93373279499999995, 2024-09-21T12:32:31.172 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:31.172 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:31.173 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:32:31.173 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:31.173 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:31.173 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:31.173 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 180773 2024-09-21T12:32:31.173 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.173 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:31.173 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.173 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:31.173 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.118125+0000", 2024-09-21T12:32:31.173 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.173 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.173 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.174 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:31.174 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.118125+0000", 2024-09-21T12:32:31.174 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.174 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.174 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.174 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:31.174 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.118127+0000", 2024-09-21T12:32:31.174 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2029999999999999e-06 2024-09-21T12:32:31.174 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.174 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.174 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:31.174 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.118133+0000", 2024-09-21T12:32:31.174 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4609999999999999e-06 2024-09-21T12:32:31.175 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.175 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.175 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:31.175 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.118146+0000", 2024-09-21T12:32:31.175 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3028e-05 2024-09-21T12:32:31.175 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.175 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.175 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:31.175 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.118154+0000", 2024-09-21T12:32:31.175 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6689999999999995e-06 2024-09-21T12:32:31.175 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.175 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.175 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:31.175 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.118215+0000", 2024-09-21T12:32:31.176 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0656999999999999e-05 2024-09-21T12:32:31.176 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.176 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.176 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:32:31.176 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.118225+0000", 2024-09-21T12:32:31.176 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.4949999999999997e-06 2024-09-21T12:32:31.176 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.176 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.176 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:31.176 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.039269+0000", 2024-09-21T12:32:31.176 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.92104478199999995 2024-09-21T12:32:31.176 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.176 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.176 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:31.177 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.039389+0000", 2024-09-21T12:32:31.177 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012013700000000001 2024-09-21T12:32:31.177 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.177 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.177 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,5", 2024-09-21T12:32:31.177 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.039663+0000", 2024-09-21T12:32:31.177 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00027322400000000003 2024-09-21T12:32:31.177 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.177 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.177 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:32:31.177 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.041271+0000", 2024-09-21T12:32:31.177 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001607898 2024-09-21T12:32:31.177 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.177 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.178 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:31.178 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.041720+0000", 2024-09-21T12:32:31.178 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00044909799999999999 2024-09-21T12:32:31.178 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.178 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.178 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:31.178 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.051742+0000", 2024-09-21T12:32:31.178 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010022771999999999 2024-09-21T12:32:31.178 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.178 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.178 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:32:31.178 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.051813+0000", 2024-09-21T12:32:31.178 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0111999999999996e-05 2024-09-21T12:32:31.178 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.179 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.179 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:31.179 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.051858+0000", 2024-09-21T12:32:31.179 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5259000000000002e-05 2024-09-21T12:32:31.179 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.179 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:31.179 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.179 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.179 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.179 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:180819 9.7 9:e0af09d4:::benchmark_data_smithi114_133214_object56785:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-21T12:32:31.179 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:03.162695+0000", 2024-09-21T12:32:31.179 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 87.960668619000003, 2024-09-21T12:32:31.179 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.88927357500000004, 2024-09-21T12:32:31.180 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:31.180 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:31.180 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:32:31.180 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:31.180 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:31.180 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:31.180 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 180819 2024-09-21T12:32:31.180 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.180 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:31.180 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.180 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:31.180 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.162695+0000", 2024-09-21T12:32:31.180 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.180 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.180 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.181 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:31.181 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.162695+0000", 2024-09-21T12:32:31.181 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.181 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.181 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.181 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:31.181 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.162696+0000", 2024-09-21T12:32:31.181 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5409999999999999e-06 2024-09-21T12:32:31.181 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.181 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.181 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:31.181 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.162702+0000", 2024-09-21T12:32:31.181 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0360000000000003e-06 2024-09-21T12:32:31.181 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.182 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.182 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:31.182 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.162715+0000", 2024-09-21T12:32:31.182 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2429000000000001e-05 2024-09-21T12:32:31.182 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.182 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.182 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:31.182 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.162723+0000", 2024-09-21T12:32:31.182 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7079999999999997e-06 2024-09-21T12:32:31.182 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.182 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.182 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:31.182 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.162790+0000", 2024-09-21T12:32:31.182 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.758e-05 2024-09-21T12:32:31.182 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.183 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.183 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:32:31.183 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.162799+0000", 2024-09-21T12:32:31.183 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6740000000000003e-06 2024-09-21T12:32:31.183 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.183 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.183 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:31.183 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.041759+0000", 2024-09-21T12:32:31.183 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.87896034000000001 2024-09-21T12:32:31.183 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.183 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.183 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:31.183 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.041890+0000", 2024-09-21T12:32:31.183 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013109299999999999 2024-09-21T12:32:31.184 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.184 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.184 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,5", 2024-09-21T12:32:31.184 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.042159+0000", 2024-09-21T12:32:31.184 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00026918800000000001 2024-09-21T12:32:31.184 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.184 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.184 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:31.184 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.045265+0000", 2024-09-21T12:32:31.184 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0031053299999999999 2024-09-21T12:32:31.184 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.184 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.184 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:32:31.185 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.045434+0000", 2024-09-21T12:32:31.185 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000169471 2024-09-21T12:32:31.185 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.185 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.185 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:31.185 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.051892+0000", 2024-09-21T12:32:31.185 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0064578580000000003 2024-09-21T12:32:31.185 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.185 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.185 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:32:31.185 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.051937+0000", 2024-09-21T12:32:31.185 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5317e-05 2024-09-21T12:32:31.185 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.185 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.186 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:31.186 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.051968+0000", 2024-09-21T12:32:31.186 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.1010000000000003e-05 2024-09-21T12:32:31.186 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.186 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:31.186 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.186 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.186 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.186 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:180821 9.7 9:e14e00c0:::benchmark_data_smithi114_133214_object56787:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-09-21T12:32:31.186 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:03.164517+0000", 2024-09-21T12:32:31.186 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 87.958846215999998, 2024-09-21T12:32:31.186 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.89406111899999996, 2024-09-21T12:32:31.186 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:31.187 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:31.187 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:32:31.187 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:31.187 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:31.187 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:31.187 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 180821 2024-09-21T12:32:31.187 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.187 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:31.187 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.187 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:31.187 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.164517+0000", 2024-09-21T12:32:31.187 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.187 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.187 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.188 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:31.188 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.164517+0000", 2024-09-21T12:32:31.188 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.188 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.188 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.188 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:31.188 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.164518+0000", 2024-09-21T12:32:31.188 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.127e-06 2024-09-21T12:32:31.188 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.188 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.188 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:31.188 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.164524+0000", 2024-09-21T12:32:31.188 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.181e-06 2024-09-21T12:32:31.188 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.189 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.189 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:31.189 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.164537+0000", 2024-09-21T12:32:31.189 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3743999999999999e-05 2024-09-21T12:32:31.189 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.189 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.189 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:31.189 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.164545+0000", 2024-09-21T12:32:31.189 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1580000000000005e-06 2024-09-21T12:32:31.189 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.189 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.189 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:31.189 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.164634+0000", 2024-09-21T12:32:31.189 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.8555999999999997e-05 2024-09-21T12:32:31.190 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.190 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.190 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:32:31.190 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:03.164642+0000", 2024-09-21T12:32:31.190 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.0229999999999992e-06 2024-09-21T12:32:31.190 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.190 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.190 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:31.190 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.043609+0000", 2024-09-21T12:32:31.190 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.87896723799999998 2024-09-21T12:32:31.190 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.190 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.190 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:31.190 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.043720+0000", 2024-09-21T12:32:31.191 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000110714 2024-09-21T12:32:31.191 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.191 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.191 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,5", 2024-09-21T12:32:31.191 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.043945+0000", 2024-09-21T12:32:31.191 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000225107 2024-09-21T12:32:31.191 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.191 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.191 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:32:31.191 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.046185+0000", 2024-09-21T12:32:31.191 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0022396019999999998 2024-09-21T12:32:31.191 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.191 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.191 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:31.192 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.047707+0000", 2024-09-21T12:32:31.192 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0015223260000000001 2024-09-21T12:32:31.192 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.192 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.192 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:31.192 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.053913+0000", 2024-09-21T12:32:31.192 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0062060780000000003 2024-09-21T12:32:31.192 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.192 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.192 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:32:31.192 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.053972+0000", 2024-09-21T12:32:31.192 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8972999999999999e-05 2024-09-21T12:32:31.192 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.192 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.193 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:31.193 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.058578+0000", 2024-09-21T12:32:31.193 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0046062919999999997 2024-09-21T12:32:31.193 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.193 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:31.193 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.193 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.193 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.193 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231208 9.7 9:e255abb1:::benchmark_data_smithi114_133214_object107174:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-09-21T12:32:31.193 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:48.662634+0000", 2024-09-21T12:32:31.193 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.460729487999998, 2024-09-21T12:32:31.193 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.033002054000000003, 2024-09-21T12:32:31.193 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:31.193 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:31.193 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:32:31.194 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:31.194 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:31.194 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:31.194 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231208 2024-09-21T12:32:31.194 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.194 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:31.194 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.194 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:31.194 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:48.662634+0000", 2024-09-21T12:32:31.194 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.194 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.194 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.194 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:31.194 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:48.662634+0000", 2024-09-21T12:32:31.195 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.195 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.195 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.195 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:31.195 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:48.662635+0000", 2024-09-21T12:32:31.195 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5200000000000001e-06 2024-09-21T12:32:31.195 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.195 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.195 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:31.195 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:48.662641+0000", 2024-09-21T12:32:31.195 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8320000000000002e-06 2024-09-21T12:32:31.195 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.195 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.195 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:31.196 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:48.662655+0000", 2024-09-21T12:32:31.196 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3906999999999999e-05 2024-09-21T12:32:31.196 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.196 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.196 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:31.196 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:48.662663+0000", 2024-09-21T12:32:31.196 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1310000000000006e-06 2024-09-21T12:32:31.196 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.196 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.196 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:31.196 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:48.666644+0000", 2024-09-21T12:32:31.196 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0039805659999999996 2024-09-21T12:32:31.196 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.196 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.196 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:31.197 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:48.666750+0000", 2024-09-21T12:32:31.197 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010648400000000001 2024-09-21T12:32:31.197 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.197 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.197 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,5", 2024-09-21T12:32:31.197 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:48.667348+0000", 2024-09-21T12:32:31.197 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00059763100000000005 2024-09-21T12:32:31.197 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.197 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.197 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:32:31.197 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:48.673726+0000", 2024-09-21T12:32:31.197 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0063776789999999998 2024-09-21T12:32:31.197 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.198 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.198 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:31.198 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:48.675622+0000", 2024-09-21T12:32:31.198 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0018960750000000001 2024-09-21T12:32:31.198 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.198 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.198 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:31.198 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:48.695413+0000", 2024-09-21T12:32:31.198 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.019791601999999998 2024-09-21T12:32:31.198 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.198 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.198 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:32:31.198 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:48.695588+0000", 2024-09-21T12:32:31.198 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017446599999999999 2024-09-21T12:32:31.199 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.199 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.199 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:31.199 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:48.695636+0000", 2024-09-21T12:32:31.199 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8161000000000002e-05 2024-09-21T12:32:31.199 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.199 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:31.199 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.199 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.199 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.199 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:232041 9.6 9:6090621d:::benchmark_data_smithi114_133214_object108007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-09-21T12:32:31.199 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:52.097737+0000", 2024-09-21T12:32:31.199 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 39.025625949000002, 2024-09-21T12:32:31.199 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.031901460999999999, 2024-09-21T12:32:31.200 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:31.200 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:31.200 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:32:31.200 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:31.200 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:31.200 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:31.200 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 232041 2024-09-21T12:32:31.200 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.200 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:31.200 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.200 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:31.200 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.097737+0000", 2024-09-21T12:32:31.200 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.200 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.201 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.201 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:31.201 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.097737+0000", 2024-09-21T12:32:31.201 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.201 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.201 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.201 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:31.201 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.097739+0000", 2024-09-21T12:32:31.201 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8959999999999999e-06 2024-09-21T12:32:31.201 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.201 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.201 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:31.201 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.097746+0000", 2024-09-21T12:32:31.201 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7139999999999996e-06 2024-09-21T12:32:31.202 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.202 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.202 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:31.202 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.097764+0000", 2024-09-21T12:32:31.202 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7433999999999999e-05 2024-09-21T12:32:31.202 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.202 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.202 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:31.202 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.097771+0000", 2024-09-21T12:32:31.202 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9179999999999994e-06 2024-09-21T12:32:31.202 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.202 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.203 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:31.203 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.097841+0000", 2024-09-21T12:32:31.203 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9898000000000001e-05 2024-09-21T12:32:31.203 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.203 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.203 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:31.203 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.097960+0000", 2024-09-21T12:32:31.203 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011861799999999999 2024-09-21T12:32:31.203 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.203 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.203 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,5", 2024-09-21T12:32:31.203 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.098318+0000", 2024-09-21T12:32:31.203 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00035760499999999999 2024-09-21T12:32:31.203 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.204 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.204 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:32:31.204 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.099882+0000", 2024-09-21T12:32:31.204 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0015645500000000001 2024-09-21T12:32:31.204 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.204 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.204 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:31.204 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.100364+0000", 2024-09-21T12:32:31.204 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00048160299999999999 2024-09-21T12:32:31.204 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.204 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.204 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:31.204 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.129519+0000", 2024-09-21T12:32:31.204 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.029154975 2024-09-21T12:32:31.204 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.205 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.205 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:32:31.205 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.129598+0000", 2024-09-21T12:32:31.205 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9021000000000001e-05 2024-09-21T12:32:31.205 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.205 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.205 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:31.205 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.129639+0000", 2024-09-21T12:32:31.205 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1229000000000001e-05 2024-09-21T12:32:31.205 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.205 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:31.205 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.205 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.205 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.206 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:233020 9.6 9:60f2de37:::benchmark_data_smithi114_133214_object108986:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-09-21T12:32:31.206 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:54.444150+0000", 2024-09-21T12:32:31.206 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 36.679213394999998, 2024-09-21T12:32:31.206 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.038971363000000002, 2024-09-21T12:32:31.206 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:31.206 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:31.206 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:32:31.206 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:31.206 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:31.206 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:31.206 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 233020 2024-09-21T12:32:31.206 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.206 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:31.206 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.206 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:31.207 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.444150+0000", 2024-09-21T12:32:31.207 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.207 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.207 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.207 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:31.207 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.444150+0000", 2024-09-21T12:32:31.207 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.207 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.207 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.207 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:31.207 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.444151+0000", 2024-09-21T12:32:31.207 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.336e-06 2024-09-21T12:32:31.207 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.208 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.208 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:31.208 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.444156+0000", 2024-09-21T12:32:31.208 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8799999999999999e-06 2024-09-21T12:32:31.208 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.208 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.208 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:31.208 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.444167+0000", 2024-09-21T12:32:31.208 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1161999999999999e-05 2024-09-21T12:32:31.208 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.208 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.208 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:31.208 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.444172+0000", 2024-09-21T12:32:31.208 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0520000000000001e-06 2024-09-21T12:32:31.209 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.209 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.209 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:31.209 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.444467+0000", 2024-09-21T12:32:31.209 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00029415099999999998 2024-09-21T12:32:31.209 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.209 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.209 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:31.209 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.444568+0000", 2024-09-21T12:32:31.209 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010112599999999999 2024-09-21T12:32:31.209 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.209 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.209 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,5", 2024-09-21T12:32:31.209 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.445277+0000", 2024-09-21T12:32:31.210 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00070975599999999997 2024-09-21T12:32:31.210 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.210 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.210 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:32:31.210 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.448259+0000", 2024-09-21T12:32:31.210 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0029813830000000002 2024-09-21T12:32:31.210 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.210 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.210 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:31.210 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.449277+0000", 2024-09-21T12:32:31.210 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001017902 2024-09-21T12:32:31.210 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.210 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.210 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:31.211 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.482970+0000", 2024-09-21T12:32:31.211 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.033693146 2024-09-21T12:32:31.211 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.211 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.211 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:32:31.211 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.483076+0000", 2024-09-21T12:32:31.211 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000105969 2024-09-21T12:32:31.211 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.211 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.211 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:31.211 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.483121+0000", 2024-09-21T12:32:31.211 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5500000000000001e-05 2024-09-21T12:32:31.211 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.211 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:31.212 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.212 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.212 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.212 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:236194 9.6 9:66c04880:::benchmark_data_smithi114_133214_object112160:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-09-21T12:32:31.212 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:56.865504+0000", 2024-09-21T12:32:31.212 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 34.257859068999998, 2024-09-21T12:32:31.212 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.032592029000000002, 2024-09-21T12:32:31.212 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:31.212 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:31.212 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:32:31.212 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:31.212 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:32:31.212 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:32:31.213 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 236194 2024-09-21T12:32:31.213 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.213 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:31.213 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.213 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:31.213 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:56.865504+0000", 2024-09-21T12:32:31.213 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.213 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.213 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.213 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:31.213 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:56.865504+0000", 2024-09-21T12:32:31.213 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.213 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.213 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.214 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:31.214 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:56.865506+0000", 2024-09-21T12:32:31.214 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9910000000000001e-06 2024-09-21T12:32:31.214 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.214 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.214 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:31.214 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:56.865512+0000", 2024-09-21T12:32:31.214 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6559999999999997e-06 2024-09-21T12:32:31.214 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.214 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.214 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:31.214 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:56.865526+0000", 2024-09-21T12:32:31.214 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3728e-05 2024-09-21T12:32:31.214 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.215 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.215 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:31.215 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:56.865534+0000", 2024-09-21T12:32:31.215 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9000000000000006e-06 2024-09-21T12:32:31.215 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.215 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.215 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:31.215 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:56.865602+0000", 2024-09-21T12:32:31.215 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7955999999999998e-05 2024-09-21T12:32:31.215 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.215 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.215 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:31.215 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:56.865707+0000", 2024-09-21T12:32:31.215 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000105388 2024-09-21T12:32:31.215 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.216 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.216 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,5", 2024-09-21T12:32:31.216 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:56.866093+0000", 2024-09-21T12:32:31.216 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00038556099999999998 2024-09-21T12:32:31.216 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.216 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.216 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:32:31.216 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:56.867871+0000", 2024-09-21T12:32:31.216 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0017787370000000001 2024-09-21T12:32:31.216 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.216 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.216 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:31.216 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:56.868213+0000", 2024-09-21T12:32:31.216 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00034127400000000003 2024-09-21T12:32:31.217 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.217 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.217 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:32:31.217 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:56.897768+0000", 2024-09-21T12:32:31.217 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.029554951999999999 2024-09-21T12:32:31.217 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.217 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.217 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:32:31.217 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:56.897956+0000", 2024-09-21T12:32:31.217 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00018814600000000001 2024-09-21T12:32:31.217 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.217 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.217 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:31.217 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:56.898096+0000", 2024-09-21T12:32:31.217 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014074000000000001 2024-09-21T12:32:31.218 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.218 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:31.218 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.218 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.218 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.218 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.14478.0:2850 5.1f 5:f95f44c2:::notify.0:head [watch ping cookie 94620848892160 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-21T12:32:31.218 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:32:10.499364+0000", 2024-09-21T12:32:31.218 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 20.623999638000001, 2024-09-21T12:32:31.218 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00075564600000000001, 2024-09-21T12:32:31.218 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:31.218 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:31.218 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:32:31.218 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:31.218 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.14478", 2024-09-21T12:32:31.219 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:32:31.219 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 2850 2024-09-21T12:32:31.219 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.219 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:31.219 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.219 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:31.219 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:10.499364+0000", 2024-09-21T12:32:31.219 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.219 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.219 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.219 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:31.219 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:10.499364+0000", 2024-09-21T12:32:31.219 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.219 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.219 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.220 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:31.220 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:10.499367+0000", 2024-09-21T12:32:31.220 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.8200000000000001e-06 2024-09-21T12:32:31.220 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.220 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.220 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:31.220 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:10.499379+0000", 2024-09-21T12:32:31.220 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1892999999999999e-05 2024-09-21T12:32:31.220 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.220 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.220 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:31.220 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:10.499422+0000", 2024-09-21T12:32:31.220 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3755999999999999e-05 2024-09-21T12:32:31.220 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.221 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.221 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:31.221 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:10.499444+0000", 2024-09-21T12:32:31.221 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1656999999999999e-05 2024-09-21T12:32:31.221 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.221 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.221 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:31.221 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:10.499996+0000", 2024-09-21T12:32:31.221 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00055236499999999997 2024-09-21T12:32:31.221 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.221 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.221 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:31.221 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:10.500057+0000", 2024-09-21T12:32:31.221 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0949000000000002e-05 2024-09-21T12:32:31.221 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.222 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.222 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:31.222 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:10.500119+0000", 2024-09-21T12:32:31.222 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2205999999999994e-05 2024-09-21T12:32:31.222 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.222 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:31.222 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.222 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.222 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.222 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.14478.0:2851 5.d 5:b4812045:::notify.4:head [watch ping cookie 94620848896768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-21T12:32:31.222 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:32:10.499538+0000", 2024-09-21T12:32:31.222 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 20.623825318000002, 2024-09-21T12:32:31.222 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000451946, 2024-09-21T12:32:31.222 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:31.223 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:31.223 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:32:31.223 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:31.223 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.14478", 2024-09-21T12:32:31.223 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:32:31.223 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 2851 2024-09-21T12:32:31.223 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.223 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:31.223 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.223 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:31.223 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:10.499538+0000", 2024-09-21T12:32:31.223 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.223 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.223 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.223 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:31.224 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:10.499538+0000", 2024-09-21T12:32:31.224 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.224 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.224 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.224 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:31.224 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:10.499539+0000", 2024-09-21T12:32:31.224 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2500000000000001e-06 2024-09-21T12:32:31.224 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.224 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.224 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:31.224 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:10.499546+0000", 2024-09-21T12:32:31.224 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3439999999999997e-06 2024-09-21T12:32:31.224 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.224 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.224 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:31.225 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:10.499564+0000", 2024-09-21T12:32:31.225 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8655999999999999e-05 2024-09-21T12:32:31.225 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.225 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.225 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:31.225 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:10.499583+0000", 2024-09-21T12:32:31.225 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8643000000000001e-05 2024-09-21T12:32:31.225 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.225 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.225 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:31.225 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:10.499797+0000", 2024-09-21T12:32:31.225 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00021393000000000001 2024-09-21T12:32:31.225 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.225 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.226 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:31.226 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:10.499897+0000", 2024-09-21T12:32:31.226 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000100505 2024-09-21T12:32:31.226 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.226 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.226 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:31.226 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:10.499990+0000", 2024-09-21T12:32:31.226 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.2618e-05 2024-09-21T12:32:31.226 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.226 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:31.226 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.226 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.226 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.226 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "MOSDScrubReserve(1.8,GRANT,e:343,from:3,reservation_nonce:0)", 2024-09-21T12:32:31.227 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:32:28.687564+0000", 2024-09-21T12:32:31.227 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 2.4357996489999998, 2024-09-21T12:32:31.227 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000106128, 2024-09-21T12:32:31.227 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:31.227 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:31.227 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:32:31.227 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:31.227 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.227 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:31.227 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.687564+0000", 2024-09-21T12:32:31.227 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.227 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.227 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.227 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:31.228 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.687564+0000", 2024-09-21T12:32:31.228 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.228 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.228 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.228 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:31.228 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.687564+0000", 2024-09-21T12:32:31.228 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1099999999999995e-07 2024-09-21T12:32:31.228 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.228 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.228 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:31.228 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.687569+0000", 2024-09-21T12:32:31.228 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3089999999999997e-06 2024-09-21T12:32:31.228 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.228 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.229 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:31.229 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.687578+0000", 2024-09-21T12:32:31.229 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0839999999999998e-06 2024-09-21T12:32:31.229 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.229 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.229 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:31.229 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.687583+0000", 2024-09-21T12:32:31.229 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4090000000000002e-06 2024-09-21T12:32:31.229 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.229 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.229 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:31.229 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.687627+0000", 2024-09-21T12:32:31.229 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3438e-05 2024-09-21T12:32:31.229 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.230 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.230 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:31.230 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.687638+0000", 2024-09-21T12:32:31.230 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1659e-05 2024-09-21T12:32:31.230 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.230 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.230 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:31.230 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.687670+0000", 2024-09-21T12:32:31.230 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.1517999999999999e-05 2024-09-21T12:32:31.230 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.230 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:31.230 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.230 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.230 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.231 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "MOSDScrubReserve(1.8,GRANT,e:343,from:5,reservation_nonce:0)", 2024-09-21T12:32:31.231 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:32:28.688165+0000", 2024-09-21T12:32:31.231 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 2.435198449, 2024-09-21T12:32:31.231 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00021958700000000001, 2024-09-21T12:32:31.231 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:31.231 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:31.231 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:32:31.231 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:31.231 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.231 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:31.231 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.688165+0000", 2024-09-21T12:32:31.231 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.231 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.231 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.232 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:31.232 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.688165+0000", 2024-09-21T12:32:31.232 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.232 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.232 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.232 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:31.232 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.688166+0000", 2024-09-21T12:32:31.232 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6000000000000003e-07 2024-09-21T12:32:31.232 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.232 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.232 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:31.232 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.688169+0000", 2024-09-21T12:32:31.232 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.05e-06 2024-09-21T12:32:31.232 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.232 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.233 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:31.233 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.688179+0000", 2024-09-21T12:32:31.233 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0040999999999999e-05 2024-09-21T12:32:31.233 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.233 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.233 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:31.233 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.688184+0000", 2024-09-21T12:32:31.233 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9130000000000002e-06 2024-09-21T12:32:31.233 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.233 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.233 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:31.233 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.688229+0000", 2024-09-21T12:32:31.233 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5058000000000001e-05 2024-09-21T12:32:31.233 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.233 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.234 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:31.234 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.688248+0000", 2024-09-21T12:32:31.234 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9000000000000001e-05 2024-09-21T12:32:31.234 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.234 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.234 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:31.234 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.688385+0000", 2024-09-21T12:32:31.234 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013686500000000001 2024-09-21T12:32:31.234 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.234 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:31.234 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.234 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.234 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.234 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "rep_scrubmap(1.8 e325 from shard 3)", 2024-09-21T12:32:31.235 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:32:28.688985+0000", 2024-09-21T12:32:31.235 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 2.4343781660000001, 2024-09-21T12:32:31.235 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000126064, 2024-09-21T12:32:31.235 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:31.235 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:31.235 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:32:31.235 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:31.235 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.235 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:31.235 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.688985+0000", 2024-09-21T12:32:31.235 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.235 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.235 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.235 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:31.235 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.688985+0000", 2024-09-21T12:32:31.236 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.236 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.236 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.236 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:31.236 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.688986+0000", 2024-09-21T12:32:31.236 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7000000000000004e-07 2024-09-21T12:32:31.236 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.236 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.236 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:31.236 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.688990+0000", 2024-09-21T12:32:31.236 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8039999999999999e-06 2024-09-21T12:32:31.236 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.236 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.236 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:31.237 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.689005+0000", 2024-09-21T12:32:31.237 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4810999999999999e-05 2024-09-21T12:32:31.237 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.237 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.237 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:31.237 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.689010+0000", 2024-09-21T12:32:31.237 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3109999999999996e-06 2024-09-21T12:32:31.237 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.237 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.237 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:31.237 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.689087+0000", 2024-09-21T12:32:31.237 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6712999999999995e-05 2024-09-21T12:32:31.237 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.237 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.237 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:31.238 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.689093+0000", 2024-09-21T12:32:31.238 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2729999999999999e-06 2024-09-21T12:32:31.238 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.238 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.238 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:31.238 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.689111+0000", 2024-09-21T12:32:31.238 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8482e-05 2024-09-21T12:32:31.238 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.238 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:31.238 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.238 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.238 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.238 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "rep_scrubmap(1.8 e325 from shard 5)", 2024-09-21T12:32:31.238 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:32:28.688986+0000", 2024-09-21T12:32:31.239 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 2.4343771329999999, 2024-09-21T12:32:31.239 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5671e-05, 2024-09-21T12:32:31.239 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:31.239 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:31.239 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:32:31.239 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:31.239 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.239 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:31.239 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.688986+0000", 2024-09-21T12:32:31.239 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.239 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.239 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.239 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:31.239 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.688986+0000", 2024-09-21T12:32:31.240 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.240 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.240 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.240 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:31.240 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.688987+0000", 2024-09-21T12:32:31.240 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2699999999999999e-07 2024-09-21T12:32:31.240 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.240 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.240 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:31.240 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.688990+0000", 2024-09-21T12:32:31.240 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2909999999999999e-06 2024-09-21T12:32:31.240 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.240 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.240 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:31.241 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.689000+0000", 2024-09-21T12:32:31.241 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7929999999999992e-06 2024-09-21T12:32:31.241 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.241 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.241 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:31.241 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.689004+0000", 2024-09-21T12:32:31.241 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3959999999999999e-06 2024-09-21T12:32:31.241 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.241 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.241 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:31.241 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.689048+0000", 2024-09-21T12:32:31.241 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3142000000000001e-05 2024-09-21T12:32:31.241 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.241 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.242 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:31.242 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.689057+0000", 2024-09-21T12:32:31.242 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.3989999999999993e-06 2024-09-21T12:32:31.242 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.242 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.242 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:31.242 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:28.689072+0000", 2024-09-21T12:32:31.242 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5023e-05 2024-09-21T12:32:31.242 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.242 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:31.242 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.242 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.242 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.242 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.14478.0:2892 5.d 5:b4812045:::notify.4:head [watch ping cookie 94620848896768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-09-21T12:32:31.243 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:32:30.502461+0000", 2024-09-21T12:32:31.243 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.62090235400000005, 2024-09-21T12:32:31.243 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00048993699999999997, 2024-09-21T12:32:31.243 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:32:31.243 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:32:31.243 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:32:31.243 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:32:31.243 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.14478", 2024-09-21T12:32:31.243 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:32:31.243 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 2892 2024-09-21T12:32:31.243 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.243 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:32:31.243 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.243 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:32:31.244 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:30.502461+0000", 2024-09-21T12:32:31.244 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.244 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.244 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.244 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:32:31.244 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:30.502461+0000", 2024-09-21T12:32:31.244 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:32:31.244 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.244 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.244 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:32:31.244 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:30.502464+0000", 2024-09-21T12:32:31.244 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.6400000000000001e-06 2024-09-21T12:32:31.244 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.244 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.244 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:32:31.245 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:30.502476+0000", 2024-09-21T12:32:31.245 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2359e-05 2024-09-21T12:32:31.245 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.245 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.245 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:32:31.245 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:30.502517+0000", 2024-09-21T12:32:31.245 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0753000000000001e-05 2024-09-21T12:32:31.245 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.245 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.245 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:32:31.245 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:30.502540+0000", 2024-09-21T12:32:31.245 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3252e-05 2024-09-21T12:32:31.245 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.245 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.246 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:32:31.246 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:30.502742+0000", 2024-09-21T12:32:31.246 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00020162500000000001 2024-09-21T12:32:31.246 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.246 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.246 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:32:31.246 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:30.502840+0000", 2024-09-21T12:32:31.246 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.8735999999999998e-05 2024-09-21T12:32:31.246 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:32:31.246 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:32:31.246 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:32:31.246 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:30.502951+0000", 2024-09-21T12:32:31.246 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000110572 2024-09-21T12:32:31.247 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.247 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:31.247 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.247 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:32:31.247 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:32:31.247 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:32:31.247 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:31.268 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:31.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:31.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:31.370 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:31.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:31.396 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:31.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:31.446 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:31 smithi114 ceph-mon[40867]: 5.13 scrub starts 2024-09-21T12:32:31.446 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:31 smithi114 ceph-mon[40867]: 5.13 scrub ok 2024-09-21T12:32:31.446 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:31 smithi114 ceph-mon[40867]: pgmap v624: 178 pgs: 16 unknown, 162 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T12:32:31.446 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:31 smithi114 ceph-mon[40867]: 5.1a scrub starts 2024-09-21T12:32:31.446 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:31 smithi114 ceph-mon[40867]: 5.1a scrub ok 2024-09-21T12:32:31.446 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:31 smithi114 ceph-mon[40867]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T12:32:31.446 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:31 smithi114 ceph-mon[40867]: osdmap e345: 8 total, 8 up, 7 in 2024-09-21T12:32:31.446 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:31 smithi114 ceph-mon[40867]: 4.13 scrub starts 2024-09-21T12:32:31.446 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:31 smithi114 ceph-mon[40867]: 4.13 scrub ok 2024-09-21T12:32:31.446 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:31 smithi114 ceph-mon[30324]: 5.13 scrub starts 2024-09-21T12:32:31.447 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:31 smithi114 ceph-mon[30324]: 5.13 scrub ok 2024-09-21T12:32:31.447 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:31 smithi114 ceph-mon[30324]: pgmap v624: 178 pgs: 16 unknown, 162 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T12:32:31.447 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:31 smithi114 ceph-mon[30324]: 5.1a scrub starts 2024-09-21T12:32:31.447 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:31 smithi114 ceph-mon[30324]: 5.1a scrub ok 2024-09-21T12:32:31.447 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:31 smithi114 ceph-mon[30324]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T12:32:31.447 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:31 smithi114 ceph-mon[30324]: osdmap e345: 8 total, 8 up, 7 in 2024-09-21T12:32:31.447 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:31 smithi114 ceph-mon[30324]: 4.13 scrub starts 2024-09-21T12:32:31.447 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:31 smithi114 ceph-mon[30324]: 4.13 scrub ok 2024-09-21T12:32:31.498 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:31.518 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:31.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:31.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:31.619 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:31.641 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:31.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:31.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:31.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:31 smithi183 ceph-mon[33277]: 5.13 scrub starts 2024-09-21T12:32:31.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:31 smithi183 ceph-mon[33277]: 5.13 scrub ok 2024-09-21T12:32:31.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:31 smithi183 ceph-mon[33277]: pgmap v624: 178 pgs: 16 unknown, 162 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T12:32:31.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:31 smithi183 ceph-mon[33277]: 5.1a scrub starts 2024-09-21T12:32:31.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:31 smithi183 ceph-mon[33277]: 5.1a scrub ok 2024-09-21T12:32:31.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:31 smithi183 ceph-mon[33277]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T12:32:31.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:31 smithi183 ceph-mon[33277]: osdmap e345: 8 total, 8 up, 7 in 2024-09-21T12:32:31.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:31 smithi183 ceph-mon[33277]: 4.13 scrub starts 2024-09-21T12:32:31.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:31 smithi183 ceph-mon[33277]: 4.13 scrub ok 2024-09-21T12:32:31.742 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:31.767 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:31.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:31.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:31.869 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:31.890 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:31.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:31.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:31.991 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:32.017 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:32.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:32.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:32.118 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:32.144 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:32.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:32.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:32.246 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:32.270 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:32.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:32.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:32.305 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-21T12:32:32.318 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T12:32:32.321 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:32.323 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:32.326 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:32.371 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:32.392 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:32.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:32.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:32.494 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:32.520 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:32.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:32.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:32.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:32 smithi114 ceph-mon[30324]: 4.9 scrub starts 2024-09-21T12:32:32.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:32 smithi114 ceph-mon[30324]: 4.9 scrub ok 2024-09-21T12:32:32.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:32 smithi114 ceph-mon[30324]: osdmap e346: 8 total, 8 up, 7 in 2024-09-21T12:32:32.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:32 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1338609870' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:32:32.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:32 smithi114 ceph-mon[30324]: 1.1b scrub starts 2024-09-21T12:32:32.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:32 smithi114 ceph-mon[30324]: 1.1b scrub ok 2024-09-21T12:32:32.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:32 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:32:32.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:32 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:32:32.550 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:32 smithi114 ceph-mon[40867]: 4.9 scrub starts 2024-09-21T12:32:32.551 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:32 smithi114 ceph-mon[40867]: 4.9 scrub ok 2024-09-21T12:32:32.551 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:32 smithi114 ceph-mon[40867]: osdmap e346: 8 total, 8 up, 7 in 2024-09-21T12:32:32.551 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:32 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1338609870' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:32:32.551 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:32 smithi114 ceph-mon[40867]: 1.1b scrub starts 2024-09-21T12:32:32.551 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:32 smithi114 ceph-mon[40867]: 1.1b scrub ok 2024-09-21T12:32:32.551 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:32 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:32:32.551 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:32 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:32:32.621 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:32.647 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:32.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:32.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:32.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:32 smithi183 ceph-mon[33277]: 4.9 scrub starts 2024-09-21T12:32:32.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:32 smithi183 ceph-mon[33277]: 4.9 scrub ok 2024-09-21T12:32:32.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:32 smithi183 ceph-mon[33277]: osdmap e346: 8 total, 8 up, 7 in 2024-09-21T12:32:32.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:32 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1338609870' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:32:32.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:32 smithi183 ceph-mon[33277]: 1.1b scrub starts 2024-09-21T12:32:32.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:32 smithi183 ceph-mon[33277]: 1.1b scrub ok 2024-09-21T12:32:32.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:32 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:32:32.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:32 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:32:32.749 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:32.774 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:32.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:32.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:32.828 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:32:32 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:32:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:32:32.876 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:32.897 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:32.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:32.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:32.999 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:33.025 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:33.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:33.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:33.126 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:33.152 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:33.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:33.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:33.254 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:33.279 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:32:33.279 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:32:33.280 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:32:33.280 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:32:33.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:33.280 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:33.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:33.293 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:32:33.293 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:32:33.312 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:32:33.312 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:32:33.329 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:32:33.329 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:32:33.346 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:32:33.346 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:32:33.346 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:33 smithi114 ceph-mon[30324]: 6.1e scrub starts 2024-09-21T12:32:33.347 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:33 smithi114 ceph-mon[30324]: 6.1e scrub ok 2024-09-21T12:32:33.347 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:33 smithi114 ceph-mon[30324]: pgmap v627: 178 pgs: 10 unknown, 168 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:33.347 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:33 smithi114 ceph-mon[30324]: 1.1d scrub starts 2024-09-21T12:32:33.347 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:33 smithi114 ceph-mon[30324]: 1.1d scrub ok 2024-09-21T12:32:33.347 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:33 smithi114 ceph-mon[30324]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:32:33.347 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:33 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1338609870' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:32:33.347 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:33 smithi114 ceph-mon[30324]: osdmap e347: 8 total, 8 up, 7 in 2024-09-21T12:32:33.347 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:33 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1772110089' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:32:33.347 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:33 smithi114 ceph-mon[40867]: 6.1e scrub starts 2024-09-21T12:32:33.347 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:33 smithi114 ceph-mon[40867]: 6.1e scrub ok 2024-09-21T12:32:33.347 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:33 smithi114 ceph-mon[40867]: pgmap v627: 178 pgs: 10 unknown, 168 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:33.347 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:33 smithi114 ceph-mon[40867]: 1.1d scrub starts 2024-09-21T12:32:33.348 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:33 smithi114 ceph-mon[40867]: 1.1d scrub ok 2024-09-21T12:32:33.348 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:33 smithi114 ceph-mon[40867]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:32:33.348 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:33 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1338609870' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:32:33.348 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:33 smithi114 ceph-mon[40867]: osdmap e347: 8 total, 8 up, 7 in 2024-09-21T12:32:33.348 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:33 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1772110089' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:32:33.363 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:32:33.363 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:32:33.379 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:32:33.379 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:32:33.382 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:33.405 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:33.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:33.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:33.507 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:33.529 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:33.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:33.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:33.630 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:33.654 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:33.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:33.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:33.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:33 smithi183 ceph-mon[33277]: 6.1e scrub starts 2024-09-21T12:32:33.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:33 smithi183 ceph-mon[33277]: 6.1e scrub ok 2024-09-21T12:32:33.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:33 smithi183 ceph-mon[33277]: pgmap v627: 178 pgs: 10 unknown, 168 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:33.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:33 smithi183 ceph-mon[33277]: 1.1d scrub starts 2024-09-21T12:32:33.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:33 smithi183 ceph-mon[33277]: 1.1d scrub ok 2024-09-21T12:32:33.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:33 smithi183 ceph-mon[33277]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:32:33.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:33 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1338609870' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:32:33.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:33 smithi183 ceph-mon[33277]: osdmap e347: 8 total, 8 up, 7 in 2024-09-21T12:32:33.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:33 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1772110089' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:32:33.756 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:33.778 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:33.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:33.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:33.879 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:33.903 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:33.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:33.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:34.004 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:34.026 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:34.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:34.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:34.127 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:34.150 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:34.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:34.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:34.251 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:34.274 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:34.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:34.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:34.316 INFO:teuthology.orchestra.run.smithi114.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-09-21T12:32:34.375 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:34.397 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:34.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:34.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:34.498 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:34.520 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:34.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:34.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:34.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:34 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1622384573' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:32:34.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:34 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:32:34.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:34 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1622384573' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:32:34.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:34 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:32:34.621 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:34.645 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:34.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:34.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:34.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:34 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1622384573' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:32:34.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:34 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:32:34.747 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:34.767 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:34.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:34.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:34.869 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:34.870 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:34.895 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:34.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:34.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:34.996 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:35.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:35.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:35.119 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:35.144 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:35.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:35.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:35.246 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:35.269 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:35.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:35.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:35.371 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:35.394 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:35.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:35.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:35.496 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:35.518 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:35.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:35.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:35.619 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:35.639 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:35 smithi183 ceph-mon[33277]: pgmap v629: 178 pgs: 178 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T12:32:35.639 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:35 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T12:32:35.640 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:35 smithi183 ceph-mon[33277]: osdmap e348: 8 total, 8 up, 7 in 2024-09-21T12:32:35.641 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:35.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:35.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:35.664 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:35 smithi114 ceph-mon[40867]: pgmap v629: 178 pgs: 178 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T12:32:35.664 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:35 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T12:32:35.664 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:35 smithi114 ceph-mon[40867]: osdmap e348: 8 total, 8 up, 7 in 2024-09-21T12:32:35.664 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:35 smithi114 ceph-mon[30324]: pgmap v629: 178 pgs: 178 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T12:32:35.664 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:35 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T12:32:35.664 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:35 smithi114 ceph-mon[30324]: osdmap e348: 8 total, 8 up, 7 in 2024-09-21T12:32:35.742 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:35.766 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:35.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:35.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:35.868 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:35.892 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:35.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:35.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:35.994 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:36.015 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:36.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:36.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:36.117 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:36.138 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:36.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:36.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:36.144 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:32:36.144 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:32:36.197 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T12:32:36.239 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:36.260 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:36.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:36.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:36.362 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:36.385 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:36.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:36.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:36.432 DEBUG:teuthology.orchestra.run.smithi114:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-09-21T12:32:36.434 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T12:32:36.434 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T12:32:36.487 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:36.513 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:36.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:36.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:36.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:36 smithi114 ceph-mon[40867]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T12:32:36.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:36 smithi114 ceph-mon[30324]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T12:32:36.615 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:36.636 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:36.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:36.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:36.645 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:hints = 1 2024-09-21T12:32:36.645 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T12:32:36.645 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Object prefix: benchmark_data_smithi114_161690 2024-09-21T12:32:36.646 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T12:32:36.646 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 0 0 0 0 0 0 - 0 2024-09-21T12:32:36.676 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:32:36.738 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:36.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:36 smithi183 ceph-mon[33277]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T12:32:36.765 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:36.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:36.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:36.867 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:36.913 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:36.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:36.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:37.017 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:37.027 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:36 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-a[30301]: 2024-09-21T12:32:36.841+0000 7f49fe685640 -1 mon.a@0(leader).osd e348 definitely_dead 0 2024-09-21T12:32:37.056 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:37.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:37.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:37.158 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:37.185 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:37.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:37.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:37.200 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T12:32:37.282 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:37.286 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:37.312 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:37.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:37.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:37.358 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:37.370 INFO:teuthology.orchestra.run.smithi114.stderr:marked in osd.4. 2024-09-21T12:32:37.394 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:37 smithi114 ceph-mon[40867]: pgmap v631: 178 pgs: 178 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 925 B/s rd, 0 op/s 2024-09-21T12:32:37.395 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:37 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2955323267' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T12:32:37.397 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:37 smithi114 ceph-mon[30324]: pgmap v631: 178 pgs: 178 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 925 B/s rd, 0 op/s 2024-09-21T12:32:37.397 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:37 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2955323267' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T12:32:37.414 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:37.436 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:37.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:37.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:37.538 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:37.559 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:37.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:37.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:37.646 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 1 16 2007 1991 124.424 124.438 0.00170751 0.00611138 2024-09-21T12:32:37.661 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:37.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:37.683 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:37.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:37.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:37 smithi183 ceph-mon[33277]: pgmap v631: 178 pgs: 178 active+clean; 583 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 925 B/s rd, 0 op/s 2024-09-21T12:32:37.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:37 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2955323267' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T12:32:37.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:32:37 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:32:37.784 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:37.806 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:37.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:37.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:37.907 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:37.928 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:37.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:37.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:38.015 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:32:38.016 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:32:38.024 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:32:38.025 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:32:38.030 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:38.041 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:32:38.041 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:32:38.054 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:38.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:38.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:38.063 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:32:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:32:38.079 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:32:38.080 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:32:38.096 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:32:38.096 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:32:38.113 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:32:38.113 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:32:38.130 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:32:38.130 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:32:38.156 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:38.178 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:38.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:38.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:38.279 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:38.300 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:38.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:38.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:38.402 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:38.435 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:38.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:38.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:38.536 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:38.557 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:38.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:38.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:38.646 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 2 16 2007 1991 62.2098 0 - 0.00611138 2024-09-21T12:32:38.659 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:38.680 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:38.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:38.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:38.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:38 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:38.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:38 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2955323267' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-21T12:32:38.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:38 smithi183 ceph-mon[33277]: osdmap e349: 8 total, 8 up, 8 in 2024-09-21T12:32:38.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:38 smithi183 ceph-mon[33277]: pgmap v633: 178 pgs: 14 peering, 164 active+clean; 45 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 7.4 MiB/s wr, 119 op/s 2024-09-21T12:32:38.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:38 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3282126243' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:32:38.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:38 smithi183 ceph-mon[33277]: osdmap e350: 8 total, 8 up, 8 in 2024-09-21T12:32:38.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:38 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:38.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:38 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2955323267' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-21T12:32:38.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:38 smithi114 ceph-mon[40867]: osdmap e349: 8 total, 8 up, 8 in 2024-09-21T12:32:38.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:38 smithi114 ceph-mon[40867]: pgmap v633: 178 pgs: 14 peering, 164 active+clean; 45 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 7.4 MiB/s wr, 119 op/s 2024-09-21T12:32:38.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:38 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3282126243' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:32:38.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:38 smithi114 ceph-mon[40867]: osdmap e350: 8 total, 8 up, 8 in 2024-09-21T12:32:38.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:38 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:38.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:38 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2955323267' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-21T12:32:38.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:38 smithi114 ceph-mon[30324]: osdmap e349: 8 total, 8 up, 8 in 2024-09-21T12:32:38.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:38 smithi114 ceph-mon[30324]: pgmap v633: 178 pgs: 14 peering, 164 active+clean; 45 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 7.4 MiB/s wr, 119 op/s 2024-09-21T12:32:38.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:38 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3282126243' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:32:38.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:38 smithi114 ceph-mon[30324]: osdmap e350: 8 total, 8 up, 8 in 2024-09-21T12:32:38.781 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:38.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:38.802 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:38.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:38.884 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-09-21T12:32:38.903 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:38.924 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:38.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:38.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:39.025 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:39.046 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:39.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:39.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:39.148 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:39.169 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:39.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:39.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:39.271 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:39.292 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:39.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:39.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:39.393 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:39.433 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:39.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:39.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:39.535 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:39.567 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:39.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:39.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:39.646 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 3 16 2775 2759 57.4713 24 0.00242527 0.0173726 2024-09-21T12:32:39.669 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:39.732 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:39.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:39.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:39.834 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:39.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:39.888 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:39.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:39.910 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:39.989 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:40.023 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:40.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:40.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:40.125 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:40.156 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:40.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:40.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:40.258 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:40.291 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:40.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:40.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:40.393 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:40.398 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-21T12:32:40.435 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:40.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:40.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:40.487 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:32:40.487 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:32:40.488 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:32:40.488 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:32:40.488 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:32:40.536 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:40.559 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:40.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:40.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:40.646 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 4 16 4806 4790 74.8344 126.938 0.00175047 0.0125835 2024-09-21T12:32:40.661 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:40.677 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:40 smithi114 ceph-mon[40867]: osdmap e351: 8 total, 8 up, 8 in 2024-09-21T12:32:40.677 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:40 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1564816655' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:32:40.677 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:40 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:32:40.677 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:40 smithi114 ceph-mon[30324]: osdmap e351: 8 total, 8 up, 8 in 2024-09-21T12:32:40.677 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:40 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1564816655' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:32:40.677 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:40 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:32:40.681 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:40.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:40.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:40.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:40 smithi183 ceph-mon[33277]: osdmap e351: 8 total, 8 up, 8 in 2024-09-21T12:32:40.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:40 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1564816655' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:32:40.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:40 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:32:40.783 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:40.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:40.805 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:40.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:40.906 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:40.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:40.927 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:40.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:41.028 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:41.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:41.052 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:41.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:41.153 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:41.175 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:41.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:41.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:41.276 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:41.299 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:41.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:41.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:41.300 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:32:41.400 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:41.430 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:41.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:41.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:41.444 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T12:32:41.532 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:41.552 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:41.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:41.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:41.646 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 5 16 4871 4855 60.68 4.0625 0.00127279 0.0157125 2024-09-21T12:32:41.654 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:41.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:41.676 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:41.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:41.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:41 smithi183 ceph-mon[33277]: pgmap v636: 178 pgs: 21 peering, 157 active+clean; 70 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 929 B/s rd, 13 MiB/s wr, 202 op/s 2024-09-21T12:32:41.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:41 smithi183 ceph-mon[33277]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:32:41.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:41 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:32:41.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:41 smithi183 ceph-mon[33277]: osdmap e352: 8 total, 8 up, 8 in 2024-09-21T12:32:41.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:41 smithi114 ceph-mon[40867]: pgmap v636: 178 pgs: 21 peering, 157 active+clean; 70 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 929 B/s rd, 13 MiB/s wr, 202 op/s 2024-09-21T12:32:41.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:41 smithi114 ceph-mon[40867]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:32:41.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:41 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:32:41.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:41 smithi114 ceph-mon[40867]: osdmap e352: 8 total, 8 up, 8 in 2024-09-21T12:32:41.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:41 smithi114 ceph-mon[30324]: pgmap v636: 178 pgs: 21 peering, 157 active+clean; 70 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 929 B/s rd, 13 MiB/s wr, 202 op/s 2024-09-21T12:32:41.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:41 smithi114 ceph-mon[30324]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:32:41.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:41 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:32:41.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:41 smithi114 ceph-mon[30324]: osdmap e352: 8 total, 8 up, 8 in 2024-09-21T12:32:41.777 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:41.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:41.798 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:41.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:41.899 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:41.907 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:41.922 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:41.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:41.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:41.930 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:42.024 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:42.059 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:42.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:42.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:42.161 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:42.204 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:42.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:42.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:42.306 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:42.339 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:42.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:42.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:42.441 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:42.479 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:42.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:42.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:42.581 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:42.614 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:42.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:42.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:42.647 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 6 16 6908 6892 71.7831 127.312 0.00594257 0.0139212 2024-09-21T12:32:42.682 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:32:42.682 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:32:42.683 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:32:42 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:32:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:32:42.683 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:42 smithi114 ceph-mon[30324]: osdmap e353: 8 total, 8 up, 8 in 2024-09-21T12:32:42.683 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:42 smithi114 ceph-mon[30324]: pgmap v639: 178 pgs: 1 creating+peering, 21 peering, 156 active+clean; 200 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 39 MiB/s wr, 619 op/s; 9.8 MiB/s, 170 objects/s recovering 2024-09-21T12:32:42.683 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:42 smithi114 ceph-mon[30324]: osdmap e354: 8 total, 8 up, 8 in 2024-09-21T12:32:42.683 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:42 smithi114 ceph-mon[40867]: osdmap e353: 8 total, 8 up, 8 in 2024-09-21T12:32:42.683 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:42 smithi114 ceph-mon[40867]: pgmap v639: 178 pgs: 1 creating+peering, 21 peering, 156 active+clean; 200 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 39 MiB/s wr, 619 op/s; 9.8 MiB/s, 170 objects/s recovering 2024-09-21T12:32:42.684 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:42 smithi114 ceph-mon[40867]: osdmap e354: 8 total, 8 up, 8 in 2024-09-21T12:32:42.716 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:42.719 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:32:42.719 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:32:42.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:42 smithi183 ceph-mon[33277]: osdmap e353: 8 total, 8 up, 8 in 2024-09-21T12:32:42.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:42 smithi183 ceph-mon[33277]: pgmap v639: 178 pgs: 1 creating+peering, 21 peering, 156 active+clean; 200 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 39 MiB/s wr, 619 op/s; 9.8 MiB/s, 170 objects/s recovering 2024-09-21T12:32:42.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:42 smithi183 ceph-mon[33277]: osdmap e354: 8 total, 8 up, 8 in 2024-09-21T12:32:42.753 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:42.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:42.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:42.756 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:32:42.756 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:32:42.791 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:32:42.792 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:32:42.829 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:32:42.829 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:32:42.855 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:42.863 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:32:42.863 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:32:42.885 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:42.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:42.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:42.895 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:32:42.896 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:32:42.925 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:32:42.926 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:32:42.987 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:43.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:43.030 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:43.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:43.131 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:43.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:43.166 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:43.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:43.267 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:43.304 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:43.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:43.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:43.406 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:43.442 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:43.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:43.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:43.544 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:43.577 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:43.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:43.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:43.647 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 7 15 9954 9939 88.7308 190.438 0.00505564 0.0112615 2024-09-21T12:32:43.678 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:43.712 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:43.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:43.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:43.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:43 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1990795331' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:32:43.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:43 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1990795331' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:32:43.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:43 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1990795331' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:32:43.814 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:43.849 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:43.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:43.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:43.868 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T12:32:43.884 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T12:32:43.885 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:32:43.885 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-21T12:32:43.885 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T12:32:43.885 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T12:32:43.885 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-21T12:32:43.903 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:32:43.903 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:32:43.950 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:43.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:43.990 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:43.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:44.091 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:44.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:44.129 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:44.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:44.230 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:44.267 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:44.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:44.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:44.280 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:32:44.369 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:44.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:44.399 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:44.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:44.500 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:44.540 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:44.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:44.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:44.642 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:44.647 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 8 15 12632 12617 98.5589 167.375 0.00593671 0.0101396 2024-09-21T12:32:44.672 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:44.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:44.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:44.695 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:44 smithi183 ceph-mon[33277]: pgmap v641: 178 pgs: 1 creating+peering, 177 active+clean; 359 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 65 MiB/s wr, 1.04k op/s; 14 MiB/s, 248 objects/s recovering 2024-09-21T12:32:44.774 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:44.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:44 smithi114 ceph-mon[40867]: pgmap v641: 178 pgs: 1 creating+peering, 177 active+clean; 359 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 65 MiB/s wr, 1.04k op/s; 14 MiB/s, 248 objects/s recovering 2024-09-21T12:32:44.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:44 smithi114 ceph-mon[30324]: pgmap v641: 178 pgs: 1 creating+peering, 177 active+clean; 359 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 65 MiB/s wr, 1.04k op/s; 14 MiB/s, 248 objects/s recovering 2024-09-21T12:32:44.810 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:44.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:44.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:44.819 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:32:44.819 INFO:teuthology.orchestra.run.smithi183.stdout: "success": "" 2024-09-21T12:32:44.819 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:32:44.912 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:44.947 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:44.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:44.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:45.049 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:45.070 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:45.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:45.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:45.171 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:45.208 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:45.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:45.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:45.310 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:45.337 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:45.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:45.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:45.438 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:45.469 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:45.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:45.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:45.571 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:45.600 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:45.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:45.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:45.647 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 9 16 14868 14852 103.127 139.688 0.00419582 0.00969119 2024-09-21T12:32:45.701 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:45.735 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:45.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:45.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:45.837 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:45.858 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:45.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:45.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:45.960 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:45.995 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:45.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:45.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:46.097 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:46.118 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:46.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:46.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:46.220 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:46.254 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:46.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:46.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:46.356 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:46.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:46.392 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:46.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:46.493 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:46.526 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:46.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:46.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:46.628 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:46.647 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 10 15 17149 17134 107.076 142.625 0.00703147 0.00933454 2024-09-21T12:32:46.661 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:46.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:46.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:46.762 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:46.796 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:46.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:46.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:46.898 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:46.934 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:46.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:46.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:47.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:46 smithi114 ceph-mon[40867]: pgmap v642: 178 pgs: 1 creating+peering, 177 active+clean; 359 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 48 MiB/s wr, 770 op/s; 11 MiB/s, 184 objects/s recovering 2024-09-21T12:32:47.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:46 smithi114 ceph-mon[30324]: pgmap v642: 178 pgs: 1 creating+peering, 177 active+clean; 359 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 48 MiB/s wr, 770 op/s; 11 MiB/s, 184 objects/s recovering 2024-09-21T12:32:47.036 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:47.072 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:47.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:47.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:47.174 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:47.207 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:47.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:47.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:47.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:46 smithi183 ceph-mon[33277]: pgmap v642: 178 pgs: 1 creating+peering, 177 active+clean; 359 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 48 MiB/s wr, 770 op/s; 11 MiB/s, 184 objects/s recovering 2024-09-21T12:32:47.309 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:47.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:47.352 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:47.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:47.361 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T12:32:47.453 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:47.459 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:47.503 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:47.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:47.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:47.605 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:47.642 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:47.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:47.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:47.647 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 11 16 19806 19790 112.431 166 0.00464802 0.00888854 2024-09-21T12:32:47.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:32:47 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:32:47.744 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:47.765 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:32:47.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:47.784 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:47.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:47.885 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:47.918 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:47.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:47.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:48.020 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:48.057 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:48.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:48.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:48.159 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:48.169 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:47 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:32:48.169 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:47 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:32:48.169 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:47 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:48.180 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:48.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:48.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:48.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:47 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:32:48.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:47 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:32:48.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:47 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:48.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:47 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:32:48.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:47 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:32:48.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:47 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:48.281 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:48.317 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:48.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:48.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:48.419 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:48.453 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:48.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:48.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:48.554 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:48.587 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:48.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:48.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:48.605 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:32:48.605 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:32:48.647 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 12 16 22122 22106 115.123 144.75 0.00597458 0.00867878 2024-09-21T12:32:48.689 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:48.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:48.713 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:48.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:48.805 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:32:48.814 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:48.846 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:48.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:48.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:48.948 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:48.979 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:48.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:48.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:49.081 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:49.105 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:49.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:49.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:49.207 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:49.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:48 smithi183 ceph-mon[33277]: pgmap v643: 178 pgs: 178 active+clean; 1006 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 126 MiB/s wr, 2.01k op/s; 8.6 MiB/s, 148 objects/s recovering 2024-09-21T12:32:49.245 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:49.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:49.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:49.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:48 smithi114 ceph-mon[40867]: pgmap v643: 178 pgs: 178 active+clean; 1006 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 126 MiB/s wr, 2.01k op/s; 8.6 MiB/s, 148 objects/s recovering 2024-09-21T12:32:49.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:48 smithi114 ceph-mon[30324]: pgmap v643: 178 pgs: 178 active+clean; 1006 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 126 MiB/s wr, 2.01k op/s; 8.6 MiB/s, 148 objects/s recovering 2024-09-21T12:32:49.347 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:49.372 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:49.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:49.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:49.474 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:49.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:49.510 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:49.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:49.611 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:49.647 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:49.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:49.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:49.648 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 13 16 24013 23997 115.35 118.188 0.00488575 0.00866297 2024-09-21T12:32:49.749 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:49.788 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:49.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:49.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:49.805 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T12:32:49.863 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:49.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:49.931 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:49.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:49.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:49.990 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-21T12:32:50.033 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:50.054 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:50.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:50.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:50.156 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:50.162 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:32:50.176 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:50.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:50.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:50.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:49 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2763008295' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:32:50.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:49 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2763008295' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:32:50.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:49 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2763008295' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:32:50.278 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:50.300 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:50.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:50.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:50.401 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:50.423 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:50.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:50.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:50.525 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:50.559 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:50.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:50.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:50.648 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 14 16 25419 25403 113.388 87.875 0.00893641 0.00881438 2024-09-21T12:32:50.661 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:50.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:50.690 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:50.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:50.711 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:32:50.711 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:32:50.790 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:50.823 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:50.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:50.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:50.924 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:50.947 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:50.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:50.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:51.029 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T12:32:51.030 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:32:51.030 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-21T12:32:51.030 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.0 'trim stale osdmaps' 2024-09-21T12:32:51.049 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:51.077 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:51.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:51.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:51.179 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:51.225 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:51.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:51.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:51.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:50 smithi183 ceph-mon[33277]: pgmap v644: 178 pgs: 178 active+clean; 1.1 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 129 MiB/s wr, 2.06k op/s; 7.6 MiB/s, 131 objects/s recovering 2024-09-21T12:32:51.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:50 smithi183 ceph-mon[33277]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:32:51.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:50 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2763008295' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:32:51.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:50 smithi183 ceph-mon[33277]: osdmap e355: 8 total, 8 up, 8 in 2024-09-21T12:32:51.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:50 smithi183 ceph-mon[33277]: 6.15 scrub starts 2024-09-21T12:32:51.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:50 smithi183 ceph-mon[33277]: 6.15 scrub ok 2024-09-21T12:32:51.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:50 smithi114 ceph-mon[40867]: pgmap v644: 178 pgs: 178 active+clean; 1.1 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 129 MiB/s wr, 2.06k op/s; 7.6 MiB/s, 131 objects/s recovering 2024-09-21T12:32:51.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:51 smithi114 ceph-mon[40867]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:32:51.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:51 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2763008295' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:32:51.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:51 smithi114 ceph-mon[40867]: osdmap e355: 8 total, 8 up, 8 in 2024-09-21T12:32:51.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:51 smithi114 ceph-mon[40867]: 6.15 scrub starts 2024-09-21T12:32:51.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:51 smithi114 ceph-mon[40867]: 6.15 scrub ok 2024-09-21T12:32:51.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:50 smithi114 ceph-mon[30324]: pgmap v644: 178 pgs: 178 active+clean; 1.1 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 129 MiB/s wr, 2.06k op/s; 7.6 MiB/s, 131 objects/s recovering 2024-09-21T12:32:51.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:51 smithi114 ceph-mon[30324]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:32:51.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:51 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2763008295' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:32:51.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:51 smithi114 ceph-mon[30324]: osdmap e355: 8 total, 8 up, 8 in 2024-09-21T12:32:51.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:51 smithi114 ceph-mon[30324]: 6.15 scrub starts 2024-09-21T12:32:51.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:51 smithi114 ceph-mon[30324]: 6.15 scrub ok 2024-09-21T12:32:51.327 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:51.351 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:51.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:51.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:51.453 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:51.488 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:51.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:51.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:51.590 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:51.622 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:51.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:51.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:51.648 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 15 16 27189 27173 113.203 110.625 0.00248858 0.00880916 2024-09-21T12:32:51.724 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:51.761 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:51.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:51.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:51.862 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:51.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:51.889 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:51.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:51.990 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:52.023 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:52.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:52.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:52.125 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:52.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:52.167 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:52.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:52.268 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:52.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:51 smithi114 ceph-mon[40867]: 4.0 scrub starts 2024-09-21T12:32:52.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:51 smithi114 ceph-mon[40867]: 4.0 scrub ok 2024-09-21T12:32:52.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:51 smithi114 ceph-mon[40867]: 3.15 scrub starts 2024-09-21T12:32:52.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:51 smithi114 ceph-mon[40867]: 3.15 scrub ok 2024-09-21T12:32:52.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:51 smithi114 ceph-mon[30324]: 4.0 scrub starts 2024-09-21T12:32:52.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:51 smithi114 ceph-mon[30324]: 4.0 scrub ok 2024-09-21T12:32:52.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:51 smithi114 ceph-mon[30324]: 3.15 scrub starts 2024-09-21T12:32:52.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:51 smithi114 ceph-mon[30324]: 3.15 scrub ok 2024-09-21T12:32:52.300 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:52.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:52.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:52.402 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:52.425 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:52.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:52.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:52.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:51 smithi183 ceph-mon[33277]: 4.0 scrub starts 2024-09-21T12:32:52.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:51 smithi183 ceph-mon[33277]: 4.0 scrub ok 2024-09-21T12:32:52.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:51 smithi183 ceph-mon[33277]: 3.15 scrub starts 2024-09-21T12:32:52.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:51 smithi183 ceph-mon[33277]: 3.15 scrub ok 2024-09-21T12:32:52.527 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:52.560 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:52.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:52.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:52.648 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 16 16 28971 28955 113.088 111.375 0.00428762 0.00883746 2024-09-21T12:32:52.662 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:52.696 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:52.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:52.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:52.798 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:52.834 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:52.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:52.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:52.935 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:52.969 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:52.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:52.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:53.024 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:32:52 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:32:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:32:53.044 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T12:32:53.071 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:53.108 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:53.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:53.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:53.210 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:53.231 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:53.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:53.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:53.298 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:32:53.333 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:53.354 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:53.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:53.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:53.419 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:53 smithi183 ceph-mon[33277]: pgmap v646: 178 pgs: 178 active+clean; 1.4 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 126 MiB/s wr, 2.02k op/s; 2.6 MiB/s, 43 objects/s recovering 2024-09-21T12:32:53.419 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:53 smithi183 ceph-mon[33277]: osdmap e356: 8 total, 8 up, 8 in 2024-09-21T12:32:53.457 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:53.481 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:53.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:53.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:53.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:53 smithi114 ceph-mon[40867]: pgmap v646: 178 pgs: 178 active+clean; 1.4 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 126 MiB/s wr, 2.02k op/s; 2.6 MiB/s, 43 objects/s recovering 2024-09-21T12:32:53.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:53 smithi114 ceph-mon[40867]: osdmap e356: 8 total, 8 up, 8 in 2024-09-21T12:32:53.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:53 smithi114 ceph-mon[30324]: pgmap v646: 178 pgs: 178 active+clean; 1.4 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 126 MiB/s wr, 2.02k op/s; 2.6 MiB/s, 43 objects/s recovering 2024-09-21T12:32:53.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:53 smithi114 ceph-mon[30324]: osdmap e356: 8 total, 8 up, 8 in 2024-09-21T12:32:53.582 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:53.617 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:53.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:53.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:53.649 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 17 16 30111 30095 110.625 71.25 0.00703077 0.00903464 2024-09-21T12:32:53.719 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:53.763 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:53.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:53.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:53.865 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:53.889 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:32:53.889 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:32:53.901 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:53.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:53.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:54.003 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:54.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:54.050 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:54.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:54.151 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:54.171 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:32:54.171 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:32:54.180 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:54.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:54.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:54.213 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:32:54.213 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:32:54.252 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:54 smithi114 ceph-mon[30324]: 1.6 scrub starts 2024-09-21T12:32:54.252 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:54 smithi114 ceph-mon[30324]: 1.6 scrub ok 2024-09-21T12:32:54.252 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:54 smithi114 ceph-mon[30324]: 5.1c scrub starts 2024-09-21T12:32:54.252 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:54 smithi114 ceph-mon[30324]: 5.1c scrub ok 2024-09-21T12:32:54.252 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:54 smithi114 ceph-mon[30324]: 6.1d scrub starts 2024-09-21T12:32:54.252 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:54 smithi114 ceph-mon[30324]: 6.1d scrub ok 2024-09-21T12:32:54.252 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:54 smithi114 ceph-mon[40867]: 1.6 scrub starts 2024-09-21T12:32:54.252 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:54 smithi114 ceph-mon[40867]: 1.6 scrub ok 2024-09-21T12:32:54.252 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:54 smithi114 ceph-mon[40867]: 5.1c scrub starts 2024-09-21T12:32:54.253 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:54 smithi114 ceph-mon[40867]: 5.1c scrub ok 2024-09-21T12:32:54.253 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:54 smithi114 ceph-mon[40867]: 6.1d scrub starts 2024-09-21T12:32:54.253 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:54 smithi114 ceph-mon[40867]: 6.1d scrub ok 2024-09-21T12:32:54.253 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:32:54.253 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:32:54.282 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:54.283 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:32:54.283 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:32:54.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:54 smithi183 ceph-mon[33277]: 1.6 scrub starts 2024-09-21T12:32:54.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:54 smithi183 ceph-mon[33277]: 1.6 scrub ok 2024-09-21T12:32:54.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:54 smithi183 ceph-mon[33277]: 5.1c scrub starts 2024-09-21T12:32:54.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:54 smithi183 ceph-mon[33277]: 5.1c scrub ok 2024-09-21T12:32:54.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:54 smithi183 ceph-mon[33277]: 6.1d scrub starts 2024-09-21T12:32:54.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:54 smithi183 ceph-mon[33277]: 6.1d scrub ok 2024-09-21T12:32:54.313 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:54.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:54.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:54.315 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:32:54.315 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:32:54.351 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:32:54.351 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:32:54.384 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:32:54.385 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:32:54.415 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:54.420 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:32:54.421 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:32:54.468 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:54.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:54.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:54.570 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:54.604 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:54.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:54.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:54.646 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T12:32:54.649 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 18 16 32167 32151 111.617 128.5 0.00946849 0.00895426 2024-09-21T12:32:54.706 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:54.767 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:54.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:54.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:54.868 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:54.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:54.905 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:54.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:55.006 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:55.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:55.048 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:55.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:55.095 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:32:55.149 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:55.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:55.185 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:55.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:55.286 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:55.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:55.320 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:55.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:55.388 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:55 smithi183 ceph-mon[33277]: 6.1f scrub starts 2024-09-21T12:32:55.388 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:55 smithi183 ceph-mon[33277]: 6.1f scrub ok 2024-09-21T12:32:55.388 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:55 smithi183 ceph-mon[33277]: pgmap v648: 178 pgs: 178 active+clean; 1.7 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 174 MiB/s wr, 2.79k op/s 2024-09-21T12:32:55.388 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:55 smithi183 ceph-mon[33277]: 3.1 scrub starts 2024-09-21T12:32:55.388 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:55 smithi183 ceph-mon[33277]: 3.1 scrub ok 2024-09-21T12:32:55.388 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:55 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/4098867107' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:32:55.388 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:55 smithi183 ceph-mon[33277]: 3.e scrub starts 2024-09-21T12:32:55.388 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:55 smithi183 ceph-mon[33277]: 3.e scrub ok 2024-09-21T12:32:55.388 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:55 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:32:55.388 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:55 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:32:55.421 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:55.457 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:55.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:55.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:55.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:55 smithi114 ceph-mon[30324]: 6.1f scrub starts 2024-09-21T12:32:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:55 smithi114 ceph-mon[30324]: 6.1f scrub ok 2024-09-21T12:32:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:55 smithi114 ceph-mon[30324]: pgmap v648: 178 pgs: 178 active+clean; 1.7 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 174 MiB/s wr, 2.79k op/s 2024-09-21T12:32:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:55 smithi114 ceph-mon[30324]: 3.1 scrub starts 2024-09-21T12:32:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:55 smithi114 ceph-mon[30324]: 3.1 scrub ok 2024-09-21T12:32:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:55 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4098867107' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:32:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:55 smithi114 ceph-mon[30324]: 3.e scrub starts 2024-09-21T12:32:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:55 smithi114 ceph-mon[30324]: 3.e scrub ok 2024-09-21T12:32:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:55 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:32:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:55 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:32:55.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:55 smithi114 ceph-mon[40867]: 6.1f scrub starts 2024-09-21T12:32:55.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:55 smithi114 ceph-mon[40867]: 6.1f scrub ok 2024-09-21T12:32:55.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:55 smithi114 ceph-mon[40867]: pgmap v648: 178 pgs: 178 active+clean; 1.7 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 174 MiB/s wr, 2.79k op/s 2024-09-21T12:32:55.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:55 smithi114 ceph-mon[40867]: 3.1 scrub starts 2024-09-21T12:32:55.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:55 smithi114 ceph-mon[40867]: 3.1 scrub ok 2024-09-21T12:32:55.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:55 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/4098867107' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:32:55.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:55 smithi114 ceph-mon[40867]: 3.e scrub starts 2024-09-21T12:32:55.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:55 smithi114 ceph-mon[40867]: 3.e scrub ok 2024-09-21T12:32:55.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:55 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:32:55.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:55 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:32:55.557 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:55.558 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:55.589 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:55.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:55.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:55.649 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 19 16 34197 34181 112.42 126.875 0.0104305 0.00888992 2024-09-21T12:32:55.690 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:55.699 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:32:55.700 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-21T12:32:55.700 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 0 2024-09-21T12:32:55.700 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:32:55.722 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:55.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:55.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:55.824 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:55.870 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:55.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:55.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:55.972 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:56.005 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:56.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:56.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:56.107 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:56.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:56.148 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:56.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:56.149 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:56 smithi114 ceph-mon[30324]: 6.c scrub starts 2024-09-21T12:32:56.149 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:56 smithi114 ceph-mon[30324]: 6.c scrub ok 2024-09-21T12:32:56.149 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:56 smithi114 ceph-mon[30324]: osdmap e357: 8 total, 8 up, 8 in 2024-09-21T12:32:56.149 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:56 smithi114 ceph-mon[40867]: 6.c scrub starts 2024-09-21T12:32:56.149 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:56 smithi114 ceph-mon[40867]: 6.c scrub ok 2024-09-21T12:32:56.150 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:56 smithi114 ceph-mon[40867]: osdmap e357: 8 total, 8 up, 8 in 2024-09-21T12:32:56.249 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:56.250 INFO:teuthology.orchestra.run.smithi114.stderr:no valid command found; 10 closest matches: 2024-09-21T12:32:56.250 INFO:teuthology.orchestra.run.smithi114.stderr:0 2024-09-21T12:32:56.251 INFO:teuthology.orchestra.run.smithi114.stderr:1 2024-09-21T12:32:56.251 INFO:teuthology.orchestra.run.smithi114.stderr:2 2024-09-21T12:32:56.251 INFO:teuthology.orchestra.run.smithi114.stderr:abort 2024-09-21T12:32:56.251 INFO:teuthology.orchestra.run.smithi114.stderr:assert 2024-09-21T12:32:56.251 INFO:teuthology.orchestra.run.smithi114.stderr:bench [] [] [] [] 2024-09-21T12:32:56.251 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs debug_inject_read_zeros 2024-09-21T12:32:56.251 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs files list 2024-09-21T12:32:56.251 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs stats 2024-09-21T12:32:56.251 INFO:teuthology.orchestra.run.smithi114.stderr:bluestore allocator dump block 2024-09-21T12:32:56.251 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: invalid command 2024-09-21T12:32:56.281 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:56.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:56.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:56.354 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:56 smithi183 ceph-mon[33277]: 6.c scrub starts 2024-09-21T12:32:56.354 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:56 smithi183 ceph-mon[33277]: 6.c scrub ok 2024-09-21T12:32:56.354 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:56 smithi183 ceph-mon[33277]: osdmap e357: 8 total, 8 up, 8 in 2024-09-21T12:32:56.383 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:56.407 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T12:32:56.420 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:56.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:56.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:56.522 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:56.554 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:56.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:56.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:56.649 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-21T12:32:56.648801+0000 min lat: 0.00124586 max lat: 2.02977 avg lat: 0.00885414 2024-09-21T12:32:56.649 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T12:32:56.649 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 20 16 36140 36124 112.87 121.438 0.00594935 0.00885414 2024-09-21T12:32:56.655 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:56.692 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:56.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:56.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:56.793 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:56.808 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T12:32:56.817 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:32:56.829 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:56.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:56.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:56.930 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:56.971 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T12:32:56.971 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T12:32:56.972 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.1 'trim stale osdmaps' 2024-09-21T12:32:56.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:56.999 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:56.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:57.099 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:57.158 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:57.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:57.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:57.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:57 smithi183 ceph-mon[33277]: pgmap v650: 178 pgs: 178 active+clean; 1.7 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-21T12:32:57.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:57 smithi183 ceph-mon[33277]: 3.1b scrub starts 2024-09-21T12:32:57.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:57 smithi183 ceph-mon[33277]: 3.1b scrub ok 2024-09-21T12:32:57.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:57 smithi183 ceph-mon[33277]: osdmap e358: 8 total, 8 up, 8 in 2024-09-21T12:32:57.260 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:57.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:57 smithi114 ceph-mon[30324]: pgmap v650: 178 pgs: 178 active+clean; 1.7 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-21T12:32:57.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:57 smithi114 ceph-mon[30324]: 3.1b scrub starts 2024-09-21T12:32:57.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:57 smithi114 ceph-mon[30324]: 3.1b scrub ok 2024-09-21T12:32:57.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:57 smithi114 ceph-mon[30324]: osdmap e358: 8 total, 8 up, 8 in 2024-09-21T12:32:57.280 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:57 smithi114 ceph-mon[40867]: pgmap v650: 178 pgs: 178 active+clean; 1.7 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-21T12:32:57.280 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:57 smithi114 ceph-mon[40867]: 3.1b scrub starts 2024-09-21T12:32:57.280 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:57 smithi114 ceph-mon[40867]: 3.1b scrub ok 2024-09-21T12:32:57.280 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:57 smithi114 ceph-mon[40867]: osdmap e358: 8 total, 8 up, 8 in 2024-09-21T12:32:57.294 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:57.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:57.311 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:57.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:57.412 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:57.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:57.453 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:57.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:57.553 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:57.586 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:57.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:57.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:57.612 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:32:57 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:32:57.614 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:32:57.649 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 21 16 37886 37870 112.691 109.125 0.00842201 0.00886763 2024-09-21T12:32:57.688 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:57.723 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:57.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:57.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:57.824 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:57.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:57.879 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:57.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:32:57.904 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:32:57.904 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-21T12:32:57.904 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-21T12:32:57.905 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-21T12:32:57.905 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:32:57.965 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:57.980 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:58.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:58.030 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:58.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:58.131 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:32:58.185 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:58.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:58.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:32:58.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:58 smithi183 ceph-mon[33277]: 4.1e scrub starts 2024-09-21T12:32:58.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:58 smithi183 ceph-mon[33277]: 4.1e scrub ok 2024-09-21T12:32:58.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:58 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:58.287 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:58.318 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:58.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:58.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:58.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:58 smithi114 ceph-mon[30324]: 4.1e scrub starts 2024-09-21T12:32:58.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:58 smithi114 ceph-mon[30324]: 4.1e scrub ok 2024-09-21T12:32:58.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:58 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:58.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:58 smithi114 ceph-mon[40867]: 4.1e scrub starts 2024-09-21T12:32:58.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:58 smithi114 ceph-mon[40867]: 4.1e scrub ok 2024-09-21T12:32:58.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:58 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:58.419 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:58.448 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:58.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:58.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:58.550 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:32:58.586 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:58.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:58.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:32:58.649 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 22 15 39602 39587 112.446 107.312 0.011784 0.00888809 2024-09-21T12:32:58.688 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:58.708 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:58.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:58.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:58.810 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:58.850 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:58.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:58.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:58.951 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:58.982 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:58.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:58.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:59.084 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:32:59.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:59.122 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:59.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:32:59.133 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-21T12:32:59.200 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T12:32:59.223 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:59.265 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:59.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:59.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:59.367 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:32:59.376 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:59 smithi114 ceph-mon[40867]: pgmap v652: 178 pgs: 178 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 130 MiB/s wr, 2.08k op/s 2024-09-21T12:32:59.376 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:59 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/814921270' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:32:59.376 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:59 smithi114 ceph-mon[40867]: 2.0 scrub starts 2024-09-21T12:32:59.376 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:59 smithi114 ceph-mon[40867]: 2.0 scrub ok 2024-09-21T12:32:59.376 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:59 smithi114 ceph-mon[40867]: 1.3 scrub starts 2024-09-21T12:32:59.376 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:32:59 smithi114 ceph-mon[40867]: 1.3 scrub ok 2024-09-21T12:32:59.377 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:59 smithi114 ceph-mon[30324]: pgmap v652: 178 pgs: 178 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 130 MiB/s wr, 2.08k op/s 2024-09-21T12:32:59.377 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:59 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/814921270' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:32:59.377 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:59 smithi114 ceph-mon[30324]: 2.0 scrub starts 2024-09-21T12:32:59.377 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:59 smithi114 ceph-mon[30324]: 2.0 scrub ok 2024-09-21T12:32:59.377 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:59 smithi114 ceph-mon[30324]: 1.3 scrub starts 2024-09-21T12:32:59.377 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:32:59 smithi114 ceph-mon[30324]: 1.3 scrub ok 2024-09-21T12:32:59.409 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:59.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:59.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:32:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:59 smithi183 ceph-mon[33277]: pgmap v652: 178 pgs: 178 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 130 MiB/s wr, 2.08k op/s 2024-09-21T12:32:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:59 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/814921270' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:32:59.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:59 smithi183 ceph-mon[33277]: 2.0 scrub starts 2024-09-21T12:32:59.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:59 smithi183 ceph-mon[33277]: 2.0 scrub ok 2024-09-21T12:32:59.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:59 smithi183 ceph-mon[33277]: 1.3 scrub starts 2024-09-21T12:32:59.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:32:59 smithi183 ceph-mon[33277]: 1.3 scrub ok 2024-09-21T12:32:59.511 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:32:59.557 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:59.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:59.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:32:59.593 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:32:59.649 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 23 16 41242 41226 112.011 102.438 0.00606956 0.00892211 2024-09-21T12:32:59.659 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:32:59.694 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:32:59.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:59.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:32:59.796 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:32:59.834 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:59.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:59.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:32:59.936 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:32:59.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:32:59.972 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:32:59.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:00.073 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:00.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:00.108 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:00.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:00.209 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:00.238 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:33:00.238 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-21T12:33:00.238 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-21T12:33:00.238 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:33:00.238 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.238 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.24695.0:116819 7.0 7:07411ed9:::benchmark_data_smithi114_78432_object12673:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T12:33:00.239 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:23:12.370997+0000", 2024-09-21T12:33:00.239 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 587.860308186, 2024-09-21T12:33:00.239 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15405449900000001, 2024-09-21T12:33:00.239 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:00.239 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:00.239 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:00.239 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:00.239 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.24695", 2024-09-21T12:33:00.239 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3620795002", 2024-09-21T12:33:00.239 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 116819 2024-09-21T12:33:00.239 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.239 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:00.239 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.240 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:00.240 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.370997+0000", 2024-09-21T12:33:00.240 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.240 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.240 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.240 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:00.240 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.370997+0000", 2024-09-21T12:33:00.240 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.240 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.240 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.240 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:00.240 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.370998+0000", 2024-09-21T12:33:00.240 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.381e-06 2024-09-21T12:33:00.240 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.241 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.241 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:00.241 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.371002+0000", 2024-09-21T12:33:00.241 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3710000000000002e-06 2024-09-21T12:33:00.241 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.241 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.241 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:00.241 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.371014+0000", 2024-09-21T12:33:00.241 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1865000000000001e-05 2024-09-21T12:33:00.241 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.241 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.241 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:00.241 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.371022+0000", 2024-09-21T12:33:00.241 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4610000000000002e-06 2024-09-21T12:33:00.241 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.242 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.242 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:00.242 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.371071+0000", 2024-09-21T12:33:00.242 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9082000000000002e-05 2024-09-21T12:33:00.242 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.242 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.242 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:00.242 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.371188+0000", 2024-09-21T12:33:00.242 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000117015 2024-09-21T12:33:00.242 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.242 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.242 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:33:00.242 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.371309+0000", 2024-09-21T12:33:00.243 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012115700000000001 2024-09-21T12:33:00.243 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.243 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.243 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:00.243 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.373478+0000", 2024-09-21T12:33:00.243 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021686840000000001 2024-09-21T12:33:00.243 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.243 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.243 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.243 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.377634+0000", 2024-09-21T12:33:00.243 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0041562589999999998 2024-09-21T12:33:00.243 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.243 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.243 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.244 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.524881+0000", 2024-09-21T12:33:00.244 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14724746899999999 2024-09-21T12:33:00.244 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.244 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.244 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:00.244 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.525006+0000", 2024-09-21T12:33:00.244 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012413300000000001 2024-09-21T12:33:00.244 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.244 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.244 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:00.244 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:23:12.525051+0000", 2024-09-21T12:33:00.244 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5621999999999998e-05 2024-09-21T12:33:00.244 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.244 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:00.244 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.245 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.245 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.245 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:38143 8.8 8:133e9789:::benchmark_data_smithi114_104936_object38142:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T12:33:00.245 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:11.302085+0000", 2024-09-21T12:33:00.245 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 468.92921994400001, 2024-09-21T12:33:00.245 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15153207799999999, 2024-09-21T12:33:00.245 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:00.245 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:00.245 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:00.245 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:00.245 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:33:00.245 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:33:00.245 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 38143 2024-09-21T12:33:00.245 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.246 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:00.246 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.246 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:00.246 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302085+0000", 2024-09-21T12:33:00.246 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.246 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.246 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.246 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:00.246 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302085+0000", 2024-09-21T12:33:00.246 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.246 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.246 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.246 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:00.246 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302086+0000", 2024-09-21T12:33:00.246 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4339999999999999e-06 2024-09-21T12:33:00.247 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.247 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.247 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:00.247 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302163+0000", 2024-09-21T12:33:00.247 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6200999999999998e-05 2024-09-21T12:33:00.247 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.247 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.247 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:00.247 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302172+0000", 2024-09-21T12:33:00.247 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5759999999999995e-06 2024-09-21T12:33:00.247 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.247 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.247 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:00.247 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302178+0000", 2024-09-21T12:33:00.248 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2260000000000002e-06 2024-09-21T12:33:00.248 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.248 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.248 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:00.248 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.303083+0000", 2024-09-21T12:33:00.248 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00090426600000000003 2024-09-21T12:33:00.248 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.248 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.248 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:00.248 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.303220+0000", 2024-09-21T12:33:00.248 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013702300000000001 2024-09-21T12:33:00.248 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.248 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.248 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-21T12:33:00.248 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.303354+0000", 2024-09-21T12:33:00.249 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013437099999999999 2024-09-21T12:33:00.249 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.249 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.249 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:00.249 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.305072+0000", 2024-09-21T12:33:00.249 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001718102 2024-09-21T12:33:00.249 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.249 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.249 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.249 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.306241+0000", 2024-09-21T12:33:00.249 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001168716 2024-09-21T12:33:00.249 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.249 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.249 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.250 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.453516+0000", 2024-09-21T12:33:00.250 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14727496200000001 2024-09-21T12:33:00.250 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.250 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.250 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:00.250 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.453572+0000", 2024-09-21T12:33:00.250 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5923e-05 2024-09-21T12:33:00.250 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.250 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.250 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:00.250 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.453617+0000", 2024-09-21T12:33:00.250 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5278000000000003e-05 2024-09-21T12:33:00.250 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.250 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:00.251 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.251 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.251 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.251 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:136741 9.f 9:f16b8898:::benchmark_data_smithi114_133214_object12707:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-09-21T12:33:00.251 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:30:12.044643+0000", 2024-09-21T12:33:00.251 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 168.18666195399999, 2024-09-21T12:33:00.251 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15204224699999999, 2024-09-21T12:33:00.251 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:00.251 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:00.251 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:00.251 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:00.251 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:00.251 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:00.251 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 136741 2024-09-21T12:33:00.252 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.252 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:00.252 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.252 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:00.252 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.044643+0000", 2024-09-21T12:33:00.252 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.252 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.252 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.252 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:00.252 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.044643+0000", 2024-09-21T12:33:00.252 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.252 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.252 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.252 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:00.252 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.044644+0000", 2024-09-21T12:33:00.253 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.043e-06 2024-09-21T12:33:00.253 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.253 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.253 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:00.253 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.044647+0000", 2024-09-21T12:33:00.253 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4199999999999999e-06 2024-09-21T12:33:00.253 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.253 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.253 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:00.253 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.044655+0000", 2024-09-21T12:33:00.253 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3649999999999998e-06 2024-09-21T12:33:00.253 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.253 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.253 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:00.253 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.044659+0000", 2024-09-21T12:33:00.254 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5800000000000002e-06 2024-09-21T12:33:00.254 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.254 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.254 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:00.254 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.044697+0000", 2024-09-21T12:33:00.254 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7499000000000001e-05 2024-09-21T12:33:00.254 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.254 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.254 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:00.254 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.044810+0000", 2024-09-21T12:33:00.254 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000113287 2024-09-21T12:33:00.254 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.254 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.254 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:33:00.255 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.044925+0000", 2024-09-21T12:33:00.255 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000115343 2024-09-21T12:33:00.255 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.255 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.255 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:00.255 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.046578+0000", 2024-09-21T12:33:00.255 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0016527989999999999 2024-09-21T12:33:00.255 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.255 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.255 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.255 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.154593+0000", 2024-09-21T12:33:00.255 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.108015135 2024-09-21T12:33:00.255 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.255 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.255 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.256 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.196609+0000", 2024-09-21T12:33:00.256 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.042015706999999999 2024-09-21T12:33:00.256 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.256 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.256 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:00.256 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.196649+0000", 2024-09-21T12:33:00.256 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9935999999999998e-05 2024-09-21T12:33:00.256 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.256 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.256 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:00.256 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.196685+0000", 2024-09-21T12:33:00.256 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6133000000000001e-05 2024-09-21T12:33:00.256 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.256 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:00.256 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.257 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.257 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.257 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:143984 9.f 9:fd48ee49:::benchmark_data_smithi114_133214_object19950:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-09-21T12:33:00.257 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:30:20.026313+0000", 2024-09-21T12:33:00.257 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 160.20499214399999, 2024-09-21T12:33:00.257 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.33116894200000002, 2024-09-21T12:33:00.257 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:00.257 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:00.257 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:00.257 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:00.257 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:00.257 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:00.257 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 143984 2024-09-21T12:33:00.257 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.258 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:00.258 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.258 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:00.258 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026313+0000", 2024-09-21T12:33:00.258 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.258 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.258 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.258 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:00.258 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026313+0000", 2024-09-21T12:33:00.258 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.258 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.258 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.258 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:00.258 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026314+0000", 2024-09-21T12:33:00.258 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.186e-06 2024-09-21T12:33:00.259 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.259 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.259 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:00.259 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026317+0000", 2024-09-21T12:33:00.259 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0989999999999999e-06 2024-09-21T12:33:00.259 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.259 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.259 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:00.259 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026329+0000", 2024-09-21T12:33:00.259 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1932e-05 2024-09-21T12:33:00.259 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.259 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.259 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:00.259 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026333+0000", 2024-09-21T12:33:00.259 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3370000000000001e-06 2024-09-21T12:33:00.260 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.260 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.260 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:00.260 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026364+0000", 2024-09-21T12:33:00.260 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0235000000000001e-05 2024-09-21T12:33:00.260 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.260 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.260 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:00.260 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026441+0000", 2024-09-21T12:33:00.260 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7663000000000005e-05 2024-09-21T12:33:00.260 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.260 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.260 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:33:00.260 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026511+0000", 2024-09-21T12:33:00.260 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9640000000000004e-05 2024-09-21T12:33:00.261 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.261 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.261 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:00.261 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.029298+0000", 2024-09-21T12:33:00.261 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0027875500000000002 2024-09-21T12:33:00.261 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.261 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.261 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.261 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.039048+0000", 2024-09-21T12:33:00.261 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0097491490000000004 2024-09-21T12:33:00.261 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.261 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.261 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.261 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.357393+0000", 2024-09-21T12:33:00.261 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31834596300000001 2024-09-21T12:33:00.262 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.262 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.262 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:00.262 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.357435+0000", 2024-09-21T12:33:00.262 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1078999999999998e-05 2024-09-21T12:33:00.262 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.262 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.262 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:00.262 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.357482+0000", 2024-09-21T12:33:00.262 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7108999999999999e-05 2024-09-21T12:33:00.262 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.262 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:00.262 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.262 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.263 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.263 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:143985 9.f 9:f12d363a:::benchmark_data_smithi114_133214_object19951:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-09-21T12:33:00.263 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:30:20.026340+0000", 2024-09-21T12:33:00.263 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 160.204964499, 2024-09-21T12:33:00.263 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.33955830100000001, 2024-09-21T12:33:00.263 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:00.263 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:00.263 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:00.263 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:00.263 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:00.263 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:00.263 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 143985 2024-09-21T12:33:00.263 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.263 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:00.263 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.264 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:00.264 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026340+0000", 2024-09-21T12:33:00.264 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.264 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.264 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.264 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:00.264 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026340+0000", 2024-09-21T12:33:00.264 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.264 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.264 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.264 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:00.264 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026341+0000", 2024-09-21T12:33:00.264 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.84e-07 2024-09-21T12:33:00.264 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.266 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.266 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:00.266 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026342+0000", 2024-09-21T12:33:00.266 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.671e-06 2024-09-21T12:33:00.266 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.266 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.266 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:00.266 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026346+0000", 2024-09-21T12:33:00.266 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5769999999999998e-06 2024-09-21T12:33:00.266 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.267 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.267 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:00.267 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026350+0000", 2024-09-21T12:33:00.267 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8029999999999998e-06 2024-09-21T12:33:00.267 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.267 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.267 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:00.267 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.027962+0000", 2024-09-21T12:33:00.267 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0016119839999999999 2024-09-21T12:33:00.267 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.267 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.267 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:00.267 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.028067+0000", 2024-09-21T12:33:00.267 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010498299999999999 2024-09-21T12:33:00.268 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.268 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.268 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:33:00.268 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.028160+0000", 2024-09-21T12:33:00.268 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3731000000000001e-05 2024-09-21T12:33:00.268 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.268 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.268 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:00.268 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.030647+0000", 2024-09-21T12:33:00.268 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024862310000000002 2024-09-21T12:33:00.268 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.268 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.268 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.269 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.039088+0000", 2024-09-21T12:33:00.269 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0084414620000000003 2024-09-21T12:33:00.269 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.269 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.269 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.269 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.365834+0000", 2024-09-21T12:33:00.269 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.32674630799999999 2024-09-21T12:33:00.269 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.269 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.269 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:00.269 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.365865+0000", 2024-09-21T12:33:00.269 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0732999999999999e-05 2024-09-21T12:33:00.269 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.269 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.270 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:00.270 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.365899+0000", 2024-09-21T12:33:00.270 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3534000000000001e-05 2024-09-21T12:33:00.270 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.270 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:00.270 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.270 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.270 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.270 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:143986 9.f 9:f1d8e647:::benchmark_data_smithi114_133214_object19952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-09-21T12:33:00.270 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:30:20.027267+0000", 2024-09-21T12:33:00.270 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 160.204037763, 2024-09-21T12:33:00.270 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.34414182799999998, 2024-09-21T12:33:00.270 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:00.270 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:00.271 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:00.271 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:00.271 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:00.271 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:00.271 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 143986 2024-09-21T12:33:00.271 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.271 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:00.271 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.271 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:00.271 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.027267+0000", 2024-09-21T12:33:00.271 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.271 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.271 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.271 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:00.272 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.027267+0000", 2024-09-21T12:33:00.272 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.272 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.272 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.272 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:00.272 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.027268+0000", 2024-09-21T12:33:00.272 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.065e-06 2024-09-21T12:33:00.272 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.272 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.272 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:00.272 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.027272+0000", 2024-09-21T12:33:00.272 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.318e-06 2024-09-21T12:33:00.272 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.272 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.273 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:00.273 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.027282+0000", 2024-09-21T12:33:00.273 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4490000000000005e-06 2024-09-21T12:33:00.273 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.273 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.273 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:00.273 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.027287+0000", 2024-09-21T12:33:00.273 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9019999999999998e-06 2024-09-21T12:33:00.273 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.273 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.273 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:00.273 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.029320+0000", 2024-09-21T12:33:00.273 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020333090000000001 2024-09-21T12:33:00.273 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.274 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.274 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:00.274 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.029383+0000", 2024-09-21T12:33:00.274 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3274000000000004e-05 2024-09-21T12:33:00.274 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.274 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.274 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:33:00.274 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.029450+0000", 2024-09-21T12:33:00.274 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6649999999999994e-05 2024-09-21T12:33:00.274 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.274 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.274 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:00.274 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.030775+0000", 2024-09-21T12:33:00.274 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001325346 2024-09-21T12:33:00.275 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.275 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.275 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.275 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.039124+0000", 2024-09-21T12:33:00.275 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0083483350000000001 2024-09-21T12:33:00.275 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.275 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.275 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.275 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.371309+0000", 2024-09-21T12:33:00.275 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.33218562400000001 2024-09-21T12:33:00.275 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.275 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.275 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:00.275 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.371364+0000", 2024-09-21T12:33:00.276 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4336999999999998e-05 2024-09-21T12:33:00.276 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.276 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.276 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:00.276 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.371409+0000", 2024-09-21T12:33:00.276 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5219000000000002e-05 2024-09-21T12:33:00.276 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.276 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:00.276 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.276 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.276 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.276 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:148258 9.f 9:fa68205d:::benchmark_data_smithi114_133214_object24224:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-09-21T12:33:00.276 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:30:25.552566+0000", 2024-09-21T12:33:00.276 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 154.678738753, 2024-09-21T12:33:00.277 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.165969116, 2024-09-21T12:33:00.277 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:00.277 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:00.277 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:00.277 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:00.277 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:00.277 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:00.277 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 148258 2024-09-21T12:33:00.277 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.277 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:00.277 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.277 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:00.277 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.552566+0000", 2024-09-21T12:33:00.277 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.277 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.278 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.278 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:00.278 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.552566+0000", 2024-09-21T12:33:00.278 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.278 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.278 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.278 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:00.278 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.552568+0000", 2024-09-21T12:33:00.278 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.406e-06 2024-09-21T12:33:00.278 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.278 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.278 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:00.278 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.552573+0000", 2024-09-21T12:33:00.278 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0030000000000002e-06 2024-09-21T12:33:00.279 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.279 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.279 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:00.279 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.552583+0000", 2024-09-21T12:33:00.279 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0379e-05 2024-09-21T12:33:00.279 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.279 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.279 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:00.279 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.552589+0000", 2024-09-21T12:33:00.279 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2480000000000001e-06 2024-09-21T12:33:00.279 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.279 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.279 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:00.279 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.552647+0000", 2024-09-21T12:33:00.279 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7422e-05 2024-09-21T12:33:00.280 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.280 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.280 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:00.280 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.552765+0000", 2024-09-21T12:33:00.280 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000118627 2024-09-21T12:33:00.280 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.280 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.280 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:33:00.280 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.552887+0000", 2024-09-21T12:33:00.280 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000121557 2024-09-21T12:33:00.280 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.280 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.280 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:00.280 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.554567+0000", 2024-09-21T12:33:00.281 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0016804249999999999 2024-09-21T12:33:00.281 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.281 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.281 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.281 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.569714+0000", 2024-09-21T12:33:00.281 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.015146980000000001 2024-09-21T12:33:00.281 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.281 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.281 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.281 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.718424+0000", 2024-09-21T12:33:00.281 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14870934699999999 2024-09-21T12:33:00.281 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.281 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.281 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:00.281 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.718488+0000", 2024-09-21T12:33:00.282 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4362000000000005e-05 2024-09-21T12:33:00.282 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.282 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.282 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:00.282 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.718535+0000", 2024-09-21T12:33:00.282 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7360000000000001e-05 2024-09-21T12:33:00.282 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.282 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:00.282 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.282 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.282 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.282 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:148262 9.f 9:f36a2eaf:::benchmark_data_smithi114_133214_object24228:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-09-21T12:33:00.282 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:30:25.564038+0000", 2024-09-21T12:33:00.282 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 154.667266944, 2024-09-21T12:33:00.283 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.154638887, 2024-09-21T12:33:00.283 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:00.283 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:00.283 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:00.283 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:00.283 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:00.283 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:00.283 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 148262 2024-09-21T12:33:00.283 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.283 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:00.283 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.283 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:00.283 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.564038+0000", 2024-09-21T12:33:00.283 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.284 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.284 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.284 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:00.284 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.564038+0000", 2024-09-21T12:33:00.284 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.284 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.284 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.284 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:00.284 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.564039+0000", 2024-09-21T12:33:00.284 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.144e-06 2024-09-21T12:33:00.284 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.284 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.284 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:00.284 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.564043+0000", 2024-09-21T12:33:00.284 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.25e-06 2024-09-21T12:33:00.285 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.285 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.285 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:00.285 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.564054+0000", 2024-09-21T12:33:00.285 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0963e-05 2024-09-21T12:33:00.285 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.285 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.285 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:00.285 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.564060+0000", 2024-09-21T12:33:00.285 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.981e-06 2024-09-21T12:33:00.285 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.285 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.285 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:00.285 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.564107+0000", 2024-09-21T12:33:00.286 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6758999999999997e-05 2024-09-21T12:33:00.286 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.286 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.286 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:00.286 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.564242+0000", 2024-09-21T12:33:00.286 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013536700000000001 2024-09-21T12:33:00.286 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.286 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.286 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:33:00.286 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.564364+0000", 2024-09-21T12:33:00.286 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000122062 2024-09-21T12:33:00.286 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.286 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.286 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:00.287 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.566033+0000", 2024-09-21T12:33:00.287 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001668444 2024-09-21T12:33:00.287 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.287 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.287 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.287 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.569752+0000", 2024-09-21T12:33:00.287 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0037187560000000001 2024-09-21T12:33:00.287 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.287 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.287 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.287 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.718589+0000", 2024-09-21T12:33:00.287 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14883724800000001 2024-09-21T12:33:00.287 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.287 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.287 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:00.288 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.718637+0000", 2024-09-21T12:33:00.288 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7763000000000003e-05 2024-09-21T12:33:00.288 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.288 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.288 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:00.288 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.718677+0000", 2024-09-21T12:33:00.288 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.015e-05 2024-09-21T12:33:00.288 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.288 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:00.288 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.288 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.288 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.288 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:148264 9.f 9:f2a805cb:::benchmark_data_smithi114_133214_object24230:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-09-21T12:33:00.288 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:30:25.566769+0000", 2024-09-21T12:33:00.289 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 154.66453632599999, 2024-09-21T12:33:00.289 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15200875599999999, 2024-09-21T12:33:00.289 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:00.289 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:00.289 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:00.289 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:00.289 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:00.289 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:00.289 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 148264 2024-09-21T12:33:00.289 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.289 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:00.289 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.289 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:00.289 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.566769+0000", 2024-09-21T12:33:00.289 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.290 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.290 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.290 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:00.290 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.566769+0000", 2024-09-21T12:33:00.290 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.290 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.290 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.290 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:00.290 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.566770+0000", 2024-09-21T12:33:00.290 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.043e-06 2024-09-21T12:33:00.290 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.290 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.290 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:00.290 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.566773+0000", 2024-09-21T12:33:00.290 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8179999999999997e-06 2024-09-21T12:33:00.291 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.291 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.291 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:00.291 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.566783+0000", 2024-09-21T12:33:00.291 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4299999999999995e-06 2024-09-21T12:33:00.291 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.291 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.291 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:00.291 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.566794+0000", 2024-09-21T12:33:00.291 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0910999999999999e-05 2024-09-21T12:33:00.291 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.291 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.291 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:00.291 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.566853+0000", 2024-09-21T12:33:00.292 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8943999999999997e-05 2024-09-21T12:33:00.292 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.292 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.292 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:00.292 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.566971+0000", 2024-09-21T12:33:00.292 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000118113 2024-09-21T12:33:00.292 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.292 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.292 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:33:00.292 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.567091+0000", 2024-09-21T12:33:00.292 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000120232 2024-09-21T12:33:00.292 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.292 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.292 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:00.293 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.568779+0000", 2024-09-21T12:33:00.293 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001688134 2024-09-21T12:33:00.293 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.293 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.293 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.293 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.569859+0000", 2024-09-21T12:33:00.293 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010798330000000001 2024-09-21T12:33:00.293 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.293 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.293 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.293 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.718708+0000", 2024-09-21T12:33:00.293 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14884867800000001 2024-09-21T12:33:00.293 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.294 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.294 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:00.294 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.718752+0000", 2024-09-21T12:33:00.294 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4379999999999999e-05 2024-09-21T12:33:00.294 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.294 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.294 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:00.294 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.718777+0000", 2024-09-21T12:33:00.294 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5239999999999999e-05 2024-09-21T12:33:00.294 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.294 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:00.294 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.294 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.294 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.295 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:175495 9.f 9:f9909850:::benchmark_data_smithi114_133214_object51461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-21T12:33:00.295 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:30:56.789850+0000", 2024-09-21T12:33:00.295 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 123.44145437, 2024-09-21T12:33:00.295 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.16364801100000001, 2024-09-21T12:33:00.295 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:00.295 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:00.295 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:00.295 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:00.295 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:00.295 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:00.295 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 175495 2024-09-21T12:33:00.295 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.295 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:00.295 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.295 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:00.296 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.789850+0000", 2024-09-21T12:33:00.296 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.296 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.296 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.296 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:00.296 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.789850+0000", 2024-09-21T12:33:00.296 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.296 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.296 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.296 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:00.296 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.789852+0000", 2024-09-21T12:33:00.296 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.573e-06 2024-09-21T12:33:00.296 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.296 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.297 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:00.297 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.789857+0000", 2024-09-21T12:33:00.297 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0209999999999999e-06 2024-09-21T12:33:00.297 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.297 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.297 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:00.297 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.789868+0000", 2024-09-21T12:33:00.297 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0496000000000001e-05 2024-09-21T12:33:00.297 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.297 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.297 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:00.297 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.789874+0000", 2024-09-21T12:33:00.297 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7599999999999997e-06 2024-09-21T12:33:00.298 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.298 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.298 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:00.298 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.789923+0000", 2024-09-21T12:33:00.298 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8188e-05 2024-09-21T12:33:00.298 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.298 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.298 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:00.298 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.790047+0000", 2024-09-21T12:33:00.298 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012412999999999999 2024-09-21T12:33:00.298 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.298 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.298 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:33:00.298 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.790161+0000", 2024-09-21T12:33:00.299 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011474800000000001 2024-09-21T12:33:00.299 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.299 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.299 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:00.299 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.791943+0000", 2024-09-21T12:33:00.299 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0017820710000000001 2024-09-21T12:33:00.299 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.299 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.299 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.299 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.943029+0000", 2024-09-21T12:33:00.299 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15108565800000001 2024-09-21T12:33:00.299 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.299 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.299 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.300 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.953435+0000", 2024-09-21T12:33:00.300 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.010406227000000001 2024-09-21T12:33:00.300 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.300 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.300 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:00.300 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.953469+0000", 2024-09-21T12:33:00.300 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3969e-05 2024-09-21T12:33:00.300 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.300 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.300 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:00.300 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.953499+0000", 2024-09-21T12:33:00.300 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.917e-05 2024-09-21T12:33:00.300 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.300 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:00.300 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.301 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.301 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.301 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:216505 9.f 9:f5ab0cee:::benchmark_data_smithi114_133214_object92471:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e306)", 2024-09-21T12:33:00.301 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:33.894635+0000", 2024-09-21T12:33:00.301 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 86.336669485000002, 2024-09-21T12:33:00.301 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.38916874499999998, 2024-09-21T12:33:00.301 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:00.301 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:00.301 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:00.301 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:00.301 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:00.301 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:00.301 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 216505 2024-09-21T12:33:00.301 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.302 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:00.302 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.302 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:00.302 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.894635+0000", 2024-09-21T12:33:00.302 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.302 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.302 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.302 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:00.302 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.894635+0000", 2024-09-21T12:33:00.302 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.302 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.302 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.302 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:00.302 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.894637+0000", 2024-09-21T12:33:00.302 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.654e-06 2024-09-21T12:33:00.303 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.303 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.303 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:00.303 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.894642+0000", 2024-09-21T12:33:00.303 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0390000000000004e-06 2024-09-21T12:33:00.303 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.303 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.303 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:00.303 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.894658+0000", 2024-09-21T12:33:00.303 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6379999999999999e-05 2024-09-21T12:33:00.303 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.303 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.303 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:00.303 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.894666+0000", 2024-09-21T12:33:00.303 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0550000000000002e-06 2024-09-21T12:33:00.304 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.304 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.304 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:00.304 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.894723+0000", 2024-09-21T12:33:00.304 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7982000000000002e-05 2024-09-21T12:33:00.304 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.304 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.304 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:00.304 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.894802+0000", 2024-09-21T12:33:00.304 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8399000000000006e-05 2024-09-21T12:33:00.304 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.304 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.304 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:33:00.304 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.894874+0000", 2024-09-21T12:33:00.305 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2423999999999996e-05 2024-09-21T12:33:00.305 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.305 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.305 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:00.305 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.896880+0000", 2024-09-21T12:33:00.305 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020054180000000001 2024-09-21T12:33:00.305 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.305 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.305 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.305 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.900439+0000", 2024-09-21T12:33:00.305 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0035589409999999999 2024-09-21T12:33:00.305 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.305 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.305 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.305 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:34.283682+0000", 2024-09-21T12:33:00.305 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.38324365500000002 2024-09-21T12:33:00.306 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.306 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.306 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:00.306 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:34.283755+0000", 2024-09-21T12:33:00.306 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2329000000000005e-05 2024-09-21T12:33:00.306 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.306 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.306 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:00.306 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:34.283804+0000", 2024-09-21T12:33:00.306 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9468999999999997e-05 2024-09-21T12:33:00.306 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.306 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:00.306 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.306 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.307 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.307 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:216514 9.f 9:f58474dd:::benchmark_data_smithi114_133214_object92480:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e306)", 2024-09-21T12:33:00.307 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:33.900257+0000", 2024-09-21T12:33:00.307 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 86.331048179000007, 2024-09-21T12:33:00.307 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.38852348399999997, 2024-09-21T12:33:00.307 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:00.307 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:00.307 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:00.307 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:00.307 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:00.307 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:00.307 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 216514 2024-09-21T12:33:00.307 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.307 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:00.308 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.308 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:00.308 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.900257+0000", 2024-09-21T12:33:00.308 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.308 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.308 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.308 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:00.308 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.900257+0000", 2024-09-21T12:33:00.308 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.308 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.308 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.308 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:00.308 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.900258+0000", 2024-09-21T12:33:00.308 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.57e-06 2024-09-21T12:33:00.309 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.309 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.309 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:00.309 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.900264+0000", 2024-09-21T12:33:00.309 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2979999999999999e-06 2024-09-21T12:33:00.309 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.309 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.309 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:00.309 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.900279+0000", 2024-09-21T12:33:00.309 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5211000000000001e-05 2024-09-21T12:33:00.309 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.309 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.309 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:00.309 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.900286+0000", 2024-09-21T12:33:00.309 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2089999999999999e-06 2024-09-21T12:33:00.310 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.310 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.310 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:00.310 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.900470+0000", 2024-09-21T12:33:00.310 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00018402699999999999 2024-09-21T12:33:00.310 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.310 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.310 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:00.310 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.900607+0000", 2024-09-21T12:33:00.310 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013691800000000001 2024-09-21T12:33:00.310 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.310 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.310 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:33:00.310 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.900732+0000", 2024-09-21T12:33:00.311 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000125081 2024-09-21T12:33:00.311 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.311 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.311 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:00.311 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.902574+0000", 2024-09-21T12:33:00.311 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0018421290000000001 2024-09-21T12:33:00.311 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.311 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.311 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.311 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.903091+0000", 2024-09-21T12:33:00.311 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000517214 2024-09-21T12:33:00.311 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.311 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.311 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.312 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:34.288673+0000", 2024-09-21T12:33:00.312 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.38558214000000002 2024-09-21T12:33:00.312 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.312 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.312 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:00.312 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:34.288738+0000", 2024-09-21T12:33:00.312 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4135999999999997e-05 2024-09-21T12:33:00.312 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.312 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.312 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:00.312 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:34.288780+0000", 2024-09-21T12:33:00.312 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2551e-05 2024-09-21T12:33:00.312 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.312 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:00.313 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.313 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.313 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.313 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:231697 9.19 9:9abf7afe:::benchmark_data_smithi114_133214_object107663:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:33:00.313 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:50.030279+0000", 2024-09-21T12:33:00.313 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 70.201025751000003, 2024-09-21T12:33:00.313 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1184929130000001, 2024-09-21T12:33:00.313 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:00.313 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:00.313 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:00.313 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:00.313 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:00.313 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:00.313 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 231697 2024-09-21T12:33:00.314 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.314 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:00.314 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.314 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:00.314 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030279+0000", 2024-09-21T12:33:00.314 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.314 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.314 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.314 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:00.314 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030279+0000", 2024-09-21T12:33:00.314 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.314 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.314 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.315 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:00.315 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030281+0000", 2024-09-21T12:33:00.315 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.689e-06 2024-09-21T12:33:00.315 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.315 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.315 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:00.315 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030285+0000", 2024-09-21T12:33:00.315 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9859999999999996e-06 2024-09-21T12:33:00.315 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.315 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.315 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:00.315 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030295+0000", 2024-09-21T12:33:00.315 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0178e-05 2024-09-21T12:33:00.315 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.315 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.316 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:00.316 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030305+0000", 2024-09-21T12:33:00.316 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7860000000000008e-06 2024-09-21T12:33:00.316 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.316 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.316 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:00.316 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030481+0000", 2024-09-21T12:33:00.316 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000175768 2024-09-21T12:33:00.316 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.316 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.316 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:00.316 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030490+0000", 2024-09-21T12:33:00.316 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7669999999999998e-06 2024-09-21T12:33:00.316 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.317 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.317 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:00.317 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.144698+0000", 2024-09-21T12:33:00.317 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1142073050000001 2024-09-21T12:33:00.317 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.317 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.317 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:00.317 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.144803+0000", 2024-09-21T12:33:00.317 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010563100000000001 2024-09-21T12:33:00.317 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.317 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.317 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,6,7", 2024-09-21T12:33:00.317 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.145159+0000", 2024-09-21T12:33:00.317 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00035557900000000002 2024-09-21T12:33:00.318 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.318 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.318 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.318 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.148533+0000", 2024-09-21T12:33:00.318 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003373741 2024-09-21T12:33:00.318 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.318 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.318 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.318 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.148585+0000", 2024-09-21T12:33:00.318 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2278999999999999e-05 2024-09-21T12:33:00.318 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.318 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.318 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.318 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.148619+0000", 2024-09-21T12:33:00.319 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.417e-05 2024-09-21T12:33:00.319 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.319 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.319 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:00.319 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.148629+0000", 2024-09-21T12:33:00.319 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8779999999999993e-06 2024-09-21T12:33:00.319 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.319 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.319 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:00.319 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.148667+0000", 2024-09-21T12:33:00.319 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7728999999999997e-05 2024-09-21T12:33:00.319 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.319 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.319 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:00.320 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.148772+0000", 2024-09-21T12:33:00.320 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010542700000000001 2024-09-21T12:33:00.320 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.320 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:00.320 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.320 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.320 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.320 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:231705 9.19 9:9e8af994:::benchmark_data_smithi114_133214_object107671:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:33:00.320 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:50.030314+0000", 2024-09-21T12:33:00.320 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 70.200990449000003, 2024-09-21T12:33:00.320 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1228952539999999, 2024-09-21T12:33:00.320 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:00.320 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:00.321 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:00.321 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:00.321 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:00.321 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:00.321 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 231705 2024-09-21T12:33:00.321 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.321 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:00.321 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.321 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:00.321 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030314+0000", 2024-09-21T12:33:00.321 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.321 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.321 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.322 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:00.322 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030314+0000", 2024-09-21T12:33:00.322 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.322 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.322 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.322 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:00.322 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030315+0000", 2024-09-21T12:33:00.322 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9299999999999998e-07 2024-09-21T12:33:00.322 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.322 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.322 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:00.322 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030317+0000", 2024-09-21T12:33:00.322 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2450000000000001e-06 2024-09-21T12:33:00.323 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.323 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.323 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:00.323 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030323+0000", 2024-09-21T12:33:00.323 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8939999999999998e-06 2024-09-21T12:33:00.323 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.323 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.323 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:00.323 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030328+0000", 2024-09-21T12:33:00.323 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1800000000000004e-06 2024-09-21T12:33:00.323 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.323 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.323 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:00.323 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030512+0000", 2024-09-21T12:33:00.324 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00018370300000000001 2024-09-21T12:33:00.324 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.324 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.324 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:00.324 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030518+0000", 2024-09-21T12:33:00.324 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2319999999999998e-06 2024-09-21T12:33:00.324 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.324 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.324 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:00.324 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.148936+0000", 2024-09-21T12:33:00.324 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1184178929999999 2024-09-21T12:33:00.324 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.324 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.324 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:00.325 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.149021+0000", 2024-09-21T12:33:00.325 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4841999999999998e-05 2024-09-21T12:33:00.325 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.325 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.325 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,6,7", 2024-09-21T12:33:00.325 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.149286+0000", 2024-09-21T12:33:00.325 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00026551899999999998 2024-09-21T12:33:00.325 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.325 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.325 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.325 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153018+0000", 2024-09-21T12:33:00.325 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0037314340000000001 2024-09-21T12:33:00.325 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.325 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.325 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.326 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153057+0000", 2024-09-21T12:33:00.326 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9365e-05 2024-09-21T12:33:00.326 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.326 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.326 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:00.326 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153068+0000", 2024-09-21T12:33:00.326 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.11e-05 2024-09-21T12:33:00.326 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.326 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.326 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.326 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153107+0000", 2024-09-21T12:33:00.326 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8305e-05 2024-09-21T12:33:00.326 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.326 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.327 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:00.327 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153144+0000", 2024-09-21T12:33:00.327 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7110000000000002e-05 2024-09-21T12:33:00.327 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.327 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.327 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:00.327 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153210+0000", 2024-09-21T12:33:00.327 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5938999999999993e-05 2024-09-21T12:33:00.327 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.327 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:00.327 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.327 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.327 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.328 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:231765 9.19 9:9e977502:::benchmark_data_smithi114_133214_object107731:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:33:00.328 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:50.030334+0000", 2024-09-21T12:33:00.328 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 70.200970859999998, 2024-09-21T12:33:00.328 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.123057872, 2024-09-21T12:33:00.328 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:00.328 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:00.328 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:00.328 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:00.328 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:00.328 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:00.328 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 231765 2024-09-21T12:33:00.328 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.328 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:00.328 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.328 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:00.328 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030334+0000", 2024-09-21T12:33:00.329 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.329 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.329 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.329 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:00.329 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030334+0000", 2024-09-21T12:33:00.329 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.329 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.329 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.329 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:00.329 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030334+0000", 2024-09-21T12:33:00.329 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2300000000000002e-07 2024-09-21T12:33:00.329 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.329 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.330 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:00.330 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030337+0000", 2024-09-21T12:33:00.330 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0949999999999998e-06 2024-09-21T12:33:00.330 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.330 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.330 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:00.330 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030342+0000", 2024-09-21T12:33:00.330 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7710000000000003e-06 2024-09-21T12:33:00.330 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.330 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.330 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:00.330 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030347+0000", 2024-09-21T12:33:00.330 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.198e-06 2024-09-21T12:33:00.330 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.330 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.331 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:00.331 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030540+0000", 2024-09-21T12:33:00.331 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019217600000000001 2024-09-21T12:33:00.331 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.331 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.331 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:00.331 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030546+0000", 2024-09-21T12:33:00.331 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.212e-06 2024-09-21T12:33:00.331 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.331 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.331 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:00.331 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.150826+0000", 2024-09-21T12:33:00.331 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.120280194 2024-09-21T12:33:00.331 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.331 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.332 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:00.332 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.150950+0000", 2024-09-21T12:33:00.332 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012343099999999999 2024-09-21T12:33:00.332 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.332 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.332 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,6,7", 2024-09-21T12:33:00.332 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.151239+0000", 2024-09-21T12:33:00.332 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00028982299999999998 2024-09-21T12:33:00.332 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.332 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.332 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:00.332 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153075+0000", 2024-09-21T12:33:00.332 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001835723 2024-09-21T12:33:00.332 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.333 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.333 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.333 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153243+0000", 2024-09-21T12:33:00.333 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000167973 2024-09-21T12:33:00.333 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.333 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.333 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.333 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153273+0000", 2024-09-21T12:33:00.334 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0176999999999999e-05 2024-09-21T12:33:00.335 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.335 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.335 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.335 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153296+0000", 2024-09-21T12:33:00.335 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2816e-05 2024-09-21T12:33:00.335 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.335 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.335 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:00.335 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153327+0000", 2024-09-21T12:33:00.335 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1031999999999997e-05 2024-09-21T12:33:00.335 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.335 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.335 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:00.336 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153392+0000", 2024-09-21T12:33:00.336 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4827999999999997e-05 2024-09-21T12:33:00.336 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.336 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:00.336 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.336 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.336 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.336 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:235406 9.f 9:fff68ecc:::benchmark_data_smithi114_133214_object111372:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-09-21T12:33:00.336 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:56.217779+0000", 2024-09-21T12:33:00.336 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 64.013525657000002, 2024-09-21T12:33:00.336 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15964376999999999, 2024-09-21T12:33:00.336 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:00.336 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:00.336 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:00.337 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:00.337 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:00.337 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:00.337 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 235406 2024-09-21T12:33:00.337 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.337 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:00.337 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.337 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:00.337 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.217779+0000", 2024-09-21T12:33:00.337 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.337 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.337 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.337 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:00.338 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.217779+0000", 2024-09-21T12:33:00.338 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.338 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.338 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.338 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:00.338 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.217780+0000", 2024-09-21T12:33:00.338 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8000000000000005e-07 2024-09-21T12:33:00.338 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.338 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.338 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:00.338 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.217784+0000", 2024-09-21T12:33:00.338 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7349999999999998e-06 2024-09-21T12:33:00.338 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.338 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.338 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:00.339 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.217795+0000", 2024-09-21T12:33:00.339 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0934000000000001e-05 2024-09-21T12:33:00.339 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.339 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.339 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:00.339 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.217801+0000", 2024-09-21T12:33:00.339 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9399999999999999e-06 2024-09-21T12:33:00.339 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.339 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.339 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:00.339 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.223367+0000", 2024-09-21T12:33:00.339 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0055666259999999999 2024-09-21T12:33:00.339 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.339 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.339 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:00.340 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.223468+0000", 2024-09-21T12:33:00.340 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100368 2024-09-21T12:33:00.340 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.340 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.340 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:33:00.340 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.223558+0000", 2024-09-21T12:33:00.340 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9945999999999996e-05 2024-09-21T12:33:00.340 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.340 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.340 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.340 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.225567+0000", 2024-09-21T12:33:00.340 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020092069999999998 2024-09-21T12:33:00.340 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.340 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.340 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:00.341 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.227739+0000", 2024-09-21T12:33:00.341 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021725160000000002 2024-09-21T12:33:00.341 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.341 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.341 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.341 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.377351+0000", 2024-09-21T12:33:00.341 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.149611789 2024-09-21T12:33:00.341 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.341 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.341 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:00.341 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.377385+0000", 2024-09-21T12:33:00.341 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3680000000000003e-05 2024-09-21T12:33:00.341 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.341 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.342 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:00.342 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.377423+0000", 2024-09-21T12:33:00.342 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8248999999999999e-05 2024-09-21T12:33:00.342 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.342 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:00.342 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.342 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.342 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.342 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:235409 9.f 9:f26d45b2:::benchmark_data_smithi114_133214_object111375:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-09-21T12:33:00.342 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:56.218056+0000", 2024-09-21T12:33:00.342 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 64.013249189000007, 2024-09-21T12:33:00.342 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.16153879900000001, 2024-09-21T12:33:00.342 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:00.343 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:00.343 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:00.343 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:00.343 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:00.343 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:00.343 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 235409 2024-09-21T12:33:00.343 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.343 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:00.343 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.343 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:00.343 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.218056+0000", 2024-09-21T12:33:00.343 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.343 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.343 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.344 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:00.344 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.218056+0000", 2024-09-21T12:33:00.344 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.344 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.344 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.344 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:00.344 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.218056+0000", 2024-09-21T12:33:00.344 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9999999999999996e-07 2024-09-21T12:33:00.344 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.344 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.344 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:00.344 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.218060+0000", 2024-09-21T12:33:00.344 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6820000000000001e-06 2024-09-21T12:33:00.344 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.345 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.345 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:00.345 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.218072+0000", 2024-09-21T12:33:00.345 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1511999999999999e-05 2024-09-21T12:33:00.345 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.345 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.345 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:00.345 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.218078+0000", 2024-09-21T12:33:00.345 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0260000000000004e-06 2024-09-21T12:33:00.345 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.345 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.345 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:00.345 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.225599+0000", 2024-09-21T12:33:00.345 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0075208879999999999 2024-09-21T12:33:00.345 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.346 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.346 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:00.346 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.225706+0000", 2024-09-21T12:33:00.346 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010780099999999999 2024-09-21T12:33:00.346 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.346 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.346 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:33:00.346 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.225831+0000", 2024-09-21T12:33:00.346 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012434100000000001 2024-09-21T12:33:00.346 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.346 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.346 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:00.346 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.227745+0000", 2024-09-21T12:33:00.347 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0019139090000000001 2024-09-21T12:33:00.347 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.347 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.347 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.347 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.228076+0000", 2024-09-21T12:33:00.347 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00033095599999999998 2024-09-21T12:33:00.347 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.347 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.347 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.347 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.379487+0000", 2024-09-21T12:33:00.347 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15141185500000001 2024-09-21T12:33:00.347 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.347 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.347 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:00.348 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.379551+0000", 2024-09-21T12:33:00.348 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3971000000000001e-05 2024-09-21T12:33:00.348 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.348 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.348 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:00.348 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.379594+0000", 2024-09-21T12:33:00.348 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3058e-05 2024-09-21T12:33:00.348 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.348 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:00.348 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.348 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.348 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.348 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:235410 9.f 9:fc9ff039:::benchmark_data_smithi114_133214_object111376:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-09-21T12:33:00.348 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:56.219546+0000", 2024-09-21T12:33:00.349 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 64.011758611000005, 2024-09-21T12:33:00.349 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.160211877, 2024-09-21T12:33:00.349 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:00.349 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:00.349 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:00.349 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:00.349 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:00.349 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:00.349 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 235410 2024-09-21T12:33:00.349 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.349 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:00.349 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.349 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:00.349 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.219546+0000", 2024-09-21T12:33:00.350 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.350 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.350 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.350 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:00.350 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.219546+0000", 2024-09-21T12:33:00.350 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.350 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.350 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.350 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:00.350 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.219547+0000", 2024-09-21T12:33:00.350 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1429999999999999e-06 2024-09-21T12:33:00.350 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.350 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.351 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:00.351 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.219552+0000", 2024-09-21T12:33:00.351 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3170000000000003e-06 2024-09-21T12:33:00.351 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.351 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.351 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:00.351 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.219561+0000", 2024-09-21T12:33:00.351 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2119999999999992e-06 2024-09-21T12:33:00.351 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.351 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.351 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:00.351 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.219568+0000", 2024-09-21T12:33:00.352 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.764e-06 2024-09-21T12:33:00.352 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.352 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.352 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:00.352 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.226862+0000", 2024-09-21T12:33:00.352 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0072939019999999997 2024-09-21T12:33:00.352 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.352 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.352 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:00.352 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.226927+0000", 2024-09-21T12:33:00.352 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5598000000000005e-05 2024-09-21T12:33:00.352 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.352 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.352 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:33:00.353 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.227004+0000", 2024-09-21T12:33:00.353 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6323999999999996e-05 2024-09-21T12:33:00.353 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.353 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.353 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:00.353 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.227859+0000", 2024-09-21T12:33:00.353 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00085563900000000005 2024-09-21T12:33:00.353 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.353 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.353 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.353 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.372977+0000", 2024-09-21T12:33:00.353 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14511786199999999 2024-09-21T12:33:00.353 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.353 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.354 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.354 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.379660+0000", 2024-09-21T12:33:00.354 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0066829469999999998 2024-09-21T12:33:00.354 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.354 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.354 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:00.354 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.379717+0000", 2024-09-21T12:33:00.354 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6575e-05 2024-09-21T12:33:00.354 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.354 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.354 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:00.354 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.379758+0000", 2024-09-21T12:33:00.354 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1594000000000002e-05 2024-09-21T12:33:00.354 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.355 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:00.355 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.355 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.355 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.355 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:239971 9.f 9:f3d2bd83:::benchmark_data_smithi114_133214_object115937:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T12:33:00.355 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:59.801376+0000", 2024-09-21T12:33:00.355 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 60.429928373000003, 2024-09-21T12:33:00.355 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.024227555000000001, 2024-09-21T12:33:00.355 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:00.355 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:00.355 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:00.355 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:00.355 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:00.355 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:00.355 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 239971 2024-09-21T12:33:00.356 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.356 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:00.356 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.356 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:00.356 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.801376+0000", 2024-09-21T12:33:00.356 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.356 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.356 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.356 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:00.356 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.801376+0000", 2024-09-21T12:33:00.356 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.356 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.356 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.356 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:00.356 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.801377+0000", 2024-09-21T12:33:00.357 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6800000000000009e-07 2024-09-21T12:33:00.357 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.357 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.357 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:00.357 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.801381+0000", 2024-09-21T12:33:00.357 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7620000000000001e-06 2024-09-21T12:33:00.357 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.357 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.357 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:00.357 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.801388+0000", 2024-09-21T12:33:00.357 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0079999999999996e-06 2024-09-21T12:33:00.357 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.357 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.357 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:00.357 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.801392+0000", 2024-09-21T12:33:00.357 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1389999999999997e-06 2024-09-21T12:33:00.358 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.358 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.358 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:00.358 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.801421+0000", 2024-09-21T12:33:00.358 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8651000000000001e-05 2024-09-21T12:33:00.358 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.358 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.358 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:00.358 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.801513+0000", 2024-09-21T12:33:00.358 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1932e-05 2024-09-21T12:33:00.358 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.358 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.358 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:33:00.358 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.801587+0000", 2024-09-21T12:33:00.359 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4507999999999999e-05 2024-09-21T12:33:00.359 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.359 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.359 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:00.359 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.805590+0000", 2024-09-21T12:33:00.359 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0040025190000000004 2024-09-21T12:33:00.359 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.359 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.359 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.359 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.812648+0000", 2024-09-21T12:33:00.359 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0070584480000000002 2024-09-21T12:33:00.359 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.359 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.359 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.359 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.825504+0000", 2024-09-21T12:33:00.359 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.012855602000000001 2024-09-21T12:33:00.360 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.360 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.360 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:00.360 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.825558+0000", 2024-09-21T12:33:00.360 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4243999999999998e-05 2024-09-21T12:33:00.360 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.360 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.360 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:00.360 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.825604+0000", 2024-09-21T12:33:00.360 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5773999999999999e-05 2024-09-21T12:33:00.360 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.360 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:00.360 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.360 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.360 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.360 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:239975 9.f 9:f9b00e91:::benchmark_data_smithi114_133214_object115941:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T12:33:00.361 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:59.801401+0000", 2024-09-21T12:33:00.361 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 60.429904022000002, 2024-09-21T12:33:00.361 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.028372571999999999, 2024-09-21T12:33:00.361 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:00.361 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:00.361 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:00.361 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:00.361 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:00.361 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:00.361 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 239975 2024-09-21T12:33:00.361 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.361 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:00.361 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.361 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:00.362 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.801401+0000", 2024-09-21T12:33:00.362 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.362 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.362 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.362 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:00.362 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.801401+0000", 2024-09-21T12:33:00.362 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:00.362 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.362 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.362 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:00.362 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.801401+0000", 2024-09-21T12:33:00.362 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1100000000000001e-07 2024-09-21T12:33:00.362 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.362 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.363 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:00.363 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.801403+0000", 2024-09-21T12:33:00.363 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1500000000000002e-06 2024-09-21T12:33:00.363 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.363 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.363 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:00.363 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.801408+0000", 2024-09-21T12:33:00.363 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9949999999999996e-06 2024-09-21T12:33:00.363 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.363 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.363 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:00.363 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.801412+0000", 2024-09-21T12:33:00.363 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0049999999999998e-06 2024-09-21T12:33:00.364 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.364 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.364 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:00.364 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.803589+0000", 2024-09-21T12:33:00.364 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021761660000000002 2024-09-21T12:33:00.364 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.364 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.364 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:00.364 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.803690+0000", 2024-09-21T12:33:00.364 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010166399999999999 2024-09-21T12:33:00.364 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.364 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.364 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:33:00.364 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.803793+0000", 2024-09-21T12:33:00.364 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000103157 2024-09-21T12:33:00.364 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.365 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.365 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:00.365 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.805758+0000", 2024-09-21T12:33:00.365 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0019647219999999999 2024-09-21T12:33:00.365 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.365 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.365 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.365 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.812711+0000", 2024-09-21T12:33:00.365 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0069532550000000002 2024-09-21T12:33:00.365 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.365 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.365 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:00.365 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.829687+0000", 2024-09-21T12:33:00.365 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.016975253999999999 2024-09-21T12:33:00.365 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.366 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.366 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:00.366 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.829737+0000", 2024-09-21T12:33:00.366 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0813000000000003e-05 2024-09-21T12:33:00.366 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:00.366 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:00.366 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:00.366 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.829773+0000", 2024-09-21T12:33:00.366 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5979999999999998e-05 2024-09-21T12:33:00.366 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.366 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:00.366 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.366 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:00.367 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:00.367 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:33:00.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:00.367 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:00.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:00.468 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:00.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:00 smithi183 ceph-mon[33277]: 5.17 scrub starts 2024-09-21T12:33:00.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:00 smithi183 ceph-mon[33277]: 5.17 scrub ok 2024-09-21T12:33:00.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:00 smithi183 ceph-mon[33277]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:33:00.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:00 smithi183 ceph-mon[33277]: Cluster is now healthy 2024-09-21T12:33:00.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:00 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/814921270' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:33:00.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:00 smithi183 ceph-mon[33277]: osdmap e359: 8 total, 8 up, 8 in 2024-09-21T12:33:00.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:00 smithi183 ceph-mon[33277]: 6.3 scrub starts 2024-09-21T12:33:00.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:00 smithi183 ceph-mon[33277]: 6.3 scrub ok 2024-09-21T12:33:00.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:00 smithi183 ceph-mon[33277]: 6.4 scrub starts 2024-09-21T12:33:00.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:00 smithi183 ceph-mon[33277]: 6.4 scrub ok 2024-09-21T12:33:00.501 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:00.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:00.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:00.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:00 smithi114 ceph-mon[40867]: 5.17 scrub starts 2024-09-21T12:33:00.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:00 smithi114 ceph-mon[40867]: 5.17 scrub ok 2024-09-21T12:33:00.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:00 smithi114 ceph-mon[40867]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:33:00.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:00 smithi114 ceph-mon[40867]: Cluster is now healthy 2024-09-21T12:33:00.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:00 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/814921270' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:33:00.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:00 smithi114 ceph-mon[40867]: osdmap e359: 8 total, 8 up, 8 in 2024-09-21T12:33:00.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:00 smithi114 ceph-mon[40867]: 6.3 scrub starts 2024-09-21T12:33:00.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:00 smithi114 ceph-mon[40867]: 6.3 scrub ok 2024-09-21T12:33:00.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:00 smithi114 ceph-mon[40867]: 6.4 scrub starts 2024-09-21T12:33:00.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:00 smithi114 ceph-mon[40867]: 6.4 scrub ok 2024-09-21T12:33:00.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:00 smithi114 ceph-mon[30324]: 5.17 scrub starts 2024-09-21T12:33:00.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:00 smithi114 ceph-mon[30324]: 5.17 scrub ok 2024-09-21T12:33:00.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:00 smithi114 ceph-mon[30324]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:33:00.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:00 smithi114 ceph-mon[30324]: Cluster is now healthy 2024-09-21T12:33:00.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:00 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/814921270' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:33:00.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:00 smithi114 ceph-mon[30324]: osdmap e359: 8 total, 8 up, 8 in 2024-09-21T12:33:00.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:00 smithi114 ceph-mon[30324]: 6.3 scrub starts 2024-09-21T12:33:00.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:00 smithi114 ceph-mon[30324]: 6.3 scrub ok 2024-09-21T12:33:00.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:00 smithi114 ceph-mon[30324]: 6.4 scrub starts 2024-09-21T12:33:00.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:00 smithi114 ceph-mon[30324]: 6.4 scrub ok 2024-09-21T12:33:00.604 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:00.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:00.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:00.649 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 24 16 42959 42943 111.815 107.312 0.0094094 0.00893833 2024-09-21T12:33:00.743 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:00.764 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:00.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:00.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:00.866 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:00.886 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:00.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:00.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:00.988 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:01.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:01.010 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:01.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:01.111 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:01.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:01.133 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:01.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:01.170 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T12:33:01.234 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:01.271 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:01.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:01.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:01.373 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:01.416 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:01.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:01.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:01 smithi183 ceph-mon[33277]: pgmap v654: 178 pgs: 178 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 93 MiB/s wr, 1.50k op/s 2024-09-21T12:33:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:01 smithi183 ceph-mon[33277]: 5.5 scrub starts 2024-09-21T12:33:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:01 smithi183 ceph-mon[33277]: 5.5 scrub ok 2024-09-21T12:33:01.520 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:01.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:01 smithi114 ceph-mon[40867]: pgmap v654: 178 pgs: 178 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 93 MiB/s wr, 1.50k op/s 2024-09-21T12:33:01.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:01 smithi114 ceph-mon[40867]: 5.5 scrub starts 2024-09-21T12:33:01.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:01 smithi114 ceph-mon[40867]: 5.5 scrub ok 2024-09-21T12:33:01.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:01 smithi114 ceph-mon[30324]: pgmap v654: 178 pgs: 178 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 93 MiB/s wr, 1.50k op/s 2024-09-21T12:33:01.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:01 smithi114 ceph-mon[30324]: 5.5 scrub starts 2024-09-21T12:33:01.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:01 smithi114 ceph-mon[30324]: 5.5 scrub ok 2024-09-21T12:33:01.568 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:01.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:01.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:01.649 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 25 16 44027 44011 110.012 66.75 0.0111968 0.00908504 2024-09-21T12:33:01.658 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:33:01.670 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:01.705 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:01.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:01.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:01.807 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:01.841 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:01.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:01.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:01.943 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:01.987 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:01.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:01.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:02.089 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:02.123 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:02.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:02.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:02.150 INFO:teuthology.orchestra.run.smithi114.stderr:no valid command found; 10 closest matches: 2024-09-21T12:33:02.150 INFO:teuthology.orchestra.run.smithi114.stderr:0 2024-09-21T12:33:02.150 INFO:teuthology.orchestra.run.smithi114.stderr:1 2024-09-21T12:33:02.150 INFO:teuthology.orchestra.run.smithi114.stderr:2 2024-09-21T12:33:02.150 INFO:teuthology.orchestra.run.smithi114.stderr:abort 2024-09-21T12:33:02.150 INFO:teuthology.orchestra.run.smithi114.stderr:assert 2024-09-21T12:33:02.150 INFO:teuthology.orchestra.run.smithi114.stderr:bench [] [] [] [] 2024-09-21T12:33:02.150 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs debug_inject_read_zeros 2024-09-21T12:33:02.150 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs files list 2024-09-21T12:33:02.150 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs stats 2024-09-21T12:33:02.150 INFO:teuthology.orchestra.run.smithi114.stderr:bluestore allocator dump block 2024-09-21T12:33:02.151 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: invalid command 2024-09-21T12:33:02.225 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:02.264 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:02.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:02.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:02.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:02 smithi114 ceph-mon[30324]: 1.5 scrub starts 2024-09-21T12:33:02.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:02 smithi114 ceph-mon[30324]: 1.5 scrub ok 2024-09-21T12:33:02.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:02 smithi114 ceph-mon[30324]: 1.11 scrub starts 2024-09-21T12:33:02.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:02 smithi114 ceph-mon[30324]: 1.11 scrub ok 2024-09-21T12:33:02.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:02 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:33:02.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:02 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:33:02.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:02 smithi114 ceph-mon[30324]: osdmap e360: 8 total, 8 up, 8 in 2024-09-21T12:33:02.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:02 smithi114 ceph-mon[40867]: 1.5 scrub starts 2024-09-21T12:33:02.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:02 smithi114 ceph-mon[40867]: 1.5 scrub ok 2024-09-21T12:33:02.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:02 smithi114 ceph-mon[40867]: 1.11 scrub starts 2024-09-21T12:33:02.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:02 smithi114 ceph-mon[40867]: 1.11 scrub ok 2024-09-21T12:33:02.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:02 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:33:02.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:02 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:33:02.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:02 smithi114 ceph-mon[40867]: osdmap e360: 8 total, 8 up, 8 in 2024-09-21T12:33:02.293 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:33:02.293 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:33:02.293 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.293 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.15924.0:45196 10.e e360/352 10:7526ad9d:::benchmark_data_smithi114_161690_object45195:head v 360'2829, mlcod=360'2827)", 2024-09-21T12:33:02.293 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:33:02.284922+0000", 2024-09-21T12:33:02.293 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0058011579999999998, 2024-09-21T12:33:02.293 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0058300330000000001, 2024-09-21T12:33:02.293 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:02.293 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:02.294 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:33:02.294 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:02.294 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.294 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:02.294 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.284922+0000", 2024-09-21T12:33:02.294 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:02.294 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.294 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.294 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:02.294 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.284922+0000", 2024-09-21T12:33:02.294 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:02.294 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.295 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.295 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:02.295 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.284923+0000", 2024-09-21T12:33:02.295 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7999999999999995e-07 2024-09-21T12:33:02.295 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.295 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.295 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:02.295 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.284948+0000", 2024-09-21T12:33:02.295 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5320000000000002e-05 2024-09-21T12:33:02.295 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.295 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.295 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:02.296 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.284960+0000", 2024-09-21T12:33:02.296 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2135e-05 2024-09-21T12:33:02.296 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.296 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.296 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:02.296 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.284967+0000", 2024-09-21T12:33:02.296 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0690000000000004e-06 2024-09-21T12:33:02.296 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.296 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.296 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:02.296 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.285023+0000", 2024-09-21T12:33:02.296 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5721999999999999e-05 2024-09-21T12:33:02.296 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.297 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.297 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:02.297 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.285042+0000", 2024-09-21T12:33:02.297 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8811999999999999e-05 2024-09-21T12:33:02.297 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:02.297 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:02.297 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:02.297 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.297 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.297 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:45195 10.f 10:ff3478b7:::benchmark_data_smithi114_161690_object45194:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-09-21T12:33:02.297 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:33:02.284910+0000", 2024-09-21T12:33:02.297 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0058126760000000001, 2024-09-21T12:33:02.297 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0059202489999999998, 2024-09-21T12:33:02.297 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:02.297 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:02.298 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:33:02.298 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:02.298 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:33:02.298 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:33:02.298 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 45195 2024-09-21T12:33:02.298 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.298 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:02.298 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.298 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:02.298 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.284910+0000", 2024-09-21T12:33:02.298 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:02.298 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.298 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.298 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:02.299 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.284910+0000", 2024-09-21T12:33:02.299 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:02.299 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.299 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.299 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:02.299 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.284912+0000", 2024-09-21T12:33:02.299 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.699e-06 2024-09-21T12:33:02.299 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.299 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.299 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:02.299 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.284986+0000", 2024-09-21T12:33:02.299 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3782999999999995e-05 2024-09-21T12:33:02.299 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.300 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.300 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:02.300 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.284997+0000", 2024-09-21T12:33:02.300 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1191e-05 2024-09-21T12:33:02.300 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.300 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.300 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:02.300 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.285005+0000", 2024-09-21T12:33:02.300 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7100000000000007e-06 2024-09-21T12:33:02.300 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.300 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.300 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:02.301 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.285074+0000", 2024-09-21T12:33:02.301 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8912999999999996e-05 2024-09-21T12:33:02.301 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.301 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.301 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:02.301 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.285214+0000", 2024-09-21T12:33:02.301 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013988 2024-09-21T12:33:02.301 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.301 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.301 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,3", 2024-09-21T12:33:02.301 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.285353+0000", 2024-09-21T12:33:02.301 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0001392 2024-09-21T12:33:02.301 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.301 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.302 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:02.302 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.289720+0000", 2024-09-21T12:33:02.302 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0043667319999999999 2024-09-21T12:33:02.302 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:02.302 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:02.302 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:02.302 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.302 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.302 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:45197 10.0 10:0156e819:::benchmark_data_smithi114_161690_object45196:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-09-21T12:33:02.302 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:33:02.285025+0000", 2024-09-21T12:33:02.302 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0056983930000000004, 2024-09-21T12:33:02.302 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0058840899999999998, 2024-09-21T12:33:02.302 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:02.303 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:02.303 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:33:02.303 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:02.303 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:33:02.303 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:33:02.303 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 45197 2024-09-21T12:33:02.303 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.303 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:02.303 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.303 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:02.303 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.285025+0000", 2024-09-21T12:33:02.303 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:02.303 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.303 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.304 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:02.304 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.285025+0000", 2024-09-21T12:33:02.304 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:02.304 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.304 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.304 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:02.304 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.285025+0000", 2024-09-21T12:33:02.304 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3799999999999997e-07 2024-09-21T12:33:02.304 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.304 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.304 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:02.304 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.285117+0000", 2024-09-21T12:33:02.304 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2020000000000003e-05 2024-09-21T12:33:02.304 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.305 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.305 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:02.305 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.285127+0000", 2024-09-21T12:33:02.305 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.268e-06 2024-09-21T12:33:02.305 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.305 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.305 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:02.305 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.285132+0000", 2024-09-21T12:33:02.305 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6389999999999997e-06 2024-09-21T12:33:02.305 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.305 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.305 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:02.305 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.285481+0000", 2024-09-21T12:33:02.305 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00034876500000000001 2024-09-21T12:33:02.306 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.306 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.306 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:02.306 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.285585+0000", 2024-09-21T12:33:02.306 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010367 2024-09-21T12:33:02.306 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.306 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.306 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,4", 2024-09-21T12:33:02.306 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.285696+0000", 2024-09-21T12:33:02.306 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000111455 2024-09-21T12:33:02.306 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.306 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.306 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:02.307 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.289193+0000", 2024-09-21T12:33:02.307 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0034965959999999998 2024-09-21T12:33:02.307 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:02.307 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:02.307 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:02.307 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.307 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.308 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:45199 10.f 10:fd26e470:::benchmark_data_smithi114_161690_object45198:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-09-21T12:33:02.308 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:33:02.285148+0000", 2024-09-21T12:33:02.308 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0055751079999999996, 2024-09-21T12:33:02.308 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0058324190000000001, 2024-09-21T12:33:02.308 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:02.308 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:02.308 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:33:02.308 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:02.308 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:33:02.308 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:33:02.308 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 45199 2024-09-21T12:33:02.308 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.308 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:02.309 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.309 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:02.309 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.285148+0000", 2024-09-21T12:33:02.309 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:02.309 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.309 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.309 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:02.309 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.285148+0000", 2024-09-21T12:33:02.309 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:02.309 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.309 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.309 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:02.309 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.285149+0000", 2024-09-21T12:33:02.309 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4499999999999997e-07 2024-09-21T12:33:02.310 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.310 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.310 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:02.310 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.285203+0000", 2024-09-21T12:33:02.310 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4107000000000002e-05 2024-09-21T12:33:02.310 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.310 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.310 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:02.310 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.285209+0000", 2024-09-21T12:33:02.310 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7800000000000003e-06 2024-09-21T12:33:02.310 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.310 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.310 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:02.310 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.285215+0000", 2024-09-21T12:33:02.311 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8459999999999996e-06 2024-09-21T12:33:02.311 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.311 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.311 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:02.311 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.287535+0000", 2024-09-21T12:33:02.311 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0023195569999999999 2024-09-21T12:33:02.311 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.311 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.311 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:02.311 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.287655+0000", 2024-09-21T12:33:02.311 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000120234 2024-09-21T12:33:02.311 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.311 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.311 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,3", 2024-09-21T12:33:02.312 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.287774+0000", 2024-09-21T12:33:02.312 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011872899999999999 2024-09-21T12:33:02.312 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:02.312 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:02.312 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:02.312 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.312 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.312 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.15924.0:45201 10.7 e360/344 10:e9124332:::benchmark_data_smithi114_161690_object45200:head v 360'2770, mlcod=360'2768)", 2024-09-21T12:33:02.312 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:33:02.285966+0000", 2024-09-21T12:33:02.312 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0047571690000000003, 2024-09-21T12:33:02.312 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0050768860000000001, 2024-09-21T12:33:02.312 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:02.312 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:02.312 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:33:02.313 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:02.313 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.313 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:02.313 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.285966+0000", 2024-09-21T12:33:02.313 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:02.313 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.313 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.313 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:02.313 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.285966+0000", 2024-09-21T12:33:02.313 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:02.313 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.313 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.313 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:02.313 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.285966+0000", 2024-09-21T12:33:02.314 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7500000000000001e-07 2024-09-21T12:33:02.314 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.314 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.314 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:02.314 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.285996+0000", 2024-09-21T12:33:02.314 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9879999999999999e-05 2024-09-21T12:33:02.314 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.314 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.314 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:02.314 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.286003+0000", 2024-09-21T12:33:02.314 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2849999999999997e-06 2024-09-21T12:33:02.314 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.314 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.314 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:02.315 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.286008+0000", 2024-09-21T12:33:02.315 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8930000000000004e-06 2024-09-21T12:33:02.315 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.315 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.315 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:02.315 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.287199+0000", 2024-09-21T12:33:02.315 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0011903020000000001 2024-09-21T12:33:02.315 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.315 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.315 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:02.315 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.287214+0000", 2024-09-21T12:33:02.315 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5211000000000001e-05 2024-09-21T12:33:02.315 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:02.315 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:02.315 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:02.316 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.316 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.316 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.15924.0:45200 10.d e360/353 10:b5d14551:::benchmark_data_smithi114_161690_object45199:head v 360'2878, mlcod=360'2874)", 2024-09-21T12:33:02.316 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:33:02.289253+0000", 2024-09-21T12:33:02.316 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0014705370000000001, 2024-09-21T12:33:02.316 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0018584960000000001, 2024-09-21T12:33:02.316 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:02.316 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:02.316 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:33:02.316 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:02.316 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.316 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:02.316 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.289253+0000", 2024-09-21T12:33:02.316 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:02.316 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.316 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.317 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:02.317 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.289253+0000", 2024-09-21T12:33:02.317 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:02.317 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.317 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.317 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:02.317 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.289253+0000", 2024-09-21T12:33:02.317 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5000000000000002e-07 2024-09-21T12:33:02.317 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.317 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.317 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:02.317 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.289277+0000", 2024-09-21T12:33:02.317 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3339000000000001e-05 2024-09-21T12:33:02.317 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.317 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.317 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:02.318 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.289288+0000", 2024-09-21T12:33:02.318 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.181e-05 2024-09-21T12:33:02.318 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.318 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.318 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:02.318 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.289296+0000", 2024-09-21T12:33:02.318 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1269999999999997e-06 2024-09-21T12:33:02.318 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.318 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.318 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:02.318 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.289377+0000", 2024-09-21T12:33:02.318 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1295e-05 2024-09-21T12:33:02.318 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.318 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.318 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:02.319 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.289394+0000", 2024-09-21T12:33:02.319 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7073000000000001e-05 2024-09-21T12:33:02.319 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:02.319 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:02.319 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:02.319 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.319 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.319 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:45210 10.0 10:0e4182e5:::benchmark_data_smithi114_161690_object45209:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-09-21T12:33:02.319 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:33:02.290642+0000", 2024-09-21T12:33:02.319 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 8.1102999999999999e-05, 2024-09-21T12:33:02.319 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00052882799999999996, 2024-09-21T12:33:02.319 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:02.319 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:02.319 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:33:02.319 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:02.320 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:33:02.320 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:33:02.320 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 45210 2024-09-21T12:33:02.320 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.320 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:02.320 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.320 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:02.320 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.290642+0000", 2024-09-21T12:33:02.320 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:02.320 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.320 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.320 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:02.320 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.290642+0000", 2024-09-21T12:33:02.320 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:02.320 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.320 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.321 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:02.321 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.290643+0000", 2024-09-21T12:33:02.321 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.362e-06 2024-09-21T12:33:02.321 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.321 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.321 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:02.321 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.290692+0000", 2024-09-21T12:33:02.321 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8131999999999999e-05 2024-09-21T12:33:02.321 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.321 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.321 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:02.321 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.290703+0000", 2024-09-21T12:33:02.321 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1375e-05 2024-09-21T12:33:02.321 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.321 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.321 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:02.322 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.290710+0000", 2024-09-21T12:33:02.322 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9020000000000002e-06 2024-09-21T12:33:02.322 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.322 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.322 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:02.322 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.290769+0000", 2024-09-21T12:33:02.322 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9110000000000002e-05 2024-09-21T12:33:02.322 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.322 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.322 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:02.322 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.290907+0000", 2024-09-21T12:33:02.322 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013825600000000001 2024-09-21T12:33:02.322 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:02.322 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:02.322 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,4", 2024-09-21T12:33:02.323 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:02.291033+0000", 2024-09-21T12:33:02.323 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000125919 2024-09-21T12:33:02.323 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:02.323 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:02.323 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:02.323 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:02.323 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-21T12:33:02.323 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 7 2024-09-21T12:33:02.323 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:33:02.323 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:02 smithi183 ceph-mon[33277]: 1.5 scrub starts 2024-09-21T12:33:02.324 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:02 smithi183 ceph-mon[33277]: 1.5 scrub ok 2024-09-21T12:33:02.324 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:02 smithi183 ceph-mon[33277]: 1.11 scrub starts 2024-09-21T12:33:02.324 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:02 smithi183 ceph-mon[33277]: 1.11 scrub ok 2024-09-21T12:33:02.324 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:02 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:33:02.324 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:02 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:33:02.324 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:02 smithi183 ceph-mon[33277]: osdmap e360: 8 total, 8 up, 8 in 2024-09-21T12:33:02.366 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:02.385 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:02.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:02.404 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:02.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:02.505 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:02.544 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:02.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:02.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:02.646 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:02.649 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 26 16 45877 45861 110.228 115.625 0.0190298 0.00906702 2024-09-21T12:33:02.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:02.677 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:02.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:02.778 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:02.815 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:02.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:02.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:02.858 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:33:02 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:33:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:33:02.917 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:02.955 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:02.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:02.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:03.057 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:03.091 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:03.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:03.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:03.158 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T12:33:03.193 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:03.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:03.227 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:03.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:03.313 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T12:33:03.313 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T12:33:03.314 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.2 'trim stale osdmaps' 2024-09-21T12:33:03.329 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:03.372 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:03.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:03.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:03.462 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:03 smithi114 ceph-mon[30324]: 3.9 scrub starts 2024-09-21T12:33:03.462 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:03 smithi114 ceph-mon[30324]: 3.9 scrub ok 2024-09-21T12:33:03.462 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:03 smithi114 ceph-mon[30324]: 4.1c scrub starts 2024-09-21T12:33:03.462 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:03 smithi114 ceph-mon[30324]: 4.1c scrub ok 2024-09-21T12:33:03.462 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:03 smithi114 ceph-mon[30324]: pgmap v655: 178 pgs: 178 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.5 KiB/s rd, 115 MiB/s wr, 1.84k op/s 2024-09-21T12:33:03.462 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:03 smithi114 ceph-mon[30324]: 3.a scrub starts 2024-09-21T12:33:03.463 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:03 smithi114 ceph-mon[30324]: 4.8 scrub starts 2024-09-21T12:33:03.463 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:03 smithi114 ceph-mon[30324]: 4.8 scrub ok 2024-09-21T12:33:03.463 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:03 smithi114 ceph-mon[30324]: 4.12 scrub starts 2024-09-21T12:33:03.463 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:03 smithi114 ceph-mon[30324]: 4.12 scrub ok 2024-09-21T12:33:03.463 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:03 smithi114 ceph-mon[40867]: 3.9 scrub starts 2024-09-21T12:33:03.463 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:03 smithi114 ceph-mon[40867]: 3.9 scrub ok 2024-09-21T12:33:03.463 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:03 smithi114 ceph-mon[40867]: 4.1c scrub starts 2024-09-21T12:33:03.463 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:03 smithi114 ceph-mon[40867]: 4.1c scrub ok 2024-09-21T12:33:03.463 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:03 smithi114 ceph-mon[40867]: pgmap v655: 178 pgs: 178 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.5 KiB/s rd, 115 MiB/s wr, 1.84k op/s 2024-09-21T12:33:03.463 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:03 smithi114 ceph-mon[40867]: 3.a scrub starts 2024-09-21T12:33:03.463 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:03 smithi114 ceph-mon[40867]: 4.8 scrub starts 2024-09-21T12:33:03.463 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:03 smithi114 ceph-mon[40867]: 4.8 scrub ok 2024-09-21T12:33:03.464 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:03 smithi114 ceph-mon[40867]: 4.12 scrub starts 2024-09-21T12:33:03.464 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:03 smithi114 ceph-mon[40867]: 4.12 scrub ok 2024-09-21T12:33:03.473 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:03.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:03 smithi183 ceph-mon[33277]: 3.9 scrub starts 2024-09-21T12:33:03.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:03 smithi183 ceph-mon[33277]: 3.9 scrub ok 2024-09-21T12:33:03.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:03 smithi183 ceph-mon[33277]: 4.1c scrub starts 2024-09-21T12:33:03.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:03 smithi183 ceph-mon[33277]: 4.1c scrub ok 2024-09-21T12:33:03.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:03 smithi183 ceph-mon[33277]: pgmap v655: 178 pgs: 178 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.5 KiB/s rd, 115 MiB/s wr, 1.84k op/s 2024-09-21T12:33:03.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:03 smithi183 ceph-mon[33277]: 3.a scrub starts 2024-09-21T12:33:03.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:03 smithi183 ceph-mon[33277]: 4.8 scrub starts 2024-09-21T12:33:03.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:03 smithi183 ceph-mon[33277]: 4.8 scrub ok 2024-09-21T12:33:03.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:03 smithi183 ceph-mon[33277]: 4.12 scrub starts 2024-09-21T12:33:03.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:03 smithi183 ceph-mon[33277]: 4.12 scrub ok 2024-09-21T12:33:03.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:03.521 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:03.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:03.557 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:33:03.622 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:03.653 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 27 16 47592 47576 110.109 107.188 0.00709477 0.00907624 2024-09-21T12:33:03.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:03.669 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:03.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:03.680 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:33:03.680 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:33:03.715 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:33:03.715 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:33:03.747 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:33:03.748 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:33:03.771 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:03.780 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:33:03.780 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:33:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:03.804 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:03.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:03.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:03.815 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:33:03.815 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:33:03.848 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:33:03.848 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:33:03.882 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:33:03.882 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:33:03.905 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:03.917 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:33:03.917 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:33:03.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:03.946 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:03.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:04.016 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T12:33:04.047 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:04.097 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:04.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:04.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:04.199 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:04.243 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:04.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:04.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:04.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:04 smithi183 ceph-mon[33277]: 3.a scrub ok 2024-09-21T12:33:04.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:04 smithi183 ceph-mon[33277]: 4.10 scrub starts 2024-09-21T12:33:04.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:04 smithi183 ceph-mon[33277]: 4.10 scrub ok 2024-09-21T12:33:04.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:04 smithi183 ceph-mon[33277]: 4.1a scrub starts 2024-09-21T12:33:04.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:04 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3827079684' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:33:04.344 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:04.353 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:33:04.353 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:33:04.384 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:04.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:04.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:04.486 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:04.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:04 smithi114 ceph-mon[40867]: 3.a scrub ok 2024-09-21T12:33:04.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:04 smithi114 ceph-mon[40867]: 4.10 scrub starts 2024-09-21T12:33:04.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:04 smithi114 ceph-mon[40867]: 4.10 scrub ok 2024-09-21T12:33:04.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:04 smithi114 ceph-mon[40867]: 4.1a scrub starts 2024-09-21T12:33:04.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:04 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3827079684' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:33:04.525 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:04.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:04 smithi114 ceph-mon[30324]: 3.a scrub ok 2024-09-21T12:33:04.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:04 smithi114 ceph-mon[30324]: 4.10 scrub starts 2024-09-21T12:33:04.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:04 smithi114 ceph-mon[30324]: 4.10 scrub ok 2024-09-21T12:33:04.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:04 smithi114 ceph-mon[30324]: 4.1a scrub starts 2024-09-21T12:33:04.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:04 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3827079684' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:33:04.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:04.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:04.627 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:04.651 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 28 16 49047 49031 109.424 90.9375 0.0129221 0.00913376 2024-09-21T12:33:04.660 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:04.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:04.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:04.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:04.796 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:04.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:04.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:04.898 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:04.933 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:04.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:04.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:05.022 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T12:33:05.035 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:05.069 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:05.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:05.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:05.172 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:05.192 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:05.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:05.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:05.294 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:05.301 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:33:05.315 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:05.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:05.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:05.416 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:05.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:05 smithi183 ceph-mon[33277]: 4.1a scrub ok 2024-09-21T12:33:05.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:05 smithi183 ceph-mon[33277]: pgmap v657: 178 pgs: 178 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 752 B/s rd, 119 MiB/s wr, 1.91k op/s 2024-09-21T12:33:05.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:05 smithi183 ceph-mon[33277]: 1.b scrub starts 2024-09-21T12:33:05.422 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:05 smithi183 ceph-mon[33277]: 1.b scrub ok 2024-09-21T12:33:05.438 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:05.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:05.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:05.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:05 smithi114 ceph-mon[40867]: 4.1a scrub ok 2024-09-21T12:33:05.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:05 smithi114 ceph-mon[40867]: pgmap v657: 178 pgs: 178 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 752 B/s rd, 119 MiB/s wr, 1.91k op/s 2024-09-21T12:33:05.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:05 smithi114 ceph-mon[40867]: 1.b scrub starts 2024-09-21T12:33:05.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:05 smithi114 ceph-mon[40867]: 1.b scrub ok 2024-09-21T12:33:05.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:05 smithi114 ceph-mon[30324]: 4.1a scrub ok 2024-09-21T12:33:05.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:05 smithi114 ceph-mon[30324]: pgmap v657: 178 pgs: 178 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 752 B/s rd, 119 MiB/s wr, 1.91k op/s 2024-09-21T12:33:05.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:05 smithi114 ceph-mon[30324]: 1.b scrub starts 2024-09-21T12:33:05.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:05 smithi114 ceph-mon[30324]: 1.b scrub ok 2024-09-21T12:33:05.539 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:05.579 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:05.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:05.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:05.651 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 29 16 49994 49978 107.692 59.1875 0.00540088 0.00928147 2024-09-21T12:33:05.681 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:05.716 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:05.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:05.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:05.818 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:05.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:05.843 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:05.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:05.877 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:33:05.877 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:33:05.944 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:05.965 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:05.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:05.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:06.067 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:06.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:06.095 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:06.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:06.196 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:06.223 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:06.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:06.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:06.326 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:06.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:06.359 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:06.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:06.460 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:06.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:06 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:33:06.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:06 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:33:06.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:06 smithi183 ceph-mon[33277]: osdmap e361: 8 total, 8 up, 8 in 2024-09-21T12:33:06.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:06 smithi183 ceph-mon[33277]: 6.1 scrub starts 2024-09-21T12:33:06.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:06 smithi183 ceph-mon[33277]: 6.1 scrub ok 2024-09-21T12:33:06.494 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:06.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:06.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:06.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:06 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:33:06.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:06 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:33:06.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:06 smithi114 ceph-mon[30324]: osdmap e361: 8 total, 8 up, 8 in 2024-09-21T12:33:06.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:06 smithi114 ceph-mon[30324]: 6.1 scrub starts 2024-09-21T12:33:06.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:06 smithi114 ceph-mon[30324]: 6.1 scrub ok 2024-09-21T12:33:06.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:06 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:33:06.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:06 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:33:06.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:06 smithi114 ceph-mon[40867]: osdmap e361: 8 total, 8 up, 8 in 2024-09-21T12:33:06.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:06 smithi114 ceph-mon[40867]: 6.1 scrub starts 2024-09-21T12:33:06.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:06 smithi114 ceph-mon[40867]: 6.1 scrub ok 2024-09-21T12:33:06.550 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T12:33:06.596 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:06.629 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:06.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:06.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:06.651 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 30 16 51135 51119 106.479 71.3125 0.0132354 0.00938628 2024-09-21T12:33:06.731 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:06.763 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:06.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:06.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:06.865 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:06.897 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:06.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:06.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:06.999 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:07.027 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:33:07.063 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:07.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:07.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:07.165 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:07.200 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:07.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:07.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:07.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:07 smithi183 ceph-mon[33277]: 6.17 scrub starts 2024-09-21T12:33:07.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:07 smithi183 ceph-mon[33277]: 6.17 scrub ok 2024-09-21T12:33:07.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:07 smithi183 ceph-mon[33277]: 1.14 scrub starts 2024-09-21T12:33:07.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:07 smithi183 ceph-mon[33277]: 1.14 scrub ok 2024-09-21T12:33:07.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:07 smithi183 ceph-mon[33277]: 3.8 scrub starts 2024-09-21T12:33:07.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:07 smithi183 ceph-mon[33277]: 3.8 scrub ok 2024-09-21T12:33:07.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:07 smithi183 ceph-mon[33277]: pgmap v659: 178 pgs: 178 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 763 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-21T12:33:07.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:07 smithi183 ceph-mon[33277]: 1.0 scrub starts 2024-09-21T12:33:07.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:07 smithi183 ceph-mon[33277]: 1.0 scrub ok 2024-09-21T12:33:07.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:07 smithi183 ceph-mon[33277]: osdmap e362: 8 total, 8 up, 8 in 2024-09-21T12:33:07.302 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:07.344 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:07 smithi114 ceph-mon[30324]: 6.17 scrub starts 2024-09-21T12:33:07.344 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:07 smithi114 ceph-mon[30324]: 6.17 scrub ok 2024-09-21T12:33:07.344 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:07 smithi114 ceph-mon[30324]: 1.14 scrub starts 2024-09-21T12:33:07.344 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:07 smithi114 ceph-mon[30324]: 1.14 scrub ok 2024-09-21T12:33:07.344 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:07 smithi114 ceph-mon[30324]: 3.8 scrub starts 2024-09-21T12:33:07.344 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:07 smithi114 ceph-mon[30324]: 3.8 scrub ok 2024-09-21T12:33:07.344 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:07 smithi114 ceph-mon[30324]: pgmap v659: 178 pgs: 178 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 763 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-21T12:33:07.344 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:07 smithi114 ceph-mon[30324]: 1.0 scrub starts 2024-09-21T12:33:07.345 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:07 smithi114 ceph-mon[30324]: 1.0 scrub ok 2024-09-21T12:33:07.345 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:07 smithi114 ceph-mon[30324]: osdmap e362: 8 total, 8 up, 8 in 2024-09-21T12:33:07.345 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:07 smithi114 ceph-mon[40867]: 6.17 scrub starts 2024-09-21T12:33:07.345 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:07 smithi114 ceph-mon[40867]: 6.17 scrub ok 2024-09-21T12:33:07.345 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:07 smithi114 ceph-mon[40867]: 1.14 scrub starts 2024-09-21T12:33:07.345 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:07 smithi114 ceph-mon[40867]: 1.14 scrub ok 2024-09-21T12:33:07.345 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:07 smithi114 ceph-mon[40867]: 3.8 scrub starts 2024-09-21T12:33:07.345 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:07 smithi114 ceph-mon[40867]: 3.8 scrub ok 2024-09-21T12:33:07.345 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:07 smithi114 ceph-mon[40867]: pgmap v659: 178 pgs: 178 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 763 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-21T12:33:07.345 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:07 smithi114 ceph-mon[40867]: 1.0 scrub starts 2024-09-21T12:33:07.345 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:07 smithi114 ceph-mon[40867]: 1.0 scrub ok 2024-09-21T12:33:07.345 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:07 smithi114 ceph-mon[40867]: osdmap e362: 8 total, 8 up, 8 in 2024-09-21T12:33:07.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:07.346 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:07.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:07.430 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:07.435 INFO:teuthology.orchestra.run.smithi114.stderr:no valid command found; 10 closest matches: 2024-09-21T12:33:07.435 INFO:teuthology.orchestra.run.smithi114.stderr:0 2024-09-21T12:33:07.435 INFO:teuthology.orchestra.run.smithi114.stderr:1 2024-09-21T12:33:07.435 INFO:teuthology.orchestra.run.smithi114.stderr:2 2024-09-21T12:33:07.435 INFO:teuthology.orchestra.run.smithi114.stderr:abort 2024-09-21T12:33:07.435 INFO:teuthology.orchestra.run.smithi114.stderr:assert 2024-09-21T12:33:07.435 INFO:teuthology.orchestra.run.smithi114.stderr:bench [] [] [] [] 2024-09-21T12:33:07.435 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs debug_inject_read_zeros 2024-09-21T12:33:07.435 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs files list 2024-09-21T12:33:07.435 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs stats 2024-09-21T12:33:07.435 INFO:teuthology.orchestra.run.smithi114.stderr:bluestore allocator dump block 2024-09-21T12:33:07.435 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: invalid command 2024-09-21T12:33:07.447 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:07.500 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:07.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:07.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:07.603 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:07.621 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:33:07 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:33:07.640 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:07.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:07.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:07.652 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 31 15 52580 52565 105.96 90.375 0.00821406 0.00943176 2024-09-21T12:33:07.676 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:33:07.676 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:33:07.742 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:07.775 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:07.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:07.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:07.876 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:07.923 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:07.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:07.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:08.024 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:08.059 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:08.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:08.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:08.143 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T12:33:08.143 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T12:33:08.144 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.3 'trim stale osdmaps' 2024-09-21T12:33:08.161 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:08.234 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:08.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:08.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:08.336 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:08.343 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:08 smithi114 ceph-mon[30324]: 6.e scrub starts 2024-09-21T12:33:08.343 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:08 smithi114 ceph-mon[30324]: 6.e scrub ok 2024-09-21T12:33:08.343 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:08 smithi114 ceph-mon[30324]: 3.1e scrub starts 2024-09-21T12:33:08.343 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:08 smithi114 ceph-mon[30324]: 3.1e scrub ok 2024-09-21T12:33:08.343 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:08 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:08.343 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:08 smithi114 ceph-mon[30324]: 1.16 scrub starts 2024-09-21T12:33:08.343 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:08 smithi114 ceph-mon[30324]: 1.16 scrub ok 2024-09-21T12:33:08.344 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:08 smithi114 ceph-mon[40867]: 6.e scrub starts 2024-09-21T12:33:08.344 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:08 smithi114 ceph-mon[40867]: 6.e scrub ok 2024-09-21T12:33:08.344 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:08 smithi114 ceph-mon[40867]: 3.1e scrub starts 2024-09-21T12:33:08.344 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:08 smithi114 ceph-mon[40867]: 3.1e scrub ok 2024-09-21T12:33:08.344 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:08 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:08.344 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:08 smithi114 ceph-mon[40867]: 1.16 scrub starts 2024-09-21T12:33:08.344 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:08 smithi114 ceph-mon[40867]: 1.16 scrub ok 2024-09-21T12:33:08.382 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:08 smithi183 ceph-mon[33277]: 6.e scrub starts 2024-09-21T12:33:08.383 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:08 smithi183 ceph-mon[33277]: 6.e scrub ok 2024-09-21T12:33:08.383 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:08 smithi183 ceph-mon[33277]: 3.1e scrub starts 2024-09-21T12:33:08.383 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:08 smithi183 ceph-mon[33277]: 3.1e scrub ok 2024-09-21T12:33:08.383 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:08 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:08.383 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:08 smithi183 ceph-mon[33277]: 1.16 scrub starts 2024-09-21T12:33:08.383 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:08 smithi183 ceph-mon[33277]: 1.16 scrub ok 2024-09-21T12:33:08.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:08.385 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:08.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:08.436 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T12:33:08.486 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:08.521 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:08.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:08.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:08.622 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:08.653 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:08.654 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:08.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:08.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:08.656 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 32 16 54132 54116 105.665 96.9375 0.00580968 0.00945795 2024-09-21T12:33:08.756 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:08.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:08.802 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:08.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:08.829 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:33:08.903 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:08.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:08.941 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:08.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:09.042 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:09.077 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:09.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:09.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:09.123 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:33:09.180 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:09.210 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-21T12:33:09.236 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:09.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:09.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:09.338 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:09.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:09.378 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:09.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:09.479 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:09.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:09 smithi183 ceph-mon[33277]: pgmap v661: 178 pgs: 178 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-21T12:33:09.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:09 smithi183 ceph-mon[33277]: 3.d scrub starts 2024-09-21T12:33:09.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:09 smithi183 ceph-mon[33277]: 3.d scrub ok 2024-09-21T12:33:09.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:09 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/936224212' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:33:09.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:09 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:33:09.524 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:09.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:09 smithi114 ceph-mon[40867]: pgmap v661: 178 pgs: 178 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-21T12:33:09.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:09 smithi114 ceph-mon[40867]: 3.d scrub starts 2024-09-21T12:33:09.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:09 smithi114 ceph-mon[40867]: 3.d scrub ok 2024-09-21T12:33:09.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:09 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/936224212' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:33:09.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:09 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:33:09.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:09.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:09.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:09 smithi114 ceph-mon[30324]: pgmap v661: 178 pgs: 178 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-21T12:33:09.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:09 smithi114 ceph-mon[30324]: 3.d scrub starts 2024-09-21T12:33:09.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:09 smithi114 ceph-mon[30324]: 3.d scrub ok 2024-09-21T12:33:09.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:09 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/936224212' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:33:09.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:09 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:33:09.626 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:09.655 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 33 16 55602 55586 105.247 91.875 0.00734099 0.00949647 2024-09-21T12:33:09.660 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:09.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:09.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:09.762 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:09.787 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:09.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:09.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:09.889 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:09.928 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:09.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:09.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:10.029 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:10.057 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:10.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:10.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:10.089 INFO:teuthology.orchestra.run.smithi114.stderr:no valid command found; 10 closest matches: 2024-09-21T12:33:10.089 INFO:teuthology.orchestra.run.smithi114.stderr:0 2024-09-21T12:33:10.089 INFO:teuthology.orchestra.run.smithi114.stderr:1 2024-09-21T12:33:10.089 INFO:teuthology.orchestra.run.smithi114.stderr:2 2024-09-21T12:33:10.089 INFO:teuthology.orchestra.run.smithi114.stderr:abort 2024-09-21T12:33:10.089 INFO:teuthology.orchestra.run.smithi114.stderr:assert 2024-09-21T12:33:10.089 INFO:teuthology.orchestra.run.smithi114.stderr:bench [] [] [] [] 2024-09-21T12:33:10.090 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs debug_inject_read_zeros 2024-09-21T12:33:10.090 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs files list 2024-09-21T12:33:10.090 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs stats 2024-09-21T12:33:10.090 INFO:teuthology.orchestra.run.smithi114.stderr:bluestore allocator dump block 2024-09-21T12:33:10.090 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: invalid command 2024-09-21T12:33:10.158 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:10.189 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:10.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:10.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:10.291 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:10.320 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:10.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:10.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:10.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:10.462 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:10.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:10.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:10.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:10 smithi183 ceph-mon[33277]: 4.d scrub starts 2024-09-21T12:33:10.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:10 smithi183 ceph-mon[33277]: 4.d scrub ok 2024-09-21T12:33:10.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:10 smithi183 ceph-mon[33277]: 6.13 scrub starts 2024-09-21T12:33:10.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:10 smithi183 ceph-mon[33277]: 6.13 scrub ok 2024-09-21T12:33:10.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:10 smithi183 ceph-mon[33277]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:33:10.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:10 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:33:10.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:10 smithi183 ceph-mon[33277]: osdmap e363: 8 total, 8 up, 8 in 2024-09-21T12:33:10.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:10 smithi114 ceph-mon[30324]: 4.d scrub starts 2024-09-21T12:33:10.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:10 smithi114 ceph-mon[30324]: 4.d scrub ok 2024-09-21T12:33:10.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:10 smithi114 ceph-mon[30324]: 6.13 scrub starts 2024-09-21T12:33:10.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:10 smithi114 ceph-mon[30324]: 6.13 scrub ok 2024-09-21T12:33:10.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:10 smithi114 ceph-mon[30324]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:33:10.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:10 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:33:10.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:10 smithi114 ceph-mon[30324]: osdmap e363: 8 total, 8 up, 8 in 2024-09-21T12:33:10.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:10 smithi114 ceph-mon[40867]: 4.d scrub starts 2024-09-21T12:33:10.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:10 smithi114 ceph-mon[40867]: 4.d scrub ok 2024-09-21T12:33:10.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:10 smithi114 ceph-mon[40867]: 6.13 scrub starts 2024-09-21T12:33:10.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:10 smithi114 ceph-mon[40867]: 6.13 scrub ok 2024-09-21T12:33:10.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:10 smithi114 ceph-mon[40867]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:33:10.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:10 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:33:10.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:10 smithi114 ceph-mon[40867]: osdmap e363: 8 total, 8 up, 8 in 2024-09-21T12:33:10.564 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:10.600 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:10.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:10.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:10.655 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 34 15 57061 57046 104.835 91.25 0.00645163 0.00953347 2024-09-21T12:33:10.702 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:10.736 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:10.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:10.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:10.838 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:10.875 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:10.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:10.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:10.930 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:33:10.930 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:33:10.977 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:11.011 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:11.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:11.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:11.113 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:11.149 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:11.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:11.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:11.250 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:11.274 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:11.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:11.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:11.376 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:11.412 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:11.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:11.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:11.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:11 smithi183 ceph-mon[33277]: pgmap v663: 178 pgs: 178 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-21T12:33:11.514 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:11.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:11 smithi114 ceph-mon[40867]: pgmap v663: 178 pgs: 178 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-21T12:33:11.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:11 smithi114 ceph-mon[30324]: pgmap v663: 178 pgs: 178 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-21T12:33:11.538 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:11.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:11.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:11.566 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T12:33:11.566 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T12:33:11.566 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.5 'trim stale osdmaps' 2024-09-21T12:33:11.587 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:11.640 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:11.655 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 35 16 58539 58523 104.477 92.3125 0.00720535 0.00956642 2024-09-21T12:33:11.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:11.713 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:11.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:11.814 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:11.860 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:11.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:11.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:11.962 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:12.001 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:12.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:12.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:12.103 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:12.143 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:12.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:12.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:12.245 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:12.245 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:12.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:12.283 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:12.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:12.288 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:33:12.288 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:33:12.326 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:33:12.326 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:33:12.356 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:33:12.357 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:33:12.384 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:12.394 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:33:12.394 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:33:12.421 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:12.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:12.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:12.426 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:33:12.426 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:33:12.457 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:33:12.458 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:33:12.493 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:33:12.493 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:33:12.524 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:12.527 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:33:12.527 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:33:12.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:12.570 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:12.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:12.646 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T12:33:12.656 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 36 16 59940 59924 104.007 87.5625 0.0081804 0.00960994 2024-09-21T12:33:12.671 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:12.700 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:12.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:12.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:12.750 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:33:12 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:33:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:33:12.802 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:12.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:12.839 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:12.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:12.940 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:12.950 INFO:teuthology.orchestra.run.smithi114.stderr:no valid command found; 10 closest matches: 2024-09-21T12:33:12.950 INFO:teuthology.orchestra.run.smithi114.stderr:0 2024-09-21T12:33:12.950 INFO:teuthology.orchestra.run.smithi114.stderr:1 2024-09-21T12:33:12.950 INFO:teuthology.orchestra.run.smithi114.stderr:2 2024-09-21T12:33:12.950 INFO:teuthology.orchestra.run.smithi114.stderr:abort 2024-09-21T12:33:12.950 INFO:teuthology.orchestra.run.smithi114.stderr:assert 2024-09-21T12:33:12.950 INFO:teuthology.orchestra.run.smithi114.stderr:bench [] [] [] [] 2024-09-21T12:33:12.951 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs debug_inject_read_zeros 2024-09-21T12:33:12.951 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs files list 2024-09-21T12:33:12.951 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs stats 2024-09-21T12:33:12.951 INFO:teuthology.orchestra.run.smithi114.stderr:bluestore allocator dump block 2024-09-21T12:33:12.951 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: invalid command 2024-09-21T12:33:12.976 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:12.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:12.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:13.065 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:33:13.078 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:13.111 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:13.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:13.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:13.213 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:13.222 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T12:33:13.243 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:13.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:13.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:13.332 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:13 smithi114 ceph-mon[30324]: pgmap v664: 178 pgs: 178 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.5 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-21T12:33:13.332 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:13 smithi114 ceph-mon[30324]: osdmap e364: 8 total, 8 up, 8 in 2024-09-21T12:33:13.332 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:13 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/597197137' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:33:13.332 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:13 smithi114 ceph-mon[40867]: pgmap v664: 178 pgs: 178 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.5 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-21T12:33:13.332 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:13 smithi114 ceph-mon[40867]: osdmap e364: 8 total, 8 up, 8 in 2024-09-21T12:33:13.332 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:13 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/597197137' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:33:13.345 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:13.378 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:13.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:13.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:13.391 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:13 smithi183 ceph-mon[33277]: pgmap v664: 178 pgs: 178 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.5 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-09-21T12:33:13.391 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:13 smithi183 ceph-mon[33277]: osdmap e364: 8 total, 8 up, 8 in 2024-09-21T12:33:13.391 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:13 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/597197137' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:33:13.479 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:13.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:13.514 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:13.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:13.615 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:13.650 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:13.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:13.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:13.656 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 37 15 61390 61375 103.647 90.6875 0.0155371 0.00964339 2024-09-21T12:33:13.704 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:33:13.704 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-21T12:33:13.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-21T12:33:13.704 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:33:13.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.704 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14478.0:2784 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94620848900224 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:33:13.705 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.038339+0000", 2024-09-21T12:33:13.705 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 95.661118923000004, 2024-09-21T12:33:13.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.027622934, 2024-09-21T12:33:13.705 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:13.705 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:13.705 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:33:13.705 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:13.705 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14478", 2024-09-21T12:33:13.705 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:33:13.705 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 2784 2024-09-21T12:33:13.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.705 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:13.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:13.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.038339+0000", 2024-09-21T12:33:13.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:13.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.038339+0000", 2024-09-21T12:33:13.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:13.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.038340+0000", 2024-09-21T12:33:13.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.164e-06 2024-09-21T12:33:13.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:13.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.038345+0000", 2024-09-21T12:33:13.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7339999999999999e-06 2024-09-21T12:33:13.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:13.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.038359+0000", 2024-09-21T12:33:13.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4577e-05 2024-09-21T12:33:13.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:13.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.038368+0000", 2024-09-21T12:33:13.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3599999999999996e-06 2024-09-21T12:33:13.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.038422+0000", 2024-09-21T12:33:13.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3999000000000003e-05 2024-09-21T12:33:13.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op must wait for map", 2024-09-21T12:33:13.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.038427+0000", 2024-09-21T12:33:13.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.789e-06 2024-09-21T12:33:13.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.045355+0000", 2024-09-21T12:33:13.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0069278600000000001 2024-09-21T12:33:13.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:13.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.045364+0000", 2024-09-21T12:33:13.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8340000000000005e-06 2024-09-21T12:33:13.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.054340+0000", 2024-09-21T12:33:13.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0089757239999999 2024-09-21T12:33:13.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for degraded object", 2024-09-21T12:33:13.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.054602+0000", 2024-09-21T12:33:13.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00026209300000000003 2024-09-21T12:33:13.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.058190+0000", 2024-09-21T12:33:13.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0035881939999999998 2024-09-21T12:33:13.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.710 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:13.710 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.058240+0000", 2024-09-21T12:33:13.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0105999999999998e-05 2024-09-21T12:33:13.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.710 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:13.710 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.065962+0000", 2024-09-21T12:33:13.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0077215000000000001 2024-09-21T12:33:13.710 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.710 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:13.710 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.711 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:221955 9.d 9:baadbb95:::benchmark_data_smithi114_133214_object97921:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:33:13.711 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.080261+0000", 2024-09-21T12:33:13.711 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 95.619197170000007, 2024-09-21T12:33:13.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0418771250000001, 2024-09-21T12:33:13.711 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:13.711 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:13.711 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:13.711 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:13.711 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:13.711 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:13.711 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 221955 2024-09-21T12:33:13.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.712 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:13.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.712 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:13.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.080261+0000", 2024-09-21T12:33:13.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.712 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:13.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.080261+0000", 2024-09-21T12:33:13.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.712 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:13.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.080262+0000", 2024-09-21T12:33:13.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1230000000000001e-06 2024-09-21T12:33:13.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:13.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.080265+0000", 2024-09-21T12:33:13.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.58e-06 2024-09-21T12:33:13.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:13.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.080277+0000", 2024-09-21T12:33:13.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1610000000000001e-05 2024-09-21T12:33:13.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:13.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.080282+0000", 2024-09-21T12:33:13.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.366e-06 2024-09-21T12:33:13.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.080329+0000", 2024-09-21T12:33:13.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7267e-05 2024-09-21T12:33:13.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:13.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.080337+0000", 2024-09-21T12:33:13.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0299999999999994e-06 2024-09-21T12:33:13.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.056546+0000", 2024-09-21T12:33:13.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.97620854599999995 2024-09-21T12:33:13.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:13.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.056558+0000", 2024-09-21T12:33:13.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1619e-05 2024-09-21T12:33:13.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.115730+0000", 2024-09-21T12:33:13.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0591722560000001 2024-09-21T12:33:13.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:13.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.115840+0000", 2024-09-21T12:33:13.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000110555 2024-09-21T12:33:13.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.716 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.716 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:33:13.716 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.115924+0000", 2024-09-21T12:33:13.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3115999999999995e-05 2024-09-21T12:33:13.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.716 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.716 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.716 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.117996+0000", 2024-09-21T12:33:13.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020720320000000001 2024-09-21T12:33:13.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.716 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.716 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:13.716 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.121834+0000", 2024-09-21T12:33:13.717 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0038379970000000001 2024-09-21T12:33:13.717 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.717 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.121889+0000", 2024-09-21T12:33:13.717 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5245999999999999e-05 2024-09-21T12:33:13.717 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.718 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.121951+0000", 2024-09-21T12:33:13.718 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1906e-05 2024-09-21T12:33:13.718 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.718 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.718 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:13.718 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122014+0000", 2024-09-21T12:33:13.718 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3441999999999993e-05 2024-09-21T12:33:13.718 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.718 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.718 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:13.718 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122138+0000", 2024-09-21T12:33:13.718 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012343400000000001 2024-09-21T12:33:13.718 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.719 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:13.719 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.719 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.719 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.719 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:221957 9.d 9:bbdc0e00:::benchmark_data_smithi114_133214_object97923:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:33:13.719 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.082731+0000", 2024-09-21T12:33:13.719 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 95.616727170999994, 2024-09-21T12:33:13.719 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0396655529999999, 2024-09-21T12:33:13.719 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:13.719 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:13.719 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:13.719 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:13.719 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:13.719 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:13.719 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 221957 2024-09-21T12:33:13.720 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.720 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:13.720 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.720 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:13.720 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.082731+0000", 2024-09-21T12:33:13.720 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.720 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.720 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.720 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:13.720 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.082731+0000", 2024-09-21T12:33:13.720 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.720 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.720 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.720 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:13.721 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.082731+0000", 2024-09-21T12:33:13.721 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0699999999999996e-07 2024-09-21T12:33:13.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.721 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:13.721 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.082735+0000", 2024-09-21T12:33:13.721 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2720000000000002e-06 2024-09-21T12:33:13.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.721 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:13.721 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.082744+0000", 2024-09-21T12:33:13.721 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0410000000000003e-06 2024-09-21T12:33:13.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.721 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:13.722 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.082748+0000", 2024-09-21T12:33:13.722 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6280000000000004e-06 2024-09-21T12:33:13.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.722 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.722 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.722 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.082805+0000", 2024-09-21T12:33:13.722 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6938000000000003e-05 2024-09-21T12:33:13.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.722 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.722 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:13.722 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.082824+0000", 2024-09-21T12:33:13.722 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8493999999999999e-05 2024-09-21T12:33:13.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.722 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.723 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.723 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.056890+0000", 2024-09-21T12:33:13.723 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.97406623800000003 2024-09-21T12:33:13.723 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.723 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.723 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:13.723 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.056898+0000", 2024-09-21T12:33:13.723 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5159999999999994e-06 2024-09-21T12:33:13.723 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.723 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.723 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.723 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.118054+0000", 2024-09-21T12:33:13.723 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.061155555 2024-09-21T12:33:13.723 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.724 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.724 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:13.724 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.118246+0000", 2024-09-21T12:33:13.724 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019225500000000001 2024-09-21T12:33:13.724 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.724 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.724 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:33:13.724 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.118379+0000", 2024-09-21T12:33:13.724 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013250299999999999 2024-09-21T12:33:13.724 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.724 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.724 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:13.724 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122152+0000", 2024-09-21T12:33:13.724 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0037734909999999999 2024-09-21T12:33:13.725 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.725 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.725 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.725 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122183+0000", 2024-09-21T12:33:13.725 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0549999999999997e-05 2024-09-21T12:33:13.725 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.725 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.725 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.725 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122237+0000", 2024-09-21T12:33:13.725 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3735e-05 2024-09-21T12:33:13.725 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.725 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.725 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.725 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122292+0000", 2024-09-21T12:33:13.725 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5507000000000002e-05 2024-09-21T12:33:13.725 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.726 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.726 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:13.726 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122328+0000", 2024-09-21T12:33:13.726 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6149000000000002e-05 2024-09-21T12:33:13.726 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.726 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.726 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:13.726 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122396+0000", 2024-09-21T12:33:13.726 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7873999999999994e-05 2024-09-21T12:33:13.726 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.726 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:13.726 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.726 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.726 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.726 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:221969 9.d 9:b7382fd3:::benchmark_data_smithi114_133214_object97935:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:33:13.727 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.094992+0000", 2024-09-21T12:33:13.727 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 95.604465777000001, 2024-09-21T12:33:13.727 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0395933770000001, 2024-09-21T12:33:13.727 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:13.727 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:13.727 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:13.727 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:13.727 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:13.727 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:13.727 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 221969 2024-09-21T12:33:13.727 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.727 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:13.727 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.727 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:13.727 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.094992+0000", 2024-09-21T12:33:13.728 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.728 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.728 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.728 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:13.728 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.094992+0000", 2024-09-21T12:33:13.728 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.728 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.728 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.728 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:13.728 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.094993+0000", 2024-09-21T12:33:13.728 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.054e-06 2024-09-21T12:33:13.728 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.728 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.728 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:13.729 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.094998+0000", 2024-09-21T12:33:13.729 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7620000000000003e-06 2024-09-21T12:33:13.729 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.729 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.729 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:13.729 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095010+0000", 2024-09-21T12:33:13.729 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2091e-05 2024-09-21T12:33:13.729 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.729 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.729 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:13.729 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095018+0000", 2024-09-21T12:33:13.729 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8029999999999997e-06 2024-09-21T12:33:13.729 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.729 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.729 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.730 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095074+0000", 2024-09-21T12:33:13.730 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6013000000000001e-05 2024-09-21T12:33:13.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.730 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:13.730 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095081+0000", 2024-09-21T12:33:13.730 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9999999999999999e-06 2024-09-21T12:33:13.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.730 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.730 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.056923+0000", 2024-09-21T12:33:13.730 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.96184202600000002 2024-09-21T12:33:13.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.730 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:13.731 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.056930+0000", 2024-09-21T12:33:13.731 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4329999999999998e-06 2024-09-21T12:33:13.731 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.731 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.731 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.731 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122692+0000", 2024-09-21T12:33:13.731 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.065761918 2024-09-21T12:33:13.731 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.731 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.731 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:13.731 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122771+0000", 2024-09-21T12:33:13.731 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9456000000000006e-05 2024-09-21T12:33:13.731 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.732 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.732 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:33:13.732 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122867+0000", 2024-09-21T12:33:13.732 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5639999999999999e-05 2024-09-21T12:33:13.732 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.732 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.732 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.732 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.129541+0000", 2024-09-21T12:33:13.732 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.006674386 2024-09-21T12:33:13.732 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.732 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.732 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.732 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.129579+0000", 2024-09-21T12:33:13.732 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7162000000000001e-05 2024-09-21T12:33:13.732 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.733 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.733 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.733 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.129693+0000", 2024-09-21T12:33:13.733 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011387099999999999 2024-09-21T12:33:13.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.733 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.733 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:13.733 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134406+0000", 2024-09-21T12:33:13.733 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0047137619999999998 2024-09-21T12:33:13.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.733 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.733 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:13.733 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134466+0000", 2024-09-21T12:33:13.733 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9317999999999997e-05 2024-09-21T12:33:13.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.734 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.734 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:13.734 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134585+0000", 2024-09-21T12:33:13.734 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000119682 2024-09-21T12:33:13.734 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.734 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:13.734 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.734 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.734 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.734 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:221970 9.d 9:b4ae3e82:::benchmark_data_smithi114_133214_object97936:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:33:13.734 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.095717+0000", 2024-09-21T12:33:13.734 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 95.603740479999999, 2024-09-21T12:33:13.734 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0390824689999998, 2024-09-21T12:33:13.734 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:13.735 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:13.735 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:13.735 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:13.735 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:13.735 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:13.735 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 221970 2024-09-21T12:33:13.735 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.735 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:13.735 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.735 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:13.735 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095717+0000", 2024-09-21T12:33:13.735 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.735 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.735 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.735 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:13.736 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095717+0000", 2024-09-21T12:33:13.736 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.736 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.736 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.736 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:13.736 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095718+0000", 2024-09-21T12:33:13.736 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4e-07 2024-09-21T12:33:13.736 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.736 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.736 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:13.736 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095723+0000", 2024-09-21T12:33:13.736 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.51e-06 2024-09-21T12:33:13.736 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.736 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.736 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:13.736 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095732+0000", 2024-09-21T12:33:13.737 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7620000000000007e-06 2024-09-21T12:33:13.737 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.737 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.737 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:13.737 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095738+0000", 2024-09-21T12:33:13.737 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2909999999999998e-06 2024-09-21T12:33:13.737 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.737 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.737 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.737 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095779+0000", 2024-09-21T12:33:13.737 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1356999999999997e-05 2024-09-21T12:33:13.737 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.737 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.737 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:13.737 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095785+0000", 2024-09-21T12:33:13.738 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7610000000000004e-06 2024-09-21T12:33:13.738 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.738 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.738 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.056979+0000", 2024-09-21T12:33:13.738 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.96119413399999998 2024-09-21T12:33:13.738 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.738 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:13.738 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.056986+0000", 2024-09-21T12:33:13.738 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1770000000000001e-06 2024-09-21T12:33:13.738 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.738 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.739 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.125121+0000", 2024-09-21T12:33:13.739 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0681344740000001 2024-09-21T12:33:13.739 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.739 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.739 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:13.739 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.125232+0000", 2024-09-21T12:33:13.739 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000111875 2024-09-21T12:33:13.739 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.739 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.739 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:33:13.739 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.125348+0000", 2024-09-21T12:33:13.739 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000115696 2024-09-21T12:33:13.739 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.739 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.740 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.740 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.129616+0000", 2024-09-21T12:33:13.740 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0042675930000000001 2024-09-21T12:33:13.740 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.740 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.740 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.740 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.129653+0000", 2024-09-21T12:33:13.740 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7786e-05 2024-09-21T12:33:13.740 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.740 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.740 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.740 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.129734+0000", 2024-09-21T12:33:13.740 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1036000000000001e-05 2024-09-21T12:33:13.740 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.741 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.741 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:13.741 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134602+0000", 2024-09-21T12:33:13.741 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0048670479999999997 2024-09-21T12:33:13.741 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.741 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.741 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:13.741 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134664+0000", 2024-09-21T12:33:13.741 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2360999999999995e-05 2024-09-21T12:33:13.741 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.741 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.741 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:13.741 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134800+0000", 2024-09-21T12:33:13.741 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013576800000000001 2024-09-21T12:33:13.742 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.742 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:13.742 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.742 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.742 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.742 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:221972 9.d 9:b9fcc29c:::benchmark_data_smithi114_133214_object97938:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:33:13.742 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.096425+0000", 2024-09-21T12:33:13.742 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 95.603032537999994, 2024-09-21T12:33:13.742 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0387005569999999, 2024-09-21T12:33:13.742 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:13.742 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:13.742 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:13.742 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:13.742 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:13.743 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:13.743 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 221972 2024-09-21T12:33:13.743 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.743 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:13.743 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.743 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:13.743 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.096425+0000", 2024-09-21T12:33:13.743 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.743 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.743 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.743 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:13.743 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.096425+0000", 2024-09-21T12:33:13.743 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.743 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.744 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.744 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:13.744 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.096427+0000", 2024-09-21T12:33:13.744 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0099999999999998e-06 2024-09-21T12:33:13.744 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.745 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.745 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:13.745 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.096437+0000", 2024-09-21T12:33:13.745 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6530000000000006e-06 2024-09-21T12:33:13.745 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.745 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.745 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:13.745 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.096465+0000", 2024-09-21T12:33:13.745 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8048999999999999e-05 2024-09-21T12:33:13.745 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.745 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.745 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:13.745 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.096479+0000", 2024-09-21T12:33:13.745 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4517e-05 2024-09-21T12:33:13.746 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.746 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.746 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.746 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.096622+0000", 2024-09-21T12:33:13.746 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000142173 2024-09-21T12:33:13.746 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.746 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.746 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:13.746 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.096637+0000", 2024-09-21T12:33:13.746 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5792999999999999e-05 2024-09-21T12:33:13.746 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.746 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.746 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.746 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057015+0000", 2024-09-21T12:33:13.746 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.96037787799999996 2024-09-21T12:33:13.747 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.747 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.747 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:13.747 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057023+0000", 2024-09-21T12:33:13.747 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6580000000000002e-06 2024-09-21T12:33:13.747 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.747 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.747 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.747 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.127356+0000", 2024-09-21T12:33:13.747 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.070332735 2024-09-21T12:33:13.747 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.747 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.747 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:13.747 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.127461+0000", 2024-09-21T12:33:13.748 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000105509 2024-09-21T12:33:13.748 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.748 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.748 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:33:13.748 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.127579+0000", 2024-09-21T12:33:13.748 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000118016 2024-09-21T12:33:13.748 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.748 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.748 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.748 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134387+0000", 2024-09-21T12:33:13.748 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0068083450000000004 2024-09-21T12:33:13.748 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.748 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.748 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:13.748 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134811+0000", 2024-09-21T12:33:13.749 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00042396600000000001 2024-09-21T12:33:13.749 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.749 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.749 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.749 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134871+0000", 2024-09-21T12:33:13.749 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9574000000000003e-05 2024-09-21T12:33:13.749 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.749 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.749 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.749 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134949+0000", 2024-09-21T12:33:13.749 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8454000000000005e-05 2024-09-21T12:33:13.749 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.749 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.749 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:13.750 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135009+0000", 2024-09-21T12:33:13.750 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9676999999999998e-05 2024-09-21T12:33:13.750 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.750 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.750 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:13.750 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135126+0000", 2024-09-21T12:33:13.750 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011655 2024-09-21T12:33:13.750 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.750 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:13.750 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.750 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.750 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.750 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:221983 9.15 9:ae2f2fb3:::benchmark_data_smithi114_133214_object97949:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:33:13.750 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.108916+0000", 2024-09-21T12:33:13.751 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 95.590541439999996, 2024-09-21T12:33:13.751 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.022601734, 2024-09-21T12:33:13.751 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:13.751 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:13.751 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:13.751 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:13.751 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:13.751 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:13.751 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 221983 2024-09-21T12:33:13.751 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.751 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:13.751 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.751 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:13.751 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.108916+0000", 2024-09-21T12:33:13.752 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.752 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.752 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.752 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:13.752 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.108916+0000", 2024-09-21T12:33:13.752 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.752 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.752 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.752 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:13.752 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.108917+0000", 2024-09-21T12:33:13.752 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.111e-06 2024-09-21T12:33:13.752 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.752 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.752 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:13.753 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.108922+0000", 2024-09-21T12:33:13.753 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8280000000000002e-06 2024-09-21T12:33:13.753 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.753 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.753 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:13.753 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.108933+0000", 2024-09-21T12:33:13.753 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0939e-05 2024-09-21T12:33:13.753 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.753 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.753 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:13.753 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.108940+0000", 2024-09-21T12:33:13.753 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8530000000000003e-06 2024-09-21T12:33:13.753 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.753 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.753 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.754 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.109007+0000", 2024-09-21T12:33:13.754 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6643000000000006e-05 2024-09-21T12:33:13.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.754 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.754 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:13.754 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.109013+0000", 2024-09-21T12:33:13.754 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9349999999999999e-06 2024-09-21T12:33:13.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.754 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.754 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.754 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.126725+0000", 2024-09-21T12:33:13.754 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0177128740000001 2024-09-21T12:33:13.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.754 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.755 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:13.755 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.126867+0000", 2024-09-21T12:33:13.755 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014166499999999999 2024-09-21T12:33:13.755 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.755 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.755 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1,3,5", 2024-09-21T12:33:13.755 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.127184+0000", 2024-09-21T12:33:13.755 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00031702800000000001 2024-09-21T12:33:13.755 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.755 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.755 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.755 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.129977+0000", 2024-09-21T12:33:13.755 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002792475 2024-09-21T12:33:13.755 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.756 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.756 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:13.756 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131139+0000", 2024-09-21T12:33:13.756 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001162777 2024-09-21T12:33:13.756 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.756 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.756 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.756 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131244+0000", 2024-09-21T12:33:13.756 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000104324 2024-09-21T12:33:13.756 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.756 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.756 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.756 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131283+0000", 2024-09-21T12:33:13.756 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9199000000000002e-05 2024-09-21T12:33:13.756 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.757 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.757 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.757 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131333+0000", 2024-09-21T12:33:13.757 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0178e-05 2024-09-21T12:33:13.757 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.757 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.757 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:13.757 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131394+0000", 2024-09-21T12:33:13.757 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0554000000000003e-05 2024-09-21T12:33:13.757 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.757 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.757 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:13.757 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131518+0000", 2024-09-21T12:33:13.758 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000124351 2024-09-21T12:33:13.758 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.758 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:13.758 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.758 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.758 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.758 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:221997 9.d 9:b9573347:::benchmark_data_smithi114_133214_object97963:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:33:13.758 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.121318+0000", 2024-09-21T12:33:13.758 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 95.578139680999996, 2024-09-21T12:33:13.758 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.01426215, 2024-09-21T12:33:13.758 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:13.758 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:13.758 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:13.758 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:13.759 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:13.759 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:13.759 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 221997 2024-09-21T12:33:13.759 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.759 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:13.759 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.759 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:13.759 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.121318+0000", 2024-09-21T12:33:13.759 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.759 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.759 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.759 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:13.759 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.121318+0000", 2024-09-21T12:33:13.760 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.760 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.760 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:13.760 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.121319+0000", 2024-09-21T12:33:13.760 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.364e-06 2024-09-21T12:33:13.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.760 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.760 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:13.760 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.121324+0000", 2024-09-21T12:33:13.760 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5929999999999999e-06 2024-09-21T12:33:13.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.760 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.761 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:13.761 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.121334+0000", 2024-09-21T12:33:13.761 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8309999999999996e-06 2024-09-21T12:33:13.761 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.761 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.761 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:13.761 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.121341+0000", 2024-09-21T12:33:13.761 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0910000000000004e-06 2024-09-21T12:33:13.761 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.761 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.761 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.761 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.121402+0000", 2024-09-21T12:33:13.761 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0875999999999998e-05 2024-09-21T12:33:13.761 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.761 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.762 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:13.762 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.121409+0000", 2024-09-21T12:33:13.762 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.498e-06 2024-09-21T12:33:13.762 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.762 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.762 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.762 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057045+0000", 2024-09-21T12:33:13.762 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.93563575200000004 2024-09-21T12:33:13.762 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.762 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.762 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:13.762 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057053+0000", 2024-09-21T12:33:13.762 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9359999999999999e-06 2024-09-21T12:33:13.762 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.763 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.763 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.763 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.129765+0000", 2024-09-21T12:33:13.763 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0727123919999999 2024-09-21T12:33:13.763 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.763 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.763 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:13.763 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.129890+0000", 2024-09-21T12:33:13.763 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012480799999999999 2024-09-21T12:33:13.763 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.763 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.763 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:33:13.763 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.130027+0000", 2024-09-21T12:33:13.763 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000136425 2024-09-21T12:33:13.764 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.764 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.764 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:13.764 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134824+0000", 2024-09-21T12:33:13.764 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0047976219999999997 2024-09-21T12:33:13.764 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.764 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.764 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.764 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135204+0000", 2024-09-21T12:33:13.764 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000380054 2024-09-21T12:33:13.764 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.764 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.764 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.764 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135259+0000", 2024-09-21T12:33:13.765 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5071000000000002e-05 2024-09-21T12:33:13.765 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.765 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.765 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.765 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135393+0000", 2024-09-21T12:33:13.765 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000133991 2024-09-21T12:33:13.765 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.765 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.765 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:13.765 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135457+0000", 2024-09-21T12:33:13.765 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3830000000000004e-05 2024-09-21T12:33:13.765 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.765 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.765 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:13.766 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135580+0000", 2024-09-21T12:33:13.766 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012301599999999999 2024-09-21T12:33:13.766 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.766 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:13.766 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.766 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.766 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.766 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:222014 9.d 9:bb4c3379:::benchmark_data_smithi114_133214_object97980:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:33:13.766 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.130717+0000", 2024-09-21T12:33:13.766 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 95.568740984000002, 2024-09-21T12:33:13.766 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0050891000000002, 2024-09-21T12:33:13.766 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:13.766 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:13.766 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:13.767 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:13.767 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:13.767 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:13.767 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 222014 2024-09-21T12:33:13.767 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.767 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:13.767 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.767 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:13.767 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.130717+0000", 2024-09-21T12:33:13.767 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.767 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.767 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.767 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:13.767 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.130717+0000", 2024-09-21T12:33:13.768 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.768 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.768 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.768 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:13.768 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.130718+0000", 2024-09-21T12:33:13.768 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.052e-06 2024-09-21T12:33:13.768 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.768 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.768 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:13.768 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.130722+0000", 2024-09-21T12:33:13.768 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8469999999999998e-06 2024-09-21T12:33:13.768 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.768 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.768 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:13.768 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.130730+0000", 2024-09-21T12:33:13.769 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5029999999999997e-06 2024-09-21T12:33:13.769 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.769 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.769 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:13.769 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.130737+0000", 2024-09-21T12:33:13.769 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6420000000000001e-06 2024-09-21T12:33:13.769 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.769 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.769 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.769 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.130793+0000", 2024-09-21T12:33:13.769 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6498e-05 2024-09-21T12:33:13.769 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.769 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.769 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:13.770 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.130802+0000", 2024-09-21T12:33:13.770 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0080000000000008e-06 2024-09-21T12:33:13.770 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.770 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.770 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.770 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057076+0000", 2024-09-21T12:33:13.770 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.92627378500000002 2024-09-21T12:33:13.770 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.770 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.770 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:13.770 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057083+0000", 2024-09-21T12:33:13.770 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7410000000000003e-06 2024-09-21T12:33:13.770 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.770 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.771 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.771 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.131785+0000", 2024-09-21T12:33:13.771 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0747017809999999 2024-09-21T12:33:13.771 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.771 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.771 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:13.771 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.131913+0000", 2024-09-21T12:33:13.771 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012813399999999999 2024-09-21T12:33:13.771 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.771 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.771 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:33:13.771 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.132034+0000", 2024-09-21T12:33:13.771 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000121236 2024-09-21T12:33:13.771 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.772 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.772 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:13.772 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134886+0000", 2024-09-21T12:33:13.772 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0028518910000000001 2024-09-21T12:33:13.772 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.772 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.772 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.772 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135303+0000", 2024-09-21T12:33:13.772 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000417279 2024-09-21T12:33:13.772 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.772 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.772 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.772 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135347+0000", 2024-09-21T12:33:13.773 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4399e-05 2024-09-21T12:33:13.773 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.773 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.773 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.773 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135633+0000", 2024-09-21T12:33:13.773 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00028544300000000002 2024-09-21T12:33:13.773 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.773 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.773 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:13.773 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135690+0000", 2024-09-21T12:33:13.773 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6577999999999999e-05 2024-09-21T12:33:13.773 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.773 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.774 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:13.774 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135806+0000", 2024-09-21T12:33:13.774 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000116283 2024-09-21T12:33:13.774 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.774 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:13.774 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.774 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.774 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.774 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:222021 9.15 9:ac1e73ee:::benchmark_data_smithi114_133214_object97987:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:33:13.774 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.135261+0000", 2024-09-21T12:33:13.774 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 95.564196976999995, 2024-09-21T12:33:13.774 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.99670988900000002, 2024-09-21T12:33:13.774 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:13.775 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:13.775 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:13.775 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:13.775 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:13.775 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:13.775 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 222021 2024-09-21T12:33:13.775 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.775 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:13.775 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.775 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:13.775 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.135261+0000", 2024-09-21T12:33:13.775 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.775 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.775 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.776 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:13.776 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.135261+0000", 2024-09-21T12:33:13.776 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.776 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.776 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.776 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:13.776 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.135261+0000", 2024-09-21T12:33:13.776 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7100000000000001e-07 2024-09-21T12:33:13.776 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.776 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.776 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:13.776 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.135264+0000", 2024-09-21T12:33:13.776 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8779999999999998e-06 2024-09-21T12:33:13.776 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.777 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.777 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:13.777 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.135279+0000", 2024-09-21T12:33:13.777 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4713e-05 2024-09-21T12:33:13.777 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.777 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.777 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:13.777 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.135284+0000", 2024-09-21T12:33:13.777 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2630000000000003e-06 2024-09-21T12:33:13.777 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.777 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.777 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.777 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.135337+0000", 2024-09-21T12:33:13.777 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3134000000000003e-05 2024-09-21T12:33:13.778 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.778 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.778 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:13.778 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.135345+0000", 2024-09-21T12:33:13.778 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6750000000000002e-06 2024-09-21T12:33:13.778 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.778 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.778 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.778 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.128709+0000", 2024-09-21T12:33:13.778 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.99336335799999997 2024-09-21T12:33:13.778 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.778 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.778 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:13.778 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.128784+0000", 2024-09-21T12:33:13.778 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5585000000000002e-05 2024-09-21T12:33:13.779 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.779 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.779 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1,3,5", 2024-09-21T12:33:13.779 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.128998+0000", 2024-09-21T12:33:13.779 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000213506 2024-09-21T12:33:13.779 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.779 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.779 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.779 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131181+0000", 2024-09-21T12:33:13.779 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002183365 2024-09-21T12:33:13.779 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.779 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.779 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:13.779 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131256+0000", 2024-09-21T12:33:13.779 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4989999999999999e-05 2024-09-21T12:33:13.779 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.780 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.780 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.780 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131691+0000", 2024-09-21T12:33:13.780 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00043519000000000001 2024-09-21T12:33:13.780 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.780 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.780 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.780 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131793+0000", 2024-09-21T12:33:13.780 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010200300000000001 2024-09-21T12:33:13.780 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.780 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.780 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.780 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131850+0000", 2024-09-21T12:33:13.780 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7299000000000001e-05 2024-09-21T12:33:13.780 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.780 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.781 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:13.781 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131900+0000", 2024-09-21T12:33:13.781 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9459999999999997e-05 2024-09-21T12:33:13.781 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.781 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.781 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:13.781 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131971+0000", 2024-09-21T12:33:13.781 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0698999999999995e-05 2024-09-21T12:33:13.781 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.781 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:13.781 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.781 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.781 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.782 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:222036 9.d 9:bf4c3996:::benchmark_data_smithi114_133214_object98002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:33:13.782 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.149247+0000", 2024-09-21T12:33:13.782 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 95.550210785000004, 2024-09-21T12:33:13.782 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9939442509999998, 2024-09-21T12:33:13.782 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:13.782 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:13.782 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:13.782 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:13.782 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:13.782 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:13.782 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 222036 2024-09-21T12:33:13.782 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.782 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:13.782 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.782 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:13.783 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.149247+0000", 2024-09-21T12:33:13.783 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.783 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.783 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.783 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:13.783 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.149247+0000", 2024-09-21T12:33:13.783 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.783 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.783 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.783 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:13.783 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.149248+0000", 2024-09-21T12:33:13.783 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3200000000000001e-06 2024-09-21T12:33:13.783 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.783 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.784 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:13.784 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.149253+0000", 2024-09-21T12:33:13.784 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2129999999999999e-06 2024-09-21T12:33:13.784 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.784 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.784 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:13.784 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.149263+0000", 2024-09-21T12:33:13.784 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2580000000000001e-06 2024-09-21T12:33:13.784 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.784 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.784 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:13.784 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.149269+0000", 2024-09-21T12:33:13.784 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5379999999999999e-06 2024-09-21T12:33:13.784 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.785 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.785 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.785 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.149332+0000", 2024-09-21T12:33:13.785 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:13.786 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2496999999999995e-05 2024-09-21T12:33:13.786 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.786 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.787 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:13.787 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.149343+0000", 2024-09-21T12:33:13.787 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1011e-05 2024-09-21T12:33:13.787 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.787 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.787 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.787 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057106+0000", 2024-09-21T12:33:13.787 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.90776296199999995 2024-09-21T12:33:13.787 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.787 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.787 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:13.787 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057113+0000", 2024-09-21T12:33:13.787 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4250000000000001e-06 2024-09-21T12:33:13.788 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.788 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.788 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.788 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135854+0000", 2024-09-21T12:33:13.788 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.078740493 2024-09-21T12:33:13.788 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.788 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.788 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:13.788 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135971+0000", 2024-09-21T12:33:13.788 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000117704 2024-09-21T12:33:13.788 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.788 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.788 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:33:13.788 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.136102+0000", 2024-09-21T12:33:13.788 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013025100000000001 2024-09-21T12:33:13.789 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.789 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.789 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:13.789 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.142810+0000", 2024-09-21T12:33:13.789 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0067082419999999997 2024-09-21T12:33:13.789 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.789 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.789 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.789 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.142861+0000", 2024-09-21T12:33:13.789 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1381000000000002e-05 2024-09-21T12:33:13.789 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.789 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.789 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.789 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.142919+0000", 2024-09-21T12:33:13.790 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7482000000000003e-05 2024-09-21T12:33:13.790 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.790 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.790 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.790 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.143004+0000", 2024-09-21T12:33:13.790 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5746000000000002e-05 2024-09-21T12:33:13.790 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.790 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.790 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:13.790 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.143072+0000", 2024-09-21T12:33:13.790 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8009000000000006e-05 2024-09-21T12:33:13.790 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.790 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.791 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:13.791 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.143191+0000", 2024-09-21T12:33:13.791 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000118719 2024-09-21T12:33:13.791 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.791 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:13.791 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.791 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.791 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.791 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:222038 9.d 9:b5ba6170:::benchmark_data_smithi114_133214_object98004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:33:13.791 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.150309+0000", 2024-09-21T12:33:13.791 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 95.549148891000002, 2024-09-21T12:33:13.791 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.995585494, 2024-09-21T12:33:13.791 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:13.791 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:13.792 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:13.792 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:13.792 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:13.792 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:13.792 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 222038 2024-09-21T12:33:13.792 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.792 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:13.792 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.792 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:13.792 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.150309+0000", 2024-09-21T12:33:13.792 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.792 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.792 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.792 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:13.793 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.150309+0000", 2024-09-21T12:33:13.793 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.793 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.793 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.793 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:13.793 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.150310+0000", 2024-09-21T12:33:13.793 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8800000000000001e-07 2024-09-21T12:33:13.793 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.793 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.793 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:13.793 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.150313+0000", 2024-09-21T12:33:13.793 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.58e-06 2024-09-21T12:33:13.793 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.793 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.793 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:13.793 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.150322+0000", 2024-09-21T12:33:13.794 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2600000000000005e-06 2024-09-21T12:33:13.794 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.794 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.794 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:13.794 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.150328+0000", 2024-09-21T12:33:13.794 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3790000000000003e-06 2024-09-21T12:33:13.794 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.794 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.794 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.794 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.150385+0000", 2024-09-21T12:33:13.794 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7188000000000002e-05 2024-09-21T12:33:13.794 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.794 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.794 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:13.794 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.150394+0000", 2024-09-21T12:33:13.795 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1360000000000003e-06 2024-09-21T12:33:13.795 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.795 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.795 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.795 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057143+0000", 2024-09-21T12:33:13.795 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.90674834100000001 2024-09-21T12:33:13.795 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.795 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.795 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:13.795 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057150+0000", 2024-09-21T12:33:13.795 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.481e-06 2024-09-21T12:33:13.795 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.795 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.796 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.796 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.138249+0000", 2024-09-21T12:33:13.796 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0810986 2024-09-21T12:33:13.796 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.796 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.796 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:13.796 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.138359+0000", 2024-09-21T12:33:13.796 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011010699999999999 2024-09-21T12:33:13.796 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.796 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.796 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:33:13.796 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.138473+0000", 2024-09-21T12:33:13.796 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000114271 2024-09-21T12:33:13.796 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.797 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.797 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:13.797 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.142879+0000", 2024-09-21T12:33:13.797 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0044062709999999998 2024-09-21T12:33:13.797 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.797 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.797 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.797 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.142962+0000", 2024-09-21T12:33:13.797 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2497999999999995e-05 2024-09-21T12:33:13.797 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.797 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.797 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.797 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.143245+0000", 2024-09-21T12:33:13.797 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00028279200000000003 2024-09-21T12:33:13.797 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.798 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.798 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.798 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.145676+0000", 2024-09-21T12:33:13.798 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024311929999999999 2024-09-21T12:33:13.798 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.798 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.798 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:13.798 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.145735+0000", 2024-09-21T12:33:13.798 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9159000000000001e-05 2024-09-21T12:33:13.798 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.798 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.798 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:13.798 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.145894+0000", 2024-09-21T12:33:13.798 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015935 2024-09-21T12:33:13.799 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.799 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:13.799 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.799 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.799 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.799 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:222040 9.15 9:afba0a7e:::benchmark_data_smithi114_133214_object98006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:33:13.799 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.152391+0000", 2024-09-21T12:33:13.799 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 95.547066735000001, 2024-09-21T12:33:13.799 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.979744434, 2024-09-21T12:33:13.799 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:13.799 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:13.799 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:13.799 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:13.799 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:13.800 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:13.800 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 222040 2024-09-21T12:33:13.800 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.800 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:13.800 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.800 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:13.800 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.152391+0000", 2024-09-21T12:33:13.800 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.800 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.800 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.800 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:13.800 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.152391+0000", 2024-09-21T12:33:13.800 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.800 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.801 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.801 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:13.801 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.152392+0000", 2024-09-21T12:33:13.801 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.09e-07 2024-09-21T12:33:13.801 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.801 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.801 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:13.801 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.152395+0000", 2024-09-21T12:33:13.801 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.534e-06 2024-09-21T12:33:13.801 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.801 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.801 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:13.801 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.152406+0000", 2024-09-21T12:33:13.801 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0779e-05 2024-09-21T12:33:13.801 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.802 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.802 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:13.802 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.152411+0000", 2024-09-21T12:33:13.802 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5079999999999999e-06 2024-09-21T12:33:13.802 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.802 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.802 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.802 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.152451+0000", 2024-09-21T12:33:13.802 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0462e-05 2024-09-21T12:33:13.802 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.802 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.802 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:13.802 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.152457+0000", 2024-09-21T12:33:13.803 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6389999999999997e-06 2024-09-21T12:33:13.803 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.803 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.803 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.803 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.130005+0000", 2024-09-21T12:33:13.803 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.97754845099999998 2024-09-21T12:33:13.803 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.803 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.803 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:13.803 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.130083+0000", 2024-09-21T12:33:13.803 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8159999999999997e-05 2024-09-21T12:33:13.803 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.803 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.803 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1,3,5", 2024-09-21T12:33:13.803 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.130252+0000", 2024-09-21T12:33:13.804 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016846799999999999 2024-09-21T12:33:13.804 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.804 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.804 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:13.804 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131532+0000", 2024-09-21T12:33:13.804 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012803300000000001 2024-09-21T12:33:13.804 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.804 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.804 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.804 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131727+0000", 2024-09-21T12:33:13.804 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019453999999999999 2024-09-21T12:33:13.804 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.804 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.804 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.805 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131758+0000", 2024-09-21T12:33:13.805 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1538000000000002e-05 2024-09-21T12:33:13.805 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.805 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.805 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.805 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.132004+0000", 2024-09-21T12:33:13.805 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000245758 2024-09-21T12:33:13.805 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.805 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.805 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.805 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.132035+0000", 2024-09-21T12:33:13.805 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0871000000000003e-05 2024-09-21T12:33:13.805 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.805 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.806 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:13.806 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.132064+0000", 2024-09-21T12:33:13.806 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9549000000000001e-05 2024-09-21T12:33:13.806 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.806 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.806 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:13.806 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.132135+0000", 2024-09-21T12:33:13.806 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1038000000000007e-05 2024-09-21T12:33:13.806 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.806 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:13.806 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.806 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.806 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.806 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:222057 9.d 9:bd682287:::benchmark_data_smithi114_133214_object98023:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:33:13.806 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.194600+0000", 2024-09-21T12:33:13.807 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 95.504857655999999, 2024-09-21T12:33:13.807 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9643363250000001, 2024-09-21T12:33:13.807 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:13.807 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:13.807 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:13.807 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:13.807 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:13.807 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:13.807 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 222057 2024-09-21T12:33:13.807 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.807 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:13.807 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.807 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:13.807 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.194600+0000", 2024-09-21T12:33:13.808 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.808 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.808 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.808 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:13.808 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.194600+0000", 2024-09-21T12:33:13.808 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.808 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.808 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.808 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:13.808 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.194601+0000", 2024-09-21T12:33:13.808 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.15e-06 2024-09-21T12:33:13.808 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.808 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.808 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:13.809 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.194605+0000", 2024-09-21T12:33:13.809 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1500000000000001e-06 2024-09-21T12:33:13.809 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.809 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.809 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:13.809 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.194616+0000", 2024-09-21T12:33:13.809 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0759e-05 2024-09-21T12:33:13.809 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.809 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.809 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:13.809 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.194622+0000", 2024-09-21T12:33:13.809 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.614e-06 2024-09-21T12:33:13.809 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.809 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.810 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.810 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.194683+0000", 2024-09-21T12:33:13.810 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0934000000000003e-05 2024-09-21T12:33:13.810 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.810 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.810 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:13.810 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.194690+0000", 2024-09-21T12:33:13.810 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3189999999999997e-06 2024-09-21T12:33:13.810 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.810 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.810 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.810 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057173+0000", 2024-09-21T12:33:13.810 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.862483322 2024-09-21T12:33:13.810 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.811 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.811 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:13.811 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057180+0000", 2024-09-21T12:33:13.811 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0600000000000002e-06 2024-09-21T12:33:13.811 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.811 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.811 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.811 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.140485+0000", 2024-09-21T12:33:13.811 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0833051250000001 2024-09-21T12:33:13.811 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.811 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.811 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:13.811 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.140621+0000", 2024-09-21T12:33:13.811 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013514899999999999 2024-09-21T12:33:13.811 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.812 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.812 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:33:13.812 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.140741+0000", 2024-09-21T12:33:13.812 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012082399999999999 2024-09-21T12:33:13.812 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.812 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.812 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.812 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.145628+0000", 2024-09-21T12:33:13.812 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0048861720000000003 2024-09-21T12:33:13.812 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.812 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.812 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.812 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.145926+0000", 2024-09-21T12:33:13.812 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00029853099999999999 2024-09-21T12:33:13.812 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.813 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.813 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:13.813 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.150100+0000", 2024-09-21T12:33:13.813 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0041734930000000003 2024-09-21T12:33:13.813 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.813 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.813 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.813 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.158801+0000", 2024-09-21T12:33:13.813 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0087013130000000004 2024-09-21T12:33:13.813 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.813 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.813 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:13.813 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.158849+0000", 2024-09-21T12:33:13.813 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7604e-05 2024-09-21T12:33:13.813 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.814 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.814 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:13.814 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.158936+0000", 2024-09-21T12:33:13.814 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7806000000000006e-05 2024-09-21T12:33:13.814 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.814 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:13.814 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.814 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.814 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.814 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:222065 9.d 9:b27ebeb8:::benchmark_data_smithi114_133214_object98031:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-09-21T12:33:13.814 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:39.140786+0000", 2024-09-21T12:33:13.814 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 94.558671451999999, 2024-09-21T12:33:13.814 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0183983160000001, 2024-09-21T12:33:13.815 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:13.815 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:13.815 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:13.815 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:13.815 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:13.815 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:13.815 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 222065 2024-09-21T12:33:13.815 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.815 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:13.815 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.815 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:13.815 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.140786+0000", 2024-09-21T12:33:13.815 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.815 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.816 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.816 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:13.816 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.140786+0000", 2024-09-21T12:33:13.816 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.816 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.816 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.816 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:13.816 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.140787+0000", 2024-09-21T12:33:13.816 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1650000000000001e-06 2024-09-21T12:33:13.816 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.816 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.816 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:13.816 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.140792+0000", 2024-09-21T12:33:13.816 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0930000000000002e-06 2024-09-21T12:33:13.816 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.817 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.817 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:13.817 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.140803+0000", 2024-09-21T12:33:13.817 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1e-05 2024-09-21T12:33:13.817 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.817 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.817 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:13.817 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.140811+0000", 2024-09-21T12:33:13.817 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1919999999999999e-06 2024-09-21T12:33:13.817 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.817 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.817 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.817 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.140887+0000", 2024-09-21T12:33:13.817 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5916999999999998e-05 2024-09-21T12:33:13.817 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.818 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.818 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:13.818 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.140899+0000", 2024-09-21T12:33:13.818 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1966e-05 2024-09-21T12:33:13.818 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.818 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.818 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.818 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.143276+0000", 2024-09-21T12:33:13.818 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.002377023 2024-09-21T12:33:13.818 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.818 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.818 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:13.818 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.143396+0000", 2024-09-21T12:33:13.818 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012068 2024-09-21T12:33:13.819 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.819 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.819 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:33:13.819 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.143523+0000", 2024-09-21T12:33:13.819 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012700799999999999 2024-09-21T12:33:13.819 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.819 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.819 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.819 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.150083+0000", 2024-09-21T12:33:13.819 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0065597399999999997 2024-09-21T12:33:13.819 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.819 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.819 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.819 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.150166+0000", 2024-09-21T12:33:13.820 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2736000000000002e-05 2024-09-21T12:33:13.820 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.820 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.820 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:13.820 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.152621+0000", 2024-09-21T12:33:13.820 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024554550000000001 2024-09-21T12:33:13.820 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.820 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.820 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.820 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.158993+0000", 2024-09-21T12:33:13.820 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0063721029999999996 2024-09-21T12:33:13.820 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.820 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.820 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:13.821 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.159060+0000", 2024-09-21T12:33:13.821 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6316999999999995e-05 2024-09-21T12:33:13.821 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.821 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.821 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:13.821 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.159185+0000", 2024-09-21T12:33:13.821 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012492100000000001 2024-09-21T12:33:13.821 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.821 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:13.821 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.821 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.821 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.821 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:222088 9.d 9:ba21c774:::benchmark_data_smithi114_133214_object98054:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-09-21T12:33:13.821 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:39.185838+0000", 2024-09-21T12:33:13.822 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 94.513619825000006, 2024-09-21T12:33:13.822 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.97349070500000001, 2024-09-21T12:33:13.822 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:13.822 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:13.822 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:13.822 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:13.822 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:33:13.822 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:13.822 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 222088 2024-09-21T12:33:13.822 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.822 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:13.822 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.822 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:13.822 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.185838+0000", 2024-09-21T12:33:13.823 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.823 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.823 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.823 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:13.823 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.185838+0000", 2024-09-21T12:33:13.823 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.823 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.823 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.823 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:13.823 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.185839+0000", 2024-09-21T12:33:13.823 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5799999999999998e-07 2024-09-21T12:33:13.823 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.823 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.823 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:13.824 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.185842+0000", 2024-09-21T12:33:13.824 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.354e-06 2024-09-21T12:33:13.824 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.824 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.824 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:13.824 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.185851+0000", 2024-09-21T12:33:13.824 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8140000000000007e-06 2024-09-21T12:33:13.824 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.824 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.824 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:13.824 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.185856+0000", 2024-09-21T12:33:13.824 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9930000000000003e-06 2024-09-21T12:33:13.824 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.824 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.824 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.825 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.185904+0000", 2024-09-21T12:33:13.825 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7979999999999998e-05 2024-09-21T12:33:13.825 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.825 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.825 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:13.825 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.185911+0000", 2024-09-21T12:33:13.825 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0729999999999999e-06 2024-09-21T12:33:13.825 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.825 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.825 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.825 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.145944+0000", 2024-09-21T12:33:13.825 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.96003328700000001 2024-09-21T12:33:13.825 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.825 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.826 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:13.826 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.146019+0000", 2024-09-21T12:33:13.826 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4240999999999996e-05 2024-09-21T12:33:13.826 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.826 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.826 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:33:13.826 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.146090+0000", 2024-09-21T12:33:13.826 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1210000000000004e-05 2024-09-21T12:33:13.826 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.826 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.826 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:13.826 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.152628+0000", 2024-09-21T12:33:13.826 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0065385490000000003 2024-09-21T12:33:13.826 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.827 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.827 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.827 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.155406+0000", 2024-09-21T12:33:13.827 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0027779979999999998 2024-09-21T12:33:13.827 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.827 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.827 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.827 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.155446+0000", 2024-09-21T12:33:13.827 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9777000000000002e-05 2024-09-21T12:33:13.827 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.827 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.827 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.827 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.159227+0000", 2024-09-21T12:33:13.827 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0037810550000000002 2024-09-21T12:33:13.828 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.828 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.828 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:13.828 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.159259+0000", 2024-09-21T12:33:13.828 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2264999999999997e-05 2024-09-21T12:33:13.828 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.828 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.828 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:13.828 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.159329+0000", 2024-09-21T12:33:13.828 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9151000000000003e-05 2024-09-21T12:33:13.828 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.828 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:13.828 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.828 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.828 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.829 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:4750 10.c 10:39bcf0f1:::benchmark_data_smithi114_161690_object4749:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-09-21T12:33:13.829 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:32:40.393690+0000", 2024-09-21T12:33:13.829 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 33.305767248999999, 2024-09-21T12:33:13.829 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.008228159, 2024-09-21T12:33:13.829 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:13.829 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:13.829 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:13.829 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:13.829 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:33:13.829 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:33:13.829 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 4750 2024-09-21T12:33:13.829 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.829 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:13.829 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.829 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:13.830 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.393690+0000", 2024-09-21T12:33:13.830 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.830 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.830 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.830 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:13.830 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.393690+0000", 2024-09-21T12:33:13.830 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.830 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.830 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.830 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:13.830 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.393693+0000", 2024-09-21T12:33:13.830 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0980000000000001e-06 2024-09-21T12:33:13.830 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.830 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.831 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:13.831 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.393750+0000", 2024-09-21T12:33:13.831 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7367999999999997e-05 2024-09-21T12:33:13.831 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.831 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.831 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:13.831 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.393760+0000", 2024-09-21T12:33:13.831 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0001e-05 2024-09-21T12:33:13.831 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.831 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.831 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:13.831 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.393766+0000", 2024-09-21T12:33:13.831 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7320000000000003e-06 2024-09-21T12:33:13.831 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.831 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.831 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.832 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.394595+0000", 2024-09-21T12:33:13.832 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00082910000000000004 2024-09-21T12:33:13.832 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.832 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.832 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:13.832 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.394605+0000", 2024-09-21T12:33:13.832 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9580000000000002e-06 2024-09-21T12:33:13.832 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.832 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.832 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.832 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.397764+0000", 2024-09-21T12:33:13.832 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0031589670000001 2024-09-21T12:33:13.832 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.832 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.832 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:13.833 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.397895+0000", 2024-09-21T12:33:13.833 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013103999999999999 2024-09-21T12:33:13.833 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.833 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.833 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,4", 2024-09-21T12:33:13.833 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.398023+0000", 2024-09-21T12:33:13.833 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000128594 2024-09-21T12:33:13.833 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.833 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.833 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:13.833 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.400685+0000", 2024-09-21T12:33:13.833 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0026614859999999998 2024-09-21T12:33:13.833 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.833 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.833 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.833 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401610+0000", 2024-09-21T12:33:13.834 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00092512699999999998 2024-09-21T12:33:13.834 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.834 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.834 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.834 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401802+0000", 2024-09-21T12:33:13.834 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019185099999999999 2024-09-21T12:33:13.834 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.834 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.834 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:13.834 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401871+0000", 2024-09-21T12:33:13.834 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9015000000000002e-05 2024-09-21T12:33:13.834 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.834 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.834 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:13.834 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401919+0000", 2024-09-21T12:33:13.835 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7821999999999997e-05 2024-09-21T12:33:13.835 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.835 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:13.835 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.835 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.835 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.835 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:4777 10.c 10:3cea9318:::benchmark_data_smithi114_161690_object4776:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-09-21T12:33:13.835 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:32:40.417960+0000", 2024-09-21T12:33:13.835 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 33.281497557000002, 2024-09-21T12:33:13.835 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.98421197500000002, 2024-09-21T12:33:13.835 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:13.835 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:13.835 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:13.835 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:13.836 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:33:13.836 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:33:13.836 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 4777 2024-09-21T12:33:13.836 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.836 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:13.836 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.836 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:13.836 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.417960+0000", 2024-09-21T12:33:13.836 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.836 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.836 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.836 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:13.836 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.417960+0000", 2024-09-21T12:33:13.836 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.837 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.837 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.837 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:13.837 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.417961+0000", 2024-09-21T12:33:13.837 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.313e-06 2024-09-21T12:33:13.837 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.837 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.837 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:13.837 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.418017+0000", 2024-09-21T12:33:13.837 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5729000000000001e-05 2024-09-21T12:33:13.837 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.837 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.837 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:13.837 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.418027+0000", 2024-09-21T12:33:13.837 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3789999999999995e-06 2024-09-21T12:33:13.838 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.838 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.838 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:13.838 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.418031+0000", 2024-09-21T12:33:13.838 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6380000000000003e-06 2024-09-21T12:33:13.838 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.838 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.838 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.838 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.418091+0000", 2024-09-21T12:33:13.838 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0069999999999999e-05 2024-09-21T12:33:13.838 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.838 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.838 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:13.838 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.418100+0000", 2024-09-21T12:33:13.839 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1500000000000005e-06 2024-09-21T12:33:13.839 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.839 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.839 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.839 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.398597+0000", 2024-09-21T12:33:13.839 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.98049644000000002 2024-09-21T12:33:13.839 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.839 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.839 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:13.840 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.398697+0000", 2024-09-21T12:33:13.841 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100028 2024-09-21T12:33:13.841 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.841 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.841 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,4", 2024-09-21T12:33:13.841 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.398833+0000", 2024-09-21T12:33:13.841 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013587000000000001 2024-09-21T12:33:13.841 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.841 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.841 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:13.841 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.400690+0000", 2024-09-21T12:33:13.841 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001857455 2024-09-21T12:33:13.841 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.841 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.842 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.842 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401652+0000", 2024-09-21T12:33:13.842 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00096136099999999999 2024-09-21T12:33:13.842 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.842 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.842 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.842 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.402070+0000", 2024-09-21T12:33:13.842 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00041827100000000001 2024-09-21T12:33:13.842 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.842 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.842 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:13.842 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.402129+0000", 2024-09-21T12:33:13.842 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9407000000000002e-05 2024-09-21T12:33:13.842 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.842 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.843 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:13.843 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.402172+0000", 2024-09-21T12:33:13.843 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2864000000000002e-05 2024-09-21T12:33:13.843 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.843 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:13.843 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.843 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.843 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.843 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:4795 10.c 10:3d88164d:::benchmark_data_smithi114_161690_object4794:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-09-21T12:33:13.843 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:32:40.430393+0000", 2024-09-21T12:33:13.843 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 33.269064743000001, 2024-09-21T12:33:13.843 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.97250018500000002, 2024-09-21T12:33:13.843 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:13.844 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:13.844 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:13.844 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:13.844 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:33:13.844 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:33:13.844 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 4795 2024-09-21T12:33:13.844 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.844 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:13.844 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.844 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:13.844 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.430393+0000", 2024-09-21T12:33:13.844 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.844 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.844 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.844 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:13.844 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.430393+0000", 2024-09-21T12:33:13.845 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.845 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.845 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.845 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:13.845 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.430395+0000", 2024-09-21T12:33:13.845 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.917e-06 2024-09-21T12:33:13.845 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.845 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.845 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:13.845 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.430505+0000", 2024-09-21T12:33:13.845 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000110171 2024-09-21T12:33:13.845 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.845 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.845 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:13.845 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.430523+0000", 2024-09-21T12:33:13.846 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8124000000000001e-05 2024-09-21T12:33:13.846 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.846 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.846 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:13.846 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.430533+0000", 2024-09-21T12:33:13.846 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3449999999999995e-06 2024-09-21T12:33:13.846 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.846 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.846 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.846 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.430612+0000", 2024-09-21T12:33:13.846 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9912000000000003e-05 2024-09-21T12:33:13.846 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.846 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.846 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:13.847 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.430623+0000", 2024-09-21T12:33:13.847 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.024e-05 2024-09-21T12:33:13.847 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.847 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.847 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.847 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.399438+0000", 2024-09-21T12:33:13.847 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.96881504399999996 2024-09-21T12:33:13.847 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.847 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.847 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:13.847 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.399548+0000", 2024-09-21T12:33:13.847 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011024 2024-09-21T12:33:13.847 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.847 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.847 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,4", 2024-09-21T12:33:13.847 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.399687+0000", 2024-09-21T12:33:13.848 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013893600000000001 2024-09-21T12:33:13.848 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.848 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.848 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:13.848 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.400696+0000", 2024-09-21T12:33:13.848 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010094800000000001 2024-09-21T12:33:13.848 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.848 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.848 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.848 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401686+0000", 2024-09-21T12:33:13.848 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00098962399999999993 2024-09-21T12:33:13.848 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.848 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.848 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.848 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.402774+0000", 2024-09-21T12:33:13.849 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010883500000000001 2024-09-21T12:33:13.849 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.849 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.849 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:13.849 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.402842+0000", 2024-09-21T12:33:13.849 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7959000000000004e-05 2024-09-21T12:33:13.849 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.849 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.849 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:13.849 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.402893+0000", 2024-09-21T12:33:13.849 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0843000000000001e-05 2024-09-21T12:33:13.849 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.849 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:13.849 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.849 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.850 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.850 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:4803 10.c 10:349a15c7:::benchmark_data_smithi114_161690_object4802:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-09-21T12:33:13.850 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:32:40.435931+0000", 2024-09-21T12:33:13.850 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 33.263527044, 2024-09-21T12:33:13.850 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.96709726100000004, 2024-09-21T12:33:13.850 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:33:13.850 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:33:13.850 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:13.850 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:33:13.850 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:33:13.850 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:33:13.850 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 4803 2024-09-21T12:33:13.850 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.850 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:33:13.850 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.851 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:33:13.851 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.435931+0000", 2024-09-21T12:33:13.851 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.851 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.851 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.851 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:33:13.851 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.435931+0000", 2024-09-21T12:33:13.851 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:33:13.851 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.851 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.851 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:33:13.851 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.435932+0000", 2024-09-21T12:33:13.851 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.206e-06 2024-09-21T12:33:13.852 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.852 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.852 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:33:13.852 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.435987+0000", 2024-09-21T12:33:13.852 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4925e-05 2024-09-21T12:33:13.852 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.852 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.852 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:33:13.852 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.435998+0000", 2024-09-21T12:33:13.852 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1589e-05 2024-09-21T12:33:13.852 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.852 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.852 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:33:13.852 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.436005+0000", 2024-09-21T12:33:13.852 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8040000000000004e-06 2024-09-21T12:33:13.853 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.853 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.853 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.853 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.436048+0000", 2024-09-21T12:33:13.853 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2719999999999998e-05 2024-09-21T12:33:13.853 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.853 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.853 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:33:13.853 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.436055+0000", 2024-09-21T12:33:13.853 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8419999999999999e-06 2024-09-21T12:33:13.853 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.853 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.853 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:33:13.853 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.400105+0000", 2024-09-21T12:33:13.853 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.964049932 2024-09-21T12:33:13.854 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.854 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.854 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:33:13.854 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.400218+0000", 2024-09-21T12:33:13.854 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011312200000000001 2024-09-21T12:33:13.854 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.854 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.854 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,4", 2024-09-21T12:33:13.854 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.400335+0000", 2024-09-21T12:33:13.854 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000117068 2024-09-21T12:33:13.854 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.854 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.854 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.854 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401732+0000", 2024-09-21T12:33:13.854 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001397421 2024-09-21T12:33:13.854 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.855 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.855 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:33:13.855 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401754+0000", 2024-09-21T12:33:13.855 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1795000000000001e-05 2024-09-21T12:33:13.855 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.855 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.855 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:13.855 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.402939+0000", 2024-09-21T12:33:13.855 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0011850369999999999 2024-09-21T12:33:13.855 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.855 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.855 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:33:13.855 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.402993+0000", 2024-09-21T12:33:13.856 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3779999999999997e-05 2024-09-21T12:33:13.856 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:33:13.856 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:33:13.856 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:33:13.856 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.403028+0000", 2024-09-21T12:33:13.856 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.502e-05 2024-09-21T12:33:13.856 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.856 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:13.856 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.856 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:33:13.856 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:33:13.856 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:33:13.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:13.857 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:13.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:13.958 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:13.994 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:13.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:13.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:14.096 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:14.132 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:14.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:14.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:14.222 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T12:33:14.223 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T12:33:14.223 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.6 'trim stale osdmaps' 2024-09-21T12:33:14.234 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:14.242 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:14.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:14.292 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:14.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:14.393 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:14.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:14.454 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:14.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:14.555 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:14.584 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:14.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:14.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:14.656 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 38 16 62772 62756 103.19 86.3125 0.00512791 0.00968586 2024-09-21T12:33:14.685 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:14.716 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:14.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:14.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:14.818 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:14.847 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:14.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:14.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:14.949 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:14.990 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:14.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:14.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:15.027 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T12:33:15.092 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:15.124 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:15.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:15.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:15.227 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:15.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:15 smithi183 ceph-mon[33277]: pgmap v666: 178 pgs: 178 active+clean; 3.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 753 B/s rd, 103 MiB/s wr, 1.64k op/s 2024-09-21T12:33:15.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:15.272 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:15.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:15.373 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:15.394 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:15.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:15.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:15.496 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:15.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:15 smithi114 ceph-mon[40867]: pgmap v666: 178 pgs: 178 active+clean; 3.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 753 B/s rd, 103 MiB/s wr, 1.64k op/s 2024-09-21T12:33:15.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:15 smithi114 ceph-mon[30324]: pgmap v666: 178 pgs: 178 active+clean; 3.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 753 B/s rd, 103 MiB/s wr, 1.64k op/s 2024-09-21T12:33:15.538 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:15.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:15.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:15.640 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:15.656 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 39 16 64021 64005 102.546 78.0625 0.0128056 0.00974653 2024-09-21T12:33:15.683 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:15.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:15.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:15.784 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:15.826 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:15.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:15.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:15.858 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:33:15.927 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:15.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:15.962 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:15.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:16.063 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:16.106 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:16.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:16.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:16.207 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:16.229 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-21T12:33:16.230 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:16 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2982578549' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:33:16.231 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:16 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2982578549' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:33:16.244 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:16.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:16.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:16.346 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:16.382 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:16.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:16.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:16.484 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:16 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2982578549' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:33:16.522 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:16.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:16.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:16.624 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:16.656 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-21T12:33:16.655762+0000 min lat: 0.00124586 max lat: 2.02977 avg lat: 0.00982379 2024-09-21T12:33:16.656 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T12:33:16.656 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 40 16 65147 65131 101.742 70.375 0.00687618 0.00982379 2024-09-21T12:33:16.658 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:16.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:16.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:16.760 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:16.780 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:16.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:16.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:16.882 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:16.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:16.916 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:16.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:17.017 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:17.039 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:17.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:17.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:17.141 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:17.181 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:17.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:17.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:17.283 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:17.304 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:17.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:17.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:17.371 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:17 smithi183 ceph-mon[33277]: pgmap v667: 178 pgs: 178 active+clean; 3.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-09-21T12:33:17.371 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:17 smithi183 ceph-mon[33277]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:33:17.371 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:17 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2982578549' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:33:17.371 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:17 smithi183 ceph-mon[33277]: osdmap e365: 8 total, 8 up, 8 in 2024-09-21T12:33:17.371 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:17 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:33:17.371 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:17 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:33:17.371 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:17 smithi183 ceph-mon[33277]: osdmap e366: 8 total, 8 up, 8 in 2024-09-21T12:33:17.406 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:17.443 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:17.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:17.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:17.516 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:17 smithi114 ceph-mon[30324]: pgmap v667: 178 pgs: 178 active+clean; 3.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-09-21T12:33:17.516 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:17 smithi114 ceph-mon[30324]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:33:17.516 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:17 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2982578549' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:33:17.516 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:17 smithi114 ceph-mon[30324]: osdmap e365: 8 total, 8 up, 8 in 2024-09-21T12:33:17.516 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:17 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:33:17.516 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:17 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:33:17.517 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:17 smithi114 ceph-mon[30324]: osdmap e366: 8 total, 8 up, 8 in 2024-09-21T12:33:17.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:17 smithi114 ceph-mon[40867]: pgmap v667: 178 pgs: 178 active+clean; 3.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-09-21T12:33:17.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:17 smithi114 ceph-mon[40867]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:33:17.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:17 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2982578549' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:33:17.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:17 smithi114 ceph-mon[40867]: osdmap e365: 8 total, 8 up, 8 in 2024-09-21T12:33:17.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:17 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:33:17.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:17 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:33:17.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:17 smithi114 ceph-mon[40867]: osdmap e366: 8 total, 8 up, 8 in 2024-09-21T12:33:17.544 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:17.573 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:17.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:17.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:17.656 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 41 16 66489 66473 101.305 83.875 0.00812371 0.00986672 2024-09-21T12:33:17.674 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:17.717 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:17.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:17.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:17.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:33:17 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:33:17.818 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:17.850 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:17.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:17.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:17.952 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:17.982 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:17.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:17.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:18.084 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:18.115 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:18.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:18.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:18.164 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:18.216 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:18.217 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:18.248 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:18.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:18.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:18.313 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:18 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:33:18.313 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:18 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:33:18.313 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:18 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:18.313 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:18 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:33:18.313 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:18 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:33:18.313 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:18 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:18.350 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:18.386 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:18.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:18.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:18.488 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:18.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:18 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:33:18.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:18 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:33:18.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:18 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:18.526 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:18.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:18.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:18.627 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:18.657 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 42 15 67846 67831 100.913 84.875 0.0157703 0.00990519 2024-09-21T12:33:18.660 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:18.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:18.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:18.762 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:18.802 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:18.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:18.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:18.904 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:18.955 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:18.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:18.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:19.056 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:19.085 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:19.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:19.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:19.178 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:19 smithi114 ceph-mon[40867]: pgmap v670: 178 pgs: 178 active+clean; 3.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 98 MiB/s wr, 1.58k op/s 2024-09-21T12:33:19.178 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:19 smithi114 ceph-mon[40867]: osdmap e367: 8 total, 8 up, 8 in 2024-09-21T12:33:19.179 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:19 smithi114 ceph-mon[30324]: pgmap v670: 178 pgs: 178 active+clean; 3.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 98 MiB/s wr, 1.58k op/s 2024-09-21T12:33:19.179 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:19 smithi114 ceph-mon[30324]: osdmap e367: 8 total, 8 up, 8 in 2024-09-21T12:33:19.186 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:19.227 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:19.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:19.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:19.271 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:33:19.272 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:33:19.305 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:33:19.305 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:33:19.329 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:19.340 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:33:19.340 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:33:19.364 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:19.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:19.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:19.376 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:33:19.376 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:33:19.408 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:33:19.409 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:33:19.443 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:33:19.443 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:33:19.466 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:19.484 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:33:19.484 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:33:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:19 smithi183 ceph-mon[33277]: pgmap v670: 178 pgs: 178 active+clean; 3.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 98 MiB/s wr, 1.58k op/s 2024-09-21T12:33:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:19 smithi183 ceph-mon[33277]: osdmap e367: 8 total, 8 up, 8 in 2024-09-21T12:33:19.495 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:19.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:19.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:19.524 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:33:19.524 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:33:19.596 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:19.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:19.636 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:19.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:19.640 INFO:teuthology.orchestra.run.smithi114.stderr:no valid command found; 10 closest matches: 2024-09-21T12:33:19.640 INFO:teuthology.orchestra.run.smithi114.stderr:0 2024-09-21T12:33:19.641 INFO:teuthology.orchestra.run.smithi114.stderr:1 2024-09-21T12:33:19.641 INFO:teuthology.orchestra.run.smithi114.stderr:2 2024-09-21T12:33:19.641 INFO:teuthology.orchestra.run.smithi114.stderr:abort 2024-09-21T12:33:19.641 INFO:teuthology.orchestra.run.smithi114.stderr:assert 2024-09-21T12:33:19.641 INFO:teuthology.orchestra.run.smithi114.stderr:bench [] [] [] [] 2024-09-21T12:33:19.641 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs debug_inject_read_zeros 2024-09-21T12:33:19.641 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs files list 2024-09-21T12:33:19.641 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs stats 2024-09-21T12:33:19.641 INFO:teuthology.orchestra.run.smithi114.stderr:bluestore allocator dump block 2024-09-21T12:33:19.641 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: invalid command 2024-09-21T12:33:19.657 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 43 16 69138 69122 100.442 80.6875 0.0125003 0.00995126 2024-09-21T12:33:19.737 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:19.771 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:19.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:19.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:19.873 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:19.910 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:19.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:19.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:20.012 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:20.045 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:20.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:20.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:20.135 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T12:33:20.147 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:20.174 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:20.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:20.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:20.276 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:20.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:20.327 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:20.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:20.428 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:20.451 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:20.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:20.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:20 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/436541009' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:33:20.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:20 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/436541009' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:33:20.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:20 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/436541009' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:33:20.553 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:20.586 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:20.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:20.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:20.651 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T12:33:20.652 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T12:33:20.652 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.7 'trim stale osdmaps' 2024-09-21T12:33:20.658 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 44 16 70366 70350 99.9033 76.75 0.0248039 0.010004 2024-09-21T12:33:20.687 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:20.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:20.729 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:20.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:20.830 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:20.870 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:20.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:20.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:20.972 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:21.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:21.004 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:21.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:21.105 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:21.127 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:21.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:21.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:21.130 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:21.195 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:21.229 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:21.261 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:21.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:21.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:21.363 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:21.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:21 smithi114 ceph-mon[30324]: pgmap v672: 178 pgs: 178 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-21T12:33:21.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:21 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:33:21.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:21 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:33:21.374 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:21 smithi114 ceph-mon[40867]: pgmap v672: 178 pgs: 178 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-21T12:33:21.374 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:21 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:33:21.374 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:21 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:33:21.412 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:21.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:21.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:21.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:21 smithi183 ceph-mon[33277]: pgmap v672: 178 pgs: 178 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-09-21T12:33:21.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:21 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:33:21.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:21 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:33:21.514 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:21.548 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:21.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:21.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:21.650 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:21.657 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 45 16 71645 71629 99.4595 79.9375 0.00488695 0.0100492 2024-09-21T12:33:21.682 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:21.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:21.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:21.784 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:21.832 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:21.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:21.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:21.934 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:21.966 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:21.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:21.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:22.068 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:22.099 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:22.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:22.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:22.131 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:22.150 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:33:22.150 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:33:22.201 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:22.246 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:22.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:22.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:22.348 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:22.380 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:22.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:22.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:22.482 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:22.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:22 smithi183 ceph-mon[33277]: osdmap e368: 8 total, 8 up, 8 in 2024-09-21T12:33:22.514 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:22.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:22.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:22.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:22 smithi114 ceph-mon[30324]: osdmap e368: 8 total, 8 up, 8 in 2024-09-21T12:33:22.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:22 smithi114 ceph-mon[40867]: osdmap e368: 8 total, 8 up, 8 in 2024-09-21T12:33:22.616 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:22.648 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:22.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:22.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:22.657 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 46 16 72967 72951 99.0934 82.625 0.0103801 0.0100865 2024-09-21T12:33:22.751 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:22.786 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:22.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:22.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:22.887 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:22.919 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:22.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:22.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:23.021 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:23.024 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:33:22 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:33:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:33:23.042 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:23.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:23.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:23.144 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:23.178 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:23.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:23.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:23.185 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:33:23.279 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:23.308 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:23.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:23.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:23.410 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:23.438 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:23.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:23.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:23.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:23 smithi183 ceph-mon[33277]: pgmap v674: 178 pgs: 178 active+clean; 4.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-09-21T12:33:23.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:23 smithi114 ceph-mon[40867]: pgmap v674: 178 pgs: 178 active+clean; 4.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-09-21T12:33:23.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:23 smithi114 ceph-mon[30324]: pgmap v674: 178 pgs: 178 active+clean; 4.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-09-21T12:33:23.541 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:23.579 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:23.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:23.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:23.657 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 47 16 74318 74302 98.7815 84.4375 0.0117902 0.0101188 2024-09-21T12:33:23.666 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T12:33:23.680 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:23.752 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:23.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:23.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:23.853 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:23.884 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:23.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:23.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:23.985 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:24.025 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:24.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:24.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:24.126 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:24.161 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:24.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:24.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:24.264 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:24.301 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:24.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:24.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:24.403 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:24.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:24.439 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:24.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:24.540 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:24.598 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:24.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:24.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:no valid command found; 10 closest matches: 2024-09-21T12:33:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:0 2024-09-21T12:33:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:1 2024-09-21T12:33:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:2 2024-09-21T12:33:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:abort 2024-09-21T12:33:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:assert 2024-09-21T12:33:24.620 INFO:teuthology.orchestra.run.smithi114.stderr:bench [] [] [] [] 2024-09-21T12:33:24.621 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs debug_inject_read_zeros 2024-09-21T12:33:24.621 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs files list 2024-09-21T12:33:24.621 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs stats 2024-09-21T12:33:24.621 INFO:teuthology.orchestra.run.smithi114.stderr:bluestore allocator dump block 2024-09-21T12:33:24.621 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: invalid command 2024-09-21T12:33:24.658 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 48 16 75556 75540 98.3345 77.375 0.0155476 0.010164 2024-09-21T12:33:24.700 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:24.727 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:24.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:24.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:24.829 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:24.869 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:24.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:24.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:24.971 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:25.008 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:25.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:25.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:25.110 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:25.119 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-21T12:33:25.145 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:25.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:25.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:25.247 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:25.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:25 smithi114 ceph-mon[30324]: pgmap v675: 178 pgs: 178 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 752 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-09-21T12:33:25.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:25 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3438600886' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:33:25.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:25 smithi114 ceph-mon[40867]: pgmap v675: 178 pgs: 178 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 752 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-09-21T12:33:25.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:25 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3438600886' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:33:25.280 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:25.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:25.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:25.382 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:25.415 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:25.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:25.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:25.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:25 smithi183 ceph-mon[33277]: pgmap v675: 178 pgs: 178 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 752 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-09-21T12:33:25.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:25 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3438600886' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:33:25.517 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:25.539 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T12:33:25.539 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T12:33:25.540 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.4 'trim stale osdmaps' 2024-09-21T12:33:25.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:25.553 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:25.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:25.655 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:25.659 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 49 16 76936 76920 98.0863 86.25 0.00733774 0.0101902 2024-09-21T12:33:25.678 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:25.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:25.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:25.780 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:25.813 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:25.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:25.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:25.915 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:25.950 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:25.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:25.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:26.052 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:26.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:26.091 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:26.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:26.192 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:26.222 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:26.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:26.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:26.278 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:26.324 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:26.358 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:26.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:26.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:26.390 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:26 smithi114 ceph-mon[30324]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:33:26.390 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:26 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3438600886' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:33:26.390 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:26 smithi114 ceph-mon[30324]: osdmap e369: 8 total, 8 up, 8 in 2024-09-21T12:33:26.390 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:26 smithi114 ceph-mon[40867]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:33:26.391 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:26 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3438600886' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:33:26.391 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:26 smithi114 ceph-mon[40867]: osdmap e369: 8 total, 8 up, 8 in 2024-09-21T12:33:26.460 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:26.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:26 smithi183 ceph-mon[33277]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:33:26.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:26 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3438600886' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:33:26.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:26 smithi183 ceph-mon[33277]: osdmap e369: 8 total, 8 up, 8 in 2024-09-21T12:33:26.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:26.496 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:26.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:26.597 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:26.638 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:26.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:26.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:26.659 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 50 16 78237 78221 97.7507 81.3125 0.00948514 0.0102254 2024-09-21T12:33:26.740 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:26.774 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:26.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:26.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:26.875 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:26.876 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:26.904 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:26.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:26.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:27.005 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:27.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:27.051 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:27.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:27.151 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:27.176 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:27.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:27.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:27.277 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:27.313 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:27.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:27.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:27.414 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:27.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:27.457 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:27.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:27.458 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:27 smithi114 ceph-mon[30324]: 4.5 scrub starts 2024-09-21T12:33:27.458 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:27 smithi114 ceph-mon[30324]: 4.5 scrub ok 2024-09-21T12:33:27.458 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:27 smithi114 ceph-mon[30324]: pgmap v677: 178 pgs: 178 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 658 B/s rd, 53 MiB/s wr, 840 op/s 2024-09-21T12:33:27.459 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:27 smithi114 ceph-mon[30324]: osdmap e370: 8 total, 8 up, 8 in 2024-09-21T12:33:27.459 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:27 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:33:27.459 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:27 smithi114 ceph-mon[40867]: 4.5 scrub starts 2024-09-21T12:33:27.459 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:27 smithi114 ceph-mon[40867]: 4.5 scrub ok 2024-09-21T12:33:27.459 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:27 smithi114 ceph-mon[40867]: pgmap v677: 178 pgs: 178 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 658 B/s rd, 53 MiB/s wr, 840 op/s 2024-09-21T12:33:27.459 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:27 smithi114 ceph-mon[40867]: osdmap e370: 8 total, 8 up, 8 in 2024-09-21T12:33:27.459 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:27 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:33:27.488 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:33:27 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:33:27.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:27 smithi183 ceph-mon[33277]: 4.5 scrub starts 2024-09-21T12:33:27.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:27 smithi183 ceph-mon[33277]: 4.5 scrub ok 2024-09-21T12:33:27.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:27 smithi183 ceph-mon[33277]: pgmap v677: 178 pgs: 178 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 658 B/s rd, 53 MiB/s wr, 840 op/s 2024-09-21T12:33:27.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:27 smithi183 ceph-mon[33277]: osdmap e370: 8 total, 8 up, 8 in 2024-09-21T12:33:27.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:27 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:33:27.559 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:27.593 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:27.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:27.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:27.659 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 51 16 79486 79470 97.3646 78.0625 0.00896741 0.0102654 2024-09-21T12:33:27.694 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:27.723 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:27.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:27.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:27.825 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:27.862 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:33:27.862 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:33:27.862 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:33:27.862 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:33:27.862 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:33:27.870 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:27.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:27.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:27.909 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:33:27.910 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:33:27.910 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:33:27.910 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:33:27.910 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:33:27.910 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:33:27.910 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:33:27.910 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:33:27.949 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:33:27.949 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:33:27.971 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:27.993 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:33:27.993 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:33:28.004 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:28.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:28.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:28.028 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:33:28.028 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:33:28.062 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:33:28.062 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:33:28.107 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:28.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:28.141 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:28.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:28.242 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:28.274 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:28.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:28.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:28.375 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:28.417 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:28.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:28.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:28.428 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:28 smithi114 ceph-mon[30324]: 4.1b scrub starts 2024-09-21T12:33:28.428 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:28 smithi114 ceph-mon[30324]: 4.1b scrub ok 2024-09-21T12:33:28.428 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:28 smithi114 ceph-mon[30324]: 6.8 scrub starts 2024-09-21T12:33:28.428 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:28 smithi114 ceph-mon[30324]: 1.18 scrub starts 2024-09-21T12:33:28.428 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:28 smithi114 ceph-mon[30324]: 1.18 scrub ok 2024-09-21T12:33:28.428 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:28 smithi114 ceph-mon[30324]: 6.8 scrub ok 2024-09-21T12:33:28.429 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:28 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:28.429 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:28 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1314411178' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:33:28.429 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:28 smithi114 ceph-mon[40867]: 4.1b scrub starts 2024-09-21T12:33:28.429 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:28 smithi114 ceph-mon[40867]: 4.1b scrub ok 2024-09-21T12:33:28.429 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:28 smithi114 ceph-mon[40867]: 6.8 scrub starts 2024-09-21T12:33:28.429 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:28 smithi114 ceph-mon[40867]: 1.18 scrub starts 2024-09-21T12:33:28.429 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:28 smithi114 ceph-mon[40867]: 1.18 scrub ok 2024-09-21T12:33:28.429 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:28 smithi114 ceph-mon[40867]: 6.8 scrub ok 2024-09-21T12:33:28.429 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:28 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:28.429 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:28 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1314411178' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:33:28.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:28 smithi183 ceph-mon[33277]: 4.1b scrub starts 2024-09-21T12:33:28.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:28 smithi183 ceph-mon[33277]: 4.1b scrub ok 2024-09-21T12:33:28.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:28 smithi183 ceph-mon[33277]: 6.8 scrub starts 2024-09-21T12:33:28.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:28 smithi183 ceph-mon[33277]: 1.18 scrub starts 2024-09-21T12:33:28.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:28 smithi183 ceph-mon[33277]: 1.18 scrub ok 2024-09-21T12:33:28.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:28 smithi183 ceph-mon[33277]: 6.8 scrub ok 2024-09-21T12:33:28.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:28 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:28.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:28 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1314411178' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:33:28.519 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:28.555 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:28.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:28.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:28.657 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:28.660 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 52 16 80751 80735 97.0126 79.0625 0.00912247 0.0103032 2024-09-21T12:33:28.692 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:28.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:28.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:28.794 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:28.835 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:28.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:28.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:28.845 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T12:33:28.936 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:28.974 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:28.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:28.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:29.076 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:29.107 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:29.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:29.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:29.209 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:29.249 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:29.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:29.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:29.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:29 smithi114 ceph-mon[40867]: 6.1a scrub starts 2024-09-21T12:33:29.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:29 smithi114 ceph-mon[40867]: 6.1a scrub ok 2024-09-21T12:33:29.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:29 smithi114 ceph-mon[40867]: 4.11 scrub starts 2024-09-21T12:33:29.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:29 smithi114 ceph-mon[40867]: 4.11 scrub ok 2024-09-21T12:33:29.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:29 smithi114 ceph-mon[40867]: pgmap v679: 178 pgs: 178 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.5 KiB/s rd, 110 MiB/s wr, 1.77k op/s 2024-09-21T12:33:29.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:29 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:33:29.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:29 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:33:29.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:29 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:33:29.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:29 smithi114 ceph-mon[30324]: 6.1a scrub starts 2024-09-21T12:33:29.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:29 smithi114 ceph-mon[30324]: 6.1a scrub ok 2024-09-21T12:33:29.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:29 smithi114 ceph-mon[30324]: 4.11 scrub starts 2024-09-21T12:33:29.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:29 smithi114 ceph-mon[30324]: 4.11 scrub ok 2024-09-21T12:33:29.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:29 smithi114 ceph-mon[30324]: pgmap v679: 178 pgs: 178 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.5 KiB/s rd, 110 MiB/s wr, 1.77k op/s 2024-09-21T12:33:29.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:29 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:33:29.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:29 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:33:29.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:29 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:33:29.353 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:29.384 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:29.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:29.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:29.486 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:29.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:29 smithi183 ceph-mon[33277]: 6.1a scrub starts 2024-09-21T12:33:29.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:29 smithi183 ceph-mon[33277]: 6.1a scrub ok 2024-09-21T12:33:29.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:29 smithi183 ceph-mon[33277]: 4.11 scrub starts 2024-09-21T12:33:29.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:29 smithi183 ceph-mon[33277]: 4.11 scrub ok 2024-09-21T12:33:29.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:29 smithi183 ceph-mon[33277]: pgmap v679: 178 pgs: 178 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.5 KiB/s rd, 110 MiB/s wr, 1.77k op/s 2024-09-21T12:33:29.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:29 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:33:29.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:29 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:33:29.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:29 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:33:29.519 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:29.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:29.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:29.620 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:29.656 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:29.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:29.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:29.659 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 53 16 82014 81998 96.6714 78.9375 0.00970095 0.0103395 2024-09-21T12:33:29.758 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:29.796 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:29.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:29.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:29.898 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:29.903 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T12:33:29.935 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:29.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:29.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:29.952 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:30.037 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:30.069 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:30.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:30.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:30.170 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:30.215 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:30.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:30.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:30.277 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:30.318 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:30.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:30.375 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:30.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:30.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:30 smithi114 ceph-mon[40867]: 3.1f scrub starts 2024-09-21T12:33:30.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:30 smithi114 ceph-mon[40867]: 3.1f scrub ok 2024-09-21T12:33:30.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:30 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:33:30.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:30 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:33:30.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:30 smithi114 ceph-mon[30324]: 3.1f scrub starts 2024-09-21T12:33:30.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:30 smithi114 ceph-mon[30324]: 3.1f scrub ok 2024-09-21T12:33:30.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:30 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:33:30.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:30 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:33:30.477 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:30.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:30 smithi183 ceph-mon[33277]: 3.1f scrub starts 2024-09-21T12:33:30.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:30 smithi183 ceph-mon[33277]: 3.1f scrub ok 2024-09-21T12:33:30.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:30 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:33:30.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:30 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:33:30.519 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:30.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:30.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:30.621 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:30.658 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:30.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:30.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:30.660 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 54 16 83270 83254 96.3346 78.5 0.00833888 0.010375 2024-09-21T12:33:30.761 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:30.798 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:30.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:30.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:30.900 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:30.972 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:30.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:30.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:31.032 INFO:teuthology.orchestra.run.smithi114.stderr:no valid command found; 10 closest matches: 2024-09-21T12:33:31.032 INFO:teuthology.orchestra.run.smithi114.stderr:0 2024-09-21T12:33:31.032 INFO:teuthology.orchestra.run.smithi114.stderr:1 2024-09-21T12:33:31.032 INFO:teuthology.orchestra.run.smithi114.stderr:2 2024-09-21T12:33:31.032 INFO:teuthology.orchestra.run.smithi114.stderr:abort 2024-09-21T12:33:31.032 INFO:teuthology.orchestra.run.smithi114.stderr:assert 2024-09-21T12:33:31.032 INFO:teuthology.orchestra.run.smithi114.stderr:bench [] [] [] [] 2024-09-21T12:33:31.032 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs debug_inject_read_zeros 2024-09-21T12:33:31.032 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs files list 2024-09-21T12:33:31.032 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs stats 2024-09-21T12:33:31.033 INFO:teuthology.orchestra.run.smithi114.stderr:bluestore allocator dump block 2024-09-21T12:33:31.033 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: invalid command 2024-09-21T12:33:31.073 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:31.101 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:31.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:31.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:31.203 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:31.239 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:31.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:31.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:31.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:31 smithi114 ceph-mon[30324]: pgmap v680: 178 pgs: 178 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-09-21T12:33:31.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:31 smithi114 ceph-mon[30324]: osdmap e371: 8 total, 8 up, 8 in 2024-09-21T12:33:31.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:31 smithi114 ceph-mon[40867]: pgmap v680: 178 pgs: 178 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-09-21T12:33:31.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:31 smithi114 ceph-mon[40867]: osdmap e371: 8 total, 8 up, 8 in 2024-09-21T12:33:31.340 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:31.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:31.378 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:31.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:31.402 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:33:31.450 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:31.479 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:31.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:31 smithi183 ceph-mon[33277]: pgmap v680: 178 pgs: 178 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-09-21T12:33:31.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:31 smithi183 ceph-mon[33277]: osdmap e371: 8 total, 8 up, 8 in 2024-09-21T12:33:31.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:31.545 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:31.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:31.646 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:31.662 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 55 16 84485 84469 95.959 75.9375 0.016496 0.0104162 2024-09-21T12:33:31.681 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:31.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:31.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:31.783 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:31.812 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:31.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:31.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:31.914 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:31.947 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:31.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:31.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:32.049 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:32.076 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:32.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:32.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:32.178 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:32.193 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-21T12:33:32.213 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:32.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:32.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:32.315 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:32.348 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:32.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:32.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:32.437 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T12:33:32.438 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T12:33:32.449 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:32.485 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:32.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:32.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:32 smithi183 ceph-mon[33277]: 3.14 scrub starts 2024-09-21T12:33:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:32 smithi183 ceph-mon[33277]: 3.14 scrub ok 2024-09-21T12:33:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:32 smithi183 ceph-mon[33277]: 4.b scrub starts 2024-09-21T12:33:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:32 smithi183 ceph-mon[33277]: 4.b scrub ok 2024-09-21T12:33:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:32 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:33:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:32 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:33:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:32 smithi183 ceph-mon[33277]: osdmap e372: 8 total, 8 up, 8 in 2024-09-21T12:33:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:32 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1071951439' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:33:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:32 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:33:32.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:32 smithi114 ceph-mon[40867]: 3.14 scrub starts 2024-09-21T12:33:32.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:32 smithi114 ceph-mon[40867]: 3.14 scrub ok 2024-09-21T12:33:32.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:32 smithi114 ceph-mon[40867]: 4.b scrub starts 2024-09-21T12:33:32.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:32 smithi114 ceph-mon[40867]: 4.b scrub ok 2024-09-21T12:33:32.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:32 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:33:32.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:32 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:33:32.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:32 smithi114 ceph-mon[40867]: osdmap e372: 8 total, 8 up, 8 in 2024-09-21T12:33:32.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:32 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1071951439' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:33:32.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:32 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:33:32.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:32 smithi114 ceph-mon[30324]: 3.14 scrub starts 2024-09-21T12:33:32.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:32 smithi114 ceph-mon[30324]: 3.14 scrub ok 2024-09-21T12:33:32.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:32 smithi114 ceph-mon[30324]: 4.b scrub starts 2024-09-21T12:33:32.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:32 smithi114 ceph-mon[30324]: 4.b scrub ok 2024-09-21T12:33:32.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:32 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:33:32.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:32 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:33:32.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:32 smithi114 ceph-mon[30324]: osdmap e372: 8 total, 8 up, 8 in 2024-09-21T12:33:32.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:32 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1071951439' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:33:32.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:32 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:33:32.587 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:32.624 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:32.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:32.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:32.662 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 56 16 85871 85855 95.7923 86.625 0.0118773 0.0104347 2024-09-21T12:33:32.726 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:32.764 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:32.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:32.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:32.849 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:33:32 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:33:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:33:32.866 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:32.905 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:32.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:32.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:33.007 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:33.023 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:33:33.023 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 20, 2024-09-21T12:33:33.023 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 600, 2024-09-21T12:33:33.023 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-21T12:33:33.023 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.023 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:221947 9.0 9:0754e963:::benchmark_data_smithi114_133214_object97913:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e308)", 2024-09-21T12:33:33.023 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:39.054189+0000", 2024-09-21T12:33:33.023 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 113.96403089099999, 2024-09-21T12:33:33.023 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.068511776, 2024-09-21T12:33:33.023 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:33.023 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:33.024 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:33.024 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:33.024 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:33:33.024 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:33.024 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 221947 2024-09-21T12:33:33.024 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.024 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:33.024 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.024 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:33.024 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.054189+0000", 2024-09-21T12:33:33.024 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.024 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.024 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.024 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:33.024 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.054189+0000", 2024-09-21T12:33:33.024 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.025 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.025 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.025 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:33.025 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.054190+0000", 2024-09-21T12:33:33.025 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.463e-06 2024-09-21T12:33:33.025 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.025 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.025 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:33.025 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.054195+0000", 2024-09-21T12:33:33.025 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4100000000000001e-06 2024-09-21T12:33:33.025 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.025 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.025 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:33.025 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.054206+0000", 2024-09-21T12:33:33.025 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0849000000000001e-05 2024-09-21T12:33:33.026 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.026 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.026 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:33.026 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.054216+0000", 2024-09-21T12:33:33.026 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.9369999999999999e-06 2024-09-21T12:33:33.026 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.026 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.026 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.026 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.054276+0000", 2024-09-21T12:33:33.026 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9939999999999999e-05 2024-09-21T12:33:33.026 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.026 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.026 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:33.026 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.054286+0000", 2024-09-21T12:33:33.026 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0560999999999999e-05 2024-09-21T12:33:33.027 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.027 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.027 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.027 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:40.116275+0000", 2024-09-21T12:33:33.027 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0619885529999999 2024-09-21T12:33:33.027 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.027 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.027 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:33.027 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:40.116393+0000", 2024-09-21T12:33:33.027 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000118628 2024-09-21T12:33:33.027 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.027 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.027 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,2,6", 2024-09-21T12:33:33.027 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:40.116712+0000", 2024-09-21T12:33:33.027 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00031853300000000002 2024-09-21T12:33:33.028 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.028 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.028 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.028 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:40.122457+0000", 2024-09-21T12:33:33.028 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0057452199999999997 2024-09-21T12:33:33.028 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.028 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.028 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.028 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:40.122517+0000", 2024-09-21T12:33:33.028 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0108000000000001e-05 2024-09-21T12:33:33.028 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.028 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.028 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.028 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:40.122566+0000", 2024-09-21T12:33:33.028 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8980000000000002e-05 2024-09-21T12:33:33.028 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.029 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.029 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:33.029 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:40.122582+0000", 2024-09-21T12:33:33.029 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5412000000000001e-05 2024-09-21T12:33:33.029 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.029 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.029 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:33.029 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:40.122623+0000", 2024-09-21T12:33:33.029 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1287e-05 2024-09-21T12:33:33.029 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.029 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.029 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:33.029 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:40.122701+0000", 2024-09-21T12:33:33.029 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7894999999999999e-05 2024-09-21T12:33:33.029 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.029 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:33.030 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.030 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.030 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.030 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:221951 9.0 9:041c0094:::benchmark_data_smithi114_133214_object97917:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e308)", 2024-09-21T12:33:33.030 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:39.054240+0000", 2024-09-21T12:33:33.030 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 113.963979665, 2024-09-21T12:33:33.030 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.081460021, 2024-09-21T12:33:33.030 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:33.030 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:33.030 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:33.030 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:33.030 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:33:33.030 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:33.030 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 221951 2024-09-21T12:33:33.030 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.030 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:33.031 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.031 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:33.031 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.054240+0000", 2024-09-21T12:33:33.031 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.031 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.031 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.031 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:33.031 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.054240+0000", 2024-09-21T12:33:33.031 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.031 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.031 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.031 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:33.031 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.054241+0000", 2024-09-21T12:33:33.031 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6199999999999998e-07 2024-09-21T12:33:33.031 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.032 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.032 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:33.032 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.054244+0000", 2024-09-21T12:33:33.032 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.1379999999999999e-06 2024-09-21T12:33:33.032 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.032 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.032 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:33.032 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.054252+0000", 2024-09-21T12:33:33.032 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7849999999999992e-06 2024-09-21T12:33:33.032 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.032 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.032 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:33.032 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.054258+0000", 2024-09-21T12:33:33.033 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8880000000000002e-06 2024-09-21T12:33:33.033 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.033 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.033 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.033 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.054309+0000", 2024-09-21T12:33:33.033 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1786999999999997e-05 2024-09-21T12:33:33.033 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.033 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.033 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:33.033 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.054316+0000", 2024-09-21T12:33:33.033 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0740000000000004e-06 2024-09-21T12:33:33.033 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.033 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.033 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.034 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:40.122938+0000", 2024-09-21T12:33:33.034 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.068621273 2024-09-21T12:33:33.034 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.034 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.034 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:33.034 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:40.123054+0000", 2024-09-21T12:33:33.034 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011609599999999999 2024-09-21T12:33:33.034 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.034 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.034 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,2,6", 2024-09-21T12:33:33.034 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:40.123298+0000", 2024-09-21T12:33:33.034 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00024458900000000003 2024-09-21T12:33:33.035 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.035 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.035 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:33.035 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:40.128733+0000", 2024-09-21T12:33:33.035 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0054345950000000004 2024-09-21T12:33:33.035 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.035 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.035 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.035 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:40.135341+0000", 2024-09-21T12:33:33.035 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0066077449999999999 2024-09-21T12:33:33.035 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.035 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.035 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.035 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:40.135405+0000", 2024-09-21T12:33:33.035 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4689000000000004e-05 2024-09-21T12:33:33.035 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.036 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.036 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.036 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:40.135524+0000", 2024-09-21T12:33:33.036 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000119078 2024-09-21T12:33:33.036 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.036 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.036 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:33.036 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:40.135584+0000", 2024-09-21T12:33:33.036 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9985999999999998e-05 2024-09-21T12:33:33.036 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.036 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.036 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:33.036 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:40.135700+0000", 2024-09-21T12:33:33.036 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000115736 2024-09-21T12:33:33.036 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.036 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:33.037 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.037 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.037 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.037 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:222000 9.0 9:04eec24c:::benchmark_data_smithi114_133214_object97966:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e308)", 2024-09-21T12:33:33.037 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:39.054266+0000", 2024-09-21T12:33:33.037 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 113.963953554, 2024-09-21T12:33:33.037 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0817169070000001, 2024-09-21T12:33:33.037 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:33.037 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:33.037 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:33.037 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:33.037 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:33:33.037 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:33.037 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 222000 2024-09-21T12:33:33.037 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.037 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:33.038 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.038 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:33.038 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.054266+0000", 2024-09-21T12:33:33.038 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.038 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.038 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.038 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:33.038 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.054266+0000", 2024-09-21T12:33:33.038 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.038 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.038 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.038 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:33.039 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.054267+0000", 2024-09-21T12:33:33.039 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.3999999999999997e-07 2024-09-21T12:33:33.039 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.039 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.039 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:33.039 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.054269+0000", 2024-09-21T12:33:33.039 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0710000000000002e-06 2024-09-21T12:33:33.039 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.039 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.039 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:33.039 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.054275+0000", 2024-09-21T12:33:33.040 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0580000000000003e-06 2024-09-21T12:33:33.040 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.040 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.040 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:33.040 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.054279+0000", 2024-09-21T12:33:33.040 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.335e-06 2024-09-21T12:33:33.040 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.040 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.040 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.040 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.054431+0000", 2024-09-21T12:33:33.040 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015221599999999999 2024-09-21T12:33:33.040 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.040 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.040 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:33.040 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.054440+0000", 2024-09-21T12:33:33.040 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6899999999999998e-06 2024-09-21T12:33:33.040 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.041 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.041 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.041 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:40.125052+0000", 2024-09-21T12:33:33.041 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.070612108 2024-09-21T12:33:33.041 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.041 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.041 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:33.041 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:40.125159+0000", 2024-09-21T12:33:33.041 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010717699999999999 2024-09-21T12:33:33.041 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.041 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.041 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,2,6", 2024-09-21T12:33:33.041 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:40.125407+0000", 2024-09-21T12:33:33.042 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00024794299999999998 2024-09-21T12:33:33.042 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.042 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.042 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:33.042 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:40.135234+0000", 2024-09-21T12:33:33.042 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0098272399999999992 2024-09-21T12:33:33.042 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.042 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.042 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.042 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:40.135475+0000", 2024-09-21T12:33:33.042 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00024009199999999999 2024-09-21T12:33:33.042 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.042 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.043 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.043 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:40.135740+0000", 2024-09-21T12:33:33.043 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00026581199999999998 2024-09-21T12:33:33.043 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.043 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.043 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.043 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:40.135822+0000", 2024-09-21T12:33:33.043 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.2008000000000005e-05 2024-09-21T12:33:33.043 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.043 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.043 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:33.043 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:40.135886+0000", 2024-09-21T12:33:33.043 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3433000000000001e-05 2024-09-21T12:33:33.043 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.044 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.044 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:33.044 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:40.135983+0000", 2024-09-21T12:33:33.044 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7384000000000001e-05 2024-09-21T12:33:33.044 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.044 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:33.044 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.044 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.044 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.044 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231698 9.8 9:133f7a8f:::benchmark_data_smithi114_133214_object107664:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:33:33.044 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029696+0000", 2024-09-21T12:33:33.044 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 102.988524234, 2024-09-21T12:33:33.044 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.180329969, 2024-09-21T12:33:33.044 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:33.045 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:33.045 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:33.045 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:33.045 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:33:33.045 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:33.045 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231698 2024-09-21T12:33:33.045 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.045 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:33.045 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.045 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:33.045 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029696+0000", 2024-09-21T12:33:33.045 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.045 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.046 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.046 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:33.046 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029696+0000", 2024-09-21T12:33:33.046 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.046 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.046 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.046 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:33.046 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029697+0000", 2024-09-21T12:33:33.046 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5209999999999999e-06 2024-09-21T12:33:33.046 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.046 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.046 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:33.046 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029702+0000", 2024-09-21T12:33:33.047 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7210000000000002e-06 2024-09-21T12:33:33.047 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.047 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.047 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:33.047 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029714+0000", 2024-09-21T12:33:33.047 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1805e-05 2024-09-21T12:33:33.047 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.047 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.047 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:33.047 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029726+0000", 2024-09-21T12:33:33.047 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.269e-05 2024-09-21T12:33:33.047 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.047 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.047 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.047 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.031749+0000", 2024-09-21T12:33:33.047 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0020226240000000002 2024-09-21T12:33:33.048 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.048 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.048 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:33.048 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.031759+0000", 2024-09-21T12:33:33.048 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0322999999999999e-05 2024-09-21T12:33:33.048 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.048 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.048 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.048 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.202924+0000", 2024-09-21T12:33:33.048 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1711650309999999 2024-09-21T12:33:33.048 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.048 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.048 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:33.048 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.203059+0000", 2024-09-21T12:33:33.048 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013477199999999999 2024-09-21T12:33:33.049 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.049 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.049 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,3,7", 2024-09-21T12:33:33.049 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.203427+0000", 2024-09-21T12:33:33.049 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00036761500000000001 2024-09-21T12:33:33.049 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.049 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.050 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.050 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.209690+0000", 2024-09-21T12:33:33.050 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0062635599999999996 2024-09-21T12:33:33.050 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.050 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.050 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:33.050 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.209705+0000", 2024-09-21T12:33:33.050 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5172e-05 2024-09-21T12:33:33.050 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.050 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.050 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.050 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.209750+0000", 2024-09-21T12:33:33.050 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4484999999999998e-05 2024-09-21T12:33:33.050 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.051 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.051 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.051 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.209799+0000", 2024-09-21T12:33:33.051 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9500999999999999e-05 2024-09-21T12:33:33.051 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.051 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.051 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:33.051 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.209869+0000", 2024-09-21T12:33:33.051 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9181000000000001e-05 2024-09-21T12:33:33.051 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.051 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.051 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:33.051 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.210026+0000", 2024-09-21T12:33:33.052 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015696800000000001 2024-09-21T12:33:33.052 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.052 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:33.052 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.052 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.052 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.052 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231716 9.8 9:10787a57:::benchmark_data_smithi114_133214_object107682:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:33:33.052 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029740+0000", 2024-09-21T12:33:33.052 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 102.988479508, 2024-09-21T12:33:33.052 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.193239296, 2024-09-21T12:33:33.052 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:33.052 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:33.052 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:33.052 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:33.052 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:33:33.053 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:33.053 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231716 2024-09-21T12:33:33.053 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.053 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:33.053 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.053 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:33.053 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029740+0000", 2024-09-21T12:33:33.053 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.053 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.053 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.053 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:33.053 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029740+0000", 2024-09-21T12:33:33.053 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.054 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.054 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.054 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:33.054 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029741+0000", 2024-09-21T12:33:33.054 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0999999999999999e-07 2024-09-21T12:33:33.054 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.054 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.054 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:33.054 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029743+0000", 2024-09-21T12:33:33.054 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.232e-06 2024-09-21T12:33:33.054 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.054 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.054 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:33.055 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029750+0000", 2024-09-21T12:33:33.055 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5409999999999997e-06 2024-09-21T12:33:33.055 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.055 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.055 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:33.055 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029755+0000", 2024-09-21T12:33:33.055 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5010000000000004e-06 2024-09-21T12:33:33.055 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.055 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.055 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.055 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.032659+0000", 2024-09-21T12:33:33.055 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0029039059999999999 2024-09-21T12:33:33.055 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.055 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.055 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:33.055 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.032669+0000", 2024-09-21T12:33:33.056 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7759999999999992e-06 2024-09-21T12:33:33.056 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.056 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.056 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.056 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.207602+0000", 2024-09-21T12:33:33.056 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.174932989 2024-09-21T12:33:33.056 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.056 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.056 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:33.056 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.207718+0000", 2024-09-21T12:33:33.056 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011654000000000001 2024-09-21T12:33:33.056 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.056 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.056 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,3,7", 2024-09-21T12:33:33.056 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.208066+0000", 2024-09-21T12:33:33.056 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00034776000000000002 2024-09-21T12:33:33.057 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.057 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.057 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.057 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.212283+0000", 2024-09-21T12:33:33.057 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0042167749999999999 2024-09-21T12:33:33.057 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.057 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.057 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.057 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.212330+0000", 2024-09-21T12:33:33.057 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7537999999999998e-05 2024-09-21T12:33:33.057 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.057 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.057 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:33.057 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.212374+0000", 2024-09-21T12:33:33.058 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3980999999999998e-05 2024-09-21T12:33:33.058 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.058 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.058 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.058 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.222783+0000", 2024-09-21T12:33:33.058 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010408327 2024-09-21T12:33:33.058 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.058 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.058 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:33.058 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.222854+0000", 2024-09-21T12:33:33.058 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1431000000000001e-05 2024-09-21T12:33:33.058 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.058 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.058 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:33.058 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.222980+0000", 2024-09-21T12:33:33.059 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012558900000000001 2024-09-21T12:33:33.059 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.059 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:33.059 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.059 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.059 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.059 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.15924.0:1998 10.d 10:b027651e:::benchmark_data_smithi114_161690_object1997:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e350)", 2024-09-21T12:33:33.059 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:32:38.378501+0000", 2024-09-21T12:33:33.059 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 54.639719016999997, 2024-09-21T12:33:33.059 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0155991470000001, 2024-09-21T12:33:33.059 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:33.059 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:33.059 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:33.059 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:33.060 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.15924", 2024-09-21T12:33:33.060 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:33:33.060 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 1998 2024-09-21T12:33:33.060 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.060 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:33.060 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.060 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:33.060 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:38.378501+0000", 2024-09-21T12:33:33.060 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.060 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.060 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.060 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:33.060 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:38.378501+0000", 2024-09-21T12:33:33.060 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.060 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.061 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.061 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:33.061 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:38.378503+0000", 2024-09-21T12:33:33.061 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9609999999999999e-06 2024-09-21T12:33:33.061 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.061 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.061 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:33.061 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:38.378552+0000", 2024-09-21T12:33:33.061 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8721999999999999e-05 2024-09-21T12:33:33.061 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.061 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.061 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:33.061 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:38.378563+0000", 2024-09-21T12:33:33.062 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1639e-05 2024-09-21T12:33:33.062 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.062 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.062 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:33.062 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:38.378572+0000", 2024-09-21T12:33:33.062 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6170000000000007e-06 2024-09-21T12:33:33.062 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.062 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.062 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.062 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:38.378711+0000", 2024-09-21T12:33:33.062 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013945599999999999 2024-09-21T12:33:33.062 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.062 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.062 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:33.063 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:38.378718+0000", 2024-09-21T12:33:33.063 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3470000000000004e-06 2024-09-21T12:33:33.063 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.063 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.063 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.063 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:39.390854+0000", 2024-09-21T12:33:33.063 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0121363800000001 2024-09-21T12:33:33.063 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.063 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.063 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:33.063 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:39.390986+0000", 2024-09-21T12:33:33.063 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013182499999999999 2024-09-21T12:33:33.063 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.063 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.063 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,6", 2024-09-21T12:33:33.064 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:39.391135+0000", 2024-09-21T12:33:33.064 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014936300000000001 2024-09-21T12:33:33.064 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.064 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.064 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.064 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:39.393173+0000", 2024-09-21T12:33:33.064 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0020383319999999999 2024-09-21T12:33:33.064 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.064 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.064 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:33.064 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:39.393238+0000", 2024-09-21T12:33:33.064 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4373000000000002e-05 2024-09-21T12:33:33.064 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.064 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.064 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.065 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:39.393808+0000", 2024-09-21T12:33:33.065 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00057054999999999996 2024-09-21T12:33:33.065 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.065 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.065 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:33.065 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:39.393853+0000", 2024-09-21T12:33:33.065 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4325e-05 2024-09-21T12:33:33.065 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.065 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.065 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:33.065 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:39.394100+0000", 2024-09-21T12:33:33.065 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00024725700000000002 2024-09-21T12:33:33.065 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.065 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:33.066 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.066 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.066 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.066 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.15924.0:2007 10.1 10:8ee93638:::benchmark_data_smithi114_161690_object2006:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e350)", 2024-09-21T12:33:33.066 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:32:38.378588+0000", 2024-09-21T12:33:33.066 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 54.639632239999997, 2024-09-21T12:33:33.066 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0139329580000001, 2024-09-21T12:33:33.066 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:33.066 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:33.066 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:33.066 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:33.066 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.15924", 2024-09-21T12:33:33.066 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:33:33.066 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 2007 2024-09-21T12:33:33.066 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.067 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:33.067 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.067 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:33.067 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:38.378588+0000", 2024-09-21T12:33:33.067 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.067 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.067 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.067 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:33.067 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:38.378588+0000", 2024-09-21T12:33:33.067 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.067 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.067 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.067 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:33.067 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:38.378588+0000", 2024-09-21T12:33:33.067 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.44e-07 2024-09-21T12:33:33.067 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.068 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.068 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:33.068 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:38.378632+0000", 2024-09-21T12:33:33.068 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3742000000000002e-05 2024-09-21T12:33:33.068 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.068 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.068 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:33.068 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:38.378641+0000", 2024-09-21T12:33:33.068 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6559999999999998e-06 2024-09-21T12:33:33.068 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.068 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.068 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:33.068 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:38.378647+0000", 2024-09-21T12:33:33.068 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2090000000000002e-06 2024-09-21T12:33:33.068 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.068 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.069 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.069 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:38.378687+0000", 2024-09-21T12:33:33.069 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.9818999999999999e-05 2024-09-21T12:33:33.069 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.069 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.069 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:33.069 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:38.378695+0000", 2024-09-21T12:33:33.069 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.242e-06 2024-09-21T12:33:33.069 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.069 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.069 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.069 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:39.389911+0000", 2024-09-21T12:33:33.069 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0112161 2024-09-21T12:33:33.069 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.070 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.070 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:33.070 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:39.390238+0000", 2024-09-21T12:33:33.070 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00032676800000000001 2024-09-21T12:33:33.070 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.070 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.070 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,7", 2024-09-21T12:33:33.070 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:39.390399+0000", 2024-09-21T12:33:33.070 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016144900000000001 2024-09-21T12:33:33.070 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.070 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.070 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:33.070 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:39.391212+0000", 2024-09-21T12:33:33.070 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00081296799999999996 2024-09-21T12:33:33.070 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.071 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.071 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.071 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:39.391633+0000", 2024-09-21T12:33:33.071 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00042059200000000001 2024-09-21T12:33:33.071 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.071 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.071 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.071 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:39.391848+0000", 2024-09-21T12:33:33.071 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00021502000000000001 2024-09-21T12:33:33.071 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.071 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.071 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:33.071 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:39.391908+0000", 2024-09-21T12:33:33.072 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0343999999999997e-05 2024-09-21T12:33:33.072 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.072 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.072 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:33.072 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:39.392521+0000", 2024-09-21T12:33:33.072 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00061250499999999995 2024-09-21T12:33:33.072 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.072 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:33.072 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.072 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.072 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.072 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.15924.0:4808 10.4 10:208035e6:::benchmark_data_smithi114_161690_object4807:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-09-21T12:33:33.073 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:32:41.401314+0000", 2024-09-21T12:33:33.073 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 51.616905770000002, 2024-09-21T12:33:33.073 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.63962072599999997, 2024-09-21T12:33:33.073 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:33.073 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:33.073 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:33.073 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:33.073 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.15924", 2024-09-21T12:33:33.073 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:33:33.073 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 4808 2024-09-21T12:33:33.073 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.074 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:33.074 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.074 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:33.074 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.401314+0000", 2024-09-21T12:33:33.074 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.074 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.074 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.074 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:33.074 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.401314+0000", 2024-09-21T12:33:33.074 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.074 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.074 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.075 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:33.075 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.401315+0000", 2024-09-21T12:33:33.075 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3820000000000001e-06 2024-09-21T12:33:33.075 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.075 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.075 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:33.075 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.401344+0000", 2024-09-21T12:33:33.075 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.8671e-05 2024-09-21T12:33:33.075 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.075 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.075 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:33.075 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.401358+0000", 2024-09-21T12:33:33.075 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4327e-05 2024-09-21T12:33:33.075 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.075 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.076 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:33.076 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.401366+0000", 2024-09-21T12:33:33.076 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1949999999999997e-06 2024-09-21T12:33:33.076 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.076 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.076 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.076 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:33.076 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:33.077 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.401547+0000", 2024-09-21T12:33:33.077 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00018157000000000001 2024-09-21T12:33:33.077 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.077 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.077 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:33.077 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.401558+0000", 2024-09-21T12:33:33.077 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1117e-05 2024-09-21T12:33:33.077 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.078 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.078 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.078 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.036816+0000", 2024-09-21T12:33:33.078 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.63525803800000002 2024-09-21T12:33:33.078 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.078 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.078 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:33.078 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.036935+0000", 2024-09-21T12:33:33.078 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000118808 2024-09-21T12:33:33.078 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.078 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.078 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,4", 2024-09-21T12:33:33.078 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.037099+0000", 2024-09-21T12:33:33.079 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016356199999999999 2024-09-21T12:33:33.079 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.079 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.079 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.079 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.038442+0000", 2024-09-21T12:33:33.079 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001342813 2024-09-21T12:33:33.079 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.079 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.079 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.079 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.040841+0000", 2024-09-21T12:33:33.079 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00239908 2024-09-21T12:33:33.079 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.079 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.080 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:33.080 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.040852+0000", 2024-09-21T12:33:33.080 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1759e-05 2024-09-21T12:33:33.080 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.080 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.080 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:33.080 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.040898+0000", 2024-09-21T12:33:33.080 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5676000000000001e-05 2024-09-21T12:33:33.080 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.080 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.080 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:33.080 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.040935+0000", 2024-09-21T12:33:33.080 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.6727999999999998e-05 2024-09-21T12:33:33.080 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.080 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:33.081 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.081 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.081 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.081 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.15924.0:4814 10.6 10:699d52d2:::benchmark_data_smithi114_161690_object4813:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-09-21T12:33:33.081 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:32:41.403173+0000", 2024-09-21T12:33:33.081 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 51.615047042, 2024-09-21T12:33:33.081 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.63976581200000004, 2024-09-21T12:33:33.081 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:33.081 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:33.081 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:33.081 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:33.081 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.15924", 2024-09-21T12:33:33.081 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:33:33.081 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 4814 2024-09-21T12:33:33.081 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.082 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:33.082 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.082 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:33.082 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.403173+0000", 2024-09-21T12:33:33.082 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.082 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.082 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.082 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:33.082 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.403173+0000", 2024-09-21T12:33:33.082 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.082 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.082 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.082 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:33.082 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.403174+0000", 2024-09-21T12:33:33.083 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.449e-06 2024-09-21T12:33:33.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.083 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.083 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:33.083 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.403193+0000", 2024-09-21T12:33:33.083 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8927e-05 2024-09-21T12:33:33.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.083 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.083 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:33.083 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.403210+0000", 2024-09-21T12:33:33.083 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6651e-05 2024-09-21T12:33:33.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.083 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.083 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:33.083 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.403217+0000", 2024-09-21T12:33:33.083 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1219999999999997e-06 2024-09-21T12:33:33.084 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.084 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.084 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.084 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.403275+0000", 2024-09-21T12:33:33.084 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7562000000000002e-05 2024-09-21T12:33:33.084 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.084 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.084 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:33.084 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.403284+0000", 2024-09-21T12:33:33.084 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1849999999999993e-06 2024-09-21T12:33:33.084 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.084 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.084 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.084 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.037282+0000", 2024-09-21T12:33:33.084 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.633998336 2024-09-21T12:33:33.084 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.085 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.085 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:33.085 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.037403+0000", 2024-09-21T12:33:33.085 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012046500000000001 2024-09-21T12:33:33.085 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.085 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.085 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,5", 2024-09-21T12:33:33.085 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.037556+0000", 2024-09-21T12:33:33.085 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015339800000000001 2024-09-21T12:33:33.085 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.085 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.085 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.085 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.039909+0000", 2024-09-21T12:33:33.086 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0023528550000000001 2024-09-21T12:33:33.086 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.086 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.086 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.086 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.039957+0000", 2024-09-21T12:33:33.086 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7797999999999999e-05 2024-09-21T12:33:33.086 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.086 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.086 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:33.086 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.042854+0000", 2024-09-21T12:33:33.086 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0028979019999999999 2024-09-21T12:33:33.086 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.086 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.087 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:33.087 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.042901+0000", 2024-09-21T12:33:33.087 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6864999999999999e-05 2024-09-21T12:33:33.087 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.087 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.087 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:33.087 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.042939+0000", 2024-09-21T12:33:33.087 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.7296999999999999e-05 2024-09-21T12:33:33.087 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.087 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:33.087 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.087 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.087 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.088 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.15924.0:4817 10.4 10:245e72db:::benchmark_data_smithi114_161690_object4816:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-09-21T12:33:33.088 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:32:41.404886+0000", 2024-09-21T12:33:33.088 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 51.613333816999997, 2024-09-21T12:33:33.088 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.63638109499999995, 2024-09-21T12:33:33.088 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:33.088 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:33.088 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:33.088 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:33.088 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.15924", 2024-09-21T12:33:33.088 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:33:33.088 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 4817 2024-09-21T12:33:33.088 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.088 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:33.088 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.088 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:33.088 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.404886+0000", 2024-09-21T12:33:33.089 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.089 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.089 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.089 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:33.089 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.404886+0000", 2024-09-21T12:33:33.089 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.089 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.089 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.089 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:33.089 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.404888+0000", 2024-09-21T12:33:33.089 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.517e-06 2024-09-21T12:33:33.089 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.089 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.089 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:33.089 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.404910+0000", 2024-09-21T12:33:33.090 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2107999999999999e-05 2024-09-21T12:33:33.090 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.090 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.090 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:33.090 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.404923+0000", 2024-09-21T12:33:33.090 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3685e-05 2024-09-21T12:33:33.090 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.090 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.090 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:33.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:33.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:33.091 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.404931+0000", 2024-09-21T12:33:33.092 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6450000000000005e-06 2024-09-21T12:33:33.092 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.092 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.092 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.092 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.405006+0000", 2024-09-21T12:33:33.092 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4742999999999999e-05 2024-09-21T12:33:33.092 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.092 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.092 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:33.092 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.405015+0000", 2024-09-21T12:33:33.092 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0650000000000005e-06 2024-09-21T12:33:33.092 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.092 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.092 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.093 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.038486+0000", 2024-09-21T12:33:33.093 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.63347095600000003 2024-09-21T12:33:33.093 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.093 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.093 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:33.093 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.038598+0000", 2024-09-21T12:33:33.093 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011255 2024-09-21T12:33:33.093 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.093 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.093 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,4", 2024-09-21T12:33:33.093 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.038712+0000", 2024-09-21T12:33:33.093 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000114021 2024-09-21T12:33:33.093 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.094 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.094 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:33.094 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.041011+0000", 2024-09-21T12:33:33.094 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0022987160000000001 2024-09-21T12:33:33.094 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.094 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.094 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.094 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.041062+0000", 2024-09-21T12:33:33.094 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1357999999999999e-05 2024-09-21T12:33:33.094 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.094 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.094 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.094 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.041142+0000", 2024-09-21T12:33:33.094 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9165000000000005e-05 2024-09-21T12:33:33.095 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.095 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.095 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:33.095 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.041226+0000", 2024-09-21T12:33:33.095 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.4361000000000001e-05 2024-09-21T12:33:33.095 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.095 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.095 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:33.095 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.041267+0000", 2024-09-21T12:33:33.095 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1205000000000003e-05 2024-09-21T12:33:33.095 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.095 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:33.095 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.095 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.095 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.096 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.15924.0:4834 10.4 10:2f333743:::benchmark_data_smithi114_161690_object4833:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-09-21T12:33:33.096 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:32:41.411548+0000", 2024-09-21T12:33:33.096 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 51.606672115999999, 2024-09-21T12:33:33.096 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.631309817, 2024-09-21T12:33:33.096 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:33.096 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:33.096 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:33.096 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:33.096 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.15924", 2024-09-21T12:33:33.096 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:33:33.096 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 4834 2024-09-21T12:33:33.096 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.096 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:33.096 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.097 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:33.097 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.411548+0000", 2024-09-21T12:33:33.097 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.097 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.097 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.097 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:33.097 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.411548+0000", 2024-09-21T12:33:33.097 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.097 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.097 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.097 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:33.097 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.411550+0000", 2024-09-21T12:33:33.097 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.79e-06 2024-09-21T12:33:33.097 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.098 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.098 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:33.098 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.411572+0000", 2024-09-21T12:33:33.098 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2816999999999999e-05 2024-09-21T12:33:33.098 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.098 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.098 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:33.098 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.411585+0000", 2024-09-21T12:33:33.098 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3011e-05 2024-09-21T12:33:33.098 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.098 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.098 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:33.098 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.411593+0000", 2024-09-21T12:33:33.099 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6899999999999992e-06 2024-09-21T12:33:33.099 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.099 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.099 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.099 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.411659+0000", 2024-09-21T12:33:33.099 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5859999999999996e-05 2024-09-21T12:33:33.099 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.099 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.099 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:33.099 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.411668+0000", 2024-09-21T12:33:33.099 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.2699999999999993e-06 2024-09-21T12:33:33.099 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.099 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.099 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.100 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.039521+0000", 2024-09-21T12:33:33.100 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.62785248400000004 2024-09-21T12:33:33.100 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.100 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.100 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:33.100 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.039746+0000", 2024-09-21T12:33:33.100 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00022511300000000001 2024-09-21T12:33:33.100 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.100 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.100 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,4", 2024-09-21T12:33:33.100 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.039978+0000", 2024-09-21T12:33:33.100 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00023195999999999999 2024-09-21T12:33:33.100 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.101 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.101 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.101 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.041102+0000", 2024-09-21T12:33:33.101 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0011243340000000001 2024-09-21T12:33:33.101 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.101 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.101 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:33.101 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.041289+0000", 2024-09-21T12:33:33.101 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000187176 2024-09-21T12:33:33.101 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.101 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.101 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.102 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.042770+0000", 2024-09-21T12:33:33.102 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001480564 2024-09-21T12:33:33.102 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.102 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.102 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:33.102 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.042820+0000", 2024-09-21T12:33:33.102 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0154999999999997e-05 2024-09-21T12:33:33.102 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.102 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.102 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:33.102 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.042858+0000", 2024-09-21T12:33:33.102 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.7592999999999997e-05 2024-09-21T12:33:33.103 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.103 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:33.103 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.103 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.103 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.103 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.15924.0:4851 10.4 10:2cfef6ee:::benchmark_data_smithi114_161690_object4850:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-09-21T12:33:33.103 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:32:41.417901+0000", 2024-09-21T12:33:33.103 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 51.600318792000003, 2024-09-21T12:33:33.103 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.62957146399999997, 2024-09-21T12:33:33.103 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:33.103 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:33.103 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:33.103 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:33.104 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.15924", 2024-09-21T12:33:33.104 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:33:33.104 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 4851 2024-09-21T12:33:33.104 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.104 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:33.104 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.104 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:33.104 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.417901+0000", 2024-09-21T12:33:33.104 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.104 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.104 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.104 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:33.104 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.417901+0000", 2024-09-21T12:33:33.104 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.105 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.105 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.105 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:33.105 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.417903+0000", 2024-09-21T12:33:33.105 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5939999999999999e-06 2024-09-21T12:33:33.105 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.105 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.105 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:33.105 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.417926+0000", 2024-09-21T12:33:33.105 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2867e-05 2024-09-21T12:33:33.105 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.105 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.105 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:33.106 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.417938+0000", 2024-09-21T12:33:33.106 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2931e-05 2024-09-21T12:33:33.106 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.106 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.106 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:33.106 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.417946+0000", 2024-09-21T12:33:33.106 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8679999999999999e-06 2024-09-21T12:33:33.106 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.106 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.106 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.106 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.418163+0000", 2024-09-21T12:33:33.106 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000216999 2024-09-21T12:33:33.106 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.107 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.107 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:33.107 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.418173+0000", 2024-09-21T12:33:33.107 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.5990000000000008e-06 2024-09-21T12:33:33.107 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.107 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.107 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.107 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.041331+0000", 2024-09-21T12:33:33.107 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.62315801299999996 2024-09-21T12:33:33.107 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.107 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.107 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:33.108 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.041418+0000", 2024-09-21T12:33:33.108 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.7459000000000004e-05 2024-09-21T12:33:33.108 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.108 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.108 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,4", 2024-09-21T12:33:33.108 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.041545+0000", 2024-09-21T12:33:33.108 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012694099999999999 2024-09-21T12:33:33.108 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.108 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.108 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.108 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.044213+0000", 2024-09-21T12:33:33.108 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0026674469999999999 2024-09-21T12:33:33.108 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.109 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.109 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:33.109 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.044226+0000", 2024-09-21T12:33:33.109 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3169999999999999e-05 2024-09-21T12:33:33.109 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.109 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.109 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.109 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.047350+0000", 2024-09-21T12:33:33.109 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0031242819999999999 2024-09-21T12:33:33.109 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.109 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.109 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:33.110 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.047429+0000", 2024-09-21T12:33:33.110 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8712000000000001e-05 2024-09-21T12:33:33.110 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.110 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.110 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:33.110 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.047473+0000", 2024-09-21T12:33:33.110 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3581999999999997e-05 2024-09-21T12:33:33.110 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.110 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:33.110 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.110 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.110 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.110 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.15924.0:4855 10.6 10:60cf57bf:::benchmark_data_smithi114_161690_object4854:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-09-21T12:33:33.111 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:32:41.419486+0000", 2024-09-21T12:33:33.111 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 51.598733379999999, 2024-09-21T12:33:33.111 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.62360211799999998, 2024-09-21T12:33:33.111 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:33.111 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:33.111 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:33.111 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:33.111 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.15924", 2024-09-21T12:33:33.111 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:33:33.111 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 4855 2024-09-21T12:33:33.111 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.111 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:33.111 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.112 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:33.112 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.419486+0000", 2024-09-21T12:33:33.112 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.112 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.112 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.112 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:33.112 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.419486+0000", 2024-09-21T12:33:33.112 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.112 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.112 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.112 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:33.112 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.419488+0000", 2024-09-21T12:33:33.112 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0899999999999999e-06 2024-09-21T12:33:33.112 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.113 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.113 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:33.113 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.419506+0000", 2024-09-21T12:33:33.113 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8929000000000001e-05 2024-09-21T12:33:33.113 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.113 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.113 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:33.113 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.419514+0000", 2024-09-21T12:33:33.113 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9869999999999999e-06 2024-09-21T12:33:33.113 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.113 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.113 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:33.113 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.419520+0000", 2024-09-21T12:33:33.114 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7690000000000002e-06 2024-09-21T12:33:33.114 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.114 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.114 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.114 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.419574+0000", 2024-09-21T12:33:33.114 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4205e-05 2024-09-21T12:33:33.114 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.114 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.114 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:33.114 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.419583+0000", 2024-09-21T12:33:33.114 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.2989999999999997e-06 2024-09-21T12:33:33.114 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.115 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.115 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.115 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.038575+0000", 2024-09-21T12:33:33.115 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.61899244399999998 2024-09-21T12:33:33.115 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.115 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.115 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:33.115 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.038734+0000", 2024-09-21T12:33:33.115 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015886900000000001 2024-09-21T12:33:33.116 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.116 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.116 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,5", 2024-09-21T12:33:33.116 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.038862+0000", 2024-09-21T12:33:33.116 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012771700000000001 2024-09-21T12:33:33.116 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.116 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.116 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.116 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.041489+0000", 2024-09-21T12:33:33.116 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0026269190000000001 2024-09-21T12:33:33.116 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.116 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.116 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.116 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.041531+0000", 2024-09-21T12:33:33.117 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.2682000000000002e-05 2024-09-21T12:33:33.117 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.117 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.117 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:33.117 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.042951+0000", 2024-09-21T12:33:33.117 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001419872 2024-09-21T12:33:33.117 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.117 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.117 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:33.117 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.043051+0000", 2024-09-21T12:33:33.117 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.9464999999999997e-05 2024-09-21T12:33:33.117 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.117 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.117 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:33.117 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.043089+0000", 2024-09-21T12:33:33.118 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.7870999999999997e-05 2024-09-21T12:33:33.118 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.118 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:33.118 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.118 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.118 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.118 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.15924.0:4859 10.6 10:6c30dc2e:::benchmark_data_smithi114_161690_object4858:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-09-21T12:33:33.118 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:32:41.420279+0000", 2024-09-21T12:33:33.118 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 51.597940469000001, 2024-09-21T12:33:33.118 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.62293885199999999, 2024-09-21T12:33:33.118 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:33.118 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:33.118 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:33.119 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:33.119 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.15924", 2024-09-21T12:33:33.119 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:33:33.119 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 4859 2024-09-21T12:33:33.119 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.119 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:33.119 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.119 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:33.119 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.420279+0000", 2024-09-21T12:33:33.119 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.119 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.119 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.119 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:33.119 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.420279+0000", 2024-09-21T12:33:33.120 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.120 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.120 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.120 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:33.120 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.420280+0000", 2024-09-21T12:33:33.120 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1000000000000001e-06 2024-09-21T12:33:33.120 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.120 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.120 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:33.120 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.420305+0000", 2024-09-21T12:33:33.120 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.4355e-05 2024-09-21T12:33:33.120 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.120 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.121 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:33.121 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.420317+0000", 2024-09-21T12:33:33.121 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1857e-05 2024-09-21T12:33:33.121 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.121 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.121 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:33.121 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.420324+0000", 2024-09-21T12:33:33.121 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.559e-06 2024-09-21T12:33:33.121 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.121 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.121 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.121 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.420374+0000", 2024-09-21T12:33:33.121 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0188999999999997e-05 2024-09-21T12:33:33.121 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.121 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.122 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:33.122 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.420382+0000", 2024-09-21T12:33:33.122 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9689999999999994e-06 2024-09-21T12:33:33.122 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.122 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.122 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.122 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.039995+0000", 2024-09-21T12:33:33.122 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.61961256799999997 2024-09-21T12:33:33.122 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.122 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.122 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:33.122 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.040108+0000", 2024-09-21T12:33:33.122 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000113443 2024-09-21T12:33:33.122 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.123 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.123 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,5", 2024-09-21T12:33:33.123 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.040227+0000", 2024-09-21T12:33:33.123 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011814 2024-09-21T12:33:33.123 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.123 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.123 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.123 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.042778+0000", 2024-09-21T12:33:33.123 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0025515350000000002 2024-09-21T12:33:33.123 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.123 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.123 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:33.123 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.043100+0000", 2024-09-21T12:33:33.123 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000321479 2024-09-21T12:33:33.123 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.124 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.124 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.124 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.043139+0000", 2024-09-21T12:33:33.124 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.9647000000000002e-05 2024-09-21T12:33:33.124 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.124 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.124 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:33.124 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.043184+0000", 2024-09-21T12:33:33.124 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4953999999999997e-05 2024-09-21T12:33:33.124 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.124 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.124 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:33.124 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.043218+0000", 2024-09-21T12:33:33.124 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4057000000000003e-05 2024-09-21T12:33:33.125 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.125 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:33.125 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.125 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.125 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.125 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.15924.0:4861 10.4 10:29f834a0:::benchmark_data_smithi114_161690_object4860:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-09-21T12:33:33.125 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:32:41.421066+0000", 2024-09-21T12:33:33.125 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 51.597154338000003, 2024-09-21T12:33:33.125 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.627260386, 2024-09-21T12:33:33.125 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:33.125 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:33.125 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:33.125 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:33.125 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.15924", 2024-09-21T12:33:33.125 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:33:33.125 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 4861 2024-09-21T12:33:33.126 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.126 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:33.126 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.126 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:33.126 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.421066+0000", 2024-09-21T12:33:33.126 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.126 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.126 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.126 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:33.126 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.421066+0000", 2024-09-21T12:33:33.126 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.126 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.126 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.126 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:33.127 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.421067+0000", 2024-09-21T12:33:33.127 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2559999999999999e-06 2024-09-21T12:33:33.127 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.127 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.127 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:33.127 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.421090+0000", 2024-09-21T12:33:33.127 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2979000000000001e-05 2024-09-21T12:33:33.127 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.127 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.127 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:33.127 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.421104+0000", 2024-09-21T12:33:33.127 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4601e-05 2024-09-21T12:33:33.127 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.127 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.127 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:33.127 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.421113+0000", 2024-09-21T12:33:33.128 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.2369999999999992e-06 2024-09-21T12:33:33.128 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.128 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.128 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.128 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.421172+0000", 2024-09-21T12:33:33.128 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9117999999999999e-05 2024-09-21T12:33:33.128 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.128 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.128 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:33.128 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.421183+0000", 2024-09-21T12:33:33.128 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1759999999999999e-05 2024-09-21T12:33:33.128 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.129 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.129 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.129 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.042890+0000", 2024-09-21T12:33:33.129 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.62170702600000005 2024-09-21T12:33:33.129 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.129 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.129 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:33.129 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.042985+0000", 2024-09-21T12:33:33.129 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.4778000000000006e-05 2024-09-21T12:33:33.129 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.129 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.129 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,4", 2024-09-21T12:33:33.129 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.043107+0000", 2024-09-21T12:33:33.129 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000122173 2024-09-21T12:33:33.130 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.130 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.130 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:33.130 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.044863+0000", 2024-09-21T12:33:33.130 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00175599 2024-09-21T12:33:33.130 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.130 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.130 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.130 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.045194+0000", 2024-09-21T12:33:33.130 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00033071799999999997 2024-09-21T12:33:33.130 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.130 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.131 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.131 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.048236+0000", 2024-09-21T12:33:33.131 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0030416990000000001 2024-09-21T12:33:33.131 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.131 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.131 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:33.131 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.048287+0000", 2024-09-21T12:33:33.131 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1620999999999999e-05 2024-09-21T12:33:33.131 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.131 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.131 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:33.131 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.048326+0000", 2024-09-21T12:33:33.131 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8430000000000003e-05 2024-09-21T12:33:33.131 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.132 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:33.132 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.132 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.132 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.132 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.15924.0:4868 10.6 10:670898e1:::benchmark_data_smithi114_161690_object4867:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-09-21T12:33:33.132 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:32:41.425562+0000", 2024-09-21T12:33:33.132 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 51.592657725999999, 2024-09-21T12:33:33.132 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.61811503000000001, 2024-09-21T12:33:33.132 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:33.132 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:33.132 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:33.132 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:33.132 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.15924", 2024-09-21T12:33:33.132 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:33:33.133 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 4868 2024-09-21T12:33:33.133 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.133 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:33.133 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.133 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:33.133 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.425562+0000", 2024-09-21T12:33:33.133 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.133 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.133 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.133 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:33.133 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.425562+0000", 2024-09-21T12:33:33.134 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.134 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.134 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.134 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:33.134 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.425564+0000", 2024-09-21T12:33:33.134 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2869999999999999e-06 2024-09-21T12:33:33.134 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.134 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.134 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:33.134 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.425592+0000", 2024-09-21T12:33:33.135 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.7889e-05 2024-09-21T12:33:33.135 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.135 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.135 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:33.135 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.425613+0000", 2024-09-21T12:33:33.135 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.054e-05 2024-09-21T12:33:33.135 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.135 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.135 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:33.135 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.425623+0000", 2024-09-21T12:33:33.135 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.9899999999999992e-06 2024-09-21T12:33:33.135 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.135 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.135 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.135 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.425692+0000", 2024-09-21T12:33:33.136 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9008000000000001e-05 2024-09-21T12:33:33.136 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.136 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.136 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:33.136 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.425704+0000", 2024-09-21T12:33:33.136 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1970000000000001e-05 2024-09-21T12:33:33.136 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.136 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.136 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.136 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.040778+0000", 2024-09-21T12:33:33.136 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.61507414100000002 2024-09-21T12:33:33.136 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.136 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.136 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:33.136 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.040886+0000", 2024-09-21T12:33:33.136 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000107978 2024-09-21T12:33:33.136 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.137 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.137 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,5", 2024-09-21T12:33:33.137 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.041015+0000", 2024-09-21T12:33:33.137 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012915999999999999 2024-09-21T12:33:33.137 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.137 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.137 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:33.137 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.043104+0000", 2024-09-21T12:33:33.137 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0020886020000000002 2024-09-21T12:33:33.137 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.137 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.137 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.137 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.043556+0000", 2024-09-21T12:33:33.137 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00045271800000000001 2024-09-21T12:33:33.137 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.137 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.138 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.138 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.043593+0000", 2024-09-21T12:33:33.138 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.6523000000000003e-05 2024-09-21T12:33:33.138 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.138 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.138 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:33.138 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.043643+0000", 2024-09-21T12:33:33.138 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9756000000000003e-05 2024-09-21T12:33:33.138 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.138 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.138 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:33.138 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.043677+0000", 2024-09-21T12:33:33.138 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4468000000000003e-05 2024-09-21T12:33:33.138 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.138 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:33.139 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.139 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.139 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.139 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.15924.0:4871 10.6 10:61df9cb6:::benchmark_data_smithi114_161690_object4870:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-09-21T12:33:33.139 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:32:41.428614+0000", 2024-09-21T12:33:33.139 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 51.589605564999999, 2024-09-21T12:33:33.139 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.61548491000000005, 2024-09-21T12:33:33.139 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:33.139 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:33.139 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:33.139 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:33.139 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.15924", 2024-09-21T12:33:33.139 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:33:33.139 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 4871 2024-09-21T12:33:33.139 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.140 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:33.140 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.140 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:33.140 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.428614+0000", 2024-09-21T12:33:33.140 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.140 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.140 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.140 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:33.140 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.428614+0000", 2024-09-21T12:33:33.140 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.140 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.140 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.140 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:33.140 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.428616+0000", 2024-09-21T12:33:33.140 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7969999999999999e-06 2024-09-21T12:33:33.140 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.140 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.141 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:33.141 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.428641+0000", 2024-09-21T12:33:33.141 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.4482e-05 2024-09-21T12:33:33.141 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.141 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.141 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:33.141 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.428656+0000", 2024-09-21T12:33:33.141 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5182e-05 2024-09-21T12:33:33.141 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.141 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.141 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:33.141 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.428664+0000", 2024-09-21T12:33:33.142 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1850000000000004e-06 2024-09-21T12:33:33.142 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.142 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.142 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.142 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.428717+0000", 2024-09-21T12:33:33.142 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3016999999999997e-05 2024-09-21T12:33:33.142 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.142 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.142 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:33.142 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:41.428723+0000", 2024-09-21T12:33:33.142 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5350000000000001e-06 2024-09-21T12:33:33.142 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.142 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.143 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.143 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.041634+0000", 2024-09-21T12:33:33.143 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.61291024199999999 2024-09-21T12:33:33.143 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.143 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.143 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:33.143 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.041730+0000", 2024-09-21T12:33:33.143 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6224999999999994e-05 2024-09-21T12:33:33.143 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.143 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.143 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,5", 2024-09-21T12:33:33.143 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.041851+0000", 2024-09-21T12:33:33.143 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000121112 2024-09-21T12:33:33.143 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.143 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.144 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:33.144 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.043228+0000", 2024-09-21T12:33:33.144 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0013771040000000001 2024-09-21T12:33:33.144 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.144 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.144 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.144 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.043713+0000", 2024-09-21T12:33:33.144 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00048459099999999998 2024-09-21T12:33:33.144 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.144 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.144 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.144 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.043942+0000", 2024-09-21T12:33:33.144 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00022905399999999999 2024-09-21T12:33:33.144 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.144 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.144 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:33.145 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.043994+0000", 2024-09-21T12:33:33.145 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2491000000000003e-05 2024-09-21T12:33:33.145 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.145 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.145 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:33.145 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:32:42.044099+0000", 2024-09-21T12:33:33.145 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000104893 2024-09-21T12:33:33.145 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.145 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:33.145 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.145 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.145 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.145 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.15924.0:43122 10.8 10:16c398dd:::benchmark_data_smithi114_161690_object43121:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-09-21T12:33:33.145 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:33:00.807359+0000", 2024-09-21T12:33:33.145 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 32.210861297000001, 2024-09-21T12:33:33.145 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.37883008099999999, 2024-09-21T12:33:33.146 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:33.146 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:33.146 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:33.146 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:33.146 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.15924", 2024-09-21T12:33:33.146 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:33:33.146 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 43122 2024-09-21T12:33:33.146 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.146 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:33.146 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.146 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:33.146 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.807359+0000", 2024-09-21T12:33:33.146 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.147 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.147 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.147 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:33.147 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.807359+0000", 2024-09-21T12:33:33.147 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.147 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.147 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.147 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:33.147 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.807360+0000", 2024-09-21T12:33:33.147 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1319999999999999e-06 2024-09-21T12:33:33.148 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.148 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.148 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:33.148 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.807380+0000", 2024-09-21T12:33:33.148 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0196000000000001e-05 2024-09-21T12:33:33.148 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.148 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.148 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:33.150 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.807388+0000", 2024-09-21T12:33:33.150 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.4910000000000005e-06 2024-09-21T12:33:33.150 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.151 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.151 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:33.151 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.807395+0000", 2024-09-21T12:33:33.151 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8210000000000004e-06 2024-09-21T12:33:33.151 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.151 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.151 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.151 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.807457+0000", 2024-09-21T12:33:33.151 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1968e-05 2024-09-21T12:33:33.151 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.151 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.151 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:33.152 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.807575+0000", 2024-09-21T12:33:33.152 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011745 2024-09-21T12:33:33.152 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.152 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.152 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,7", 2024-09-21T12:33:33.152 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.807697+0000", 2024-09-21T12:33:33.152 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012284700000000001 2024-09-21T12:33:33.152 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.152 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.152 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:33.152 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.809294+0000", 2024-09-21T12:33:33.152 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0015961529999999999 2024-09-21T12:33:33.152 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.152 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.153 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.153 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.810540+0000", 2024-09-21T12:33:33.153 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001246619 2024-09-21T12:33:33.153 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.153 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.153 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.153 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:01.185293+0000", 2024-09-21T12:33:33.153 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.374752476 2024-09-21T12:33:33.153 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.153 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.153 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:33.153 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:01.185346+0000", 2024-09-21T12:33:33.153 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3644999999999999e-05 2024-09-21T12:33:33.153 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.153 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.154 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:33.154 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:01.186189+0000", 2024-09-21T12:33:33.154 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00084228299999999999 2024-09-21T12:33:33.154 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.154 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:33.154 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.154 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.154 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.154 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.15924.0:43131 10.8 10:1b8763d4:::benchmark_data_smithi114_161690_object43130:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-09-21T12:33:33.154 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:33:00.810533+0000", 2024-09-21T12:33:33.155 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 32.207686871999996, 2024-09-21T12:33:33.155 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.37579112100000001, 2024-09-21T12:33:33.155 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:33.155 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:33.155 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:33.155 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:33.155 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.15924", 2024-09-21T12:33:33.155 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:33:33.155 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 43131 2024-09-21T12:33:33.155 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.155 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:33.155 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.155 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:33.156 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.810533+0000", 2024-09-21T12:33:33.156 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.156 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.156 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.156 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:33.156 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.810533+0000", 2024-09-21T12:33:33.156 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.156 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.156 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.156 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:33.156 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.810534+0000", 2024-09-21T12:33:33.157 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.283e-06 2024-09-21T12:33:33.157 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.157 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.157 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:33.157 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.810564+0000", 2024-09-21T12:33:33.157 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.9601e-05 2024-09-21T12:33:33.157 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.157 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.157 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:33.157 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.810585+0000", 2024-09-21T12:33:33.157 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1515e-05 2024-09-21T12:33:33.157 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.158 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.158 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:33.158 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.810602+0000", 2024-09-21T12:33:33.158 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6381000000000001e-05 2024-09-21T12:33:33.158 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.158 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.158 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.158 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.810648+0000", 2024-09-21T12:33:33.158 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6301999999999998e-05 2024-09-21T12:33:33.158 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.158 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.158 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:33.159 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.810828+0000", 2024-09-21T12:33:33.159 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017946499999999999 2024-09-21T12:33:33.159 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.159 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.159 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,7", 2024-09-21T12:33:33.159 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.810926+0000", 2024-09-21T12:33:33.159 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.8537999999999997e-05 2024-09-21T12:33:33.159 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.159 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.159 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:33.159 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.812456+0000", 2024-09-21T12:33:33.159 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001529751 2024-09-21T12:33:33.159 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.159 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.160 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.160 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.813140+0000", 2024-09-21T12:33:33.160 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00068376799999999996 2024-09-21T12:33:33.160 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.160 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.160 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.160 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:01.186237+0000", 2024-09-21T12:33:33.160 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.37309772600000002 2024-09-21T12:33:33.160 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.160 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.160 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:33.160 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:01.186286+0000", 2024-09-21T12:33:33.160 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8267999999999999e-05 2024-09-21T12:33:33.161 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.161 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.161 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:33.161 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:01.186324+0000", 2024-09-21T12:33:33.161 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8522999999999997e-05 2024-09-21T12:33:33.161 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.161 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:33.161 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.161 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.161 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.161 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.15924.0:43141 10.8 10:16aadd5e:::benchmark_data_smithi114_161690_object43140:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-09-21T12:33:33.161 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:33:00.816979+0000", 2024-09-21T12:33:33.161 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 32.201240370999997, 2024-09-21T12:33:33.161 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.36949421900000001, 2024-09-21T12:33:33.162 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:33.162 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:33.162 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:33.162 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:33.162 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.15924", 2024-09-21T12:33:33.162 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:33:33.162 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 43141 2024-09-21T12:33:33.162 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.162 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:33.162 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.162 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:33.162 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.816979+0000", 2024-09-21T12:33:33.162 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.163 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.163 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.163 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:33.163 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.816979+0000", 2024-09-21T12:33:33.163 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:33.163 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.163 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.163 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:33.163 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.816981+0000", 2024-09-21T12:33:33.163 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.426e-06 2024-09-21T12:33:33.163 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.163 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.163 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:33.164 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.817005+0000", 2024-09-21T12:33:33.164 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3762000000000001e-05 2024-09-21T12:33:33.164 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.164 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.164 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:33.164 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.817017+0000", 2024-09-21T12:33:33.164 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2765000000000001e-05 2024-09-21T12:33:33.164 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.164 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.164 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:33.164 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.817030+0000", 2024-09-21T12:33:33.164 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2194999999999999e-05 2024-09-21T12:33:33.164 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.164 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.164 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:33.165 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.817088+0000", 2024-09-21T12:33:33.165 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8828e-05 2024-09-21T12:33:33.165 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.165 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.165 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:33.165 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.817217+0000", 2024-09-21T12:33:33.165 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012878 2024-09-21T12:33:33.165 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.165 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.165 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,7", 2024-09-21T12:33:33.165 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.817314+0000", 2024-09-21T12:33:33.165 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6722e-05 2024-09-21T12:33:33.165 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.165 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.165 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:33.166 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.818693+0000", 2024-09-21T12:33:33.166 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001378663 2024-09-21T12:33:33.166 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.166 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.166 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.166 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:00.819475+0000", 2024-09-21T12:33:33.166 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00078237000000000001 2024-09-21T12:33:33.166 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.166 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.166 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:33.166 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:01.186401+0000", 2024-09-21T12:33:33.166 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.36692581000000002 2024-09-21T12:33:33.166 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.167 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.167 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:33.167 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:01.186442+0000", 2024-09-21T12:33:33.167 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0828999999999998e-05 2024-09-21T12:33:33.167 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:33.167 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:33.167 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:33.167 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:33:01.186474+0000", 2024-09-21T12:33:33.167 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2069000000000001e-05 2024-09-21T12:33:33.167 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.167 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:33.167 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.167 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:33.167 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:33.167 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:33:33.192 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:33.234 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:33.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:33.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:33.335 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:33.371 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:33.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:33.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:33.472 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:33.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:33 smithi183 ceph-mon[33277]: 4.14 scrub starts 2024-09-21T12:33:33.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:33 smithi183 ceph-mon[33277]: 4.14 scrub ok 2024-09-21T12:33:33.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:33 smithi183 ceph-mon[33277]: pgmap v683: 178 pgs: 178 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-21T12:33:33.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:33 smithi183 ceph-mon[33277]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:33:33.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:33 smithi183 ceph-mon[33277]: Cluster is now healthy 2024-09-21T12:33:33.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:33 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1071951439' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:33:33.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:33 smithi183 ceph-mon[33277]: osdmap e373: 8 total, 8 up, 8 in 2024-09-21T12:33:33.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:33 smithi183 ceph-mon[33277]: 1.d scrub starts 2024-09-21T12:33:33.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:33 smithi183 ceph-mon[33277]: 1.d scrub ok 2024-09-21T12:33:33.507 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:33.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:33.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:33.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:33 smithi114 ceph-mon[40867]: 4.14 scrub starts 2024-09-21T12:33:33.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:33 smithi114 ceph-mon[40867]: 4.14 scrub ok 2024-09-21T12:33:33.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:33 smithi114 ceph-mon[40867]: pgmap v683: 178 pgs: 178 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-21T12:33:33.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:33 smithi114 ceph-mon[40867]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:33:33.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:33 smithi114 ceph-mon[40867]: Cluster is now healthy 2024-09-21T12:33:33.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:33 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1071951439' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:33:33.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:33 smithi114 ceph-mon[40867]: osdmap e373: 8 total, 8 up, 8 in 2024-09-21T12:33:33.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:33 smithi114 ceph-mon[40867]: 1.d scrub starts 2024-09-21T12:33:33.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:33 smithi114 ceph-mon[40867]: 1.d scrub ok 2024-09-21T12:33:33.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:33 smithi114 ceph-mon[30324]: 4.14 scrub starts 2024-09-21T12:33:33.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:33 smithi114 ceph-mon[30324]: 4.14 scrub ok 2024-09-21T12:33:33.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:33 smithi114 ceph-mon[30324]: pgmap v683: 178 pgs: 178 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-21T12:33:33.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:33 smithi114 ceph-mon[30324]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:33:33.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:33 smithi114 ceph-mon[30324]: Cluster is now healthy 2024-09-21T12:33:33.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:33 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1071951439' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:33:33.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:33 smithi114 ceph-mon[30324]: osdmap e373: 8 total, 8 up, 8 in 2024-09-21T12:33:33.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:33 smithi114 ceph-mon[30324]: 1.d scrub starts 2024-09-21T12:33:33.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:33 smithi114 ceph-mon[30324]: 1.d scrub ok 2024-09-21T12:33:33.609 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:33.646 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:33.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:33.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:33.662 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 57 16 87199 87183 95.5678 83 0.00945446 0.0104588 2024-09-21T12:33:33.748 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:33.780 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:33.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:33.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:33.881 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:33.917 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:33.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:33.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:34.018 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:34.059 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:34.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:34.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:34.161 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:34.193 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:34.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:34.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:34.294 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:34.332 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:34.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:34.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:34.434 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:34.466 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:34.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:34.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:34.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:34 smithi183 ceph-mon[33277]: 3.1d scrub starts 2024-09-21T12:33:34.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:34 smithi183 ceph-mon[33277]: 3.1d scrub ok 2024-09-21T12:33:34.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:34 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:33:34.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:34 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:33:34.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:34 smithi183 ceph-mon[33277]: osdmap e374: 8 total, 8 up, 8 in 2024-09-21T12:33:34.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:34 smithi114 ceph-mon[30324]: 3.1d scrub starts 2024-09-21T12:33:34.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:34 smithi114 ceph-mon[30324]: 3.1d scrub ok 2024-09-21T12:33:34.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:34 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:33:34.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:34 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:33:34.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:34 smithi114 ceph-mon[30324]: osdmap e374: 8 total, 8 up, 8 in 2024-09-21T12:33:34.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:34 smithi114 ceph-mon[40867]: 3.1d scrub starts 2024-09-21T12:33:34.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:34 smithi114 ceph-mon[40867]: 3.1d scrub ok 2024-09-21T12:33:34.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:34 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:33:34.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:34 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:33:34.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:34 smithi114 ceph-mon[40867]: osdmap e374: 8 total, 8 up, 8 in 2024-09-21T12:33:34.567 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:34.601 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:34.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:34.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:34.662 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 58 16 88515 88499 95.3381 82.25 0.0198571 0.0104842 2024-09-21T12:33:34.703 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:34.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:34.737 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:34.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:34.838 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:34.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:34.868 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:34.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:34.969 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:35.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:35.005 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:35.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:35.106 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:35.138 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:35.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:35.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:35.240 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:35.243 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T12:33:35.272 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:35.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:35.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:35.374 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:35.404 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:33:35.404 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:33:35.404 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:33:35.404 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:33:35.404 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:33:35.404 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:33:35.404 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:33:35.404 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:33:35.405 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:33:35.405 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:33:35.405 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:33:35.405 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:33:35.405 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:33:35.405 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:33:35.405 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:33:35.405 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:33:35.406 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:35.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:35.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:35.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:35 smithi183 ceph-mon[33277]: 1.4 scrub starts 2024-09-21T12:33:35.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:35 smithi183 ceph-mon[33277]: 1.4 scrub ok 2024-09-21T12:33:35.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:35 smithi183 ceph-mon[33277]: pgmap v686: 178 pgs: 178 active+clean; 5.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-09-21T12:33:35.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:35 smithi183 ceph-mon[33277]: 1.1 scrub starts 2024-09-21T12:33:35.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:35 smithi183 ceph-mon[33277]: 1.1 scrub ok 2024-09-21T12:33:35.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:35 smithi183 ceph-mon[33277]: 5.d scrub starts 2024-09-21T12:33:35.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:35 smithi183 ceph-mon[33277]: 5.d scrub ok 2024-09-21T12:33:35.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:35 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3737964775' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:33:35.508 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:35.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:35 smithi114 ceph-mon[40867]: 1.4 scrub starts 2024-09-21T12:33:35.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:35 smithi114 ceph-mon[40867]: 1.4 scrub ok 2024-09-21T12:33:35.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:35 smithi114 ceph-mon[40867]: pgmap v686: 178 pgs: 178 active+clean; 5.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-09-21T12:33:35.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:35 smithi114 ceph-mon[40867]: 1.1 scrub starts 2024-09-21T12:33:35.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:35 smithi114 ceph-mon[40867]: 1.1 scrub ok 2024-09-21T12:33:35.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:35 smithi114 ceph-mon[40867]: 5.d scrub starts 2024-09-21T12:33:35.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:35 smithi114 ceph-mon[40867]: 5.d scrub ok 2024-09-21T12:33:35.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:35 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3737964775' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:33:35.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:35 smithi114 ceph-mon[30324]: 1.4 scrub starts 2024-09-21T12:33:35.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:35 smithi114 ceph-mon[30324]: 1.4 scrub ok 2024-09-21T12:33:35.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:35 smithi114 ceph-mon[30324]: pgmap v686: 178 pgs: 178 active+clean; 5.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-09-21T12:33:35.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:35 smithi114 ceph-mon[30324]: 1.1 scrub starts 2024-09-21T12:33:35.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:35 smithi114 ceph-mon[30324]: 1.1 scrub ok 2024-09-21T12:33:35.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:35 smithi114 ceph-mon[30324]: 5.d scrub starts 2024-09-21T12:33:35.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:35 smithi114 ceph-mon[30324]: 5.d scrub ok 2024-09-21T12:33:35.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:35 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3737964775' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:33:35.547 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:35.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:35.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:35.649 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:35.664 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 59 16 89734 89718 95.0112 76.1875 0.0101734 0.01052 2024-09-21T12:33:35.688 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:35.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:35.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:35.734 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T12:33:35.790 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:35.827 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:35.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:35.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:35.929 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:35.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:35.967 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:35.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:36.068 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:36.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:36.101 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:36.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:36.187 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:36.201 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:36.202 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:36.240 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:36.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:36.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:36.322 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:36 smithi114 ceph-mon[30324]: 5.16 scrub starts 2024-09-21T12:33:36.322 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:36 smithi114 ceph-mon[30324]: 5.16 scrub ok 2024-09-21T12:33:36.322 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:36 smithi114 ceph-mon[30324]: 10.f scrub starts 2024-09-21T12:33:36.323 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:36 smithi114 ceph-mon[40867]: 5.16 scrub starts 2024-09-21T12:33:36.323 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:36 smithi114 ceph-mon[40867]: 5.16 scrub ok 2024-09-21T12:33:36.323 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:36 smithi114 ceph-mon[40867]: 10.f scrub starts 2024-09-21T12:33:36.342 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:36.383 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:36.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:36.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:36.484 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:36.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:36 smithi183 ceph-mon[33277]: 5.16 scrub starts 2024-09-21T12:33:36.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:36 smithi183 ceph-mon[33277]: 5.16 scrub ok 2024-09-21T12:33:36.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:36 smithi183 ceph-mon[33277]: 10.f scrub starts 2024-09-21T12:33:36.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:36.522 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:36.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:36.623 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:36.659 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:36.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:36.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:36.664 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-21T12:33:36.663863+0000 min lat: 0.00124586 max lat: 2.02977 avg lat: 0.0105594 2024-09-21T12:33:36.664 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T12:33:36.664 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 60 16 90920 90904 94.663 74.125 0.0194862 0.0105594 2024-09-21T12:33:36.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:36.806 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:36.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:36.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:36.908 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:36.963 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:36.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:36.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:37.002 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:33:37.002 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:33:37.065 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:37.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:37.102 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:37.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:37.202 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:37.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:37.238 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:37.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:37.339 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:37.377 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:37.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:37.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:37.438 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T12:33:37.438 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:33:37.439 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T12:33:37.439 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-21T12:33:37.479 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:37 smithi114 ceph-mon[40867]: 4.e scrub starts 2024-09-21T12:33:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:37 smithi114 ceph-mon[40867]: 4.e scrub ok 2024-09-21T12:33:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:37 smithi114 ceph-mon[40867]: pgmap v687: 178 pgs: 178 active+clean; 5.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 902 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-09-21T12:33:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:37 smithi114 ceph-mon[40867]: 5.1e scrub starts 2024-09-21T12:33:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:37 smithi114 ceph-mon[40867]: 5.1e scrub ok 2024-09-21T12:33:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:37 smithi114 ceph-mon[40867]: 6.19 scrub starts 2024-09-21T12:33:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:37 smithi114 ceph-mon[40867]: 6.19 scrub ok 2024-09-21T12:33:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:37 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:33:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:37 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:33:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:37 smithi114 ceph-mon[40867]: 6.9 scrub starts 2024-09-21T12:33:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:37 smithi114 ceph-mon[40867]: 6.9 scrub ok 2024-09-21T12:33:37.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:37 smithi114 ceph-mon[30324]: 4.e scrub starts 2024-09-21T12:33:37.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:37 smithi114 ceph-mon[30324]: 4.e scrub ok 2024-09-21T12:33:37.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:37 smithi114 ceph-mon[30324]: pgmap v687: 178 pgs: 178 active+clean; 5.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 902 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-09-21T12:33:37.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:37 smithi114 ceph-mon[30324]: 5.1e scrub starts 2024-09-21T12:33:37.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:37 smithi114 ceph-mon[30324]: 5.1e scrub ok 2024-09-21T12:33:37.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:37 smithi114 ceph-mon[30324]: 6.19 scrub starts 2024-09-21T12:33:37.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:37 smithi114 ceph-mon[30324]: 6.19 scrub ok 2024-09-21T12:33:37.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:37 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:33:37.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:37 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:33:37.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:37 smithi114 ceph-mon[30324]: 6.9 scrub starts 2024-09-21T12:33:37.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:37 smithi114 ceph-mon[30324]: 6.9 scrub ok 2024-09-21T12:33:37.543 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:37.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:37.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:37.645 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:37.664 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 61 16 92109 92093 94.3294 74.3125 0.016078 0.0105961 2024-09-21T12:33:37.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:37.690 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:37.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:37.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:33:37 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:33:37.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:37 smithi183 ceph-mon[33277]: 4.e scrub starts 2024-09-21T12:33:37.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:37 smithi183 ceph-mon[33277]: 4.e scrub ok 2024-09-21T12:33:37.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:37 smithi183 ceph-mon[33277]: pgmap v687: 178 pgs: 178 active+clean; 5.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 902 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-09-21T12:33:37.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:37 smithi183 ceph-mon[33277]: 5.1e scrub starts 2024-09-21T12:33:37.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:37 smithi183 ceph-mon[33277]: 5.1e scrub ok 2024-09-21T12:33:37.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:37 smithi183 ceph-mon[33277]: 6.19 scrub starts 2024-09-21T12:33:37.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:37 smithi183 ceph-mon[33277]: 6.19 scrub ok 2024-09-21T12:33:37.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:37 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:33:37.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:37 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:33:37.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:37 smithi183 ceph-mon[33277]: 6.9 scrub starts 2024-09-21T12:33:37.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:37 smithi183 ceph-mon[33277]: 6.9 scrub ok 2024-09-21T12:33:37.791 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:37.827 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:37.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:37.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:37.928 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:37.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:37.973 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:37.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:38.074 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:38.109 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:38.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:38.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:38.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:33:38.211 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:38.232 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:38.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:38.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:38.263 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-21T12:33:38.334 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:38.387 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:38.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:38.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:38.489 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:38.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:38.525 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:38.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:38.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:38 smithi114 ceph-mon[30324]: 1.1e scrub starts 2024-09-21T12:33:38.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:38 smithi114 ceph-mon[30324]: 1.1e scrub ok 2024-09-21T12:33:38.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:38 smithi114 ceph-mon[30324]: 10.f scrub ok 2024-09-21T12:33:38.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:38 smithi114 ceph-mon[30324]: osdmap e375: 8 total, 8 up, 8 in 2024-09-21T12:33:38.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:38 smithi114 ceph-mon[30324]: 6.1c scrub starts 2024-09-21T12:33:38.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:38 smithi114 ceph-mon[30324]: 6.1c scrub ok 2024-09-21T12:33:38.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:38 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:38.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:38 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2922233742' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:33:38.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:38 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:33:38.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:38 smithi114 ceph-mon[40867]: 1.1e scrub starts 2024-09-21T12:33:38.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:38 smithi114 ceph-mon[40867]: 1.1e scrub ok 2024-09-21T12:33:38.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:38 smithi114 ceph-mon[40867]: 10.f scrub ok 2024-09-21T12:33:38.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:38 smithi114 ceph-mon[40867]: osdmap e375: 8 total, 8 up, 8 in 2024-09-21T12:33:38.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:38 smithi114 ceph-mon[40867]: 6.1c scrub starts 2024-09-21T12:33:38.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:38 smithi114 ceph-mon[40867]: 6.1c scrub ok 2024-09-21T12:33:38.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:38 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:38.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:38 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2922233742' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:33:38.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:38 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:33:38.625 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:38.664 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:38.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:38.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:38.665 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 62 16 93257 93241 93.9642 71.75 0.0300064 0.0106372 2024-09-21T12:33:38.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:38 smithi183 ceph-mon[33277]: 1.1e scrub starts 2024-09-21T12:33:38.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:38 smithi183 ceph-mon[33277]: 1.1e scrub ok 2024-09-21T12:33:38.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:38 smithi183 ceph-mon[33277]: 10.f scrub ok 2024-09-21T12:33:38.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:38 smithi183 ceph-mon[33277]: osdmap e375: 8 total, 8 up, 8 in 2024-09-21T12:33:38.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:38 smithi183 ceph-mon[33277]: 6.1c scrub starts 2024-09-21T12:33:38.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:38 smithi183 ceph-mon[33277]: 6.1c scrub ok 2024-09-21T12:33:38.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:38 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:38.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:38 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2922233742' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:33:38.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:38 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:33:38.766 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:38.788 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:38.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:38.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:38.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:38.925 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:38.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:38.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:39.027 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:39.061 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:39.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:39.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:39.163 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:39.193 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:39.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:39.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:39.295 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:39.332 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:39.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:39.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:39.434 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:39.459 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:39.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:39.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:39 smithi114 ceph-mon[30324]: 1.c scrub starts 2024-09-21T12:33:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:39 smithi114 ceph-mon[30324]: 1.c scrub ok 2024-09-21T12:33:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:39 smithi114 ceph-mon[30324]: pgmap v689: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 5.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-21T12:33:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:39 smithi114 ceph-mon[30324]: 4.3 scrub starts 2024-09-21T12:33:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:39 smithi114 ceph-mon[30324]: 4.3 scrub ok 2024-09-21T12:33:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:39 smithi114 ceph-mon[30324]: 1.8 scrub starts 2024-09-21T12:33:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:39 smithi114 ceph-mon[30324]: 1.8 scrub ok 2024-09-21T12:33:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:39 smithi114 ceph-mon[30324]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:33:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:39 smithi114 ceph-mon[30324]: 3.4 scrub starts 2024-09-21T12:33:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:39 smithi114 ceph-mon[30324]: 3.4 scrub ok 2024-09-21T12:33:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:39 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:33:39.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:39 smithi114 ceph-mon[30324]: osdmap e376: 8 total, 8 up, 8 in 2024-09-21T12:33:39.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:39 smithi114 ceph-mon[40867]: 1.c scrub starts 2024-09-21T12:33:39.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:39 smithi114 ceph-mon[40867]: 1.c scrub ok 2024-09-21T12:33:39.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:39 smithi114 ceph-mon[40867]: pgmap v689: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 5.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-21T12:33:39.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:39 smithi114 ceph-mon[40867]: 4.3 scrub starts 2024-09-21T12:33:39.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:39 smithi114 ceph-mon[40867]: 4.3 scrub ok 2024-09-21T12:33:39.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:39 smithi114 ceph-mon[40867]: 1.8 scrub starts 2024-09-21T12:33:39.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:39 smithi114 ceph-mon[40867]: 1.8 scrub ok 2024-09-21T12:33:39.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:39 smithi114 ceph-mon[40867]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:33:39.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:39 smithi114 ceph-mon[40867]: 3.4 scrub starts 2024-09-21T12:33:39.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:39 smithi114 ceph-mon[40867]: 3.4 scrub ok 2024-09-21T12:33:39.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:39 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:33:39.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:39 smithi114 ceph-mon[40867]: osdmap e376: 8 total, 8 up, 8 in 2024-09-21T12:33:39.561 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:39.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:39.599 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:39.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:39.665 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 63 16 94442 94426 93.6482 74.0625 0.00816026 0.0106741 2024-09-21T12:33:39.699 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:39.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:39 smithi183 ceph-mon[33277]: 1.c scrub starts 2024-09-21T12:33:39.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:39 smithi183 ceph-mon[33277]: 1.c scrub ok 2024-09-21T12:33:39.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:39 smithi183 ceph-mon[33277]: pgmap v689: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 5.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-21T12:33:39.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:39 smithi183 ceph-mon[33277]: 4.3 scrub starts 2024-09-21T12:33:39.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:39 smithi183 ceph-mon[33277]: 4.3 scrub ok 2024-09-21T12:33:39.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:39 smithi183 ceph-mon[33277]: 1.8 scrub starts 2024-09-21T12:33:39.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:39 smithi183 ceph-mon[33277]: 1.8 scrub ok 2024-09-21T12:33:39.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:39 smithi183 ceph-mon[33277]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:33:39.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:39 smithi183 ceph-mon[33277]: 3.4 scrub starts 2024-09-21T12:33:39.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:39 smithi183 ceph-mon[33277]: 3.4 scrub ok 2024-09-21T12:33:39.740 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:39 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:33:39.740 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:39 smithi183 ceph-mon[33277]: osdmap e376: 8 total, 8 up, 8 in 2024-09-21T12:33:39.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:39.742 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:39.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:39.843 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:39.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:39.881 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:39.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:39.950 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T12:33:39.982 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:40.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:40.034 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:40.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:40.135 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:40.186 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:40.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:40.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:40.288 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:40.320 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:40.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:40.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:40.422 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:40.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:40.460 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:40.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:40.561 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:40.599 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:40.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:40.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:40.665 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 64 16 95715 95699 93.4281 79.5625 0.00687357 0.010699 2024-09-21T12:33:40.684 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:40.685 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:40.701 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:40.728 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:40.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:40.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:40.830 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:40.871 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:40.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:40.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:40.973 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:41.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:41.011 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:41.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:41.112 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:41.146 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:41.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:41.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:41.248 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:41.277 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:41.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:41.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:41.372 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:33:41.373 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:33:41.379 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:41.411 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:33:41.411 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:33:41.421 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:41.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:41.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:41.443 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:33:41.443 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:33:41.444 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:41 smithi114 ceph-mon[40867]: pgmap v691: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 770 B/s rd, 60 MiB/s wr, 956 op/s 2024-09-21T12:33:41.444 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:41 smithi114 ceph-mon[30324]: pgmap v691: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 770 B/s rd, 60 MiB/s wr, 956 op/s 2024-09-21T12:33:41.485 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:33:41.485 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:33:41.519 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:33:41.519 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:33:41.524 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:41.550 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:33:41.551 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:33:41.553 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:41.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:41.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:41.590 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:33:41.590 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:33:41.624 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:33:41.624 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:33:41.655 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:41.665 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 65 16 96966 96950 93.1936 78.1875 0.0186161 0.0107259 2024-09-21T12:33:41.688 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:41.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:41.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:41.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:41 smithi183 ceph-mon[33277]: pgmap v691: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 770 B/s rd, 60 MiB/s wr, 956 op/s 2024-09-21T12:33:41.790 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:41.821 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:41.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:41.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:41.922 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:41.962 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:41.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:41.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:42.064 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:42.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:42.103 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:42.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:42.204 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:42.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:42.239 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:42.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:42.340 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:42.377 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:42.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:42.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:42.478 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:42.518 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:42.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:42.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:42.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:42 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1302276288' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:33:42.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:42 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1302276288' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:33:42.620 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:42.661 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:42.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:42.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:42.665 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 66 16 98326 98310 93.0694 85 0.00767232 0.01074 2024-09-21T12:33:42.674 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T12:33:42.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:42 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1302276288' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:33:42.763 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:42.819 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:42.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:42.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:42.921 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:42.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:42.964 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:42.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:43.024 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:33:42 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:33:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:33:43.065 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:43.098 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:43.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:43.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:43.200 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:43.243 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:43.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:43.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:43.345 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:43.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:43.382 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:43.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:43.483 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:43.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:43.524 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:43.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:43.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:43 smithi114 ceph-mon[30324]: pgmap v692: 178 pgs: 178 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.13k op/s 2024-09-21T12:33:43.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:43 smithi114 ceph-mon[40867]: pgmap v692: 178 pgs: 178 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.13k op/s 2024-09-21T12:33:43.624 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:43.665 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 67 16 99618 99602 92.8854 80.75 0.0112156 0.0107615 2024-09-21T12:33:43.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:43.672 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:43.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:43.696 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:43.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:43 smithi183 ceph-mon[33277]: pgmap v692: 178 pgs: 178 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.13k op/s 2024-09-21T12:33:43.773 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:43.833 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:43.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:43.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:43.935 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:43.974 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:43.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:43.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:44.006 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:44.076 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:44.095 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:33:44.095 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:33:44.096 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 376, 2024-09-21T12:33:44.096 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "e2ad975c-7812-11ef-baf6-efdc52797490", 2024-09-21T12:33:44.096 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-21T12:14:24.181227+0000", 2024-09-21T12:33:44.096 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-21T12:33:38.241389+0000", 2024-09-21T12:33:44.096 INFO:teuthology.orchestra.run.smithi114.stdout: "last_up_change": "2024-09-21T12:20:03.702987+0000", 2024-09-21T12:33:44.096 INFO:teuthology.orchestra.run.smithi114.stdout: "last_in_change": "2024-09-21T12:32:37.364926+0000", 2024-09-21T12:33:44.096 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T12:33:44.096 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_num": 5801984, 2024-09-21T12:33:44.096 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_set": [ 2024-09-21T12:33:44.096 INFO:teuthology.orchestra.run.smithi114.stdout: "noscrub", 2024-09-21T12:33:44.096 INFO:teuthology.orchestra.run.smithi114.stdout: "pglog_hardlimit", 2024-09-21T12:33:44.096 INFO:teuthology.orchestra.run.smithi114.stdout: "purged_snapdirs", 2024-09-21T12:33:44.097 INFO:teuthology.orchestra.run.smithi114.stdout: "recovery_deletes", 2024-09-21T12:33:44.097 INFO:teuthology.orchestra.run.smithi114.stdout: "sortbitwise" 2024-09-21T12:33:44.097 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:33:44.097 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_version": 18, 2024-09-21T12:33:44.097 INFO:teuthology.orchestra.run.smithi114.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T12:33:44.097 INFO:teuthology.orchestra.run.smithi114.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T12:33:44.097 INFO:teuthology.orchestra.run.smithi114.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T12:33:44.097 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_snapshot": "", 2024-09-21T12:33:44.097 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_max": 10, 2024-09-21T12:33:44.097 INFO:teuthology.orchestra.run.smithi114.stdout: "max_osd": 8, 2024-09-21T12:33:44.097 INFO:teuthology.orchestra.run.smithi114.stdout: "require_min_compat_client": "luminous", 2024-09-21T12:33:44.097 INFO:teuthology.orchestra.run.smithi114.stdout: "min_compat_client": "luminous", 2024-09-21T12:33:44.097 INFO:teuthology.orchestra.run.smithi114.stdout: "require_osd_release": "squid", 2024-09-21T12:33:44.097 INFO:teuthology.orchestra.run.smithi114.stdout: "allow_crimson": false, 2024-09-21T12:33:44.097 INFO:teuthology.orchestra.run.smithi114.stdout: "pools": [ 2024-09-21T12:33:44.098 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.098 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 1, 2024-09-21T12:33:44.098 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": ".mgr", 2024-09-21T12:33:44.098 INFO:teuthology.orchestra.run.smithi114.stdout: "create_time": "2024-09-21T12:17:38.357647+0000", 2024-09-21T12:33:44.098 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-09-21T12:33:44.098 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_names": "hashpspool", 2024-09-21T12:33:44.098 INFO:teuthology.orchestra.run.smithi114.stdout: "type": 1, 2024-09-21T12:33:44.098 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 3, 2024-09-21T12:33:44.098 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": 2, 2024-09-21T12:33:44.098 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_rule": 0, 2024-09-21T12:33:44.098 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_count": 0, 2024-09-21T12:33:44.098 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_target": 0, 2024-09-21T12:33:44.098 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T12:33:44.098 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T12:33:44.099 INFO:teuthology.orchestra.run.smithi114.stdout: "is_stretch_pool": false, 2024-09-21T12:33:44.099 INFO:teuthology.orchestra.run.smithi114.stdout: "object_hash": 2, 2024-09-21T12:33:44.099 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_mode": "off", 2024-09-21T12:33:44.099 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": 31, 2024-09-21T12:33:44.099 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num": 31, 2024-09-21T12:33:44.099 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num_target": 31, 2024-09-21T12:33:44.099 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_target": 31, 2024-09-21T12:33:44.099 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_pending": 31, 2024-09-21T12:33:44.099 INFO:teuthology.orchestra.run.smithi114.stdout: "last_pg_merge_meta": { 2024-09-21T12:33:44.099 INFO:teuthology.orchestra.run.smithi114.stdout: "source_pgid": "0.0", 2024-09-21T12:33:44.099 INFO:teuthology.orchestra.run.smithi114.stdout: "ready_epoch": 0, 2024-09-21T12:33:44.099 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_started": 0, 2024-09-21T12:33:44.099 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_clean": 0, 2024-09-21T12:33:44.099 INFO:teuthology.orchestra.run.smithi114.stdout: "source_version": "0'0", 2024-09-21T12:33:44.099 INFO:teuthology.orchestra.run.smithi114.stdout: "target_version": "0'0" 2024-09-21T12:33:44.100 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.100 INFO:teuthology.orchestra.run.smithi114.stdout: "last_change": "218", 2024-09-21T12:33:44.100 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend": "0", 2024-09-21T12:33:44.100 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T12:33:44.100 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_preluminous": "216", 2024-09-21T12:33:44.100 INFO:teuthology.orchestra.run.smithi114.stdout: "auid": 0, 2024-09-21T12:33:44.100 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_mode": "selfmanaged", 2024-09-21T12:33:44.100 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_seq": 0, 2024-09-21T12:33:44.100 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_epoch": 0, 2024-09-21T12:33:44.100 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_snaps": [], 2024-09-21T12:33:44.100 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps": "[]", 2024-09-21T12:33:44.100 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_bytes": 0, 2024-09-21T12:33:44.100 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_objects": 0, 2024-09-21T12:33:44.100 INFO:teuthology.orchestra.run.smithi114.stdout: "tiers": [], 2024-09-21T12:33:44.100 INFO:teuthology.orchestra.run.smithi114.stdout: "tier_of": -1, 2024-09-21T12:33:44.101 INFO:teuthology.orchestra.run.smithi114.stdout: "read_tier": -1, 2024-09-21T12:33:44.101 INFO:teuthology.orchestra.run.smithi114.stdout: "write_tier": -1, 2024-09-21T12:33:44.101 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_mode": "none", 2024-09-21T12:33:44.101 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_bytes": 0, 2024-09-21T12:33:44.101 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_objects": 0, 2024-09-21T12:33:44.101 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T12:33:44.101 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T12:33:44.101 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T12:33:44.101 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_flush_age": 0, 2024-09-21T12:33:44.101 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_evict_age": 0, 2024-09-21T12:33:44.101 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profile": "", 2024-09-21T12:33:44.101 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_params": { 2024-09-21T12:33:44.101 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "none" 2024-09-21T12:33:44.101 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.101 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_period": 0, 2024-09-21T12:33:44.102 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_count": 0, 2024-09-21T12:33:44.102 INFO:teuthology.orchestra.run.smithi114.stdout: "use_gmt_hitset": true, 2024-09-21T12:33:44.102 INFO:teuthology.orchestra.run.smithi114.stdout: "min_read_recency_for_promote": 0, 2024-09-21T12:33:44.102 INFO:teuthology.orchestra.run.smithi114.stdout: "min_write_recency_for_promote": 0, 2024-09-21T12:33:44.102 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T12:33:44.102 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_search_last_n": 0, 2024-09-21T12:33:44.102 INFO:teuthology.orchestra.run.smithi114.stdout: "grade_table": [], 2024-09-21T12:33:44.102 INFO:teuthology.orchestra.run.smithi114.stdout: "stripe_width": 0, 2024-09-21T12:33:44.102 INFO:teuthology.orchestra.run.smithi114.stdout: "expected_num_objects": 0, 2024-09-21T12:33:44.102 INFO:teuthology.orchestra.run.smithi114.stdout: "fast_read": false, 2024-09-21T12:33:44.102 INFO:teuthology.orchestra.run.smithi114.stdout: "options": { 2024-09-21T12:33:44.102 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_max": 32, 2024-09-21T12:33:44.102 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_min": 1 2024-09-21T12:33:44.102 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.103 INFO:teuthology.orchestra.run.smithi114.stdout: "application_metadata": { 2024-09-21T12:33:44.103 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": {} 2024-09-21T12:33:44.103 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.103 INFO:teuthology.orchestra.run.smithi114.stdout: "read_balance": { 2024-09-21T12:33:44.103 INFO:teuthology.orchestra.run.smithi114.stdout: "score_type": "Fair distribution", 2024-09-21T12:33:44.103 INFO:teuthology.orchestra.run.smithi114.stdout: "score_acting": 1.809999942779541, 2024-09-21T12:33:44.103 INFO:teuthology.orchestra.run.smithi114.stdout: "score_stable": 1.809999942779541, 2024-09-21T12:33:44.103 INFO:teuthology.orchestra.run.smithi114.stdout: "optimal_score": 1, 2024-09-21T12:33:44.103 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_acting": 1.809999942779541, 2024-09-21T12:33:44.103 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_stable": 1.809999942779541, 2024-09-21T12:33:44.103 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity_weighted": 0.86082649230957031, 2024-09-21T12:33:44.103 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity": 0.86000001430511475, 2024-09-21T12:33:44.103 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity_weighted": 0.86000001430511475 2024-09-21T12:33:44.103 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.103 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.104 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.104 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 2, 2024-09-21T12:33:44.104 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": "datapool", 2024-09-21T12:33:44.104 INFO:teuthology.orchestra.run.smithi114.stdout: "create_time": "2024-09-21T12:20:12.761046+0000", 2024-09-21T12:33:44.104 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 8193, 2024-09-21T12:33:44.104 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T12:33:44.104 INFO:teuthology.orchestra.run.smithi114.stdout: "type": 1, 2024-09-21T12:33:44.104 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 3, 2024-09-21T12:33:44.104 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": 2, 2024-09-21T12:33:44.104 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_rule": 0, 2024-09-21T12:33:44.104 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_count": 0, 2024-09-21T12:33:44.104 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_target": 0, 2024-09-21T12:33:44.104 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T12:33:44.104 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T12:33:44.104 INFO:teuthology.orchestra.run.smithi114.stdout: "is_stretch_pool": false, 2024-09-21T12:33:44.104 INFO:teuthology.orchestra.run.smithi114.stdout: "object_hash": 2, 2024-09-21T12:33:44.105 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_mode": "off", 2024-09-21T12:33:44.105 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": 3, 2024-09-21T12:33:44.105 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num": 3, 2024-09-21T12:33:44.105 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num_target": 3, 2024-09-21T12:33:44.105 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_target": 3, 2024-09-21T12:33:44.105 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_pending": 3, 2024-09-21T12:33:44.105 INFO:teuthology.orchestra.run.smithi114.stdout: "last_pg_merge_meta": { 2024-09-21T12:33:44.105 INFO:teuthology.orchestra.run.smithi114.stdout: "source_pgid": "0.0", 2024-09-21T12:33:44.105 INFO:teuthology.orchestra.run.smithi114.stdout: "ready_epoch": 0, 2024-09-21T12:33:44.105 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_started": 0, 2024-09-21T12:33:44.105 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_clean": 0, 2024-09-21T12:33:44.105 INFO:teuthology.orchestra.run.smithi114.stdout: "source_version": "0'0", 2024-09-21T12:33:44.105 INFO:teuthology.orchestra.run.smithi114.stdout: "target_version": "0'0" 2024-09-21T12:33:44.105 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.105 INFO:teuthology.orchestra.run.smithi114.stdout: "last_change": "61", 2024-09-21T12:33:44.106 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend": "0", 2024-09-21T12:33:44.106 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T12:33:44.106 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T12:33:44.106 INFO:teuthology.orchestra.run.smithi114.stdout: "auid": 0, 2024-09-21T12:33:44.106 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_mode": "selfmanaged", 2024-09-21T12:33:44.106 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_seq": 2, 2024-09-21T12:33:44.106 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_epoch": 61, 2024-09-21T12:33:44.106 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_snaps": [], 2024-09-21T12:33:44.106 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps": "[]", 2024-09-21T12:33:44.106 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_bytes": 0, 2024-09-21T12:33:44.106 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_objects": 0, 2024-09-21T12:33:44.106 INFO:teuthology.orchestra.run.smithi114.stdout: "tiers": [], 2024-09-21T12:33:44.106 INFO:teuthology.orchestra.run.smithi114.stdout: "tier_of": -1, 2024-09-21T12:33:44.106 INFO:teuthology.orchestra.run.smithi114.stdout: "read_tier": -1, 2024-09-21T12:33:44.107 INFO:teuthology.orchestra.run.smithi114.stdout: "write_tier": -1, 2024-09-21T12:33:44.107 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_mode": "none", 2024-09-21T12:33:44.107 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_bytes": 0, 2024-09-21T12:33:44.107 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_objects": 0, 2024-09-21T12:33:44.107 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T12:33:44.107 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T12:33:44.107 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T12:33:44.107 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_flush_age": 0, 2024-09-21T12:33:44.107 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_evict_age": 0, 2024-09-21T12:33:44.107 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profile": "", 2024-09-21T12:33:44.107 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_params": { 2024-09-21T12:33:44.107 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "none" 2024-09-21T12:33:44.107 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.107 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_period": 0, 2024-09-21T12:33:44.107 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_count": 0, 2024-09-21T12:33:44.107 INFO:teuthology.orchestra.run.smithi114.stdout: "use_gmt_hitset": true, 2024-09-21T12:33:44.108 INFO:teuthology.orchestra.run.smithi114.stdout: "min_read_recency_for_promote": 0, 2024-09-21T12:33:44.108 INFO:teuthology.orchestra.run.smithi114.stdout: "min_write_recency_for_promote": 0, 2024-09-21T12:33:44.108 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T12:33:44.108 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_search_last_n": 0, 2024-09-21T12:33:44.108 INFO:teuthology.orchestra.run.smithi114.stdout: "grade_table": [], 2024-09-21T12:33:44.108 INFO:teuthology.orchestra.run.smithi114.stdout: "stripe_width": 0, 2024-09-21T12:33:44.108 INFO:teuthology.orchestra.run.smithi114.stdout: "expected_num_objects": 0, 2024-09-21T12:33:44.108 INFO:teuthology.orchestra.run.smithi114.stdout: "fast_read": false, 2024-09-21T12:33:44.108 INFO:teuthology.orchestra.run.smithi114.stdout: "options": {}, 2024-09-21T12:33:44.108 INFO:teuthology.orchestra.run.smithi114.stdout: "application_metadata": { 2024-09-21T12:33:44.108 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd": {} 2024-09-21T12:33:44.108 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.108 INFO:teuthology.orchestra.run.smithi114.stdout: "read_balance": { 2024-09-21T12:33:44.108 INFO:teuthology.orchestra.run.smithi114.stdout: "score_type": "Fair distribution", 2024-09-21T12:33:44.109 INFO:teuthology.orchestra.run.smithi114.stdout: "score_acting": 2.6500000953674316, 2024-09-21T12:33:44.109 INFO:teuthology.orchestra.run.smithi114.stdout: "score_stable": 2.6500000953674316, 2024-09-21T12:33:44.109 INFO:teuthology.orchestra.run.smithi114.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T12:33:44.109 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-21T12:33:44.109 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-21T12:33:44.109 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity_weighted": 0.86082649230957031, 2024-09-21T12:33:44.109 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity": 0.86000001430511475, 2024-09-21T12:33:44.109 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity_weighted": 0.86000001430511475 2024-09-21T12:33:44.109 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.109 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.109 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.109 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 3, 2024-09-21T12:33:44.109 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": ".rgw.root", 2024-09-21T12:33:44.109 INFO:teuthology.orchestra.run.smithi114.stdout: "create_time": "2024-09-21T12:20:13.713971+0000", 2024-09-21T12:33:44.110 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-09-21T12:33:44.110 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_names": "hashpspool", 2024-09-21T12:33:44.110 INFO:teuthology.orchestra.run.smithi114.stdout: "type": 1, 2024-09-21T12:33:44.110 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 3, 2024-09-21T12:33:44.110 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": 2, 2024-09-21T12:33:44.110 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_rule": 0, 2024-09-21T12:33:44.110 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_count": 0, 2024-09-21T12:33:44.110 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_target": 0, 2024-09-21T12:33:44.110 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T12:33:44.110 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T12:33:44.110 INFO:teuthology.orchestra.run.smithi114.stdout: "is_stretch_pool": false, 2024-09-21T12:33:44.110 INFO:teuthology.orchestra.run.smithi114.stdout: "object_hash": 2, 2024-09-21T12:33:44.110 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_mode": "off", 2024-09-21T12:33:44.111 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": 32, 2024-09-21T12:33:44.111 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num": 32, 2024-09-21T12:33:44.111 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num_target": 32, 2024-09-21T12:33:44.111 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_target": 32, 2024-09-21T12:33:44.111 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_pending": 32, 2024-09-21T12:33:44.111 INFO:teuthology.orchestra.run.smithi114.stdout: "last_pg_merge_meta": { 2024-09-21T12:33:44.111 INFO:teuthology.orchestra.run.smithi114.stdout: "source_pgid": "0.0", 2024-09-21T12:33:44.111 INFO:teuthology.orchestra.run.smithi114.stdout: "ready_epoch": 0, 2024-09-21T12:33:44.111 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_started": 0, 2024-09-21T12:33:44.111 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_clean": 0, 2024-09-21T12:33:44.111 INFO:teuthology.orchestra.run.smithi114.stdout: "source_version": "0'0", 2024-09-21T12:33:44.111 INFO:teuthology.orchestra.run.smithi114.stdout: "target_version": "0'0" 2024-09-21T12:33:44.111 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.111 INFO:teuthology.orchestra.run.smithi114.stdout: "last_change": "60", 2024-09-21T12:33:44.112 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend": "0", 2024-09-21T12:33:44.112 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T12:33:44.112 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T12:33:44.112 INFO:teuthology.orchestra.run.smithi114.stdout: "auid": 0, 2024-09-21T12:33:44.112 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_mode": "selfmanaged", 2024-09-21T12:33:44.112 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_seq": 0, 2024-09-21T12:33:44.112 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_epoch": 0, 2024-09-21T12:33:44.112 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_snaps": [], 2024-09-21T12:33:44.112 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps": "[]", 2024-09-21T12:33:44.112 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_bytes": 0, 2024-09-21T12:33:44.112 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_objects": 0, 2024-09-21T12:33:44.112 INFO:teuthology.orchestra.run.smithi114.stdout: "tiers": [], 2024-09-21T12:33:44.112 INFO:teuthology.orchestra.run.smithi114.stdout: "tier_of": -1, 2024-09-21T12:33:44.112 INFO:teuthology.orchestra.run.smithi114.stdout: "read_tier": -1, 2024-09-21T12:33:44.113 INFO:teuthology.orchestra.run.smithi114.stdout: "write_tier": -1, 2024-09-21T12:33:44.113 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_mode": "none", 2024-09-21T12:33:44.113 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_bytes": 0, 2024-09-21T12:33:44.113 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_objects": 0, 2024-09-21T12:33:44.113 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T12:33:44.113 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T12:33:44.113 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T12:33:44.113 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_flush_age": 0, 2024-09-21T12:33:44.113 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_evict_age": 0, 2024-09-21T12:33:44.113 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profile": "", 2024-09-21T12:33:44.113 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_params": { 2024-09-21T12:33:44.113 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "none" 2024-09-21T12:33:44.113 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.113 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_period": 0, 2024-09-21T12:33:44.113 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_count": 0, 2024-09-21T12:33:44.114 INFO:teuthology.orchestra.run.smithi114.stdout: "use_gmt_hitset": true, 2024-09-21T12:33:44.114 INFO:teuthology.orchestra.run.smithi114.stdout: "min_read_recency_for_promote": 0, 2024-09-21T12:33:44.114 INFO:teuthology.orchestra.run.smithi114.stdout: "min_write_recency_for_promote": 0, 2024-09-21T12:33:44.114 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T12:33:44.114 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_search_last_n": 0, 2024-09-21T12:33:44.114 INFO:teuthology.orchestra.run.smithi114.stdout: "grade_table": [], 2024-09-21T12:33:44.114 INFO:teuthology.orchestra.run.smithi114.stdout: "stripe_width": 0, 2024-09-21T12:33:44.114 INFO:teuthology.orchestra.run.smithi114.stdout: "expected_num_objects": 0, 2024-09-21T12:33:44.114 INFO:teuthology.orchestra.run.smithi114.stdout: "fast_read": false, 2024-09-21T12:33:44.114 INFO:teuthology.orchestra.run.smithi114.stdout: "options": {}, 2024-09-21T12:33:44.114 INFO:teuthology.orchestra.run.smithi114.stdout: "application_metadata": { 2024-09-21T12:33:44.114 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw": {} 2024-09-21T12:33:44.115 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.115 INFO:teuthology.orchestra.run.smithi114.stdout: "read_balance": { 2024-09-21T12:33:44.115 INFO:teuthology.orchestra.run.smithi114.stdout: "score_type": "Fair distribution", 2024-09-21T12:33:44.115 INFO:teuthology.orchestra.run.smithi114.stdout: "score_acting": 1.75, 2024-09-21T12:33:44.115 INFO:teuthology.orchestra.run.smithi114.stdout: "score_stable": 1.75, 2024-09-21T12:33:44.115 INFO:teuthology.orchestra.run.smithi114.stdout: "optimal_score": 1, 2024-09-21T12:33:44.115 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_acting": 1.75, 2024-09-21T12:33:44.115 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_stable": 1.75, 2024-09-21T12:33:44.115 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity_weighted": 0.86082649230957031, 2024-09-21T12:33:44.115 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity": 0.86000001430511475, 2024-09-21T12:33:44.115 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity_weighted": 0.86000001430511475 2024-09-21T12:33:44.115 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.115 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.116 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.116 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 4, 2024-09-21T12:33:44.116 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": "default.rgw.log", 2024-09-21T12:33:44.116 INFO:teuthology.orchestra.run.smithi114.stdout: "create_time": "2024-09-21T12:20:15.368765+0000", 2024-09-21T12:33:44.116 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-09-21T12:33:44.116 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_names": "hashpspool", 2024-09-21T12:33:44.116 INFO:teuthology.orchestra.run.smithi114.stdout: "type": 1, 2024-09-21T12:33:44.116 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 3, 2024-09-21T12:33:44.116 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": 2, 2024-09-21T12:33:44.116 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_rule": 0, 2024-09-21T12:33:44.116 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_count": 0, 2024-09-21T12:33:44.116 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_target": 0, 2024-09-21T12:33:44.116 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T12:33:44.116 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T12:33:44.116 INFO:teuthology.orchestra.run.smithi114.stdout: "is_stretch_pool": false, 2024-09-21T12:33:44.117 INFO:teuthology.orchestra.run.smithi114.stdout: "object_hash": 2, 2024-09-21T12:33:44.117 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_mode": "off", 2024-09-21T12:33:44.117 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": 32, 2024-09-21T12:33:44.117 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num": 32, 2024-09-21T12:33:44.117 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num_target": 32, 2024-09-21T12:33:44.117 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_target": 32, 2024-09-21T12:33:44.117 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_pending": 32, 2024-09-21T12:33:44.117 INFO:teuthology.orchestra.run.smithi114.stdout: "last_pg_merge_meta": { 2024-09-21T12:33:44.117 INFO:teuthology.orchestra.run.smithi114.stdout: "source_pgid": "0.0", 2024-09-21T12:33:44.117 INFO:teuthology.orchestra.run.smithi114.stdout: "ready_epoch": 0, 2024-09-21T12:33:44.117 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_started": 0, 2024-09-21T12:33:44.117 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_clean": 0, 2024-09-21T12:33:44.117 INFO:teuthology.orchestra.run.smithi114.stdout: "source_version": "0'0", 2024-09-21T12:33:44.117 INFO:teuthology.orchestra.run.smithi114.stdout: "target_version": "0'0" 2024-09-21T12:33:44.117 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.118 INFO:teuthology.orchestra.run.smithi114.stdout: "last_change": "62", 2024-09-21T12:33:44.118 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend": "0", 2024-09-21T12:33:44.118 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T12:33:44.118 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T12:33:44.118 INFO:teuthology.orchestra.run.smithi114.stdout: "auid": 0, 2024-09-21T12:33:44.118 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_mode": "selfmanaged", 2024-09-21T12:33:44.118 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_seq": 0, 2024-09-21T12:33:44.118 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_epoch": 0, 2024-09-21T12:33:44.118 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_snaps": [], 2024-09-21T12:33:44.118 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps": "[]", 2024-09-21T12:33:44.118 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_bytes": 0, 2024-09-21T12:33:44.118 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_objects": 0, 2024-09-21T12:33:44.118 INFO:teuthology.orchestra.run.smithi114.stdout: "tiers": [], 2024-09-21T12:33:44.118 INFO:teuthology.orchestra.run.smithi114.stdout: "tier_of": -1, 2024-09-21T12:33:44.119 INFO:teuthology.orchestra.run.smithi114.stdout: "read_tier": -1, 2024-09-21T12:33:44.119 INFO:teuthology.orchestra.run.smithi114.stdout: "write_tier": -1, 2024-09-21T12:33:44.119 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_mode": "none", 2024-09-21T12:33:44.119 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_bytes": 0, 2024-09-21T12:33:44.119 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_objects": 0, 2024-09-21T12:33:44.119 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T12:33:44.119 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T12:33:44.119 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T12:33:44.119 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_flush_age": 0, 2024-09-21T12:33:44.119 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_evict_age": 0, 2024-09-21T12:33:44.119 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profile": "", 2024-09-21T12:33:44.119 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_params": { 2024-09-21T12:33:44.119 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "none" 2024-09-21T12:33:44.119 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.120 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_period": 0, 2024-09-21T12:33:44.120 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_count": 0, 2024-09-21T12:33:44.120 INFO:teuthology.orchestra.run.smithi114.stdout: "use_gmt_hitset": true, 2024-09-21T12:33:44.120 INFO:teuthology.orchestra.run.smithi114.stdout: "min_read_recency_for_promote": 0, 2024-09-21T12:33:44.120 INFO:teuthology.orchestra.run.smithi114.stdout: "min_write_recency_for_promote": 0, 2024-09-21T12:33:44.120 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T12:33:44.120 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_search_last_n": 0, 2024-09-21T12:33:44.120 INFO:teuthology.orchestra.run.smithi114.stdout: "grade_table": [], 2024-09-21T12:33:44.120 INFO:teuthology.orchestra.run.smithi114.stdout: "stripe_width": 0, 2024-09-21T12:33:44.120 INFO:teuthology.orchestra.run.smithi114.stdout: "expected_num_objects": 0, 2024-09-21T12:33:44.120 INFO:teuthology.orchestra.run.smithi114.stdout: "fast_read": false, 2024-09-21T12:33:44.120 INFO:teuthology.orchestra.run.smithi114.stdout: "options": {}, 2024-09-21T12:33:44.120 INFO:teuthology.orchestra.run.smithi114.stdout: "application_metadata": { 2024-09-21T12:33:44.120 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw": {} 2024-09-21T12:33:44.121 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.121 INFO:teuthology.orchestra.run.smithi114.stdout: "read_balance": { 2024-09-21T12:33:44.121 INFO:teuthology.orchestra.run.smithi114.stdout: "score_type": "Fair distribution", 2024-09-21T12:33:44.121 INFO:teuthology.orchestra.run.smithi114.stdout: "score_acting": 2.5, 2024-09-21T12:33:44.121 INFO:teuthology.orchestra.run.smithi114.stdout: "score_stable": 2.5, 2024-09-21T12:33:44.121 INFO:teuthology.orchestra.run.smithi114.stdout: "optimal_score": 1, 2024-09-21T12:33:44.121 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_acting": 2.5, 2024-09-21T12:33:44.121 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_stable": 2.5, 2024-09-21T12:33:44.121 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity_weighted": 0.86082649230957031, 2024-09-21T12:33:44.121 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity": 0.86000001430511475, 2024-09-21T12:33:44.121 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity_weighted": 0.86000001430511475 2024-09-21T12:33:44.121 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.121 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.121 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.121 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 5, 2024-09-21T12:33:44.122 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": "default.rgw.control", 2024-09-21T12:33:44.122 INFO:teuthology.orchestra.run.smithi114.stdout: "create_time": "2024-09-21T12:20:17.314275+0000", 2024-09-21T12:33:44.122 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-09-21T12:33:44.122 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_names": "hashpspool", 2024-09-21T12:33:44.122 INFO:teuthology.orchestra.run.smithi114.stdout: "type": 1, 2024-09-21T12:33:44.122 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 3, 2024-09-21T12:33:44.122 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": 2, 2024-09-21T12:33:44.122 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_rule": 0, 2024-09-21T12:33:44.122 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_count": 0, 2024-09-21T12:33:44.122 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_target": 0, 2024-09-21T12:33:44.122 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T12:33:44.122 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T12:33:44.122 INFO:teuthology.orchestra.run.smithi114.stdout: "is_stretch_pool": false, 2024-09-21T12:33:44.122 INFO:teuthology.orchestra.run.smithi114.stdout: "object_hash": 2, 2024-09-21T12:33:44.123 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_mode": "off", 2024-09-21T12:33:44.123 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": 32, 2024-09-21T12:33:44.123 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num": 32, 2024-09-21T12:33:44.123 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num_target": 32, 2024-09-21T12:33:44.123 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_target": 32, 2024-09-21T12:33:44.123 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_pending": 32, 2024-09-21T12:33:44.123 INFO:teuthology.orchestra.run.smithi114.stdout: "last_pg_merge_meta": { 2024-09-21T12:33:44.123 INFO:teuthology.orchestra.run.smithi114.stdout: "source_pgid": "0.0", 2024-09-21T12:33:44.123 INFO:teuthology.orchestra.run.smithi114.stdout: "ready_epoch": 0, 2024-09-21T12:33:44.123 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_started": 0, 2024-09-21T12:33:44.123 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_clean": 0, 2024-09-21T12:33:44.123 INFO:teuthology.orchestra.run.smithi114.stdout: "source_version": "0'0", 2024-09-21T12:33:44.123 INFO:teuthology.orchestra.run.smithi114.stdout: "target_version": "0'0" 2024-09-21T12:33:44.123 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.124 INFO:teuthology.orchestra.run.smithi114.stdout: "last_change": "64", 2024-09-21T12:33:44.124 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend": "0", 2024-09-21T12:33:44.124 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T12:33:44.124 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T12:33:44.124 INFO:teuthology.orchestra.run.smithi114.stdout: "auid": 0, 2024-09-21T12:33:44.124 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_mode": "selfmanaged", 2024-09-21T12:33:44.124 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_seq": 0, 2024-09-21T12:33:44.124 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_epoch": 0, 2024-09-21T12:33:44.124 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_snaps": [], 2024-09-21T12:33:44.124 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps": "[]", 2024-09-21T12:33:44.124 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_bytes": 0, 2024-09-21T12:33:44.124 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_objects": 0, 2024-09-21T12:33:44.124 INFO:teuthology.orchestra.run.smithi114.stdout: "tiers": [], 2024-09-21T12:33:44.124 INFO:teuthology.orchestra.run.smithi114.stdout: "tier_of": -1, 2024-09-21T12:33:44.124 INFO:teuthology.orchestra.run.smithi114.stdout: "read_tier": -1, 2024-09-21T12:33:44.125 INFO:teuthology.orchestra.run.smithi114.stdout: "write_tier": -1, 2024-09-21T12:33:44.125 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_mode": "none", 2024-09-21T12:33:44.125 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_bytes": 0, 2024-09-21T12:33:44.125 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_objects": 0, 2024-09-21T12:33:44.125 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T12:33:44.125 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T12:33:44.125 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T12:33:44.125 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_flush_age": 0, 2024-09-21T12:33:44.125 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_evict_age": 0, 2024-09-21T12:33:44.125 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profile": "", 2024-09-21T12:33:44.125 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_params": { 2024-09-21T12:33:44.125 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "none" 2024-09-21T12:33:44.125 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.125 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_period": 0, 2024-09-21T12:33:44.125 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_count": 0, 2024-09-21T12:33:44.126 INFO:teuthology.orchestra.run.smithi114.stdout: "use_gmt_hitset": true, 2024-09-21T12:33:44.126 INFO:teuthology.orchestra.run.smithi114.stdout: "min_read_recency_for_promote": 0, 2024-09-21T12:33:44.126 INFO:teuthology.orchestra.run.smithi114.stdout: "min_write_recency_for_promote": 0, 2024-09-21T12:33:44.126 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T12:33:44.126 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_search_last_n": 0, 2024-09-21T12:33:44.126 INFO:teuthology.orchestra.run.smithi114.stdout: "grade_table": [], 2024-09-21T12:33:44.126 INFO:teuthology.orchestra.run.smithi114.stdout: "stripe_width": 0, 2024-09-21T12:33:44.126 INFO:teuthology.orchestra.run.smithi114.stdout: "expected_num_objects": 0, 2024-09-21T12:33:44.126 INFO:teuthology.orchestra.run.smithi114.stdout: "fast_read": false, 2024-09-21T12:33:44.126 INFO:teuthology.orchestra.run.smithi114.stdout: "options": {}, 2024-09-21T12:33:44.126 INFO:teuthology.orchestra.run.smithi114.stdout: "application_metadata": { 2024-09-21T12:33:44.126 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw": {} 2024-09-21T12:33:44.126 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.126 INFO:teuthology.orchestra.run.smithi114.stdout: "read_balance": { 2024-09-21T12:33:44.126 INFO:teuthology.orchestra.run.smithi114.stdout: "score_type": "Fair distribution", 2024-09-21T12:33:44.127 INFO:teuthology.orchestra.run.smithi114.stdout: "score_acting": 1.5, 2024-09-21T12:33:44.127 INFO:teuthology.orchestra.run.smithi114.stdout: "score_stable": 1.5, 2024-09-21T12:33:44.127 INFO:teuthology.orchestra.run.smithi114.stdout: "optimal_score": 1, 2024-09-21T12:33:44.127 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_acting": 1.5, 2024-09-21T12:33:44.127 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_stable": 1.5, 2024-09-21T12:33:44.127 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity_weighted": 0.86082649230957031, 2024-09-21T12:33:44.127 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity": 0.86000001430511475, 2024-09-21T12:33:44.127 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity_weighted": 0.86000001430511475 2024-09-21T12:33:44.128 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.128 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.128 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.128 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 6, 2024-09-21T12:33:44.128 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": "default.rgw.meta", 2024-09-21T12:33:44.128 INFO:teuthology.orchestra.run.smithi114.stdout: "create_time": "2024-09-21T12:20:19.370218+0000", 2024-09-21T12:33:44.128 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-09-21T12:33:44.128 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_names": "hashpspool", 2024-09-21T12:33:44.128 INFO:teuthology.orchestra.run.smithi114.stdout: "type": 1, 2024-09-21T12:33:44.128 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 3, 2024-09-21T12:33:44.128 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": 2, 2024-09-21T12:33:44.128 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_rule": 0, 2024-09-21T12:33:44.128 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_count": 0, 2024-09-21T12:33:44.128 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_target": 0, 2024-09-21T12:33:44.129 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T12:33:44.129 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T12:33:44.129 INFO:teuthology.orchestra.run.smithi114.stdout: "is_stretch_pool": false, 2024-09-21T12:33:44.129 INFO:teuthology.orchestra.run.smithi114.stdout: "object_hash": 2, 2024-09-21T12:33:44.129 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_mode": "off", 2024-09-21T12:33:44.129 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": 32, 2024-09-21T12:33:44.129 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num": 32, 2024-09-21T12:33:44.129 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num_target": 32, 2024-09-21T12:33:44.129 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_target": 32, 2024-09-21T12:33:44.129 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_pending": 32, 2024-09-21T12:33:44.129 INFO:teuthology.orchestra.run.smithi114.stdout: "last_pg_merge_meta": { 2024-09-21T12:33:44.129 INFO:teuthology.orchestra.run.smithi114.stdout: "source_pgid": "0.0", 2024-09-21T12:33:44.129 INFO:teuthology.orchestra.run.smithi114.stdout: "ready_epoch": 0, 2024-09-21T12:33:44.129 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_started": 0, 2024-09-21T12:33:44.130 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_clean": 0, 2024-09-21T12:33:44.130 INFO:teuthology.orchestra.run.smithi114.stdout: "source_version": "0'0", 2024-09-21T12:33:44.130 INFO:teuthology.orchestra.run.smithi114.stdout: "target_version": "0'0" 2024-09-21T12:33:44.130 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.130 INFO:teuthology.orchestra.run.smithi114.stdout: "last_change": "66", 2024-09-21T12:33:44.130 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend": "0", 2024-09-21T12:33:44.130 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T12:33:44.130 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T12:33:44.130 INFO:teuthology.orchestra.run.smithi114.stdout: "auid": 0, 2024-09-21T12:33:44.130 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_mode": "selfmanaged", 2024-09-21T12:33:44.130 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_seq": 0, 2024-09-21T12:33:44.130 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_epoch": 0, 2024-09-21T12:33:44.130 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_snaps": [], 2024-09-21T12:33:44.130 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps": "[]", 2024-09-21T12:33:44.131 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_bytes": 0, 2024-09-21T12:33:44.131 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_objects": 0, 2024-09-21T12:33:44.131 INFO:teuthology.orchestra.run.smithi114.stdout: "tiers": [], 2024-09-21T12:33:44.131 INFO:teuthology.orchestra.run.smithi114.stdout: "tier_of": -1, 2024-09-21T12:33:44.131 INFO:teuthology.orchestra.run.smithi114.stdout: "read_tier": -1, 2024-09-21T12:33:44.131 INFO:teuthology.orchestra.run.smithi114.stdout: "write_tier": -1, 2024-09-21T12:33:44.131 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_mode": "none", 2024-09-21T12:33:44.131 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_bytes": 0, 2024-09-21T12:33:44.131 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_objects": 0, 2024-09-21T12:33:44.131 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T12:33:44.131 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T12:33:44.131 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T12:33:44.131 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_flush_age": 0, 2024-09-21T12:33:44.132 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_evict_age": 0, 2024-09-21T12:33:44.132 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profile": "", 2024-09-21T12:33:44.132 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_params": { 2024-09-21T12:33:44.132 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "none" 2024-09-21T12:33:44.132 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.132 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_period": 0, 2024-09-21T12:33:44.132 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_count": 0, 2024-09-21T12:33:44.132 INFO:teuthology.orchestra.run.smithi114.stdout: "use_gmt_hitset": true, 2024-09-21T12:33:44.132 INFO:teuthology.orchestra.run.smithi114.stdout: "min_read_recency_for_promote": 0, 2024-09-21T12:33:44.132 INFO:teuthology.orchestra.run.smithi114.stdout: "min_write_recency_for_promote": 0, 2024-09-21T12:33:44.132 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T12:33:44.132 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_search_last_n": 0, 2024-09-21T12:33:44.132 INFO:teuthology.orchestra.run.smithi114.stdout: "grade_table": [], 2024-09-21T12:33:44.132 INFO:teuthology.orchestra.run.smithi114.stdout: "stripe_width": 0, 2024-09-21T12:33:44.133 INFO:teuthology.orchestra.run.smithi114.stdout: "expected_num_objects": 0, 2024-09-21T12:33:44.133 INFO:teuthology.orchestra.run.smithi114.stdout: "fast_read": false, 2024-09-21T12:33:44.133 INFO:teuthology.orchestra.run.smithi114.stdout: "options": { 2024-09-21T12:33:44.133 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_bias": 4 2024-09-21T12:33:44.133 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.133 INFO:teuthology.orchestra.run.smithi114.stdout: "application_metadata": { 2024-09-21T12:33:44.133 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw": {} 2024-09-21T12:33:44.133 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.133 INFO:teuthology.orchestra.run.smithi114.stdout: "read_balance": { 2024-09-21T12:33:44.133 INFO:teuthology.orchestra.run.smithi114.stdout: "score_type": "Fair distribution", 2024-09-21T12:33:44.133 INFO:teuthology.orchestra.run.smithi114.stdout: "score_acting": 2, 2024-09-21T12:33:44.133 INFO:teuthology.orchestra.run.smithi114.stdout: "score_stable": 2, 2024-09-21T12:33:44.133 INFO:teuthology.orchestra.run.smithi114.stdout: "optimal_score": 1, 2024-09-21T12:33:44.133 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_acting": 2, 2024-09-21T12:33:44.133 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_stable": 2, 2024-09-21T12:33:44.134 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity_weighted": 0.86082649230957031, 2024-09-21T12:33:44.134 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity": 0.86000001430511475, 2024-09-21T12:33:44.134 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity_weighted": 0.86000001430511475 2024-09-21T12:33:44.134 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.134 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.134 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.134 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 10, 2024-09-21T12:33:44.134 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": "unique_pool_3", 2024-09-21T12:33:44.134 INFO:teuthology.orchestra.run.smithi114.stdout: "create_time": "2024-09-21T12:32:28.476438+0000", 2024-09-21T12:33:44.134 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-09-21T12:33:44.134 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_names": "hashpspool", 2024-09-21T12:33:44.134 INFO:teuthology.orchestra.run.smithi114.stdout: "type": 1, 2024-09-21T12:33:44.134 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 3, 2024-09-21T12:33:44.134 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": 2, 2024-09-21T12:33:44.135 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_rule": 0, 2024-09-21T12:33:44.135 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_count": 0, 2024-09-21T12:33:44.135 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_target": 0, 2024-09-21T12:33:44.135 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T12:33:44.135 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T12:33:44.135 INFO:teuthology.orchestra.run.smithi114.stdout: "is_stretch_pool": false, 2024-09-21T12:33:44.135 INFO:teuthology.orchestra.run.smithi114.stdout: "object_hash": 2, 2024-09-21T12:33:44.135 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_mode": "off", 2024-09-21T12:33:44.135 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": 16, 2024-09-21T12:33:44.135 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num": 16, 2024-09-21T12:33:44.135 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num_target": 16, 2024-09-21T12:33:44.135 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_target": 16, 2024-09-21T12:33:44.135 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_pending": 16, 2024-09-21T12:33:44.135 INFO:teuthology.orchestra.run.smithi114.stdout: "last_pg_merge_meta": { 2024-09-21T12:33:44.135 INFO:teuthology.orchestra.run.smithi114.stdout: "source_pgid": "0.0", 2024-09-21T12:33:44.136 INFO:teuthology.orchestra.run.smithi114.stdout: "ready_epoch": 0, 2024-09-21T12:33:44.136 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_started": 0, 2024-09-21T12:33:44.136 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_clean": 0, 2024-09-21T12:33:44.136 INFO:teuthology.orchestra.run.smithi114.stdout: "source_version": "0'0", 2024-09-21T12:33:44.136 INFO:teuthology.orchestra.run.smithi114.stdout: "target_version": "0'0" 2024-09-21T12:33:44.136 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.136 INFO:teuthology.orchestra.run.smithi114.stdout: "last_change": "348", 2024-09-21T12:33:44.136 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend": "0", 2024-09-21T12:33:44.136 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T12:33:44.136 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T12:33:44.136 INFO:teuthology.orchestra.run.smithi114.stdout: "auid": 0, 2024-09-21T12:33:44.136 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_mode": "selfmanaged", 2024-09-21T12:33:44.136 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_seq": 0, 2024-09-21T12:33:44.136 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_epoch": 0, 2024-09-21T12:33:44.137 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_snaps": [], 2024-09-21T12:33:44.137 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps": "[]", 2024-09-21T12:33:44.137 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_bytes": 0, 2024-09-21T12:33:44.137 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_objects": 0, 2024-09-21T12:33:44.137 INFO:teuthology.orchestra.run.smithi114.stdout: "tiers": [], 2024-09-21T12:33:44.137 INFO:teuthology.orchestra.run.smithi114.stdout: "tier_of": -1, 2024-09-21T12:33:44.137 INFO:teuthology.orchestra.run.smithi114.stdout: "read_tier": -1, 2024-09-21T12:33:44.137 INFO:teuthology.orchestra.run.smithi114.stdout: "write_tier": -1, 2024-09-21T12:33:44.137 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_mode": "none", 2024-09-21T12:33:44.137 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_bytes": 0, 2024-09-21T12:33:44.137 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_objects": 0, 2024-09-21T12:33:44.137 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T12:33:44.137 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T12:33:44.137 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T12:33:44.138 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_flush_age": 0, 2024-09-21T12:33:44.138 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_evict_age": 0, 2024-09-21T12:33:44.138 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profile": "", 2024-09-21T12:33:44.138 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_params": { 2024-09-21T12:33:44.138 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "none" 2024-09-21T12:33:44.138 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.138 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_period": 0, 2024-09-21T12:33:44.138 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_count": 0, 2024-09-21T12:33:44.138 INFO:teuthology.orchestra.run.smithi114.stdout: "use_gmt_hitset": true, 2024-09-21T12:33:44.138 INFO:teuthology.orchestra.run.smithi114.stdout: "min_read_recency_for_promote": 0, 2024-09-21T12:33:44.138 INFO:teuthology.orchestra.run.smithi114.stdout: "min_write_recency_for_promote": 0, 2024-09-21T12:33:44.138 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T12:33:44.138 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_search_last_n": 0, 2024-09-21T12:33:44.138 INFO:teuthology.orchestra.run.smithi114.stdout: "grade_table": [], 2024-09-21T12:33:44.139 INFO:teuthology.orchestra.run.smithi114.stdout: "stripe_width": 0, 2024-09-21T12:33:44.139 INFO:teuthology.orchestra.run.smithi114.stdout: "expected_num_objects": 0, 2024-09-21T12:33:44.139 INFO:teuthology.orchestra.run.smithi114.stdout: "fast_read": false, 2024-09-21T12:33:44.139 INFO:teuthology.orchestra.run.smithi114.stdout: "options": {}, 2024-09-21T12:33:44.139 INFO:teuthology.orchestra.run.smithi114.stdout: "application_metadata": { 2024-09-21T12:33:44.139 INFO:teuthology.orchestra.run.smithi114.stdout: "rados": {} 2024-09-21T12:33:44.139 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.139 INFO:teuthology.orchestra.run.smithi114.stdout: "read_balance": { 2024-09-21T12:33:44.139 INFO:teuthology.orchestra.run.smithi114.stdout: "score_type": "Fair distribution", 2024-09-21T12:33:44.139 INFO:teuthology.orchestra.run.smithi114.stdout: "score_acting": 1.5, 2024-09-21T12:33:44.139 INFO:teuthology.orchestra.run.smithi114.stdout: "score_stable": 1.5, 2024-09-21T12:33:44.139 INFO:teuthology.orchestra.run.smithi114.stdout: "optimal_score": 1, 2024-09-21T12:33:44.139 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_acting": 1.5, 2024-09-21T12:33:44.139 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_stable": 1.5, 2024-09-21T12:33:44.140 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity_weighted": 0.86082649230957031, 2024-09-21T12:33:44.140 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity": 0.86000001430511475, 2024-09-21T12:33:44.140 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity_weighted": 0.86000001430511475 2024-09-21T12:33:44.140 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.140 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.140 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:33:44.140 INFO:teuthology.orchestra.run.smithi114.stdout: "osds": [ 2024-09-21T12:33:44.140 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.140 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 0, 2024-09-21T12:33:44.140 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "5715e521-c7dc-41dd-a075-5c96a88fa9b3", 2024-09-21T12:33:44.140 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:33:44.140 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 1, 2024-09-21T12:33:44.140 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 1, 2024-09-21T12:33:44.140 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 1, 2024-09-21T12:33:44.140 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:33:44.141 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:33:44.141 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 8, 2024-09-21T12:33:44.141 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 353, 2024-09-21T12:33:44.141 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:33:44.141 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:33:44.141 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:33:44.141 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.141 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.141 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.141 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6802", 2024-09-21T12:33:44.141 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:33:44.141 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.141 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.141 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.142 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6803", 2024-09-21T12:33:44.142 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:33:44.142 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.142 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.142 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.142 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:33:44.142 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.142 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.142 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.142 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6804", 2024-09-21T12:33:44.142 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:33:44.142 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.142 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.142 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.143 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6805", 2024-09-21T12:33:44.143 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:33:44.143 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.143 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.143 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.143 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:33:44.143 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.143 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.143 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.143 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6808", 2024-09-21T12:33:44.143 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:33:44.143 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.143 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.143 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:44.145 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:44.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:44.145 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6809", 2024-09-21T12:33:44.145 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:33:44.145 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.145 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.146 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.146 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:33:44.146 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.146 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.146 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.146 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6806", 2024-09-21T12:33:44.146 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:33:44.146 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.146 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.146 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.146 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6807", 2024-09-21T12:33:44.146 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:33:44.146 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.147 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.147 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.147 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6803/3555869816", 2024-09-21T12:33:44.147 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.114:6805/3555869816", 2024-09-21T12:33:44.147 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.114:6809/3555869816", 2024-09-21T12:33:44.147 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.114:6807/3555869816", 2024-09-21T12:33:44.147 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:33:44.147 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:33:44.147 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:33:44.147 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.147 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.147 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.147 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 1, 2024-09-21T12:33:44.147 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "1fe160a7-5972-4e3e-99ce-15e2311ba4c5", 2024-09-21T12:33:44.148 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:33:44.148 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 1, 2024-09-21T12:33:44.148 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 1, 2024-09-21T12:33:44.148 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 1, 2024-09-21T12:33:44.148 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:33:44.148 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:33:44.148 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 13, 2024-09-21T12:33:44.148 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 353, 2024-09-21T12:33:44.148 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:33:44.148 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:33:44.148 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:33:44.148 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.148 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.148 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.149 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6810", 2024-09-21T12:33:44.149 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:33:44.149 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.149 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.149 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.149 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6811", 2024-09-21T12:33:44.149 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:33:44.149 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.149 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.149 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.149 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:33:44.149 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.149 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.149 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.150 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6812", 2024-09-21T12:33:44.150 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:33:44.150 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.150 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.150 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.150 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6813", 2024-09-21T12:33:44.150 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:33:44.150 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.150 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.150 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.150 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:33:44.150 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.150 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.150 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.151 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6816", 2024-09-21T12:33:44.151 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:33:44.151 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.151 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.151 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.151 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6817", 2024-09-21T12:33:44.151 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:33:44.151 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.151 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.151 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.151 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:33:44.151 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.151 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.151 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.152 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6814", 2024-09-21T12:33:44.152 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:33:44.152 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.152 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.152 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.152 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6815", 2024-09-21T12:33:44.152 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:33:44.152 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.152 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.152 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.152 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6811/2744663863", 2024-09-21T12:33:44.152 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.114:6813/2744663863", 2024-09-21T12:33:44.152 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.114:6817/2744663863", 2024-09-21T12:33:44.152 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.114:6815/2744663863", 2024-09-21T12:33:44.153 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:33:44.153 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:33:44.153 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:33:44.153 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.153 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.153 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.153 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 2, 2024-09-21T12:33:44.153 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "6a63581f-7603-49ae-9ab3-b37e7c91ab91", 2024-09-21T12:33:44.153 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:33:44.153 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 1, 2024-09-21T12:33:44.153 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 1, 2024-09-21T12:33:44.153 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 1, 2024-09-21T12:33:44.153 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:33:44.153 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:33:44.153 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 18, 2024-09-21T12:33:44.153 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 351, 2024-09-21T12:33:44.154 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:33:44.154 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:33:44.154 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:33:44.154 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.154 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.154 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.154 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6818", 2024-09-21T12:33:44.154 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:33:44.154 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.154 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.154 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.154 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6819", 2024-09-21T12:33:44.154 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:33:44.155 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.155 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.155 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.155 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:33:44.155 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.155 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.155 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.155 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6820", 2024-09-21T12:33:44.155 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:33:44.155 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.155 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.155 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.155 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6821", 2024-09-21T12:33:44.155 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:33:44.155 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.156 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.156 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.156 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:33:44.156 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.156 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.156 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.156 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6824", 2024-09-21T12:33:44.156 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:33:44.156 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.156 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.156 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.156 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6825", 2024-09-21T12:33:44.156 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:33:44.156 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.157 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.157 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.157 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:33:44.157 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.157 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.157 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.157 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6822", 2024-09-21T12:33:44.157 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:33:44.157 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.157 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.157 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.157 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6823", 2024-09-21T12:33:44.157 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:33:44.157 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.157 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.158 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.158 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6819/3740045173", 2024-09-21T12:33:44.158 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.114:6821/3740045173", 2024-09-21T12:33:44.158 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.114:6825/3740045173", 2024-09-21T12:33:44.158 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.114:6823/3740045173", 2024-09-21T12:33:44.158 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:33:44.158 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:33:44.158 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:33:44.158 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.158 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.158 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.158 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 3, 2024-09-21T12:33:44.158 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "0ad68cdb-ae5c-4090-8a8d-b67c1177ae42", 2024-09-21T12:33:44.159 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:33:44.159 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 1, 2024-09-21T12:33:44.159 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 1, 2024-09-21T12:33:44.159 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 1, 2024-09-21T12:33:44.159 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:33:44.159 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:33:44.159 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 26, 2024-09-21T12:33:44.159 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 349, 2024-09-21T12:33:44.159 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:33:44.159 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:33:44.159 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:33:44.159 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.159 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.159 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.159 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6826", 2024-09-21T12:33:44.160 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:33:44.160 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.160 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.160 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.160 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6827", 2024-09-21T12:33:44.160 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:33:44.160 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.160 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.160 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.160 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:33:44.160 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.160 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.160 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.160 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6828", 2024-09-21T12:33:44.161 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:33:44.161 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.161 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.161 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.161 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6829", 2024-09-21T12:33:44.161 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:33:44.161 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.161 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.161 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.161 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:33:44.161 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.161 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.161 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.161 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6832", 2024-09-21T12:33:44.161 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:33:44.162 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.162 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.162 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.162 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6833", 2024-09-21T12:33:44.162 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:33:44.162 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.162 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.162 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.162 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:33:44.162 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.162 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.162 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.162 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6830", 2024-09-21T12:33:44.162 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:33:44.163 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.163 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.163 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.163 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6831", 2024-09-21T12:33:44.163 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:33:44.163 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.163 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.163 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.163 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6827/1312590318", 2024-09-21T12:33:44.163 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.114:6829/1312590318", 2024-09-21T12:33:44.163 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.114:6833/1312590318", 2024-09-21T12:33:44.163 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.114:6831/1312590318", 2024-09-21T12:33:44.163 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:33:44.164 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:33:44.164 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:33:44.164 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.164 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.164 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.164 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 4, 2024-09-21T12:33:44.164 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "26bb8a84-e41d-4425-8491-2dea9764b7cf", 2024-09-21T12:33:44.164 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:33:44.164 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 1, 2024-09-21T12:33:44.164 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 1, 2024-09-21T12:33:44.164 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 1, 2024-09-21T12:33:44.164 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:33:44.164 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:33:44.164 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 32, 2024-09-21T12:33:44.164 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 353, 2024-09-21T12:33:44.165 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:33:44.165 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:33:44.165 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:33:44.165 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.165 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.165 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.165 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6800", 2024-09-21T12:33:44.165 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:33:44.165 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.165 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.165 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.165 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6801", 2024-09-21T12:33:44.165 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:33:44.165 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.165 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.166 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.166 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:33:44.166 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.166 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.166 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.166 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6802", 2024-09-21T12:33:44.166 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:33:44.166 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.166 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.166 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.166 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6803", 2024-09-21T12:33:44.166 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:33:44.166 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.166 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.167 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.167 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:33:44.167 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.167 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.167 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.167 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6806", 2024-09-21T12:33:44.167 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:33:44.167 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.167 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.167 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.167 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6807", 2024-09-21T12:33:44.167 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:33:44.167 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.168 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.168 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.168 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:33:44.168 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.168 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.168 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.168 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6804", 2024-09-21T12:33:44.168 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:33:44.168 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.168 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.168 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.168 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6805", 2024-09-21T12:33:44.168 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:33:44.168 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.168 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.169 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.169 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.183:6801/1391644836", 2024-09-21T12:33:44.169 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.183:6803/1391644836", 2024-09-21T12:33:44.169 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.183:6807/1391644836", 2024-09-21T12:33:44.169 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.183:6805/1391644836", 2024-09-21T12:33:44.169 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:33:44.169 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:33:44.169 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:33:44.169 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.169 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.169 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.169 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 5, 2024-09-21T12:33:44.170 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "33f96871-0869-4281-82b9-bd518987c551", 2024-09-21T12:33:44.170 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:33:44.170 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 1, 2024-09-21T12:33:44.170 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 1, 2024-09-21T12:33:44.170 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 0.476898193359375, 2024-09-21T12:33:44.170 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:33:44.170 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:33:44.170 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 39, 2024-09-21T12:33:44.170 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 349, 2024-09-21T12:33:44.170 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:33:44.170 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:33:44.170 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:33:44.170 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.170 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.171 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.171 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6808", 2024-09-21T12:33:44.171 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:33:44.171 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.171 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.171 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.171 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6809", 2024-09-21T12:33:44.171 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:33:44.171 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.171 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.171 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.172 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:33:44.172 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.172 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.172 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.172 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6810", 2024-09-21T12:33:44.172 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:33:44.172 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.172 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.172 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.172 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6811", 2024-09-21T12:33:44.172 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:33:44.172 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.172 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.172 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.173 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:33:44.173 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.173 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.173 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.173 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6814", 2024-09-21T12:33:44.173 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:33:44.173 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.173 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.173 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.173 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6815", 2024-09-21T12:33:44.173 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:33:44.173 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.173 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.173 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.174 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:33:44.174 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.174 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.174 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.174 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6812", 2024-09-21T12:33:44.174 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:33:44.174 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.174 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.174 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.174 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6813", 2024-09-21T12:33:44.174 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:33:44.174 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.174 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.174 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.175 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.183:6809/3187643473", 2024-09-21T12:33:44.175 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.183:6811/3187643473", 2024-09-21T12:33:44.175 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.183:6815/3187643473", 2024-09-21T12:33:44.175 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.183:6813/3187643473", 2024-09-21T12:33:44.175 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:33:44.175 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:33:44.175 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:33:44.175 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.175 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.175 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.175 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 6, 2024-09-21T12:33:44.175 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "e09c72b8-ad5a-4779-a874-b149fce6ed70", 2024-09-21T12:33:44.175 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:33:44.175 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 1, 2024-09-21T12:33:44.176 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 1, 2024-09-21T12:33:44.176 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 0.4097137451171875, 2024-09-21T12:33:44.176 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:33:44.176 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:33:44.176 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 47, 2024-09-21T12:33:44.176 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 352, 2024-09-21T12:33:44.176 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:33:44.176 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:33:44.176 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:33:44.176 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.176 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.176 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.176 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6816", 2024-09-21T12:33:44.176 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:33:44.177 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.177 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.177 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.177 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6817", 2024-09-21T12:33:44.177 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:33:44.177 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.177 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.177 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.177 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:33:44.177 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.177 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.177 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.177 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6818", 2024-09-21T12:33:44.177 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:33:44.177 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.178 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.178 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.178 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6819", 2024-09-21T12:33:44.178 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:33:44.178 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.178 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.178 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.178 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:33:44.178 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.178 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.178 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.178 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6822", 2024-09-21T12:33:44.178 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:33:44.178 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.179 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.179 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.179 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6823", 2024-09-21T12:33:44.179 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:33:44.179 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.179 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.179 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.179 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:33:44.179 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.179 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.179 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.179 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6820", 2024-09-21T12:33:44.179 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:33:44.179 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.180 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.180 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.180 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6821", 2024-09-21T12:33:44.180 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:33:44.180 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.180 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.180 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.180 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.183:6817/3025199631", 2024-09-21T12:33:44.180 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.183:6819/3025199631", 2024-09-21T12:33:44.180 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.183:6823/3025199631", 2024-09-21T12:33:44.180 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.183:6821/3025199631", 2024-09-21T12:33:44.180 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:33:44.180 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:33:44.180 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:33:44.180 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.181 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.181 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.181 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 7, 2024-09-21T12:33:44.181 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "1a24194d-1af6-485f-8856-98844a02dd9d", 2024-09-21T12:33:44.181 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:33:44.181 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 1, 2024-09-21T12:33:44.181 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 1, 2024-09-21T12:33:44.182 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 1, 2024-09-21T12:33:44.182 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:33:44.182 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:33:44.182 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 54, 2024-09-21T12:33:44.182 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 352, 2024-09-21T12:33:44.182 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:33:44.182 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:33:44.182 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:33:44.182 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.182 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.182 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.182 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6824", 2024-09-21T12:33:44.182 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:33:44.183 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.183 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.183 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.183 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6825", 2024-09-21T12:33:44.183 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:33:44.183 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.183 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.183 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.183 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:33:44.183 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.183 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.183 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.183 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6826", 2024-09-21T12:33:44.183 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:33:44.184 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.184 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.184 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.184 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6827", 2024-09-21T12:33:44.184 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:33:44.184 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.184 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.184 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.184 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:33:44.184 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.184 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.184 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.184 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6830", 2024-09-21T12:33:44.185 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:33:44.185 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.185 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.185 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.185 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6831", 2024-09-21T12:33:44.185 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:33:44.185 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.185 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.185 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.185 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:33:44.185 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:33:44.185 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.185 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:33:44.185 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6828", 2024-09-21T12:33:44.185 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:33:44.186 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.186 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.186 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:33:44.186 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6829", 2024-09-21T12:33:44.186 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:33:44.186 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.186 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.186 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.186 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.183:6825/3203337442", 2024-09-21T12:33:44.186 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.183:6827/3203337442", 2024-09-21T12:33:44.186 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.183:6831/3203337442", 2024-09-21T12:33:44.186 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.183:6829/3203337442", 2024-09-21T12:33:44.186 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:33:44.187 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:33:44.187 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:33:44.187 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.187 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.187 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:33:44.187 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_xinfo": [ 2024-09-21T12:33:44.187 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.187 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 0, 2024-09-21T12:33:44.187 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:33:44.187 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:33:44.187 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:33:44.187 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:33:44.187 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 0, 2024-09-21T12:33:44.187 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:33:04.792722+0000", 2024-09-21T12:33:44.188 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:33:44.188 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.188 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.188 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 1, 2024-09-21T12:33:44.188 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:33:44.188 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:33:44.188 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:33:44.188 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:33:44.188 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 0, 2024-09-21T12:33:44.188 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:32:54.736840+0000", 2024-09-21T12:33:44.188 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:33:44.188 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.188 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.189 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 2, 2024-09-21T12:33:44.189 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:33:44.189 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:33:44.189 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:33:44.189 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:33:44.189 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 0, 2024-09-21T12:33:44.189 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:33:33.127156+0000", 2024-09-21T12:33:44.189 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:33:44.189 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.189 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.189 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 3, 2024-09-21T12:33:44.189 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:33:44.189 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:33:44.189 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:33:44.190 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:33:44.190 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 0, 2024-09-21T12:33:44.190 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:33:17.347963+0000", 2024-09-21T12:33:44.190 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:33:44.190 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.190 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.190 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 4, 2024-09-21T12:33:44.190 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:33:44.190 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:33:44.190 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:33:44.190 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:33:44.190 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 0, 2024-09-21T12:33:44.190 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:33:31.062655+0000", 2024-09-21T12:33:44.190 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:33:44.191 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.191 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.191 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 5, 2024-09-21T12:33:44.191 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:33:44.191 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:33:44.191 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:33:44.191 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:33:44.191 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 0, 2024-09-21T12:33:44.191 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:33:20.194600+0000", 2024-09-21T12:33:44.191 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:33:44.191 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.191 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.191 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 6, 2024-09-21T12:33:44.192 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:33:44.192 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:33:44.192 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:33:44.192 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:33:44.192 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 0, 2024-09-21T12:33:44.192 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:33:36.228827+0000", 2024-09-21T12:33:44.192 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:33:44.192 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.192 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.192 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 7, 2024-09-21T12:33:44.192 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:33:44.192 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:33:44.192 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:33:44.192 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:33:44.193 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 0, 2024-09-21T12:33:44.193 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:33:29.492142+0000", 2024-09-21T12:33:44.193 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:33:44.193 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.193 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:33:44.193 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_upmap": [], 2024-09-21T12:33:44.193 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_upmap_items": [ 2024-09-21T12:33:44.193 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.193 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "4.c", 2024-09-21T12:33:44.193 INFO:teuthology.orchestra.run.smithi114.stdout: "mappings": [ 2024-09-21T12:33:44.193 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.193 INFO:teuthology.orchestra.run.smithi114.stdout: "from": 3, 2024-09-21T12:33:44.193 INFO:teuthology.orchestra.run.smithi114.stdout: "to": 2 2024-09-21T12:33:44.193 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.194 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.194 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.194 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.194 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "4.d", 2024-09-21T12:33:44.194 INFO:teuthology.orchestra.run.smithi114.stdout: "mappings": [ 2024-09-21T12:33:44.194 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.194 INFO:teuthology.orchestra.run.smithi114.stdout: "from": 1, 2024-09-21T12:33:44.194 INFO:teuthology.orchestra.run.smithi114.stdout: "to": 5 2024-09-21T12:33:44.194 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.194 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.194 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.194 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.194 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "4.f", 2024-09-21T12:33:44.194 INFO:teuthology.orchestra.run.smithi114.stdout: "mappings": [ 2024-09-21T12:33:44.195 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.195 INFO:teuthology.orchestra.run.smithi114.stdout: "from": 1, 2024-09-21T12:33:44.195 INFO:teuthology.orchestra.run.smithi114.stdout: "to": 2 2024-09-21T12:33:44.195 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.195 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.195 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.195 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.195 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "4.15", 2024-09-21T12:33:44.195 INFO:teuthology.orchestra.run.smithi114.stdout: "mappings": [ 2024-09-21T12:33:44.195 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.195 INFO:teuthology.orchestra.run.smithi114.stdout: "from": 3, 2024-09-21T12:33:44.195 INFO:teuthology.orchestra.run.smithi114.stdout: "to": 2 2024-09-21T12:33:44.195 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.195 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.195 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.196 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.196 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "5.1d", 2024-09-21T12:33:44.196 INFO:teuthology.orchestra.run.smithi114.stdout: "mappings": [ 2024-09-21T12:33:44.196 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:44.196 INFO:teuthology.orchestra.run.smithi114.stdout: "from": 0, 2024-09-21T12:33:44.196 INFO:teuthology.orchestra.run.smithi114.stdout: "to": 3 2024-09-21T12:33:44.196 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.196 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:44.196 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.196 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:33:44.196 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_upmap_primaries": [], 2024-09-21T12:33:44.196 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_temp": [], 2024-09-21T12:33:44.196 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_temp": [], 2024-09-21T12:33:44.196 INFO:teuthology.orchestra.run.smithi114.stdout: "blocklist": { 2024-09-21T12:33:44.197 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:6801/237768393": "2024-09-22T12:20:45.528856+0000", 2024-09-21T12:33:44.197 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:6800/237768393": "2024-09-22T12:20:45.528856+0000", 2024-09-21T12:33:44.197 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/898990943": "2024-09-22T12:14:53.998653+0000", 2024-09-21T12:33:44.197 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/3866305956": "2024-09-22T12:20:45.528856+0000", 2024-09-21T12:33:44.197 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/3598782088": "2024-09-22T12:20:45.528856+0000", 2024-09-21T12:33:44.197 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/2620151541": "2024-09-22T12:15:20.045217+0000", 2024-09-21T12:33:44.197 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:6800/1725148595": "2024-09-22T12:14:53.998653+0000", 2024-09-21T12:33:44.197 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/442404805": "2024-09-22T12:14:53.998653+0000", 2024-09-21T12:33:44.197 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:6801/1725148595": "2024-09-22T12:14:53.998653+0000", 2024-09-21T12:33:44.197 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/4173909019": "2024-09-22T12:20:45.528856+0000", 2024-09-21T12:33:44.197 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/2106467614": "2024-09-22T12:15:20.045217+0000", 2024-09-21T12:33:44.197 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/139803357": "2024-09-22T12:15:20.045217+0000", 2024-09-21T12:33:44.197 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:6801/770932721": "2024-09-22T12:15:20.045217+0000", 2024-09-21T12:33:44.197 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:6800/770932721": "2024-09-22T12:15:20.045217+0000", 2024-09-21T12:33:44.197 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/1457547294": "2024-09-22T12:20:45.528856+0000", 2024-09-21T12:33:44.198 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/3416259228": "2024-09-22T12:14:53.998653+0000", 2024-09-21T12:33:44.198 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/995404371": "2024-09-22T12:20:45.528856+0000" 2024-09-21T12:33:44.198 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.198 INFO:teuthology.orchestra.run.smithi114.stdout: "range_blocklist": {}, 2024-09-21T12:33:44.198 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profiles": { 2024-09-21T12:33:44.198 INFO:teuthology.orchestra.run.smithi114.stdout: "default": { 2024-09-21T12:33:44.198 INFO:teuthology.orchestra.run.smithi114.stdout: "crush-failure-domain": "osd", 2024-09-21T12:33:44.198 INFO:teuthology.orchestra.run.smithi114.stdout: "k": "2", 2024-09-21T12:33:44.198 INFO:teuthology.orchestra.run.smithi114.stdout: "m": "1", 2024-09-21T12:33:44.198 INFO:teuthology.orchestra.run.smithi114.stdout: "plugin": "jerasure", 2024-09-21T12:33:44.198 INFO:teuthology.orchestra.run.smithi114.stdout: "technique": "reed_sol_van" 2024-09-21T12:33:44.198 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.198 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:44.199 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps_queue": [], 2024-09-21T12:33:44.199 INFO:teuthology.orchestra.run.smithi114.stdout: "new_removed_snaps": [], 2024-09-21T12:33:44.199 INFO:teuthology.orchestra.run.smithi114.stdout: "new_purged_snaps": [], 2024-09-21T12:33:44.199 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_node_flags": {}, 2024-09-21T12:33:44.199 INFO:teuthology.orchestra.run.smithi114.stdout: "device_class_flags": {}, 2024-09-21T12:33:44.199 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": { 2024-09-21T12:33:44.199 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode_enabled": false, 2024-09-21T12:33:44.199 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_bucket_count": 0, 2024-09-21T12:33:44.199 INFO:teuthology.orchestra.run.smithi114.stdout: "degraded_stretch_mode": 0, 2024-09-21T12:33:44.199 INFO:teuthology.orchestra.run.smithi114.stdout: "recovering_stretch_mode": 0, 2024-09-21T12:33:44.199 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode_bucket": 0 2024-09-21T12:33:44.199 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:44.199 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:33:44.246 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:44.279 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:44.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:44.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:44.344 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:33:44.344 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:33:44.344 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:33:44.344 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:33:44.344 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:33:44.360 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:44 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/70500991' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T12:33:44.360 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:44 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/70500991' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T12:33:44.381 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:44.415 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:44.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:44.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:44.516 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:44.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:44.555 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:44.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:44.656 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:44.666 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 68 16 100780 100764 92.5874 72.625 0.00769866 0.0107953 2024-09-21T12:33:44.690 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:44.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:44.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:44.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:44 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/70500991' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T12:33:44.791 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:44.824 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:44.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:44.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:44.925 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:44.959 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:44.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:44.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:45.060 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:45.094 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:45.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:45.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:45.195 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:45.222 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:45.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:45.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:45.324 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:45.356 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:45.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:45.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:45.458 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:45.461 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:33:45.498 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:45.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:45.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:45.540 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:45 smithi114 ceph-mon[40867]: pgmap v693: 178 pgs: 178 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.56k op/s 2024-09-21T12:33:45.540 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:45 smithi114 ceph-mon[30324]: pgmap v693: 178 pgs: 178 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.56k op/s 2024-09-21T12:33:45.600 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:45.653 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:45.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:45.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:45.666 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 69 16 101973 101957 92.3262 74.5625 0.00840895 0.0108258 2024-09-21T12:33:45.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:45 smithi183 ceph-mon[33277]: pgmap v693: 178 pgs: 178 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.56k op/s 2024-09-21T12:33:45.754 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:45.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:45.801 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:45.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:45.902 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:45.946 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:45.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:45.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:46.048 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:46.081 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:46.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:46.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:46.183 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:46.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:46.205 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:46.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:46.306 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:46.337 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:46.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:46.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:46.439 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:46.475 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:46.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:46.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:46.576 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:46.617 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:46.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:46.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:46.665 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 70 16 103186 103170 92.0902 75.8125 0.0129681 0.0108545 2024-09-21T12:33:46.719 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:46.754 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:46.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:46.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:46.856 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:46.893 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:46.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:46.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:46.994 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:47.032 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:47.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:47.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:47.133 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:47.172 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:47.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:47.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:47.274 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:47.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:47.311 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:47.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:47.412 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:47.450 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:47.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:47.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:47.552 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:47.576 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:47.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:47.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:47.665 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 71 16 104426 104410 91.8847 77.5 0.018449 0.0108785 2024-09-21T12:33:47.678 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:47.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:47.716 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:47.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:47.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:33:47 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:33:47.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:47 smithi183 ceph-mon[33277]: pgmap v694: 178 pgs: 178 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 594 B/s rd, 61 MiB/s wr, 982 op/s 2024-09-21T12:33:47.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:47 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]} : dispatch 2024-09-21T12:33:47.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:47 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]} : dispatch 2024-09-21T12:33:47.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:47 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:33:47.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:47 smithi114 ceph-mon[40867]: pgmap v694: 178 pgs: 178 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 594 B/s rd, 61 MiB/s wr, 982 op/s 2024-09-21T12:33:47.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:47 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]} : dispatch 2024-09-21T12:33:47.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:47 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]} : dispatch 2024-09-21T12:33:47.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:47 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:33:47.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:47 smithi114 ceph-mon[30324]: pgmap v694: 178 pgs: 178 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 594 B/s rd, 61 MiB/s wr, 982 op/s 2024-09-21T12:33:47.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:47 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]} : dispatch 2024-09-21T12:33:47.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:47 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]} : dispatch 2024-09-21T12:33:47.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:47 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:33:47.817 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:47.849 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:47.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:47.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:47.950 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:47.981 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:47.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:47.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:48.048 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T12:33:48.083 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:48.117 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:48.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:48.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:48.219 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:48.271 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:48.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:48.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:48.372 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:48.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:48.431 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:48.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:48.532 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:48.568 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:48.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:48.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:48.667 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 72 16 105599 105583 91.6252 73.3125 0.0312561 0.0109075 2024-09-21T12:33:48.670 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:48.698 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:48.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:48.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:48.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:48 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-09-21T12:33:48.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:48 smithi183 ceph-mon[33277]: osdmap e377: 8 total, 8 up, 8 in 2024-09-21T12:33:48.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:48 smithi183 ceph-mon[33277]: osdmap e378: 8 total, 8 up, 8 in 2024-09-21T12:33:48.759 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:48 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-09-21T12:33:48.759 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:48 smithi114 ceph-mon[30324]: osdmap e377: 8 total, 8 up, 8 in 2024-09-21T12:33:48.759 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:48 smithi114 ceph-mon[30324]: osdmap e378: 8 total, 8 up, 8 in 2024-09-21T12:33:48.763 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:48 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-09-21T12:33:48.763 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:48 smithi114 ceph-mon[40867]: osdmap e377: 8 total, 8 up, 8 in 2024-09-21T12:33:48.763 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:48 smithi114 ceph-mon[40867]: osdmap e378: 8 total, 8 up, 8 in 2024-09-21T12:33:48.800 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:48.833 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:48.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:48.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:48.935 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:48.964 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:48.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:48.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:49.065 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:49.100 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:49.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:49.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:49.202 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:49.260 INFO:tasks.thrashosds.thrasher:j is {'epoch': 376, 'fsid': 'e2ad975c-7812-11ef-baf6-efdc52797490', 'created': '2024-09-21T12:14:24.181227+0000', 'modified': '2024-09-21T12:33:38.241389+0000', 'last_up_change': '2024-09-21T12:20:03.702987+0000', 'last_in_change': '2024-09-21T12:32:37.364926+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T12:17:38.357647+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '218', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '216', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.809999942779541, 'score_stable': 1.809999942779541, 'optimal_score': 1, 'raw_score_acting': 1.809999942779541, 'raw_score_stable': 1.809999942779541, 'primary_affinity_weighted': 0.8608264923095703, 'average_primary_affinity': 0.8600000143051147, 'average_primary_affinity_weighted': 0.8600000143051147}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-21T12:20:12.761046+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 61, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.8608264923095703, 'average_primary_affinity': 0.8600000143051147, 'average_primary_affinity_weighted': 0.8600000143051147}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-21T12:20:13.713971+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8608264923095703, 'average_primary_affinity': 0.8600000143051147, 'average_primary_affinity_weighted': 0.8600000143051147}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-21T12:20:15.368765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.8608264923095703, 'average_primary_affinity': 0.8600000143051147, 'average_primary_affinity_weighted': 0.8600000143051147}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-21T12:20:17.314275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.8608264923095703, 'average_primary_affinity': 0.8600000143051147, 'average_primary_affinity_weighted': 0.8600000143051147}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-21T12:20:19.370218+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8608264923095703, 'average_primary_affinity': 0.8600000143051147, 'average_primary_affinity_weighted': 0.8600000143051147}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-09-21T12:32:28.476438+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '348', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.8608264923095703, 'average_primary_affinity': 0.8600000143051147, 'average_primary_affinity_weighted': 0.8600000143051147}}], 'osds': [{'osd': 0, 'uuid': '5715e521-c7dc-41dd-a075-5c96a88fa9b3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 353, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6802', 'nonce': 3555869816}, {'type': 'v1', 'addr': '172.21.15.114:6803', 'nonce': 3555869816}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6804', 'nonce': 3555869816}, {'type': 'v1', 'addr': '172.21.15.114:6805', 'nonce': 3555869816}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6808', 'nonce': 3555869816}, {'type': 'v1', 'addr': '172.21.15.114:6809', 'nonce': 3555869816}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6806', 'nonce': 3555869816}, {'type': 'v1', 'addr': '172.21.15.114:6807', 'nonce': 3555869816}]}, 'public_addr': '172.21.15.114:6803/3555869816', 'cluster_addr': '172.21.15.114:6805/3555869816', 'heartbeat_back_addr': '172.21.15.114:6809/3555869816', 'heartbeat_front_addr': '172.21.15.114:6807/3555869816', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '1fe160a7-5972-4e3e-99ce-15e2311ba4c5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 353, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6810', 'nonce': 2744663863}, {'type': 'v1', 'addr': '172.21.15.114:6811', 'nonce': 2744663863}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6812', 'nonce': 2744663863}, {'type': 'v1', 'addr': '172.21.15.114:6813', 'nonce': 2744663863}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6816', 'nonce': 2744663863}, {'type': 'v1', 'addr': '172.21.15.114:6817', 'nonce': 2744663863}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6814', 'nonce': 2744663863}, {'type': 'v1', 'addr': '172.21.15.114:6815', 'nonce': 2744663863}]}, 'public_addr': '172.21.15.114:6811/2744663863', 'cluster_addr': '172.21.15.114:6813/2744663863', 'heartbeat_back_addr': '172.21.15.114:6817/2744663863', 'heartbeat_front_addr': '172.21.15.114:6815/2744663863', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '6a63581f-7603-49ae-9ab3-b37e7c91ab91', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 351, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6818', 'nonce': 3740045173}, {'type': 'v1', 'addr': '172.21.15.114:6819', 'nonce': 3740045173}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6820', 'nonce': 3740045173}, {'type': 'v1', 'addr': '172.21.15.114:6821', 'nonce': 3740045173}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6824', 'nonce': 3740045173}, {'type': 'v1', 'addr': '172.21.15.114:6825', 'nonce': 3740045173}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6822', 'nonce': 3740045173}, {'type': 'v1', 'addr': '172.21.15.114:6823', 'nonce': 3740045173}]}, 'public_addr': '172.21.15.114:6819/3740045173', 'cluster_addr': '172.21.15.114:6821/3740045173', 'heartbeat_back_addr': '172.21.15.114:6825/3740045173', 'heartbeat_front_addr': '172.21.15.114:6823/3740045173', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '0ad68cdb-ae5c-4090-8a8d-b67c1177ae42', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 349, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6826', 'nonce': 1312590318}, {'type': 'v1', 'addr': '172.21.15.114:6827', 'nonce': 1312590318}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6828', 'nonce': 1312590318}, {'type': 'v1', 'addr': '172.21.15.114:6829', 'nonce': 1312590318}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6832', 'nonce': 1312590318}, {'type': 'v1', 'addr': '172.21.15.114:6833', 'nonce': 1312590318}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6830', 'nonce': 1312590318}, {'type': 'v1', 'addr': '172.21.15.114:6831', 'nonce': 1312590318}]}, 'public_addr': '172.21.15.114:6827/1312590318', 'cluster_addr': '172.21.15.114:6829/1312590318', 'heartbeat_back_addr': '172.21.15.114:6833/1312590318', 'heartbeat_front_addr': '172.21.15.114:6831/1312590318', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '26bb8a84-e41d-4425-8491-2dea9764b7cf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 353, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6800', 'nonce': 1391644836}, {'type': 'v1', 'addr': '172.21.15.183:6801', 'nonce': 1391644836}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6802', 'nonce': 1391644836}, {'type': 'v1', 'addr': '172.21.15.183:6803', 'nonce': 1391644836}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6806', 'nonce': 1391644836}, {'type': 'v1', 'addr': '172.21.15.183:6807', 'nonce': 1391644836}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6804', 'nonce': 1391644836}, {'type': 'v1', 'addr': '172.21.15.183:6805', 'nonce': 1391644836}]}, 'public_addr': '172.21.15.183:6801/1391644836', 'cluster_addr': '172.21.15.183:6803/1391644836', 'heartbeat_back_addr': '172.21.15.183:6807/1391644836', 'heartbeat_front_addr': '172.21.15.183:6805/1391644836', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '33f96871-0869-4281-82b9-bd518987c551', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.476898193359375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 349, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6808', 'nonce': 3187643473}, {'type': 'v1', 'addr': '172.21.15.183:6809', 'nonce': 3187643473}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6810', 'nonce': 3187643473}, {'type': 'v1', 'addr': '172.21.15.183:6811', 'nonce': 3187643473}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6814', 'nonce': 3187643473}, {'type': 'v1', 'addr': '172.21.15.183:6815', 'nonce': 3187643473}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6812', 'nonce': 3187643473}, {'type': 'v1', 'addr': '172.21.15.183:6813', 'nonce': 3187643473}]}, 'public_addr': '172.21.15.183:6809/3187643473', 'cluster_addr': '172.21.15.183:6811/3187643473', 'heartbeat_back_addr': '172.21.15.183:6815/3187643473', 'heartbeat_front_addr': '172.21.15.183:6813/3187643473', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e09c72b8-ad5a-4779-a874-b149fce6ed70', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.4097137451171875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 352, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6816', 'nonce': 3025199631}, {'type': 'v1', 'addr': '172.21.15.183:6817', 'nonce': 3025199631}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6818', 'nonce': 3025199631}, {'type': 'v1', 'addr': '172.21.15.183:6819', 'nonce': 3025199631}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6822', 'nonce': 3025199631}, {'type': 'v1', 'addr': '172.21.15.183:6823', 'nonce': 3025199631}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6820', 'nonce': 3025199631}, {'type': 'v1', 'addr': '172.21.15.183:6821', 'nonce': 3025199631}]}, 'public_addr': '172.21.15.183:6817/3025199631', 'cluster_addr': '172.21.15.183:6819/3025199631', 'heartbeat_back_addr': '172.21.15.183:6823/3025199631', 'heartbeat_front_addr': '172.21.15.183:6821/3025199631', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1a24194d-1af6-485f-8856-98844a02dd9d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 352, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6824', 'nonce': 3203337442}, {'type': 'v1', 'addr': '172.21.15.183:6825', 'nonce': 3203337442}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6826', 'nonce': 3203337442}, {'type': 'v1', 'addr': '172.21.15.183:6827', 'nonce': 3203337442}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6830', 'nonce': 3203337442}, {'type': 'v1', 'addr': '172.21.15.183:6831', 'nonce': 3203337442}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6828', 'nonce': 3203337442}, {'type': 'v1', 'addr': '172.21.15.183:6829', 'nonce': 3203337442}]}, 'public_addr': '172.21.15.183:6825/3203337442', 'cluster_addr': '172.21.15.183:6827/3203337442', 'heartbeat_back_addr': '172.21.15.183:6831/3203337442', 'heartbeat_front_addr': '172.21.15.183:6829/3203337442', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T12:33:04.792722+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T12:32:54.736840+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T12:33:33.127156+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T12:33:17.347963+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T12:33:31.062655+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T12:33:20.194600+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T12:33:36.228827+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T12:33:29.492142+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.c', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.15', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '5.1d', 'mappings': [{'from': 0, 'to': 3}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.114:6801/237768393': '2024-09-22T12:20:45.528856+0000', '172.21.15.114:6800/237768393': '2024-09-22T12:20:45.528856+0000', '172.21.15.114:0/898990943': '2024-09-22T12:14:53.998653+0000', '172.21.15.114:0/3866305956': '2024-09-22T12:20:45.528856+0000', '172.21.15.114:0/3598782088': '2024-09-22T12:20:45.528856+0000', '172.21.15.114:0/2620151541': '2024-09-22T12:15:20.045217+0000', '172.21.15.114:6800/1725148595': '2024-09-22T12:14:53.998653+0000', '172.21.15.114:0/442404805': '2024-09-22T12:14:53.998653+0000', '172.21.15.114:6801/1725148595': '2024-09-22T12:14:53.998653+0000', '172.21.15.114:0/4173909019': '2024-09-22T12:20:45.528856+0000', '172.21.15.114:0/2106467614': '2024-09-22T12:15:20.045217+0000', '172.21.15.114:0/139803357': '2024-09-22T12:15:20.045217+0000', '172.21.15.114:6801/770932721': '2024-09-22T12:15:20.045217+0000', '172.21.15.114:6800/770932721': '2024-09-22T12:15:20.045217+0000', '172.21.15.114:0/1457547294': '2024-09-22T12:20:45.528856+0000', '172.21.15.114:0/3416259228': '2024-09-22T12:14:53.998653+0000', '172.21.15.114:0/995404371': '2024-09-22T12:20:45.528856+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T12:33:49.261 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:33:49.282 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:49.282 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:49.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:49.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:49.313 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:49.383 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:49.389 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-21T12:33:49.407 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:49.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:49.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:49.456 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:49 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:49.456 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:49 smithi114 ceph-mon[30324]: pgmap v696: 178 pgs: 178 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-09-21T12:33:49.456 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:49 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2293423776' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:33:49.456 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:49 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:33:49.456 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:49 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:49.456 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:49 smithi114 ceph-mon[40867]: pgmap v696: 178 pgs: 178 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-09-21T12:33:49.456 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:49 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2293423776' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:33:49.457 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:49 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:33:49.509 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:49.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:49.538 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:49.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:49.639 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:49.668 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 73 16 106776 106760 91.3774 73.5625 0.0113601 0.0109384 2024-09-21T12:33:49.675 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:49.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:49.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:49.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:49 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:49.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:49 smithi183 ceph-mon[33277]: pgmap v696: 178 pgs: 178 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-09-21T12:33:49.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:49 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2293423776' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:33:49.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:49 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:33:49.777 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:49.812 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:49.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:49.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:49.914 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:49.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:49.955 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:49.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:50.056 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:50.097 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:50.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:50.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:50.199 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:50.236 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:50.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:50.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:50.338 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:50.367 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:50.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:50.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:50.461 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:50 smithi114 ceph-mon[40867]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:33:50.461 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:50 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:33:50.462 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:50 smithi114 ceph-mon[40867]: osdmap e379: 8 total, 8 up, 8 in 2024-09-21T12:33:50.462 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:50 smithi114 ceph-mon[30324]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:33:50.462 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:50 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:33:50.462 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:50 smithi114 ceph-mon[30324]: osdmap e379: 8 total, 8 up, 8 in 2024-09-21T12:33:50.468 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:50.496 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:50.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:50.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:50.563 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:33:50.564 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 20, 2024-09-21T12:33:50.564 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 600, 2024-09-21T12:33:50.564 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-21T12:33:50.564 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.564 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:179611 9.16 9:6ab28fd3:::benchmark_data_smithi114_133214_object55577:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-21T12:33:50.565 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:01.026416+0000", 2024-09-21T12:33:50.565 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 169.48818578300001, 2024-09-21T12:33:50.565 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0040347110000001, 2024-09-21T12:33:50.565 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:50.565 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:50.565 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:33:50.566 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:50.566 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:33:50.566 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:50.566 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 179611 2024-09-21T12:33:50.566 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.567 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:50.567 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.567 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:50.567 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.026416+0000", 2024-09-21T12:33:50.567 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.567 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.568 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.568 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:50.568 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.026416+0000", 2024-09-21T12:33:50.568 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.568 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.568 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.569 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:50.569 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.026417+0000", 2024-09-21T12:33:50.569 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2240000000000001e-06 2024-09-21T12:33:50.569 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.569 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.569 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:50.570 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.026423+0000", 2024-09-21T12:33:50.570 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0679999999999996e-06 2024-09-21T12:33:50.570 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.570 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.570 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:50.570 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.026436+0000", 2024-09-21T12:33:50.571 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.34e-05 2024-09-21T12:33:50.571 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.571 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.571 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:50.571 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.026443+0000", 2024-09-21T12:33:50.571 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5079999999999999e-06 2024-09-21T12:33:50.572 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.572 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.572 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.572 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.026505+0000", 2024-09-21T12:33:50.572 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1450000000000003e-05 2024-09-21T12:33:50.572 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.573 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.573 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:50.573 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.026632+0000", 2024-09-21T12:33:50.573 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012670800000000001 2024-09-21T12:33:50.573 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.573 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.574 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,5,6", 2024-09-21T12:33:50.574 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.026929+0000", 2024-09-21T12:33:50.574 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00029765500000000002 2024-09-21T12:33:50.574 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.574 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.574 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.575 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.034732+0000", 2024-09-21T12:33:50.575 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.007802425 2024-09-21T12:33:50.575 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.575 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.575 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:50.575 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.034741+0000", 2024-09-21T12:33:50.576 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1020000000000003e-06 2024-09-21T12:33:50.576 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.576 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.576 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.576 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.030372+0000", 2024-09-21T12:33:50.576 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99563096699999998 2024-09-21T12:33:50.576 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.577 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.577 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:50.577 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.030451+0000", 2024-09-21T12:33:50.577 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9203999999999996e-05 2024-09-21T12:33:50.577 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.577 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:50.578 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.578 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.578 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.578 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:179657 9.16 9:6bdfd1a2:::benchmark_data_smithi114_133214_object55623:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-21T12:33:50.578 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:01.064476+0000", 2024-09-21T12:33:50.578 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 169.45012555900001, 2024-09-21T12:33:50.579 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98292075899999998, 2024-09-21T12:33:50.579 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:50.579 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:50.579 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:50.579 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:50.579 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:33:50.579 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:50.580 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 179657 2024-09-21T12:33:50.580 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.580 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:50.580 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.580 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:50.580 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.064476+0000", 2024-09-21T12:33:50.581 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.581 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.581 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.581 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:50.581 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.064476+0000", 2024-09-21T12:33:50.581 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.581 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.582 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.582 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:50.582 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.064478+0000", 2024-09-21T12:33:50.582 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4079999999999999e-06 2024-09-21T12:33:50.582 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.582 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.583 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:50.583 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.064482+0000", 2024-09-21T12:33:50.583 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.2089999999999999e-06 2024-09-21T12:33:50.583 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.583 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.583 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:50.583 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.064491+0000", 2024-09-21T12:33:50.583 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1740000000000006e-06 2024-09-21T12:33:50.584 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.584 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.584 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:50.584 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.064497+0000", 2024-09-21T12:33:50.584 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7540000000000003e-06 2024-09-21T12:33:50.584 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.585 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.585 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.585 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.067484+0000", 2024-09-21T12:33:50.585 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0029873439999999999 2024-09-21T12:33:50.585 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.585 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.585 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:50.586 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.067493+0000", 2024-09-21T12:33:50.586 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.3890000000000005e-06 2024-09-21T12:33:50.586 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.586 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.586 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.586 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.031059+0000", 2024-09-21T12:33:50.586 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.96356651000000004 2024-09-21T12:33:50.586 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.587 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.587 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:50.587 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.031150+0000", 2024-09-21T12:33:50.587 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0994000000000003e-05 2024-09-21T12:33:50.587 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.587 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.587 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:33:50.587 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.031439+0000", 2024-09-21T12:33:50.587 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00028878699999999998 2024-09-21T12:33:50.587 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.587 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.587 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.588 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.034468+0000", 2024-09-21T12:33:50.588 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003029482 2024-09-21T12:33:50.588 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.588 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.588 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:50.588 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.041480+0000", 2024-09-21T12:33:50.588 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0070113199999999997 2024-09-21T12:33:50.588 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.588 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.588 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.588 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.047273+0000", 2024-09-21T12:33:50.588 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0057935139999999996 2024-09-21T12:33:50.588 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.589 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.589 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:50.589 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.047353+0000", 2024-09-21T12:33:50.589 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9925999999999993e-05 2024-09-21T12:33:50.589 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.589 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.589 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:50.589 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.047397+0000", 2024-09-21T12:33:50.589 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3948e-05 2024-09-21T12:33:50.589 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.589 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:50.589 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.589 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.589 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.589 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:179691 9.16 9:693ea84b:::benchmark_data_smithi114_133214_object55657:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-21T12:33:50.590 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:01.086828+0000", 2024-09-21T12:33:50.590 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 169.42777358199999, 2024-09-21T12:33:50.590 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.96072276199999995, 2024-09-21T12:33:50.590 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:50.590 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:50.590 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:50.590 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:50.590 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:33:50.590 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:50.590 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 179691 2024-09-21T12:33:50.590 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.590 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:50.590 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.591 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:50.591 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.086828+0000", 2024-09-21T12:33:50.591 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.591 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.591 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.591 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:50.591 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.086828+0000", 2024-09-21T12:33:50.591 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.591 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.591 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.591 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:50.592 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.086830+0000", 2024-09-21T12:33:50.592 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2270000000000001e-06 2024-09-21T12:33:50.592 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.592 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.592 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:50.592 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.086834+0000", 2024-09-21T12:33:50.592 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.9990000000000002e-06 2024-09-21T12:33:50.592 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.592 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.592 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:50.592 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.086843+0000", 2024-09-21T12:33:50.592 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9090000000000006e-06 2024-09-21T12:33:50.592 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.593 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.593 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:50.593 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.086849+0000", 2024-09-21T12:33:50.593 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7440000000000001e-06 2024-09-21T12:33:50.593 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.593 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.593 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.593 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.101158+0000", 2024-09-21T12:33:50.593 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.014308922999999999 2024-09-21T12:33:50.593 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.593 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.593 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:50.593 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.101176+0000", 2024-09-21T12:33:50.594 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7980999999999999e-05 2024-09-21T12:33:50.594 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.594 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.594 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.594 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.032538+0000", 2024-09-21T12:33:50.594 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.93136199600000003 2024-09-21T12:33:50.594 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.594 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.594 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:50.594 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.032634+0000", 2024-09-21T12:33:50.594 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.5873999999999997e-05 2024-09-21T12:33:50.594 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.594 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.595 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:33:50.595 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.032790+0000", 2024-09-21T12:33:50.595 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015569999999999999 2024-09-21T12:33:50.595 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.595 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.595 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.595 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.034503+0000", 2024-09-21T12:33:50.595 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0017132219999999999 2024-09-21T12:33:50.595 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.595 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.595 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:50.595 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.041484+0000", 2024-09-21T12:33:50.596 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0069813119999999999 2024-09-21T12:33:50.596 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.596 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.596 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.596 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.047470+0000", 2024-09-21T12:33:50.596 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0059858710000000002 2024-09-21T12:33:50.596 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.596 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.596 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:50.596 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.047517+0000", 2024-09-21T12:33:50.596 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6820000000000002e-05 2024-09-21T12:33:50.596 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.596 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.597 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:50.597 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.047551+0000", 2024-09-21T12:33:50.597 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4183999999999997e-05 2024-09-21T12:33:50.597 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.597 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:50.597 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.597 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.597 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.597 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:180848 9.12 9:4ecd9eaa:::benchmark_data_smithi114_133214_object56814:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-09-21T12:33:50.597 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:04.045860+0000", 2024-09-21T12:33:50.597 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 166.46874162099999, 2024-09-21T12:33:50.598 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99015821699999995, 2024-09-21T12:33:50.598 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:50.598 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:50.598 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:50.598 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:50.598 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:33:50.598 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:50.598 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 180848 2024-09-21T12:33:50.598 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.598 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:50.598 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.598 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:50.598 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.045860+0000", 2024-09-21T12:33:50.599 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.599 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.599 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.599 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:50.599 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.045860+0000", 2024-09-21T12:33:50.599 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.599 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.599 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.599 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:50.599 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.045862+0000", 2024-09-21T12:33:50.599 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.466e-06 2024-09-21T12:33:50.599 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.599 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.600 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:50.600 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.045866+0000", 2024-09-21T12:33:50.600 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.89e-06 2024-09-21T12:33:50.600 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.600 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.600 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:50.600 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.045887+0000", 2024-09-21T12:33:50.600 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0801e-05 2024-09-21T12:33:50.600 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.600 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.600 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:50.601 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.045894+0000", 2024-09-21T12:33:50.601 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2830000000000004e-06 2024-09-21T12:33:50.601 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.601 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.601 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.601 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.045948+0000", 2024-09-21T12:33:50.601 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3677999999999997e-05 2024-09-21T12:33:50.601 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.601 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.601 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:50.601 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.045956+0000", 2024-09-21T12:33:50.601 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1079999999999993e-06 2024-09-21T12:33:50.601 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.601 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.602 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.602 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:05.032414+0000", 2024-09-21T12:33:50.602 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98645813900000001 2024-09-21T12:33:50.602 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.602 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.602 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:50.602 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:05.032550+0000", 2024-09-21T12:33:50.602 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013655699999999999 2024-09-21T12:33:50.602 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.602 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.602 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,5", 2024-09-21T12:33:50.603 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:05.032837+0000", 2024-09-21T12:33:50.603 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00028715899999999999 2024-09-21T12:33:50.603 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.603 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.603 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.603 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:05.035846+0000", 2024-09-21T12:33:50.603 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0030088020000000001 2024-09-21T12:33:50.603 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.603 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.603 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:50.603 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:05.035860+0000", 2024-09-21T12:33:50.603 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3366e-05 2024-09-21T12:33:50.603 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.603 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.603 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.604 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:05.035895+0000", 2024-09-21T12:33:50.604 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5751999999999999e-05 2024-09-21T12:33:50.604 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.604 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.604 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:50.604 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:05.035973+0000", 2024-09-21T12:33:50.604 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7999999999999999e-05 2024-09-21T12:33:50.604 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.604 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.604 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:50.604 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:05.036019+0000", 2024-09-21T12:33:50.604 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5216000000000002e-05 2024-09-21T12:33:50.605 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.605 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:50.605 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.605 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.605 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.605 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:221947 9.0 9.c6972ae0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:33:50.605 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:38.070413+0000", 2024-09-21T12:33:50.605 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 132.44418852199999, 2024-09-21T12:33:50.605 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97957693599999995, 2024-09-21T12:33:50.605 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:50.605 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:50.605 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:33:50.605 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:50.605 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:33:50.606 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:50.606 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 221947 2024-09-21T12:33:50.606 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.606 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:50.606 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.606 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:50.606 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.070413+0000", 2024-09-21T12:33:50.606 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.606 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.606 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.606 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:50.606 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.070413+0000", 2024-09-21T12:33:50.606 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.607 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.607 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.607 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:50.607 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.070414+0000", 2024-09-21T12:33:50.607 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4000000000000002e-07 2024-09-21T12:33:50.607 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.607 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.607 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:50.607 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.070416+0000", 2024-09-21T12:33:50.607 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1730000000000002e-06 2024-09-21T12:33:50.607 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.607 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.607 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:50.607 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.070423+0000", 2024-09-21T12:33:50.607 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7320000000000001e-06 2024-09-21T12:33:50.608 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.608 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.608 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:50.608 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.070428+0000", 2024-09-21T12:33:50.608 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0100000000000003e-06 2024-09-21T12:33:50.608 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.608 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.608 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.608 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.071108+0000", 2024-09-21T12:33:50.608 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00067975200000000005 2024-09-21T12:33:50.608 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.608 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.608 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:50.608 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.071117+0000", 2024-09-21T12:33:50.609 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0880000000000001e-06 2024-09-21T12:33:50.609 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.609 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.609 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.609 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.049979+0000", 2024-09-21T12:33:50.609 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97886278999999998 2024-09-21T12:33:50.609 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.609 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.609 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:50.609 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.049990+0000", 2024-09-21T12:33:50.609 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0951000000000001e-05 2024-09-21T12:33:50.609 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.609 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:50.609 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.610 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.610 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.610 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:221951 9.0 9.29003820 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:33:50.610 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:38.078421+0000", 2024-09-21T12:33:50.610 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 132.43618110700001, 2024-09-21T12:33:50.610 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97160824800000001, 2024-09-21T12:33:50.610 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:50.610 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:50.610 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:33:50.610 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:50.610 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:33:50.610 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:50.610 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 221951 2024-09-21T12:33:50.611 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.611 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:50.611 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.611 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:50.611 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.078421+0000", 2024-09-21T12:33:50.611 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.611 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.611 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.611 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:50.611 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.078421+0000", 2024-09-21T12:33:50.611 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.611 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.612 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.612 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:50.612 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.078422+0000", 2024-09-21T12:33:50.612 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4300000000000001e-06 2024-09-21T12:33:50.612 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.612 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.612 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:50.612 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.078426+0000", 2024-09-21T12:33:50.612 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5140000000000002e-06 2024-09-21T12:33:50.612 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.612 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.612 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:50.612 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.078435+0000", 2024-09-21T12:33:50.613 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6570000000000003e-06 2024-09-21T12:33:50.613 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.613 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.613 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:50.613 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.078440+0000", 2024-09-21T12:33:50.613 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2660000000000001e-06 2024-09-21T12:33:50.613 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.613 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.613 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.613 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.078657+0000", 2024-09-21T12:33:50.613 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00021765200000000001 2024-09-21T12:33:50.613 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.613 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.613 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:50.614 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:38.078666+0000", 2024-09-21T12:33:50.614 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.2109999999999998e-06 2024-09-21T12:33:50.614 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.614 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.614 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.614 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.050016+0000", 2024-09-21T12:33:50.614 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97134992799999997 2024-09-21T12:33:50.614 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.614 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.614 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:50.614 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:39.050029+0000", 2024-09-21T12:33:50.614 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.359e-05 2024-09-21T12:33:50.614 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.614 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:50.615 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.615 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.615 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.615 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231723 9.14 9:2de8fd01:::benchmark_data_smithi114_133214_object107689:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T12:33:50.615 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:49.066280+0000", 2024-09-21T12:33:50.615 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 121.44832245000001, 2024-09-21T12:33:50.615 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1384219230000001, 2024-09-21T12:33:50.615 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:50.615 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:50.615 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:50.615 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:50.615 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:33:50.615 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:50.616 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231723 2024-09-21T12:33:50.616 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.616 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:50.616 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.616 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:50.616 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.066280+0000", 2024-09-21T12:33:50.616 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.616 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.616 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.616 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:50.616 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.066280+0000", 2024-09-21T12:33:50.617 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.617 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.617 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.617 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:50.617 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.066281+0000", 2024-09-21T12:33:50.617 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5939999999999999e-06 2024-09-21T12:33:50.617 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.617 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.617 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:50.617 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.066286+0000", 2024-09-21T12:33:50.617 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0189999999999997e-06 2024-09-21T12:33:50.617 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.618 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.618 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:50.618 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.066298+0000", 2024-09-21T12:33:50.618 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2024999999999999e-05 2024-09-21T12:33:50.618 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.618 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.618 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:50.618 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.066305+0000", 2024-09-21T12:33:50.618 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9870000000000002e-06 2024-09-21T12:33:50.619 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.619 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.619 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.619 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.066371+0000", 2024-09-21T12:33:50.619 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6025999999999994e-05 2024-09-21T12:33:50.619 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.619 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.619 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:50.619 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.066382+0000", 2024-09-21T12:33:50.620 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0311e-05 2024-09-21T12:33:50.620 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.620 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.620 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.620 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.038211+0000", 2024-09-21T12:33:50.620 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97182923300000001 2024-09-21T12:33:50.620 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.620 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.620 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:50.621 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.038222+0000", 2024-09-21T12:33:50.621 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1032e-05 2024-09-21T12:33:50.621 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.621 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.621 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.621 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.201658+0000", 2024-09-21T12:33:50.621 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1634362899999999 2024-09-21T12:33:50.621 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.622 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.622 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:50.622 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.201745+0000", 2024-09-21T12:33:50.622 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6829000000000005e-05 2024-09-21T12:33:50.622 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.622 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.622 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,3,5", 2024-09-21T12:33:50.622 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.201998+0000", 2024-09-21T12:33:50.622 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000253212 2024-09-21T12:33:50.623 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.623 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.623 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:50.623 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.203777+0000", 2024-09-21T12:33:50.623 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001778565 2024-09-21T12:33:50.623 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.623 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.623 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.623 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.204187+0000", 2024-09-21T12:33:50.624 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000409873 2024-09-21T12:33:50.624 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.624 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.624 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.624 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.204239+0000", 2024-09-21T12:33:50.624 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2166999999999997e-05 2024-09-21T12:33:50.624 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.624 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.625 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.625 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.204276+0000", 2024-09-21T12:33:50.625 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.7181999999999997e-05 2024-09-21T12:33:50.625 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.625 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.625 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:50.625 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.204485+0000", 2024-09-21T12:33:50.625 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00020932399999999999 2024-09-21T12:33:50.625 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.626 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.626 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:50.626 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.204701+0000", 2024-09-21T12:33:50.626 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00021625400000000001 2024-09-21T12:33:50.626 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.626 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:50.626 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.626 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.627 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.627 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231668 9.7 9:e40b995a:::benchmark_data_smithi114_133214_object107634:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:33:50.627 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029737+0000", 2024-09-21T12:33:50.627 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 120.484865101, 2024-09-21T12:33:50.627 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1844050580000001, 2024-09-21T12:33:50.627 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:50.627 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:50.627 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:33:50.627 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:50.628 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:33:50.628 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:50.628 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231668 2024-09-21T12:33:50.628 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.628 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:50.628 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.628 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:50.628 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029737+0000", 2024-09-21T12:33:50.629 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.629 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.629 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.629 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:50.629 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029737+0000", 2024-09-21T12:33:50.629 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.629 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.629 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.629 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:50.629 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029739+0000", 2024-09-21T12:33:50.630 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9300000000000002e-06 2024-09-21T12:33:50.630 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.630 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.630 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:50.630 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029744+0000", 2024-09-21T12:33:50.630 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6970000000000001e-06 2024-09-21T12:33:50.630 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.630 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.630 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:50.630 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029764+0000", 2024-09-21T12:33:50.630 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0959e-05 2024-09-21T12:33:50.630 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.630 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.630 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:50.631 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029773+0000", 2024-09-21T12:33:50.631 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9390000000000003e-06 2024-09-21T12:33:50.631 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.631 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.631 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.631 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.030032+0000", 2024-09-21T12:33:50.631 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00025845499999999999 2024-09-21T12:33:50.631 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.631 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.631 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op must wait for map", 2024-09-21T12:33:50.631 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.030039+0000", 2024-09-21T12:33:50.631 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5310000000000004e-06 2024-09-21T12:33:50.631 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.631 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.632 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.632 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033252+0000", 2024-09-21T12:33:50.632 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0032126519999999999 2024-09-21T12:33:50.632 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.632 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.632 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:50.632 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033262+0000", 2024-09-21T12:33:50.632 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6039999999999999e-06 2024-09-21T12:33:50.632 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.632 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.632 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.632 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.214079+0000", 2024-09-21T12:33:50.632 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1808168320000001 2024-09-21T12:33:50.632 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.633 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.633 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:50.633 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.214142+0000", 2024-09-21T12:33:50.633 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3459000000000003e-05 2024-09-21T12:33:50.633 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.633 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:50.633 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.633 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.633 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.633 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231684 9.2 9:431c20d3:::benchmark_data_smithi114_133214_object107650:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:33:50.633 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029798+0000", 2024-09-21T12:33:50.633 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 120.484804238, 2024-09-21T12:33:50.633 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.174456956, 2024-09-21T12:33:50.633 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:50.634 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:50.634 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:33:50.634 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:50.634 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:33:50.634 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:50.634 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231684 2024-09-21T12:33:50.634 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.634 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:50.634 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.634 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:50.634 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029798+0000", 2024-09-21T12:33:50.634 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.634 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.634 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.635 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:50.635 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029798+0000", 2024-09-21T12:33:50.635 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.635 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.635 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.635 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:50.635 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029798+0000", 2024-09-21T12:33:50.635 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4600000000000005e-07 2024-09-21T12:33:50.635 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.635 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.635 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:50.636 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029802+0000", 2024-09-21T12:33:50.636 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.399e-06 2024-09-21T12:33:50.636 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.636 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.636 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:50.636 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029813+0000", 2024-09-21T12:33:50.636 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0927999999999999e-05 2024-09-21T12:33:50.636 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.636 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.636 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:50.636 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029819+0000", 2024-09-21T12:33:50.636 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3840000000000002e-06 2024-09-21T12:33:50.636 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.637 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.637 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.637 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029883+0000", 2024-09-21T12:33:50.637 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3903000000000001e-05 2024-09-21T12:33:50.637 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.637 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.637 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op must wait for map", 2024-09-21T12:33:50.637 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029891+0000", 2024-09-21T12:33:50.637 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1929999999999993e-06 2024-09-21T12:33:50.637 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.637 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.637 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.637 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033472+0000", 2024-09-21T12:33:50.638 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003581009 2024-09-21T12:33:50.638 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.638 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.638 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:50.638 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033481+0000", 2024-09-21T12:33:50.638 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5320000000000006e-06 2024-09-21T12:33:50.638 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.638 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.638 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.638 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.204173+0000", 2024-09-21T12:33:50.638 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.170692377 2024-09-21T12:33:50.638 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.639 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.639 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:50.639 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.204255+0000", 2024-09-21T12:33:50.639 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1685000000000001e-05 2024-09-21T12:33:50.639 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.639 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:50.639 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.639 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.639 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.639 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231688 9.7 9:e423c016:::benchmark_data_smithi114_133214_object107654:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:33:50.639 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029833+0000", 2024-09-21T12:33:50.639 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 120.484768848, 2024-09-21T12:33:50.640 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.184627359, 2024-09-21T12:33:50.640 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:50.640 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:50.640 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:33:50.640 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:50.640 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:33:50.640 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:50.640 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231688 2024-09-21T12:33:50.640 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.640 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:50.640 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.640 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:50.640 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029833+0000", 2024-09-21T12:33:50.640 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.641 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.641 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.641 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:50.641 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029833+0000", 2024-09-21T12:33:50.641 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.641 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.641 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.641 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:50.641 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029834+0000", 2024-09-21T12:33:50.641 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6600000000000002e-07 2024-09-21T12:33:50.641 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.641 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.641 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:50.641 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029836+0000", 2024-09-21T12:33:50.642 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.6450000000000001e-06 2024-09-21T12:33:50.642 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.642 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.642 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:50.642 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029843+0000", 2024-09-21T12:33:50.642 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6440000000000003e-06 2024-09-21T12:33:50.642 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.642 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.642 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:50.642 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029848+0000", 2024-09-21T12:33:50.642 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0549999999999999e-06 2024-09-21T12:33:50.642 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.643 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.643 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.643 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033286+0000", 2024-09-21T12:33:50.643 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0034382089999999998 2024-09-21T12:33:50.643 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.643 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.643 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:50.643 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033294+0000", 2024-09-21T12:33:50.643 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9279999999999993e-06 2024-09-21T12:33:50.643 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.643 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.643 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.643 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.214400+0000", 2024-09-21T12:33:50.644 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.181105869 2024-09-21T12:33:50.644 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.644 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.644 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:50.644 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.214461+0000", 2024-09-21T12:33:50.644 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0543e-05 2024-09-21T12:33:50.644 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.644 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:50.644 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.644 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.644 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.644 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231694 9.7 9:e4532815:::benchmark_data_smithi114_133214_object107660:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:33:50.644 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029855+0000", 2024-09-21T12:33:50.644 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 120.484746923, 2024-09-21T12:33:50.645 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.212233884, 2024-09-21T12:33:50.645 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:50.645 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:50.645 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:50.645 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:50.645 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:33:50.645 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:50.645 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231694 2024-09-21T12:33:50.645 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.645 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:50.645 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.645 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:50.645 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029855+0000", 2024-09-21T12:33:50.646 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.646 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.646 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.646 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:50.646 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029855+0000", 2024-09-21T12:33:50.646 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.646 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.646 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.646 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:50.646 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029856+0000", 2024-09-21T12:33:50.646 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4499999999999997e-07 2024-09-21T12:33:50.646 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.646 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.646 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:50.647 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029858+0000", 2024-09-21T12:33:50.647 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.131e-06 2024-09-21T12:33:50.647 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.647 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.647 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:50.647 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029864+0000", 2024-09-21T12:33:50.647 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1410000000000002e-06 2024-09-21T12:33:50.647 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.647 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.647 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:50.647 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029869+0000", 2024-09-21T12:33:50.647 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7829999999999998e-06 2024-09-21T12:33:50.647 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.647 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.648 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.648 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033319+0000", 2024-09-21T12:33:50.648 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003450304 2024-09-21T12:33:50.648 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.648 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.648 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:50.648 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033326+0000", 2024-09-21T12:33:50.648 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3590000000000002e-06 2024-09-21T12:33:50.648 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.648 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.648 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.648 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.214613+0000", 2024-09-21T12:33:50.648 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1812871789999999 2024-09-21T12:33:50.648 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.649 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.649 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:50.649 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.214822+0000", 2024-09-21T12:33:50.649 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00020822599999999999 2024-09-21T12:33:50.649 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.649 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.649 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,3,5", 2024-09-21T12:33:50.649 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.215218+0000", 2024-09-21T12:33:50.649 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000396728 2024-09-21T12:33:50.649 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.649 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.649 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:50.649 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.229559+0000", 2024-09-21T12:33:50.649 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.014340258 2024-09-21T12:33:50.649 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.650 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.650 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.650 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.232468+0000", 2024-09-21T12:33:50.650 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0029092469999999998 2024-09-21T12:33:50.650 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.650 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.650 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.650 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.232552+0000", 2024-09-21T12:33:50.650 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.3855000000000003e-05 2024-09-21T12:33:50.650 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.650 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.650 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.650 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.241712+0000", 2024-09-21T12:33:50.651 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0091604219999999997 2024-09-21T12:33:50.651 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.651 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.651 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:50.651 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.241968+0000", 2024-09-21T12:33:50.651 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00025556599999999999 2024-09-21T12:33:50.651 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.651 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.651 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:50.651 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.242089+0000", 2024-09-21T12:33:50.651 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012124 2024-09-21T12:33:50.651 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.652 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:50.652 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.652 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.652 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.652 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231696 9.2 9:4615e034:::benchmark_data_smithi114_133214_object107662:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:33:50.652 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029874+0000", 2024-09-21T12:33:50.652 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 120.484727944, 2024-09-21T12:33:50.652 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.181378748, 2024-09-21T12:33:50.652 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:50.652 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:50.652 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:50.652 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:50.652 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:33:50.652 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:50.653 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231696 2024-09-21T12:33:50.653 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.653 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:50.653 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.653 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:50.653 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029874+0000", 2024-09-21T12:33:50.653 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.653 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.653 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.653 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:50.653 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029874+0000", 2024-09-21T12:33:50.653 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.653 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.653 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.654 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:50.654 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029874+0000", 2024-09-21T12:33:50.654 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5600000000000001e-07 2024-09-21T12:33:50.654 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.654 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.654 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:50.654 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029876+0000", 2024-09-21T12:33:50.654 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.99e-06 2024-09-21T12:33:50.654 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.654 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.654 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:50.654 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029882+0000", 2024-09-21T12:33:50.655 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4750000000000001e-06 2024-09-21T12:33:50.655 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.655 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.655 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:50.655 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029887+0000", 2024-09-21T12:33:50.655 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2839999999999997e-06 2024-09-21T12:33:50.655 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.655 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.655 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.655 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033506+0000", 2024-09-21T12:33:50.655 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00361875 2024-09-21T12:33:50.656 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.656 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.656 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:50.656 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033514+0000", 2024-09-21T12:33:50.656 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6850000000000001e-06 2024-09-21T12:33:50.656 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.656 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.656 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.656 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.204539+0000", 2024-09-21T12:33:50.656 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1710255940000001 2024-09-21T12:33:50.656 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.656 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.656 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:50.657 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.204647+0000", 2024-09-21T12:33:50.657 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000107538 2024-09-21T12:33:50.657 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.657 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.657 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,3,7", 2024-09-21T12:33:50.657 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.204970+0000", 2024-09-21T12:33:50.657 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00032291999999999998 2024-09-21T12:33:50.657 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.657 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.657 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:50.657 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.207582+0000", 2024-09-21T12:33:50.657 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0026120230000000002 2024-09-21T12:33:50.657 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.657 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.658 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.658 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.210625+0000", 2024-09-21T12:33:50.658 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0030428959999999998 2024-09-21T12:33:50.658 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.658 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.658 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.658 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.210912+0000", 2024-09-21T12:33:50.658 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00028736700000000002 2024-09-21T12:33:50.658 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.658 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.658 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.658 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.211091+0000", 2024-09-21T12:33:50.659 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017910199999999999 2024-09-21T12:33:50.659 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.659 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.659 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:50.659 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.211150+0000", 2024-09-21T12:33:50.659 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8934999999999997e-05 2024-09-21T12:33:50.659 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.659 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.659 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:50.659 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.211253+0000", 2024-09-21T12:33:50.659 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000102833 2024-09-21T12:33:50.659 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.659 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:50.659 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.660 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.660 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.660 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231712 9.2 9:41d44c3f:::benchmark_data_smithi114_133214_object107678:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:33:50.660 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029893+0000", 2024-09-21T12:33:50.660 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 120.48470896000001, 2024-09-21T12:33:50.660 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.194669177, 2024-09-21T12:33:50.660 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:50.660 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:50.660 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:50.660 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:50.660 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:33:50.660 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:50.660 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231712 2024-09-21T12:33:50.661 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.661 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:50.661 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.661 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:50.661 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029893+0000", 2024-09-21T12:33:50.661 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.661 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.661 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.661 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:50.661 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029893+0000", 2024-09-21T12:33:50.661 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.661 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.661 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.662 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:50.662 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029893+0000", 2024-09-21T12:33:50.662 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5499999999999999e-07 2024-09-21T12:33:50.662 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.662 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.662 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:50.662 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029895+0000", 2024-09-21T12:33:50.662 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7859999999999999e-06 2024-09-21T12:33:50.662 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.662 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.662 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:50.662 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029901+0000", 2024-09-21T12:33:50.662 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7239999999999997e-06 2024-09-21T12:33:50.662 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.663 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.663 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:50.663 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029905+0000", 2024-09-21T12:33:50.663 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5210000000000004e-06 2024-09-21T12:33:50.663 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.663 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.663 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.663 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033538+0000", 2024-09-21T12:33:50.663 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0036328900000000002 2024-09-21T12:33:50.663 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.663 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.663 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:50.663 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033546+0000", 2024-09-21T12:33:50.664 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9170000000000006e-06 2024-09-21T12:33:50.664 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.664 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.664 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.664 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.207834+0000", 2024-09-21T12:33:50.664 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.17428818 2024-09-21T12:33:50.664 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.664 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.664 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:50.664 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.207946+0000", 2024-09-21T12:33:50.664 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000111501 2024-09-21T12:33:50.664 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.664 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.664 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,3,7", 2024-09-21T12:33:50.664 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.209056+0000", 2024-09-21T12:33:50.665 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0011098670000000001 2024-09-21T12:33:50.665 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.665 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.665 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.665 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.212301+0000", 2024-09-21T12:33:50.665 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0032456199999999998 2024-09-21T12:33:50.665 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.665 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.665 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:50.665 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.212318+0000", 2024-09-21T12:33:50.665 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7031999999999999e-05 2024-09-21T12:33:50.665 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.666 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.666 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.666 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.214089+0000", 2024-09-21T12:33:50.666 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001770726 2024-09-21T12:33:50.666 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.666 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.666 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.666 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.224405+0000", 2024-09-21T12:33:50.666 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010315658 2024-09-21T12:33:50.666 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.666 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.666 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:50.666 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.224461+0000", 2024-09-21T12:33:50.667 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6341999999999997e-05 2024-09-21T12:33:50.667 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.667 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.667 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:50.667 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.224562+0000", 2024-09-21T12:33:50.667 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:50.669 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010105799999999999 2024-09-21T12:33:50.669 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.669 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:50.669 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.669 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.669 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.669 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231746 9.7 9:e6f0ab86:::benchmark_data_smithi114_133214_object107712:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:33:50.669 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029911+0000", 2024-09-21T12:33:50.669 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 120.484690965, 2024-09-21T12:33:50.669 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2150179190000001, 2024-09-21T12:33:50.670 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:50.670 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:50.670 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:50.670 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:50.670 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:33:50.670 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:50.670 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231746 2024-09-21T12:33:50.670 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.670 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:50.670 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.670 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:50.670 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029911+0000", 2024-09-21T12:33:50.671 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.671 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.671 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.671 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:50.671 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029911+0000", 2024-09-21T12:33:50.671 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.671 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.671 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.671 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:50.671 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029911+0000", 2024-09-21T12:33:50.671 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4700000000000002e-07 2024-09-21T12:33:50.671 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.672 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.672 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:50.672 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029913+0000", 2024-09-21T12:33:50.672 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.889e-06 2024-09-21T12:33:50.672 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.672 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.672 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:50.672 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029919+0000", 2024-09-21T12:33:50.672 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3630000000000002e-06 2024-09-21T12:33:50.672 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.672 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.672 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:50.672 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029923+0000", 2024-09-21T12:33:50.672 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4379999999999997e-06 2024-09-21T12:33:50.672 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033349+0000", 2024-09-21T12:33:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0034264059999999999 2024-09-21T12:33:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033357+0000", 2024-09-21T12:33:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4590000000000001e-06 2024-09-21T12:33:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.225869+0000", 2024-09-21T12:33:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.192512056 2024-09-21T12:33:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.225992+0000", 2024-09-21T12:33:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012313800000000001 2024-09-21T12:33:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,3,5", 2024-09-21T12:33:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.226276+0000", 2024-09-21T12:33:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00028393899999999998 2024-09-21T12:33:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.232512+0000", 2024-09-21T12:33:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0062360330000000002 2024-09-21T12:33:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.232592+0000", 2024-09-21T12:33:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9430000000000004e-05 2024-09-21T12:33:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.241522+0000", 2024-09-21T12:33:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0089301380000000007 2024-09-21T12:33:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.244727+0000", 2024-09-21T12:33:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0032050389999999998 2024-09-21T12:33:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.244780+0000", 2024-09-21T12:33:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3777999999999999e-05 2024-09-21T12:33:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.244929+0000", 2024-09-21T12:33:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014846600000000001 2024-09-21T12:33:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231751 9.7 9:e24ce5a0:::benchmark_data_smithi114_133214_object107717:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:33:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029929+0000", 2024-09-21T12:33:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 120.48467334999999, 2024-09-21T12:33:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2152077539999999, 2024-09-21T12:33:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:33:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231751 2024-09-21T12:33:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029929+0000", 2024-09-21T12:33:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029929+0000", 2024-09-21T12:33:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029929+0000", 2024-09-21T12:33:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5199999999999998e-07 2024-09-21T12:33:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029931+0000", 2024-09-21T12:33:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0090000000000002e-06 2024-09-21T12:33:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029936+0000", 2024-09-21T12:33:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.215e-06 2024-09-21T12:33:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029940+0000", 2024-09-21T12:33:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.2570000000000001e-06 2024-09-21T12:33:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033382+0000", 2024-09-21T12:33:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0034412370000000002 2024-09-21T12:33:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033388+0000", 2024-09-21T12:33:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6780000000000003e-06 2024-09-21T12:33:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.233447+0000", 2024-09-21T12:33:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.200058297 2024-09-21T12:33:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.233585+0000", 2024-09-21T12:33:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000138429 2024-09-21T12:33:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,3,5", 2024-09-21T12:33:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.233890+0000", 2024-09-21T12:33:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00030514399999999998 2024-09-21T12:33:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.237017+0000", 2024-09-21T12:33:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003126789 2024-09-21T12:33:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.237064+0000", 2024-09-21T12:33:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7287000000000003e-05 2024-09-21T12:33:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.242303+0000", 2024-09-21T12:33:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.005238599 2024-09-21T12:33:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.244979+0000", 2024-09-21T12:33:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0026761549999999999 2024-09-21T12:33:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.245035+0000", 2024-09-21T12:33:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6159000000000002e-05 2024-09-21T12:33:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.245136+0000", 2024-09-21T12:33:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000101147 2024-09-21T12:33:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231758 9.7 9:e06abb50:::benchmark_data_smithi114_133214_object107724:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:33:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029948+0000", 2024-09-21T12:33:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 120.484654285, 2024-09-21T12:33:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2159567739999999, 2024-09-21T12:33:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:33:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231758 2024-09-21T12:33:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:50.686 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.686 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:50.686 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029948+0000", 2024-09-21T12:33:50.686 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.686 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.686 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.686 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:50.686 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029948+0000", 2024-09-21T12:33:50.686 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.686 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.686 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.686 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:50.686 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029948+0000", 2024-09-21T12:33:50.686 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.41e-07 2024-09-21T12:33:50.687 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.687 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.687 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:50.687 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029950+0000", 2024-09-21T12:33:50.687 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0470000000000001e-06 2024-09-21T12:33:50.687 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.687 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.687 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:50.687 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029956+0000", 2024-09-21T12:33:50.687 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7069999999999997e-06 2024-09-21T12:33:50.687 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.687 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.687 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:50.687 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029961+0000", 2024-09-21T12:33:50.688 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1320000000000002e-06 2024-09-21T12:33:50.688 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.688 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.688 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.688 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033413+0000", 2024-09-21T12:33:50.688 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0034517609999999998 2024-09-21T12:33:50.688 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.689 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.689 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:50.689 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033420+0000", 2024-09-21T12:33:50.689 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6240000000000002e-06 2024-09-21T12:33:50.689 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.689 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.689 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.689 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.235474+0000", 2024-09-21T12:33:50.689 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.202053832 2024-09-21T12:33:50.689 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.689 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.689 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:50.689 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.235573+0000", 2024-09-21T12:33:50.689 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.9216999999999996e-05 2024-09-21T12:33:50.689 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.690 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.690 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,3,5", 2024-09-21T12:33:50.690 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.235777+0000", 2024-09-21T12:33:50.690 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000203687 2024-09-21T12:33:50.690 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.690 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.690 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.690 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.239048+0000", 2024-09-21T12:33:50.690 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003271173 2024-09-21T12:33:50.690 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.690 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.690 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.690 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.241655+0000", 2024-09-21T12:33:50.690 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002607023 2024-09-21T12:33:50.691 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.691 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.691 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:50.691 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.242308+0000", 2024-09-21T12:33:50.691 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00065309099999999996 2024-09-21T12:33:50.691 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.691 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.691 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.691 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.245732+0000", 2024-09-21T12:33:50.691 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003423461 2024-09-21T12:33:50.691 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.691 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.691 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:50.691 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.245786+0000", 2024-09-21T12:33:50.692 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4289999999999997e-05 2024-09-21T12:33:50.692 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.692 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.692 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:50.692 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.245904+0000", 2024-09-21T12:33:50.692 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011838800000000001 2024-09-21T12:33:50.692 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.692 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:50.692 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.692 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.692 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.692 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231761 9.7 9:e362f0a9:::benchmark_data_smithi114_133214_object107727:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:33:50.692 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029967+0000", 2024-09-21T12:33:50.692 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 120.48463548700001, 2024-09-21T12:33:50.692 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.217986754, 2024-09-21T12:33:50.693 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:50.693 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:50.693 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:50.693 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:50.693 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:33:50.693 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:50.693 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231761 2024-09-21T12:33:50.693 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.693 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:50.693 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.693 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:50.693 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029967+0000", 2024-09-21T12:33:50.693 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.693 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.694 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.694 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:50.694 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029967+0000", 2024-09-21T12:33:50.694 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.694 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.694 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.694 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:50.694 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029967+0000", 2024-09-21T12:33:50.694 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4900000000000001e-07 2024-09-21T12:33:50.694 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.694 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.694 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:50.694 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029969+0000", 2024-09-21T12:33:50.694 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.68e-06 2024-09-21T12:33:50.695 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.695 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.695 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:50.695 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029974+0000", 2024-09-21T12:33:50.695 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5899999999999998e-06 2024-09-21T12:33:50.695 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.695 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.695 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:50.695 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029979+0000", 2024-09-21T12:33:50.695 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6530000000000002e-06 2024-09-21T12:33:50.695 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.695 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.695 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.695 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033442+0000", 2024-09-21T12:33:50.696 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0034634930000000002 2024-09-21T12:33:50.696 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.696 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.696 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:50.696 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033449+0000", 2024-09-21T12:33:50.696 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9290000000000001e-06 2024-09-21T12:33:50.696 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.696 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.696 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.696 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.237181+0000", 2024-09-21T12:33:50.696 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2037321190000001 2024-09-21T12:33:50.696 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.696 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.696 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:50.696 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.237313+0000", 2024-09-21T12:33:50.697 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013190700000000001 2024-09-21T12:33:50.697 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.697 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.697 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,3,5", 2024-09-21T12:33:50.697 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.237544+0000", 2024-09-21T12:33:50.697 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00023036199999999999 2024-09-21T12:33:50.697 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.697 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.697 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.697 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.241579+0000", 2024-09-21T12:33:50.697 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.004035294 2024-09-21T12:33:50.697 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.697 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.697 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:50.698 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.242313+0000", 2024-09-21T12:33:50.698 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00073439600000000001 2024-09-21T12:33:50.698 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.698 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.698 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.698 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.245175+0000", 2024-09-21T12:33:50.698 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0028617299999999998 2024-09-21T12:33:50.698 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.699 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.699 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.699 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.247800+0000", 2024-09-21T12:33:50.699 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002624548 2024-09-21T12:33:50.699 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.699 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.699 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:50.699 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.247850+0000", 2024-09-21T12:33:50.699 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0158999999999999e-05 2024-09-21T12:33:50.699 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.699 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.699 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:50.699 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.247953+0000", 2024-09-21T12:33:50.699 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000103545 2024-09-21T12:33:50.700 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.700 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:50.700 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.700 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.700 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.700 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231982 9.14 9:2d540493:::benchmark_data_smithi114_133214_object107948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-09-21T12:33:50.700 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:52.048929+0000", 2024-09-21T12:33:50.700 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 118.46567335899999, 2024-09-21T12:33:50.700 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99998082899999996, 2024-09-21T12:33:50.700 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:50.700 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:50.700 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:50.701 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:50.701 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:33:50.701 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:50.701 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231982 2024-09-21T12:33:50.701 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.701 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:50.701 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.701 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:50.701 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.048929+0000", 2024-09-21T12:33:50.701 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.701 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.701 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.701 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:50.702 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.048929+0000", 2024-09-21T12:33:50.702 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.702 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.702 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.702 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:50.702 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.048931+0000", 2024-09-21T12:33:50.702 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.6589999999999999e-06 2024-09-21T12:33:50.702 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.702 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.702 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:50.702 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.048936+0000", 2024-09-21T12:33:50.702 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0869999999999997e-06 2024-09-21T12:33:50.702 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.702 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.703 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:50.703 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.048950+0000", 2024-09-21T12:33:50.703 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3976e-05 2024-09-21T12:33:50.703 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.703 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.703 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:50.703 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.048962+0000", 2024-09-21T12:33:50.703 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1225e-05 2024-09-21T12:33:50.703 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.703 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.703 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.703 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.049061+0000", 2024-09-21T12:33:50.703 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.9086999999999995e-05 2024-09-21T12:33:50.704 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.704 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.704 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:50.704 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.049068+0000", 2024-09-21T12:33:50.704 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6539999999999999e-06 2024-09-21T12:33:50.704 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.704 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.704 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.704 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.045781+0000", 2024-09-21T12:33:50.704 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99671266999999997 2024-09-21T12:33:50.704 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.704 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.704 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:50.704 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.045859+0000", 2024-09-21T12:33:50.705 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7891000000000004e-05 2024-09-21T12:33:50.705 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.705 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.705 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,5", 2024-09-21T12:33:50.705 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.046118+0000", 2024-09-21T12:33:50.705 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00025944599999999998 2024-09-21T12:33:50.705 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.705 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.705 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.705 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.048757+0000", 2024-09-21T12:33:50.705 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0026386069999999998 2024-09-21T12:33:50.705 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.705 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.706 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:50.706 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.048771+0000", 2024-09-21T12:33:50.706 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4440999999999999e-05 2024-09-21T12:33:50.706 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.706 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.706 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.706 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.048824+0000", 2024-09-21T12:33:50.706 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2460999999999998e-05 2024-09-21T12:33:50.706 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.706 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.706 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:50.706 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.048880+0000", 2024-09-21T12:33:50.706 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6067999999999999e-05 2024-09-21T12:33:50.707 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.707 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.707 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:50.707 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.048909+0000", 2024-09-21T12:33:50.707 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.9556999999999998e-05 2024-09-21T12:33:50.707 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.707 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:50.707 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.707 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.707 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.707 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231987 9.14 9:2dd07ba6:::benchmark_data_smithi114_133214_object107953:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-09-21T12:33:50.707 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:52.052061+0000", 2024-09-21T12:33:50.707 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 118.46254137699999, 2024-09-21T12:33:50.707 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99820888500000005, 2024-09-21T12:33:50.707 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:50.708 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:50.708 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:50.708 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:50.708 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:33:50.708 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:50.708 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231987 2024-09-21T12:33:50.708 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.708 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:50.708 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.708 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:50.708 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.052061+0000", 2024-09-21T12:33:50.708 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.708 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.708 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.709 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:50.709 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.052061+0000", 2024-09-21T12:33:50.709 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.709 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.709 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.709 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:50.709 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.052062+0000", 2024-09-21T12:33:50.709 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.026e-06 2024-09-21T12:33:50.709 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.709 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.709 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:50.709 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.052479+0000", 2024-09-21T12:33:50.710 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00041693600000000002 2024-09-21T12:33:50.710 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.710 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.710 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:50.710 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.052490+0000", 2024-09-21T12:33:50.710 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1270000000000001e-05 2024-09-21T12:33:50.710 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.710 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.710 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:50.710 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.052496+0000", 2024-09-21T12:33:50.710 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4330000000000001e-06 2024-09-21T12:33:50.710 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.710 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.710 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.710 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.052654+0000", 2024-09-21T12:33:50.711 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015731400000000001 2024-09-21T12:33:50.711 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.711 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.711 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:50.711 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.052663+0000", 2024-09-21T12:33:50.711 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9760000000000001e-06 2024-09-21T12:33:50.711 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.711 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.711 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.711 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.047676+0000", 2024-09-21T12:33:50.711 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99501326700000003 2024-09-21T12:33:50.711 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.711 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.711 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:50.712 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.047793+0000", 2024-09-21T12:33:50.712 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000117099 2024-09-21T12:33:50.712 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.712 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.712 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,5", 2024-09-21T12:33:50.712 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.047970+0000", 2024-09-21T12:33:50.712 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017714599999999999 2024-09-21T12:33:50.712 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.712 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.712 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.712 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.050154+0000", 2024-09-21T12:33:50.713 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0021837110000000001 2024-09-21T12:33:50.713 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.713 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.713 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:50.713 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.050164+0000", 2024-09-21T12:33:50.713 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0156000000000001e-05 2024-09-21T12:33:50.713 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.713 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.713 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.713 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.050195+0000", 2024-09-21T12:33:50.713 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.0966999999999997e-05 2024-09-21T12:33:50.713 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.713 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.714 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:50.714 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.050237+0000", 2024-09-21T12:33:50.714 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1909999999999997e-05 2024-09-21T12:33:50.714 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.714 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.714 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:50.714 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.050270+0000", 2024-09-21T12:33:50.714 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2673999999999999e-05 2024-09-21T12:33:50.714 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.714 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:50.714 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.714 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.714 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.715 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:232023 9.14 9:2a2d427f:::benchmark_data_smithi114_133214_object107989:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-09-21T12:33:50.715 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:52.084360+0000", 2024-09-21T12:33:50.715 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 118.430242019, 2024-09-21T12:33:50.715 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.96746416199999996, 2024-09-21T12:33:50.715 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:33:50.715 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:33:50.715 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:33:50.715 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:33:50.715 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:33:50.715 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:33:50.715 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 232023 2024-09-21T12:33:50.715 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.715 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:33:50.715 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.716 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:33:50.716 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.084360+0000", 2024-09-21T12:33:50.716 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.716 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.716 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.716 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:33:50.716 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.084360+0000", 2024-09-21T12:33:50.716 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:33:50.716 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.716 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.716 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:33:50.716 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.084361+0000", 2024-09-21T12:33:50.716 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0589999999999999e-06 2024-09-21T12:33:50.716 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.716 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.717 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:33:50.717 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.084364+0000", 2024-09-21T12:33:50.717 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.3050000000000001e-06 2024-09-21T12:33:50.717 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.717 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.717 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:33:50.717 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.084373+0000", 2024-09-21T12:33:50.717 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.2269999999999993e-06 2024-09-21T12:33:50.717 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.717 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.717 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:33:50.717 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.084378+0000", 2024-09-21T12:33:50.717 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7400000000000001e-06 2024-09-21T12:33:50.717 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.717 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.718 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.718 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.084461+0000", 2024-09-21T12:33:50.718 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.2650000000000003e-05 2024-09-21T12:33:50.718 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.718 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.718 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:33:50.718 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.084469+0000", 2024-09-21T12:33:50.718 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.695e-06 2024-09-21T12:33:50.718 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.718 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.718 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:33:50.718 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.048948+0000", 2024-09-21T12:33:50.718 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.96447884399999995 2024-09-21T12:33:50.718 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.719 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.719 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:33:50.719 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.049009+0000", 2024-09-21T12:33:50.719 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1017999999999998e-05 2024-09-21T12:33:50.719 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.719 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.719 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,5", 2024-09-21T12:33:50.719 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.049163+0000", 2024-09-21T12:33:50.719 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000154375 2024-09-21T12:33:50.719 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.720 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.720 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.720 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.051692+0000", 2024-09-21T12:33:50.720 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002528831 2024-09-21T12:33:50.720 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.720 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.720 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:33:50.720 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.051702+0000", 2024-09-21T12:33:50.720 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0161e-05 2024-09-21T12:33:50.720 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.720 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.720 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:33:50.720 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.051739+0000", 2024-09-21T12:33:50.720 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.6704e-05 2024-09-21T12:33:50.720 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.721 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.721 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:33:50.721 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.051790+0000", 2024-09-21T12:33:50.721 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0979000000000001e-05 2024-09-21T12:33:50.721 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:33:50.721 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:33:50.721 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:33:50.721 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.051824+0000", 2024-09-21T12:33:50.721 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4573999999999998e-05 2024-09-21T12:33:50.721 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.721 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:50.721 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.722 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:33:50.722 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:33:50.722 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:33:50.722 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 74 16 107943 107927 91.1282 72.9375 0.0111374 0.010969 2024-09-21T12:33:50.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:50.723 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:50.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:50.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:50 smithi183 ceph-mon[33277]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:33:50.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:50 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:33:50.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:50 smithi183 ceph-mon[33277]: osdmap e379: 8 total, 8 up, 8 in 2024-09-21T12:33:50.825 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:50.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:50.867 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:50.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:50.968 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:50.997 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:50.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:50.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:51.099 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:51.127 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:51.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:51.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:51.228 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:51.259 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:51.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:51.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:51.361 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:51.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:51.403 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:51.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:51.470 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:33:51.471 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:33:51.504 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:51.514 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:33:51.514 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:33:51.530 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:51.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:51.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:51.559 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:33:51.559 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:33:51.602 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:33:51.602 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:33:51.631 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:51.637 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:33:51.637 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:33:51.668 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 75 16 109191 109175 90.9522 78 0.0212942 0.0109899 2024-09-21T12:33:51.669 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:51.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:51.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:51.671 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:51 smithi114 ceph-mon[30324]: pgmap v699: 178 pgs: 178 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-09-21T12:33:51.672 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:51 smithi114 ceph-mon[40867]: pgmap v699: 178 pgs: 178 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-09-21T12:33:51.672 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:33:51.672 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:33:51.703 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:33:51.703 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:33:51.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:51 smithi183 ceph-mon[33277]: pgmap v699: 178 pgs: 178 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-09-21T12:33:51.739 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:33:51.740 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:33:51.771 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:51.816 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:51.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:51.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:51.918 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:51.952 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:51.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:51.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:52.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:52.090 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:52.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:52.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:52.192 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:52.226 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:52.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:52.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:52.328 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:52.363 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:52.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:52.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:52.465 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:52.481 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T12:33:52.493 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:52.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:52.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:52.595 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:52.636 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:52.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:52.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:52.668 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 76 16 110462 110446 90.8006 79.4375 0.0113753 0.0110088 2024-09-21T12:33:52.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:52 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3490360432' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:33:52.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:52 smithi183 ceph-mon[33277]: pgmap v700: 178 pgs: 178 active+clean; 6.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.7 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-21T12:33:52.738 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:52.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:52 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3490360432' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:33:52.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:52 smithi114 ceph-mon[30324]: pgmap v700: 178 pgs: 178 active+clean; 6.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.7 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-21T12:33:52.775 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:33:52 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:33:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:33:52.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:52 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3490360432' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:33:52.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:52 smithi114 ceph-mon[40867]: pgmap v700: 178 pgs: 178 active+clean; 6.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.7 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-21T12:33:52.780 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:52.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:52.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:52.883 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:52.912 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:52.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:52.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:53.014 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:53.063 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:53.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:53.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:53.130 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T12:33:53.164 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:53.189 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:53.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:53.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:53.203 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:53.231 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:53.291 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:53.324 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:53.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:53.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:53.426 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:53.480 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:53.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:53.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:53.582 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:53.611 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:53.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:53.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:53.668 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 77 16 111610 111594 90.5531 71.75 0.00642004 0.0110387 2024-09-21T12:33:53.712 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:53.748 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:53.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:53.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:53.850 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:53.880 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:53.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:53.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:53.982 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:54.015 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:54.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:54.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:54.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:54.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:54.154 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:54.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:54.255 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:54.297 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:54.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:54.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:54.399 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:54.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:54.454 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:54.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:54.555 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:54.606 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:54.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:54.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:54.668 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:33:54.672 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:33:54.672 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 78 16 112880 112864 90.4098 79.375 0.0149681 0.0110559 2024-09-21T12:33:54.710 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:54.747 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:54.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:54.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:54.849 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:54.876 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:54.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:54.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:54.905 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:33:54.905 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:33:54.978 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:55.010 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:55.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:55.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:55.112 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:55.148 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:55.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:55.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:55.168 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:33:55.226 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:54 smithi114 ceph-mon[30324]: pgmap v701: 178 pgs: 178 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 780 B/s rd, 79 MiB/s wr, 1.27k op/s; 16 B/s, 1 objects/s recovering 2024-09-21T12:33:55.226 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:54 smithi114 ceph-mon[40867]: pgmap v701: 178 pgs: 178 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 780 B/s rd, 79 MiB/s wr, 1.27k op/s; 16 B/s, 1 objects/s recovering 2024-09-21T12:33:55.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:54 smithi183 ceph-mon[33277]: pgmap v701: 178 pgs: 178 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 780 B/s rd, 79 MiB/s wr, 1.27k op/s; 16 B/s, 1 objects/s recovering 2024-09-21T12:33:55.250 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:55.291 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:55.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:55.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:55.393 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:55.447 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:55.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:55.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:55.549 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:55.592 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:55.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:55.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:55.668 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 79 16 114135 114119 90.2582 78.4375 0.00427387 0.0110749 2024-09-21T12:33:55.694 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:55.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:55.725 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:55.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:55.826 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:55.865 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:55.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:55.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:55.966 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:56.003 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:56.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:56.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:56.066 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":701,"stamp":"2024-09-21T12:33:53.856928+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7203868292,"num_objects":110112,"num_object_clones":0,"num_object_copies":330336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":110112,"num_whiteouts":0,"num_read":1942,"num_read_kb":1697,"num_write":110637,"num_write_kb":7034514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1454,"num_bytes_recovered":67327497,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78452,"ondisk_log_size":78452,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":534,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":25501248,"kb_used_data":20990908,"kb_used_omap":12,"kb_used_meta":4510003,"kb_avail":724459968,"statfs":{"total":767960285184,"available":741847007232,"internally_reserved":0,"allocated":21494689792,"data_stored":21491578825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":4618243632},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,2,2,6,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":546832384,"num_objects":8344,"num_object_clones":0,"num_object_copies":25032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8344,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8344,"num_write_kb":534016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.559826"},"pg_stats":[{"pgid":"1.1d","version":"76'79","reported_seq":380,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.388737+0000","last_change":"2024-09-21T12:32:38.427111+0000","last_active":"2024-09-21T12:33:49.388737+0000","last_peered":"2024-09-21T12:33:49.388737+0000","last_clean":"2024-09-21T12:33:49.388737+0000","last_became_active":"2024-09-21T12:32:38.426383+0000","last_became_peered":"2024-09-21T12:32:38.426383+0000","last_unstale":"2024-09-21T12:33:49.388737+0000","last_undegraded":"2024-09-21T12:33:49.388737+0000","last_fullsized":"2024-09-21T12:33:49.388737+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:32.060116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:32.060116+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":690,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.388843+0000","last_change":"2024-09-21T12:33:29.086898+0000","last_active":"2024-09-21T12:33:49.388843+0000","last_peered":"2024-09-21T12:33:49.388843+0000","last_clean":"2024-09-21T12:33:49.388843+0000","last_became_active":"2024-09-21T12:20:15.309431+0000","last_became_peered":"2024-09-21T12:20:15.309431+0000","last_unstale":"2024-09-21T12:33:49.388843+0000","last_undegraded":"2024-09-21T12:33:49.388843+0000","last_fullsized":"2024-09-21T12:33:49.388843+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:29.086830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:29.086830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:54.486629+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":80,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.374152+0000","last_change":"2024-09-21T12:32:38.443995+0000","last_active":"2024-09-21T12:33:49.374152+0000","last_peered":"2024-09-21T12:33:49.374152+0000","last_clean":"2024-09-21T12:33:49.374152+0000","last_became_active":"2024-09-21T12:32:38.419081+0000","last_became_peered":"2024-09-21T12:32:38.419081+0000","last_unstale":"2024-09-21T12:33:49.374152+0000","last_undegraded":"2024-09-21T12:33:49.374152+0000","last_fullsized":"2024-09-21T12:33:49.374152+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:29:08.671723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:29:08.671723+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:20.264381+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":671,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.387304+0000","last_change":"2024-09-21T12:29:10.672659+0000","last_active":"2024-09-21T12:33:49.387304+0000","last_peered":"2024-09-21T12:33:49.387304+0000","last_clean":"2024-09-21T12:33:49.387304+0000","last_became_active":"2024-09-21T12:20:19.314908+0000","last_became_peered":"2024-09-21T12:20:19.314908+0000","last_unstale":"2024-09-21T12:33:49.387304+0000","last_undegraded":"2024-09-21T12:33:49.387304+0000","last_fullsized":"2024-09-21T12:33:49.387304+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:10.672599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:29:10.672599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":78,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.375164+0000","last_change":"2024-09-21T12:33:27.411258+0000","last_active":"2024-09-21T12:33:49.375164+0000","last_peered":"2024-09-21T12:33:49.375164+0000","last_clean":"2024-09-21T12:33:49.375164+0000","last_became_active":"2024-09-21T12:32:38.425588+0000","last_became_peered":"2024-09-21T12:32:38.425588+0000","last_unstale":"2024-09-21T12:33:49.375164+0000","last_undegraded":"2024-09-21T12:33:49.375164+0000","last_fullsized":"2024-09-21T12:33:49.375164+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:27.411142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:27.411142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:51.940910+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"76'79","reported_seq":341,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.384863+0000","last_change":"2024-09-21T12:27:28.374689+0000","last_active":"2024-09-21T12:33:49.384863+0000","last_peered":"2024-09-21T12:33:49.384863+0000","last_clean":"2024-09-21T12:33:49.384863+0000","last_became_active":"2024-09-21T12:27:16.920545+0000","last_became_peered":"2024-09-21T12:27:16.920545+0000","last_unstale":"2024-09-21T12:33:49.384863+0000","last_undegraded":"2024-09-21T12:33:49.384863+0000","last_fullsized":"2024-09-21T12:33:49.384863+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:28.374616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:28.374616+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:13.402077+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":658,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.390649+0000","last_change":"2024-09-21T12:32:17.528064+0000","last_active":"2024-09-21T12:33:49.390649+0000","last_peered":"2024-09-21T12:33:49.390649+0000","last_clean":"2024-09-21T12:33:49.390649+0000","last_became_active":"2024-09-21T12:20:21.595301+0000","last_became_peered":"2024-09-21T12:20:21.595301+0000","last_unstale":"2024-09-21T12:33:49.390649+0000","last_undegraded":"2024-09-21T12:33:49.390649+0000","last_fullsized":"2024-09-21T12:33:49.390649+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:17.527992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:32:17.527992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":687,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.389585+0000","last_change":"2024-09-21T12:33:06.699081+0000","last_active":"2024-09-21T12:33:49.389585+0000","last_peered":"2024-09-21T12:33:49.389585+0000","last_clean":"2024-09-21T12:33:49.389585+0000","last_became_active":"2024-09-21T12:20:15.308655+0000","last_became_peered":"2024-09-21T12:20:15.308655+0000","last_unstale":"2024-09-21T12:33:49.389585+0000","last_undegraded":"2024-09-21T12:33:49.389585+0000","last_fullsized":"2024-09-21T12:33:49.389585+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:06.698999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:06.698999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:22.244932+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":725,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.389869+0000","last_change":"2024-09-21T12:31:48.294386+0000","last_active":"2024-09-21T12:33:49.389869+0000","last_peered":"2024-09-21T12:33:49.389869+0000","last_clean":"2024-09-21T12:33:49.389869+0000","last_became_active":"2024-09-21T12:20:17.305220+0000","last_became_peered":"2024-09-21T12:20:17.305220+0000","last_unstale":"2024-09-21T12:33:49.389869+0000","last_undegraded":"2024-09-21T12:33:49.389869+0000","last_fullsized":"2024-09-21T12:33:49.389869+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:31:48.294300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:48.294300+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:40.758527+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":70,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.374102+0000","last_change":"2024-09-21T12:32:38.418041+0000","last_active":"2024-09-21T12:33:49.374102+0000","last_peered":"2024-09-21T12:33:49.374102+0000","last_clean":"2024-09-21T12:33:49.374102+0000","last_became_active":"2024-09-21T12:32:38.417724+0000","last_became_peered":"2024-09-21T12:32:38.417724+0000","last_unstale":"2024-09-21T12:33:49.374102+0000","last_undegraded":"2024-09-21T12:33:49.374102+0000","last_fullsized":"2024-09-21T12:33:49.374102+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:39.179665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:39.179665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:10.351688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":78,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.376289+0000","last_change":"2024-09-21T12:33:32.373658+0000","last_active":"2024-09-21T12:33:49.376289+0000","last_peered":"2024-09-21T12:33:49.376289+0000","last_clean":"2024-09-21T12:33:49.376289+0000","last_became_active":"2024-09-21T12:32:38.428077+0000","last_became_peered":"2024-09-21T12:32:38.428077+0000","last_unstale":"2024-09-21T12:33:49.376289+0000","last_undegraded":"2024-09-21T12:33:49.376289+0000","last_fullsized":"2024-09-21T12:33:49.376289+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:32.372832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:32.372832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:56.425700+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":88,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.375170+0000","last_change":"2024-09-21T12:33:03.312402+0000","last_active":"2024-09-21T12:33:49.375170+0000","last_peered":"2024-09-21T12:33:49.375170+0000","last_clean":"2024-09-21T12:33:49.375170+0000","last_became_active":"2024-09-21T12:32:38.426121+0000","last_became_peered":"2024-09-21T12:32:38.426121+0000","last_unstale":"2024-09-21T12:33:49.375170+0000","last_undegraded":"2024-09-21T12:33:49.375170+0000","last_fullsized":"2024-09-21T12:33:49.375170+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:33:03.312251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:03.312251+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:32.603832+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":709,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.393692+0000","last_change":"2024-09-21T12:32:14.238277+0000","last_active":"2024-09-21T12:33:49.393692+0000","last_peered":"2024-09-21T12:33:49.393692+0000","last_clean":"2024-09-21T12:33:49.393692+0000","last_became_active":"2024-09-21T12:32:14.237803+0000","last_became_peered":"2024-09-21T12:32:14.237803+0000","last_unstale":"2024-09-21T12:33:49.393692+0000","last_undegraded":"2024-09-21T12:33:49.393692+0000","last_fullsized":"2024-09-21T12:33:49.393692+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:56.620241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:31:56.620241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:09.306639+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":669,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.382994+0000","last_change":"2024-09-21T12:27:54.959865+0000","last_active":"2024-09-21T12:33:49.382994+0000","last_peered":"2024-09-21T12:33:49.382994+0000","last_clean":"2024-09-21T12:33:49.382994+0000","last_became_active":"2024-09-21T12:20:21.309190+0000","last_became_peered":"2024-09-21T12:20:21.309190+0000","last_unstale":"2024-09-21T12:33:49.382994+0000","last_undegraded":"2024-09-21T12:33:49.382994+0000","last_fullsized":"2024-09-21T12:33:49.382994+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:27:54.959806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:27:54.959806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:15.009786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"76'79","reported_seq":451,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.389398+0000","last_change":"2024-09-21T12:33:36.752640+0000","last_active":"2024-09-21T12:33:49.389398+0000","last_peered":"2024-09-21T12:33:49.389398+0000","last_clean":"2024-09-21T12:33:49.389398+0000","last_became_active":"2024-09-21T12:27:16.937711+0000","last_became_peered":"2024-09-21T12:27:16.937711+0000","last_unstale":"2024-09-21T12:33:49.389398+0000","last_undegraded":"2024-09-21T12:33:49.389398+0000","last_fullsized":"2024-09-21T12:33:49.389398+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:36.752562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:36.752562+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:38.559129+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":70,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.375028+0000","last_change":"2024-09-21T12:32:38.426448+0000","last_active":"2024-09-21T12:33:49.375028+0000","last_peered":"2024-09-21T12:33:49.375028+0000","last_clean":"2024-09-21T12:33:49.375028+0000","last_became_active":"2024-09-21T12:32:38.426142+0000","last_became_peered":"2024-09-21T12:32:38.426142+0000","last_unstale":"2024-09-21T12:33:49.375028+0000","last_undegraded":"2024-09-21T12:33:49.375028+0000","last_fullsized":"2024-09-21T12:33:49.375028+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:29.101141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:29.101141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":84,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.374780+0000","last_change":"2024-09-21T12:33:26.395059+0000","last_active":"2024-09-21T12:33:49.374780+0000","last_peered":"2024-09-21T12:33:49.374780+0000","last_clean":"2024-09-21T12:33:49.374780+0000","last_became_active":"2024-09-21T12:32:38.422912+0000","last_became_peered":"2024-09-21T12:32:38.422912+0000","last_unstale":"2024-09-21T12:33:49.374780+0000","last_undegraded":"2024-09-21T12:33:49.374780+0000","last_fullsized":"2024-09-21T12:33:49.374780+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'7","last_scrub_stamp":"2024-09-21T12:33:26.394982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:26.394982+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:55.174938+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":699,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.383543+0000","last_change":"2024-09-21T12:32:38.419413+0000","last_active":"2024-09-21T12:33:49.383543+0000","last_peered":"2024-09-21T12:33:49.383543+0000","last_clean":"2024-09-21T12:33:49.383543+0000","last_became_active":"2024-09-21T12:32:38.417472+0000","last_became_peered":"2024-09-21T12:32:38.417472+0000","last_unstale":"2024-09-21T12:33:49.383543+0000","last_undegraded":"2024-09-21T12:33:49.383543+0000","last_fullsized":"2024-09-21T12:33:49.383543+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:30.251514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:32:30.251514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":691,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.393300+0000","last_change":"2024-09-21T12:33:36.125346+0000","last_active":"2024-09-21T12:33:49.393300+0000","last_peered":"2024-09-21T12:33:49.393300+0000","last_clean":"2024-09-21T12:33:49.393300+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:33:49.393300+0000","last_undegraded":"2024-09-21T12:33:49.393300+0000","last_fullsized":"2024-09-21T12:33:49.393300+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:36.124095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:36.124095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:59.725363+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"76'79","reported_seq":505,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.390451+0000","last_change":"2024-09-21T12:27:22.289065+0000","last_active":"2024-09-21T12:33:49.390451+0000","last_peered":"2024-09-21T12:33:49.390451+0000","last_clean":"2024-09-21T12:33:49.390451+0000","last_became_active":"2024-09-21T12:27:16.957569+0000","last_became_peered":"2024-09-21T12:27:16.957569+0000","last_unstale":"2024-09-21T12:33:49.390451+0000","last_undegraded":"2024-09-21T12:33:49.390451+0000","last_fullsized":"2024-09-21T12:33:49.390451+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:22.288989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:22.288989+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:42.764336+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":70,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.377735+0000","last_change":"2024-09-21T12:32:38.429316+0000","last_active":"2024-09-21T12:33:49.377735+0000","last_peered":"2024-09-21T12:33:49.377735+0000","last_clean":"2024-09-21T12:33:49.377735+0000","last_became_active":"2024-09-21T12:32:38.428997+0000","last_became_peered":"2024-09-21T12:32:38.428997+0000","last_unstale":"2024-09-21T12:33:49.377735+0000","last_undegraded":"2024-09-21T12:33:49.377735+0000","last_fullsized":"2024-09-21T12:33:49.377735+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:31.581641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:32:31.581641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:49.768565+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":739,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.382603+0000","last_change":"2024-09-21T12:32:56.016897+0000","last_active":"2024-09-21T12:33:49.382603+0000","last_peered":"2024-09-21T12:33:49.382603+0000","last_clean":"2024-09-21T12:33:49.382603+0000","last_became_active":"2024-09-21T12:32:38.410246+0000","last_became_peered":"2024-09-21T12:32:38.410246+0000","last_unstale":"2024-09-21T12:33:49.382603+0000","last_undegraded":"2024-09-21T12:33:49.382603+0000","last_fullsized":"2024-09-21T12:33:49.382603+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:56.016840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:56.016840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:12.575751+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":758,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.393598+0000","last_change":"2024-09-21T12:33:48.390232+0000","last_active":"2024-09-21T12:33:49.393598+0000","last_peered":"2024-09-21T12:33:49.393598+0000","last_clean":"2024-09-21T12:33:49.393598+0000","last_became_active":"2024-09-21T12:33:48.333872+0000","last_became_peered":"2024-09-21T12:33:48.333872+0000","last_unstale":"2024-09-21T12:33:49.393598+0000","last_undegraded":"2024-09-21T12:33:49.393598+0000","last_fullsized":"2024-09-21T12:33:49.393598+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:33:01.668584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:01.668584+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:26.784149+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":706,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.389026+0000","last_change":"2024-09-21T12:32:38.426714+0000","last_active":"2024-09-21T12:33:49.389026+0000","last_peered":"2024-09-21T12:33:49.389026+0000","last_clean":"2024-09-21T12:33:49.389026+0000","last_became_active":"2024-09-21T12:32:38.425958+0000","last_became_peered":"2024-09-21T12:32:38.425958+0000","last_unstale":"2024-09-21T12:33:49.389026+0000","last_undegraded":"2024-09-21T12:33:49.389026+0000","last_fullsized":"2024-09-21T12:33:49.389026+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:06.636729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:06.636729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:48.229340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"76'79","reported_seq":622,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.393386+0000","last_change":"2024-09-21T12:33:27.294757+0000","last_active":"2024-09-21T12:33:49.393386+0000","last_peered":"2024-09-21T12:33:49.393386+0000","last_clean":"2024-09-21T12:33:49.393386+0000","last_became_active":"2024-09-21T12:27:16.927791+0000","last_became_peered":"2024-09-21T12:27:16.927791+0000","last_unstale":"2024-09-21T12:33:49.393386+0000","last_undegraded":"2024-09-21T12:33:49.393386+0000","last_fullsized":"2024-09-21T12:33:49.393386+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:27.294674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:27.294674+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:30.531743+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":688,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.390652+0000","last_change":"2024-09-21T12:32:53.765704+0000","last_active":"2024-09-21T12:33:49.390652+0000","last_peered":"2024-09-21T12:33:49.390652+0000","last_clean":"2024-09-21T12:33:49.390652+0000","last_became_active":"2024-09-21T12:32:02.033276+0000","last_became_peered":"2024-09-21T12:32:02.033276+0000","last_unstale":"2024-09-21T12:33:49.390652+0000","last_undegraded":"2024-09-21T12:33:49.390652+0000","last_fullsized":"2024-09-21T12:33:49.390652+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:53.765638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:32:53.765638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:54.321629+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":70,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.374578+0000","last_change":"2024-09-21T12:32:38.424504+0000","last_active":"2024-09-21T12:33:49.374578+0000","last_peered":"2024-09-21T12:33:49.374578+0000","last_clean":"2024-09-21T12:33:49.374578+0000","last_became_active":"2024-09-21T12:32:38.424217+0000","last_became_peered":"2024-09-21T12:32:38.424217+0000","last_unstale":"2024-09-21T12:33:49.374578+0000","last_undegraded":"2024-09-21T12:33:49.374578+0000","last_fullsized":"2024-09-21T12:33:49.374578+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:45.725968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.725968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:17.243492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":782,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.391193+0000","last_change":"2024-09-21T12:32:38.437419+0000","last_active":"2024-09-21T12:33:49.391193+0000","last_peered":"2024-09-21T12:33:49.391193+0000","last_clean":"2024-09-21T12:33:49.391193+0000","last_became_active":"2024-09-21T12:32:38.409163+0000","last_became_peered":"2024-09-21T12:32:38.409163+0000","last_unstale":"2024-09-21T12:33:49.391193+0000","last_undegraded":"2024-09-21T12:33:49.391193+0000","last_fullsized":"2024-09-21T12:33:49.391193+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:28:30.619954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:30.619954+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":78,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.374880+0000","last_change":"2024-09-21T12:32:53.270679+0000","last_active":"2024-09-21T12:33:49.374880+0000","last_peered":"2024-09-21T12:33:49.374880+0000","last_clean":"2024-09-21T12:33:49.374880+0000","last_became_active":"2024-09-21T12:32:38.419143+0000","last_became_peered":"2024-09-21T12:32:38.419143+0000","last_unstale":"2024-09-21T12:33:49.374880+0000","last_undegraded":"2024-09-21T12:33:49.374880+0000","last_fullsized":"2024-09-21T12:33:49.374880+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:53.270634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:32:53.270634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:07.052948+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"76'79","reported_seq":421,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.393521+0000","last_change":"2024-09-21T12:32:38.414543+0000","last_active":"2024-09-21T12:33:49.393521+0000","last_peered":"2024-09-21T12:33:49.393521+0000","last_clean":"2024-09-21T12:33:49.393521+0000","last_became_active":"2024-09-21T12:32:38.414204+0000","last_became_peered":"2024-09-21T12:32:38.414204+0000","last_unstale":"2024-09-21T12:33:49.393521+0000","last_undegraded":"2024-09-21T12:33:49.393521+0000","last_fullsized":"2024-09-21T12:33:49.393521+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:31.720475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:31.720475+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:45.094510+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":660,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.384777+0000","last_change":"2024-09-21T12:33:37.344930+0000","last_active":"2024-09-21T12:33:49.384777+0000","last_peered":"2024-09-21T12:33:49.384777+0000","last_clean":"2024-09-21T12:33:49.384777+0000","last_became_active":"2024-09-21T12:20:21.312976+0000","last_became_peered":"2024-09-21T12:20:21.312976+0000","last_unstale":"2024-09-21T12:33:49.384777+0000","last_undegraded":"2024-09-21T12:33:49.384777+0000","last_fullsized":"2024-09-21T12:33:49.384777+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:33:37.344821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:37.344821+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:06.688452+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":724,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.387502+0000","last_change":"2024-09-21T12:32:38.441487+0000","last_active":"2024-09-21T12:33:49.387502+0000","last_peered":"2024-09-21T12:33:49.387502+0000","last_clean":"2024-09-21T12:33:49.387502+0000","last_became_active":"2024-09-21T12:32:38.425335+0000","last_became_peered":"2024-09-21T12:32:38.425335+0000","last_unstale":"2024-09-21T12:33:49.387502+0000","last_undegraded":"2024-09-21T12:33:49.387502+0000","last_fullsized":"2024-09-21T12:33:49.387502+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:27:56.597473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:27:56.597473+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:42.251918+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":770,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.385923+0000","last_change":"2024-09-21T12:32:57.082602+0000","last_active":"2024-09-21T12:33:49.385923+0000","last_peered":"2024-09-21T12:33:49.385923+0000","last_clean":"2024-09-21T12:33:49.385923+0000","last_became_active":"2024-09-21T12:32:14.238429+0000","last_became_peered":"2024-09-21T12:32:14.238429+0000","last_unstale":"2024-09-21T12:33:49.385923+0000","last_undegraded":"2024-09-21T12:33:49.385923+0000","last_fullsized":"2024-09-21T12:33:49.385923+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:32:57.082525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:57.082525+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:16.381207+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"352'15","reported_seq":89,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.376424+0000","last_change":"2024-09-21T12:32:38.459772+0000","last_active":"2024-09-21T12:33:49.376424+0000","last_peered":"2024-09-21T12:33:49.376424+0000","last_clean":"2024-09-21T12:33:49.376424+0000","last_became_active":"2024-09-21T12:32:38.432386+0000","last_became_peered":"2024-09-21T12:32:38.432386+0000","last_unstale":"2024-09-21T12:33:49.376424+0000","last_undegraded":"2024-09-21T12:33:49.376424+0000","last_fullsized":"2024-09-21T12:33:49.376424+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"335'14","last_scrub_stamp":"2024-09-21T12:32:17.332448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:32:17.332448+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:39.320931+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"379'6788","reported_seq":7152,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:51.599202+0000","last_change":"2024-09-21T12:32:41.411402+0000","last_active":"2024-09-21T12:33:51.599202+0000","last_peered":"2024-09-21T12:33:51.599202+0000","last_clean":"2024-09-21T12:33:51.599202+0000","last_became_active":"2024-09-21T12:32:41.394385+0000","last_became_peered":"2024-09-21T12:32:41.394385+0000","last_unstale":"2024-09-21T12:33:51.599202+0000","last_undegraded":"2024-09-21T12:33:51.599202+0000","last_fullsized":"2024-09-21T12:33:51.599202+0000","mapping_epoch":352,"log_start":"359'2800","ondisk_log_start":"359'2800","created":344,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:32:29.279916+0000","objects_scrubbed":0,"log_size":3988,"log_dups_size":0,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444858368,"num_objects":6788,"num_object_clones":0,"num_object_copies":20364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6788,"num_write_kb":434432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":8847360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"76'79","reported_seq":390,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.394566+0000","last_change":"2024-09-21T12:28:29.446170+0000","last_active":"2024-09-21T12:33:49.394566+0000","last_peered":"2024-09-21T12:33:49.394566+0000","last_clean":"2024-09-21T12:33:49.394566+0000","last_became_active":"2024-09-21T12:27:14.943286+0000","last_became_peered":"2024-09-21T12:27:14.943286+0000","last_unstale":"2024-09-21T12:33:49.394566+0000","last_undegraded":"2024-09-21T12:33:49.394566+0000","last_fullsized":"2024-09-21T12:33:49.394566+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:28:29.446103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:28:29.446103+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:11.576750+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":580,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.396151+0000","last_change":"2024-09-21T12:32:38.444104+0000","last_active":"2024-09-21T12:33:49.396151+0000","last_peered":"2024-09-21T12:33:49.396151+0000","last_clean":"2024-09-21T12:33:49.396151+0000","last_became_active":"2024-09-21T12:32:38.414030+0000","last_became_peered":"2024-09-21T12:32:38.414030+0000","last_unstale":"2024-09-21T12:33:49.396151+0000","last_undegraded":"2024-09-21T12:33:49.396151+0000","last_fullsized":"2024-09-21T12:33:49.396151+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:31:56.722241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:56.722241+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:14.519526+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":146,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.386490+0000","last_change":"2024-09-21T12:33:05.703536+0000","last_active":"2024-09-21T12:33:49.386490+0000","last_peered":"2024-09-21T12:33:49.386490+0000","last_clean":"2024-09-21T12:33:49.386490+0000","last_became_active":"2024-09-21T12:31:50.039390+0000","last_became_peered":"2024-09-21T12:31:50.039390+0000","last_unstale":"2024-09-21T12:33:49.386490+0000","last_undegraded":"2024-09-21T12:33:49.386490+0000","last_fullsized":"2024-09-21T12:33:49.386490+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:05.703464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:05.703464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:22.763784+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"352'15","reported_seq":89,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.377738+0000","last_change":"2024-09-21T12:32:38.458099+0000","last_active":"2024-09-21T12:33:49.377738+0000","last_peered":"2024-09-21T12:33:49.377738+0000","last_clean":"2024-09-21T12:33:49.377738+0000","last_became_active":"2024-09-21T12:32:38.429092+0000","last_became_peered":"2024-09-21T12:32:38.429092+0000","last_unstale":"2024-09-21T12:33:49.377738+0000","last_undegraded":"2024-09-21T12:33:49.377738+0000","last_fullsized":"2024-09-21T12:33:49.377738+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'13","last_scrub_stamp":"2024-09-21T12:25:33.474391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:33.474391+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:48.191356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":158,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.388923+0000","last_change":"2024-09-21T12:31:39.062941+0000","last_active":"2024-09-21T12:33:49.388923+0000","last_peered":"2024-09-21T12:33:49.388923+0000","last_clean":"2024-09-21T12:33:49.388923+0000","last_became_active":"2024-09-21T12:31:39.054600+0000","last_became_peered":"2024-09-21T12:31:39.054600+0000","last_unstale":"2024-09-21T12:33:49.388923+0000","last_undegraded":"2024-09-21T12:33:49.388923+0000","last_fullsized":"2024-09-21T12:33:49.388923+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:28:31.580245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:31.580245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:09.537988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.e","version":"379'6926","reported_seq":7244,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:51.717461+0000","last_change":"2024-09-21T12:32:41.396233+0000","last_active":"2024-09-21T12:33:51.717461+0000","last_peered":"2024-09-21T12:33:51.717461+0000","last_clean":"2024-09-21T12:33:51.717461+0000","last_became_active":"2024-09-21T12:32:41.393651+0000","last_became_peered":"2024-09-21T12:32:41.393651+0000","last_unstale":"2024-09-21T12:33:51.717461+0000","last_undegraded":"2024-09-21T12:33:51.717461+0000","last_fullsized":"2024-09-21T12:33:51.717461+0000","mapping_epoch":352,"log_start":"360'3000","ondisk_log_start":"360'3000","created":344,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:32:29.279916+0000","objects_scrubbed":0,"log_size":3926,"log_dups_size":0,"ondisk_log_size":3926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:53.568163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453902336,"num_objects":6926,"num_object_clones":0,"num_object_copies":20778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6926,"num_write_kb":443264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":7405568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"76'79","reported_seq":78,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.375951+0000","last_change":"2024-09-21T12:33:00.343986+0000","last_active":"2024-09-21T12:33:49.375951+0000","last_peered":"2024-09-21T12:33:49.375951+0000","last_clean":"2024-09-21T12:33:49.375951+0000","last_became_active":"2024-09-21T12:32:38.420379+0000","last_became_peered":"2024-09-21T12:32:38.420379+0000","last_unstale":"2024-09-21T12:33:49.375951+0000","last_undegraded":"2024-09-21T12:33:49.375951+0000","last_fullsized":"2024-09-21T12:33:49.375951+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:00.343916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:00.343916+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:01.835131+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":783,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.388141+0000","last_change":"2024-09-21T12:32:50.746240+0000","last_active":"2024-09-21T12:33:49.388141+0000","last_peered":"2024-09-21T12:33:49.388141+0000","last_clean":"2024-09-21T12:33:49.388141+0000","last_became_active":"2024-09-21T12:32:14.242826+0000","last_became_peered":"2024-09-21T12:32:14.242826+0000","last_unstale":"2024-09-21T12:33:49.388141+0000","last_undegraded":"2024-09-21T12:33:49.388141+0000","last_fullsized":"2024-09-21T12:33:49.388141+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T12:32:50.746162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:50.746162+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:06.131744+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":671,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.391650+0000","last_change":"2024-09-21T12:29:10.089053+0000","last_active":"2024-09-21T12:33:49.391650+0000","last_peered":"2024-09-21T12:33:49.391650+0000","last_clean":"2024-09-21T12:33:49.391650+0000","last_became_active":"2024-09-21T12:20:15.297340+0000","last_became_peered":"2024-09-21T12:20:15.297340+0000","last_unstale":"2024-09-21T12:33:49.391650+0000","last_undegraded":"2024-09-21T12:33:49.391650+0000","last_fullsized":"2024-09-21T12:33:49.391650+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:10.088986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:10.088986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:12.727092+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":70,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.374613+0000","last_change":"2024-09-21T12:32:38.433724+0000","last_active":"2024-09-21T12:33:49.374613+0000","last_peered":"2024-09-21T12:33:49.374613+0000","last_clean":"2024-09-21T12:33:49.374613+0000","last_became_active":"2024-09-21T12:32:38.433417+0000","last_became_peered":"2024-09-21T12:32:38.433417+0000","last_unstale":"2024-09-21T12:33:49.374613+0000","last_undegraded":"2024-09-21T12:33:49.374613+0000","last_fullsized":"2024-09-21T12:33:49.374613+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:33.697208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:33.697208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:09.307597+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":70,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.375246+0000","last_change":"2024-09-21T12:32:38.416105+0000","last_active":"2024-09-21T12:33:49.375246+0000","last_peered":"2024-09-21T12:33:49.375246+0000","last_clean":"2024-09-21T12:33:49.375246+0000","last_became_active":"2024-09-21T12:32:38.415143+0000","last_became_peered":"2024-09-21T12:32:38.415143+0000","last_unstale":"2024-09-21T12:33:49.375246+0000","last_undegraded":"2024-09-21T12:33:49.375246+0000","last_fullsized":"2024-09-21T12:33:49.375246+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:05.143736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:29:05.143736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:18.203053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"379'6795","reported_seq":6942,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:51.768467+0000","last_change":"2024-09-21T12:33:36.827153+0000","last_active":"2024-09-21T12:33:51.768467+0000","last_peered":"2024-09-21T12:33:51.768467+0000","last_clean":"2024-09-21T12:33:51.768467+0000","last_became_active":"2024-09-21T12:32:30.305942+0000","last_became_peered":"2024-09-21T12:32:30.305942+0000","last_unstale":"2024-09-21T12:33:51.768467+0000","last_undegraded":"2024-09-21T12:33:51.768467+0000","last_fullsized":"2024-09-21T12:33:51.768467+0000","mapping_epoch":344,"log_start":"356'2000","ondisk_log_start":"356'2000","created":344,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'5675","last_scrub_stamp":"2024-09-21T12:33:36.827072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:33:36.827072+0000","objects_scrubbed":5605,"log_size":4795,"log_dups_size":0,"ondisk_log_size":4795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:37.893142+0000","scrub_duration":1900,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445317120,"num_objects":6795,"num_object_clones":0,"num_object_copies":20385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6795,"num_write_kb":434880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"76'79","reported_seq":606,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.388570+0000","last_change":"2024-09-21T12:33:33.718690+0000","last_active":"2024-09-21T12:33:49.388570+0000","last_peered":"2024-09-21T12:33:49.388570+0000","last_clean":"2024-09-21T12:33:49.388570+0000","last_became_active":"2024-09-21T12:26:44.960093+0000","last_became_peered":"2024-09-21T12:26:44.960093+0000","last_unstale":"2024-09-21T12:33:49.388570+0000","last_undegraded":"2024-09-21T12:33:49.388570+0000","last_fullsized":"2024-09-21T12:33:49.388570+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:33.718619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:33.718619+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:47.681797+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":86,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.374343+0000","last_change":"2024-09-21T12:32:38.457999+0000","last_active":"2024-09-21T12:33:49.374343+0000","last_peered":"2024-09-21T12:33:49.374343+0000","last_clean":"2024-09-21T12:33:49.374343+0000","last_became_active":"2024-09-21T12:32:38.425453+0000","last_became_peered":"2024-09-21T12:32:38.425453+0000","last_unstale":"2024-09-21T12:33:49.374343+0000","last_undegraded":"2024-09-21T12:33:49.374343+0000","last_fullsized":"2024-09-21T12:33:49.374343+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-21T12:26:51.463698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:51.463698+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:49.235792+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":40,"num_read_kb":33,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":760,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.388590+0000","last_change":"2024-09-21T12:32:38.438805+0000","last_active":"2024-09-21T12:33:49.388590+0000","last_peered":"2024-09-21T12:33:49.388590+0000","last_clean":"2024-09-21T12:33:49.388590+0000","last_became_active":"2024-09-21T12:32:38.424412+0000","last_became_peered":"2024-09-21T12:32:38.424412+0000","last_unstale":"2024-09-21T12:33:49.388590+0000","last_undegraded":"2024-09-21T12:33:49.388590+0000","last_fullsized":"2024-09-21T12:33:49.388590+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:28:32.727727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:32.727727+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:45.106117+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"352'12","reported_seq":887,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.387884+0000","last_change":"2024-09-21T12:32:38.426580+0000","last_active":"2024-09-21T12:33:49.387884+0000","last_peered":"2024-09-21T12:33:49.387884+0000","last_clean":"2024-09-21T12:33:49.387884+0000","last_became_active":"2024-09-21T12:32:38.409177+0000","last_became_peered":"2024-09-21T12:32:38.409177+0000","last_unstale":"2024-09-21T12:33:49.387884+0000","last_undegraded":"2024-09-21T12:33:49.387884+0000","last_fullsized":"2024-09-21T12:33:49.387884+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'11","last_scrub_stamp":"2024-09-21T12:32:27.612973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:32:27.612973+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":668,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.383593+0000","last_change":"2024-09-21T12:32:59.406284+0000","last_active":"2024-09-21T12:33:49.383593+0000","last_peered":"2024-09-21T12:33:49.383593+0000","last_clean":"2024-09-21T12:33:49.383593+0000","last_became_active":"2024-09-21T12:20:21.593367+0000","last_became_peered":"2024-09-21T12:20:21.593367+0000","last_unstale":"2024-09-21T12:33:49.383593+0000","last_undegraded":"2024-09-21T12:33:49.383593+0000","last_fullsized":"2024-09-21T12:33:49.383593+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:59.406201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:32:59.406201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:23.753561+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"379'6826","reported_seq":7165,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:51.799599+0000","last_change":"2024-09-21T12:32:41.397519+0000","last_active":"2024-09-21T12:33:51.799599+0000","last_peered":"2024-09-21T12:33:51.799599+0000","last_clean":"2024-09-21T12:33:51.799599+0000","last_became_active":"2024-09-21T12:32:41.396828+0000","last_became_peered":"2024-09-21T12:32:41.396828+0000","last_unstale":"2024-09-21T12:33:51.799599+0000","last_undegraded":"2024-09-21T12:33:51.799599+0000","last_fullsized":"2024-09-21T12:33:51.799599+0000","mapping_epoch":352,"log_start":"358'2500","ondisk_log_start":"358'2500","created":344,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:32:29.279916+0000","objects_scrubbed":0,"log_size":4326,"log_dups_size":0,"ondisk_log_size":4326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447348736,"num_objects":6826,"num_object_clones":0,"num_object_copies":20478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6826,"num_write_kb":436864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":8060928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"76'79","reported_seq":146,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.391499+0000","last_change":"2024-09-21T12:31:50.320881+0000","last_active":"2024-09-21T12:33:49.391499+0000","last_peered":"2024-09-21T12:33:49.391499+0000","last_clean":"2024-09-21T12:33:49.391499+0000","last_became_active":"2024-09-21T12:31:50.040087+0000","last_became_peered":"2024-09-21T12:31:50.040087+0000","last_unstale":"2024-09-21T12:33:49.391499+0000","last_undegraded":"2024-09-21T12:33:49.391499+0000","last_fullsized":"2024-09-21T12:33:49.391499+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:31:50.320834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:31:50.320834+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":213,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.388758+0000","last_change":"2024-09-21T12:32:38.446328+0000","last_active":"2024-09-21T12:33:49.388758+0000","last_peered":"2024-09-21T12:33:49.388758+0000","last_clean":"2024-09-21T12:33:49.388758+0000","last_became_active":"2024-09-21T12:32:38.424412+0000","last_became_peered":"2024-09-21T12:32:38.424412+0000","last_unstale":"2024-09-21T12:33:49.388758+0000","last_undegraded":"2024-09-21T12:33:49.388758+0000","last_fullsized":"2024-09-21T12:33:49.388758+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:31:59.022007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:59.022007+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:15.384785+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":696,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.394195+0000","last_change":"2024-09-21T12:32:19.629135+0000","last_active":"2024-09-21T12:33:49.394195+0000","last_peered":"2024-09-21T12:33:49.394195+0000","last_clean":"2024-09-21T12:33:49.394195+0000","last_became_active":"2024-09-21T12:20:15.308518+0000","last_became_peered":"2024-09-21T12:20:15.308518+0000","last_unstale":"2024-09-21T12:33:49.394195+0000","last_undegraded":"2024-09-21T12:33:49.394195+0000","last_fullsized":"2024-09-21T12:33:49.394195+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:19.629083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:19.629083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:43.126139+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":849,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.383930+0000","last_change":"2024-09-21T12:27:58.871800+0000","last_active":"2024-09-21T12:33:49.383930+0000","last_peered":"2024-09-21T12:33:49.383930+0000","last_clean":"2024-09-21T12:33:49.383930+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:33:49.383930+0000","last_undegraded":"2024-09-21T12:33:49.383930+0000","last_fullsized":"2024-09-21T12:33:49.383930+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:27:58.871710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:27:58.871710+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:16.027461+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":670,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.382821+0000","last_change":"2024-09-21T12:28:37.850166+0000","last_active":"2024-09-21T12:33:49.382821+0000","last_peered":"2024-09-21T12:33:49.382821+0000","last_clean":"2024-09-21T12:33:49.382821+0000","last_became_active":"2024-09-21T12:20:21.315801+0000","last_became_peered":"2024-09-21T12:20:21.315801+0000","last_unstale":"2024-09-21T12:33:49.382821+0000","last_undegraded":"2024-09-21T12:33:49.382821+0000","last_fullsized":"2024-09-21T12:33:49.382821+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:37.850100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:37.850100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:39.093148+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"379'7057","reported_seq":7363,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:51.720512+0000","last_change":"2024-09-21T12:32:42.037526+0000","last_active":"2024-09-21T12:33:51.720512+0000","last_peered":"2024-09-21T12:33:51.720512+0000","last_clean":"2024-09-21T12:33:51.720512+0000","last_became_active":"2024-09-21T12:32:42.035934+0000","last_became_peered":"2024-09-21T12:32:42.035934+0000","last_unstale":"2024-09-21T12:33:51.720512+0000","last_undegraded":"2024-09-21T12:33:51.720512+0000","last_fullsized":"2024-09-21T12:33:51.720512+0000","mapping_epoch":353,"log_start":"360'3100","ondisk_log_start":"360'3100","created":344,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:32:29.279916+0000","objects_scrubbed":0,"log_size":3957,"log_dups_size":0,"ondisk_log_size":3957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462487552,"num_objects":7057,"num_object_clones":0,"num_object_copies":21171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7057,"num_write_kb":451648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"76'79","reported_seq":364,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.385675+0000","last_change":"2024-09-21T12:32:52.986242+0000","last_active":"2024-09-21T12:33:49.385675+0000","last_peered":"2024-09-21T12:33:49.385675+0000","last_clean":"2024-09-21T12:33:49.385675+0000","last_became_active":"2024-09-21T12:29:05.835884+0000","last_became_peered":"2024-09-21T12:29:05.835884+0000","last_unstale":"2024-09-21T12:33:49.385675+0000","last_undegraded":"2024-09-21T12:33:49.385675+0000","last_fullsized":"2024-09-21T12:33:49.385675+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:52.986175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:52.986175+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:15.140598+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":729,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.383734+0000","last_change":"2024-09-21T12:33:38.081629+0000","last_active":"2024-09-21T12:33:49.383734+0000","last_peered":"2024-09-21T12:33:49.383734+0000","last_clean":"2024-09-21T12:33:49.383734+0000","last_became_active":"2024-09-21T12:20:17.693067+0000","last_became_peered":"2024-09-21T12:20:17.693067+0000","last_unstale":"2024-09-21T12:33:49.383734+0000","last_undegraded":"2024-09-21T12:33:49.383734+0000","last_fullsized":"2024-09-21T12:33:49.383734+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'29","last_scrub_stamp":"2024-09-21T12:33:38.081452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:38.081452+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:49.130782+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":672,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.388521+0000","last_change":"2024-09-21T12:33:38.250890+0000","last_active":"2024-09-21T12:33:49.388521+0000","last_peered":"2024-09-21T12:33:49.388521+0000","last_clean":"2024-09-21T12:33:49.388521+0000","last_became_active":"2024-09-21T12:20:15.308441+0000","last_became_peered":"2024-09-21T12:20:15.308441+0000","last_unstale":"2024-09-21T12:33:49.388521+0000","last_undegraded":"2024-09-21T12:33:49.388521+0000","last_fullsized":"2024-09-21T12:33:49.388521+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:38.250806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:38.250806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:53.341958+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":260,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.383639+0000","last_change":"2024-09-21T12:29:05.826167+0000","last_active":"2024-09-21T12:33:49.383639+0000","last_peered":"2024-09-21T12:33:49.383639+0000","last_clean":"2024-09-21T12:33:49.383639+0000","last_became_active":"2024-09-21T12:29:05.825869+0000","last_became_peered":"2024-09-21T12:29:05.825869+0000","last_unstale":"2024-09-21T12:33:49.383639+0000","last_undegraded":"2024-09-21T12:33:49.383639+0000","last_fullsized":"2024-09-21T12:33:49.383639+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:48.999878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:48.999878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":668,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.387219+0000","last_change":"2024-09-21T12:33:05.510687+0000","last_active":"2024-09-21T12:33:49.387219+0000","last_peered":"2024-09-21T12:33:49.387219+0000","last_clean":"2024-09-21T12:33:49.387219+0000","last_became_active":"2024-09-21T12:20:21.592828+0000","last_became_peered":"2024-09-21T12:20:21.592828+0000","last_unstale":"2024-09-21T12:33:49.387219+0000","last_undegraded":"2024-09-21T12:33:49.387219+0000","last_fullsized":"2024-09-21T12:33:49.387219+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:05.510620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:05.510620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:23.121767+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"379'6932","reported_seq":7013,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:51.770070+0000","last_change":"2024-09-21T12:32:30.352578+0000","last_active":"2024-09-21T12:33:51.770070+0000","last_peered":"2024-09-21T12:33:51.770070+0000","last_clean":"2024-09-21T12:33:51.770070+0000","last_became_active":"2024-09-21T12:32:30.352289+0000","last_became_peered":"2024-09-21T12:32:30.352289+0000","last_unstale":"2024-09-21T12:33:51.770070+0000","last_undegraded":"2024-09-21T12:33:51.770070+0000","last_fullsized":"2024-09-21T12:33:51.770070+0000","mapping_epoch":344,"log_start":"356'2100","ondisk_log_start":"356'2100","created":344,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:32:29.279916+0000","objects_scrubbed":0,"log_size":4832,"log_dups_size":0,"ondisk_log_size":4832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:49.829995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454295552,"num_objects":6932,"num_object_clones":0,"num_object_copies":20796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6932,"num_write_kb":443648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"76'79","reported_seq":472,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.386887+0000","last_change":"2024-09-21T12:33:34.055788+0000","last_active":"2024-09-21T12:33:49.386887+0000","last_peered":"2024-09-21T12:33:49.386887+0000","last_clean":"2024-09-21T12:33:49.386887+0000","last_became_active":"2024-09-21T12:26:54.092726+0000","last_became_peered":"2024-09-21T12:26:54.092726+0000","last_unstale":"2024-09-21T12:33:49.386887+0000","last_undegraded":"2024-09-21T12:33:49.386887+0000","last_fullsized":"2024-09-21T12:33:49.386887+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:34.055720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:34.055720+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:44.412164+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"276'40","reported_seq":898,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.388413+0000","last_change":"2024-09-21T12:32:25.139492+0000","last_active":"2024-09-21T12:33:49.388413+0000","last_peered":"2024-09-21T12:33:49.388413+0000","last_clean":"2024-09-21T12:33:49.388413+0000","last_became_active":"2024-09-21T12:20:17.307769+0000","last_became_peered":"2024-09-21T12:20:17.307769+0000","last_unstale":"2024-09-21T12:33:49.388413+0000","last_undegraded":"2024-09-21T12:33:49.388413+0000","last_fullsized":"2024-09-21T12:33:49.388413+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'40","last_scrub_stamp":"2024-09-21T12:32:25.139435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:25.139435+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":70,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.374931+0000","last_change":"2024-09-21T12:32:38.428085+0000","last_active":"2024-09-21T12:33:49.374931+0000","last_peered":"2024-09-21T12:33:49.374931+0000","last_clean":"2024-09-21T12:33:49.374931+0000","last_became_active":"2024-09-21T12:32:38.426309+0000","last_became_peered":"2024-09-21T12:32:38.426309+0000","last_unstale":"2024-09-21T12:33:49.374931+0000","last_undegraded":"2024-09-21T12:33:49.374931+0000","last_fullsized":"2024-09-21T12:33:49.374931+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:27.888717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:27.888717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:38.361903+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":398,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.388993+0000","last_change":"2024-09-21T12:32:18.988488+0000","last_active":"2024-09-21T12:33:49.388993+0000","last_peered":"2024-09-21T12:33:49.388993+0000","last_clean":"2024-09-21T12:33:49.388993+0000","last_became_active":"2024-09-21T12:26:54.083584+0000","last_became_peered":"2024-09-21T12:26:54.083584+0000","last_unstale":"2024-09-21T12:33:49.388993+0000","last_undegraded":"2024-09-21T12:33:49.388993+0000","last_fullsized":"2024-09-21T12:33:49.388993+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T12:32:18.988439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:32:18.988439+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:24.258290+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":718,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.384432+0000","last_change":"2024-09-21T12:33:00.087826+0000","last_active":"2024-09-21T12:33:49.384432+0000","last_peered":"2024-09-21T12:33:49.384432+0000","last_clean":"2024-09-21T12:33:49.384432+0000","last_became_active":"2024-09-21T12:32:38.424402+0000","last_became_peered":"2024-09-21T12:32:38.424402+0000","last_unstale":"2024-09-21T12:33:49.384432+0000","last_undegraded":"2024-09-21T12:33:49.384432+0000","last_fullsized":"2024-09-21T12:33:49.384432+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:00.087746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:33:00.087746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:10.178197+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":695,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.389306+0000","last_change":"2024-09-21T12:32:38.409478+0000","last_active":"2024-09-21T12:33:49.389306+0000","last_peered":"2024-09-21T12:33:49.389306+0000","last_clean":"2024-09-21T12:33:49.389306+0000","last_became_active":"2024-09-21T12:32:38.409159+0000","last_became_peered":"2024-09-21T12:32:38.409159+0000","last_unstale":"2024-09-21T12:33:49.389306+0000","last_undegraded":"2024-09-21T12:33:49.389306+0000","last_fullsized":"2024-09-21T12:33:49.389306+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:21.620769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:32:21.620769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"379'6755","reported_seq":6836,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:52.190596+0000","last_change":"2024-09-21T12:32:30.352377+0000","last_active":"2024-09-21T12:33:52.190596+0000","last_peered":"2024-09-21T12:33:52.190596+0000","last_clean":"2024-09-21T12:33:52.190596+0000","last_became_active":"2024-09-21T12:32:30.352017+0000","last_became_peered":"2024-09-21T12:32:30.352017+0000","last_unstale":"2024-09-21T12:33:52.190596+0000","last_undegraded":"2024-09-21T12:33:52.190596+0000","last_fullsized":"2024-09-21T12:33:52.190596+0000","mapping_epoch":344,"log_start":"354'900","ondisk_log_start":"354'900","created":344,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:32:29.279916+0000","objects_scrubbed":0,"log_size":5855,"log_dups_size":0,"ondisk_log_size":5855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:52.441778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442695680,"num_objects":6755,"num_object_clones":0,"num_object_copies":20265,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6755,"num_write_kb":432320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"76'79","reported_seq":109,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.383364+0000","last_change":"2024-09-21T12:32:38.420376+0000","last_active":"2024-09-21T12:33:49.383364+0000","last_peered":"2024-09-21T12:33:49.383364+0000","last_clean":"2024-09-21T12:33:49.383364+0000","last_became_active":"2024-09-21T12:32:38.417915+0000","last_became_peered":"2024-09-21T12:32:38.417915+0000","last_unstale":"2024-09-21T12:33:49.383364+0000","last_undegraded":"2024-09-21T12:33:49.383364+0000","last_fullsized":"2024-09-21T12:33:49.383364+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:30:39.179014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:30:39.179014+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:39.350016+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":167,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.389846+0000","last_change":"2024-09-21T12:31:39.072454+0000","last_active":"2024-09-21T12:33:49.389846+0000","last_peered":"2024-09-21T12:33:49.389846+0000","last_clean":"2024-09-21T12:33:49.389846+0000","last_became_active":"2024-09-21T12:31:39.057316+0000","last_became_peered":"2024-09-21T12:31:39.057316+0000","last_unstale":"2024-09-21T12:33:49.389846+0000","last_undegraded":"2024-09-21T12:33:49.389846+0000","last_fullsized":"2024-09-21T12:33:49.389846+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'13","last_scrub_stamp":"2024-09-21T12:28:25.549320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:25.549320+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:10.490444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":696,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.386793+0000","last_change":"2024-09-21T12:32:21.077501+0000","last_active":"2024-09-21T12:33:49.386793+0000","last_peered":"2024-09-21T12:33:49.386793+0000","last_clean":"2024-09-21T12:33:49.386793+0000","last_became_active":"2024-09-21T12:20:15.308356+0000","last_became_peered":"2024-09-21T12:20:15.308356+0000","last_unstale":"2024-09-21T12:33:49.386793+0000","last_undegraded":"2024-09-21T12:33:49.386793+0000","last_fullsized":"2024-09-21T12:33:49.386793+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:21.077447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:21.077447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":706,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.393389+0000","last_change":"2024-09-21T12:32:20.845517+0000","last_active":"2024-09-21T12:33:49.393389+0000","last_peered":"2024-09-21T12:33:49.393389+0000","last_clean":"2024-09-21T12:33:49.393389+0000","last_became_active":"2024-09-21T12:20:14.288358+0000","last_became_peered":"2024-09-21T12:20:14.288358+0000","last_unstale":"2024-09-21T12:33:49.393389+0000","last_undegraded":"2024-09-21T12:33:49.393389+0000","last_fullsized":"2024-09-21T12:33:49.393389+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:32:20.845428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:32:20.845428+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0019042449999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":689,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.394076+0000","last_change":"2024-09-21T12:32:22.799647+0000","last_active":"2024-09-21T12:33:49.394076+0000","last_peered":"2024-09-21T12:33:49.394076+0000","last_clean":"2024-09-21T12:33:49.394076+0000","last_became_active":"2024-09-21T12:20:19.309724+0000","last_became_peered":"2024-09-21T12:20:19.309724+0000","last_unstale":"2024-09-21T12:33:49.394076+0000","last_undegraded":"2024-09-21T12:33:49.394076+0000","last_fullsized":"2024-09-21T12:33:49.394076+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:22.799600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:32:22.799600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":660,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.385207+0000","last_change":"2024-09-21T12:32:24.398751+0000","last_active":"2024-09-21T12:33:49.385207+0000","last_peered":"2024-09-21T12:33:49.385207+0000","last_clean":"2024-09-21T12:33:49.385207+0000","last_became_active":"2024-09-21T12:20:21.592982+0000","last_became_peered":"2024-09-21T12:20:21.592982+0000","last_unstale":"2024-09-21T12:33:49.385207+0000","last_undegraded":"2024-09-21T12:33:49.385207+0000","last_fullsized":"2024-09-21T12:33:49.385207+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:24.398686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:32:24.398686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"379'6976","reported_seq":7057,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:52.974354+0000","last_change":"2024-09-21T12:32:30.307422+0000","last_active":"2024-09-21T12:33:52.974354+0000","last_peered":"2024-09-21T12:33:52.974354+0000","last_clean":"2024-09-21T12:33:52.974354+0000","last_became_active":"2024-09-21T12:32:30.306994+0000","last_became_peered":"2024-09-21T12:32:30.306994+0000","last_unstale":"2024-09-21T12:33:52.974354+0000","last_undegraded":"2024-09-21T12:33:52.974354+0000","last_fullsized":"2024-09-21T12:33:52.974354+0000","mapping_epoch":344,"log_start":"357'2300","ondisk_log_start":"357'2300","created":344,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:32:29.279916+0000","objects_scrubbed":0,"log_size":4676,"log_dups_size":0,"ondisk_log_size":4676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:41.082636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457179136,"num_objects":6976,"num_object_clones":0,"num_object_copies":20928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6976,"num_write_kb":446464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"76'79","reported_seq":637,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.387959+0000","last_change":"2024-09-21T12:32:58.503527+0000","last_active":"2024-09-21T12:33:49.387959+0000","last_peered":"2024-09-21T12:33:49.387959+0000","last_clean":"2024-09-21T12:33:49.387959+0000","last_became_active":"2024-09-21T12:32:38.424710+0000","last_became_peered":"2024-09-21T12:32:38.424710+0000","last_unstale":"2024-09-21T12:33:49.387959+0000","last_undegraded":"2024-09-21T12:33:49.387959+0000","last_fullsized":"2024-09-21T12:33:49.387959+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:58.503445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:58.503445+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:07.515803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":714,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.386000+0000","last_change":"2024-09-21T12:28:45.783033+0000","last_active":"2024-09-21T12:33:49.386000+0000","last_peered":"2024-09-21T12:33:49.386000+0000","last_clean":"2024-09-21T12:33:49.386000+0000","last_became_active":"2024-09-21T12:20:17.299325+0000","last_became_peered":"2024-09-21T12:20:17.299325+0000","last_unstale":"2024-09-21T12:33:49.386000+0000","last_undegraded":"2024-09-21T12:33:49.386000+0000","last_fullsized":"2024-09-21T12:33:49.386000+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:28:45.782955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.782955+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":732,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.385561+0000","last_change":"2024-09-21T12:32:53.934634+0000","last_active":"2024-09-21T12:33:49.385561+0000","last_peered":"2024-09-21T12:33:49.385561+0000","last_clean":"2024-09-21T12:33:49.385561+0000","last_became_active":"2024-09-21T12:32:38.411004+0000","last_became_peered":"2024-09-21T12:32:38.411004+0000","last_unstale":"2024-09-21T12:33:49.385561+0000","last_undegraded":"2024-09-21T12:33:49.385561+0000","last_fullsized":"2024-09-21T12:33:49.385561+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:53.934583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:53.934583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:12.010647+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":1509,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:50.109207+0000","last_change":"2024-09-21T12:32:58.369907+0000","last_active":"2024-09-21T12:33:50.109207+0000","last_peered":"2024-09-21T12:33:50.109207+0000","last_clean":"2024-09-21T12:33:50.109207+0000","last_became_active":"2024-09-21T12:20:14.287543+0000","last_became_peered":"2024-09-21T12:20:14.287543+0000","last_unstale":"2024-09-21T12:33:50.109207+0000","last_undegraded":"2024-09-21T12:33:50.109207+0000","last_fullsized":"2024-09-21T12:33:50.109207+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-21T12:32:58.369821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:32:58.369821+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:27.055101+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":810,"num_read_kb":805,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":118,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.393705+0000","last_change":"2024-09-21T12:32:02.030536+0000","last_active":"2024-09-21T12:33:49.393705+0000","last_peered":"2024-09-21T12:33:49.393705+0000","last_clean":"2024-09-21T12:33:49.393705+0000","last_became_active":"2024-09-21T12:32:02.030240+0000","last_became_peered":"2024-09-21T12:32:02.030240+0000","last_unstale":"2024-09-21T12:33:49.393705+0000","last_undegraded":"2024-09-21T12:33:49.393705+0000","last_fullsized":"2024-09-21T12:33:49.393705+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:11.422293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:11.422293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:16.252475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":163,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.388112+0000","last_change":"2024-09-21T12:32:59.551630+0000","last_active":"2024-09-21T12:33:49.388112+0000","last_peered":"2024-09-21T12:33:49.388112+0000","last_clean":"2024-09-21T12:33:49.388112+0000","last_became_active":"2024-09-21T12:31:39.060160+0000","last_became_peered":"2024-09-21T12:31:39.060160+0000","last_unstale":"2024-09-21T12:33:49.388112+0000","last_undegraded":"2024-09-21T12:33:49.388112+0000","last_fullsized":"2024-09-21T12:33:49.388112+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:59.551552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:32:59.551552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:05.100349+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"379'6921","reported_seq":7002,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:52.595996+0000","last_change":"2024-09-21T12:32:30.306788+0000","last_active":"2024-09-21T12:33:52.595996+0000","last_peered":"2024-09-21T12:33:52.595996+0000","last_clean":"2024-09-21T12:33:52.595996+0000","last_became_active":"2024-09-21T12:32:30.306490+0000","last_became_peered":"2024-09-21T12:32:30.306490+0000","last_unstale":"2024-09-21T12:33:52.595996+0000","last_undegraded":"2024-09-21T12:33:52.595996+0000","last_fullsized":"2024-09-21T12:33:52.595996+0000","mapping_epoch":344,"log_start":"358'2400","ondisk_log_start":"358'2400","created":344,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:32:29.279916+0000","objects_scrubbed":0,"log_size":4521,"log_dups_size":0,"ondisk_log_size":4521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:55.678477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453574656,"num_objects":6921,"num_object_clones":0,"num_object_copies":20763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6921,"num_write_kb":442944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"352'33","reported_seq":111,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.375899+0000","last_change":"2024-09-21T12:33:25.426978+0000","last_active":"2024-09-21T12:33:49.375899+0000","last_peered":"2024-09-21T12:33:49.375899+0000","last_clean":"2024-09-21T12:33:49.375899+0000","last_became_active":"2024-09-21T12:32:38.433558+0000","last_became_peered":"2024-09-21T12:32:38.433558+0000","last_unstale":"2024-09-21T12:33:49.375899+0000","last_undegraded":"2024-09-21T12:33:49.375899+0000","last_fullsized":"2024-09-21T12:33:49.375899+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'33","last_scrub_stamp":"2024-09-21T12:33:25.426799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:25.426799+0000","objects_scrubbed":8,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:51.190454+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":685,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.388734+0000","last_change":"2024-09-21T12:32:19.534660+0000","last_active":"2024-09-21T12:33:49.388734+0000","last_peered":"2024-09-21T12:33:49.388734+0000","last_clean":"2024-09-21T12:33:49.388734+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:33:49.388734+0000","last_undegraded":"2024-09-21T12:33:49.388734+0000","last_fullsized":"2024-09-21T12:33:49.388734+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:32:19.534573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:19.534573+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":764,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.384613+0000","last_change":"2024-09-21T12:33:06.338970+0000","last_active":"2024-09-21T12:33:49.384613+0000","last_peered":"2024-09-21T12:33:49.384613+0000","last_clean":"2024-09-21T12:33:49.384613+0000","last_became_active":"2024-09-21T12:26:44.950132+0000","last_became_peered":"2024-09-21T12:26:44.950132+0000","last_unstale":"2024-09-21T12:33:49.384613+0000","last_undegraded":"2024-09-21T12:33:49.384613+0000","last_fullsized":"2024-09-21T12:33:49.384613+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:06.338707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:06.338707+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:11.186546+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":102,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.384297+0000","last_change":"2024-09-21T12:32:14.233428+0000","last_active":"2024-09-21T12:33:49.384297+0000","last_peered":"2024-09-21T12:33:49.384297+0000","last_clean":"2024-09-21T12:33:49.384297+0000","last_became_active":"2024-09-21T12:32:14.233034+0000","last_became_peered":"2024-09-21T12:32:14.233034+0000","last_unstale":"2024-09-21T12:33:49.384297+0000","last_undegraded":"2024-09-21T12:33:49.384297+0000","last_fullsized":"2024-09-21T12:33:49.384297+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:36.684459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:36.684459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:18.337165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":527,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.390497+0000","last_change":"2024-09-21T12:32:38.409519+0000","last_active":"2024-09-21T12:33:49.390497+0000","last_peered":"2024-09-21T12:33:49.390497+0000","last_clean":"2024-09-21T12:33:49.390497+0000","last_became_active":"2024-09-21T12:32:38.409187+0000","last_became_peered":"2024-09-21T12:32:38.409187+0000","last_unstale":"2024-09-21T12:33:49.390497+0000","last_undegraded":"2024-09-21T12:33:49.390497+0000","last_fullsized":"2024-09-21T12:33:49.390497+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:28.647287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:28.647287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"379'6867","reported_seq":7211,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:51.757093+0000","last_change":"2024-09-21T12:32:41.394500+0000","last_active":"2024-09-21T12:33:51.757093+0000","last_peered":"2024-09-21T12:33:51.757093+0000","last_clean":"2024-09-21T12:33:51.757093+0000","last_became_active":"2024-09-21T12:32:41.393857+0000","last_became_peered":"2024-09-21T12:32:41.393857+0000","last_unstale":"2024-09-21T12:33:51.757093+0000","last_undegraded":"2024-09-21T12:33:51.757093+0000","last_fullsized":"2024-09-21T12:33:51.757093+0000","mapping_epoch":352,"log_start":"357'2100","ondisk_log_start":"357'2100","created":344,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:32:29.279916+0000","objects_scrubbed":0,"log_size":4767,"log_dups_size":0,"ondisk_log_size":4767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:50.783572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450035712,"num_objects":6867,"num_object_clones":0,"num_object_copies":20601,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6867,"num_write_kb":439488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":8257536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"76'79","reported_seq":634,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.388198+0000","last_change":"2024-09-21T12:33:04.508642+0000","last_active":"2024-09-21T12:33:49.388198+0000","last_peered":"2024-09-21T12:33:49.388198+0000","last_clean":"2024-09-21T12:33:49.388198+0000","last_became_active":"2024-09-21T12:27:14.954351+0000","last_became_peered":"2024-09-21T12:27:14.954351+0000","last_unstale":"2024-09-21T12:33:49.388198+0000","last_undegraded":"2024-09-21T12:33:49.388198+0000","last_fullsized":"2024-09-21T12:33:49.388198+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:04.508319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:04.508319+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:18.265775+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":90,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.377789+0000","last_change":"2024-09-21T12:33:35.410866+0000","last_active":"2024-09-21T12:33:49.377789+0000","last_peered":"2024-09-21T12:33:49.377789+0000","last_clean":"2024-09-21T12:33:49.377789+0000","last_became_active":"2024-09-21T12:32:38.419714+0000","last_became_peered":"2024-09-21T12:32:38.419714+0000","last_unstale":"2024-09-21T12:33:49.377789+0000","last_undegraded":"2024-09-21T12:33:49.377789+0000","last_fullsized":"2024-09-21T12:33:49.377789+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:33:35.410528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:35.410528+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:46.795723+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":78,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.374530+0000","last_change":"2024-09-21T12:33:01.343188+0000","last_active":"2024-09-21T12:33:49.374530+0000","last_peered":"2024-09-21T12:33:49.374530+0000","last_clean":"2024-09-21T12:33:49.374530+0000","last_became_active":"2024-09-21T12:32:38.432592+0000","last_became_peered":"2024-09-21T12:32:38.432592+0000","last_unstale":"2024-09-21T12:33:49.374530+0000","last_undegraded":"2024-09-21T12:33:49.374530+0000","last_fullsized":"2024-09-21T12:33:49.374530+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:01.342864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:01.342864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:19.155035+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":70,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.376315+0000","last_change":"2024-09-21T12:32:38.425995+0000","last_active":"2024-09-21T12:33:49.376315+0000","last_peered":"2024-09-21T12:33:49.376315+0000","last_clean":"2024-09-21T12:33:49.376315+0000","last_became_active":"2024-09-21T12:32:38.425683+0000","last_became_peered":"2024-09-21T12:32:38.425683+0000","last_unstale":"2024-09-21T12:33:49.376315+0000","last_undegraded":"2024-09-21T12:33:49.376315+0000","last_fullsized":"2024-09-21T12:33:49.376315+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:21.847336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:32:21.847336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":667,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.388700+0000","last_change":"2024-09-21T12:32:54.734001+0000","last_active":"2024-09-21T12:33:49.388700+0000","last_peered":"2024-09-21T12:33:49.388700+0000","last_clean":"2024-09-21T12:33:49.388700+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:33:49.388700+0000","last_undegraded":"2024-09-21T12:33:49.388700+0000","last_fullsized":"2024-09-21T12:33:49.388700+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:54.733930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:32:54.733930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:22.095119+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"379'6720","reported_seq":6801,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:52.185209+0000","last_change":"2024-09-21T12:32:30.352552+0000","last_active":"2024-09-21T12:33:52.185209+0000","last_peered":"2024-09-21T12:33:52.185209+0000","last_clean":"2024-09-21T12:33:52.185209+0000","last_became_active":"2024-09-21T12:32:30.351805+0000","last_became_peered":"2024-09-21T12:32:30.351805+0000","last_unstale":"2024-09-21T12:33:52.185209+0000","last_undegraded":"2024-09-21T12:33:52.185209+0000","last_fullsized":"2024-09-21T12:33:52.185209+0000","mapping_epoch":344,"log_start":"354'900","ondisk_log_start":"354'900","created":344,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:32:29.279916+0000","objects_scrubbed":0,"log_size":5820,"log_dups_size":0,"ondisk_log_size":5820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:48.268247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440401920,"num_objects":6720,"num_object_clones":0,"num_object_copies":20160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6720,"num_write_kb":430080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"76'79","reported_seq":412,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.393397+0000","last_change":"2024-09-21T12:33:38.223142+0000","last_active":"2024-09-21T12:33:49.393397+0000","last_peered":"2024-09-21T12:33:49.393397+0000","last_clean":"2024-09-21T12:33:49.393397+0000","last_became_active":"2024-09-21T12:32:02.026943+0000","last_became_peered":"2024-09-21T12:32:02.026943+0000","last_unstale":"2024-09-21T12:33:49.393397+0000","last_undegraded":"2024-09-21T12:33:49.393397+0000","last_fullsized":"2024-09-21T12:33:49.393397+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:38.223087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:38.223087+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:53.336483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":96,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.376113+0000","last_change":"2024-09-21T12:33:08.360079+0000","last_active":"2024-09-21T12:33:49.376113+0000","last_peered":"2024-09-21T12:33:49.376113+0000","last_clean":"2024-09-21T12:33:49.376113+0000","last_became_active":"2024-09-21T12:32:38.423532+0000","last_became_peered":"2024-09-21T12:32:38.423532+0000","last_unstale":"2024-09-21T12:33:49.376113+0000","last_undegraded":"2024-09-21T12:33:49.376113+0000","last_fullsized":"2024-09-21T12:33:49.376113+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'31","last_scrub_stamp":"2024-09-21T12:33:08.360012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:08.360012+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:22.046985+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":78,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.374991+0000","last_change":"2024-09-21T12:33:02.315298+0000","last_active":"2024-09-21T12:33:49.374991+0000","last_peered":"2024-09-21T12:33:49.374991+0000","last_clean":"2024-09-21T12:33:49.374991+0000","last_became_active":"2024-09-21T12:32:38.424451+0000","last_became_peered":"2024-09-21T12:32:38.424451+0000","last_unstale":"2024-09-21T12:33:49.374991+0000","last_undegraded":"2024-09-21T12:33:49.374991+0000","last_fullsized":"2024-09-21T12:33:49.374991+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:02.315122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:02.315122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:03.062338+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":692,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.388226+0000","last_change":"2024-09-21T12:32:38.425510+0000","last_active":"2024-09-21T12:33:49.388226+0000","last_peered":"2024-09-21T12:33:49.388226+0000","last_clean":"2024-09-21T12:33:49.388226+0000","last_became_active":"2024-09-21T12:32:38.425177+0000","last_became_peered":"2024-09-21T12:32:38.425177+0000","last_unstale":"2024-09-21T12:33:49.388226+0000","last_undegraded":"2024-09-21T12:33:49.388226+0000","last_fullsized":"2024-09-21T12:33:49.388226+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:03.570984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:03.570984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:45.297014+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":680,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.396249+0000","last_change":"2024-09-21T12:32:38.414367+0000","last_active":"2024-09-21T12:33:49.396249+0000","last_peered":"2024-09-21T12:33:49.396249+0000","last_clean":"2024-09-21T12:33:49.396249+0000","last_became_active":"2024-09-21T12:32:38.414053+0000","last_became_peered":"2024-09-21T12:32:38.414053+0000","last_unstale":"2024-09-21T12:33:49.396249+0000","last_undegraded":"2024-09-21T12:33:49.396249+0000","last_fullsized":"2024-09-21T12:33:49.396249+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:25:39.079015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:25:39.079015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:43.271537+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"379'6878","reported_seq":7254,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:52.581974+0000","last_change":"2024-09-21T12:32:42.036554+0000","last_active":"2024-09-21T12:33:52.581974+0000","last_peered":"2024-09-21T12:33:52.581974+0000","last_clean":"2024-09-21T12:33:52.581974+0000","last_became_active":"2024-09-21T12:32:42.035883+0000","last_became_peered":"2024-09-21T12:32:42.035883+0000","last_unstale":"2024-09-21T12:33:52.581974+0000","last_undegraded":"2024-09-21T12:33:52.581974+0000","last_fullsized":"2024-09-21T12:33:52.581974+0000","mapping_epoch":353,"log_start":"358'2400","ondisk_log_start":"358'2400","created":344,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:32:29.279916+0000","objects_scrubbed":0,"log_size":4478,"log_dups_size":0,"ondisk_log_size":4478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:57.371064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450756608,"num_objects":6878,"num_object_clones":0,"num_object_copies":20634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6878,"num_write_kb":440192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":9306112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"76'79","reported_seq":754,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.384552+0000","last_change":"2024-09-21T12:29:05.501052+0000","last_active":"2024-09-21T12:33:49.384552+0000","last_peered":"2024-09-21T12:33:49.384552+0000","last_clean":"2024-09-21T12:33:49.384552+0000","last_became_active":"2024-09-21T12:27:14.943705+0000","last_became_peered":"2024-09-21T12:27:14.943705+0000","last_unstale":"2024-09-21T12:33:49.384552+0000","last_undegraded":"2024-09-21T12:33:49.384552+0000","last_fullsized":"2024-09-21T12:33:49.384552+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:05.500994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:05.500994+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:19.339677+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":82,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.374872+0000","last_change":"2024-09-21T12:32:38.441207+0000","last_active":"2024-09-21T12:33:49.374872+0000","last_peered":"2024-09-21T12:33:49.374872+0000","last_clean":"2024-09-21T12:33:49.374872+0000","last_became_active":"2024-09-21T12:32:38.418750+0000","last_became_peered":"2024-09-21T12:32:38.418750+0000","last_unstale":"2024-09-21T12:33:49.374872+0000","last_undegraded":"2024-09-21T12:33:49.374872+0000","last_fullsized":"2024-09-21T12:33:49.374872+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:32:26.690403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:26.690403+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":707,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.391092+0000","last_change":"2024-09-21T12:32:38.404305+0000","last_active":"2024-09-21T12:33:49.391092+0000","last_peered":"2024-09-21T12:33:49.391092+0000","last_clean":"2024-09-21T12:33:49.391092+0000","last_became_active":"2024-09-21T12:32:38.403997+0000","last_became_peered":"2024-09-21T12:32:38.403997+0000","last_unstale":"2024-09-21T12:33:49.391092+0000","last_undegraded":"2024-09-21T12:33:49.391092+0000","last_fullsized":"2024-09-21T12:33:49.391092+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:47.248389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:31:47.248389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"66'8","reported_seq":869,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:50.521739+0000","last_change":"2024-09-21T12:33:34.270676+0000","last_active":"2024-09-21T12:33:50.521739+0000","last_peered":"2024-09-21T12:33:50.521739+0000","last_clean":"2024-09-21T12:33:50.521739+0000","last_became_active":"2024-09-21T12:20:19.307924+0000","last_became_peered":"2024-09-21T12:20:19.307924+0000","last_unstale":"2024-09-21T12:33:50.521739+0000","last_undegraded":"2024-09-21T12:33:50.521739+0000","last_fullsized":"2024-09-21T12:33:50.521739+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:33:34.270590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:33:34.270590+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:35.866548+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":78,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.377641+0000","last_change":"2024-09-21T12:33:06.369571+0000","last_active":"2024-09-21T12:33:49.377641+0000","last_peered":"2024-09-21T12:33:49.377641+0000","last_clean":"2024-09-21T12:33:49.377641+0000","last_became_active":"2024-09-21T12:32:38.419743+0000","last_became_peered":"2024-09-21T12:32:38.419743+0000","last_unstale":"2024-09-21T12:33:49.377641+0000","last_undegraded":"2024-09-21T12:33:49.377641+0000","last_fullsized":"2024-09-21T12:33:49.377641+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:06.369436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:06.369436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:31.986378+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"379'6834","reported_seq":6912,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:52.183821+0000","last_change":"2024-09-21T12:32:30.311967+0000","last_active":"2024-09-21T12:33:52.183821+0000","last_peered":"2024-09-21T12:33:52.183821+0000","last_clean":"2024-09-21T12:33:52.183821+0000","last_became_active":"2024-09-21T12:32:30.311511+0000","last_became_peered":"2024-09-21T12:32:30.311511+0000","last_unstale":"2024-09-21T12:33:52.183821+0000","last_undegraded":"2024-09-21T12:33:52.183821+0000","last_fullsized":"2024-09-21T12:33:52.183821+0000","mapping_epoch":344,"log_start":"354'1000","ondisk_log_start":"354'1000","created":344,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:32:29.279916+0000","objects_scrubbed":0,"log_size":5834,"log_dups_size":0,"ondisk_log_size":5834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:58.226151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447873024,"num_objects":6834,"num_object_clones":0,"num_object_copies":20502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6834,"num_write_kb":437376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"76'79","reported_seq":70,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.377843+0000","last_change":"2024-09-21T12:32:38.433715+0000","last_active":"2024-09-21T12:33:49.377843+0000","last_peered":"2024-09-21T12:33:49.377843+0000","last_clean":"2024-09-21T12:33:49.377843+0000","last_became_active":"2024-09-21T12:32:38.433365+0000","last_became_peered":"2024-09-21T12:32:38.433365+0000","last_unstale":"2024-09-21T12:33:49.377843+0000","last_undegraded":"2024-09-21T12:33:49.377843+0000","last_fullsized":"2024-09-21T12:33:49.377843+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:21.345364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:21.345364+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:34:46.252764+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13417,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":5,"num_write":2,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":13417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":787,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.383812+0000","last_change":"2024-09-21T12:33:31.018209+0000","last_active":"2024-09-21T12:33:49.383812+0000","last_peered":"2024-09-21T12:33:49.383812+0000","last_clean":"2024-09-21T12:33:49.383812+0000","last_became_active":"2024-09-21T12:32:38.424376+0000","last_became_peered":"2024-09-21T12:32:38.424376+0000","last_unstale":"2024-09-21T12:33:49.383812+0000","last_undegraded":"2024-09-21T12:33:49.383812+0000","last_fullsized":"2024-09-21T12:33:49.383812+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:33:31.018128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:31.018128+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:57.269413+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":363,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.389169+0000","last_change":"2024-09-21T12:31:12.884054+0000","last_active":"2024-09-21T12:33:49.389169+0000","last_peered":"2024-09-21T12:33:49.389169+0000","last_clean":"2024-09-21T12:33:49.389169+0000","last_became_active":"2024-09-21T12:26:54.084228+0000","last_became_peered":"2024-09-21T12:26:54.084228+0000","last_unstale":"2024-09-21T12:33:49.389169+0000","last_undegraded":"2024-09-21T12:33:49.389169+0000","last_fullsized":"2024-09-21T12:33:49.389169+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:12.883983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:31:12.883983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:24.829777+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":709,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.395850+0000","last_change":"2024-09-21T12:32:38.414230+0000","last_active":"2024-09-21T12:33:49.395850+0000","last_peered":"2024-09-21T12:33:49.395850+0000","last_clean":"2024-09-21T12:33:49.395850+0000","last_became_active":"2024-09-21T12:32:38.413905+0000","last_became_peered":"2024-09-21T12:32:38.413905+0000","last_unstale":"2024-09-21T12:33:49.395850+0000","last_undegraded":"2024-09-21T12:33:49.395850+0000","last_fullsized":"2024-09-21T12:33:49.395850+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:47.361115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:47.361115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":670,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.386471+0000","last_change":"2024-09-21T12:33:37.119882+0000","last_active":"2024-09-21T12:33:49.386471+0000","last_peered":"2024-09-21T12:33:49.386471+0000","last_clean":"2024-09-21T12:33:49.386471+0000","last_became_active":"2024-09-21T12:20:21.311493+0000","last_became_peered":"2024-09-21T12:20:21.311493+0000","last_unstale":"2024-09-21T12:33:49.386471+0000","last_undegraded":"2024-09-21T12:33:49.386471+0000","last_fullsized":"2024-09-21T12:33:49.386471+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:37.119815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:37.119815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:43.800267+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"379'6997","reported_seq":7348,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:52.960911+0000","last_change":"2024-09-21T12:32:42.038197+0000","last_active":"2024-09-21T12:33:52.960911+0000","last_peered":"2024-09-21T12:33:52.960911+0000","last_clean":"2024-09-21T12:33:52.960911+0000","last_became_active":"2024-09-21T12:32:42.036249+0000","last_became_peered":"2024-09-21T12:32:42.036249+0000","last_unstale":"2024-09-21T12:33:52.960911+0000","last_undegraded":"2024-09-21T12:33:52.960911+0000","last_fullsized":"2024-09-21T12:33:52.960911+0000","mapping_epoch":353,"log_start":"358'2300","ondisk_log_start":"358'2300","created":344,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:32:29.279916+0000","objects_scrubbed":0,"log_size":4697,"log_dups_size":0,"ondisk_log_size":4697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458555392,"num_objects":6997,"num_object_clones":0,"num_object_copies":20991,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6997,"num_write_kb":447808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":8454144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"76'79","reported_seq":610,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.394164+0000","last_change":"2024-09-21T12:32:38.415231+0000","last_active":"2024-09-21T12:33:49.394164+0000","last_peered":"2024-09-21T12:33:49.394164+0000","last_clean":"2024-09-21T12:33:49.394164+0000","last_became_active":"2024-09-21T12:32:38.414877+0000","last_became_peered":"2024-09-21T12:32:38.414877+0000","last_unstale":"2024-09-21T12:33:49.394164+0000","last_undegraded":"2024-09-21T12:33:49.394164+0000","last_fullsized":"2024-09-21T12:33:49.394164+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:25.729795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:25.729795+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:52.053506+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":793,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.376896+0000","last_change":"2024-09-21T12:32:14.251878+0000","last_active":"2024-09-21T12:33:49.376896+0000","last_peered":"2024-09-21T12:33:49.376896+0000","last_clean":"2024-09-21T12:33:49.376896+0000","last_became_active":"2024-09-21T12:32:14.236905+0000","last_became_peered":"2024-09-21T12:32:14.236905+0000","last_unstale":"2024-09-21T12:33:49.376896+0000","last_undegraded":"2024-09-21T12:33:49.376896+0000","last_fullsized":"2024-09-21T12:33:49.376896+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'35","last_scrub_stamp":"2024-09-21T12:31:08.327617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:08.327617+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:43.921534+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":694,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.385594+0000","last_change":"2024-09-21T12:33:08.316386+0000","last_active":"2024-09-21T12:33:49.385594+0000","last_peered":"2024-09-21T12:33:49.385594+0000","last_clean":"2024-09-21T12:33:49.385594+0000","last_became_active":"2024-09-21T12:32:02.025668+0000","last_became_peered":"2024-09-21T12:32:02.025668+0000","last_unstale":"2024-09-21T12:33:49.385594+0000","last_undegraded":"2024-09-21T12:33:49.385594+0000","last_fullsized":"2024-09-21T12:33:49.385594+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:08.316315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:08.316315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:20.301418+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":687,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.392921+0000","last_change":"2024-09-21T12:32:02.043989+0000","last_active":"2024-09-21T12:33:49.392921+0000","last_peered":"2024-09-21T12:33:49.392921+0000","last_clean":"2024-09-21T12:33:49.392921+0000","last_became_active":"2024-09-21T12:32:02.043597+0000","last_became_peered":"2024-09-21T12:32:02.043597+0000","last_unstale":"2024-09-21T12:33:49.392921+0000","last_undegraded":"2024-09-21T12:33:49.392921+0000","last_fullsized":"2024-09-21T12:33:49.392921+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:39.732474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:31:39.732474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:09.620517+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":660,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.384553+0000","last_change":"2024-09-21T12:33:27.297692+0000","last_active":"2024-09-21T12:33:49.384553+0000","last_peered":"2024-09-21T12:33:49.384553+0000","last_clean":"2024-09-21T12:33:49.384553+0000","last_became_active":"2024-09-21T12:20:21.314545+0000","last_became_peered":"2024-09-21T12:20:21.314545+0000","last_unstale":"2024-09-21T12:33:49.384553+0000","last_undegraded":"2024-09-21T12:33:49.384553+0000","last_fullsized":"2024-09-21T12:33:49.384553+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:27.297558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:27.297558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:50.029655+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"379'6822","reported_seq":6903,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:52.660638+0000","last_change":"2024-09-21T12:32:30.352070+0000","last_active":"2024-09-21T12:33:52.660638+0000","last_peered":"2024-09-21T12:33:52.660638+0000","last_clean":"2024-09-21T12:33:52.660638+0000","last_became_active":"2024-09-21T12:32:30.351779+0000","last_became_peered":"2024-09-21T12:32:30.351779+0000","last_unstale":"2024-09-21T12:33:52.660638+0000","last_undegraded":"2024-09-21T12:33:52.660638+0000","last_fullsized":"2024-09-21T12:33:52.660638+0000","mapping_epoch":344,"log_start":"359'2700","ondisk_log_start":"359'2700","created":344,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:32:29.279916+0000","objects_scrubbed":0,"log_size":4122,"log_dups_size":0,"ondisk_log_size":4122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447086592,"num_objects":6822,"num_object_clones":0,"num_object_copies":20466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6822,"num_write_kb":436608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"76'79","reported_seq":633,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.388858+0000","last_change":"2024-09-21T12:33:37.767771+0000","last_active":"2024-09-21T12:33:49.388858+0000","last_peered":"2024-09-21T12:33:49.388858+0000","last_clean":"2024-09-21T12:33:49.388858+0000","last_became_active":"2024-09-21T12:27:14.951375+0000","last_became_peered":"2024-09-21T12:27:14.951375+0000","last_unstale":"2024-09-21T12:33:49.388858+0000","last_undegraded":"2024-09-21T12:33:49.388858+0000","last_fullsized":"2024-09-21T12:33:49.388858+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:37.767584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:37.767584+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:42.116434+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":82,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.374761+0000","last_change":"2024-09-21T12:32:38.451446+0000","last_active":"2024-09-21T12:33:49.374761+0000","last_peered":"2024-09-21T12:33:49.374761+0000","last_clean":"2024-09-21T12:33:49.374761+0000","last_became_active":"2024-09-21T12:32:38.421609+0000","last_became_peered":"2024-09-21T12:32:38.421609+0000","last_unstale":"2024-09-21T12:33:49.374761+0000","last_undegraded":"2024-09-21T12:33:49.374761+0000","last_fullsized":"2024-09-21T12:33:49.374761+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T12:32:31.075451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:31.075451+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":716,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.384259+0000","last_change":"2024-09-21T12:32:54.364388+0000","last_active":"2024-09-21T12:33:49.384259+0000","last_peered":"2024-09-21T12:33:49.384259+0000","last_clean":"2024-09-21T12:33:49.384259+0000","last_became_active":"2024-09-21T12:32:38.417869+0000","last_became_peered":"2024-09-21T12:32:38.417869+0000","last_unstale":"2024-09-21T12:33:49.384259+0000","last_undegraded":"2024-09-21T12:33:49.384259+0000","last_fullsized":"2024-09-21T12:33:49.384259+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:54.364340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:54.364340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:09.510649+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":689,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.393581+0000","last_change":"2024-09-21T12:32:23.769797+0000","last_active":"2024-09-21T12:33:49.393581+0000","last_peered":"2024-09-21T12:33:49.393581+0000","last_clean":"2024-09-21T12:33:49.393581+0000","last_became_active":"2024-09-21T12:20:19.312572+0000","last_became_peered":"2024-09-21T12:20:19.312572+0000","last_unstale":"2024-09-21T12:33:49.393581+0000","last_undegraded":"2024-09-21T12:33:49.393581+0000","last_fullsized":"2024-09-21T12:33:49.393581+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:23.769748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:32:23.769748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:47.219530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":681,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.390756+0000","last_change":"2024-09-21T12:32:15.555334+0000","last_active":"2024-09-21T12:33:49.390756+0000","last_peered":"2024-09-21T12:33:49.390756+0000","last_clean":"2024-09-21T12:33:49.390756+0000","last_became_active":"2024-09-21T12:32:14.242180+0000","last_became_peered":"2024-09-21T12:32:14.242180+0000","last_unstale":"2024-09-21T12:33:49.390756+0000","last_undegraded":"2024-09-21T12:33:49.390756+0000","last_fullsized":"2024-09-21T12:33:49.390756+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:15.555279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:32:15.555279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"379'6819","reported_seq":7172,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:52.891087+0000","last_change":"2024-09-21T12:32:42.036962+0000","last_active":"2024-09-21T12:33:52.891087+0000","last_peered":"2024-09-21T12:33:52.891087+0000","last_clean":"2024-09-21T12:33:52.891087+0000","last_became_active":"2024-09-21T12:32:42.036291+0000","last_became_peered":"2024-09-21T12:32:42.036291+0000","last_unstale":"2024-09-21T12:33:52.891087+0000","last_undegraded":"2024-09-21T12:33:52.891087+0000","last_fullsized":"2024-09-21T12:33:52.891087+0000","mapping_epoch":353,"log_start":"357'2200","ondisk_log_start":"357'2200","created":344,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:32:29.279916+0000","objects_scrubbed":0,"log_size":4619,"log_dups_size":0,"ondisk_log_size":4619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446889984,"num_objects":6819,"num_object_clones":0,"num_object_copies":20457,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6819,"num_write_kb":436416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":8585216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"76'79","reported_seq":117,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.383481+0000","last_change":"2024-09-21T12:33:32.272999+0000","last_active":"2024-09-21T12:33:49.383481+0000","last_peered":"2024-09-21T12:33:49.383481+0000","last_clean":"2024-09-21T12:33:49.383481+0000","last_became_active":"2024-09-21T12:32:38.418574+0000","last_became_peered":"2024-09-21T12:32:38.418574+0000","last_unstale":"2024-09-21T12:33:49.383481+0000","last_undegraded":"2024-09-21T12:33:49.383481+0000","last_fullsized":"2024-09-21T12:33:49.383481+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:32.272927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:32.272927+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:57.100943+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":165,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.393314+0000","last_change":"2024-09-21T12:33:02.333599+0000","last_active":"2024-09-21T12:33:49.393314+0000","last_peered":"2024-09-21T12:33:49.393314+0000","last_clean":"2024-09-21T12:33:49.393314+0000","last_became_active":"2024-09-21T12:32:14.234897+0000","last_became_peered":"2024-09-21T12:32:14.234897+0000","last_unstale":"2024-09-21T12:33:49.393314+0000","last_undegraded":"2024-09-21T12:33:49.393314+0000","last_fullsized":"2024-09-21T12:33:49.393314+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:33:02.333507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:02.333507+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:11.346228+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":770,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.384716+0000","last_change":"2024-09-21T12:32:38.444277+0000","last_active":"2024-09-21T12:33:49.384716+0000","last_peered":"2024-09-21T12:33:49.384716+0000","last_clean":"2024-09-21T12:33:49.384716+0000","last_became_active":"2024-09-21T12:32:38.417302+0000","last_became_peered":"2024-09-21T12:32:38.417302+0000","last_unstale":"2024-09-21T12:33:49.384716+0000","last_undegraded":"2024-09-21T12:33:49.384716+0000","last_fullsized":"2024-09-21T12:33:49.384716+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-21T12:31:53.103575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:31:53.103575+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:51.468681+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"352'14","reported_seq":903,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.384502+0000","last_change":"2024-09-21T12:32:38.446438+0000","last_active":"2024-09-21T12:33:49.384502+0000","last_peered":"2024-09-21T12:33:49.384502+0000","last_clean":"2024-09-21T12:33:49.384502+0000","last_became_active":"2024-09-21T12:32:38.418813+0000","last_became_peered":"2024-09-21T12:32:38.418813+0000","last_unstale":"2024-09-21T12:33:49.384502+0000","last_undegraded":"2024-09-21T12:33:49.384502+0000","last_fullsized":"2024-09-21T12:33:49.384502+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'13","last_scrub_stamp":"2024-09-21T12:31:57.223134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:31:57.223134+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":371,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.376850+0000","last_change":"2024-09-21T12:32:27.717252+0000","last_active":"2024-09-21T12:33:49.376850+0000","last_peered":"2024-09-21T12:33:49.376850+0000","last_clean":"2024-09-21T12:33:49.376850+0000","last_became_active":"2024-09-21T12:26:54.089155+0000","last_became_peered":"2024-09-21T12:26:54.089155+0000","last_unstale":"2024-09-21T12:33:49.376850+0000","last_undegraded":"2024-09-21T12:33:49.376850+0000","last_fullsized":"2024-09-21T12:33:49.376850+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:27.717157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:32:27.717157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:56.618130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"76'79","reported_seq":417,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.393839+0000","last_change":"2024-09-21T12:32:26.698667+0000","last_active":"2024-09-21T12:33:49.393839+0000","last_peered":"2024-09-21T12:33:49.393839+0000","last_clean":"2024-09-21T12:33:49.393839+0000","last_became_active":"2024-09-21T12:31:39.060422+0000","last_became_peered":"2024-09-21T12:31:39.060422+0000","last_unstale":"2024-09-21T12:33:49.393839+0000","last_undegraded":"2024-09-21T12:33:49.393839+0000","last_fullsized":"2024-09-21T12:33:49.393839+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:26.698622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:26.698622+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:53.950707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":717,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.376746+0000","last_change":"2024-09-21T12:32:02.027084+0000","last_active":"2024-09-21T12:33:49.376746+0000","last_peered":"2024-09-21T12:33:49.376746+0000","last_clean":"2024-09-21T12:33:49.376746+0000","last_became_active":"2024-09-21T12:32:02.026693+0000","last_became_peered":"2024-09-21T12:32:02.026693+0000","last_unstale":"2024-09-21T12:33:49.376746+0000","last_undegraded":"2024-09-21T12:33:49.376746+0000","last_fullsized":"2024-09-21T12:33:49.376746+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:07.375741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:31:07.375741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:42.928508+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":169,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.390933+0000","last_change":"2024-09-21T12:32:20.584288+0000","last_active":"2024-09-21T12:33:49.390933+0000","last_peered":"2024-09-21T12:33:49.390933+0000","last_clean":"2024-09-21T12:33:49.390933+0000","last_became_active":"2024-09-21T12:32:14.244680+0000","last_became_peered":"2024-09-21T12:32:14.244680+0000","last_unstale":"2024-09-21T12:33:49.390933+0000","last_undegraded":"2024-09-21T12:33:49.390933+0000","last_fullsized":"2024-09-21T12:33:49.390933+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T12:32:20.584227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:20.584227+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":371,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.389160+0000","last_change":"2024-09-21T12:33:34.706596+0000","last_active":"2024-09-21T12:33:49.389160+0000","last_peered":"2024-09-21T12:33:49.389160+0000","last_clean":"2024-09-21T12:33:49.389160+0000","last_became_active":"2024-09-21T12:26:54.084109+0000","last_became_peered":"2024-09-21T12:26:54.084109+0000","last_unstale":"2024-09-21T12:33:49.389160+0000","last_undegraded":"2024-09-21T12:33:49.389160+0000","last_fullsized":"2024-09-21T12:33:49.389160+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:34.706531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:33:34.706531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:50.815981+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":687,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.385120+0000","last_change":"2024-09-21T12:32:50.355531+0000","last_active":"2024-09-21T12:33:49.385120+0000","last_peered":"2024-09-21T12:33:49.385120+0000","last_clean":"2024-09-21T12:33:49.385120+0000","last_became_active":"2024-09-21T12:32:38.417522+0000","last_became_peered":"2024-09-21T12:32:38.417522+0000","last_unstale":"2024-09-21T12:33:49.385120+0000","last_undegraded":"2024-09-21T12:33:49.385120+0000","last_fullsized":"2024-09-21T12:33:49.385120+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:50.355485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:32:50.355485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:55.079521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'79","reported_seq":649,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.388059+0000","last_change":"2024-09-21T12:32:38.425852+0000","last_active":"2024-09-21T12:33:49.388059+0000","last_peered":"2024-09-21T12:33:49.388059+0000","last_clean":"2024-09-21T12:33:49.388059+0000","last_became_active":"2024-09-21T12:32:38.425495+0000","last_became_peered":"2024-09-21T12:32:38.425495+0000","last_unstale":"2024-09-21T12:33:49.388059+0000","last_undegraded":"2024-09-21T12:33:49.388059+0000","last_fullsized":"2024-09-21T12:33:49.388059+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:28.139007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:28.139007+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:57.563096+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":744,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.388491+0000","last_change":"2024-09-21T12:32:19.786992+0000","last_active":"2024-09-21T12:33:49.388491+0000","last_peered":"2024-09-21T12:33:49.388491+0000","last_clean":"2024-09-21T12:33:49.388491+0000","last_became_active":"2024-09-21T12:32:14.242339+0000","last_became_peered":"2024-09-21T12:32:14.242339+0000","last_unstale":"2024-09-21T12:33:49.388491+0000","last_undegraded":"2024-09-21T12:33:49.388491+0000","last_fullsized":"2024-09-21T12:33:49.388491+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:32:19.786928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:19.786928+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:22.101928+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":715,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.384856+0000","last_change":"2024-09-21T12:32:38.421009+0000","last_active":"2024-09-21T12:33:49.384856+0000","last_peered":"2024-09-21T12:33:49.384856+0000","last_clean":"2024-09-21T12:33:49.384856+0000","last_became_active":"2024-09-21T12:32:38.416992+0000","last_became_peered":"2024-09-21T12:32:38.416992+0000","last_unstale":"2024-09-21T12:33:49.384856+0000","last_undegraded":"2024-09-21T12:33:49.384856+0000","last_fullsized":"2024-09-21T12:33:49.384856+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:43.545197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:43.545197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:16.354458+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":670,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.390799+0000","last_change":"2024-09-21T12:32:58.720897+0000","last_active":"2024-09-21T12:33:49.390799+0000","last_peered":"2024-09-21T12:33:49.390799+0000","last_clean":"2024-09-21T12:33:49.390799+0000","last_became_active":"2024-09-21T12:20:19.311947+0000","last_became_peered":"2024-09-21T12:20:19.311947+0000","last_unstale":"2024-09-21T12:33:49.390799+0000","last_undegraded":"2024-09-21T12:33:49.390799+0000","last_fullsized":"2024-09-21T12:33:49.390799+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:58.720834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:32:58.720834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:22.733283+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":704,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.393264+0000","last_change":"2024-09-21T12:32:38.414319+0000","last_active":"2024-09-21T12:33:49.393264+0000","last_peered":"2024-09-21T12:33:49.393264+0000","last_clean":"2024-09-21T12:33:49.393264+0000","last_became_active":"2024-09-21T12:32:38.413983+0000","last_became_peered":"2024-09-21T12:32:38.413983+0000","last_unstale":"2024-09-21T12:33:49.393264+0000","last_undegraded":"2024-09-21T12:33:49.393264+0000","last_fullsized":"2024-09-21T12:33:49.393264+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:11.105583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:11.105583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:45.271665+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"76'79","reported_seq":755,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.383952+0000","last_change":"2024-09-21T12:31:03.293976+0000","last_active":"2024-09-21T12:33:49.383952+0000","last_peered":"2024-09-21T12:33:49.383952+0000","last_clean":"2024-09-21T12:33:49.383952+0000","last_became_active":"2024-09-21T12:26:54.084939+0000","last_became_peered":"2024-09-21T12:26:54.084939+0000","last_unstale":"2024-09-21T12:33:49.383952+0000","last_undegraded":"2024-09-21T12:33:49.383952+0000","last_fullsized":"2024-09-21T12:33:49.383952+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:31:03.293909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:31:03.293909+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:45.453062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":441,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.385352+0000","last_change":"2024-09-21T12:32:02.034771+0000","last_active":"2024-09-21T12:33:49.385352+0000","last_peered":"2024-09-21T12:33:49.385352+0000","last_clean":"2024-09-21T12:33:49.385352+0000","last_became_active":"2024-09-21T12:32:02.027460+0000","last_became_peered":"2024-09-21T12:32:02.027460+0000","last_unstale":"2024-09-21T12:33:49.385352+0000","last_undegraded":"2024-09-21T12:33:49.385352+0000","last_fullsized":"2024-09-21T12:33:49.385352+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:28:42.511874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:42.511874+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:15.356193+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":710,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.383776+0000","last_change":"2024-09-21T12:32:17.813348+0000","last_active":"2024-09-21T12:33:49.383776+0000","last_peered":"2024-09-21T12:33:49.383776+0000","last_clean":"2024-09-21T12:33:49.383776+0000","last_became_active":"2024-09-21T12:32:14.236220+0000","last_became_peered":"2024-09-21T12:32:14.236220+0000","last_unstale":"2024-09-21T12:33:49.383776+0000","last_undegraded":"2024-09-21T12:33:49.383776+0000","last_fullsized":"2024-09-21T12:33:49.383776+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:17.813293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:17.813293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"316'10","reported_seq":167,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.388516+0000","last_change":"2024-09-21T12:31:50.050850+0000","last_active":"2024-09-21T12:33:49.388516+0000","last_peered":"2024-09-21T12:33:49.388516+0000","last_clean":"2024-09-21T12:33:49.388516+0000","last_became_active":"2024-09-21T12:31:50.040167+0000","last_became_peered":"2024-09-21T12:31:50.040167+0000","last_unstale":"2024-09-21T12:33:49.388516+0000","last_undegraded":"2024-09-21T12:33:49.388516+0000","last_fullsized":"2024-09-21T12:33:49.388516+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:28:43.853712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:43.853712+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:13.732171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":78,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.376182+0000","last_change":"2024-09-21T12:33:05.370312+0000","last_active":"2024-09-21T12:33:49.376182+0000","last_peered":"2024-09-21T12:33:49.376182+0000","last_clean":"2024-09-21T12:33:49.376182+0000","last_became_active":"2024-09-21T12:32:38.425776+0000","last_became_peered":"2024-09-21T12:32:38.425776+0000","last_unstale":"2024-09-21T12:33:49.376182+0000","last_undegraded":"2024-09-21T12:33:49.376182+0000","last_fullsized":"2024-09-21T12:33:49.376182+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:05.370263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:05.370263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:16.526109+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"76'79","reported_seq":492,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.386680+0000","last_change":"2024-09-21T12:33:01.101159+0000","last_active":"2024-09-21T12:33:49.386680+0000","last_peered":"2024-09-21T12:33:49.386680+0000","last_clean":"2024-09-21T12:33:49.386680+0000","last_became_active":"2024-09-21T12:32:02.034387+0000","last_became_peered":"2024-09-21T12:32:02.034387+0000","last_unstale":"2024-09-21T12:33:49.386680+0000","last_undegraded":"2024-09-21T12:33:49.386680+0000","last_fullsized":"2024-09-21T12:33:49.386680+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:01.101109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:01.101109+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:08.644169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":750,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.389026+0000","last_change":"2024-09-21T12:33:31.734043+0000","last_active":"2024-09-21T12:33:49.389026+0000","last_peered":"2024-09-21T12:33:49.389026+0000","last_clean":"2024-09-21T12:33:49.389026+0000","last_became_active":"2024-09-21T12:31:39.058798+0000","last_became_peered":"2024-09-21T12:31:39.058798+0000","last_unstale":"2024-09-21T12:33:49.389026+0000","last_undegraded":"2024-09-21T12:33:49.389026+0000","last_fullsized":"2024-09-21T12:33:49.389026+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:33:31.733776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:31.733776+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:58.954899+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":707,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.383869+0000","last_change":"2024-09-21T12:32:38.418862+0000","last_active":"2024-09-21T12:33:49.383869+0000","last_peered":"2024-09-21T12:33:49.383869+0000","last_clean":"2024-09-21T12:33:49.383869+0000","last_became_active":"2024-09-21T12:32:38.418517+0000","last_became_peered":"2024-09-21T12:32:38.418517+0000","last_unstale":"2024-09-21T12:33:49.383869+0000","last_undegraded":"2024-09-21T12:33:49.383869+0000","last_fullsized":"2024-09-21T12:33:49.383869+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:32.630261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:32.630261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:17.324562+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":843,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.394398+0000","last_change":"2024-09-21T12:31:07.510064+0000","last_active":"2024-09-21T12:33:49.394398+0000","last_peered":"2024-09-21T12:33:49.394398+0000","last_clean":"2024-09-21T12:33:49.394398+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:33:49.394398+0000","last_undegraded":"2024-09-21T12:33:49.394398+0000","last_fullsized":"2024-09-21T12:33:49.394398+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:31:07.509980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:31:07.509980+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:42.079733+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":675,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.385767+0000","last_change":"2024-09-21T12:31:52.608520+0000","last_active":"2024-09-21T12:33:49.385767+0000","last_peered":"2024-09-21T12:33:49.385767+0000","last_clean":"2024-09-21T12:33:49.385767+0000","last_became_active":"2024-09-21T12:31:50.037934+0000","last_became_peered":"2024-09-21T12:31:50.037934+0000","last_unstale":"2024-09-21T12:33:49.385767+0000","last_undegraded":"2024-09-21T12:33:49.385767+0000","last_fullsized":"2024-09-21T12:33:49.385767+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:52.608467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:31:52.608467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:17.049532+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"76'79","reported_seq":356,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.389395+0000","last_change":"2024-09-21T12:33:07.561773+0000","last_active":"2024-09-21T12:33:49.389395+0000","last_peered":"2024-09-21T12:33:49.389395+0000","last_clean":"2024-09-21T12:33:49.389395+0000","last_became_active":"2024-09-21T12:27:16.940347+0000","last_became_peered":"2024-09-21T12:27:16.940347+0000","last_unstale":"2024-09-21T12:33:49.389395+0000","last_undegraded":"2024-09-21T12:33:49.389395+0000","last_fullsized":"2024-09-21T12:33:49.389395+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:07.561702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:07.561702+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:34.783593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":82,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.375009+0000","last_change":"2024-09-21T12:32:38.448447+0000","last_active":"2024-09-21T12:33:49.375009+0000","last_peered":"2024-09-21T12:33:49.375009+0000","last_clean":"2024-09-21T12:33:49.375009+0000","last_became_active":"2024-09-21T12:32:38.421008+0000","last_became_peered":"2024-09-21T12:32:38.421008+0000","last_unstale":"2024-09-21T12:33:49.375009+0000","last_undegraded":"2024-09-21T12:33:49.375009+0000","last_fullsized":"2024-09-21T12:33:49.375009+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:32:30.775392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:30.775392+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":78,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.375245+0000","last_change":"2024-09-21T12:33:30.406961+0000","last_active":"2024-09-21T12:33:49.375245+0000","last_peered":"2024-09-21T12:33:49.375245+0000","last_clean":"2024-09-21T12:33:49.375245+0000","last_became_active":"2024-09-21T12:32:38.432970+0000","last_became_peered":"2024-09-21T12:32:38.432970+0000","last_unstale":"2024-09-21T12:33:49.375245+0000","last_undegraded":"2024-09-21T12:33:49.375245+0000","last_fullsized":"2024-09-21T12:33:49.375245+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:30.406915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:30.406915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:59.596160+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":680,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.389254+0000","last_change":"2024-09-21T12:31:51.006655+0000","last_active":"2024-09-21T12:33:49.389254+0000","last_peered":"2024-09-21T12:33:49.389254+0000","last_clean":"2024-09-21T12:33:49.389254+0000","last_became_active":"2024-09-21T12:31:50.040754+0000","last_became_peered":"2024-09-21T12:31:50.040754+0000","last_unstale":"2024-09-21T12:33:49.389254+0000","last_undegraded":"2024-09-21T12:33:49.389254+0000","last_fullsized":"2024-09-21T12:33:49.389254+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:51.006611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:31:51.006611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:17.310752+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":146,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.389684+0000","last_change":"2024-09-21T12:31:53.359206+0000","last_active":"2024-09-21T12:33:49.389684+0000","last_peered":"2024-09-21T12:33:49.389684+0000","last_clean":"2024-09-21T12:33:49.389684+0000","last_became_active":"2024-09-21T12:31:50.036150+0000","last_became_peered":"2024-09-21T12:31:50.036150+0000","last_unstale":"2024-09-21T12:33:49.389684+0000","last_undegraded":"2024-09-21T12:33:49.389684+0000","last_fullsized":"2024-09-21T12:33:49.389684+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:53.359159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:31:53.359159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:44.786164+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"76'79","reported_seq":373,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.394420+0000","last_change":"2024-09-21T12:32:24.750378+0000","last_active":"2024-09-21T12:33:49.394420+0000","last_peered":"2024-09-21T12:33:49.394420+0000","last_clean":"2024-09-21T12:33:49.394420+0000","last_became_active":"2024-09-21T12:27:16.921880+0000","last_became_peered":"2024-09-21T12:27:16.921880+0000","last_unstale":"2024-09-21T12:33:49.394420+0000","last_undegraded":"2024-09-21T12:33:49.394420+0000","last_fullsized":"2024-09-21T12:33:49.394420+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:24.750320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:24.750320+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:54.665474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":172,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.388902+0000","last_change":"2024-09-21T12:33:02.544972+0000","last_active":"2024-09-21T12:33:49.388902+0000","last_peered":"2024-09-21T12:33:49.388902+0000","last_clean":"2024-09-21T12:33:49.388902+0000","last_became_active":"2024-09-21T12:31:39.058221+0000","last_became_peered":"2024-09-21T12:31:39.058221+0000","last_unstale":"2024-09-21T12:33:49.388902+0000","last_undegraded":"2024-09-21T12:33:49.388902+0000","last_fullsized":"2024-09-21T12:33:49.388902+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:33:02.544885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:02.544885+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:21.643673+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":715,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.385435+0000","last_change":"2024-09-21T12:32:51.311159+0000","last_active":"2024-09-21T12:33:49.385435+0000","last_peered":"2024-09-21T12:33:49.385435+0000","last_clean":"2024-09-21T12:33:49.385435+0000","last_became_active":"2024-09-21T12:32:38.417928+0000","last_became_peered":"2024-09-21T12:32:38.417928+0000","last_unstale":"2024-09-21T12:33:49.385435+0000","last_undegraded":"2024-09-21T12:33:49.385435+0000","last_fullsized":"2024-09-21T12:33:49.385435+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:51.311112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:51.311112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:13.393300+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":670,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.390846+0000","last_change":"2024-09-21T12:32:29.582857+0000","last_active":"2024-09-21T12:33:49.390846+0000","last_peered":"2024-09-21T12:33:49.390846+0000","last_clean":"2024-09-21T12:33:49.390846+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:33:49.390846+0000","last_undegraded":"2024-09-21T12:33:49.390846+0000","last_fullsized":"2024-09-21T12:33:49.390846+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:29.582800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:32:29.582800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:33:51.045263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":661,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.383122+0000","last_change":"2024-09-21T12:28:41.864711+0000","last_active":"2024-09-21T12:33:49.383122+0000","last_peered":"2024-09-21T12:33:49.383122+0000","last_clean":"2024-09-21T12:33:49.383122+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:33:49.383122+0000","last_undegraded":"2024-09-21T12:33:49.383122+0000","last_fullsized":"2024-09-21T12:33:49.383122+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:41.864630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:41.864630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"76'79","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.392776+0000","last_change":"2024-09-21T12:33:05.658628+0000","last_active":"2024-09-21T12:33:49.392776+0000","last_peered":"2024-09-21T12:33:49.392776+0000","last_clean":"2024-09-21T12:33:49.392776+0000","last_became_active":"2024-09-21T12:32:38.414108+0000","last_became_peered":"2024-09-21T12:32:38.414108+0000","last_unstale":"2024-09-21T12:33:49.392776+0000","last_undegraded":"2024-09-21T12:33:49.392776+0000","last_fullsized":"2024-09-21T12:33:49.392776+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:05.658561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:05.658561+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:32.822687+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":713,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.390080+0000","last_change":"2024-09-21T12:33:27.793529+0000","last_active":"2024-09-21T12:33:49.390080+0000","last_peered":"2024-09-21T12:33:49.390080+0000","last_clean":"2024-09-21T12:33:49.390080+0000","last_became_active":"2024-09-21T12:20:17.694794+0000","last_became_peered":"2024-09-21T12:20:17.694794+0000","last_unstale":"2024-09-21T12:33:49.390080+0000","last_undegraded":"2024-09-21T12:33:49.390080+0000","last_fullsized":"2024-09-21T12:33:49.390080+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:33:27.793452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:27.793452+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:38.100197+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":695,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.395389+0000","last_change":"2024-09-21T12:29:54.303655+0000","last_active":"2024-09-21T12:33:49.395389+0000","last_peered":"2024-09-21T12:33:49.395389+0000","last_clean":"2024-09-21T12:33:49.395389+0000","last_became_active":"2024-09-21T12:20:15.295968+0000","last_became_peered":"2024-09-21T12:20:15.295968+0000","last_unstale":"2024-09-21T12:33:49.395389+0000","last_undegraded":"2024-09-21T12:33:49.395389+0000","last_fullsized":"2024-09-21T12:33:49.395389+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:54.303590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:54.303590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:39.147758+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":683,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.383245+0000","last_change":"2024-09-21T12:32:38.416206+0000","last_active":"2024-09-21T12:33:49.383245+0000","last_peered":"2024-09-21T12:33:49.383245+0000","last_clean":"2024-09-21T12:33:49.383245+0000","last_became_active":"2024-09-21T12:32:38.415887+0000","last_became_peered":"2024-09-21T12:32:38.415887+0000","last_unstale":"2024-09-21T12:33:49.383245+0000","last_undegraded":"2024-09-21T12:33:49.383245+0000","last_fullsized":"2024-09-21T12:33:49.383245+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:23:57.018749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:23:57.018749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:41.404317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":666,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.391011+0000","last_change":"2024-09-21T12:33:08.711630+0000","last_active":"2024-09-21T12:33:49.391011+0000","last_peered":"2024-09-21T12:33:49.391011+0000","last_clean":"2024-09-21T12:33:49.391011+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:33:49.391011+0000","last_undegraded":"2024-09-21T12:33:49.391011+0000","last_fullsized":"2024-09-21T12:33:49.391011+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:08.711439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:08.711439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:26.365998+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"76'79","reported_seq":349,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.384797+0000","last_change":"2024-09-21T12:32:20.425343+0000","last_active":"2024-09-21T12:33:49.384797+0000","last_peered":"2024-09-21T12:33:49.384797+0000","last_clean":"2024-09-21T12:33:49.384797+0000","last_became_active":"2024-09-21T12:27:16.914713+0000","last_became_peered":"2024-09-21T12:27:16.914713+0000","last_unstale":"2024-09-21T12:33:49.384797+0000","last_undegraded":"2024-09-21T12:33:49.384797+0000","last_fullsized":"2024-09-21T12:33:49.384797+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:20.417296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:20.417296+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":686,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.388040+0000","last_change":"2024-09-21T12:33:02.665954+0000","last_active":"2024-09-21T12:33:49.388040+0000","last_peered":"2024-09-21T12:33:49.388040+0000","last_clean":"2024-09-21T12:33:49.388040+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:33:49.388040+0000","last_undegraded":"2024-09-21T12:33:49.388040+0000","last_fullsized":"2024-09-21T12:33:49.388040+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'8","last_scrub_stamp":"2024-09-21T12:33:02.665881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:02.665881+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:15.567861+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":695,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.388996+0000","last_change":"2024-09-21T12:31:57.681487+0000","last_active":"2024-09-21T12:33:49.388996+0000","last_peered":"2024-09-21T12:33:49.388996+0000","last_clean":"2024-09-21T12:33:49.388996+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:33:49.388996+0000","last_undegraded":"2024-09-21T12:33:49.388996+0000","last_fullsized":"2024-09-21T12:33:49.388996+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:31:57.681417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:31:57.681417+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:19.076624+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":699,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.376717+0000","last_change":"2024-09-21T12:32:38.422189+0000","last_active":"2024-09-21T12:33:49.376717+0000","last_peered":"2024-09-21T12:33:49.376717+0000","last_clean":"2024-09-21T12:33:49.376717+0000","last_became_active":"2024-09-21T12:32:38.421807+0000","last_became_peered":"2024-09-21T12:32:38.421807+0000","last_unstale":"2024-09-21T12:33:49.376717+0000","last_undegraded":"2024-09-21T12:33:49.376717+0000","last_fullsized":"2024-09-21T12:33:49.376717+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:59.216442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:29:59.216442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:38.893095+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":687,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.383461+0000","last_change":"2024-09-21T12:32:38.419151+0000","last_active":"2024-09-21T12:33:49.383461+0000","last_peered":"2024-09-21T12:33:49.383461+0000","last_clean":"2024-09-21T12:33:49.383461+0000","last_became_active":"2024-09-21T12:32:38.417062+0000","last_became_peered":"2024-09-21T12:32:38.417062+0000","last_unstale":"2024-09-21T12:33:49.383461+0000","last_undegraded":"2024-09-21T12:33:49.383461+0000","last_fullsized":"2024-09-21T12:33:49.383461+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:41.537786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:41.537786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:09.342621+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"76'79","reported_seq":70,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.375085+0000","last_change":"2024-09-21T12:32:38.432700+0000","last_active":"2024-09-21T12:33:49.375085+0000","last_peered":"2024-09-21T12:33:49.375085+0000","last_clean":"2024-09-21T12:33:49.375085+0000","last_became_active":"2024-09-21T12:32:38.432302+0000","last_became_peered":"2024-09-21T12:32:38.432302+0000","last_unstale":"2024-09-21T12:33:49.375085+0000","last_undegraded":"2024-09-21T12:33:49.375085+0000","last_fullsized":"2024-09-21T12:33:49.375085+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:27:29.341268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:27:29.341268+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:34:43.286878+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":695,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.388832+0000","last_change":"2024-09-21T12:32:53.520137+0000","last_active":"2024-09-21T12:33:49.388832+0000","last_peered":"2024-09-21T12:33:49.388832+0000","last_clean":"2024-09-21T12:33:49.388832+0000","last_became_active":"2024-09-21T12:32:38.426822+0000","last_became_peered":"2024-09-21T12:32:38.426822+0000","last_unstale":"2024-09-21T12:33:49.388832+0000","last_undegraded":"2024-09-21T12:33:49.388832+0000","last_fullsized":"2024-09-21T12:33:49.388832+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:53.520065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:32:53.520065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:16.034350+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":692,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.388303+0000","last_change":"2024-09-21T12:31:39.052864+0000","last_active":"2024-09-21T12:33:49.388303+0000","last_peered":"2024-09-21T12:33:49.388303+0000","last_clean":"2024-09-21T12:33:49.388303+0000","last_became_active":"2024-09-21T12:31:39.052486+0000","last_became_peered":"2024-09-21T12:31:39.052486+0000","last_unstale":"2024-09-21T12:33:49.388303+0000","last_undegraded":"2024-09-21T12:33:49.388303+0000","last_fullsized":"2024-09-21T12:33:49.388303+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:15.071234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:15.071234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:19.718425+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":707,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.377991+0000","last_change":"2024-09-21T12:30:02.290006+0000","last_active":"2024-09-21T12:33:49.377991+0000","last_peered":"2024-09-21T12:33:49.377991+0000","last_clean":"2024-09-21T12:33:49.377991+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:33:49.377991+0000","last_undegraded":"2024-09-21T12:33:49.377991+0000","last_fullsized":"2024-09-21T12:33:49.377991+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-21T12:30:02.289930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:30:02.289930+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:41.910865+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":698,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T12:33:49.385854+0000","last_change":"2024-09-21T12:33:36.112295+0000","last_active":"2024-09-21T12:33:49.385854+0000","last_peered":"2024-09-21T12:33:49.385854+0000","last_clean":"2024-09-21T12:33:49.385854+0000","last_became_active":"2024-09-21T12:20:19.308247+0000","last_became_peered":"2024-09-21T12:20:19.308247+0000","last_unstale":"2024-09-21T12:33:49.385854+0000","last_undegraded":"2024-09-21T12:33:49.385854+0000","last_fullsized":"2024-09-21T12:33:49.385854+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:36.112155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:33:36.112155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:34:57.742371+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":7203258368,"num_objects":109913,"num_object_clones":0,"num_object_copies":329739,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109913,"num_write_kb":7034432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1027,"num_bytes_recovered":67305472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21605253120,"data_stored":21605253120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75213,"ondisk_log_size":75213,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1106,"num_read_kb":863,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":386,"num_bytes_recovered":7524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":684,"ondisk_log_size":684,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1058,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":810,"num_read_kb":805,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":603785,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":5,"num_write":2,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":13417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2449,"ondisk_log_size":2449,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234151,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2346420,"kb_used_data":1737700,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":91398732,"statfs":{"total":95995035648,"available":93592301568,"internally_reserved":0,"allocated":1779404800,"data_stored":1779019689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":623311306},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463087,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3682784,"kb_used_data":3083792,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":90062368,"statfs":{"total":95995035648,"available":92223864832,"internally_reserved":0,"allocated":3157803008,"data_stored":3157437184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":613349830},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724722,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2727740,"kb_used_data":2150760,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":91017412,"statfs":{"total":95995035648,"available":93201829888,"internally_reserved":0,"allocated":2202378240,"data_stored":2201989969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953658,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3676856,"kb_used_data":3492968,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":90068296,"statfs":{"total":95995035648,"available":92229935104,"internally_reserved":0,"allocated":3576799232,"data_stored":3576409001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149888,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2842948,"kb_used_data":2175924,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":90902204,"statfs":{"total":95995035648,"available":93083856896,"internally_reserved":0,"allocated":2228146176,"data_stored":2227745262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":683014599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411526,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3160484,"kb_used_data":2623956,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":90584668,"statfs":{"total":95995035648,"available":92758700032,"internally_reserved":0,"allocated":2686930944,"data_stored":2686561500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575052,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2007380,"kb_used_data":1322308,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":91737772,"statfs":{"total":95995035648,"available":93939478528,"internally_reserved":0,"allocated":1354043392,"data_stored":1353625352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738578,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5056636,"kb_used_data":4403500,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":88688516,"statfs":{"total":95995035648,"available":90817040384,"internally_reserved":0,"allocated":4509184000,"data_stored":4508790868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,5,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4558290944,"data_stored":4558290944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1358888960,"data_stored":1358888960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2686844928,"data_stored":2686844928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2245918720,"data_stored":2245918720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3596156928,"data_stored":3596156928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2217017344,"data_stored":2217017344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3157721088,"data_stored":3157721088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1784414208,"data_stored":1784414208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:33:56.073 INFO:tasks.thrashosds.thrasher:Setting 4.3 to [6, 2, 0, 5, 3, 1] 2024-09-21T12:33:56.074 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '4.3', '6', '2', '0', '5', '3', '1'] 2024-09-21T12:33:56.074 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pg-upmap-items 4.3 6 2 0 5 3 1 2024-09-21T12:33:56.105 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:56.141 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:56.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:56.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:56.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:55 smithi183 ceph-mon[33277]: from='client.16032 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:33:56.242 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:56.246 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:55 smithi114 ceph-mon[30324]: from='client.16032 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:33:56.246 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:55 smithi114 ceph-mon[40867]: from='client.16032 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:33:56.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:56.284 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:56.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:56.385 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:56.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:56.419 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:56.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:56.520 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:56.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:56.558 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:56.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:56.659 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:56.668 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-21T12:33:56.668433+0000 min lat: 0.00124586 max lat: 2.02977 avg lat: 0.0110857 2024-09-21T12:33:56.669 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T12:33:56.669 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 80 16 115464 115448 90.1682 83.0625 0.00644256 0.0110857 2024-09-21T12:33:56.693 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:56.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:56.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:56.731 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T12:33:56.795 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:56.824 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:56.845 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:56.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:56.881 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:56.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:56.917 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:33:56.983 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:57.001 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:56 smithi114 ceph-mon[30324]: pgmap v702: 178 pgs: 178 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 904 op/s; 13 B/s, 0 objects/s recovering 2024-09-21T12:33:57.001 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:56 smithi114 ceph-mon[40867]: pgmap v702: 178 pgs: 178 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 904 op/s; 13 B/s, 0 objects/s recovering 2024-09-21T12:33:57.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:57.039 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:57.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:57.139 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:57.184 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:57.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:57.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:57.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:56 smithi183 ceph-mon[33277]: pgmap v702: 178 pgs: 178 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 904 op/s; 13 B/s, 0 objects/s recovering 2024-09-21T12:33:57.285 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:57.317 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:57.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:57.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:57.418 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:57.468 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:57.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:57.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:57.570 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:57.608 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:57.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:57.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:57.669 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 81 16 116652 116636 89.9716 74.25 0.00966541 0.0111104 2024-09-21T12:33:57.710 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:57.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:57.739 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:57.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:57.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:33:57 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:33:57.841 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:57.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:57.880 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:57.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:57.980 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:33:58.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:58.010 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:58.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:33:58.111 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:58.133 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:33:58.133 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:33:58.133 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:33:58.133 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:33:58.133 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:33:58.133 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:33:58.133 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:33:58.133 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:33:58.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:58.160 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:58.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:58.175 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:33:58.175 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:33:58.178 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:58 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3823432124' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:33:58.178 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:58 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3823432124' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:33:58.213 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:33:58.213 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:33:58.250 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:33:58.250 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:33:58.261 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:58.288 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:33:58.289 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:33:58.292 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:58.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:58.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:58.394 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:58.438 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:58.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:58.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:58.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:58 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3823432124' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:33:58.539 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:58.566 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:58.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:58.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:58.668 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:33:58.672 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 82 15 117801 117786 89.7471 71.875 0.00854543 0.011138 2024-09-21T12:33:58.714 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:58.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:58.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:33:58.816 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:58.846 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:58.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:58.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:58.947 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:33:58.985 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:58.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:58.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:33:59.087 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:33:59.101 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-21T12:33:59.104 INFO:teuthology.orchestra.run.smithi114.stderr:set 4.3 pg_upmap_items mapping to [6->2,0->5,3->1] 2024-09-21T12:33:59.122 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:59.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:59.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:33:59.224 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:33:59.259 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:59.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:59.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:33:59.360 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:33:59.392 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:33:59.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:59.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:33:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:59 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:59 smithi183 ceph-mon[33277]: pgmap v703: 178 pgs: 178 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.0 KiB/s rd, 72 MiB/s wr, 1.16k op/s; 11 B/s, 0 objects/s recovering 2024-09-21T12:33:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:59 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3811952688' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:33:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:59 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:33:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:59 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/935549427' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "4.3", "id": [6, 2, 0, 5, 3, 1]} : dispatch 2024-09-21T12:33:59.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:33:59 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "4.3", "id": [6, 2, 0, 5, 3, 1]} : dispatch 2024-09-21T12:33:59.494 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:59.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:59 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:59.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:59 smithi114 ceph-mon[40867]: pgmap v703: 178 pgs: 178 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.0 KiB/s rd, 72 MiB/s wr, 1.16k op/s; 11 B/s, 0 objects/s recovering 2024-09-21T12:33:59.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:59 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3811952688' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:33:59.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:59 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:33:59.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:59 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/935549427' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "4.3", "id": [6, 2, 0, 5, 3, 1]} : dispatch 2024-09-21T12:33:59.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:33:59 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "4.3", "id": [6, 2, 0, 5, 3, 1]} : dispatch 2024-09-21T12:33:59.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:59 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:59.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:59 smithi114 ceph-mon[30324]: pgmap v703: 178 pgs: 178 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.0 KiB/s rd, 72 MiB/s wr, 1.16k op/s; 11 B/s, 0 objects/s recovering 2024-09-21T12:33:59.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:59 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3811952688' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:33:59.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:59 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:33:59.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:59 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/935549427' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "4.3", "id": [6, 2, 0, 5, 3, 1]} : dispatch 2024-09-21T12:33:59.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:33:59 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "4.3", "id": [6, 2, 0, 5, 3, 1]} : dispatch 2024-09-21T12:33:59.528 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:59.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:59.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:59.630 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:59.664 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:59.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:59.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:33:59.672 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 83 16 119131 119115 89.6665 83.0625 0.0205509 0.0111479 2024-09-21T12:33:59.766 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:33:59.805 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:59.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:59.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:33:59.906 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:33:59.943 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:33:59.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:33:59.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:00.045 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:34:00.086 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:00.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:00.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:34:00.187 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:00.220 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:00.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:00.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:00.322 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:34:00.355 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:00.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:00.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:34:00.457 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:00.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:00 smithi183 ceph-mon[33277]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:34:00.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:00 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:34:00.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:00 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.3", "id": [6, 2, 0, 5, 3, 1]}]': finished 2024-09-21T12:34:00.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:00 smithi183 ceph-mon[33277]: osdmap e380: 8 total, 8 up, 8 in 2024-09-21T12:34:00.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:00 smithi183 ceph-mon[33277]: 5.f scrub starts 2024-09-21T12:34:00.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:00 smithi183 ceph-mon[33277]: 5.f scrub ok 2024-09-21T12:34:00.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:00 smithi183 ceph-mon[33277]: 1.17 scrub starts 2024-09-21T12:34:00.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:00 smithi183 ceph-mon[33277]: 1.17 scrub ok 2024-09-21T12:34:00.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:00.489 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:00.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:00.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:00 smithi114 ceph-mon[30324]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:34:00.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:00 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:34:00.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:00 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.3", "id": [6, 2, 0, 5, 3, 1]}]': finished 2024-09-21T12:34:00.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:00 smithi114 ceph-mon[30324]: osdmap e380: 8 total, 8 up, 8 in 2024-09-21T12:34:00.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:00 smithi114 ceph-mon[30324]: 5.f scrub starts 2024-09-21T12:34:00.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:00 smithi114 ceph-mon[30324]: 5.f scrub ok 2024-09-21T12:34:00.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:00 smithi114 ceph-mon[30324]: 1.17 scrub starts 2024-09-21T12:34:00.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:00 smithi114 ceph-mon[30324]: 1.17 scrub ok 2024-09-21T12:34:00.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:00 smithi114 ceph-mon[40867]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:34:00.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:00 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:34:00.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:00 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.3", "id": [6, 2, 0, 5, 3, 1]}]': finished 2024-09-21T12:34:00.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:00 smithi114 ceph-mon[40867]: osdmap e380: 8 total, 8 up, 8 in 2024-09-21T12:34:00.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:00 smithi114 ceph-mon[40867]: 5.f scrub starts 2024-09-21T12:34:00.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:00 smithi114 ceph-mon[40867]: 5.f scrub ok 2024-09-21T12:34:00.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:00 smithi114 ceph-mon[40867]: 1.17 scrub starts 2024-09-21T12:34:00.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:00 smithi114 ceph-mon[40867]: 1.17 scrub ok 2024-09-21T12:34:00.590 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:00.625 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:00.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:00.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:00.672 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 84 16 120505 120489 89.6213 85.875 0.0149437 0.0111536 2024-09-21T12:34:00.727 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:00.759 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:00.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:00.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:00.861 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:00.881 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:00.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:00.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:00.983 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:01.016 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:01.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:01.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:01.118 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:01.153 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:01.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:01.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:01.255 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:01.290 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:01.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:01.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:01.321 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:34:01.366 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:01 smithi114 ceph-mon[30324]: pgmap v705: 178 pgs: 178 active+clean; 7.1 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s; 10 B/s, 0 objects/s recovering 2024-09-21T12:34:01.366 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:01 smithi114 ceph-mon[30324]: 6.6 scrub starts 2024-09-21T12:34:01.366 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:01 smithi114 ceph-mon[30324]: 6.6 scrub ok 2024-09-21T12:34:01.366 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:01 smithi114 ceph-mon[30324]: 4.13 scrub starts 2024-09-21T12:34:01.366 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:01 smithi114 ceph-mon[30324]: 4.13 scrub ok 2024-09-21T12:34:01.367 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:01 smithi114 ceph-mon[30324]: 3.0 scrub starts 2024-09-21T12:34:01.367 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:01 smithi114 ceph-mon[30324]: 3.0 scrub ok 2024-09-21T12:34:01.367 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:01 smithi114 ceph-mon[40867]: pgmap v705: 178 pgs: 178 active+clean; 7.1 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s; 10 B/s, 0 objects/s recovering 2024-09-21T12:34:01.367 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:01 smithi114 ceph-mon[40867]: 6.6 scrub starts 2024-09-21T12:34:01.367 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:01 smithi114 ceph-mon[40867]: 6.6 scrub ok 2024-09-21T12:34:01.367 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:01 smithi114 ceph-mon[40867]: 4.13 scrub starts 2024-09-21T12:34:01.367 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:01 smithi114 ceph-mon[40867]: 4.13 scrub ok 2024-09-21T12:34:01.367 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:01 smithi114 ceph-mon[40867]: 3.0 scrub starts 2024-09-21T12:34:01.367 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:01 smithi114 ceph-mon[40867]: 3.0 scrub ok 2024-09-21T12:34:01.392 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:34:01.413 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:01.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:01.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:34:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:01 smithi183 ceph-mon[33277]: pgmap v705: 178 pgs: 178 active+clean; 7.1 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s; 10 B/s, 0 objects/s recovering 2024-09-21T12:34:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:01 smithi183 ceph-mon[33277]: 6.6 scrub starts 2024-09-21T12:34:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:01 smithi183 ceph-mon[33277]: 6.6 scrub ok 2024-09-21T12:34:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:01 smithi183 ceph-mon[33277]: 4.13 scrub starts 2024-09-21T12:34:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:01 smithi183 ceph-mon[33277]: 4.13 scrub ok 2024-09-21T12:34:01.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:01 smithi183 ceph-mon[33277]: 3.0 scrub starts 2024-09-21T12:34:01.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:01 smithi183 ceph-mon[33277]: 3.0 scrub ok 2024-09-21T12:34:01.515 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:01.546 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:01.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:01.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:01.648 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:01.673 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 85 16 121707 121691 89.4503 75.125 0.012345 0.0111752 2024-09-21T12:34:01.692 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:01.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:01.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:01.794 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:01.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:01.817 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:01.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:01.918 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:01.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:01.952 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:01.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:02.053 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:02.096 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:02.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:02.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:02.146 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:34:02.197 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:02.225 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T12:34:02.232 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:02.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:02.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:02.335 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:02.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:02.367 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:02.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:02.468 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:02.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:02 smithi183 ceph-mon[33277]: 10.d scrub starts 2024-09-21T12:34:02.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:02 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:34:02.503 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:02.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:02.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:02.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:02 smithi114 ceph-mon[40867]: 10.d scrub starts 2024-09-21T12:34:02.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:02 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:34:02.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:02 smithi114 ceph-mon[30324]: 10.d scrub starts 2024-09-21T12:34:02.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:02 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:34:02.604 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:02.636 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:02.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:02.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:02.673 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 86 16 122859 122843 89.2473 72 0.00950011 0.0112003 2024-09-21T12:34:02.738 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:02.779 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:02.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:02.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:02.880 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:02.900 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:34:02 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:34:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:34:02.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:02.915 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:02.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:03.016 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:03.037 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:03.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:03.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:03.139 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:34:03.177 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:03.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:03.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:34:03.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:03 smithi114 ceph-mon[40867]: 3.2 scrub starts 2024-09-21T12:34:03.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:03 smithi114 ceph-mon[40867]: 3.2 scrub ok 2024-09-21T12:34:03.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:03 smithi114 ceph-mon[40867]: pgmap v706: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.24k op/s; 10 B/s, 0 objects/s recovering 2024-09-21T12:34:03.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:03 smithi114 ceph-mon[40867]: 3.12 scrub starts 2024-09-21T12:34:03.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:03 smithi114 ceph-mon[40867]: 3.12 scrub ok 2024-09-21T12:34:03.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:03 smithi114 ceph-mon[30324]: 3.2 scrub starts 2024-09-21T12:34:03.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:03 smithi114 ceph-mon[30324]: 3.2 scrub ok 2024-09-21T12:34:03.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:03 smithi114 ceph-mon[30324]: pgmap v706: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.24k op/s; 10 B/s, 0 objects/s recovering 2024-09-21T12:34:03.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:03 smithi114 ceph-mon[30324]: 3.12 scrub starts 2024-09-21T12:34:03.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:03 smithi114 ceph-mon[30324]: 3.12 scrub ok 2024-09-21T12:34:03.279 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:03.319 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:03.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:03.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:03.421 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:03.456 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:03.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:03.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:03.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:03 smithi183 ceph-mon[33277]: 3.2 scrub starts 2024-09-21T12:34:03.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:03 smithi183 ceph-mon[33277]: 3.2 scrub ok 2024-09-21T12:34:03.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:03 smithi183 ceph-mon[33277]: pgmap v706: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.24k op/s; 10 B/s, 0 objects/s recovering 2024-09-21T12:34:03.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:03 smithi183 ceph-mon[33277]: 3.12 scrub starts 2024-09-21T12:34:03.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:03 smithi183 ceph-mon[33277]: 3.12 scrub ok 2024-09-21T12:34:03.558 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:03.595 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:03.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:03.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:03.673 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 87 15 123990 123975 89.0347 70.75 0.0101096 0.0112274 2024-09-21T12:34:03.697 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:03.722 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:03.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:03.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:03.801 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:34:03.819 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:34:03.819 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:34:03.819 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:34:03.819 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:34:03.820 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:34:03.824 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:03.848 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:03.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:03.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:03.951 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:03.979 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:03.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:03.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:04.080 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:04.126 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:04.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:04.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:04.227 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:04.269 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:04.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:04.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:04.371 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:04.400 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:04.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:04.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:04.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:04 smithi183 ceph-mon[33277]: 5.6 scrub starts 2024-09-21T12:34:04.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:04 smithi183 ceph-mon[33277]: 5.6 scrub ok 2024-09-21T12:34:04.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:04 smithi183 ceph-mon[33277]: 10.7 scrub starts 2024-09-21T12:34:04.501 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:04 smithi114 ceph-mon[30324]: 5.6 scrub starts 2024-09-21T12:34:04.501 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:04 smithi114 ceph-mon[30324]: 5.6 scrub ok 2024-09-21T12:34:04.501 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:04 smithi114 ceph-mon[30324]: 10.7 scrub starts 2024-09-21T12:34:04.502 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:04 smithi114 ceph-mon[40867]: 5.6 scrub starts 2024-09-21T12:34:04.502 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:04 smithi114 ceph-mon[40867]: 5.6 scrub ok 2024-09-21T12:34:04.502 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:04 smithi114 ceph-mon[40867]: 10.7 scrub starts 2024-09-21T12:34:04.502 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:04.530 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:04.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:04.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:04.555 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T12:34:04.632 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:04.659 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:04.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:04.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:04.676 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 88 16 125064 125048 88.782 67.0625 0.0221799 0.0112584 2024-09-21T12:34:04.761 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:04.788 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:34:04.791 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:04.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:04.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:04.894 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:04.924 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:04.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:04.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:05.026 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:05.104 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:05.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:05.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:05.206 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:05.231 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:05.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:05.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:05.332 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:05.365 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:05.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:05.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:05.376 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:34:05.376 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:34:05.376 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:34:05.376 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:34:05.377 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:34:05.377 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:34:05.377 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:34:05.377 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:34:05.377 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:05 smithi114 ceph-mon[40867]: 10.d scrub ok 2024-09-21T12:34:05.377 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:05 smithi114 ceph-mon[40867]: pgmap v707: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.5 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-21T12:34:05.377 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:05 smithi114 ceph-mon[40867]: 10.4 scrub starts 2024-09-21T12:34:05.378 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:05 smithi114 ceph-mon[30324]: 10.d scrub ok 2024-09-21T12:34:05.378 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:05 smithi114 ceph-mon[30324]: pgmap v707: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.5 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-21T12:34:05.378 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:05 smithi114 ceph-mon[30324]: 10.4 scrub starts 2024-09-21T12:34:05.401 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:34:05.401 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:34:05.433 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:34:05.434 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:34:05.467 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:05.468 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:34:05.469 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:34:05.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:05 smithi183 ceph-mon[33277]: 10.d scrub ok 2024-09-21T12:34:05.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:05 smithi183 ceph-mon[33277]: pgmap v707: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.5 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-21T12:34:05.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:05 smithi183 ceph-mon[33277]: 10.4 scrub starts 2024-09-21T12:34:05.498 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:05.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:05.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:05.506 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:34:05.507 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:34:05.600 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:05.631 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:05.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:05.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:05.676 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 89 16 125998 125982 88.4404 58.375 0.00848076 0.011302 2024-09-21T12:34:05.733 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:05.769 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:05.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:05.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:05.871 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:05.901 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:05.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:05.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:06.003 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:06.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:06.047 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:06.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:06.148 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:06.174 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-21T12:34:06.182 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:06.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:06.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:06.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:06 smithi114 ceph-mon[40867]: 10.9 scrub starts 2024-09-21T12:34:06.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:06 smithi114 ceph-mon[40867]: 6.b scrub starts 2024-09-21T12:34:06.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:06 smithi114 ceph-mon[40867]: 6.b scrub ok 2024-09-21T12:34:06.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:06 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:34:06.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:06 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:34:06.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:06 smithi114 ceph-mon[40867]: osdmap e381: 8 total, 8 up, 8 in 2024-09-21T12:34:06.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:06 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/883338608' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:34:06.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:06 smithi114 ceph-mon[40867]: 6.1e scrub starts 2024-09-21T12:34:06.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:06 smithi114 ceph-mon[40867]: 6.1e scrub ok 2024-09-21T12:34:06.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:06 smithi114 ceph-mon[40867]: 10.7 scrub ok 2024-09-21T12:34:06.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:06 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/87769990' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:34:06.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:06 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:34:06.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:06 smithi114 ceph-mon[30324]: 10.9 scrub starts 2024-09-21T12:34:06.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:06 smithi114 ceph-mon[30324]: 6.b scrub starts 2024-09-21T12:34:06.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:06 smithi114 ceph-mon[30324]: 6.b scrub ok 2024-09-21T12:34:06.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:06 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:34:06.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:06 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:34:06.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:06 smithi114 ceph-mon[30324]: osdmap e381: 8 total, 8 up, 8 in 2024-09-21T12:34:06.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:06 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/883338608' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:34:06.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:06 smithi114 ceph-mon[30324]: 6.1e scrub starts 2024-09-21T12:34:06.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:06 smithi114 ceph-mon[30324]: 6.1e scrub ok 2024-09-21T12:34:06.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:06 smithi114 ceph-mon[30324]: 10.7 scrub ok 2024-09-21T12:34:06.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:06 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/87769990' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:34:06.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:06 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:34:06.284 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:06.311 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:06.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:06.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:06.413 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:06.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:06.448 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:06.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:06.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:06 smithi183 ceph-mon[33277]: 10.9 scrub starts 2024-09-21T12:34:06.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:06 smithi183 ceph-mon[33277]: 6.b scrub starts 2024-09-21T12:34:06.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:06 smithi183 ceph-mon[33277]: 6.b scrub ok 2024-09-21T12:34:06.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:06 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:34:06.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:06 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:34:06.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:06 smithi183 ceph-mon[33277]: osdmap e381: 8 total, 8 up, 8 in 2024-09-21T12:34:06.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:06 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/883338608' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:34:06.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:06 smithi183 ceph-mon[33277]: 6.1e scrub starts 2024-09-21T12:34:06.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:06 smithi183 ceph-mon[33277]: 6.1e scrub ok 2024-09-21T12:34:06.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:06 smithi183 ceph-mon[33277]: 10.7 scrub ok 2024-09-21T12:34:06.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:06 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/87769990' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:34:06.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:06 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:34:06.549 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:06.579 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:06.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:06.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:06.676 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Total time run: 90.0169 2024-09-21T12:34:06.677 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Total writes made: 127049 2024-09-21T12:34:06.677 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Write size: 65536 2024-09-21T12:34:06.677 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Object size: 65536 2024-09-21T12:34:06.677 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Bandwidth (MB/sec): 88.2119 2024-09-21T12:34:06.677 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev Bandwidth: 28.7821 2024-09-21T12:34:06.677 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max bandwidth (MB/sec): 190.438 2024-09-21T12:34:06.677 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min bandwidth (MB/sec): 0 2024-09-21T12:34:06.677 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Average IOPS: 1411 2024-09-21T12:34:06.677 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev IOPS: 460.513 2024-09-21T12:34:06.678 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max IOPS: 3047 2024-09-21T12:34:06.678 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min IOPS: 0 2024-09-21T12:34:06.678 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Average Latency(s): 0.0113315 2024-09-21T12:34:06.678 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev Latency(s): 0.0281733 2024-09-21T12:34:06.678 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max latency(s): 2.02977 2024-09-21T12:34:06.678 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min latency(s): 0.00124586 2024-09-21T12:34:06.680 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T12:34:06.682 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:06.714 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:06.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:06.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:06.816 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:06.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:06.852 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:06.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:06.953 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:06.986 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:06.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:06.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:07.087 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:07.124 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:07.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:07.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:07.143 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:34:07.192 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T12:34:07.193 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:34:07.193 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T12:34:07.193 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.000000 2024-09-21T12:34:07.193 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 2 0 2024-09-21T12:34:07.225 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:07.256 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:07.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:07.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:07.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:07 smithi114 ceph-mon[30324]: pgmap v709: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.5 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 58 MiB/s wr, 926 op/s 2024-09-21T12:34:07.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:07 smithi114 ceph-mon[30324]: 6.1b scrub starts 2024-09-21T12:34:07.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:07 smithi114 ceph-mon[30324]: 6.1b scrub ok 2024-09-21T12:34:07.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:07 smithi114 ceph-mon[30324]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:34:07.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:07 smithi114 ceph-mon[30324]: Cluster is now healthy 2024-09-21T12:34:07.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:07 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:34:07.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:07 smithi114 ceph-mon[30324]: osdmap e382: 8 total, 8 up, 8 in 2024-09-21T12:34:07.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:07 smithi114 ceph-mon[30324]: 1.15 scrub starts 2024-09-21T12:34:07.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:07 smithi114 ceph-mon[30324]: 1.15 scrub ok 2024-09-21T12:34:07.301 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:07 smithi114 ceph-mon[40867]: pgmap v709: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.5 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 58 MiB/s wr, 926 op/s 2024-09-21T12:34:07.301 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:07 smithi114 ceph-mon[40867]: 6.1b scrub starts 2024-09-21T12:34:07.301 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:07 smithi114 ceph-mon[40867]: 6.1b scrub ok 2024-09-21T12:34:07.301 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:07 smithi114 ceph-mon[40867]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:34:07.301 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:07 smithi114 ceph-mon[40867]: Cluster is now healthy 2024-09-21T12:34:07.301 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:07 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:34:07.302 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:07 smithi114 ceph-mon[40867]: osdmap e382: 8 total, 8 up, 8 in 2024-09-21T12:34:07.302 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:07 smithi114 ceph-mon[40867]: 1.15 scrub starts 2024-09-21T12:34:07.302 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:07 smithi114 ceph-mon[40867]: 1.15 scrub ok 2024-09-21T12:34:07.358 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:07.409 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:07.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:07.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:07.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:07 smithi183 ceph-mon[33277]: pgmap v709: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.5 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 58 MiB/s wr, 926 op/s 2024-09-21T12:34:07.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:07 smithi183 ceph-mon[33277]: 6.1b scrub starts 2024-09-21T12:34:07.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:07 smithi183 ceph-mon[33277]: 6.1b scrub ok 2024-09-21T12:34:07.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:07 smithi183 ceph-mon[33277]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:34:07.424 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:07 smithi183 ceph-mon[33277]: Cluster is now healthy 2024-09-21T12:34:07.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:07 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:34:07.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:07 smithi183 ceph-mon[33277]: osdmap e382: 8 total, 8 up, 8 in 2024-09-21T12:34:07.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:07 smithi183 ceph-mon[33277]: 1.15 scrub starts 2024-09-21T12:34:07.425 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:07 smithi183 ceph-mon[33277]: 1.15 scrub ok 2024-09-21T12:34:07.511 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:07.539 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:07.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:07.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:07.641 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:07.670 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:07.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:07.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:07.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:34:07 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:34:07.773 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:07.797 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:34:07.797 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:34:07.823 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:07.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:07.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:07.924 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:07.956 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:07.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:07.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:08.058 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:08.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:08.092 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:08.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:08.193 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:08.228 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:08.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:08.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:08.330 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:08.368 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:08.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:08.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:08.416 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:34:08.454 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:08 smithi114 ceph-mon[30324]: 10.9 scrub ok 2024-09-21T12:34:08.454 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:08 smithi114 ceph-mon[30324]: 10.4 scrub ok 2024-09-21T12:34:08.454 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:08 smithi114 ceph-mon[30324]: 10.1 scrub starts 2024-09-21T12:34:08.455 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:08 smithi114 ceph-mon[40867]: 10.9 scrub ok 2024-09-21T12:34:08.455 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:08 smithi114 ceph-mon[40867]: 10.4 scrub ok 2024-09-21T12:34:08.455 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:08 smithi114 ceph-mon[40867]: 10.1 scrub starts 2024-09-21T12:34:08.471 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:08.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:08 smithi183 ceph-mon[33277]: 10.9 scrub ok 2024-09-21T12:34:08.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:08 smithi183 ceph-mon[33277]: 10.4 scrub ok 2024-09-21T12:34:08.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:08 smithi183 ceph-mon[33277]: 10.1 scrub starts 2024-09-21T12:34:08.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:08.510 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:08.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:08.611 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:08.645 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:08.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:08.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:08.747 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:08.792 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:08.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:08.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:08.894 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:34:08.923 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:08.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:08.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:34:09.025 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:09.058 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:09.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:09.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:09.082 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T12:34:09.160 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:09.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:09.209 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:09.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:09.310 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:09.352 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:09.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:09.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:09.453 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:09.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:09.486 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:09.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:09.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:09 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:34:09.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:09 smithi183 ceph-mon[33277]: 5.0 scrub starts 2024-09-21T12:34:09.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:09 smithi183 ceph-mon[33277]: 5.0 scrub ok 2024-09-21T12:34:09.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:09 smithi183 ceph-mon[33277]: pgmap v711: 178 pgs: 178 active+clean; 7.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.4 KiB/s rd, 80 MiB/s wr, 1.33k op/s 2024-09-21T12:34:09.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:09 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:34:09.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:09 smithi114 ceph-mon[30324]: 5.0 scrub starts 2024-09-21T12:34:09.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:09 smithi114 ceph-mon[30324]: 5.0 scrub ok 2024-09-21T12:34:09.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:09 smithi114 ceph-mon[30324]: pgmap v711: 178 pgs: 178 active+clean; 7.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.4 KiB/s rd, 80 MiB/s wr, 1.33k op/s 2024-09-21T12:34:09.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:09 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:34:09.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:09 smithi114 ceph-mon[40867]: 5.0 scrub starts 2024-09-21T12:34:09.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:09 smithi114 ceph-mon[40867]: 5.0 scrub ok 2024-09-21T12:34:09.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:09 smithi114 ceph-mon[40867]: pgmap v711: 178 pgs: 178 active+clean; 7.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.4 KiB/s rd, 80 MiB/s wr, 1.33k op/s 2024-09-21T12:34:09.587 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:09.624 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:09.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:09.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:09.726 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:34:09.761 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:09.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:09.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:34:09.863 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:34:09.900 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:09.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:09.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:34:10.002 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:10.031 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:10.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:10.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:10.132 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:10.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:10.168 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:10.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:10.269 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:10.310 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:10.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:10.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:10.412 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:10.443 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:10.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:10.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:10.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:10 smithi183 ceph-mon[33277]: 10.5 scrub starts 2024-09-21T12:34:10.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:10 smithi183 ceph-mon[33277]: 10.1 scrub ok 2024-09-21T12:34:10.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:10 smithi183 ceph-mon[33277]: 10.6 scrub starts 2024-09-21T12:34:10.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:10 smithi183 ceph-mon[33277]: 10.b scrub starts 2024-09-21T12:34:10.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:10 smithi183 ceph-mon[33277]: 10.e scrub starts 2024-09-21T12:34:10.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:10 smithi114 ceph-mon[30324]: 10.5 scrub starts 2024-09-21T12:34:10.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:10 smithi114 ceph-mon[30324]: 10.1 scrub ok 2024-09-21T12:34:10.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:10 smithi114 ceph-mon[30324]: 10.6 scrub starts 2024-09-21T12:34:10.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:10 smithi114 ceph-mon[30324]: 10.b scrub starts 2024-09-21T12:34:10.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:10 smithi114 ceph-mon[30324]: 10.e scrub starts 2024-09-21T12:34:10.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:10 smithi114 ceph-mon[40867]: 10.5 scrub starts 2024-09-21T12:34:10.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:10 smithi114 ceph-mon[40867]: 10.1 scrub ok 2024-09-21T12:34:10.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:10 smithi114 ceph-mon[40867]: 10.6 scrub starts 2024-09-21T12:34:10.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:10 smithi114 ceph-mon[40867]: 10.b scrub starts 2024-09-21T12:34:10.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:10 smithi114 ceph-mon[40867]: 10.e scrub starts 2024-09-21T12:34:10.545 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:10.584 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:34:10.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:10.588 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:10.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:10.639 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T12:34:10.689 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:10.733 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:34:10.755 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:10.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:10.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:10.864 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:10.915 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:10.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:10.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:11.017 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:11.047 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:11.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:11.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:11.148 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:11.169 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:11.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:11.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:11.271 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:11.309 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:11.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:11.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:11.410 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:11.435 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:11.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:11.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:11.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:11 smithi183 ceph-mon[33277]: pgmap v712: 178 pgs: 178 active+clean; 7.7 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 767 B/s rd, 64 MiB/s wr, 1.11k op/s 2024-09-21T12:34:11.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:11 smithi183 ceph-mon[33277]: 4.17 scrub starts 2024-09-21T12:34:11.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:11 smithi183 ceph-mon[33277]: 4.17 scrub ok 2024-09-21T12:34:11.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:11 smithi183 ceph-mon[33277]: 6.d scrub starts 2024-09-21T12:34:11.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:11 smithi183 ceph-mon[33277]: 6.d scrub ok 2024-09-21T12:34:11.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:11 smithi114 ceph-mon[30324]: pgmap v712: 178 pgs: 178 active+clean; 7.7 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 767 B/s rd, 64 MiB/s wr, 1.11k op/s 2024-09-21T12:34:11.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:11 smithi114 ceph-mon[30324]: 4.17 scrub starts 2024-09-21T12:34:11.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:11 smithi114 ceph-mon[30324]: 4.17 scrub ok 2024-09-21T12:34:11.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:11 smithi114 ceph-mon[30324]: 6.d scrub starts 2024-09-21T12:34:11.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:11 smithi114 ceph-mon[30324]: 6.d scrub ok 2024-09-21T12:34:11.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:11 smithi114 ceph-mon[40867]: pgmap v712: 178 pgs: 178 active+clean; 7.7 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 767 B/s rd, 64 MiB/s wr, 1.11k op/s 2024-09-21T12:34:11.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:11 smithi114 ceph-mon[40867]: 4.17 scrub starts 2024-09-21T12:34:11.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:11 smithi114 ceph-mon[40867]: 4.17 scrub ok 2024-09-21T12:34:11.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:11 smithi114 ceph-mon[40867]: 6.d scrub starts 2024-09-21T12:34:11.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:11 smithi114 ceph-mon[40867]: 6.d scrub ok 2024-09-21T12:34:11.536 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:11.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:11.573 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:11.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:11.674 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:11.719 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:11.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:11.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:11.821 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:11.859 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:11.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:11.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:11.961 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:11.992 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:11.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:11.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:12.094 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:12.118 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:12.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:12.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:12.220 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:12.250 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:12.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:12.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:12.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:12 smithi114 ceph-mon[30324]: 10.e scrub ok 2024-09-21T12:34:12.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:12 smithi114 ceph-mon[30324]: 4.7 scrub starts 2024-09-21T12:34:12.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:12 smithi114 ceph-mon[30324]: 4.7 scrub ok 2024-09-21T12:34:12.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:12 smithi114 ceph-mon[30324]: 10.b scrub ok 2024-09-21T12:34:12.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:12 smithi114 ceph-mon[40867]: 10.e scrub ok 2024-09-21T12:34:12.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:12 smithi114 ceph-mon[40867]: 4.7 scrub starts 2024-09-21T12:34:12.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:12 smithi114 ceph-mon[40867]: 4.7 scrub ok 2024-09-21T12:34:12.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:12 smithi114 ceph-mon[40867]: 10.b scrub ok 2024-09-21T12:34:12.352 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:12.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:12.376 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:12.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:12.477 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:12 smithi183 ceph-mon[33277]: 10.e scrub ok 2024-09-21T12:34:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:12 smithi183 ceph-mon[33277]: 4.7 scrub starts 2024-09-21T12:34:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:12 smithi183 ceph-mon[33277]: 4.7 scrub ok 2024-09-21T12:34:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:12 smithi183 ceph-mon[33277]: 10.b scrub ok 2024-09-21T12:34:12.515 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:12.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:12.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:12.616 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:12.646 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:12.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:12.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:12.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:12.782 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:12.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:12.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:12.884 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:12.914 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:12.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:12.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:13.016 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:34:13.025 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:34:12 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:34:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:34:13.052 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:13.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:13.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:34:13.154 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:13.181 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:13.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:13.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:13.283 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:34:13.318 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:13.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:13.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:34:13.420 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:13.442 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:13.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:13.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:13.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:13 smithi183 ceph-mon[33277]: 10.5 scrub ok 2024-09-21T12:34:13.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:13 smithi183 ceph-mon[33277]: pgmap v713: 178 pgs: 178 active+clean; 7.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.4 KiB/s rd, 37 MiB/s wr, 988 op/s 2024-09-21T12:34:13.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:13 smithi183 ceph-mon[33277]: 5.13 scrub starts 2024-09-21T12:34:13.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:13 smithi183 ceph-mon[33277]: 5.13 scrub ok 2024-09-21T12:34:13.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:13 smithi183 ceph-mon[33277]: 10.6 scrub ok 2024-09-21T12:34:13.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:13 smithi183 ceph-mon[33277]: 5.1 scrub starts 2024-09-21T12:34:13.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:13 smithi183 ceph-mon[33277]: 5.1 scrub ok 2024-09-21T12:34:13.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:13 smithi183 ceph-mon[33277]: 10.2 scrub starts 2024-09-21T12:34:13.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:13 smithi114 ceph-mon[40867]: 10.5 scrub ok 2024-09-21T12:34:13.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:13 smithi114 ceph-mon[40867]: pgmap v713: 178 pgs: 178 active+clean; 7.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.4 KiB/s rd, 37 MiB/s wr, 988 op/s 2024-09-21T12:34:13.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:13 smithi114 ceph-mon[40867]: 5.13 scrub starts 2024-09-21T12:34:13.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:13 smithi114 ceph-mon[40867]: 5.13 scrub ok 2024-09-21T12:34:13.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:13 smithi114 ceph-mon[40867]: 10.6 scrub ok 2024-09-21T12:34:13.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:13 smithi114 ceph-mon[40867]: 5.1 scrub starts 2024-09-21T12:34:13.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:13 smithi114 ceph-mon[40867]: 5.1 scrub ok 2024-09-21T12:34:13.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:13 smithi114 ceph-mon[40867]: 10.2 scrub starts 2024-09-21T12:34:13.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:13 smithi114 ceph-mon[30324]: 10.5 scrub ok 2024-09-21T12:34:13.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:13 smithi114 ceph-mon[30324]: pgmap v713: 178 pgs: 178 active+clean; 7.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.4 KiB/s rd, 37 MiB/s wr, 988 op/s 2024-09-21T12:34:13.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:13 smithi114 ceph-mon[30324]: 5.13 scrub starts 2024-09-21T12:34:13.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:13 smithi114 ceph-mon[30324]: 5.13 scrub ok 2024-09-21T12:34:13.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:13 smithi114 ceph-mon[30324]: 10.6 scrub ok 2024-09-21T12:34:13.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:13 smithi114 ceph-mon[30324]: 5.1 scrub starts 2024-09-21T12:34:13.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:13 smithi114 ceph-mon[30324]: 5.1 scrub ok 2024-09-21T12:34:13.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:13 smithi114 ceph-mon[30324]: 10.2 scrub starts 2024-09-21T12:34:13.544 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:13.581 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:13.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:13.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:13.682 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:13.723 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:13.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:13.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:13.825 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:13.867 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:13.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:13.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:13.968 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:14.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:14.017 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:14.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:14.118 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:14.149 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:14.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:14.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:14.251 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:14.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:14 smithi114 ceph-mon[30324]: 1.11 scrub starts 2024-09-21T12:34:14.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:14 smithi114 ceph-mon[30324]: 1.11 scrub ok 2024-09-21T12:34:14.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:14 smithi114 ceph-mon[30324]: 5.1b scrub starts 2024-09-21T12:34:14.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:14 smithi114 ceph-mon[30324]: 5.1b scrub ok 2024-09-21T12:34:14.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:14 smithi114 ceph-mon[30324]: 1.a scrub starts 2024-09-21T12:34:14.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:14 smithi114 ceph-mon[40867]: 1.11 scrub starts 2024-09-21T12:34:14.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:14 smithi114 ceph-mon[40867]: 1.11 scrub ok 2024-09-21T12:34:14.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:14 smithi114 ceph-mon[40867]: 5.1b scrub starts 2024-09-21T12:34:14.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:14 smithi114 ceph-mon[40867]: 5.1b scrub ok 2024-09-21T12:34:14.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:14 smithi114 ceph-mon[40867]: 1.a scrub starts 2024-09-21T12:34:14.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:14.293 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:14.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:14.394 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:14.421 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:14.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:14.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:14.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:14 smithi183 ceph-mon[33277]: 1.11 scrub starts 2024-09-21T12:34:14.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:14 smithi183 ceph-mon[33277]: 1.11 scrub ok 2024-09-21T12:34:14.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:14 smithi183 ceph-mon[33277]: 5.1b scrub starts 2024-09-21T12:34:14.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:14 smithi183 ceph-mon[33277]: 5.1b scrub ok 2024-09-21T12:34:14.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:14 smithi183 ceph-mon[33277]: 1.a scrub starts 2024-09-21T12:34:14.523 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:34:14.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:14.557 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:14.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:34:14.658 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:14.692 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:14.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:14.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:14.773 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:34:14.794 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:14.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:14.825 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:14.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:14.926 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:14.954 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:14.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:14.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:15.056 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:15.087 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:15.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:15.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:15.189 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:15.210 INFO:teuthology.orchestra.run.smithi114.stderr:set osd.2 primary-affinity to 0 (802) 2024-09-21T12:34:15.228 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:15.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:15.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:15.329 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:15.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:15.351 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:15.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:15.452 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:15.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:15.474 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:15.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:15.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:15 smithi183 ceph-mon[33277]: 1.a scrub ok 2024-09-21T12:34:15.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:15 smithi183 ceph-mon[33277]: pgmap v714: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.4 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.3 KiB/s rd, 33 MiB/s wr, 1.24k op/s 2024-09-21T12:34:15.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:15 smithi183 ceph-mon[33277]: 5.14 scrub starts 2024-09-21T12:34:15.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:15 smithi183 ceph-mon[33277]: 5.14 scrub ok 2024-09-21T12:34:15.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:15 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:34:15.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:15 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:34:15.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:15 smithi183 ceph-mon[33277]: osdmap e383: 8 total, 8 up, 8 in 2024-09-21T12:34:15.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:15 smithi183 ceph-mon[33277]: 1.1c scrub starts 2024-09-21T12:34:15.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:15 smithi183 ceph-mon[33277]: 1.1c scrub ok 2024-09-21T12:34:15.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:15 smithi183 ceph-mon[33277]: 10.2 scrub ok 2024-09-21T12:34:15.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:15 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/4158947231' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.0} : dispatch 2024-09-21T12:34:15.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:15 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.0} : dispatch 2024-09-21T12:34:15.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:15 smithi114 ceph-mon[40867]: 1.a scrub ok 2024-09-21T12:34:15.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:15 smithi114 ceph-mon[40867]: pgmap v714: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.4 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.3 KiB/s rd, 33 MiB/s wr, 1.24k op/s 2024-09-21T12:34:15.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:15 smithi114 ceph-mon[40867]: 5.14 scrub starts 2024-09-21T12:34:15.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:15 smithi114 ceph-mon[40867]: 5.14 scrub ok 2024-09-21T12:34:15.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:15 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:34:15.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:15 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:34:15.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:15 smithi114 ceph-mon[40867]: osdmap e383: 8 total, 8 up, 8 in 2024-09-21T12:34:15.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:15 smithi114 ceph-mon[40867]: 1.1c scrub starts 2024-09-21T12:34:15.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:15 smithi114 ceph-mon[40867]: 1.1c scrub ok 2024-09-21T12:34:15.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:15 smithi114 ceph-mon[40867]: 10.2 scrub ok 2024-09-21T12:34:15.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:15 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/4158947231' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.0} : dispatch 2024-09-21T12:34:15.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:15 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.0} : dispatch 2024-09-21T12:34:15.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:15 smithi114 ceph-mon[30324]: 1.a scrub ok 2024-09-21T12:34:15.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:15 smithi114 ceph-mon[30324]: pgmap v714: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.4 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.3 KiB/s rd, 33 MiB/s wr, 1.24k op/s 2024-09-21T12:34:15.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:15 smithi114 ceph-mon[30324]: 5.14 scrub starts 2024-09-21T12:34:15.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:15 smithi114 ceph-mon[30324]: 5.14 scrub ok 2024-09-21T12:34:15.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:15 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:34:15.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:15 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:34:15.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:15 smithi114 ceph-mon[30324]: osdmap e383: 8 total, 8 up, 8 in 2024-09-21T12:34:15.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:15 smithi114 ceph-mon[30324]: 1.1c scrub starts 2024-09-21T12:34:15.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:15 smithi114 ceph-mon[30324]: 1.1c scrub ok 2024-09-21T12:34:15.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:15 smithi114 ceph-mon[30324]: 10.2 scrub ok 2024-09-21T12:34:15.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:15 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4158947231' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.0} : dispatch 2024-09-21T12:34:15.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:15 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.0} : dispatch 2024-09-21T12:34:15.575 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:15.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:15.597 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:15.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:15.698 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:34:15.719 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:15.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:15.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:34:15.821 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:34:15.821 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:34:15.821 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:34:15.821 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:34:15.821 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:34:15.821 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:34:15.821 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:34:15.821 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:34:15.822 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:34:15.822 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:34:15.822 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:34:15.822 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:34:15.822 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:34:15.822 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:34:15.822 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:15.831 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:34:15.831 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:34:15.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:15.844 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:15.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:15.945 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:15.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:15.967 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:15.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:16.067 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:16.088 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:16.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:16.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:16.190 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:16.216 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:16.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:16.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:16.318 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:16.342 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:16.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:16.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:16.444 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:16.475 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:16.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:16.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:16 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]': finished 2024-09-21T12:34:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:16 smithi183 ceph-mon[33277]: osdmap e384: 8 total, 8 up, 8 in 2024-09-21T12:34:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:16 smithi183 ceph-mon[33277]: 6.12 scrub starts 2024-09-21T12:34:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:16 smithi183 ceph-mon[33277]: 6.12 scrub ok 2024-09-21T12:34:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:16 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3876202686' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:34:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:16 smithi183 ceph-mon[33277]: osdmap e385: 8 total, 8 up, 8 in 2024-09-21T12:34:16.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:16 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]': finished 2024-09-21T12:34:16.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:16 smithi114 ceph-mon[30324]: osdmap e384: 8 total, 8 up, 8 in 2024-09-21T12:34:16.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:16 smithi114 ceph-mon[30324]: 6.12 scrub starts 2024-09-21T12:34:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:16 smithi114 ceph-mon[30324]: 6.12 scrub ok 2024-09-21T12:34:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:16 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3876202686' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:34:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:16 smithi114 ceph-mon[30324]: osdmap e385: 8 total, 8 up, 8 in 2024-09-21T12:34:16.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:16 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]': finished 2024-09-21T12:34:16.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:16 smithi114 ceph-mon[40867]: osdmap e384: 8 total, 8 up, 8 in 2024-09-21T12:34:16.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:16 smithi114 ceph-mon[40867]: 6.12 scrub starts 2024-09-21T12:34:16.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:16 smithi114 ceph-mon[40867]: 6.12 scrub ok 2024-09-21T12:34:16.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:16 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3876202686' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:34:16.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:16 smithi114 ceph-mon[40867]: osdmap e385: 8 total, 8 up, 8 in 2024-09-21T12:34:16.577 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:16.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:16.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:16.706 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:16.740 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:16.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:16.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:16.842 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:34:16.871 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:16.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:16.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:34:16.972 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:17.001 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:17.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:17.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:17.103 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:17.136 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:17.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:17.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:17.238 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:34:17.273 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:17.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:17.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:34:17.375 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:17.407 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:17.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:17.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:17.488 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:34:17 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:34:17.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:17 smithi183 ceph-mon[33277]: pgmap v717: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.4 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 5.3 MiB/s wr, 795 op/s 2024-09-21T12:34:17.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:17 smithi183 ceph-mon[33277]: 6.18 scrub starts 2024-09-21T12:34:17.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:17 smithi183 ceph-mon[33277]: 6.18 scrub ok 2024-09-21T12:34:17.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:17 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:34:17.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:17 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:34:17.509 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:17.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:17 smithi114 ceph-mon[40867]: pgmap v717: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.4 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 5.3 MiB/s wr, 795 op/s 2024-09-21T12:34:17.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:17 smithi114 ceph-mon[40867]: 6.18 scrub starts 2024-09-21T12:34:17.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:17 smithi114 ceph-mon[40867]: 6.18 scrub ok 2024-09-21T12:34:17.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:17 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:34:17.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:17 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:34:17.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:17 smithi114 ceph-mon[30324]: pgmap v717: 178 pgs: 1 active+clean+scrubbing, 177 active+clean; 7.4 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 5.3 MiB/s wr, 795 op/s 2024-09-21T12:34:17.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:17 smithi114 ceph-mon[30324]: 6.18 scrub starts 2024-09-21T12:34:17.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:17 smithi114 ceph-mon[30324]: 6.18 scrub ok 2024-09-21T12:34:17.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:17 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:34:17.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:17 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:34:17.547 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:17.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:17.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:17.649 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:17.680 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:17.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:17.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:17.782 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:17.818 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:17.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:17.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:17.920 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:17.954 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:17.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:17.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:18.056 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:18.093 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:18.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:18.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:18.195 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:18.226 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:18.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:18.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:18.327 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:18.362 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:18.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:18.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:18.464 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:18.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:18 smithi183 ceph-mon[33277]: 1.8 scrub starts 2024-09-21T12:34:18.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:18 smithi183 ceph-mon[33277]: 1.8 scrub ok 2024-09-21T12:34:18.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:18 smithi183 ceph-mon[33277]: 5.3 scrub starts 2024-09-21T12:34:18.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:18 smithi183 ceph-mon[33277]: 5.3 scrub ok 2024-09-21T12:34:18.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:18 smithi183 ceph-mon[33277]: 4.2 scrub starts 2024-09-21T12:34:18.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:18 smithi183 ceph-mon[33277]: 4.2 scrub ok 2024-09-21T12:34:18.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:18 smithi183 ceph-mon[33277]: 1.14 scrub starts 2024-09-21T12:34:18.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:18 smithi183 ceph-mon[33277]: 1.14 scrub ok 2024-09-21T12:34:18.500 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:18.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:18.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:18.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:18 smithi114 ceph-mon[30324]: 1.8 scrub starts 2024-09-21T12:34:18.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:18 smithi114 ceph-mon[30324]: 1.8 scrub ok 2024-09-21T12:34:18.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:18 smithi114 ceph-mon[30324]: 5.3 scrub starts 2024-09-21T12:34:18.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:18 smithi114 ceph-mon[30324]: 5.3 scrub ok 2024-09-21T12:34:18.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:18 smithi114 ceph-mon[30324]: 4.2 scrub starts 2024-09-21T12:34:18.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:18 smithi114 ceph-mon[30324]: 4.2 scrub ok 2024-09-21T12:34:18.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:18 smithi114 ceph-mon[30324]: 1.14 scrub starts 2024-09-21T12:34:18.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:18 smithi114 ceph-mon[30324]: 1.14 scrub ok 2024-09-21T12:34:18.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:18 smithi114 ceph-mon[40867]: 1.8 scrub starts 2024-09-21T12:34:18.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:18 smithi114 ceph-mon[40867]: 1.8 scrub ok 2024-09-21T12:34:18.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:18 smithi114 ceph-mon[40867]: 5.3 scrub starts 2024-09-21T12:34:18.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:18 smithi114 ceph-mon[40867]: 5.3 scrub ok 2024-09-21T12:34:18.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:18 smithi114 ceph-mon[40867]: 4.2 scrub starts 2024-09-21T12:34:18.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:18 smithi114 ceph-mon[40867]: 4.2 scrub ok 2024-09-21T12:34:18.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:18 smithi114 ceph-mon[40867]: 1.14 scrub starts 2024-09-21T12:34:18.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:18 smithi114 ceph-mon[40867]: 1.14 scrub ok 2024-09-21T12:34:18.601 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:18.635 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:18.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:18.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:18.737 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:18.771 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:18.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:18.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:18.872 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:18.914 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:18.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:18.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:19.016 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:19.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:19.046 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:19.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:19.147 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:19.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:19.182 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:19.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:19.283 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:19.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:19.327 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:19.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:19.428 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:19.461 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:19.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:19.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:19 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:34:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:19 smithi183 ceph-mon[33277]: pgmap v719: 178 pgs: 178 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.8 KiB/s rd, 8.2 KiB/s wr, 1.10k op/s; 2.9 KiB/s, 0 objects/s recovering 2024-09-21T12:34:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:19 smithi183 ceph-mon[33277]: 1.a scrub starts 2024-09-21T12:34:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:19 smithi183 ceph-mon[33277]: 1.a scrub ok 2024-09-21T12:34:19.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:19 smithi183 ceph-mon[33277]: 5.12 scrub starts 2024-09-21T12:34:19.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:19 smithi183 ceph-mon[33277]: 5.12 scrub ok 2024-09-21T12:34:19.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:19 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:34:19.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:19 smithi114 ceph-mon[40867]: pgmap v719: 178 pgs: 178 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.8 KiB/s rd, 8.2 KiB/s wr, 1.10k op/s; 2.9 KiB/s, 0 objects/s recovering 2024-09-21T12:34:19.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:19 smithi114 ceph-mon[40867]: 1.a scrub starts 2024-09-21T12:34:19.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:19 smithi114 ceph-mon[40867]: 1.a scrub ok 2024-09-21T12:34:19.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:19 smithi114 ceph-mon[40867]: 5.12 scrub starts 2024-09-21T12:34:19.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:19 smithi114 ceph-mon[40867]: 5.12 scrub ok 2024-09-21T12:34:19.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:19 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:34:19.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:19 smithi114 ceph-mon[30324]: pgmap v719: 178 pgs: 178 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.8 KiB/s rd, 8.2 KiB/s wr, 1.10k op/s; 2.9 KiB/s, 0 objects/s recovering 2024-09-21T12:34:19.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:19 smithi114 ceph-mon[30324]: 1.a scrub starts 2024-09-21T12:34:19.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:19 smithi114 ceph-mon[30324]: 1.a scrub ok 2024-09-21T12:34:19.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:19 smithi114 ceph-mon[30324]: 5.12 scrub starts 2024-09-21T12:34:19.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:19 smithi114 ceph-mon[30324]: 5.12 scrub ok 2024-09-21T12:34:19.563 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:34:19.595 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:19.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:19.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:34:19.697 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:19.735 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:19.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:19.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:19.837 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:19.882 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:19.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:19.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:19.984 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:20.029 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:20.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:20.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:20.131 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:20.165 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:20.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:20.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:20.267 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:20.287 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:34:20.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:20.304 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:20.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:20.405 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:20.442 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:20.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:20.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:20.544 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:34:20.579 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:20.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:20.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:34:20.681 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:20.714 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:20.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:20.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:20.816 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:20.850 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:20.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:20.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:20.905 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:34:20.906 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-21T12:34:20.906 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.906 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.15924.0:143456 10.3 10:cd993737:::benchmark_data_smithi114_161690_object16404:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-09-21T12:34:20.906 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:34:20.892970+0000", 2024-09-21T12:34:20.906 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.010727360999999999, 2024-09-21T12:34:20.906 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010757432000000001, 2024-09-21T12:34:20.906 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:34:20.906 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:34:20.906 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:34:20.906 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:34:20.906 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.15924", 2024-09-21T12:34:20.907 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:34:20.907 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 143456 2024-09-21T12:34:20.907 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.907 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:34:20.907 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.907 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:34:20.907 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.892970+0000", 2024-09-21T12:34:20.907 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:34:20.907 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.907 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.907 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:34:20.907 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.892970+0000", 2024-09-21T12:34:20.907 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:34:20.907 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.908 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.908 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:34:20.908 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.892972+0000", 2024-09-21T12:34:20.908 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6160000000000001e-06 2024-09-21T12:34:20.908 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.908 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.908 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:34:20.908 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.892978+0000", 2024-09-21T12:34:20.908 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8830000000000003e-06 2024-09-21T12:34:20.908 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.908 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.908 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:34:20.908 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.892994+0000", 2024-09-21T12:34:20.908 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5811000000000002e-05 2024-09-21T12:34:20.909 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.909 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.909 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:34:20.909 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.893001+0000", 2024-09-21T12:34:20.909 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5360000000000004e-06 2024-09-21T12:34:20.909 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.909 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.909 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:34:20.909 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.893783+0000", 2024-09-21T12:34:20.909 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00078143400000000001 2024-09-21T12:34:20.909 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.909 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.909 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:34:20.909 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.893927+0000", 2024-09-21T12:34:20.910 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014381999999999999 2024-09-21T12:34:20.910 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.910 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.910 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,5", 2024-09-21T12:34:20.910 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.894072+0000", 2024-09-21T12:34:20.910 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000145597 2024-09-21T12:34:20.910 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.910 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.910 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:20.910 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.896430+0000", 2024-09-21T12:34:20.910 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002357423 2024-09-21T12:34:20.910 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.910 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.910 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:34:20.911 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.902960+0000", 2024-09-21T12:34:20.911 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0065307689999999996 2024-09-21T12:34:20.911 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:34:20.911 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:34:20.911 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:34:20.911 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.911 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.911 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.15924.0:143457 10.3 10:c452cb4b:::benchmark_data_smithi114_161690_object16405:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-09-21T12:34:20.911 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:34:20.893573+0000", 2024-09-21T12:34:20.911 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.010124921, 2024-09-21T12:34:20.911 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010231239, 2024-09-21T12:34:20.911 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:34:20.911 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:34:20.911 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:34:20.912 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:34:20.912 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.15924", 2024-09-21T12:34:20.912 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:34:20.912 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 143457 2024-09-21T12:34:20.912 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.912 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:34:20.912 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.912 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:34:20.912 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.893573+0000", 2024-09-21T12:34:20.912 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:34:20.912 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.912 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.912 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:34:20.912 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.893573+0000", 2024-09-21T12:34:20.913 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:34:20.913 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.913 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.913 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:34:20.913 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.893574+0000", 2024-09-21T12:34:20.913 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.54e-07 2024-09-21T12:34:20.913 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.913 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.913 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:34:20.913 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.893578+0000", 2024-09-21T12:34:20.913 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.284e-06 2024-09-21T12:34:20.913 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.913 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.913 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:34:20.914 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.893591+0000", 2024-09-21T12:34:20.914 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2669e-05 2024-09-21T12:34:20.914 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.914 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.914 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:34:20.914 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.893597+0000", 2024-09-21T12:34:20.914 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6309999999999997e-06 2024-09-21T12:34:20.914 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.914 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.914 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:34:20.914 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.896456+0000", 2024-09-21T12:34:20.914 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0028581549999999998 2024-09-21T12:34:20.914 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.914 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.915 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:34:20.915 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.896551+0000", 2024-09-21T12:34:20.915 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.5130000000000006e-05 2024-09-21T12:34:20.915 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.915 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.915 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,5", 2024-09-21T12:34:20.915 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.896649+0000", 2024-09-21T12:34:20.915 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7975999999999998e-05 2024-09-21T12:34:20.915 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.915 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.915 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:34:20.915 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.903307+0000", 2024-09-21T12:34:20.915 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0066588710000000002 2024-09-21T12:34:20.915 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:34:20.916 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:34:20.916 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:34:20.916 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.916 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.916 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.15924.0:143461 10.3 10:c3bab772:::benchmark_data_smithi114_161690_object16409:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-09-21T12:34:20.916 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:34:20.895249+0000", 2024-09-21T12:34:20.916 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.008448433, 2024-09-21T12:34:20.916 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0086171930000000004, 2024-09-21T12:34:20.916 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:34:20.916 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:34:20.916 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:34:20.916 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:34:20.916 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.15924", 2024-09-21T12:34:20.916 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:34:20.917 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 143461 2024-09-21T12:34:20.917 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.917 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:34:20.917 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.917 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:34:20.917 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.895249+0000", 2024-09-21T12:34:20.917 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:34:20.917 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.917 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.917 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:34:20.917 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.895249+0000", 2024-09-21T12:34:20.917 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:34:20.917 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.918 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.918 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:34:20.918 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.895251+0000", 2024-09-21T12:34:20.918 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2470000000000001e-06 2024-09-21T12:34:20.918 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.918 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.918 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:34:20.918 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.895255+0000", 2024-09-21T12:34:20.918 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4089999999999996e-06 2024-09-21T12:34:20.918 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.918 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.918 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:34:20.918 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.895268+0000", 2024-09-21T12:34:20.918 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2697999999999999e-05 2024-09-21T12:34:20.918 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.919 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.919 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:34:20.919 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.895276+0000", 2024-09-21T12:34:20.919 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.7020000000000007e-06 2024-09-21T12:34:20.919 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.919 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.919 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:34:20.919 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.900636+0000", 2024-09-21T12:34:20.919 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0053594890000000003 2024-09-21T12:34:20.919 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.919 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.919 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:34:20.919 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.900740+0000", 2024-09-21T12:34:20.920 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000103835 2024-09-21T12:34:20.920 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.920 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.920 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,5", 2024-09-21T12:34:20.920 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.900843+0000", 2024-09-21T12:34:20.920 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000103168 2024-09-21T12:34:20.920 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.920 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.920 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:34:20.920 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.903312+0000", 2024-09-21T12:34:20.920 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0024692440000000002 2024-09-21T12:34:20.920 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:34:20.920 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:34:20.920 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:34:20.920 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.921 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.921 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop_reply(client.15924.0:143457 10.3 e385/384)", 2024-09-21T12:34:20.921 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:34:20.903166+0000", 2024-09-21T12:34:20.921 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.00053138899999999995, 2024-09-21T12:34:20.921 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00075797600000000005, 2024-09-21T12:34:20.921 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:34:20.921 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:34:20.921 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "queued for pg", 2024-09-21T12:34:20.921 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:34:20.921 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.921 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:34:20.921 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.903166+0000", 2024-09-21T12:34:20.921 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:34:20.921 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.922 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.922 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:34:20.922 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.903166+0000", 2024-09-21T12:34:20.922 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:34:20.922 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.922 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.922 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:34:20.922 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.903168+0000", 2024-09-21T12:34:20.922 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0699999999999999e-06 2024-09-21T12:34:20.922 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.922 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.922 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:34:20.922 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.903173+0000", 2024-09-21T12:34:20.922 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5929999999999996e-06 2024-09-21T12:34:20.922 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.923 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.923 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:34:20.923 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.903184+0000", 2024-09-21T12:34:20.923 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.113e-05 2024-09-21T12:34:20.923 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:34:20.923 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:34:20.923 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:34:20.923 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:20.903191+0000", 2024-09-21T12:34:20.923 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.165e-06 2024-09-21T12:34:20.923 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:34:20.923 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:34:20.923 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:34:20.923 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:34:20.923 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:34:20.923 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 4 2024-09-21T12:34:20.923 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:34:20.944 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:34:20.952 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:20.982 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:20.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:20.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:21.084 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:21.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:21.114 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:21.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:21.215 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:21.247 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:21.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:21.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:21.254 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-21T12:34:21.349 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:21.382 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:21.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:21.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:21.484 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:21.517 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:21.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:21.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:21.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:21 smithi114 ceph-mon[40867]: pgmap v720: 178 pgs: 178 active+clean; 7.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.8 KiB/s rd, 8.2 KiB/s wr, 862 op/s; 2.9 KiB/s, 0 objects/s recovering 2024-09-21T12:34:21.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:21 smithi114 ceph-mon[40867]: 6.f scrub starts 2024-09-21T12:34:21.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:21 smithi114 ceph-mon[40867]: 6.f scrub ok 2024-09-21T12:34:21.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:21 smithi114 ceph-mon[40867]: 5.b scrub starts 2024-09-21T12:34:21.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:21 smithi114 ceph-mon[40867]: 5.b scrub ok 2024-09-21T12:34:21.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:21 smithi114 ceph-mon[40867]: 1.13 scrub starts 2024-09-21T12:34:21.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:21 smithi114 ceph-mon[40867]: 1.13 scrub ok 2024-09-21T12:34:21.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:21 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1479163936' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:34:21.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:21 smithi114 ceph-mon[30324]: pgmap v720: 178 pgs: 178 active+clean; 7.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.8 KiB/s rd, 8.2 KiB/s wr, 862 op/s; 2.9 KiB/s, 0 objects/s recovering 2024-09-21T12:34:21.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:21 smithi114 ceph-mon[30324]: 6.f scrub starts 2024-09-21T12:34:21.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:21 smithi114 ceph-mon[30324]: 6.f scrub ok 2024-09-21T12:34:21.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:21 smithi114 ceph-mon[30324]: 5.b scrub starts 2024-09-21T12:34:21.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:21 smithi114 ceph-mon[30324]: 5.b scrub ok 2024-09-21T12:34:21.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:21 smithi114 ceph-mon[30324]: 1.13 scrub starts 2024-09-21T12:34:21.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:21 smithi114 ceph-mon[30324]: 1.13 scrub ok 2024-09-21T12:34:21.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:21 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1479163936' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:34:21.619 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:21.652 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:21.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:21.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:21.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:21 smithi183 ceph-mon[33277]: pgmap v720: 178 pgs: 178 active+clean; 7.1 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.8 KiB/s rd, 8.2 KiB/s wr, 862 op/s; 2.9 KiB/s, 0 objects/s recovering 2024-09-21T12:34:21.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:21 smithi183 ceph-mon[33277]: 6.f scrub starts 2024-09-21T12:34:21.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:21 smithi183 ceph-mon[33277]: 6.f scrub ok 2024-09-21T12:34:21.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:21 smithi183 ceph-mon[33277]: 5.b scrub starts 2024-09-21T12:34:21.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:21 smithi183 ceph-mon[33277]: 5.b scrub ok 2024-09-21T12:34:21.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:21 smithi183 ceph-mon[33277]: 1.13 scrub starts 2024-09-21T12:34:21.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:21 smithi183 ceph-mon[33277]: 1.13 scrub ok 2024-09-21T12:34:21.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:21 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1479163936' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:34:21.754 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:21.779 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:21.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:21.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:21.880 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:21.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:21.916 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:21.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:22.017 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:22.050 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:22.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:22.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:22.151 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:22.175 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:22.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:22.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:22.277 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:22.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:22.305 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:22.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:22.406 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:22.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:22.435 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:22.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:22.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:22 smithi114 ceph-mon[30324]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:34:22.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:22 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1479163936' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:34:22.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:22 smithi114 ceph-mon[30324]: osdmap e386: 8 total, 8 up, 8 in 2024-09-21T12:34:22.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:22 smithi114 ceph-mon[40867]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:34:22.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:22 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1479163936' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:34:22.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:22 smithi114 ceph-mon[40867]: osdmap e386: 8 total, 8 up, 8 in 2024-09-21T12:34:22.536 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:22.557 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:22.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:22.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:22.658 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:34:22.692 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:22.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:22.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:34:22.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:22 smithi183 ceph-mon[33277]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:34:22.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:22 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1479163936' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:34:22.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:22 smithi183 ceph-mon[33277]: osdmap e386: 8 total, 8 up, 8 in 2024-09-21T12:34:22.793 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:22.819 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:22.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:22.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:22.921 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:22.958 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:22.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:22.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:23.008 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:34:22 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:34:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:34:23.060 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:23.103 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:23.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:23.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:23.207 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:23.244 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:23.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:23.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:23.346 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:23.380 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:23.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:23.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:23.482 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:23.519 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:23.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:23.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:23.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:23 smithi114 ceph-mon[40867]: pgmap v722: 178 pgs: 178 active+clean; 6.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-09-21T12:34:23.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:23 smithi114 ceph-mon[30324]: pgmap v722: 178 pgs: 178 active+clean; 6.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-09-21T12:34:23.621 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:23.653 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:23.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:23.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:23.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:23 smithi183 ceph-mon[33277]: pgmap v722: 178 pgs: 178 active+clean; 6.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-09-21T12:34:23.754 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:23.803 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T12:34:23.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:23.811 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:23.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:23.911 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:23.945 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:23.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:23.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:24.047 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:24.086 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:24.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:24.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:24.188 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:34:24.232 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:24.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:24.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:34:24.295 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:34:24.334 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:24.386 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:24.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:24.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:24.488 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:24.518 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:24.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:24.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:24.620 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:24.661 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:24.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:24.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:24.763 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:24.802 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:24.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:24.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:24.873 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T12:34:24.873 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:34:24.873 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-21T12:34:24.873 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T12:34:24.873 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T12:34:24.873 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-21T12:34:24.904 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:24.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:24.961 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:24.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:24.969 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:34:24.969 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:34:25.000 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:34:25.001 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:34:25.001 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:34:25.037 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:34:25.037 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:34:25.062 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:25.073 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:34:25.073 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:34:25.110 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:34:25.110 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:34:25.115 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:25.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:25.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:25.140 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:34:25.140 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:34:25.170 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:34:25.171 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:34:25.202 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:34:25.202 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:34:25.217 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:25.257 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:25.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:25.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:25.280 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:34:25.334 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:25 smithi114 ceph-mon[40867]: pgmap v723: 178 pgs: 178 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.52k op/s 2024-09-21T12:34:25.334 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:25 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2391864883' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:34:25.334 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:25 smithi114 ceph-mon[30324]: pgmap v723: 178 pgs: 178 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.52k op/s 2024-09-21T12:34:25.334 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:25 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2391864883' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:34:25.358 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:25.393 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:25.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:25.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:25.495 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:34:25.529 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:25.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:25.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:34:25.553 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:25 smithi183 ceph-mon[33277]: pgmap v723: 178 pgs: 178 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.52k op/s 2024-09-21T12:34:25.553 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:25 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2391864883' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:34:25.631 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:25.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:25.674 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:25.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:25.775 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:25.811 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:25.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:25.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:25.913 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:25.914 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:34:25.914 INFO:teuthology.orchestra.run.smithi183.stdout: "success": "" 2024-09-21T12:34:25.914 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:34:25.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:25.949 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:25.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:26.051 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:26.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:26.081 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:26.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:26.182 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:26.214 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:26.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:26.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:26.285 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T12:34:26.315 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:26.357 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:34:26.357 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:34:26.370 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:26.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:26.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:26.472 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:26.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:26.523 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:26.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:26.624 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:26.657 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:26.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:26.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:26.758 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:26.791 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:26.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:26.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:26.893 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:26.927 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:26.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:26.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:27.028 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:27.066 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:27.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:27.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:27.086 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T12:34:27.168 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:27.168 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:34:27.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:27.199 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:27.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:27.300 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:27.304 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:27 smithi114 ceph-mon[30324]: pgmap v724: 178 pgs: 178 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 953 B/s rd, 0 B/s wr, 1.26k op/s 2024-09-21T12:34:27.304 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:27 smithi114 ceph-mon[40867]: pgmap v724: 178 pgs: 178 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 953 B/s rd, 0 B/s wr, 1.26k op/s 2024-09-21T12:34:27.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:27.334 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:27.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:27.435 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:27.466 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:27.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:27.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:27.567 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:27.599 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:27.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:27.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:27.701 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:27.736 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:27.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:27.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:27.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:27 smithi183 ceph-mon[33277]: pgmap v724: 178 pgs: 178 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 953 B/s rd, 0 B/s wr, 1.26k op/s 2024-09-21T12:34:27.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:34:27 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:34:27.839 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:27.873 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:27.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:27.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:27.975 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:28.017 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:28.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:28.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:28.066 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:34:28.118 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:28.145 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:28.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:28.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:28.247 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:28.278 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:28.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:28.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:28.379 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:28.411 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:28.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:28.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:28.513 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:28.520 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:34:28.542 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:28.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:28.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:28.644 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:28.689 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:28.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:28.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:28.791 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:28.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:28.835 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:28.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:28.843 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:34:28.936 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:28.966 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:28.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:28.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:29.068 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:29.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:29.105 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:29.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:29.178 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:34:29.178 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:34:29.206 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:29.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:29.239 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:29.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:29.284 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-21T12:34:29.339 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:34:29.396 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:29.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:29.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:34:29.497 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:29.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:29 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:34:29.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:29 smithi114 ceph-mon[40867]: pgmap v725: 178 pgs: 178 active+clean; 6.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 307 B/s rd, 0 B/s wr, 1.38k op/s 2024-09-21T12:34:29.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:29 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/408456758' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:34:29.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:29 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:34:29.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:29 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:34:29.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:29 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:34:29.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:29 smithi114 ceph-mon[30324]: pgmap v725: 178 pgs: 178 active+clean; 6.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 307 B/s rd, 0 B/s wr, 1.38k op/s 2024-09-21T12:34:29.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:29 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/408456758' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:34:29.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:29 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:34:29.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:29 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:34:29.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:29.532 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:29.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:29.633 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:29.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:29.665 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:29.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:29 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:34:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:29 smithi183 ceph-mon[33277]: pgmap v725: 178 pgs: 178 active+clean; 6.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 307 B/s rd, 0 B/s wr, 1.38k op/s 2024-09-21T12:34:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:29 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/408456758' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:34:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:29 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:34:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:29 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:34:29.766 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:29.801 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:29.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:29.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:29.902 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:29.944 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:29.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:29.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:30.045 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:30.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:30.084 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:30.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:30.184 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:30.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:30.222 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:30.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:30.235 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T12:34:30.323 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:30.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:30.360 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:30.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:30.437 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:30 smithi114 ceph-mon[40867]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:34:30.437 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:30 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:34:30.437 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:30 smithi114 ceph-mon[40867]: osdmap e387: 8 total, 8 up, 8 in 2024-09-21T12:34:30.438 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:30 smithi114 ceph-mon[30324]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:34:30.438 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:30 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:34:30.438 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:30 smithi114 ceph-mon[30324]: osdmap e387: 8 total, 8 up, 8 in 2024-09-21T12:34:30.457 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:34:30.457 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:34:30.461 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:30.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:30 smithi183 ceph-mon[33277]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:34:30.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:30 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:34:30.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:30 smithi183 ceph-mon[33277]: osdmap e387: 8 total, 8 up, 8 in 2024-09-21T12:34:30.489 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:34:30.490 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:34:30.503 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:30.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:30.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:30.523 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:34:30.523 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:34:30.553 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:34:30.553 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:34:30.593 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:34:30.594 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:34:30.605 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:30.625 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:34:30.625 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:34:30.653 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:30.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:30.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:30.661 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:34:30.661 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:34:30.691 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:34:30.698 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:34:30.698 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:34:30.754 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:30.791 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:30.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:30.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:30.893 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:30.942 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:30.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:30.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:31.043 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:31.072 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:31.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:31.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:31.174 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:31.204 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:31.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:31.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:31.306 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:31.341 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:31.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:31.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:31.345 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:34:31.345 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:34:31.345 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.345 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:157764 10.e 10:7de68364:::benchmark_data_smithi114_161690_object30712:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-21T12:34:31.345 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:34:31.339748+0000", 2024-09-21T12:34:31.345 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0026865050000000001, 2024-09-21T12:34:31.345 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0027134809999999998, 2024-09-21T12:34:31.345 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:31.346 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:31.346 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:34:31.346 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:31.346 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:34:31.346 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:34:31.346 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 157764 2024-09-21T12:34:31.346 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.346 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:31.346 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.346 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:31.346 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.339748+0000", 2024-09-21T12:34:31.346 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:31.346 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.347 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.347 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:31.347 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.339748+0000", 2024-09-21T12:34:31.347 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:31.347 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.347 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.347 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:31.347 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.339750+0000", 2024-09-21T12:34:31.347 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.412e-06 2024-09-21T12:34:31.347 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.347 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.347 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:31.347 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.339755+0000", 2024-09-21T12:34:31.347 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9269999999999996e-06 2024-09-21T12:34:31.347 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.348 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.348 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:31.348 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.339768+0000", 2024-09-21T12:34:31.348 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3910999999999999e-05 2024-09-21T12:34:31.348 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.348 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.348 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:31.348 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.339775+0000", 2024-09-21T12:34:31.348 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6209999999999998e-06 2024-09-21T12:34:31.348 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.348 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.348 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:31.348 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.339877+0000", 2024-09-21T12:34:31.348 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010151 2024-09-21T12:34:31.348 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.349 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.349 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:31.349 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.339985+0000", 2024-09-21T12:34:31.349 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010822100000000001 2024-09-21T12:34:31.349 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.349 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.349 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:34:31.349 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.340094+0000", 2024-09-21T12:34:31.349 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010937 2024-09-21T12:34:31.349 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:31.349 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:31.349 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:31.349 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.349 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.350 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:157765 10.1 10:8d9633cc:::benchmark_data_smithi114_161690_object30713:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-21T12:34:31.350 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:34:31.340007+0000", 2024-09-21T12:34:31.350 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.00242771, 2024-09-21T12:34:31.350 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0025195450000000002, 2024-09-21T12:34:31.350 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:31.350 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:31.350 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:34:31.350 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:31.350 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:34:31.350 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:34:31.350 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 157765 2024-09-21T12:34:31.350 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.350 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:31.351 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.351 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:31.351 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.340007+0000", 2024-09-21T12:34:31.351 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:31.351 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.351 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.351 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:31.351 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.340007+0000", 2024-09-21T12:34:31.351 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:31.351 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.351 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.351 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:31.351 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.340008+0000", 2024-09-21T12:34:31.351 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.158e-06 2024-09-21T12:34:31.352 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.352 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.352 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:31.352 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.340013+0000", 2024-09-21T12:34:31.352 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5900000000000001e-06 2024-09-21T12:34:31.352 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.352 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.352 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:31.352 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.340025+0000", 2024-09-21T12:34:31.352 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2551000000000001e-05 2024-09-21T12:34:31.352 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.352 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.352 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:31.352 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.340033+0000", 2024-09-21T12:34:31.353 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6580000000000002e-06 2024-09-21T12:34:31.353 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.353 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.353 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:31.353 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.341569+0000", 2024-09-21T12:34:31.353 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001535593 2024-09-21T12:34:31.353 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.353 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.353 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:31.353 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.341678+0000", 2024-09-21T12:34:31.353 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000109768 2024-09-21T12:34:31.353 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.353 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.353 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,7", 2024-09-21T12:34:31.354 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.341787+0000", 2024-09-21T12:34:31.354 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000108885 2024-09-21T12:34:31.354 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:31.354 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:31.354 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:31.354 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.354 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.354 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:157770 10.e 10.e54eb09e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-21T12:34:31.354 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:34:31.341437+0000", 2024-09-21T12:34:31.354 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.00099784900000000009, 2024-09-21T12:34:31.354 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0011556719999999999, 2024-09-21T12:34:31.354 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:31.354 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:31.355 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-09-21T12:34:31.355 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:31.355 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:34:31.355 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:34:31.355 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 157770 2024-09-21T12:34:31.355 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.355 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:31.355 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.355 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:31.355 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.341437+0000", 2024-09-21T12:34:31.355 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:31.355 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.355 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.355 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:31.355 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.341437+0000", 2024-09-21T12:34:31.356 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:31.356 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.356 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.356 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:31.356 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.341438+0000", 2024-09-21T12:34:31.356 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.209e-06 2024-09-21T12:34:31.356 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.356 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.356 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:31.356 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.341443+0000", 2024-09-21T12:34:31.356 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0019999999999997e-06 2024-09-21T12:34:31.356 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.356 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.356 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:31.356 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.341458+0000", 2024-09-21T12:34:31.357 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4814000000000001e-05 2024-09-21T12:34:31.357 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.357 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.357 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:31.357 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.341465+0000", 2024-09-21T12:34:31.357 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4660000000000002e-06 2024-09-21T12:34:31.357 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:31.357 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:31.357 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:31.357 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.357 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.357 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.15924.0:157767 10.0 e387/352 10:04aa37ef:::benchmark_data_smithi114_161690_object30715:head v 387'9973, mlcod=387'9971)", 2024-09-21T12:34:31.357 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:34:31.341738+0000", 2024-09-21T12:34:31.357 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.00069683199999999999, 2024-09-21T12:34:31.358 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00089692600000000004, 2024-09-21T12:34:31.358 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:31.358 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:31.358 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:34:31.358 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:31.358 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.358 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:31.358 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.341738+0000", 2024-09-21T12:34:31.358 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:31.358 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.358 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.358 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:31.358 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.341738+0000", 2024-09-21T12:34:31.358 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:31.358 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.358 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.359 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:31.359 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.341739+0000", 2024-09-21T12:34:31.359 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5000000000000001e-07 2024-09-21T12:34:31.359 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.359 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.359 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:31.359 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.341751+0000", 2024-09-21T12:34:31.359 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2429000000000001e-05 2024-09-21T12:34:31.359 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.359 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.360 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:31.360 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.341762+0000", 2024-09-21T12:34:31.360 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.044e-05 2024-09-21T12:34:31.360 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.360 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.360 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:31.360 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.341768+0000", 2024-09-21T12:34:31.360 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8619999999999997e-06 2024-09-21T12:34:31.360 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.360 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.360 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:31.360 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.341916+0000", 2024-09-21T12:34:31.360 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014771200000000001 2024-09-21T12:34:31.360 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.360 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.361 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:31.361 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.341934+0000", 2024-09-21T12:34:31.361 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7898999999999999e-05 2024-09-21T12:34:31.361 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:31.361 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:31.361 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:31.361 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.361 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.361 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.15924.0:157768 10.c e387/352)", 2024-09-21T12:34:31.361 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:34:31.342404+0000", 2024-09-21T12:34:31.361 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 3.0759000000000001e-05, 2024-09-21T12:34:31.361 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00027283100000000002, 2024-09-21T12:34:31.361 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:31.361 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:31.361 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-09-21T12:34:31.362 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:31.362 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.362 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:31.362 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.342404+0000", 2024-09-21T12:34:31.362 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:31.362 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.362 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.362 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:31.362 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.342404+0000", 2024-09-21T12:34:31.362 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:31.362 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.362 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.362 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:31.362 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.342405+0000", 2024-09-21T12:34:31.362 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1900000000000002e-07 2024-09-21T12:34:31.363 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.363 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.363 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:31.363 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.342410+0000", 2024-09-21T12:34:31.363 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2959999999999998e-06 2024-09-21T12:34:31.363 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.363 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.363 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:31.363 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.342419+0000", 2024-09-21T12:34:31.363 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4700000000000008e-06 2024-09-21T12:34:31.363 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.363 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.363 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:31.363 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.342426+0000", 2024-09-21T12:34:31.363 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3550000000000001e-06 2024-09-21T12:34:31.363 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:31.364 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:31.364 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:31.364 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.364 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.364 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:157762 10.d 10:b966d3c0:::benchmark_data_smithi114_161690_object30710:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-21T12:34:31.364 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:34:31.335932+0000", 2024-09-21T12:34:31.364 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0065029559999999998, 2024-09-21T12:34:31.364 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0067908589999999998, 2024-09-21T12:34:31.364 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:31.364 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:31.364 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:34:31.364 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:31.364 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:34:31.364 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:34:31.364 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 157762 2024-09-21T12:34:31.364 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.365 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:31.365 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.365 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:31.365 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.335932+0000", 2024-09-21T12:34:31.365 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:31.365 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.365 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.365 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:31.365 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.335932+0000", 2024-09-21T12:34:31.365 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:31.365 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.365 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.365 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:31.365 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.335933+0000", 2024-09-21T12:34:31.365 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.556e-06 2024-09-21T12:34:31.365 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.366 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.366 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:31.366 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.335939+0000", 2024-09-21T12:34:31.366 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7209999999999999e-06 2024-09-21T12:34:31.366 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.366 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.366 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:31.366 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.335953+0000", 2024-09-21T12:34:31.366 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3609e-05 2024-09-21T12:34:31.366 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.366 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.366 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:31.366 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.335960+0000", 2024-09-21T12:34:31.366 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7279999999999995e-06 2024-09-21T12:34:31.367 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.367 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.367 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:31.367 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.336017+0000", 2024-09-21T12:34:31.367 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7002000000000001e-05 2024-09-21T12:34:31.367 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.367 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.367 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:31.367 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.336125+0000", 2024-09-21T12:34:31.367 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000107667 2024-09-21T12:34:31.367 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.367 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.367 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,6", 2024-09-21T12:34:31.367 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.336223+0000", 2024-09-21T12:34:31.367 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8288000000000005e-05 2024-09-21T12:34:31.368 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.368 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.368 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:31.368 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.338701+0000", 2024-09-21T12:34:31.368 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024776059999999998 2024-09-21T12:34:31.368 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:31.368 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:31.368 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:31.368 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:31.339167+0000", 2024-09-21T12:34:31.368 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00046588599999999999 2024-09-21T12:34:31.368 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:31.368 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:31.368 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:31.368 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:31.368 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-21T12:34:31.368 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 6 2024-09-21T12:34:31.369 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:34:31.443 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:34:31.480 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:31.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:31.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:34:31.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:31 smithi183 ceph-mon[33277]: pgmap v727: 178 pgs: 178 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 356 B/s rd, 0 B/s wr, 1.63k op/s 2024-09-21T12:34:31.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:31 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/279798427' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:34:31.521 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:31 smithi114 ceph-mon[40867]: pgmap v727: 178 pgs: 178 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 356 B/s rd, 0 B/s wr, 1.63k op/s 2024-09-21T12:34:31.521 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:31 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/279798427' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:34:31.521 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:31 smithi114 ceph-mon[30324]: pgmap v727: 178 pgs: 178 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 356 B/s rd, 0 B/s wr, 1.63k op/s 2024-09-21T12:34:31.522 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:31 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/279798427' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:34:31.582 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:31.632 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:31.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:31.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:31.733 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:31.766 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:31.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:31.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:31.867 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:31.903 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:31.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:31.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:32.005 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:32.011 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T12:34:32.011 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:34:32.011 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T12:34:32.011 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-21T12:34:32.011 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-e2ad975c-7812-11ef-baf6-efdc52797490@osd.2 2024-09-21T12:34:32.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:32.056 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:32.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:32.157 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:32.189 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:32.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:32.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:32.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:34:32 smithi114 systemd[1]: Stopping Ceph osd.2 for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:34:32.291 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:32.325 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:32.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:32.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:32.445 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:32.486 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:32.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:32.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:32.587 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:32.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:32.619 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:32.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:32.720 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:32.741 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:32.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:32.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:32.774 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:34:32 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:34:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:34:32.843 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:32.871 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:32.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:32.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:32.973 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:33.010 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:33.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:33.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:33.112 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:33.122 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T12:34:33.145 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:33.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:33.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:33.246 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:33.276 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:33.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:33.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:33.378 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:33.398 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:33.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:33.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:33.437 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:33 smithi183 ceph-mon[33277]: pgmap v728: 178 pgs: 178 active+clean; 6.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.39k op/s 2024-09-21T12:34:33.438 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:33 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:34:33.438 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:33 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:34:33.438 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:33 smithi183 ceph-mon[33277]: osd.2 marked itself down and dead 2024-09-21T12:34:33.464 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T12:34:33.497 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:34:33.500 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:33.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:33 smithi114 ceph-mon[40867]: pgmap v728: 178 pgs: 178 active+clean; 6.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.39k op/s 2024-09-21T12:34:33.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:33 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:34:33.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:33 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:34:33.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:33 smithi114 ceph-mon[40867]: osd.2 marked itself down and dead 2024-09-21T12:34:33.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:33 smithi114 ceph-mon[30324]: pgmap v728: 178 pgs: 178 active+clean; 6.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.39k op/s 2024-09-21T12:34:33.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:33 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:34:33.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:33 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:34:33.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:33 smithi114 ceph-mon[30324]: osd.2 marked itself down and dead 2024-09-21T12:34:33.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:34:33 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2[56527]: 2024-09-21T12:34:33.070+0000 7fc6b6065640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T12:34:33.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:34:33 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2[56527]: 2024-09-21T12:34:33.070+0000 7fc6b6065640 -1 osd.2 387 *** Got signal Terminated *** 2024-09-21T12:34:33.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:34:33 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2[56527]: 2024-09-21T12:34:33.070+0000 7fc6b6065640 -1 osd.2 387 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T12:34:33.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:33.526 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:33.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:33.627 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:33.649 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:33.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:33.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:33.750 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:33.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:33.774 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:33.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:33.875 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:33.896 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:33.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:33.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:33.945 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:34:33.999 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:34.023 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:34.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:34.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:34.125 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:34.146 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:34.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:34.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:34.158 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:34 smithi114 ceph-mon[40867]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T12:34:34.158 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:34 smithi114 ceph-mon[40867]: osdmap e388: 8 total, 7 up, 8 in 2024-09-21T12:34:34.158 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:34 smithi114 ceph-mon[30324]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T12:34:34.159 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:34 smithi114 ceph-mon[30324]: osdmap e388: 8 total, 7 up, 8 in 2024-09-21T12:34:34.248 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:34.272 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:34.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:34.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:34.374 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:34.409 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:34.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:34.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:34.471 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:34:34 smithi114 podman[176209]: 2024-09-21 12:34:34.156834514 +0000 UTC m=+1.235167669 container died 6b04b32d27f6a3e5bd511f3c6ffc2353eb61e3b01c0b5d5ebefde56dddfdb928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3) 2024-09-21T12:34:34.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:34 smithi183 ceph-mon[33277]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T12:34:34.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:34 smithi183 ceph-mon[33277]: osdmap e388: 8 total, 7 up, 8 in 2024-09-21T12:34:34.511 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:34.536 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:34.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:34.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:34.638 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:34.677 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:34.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:34.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:34.774 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:34:34 smithi114 podman[176209]: 2024-09-21 12:34:34.570448176 +0000 UTC m=+1.648781320 container cleanup 6b04b32d27f6a3e5bd511f3c6ffc2353eb61e3b01c0b5d5ebefde56dddfdb928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-21T12:34:34.775 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:34:34 smithi114 bash[176209]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2 2024-09-21T12:34:34.779 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:34.807 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:34.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:34.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:34.870 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:34:34.908 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:34.944 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:34.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:34.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:35.046 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:35.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:35.084 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:35.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:35.108 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:34:35 smithi114 podman[176498]: 2024-09-21 12:34:35.050292956 +0000 UTC m=+1.118101294 container remove 6b04b32d27f6a3e5bd511f3c6ffc2353eb61e3b01c0b5d5ebefde56dddfdb928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-21T12:34:35.186 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:35.232 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:35.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:35.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:35.334 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:35.358 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-21T12:34:35.361 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:35 smithi114 ceph-mon[40867]: pgmap v730: 178 pgs: 178 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.73k op/s 2024-09-21T12:34:35.361 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:35 smithi114 ceph-mon[40867]: osdmap e389: 8 total, 7 up, 8 in 2024-09-21T12:34:35.361 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:35 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2957957250' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:34:35.361 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:35 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:34:35.362 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:35 smithi114 ceph-mon[30324]: pgmap v730: 178 pgs: 178 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.73k op/s 2024-09-21T12:34:35.362 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:35 smithi114 ceph-mon[30324]: osdmap e389: 8 total, 7 up, 8 in 2024-09-21T12:34:35.362 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:35 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2957957250' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:34:35.362 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:35 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:34:35.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:35.374 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:35.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:35.458 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:35 smithi183 ceph-mon[33277]: pgmap v730: 178 pgs: 178 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.73k op/s 2024-09-21T12:34:35.458 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:35 smithi183 ceph-mon[33277]: osdmap e389: 8 total, 7 up, 8 in 2024-09-21T12:34:35.458 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:35 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2957957250' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:34:35.459 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:35 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:34:35.478 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:35.515 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:35.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:35.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:35.616 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:35.659 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:35.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:35.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:35.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:35.789 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:35.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:35.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:35.891 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:35.920 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:35.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:35.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:36.022 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:36.058 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:36.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:36.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:36.160 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:36.198 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:36.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:36.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:36.212 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:34:36.300 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:36.328 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:36.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:36.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:36.418 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:34:36.418 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-21T12:34:36.418 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-21T12:34:36.418 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-21T12:34:36.419 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:34:36.419 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:36 smithi183 ceph-mon[33277]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:34:36.419 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:36 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:34:36.419 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:36 smithi183 ceph-mon[33277]: osdmap e390: 8 total, 7 up, 8 in 2024-09-21T12:34:36.419 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:36 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:34:36.419 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:36 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:34:36.430 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:36.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:36.470 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:36.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:36.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:36 smithi114 ceph-mon[30324]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:34:36.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:36 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:34:36.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:36 smithi114 ceph-mon[30324]: osdmap e390: 8 total, 7 up, 8 in 2024-09-21T12:34:36.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:36 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:34:36.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:36 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:34:36.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:36 smithi114 ceph-mon[40867]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:34:36.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:36 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:34:36.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:36 smithi114 ceph-mon[40867]: osdmap e390: 8 total, 7 up, 8 in 2024-09-21T12:34:36.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:36 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:34:36.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:36 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:34:36.570 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:36.610 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:36.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:36.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:36.712 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:36.741 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:36.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:36.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:36.843 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:36.876 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:36.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:36.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:36.977 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:37.015 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:37.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:37.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:37.117 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:37.132 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T12:34:37.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:37.165 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:37.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:37.266 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:37.298 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:37.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:37.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:37.399 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:37.436 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:37.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:37.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:37.478 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:34:37.478 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:34:37.478 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:34:37.478 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:34:37.478 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:34:37.479 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:34:37.479 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:34:37.479 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T12:34:37.479 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:34:37.479 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:34:37.479 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:34:37.479 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:34:37.479 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:34:37.479 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:34:37.480 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:34:37.537 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:37.568 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:37.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:37.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:37.669 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:37.697 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:37.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:37.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:37.727 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:34:37.740 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:37 smithi183 ceph-mon[33277]: pgmap v733: 178 pgs: 178 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-21T12:34:37.741 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:37 smithi183 ceph-mon[33277]: 3.c scrub starts 2024-09-21T12:34:37.741 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:37 smithi183 ceph-mon[33277]: 3.c scrub ok 2024-09-21T12:34:37.741 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:37 smithi183 ceph-mon[33277]: osdmap e391: 8 total, 7 up, 8 in 2024-09-21T12:34:37.741 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:37 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/4266595134' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:34:37.741 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:34:37 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:34:37.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:37 smithi114 ceph-mon[40867]: pgmap v733: 178 pgs: 178 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-21T12:34:37.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:37 smithi114 ceph-mon[40867]: 3.c scrub starts 2024-09-21T12:34:37.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:37 smithi114 ceph-mon[40867]: 3.c scrub ok 2024-09-21T12:34:37.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:37 smithi114 ceph-mon[40867]: osdmap e391: 8 total, 7 up, 8 in 2024-09-21T12:34:37.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:37 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/4266595134' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:34:37.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:37 smithi114 ceph-mon[30324]: pgmap v733: 178 pgs: 178 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.13k op/s 2024-09-21T12:34:37.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:37 smithi114 ceph-mon[30324]: 3.c scrub starts 2024-09-21T12:34:37.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:37 smithi114 ceph-mon[30324]: 3.c scrub ok 2024-09-21T12:34:37.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:37 smithi114 ceph-mon[30324]: osdmap e391: 8 total, 7 up, 8 in 2024-09-21T12:34:37.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:37 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4266595134' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:34:37.799 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:37.846 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:37.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:37.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:37.948 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:37.979 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:37.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:37.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:38.081 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:38.111 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:38.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:38.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:38.212 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:38.241 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:38.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:38.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:38.252 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T12:34:38.343 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:38.383 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:38.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:38.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:38.485 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:38.524 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:38.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:38.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:38.548 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T12:34:38.548 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T12:34:38.626 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:38.659 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:38.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:38.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:38.709 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:38 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:34:38.709 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:38 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:34:38.731 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:38 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:34:38.731 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:38 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:34:38.731 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:38 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:34:38.732 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:38 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:34:38.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:38.793 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:38.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:38.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:38.894 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:38.929 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:38.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:38.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:39.024 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:34:38 smithi114 podman[177231]: 2024-09-21 12:34:38.637464102 +0000 UTC m=+0.033360609 image pull 8f4d013db793795fef09ff5edf96e6b42ba8cd2c9158b2f0b7a5d7c1e657c063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472 2024-09-21T12:34:39.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:34:38 smithi114 podman[177231]: 2024-09-21 12:34:38.784870719 +0000 UTC m=+0.180767226 container create dce338dbe7c961b5607cbfed3eb223f87c6437ec8a0b715aee4144b1f00e8f90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731) 2024-09-21T12:34:39.031 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:39.066 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:39.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:39.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:39.168 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:39.168 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:34:39.204 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:39.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:39.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:39.300 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:34:39 smithi114 podman[177231]: 2024-09-21 12:34:39.046499886 +0000 UTC m=+0.442396405 container init dce338dbe7c961b5607cbfed3eb223f87c6437ec8a0b715aee4144b1f00e8f90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2-deactivate, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD) 2024-09-21T12:34:39.301 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:34:39 smithi114 podman[177231]: 2024-09-21 12:34:39.05397438 +0000 UTC m=+0.449870888 container start dce338dbe7c961b5607cbfed3eb223f87c6437ec8a0b715aee4144b1f00e8f90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T12:34:39.301 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:34:39 smithi114 podman[177231]: 2024-09-21 12:34:39.156800061 +0000 UTC m=+0.552696584 container attach dce338dbe7c961b5607cbfed3eb223f87c6437ec8a0b715aee4144b1f00e8f90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3) 2024-09-21T12:34:39.306 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:39.344 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:39.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:39.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:39.353 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:34:39.353 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:34:39.446 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:39.477 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:39.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:39.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:39.570 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:39 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:34:39.570 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:39 smithi114 ceph-mon[30324]: pgmap v735: 178 pgs: 25 active+undersized, 17 active+undersized+degraded, 136 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 886 op/s; 22370/273465 objects degraded (8.180%) 2024-09-21T12:34:39.570 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:39 smithi114 ceph-mon[30324]: Health check failed: Degraded data redundancy: 22370/273465 objects degraded (8.180%), 17 pgs degraded (PG_DEGRADED) 2024-09-21T12:34:39.571 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:39 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:34:39.571 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:39 smithi114 ceph-mon[40867]: pgmap v735: 178 pgs: 25 active+undersized, 17 active+undersized+degraded, 136 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 886 op/s; 22370/273465 objects degraded (8.180%) 2024-09-21T12:34:39.571 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:39 smithi114 ceph-mon[40867]: Health check failed: Degraded data redundancy: 22370/273465 objects degraded (8.180%), 17 pgs degraded (PG_DEGRADED) 2024-09-21T12:34:39.571 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:34:39 smithi114 podman[177231]: 2024-09-21 12:34:39.511367155 +0000 UTC m=+0.907263673 container died dce338dbe7c961b5607cbfed3eb223f87c6437ec8a0b715aee4144b1f00e8f90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T12:34:39.579 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:39.609 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:39.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:39.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:39.701 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:39 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:34:39.701 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:39 smithi183 ceph-mon[33277]: pgmap v735: 178 pgs: 25 active+undersized, 17 active+undersized+degraded, 136 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 886 op/s; 22370/273465 objects degraded (8.180%) 2024-09-21T12:34:39.701 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:39 smithi183 ceph-mon[33277]: Health check failed: Degraded data redundancy: 22370/273465 objects degraded (8.180%), 17 pgs degraded (PG_DEGRADED) 2024-09-21T12:34:39.711 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:39.738 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:39.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:39.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:39.840 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:39.872 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:39.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:39.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:39.974 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:40.007 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:40.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:40.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:40.109 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:40.141 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:40.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:40.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:40.243 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:40.286 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:40.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:40.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:40.381 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T12:34:40.387 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:40.413 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:40.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:40.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:40.515 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:40.549 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:34:40.553 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@osd.2.service' 2024-09-21T12:34:40.562 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:40.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:40.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:40.662 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T12:34:40.663 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-21T12:34:40.663 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.2 2024-09-21T12:34:40.663 INFO:tasks.thrashosds.thrasher:remote for osd 2 is ubuntu@smithi114.front.sepia.ceph.com 2024-09-21T12:34:40.663 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -n osd.2 --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op list-pgs 2024-09-21T12:34:40.663 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:40.719 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:40.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:40.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:40.820 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:40.826 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:34:40.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:40.870 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:40.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:40.972 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:40.975 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:34:40.990 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/osd.2/config 2024-09-21T12:34:41.029 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:41.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:41.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:41.093 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-21T12:34:41.131 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:41.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:41.180 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:41.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:41.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:41 smithi183 ceph-mon[33277]: pgmap v736: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 902 op/s; 33381/269307 objects degraded (12.395%) 2024-09-21T12:34:41.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:41 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:34:41.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:41 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:34:41.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:41 smithi183 ceph-mon[33277]: 3.16 scrub starts 2024-09-21T12:34:41.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:41 smithi183 ceph-mon[33277]: 3.16 scrub ok 2024-09-21T12:34:41.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:41 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2411879483' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:34:41.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:41 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:34:41.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:41 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:34:41.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:41 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:34:41.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:41 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:34:41.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:41 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:34:41.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:41 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:34:41.281 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:41.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:41.327 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:41.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:41.328 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:41 smithi114 ceph-mon[30324]: pgmap v736: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 902 op/s; 33381/269307 objects degraded (12.395%) 2024-09-21T12:34:41.328 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:41 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:34:41.328 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:41 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:34:41.328 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:41 smithi114 ceph-mon[30324]: 3.16 scrub starts 2024-09-21T12:34:41.329 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:41 smithi114 ceph-mon[30324]: 3.16 scrub ok 2024-09-21T12:34:41.329 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:41 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2411879483' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:34:41.329 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:41 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:34:41.329 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:41 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:34:41.329 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:41 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:34:41.329 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:41 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:34:41.329 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:41 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:34:41.329 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:41 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:34:41.329 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:41 smithi114 ceph-mon[40867]: pgmap v736: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 902 op/s; 33381/269307 objects degraded (12.395%) 2024-09-21T12:34:41.329 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:41 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:34:41.329 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:41 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:34:41.330 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:41 smithi114 ceph-mon[40867]: 3.16 scrub starts 2024-09-21T12:34:41.330 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:41 smithi114 ceph-mon[40867]: 3.16 scrub ok 2024-09-21T12:34:41.330 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:41 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2411879483' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:34:41.330 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:41 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:34:41.330 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:41 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:34:41.330 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:41 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:34:41.330 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:41 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:34:41.330 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:41 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:34:41.330 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:41 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:34:41.429 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:41.468 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:41.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:41.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:41.501 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:34:41.501 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:34:41.569 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:41.604 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:41.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:41.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:41.705 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:41.727 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:34:41.728 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:34:41.728 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:34:41.728 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:34:41.728 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T12:34:41.739 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:41.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:41.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:41.754 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:34:41.754 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:34:41.787 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:34:41.787 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:34:41.816 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:34:41.816 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:34:41.841 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:41.849 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:34:41.849 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:34:41.878 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:41.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:41.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:41.880 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:34:41.880 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:34:41.980 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:42.010 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:42.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:42.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:42.112 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:42.142 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:42.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:42.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:42.244 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:42.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:42 smithi114 ceph-mon[30324]: pgmap v737: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 810 B/s rd, 0 B/s wr, 857 op/s; 33381/269307 objects degraded (12.395%) 2024-09-21T12:34:42.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:42 smithi114 ceph-mon[30324]: 5.11 scrub starts 2024-09-21T12:34:42.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:42 smithi114 ceph-mon[30324]: 5.11 scrub ok 2024-09-21T12:34:42.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:42 smithi114 ceph-mon[30324]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:34:42.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:42 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:34:42.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:42 smithi114 ceph-mon[30324]: osdmap e392: 8 total, 7 up, 8 in 2024-09-21T12:34:42.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:42 smithi114 ceph-mon[30324]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T12:34:42.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:42 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/617394947' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:34:42.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:42 smithi114 ceph-mon[40867]: pgmap v737: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 810 B/s rd, 0 B/s wr, 857 op/s; 33381/269307 objects degraded (12.395%) 2024-09-21T12:34:42.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:42 smithi114 ceph-mon[40867]: 5.11 scrub starts 2024-09-21T12:34:42.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:42 smithi114 ceph-mon[40867]: 5.11 scrub ok 2024-09-21T12:34:42.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:42 smithi114 ceph-mon[40867]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:34:42.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:42 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:34:42.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:42 smithi114 ceph-mon[40867]: osdmap e392: 8 total, 7 up, 8 in 2024-09-21T12:34:42.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:42 smithi114 ceph-mon[40867]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T12:34:42.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:42 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/617394947' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:34:42.280 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:42.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:42.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:42.382 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:42.412 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:42.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:42.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:42.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:42 smithi183 ceph-mon[33277]: pgmap v737: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 810 B/s rd, 0 B/s wr, 857 op/s; 33381/269307 objects degraded (12.395%) 2024-09-21T12:34:42.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:42 smithi183 ceph-mon[33277]: 5.11 scrub starts 2024-09-21T12:34:42.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:42 smithi183 ceph-mon[33277]: 5.11 scrub ok 2024-09-21T12:34:42.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:42 smithi183 ceph-mon[33277]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:34:42.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:42 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:34:42.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:42 smithi183 ceph-mon[33277]: osdmap e392: 8 total, 7 up, 8 in 2024-09-21T12:34:42.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:42 smithi183 ceph-mon[33277]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T12:34:42.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:42 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/617394947' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:34:42.514 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:42.541 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:42.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:42.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:42.643 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:42.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:42.676 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:42.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:42.777 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:42.800 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:42.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:42.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:42.845 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:34:42 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:34:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:34:42.901 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:42.938 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:42.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:42.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:43.039 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:43.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:43.076 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:43.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:43.172 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:43 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:34:43.172 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:43 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:34:43.172 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:43 smithi114 ceph-mon[30324]: 3.19 scrub starts 2024-09-21T12:34:43.172 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:43 smithi114 ceph-mon[30324]: 3.19 scrub ok 2024-09-21T12:34:43.172 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:43 smithi114 ceph-mon[30324]: osdmap e393: 8 total, 7 up, 8 in 2024-09-21T12:34:43.172 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:43 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:34:43.172 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:43 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:34:43.173 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:43 smithi114 ceph-mon[40867]: 3.19 scrub starts 2024-09-21T12:34:43.173 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:43 smithi114 ceph-mon[40867]: 3.19 scrub ok 2024-09-21T12:34:43.173 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:43 smithi114 ceph-mon[40867]: osdmap e393: 8 total, 7 up, 8 in 2024-09-21T12:34:43.177 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:43.212 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:43.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:43.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:43.314 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:43.346 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:43.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:43.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:43.448 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:43.478 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:43.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:43.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:43.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:43 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:34:43.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:43 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:34:43.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:43 smithi183 ceph-mon[33277]: 3.19 scrub starts 2024-09-21T12:34:43.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:43 smithi183 ceph-mon[33277]: 3.19 scrub ok 2024-09-21T12:34:43.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:43 smithi183 ceph-mon[33277]: osdmap e393: 8 total, 7 up, 8 in 2024-09-21T12:34:43.580 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:43.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:43.626 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:43.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:43.644 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T12:34:43.727 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:43.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:43.777 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:43.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:43.878 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:43.907 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:43.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:43.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:44.008 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:44.035 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:44.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:44.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:44.095 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:34:44.136 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:44.176 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:44.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:44.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:44.278 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:44.303 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:44.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:44.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:44.405 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:44.408 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:44 smithi183 ceph-mon[33277]: pgmap v740: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.58k op/s; 31998/255708 objects degraded (12.513%) 2024-09-21T12:34:44.408 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:44 smithi183 ceph-mon[33277]: 1.1a scrub starts 2024-09-21T12:34:44.408 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:44 smithi183 ceph-mon[33277]: 1.1a scrub ok 2024-09-21T12:34:44.408 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:44 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 31998/255708 objects degraded (12.513%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T12:34:44.445 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:44.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:44.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:44.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:44 smithi114 ceph-mon[40867]: pgmap v740: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.58k op/s; 31998/255708 objects degraded (12.513%) 2024-09-21T12:34:44.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:44 smithi114 ceph-mon[40867]: 1.1a scrub starts 2024-09-21T12:34:44.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:44 smithi114 ceph-mon[40867]: 1.1a scrub ok 2024-09-21T12:34:44.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:44 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 31998/255708 objects degraded (12.513%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T12:34:44.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:44 smithi114 ceph-mon[30324]: pgmap v740: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.58k op/s; 31998/255708 objects degraded (12.513%) 2024-09-21T12:34:44.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:44 smithi114 ceph-mon[30324]: 1.1a scrub starts 2024-09-21T12:34:44.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:44 smithi114 ceph-mon[30324]: 1.1a scrub ok 2024-09-21T12:34:44.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:44 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 31998/255708 objects degraded (12.513%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T12:34:44.546 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:44.576 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:44.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:44.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:44.678 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:44.707 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:44.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:44.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:44.749 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:34:44.750 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-21T12:34:44.750 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-21T12:34:44.750 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-21T12:34:44.750 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:34:44.808 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:44.841 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:44.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:44.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:44.942 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:44.965 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:44.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:44.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:45.066 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:45.106 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:45.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:45.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:45.208 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:45.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:45.242 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:45.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:45.343 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:45.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:45.384 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:45.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:45.485 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:45.491 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T12:34:45.522 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:45.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:45.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:45.624 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:45.672 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:45.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:45.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:45.774 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:45.779 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T12:34:45.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:45.802 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:45.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:45.903 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:45.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:45.947 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:45.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:45.952 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:34:46.048 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:46.082 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:46.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:46.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:46.184 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:46.214 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:46.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:46.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:46.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:46 smithi114 ceph-mon[40867]: pgmap v741: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 755 B/s rd, 0 B/s wr, 1.28k op/s; 30615/247446 objects degraded (12.372%) 2024-09-21T12:34:46.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:46 smithi114 ceph-mon[40867]: 4.1f scrub starts 2024-09-21T12:34:46.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:46 smithi114 ceph-mon[40867]: 4.1f scrub ok 2024-09-21T12:34:46.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:46 smithi114 ceph-mon[40867]: 5.15 scrub starts 2024-09-21T12:34:46.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:46 smithi114 ceph-mon[40867]: 5.15 scrub ok 2024-09-21T12:34:46.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:46 smithi114 ceph-mon[30324]: pgmap v741: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 755 B/s rd, 0 B/s wr, 1.28k op/s; 30615/247446 objects degraded (12.372%) 2024-09-21T12:34:46.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:46 smithi114 ceph-mon[30324]: 4.1f scrub starts 2024-09-21T12:34:46.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:46 smithi114 ceph-mon[30324]: 4.1f scrub ok 2024-09-21T12:34:46.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:46 smithi114 ceph-mon[30324]: 5.15 scrub starts 2024-09-21T12:34:46.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:46 smithi114 ceph-mon[30324]: 5.15 scrub ok 2024-09-21T12:34:46.316 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:46.349 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:46.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:46.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:46.437 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:46 smithi183 ceph-mon[33277]: pgmap v741: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 755 B/s rd, 0 B/s wr, 1.28k op/s; 30615/247446 objects degraded (12.372%) 2024-09-21T12:34:46.437 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:46 smithi183 ceph-mon[33277]: 4.1f scrub starts 2024-09-21T12:34:46.437 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:46 smithi183 ceph-mon[33277]: 4.1f scrub ok 2024-09-21T12:34:46.437 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:46 smithi183 ceph-mon[33277]: 5.15 scrub starts 2024-09-21T12:34:46.437 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:46 smithi183 ceph-mon[33277]: 5.15 scrub ok 2024-09-21T12:34:46.450 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:46.496 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:46.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:46.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:46.587 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:34:46.587 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-21T12:34:46.588 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-21T12:34:46.588 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:34:46.588 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.588 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:32454 8.5 8:afb3ecb0:::benchmark_data_smithi114_104936_object32453:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T12:34:46.588 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:07.550337+0000", 2024-09-21T12:34:46.588 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 579.03175671099996, 2024-09-21T12:34:46.588 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15084788399999999, 2024-09-21T12:34:46.588 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:46.588 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:46.588 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:46.588 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:46.588 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:34:46.588 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:34:46.589 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 32454 2024-09-21T12:34:46.589 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.589 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:46.589 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.589 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:46.589 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.550337+0000", 2024-09-21T12:34:46.589 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.589 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.589 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.589 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:46.589 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.550337+0000", 2024-09-21T12:34:46.589 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.589 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.589 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.590 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:46.590 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.550338+0000", 2024-09-21T12:34:46.590 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.525e-06 2024-09-21T12:34:46.590 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.590 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.590 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:46.590 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.550385+0000", 2024-09-21T12:34:46.590 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6434999999999998e-05 2024-09-21T12:34:46.590 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.590 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.590 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:46.590 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.550398+0000", 2024-09-21T12:34:46.590 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3502999999999999e-05 2024-09-21T12:34:46.591 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.591 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.591 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:46.591 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.550406+0000", 2024-09-21T12:34:46.591 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.187e-06 2024-09-21T12:34:46.591 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.591 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.591 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:46.591 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.550466+0000", 2024-09-21T12:34:46.591 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0708000000000002e-05 2024-09-21T12:34:46.591 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.591 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.591 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:46.591 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.550611+0000", 2024-09-21T12:34:46.592 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000144864 2024-09-21T12:34:46.592 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.592 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.592 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,7", 2024-09-21T12:34:46.592 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.550704+0000", 2024-09-21T12:34:46.592 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2888999999999998e-05 2024-09-21T12:34:46.592 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.592 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.592 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:46.592 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.552093+0000", 2024-09-21T12:34:46.592 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0013884590000000001 2024-09-21T12:34:46.592 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.592 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.592 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.592 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.553102+0000", 2024-09-21T12:34:46.592 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010093299999999999 2024-09-21T12:34:46.593 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.593 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.593 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.593 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.701083+0000", 2024-09-21T12:34:46.593 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14798086099999999 2024-09-21T12:34:46.593 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.593 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.593 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:46.593 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.701142+0000", 2024-09-21T12:34:46.593 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8788e-05 2024-09-21T12:34:46.593 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.593 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.593 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:46.593 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:07.701185+0000", 2024-09-21T12:34:46.593 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3334999999999998e-05 2024-09-21T12:34:46.594 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.594 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:46.594 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.594 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.594 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.594 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:38141 8.9 8:961cc3e2:::benchmark_data_smithi114_104936_object38140:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T12:34:46.594 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:11.301060+0000", 2024-09-21T12:34:46.594 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 575.28103387800002, 2024-09-21T12:34:46.594 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14064707500000001, 2024-09-21T12:34:46.594 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:46.594 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:46.594 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:46.594 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:46.594 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:34:46.594 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:34:46.595 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 38141 2024-09-21T12:34:46.595 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.595 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:46.595 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.595 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:46.595 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.301060+0000", 2024-09-21T12:34:46.595 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.595 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.595 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.595 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:46.595 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.301060+0000", 2024-09-21T12:34:46.595 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.595 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.595 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.595 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:46.595 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.301061+0000", 2024-09-21T12:34:46.596 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3209999999999999e-06 2024-09-21T12:34:46.596 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.596 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.596 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:46.596 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.301107+0000", 2024-09-21T12:34:46.596 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5577000000000001e-05 2024-09-21T12:34:46.596 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.596 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.596 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:46.596 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.301121+0000", 2024-09-21T12:34:46.596 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4518999999999999e-05 2024-09-21T12:34:46.596 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.596 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.596 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:46.596 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.301129+0000", 2024-09-21T12:34:46.596 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6739999999999997e-06 2024-09-21T12:34:46.597 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.597 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.597 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:46.597 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.301183+0000", 2024-09-21T12:34:46.597 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4363000000000001e-05 2024-09-21T12:34:46.597 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.597 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.597 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:46.597 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.301317+0000", 2024-09-21T12:34:46.597 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013395099999999999 2024-09-21T12:34:46.597 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.597 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.597 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-21T12:34:46.597 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.301459+0000", 2024-09-21T12:34:46.597 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014159599999999999 2024-09-21T12:34:46.598 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.598 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.598 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.598 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.304358+0000", 2024-09-21T12:34:46.598 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0028990130000000002 2024-09-21T12:34:46.598 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.598 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.598 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:46.598 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.304497+0000", 2024-09-21T12:34:46.598 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013938500000000001 2024-09-21T12:34:46.598 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.598 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.598 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.598 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.441590+0000", 2024-09-21T12:34:46.598 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13709248099999999 2024-09-21T12:34:46.599 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.599 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.599 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:46.599 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.441667+0000", 2024-09-21T12:34:46.599 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7088999999999993e-05 2024-09-21T12:34:46.599 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.599 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.599 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:46.599 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.441707+0000", 2024-09-21T12:34:46.599 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0105999999999999e-05 2024-09-21T12:34:46.599 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.599 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:46.599 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.599 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.599 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.600 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:38144 8.5 8:a00fd695:::benchmark_data_smithi114_104936_object38143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-21T12:34:46.600 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:25:11.302034+0000", 2024-09-21T12:34:46.600 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 575.28005970499999, 2024-09-21T12:34:46.600 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13963275, 2024-09-21T12:34:46.600 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:46.600 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:46.600 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:46.600 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:46.600 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:34:46.600 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:34:46.600 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 38144 2024-09-21T12:34:46.600 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.600 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:46.600 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.600 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:46.601 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302034+0000", 2024-09-21T12:34:46.601 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.601 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.601 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.601 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:46.601 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302034+0000", 2024-09-21T12:34:46.601 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.601 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.601 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.601 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:46.601 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302035+0000", 2024-09-21T12:34:46.601 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1349999999999999e-06 2024-09-21T12:34:46.601 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.601 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.602 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:46.602 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302074+0000", 2024-09-21T12:34:46.602 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9057999999999998e-05 2024-09-21T12:34:46.602 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.602 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.602 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:46.602 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302090+0000", 2024-09-21T12:34:46.602 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6007000000000001e-05 2024-09-21T12:34:46.602 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.602 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.602 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:46.603 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302098+0000", 2024-09-21T12:34:46.603 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9070000000000007e-06 2024-09-21T12:34:46.603 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.603 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.603 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:46.603 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302165+0000", 2024-09-21T12:34:46.603 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7008999999999995e-05 2024-09-21T12:34:46.603 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.603 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.603 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:46.603 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302300+0000", 2024-09-21T12:34:46.603 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013537900000000001 2024-09-21T12:34:46.603 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.603 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.604 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,7", 2024-09-21T12:34:46.604 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.302451+0000", 2024-09-21T12:34:46.604 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015088600000000001 2024-09-21T12:34:46.604 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.604 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.604 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:46.604 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.304486+0000", 2024-09-21T12:34:46.604 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020348950000000001 2024-09-21T12:34:46.604 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.604 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.604 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.604 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.304554+0000", 2024-09-21T12:34:46.604 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8201999999999995e-05 2024-09-21T12:34:46.604 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.604 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.604 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.605 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.441565+0000", 2024-09-21T12:34:46.605 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.137010256 2024-09-21T12:34:46.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.605 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.605 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:46.605 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.441624+0000", 2024-09-21T12:34:46.605 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9367999999999998e-05 2024-09-21T12:34:46.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.605 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.605 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:46.605 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:25:11.441667+0000", 2024-09-21T12:34:46.605 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2648000000000002e-05 2024-09-21T12:34:46.605 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.605 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:46.605 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.606 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.606 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:223832 8.9 8:95f724a3:::benchmark_data_smithi114_104936_object93416:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-09-21T12:34:46.606 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:27:35.314953+0000", 2024-09-21T12:34:46.606 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 431.26714114800001, 2024-09-21T12:34:46.606 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13631865800000001, 2024-09-21T12:34:46.606 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:46.606 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:46.606 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:46.606 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:46.606 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:34:46.606 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:34:46.606 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 223832 2024-09-21T12:34:46.606 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.606 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:46.607 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.607 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:46.607 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.314953+0000", 2024-09-21T12:34:46.607 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.607 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.607 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:46.607 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.314953+0000", 2024-09-21T12:34:46.607 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.607 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.607 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:46.607 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.314954+0000", 2024-09-21T12:34:46.607 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4249999999999999e-06 2024-09-21T12:34:46.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.608 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:46.608 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.314959+0000", 2024-09-21T12:34:46.608 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6149999999999999e-06 2024-09-21T12:34:46.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.608 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:46.608 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.314973+0000", 2024-09-21T12:34:46.608 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4134999999999999e-05 2024-09-21T12:34:46.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.608 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:46.608 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.314979+0000", 2024-09-21T12:34:46.608 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3489999999999997e-06 2024-09-21T12:34:46.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.609 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:46.609 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.315029+0000", 2024-09-21T12:34:46.609 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0272999999999998e-05 2024-09-21T12:34:46.609 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.609 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.609 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:46.609 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.315137+0000", 2024-09-21T12:34:46.609 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000107878 2024-09-21T12:34:46.609 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.609 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.609 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-21T12:34:46.609 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.315251+0000", 2024-09-21T12:34:46.609 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000113453 2024-09-21T12:34:46.609 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.609 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.609 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:46.610 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.317452+0000", 2024-09-21T12:34:46.610 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0022017080000000001 2024-09-21T12:34:46.610 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.610 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.610 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.610 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.317539+0000", 2024-09-21T12:34:46.610 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6935999999999996e-05 2024-09-21T12:34:46.610 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.610 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.610 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.610 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.451201+0000", 2024-09-21T12:34:46.610 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13366187199999999 2024-09-21T12:34:46.610 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.610 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.611 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:46.611 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.451237+0000", 2024-09-21T12:34:46.611 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6273999999999999e-05 2024-09-21T12:34:46.611 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.611 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.611 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:46.611 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.451271+0000", 2024-09-21T12:34:46.611 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3739999999999999e-05 2024-09-21T12:34:46.611 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.611 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:46.611 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.611 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.611 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.611 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:4793 10.0 10:0db935dd:::benchmark_data_smithi114_161690_object4792:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-09-21T12:34:46.611 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:32:40.429237+0000", 2024-09-21T12:34:46.612 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 126.152856439, 2024-09-21T12:34:46.612 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.97170812799999995, 2024-09-21T12:34:46.612 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:46.612 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:46.612 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:46.612 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:46.612 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:34:46.612 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:34:46.612 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 4793 2024-09-21T12:34:46.612 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.612 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:46.612 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.612 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:46.612 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.429237+0000", 2024-09-21T12:34:46.612 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.613 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.613 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.613 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:46.613 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.429237+0000", 2024-09-21T12:34:46.613 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.613 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.613 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:46.614 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.614 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:46.614 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.429241+0000", 2024-09-21T12:34:46.614 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4620000000000001e-06 2024-09-21T12:34:46.614 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.614 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.614 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:46.614 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.429321+0000", 2024-09-21T12:34:46.615 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0127e-05 2024-09-21T12:34:46.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.615 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.615 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:46.615 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.429332+0000", 2024-09-21T12:34:46.615 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.081e-05 2024-09-21T12:34:46.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.615 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.615 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:46.615 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.429337+0000", 2024-09-21T12:34:46.615 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2000000000000002e-06 2024-09-21T12:34:46.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.615 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.615 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:46.615 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.429383+0000", 2024-09-21T12:34:46.616 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6369999999999998e-05 2024-09-21T12:34:46.616 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.616 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.616 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:46.616 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.429390+0000", 2024-09-21T12:34:46.616 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0729999999999999e-06 2024-09-21T12:34:46.616 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.616 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.616 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:46.616 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.394738+0000", 2024-09-21T12:34:46.616 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.96534762399999996 2024-09-21T12:34:46.616 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.616 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.616 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:46.616 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.394860+0000", 2024-09-21T12:34:46.616 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012222200000000001 2024-09-21T12:34:46.617 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.617 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.617 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,4", 2024-09-21T12:34:46.617 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.394986+0000", 2024-09-21T12:34:46.617 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012562900000000001 2024-09-21T12:34:46.617 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.617 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.617 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:46.617 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.396494+0000", 2024-09-21T12:34:46.617 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001508704 2024-09-21T12:34:46.617 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.617 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.617 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.617 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.399005+0000", 2024-09-21T12:34:46.617 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0025109709999999999 2024-09-21T12:34:46.618 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.618 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.618 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.618 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.400827+0000", 2024-09-21T12:34:46.618 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001821533 2024-09-21T12:34:46.618 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.618 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.618 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:46.618 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.400894+0000", 2024-09-21T12:34:46.618 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7377000000000002e-05 2024-09-21T12:34:46.618 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.618 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.618 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:46.618 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.400945+0000", 2024-09-21T12:34:46.618 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1026000000000002e-05 2024-09-21T12:34:46.619 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.619 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:46.619 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.619 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.619 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.619 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:4801 10.0 10:0f176bf4:::benchmark_data_smithi114_161690_object4800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-09-21T12:34:46.619 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:32:40.434971+0000", 2024-09-21T12:34:46.619 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 126.147122424, 2024-09-21T12:34:46.619 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.96613722999999996, 2024-09-21T12:34:46.619 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:46.619 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:46.619 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:46.619 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:46.620 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:34:46.620 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:34:46.620 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 4801 2024-09-21T12:34:46.620 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.620 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:46.620 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.620 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:46.620 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.434971+0000", 2024-09-21T12:34:46.620 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.620 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.620 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.620 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:46.620 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.434971+0000", 2024-09-21T12:34:46.621 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.621 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:46.621 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.434978+0000", 2024-09-21T12:34:46.621 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9670000000000004e-06 2024-09-21T12:34:46.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.621 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:46.621 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.435070+0000", 2024-09-21T12:34:46.621 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1558999999999995e-05 2024-09-21T12:34:46.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.621 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:46.621 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.435086+0000", 2024-09-21T12:34:46.621 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6447e-05 2024-09-21T12:34:46.622 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.622 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.622 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:46.622 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.435095+0000", 2024-09-21T12:34:46.622 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6719999999999993e-06 2024-09-21T12:34:46.622 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.622 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.622 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:46.622 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.435169+0000", 2024-09-21T12:34:46.622 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3708999999999995e-05 2024-09-21T12:34:46.622 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.622 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.622 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:46.622 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.435179+0000", 2024-09-21T12:34:46.622 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0409e-05 2024-09-21T12:34:46.623 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.623 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.623 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:46.623 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.395628+0000", 2024-09-21T12:34:46.623 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.960448686 2024-09-21T12:34:46.623 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.623 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.623 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:46.623 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.395750+0000", 2024-09-21T12:34:46.623 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000121984 2024-09-21T12:34:46.623 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.623 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.623 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,4", 2024-09-21T12:34:46.623 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.395914+0000", 2024-09-21T12:34:46.623 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000163857 2024-09-21T12:34:46.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.624 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:46.624 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.396701+0000", 2024-09-21T12:34:46.624 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00078757800000000004 2024-09-21T12:34:46.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.624 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.624 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.399098+0000", 2024-09-21T12:34:46.624 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0023964110000000002 2024-09-21T12:34:46.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.624 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.624 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401013+0000", 2024-09-21T12:34:46.625 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0019155540000000001 2024-09-21T12:34:46.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:46.625 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401072+0000", 2024-09-21T12:34:46.625 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8999999999999998e-05 2024-09-21T12:34:46.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:46.625 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401108+0000", 2024-09-21T12:34:46.625 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6396999999999997e-05 2024-09-21T12:34:46.625 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.626 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:46.626 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.626 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:43123 10.f 10:f5348dda:::benchmark_data_smithi114_161690_object43122:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-09-21T12:34:46.626 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:33:00.807719+0000", 2024-09-21T12:34:46.626 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 105.774374932, 2024-09-21T12:34:46.626 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.37280072400000003, 2024-09-21T12:34:46.626 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:46.626 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:46.626 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:46.626 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:46.626 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:34:46.626 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:34:46.627 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 43123 2024-09-21T12:34:46.627 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.627 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:46.627 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.627 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:46.627 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.807719+0000", 2024-09-21T12:34:46.627 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.627 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.627 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.627 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:46.627 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.807719+0000", 2024-09-21T12:34:46.627 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.627 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.628 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.628 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:46.628 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.807720+0000", 2024-09-21T12:34:46.628 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.46e-06 2024-09-21T12:34:46.628 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.628 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.628 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:46.628 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.807752+0000", 2024-09-21T12:34:46.628 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1424000000000003e-05 2024-09-21T12:34:46.628 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.628 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.628 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:46.628 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.807766+0000", 2024-09-21T12:34:46.629 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4498000000000001e-05 2024-09-21T12:34:46.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.629 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.629 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:46.629 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.807774+0000", 2024-09-21T12:34:46.629 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9780000000000005e-06 2024-09-21T12:34:46.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.629 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.629 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:46.629 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.808091+0000", 2024-09-21T12:34:46.629 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00031702100000000001 2024-09-21T12:34:46.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.629 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.629 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:46.630 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.808233+0000", 2024-09-21T12:34:46.630 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014201 2024-09-21T12:34:46.630 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.630 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.630 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,3", 2024-09-21T12:34:46.630 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.808361+0000", 2024-09-21T12:34:46.630 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000127813 2024-09-21T12:34:46.630 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.630 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.630 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.630 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.810940+0000", 2024-09-21T12:34:46.630 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0025795179999999998 2024-09-21T12:34:46.630 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.631 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.631 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:46.631 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.812556+0000", 2024-09-21T12:34:46.631 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001615184 2024-09-21T12:34:46.631 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.631 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.631 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.631 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:01.180397+0000", 2024-09-21T12:34:46.631 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.36784113099999999 2024-09-21T12:34:46.631 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.631 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.631 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:46.631 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:01.180472+0000", 2024-09-21T12:34:46.632 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5068999999999996e-05 2024-09-21T12:34:46.632 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.632 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.632 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:46.632 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:01.180519+0000", 2024-09-21T12:34:46.632 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7617999999999997e-05 2024-09-21T12:34:46.632 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.632 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:46.632 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.632 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.632 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.632 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:43148 10.f 10:f122d938:::benchmark_data_smithi114_161690_object43147:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-09-21T12:34:46.632 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:33:00.820371+0000", 2024-09-21T12:34:46.633 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 105.761722378, 2024-09-21T12:34:46.633 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.36041693699999999, 2024-09-21T12:34:46.633 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:46.633 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:46.633 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:46.633 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:46.633 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:34:46.633 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:34:46.633 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 43148 2024-09-21T12:34:46.633 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.633 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:46.633 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.633 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:46.634 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.820371+0000", 2024-09-21T12:34:46.634 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.634 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.634 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:46.634 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.820371+0000", 2024-09-21T12:34:46.634 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.634 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.634 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:46.634 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.820373+0000", 2024-09-21T12:34:46.634 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5290000000000001e-06 2024-09-21T12:34:46.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.635 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.635 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:46.635 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.820402+0000", 2024-09-21T12:34:46.635 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9646e-05 2024-09-21T12:34:46.635 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.635 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.635 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:46.635 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.820415+0000", 2024-09-21T12:34:46.635 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2773e-05 2024-09-21T12:34:46.635 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.635 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.635 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:46.635 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.820423+0000", 2024-09-21T12:34:46.636 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5279999999999998e-06 2024-09-21T12:34:46.636 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.636 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:46.636 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.820517+0000", 2024-09-21T12:34:46.636 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4477999999999999e-05 2024-09-21T12:34:46.636 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.636 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:46.636 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.820658+0000", 2024-09-21T12:34:46.636 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000140507 2024-09-21T12:34:46.636 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.637 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,3", 2024-09-21T12:34:46.637 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.820792+0000", 2024-09-21T12:34:46.637 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013416799999999999 2024-09-21T12:34:46.637 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.637 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.637 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:46.637 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.822512+0000", 2024-09-21T12:34:46.637 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001720023 2024-09-21T12:34:46.637 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.637 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.637 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.637 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.822900+0000", 2024-09-21T12:34:46.637 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00038832100000000001 2024-09-21T12:34:46.637 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.638 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.638 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.638 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:01.180699+0000", 2024-09-21T12:34:46.638 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.35779872600000001 2024-09-21T12:34:46.638 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.638 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.638 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:46.638 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:01.180751+0000", 2024-09-21T12:34:46.638 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2394e-05 2024-09-21T12:34:46.638 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.638 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.638 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:46.638 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:01.180788+0000", 2024-09-21T12:34:46.638 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6844000000000002e-05 2024-09-21T12:34:46.639 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.639 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:46.639 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.639 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.639 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.639 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:64539 10.0 10:0a4a2714:::benchmark_data_smithi114_161690_object64538:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-09-21T12:34:46.639 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:33:16.091235+0000", 2024-09-21T12:34:46.639 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 90.490858282999994, 2024-09-21T12:34:46.639 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.12744955099999999, 2024-09-21T12:34:46.639 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:46.639 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:46.639 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:46.639 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:46.640 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:34:46.640 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:34:46.640 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 64539 2024-09-21T12:34:46.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.640 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:46.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.640 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:46.640 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.091235+0000", 2024-09-21T12:34:46.640 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.640 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:46.640 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.091235+0000", 2024-09-21T12:34:46.640 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.641 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.641 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.641 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:46.641 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.091237+0000", 2024-09-21T12:34:46.641 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7090000000000001e-06 2024-09-21T12:34:46.641 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.641 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.641 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:46.641 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.091267+0000", 2024-09-21T12:34:46.641 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9697999999999999e-05 2024-09-21T12:34:46.641 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.641 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.641 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:46.641 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.091278+0000", 2024-09-21T12:34:46.642 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1452e-05 2024-09-21T12:34:46.642 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.642 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.642 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:46.642 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.091284+0000", 2024-09-21T12:34:46.642 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.327e-06 2024-09-21T12:34:46.642 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.642 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.642 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:46.642 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.091360+0000", 2024-09-21T12:34:46.642 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6860000000000006e-05 2024-09-21T12:34:46.642 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.642 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.642 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:46.642 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.091457+0000", 2024-09-21T12:34:46.643 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6650999999999993e-05 2024-09-21T12:34:46.643 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.643 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.643 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,4", 2024-09-21T12:34:46.643 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.091542+0000", 2024-09-21T12:34:46.643 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5358999999999994e-05 2024-09-21T12:34:46.643 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.643 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.643 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:46.643 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.093589+0000", 2024-09-21T12:34:46.643 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020467150000000002 2024-09-21T12:34:46.643 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.643 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.643 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.644 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.093785+0000", 2024-09-21T12:34:46.644 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000195484 2024-09-21T12:34:46.644 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.644 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.644 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.644 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.218607+0000", 2024-09-21T12:34:46.644 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.124822291 2024-09-21T12:34:46.644 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.644 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.644 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:46.644 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.218646+0000", 2024-09-21T12:34:46.644 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8739000000000003e-05 2024-09-21T12:34:46.644 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.644 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.645 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:46.645 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.218685+0000", 2024-09-21T12:34:46.645 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9266e-05 2024-09-21T12:34:46.645 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.645 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:46.645 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.645 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.645 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.645 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:64554 10.0 10:0544a528:::benchmark_data_smithi114_161690_object64553:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-09-21T12:34:46.645 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:33:16.105855+0000", 2024-09-21T12:34:46.645 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 90.476238232, 2024-09-21T12:34:46.645 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.118817404, 2024-09-21T12:34:46.645 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:46.645 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:46.645 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:46.646 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:46.646 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:34:46.646 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:34:46.646 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 64554 2024-09-21T12:34:46.646 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.646 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:46.646 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.646 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:46.646 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.105855+0000", 2024-09-21T12:34:46.646 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.646 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.646 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:46.647 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.105855+0000", 2024-09-21T12:34:46.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:46.647 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.105857+0000", 2024-09-21T12:34:46.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3829999999999999e-06 2024-09-21T12:34:46.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:46.647 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.105920+0000", 2024-09-21T12:34:46.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3225999999999994e-05 2024-09-21T12:34:46.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:46.648 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.105934+0000", 2024-09-21T12:34:46.648 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3908e-05 2024-09-21T12:34:46.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:46.648 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.105942+0000", 2024-09-21T12:34:46.648 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6979999999999998e-06 2024-09-21T12:34:46.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:46.649 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.106002+0000", 2024-09-21T12:34:46.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0640999999999998e-05 2024-09-21T12:34:46.649 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.649 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:46.649 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.106156+0000", 2024-09-21T12:34:46.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015419899999999999 2024-09-21T12:34:46.649 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.649 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,4", 2024-09-21T12:34:46.649 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.106254+0000", 2024-09-21T12:34:46.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7637e-05 2024-09-21T12:34:46.649 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.649 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:46.650 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.111103+0000", 2024-09-21T12:34:46.650 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0048488699999999999 2024-09-21T12:34:46.650 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.650 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.650 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.650 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.112249+0000", 2024-09-21T12:34:46.650 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0011458410000000001 2024-09-21T12:34:46.650 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.650 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.650 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.650 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.224604+0000", 2024-09-21T12:34:46.650 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.112355257 2024-09-21T12:34:46.650 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.650 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.651 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:46.651 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.224641+0000", 2024-09-21T12:34:46.651 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7239e-05 2024-09-21T12:34:46.651 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.651 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.651 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:46.651 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.224673+0000", 2024-09-21T12:34:46.651 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1504999999999997e-05 2024-09-21T12:34:46.651 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.651 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:46.651 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.651 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.651 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.651 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:64556 10.0 10:0c999897:::benchmark_data_smithi114_161690_object64555:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-09-21T12:34:46.652 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:33:16.106329+0000", 2024-09-21T12:34:46.652 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 90.475764384000001, 2024-09-21T12:34:46.652 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.118436055, 2024-09-21T12:34:46.652 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:46.652 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:46.652 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:46.652 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:46.652 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:34:46.652 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:34:46.652 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 64556 2024-09-21T12:34:46.652 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.652 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:46.652 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.652 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:46.653 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.106329+0000", 2024-09-21T12:34:46.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.653 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.653 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:46.653 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.106329+0000", 2024-09-21T12:34:46.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.653 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.653 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:46.653 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.106330+0000", 2024-09-21T12:34:46.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7899999999999997e-07 2024-09-21T12:34:46.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.653 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.654 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:46.654 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.106370+0000", 2024-09-21T12:34:46.654 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0163000000000002e-05 2024-09-21T12:34:46.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.654 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:46.654 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.106386+0000", 2024-09-21T12:34:46.654 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5838999999999999e-05 2024-09-21T12:34:46.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.654 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:46.654 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.106394+0000", 2024-09-21T12:34:46.654 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7810000000000005e-06 2024-09-21T12:34:46.655 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.655 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.655 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:46.655 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.108370+0000", 2024-09-21T12:34:46.656 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001976261 2024-09-21T12:34:46.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.656 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.656 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:46.656 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.108528+0000", 2024-09-21T12:34:46.656 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000158221 2024-09-21T12:34:46.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.656 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.656 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,4", 2024-09-21T12:34:46.656 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.108654+0000", 2024-09-21T12:34:46.656 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012600900000000001 2024-09-21T12:34:46.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.657 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.657 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:46.657 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.111167+0000", 2024-09-21T12:34:46.657 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0025126879999999999 2024-09-21T12:34:46.657 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.657 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.657 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.657 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.112447+0000", 2024-09-21T12:34:46.657 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012798429999999999 2024-09-21T12:34:46.657 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.657 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.657 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.657 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.224712+0000", 2024-09-21T12:34:46.657 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.112264871 2024-09-21T12:34:46.657 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.658 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.658 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:46.658 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.224743+0000", 2024-09-21T12:34:46.658 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0738000000000003e-05 2024-09-21T12:34:46.658 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.658 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.658 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:46.658 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.224765+0000", 2024-09-21T12:34:46.658 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2762e-05 2024-09-21T12:34:46.658 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.658 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:46.658 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.658 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.658 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.658 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:64566 10.0 10:0b316e55:::benchmark_data_smithi114_161690_object64565:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-09-21T12:34:46.659 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:33:16.115324+0000", 2024-09-21T12:34:46.659 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 90.466769735, 2024-09-21T12:34:46.659 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.109535061, 2024-09-21T12:34:46.659 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:46.659 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:46.659 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:46.659 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:46.659 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:34:46.659 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:34:46.659 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 64566 2024-09-21T12:34:46.659 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.659 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:46.659 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.659 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:46.659 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.115324+0000", 2024-09-21T12:34:46.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:46.660 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.115324+0000", 2024-09-21T12:34:46.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:46.660 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.115326+0000", 2024-09-21T12:34:46.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6899999999999999e-06 2024-09-21T12:34:46.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:46.660 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.115379+0000", 2024-09-21T12:34:46.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3257000000000002e-05 2024-09-21T12:34:46.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.661 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:46.661 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.115389+0000", 2024-09-21T12:34:46.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.06e-05 2024-09-21T12:34:46.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.661 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:46.661 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.115395+0000", 2024-09-21T12:34:46.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5949999999999998e-06 2024-09-21T12:34:46.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.661 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:46.661 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.115438+0000", 2024-09-21T12:34:46.662 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3220000000000003e-05 2024-09-21T12:34:46.662 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.662 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:46.662 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.115545+0000", 2024-09-21T12:34:46.662 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000106473 2024-09-21T12:34:46.662 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.662 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,4", 2024-09-21T12:34:46.662 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.115630+0000", 2024-09-21T12:34:46.662 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5495999999999996e-05 2024-09-21T12:34:46.662 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.662 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:46.663 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.117969+0000", 2024-09-21T12:34:46.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0023391739999999999 2024-09-21T12:34:46.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.663 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.663 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.663 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.118163+0000", 2024-09-21T12:34:46.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019349300000000001 2024-09-21T12:34:46.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.663 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.663 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.663 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.224798+0000", 2024-09-21T12:34:46.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10663518600000001 2024-09-21T12:34:46.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.663 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.664 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:46.664 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.224838+0000", 2024-09-21T12:34:46.664 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9529e-05 2024-09-21T12:34:46.664 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.664 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.664 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:46.664 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.224859+0000", 2024-09-21T12:34:46.664 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1348e-05 2024-09-21T12:34:46.664 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.664 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:46.664 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.664 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.664 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.664 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:102611 10.a 10:59e6040d:::benchmark_data_smithi114_161690_object102610:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-21T12:34:46.664 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:33:46.154907+0000", 2024-09-21T12:34:46.665 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 60.427187038, 2024-09-21T12:34:46.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.11589032, 2024-09-21T12:34:46.665 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:46.665 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:46.665 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:46.665 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:46.665 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:34:46.665 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:34:46.665 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 102611 2024-09-21T12:34:46.665 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.665 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:46.665 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.665 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:46.665 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.154907+0000", 2024-09-21T12:34:46.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:46.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.154907+0000", 2024-09-21T12:34:46.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:46.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.154907+0000", 2024-09-21T12:34:46.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5799999999999998e-07 2024-09-21T12:34:46.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:46.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.154974+0000", 2024-09-21T12:34:46.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6333000000000003e-05 2024-09-21T12:34:46.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:46.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.154985+0000", 2024-09-21T12:34:46.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1455e-05 2024-09-21T12:34:46.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:46.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.154993+0000", 2024-09-21T12:34:46.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7600000000000002e-06 2024-09-21T12:34:46.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:46.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.155048+0000", 2024-09-21T12:34:46.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4540999999999998e-05 2024-09-21T12:34:46.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:46.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.155136+0000", 2024-09-21T12:34:46.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7979999999999995e-05 2024-09-21T12:34:46.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-21T12:34:46.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.155222+0000", 2024-09-21T12:34:46.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6766000000000002e-05 2024-09-21T12:34:46.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:46.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.157262+0000", 2024-09-21T12:34:46.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020395130000000002 2024-09-21T12:34:46.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.160221+0000", 2024-09-21T12:34:46.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0029590829999999999 2024-09-21T12:34:46.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.270732+0000", 2024-09-21T12:34:46.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.11051095599999999 2024-09-21T12:34:46.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:46.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.270768+0000", 2024-09-21T12:34:46.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6316000000000003e-05 2024-09-21T12:34:46.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:46.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.270797+0000", 2024-09-21T12:34:46.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8759e-05 2024-09-21T12:34:46.671 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.671 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:46.671 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.671 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:102616 10.a 10:5af58290:::benchmark_data_smithi114_161690_object102615:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-21T12:34:46.671 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:33:46.157611+0000", 2024-09-21T12:34:46.671 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 60.424482484000002, 2024-09-21T12:34:46.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.117902597, 2024-09-21T12:34:46.671 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:46.671 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:46.671 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:46.672 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:46.672 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:34:46.672 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:34:46.672 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 102616 2024-09-21T12:34:46.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.672 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:46.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:46.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.157611+0000", 2024-09-21T12:34:46.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:46.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.157611+0000", 2024-09-21T12:34:46.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:46.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.157612+0000", 2024-09-21T12:34:46.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9800000000000003e-07 2024-09-21T12:34:46.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:46.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.157684+0000", 2024-09-21T12:34:46.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1779999999999994e-05 2024-09-21T12:34:46.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:46.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.157696+0000", 2024-09-21T12:34:46.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2368000000000001e-05 2024-09-21T12:34:46.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:46.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.157704+0000", 2024-09-21T12:34:46.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7549999999999994e-06 2024-09-21T12:34:46.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:46.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.157749+0000", 2024-09-21T12:34:46.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5467999999999999e-05 2024-09-21T12:34:46.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:46.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.157859+0000", 2024-09-21T12:34:46.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000110098 2024-09-21T12:34:46.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-21T12:34:46.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.157963+0000", 2024-09-21T12:34:46.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010394400000000001 2024-09-21T12:34:46.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:46.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.163127+0000", 2024-09-21T12:34:46.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0051632450000000003 2024-09-21T12:34:46.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.165350+0000", 2024-09-21T12:34:46.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0022236109999999999 2024-09-21T12:34:46.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.275450+0000", 2024-09-21T12:34:46.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.110099602 2024-09-21T12:34:46.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:46.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.275488+0000", 2024-09-21T12:34:46.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8247999999999997e-05 2024-09-21T12:34:46.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:46.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.275514+0000", 2024-09-21T12:34:46.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5680000000000001e-05 2024-09-21T12:34:46.677 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.677 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:46.677 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.678 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:102617 10.a 10:598d3175:::benchmark_data_smithi114_161690_object102616:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-21T12:34:46.678 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:33:46.157720+0000", 2024-09-21T12:34:46.678 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 60.424373936000002, 2024-09-21T12:34:46.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.11789568, 2024-09-21T12:34:46.678 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:46.678 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:46.678 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:46.678 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:46.678 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:34:46.678 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:34:46.678 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 102617 2024-09-21T12:34:46.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.679 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:46.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:46.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.157720+0000", 2024-09-21T12:34:46.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:46.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.157720+0000", 2024-09-21T12:34:46.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:46.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.157720+0000", 2024-09-21T12:34:46.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3600000000000004e-07 2024-09-21T12:34:46.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:46.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.157752+0000", 2024-09-21T12:34:46.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1482000000000001e-05 2024-09-21T12:34:46.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:46.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.157758+0000", 2024-09-21T12:34:46.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.421e-06 2024-09-21T12:34:46.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:46.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.157764+0000", 2024-09-21T12:34:46.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9200000000000001e-06 2024-09-21T12:34:46.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:46.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.160251+0000", 2024-09-21T12:34:46.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024871839999999999 2024-09-21T12:34:46.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:46.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.160362+0000", 2024-09-21T12:34:46.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011069 2024-09-21T12:34:46.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-21T12:34:46.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.160497+0000", 2024-09-21T12:34:46.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013463799999999999 2024-09-21T12:34:46.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:46.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.164624+0000", 2024-09-21T12:34:46.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0041272740000000002 2024-09-21T12:34:46.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.165598+0000", 2024-09-21T12:34:46.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00097409999999999999 2024-09-21T12:34:46.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.275563+0000", 2024-09-21T12:34:46.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10996460600000001 2024-09-21T12:34:46.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:46.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.275593+0000", 2024-09-21T12:34:46.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0568000000000003e-05 2024-09-21T12:34:46.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:46.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.275615+0000", 2024-09-21T12:34:46.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2260999999999999e-05 2024-09-21T12:34:46.684 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.684 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:46.684 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.684 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:102627 10.a 10:5c65779b:::benchmark_data_smithi114_161690_object102626:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-21T12:34:46.684 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:33:46.166176+0000", 2024-09-21T12:34:46.684 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 60.415917473999997, 2024-09-21T12:34:46.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.11281720000000001, 2024-09-21T12:34:46.684 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:46.685 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:46.685 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:46.685 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:46.685 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:34:46.685 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:34:46.685 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 102627 2024-09-21T12:34:46.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.685 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:46.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:46.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.166176+0000", 2024-09-21T12:34:46.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:46.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.166176+0000", 2024-09-21T12:34:46.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:46.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.166178+0000", 2024-09-21T12:34:46.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.387e-06 2024-09-21T12:34:46.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:46.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.166217+0000", 2024-09-21T12:34:46.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9771999999999998e-05 2024-09-21T12:34:46.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:46.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.166230+0000", 2024-09-21T12:34:46.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2537999999999999e-05 2024-09-21T12:34:46.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:46.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.166238+0000", 2024-09-21T12:34:46.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7100000000000007e-06 2024-09-21T12:34:46.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:46.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.166299+0000", 2024-09-21T12:34:46.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1449000000000001e-05 2024-09-21T12:34:46.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:46.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.166427+0000", 2024-09-21T12:34:46.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012783299999999999 2024-09-21T12:34:46.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-21T12:34:46.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.166588+0000", 2024-09-21T12:34:46.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016142899999999999 2024-09-21T12:34:46.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.169855+0000", 2024-09-21T12:34:46.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0032667500000000001 2024-09-21T12:34:46.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:46.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.169917+0000", 2024-09-21T12:34:46.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2373999999999996e-05 2024-09-21T12:34:46.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.278874+0000", 2024-09-21T12:34:46.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.108956701 2024-09-21T12:34:46.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:46.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.278932+0000", 2024-09-21T12:34:46.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7374999999999999e-05 2024-09-21T12:34:46.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:46.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.278993+0000", 2024-09-21T12:34:46.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1882000000000001e-05 2024-09-21T12:34:46.690 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.690 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:46.690 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.690 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:102633 10.a 10:58b4922b:::benchmark_data_smithi114_161690_object102632:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-21T12:34:46.690 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:33:46.171548+0000", 2024-09-21T12:34:46.690 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 60.410545823, 2024-09-21T12:34:46.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10761604, 2024-09-21T12:34:46.691 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:46.691 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:46.691 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:46.691 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:46.691 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:34:46.691 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:34:46.691 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 102633 2024-09-21T12:34:46.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.691 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:46.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.691 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:46.691 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.171548+0000", 2024-09-21T12:34:46.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:46.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.171548+0000", 2024-09-21T12:34:46.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:46.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.171549+0000", 2024-09-21T12:34:46.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.37e-06 2024-09-21T12:34:46.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:46.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.171675+0000", 2024-09-21T12:34:46.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000125897 2024-09-21T12:34:46.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:46.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.171689+0000", 2024-09-21T12:34:46.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3756e-05 2024-09-21T12:34:46.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:46.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.171694+0000", 2024-09-21T12:34:46.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5369999999999997e-06 2024-09-21T12:34:46.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:46.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.171776+0000", 2024-09-21T12:34:46.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.161e-05 2024-09-21T12:34:46.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:46.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.171922+0000", 2024-09-21T12:34:46.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014635200000000001 2024-09-21T12:34:46.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-21T12:34:46.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.172013+0000", 2024-09-21T12:34:46.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1020000000000006e-05 2024-09-21T12:34:46.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:46.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.178445+0000", 2024-09-21T12:34:46.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0064314330000000003 2024-09-21T12:34:46.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.178488+0000", 2024-09-21T12:34:46.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2874000000000003e-05 2024-09-21T12:34:46.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.279050+0000", 2024-09-21T12:34:46.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10056219399999999 2024-09-21T12:34:46.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:46.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.279104+0000", 2024-09-21T12:34:46.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4196000000000001e-05 2024-09-21T12:34:46.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:46.696 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.279164+0000", 2024-09-21T12:34:46.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9800999999999999e-05 2024-09-21T12:34:46.696 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.696 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:46.696 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:46.697 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:102634 10.a 10:5d2455ee:::benchmark_data_smithi114_161690_object102633:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-21T12:34:46.697 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:33:46.171711+0000", 2024-09-21T12:34:46.697 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 60.410382710999997, 2024-09-21T12:34:46.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.107839398, 2024-09-21T12:34:46.698 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:46.698 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:46.698 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:46.698 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:46.698 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:34:46.698 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:34:46.698 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 102634 2024-09-21T12:34:46.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.698 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:46.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:46.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.171711+0000", 2024-09-21T12:34:46.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:46.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.171711+0000", 2024-09-21T12:34:46.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:46.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.171712+0000", 2024-09-21T12:34:46.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4799999999999998e-07 2024-09-21T12:34:46.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:46.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.171785+0000", 2024-09-21T12:34:46.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3505999999999997e-05 2024-09-21T12:34:46.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:46.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.171792+0000", 2024-09-21T12:34:46.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0639999999999996e-06 2024-09-21T12:34:46.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:46.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.171799+0000", 2024-09-21T12:34:46.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1670000000000002e-06 2024-09-21T12:34:46.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:46.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.175845+0000", 2024-09-21T12:34:46.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0040454289999999997 2024-09-21T12:34:46.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:46.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.175979+0000", 2024-09-21T12:34:46.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013398699999999999 2024-09-21T12:34:46.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-21T12:34:46.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.176118+0000", 2024-09-21T12:34:46.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013925299999999999 2024-09-21T12:34:46.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:46.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.178549+0000", 2024-09-21T12:34:46.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024309209999999999 2024-09-21T12:34:46.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.178645+0000", 2024-09-21T12:34:46.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5798999999999995e-05 2024-09-21T12:34:46.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.279460+0000", 2024-09-21T12:34:46.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10081504299999999 2024-09-21T12:34:46.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:46.703 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.279514+0000", 2024-09-21T12:34:46.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4409e-05 2024-09-21T12:34:46.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:46.703 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:46.279550+0000", 2024-09-21T12:34:46.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6171999999999999e-05 2024-09-21T12:34:46.703 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.703 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:46.703 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.703 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:145069 10.0 10:02dd3161:::benchmark_data_smithi114_161690_object18017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-09-21T12:34:46.703 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:34:22.114030+0000", 2024-09-21T12:34:46.703 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 24.468063367999999, 2024-09-21T12:34:46.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.119060417, 2024-09-21T12:34:46.704 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:46.704 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:46.704 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:46.704 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:46.704 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:34:46.704 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:34:46.704 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 145069 2024-09-21T12:34:46.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.704 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:46.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:46.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:22.114030+0000", 2024-09-21T12:34:46.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:46.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:22.114030+0000", 2024-09-21T12:34:46.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:46.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:22.114032+0000", 2024-09-21T12:34:46.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4050000000000001e-06 2024-09-21T12:34:46.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:46.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:22.114037+0000", 2024-09-21T12:34:46.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1039999999999998e-06 2024-09-21T12:34:46.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:46.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:22.114051+0000", 2024-09-21T12:34:46.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4479e-05 2024-09-21T12:34:46.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:46.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:22.114059+0000", 2024-09-21T12:34:46.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2120000000000003e-06 2024-09-21T12:34:46.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:46.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:22.114138+0000", 2024-09-21T12:34:46.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8378000000000001e-05 2024-09-21T12:34:46.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:46.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:22.114244+0000", 2024-09-21T12:34:46.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000106011 2024-09-21T12:34:46.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,4", 2024-09-21T12:34:46.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:22.114343+0000", 2024-09-21T12:34:46.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8770000000000005e-05 2024-09-21T12:34:46.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:46.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:22.119407+0000", 2024-09-21T12:34:46.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0050647380000000001 2024-09-21T12:34:46.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:22.121456+0000", 2024-09-21T12:34:46.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020482059999999999 2024-09-21T12:34:46.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:22.232997+0000", 2024-09-21T12:34:46.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.111541191 2024-09-21T12:34:46.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:46.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:22.233048+0000", 2024-09-21T12:34:46.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0893999999999998e-05 2024-09-21T12:34:46.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:46.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:22.233091+0000", 2024-09-21T12:34:46.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3028999999999997e-05 2024-09-21T12:34:46.709 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.709 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:46.709 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.709 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:160307 10.f 10:f936c05c:::benchmark_data_smithi114_161690_object33255:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-21T12:34:46.709 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:34:33.358322+0000", 2024-09-21T12:34:46.709 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 13.223771788000001, 2024-09-21T12:34:46.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.010655659, 2024-09-21T12:34:46.710 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:46.710 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:46.710 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:46.710 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:46.710 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:34:46.710 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:34:46.710 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 160307 2024-09-21T12:34:46.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.710 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:46.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.710 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:46.710 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.358322+0000", 2024-09-21T12:34:46.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:46.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.358322+0000", 2024-09-21T12:34:46.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:46.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:46.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.358324+0000", 2024-09-21T12:34:46.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6819999999999999e-06 2024-09-21T12:34:46.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:46.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.358329+0000", 2024-09-21T12:34:46.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2530000000000004e-06 2024-09-21T12:34:46.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.712 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:46.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.358342+0000", 2024-09-21T12:34:46.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3258000000000001e-05 2024-09-21T12:34:46.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.712 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:46.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.358349+0000", 2024-09-21T12:34:46.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8419999999999999e-06 2024-09-21T12:34:46.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:46.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.358395+0000", 2024-09-21T12:34:46.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6254000000000001e-05 2024-09-21T12:34:46.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:46.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.358402+0000", 2024-09-21T12:34:46.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5540000000000003e-06 2024-09-21T12:34:46.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:46.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.364082+0000", 2024-09-21T12:34:46.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.005680286 2024-09-21T12:34:46.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:46.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.364229+0000", 2024-09-21T12:34:46.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014693700000000001 2024-09-21T12:34:46.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3", 2024-09-21T12:34:46.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.364330+0000", 2024-09-21T12:34:46.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010115000000000001 2024-09-21T12:34:46.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:46.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.368799+0000", 2024-09-21T12:34:46.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0044687160000000002 2024-09-21T12:34:46.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:46.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.368853+0000", 2024-09-21T12:34:46.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.376e-05 2024-09-21T12:34:46.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:46.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.368906+0000", 2024-09-21T12:34:46.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3761999999999998e-05 2024-09-21T12:34:46.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:46.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:46.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:46.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.368978+0000", 2024-09-21T12:34:46.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1204999999999993e-05 2024-09-21T12:34:46.716 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.716 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:46.716 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.716 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:46.716 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:46.716 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:34:46.716 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:46.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:46.817 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:46.849 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:46.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:46.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:46.951 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:46.982 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:46.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:46.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:47.084 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:47.116 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:47.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:47.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:47.218 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:47.231 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:47 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:34:47.231 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:47 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:34:47.232 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:47 smithi183 ceph-mon[33277]: 3.10 scrub starts 2024-09-21T12:34:47.232 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:47 smithi183 ceph-mon[33277]: 3.10 scrub ok 2024-09-21T12:34:47.232 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:47 smithi183 ceph-mon[33277]: osdmap e394: 8 total, 7 up, 8 in 2024-09-21T12:34:47.232 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:47 smithi183 ceph-mon[33277]: 1.1b scrub starts 2024-09-21T12:34:47.232 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:47 smithi183 ceph-mon[33277]: 1.1b scrub ok 2024-09-21T12:34:47.232 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:47 smithi183 ceph-mon[33277]: 5.c scrub starts 2024-09-21T12:34:47.232 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:47 smithi183 ceph-mon[33277]: 5.c scrub ok 2024-09-21T12:34:47.248 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:47.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:47.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:47.304 INFO:teuthology.orchestra.run.smithi114.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T12:34:47.350 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:47.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:47 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:34:47.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:47 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:34:47.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:47 smithi114 ceph-mon[30324]: 3.10 scrub starts 2024-09-21T12:34:47.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:47 smithi114 ceph-mon[30324]: 3.10 scrub ok 2024-09-21T12:34:47.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:47 smithi114 ceph-mon[30324]: osdmap e394: 8 total, 7 up, 8 in 2024-09-21T12:34:47.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:47 smithi114 ceph-mon[30324]: 1.1b scrub starts 2024-09-21T12:34:47.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:47 smithi114 ceph-mon[30324]: 1.1b scrub ok 2024-09-21T12:34:47.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:47 smithi114 ceph-mon[30324]: 5.c scrub starts 2024-09-21T12:34:47.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:47 smithi114 ceph-mon[30324]: 5.c scrub ok 2024-09-21T12:34:47.375 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:47 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:34:47.375 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:47 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:34:47.375 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:47 smithi114 ceph-mon[40867]: 3.10 scrub starts 2024-09-21T12:34:47.375 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:47 smithi114 ceph-mon[40867]: 3.10 scrub ok 2024-09-21T12:34:47.375 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:47 smithi114 ceph-mon[40867]: osdmap e394: 8 total, 7 up, 8 in 2024-09-21T12:34:47.375 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:47 smithi114 ceph-mon[40867]: 1.1b scrub starts 2024-09-21T12:34:47.375 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:47 smithi114 ceph-mon[40867]: 1.1b scrub ok 2024-09-21T12:34:47.375 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:47 smithi114 ceph-mon[40867]: 5.c scrub starts 2024-09-21T12:34:47.376 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:47 smithi114 ceph-mon[40867]: 5.c scrub ok 2024-09-21T12:34:47.383 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:47.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:47.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:47.423 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T12:34:47.423 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T12:34:47.485 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:47.488 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:34:47 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:34:47.518 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:47.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:47.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:47.620 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:47.663 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:47.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:47.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:47.698 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T12:34:47.765 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:47.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:47.805 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:47.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:47.906 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:47.953 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:47.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:47.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:48.055 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:48.085 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:48.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:48.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:48.135 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:34:48.164 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:34:48.187 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:48.231 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:48.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:48.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:48.333 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:48.370 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:48.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:48.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:48.452 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:48 smithi183 ceph-mon[33277]: pgmap v743: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.21k op/s; 30615/247446 objects degraded (12.372%) 2024-09-21T12:34:48.452 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:48 smithi183 ceph-mon[33277]: 4.a scrub starts 2024-09-21T12:34:48.452 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:48 smithi183 ceph-mon[33277]: 4.a scrub ok 2024-09-21T12:34:48.453 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:48 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:34:48.453 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:48 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:34:48.453 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:48 smithi183 ceph-mon[33277]: 5.10 scrub starts 2024-09-21T12:34:48.453 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:48 smithi183 ceph-mon[33277]: 5.10 scrub ok 2024-09-21T12:34:48.460 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:48 smithi114 ceph-mon[30324]: pgmap v743: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.21k op/s; 30615/247446 objects degraded (12.372%) 2024-09-21T12:34:48.460 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:48 smithi114 ceph-mon[30324]: 4.a scrub starts 2024-09-21T12:34:48.460 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:48 smithi114 ceph-mon[30324]: 4.a scrub ok 2024-09-21T12:34:48.460 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:48 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:34:48.460 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:48 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:34:48.460 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:48 smithi114 ceph-mon[30324]: 5.10 scrub starts 2024-09-21T12:34:48.460 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:48 smithi114 ceph-mon[30324]: 5.10 scrub ok 2024-09-21T12:34:48.461 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:48 smithi114 ceph-mon[40867]: pgmap v743: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.21k op/s; 30615/247446 objects degraded (12.372%) 2024-09-21T12:34:48.461 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:48 smithi114 ceph-mon[40867]: 4.a scrub starts 2024-09-21T12:34:48.461 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:48 smithi114 ceph-mon[40867]: 4.a scrub ok 2024-09-21T12:34:48.461 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:48 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:34:48.461 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:48 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:34:48.461 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:48 smithi114 ceph-mon[40867]: 5.10 scrub starts 2024-09-21T12:34:48.461 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:48 smithi114 ceph-mon[40867]: 5.10 scrub ok 2024-09-21T12:34:48.472 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:48.507 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:48.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:48.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:48.609 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:48.634 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:48.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:48.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:48.736 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:48.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:48.769 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:48.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:48.871 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:48.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:48.903 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:48.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:49.004 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:49.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:49.040 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:49.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:49.064 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:34:49.064 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:34:49.064 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.064 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:181780 10.c 10:3333c472:::benchmark_data_smithi114_161690_object54728:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T12:34:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:34:49.054539+0000", 2024-09-21T12:34:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.007533845, 2024-09-21T12:34:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0075645039999999997, 2024-09-21T12:34:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:34:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:34:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:34:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 181780 2024-09-21T12:34:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:49.054539+0000", 2024-09-21T12:34:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:49.054539+0000", 2024-09-21T12:34:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:49.054541+0000", 2024-09-21T12:34:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.294e-06 2024-09-21T12:34:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:49.054545+0000", 2024-09-21T12:34:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0099999999999997e-06 2024-09-21T12:34:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:49.054553+0000", 2024-09-21T12:34:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5920000000000001e-06 2024-09-21T12:34:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:49.054559+0000", 2024-09-21T12:34:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5809999999999996e-06 2024-09-21T12:34:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:49.055676+0000", 2024-09-21T12:34:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0011168180000000001 2024-09-21T12:34:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:49.055803+0000", 2024-09-21T12:34:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012770899999999999 2024-09-21T12:34:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,4", 2024-09-21T12:34:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:49.055920+0000", 2024-09-21T12:34:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000116767 2024-09-21T12:34:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:181784 10.c 10:3326e710:::benchmark_data_smithi114_161690_object54732:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T12:34:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:34:49.056663+0000", 2024-09-21T12:34:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0054102719999999998, 2024-09-21T12:34:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0055099889999999999, 2024-09-21T12:34:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:34:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:34:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:34:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 181784 2024-09-21T12:34:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:49.056663+0000", 2024-09-21T12:34:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:49.056663+0000", 2024-09-21T12:34:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:49.056664+0000", 2024-09-21T12:34:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.24e-06 2024-09-21T12:34:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:49.056668+0000", 2024-09-21T12:34:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8079999999999998e-06 2024-09-21T12:34:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:49.072 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:49.056677+0000", 2024-09-21T12:34:49.072 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1419999999999999e-06 2024-09-21T12:34:49.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:49.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.072 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:49.072 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:49.056683+0000", 2024-09-21T12:34:49.072 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9159999999999998e-06 2024-09-21T12:34:49.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:49.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.072 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:49.072 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:49.059068+0000", 2024-09-21T12:34:49.072 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0023847389999999999 2024-09-21T12:34:49.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:49.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.072 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:49.073 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:49.059174+0000", 2024-09-21T12:34:49.073 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010621899999999999 2024-09-21T12:34:49.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:49.073 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.073 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,4", 2024-09-21T12:34:49.073 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:49.059277+0000", 2024-09-21T12:34:49.073 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010301099999999999 2024-09-21T12:34:49.073 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:49.073 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:49.073 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:49.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:49.073 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.074 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:181785 10.9 10:91760868:::benchmark_data_smithi114_161690_object54733:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T12:34:49.074 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:34:49.056906+0000", 2024-09-21T12:34:49.074 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0051668979999999996, 2024-09-21T12:34:49.074 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0053300939999999996, 2024-09-21T12:34:49.074 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:49.074 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:49.074 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:34:49.074 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:49.074 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:34:49.074 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:34:49.074 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 181785 2024-09-21T12:34:49.074 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:49.074 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:49.075 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.075 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:49.075 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:49.056906+0000", 2024-09-21T12:34:49.075 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:49.075 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:49.075 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.075 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:49.075 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:49.056906+0000", 2024-09-21T12:34:49.075 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:49.075 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:49.075 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.075 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:49.075 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:49.056907+0000", 2024-09-21T12:34:49.075 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1600000000000001e-07 2024-09-21T12:34:49.076 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:49.076 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.076 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:49.076 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:49.056910+0000", 2024-09-21T12:34:49.076 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2600000000000001e-06 2024-09-21T12:34:49.076 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:49.076 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.076 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:49.076 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:49.056919+0000", 2024-09-21T12:34:49.076 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7509999999999997e-06 2024-09-21T12:34:49.076 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:49.076 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.076 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:49.076 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:49.056925+0000", 2024-09-21T12:34:49.076 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0009999999999998e-06 2024-09-21T12:34:49.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:49.077 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.077 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:49.077 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:49.056980+0000", 2024-09-21T12:34:49.077 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4497999999999999e-05 2024-09-21T12:34:49.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:49.077 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.077 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:49.077 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:49.057094+0000", 2024-09-21T12:34:49.077 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000114864 2024-09-21T12:34:49.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:49.077 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.077 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5", 2024-09-21T12:34:49.077 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:49.057216+0000", 2024-09-21T12:34:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000121761 2024-09-21T12:34:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:49.059810+0000", 2024-09-21T12:34:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002593452 2024-09-21T12:34:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:49.078 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-21T12:34:49.079 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 3 2024-09-21T12:34:49.079 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:34:49.141 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:49.176 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:49.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:49.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:49.204 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-21T12:34:49.278 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:49.310 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:49.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:49.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:49.412 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:49.424 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:34:49.449 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:49.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:49.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:49.457 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:49 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:34:49.457 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:49 smithi114 ceph-mon[30324]: 3.6 scrub starts 2024-09-21T12:34:49.457 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:49 smithi114 ceph-mon[30324]: 3.6 scrub ok 2024-09-21T12:34:49.457 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:49 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1371062096' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:34:49.457 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:49 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:34:49.458 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:49 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:34:49.458 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:49 smithi114 ceph-mon[40867]: 3.6 scrub starts 2024-09-21T12:34:49.458 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:49 smithi114 ceph-mon[40867]: 3.6 scrub ok 2024-09-21T12:34:49.458 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:49 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1371062096' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:34:49.458 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:49 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:34:49.466 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:49 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:34:49.467 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:49 smithi183 ceph-mon[33277]: 3.6 scrub starts 2024-09-21T12:34:49.467 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:49 smithi183 ceph-mon[33277]: 3.6 scrub ok 2024-09-21T12:34:49.467 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:49 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1371062096' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:34:49.467 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:49 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:34:49.551 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:49.583 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:49.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:49.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:49.685 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:49.721 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:49.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:49.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:49.823 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:49.851 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T12:34:49.872 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:49.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:49.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:49.975 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:50.016 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:50.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:50.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:50.118 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:50.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:50.153 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:50.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:50.254 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:50.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:50.286 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:50.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:50.303 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:34:50.387 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:50.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:50.421 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:50.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:50.469 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:50 smithi183 ceph-mon[33277]: pgmap v744: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.95k op/s; 27809/225096 objects degraded (12.354%) 2024-09-21T12:34:50.470 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:50 smithi183 ceph-mon[33277]: 1.19 scrub starts 2024-09-21T12:34:50.470 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:50 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:34:50.470 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:50 smithi183 ceph-mon[33277]: 1.19 scrub ok 2024-09-21T12:34:50.470 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:50 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:34:50.470 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:50 smithi183 ceph-mon[33277]: 1.e scrub starts 2024-09-21T12:34:50.471 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:50 smithi183 ceph-mon[33277]: 1.e scrub ok 2024-09-21T12:34:50.471 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:50 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 27809/225096 objects degraded (12.354%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T12:34:50.471 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:50 smithi183 ceph-mon[33277]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:34:50.471 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:50 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:34:50.471 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:50 smithi183 ceph-mon[33277]: osdmap e395: 8 total, 7 up, 8 in 2024-09-21T12:34:50.522 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:50.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:50 smithi114 ceph-mon[30324]: pgmap v744: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.95k op/s; 27809/225096 objects degraded (12.354%) 2024-09-21T12:34:50.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:50 smithi114 ceph-mon[30324]: 1.19 scrub starts 2024-09-21T12:34:50.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:50 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:34:50.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:50 smithi114 ceph-mon[30324]: 1.19 scrub ok 2024-09-21T12:34:50.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:50 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:34:50.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:50 smithi114 ceph-mon[30324]: 1.e scrub starts 2024-09-21T12:34:50.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:50 smithi114 ceph-mon[30324]: 1.e scrub ok 2024-09-21T12:34:50.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:50 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 27809/225096 objects degraded (12.354%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T12:34:50.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:50 smithi114 ceph-mon[30324]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:34:50.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:50 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:34:50.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:50 smithi114 ceph-mon[30324]: osdmap e395: 8 total, 7 up, 8 in 2024-09-21T12:34:50.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:50 smithi114 ceph-mon[40867]: pgmap v744: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.95k op/s; 27809/225096 objects degraded (12.354%) 2024-09-21T12:34:50.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:50 smithi114 ceph-mon[40867]: 1.19 scrub starts 2024-09-21T12:34:50.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:50 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:34:50.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:50 smithi114 ceph-mon[40867]: 1.19 scrub ok 2024-09-21T12:34:50.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:50 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:34:50.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:50 smithi114 ceph-mon[40867]: 1.e scrub starts 2024-09-21T12:34:50.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:50 smithi114 ceph-mon[40867]: 1.e scrub ok 2024-09-21T12:34:50.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:50 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 27809/225096 objects degraded (12.354%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T12:34:50.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:50 smithi114 ceph-mon[40867]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:34:50.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:50 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:34:50.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:50 smithi114 ceph-mon[40867]: osdmap e395: 8 total, 7 up, 8 in 2024-09-21T12:34:50.558 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:50.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:50.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:50.660 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:50.690 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:50.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:50.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:50.792 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:50.843 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:50.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:50.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:50.910 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:34:50.910 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-21T12:34:50.911 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-21T12:34:50.911 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-21T12:34:50.911 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:34:50.945 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:50.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:50.974 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:50.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:51.075 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:51.120 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:51.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:51.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:51.223 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:51.257 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:51.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:51.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:51.358 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:51.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:51.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:51.489 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:51.519 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:51.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:51.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:51.620 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:51.627 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T12:34:51.651 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:51.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:51.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:51.654 INFO:teuthology.orchestra.run.smithi114.stdout:6.f 2024-09-21T12:34:51.655 INFO:teuthology.orchestra.run.smithi114.stdout:6.e 2024-09-21T12:34:51.655 INFO:teuthology.orchestra.run.smithi114.stdout:6.9 2024-09-21T12:34:51.655 INFO:teuthology.orchestra.run.smithi114.stdout:6.3 2024-09-21T12:34:51.655 INFO:teuthology.orchestra.run.smithi114.stdout:6.1 2024-09-21T12:34:51.655 INFO:teuthology.orchestra.run.smithi114.stdout:6.17 2024-09-21T12:34:51.655 INFO:teuthology.orchestra.run.smithi114.stdout:6.12 2024-09-21T12:34:51.656 INFO:teuthology.orchestra.run.smithi114.stdout:6.14 2024-09-21T12:34:51.656 INFO:teuthology.orchestra.run.smithi114.stdout:6.0 2024-09-21T12:34:51.656 INFO:teuthology.orchestra.run.smithi114.stdout:5.d 2024-09-21T12:34:51.656 INFO:teuthology.orchestra.run.smithi114.stdout:5.b 2024-09-21T12:34:51.656 INFO:teuthology.orchestra.run.smithi114.stdout:5.8 2024-09-21T12:34:51.656 INFO:teuthology.orchestra.run.smithi114.stdout:5.1e 2024-09-21T12:34:51.656 INFO:teuthology.orchestra.run.smithi114.stdout:5.1c 2024-09-21T12:34:51.656 INFO:teuthology.orchestra.run.smithi114.stdout:6.1c 2024-09-21T12:34:51.656 INFO:teuthology.orchestra.run.smithi114.stdout:10.8 2024-09-21T12:34:51.656 INFO:teuthology.orchestra.run.smithi114.stdout:10.e 2024-09-21T12:34:51.656 INFO:teuthology.orchestra.run.smithi114.stdout:10.5 2024-09-21T12:34:51.656 INFO:teuthology.orchestra.run.smithi114.stdout:10.3 2024-09-21T12:34:51.656 INFO:teuthology.orchestra.run.smithi114.stdout:5.4 2024-09-21T12:34:51.656 INFO:teuthology.orchestra.run.smithi114.stdout:4.6 2024-09-21T12:34:51.657 INFO:teuthology.orchestra.run.smithi114.stdout:6.2 2024-09-21T12:34:51.657 INFO:teuthology.orchestra.run.smithi114.stdout:4.4 2024-09-21T12:34:51.657 INFO:teuthology.orchestra.run.smithi114.stdout:1.9 2024-09-21T12:34:51.657 INFO:teuthology.orchestra.run.smithi114.stdout:1.a 2024-09-21T12:34:51.657 INFO:teuthology.orchestra.run.smithi114.stdout:5.14 2024-09-21T12:34:51.657 INFO:teuthology.orchestra.run.smithi114.stdout:1.8 2024-09-21T12:34:51.658 INFO:teuthology.orchestra.run.smithi114.stdout:1.5 2024-09-21T12:34:51.658 INFO:teuthology.orchestra.run.smithi114.stdout:1.1 2024-09-21T12:34:51.658 INFO:teuthology.orchestra.run.smithi114.stdout:3.1f 2024-09-21T12:34:51.658 INFO:teuthology.orchestra.run.smithi114.stdout:10.f 2024-09-21T12:34:51.658 INFO:teuthology.orchestra.run.smithi114.stdout:5.a 2024-09-21T12:34:51.658 INFO:teuthology.orchestra.run.smithi114.stdout:1.14 2024-09-21T12:34:51.658 INFO:teuthology.orchestra.run.smithi114.stdout:2.1 2024-09-21T12:34:51.658 INFO:teuthology.orchestra.run.smithi114.stdout:3.0 2024-09-21T12:34:51.658 INFO:teuthology.orchestra.run.smithi114.stdout:3.13 2024-09-21T12:34:51.658 INFO:teuthology.orchestra.run.smithi114.stdout:4.f 2024-09-21T12:34:51.659 INFO:teuthology.orchestra.run.smithi114.stdout:4.19 2024-09-21T12:34:51.659 INFO:teuthology.orchestra.run.smithi114.stdout:6.8 2024-09-21T12:34:51.659 INFO:teuthology.orchestra.run.smithi114.stdout:4.d 2024-09-21T12:34:51.659 INFO:teuthology.orchestra.run.smithi114.stdout:10.9 2024-09-21T12:34:51.659 INFO:teuthology.orchestra.run.smithi114.stdout:3.1e 2024-09-21T12:34:51.659 INFO:teuthology.orchestra.run.smithi114.stdout:1.18 2024-09-21T12:34:51.659 INFO:teuthology.orchestra.run.smithi114.stdout:3.5 2024-09-21T12:34:51.659 INFO:teuthology.orchestra.run.smithi114.stdout:4.1c 2024-09-21T12:34:51.659 INFO:teuthology.orchestra.run.smithi114.stdout:3.4 2024-09-21T12:34:51.659 INFO:teuthology.orchestra.run.smithi114.stdout:3.9 2024-09-21T12:34:51.659 INFO:teuthology.orchestra.run.smithi114.stdout:4.12 2024-09-21T12:34:51.659 INFO:teuthology.orchestra.run.smithi114.stdout:5.6 2024-09-21T12:34:51.660 INFO:teuthology.orchestra.run.smithi114.stdout:4.15 2024-09-21T12:34:51.660 INFO:teuthology.orchestra.run.smithi114.stdout:3.1a 2024-09-21T12:34:51.660 INFO:teuthology.orchestra.run.smithi114.stdout:4.c 2024-09-21T12:34:51.753 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:51.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:51.798 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:51.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:51.899 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:51.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:51.933 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:51.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:52.034 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:52.066 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:34:52.086 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:52.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:52.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:52.188 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:52.225 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:52.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:52.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:52.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:52 smithi114 ceph-mon[30324]: pgmap v746: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.27k op/s; 27809/225096 objects degraded (12.354%) 2024-09-21T12:34:52.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:52 smithi114 ceph-mon[40867]: pgmap v746: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.27k op/s; 27809/225096 objects degraded (12.354%) 2024-09-21T12:34:52.327 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:52.368 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:52.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:52.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:52.470 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:52.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:52 smithi183 ceph-mon[33277]: pgmap v746: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.27k op/s; 27809/225096 objects degraded (12.354%) 2024-09-21T12:34:52.508 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:52.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:52.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:52.609 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:52.636 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:52.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:52.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:52.709 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:34:52.709 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-21T12:34:52.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-21T12:34:52.709 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:34:52.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.709 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14478.0:2784 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94620848900224 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:34:52.709 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.038339+0000", 2024-09-21T12:34:52.709 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 194.666053017, 2024-09-21T12:34:52.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.027622934, 2024-09-21T12:34:52.709 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:52.710 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:52.710 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:34:52.710 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:52.710 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14478", 2024-09-21T12:34:52.710 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:34:52.710 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 2784 2024-09-21T12:34:52.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.710 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:52.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.710 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:52.710 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.038339+0000", 2024-09-21T12:34:52.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:52.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.038339+0000", 2024-09-21T12:34:52.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:52.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.038340+0000", 2024-09-21T12:34:52.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.164e-06 2024-09-21T12:34:52.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:52.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.038345+0000", 2024-09-21T12:34:52.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7339999999999999e-06 2024-09-21T12:34:52.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.712 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:52.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.038359+0000", 2024-09-21T12:34:52.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4577e-05 2024-09-21T12:34:52.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.712 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:52.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.038368+0000", 2024-09-21T12:34:52.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3599999999999996e-06 2024-09-21T12:34:52.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.712 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.038422+0000", 2024-09-21T12:34:52.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3999000000000003e-05 2024-09-21T12:34:52.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op must wait for map", 2024-09-21T12:34:52.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.038427+0000", 2024-09-21T12:34:52.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.789e-06 2024-09-21T12:34:52.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.045355+0000", 2024-09-21T12:34:52.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0069278600000000001 2024-09-21T12:34:52.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:52.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.045364+0000", 2024-09-21T12:34:52.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8340000000000005e-06 2024-09-21T12:34:52.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.054340+0000", 2024-09-21T12:34:52.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0089757239999999 2024-09-21T12:34:52.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for degraded object", 2024-09-21T12:34:52.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.054602+0000", 2024-09-21T12:34:52.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00026209300000000003 2024-09-21T12:34:52.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.058190+0000", 2024-09-21T12:34:52.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0035881939999999998 2024-09-21T12:34:52.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:52.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.058240+0000", 2024-09-21T12:34:52.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0105999999999998e-05 2024-09-21T12:34:52.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:52.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.065962+0000", 2024-09-21T12:34:52.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0077215000000000001 2024-09-21T12:34:52.715 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.716 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:52.716 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.716 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.716 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:221955 9.d 9:baadbb95:::benchmark_data_smithi114_133214_object97921:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:34:52.716 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.080261+0000", 2024-09-21T12:34:52.716 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 194.624131264, 2024-09-21T12:34:52.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0418771250000001, 2024-09-21T12:34:52.716 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:52.716 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:52.716 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:52.716 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:52.716 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:34:52.716 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:34:52.717 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 221955 2024-09-21T12:34:52.717 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.717 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:52.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:52.717 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.080261+0000", 2024-09-21T12:34:52.717 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.717 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:52.717 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.080261+0000", 2024-09-21T12:34:52.717 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.717 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.718 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:52.718 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.080262+0000", 2024-09-21T12:34:52.718 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1230000000000001e-06 2024-09-21T12:34:52.718 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.718 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.718 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:52.718 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.080265+0000", 2024-09-21T12:34:52.718 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.58e-06 2024-09-21T12:34:52.718 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.718 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.718 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:52.718 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.080277+0000", 2024-09-21T12:34:52.718 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1610000000000001e-05 2024-09-21T12:34:52.719 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.719 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.719 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:52.719 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.080282+0000", 2024-09-21T12:34:52.719 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.366e-06 2024-09-21T12:34:52.719 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.719 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.719 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.719 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.080329+0000", 2024-09-21T12:34:52.719 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7267e-05 2024-09-21T12:34:52.719 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.719 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.719 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:52.719 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.080337+0000", 2024-09-21T12:34:52.719 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0299999999999994e-06 2024-09-21T12:34:52.720 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.720 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.720 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.720 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.056546+0000", 2024-09-21T12:34:52.720 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.97620854599999995 2024-09-21T12:34:52.720 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.720 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.720 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:52.720 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.056558+0000", 2024-09-21T12:34:52.720 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1619e-05 2024-09-21T12:34:52.720 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.720 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.720 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.720 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.115730+0000", 2024-09-21T12:34:52.720 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0591722560000001 2024-09-21T12:34:52.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.721 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:52.721 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.115840+0000", 2024-09-21T12:34:52.721 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000110555 2024-09-21T12:34:52.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.721 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:34:52.721 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.115924+0000", 2024-09-21T12:34:52.721 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3115999999999995e-05 2024-09-21T12:34:52.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.721 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.722 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.117996+0000", 2024-09-21T12:34:52.722 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020720320000000001 2024-09-21T12:34:52.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.722 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.722 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:52.722 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.121834+0000", 2024-09-21T12:34:52.722 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0038379970000000001 2024-09-21T12:34:52.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.722 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.722 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.722 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.121889+0000", 2024-09-21T12:34:52.722 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5245999999999999e-05 2024-09-21T12:34:52.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.722 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.723 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.723 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.121951+0000", 2024-09-21T12:34:52.724 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1906e-05 2024-09-21T12:34:52.724 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.724 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.724 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:52.724 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122014+0000", 2024-09-21T12:34:52.724 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3441999999999993e-05 2024-09-21T12:34:52.724 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.724 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.724 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:52.724 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122138+0000", 2024-09-21T12:34:52.724 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012343400000000001 2024-09-21T12:34:52.724 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.725 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:52.725 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.725 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.725 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.725 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:221957 9.d 9:bbdc0e00:::benchmark_data_smithi114_133214_object97923:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:34:52.725 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.082731+0000", 2024-09-21T12:34:52.725 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 194.621661265, 2024-09-21T12:34:52.725 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0396655529999999, 2024-09-21T12:34:52.725 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:52.725 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:52.725 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:52.725 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:52.725 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:34:52.726 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:34:52.726 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 221957 2024-09-21T12:34:52.726 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.726 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:52.726 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.726 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:52.726 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.082731+0000", 2024-09-21T12:34:52.726 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.726 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.726 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.726 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:52.726 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.082731+0000", 2024-09-21T12:34:52.726 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.727 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.727 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.727 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:52.727 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.082731+0000", 2024-09-21T12:34:52.727 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0699999999999996e-07 2024-09-21T12:34:52.727 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.727 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.727 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:52.727 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.082735+0000", 2024-09-21T12:34:52.727 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2720000000000002e-06 2024-09-21T12:34:52.727 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.727 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.727 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:52.728 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.082744+0000", 2024-09-21T12:34:52.728 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0410000000000003e-06 2024-09-21T12:34:52.728 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.728 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.728 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:52.728 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.082748+0000", 2024-09-21T12:34:52.728 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6280000000000004e-06 2024-09-21T12:34:52.728 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.728 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.728 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.728 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.082805+0000", 2024-09-21T12:34:52.728 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6938000000000003e-05 2024-09-21T12:34:52.728 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.729 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.729 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:52.729 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.082824+0000", 2024-09-21T12:34:52.729 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8493999999999999e-05 2024-09-21T12:34:52.729 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.729 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.729 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.729 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.056890+0000", 2024-09-21T12:34:52.729 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.97406623800000003 2024-09-21T12:34:52.729 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.729 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.729 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:52.729 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.056898+0000", 2024-09-21T12:34:52.730 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5159999999999994e-06 2024-09-21T12:34:52.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.730 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.730 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.118054+0000", 2024-09-21T12:34:52.730 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.061155555 2024-09-21T12:34:52.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.730 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:52.730 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.118246+0000", 2024-09-21T12:34:52.730 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019225500000000001 2024-09-21T12:34:52.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.731 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:34:52.731 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.118379+0000", 2024-09-21T12:34:52.731 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013250299999999999 2024-09-21T12:34:52.731 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.731 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.731 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:52.731 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122152+0000", 2024-09-21T12:34:52.731 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0037734909999999999 2024-09-21T12:34:52.731 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.731 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.731 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.731 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122183+0000", 2024-09-21T12:34:52.731 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0549999999999997e-05 2024-09-21T12:34:52.731 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.732 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.732 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.732 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122237+0000", 2024-09-21T12:34:52.732 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3735e-05 2024-09-21T12:34:52.732 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.732 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.732 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.732 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122292+0000", 2024-09-21T12:34:52.732 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5507000000000002e-05 2024-09-21T12:34:52.732 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.732 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.732 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:52.733 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122328+0000", 2024-09-21T12:34:52.733 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6149000000000002e-05 2024-09-21T12:34:52.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.733 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.733 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:52.733 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122396+0000", 2024-09-21T12:34:52.733 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7873999999999994e-05 2024-09-21T12:34:52.733 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.733 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:52.733 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.733 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.733 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:221969 9.d 9:b7382fd3:::benchmark_data_smithi114_133214_object97935:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:34:52.734 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.094992+0000", 2024-09-21T12:34:52.734 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 194.60939987099999, 2024-09-21T12:34:52.734 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0395933770000001, 2024-09-21T12:34:52.734 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:52.734 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:52.734 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:52.734 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:52.734 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:34:52.734 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:34:52.734 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 221969 2024-09-21T12:34:52.734 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.734 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:52.734 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.734 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:52.735 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.094992+0000", 2024-09-21T12:34:52.735 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.735 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.735 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.735 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:52.735 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.094992+0000", 2024-09-21T12:34:52.735 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.735 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.735 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.735 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:52.735 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.094993+0000", 2024-09-21T12:34:52.735 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.054e-06 2024-09-21T12:34:52.735 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.736 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.736 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:52.736 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.094998+0000", 2024-09-21T12:34:52.736 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7620000000000003e-06 2024-09-21T12:34:52.736 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.736 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.736 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:52.736 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095010+0000", 2024-09-21T12:34:52.736 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2091e-05 2024-09-21T12:34:52.736 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.736 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.736 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:52.736 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095018+0000", 2024-09-21T12:34:52.736 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8029999999999997e-06 2024-09-21T12:34:52.737 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.737 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.737 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.737 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095074+0000", 2024-09-21T12:34:52.737 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6013000000000001e-05 2024-09-21T12:34:52.737 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.737 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.737 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:52.737 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095081+0000", 2024-09-21T12:34:52.737 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9999999999999999e-06 2024-09-21T12:34:52.737 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.737 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.737 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.737 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.056923+0000", 2024-09-21T12:34:52.737 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.96184202600000002 2024-09-21T12:34:52.738 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.738 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:52.738 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.056930+0000", 2024-09-21T12:34:52.738 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4329999999999998e-06 2024-09-21T12:34:52.738 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.738 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.738 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122692+0000", 2024-09-21T12:34:52.738 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.065761918 2024-09-21T12:34:52.738 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.739 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:52.739 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122771+0000", 2024-09-21T12:34:52.739 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9456000000000006e-05 2024-09-21T12:34:52.739 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.739 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.739 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:34:52.739 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122867+0000", 2024-09-21T12:34:52.739 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5639999999999999e-05 2024-09-21T12:34:52.739 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.739 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.739 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.739 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.129541+0000", 2024-09-21T12:34:52.739 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.006674386 2024-09-21T12:34:52.740 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.740 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.740 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.740 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.129579+0000", 2024-09-21T12:34:52.740 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7162000000000001e-05 2024-09-21T12:34:52.740 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.740 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.740 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.740 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.129693+0000", 2024-09-21T12:34:52.740 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011387099999999999 2024-09-21T12:34:52.740 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.740 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.740 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:52.740 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134406+0000", 2024-09-21T12:34:52.741 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0047137619999999998 2024-09-21T12:34:52.741 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.741 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.741 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:52.741 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134466+0000", 2024-09-21T12:34:52.741 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9317999999999997e-05 2024-09-21T12:34:52.741 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.741 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.741 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:52.741 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134585+0000", 2024-09-21T12:34:52.741 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000119682 2024-09-21T12:34:52.741 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.741 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:52.742 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.742 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.742 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.742 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:221970 9.d 9:b4ae3e82:::benchmark_data_smithi114_133214_object97936:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:34:52.742 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.095717+0000", 2024-09-21T12:34:52.742 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 194.60867457399999, 2024-09-21T12:34:52.742 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0390824689999998, 2024-09-21T12:34:52.742 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:52.742 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:52.742 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:52.742 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:52.742 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:34:52.742 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:34:52.742 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 221970 2024-09-21T12:34:52.743 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.743 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:52.743 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.743 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:52.743 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095717+0000", 2024-09-21T12:34:52.743 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.743 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.743 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.743 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:52.743 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095717+0000", 2024-09-21T12:34:52.743 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.743 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.743 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.743 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:52.744 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095718+0000", 2024-09-21T12:34:52.744 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4e-07 2024-09-21T12:34:52.744 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.744 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.744 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:52.744 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095723+0000", 2024-09-21T12:34:52.744 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.51e-06 2024-09-21T12:34:52.744 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.744 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.744 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:52.744 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095732+0000", 2024-09-21T12:34:52.744 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7620000000000007e-06 2024-09-21T12:34:52.744 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.744 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.745 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:52.745 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095738+0000", 2024-09-21T12:34:52.745 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2909999999999998e-06 2024-09-21T12:34:52.745 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.745 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.745 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.745 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095779+0000", 2024-09-21T12:34:52.745 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1356999999999997e-05 2024-09-21T12:34:52.745 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.745 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.745 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:52.745 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095785+0000", 2024-09-21T12:34:52.745 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7610000000000004e-06 2024-09-21T12:34:52.746 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.746 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.746 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.746 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.056979+0000", 2024-09-21T12:34:52.746 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.96119413399999998 2024-09-21T12:34:52.746 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.746 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.746 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:52.746 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.056986+0000", 2024-09-21T12:34:52.746 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1770000000000001e-06 2024-09-21T12:34:52.746 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.746 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.746 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.747 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.125121+0000", 2024-09-21T12:34:52.747 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0681344740000001 2024-09-21T12:34:52.747 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.747 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.747 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:52.747 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.125232+0000", 2024-09-21T12:34:52.747 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000111875 2024-09-21T12:34:52.747 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.747 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.747 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:34:52.747 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.125348+0000", 2024-09-21T12:34:52.747 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000115696 2024-09-21T12:34:52.747 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.748 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.748 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.748 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.129616+0000", 2024-09-21T12:34:52.748 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0042675930000000001 2024-09-21T12:34:52.748 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.748 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.748 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.748 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.129653+0000", 2024-09-21T12:34:52.748 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7786e-05 2024-09-21T12:34:52.748 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.748 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.748 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.748 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.129734+0000", 2024-09-21T12:34:52.749 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1036000000000001e-05 2024-09-21T12:34:52.749 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.749 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.749 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:52.749 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134602+0000", 2024-09-21T12:34:52.749 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0048670479999999997 2024-09-21T12:34:52.749 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.749 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.749 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:52.749 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134664+0000", 2024-09-21T12:34:52.749 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2360999999999995e-05 2024-09-21T12:34:52.749 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.749 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.750 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:52.750 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134800+0000", 2024-09-21T12:34:52.750 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013576800000000001 2024-09-21T12:34:52.750 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.750 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:52.750 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.750 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.750 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.750 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:221972 9.d 9:b9fcc29c:::benchmark_data_smithi114_133214_object97938:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:34:52.750 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.096425+0000", 2024-09-21T12:34:52.750 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 194.607966632, 2024-09-21T12:34:52.750 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0387005569999999, 2024-09-21T12:34:52.750 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:52.751 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:52.751 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:52.751 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:52.751 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:34:52.751 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:34:52.751 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 221972 2024-09-21T12:34:52.751 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.751 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:52.751 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.751 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:52.751 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.096425+0000", 2024-09-21T12:34:52.751 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.752 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.752 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.752 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:52.752 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.096425+0000", 2024-09-21T12:34:52.752 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.752 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.752 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.752 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:52.752 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.096427+0000", 2024-09-21T12:34:52.753 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:52.754 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0099999999999998e-06 2024-09-21T12:34:52.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.754 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.754 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:52.754 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.096437+0000", 2024-09-21T12:34:52.754 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6530000000000006e-06 2024-09-21T12:34:52.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.754 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.754 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:52.755 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.096465+0000", 2024-09-21T12:34:52.755 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8048999999999999e-05 2024-09-21T12:34:52.755 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.755 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.755 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:52.755 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.096479+0000", 2024-09-21T12:34:52.755 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4517e-05 2024-09-21T12:34:52.755 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.755 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.755 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.755 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.096622+0000", 2024-09-21T12:34:52.755 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000142173 2024-09-21T12:34:52.755 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.755 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.756 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:52.756 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.096637+0000", 2024-09-21T12:34:52.756 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5792999999999999e-05 2024-09-21T12:34:52.756 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.756 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.756 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.756 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057015+0000", 2024-09-21T12:34:52.756 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.96037787799999996 2024-09-21T12:34:52.756 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.756 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.756 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:52.756 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057023+0000", 2024-09-21T12:34:52.756 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6580000000000002e-06 2024-09-21T12:34:52.757 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.757 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.757 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.757 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.127356+0000", 2024-09-21T12:34:52.757 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.070332735 2024-09-21T12:34:52.757 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.757 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.757 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:52.757 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.127461+0000", 2024-09-21T12:34:52.757 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000105509 2024-09-21T12:34:52.757 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.757 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.757 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:34:52.758 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.127579+0000", 2024-09-21T12:34:52.758 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000118016 2024-09-21T12:34:52.758 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.758 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.758 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.758 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134387+0000", 2024-09-21T12:34:52.758 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0068083450000000004 2024-09-21T12:34:52.758 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.758 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.758 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:52.758 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134811+0000", 2024-09-21T12:34:52.758 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00042396600000000001 2024-09-21T12:34:52.758 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.759 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.759 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.759 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134871+0000", 2024-09-21T12:34:52.759 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9574000000000003e-05 2024-09-21T12:34:52.759 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.759 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.759 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.759 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134949+0000", 2024-09-21T12:34:52.759 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8454000000000005e-05 2024-09-21T12:34:52.759 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.759 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.759 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:52.759 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135009+0000", 2024-09-21T12:34:52.759 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9676999999999998e-05 2024-09-21T12:34:52.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.760 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.760 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:52.760 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135126+0000", 2024-09-21T12:34:52.760 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011655 2024-09-21T12:34:52.760 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.760 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:52.760 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.761 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.761 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:221983 9.15 9:ae2f2fb3:::benchmark_data_smithi114_133214_object97949:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:34:52.761 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.108916+0000", 2024-09-21T12:34:52.761 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 194.595475534, 2024-09-21T12:34:52.761 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.022601734, 2024-09-21T12:34:52.761 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:52.761 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:52.761 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:52.761 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:52.761 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:34:52.761 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:34:52.761 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 221983 2024-09-21T12:34:52.761 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.762 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:52.762 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.762 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:52.762 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.108916+0000", 2024-09-21T12:34:52.762 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.762 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.762 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.762 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:52.762 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.108916+0000", 2024-09-21T12:34:52.762 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.762 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.762 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.762 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:52.763 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.108917+0000", 2024-09-21T12:34:52.763 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.111e-06 2024-09-21T12:34:52.763 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.763 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.763 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:52.763 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.108922+0000", 2024-09-21T12:34:52.763 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8280000000000002e-06 2024-09-21T12:34:52.763 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.763 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.763 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:52.763 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.108933+0000", 2024-09-21T12:34:52.763 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0939e-05 2024-09-21T12:34:52.763 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.763 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.763 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:52.763 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.108940+0000", 2024-09-21T12:34:52.764 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8530000000000003e-06 2024-09-21T12:34:52.764 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.764 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.764 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.764 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.109007+0000", 2024-09-21T12:34:52.764 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6643000000000006e-05 2024-09-21T12:34:52.764 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.764 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.764 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:52.764 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.109013+0000", 2024-09-21T12:34:52.764 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9349999999999999e-06 2024-09-21T12:34:52.764 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.765 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.765 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.765 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.126725+0000", 2024-09-21T12:34:52.765 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0177128740000001 2024-09-21T12:34:52.765 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.765 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.765 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:52.765 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.126867+0000", 2024-09-21T12:34:52.765 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014166499999999999 2024-09-21T12:34:52.765 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.765 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.765 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1,3,5", 2024-09-21T12:34:52.765 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.127184+0000", 2024-09-21T12:34:52.765 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00031702800000000001 2024-09-21T12:34:52.766 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.766 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.766 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.766 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.129977+0000", 2024-09-21T12:34:52.766 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002792475 2024-09-21T12:34:52.766 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.766 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.766 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:52.766 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131139+0000", 2024-09-21T12:34:52.766 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001162777 2024-09-21T12:34:52.766 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.766 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.766 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.766 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131244+0000", 2024-09-21T12:34:52.767 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000104324 2024-09-21T12:34:52.767 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.767 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.767 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.767 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131283+0000", 2024-09-21T12:34:52.767 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9199000000000002e-05 2024-09-21T12:34:52.767 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.767 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.767 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.767 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131333+0000", 2024-09-21T12:34:52.767 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0178e-05 2024-09-21T12:34:52.767 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.767 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.767 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:52.768 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131394+0000", 2024-09-21T12:34:52.768 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0554000000000003e-05 2024-09-21T12:34:52.768 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.768 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.768 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:52.768 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131518+0000", 2024-09-21T12:34:52.768 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000124351 2024-09-21T12:34:52.768 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.768 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:52.768 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.768 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.768 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.769 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:221997 9.d 9:b9573347:::benchmark_data_smithi114_133214_object97963:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:34:52.769 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.121318+0000", 2024-09-21T12:34:52.769 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 194.583073775, 2024-09-21T12:34:52.769 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.01426215, 2024-09-21T12:34:52.769 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:52.769 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:52.769 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:52.769 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:52.769 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:34:52.769 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:34:52.769 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 221997 2024-09-21T12:34:52.769 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.769 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:52.770 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.770 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:52.770 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.121318+0000", 2024-09-21T12:34:52.770 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.770 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.770 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.770 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:52.770 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.121318+0000", 2024-09-21T12:34:52.770 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.770 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.770 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.770 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:52.770 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.121319+0000", 2024-09-21T12:34:52.770 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.364e-06 2024-09-21T12:34:52.771 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.771 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.771 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:52.771 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.121324+0000", 2024-09-21T12:34:52.771 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5929999999999999e-06 2024-09-21T12:34:52.771 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.771 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.771 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:52.771 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.121334+0000", 2024-09-21T12:34:52.771 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8309999999999996e-06 2024-09-21T12:34:52.771 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.771 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.771 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:52.772 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.121341+0000", 2024-09-21T12:34:52.772 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0910000000000004e-06 2024-09-21T12:34:52.772 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.772 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.772 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.772 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.121402+0000", 2024-09-21T12:34:52.772 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0875999999999998e-05 2024-09-21T12:34:52.772 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.772 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.772 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:52.772 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.121409+0000", 2024-09-21T12:34:52.772 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.498e-06 2024-09-21T12:34:52.772 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.772 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.773 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.773 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057045+0000", 2024-09-21T12:34:52.773 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.93563575200000004 2024-09-21T12:34:52.773 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.773 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.773 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:52.773 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057053+0000", 2024-09-21T12:34:52.773 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9359999999999999e-06 2024-09-21T12:34:52.773 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.773 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.773 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.773 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.129765+0000", 2024-09-21T12:34:52.773 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0727123919999999 2024-09-21T12:34:52.773 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.774 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.774 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:52.774 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.129890+0000", 2024-09-21T12:34:52.774 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012480799999999999 2024-09-21T12:34:52.774 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.774 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.774 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:34:52.774 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.130027+0000", 2024-09-21T12:34:52.774 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000136425 2024-09-21T12:34:52.774 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.774 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.774 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:52.774 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134824+0000", 2024-09-21T12:34:52.774 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0047976219999999997 2024-09-21T12:34:52.775 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.775 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.775 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.775 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135204+0000", 2024-09-21T12:34:52.775 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000380054 2024-09-21T12:34:52.775 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.775 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.775 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.775 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135259+0000", 2024-09-21T12:34:52.775 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5071000000000002e-05 2024-09-21T12:34:52.775 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.775 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.776 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.776 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135393+0000", 2024-09-21T12:34:52.776 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000133991 2024-09-21T12:34:52.776 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.776 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.776 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:52.776 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135457+0000", 2024-09-21T12:34:52.776 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3830000000000004e-05 2024-09-21T12:34:52.776 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.776 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.776 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:52.776 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135580+0000", 2024-09-21T12:34:52.776 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012301599999999999 2024-09-21T12:34:52.776 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.777 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:52.777 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.777 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.777 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.777 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:222014 9.d 9:bb4c3379:::benchmark_data_smithi114_133214_object97980:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:34:52.777 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.130717+0000", 2024-09-21T12:34:52.777 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 194.57367507800001, 2024-09-21T12:34:52.777 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0050891000000002, 2024-09-21T12:34:52.777 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:52.777 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:52.777 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:52.777 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:52.777 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:34:52.778 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:34:52.778 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 222014 2024-09-21T12:34:52.778 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.778 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:52.778 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.778 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:52.778 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.130717+0000", 2024-09-21T12:34:52.778 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.778 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.778 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.778 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:52.778 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.130717+0000", 2024-09-21T12:34:52.778 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.779 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.779 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.779 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:52.779 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.130718+0000", 2024-09-21T12:34:52.779 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.052e-06 2024-09-21T12:34:52.779 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.779 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.779 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:52.779 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.130722+0000", 2024-09-21T12:34:52.779 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8469999999999998e-06 2024-09-21T12:34:52.779 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.779 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.779 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:52.779 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.130730+0000", 2024-09-21T12:34:52.780 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5029999999999997e-06 2024-09-21T12:34:52.780 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.780 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.780 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:52.780 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.130737+0000", 2024-09-21T12:34:52.781 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6420000000000001e-06 2024-09-21T12:34:52.781 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.781 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.781 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.782 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.130793+0000", 2024-09-21T12:34:52.782 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6498e-05 2024-09-21T12:34:52.782 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.782 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.782 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:52.782 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.130802+0000", 2024-09-21T12:34:52.782 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0080000000000008e-06 2024-09-21T12:34:52.782 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.782 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.782 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.782 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057076+0000", 2024-09-21T12:34:52.782 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.92627378500000002 2024-09-21T12:34:52.783 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.783 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.783 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:52.783 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057083+0000", 2024-09-21T12:34:52.783 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7410000000000003e-06 2024-09-21T12:34:52.783 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.783 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.783 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.783 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.131785+0000", 2024-09-21T12:34:52.783 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0747017809999999 2024-09-21T12:34:52.783 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.783 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.783 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:52.784 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.131913+0000", 2024-09-21T12:34:52.784 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012813399999999999 2024-09-21T12:34:52.784 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.784 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.784 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:34:52.784 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.132034+0000", 2024-09-21T12:34:52.784 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000121236 2024-09-21T12:34:52.784 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.784 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.784 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:52.784 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134886+0000", 2024-09-21T12:34:52.784 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0028518910000000001 2024-09-21T12:34:52.784 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.785 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.785 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.785 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135303+0000", 2024-09-21T12:34:52.785 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000417279 2024-09-21T12:34:52.787 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.787 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.787 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.787 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135347+0000", 2024-09-21T12:34:52.787 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4399e-05 2024-09-21T12:34:52.787 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.787 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.787 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.787 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135633+0000", 2024-09-21T12:34:52.787 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00028544300000000002 2024-09-21T12:34:52.787 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.787 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.788 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:52.788 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135690+0000", 2024-09-21T12:34:52.788 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6577999999999999e-05 2024-09-21T12:34:52.788 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.788 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.788 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:52.788 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135806+0000", 2024-09-21T12:34:52.788 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000116283 2024-09-21T12:34:52.788 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.788 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:52.788 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.788 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.789 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.789 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:222021 9.15 9:ac1e73ee:::benchmark_data_smithi114_133214_object97987:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:34:52.789 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.135261+0000", 2024-09-21T12:34:52.789 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 194.56913107099999, 2024-09-21T12:34:52.789 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.99670988900000002, 2024-09-21T12:34:52.789 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:52.789 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:52.789 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:52.789 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:52.789 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:34:52.789 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:34:52.789 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 222021 2024-09-21T12:34:52.790 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.790 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:52.790 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.790 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:52.790 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.135261+0000", 2024-09-21T12:34:52.790 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.790 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.790 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.790 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:52.790 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.135261+0000", 2024-09-21T12:34:52.790 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.791 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.791 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.791 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:52.791 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.135261+0000", 2024-09-21T12:34:52.791 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7100000000000001e-07 2024-09-21T12:34:52.792 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.792 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.792 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:52.792 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.135264+0000", 2024-09-21T12:34:52.792 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8779999999999998e-06 2024-09-21T12:34:52.792 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.792 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.792 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:52.792 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.135279+0000", 2024-09-21T12:34:52.792 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4713e-05 2024-09-21T12:34:52.792 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.792 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.792 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:52.792 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.135284+0000", 2024-09-21T12:34:52.793 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2630000000000003e-06 2024-09-21T12:34:52.793 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.793 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.793 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.793 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.135337+0000", 2024-09-21T12:34:52.793 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3134000000000003e-05 2024-09-21T12:34:52.793 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.793 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.793 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:52.793 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.135345+0000", 2024-09-21T12:34:52.793 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6750000000000002e-06 2024-09-21T12:34:52.793 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.793 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.793 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.793 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.128709+0000", 2024-09-21T12:34:52.794 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.99336335799999997 2024-09-21T12:34:52.794 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.794 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.794 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:52.794 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.128784+0000", 2024-09-21T12:34:52.794 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5585000000000002e-05 2024-09-21T12:34:52.794 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.794 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.794 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1,3,5", 2024-09-21T12:34:52.794 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.128998+0000", 2024-09-21T12:34:52.794 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000213506 2024-09-21T12:34:52.794 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.794 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.795 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.795 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131181+0000", 2024-09-21T12:34:52.795 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002183365 2024-09-21T12:34:52.795 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.795 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.795 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:52.795 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131256+0000", 2024-09-21T12:34:52.795 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4989999999999999e-05 2024-09-21T12:34:52.795 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.795 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.795 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.795 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131691+0000", 2024-09-21T12:34:52.795 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00043519000000000001 2024-09-21T12:34:52.796 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.796 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.796 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.796 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131793+0000", 2024-09-21T12:34:52.796 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010200300000000001 2024-09-21T12:34:52.796 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.796 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.796 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.796 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131850+0000", 2024-09-21T12:34:52.796 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7299000000000001e-05 2024-09-21T12:34:52.796 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.796 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.796 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:52.796 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131900+0000", 2024-09-21T12:34:52.797 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9459999999999997e-05 2024-09-21T12:34:52.797 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.797 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.797 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:52.797 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131971+0000", 2024-09-21T12:34:52.797 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0698999999999995e-05 2024-09-21T12:34:52.797 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.797 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:52.797 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.797 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.797 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.797 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:222036 9.d 9:bf4c3996:::benchmark_data_smithi114_133214_object98002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:34:52.797 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.149247+0000", 2024-09-21T12:34:52.797 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 194.55514487900001, 2024-09-21T12:34:52.797 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9939442509999998, 2024-09-21T12:34:52.798 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:52.798 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:52.798 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:52.798 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:52.798 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:34:52.798 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:34:52.798 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 222036 2024-09-21T12:34:52.798 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.798 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:52.798 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.798 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:52.798 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.149247+0000", 2024-09-21T12:34:52.798 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.799 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.799 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.799 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:52.799 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.149247+0000", 2024-09-21T12:34:52.799 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.799 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.799 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.799 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:52.799 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.149248+0000", 2024-09-21T12:34:52.799 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3200000000000001e-06 2024-09-21T12:34:52.799 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.799 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.799 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:52.799 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.149253+0000", 2024-09-21T12:34:52.799 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2129999999999999e-06 2024-09-21T12:34:52.800 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.800 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.800 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:52.800 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.149263+0000", 2024-09-21T12:34:52.800 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2580000000000001e-06 2024-09-21T12:34:52.800 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.800 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.800 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:52.800 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.149269+0000", 2024-09-21T12:34:52.800 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5379999999999999e-06 2024-09-21T12:34:52.800 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.800 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.800 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.801 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.149332+0000", 2024-09-21T12:34:52.801 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2496999999999995e-05 2024-09-21T12:34:52.802 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.802 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.802 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:52.802 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.149343+0000", 2024-09-21T12:34:52.802 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1011e-05 2024-09-21T12:34:52.802 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.802 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.802 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.802 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057106+0000", 2024-09-21T12:34:52.802 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.90776296199999995 2024-09-21T12:34:52.803 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.803 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.803 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:52.803 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057113+0000", 2024-09-21T12:34:52.803 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4250000000000001e-06 2024-09-21T12:34:52.803 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.803 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.803 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.803 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135854+0000", 2024-09-21T12:34:52.803 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.078740493 2024-09-21T12:34:52.803 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.803 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.803 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:52.804 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135971+0000", 2024-09-21T12:34:52.804 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000117704 2024-09-21T12:34:52.804 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.804 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.804 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:34:52.804 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.136102+0000", 2024-09-21T12:34:52.804 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013025100000000001 2024-09-21T12:34:52.804 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.804 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.804 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:52.804 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.142810+0000", 2024-09-21T12:34:52.804 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0067082419999999997 2024-09-21T12:34:52.805 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.805 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.805 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.805 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.142861+0000", 2024-09-21T12:34:52.805 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1381000000000002e-05 2024-09-21T12:34:52.805 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.805 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.805 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.805 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.142919+0000", 2024-09-21T12:34:52.805 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7482000000000003e-05 2024-09-21T12:34:52.805 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.805 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.805 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.805 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.143004+0000", 2024-09-21T12:34:52.806 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5746000000000002e-05 2024-09-21T12:34:52.806 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.806 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.806 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:52.806 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.143072+0000", 2024-09-21T12:34:52.806 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8009000000000006e-05 2024-09-21T12:34:52.806 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.806 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.806 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:52.806 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.143191+0000", 2024-09-21T12:34:52.806 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000118719 2024-09-21T12:34:52.807 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.807 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:52.807 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.807 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.807 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.807 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:222038 9.d 9:b5ba6170:::benchmark_data_smithi114_133214_object98004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:34:52.807 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.150309+0000", 2024-09-21T12:34:52.807 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 194.55408298500001, 2024-09-21T12:34:52.807 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.995585494, 2024-09-21T12:34:52.807 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:52.807 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:52.808 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:52.808 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:52.808 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:34:52.808 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:34:52.808 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 222038 2024-09-21T12:34:52.808 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.808 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:52.808 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.808 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:52.808 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.150309+0000", 2024-09-21T12:34:52.808 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.808 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.808 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.808 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:52.809 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.150309+0000", 2024-09-21T12:34:52.809 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.809 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.809 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.809 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:52.809 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.150310+0000", 2024-09-21T12:34:52.809 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8800000000000001e-07 2024-09-21T12:34:52.809 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.809 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.809 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:52.809 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.150313+0000", 2024-09-21T12:34:52.809 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.58e-06 2024-09-21T12:34:52.809 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.810 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.810 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:52.810 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.150322+0000", 2024-09-21T12:34:52.810 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2600000000000005e-06 2024-09-21T12:34:52.810 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.810 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.810 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:52.810 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.150328+0000", 2024-09-21T12:34:52.810 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3790000000000003e-06 2024-09-21T12:34:52.810 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.810 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.810 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.810 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.150385+0000", 2024-09-21T12:34:52.810 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7188000000000002e-05 2024-09-21T12:34:52.811 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.811 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.811 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:52.811 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.150394+0000", 2024-09-21T12:34:52.811 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1360000000000003e-06 2024-09-21T12:34:52.811 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.811 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.811 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.811 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057143+0000", 2024-09-21T12:34:52.811 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.90674834100000001 2024-09-21T12:34:52.811 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.811 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.811 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:52.811 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057150+0000", 2024-09-21T12:34:52.811 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.481e-06 2024-09-21T12:34:52.812 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.812 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.812 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.812 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.138249+0000", 2024-09-21T12:34:52.812 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0810986 2024-09-21T12:34:52.812 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.812 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.812 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:52.812 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.138359+0000", 2024-09-21T12:34:52.812 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011010699999999999 2024-09-21T12:34:52.812 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.812 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.812 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:34:52.812 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.138473+0000", 2024-09-21T12:34:52.813 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000114271 2024-09-21T12:34:52.813 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.813 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.813 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:52.813 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.142879+0000", 2024-09-21T12:34:52.813 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0044062709999999998 2024-09-21T12:34:52.813 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.813 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.813 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.813 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.142962+0000", 2024-09-21T12:34:52.813 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2497999999999995e-05 2024-09-21T12:34:52.813 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.813 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.814 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.814 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.143245+0000", 2024-09-21T12:34:52.814 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00028279200000000003 2024-09-21T12:34:52.814 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.814 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.814 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.814 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.145676+0000", 2024-09-21T12:34:52.814 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024311929999999999 2024-09-21T12:34:52.814 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.814 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.814 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:52.814 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.145735+0000", 2024-09-21T12:34:52.814 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9159000000000001e-05 2024-09-21T12:34:52.814 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.815 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.815 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:52.815 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.145894+0000", 2024-09-21T12:34:52.815 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015935 2024-09-21T12:34:52.815 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.815 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:52.815 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.815 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.815 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.815 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:222040 9.15 9:afba0a7e:::benchmark_data_smithi114_133214_object98006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:34:52.815 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.152391+0000", 2024-09-21T12:34:52.815 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 194.55200082900001, 2024-09-21T12:34:52.815 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.979744434, 2024-09-21T12:34:52.815 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:52.816 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:52.816 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:52.816 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:52.816 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:34:52.816 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:34:52.816 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 222040 2024-09-21T12:34:52.816 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.816 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:52.816 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.816 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:52.816 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.152391+0000", 2024-09-21T12:34:52.816 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.816 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.817 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.817 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:52.817 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.152391+0000", 2024-09-21T12:34:52.817 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.817 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.817 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.817 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:52.817 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.152392+0000", 2024-09-21T12:34:52.817 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.09e-07 2024-09-21T12:34:52.817 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.817 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.817 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:52.817 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.152395+0000", 2024-09-21T12:34:52.817 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.534e-06 2024-09-21T12:34:52.818 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.818 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.818 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:52.818 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.152406+0000", 2024-09-21T12:34:52.818 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0779e-05 2024-09-21T12:34:52.818 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.818 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.818 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:52.818 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.152411+0000", 2024-09-21T12:34:52.818 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5079999999999999e-06 2024-09-21T12:34:52.818 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.818 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.818 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.818 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.152451+0000", 2024-09-21T12:34:52.819 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0462e-05 2024-09-21T12:34:52.819 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.819 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.819 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:52.819 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.152457+0000", 2024-09-21T12:34:52.819 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6389999999999997e-06 2024-09-21T12:34:52.819 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.819 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.819 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.819 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.130005+0000", 2024-09-21T12:34:52.819 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.97754845099999998 2024-09-21T12:34:52.820 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.820 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.820 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:52.820 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.130083+0000", 2024-09-21T12:34:52.820 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8159999999999997e-05 2024-09-21T12:34:52.820 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.820 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.820 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1,3,5", 2024-09-21T12:34:52.820 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.130252+0000", 2024-09-21T12:34:52.820 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016846799999999999 2024-09-21T12:34:52.820 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.820 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.820 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:52.820 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131532+0000", 2024-09-21T12:34:52.821 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012803300000000001 2024-09-21T12:34:52.821 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.821 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.821 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.821 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131727+0000", 2024-09-21T12:34:52.821 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019453999999999999 2024-09-21T12:34:52.821 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.821 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.821 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.821 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.131758+0000", 2024-09-21T12:34:52.821 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1538000000000002e-05 2024-09-21T12:34:52.821 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.822 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.822 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.822 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.132004+0000", 2024-09-21T12:34:52.822 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000245758 2024-09-21T12:34:52.822 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.822 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.822 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.822 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.132035+0000", 2024-09-21T12:34:52.822 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0871000000000003e-05 2024-09-21T12:34:52.822 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.822 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.822 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:52.822 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.132064+0000", 2024-09-21T12:34:52.822 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9549000000000001e-05 2024-09-21T12:34:52.823 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.823 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.823 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:52.823 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.132135+0000", 2024-09-21T12:34:52.823 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1038000000000007e-05 2024-09-21T12:34:52.823 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.823 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:52.823 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.823 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.823 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.823 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:222057 9.d 9:bd682287:::benchmark_data_smithi114_133214_object98023:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:34:52.823 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.194600+0000", 2024-09-21T12:34:52.823 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 194.50979175000001, 2024-09-21T12:34:52.823 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9643363250000001, 2024-09-21T12:34:52.824 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:52.824 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:52.824 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:52.824 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:52.824 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:34:52.824 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:34:52.824 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 222057 2024-09-21T12:34:52.824 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.824 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:52.824 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.824 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:52.824 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.194600+0000", 2024-09-21T12:34:52.824 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.825 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.825 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.825 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:52.825 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.194600+0000", 2024-09-21T12:34:52.825 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.825 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.825 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.825 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:52.825 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.194601+0000", 2024-09-21T12:34:52.825 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.15e-06 2024-09-21T12:34:52.825 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.825 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.825 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:52.825 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.194605+0000", 2024-09-21T12:34:52.826 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1500000000000001e-06 2024-09-21T12:34:52.826 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.826 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.826 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:52.826 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.194616+0000", 2024-09-21T12:34:52.826 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0759e-05 2024-09-21T12:34:52.826 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.826 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.826 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:52.826 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.194622+0000", 2024-09-21T12:34:52.826 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.614e-06 2024-09-21T12:34:52.826 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.826 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.827 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.827 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.194683+0000", 2024-09-21T12:34:52.827 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0934000000000003e-05 2024-09-21T12:34:52.827 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.827 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.827 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:52.827 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.194690+0000", 2024-09-21T12:34:52.827 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3189999999999997e-06 2024-09-21T12:34:52.827 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.827 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.827 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.827 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057173+0000", 2024-09-21T12:34:52.827 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.862483322 2024-09-21T12:34:52.828 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.828 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.828 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:52.828 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057180+0000", 2024-09-21T12:34:52.828 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0600000000000002e-06 2024-09-21T12:34:52.828 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.828 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.828 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.828 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.140485+0000", 2024-09-21T12:34:52.828 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0833051250000001 2024-09-21T12:34:52.828 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.828 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.828 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:52.828 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.140621+0000", 2024-09-21T12:34:52.829 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013514899999999999 2024-09-21T12:34:52.829 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.829 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.829 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:34:52.829 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.140741+0000", 2024-09-21T12:34:52.829 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012082399999999999 2024-09-21T12:34:52.829 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.829 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.829 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.829 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.145628+0000", 2024-09-21T12:34:52.829 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0048861720000000003 2024-09-21T12:34:52.829 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.829 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.830 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.830 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.145926+0000", 2024-09-21T12:34:52.830 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00029853099999999999 2024-09-21T12:34:52.830 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.830 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.830 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:52.830 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.150100+0000", 2024-09-21T12:34:52.830 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0041734930000000003 2024-09-21T12:34:52.830 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.830 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.830 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.831 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.158801+0000", 2024-09-21T12:34:52.831 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0087013130000000004 2024-09-21T12:34:52.831 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.831 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.831 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:52.831 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.158849+0000", 2024-09-21T12:34:52.831 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7604e-05 2024-09-21T12:34:52.831 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.831 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.831 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:52.831 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.158936+0000", 2024-09-21T12:34:52.831 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7806000000000006e-05 2024-09-21T12:34:52.831 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.832 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:52.832 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.832 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.832 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.832 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:222065 9.d 9:b27ebeb8:::benchmark_data_smithi114_133214_object98031:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-09-21T12:34:52.832 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:39.140786+0000", 2024-09-21T12:34:52.832 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 193.56360554599999, 2024-09-21T12:34:52.832 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0183983160000001, 2024-09-21T12:34:52.832 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:52.832 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:52.832 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:52.832 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:52.832 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:34:52.832 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:34:52.833 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 222065 2024-09-21T12:34:52.833 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.833 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:52.833 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.833 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:52.833 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.140786+0000", 2024-09-21T12:34:52.833 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.833 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.833 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.833 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:52.833 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.140786+0000", 2024-09-21T12:34:52.833 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.833 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.834 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.834 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:52.834 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.140787+0000", 2024-09-21T12:34:52.834 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1650000000000001e-06 2024-09-21T12:34:52.834 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.834 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.834 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:52.834 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.140792+0000", 2024-09-21T12:34:52.834 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0930000000000002e-06 2024-09-21T12:34:52.834 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.834 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.834 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:52.834 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.140803+0000", 2024-09-21T12:34:52.835 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1e-05 2024-09-21T12:34:52.835 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.835 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.835 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:52.835 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.140811+0000", 2024-09-21T12:34:52.835 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1919999999999999e-06 2024-09-21T12:34:52.835 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.835 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.835 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.835 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.140887+0000", 2024-09-21T12:34:52.835 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5916999999999998e-05 2024-09-21T12:34:52.835 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.835 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.836 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:52.836 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.140899+0000", 2024-09-21T12:34:52.836 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1966e-05 2024-09-21T12:34:52.836 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.836 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.836 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.836 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.143276+0000", 2024-09-21T12:34:52.836 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.002377023 2024-09-21T12:34:52.836 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.836 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.836 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:52.836 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.143396+0000", 2024-09-21T12:34:52.836 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012068 2024-09-21T12:34:52.836 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.837 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.837 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:34:52.837 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.143523+0000", 2024-09-21T12:34:52.837 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012700799999999999 2024-09-21T12:34:52.837 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.837 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.837 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.837 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.150083+0000", 2024-09-21T12:34:52.837 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0065597399999999997 2024-09-21T12:34:52.837 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.837 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.838 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.838 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.150166+0000", 2024-09-21T12:34:52.838 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2736000000000002e-05 2024-09-21T12:34:52.838 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.838 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.838 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:52.838 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.152621+0000", 2024-09-21T12:34:52.838 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024554550000000001 2024-09-21T12:34:52.838 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.838 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.838 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.838 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.158993+0000", 2024-09-21T12:34:52.838 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0063721029999999996 2024-09-21T12:34:52.839 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.839 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.839 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:52.839 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.159060+0000", 2024-09-21T12:34:52.839 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6316999999999995e-05 2024-09-21T12:34:52.839 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.839 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.839 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:52.839 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.159185+0000", 2024-09-21T12:34:52.839 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012492100000000001 2024-09-21T12:34:52.839 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.839 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:52.839 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.839 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.839 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.840 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:4750 10.c 10:39bcf0f1:::benchmark_data_smithi114_161690_object4749:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-09-21T12:34:52.840 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:32:40.393690+0000", 2024-09-21T12:34:52.840 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 132.31070134300001, 2024-09-21T12:34:52.840 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.008228159, 2024-09-21T12:34:52.840 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:52.840 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:52.840 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:52.840 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:52.840 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:34:52.840 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:34:52.840 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 4750 2024-09-21T12:34:52.840 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.840 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:52.840 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.841 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:52.841 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.393690+0000", 2024-09-21T12:34:52.841 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.841 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.841 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.841 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:52.841 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.393690+0000", 2024-09-21T12:34:52.841 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.841 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.841 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.841 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:52.841 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.393693+0000", 2024-09-21T12:34:52.841 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0980000000000001e-06 2024-09-21T12:34:52.842 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.842 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.842 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:52.842 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.393750+0000", 2024-09-21T12:34:52.842 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7367999999999997e-05 2024-09-21T12:34:52.842 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.842 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.842 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:52.842 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.393760+0000", 2024-09-21T12:34:52.842 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0001e-05 2024-09-21T12:34:52.842 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.842 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.842 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:52.843 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.393766+0000", 2024-09-21T12:34:52.843 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7320000000000003e-06 2024-09-21T12:34:52.843 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.843 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.843 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.843 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.394595+0000", 2024-09-21T12:34:52.843 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00082910000000000004 2024-09-21T12:34:52.843 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.843 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.843 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:52.843 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.394605+0000", 2024-09-21T12:34:52.843 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9580000000000002e-06 2024-09-21T12:34:52.843 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.844 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.844 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.844 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.397764+0000", 2024-09-21T12:34:52.844 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0031589670000001 2024-09-21T12:34:52.844 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.844 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.844 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:52.844 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.397895+0000", 2024-09-21T12:34:52.844 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013103999999999999 2024-09-21T12:34:52.844 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.844 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.844 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,4", 2024-09-21T12:34:52.844 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.398023+0000", 2024-09-21T12:34:52.844 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000128594 2024-09-21T12:34:52.844 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.845 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.845 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:52.845 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.400685+0000", 2024-09-21T12:34:52.845 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0026614859999999998 2024-09-21T12:34:52.845 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.845 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.845 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.845 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401610+0000", 2024-09-21T12:34:52.845 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00092512699999999998 2024-09-21T12:34:52.845 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.845 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.845 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.845 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401802+0000", 2024-09-21T12:34:52.845 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019185099999999999 2024-09-21T12:34:52.845 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.846 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.846 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:52.846 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401871+0000", 2024-09-21T12:34:52.846 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9015000000000002e-05 2024-09-21T12:34:52.846 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.846 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.846 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:52.846 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401919+0000", 2024-09-21T12:34:52.846 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7821999999999997e-05 2024-09-21T12:34:52.846 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.846 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:52.846 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.846 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.846 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.847 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:4777 10.c 10:3cea9318:::benchmark_data_smithi114_161690_object4776:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-09-21T12:34:52.847 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:32:40.417960+0000", 2024-09-21T12:34:52.847 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 132.28643165099999, 2024-09-21T12:34:52.847 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.98421197500000002, 2024-09-21T12:34:52.847 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:52.847 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:52.847 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:52.847 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:52.847 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:34:52.847 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:34:52.847 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 4777 2024-09-21T12:34:52.847 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.847 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:52.847 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.848 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:52.848 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.417960+0000", 2024-09-21T12:34:52.848 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.848 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.848 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.848 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:52.848 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.417960+0000", 2024-09-21T12:34:52.848 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.848 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.848 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.848 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:52.848 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.417961+0000", 2024-09-21T12:34:52.848 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.313e-06 2024-09-21T12:34:52.848 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.849 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.849 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:52.849 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.418017+0000", 2024-09-21T12:34:52.849 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5729000000000001e-05 2024-09-21T12:34:52.849 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.849 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.849 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:52.849 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.418027+0000", 2024-09-21T12:34:52.849 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3789999999999995e-06 2024-09-21T12:34:52.849 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.849 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.849 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:52.849 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.418031+0000", 2024-09-21T12:34:52.849 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6380000000000003e-06 2024-09-21T12:34:52.849 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.850 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.850 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.850 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.418091+0000", 2024-09-21T12:34:52.850 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0069999999999999e-05 2024-09-21T12:34:52.850 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.850 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.850 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:52.850 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.418100+0000", 2024-09-21T12:34:52.850 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1500000000000005e-06 2024-09-21T12:34:52.850 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.850 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.850 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.850 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.398597+0000", 2024-09-21T12:34:52.850 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.98049644000000002 2024-09-21T12:34:52.851 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.851 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.851 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:52.851 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.398697+0000", 2024-09-21T12:34:52.851 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100028 2024-09-21T12:34:52.851 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.851 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.851 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,4", 2024-09-21T12:34:52.851 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.398833+0000", 2024-09-21T12:34:52.851 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013587000000000001 2024-09-21T12:34:52.851 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.851 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.851 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:52.851 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.400690+0000", 2024-09-21T12:34:52.851 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001857455 2024-09-21T12:34:52.852 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.852 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.852 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.852 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401652+0000", 2024-09-21T12:34:52.852 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00096136099999999999 2024-09-21T12:34:52.852 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.852 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.852 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.852 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.402070+0000", 2024-09-21T12:34:52.852 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00041827100000000001 2024-09-21T12:34:52.852 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.852 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.853 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:52.853 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.402129+0000", 2024-09-21T12:34:52.853 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9407000000000002e-05 2024-09-21T12:34:52.853 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.853 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.853 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:52.853 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.402172+0000", 2024-09-21T12:34:52.853 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2864000000000002e-05 2024-09-21T12:34:52.853 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.853 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:52.853 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.853 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.853 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.853 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14478.0:3151 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94620848896768 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-21T12:34:52.853 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:34:33.327034+0000", 2024-09-21T12:34:52.854 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 19.377357983, 2024-09-21T12:34:52.854 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.014477877, 2024-09-21T12:34:52.854 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:52.854 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:52.854 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:34:52.854 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:52.854 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14478", 2024-09-21T12:34:52.854 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:34:52.854 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 3151 2024-09-21T12:34:52.854 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.854 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:52.854 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.854 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:52.854 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.327034+0000", 2024-09-21T12:34:52.855 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.855 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.855 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.855 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:52.855 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.327034+0000", 2024-09-21T12:34:52.855 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.855 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.855 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.855 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:52.855 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.327035+0000", 2024-09-21T12:34:52.855 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4649999999999999e-06 2024-09-21T12:34:52.855 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.855 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.855 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:52.856 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.327040+0000", 2024-09-21T12:34:52.856 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.578e-06 2024-09-21T12:34:52.856 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.856 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.856 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:52.856 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.327053+0000", 2024-09-21T12:34:52.856 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3477e-05 2024-09-21T12:34:52.856 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.856 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.856 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:52.856 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.327061+0000", 2024-09-21T12:34:52.856 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2150000000000004e-06 2024-09-21T12:34:52.856 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.856 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.856 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.857 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.327129+0000", 2024-09-21T12:34:52.857 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8140999999999997e-05 2024-09-21T12:34:52.857 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.857 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.857 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op must wait for map", 2024-09-21T12:34:52.857 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.327134+0000", 2024-09-21T12:34:52.857 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.339e-06 2024-09-21T12:34:52.857 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.857 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.857 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.857 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.333808+0000", 2024-09-21T12:34:52.857 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0066744309999999998 2024-09-21T12:34:52.857 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.857 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.857 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:52.857 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.333823+0000", 2024-09-21T12:34:52.858 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4929000000000001e-05 2024-09-21T12:34:52.858 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.858 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.858 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.858 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.338567+0000", 2024-09-21T12:34:52.859 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0047435229999999 2024-09-21T12:34:52.859 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.859 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.859 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:52.859 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.338733+0000", 2024-09-21T12:34:52.859 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016607899999999999 2024-09-21T12:34:52.859 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.859 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.860 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:52.860 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.341512+0000", 2024-09-21T12:34:52.860 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0027786999999999998 2024-09-21T12:34:52.860 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.860 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:52.860 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.860 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.860 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.860 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:160304 10.9 10:929fb0d0:::benchmark_data_smithi114_161690_object33252:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-21T12:34:52.860 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:34:33.352195+0000", 2024-09-21T12:34:52.860 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 19.352196822, 2024-09-21T12:34:52.860 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0185030420000001, 2024-09-21T12:34:52.860 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:52.861 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:52.861 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:52.861 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:52.861 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:34:52.861 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:34:52.861 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 160304 2024-09-21T12:34:52.861 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.861 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:52.861 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.861 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:52.861 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.352195+0000", 2024-09-21T12:34:52.861 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.862 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.862 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.862 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:52.862 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.352195+0000", 2024-09-21T12:34:52.862 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.862 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.862 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.862 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:52.862 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.352196+0000", 2024-09-21T12:34:52.862 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.279e-06 2024-09-21T12:34:52.862 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.862 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.863 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:52.863 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.352201+0000", 2024-09-21T12:34:52.863 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5719999999999996e-06 2024-09-21T12:34:52.863 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.863 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.863 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:52.863 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.352212+0000", 2024-09-21T12:34:52.863 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1657e-05 2024-09-21T12:34:52.863 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.863 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.863 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:52.863 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.352218+0000", 2024-09-21T12:34:52.863 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7019999999999998e-06 2024-09-21T12:34:52.863 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.864 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.864 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.864 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.352263+0000", 2024-09-21T12:34:52.864 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5308000000000001e-05 2024-09-21T12:34:52.864 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.864 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.864 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:52.864 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.352272+0000", 2024-09-21T12:34:52.864 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1610000000000003e-06 2024-09-21T12:34:52.864 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.864 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.864 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.864 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.365346+0000", 2024-09-21T12:34:52.865 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0130743250000001 2024-09-21T12:34:52.865 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.865 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.865 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:52.865 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.365443+0000", 2024-09-21T12:34:52.865 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7501e-05 2024-09-21T12:34:52.865 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.865 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.865 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5", 2024-09-21T12:34:52.865 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.365545+0000", 2024-09-21T12:34:52.865 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000101523 2024-09-21T12:34:52.865 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.865 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.865 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:52.866 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.370110+0000", 2024-09-21T12:34:52.866 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0045650250000000003 2024-09-21T12:34:52.866 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.866 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.866 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.866 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.370576+0000", 2024-09-21T12:34:52.866 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00046550299999999998 2024-09-21T12:34:52.866 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.866 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.866 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:52.866 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.370627+0000", 2024-09-21T12:34:52.866 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1088000000000003e-05 2024-09-21T12:34:52.867 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.867 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.867 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:52.867 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.370698+0000", 2024-09-21T12:34:52.867 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1397999999999996e-05 2024-09-21T12:34:52.867 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.867 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:52.867 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.867 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.867 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.867 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:160317 10.9 10:9d9a98a7:::benchmark_data_smithi114_161690_object33265:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-21T12:34:52.867 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:34:33.375525+0000", 2024-09-21T12:34:52.867 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 19.328866654999999, 2024-09-21T12:34:52.867 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0011166, 2024-09-21T12:34:52.868 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:34:52.868 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:34:52.868 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:34:52.868 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:34:52.868 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:34:52.868 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:34:52.868 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 160317 2024-09-21T12:34:52.868 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.868 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:34:52.868 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.868 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:34:52.868 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.375525+0000", 2024-09-21T12:34:52.868 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.868 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.869 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.869 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:34:52.869 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.375525+0000", 2024-09-21T12:34:52.869 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:34:52.869 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.869 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.869 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:34:52.869 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.375526+0000", 2024-09-21T12:34:52.869 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3629999999999999e-06 2024-09-21T12:34:52.869 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.869 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.869 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:34:52.869 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.375531+0000", 2024-09-21T12:34:52.870 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5689999999999998e-06 2024-09-21T12:34:52.870 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.870 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.870 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:34:52.870 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.375544+0000", 2024-09-21T12:34:52.870 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2574e-05 2024-09-21T12:34:52.870 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.870 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.870 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:34:52.870 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.375551+0000", 2024-09-21T12:34:52.870 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6399999999999997e-06 2024-09-21T12:34:52.870 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.870 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.870 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.871 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.375598+0000", 2024-09-21T12:34:52.871 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7095000000000002e-05 2024-09-21T12:34:52.871 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.871 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.871 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:34:52.871 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.375605+0000", 2024-09-21T12:34:52.871 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8170000000000001e-06 2024-09-21T12:34:52.871 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.871 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.871 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:34:52.871 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.370726+0000", 2024-09-21T12:34:52.871 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.99512048600000003 2024-09-21T12:34:52.871 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.871 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.872 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:34:52.872 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.370832+0000", 2024-09-21T12:34:52.872 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000106351 2024-09-21T12:34:52.872 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.872 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.872 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5", 2024-09-21T12:34:52.872 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.370937+0000", 2024-09-21T12:34:52.872 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000105394 2024-09-21T12:34:52.872 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.872 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.872 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:34:52.872 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.374227+0000", 2024-09-21T12:34:52.872 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0032890990000000002 2024-09-21T12:34:52.872 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.873 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.873 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:34:52.873 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.376339+0000", 2024-09-21T12:34:52.873 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021121489999999998 2024-09-21T12:34:52.873 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.873 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.873 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:34:52.873 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.376549+0000", 2024-09-21T12:34:52.873 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00021054 2024-09-21T12:34:52.873 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:34:52.873 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:34:52.873 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:34:52.873 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.376642+0000", 2024-09-21T12:34:52.874 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2522999999999995e-05 2024-09-21T12:34:52.874 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.874 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:52.874 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.874 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:34:52.874 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:34:52.874 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:34:52.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:52.875 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:52.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:52.910 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:34:52.976 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:53.009 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:53.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:53.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:53.024 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:34:52 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:34:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:34:53.110 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:53.149 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:53.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:53.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:53.251 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:53.272 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:53.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:53.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:53.374 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:53.409 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:53.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:53.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:53.476 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -n osd.2 --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op export --pgid 4.c --file /var/log/ceph/exp.4.c.2 2024-09-21T12:34:53.511 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:53.538 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:53.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:53.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:53.640 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:53.673 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:53.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:53.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:53.757 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/osd.2/config 2024-09-21T12:34:53.775 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:53.797 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:53.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:53.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:53.899 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:53.941 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:53.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:53.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:54.043 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:54.056 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:34:54.056 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:34:54.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:54.070 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:54.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:54.079 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:34:54.080 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:34:54.080 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T12:34:54.106 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:34:54.106 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:34:54.140 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:34:54.140 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:34:54.170 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:34:54.170 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:34:54.170 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:54.193 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:34:54.193 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:34:54.201 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:54.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:54.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:54.228 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:34:54.228 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:34:54.303 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:54.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:54 smithi114 ceph-mon[30324]: pgmap v747: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.51k op/s; 26438/211203 objects degraded (12.518%) 2024-09-21T12:34:54.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:54 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2603145212' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:34:54.308 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:54 smithi114 ceph-mon[40867]: pgmap v747: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.51k op/s; 26438/211203 objects degraded (12.518%) 2024-09-21T12:34:54.308 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:54 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2603145212' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:34:54.335 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:54.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:54.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:54.437 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:54.472 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:54.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:54.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:54.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:54 smithi183 ceph-mon[33277]: pgmap v747: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.51k op/s; 26438/211203 objects degraded (12.518%) 2024-09-21T12:34:54.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:54 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2603145212' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:34:54.574 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:54.616 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:54.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:54.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:54.718 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:54.749 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:54.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:54.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:54.754 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T12:34:54.851 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:54.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:54.887 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:54.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:54.988 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:55.010 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T12:34:55.011 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T12:34:55.024 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:55.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:55.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:55.126 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:55.158 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:55.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:55.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:55.260 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:55.285 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:55.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:55.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:55.386 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:55.413 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:55.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:55.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:55.515 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:55.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:55 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 25041/202896 objects degraded (12.342%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T12:34:55.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:55 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 25041/202896 objects degraded (12.342%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T12:34:55.543 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:55.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:55.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:55.645 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:55.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:55.671 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:55.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:55.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:55 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 25041/202896 objects degraded (12.342%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T12:34:55.772 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:55.807 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:55.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:55.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:55.907 INFO:teuthology.orchestra.run.smithi114.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T12:34:55.909 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:55.940 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:34:55.940 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:55.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:55.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:56.042 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:56.073 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:56.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:56.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:56.175 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:56.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:56.197 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:56.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:56.298 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:56.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:56.330 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:56.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:56.431 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:56.469 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:56.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:56.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:56.521 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:56 smithi114 ceph-mon[40867]: pgmap v748: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 4.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 598 B/s rd, 0 B/s wr, 1.74k op/s; 25041/202896 objects degraded (12.342%) 2024-09-21T12:34:56.521 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:56 smithi114 ceph-mon[30324]: pgmap v748: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 4.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 598 B/s rd, 0 B/s wr, 1.74k op/s; 25041/202896 objects degraded (12.342%) 2024-09-21T12:34:56.571 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:56.604 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:56.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:56.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:56.706 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:56.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:56.736 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:56.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:56.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:56 smithi183 ceph-mon[33277]: pgmap v748: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 4.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 598 B/s rd, 0 B/s wr, 1.74k op/s; 25041/202896 objects degraded (12.342%) 2024-09-21T12:34:56.837 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:56.871 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:56.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:56.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:56.974 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:57.011 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:57.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:57.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:57.012 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:34:57.112 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:57.143 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:57.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:57.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:57.245 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:57.267 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:57.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:57.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:57.369 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:57.399 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:34:57.401 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:57.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:57.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:57.503 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:57.533 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:57.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:57.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:57.634 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:57.666 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:57.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:57.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:57.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:34:57 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:34:57.768 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:57.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:57.804 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:57.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:57.905 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:57.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:57.938 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:57.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:58.012 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:34:58.012 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:34:58.039 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:58.051 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T12:34:58.052 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:34:58.053 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:34:58.055 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T12:34:58.073 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:58.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:58.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:58.094 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:34:58.094 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:34:58.123 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:34:58.124 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:34:58.160 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:34:58.160 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:34:58.174 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:58.195 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:58.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:58.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:58.249 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:34:58.250 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:34:58.268 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-21T12:34:58.296 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:34:58.296 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:34:58.297 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:58.354 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:58.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:58.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:58.457 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:34:58.499 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:58.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:58.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:34:58.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:58 smithi114 ceph-mon[30324]: pgmap v749: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 4.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 1.49k op/s; 25041/202896 objects degraded (12.342%) 2024-09-21T12:34:58.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:58 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2225220899' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:34:58.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:58 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3620215418' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:34:58.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:58 smithi114 ceph-mon[40867]: pgmap v749: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 4.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 1.49k op/s; 25041/202896 objects degraded (12.342%) 2024-09-21T12:34:58.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:58 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2225220899' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:34:58.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:58 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3620215418' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:34:58.601 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:58.635 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:58.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:58.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:58.737 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:58.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:58 smithi183 ceph-mon[33277]: pgmap v749: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 4.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 1.49k op/s; 25041/202896 objects degraded (12.342%) 2024-09-21T12:34:58.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:58 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2225220899' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:34:58.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:58 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3620215418' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:34:58.777 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:58.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:58.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:58.879 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:34:58.909 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:58.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:58.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:34:59.011 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:34:59.047 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:59.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:59.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:34:59.149 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:59.180 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:59.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:59.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:59.281 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:34:59.315 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:59.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:59.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:34:59.417 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:59.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:59.449 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:59.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:59.550 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:34:59.585 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:59.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:59.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:34:59.687 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:59.708 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:59.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:59.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:34:59.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:59 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:34:59.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:59 smithi183 ceph-mon[33277]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:34:59.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:59 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2225220899' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:34:59.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:34:59 smithi183 ceph-mon[33277]: osdmap e396: 8 total, 7 up, 8 in 2024-09-21T12:34:59.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:59 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:34:59.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:59 smithi114 ceph-mon[30324]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:34:59.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:59 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2225220899' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:34:59.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:34:59 smithi114 ceph-mon[30324]: osdmap e396: 8 total, 7 up, 8 in 2024-09-21T12:34:59.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:59 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:34:59.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:59 smithi114 ceph-mon[40867]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:34:59.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:59 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2225220899' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:34:59.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:34:59 smithi114 ceph-mon[40867]: osdmap e396: 8 total, 7 up, 8 in 2024-09-21T12:34:59.810 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:34:59.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:59.848 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:34:59.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:34:59.949 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:34:59.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:34:59.984 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:34:59.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:00.085 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:00.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:00.121 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:00.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:00.222 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:00.258 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:00.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:00.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:00.361 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:00.393 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:00.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:00.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:00.495 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:00.514 INFO:teuthology.orchestra.run.smithi114.stderr:Exporting 4.c info 4.c( v 276'20 (0'0,276'20] local-lis/les=349/350 n=5 ec=60/60 lis/c=349/349 les/c/f=350/350/0 sis=349) 2024-09-21T12:35:00.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:00 smithi114 ceph-mon[40867]: pgmap v751: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 3.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 541 B/s rd, 0 B/s wr, 1.52k op/s; 22507/181992 objects degraded (12.367%) 2024-09-21T12:35:00.515 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:00 smithi114 ceph-mon[30324]: pgmap v751: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 3.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 541 B/s rd, 0 B/s wr, 1.52k op/s; 22507/181992 objects degraded (12.367%) 2024-09-21T12:35:00.516 INFO:teuthology.orchestra.run.smithi114.stderr:Read #4:31288c1b:::obj_delete_at_hint.0000000096:head# 2024-09-21T12:35:00.516 INFO:teuthology.orchestra.run.smithi114.stderr:Read #4:330b1641:::obj_delete_at_hint.0000000087:head# 2024-09-21T12:35:00.516 INFO:teuthology.orchestra.run.smithi114.stderr:Read #4:338eb7e9:::obj_delete_at_hint.0000000008:head# 2024-09-21T12:35:00.517 INFO:teuthology.orchestra.run.smithi114.stderr:Read #4:3677c228:::obj_delete_at_hint.0000000006:head# 2024-09-21T12:35:00.517 INFO:teuthology.orchestra.run.smithi114.stderr:Read #4:36abfd60:::obj_delete_at_hint.0000000029:head# 2024-09-21T12:35:00.517 INFO:teuthology.orchestra.run.smithi114.stderr:Export successful 2024-09-21T12:35:00.526 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:00.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:00.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:00.628 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:00.659 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:00.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:00.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:00.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:00 smithi183 ceph-mon[33277]: pgmap v751: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 3.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 541 B/s rd, 0 B/s wr, 1.52k op/s; 22507/181992 objects degraded (12.367%) 2024-09-21T12:35:00.760 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:00.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:00.798 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:00.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:00.899 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:00.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:00.928 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:00.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:01.028 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:01.063 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:01.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:01.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:01.108 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T12:35:01.108 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T12:35:01.165 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:01.199 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:01.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:01.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:01.301 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:01.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:01.341 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:01.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:01.442 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:01.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:01.480 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:01.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:01.581 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:01.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:01.611 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:01.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:01.713 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:01.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:01.747 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:01.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:01.759 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:01.806 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -n osd.2 --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --force --op remove --pgid 4.c 2024-09-21T12:35:01.848 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:01.888 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:01.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:01.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:01.990 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:02.025 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:02.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:02.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:02.127 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:02.158 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:02.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:02.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:02.224 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T12:35:02.259 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:02.260 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/osd.2/config 2024-09-21T12:35:02.286 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:02.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:02.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:02.367 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:02 smithi114 ceph-mon[30324]: pgmap v752: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 3.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 1.44k op/s; 22507/181992 objects degraded (12.367%) 2024-09-21T12:35:02.367 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:02 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 22507/181992 objects degraded (12.367%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T12:35:02.367 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:02 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:35:02.367 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:02 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:35:02.367 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:02 smithi114 ceph-mon[40867]: pgmap v752: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 3.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 1.44k op/s; 22507/181992 objects degraded (12.367%) 2024-09-21T12:35:02.367 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:02 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 22507/181992 objects degraded (12.367%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T12:35:02.367 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:02 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:35:02.367 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:02 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:35:02.388 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:02.417 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:02.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:02.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:02.425 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:35:02.425 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-21T12:35:02.425 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.425 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.15924.0:201275 10.b e396/344 10:d8a412e1:::benchmark_data_smithi114_161690_object74223:head v 396'12651, mlcod=396'12649)", 2024-09-21T12:35:02.425 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:35:02.414157+0000", 2024-09-21T12:35:02.425 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0074059260000000002, 2024-09-21T12:35:02.425 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0074263869999999996, 2024-09-21T12:35:02.425 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:35:02.425 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:35:02.425 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:35:02.425 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:35:02.426 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.426 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:35:02.426 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.414157+0000", 2024-09-21T12:35:02.426 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:02.426 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.426 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.426 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:35:02.426 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.414157+0000", 2024-09-21T12:35:02.426 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:02.426 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.426 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.426 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:35:02.426 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.414158+0000", 2024-09-21T12:35:02.426 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9599999999999999e-07 2024-09-21T12:35:02.427 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.427 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.427 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:35:02.427 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.414162+0000", 2024-09-21T12:35:02.427 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5040000000000004e-06 2024-09-21T12:35:02.427 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.427 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.427 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:35:02.427 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.414172+0000", 2024-09-21T12:35:02.427 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0321e-05 2024-09-21T12:35:02.427 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.427 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.427 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:35:02.427 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.414179+0000", 2024-09-21T12:35:02.428 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1330000000000004e-06 2024-09-21T12:35:02.428 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.428 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.428 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:02.428 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.414323+0000", 2024-09-21T12:35:02.428 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014494900000000001 2024-09-21T12:35:02.428 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.428 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.428 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:35:02.428 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.414341+0000", 2024-09-21T12:35:02.428 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7428999999999999e-05 2024-09-21T12:35:02.428 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:02.428 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:02.428 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:02.429 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.429 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.429 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.15924.0:201279 10.b e396/344 10:d41504b6:::benchmark_data_smithi114_161690_object74227:head v 396'12652, mlcod=396'12649)", 2024-09-21T12:35:02.429 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:35:02.414197+0000", 2024-09-21T12:35:02.429 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.007365562, 2024-09-21T12:35:02.429 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0074182609999999998, 2024-09-21T12:35:02.429 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:35:02.429 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:35:02.429 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:35:02.429 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:35:02.429 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.429 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:35:02.429 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.414197+0000", 2024-09-21T12:35:02.429 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:02.429 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.430 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.430 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:35:02.430 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.414197+0000", 2024-09-21T12:35:02.430 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:02.430 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.430 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.430 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:35:02.430 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.414198+0000", 2024-09-21T12:35:02.430 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4999999999999998e-07 2024-09-21T12:35:02.430 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.430 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.430 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:35:02.430 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.414201+0000", 2024-09-21T12:35:02.430 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.3519999999999998e-06 2024-09-21T12:35:02.431 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.431 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.431 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:35:02.431 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.414206+0000", 2024-09-21T12:35:02.431 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4739999999999996e-06 2024-09-21T12:35:02.431 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.431 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.431 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:35:02.431 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.414210+0000", 2024-09-21T12:35:02.431 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0670000000000002e-06 2024-09-21T12:35:02.431 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.431 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.431 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:02.432 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.417233+0000", 2024-09-21T12:35:02.432 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0030226189999999998 2024-09-21T12:35:02.432 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.432 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.432 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:35:02.432 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.417253+0000", 2024-09-21T12:35:02.432 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9539e-05 2024-09-21T12:35:02.432 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:02.432 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:02.432 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:02.432 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.432 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.432 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.15924.0:201282 10.b e396/344 10:d698b8fa:::benchmark_data_smithi114_161690_object74230:head v 396'12653, mlcod=396'12649)", 2024-09-21T12:35:02.432 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:35:02.414216+0000", 2024-09-21T12:35:02.432 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0073463909999999999, 2024-09-21T12:35:02.433 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.007425498, 2024-09-21T12:35:02.433 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:35:02.433 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:35:02.433 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:35:02.433 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:35:02.433 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.433 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:35:02.433 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.414216+0000", 2024-09-21T12:35:02.433 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:02.433 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.433 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.433 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:35:02.433 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.414216+0000", 2024-09-21T12:35:02.434 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:02.434 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.434 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.434 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:35:02.434 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.414217+0000", 2024-09-21T12:35:02.434 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9399999999999999e-07 2024-09-21T12:35:02.434 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.434 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.434 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:35:02.434 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.414222+0000", 2024-09-21T12:35:02.434 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9050000000000002e-06 2024-09-21T12:35:02.434 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.434 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.434 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:35:02.435 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.414227+0000", 2024-09-21T12:35:02.435 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7119999999999999e-06 2024-09-21T12:35:02.435 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.435 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.435 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:35:02.435 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.414231+0000", 2024-09-21T12:35:02.435 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8469999999999998e-06 2024-09-21T12:35:02.435 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.435 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.435 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:02.435 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.419872+0000", 2024-09-21T12:35:02.435 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0056406720000000002 2024-09-21T12:35:02.435 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.436 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.436 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:35:02.436 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.419893+0000", 2024-09-21T12:35:02.436 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1106000000000001e-05 2024-09-21T12:35:02.436 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:02.436 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:02.436 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:02.436 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.436 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.436 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.15924.0:201296 10.4 10:24d64bfc:::benchmark_data_smithi114_161690_object74244:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e396)", 2024-09-21T12:35:02.436 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:35:02.419769+0000", 2024-09-21T12:35:02.437 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0017941859999999999, 2024-09-21T12:35:02.437 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0019016059999999999, 2024-09-21T12:35:02.437 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:35:02.437 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:35:02.437 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:35:02.437 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:35:02.437 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.15924", 2024-09-21T12:35:02.437 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:35:02.438 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 201296 2024-09-21T12:35:02.438 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.438 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:35:02.438 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.438 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:35:02.438 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.419769+0000", 2024-09-21T12:35:02.438 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:02.438 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.438 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.438 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:35:02.438 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.419769+0000", 2024-09-21T12:35:02.438 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:02.438 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.439 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.439 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:35:02.439 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.419770+0000", 2024-09-21T12:35:02.439 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2890000000000001e-06 2024-09-21T12:35:02.439 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.439 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.439 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:35:02.439 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.419775+0000", 2024-09-21T12:35:02.439 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1030000000000001e-06 2024-09-21T12:35:02.439 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.439 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.439 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:35:02.439 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.419785+0000", 2024-09-21T12:35:02.440 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0285999999999999e-05 2024-09-21T12:35:02.440 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.440 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.440 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:35:02.440 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.419792+0000", 2024-09-21T12:35:02.440 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6379999999999998e-06 2024-09-21T12:35:02.440 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.440 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.440 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:02.440 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.419854+0000", 2024-09-21T12:35:02.440 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2081000000000004e-05 2024-09-21T12:35:02.440 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.440 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.440 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:35:02.441 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.419937+0000", 2024-09-21T12:35:02.441 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.2829000000000003e-05 2024-09-21T12:35:02.441 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.441 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.441 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,4", 2024-09-21T12:35:02.441 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.420010+0000", 2024-09-21T12:35:02.441 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2871000000000001e-05 2024-09-21T12:35:02.441 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:02.441 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:02.441 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:02.441 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.441 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.441 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.15924.0:201295 10.a e396/344 10:506a96f9:::benchmark_data_smithi114_161690_object74243:head v 396'12742, mlcod=396'12740)", 2024-09-21T12:35:02.442 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:35:02.420017+0000", 2024-09-21T12:35:02.442 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.001545659, 2024-09-21T12:35:02.442 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0016868600000000001, 2024-09-21T12:35:02.442 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:35:02.442 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:35:02.442 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:35:02.442 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:35:02.442 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.442 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:35:02.442 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.420017+0000", 2024-09-21T12:35:02.442 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:02.442 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.442 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.443 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:35:02.443 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.420017+0000", 2024-09-21T12:35:02.443 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:02.443 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.443 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.443 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:35:02.443 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.420018+0000", 2024-09-21T12:35:02.443 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2900000000000003e-07 2024-09-21T12:35:02.443 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.443 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.443 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:35:02.443 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.420029+0000", 2024-09-21T12:35:02.443 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1389000000000001e-05 2024-09-21T12:35:02.444 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.444 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.444 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:35:02.444 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.420038+0000", 2024-09-21T12:35:02.444 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5189999999999992e-06 2024-09-21T12:35:02.444 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.444 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.444 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:35:02.444 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.420044+0000", 2024-09-21T12:35:02.444 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7150000000000001e-06 2024-09-21T12:35:02.444 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.444 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.444 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:02.444 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.420093+0000", 2024-09-21T12:35:02.444 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8696999999999998e-05 2024-09-21T12:35:02.445 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.445 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.445 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:35:02.445 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.420111+0000", 2024-09-21T12:35:02.445 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7901e-05 2024-09-21T12:35:02.445 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:02.445 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:02.445 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:02.445 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.445 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.445 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop_reply(client.15924.0:201276 10.4 e396/353)", 2024-09-21T12:35:02.445 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:35:02.420518+0000", 2024-09-21T12:35:02.445 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.001044513, 2024-09-21T12:35:02.446 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0012089379999999999, 2024-09-21T12:35:02.446 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:35:02.446 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:35:02.446 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "queued for pg", 2024-09-21T12:35:02.446 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:35:02.446 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.446 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:35:02.446 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.420518+0000", 2024-09-21T12:35:02.446 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:02.446 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.446 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.446 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:35:02.446 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.420518+0000", 2024-09-21T12:35:02.446 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:02.446 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.447 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.447 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:35:02.447 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.420519+0000", 2024-09-21T12:35:02.447 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.92e-07 2024-09-21T12:35:02.447 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.447 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.447 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:35:02.447 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.420523+0000", 2024-09-21T12:35:02.447 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8179999999999997e-06 2024-09-21T12:35:02.447 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.447 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.447 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:35:02.447 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.420532+0000", 2024-09-21T12:35:02.447 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.2140000000000002e-06 2024-09-21T12:35:02.447 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.448 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.448 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:35:02.448 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.420538+0000", 2024-09-21T12:35:02.448 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8989999999999998e-06 2024-09-21T12:35:02.448 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:02.448 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:02.448 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:02.448 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.448 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.448 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.15924.0:201297 10.1 e396/352 10:8065d899:::benchmark_data_smithi114_161690_object74245:head v 396'12551, mlcod=396'12549)", 2024-09-21T12:35:02.448 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:35:02.420554+0000", 2024-09-21T12:35:02.448 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0010088429999999999, 2024-09-21T12:35:02.448 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0011937499999999999, 2024-09-21T12:35:02.448 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:35:02.448 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:35:02.449 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:35:02.449 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:35:02.449 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.449 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:35:02.449 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.420554+0000", 2024-09-21T12:35:02.449 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:02.449 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.449 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.449 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:35:02.449 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.420554+0000", 2024-09-21T12:35:02.449 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:02.449 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.449 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.449 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:35:02.449 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.420554+0000", 2024-09-21T12:35:02.450 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4999999999999998e-07 2024-09-21T12:35:02.450 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.450 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.450 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:35:02.450 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.420557+0000", 2024-09-21T12:35:02.450 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.1360000000000001e-06 2024-09-21T12:35:02.450 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.450 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.450 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:35:02.450 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.420563+0000", 2024-09-21T12:35:02.450 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6149999999999996e-06 2024-09-21T12:35:02.450 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.450 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.450 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:35:02.450 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.420567+0000", 2024-09-21T12:35:02.451 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0579999999999999e-06 2024-09-21T12:35:02.451 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.451 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.451 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:02.451 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.420642+0000", 2024-09-21T12:35:02.451 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4552e-05 2024-09-21T12:35:02.451 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.451 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.451 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:35:02.451 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.420662+0000", 2024-09-21T12:35:02.451 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0380000000000001e-05 2024-09-21T12:35:02.451 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:02.451 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:02.451 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:02.451 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.451 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.452 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.15924.0:201276 10.4 10:22f8756f:::benchmark_data_smithi114_161690_object74224:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e396)", 2024-09-21T12:35:02.452 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:35:02.404926+0000", 2024-09-21T12:35:02.452 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.016636730999999998, 2024-09-21T12:35:02.452 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.016848063999999999, 2024-09-21T12:35:02.452 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:35:02.452 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:35:02.452 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:35:02.452 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:35:02.452 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.15924", 2024-09-21T12:35:02.452 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:35:02.452 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 201276 2024-09-21T12:35:02.452 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.452 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:35:02.452 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.452 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:35:02.452 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.404926+0000", 2024-09-21T12:35:02.453 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:02.453 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.453 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.453 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:35:02.453 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.404926+0000", 2024-09-21T12:35:02.453 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:02.453 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.453 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.453 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:35:02.453 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.404927+0000", 2024-09-21T12:35:02.453 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2669999999999999e-06 2024-09-21T12:35:02.453 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.453 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.453 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:35:02.453 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.404932+0000", 2024-09-21T12:35:02.454 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0409999999999997e-06 2024-09-21T12:35:02.454 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.454 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.454 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:35:02.454 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.404944+0000", 2024-09-21T12:35:02.454 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1722000000000001e-05 2024-09-21T12:35:02.454 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.454 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.454 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:35:02.454 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.404952+0000", 2024-09-21T12:35:02.454 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.2500000000000006e-06 2024-09-21T12:35:02.454 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.454 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.454 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:02.455 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.405007+0000", 2024-09-21T12:35:02.455 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4477e-05 2024-09-21T12:35:02.455 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.455 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.455 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:35:02.455 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.405148+0000", 2024-09-21T12:35:02.455 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014157899999999999 2024-09-21T12:35:02.455 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.455 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.455 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,4", 2024-09-21T12:35:02.455 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.405267+0000", 2024-09-21T12:35:02.455 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000118616 2024-09-21T12:35:02.455 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.455 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.456 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:35:02.456 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.412112+0000", 2024-09-21T12:35:02.456 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0068452740000000002 2024-09-21T12:35:02.456 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:02.456 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:02.456 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:02.456 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:02.413690+0000", 2024-09-21T12:35:02.456 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001577815 2024-09-21T12:35:02.456 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:02.456 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:02.456 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:02.456 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:02.456 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:35:02.456 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 8 2024-09-21T12:35:02.457 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:35:02.519 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:02.548 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:02.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:02.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:02.647 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:35:02 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:35:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:35:02.649 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:02.682 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:02.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:02.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:02.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:02 smithi183 ceph-mon[33277]: pgmap v752: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 3.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 1.44k op/s; 22507/181992 objects degraded (12.367%) 2024-09-21T12:35:02.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:02 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 22507/181992 objects degraded (12.367%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T12:35:02.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:02 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:35:02.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:02 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:35:02.784 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:02.818 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:02.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:02.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:02.920 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:02.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:02.956 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:02.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:03.057 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:03.101 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:03.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:03.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:03.109 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:35:03.203 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:03.237 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:03.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:03.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:03.339 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:03.383 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:03.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:03.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:03.393 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T12:35:03.485 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:03.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:03.519 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:03.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:03.620 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:03.652 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:03.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:03.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:03.754 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:03.789 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:03.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:03.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:03.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:03.920 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:03.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:03.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:04.022 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:04.055 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:04.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:04.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:04.157 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:04.190 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:04.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:04.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:04.292 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:04.323 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:04.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:04.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:04.405 INFO:teuthology.orchestra.run.smithi114.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T12:35:04.425 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:04.427 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:04.453 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:04.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:04.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:04.467 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:04.555 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:04.588 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:04 smithi114 ceph-mon[30324]: pgmap v753: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s; 21108/167847 objects degraded (12.576%) 2024-09-21T12:35:04.588 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:04 smithi114 ceph-mon[40867]: pgmap v753: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s; 21108/167847 objects degraded (12.576%) 2024-09-21T12:35:04.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:04.590 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:04.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:04.691 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:04.730 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:04.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:04.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:04.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:04 smithi183 ceph-mon[33277]: pgmap v753: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s; 21108/167847 objects degraded (12.576%) 2024-09-21T12:35:04.832 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:04.866 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:04.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:04.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:04.967 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:05.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:05.002 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:05.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:05.103 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:05.136 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:05.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:05.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:05.238 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:05.272 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:05.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:05.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:05.373 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:05.395 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:05.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:05.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:05.497 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:05.533 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:05.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:05.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:05.635 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:05.669 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:05.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:05.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:05.771 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:05.805 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:05.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:05.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:05.907 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:05.952 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:05.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:05.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:06.053 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:06.083 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:06.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:06.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:06.089 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:06.185 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:06.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:06.225 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:06.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:06.325 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:06.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:06.366 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:06.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:06.467 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:06.499 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:06.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:06.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:06.601 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:06.628 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:06.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:06.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:06.730 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:06.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:06 smithi183 ceph-mon[33277]: pgmap v754: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.46k op/s; 19680/159102 objects degraded (12.369%) 2024-09-21T12:35:06.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:06 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/724342636' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:35:06.765 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:06.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:06.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:06.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:06 smithi114 ceph-mon[40867]: pgmap v754: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.46k op/s; 19680/159102 objects degraded (12.369%) 2024-09-21T12:35:06.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:06 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/724342636' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:35:06.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:06 smithi114 ceph-mon[30324]: pgmap v754: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.46k op/s; 19680/159102 objects degraded (12.369%) 2024-09-21T12:35:06.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:06 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/724342636' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:35:06.868 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:06.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:06.902 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:06.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:07.003 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:07.034 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:07.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:07.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:07.136 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:07.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:07.168 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:07.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:07.239 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-21T12:35:07.269 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:07.306 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:07.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:07.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:07.408 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:07.440 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:07.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:07.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:07.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:07 smithi114 ceph-mon[30324]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:35:07.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:07 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/724342636' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:35:07.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:07 smithi114 ceph-mon[30324]: osdmap e397: 8 total, 7 up, 8 in 2024-09-21T12:35:07.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:07 smithi114 ceph-mon[30324]: 5.9 scrub starts 2024-09-21T12:35:07.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:07 smithi114 ceph-mon[30324]: 5.9 scrub ok 2024-09-21T12:35:07.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:07 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 19680/159102 objects degraded (12.369%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T12:35:07.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:07 smithi114 ceph-mon[40867]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:35:07.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:07 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/724342636' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:35:07.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:07 smithi114 ceph-mon[40867]: osdmap e397: 8 total, 7 up, 8 in 2024-09-21T12:35:07.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:07 smithi114 ceph-mon[40867]: 5.9 scrub starts 2024-09-21T12:35:07.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:07 smithi114 ceph-mon[40867]: 5.9 scrub ok 2024-09-21T12:35:07.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:07 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 19680/159102 objects degraded (12.369%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T12:35:07.542 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:07.573 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:07.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:07.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:07.674 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:07.709 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:07.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:07.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:07.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:35:07 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:35:07.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:07 smithi183 ceph-mon[33277]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:35:07.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:07 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/724342636' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:35:07.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:07 smithi183 ceph-mon[33277]: osdmap e397: 8 total, 7 up, 8 in 2024-09-21T12:35:07.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:07 smithi183 ceph-mon[33277]: 5.9 scrub starts 2024-09-21T12:35:07.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:07 smithi183 ceph-mon[33277]: 5.9 scrub ok 2024-09-21T12:35:07.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:07 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 19680/159102 objects degraded (12.369%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T12:35:07.811 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:07.848 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:07.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:07.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:07.950 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:07.984 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:07.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:07.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:08.085 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:08.124 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:08.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:08.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:08.226 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:08.264 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:08.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:08.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:08.365 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:08.400 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:08.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:08.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:08.501 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:08.530 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:08.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:08.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:08.632 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:08.668 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:08.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:08.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:08.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:08 smithi183 ceph-mon[33277]: pgmap v756: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 610 B/s rd, 0 B/s wr, 1.11k op/s; 19680/159102 objects degraded (12.369%) 2024-09-21T12:35:08.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:08 smithi183 ceph-mon[33277]: 6.1f scrub starts 2024-09-21T12:35:08.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:08 smithi183 ceph-mon[33277]: 6.1f scrub ok 2024-09-21T12:35:08.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:08 smithi183 ceph-mon[33277]: 6.5 scrub starts 2024-09-21T12:35:08.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:08 smithi183 ceph-mon[33277]: 6.5 scrub ok 2024-09-21T12:35:08.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:08 smithi183 ceph-mon[33277]: 6.a scrub starts 2024-09-21T12:35:08.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:08 smithi183 ceph-mon[33277]: 6.a scrub ok 2024-09-21T12:35:08.769 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:08.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:08 smithi114 ceph-mon[40867]: pgmap v756: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 610 B/s rd, 0 B/s wr, 1.11k op/s; 19680/159102 objects degraded (12.369%) 2024-09-21T12:35:08.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:08 smithi114 ceph-mon[40867]: 6.1f scrub starts 2024-09-21T12:35:08.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:08 smithi114 ceph-mon[40867]: 6.1f scrub ok 2024-09-21T12:35:08.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:08 smithi114 ceph-mon[40867]: 6.5 scrub starts 2024-09-21T12:35:08.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:08 smithi114 ceph-mon[40867]: 6.5 scrub ok 2024-09-21T12:35:08.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:08 smithi114 ceph-mon[40867]: 6.a scrub starts 2024-09-21T12:35:08.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:08 smithi114 ceph-mon[40867]: 6.a scrub ok 2024-09-21T12:35:08.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:08 smithi114 ceph-mon[30324]: pgmap v756: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 610 B/s rd, 0 B/s wr, 1.11k op/s; 19680/159102 objects degraded (12.369%) 2024-09-21T12:35:08.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:08 smithi114 ceph-mon[30324]: 6.1f scrub starts 2024-09-21T12:35:08.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:08 smithi114 ceph-mon[30324]: 6.1f scrub ok 2024-09-21T12:35:08.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:08 smithi114 ceph-mon[30324]: 6.5 scrub starts 2024-09-21T12:35:08.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:08 smithi114 ceph-mon[30324]: 6.5 scrub ok 2024-09-21T12:35:08.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:08 smithi114 ceph-mon[30324]: 6.a scrub starts 2024-09-21T12:35:08.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:08 smithi114 ceph-mon[30324]: 6.a scrub ok 2024-09-21T12:35:08.803 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:08.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:08.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:08.905 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:08.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:08.936 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:08.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:09.037 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:09.069 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:09.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:09.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:09.170 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:09.207 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:09.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:09.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:09.308 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:09.340 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:09.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:09.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:09.442 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:09.474 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:09.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:09.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:09.575 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:09.616 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:09.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:09.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:09.706 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:09 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:35:09.706 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:09 smithi114 ceph-mon[30324]: 10.c scrub starts 2024-09-21T12:35:09.706 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:09 smithi114 ceph-mon[30324]: 3.3 scrub starts 2024-09-21T12:35:09.706 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:09 smithi114 ceph-mon[30324]: 3.3 scrub ok 2024-09-21T12:35:09.707 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:09 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:35:09.707 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:09 smithi114 ceph-mon[40867]: 10.c scrub starts 2024-09-21T12:35:09.707 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:09 smithi114 ceph-mon[40867]: 3.3 scrub starts 2024-09-21T12:35:09.707 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:09 smithi114 ceph-mon[40867]: 3.3 scrub ok 2024-09-21T12:35:09.718 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:09.740 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:09 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:35:09.740 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:09 smithi183 ceph-mon[33277]: 10.c scrub starts 2024-09-21T12:35:09.740 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:09 smithi183 ceph-mon[33277]: 3.3 scrub starts 2024-09-21T12:35:09.740 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:09 smithi183 ceph-mon[33277]: 3.3 scrub ok 2024-09-21T12:35:09.755 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:09.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:09.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:09.858 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:09.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:09.893 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:09.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:09.994 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:10.036 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:10.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:10.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:10.103 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:35:10.103 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:35:10.138 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:10.159 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:35:10.159 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:35:10.159 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:35:10.159 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:35:10.160 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:35:10.160 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:35:10.160 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T12:35:10.160 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:35:10.160 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:35:10.172 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:10.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:10.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:10.188 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:35:10.188 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:35:10.220 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:35:10.220 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:35:10.260 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:35:10.260 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:35:10.274 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:10.296 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:10.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:10.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:10.398 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:10.421 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:10.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:10.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:10.523 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:10.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:10 smithi114 ceph-mon[40867]: pgmap v757: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.55k op/s; 16732/135462 objects degraded (12.352%) 2024-09-21T12:35:10.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:10 smithi114 ceph-mon[40867]: 10.c scrub ok 2024-09-21T12:35:10.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:10 smithi114 ceph-mon[40867]: 5.1a scrub starts 2024-09-21T12:35:10.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:10 smithi114 ceph-mon[40867]: 5.1a scrub ok 2024-09-21T12:35:10.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:10 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1551209324' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:35:10.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:10 smithi114 ceph-mon[30324]: pgmap v757: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.55k op/s; 16732/135462 objects degraded (12.352%) 2024-09-21T12:35:10.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:10 smithi114 ceph-mon[30324]: 10.c scrub ok 2024-09-21T12:35:10.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:10 smithi114 ceph-mon[30324]: 5.1a scrub starts 2024-09-21T12:35:10.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:10 smithi114 ceph-mon[30324]: 5.1a scrub ok 2024-09-21T12:35:10.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:10 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1551209324' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:35:10.546 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:10.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:10.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:10.648 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:10.684 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:10.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:10.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:10.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:10 smithi183 ceph-mon[33277]: pgmap v757: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.55k op/s; 16732/135462 objects degraded (12.352%) 2024-09-21T12:35:10.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:10 smithi183 ceph-mon[33277]: 10.c scrub ok 2024-09-21T12:35:10.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:10 smithi183 ceph-mon[33277]: 5.1a scrub starts 2024-09-21T12:35:10.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:10 smithi183 ceph-mon[33277]: 5.1a scrub ok 2024-09-21T12:35:10.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:10 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1551209324' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:35:10.786 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:10.816 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:10.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:10.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:10.918 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:10.953 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:10.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:10.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:10.984 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T12:35:11.055 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:11.117 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:11.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:11.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:11.218 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:11.249 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:11.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:11.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:11.351 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:11.391 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:11.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:11.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:11.493 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:11.527 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:11.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:11.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:11.629 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:11.663 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:11.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:11.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:11.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:11 smithi183 ceph-mon[33277]: pgmap v758: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.55k op/s; 16732/135462 objects degraded (12.352%) 2024-09-21T12:35:11.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:11 smithi183 ceph-mon[33277]: 5.1f scrub starts 2024-09-21T12:35:11.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:11 smithi183 ceph-mon[33277]: 5.1f scrub ok 2024-09-21T12:35:11.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:11 smithi183 ceph-mon[33277]: 4.b scrub starts 2024-09-21T12:35:11.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:11 smithi183 ceph-mon[33277]: 4.b scrub ok 2024-09-21T12:35:11.765 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:11.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:11 smithi114 ceph-mon[30324]: pgmap v758: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.55k op/s; 16732/135462 objects degraded (12.352%) 2024-09-21T12:35:11.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:11 smithi114 ceph-mon[30324]: 5.1f scrub starts 2024-09-21T12:35:11.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:11 smithi114 ceph-mon[30324]: 5.1f scrub ok 2024-09-21T12:35:11.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:11 smithi114 ceph-mon[30324]: 4.b scrub starts 2024-09-21T12:35:11.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:11 smithi114 ceph-mon[30324]: 4.b scrub ok 2024-09-21T12:35:11.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:11 smithi114 ceph-mon[40867]: pgmap v758: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.55k op/s; 16732/135462 objects degraded (12.352%) 2024-09-21T12:35:11.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:11 smithi114 ceph-mon[40867]: 5.1f scrub starts 2024-09-21T12:35:11.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:11 smithi114 ceph-mon[40867]: 5.1f scrub ok 2024-09-21T12:35:11.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:11 smithi114 ceph-mon[40867]: 4.b scrub starts 2024-09-21T12:35:11.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:11 smithi114 ceph-mon[40867]: 4.b scrub ok 2024-09-21T12:35:11.798 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:11.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:11.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:11.900 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:11.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:11.936 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:11.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:12.036 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:12.069 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:12.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:12.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:12.171 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:12.204 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:12.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:12.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:12.306 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:12.308 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T12:35:12.308 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T12:35:12.351 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:12.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:12.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:12.362 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:12.454 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:12.493 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:12.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:12.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:12.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:12 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 16732/135462 objects degraded (12.352%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T12:35:12.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:12 smithi114 ceph-mon[30324]: 4.9 scrub starts 2024-09-21T12:35:12.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:12 smithi114 ceph-mon[30324]: 4.9 scrub ok 2024-09-21T12:35:12.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:12 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 16732/135462 objects degraded (12.352%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T12:35:12.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:12 smithi114 ceph-mon[40867]: 4.9 scrub starts 2024-09-21T12:35:12.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:12 smithi114 ceph-mon[40867]: 4.9 scrub ok 2024-09-21T12:35:12.563 INFO:teuthology.orchestra.run.smithi114.stdout: marking collection for removal 2024-09-21T12:35:12.563 INFO:teuthology.orchestra.run.smithi114.stdout:setting '_remove' omap key 2024-09-21T12:35:12.564 INFO:teuthology.orchestra.run.smithi114.stdout:finish_remove_pgs 4.c_head removing 4.c 2024-09-21T12:35:12.569 INFO:teuthology.orchestra.run.smithi114.stdout:Remove successful 2024-09-21T12:35:12.594 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:12.626 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:12.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:12.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:12.727 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:12.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:12 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 16732/135462 objects degraded (12.352%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T12:35:12.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:12 smithi183 ceph-mon[33277]: 4.9 scrub starts 2024-09-21T12:35:12.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:12 smithi183 ceph-mon[33277]: 4.9 scrub ok 2024-09-21T12:35:12.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:12.769 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:12.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:12.830 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:35:12 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:35:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:35:12.870 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:12.906 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:12.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:12.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:12.989 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:35:12.989 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:35:13.008 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:13.039 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:13.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:13.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:13.141 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:13.176 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:13.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:13.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:13.278 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:13.309 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:13.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:13.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:13.411 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:13.441 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:13.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:13.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:13.543 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:13.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:13.575 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:13.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:13.591 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T12:35:13.677 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:13.712 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:13.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:13.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:13.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:13 smithi183 ceph-mon[33277]: pgmap v759: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.54k op/s; 15380/121746 objects degraded (12.633%) 2024-09-21T12:35:13.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:13 smithi114 ceph-mon[40867]: pgmap v759: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.54k op/s; 15380/121746 objects degraded (12.633%) 2024-09-21T12:35:13.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:13 smithi114 ceph-mon[30324]: pgmap v759: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.54k op/s; 15380/121746 objects degraded (12.633%) 2024-09-21T12:35:13.814 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:13.853 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:13.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:13.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:13.880 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell -n osd.2 --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.c.2 2024-09-21T12:35:13.955 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:14.004 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:14.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:14.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:14.106 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:14.126 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:14.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:14.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:14.228 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:14.261 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:14.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:14.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:14.309 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:35:14.363 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:14.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:14.398 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:14.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:14.499 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:14.537 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:14.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:14.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:14.639 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:14.679 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:14.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:14.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:14.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:14 smithi183 ceph-mon[33277]: 4.0 scrub starts 2024-09-21T12:35:14.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:14 smithi183 ceph-mon[33277]: 4.0 scrub ok 2024-09-21T12:35:14.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:14 smithi183 ceph-mon[33277]: 3.1c scrub starts 2024-09-21T12:35:14.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:14 smithi183 ceph-mon[33277]: 3.1c scrub ok 2024-09-21T12:35:14.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:14 smithi114 ceph-mon[30324]: 4.0 scrub starts 2024-09-21T12:35:14.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:14 smithi114 ceph-mon[30324]: 4.0 scrub ok 2024-09-21T12:35:14.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:14 smithi114 ceph-mon[30324]: 3.1c scrub starts 2024-09-21T12:35:14.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:14 smithi114 ceph-mon[30324]: 3.1c scrub ok 2024-09-21T12:35:14.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:14 smithi114 ceph-mon[40867]: 4.0 scrub starts 2024-09-21T12:35:14.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:14 smithi114 ceph-mon[40867]: 4.0 scrub ok 2024-09-21T12:35:14.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:14 smithi114 ceph-mon[40867]: 3.1c scrub starts 2024-09-21T12:35:14.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:14 smithi114 ceph-mon[40867]: 3.1c scrub ok 2024-09-21T12:35:14.781 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:14.812 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:14.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:14.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:14.914 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:14.944 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:14.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:14.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:15.046 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:15.080 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:15.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:15.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:15.181 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:15.220 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:15.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:15.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:15.285 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T12:35:15.317 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/osd.2/config 2024-09-21T12:35:15.322 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:15.349 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:15.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:15.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:15.357 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:15.450 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:15.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:15.497 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:15.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:15.498 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:15 smithi114 ceph-mon[30324]: 6.15 scrub starts 2024-09-21T12:35:15.498 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:15 smithi114 ceph-mon[30324]: 6.15 scrub ok 2024-09-21T12:35:15.498 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:15 smithi114 ceph-mon[30324]: pgmap v760: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.52k op/s; 14008/113457 objects degraded (12.347%) 2024-09-21T12:35:15.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:15 smithi114 ceph-mon[40867]: 6.15 scrub starts 2024-09-21T12:35:15.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:15 smithi114 ceph-mon[40867]: 6.15 scrub ok 2024-09-21T12:35:15.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:15 smithi114 ceph-mon[40867]: pgmap v760: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.52k op/s; 14008/113457 objects degraded (12.347%) 2024-09-21T12:35:15.598 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:15.636 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:15.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:15.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:15.738 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:15.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:15 smithi183 ceph-mon[33277]: 6.15 scrub starts 2024-09-21T12:35:15.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:15 smithi183 ceph-mon[33277]: 6.15 scrub ok 2024-09-21T12:35:15.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:15 smithi183 ceph-mon[33277]: pgmap v760: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.52k op/s; 14008/113457 objects degraded (12.347%) 2024-09-21T12:35:15.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:15.772 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:15.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:15.873 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:15.911 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:15.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:15.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:16.012 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:16.050 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:16.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:16.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:16.151 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:16.184 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:16.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:16.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:16.286 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:16.316 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:16.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:16.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:16.418 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:16.450 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:16.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:16.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:16.551 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:16.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:16.587 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:16.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:16.688 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:16.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:16.727 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:16.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:16.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:16 smithi183 ceph-mon[33277]: 4.10 scrub starts 2024-09-21T12:35:16.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:16 smithi183 ceph-mon[33277]: 4.10 scrub ok 2024-09-21T12:35:16.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:16 smithi183 ceph-mon[33277]: 10.4 scrub starts 2024-09-21T12:35:16.765 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:16 smithi114 ceph-mon[40867]: 4.10 scrub starts 2024-09-21T12:35:16.765 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:16 smithi114 ceph-mon[40867]: 4.10 scrub ok 2024-09-21T12:35:16.765 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:16 smithi114 ceph-mon[40867]: 10.4 scrub starts 2024-09-21T12:35:16.765 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:16 smithi114 ceph-mon[30324]: 4.10 scrub starts 2024-09-21T12:35:16.765 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:16 smithi114 ceph-mon[30324]: 4.10 scrub ok 2024-09-21T12:35:16.765 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:16 smithi114 ceph-mon[30324]: 10.4 scrub starts 2024-09-21T12:35:16.828 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:16.862 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:16.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:16.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:16.964 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:17.002 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:17.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:17.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:17.104 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:17.121 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:17.127 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:17.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:17.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:17.145 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:17.158 INFO:teuthology.orchestra.run.smithi114.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T12:35:17.229 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:17.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:17.270 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:17.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:17.370 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:17.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:17.409 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:17.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:17.506 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-21T12:35:17.510 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:17.545 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:17.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:17.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:17.647 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:17.656 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:17 smithi114 ceph-mon[40867]: pgmap v761: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 992 B/s rd, 0 B/s wr, 1.48k op/s; 14008/113457 objects degraded (12.347%) 2024-09-21T12:35:17.656 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:17 smithi114 ceph-mon[40867]: 10.4 scrub ok 2024-09-21T12:35:17.656 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:17 smithi114 ceph-mon[40867]: 1.17 scrub starts 2024-09-21T12:35:17.656 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:17 smithi114 ceph-mon[40867]: 1.17 scrub ok 2024-09-21T12:35:17.657 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:17 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/434386924' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:35:17.657 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:17 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:35:17.657 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:17 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 14008/113457 objects degraded (12.347%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T12:35:17.657 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:17 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:35:17.657 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:17 smithi114 ceph-mon[40867]: 1.11 scrub starts 2024-09-21T12:35:17.657 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:17 smithi114 ceph-mon[40867]: 1.11 scrub ok 2024-09-21T12:35:17.657 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:17 smithi114 ceph-mon[30324]: pgmap v761: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 992 B/s rd, 0 B/s wr, 1.48k op/s; 14008/113457 objects degraded (12.347%) 2024-09-21T12:35:17.657 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:17 smithi114 ceph-mon[30324]: 10.4 scrub ok 2024-09-21T12:35:17.657 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:17 smithi114 ceph-mon[30324]: 1.17 scrub starts 2024-09-21T12:35:17.657 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:17 smithi114 ceph-mon[30324]: 1.17 scrub ok 2024-09-21T12:35:17.658 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:17 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/434386924' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:35:17.658 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:17 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:35:17.658 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:17 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 14008/113457 objects degraded (12.347%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T12:35:17.658 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:17 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:35:17.658 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:17 smithi114 ceph-mon[30324]: 1.11 scrub starts 2024-09-21T12:35:17.658 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:17 smithi114 ceph-mon[30324]: 1.11 scrub ok 2024-09-21T12:35:17.680 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:17.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:17.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:17.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:35:17 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:35:17.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:17 smithi183 ceph-mon[33277]: pgmap v761: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 992 B/s rd, 0 B/s wr, 1.48k op/s; 14008/113457 objects degraded (12.347%) 2024-09-21T12:35:17.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:17 smithi183 ceph-mon[33277]: 10.4 scrub ok 2024-09-21T12:35:17.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:17 smithi183 ceph-mon[33277]: 1.17 scrub starts 2024-09-21T12:35:17.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:17 smithi183 ceph-mon[33277]: 1.17 scrub ok 2024-09-21T12:35:17.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:17 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/434386924' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:35:17.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:17 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:35:17.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:17 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 14008/113457 objects degraded (12.347%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T12:35:17.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:17 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:35:17.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:17 smithi183 ceph-mon[33277]: 1.11 scrub starts 2024-09-21T12:35:17.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:17 smithi183 ceph-mon[33277]: 1.11 scrub ok 2024-09-21T12:35:17.782 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:17.815 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:17.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:17.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:17.916 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:17.954 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:17.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:17.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:18.056 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:18.087 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:18.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:18.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:18.133 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:35:18.133 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:35:18.164 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:35:18.165 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:35:18.169 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T12:35:18.189 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:18.200 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:35:18.200 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:35:18.222 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:18.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:18.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:18.239 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:35:18.239 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:35:18.269 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:35:18.270 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:35:18.302 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:35:18.302 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:35:18.324 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:18.330 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:35:18.331 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:35:18.363 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:18.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:18.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:18.465 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:18.507 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:18.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:18.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:18.609 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:18.642 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:18.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:18.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:18.744 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:18.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:18 smithi114 ceph-mon[40867]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:35:18.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:18 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:35:18.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:18 smithi114 ceph-mon[40867]: osdmap e398: 8 total, 7 up, 8 in 2024-09-21T12:35:18.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:18 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:35:18.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:18 smithi114 ceph-mon[40867]: 3.a scrub starts 2024-09-21T12:35:18.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:18 smithi114 ceph-mon[40867]: 3.a scrub ok 2024-09-21T12:35:18.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:18 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3652165386' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:35:18.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:18 smithi114 ceph-mon[30324]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:35:18.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:18 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:35:18.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:18 smithi114 ceph-mon[30324]: osdmap e398: 8 total, 7 up, 8 in 2024-09-21T12:35:18.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:18 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:35:18.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:18 smithi114 ceph-mon[30324]: 3.a scrub starts 2024-09-21T12:35:18.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:18 smithi114 ceph-mon[30324]: 3.a scrub ok 2024-09-21T12:35:18.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:18 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3652165386' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:35:18.778 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:18.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:18.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:18.880 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:18.912 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:18.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:18.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:18.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:18 smithi183 ceph-mon[33277]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:35:18.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:18 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:35:18.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:18 smithi183 ceph-mon[33277]: osdmap e398: 8 total, 7 up, 8 in 2024-09-21T12:35:18.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:18 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:35:18.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:18 smithi183 ceph-mon[33277]: 3.a scrub starts 2024-09-21T12:35:18.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:18 smithi183 ceph-mon[33277]: 3.a scrub ok 2024-09-21T12:35:18.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:18 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3652165386' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:35:19.014 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:19.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:19.050 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:19.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:19.124 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:35:19.124 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-21T12:35:19.124 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:19.124 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.15924.0:227019 10.b 10:d0ee2899:::benchmark_data_smithi114_161690_object99967:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T12:35:19.124 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:35:19.106043+0000", 2024-09-21T12:35:19.125 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0096245299999999992, 2024-09-21T12:35:19.125 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0096547149999999995, 2024-09-21T12:35:19.125 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:35:19.125 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:35:19.125 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:35:19.125 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:35:19.125 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.15924", 2024-09-21T12:35:19.125 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:35:19.125 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 227019 2024-09-21T12:35:19.125 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:19.125 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:35:19.125 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:19.125 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:35:19.125 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:19.106043+0000", 2024-09-21T12:35:19.126 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:19.126 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:19.126 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:19.126 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:35:19.126 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:19.106043+0000", 2024-09-21T12:35:19.126 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:19.126 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:19.126 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:19.126 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:35:19.126 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:19.106044+0000", 2024-09-21T12:35:19.126 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.001e-06 2024-09-21T12:35:19.126 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:19.126 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:19.126 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:35:19.127 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:19.106048+0000", 2024-09-21T12:35:19.127 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4989999999999996e-06 2024-09-21T12:35:19.127 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:19.127 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:19.127 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:35:19.127 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:19.106060+0000", 2024-09-21T12:35:19.127 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1593999999999999e-05 2024-09-21T12:35:19.127 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:19.127 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:19.127 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:35:19.127 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:19.106072+0000", 2024-09-21T12:35:19.127 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1602e-05 2024-09-21T12:35:19.127 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:19.127 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:19.127 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:19.127 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:19.106125+0000", 2024-09-21T12:35:19.128 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3189999999999997e-05 2024-09-21T12:35:19.128 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:19.128 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:19.128 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:35:19.128 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:19.106238+0000", 2024-09-21T12:35:19.128 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000112737 2024-09-21T12:35:19.128 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:19.128 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:19.128 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,6", 2024-09-21T12:35:19.128 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:19.106360+0000", 2024-09-21T12:35:19.128 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000122137 2024-09-21T12:35:19.128 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:19.128 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:19.128 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:35:19.128 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:19.109172+0000", 2024-09-21T12:35:19.128 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0028123750000000002 2024-09-21T12:35:19.129 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:19.129 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:19.129 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:19.129 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:19.109598+0000", 2024-09-21T12:35:19.129 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00042574300000000002 2024-09-21T12:35:19.129 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:19.129 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:19.129 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:19.129 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:19.129 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:19.129 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.15924.0:227029 10.b 10:d5d6fa57:::benchmark_data_smithi114_161690_object99977:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T12:35:19.129 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:35:19.113403+0000", 2024-09-21T12:35:19.129 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.00226451, 2024-09-21T12:35:19.129 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002370413, 2024-09-21T12:35:19.129 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:35:19.130 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:35:19.130 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:35:19.130 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:35:19.130 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.15924", 2024-09-21T12:35:19.130 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:35:19.130 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 227029 2024-09-21T12:35:19.130 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:19.130 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:35:19.130 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:19.130 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:35:19.130 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:19.113403+0000", 2024-09-21T12:35:19.130 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:19.130 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:19.130 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:19.130 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:35:19.131 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:19.113403+0000", 2024-09-21T12:35:19.131 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:19.131 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:19.131 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:19.131 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:35:19.131 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:19.113405+0000", 2024-09-21T12:35:19.131 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6130000000000001e-06 2024-09-21T12:35:19.131 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:19.131 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:19.131 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:35:19.131 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:19.113411+0000", 2024-09-21T12:35:19.131 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9059999999999999e-06 2024-09-21T12:35:19.131 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:19.131 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:19.131 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:35:19.132 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:19.113426+0000", 2024-09-21T12:35:19.132 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5065999999999999e-05 2024-09-21T12:35:19.132 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:19.132 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:19.132 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:35:19.132 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:19.113433+0000", 2024-09-21T12:35:19.132 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3479999999999998e-06 2024-09-21T12:35:19.132 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:19.132 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:19.132 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:19.132 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:19.113494+0000", 2024-09-21T12:35:19.132 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1378999999999996e-05 2024-09-21T12:35:19.132 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:19.132 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:19.132 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:35:19.132 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:19.113590+0000", 2024-09-21T12:35:19.133 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.5198000000000006e-05 2024-09-21T12:35:19.133 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:19.133 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:19.133 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,6", 2024-09-21T12:35:19.133 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:19.113693+0000", 2024-09-21T12:35:19.133 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010390700000000001 2024-09-21T12:35:19.133 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:19.133 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:19.133 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:19.133 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:19.133 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:35:19.133 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 2 2024-09-21T12:35:19.133 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:35:19.151 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:19.185 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:19.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:19.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:19.286 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:19.324 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:19.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:19.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:19.426 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:19.460 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:19.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:19.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:19.562 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:19.599 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:19.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:19.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:19.601 INFO:teuthology.orchestra.run.smithi114.stdout:get_pg_num_history pg_num_history pg_num_history(e387 pg_nums {1={216=31},2={57=3},3={58=32},4={60=32},5={62=32},6={64=32},9={281=26},10={344=16}} deleted_pools 338,9) 2024-09-21T12:35:19.601 INFO:teuthology.orchestra.run.smithi114.stdout:Importing pgid 4.c 2024-09-21T12:35:19.601 INFO:teuthology.orchestra.run.smithi114.stdout:Write #4:31288c1b:::obj_delete_at_hint.0000000096:head# 2024-09-21T12:35:19.601 INFO:teuthology.orchestra.run.smithi114.stdout:snapset 0=[]:{} 2024-09-21T12:35:19.601 INFO:teuthology.orchestra.run.smithi114.stdout:Write #4:330b1641:::obj_delete_at_hint.0000000087:head# 2024-09-21T12:35:19.602 INFO:teuthology.orchestra.run.smithi114.stdout:snapset 0=[]:{} 2024-09-21T12:35:19.602 INFO:teuthology.orchestra.run.smithi114.stdout:Write #4:338eb7e9:::obj_delete_at_hint.0000000008:head# 2024-09-21T12:35:19.606 INFO:teuthology.orchestra.run.smithi114.stdout:snapset 0=[]:{} 2024-09-21T12:35:19.606 INFO:teuthology.orchestra.run.smithi114.stdout:Write #4:3677c228:::obj_delete_at_hint.0000000006:head# 2024-09-21T12:35:19.606 INFO:teuthology.orchestra.run.smithi114.stdout:snapset 0=[]:{} 2024-09-21T12:35:19.607 INFO:teuthology.orchestra.run.smithi114.stdout:Write #4:36abfd60:::obj_delete_at_hint.0000000029:head# 2024-09-21T12:35:19.607 INFO:teuthology.orchestra.run.smithi114.stdout:snapset 0=[]:{} 2024-09-21T12:35:19.607 INFO:teuthology.orchestra.run.smithi114.stdout:write_pg epoch 350 info 4.c( v 276'20 (0'0,276'20] local-lis/les=349/350 n=5 ec=60/60 lis/c=349/349 les/c/f=350/350/0 sis=349) 2024-09-21T12:35:19.609 INFO:teuthology.orchestra.run.smithi114.stdout:Import successful 2024-09-21T12:35:19.700 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:19.739 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:19.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:19.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:19.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:19 smithi114 ceph-mon[40867]: 4.7 scrub starts 2024-09-21T12:35:19.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:19 smithi114 ceph-mon[40867]: 4.7 scrub ok 2024-09-21T12:35:19.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:19 smithi114 ceph-mon[40867]: pgmap v763: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.52k op/s; 11073/89862 objects degraded (12.322%) 2024-09-21T12:35:19.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:19 smithi114 ceph-mon[30324]: 4.7 scrub starts 2024-09-21T12:35:19.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:19 smithi114 ceph-mon[30324]: 4.7 scrub ok 2024-09-21T12:35:19.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:19 smithi114 ceph-mon[30324]: pgmap v763: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.52k op/s; 11073/89862 objects degraded (12.322%) 2024-09-21T12:35:19.841 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:19.877 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:19.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:19.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:19.941 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T12:35:19.941 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T12:35:19.979 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:19.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:19 smithi183 ceph-mon[33277]: 4.7 scrub starts 2024-09-21T12:35:19.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:19 smithi183 ceph-mon[33277]: 4.7 scrub ok 2024-09-21T12:35:19.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:19 smithi183 ceph-mon[33277]: pgmap v763: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.52k op/s; 11073/89862 objects degraded (12.322%) 2024-09-21T12:35:20.014 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:20.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:20.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:20.117 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:20.154 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:20.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:20.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:20.255 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:20.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:20.294 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:20.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:20.395 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:20.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:20.429 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:20.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:20.530 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:20.559 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:20.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:20.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:20.661 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:20.698 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:20.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:20.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:20.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:20 smithi114 ceph-mon[40867]: 3.e scrub starts 2024-09-21T12:35:20.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:20 smithi114 ceph-mon[40867]: 3.e scrub ok 2024-09-21T12:35:20.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:20 smithi114 ceph-mon[30324]: 3.e scrub starts 2024-09-21T12:35:20.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:20 smithi114 ceph-mon[30324]: 3.e scrub ok 2024-09-21T12:35:20.800 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:20.835 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T12:35:20.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:20.840 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:20.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:20.941 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:20.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:20 smithi183 ceph-mon[33277]: 3.e scrub starts 2024-09-21T12:35:20.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:20 smithi183 ceph-mon[33277]: 3.e scrub ok 2024-09-21T12:35:20.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:20.989 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:20.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:21.090 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:21.124 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:21.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:21.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:21.226 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:21.263 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:21.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:21.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:21.300 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/exp.4.c.2 2024-09-21T12:35:21.365 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:21.417 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:21.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:21.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:21.519 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:21.559 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:21.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:21.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:21.661 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:21.692 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:21.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:21.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:21.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:21 smithi114 ceph-mon[30324]: 10.b scrub starts 2024-09-21T12:35:21.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:21 smithi114 ceph-mon[30324]: 10.b scrub ok 2024-09-21T12:35:21.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:21 smithi114 ceph-mon[30324]: pgmap v764: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.52k op/s; 11073/89862 objects degraded (12.322%) 2024-09-21T12:35:21.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:21 smithi114 ceph-mon[40867]: 10.b scrub starts 2024-09-21T12:35:21.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:21 smithi114 ceph-mon[40867]: 10.b scrub ok 2024-09-21T12:35:21.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:21 smithi114 ceph-mon[40867]: pgmap v764: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.52k op/s; 11073/89862 objects degraded (12.322%) 2024-09-21T12:35:21.794 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:21.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:21.832 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:21.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:21.918 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:21.933 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:21.942 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:35:21.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:21.969 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:21.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:21.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:21 smithi183 ceph-mon[33277]: 10.b scrub starts 2024-09-21T12:35:21.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:21 smithi183 ceph-mon[33277]: 10.b scrub ok 2024-09-21T12:35:21.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:21 smithi183 ceph-mon[33277]: pgmap v764: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.52k op/s; 11073/89862 objects degraded (12.322%) 2024-09-21T12:35:22.070 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:22.109 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:22.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:22.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:22.211 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:22.246 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:22.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:22.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:22.348 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:22.386 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:22.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:22.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:22.487 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:22.521 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:22.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:22.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:22.554 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:35:22.555 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:35:22.555 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:35:22.555 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:35:22.555 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:35:22.622 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:22.654 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:22.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:22.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:22.756 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:22.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:22 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 11073/89862 objects degraded (12.322%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T12:35:22.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:22 smithi114 ceph-mon[30324]: 3.12 scrub starts 2024-09-21T12:35:22.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:22 smithi114 ceph-mon[30324]: 3.12 scrub ok 2024-09-21T12:35:22.775 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:35:22 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:35:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:35:22.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:22 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 11073/89862 objects degraded (12.322%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T12:35:22.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:22 smithi114 ceph-mon[40867]: 3.12 scrub starts 2024-09-21T12:35:22.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:22 smithi114 ceph-mon[40867]: 3.12 scrub ok 2024-09-21T12:35:22.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:22.790 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:22.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:22.891 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:22.920 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:22.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:22.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:22.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:22 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 11073/89862 objects degraded (12.322%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T12:35:22.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:22 smithi183 ceph-mon[33277]: 3.12 scrub starts 2024-09-21T12:35:22.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:22 smithi183 ceph-mon[33277]: 3.12 scrub ok 2024-09-21T12:35:23.023 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:23.056 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:23.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:23.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:23.157 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:23.187 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:23.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:23.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:23.228 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:23.251 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T12:35:23.289 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:23.330 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:23.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:23.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:23.432 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:23.466 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:23.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:23.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:23.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:23 smithi114 ceph-mon[30324]: pgmap v765: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.57k op/s; 9503/74724 objects degraded (12.717%) 2024-09-21T12:35:23.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:23 smithi114 ceph-mon[30324]: 3.7 scrub starts 2024-09-21T12:35:23.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:23 smithi114 ceph-mon[30324]: 3.7 scrub ok 2024-09-21T12:35:23.567 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:23.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:23.618 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:23.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:23.719 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:23.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:23.751 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:23.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:23.852 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:23.855 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:23 smithi114 ceph-mon[40867]: pgmap v765: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.57k op/s; 9503/74724 objects degraded (12.717%) 2024-09-21T12:35:23.855 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:23 smithi114 ceph-mon[40867]: 3.7 scrub starts 2024-09-21T12:35:23.855 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:23 smithi114 ceph-mon[40867]: 3.7 scrub ok 2024-09-21T12:35:23.887 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:23.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:23.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:23.897 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T12:35:23.913 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:35:23.913 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:35:23.939 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:23.947 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:35:23.947 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:35:23.954 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T12:35:23.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:23 smithi183 ceph-mon[33277]: pgmap v765: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.57k op/s; 9503/74724 objects degraded (12.717%) 2024-09-21T12:35:23.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:23 smithi183 ceph-mon[33277]: 3.7 scrub starts 2024-09-21T12:35:23.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:23 smithi183 ceph-mon[33277]: 3.7 scrub ok 2024-09-21T12:35:23.989 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:35:23.989 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:35:23.989 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:24.025 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:24.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:24.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:24.032 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:35:24.032 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:35:24.070 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:35:24.070 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:35:24.109 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:35:24.109 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:35:24.127 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:24.148 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:35:24.148 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:35:24.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:24.178 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:24.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:24.279 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:24.309 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:24.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:24.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:24.410 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:24.448 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:24.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:24.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:24.550 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:24.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:24.583 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:24.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:24.684 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:24.715 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:35:24.715 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:35:24.719 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:24.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:24.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:24.821 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:24.858 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:24.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:24.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:24.960 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:24.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:24 smithi183 ceph-mon[33277]: 6.4 scrub starts 2024-09-21T12:35:24.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:24 smithi183 ceph-mon[33277]: 6.4 scrub ok 2024-09-21T12:35:24.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:24 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/521179874' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:35:24.990 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:24.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:24.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:25.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:24 smithi114 ceph-mon[40867]: 6.4 scrub starts 2024-09-21T12:35:25.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:24 smithi114 ceph-mon[40867]: 6.4 scrub ok 2024-09-21T12:35:25.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:24 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/521179874' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:35:25.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:24 smithi114 ceph-mon[30324]: 6.4 scrub starts 2024-09-21T12:35:25.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:24 smithi114 ceph-mon[30324]: 6.4 scrub ok 2024-09-21T12:35:25.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:24 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/521179874' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:35:25.092 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:25.127 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:25.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:25.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:25.228 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:25.260 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:25.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:25.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:25.361 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:25.389 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:25.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:25.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:25.490 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:25.525 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:25.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:25.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:25.627 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:25.659 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:25.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:25.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:25.761 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:25.785 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:25.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:25.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:25.886 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:25.927 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:25.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:25.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:25.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:25 smithi183 ceph-mon[33277]: pgmap v766: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 1.3 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.60k op/s; 7983/65466 objects degraded (12.194%) 2024-09-21T12:35:25.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:25 smithi183 ceph-mon[33277]: 1.0 scrub starts 2024-09-21T12:35:25.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:25 smithi183 ceph-mon[33277]: 1.0 scrub ok 2024-09-21T12:35:25.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:25 smithi183 ceph-mon[33277]: 6.c scrub starts 2024-09-21T12:35:25.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:25 smithi183 ceph-mon[33277]: 6.c scrub ok 2024-09-21T12:35:26.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:25 smithi114 ceph-mon[40867]: pgmap v766: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 1.3 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.60k op/s; 7983/65466 objects degraded (12.194%) 2024-09-21T12:35:26.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:25 smithi114 ceph-mon[40867]: 1.0 scrub starts 2024-09-21T12:35:26.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:25 smithi114 ceph-mon[40867]: 1.0 scrub ok 2024-09-21T12:35:26.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:25 smithi114 ceph-mon[40867]: 6.c scrub starts 2024-09-21T12:35:26.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:25 smithi114 ceph-mon[40867]: 6.c scrub ok 2024-09-21T12:35:26.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:25 smithi114 ceph-mon[30324]: pgmap v766: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 1.3 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.60k op/s; 7983/65466 objects degraded (12.194%) 2024-09-21T12:35:26.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:25 smithi114 ceph-mon[30324]: 1.0 scrub starts 2024-09-21T12:35:26.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:25 smithi114 ceph-mon[30324]: 1.0 scrub ok 2024-09-21T12:35:26.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:25 smithi114 ceph-mon[30324]: 6.c scrub starts 2024-09-21T12:35:26.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:25 smithi114 ceph-mon[30324]: 6.c scrub ok 2024-09-21T12:35:26.029 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:26.066 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:26.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:26.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:26.168 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:26.202 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:26.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:26.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:26.304 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:26.337 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:26.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:26.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:26.344 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-09-21T12:35:26.345 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:35:26.345 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-21T12:35:26.345 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:35:26.439 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:26.474 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:26.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:26.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:26.576 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:26.615 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:26.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:26.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:26.716 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:26.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:26.750 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:26.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:26.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:26 smithi114 ceph-mon[30324]: 6.d scrub starts 2024-09-21T12:35:26.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:26 smithi114 ceph-mon[30324]: 6.d scrub ok 2024-09-21T12:35:26.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:26 smithi114 ceph-mon[30324]: 6.16 scrub starts 2024-09-21T12:35:26.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:26 smithi114 ceph-mon[30324]: 6.16 scrub ok 2024-09-21T12:35:26.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:26 smithi114 ceph-mon[40867]: 6.d scrub starts 2024-09-21T12:35:26.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:26 smithi114 ceph-mon[40867]: 6.d scrub ok 2024-09-21T12:35:26.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:26 smithi114 ceph-mon[40867]: 6.16 scrub starts 2024-09-21T12:35:26.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:26 smithi114 ceph-mon[40867]: 6.16 scrub ok 2024-09-21T12:35:26.852 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:26.883 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:26.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:26.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:26.985 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:26.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:26 smithi183 ceph-mon[33277]: 6.d scrub starts 2024-09-21T12:35:26.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:26 smithi183 ceph-mon[33277]: 6.d scrub ok 2024-09-21T12:35:26.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:26 smithi183 ceph-mon[33277]: 6.16 scrub starts 2024-09-21T12:35:26.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:26 smithi183 ceph-mon[33277]: 6.16 scrub ok 2024-09-21T12:35:27.006 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:27.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:27.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:27.107 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:27.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:27.141 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:27.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:27.242 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:27.273 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:27.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:27.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:27.375 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:27.405 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:27.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:27.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:27.507 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:27.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:27.544 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:27.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:27.644 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:27.678 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:27.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:27.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:27.780 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:27.801 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:27.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:27.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:27.903 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:27.934 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:27.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:27.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:27.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:27 smithi114 ceph-mon[30324]: pgmap v767: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 1.3 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.60k op/s; 7983/65466 objects degraded (12.194%) 2024-09-21T12:35:27.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:27 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 7983/65466 objects degraded (12.194%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T12:35:27.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:27 smithi114 ceph-mon[30324]: 3.17 scrub starts 2024-09-21T12:35:27.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:27 smithi114 ceph-mon[30324]: 3.17 scrub ok 2024-09-21T12:35:27.969 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:27 smithi114 ceph-mon[40867]: pgmap v767: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 1.3 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.60k op/s; 7983/65466 objects degraded (12.194%) 2024-09-21T12:35:27.969 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:27 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 7983/65466 objects degraded (12.194%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T12:35:27.969 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:27 smithi114 ceph-mon[40867]: 3.17 scrub starts 2024-09-21T12:35:27.970 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:27 smithi114 ceph-mon[40867]: 3.17 scrub ok 2024-09-21T12:35:27.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:35:27 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:35:27.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:27 smithi183 ceph-mon[33277]: pgmap v767: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 1.3 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.60k op/s; 7983/65466 objects degraded (12.194%) 2024-09-21T12:35:27.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:27 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 7983/65466 objects degraded (12.194%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T12:35:27.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:27 smithi183 ceph-mon[33277]: 3.17 scrub starts 2024-09-21T12:35:27.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:27 smithi183 ceph-mon[33277]: 3.17 scrub ok 2024-09-21T12:35:28.036 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:28.067 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:28.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:28.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:28.169 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:28.206 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:28.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:28.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:28.308 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:28.338 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:28.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:28.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:28.440 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:28.461 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:28.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:28.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:28.562 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:28.592 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:28.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:28.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:28.694 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:28.726 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:28.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:28.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:28.828 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:28.860 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:28.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:28.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:28.962 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:28.984 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:28.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:28.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:28.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:28 smithi183 ceph-mon[33277]: 2.2 scrub starts 2024-09-21T12:35:28.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:28 smithi183 ceph-mon[33277]: 2.2 scrub ok 2024-09-21T12:35:28.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:28 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:35:28.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:28 smithi183 ceph-mon[33277]: 3.11 scrub starts 2024-09-21T12:35:28.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:28 smithi183 ceph-mon[33277]: 3.11 scrub ok 2024-09-21T12:35:29.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:28 smithi114 ceph-mon[40867]: 2.2 scrub starts 2024-09-21T12:35:29.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:28 smithi114 ceph-mon[40867]: 2.2 scrub ok 2024-09-21T12:35:29.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:28 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:35:29.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:28 smithi114 ceph-mon[40867]: 3.11 scrub starts 2024-09-21T12:35:29.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:28 smithi114 ceph-mon[40867]: 3.11 scrub ok 2024-09-21T12:35:29.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:28 smithi114 ceph-mon[30324]: 2.2 scrub starts 2024-09-21T12:35:29.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:28 smithi114 ceph-mon[30324]: 2.2 scrub ok 2024-09-21T12:35:29.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:28 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:35:29.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:28 smithi114 ceph-mon[30324]: 3.11 scrub starts 2024-09-21T12:35:29.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:28 smithi114 ceph-mon[30324]: 3.11 scrub ok 2024-09-21T12:35:29.086 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:29.110 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:29.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:29.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:29.211 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:29.239 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:29.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:29.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:29.341 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:29.361 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:29.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:29.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:29.463 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:29.495 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:29.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:29.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:29.597 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:29.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:29.634 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:29.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:29.735 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:29.776 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:29.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:29.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:29.803 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T12:35:29.875 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:29 smithi114 ceph-mon[40867]: pgmap v768: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 869 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 916 B/s rd, 0 B/s wr, 1.77k op/s; 5161/42282 objects degraded (12.206%) 2024-09-21T12:35:29.875 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:29 smithi114 ceph-mon[30324]: pgmap v768: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 869 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 916 B/s rd, 0 B/s wr, 1.77k op/s; 5161/42282 objects degraded (12.206%) 2024-09-21T12:35:29.878 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:29.912 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:29.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:29.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:29.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:29 smithi183 ceph-mon[33277]: pgmap v768: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 869 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 916 B/s rd, 0 B/s wr, 1.77k op/s; 5161/42282 objects degraded (12.206%) 2024-09-21T12:35:30.013 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:30.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:30.046 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:30.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:30.147 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:30.178 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:30.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:30.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:30.280 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:30.305 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:30.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:30.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:30.314 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T12:35:30.315 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T12:35:30.335 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:30.340 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:30.365 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:30.407 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:30.439 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:30.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:30.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:30.540 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:30.580 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:30.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:30.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:30.682 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:30.718 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:30.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:30.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:30.820 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:30.856 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:30.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:30.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:30.894 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:30 smithi114 ceph-mon[40867]: 3.15 scrub starts 2024-09-21T12:35:30.894 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:30 smithi114 ceph-mon[40867]: 3.15 scrub ok 2024-09-21T12:35:30.894 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:30 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:35:30.894 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:30 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:35:30.894 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:30 smithi114 ceph-mon[30324]: 3.15 scrub starts 2024-09-21T12:35:30.894 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:30 smithi114 ceph-mon[30324]: 3.15 scrub ok 2024-09-21T12:35:30.894 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:30 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:35:30.894 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:30 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:35:30.960 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:30.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:30 smithi183 ceph-mon[33277]: 3.15 scrub starts 2024-09-21T12:35:30.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:30 smithi183 ceph-mon[33277]: 3.15 scrub ok 2024-09-21T12:35:30.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:30 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:35:30.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:30 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:35:30.993 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:30.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:30.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:31.094 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:31.130 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:31.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:31.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:31.232 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:31.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:31.270 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:31.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:31.371 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:31.406 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:31.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:31.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:31.508 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:31.545 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:31.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:31.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:31.647 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:31.657 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:35:31.657 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:35:31.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:31.675 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:31.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:31.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:31 smithi114 ceph-mon[40867]: osdmap e399: 8 total, 7 up, 8 in 2024-09-21T12:35:31.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:31 smithi114 ceph-mon[40867]: pgmap v770: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 869 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.59k op/s; 5161/42282 objects degraded (12.206%) 2024-09-21T12:35:31.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:31 smithi114 ceph-mon[40867]: 3.18 scrub starts 2024-09-21T12:35:31.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:31 smithi114 ceph-mon[40867]: 3.18 scrub ok 2024-09-21T12:35:31.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:31 smithi114 ceph-mon[30324]: osdmap e399: 8 total, 7 up, 8 in 2024-09-21T12:35:31.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:31 smithi114 ceph-mon[30324]: pgmap v770: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 869 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.59k op/s; 5161/42282 objects degraded (12.206%) 2024-09-21T12:35:31.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:31 smithi114 ceph-mon[30324]: 3.18 scrub starts 2024-09-21T12:35:31.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:31 smithi114 ceph-mon[30324]: 3.18 scrub ok 2024-09-21T12:35:31.775 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:31.806 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:31.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:31.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:31.908 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:31.940 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:31.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:31.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:31.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:31 smithi183 ceph-mon[33277]: osdmap e399: 8 total, 7 up, 8 in 2024-09-21T12:35:31.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:31 smithi183 ceph-mon[33277]: pgmap v770: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 869 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.59k op/s; 5161/42282 objects degraded (12.206%) 2024-09-21T12:35:31.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:31 smithi183 ceph-mon[33277]: 3.18 scrub starts 2024-09-21T12:35:31.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:31 smithi183 ceph-mon[33277]: 3.18 scrub ok 2024-09-21T12:35:32.043 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:32.076 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:32.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:32.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:32.178 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:32.220 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:32.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:32.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:32.239 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:35:32.244 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:35:32.316 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:35:32.322 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:32.396 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:32.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:32.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:32.498 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:32.524 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:32.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:32.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:32.592 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-21T12:35:32.625 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:32.658 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:32.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:32.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:32.760 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:32.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:32.791 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:32.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:32.892 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:32.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:32.935 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:32.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:32.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:32 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/239153127' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:35:32.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:32 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 5161/42282 objects degraded (12.206%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T12:35:32.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:32 smithi183 ceph-mon[33277]: from='client.26072 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:35:32.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:32 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:35:32.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:32 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:35:33.024 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:35:32 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:35:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:35:33.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:32 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/239153127' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:35:33.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:32 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 5161/42282 objects degraded (12.206%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T12:35:33.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:32 smithi114 ceph-mon[30324]: from='client.26072 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:35:33.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:32 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:35:33.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:32 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:35:33.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:32 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/239153127' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:35:33.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:32 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 5161/42282 objects degraded (12.206%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T12:35:33.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:32 smithi114 ceph-mon[40867]: from='client.26072 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:35:33.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:32 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:35:33.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:32 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:35:33.036 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:33.070 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:33.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:33.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:33.172 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:33.204 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:33.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:33.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:33.306 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:33.339 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:33.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:33.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:33.441 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:33.467 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:33.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:33.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:33.569 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:33.604 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:33.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:33.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:33.706 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:33.728 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:33.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:33.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:33.829 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:33.867 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:33.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:33.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:33.968 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:33.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:33 smithi183 ceph-mon[33277]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:35:33.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:33 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/239153127' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:35:33.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:33 smithi183 ceph-mon[33277]: osdmap e400: 8 total, 7 up, 8 in 2024-09-21T12:35:33.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:33 smithi183 ceph-mon[33277]: pgmap v772: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 639 MiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s; 3822/31245 objects degraded (12.232%) 2024-09-21T12:35:33.997 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:33.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:33.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:34.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:33 smithi114 ceph-mon[40867]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:35:34.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:33 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/239153127' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:35:34.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:33 smithi114 ceph-mon[40867]: osdmap e400: 8 total, 7 up, 8 in 2024-09-21T12:35:34.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:33 smithi114 ceph-mon[40867]: pgmap v772: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 639 MiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s; 3822/31245 objects degraded (12.232%) 2024-09-21T12:35:34.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:33 smithi114 ceph-mon[30324]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:35:34.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:33 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/239153127' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:35:34.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:33 smithi114 ceph-mon[30324]: osdmap e400: 8 total, 7 up, 8 in 2024-09-21T12:35:34.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:33 smithi114 ceph-mon[30324]: pgmap v772: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 639 MiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s; 3822/31245 objects degraded (12.232%) 2024-09-21T12:35:34.100 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:34.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:34.130 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:34.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:34.231 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:34.262 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:34.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:34.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:34.364 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:34.397 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:34.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:34.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:34.499 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:34.534 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:34.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:34.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:34.553 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T12:35:34.636 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:34.672 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:34.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:34.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:34.774 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:34.804 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:34.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:34.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:34.905 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:34.937 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:34.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:34.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:34.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:34 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:35:34.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:34 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:35:34.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:34 smithi183 ceph-mon[33277]: osdmap e401: 8 total, 7 up, 8 in 2024-09-21T12:35:35.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:34 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:35:35.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:34 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:35:35.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:34 smithi114 ceph-mon[30324]: osdmap e401: 8 total, 7 up, 8 in 2024-09-21T12:35:35.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:34 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:35:35.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:34 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:35:35.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:34 smithi114 ceph-mon[40867]: osdmap e401: 8 total, 7 up, 8 in 2024-09-21T12:35:35.039 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:35.076 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:35.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:35.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:35.178 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:35.213 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:35.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:35.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:35.315 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:35.344 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:35.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:35.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:35.446 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:35.485 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:35.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:35.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:35.587 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:35.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:35.625 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:35.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:35.726 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:35.757 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:35.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:35.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:35.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:35 smithi114 ceph-mon[30324]: pgmap v774: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 412 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.22k op/s; 2459/20367 objects degraded (12.073%) 2024-09-21T12:35:35.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:35 smithi114 ceph-mon[40867]: pgmap v774: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 412 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.22k op/s; 2459/20367 objects degraded (12.073%) 2024-09-21T12:35:35.859 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:35.895 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:35.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:35.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:35.982 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":770,"stamp":"2024-09-21T12:35:30.665895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":911153715,"num_objects":14094,"num_object_clones":0,"num_object_copies":42282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14094,"num_whiteouts":0,"num_read":2035,"num_read_kb":1787,"num_write":240927,"num_write_kb":8131182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1440,"num_bytes_recovered":67313860,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74396,"ondisk_log_size":74396,"up":482,"acting":482,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":534,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":9787900,"kb_used_data":4804348,"kb_used_omap":12,"kb_used_meta":4983411,"kb_avail":740173316,"statfs":{"total":767960285184,"available":757937475584,"internally_reserved":0,"allocated":4919652352,"data_stored":4917231022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":5103013424},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1039400960,"num_objects":-15860,"num_object_clones":0,"num_object_copies":-47580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-5912,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-15860,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":15860,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004076"},"pg_stats":[{"pgid":"1.1d","version":"76'79","reported_seq":419,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.503732+0000","last_change":"2024-09-21T12:32:38.427111+0000","last_active":"2024-09-21T12:35:17.503732+0000","last_peered":"2024-09-21T12:35:17.503732+0000","last_clean":"2024-09-21T12:35:17.503732+0000","last_became_active":"2024-09-21T12:32:38.426383+0000","last_became_peered":"2024-09-21T12:32:38.426383+0000","last_unstale":"2024-09-21T12:35:17.503732+0000","last_undegraded":"2024-09-21T12:35:17.503732+0000","last_fullsized":"2024-09-21T12:35:17.503732+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:32.060116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:32.060116+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:58.209751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":733,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.507433+0000","last_change":"2024-09-21T12:34:34.340741+0000","last_active":"2024-09-21T12:35:17.507433+0000","last_peered":"2024-09-21T12:35:17.507433+0000","last_clean":"2024-09-21T12:34:29.289357+0000","last_became_active":"2024-09-21T12:34:34.340741+0000","last_became_peered":"2024-09-21T12:34:34.340741+0000","last_unstale":"2024-09-21T12:35:17.507433+0000","last_undegraded":"2024-09-21T12:35:17.507433+0000","last_fullsized":"2024-09-21T12:34:34.332209+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:29.086830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:29.086830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:04.246658+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":119,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.498719+0000","last_change":"2024-09-21T12:32:38.443995+0000","last_active":"2024-09-21T12:35:17.498719+0000","last_peered":"2024-09-21T12:35:17.498719+0000","last_clean":"2024-09-21T12:35:17.498719+0000","last_became_active":"2024-09-21T12:32:38.419081+0000","last_became_peered":"2024-09-21T12:32:38.419081+0000","last_unstale":"2024-09-21T12:35:17.498719+0000","last_undegraded":"2024-09-21T12:35:17.498719+0000","last_fullsized":"2024-09-21T12:35:17.498719+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:29:08.671723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:29:08.671723+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:04.123552+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":710,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.502716+0000","last_change":"2024-09-21T12:29:10.672659+0000","last_active":"2024-09-21T12:35:17.502716+0000","last_peered":"2024-09-21T12:35:17.502716+0000","last_clean":"2024-09-21T12:35:17.502716+0000","last_became_active":"2024-09-21T12:20:19.314908+0000","last_became_peered":"2024-09-21T12:20:19.314908+0000","last_unstale":"2024-09-21T12:35:17.502716+0000","last_undegraded":"2024-09-21T12:35:17.502716+0000","last_fullsized":"2024-09-21T12:35:17.502716+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:10.672599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:29:10.672599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:54.252919+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":117,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.499634+0000","last_change":"2024-09-21T12:33:27.411258+0000","last_active":"2024-09-21T12:35:17.499634+0000","last_peered":"2024-09-21T12:35:17.499634+0000","last_clean":"2024-09-21T12:35:17.499634+0000","last_became_active":"2024-09-21T12:32:38.425588+0000","last_became_peered":"2024-09-21T12:32:38.425588+0000","last_unstale":"2024-09-21T12:35:17.499634+0000","last_undegraded":"2024-09-21T12:35:17.499634+0000","last_fullsized":"2024-09-21T12:35:17.499634+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:27.411142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:27.411142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"76'79","reported_seq":388,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.506067+0000","last_change":"2024-09-21T12:34:14.420616+0000","last_active":"2024-09-21T12:35:17.506067+0000","last_peered":"2024-09-21T12:35:17.506067+0000","last_clean":"2024-09-21T12:35:17.506067+0000","last_became_active":"2024-09-21T12:27:16.920545+0000","last_became_peered":"2024-09-21T12:27:16.920545+0000","last_unstale":"2024-09-21T12:35:17.506067+0000","last_undegraded":"2024-09-21T12:35:17.506067+0000","last_fullsized":"2024-09-21T12:35:17.506067+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:14.420534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:14.420534+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:40.003098+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":705,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.508991+0000","last_change":"2024-09-21T12:34:05.904633+0000","last_active":"2024-09-21T12:35:17.508991+0000","last_peered":"2024-09-21T12:35:17.508991+0000","last_clean":"2024-09-21T12:35:17.508991+0000","last_became_active":"2024-09-21T12:20:21.595301+0000","last_became_peered":"2024-09-21T12:20:21.595301+0000","last_unstale":"2024-09-21T12:35:17.508991+0000","last_undegraded":"2024-09-21T12:35:17.508991+0000","last_fullsized":"2024-09-21T12:35:17.508991+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:05.904553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:34:05.904553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:31.688471+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":730,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.512218+0000","last_change":"2024-09-21T12:34:34.346389+0000","last_active":"2024-09-21T12:35:17.512218+0000","last_peered":"2024-09-21T12:35:17.512218+0000","last_clean":"2024-09-21T12:34:29.294384+0000","last_became_active":"2024-09-21T12:34:34.346389+0000","last_became_peered":"2024-09-21T12:34:34.346389+0000","last_unstale":"2024-09-21T12:35:17.512218+0000","last_undegraded":"2024-09-21T12:35:17.512218+0000","last_fullsized":"2024-09-21T12:34:34.336359+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:06.698999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:06.698999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":768,"reported_epoch":398,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:35:17.510146+0000","last_change":"2024-09-21T12:34:34.337065+0000","last_active":"2024-09-21T12:35:17.510146+0000","last_peered":"2024-09-21T12:35:17.510146+0000","last_clean":"2024-09-21T12:34:29.291199+0000","last_became_active":"2024-09-21T12:34:34.337065+0000","last_became_peered":"2024-09-21T12:34:34.337065+0000","last_unstale":"2024-09-21T12:35:17.510146+0000","last_undegraded":"2024-09-21T12:34:34.335679+0000","last_fullsized":"2024-09-21T12:34:34.335428+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:31:48.294300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:48.294300+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:37.100442+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":109,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.498595+0000","last_change":"2024-09-21T12:32:38.418041+0000","last_active":"2024-09-21T12:35:17.498595+0000","last_peered":"2024-09-21T12:35:17.498595+0000","last_clean":"2024-09-21T12:35:17.498595+0000","last_became_active":"2024-09-21T12:32:38.417724+0000","last_became_peered":"2024-09-21T12:32:38.417724+0000","last_unstale":"2024-09-21T12:35:17.498595+0000","last_undegraded":"2024-09-21T12:35:17.498595+0000","last_fullsized":"2024-09-21T12:35:17.498595+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:39.179665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:39.179665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:31.224881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":117,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.500475+0000","last_change":"2024-09-21T12:33:32.373658+0000","last_active":"2024-09-21T12:35:17.500475+0000","last_peered":"2024-09-21T12:35:17.500475+0000","last_clean":"2024-09-21T12:35:17.500475+0000","last_became_active":"2024-09-21T12:32:38.428077+0000","last_became_peered":"2024-09-21T12:32:38.428077+0000","last_unstale":"2024-09-21T12:35:17.500475+0000","last_undegraded":"2024-09-21T12:35:17.500475+0000","last_fullsized":"2024-09-21T12:35:17.500475+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:32.372832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:32.372832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":127,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.499793+0000","last_change":"2024-09-21T12:33:03.312402+0000","last_active":"2024-09-21T12:35:17.499793+0000","last_peered":"2024-09-21T12:35:17.499793+0000","last_clean":"2024-09-21T12:35:17.499793+0000","last_became_active":"2024-09-21T12:32:38.426121+0000","last_became_peered":"2024-09-21T12:32:38.426121+0000","last_unstale":"2024-09-21T12:35:17.499793+0000","last_undegraded":"2024-09-21T12:35:17.499793+0000","last_fullsized":"2024-09-21T12:35:17.499793+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:33:03.312251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:03.312251+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":756,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.511004+0000","last_change":"2024-09-21T12:34:13.198443+0000","last_active":"2024-09-21T12:35:17.511004+0000","last_peered":"2024-09-21T12:35:17.511004+0000","last_clean":"2024-09-21T12:35:17.511004+0000","last_became_active":"2024-09-21T12:32:14.237803+0000","last_became_peered":"2024-09-21T12:32:14.237803+0000","last_unstale":"2024-09-21T12:35:17.511004+0000","last_undegraded":"2024-09-21T12:35:17.511004+0000","last_fullsized":"2024-09-21T12:35:17.511004+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:13.198376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:13.198376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:41.988938+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":716,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.505345+0000","last_change":"2024-09-21T12:34:16.147749+0000","last_active":"2024-09-21T12:35:17.505345+0000","last_peered":"2024-09-21T12:35:17.505345+0000","last_clean":"2024-09-21T12:35:17.505345+0000","last_became_active":"2024-09-21T12:20:21.309190+0000","last_became_peered":"2024-09-21T12:20:21.309190+0000","last_unstale":"2024-09-21T12:35:17.505345+0000","last_undegraded":"2024-09-21T12:35:17.505345+0000","last_fullsized":"2024-09-21T12:35:17.505345+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:16.147705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:34:16.147705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:36.506746+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"76'79","reported_seq":490,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.512081+0000","last_change":"2024-09-21T12:33:36.752640+0000","last_active":"2024-09-21T12:35:17.512081+0000","last_peered":"2024-09-21T12:35:17.512081+0000","last_clean":"2024-09-21T12:35:17.512081+0000","last_became_active":"2024-09-21T12:27:16.937711+0000","last_became_peered":"2024-09-21T12:27:16.937711+0000","last_unstale":"2024-09-21T12:35:17.512081+0000","last_undegraded":"2024-09-21T12:35:17.512081+0000","last_fullsized":"2024-09-21T12:35:17.512081+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:36.752562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:36.752562+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":117,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.500507+0000","last_change":"2024-09-21T12:35:14.244213+0000","last_active":"2024-09-21T12:35:17.500507+0000","last_peered":"2024-09-21T12:35:17.500507+0000","last_clean":"2024-09-21T12:35:17.500507+0000","last_became_active":"2024-09-21T12:32:38.426142+0000","last_became_peered":"2024-09-21T12:32:38.426142+0000","last_unstale":"2024-09-21T12:35:17.500507+0000","last_undegraded":"2024-09-21T12:35:17.500507+0000","last_fullsized":"2024-09-21T12:35:17.500507+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:14.243954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:14.243954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:40.353667+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":123,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.499391+0000","last_change":"2024-09-21T12:33:26.395059+0000","last_active":"2024-09-21T12:35:17.499391+0000","last_peered":"2024-09-21T12:35:17.499391+0000","last_clean":"2024-09-21T12:35:17.499391+0000","last_became_active":"2024-09-21T12:32:38.422912+0000","last_became_peered":"2024-09-21T12:32:38.422912+0000","last_unstale":"2024-09-21T12:35:17.499391+0000","last_undegraded":"2024-09-21T12:35:17.499391+0000","last_fullsized":"2024-09-21T12:35:17.499391+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'7","last_scrub_stamp":"2024-09-21T12:33:26.394982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:26.394982+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":746,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.503545+0000","last_change":"2024-09-21T12:35:09.558490+0000","last_active":"2024-09-21T12:35:17.503545+0000","last_peered":"2024-09-21T12:35:17.503545+0000","last_clean":"2024-09-21T12:35:17.503545+0000","last_became_active":"2024-09-21T12:32:38.417472+0000","last_became_peered":"2024-09-21T12:32:38.417472+0000","last_unstale":"2024-09-21T12:35:17.503545+0000","last_undegraded":"2024-09-21T12:35:17.503545+0000","last_fullsized":"2024-09-21T12:35:17.503545+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:09.558418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:35:09.558418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:27.798085+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":730,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.510671+0000","last_change":"2024-09-21T12:33:36.125346+0000","last_active":"2024-09-21T12:35:17.510671+0000","last_peered":"2024-09-21T12:35:17.510671+0000","last_clean":"2024-09-21T12:35:17.510671+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:35:17.510671+0000","last_undegraded":"2024-09-21T12:35:17.510671+0000","last_fullsized":"2024-09-21T12:35:17.510671+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:36.124095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:36.124095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:00.976265+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"76'79","reported_seq":552,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.510680+0000","last_change":"2024-09-21T12:34:48.959991+0000","last_active":"2024-09-21T12:35:17.510680+0000","last_peered":"2024-09-21T12:35:17.510680+0000","last_clean":"2024-09-21T12:35:17.510680+0000","last_became_active":"2024-09-21T12:27:16.957569+0000","last_became_peered":"2024-09-21T12:27:16.957569+0000","last_unstale":"2024-09-21T12:35:17.510680+0000","last_undegraded":"2024-09-21T12:35:17.510680+0000","last_fullsized":"2024-09-21T12:35:17.510680+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:48.959916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:48.959916+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:18.731109+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":117,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.500931+0000","last_change":"2024-09-21T12:34:05.193021+0000","last_active":"2024-09-21T12:35:17.500931+0000","last_peered":"2024-09-21T12:35:17.500931+0000","last_clean":"2024-09-21T12:35:17.500931+0000","last_became_active":"2024-09-21T12:32:38.428997+0000","last_became_peered":"2024-09-21T12:32:38.428997+0000","last_unstale":"2024-09-21T12:35:17.500931+0000","last_undegraded":"2024-09-21T12:35:17.500931+0000","last_fullsized":"2024-09-21T12:35:17.500931+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:05.192971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:34:05.192971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":778,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.508508+0000","last_change":"2024-09-21T12:32:56.016897+0000","last_active":"2024-09-21T12:35:17.508508+0000","last_peered":"2024-09-21T12:35:17.508508+0000","last_clean":"2024-09-21T12:35:17.508508+0000","last_became_active":"2024-09-21T12:32:38.410246+0000","last_became_peered":"2024-09-21T12:32:38.410246+0000","last_unstale":"2024-09-21T12:35:17.508508+0000","last_undegraded":"2024-09-21T12:35:17.508508+0000","last_fullsized":"2024-09-21T12:35:17.508508+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:56.016840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:56.016840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:54.118245+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":773,"reported_epoch":398,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:35:17.509926+0000","last_change":"2024-09-21T12:34:34.341728+0000","last_active":"2024-09-21T12:35:17.509926+0000","last_peered":"2024-09-21T12:35:17.509926+0000","last_clean":"2024-09-21T12:34:29.305282+0000","last_became_active":"2024-09-21T12:34:34.341728+0000","last_became_peered":"2024-09-21T12:34:34.341728+0000","last_unstale":"2024-09-21T12:35:17.509926+0000","last_undegraded":"2024-09-21T12:34:34.336150+0000","last_fullsized":"2024-09-21T12:34:34.336049+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:33:01.668584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:01.668584+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:41.110659+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":745,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.504244+0000","last_change":"2024-09-21T12:32:38.426714+0000","last_active":"2024-09-21T12:35:17.504244+0000","last_peered":"2024-09-21T12:35:17.504244+0000","last_clean":"2024-09-21T12:35:17.504244+0000","last_became_active":"2024-09-21T12:32:38.425958+0000","last_became_peered":"2024-09-21T12:32:38.425958+0000","last_unstale":"2024-09-21T12:35:17.504244+0000","last_undegraded":"2024-09-21T12:35:17.504244+0000","last_fullsized":"2024-09-21T12:35:17.504244+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:06.636729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:06.636729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:50.282882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"76'79","reported_seq":665,"reported_epoch":398,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:35:17.509438+0000","last_change":"2024-09-21T12:34:34.345604+0000","last_active":"2024-09-21T12:35:17.509438+0000","last_peered":"2024-09-21T12:35:17.509438+0000","last_clean":"2024-09-21T12:34:31.810778+0000","last_became_active":"2024-09-21T12:34:34.345604+0000","last_became_peered":"2024-09-21T12:34:34.345604+0000","last_unstale":"2024-09-21T12:35:17.509438+0000","last_undegraded":"2024-09-21T12:34:34.335173+0000","last_fullsized":"2024-09-21T12:34:34.335050+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:27.294674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:27.294674+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:49.044523+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":735,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.512487+0000","last_change":"2024-09-21T12:35:06.944278+0000","last_active":"2024-09-21T12:35:17.512487+0000","last_peered":"2024-09-21T12:35:17.512487+0000","last_clean":"2024-09-21T12:35:17.512487+0000","last_became_active":"2024-09-21T12:32:02.033276+0000","last_became_peered":"2024-09-21T12:32:02.033276+0000","last_unstale":"2024-09-21T12:35:17.512487+0000","last_undegraded":"2024-09-21T12:35:17.512487+0000","last_fullsized":"2024-09-21T12:35:17.512487+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:06.944209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:06.944209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:36.548698+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":113,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.499106+0000","last_change":"2024-09-21T12:34:34.349252+0000","last_active":"2024-09-21T12:35:17.499106+0000","last_peered":"2024-09-21T12:35:17.499106+0000","last_clean":"2024-09-21T12:34:29.278130+0000","last_became_active":"2024-09-21T12:34:34.349252+0000","last_became_peered":"2024-09-21T12:34:34.349252+0000","last_unstale":"2024-09-21T12:35:17.499106+0000","last_undegraded":"2024-09-21T12:35:17.499106+0000","last_fullsized":"2024-09-21T12:34:34.329566+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:45.725968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.725968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:02.111360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":821,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.514166+0000","last_change":"2024-09-21T12:32:38.437419+0000","last_active":"2024-09-21T12:35:17.514166+0000","last_peered":"2024-09-21T12:35:17.514166+0000","last_clean":"2024-09-21T12:35:17.514166+0000","last_became_active":"2024-09-21T12:32:38.409163+0000","last_became_peered":"2024-09-21T12:32:38.409163+0000","last_unstale":"2024-09-21T12:35:17.514166+0000","last_undegraded":"2024-09-21T12:35:17.514166+0000","last_fullsized":"2024-09-21T12:35:17.514166+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:28:30.619954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:30.619954+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:54.938711+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":121,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.499729+0000","last_change":"2024-09-21T12:34:34.349550+0000","last_active":"2024-09-21T12:35:17.499729+0000","last_peered":"2024-09-21T12:35:17.499729+0000","last_clean":"2024-09-21T12:34:29.281006+0000","last_became_active":"2024-09-21T12:34:34.349550+0000","last_became_peered":"2024-09-21T12:34:34.349550+0000","last_unstale":"2024-09-21T12:35:17.499729+0000","last_undegraded":"2024-09-21T12:35:17.499729+0000","last_fullsized":"2024-09-21T12:34:34.331038+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:53.270634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:32:53.270634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"76'79","reported_seq":468,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.510801+0000","last_change":"2024-09-21T12:34:46.125581+0000","last_active":"2024-09-21T12:35:17.510801+0000","last_peered":"2024-09-21T12:35:17.510801+0000","last_clean":"2024-09-21T12:35:17.510801+0000","last_became_active":"2024-09-21T12:32:38.414204+0000","last_became_peered":"2024-09-21T12:32:38.414204+0000","last_unstale":"2024-09-21T12:35:17.510801+0000","last_undegraded":"2024-09-21T12:35:17.510801+0000","last_fullsized":"2024-09-21T12:35:17.510801+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:46.125518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:46.125518+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:15.895131+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":703,"reported_epoch":398,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:35:17.505920+0000","last_change":"2024-09-21T12:34:34.339462+0000","last_active":"2024-09-21T12:35:17.505920+0000","last_peered":"2024-09-21T12:35:17.505920+0000","last_clean":"2024-09-21T12:34:29.282696+0000","last_became_active":"2024-09-21T12:34:34.339462+0000","last_became_peered":"2024-09-21T12:34:34.339462+0000","last_unstale":"2024-09-21T12:35:17.505920+0000","last_undegraded":"2024-09-21T12:34:34.334227+0000","last_fullsized":"2024-09-21T12:34:34.334120+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:33:37.344821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:37.344821+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":771,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.502843+0000","last_change":"2024-09-21T12:34:42.279770+0000","last_active":"2024-09-21T12:35:17.502843+0000","last_peered":"2024-09-21T12:35:17.502843+0000","last_clean":"2024-09-21T12:35:17.502843+0000","last_became_active":"2024-09-21T12:32:38.425335+0000","last_became_peered":"2024-09-21T12:32:38.425335+0000","last_unstale":"2024-09-21T12:35:17.502843+0000","last_undegraded":"2024-09-21T12:35:17.502843+0000","last_fullsized":"2024-09-21T12:35:17.502843+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:34:42.279704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:34:42.279704+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:06.847042+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":809,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.507376+0000","last_change":"2024-09-21T12:32:57.082602+0000","last_active":"2024-09-21T12:35:17.507376+0000","last_peered":"2024-09-21T12:35:17.507376+0000","last_clean":"2024-09-21T12:35:17.507376+0000","last_became_active":"2024-09-21T12:32:14.238429+0000","last_became_peered":"2024-09-21T12:32:14.238429+0000","last_unstale":"2024-09-21T12:35:17.507376+0000","last_undegraded":"2024-09-21T12:35:17.507376+0000","last_fullsized":"2024-09-21T12:35:17.507376+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:32:57.082525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:57.082525+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:01.231017+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"352'15","reported_seq":155,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:20.541011+0000","last_change":"2024-09-21T12:35:11.182400+0000","last_active":"2024-09-21T12:35:20.541011+0000","last_peered":"2024-09-21T12:35:20.541011+0000","last_clean":"2024-09-21T12:35:20.541011+0000","last_became_active":"2024-09-21T12:32:38.432386+0000","last_became_peered":"2024-09-21T12:32:38.432386+0000","last_unstale":"2024-09-21T12:35:20.541011+0000","last_undegraded":"2024-09-21T12:35:20.541011+0000","last_fullsized":"2024-09-21T12:35:20.541011+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'15","last_scrub_stamp":"2024-09-21T12:35:11.182313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:35:11.182313+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:40.169044+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"398'14894","reported_seq":15384,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:26.824796+0000","last_change":"2024-09-21T12:34:09.186254+0000","last_active":"2024-09-21T12:35:26.824796+0000","last_peered":"2024-09-21T12:35:26.824796+0000","last_clean":"2024-09-21T12:35:26.824796+0000","last_became_active":"2024-09-21T12:32:41.394385+0000","last_became_peered":"2024-09-21T12:32:41.394385+0000","last_unstale":"2024-09-21T12:35:26.824796+0000","last_undegraded":"2024-09-21T12:35:26.824796+0000","last_fullsized":"2024-09-21T12:35:26.824796+0000","mapping_epoch":352,"log_start":"393'10900","ondisk_log_start":"393'10900","created":344,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8142","last_scrub_stamp":"2024-09-21T12:34:09.186087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:34:09.186087+0000","objects_scrubbed":7800,"log_size":3994,"log_dups_size":0,"ondisk_log_size":3994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:34.332388+0000","scrub_duration":1969,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63832064,"num_objects":974,"num_object_clones":0,"num_object_copies":2922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14894,"num_write_kb":507776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":8847360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"76'79","reported_seq":280,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.510254+0000","last_change":"2024-09-21T12:34:34.347159+0000","last_active":"2024-09-21T12:35:17.510254+0000","last_peered":"2024-09-21T12:35:17.510254+0000","last_clean":"2024-09-21T12:34:29.305642+0000","last_became_active":"2024-09-21T12:34:34.347159+0000","last_became_peered":"2024-09-21T12:34:34.347159+0000","last_unstale":"2024-09-21T12:35:17.510254+0000","last_undegraded":"2024-09-21T12:35:17.510254+0000","last_fullsized":"2024-09-21T12:34:34.340605+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:18.151644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:18.151644+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:39.813235+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":382,"reported_epoch":398,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:35:17.512646+0000","last_change":"2024-09-21T12:34:34.345532+0000","last_active":"2024-09-21T12:35:17.512646+0000","last_peered":"2024-09-21T12:35:17.512646+0000","last_clean":"2024-09-21T12:34:29.295616+0000","last_became_active":"2024-09-21T12:34:34.345532+0000","last_became_peered":"2024-09-21T12:34:34.345532+0000","last_unstale":"2024-09-21T12:35:17.512646+0000","last_undegraded":"2024-09-21T12:34:34.334307+0000","last_fullsized":"2024-09-21T12:34:34.334181+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:31:56.722241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:56.722241+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:39.150345+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":185,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.510788+0000","last_change":"2024-09-21T12:33:05.703536+0000","last_active":"2024-09-21T12:35:17.510788+0000","last_peered":"2024-09-21T12:35:17.510788+0000","last_clean":"2024-09-21T12:35:17.510788+0000","last_became_active":"2024-09-21T12:31:50.039390+0000","last_became_peered":"2024-09-21T12:31:50.039390+0000","last_unstale":"2024-09-21T12:35:17.510788+0000","last_undegraded":"2024-09-21T12:35:17.510788+0000","last_fullsized":"2024-09-21T12:35:17.510788+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:05.703464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:05.703464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"352'15","reported_seq":147,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:20.540555+0000","last_change":"2024-09-21T12:32:38.458099+0000","last_active":"2024-09-21T12:35:20.540555+0000","last_peered":"2024-09-21T12:35:20.540555+0000","last_clean":"2024-09-21T12:35:20.540555+0000","last_became_active":"2024-09-21T12:32:38.429092+0000","last_became_peered":"2024-09-21T12:32:38.429092+0000","last_unstale":"2024-09-21T12:35:20.540555+0000","last_undegraded":"2024-09-21T12:35:20.540555+0000","last_fullsized":"2024-09-21T12:35:20.540555+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'13","last_scrub_stamp":"2024-09-21T12:25:33.474391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:25:33.474391+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:50.125861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":213,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:24.989679+0000","last_change":"2024-09-21T12:35:24.989679+0000","last_active":"2024-09-21T12:35:24.989679+0000","last_peered":"2024-09-21T12:35:24.989679+0000","last_clean":"2024-09-21T12:35:24.989679+0000","last_became_active":"2024-09-21T12:31:39.054600+0000","last_became_peered":"2024-09-21T12:31:39.054600+0000","last_unstale":"2024-09-21T12:35:24.989679+0000","last_undegraded":"2024-09-21T12:35:24.989679+0000","last_fullsized":"2024-09-21T12:35:24.989679+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:35:24.989615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:24.989615+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:51.845745+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.e","version":"398'15124","reported_seq":15566,"reported_epoch":398,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:35:26.829926+0000","last_change":"2024-09-21T12:34:34.348181+0000","last_active":"2024-09-21T12:35:26.829926+0000","last_peered":"2024-09-21T12:35:26.829926+0000","last_clean":"2024-09-21T12:34:33.276141+0000","last_became_active":"2024-09-21T12:34:34.348181+0000","last_became_peered":"2024-09-21T12:34:34.348181+0000","last_unstale":"2024-09-21T12:35:26.829926+0000","last_undegraded":"2024-09-21T12:34:34.332718+0000","last_fullsized":"2024-09-21T12:34:34.332605+0000","mapping_epoch":388,"log_start":"394'11200","ondisk_log_start":"394'11200","created":344,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8308","last_scrub_stamp":"2024-09-21T12:34:11.235464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:34:11.235464+0000","objects_scrubbed":7770,"log_size":3924,"log_dups_size":0,"ondisk_log_size":3924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:33.585260+0000","scrub_duration":1980,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58851328,"num_objects":898,"num_object_clones":0,"num_object_copies":2694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":898,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15124,"num_write_kb":512704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":7405568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":898}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"76'79","reported_seq":121,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.500916+0000","last_change":"2024-09-21T12:34:34.348261+0000","last_active":"2024-09-21T12:35:17.500916+0000","last_peered":"2024-09-21T12:35:17.500916+0000","last_clean":"2024-09-21T12:34:29.279263+0000","last_became_active":"2024-09-21T12:34:34.348261+0000","last_became_peered":"2024-09-21T12:34:34.348261+0000","last_unstale":"2024-09-21T12:35:17.500916+0000","last_undegraded":"2024-09-21T12:35:17.500916+0000","last_fullsized":"2024-09-21T12:34:34.332145+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:00.343916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:00.343916+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":830,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.510944+0000","last_change":"2024-09-21T12:35:12.790887+0000","last_active":"2024-09-21T12:35:17.510944+0000","last_peered":"2024-09-21T12:35:17.510944+0000","last_clean":"2024-09-21T12:35:17.510944+0000","last_became_active":"2024-09-21T12:32:14.242826+0000","last_became_peered":"2024-09-21T12:32:14.242826+0000","last_unstale":"2024-09-21T12:35:17.510944+0000","last_undegraded":"2024-09-21T12:35:17.510944+0000","last_fullsized":"2024-09-21T12:35:17.510944+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T12:35:12.790809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:12.790809+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:24.019626+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":718,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:22.753104+0000","last_change":"2024-09-21T12:35:22.753036+0000","last_active":"2024-09-21T12:35:22.753104+0000","last_peered":"2024-09-21T12:35:22.753104+0000","last_clean":"2024-09-21T12:35:22.753104+0000","last_became_active":"2024-09-21T12:20:15.297340+0000","last_became_peered":"2024-09-21T12:20:15.297340+0000","last_unstale":"2024-09-21T12:35:22.753104+0000","last_undegraded":"2024-09-21T12:35:22.753104+0000","last_fullsized":"2024-09-21T12:35:22.753104+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:22.752960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:22.752960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:47.709039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":117,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.499508+0000","last_change":"2024-09-21T12:34:12.216251+0000","last_active":"2024-09-21T12:35:17.499508+0000","last_peered":"2024-09-21T12:35:17.499508+0000","last_clean":"2024-09-21T12:35:17.499508+0000","last_became_active":"2024-09-21T12:32:38.433417+0000","last_became_peered":"2024-09-21T12:32:38.433417+0000","last_unstale":"2024-09-21T12:35:17.499508+0000","last_undegraded":"2024-09-21T12:35:17.499508+0000","last_fullsized":"2024-09-21T12:35:17.499508+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:12.216104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:12.216104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:39.310092+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":113,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.499885+0000","last_change":"2024-09-21T12:34:34.347165+0000","last_active":"2024-09-21T12:35:17.499885+0000","last_peered":"2024-09-21T12:35:17.499885+0000","last_clean":"2024-09-21T12:34:29.278468+0000","last_became_active":"2024-09-21T12:34:34.347165+0000","last_became_peered":"2024-09-21T12:34:34.347165+0000","last_unstale":"2024-09-21T12:35:17.499885+0000","last_undegraded":"2024-09-21T12:35:17.499885+0000","last_fullsized":"2024-09-21T12:34:34.331768+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:05.143736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:29:05.143736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:07.200140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"398'14786","reported_seq":14975,"reported_epoch":398,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:35:26.854410+0000","last_change":"2024-09-21T12:34:34.341756+0000","last_active":"2024-09-21T12:35:26.854410+0000","last_peered":"2024-09-21T12:35:26.854410+0000","last_clean":"2024-09-21T12:34:33.322295+0000","last_became_active":"2024-09-21T12:34:34.341756+0000","last_became_peered":"2024-09-21T12:34:34.341756+0000","last_unstale":"2024-09-21T12:35:26.854410+0000","last_undegraded":"2024-09-21T12:34:34.339006+0000","last_fullsized":"2024-09-21T12:34:34.338896+0000","mapping_epoch":388,"log_start":"389'10000","ondisk_log_start":"389'10000","created":344,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'5675","last_scrub_stamp":"2024-09-21T12:33:36.827072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:33:36.827072+0000","objects_scrubbed":5605,"log_size":4786,"log_dups_size":0,"ondisk_log_size":4786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:49.009376+0000","scrub_duration":1900,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59506688,"num_objects":908,"num_object_clones":0,"num_object_copies":2724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":908,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14786,"num_write_kb":502208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":908}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"76'79","reported_seq":645,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.512590+0000","last_change":"2024-09-21T12:33:33.718690+0000","last_active":"2024-09-21T12:35:17.512590+0000","last_peered":"2024-09-21T12:35:17.512590+0000","last_clean":"2024-09-21T12:35:17.512590+0000","last_became_active":"2024-09-21T12:26:44.960093+0000","last_became_peered":"2024-09-21T12:26:44.960093+0000","last_unstale":"2024-09-21T12:35:17.512590+0000","last_undegraded":"2024-09-21T12:35:17.512590+0000","last_fullsized":"2024-09-21T12:35:17.512590+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:33.718619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:33.718619+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":125,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.498946+0000","last_change":"2024-09-21T12:32:38.457999+0000","last_active":"2024-09-21T12:35:17.498946+0000","last_peered":"2024-09-21T12:35:17.498946+0000","last_clean":"2024-09-21T12:35:17.498946+0000","last_became_active":"2024-09-21T12:32:38.425453+0000","last_became_peered":"2024-09-21T12:32:38.425453+0000","last_unstale":"2024-09-21T12:35:17.498946+0000","last_undegraded":"2024-09-21T12:35:17.498946+0000","last_fullsized":"2024-09-21T12:35:17.498946+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-21T12:26:51.463698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:51.463698+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:57.149094+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":40,"num_read_kb":33,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":807,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.507761+0000","last_change":"2024-09-21T12:34:48.162040+0000","last_active":"2024-09-21T12:35:17.507761+0000","last_peered":"2024-09-21T12:35:17.507761+0000","last_clean":"2024-09-21T12:35:17.507761+0000","last_became_active":"2024-09-21T12:32:38.424412+0000","last_became_peered":"2024-09-21T12:32:38.424412+0000","last_unstale":"2024-09-21T12:35:17.507761+0000","last_undegraded":"2024-09-21T12:35:17.507761+0000","last_fullsized":"2024-09-21T12:35:17.507761+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:34:48.161956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:34:48.161956+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:16.468764+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"352'12","reported_seq":953,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:20.540870+0000","last_change":"2024-09-21T12:34:07.827409+0000","last_active":"2024-09-21T12:35:20.540870+0000","last_peered":"2024-09-21T12:35:20.540870+0000","last_clean":"2024-09-21T12:35:20.540870+0000","last_became_active":"2024-09-21T12:32:38.409177+0000","last_became_peered":"2024-09-21T12:32:38.409177+0000","last_unstale":"2024-09-21T12:35:20.540870+0000","last_undegraded":"2024-09-21T12:35:20.540870+0000","last_fullsized":"2024-09-21T12:35:20.540870+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'12","last_scrub_stamp":"2024-09-21T12:34:07.827315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:07.827315+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":711,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.502721+0000","last_change":"2024-09-21T12:34:34.345290+0000","last_active":"2024-09-21T12:35:17.502721+0000","last_peered":"2024-09-21T12:35:17.502721+0000","last_clean":"2024-09-21T12:34:29.282268+0000","last_became_active":"2024-09-21T12:34:34.345290+0000","last_became_peered":"2024-09-21T12:34:34.345290+0000","last_unstale":"2024-09-21T12:35:17.502721+0000","last_undegraded":"2024-09-21T12:35:17.502721+0000","last_fullsized":"2024-09-21T12:34:34.331990+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:59.406201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:32:59.406201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"398'14971","reported_seq":15383,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:26.848256+0000","last_change":"2024-09-21T12:35:09.333805+0000","last_active":"2024-09-21T12:35:26.848256+0000","last_peered":"2024-09-21T12:35:26.848256+0000","last_clean":"2024-09-21T12:35:26.848256+0000","last_became_active":"2024-09-21T12:32:41.396828+0000","last_became_peered":"2024-09-21T12:32:41.396828+0000","last_unstale":"2024-09-21T12:35:26.848256+0000","last_undegraded":"2024-09-21T12:35:26.848256+0000","last_fullsized":"2024-09-21T12:35:26.848256+0000","mapping_epoch":352,"log_start":"391'10600","ondisk_log_start":"391'10600","created":344,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'13253","last_scrub_stamp":"2024-09-21T12:35:09.333331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:35:09.333331+0000","objects_scrubbed":2651,"log_size":4371,"log_dups_size":0,"ondisk_log_size":4371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:16.560386+0000","scrub_duration":804,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58654720,"num_objects":895,"num_object_clones":0,"num_object_copies":2685,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14971,"num_write_kb":507712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":8060928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"76'79","reported_seq":185,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.513258+0000","last_change":"2024-09-21T12:31:50.320881+0000","last_active":"2024-09-21T12:35:17.513258+0000","last_peered":"2024-09-21T12:35:17.513258+0000","last_clean":"2024-09-21T12:35:17.513258+0000","last_became_active":"2024-09-21T12:31:50.040087+0000","last_became_peered":"2024-09-21T12:31:50.040087+0000","last_unstale":"2024-09-21T12:35:17.513258+0000","last_undegraded":"2024-09-21T12:35:17.513258+0000","last_fullsized":"2024-09-21T12:35:17.513258+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:31:50.320834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:31:50.320834+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:03.952447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":259,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.502967+0000","last_change":"2024-09-21T12:34:17.380869+0000","last_active":"2024-09-21T12:35:17.502967+0000","last_peered":"2024-09-21T12:35:17.502967+0000","last_clean":"2024-09-21T12:35:17.502967+0000","last_became_active":"2024-09-21T12:32:38.424412+0000","last_became_peered":"2024-09-21T12:32:38.424412+0000","last_unstale":"2024-09-21T12:35:17.502967+0000","last_undegraded":"2024-09-21T12:35:17.502967+0000","last_fullsized":"2024-09-21T12:35:17.502967+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:34:17.380538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:34:17.380538+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:28.909301+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":739,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.510256+0000","last_change":"2024-09-21T12:34:34.339986+0000","last_active":"2024-09-21T12:35:17.510256+0000","last_peered":"2024-09-21T12:35:17.510256+0000","last_clean":"2024-09-21T12:34:29.305656+0000","last_became_active":"2024-09-21T12:34:34.339986+0000","last_became_peered":"2024-09-21T12:34:34.339986+0000","last_unstale":"2024-09-21T12:35:17.510256+0000","last_undegraded":"2024-09-21T12:35:17.510256+0000","last_fullsized":"2024-09-21T12:34:34.336347+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:19.629083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:19.629083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:36.225737+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":915,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:20.541526+0000","last_change":"2024-09-21T12:34:17.207726+0000","last_active":"2024-09-21T12:35:20.541526+0000","last_peered":"2024-09-21T12:35:20.541526+0000","last_clean":"2024-09-21T12:35:20.541526+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:35:20.541526+0000","last_undegraded":"2024-09-21T12:35:20.541526+0000","last_fullsized":"2024-09-21T12:35:20.541526+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:34:17.207117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:17.207117+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:31.149371+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":713,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.505211+0000","last_change":"2024-09-21T12:34:34.335260+0000","last_active":"2024-09-21T12:35:17.505211+0000","last_peered":"2024-09-21T12:35:17.505211+0000","last_clean":"2024-09-21T12:34:29.288868+0000","last_became_active":"2024-09-21T12:34:34.335260+0000","last_became_peered":"2024-09-21T12:34:34.335260+0000","last_unstale":"2024-09-21T12:35:17.505211+0000","last_undegraded":"2024-09-21T12:35:17.505211+0000","last_fullsized":"2024-09-21T12:34:34.331914+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:37.850100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:37.850100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:40.243972+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"398'15448","reported_seq":15880,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:26.830553+0000","last_change":"2024-09-21T12:34:03.588333+0000","last_active":"2024-09-21T12:35:26.830553+0000","last_peered":"2024-09-21T12:35:26.830553+0000","last_clean":"2024-09-21T12:35:26.830553+0000","last_became_active":"2024-09-21T12:32:42.035934+0000","last_became_peered":"2024-09-21T12:32:42.035934+0000","last_unstale":"2024-09-21T12:35:26.830553+0000","last_undegraded":"2024-09-21T12:35:26.830553+0000","last_fullsized":"2024-09-21T12:35:26.830553+0000","mapping_epoch":353,"log_start":"394'11500","ondisk_log_start":"394'11500","created":344,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'7988","last_scrub_stamp":"2024-09-21T12:34:03.588033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:34:03.588033+0000","objects_scrubbed":7906,"log_size":3948,"log_dups_size":0,"ondisk_log_size":3948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2385,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60620824,"num_objects":926,"num_object_clones":0,"num_object_copies":2778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":926,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15448,"num_write_kb":523905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"76'79","reported_seq":402,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.507037+0000","last_change":"2024-09-21T12:32:52.986242+0000","last_active":"2024-09-21T12:35:17.507037+0000","last_peered":"2024-09-21T12:35:17.507037+0000","last_clean":"2024-09-21T12:35:17.507037+0000","last_became_active":"2024-09-21T12:29:05.835884+0000","last_became_peered":"2024-09-21T12:29:05.835884+0000","last_unstale":"2024-09-21T12:35:17.507037+0000","last_undegraded":"2024-09-21T12:35:17.507037+0000","last_fullsized":"2024-09-21T12:35:17.507037+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:52.986175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:52.986175+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:55.111405+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":768,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.508924+0000","last_change":"2024-09-21T12:33:38.081629+0000","last_active":"2024-09-21T12:35:17.508924+0000","last_peered":"2024-09-21T12:35:17.508924+0000","last_clean":"2024-09-21T12:35:17.508924+0000","last_became_active":"2024-09-21T12:20:17.693067+0000","last_became_peered":"2024-09-21T12:20:17.693067+0000","last_unstale":"2024-09-21T12:35:17.508924+0000","last_undegraded":"2024-09-21T12:35:17.508924+0000","last_fullsized":"2024-09-21T12:35:17.508924+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'29","last_scrub_stamp":"2024-09-21T12:33:38.081452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:38.081452+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:03.222822+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":715,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.503989+0000","last_change":"2024-09-21T12:34:34.337276+0000","last_active":"2024-09-21T12:35:17.503989+0000","last_peered":"2024-09-21T12:35:17.503989+0000","last_clean":"2024-09-21T12:34:29.286433+0000","last_became_active":"2024-09-21T12:34:34.337276+0000","last_became_peered":"2024-09-21T12:34:34.337276+0000","last_unstale":"2024-09-21T12:35:17.503989+0000","last_undegraded":"2024-09-21T12:35:17.503989+0000","last_fullsized":"2024-09-21T12:34:34.330247+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:38.250806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:38.250806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:57.259384+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":299,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.506435+0000","last_change":"2024-09-21T12:29:05.826167+0000","last_active":"2024-09-21T12:35:17.506435+0000","last_peered":"2024-09-21T12:35:17.506435+0000","last_clean":"2024-09-21T12:35:17.506435+0000","last_became_active":"2024-09-21T12:29:05.825869+0000","last_became_peered":"2024-09-21T12:29:05.825869+0000","last_unstale":"2024-09-21T12:35:17.506435+0000","last_undegraded":"2024-09-21T12:35:17.506435+0000","last_fullsized":"2024-09-21T12:35:17.506435+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:48.999878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:48.999878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:58.170092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":711,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.502546+0000","last_change":"2024-09-21T12:34:34.339352+0000","last_active":"2024-09-21T12:35:17.502546+0000","last_peered":"2024-09-21T12:35:17.502546+0000","last_clean":"2024-09-21T12:34:29.284095+0000","last_became_active":"2024-09-21T12:34:34.339352+0000","last_became_peered":"2024-09-21T12:34:34.339352+0000","last_unstale":"2024-09-21T12:35:17.502546+0000","last_undegraded":"2024-09-21T12:35:17.502546+0000","last_fullsized":"2024-09-21T12:34:34.329336+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:05.510620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:05.510620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:53.299822+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"398'15141","reported_seq":15261,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:26.873996+0000","last_change":"2024-09-21T12:32:30.352578+0000","last_active":"2024-09-21T12:35:26.873996+0000","last_peered":"2024-09-21T12:35:26.873996+0000","last_clean":"2024-09-21T12:35:26.873996+0000","last_became_active":"2024-09-21T12:32:30.352289+0000","last_became_peered":"2024-09-21T12:32:30.352289+0000","last_unstale":"2024-09-21T12:35:26.873996+0000","last_undegraded":"2024-09-21T12:35:26.873996+0000","last_fullsized":"2024-09-21T12:35:26.873996+0000","mapping_epoch":344,"log_start":"390'10300","ondisk_log_start":"390'10300","created":344,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:32:29.279916+0000","objects_scrubbed":0,"log_size":4841,"log_dups_size":0,"ondisk_log_size":4841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:53.955623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60358656,"num_objects":921,"num_object_clones":0,"num_object_copies":2763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15141,"num_write_kb":513984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"76'79","reported_seq":515,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.506312+0000","last_change":"2024-09-21T12:34:34.337922+0000","last_active":"2024-09-21T12:35:17.506312+0000","last_peered":"2024-09-21T12:35:17.506312+0000","last_clean":"2024-09-21T12:34:29.289671+0000","last_became_active":"2024-09-21T12:34:34.337922+0000","last_became_peered":"2024-09-21T12:34:34.337922+0000","last_unstale":"2024-09-21T12:35:17.506312+0000","last_undegraded":"2024-09-21T12:35:17.506312+0000","last_fullsized":"2024-09-21T12:34:34.332211+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:34.055720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:34.055720+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:12.184786+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"389'41","reported_seq":961,"reported_epoch":398,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:35:20.540630+0000","last_change":"2024-09-21T12:34:34.339316+0000","last_active":"2024-09-21T12:35:20.540630+0000","last_peered":"2024-09-21T12:35:20.540630+0000","last_clean":"2024-09-21T12:34:30.529659+0000","last_became_active":"2024-09-21T12:34:34.339316+0000","last_became_peered":"2024-09-21T12:34:34.339316+0000","last_unstale":"2024-09-21T12:35:20.540630+0000","last_undegraded":"2024-09-21T12:34:34.330286+0000","last_fullsized":"2024-09-21T12:34:34.330177+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'40","last_scrub_stamp":"2024-09-21T12:32:25.139435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:25.139435+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:34.351460+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":117,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.499494+0000","last_change":"2024-09-21T12:35:09.182021+0000","last_active":"2024-09-21T12:35:17.499494+0000","last_peered":"2024-09-21T12:35:17.499494+0000","last_clean":"2024-09-21T12:35:17.499494+0000","last_became_active":"2024-09-21T12:32:38.426309+0000","last_became_peered":"2024-09-21T12:32:38.426309+0000","last_unstale":"2024-09-21T12:35:17.499494+0000","last_undegraded":"2024-09-21T12:35:17.499494+0000","last_fullsized":"2024-09-21T12:35:17.499494+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:09.181952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:09.181952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:15.372019+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":445,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:27.114524+0000","last_change":"2024-09-21T12:35:27.114524+0000","last_active":"2024-09-21T12:35:27.114524+0000","last_peered":"2024-09-21T12:35:27.114524+0000","last_clean":"2024-09-21T12:35:27.114524+0000","last_became_active":"2024-09-21T12:26:54.083584+0000","last_became_peered":"2024-09-21T12:26:54.083584+0000","last_unstale":"2024-09-21T12:35:27.114524+0000","last_undegraded":"2024-09-21T12:35:27.114524+0000","last_fullsized":"2024-09-21T12:35:27.114524+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T12:35:27.114442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:35:27.114442+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:35.507665+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":757,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.507434+0000","last_change":"2024-09-21T12:33:00.087826+0000","last_active":"2024-09-21T12:35:17.507434+0000","last_peered":"2024-09-21T12:35:17.507434+0000","last_clean":"2024-09-21T12:35:17.507434+0000","last_became_active":"2024-09-21T12:32:38.424402+0000","last_became_peered":"2024-09-21T12:32:38.424402+0000","last_unstale":"2024-09-21T12:35:17.507434+0000","last_undegraded":"2024-09-21T12:35:17.507434+0000","last_fullsized":"2024-09-21T12:35:17.507434+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:00.087746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:33:00.087746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:52.163251+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":741,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.511960+0000","last_change":"2024-09-21T12:33:59.899574+0000","last_active":"2024-09-21T12:35:17.511960+0000","last_peered":"2024-09-21T12:35:17.511960+0000","last_clean":"2024-09-21T12:35:17.511960+0000","last_became_active":"2024-09-21T12:32:38.409159+0000","last_became_peered":"2024-09-21T12:32:38.409159+0000","last_unstale":"2024-09-21T12:35:17.511960+0000","last_undegraded":"2024-09-21T12:35:17.511960+0000","last_fullsized":"2024-09-21T12:35:17.511960+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:59.899499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:59.899499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:28.715289+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"398'14653","reported_seq":14868,"reported_epoch":398,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:35:26.844567+0000","last_change":"2024-09-21T12:34:34.344990+0000","last_active":"2024-09-21T12:35:26.844567+0000","last_peered":"2024-09-21T12:35:26.844567+0000","last_clean":"2024-09-21T12:34:33.324393+0000","last_became_active":"2024-09-21T12:34:34.344990+0000","last_became_peered":"2024-09-21T12:34:34.344990+0000","last_unstale":"2024-09-21T12:35:26.844567+0000","last_undegraded":"2024-09-21T12:34:34.332975+0000","last_fullsized":"2024-09-21T12:34:34.332347+0000","mapping_epoch":388,"log_start":"391'10268","ondisk_log_start":"391'10268","created":344,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'7776","last_scrub_stamp":"2024-09-21T12:34:06.678624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:34:06.678624+0000","objects_scrubbed":7706,"log_size":4385,"log_dups_size":0,"ondisk_log_size":4385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2387,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58916864,"num_objects":899,"num_object_clones":0,"num_object_copies":2697,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":899,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14653,"num_write_kb":497664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":899}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"76'79","reported_seq":148,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.503288+0000","last_change":"2024-09-21T12:32:38.420376+0000","last_active":"2024-09-21T12:35:17.503288+0000","last_peered":"2024-09-21T12:35:17.503288+0000","last_clean":"2024-09-21T12:35:17.503288+0000","last_became_active":"2024-09-21T12:32:38.417915+0000","last_became_peered":"2024-09-21T12:32:38.417915+0000","last_unstale":"2024-09-21T12:35:17.503288+0000","last_undegraded":"2024-09-21T12:35:17.503288+0000","last_fullsized":"2024-09-21T12:35:17.503288+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:30:39.179014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:30:39.179014+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:02.473143+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":221,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:18.061854+0000","last_change":"2024-09-21T12:35:18.061854+0000","last_active":"2024-09-21T12:35:18.061854+0000","last_peered":"2024-09-21T12:35:18.061854+0000","last_clean":"2024-09-21T12:35:18.061854+0000","last_became_active":"2024-09-21T12:31:39.057316+0000","last_became_peered":"2024-09-21T12:31:39.057316+0000","last_unstale":"2024-09-21T12:35:18.061854+0000","last_undegraded":"2024-09-21T12:35:18.061854+0000","last_fullsized":"2024-09-21T12:35:18.061854+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'17","last_scrub_stamp":"2024-09-21T12:35:18.061753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:18.061753+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:20.057668+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":747,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.503072+0000","last_change":"2024-09-21T12:34:34.343641+0000","last_active":"2024-09-21T12:35:17.503072+0000","last_peered":"2024-09-21T12:35:17.503072+0000","last_clean":"2024-09-21T12:34:29.286935+0000","last_became_active":"2024-09-21T12:34:34.343641+0000","last_became_peered":"2024-09-21T12:34:34.343641+0000","last_unstale":"2024-09-21T12:35:17.503072+0000","last_undegraded":"2024-09-21T12:35:17.503072+0000","last_fullsized":"2024-09-21T12:34:34.329247+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:00.211563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:34:00.211563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:16.079303+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":55,"reported_epoch":398,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:35:17.509949+0000","last_change":"2024-09-21T12:34:34.338167+0000","last_active":"2024-09-21T12:35:17.509949+0000","last_peered":"2024-09-21T12:35:17.509949+0000","last_clean":"2024-09-21T12:34:29.291114+0000","last_became_active":"2024-09-21T12:34:34.338167+0000","last_became_peered":"2024-09-21T12:34:34.338167+0000","last_unstale":"2024-09-21T12:35:17.509949+0000","last_undegraded":"2024-09-21T12:34:34.335226+0000","last_fullsized":"2024-09-21T12:34:34.335117+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:32:20.845428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:32:20.845428+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:05.893587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":43,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.511482+0000","last_change":"2024-09-21T12:34:34.337576+0000","last_active":"2024-09-21T12:35:17.511482+0000","last_peered":"2024-09-21T12:35:17.511482+0000","last_clean":"2024-09-21T12:34:29.306536+0000","last_became_active":"2024-09-21T12:34:34.337576+0000","last_became_peered":"2024-09-21T12:34:34.337576+0000","last_unstale":"2024-09-21T12:35:17.511482+0000","last_undegraded":"2024-09-21T12:35:17.511482+0000","last_fullsized":"2024-09-21T12:34:34.335266+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:02.295255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:02.295255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:27.672745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":707,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.504250+0000","last_change":"2024-09-21T12:35:07.541730+0000","last_active":"2024-09-21T12:35:17.504250+0000","last_peered":"2024-09-21T12:35:17.504250+0000","last_clean":"2024-09-21T12:35:17.504250+0000","last_became_active":"2024-09-21T12:20:21.592982+0000","last_became_peered":"2024-09-21T12:20:21.592982+0000","last_unstale":"2024-09-21T12:35:17.504250+0000","last_undegraded":"2024-09-21T12:35:17.504250+0000","last_fullsized":"2024-09-21T12:35:17.504250+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:07.541665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:07.541665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:26.182451+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"398'15160","reported_seq":15283,"reported_epoch":398,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:35:28.386836+0000","last_change":"2024-09-21T12:34:34.345677+0000","last_active":"2024-09-21T12:35:28.386836+0000","last_peered":"2024-09-21T12:35:28.386836+0000","last_clean":"2024-09-21T12:34:33.319926+0000","last_became_active":"2024-09-21T12:34:34.345677+0000","last_became_peered":"2024-09-21T12:34:34.345677+0000","last_unstale":"2024-09-21T12:35:28.386836+0000","last_undegraded":"2024-09-21T12:34:34.331580+0000","last_fullsized":"2024-09-21T12:34:34.331472+0000","mapping_epoch":388,"log_start":"391'10500","ondisk_log_start":"391'10500","created":344,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:32:29.279916+0000","objects_scrubbed":0,"log_size":4660,"log_dups_size":0,"ondisk_log_size":4660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:49.110056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51511296,"num_objects":786,"num_object_clones":0,"num_object_copies":2358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":786,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15160,"num_write_kb":510272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":786}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"76'79","reported_seq":676,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.503416+0000","last_change":"2024-09-21T12:32:58.503527+0000","last_active":"2024-09-21T12:35:17.503416+0000","last_peered":"2024-09-21T12:35:17.503416+0000","last_clean":"2024-09-21T12:35:17.503416+0000","last_became_active":"2024-09-21T12:32:38.424710+0000","last_became_peered":"2024-09-21T12:32:38.424710+0000","last_unstale":"2024-09-21T12:35:17.503416+0000","last_undegraded":"2024-09-21T12:35:17.503416+0000","last_fullsized":"2024-09-21T12:35:17.503416+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:58.503445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:58.503445+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:29.296475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":757,"reported_epoch":398,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:35:17.507551+0000","last_change":"2024-09-21T12:34:34.334240+0000","last_active":"2024-09-21T12:35:17.507551+0000","last_peered":"2024-09-21T12:35:17.507551+0000","last_clean":"2024-09-21T12:34:29.290312+0000","last_became_active":"2024-09-21T12:34:34.334240+0000","last_became_peered":"2024-09-21T12:34:34.334240+0000","last_unstale":"2024-09-21T12:35:17.507551+0000","last_undegraded":"2024-09-21T12:34:34.331492+0000","last_fullsized":"2024-09-21T12:34:34.331281+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:28:45.782955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.782955+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:42.248203+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":771,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.505700+0000","last_change":"2024-09-21T12:32:53.934634+0000","last_active":"2024-09-21T12:35:17.505700+0000","last_peered":"2024-09-21T12:35:17.505700+0000","last_clean":"2024-09-21T12:35:17.505700+0000","last_became_active":"2024-09-21T12:32:38.411004+0000","last_became_peered":"2024-09-21T12:32:38.411004+0000","last_unstale":"2024-09-21T12:35:17.505700+0000","last_undegraded":"2024-09-21T12:35:17.505700+0000","last_fullsized":"2024-09-21T12:35:17.505700+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:53.934583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:53.934583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:53.166913+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":1643,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:25.243663+0000","last_change":"2024-09-21T12:32:58.369907+0000","last_active":"2024-09-21T12:35:25.243663+0000","last_peered":"2024-09-21T12:35:25.243663+0000","last_clean":"2024-09-21T12:35:25.243663+0000","last_became_active":"2024-09-21T12:20:14.287543+0000","last_became_peered":"2024-09-21T12:20:14.287543+0000","last_unstale":"2024-09-21T12:35:25.243663+0000","last_undegraded":"2024-09-21T12:35:25.243663+0000","last_fullsized":"2024-09-21T12:35:25.243663+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-21T12:32:58.369821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:32:58.369821+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":905,"num_read_kb":900,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":157,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.510735+0000","last_change":"2024-09-21T12:32:02.030536+0000","last_active":"2024-09-21T12:35:17.510735+0000","last_peered":"2024-09-21T12:35:17.510735+0000","last_clean":"2024-09-21T12:35:17.510735+0000","last_became_active":"2024-09-21T12:32:02.030240+0000","last_became_peered":"2024-09-21T12:32:02.030240+0000","last_unstale":"2024-09-21T12:35:17.510735+0000","last_undegraded":"2024-09-21T12:35:17.510735+0000","last_fullsized":"2024-09-21T12:35:17.510735+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:11.422293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:11.422293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:22.090469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":209,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:22.978507+0000","last_change":"2024-09-21T12:35:22.978507+0000","last_active":"2024-09-21T12:35:22.978507+0000","last_peered":"2024-09-21T12:35:22.978507+0000","last_clean":"2024-09-21T12:35:22.978507+0000","last_became_active":"2024-09-21T12:31:39.060160+0000","last_became_peered":"2024-09-21T12:31:39.060160+0000","last_unstale":"2024-09-21T12:35:22.978507+0000","last_undegraded":"2024-09-21T12:35:22.978507+0000","last_fullsized":"2024-09-21T12:35:22.978507+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:22.978438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:22.978438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:39.902333+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"398'15076","reported_seq":15304,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:27.643198+0000","last_change":"2024-09-21T12:35:20.465485+0000","last_active":"2024-09-21T12:35:27.643198+0000","last_peered":"2024-09-21T12:35:27.643198+0000","last_clean":"2024-09-21T12:35:27.643198+0000","last_became_active":"2024-09-21T12:32:30.306490+0000","last_became_peered":"2024-09-21T12:32:30.306490+0000","last_unstale":"2024-09-21T12:35:27.643198+0000","last_undegraded":"2024-09-21T12:35:27.643198+0000","last_fullsized":"2024-09-21T12:35:27.643198+0000","mapping_epoch":344,"log_start":"391'10500","ondisk_log_start":"391'10500","created":344,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'14385","last_scrub_stamp":"2024-09-21T12:35:20.465241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:35:20.465241+0000","objects_scrubbed":1536,"log_size":4576,"log_dups_size":0,"ondisk_log_size":4576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:44.270073+0000","scrub_duration":405,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":832,"num_object_clones":0,"num_object_copies":2496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15076,"num_write_kb":509056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"352'33","reported_seq":169,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:20.540572+0000","last_change":"2024-09-21T12:33:25.426978+0000","last_active":"2024-09-21T12:35:20.540572+0000","last_peered":"2024-09-21T12:35:20.540572+0000","last_clean":"2024-09-21T12:35:20.540572+0000","last_became_active":"2024-09-21T12:32:38.433558+0000","last_became_peered":"2024-09-21T12:32:38.433558+0000","last_unstale":"2024-09-21T12:35:20.540572+0000","last_undegraded":"2024-09-21T12:35:20.540572+0000","last_fullsized":"2024-09-21T12:35:20.540572+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'33","last_scrub_stamp":"2024-09-21T12:33:25.426799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:25.426799+0000","objects_scrubbed":8,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":732,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.509727+0000","last_change":"2024-09-21T12:34:01.815496+0000","last_active":"2024-09-21T12:35:17.509727+0000","last_peered":"2024-09-21T12:35:17.509727+0000","last_clean":"2024-09-21T12:35:17.509727+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:35:17.509727+0000","last_undegraded":"2024-09-21T12:35:17.509727+0000","last_fullsized":"2024-09-21T12:35:17.509727+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:34:01.815445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:34:01.815445+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:27.899487+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":811,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:24.687886+0000","last_change":"2024-09-21T12:35:24.687886+0000","last_active":"2024-09-21T12:35:24.687886+0000","last_peered":"2024-09-21T12:35:24.687886+0000","last_clean":"2024-09-21T12:35:24.687886+0000","last_became_active":"2024-09-21T12:26:44.950132+0000","last_became_peered":"2024-09-21T12:26:44.950132+0000","last_unstale":"2024-09-21T12:35:24.687886+0000","last_undegraded":"2024-09-21T12:35:24.687886+0000","last_fullsized":"2024-09-21T12:35:24.687886+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:35:24.687711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:35:24.687711+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:44.588456+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":145,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.504060+0000","last_change":"2024-09-21T12:34:34.339091+0000","last_active":"2024-09-21T12:35:17.504060+0000","last_peered":"2024-09-21T12:35:17.504060+0000","last_clean":"2024-09-21T12:34:29.282391+0000","last_became_active":"2024-09-21T12:34:34.339091+0000","last_became_peered":"2024-09-21T12:34:34.339091+0000","last_unstale":"2024-09-21T12:35:17.504060+0000","last_undegraded":"2024-09-21T12:35:17.504060+0000","last_fullsized":"2024-09-21T12:34:34.333720+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:36.684459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:36.684459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:35.417864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":565,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.512432+0000","last_change":"2024-09-21T12:32:38.409519+0000","last_active":"2024-09-21T12:35:17.512432+0000","last_peered":"2024-09-21T12:35:17.512432+0000","last_clean":"2024-09-21T12:35:17.512432+0000","last_became_active":"2024-09-21T12:32:38.409187+0000","last_became_peered":"2024-09-21T12:32:38.409187+0000","last_unstale":"2024-09-21T12:35:17.512432+0000","last_undegraded":"2024-09-21T12:35:17.512432+0000","last_fullsized":"2024-09-21T12:35:17.512432+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:28.647287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:28.647287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:55.954255+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"398'15100","reported_seq":15481,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:26.862468+0000","last_change":"2024-09-21T12:32:41.394500+0000","last_active":"2024-09-21T12:35:26.862468+0000","last_peered":"2024-09-21T12:35:26.862468+0000","last_clean":"2024-09-21T12:35:26.862468+0000","last_became_active":"2024-09-21T12:32:41.393857+0000","last_became_peered":"2024-09-21T12:32:41.393857+0000","last_unstale":"2024-09-21T12:35:26.862468+0000","last_undegraded":"2024-09-21T12:35:26.862468+0000","last_fullsized":"2024-09-21T12:35:26.862468+0000","mapping_epoch":352,"log_start":"390'10300","ondisk_log_start":"390'10300","created":344,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:32:29.279916+0000","objects_scrubbed":0,"log_size":4800,"log_dups_size":0,"ondisk_log_size":4800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:05.887743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64749568,"num_objects":988,"num_object_clones":0,"num_object_copies":2964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15100,"num_write_kb":514816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":8257536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"76'79","reported_seq":672,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.504019+0000","last_change":"2024-09-21T12:33:04.508642+0000","last_active":"2024-09-21T12:35:17.504019+0000","last_peered":"2024-09-21T12:35:17.504019+0000","last_clean":"2024-09-21T12:35:17.504019+0000","last_became_active":"2024-09-21T12:27:14.954351+0000","last_became_peered":"2024-09-21T12:27:14.954351+0000","last_unstale":"2024-09-21T12:35:17.504019+0000","last_undegraded":"2024-09-21T12:35:17.504019+0000","last_fullsized":"2024-09-21T12:35:17.504019+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:04.508319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:04.508319+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:31.308296+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":129,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.501024+0000","last_change":"2024-09-21T12:33:35.410866+0000","last_active":"2024-09-21T12:35:17.501024+0000","last_peered":"2024-09-21T12:35:17.501024+0000","last_clean":"2024-09-21T12:35:17.501024+0000","last_became_active":"2024-09-21T12:32:38.419714+0000","last_became_peered":"2024-09-21T12:32:38.419714+0000","last_unstale":"2024-09-21T12:35:17.501024+0000","last_undegraded":"2024-09-21T12:35:17.501024+0000","last_fullsized":"2024-09-21T12:35:17.501024+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:33:35.410528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:35.410528+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":121,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.499281+0000","last_change":"2024-09-21T12:34:34.334421+0000","last_active":"2024-09-21T12:35:17.499281+0000","last_peered":"2024-09-21T12:35:17.499281+0000","last_clean":"2024-09-21T12:34:29.279466+0000","last_became_active":"2024-09-21T12:34:34.334421+0000","last_became_peered":"2024-09-21T12:34:34.334421+0000","last_unstale":"2024-09-21T12:35:17.499281+0000","last_undegraded":"2024-09-21T12:35:17.499281+0000","last_fullsized":"2024-09-21T12:34:34.329501+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:01.342864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:01.342864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":117,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.500516+0000","last_change":"2024-09-21T12:33:59.196744+0000","last_active":"2024-09-21T12:35:17.500516+0000","last_peered":"2024-09-21T12:35:17.500516+0000","last_clean":"2024-09-21T12:35:17.500516+0000","last_became_active":"2024-09-21T12:32:38.425683+0000","last_became_peered":"2024-09-21T12:32:38.425683+0000","last_unstale":"2024-09-21T12:35:17.500516+0000","last_undegraded":"2024-09-21T12:35:17.500516+0000","last_fullsized":"2024-09-21T12:35:17.500516+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:59.196667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:33:59.196667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":713,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:24.786713+0000","last_change":"2024-09-21T12:35:24.786713+0000","last_active":"2024-09-21T12:35:24.786713+0000","last_peered":"2024-09-21T12:35:24.786713+0000","last_clean":"2024-09-21T12:35:24.786713+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:35:24.786713+0000","last_undegraded":"2024-09-21T12:35:24.786713+0000","last_fullsized":"2024-09-21T12:35:24.786713+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:24.786609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:24.786609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:26.570879+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"398'14808","reported_seq":14930,"reported_epoch":398,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:35:27.765708+0000","last_change":"2024-09-21T12:34:34.343294+0000","last_active":"2024-09-21T12:35:27.765708+0000","last_peered":"2024-09-21T12:35:27.765708+0000","last_clean":"2024-09-21T12:34:33.321340+0000","last_became_active":"2024-09-21T12:34:34.343294+0000","last_became_peered":"2024-09-21T12:34:34.343294+0000","last_unstale":"2024-09-21T12:35:27.765708+0000","last_undegraded":"2024-09-21T12:34:34.335818+0000","last_fullsized":"2024-09-21T12:34:34.335573+0000","mapping_epoch":388,"log_start":"393'10602","ondisk_log_start":"393'10602","created":344,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:32:29.279916+0000","objects_scrubbed":0,"log_size":4206,"log_dups_size":0,"ondisk_log_size":4206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:11.810118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56229888,"num_objects":858,"num_object_clones":0,"num_object_copies":2574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":858,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14808,"num_write_kb":501312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3,5","objects":858}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"76'79","reported_seq":53,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.509132+0000","last_change":"2024-09-21T12:34:34.338192+0000","last_active":"2024-09-21T12:35:17.509132+0000","last_peered":"2024-09-21T12:35:17.509132+0000","last_clean":"2024-09-21T12:34:29.289953+0000","last_became_active":"2024-09-21T12:34:34.338192+0000","last_became_peered":"2024-09-21T12:34:34.338192+0000","last_unstale":"2024-09-21T12:35:17.509132+0000","last_undegraded":"2024-09-21T12:35:17.509132+0000","last_fullsized":"2024-09-21T12:34:34.334873+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:17.035216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:17.035216+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:19.297400+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":139,"reported_epoch":398,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:35:17.500627+0000","last_change":"2024-09-21T12:34:34.341886+0000","last_active":"2024-09-21T12:35:17.500627+0000","last_peered":"2024-09-21T12:35:17.500627+0000","last_clean":"2024-09-21T12:34:29.279382+0000","last_became_active":"2024-09-21T12:34:34.341886+0000","last_became_peered":"2024-09-21T12:34:34.341886+0000","last_unstale":"2024-09-21T12:35:17.500627+0000","last_undegraded":"2024-09-21T12:34:34.333296+0000","last_fullsized":"2024-09-21T12:34:34.332270+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'31","last_scrub_stamp":"2024-09-21T12:33:08.360012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:08.360012+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":125,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:18.088890+0000","last_change":"2024-09-21T12:35:18.088890+0000","last_active":"2024-09-21T12:35:18.088890+0000","last_peered":"2024-09-21T12:35:18.088890+0000","last_clean":"2024-09-21T12:35:18.088890+0000","last_became_active":"2024-09-21T12:32:38.424451+0000","last_became_peered":"2024-09-21T12:32:38.424451+0000","last_unstale":"2024-09-21T12:35:18.088890+0000","last_undegraded":"2024-09-21T12:35:18.088890+0000","last_fullsized":"2024-09-21T12:35:18.088890+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:18.088831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:18.088831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:32.381939+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":738,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.503426+0000","last_change":"2024-09-21T12:34:46.278390+0000","last_active":"2024-09-21T12:35:17.503426+0000","last_peered":"2024-09-21T12:35:17.503426+0000","last_clean":"2024-09-21T12:35:17.503426+0000","last_became_active":"2024-09-21T12:32:38.425177+0000","last_became_peered":"2024-09-21T12:32:38.425177+0000","last_unstale":"2024-09-21T12:35:17.503426+0000","last_undegraded":"2024-09-21T12:35:17.503426+0000","last_fullsized":"2024-09-21T12:35:17.503426+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:46.278330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:46.278330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:06.754141+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":663,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.512312+0000","last_change":"2024-09-21T12:34:34.345710+0000","last_active":"2024-09-21T12:35:17.512312+0000","last_peered":"2024-09-21T12:35:17.512312+0000","last_clean":"2024-09-21T12:34:29.294810+0000","last_became_active":"2024-09-21T12:34:34.345710+0000","last_became_peered":"2024-09-21T12:34:34.345710+0000","last_unstale":"2024-09-21T12:35:17.512312+0000","last_undegraded":"2024-09-21T12:35:17.512312+0000","last_fullsized":"2024-09-21T12:34:34.334512+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:20.005906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:34:20.005906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:44.252661+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"398'15009","reported_seq":15508,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:27.649637+0000","last_change":"2024-09-21T12:34:14.480995+0000","last_active":"2024-09-21T12:35:27.649637+0000","last_peered":"2024-09-21T12:35:27.649637+0000","last_clean":"2024-09-21T12:35:27.649637+0000","last_became_active":"2024-09-21T12:32:42.035883+0000","last_became_peered":"2024-09-21T12:32:42.035883+0000","last_unstale":"2024-09-21T12:35:27.649637+0000","last_undegraded":"2024-09-21T12:35:27.649637+0000","last_fullsized":"2024-09-21T12:35:27.649637+0000","mapping_epoch":353,"log_start":"391'10500","ondisk_log_start":"391'10500","created":344,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"383'8450","last_scrub_stamp":"2024-09-21T12:34:14.480660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:34:14.480660+0000","objects_scrubbed":7514,"log_size":4509,"log_dups_size":0,"ondisk_log_size":4509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:43.919206+0000","scrub_duration":2231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57606144,"num_objects":879,"num_object_clones":0,"num_object_copies":2637,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15009,"num_write_kb":508416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":9306112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"76'79","reported_seq":797,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.506003+0000","last_change":"2024-09-21T12:34:34.338374+0000","last_active":"2024-09-21T12:35:17.506003+0000","last_peered":"2024-09-21T12:35:17.506003+0000","last_clean":"2024-09-21T12:34:29.282201+0000","last_became_active":"2024-09-21T12:34:34.338374+0000","last_became_peered":"2024-09-21T12:34:34.338374+0000","last_unstale":"2024-09-21T12:35:17.506003+0000","last_undegraded":"2024-09-21T12:35:17.506003+0000","last_fullsized":"2024-09-21T12:34:34.332296+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:05.500994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:05.500994+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:59.426015+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":125,"reported_epoch":398,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:35:17.499542+0000","last_change":"2024-09-21T12:34:34.347138+0000","last_active":"2024-09-21T12:35:17.499542+0000","last_peered":"2024-09-21T12:35:17.499542+0000","last_clean":"2024-09-21T12:34:29.280929+0000","last_became_active":"2024-09-21T12:34:34.347138+0000","last_became_peered":"2024-09-21T12:34:34.347138+0000","last_unstale":"2024-09-21T12:35:17.499542+0000","last_undegraded":"2024-09-21T12:34:34.332331+0000","last_fullsized":"2024-09-21T12:34:34.330784+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:32:26.690403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:26.690403+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:17.124255+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":745,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.509333+0000","last_change":"2024-09-21T12:32:38.404305+0000","last_active":"2024-09-21T12:35:17.509333+0000","last_peered":"2024-09-21T12:35:17.509333+0000","last_clean":"2024-09-21T12:35:17.509333+0000","last_became_active":"2024-09-21T12:32:38.403997+0000","last_became_peered":"2024-09-21T12:32:38.403997+0000","last_unstale":"2024-09-21T12:35:17.509333+0000","last_undegraded":"2024-09-21T12:35:17.509333+0000","last_fullsized":"2024-09-21T12:35:17.509333+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:47.248389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:31:47.248389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:01.962290+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"389'10","reported_seq":84,"reported_epoch":398,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:35:20.540628+0000","last_change":"2024-09-21T12:34:34.337894+0000","last_active":"2024-09-21T12:35:20.540628+0000","last_peered":"2024-09-21T12:35:20.540628+0000","last_clean":"2024-09-21T12:34:30.531145+0000","last_became_active":"2024-09-21T12:34:34.337894+0000","last_became_peered":"2024-09-21T12:34:34.337894+0000","last_unstale":"2024-09-21T12:35:20.540628+0000","last_undegraded":"2024-09-21T12:34:34.333978+0000","last_fullsized":"2024-09-21T12:34:34.333743+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:33:34.270590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:33:34.270590+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":121,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.500474+0000","last_change":"2024-09-21T12:34:34.348303+0000","last_active":"2024-09-21T12:35:17.500474+0000","last_peered":"2024-09-21T12:35:17.500474+0000","last_clean":"2024-09-21T12:34:29.279670+0000","last_became_active":"2024-09-21T12:34:34.348303+0000","last_became_peered":"2024-09-21T12:34:34.348303+0000","last_unstale":"2024-09-21T12:35:17.500474+0000","last_undegraded":"2024-09-21T12:35:17.500474+0000","last_fullsized":"2024-09-21T12:34:34.331636+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:06.369436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:06.369436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"398'15023","reported_seq":15235,"reported_epoch":398,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:35:28.402018+0000","last_change":"2024-09-21T12:34:34.338893+0000","last_active":"2024-09-21T12:35:28.402018+0000","last_peered":"2024-09-21T12:35:28.402018+0000","last_clean":"2024-09-21T12:34:33.294131+0000","last_became_active":"2024-09-21T12:34:34.338893+0000","last_became_peered":"2024-09-21T12:34:34.338893+0000","last_unstale":"2024-09-21T12:35:28.402018+0000","last_undegraded":"2024-09-21T12:34:34.331926+0000","last_fullsized":"2024-09-21T12:34:34.331816+0000","mapping_epoch":388,"log_start":"391'10382","ondisk_log_start":"391'10382","created":344,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8210","last_scrub_stamp":"2024-09-21T12:34:11.614253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:34:11.614253+0000","objects_scrubbed":7651,"log_size":4641,"log_dups_size":0,"ondisk_log_size":4641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:14.135842+0000","scrub_duration":3304,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49086464,"num_objects":749,"num_object_clones":0,"num_object_copies":2247,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":749,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15023,"num_write_kb":504704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":749}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"76'79","reported_seq":117,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.501106+0000","last_change":"2024-09-21T12:34:49.155650+0000","last_active":"2024-09-21T12:35:17.501106+0000","last_peered":"2024-09-21T12:35:17.501106+0000","last_clean":"2024-09-21T12:35:17.501106+0000","last_became_active":"2024-09-21T12:32:38.433365+0000","last_became_peered":"2024-09-21T12:32:38.433365+0000","last_unstale":"2024-09-21T12:35:17.501106+0000","last_undegraded":"2024-09-21T12:35:17.501106+0000","last_fullsized":"2024-09-21T12:35:17.501106+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:49.155069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:49.155069+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:17.664539+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":833,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.508962+0000","last_change":"2024-09-21T12:35:11.198389+0000","last_active":"2024-09-21T12:35:17.508962+0000","last_peered":"2024-09-21T12:35:17.508962+0000","last_clean":"2024-09-21T12:35:17.508962+0000","last_became_active":"2024-09-21T12:32:38.424376+0000","last_became_peered":"2024-09-21T12:32:38.424376+0000","last_unstale":"2024-09-21T12:35:17.508962+0000","last_undegraded":"2024-09-21T12:35:17.508962+0000","last_fullsized":"2024-09-21T12:35:17.508962+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:35:11.197905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:11.197905+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:17.072773+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":409,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.513255+0000","last_change":"2024-09-21T12:34:36.079650+0000","last_active":"2024-09-21T12:35:17.513255+0000","last_peered":"2024-09-21T12:35:17.513255+0000","last_clean":"2024-09-21T12:35:17.513255+0000","last_became_active":"2024-09-21T12:26:54.084228+0000","last_became_peered":"2024-09-21T12:26:54.084228+0000","last_unstale":"2024-09-21T12:35:17.513255+0000","last_undegraded":"2024-09-21T12:35:17.513255+0000","last_fullsized":"2024-09-21T12:35:17.513255+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:36.079583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:34:36.079583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:58.142377+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":683,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.499455+0000","last_change":"2024-09-21T12:34:34.346822+0000","last_active":"2024-09-21T12:35:17.499455+0000","last_peered":"2024-09-21T12:35:17.499455+0000","last_clean":"2024-09-21T12:34:29.278339+0000","last_became_active":"2024-09-21T12:34:34.346822+0000","last_became_peered":"2024-09-21T12:34:34.346822+0000","last_unstale":"2024-09-21T12:35:17.499455+0000","last_undegraded":"2024-09-21T12:35:17.499455+0000","last_fullsized":"2024-09-21T12:34:34.330172+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:47.361115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:47.361115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:01.083478+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":712,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.505786+0000","last_change":"2024-09-21T12:34:34.343541+0000","last_active":"2024-09-21T12:35:17.505786+0000","last_peered":"2024-09-21T12:35:17.505786+0000","last_clean":"2024-09-21T12:34:29.289308+0000","last_became_active":"2024-09-21T12:34:34.343541+0000","last_became_peered":"2024-09-21T12:34:34.343541+0000","last_unstale":"2024-09-21T12:35:17.505786+0000","last_undegraded":"2024-09-21T12:35:17.505786+0000","last_fullsized":"2024-09-21T12:34:34.331631+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:37.119815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:37.119815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:09.193346+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"398'15228","reported_seq":15734,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:28.399060+0000","last_change":"2024-09-21T12:35:16.787414+0000","last_active":"2024-09-21T12:35:28.399060+0000","last_peered":"2024-09-21T12:35:28.399060+0000","last_clean":"2024-09-21T12:35:28.399060+0000","last_became_active":"2024-09-21T12:32:42.036249+0000","last_became_peered":"2024-09-21T12:32:42.036249+0000","last_unstale":"2024-09-21T12:35:28.399060+0000","last_undegraded":"2024-09-21T12:35:28.399060+0000","last_fullsized":"2024-09-21T12:35:28.399060+0000","mapping_epoch":353,"log_start":"391'10600","ondisk_log_start":"391'10600","created":344,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'14018","last_scrub_stamp":"2024-09-21T12:35:16.787195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:35:16.787195+0000","objects_scrubbed":1989,"log_size":4628,"log_dups_size":0,"ondisk_log_size":4628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:40.953429+0000","scrub_duration":596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50200576,"num_objects":766,"num_object_clones":0,"num_object_copies":2298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15228,"num_write_kb":511808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":8454144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"76'79","reported_seq":648,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.511087+0000","last_change":"2024-09-21T12:32:38.415231+0000","last_active":"2024-09-21T12:35:17.511087+0000","last_peered":"2024-09-21T12:35:17.511087+0000","last_clean":"2024-09-21T12:35:17.511087+0000","last_became_active":"2024-09-21T12:32:38.414877+0000","last_became_peered":"2024-09-21T12:32:38.414877+0000","last_unstale":"2024-09-21T12:35:17.511087+0000","last_undegraded":"2024-09-21T12:35:17.511087+0000","last_fullsized":"2024-09-21T12:35:17.511087+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:25.729795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:25.729795+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:57.023502+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":839,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.508275+0000","last_change":"2024-09-21T12:34:46.956902+0000","last_active":"2024-09-21T12:35:17.508275+0000","last_peered":"2024-09-21T12:35:17.508275+0000","last_clean":"2024-09-21T12:35:17.508275+0000","last_became_active":"2024-09-21T12:32:14.236905+0000","last_became_peered":"2024-09-21T12:32:14.236905+0000","last_unstale":"2024-09-21T12:35:17.508275+0000","last_undegraded":"2024-09-21T12:35:17.508275+0000","last_fullsized":"2024-09-21T12:35:17.508275+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'35","last_scrub_stamp":"2024-09-21T12:34:46.956618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:34:46.956618+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:03.404602+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":732,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.506387+0000","last_change":"2024-09-21T12:33:08.316386+0000","last_active":"2024-09-21T12:35:17.506387+0000","last_peered":"2024-09-21T12:35:17.506387+0000","last_clean":"2024-09-21T12:35:17.506387+0000","last_became_active":"2024-09-21T12:32:02.025668+0000","last_became_peered":"2024-09-21T12:32:02.025668+0000","last_unstale":"2024-09-21T12:35:17.506387+0000","last_undegraded":"2024-09-21T12:35:17.506387+0000","last_fullsized":"2024-09-21T12:35:17.506387+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:08.316315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:08.316315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":621,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.509345+0000","last_change":"2024-09-21T12:34:34.343106+0000","last_active":"2024-09-21T12:35:17.509345+0000","last_peered":"2024-09-21T12:35:17.509345+0000","last_clean":"2024-09-21T12:34:29.291366+0000","last_became_active":"2024-09-21T12:34:34.343106+0000","last_became_peered":"2024-09-21T12:34:34.343106+0000","last_unstale":"2024-09-21T12:35:17.509345+0000","last_undegraded":"2024-09-21T12:35:17.509345+0000","last_fullsized":"2024-09-21T12:34:34.333287+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:20.243774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:20.243774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:26.174978+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":702,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.503791+0000","last_change":"2024-09-21T12:34:34.343105+0000","last_active":"2024-09-21T12:35:17.503791+0000","last_peered":"2024-09-21T12:35:17.503791+0000","last_clean":"2024-09-21T12:34:29.284141+0000","last_became_active":"2024-09-21T12:34:34.343105+0000","last_became_peered":"2024-09-21T12:34:34.343105+0000","last_unstale":"2024-09-21T12:35:17.503791+0000","last_undegraded":"2024-09-21T12:35:17.503791+0000","last_fullsized":"2024-09-21T12:34:34.332503+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:27.297558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:27.297558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"398'14900","reported_seq":15106,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:27.754829+0000","last_change":"2024-09-21T12:34:05.360222+0000","last_active":"2024-09-21T12:35:27.754829+0000","last_peered":"2024-09-21T12:35:27.754829+0000","last_clean":"2024-09-21T12:35:27.754829+0000","last_became_active":"2024-09-21T12:32:30.351779+0000","last_became_peered":"2024-09-21T12:32:30.351779+0000","last_unstale":"2024-09-21T12:35:27.754829+0000","last_undegraded":"2024-09-21T12:35:27.754829+0000","last_fullsized":"2024-09-21T12:35:27.754829+0000","mapping_epoch":344,"log_start":"393'10700","ondisk_log_start":"393'10700","created":344,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'7793","last_scrub_stamp":"2024-09-21T12:34:05.360110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:34:05.360110+0000","objects_scrubbed":7718,"log_size":4200,"log_dups_size":0,"ondisk_log_size":4200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2513,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57278464,"num_objects":874,"num_object_clones":0,"num_object_copies":2622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14900,"num_write_kb":504768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"76'79","reported_seq":671,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.512951+0000","last_change":"2024-09-21T12:33:37.767771+0000","last_active":"2024-09-21T12:35:17.512951+0000","last_peered":"2024-09-21T12:35:17.512951+0000","last_clean":"2024-09-21T12:35:17.512951+0000","last_became_active":"2024-09-21T12:27:14.951375+0000","last_became_peered":"2024-09-21T12:27:14.951375+0000","last_unstale":"2024-09-21T12:35:17.512951+0000","last_undegraded":"2024-09-21T12:35:17.512951+0000","last_fullsized":"2024-09-21T12:35:17.512951+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:37.767584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:37.767584+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":129,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.499695+0000","last_change":"2024-09-21T12:35:12.197153+0000","last_active":"2024-09-21T12:35:17.499695+0000","last_peered":"2024-09-21T12:35:17.499695+0000","last_clean":"2024-09-21T12:35:17.499695+0000","last_became_active":"2024-09-21T12:32:38.421609+0000","last_became_peered":"2024-09-21T12:32:38.421609+0000","last_unstale":"2024-09-21T12:35:17.499695+0000","last_undegraded":"2024-09-21T12:35:17.499695+0000","last_fullsized":"2024-09-21T12:35:17.499695+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T12:35:12.197060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:12.197060+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:26.198288+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":762,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:19.611605+0000","last_change":"2024-09-21T12:35:19.611605+0000","last_active":"2024-09-21T12:35:19.611605+0000","last_peered":"2024-09-21T12:35:19.611605+0000","last_clean":"2024-09-21T12:35:19.611605+0000","last_became_active":"2024-09-21T12:32:38.417869+0000","last_became_peered":"2024-09-21T12:32:38.417869+0000","last_unstale":"2024-09-21T12:35:19.611605+0000","last_undegraded":"2024-09-21T12:35:19.611605+0000","last_fullsized":"2024-09-21T12:35:19.611605+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:19.611537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:19.611537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:39.443426+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":43,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.505352+0000","last_change":"2024-09-21T12:34:34.333741+0000","last_active":"2024-09-21T12:35:17.505352+0000","last_peered":"2024-09-21T12:35:17.505352+0000","last_clean":"2024-09-21T12:34:29.289730+0000","last_became_active":"2024-09-21T12:34:34.333741+0000","last_became_peered":"2024-09-21T12:34:34.333741+0000","last_unstale":"2024-09-21T12:35:17.505352+0000","last_undegraded":"2024-09-21T12:35:17.505352+0000","last_fullsized":"2024-09-21T12:34:34.330975+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:23.769748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:32:23.769748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:43.251313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":727,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.508862+0000","last_change":"2024-09-21T12:34:04.876726+0000","last_active":"2024-09-21T12:35:17.508862+0000","last_peered":"2024-09-21T12:35:17.508862+0000","last_clean":"2024-09-21T12:35:17.508862+0000","last_became_active":"2024-09-21T12:32:14.242180+0000","last_became_peered":"2024-09-21T12:32:14.242180+0000","last_unstale":"2024-09-21T12:35:17.508862+0000","last_undegraded":"2024-09-21T12:35:17.508862+0000","last_fullsized":"2024-09-21T12:35:17.508862+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:04.876656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:34:04.876656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:27.205484+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"398'14884","reported_seq":15359,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:28.391086+0000","last_change":"2024-09-21T12:34:12.121964+0000","last_active":"2024-09-21T12:35:28.391086+0000","last_peered":"2024-09-21T12:35:28.391086+0000","last_clean":"2024-09-21T12:35:28.391086+0000","last_became_active":"2024-09-21T12:32:42.036291+0000","last_became_peered":"2024-09-21T12:32:42.036291+0000","last_unstale":"2024-09-21T12:35:28.391086+0000","last_undegraded":"2024-09-21T12:35:28.391086+0000","last_fullsized":"2024-09-21T12:35:28.391086+0000","mapping_epoch":353,"log_start":"391'10200","ondisk_log_start":"391'10200","created":344,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'8181","last_scrub_stamp":"2024-09-21T12:34:12.121742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:32:29.279916+0000","last_clean_scrub_stamp":"2024-09-21T12:34:12.121742+0000","objects_scrubbed":7522,"log_size":4684,"log_dups_size":0,"ondisk_log_size":4684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2896,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48627712,"num_objects":742,"num_object_clones":0,"num_object_copies":2226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14884,"num_write_kb":500032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":8585216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"76'79","reported_seq":156,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.502624+0000","last_change":"2024-09-21T12:33:32.272999+0000","last_active":"2024-09-21T12:35:17.502624+0000","last_peered":"2024-09-21T12:35:17.502624+0000","last_clean":"2024-09-21T12:35:17.502624+0000","last_became_active":"2024-09-21T12:32:38.418574+0000","last_became_peered":"2024-09-21T12:32:38.418574+0000","last_unstale":"2024-09-21T12:35:17.502624+0000","last_undegraded":"2024-09-21T12:35:17.502624+0000","last_fullsized":"2024-09-21T12:35:17.502624+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:32.272927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:32.272927+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":204,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.509351+0000","last_change":"2024-09-21T12:33:02.333599+0000","last_active":"2024-09-21T12:35:17.509351+0000","last_peered":"2024-09-21T12:35:17.509351+0000","last_clean":"2024-09-21T12:35:17.509351+0000","last_became_active":"2024-09-21T12:32:14.234897+0000","last_became_peered":"2024-09-21T12:32:14.234897+0000","last_unstale":"2024-09-21T12:35:17.509351+0000","last_undegraded":"2024-09-21T12:35:17.509351+0000","last_fullsized":"2024-09-21T12:35:17.509351+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:33:02.333507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:02.333507+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:24.079126+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":808,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.503794+0000","last_change":"2024-09-21T12:32:38.444277+0000","last_active":"2024-09-21T12:35:17.503794+0000","last_peered":"2024-09-21T12:35:17.503794+0000","last_clean":"2024-09-21T12:35:17.503794+0000","last_became_active":"2024-09-21T12:32:38.417302+0000","last_became_peered":"2024-09-21T12:32:38.417302+0000","last_unstale":"2024-09-21T12:35:17.503794+0000","last_undegraded":"2024-09-21T12:35:17.503794+0000","last_fullsized":"2024-09-21T12:35:17.503794+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-21T12:31:53.103575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:31:53.103575+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:05.526614+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"352'14","reported_seq":968,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:20.546655+0000","last_change":"2024-09-21T12:35:06.542878+0000","last_active":"2024-09-21T12:35:20.546655+0000","last_peered":"2024-09-21T12:35:20.546655+0000","last_clean":"2024-09-21T12:35:20.546655+0000","last_became_active":"2024-09-21T12:32:38.418813+0000","last_became_peered":"2024-09-21T12:32:38.418813+0000","last_unstale":"2024-09-21T12:35:20.546655+0000","last_undegraded":"2024-09-21T12:35:20.546655+0000","last_fullsized":"2024-09-21T12:35:20.546655+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'14","last_scrub_stamp":"2024-09-21T12:35:06.542700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:35:06.542700+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:11.732554+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":417,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.508085+0000","last_change":"2024-09-21T12:35:07.828972+0000","last_active":"2024-09-21T12:35:17.508085+0000","last_peered":"2024-09-21T12:35:17.508085+0000","last_clean":"2024-09-21T12:35:17.508085+0000","last_became_active":"2024-09-21T12:26:54.089155+0000","last_became_peered":"2024-09-21T12:26:54.089155+0000","last_unstale":"2024-09-21T12:35:17.508085+0000","last_undegraded":"2024-09-21T12:35:17.508085+0000","last_fullsized":"2024-09-21T12:35:17.508085+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:07.828886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:07.828886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:33.632460+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"76'79","reported_seq":455,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.509958+0000","last_change":"2024-09-21T12:32:26.698667+0000","last_active":"2024-09-21T12:35:17.509958+0000","last_peered":"2024-09-21T12:35:17.509958+0000","last_clean":"2024-09-21T12:35:17.509958+0000","last_became_active":"2024-09-21T12:31:39.060422+0000","last_became_peered":"2024-09-21T12:31:39.060422+0000","last_unstale":"2024-09-21T12:35:17.509958+0000","last_undegraded":"2024-09-21T12:35:17.509958+0000","last_fullsized":"2024-09-21T12:35:17.509958+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:26.698622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:26.698622+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:00.014991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":764,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.507390+0000","last_change":"2024-09-21T12:34:45.969199+0000","last_active":"2024-09-21T12:35:17.507390+0000","last_peered":"2024-09-21T12:35:17.507390+0000","last_clean":"2024-09-21T12:35:17.507390+0000","last_became_active":"2024-09-21T12:32:02.026693+0000","last_became_peered":"2024-09-21T12:32:02.026693+0000","last_unstale":"2024-09-21T12:35:17.507390+0000","last_undegraded":"2024-09-21T12:35:17.507390+0000","last_fullsized":"2024-09-21T12:35:17.507390+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:45.968651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:34:45.968651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:56.177081+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":216,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.513138+0000","last_change":"2024-09-21T12:34:09.872532+0000","last_active":"2024-09-21T12:35:17.513138+0000","last_peered":"2024-09-21T12:35:17.513138+0000","last_clean":"2024-09-21T12:35:17.513138+0000","last_became_active":"2024-09-21T12:32:14.244680+0000","last_became_peered":"2024-09-21T12:32:14.244680+0000","last_unstale":"2024-09-21T12:35:17.513138+0000","last_undegraded":"2024-09-21T12:35:17.513138+0000","last_fullsized":"2024-09-21T12:35:17.513138+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T12:34:09.872320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:34:09.872320+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:29.872094+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":410,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.511839+0000","last_change":"2024-09-21T12:33:34.706596+0000","last_active":"2024-09-21T12:35:17.511839+0000","last_peered":"2024-09-21T12:35:17.511839+0000","last_clean":"2024-09-21T12:35:17.511839+0000","last_became_active":"2024-09-21T12:26:54.084109+0000","last_became_peered":"2024-09-21T12:26:54.084109+0000","last_unstale":"2024-09-21T12:35:17.511839+0000","last_undegraded":"2024-09-21T12:35:17.511839+0000","last_fullsized":"2024-09-21T12:35:17.511839+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:34.706531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:33:34.706531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":734,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.503938+0000","last_change":"2024-09-21T12:35:14.567031+0000","last_active":"2024-09-21T12:35:17.503938+0000","last_peered":"2024-09-21T12:35:17.503938+0000","last_clean":"2024-09-21T12:35:17.503938+0000","last_became_active":"2024-09-21T12:32:38.417522+0000","last_became_peered":"2024-09-21T12:32:38.417522+0000","last_unstale":"2024-09-21T12:35:17.503938+0000","last_undegraded":"2024-09-21T12:35:17.503938+0000","last_fullsized":"2024-09-21T12:35:17.503938+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:14.566963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:14.566963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:26.001835+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'79","reported_seq":695,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.503508+0000","last_change":"2024-09-21T12:34:20.427215+0000","last_active":"2024-09-21T12:35:17.503508+0000","last_peered":"2024-09-21T12:35:17.503508+0000","last_clean":"2024-09-21T12:35:17.503508+0000","last_became_active":"2024-09-21T12:32:38.425495+0000","last_became_peered":"2024-09-21T12:32:38.425495+0000","last_unstale":"2024-09-21T12:35:17.503508+0000","last_undegraded":"2024-09-21T12:35:17.503508+0000","last_fullsized":"2024-09-21T12:35:17.503508+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:20.427138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:20.427138+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:50.177264+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":783,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.507637+0000","last_change":"2024-09-21T12:32:19.786992+0000","last_active":"2024-09-21T12:35:17.507637+0000","last_peered":"2024-09-21T12:35:17.507637+0000","last_clean":"2024-09-21T12:35:17.507637+0000","last_became_active":"2024-09-21T12:32:14.242339+0000","last_became_peered":"2024-09-21T12:32:14.242339+0000","last_unstale":"2024-09-21T12:35:17.507637+0000","last_undegraded":"2024-09-21T12:35:17.507637+0000","last_fullsized":"2024-09-21T12:35:17.507637+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:32:19.786928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:19.786928+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:28.229444+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":754,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.506187+0000","last_change":"2024-09-21T12:32:38.421009+0000","last_active":"2024-09-21T12:35:17.506187+0000","last_peered":"2024-09-21T12:35:17.506187+0000","last_clean":"2024-09-21T12:35:17.506187+0000","last_became_active":"2024-09-21T12:32:38.416992+0000","last_became_peered":"2024-09-21T12:32:38.416992+0000","last_unstale":"2024-09-21T12:35:17.506187+0000","last_undegraded":"2024-09-21T12:35:17.506187+0000","last_fullsized":"2024-09-21T12:35:17.506187+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:43.545197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:43.545197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:25.230560+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":709,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.513178+0000","last_change":"2024-09-21T12:32:58.720897+0000","last_active":"2024-09-21T12:35:17.513178+0000","last_peered":"2024-09-21T12:35:17.513178+0000","last_clean":"2024-09-21T12:35:17.513178+0000","last_became_active":"2024-09-21T12:20:19.311947+0000","last_became_peered":"2024-09-21T12:20:19.311947+0000","last_unstale":"2024-09-21T12:35:17.513178+0000","last_undegraded":"2024-09-21T12:35:17.513178+0000","last_fullsized":"2024-09-21T12:35:17.513178+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:58.720834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:32:58.720834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":679,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.499991+0000","last_change":"2024-09-21T12:34:34.341069+0000","last_active":"2024-09-21T12:35:17.499991+0000","last_peered":"2024-09-21T12:35:17.499991+0000","last_clean":"2024-09-21T12:34:29.281161+0000","last_became_active":"2024-09-21T12:34:34.341069+0000","last_became_peered":"2024-09-21T12:34:34.341069+0000","last_unstale":"2024-09-21T12:35:17.499991+0000","last_undegraded":"2024-09-21T12:35:17.499991+0000","last_fullsized":"2024-09-21T12:34:34.331679+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:11.105583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:26:11.105583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:42.273955+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.10","version":"76'79","reported_seq":794,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.502409+0000","last_change":"2024-09-21T12:31:03.293976+0000","last_active":"2024-09-21T12:35:17.502409+0000","last_peered":"2024-09-21T12:35:17.502409+0000","last_clean":"2024-09-21T12:35:17.502409+0000","last_became_active":"2024-09-21T12:26:54.084939+0000","last_became_peered":"2024-09-21T12:26:54.084939+0000","last_unstale":"2024-09-21T12:35:17.502409+0000","last_undegraded":"2024-09-21T12:35:17.502409+0000","last_fullsized":"2024-09-21T12:35:17.502409+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:31:03.293909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:31:03.293909+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:04.496715+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":484,"reported_epoch":398,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:35:17.504425+0000","last_change":"2024-09-21T12:34:34.338029+0000","last_active":"2024-09-21T12:35:17.504425+0000","last_peered":"2024-09-21T12:35:17.504425+0000","last_clean":"2024-09-21T12:34:29.283333+0000","last_became_active":"2024-09-21T12:34:34.338029+0000","last_became_peered":"2024-09-21T12:34:34.338029+0000","last_unstale":"2024-09-21T12:35:17.504425+0000","last_undegraded":"2024-09-21T12:34:34.334105+0000","last_fullsized":"2024-09-21T12:34:34.333950+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:28:42.511874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:42.511874+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:55.419223+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":765,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:22.225198+0000","last_change":"2024-09-21T12:35:22.225198+0000","last_active":"2024-09-21T12:35:22.225198+0000","last_peered":"2024-09-21T12:35:22.225198+0000","last_clean":"2024-09-21T12:35:22.225198+0000","last_became_active":"2024-09-21T12:32:14.236220+0000","last_became_peered":"2024-09-21T12:32:14.236220+0000","last_unstale":"2024-09-21T12:35:22.225198+0000","last_undegraded":"2024-09-21T12:35:22.225198+0000","last_fullsized":"2024-09-21T12:35:22.225198+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:22.225122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:22.225122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:30.053517+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"389'11","reported_seq":238,"reported_epoch":398,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:35:20.540985+0000","last_change":"2024-09-21T12:34:34.343117+0000","last_active":"2024-09-21T12:35:20.540985+0000","last_peered":"2024-09-21T12:35:20.540985+0000","last_clean":"2024-09-21T12:34:30.530125+0000","last_became_active":"2024-09-21T12:34:34.343117+0000","last_became_peered":"2024-09-21T12:34:34.343117+0000","last_unstale":"2024-09-21T12:35:20.540985+0000","last_undegraded":"2024-09-21T12:34:34.335049+0000","last_fullsized":"2024-09-21T12:34:34.334908+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'10","last_scrub_stamp":"2024-09-21T12:34:14.003545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:14.003545+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:37.456798+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":121,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.500125+0000","last_change":"2024-09-21T12:34:34.340859+0000","last_active":"2024-09-21T12:35:17.500125+0000","last_peered":"2024-09-21T12:35:17.500125+0000","last_clean":"2024-09-21T12:34:29.279088+0000","last_became_active":"2024-09-21T12:34:34.340859+0000","last_became_peered":"2024-09-21T12:34:34.340859+0000","last_unstale":"2024-09-21T12:35:17.500125+0000","last_undegraded":"2024-09-21T12:35:17.500125+0000","last_fullsized":"2024-09-21T12:34:34.330912+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:05.370263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:05.370263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"76'79","reported_seq":546,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.506015+0000","last_change":"2024-09-21T12:35:17.231922+0000","last_active":"2024-09-21T12:35:17.506015+0000","last_peered":"2024-09-21T12:35:17.506015+0000","last_clean":"2024-09-21T12:35:17.506015+0000","last_became_active":"2024-09-21T12:32:02.034387+0000","last_became_peered":"2024-09-21T12:32:02.034387+0000","last_unstale":"2024-09-21T12:35:17.506015+0000","last_undegraded":"2024-09-21T12:35:17.506015+0000","last_fullsized":"2024-09-21T12:35:17.506015+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:35:17.231860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:35:17.231860+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:40.387153+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":789,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.513097+0000","last_change":"2024-09-21T12:33:31.734043+0000","last_active":"2024-09-21T12:35:17.513097+0000","last_peered":"2024-09-21T12:35:17.513097+0000","last_clean":"2024-09-21T12:35:17.513097+0000","last_became_active":"2024-09-21T12:31:39.058798+0000","last_became_peered":"2024-09-21T12:31:39.058798+0000","last_unstale":"2024-09-21T12:35:17.513097+0000","last_undegraded":"2024-09-21T12:35:17.513097+0000","last_fullsized":"2024-09-21T12:35:17.513097+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:33:31.733776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:31.733776+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":750,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.503081+0000","last_change":"2024-09-21T12:34:34.344409+0000","last_active":"2024-09-21T12:35:17.503081+0000","last_peered":"2024-09-21T12:35:17.503081+0000","last_clean":"2024-09-21T12:34:29.282494+0000","last_became_active":"2024-09-21T12:34:34.344409+0000","last_became_peered":"2024-09-21T12:34:34.344409+0000","last_unstale":"2024-09-21T12:35:17.503081+0000","last_undegraded":"2024-09-21T12:35:17.503081+0000","last_fullsized":"2024-09-21T12:34:34.331019+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:32.630261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:32.630261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:41.309689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":909,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:20.540561+0000","last_change":"2024-09-21T12:34:45.114379+0000","last_active":"2024-09-21T12:35:20.540561+0000","last_peered":"2024-09-21T12:35:20.540561+0000","last_clean":"2024-09-21T12:35:20.540561+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:35:20.540561+0000","last_undegraded":"2024-09-21T12:35:20.540561+0000","last_fullsized":"2024-09-21T12:35:20.540561+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:34:45.114312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:45.114312+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:46.093364+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":722,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:26.208530+0000","last_change":"2024-09-21T12:35:26.208530+0000","last_active":"2024-09-21T12:35:26.208530+0000","last_peered":"2024-09-21T12:35:26.208530+0000","last_clean":"2024-09-21T12:35:26.208530+0000","last_became_active":"2024-09-21T12:31:50.037934+0000","last_became_peered":"2024-09-21T12:31:50.037934+0000","last_unstale":"2024-09-21T12:35:26.208530+0000","last_undegraded":"2024-09-21T12:35:26.208530+0000","last_fullsized":"2024-09-21T12:35:26.208530+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:26.208487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:26.208487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:27.168884+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"76'79","reported_seq":395,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.503801+0000","last_change":"2024-09-21T12:33:07.561773+0000","last_active":"2024-09-21T12:35:17.503801+0000","last_peered":"2024-09-21T12:35:17.503801+0000","last_clean":"2024-09-21T12:35:17.503801+0000","last_became_active":"2024-09-21T12:27:16.940347+0000","last_became_peered":"2024-09-21T12:27:16.940347+0000","last_unstale":"2024-09-21T12:35:17.503801+0000","last_undegraded":"2024-09-21T12:35:17.503801+0000","last_fullsized":"2024-09-21T12:35:17.503801+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:07.561702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:07.561702+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:56.280977+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":129,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.499579+0000","last_change":"2024-09-21T12:34:00.191190+0000","last_active":"2024-09-21T12:35:17.499579+0000","last_peered":"2024-09-21T12:35:17.499579+0000","last_clean":"2024-09-21T12:35:17.499579+0000","last_became_active":"2024-09-21T12:32:38.421008+0000","last_became_peered":"2024-09-21T12:32:38.421008+0000","last_unstale":"2024-09-21T12:35:17.499579+0000","last_undegraded":"2024-09-21T12:35:17.499579+0000","last_fullsized":"2024-09-21T12:35:17.499579+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:34:00.191106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:34:00.191106+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":117,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.499904+0000","last_change":"2024-09-21T12:33:30.406961+0000","last_active":"2024-09-21T12:35:17.499904+0000","last_peered":"2024-09-21T12:35:17.499904+0000","last_clean":"2024-09-21T12:35:17.499904+0000","last_became_active":"2024-09-21T12:32:38.432970+0000","last_became_peered":"2024-09-21T12:32:38.432970+0000","last_unstale":"2024-09-21T12:35:17.499904+0000","last_undegraded":"2024-09-21T12:35:17.499904+0000","last_fullsized":"2024-09-21T12:35:17.499904+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:30.406915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:30.406915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":727,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.506577+0000","last_change":"2024-09-21T12:34:18.421712+0000","last_active":"2024-09-21T12:35:17.506577+0000","last_peered":"2024-09-21T12:35:17.506577+0000","last_clean":"2024-09-21T12:35:17.506577+0000","last_became_active":"2024-09-21T12:31:50.040754+0000","last_became_peered":"2024-09-21T12:31:50.040754+0000","last_unstale":"2024-09-21T12:35:17.506577+0000","last_undegraded":"2024-09-21T12:35:17.506577+0000","last_fullsized":"2024-09-21T12:35:17.506577+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:18.421650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:18.421650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:34.554916+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":185,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.509870+0000","last_change":"2024-09-21T12:31:53.359206+0000","last_active":"2024-09-21T12:35:17.509870+0000","last_peered":"2024-09-21T12:35:17.509870+0000","last_clean":"2024-09-21T12:35:17.509870+0000","last_became_active":"2024-09-21T12:31:50.036150+0000","last_became_peered":"2024-09-21T12:31:50.036150+0000","last_unstale":"2024-09-21T12:35:17.509870+0000","last_undegraded":"2024-09-21T12:35:17.509870+0000","last_fullsized":"2024-09-21T12:35:17.509870+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:53.359159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:31:53.359159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:59.999393+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"76'79","reported_seq":428,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.511273+0000","last_change":"2024-09-21T12:35:16.997515+0000","last_active":"2024-09-21T12:35:17.511273+0000","last_peered":"2024-09-21T12:35:17.511273+0000","last_clean":"2024-09-21T12:35:17.511273+0000","last_became_active":"2024-09-21T12:27:16.921880+0000","last_became_peered":"2024-09-21T12:27:16.921880+0000","last_unstale":"2024-09-21T12:35:17.511273+0000","last_undegraded":"2024-09-21T12:35:17.511273+0000","last_fullsized":"2024-09-21T12:35:17.511273+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:35:16.997373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:35:16.997373+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:44.016895+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":215,"reported_epoch":398,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:35:17.503107+0000","last_change":"2024-09-21T12:34:34.345279+0000","last_active":"2024-09-21T12:35:17.503107+0000","last_peered":"2024-09-21T12:35:17.503107+0000","last_clean":"2024-09-21T12:34:29.286797+0000","last_became_active":"2024-09-21T12:34:34.345279+0000","last_became_peered":"2024-09-21T12:34:34.345279+0000","last_unstale":"2024-09-21T12:35:17.503107+0000","last_undegraded":"2024-09-21T12:34:34.331120+0000","last_fullsized":"2024-09-21T12:34:34.331029+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:33:02.544885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:02.544885+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:36.366202+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":754,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.506244+0000","last_change":"2024-09-21T12:32:51.311159+0000","last_active":"2024-09-21T12:35:17.506244+0000","last_peered":"2024-09-21T12:35:17.506244+0000","last_clean":"2024-09-21T12:35:17.506244+0000","last_became_active":"2024-09-21T12:32:38.417928+0000","last_became_peered":"2024-09-21T12:32:38.417928+0000","last_unstale":"2024-09-21T12:35:17.506244+0000","last_undegraded":"2024-09-21T12:35:17.506244+0000","last_fullsized":"2024-09-21T12:35:17.506244+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:51.311112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:51.311112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":717,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.509160+0000","last_change":"2024-09-21T12:34:11.934886+0000","last_active":"2024-09-21T12:35:17.509160+0000","last_peered":"2024-09-21T12:35:17.509160+0000","last_clean":"2024-09-21T12:35:17.509160+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:35:17.509160+0000","last_undegraded":"2024-09-21T12:35:17.509160+0000","last_fullsized":"2024-09-21T12:35:17.509160+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:11.934826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:11.934826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:23.395786+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":700,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.505443+0000","last_change":"2024-09-21T12:28:41.864711+0000","last_active":"2024-09-21T12:35:17.505443+0000","last_peered":"2024-09-21T12:35:17.505443+0000","last_clean":"2024-09-21T12:35:17.505443+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:35:17.505443+0000","last_undegraded":"2024-09-21T12:35:17.505443+0000","last_fullsized":"2024-09-21T12:35:17.505443+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:41.864630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:41.864630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:57.132712+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"76'79","reported_seq":211,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.504188+0000","last_change":"2024-09-21T12:34:34.342536+0000","last_active":"2024-09-21T12:35:17.504188+0000","last_peered":"2024-09-21T12:35:17.504188+0000","last_clean":"2024-09-21T12:34:29.282503+0000","last_became_active":"2024-09-21T12:34:34.342536+0000","last_became_peered":"2024-09-21T12:34:34.342536+0000","last_unstale":"2024-09-21T12:35:17.504188+0000","last_undegraded":"2024-09-21T12:35:17.504188+0000","last_fullsized":"2024-09-21T12:34:34.331363+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:17.398972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:17.398972+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:23.834472+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":752,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.510424+0000","last_change":"2024-09-21T12:33:27.793529+0000","last_active":"2024-09-21T12:35:17.510424+0000","last_peered":"2024-09-21T12:35:17.510424+0000","last_clean":"2024-09-21T12:35:17.510424+0000","last_became_active":"2024-09-21T12:20:17.694794+0000","last_became_peered":"2024-09-21T12:20:17.694794+0000","last_unstale":"2024-09-21T12:35:17.510424+0000","last_undegraded":"2024-09-21T12:35:17.510424+0000","last_fullsized":"2024-09-21T12:35:17.510424+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:33:27.793452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:27.793452+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":742,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.511678+0000","last_change":"2024-09-21T12:34:40.142628+0000","last_active":"2024-09-21T12:35:17.511678+0000","last_peered":"2024-09-21T12:35:17.511678+0000","last_clean":"2024-09-21T12:35:17.511678+0000","last_became_active":"2024-09-21T12:20:15.295968+0000","last_became_peered":"2024-09-21T12:20:15.295968+0000","last_unstale":"2024-09-21T12:35:17.511678+0000","last_undegraded":"2024-09-21T12:35:17.511678+0000","last_fullsized":"2024-09-21T12:35:17.511678+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:40.142561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:34:40.142561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:46.956523+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":730,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.502263+0000","last_change":"2024-09-21T12:34:47.316150+0000","last_active":"2024-09-21T12:35:17.502263+0000","last_peered":"2024-09-21T12:35:17.502263+0000","last_clean":"2024-09-21T12:35:17.502263+0000","last_became_active":"2024-09-21T12:32:38.415887+0000","last_became_peered":"2024-09-21T12:32:38.415887+0000","last_unstale":"2024-09-21T12:35:17.502263+0000","last_undegraded":"2024-09-21T12:35:17.502263+0000","last_fullsized":"2024-09-21T12:35:17.502263+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:47.316065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:47.316065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:52.304715+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":705,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.509322+0000","last_change":"2024-09-21T12:33:08.711630+0000","last_active":"2024-09-21T12:35:17.509322+0000","last_peered":"2024-09-21T12:35:17.509322+0000","last_clean":"2024-09-21T12:35:17.509322+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:35:17.509322+0000","last_undegraded":"2024-09-21T12:35:17.509322+0000","last_fullsized":"2024-09-21T12:35:17.509322+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:08.711439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:08.711439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"76'79","reported_seq":396,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.502689+0000","last_change":"2024-09-21T12:34:06.362921+0000","last_active":"2024-09-21T12:35:17.502689+0000","last_peered":"2024-09-21T12:35:17.502689+0000","last_clean":"2024-09-21T12:35:17.502689+0000","last_became_active":"2024-09-21T12:27:16.914713+0000","last_became_peered":"2024-09-21T12:27:16.914713+0000","last_unstale":"2024-09-21T12:35:17.502689+0000","last_undegraded":"2024-09-21T12:35:17.502689+0000","last_fullsized":"2024-09-21T12:35:17.502689+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:06.362857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:06.362857+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":733,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.510867+0000","last_change":"2024-09-21T12:35:14.789148+0000","last_active":"2024-09-21T12:35:17.510867+0000","last_peered":"2024-09-21T12:35:17.510867+0000","last_clean":"2024-09-21T12:35:17.510867+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:35:17.510867+0000","last_undegraded":"2024-09-21T12:35:17.510867+0000","last_fullsized":"2024-09-21T12:35:17.510867+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'8","last_scrub_stamp":"2024-09-21T12:35:14.789070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:14.789070+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:16.751477+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":742,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:27.237430+0000","last_change":"2024-09-21T12:35:27.237430+0000","last_active":"2024-09-21T12:35:27.237430+0000","last_peered":"2024-09-21T12:35:27.237430+0000","last_clean":"2024-09-21T12:35:27.237430+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:35:27.237430+0000","last_undegraded":"2024-09-21T12:35:27.237430+0000","last_fullsized":"2024-09-21T12:35:27.237430+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:35:27.236980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:27.236980+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:49.341969+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":746,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.506669+0000","last_change":"2024-09-21T12:34:40.861569+0000","last_active":"2024-09-21T12:35:17.506669+0000","last_peered":"2024-09-21T12:35:17.506669+0000","last_clean":"2024-09-21T12:35:17.506669+0000","last_became_active":"2024-09-21T12:32:38.421807+0000","last_became_peered":"2024-09-21T12:32:38.421807+0000","last_unstale":"2024-09-21T12:35:17.506669+0000","last_undegraded":"2024-09-21T12:35:17.506669+0000","last_fullsized":"2024-09-21T12:35:17.506669+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:40.861493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:40.861493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:56.164524+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":738,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.503380+0000","last_change":"2024-09-21T12:34:34.347842+0000","last_active":"2024-09-21T12:35:17.503380+0000","last_peered":"2024-09-21T12:35:17.503380+0000","last_clean":"2024-09-21T12:34:29.282053+0000","last_became_active":"2024-09-21T12:34:34.347842+0000","last_became_peered":"2024-09-21T12:34:34.347842+0000","last_unstale":"2024-09-21T12:35:17.503380+0000","last_undegraded":"2024-09-21T12:35:17.503380+0000","last_fullsized":"2024-09-21T12:34:34.331116+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:15.385699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:34:15.385699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:29.045233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"76'79","reported_seq":117,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.499353+0000","last_change":"2024-09-21T12:34:43.304674+0000","last_active":"2024-09-21T12:35:17.499353+0000","last_peered":"2024-09-21T12:35:17.499353+0000","last_clean":"2024-09-21T12:35:17.499353+0000","last_became_active":"2024-09-21T12:32:38.432302+0000","last_became_peered":"2024-09-21T12:32:38.432302+0000","last_unstale":"2024-09-21T12:35:17.499353+0000","last_undegraded":"2024-09-21T12:35:17.499353+0000","last_fullsized":"2024-09-21T12:35:17.499353+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:43.304606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:43.304606+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:08.889287+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":734,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.503865+0000","last_change":"2024-09-21T12:32:53.520137+0000","last_active":"2024-09-21T12:35:17.503865+0000","last_peered":"2024-09-21T12:35:17.503865+0000","last_clean":"2024-09-21T12:35:17.503865+0000","last_became_active":"2024-09-21T12:32:38.426822+0000","last_became_peered":"2024-09-21T12:32:38.426822+0000","last_unstale":"2024-09-21T12:35:17.503865+0000","last_undegraded":"2024-09-21T12:35:17.503865+0000","last_fullsized":"2024-09-21T12:35:17.503865+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:53.520065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:32:53.520065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:30.273260+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":731,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.511076+0000","last_change":"2024-09-21T12:31:39.052864+0000","last_active":"2024-09-21T12:35:17.511076+0000","last_peered":"2024-09-21T12:35:17.511076+0000","last_clean":"2024-09-21T12:35:17.511076+0000","last_became_active":"2024-09-21T12:31:39.052486+0000","last_became_peered":"2024-09-21T12:31:39.052486+0000","last_unstale":"2024-09-21T12:35:17.511076+0000","last_undegraded":"2024-09-21T12:35:17.511076+0000","last_fullsized":"2024-09-21T12:35:17.511076+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:15.071234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:29:15.071234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:35:29.047015+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":754,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T12:35:17.514581+0000","last_change":"2024-09-21T12:34:44.919918+0000","last_active":"2024-09-21T12:35:17.514581+0000","last_peered":"2024-09-21T12:35:17.514581+0000","last_clean":"2024-09-21T12:35:17.514581+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:35:17.514581+0000","last_undegraded":"2024-09-21T12:35:17.514581+0000","last_fullsized":"2024-09-21T12:35:17.514581+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:34:44.919796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:34:44.919796+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:05.730220+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":741,"reported_epoch":398,"state":"active+undersized","last_fresh":"2024-09-21T12:35:17.507219+0000","last_change":"2024-09-21T12:34:34.337387+0000","last_active":"2024-09-21T12:35:17.507219+0000","last_peered":"2024-09-21T12:35:17.507219+0000","last_clean":"2024-09-21T12:34:29.290158+0000","last_became_active":"2024-09-21T12:34:34.337387+0000","last_became_peered":"2024-09-21T12:34:34.337387+0000","last_unstale":"2024-09-21T12:35:17.507219+0000","last_undegraded":"2024-09-21T12:35:17.507219+0000","last_fullsized":"2024-09-21T12:34:34.332508+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:36.112155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:33:36.112155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:13.145641+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":910557208,"num_objects":13895,"num_object_clones":0,"num_object_copies":41685,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13895,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":240205,"num_write_kb":8131137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1027,"num_bytes_recovered":67305472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4735250432,"data_stored":4735238216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71153,"ondisk_log_size":71153,"up":42,"acting":42,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":84,"acting":84,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93,"ondisk_log_size":93,"up":87,"acting":87,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":58,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1105,"num_read_kb":862,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":372,"num_bytes_recovered":7304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":685,"ondisk_log_size":685,"up":87,"acting":87,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1058,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":88,"acting":88,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":905,"num_read_kb":900,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2449,"ondisk_log_size":2449,"up":86,"acting":86,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234170,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":890796,"kb_used_data":237660,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":92854356,"statfs":{"total":95995035648,"available":95082860544,"internally_reserved":0,"allocated":243363840,"data_stored":243066369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":668793290},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463106,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1099292,"kb_used_data":454476,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":92645860,"statfs":{"total":95995035648,"available":94869360640,"internally_reserved":0,"allocated":465383424,"data_stored":465101168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":660273606},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724741,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":962608,"kb_used_data":327520,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":92782544,"statfs":{"total":95995035648,"available":95009325056,"internally_reserved":0,"allocated":335380480,"data_stored":335079849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953677,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":814452,"kb_used_data":522276,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":92930700,"statfs":{"total":95995035648,"available":95161036800,"internally_reserved":0,"allocated":534810624,"data_stored":534503961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149907,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1026108,"kb_used_data":296620,"kb_used_omap":1,"kb_used_meta":729470,"kb_avail":92719044,"statfs":{"total":95995035648,"available":94944301056,"internally_reserved":0,"allocated":303738880,"data_stored":303425606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":746977735},"hb_peers":[0,1,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411534,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2808848,"kb_used_data":2272320,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":90936304,"statfs":{"total":95995035648,"available":93118775296,"internally_reserved":0,"allocated":2326855680,"data_stored":2326574609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575071,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":900364,"kb_used_data":179196,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":92844788,"statfs":{"total":95995035648,"available":95073062912,"internally_reserved":0,"allocated":183496704,"data_stored":183166304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738597,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1285432,"kb_used_data":514280,"kb_used_omap":1,"kb_used_meta":771134,"kb_avail":92459720,"statfs":{"total":95995035648,"available":94678753280,"internally_reserved":0,"allocated":526622720,"data_stored":526313156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":789641669},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":510529536,"data_stored":510525464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":172818432,"data_stored":172818432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2320039936,"data_stored":2320039936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":276496384,"data_stored":276496384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":487788544,"data_stored":487784472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":302907392,"data_stored":302907392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":424153088,"data_stored":424149016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":240517120,"data_stored":240517120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:35:35.990 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T12:35:35.990 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pg_num 41 2024-09-21T12:35:35.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:35 smithi183 ceph-mon[33277]: pgmap v774: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 412 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.22k op/s; 2459/20367 objects degraded (12.073%) 2024-09-21T12:35:35.997 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:36.032 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:36.042 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:36.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:36.095 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:36.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:36.196 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:36.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:36.240 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:36.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:36.340 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:36.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:36.375 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:36.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:36.476 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:36.508 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:36.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:36.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:36.610 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:36.643 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:36.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:36.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:36.708 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:35:36.708 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:35:36.739 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:35:36.740 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:35:36.743 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T12:35:36.745 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:36.775 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:35:36.776 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:35:36.787 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:36.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:36.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:36.807 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:35:36.807 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:35:36.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:36 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1154762023' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:35:36.808 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:36 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1154762023' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:35:36.836 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Removed 127049 objects 2024-09-21T12:35:36.836 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Clean up completed and total clean up time :90.1562 2024-09-21T12:35:36.837 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:35:36.837 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:35:36.854 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:35:36.854 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:35:36.870 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:35:36.870 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:35:36.889 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:36.911 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:36.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:36.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:36.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:36 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1154762023' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:35:37.013 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:37.034 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:37.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:37.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:37.136 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:37.158 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:37.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:37.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:37.259 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:37.280 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:37.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:37.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:37.368 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:35:37.368 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:35:37.368 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:35:37.368 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:35:37.368 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:35:37.381 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:37.402 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:37.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:37.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:37.504 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:37.524 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T12:35:37.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:37.525 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:37.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:37.627 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:37.647 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:37.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:37.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:37.749 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:37.770 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:37.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:37.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:37.872 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:37.893 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:37.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:37.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:37.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:35:37 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:35:37.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:37 smithi183 ceph-mon[33277]: pgmap v775: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 412 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 838 B/s rd, 0 B/s wr, 1.20k op/s; 2459/20367 objects degraded (12.073%) 2024-09-21T12:35:37.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:37 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 2459/20367 objects degraded (12.073%), 20 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-09-21T12:35:37.995 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:38.016 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:38.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:38.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:38.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:37 smithi114 ceph-mon[30324]: pgmap v775: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 412 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 838 B/s rd, 0 B/s wr, 1.20k op/s; 2459/20367 objects degraded (12.073%) 2024-09-21T12:35:38.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:37 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 2459/20367 objects degraded (12.073%), 20 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-09-21T12:35:38.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:37 smithi114 ceph-mon[40867]: pgmap v775: 178 pgs: 32 active+undersized, 20 active+undersized+degraded, 126 active+clean; 412 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 838 B/s rd, 0 B/s wr, 1.20k op/s; 2459/20367 objects degraded (12.073%) 2024-09-21T12:35:38.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:37 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 2459/20367 objects degraded (12.073%), 20 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-09-21T12:35:38.117 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:38.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:38.139 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:38.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:38.227 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T12:35:38.227 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T12:35:38.239 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:38.262 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:38.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:38.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:38.363 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:38.385 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:38.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:38.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:38.487 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:38.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:38.509 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:38.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:38.610 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:38.631 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:38.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:38.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:38.659 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:38.689 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T12:35:38.732 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:38.755 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:38.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:38.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:38.857 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:38.879 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:38.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:38.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:38.906 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:38 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:35:38.906 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:38 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:35:38.980 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:39.004 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:39.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:39.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:39.105 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:39.129 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:39.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:39.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:39.218 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:39.226 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:39.230 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:39.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:38 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:35:39.253 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:39.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:39.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:39.321 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: specified pg_num 41 < pg_num_max 32 2024-09-21T12:35:39.355 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:39.376 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:39.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:39.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:39.478 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:39.500 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:39.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:39.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:39.601 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:39.623 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:39.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:39.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:39.724 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:39.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:39.747 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:39.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:39.848 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:39.854 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:39 smithi114 ceph-mon[30324]: pgmap v776: 178 pgs: 35 active+undersized, 17 active+undersized+degraded, 126 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.74k op/s; 63/597 objects degraded (10.553%) 2024-09-21T12:35:39.854 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:39 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1237866479' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"} : dispatch 2024-09-21T12:35:39.854 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:39 smithi114 ceph-mon[40867]: pgmap v776: 178 pgs: 35 active+undersized, 17 active+undersized+degraded, 126 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.74k op/s; 63/597 objects degraded (10.553%) 2024-09-21T12:35:39.854 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:39 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1237866479' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"} : dispatch 2024-09-21T12:35:39.869 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:39.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:39.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:39.971 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:39.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:39.993 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:39.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:40.094 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:40.115 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:40.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:40.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:40.217 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:40.228 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:35:40.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:39 smithi183 ceph-mon[33277]: pgmap v776: 178 pgs: 35 active+undersized, 17 active+undersized+degraded, 126 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.74k op/s; 63/597 objects degraded (10.553%) 2024-09-21T12:35:40.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:39 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1237866479' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"} : dispatch 2024-09-21T12:35:40.239 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:40.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:40.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:40.341 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:40.355 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:35:40.356 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 20, 2024-09-21T12:35:40.356 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 600, 2024-09-21T12:35:40.356 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-21T12:35:40.356 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.356 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231694 9.7 9.a814ca27 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T12:35:40.356 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:49.030271+0000", 2024-09-21T12:35:40.356 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 231.209881897, 2024-09-21T12:35:40.356 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0062914110000001, 2024-09-21T12:35:40.356 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:35:40.356 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:35:40.356 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:35:40.356 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:35:40.356 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:35:40.357 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:35:40.357 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231694 2024-09-21T12:35:40.357 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.357 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:35:40.357 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.357 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:35:40.357 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.030271+0000", 2024-09-21T12:35:40.357 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.357 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.357 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.357 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:35:40.357 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.030271+0000", 2024-09-21T12:35:40.357 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.357 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.357 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.357 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:35:40.358 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.030272+0000", 2024-09-21T12:35:40.358 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.747e-06 2024-09-21T12:35:40.358 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.358 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.358 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:35:40.358 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.030278+0000", 2024-09-21T12:35:40.358 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6500000000000001e-06 2024-09-21T12:35:40.358 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.358 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.358 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:35:40.358 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.030290+0000", 2024-09-21T12:35:40.358 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1887e-05 2024-09-21T12:35:40.358 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.358 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.358 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:35:40.359 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.030300+0000", 2024-09-21T12:35:40.359 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6360000000000006e-06 2024-09-21T12:35:40.359 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.359 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.359 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.359 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.035989+0000", 2024-09-21T12:35:40.359 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.005689637 2024-09-21T12:35:40.359 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.359 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.359 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:35:40.359 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.035999+0000", 2024-09-21T12:35:40.359 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.2369999999999998e-06 2024-09-21T12:35:40.359 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.359 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.360 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.360 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.036551+0000", 2024-09-21T12:35:40.360 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.000552042 2024-09-21T12:35:40.360 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.360 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.360 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:35:40.360 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.036562+0000", 2024-09-21T12:35:40.360 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1575e-05 2024-09-21T12:35:40.360 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.360 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:40.360 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.360 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.360 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.360 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231696 9.2 9.2c07a862 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T12:35:40.360 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:49.038118+0000", 2024-09-21T12:35:40.361 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 231.20203470300001, 2024-09-21T12:35:40.361 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99215267900000004, 2024-09-21T12:35:40.361 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:35:40.361 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:35:40.361 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:35:40.361 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:35:40.361 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:35:40.361 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:35:40.361 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231696 2024-09-21T12:35:40.361 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.361 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:35:40.361 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.361 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:35:40.361 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.038118+0000", 2024-09-21T12:35:40.361 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.362 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.362 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.362 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:35:40.362 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.038118+0000", 2024-09-21T12:35:40.362 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.362 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.362 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.362 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:35:40.362 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.038120+0000", 2024-09-21T12:35:40.362 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8309999999999999e-06 2024-09-21T12:35:40.362 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.362 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.362 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:35:40.362 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.038123+0000", 2024-09-21T12:35:40.363 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.275e-06 2024-09-21T12:35:40.363 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.363 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.363 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:35:40.363 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.038132+0000", 2024-09-21T12:35:40.363 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6999999999999997e-06 2024-09-21T12:35:40.363 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.363 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.363 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:35:40.363 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.038140+0000", 2024-09-21T12:35:40.363 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.7190000000000007e-06 2024-09-21T12:35:40.363 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.363 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.363 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.363 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.038187+0000", 2024-09-21T12:35:40.364 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6276000000000002e-05 2024-09-21T12:35:40.364 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.364 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.364 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:35:40.364 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.038205+0000", 2024-09-21T12:35:40.364 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7805999999999999e-05 2024-09-21T12:35:40.364 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.364 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.364 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.364 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.030259+0000", 2024-09-21T12:35:40.364 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99205480199999996 2024-09-21T12:35:40.364 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.364 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.364 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:35:40.364 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.030271+0000", 2024-09-21T12:35:40.365 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1270000000000001e-05 2024-09-21T12:35:40.365 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.365 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:40.365 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.365 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.365 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.365 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231697 9.19 9.7f5efd59 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T12:35:40.365 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:49.048873+0000", 2024-09-21T12:35:40.365 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 231.19127931, 2024-09-21T12:35:40.365 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98772340000000003, 2024-09-21T12:35:40.365 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:35:40.365 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:35:40.365 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:35:40.365 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:35:40.365 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:35:40.365 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:35:40.366 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231697 2024-09-21T12:35:40.366 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.366 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:35:40.366 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.366 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:35:40.366 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.048873+0000", 2024-09-21T12:35:40.366 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.366 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.366 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.366 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:35:40.366 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.048873+0000", 2024-09-21T12:35:40.366 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.366 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.366 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.366 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:35:40.367 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.048875+0000", 2024-09-21T12:35:40.367 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4589999999999999e-06 2024-09-21T12:35:40.367 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.367 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.367 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:35:40.367 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.048878+0000", 2024-09-21T12:35:40.367 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.7239999999999998e-06 2024-09-21T12:35:40.367 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.367 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.367 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:35:40.367 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.048887+0000", 2024-09-21T12:35:40.367 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9339999999999995e-06 2024-09-21T12:35:40.367 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.367 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.367 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:35:40.368 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.048894+0000", 2024-09-21T12:35:40.368 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7349999999999999e-06 2024-09-21T12:35:40.368 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.368 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.368 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.368 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.049014+0000", 2024-09-21T12:35:40.368 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011964900000000001 2024-09-21T12:35:40.368 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.368 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.368 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:35:40.368 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.049026+0000", 2024-09-21T12:35:40.368 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2199e-05 2024-09-21T12:35:40.368 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.368 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.368 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.368 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.036588+0000", 2024-09-21T12:35:40.368 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98756192399999998 2024-09-21T12:35:40.369 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.369 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.369 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:35:40.369 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.036597+0000", 2024-09-21T12:35:40.369 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.7760000000000003e-06 2024-09-21T12:35:40.369 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.369 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:40.369 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.369 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.369 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.369 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231698 9.8 9.f15efcc8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T12:35:40.369 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:49.048909+0000", 2024-09-21T12:35:40.369 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 231.19124340299999, 2024-09-21T12:35:40.369 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97906013199999997, 2024-09-21T12:35:40.370 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:35:40.370 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:35:40.370 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:35:40.370 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:35:40.370 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:35:40.370 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:35:40.370 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231698 2024-09-21T12:35:40.370 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.370 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:35:40.370 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.370 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:35:40.370 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.048909+0000", 2024-09-21T12:35:40.370 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.370 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.370 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.371 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:35:40.371 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.048909+0000", 2024-09-21T12:35:40.371 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.371 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.371 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.371 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:35:40.371 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.048910+0000", 2024-09-21T12:35:40.371 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.2599999999999998e-07 2024-09-21T12:35:40.371 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.371 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.371 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:35:40.371 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.048912+0000", 2024-09-21T12:35:40.371 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.576e-06 2024-09-21T12:35:40.371 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.371 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.372 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:35:40.372 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.048918+0000", 2024-09-21T12:35:40.372 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2530000000000001e-06 2024-09-21T12:35:40.372 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.372 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.372 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:35:40.372 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.048924+0000", 2024-09-21T12:35:40.372 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3290000000000001e-06 2024-09-21T12:35:40.372 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.372 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.372 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.372 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.048966+0000", 2024-09-21T12:35:40.372 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.2379999999999997e-05 2024-09-21T12:35:40.372 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.372 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.372 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:35:40.373 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.048976+0000", 2024-09-21T12:35:40.373 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.5060000000000001e-06 2024-09-21T12:35:40.373 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.373 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.373 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.373 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.027955+0000", 2024-09-21T12:35:40.373 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97897941399999999 2024-09-21T12:35:40.373 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.373 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.373 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:35:40.373 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.027969+0000", 2024-09-21T12:35:40.373 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4248e-05 2024-09-21T12:35:40.373 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.373 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:40.374 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.374 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.374 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.374 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231705 9.19 9.299f5179 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T12:35:40.374 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:49.052946+0000", 2024-09-21T12:35:40.374 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 231.18720636800001, 2024-09-21T12:35:40.374 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98368246000000004, 2024-09-21T12:35:40.374 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:35:40.374 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:35:40.374 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:35:40.374 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:35:40.374 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:35:40.374 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:35:40.375 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231705 2024-09-21T12:35:40.375 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.375 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:35:40.375 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.375 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:35:40.375 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.052946+0000", 2024-09-21T12:35:40.375 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.375 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.375 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.375 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:35:40.375 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.052946+0000", 2024-09-21T12:35:40.375 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.375 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.375 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.375 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:35:40.376 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.052948+0000", 2024-09-21T12:35:40.376 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2089999999999999e-06 2024-09-21T12:35:40.376 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.376 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.376 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:35:40.376 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.052955+0000", 2024-09-21T12:35:40.376 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7349999999999999e-06 2024-09-21T12:35:40.376 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.376 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.376 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:35:40.376 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.052968+0000", 2024-09-21T12:35:40.376 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3025e-05 2024-09-21T12:35:40.376 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.376 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.376 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:35:40.376 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.052977+0000", 2024-09-21T12:35:40.377 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5429999999999994e-06 2024-09-21T12:35:40.377 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.377 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.377 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.377 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.053216+0000", 2024-09-21T12:35:40.377 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00023908999999999999 2024-09-21T12:35:40.377 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.377 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.377 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:35:40.377 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.053225+0000", 2024-09-21T12:35:40.377 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.8300000000000002e-06 2024-09-21T12:35:40.377 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.377 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.377 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.377 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.036620+0000", 2024-09-21T12:35:40.377 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98339553899999999 2024-09-21T12:35:40.378 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.378 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.378 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:35:40.378 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.036629+0000", 2024-09-21T12:35:40.378 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.4889999999999995e-06 2024-09-21T12:35:40.378 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.378 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:40.378 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.378 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.378 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.378 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231712 9.2 9.fc322b82 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T12:35:40.378 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:49.057448+0000", 2024-09-21T12:35:40.378 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 231.18270462999999, 2024-09-21T12:35:40.378 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97285675500000002, 2024-09-21T12:35:40.378 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:35:40.379 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:35:40.379 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:35:40.379 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:35:40.379 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:35:40.379 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:35:40.379 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231712 2024-09-21T12:35:40.379 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.379 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:35:40.379 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.379 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:35:40.379 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.057448+0000", 2024-09-21T12:35:40.379 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.379 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.379 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.379 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:35:40.380 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.057448+0000", 2024-09-21T12:35:40.380 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.380 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.380 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.380 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:35:40.380 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.057450+0000", 2024-09-21T12:35:40.380 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5939999999999999e-06 2024-09-21T12:35:40.380 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.380 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.380 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:35:40.380 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.057454+0000", 2024-09-21T12:35:40.380 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6110000000000004e-06 2024-09-21T12:35:40.380 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.380 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.380 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:35:40.380 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.057468+0000", 2024-09-21T12:35:40.380 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4226999999999999e-05 2024-09-21T12:35:40.381 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.381 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.381 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:35:40.381 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.057476+0000", 2024-09-21T12:35:40.381 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4749999999999996e-06 2024-09-21T12:35:40.381 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.381 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.381 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.381 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.057676+0000", 2024-09-21T12:35:40.381 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000200571 2024-09-21T12:35:40.381 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.381 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.381 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:35:40.381 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.057687+0000", 2024-09-21T12:35:40.382 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0064999999999999e-05 2024-09-21T12:35:40.382 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.382 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.382 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.382 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.030294+0000", 2024-09-21T12:35:40.382 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97260765699999996 2024-09-21T12:35:40.382 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.382 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.382 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:35:40.382 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.030305+0000", 2024-09-21T12:35:40.382 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0555e-05 2024-09-21T12:35:40.382 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.382 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:40.382 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.383 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.383 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.383 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:232383 9.2 9:4387765c:::benchmark_data_smithi114_133214_object108349:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-09-21T12:35:40.383 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:53.083027+0000", 2024-09-21T12:35:40.383 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 227.15712556299999, 2024-09-21T12:35:40.383 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99262139299999996, 2024-09-21T12:35:40.383 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:35:40.383 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:35:40.383 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:35:40.383 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:35:40.383 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:35:40.383 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:35:40.383 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 232383 2024-09-21T12:35:40.383 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.383 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:35:40.384 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.384 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:35:40.384 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.083027+0000", 2024-09-21T12:35:40.384 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.384 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.384 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.384 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:35:40.384 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.083027+0000", 2024-09-21T12:35:40.384 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.384 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.384 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.384 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:35:40.384 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.083028+0000", 2024-09-21T12:35:40.384 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7100000000000011e-07 2024-09-21T12:35:40.384 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.384 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.385 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:35:40.385 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.083037+0000", 2024-09-21T12:35:40.385 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.8789999999999992e-06 2024-09-21T12:35:40.385 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.385 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.385 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:35:40.385 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.083052+0000", 2024-09-21T12:35:40.385 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5258e-05 2024-09-21T12:35:40.385 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.385 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.385 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:35:40.385 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.083060+0000", 2024-09-21T12:35:40.385 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6909999999999997e-06 2024-09-21T12:35:40.385 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.385 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.385 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.386 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.083167+0000", 2024-09-21T12:35:40.386 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000107389 2024-09-21T12:35:40.386 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.386 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.386 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:35:40.386 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.083176+0000", 2024-09-21T12:35:40.386 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6009999999999995e-06 2024-09-21T12:35:40.386 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.386 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.386 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.386 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.068403+0000", 2024-09-21T12:35:40.386 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98522679700000004 2024-09-21T12:35:40.386 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.386 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.387 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:35:40.387 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.068560+0000", 2024-09-21T12:35:40.387 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015688099999999999 2024-09-21T12:35:40.387 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.387 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.387 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,2", 2024-09-21T12:35:40.387 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.068855+0000", 2024-09-21T12:35:40.387 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000295708 2024-09-21T12:35:40.387 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.387 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.387 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:40.387 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.075433+0000", 2024-09-21T12:35:40.387 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0065781720000000002 2024-09-21T12:35:40.387 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.388 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.388 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:40.388 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.075494+0000", 2024-09-21T12:35:40.388 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0723000000000001e-05 2024-09-21T12:35:40.388 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.388 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.388 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:35:40.388 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.075550+0000", 2024-09-21T12:35:40.388 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5813999999999998e-05 2024-09-21T12:35:40.388 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.388 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.388 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:35:40.388 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.075601+0000", 2024-09-21T12:35:40.388 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1248999999999997e-05 2024-09-21T12:35:40.389 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.389 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.389 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:35:40.389 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.075648+0000", 2024-09-21T12:35:40.389 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7259999999999998e-05 2024-09-21T12:35:40.389 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.389 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:40.389 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.389 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.389 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.389 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:232402 9.2 9:42f7e5a1:::benchmark_data_smithi114_133214_object108368:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-09-21T12:35:40.389 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:53.091828+0000", 2024-09-21T12:35:40.389 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 227.148325027, 2024-09-21T12:35:40.389 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98406757600000005, 2024-09-21T12:35:40.390 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:35:40.390 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:35:40.390 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:35:40.390 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:35:40.390 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:35:40.390 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:35:40.390 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 232402 2024-09-21T12:35:40.390 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.390 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:35:40.390 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.390 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:35:40.390 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.091828+0000", 2024-09-21T12:35:40.390 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.390 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.391 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.391 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:35:40.391 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.091828+0000", 2024-09-21T12:35:40.391 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.391 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.391 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.391 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:35:40.391 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.091829+0000", 2024-09-21T12:35:40.391 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7850000000000001e-06 2024-09-21T12:35:40.391 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.391 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.391 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:35:40.391 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.091835+0000", 2024-09-21T12:35:40.391 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2020000000000003e-06 2024-09-21T12:35:40.391 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.392 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.392 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:35:40.392 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.091850+0000", 2024-09-21T12:35:40.392 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5577e-05 2024-09-21T12:35:40.392 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.392 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.392 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:35:40.392 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.091858+0000", 2024-09-21T12:35:40.392 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1049999999999995e-06 2024-09-21T12:35:40.392 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.392 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.392 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.392 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.091905+0000", 2024-09-21T12:35:40.392 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6387999999999997e-05 2024-09-21T12:35:40.393 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.393 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.393 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:35:40.393 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.091916+0000", 2024-09-21T12:35:40.393 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1379999999999999e-05 2024-09-21T12:35:40.393 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.393 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.393 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.393 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.071179+0000", 2024-09-21T12:35:40.393 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97926264600000001 2024-09-21T12:35:40.393 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.393 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.393 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:35:40.393 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.071281+0000", 2024-09-21T12:35:40.393 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010245499999999999 2024-09-21T12:35:40.394 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.394 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.394 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,2", 2024-09-21T12:35:40.394 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.071533+0000", 2024-09-21T12:35:40.394 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00025137599999999999 2024-09-21T12:35:40.394 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.394 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.394 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:40.394 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.075539+0000", 2024-09-21T12:35:40.394 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.004006262 2024-09-21T12:35:40.394 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.394 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.394 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:40.394 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.075698+0000", 2024-09-21T12:35:40.395 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015942800000000001 2024-09-21T12:35:40.395 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.395 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.395 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:35:40.395 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.075818+0000", 2024-09-21T12:35:40.395 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000119462 2024-09-21T12:35:40.395 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.395 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.395 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:35:40.395 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.075861+0000", 2024-09-21T12:35:40.395 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3297000000000002e-05 2024-09-21T12:35:40.395 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.395 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.395 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:35:40.395 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.075895+0000", 2024-09-21T12:35:40.396 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4212999999999999e-05 2024-09-21T12:35:40.396 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.396 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:40.396 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.396 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.396 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.396 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:232405 9.8 9:14aa08f4:::benchmark_data_smithi114_133214_object108371:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-09-21T12:35:40.396 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:53.094407+0000", 2024-09-21T12:35:40.396 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 227.145745957, 2024-09-21T12:35:40.396 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98245775499999999, 2024-09-21T12:35:40.396 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:35:40.396 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:35:40.396 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:35:40.396 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:35:40.396 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:35:40.396 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:35:40.397 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 232405 2024-09-21T12:35:40.397 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.397 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:35:40.397 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.397 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:35:40.397 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.094407+0000", 2024-09-21T12:35:40.397 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.397 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.397 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.397 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:35:40.397 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.094407+0000", 2024-09-21T12:35:40.397 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.397 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.397 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.397 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:35:40.397 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.094408+0000", 2024-09-21T12:35:40.397 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4780000000000001e-06 2024-09-21T12:35:40.398 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.398 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.398 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:35:40.398 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.094413+0000", 2024-09-21T12:35:40.398 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2619999999999998e-06 2024-09-21T12:35:40.398 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.398 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.398 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:35:40.398 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.094425+0000", 2024-09-21T12:35:40.398 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2004000000000001e-05 2024-09-21T12:35:40.398 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.398 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.398 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:35:40.398 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.094443+0000", 2024-09-21T12:35:40.398 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7146000000000002e-05 2024-09-21T12:35:40.399 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.399 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.399 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.399 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.094606+0000", 2024-09-21T12:35:40.399 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000163447 2024-09-21T12:35:40.399 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.399 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.399 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:35:40.399 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.094613+0000", 2024-09-21T12:35:40.399 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6640000000000001e-06 2024-09-21T12:35:40.399 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.399 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.400 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.400 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.068722+0000", 2024-09-21T12:35:40.400 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97410885899999999 2024-09-21T12:35:40.400 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.400 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.400 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:35:40.400 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.068829+0000", 2024-09-21T12:35:40.400 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010755 2024-09-21T12:35:40.400 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.400 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.400 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,7", 2024-09-21T12:35:40.400 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.069136+0000", 2024-09-21T12:35:40.400 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00030712600000000001 2024-09-21T12:35:40.400 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.401 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.401 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:40.401 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.076551+0000", 2024-09-21T12:35:40.401 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0074152039999999999 2024-09-21T12:35:40.401 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.401 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.401 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:35:40.401 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.076617+0000", 2024-09-21T12:35:40.401 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5560000000000002e-05 2024-09-21T12:35:40.401 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.401 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.401 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:40.401 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.076769+0000", 2024-09-21T12:35:40.401 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000151996 2024-09-21T12:35:40.401 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.402 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.402 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:35:40.402 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.076827+0000", 2024-09-21T12:35:40.402 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8505999999999999e-05 2024-09-21T12:35:40.402 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.402 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.402 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:35:40.402 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.076864+0000", 2024-09-21T12:35:40.402 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.6952999999999997e-05 2024-09-21T12:35:40.402 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.402 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:40.402 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.402 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.402 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.402 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:232419 9.8 9:15947b8c:::benchmark_data_smithi114_133214_object108385:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-09-21T12:35:40.403 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:53.103794+0000", 2024-09-21T12:35:40.403 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 227.13635832700001, 2024-09-21T12:35:40.403 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97848711399999999, 2024-09-21T12:35:40.403 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:35:40.403 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:35:40.403 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:35:40.403 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:35:40.403 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:35:40.403 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:35:40.403 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 232419 2024-09-21T12:35:40.403 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.403 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:35:40.403 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.403 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:35:40.403 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.103794+0000", 2024-09-21T12:35:40.404 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.404 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.404 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.404 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:35:40.404 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.103794+0000", 2024-09-21T12:35:40.404 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.404 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.404 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.404 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:35:40.404 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.103796+0000", 2024-09-21T12:35:40.404 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6980000000000001e-06 2024-09-21T12:35:40.404 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.404 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.404 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:35:40.405 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.103801+0000", 2024-09-21T12:35:40.405 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2789999999999998e-06 2024-09-21T12:35:40.405 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.405 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.405 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:35:40.405 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.103815+0000", 2024-09-21T12:35:40.405 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3701e-05 2024-09-21T12:35:40.405 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.405 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.405 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:35:40.405 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.103823+0000", 2024-09-21T12:35:40.405 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6210000000000004e-06 2024-09-21T12:35:40.405 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.405 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.405 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.405 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.103890+0000", 2024-09-21T12:35:40.405 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7819999999999998e-05 2024-09-21T12:35:40.406 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.406 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.406 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:35:40.406 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.103899+0000", 2024-09-21T12:35:40.406 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6459999999999999e-06 2024-09-21T12:35:40.406 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.406 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.406 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.406 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.072738+0000", 2024-09-21T12:35:40.406 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.96883906600000003 2024-09-21T12:35:40.406 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.406 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.406 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:35:40.406 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.072838+0000", 2024-09-21T12:35:40.407 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.9920999999999994e-05 2024-09-21T12:35:40.407 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.407 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.407 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,7", 2024-09-21T12:35:40.407 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.073310+0000", 2024-09-21T12:35:40.407 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00047227 2024-09-21T12:35:40.407 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.407 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.407 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:40.407 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.076603+0000", 2024-09-21T12:35:40.407 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003292868 2024-09-21T12:35:40.407 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.407 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.407 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:35:40.407 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.076707+0000", 2024-09-21T12:35:40.407 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000104107 2024-09-21T12:35:40.408 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.408 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.408 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:40.408 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.082196+0000", 2024-09-21T12:35:40.408 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0054885790000000004 2024-09-21T12:35:40.408 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.408 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.408 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:35:40.408 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.082240+0000", 2024-09-21T12:35:40.408 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4022e-05 2024-09-21T12:35:40.408 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.408 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.408 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:35:40.408 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.082281+0000", 2024-09-21T12:35:40.409 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1516e-05 2024-09-21T12:35:40.409 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.409 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:40.409 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.409 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.409 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.409 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:232425 9.8 9:10af9b7e:::benchmark_data_smithi114_133214_object108391:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-09-21T12:35:40.409 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:53.108407+0000", 2024-09-21T12:35:40.409 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 227.13174583700001, 2024-09-21T12:35:40.409 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97965131900000002, 2024-09-21T12:35:40.409 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:35:40.409 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:35:40.409 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:35:40.409 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:35:40.409 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:35:40.409 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:35:40.410 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 232425 2024-09-21T12:35:40.410 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.410 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:35:40.410 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.410 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:35:40.410 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.108407+0000", 2024-09-21T12:35:40.410 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.410 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.410 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.410 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:35:40.410 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.108407+0000", 2024-09-21T12:35:40.410 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.410 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.410 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.410 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:35:40.411 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.108407+0000", 2024-09-21T12:35:40.411 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8299999999999997e-07 2024-09-21T12:35:40.411 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.411 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.411 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:35:40.411 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.108410+0000", 2024-09-21T12:35:40.411 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.4399999999999999e-06 2024-09-21T12:35:40.411 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.411 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.411 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:35:40.411 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.108416+0000", 2024-09-21T12:35:40.411 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2339999999999999e-06 2024-09-21T12:35:40.411 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.411 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.411 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:35:40.412 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.108421+0000", 2024-09-21T12:35:40.412 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2320000000000001e-06 2024-09-21T12:35:40.412 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.412 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.412 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.412 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.108474+0000", 2024-09-21T12:35:40.412 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2920999999999997e-05 2024-09-21T12:35:40.412 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.412 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.412 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:35:40.412 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.108482+0000", 2024-09-21T12:35:40.412 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6760000000000007e-06 2024-09-21T12:35:40.412 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.412 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.412 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.412 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.074870+0000", 2024-09-21T12:35:40.413 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.96638845799999995 2024-09-21T12:35:40.413 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.413 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.413 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:35:40.413 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.074975+0000", 2024-09-21T12:35:40.413 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000104726 2024-09-21T12:35:40.413 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.413 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.413 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,7", 2024-09-21T12:35:40.413 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.075237+0000", 2024-09-21T12:35:40.413 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000261616 2024-09-21T12:35:40.413 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.413 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.413 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:40.413 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.082317+0000", 2024-09-21T12:35:40.413 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0070807320000000002 2024-09-21T12:35:40.413 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.414 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.414 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:40.414 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.082353+0000", 2024-09-21T12:35:40.414 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5542e-05 2024-09-21T12:35:40.414 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.414 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.414 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:35:40.414 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.087953+0000", 2024-09-21T12:35:40.414 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0056001999999999996 2024-09-21T12:35:40.414 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.414 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.414 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:35:40.414 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.088000+0000", 2024-09-21T12:35:40.414 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6940999999999997e-05 2024-09-21T12:35:40.414 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.415 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.415 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:35:40.415 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:54.088058+0000", 2024-09-21T12:35:40.415 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8118000000000002e-05 2024-09-21T12:35:40.415 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.415 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:40.415 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.415 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.415 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.415 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.15924.0:123252 10.7 10:e0f4bf27:::benchmark_data_smithi114_161690_object123251:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T12:35:40.415 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:34:02.987833+0000", 2024-09-21T12:35:40.415 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 97.252320065000006, 2024-09-21T12:35:40.415 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.10727073500000001, 2024-09-21T12:35:40.415 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:35:40.416 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:35:40.416 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:35:40.416 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:35:40.416 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.15924", 2024-09-21T12:35:40.416 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:35:40.416 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 123252 2024-09-21T12:35:40.416 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.416 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:35:40.416 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.416 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:35:40.416 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:02.987833+0000", 2024-09-21T12:35:40.416 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.416 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.417 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.417 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:35:40.417 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:02.987833+0000", 2024-09-21T12:35:40.417 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.417 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.417 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.417 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:35:40.417 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:02.987835+0000", 2024-09-21T12:35:40.417 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1909999999999999e-06 2024-09-21T12:35:40.417 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.417 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.417 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:35:40.417 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:02.987858+0000", 2024-09-21T12:35:40.417 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3519e-05 2024-09-21T12:35:40.417 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.417 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.418 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:35:40.418 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:02.987870+0000", 2024-09-21T12:35:40.418 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1647999999999999e-05 2024-09-21T12:35:40.418 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.418 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.418 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:35:40.418 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:02.987878+0000", 2024-09-21T12:35:40.418 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.2549999999999997e-06 2024-09-21T12:35:40.418 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.418 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.418 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.418 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:03.080961+0000", 2024-09-21T12:35:40.418 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.093083162999999997 2024-09-21T12:35:40.418 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.418 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.418 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:35:40.419 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:03.081123+0000", 2024-09-21T12:35:40.419 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016159900000000001 2024-09-21T12:35:40.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.419 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.419 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 5,6", 2024-09-21T12:35:40.419 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:03.081277+0000", 2024-09-21T12:35:40.419 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000154382 2024-09-21T12:35:40.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.419 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.419 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:40.419 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:03.086307+0000", 2024-09-21T12:35:40.419 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.005029609 2024-09-21T12:35:40.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.419 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.419 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:35:40.419 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:03.094848+0000", 2024-09-21T12:35:40.420 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0085411689999999995 2024-09-21T12:35:40.420 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.420 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.420 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:40.420 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:03.094965+0000", 2024-09-21T12:35:40.420 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011661400000000001 2024-09-21T12:35:40.420 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.420 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.420 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:35:40.420 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:03.095025+0000", 2024-09-21T12:35:40.420 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0597000000000003e-05 2024-09-21T12:35:40.420 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.420 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.420 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:35:40.420 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:03.095103+0000", 2024-09-21T12:35:40.420 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7989000000000002e-05 2024-09-21T12:35:40.421 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.421 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:40.421 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.421 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.421 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.421 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.15924.0:123257 10.7 10:eb1a1d3d:::benchmark_data_smithi114_161690_object123256:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T12:35:40.421 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:34:02.989369+0000", 2024-09-21T12:35:40.421 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 97.250783815999995, 2024-09-21T12:35:40.421 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.106037646, 2024-09-21T12:35:40.421 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:35:40.421 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:35:40.421 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:35:40.421 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:35:40.421 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.15924", 2024-09-21T12:35:40.421 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:35:40.421 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 123257 2024-09-21T12:35:40.422 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.422 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:35:40.422 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.422 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:35:40.422 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:02.989369+0000", 2024-09-21T12:35:40.422 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.422 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.422 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.422 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:35:40.422 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:02.989369+0000", 2024-09-21T12:35:40.422 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.422 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.422 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.422 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:35:40.423 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:02.989370+0000", 2024-09-21T12:35:40.423 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.184e-06 2024-09-21T12:35:40.423 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.423 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.423 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:35:40.423 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:02.989417+0000", 2024-09-21T12:35:40.423 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6788e-05 2024-09-21T12:35:40.423 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.423 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.423 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:35:40.423 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:02.989426+0000", 2024-09-21T12:35:40.423 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.8769999999999999e-06 2024-09-21T12:35:40.423 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.423 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.424 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:35:40.424 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:02.989432+0000", 2024-09-21T12:35:40.424 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4729999999999997e-06 2024-09-21T12:35:40.424 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.424 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.424 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.424 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:03.086338+0000", 2024-09-21T12:35:40.424 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.096906081000000005 2024-09-21T12:35:40.424 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.424 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.424 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:35:40.424 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:03.086485+0000", 2024-09-21T12:35:40.424 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014634500000000001 2024-09-21T12:35:40.424 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.424 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.425 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 5,6", 2024-09-21T12:35:40.425 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:03.086635+0000", 2024-09-21T12:35:40.425 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015028 2024-09-21T12:35:40.425 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.425 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.425 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:35:40.425 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:03.095199+0000", 2024-09-21T12:35:40.425 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0085638610000000007 2024-09-21T12:35:40.425 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.425 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.425 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:40.425 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:03.095244+0000", 2024-09-21T12:35:40.425 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5000999999999998e-05 2024-09-21T12:35:40.425 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.425 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.426 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:40.426 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:03.095304+0000", 2024-09-21T12:35:40.426 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0680000000000002e-05 2024-09-21T12:35:40.426 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.426 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.426 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:35:40.426 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:03.095357+0000", 2024-09-21T12:35:40.426 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3041000000000003e-05 2024-09-21T12:35:40.426 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.426 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.426 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:35:40.426 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:03.095406+0000", 2024-09-21T12:35:40.426 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9035000000000001e-05 2024-09-21T12:35:40.426 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.426 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:40.426 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.427 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.427 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.427 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.15924.0:123263 10.f e380/344 10:f8e87803:::benchmark_data_smithi114_161690_object123262:head v 380'7605, mlcod=380'7603)", 2024-09-21T12:35:40.427 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:34:02.995057+0000", 2024-09-21T12:35:40.427 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 97.245095264, 2024-09-21T12:35:40.427 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.099868359000000004, 2024-09-21T12:35:40.427 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:35:40.427 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:35:40.427 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:35:40.427 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:35:40.427 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.427 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:35:40.427 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:02.995057+0000", 2024-09-21T12:35:40.427 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.427 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.427 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.427 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:35:40.428 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:02.995057+0000", 2024-09-21T12:35:40.428 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.428 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.428 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.428 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:35:40.428 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:02.995059+0000", 2024-09-21T12:35:40.428 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.198e-06 2024-09-21T12:35:40.428 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.428 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.428 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:35:40.428 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:02.995099+0000", 2024-09-21T12:35:40.428 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0188999999999998e-05 2024-09-21T12:35:40.428 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.428 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.428 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:35:40.429 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:03.061721+0000", 2024-09-21T12:35:40.429 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.066622418000000003 2024-09-21T12:35:40.429 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.429 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.429 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:35:40.429 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:03.061731+0000", 2024-09-21T12:35:40.429 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.978e-06 2024-09-21T12:35:40.429 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.429 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.429 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.429 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:03.080906+0000", 2024-09-21T12:35:40.429 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.019174574 2024-09-21T12:35:40.429 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.429 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.430 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:35:40.430 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:03.080923+0000", 2024-09-21T12:35:40.430 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7229000000000001e-05 2024-09-21T12:35:40.430 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.430 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.430 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:35:40.430 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:03.094850+0000", 2024-09-21T12:35:40.430 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.013927442999999999 2024-09-21T12:35:40.430 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.430 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.430 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:35:40.430 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:03.094926+0000", 2024-09-21T12:35:40.430 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5329999999999999e-05 2024-09-21T12:35:40.430 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.430 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:40.430 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.431 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.431 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.431 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.15924.0:160279 10.3 10:c0d2ee74:::benchmark_data_smithi114_161690_object33227:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-21T12:35:40.431 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:34:33.315863+0000", 2024-09-21T12:35:40.431 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 66.924290024000001, 2024-09-21T12:35:40.431 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.039515446, 2024-09-21T12:35:40.431 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:35:40.431 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:35:40.431 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:35:40.431 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:35:40.431 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.15924", 2024-09-21T12:35:40.431 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:35:40.431 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 160279 2024-09-21T12:35:40.431 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.431 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:35:40.432 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.432 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:35:40.432 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.315863+0000", 2024-09-21T12:35:40.432 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.432 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.432 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.432 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:35:40.432 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.315863+0000", 2024-09-21T12:35:40.432 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.432 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.432 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.432 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:35:40.432 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.315864+0000", 2024-09-21T12:35:40.432 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.79e-06 2024-09-21T12:35:40.432 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.432 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.433 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:35:40.433 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.315869+0000", 2024-09-21T12:35:40.433 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.023e-06 2024-09-21T12:35:40.433 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.433 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.433 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:35:40.433 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.315880+0000", 2024-09-21T12:35:40.433 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0669999999999999e-05 2024-09-21T12:35:40.433 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.433 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.433 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:35:40.433 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.315887+0000", 2024-09-21T12:35:40.433 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9059999999999996e-06 2024-09-21T12:35:40.434 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.434 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.434 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.434 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.316107+0000", 2024-09-21T12:35:40.434 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00021952500000000001 2024-09-21T12:35:40.434 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.434 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.434 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:35:40.434 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.318166+0000", 2024-09-21T12:35:40.434 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0020598589999999998 2024-09-21T12:35:40.434 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.434 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.434 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,5", 2024-09-21T12:35:40.434 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.318290+0000", 2024-09-21T12:35:40.434 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012345499999999999 2024-09-21T12:35:40.435 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.435 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.435 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:35:40.435 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.321390+0000", 2024-09-21T12:35:40.435 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003100106 2024-09-21T12:35:40.435 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.435 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.435 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:40.435 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.324999+0000", 2024-09-21T12:35:40.435 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003608797 2024-09-21T12:35:40.435 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.435 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.435 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.435 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.349181+0000", 2024-09-21T12:35:40.435 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.024181904000000001 2024-09-21T12:35:40.435 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.436 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.436 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:35:40.436 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.349196+0000", 2024-09-21T12:35:40.436 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5376999999999999e-05 2024-09-21T12:35:40.436 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.436 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.436 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.436 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:34.355313+0000", 2024-09-21T12:35:40.436 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0061168730000001 2024-09-21T12:35:40.436 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.436 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.436 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:35:40.436 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:34.355378+0000", 2024-09-21T12:35:40.436 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5160999999999995e-05 2024-09-21T12:35:40.436 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.436 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:40.437 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.437 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.437 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.437 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.15924.0:160290 10.3 10:cb813957:::benchmark_data_smithi114_161690_object33238:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-21T12:35:40.437 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:34:33.324912+0000", 2024-09-21T12:35:40.437 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 66.915240978, 2024-09-21T12:35:40.437 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0308570189999999, 2024-09-21T12:35:40.437 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:35:40.437 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:35:40.437 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:35:40.437 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:35:40.437 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.15924", 2024-09-21T12:35:40.437 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:35:40.437 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 160290 2024-09-21T12:35:40.437 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.437 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:35:40.438 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.438 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:35:40.438 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.324912+0000", 2024-09-21T12:35:40.438 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.438 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.438 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.438 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:35:40.438 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.324912+0000", 2024-09-21T12:35:40.438 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.438 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.438 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.438 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:35:40.438 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.324912+0000", 2024-09-21T12:35:40.438 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5499999999999999e-07 2024-09-21T12:35:40.438 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.439 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.439 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:35:40.439 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.324914+0000", 2024-09-21T12:35:40.439 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.069e-06 2024-09-21T12:35:40.439 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.439 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.439 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:35:40.439 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.324922+0000", 2024-09-21T12:35:40.439 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.0160000000000008e-06 2024-09-21T12:35:40.439 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.439 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.439 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:35:40.439 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.324927+0000", 2024-09-21T12:35:40.439 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6890000000000003e-06 2024-09-21T12:35:40.440 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.440 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.440 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.440 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.325034+0000", 2024-09-21T12:35:40.440 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000107458 2024-09-21T12:35:40.440 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.440 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.440 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:35:40.440 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.325150+0000", 2024-09-21T12:35:40.440 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000116106 2024-09-21T12:35:40.440 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.440 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.440 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,5", 2024-09-21T12:35:40.440 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.325264+0000", 2024-09-21T12:35:40.441 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011396300000000001 2024-09-21T12:35:40.441 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.441 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.441 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:35:40.441 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.346903+0000", 2024-09-21T12:35:40.441 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.021638654 2024-09-21T12:35:40.441 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.441 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.441 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:40.441 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.347040+0000", 2024-09-21T12:35:40.441 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013753800000000001 2024-09-21T12:35:40.441 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.441 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.441 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.441 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.349278+0000", 2024-09-21T12:35:40.441 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0022378799999999998 2024-09-21T12:35:40.442 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.442 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.442 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:35:40.442 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.349285+0000", 2024-09-21T12:35:40.442 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0570000000000004e-06 2024-09-21T12:35:40.442 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.442 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.442 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.442 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:34.355727+0000", 2024-09-21T12:35:40.442 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0064413080000001 2024-09-21T12:35:40.442 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.442 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.442 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:35:40.442 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:34.355769+0000", 2024-09-21T12:35:40.442 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1925999999999998e-05 2024-09-21T12:35:40.443 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.443 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:40.443 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.443 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.443 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.443 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.15924.0:160292 10.3 10:c093ed0c:::benchmark_data_smithi114_161690_object33240:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-21T12:35:40.443 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:34:33.326250+0000", 2024-09-21T12:35:40.443 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 66.913902594000007, 2024-09-21T12:35:40.443 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.029599913, 2024-09-21T12:35:40.443 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:35:40.443 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:35:40.443 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:35:40.443 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:35:40.443 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.15924", 2024-09-21T12:35:40.443 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:35:40.444 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 160292 2024-09-21T12:35:40.444 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.444 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:35:40.444 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.444 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:35:40.444 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.326250+0000", 2024-09-21T12:35:40.444 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.444 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.444 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.444 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:35:40.444 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.326250+0000", 2024-09-21T12:35:40.444 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.444 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.444 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.444 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:35:40.445 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.326252+0000", 2024-09-21T12:35:40.445 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5030000000000001e-06 2024-09-21T12:35:40.445 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.445 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.445 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:35:40.445 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.326256+0000", 2024-09-21T12:35:40.445 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7269999999999998e-06 2024-09-21T12:35:40.445 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.445 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.445 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:35:40.445 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.326271+0000", 2024-09-21T12:35:40.445 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5158e-05 2024-09-21T12:35:40.445 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.445 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.445 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:35:40.445 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.326279+0000", 2024-09-21T12:35:40.446 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1690000000000003e-06 2024-09-21T12:35:40.446 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.446 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.446 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.446 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.329114+0000", 2024-09-21T12:35:40.446 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0028352210000000002 2024-09-21T12:35:40.446 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.446 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.446 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:35:40.446 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.329218+0000", 2024-09-21T12:35:40.446 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000104361 2024-09-21T12:35:40.446 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.446 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.446 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,5", 2024-09-21T12:35:40.446 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.329327+0000", 2024-09-21T12:35:40.446 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010876500000000001 2024-09-21T12:35:40.447 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.447 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.447 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:35:40.447 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.346991+0000", 2024-09-21T12:35:40.447 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.017664279000000001 2024-09-21T12:35:40.447 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.447 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.447 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.447 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.349314+0000", 2024-09-21T12:35:40.447 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0023231319999999999 2024-09-21T12:35:40.447 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.447 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.447 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:35:40.447 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.349321+0000", 2024-09-21T12:35:40.447 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7440000000000001e-06 2024-09-21T12:35:40.447 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.448 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.448 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.448 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:34.355801+0000", 2024-09-21T12:35:40.448 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.00647943 2024-09-21T12:35:40.448 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.448 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.448 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:35:40.448 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:34.355850+0000", 2024-09-21T12:35:40.448 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9424e-05 2024-09-21T12:35:40.448 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.448 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:40.448 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.448 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.448 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.448 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.14478.0:3150 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 94620848899072 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-21T12:35:40.448 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:34:33.327019+0000", 2024-09-21T12:35:40.449 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 66.913133841000004, 2024-09-21T12:35:40.449 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0195015810000001, 2024-09-21T12:35:40.449 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:35:40.449 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:35:40.449 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:35:40.449 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:35:40.449 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.14478", 2024-09-21T12:35:40.449 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:35:40.449 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 3150 2024-09-21T12:35:40.449 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.449 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:35:40.449 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.449 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:35:40.449 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.327019+0000", 2024-09-21T12:35:40.449 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.449 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.449 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.450 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:35:40.450 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.327019+0000", 2024-09-21T12:35:40.450 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.450 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.450 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.450 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:35:40.450 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.327025+0000", 2024-09-21T12:35:40.450 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1580000000000002e-06 2024-09-21T12:35:40.450 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.450 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.450 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:35:40.450 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.327031+0000", 2024-09-21T12:35:40.450 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9220000000000002e-06 2024-09-21T12:35:40.450 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.451 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.451 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:35:40.451 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.327042+0000", 2024-09-21T12:35:40.451 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0806e-05 2024-09-21T12:35:40.451 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.451 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.451 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:35:40.451 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.327049+0000", 2024-09-21T12:35:40.451 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9750000000000001e-06 2024-09-21T12:35:40.451 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.451 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.451 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.451 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.327567+0000", 2024-09-21T12:35:40.451 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00051809600000000001 2024-09-21T12:35:40.452 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.452 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.452 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op must wait for map", 2024-09-21T12:35:40.452 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.327576+0000", 2024-09-21T12:35:40.452 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7229999999999999e-06 2024-09-21T12:35:40.452 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.452 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.452 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.452 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.345947+0000", 2024-09-21T12:35:40.452 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.018370653000000001 2024-09-21T12:35:40.452 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.452 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.452 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:35:40.452 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.345957+0000", 2024-09-21T12:35:40.452 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6709999999999994e-06 2024-09-21T12:35:40.452 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.453 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.453 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.453 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:34.344161+0000", 2024-09-21T12:35:40.453 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99820439699999997 2024-09-21T12:35:40.453 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.453 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.453 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:35:40.453 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:34.344357+0000", 2024-09-21T12:35:40.453 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000196176 2024-09-21T12:35:40.453 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.453 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.453 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:35:40.453 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:34.346520+0000", 2024-09-21T12:35:40.453 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0021630040000000001 2024-09-21T12:35:40.453 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.454 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:40.454 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.454 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.454 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.454 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.15924.0:160297 10.3 10:c4bf865c:::benchmark_data_smithi114_161690_object33245:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-21T12:35:40.454 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:34:33.350814+0000", 2024-09-21T12:35:40.454 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 66.889338727999998, 2024-09-21T12:35:40.454 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0106960110000001, 2024-09-21T12:35:40.454 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:35:40.454 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:35:40.454 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:35:40.454 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:35:40.454 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.15924", 2024-09-21T12:35:40.454 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:35:40.454 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 160297 2024-09-21T12:35:40.455 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.455 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:35:40.455 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.455 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:35:40.455 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.350814+0000", 2024-09-21T12:35:40.455 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.455 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.455 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.455 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:35:40.455 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.350814+0000", 2024-09-21T12:35:40.455 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.455 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.455 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.455 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:35:40.456 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.350815+0000", 2024-09-21T12:35:40.456 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5740000000000001e-06 2024-09-21T12:35:40.456 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.456 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.456 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:35:40.456 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.350820+0000", 2024-09-21T12:35:40.456 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3640000000000001e-06 2024-09-21T12:35:40.457 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.457 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.458 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:35:40.458 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.350829+0000", 2024-09-21T12:35:40.458 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.8270000000000004e-06 2024-09-21T12:35:40.458 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.458 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.458 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:35:40.458 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.350836+0000", 2024-09-21T12:35:40.458 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7209999999999994e-06 2024-09-21T12:35:40.458 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.458 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.458 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.458 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.350988+0000", 2024-09-21T12:35:40.458 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015201099999999999 2024-09-21T12:35:40.458 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.458 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.459 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:35:40.459 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.350995+0000", 2024-09-21T12:35:40.459 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.934e-06 2024-09-21T12:35:40.459 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.459 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.459 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.459 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:34.355876+0000", 2024-09-21T12:35:40.459 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0048808380000001 2024-09-21T12:35:40.459 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.459 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.459 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:35:40.459 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:34.355966+0000", 2024-09-21T12:35:40.459 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9665000000000003e-05 2024-09-21T12:35:40.459 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.459 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.460 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 5", 2024-09-21T12:35:40.460 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:34.356098+0000", 2024-09-21T12:35:40.460 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013210199999999999 2024-09-21T12:35:40.460 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.460 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.460 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:40.460 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:34.361371+0000", 2024-09-21T12:35:40.460 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0052735359999999997 2024-09-21T12:35:40.460 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.460 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.460 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:35:40.460 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:34.361390+0000", 2024-09-21T12:35:40.460 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8681999999999998e-05 2024-09-21T12:35:40.460 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.460 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.460 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:35:40.461 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:34.361425+0000", 2024-09-21T12:35:40.461 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5199e-05 2024-09-21T12:35:40.461 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.461 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.461 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:35:40.461 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:34.361510+0000", 2024-09-21T12:35:40.461 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.4557999999999999e-05 2024-09-21T12:35:40.461 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.461 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:40.461 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.461 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.461 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.461 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.15924.0:160311 10.3 10:c49dddeb:::benchmark_data_smithi114_161690_object33259:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-21T12:35:40.461 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:34:33.360384+0000", 2024-09-21T12:35:40.461 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 66.879769096999993, 2024-09-21T12:35:40.461 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0025336119999999, 2024-09-21T12:35:40.462 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:35:40.462 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:35:40.462 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:35:40.462 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:35:40.462 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.15924", 2024-09-21T12:35:40.462 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:35:40.462 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 160311 2024-09-21T12:35:40.462 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.462 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:35:40.462 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.462 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:35:40.462 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.360384+0000", 2024-09-21T12:35:40.462 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.462 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.462 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.462 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:35:40.463 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.360384+0000", 2024-09-21T12:35:40.463 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:35:40.463 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.463 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.463 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:35:40.463 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.360385+0000", 2024-09-21T12:35:40.463 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3009999999999999e-06 2024-09-21T12:35:40.463 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.463 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.463 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:35:40.463 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.360390+0000", 2024-09-21T12:35:40.463 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1270000000000002e-06 2024-09-21T12:35:40.463 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.463 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.463 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:35:40.463 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.360404+0000", 2024-09-21T12:35:40.464 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4262e-05 2024-09-21T12:35:40.464 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.464 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.464 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:35:40.464 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.360411+0000", 2024-09-21T12:35:40.464 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0600000000000002e-06 2024-09-21T12:35:40.464 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.464 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.464 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.464 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.360469+0000", 2024-09-21T12:35:40.464 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7781000000000001e-05 2024-09-21T12:35:40.464 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.464 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.464 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:35:40.464 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.360477+0000", 2024-09-21T12:35:40.464 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4420000000000001e-06 2024-09-21T12:35:40.465 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.465 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.465 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:35:40.465 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:34.357815+0000", 2024-09-21T12:35:40.465 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99733891399999997 2024-09-21T12:35:40.465 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.465 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.465 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:35:40.465 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:34.357926+0000", 2024-09-21T12:35:40.465 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000110862 2024-09-21T12:35:40.465 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.465 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.465 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 5", 2024-09-21T12:35:40.465 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:34.358054+0000", 2024-09-21T12:35:40.465 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000127305 2024-09-21T12:35:40.466 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.466 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.466 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:35:40.466 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:34.361517+0000", 2024-09-21T12:35:40.466 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0034637520000000001 2024-09-21T12:35:40.466 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.466 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.466 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:40.466 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:34.362769+0000", 2024-09-21T12:35:40.466 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001251902 2024-09-21T12:35:40.466 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.466 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.466 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:35:40.466 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:34.362821+0000", 2024-09-21T12:35:40.466 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1499000000000003e-05 2024-09-21T12:35:40.467 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:35:40.467 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:35:40.467 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:35:40.467 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:34.362917+0000", 2024-09-21T12:35:40.467 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6404999999999996e-05 2024-09-21T12:35:40.467 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.467 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:40.467 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.467 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:35:40.467 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:35:40.467 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:35:40.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:40.468 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:40.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:40.569 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:40.592 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:40.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:40.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:40.693 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:40.715 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:40.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:40.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:40.762 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-21T12:35:40.817 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:40.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:40.841 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:40.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:40.942 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:40.964 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:40.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:40.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:41.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:40 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:35:41.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:40 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:35:41.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:40 smithi114 ceph-mon[40867]: osdmap e402: 8 total, 7 up, 8 in 2024-09-21T12:35:41.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:40 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1151736338' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:35:41.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:40 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:35:41.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:40 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:35:41.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:40 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:35:41.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:40 smithi114 ceph-mon[30324]: osdmap e402: 8 total, 7 up, 8 in 2024-09-21T12:35:41.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:40 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1151736338' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:35:41.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:40 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:35:41.066 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:41.088 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:41.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:41.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:41.190 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:41.212 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:41.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:41.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:41.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:40 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:35:41.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:40 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:35:41.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:40 smithi183 ceph-mon[33277]: osdmap e402: 8 total, 7 up, 8 in 2024-09-21T12:35:41.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:40 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1151736338' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:35:41.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:40 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:35:41.262 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T12:35:41.313 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:41.335 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:41.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:41.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:41.436 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:41.458 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:41.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:41.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:41.559 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:41.581 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:41.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:41.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:41.597 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T12:35:41.597 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-09-21T12:35:41.598 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-09-21T12:35:41.683 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:41.705 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:41.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:41.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:41.746 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:35:41.806 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:41.827 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:41.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:41.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:41.929 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:41.938 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:41 smithi114 ceph-mon[40867]: pgmap v778: 178 pgs: 35 active+undersized, 17 active+undersized+degraded, 126 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.28k op/s; 63/597 objects degraded (10.553%) 2024-09-21T12:35:41.938 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:41 smithi114 ceph-mon[40867]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:35:41.939 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:41 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:35:41.939 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:41 smithi114 ceph-mon[40867]: osdmap e403: 8 total, 7 up, 8 in 2024-09-21T12:35:41.939 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:41 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:35:41.939 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:41 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:35:41.939 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:41 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:35:41.939 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:41 smithi114 ceph-mon[30324]: pgmap v778: 178 pgs: 35 active+undersized, 17 active+undersized+degraded, 126 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.28k op/s; 63/597 objects degraded (10.553%) 2024-09-21T12:35:41.939 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:41 smithi114 ceph-mon[30324]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:35:41.939 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:41 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:35:41.939 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:41 smithi114 ceph-mon[30324]: osdmap e403: 8 total, 7 up, 8 in 2024-09-21T12:35:41.939 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:41 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:35:41.940 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:41 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:35:41.940 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:41 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:35:41.950 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:41.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:41.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:42.023 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:41 smithi183 ceph-mon[33277]: pgmap v778: 178 pgs: 35 active+undersized, 17 active+undersized+degraded, 126 active+clean; 583 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.28k op/s; 63/597 objects degraded (10.553%) 2024-09-21T12:35:42.024 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:41 smithi183 ceph-mon[33277]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:35:42.024 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:41 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:35:42.024 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:41 smithi183 ceph-mon[33277]: osdmap e403: 8 total, 7 up, 8 in 2024-09-21T12:35:42.024 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:41 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:35:42.024 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:41 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:35:42.024 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:41 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:35:42.051 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:42.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:42.074 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:42.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:42.175 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:42.197 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:42.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:42.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:42.298 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:42.320 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:42.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:42.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:42.391 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:42.410 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:42.421 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:42.445 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:42.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:42.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:42.459 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:35:42.459 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-21T12:35:42.459 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 0 2024-09-21T12:35:42.459 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:35:42.547 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:42.568 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:42.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:42.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:42.670 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:42.691 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:42.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:42.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:42.793 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:42.809 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:35:42 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:35:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:35:42.809 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:42 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 63/597 objects degraded (10.553%), 17 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-09-21T12:35:42.810 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:42 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:35:42.810 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:42 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:35:42.810 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:42 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:35:42.810 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:42 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 63/597 objects degraded (10.553%), 17 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-09-21T12:35:42.810 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:42 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:35:42.810 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:42 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:35:42.810 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:42 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:35:42.814 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:42.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:42.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:42.916 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:42.926 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:35:42.926 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:35:42.926 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:35:42.926 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:35:42.926 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T12:35:42.937 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:42.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:42.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:42.942 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:35:42.943 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:35:42.959 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:35:42.959 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:35:42.981 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:35:42.981 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:35:42.998 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:35:42.998 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:35:43.014 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:35:43.015 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:35:43.038 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:43.060 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:43.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:43.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:43.114 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:42 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 63/597 objects degraded (10.553%), 17 pgs degraded, 52 pgs undersized (PG_DEGRADED) 2024-09-21T12:35:43.114 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:42 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:35:43.114 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:42 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:35:43.114 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:42 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:35:43.141 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T12:35:43.161 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:43.204 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:43.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:43.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:43.305 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:43.326 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:43.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:43.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:43.418 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:35:43.428 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:43.450 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:43.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:43.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:43.551 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:43.572 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:43.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:43.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:43.674 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:43.696 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:43.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:43.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:43.798 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:43.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:43.820 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:43.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:43.843 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:43 smithi183 ceph-mon[33277]: pgmap v780: 178 pgs: 36 active+undersized, 16 active+undersized+degraded, 126 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 839 op/s; 63/597 objects degraded (10.553%) 2024-09-21T12:35:43.843 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:43 smithi183 ceph-mon[33277]: pgmap v781: 178 pgs: 36 active+undersized, 16 active+undersized+degraded, 126 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.12k op/s; 63/597 objects degraded (10.553%) 2024-09-21T12:35:43.843 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:43 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2484040721' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:35:43.922 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:43.943 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:43.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:43.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:43.944 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:35:43.944 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:35:44.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:43 smithi114 ceph-mon[30324]: pgmap v780: 178 pgs: 36 active+undersized, 16 active+undersized+degraded, 126 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 839 op/s; 63/597 objects degraded (10.553%) 2024-09-21T12:35:44.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:43 smithi114 ceph-mon[30324]: pgmap v781: 178 pgs: 36 active+undersized, 16 active+undersized+degraded, 126 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.12k op/s; 63/597 objects degraded (10.553%) 2024-09-21T12:35:44.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:43 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2484040721' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:35:44.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:43 smithi114 ceph-mon[40867]: pgmap v780: 178 pgs: 36 active+undersized, 16 active+undersized+degraded, 126 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 839 op/s; 63/597 objects degraded (10.553%) 2024-09-21T12:35:44.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:43 smithi114 ceph-mon[40867]: pgmap v781: 178 pgs: 36 active+undersized, 16 active+undersized+degraded, 126 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.12k op/s; 63/597 objects degraded (10.553%) 2024-09-21T12:35:44.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:43 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2484040721' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:35:44.044 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:44.066 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:44.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:44.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:44.168 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:44.189 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:44.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:44.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:44.264 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T12:35:44.264 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T12:35:44.290 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:44.311 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:44.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:44.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:44.324 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T12:35:44.412 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:44.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:44.434 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:44.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:44.535 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:44.550 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T12:35:44.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:44.557 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:44.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:44.570 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:44.657 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:44.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:44.680 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:44.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:44.766 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'unique_pool_3' removed 2024-09-21T12:35:44.781 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:44.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:44.810 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:44.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:44.817 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:35:44.911 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:44.932 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:44.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:44.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:44.939 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:44 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1152971434' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T12:35:45.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:44 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1152971434' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T12:35:45.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:44 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1152971434' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T12:35:45.034 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:45.058 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:45.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:45.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:45.160 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:45.182 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:45.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:45.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:45.283 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:45.291 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:35:45.291 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:35:45.306 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:45.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:45.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:45.408 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:45.430 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:45.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:45.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:45.470 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T12:35:45.471 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T12:35:45.471 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-09-21T12:35:45.471 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_4 16 2024-09-21T12:35:45.532 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:45.553 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:45.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:45.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:45.655 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:45.678 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:45.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:45.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:45.780 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:45.801 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:45.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:45.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:45.852 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:45.903 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:45.924 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:45.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:45.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:45.971 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:45 smithi183 ceph-mon[33277]: pgmap v782: 178 pgs: 38 active+undersized, 14 active+undersized+degraded, 126 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 873 B/s rd, 0 op/s; 63/597 objects degraded (10.553%) 2024-09-21T12:35:45.971 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:45 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1152971434' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T12:35:45.971 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:45 smithi183 ceph-mon[33277]: osdmap e404: 8 total, 7 up, 8 in 2024-09-21T12:35:45.999 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T12:35:46.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:45 smithi114 ceph-mon[40867]: pgmap v782: 178 pgs: 38 active+undersized, 14 active+undersized+degraded, 126 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 873 B/s rd, 0 op/s; 63/597 objects degraded (10.553%) 2024-09-21T12:35:46.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:45 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1152971434' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T12:35:46.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:45 smithi114 ceph-mon[40867]: osdmap e404: 8 total, 7 up, 8 in 2024-09-21T12:35:46.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:45 smithi114 ceph-mon[30324]: pgmap v782: 178 pgs: 38 active+undersized, 14 active+undersized+degraded, 126 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 873 B/s rd, 0 op/s; 63/597 objects degraded (10.553%) 2024-09-21T12:35:46.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:45 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1152971434' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T12:35:46.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:45 smithi114 ceph-mon[30324]: osdmap e404: 8 total, 7 up, 8 in 2024-09-21T12:35:46.026 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:46.047 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:46.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:46.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:46.149 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:46.170 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:46.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:46.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:46.244 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:35:46.266 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:35:46.272 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:46.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:46.294 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:46.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:46.394 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:46.416 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:46.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:46.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:46.517 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:46.520 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:46.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:46.540 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:46.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:46.598 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-09-21T12:35:46.598 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:35:46.598 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-21T12:35:46.598 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd out 7 2024-09-21T12:35:46.642 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:46.663 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:46.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:46.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:46.733 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:35:46.733 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:35:46.765 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:46.778 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'unique_pool_4' created 2024-09-21T12:35:46.789 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-21T12:35:46.789 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:46.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:46.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:46.891 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:46.912 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:46.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:46.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:46.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:46 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/4155500930' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:35:46.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:46 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:35:46.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:46 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2978365541' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-21T12:35:47.014 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:47.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:46 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4155500930' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:35:47.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:46 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:35:47.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:46 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2978365541' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-21T12:35:47.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:46 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/4155500930' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:35:47.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:46 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:35:47.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:46 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2978365541' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-21T12:35:47.035 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:47.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:47.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:47.136 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:47.158 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:47.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:47.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:47.260 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:47.282 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:47.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:47.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:47.358 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:35:47.359 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:35:47.376 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:35:47.376 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:35:47.376 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T12:35:47.385 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:47.392 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:35:47.392 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:35:47.406 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:47.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:47.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:47.410 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T12:35:47.410 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:35:47.411 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:35:47.426 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:35:47.426 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:35:47.443 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:35:47.443 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:35:47.460 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:35:47.461 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:35:47.508 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:47.529 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:47.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:47.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:47.631 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:47.653 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:47.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:47.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:47.697 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:35:47.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:35:47 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:35:47.755 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:47.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:47.777 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:47.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:47.817 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-09-21T12:35:47.878 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:47.923 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:47.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:47.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:48.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:47 smithi114 ceph-mon[30324]: pgmap v784: 162 pgs: 32 active+undersized, 14 active+undersized+degraded, 116 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 698 B/s rd, 0 op/s; 63/597 objects degraded (10.553%) 2024-09-21T12:35:48.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:47 smithi114 ceph-mon[30324]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:35:48.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:47 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:35:48.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:47 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2978365541' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-21T12:35:48.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:47 smithi114 ceph-mon[30324]: osdmap e405: 8 total, 7 up, 8 in 2024-09-21T12:35:48.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:47 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 63/597 objects degraded (10.553%), 14 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-09-21T12:35:48.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:47 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:35:48.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:47 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:35:48.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:47 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2162954786' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:35:48.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:47 smithi114 ceph-mon[40867]: pgmap v784: 162 pgs: 32 active+undersized, 14 active+undersized+degraded, 116 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 698 B/s rd, 0 op/s; 63/597 objects degraded (10.553%) 2024-09-21T12:35:48.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:47 smithi114 ceph-mon[40867]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:35:48.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:47 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:35:48.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:47 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2978365541' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-21T12:35:48.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:47 smithi114 ceph-mon[40867]: osdmap e405: 8 total, 7 up, 8 in 2024-09-21T12:35:48.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:47 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 63/597 objects degraded (10.553%), 14 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-09-21T12:35:48.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:47 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:35:48.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:47 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:35:48.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:47 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2162954786' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:35:48.026 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:48.047 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:48.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:48.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:48.091 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:47 smithi183 ceph-mon[33277]: pgmap v784: 162 pgs: 32 active+undersized, 14 active+undersized+degraded, 116 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 698 B/s rd, 0 op/s; 63/597 objects degraded (10.553%) 2024-09-21T12:35:48.091 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:47 smithi183 ceph-mon[33277]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:35:48.091 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:47 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:35:48.092 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:47 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2978365541' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-21T12:35:48.092 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:47 smithi183 ceph-mon[33277]: osdmap e405: 8 total, 7 up, 8 in 2024-09-21T12:35:48.092 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:47 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 63/597 objects degraded (10.553%), 14 pgs degraded, 46 pgs undersized (PG_DEGRADED) 2024-09-21T12:35:48.092 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:47 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:35:48.092 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:47 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:35:48.092 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:47 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2162954786' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:35:48.148 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:48.170 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:48.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:48.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:48.195 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:35:48.195 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-21T12:35:48.195 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-21T12:35:48.195 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-21T12:35:48.195 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:35:48.272 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:48.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:48.293 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:48.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:48.394 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:48.404 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T12:35:48.404 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T12:35:48.416 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:48.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:48.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:48.517 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:48.539 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:48.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:48.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:48.640 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:48.661 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:48.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:48.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:48.763 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:48.786 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:48.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:48.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:48.815 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T12:35:48.888 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:48.919 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:48.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:48.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:49.021 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:49.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:48 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:35:49.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:48 smithi114 ceph-mon[40867]: osdmap e406: 8 total, 7 up, 8 in 2024-09-21T12:35:49.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:48 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:35:49.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:48 smithi114 ceph-mon[30324]: osdmap e406: 8 total, 7 up, 8 in 2024-09-21T12:35:49.047 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:49.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:49.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:49.100 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:49.108 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:35:49.118 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:49.149 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:49.170 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:49.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:49.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:49.229 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:48 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:35:49.229 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:48 smithi183 ceph-mon[33277]: osdmap e406: 8 total, 7 up, 8 in 2024-09-21T12:35:49.272 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:49.293 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:49.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:49.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:49.394 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:49.416 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:49.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:49.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:49.517 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:49.539 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:49.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:49.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:49.632 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:35:49.632 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-21T12:35:49.632 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-21T12:35:49.632 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:35:49.632 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.632 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:136741 9.f 9:f16b8898:::benchmark_data_smithi114_133214_object12707:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-09-21T12:35:49.632 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:30:12.044643+0000", 2024-09-21T12:35:49.632 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 337.58024035800003, 2024-09-21T12:35:49.632 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15204224699999999, 2024-09-21T12:35:49.633 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:35:49.633 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:35:49.633 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:35:49.633 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:35:49.633 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:35:49.633 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:35:49.633 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 136741 2024-09-21T12:35:49.633 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.633 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:35:49.633 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.633 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:35:49.633 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.044643+0000", 2024-09-21T12:35:49.633 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.633 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.633 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.634 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:35:49.634 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.044643+0000", 2024-09-21T12:35:49.634 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.634 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.634 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:35:49.634 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.044644+0000", 2024-09-21T12:35:49.634 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.043e-06 2024-09-21T12:35:49.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.634 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.634 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:35:49.634 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.044647+0000", 2024-09-21T12:35:49.634 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4199999999999999e-06 2024-09-21T12:35:49.635 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.635 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.635 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:35:49.635 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.044655+0000", 2024-09-21T12:35:49.635 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3649999999999998e-06 2024-09-21T12:35:49.635 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.635 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.635 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:35:49.635 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.044659+0000", 2024-09-21T12:35:49.635 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5800000000000002e-06 2024-09-21T12:35:49.635 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.635 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.635 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:35:49.635 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.044697+0000", 2024-09-21T12:35:49.635 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7499000000000001e-05 2024-09-21T12:35:49.636 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.636 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:35:49.636 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.044810+0000", 2024-09-21T12:35:49.636 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000113287 2024-09-21T12:35:49.636 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.636 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:35:49.636 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.044925+0000", 2024-09-21T12:35:49.636 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000115343 2024-09-21T12:35:49.636 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.636 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:35:49.637 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.046578+0000", 2024-09-21T12:35:49.637 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0016527989999999999 2024-09-21T12:35:49.637 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.637 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.637 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.637 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.154593+0000", 2024-09-21T12:35:49.637 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.108015135 2024-09-21T12:35:49.637 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.637 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.637 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.637 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.196609+0000", 2024-09-21T12:35:49.637 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.042015706999999999 2024-09-21T12:35:49.637 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.637 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.637 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:35:49.638 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.196649+0000", 2024-09-21T12:35:49.638 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9935999999999998e-05 2024-09-21T12:35:49.638 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.638 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.638 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:35:49.638 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.196685+0000", 2024-09-21T12:35:49.638 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6133000000000001e-05 2024-09-21T12:35:49.638 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.638 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:35:49.638 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.638 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.638 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.638 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:143984 9.f 9:fd48ee49:::benchmark_data_smithi114_133214_object19950:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-09-21T12:35:49.638 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:30:20.026313+0000", 2024-09-21T12:35:49.638 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 329.598570548, 2024-09-21T12:35:49.639 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.33116894200000002, 2024-09-21T12:35:49.639 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:35:49.639 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:35:49.639 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:35:49.639 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:35:49.639 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:35:49.639 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:35:49.639 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 143984 2024-09-21T12:35:49.639 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.639 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:35:49.639 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.639 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:35:49.639 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026313+0000", 2024-09-21T12:35:49.639 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.640 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:35:49.640 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026313+0000", 2024-09-21T12:35:49.640 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.640 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:35:49.640 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026314+0000", 2024-09-21T12:35:49.640 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.186e-06 2024-09-21T12:35:49.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.640 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:35:49.641 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026317+0000", 2024-09-21T12:35:49.641 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0989999999999999e-06 2024-09-21T12:35:49.641 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.641 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.641 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:35:49.641 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026329+0000", 2024-09-21T12:35:49.641 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1932e-05 2024-09-21T12:35:49.641 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.641 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.641 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:35:49.641 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026333+0000", 2024-09-21T12:35:49.641 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3370000000000001e-06 2024-09-21T12:35:49.641 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.641 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.642 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:35:49.642 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026364+0000", 2024-09-21T12:35:49.642 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0235000000000001e-05 2024-09-21T12:35:49.642 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.642 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.642 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:35:49.642 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026441+0000", 2024-09-21T12:35:49.642 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7663000000000005e-05 2024-09-21T12:35:49.642 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.642 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.642 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:35:49.642 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026511+0000", 2024-09-21T12:35:49.642 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9640000000000004e-05 2024-09-21T12:35:49.643 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.643 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.643 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:35:49.643 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.029298+0000", 2024-09-21T12:35:49.643 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0027875500000000002 2024-09-21T12:35:49.643 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.643 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.643 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.643 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.039048+0000", 2024-09-21T12:35:49.643 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0097491490000000004 2024-09-21T12:35:49.643 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.643 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.643 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.644 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.357393+0000", 2024-09-21T12:35:49.644 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31834596300000001 2024-09-21T12:35:49.644 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.644 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.644 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:35:49.644 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.357435+0000", 2024-09-21T12:35:49.644 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1078999999999998e-05 2024-09-21T12:35:49.644 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.644 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.644 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:35:49.644 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.357482+0000", 2024-09-21T12:35:49.644 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7108999999999999e-05 2024-09-21T12:35:49.644 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.644 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:35:49.645 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.645 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.645 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.645 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:143985 9.f 9:f12d363a:::benchmark_data_smithi114_133214_object19951:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-09-21T12:35:49.645 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:30:20.026340+0000", 2024-09-21T12:35:49.645 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 329.59854290300001, 2024-09-21T12:35:49.645 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.33955830100000001, 2024-09-21T12:35:49.645 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:35:49.645 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:35:49.645 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:35:49.645 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:35:49.645 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:35:49.645 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:35:49.646 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 143985 2024-09-21T12:35:49.646 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.646 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:35:49.646 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.646 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:35:49.646 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026340+0000", 2024-09-21T12:35:49.646 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.646 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.646 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.646 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:35:49.646 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026340+0000", 2024-09-21T12:35:49.646 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:35:49.647 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026341+0000", 2024-09-21T12:35:49.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.84e-07 2024-09-21T12:35:49.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:35:49.647 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026342+0000", 2024-09-21T12:35:49.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.671e-06 2024-09-21T12:35:49.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:35:49.647 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026346+0000", 2024-09-21T12:35:49.648 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5769999999999998e-06 2024-09-21T12:35:49.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:35:49.648 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026350+0000", 2024-09-21T12:35:49.648 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8029999999999998e-06 2024-09-21T12:35:49.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:35:49.648 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.027962+0000", 2024-09-21T12:35:49.648 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0016119839999999999 2024-09-21T12:35:49.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.649 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:35:49.649 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.028067+0000", 2024-09-21T12:35:49.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010498299999999999 2024-09-21T12:35:49.649 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.649 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:35:49.649 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.028160+0000", 2024-09-21T12:35:49.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3731000000000001e-05 2024-09-21T12:35:49.649 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.649 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:35:49.649 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.030647+0000", 2024-09-21T12:35:49.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024862310000000002 2024-09-21T12:35:49.649 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.650 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.650 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.650 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.039088+0000", 2024-09-21T12:35:49.650 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0084414620000000003 2024-09-21T12:35:49.650 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.650 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.650 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.650 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.365834+0000", 2024-09-21T12:35:49.650 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.32674630799999999 2024-09-21T12:35:49.650 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.650 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.650 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:35:49.650 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.365865+0000", 2024-09-21T12:35:49.650 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0732999999999999e-05 2024-09-21T12:35:49.651 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.651 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.651 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:35:49.651 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.365899+0000", 2024-09-21T12:35:49.651 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3534000000000001e-05 2024-09-21T12:35:49.651 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.651 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:35:49.651 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.651 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.651 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.651 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:143986 9.f 9:f1d8e647:::benchmark_data_smithi114_133214_object19952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-09-21T12:35:49.651 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:30:20.027267+0000", 2024-09-21T12:35:49.651 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 329.59761616700001, 2024-09-21T12:35:49.651 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.34414182799999998, 2024-09-21T12:35:49.651 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:35:49.652 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:35:49.652 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:35:49.652 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:35:49.652 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:35:49.652 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:35:49.652 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 143986 2024-09-21T12:35:49.652 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.652 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:35:49.652 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.652 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:35:49.652 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.027267+0000", 2024-09-21T12:35:49.652 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.652 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.652 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.653 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:35:49.653 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.027267+0000", 2024-09-21T12:35:49.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.653 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.653 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:35:49.653 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.027268+0000", 2024-09-21T12:35:49.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.065e-06 2024-09-21T12:35:49.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.653 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.653 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:35:49.653 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.027272+0000", 2024-09-21T12:35:49.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.318e-06 2024-09-21T12:35:49.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.654 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:35:49.654 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.027282+0000", 2024-09-21T12:35:49.654 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4490000000000005e-06 2024-09-21T12:35:49.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.654 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:35:49.654 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.027287+0000", 2024-09-21T12:35:49.654 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9019999999999998e-06 2024-09-21T12:35:49.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.654 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:35:49.654 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.029320+0000", 2024-09-21T12:35:49.654 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020333090000000001 2024-09-21T12:35:49.655 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.655 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.655 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:35:49.655 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.029383+0000", 2024-09-21T12:35:49.655 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3274000000000004e-05 2024-09-21T12:35:49.655 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.655 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.655 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:35:49.655 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.029450+0000", 2024-09-21T12:35:49.655 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6649999999999994e-05 2024-09-21T12:35:49.655 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.655 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.655 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:35:49.655 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.030775+0000", 2024-09-21T12:35:49.655 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001325346 2024-09-21T12:35:49.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.656 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.656 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.656 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.039124+0000", 2024-09-21T12:35:49.656 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0083483350000000001 2024-09-21T12:35:49.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.656 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.656 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.656 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.371309+0000", 2024-09-21T12:35:49.656 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.33218562400000001 2024-09-21T12:35:49.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.656 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.656 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:35:49.657 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.371364+0000", 2024-09-21T12:35:49.657 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4336999999999998e-05 2024-09-21T12:35:49.657 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.657 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.657 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:35:49.657 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.371409+0000", 2024-09-21T12:35:49.657 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5219000000000002e-05 2024-09-21T12:35:49.657 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.657 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:35:49.657 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.657 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.657 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.657 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:148258 9.f 9:fa68205d:::benchmark_data_smithi114_133214_object24224:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-09-21T12:35:49.657 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:30:25.552566+0000", 2024-09-21T12:35:49.658 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 324.07231715699999, 2024-09-21T12:35:49.658 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.165969116, 2024-09-21T12:35:49.658 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:35:49.658 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:35:49.658 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:35:49.658 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:35:49.658 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:35:49.658 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:35:49.658 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 148258 2024-09-21T12:35:49.658 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.658 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:35:49.658 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.658 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:35:49.659 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.552566+0000", 2024-09-21T12:35:49.659 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.659 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.659 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.659 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:35:49.659 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.552566+0000", 2024-09-21T12:35:49.659 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.659 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.659 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.659 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:35:49.659 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.552568+0000", 2024-09-21T12:35:49.659 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.406e-06 2024-09-21T12:35:49.659 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:35:49.660 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.552573+0000", 2024-09-21T12:35:49.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0030000000000002e-06 2024-09-21T12:35:49.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:35:49.660 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.552583+0000", 2024-09-21T12:35:49.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0379e-05 2024-09-21T12:35:49.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:35:49.661 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.552589+0000", 2024-09-21T12:35:49.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2480000000000001e-06 2024-09-21T12:35:49.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.661 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:35:49.661 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.552647+0000", 2024-09-21T12:35:49.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7422e-05 2024-09-21T12:35:49.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.661 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:35:49.661 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.552765+0000", 2024-09-21T12:35:49.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000118627 2024-09-21T12:35:49.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:35:49.662 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.552887+0000", 2024-09-21T12:35:49.662 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000121557 2024-09-21T12:35:49.662 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.662 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:35:49.662 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.554567+0000", 2024-09-21T12:35:49.662 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0016804249999999999 2024-09-21T12:35:49.662 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.662 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.662 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.569714+0000", 2024-09-21T12:35:49.662 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.015146980000000001 2024-09-21T12:35:49.662 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.662 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.663 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.663 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.718424+0000", 2024-09-21T12:35:49.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14870934699999999 2024-09-21T12:35:49.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.663 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.663 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:35:49.663 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.718488+0000", 2024-09-21T12:35:49.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4362000000000005e-05 2024-09-21T12:35:49.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.663 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.663 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:35:49.663 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.718535+0000", 2024-09-21T12:35:49.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7360000000000001e-05 2024-09-21T12:35:49.664 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.664 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:35:49.664 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.664 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.664 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.664 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:148262 9.f 9:f36a2eaf:::benchmark_data_smithi114_133214_object24228:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-09-21T12:35:49.664 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:30:25.564038+0000", 2024-09-21T12:35:49.664 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 324.06084534799999, 2024-09-21T12:35:49.664 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.154638887, 2024-09-21T12:35:49.664 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:35:49.664 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:35:49.664 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:35:49.664 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:35:49.664 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:35:49.665 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:35:49.665 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 148262 2024-09-21T12:35:49.665 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.665 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:35:49.665 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.665 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:35:49.665 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.564038+0000", 2024-09-21T12:35:49.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.665 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.665 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.665 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:35:49.665 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.564038+0000", 2024-09-21T12:35:49.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.665 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.665 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.665 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:35:49.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.564039+0000", 2024-09-21T12:35:49.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.144e-06 2024-09-21T12:35:49.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:35:49.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.564043+0000", 2024-09-21T12:35:49.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.25e-06 2024-09-21T12:35:49.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:35:49.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.564054+0000", 2024-09-21T12:35:49.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0963e-05 2024-09-21T12:35:49.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:35:49.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.564060+0000", 2024-09-21T12:35:49.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.981e-06 2024-09-21T12:35:49.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:35:49.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.564107+0000", 2024-09-21T12:35:49.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6758999999999997e-05 2024-09-21T12:35:49.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:35:49.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.564242+0000", 2024-09-21T12:35:49.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013536700000000001 2024-09-21T12:35:49.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:35:49.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.564364+0000", 2024-09-21T12:35:49.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000122062 2024-09-21T12:35:49.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:35:49.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.566033+0000", 2024-09-21T12:35:49.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001668444 2024-09-21T12:35:49.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.569752+0000", 2024-09-21T12:35:49.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0037187560000000001 2024-09-21T12:35:49.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.718589+0000", 2024-09-21T12:35:49.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14883724800000001 2024-09-21T12:35:49.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:35:49.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.718637+0000", 2024-09-21T12:35:49.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7763000000000003e-05 2024-09-21T12:35:49.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:35:49.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.718677+0000", 2024-09-21T12:35:49.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.015e-05 2024-09-21T12:35:49.669 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.669 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:35:49.670 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.670 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:148264 9.f 9:f2a805cb:::benchmark_data_smithi114_133214_object24230:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-09-21T12:35:49.670 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:30:25.566769+0000", 2024-09-21T12:35:49.670 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 324.05811473, 2024-09-21T12:35:49.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15200875599999999, 2024-09-21T12:35:49.670 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:35:49.670 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:35:49.670 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:35:49.670 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:35:49.670 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:35:49.670 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:35:49.670 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 148264 2024-09-21T12:35:49.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.671 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:35:49.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:35:49.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.566769+0000", 2024-09-21T12:35:49.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:35:49.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.566769+0000", 2024-09-21T12:35:49.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:35:49.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.566770+0000", 2024-09-21T12:35:49.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.043e-06 2024-09-21T12:35:49.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:35:49.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.566773+0000", 2024-09-21T12:35:49.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8179999999999997e-06 2024-09-21T12:35:49.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:35:49.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.566783+0000", 2024-09-21T12:35:49.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4299999999999995e-06 2024-09-21T12:35:49.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:35:49.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.566794+0000", 2024-09-21T12:35:49.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0910999999999999e-05 2024-09-21T12:35:49.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:35:49.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.566853+0000", 2024-09-21T12:35:49.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8943999999999997e-05 2024-09-21T12:35:49.673 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:49.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:35:49.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.566971+0000", 2024-09-21T12:35:49.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000118113 2024-09-21T12:35:49.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:35:49.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.567091+0000", 2024-09-21T12:35:49.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000120232 2024-09-21T12:35:49.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:35:49.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.568779+0000", 2024-09-21T12:35:49.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001688134 2024-09-21T12:35:49.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.569859+0000", 2024-09-21T12:35:49.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010798330000000001 2024-09-21T12:35:49.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.718708+0000", 2024-09-21T12:35:49.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14884867800000001 2024-09-21T12:35:49.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:35:49.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.718752+0000", 2024-09-21T12:35:49.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4379999999999999e-05 2024-09-21T12:35:49.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:35:49.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.718777+0000", 2024-09-21T12:35:49.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5239999999999999e-05 2024-09-21T12:35:49.677 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.677 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:35:49.677 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.678 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:175495 9.f 9:f9909850:::benchmark_data_smithi114_133214_object51461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-21T12:35:49.678 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:30:56.789850+0000", 2024-09-21T12:35:49.678 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 292.83503277400001, 2024-09-21T12:35:49.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.16364801100000001, 2024-09-21T12:35:49.678 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:35:49.678 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:35:49.678 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:35:49.678 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:35:49.678 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:35:49.678 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:35:49.678 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 175495 2024-09-21T12:35:49.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.678 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:35:49.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:35:49.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.789850+0000", 2024-09-21T12:35:49.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:35:49.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.789850+0000", 2024-09-21T12:35:49.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:35:49.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.789852+0000", 2024-09-21T12:35:49.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.573e-06 2024-09-21T12:35:49.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:35:49.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.789857+0000", 2024-09-21T12:35:49.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0209999999999999e-06 2024-09-21T12:35:49.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:35:49.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.789868+0000", 2024-09-21T12:35:49.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0496000000000001e-05 2024-09-21T12:35:49.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:35:49.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.789874+0000", 2024-09-21T12:35:49.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7599999999999997e-06 2024-09-21T12:35:49.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:35:49.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.789923+0000", 2024-09-21T12:35:49.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8188e-05 2024-09-21T12:35:49.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:35:49.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.790047+0000", 2024-09-21T12:35:49.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012412999999999999 2024-09-21T12:35:49.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:35:49.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.790161+0000", 2024-09-21T12:35:49.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011474800000000001 2024-09-21T12:35:49.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:35:49.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.791943+0000", 2024-09-21T12:35:49.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0017820710000000001 2024-09-21T12:35:49.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.943029+0000", 2024-09-21T12:35:49.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15108565800000001 2024-09-21T12:35:49.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.953435+0000", 2024-09-21T12:35:49.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.010406227000000001 2024-09-21T12:35:49.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:35:49.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.953469+0000", 2024-09-21T12:35:49.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3969e-05 2024-09-21T12:35:49.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:35:49.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.953499+0000", 2024-09-21T12:35:49.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.917e-05 2024-09-21T12:35:49.683 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.683 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:35:49.683 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.684 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:216505 9.f 9:f5ab0cee:::benchmark_data_smithi114_133214_object92471:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e306)", 2024-09-21T12:35:49.684 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:33.894635+0000", 2024-09-21T12:35:49.684 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 255.730247889, 2024-09-21T12:35:49.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.38916874499999998, 2024-09-21T12:35:49.684 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:35:49.684 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:35:49.684 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:35:49.684 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:35:49.684 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:35:49.684 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:35:49.684 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 216505 2024-09-21T12:35:49.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.684 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:35:49.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:35:49.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.894635+0000", 2024-09-21T12:35:49.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:35:49.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.894635+0000", 2024-09-21T12:35:49.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:35:49.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.894637+0000", 2024-09-21T12:35:49.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.654e-06 2024-09-21T12:35:49.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:35:49.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.894642+0000", 2024-09-21T12:35:49.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0390000000000004e-06 2024-09-21T12:35:49.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:35:49.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.894658+0000", 2024-09-21T12:35:49.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6379999999999999e-05 2024-09-21T12:35:49.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:35:49.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.894666+0000", 2024-09-21T12:35:49.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0550000000000002e-06 2024-09-21T12:35:49.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:35:49.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.894723+0000", 2024-09-21T12:35:49.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7982000000000002e-05 2024-09-21T12:35:49.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:35:49.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.894802+0000", 2024-09-21T12:35:49.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8399000000000006e-05 2024-09-21T12:35:49.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:35:49.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.894874+0000", 2024-09-21T12:35:49.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2423999999999996e-05 2024-09-21T12:35:49.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:35:49.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.896880+0000", 2024-09-21T12:35:49.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020054180000000001 2024-09-21T12:35:49.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.900439+0000", 2024-09-21T12:35:49.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0035589409999999999 2024-09-21T12:35:49.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:34.283682+0000", 2024-09-21T12:35:49.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.38324365500000002 2024-09-21T12:35:49.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:35:49.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:34.283755+0000", 2024-09-21T12:35:49.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2329000000000005e-05 2024-09-21T12:35:49.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:35:49.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:34.283804+0000", 2024-09-21T12:35:49.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9468999999999997e-05 2024-09-21T12:35:49.689 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.689 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:35:49.689 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.690 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:216514 9.f 9:f58474dd:::benchmark_data_smithi114_133214_object92480:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e306)", 2024-09-21T12:35:49.690 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:33.900257+0000", 2024-09-21T12:35:49.690 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 255.724626583, 2024-09-21T12:35:49.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.38852348399999997, 2024-09-21T12:35:49.690 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:35:49.690 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:35:49.690 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:35:49.690 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:35:49.690 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:35:49.690 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:35:49.690 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 216514 2024-09-21T12:35:49.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.690 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:35:49.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.691 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:35:49.691 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.900257+0000", 2024-09-21T12:35:49.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.691 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:35:49.691 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.900257+0000", 2024-09-21T12:35:49.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.691 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:35:49.691 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.900258+0000", 2024-09-21T12:35:49.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.57e-06 2024-09-21T12:35:49.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:35:49.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.900264+0000", 2024-09-21T12:35:49.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2979999999999999e-06 2024-09-21T12:35:49.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:35:49.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.900279+0000", 2024-09-21T12:35:49.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5211000000000001e-05 2024-09-21T12:35:49.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:35:49.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.900286+0000", 2024-09-21T12:35:49.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2089999999999999e-06 2024-09-21T12:35:49.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:35:49.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.900470+0000", 2024-09-21T12:35:49.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00018402699999999999 2024-09-21T12:35:49.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:35:49.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.900607+0000", 2024-09-21T12:35:49.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013691800000000001 2024-09-21T12:35:49.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:35:49.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.900732+0000", 2024-09-21T12:35:49.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000125081 2024-09-21T12:35:49.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:35:49.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.902574+0000", 2024-09-21T12:35:49.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0018421290000000001 2024-09-21T12:35:49.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.903091+0000", 2024-09-21T12:35:49.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000517214 2024-09-21T12:35:49.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:34.288673+0000", 2024-09-21T12:35:49.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.38558214000000002 2024-09-21T12:35:49.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:35:49.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:34.288738+0000", 2024-09-21T12:35:49.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4135999999999997e-05 2024-09-21T12:35:49.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:35:49.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:34.288780+0000", 2024-09-21T12:35:49.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2551e-05 2024-09-21T12:35:49.695 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.696 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:35:49.696 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.696 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:231697 9.19 9:9abf7afe:::benchmark_data_smithi114_133214_object107663:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:35:49.696 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:50.030279+0000", 2024-09-21T12:35:49.696 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 239.59460415500001, 2024-09-21T12:35:49.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1184929130000001, 2024-09-21T12:35:49.696 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:35:49.696 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:35:49.696 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:35:49.696 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:35:49.696 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:35:49.696 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:35:49.696 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 231697 2024-09-21T12:35:49.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.697 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:35:49.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:35:49.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030279+0000", 2024-09-21T12:35:49.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:35:49.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030279+0000", 2024-09-21T12:35:49.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:35:49.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030281+0000", 2024-09-21T12:35:49.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.689e-06 2024-09-21T12:35:49.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:35:49.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030285+0000", 2024-09-21T12:35:49.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9859999999999996e-06 2024-09-21T12:35:49.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:35:49.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030295+0000", 2024-09-21T12:35:49.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0178e-05 2024-09-21T12:35:49.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:35:49.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030305+0000", 2024-09-21T12:35:49.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7860000000000008e-06 2024-09-21T12:35:49.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:35:49.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030481+0000", 2024-09-21T12:35:49.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000175768 2024-09-21T12:35:49.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:35:49.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030490+0000", 2024-09-21T12:35:49.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7669999999999998e-06 2024-09-21T12:35:49.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:35:49.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.144698+0000", 2024-09-21T12:35:49.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1142073050000001 2024-09-21T12:35:49.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:35:49.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.144803+0000", 2024-09-21T12:35:49.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010563100000000001 2024-09-21T12:35:49.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,6,7", 2024-09-21T12:35:49.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.145159+0000", 2024-09-21T12:35:49.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00035557900000000002 2024-09-21T12:35:49.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.148533+0000", 2024-09-21T12:35:49.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003373741 2024-09-21T12:35:49.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.148585+0000", 2024-09-21T12:35:49.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2278999999999999e-05 2024-09-21T12:35:49.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.148619+0000", 2024-09-21T12:35:49.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.417e-05 2024-09-21T12:35:49.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:35:49.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.148629+0000", 2024-09-21T12:35:49.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8779999999999993e-06 2024-09-21T12:35:49.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:35:49.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.148667+0000", 2024-09-21T12:35:49.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7728999999999997e-05 2024-09-21T12:35:49.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:35:49.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.148772+0000", 2024-09-21T12:35:49.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010542700000000001 2024-09-21T12:35:49.702 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.702 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:35:49.703 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.703 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:231705 9.19 9:9e8af994:::benchmark_data_smithi114_133214_object107671:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:35:49.703 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:50.030314+0000", 2024-09-21T12:35:49.703 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 239.594568853, 2024-09-21T12:35:49.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1228952539999999, 2024-09-21T12:35:49.703 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:35:49.703 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:35:49.703 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:35:49.703 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:35:49.703 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:35:49.703 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:35:49.703 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 231705 2024-09-21T12:35:49.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.704 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:35:49.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:35:49.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030314+0000", 2024-09-21T12:35:49.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:35:49.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030314+0000", 2024-09-21T12:35:49.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:35:49.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030315+0000", 2024-09-21T12:35:49.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9299999999999998e-07 2024-09-21T12:35:49.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:35:49.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030317+0000", 2024-09-21T12:35:49.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2450000000000001e-06 2024-09-21T12:35:49.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:35:49.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030323+0000", 2024-09-21T12:35:49.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8939999999999998e-06 2024-09-21T12:35:49.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:35:49.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030328+0000", 2024-09-21T12:35:49.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1800000000000004e-06 2024-09-21T12:35:49.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:35:49.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030512+0000", 2024-09-21T12:35:49.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00018370300000000001 2024-09-21T12:35:49.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:35:49.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030518+0000", 2024-09-21T12:35:49.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2319999999999998e-06 2024-09-21T12:35:49.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:35:49.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.148936+0000", 2024-09-21T12:35:49.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1184178929999999 2024-09-21T12:35:49.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:35:49.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.149021+0000", 2024-09-21T12:35:49.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4841999999999998e-05 2024-09-21T12:35:49.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,6,7", 2024-09-21T12:35:49.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.149286+0000", 2024-09-21T12:35:49.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00026551899999999998 2024-09-21T12:35:49.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153018+0000", 2024-09-21T12:35:49.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0037314340000000001 2024-09-21T12:35:49.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153057+0000", 2024-09-21T12:35:49.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9365e-05 2024-09-21T12:35:49.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:35:49.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153068+0000", 2024-09-21T12:35:49.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.11e-05 2024-09-21T12:35:49.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153107+0000", 2024-09-21T12:35:49.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8305e-05 2024-09-21T12:35:49.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:35:49.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153144+0000", 2024-09-21T12:35:49.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7110000000000002e-05 2024-09-21T12:35:49.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:35:49.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153210+0000", 2024-09-21T12:35:49.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5938999999999993e-05 2024-09-21T12:35:49.709 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.709 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:35:49.709 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.710 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:231765 9.19 9:9e977502:::benchmark_data_smithi114_133214_object107731:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:35:49.710 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:50.030334+0000", 2024-09-21T12:35:49.710 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 239.59454926399999, 2024-09-21T12:35:49.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.123057872, 2024-09-21T12:35:49.710 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:35:49.710 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:35:49.710 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:35:49.710 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:35:49.710 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:35:49.710 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:35:49.710 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 231765 2024-09-21T12:35:49.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.711 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:35:49.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:35:49.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030334+0000", 2024-09-21T12:35:49.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:35:49.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030334+0000", 2024-09-21T12:35:49.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:35:49.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030334+0000", 2024-09-21T12:35:49.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2300000000000002e-07 2024-09-21T12:35:49.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.712 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:35:49.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030337+0000", 2024-09-21T12:35:49.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0949999999999998e-06 2024-09-21T12:35:49.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.712 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:35:49.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030342+0000", 2024-09-21T12:35:49.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7710000000000003e-06 2024-09-21T12:35:49.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:35:49.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030347+0000", 2024-09-21T12:35:49.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.198e-06 2024-09-21T12:35:49.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:35:49.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030540+0000", 2024-09-21T12:35:49.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019217600000000001 2024-09-21T12:35:49.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:35:49.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030546+0000", 2024-09-21T12:35:49.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.212e-06 2024-09-21T12:35:49.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:35:49.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.150826+0000", 2024-09-21T12:35:49.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.120280194 2024-09-21T12:35:49.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:35:49.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.150950+0000", 2024-09-21T12:35:49.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012343099999999999 2024-09-21T12:35:49.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,6,7", 2024-09-21T12:35:49.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.151239+0000", 2024-09-21T12:35:49.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00028982299999999998 2024-09-21T12:35:49.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:35:49.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153075+0000", 2024-09-21T12:35:49.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001835723 2024-09-21T12:35:49.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153243+0000", 2024-09-21T12:35:49.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000167973 2024-09-21T12:35:49.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153273+0000", 2024-09-21T12:35:49.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0176999999999999e-05 2024-09-21T12:35:49.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.716 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.716 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.716 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153296+0000", 2024-09-21T12:35:49.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2816e-05 2024-09-21T12:35:49.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.716 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.716 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:35:49.716 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153327+0000", 2024-09-21T12:35:49.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1031999999999997e-05 2024-09-21T12:35:49.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.716 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.716 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:35:49.716 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153392+0000", 2024-09-21T12:35:49.717 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4827999999999997e-05 2024-09-21T12:35:49.717 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.717 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:35:49.717 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.717 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.717 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:235406 9.f 9:fff68ecc:::benchmark_data_smithi114_133214_object111372:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-09-21T12:35:49.717 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:56.217779+0000", 2024-09-21T12:35:49.717 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 233.40710406100001, 2024-09-21T12:35:49.717 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15964376999999999, 2024-09-21T12:35:49.717 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:35:49.717 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:35:49.717 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:35:49.717 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:35:49.717 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:35:49.718 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:35:49.718 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 235406 2024-09-21T12:35:49.718 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.718 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:35:49.718 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.718 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:35:49.718 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.217779+0000", 2024-09-21T12:35:49.718 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.718 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.718 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.718 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:35:49.718 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.217779+0000", 2024-09-21T12:35:49.718 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.719 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.719 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.719 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:35:49.719 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.217780+0000", 2024-09-21T12:35:49.719 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8000000000000005e-07 2024-09-21T12:35:49.719 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.719 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.719 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:35:49.719 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.217784+0000", 2024-09-21T12:35:49.719 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7349999999999998e-06 2024-09-21T12:35:49.719 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.719 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.719 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:35:49.720 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.217795+0000", 2024-09-21T12:35:49.720 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0934000000000001e-05 2024-09-21T12:35:49.720 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.720 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.720 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:35:49.720 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.217801+0000", 2024-09-21T12:35:49.720 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9399999999999999e-06 2024-09-21T12:35:49.720 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.720 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.720 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:35:49.720 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.223367+0000", 2024-09-21T12:35:49.720 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0055666259999999999 2024-09-21T12:35:49.720 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.720 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.721 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:35:49.721 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.223468+0000", 2024-09-21T12:35:49.721 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100368 2024-09-21T12:35:49.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.721 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:35:49.721 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.223558+0000", 2024-09-21T12:35:49.721 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9945999999999996e-05 2024-09-21T12:35:49.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.721 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.721 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.225567+0000", 2024-09-21T12:35:49.721 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020092069999999998 2024-09-21T12:35:49.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.722 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:35:49.722 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.227739+0000", 2024-09-21T12:35:49.722 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021725160000000002 2024-09-21T12:35:49.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.722 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.722 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.722 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.377351+0000", 2024-09-21T12:35:49.722 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.149611789 2024-09-21T12:35:49.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.722 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.722 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:35:49.722 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.377385+0000", 2024-09-21T12:35:49.722 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3680000000000003e-05 2024-09-21T12:35:49.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.723 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.723 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:35:49.723 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.377423+0000", 2024-09-21T12:35:49.723 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8248999999999999e-05 2024-09-21T12:35:49.723 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.723 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:35:49.723 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.723 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.723 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.723 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:235409 9.f 9:f26d45b2:::benchmark_data_smithi114_133214_object111375:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-09-21T12:35:49.723 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:56.218056+0000", 2024-09-21T12:35:49.723 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 233.406827593, 2024-09-21T12:35:49.723 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.16153879900000001, 2024-09-21T12:35:49.724 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:35:49.724 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:35:49.724 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:35:49.724 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:35:49.724 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:35:49.724 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:35:49.724 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 235409 2024-09-21T12:35:49.724 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.724 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:35:49.724 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.724 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:35:49.724 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.218056+0000", 2024-09-21T12:35:49.724 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.725 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.725 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.725 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:35:49.725 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.218056+0000", 2024-09-21T12:35:49.725 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.725 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.725 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.725 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:35:49.725 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.218056+0000", 2024-09-21T12:35:49.725 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9999999999999996e-07 2024-09-21T12:35:49.725 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.725 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.725 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:35:49.726 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.218060+0000", 2024-09-21T12:35:49.726 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6820000000000001e-06 2024-09-21T12:35:49.726 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.726 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.726 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:35:49.726 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.218072+0000", 2024-09-21T12:35:49.726 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1511999999999999e-05 2024-09-21T12:35:49.726 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.726 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.726 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:35:49.726 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.218078+0000", 2024-09-21T12:35:49.726 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0260000000000004e-06 2024-09-21T12:35:49.726 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.727 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.727 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:35:49.727 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.225599+0000", 2024-09-21T12:35:49.727 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0075208879999999999 2024-09-21T12:35:49.727 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.727 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.727 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:35:49.727 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.225706+0000", 2024-09-21T12:35:49.727 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010780099999999999 2024-09-21T12:35:49.727 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.727 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.727 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:35:49.727 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.225831+0000", 2024-09-21T12:35:49.727 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012434100000000001 2024-09-21T12:35:49.727 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.728 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.728 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:35:49.728 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.227745+0000", 2024-09-21T12:35:49.728 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0019139090000000001 2024-09-21T12:35:49.728 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.728 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.728 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.728 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.228076+0000", 2024-09-21T12:35:49.728 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00033095599999999998 2024-09-21T12:35:49.728 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.728 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.728 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.728 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.379487+0000", 2024-09-21T12:35:49.728 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15141185500000001 2024-09-21T12:35:49.729 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.729 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.729 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:35:49.729 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.379551+0000", 2024-09-21T12:35:49.729 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3971000000000001e-05 2024-09-21T12:35:49.729 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.729 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.729 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:35:49.729 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.379594+0000", 2024-09-21T12:35:49.729 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3058e-05 2024-09-21T12:35:49.729 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.729 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:35:49.729 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.730 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:235410 9.f 9:fc9ff039:::benchmark_data_smithi114_133214_object111376:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-09-21T12:35:49.730 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:56.219546+0000", 2024-09-21T12:35:49.730 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 233.40533701499999, 2024-09-21T12:35:49.730 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.160211877, 2024-09-21T12:35:49.730 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:35:49.730 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:35:49.730 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:35:49.730 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:35:49.730 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:35:49.730 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:35:49.730 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 235410 2024-09-21T12:35:49.731 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.731 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:35:49.731 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.731 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:35:49.731 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.219546+0000", 2024-09-21T12:35:49.731 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.731 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.731 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.731 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:35:49.731 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.219546+0000", 2024-09-21T12:35:49.731 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.731 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.731 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.731 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:35:49.732 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.219547+0000", 2024-09-21T12:35:49.732 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1429999999999999e-06 2024-09-21T12:35:49.732 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.732 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.732 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:35:49.732 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.219552+0000", 2024-09-21T12:35:49.732 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3170000000000003e-06 2024-09-21T12:35:49.732 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.732 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.732 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:35:49.732 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.219561+0000", 2024-09-21T12:35:49.732 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2119999999999992e-06 2024-09-21T12:35:49.732 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.732 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.733 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:35:49.733 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.219568+0000", 2024-09-21T12:35:49.733 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.764e-06 2024-09-21T12:35:49.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.733 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.733 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:35:49.733 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.226862+0000", 2024-09-21T12:35:49.733 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0072939019999999997 2024-09-21T12:35:49.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.733 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.733 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:35:49.733 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.226927+0000", 2024-09-21T12:35:49.733 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5598000000000005e-05 2024-09-21T12:35:49.734 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.734 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.734 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:35:49.734 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.227004+0000", 2024-09-21T12:35:49.734 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6323999999999996e-05 2024-09-21T12:35:49.734 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.734 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.734 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:35:49.734 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.227859+0000", 2024-09-21T12:35:49.734 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00085563900000000005 2024-09-21T12:35:49.734 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.734 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.734 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.734 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.372977+0000", 2024-09-21T12:35:49.735 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14511786199999999 2024-09-21T12:35:49.735 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.735 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.735 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.735 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.379660+0000", 2024-09-21T12:35:49.735 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0066829469999999998 2024-09-21T12:35:49.735 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.735 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.735 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:35:49.735 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.379717+0000", 2024-09-21T12:35:49.735 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6575e-05 2024-09-21T12:35:49.735 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.735 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.735 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:35:49.736 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.379758+0000", 2024-09-21T12:35:49.736 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1594000000000002e-05 2024-09-21T12:35:49.736 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.736 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:35:49.736 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.736 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.736 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.736 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:239971 9.f 9:f3d2bd83:::benchmark_data_smithi114_133214_object115937:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T12:35:49.736 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:59.801376+0000", 2024-09-21T12:35:49.736 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 229.82350677700001, 2024-09-21T12:35:49.736 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.024227555000000001, 2024-09-21T12:35:49.736 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:35:49.736 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:35:49.736 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:35:49.737 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:35:49.737 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:35:49.737 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:35:49.737 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 239971 2024-09-21T12:35:49.737 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.737 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:35:49.737 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.737 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:35:49.737 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.801376+0000", 2024-09-21T12:35:49.737 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.737 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.737 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.737 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:35:49.737 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.801376+0000", 2024-09-21T12:35:49.737 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.738 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.738 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:35:49.738 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.801377+0000", 2024-09-21T12:35:49.738 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6800000000000009e-07 2024-09-21T12:35:49.738 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.738 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:35:49.738 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.801381+0000", 2024-09-21T12:35:49.738 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7620000000000001e-06 2024-09-21T12:35:49.738 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.738 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:35:49.739 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.801388+0000", 2024-09-21T12:35:49.739 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0079999999999996e-06 2024-09-21T12:35:49.739 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.739 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.739 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:35:49.739 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.801392+0000", 2024-09-21T12:35:49.739 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1389999999999997e-06 2024-09-21T12:35:49.739 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.739 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.739 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:35:49.739 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.801421+0000", 2024-09-21T12:35:49.739 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8651000000000001e-05 2024-09-21T12:35:49.739 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.739 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.740 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:35:49.740 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.801513+0000", 2024-09-21T12:35:49.740 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1932e-05 2024-09-21T12:35:49.740 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.740 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.740 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:35:49.740 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.801587+0000", 2024-09-21T12:35:49.740 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4507999999999999e-05 2024-09-21T12:35:49.740 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.740 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.740 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:35:49.740 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.805590+0000", 2024-09-21T12:35:49.740 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0040025190000000004 2024-09-21T12:35:49.740 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.740 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.740 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.741 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.812648+0000", 2024-09-21T12:35:49.741 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0070584480000000002 2024-09-21T12:35:49.741 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.741 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.741 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.741 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.825504+0000", 2024-09-21T12:35:49.741 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.012855602000000001 2024-09-21T12:35:49.741 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.741 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.741 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:35:49.741 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.825558+0000", 2024-09-21T12:35:49.741 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4243999999999998e-05 2024-09-21T12:35:49.741 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.741 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.741 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:35:49.742 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.825604+0000", 2024-09-21T12:35:49.742 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5773999999999999e-05 2024-09-21T12:35:49.742 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.742 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:35:49.742 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.742 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.742 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.742 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:239975 9.f 9:f9b00e91:::benchmark_data_smithi114_133214_object115941:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T12:35:49.742 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:59.801401+0000", 2024-09-21T12:35:49.742 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 229.823482426, 2024-09-21T12:35:49.742 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.028372571999999999, 2024-09-21T12:35:49.742 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:35:49.742 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:35:49.742 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:35:49.742 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:35:49.743 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:35:49.743 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:35:49.743 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 239975 2024-09-21T12:35:49.743 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.743 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:35:49.743 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.743 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:35:49.743 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.801401+0000", 2024-09-21T12:35:49.743 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.743 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.743 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.743 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:35:49.743 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.801401+0000", 2024-09-21T12:35:49.744 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.744 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.744 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.744 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:35:49.744 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.801401+0000", 2024-09-21T12:35:49.744 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1100000000000001e-07 2024-09-21T12:35:49.744 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.744 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.744 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:35:49.744 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.801403+0000", 2024-09-21T12:35:49.744 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1500000000000002e-06 2024-09-21T12:35:49.744 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.744 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.744 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:35:49.745 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.801408+0000", 2024-09-21T12:35:49.745 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9949999999999996e-06 2024-09-21T12:35:49.745 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.745 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.745 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:35:49.745 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.801412+0000", 2024-09-21T12:35:49.745 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0049999999999998e-06 2024-09-21T12:35:49.745 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.745 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.745 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:35:49.745 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.803589+0000", 2024-09-21T12:35:49.745 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021761660000000002 2024-09-21T12:35:49.745 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.745 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.745 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:35:49.746 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.803690+0000", 2024-09-21T12:35:49.746 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010166399999999999 2024-09-21T12:35:49.746 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.746 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.746 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:35:49.746 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.803793+0000", 2024-09-21T12:35:49.746 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000103157 2024-09-21T12:35:49.746 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.746 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.746 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:35:49.746 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.805758+0000", 2024-09-21T12:35:49.746 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0019647219999999999 2024-09-21T12:35:49.747 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.747 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.747 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.747 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.812711+0000", 2024-09-21T12:35:49.747 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0069532550000000002 2024-09-21T12:35:49.747 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.747 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.747 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:35:49.747 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.829687+0000", 2024-09-21T12:35:49.747 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.016975253999999999 2024-09-21T12:35:49.747 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.747 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.747 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:35:49.748 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.829737+0000", 2024-09-21T12:35:49.748 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0813000000000003e-05 2024-09-21T12:35:49.748 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.748 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.748 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:35:49.748 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:59.829773+0000", 2024-09-21T12:35:49.748 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5979999999999998e-05 2024-09-21T12:35:49.748 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.748 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:35:49.748 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.748 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.748 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.748 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.15924.0:162923 10.6 e390/353 10:6f6ffe60:::benchmark_data_smithi114_161690_object35871:head v 390'10008, mlcod=390'10005)", 2024-09-21T12:35:49.749 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:34:36.146436+0000", 2024-09-21T12:35:49.749 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 73.478447461000002, 2024-09-21T12:35:49.749 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.027174997999999999, 2024-09-21T12:35:49.749 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:35:49.749 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:35:49.749 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:35:49.749 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:35:49.749 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.749 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:35:49.749 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:36.146436+0000", 2024-09-21T12:35:49.749 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.749 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.749 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.750 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:35:49.750 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:36.146436+0000", 2024-09-21T12:35:49.750 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.750 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.750 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.750 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:35:49.750 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:36.146437+0000", 2024-09-21T12:35:49.750 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2099999999999996e-07 2024-09-21T12:35:49.750 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.750 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.750 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:35:49.750 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:36.146442+0000", 2024-09-21T12:35:49.750 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7189999999999998e-06 2024-09-21T12:35:49.751 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.751 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.751 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:35:49.751 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:36.146454+0000", 2024-09-21T12:35:49.751 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2232000000000001e-05 2024-09-21T12:35:49.751 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.751 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.751 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:35:49.751 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:36.146462+0000", 2024-09-21T12:35:49.751 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1350000000000003e-06 2024-09-21T12:35:49.751 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.751 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.751 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:35:49.751 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:36.151309+0000", 2024-09-21T12:35:49.751 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0048475150000000002 2024-09-21T12:35:49.752 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.752 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.752 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:35:49.752 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:36.151356+0000", 2024-09-21T12:35:49.752 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7250999999999999e-05 2024-09-21T12:35:49.752 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.752 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.752 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:35:49.752 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:36.173465+0000", 2024-09-21T12:35:49.752 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.022108630000000001 2024-09-21T12:35:49.752 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.752 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.752 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:35:49.752 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:36.173611+0000", 2024-09-21T12:35:49.752 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000145795 2024-09-21T12:35:49.753 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.753 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:35:49.753 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.753 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.753 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.753 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.15924.0:233517 10.a e398/344 10:55ae3598:::benchmark_data_smithi114_161690_object106465:head v 398'14778, mlcod=398'14776)", 2024-09-21T12:35:49.753 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:35:23.076032+0000", 2024-09-21T12:35:49.753 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 26.548851525, 2024-09-21T12:35:49.753 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.020376898000000001, 2024-09-21T12:35:49.753 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:35:49.753 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:35:49.753 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:35:49.753 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:35:49.753 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.754 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:35:49.754 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:23.076032+0000", 2024-09-21T12:35:49.754 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.754 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.754 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:35:49.754 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:23.076032+0000", 2024-09-21T12:35:49.754 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:35:49.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.754 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.754 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:35:49.754 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:23.076033+0000", 2024-09-21T12:35:49.754 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9900000000000009e-07 2024-09-21T12:35:49.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.755 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.755 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:35:49.755 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:23.076038+0000", 2024-09-21T12:35:49.755 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7189999999999998e-06 2024-09-21T12:35:49.755 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.755 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.755 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:35:49.755 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:23.076049+0000", 2024-09-21T12:35:49.755 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0869e-05 2024-09-21T12:35:49.755 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.755 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.755 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:35:49.756 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:23.076056+0000", 2024-09-21T12:35:49.756 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4479999999999999e-06 2024-09-21T12:35:49.756 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.756 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.756 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:35:49.756 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:23.076111+0000", 2024-09-21T12:35:49.756 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5245999999999999e-05 2024-09-21T12:35:49.756 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.756 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.756 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:35:49.756 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:23.076133+0000", 2024-09-21T12:35:49.756 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1654e-05 2024-09-21T12:35:49.756 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.756 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.756 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:35:49.757 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:23.096337+0000", 2024-09-21T12:35:49.757 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.020204318999999998 2024-09-21T12:35:49.757 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:35:49.757 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:35:49.757 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:35:49.757 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:23.096409+0000", 2024-09-21T12:35:49.757 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1643999999999994e-05 2024-09-21T12:35:49.757 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.757 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:35:49.757 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.757 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:35:49.757 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:35:49.757 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:35:49.778 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:49.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:49.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:49.879 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:49.900 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:49.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:49.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:49.984 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:49 smithi114 ceph-mon[30324]: pgmap v787: 178 pgs: 5 creating+activating, 6 creating+peering, 3 unknown, 34 active+undersized, 14 active+undersized+degraded, 116 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 63/597 objects degraded (10.553%) 2024-09-21T12:35:49.984 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:49 smithi114 ceph-mon[30324]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T12:35:49.984 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:49 smithi114 ceph-mon[30324]: osdmap e407: 8 total, 7 up, 8 in 2024-09-21T12:35:49.984 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:49 smithi114 ceph-mon[30324]: 6.1d scrub starts 2024-09-21T12:35:49.985 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:49 smithi114 ceph-mon[30324]: 6.1d scrub ok 2024-09-21T12:35:49.985 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:49 smithi114 ceph-mon[40867]: pgmap v787: 178 pgs: 5 creating+activating, 6 creating+peering, 3 unknown, 34 active+undersized, 14 active+undersized+degraded, 116 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 63/597 objects degraded (10.553%) 2024-09-21T12:35:49.985 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:49 smithi114 ceph-mon[40867]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T12:35:49.985 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:49 smithi114 ceph-mon[40867]: osdmap e407: 8 total, 7 up, 8 in 2024-09-21T12:35:49.985 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:49 smithi114 ceph-mon[40867]: 6.1d scrub starts 2024-09-21T12:35:49.985 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:49 smithi114 ceph-mon[40867]: 6.1d scrub ok 2024-09-21T12:35:50.001 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:50.023 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:50.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:50.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:50.096 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:49 smithi183 ceph-mon[33277]: pgmap v787: 178 pgs: 5 creating+activating, 6 creating+peering, 3 unknown, 34 active+undersized, 14 active+undersized+degraded, 116 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 63/597 objects degraded (10.553%) 2024-09-21T12:35:50.096 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:49 smithi183 ceph-mon[33277]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T12:35:50.096 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:49 smithi183 ceph-mon[33277]: osdmap e407: 8 total, 7 up, 8 in 2024-09-21T12:35:50.096 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:49 smithi183 ceph-mon[33277]: 6.1d scrub starts 2024-09-21T12:35:50.096 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:49 smithi183 ceph-mon[33277]: 6.1d scrub ok 2024-09-21T12:35:50.125 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:50.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:50.148 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:50.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:50.249 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:50.271 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:50.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:50.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:50.272 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:49 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-a[30301]: 2024-09-21T12:35:49.953+0000 7f49fe685640 -1 mon.a@0(leader).osd e407 definitely_dead 0 2024-09-21T12:35:50.372 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:50.394 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:50.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:50.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:50.405 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:35:50.495 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:50.516 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:50.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:50.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:50.618 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:50.636 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T12:35:50.639 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:50.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:50.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:50.683 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:50.741 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:50.762 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:50.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:50.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:50.803 INFO:teuthology.orchestra.run.smithi114.stderr:marked out osd.7. 2024-09-21T12:35:50.804 INFO:teuthology.orchestra.run.smithi114.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-09-21T12:35:50.864 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:50.886 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:50.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:50.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:50.897 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:35:50.987 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:50.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:50 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1141700510' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-21T12:35:50.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:50 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-21T12:35:50.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:50 smithi183 ceph-mon[33277]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-21T12:35:50.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:50 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2059751061' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:35:50.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:50 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:35:51.009 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:51.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:51.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:51.058 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T12:35:51.079 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:50 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1141700510' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-21T12:35:51.079 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:50 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-21T12:35:51.080 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:50 smithi114 ceph-mon[30324]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-21T12:35:51.080 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:50 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2059751061' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:35:51.080 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:50 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:35:51.080 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:50 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1141700510' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-21T12:35:51.080 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:50 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-21T12:35:51.080 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:50 smithi114 ceph-mon[40867]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-21T12:35:51.080 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:50 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2059751061' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:35:51.080 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:50 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T12:35:51.111 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:51.133 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:51.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:51.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:51.234 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:51.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:51.258 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:51.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:51.313 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:35:51.313 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:35:51.330 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:35:51.331 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:35:51.331 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T12:35:51.353 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:35:51.354 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:35:51.359 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:51.370 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:35:51.370 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:35:51.380 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:51.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:51.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:51.387 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:35:51.387 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:35:51.403 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:35:51.403 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:35:51.404 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:35:51.404 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:35:51.420 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:35:51.420 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:35:51.481 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:51.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:51.504 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:51.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:51.605 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:51.627 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:51.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:51.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:51.729 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:51.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:51.751 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:51.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:51.852 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:51.873 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:51.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:51.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:51.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:51 smithi114 ceph-mon[40867]: pgmap v789: 178 pgs: 5 creating+activating, 6 creating+peering, 3 unknown, 34 active+undersized, 14 active+undersized+degraded, 116 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 887 B/s rd, 0 op/s; 63/597 objects degraded (10.553%) 2024-09-21T12:35:51.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:51 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-21T12:35:51.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:51 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T12:35:51.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:51 smithi114 ceph-mon[40867]: osdmap e408: 8 total, 7 up, 7 in 2024-09-21T12:35:51.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:51 smithi114 ceph-mon[40867]: 5.1d scrub starts 2024-09-21T12:35:51.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:51 smithi114 ceph-mon[40867]: 5.1d scrub ok 2024-09-21T12:35:51.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:51 smithi114 ceph-mon[40867]: 5.e scrub starts 2024-09-21T12:35:51.974 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:51 smithi114 ceph-mon[40867]: 5.e scrub ok 2024-09-21T12:35:51.975 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:51 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/218358342' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:35:51.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:51 smithi114 ceph-mon[30324]: pgmap v789: 178 pgs: 5 creating+activating, 6 creating+peering, 3 unknown, 34 active+undersized, 14 active+undersized+degraded, 116 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 887 B/s rd, 0 op/s; 63/597 objects degraded (10.553%) 2024-09-21T12:35:51.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:51 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-21T12:35:51.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:51 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T12:35:51.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:51 smithi114 ceph-mon[30324]: osdmap e408: 8 total, 7 up, 7 in 2024-09-21T12:35:51.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:51 smithi114 ceph-mon[30324]: 5.1d scrub starts 2024-09-21T12:35:51.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:51 smithi114 ceph-mon[30324]: 5.1d scrub ok 2024-09-21T12:35:51.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:51 smithi114 ceph-mon[30324]: 5.e scrub starts 2024-09-21T12:35:51.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:51 smithi114 ceph-mon[30324]: 5.e scrub ok 2024-09-21T12:35:51.975 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:51 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/218358342' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:35:51.976 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:51.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:51 smithi183 ceph-mon[33277]: pgmap v789: 178 pgs: 5 creating+activating, 6 creating+peering, 3 unknown, 34 active+undersized, 14 active+undersized+degraded, 116 active+clean; 583 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 887 B/s rd, 0 op/s; 63/597 objects degraded (10.553%) 2024-09-21T12:35:51.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:51 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-21T12:35:51.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:51 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T12:35:51.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:51 smithi183 ceph-mon[33277]: osdmap e408: 8 total, 7 up, 7 in 2024-09-21T12:35:51.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:51 smithi183 ceph-mon[33277]: 5.1d scrub starts 2024-09-21T12:35:51.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:51 smithi183 ceph-mon[33277]: 5.1d scrub ok 2024-09-21T12:35:51.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:51 smithi183 ceph-mon[33277]: 5.e scrub starts 2024-09-21T12:35:51.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:51 smithi183 ceph-mon[33277]: 5.e scrub ok 2024-09-21T12:35:51.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:51 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/218358342' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:35:51.996 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:51.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:51.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:52.098 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:52.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:52.122 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:52.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:52.130 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T12:35:52.223 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:52.244 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:52.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:52.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:52.345 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:52.369 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:52.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:52.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:52.417 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:35:52.471 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:52.492 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:52.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:52.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:52.581 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:52.594 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:52.615 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:52.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:52.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:52.688 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:35:52 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:35:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:35:52.716 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:52.738 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:52.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:52.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:52.811 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:52 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:35:52.811 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:52 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:35:52.811 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:52 smithi183 ceph-mon[33277]: 6.13 scrub starts 2024-09-21T12:35:52.811 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:52 smithi183 ceph-mon[33277]: 6.13 scrub ok 2024-09-21T12:35:52.811 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:52 smithi183 ceph-mon[33277]: osdmap e409: 8 total, 7 up, 7 in 2024-09-21T12:35:52.811 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:52 smithi183 ceph-mon[33277]: 3.f scrub starts 2024-09-21T12:35:52.811 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:52 smithi183 ceph-mon[33277]: 3.f scrub ok 2024-09-21T12:35:52.812 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:52 smithi183 ceph-mon[33277]: 1.10 scrub starts 2024-09-21T12:35:52.812 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:52 smithi183 ceph-mon[33277]: 1.10 scrub ok 2024-09-21T12:35:52.840 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:52.861 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:52.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:52.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:52.890 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:35:52.890 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:35:52.964 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:52.986 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:52.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:52.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:53.006 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:52 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:35:53.006 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:52 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:35:53.006 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:52 smithi114 ceph-mon[40867]: 6.13 scrub starts 2024-09-21T12:35:53.006 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:52 smithi114 ceph-mon[40867]: 6.13 scrub ok 2024-09-21T12:35:53.007 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:52 smithi114 ceph-mon[40867]: osdmap e409: 8 total, 7 up, 7 in 2024-09-21T12:35:53.007 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:52 smithi114 ceph-mon[40867]: 3.f scrub starts 2024-09-21T12:35:53.007 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:52 smithi114 ceph-mon[40867]: 3.f scrub ok 2024-09-21T12:35:53.007 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:52 smithi114 ceph-mon[40867]: 1.10 scrub starts 2024-09-21T12:35:53.007 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:52 smithi114 ceph-mon[40867]: 1.10 scrub ok 2024-09-21T12:35:53.007 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:52 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:35:53.007 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:52 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:35:53.007 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:52 smithi114 ceph-mon[30324]: 6.13 scrub starts 2024-09-21T12:35:53.007 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:52 smithi114 ceph-mon[30324]: 6.13 scrub ok 2024-09-21T12:35:53.007 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:52 smithi114 ceph-mon[30324]: osdmap e409: 8 total, 7 up, 7 in 2024-09-21T12:35:53.007 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:52 smithi114 ceph-mon[30324]: 3.f scrub starts 2024-09-21T12:35:53.007 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:52 smithi114 ceph-mon[30324]: 3.f scrub ok 2024-09-21T12:35:53.008 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:52 smithi114 ceph-mon[30324]: 1.10 scrub starts 2024-09-21T12:35:53.008 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:52 smithi114 ceph-mon[30324]: 1.10 scrub ok 2024-09-21T12:35:53.034 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T12:35:53.034 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T12:35:53.088 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:53.109 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:53.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:53.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:53.211 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:53.232 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:53.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:53.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:53.333 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:53.354 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:53.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:53.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:53.377 DEBUG:teuthology.orchestra.run.smithi114:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-09-21T12:35:53.397 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T12:35:53.398 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T12:35:53.456 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:53.478 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:53.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:53.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:53.499 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:hints = 1 2024-09-21T12:35:53.499 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T12:35:53.500 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Object prefix: benchmark_data_smithi114_187622 2024-09-21T12:35:53.500 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T12:35:53.500 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 0 0 0 0 0 0 - 0 2024-09-21T12:35:53.580 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:53.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:53.617 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:53.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:53.718 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:53.719 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T12:35:53.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:53.762 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:53.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:35:53.863 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:53.896 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:53.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:53.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:53.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:53 smithi183 ceph-mon[33277]: 1.15 scrub starts 2024-09-21T12:35:53.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:53 smithi183 ceph-mon[33277]: 1.15 scrub ok 2024-09-21T12:35:53.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:53 smithi183 ceph-mon[33277]: 4.8 scrub starts 2024-09-21T12:35:53.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:53 smithi183 ceph-mon[33277]: 4.8 scrub ok 2024-09-21T12:35:53.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:53 smithi183 ceph-mon[33277]: pgmap v792: 178 pgs: 5 peering, 5 creating+activating, 14 remapped+peering, 2 unknown, 29 active+undersized, 12 active+undersized+degraded, 111 active+clean; 662 KiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 16 KiB/s wr, 1 op/s; 58/600 objects degraded (9.667%); 5.6 KiB/s, 4 objects/s recovering 2024-09-21T12:35:53.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:53 smithi183 ceph-mon[33277]: Health check failed: Reduced data availability: 8 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-09-21T12:35:53.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:53 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 58/600 objects degraded (9.667%), 12 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-09-21T12:35:53.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:53 smithi183 ceph-mon[33277]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T12:35:53.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:53 smithi183 ceph-mon[33277]: osdmap e410: 8 total, 7 up, 7 in 2024-09-21T12:35:53.998 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:54.032 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:54.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:54.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:54.128 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:35:54.133 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:54.171 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:54.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:54.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:54.273 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:54.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:53 smithi114 ceph-mon[40867]: 1.15 scrub starts 2024-09-21T12:35:54.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:53 smithi114 ceph-mon[40867]: 1.15 scrub ok 2024-09-21T12:35:54.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:53 smithi114 ceph-mon[40867]: 4.8 scrub starts 2024-09-21T12:35:54.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:53 smithi114 ceph-mon[40867]: 4.8 scrub ok 2024-09-21T12:35:54.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:53 smithi114 ceph-mon[40867]: pgmap v792: 178 pgs: 5 peering, 5 creating+activating, 14 remapped+peering, 2 unknown, 29 active+undersized, 12 active+undersized+degraded, 111 active+clean; 662 KiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 16 KiB/s wr, 1 op/s; 58/600 objects degraded (9.667%); 5.6 KiB/s, 4 objects/s recovering 2024-09-21T12:35:54.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:53 smithi114 ceph-mon[40867]: Health check failed: Reduced data availability: 8 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-09-21T12:35:54.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:53 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 58/600 objects degraded (9.667%), 12 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-09-21T12:35:54.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:53 smithi114 ceph-mon[40867]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T12:35:54.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:53 smithi114 ceph-mon[40867]: osdmap e410: 8 total, 7 up, 7 in 2024-09-21T12:35:54.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:53 smithi114 ceph-mon[30324]: 1.15 scrub starts 2024-09-21T12:35:54.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:53 smithi114 ceph-mon[30324]: 1.15 scrub ok 2024-09-21T12:35:54.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:53 smithi114 ceph-mon[30324]: 4.8 scrub starts 2024-09-21T12:35:54.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:53 smithi114 ceph-mon[30324]: 4.8 scrub ok 2024-09-21T12:35:54.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:53 smithi114 ceph-mon[30324]: pgmap v792: 178 pgs: 5 peering, 5 creating+activating, 14 remapped+peering, 2 unknown, 29 active+undersized, 12 active+undersized+degraded, 111 active+clean; 662 KiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 16 KiB/s wr, 1 op/s; 58/600 objects degraded (9.667%); 5.6 KiB/s, 4 objects/s recovering 2024-09-21T12:35:54.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:53 smithi114 ceph-mon[30324]: Health check failed: Reduced data availability: 8 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-09-21T12:35:54.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:53 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 58/600 objects degraded (9.667%), 12 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-09-21T12:35:54.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:53 smithi114 ceph-mon[30324]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T12:35:54.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:53 smithi114 ceph-mon[30324]: osdmap e410: 8 total, 7 up, 7 in 2024-09-21T12:35:54.306 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:54.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:54.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:54.407 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:54.440 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:54.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:54.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:54.500 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 1 16 3837 3821 238.797 238.812 0.00486269 0.00417251 2024-09-21T12:35:54.543 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:54.574 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:54.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:54.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:54.676 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:54.712 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:54.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:54.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:54.732 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:35:54.732 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:35:54.815 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:54.830 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-21T12:35:54.848 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:54.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:54.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:54.950 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:54.981 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:54.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:54.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:54.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:54 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3322366219' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:35:54.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:54 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:35:55.035 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:35:55.072 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:54 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3322366219' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:35:55.072 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:54 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:35:55.073 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:54 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3322366219' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:35:55.073 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:54 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:35:55.083 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:55.113 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:55.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:55.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:55.214 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:55.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:55.262 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:55.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:55.363 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:55.392 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:55.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:55.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:55.440 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T12:35:55.493 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:55.500 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 2 16 7333 7317 228.639 218.5 0.00482963 0.00436573 2024-09-21T12:35:55.526 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:55.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:55.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:55.628 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:55.659 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:55.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:55.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:55.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:55.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:55.811 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:55.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:55.829 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:55.833 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:35:55.912 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:55.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:55.953 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:55.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:55.964 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:55 smithi114 ceph-mon[40867]: 1.1c scrub starts 2024-09-21T12:35:55.964 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:55 smithi114 ceph-mon[40867]: 1.1c scrub ok 2024-09-21T12:35:55.964 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:55 smithi114 ceph-mon[40867]: 4.15 scrub starts 2024-09-21T12:35:55.964 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:55 smithi114 ceph-mon[40867]: 4.15 scrub ok 2024-09-21T12:35:55.964 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:55 smithi114 ceph-mon[40867]: pgmap v794: 178 pgs: 7 active+clean+remapped, 5 peering, 20 remapped+peering, 2 unknown, 24 active+undersized, 9 active+undersized+degraded, 111 active+clean; 714 KiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 39 KiB/s wr, 2 op/s; 55/600 objects degraded (9.167%); 5/600 objects misplaced (0.833%); 14 KiB/s, 8 objects/s recovering 2024-09-21T12:35:55.964 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:55 smithi114 ceph-mon[40867]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:35:55.964 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:55 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:35:55.965 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:55 smithi114 ceph-mon[40867]: osdmap e411: 8 total, 7 up, 7 in 2024-09-21T12:35:55.965 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:55 smithi114 ceph-mon[40867]: 5.19 scrub starts 2024-09-21T12:35:55.965 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:55 smithi114 ceph-mon[30324]: 1.1c scrub starts 2024-09-21T12:35:55.965 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:55 smithi114 ceph-mon[30324]: 1.1c scrub ok 2024-09-21T12:35:55.965 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:55 smithi114 ceph-mon[30324]: 4.15 scrub starts 2024-09-21T12:35:55.965 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:55 smithi114 ceph-mon[30324]: 4.15 scrub ok 2024-09-21T12:35:55.965 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:55 smithi114 ceph-mon[30324]: pgmap v794: 178 pgs: 7 active+clean+remapped, 5 peering, 20 remapped+peering, 2 unknown, 24 active+undersized, 9 active+undersized+degraded, 111 active+clean; 714 KiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 39 KiB/s wr, 2 op/s; 55/600 objects degraded (9.167%); 5/600 objects misplaced (0.833%); 14 KiB/s, 8 objects/s recovering 2024-09-21T12:35:55.965 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:55 smithi114 ceph-mon[30324]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:35:55.965 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:55 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:35:55.965 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:55 smithi114 ceph-mon[30324]: osdmap e411: 8 total, 7 up, 7 in 2024-09-21T12:35:55.965 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:55 smithi114 ceph-mon[30324]: 5.19 scrub starts 2024-09-21T12:35:56.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:56.089 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:56.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:56.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:56.117 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:55 smithi183 ceph-mon[33277]: 1.1c scrub starts 2024-09-21T12:35:56.117 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:55 smithi183 ceph-mon[33277]: 1.1c scrub ok 2024-09-21T12:35:56.117 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:55 smithi183 ceph-mon[33277]: 4.15 scrub starts 2024-09-21T12:35:56.117 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:55 smithi183 ceph-mon[33277]: 4.15 scrub ok 2024-09-21T12:35:56.117 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:55 smithi183 ceph-mon[33277]: pgmap v794: 178 pgs: 7 active+clean+remapped, 5 peering, 20 remapped+peering, 2 unknown, 24 active+undersized, 9 active+undersized+degraded, 111 active+clean; 714 KiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 39 KiB/s wr, 2 op/s; 55/600 objects degraded (9.167%); 5/600 objects misplaced (0.833%); 14 KiB/s, 8 objects/s recovering 2024-09-21T12:35:56.117 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:55 smithi183 ceph-mon[33277]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:35:56.117 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:55 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:35:56.117 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:55 smithi183 ceph-mon[33277]: osdmap e411: 8 total, 7 up, 7 in 2024-09-21T12:35:56.117 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:55 smithi183 ceph-mon[33277]: 5.19 scrub starts 2024-09-21T12:35:56.191 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:56.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:56.231 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:56.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:56.332 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:56.371 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:56.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:56.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:56.424 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:35:56.424 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:35:56.473 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:56.494 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:35:56.494 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:35:56.494 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:56.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:56.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:56.500 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 3 15 10413 10398 216.604 192.562 0.0358385 0.00460444 2024-09-21T12:35:56.526 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:35:56.526 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:35:56.527 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T12:35:56.559 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:35:56.559 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:35:56.592 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:35:56.592 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:35:56.596 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:56.626 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:35:56.626 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:35:56.627 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:56.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:56.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:56.660 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:35:56.660 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:35:56.694 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:35:56.694 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:35:56.728 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:56.764 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:56.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:56.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:56.866 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:56.896 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:56.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:56.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:56.997 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:57.004 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 4] out_osds: [7] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-09-21T12:35:57.004 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:35:57.005 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-09-21T12:35:57.005 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd in 7 2024-09-21T12:35:57.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:56 smithi114 ceph-mon[30324]: 5.19 scrub ok 2024-09-21T12:35:57.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:56 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:35:57.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:56 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:35:57.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:56 smithi114 ceph-mon[30324]: osdmap e412: 8 total, 7 up, 7 in 2024-09-21T12:35:57.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:56 smithi114 ceph-mon[30324]: 6.14 scrub starts 2024-09-21T12:35:57.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:56 smithi114 ceph-mon[30324]: 6.14 scrub ok 2024-09-21T12:35:57.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:56 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2854033719' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:35:57.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:56 smithi114 ceph-mon[40867]: 5.19 scrub ok 2024-09-21T12:35:57.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:56 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:35:57.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:56 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:35:57.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:56 smithi114 ceph-mon[40867]: osdmap e412: 8 total, 7 up, 7 in 2024-09-21T12:35:57.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:56 smithi114 ceph-mon[40867]: 6.14 scrub starts 2024-09-21T12:35:57.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:56 smithi114 ceph-mon[40867]: 6.14 scrub ok 2024-09-21T12:35:57.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:56 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2854033719' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:35:57.027 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:57.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:57.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:57.129 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:57.158 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:57.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:57.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:57.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:56 smithi183 ceph-mon[33277]: 5.19 scrub ok 2024-09-21T12:35:57.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:56 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:35:57.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:56 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:35:57.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:56 smithi183 ceph-mon[33277]: osdmap e412: 8 total, 7 up, 7 in 2024-09-21T12:35:57.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:56 smithi183 ceph-mon[33277]: 6.14 scrub starts 2024-09-21T12:35:57.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:56 smithi183 ceph-mon[33277]: 6.14 scrub ok 2024-09-21T12:35:57.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:56 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2854033719' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:35:57.260 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:57.288 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:57.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:57.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:57.390 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:57.420 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:57.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:57.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:57.458 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T12:35:57.458 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T12:35:57.500 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 4 16 13220 13204 206.293 175.375 0.0034224 0.00483785 2024-09-21T12:35:57.521 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:57.535 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:57.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:57.564 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:57.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:57.665 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:57.683 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T12:35:57.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:57.700 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:57.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:57.728 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:35:57 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:35:57.777 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T12:35:57.801 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:57.848 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:57.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:57.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:57.949 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:57.980 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:57.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:57.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:58.082 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:58.115 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:58.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:58.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:58.176 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:35:58.217 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:35:58.255 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:58.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:58.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:35:58.357 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:58.387 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:58.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:58.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:58.433 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:58 smithi183 ceph-mon[33277]: pgmap v797: 178 pgs: 7 active+clean+remapped, 5 peering, 20 remapped+peering, 2 unknown, 24 active+undersized, 9 active+undersized+degraded, 111 active+clean; 714 KiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 6.1 KiB/s rd, 47 KiB/s wr, 6 op/s; 55/600 objects degraded (9.167%); 5/600 objects misplaced (0.833%); 17 KiB/s, 9 objects/s recovering 2024-09-21T12:35:58.433 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:58 smithi183 ceph-mon[33277]: osdmap e413: 8 total, 7 up, 7 in 2024-09-21T12:35:58.433 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:58 smithi183 ceph-mon[33277]: 1.f scrub starts 2024-09-21T12:35:58.433 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:58 smithi183 ceph-mon[33277]: 1.f scrub ok 2024-09-21T12:35:58.489 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:58.500 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 5 16 15735 15719 196.469 157.188 0.00170389 0.00505919 2024-09-21T12:35:58.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:58.518 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:58.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:58.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:58 smithi114 ceph-mon[30324]: pgmap v797: 178 pgs: 7 active+clean+remapped, 5 peering, 20 remapped+peering, 2 unknown, 24 active+undersized, 9 active+undersized+degraded, 111 active+clean; 714 KiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 6.1 KiB/s rd, 47 KiB/s wr, 6 op/s; 55/600 objects degraded (9.167%); 5/600 objects misplaced (0.833%); 17 KiB/s, 9 objects/s recovering 2024-09-21T12:35:58.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:58 smithi114 ceph-mon[30324]: osdmap e413: 8 total, 7 up, 7 in 2024-09-21T12:35:58.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:58 smithi114 ceph-mon[30324]: 1.f scrub starts 2024-09-21T12:35:58.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:58 smithi114 ceph-mon[30324]: 1.f scrub ok 2024-09-21T12:35:58.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:58 smithi114 ceph-mon[40867]: pgmap v797: 178 pgs: 7 active+clean+remapped, 5 peering, 20 remapped+peering, 2 unknown, 24 active+undersized, 9 active+undersized+degraded, 111 active+clean; 714 KiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 6.1 KiB/s rd, 47 KiB/s wr, 6 op/s; 55/600 objects degraded (9.167%); 5/600 objects misplaced (0.833%); 17 KiB/s, 9 objects/s recovering 2024-09-21T12:35:58.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:58 smithi114 ceph-mon[40867]: osdmap e413: 8 total, 7 up, 7 in 2024-09-21T12:35:58.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:58 smithi114 ceph-mon[40867]: 1.f scrub starts 2024-09-21T12:35:58.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:58 smithi114 ceph-mon[40867]: 1.f scrub ok 2024-09-21T12:35:58.619 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:58.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:58.657 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:58.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:58.758 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:58.796 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:58.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:58.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:58.802 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:35:58.802 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-21T12:35:58.802 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-21T12:35:58.802 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-21T12:35:58.802 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:35:58.898 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:35:58.930 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:58.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:58.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:35:58.999 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:58 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-a[30301]: 2024-09-21T12:35:58.562+0000 7f49fe685640 -1 mon.a@0(leader).osd e413 definitely_dead 0 2024-09-21T12:35:59.032 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:59.053 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:59.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:59.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:59.125 INFO:teuthology.orchestra.run.smithi114.stderr:marked in osd.7. 2024-09-21T12:35:59.155 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:59.177 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:59.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:59.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:59 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:35:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:59 smithi114 ceph-mon[40867]: 3.d scrub starts 2024-09-21T12:35:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:59 smithi114 ceph-mon[40867]: 3.d scrub ok 2024-09-21T12:35:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:59 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/486073292' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-21T12:35:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:35:59 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-21T12:35:59.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:59 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:35:59.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:59 smithi114 ceph-mon[30324]: 3.d scrub starts 2024-09-21T12:35:59.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:59 smithi114 ceph-mon[30324]: 3.d scrub ok 2024-09-21T12:35:59.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:59 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/486073292' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-21T12:35:59.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:35:59 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-21T12:35:59.279 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:35:59.300 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:59.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:59.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:35:59.401 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:35:59.423 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:59.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:59.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:35:59.459 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:35:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:59 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:35:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:59 smithi183 ceph-mon[33277]: 3.d scrub starts 2024-09-21T12:35:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:59 smithi183 ceph-mon[33277]: 3.d scrub ok 2024-09-21T12:35:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:59 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/486073292' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-21T12:35:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:35:59 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-21T12:35:59.501 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 6 16 17086 17070 177.796 84.4375 0.00148891 0.00528042 2024-09-21T12:35:59.524 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:59.546 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:59.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:59.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:59.647 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:35:59.670 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:35:59.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:59.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:35:59.755 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-09-21T12:35:59.771 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:35:59.795 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:59.798 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:59.816 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:59.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:59.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:35:59.918 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:35:59.939 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:35:59.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:59.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:00.041 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:00.067 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:00.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:00.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:00.168 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:00.191 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:00.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:00.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:00.293 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:00.315 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:00 smithi114 ceph-mon[30324]: pgmap v799: 178 pgs: 30 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 115 active+clean; 779 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 713 B/s rd, 136 MiB/s wr, 2.17k op/s; 3221/37962 objects degraded (8.485%); 1426/37962 objects misplaced (3.756%); 45 KiB/s, 3 objects/s recovering 2024-09-21T12:36:00.315 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:00 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 3221/37962 objects degraded (8.485%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:00.315 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:00 smithi114 ceph-mon[30324]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 12 pgs peering) 2024-09-21T12:36:00.315 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:00 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-21T12:36:00.315 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:00 smithi114 ceph-mon[30324]: osdmap e414: 8 total, 7 up, 8 in 2024-09-21T12:36:00.315 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:00 smithi114 ceph-mon[30324]: 3.10 scrub starts 2024-09-21T12:36:00.315 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:00 smithi114 ceph-mon[30324]: 3.10 scrub ok 2024-09-21T12:36:00.315 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:00 smithi114 ceph-mon[40867]: pgmap v799: 178 pgs: 30 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 115 active+clean; 779 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 713 B/s rd, 136 MiB/s wr, 2.17k op/s; 3221/37962 objects degraded (8.485%); 1426/37962 objects misplaced (3.756%); 45 KiB/s, 3 objects/s recovering 2024-09-21T12:36:00.315 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:00 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 3221/37962 objects degraded (8.485%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:00.315 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:00 smithi114 ceph-mon[40867]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 12 pgs peering) 2024-09-21T12:36:00.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:00 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-21T12:36:00.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:00 smithi114 ceph-mon[40867]: osdmap e414: 8 total, 7 up, 8 in 2024-09-21T12:36:00.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:00 smithi114 ceph-mon[40867]: 3.10 scrub starts 2024-09-21T12:36:00.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:00 smithi114 ceph-mon[40867]: 3.10 scrub ok 2024-09-21T12:36:00.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:00.317 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:00.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:00.417 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:00.439 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:00.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:00.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:00.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:00 smithi183 ceph-mon[33277]: pgmap v799: 178 pgs: 30 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 115 active+clean; 779 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 713 B/s rd, 136 MiB/s wr, 2.17k op/s; 3221/37962 objects degraded (8.485%); 1426/37962 objects misplaced (3.756%); 45 KiB/s, 3 objects/s recovering 2024-09-21T12:36:00.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:00 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 3221/37962 objects degraded (8.485%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:00.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:00 smithi183 ceph-mon[33277]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 12 pgs peering) 2024-09-21T12:36:00.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:00 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-21T12:36:00.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:00 smithi183 ceph-mon[33277]: osdmap e414: 8 total, 7 up, 8 in 2024-09-21T12:36:00.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:00 smithi183 ceph-mon[33277]: 3.10 scrub starts 2024-09-21T12:36:00.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:00 smithi183 ceph-mon[33277]: 3.10 scrub ok 2024-09-21T12:36:00.501 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 7 16 17086 17070 152.397 0 - 0.00528042 2024-09-21T12:36:00.534 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:36:00.534 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:36:00.541 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:00.544 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:36:00.544 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:36:00.544 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T12:36:00.560 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:36:00.560 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:36:00.567 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:00.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:00.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:00.577 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:36:00.577 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:36:00.594 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:36:00.594 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:36:00.611 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:36:00.611 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:36:00.628 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:36:00.628 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:36:00.669 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:00.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:00.691 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:00.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:00.792 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:00.813 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:00.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:00.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:00.914 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:00.935 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:00.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:00.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:01.037 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:01.062 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:01.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:01.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:01.110 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-21T12:36:01.164 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:01.196 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:01.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:01.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:01.297 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:01.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:01.334 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:01.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:01.436 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:01.474 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:01.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:01.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:01 smithi183 ceph-mon[33277]: 4.1d scrub starts 2024-09-21T12:36:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:01 smithi183 ceph-mon[33277]: 4.1d scrub ok 2024-09-21T12:36:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:01 smithi183 ceph-mon[33277]: 6.10 scrub starts 2024-09-21T12:36:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:01 smithi183 ceph-mon[33277]: 6.10 scrub ok 2024-09-21T12:36:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:01 smithi183 ceph-mon[33277]: osdmap e415: 8 total, 7 up, 8 in 2024-09-21T12:36:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:01 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3126912339' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:36:01.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:01 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/4041320409' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:36:01.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:01 smithi183 ceph-mon[33277]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:36:01.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:01 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/4041320409' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:36:01.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:01 smithi114 ceph-mon[40867]: 4.1d scrub starts 2024-09-21T12:36:01.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:01 smithi114 ceph-mon[40867]: 4.1d scrub ok 2024-09-21T12:36:01.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:01 smithi114 ceph-mon[40867]: 6.10 scrub starts 2024-09-21T12:36:01.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:01 smithi114 ceph-mon[40867]: 6.10 scrub ok 2024-09-21T12:36:01.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:01 smithi114 ceph-mon[40867]: osdmap e415: 8 total, 7 up, 8 in 2024-09-21T12:36:01.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:01 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3126912339' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:36:01.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:01 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/4041320409' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:36:01.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:01 smithi114 ceph-mon[40867]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:36:01.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:01 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/4041320409' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:36:01.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:01 smithi114 ceph-mon[30324]: 4.1d scrub starts 2024-09-21T12:36:01.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:01 smithi114 ceph-mon[30324]: 4.1d scrub ok 2024-09-21T12:36:01.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:01 smithi114 ceph-mon[30324]: 6.10 scrub starts 2024-09-21T12:36:01.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:01 smithi114 ceph-mon[30324]: 6.10 scrub ok 2024-09-21T12:36:01.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:01 smithi114 ceph-mon[30324]: osdmap e415: 8 total, 7 up, 8 in 2024-09-21T12:36:01.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:01 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3126912339' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:36:01.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:01 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4041320409' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:36:01.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:01 smithi114 ceph-mon[30324]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:36:01.495 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:01 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4041320409' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:36:01.501 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 8 16 17824 17808 139.113 23.0625 0.00466298 0.00717965 2024-09-21T12:36:01.547 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T12:36:01.547 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T12:36:01.576 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:01.621 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:01.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:01.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:01.723 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:01.744 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:01.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:01.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:01.846 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:01.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:01.882 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:01.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:01.983 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:02.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:02.017 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:02.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:02.095 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T12:36:02.118 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:02.154 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:02.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:02.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:02.256 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:02.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:02.291 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:02.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:02.392 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:02.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:02.436 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:02.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:02.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:02 smithi183 ceph-mon[33277]: pgmap v802: 178 pgs: 30 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 115 active+clean; 779 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 165 MiB/s wr, 2.64k op/s; 3221/37962 objects degraded (8.485%); 1426/37962 objects misplaced (3.756%); 49 KiB/s, 1 objects/s recovering 2024-09-21T12:36:02.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:02 smithi183 ceph-mon[33277]: 6.7 scrub starts 2024-09-21T12:36:02.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:02 smithi183 ceph-mon[33277]: 6.7 scrub ok 2024-09-21T12:36:02.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:02 smithi183 ceph-mon[33277]: 1.10 scrub starts 2024-09-21T12:36:02.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:02 smithi183 ceph-mon[33277]: 1.10 scrub ok 2024-09-21T12:36:02.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:02 smithi183 ceph-mon[33277]: 5.2 scrub starts 2024-09-21T12:36:02.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:02 smithi183 ceph-mon[33277]: 5.2 scrub ok 2024-09-21T12:36:02.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:02 smithi183 ceph-mon[33277]: osdmap e416: 8 total, 7 up, 8 in 2024-09-21T12:36:02.501 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 9 16 19966 19950 138.529 133.875 0.00256199 0.0072104 2024-09-21T12:36:02.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:02 smithi114 ceph-mon[40867]: pgmap v802: 178 pgs: 30 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 115 active+clean; 779 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 165 MiB/s wr, 2.64k op/s; 3221/37962 objects degraded (8.485%); 1426/37962 objects misplaced (3.756%); 49 KiB/s, 1 objects/s recovering 2024-09-21T12:36:02.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:02 smithi114 ceph-mon[40867]: 6.7 scrub starts 2024-09-21T12:36:02.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:02 smithi114 ceph-mon[40867]: 6.7 scrub ok 2024-09-21T12:36:02.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:02 smithi114 ceph-mon[40867]: 1.10 scrub starts 2024-09-21T12:36:02.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:02 smithi114 ceph-mon[40867]: 1.10 scrub ok 2024-09-21T12:36:02.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:02 smithi114 ceph-mon[40867]: 5.2 scrub starts 2024-09-21T12:36:02.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:02 smithi114 ceph-mon[40867]: 5.2 scrub ok 2024-09-21T12:36:02.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:02 smithi114 ceph-mon[40867]: osdmap e416: 8 total, 7 up, 8 in 2024-09-21T12:36:02.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:02 smithi114 ceph-mon[30324]: pgmap v802: 178 pgs: 30 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 115 active+clean; 779 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 165 MiB/s wr, 2.64k op/s; 3221/37962 objects degraded (8.485%); 1426/37962 objects misplaced (3.756%); 49 KiB/s, 1 objects/s recovering 2024-09-21T12:36:02.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:02 smithi114 ceph-mon[30324]: 6.7 scrub starts 2024-09-21T12:36:02.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:02 smithi114 ceph-mon[30324]: 6.7 scrub ok 2024-09-21T12:36:02.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:02 smithi114 ceph-mon[30324]: 1.10 scrub starts 2024-09-21T12:36:02.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:02 smithi114 ceph-mon[30324]: 1.10 scrub ok 2024-09-21T12:36:02.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:02 smithi114 ceph-mon[30324]: 5.2 scrub starts 2024-09-21T12:36:02.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:02 smithi114 ceph-mon[30324]: 5.2 scrub ok 2024-09-21T12:36:02.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:02 smithi114 ceph-mon[30324]: osdmap e416: 8 total, 7 up, 8 in 2024-09-21T12:36:02.530 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:36:02.537 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:02.572 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:02.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:02.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:02.673 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:02.708 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:02.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:02.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:02.809 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:02.850 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:02.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:02.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:02.952 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:02.985 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:02.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:02.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:03.024 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:36:02 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:36:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:36:03.087 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:03.124 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:03.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:03.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:03.159 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:36:03.159 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:36:03.227 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:03.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:03 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:36:03.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:03.262 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:03.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:03.363 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:03.397 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:03.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:03.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:03.498 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:03.501 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 10 16 22297 22281 139.244 145.688 0.00617154 0.00717477 2024-09-21T12:36:03.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:03 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:36:03.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:03 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:36:03.545 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:03.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:03.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:03.548 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:36:03.646 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:03.677 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:03.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:03.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:03.779 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:03.820 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:03.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:03.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:03.920 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T12:36:03.922 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:03.957 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:03.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:03.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:03.984 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:36:04.058 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:04.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:04.104 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:04.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:04.205 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:04.240 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:04.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:04.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:04.241 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T12:36:04.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:04 smithi114 ceph-mon[30324]: pgmap v804: 178 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 23 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 119 active+clean; 1009 MiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 114 MiB/s wr, 1.83k op/s; 5196/49008 objects degraded (10.602%); 4381/49008 objects misplaced (8.939%); 1.2 MiB/s, 21 objects/s recovering 2024-09-21T12:36:04.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:04 smithi114 ceph-mon[40867]: pgmap v804: 178 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 23 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 119 active+clean; 1009 MiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 114 MiB/s wr, 1.83k op/s; 5196/49008 objects degraded (10.602%); 4381/49008 objects misplaced (8.939%); 1.2 MiB/s, 21 objects/s recovering 2024-09-21T12:36:04.341 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:04.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:04.374 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:04.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:04.474 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:04.510 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:04 smithi183 ceph-mon[33277]: pgmap v804: 178 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 23 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 119 active+clean; 1009 MiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 114 MiB/s wr, 1.83k op/s; 5196/49008 objects degraded (10.602%); 4381/49008 objects misplaced (8.939%); 1.2 MiB/s, 21 objects/s recovering 2024-09-21T12:36:04.510 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 11 15 24210 24195 137.459 119.625 0.00442416 0.00726874 2024-09-21T12:36:04.551 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:04.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:04.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:04.653 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:04.690 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:04.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:04.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:04.756 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 4, 7] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-09-21T12:36:04.756 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:36:04.756 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-21T12:36:04.756 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-21T12:36:04.792 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:04.824 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:04.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:04.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:04.926 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:04.961 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:04.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:04.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:05.062 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:05.103 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:05.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:05.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:05.205 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:05.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:05.251 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:05.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:05.351 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:05.387 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:05.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:05.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:05.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:05 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 6040/59340 objects degraded (10.179%), 14 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:05.489 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:05.501 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 12 16 25988 25972 135.258 111.062 0.00467974 0.00737162 2024-09-21T12:36:05.511 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:05.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:05.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:05.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:05 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 6040/59340 objects degraded (10.179%), 14 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:05.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:05 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 6040/59340 objects degraded (10.179%), 14 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:05.612 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:05.646 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:05.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:05.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:05.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:05.781 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:05.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:05.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:05.883 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:05.905 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:05.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:05.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:06.007 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:06.041 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:06.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:06.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:06.143 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:06.174 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:06.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:06.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:06.276 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:06.312 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:06.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:06.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:06.414 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:06.448 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:06.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:06.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:06.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:06 smithi183 ceph-mon[33277]: pgmap v805: 178 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 122 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s; 6040/59340 objects degraded (10.179%); 5271/59340 objects misplaced (8.883%); 2.6 MiB/s, 47 objects/s recovering 2024-09-21T12:36:06.501 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 13 16 27627 27611 132.733 102.438 0.0073885 0.00750575 2024-09-21T12:36:06.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:06 smithi114 ceph-mon[40867]: pgmap v805: 178 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 122 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s; 6040/59340 objects degraded (10.179%); 5271/59340 objects misplaced (8.883%); 2.6 MiB/s, 47 objects/s recovering 2024-09-21T12:36:06.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:06 smithi114 ceph-mon[30324]: pgmap v805: 178 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 122 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s; 6040/59340 objects degraded (10.179%); 5271/59340 objects misplaced (8.883%); 2.6 MiB/s, 47 objects/s recovering 2024-09-21T12:36:06.550 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:06.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:06.590 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:06.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:06.691 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:06.725 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:06.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:06.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:06.827 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:06.860 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:06.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:06.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:06.961 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:06.996 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:06.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:06.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:07.098 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:07.132 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:07.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:07.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:07.234 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:07.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:07.271 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:07.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:07.372 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:07.398 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:07.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:07.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:07.500 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:07.501 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 14 16 29122 29106 129.925 93.4375 0.00189984 0.007596 2024-09-21T12:36:07.523 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:07.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:07.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:07.625 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:07.649 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:07.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:07.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:07.750 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:07.775 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:07.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:07.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:07.876 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:07.901 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:07.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:07.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:07.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:36:07 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:36:08.002 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:08.027 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:08.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:08.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:08.128 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:08.165 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:08.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:08.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:08.266 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:08.287 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:08.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:08.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:08.389 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:08.424 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:08.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:08.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:08.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:08 smithi183 ceph-mon[33277]: pgmap v806: 178 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 122 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 60 MiB/s wr, 957 op/s; 6040/59340 objects degraded (10.179%); 5271/59340 objects misplaced (8.883%); 2.1 MiB/s, 38 objects/s recovering 2024-09-21T12:36:08.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:08 smithi183 ceph-mon[33277]: osdmap e417: 8 total, 7 up, 8 in 2024-09-21T12:36:08.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:08 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:36:08.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:08 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:36:08.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:08 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3694016354' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:36:08.501 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 15 16 29796 29780 124.071 42.125 0.00483075 0.00805371 2024-09-21T12:36:08.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:08 smithi114 ceph-mon[30324]: pgmap v806: 178 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 122 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 60 MiB/s wr, 957 op/s; 6040/59340 objects degraded (10.179%); 5271/59340 objects misplaced (8.883%); 2.1 MiB/s, 38 objects/s recovering 2024-09-21T12:36:08.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:08 smithi114 ceph-mon[30324]: osdmap e417: 8 total, 7 up, 8 in 2024-09-21T12:36:08.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:08 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:36:08.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:08 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:36:08.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:08 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3694016354' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:36:08.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:08 smithi114 ceph-mon[40867]: pgmap v806: 178 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 122 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 60 MiB/s wr, 957 op/s; 6040/59340 objects degraded (10.179%); 5271/59340 objects misplaced (8.883%); 2.1 MiB/s, 38 objects/s recovering 2024-09-21T12:36:08.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:08 smithi114 ceph-mon[40867]: osdmap e417: 8 total, 7 up, 8 in 2024-09-21T12:36:08.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:08 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:36:08.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:08 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:36:08.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:08 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3694016354' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:36:08.526 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:08.548 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:08.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:08.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:08.650 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:08.684 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:08.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:08.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:08.747 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:36:08.747 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:36:08.747 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:36:08.747 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:36:08.747 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:36:08.747 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:36:08.748 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:36:08.748 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:36:08.748 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:36:08.748 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T12:36:08.748 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:36:08.748 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:36:08.748 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:36:08.748 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:36:08.748 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:36:08.786 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:08.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:08.808 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:08.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:08.909 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:08.942 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:08.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:08.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:09.044 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:09.074 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:09.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:09.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:09.175 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:09.199 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:09.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:09.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:09.301 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:09.336 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:09.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:09.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:09.437 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:09.459 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:09.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:09.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:09.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:09 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:36:09.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:09 smithi183 ceph-mon[33277]: osdmap e418: 8 total, 7 up, 8 in 2024-09-21T12:36:09.501 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 16 16 31405 31389 122.602 100.562 0.00210716 0.00810906 2024-09-21T12:36:09.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:09 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:36:09.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:09 smithi114 ceph-mon[40867]: osdmap e418: 8 total, 7 up, 8 in 2024-09-21T12:36:09.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:09 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:36:09.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:09 smithi114 ceph-mon[30324]: osdmap e418: 8 total, 7 up, 8 in 2024-09-21T12:36:09.561 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:09.596 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:09.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:09.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:09.698 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:09.733 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:09.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:09.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:09.835 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:09.877 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:09.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:09.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:09.978 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:10.015 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:10.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:10.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:10.117 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:10.138 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:10.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:10.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:10.240 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:10.277 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:10.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:10.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:10.379 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:10.401 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:10.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:10.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:10.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:10 smithi183 ceph-mon[33277]: pgmap v809: 178 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 122 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 136 MiB/s wr, 2.17k op/s; 8247/86406 objects degraded (9.544%); 4622/86406 objects misplaced (5.349%); 18 MiB/s, 289 objects/s recovering 2024-09-21T12:36:10.501 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 17 16 33107 33091 121.646 106.375 0.00735599 0.00821473 2024-09-21T12:36:10.502 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:10.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:10 smithi114 ceph-mon[40867]: pgmap v809: 178 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 122 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 136 MiB/s wr, 2.17k op/s; 8247/86406 objects degraded (9.544%); 4622/86406 objects misplaced (5.349%); 18 MiB/s, 289 objects/s recovering 2024-09-21T12:36:10.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:10 smithi114 ceph-mon[30324]: pgmap v809: 178 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 122 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 136 MiB/s wr, 2.17k op/s; 8247/86406 objects degraded (9.544%); 4622/86406 objects misplaced (5.349%); 18 MiB/s, 289 objects/s recovering 2024-09-21T12:36:10.536 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:10.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:10.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:10.638 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:10.660 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:10.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:10.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:10.762 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:10.795 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:10.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:10.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:10.897 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:10.932 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:10.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:10.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:11.034 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:11.069 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:11.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:11.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:11.170 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:11.200 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:11.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:11.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:11.302 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:11.324 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:11.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:11.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:11.426 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:11.448 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:11.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:11.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:11.502 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 18 16 34455 34439 119.568 84.25 0.00179047 0.00822037 2024-09-21T12:36:11.549 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:11.571 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:11.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:11.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:11.672 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:11.694 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:11.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:11.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:11.796 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:11.818 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:11.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:11.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:11.919 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:11.941 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:11.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:11.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:12.043 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:12.064 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:12.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:12.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:12.166 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:12.187 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:12.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:12.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:12.289 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:12.310 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:12.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:12.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:12.412 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:12.435 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:12.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:12.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:12 smithi183 ceph-mon[33277]: pgmap v810: 178 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 122 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 97 MiB/s wr, 1.56k op/s; 8247/86406 objects degraded (9.544%); 4622/86406 objects misplaced (5.349%); 16 MiB/s, 253 objects/s recovering 2024-09-21T12:36:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:12 smithi183 ceph-mon[33277]: osdmap e419: 8 total, 7 up, 8 in 2024-09-21T12:36:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:12 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 8247/86406 objects degraded (9.544%), 14 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:12 smithi183 ceph-mon[33277]: osdmap e420: 8 total, 7 up, 8 in 2024-09-21T12:36:12.502 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 19 16 34535 34519 113.538 5 0.00202687 0.0086461 2024-09-21T12:36:12.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:12 smithi114 ceph-mon[40867]: pgmap v810: 178 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 122 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 97 MiB/s wr, 1.56k op/s; 8247/86406 objects degraded (9.544%); 4622/86406 objects misplaced (5.349%); 16 MiB/s, 253 objects/s recovering 2024-09-21T12:36:12.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:12 smithi114 ceph-mon[40867]: osdmap e419: 8 total, 7 up, 8 in 2024-09-21T12:36:12.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:12 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 8247/86406 objects degraded (9.544%), 14 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:12.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:12 smithi114 ceph-mon[40867]: osdmap e420: 8 total, 7 up, 8 in 2024-09-21T12:36:12.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:12 smithi114 ceph-mon[30324]: pgmap v810: 178 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 122 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 97 MiB/s wr, 1.56k op/s; 8247/86406 objects degraded (9.544%); 4622/86406 objects misplaced (5.349%); 16 MiB/s, 253 objects/s recovering 2024-09-21T12:36:12.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:12 smithi114 ceph-mon[30324]: osdmap e419: 8 total, 7 up, 8 in 2024-09-21T12:36:12.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:12 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 8247/86406 objects degraded (9.544%), 14 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:12.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:12 smithi114 ceph-mon[30324]: osdmap e420: 8 total, 7 up, 8 in 2024-09-21T12:36:12.537 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:12.562 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:12.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:12.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:12.664 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:12.689 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:12.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:12.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:12.790 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:12.814 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:12.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:12.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:12.916 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:12.939 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:12.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:12.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:13.024 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:36:12 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:36:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:36:13.041 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:13.065 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:13.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:13.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:13.166 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:13.201 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:13.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:13.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:13.302 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:13.332 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:13.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:13.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:13.434 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:13.468 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:13.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:13.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:13.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:13 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:36:13.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:13 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:36:13.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:13 smithi183 ceph-mon[33277]: osdmap e421: 8 total, 7 up, 8 in 2024-09-21T12:36:13.502 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-21T12:36:13.501947+0000 min lat: 0.00111509 max lat: 2.08308 avg lat: 0.00906902 2024-09-21T12:36:13.502 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T12:36:13.502 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 20 16 35284 35268 110.202 46.8125 0.0024627 0.00906902 2024-09-21T12:36:13.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:13 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:36:13.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:13 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:36:13.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:13 smithi114 ceph-mon[30324]: osdmap e421: 8 total, 7 up, 8 in 2024-09-21T12:36:13.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:13 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:36:13.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:13 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:36:13.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:13 smithi114 ceph-mon[40867]: osdmap e421: 8 total, 7 up, 8 in 2024-09-21T12:36:13.570 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:13.601 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:13.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:13.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:13.702 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:13.733 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:13.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:13.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:13.835 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:13.870 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:13.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:13.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:13.972 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:14.006 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:14.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:14.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:14.107 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:14.140 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:14.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:14.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:14.242 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:14.275 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:14.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:14.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:14.377 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:14.398 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:14.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:14.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:14.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:14 smithi183 ceph-mon[33277]: pgmap v813: 178 pgs: 1 creating+peering, 2 active+undersized+remapped, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 123 active+clean; 2.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 939 B/s rd, 87 MiB/s wr, 1.40k op/s; 7982/96822 objects degraded (8.244%); 4352/96822 objects misplaced (4.495%); 30 MiB/s, 481 objects/s recovering 2024-09-21T12:36:14.500 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:14.502 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 21 16 37168 37152 110.56 117.75 0.00763941 0.00904045 2024-09-21T12:36:14.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:14 smithi114 ceph-mon[30324]: pgmap v813: 178 pgs: 1 creating+peering, 2 active+undersized+remapped, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 123 active+clean; 2.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 939 B/s rd, 87 MiB/s wr, 1.40k op/s; 7982/96822 objects degraded (8.244%); 4352/96822 objects misplaced (4.495%); 30 MiB/s, 481 objects/s recovering 2024-09-21T12:36:14.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:14 smithi114 ceph-mon[40867]: pgmap v813: 178 pgs: 1 creating+peering, 2 active+undersized+remapped, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 123 active+clean; 2.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 939 B/s rd, 87 MiB/s wr, 1.40k op/s; 7982/96822 objects degraded (8.244%); 4352/96822 objects misplaced (4.495%); 30 MiB/s, 481 objects/s recovering 2024-09-21T12:36:14.534 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:14.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:14.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:14.636 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:14.670 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:14.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:14.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:14.771 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:14.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:14.809 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:14.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:14.910 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:14.937 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:14.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:14.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:15.039 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:15.071 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:15.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:15.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:15.174 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:15.195 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:15.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:15.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:15.296 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:15.328 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:15.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:15.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:15.430 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:15.450 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:15.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:15.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:15.502 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 22 16 38826 38810 110.245 103.625 0.00199731 0.00903283 2024-09-21T12:36:15.551 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:15.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:15.586 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:15.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:15.687 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:15.724 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:15.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:15.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:15.825 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:15.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:15.859 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:15.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:15.960 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:15.984 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:15.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:15.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:16.086 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:16.119 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:16.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:16.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:16.220 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:16.251 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:16.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:16.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:16.353 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:16.387 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:16.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:16.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:16.390 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:16 smithi114 ceph-mon[30324]: pgmap v815: 178 pgs: 1 creating+peering, 2 active+undersized+remapped, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 123 active+clean; 2.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 985 op/s; 8686/104133 objects degraded (8.341%); 4352/104133 objects misplaced (4.179%); 22 MiB/s, 347 objects/s recovering 2024-09-21T12:36:16.390 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:16 smithi114 ceph-mon[40867]: pgmap v815: 178 pgs: 1 creating+peering, 2 active+undersized+remapped, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 123 active+clean; 2.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 985 op/s; 8686/104133 objects degraded (8.341%); 4352/104133 objects misplaced (4.179%); 22 MiB/s, 347 objects/s recovering 2024-09-21T12:36:16.448 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T12:36:16.449 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T12:36:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:16 smithi183 ceph-mon[33277]: pgmap v815: 178 pgs: 1 creating+peering, 2 active+undersized+remapped, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 123 active+clean; 2.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 985 op/s; 8686/104133 objects degraded (8.341%); 4352/104133 objects misplaced (4.179%); 22 MiB/s, 347 objects/s recovering 2024-09-21T12:36:16.489 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:16.503 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 23 16 40809 40793 110.836 123.938 0.00681968 0.00901707 2024-09-21T12:36:16.536 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:36:16.542 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:16.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:16.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:16.547 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:36:16.553 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:36:16.644 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:16.688 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:16.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:16.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:16.790 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:16.828 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:16.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:16.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:16.930 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:16.966 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:16.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:16.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:17.068 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:17.108 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:17.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:17.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:17.210 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:17.241 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:17.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:17.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:17.283 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:17 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 8686/104133 objects degraded (8.341%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:17.283 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:17 smithi114 ceph-mon[30324]: osdmap e422: 8 total, 7 up, 8 in 2024-09-21T12:36:17.283 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:17 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 8686/104133 objects degraded (8.341%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:17.283 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:17 smithi114 ceph-mon[40867]: osdmap e422: 8 total, 7 up, 8 in 2024-09-21T12:36:17.343 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:17.375 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:17.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:17.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:17.476 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:17.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:17 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 8686/104133 objects degraded (8.341%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:17.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:17 smithi183 ceph-mon[33277]: osdmap e422: 8 total, 7 up, 8 in 2024-09-21T12:36:17.503 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 24 16 42797 42781 111.395 124.25 0.00783408 0.0089713 2024-09-21T12:36:17.508 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:17.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:17.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:17.610 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:17.640 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:17.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:17.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:17.742 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:17.771 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:17.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:17.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:17.873 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:17.900 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:17.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:17.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:17.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:36:17 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:36:18.002 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:18.050 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:18.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:18.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:18.152 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:18.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:18.200 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:18.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:18.224 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:36:18.224 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-21T12:36:18.224 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.224 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.26147.0:42691 11.6 11:6af6d8a1:::benchmark_data_smithi114_187622_object42690:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-09-21T12:36:18.224 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:36:17.441377+0000", 2024-09-21T12:36:18.224 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.013123958, 2024-09-21T12:36:18.224 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.013157319000000001, 2024-09-21T12:36:18.224 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:36:18.224 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:36:18.224 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:36:18.225 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:36:18.225 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.26147", 2024-09-21T12:36:18.225 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:36:18.225 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 42691 2024-09-21T12:36:18.225 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.225 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:36:18.225 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.225 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:36:18.225 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.441377+0000", 2024-09-21T12:36:18.225 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:18.225 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.225 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.226 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:36:18.226 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.441377+0000", 2024-09-21T12:36:18.226 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:18.226 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.226 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.226 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:36:18.226 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.441378+0000", 2024-09-21T12:36:18.226 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2669999999999999e-06 2024-09-21T12:36:18.226 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.226 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.226 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:36:18.227 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.441401+0000", 2024-09-21T12:36:18.227 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2143000000000001e-05 2024-09-21T12:36:18.227 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.227 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.227 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:36:18.227 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.441408+0000", 2024-09-21T12:36:18.227 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6410000000000002e-06 2024-09-21T12:36:18.227 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.227 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.227 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:36:18.228 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.441414+0000", 2024-09-21T12:36:18.228 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0900000000000001e-06 2024-09-21T12:36:18.228 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.228 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.228 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:36:18.228 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.447414+0000", 2024-09-21T12:36:18.228 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0059997879999999998 2024-09-21T12:36:18.228 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.228 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.228 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:36:18.228 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.447551+0000", 2024-09-21T12:36:18.228 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013710299999999999 2024-09-21T12:36:18.228 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.229 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.229 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1", 2024-09-21T12:36:18.229 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.447693+0000", 2024-09-21T12:36:18.229 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014168199999999999 2024-09-21T12:36:18.229 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.229 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.229 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:36:18.229 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.452346+0000", 2024-09-21T12:36:18.229 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0046534899999999997 2024-09-21T12:36:18.229 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.229 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.230 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.230 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.230 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.230 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.26147.0:42705 11.6 11:6a6444e2:::benchmark_data_smithi114_187622_object42704:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-09-21T12:36:18.230 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:36:17.449378+0000", 2024-09-21T12:36:18.230 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0051230429999999999, 2024-09-21T12:36:18.230 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0052444350000000004, 2024-09-21T12:36:18.230 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:36:18.230 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:36:18.230 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:36:18.231 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:36:18.231 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.26147", 2024-09-21T12:36:18.231 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:36:18.231 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 42705 2024-09-21T12:36:18.231 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.231 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:36:18.231 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.231 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:36:18.231 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.449378+0000", 2024-09-21T12:36:18.231 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:18.231 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.232 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.232 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:36:18.232 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.449378+0000", 2024-09-21T12:36:18.232 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:18.232 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.232 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.232 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:36:18.232 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.449380+0000", 2024-09-21T12:36:18.232 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0789999999999999e-06 2024-09-21T12:36:18.232 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.232 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.233 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:36:18.233 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.449413+0000", 2024-09-21T12:36:18.233 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.3120999999999997e-05 2024-09-21T12:36:18.233 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.233 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.233 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:36:18.233 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.449430+0000", 2024-09-21T12:36:18.233 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6611000000000001e-05 2024-09-21T12:36:18.233 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.233 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.233 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:36:18.233 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.449438+0000", 2024-09-21T12:36:18.234 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5189999999999992e-06 2024-09-21T12:36:18.234 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.234 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.234 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:36:18.234 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.449682+0000", 2024-09-21T12:36:18.234 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000243957 2024-09-21T12:36:18.234 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.234 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.234 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:36:18.234 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.449841+0000", 2024-09-21T12:36:18.234 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015840200000000001 2024-09-21T12:36:18.234 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.235 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.235 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1", 2024-09-21T12:36:18.235 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.449965+0000", 2024-09-21T12:36:18.235 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012437099999999999 2024-09-21T12:36:18.235 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.235 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.235 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:36:18.235 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.452352+0000", 2024-09-21T12:36:18.235 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0023864110000000002 2024-09-21T12:36:18.235 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.235 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.236 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.236 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.236 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.236 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.26147.0:42694 11.1 e422/405 11:82504005:::benchmark_data_smithi114_187622_object42693:head v 422'2694, mlcod=422'2691)", 2024-09-21T12:36:18.236 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:36:17.451969+0000", 2024-09-21T12:36:18.236 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.002531854, 2024-09-21T12:36:18.236 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0027488899999999999, 2024-09-21T12:36:18.236 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:36:18.236 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:36:18.236 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:36:18.236 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:36:18.237 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.237 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:36:18.237 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.451969+0000", 2024-09-21T12:36:18.237 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:18.237 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.237 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.237 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:36:18.237 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.451969+0000", 2024-09-21T12:36:18.237 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:18.237 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.237 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.238 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:36:18.238 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.451970+0000", 2024-09-21T12:36:18.238 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7000000000000003e-07 2024-09-21T12:36:18.238 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.238 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.238 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:36:18.238 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.451997+0000", 2024-09-21T12:36:18.238 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.6322e-05 2024-09-21T12:36:18.238 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.238 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.238 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:36:18.239 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.452010+0000", 2024-09-21T12:36:18.239 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3169e-05 2024-09-21T12:36:18.239 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.239 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.239 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:36:18.239 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.452017+0000", 2024-09-21T12:36:18.239 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2760000000000003e-06 2024-09-21T12:36:18.239 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.239 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.239 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:36:18.239 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.452403+0000", 2024-09-21T12:36:18.239 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00038567199999999999 2024-09-21T12:36:18.240 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.240 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.240 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:36:18.240 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.452419+0000", 2024-09-21T12:36:18.240 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.596e-05 2024-09-21T12:36:18.240 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.240 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.240 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.240 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.240 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.240 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.26147.0:42697 11.1 e422/405)", 2024-09-21T12:36:18.240 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:36:17.452157+0000", 2024-09-21T12:36:18.241 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0023438869999999998, 2024-09-21T12:36:18.241 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0026092899999999998, 2024-09-21T12:36:18.241 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:36:18.241 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:36:18.241 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "queued for pg", 2024-09-21T12:36:18.241 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:36:18.241 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.241 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:36:18.241 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.452157+0000", 2024-09-21T12:36:18.241 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:18.241 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.242 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.242 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:36:18.242 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.452157+0000", 2024-09-21T12:36:18.243 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:18.243 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.243 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.243 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:36:18.243 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.452158+0000", 2024-09-21T12:36:18.243 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5199999999999998e-07 2024-09-21T12:36:18.243 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.243 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.243 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:36:18.243 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.452181+0000", 2024-09-21T12:36:18.244 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2996000000000001e-05 2024-09-21T12:36:18.244 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.244 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.244 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:36:18.244 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.452187+0000", 2024-09-21T12:36:18.244 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1890000000000004e-06 2024-09-21T12:36:18.244 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.244 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.244 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:36:18.244 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:17.452192+0000", 2024-09-21T12:36:18.244 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8199999999999996e-06 2024-09-21T12:36:18.244 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.245 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.245 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.245 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.245 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:36:18.245 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 4 2024-09-21T12:36:18.245 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:36:18.301 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:18.331 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:18.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:18.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:18.391 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:36:18.391 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:36:18.392 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 422, 2024-09-21T12:36:18.392 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "e2ad975c-7812-11ef-baf6-efdc52797490", 2024-09-21T12:36:18.392 INFO:teuthology.orchestra.run.smithi114.stdout: "created": "2024-09-21T12:14:24.181227+0000", 2024-09-21T12:36:18.392 INFO:teuthology.orchestra.run.smithi114.stdout: "modified": "2024-09-21T12:36:17.094187+0000", 2024-09-21T12:36:18.392 INFO:teuthology.orchestra.run.smithi114.stdout: "last_up_change": "2024-09-21T12:34:33.320113+0000", 2024-09-21T12:36:18.392 INFO:teuthology.orchestra.run.smithi114.stdout: "last_in_change": "2024-09-21T12:35:59.095504+0000", 2024-09-21T12:36:18.392 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T12:36:18.392 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_num": 5801984, 2024-09-21T12:36:18.392 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_set": [ 2024-09-21T12:36:18.392 INFO:teuthology.orchestra.run.smithi114.stdout: "noscrub", 2024-09-21T12:36:18.392 INFO:teuthology.orchestra.run.smithi114.stdout: "pglog_hardlimit", 2024-09-21T12:36:18.392 INFO:teuthology.orchestra.run.smithi114.stdout: "purged_snapdirs", 2024-09-21T12:36:18.392 INFO:teuthology.orchestra.run.smithi114.stdout: "recovery_deletes", 2024-09-21T12:36:18.392 INFO:teuthology.orchestra.run.smithi114.stdout: "sortbitwise" 2024-09-21T12:36:18.393 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:36:18.393 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_version": 18, 2024-09-21T12:36:18.393 INFO:teuthology.orchestra.run.smithi114.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T12:36:18.393 INFO:teuthology.orchestra.run.smithi114.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T12:36:18.393 INFO:teuthology.orchestra.run.smithi114.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T12:36:18.393 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_snapshot": "", 2024-09-21T12:36:18.393 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_max": 11, 2024-09-21T12:36:18.393 INFO:teuthology.orchestra.run.smithi114.stdout: "max_osd": 8, 2024-09-21T12:36:18.393 INFO:teuthology.orchestra.run.smithi114.stdout: "require_min_compat_client": "luminous", 2024-09-21T12:36:18.393 INFO:teuthology.orchestra.run.smithi114.stdout: "min_compat_client": "luminous", 2024-09-21T12:36:18.393 INFO:teuthology.orchestra.run.smithi114.stdout: "require_osd_release": "squid", 2024-09-21T12:36:18.393 INFO:teuthology.orchestra.run.smithi114.stdout: "allow_crimson": false, 2024-09-21T12:36:18.393 INFO:teuthology.orchestra.run.smithi114.stdout: "pools": [ 2024-09-21T12:36:18.394 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.394 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 1, 2024-09-21T12:36:18.394 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": ".mgr", 2024-09-21T12:36:18.394 INFO:teuthology.orchestra.run.smithi114.stdout: "create_time": "2024-09-21T12:17:38.357647+0000", 2024-09-21T12:36:18.394 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-09-21T12:36:18.394 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_names": "hashpspool", 2024-09-21T12:36:18.394 INFO:teuthology.orchestra.run.smithi114.stdout: "type": 1, 2024-09-21T12:36:18.394 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 3, 2024-09-21T12:36:18.394 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": 2, 2024-09-21T12:36:18.394 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_rule": 0, 2024-09-21T12:36:18.394 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_count": 0, 2024-09-21T12:36:18.394 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_target": 0, 2024-09-21T12:36:18.394 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T12:36:18.395 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T12:36:18.395 INFO:teuthology.orchestra.run.smithi114.stdout: "is_stretch_pool": false, 2024-09-21T12:36:18.395 INFO:teuthology.orchestra.run.smithi114.stdout: "object_hash": 2, 2024-09-21T12:36:18.395 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_mode": "off", 2024-09-21T12:36:18.395 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": 31, 2024-09-21T12:36:18.395 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num": 31, 2024-09-21T12:36:18.395 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num_target": 31, 2024-09-21T12:36:18.395 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_target": 31, 2024-09-21T12:36:18.395 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_pending": 31, 2024-09-21T12:36:18.395 INFO:teuthology.orchestra.run.smithi114.stdout: "last_pg_merge_meta": { 2024-09-21T12:36:18.395 INFO:teuthology.orchestra.run.smithi114.stdout: "source_pgid": "0.0", 2024-09-21T12:36:18.395 INFO:teuthology.orchestra.run.smithi114.stdout: "ready_epoch": 0, 2024-09-21T12:36:18.396 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_started": 0, 2024-09-21T12:36:18.396 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_clean": 0, 2024-09-21T12:36:18.396 INFO:teuthology.orchestra.run.smithi114.stdout: "source_version": "0'0", 2024-09-21T12:36:18.396 INFO:teuthology.orchestra.run.smithi114.stdout: "target_version": "0'0" 2024-09-21T12:36:18.396 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.396 INFO:teuthology.orchestra.run.smithi114.stdout: "last_change": "218", 2024-09-21T12:36:18.396 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend": "0", 2024-09-21T12:36:18.396 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T12:36:18.396 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_preluminous": "216", 2024-09-21T12:36:18.396 INFO:teuthology.orchestra.run.smithi114.stdout: "auid": 0, 2024-09-21T12:36:18.396 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_mode": "selfmanaged", 2024-09-21T12:36:18.396 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_seq": 0, 2024-09-21T12:36:18.396 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_epoch": 0, 2024-09-21T12:36:18.396 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_snaps": [], 2024-09-21T12:36:18.397 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps": "[]", 2024-09-21T12:36:18.397 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_bytes": 0, 2024-09-21T12:36:18.397 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_objects": 0, 2024-09-21T12:36:18.397 INFO:teuthology.orchestra.run.smithi114.stdout: "tiers": [], 2024-09-21T12:36:18.397 INFO:teuthology.orchestra.run.smithi114.stdout: "tier_of": -1, 2024-09-21T12:36:18.397 INFO:teuthology.orchestra.run.smithi114.stdout: "read_tier": -1, 2024-09-21T12:36:18.397 INFO:teuthology.orchestra.run.smithi114.stdout: "write_tier": -1, 2024-09-21T12:36:18.397 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_mode": "none", 2024-09-21T12:36:18.397 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_bytes": 0, 2024-09-21T12:36:18.397 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_objects": 0, 2024-09-21T12:36:18.397 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T12:36:18.397 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T12:36:18.397 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T12:36:18.397 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_flush_age": 0, 2024-09-21T12:36:18.398 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_evict_age": 0, 2024-09-21T12:36:18.398 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profile": "", 2024-09-21T12:36:18.398 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_params": { 2024-09-21T12:36:18.398 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "none" 2024-09-21T12:36:18.398 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.398 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_period": 0, 2024-09-21T12:36:18.398 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_count": 0, 2024-09-21T12:36:18.398 INFO:teuthology.orchestra.run.smithi114.stdout: "use_gmt_hitset": true, 2024-09-21T12:36:18.398 INFO:teuthology.orchestra.run.smithi114.stdout: "min_read_recency_for_promote": 0, 2024-09-21T12:36:18.398 INFO:teuthology.orchestra.run.smithi114.stdout: "min_write_recency_for_promote": 0, 2024-09-21T12:36:18.398 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T12:36:18.398 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_search_last_n": 0, 2024-09-21T12:36:18.398 INFO:teuthology.orchestra.run.smithi114.stdout: "grade_table": [], 2024-09-21T12:36:18.398 INFO:teuthology.orchestra.run.smithi114.stdout: "stripe_width": 0, 2024-09-21T12:36:18.399 INFO:teuthology.orchestra.run.smithi114.stdout: "expected_num_objects": 0, 2024-09-21T12:36:18.399 INFO:teuthology.orchestra.run.smithi114.stdout: "fast_read": false, 2024-09-21T12:36:18.399 INFO:teuthology.orchestra.run.smithi114.stdout: "options": { 2024-09-21T12:36:18.399 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_max": 32, 2024-09-21T12:36:18.399 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_min": 1 2024-09-21T12:36:18.399 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.399 INFO:teuthology.orchestra.run.smithi114.stdout: "application_metadata": { 2024-09-21T12:36:18.399 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": {} 2024-09-21T12:36:18.399 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.399 INFO:teuthology.orchestra.run.smithi114.stdout: "read_balance": { 2024-09-21T12:36:18.399 INFO:teuthology.orchestra.run.smithi114.stdout: "score_type": "Fair distribution", 2024-09-21T12:36:18.399 INFO:teuthology.orchestra.run.smithi114.stdout: "score_acting": 1.809999942779541, 2024-09-21T12:36:18.399 INFO:teuthology.orchestra.run.smithi114.stdout: "score_stable": 1.809999942779541, 2024-09-21T12:36:18.400 INFO:teuthology.orchestra.run.smithi114.stdout: "optimal_score": 1, 2024-09-21T12:36:18.400 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_acting": 1.809999942779541, 2024-09-21T12:36:18.400 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_stable": 1.809999942779541, 2024-09-21T12:36:18.400 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity_weighted": 0.73582649230957031, 2024-09-21T12:36:18.400 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity": 0.8399999737739563, 2024-09-21T12:36:18.400 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity_weighted": 0.74000000953674316 2024-09-21T12:36:18.400 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.400 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.400 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.400 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 2, 2024-09-21T12:36:18.400 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": "datapool", 2024-09-21T12:36:18.400 INFO:teuthology.orchestra.run.smithi114.stdout: "create_time": "2024-09-21T12:20:12.761046+0000", 2024-09-21T12:36:18.400 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 8193, 2024-09-21T12:36:18.401 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T12:36:18.401 INFO:teuthology.orchestra.run.smithi114.stdout: "type": 1, 2024-09-21T12:36:18.401 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 3, 2024-09-21T12:36:18.401 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": 2, 2024-09-21T12:36:18.401 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_rule": 0, 2024-09-21T12:36:18.401 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_count": 0, 2024-09-21T12:36:18.401 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_target": 0, 2024-09-21T12:36:18.401 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T12:36:18.401 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T12:36:18.401 INFO:teuthology.orchestra.run.smithi114.stdout: "is_stretch_pool": false, 2024-09-21T12:36:18.401 INFO:teuthology.orchestra.run.smithi114.stdout: "object_hash": 2, 2024-09-21T12:36:18.401 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_mode": "off", 2024-09-21T12:36:18.401 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": 3, 2024-09-21T12:36:18.401 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num": 3, 2024-09-21T12:36:18.402 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num_target": 3, 2024-09-21T12:36:18.402 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_target": 3, 2024-09-21T12:36:18.402 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_pending": 3, 2024-09-21T12:36:18.402 INFO:teuthology.orchestra.run.smithi114.stdout: "last_pg_merge_meta": { 2024-09-21T12:36:18.402 INFO:teuthology.orchestra.run.smithi114.stdout: "source_pgid": "0.0", 2024-09-21T12:36:18.402 INFO:teuthology.orchestra.run.smithi114.stdout: "ready_epoch": 0, 2024-09-21T12:36:18.402 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_started": 0, 2024-09-21T12:36:18.402 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_clean": 0, 2024-09-21T12:36:18.402 INFO:teuthology.orchestra.run.smithi114.stdout: "source_version": "0'0", 2024-09-21T12:36:18.402 INFO:teuthology.orchestra.run.smithi114.stdout: "target_version": "0'0" 2024-09-21T12:36:18.402 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.402 INFO:teuthology.orchestra.run.smithi114.stdout: "last_change": "61", 2024-09-21T12:36:18.402 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend": "0", 2024-09-21T12:36:18.403 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T12:36:18.403 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T12:36:18.403 INFO:teuthology.orchestra.run.smithi114.stdout: "auid": 0, 2024-09-21T12:36:18.403 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_mode": "selfmanaged", 2024-09-21T12:36:18.403 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_seq": 2, 2024-09-21T12:36:18.403 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_epoch": 61, 2024-09-21T12:36:18.403 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_snaps": [], 2024-09-21T12:36:18.403 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps": "[]", 2024-09-21T12:36:18.403 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_bytes": 0, 2024-09-21T12:36:18.403 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_objects": 0, 2024-09-21T12:36:18.403 INFO:teuthology.orchestra.run.smithi114.stdout: "tiers": [], 2024-09-21T12:36:18.403 INFO:teuthology.orchestra.run.smithi114.stdout: "tier_of": -1, 2024-09-21T12:36:18.403 INFO:teuthology.orchestra.run.smithi114.stdout: "read_tier": -1, 2024-09-21T12:36:18.403 INFO:teuthology.orchestra.run.smithi114.stdout: "write_tier": -1, 2024-09-21T12:36:18.404 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_mode": "none", 2024-09-21T12:36:18.404 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_bytes": 0, 2024-09-21T12:36:18.404 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_objects": 0, 2024-09-21T12:36:18.404 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T12:36:18.404 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T12:36:18.404 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T12:36:18.404 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_flush_age": 0, 2024-09-21T12:36:18.404 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_evict_age": 0, 2024-09-21T12:36:18.404 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profile": "", 2024-09-21T12:36:18.404 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_params": { 2024-09-21T12:36:18.404 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "none" 2024-09-21T12:36:18.404 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.404 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_period": 0, 2024-09-21T12:36:18.404 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_count": 0, 2024-09-21T12:36:18.405 INFO:teuthology.orchestra.run.smithi114.stdout: "use_gmt_hitset": true, 2024-09-21T12:36:18.405 INFO:teuthology.orchestra.run.smithi114.stdout: "min_read_recency_for_promote": 0, 2024-09-21T12:36:18.405 INFO:teuthology.orchestra.run.smithi114.stdout: "min_write_recency_for_promote": 0, 2024-09-21T12:36:18.405 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T12:36:18.405 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_search_last_n": 0, 2024-09-21T12:36:18.405 INFO:teuthology.orchestra.run.smithi114.stdout: "grade_table": [], 2024-09-21T12:36:18.405 INFO:teuthology.orchestra.run.smithi114.stdout: "stripe_width": 0, 2024-09-21T12:36:18.405 INFO:teuthology.orchestra.run.smithi114.stdout: "expected_num_objects": 0, 2024-09-21T12:36:18.405 INFO:teuthology.orchestra.run.smithi114.stdout: "fast_read": false, 2024-09-21T12:36:18.405 INFO:teuthology.orchestra.run.smithi114.stdout: "options": {}, 2024-09-21T12:36:18.405 INFO:teuthology.orchestra.run.smithi114.stdout: "application_metadata": { 2024-09-21T12:36:18.405 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd": {} 2024-09-21T12:36:18.405 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.405 INFO:teuthology.orchestra.run.smithi114.stdout: "read_balance": { 2024-09-21T12:36:18.406 INFO:teuthology.orchestra.run.smithi114.stdout: "score_type": "Fair distribution", 2024-09-21T12:36:18.406 INFO:teuthology.orchestra.run.smithi114.stdout: "score_acting": 2.3299999237060547, 2024-09-21T12:36:18.406 INFO:teuthology.orchestra.run.smithi114.stdout: "score_stable": 2.3299999237060547, 2024-09-21T12:36:18.406 INFO:teuthology.orchestra.run.smithi114.stdout: "optimal_score": 0.86000001430511475, 2024-09-21T12:36:18.406 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_acting": 2, 2024-09-21T12:36:18.406 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_stable": 2, 2024-09-21T12:36:18.406 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity_weighted": 0.73582649230957031, 2024-09-21T12:36:18.406 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity": 0.8399999737739563, 2024-09-21T12:36:18.406 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity_weighted": 0.74000000953674316 2024-09-21T12:36:18.406 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.406 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.406 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.406 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 3, 2024-09-21T12:36:18.407 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": ".rgw.root", 2024-09-21T12:36:18.407 INFO:teuthology.orchestra.run.smithi114.stdout: "create_time": "2024-09-21T12:20:13.713971+0000", 2024-09-21T12:36:18.407 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-09-21T12:36:18.407 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_names": "hashpspool", 2024-09-21T12:36:18.407 INFO:teuthology.orchestra.run.smithi114.stdout: "type": 1, 2024-09-21T12:36:18.407 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 3, 2024-09-21T12:36:18.407 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": 2, 2024-09-21T12:36:18.407 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_rule": 0, 2024-09-21T12:36:18.407 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_count": 0, 2024-09-21T12:36:18.407 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_target": 0, 2024-09-21T12:36:18.407 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T12:36:18.407 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T12:36:18.407 INFO:teuthology.orchestra.run.smithi114.stdout: "is_stretch_pool": false, 2024-09-21T12:36:18.407 INFO:teuthology.orchestra.run.smithi114.stdout: "object_hash": 2, 2024-09-21T12:36:18.408 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_mode": "off", 2024-09-21T12:36:18.408 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": 32, 2024-09-21T12:36:18.408 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num": 32, 2024-09-21T12:36:18.408 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num_target": 32, 2024-09-21T12:36:18.408 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_target": 32, 2024-09-21T12:36:18.408 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_pending": 32, 2024-09-21T12:36:18.408 INFO:teuthology.orchestra.run.smithi114.stdout: "last_pg_merge_meta": { 2024-09-21T12:36:18.408 INFO:teuthology.orchestra.run.smithi114.stdout: "source_pgid": "0.0", 2024-09-21T12:36:18.408 INFO:teuthology.orchestra.run.smithi114.stdout: "ready_epoch": 0, 2024-09-21T12:36:18.409 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_started": 0, 2024-09-21T12:36:18.409 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_clean": 0, 2024-09-21T12:36:18.409 INFO:teuthology.orchestra.run.smithi114.stdout: "source_version": "0'0", 2024-09-21T12:36:18.409 INFO:teuthology.orchestra.run.smithi114.stdout: "target_version": "0'0" 2024-09-21T12:36:18.409 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.409 INFO:teuthology.orchestra.run.smithi114.stdout: "last_change": "60", 2024-09-21T12:36:18.409 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend": "0", 2024-09-21T12:36:18.409 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T12:36:18.409 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T12:36:18.409 INFO:teuthology.orchestra.run.smithi114.stdout: "auid": 0, 2024-09-21T12:36:18.409 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_mode": "selfmanaged", 2024-09-21T12:36:18.409 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_seq": 0, 2024-09-21T12:36:18.409 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_epoch": 0, 2024-09-21T12:36:18.409 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_snaps": [], 2024-09-21T12:36:18.410 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps": "[]", 2024-09-21T12:36:18.410 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_bytes": 0, 2024-09-21T12:36:18.410 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_objects": 0, 2024-09-21T12:36:18.410 INFO:teuthology.orchestra.run.smithi114.stdout: "tiers": [], 2024-09-21T12:36:18.410 INFO:teuthology.orchestra.run.smithi114.stdout: "tier_of": -1, 2024-09-21T12:36:18.410 INFO:teuthology.orchestra.run.smithi114.stdout: "read_tier": -1, 2024-09-21T12:36:18.410 INFO:teuthology.orchestra.run.smithi114.stdout: "write_tier": -1, 2024-09-21T12:36:18.410 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_mode": "none", 2024-09-21T12:36:18.410 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_bytes": 0, 2024-09-21T12:36:18.410 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_objects": 0, 2024-09-21T12:36:18.410 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T12:36:18.410 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T12:36:18.410 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T12:36:18.411 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_flush_age": 0, 2024-09-21T12:36:18.411 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_evict_age": 0, 2024-09-21T12:36:18.411 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profile": "", 2024-09-21T12:36:18.411 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_params": { 2024-09-21T12:36:18.411 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "none" 2024-09-21T12:36:18.411 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.411 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_period": 0, 2024-09-21T12:36:18.411 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_count": 0, 2024-09-21T12:36:18.411 INFO:teuthology.orchestra.run.smithi114.stdout: "use_gmt_hitset": true, 2024-09-21T12:36:18.411 INFO:teuthology.orchestra.run.smithi114.stdout: "min_read_recency_for_promote": 0, 2024-09-21T12:36:18.411 INFO:teuthology.orchestra.run.smithi114.stdout: "min_write_recency_for_promote": 0, 2024-09-21T12:36:18.411 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T12:36:18.411 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_search_last_n": 0, 2024-09-21T12:36:18.411 INFO:teuthology.orchestra.run.smithi114.stdout: "grade_table": [], 2024-09-21T12:36:18.412 INFO:teuthology.orchestra.run.smithi114.stdout: "stripe_width": 0, 2024-09-21T12:36:18.412 INFO:teuthology.orchestra.run.smithi114.stdout: "expected_num_objects": 0, 2024-09-21T12:36:18.412 INFO:teuthology.orchestra.run.smithi114.stdout: "fast_read": false, 2024-09-21T12:36:18.412 INFO:teuthology.orchestra.run.smithi114.stdout: "options": {}, 2024-09-21T12:36:18.412 INFO:teuthology.orchestra.run.smithi114.stdout: "application_metadata": { 2024-09-21T12:36:18.412 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw": {} 2024-09-21T12:36:18.412 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.412 INFO:teuthology.orchestra.run.smithi114.stdout: "read_balance": { 2024-09-21T12:36:18.412 INFO:teuthology.orchestra.run.smithi114.stdout: "score_type": "Fair distribution", 2024-09-21T12:36:18.412 INFO:teuthology.orchestra.run.smithi114.stdout: "score_acting": 1.5299999713897705, 2024-09-21T12:36:18.412 INFO:teuthology.orchestra.run.smithi114.stdout: "score_stable": 1.5299999713897705, 2024-09-21T12:36:18.412 INFO:teuthology.orchestra.run.smithi114.stdout: "optimal_score": 1, 2024-09-21T12:36:18.412 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_acting": 1.5299999713897705, 2024-09-21T12:36:18.413 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_stable": 1.5299999713897705, 2024-09-21T12:36:18.413 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity_weighted": 0.73582649230957031, 2024-09-21T12:36:18.413 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity": 0.8399999737739563, 2024-09-21T12:36:18.413 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity_weighted": 0.74000000953674316 2024-09-21T12:36:18.413 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.413 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.413 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.413 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 4, 2024-09-21T12:36:18.413 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": "default.rgw.log", 2024-09-21T12:36:18.413 INFO:teuthology.orchestra.run.smithi114.stdout: "create_time": "2024-09-21T12:20:15.368765+0000", 2024-09-21T12:36:18.413 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-09-21T12:36:18.413 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_names": "hashpspool", 2024-09-21T12:36:18.413 INFO:teuthology.orchestra.run.smithi114.stdout: "type": 1, 2024-09-21T12:36:18.413 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 3, 2024-09-21T12:36:18.414 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": 2, 2024-09-21T12:36:18.414 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_rule": 0, 2024-09-21T12:36:18.414 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_count": 0, 2024-09-21T12:36:18.414 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_target": 0, 2024-09-21T12:36:18.414 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T12:36:18.414 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T12:36:18.414 INFO:teuthology.orchestra.run.smithi114.stdout: "is_stretch_pool": false, 2024-09-21T12:36:18.414 INFO:teuthology.orchestra.run.smithi114.stdout: "object_hash": 2, 2024-09-21T12:36:18.414 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_mode": "off", 2024-09-21T12:36:18.414 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": 32, 2024-09-21T12:36:18.414 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num": 32, 2024-09-21T12:36:18.414 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num_target": 32, 2024-09-21T12:36:18.414 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_target": 32, 2024-09-21T12:36:18.415 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_pending": 32, 2024-09-21T12:36:18.415 INFO:teuthology.orchestra.run.smithi114.stdout: "last_pg_merge_meta": { 2024-09-21T12:36:18.415 INFO:teuthology.orchestra.run.smithi114.stdout: "source_pgid": "0.0", 2024-09-21T12:36:18.415 INFO:teuthology.orchestra.run.smithi114.stdout: "ready_epoch": 0, 2024-09-21T12:36:18.415 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_started": 0, 2024-09-21T12:36:18.415 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_clean": 0, 2024-09-21T12:36:18.415 INFO:teuthology.orchestra.run.smithi114.stdout: "source_version": "0'0", 2024-09-21T12:36:18.415 INFO:teuthology.orchestra.run.smithi114.stdout: "target_version": "0'0" 2024-09-21T12:36:18.415 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.415 INFO:teuthology.orchestra.run.smithi114.stdout: "last_change": "62", 2024-09-21T12:36:18.415 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend": "0", 2024-09-21T12:36:18.415 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T12:36:18.415 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T12:36:18.415 INFO:teuthology.orchestra.run.smithi114.stdout: "auid": 0, 2024-09-21T12:36:18.416 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_mode": "selfmanaged", 2024-09-21T12:36:18.416 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_seq": 0, 2024-09-21T12:36:18.416 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_epoch": 0, 2024-09-21T12:36:18.416 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_snaps": [], 2024-09-21T12:36:18.416 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps": "[]", 2024-09-21T12:36:18.416 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_bytes": 0, 2024-09-21T12:36:18.416 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_objects": 0, 2024-09-21T12:36:18.416 INFO:teuthology.orchestra.run.smithi114.stdout: "tiers": [], 2024-09-21T12:36:18.416 INFO:teuthology.orchestra.run.smithi114.stdout: "tier_of": -1, 2024-09-21T12:36:18.416 INFO:teuthology.orchestra.run.smithi114.stdout: "read_tier": -1, 2024-09-21T12:36:18.416 INFO:teuthology.orchestra.run.smithi114.stdout: "write_tier": -1, 2024-09-21T12:36:18.416 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_mode": "none", 2024-09-21T12:36:18.416 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_bytes": 0, 2024-09-21T12:36:18.417 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_objects": 0, 2024-09-21T12:36:18.417 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T12:36:18.417 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T12:36:18.417 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T12:36:18.417 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_flush_age": 0, 2024-09-21T12:36:18.417 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_evict_age": 0, 2024-09-21T12:36:18.417 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profile": "", 2024-09-21T12:36:18.417 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_params": { 2024-09-21T12:36:18.417 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "none" 2024-09-21T12:36:18.417 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.417 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_period": 0, 2024-09-21T12:36:18.417 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_count": 0, 2024-09-21T12:36:18.417 INFO:teuthology.orchestra.run.smithi114.stdout: "use_gmt_hitset": true, 2024-09-21T12:36:18.417 INFO:teuthology.orchestra.run.smithi114.stdout: "min_read_recency_for_promote": 0, 2024-09-21T12:36:18.418 INFO:teuthology.orchestra.run.smithi114.stdout: "min_write_recency_for_promote": 0, 2024-09-21T12:36:18.418 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T12:36:18.418 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_search_last_n": 0, 2024-09-21T12:36:18.418 INFO:teuthology.orchestra.run.smithi114.stdout: "grade_table": [], 2024-09-21T12:36:18.418 INFO:teuthology.orchestra.run.smithi114.stdout: "stripe_width": 0, 2024-09-21T12:36:18.418 INFO:teuthology.orchestra.run.smithi114.stdout: "expected_num_objects": 0, 2024-09-21T12:36:18.418 INFO:teuthology.orchestra.run.smithi114.stdout: "fast_read": false, 2024-09-21T12:36:18.418 INFO:teuthology.orchestra.run.smithi114.stdout: "options": {}, 2024-09-21T12:36:18.418 INFO:teuthology.orchestra.run.smithi114.stdout: "application_metadata": { 2024-09-21T12:36:18.418 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw": {} 2024-09-21T12:36:18.418 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.418 INFO:teuthology.orchestra.run.smithi114.stdout: "read_balance": { 2024-09-21T12:36:18.418 INFO:teuthology.orchestra.run.smithi114.stdout: "score_type": "Fair distribution", 2024-09-21T12:36:18.418 INFO:teuthology.orchestra.run.smithi114.stdout: "score_acting": 2.190000057220459, 2024-09-21T12:36:18.419 INFO:teuthology.orchestra.run.smithi114.stdout: "score_stable": 2.190000057220459, 2024-09-21T12:36:18.419 INFO:teuthology.orchestra.run.smithi114.stdout: "optimal_score": 1, 2024-09-21T12:36:18.419 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_acting": 2.190000057220459, 2024-09-21T12:36:18.419 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_stable": 2.190000057220459, 2024-09-21T12:36:18.419 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity_weighted": 0.73582649230957031, 2024-09-21T12:36:18.419 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity": 0.8399999737739563, 2024-09-21T12:36:18.419 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity_weighted": 0.74000000953674316 2024-09-21T12:36:18.419 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.419 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.419 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 5, 2024-09-21T12:36:18.419 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": "default.rgw.control", 2024-09-21T12:36:18.419 INFO:teuthology.orchestra.run.smithi114.stdout: "create_time": "2024-09-21T12:20:17.314275+0000", 2024-09-21T12:36:18.420 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-09-21T12:36:18.420 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_names": "hashpspool", 2024-09-21T12:36:18.420 INFO:teuthology.orchestra.run.smithi114.stdout: "type": 1, 2024-09-21T12:36:18.420 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 3, 2024-09-21T12:36:18.420 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": 2, 2024-09-21T12:36:18.420 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_rule": 0, 2024-09-21T12:36:18.420 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_count": 0, 2024-09-21T12:36:18.420 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_target": 0, 2024-09-21T12:36:18.420 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T12:36:18.420 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T12:36:18.420 INFO:teuthology.orchestra.run.smithi114.stdout: "is_stretch_pool": false, 2024-09-21T12:36:18.420 INFO:teuthology.orchestra.run.smithi114.stdout: "object_hash": 2, 2024-09-21T12:36:18.420 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_mode": "off", 2024-09-21T12:36:18.420 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": 32, 2024-09-21T12:36:18.421 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num": 32, 2024-09-21T12:36:18.421 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num_target": 32, 2024-09-21T12:36:18.421 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_target": 32, 2024-09-21T12:36:18.421 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_pending": 32, 2024-09-21T12:36:18.421 INFO:teuthology.orchestra.run.smithi114.stdout: "last_pg_merge_meta": { 2024-09-21T12:36:18.421 INFO:teuthology.orchestra.run.smithi114.stdout: "source_pgid": "0.0", 2024-09-21T12:36:18.421 INFO:teuthology.orchestra.run.smithi114.stdout: "ready_epoch": 0, 2024-09-21T12:36:18.421 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_started": 0, 2024-09-21T12:36:18.421 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_clean": 0, 2024-09-21T12:36:18.421 INFO:teuthology.orchestra.run.smithi114.stdout: "source_version": "0'0", 2024-09-21T12:36:18.421 INFO:teuthology.orchestra.run.smithi114.stdout: "target_version": "0'0" 2024-09-21T12:36:18.421 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.421 INFO:teuthology.orchestra.run.smithi114.stdout: "last_change": "64", 2024-09-21T12:36:18.421 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend": "0", 2024-09-21T12:36:18.422 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T12:36:18.422 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T12:36:18.422 INFO:teuthology.orchestra.run.smithi114.stdout: "auid": 0, 2024-09-21T12:36:18.422 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_mode": "selfmanaged", 2024-09-21T12:36:18.422 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_seq": 0, 2024-09-21T12:36:18.422 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_epoch": 0, 2024-09-21T12:36:18.422 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_snaps": [], 2024-09-21T12:36:18.422 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps": "[]", 2024-09-21T12:36:18.422 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_bytes": 0, 2024-09-21T12:36:18.422 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_objects": 0, 2024-09-21T12:36:18.422 INFO:teuthology.orchestra.run.smithi114.stdout: "tiers": [], 2024-09-21T12:36:18.422 INFO:teuthology.orchestra.run.smithi114.stdout: "tier_of": -1, 2024-09-21T12:36:18.422 INFO:teuthology.orchestra.run.smithi114.stdout: "read_tier": -1, 2024-09-21T12:36:18.422 INFO:teuthology.orchestra.run.smithi114.stdout: "write_tier": -1, 2024-09-21T12:36:18.423 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_mode": "none", 2024-09-21T12:36:18.423 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_bytes": 0, 2024-09-21T12:36:18.423 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_objects": 0, 2024-09-21T12:36:18.423 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T12:36:18.423 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T12:36:18.423 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T12:36:18.423 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_flush_age": 0, 2024-09-21T12:36:18.423 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_evict_age": 0, 2024-09-21T12:36:18.423 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profile": "", 2024-09-21T12:36:18.423 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_params": { 2024-09-21T12:36:18.423 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "none" 2024-09-21T12:36:18.423 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.423 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_period": 0, 2024-09-21T12:36:18.423 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_count": 0, 2024-09-21T12:36:18.424 INFO:teuthology.orchestra.run.smithi114.stdout: "use_gmt_hitset": true, 2024-09-21T12:36:18.424 INFO:teuthology.orchestra.run.smithi114.stdout: "min_read_recency_for_promote": 0, 2024-09-21T12:36:18.424 INFO:teuthology.orchestra.run.smithi114.stdout: "min_write_recency_for_promote": 0, 2024-09-21T12:36:18.424 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T12:36:18.424 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_search_last_n": 0, 2024-09-21T12:36:18.424 INFO:teuthology.orchestra.run.smithi114.stdout: "grade_table": [], 2024-09-21T12:36:18.424 INFO:teuthology.orchestra.run.smithi114.stdout: "stripe_width": 0, 2024-09-21T12:36:18.424 INFO:teuthology.orchestra.run.smithi114.stdout: "expected_num_objects": 0, 2024-09-21T12:36:18.424 INFO:teuthology.orchestra.run.smithi114.stdout: "fast_read": false, 2024-09-21T12:36:18.424 INFO:teuthology.orchestra.run.smithi114.stdout: "options": {}, 2024-09-21T12:36:18.424 INFO:teuthology.orchestra.run.smithi114.stdout: "application_metadata": { 2024-09-21T12:36:18.424 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw": {} 2024-09-21T12:36:18.424 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.424 INFO:teuthology.orchestra.run.smithi114.stdout: "read_balance": { 2024-09-21T12:36:18.424 INFO:teuthology.orchestra.run.smithi114.stdout: "score_type": "Fair distribution", 2024-09-21T12:36:18.425 INFO:teuthology.orchestra.run.smithi114.stdout: "score_acting": 1.5299999713897705, 2024-09-21T12:36:18.425 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:18 smithi114 ceph-mon[40867]: pgmap v816: 178 pgs: 1 creating+peering, 2 active+undersized+remapped, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 123 active+clean; 2.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 62 MiB/s wr, 985 op/s; 8686/104133 objects degraded (8.341%); 4352/104133 objects misplaced (4.179%); 22 MiB/s, 347 objects/s recovering 2024-09-21T12:36:18.425 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:18 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:36:18.425 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:18 smithi114 ceph-mon[30324]: pgmap v816: 178 pgs: 1 creating+peering, 2 active+undersized+remapped, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 123 active+clean; 2.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 62 MiB/s wr, 985 op/s; 8686/104133 objects degraded (8.341%); 4352/104133 objects misplaced (4.179%); 22 MiB/s, 347 objects/s recovering 2024-09-21T12:36:18.425 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:18 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:36:18.426 INFO:teuthology.orchestra.run.smithi114.stdout: "score_stable": 1.5299999713897705, 2024-09-21T12:36:18.426 INFO:teuthology.orchestra.run.smithi114.stdout: "optimal_score": 1, 2024-09-21T12:36:18.426 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_acting": 1.5299999713897705, 2024-09-21T12:36:18.426 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_stable": 1.5299999713897705, 2024-09-21T12:36:18.426 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity_weighted": 0.73582649230957031, 2024-09-21T12:36:18.426 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity": 0.8399999737739563, 2024-09-21T12:36:18.426 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity_weighted": 0.74000000953674316 2024-09-21T12:36:18.426 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.426 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.426 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.427 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 6, 2024-09-21T12:36:18.427 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": "default.rgw.meta", 2024-09-21T12:36:18.427 INFO:teuthology.orchestra.run.smithi114.stdout: "create_time": "2024-09-21T12:20:19.370218+0000", 2024-09-21T12:36:18.427 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-09-21T12:36:18.427 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_names": "hashpspool", 2024-09-21T12:36:18.427 INFO:teuthology.orchestra.run.smithi114.stdout: "type": 1, 2024-09-21T12:36:18.427 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 3, 2024-09-21T12:36:18.427 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": 2, 2024-09-21T12:36:18.427 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_rule": 0, 2024-09-21T12:36:18.427 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_count": 0, 2024-09-21T12:36:18.427 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_target": 0, 2024-09-21T12:36:18.427 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T12:36:18.427 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T12:36:18.428 INFO:teuthology.orchestra.run.smithi114.stdout: "is_stretch_pool": false, 2024-09-21T12:36:18.428 INFO:teuthology.orchestra.run.smithi114.stdout: "object_hash": 2, 2024-09-21T12:36:18.428 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_mode": "off", 2024-09-21T12:36:18.428 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": 32, 2024-09-21T12:36:18.428 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num": 32, 2024-09-21T12:36:18.428 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num_target": 32, 2024-09-21T12:36:18.428 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_target": 32, 2024-09-21T12:36:18.428 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_pending": 32, 2024-09-21T12:36:18.428 INFO:teuthology.orchestra.run.smithi114.stdout: "last_pg_merge_meta": { 2024-09-21T12:36:18.428 INFO:teuthology.orchestra.run.smithi114.stdout: "source_pgid": "0.0", 2024-09-21T12:36:18.428 INFO:teuthology.orchestra.run.smithi114.stdout: "ready_epoch": 0, 2024-09-21T12:36:18.428 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_started": 0, 2024-09-21T12:36:18.428 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_clean": 0, 2024-09-21T12:36:18.429 INFO:teuthology.orchestra.run.smithi114.stdout: "source_version": "0'0", 2024-09-21T12:36:18.429 INFO:teuthology.orchestra.run.smithi114.stdout: "target_version": "0'0" 2024-09-21T12:36:18.429 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.429 INFO:teuthology.orchestra.run.smithi114.stdout: "last_change": "66", 2024-09-21T12:36:18.429 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend": "0", 2024-09-21T12:36:18.429 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T12:36:18.429 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T12:36:18.429 INFO:teuthology.orchestra.run.smithi114.stdout: "auid": 0, 2024-09-21T12:36:18.429 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_mode": "selfmanaged", 2024-09-21T12:36:18.429 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_seq": 0, 2024-09-21T12:36:18.429 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_epoch": 0, 2024-09-21T12:36:18.429 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_snaps": [], 2024-09-21T12:36:18.429 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps": "[]", 2024-09-21T12:36:18.429 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_bytes": 0, 2024-09-21T12:36:18.429 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_objects": 0, 2024-09-21T12:36:18.430 INFO:teuthology.orchestra.run.smithi114.stdout: "tiers": [], 2024-09-21T12:36:18.430 INFO:teuthology.orchestra.run.smithi114.stdout: "tier_of": -1, 2024-09-21T12:36:18.430 INFO:teuthology.orchestra.run.smithi114.stdout: "read_tier": -1, 2024-09-21T12:36:18.430 INFO:teuthology.orchestra.run.smithi114.stdout: "write_tier": -1, 2024-09-21T12:36:18.430 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_mode": "none", 2024-09-21T12:36:18.430 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_bytes": 0, 2024-09-21T12:36:18.430 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_objects": 0, 2024-09-21T12:36:18.430 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T12:36:18.430 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T12:36:18.430 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T12:36:18.430 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_flush_age": 0, 2024-09-21T12:36:18.430 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_evict_age": 0, 2024-09-21T12:36:18.430 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profile": "", 2024-09-21T12:36:18.431 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_params": { 2024-09-21T12:36:18.431 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "none" 2024-09-21T12:36:18.431 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.431 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_period": 0, 2024-09-21T12:36:18.431 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_count": 0, 2024-09-21T12:36:18.431 INFO:teuthology.orchestra.run.smithi114.stdout: "use_gmt_hitset": true, 2024-09-21T12:36:18.431 INFO:teuthology.orchestra.run.smithi114.stdout: "min_read_recency_for_promote": 0, 2024-09-21T12:36:18.431 INFO:teuthology.orchestra.run.smithi114.stdout: "min_write_recency_for_promote": 0, 2024-09-21T12:36:18.431 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T12:36:18.431 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_search_last_n": 0, 2024-09-21T12:36:18.431 INFO:teuthology.orchestra.run.smithi114.stdout: "grade_table": [], 2024-09-21T12:36:18.431 INFO:teuthology.orchestra.run.smithi114.stdout: "stripe_width": 0, 2024-09-21T12:36:18.431 INFO:teuthology.orchestra.run.smithi114.stdout: "expected_num_objects": 0, 2024-09-21T12:36:18.432 INFO:teuthology.orchestra.run.smithi114.stdout: "fast_read": false, 2024-09-21T12:36:18.432 INFO:teuthology.orchestra.run.smithi114.stdout: "options": { 2024-09-21T12:36:18.432 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_bias": 4 2024-09-21T12:36:18.432 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.432 INFO:teuthology.orchestra.run.smithi114.stdout: "application_metadata": { 2024-09-21T12:36:18.432 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw": {} 2024-09-21T12:36:18.432 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.432 INFO:teuthology.orchestra.run.smithi114.stdout: "read_balance": { 2024-09-21T12:36:18.432 INFO:teuthology.orchestra.run.smithi114.stdout: "score_type": "Fair distribution", 2024-09-21T12:36:18.432 INFO:teuthology.orchestra.run.smithi114.stdout: "score_acting": 1.9700000286102295, 2024-09-21T12:36:18.432 INFO:teuthology.orchestra.run.smithi114.stdout: "score_stable": 1.9700000286102295, 2024-09-21T12:36:18.432 INFO:teuthology.orchestra.run.smithi114.stdout: "optimal_score": 1, 2024-09-21T12:36:18.432 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_acting": 1.9700000286102295, 2024-09-21T12:36:18.432 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_stable": 1.9700000286102295, 2024-09-21T12:36:18.433 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity_weighted": 0.73582649230957031, 2024-09-21T12:36:18.433 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity": 0.8399999737739563, 2024-09-21T12:36:18.433 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity_weighted": 0.74000000953674316 2024-09-21T12:36:18.433 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.433 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.433 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.433 INFO:teuthology.orchestra.run.smithi114.stdout: "pool": 11, 2024-09-21T12:36:18.433 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": "unique_pool_4", 2024-09-21T12:36:18.433 INFO:teuthology.orchestra.run.smithi114.stdout: "create_time": "2024-09-21T12:35:46.562644+0000", 2024-09-21T12:36:18.433 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-09-21T12:36:18.433 INFO:teuthology.orchestra.run.smithi114.stdout: "flags_names": "hashpspool", 2024-09-21T12:36:18.433 INFO:teuthology.orchestra.run.smithi114.stdout: "type": 1, 2024-09-21T12:36:18.433 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 3, 2024-09-21T12:36:18.434 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": 2, 2024-09-21T12:36:18.434 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_rule": 0, 2024-09-21T12:36:18.434 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_count": 0, 2024-09-21T12:36:18.434 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_target": 0, 2024-09-21T12:36:18.434 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T12:36:18.434 INFO:teuthology.orchestra.run.smithi114.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T12:36:18.434 INFO:teuthology.orchestra.run.smithi114.stdout: "is_stretch_pool": false, 2024-09-21T12:36:18.434 INFO:teuthology.orchestra.run.smithi114.stdout: "object_hash": 2, 2024-09-21T12:36:18.434 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscale_mode": "off", 2024-09-21T12:36:18.434 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": 16, 2024-09-21T12:36:18.434 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num": 16, 2024-09-21T12:36:18.434 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_placement_num_target": 16, 2024-09-21T12:36:18.435 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_target": 16, 2024-09-21T12:36:18.435 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num_pending": 16, 2024-09-21T12:36:18.435 INFO:teuthology.orchestra.run.smithi114.stdout: "last_pg_merge_meta": { 2024-09-21T12:36:18.435 INFO:teuthology.orchestra.run.smithi114.stdout: "source_pgid": "0.0", 2024-09-21T12:36:18.435 INFO:teuthology.orchestra.run.smithi114.stdout: "ready_epoch": 0, 2024-09-21T12:36:18.435 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_started": 0, 2024-09-21T12:36:18.435 INFO:teuthology.orchestra.run.smithi114.stdout: "last_epoch_clean": 0, 2024-09-21T12:36:18.435 INFO:teuthology.orchestra.run.smithi114.stdout: "source_version": "0'0", 2024-09-21T12:36:18.435 INFO:teuthology.orchestra.run.smithi114.stdout: "target_version": "0'0" 2024-09-21T12:36:18.435 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.435 INFO:teuthology.orchestra.run.smithi114.stdout: "last_change": "408", 2024-09-21T12:36:18.435 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend": "0", 2024-09-21T12:36:18.436 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T12:36:18.436 INFO:teuthology.orchestra.run.smithi114.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T12:36:18.436 INFO:teuthology.orchestra.run.smithi114.stdout: "auid": 0, 2024-09-21T12:36:18.436 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_mode": "selfmanaged", 2024-09-21T12:36:18.436 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_seq": 0, 2024-09-21T12:36:18.436 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_epoch": 0, 2024-09-21T12:36:18.436 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_snaps": [], 2024-09-21T12:36:18.436 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps": "[]", 2024-09-21T12:36:18.436 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_bytes": 0, 2024-09-21T12:36:18.436 INFO:teuthology.orchestra.run.smithi114.stdout: "quota_max_objects": 0, 2024-09-21T12:36:18.436 INFO:teuthology.orchestra.run.smithi114.stdout: "tiers": [], 2024-09-21T12:36:18.436 INFO:teuthology.orchestra.run.smithi114.stdout: "tier_of": -1, 2024-09-21T12:36:18.436 INFO:teuthology.orchestra.run.smithi114.stdout: "read_tier": -1, 2024-09-21T12:36:18.436 INFO:teuthology.orchestra.run.smithi114.stdout: "write_tier": -1, 2024-09-21T12:36:18.436 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_mode": "none", 2024-09-21T12:36:18.437 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_bytes": 0, 2024-09-21T12:36:18.437 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_objects": 0, 2024-09-21T12:36:18.437 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T12:36:18.437 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T12:36:18.437 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T12:36:18.437 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_flush_age": 0, 2024-09-21T12:36:18.437 INFO:teuthology.orchestra.run.smithi114.stdout: "cache_min_evict_age": 0, 2024-09-21T12:36:18.437 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profile": "", 2024-09-21T12:36:18.437 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_params": { 2024-09-21T12:36:18.437 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "none" 2024-09-21T12:36:18.437 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.437 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_period": 0, 2024-09-21T12:36:18.437 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_count": 0, 2024-09-21T12:36:18.437 INFO:teuthology.orchestra.run.smithi114.stdout: "use_gmt_hitset": true, 2024-09-21T12:36:18.437 INFO:teuthology.orchestra.run.smithi114.stdout: "min_read_recency_for_promote": 0, 2024-09-21T12:36:18.438 INFO:teuthology.orchestra.run.smithi114.stdout: "min_write_recency_for_promote": 0, 2024-09-21T12:36:18.438 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T12:36:18.438 INFO:teuthology.orchestra.run.smithi114.stdout: "hit_set_search_last_n": 0, 2024-09-21T12:36:18.438 INFO:teuthology.orchestra.run.smithi114.stdout: "grade_table": [], 2024-09-21T12:36:18.438 INFO:teuthology.orchestra.run.smithi114.stdout: "stripe_width": 0, 2024-09-21T12:36:18.438 INFO:teuthology.orchestra.run.smithi114.stdout: "expected_num_objects": 0, 2024-09-21T12:36:18.438 INFO:teuthology.orchestra.run.smithi114.stdout: "fast_read": false, 2024-09-21T12:36:18.438 INFO:teuthology.orchestra.run.smithi114.stdout: "options": {}, 2024-09-21T12:36:18.438 INFO:teuthology.orchestra.run.smithi114.stdout: "application_metadata": { 2024-09-21T12:36:18.438 INFO:teuthology.orchestra.run.smithi114.stdout: "rados": {} 2024-09-21T12:36:18.438 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.438 INFO:teuthology.orchestra.run.smithi114.stdout: "read_balance": { 2024-09-21T12:36:18.438 INFO:teuthology.orchestra.run.smithi114.stdout: "score_type": "Fair distribution", 2024-09-21T12:36:18.438 INFO:teuthology.orchestra.run.smithi114.stdout: "score_acting": 2.630000114440918, 2024-09-21T12:36:18.439 INFO:teuthology.orchestra.run.smithi114.stdout: "score_stable": 2.630000114440918, 2024-09-21T12:36:18.439 INFO:teuthology.orchestra.run.smithi114.stdout: "optimal_score": 1, 2024-09-21T12:36:18.439 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_acting": 2.630000114440918, 2024-09-21T12:36:18.439 INFO:teuthology.orchestra.run.smithi114.stdout: "raw_score_stable": 2.630000114440918, 2024-09-21T12:36:18.439 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity_weighted": 0.73582649230957031, 2024-09-21T12:36:18.439 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity": 0.8399999737739563, 2024-09-21T12:36:18.439 INFO:teuthology.orchestra.run.smithi114.stdout: "average_primary_affinity_weighted": 0.74000000953674316 2024-09-21T12:36:18.439 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.439 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.439 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:36:18.439 INFO:teuthology.orchestra.run.smithi114.stdout: "osds": [ 2024-09-21T12:36:18.439 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.440 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 0, 2024-09-21T12:36:18.440 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "5715e521-c7dc-41dd-a075-5c96a88fa9b3", 2024-09-21T12:36:18.440 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:36:18.440 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 1, 2024-09-21T12:36:18.440 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 1, 2024-09-21T12:36:18.440 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 1, 2024-09-21T12:36:18.440 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:36:18.440 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:36:18.440 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 8, 2024-09-21T12:36:18.440 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 415, 2024-09-21T12:36:18.440 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:36:18.440 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:36:18.440 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:36:18.440 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.441 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.441 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.441 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6802", 2024-09-21T12:36:18.441 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:36:18.441 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.441 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.441 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.441 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6803", 2024-09-21T12:36:18.441 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:36:18.441 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.441 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.441 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.441 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:36:18.441 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.442 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.442 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.442 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6804", 2024-09-21T12:36:18.442 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:36:18.442 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.442 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.442 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.442 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6805", 2024-09-21T12:36:18.442 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:36:18.442 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.442 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.442 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.442 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:36:18.442 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.443 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.443 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.443 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6808", 2024-09-21T12:36:18.443 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:36:18.443 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.443 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.443 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.443 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6809", 2024-09-21T12:36:18.443 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:36:18.443 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.443 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.443 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.444 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:36:18.444 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.444 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.444 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.444 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6806", 2024-09-21T12:36:18.444 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:36:18.444 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.444 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.444 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.444 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6807", 2024-09-21T12:36:18.444 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3555869816 2024-09-21T12:36:18.444 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.444 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.444 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.445 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6803/3555869816", 2024-09-21T12:36:18.445 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.114:6805/3555869816", 2024-09-21T12:36:18.445 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.114:6809/3555869816", 2024-09-21T12:36:18.445 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.114:6807/3555869816", 2024-09-21T12:36:18.445 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:36:18.445 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:36:18.445 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:36:18.445 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.445 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.445 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.445 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 1, 2024-09-21T12:36:18.445 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "1fe160a7-5972-4e3e-99ce-15e2311ba4c5", 2024-09-21T12:36:18.445 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:36:18.445 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 1, 2024-09-21T12:36:18.446 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 1, 2024-09-21T12:36:18.446 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 1, 2024-09-21T12:36:18.446 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:36:18.446 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:36:18.446 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 13, 2024-09-21T12:36:18.446 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 420, 2024-09-21T12:36:18.446 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:36:18.446 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:36:18.446 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:36:18.446 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.446 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.446 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.446 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6810", 2024-09-21T12:36:18.447 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:36:18.447 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.447 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.447 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.447 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6811", 2024-09-21T12:36:18.447 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:36:18.447 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.447 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.447 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.447 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:36:18.447 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.447 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.447 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.447 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6812", 2024-09-21T12:36:18.448 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:36:18.448 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.448 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.448 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.448 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6813", 2024-09-21T12:36:18.448 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:36:18.448 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.448 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.448 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.448 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:36:18.448 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.448 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.448 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.448 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6816", 2024-09-21T12:36:18.449 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:36:18.449 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.449 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.449 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.449 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6817", 2024-09-21T12:36:18.449 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:36:18.449 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.449 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.449 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.449 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:36:18.449 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.449 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.449 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.449 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6814", 2024-09-21T12:36:18.450 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:36:18.450 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.450 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.450 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.450 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6815", 2024-09-21T12:36:18.450 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2744663863 2024-09-21T12:36:18.450 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.450 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.450 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.450 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6811/2744663863", 2024-09-21T12:36:18.450 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.114:6813/2744663863", 2024-09-21T12:36:18.450 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.114:6817/2744663863", 2024-09-21T12:36:18.450 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.114:6815/2744663863", 2024-09-21T12:36:18.450 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:36:18.451 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:36:18.451 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:36:18.451 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.451 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.451 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.451 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 2, 2024-09-21T12:36:18.451 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "6a63581f-7603-49ae-9ab3-b37e7c91ab91", 2024-09-21T12:36:18.451 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 0, 2024-09-21T12:36:18.451 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 1, 2024-09-21T12:36:18.451 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 1, 2024-09-21T12:36:18.451 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 0, 2024-09-21T12:36:18.451 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:36:18.451 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:36:18.452 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 18, 2024-09-21T12:36:18.452 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 377, 2024-09-21T12:36:18.452 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 388, 2024-09-21T12:36:18.452 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:36:18.452 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:36:18.452 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.452 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.452 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.452 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6818", 2024-09-21T12:36:18.452 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:36:18.452 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.452 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.452 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.452 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6819", 2024-09-21T12:36:18.453 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:36:18.453 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.453 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.453 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.453 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:36:18.453 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.453 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.453 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.453 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6820", 2024-09-21T12:36:18.453 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:36:18.453 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.453 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.453 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.453 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6821", 2024-09-21T12:36:18.454 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:36:18.454 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.454 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.454 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.454 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:36:18.454 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.454 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.454 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.454 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6824", 2024-09-21T12:36:18.454 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:36:18.454 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.454 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.454 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.455 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6825", 2024-09-21T12:36:18.455 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:36:18.455 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.455 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.455 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.455 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:36:18.455 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.455 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.455 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.455 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6822", 2024-09-21T12:36:18.455 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:36:18.455 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.455 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.456 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.456 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6823", 2024-09-21T12:36:18.456 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3740045173 2024-09-21T12:36:18.456 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.456 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.456 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.456 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6819/3740045173", 2024-09-21T12:36:18.456 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.114:6821/3740045173", 2024-09-21T12:36:18.456 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.114:6825/3740045173", 2024-09-21T12:36:18.456 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.114:6823/3740045173", 2024-09-21T12:36:18.456 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:36:18.456 INFO:teuthology.orchestra.run.smithi114.stdout: "exists" 2024-09-21T12:36:18.456 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.457 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.457 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.457 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 3, 2024-09-21T12:36:18.457 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "0ad68cdb-ae5c-4090-8a8d-b67c1177ae42", 2024-09-21T12:36:18.457 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:36:18.457 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 1, 2024-09-21T12:36:18.457 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 1, 2024-09-21T12:36:18.457 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 1, 2024-09-21T12:36:18.457 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:36:18.457 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:36:18.457 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 26, 2024-09-21T12:36:18.457 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 418, 2024-09-21T12:36:18.457 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:36:18.457 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:36:18.458 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:36:18.458 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.458 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.458 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.458 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6826", 2024-09-21T12:36:18.458 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:36:18.458 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.458 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.458 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.458 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6827", 2024-09-21T12:36:18.458 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:36:18.458 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.459 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.459 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.459 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:36:18.459 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.459 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.459 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.459 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6828", 2024-09-21T12:36:18.459 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:36:18.459 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.459 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.459 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.459 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6829", 2024-09-21T12:36:18.459 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:36:18.459 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.460 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.460 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.460 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:36:18.460 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.460 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.460 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.460 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6832", 2024-09-21T12:36:18.460 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:36:18.460 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.460 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.460 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.460 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6833", 2024-09-21T12:36:18.460 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:36:18.460 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.461 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.461 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.461 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:36:18.461 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.461 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.461 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.461 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6830", 2024-09-21T12:36:18.461 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:18.462 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:36:18.463 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:36:18.463 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.463 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.463 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.463 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6831", 2024-09-21T12:36:18.463 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1312590318 2024-09-21T12:36:18.463 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.463 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.464 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.464 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.114:6827/1312590318", 2024-09-21T12:36:18.464 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.114:6829/1312590318", 2024-09-21T12:36:18.464 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.114:6833/1312590318", 2024-09-21T12:36:18.464 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.114:6831/1312590318", 2024-09-21T12:36:18.464 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:36:18.464 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:36:18.464 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:36:18.464 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.464 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.464 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.464 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 4, 2024-09-21T12:36:18.464 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "26bb8a84-e41d-4425-8491-2dea9764b7cf", 2024-09-21T12:36:18.464 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:36:18.465 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 1, 2024-09-21T12:36:18.465 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 1, 2024-09-21T12:36:18.465 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 1, 2024-09-21T12:36:18.465 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:36:18.465 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:36:18.465 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 32, 2024-09-21T12:36:18.465 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 417, 2024-09-21T12:36:18.465 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:36:18.465 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:36:18.465 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:36:18.465 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.465 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.465 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.465 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6800", 2024-09-21T12:36:18.466 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:36:18.466 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.466 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.466 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.466 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6801", 2024-09-21T12:36:18.466 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:36:18.466 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.466 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.466 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.466 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:36:18.466 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.466 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.466 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.467 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6802", 2024-09-21T12:36:18.467 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:36:18.467 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.467 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.467 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.467 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6803", 2024-09-21T12:36:18.467 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:36:18.467 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.467 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.467 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.467 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:36:18.467 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.467 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.467 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.468 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6806", 2024-09-21T12:36:18.468 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:36:18.468 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.468 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.468 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.468 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6807", 2024-09-21T12:36:18.468 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:36:18.468 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.468 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.468 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.468 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:36:18.468 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.468 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.469 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.469 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6804", 2024-09-21T12:36:18.469 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:36:18.469 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.469 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.469 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.469 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6805", 2024-09-21T12:36:18.469 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1391644836 2024-09-21T12:36:18.469 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.469 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.469 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.469 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.183:6801/1391644836", 2024-09-21T12:36:18.469 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.183:6803/1391644836", 2024-09-21T12:36:18.469 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.183:6807/1391644836", 2024-09-21T12:36:18.470 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.183:6805/1391644836", 2024-09-21T12:36:18.470 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:36:18.470 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:36:18.470 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:36:18.470 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.470 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.470 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.470 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 5, 2024-09-21T12:36:18.470 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "33f96871-0869-4281-82b9-bd518987c551", 2024-09-21T12:36:18.470 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:36:18.470 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 1, 2024-09-21T12:36:18.470 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 1, 2024-09-21T12:36:18.470 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 0.476898193359375, 2024-09-21T12:36:18.470 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:36:18.471 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:36:18.471 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 39, 2024-09-21T12:36:18.471 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 415, 2024-09-21T12:36:18.471 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:36:18.471 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:36:18.471 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:36:18.471 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.471 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.471 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.471 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6808", 2024-09-21T12:36:18.471 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:36:18.471 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.471 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.471 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.472 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6809", 2024-09-21T12:36:18.472 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:36:18.472 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.472 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.472 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.472 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:36:18.472 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.472 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.472 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.472 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6810", 2024-09-21T12:36:18.472 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:36:18.472 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.472 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.472 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.473 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6811", 2024-09-21T12:36:18.473 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:36:18.473 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.473 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.473 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.473 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:36:18.473 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.473 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.473 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.473 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6814", 2024-09-21T12:36:18.473 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:36:18.473 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.473 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.473 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.473 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6815", 2024-09-21T12:36:18.474 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:36:18.474 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.474 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.474 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.474 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:36:18.474 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.474 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.474 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.474 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6812", 2024-09-21T12:36:18.474 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:36:18.474 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.474 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.474 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.475 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6813", 2024-09-21T12:36:18.475 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3187643473 2024-09-21T12:36:18.475 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.475 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.475 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.475 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.183:6809/3187643473", 2024-09-21T12:36:18.475 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.183:6811/3187643473", 2024-09-21T12:36:18.475 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.183:6815/3187643473", 2024-09-21T12:36:18.475 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.183:6813/3187643473", 2024-09-21T12:36:18.475 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:36:18.475 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:36:18.475 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:36:18.475 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.475 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.475 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.476 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 6, 2024-09-21T12:36:18.476 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "e09c72b8-ad5a-4779-a874-b149fce6ed70", 2024-09-21T12:36:18.476 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:36:18.476 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 1, 2024-09-21T12:36:18.476 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 1, 2024-09-21T12:36:18.476 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 0.4097137451171875, 2024-09-21T12:36:18.476 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:36:18.476 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:36:18.476 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 47, 2024-09-21T12:36:18.476 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 420, 2024-09-21T12:36:18.476 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:36:18.476 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:36:18.476 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:36:18.477 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.477 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.477 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.477 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6816", 2024-09-21T12:36:18.477 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:36:18.477 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.477 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.477 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.477 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6817", 2024-09-21T12:36:18.477 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:36:18.477 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.477 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.477 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.477 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:36:18.477 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.478 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.478 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.478 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6818", 2024-09-21T12:36:18.478 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:36:18.478 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.478 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.478 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.478 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6819", 2024-09-21T12:36:18.478 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:36:18.478 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.478 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.478 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.478 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:36:18.479 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.479 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.479 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.479 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6822", 2024-09-21T12:36:18.479 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:36:18.479 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.479 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.479 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.479 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6823", 2024-09-21T12:36:18.479 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:36:18.479 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.479 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.479 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.479 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:36:18.480 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.480 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.480 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.480 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6820", 2024-09-21T12:36:18.480 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:36:18.480 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.480 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.480 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.480 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6821", 2024-09-21T12:36:18.480 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3025199631 2024-09-21T12:36:18.480 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.480 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.480 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.481 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.183:6817/3025199631", 2024-09-21T12:36:18.481 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.183:6819/3025199631", 2024-09-21T12:36:18.481 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.183:6823/3025199631", 2024-09-21T12:36:18.481 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.183:6821/3025199631", 2024-09-21T12:36:18.481 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:36:18.481 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:36:18.481 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:36:18.481 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.481 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.481 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.481 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 7, 2024-09-21T12:36:18.481 INFO:teuthology.orchestra.run.smithi114.stdout: "uuid": "1a24194d-1af6-485f-8856-98844a02dd9d", 2024-09-21T12:36:18.481 INFO:teuthology.orchestra.run.smithi114.stdout: "up": 1, 2024-09-21T12:36:18.481 INFO:teuthology.orchestra.run.smithi114.stdout: "in": 1, 2024-09-21T12:36:18.482 INFO:teuthology.orchestra.run.smithi114.stdout: "weight": 1, 2024-09-21T12:36:18.482 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_affinity": 1, 2024-09-21T12:36:18.482 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_begin": 0, 2024-09-21T12:36:18.482 INFO:teuthology.orchestra.run.smithi114.stdout: "last_clean_end": 0, 2024-09-21T12:36:18.482 INFO:teuthology.orchestra.run.smithi114.stdout: "up_from": 54, 2024-09-21T12:36:18.482 INFO:teuthology.orchestra.run.smithi114.stdout: "up_thru": 419, 2024-09-21T12:36:18.482 INFO:teuthology.orchestra.run.smithi114.stdout: "down_at": 0, 2024-09-21T12:36:18.482 INFO:teuthology.orchestra.run.smithi114.stdout: "lost_at": 0, 2024-09-21T12:36:18.482 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrs": { 2024-09-21T12:36:18.482 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.482 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.482 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.482 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6824", 2024-09-21T12:36:18.482 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:36:18.483 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.483 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.483 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.483 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6825", 2024-09-21T12:36:18.483 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:36:18.483 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.483 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.483 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.483 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addrs": { 2024-09-21T12:36:18.483 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.483 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.483 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.483 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6826", 2024-09-21T12:36:18.483 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:36:18.484 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.484 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.484 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.484 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6827", 2024-09-21T12:36:18.484 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:36:18.484 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.484 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.484 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.484 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addrs": { 2024-09-21T12:36:18.484 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.484 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.484 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.484 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6830", 2024-09-21T12:36:18.484 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:36:18.485 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.485 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.485 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.485 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6831", 2024-09-21T12:36:18.485 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:36:18.485 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.485 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.485 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.485 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addrs": { 2024-09-21T12:36:18.485 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-09-21T12:36:18.485 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.485 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-09-21T12:36:18.485 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6828", 2024-09-21T12:36:18.486 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:36:18.486 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.486 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.486 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-21T12:36:18.486 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.183:6829", 2024-09-21T12:36:18.486 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3203337442 2024-09-21T12:36:18.486 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.486 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.486 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.486 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "172.21.15.183:6825/3203337442", 2024-09-21T12:36:18.486 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "172.21.15.183:6827/3203337442", 2024-09-21T12:36:18.486 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_back_addr": "172.21.15.183:6831/3203337442", 2024-09-21T12:36:18.486 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_front_addr": "172.21.15.183:6829/3203337442", 2024-09-21T12:36:18.487 INFO:teuthology.orchestra.run.smithi114.stdout: "state": [ 2024-09-21T12:36:18.487 INFO:teuthology.orchestra.run.smithi114.stdout: "exists", 2024-09-21T12:36:18.487 INFO:teuthology.orchestra.run.smithi114.stdout: "up" 2024-09-21T12:36:18.487 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.487 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.487 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:36:18.487 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_xinfo": [ 2024-09-21T12:36:18.487 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.487 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 0, 2024-09-21T12:36:18.487 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:36:18.487 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:36:18.487 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:36:18.487 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:36:18.487 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 0, 2024-09-21T12:36:18.488 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:35:30.398140+0000", 2024-09-21T12:36:18.488 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:36:18.488 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.488 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.488 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 1, 2024-09-21T12:36:18.488 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:36:18.488 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:36:18.488 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:36:18.488 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:36:18.488 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 0, 2024-09-21T12:36:18.488 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:35:33.349492+0000", 2024-09-21T12:36:18.488 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:36:18.489 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.489 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.489 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 2, 2024-09-21T12:36:18.489 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:36:18.489 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:36:18.489 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:36:18.489 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:36:18.489 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 0, 2024-09-21T12:36:18.489 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:33:33.127156+0000", 2024-09-21T12:36:18.489 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 387 2024-09-21T12:36:18.489 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.489 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.489 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 3, 2024-09-21T12:36:18.489 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:36:18.490 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:36:18.490 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:36:18.490 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:36:18.490 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 0, 2024-09-21T12:36:18.490 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:35:55.565509+0000", 2024-09-21T12:36:18.490 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:36:18.490 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.490 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.490 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 4, 2024-09-21T12:36:18.490 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:36:18.490 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:36:18.490 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:36:18.490 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:36:18.490 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 0, 2024-09-21T12:36:18.491 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:36:11.567661+0000", 2024-09-21T12:36:18.491 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:36:18.491 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.491 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.491 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 5, 2024-09-21T12:36:18.491 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:36:18.491 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:36:18.491 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:36:18.491 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:36:18.491 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 0, 2024-09-21T12:36:18.491 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:35:39.681450+0000", 2024-09-21T12:36:18.491 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:36:18.491 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.491 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.491 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 6, 2024-09-21T12:36:18.492 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:36:18.492 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:36:18.492 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:36:18.492 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:36:18.492 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 0, 2024-09-21T12:36:18.492 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:35:51.483786+0000", 2024-09-21T12:36:18.492 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:36:18.492 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.492 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.492 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": 7, 2024-09-21T12:36:18.492 INFO:teuthology.orchestra.run.smithi114.stdout: "down_stamp": "0.000000", 2024-09-21T12:36:18.492 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_probability": 0, 2024-09-21T12:36:18.492 INFO:teuthology.orchestra.run.smithi114.stdout: "laggy_interval": 0, 2024-09-21T12:36:18.492 INFO:teuthology.orchestra.run.smithi114.stdout: "features": 4540719139924082687, 2024-09-21T12:36:18.492 INFO:teuthology.orchestra.run.smithi114.stdout: "old_weight": 0, 2024-09-21T12:36:18.493 INFO:teuthology.orchestra.run.smithi114.stdout: "last_purged_snaps_scrub": "2024-09-21T12:36:07.434738+0000", 2024-09-21T12:36:18.493 INFO:teuthology.orchestra.run.smithi114.stdout: "dead_epoch": 0 2024-09-21T12:36:18.493 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.493 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:36:18.493 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_upmap": [], 2024-09-21T12:36:18.493 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_upmap_items": [ 2024-09-21T12:36:18.493 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.493 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "4.3", 2024-09-21T12:36:18.493 INFO:teuthology.orchestra.run.smithi114.stdout: "mappings": [ 2024-09-21T12:36:18.493 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.493 INFO:teuthology.orchestra.run.smithi114.stdout: "from": 0, 2024-09-21T12:36:18.493 INFO:teuthology.orchestra.run.smithi114.stdout: "to": 5 2024-09-21T12:36:18.493 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.493 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.493 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.494 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.494 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "4.c", 2024-09-21T12:36:18.494 INFO:teuthology.orchestra.run.smithi114.stdout: "mappings": [ 2024-09-21T12:36:18.494 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.494 INFO:teuthology.orchestra.run.smithi114.stdout: "from": 3, 2024-09-21T12:36:18.494 INFO:teuthology.orchestra.run.smithi114.stdout: "to": 2 2024-09-21T12:36:18.494 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.494 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.494 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.494 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.494 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "4.d", 2024-09-21T12:36:18.494 INFO:teuthology.orchestra.run.smithi114.stdout: "mappings": [ 2024-09-21T12:36:18.494 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.494 INFO:teuthology.orchestra.run.smithi114.stdout: "from": 1, 2024-09-21T12:36:18.495 INFO:teuthology.orchestra.run.smithi114.stdout: "to": 5 2024-09-21T12:36:18.495 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.495 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.495 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.495 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.495 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "4.f", 2024-09-21T12:36:18.495 INFO:teuthology.orchestra.run.smithi114.stdout: "mappings": [ 2024-09-21T12:36:18.495 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.495 INFO:teuthology.orchestra.run.smithi114.stdout: "from": 1, 2024-09-21T12:36:18.495 INFO:teuthology.orchestra.run.smithi114.stdout: "to": 2 2024-09-21T12:36:18.495 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.495 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.495 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.495 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.496 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "4.15", 2024-09-21T12:36:18.496 INFO:teuthology.orchestra.run.smithi114.stdout: "mappings": [ 2024-09-21T12:36:18.496 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.496 INFO:teuthology.orchestra.run.smithi114.stdout: "from": 3, 2024-09-21T12:36:18.496 INFO:teuthology.orchestra.run.smithi114.stdout: "to": 2 2024-09-21T12:36:18.496 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.496 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.496 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.496 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.496 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "4.1c", 2024-09-21T12:36:18.496 INFO:teuthology.orchestra.run.smithi114.stdout: "mappings": [ 2024-09-21T12:36:18.496 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.496 INFO:teuthology.orchestra.run.smithi114.stdout: "from": 1, 2024-09-21T12:36:18.496 INFO:teuthology.orchestra.run.smithi114.stdout: "to": 5 2024-09-21T12:36:18.496 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.497 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.497 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.497 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.497 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "5.1d", 2024-09-21T12:36:18.497 INFO:teuthology.orchestra.run.smithi114.stdout: "mappings": [ 2024-09-21T12:36:18.497 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.497 INFO:teuthology.orchestra.run.smithi114.stdout: "from": 0, 2024-09-21T12:36:18.497 INFO:teuthology.orchestra.run.smithi114.stdout: "to": 3 2024-09-21T12:36:18.497 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.497 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.497 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.497 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:36:18.497 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_upmap_primaries": [], 2024-09-21T12:36:18.497 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_temp": [ 2024-09-21T12:36:18.497 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.498 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "1.9", 2024-09-21T12:36:18.498 INFO:teuthology.orchestra.run.smithi114.stdout: "osds": [ 2024-09-21T12:36:18.498 INFO:teuthology.orchestra.run.smithi114.stdout: 7, 2024-09-21T12:36:18.498 INFO:teuthology.orchestra.run.smithi114.stdout: 3, 2024-09-21T12:36:18.498 INFO:teuthology.orchestra.run.smithi114.stdout: 5 2024-09-21T12:36:18.498 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.498 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.498 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.498 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "1.14", 2024-09-21T12:36:18.498 INFO:teuthology.orchestra.run.smithi114.stdout: "osds": [ 2024-09-21T12:36:18.498 INFO:teuthology.orchestra.run.smithi114.stdout: 7, 2024-09-21T12:36:18.498 INFO:teuthology.orchestra.run.smithi114.stdout: 4, 2024-09-21T12:36:18.498 INFO:teuthology.orchestra.run.smithi114.stdout: 1 2024-09-21T12:36:18.498 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.499 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.499 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.499 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "1.18", 2024-09-21T12:36:18.499 INFO:teuthology.orchestra.run.smithi114.stdout: "osds": [ 2024-09-21T12:36:18.499 INFO:teuthology.orchestra.run.smithi114.stdout: 5, 2024-09-21T12:36:18.499 INFO:teuthology.orchestra.run.smithi114.stdout: 7, 2024-09-21T12:36:18.499 INFO:teuthology.orchestra.run.smithi114.stdout: 1 2024-09-21T12:36:18.499 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.499 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.499 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.499 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "3.9", 2024-09-21T12:36:18.499 INFO:teuthology.orchestra.run.smithi114.stdout: "osds": [ 2024-09-21T12:36:18.499 INFO:teuthology.orchestra.run.smithi114.stdout: 4, 2024-09-21T12:36:18.499 INFO:teuthology.orchestra.run.smithi114.stdout: 7, 2024-09-21T12:36:18.500 INFO:teuthology.orchestra.run.smithi114.stdout: 1 2024-09-21T12:36:18.500 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.500 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.500 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.500 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "3.13", 2024-09-21T12:36:18.500 INFO:teuthology.orchestra.run.smithi114.stdout: "osds": [ 2024-09-21T12:36:18.500 INFO:teuthology.orchestra.run.smithi114.stdout: 7, 2024-09-21T12:36:18.500 INFO:teuthology.orchestra.run.smithi114.stdout: 4, 2024-09-21T12:36:18.500 INFO:teuthology.orchestra.run.smithi114.stdout: 6 2024-09-21T12:36:18.500 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.500 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.500 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.500 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "4.15", 2024-09-21T12:36:18.500 INFO:teuthology.orchestra.run.smithi114.stdout: "osds": [ 2024-09-21T12:36:18.500 INFO:teuthology.orchestra.run.smithi114.stdout: 7, 2024-09-21T12:36:18.501 INFO:teuthology.orchestra.run.smithi114.stdout: 5, 2024-09-21T12:36:18.501 INFO:teuthology.orchestra.run.smithi114.stdout: 6 2024-09-21T12:36:18.501 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.501 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.501 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.501 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "5.4", 2024-09-21T12:36:18.501 INFO:teuthology.orchestra.run.smithi114.stdout: "osds": [ 2024-09-21T12:36:18.501 INFO:teuthology.orchestra.run.smithi114.stdout: 7, 2024-09-21T12:36:18.501 INFO:teuthology.orchestra.run.smithi114.stdout: 5, 2024-09-21T12:36:18.501 INFO:teuthology.orchestra.run.smithi114.stdout: 0 2024-09-21T12:36:18.501 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.501 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.501 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.501 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "5.6", 2024-09-21T12:36:18.501 INFO:teuthology.orchestra.run.smithi114.stdout: "osds": [ 2024-09-21T12:36:18.502 INFO:teuthology.orchestra.run.smithi114.stdout: 5, 2024-09-21T12:36:18.502 INFO:teuthology.orchestra.run.smithi114.stdout: 7, 2024-09-21T12:36:18.502 INFO:teuthology.orchestra.run.smithi114.stdout: 4 2024-09-21T12:36:18.502 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.502 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.502 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.502 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "5.d", 2024-09-21T12:36:18.502 INFO:teuthology.orchestra.run.smithi114.stdout: "osds": [ 2024-09-21T12:36:18.502 INFO:teuthology.orchestra.run.smithi114.stdout: 7, 2024-09-21T12:36:18.502 INFO:teuthology.orchestra.run.smithi114.stdout: 5, 2024-09-21T12:36:18.502 INFO:teuthology.orchestra.run.smithi114.stdout: 0 2024-09-21T12:36:18.502 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.502 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.502 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.502 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "5.14", 2024-09-21T12:36:18.503 INFO:teuthology.orchestra.run.smithi114.stdout: "osds": [ 2024-09-21T12:36:18.503 INFO:teuthology.orchestra.run.smithi114.stdout: 3, 2024-09-21T12:36:18.503 INFO:teuthology.orchestra.run.smithi114.stdout: 7, 2024-09-21T12:36:18.503 INFO:teuthology.orchestra.run.smithi114.stdout: 1 2024-09-21T12:36:18.503 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.503 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.503 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.503 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "5.1e", 2024-09-21T12:36:18.503 INFO:teuthology.orchestra.run.smithi114.stdout: "osds": [ 2024-09-21T12:36:18.503 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-21T12:36:18.503 INFO:teuthology.orchestra.run.smithi114.stdout: 7, 2024-09-21T12:36:18.504 INFO:teuthology.orchestra.run.smithi114.stdout: 1 2024-09-21T12:36:18.504 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.504 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.505 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.505 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "6.3", 2024-09-21T12:36:18.505 INFO:teuthology.orchestra.run.smithi114.stdout: "osds": [ 2024-09-21T12:36:18.505 INFO:teuthology.orchestra.run.smithi114.stdout: 7, 2024-09-21T12:36:18.505 INFO:teuthology.orchestra.run.smithi114.stdout: 6, 2024-09-21T12:36:18.505 INFO:teuthology.orchestra.run.smithi114.stdout: 1 2024-09-21T12:36:18.505 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.505 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.505 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.505 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "6.8", 2024-09-21T12:36:18.505 INFO:teuthology.orchestra.run.smithi114.stdout: "osds": [ 2024-09-21T12:36:18.505 INFO:teuthology.orchestra.run.smithi114.stdout: 7, 2024-09-21T12:36:18.505 INFO:teuthology.orchestra.run.smithi114.stdout: 3, 2024-09-21T12:36:18.505 INFO:teuthology.orchestra.run.smithi114.stdout: 6 2024-09-21T12:36:18.505 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.506 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.506 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.506 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "6.9", 2024-09-21T12:36:18.506 INFO:teuthology.orchestra.run.smithi114.stdout: "osds": [ 2024-09-21T12:36:18.506 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-09-21T12:36:18.506 INFO:teuthology.orchestra.run.smithi114.stdout: 7, 2024-09-21T12:36:18.506 INFO:teuthology.orchestra.run.smithi114.stdout: 3 2024-09-21T12:36:18.506 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.506 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.506 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.506 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "6.12", 2024-09-21T12:36:18.506 INFO:teuthology.orchestra.run.smithi114.stdout: "osds": [ 2024-09-21T12:36:18.506 INFO:teuthology.orchestra.run.smithi114.stdout: 7, 2024-09-21T12:36:18.506 INFO:teuthology.orchestra.run.smithi114.stdout: 4, 2024-09-21T12:36:18.506 INFO:teuthology.orchestra.run.smithi114.stdout: 3 2024-09-21T12:36:18.507 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.507 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.507 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.507 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "6.14", 2024-09-21T12:36:18.507 INFO:teuthology.orchestra.run.smithi114.stdout: "osds": [ 2024-09-21T12:36:18.507 INFO:teuthology.orchestra.run.smithi114.stdout: 4, 2024-09-21T12:36:18.507 INFO:teuthology.orchestra.run.smithi114.stdout: 7, 2024-09-21T12:36:18.507 INFO:teuthology.orchestra.run.smithi114.stdout: 0 2024-09-21T12:36:18.507 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.507 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.507 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.507 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "6.1c", 2024-09-21T12:36:18.507 INFO:teuthology.orchestra.run.smithi114.stdout: "osds": [ 2024-09-21T12:36:18.507 INFO:teuthology.orchestra.run.smithi114.stdout: 7, 2024-09-21T12:36:18.508 INFO:teuthology.orchestra.run.smithi114.stdout: 5, 2024-09-21T12:36:18.508 INFO:teuthology.orchestra.run.smithi114.stdout: 3 2024-09-21T12:36:18.508 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.508 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.508 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.508 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "11.4", 2024-09-21T12:36:18.508 INFO:teuthology.orchestra.run.smithi114.stdout: "osds": [ 2024-09-21T12:36:18.508 INFO:teuthology.orchestra.run.smithi114.stdout: 7, 2024-09-21T12:36:18.508 INFO:teuthology.orchestra.run.smithi114.stdout: 6, 2024-09-21T12:36:18.508 INFO:teuthology.orchestra.run.smithi114.stdout: 1 2024-09-21T12:36:18.508 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.508 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.508 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:18.508 INFO:teuthology.orchestra.run.smithi114.stdout: "pgid": "11.f", 2024-09-21T12:36:18.509 INFO:teuthology.orchestra.run.smithi114.stdout: "osds": [ 2024-09-21T12:36:18.509 INFO:teuthology.orchestra.run.smithi114.stdout: 7, 2024-09-21T12:36:18.509 INFO:teuthology.orchestra.run.smithi114.stdout: 1, 2024-09-21T12:36:18.509 INFO:teuthology.orchestra.run.smithi114.stdout: 0 2024-09-21T12:36:18.509 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:18.509 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.509 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:36:18.509 INFO:teuthology.orchestra.run.smithi114.stdout: "primary_temp": [], 2024-09-21T12:36:18.509 INFO:teuthology.orchestra.run.smithi114.stdout: "blocklist": { 2024-09-21T12:36:18.509 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:6801/237768393": "2024-09-22T12:20:45.528856+0000", 2024-09-21T12:36:18.509 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:6800/237768393": "2024-09-22T12:20:45.528856+0000", 2024-09-21T12:36:18.509 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/898990943": "2024-09-22T12:14:53.998653+0000", 2024-09-21T12:36:18.509 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/3866305956": "2024-09-22T12:20:45.528856+0000", 2024-09-21T12:36:18.509 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/3598782088": "2024-09-22T12:20:45.528856+0000", 2024-09-21T12:36:18.510 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/2620151541": "2024-09-22T12:15:20.045217+0000", 2024-09-21T12:36:18.510 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:6800/1725148595": "2024-09-22T12:14:53.998653+0000", 2024-09-21T12:36:18.510 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/442404805": "2024-09-22T12:14:53.998653+0000", 2024-09-21T12:36:18.510 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:6801/1725148595": "2024-09-22T12:14:53.998653+0000", 2024-09-21T12:36:18.510 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/4173909019": "2024-09-22T12:20:45.528856+0000", 2024-09-21T12:36:18.510 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/2106467614": "2024-09-22T12:15:20.045217+0000", 2024-09-21T12:36:18.510 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/139803357": "2024-09-22T12:15:20.045217+0000", 2024-09-21T12:36:18.510 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:6801/770932721": "2024-09-22T12:15:20.045217+0000", 2024-09-21T12:36:18.510 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:6800/770932721": "2024-09-22T12:15:20.045217+0000", 2024-09-21T12:36:18.510 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/1457547294": "2024-09-22T12:20:45.528856+0000", 2024-09-21T12:36:18.510 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/3416259228": "2024-09-22T12:14:53.998653+0000", 2024-09-21T12:36:18.510 INFO:teuthology.orchestra.run.smithi114.stdout: "172.21.15.114:0/995404371": "2024-09-22T12:20:45.528856+0000" 2024-09-21T12:36:18.510 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.511 INFO:teuthology.orchestra.run.smithi114.stdout: "range_blocklist": {}, 2024-09-21T12:36:18.511 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_profiles": { 2024-09-21T12:36:18.511 INFO:teuthology.orchestra.run.smithi114.stdout: "default": { 2024-09-21T12:36:18.511 INFO:teuthology.orchestra.run.smithi114.stdout: "crush-failure-domain": "osd", 2024-09-21T12:36:18.511 INFO:teuthology.orchestra.run.smithi114.stdout: "k": "2", 2024-09-21T12:36:18.511 INFO:teuthology.orchestra.run.smithi114.stdout: "m": "1", 2024-09-21T12:36:18.511 INFO:teuthology.orchestra.run.smithi114.stdout: "plugin": "jerasure", 2024-09-21T12:36:18.511 INFO:teuthology.orchestra.run.smithi114.stdout: "technique": "reed_sol_van" 2024-09-21T12:36:18.511 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.511 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:18.511 INFO:teuthology.orchestra.run.smithi114.stdout: "removed_snaps_queue": [], 2024-09-21T12:36:18.511 INFO:teuthology.orchestra.run.smithi114.stdout: "new_removed_snaps": [], 2024-09-21T12:36:18.511 INFO:teuthology.orchestra.run.smithi114.stdout: "new_purged_snaps": [], 2024-09-21T12:36:18.511 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_node_flags": {}, 2024-09-21T12:36:18.512 INFO:teuthology.orchestra.run.smithi114.stdout: "device_class_flags": {}, 2024-09-21T12:36:18.512 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode": { 2024-09-21T12:36:18.512 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode_enabled": false, 2024-09-21T12:36:18.512 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_bucket_count": 0, 2024-09-21T12:36:18.512 INFO:teuthology.orchestra.run.smithi114.stdout: "degraded_stretch_mode": 0, 2024-09-21T12:36:18.512 INFO:teuthology.orchestra.run.smithi114.stdout: "recovering_stretch_mode": 0, 2024-09-21T12:36:18.512 INFO:teuthology.orchestra.run.smithi114.stdout: "stretch_mode_bucket": 0 2024-09-21T12:36:18.512 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:18.512 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:36:18.513 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:18 smithi183 ceph-mon[33277]: pgmap v816: 178 pgs: 1 creating+peering, 2 active+undersized+remapped, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 123 active+clean; 2.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 62 MiB/s wr, 985 op/s; 8686/104133 objects degraded (8.341%); 4352/104133 objects misplaced (4.179%); 22 MiB/s, 347 objects/s recovering 2024-09-21T12:36:18.513 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:18 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:36:18.513 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 25 16 44608 44592 111.466 113.188 0.00627465 0.00896593 2024-09-21T12:36:18.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:18.514 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:18.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:18.615 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:18.650 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:18.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:18.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:18.752 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:18.782 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:18.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:18.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:18.884 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:18.919 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:18.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:18.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:19.021 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:19.058 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:19.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:19.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:19.160 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:19.162 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-21T12:36:19.200 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:19.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:19.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:19.302 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:19.312 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T12:36:19.334 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:19.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:19.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:19.436 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:19.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:19.468 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:19.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:19 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:36:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:19 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3939508439' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:36:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:19 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:36:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:19 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2075493290' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T12:36:19.503 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 26 16 46506 46490 111.741 118.625 0.00698879 0.00894432 2024-09-21T12:36:19.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:19 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:36:19.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:19 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3939508439' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:36:19.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:19 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:36:19.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:19 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2075493290' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T12:36:19.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:19 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:36:19.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:19 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3939508439' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:36:19.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:19 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:36:19.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:19 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2075493290' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T12:36:19.569 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:19.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:19.601 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:19.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:19.702 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:19.738 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:19.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:19.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:19.840 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:19.869 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:19.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:19.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:19.971 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:20.002 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:20.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:20.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:20.104 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:20.137 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:20.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:20.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:20.239 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:20.269 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:20.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:20.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:20.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:20 smithi114 ceph-mon[30324]: pgmap v818: 178 pgs: 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 126 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 793 B/s rd, 99 MiB/s wr, 1.58k op/s; 10628/127425 objects degraded (8.341%); 5274/127425 objects misplaced (4.139%) 2024-09-21T12:36:20.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:20 smithi114 ceph-mon[30324]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:36:20.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:20 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:36:20.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:20 smithi114 ceph-mon[30324]: osdmap e423: 8 total, 7 up, 8 in 2024-09-21T12:36:20.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:20 smithi114 ceph-mon[40867]: pgmap v818: 178 pgs: 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 126 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 793 B/s rd, 99 MiB/s wr, 1.58k op/s; 10628/127425 objects degraded (8.341%); 5274/127425 objects misplaced (4.139%) 2024-09-21T12:36:20.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:20 smithi114 ceph-mon[40867]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:36:20.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:20 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:36:20.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:20 smithi114 ceph-mon[40867]: osdmap e423: 8 total, 7 up, 8 in 2024-09-21T12:36:20.371 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:20.402 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:20.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:20.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:20 smithi183 ceph-mon[33277]: pgmap v818: 178 pgs: 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 126 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 793 B/s rd, 99 MiB/s wr, 1.58k op/s; 10628/127425 objects degraded (8.341%); 5274/127425 objects misplaced (4.139%) 2024-09-21T12:36:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:20 smithi183 ceph-mon[33277]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:36:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:20 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:36:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:20 smithi183 ceph-mon[33277]: osdmap e423: 8 total, 7 up, 8 in 2024-09-21T12:36:20.503 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 27 15 48469 48454 112.148 122.75 0.00814486 0.0089121 2024-09-21T12:36:20.503 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:20.536 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:20.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:20.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:20.637 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:20.668 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:20.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:20.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:20.770 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:20.803 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:20.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:20.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:20.901 INFO:tasks.thrashosds.thrasher:j is {'epoch': 422, 'fsid': 'e2ad975c-7812-11ef-baf6-efdc52797490', 'created': '2024-09-21T12:14:24.181227+0000', 'modified': '2024-09-21T12:36:17.094187+0000', 'last_up_change': '2024-09-21T12:34:33.320113+0000', 'last_in_change': '2024-09-21T12:35:59.095504+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T12:17:38.357647+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '218', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '216', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.809999942779541, 'score_stable': 1.809999942779541, 'optimal_score': 1, 'raw_score_acting': 1.809999942779541, 'raw_score_stable': 1.809999942779541, 'primary_affinity_weighted': 0.7358264923095703, 'average_primary_affinity': 0.8399999737739563, 'average_primary_affinity_weighted': 0.7400000095367432}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-21T12:20:12.761046+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 61, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3299999237060547, 'score_stable': 2.3299999237060547, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.7358264923095703, 'average_primary_affinity': 0.8399999737739563, 'average_primary_affinity_weighted': 0.7400000095367432}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-21T12:20:13.713971+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5299999713897705, 'score_stable': 1.5299999713897705, 'optimal_score': 1, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.7358264923095703, 'average_primary_affinity': 0.8399999737739563, 'average_primary_affinity_weighted': 0.7400000095367432}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-21T12:20:15.368765+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 1, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 0.7358264923095703, 'average_primary_affinity': 0.8399999737739563, 'average_primary_affinity_weighted': 0.7400000095367432}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-21T12:20:17.314275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5299999713897705, 'score_stable': 1.5299999713897705, 'optimal_score': 1, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.7358264923095703, 'average_primary_affinity': 0.8399999737739563, 'average_primary_affinity_weighted': 0.7400000095367432}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-21T12:20:19.370218+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 1, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.7358264923095703, 'average_primary_affinity': 0.8399999737739563, 'average_primary_affinity_weighted': 0.7400000095367432}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-09-21T12:35:46.562644+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '408', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.630000114440918, 'score_stable': 2.630000114440918, 'optimal_score': 1, 'raw_score_acting': 2.630000114440918, 'raw_score_stable': 2.630000114440918, 'primary_affinity_weighted': 0.7358264923095703, 'average_primary_affinity': 0.8399999737739563, 'average_primary_affinity_weighted': 0.7400000095367432}}], 'osds': [{'osd': 0, 'uuid': '5715e521-c7dc-41dd-a075-5c96a88fa9b3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 415, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6802', 'nonce': 3555869816}, {'type': 'v1', 'addr': '172.21.15.114:6803', 'nonce': 3555869816}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6804', 'nonce': 3555869816}, {'type': 'v1', 'addr': '172.21.15.114:6805', 'nonce': 3555869816}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6808', 'nonce': 3555869816}, {'type': 'v1', 'addr': '172.21.15.114:6809', 'nonce': 3555869816}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6806', 'nonce': 3555869816}, {'type': 'v1', 'addr': '172.21.15.114:6807', 'nonce': 3555869816}]}, 'public_addr': '172.21.15.114:6803/3555869816', 'cluster_addr': '172.21.15.114:6805/3555869816', 'heartbeat_back_addr': '172.21.15.114:6809/3555869816', 'heartbeat_front_addr': '172.21.15.114:6807/3555869816', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '1fe160a7-5972-4e3e-99ce-15e2311ba4c5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 420, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6810', 'nonce': 2744663863}, {'type': 'v1', 'addr': '172.21.15.114:6811', 'nonce': 2744663863}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6812', 'nonce': 2744663863}, {'type': 'v1', 'addr': '172.21.15.114:6813', 'nonce': 2744663863}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6816', 'nonce': 2744663863}, {'type': 'v1', 'addr': '172.21.15.114:6817', 'nonce': 2744663863}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6814', 'nonce': 2744663863}, {'type': 'v1', 'addr': '172.21.15.114:6815', 'nonce': 2744663863}]}, 'public_addr': '172.21.15.114:6811/2744663863', 'cluster_addr': '172.21.15.114:6813/2744663863', 'heartbeat_back_addr': '172.21.15.114:6817/2744663863', 'heartbeat_front_addr': '172.21.15.114:6815/2744663863', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '6a63581f-7603-49ae-9ab3-b37e7c91ab91', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 377, 'down_at': 388, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6818', 'nonce': 3740045173}, {'type': 'v1', 'addr': '172.21.15.114:6819', 'nonce': 3740045173}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6820', 'nonce': 3740045173}, {'type': 'v1', 'addr': '172.21.15.114:6821', 'nonce': 3740045173}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6824', 'nonce': 3740045173}, {'type': 'v1', 'addr': '172.21.15.114:6825', 'nonce': 3740045173}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6822', 'nonce': 3740045173}, {'type': 'v1', 'addr': '172.21.15.114:6823', 'nonce': 3740045173}]}, 'public_addr': '172.21.15.114:6819/3740045173', 'cluster_addr': '172.21.15.114:6821/3740045173', 'heartbeat_back_addr': '172.21.15.114:6825/3740045173', 'heartbeat_front_addr': '172.21.15.114:6823/3740045173', 'state': ['exists']}, {'osd': 3, 'uuid': '0ad68cdb-ae5c-4090-8a8d-b67c1177ae42', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 418, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6826', 'nonce': 1312590318}, {'type': 'v1', 'addr': '172.21.15.114:6827', 'nonce': 1312590318}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6828', 'nonce': 1312590318}, {'type': 'v1', 'addr': '172.21.15.114:6829', 'nonce': 1312590318}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6832', 'nonce': 1312590318}, {'type': 'v1', 'addr': '172.21.15.114:6833', 'nonce': 1312590318}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6830', 'nonce': 1312590318}, {'type': 'v1', 'addr': '172.21.15.114:6831', 'nonce': 1312590318}]}, 'public_addr': '172.21.15.114:6827/1312590318', 'cluster_addr': '172.21.15.114:6829/1312590318', 'heartbeat_back_addr': '172.21.15.114:6833/1312590318', 'heartbeat_front_addr': '172.21.15.114:6831/1312590318', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '26bb8a84-e41d-4425-8491-2dea9764b7cf', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 417, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6800', 'nonce': 1391644836}, {'type': 'v1', 'addr': '172.21.15.183:6801', 'nonce': 1391644836}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6802', 'nonce': 1391644836}, {'type': 'v1', 'addr': '172.21.15.183:6803', 'nonce': 1391644836}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6806', 'nonce': 1391644836}, {'type': 'v1', 'addr': '172.21.15.183:6807', 'nonce': 1391644836}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6804', 'nonce': 1391644836}, {'type': 'v1', 'addr': '172.21.15.183:6805', 'nonce': 1391644836}]}, 'public_addr': '172.21.15.183:6801/1391644836', 'cluster_addr': '172.21.15.183:6803/1391644836', 'heartbeat_back_addr': '172.21.15.183:6807/1391644836', 'heartbeat_front_addr': '172.21.15.183:6805/1391644836', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '33f96871-0869-4281-82b9-bd518987c551', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.476898193359375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 415, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6808', 'nonce': 3187643473}, {'type': 'v1', 'addr': '172.21.15.183:6809', 'nonce': 3187643473}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6810', 'nonce': 3187643473}, {'type': 'v1', 'addr': '172.21.15.183:6811', 'nonce': 3187643473}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6814', 'nonce': 3187643473}, {'type': 'v1', 'addr': '172.21.15.183:6815', 'nonce': 3187643473}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6812', 'nonce': 3187643473}, {'type': 'v1', 'addr': '172.21.15.183:6813', 'nonce': 3187643473}]}, 'public_addr': '172.21.15.183:6809/3187643473', 'cluster_addr': '172.21.15.183:6811/3187643473', 'heartbeat_back_addr': '172.21.15.183:6815/3187643473', 'heartbeat_front_addr': '172.21.15.183:6813/3187643473', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e09c72b8-ad5a-4779-a874-b149fce6ed70', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.4097137451171875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 420, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6816', 'nonce': 3025199631}, {'type': 'v1', 'addr': '172.21.15.183:6817', 'nonce': 3025199631}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6818', 'nonce': 3025199631}, {'type': 'v1', 'addr': '172.21.15.183:6819', 'nonce': 3025199631}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6822', 'nonce': 3025199631}, {'type': 'v1', 'addr': '172.21.15.183:6823', 'nonce': 3025199631}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6820', 'nonce': 3025199631}, {'type': 'v1', 'addr': '172.21.15.183:6821', 'nonce': 3025199631}]}, 'public_addr': '172.21.15.183:6817/3025199631', 'cluster_addr': '172.21.15.183:6819/3025199631', 'heartbeat_back_addr': '172.21.15.183:6823/3025199631', 'heartbeat_front_addr': '172.21.15.183:6821/3025199631', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1a24194d-1af6-485f-8856-98844a02dd9d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 419, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6824', 'nonce': 3203337442}, {'type': 'v1', 'addr': '172.21.15.183:6825', 'nonce': 3203337442}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6826', 'nonce': 3203337442}, {'type': 'v1', 'addr': '172.21.15.183:6827', 'nonce': 3203337442}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6830', 'nonce': 3203337442}, {'type': 'v1', 'addr': '172.21.15.183:6831', 'nonce': 3203337442}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6828', 'nonce': 3203337442}, {'type': 'v1', 'addr': '172.21.15.183:6829', 'nonce': 3203337442}]}, 'public_addr': '172.21.15.183:6825/3203337442', 'cluster_addr': '172.21.15.183:6827/3203337442', 'heartbeat_back_addr': '172.21.15.183:6831/3203337442', 'heartbeat_front_addr': '172.21.15.183:6829/3203337442', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T12:35:30.398140+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T12:35:33.349492+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T12:33:33.127156+0000', 'dead_epoch': 387}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T12:35:55.565509+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T12:36:11.567661+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T12:35:39.681450+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T12:35:51.483786+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T12:36:07.434738+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.3', 'mappings': [{'from': 0, 'to': 5}]}, {'pgid': '4.c', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.15', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.1c', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '5.1d', 'mappings': [{'from': 0, 'to': 3}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '1.9', 'osds': [7, 3, 5]}, {'pgid': '1.14', 'osds': [7, 4, 1]}, {'pgid': '1.18', 'osds': [5, 7, 1]}, {'pgid': '3.9', 'osds': [4, 7, 1]}, {'pgid': '3.13', 'osds': [7, 4, 6]}, {'pgid': '4.15', 'osds': [7, 5, 6]}, {'pgid': '5.4', 'osds': [7, 5, 0]}, {'pgid': '5.6', 'osds': [5, 7, 4]}, {'pgid': '5.d', 'osds': [7, 5, 0]}, {'pgid': '5.14', 'osds': [3, 7, 1]}, {'pgid': '5.1e', 'osds': [0, 7, 1]}, {'pgid': '6.3', 'osds': [7, 6, 1]}, {'pgid': '6.8', 'osds': [7, 3, 6]}, {'pgid': '6.9', 'osds': [0, 7, 3]}, {'pgid': '6.12', 'osds': [7, 4, 3]}, {'pgid': '6.14', 'osds': [4, 7, 0]}, {'pgid': '6.1c', 'osds': [7, 5, 3]}, {'pgid': '11.4', 'osds': [7, 6, 1]}, {'pgid': '11.f', 'osds': [7, 1, 0]}], 'primary_temp': [], 'blocklist': {'172.21.15.114:6801/237768393': '2024-09-22T12:20:45.528856+0000', '172.21.15.114:6800/237768393': '2024-09-22T12:20:45.528856+0000', '172.21.15.114:0/898990943': '2024-09-22T12:14:53.998653+0000', '172.21.15.114:0/3866305956': '2024-09-22T12:20:45.528856+0000', '172.21.15.114:0/3598782088': '2024-09-22T12:20:45.528856+0000', '172.21.15.114:0/2620151541': '2024-09-22T12:15:20.045217+0000', '172.21.15.114:6800/1725148595': '2024-09-22T12:14:53.998653+0000', '172.21.15.114:0/442404805': '2024-09-22T12:14:53.998653+0000', '172.21.15.114:6801/1725148595': '2024-09-22T12:14:53.998653+0000', '172.21.15.114:0/4173909019': '2024-09-22T12:20:45.528856+0000', '172.21.15.114:0/2106467614': '2024-09-22T12:15:20.045217+0000', '172.21.15.114:0/139803357': '2024-09-22T12:15:20.045217+0000', '172.21.15.114:6801/770932721': '2024-09-22T12:15:20.045217+0000', '172.21.15.114:6800/770932721': '2024-09-22T12:15:20.045217+0000', '172.21.15.114:0/1457547294': '2024-09-22T12:20:45.528856+0000', '172.21.15.114:0/3416259228': '2024-09-22T12:14:53.998653+0000', '172.21.15.114:0/995404371': '2024-09-22T12:20:45.528856+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T12:36:20.901 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:36:20.905 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:20.920 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:36:20.943 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:36:20.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:20.955 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:20.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:21.056 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:21.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:21.085 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:21.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:21.186 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:21.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:21.245 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:21.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:21.346 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:21.375 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:21.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:21.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:21.477 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:21.503 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 28 16 50248 50232 112.111 111.125 0.0116031 0.00891482 2024-09-21T12:36:21.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:21.510 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:21.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:21.611 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:21.642 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:21.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:21.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:21.744 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:21.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:21.779 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:21.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:21.858 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:36:21.858 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:36:21.858 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:36:21.858 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:36:21.859 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:36:21.880 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:21.916 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:21.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:21.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:22.018 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:22.053 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:22.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:22.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:22.155 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:22.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:22.180 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:22.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:22.281 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:22.315 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:22.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:22.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:22.417 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:22.452 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:22.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:22.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:22.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:22 smithi183 ceph-mon[33277]: pgmap v820: 178 pgs: 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 126 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 687 B/s rd, 68 MiB/s wr, 1.09k op/s; 10628/127425 objects degraded (8.341%); 5274/127425 objects misplaced (4.139%) 2024-09-21T12:36:22.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:22 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/139205790' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:36:22.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:22 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 10628/127425 objects degraded (8.341%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:22.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:22 smithi183 ceph-mon[33277]: osdmap e424: 8 total, 7 up, 8 in 2024-09-21T12:36:22.504 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 29 16 51844 51828 111.685 99.75 0.00811052 0.00894245 2024-09-21T12:36:22.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:22 smithi114 ceph-mon[40867]: pgmap v820: 178 pgs: 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 126 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 687 B/s rd, 68 MiB/s wr, 1.09k op/s; 10628/127425 objects degraded (8.341%); 5274/127425 objects misplaced (4.139%) 2024-09-21T12:36:22.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:22 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/139205790' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:36:22.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:22 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 10628/127425 objects degraded (8.341%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:22.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:22 smithi114 ceph-mon[40867]: osdmap e424: 8 total, 7 up, 8 in 2024-09-21T12:36:22.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:22 smithi114 ceph-mon[30324]: pgmap v820: 178 pgs: 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 126 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 687 B/s rd, 68 MiB/s wr, 1.09k op/s; 10628/127425 objects degraded (8.341%); 5274/127425 objects misplaced (4.139%) 2024-09-21T12:36:22.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:22 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/139205790' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:36:22.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:22 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 10628/127425 objects degraded (8.341%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:22.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:22 smithi114 ceph-mon[30324]: osdmap e424: 8 total, 7 up, 8 in 2024-09-21T12:36:22.554 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:22.589 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:22.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:22.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:22.690 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:22.724 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:22.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:22.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:22.726 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T12:36:22.826 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:22.852 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:22.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:22.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:22.883 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:36:22.883 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:36:22.883 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:36:22.883 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:36:22.883 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:36:22.883 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:36:22.883 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T12:36:22.883 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:36:22.883 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:36:22.883 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:36:22.884 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:36:22.884 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:36:22.884 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:36:22.884 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:36:22.884 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:36:22.884 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:36:22 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:36:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:36:22.954 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:22.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:22.994 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:22.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:23.095 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:23.124 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:23.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:23.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:23.225 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:23.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:23.262 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:23.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:23.362 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:23.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:23.396 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:23.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:23.497 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:23.503 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 30 15 53587 53572 111.595 109 0.0162362 0.0089561 2024-09-21T12:36:23.532 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:23.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:23.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:23.633 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:23.669 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:23.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:23.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:23.745 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:36:23.771 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:23.791 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T12:36:23.792 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T12:36:23.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:23.809 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:23.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:23.910 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:23.946 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:23.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:23.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:24.048 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:24.080 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:24.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:24.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:24.182 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:24.216 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:24.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:24.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:24.318 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:24.355 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:24.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:24.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:24.395 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T12:36:24.444 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:24 smithi114 ceph-mon[40867]: pgmap v822: 178 pgs: 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 126 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 137 MiB/s wr, 2.19k op/s; 11821/143484 objects degraded (8.239%); 6443/143484 objects misplaced (4.490%) 2024-09-21T12:36:24.445 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:24 smithi114 ceph-mon[30324]: pgmap v822: 178 pgs: 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 126 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 137 MiB/s wr, 2.19k op/s; 11821/143484 objects degraded (8.239%); 6443/143484 objects misplaced (4.490%) 2024-09-21T12:36:24.457 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:24.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:24 smithi183 ceph-mon[33277]: pgmap v822: 178 pgs: 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 126 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 137 MiB/s wr, 2.19k op/s; 11821/143484 objects degraded (8.239%); 6443/143484 objects misplaced (4.490%) 2024-09-21T12:36:24.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:24.498 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:24.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:24.504 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 31 16 55372 55356 111.591 111.5 0.0131166 0.00895536 2024-09-21T12:36:24.598 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:24.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:24.653 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:24.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:24.754 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:24.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:24.812 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:24.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:24.884 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:36:24.912 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:24.937 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:24.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:24.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:25.039 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:25.069 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:25.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:25.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:25.171 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:25.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:25.218 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:25.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:25.293 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:36:25.296 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:36:25.319 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:25.351 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:25.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:25.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:25.452 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:25.480 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:25.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:25.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:25.504 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 32 16 56934 56918 111.155 97.625 0.0130642 0.0089916 2024-09-21T12:36:25.582 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:25.623 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:25.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:25.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:25.725 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:25.734 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:36:25.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:25.767 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:25.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:25.792 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:36:25.868 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:25.899 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:25.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:25.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:26.001 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:26.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:26.039 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:26.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:26.140 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:26.173 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:26.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:26.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:26.190 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-21T12:36:26.274 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:26.311 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:26.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:26.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:26.394 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:26 smithi114 ceph-mon[30324]: pgmap v823: 178 pgs: 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 126 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 687 B/s rd, 110 MiB/s wr, 1.76k op/s; 12893/155127 objects degraded (8.311%); 6443/155127 objects misplaced (4.153%) 2024-09-21T12:36:26.394 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:26 smithi114 ceph-mon[30324]: from='client.26195 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:36:26.394 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:26 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1939901651' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:36:26.394 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:26 smithi114 ceph-mon[40867]: pgmap v823: 178 pgs: 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 126 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 687 B/s rd, 110 MiB/s wr, 1.76k op/s; 12893/155127 objects degraded (8.311%); 6443/155127 objects misplaced (4.153%) 2024-09-21T12:36:26.394 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:26 smithi114 ceph-mon[40867]: from='client.26195 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:36:26.394 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:26 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1939901651' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:36:26.414 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:26.442 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:26.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:26.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:26.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:26 smithi183 ceph-mon[33277]: pgmap v823: 178 pgs: 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 126 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 687 B/s rd, 110 MiB/s wr, 1.76k op/s; 12893/155127 objects degraded (8.311%); 6443/155127 objects misplaced (4.153%) 2024-09-21T12:36:26.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:26 smithi183 ceph-mon[33277]: from='client.26195 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:36:26.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:26 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1939901651' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:36:26.504 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 33 16 58490 58474 110.733 97.25 0.0101239 0.00902599 2024-09-21T12:36:26.544 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:26.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:26.581 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:26.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:26.682 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:26.718 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:26.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:26.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:26.820 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:26.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:26.856 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:26.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:26.958 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:26.987 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:26.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:26.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:27.089 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:27.130 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:27.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:27.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:27.232 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:27.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:27 smithi183 ceph-mon[33277]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:36:27.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:27 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1939901651' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:36:27.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:27 smithi183 ceph-mon[33277]: osdmap e425: 8 total, 7 up, 8 in 2024-09-21T12:36:27.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:27 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 12893/155127 objects degraded (8.311%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:27.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:27 smithi183 ceph-mon[33277]: osdmap e426: 8 total, 7 up, 8 in 2024-09-21T12:36:27.239 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:36:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=sqlstore.transactions t=2024-09-21T12:36:26.931304437Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T12:36:27.239 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:36:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=sqlstore.transactions t=2024-09-21T12:36:26.942436362Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-21T12:36:27.239 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:36:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=sqlstore.transactions t=2024-09-21T12:36:26.953246206Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-09-21T12:36:27.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:27.266 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:27.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:27.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:27 smithi114 ceph-mon[40867]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:36:27.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:27 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1939901651' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:36:27.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:27 smithi114 ceph-mon[40867]: osdmap e425: 8 total, 7 up, 8 in 2024-09-21T12:36:27.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:27 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 12893/155127 objects degraded (8.311%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:27.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:27 smithi114 ceph-mon[40867]: osdmap e426: 8 total, 7 up, 8 in 2024-09-21T12:36:27.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:27 smithi114 ceph-mon[30324]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:36:27.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:27 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1939901651' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:36:27.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:27 smithi114 ceph-mon[30324]: osdmap e425: 8 total, 7 up, 8 in 2024-09-21T12:36:27.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:27 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 12893/155127 objects degraded (8.311%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:27.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:27 smithi114 ceph-mon[30324]: osdmap e426: 8 total, 7 up, 8 in 2024-09-21T12:36:27.368 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:27.403 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:27.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:27.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:27.504 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 34 16 60214 60198 110.645 107.75 0.00479528 0.00903222 2024-09-21T12:36:27.504 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:27.544 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:27.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:27.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:27.546 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":823,"stamp":"2024-09-21T12:36:24.549881+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3376382701,"num_objects":51709,"num_object_clones":0,"num_object_copies":155127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12893,"num_objects_misplaced":6443,"num_objects_unfound":0,"num_objects_dirty":51709,"num_whiteouts":0,"num_read":1152,"num_read_kb":908,"num_write":52237,"num_write_kb":3296759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4699,"num_bytes_recovered":275092355,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54768,"ondisk_log_size":54768,"up":482,"acting":501,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":556,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":16669216,"kb_used_data":11371856,"kb_used_omap":12,"kb_used_meta":5297203,"kb_avail":733292000,"statfs":{"total":767960285184,"available":750891008000,"internally_reserved":0,"allocated":11644780544,"data_stored":11641301616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":5424336432},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,2,8,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":859242496,"num_objects":13111,"num_object_clones":0,"num_object_copies":39333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3372,"num_objects_misplaced":1169,"num_objects_unfound":0,"num_objects_dirty":13111,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":13111,"num_write_kb":839104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.451647"},"pg_stats":[{"pgid":"1.1d","version":"76'79","reported_seq":469,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.167107+0000","last_change":"2024-09-21T12:32:38.427111+0000","last_active":"2024-09-21T12:36:19.167107+0000","last_peered":"2024-09-21T12:36:19.167107+0000","last_clean":"2024-09-21T12:36:19.167107+0000","last_became_active":"2024-09-21T12:32:38.426383+0000","last_became_peered":"2024-09-21T12:32:38.426383+0000","last_unstale":"2024-09-21T12:36:19.167107+0000","last_undegraded":"2024-09-21T12:36:19.167107+0000","last_fullsized":"2024-09-21T12:36:19.167107+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:32.060116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:32.060116+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:14.142065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":785,"reported_epoch":424,"state":"active+undersized","last_fresh":"2024-09-21T12:36:22.179725+0000","last_change":"2024-09-21T12:34:34.340741+0000","last_active":"2024-09-21T12:36:22.179725+0000","last_peered":"2024-09-21T12:36:22.179725+0000","last_clean":"2024-09-21T12:34:29.289357+0000","last_became_active":"2024-09-21T12:34:34.340741+0000","last_became_peered":"2024-09-21T12:34:34.340741+0000","last_unstale":"2024-09-21T12:36:22.179725+0000","last_undegraded":"2024-09-21T12:36:22.179725+0000","last_fullsized":"2024-09-21T12:34:34.332209+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:29.086830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:29.086830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":169,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.158291+0000","last_change":"2024-09-21T12:32:38.443995+0000","last_active":"2024-09-21T12:36:19.158291+0000","last_peered":"2024-09-21T12:36:19.158291+0000","last_clean":"2024-09-21T12:36:19.158291+0000","last_became_active":"2024-09-21T12:32:38.419081+0000","last_became_peered":"2024-09-21T12:32:38.419081+0000","last_unstale":"2024-09-21T12:36:19.158291+0000","last_undegraded":"2024-09-21T12:36:19.158291+0000","last_fullsized":"2024-09-21T12:36:19.158291+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:29:08.671723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:29:08.671723+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:18.917504+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":781,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.165733+0000","last_change":"2024-09-21T12:36:00.136989+0000","last_active":"2024-09-21T12:36:19.165733+0000","last_peered":"2024-09-21T12:36:19.165733+0000","last_clean":"2024-09-21T12:36:19.165733+0000","last_became_active":"2024-09-21T12:36:00.136673+0000","last_became_peered":"2024-09-21T12:36:00.136673+0000","last_unstale":"2024-09-21T12:36:19.165733+0000","last_undegraded":"2024-09-21T12:36:19.165733+0000","last_fullsized":"2024-09-21T12:36:19.165733+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:54.969048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:35:54.969048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:20.733223+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":167,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.158843+0000","last_change":"2024-09-21T12:33:27.411258+0000","last_active":"2024-09-21T12:36:19.158843+0000","last_peered":"2024-09-21T12:36:19.158843+0000","last_clean":"2024-09-21T12:36:19.158843+0000","last_became_active":"2024-09-21T12:32:38.425588+0000","last_became_peered":"2024-09-21T12:32:38.425588+0000","last_unstale":"2024-09-21T12:36:19.158843+0000","last_undegraded":"2024-09-21T12:36:19.158843+0000","last_fullsized":"2024-09-21T12:36:19.158843+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:27.411142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:27.411142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"76'79","reported_seq":28,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.170995+0000","last_change":"2024-09-21T12:36:00.154588+0000","last_active":"2024-09-21T12:36:19.170995+0000","last_peered":"2024-09-21T12:36:19.170995+0000","last_clean":"2024-09-21T12:36:19.170995+0000","last_became_active":"2024-09-21T12:36:00.154209+0000","last_became_peered":"2024-09-21T12:36:00.154209+0000","last_unstale":"2024-09-21T12:36:19.170995+0000","last_undegraded":"2024-09-21T12:36:19.170995+0000","last_fullsized":"2024-09-21T12:36:19.170995+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:35:54.010909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:35:54.010909+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:12.176158+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":771,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.180232+0000","last_change":"2024-09-21T12:36:00.139483+0000","last_active":"2024-09-21T12:36:22.180232+0000","last_peered":"2024-09-21T12:36:22.180232+0000","last_clean":"2024-09-21T12:36:22.180232+0000","last_became_active":"2024-09-21T12:36:00.139167+0000","last_became_peered":"2024-09-21T12:36:00.139167+0000","last_unstale":"2024-09-21T12:36:22.180232+0000","last_undegraded":"2024-09-21T12:36:22.180232+0000","last_fullsized":"2024-09-21T12:36:22.180232+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:05.904553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:34:05.904553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":783,"reported_epoch":424,"state":"active+undersized","last_fresh":"2024-09-21T12:36:22.178955+0000","last_change":"2024-09-21T12:34:34.346389+0000","last_active":"2024-09-21T12:36:22.178955+0000","last_peered":"2024-09-21T12:36:22.178955+0000","last_clean":"2024-09-21T12:34:29.294384+0000","last_became_active":"2024-09-21T12:34:34.346389+0000","last_became_peered":"2024-09-21T12:34:34.346389+0000","last_unstale":"2024-09-21T12:36:22.178955+0000","last_undegraded":"2024-09-21T12:36:22.178955+0000","last_fullsized":"2024-09-21T12:34:34.336359+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:06.698999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:06.698999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:33.676355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":820,"reported_epoch":424,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:22.182608+0000","last_change":"2024-09-21T12:34:34.337065+0000","last_active":"2024-09-21T12:36:22.182608+0000","last_peered":"2024-09-21T12:36:22.182608+0000","last_clean":"2024-09-21T12:34:29.291199+0000","last_became_active":"2024-09-21T12:34:34.337065+0000","last_became_peered":"2024-09-21T12:34:34.337065+0000","last_unstale":"2024-09-21T12:36:22.182608+0000","last_undegraded":"2024-09-21T12:34:34.335679+0000","last_fullsized":"2024-09-21T12:34:34.335428+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:31:48.294300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:48.294300+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:37.589687+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":159,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.158214+0000","last_change":"2024-09-21T12:32:38.418041+0000","last_active":"2024-09-21T12:36:19.158214+0000","last_peered":"2024-09-21T12:36:19.158214+0000","last_clean":"2024-09-21T12:36:19.158214+0000","last_became_active":"2024-09-21T12:32:38.417724+0000","last_became_peered":"2024-09-21T12:32:38.417724+0000","last_unstale":"2024-09-21T12:36:19.158214+0000","last_undegraded":"2024-09-21T12:36:19.158214+0000","last_fullsized":"2024-09-21T12:36:19.158214+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:39.179665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:39.179665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:32.962894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":167,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.159916+0000","last_change":"2024-09-21T12:33:32.373658+0000","last_active":"2024-09-21T12:36:19.159916+0000","last_peered":"2024-09-21T12:36:19.159916+0000","last_clean":"2024-09-21T12:36:19.159916+0000","last_became_active":"2024-09-21T12:32:38.428077+0000","last_became_peered":"2024-09-21T12:32:38.428077+0000","last_unstale":"2024-09-21T12:36:19.159916+0000","last_undegraded":"2024-09-21T12:36:19.159916+0000","last_fullsized":"2024-09-21T12:36:19.159916+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:32.372832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:32.372832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":177,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.159181+0000","last_change":"2024-09-21T12:33:03.312402+0000","last_active":"2024-09-21T12:36:19.159181+0000","last_peered":"2024-09-21T12:36:19.159181+0000","last_clean":"2024-09-21T12:36:19.159181+0000","last_became_active":"2024-09-21T12:32:38.426121+0000","last_became_peered":"2024-09-21T12:32:38.426121+0000","last_unstale":"2024-09-21T12:36:19.159181+0000","last_undegraded":"2024-09-21T12:36:19.159181+0000","last_fullsized":"2024-09-21T12:36:19.159181+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:33:03.312251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:03.312251+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:41.849595+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":820,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.173915+0000","last_change":"2024-09-21T12:36:00.138341+0000","last_active":"2024-09-21T12:36:19.173915+0000","last_peered":"2024-09-21T12:36:19.173915+0000","last_clean":"2024-09-21T12:36:19.173915+0000","last_became_active":"2024-09-21T12:36:00.137364+0000","last_became_peered":"2024-09-21T12:36:00.137364+0000","last_unstale":"2024-09-21T12:36:19.173915+0000","last_undegraded":"2024-09-21T12:36:19.173915+0000","last_fullsized":"2024-09-21T12:36:19.173915+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:13.198376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:13.198376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":781,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.171162+0000","last_change":"2024-09-21T12:36:00.135034+0000","last_active":"2024-09-21T12:36:22.171162+0000","last_peered":"2024-09-21T12:36:22.171162+0000","last_clean":"2024-09-21T12:36:22.171162+0000","last_became_active":"2024-09-21T12:36:00.134723+0000","last_became_peered":"2024-09-21T12:36:00.134723+0000","last_unstale":"2024-09-21T12:36:22.171162+0000","last_undegraded":"2024-09-21T12:36:22.171162+0000","last_fullsized":"2024-09-21T12:36:22.171162+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:16.147705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:34:16.147705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"76'79","reported_seq":556,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.178684+0000","last_change":"2024-09-21T12:36:00.148161+0000","last_active":"2024-09-21T12:36:22.178684+0000","last_peered":"2024-09-21T12:36:22.178684+0000","last_clean":"2024-09-21T12:36:22.178684+0000","last_became_active":"2024-09-21T12:36:00.147795+0000","last_became_peered":"2024-09-21T12:36:00.147795+0000","last_unstale":"2024-09-21T12:36:22.178684+0000","last_undegraded":"2024-09-21T12:36:22.178684+0000","last_fullsized":"2024-09-21T12:36:22.178684+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:36.752562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:36.752562+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":167,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.159643+0000","last_change":"2024-09-21T12:35:14.244213+0000","last_active":"2024-09-21T12:36:19.159643+0000","last_peered":"2024-09-21T12:36:19.159643+0000","last_clean":"2024-09-21T12:36:19.159643+0000","last_became_active":"2024-09-21T12:32:38.426142+0000","last_became_peered":"2024-09-21T12:32:38.426142+0000","last_unstale":"2024-09-21T12:36:19.159643+0000","last_undegraded":"2024-09-21T12:36:19.159643+0000","last_fullsized":"2024-09-21T12:36:19.159643+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:14.243954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:14.243954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:40.353667+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":173,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.158960+0000","last_change":"2024-09-21T12:33:26.395059+0000","last_active":"2024-09-21T12:36:19.158960+0000","last_peered":"2024-09-21T12:36:19.158960+0000","last_clean":"2024-09-21T12:36:19.158960+0000","last_became_active":"2024-09-21T12:32:38.422912+0000","last_became_peered":"2024-09-21T12:32:38.422912+0000","last_unstale":"2024-09-21T12:36:19.158960+0000","last_undegraded":"2024-09-21T12:36:19.158960+0000","last_fullsized":"2024-09-21T12:36:19.158960+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'7","last_scrub_stamp":"2024-09-21T12:33:26.394982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:26.394982+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":28,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.170575+0000","last_change":"2024-09-21T12:36:00.143221+0000","last_active":"2024-09-21T12:36:19.170575+0000","last_peered":"2024-09-21T12:36:19.170575+0000","last_clean":"2024-09-21T12:36:19.170575+0000","last_became_active":"2024-09-21T12:36:00.142106+0000","last_became_peered":"2024-09-21T12:36:00.142106+0000","last_unstale":"2024-09-21T12:36:19.170575+0000","last_undegraded":"2024-09-21T12:36:19.170575+0000","last_fullsized":"2024-09-21T12:36:19.170575+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:09.558418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:35:09.558418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":780,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.173192+0000","last_change":"2024-09-21T12:33:36.125346+0000","last_active":"2024-09-21T12:36:19.173192+0000","last_peered":"2024-09-21T12:36:19.173192+0000","last_clean":"2024-09-21T12:36:19.173192+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:36:19.173192+0000","last_undegraded":"2024-09-21T12:36:19.173192+0000","last_fullsized":"2024-09-21T12:36:19.173192+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:36.124095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:36.124095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"76'79","reported_seq":604,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.184493+0000","last_change":"2024-09-21T12:34:48.959991+0000","last_active":"2024-09-21T12:36:22.184493+0000","last_peered":"2024-09-21T12:36:22.184493+0000","last_clean":"2024-09-21T12:36:22.184493+0000","last_became_active":"2024-09-21T12:27:16.957569+0000","last_became_peered":"2024-09-21T12:27:16.957569+0000","last_unstale":"2024-09-21T12:36:22.184493+0000","last_undegraded":"2024-09-21T12:36:22.184493+0000","last_fullsized":"2024-09-21T12:36:22.184493+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:48.959916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:48.959916+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":167,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.159933+0000","last_change":"2024-09-21T12:34:05.193021+0000","last_active":"2024-09-21T12:36:19.159933+0000","last_peered":"2024-09-21T12:36:19.159933+0000","last_clean":"2024-09-21T12:36:19.159933+0000","last_became_active":"2024-09-21T12:32:38.428997+0000","last_became_peered":"2024-09-21T12:32:38.428997+0000","last_unstale":"2024-09-21T12:36:19.159933+0000","last_undegraded":"2024-09-21T12:36:19.159933+0000","last_fullsized":"2024-09-21T12:36:19.159933+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:05.192971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:34:05.192971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":844,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.171933+0000","last_change":"2024-09-21T12:36:00.127510+0000","last_active":"2024-09-21T12:36:22.171933+0000","last_peered":"2024-09-21T12:36:22.171933+0000","last_clean":"2024-09-21T12:36:22.171933+0000","last_became_active":"2024-09-21T12:36:00.127153+0000","last_became_peered":"2024-09-21T12:36:00.127153+0000","last_unstale":"2024-09-21T12:36:22.171933+0000","last_undegraded":"2024-09-21T12:36:22.171933+0000","last_fullsized":"2024-09-21T12:36:22.171933+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:56.016840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:56.016840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:07.001046+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":823,"reported_epoch":423,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:19.174050+0000","last_change":"2024-09-21T12:34:34.341728+0000","last_active":"2024-09-21T12:36:19.174050+0000","last_peered":"2024-09-21T12:36:19.174050+0000","last_clean":"2024-09-21T12:34:29.305282+0000","last_became_active":"2024-09-21T12:34:34.341728+0000","last_became_peered":"2024-09-21T12:34:34.341728+0000","last_unstale":"2024-09-21T12:36:19.174050+0000","last_undegraded":"2024-09-21T12:34:34.336150+0000","last_fullsized":"2024-09-21T12:34:34.336049+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:33:01.668584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:01.668584+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:47.258634+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":803,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.167377+0000","last_change":"2024-09-21T12:35:50.957724+0000","last_active":"2024-09-21T12:36:19.167377+0000","last_peered":"2024-09-21T12:36:19.167377+0000","last_clean":"2024-09-21T12:36:19.167377+0000","last_became_active":"2024-09-21T12:32:38.425958+0000","last_became_peered":"2024-09-21T12:32:38.425958+0000","last_unstale":"2024-09-21T12:36:19.167377+0000","last_undegraded":"2024-09-21T12:36:19.167377+0000","last_fullsized":"2024-09-21T12:36:19.167377+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:50.957677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:35:50.957677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:02.566364+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"76'79","reported_seq":731,"reported_epoch":423,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:19.173003+0000","last_change":"2024-09-21T12:36:01.149228+0000","last_active":"2024-09-21T12:36:19.173003+0000","last_peered":"2024-09-21T12:36:19.173003+0000","last_clean":"2024-09-21T12:36:19.173003+0000","last_became_active":"2024-09-21T12:36:01.148686+0000","last_became_peered":"2024-09-21T12:36:01.148686+0000","last_unstale":"2024-09-21T12:36:19.173003+0000","last_undegraded":"2024-09-21T12:36:19.173003+0000","last_fullsized":"2024-09-21T12:36:19.173003+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:27.294674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:27.294674+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:14.322523+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7,1],"avail_no_missing":["5","1","7"],"object_location_counts":[{"shards":"1,5,7","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":787,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.177745+0000","last_change":"2024-09-21T12:35:06.944278+0000","last_active":"2024-09-21T12:36:22.177745+0000","last_peered":"2024-09-21T12:36:22.177745+0000","last_clean":"2024-09-21T12:36:22.177745+0000","last_became_active":"2024-09-21T12:32:02.033276+0000","last_became_peered":"2024-09-21T12:32:02.033276+0000","last_unstale":"2024-09-21T12:36:22.177745+0000","last_undegraded":"2024-09-21T12:36:22.177745+0000","last_fullsized":"2024-09-21T12:36:22.177745+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:06.944209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:06.944209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:36.548698+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":163,"reported_epoch":423,"state":"active+undersized","last_fresh":"2024-09-21T12:36:19.158578+0000","last_change":"2024-09-21T12:34:34.349252+0000","last_active":"2024-09-21T12:36:19.158578+0000","last_peered":"2024-09-21T12:36:19.158578+0000","last_clean":"2024-09-21T12:34:29.278130+0000","last_became_active":"2024-09-21T12:34:34.349252+0000","last_became_peered":"2024-09-21T12:34:34.349252+0000","last_unstale":"2024-09-21T12:36:19.158578+0000","last_undegraded":"2024-09-21T12:36:19.158578+0000","last_fullsized":"2024-09-21T12:34:34.329566+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:45.725968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.725968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:14.888145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":881,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.180176+0000","last_change":"2024-09-21T12:35:59.641966+0000","last_active":"2024-09-21T12:36:22.180176+0000","last_peered":"2024-09-21T12:36:22.180176+0000","last_clean":"2024-09-21T12:36:22.180176+0000","last_became_active":"2024-09-21T12:32:38.409163+0000","last_became_peered":"2024-09-21T12:32:38.409163+0000","last_unstale":"2024-09-21T12:36:22.180176+0000","last_undegraded":"2024-09-21T12:36:22.180176+0000","last_fullsized":"2024-09-21T12:36:22.180176+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T12:35:59.641895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:59.641895+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:19.348495+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":171,"reported_epoch":423,"state":"active+undersized","last_fresh":"2024-09-21T12:36:19.159378+0000","last_change":"2024-09-21T12:34:34.349550+0000","last_active":"2024-09-21T12:36:19.159378+0000","last_peered":"2024-09-21T12:36:19.159378+0000","last_clean":"2024-09-21T12:34:29.281006+0000","last_became_active":"2024-09-21T12:34:34.349550+0000","last_became_peered":"2024-09-21T12:34:34.349550+0000","last_unstale":"2024-09-21T12:36:19.159378+0000","last_undegraded":"2024-09-21T12:36:19.159378+0000","last_fullsized":"2024-09-21T12:34:34.331038+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:53.270634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:32:53.270634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:29.979680+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"76'79","reported_seq":518,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.173712+0000","last_change":"2024-09-21T12:34:46.125581+0000","last_active":"2024-09-21T12:36:19.173712+0000","last_peered":"2024-09-21T12:36:19.173712+0000","last_clean":"2024-09-21T12:36:19.173712+0000","last_became_active":"2024-09-21T12:32:38.414204+0000","last_became_peered":"2024-09-21T12:32:38.414204+0000","last_unstale":"2024-09-21T12:36:19.173712+0000","last_undegraded":"2024-09-21T12:36:19.173712+0000","last_fullsized":"2024-09-21T12:36:19.173712+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:46.125518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:46.125518+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":746,"reported_epoch":423,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:19.170772+0000","last_change":"2024-09-21T12:36:01.127357+0000","last_active":"2024-09-21T12:36:19.170772+0000","last_peered":"2024-09-21T12:36:19.170772+0000","last_clean":"2024-09-21T12:36:19.170772+0000","last_became_active":"2024-09-21T12:36:01.126897+0000","last_became_peered":"2024-09-21T12:36:01.126897+0000","last_unstale":"2024-09-21T12:36:19.170772+0000","last_undegraded":"2024-09-21T12:36:19.170772+0000","last_fullsized":"2024-09-21T12:36:19.170772+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:33:37.344821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:37.344821+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5,3],"avail_no_missing":["7","3","5"],"object_location_counts":[{"shards":"3,5,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":821,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.165853+0000","last_change":"2024-09-21T12:34:42.279770+0000","last_active":"2024-09-21T12:36:19.165853+0000","last_peered":"2024-09-21T12:36:19.165853+0000","last_clean":"2024-09-21T12:36:19.165853+0000","last_became_active":"2024-09-21T12:32:38.425335+0000","last_became_peered":"2024-09-21T12:32:38.425335+0000","last_unstale":"2024-09-21T12:36:19.165853+0000","last_undegraded":"2024-09-21T12:36:19.165853+0000","last_fullsized":"2024-09-21T12:36:19.165853+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:34:42.279704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:34:42.279704+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":900,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.181119+0000","last_change":"2024-09-21T12:36:00.173918+0000","last_active":"2024-09-21T12:36:22.181119+0000","last_peered":"2024-09-21T12:36:22.181119+0000","last_clean":"2024-09-21T12:36:22.181119+0000","last_became_active":"2024-09-21T12:36:00.148761+0000","last_became_peered":"2024-09-21T12:36:00.148761+0000","last_unstale":"2024-09-21T12:36:22.181119+0000","last_undegraded":"2024-09-21T12:36:22.181119+0000","last_fullsized":"2024-09-21T12:36:22.181119+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:32:57.082525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:57.082525+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:19.889528+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"415'18","reported_seq":230,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.160302+0000","last_change":"2024-09-21T12:36:00.148175+0000","last_active":"2024-09-21T12:36:19.160302+0000","last_peered":"2024-09-21T12:36:19.160302+0000","last_clean":"2024-09-21T12:36:19.160302+0000","last_became_active":"2024-09-21T12:36:00.147676+0000","last_became_peered":"2024-09-21T12:36:00.147676+0000","last_unstale":"2024-09-21T12:36:19.160302+0000","last_undegraded":"2024-09-21T12:36:19.160302+0000","last_fullsized":"2024-09-21T12:36:19.160302+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'15","last_scrub_stamp":"2024-09-21T12:35:11.182313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:35:11.182313+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:37.713379+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"423'3179","reported_seq":5399,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:21.545825+0000","last_change":"2024-09-21T12:36:13.108581+0000","last_active":"2024-09-21T12:36:21.545825+0000","last_peered":"2024-09-21T12:36:21.545825+0000","last_clean":"2024-09-21T12:36:21.545825+0000","last_became_active":"2024-09-21T12:36:13.105161+0000","last_became_peered":"2024-09-21T12:36:13.105161+0000","last_unstale":"2024-09-21T12:36:21.545825+0000","last_undegraded":"2024-09-21T12:36:21.545825+0000","last_fullsized":"2024-09-21T12:36:21.545825+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3179,"log_dups_size":0,"ondisk_log_size":3179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:51.244739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208338944,"num_objects":3179,"num_object_clones":0,"num_object_copies":9537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3179,"num_write_kb":203456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1078,"num_bytes_recovered":70647808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"76'79","reported_seq":330,"reported_epoch":423,"state":"active+undersized","last_fresh":"2024-09-21T12:36:19.173866+0000","last_change":"2024-09-21T12:34:34.347159+0000","last_active":"2024-09-21T12:36:19.173866+0000","last_peered":"2024-09-21T12:36:19.173866+0000","last_clean":"2024-09-21T12:34:29.305642+0000","last_became_active":"2024-09-21T12:34:34.347159+0000","last_became_peered":"2024-09-21T12:34:34.347159+0000","last_unstale":"2024-09-21T12:36:19.173866+0000","last_undegraded":"2024-09-21T12:36:19.173866+0000","last_fullsized":"2024-09-21T12:34:34.340605+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:18.151644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:18.151644+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":434,"reported_epoch":424,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:22.177909+0000","last_change":"2024-09-21T12:34:34.345532+0000","last_active":"2024-09-21T12:36:22.177909+0000","last_peered":"2024-09-21T12:36:22.177909+0000","last_clean":"2024-09-21T12:34:29.295616+0000","last_became_active":"2024-09-21T12:34:34.345532+0000","last_became_peered":"2024-09-21T12:34:34.345532+0000","last_unstale":"2024-09-21T12:36:22.177909+0000","last_undegraded":"2024-09-21T12:34:34.334307+0000","last_fullsized":"2024-09-21T12:34:34.334181+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:31:56.722241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:56.722241+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:43.647722+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":252,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.180424+0000","last_change":"2024-09-21T12:36:00.140935+0000","last_active":"2024-09-21T12:36:22.180424+0000","last_peered":"2024-09-21T12:36:22.180424+0000","last_clean":"2024-09-21T12:36:22.180424+0000","last_became_active":"2024-09-21T12:36:00.140455+0000","last_became_peered":"2024-09-21T12:36:00.140455+0000","last_unstale":"2024-09-21T12:36:22.180424+0000","last_undegraded":"2024-09-21T12:36:22.180424+0000","last_fullsized":"2024-09-21T12:36:22.180424+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:05.703464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:05.703464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:21.552824+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"352'15","reported_seq":216,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.159883+0000","last_change":"2024-09-21T12:35:50.989423+0000","last_active":"2024-09-21T12:36:19.159883+0000","last_peered":"2024-09-21T12:36:19.159883+0000","last_clean":"2024-09-21T12:36:19.159883+0000","last_became_active":"2024-09-21T12:32:38.429092+0000","last_became_peered":"2024-09-21T12:32:38.429092+0000","last_unstale":"2024-09-21T12:36:19.159883+0000","last_undegraded":"2024-09-21T12:36:19.159883+0000","last_fullsized":"2024-09-21T12:36:19.159883+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'15","last_scrub_stamp":"2024-09-21T12:35:50.989377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:35:50.989377+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:19.146860+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":264,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.167281+0000","last_change":"2024-09-21T12:35:24.989679+0000","last_active":"2024-09-21T12:36:19.167281+0000","last_peered":"2024-09-21T12:36:19.167281+0000","last_clean":"2024-09-21T12:36:19.167281+0000","last_became_active":"2024-09-21T12:31:39.054600+0000","last_became_peered":"2024-09-21T12:31:39.054600+0000","last_unstale":"2024-09-21T12:36:19.167281+0000","last_undegraded":"2024-09-21T12:36:19.167281+0000","last_fullsized":"2024-09-21T12:36:19.167281+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:35:24.989615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:24.989615+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:51.845745+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.f","version":"423'3277","reported_seq":3325,"reported_epoch":423,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:21.885149+0000","last_change":"2024-09-21T12:36:01.154474+0000","last_active":"2024-09-21T12:36:21.885149+0000","last_peered":"2024-09-21T12:36:21.885149+0000","last_clean":"2024-09-21T12:36:21.885149+0000","last_became_active":"2024-09-21T12:36:01.133058+0000","last_became_peered":"2024-09-21T12:36:01.133058+0000","last_unstale":"2024-09-21T12:36:21.885149+0000","last_undegraded":"2024-09-21T12:36:21.885149+0000","last_fullsized":"2024-09-21T12:36:21.885149+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3277,"log_dups_size":0,"ondisk_log_size":3277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:53.471384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214761472,"num_objects":3277,"num_object_clones":0,"num_object_copies":9831,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3277,"num_objects_unfound":0,"num_objects_dirty":3277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3277,"num_write_kb":209728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,0],"avail_no_missing":["7","0","1"],"object_location_counts":[{"shards":"0,1,7","objects":3277}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"76'79","reported_seq":171,"reported_epoch":423,"state":"active+undersized","last_fresh":"2024-09-21T12:36:19.159824+0000","last_change":"2024-09-21T12:34:34.348261+0000","last_active":"2024-09-21T12:36:19.159824+0000","last_peered":"2024-09-21T12:36:19.159824+0000","last_clean":"2024-09-21T12:34:29.279263+0000","last_became_active":"2024-09-21T12:34:34.348261+0000","last_became_peered":"2024-09-21T12:34:34.348261+0000","last_unstale":"2024-09-21T12:36:19.159824+0000","last_undegraded":"2024-09-21T12:36:19.159824+0000","last_fullsized":"2024-09-21T12:34:34.332145+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:00.343916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:00.343916+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:34.992366+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":933,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.181004+0000","last_change":"2024-09-21T12:36:00.163117+0000","last_active":"2024-09-21T12:36:22.181004+0000","last_peered":"2024-09-21T12:36:22.181004+0000","last_clean":"2024-09-21T12:36:22.181004+0000","last_became_active":"2024-09-21T12:36:00.139849+0000","last_became_peered":"2024-09-21T12:36:00.139849+0000","last_unstale":"2024-09-21T12:36:22.181004+0000","last_undegraded":"2024-09-21T12:36:22.181004+0000","last_fullsized":"2024-09-21T12:36:22.181004+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T12:35:12.790809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:12.790809+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:32.833661+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":784,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.178352+0000","last_change":"2024-09-21T12:36:00.152050+0000","last_active":"2024-09-21T12:36:22.178352+0000","last_peered":"2024-09-21T12:36:22.178352+0000","last_clean":"2024-09-21T12:36:22.178352+0000","last_became_active":"2024-09-21T12:36:00.151488+0000","last_became_peered":"2024-09-21T12:36:00.151488+0000","last_unstale":"2024-09-21T12:36:22.178352+0000","last_undegraded":"2024-09-21T12:36:22.178352+0000","last_fullsized":"2024-09-21T12:36:22.178352+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:22.752960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:22.752960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:38.395609+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":181,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.158405+0000","last_change":"2024-09-21T12:36:00.137014+0000","last_active":"2024-09-21T12:36:19.158405+0000","last_peered":"2024-09-21T12:36:19.158405+0000","last_clean":"2024-09-21T12:36:19.158405+0000","last_became_active":"2024-09-21T12:36:00.136650+0000","last_became_peered":"2024-09-21T12:36:00.136650+0000","last_unstale":"2024-09-21T12:36:19.158405+0000","last_undegraded":"2024-09-21T12:36:19.158405+0000","last_fullsized":"2024-09-21T12:36:19.158405+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:12.216104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:12.216104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":163,"reported_epoch":423,"state":"active+undersized","last_fresh":"2024-09-21T12:36:19.158939+0000","last_change":"2024-09-21T12:34:34.347165+0000","last_active":"2024-09-21T12:36:19.158939+0000","last_peered":"2024-09-21T12:36:19.158939+0000","last_clean":"2024-09-21T12:34:29.278468+0000","last_became_active":"2024-09-21T12:34:34.347165+0000","last_became_peered":"2024-09-21T12:34:34.347165+0000","last_unstale":"2024-09-21T12:36:19.158939+0000","last_undegraded":"2024-09-21T12:36:19.158939+0000","last_fullsized":"2024-09-21T12:34:34.331768+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:05.143736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:29:05.143736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"423'3170","reported_seq":5268,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:21.864871+0000","last_change":"2024-09-21T12:36:08.103615+0000","last_active":"2024-09-21T12:36:21.864871+0000","last_peered":"2024-09-21T12:36:21.864871+0000","last_clean":"2024-09-21T12:36:21.864871+0000","last_became_active":"2024-09-21T12:36:08.101796+0000","last_became_peered":"2024-09-21T12:36:08.101796+0000","last_unstale":"2024-09-21T12:36:21.864871+0000","last_undegraded":"2024-09-21T12:36:21.864871+0000","last_fullsized":"2024-09-21T12:36:21.864871+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3170,"log_dups_size":0,"ondisk_log_size":3170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:16.601019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207749120,"num_objects":3170,"num_object_clones":0,"num_object_copies":9510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3170,"num_write_kb":202880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1016,"num_bytes_recovered":66584576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"76'79","reported_seq":711,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.174077+0000","last_change":"2024-09-21T12:36:00.155500+0000","last_active":"2024-09-21T12:36:22.174077+0000","last_peered":"2024-09-21T12:36:22.174077+0000","last_clean":"2024-09-21T12:36:22.174077+0000","last_became_active":"2024-09-21T12:36:00.155047+0000","last_became_peered":"2024-09-21T12:36:00.155047+0000","last_unstale":"2024-09-21T12:36:22.174077+0000","last_undegraded":"2024-09-21T12:36:22.174077+0000","last_fullsized":"2024-09-21T12:36:22.174077+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:33.718619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:33.718619+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":175,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.158314+0000","last_change":"2024-09-21T12:32:38.457999+0000","last_active":"2024-09-21T12:36:19.158314+0000","last_peered":"2024-09-21T12:36:19.158314+0000","last_clean":"2024-09-21T12:36:19.158314+0000","last_became_active":"2024-09-21T12:32:38.425453+0000","last_became_peered":"2024-09-21T12:32:38.425453+0000","last_unstale":"2024-09-21T12:36:19.158314+0000","last_undegraded":"2024-09-21T12:36:19.158314+0000","last_fullsized":"2024-09-21T12:36:19.158314+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-21T12:26:51.463698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:51.463698+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:07.901778+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":40,"num_read_kb":33,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":859,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.181426+0000","last_change":"2024-09-21T12:34:48.162040+0000","last_active":"2024-09-21T12:36:22.181426+0000","last_peered":"2024-09-21T12:36:22.181426+0000","last_clean":"2024-09-21T12:36:22.181426+0000","last_became_active":"2024-09-21T12:32:38.424412+0000","last_became_peered":"2024-09-21T12:32:38.424412+0000","last_unstale":"2024-09-21T12:36:22.181426+0000","last_undegraded":"2024-09-21T12:36:22.181426+0000","last_fullsized":"2024-09-21T12:36:22.181426+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:34:48.161956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:34:48.161956+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"352'12","reported_seq":1016,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.180775+0000","last_change":"2024-09-21T12:34:07.827409+0000","last_active":"2024-09-21T12:36:22.180775+0000","last_peered":"2024-09-21T12:36:22.180775+0000","last_clean":"2024-09-21T12:36:22.180775+0000","last_became_active":"2024-09-21T12:32:38.409177+0000","last_became_peered":"2024-09-21T12:32:38.409177+0000","last_unstale":"2024-09-21T12:36:22.180775+0000","last_undegraded":"2024-09-21T12:36:22.180775+0000","last_fullsized":"2024-09-21T12:36:22.180775+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'12","last_scrub_stamp":"2024-09-21T12:34:07.827315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:07.827315+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":754,"reported_epoch":423,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:19.171051+0000","last_change":"2024-09-21T12:36:01.138656+0000","last_active":"2024-09-21T12:36:19.171051+0000","last_peered":"2024-09-21T12:36:19.171051+0000","last_clean":"2024-09-21T12:36:19.171051+0000","last_became_active":"2024-09-21T12:36:01.135064+0000","last_became_peered":"2024-09-21T12:36:01.135064+0000","last_unstale":"2024-09-21T12:36:19.171051+0000","last_undegraded":"2024-09-21T12:36:19.171051+0000","last_fullsized":"2024-09-21T12:36:19.171051+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:59.406201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:32:59.406201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6,1],"avail_no_missing":["7","1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"423'3086","reported_seq":3130,"reported_epoch":423,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:21.858739+0000","last_change":"2024-09-21T12:35:53.507442+0000","last_active":"2024-09-21T12:36:21.858739+0000","last_peered":"2024-09-21T12:36:21.858739+0000","last_clean":"2024-09-21T12:35:46.767214+0000","last_became_active":"2024-09-21T12:35:47.791328+0000","last_became_peered":"2024-09-21T12:35:47.791328+0000","last_unstale":"2024-09-21T12:36:21.858739+0000","last_undegraded":"2024-09-21T12:35:53.507160+0000","last_fullsized":"2024-09-21T12:35:47.787870+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3086,"log_dups_size":0,"ondisk_log_size":3086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:03.374268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202244096,"num_objects":3086,"num_object_clones":0,"num_object_copies":9258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3086,"num_write_kb":197504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":3086}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"76'79","reported_seq":252,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.178202+0000","last_change":"2024-09-21T12:36:00.151734+0000","last_active":"2024-09-21T12:36:22.178202+0000","last_peered":"2024-09-21T12:36:22.178202+0000","last_clean":"2024-09-21T12:36:22.178202+0000","last_became_active":"2024-09-21T12:36:00.151076+0000","last_became_peered":"2024-09-21T12:36:00.151076+0000","last_unstale":"2024-09-21T12:36:22.178202+0000","last_undegraded":"2024-09-21T12:36:22.178202+0000","last_fullsized":"2024-09-21T12:36:22.178202+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:31:50.320834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:31:50.320834+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:13.696624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":310,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.166458+0000","last_change":"2024-09-21T12:34:17.380869+0000","last_active":"2024-09-21T12:36:19.166458+0000","last_peered":"2024-09-21T12:36:19.166458+0000","last_clean":"2024-09-21T12:36:19.166458+0000","last_became_active":"2024-09-21T12:32:38.424412+0000","last_became_peered":"2024-09-21T12:32:38.424412+0000","last_unstale":"2024-09-21T12:36:19.166458+0000","last_undegraded":"2024-09-21T12:36:19.166458+0000","last_fullsized":"2024-09-21T12:36:19.166458+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:34:17.380538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:34:17.380538+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":789,"reported_epoch":423,"state":"active+undersized","last_fresh":"2024-09-21T12:36:19.174485+0000","last_change":"2024-09-21T12:34:34.339986+0000","last_active":"2024-09-21T12:36:19.174485+0000","last_peered":"2024-09-21T12:36:19.174485+0000","last_clean":"2024-09-21T12:34:29.305656+0000","last_became_active":"2024-09-21T12:34:34.339986+0000","last_became_peered":"2024-09-21T12:34:34.339986+0000","last_unstale":"2024-09-21T12:36:19.174485+0000","last_undegraded":"2024-09-21T12:36:19.174485+0000","last_fullsized":"2024-09-21T12:34:34.336347+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:19.629083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:19.629083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:42.094853+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":978,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.180113+0000","last_change":"2024-09-21T12:34:17.207726+0000","last_active":"2024-09-21T12:36:22.180113+0000","last_peered":"2024-09-21T12:36:22.180113+0000","last_clean":"2024-09-21T12:36:22.180113+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:36:22.180113+0000","last_undegraded":"2024-09-21T12:36:22.180113+0000","last_fullsized":"2024-09-21T12:36:22.180113+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:34:17.207117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:17.207117+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:42.051141+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":765,"reported_epoch":424,"state":"active+undersized","last_fresh":"2024-09-21T12:36:22.171530+0000","last_change":"2024-09-21T12:34:34.335260+0000","last_active":"2024-09-21T12:36:22.171530+0000","last_peered":"2024-09-21T12:36:22.171530+0000","last_clean":"2024-09-21T12:34:29.288868+0000","last_became_active":"2024-09-21T12:34:34.335260+0000","last_became_peered":"2024-09-21T12:34:34.335260+0000","last_unstale":"2024-09-21T12:36:22.171530+0000","last_undegraded":"2024-09-21T12:36:22.171530+0000","last_fullsized":"2024-09-21T12:34:34.331914+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:37.850100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:37.850100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:41.003975+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"423'3244","reported_seq":3288,"reported_epoch":423,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:21.556903+0000","last_change":"2024-09-21T12:35:53.515408+0000","last_active":"2024-09-21T12:36:21.556903+0000","last_peered":"2024-09-21T12:36:21.556903+0000","last_clean":"2024-09-21T12:35:46.767214+0000","last_became_active":"2024-09-21T12:35:47.791289+0000","last_became_peered":"2024-09-21T12:35:47.791289+0000","last_unstale":"2024-09-21T12:36:21.556903+0000","last_undegraded":"2024-09-21T12:35:53.506389+0000","last_fullsized":"2024-09-21T12:35:47.788809+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3244,"log_dups_size":0,"ondisk_log_size":3244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:03.677425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212598784,"num_objects":3244,"num_object_clones":0,"num_object_copies":9732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3244,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3244,"num_write_kb":207616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":3244}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"76'79","reported_seq":455,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.172550+0000","last_change":"2024-09-21T12:32:52.986242+0000","last_active":"2024-09-21T12:36:22.172550+0000","last_peered":"2024-09-21T12:36:22.172550+0000","last_clean":"2024-09-21T12:36:22.172550+0000","last_became_active":"2024-09-21T12:29:05.835884+0000","last_became_peered":"2024-09-21T12:29:05.835884+0000","last_unstale":"2024-09-21T12:36:22.172550+0000","last_undegraded":"2024-09-21T12:36:22.172550+0000","last_fullsized":"2024-09-21T12:36:22.172550+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:52.986175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:52.986175+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":870,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.180012+0000","last_change":"2024-09-21T12:36:00.161900+0000","last_active":"2024-09-21T12:36:22.180012+0000","last_peered":"2024-09-21T12:36:22.180012+0000","last_clean":"2024-09-21T12:36:22.180012+0000","last_became_active":"2024-09-21T12:36:00.127312+0000","last_became_peered":"2024-09-21T12:36:00.127312+0000","last_unstale":"2024-09-21T12:36:22.180012+0000","last_undegraded":"2024-09-21T12:36:22.180012+0000","last_fullsized":"2024-09-21T12:36:22.180012+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'29","last_scrub_stamp":"2024-09-21T12:33:38.081452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:38.081452+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":765,"reported_epoch":423,"state":"active+undersized","last_fresh":"2024-09-21T12:36:19.168671+0000","last_change":"2024-09-21T12:34:34.337276+0000","last_active":"2024-09-21T12:36:19.168671+0000","last_peered":"2024-09-21T12:36:19.168671+0000","last_clean":"2024-09-21T12:34:29.286433+0000","last_became_active":"2024-09-21T12:34:34.337276+0000","last_became_peered":"2024-09-21T12:34:34.337276+0000","last_unstale":"2024-09-21T12:36:19.168671+0000","last_undegraded":"2024-09-21T12:36:19.168671+0000","last_fullsized":"2024-09-21T12:34:34.330247+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:38.250806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:38.250806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":359,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.172367+0000","last_change":"2024-09-21T12:36:00.924159+0000","last_active":"2024-09-21T12:36:22.172367+0000","last_peered":"2024-09-21T12:36:22.172367+0000","last_clean":"2024-09-21T12:36:22.172367+0000","last_became_active":"2024-09-21T12:29:05.825869+0000","last_became_peered":"2024-09-21T12:29:05.825869+0000","last_unstale":"2024-09-21T12:36:22.172367+0000","last_undegraded":"2024-09-21T12:36:22.172367+0000","last_fullsized":"2024-09-21T12:36:22.172367+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:00.924084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:00.924084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:28.472139+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":761,"reported_epoch":423,"state":"active+undersized","last_fresh":"2024-09-21T12:36:19.165547+0000","last_change":"2024-09-21T12:34:34.339352+0000","last_active":"2024-09-21T12:36:19.165547+0000","last_peered":"2024-09-21T12:36:19.165547+0000","last_clean":"2024-09-21T12:34:29.284095+0000","last_became_active":"2024-09-21T12:34:34.339352+0000","last_became_peered":"2024-09-21T12:34:34.339352+0000","last_unstale":"2024-09-21T12:36:19.165547+0000","last_undegraded":"2024-09-21T12:36:19.165547+0000","last_fullsized":"2024-09-21T12:34:34.329336+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:05.510620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:05.510620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"424'3250","reported_seq":3299,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.789757+0000","last_change":"2024-09-21T12:35:47.791371+0000","last_active":"2024-09-21T12:36:22.789757+0000","last_peered":"2024-09-21T12:36:22.789757+0000","last_clean":"2024-09-21T12:36:22.789757+0000","last_became_active":"2024-09-21T12:35:47.791120+0000","last_became_peered":"2024-09-21T12:35:47.791120+0000","last_unstale":"2024-09-21T12:36:22.789757+0000","last_undegraded":"2024-09-21T12:36:22.789757+0000","last_fullsized":"2024-09-21T12:36:22.789757+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3250,"log_dups_size":0,"ondisk_log_size":3250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:02.202057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212992000,"num_objects":3250,"num_object_clones":0,"num_object_copies":9750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3250,"num_write_kb":208000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"76'79","reported_seq":567,"reported_epoch":424,"state":"active+undersized","last_fresh":"2024-09-21T12:36:22.179460+0000","last_change":"2024-09-21T12:34:34.337922+0000","last_active":"2024-09-21T12:36:22.179460+0000","last_peered":"2024-09-21T12:36:22.179460+0000","last_clean":"2024-09-21T12:34:29.289671+0000","last_became_active":"2024-09-21T12:34:34.337922+0000","last_became_peered":"2024-09-21T12:34:34.337922+0000","last_unstale":"2024-09-21T12:36:22.179460+0000","last_undegraded":"2024-09-21T12:36:22.179460+0000","last_fullsized":"2024-09-21T12:34:34.332211+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:34.055720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:34.055720+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"389'41","reported_seq":1022,"reported_epoch":423,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:19.168389+0000","last_change":"2024-09-21T12:34:34.339316+0000","last_active":"2024-09-21T12:36:19.168389+0000","last_peered":"2024-09-21T12:36:19.168389+0000","last_clean":"2024-09-21T12:34:30.529659+0000","last_became_active":"2024-09-21T12:34:34.339316+0000","last_became_peered":"2024-09-21T12:34:34.339316+0000","last_unstale":"2024-09-21T12:36:19.168389+0000","last_undegraded":"2024-09-21T12:34:34.330286+0000","last_fullsized":"2024-09-21T12:34:34.330177+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'40","last_scrub_stamp":"2024-09-21T12:32:25.139435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:25.139435+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:39.844521+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":167,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.159098+0000","last_change":"2024-09-21T12:35:09.182021+0000","last_active":"2024-09-21T12:36:19.159098+0000","last_peered":"2024-09-21T12:36:19.159098+0000","last_clean":"2024-09-21T12:36:19.159098+0000","last_became_active":"2024-09-21T12:32:38.426309+0000","last_became_peered":"2024-09-21T12:32:38.426309+0000","last_unstale":"2024-09-21T12:36:19.159098+0000","last_undegraded":"2024-09-21T12:36:19.159098+0000","last_fullsized":"2024-09-21T12:36:19.159098+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:09.181952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:09.181952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":495,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.166757+0000","last_change":"2024-09-21T12:35:27.114524+0000","last_active":"2024-09-21T12:36:19.166757+0000","last_peered":"2024-09-21T12:36:19.166757+0000","last_clean":"2024-09-21T12:36:19.166757+0000","last_became_active":"2024-09-21T12:26:54.083584+0000","last_became_peered":"2024-09-21T12:26:54.083584+0000","last_unstale":"2024-09-21T12:36:19.166757+0000","last_undegraded":"2024-09-21T12:36:19.166757+0000","last_fullsized":"2024-09-21T12:36:19.166757+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T12:35:27.114442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:35:27.114442+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:35.507665+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":809,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.181929+0000","last_change":"2024-09-21T12:33:00.087826+0000","last_active":"2024-09-21T12:36:22.181929+0000","last_peered":"2024-09-21T12:36:22.181929+0000","last_clean":"2024-09-21T12:36:22.181929+0000","last_became_active":"2024-09-21T12:32:38.424402+0000","last_became_peered":"2024-09-21T12:32:38.424402+0000","last_unstale":"2024-09-21T12:36:22.181929+0000","last_undegraded":"2024-09-21T12:36:22.181929+0000","last_fullsized":"2024-09-21T12:36:22.181929+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:00.087746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:33:00.087746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:10.889168+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":808,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.178599+0000","last_change":"2024-09-21T12:36:00.144733+0000","last_active":"2024-09-21T12:36:22.178599+0000","last_peered":"2024-09-21T12:36:22.178599+0000","last_clean":"2024-09-21T12:36:22.178599+0000","last_became_active":"2024-09-21T12:36:00.144526+0000","last_became_peered":"2024-09-21T12:36:00.144526+0000","last_unstale":"2024-09-21T12:36:22.178599+0000","last_undegraded":"2024-09-21T12:36:22.178599+0000","last_fullsized":"2024-09-21T12:36:22.178599+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:59.899499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:59.899499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"424'3317","reported_seq":3366,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.730982+0000","last_change":"2024-09-21T12:35:47.848839+0000","last_active":"2024-09-21T12:36:22.730982+0000","last_peered":"2024-09-21T12:36:22.730982+0000","last_clean":"2024-09-21T12:36:22.730982+0000","last_became_active":"2024-09-21T12:35:47.848543+0000","last_became_peered":"2024-09-21T12:35:47.848543+0000","last_unstale":"2024-09-21T12:36:22.730982+0000","last_undegraded":"2024-09-21T12:36:22.730982+0000","last_fullsized":"2024-09-21T12:36:22.730982+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3317,"log_dups_size":0,"ondisk_log_size":3317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:50.416027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217382912,"num_objects":3317,"num_object_clones":0,"num_object_copies":9951,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3317,"num_write_kb":212288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'79","reported_seq":191,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.170628+0000","last_change":"2024-09-21T12:36:01.132046+0000","last_active":"2024-09-21T12:36:19.170628+0000","last_peered":"2024-09-21T12:36:19.170628+0000","last_clean":"2024-09-21T12:36:19.170628+0000","last_became_active":"2024-09-21T12:36:01.131622+0000","last_became_peered":"2024-09-21T12:36:01.131622+0000","last_unstale":"2024-09-21T12:36:19.170628+0000","last_undegraded":"2024-09-21T12:36:19.170628+0000","last_fullsized":"2024-09-21T12:36:19.170628+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:30:39.179014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:30:39.179014+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":272,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.167201+0000","last_change":"2024-09-21T12:35:18.061854+0000","last_active":"2024-09-21T12:36:19.167201+0000","last_peered":"2024-09-21T12:36:19.167201+0000","last_clean":"2024-09-21T12:36:19.167201+0000","last_became_active":"2024-09-21T12:31:39.057316+0000","last_became_peered":"2024-09-21T12:31:39.057316+0000","last_unstale":"2024-09-21T12:36:19.167201+0000","last_undegraded":"2024-09-21T12:36:19.167201+0000","last_fullsized":"2024-09-21T12:36:19.167201+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'17","last_scrub_stamp":"2024-09-21T12:35:18.061753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:18.061753+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:20.057668+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":797,"reported_epoch":423,"state":"active+undersized","last_fresh":"2024-09-21T12:36:19.165288+0000","last_change":"2024-09-21T12:34:34.343641+0000","last_active":"2024-09-21T12:36:19.165288+0000","last_peered":"2024-09-21T12:36:19.165288+0000","last_clean":"2024-09-21T12:34:29.286935+0000","last_became_active":"2024-09-21T12:34:34.343641+0000","last_became_peered":"2024-09-21T12:34:34.343641+0000","last_unstale":"2024-09-21T12:36:19.165288+0000","last_undegraded":"2024-09-21T12:36:19.165288+0000","last_fullsized":"2024-09-21T12:34:34.329247+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:00.211563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:34:00.211563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":108,"reported_epoch":424,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:22.182516+0000","last_change":"2024-09-21T12:34:34.338167+0000","last_active":"2024-09-21T12:36:22.182516+0000","last_peered":"2024-09-21T12:36:22.182516+0000","last_clean":"2024-09-21T12:34:29.291114+0000","last_became_active":"2024-09-21T12:34:34.338167+0000","last_became_peered":"2024-09-21T12:34:34.338167+0000","last_unstale":"2024-09-21T12:36:22.182516+0000","last_undegraded":"2024-09-21T12:34:34.335226+0000","last_fullsized":"2024-09-21T12:34:34.335117+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:32:20.845428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:32:20.845428+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:19.597743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":105,"reported_epoch":423,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:19.174166+0000","last_change":"2024-09-21T12:36:01.125293+0000","last_active":"2024-09-21T12:36:19.174166+0000","last_peered":"2024-09-21T12:36:19.174166+0000","last_clean":"2024-09-21T12:36:19.174166+0000","last_became_active":"2024-09-21T12:36:01.124868+0000","last_became_peered":"2024-09-21T12:36:01.124868+0000","last_unstale":"2024-09-21T12:36:19.174166+0000","last_undegraded":"2024-09-21T12:36:19.174166+0000","last_fullsized":"2024-09-21T12:36:19.174166+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:02.295255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:02.295255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7,4],"avail_no_missing":["5","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":750,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.171135+0000","last_change":"2024-09-21T12:36:01.139453+0000","last_active":"2024-09-21T12:36:19.171135+0000","last_peered":"2024-09-21T12:36:19.171135+0000","last_clean":"2024-09-21T12:36:19.171135+0000","last_became_active":"2024-09-21T12:36:01.139034+0000","last_became_peered":"2024-09-21T12:36:01.139034+0000","last_unstale":"2024-09-21T12:36:19.171135+0000","last_undegraded":"2024-09-21T12:36:19.171135+0000","last_fullsized":"2024-09-21T12:36:19.171135+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:07.541665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:07.541665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"424'3290","reported_seq":3339,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.790549+0000","last_change":"2024-09-21T12:35:47.791375+0000","last_active":"2024-09-21T12:36:22.790549+0000","last_peered":"2024-09-21T12:36:22.790549+0000","last_clean":"2024-09-21T12:36:22.790549+0000","last_became_active":"2024-09-21T12:35:47.791073+0000","last_became_peered":"2024-09-21T12:35:47.791073+0000","last_unstale":"2024-09-21T12:36:22.790549+0000","last_undegraded":"2024-09-21T12:36:22.790549+0000","last_fullsized":"2024-09-21T12:36:22.790549+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3290,"log_dups_size":0,"ondisk_log_size":3290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:52.928312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215613440,"num_objects":3290,"num_object_clones":0,"num_object_copies":9870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3290,"num_write_kb":210560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"76'79","reported_seq":726,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.172237+0000","last_change":"2024-09-21T12:32:58.503527+0000","last_active":"2024-09-21T12:36:19.172237+0000","last_peered":"2024-09-21T12:36:19.172237+0000","last_clean":"2024-09-21T12:36:19.172237+0000","last_became_active":"2024-09-21T12:32:38.424710+0000","last_became_peered":"2024-09-21T12:32:38.424710+0000","last_unstale":"2024-09-21T12:36:19.172237+0000","last_undegraded":"2024-09-21T12:36:19.172237+0000","last_fullsized":"2024-09-21T12:36:19.172237+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:58.503445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:58.503445+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:40.894362+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":809,"reported_epoch":424,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:22.181164+0000","last_change":"2024-09-21T12:34:34.334240+0000","last_active":"2024-09-21T12:36:22.181164+0000","last_peered":"2024-09-21T12:36:22.181164+0000","last_clean":"2024-09-21T12:34:29.290312+0000","last_became_active":"2024-09-21T12:34:34.334240+0000","last_became_peered":"2024-09-21T12:34:34.334240+0000","last_unstale":"2024-09-21T12:36:22.181164+0000","last_undegraded":"2024-09-21T12:34:34.331492+0000","last_fullsized":"2024-09-21T12:34:34.331281+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:28:45.782955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.782955+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:47.864929+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":823,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.172511+0000","last_change":"2024-09-21T12:32:53.934634+0000","last_active":"2024-09-21T12:36:22.172511+0000","last_peered":"2024-09-21T12:36:22.172511+0000","last_clean":"2024-09-21T12:36:22.172511+0000","last_became_active":"2024-09-21T12:32:38.411004+0000","last_became_peered":"2024-09-21T12:32:38.411004+0000","last_unstale":"2024-09-21T12:36:22.172511+0000","last_undegraded":"2024-09-21T12:36:22.172511+0000","last_fullsized":"2024-09-21T12:36:22.172511+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:53.934583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:53.934583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:17.842220+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":54,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:20.181168+0000","last_change":"2024-09-21T12:36:00.155598+0000","last_active":"2024-09-21T12:36:20.181168+0000","last_peered":"2024-09-21T12:36:20.181168+0000","last_clean":"2024-09-21T12:36:20.181168+0000","last_became_active":"2024-09-21T12:36:00.144053+0000","last_became_peered":"2024-09-21T12:36:00.144053+0000","last_unstale":"2024-09-21T12:36:20.181168+0000","last_undegraded":"2024-09-21T12:36:20.181168+0000","last_fullsized":"2024-09-21T12:36:20.181168+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-21T12:32:58.369821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:32:58.369821+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":208,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.174778+0000","last_change":"2024-09-21T12:32:02.030536+0000","last_active":"2024-09-21T12:36:19.174778+0000","last_peered":"2024-09-21T12:36:19.174778+0000","last_clean":"2024-09-21T12:36:19.174778+0000","last_became_active":"2024-09-21T12:32:02.030240+0000","last_became_peered":"2024-09-21T12:32:02.030240+0000","last_unstale":"2024-09-21T12:36:19.174778+0000","last_undegraded":"2024-09-21T12:36:19.174778+0000","last_fullsized":"2024-09-21T12:36:19.174778+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:11.422293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:26:11.422293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:34.940322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":260,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.167906+0000","last_change":"2024-09-21T12:35:22.978507+0000","last_active":"2024-09-21T12:36:19.167906+0000","last_peered":"2024-09-21T12:36:19.167906+0000","last_clean":"2024-09-21T12:36:19.167906+0000","last_became_active":"2024-09-21T12:31:39.060160+0000","last_became_peered":"2024-09-21T12:31:39.060160+0000","last_unstale":"2024-09-21T12:36:19.167906+0000","last_undegraded":"2024-09-21T12:36:19.167906+0000","last_fullsized":"2024-09-21T12:36:19.167906+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:22.978438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:22.978438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:39.902333+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"424'3286","reported_seq":3335,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.777456+0000","last_change":"2024-09-21T12:35:47.849177+0000","last_active":"2024-09-21T12:36:22.777456+0000","last_peered":"2024-09-21T12:36:22.777456+0000","last_clean":"2024-09-21T12:36:22.777456+0000","last_became_active":"2024-09-21T12:35:47.848768+0000","last_became_peered":"2024-09-21T12:35:47.848768+0000","last_unstale":"2024-09-21T12:36:22.777456+0000","last_undegraded":"2024-09-21T12:36:22.777456+0000","last_fullsized":"2024-09-21T12:36:22.777456+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3286,"log_dups_size":0,"ondisk_log_size":3286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:56.898376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215351296,"num_objects":3286,"num_object_clones":0,"num_object_copies":9858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3286,"num_write_kb":210304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"415'35","reported_seq":280,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.160050+0000","last_change":"2024-09-21T12:36:00.163452+0000","last_active":"2024-09-21T12:36:19.160050+0000","last_peered":"2024-09-21T12:36:19.160050+0000","last_clean":"2024-09-21T12:36:19.160050+0000","last_became_active":"2024-09-21T12:36:00.139342+0000","last_became_peered":"2024-09-21T12:36:00.139342+0000","last_unstale":"2024-09-21T12:36:19.160050+0000","last_undegraded":"2024-09-21T12:36:19.160050+0000","last_fullsized":"2024-09-21T12:36:19.160050+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'33","last_scrub_stamp":"2024-09-21T12:33:25.426799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:25.426799+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":784,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.181695+0000","last_change":"2024-09-21T12:34:01.815496+0000","last_active":"2024-09-21T12:36:22.181695+0000","last_peered":"2024-09-21T12:36:22.181695+0000","last_clean":"2024-09-21T12:36:22.181695+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:36:22.181695+0000","last_undegraded":"2024-09-21T12:36:22.181695+0000","last_fullsized":"2024-09-21T12:36:22.181695+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:34:01.815445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:34:01.815445+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":28,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.171678+0000","last_change":"2024-09-21T12:36:00.144564+0000","last_active":"2024-09-21T12:36:19.171678+0000","last_peered":"2024-09-21T12:36:19.171678+0000","last_clean":"2024-09-21T12:36:19.171678+0000","last_became_active":"2024-09-21T12:36:00.134878+0000","last_became_peered":"2024-09-21T12:36:00.134878+0000","last_unstale":"2024-09-21T12:36:19.171678+0000","last_undegraded":"2024-09-21T12:36:19.171678+0000","last_fullsized":"2024-09-21T12:36:19.171678+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:35:24.687711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:35:24.687711+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:37:11.734028+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26834,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":9,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26834,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":188,"reported_epoch":423,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:19.170273+0000","last_change":"2024-09-21T12:36:01.130402+0000","last_active":"2024-09-21T12:36:19.170273+0000","last_peered":"2024-09-21T12:36:19.170273+0000","last_clean":"2024-09-21T12:36:19.170273+0000","last_became_active":"2024-09-21T12:36:01.129957+0000","last_became_peered":"2024-09-21T12:36:01.129957+0000","last_unstale":"2024-09-21T12:36:19.170273+0000","last_undegraded":"2024-09-21T12:36:19.170273+0000","last_fullsized":"2024-09-21T12:36:19.170273+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:36.684459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:36.684459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5,0],"avail_no_missing":["7","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":626,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.177665+0000","last_change":"2024-09-21T12:36:00.653180+0000","last_active":"2024-09-21T12:36:22.177665+0000","last_peered":"2024-09-21T12:36:22.177665+0000","last_clean":"2024-09-21T12:36:22.177665+0000","last_became_active":"2024-09-21T12:32:38.409187+0000","last_became_peered":"2024-09-21T12:32:38.409187+0000","last_unstale":"2024-09-21T12:36:22.177665+0000","last_undegraded":"2024-09-21T12:36:22.177665+0000","last_fullsized":"2024-09-21T12:36:22.177665+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:00.653130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:00.653130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:10.867677+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1","version":"424'3310","reported_seq":3359,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.785300+0000","last_change":"2024-09-21T12:35:47.788808+0000","last_active":"2024-09-21T12:36:22.785300+0000","last_peered":"2024-09-21T12:36:22.785300+0000","last_clean":"2024-09-21T12:36:22.785300+0000","last_became_active":"2024-09-21T12:35:47.788318+0000","last_became_peered":"2024-09-21T12:35:47.788318+0000","last_unstale":"2024-09-21T12:36:22.785300+0000","last_undegraded":"2024-09-21T12:36:22.785300+0000","last_fullsized":"2024-09-21T12:36:22.785300+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3310,"log_dups_size":0,"ondisk_log_size":3310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:02.330107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216924160,"num_objects":3310,"num_object_clones":0,"num_object_copies":9930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3310,"num_write_kb":211840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"76'79","reported_seq":723,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.172473+0000","last_change":"2024-09-21T12:33:04.508642+0000","last_active":"2024-09-21T12:36:19.172473+0000","last_peered":"2024-09-21T12:36:19.172473+0000","last_clean":"2024-09-21T12:36:19.172473+0000","last_became_active":"2024-09-21T12:27:14.954351+0000","last_became_peered":"2024-09-21T12:27:14.954351+0000","last_unstale":"2024-09-21T12:36:19.172473+0000","last_undegraded":"2024-09-21T12:36:19.172473+0000","last_fullsized":"2024-09-21T12:36:19.172473+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:04.508319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:04.508319+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":179,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.160021+0000","last_change":"2024-09-21T12:33:35.410866+0000","last_active":"2024-09-21T12:36:19.160021+0000","last_peered":"2024-09-21T12:36:19.160021+0000","last_clean":"2024-09-21T12:36:19.160021+0000","last_became_active":"2024-09-21T12:32:38.419714+0000","last_became_peered":"2024-09-21T12:32:38.419714+0000","last_unstale":"2024-09-21T12:36:19.160021+0000","last_undegraded":"2024-09-21T12:36:19.160021+0000","last_fullsized":"2024-09-21T12:36:19.160021+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:33:35.410528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:35.410528+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":183,"reported_epoch":423,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:19.158364+0000","last_change":"2024-09-21T12:36:01.131317+0000","last_active":"2024-09-21T12:36:19.158364+0000","last_peered":"2024-09-21T12:36:19.158364+0000","last_clean":"2024-09-21T12:36:19.158364+0000","last_became_active":"2024-09-21T12:36:01.130874+0000","last_became_peered":"2024-09-21T12:36:01.130874+0000","last_unstale":"2024-09-21T12:36:19.158364+0000","last_undegraded":"2024-09-21T12:36:19.158364+0000","last_fullsized":"2024-09-21T12:36:19.158364+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:01.342864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:01.342864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7,1],"avail_no_missing":["4","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":167,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.160250+0000","last_change":"2024-09-21T12:33:59.196744+0000","last_active":"2024-09-21T12:36:19.160250+0000","last_peered":"2024-09-21T12:36:19.160250+0000","last_clean":"2024-09-21T12:36:19.160250+0000","last_became_active":"2024-09-21T12:32:38.425683+0000","last_became_peered":"2024-09-21T12:32:38.425683+0000","last_unstale":"2024-09-21T12:36:19.160250+0000","last_undegraded":"2024-09-21T12:36:19.160250+0000","last_fullsized":"2024-09-21T12:36:19.160250+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:59.196667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:33:59.196667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":766,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.179724+0000","last_change":"2024-09-21T12:35:24.786713+0000","last_active":"2024-09-21T12:36:22.179724+0000","last_peered":"2024-09-21T12:36:22.179724+0000","last_clean":"2024-09-21T12:36:22.179724+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:36:22.179724+0000","last_undegraded":"2024-09-21T12:36:22.179724+0000","last_fullsized":"2024-09-21T12:36:22.179724+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:24.786609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:24.786609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:26.570879+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"424'3274","reported_seq":3320,"reported_epoch":424,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:22.801454+0000","last_change":"2024-09-21T12:35:53.505439+0000","last_active":"2024-09-21T12:36:22.801454+0000","last_peered":"2024-09-21T12:36:22.801454+0000","last_clean":"2024-09-21T12:35:46.767214+0000","last_became_active":"2024-09-21T12:35:47.787239+0000","last_became_peered":"2024-09-21T12:35:47.787239+0000","last_unstale":"2024-09-21T12:36:22.801454+0000","last_undegraded":"2024-09-21T12:35:53.504658+0000","last_fullsized":"2024-09-21T12:35:47.783070+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3274,"log_dups_size":0,"ondisk_log_size":3274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:16.320842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214564864,"num_objects":3274,"num_object_clones":0,"num_object_copies":9822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3274,"num_write_kb":209536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":3274}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"76'79","reported_seq":105,"reported_epoch":424,"state":"active+undersized","last_fresh":"2024-09-21T12:36:22.180510+0000","last_change":"2024-09-21T12:34:34.338192+0000","last_active":"2024-09-21T12:36:22.180510+0000","last_peered":"2024-09-21T12:36:22.180510+0000","last_clean":"2024-09-21T12:34:29.289953+0000","last_became_active":"2024-09-21T12:34:34.338192+0000","last_became_peered":"2024-09-21T12:34:34.338192+0000","last_unstale":"2024-09-21T12:36:22.180510+0000","last_undegraded":"2024-09-21T12:36:22.180510+0000","last_fullsized":"2024-09-21T12:34:34.334873+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:17.035216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:17.035216+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":189,"reported_epoch":423,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:19.159559+0000","last_change":"2024-09-21T12:34:34.341886+0000","last_active":"2024-09-21T12:36:19.159559+0000","last_peered":"2024-09-21T12:36:19.159559+0000","last_clean":"2024-09-21T12:34:29.279382+0000","last_became_active":"2024-09-21T12:34:34.341886+0000","last_became_peered":"2024-09-21T12:34:34.341886+0000","last_unstale":"2024-09-21T12:36:19.159559+0000","last_undegraded":"2024-09-21T12:34:34.333296+0000","last_fullsized":"2024-09-21T12:34:34.332270+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'31","last_scrub_stamp":"2024-09-21T12:33:08.360012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:08.360012+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:21.068003+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":175,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.158619+0000","last_change":"2024-09-21T12:35:18.088890+0000","last_active":"2024-09-21T12:36:19.158619+0000","last_peered":"2024-09-21T12:36:19.158619+0000","last_clean":"2024-09-21T12:36:19.158619+0000","last_became_active":"2024-09-21T12:32:38.424451+0000","last_became_peered":"2024-09-21T12:32:38.424451+0000","last_unstale":"2024-09-21T12:36:19.158619+0000","last_undegraded":"2024-09-21T12:36:19.158619+0000","last_fullsized":"2024-09-21T12:36:19.158619+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:18.088831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:18.088831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:32.381939+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":789,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.167989+0000","last_change":"2024-09-21T12:34:46.278390+0000","last_active":"2024-09-21T12:36:19.167989+0000","last_peered":"2024-09-21T12:36:19.167989+0000","last_clean":"2024-09-21T12:36:19.167989+0000","last_became_active":"2024-09-21T12:32:38.425177+0000","last_became_peered":"2024-09-21T12:32:38.425177+0000","last_unstale":"2024-09-21T12:36:19.167989+0000","last_undegraded":"2024-09-21T12:36:19.167989+0000","last_fullsized":"2024-09-21T12:36:19.167989+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:46.278330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:46.278330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":715,"reported_epoch":424,"state":"active+undersized","last_fresh":"2024-09-21T12:36:22.177484+0000","last_change":"2024-09-21T12:34:34.345710+0000","last_active":"2024-09-21T12:36:22.177484+0000","last_peered":"2024-09-21T12:36:22.177484+0000","last_clean":"2024-09-21T12:34:29.294810+0000","last_became_active":"2024-09-21T12:34:34.345710+0000","last_became_peered":"2024-09-21T12:34:34.345710+0000","last_unstale":"2024-09-21T12:36:22.177484+0000","last_undegraded":"2024-09-21T12:36:22.177484+0000","last_fullsized":"2024-09-21T12:34:34.334512+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:20.005906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:34:20.005906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3","version":"423'3162","reported_seq":5329,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:21.899525+0000","last_change":"2024-09-21T12:36:13.109138+0000","last_active":"2024-09-21T12:36:21.899525+0000","last_peered":"2024-09-21T12:36:21.899525+0000","last_clean":"2024-09-21T12:36:21.899525+0000","last_became_active":"2024-09-21T12:36:13.105381+0000","last_became_peered":"2024-09-21T12:36:13.105381+0000","last_unstale":"2024-09-21T12:36:21.899525+0000","last_undegraded":"2024-09-21T12:36:21.899525+0000","last_fullsized":"2024-09-21T12:36:21.899525+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3162,"log_dups_size":0,"ondisk_log_size":3162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:57.457946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207224832,"num_objects":3162,"num_object_clones":0,"num_object_copies":9486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3162,"num_write_kb":202368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1051,"num_bytes_recovered":68878336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"76'79","reported_seq":840,"reported_epoch":423,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:19.170375+0000","last_change":"2024-09-21T12:36:01.128488+0000","last_active":"2024-09-21T12:36:19.170375+0000","last_peered":"2024-09-21T12:36:19.170375+0000","last_clean":"2024-09-21T12:36:19.170375+0000","last_became_active":"2024-09-21T12:36:01.127851+0000","last_became_peered":"2024-09-21T12:36:01.127851+0000","last_unstale":"2024-09-21T12:36:19.170375+0000","last_undegraded":"2024-09-21T12:36:19.170375+0000","last_fullsized":"2024-09-21T12:36:19.170375+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:05.500994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:05.500994+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,5],"avail_no_missing":["7","3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":175,"reported_epoch":423,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:19.159308+0000","last_change":"2024-09-21T12:34:34.347138+0000","last_active":"2024-09-21T12:36:19.159308+0000","last_peered":"2024-09-21T12:36:19.159308+0000","last_clean":"2024-09-21T12:34:29.280929+0000","last_became_active":"2024-09-21T12:34:34.347138+0000","last_became_peered":"2024-09-21T12:34:34.347138+0000","last_unstale":"2024-09-21T12:36:19.159308+0000","last_undegraded":"2024-09-21T12:34:34.332331+0000","last_fullsized":"2024-09-21T12:34:34.330784+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:32:26.690403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:26.690403+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":798,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.180624+0000","last_change":"2024-09-21T12:32:38.404305+0000","last_active":"2024-09-21T12:36:22.180624+0000","last_peered":"2024-09-21T12:36:22.180624+0000","last_clean":"2024-09-21T12:36:22.180624+0000","last_became_active":"2024-09-21T12:32:38.403997+0000","last_became_peered":"2024-09-21T12:32:38.403997+0000","last_unstale":"2024-09-21T12:36:22.180624+0000","last_undegraded":"2024-09-21T12:36:22.180624+0000","last_fullsized":"2024-09-21T12:36:22.180624+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:47.248389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:31:47.248389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:18.590240+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"416'14","reported_seq":138,"reported_epoch":423,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:19.171322+0000","last_change":"2024-09-21T12:36:01.131051+0000","last_active":"2024-09-21T12:36:19.171322+0000","last_peered":"2024-09-21T12:36:19.171322+0000","last_clean":"2024-09-21T12:36:19.171322+0000","last_became_active":"2024-09-21T12:36:01.130569+0000","last_became_peered":"2024-09-21T12:36:01.130569+0000","last_unstale":"2024-09-21T12:36:19.171322+0000","last_undegraded":"2024-09-21T12:36:19.171322+0000","last_fullsized":"2024-09-21T12:36:19.171322+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:33:34.270590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:33:34.270590+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5,0],"avail_no_missing":["7","0","5"],"object_location_counts":[{"shards":"0,5,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":171,"reported_epoch":423,"state":"active+undersized","last_fresh":"2024-09-21T12:36:19.159832+0000","last_change":"2024-09-21T12:34:34.348303+0000","last_active":"2024-09-21T12:36:19.159832+0000","last_peered":"2024-09-21T12:36:19.159832+0000","last_clean":"2024-09-21T12:34:29.279670+0000","last_became_active":"2024-09-21T12:34:34.348303+0000","last_became_peered":"2024-09-21T12:34:34.348303+0000","last_unstale":"2024-09-21T12:36:19.159832+0000","last_undegraded":"2024-09-21T12:36:19.159832+0000","last_fullsized":"2024-09-21T12:34:34.331636+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:06.369436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:06.369436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"423'3158","reported_seq":3203,"reported_epoch":423,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:21.855857+0000","last_change":"2024-09-21T12:36:01.140837+0000","last_active":"2024-09-21T12:36:21.855857+0000","last_peered":"2024-09-21T12:36:21.855857+0000","last_clean":"2024-09-21T12:36:21.855857+0000","last_became_active":"2024-09-21T12:36:01.132703+0000","last_became_peered":"2024-09-21T12:36:01.132703+0000","last_unstale":"2024-09-21T12:36:21.855857+0000","last_undegraded":"2024-09-21T12:36:21.855857+0000","last_fullsized":"2024-09-21T12:36:21.855857+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3158,"log_dups_size":0,"ondisk_log_size":3158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:52.238296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206962688,"num_objects":3158,"num_object_clones":0,"num_object_copies":9474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3158,"num_objects_unfound":0,"num_objects_dirty":3158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3158,"num_write_kb":202112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6,1],"avail_no_missing":["7","1","6"],"object_location_counts":[{"shards":"1,6,7","objects":3158}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"76'79","reported_seq":179,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.160107+0000","last_change":"2024-09-21T12:36:01.159899+0000","last_active":"2024-09-21T12:36:19.160107+0000","last_peered":"2024-09-21T12:36:19.160107+0000","last_clean":"2024-09-21T12:36:19.160107+0000","last_became_active":"2024-09-21T12:36:01.159481+0000","last_became_peered":"2024-09-21T12:36:01.159481+0000","last_unstale":"2024-09-21T12:36:19.160107+0000","last_undegraded":"2024-09-21T12:36:19.160107+0000","last_fullsized":"2024-09-21T12:36:19.160107+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:49.155069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:49.155069+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":886,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.182087+0000","last_change":"2024-09-21T12:35:11.198389+0000","last_active":"2024-09-21T12:36:22.182087+0000","last_peered":"2024-09-21T12:36:22.182087+0000","last_clean":"2024-09-21T12:36:22.182087+0000","last_became_active":"2024-09-21T12:32:38.424376+0000","last_became_peered":"2024-09-21T12:32:38.424376+0000","last_unstale":"2024-09-21T12:36:22.182087+0000","last_undegraded":"2024-09-21T12:36:22.182087+0000","last_fullsized":"2024-09-21T12:36:22.182087+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:35:11.197905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:11.197905+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":462,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.180314+0000","last_change":"2024-09-21T12:34:36.079650+0000","last_active":"2024-09-21T12:36:22.180314+0000","last_peered":"2024-09-21T12:36:22.180314+0000","last_clean":"2024-09-21T12:36:22.180314+0000","last_became_active":"2024-09-21T12:26:54.084228+0000","last_became_peered":"2024-09-21T12:26:54.084228+0000","last_unstale":"2024-09-21T12:36:22.180314+0000","last_undegraded":"2024-09-21T12:36:22.180314+0000","last_fullsized":"2024-09-21T12:36:22.180314+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:36.079583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:34:36.079583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":733,"reported_epoch":423,"state":"active+undersized","last_fresh":"2024-09-21T12:36:19.158759+0000","last_change":"2024-09-21T12:34:34.346822+0000","last_active":"2024-09-21T12:36:19.158759+0000","last_peered":"2024-09-21T12:36:19.158759+0000","last_clean":"2024-09-21T12:34:29.278339+0000","last_became_active":"2024-09-21T12:34:34.346822+0000","last_became_peered":"2024-09-21T12:34:34.346822+0000","last_unstale":"2024-09-21T12:36:19.158759+0000","last_undegraded":"2024-09-21T12:36:19.158759+0000","last_fullsized":"2024-09-21T12:34:34.330172+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:47.361115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:47.361115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:10.874140+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":777,"reported_epoch":424,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:22.172591+0000","last_change":"2024-09-21T12:36:01.130956+0000","last_active":"2024-09-21T12:36:22.172591+0000","last_peered":"2024-09-21T12:36:22.172591+0000","last_clean":"2024-09-21T12:36:22.172591+0000","last_became_active":"2024-09-21T12:36:01.130370+0000","last_became_peered":"2024-09-21T12:36:01.130370+0000","last_unstale":"2024-09-21T12:36:22.172591+0000","last_undegraded":"2024-09-21T12:36:22.172591+0000","last_fullsized":"2024-09-21T12:36:22.172591+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:37.119815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:37.119815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7,3],"avail_no_missing":["0","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"423'3125","reported_seq":5275,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:21.846535+0000","last_change":"2024-09-21T12:36:12.108511+0000","last_active":"2024-09-21T12:36:21.846535+0000","last_peered":"2024-09-21T12:36:21.846535+0000","last_clean":"2024-09-21T12:36:21.846535+0000","last_became_active":"2024-09-21T12:36:12.104630+0000","last_became_peered":"2024-09-21T12:36:12.104630+0000","last_unstale":"2024-09-21T12:36:21.846535+0000","last_undegraded":"2024-09-21T12:36:21.846535+0000","last_fullsized":"2024-09-21T12:36:21.846535+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3125,"log_dups_size":0,"ondisk_log_size":3125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:06.896900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204800000,"num_objects":3125,"num_object_clones":0,"num_object_copies":9375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3125,"num_write_kb":200000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1047,"num_bytes_recovered":68616192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"76'79","reported_seq":707,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.174992+0000","last_change":"2024-09-21T12:35:57.172169+0000","last_active":"2024-09-21T12:36:19.174992+0000","last_peered":"2024-09-21T12:36:19.174992+0000","last_clean":"2024-09-21T12:36:19.174992+0000","last_became_active":"2024-09-21T12:32:38.414877+0000","last_became_peered":"2024-09-21T12:32:38.414877+0000","last_unstale":"2024-09-21T12:36:19.174992+0000","last_undegraded":"2024-09-21T12:36:19.174992+0000","last_fullsized":"2024-09-21T12:36:19.174992+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:35:57.172103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:35:57.172103+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:06.814669+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":944,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.173342+0000","last_change":"2024-09-21T12:36:00.174641+0000","last_active":"2024-09-21T12:36:19.173342+0000","last_peered":"2024-09-21T12:36:19.173342+0000","last_clean":"2024-09-21T12:36:19.173342+0000","last_became_active":"2024-09-21T12:36:00.148392+0000","last_became_peered":"2024-09-21T12:36:00.148392+0000","last_unstale":"2024-09-21T12:36:19.173342+0000","last_undegraded":"2024-09-21T12:36:19.173342+0000","last_fullsized":"2024-09-21T12:36:19.173342+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'35","last_scrub_stamp":"2024-09-21T12:34:46.956618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:34:46.956618+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:17.045549+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":28,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.170853+0000","last_change":"2024-09-21T12:36:00.145588+0000","last_active":"2024-09-21T12:36:19.170853+0000","last_peered":"2024-09-21T12:36:19.170853+0000","last_clean":"2024-09-21T12:36:19.170853+0000","last_became_active":"2024-09-21T12:36:00.145287+0000","last_became_peered":"2024-09-21T12:36:00.145287+0000","last_unstale":"2024-09-21T12:36:19.170853+0000","last_undegraded":"2024-09-21T12:36:19.170853+0000","last_fullsized":"2024-09-21T12:36:19.170853+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:58.197610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:58.197610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:14.038278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":673,"reported_epoch":424,"state":"active+undersized","last_fresh":"2024-09-21T12:36:22.180291+0000","last_change":"2024-09-21T12:34:34.343106+0000","last_active":"2024-09-21T12:36:22.180291+0000","last_peered":"2024-09-21T12:36:22.180291+0000","last_clean":"2024-09-21T12:34:29.291366+0000","last_became_active":"2024-09-21T12:34:34.343106+0000","last_became_peered":"2024-09-21T12:34:34.343106+0000","last_unstale":"2024-09-21T12:36:22.180291+0000","last_undegraded":"2024-09-21T12:36:22.180291+0000","last_fullsized":"2024-09-21T12:34:34.333287+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:20.243774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:20.243774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:35.146338+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":746,"reported_epoch":423,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:19.171736+0000","last_change":"2024-09-21T12:36:01.140977+0000","last_active":"2024-09-21T12:36:19.171736+0000","last_peered":"2024-09-21T12:36:19.171736+0000","last_clean":"2024-09-21T12:36:19.171736+0000","last_became_active":"2024-09-21T12:36:01.135641+0000","last_became_peered":"2024-09-21T12:36:01.135641+0000","last_unstale":"2024-09-21T12:36:19.171736+0000","last_undegraded":"2024-09-21T12:36:19.171736+0000","last_fullsized":"2024-09-21T12:36:19.171736+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:27.297558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:27.297558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,6],"avail_no_missing":["7","3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"424'3234","reported_seq":3280,"reported_epoch":424,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:23.426409+0000","last_change":"2024-09-21T12:35:53.511573+0000","last_active":"2024-09-21T12:36:23.426409+0000","last_peered":"2024-09-21T12:36:23.426409+0000","last_clean":"2024-09-21T12:35:46.767214+0000","last_became_active":"2024-09-21T12:35:47.793577+0000","last_became_peered":"2024-09-21T12:35:47.793577+0000","last_unstale":"2024-09-21T12:36:23.426409+0000","last_undegraded":"2024-09-21T12:35:53.504929+0000","last_fullsized":"2024-09-21T12:35:47.782699+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3234,"log_dups_size":0,"ondisk_log_size":3234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:01.401191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211943424,"num_objects":3234,"num_object_clones":0,"num_object_copies":9702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3234,"num_write_kb":206976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":3234}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"76'79","reported_seq":724,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.179994+0000","last_change":"2024-09-21T12:33:37.767771+0000","last_active":"2024-09-21T12:36:22.179994+0000","last_peered":"2024-09-21T12:36:22.179994+0000","last_clean":"2024-09-21T12:36:22.179994+0000","last_became_active":"2024-09-21T12:27:14.951375+0000","last_became_peered":"2024-09-21T12:27:14.951375+0000","last_unstale":"2024-09-21T12:36:22.179994+0000","last_undegraded":"2024-09-21T12:36:22.179994+0000","last_fullsized":"2024-09-21T12:36:22.179994+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:37.767584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:37.767584+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":179,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.158549+0000","last_change":"2024-09-21T12:35:12.197153+0000","last_active":"2024-09-21T12:36:19.158549+0000","last_peered":"2024-09-21T12:36:19.158549+0000","last_clean":"2024-09-21T12:36:19.158549+0000","last_became_active":"2024-09-21T12:32:38.421609+0000","last_became_peered":"2024-09-21T12:32:38.421609+0000","last_unstale":"2024-09-21T12:36:19.158549+0000","last_undegraded":"2024-09-21T12:36:19.158549+0000","last_fullsized":"2024-09-21T12:36:19.158549+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T12:35:12.197060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:12.197060+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:26.198288+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":28,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.170983+0000","last_change":"2024-09-21T12:36:00.151126+0000","last_active":"2024-09-21T12:36:19.170983+0000","last_peered":"2024-09-21T12:36:19.170983+0000","last_clean":"2024-09-21T12:36:19.170983+0000","last_became_active":"2024-09-21T12:36:00.150832+0000","last_became_peered":"2024-09-21T12:36:00.150832+0000","last_unstale":"2024-09-21T12:36:19.170983+0000","last_undegraded":"2024-09-21T12:36:19.170983+0000","last_fullsized":"2024-09-21T12:36:19.170983+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:19.611537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:19.611537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:20.075332+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":95,"reported_epoch":424,"state":"active+undersized","last_fresh":"2024-09-21T12:36:22.171268+0000","last_change":"2024-09-21T12:34:34.333741+0000","last_active":"2024-09-21T12:36:22.171268+0000","last_peered":"2024-09-21T12:36:22.171268+0000","last_clean":"2024-09-21T12:34:29.289730+0000","last_became_active":"2024-09-21T12:34:34.333741+0000","last_became_peered":"2024-09-21T12:34:34.333741+0000","last_unstale":"2024-09-21T12:36:22.171268+0000","last_undegraded":"2024-09-21T12:36:22.171268+0000","last_fullsized":"2024-09-21T12:34:34.330975+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:23.769748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:32:23.769748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:04.957773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":794,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.180414+0000","last_change":"2024-09-21T12:36:00.139665+0000","last_active":"2024-09-21T12:36:22.180414+0000","last_peered":"2024-09-21T12:36:22.180414+0000","last_clean":"2024-09-21T12:36:22.180414+0000","last_became_active":"2024-09-21T12:36:00.139300+0000","last_became_peered":"2024-09-21T12:36:00.139300+0000","last_unstale":"2024-09-21T12:36:22.180414+0000","last_undegraded":"2024-09-21T12:36:22.180414+0000","last_fullsized":"2024-09-21T12:36:22.180414+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:04.876656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:34:04.876656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"423'3148","reported_seq":3195,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:21.562912+0000","last_change":"2024-09-21T12:35:47.849125+0000","last_active":"2024-09-21T12:36:21.562912+0000","last_peered":"2024-09-21T12:36:21.562912+0000","last_clean":"2024-09-21T12:36:21.562912+0000","last_became_active":"2024-09-21T12:35:47.848708+0000","last_became_peered":"2024-09-21T12:35:47.848708+0000","last_unstale":"2024-09-21T12:36:21.562912+0000","last_undegraded":"2024-09-21T12:36:21.562912+0000","last_fullsized":"2024-09-21T12:36:21.562912+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3148,"log_dups_size":0,"ondisk_log_size":3148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:47.277063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206307328,"num_objects":3148,"num_object_clones":0,"num_object_copies":9444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3148,"num_write_kb":201472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"76'79","reported_seq":199,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.171097+0000","last_change":"2024-09-21T12:36:01.143284+0000","last_active":"2024-09-21T12:36:19.171097+0000","last_peered":"2024-09-21T12:36:19.171097+0000","last_clean":"2024-09-21T12:36:19.171097+0000","last_became_active":"2024-09-21T12:36:01.141807+0000","last_became_peered":"2024-09-21T12:36:01.141807+0000","last_unstale":"2024-09-21T12:36:19.171097+0000","last_undegraded":"2024-09-21T12:36:19.171097+0000","last_fullsized":"2024-09-21T12:36:19.171097+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:32.272927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:32.272927+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":309,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.173000+0000","last_change":"2024-09-21T12:36:00.171576+0000","last_active":"2024-09-21T12:36:19.173000+0000","last_peered":"2024-09-21T12:36:19.173000+0000","last_clean":"2024-09-21T12:36:19.173000+0000","last_became_active":"2024-09-21T12:36:00.137597+0000","last_became_peered":"2024-09-21T12:36:00.137597+0000","last_unstale":"2024-09-21T12:36:19.173000+0000","last_undegraded":"2024-09-21T12:36:19.173000+0000","last_fullsized":"2024-09-21T12:36:19.173000+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:35:52.226406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:52.226406+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:52.655311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":34,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.171008+0000","last_change":"2024-09-21T12:36:00.161052+0000","last_active":"2024-09-21T12:36:19.171008+0000","last_peered":"2024-09-21T12:36:19.171008+0000","last_clean":"2024-09-21T12:36:19.171008+0000","last_became_active":"2024-09-21T12:36:00.148899+0000","last_became_peered":"2024-09-21T12:36:00.148899+0000","last_unstale":"2024-09-21T12:36:19.171008+0000","last_undegraded":"2024-09-21T12:36:19.171008+0000","last_fullsized":"2024-09-21T12:36:19.171008+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-21T12:35:51.961868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:51.961868+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:54.987102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"415'16","reported_seq":37,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.170203+0000","last_change":"2024-09-21T12:36:00.165364+0000","last_active":"2024-09-21T12:36:19.170203+0000","last_peered":"2024-09-21T12:36:19.170203+0000","last_clean":"2024-09-21T12:36:19.170203+0000","last_became_active":"2024-09-21T12:36:00.143091+0000","last_became_peered":"2024-09-21T12:36:00.143091+0000","last_unstale":"2024-09-21T12:36:19.170203+0000","last_undegraded":"2024-09-21T12:36:19.170203+0000","last_fullsized":"2024-09-21T12:36:19.170203+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'14","last_scrub_stamp":"2024-09-21T12:35:06.542700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:35:06.542700+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:22.185120+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":468,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.173171+0000","last_change":"2024-09-21T12:35:07.828972+0000","last_active":"2024-09-21T12:36:19.173171+0000","last_peered":"2024-09-21T12:36:19.173171+0000","last_clean":"2024-09-21T12:36:19.173171+0000","last_became_active":"2024-09-21T12:26:54.089155+0000","last_became_peered":"2024-09-21T12:26:54.089155+0000","last_unstale":"2024-09-21T12:36:19.173171+0000","last_undegraded":"2024-09-21T12:36:19.173171+0000","last_fullsized":"2024-09-21T12:36:19.173171+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:07.828886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:07.828886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:33.632460+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"76'79","reported_seq":506,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.173539+0000","last_change":"2024-09-21T12:32:26.698667+0000","last_active":"2024-09-21T12:36:19.173539+0000","last_peered":"2024-09-21T12:36:19.173539+0000","last_clean":"2024-09-21T12:36:19.173539+0000","last_became_active":"2024-09-21T12:31:39.060422+0000","last_became_peered":"2024-09-21T12:31:39.060422+0000","last_unstale":"2024-09-21T12:36:19.173539+0000","last_undegraded":"2024-09-21T12:36:19.173539+0000","last_fullsized":"2024-09-21T12:36:19.173539+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:26.698622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:26.698622+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:01.158707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":822,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.173279+0000","last_change":"2024-09-21T12:35:59.183091+0000","last_active":"2024-09-21T12:36:19.173279+0000","last_peered":"2024-09-21T12:36:19.173279+0000","last_clean":"2024-09-21T12:36:19.173279+0000","last_became_active":"2024-09-21T12:32:02.026693+0000","last_became_peered":"2024-09-21T12:32:02.026693+0000","last_unstale":"2024-09-21T12:36:19.173279+0000","last_undegraded":"2024-09-21T12:36:19.173279+0000","last_fullsized":"2024-09-21T12:36:19.173279+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:59.183047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:59.183047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:03.346089+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":280,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.178111+0000","last_change":"2024-09-21T12:36:01.139003+0000","last_active":"2024-09-21T12:36:22.178111+0000","last_peered":"2024-09-21T12:36:22.178111+0000","last_clean":"2024-09-21T12:36:22.178111+0000","last_became_active":"2024-09-21T12:36:01.138626+0000","last_became_peered":"2024-09-21T12:36:01.138626+0000","last_unstale":"2024-09-21T12:36:22.178111+0000","last_undegraded":"2024-09-21T12:36:22.178111+0000","last_fullsized":"2024-09-21T12:36:22.178111+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T12:34:09.872320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:34:09.872320+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":462,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.178470+0000","last_change":"2024-09-21T12:33:34.706596+0000","last_active":"2024-09-21T12:36:22.178470+0000","last_peered":"2024-09-21T12:36:22.178470+0000","last_clean":"2024-09-21T12:36:22.178470+0000","last_became_active":"2024-09-21T12:26:54.084109+0000","last_became_peered":"2024-09-21T12:26:54.084109+0000","last_unstale":"2024-09-21T12:36:22.178470+0000","last_undegraded":"2024-09-21T12:36:22.178470+0000","last_fullsized":"2024-09-21T12:36:22.178470+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:34.706531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:33:34.706531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":28,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.170165+0000","last_change":"2024-09-21T12:36:00.149496+0000","last_active":"2024-09-21T12:36:19.170165+0000","last_peered":"2024-09-21T12:36:19.170165+0000","last_clean":"2024-09-21T12:36:19.170165+0000","last_became_active":"2024-09-21T12:36:00.149086+0000","last_became_peered":"2024-09-21T12:36:00.149086+0000","last_unstale":"2024-09-21T12:36:19.170165+0000","last_undegraded":"2024-09-21T12:36:19.170165+0000","last_fullsized":"2024-09-21T12:36:19.170165+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:14.566963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:14.566963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:34.794586+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'79","reported_seq":746,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.172333+0000","last_change":"2024-09-21T12:34:20.427215+0000","last_active":"2024-09-21T12:36:19.172333+0000","last_peered":"2024-09-21T12:36:19.172333+0000","last_clean":"2024-09-21T12:36:19.172333+0000","last_became_active":"2024-09-21T12:32:38.425495+0000","last_became_peered":"2024-09-21T12:32:38.425495+0000","last_unstale":"2024-09-21T12:36:19.172333+0000","last_undegraded":"2024-09-21T12:36:19.172333+0000","last_fullsized":"2024-09-21T12:36:19.172333+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:20.427138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:20.427138+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":848,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.181347+0000","last_change":"2024-09-21T12:36:00.142630+0000","last_active":"2024-09-21T12:36:22.181347+0000","last_peered":"2024-09-21T12:36:22.181347+0000","last_clean":"2024-09-21T12:36:22.181347+0000","last_became_active":"2024-09-21T12:36:00.142274+0000","last_became_peered":"2024-09-21T12:36:00.142274+0000","last_unstale":"2024-09-21T12:36:22.181347+0000","last_undegraded":"2024-09-21T12:36:22.181347+0000","last_fullsized":"2024-09-21T12:36:22.181347+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:32:19.786928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:19.786928+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:01.953764+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":28,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.170249+0000","last_change":"2024-09-21T12:36:00.140779+0000","last_active":"2024-09-21T12:36:19.170249+0000","last_peered":"2024-09-21T12:36:19.170249+0000","last_clean":"2024-09-21T12:36:19.170249+0000","last_became_active":"2024-09-21T12:36:00.140477+0000","last_became_peered":"2024-09-21T12:36:00.140477+0000","last_unstale":"2024-09-21T12:36:19.170249+0000","last_undegraded":"2024-09-21T12:36:19.170249+0000","last_fullsized":"2024-09-21T12:36:19.170249+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:27.798236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:27.798236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:51.374052+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":775,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.177827+0000","last_change":"2024-09-21T12:36:00.151444+0000","last_active":"2024-09-21T12:36:22.177827+0000","last_peered":"2024-09-21T12:36:22.177827+0000","last_clean":"2024-09-21T12:36:22.177827+0000","last_became_active":"2024-09-21T12:36:00.150975+0000","last_became_peered":"2024-09-21T12:36:00.150975+0000","last_unstale":"2024-09-21T12:36:22.177827+0000","last_undegraded":"2024-09-21T12:36:22.177827+0000","last_fullsized":"2024-09-21T12:36:22.177827+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:58.720834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:32:58.720834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:20.560140+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":749,"reported_epoch":423,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:19.159564+0000","last_change":"2024-09-21T12:36:01.138863+0000","last_active":"2024-09-21T12:36:19.159564+0000","last_peered":"2024-09-21T12:36:19.159564+0000","last_clean":"2024-09-21T12:36:19.159564+0000","last_became_active":"2024-09-21T12:36:01.138205+0000","last_became_peered":"2024-09-21T12:36:01.138205+0000","last_unstale":"2024-09-21T12:36:19.159564+0000","last_undegraded":"2024-09-21T12:36:19.159564+0000","last_fullsized":"2024-09-21T12:36:19.159564+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:55.936030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:55.936030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:09.145667+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7,0],"avail_no_missing":["4","0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.10","version":"76'79","reported_seq":36,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.171425+0000","last_change":"2024-09-21T12:36:00.829177+0000","last_active":"2024-09-21T12:36:19.171425+0000","last_peered":"2024-09-21T12:36:19.171425+0000","last_clean":"2024-09-21T12:36:19.171425+0000","last_became_active":"2024-09-21T12:36:00.140266+0000","last_became_peered":"2024-09-21T12:36:00.140266+0000","last_unstale":"2024-09-21T12:36:19.171425+0000","last_undegraded":"2024-09-21T12:36:19.171425+0000","last_fullsized":"2024-09-21T12:36:19.171425+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:00.829119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:00.829119+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:19.526550+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":527,"reported_epoch":423,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:19.171258+0000","last_change":"2024-09-21T12:36:01.130787+0000","last_active":"2024-09-21T12:36:19.171258+0000","last_peered":"2024-09-21T12:36:19.171258+0000","last_clean":"2024-09-21T12:36:19.171258+0000","last_became_active":"2024-09-21T12:36:01.129843+0000","last_became_peered":"2024-09-21T12:36:01.129843+0000","last_unstale":"2024-09-21T12:36:19.171258+0000","last_undegraded":"2024-09-21T12:36:19.171258+0000","last_fullsized":"2024-09-21T12:36:19.171258+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:35:54.238498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:54.238498+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:14.079285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5,6],"avail_no_missing":["7","5","6"],"object_location_counts":[{"shards":"5,6,7","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":831,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.172491+0000","last_change":"2024-09-21T12:36:00.127553+0000","last_active":"2024-09-21T12:36:22.172491+0000","last_peered":"2024-09-21T12:36:22.172491+0000","last_clean":"2024-09-21T12:36:22.172491+0000","last_became_active":"2024-09-21T12:36:00.127225+0000","last_became_peered":"2024-09-21T12:36:00.127225+0000","last_unstale":"2024-09-21T12:36:22.172491+0000","last_undegraded":"2024-09-21T12:36:22.172491+0000","last_fullsized":"2024-09-21T12:36:22.172491+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:22.225122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:22.225122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:41.794207+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"416'15","reported_seq":319,"reported_epoch":424,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:22.173853+0000","last_change":"2024-09-21T12:36:01.139488+0000","last_active":"2024-09-21T12:36:22.173853+0000","last_peered":"2024-09-21T12:36:22.173853+0000","last_clean":"2024-09-21T12:36:22.173853+0000","last_became_active":"2024-09-21T12:36:01.138017+0000","last_became_peered":"2024-09-21T12:36:01.138017+0000","last_unstale":"2024-09-21T12:36:22.173853+0000","last_undegraded":"2024-09-21T12:36:22.173853+0000","last_fullsized":"2024-09-21T12:36:22.173853+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'10","last_scrub_stamp":"2024-09-21T12:34:14.003545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:14.003545+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7,1],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":171,"reported_epoch":423,"state":"active+undersized","last_fresh":"2024-09-21T12:36:19.160069+0000","last_change":"2024-09-21T12:34:34.340859+0000","last_active":"2024-09-21T12:36:19.160069+0000","last_peered":"2024-09-21T12:36:19.160069+0000","last_clean":"2024-09-21T12:34:29.279088+0000","last_became_active":"2024-09-21T12:34:34.340859+0000","last_became_peered":"2024-09-21T12:34:34.340859+0000","last_unstale":"2024-09-21T12:36:19.160069+0000","last_undegraded":"2024-09-21T12:36:19.160069+0000","last_fullsized":"2024-09-21T12:34:34.330912+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:05.370263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:05.370263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:24.973131+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"76'79","reported_seq":599,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.176836+0000","last_change":"2024-09-21T12:35:17.231922+0000","last_active":"2024-09-21T12:36:22.176836+0000","last_peered":"2024-09-21T12:36:22.176836+0000","last_clean":"2024-09-21T12:36:22.176836+0000","last_became_active":"2024-09-21T12:32:02.034387+0000","last_became_peered":"2024-09-21T12:32:02.034387+0000","last_unstale":"2024-09-21T12:36:22.176836+0000","last_undegraded":"2024-09-21T12:36:22.176836+0000","last_fullsized":"2024-09-21T12:36:22.176836+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:35:17.231860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:35:17.231860+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:40.387153+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":854,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.180157+0000","last_change":"2024-09-21T12:36:00.155407+0000","last_active":"2024-09-21T12:36:22.180157+0000","last_peered":"2024-09-21T12:36:22.180157+0000","last_clean":"2024-09-21T12:36:22.180157+0000","last_became_active":"2024-09-21T12:36:00.155033+0000","last_became_peered":"2024-09-21T12:36:00.155033+0000","last_unstale":"2024-09-21T12:36:22.180157+0000","last_undegraded":"2024-09-21T12:36:22.180157+0000","last_fullsized":"2024-09-21T12:36:22.180157+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:33:31.733776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:31.733776+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":793,"reported_epoch":423,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:19.171338+0000","last_change":"2024-09-21T12:36:01.139279+0000","last_active":"2024-09-21T12:36:19.171338+0000","last_peered":"2024-09-21T12:36:19.171338+0000","last_clean":"2024-09-21T12:36:19.171338+0000","last_became_active":"2024-09-21T12:36:01.135527+0000","last_became_peered":"2024-09-21T12:36:01.135527+0000","last_unstale":"2024-09-21T12:36:19.171338+0000","last_undegraded":"2024-09-21T12:36:19.171338+0000","last_fullsized":"2024-09-21T12:36:19.171338+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:32.630261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:32.630261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:17.760444+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4,6],"avail_no_missing":["7","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":970,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.174726+0000","last_change":"2024-09-21T12:34:45.114379+0000","last_active":"2024-09-21T12:36:19.174726+0000","last_peered":"2024-09-21T12:36:19.174726+0000","last_clean":"2024-09-21T12:36:19.174726+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:36:19.174726+0000","last_undegraded":"2024-09-21T12:36:19.174726+0000","last_fullsized":"2024-09-21T12:36:19.174726+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:34:45.114312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:45.114312+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":788,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.172645+0000","last_change":"2024-09-21T12:36:00.149113+0000","last_active":"2024-09-21T12:36:22.172645+0000","last_peered":"2024-09-21T12:36:22.172645+0000","last_clean":"2024-09-21T12:36:22.172645+0000","last_became_active":"2024-09-21T12:36:00.148332+0000","last_became_peered":"2024-09-21T12:36:00.148332+0000","last_unstale":"2024-09-21T12:36:22.172645+0000","last_undegraded":"2024-09-21T12:36:22.172645+0000","last_fullsized":"2024-09-21T12:36:22.172645+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:26.208487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:26.208487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:31.632437+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"76'79","reported_seq":445,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.166287+0000","last_change":"2024-09-21T12:33:07.561773+0000","last_active":"2024-09-21T12:36:19.166287+0000","last_peered":"2024-09-21T12:36:19.166287+0000","last_clean":"2024-09-21T12:36:19.166287+0000","last_became_active":"2024-09-21T12:27:16.940347+0000","last_became_peered":"2024-09-21T12:27:16.940347+0000","last_unstale":"2024-09-21T12:36:19.166287+0000","last_undegraded":"2024-09-21T12:36:19.166287+0000","last_fullsized":"2024-09-21T12:36:19.166287+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:07.561702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:07.561702+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":179,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.159181+0000","last_change":"2024-09-21T12:34:00.191190+0000","last_active":"2024-09-21T12:36:19.159181+0000","last_peered":"2024-09-21T12:36:19.159181+0000","last_clean":"2024-09-21T12:36:19.159181+0000","last_became_active":"2024-09-21T12:32:38.421008+0000","last_became_peered":"2024-09-21T12:32:38.421008+0000","last_unstale":"2024-09-21T12:36:19.159181+0000","last_undegraded":"2024-09-21T12:36:19.159181+0000","last_fullsized":"2024-09-21T12:36:19.159181+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:34:00.191106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:34:00.191106+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":179,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.159486+0000","last_change":"2024-09-21T12:36:01.124070+0000","last_active":"2024-09-21T12:36:19.159486+0000","last_peered":"2024-09-21T12:36:19.159486+0000","last_clean":"2024-09-21T12:36:19.159486+0000","last_became_active":"2024-09-21T12:36:01.123399+0000","last_became_peered":"2024-09-21T12:36:01.123399+0000","last_unstale":"2024-09-21T12:36:19.159486+0000","last_undegraded":"2024-09-21T12:36:19.159486+0000","last_fullsized":"2024-09-21T12:36:19.159486+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:30.406915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:30.406915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":777,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.166984+0000","last_change":"2024-09-21T12:34:18.421712+0000","last_active":"2024-09-21T12:36:19.166984+0000","last_peered":"2024-09-21T12:36:19.166984+0000","last_clean":"2024-09-21T12:36:19.166984+0000","last_became_active":"2024-09-21T12:31:50.040754+0000","last_became_peered":"2024-09-21T12:31:50.040754+0000","last_unstale":"2024-09-21T12:36:19.166984+0000","last_undegraded":"2024-09-21T12:36:19.166984+0000","last_fullsized":"2024-09-21T12:36:19.166984+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:18.421650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:18.421650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":237,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.182335+0000","last_change":"2024-09-21T12:31:53.359206+0000","last_active":"2024-09-21T12:36:22.182335+0000","last_peered":"2024-09-21T12:36:22.182335+0000","last_clean":"2024-09-21T12:36:22.182335+0000","last_became_active":"2024-09-21T12:31:50.036150+0000","last_became_peered":"2024-09-21T12:31:50.036150+0000","last_unstale":"2024-09-21T12:36:22.182335+0000","last_undegraded":"2024-09-21T12:36:22.182335+0000","last_fullsized":"2024-09-21T12:36:22.182335+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:53.359159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:31:53.359159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:17.628056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"76'79","reported_seq":478,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.175090+0000","last_change":"2024-09-21T12:35:16.997515+0000","last_active":"2024-09-21T12:36:19.175090+0000","last_peered":"2024-09-21T12:36:19.175090+0000","last_clean":"2024-09-21T12:36:19.175090+0000","last_became_active":"2024-09-21T12:27:16.921880+0000","last_became_peered":"2024-09-21T12:27:16.921880+0000","last_unstale":"2024-09-21T12:36:19.175090+0000","last_undegraded":"2024-09-21T12:36:19.175090+0000","last_fullsized":"2024-09-21T12:36:19.175090+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:35:16.997373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:35:16.997373+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:44.016895+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":266,"reported_epoch":423,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:19.166676+0000","last_change":"2024-09-21T12:34:34.345279+0000","last_active":"2024-09-21T12:36:19.166676+0000","last_peered":"2024-09-21T12:36:19.166676+0000","last_clean":"2024-09-21T12:34:29.286797+0000","last_became_active":"2024-09-21T12:34:34.345279+0000","last_became_peered":"2024-09-21T12:34:34.345279+0000","last_unstale":"2024-09-21T12:36:19.166676+0000","last_undegraded":"2024-09-21T12:34:34.331120+0000","last_fullsized":"2024-09-21T12:34:34.331029+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:33:02.544885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:02.544885+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:44.914352+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":28,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.171010+0000","last_change":"2024-09-21T12:36:00.147458+0000","last_active":"2024-09-21T12:36:19.171010+0000","last_peered":"2024-09-21T12:36:19.171010+0000","last_clean":"2024-09-21T12:36:19.171010+0000","last_became_active":"2024-09-21T12:36:00.147063+0000","last_became_peered":"2024-09-21T12:36:00.147063+0000","last_unstale":"2024-09-21T12:36:19.171010+0000","last_undegraded":"2024-09-21T12:36:19.171010+0000","last_fullsized":"2024-09-21T12:36:19.171010+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:28.849200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:28.849200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:34.912933+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":769,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.180461+0000","last_change":"2024-09-21T12:34:11.934886+0000","last_active":"2024-09-21T12:36:22.180461+0000","last_peered":"2024-09-21T12:36:22.180461+0000","last_clean":"2024-09-21T12:36:22.180461+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:36:22.180461+0000","last_undegraded":"2024-09-21T12:36:22.180461+0000","last_fullsized":"2024-09-21T12:36:22.180461+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:11.934826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:11.934826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":760,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.171438+0000","last_change":"2024-09-21T12:35:59.959597+0000","last_active":"2024-09-21T12:36:22.171438+0000","last_peered":"2024-09-21T12:36:22.171438+0000","last_clean":"2024-09-21T12:36:22.171438+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:36:22.171438+0000","last_undegraded":"2024-09-21T12:36:22.171438+0000","last_fullsized":"2024-09-21T12:36:22.171438+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:59.959545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:59.959545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:04.514896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"76'79","reported_seq":254,"reported_epoch":423,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:19.170443+0000","last_change":"2024-09-21T12:36:01.135676+0000","last_active":"2024-09-21T12:36:19.170443+0000","last_peered":"2024-09-21T12:36:19.170443+0000","last_clean":"2024-09-21T12:36:19.170443+0000","last_became_active":"2024-09-21T12:36:01.135246+0000","last_became_peered":"2024-09-21T12:36:01.135246+0000","last_unstale":"2024-09-21T12:36:19.170443+0000","last_undegraded":"2024-09-21T12:36:19.170443+0000","last_fullsized":"2024-09-21T12:36:19.170443+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:17.398972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:17.398972+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4,1],"avail_no_missing":["7","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":842,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.182911+0000","last_change":"2024-09-21T12:36:00.168176+0000","last_active":"2024-09-21T12:36:22.182911+0000","last_peered":"2024-09-21T12:36:22.182911+0000","last_clean":"2024-09-21T12:36:22.182911+0000","last_became_active":"2024-09-21T12:36:00.139682+0000","last_became_peered":"2024-09-21T12:36:00.139682+0000","last_unstale":"2024-09-21T12:36:22.182911+0000","last_undegraded":"2024-09-21T12:36:22.182911+0000","last_fullsized":"2024-09-21T12:36:22.182911+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:33:27.793452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:27.793452+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":806,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.174393+0000","last_change":"2024-09-21T12:36:00.151927+0000","last_active":"2024-09-21T12:36:19.174393+0000","last_peered":"2024-09-21T12:36:19.174393+0000","last_clean":"2024-09-21T12:36:19.174393+0000","last_became_active":"2024-09-21T12:36:00.151584+0000","last_became_peered":"2024-09-21T12:36:00.151584+0000","last_unstale":"2024-09-21T12:36:19.174393+0000","last_undegraded":"2024-09-21T12:36:19.174393+0000","last_fullsized":"2024-09-21T12:36:19.174393+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:40.142561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:34:40.142561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":28,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.171226+0000","last_change":"2024-09-21T12:36:00.145015+0000","last_active":"2024-09-21T12:36:19.171226+0000","last_peered":"2024-09-21T12:36:19.171226+0000","last_clean":"2024-09-21T12:36:19.171226+0000","last_became_active":"2024-09-21T12:36:00.144521+0000","last_became_peered":"2024-09-21T12:36:00.144521+0000","last_unstale":"2024-09-21T12:36:19.171226+0000","last_undegraded":"2024-09-21T12:36:19.171226+0000","last_fullsized":"2024-09-21T12:36:19.171226+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:47.316065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:47.316065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":765,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.180705+0000","last_change":"2024-09-21T12:35:51.628370+0000","last_active":"2024-09-21T12:36:22.180705+0000","last_peered":"2024-09-21T12:36:22.180705+0000","last_clean":"2024-09-21T12:36:22.180705+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:36:22.180705+0000","last_undegraded":"2024-09-21T12:36:22.180705+0000","last_fullsized":"2024-09-21T12:36:22.180705+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:51.628326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:51.628326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:05.095925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"76'79","reported_seq":28,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.170056+0000","last_change":"2024-09-21T12:36:00.151654+0000","last_active":"2024-09-21T12:36:19.170056+0000","last_peered":"2024-09-21T12:36:19.170056+0000","last_clean":"2024-09-21T12:36:19.170056+0000","last_became_active":"2024-09-21T12:36:00.151043+0000","last_became_peered":"2024-09-21T12:36:00.151043+0000","last_unstale":"2024-09-21T12:36:19.170056+0000","last_undegraded":"2024-09-21T12:36:19.170056+0000","last_fullsized":"2024-09-21T12:36:19.170056+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:35:51.976777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:35:51.976777+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:17.768008+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":785,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.180876+0000","last_change":"2024-09-21T12:35:14.789148+0000","last_active":"2024-09-21T12:36:22.180876+0000","last_peered":"2024-09-21T12:36:22.180876+0000","last_clean":"2024-09-21T12:36:22.180876+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:36:22.180876+0000","last_undegraded":"2024-09-21T12:36:22.180876+0000","last_fullsized":"2024-09-21T12:36:22.180876+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'8","last_scrub_stamp":"2024-09-21T12:35:14.789070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:14.789070+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":794,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.179896+0000","last_change":"2024-09-21T12:35:27.237430+0000","last_active":"2024-09-21T12:36:22.179896+0000","last_peered":"2024-09-21T12:36:22.179896+0000","last_clean":"2024-09-21T12:36:22.179896+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:36:22.179896+0000","last_undegraded":"2024-09-21T12:36:22.179896+0000","last_fullsized":"2024-09-21T12:36:22.179896+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:35:27.236980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:27.236980+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:49.341969+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":810,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.174567+0000","last_change":"2024-09-21T12:36:00.135914+0000","last_active":"2024-09-21T12:36:19.174567+0000","last_peered":"2024-09-21T12:36:19.174567+0000","last_clean":"2024-09-21T12:36:19.174567+0000","last_became_active":"2024-09-21T12:36:00.135537+0000","last_became_peered":"2024-09-21T12:36:00.135537+0000","last_unstale":"2024-09-21T12:36:19.174567+0000","last_undegraded":"2024-09-21T12:36:19.174567+0000","last_fullsized":"2024-09-21T12:36:19.174567+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:40.861493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:40.861493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":781,"reported_epoch":423,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:19.170659+0000","last_change":"2024-09-21T12:36:01.134927+0000","last_active":"2024-09-21T12:36:19.170659+0000","last_peered":"2024-09-21T12:36:19.170659+0000","last_clean":"2024-09-21T12:36:19.170659+0000","last_became_active":"2024-09-21T12:36:01.134484+0000","last_became_peered":"2024-09-21T12:36:01.134484+0000","last_unstale":"2024-09-21T12:36:19.170659+0000","last_undegraded":"2024-09-21T12:36:19.170659+0000","last_fullsized":"2024-09-21T12:36:19.170659+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:15.385699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:34:15.385699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4,3],"avail_no_missing":["7","3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"76'79","reported_seq":181,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.158741+0000","last_change":"2024-09-21T12:36:00.137642+0000","last_active":"2024-09-21T12:36:19.158741+0000","last_peered":"2024-09-21T12:36:19.158741+0000","last_clean":"2024-09-21T12:36:19.158741+0000","last_became_active":"2024-09-21T12:36:00.137290+0000","last_became_peered":"2024-09-21T12:36:00.137290+0000","last_unstale":"2024-09-21T12:36:19.158741+0000","last_undegraded":"2024-09-21T12:36:19.158741+0000","last_fullsized":"2024-09-21T12:36:19.158741+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:43.304606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:43.304606+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":792,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.167201+0000","last_change":"2024-09-21T12:35:48.922444+0000","last_active":"2024-09-21T12:36:19.167201+0000","last_peered":"2024-09-21T12:36:19.167201+0000","last_clean":"2024-09-21T12:36:19.167201+0000","last_became_active":"2024-09-21T12:32:38.426822+0000","last_became_peered":"2024-09-21T12:32:38.426822+0000","last_unstale":"2024-09-21T12:36:19.167201+0000","last_undegraded":"2024-09-21T12:36:19.167201+0000","last_fullsized":"2024-09-21T12:36:19.167201+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:48.922389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:48.922389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:56.376647+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":791,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T12:36:22.181122+0000","last_change":"2024-09-21T12:35:30.730640+0000","last_active":"2024-09-21T12:36:22.181122+0000","last_peered":"2024-09-21T12:36:22.181122+0000","last_clean":"2024-09-21T12:36:22.181122+0000","last_became_active":"2024-09-21T12:31:39.052486+0000","last_became_peered":"2024-09-21T12:31:39.052486+0000","last_unstale":"2024-09-21T12:36:22.181122+0000","last_undegraded":"2024-09-21T12:36:22.181122+0000","last_fullsized":"2024-09-21T12:36:22.181122+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:30.730574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:30.730574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:50.338601+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":804,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T12:36:19.173993+0000","last_change":"2024-09-21T12:34:44.919918+0000","last_active":"2024-09-21T12:36:19.173993+0000","last_peered":"2024-09-21T12:36:19.173993+0000","last_clean":"2024-09-21T12:36:19.173993+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:36:19.173993+0000","last_undegraded":"2024-09-21T12:36:19.173993+0000","last_fullsized":"2024-09-21T12:36:19.173993+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:34:44.919796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:34:44.919796+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":805,"reported_epoch":424,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:22.178443+0000","last_change":"2024-09-21T12:36:01.131503+0000","last_active":"2024-09-21T12:36:22.178443+0000","last_peered":"2024-09-21T12:36:22.178443+0000","last_clean":"2024-09-21T12:36:22.178443+0000","last_became_active":"2024-09-21T12:36:01.130870+0000","last_became_peered":"2024-09-21T12:36:01.130870+0000","last_unstale":"2024-09-21T12:36:22.178443+0000","last_undegraded":"2024-09-21T12:36:22.178443+0000","last_fullsized":"2024-09-21T12:36:22.178443+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:36.112155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:33:36.112155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7,1],"avail_no_missing":["0","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":3375759360,"num_objects":51510,"num_object_clones":0,"num_object_copies":154530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12838,"num_objects_misplaced":6435,"num_objects_unfound":0,"num_objects_dirty":51510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51510,"num_write_kb":3296640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4193,"num_bytes_recovered":274792448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9469558784,"data_stored":9469558784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51510,"ondisk_log_size":51510,"up":42,"acting":44,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":84,"acting":90,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106,"ondisk_log_size":106,"up":87,"acting":92,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":54,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1105,"num_read_kb":862,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":472,"num_bytes_recovered":9372,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":11216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":687,"ondisk_log_size":687,"up":87,"acting":88,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":598,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":88,"acting":90,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":617202,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":9,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":289522,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2048000,"data_stored":2033792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2449,"ondisk_log_size":2449,"up":86,"acting":89,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234181,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1913560,"kb_used_data":1218760,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":91831592,"statfs":{"total":95995035648,"available":94035550208,"internally_reserved":0,"allocated":1248010240,"data_stored":1247563696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":711457226},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463117,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2276296,"kb_used_data":1577336,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":91468856,"statfs":{"total":95995035648,"available":93664108544,"internally_reserved":0,"allocated":1615192064,"data_stored":1614760821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":715717062},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724752,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1496796,"kb_used_data":851980,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":92248356,"statfs":{"total":95995035648,"available":94462316544,"internally_reserved":0,"allocated":872427520,"data_stored":871977816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953688,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1126752,"kb_used_data":784588,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":92618400,"statfs":{"total":95995035648,"available":94841241600,"internally_reserved":0,"allocated":803418112,"data_stored":802966448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149918,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2365696,"kb_used_data":1576492,"kb_used_omap":1,"kb_used_meta":789182,"kb_avail":91379456,"statfs":{"total":95995035648,"available":93572562944,"internally_reserved":0,"allocated":1614327808,"data_stored":1613854232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":808122823},"hb_peers":[0,1,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411534,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2808848,"kb_used_data":2272320,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":90936304,"statfs":{"total":95995035648,"available":93118775296,"internally_reserved":0,"allocated":2326855680,"data_stored":2326574609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575082,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2570880,"kb_used_data":1791404,"kb_used_omap":1,"kb_used_meta":779454,"kb_avail":91174272,"statfs":{"total":95995035648,"available":93362454528,"internally_reserved":0,"allocated":1834397696,"data_stored":1833914671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":798161349},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738608,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2110388,"kb_used_data":1298976,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":91634764,"statfs":{"total":95995035648,"available":93833998336,"internally_reserved":0,"allocated":1330151424,"data_stored":1329689323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":830863813},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1349189632,"data_stored":1349189632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1859911680,"data_stored":1859911680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1624047616,"data_stored":1624047616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":833814528,"data_stored":833814528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":885784576,"data_stored":885784576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1664745472,"data_stored":1664745472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1252065280,"data_stored":1252065280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:36:27.555 INFO:tasks.thrashosds.thrasher:Setting 6.5 to [2, 0, 6] 2024-09-21T12:36:27.556 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '6.5', '2', '0', '6'] 2024-09-21T12:36:27.556 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pg-upmap 6.5 2 0 6 2024-09-21T12:36:27.646 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:27.669 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:36:27.669 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:36:27.680 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:27.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:27.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:27.781 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:27.806 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:27.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:27.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:27.908 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:27.946 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:27.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:27.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:27.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:36:27 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:36:28.050 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:28.085 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:28.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:28.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:28.186 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:28.219 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:28.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:28.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:28.321 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:28.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:28.356 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:28.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:28.458 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:28.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:28 smithi183 ceph-mon[33277]: pgmap v825: 178 pgs: 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 126 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 692 B/s rd, 78 MiB/s wr, 1.25k op/s; 12893/155127 objects degraded (8.311%); 6443/155127 objects misplaced (4.153%) 2024-09-21T12:36:28.491 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:28.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:28.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:28.504 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 35 16 61626 61610 110.005 88.25 0.00576982 0.00908596 2024-09-21T12:36:28.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:28 smithi114 ceph-mon[40867]: pgmap v825: 178 pgs: 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 126 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 692 B/s rd, 78 MiB/s wr, 1.25k op/s; 12893/155127 objects degraded (8.311%); 6443/155127 objects misplaced (4.153%) 2024-09-21T12:36:28.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:28 smithi114 ceph-mon[30324]: pgmap v825: 178 pgs: 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 126 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 692 B/s rd, 78 MiB/s wr, 1.25k op/s; 12893/155127 objects degraded (8.311%); 6443/155127 objects misplaced (4.153%) 2024-09-21T12:36:28.594 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:28.614 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:28.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:28.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:28.716 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:28.753 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:28.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:28.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:28.855 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:28.889 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:28.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:28.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:28.991 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:29.025 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:29.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:29.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:29.044 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T12:36:29.126 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:29.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:29.152 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:29.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:29.253 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:29.290 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:29.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:29.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:29.391 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:29.431 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:29.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:29.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:29.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:29 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:36:29.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:29 smithi183 ceph-mon[33277]: 2.0 scrub starts 2024-09-21T12:36:29.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:29 smithi183 ceph-mon[33277]: 2.0 scrub ok 2024-09-21T12:36:29.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:29 smithi183 ceph-mon[33277]: 1.16 scrub starts 2024-09-21T12:36:29.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:29 smithi183 ceph-mon[33277]: 1.16 scrub ok 2024-09-21T12:36:29.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:29 smithi183 ceph-mon[33277]: 6.8 scrub starts 2024-09-21T12:36:29.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:29 smithi183 ceph-mon[33277]: 6.8 scrub ok 2024-09-21T12:36:29.504 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 36 16 62996 62980 109.328 85.625 0.00651271 0.0091428 2024-09-21T12:36:29.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:29 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:36:29.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:29 smithi114 ceph-mon[30324]: 2.0 scrub starts 2024-09-21T12:36:29.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:29 smithi114 ceph-mon[30324]: 2.0 scrub ok 2024-09-21T12:36:29.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:29 smithi114 ceph-mon[30324]: 1.16 scrub starts 2024-09-21T12:36:29.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:29 smithi114 ceph-mon[30324]: 1.16 scrub ok 2024-09-21T12:36:29.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:29 smithi114 ceph-mon[30324]: 6.8 scrub starts 2024-09-21T12:36:29.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:29 smithi114 ceph-mon[30324]: 6.8 scrub ok 2024-09-21T12:36:29.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:29 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:36:29.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:29 smithi114 ceph-mon[40867]: 2.0 scrub starts 2024-09-21T12:36:29.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:29 smithi114 ceph-mon[40867]: 2.0 scrub ok 2024-09-21T12:36:29.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:29 smithi114 ceph-mon[40867]: 1.16 scrub starts 2024-09-21T12:36:29.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:29 smithi114 ceph-mon[40867]: 1.16 scrub ok 2024-09-21T12:36:29.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:29 smithi114 ceph-mon[40867]: 6.8 scrub starts 2024-09-21T12:36:29.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:29 smithi114 ceph-mon[40867]: 6.8 scrub ok 2024-09-21T12:36:29.532 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:29.564 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:29.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:29.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:29.666 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:29.687 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:29.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:29.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:29.789 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:29.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:29.828 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:29.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:29.929 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:29.963 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:29.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:29.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:30.065 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:30.096 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:30.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:30.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:30.198 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:30.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:30.231 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:30.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:30.331 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:30.370 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:30.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:30.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:30.472 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:30.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:30 smithi183 ceph-mon[33277]: pgmap v827: 178 pgs: 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 126 active+clean; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 793 B/s rd, 118 MiB/s wr, 1.89k op/s; 14983/179964 objects degraded (8.326%); 7466/179964 objects misplaced (4.149%) 2024-09-21T12:36:30.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:30 smithi183 ceph-mon[33277]: 5.1e scrub starts 2024-09-21T12:36:30.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:30 smithi183 ceph-mon[33277]: 5.1e scrub ok 2024-09-21T12:36:30.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:30 smithi183 ceph-mon[33277]: 5.15 scrub starts 2024-09-21T12:36:30.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:30 smithi183 ceph-mon[33277]: 5.15 scrub ok 2024-09-21T12:36:30.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:30.499 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:30.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:30.504 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 37 16 64205 64189 108.415 75.5625 0.00732576 0.00921917 2024-09-21T12:36:30.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:30 smithi114 ceph-mon[40867]: pgmap v827: 178 pgs: 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 126 active+clean; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 793 B/s rd, 118 MiB/s wr, 1.89k op/s; 14983/179964 objects degraded (8.326%); 7466/179964 objects misplaced (4.149%) 2024-09-21T12:36:30.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:30 smithi114 ceph-mon[40867]: 5.1e scrub starts 2024-09-21T12:36:30.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:30 smithi114 ceph-mon[40867]: 5.1e scrub ok 2024-09-21T12:36:30.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:30 smithi114 ceph-mon[40867]: 5.15 scrub starts 2024-09-21T12:36:30.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:30 smithi114 ceph-mon[40867]: 5.15 scrub ok 2024-09-21T12:36:30.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:30 smithi114 ceph-mon[30324]: pgmap v827: 178 pgs: 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 126 active+clean; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 793 B/s rd, 118 MiB/s wr, 1.89k op/s; 14983/179964 objects degraded (8.326%); 7466/179964 objects misplaced (4.149%) 2024-09-21T12:36:30.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:30 smithi114 ceph-mon[30324]: 5.1e scrub starts 2024-09-21T12:36:30.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:30 smithi114 ceph-mon[30324]: 5.1e scrub ok 2024-09-21T12:36:30.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:30 smithi114 ceph-mon[30324]: 5.15 scrub starts 2024-09-21T12:36:30.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:30 smithi114 ceph-mon[30324]: 5.15 scrub ok 2024-09-21T12:36:30.601 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:30.637 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:30.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:30.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:30.738 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:30.768 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:30.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:30.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:30.869 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:30.906 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:30.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:30.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:31.008 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:31.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:31.045 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:31.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:31.146 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:31.180 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:31.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:31.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:31.282 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:31.312 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:31.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:31.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:31.413 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:31.450 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:31.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:31.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:31.504 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 38 15 65727 65712 108.067 95.1875 0.00571186 0.00924925 2024-09-21T12:36:31.552 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:31.583 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:31.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:31.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:31.685 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:31.717 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:31.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:31.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:31.818 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:31.853 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:31.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:31.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:31.954 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:31.989 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:31.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:31.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:32.090 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:32.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:32.129 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:32.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:32.229 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:32.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:32.275 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:32.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:32.376 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:32.412 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:32.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:32.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:32 smithi183 ceph-mon[33277]: pgmap v828: 178 pgs: 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 126 active+clean; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 95 MiB/s wr, 1.52k op/s; 14983/179964 objects degraded (8.326%); 7466/179964 objects misplaced (4.149%) 2024-09-21T12:36:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:32 smithi183 ceph-mon[33277]: 1.c scrub starts 2024-09-21T12:36:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:32 smithi183 ceph-mon[33277]: 1.c scrub ok 2024-09-21T12:36:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:32 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 14983/179964 objects degraded (8.326%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:32.504 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 39 16 67117 67101 107.522 86.8125 0.00631047 0.00929564 2024-09-21T12:36:32.514 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:32.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:32 smithi114 ceph-mon[40867]: pgmap v828: 178 pgs: 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 126 active+clean; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 95 MiB/s wr, 1.52k op/s; 14983/179964 objects degraded (8.326%); 7466/179964 objects misplaced (4.149%) 2024-09-21T12:36:32.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:32 smithi114 ceph-mon[40867]: 1.c scrub starts 2024-09-21T12:36:32.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:32 smithi114 ceph-mon[40867]: 1.c scrub ok 2024-09-21T12:36:32.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:32 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 14983/179964 objects degraded (8.326%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:32.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:32 smithi114 ceph-mon[30324]: pgmap v828: 178 pgs: 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 126 active+clean; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 95 MiB/s wr, 1.52k op/s; 14983/179964 objects degraded (8.326%); 7466/179964 objects misplaced (4.149%) 2024-09-21T12:36:32.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:32 smithi114 ceph-mon[30324]: 1.c scrub starts 2024-09-21T12:36:32.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:32 smithi114 ceph-mon[30324]: 1.c scrub ok 2024-09-21T12:36:32.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:32 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 14983/179964 objects degraded (8.326%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:32.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:32.544 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:32.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:32.644 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:32.674 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:32.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:32.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:32.776 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:32.805 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:32.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:32.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:32.907 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:32.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:32.944 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:32.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:33.024 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:36:32 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:36:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:36:33.044 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:33.065 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:33.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:33.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:33.166 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:33.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:33.196 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:33.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:33.298 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:33.320 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:33.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:33.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:33.422 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:33.443 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:33.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:33.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:33.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:33 smithi183 ceph-mon[33277]: 5.d scrub starts 2024-09-21T12:36:33.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:33 smithi183 ceph-mon[33277]: 5.d scrub ok 2024-09-21T12:36:33.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:33 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:36:33.504 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-21T12:36:33.504454+0000 min lat: 0.00111509 max lat: 2.08308 avg lat: 0.00938435 2024-09-21T12:36:33.505 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T12:36:33.505 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 40 16 68190 68174 106.51 67.0625 0.0138995 0.00938435 2024-09-21T12:36:33.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:33 smithi114 ceph-mon[40867]: 5.d scrub starts 2024-09-21T12:36:33.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:33 smithi114 ceph-mon[40867]: 5.d scrub ok 2024-09-21T12:36:33.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:33 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:36:33.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:33 smithi114 ceph-mon[30324]: 5.d scrub starts 2024-09-21T12:36:33.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:33 smithi114 ceph-mon[30324]: 5.d scrub ok 2024-09-21T12:36:33.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:33 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:36:33.544 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:33.582 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:33.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:33.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:33.684 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:33.720 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:33.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:33.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:33.822 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:33.842 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:33.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:33.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:33.944 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:33.977 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:33.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:33.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:34.079 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:34.111 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:34.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:34.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:34.216 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:34.251 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:34.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:34.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:34.353 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:34.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:34.388 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:34.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:34.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:34 smithi183 ceph-mon[33277]: pgmap v829: 178 pgs: 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 126 active+clean; 3.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s; 15868/191955 objects degraded (8.267%); 8344/191955 objects misplaced (4.347%) 2024-09-21T12:36:34.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:34 smithi183 ceph-mon[33277]: 4.2 scrub starts 2024-09-21T12:36:34.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:34 smithi183 ceph-mon[33277]: 4.2 scrub ok 2024-09-21T12:36:34.489 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:34.505 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 41 16 69654 69638 106.144 91.5 0.00661719 0.00941632 2024-09-21T12:36:34.520 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:34.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:34.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:34.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:34 smithi114 ceph-mon[40867]: pgmap v829: 178 pgs: 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 126 active+clean; 3.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s; 15868/191955 objects degraded (8.267%); 8344/191955 objects misplaced (4.347%) 2024-09-21T12:36:34.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:34 smithi114 ceph-mon[40867]: 4.2 scrub starts 2024-09-21T12:36:34.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:34 smithi114 ceph-mon[40867]: 4.2 scrub ok 2024-09-21T12:36:34.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:34 smithi114 ceph-mon[30324]: pgmap v829: 178 pgs: 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 126 active+clean; 3.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s; 15868/191955 objects degraded (8.267%); 8344/191955 objects misplaced (4.347%) 2024-09-21T12:36:34.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:34 smithi114 ceph-mon[30324]: 4.2 scrub starts 2024-09-21T12:36:34.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:34 smithi114 ceph-mon[30324]: 4.2 scrub ok 2024-09-21T12:36:34.621 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:34.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:34.653 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:34.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:34.666 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T12:36:34.682 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:36:34.754 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:34.763 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:36:34.782 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:34.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:34.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:34.884 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:34.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:34.929 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:34.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:35.030 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:35.064 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:35.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:35.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:35.113 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:36:35.165 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:35.199 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:35.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:35.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:35.355 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:35.385 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:35.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:35.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:35.436 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:36:35.436 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:36:35.468 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:36:35.468 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:36:35.471 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T12:36:35.486 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:35.487 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:35 smithi114 ceph-mon[40867]: 5.12 scrub starts 2024-09-21T12:36:35.487 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:35 smithi114 ceph-mon[40867]: 5.12 scrub ok 2024-09-21T12:36:35.487 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:35 smithi114 ceph-mon[40867]: 3.16 scrub starts 2024-09-21T12:36:35.487 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:35 smithi114 ceph-mon[40867]: 3.16 scrub ok 2024-09-21T12:36:35.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:35 smithi114 ceph-mon[30324]: 5.12 scrub starts 2024-09-21T12:36:35.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:35 smithi114 ceph-mon[30324]: 5.12 scrub ok 2024-09-21T12:36:35.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:35 smithi114 ceph-mon[30324]: 3.16 scrub starts 2024-09-21T12:36:35.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:35 smithi114 ceph-mon[30324]: 3.16 scrub ok 2024-09-21T12:36:35.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:35 smithi183 ceph-mon[33277]: 5.12 scrub starts 2024-09-21T12:36:35.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:35 smithi183 ceph-mon[33277]: 5.12 scrub ok 2024-09-21T12:36:35.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:35 smithi183 ceph-mon[33277]: 3.16 scrub starts 2024-09-21T12:36:35.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:35 smithi183 ceph-mon[33277]: 3.16 scrub ok 2024-09-21T12:36:35.501 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:36:35.501 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:36:35.504 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 42 16 71103 71087 105.772 90.5625 0.0123373 0.00944948 2024-09-21T12:36:35.523 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:35.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:35.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:35.535 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:36:35.536 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:36:35.568 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:36:35.568 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:36:35.606 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:36:35.606 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:36:35.626 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:35.643 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:36:35.644 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:36:35.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:35.667 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:35.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:35.768 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:35.805 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:35.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:35.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:35.907 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:35.938 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:35.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:35.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:36.040 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:36.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:36.079 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:36.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:36.180 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:36.211 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:36.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:36.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:36.312 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:36.342 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:36.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:36.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:36.443 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:36.480 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:36.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:36.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:36.505 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 43 16 72658 72642 105.573 97.1875 0.0081939 0.00946792 2024-09-21T12:36:36.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:36 smithi114 ceph-mon[30324]: pgmap v830: 178 pgs: 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 126 active+clean; 4.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 114 MiB/s wr, 1.83k op/s; 16755/201003 objects degraded (8.336%); 8344/201003 objects misplaced (4.151%) 2024-09-21T12:36:36.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:36 smithi114 ceph-mon[30324]: 6.9 scrub starts 2024-09-21T12:36:36.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:36 smithi114 ceph-mon[30324]: 6.9 scrub ok 2024-09-21T12:36:36.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:36 smithi114 ceph-mon[30324]: 6.a scrub starts 2024-09-21T12:36:36.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:36 smithi114 ceph-mon[30324]: 6.a scrub ok 2024-09-21T12:36:36.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:36 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2156975096' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:36:36.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:36 smithi114 ceph-mon[40867]: pgmap v830: 178 pgs: 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 126 active+clean; 4.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 114 MiB/s wr, 1.83k op/s; 16755/201003 objects degraded (8.336%); 8344/201003 objects misplaced (4.151%) 2024-09-21T12:36:36.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:36 smithi114 ceph-mon[40867]: 6.9 scrub starts 2024-09-21T12:36:36.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:36 smithi114 ceph-mon[40867]: 6.9 scrub ok 2024-09-21T12:36:36.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:36 smithi114 ceph-mon[40867]: 6.a scrub starts 2024-09-21T12:36:36.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:36 smithi114 ceph-mon[40867]: 6.a scrub ok 2024-09-21T12:36:36.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:36 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2156975096' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:36:36.582 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:36.615 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T12:36:36.616 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T12:36:36.622 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:36.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:36.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:36.724 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:36.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:36 smithi183 ceph-mon[33277]: pgmap v830: 178 pgs: 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 126 active+clean; 4.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 114 MiB/s wr, 1.83k op/s; 16755/201003 objects degraded (8.336%); 8344/201003 objects misplaced (4.151%) 2024-09-21T12:36:36.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:36 smithi183 ceph-mon[33277]: 6.9 scrub starts 2024-09-21T12:36:36.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:36 smithi183 ceph-mon[33277]: 6.9 scrub ok 2024-09-21T12:36:36.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:36 smithi183 ceph-mon[33277]: 6.a scrub starts 2024-09-21T12:36:36.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:36 smithi183 ceph-mon[33277]: 6.a scrub ok 2024-09-21T12:36:36.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:36 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2156975096' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:36:36.757 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:36.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:36.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:36.859 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:36.896 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:36.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:36.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:36.923 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:36:36.997 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:37.031 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:37.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:37.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:37.133 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:37.164 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:37.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:37.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:37.266 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:37.270 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-21T12:36:37.296 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:37.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:37.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:37.397 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:37.436 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:37.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:37.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:37.450 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:37 smithi114 ceph-mon[30324]: 5.18 scrub starts 2024-09-21T12:36:37.450 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:37 smithi114 ceph-mon[30324]: 5.18 scrub ok 2024-09-21T12:36:37.450 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:37 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3892817529' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:36:37.450 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:37 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:36:37.450 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:37 smithi114 ceph-mon[30324]: 4.14 scrub starts 2024-09-21T12:36:37.450 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:37 smithi114 ceph-mon[30324]: 4.14 scrub ok 2024-09-21T12:36:37.450 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:37 smithi114 ceph-mon[30324]: 5.7 scrub starts 2024-09-21T12:36:37.450 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:37 smithi114 ceph-mon[30324]: 5.7 scrub ok 2024-09-21T12:36:37.450 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:37 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 16755/201003 objects degraded (8.336%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:37.451 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:37 smithi114 ceph-mon[40867]: 5.18 scrub starts 2024-09-21T12:36:37.451 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:37 smithi114 ceph-mon[40867]: 5.18 scrub ok 2024-09-21T12:36:37.451 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:37 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3892817529' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:36:37.451 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:37 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:36:37.451 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:37 smithi114 ceph-mon[40867]: 4.14 scrub starts 2024-09-21T12:36:37.451 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:37 smithi114 ceph-mon[40867]: 4.14 scrub ok 2024-09-21T12:36:37.451 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:37 smithi114 ceph-mon[40867]: 5.7 scrub starts 2024-09-21T12:36:37.451 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:37 smithi114 ceph-mon[40867]: 5.7 scrub ok 2024-09-21T12:36:37.451 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:37 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 16755/201003 objects degraded (8.336%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:37.505 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 44 16 74138 74122 105.275 92.5 0.0166768 0.00949403 2024-09-21T12:36:37.538 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:37.569 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:37.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:37.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:37.593 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:37 smithi183 ceph-mon[33277]: 5.18 scrub starts 2024-09-21T12:36:37.593 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:37 smithi183 ceph-mon[33277]: 5.18 scrub ok 2024-09-21T12:36:37.593 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:37 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3892817529' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:36:37.593 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:37 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:36:37.593 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:37 smithi183 ceph-mon[33277]: 4.14 scrub starts 2024-09-21T12:36:37.593 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:37 smithi183 ceph-mon[33277]: 4.14 scrub ok 2024-09-21T12:36:37.593 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:37 smithi183 ceph-mon[33277]: 5.7 scrub starts 2024-09-21T12:36:37.593 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:37 smithi183 ceph-mon[33277]: 5.7 scrub ok 2024-09-21T12:36:37.593 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:37 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 16755/201003 objects degraded (8.336%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:37.670 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:37.706 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:37.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:37.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:37.808 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:37.839 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:37.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:37.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:37.940 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:37.975 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:37.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:37.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:37.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:36:37 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:36:38.077 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:38.114 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:38.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:38.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:38.215 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:38.248 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:38.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:38.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:38.274 INFO:teuthology.orchestra.run.smithi114.stderr:set 6.5 pg_upmap mapping to [2,0,6] 2024-09-21T12:36:38.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:38 smithi114 ceph-mon[40867]: pgmap v831: 178 pgs: 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 126 active+clean; 4.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 96 MiB/s wr, 1.53k op/s; 16755/201003 objects degraded (8.336%); 8344/201003 objects misplaced (4.151%) 2024-09-21T12:36:38.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:38 smithi114 ceph-mon[40867]: 4.3 scrub starts 2024-09-21T12:36:38.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:38 smithi114 ceph-mon[40867]: 4.3 scrub ok 2024-09-21T12:36:38.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:38 smithi114 ceph-mon[40867]: 1.d scrub starts 2024-09-21T12:36:38.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:38 smithi114 ceph-mon[40867]: 1.d scrub ok 2024-09-21T12:36:38.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:38 smithi114 ceph-mon[40867]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:36:38.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:38 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:36:38.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:38 smithi114 ceph-mon[40867]: osdmap e427: 8 total, 7 up, 8 in 2024-09-21T12:36:38.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:38 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3204887675' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "6.5", "id": [2, 0, 6]} : dispatch 2024-09-21T12:36:38.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:38 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "6.5", "id": [2, 0, 6]} : dispatch 2024-09-21T12:36:38.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:38 smithi114 ceph-mon[30324]: pgmap v831: 178 pgs: 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 126 active+clean; 4.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 96 MiB/s wr, 1.53k op/s; 16755/201003 objects degraded (8.336%); 8344/201003 objects misplaced (4.151%) 2024-09-21T12:36:38.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:38 smithi114 ceph-mon[30324]: 4.3 scrub starts 2024-09-21T12:36:38.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:38 smithi114 ceph-mon[30324]: 4.3 scrub ok 2024-09-21T12:36:38.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:38 smithi114 ceph-mon[30324]: 1.d scrub starts 2024-09-21T12:36:38.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:38 smithi114 ceph-mon[30324]: 1.d scrub ok 2024-09-21T12:36:38.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:38 smithi114 ceph-mon[30324]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:36:38.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:38 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:36:38.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:38 smithi114 ceph-mon[30324]: osdmap e427: 8 total, 7 up, 8 in 2024-09-21T12:36:38.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:38 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3204887675' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "6.5", "id": [2, 0, 6]} : dispatch 2024-09-21T12:36:38.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:38 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "6.5", "id": [2, 0, 6]} : dispatch 2024-09-21T12:36:38.350 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:38.385 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:38.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:38.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:38.486 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:38.506 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 45 15 75638 75623 105.02 93.8125 0.0117428 0.00951773 2024-09-21T12:36:38.520 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:38.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:38.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:38.617 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:36:38.622 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:38.652 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:38.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:38.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:38.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:38 smithi183 ceph-mon[33277]: pgmap v831: 178 pgs: 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 126 active+clean; 4.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 96 MiB/s wr, 1.53k op/s; 16755/201003 objects degraded (8.336%); 8344/201003 objects misplaced (4.151%) 2024-09-21T12:36:38.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:38 smithi183 ceph-mon[33277]: 4.3 scrub starts 2024-09-21T12:36:38.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:38 smithi183 ceph-mon[33277]: 4.3 scrub ok 2024-09-21T12:36:38.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:38 smithi183 ceph-mon[33277]: 1.d scrub starts 2024-09-21T12:36:38.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:38 smithi183 ceph-mon[33277]: 1.d scrub ok 2024-09-21T12:36:38.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:38 smithi183 ceph-mon[33277]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:36:38.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:38 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:36:38.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:38 smithi183 ceph-mon[33277]: osdmap e427: 8 total, 7 up, 8 in 2024-09-21T12:36:38.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:38 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3204887675' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "6.5", "id": [2, 0, 6]} : dispatch 2024-09-21T12:36:38.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:38 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "6.5", "id": [2, 0, 6]} : dispatch 2024-09-21T12:36:38.754 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:38.789 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:38.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:38.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:38.891 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:38.912 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:38.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:38.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:39.013 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:39.049 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:39.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:39.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:39.151 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:39.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:39.192 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:39.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:39.293 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:39.332 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:39.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:39.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:39.433 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:39.472 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:39.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:39.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:39.505 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 46 16 77083 77067 104.699 90.25 0.00958503 0.00954605 2024-09-21T12:36:39.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:39 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:36:39.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:39 smithi114 ceph-mon[30324]: 6.1c scrub starts 2024-09-21T12:36:39.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:39 smithi114 ceph-mon[30324]: 6.1c scrub ok 2024-09-21T12:36:39.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:39 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.5", "id": [2, 0, 6]}]': finished 2024-09-21T12:36:39.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:39 smithi114 ceph-mon[30324]: osdmap e428: 8 total, 7 up, 8 in 2024-09-21T12:36:39.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:39 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:36:39.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:39 smithi114 ceph-mon[40867]: 6.1c scrub starts 2024-09-21T12:36:39.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:39 smithi114 ceph-mon[40867]: 6.1c scrub ok 2024-09-21T12:36:39.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:39 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.5", "id": [2, 0, 6]}]': finished 2024-09-21T12:36:39.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:39 smithi114 ceph-mon[40867]: osdmap e428: 8 total, 7 up, 8 in 2024-09-21T12:36:39.574 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:39.609 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:39.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:39.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:39.639 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:36:39.639 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-21T12:36:39.639 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.640 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.26147.0:74904 11.e e427/417 11:73e16d58:::benchmark_data_smithi114_187622_object74903:head v 427'4713, mlcod=427'4711)", 2024-09-21T12:36:39.640 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:36:37.995413+0000", 2024-09-21T12:36:39.640 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.011275212999999999, 2024-09-21T12:36:39.640 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.011302856999999999, 2024-09-21T12:36:39.640 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:36:39.640 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:36:39.640 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:36:39.640 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:36:39.640 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.640 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:36:39.640 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.995413+0000", 2024-09-21T12:36:39.641 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:39.641 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.641 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.641 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:36:39.641 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.995413+0000", 2024-09-21T12:36:39.641 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:39.641 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.641 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.641 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:36:39.641 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.995413+0000", 2024-09-21T12:36:39.641 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8099999999999998e-07 2024-09-21T12:36:39.641 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.642 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.642 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:36:39.642 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.995453+0000", 2024-09-21T12:36:39.642 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.9632999999999998e-05 2024-09-21T12:36:39.642 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.642 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.642 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:36:39.642 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.995461+0000", 2024-09-21T12:36:39.642 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.7860000000000002e-06 2024-09-21T12:36:39.642 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.642 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.642 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:36:39.643 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.995467+0000", 2024-09-21T12:36:39.643 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4319999999999998e-06 2024-09-21T12:36:39.643 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.643 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.643 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:36:39.643 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.995606+0000", 2024-09-21T12:36:39.643 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013931999999999999 2024-09-21T12:36:39.643 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.643 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.643 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:36:39.643 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.995626+0000", 2024-09-21T12:36:39.643 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0245999999999999e-05 2024-09-21T12:36:39.644 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:39.644 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:39.644 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:39.644 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.644 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.644 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.26147.0:74907 11.6 e427/405 11:645805ef:::benchmark_data_smithi114_187622_object74906:head v 427'4642, mlcod=427'4640)", 2024-09-21T12:36:39.644 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:36:37.997092+0000", 2024-09-21T12:36:39.644 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0095962129999999993, 2024-09-21T12:36:39.644 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0096762020000000001, 2024-09-21T12:36:39.644 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:36:39.644 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:36:39.644 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:36:39.644 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:36:39.645 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.645 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:36:39.645 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.997092+0000", 2024-09-21T12:36:39.645 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:39.645 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.645 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.645 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:36:39.645 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.997092+0000", 2024-09-21T12:36:39.645 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:39.645 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.645 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.645 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:36:39.646 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.997093+0000", 2024-09-21T12:36:39.646 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5300000000000003e-07 2024-09-21T12:36:39.646 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.646 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.646 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:36:39.646 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.997118+0000", 2024-09-21T12:36:39.646 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5029e-05 2024-09-21T12:36:39.646 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.646 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.646 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:36:39.646 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.997128+0000", 2024-09-21T12:36:39.646 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0947e-05 2024-09-21T12:36:39.647 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.647 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.647 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:36:39.647 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.997135+0000", 2024-09-21T12:36:39.647 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1029999999999998e-06 2024-09-21T12:36:39.647 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.647 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.647 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:36:39.647 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.997284+0000", 2024-09-21T12:36:39.647 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014892700000000001 2024-09-21T12:36:39.647 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.647 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.648 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:36:39.648 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.997304+0000", 2024-09-21T12:36:39.648 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0109999999999999e-05 2024-09-21T12:36:39.648 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:39.648 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:39.648 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:39.648 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.648 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.648 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.26147.0:74912 11.c 11:3920d37d:::benchmark_data_smithi114_187622_object74911:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e427)", 2024-09-21T12:36:39.648 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:36:37.998903+0000", 2024-09-21T12:36:39.648 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0077848780000000003, 2024-09-21T12:36:39.648 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0079169180000000002, 2024-09-21T12:36:39.648 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:36:39.649 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:36:39.649 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:36:39.649 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:36:39.649 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.26147", 2024-09-21T12:36:39.649 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:36:39.649 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 74912 2024-09-21T12:36:39.649 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.649 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:36:39.649 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.649 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:36:39.649 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.998903+0000", 2024-09-21T12:36:39.649 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:39.650 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.650 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.650 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:36:39.650 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.998903+0000", 2024-09-21T12:36:39.650 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:39.650 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.650 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.650 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:36:39.650 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.998904+0000", 2024-09-21T12:36:39.650 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4109999999999999e-06 2024-09-21T12:36:39.650 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.651 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.651 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:36:39.651 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.998926+0000", 2024-09-21T12:36:39.651 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1489999999999999e-05 2024-09-21T12:36:39.651 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.651 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.651 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:36:39.651 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.998947+0000", 2024-09-21T12:36:39.651 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1449000000000001e-05 2024-09-21T12:36:39.651 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.651 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.651 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:36:39.652 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.998954+0000", 2024-09-21T12:36:39.652 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1359999999999999e-06 2024-09-21T12:36:39.652 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.652 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.652 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:36:39.652 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.000614+0000", 2024-09-21T12:36:39.652 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0016591279999999999 2024-09-21T12:36:39.652 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.652 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.652 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:36:39.652 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.000709+0000", 2024-09-21T12:36:39.653 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.4927999999999996e-05 2024-09-21T12:36:39.653 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.653 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.653 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4", 2024-09-21T12:36:39.653 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.000799+0000", 2024-09-21T12:36:39.653 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0218999999999998e-05 2024-09-21T12:36:39.653 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.653 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.653 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:36:39.653 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.004934+0000", 2024-09-21T12:36:39.653 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0041349330000000004 2024-09-21T12:36:39.653 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:39.654 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:39.654 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:39.654 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.654 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.654 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.26147.0:74910 11.4 e427/415 11:2ad43b5b:::benchmark_data_smithi114_187622_object74909:head v 427'4665, mlcod=427'4662)", 2024-09-21T12:36:39.654 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:36:37.999375+0000", 2024-09-21T12:36:39.654 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0073128560000000004, 2024-09-21T12:36:39.654 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0075118060000000002, 2024-09-21T12:36:39.654 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:36:39.654 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:36:39.654 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:36:39.654 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:36:39.655 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.655 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:36:39.655 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.999375+0000", 2024-09-21T12:36:39.655 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:39.655 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.655 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.655 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:36:39.655 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.999375+0000", 2024-09-21T12:36:39.655 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:39.655 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.655 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.655 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:36:39.656 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.999376+0000", 2024-09-21T12:36:39.656 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.37e-07 2024-09-21T12:36:39.656 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.656 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.656 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:36:39.656 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.999430+0000", 2024-09-21T12:36:39.656 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4153999999999997e-05 2024-09-21T12:36:39.656 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.656 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.656 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:36:39.656 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.999438+0000", 2024-09-21T12:36:39.657 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6990000000000003e-06 2024-09-21T12:36:39.657 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.657 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.657 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:36:39.657 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.999443+0000", 2024-09-21T12:36:39.657 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3170000000000001e-06 2024-09-21T12:36:39.657 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.657 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.657 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:36:39.657 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.002625+0000", 2024-09-21T12:36:39.657 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0031825069999999998 2024-09-21T12:36:39.657 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.657 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.658 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:36:39.658 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.002645+0000", 2024-09-21T12:36:39.658 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9590999999999999e-05 2024-09-21T12:36:39.658 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:39.658 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:39.658 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:39.658 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.658 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.658 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.26147.0:74915 11.c 11:32f53b55:::benchmark_data_smithi114_187622_object74914:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e427)", 2024-09-21T12:36:39.658 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:36:37.999565+0000", 2024-09-21T12:36:39.658 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0071230080000000001, 2024-09-21T12:36:39.659 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.007374037, 2024-09-21T12:36:39.659 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:36:39.659 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:36:39.659 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:36:39.659 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:36:39.659 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.26147", 2024-09-21T12:36:39.659 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:36:39.659 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 74915 2024-09-21T12:36:39.659 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.659 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:36:39.659 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.660 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:36:39.660 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.999565+0000", 2024-09-21T12:36:39.660 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:39.660 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.660 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.660 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:36:39.660 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.999565+0000", 2024-09-21T12:36:39.660 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:39.660 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.660 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.660 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:36:39.660 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.999566+0000", 2024-09-21T12:36:39.661 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5099999999999999e-07 2024-09-21T12:36:39.661 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.661 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.661 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:36:39.661 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.999584+0000", 2024-09-21T12:36:39.661 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8493000000000001e-05 2024-09-21T12:36:39.661 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.661 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.661 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:36:39.661 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.999596+0000", 2024-09-21T12:36:39.661 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1715e-05 2024-09-21T12:36:39.661 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.662 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.662 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:36:39.662 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.999602+0000", 2024-09-21T12:36:39.662 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7089999999999999e-06 2024-09-21T12:36:39.662 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.662 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.662 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:36:39.662 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.004982+0000", 2024-09-21T12:36:39.662 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0053802570000000003 2024-09-21T12:36:39.662 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.662 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.662 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:36:39.663 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.005143+0000", 2024-09-21T12:36:39.663 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000161597 2024-09-21T12:36:39.663 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.663 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.663 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4", 2024-09-21T12:36:39.663 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.005287+0000", 2024-09-21T12:36:39.663 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014311200000000001 2024-09-21T12:36:39.663 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:39.663 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:39.663 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:39.663 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.663 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.663 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.26147.0:74913 11.e e427/417 11:74dc06ed:::benchmark_data_smithi114_187622_object74912:head v 427'4714, mlcod=427'4711)", 2024-09-21T12:36:39.664 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:36:37.999771+0000", 2024-09-21T12:36:39.664 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0069168750000000003, 2024-09-21T12:36:39.664 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0072247409999999998, 2024-09-21T12:36:39.664 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:36:39.664 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:36:39.664 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:36:39.664 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:36:39.664 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.664 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:36:39.664 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.999771+0000", 2024-09-21T12:36:39.664 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:39.664 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.665 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.665 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:36:39.665 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.999771+0000", 2024-09-21T12:36:39.665 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:39.665 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.665 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.665 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:36:39.665 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.999772+0000", 2024-09-21T12:36:39.665 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1900000000000003e-07 2024-09-21T12:36:39.665 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.665 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.665 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:36:39.666 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.999876+0000", 2024-09-21T12:36:39.666 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000104818 2024-09-21T12:36:39.666 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.666 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.666 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:36:39.666 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.999886+0000", 2024-09-21T12:36:39.666 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1689999999999998e-06 2024-09-21T12:36:39.666 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.666 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.666 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:36:39.666 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.999891+0000", 2024-09-21T12:36:39.666 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6509999999999998e-06 2024-09-21T12:36:39.666 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.667 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.667 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:36:39.667 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.001657+0000", 2024-09-21T12:36:39.667 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001765846 2024-09-21T12:36:39.667 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.667 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.667 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:36:39.667 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.001689+0000", 2024-09-21T12:36:39.667 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2051000000000002e-05 2024-09-21T12:36:39.667 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:39.667 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:39.667 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:39.668 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.668 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.668 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.26147.0:74916 11.e e427/417 11:7f0a8e93:::benchmark_data_smithi114_187622_object74915:head v 427'4715, mlcod=427'4711)", 2024-09-21T12:36:39.668 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:36:38.002261+0000", 2024-09-21T12:36:39.668 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0044272019999999999, 2024-09-21T12:36:39.668 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0047948770000000003, 2024-09-21T12:36:39.668 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:36:39.668 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:36:39.668 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:36:39.668 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:36:39.668 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.668 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:36:39.669 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.002261+0000", 2024-09-21T12:36:39.669 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:39.669 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.669 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.669 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:36:39.669 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.002261+0000", 2024-09-21T12:36:39.669 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:39.669 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.669 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.669 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:36:39.669 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.002261+0000", 2024-09-21T12:36:39.669 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8999999999999995e-07 2024-09-21T12:36:39.670 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.670 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.670 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:36:39.670 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.002321+0000", 2024-09-21T12:36:39.670 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9559999999999999e-05 2024-09-21T12:36:39.670 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.670 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.670 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:36:39.670 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.002337+0000", 2024-09-21T12:36:39.670 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5534999999999999e-05 2024-09-21T12:36:39.670 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.671 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.671 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:36:39.671 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.002344+0000", 2024-09-21T12:36:39.671 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1029999999999996e-06 2024-09-21T12:36:39.671 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.671 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.671 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:36:39.671 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.004496+0000", 2024-09-21T12:36:39.671 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002152462 2024-09-21T12:36:39.671 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.671 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.671 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:36:39.672 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.004516+0000", 2024-09-21T12:36:39.672 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9538000000000002e-05 2024-09-21T12:36:39.672 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:39.672 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:39.672 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:39.672 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.672 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.672 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.26147.0:74918 11.6 e427/405 11:68190b63:::benchmark_data_smithi114_187622_object74917:head v 427'4643, mlcod=427'4640)", 2024-09-21T12:36:39.672 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:36:38.003651+0000", 2024-09-21T12:36:39.672 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0030366659999999999, 2024-09-21T12:36:39.672 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003453069, 2024-09-21T12:36:39.672 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:36:39.673 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:36:39.673 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:36:39.673 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:36:39.673 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.673 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:36:39.673 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.003651+0000", 2024-09-21T12:36:39.673 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:39.673 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.673 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.673 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:36:39.673 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.003651+0000", 2024-09-21T12:36:39.674 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:39.674 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.674 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.674 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:36:39.674 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.003652+0000", 2024-09-21T12:36:39.674 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.4399999999999999e-07 2024-09-21T12:36:39.674 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.674 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.674 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:36:39.674 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.003676+0000", 2024-09-21T12:36:39.674 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3634000000000001e-05 2024-09-21T12:36:39.674 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.675 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.675 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:36:39.675 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.003691+0000", 2024-09-21T12:36:39.675 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5145e-05 2024-09-21T12:36:39.675 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.675 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.675 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:36:39.675 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.003697+0000", 2024-09-21T12:36:39.675 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6460000000000004e-06 2024-09-21T12:36:39.675 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.675 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.675 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:36:39.676 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.004445+0000", 2024-09-21T12:36:39.676 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00074721300000000004 2024-09-21T12:36:39.676 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.676 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.676 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:36:39.676 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.004468+0000", 2024-09-21T12:36:39.676 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3197000000000002e-05 2024-09-21T12:36:39.676 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:39.676 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:39.676 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:39.676 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.676 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.677 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.26147.0:74920 11.4 e427/415 11:21add17b:::benchmark_data_smithi114_187622_object74919:head v 427'4666, mlcod=427'4662)", 2024-09-21T12:36:39.677 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:36:38.004731+0000", 2024-09-21T12:36:39.677 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0019570529999999998, 2024-09-21T12:36:39.677 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002422156, 2024-09-21T12:36:39.677 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:36:39.677 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:36:39.677 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:36:39.677 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:36:39.677 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.677 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:36:39.677 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.004731+0000", 2024-09-21T12:36:39.677 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:39.678 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.678 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.678 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:36:39.678 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.004731+0000", 2024-09-21T12:36:39.678 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:39.678 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.678 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.678 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:36:39.678 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.004732+0000", 2024-09-21T12:36:39.678 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6400000000000001e-07 2024-09-21T12:36:39.678 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.678 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.679 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:36:39.679 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.004789+0000", 2024-09-21T12:36:39.679 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7271000000000001e-05 2024-09-21T12:36:39.679 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.679 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.679 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:36:39.679 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.004797+0000", 2024-09-21T12:36:39.679 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9959999999999993e-06 2024-09-21T12:36:39.679 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.679 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.679 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:36:39.679 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.004803+0000", 2024-09-21T12:36:39.680 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7910000000000001e-06 2024-09-21T12:36:39.680 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.680 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.680 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:36:39.680 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.004914+0000", 2024-09-21T12:36:39.680 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011181800000000001 2024-09-21T12:36:39.680 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.680 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.680 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:36:39.680 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.004935+0000", 2024-09-21T12:36:39.680 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0492e-05 2024-09-21T12:36:39.681 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:39.681 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:39.681 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:39.681 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.681 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.681 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.26147.0:74906 11.c 11:3a6dd32e:::benchmark_data_smithi114_187622_object74905:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e427)", 2024-09-21T12:36:39.681 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:36:37.995136+0000", 2024-09-21T12:36:39.681 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.011551924, 2024-09-21T12:36:39.681 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.012075028, 2024-09-21T12:36:39.681 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:36:39.681 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:36:39.681 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:36:39.681 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:36:39.682 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.26147", 2024-09-21T12:36:39.682 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:36:39.682 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 74906 2024-09-21T12:36:39.682 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.682 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:36:39.682 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.682 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:36:39.682 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.995136+0000", 2024-09-21T12:36:39.682 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:39.682 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.682 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.683 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:36:39.683 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.995136+0000", 2024-09-21T12:36:39.683 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:39.683 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.683 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.683 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:36:39.683 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.995137+0000", 2024-09-21T12:36:39.683 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6700000000000003e-07 2024-09-21T12:36:39.683 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.683 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.683 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:36:39.684 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.995155+0000", 2024-09-21T12:36:39.684 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8219e-05 2024-09-21T12:36:39.684 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.684 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.684 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:36:39.684 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.995163+0000", 2024-09-21T12:36:39.684 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8849999999999999e-06 2024-09-21T12:36:39.684 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.684 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.684 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:36:39.684 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.995168+0000", 2024-09-21T12:36:39.684 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8039999999999998e-06 2024-09-21T12:36:39.685 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.685 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.685 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:36:39.685 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.998482+0000", 2024-09-21T12:36:39.685 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0033136300000000001 2024-09-21T12:36:39.685 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.685 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.685 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:36:39.685 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.998621+0000", 2024-09-21T12:36:39.685 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013866900000000001 2024-09-21T12:36:39.686 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.686 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.686 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4", 2024-09-21T12:36:39.686 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:37.998705+0000", 2024-09-21T12:36:39.686 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.4263000000000003e-05 2024-09-21T12:36:39.686 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:39.686 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:39.686 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:36:39.686 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:38.004633+0000", 2024-09-21T12:36:39.686 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0059275819999999998 2024-09-21T12:36:39.686 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:39.686 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:39.687 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:39.687 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:39.687 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:36:39.687 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 10 2024-09-21T12:36:39.687 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:36:39.711 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:39.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:39 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:36:39.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:39 smithi183 ceph-mon[33277]: 6.1c scrub starts 2024-09-21T12:36:39.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:39 smithi183 ceph-mon[33277]: 6.1c scrub ok 2024-09-21T12:36:39.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:39 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.5", "id": [2, 0, 6]}]': finished 2024-09-21T12:36:39.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:39 smithi183 ceph-mon[33277]: osdmap e428: 8 total, 7 up, 8 in 2024-09-21T12:36:39.751 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:39.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:39.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:39.853 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:39.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:39.888 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:39.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:39.989 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:40.023 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:40.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:40.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:40.124 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:40.162 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:40.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:40.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:40.264 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:40.314 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T12:36:40.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:40.318 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:40.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:40.419 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:40.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:40.447 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:40.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:40.505 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 47 16 78709 78693 104.634 101.625 0.00950251 0.00955248 2024-09-21T12:36:40.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:40 smithi114 ceph-mon[40867]: pgmap v834: 178 pgs: 1 unknown, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.76k op/s; 18528/222204 objects degraded (8.338%); 9261/222204 objects misplaced (4.168%) 2024-09-21T12:36:40.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:40 smithi114 ceph-mon[40867]: 6.18 scrub starts 2024-09-21T12:36:40.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:40 smithi114 ceph-mon[40867]: 6.18 scrub ok 2024-09-21T12:36:40.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:40 smithi114 ceph-mon[40867]: 6.12 scrub starts 2024-09-21T12:36:40.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:40 smithi114 ceph-mon[40867]: 6.12 scrub ok 2024-09-21T12:36:40.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:40 smithi114 ceph-mon[40867]: osdmap e429: 8 total, 7 up, 8 in 2024-09-21T12:36:40.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:40 smithi114 ceph-mon[40867]: 1.13 scrub starts 2024-09-21T12:36:40.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:40 smithi114 ceph-mon[40867]: 1.13 scrub ok 2024-09-21T12:36:40.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:40 smithi114 ceph-mon[30324]: pgmap v834: 178 pgs: 1 unknown, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.76k op/s; 18528/222204 objects degraded (8.338%); 9261/222204 objects misplaced (4.168%) 2024-09-21T12:36:40.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:40 smithi114 ceph-mon[30324]: 6.18 scrub starts 2024-09-21T12:36:40.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:40 smithi114 ceph-mon[30324]: 6.18 scrub ok 2024-09-21T12:36:40.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:40 smithi114 ceph-mon[30324]: 6.12 scrub starts 2024-09-21T12:36:40.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:40 smithi114 ceph-mon[30324]: 6.12 scrub ok 2024-09-21T12:36:40.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:40 smithi114 ceph-mon[30324]: osdmap e429: 8 total, 7 up, 8 in 2024-09-21T12:36:40.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:40 smithi114 ceph-mon[30324]: 1.13 scrub starts 2024-09-21T12:36:40.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:40 smithi114 ceph-mon[30324]: 1.13 scrub ok 2024-09-21T12:36:40.548 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:40.579 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:40.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:40.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:40.681 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:40.711 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:40.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:40.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:40.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:40 smithi183 ceph-mon[33277]: pgmap v834: 178 pgs: 1 unknown, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.76k op/s; 18528/222204 objects degraded (8.338%); 9261/222204 objects misplaced (4.168%) 2024-09-21T12:36:40.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:40 smithi183 ceph-mon[33277]: 6.18 scrub starts 2024-09-21T12:36:40.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:40 smithi183 ceph-mon[33277]: 6.18 scrub ok 2024-09-21T12:36:40.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:40 smithi183 ceph-mon[33277]: 6.12 scrub starts 2024-09-21T12:36:40.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:40 smithi183 ceph-mon[33277]: 6.12 scrub ok 2024-09-21T12:36:40.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:40 smithi183 ceph-mon[33277]: osdmap e429: 8 total, 7 up, 8 in 2024-09-21T12:36:40.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:40 smithi183 ceph-mon[33277]: 1.13 scrub starts 2024-09-21T12:36:40.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:40 smithi183 ceph-mon[33277]: 1.13 scrub ok 2024-09-21T12:36:40.813 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:40.844 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:40.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:40.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:40.946 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:40.977 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:40.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:40.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:41.079 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:41.103 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:36:41.105 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:41.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:41.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:41.166 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:36:41.206 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:41.242 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:41.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:41.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:41.344 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:41.384 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:41.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:41.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:41.486 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:41.505 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 48 16 80267 80251 104.482 97.375 0.0156133 0.00956646 2024-09-21T12:36:41.522 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:41.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:41.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:41.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:41 smithi114 ceph-mon[40867]: 3.6 scrub starts 2024-09-21T12:36:41.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:41 smithi114 ceph-mon[40867]: 3.6 scrub ok 2024-09-21T12:36:41.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:41 smithi114 ceph-mon[40867]: osdmap e430: 8 total, 7 up, 8 in 2024-09-21T12:36:41.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:41 smithi114 ceph-mon[30324]: 3.6 scrub starts 2024-09-21T12:36:41.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:41 smithi114 ceph-mon[30324]: 3.6 scrub ok 2024-09-21T12:36:41.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:41 smithi114 ceph-mon[30324]: osdmap e430: 8 total, 7 up, 8 in 2024-09-21T12:36:41.534 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T12:36:41.624 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:41.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:41.665 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:41.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:41.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:41 smithi183 ceph-mon[33277]: 3.6 scrub starts 2024-09-21T12:36:41.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:41 smithi183 ceph-mon[33277]: 3.6 scrub ok 2024-09-21T12:36:41.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:41 smithi183 ceph-mon[33277]: osdmap e430: 8 total, 7 up, 8 in 2024-09-21T12:36:41.766 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:41.796 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:41.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:41.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:41.898 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:41.938 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:41.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:41.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:42.041 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:42.073 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:42.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:42.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:42.126 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:36:42.126 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:36:42.126 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:36:42.126 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:36:42.126 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:36:42.174 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:42.209 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:42.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:42.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:42.311 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:42.353 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:42.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:42.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:42.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:42 smithi114 ceph-mon[30324]: pgmap v837: 178 pgs: 1 unknown, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.77k op/s; 18528/222204 objects degraded (8.338%); 9261/222204 objects misplaced (4.168%) 2024-09-21T12:36:42.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:42 smithi114 ceph-mon[30324]: 3.9 scrub starts 2024-09-21T12:36:42.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:42 smithi114 ceph-mon[30324]: 3.9 scrub ok 2024-09-21T12:36:42.383 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:42 smithi114 ceph-mon[30324]: 4.b scrub starts 2024-09-21T12:36:42.383 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:42 smithi114 ceph-mon[30324]: 4.b scrub ok 2024-09-21T12:36:42.383 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:42 smithi114 ceph-mon[30324]: 1.3 scrub starts 2024-09-21T12:36:42.383 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:42 smithi114 ceph-mon[30324]: 1.3 scrub ok 2024-09-21T12:36:42.383 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:42 smithi114 ceph-mon[30324]: 4.11 scrub starts 2024-09-21T12:36:42.383 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:42 smithi114 ceph-mon[30324]: 4.11 scrub ok 2024-09-21T12:36:42.383 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:42 smithi114 ceph-mon[30324]: 1.1b scrub starts 2024-09-21T12:36:42.383 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:42 smithi114 ceph-mon[30324]: 1.1b scrub ok 2024-09-21T12:36:42.383 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:42 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 18528/222204 objects degraded (8.338%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:42.383 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:42 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1193512758' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:36:42.383 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:42 smithi114 ceph-mon[40867]: pgmap v837: 178 pgs: 1 unknown, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.77k op/s; 18528/222204 objects degraded (8.338%); 9261/222204 objects misplaced (4.168%) 2024-09-21T12:36:42.384 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:42 smithi114 ceph-mon[40867]: 3.9 scrub starts 2024-09-21T12:36:42.384 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:42 smithi114 ceph-mon[40867]: 3.9 scrub ok 2024-09-21T12:36:42.384 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:42 smithi114 ceph-mon[40867]: 4.b scrub starts 2024-09-21T12:36:42.384 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:42 smithi114 ceph-mon[40867]: 4.b scrub ok 2024-09-21T12:36:42.384 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:42 smithi114 ceph-mon[40867]: 1.3 scrub starts 2024-09-21T12:36:42.384 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:42 smithi114 ceph-mon[40867]: 1.3 scrub ok 2024-09-21T12:36:42.384 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:42 smithi114 ceph-mon[40867]: 4.11 scrub starts 2024-09-21T12:36:42.384 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:42 smithi114 ceph-mon[40867]: 4.11 scrub ok 2024-09-21T12:36:42.384 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:42 smithi114 ceph-mon[40867]: 1.1b scrub starts 2024-09-21T12:36:42.384 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:42 smithi114 ceph-mon[40867]: 1.1b scrub ok 2024-09-21T12:36:42.384 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:42 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 18528/222204 objects degraded (8.338%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:42.384 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:42 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1193512758' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:36:42.455 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:42.491 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:42.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:42.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:42.507 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 49 16 81688 81672 104.161 88.8125 0.00916205 0.00959644 2024-09-21T12:36:42.592 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:42.622 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:42.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:42.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:42.725 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:42.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:42 smithi183 ceph-mon[33277]: pgmap v837: 178 pgs: 1 unknown, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.77k op/s; 18528/222204 objects degraded (8.338%); 9261/222204 objects misplaced (4.168%) 2024-09-21T12:36:42.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:42 smithi183 ceph-mon[33277]: 3.9 scrub starts 2024-09-21T12:36:42.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:42 smithi183 ceph-mon[33277]: 3.9 scrub ok 2024-09-21T12:36:42.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:42 smithi183 ceph-mon[33277]: 4.b scrub starts 2024-09-21T12:36:42.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:42 smithi183 ceph-mon[33277]: 4.b scrub ok 2024-09-21T12:36:42.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:42 smithi183 ceph-mon[33277]: 1.3 scrub starts 2024-09-21T12:36:42.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:42 smithi183 ceph-mon[33277]: 1.3 scrub ok 2024-09-21T12:36:42.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:42 smithi183 ceph-mon[33277]: 4.11 scrub starts 2024-09-21T12:36:42.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:42 smithi183 ceph-mon[33277]: 4.11 scrub ok 2024-09-21T12:36:42.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:42 smithi183 ceph-mon[33277]: 1.1b scrub starts 2024-09-21T12:36:42.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:42 smithi183 ceph-mon[33277]: 1.1b scrub ok 2024-09-21T12:36:42.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:42 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 18528/222204 objects degraded (8.338%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:42.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:42 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1193512758' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:36:42.761 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:42.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:42.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:42.774 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:36:42 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:36:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:36:42.863 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:42.898 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:42.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:42.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:43.000 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:43.015 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:36:43.015 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:36:43.015 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:36:43.015 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:36:43.015 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:36:43.015 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:36:43.015 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:36:43.016 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:36:43.016 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:36:43.016 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T12:36:43.016 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:36:43.016 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:36:43.016 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:36:43.016 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:36:43.016 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:36:43.036 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:43.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:43.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:43.138 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:43.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:43.180 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:43.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:43.281 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:43.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:43.321 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:43.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:43.413 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T12:36:43.422 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:43.466 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:43.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:43.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:43.507 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 50 16 83168 83152 103.925 92.5 0.0143066 0.00961776 2024-09-21T12:36:43.568 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:43.604 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:43.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:43.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:43.706 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:43.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:43 smithi183 ceph-mon[33277]: 4.e scrub starts 2024-09-21T12:36:43.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:43 smithi183 ceph-mon[33277]: 4.e scrub ok 2024-09-21T12:36:43.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:43 smithi183 ceph-mon[33277]: 5.16 scrub starts 2024-09-21T12:36:43.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:43 smithi183 ceph-mon[33277]: 5.16 scrub ok 2024-09-21T12:36:43.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:43 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:36:43.746 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:43.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:43.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:43.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:43 smithi114 ceph-mon[40867]: 4.e scrub starts 2024-09-21T12:36:43.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:43 smithi114 ceph-mon[40867]: 4.e scrub ok 2024-09-21T12:36:43.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:43 smithi114 ceph-mon[40867]: 5.16 scrub starts 2024-09-21T12:36:43.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:43 smithi114 ceph-mon[40867]: 5.16 scrub ok 2024-09-21T12:36:43.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:43 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:36:43.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:43 smithi114 ceph-mon[30324]: 4.e scrub starts 2024-09-21T12:36:43.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:43 smithi114 ceph-mon[30324]: 4.e scrub ok 2024-09-21T12:36:43.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:43 smithi114 ceph-mon[30324]: 5.16 scrub starts 2024-09-21T12:36:43.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:43 smithi114 ceph-mon[30324]: 5.16 scrub ok 2024-09-21T12:36:43.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:43 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:36:43.848 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:43.873 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:43.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:43.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:43.975 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:44.009 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:44.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:44.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:44.111 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:44.150 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:44.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:44.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:44.252 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:44.286 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:44.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:44.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:44.388 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:44.414 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:44.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:44.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:44.507 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 51 16 84499 84483 103.519 83.1875 0.00618947 0.00965553 2024-09-21T12:36:44.517 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:44.553 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:44.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:44.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:44.655 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:44.676 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:44.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:44.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:44.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:44 smithi183 ceph-mon[33277]: pgmap v838: 178 pgs: 1 unknown, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 4.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 966 B/s rd, 87 MiB/s wr, 1.39k op/s; 19479/235002 objects degraded (8.289%); 10233/235002 objects misplaced (4.354%) 2024-09-21T12:36:44.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:44 smithi183 ceph-mon[33277]: 4.1a scrub starts 2024-09-21T12:36:44.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:44 smithi183 ceph-mon[33277]: 4.1a scrub ok 2024-09-21T12:36:44.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:44 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:36:44.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:44 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:36:44.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:44 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:36:44.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:44 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:36:44.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:44 smithi183 ceph-mon[33277]: 1.4 scrub starts 2024-09-21T12:36:44.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:44 smithi183 ceph-mon[33277]: 1.4 scrub ok 2024-09-21T12:36:44.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:44 smithi183 ceph-mon[33277]: 5.c scrub starts 2024-09-21T12:36:44.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:44 smithi183 ceph-mon[33277]: 5.c scrub ok 2024-09-21T12:36:44.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:44 smithi183 ceph-mon[33277]: 1.14 scrub starts 2024-09-21T12:36:44.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:44 smithi183 ceph-mon[33277]: 1.14 scrub ok 2024-09-21T12:36:44.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:44 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:36:44.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:44 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:36:44.740 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:44 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:36:44.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[40867]: pgmap v838: 178 pgs: 1 unknown, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 4.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 966 B/s rd, 87 MiB/s wr, 1.39k op/s; 19479/235002 objects degraded (8.289%); 10233/235002 objects misplaced (4.354%) 2024-09-21T12:36:44.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[40867]: 4.1a scrub starts 2024-09-21T12:36:44.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[40867]: 4.1a scrub ok 2024-09-21T12:36:44.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:36:44.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:36:44.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:36:44.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:36:44.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[40867]: 1.4 scrub starts 2024-09-21T12:36:44.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[40867]: 1.4 scrub ok 2024-09-21T12:36:44.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[40867]: 5.c scrub starts 2024-09-21T12:36:44.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[40867]: 5.c scrub ok 2024-09-21T12:36:44.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[40867]: 1.14 scrub starts 2024-09-21T12:36:44.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[40867]: 1.14 scrub ok 2024-09-21T12:36:44.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:36:44.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:36:44.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:36:44.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[30324]: pgmap v838: 178 pgs: 1 unknown, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 4.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 966 B/s rd, 87 MiB/s wr, 1.39k op/s; 19479/235002 objects degraded (8.289%); 10233/235002 objects misplaced (4.354%) 2024-09-21T12:36:44.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[30324]: 4.1a scrub starts 2024-09-21T12:36:44.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[30324]: 4.1a scrub ok 2024-09-21T12:36:44.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:36:44.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:36:44.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:36:44.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:36:44.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[30324]: 1.4 scrub starts 2024-09-21T12:36:44.777 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[30324]: 1.4 scrub ok 2024-09-21T12:36:44.777 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[30324]: 5.c scrub starts 2024-09-21T12:36:44.777 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[30324]: 5.c scrub ok 2024-09-21T12:36:44.777 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[30324]: 1.14 scrub starts 2024-09-21T12:36:44.777 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[30324]: 1.14 scrub ok 2024-09-21T12:36:44.777 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:36:44.777 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:36:44.777 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:44 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:36:44.777 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:44.806 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:44.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:44.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:44.908 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:44.929 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:44.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:44.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:45.031 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:45.061 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:45.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:45.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:45.163 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:45.184 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:45.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:45.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:45.286 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:45.319 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:45.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:45.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:45.421 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:45.455 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:45.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:45.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:45.507 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 52 16 85886 85870 103.195 86.6875 0.0050274 0.00968578 2024-09-21T12:36:45.556 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:45.597 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:45.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:45.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:45.699 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:45.729 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:45.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:45.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:45.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:45 smithi183 ceph-mon[33277]: 4.5 scrub starts 2024-09-21T12:36:45.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:45 smithi183 ceph-mon[33277]: 4.5 scrub ok 2024-09-21T12:36:45.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:45 smithi183 ceph-mon[33277]: pgmap v839: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 914 B/s rd, 89 MiB/s wr, 1.43k op/s; 20403/244794 objects degraded (8.335%); 10233/244794 objects misplaced (4.180%) 2024-09-21T12:36:45.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:45 smithi183 ceph-mon[33277]: pgmap v840: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 963 B/s rd, 89 MiB/s wr, 1.42k op/s; 20403/244794 objects degraded (8.335%); 10233/244794 objects misplaced (4.180%) 2024-09-21T12:36:45.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:45 smithi183 ceph-mon[33277]: osdmap e431: 8 total, 7 up, 8 in 2024-09-21T12:36:45.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:45 smithi183 ceph-mon[33277]: 3.19 scrub starts 2024-09-21T12:36:45.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:45 smithi183 ceph-mon[33277]: 3.19 scrub ok 2024-09-21T12:36:45.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:45 smithi114 ceph-mon[30324]: 4.5 scrub starts 2024-09-21T12:36:45.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:45 smithi114 ceph-mon[30324]: 4.5 scrub ok 2024-09-21T12:36:45.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:45 smithi114 ceph-mon[30324]: pgmap v839: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 914 B/s rd, 89 MiB/s wr, 1.43k op/s; 20403/244794 objects degraded (8.335%); 10233/244794 objects misplaced (4.180%) 2024-09-21T12:36:45.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:45 smithi114 ceph-mon[30324]: pgmap v840: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 963 B/s rd, 89 MiB/s wr, 1.42k op/s; 20403/244794 objects degraded (8.335%); 10233/244794 objects misplaced (4.180%) 2024-09-21T12:36:45.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:45 smithi114 ceph-mon[30324]: osdmap e431: 8 total, 7 up, 8 in 2024-09-21T12:36:45.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:45 smithi114 ceph-mon[30324]: 3.19 scrub starts 2024-09-21T12:36:45.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:45 smithi114 ceph-mon[30324]: 3.19 scrub ok 2024-09-21T12:36:45.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:45 smithi114 ceph-mon[40867]: 4.5 scrub starts 2024-09-21T12:36:45.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:45 smithi114 ceph-mon[40867]: 4.5 scrub ok 2024-09-21T12:36:45.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:45 smithi114 ceph-mon[40867]: pgmap v839: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 914 B/s rd, 89 MiB/s wr, 1.43k op/s; 20403/244794 objects degraded (8.335%); 10233/244794 objects misplaced (4.180%) 2024-09-21T12:36:45.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:45 smithi114 ceph-mon[40867]: pgmap v840: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 963 B/s rd, 89 MiB/s wr, 1.42k op/s; 20403/244794 objects degraded (8.335%); 10233/244794 objects misplaced (4.180%) 2024-09-21T12:36:45.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:45 smithi114 ceph-mon[40867]: osdmap e431: 8 total, 7 up, 8 in 2024-09-21T12:36:45.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:45 smithi114 ceph-mon[40867]: 3.19 scrub starts 2024-09-21T12:36:45.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:45 smithi114 ceph-mon[40867]: 3.19 scrub ok 2024-09-21T12:36:45.831 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:45.861 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:45.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:45.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:45.964 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:45.996 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:45.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:45.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:46.006 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 4, 7] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-09-21T12:36:46.006 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:36:46.007 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-21T12:36:46.007 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:36:46.098 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:46.133 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:46.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:46.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:46.204 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T12:36:46.204 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T12:36:46.222 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:36:46.235 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:46.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:46.276 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:46.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:46.377 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:46.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:46.415 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:46.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:46.508 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 53 16 87406 87390 103.04 95 0.00560098 0.00970057 2024-09-21T12:36:46.516 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:46.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:46.561 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:46.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:46.662 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:46.695 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:46.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:46.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:46.703 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:46 smithi114 ceph-mon[30324]: 6.1a scrub starts 2024-09-21T12:36:46.703 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:46 smithi114 ceph-mon[30324]: 6.1a scrub ok 2024-09-21T12:36:46.703 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:46 smithi114 ceph-mon[30324]: 5.3 scrub starts 2024-09-21T12:36:46.703 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:46 smithi114 ceph-mon[30324]: 5.3 scrub ok 2024-09-21T12:36:46.703 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:46 smithi114 ceph-mon[30324]: 1.17 scrub starts 2024-09-21T12:36:46.703 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:46 smithi114 ceph-mon[30324]: 1.17 scrub ok 2024-09-21T12:36:46.703 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:46 smithi114 ceph-mon[40867]: 6.1a scrub starts 2024-09-21T12:36:46.703 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:46 smithi114 ceph-mon[40867]: 6.1a scrub ok 2024-09-21T12:36:46.704 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:46 smithi114 ceph-mon[40867]: 5.3 scrub starts 2024-09-21T12:36:46.704 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:46 smithi114 ceph-mon[40867]: 5.3 scrub ok 2024-09-21T12:36:46.704 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:46 smithi114 ceph-mon[40867]: 1.17 scrub starts 2024-09-21T12:36:46.704 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:46 smithi114 ceph-mon[40867]: 1.17 scrub ok 2024-09-21T12:36:46.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:46 smithi183 ceph-mon[33277]: 6.1a scrub starts 2024-09-21T12:36:46.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:46 smithi183 ceph-mon[33277]: 6.1a scrub ok 2024-09-21T12:36:46.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:46 smithi183 ceph-mon[33277]: 5.3 scrub starts 2024-09-21T12:36:46.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:46 smithi183 ceph-mon[33277]: 5.3 scrub ok 2024-09-21T12:36:46.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:46 smithi183 ceph-mon[33277]: 1.17 scrub starts 2024-09-21T12:36:46.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:46 smithi183 ceph-mon[33277]: 1.17 scrub ok 2024-09-21T12:36:46.797 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:46.840 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:36:46.841 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:46.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:46.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:46.943 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:46.978 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:46.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:46.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:47.080 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:47.121 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:47.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:47.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:47.223 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:47.254 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:47.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:47.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:47.356 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:47.384 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:47.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:47.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:47.453 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:36:47.485 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:47.506 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:36:47.507 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:36:47.507 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 54 16 88815 88799 102.762 88.0625 0.0143543 0.00972665 2024-09-21T12:36:47.537 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:47.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:47.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:47.587 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:47 smithi114 ceph-mon[30324]: pgmap v842: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 916 B/s rd, 84 MiB/s wr, 1.35k op/s; 20403/244794 objects degraded (8.335%); 10233/244794 objects misplaced (4.180%) 2024-09-21T12:36:47.587 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:47 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 20403/244794 objects degraded (8.335%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:47.587 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:47 smithi114 ceph-mon[30324]: osdmap e432: 8 total, 7 up, 8 in 2024-09-21T12:36:47.587 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:47 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/257988649' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:36:47.587 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:47 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:36:47.587 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:47 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:36:47.587 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:47 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:36:47.588 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:47 smithi114 ceph-mon[40867]: pgmap v842: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 916 B/s rd, 84 MiB/s wr, 1.35k op/s; 20403/244794 objects degraded (8.335%); 10233/244794 objects misplaced (4.180%) 2024-09-21T12:36:47.588 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:47 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 20403/244794 objects degraded (8.335%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:47.588 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:47 smithi114 ceph-mon[40867]: osdmap e432: 8 total, 7 up, 8 in 2024-09-21T12:36:47.588 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:47 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/257988649' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:36:47.588 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:47 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:36:47.588 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:47 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:36:47.588 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:47 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:36:47.639 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:47.670 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:47.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:47.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:47.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:36:47 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:36:47.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:47 smithi183 ceph-mon[33277]: pgmap v842: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 916 B/s rd, 84 MiB/s wr, 1.35k op/s; 20403/244794 objects degraded (8.335%); 10233/244794 objects misplaced (4.180%) 2024-09-21T12:36:47.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:47 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 20403/244794 objects degraded (8.335%), 13 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:47.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:47 smithi183 ceph-mon[33277]: osdmap e432: 8 total, 7 up, 8 in 2024-09-21T12:36:47.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:47 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/257988649' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:36:47.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:47 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:36:47.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:47 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:36:47.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:47 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:36:47.772 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:47.803 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:47.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:47.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:47.904 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:47.944 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:47.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:47.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:48.045 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:48.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:48.079 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:48.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:48.142 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-21T12:36:48.180 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:48.205 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:36:48.206 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:48.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:48.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:48.308 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:48.340 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:48.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:48.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:48.442 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:48.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:48.485 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:48.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:48.507 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 55 16 90398 90382 102.693 98.9375 0.00678038 0.00973326 2024-09-21T12:36:48.586 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:48.593 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T12:36:48.618 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:48.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:48.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:48.720 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:48.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:48 smithi183 ceph-mon[33277]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:36:48.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:48 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:36:48.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:48 smithi183 ceph-mon[33277]: osdmap e433: 8 total, 7 up, 8 in 2024-09-21T12:36:48.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:48.759 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:48.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:48.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:48 smithi114 ceph-mon[30324]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:36:48.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:48 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:36:48.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:48 smithi114 ceph-mon[30324]: osdmap e433: 8 total, 7 up, 8 in 2024-09-21T12:36:48.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:48 smithi114 ceph-mon[40867]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:36:48.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:48 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:36:48.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:48 smithi114 ceph-mon[40867]: osdmap e433: 8 total, 7 up, 8 in 2024-09-21T12:36:48.861 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:48.886 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:48.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:48.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:48.988 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:49.020 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:49.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:49.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:49.122 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:49.161 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:49.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:49.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:49.186 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:36:49.189 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:36:49.263 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:49.299 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:49.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:49.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:49.401 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:49.433 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:49.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:49.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:49.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:49 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:36:49.508 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 56 16 91837 91821 102.465 89.9375 0.00599657 0.00975502 2024-09-21T12:36:49.534 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:49.572 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:49.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:49.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:49.674 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:49.705 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:49.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:49.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:49.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:49 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:36:49.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:49 smithi183 ceph-mon[33277]: pgmap v844: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 963 B/s rd, 117 MiB/s wr, 1.88k op/s; 21730/264942 objects degraded (8.202%); 11162/264942 objects misplaced (4.213%) 2024-09-21T12:36:49.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:49 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:36:49.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:49 smithi114 ceph-mon[40867]: pgmap v844: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 963 B/s rd, 117 MiB/s wr, 1.88k op/s; 21730/264942 objects degraded (8.202%); 11162/264942 objects misplaced (4.213%) 2024-09-21T12:36:49.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:49 smithi114 ceph-mon[30324]: pgmap v844: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 963 B/s rd, 117 MiB/s wr, 1.88k op/s; 21730/264942 objects degraded (8.202%); 11162/264942 objects misplaced (4.213%) 2024-09-21T12:36:49.807 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:49.834 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:49.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:49.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:49.936 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:49.967 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:49.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:49.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:50.069 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:50.110 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:50.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:50.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:50.212 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:50.247 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:50.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:50.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:50.349 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:50.384 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:50.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:50.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:50.486 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:50.508 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 57 16 93460 93444 102.447 101.438 0.00523337 0.00975676 2024-09-21T12:36:50.517 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:50.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:50.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:50.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:50 smithi114 ceph-mon[40867]: from='client.26228 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:36:50.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:50 smithi114 ceph-mon[30324]: from='client.26228 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:36:50.543 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:36:50.579 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:36:50.604 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":845,"stamp":"2024-09-21T12:36:48.136050+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5775327981,"num_objects":88314,"num_object_clones":0,"num_object_copies":264942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21730,"num_objects_misplaced":11162,"num_objects_unfound":0,"num_objects_dirty":88314,"num_whiteouts":0,"num_read":1177,"num_read_kb":933,"num_write":88842,"num_write_kb":5639479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4699,"num_bytes_recovered":275092355,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67850,"ondisk_log_size":67850,"up":481,"acting":501,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":556,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":23141624,"kb_used_data":17641584,"kb_used_omap":12,"kb_used_meta":5499891,"kb_avail":726819592,"statfs":{"total":767960285184,"available":744263262208,"internally_reserved":0,"allocated":18064982016,"data_stored":18061098635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":5631888944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":440139776,"num_objects":6716,"num_object_clones":0,"num_object_copies":20148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1327,"num_objects_misplaced":929,"num_objects_unfound":0,"num_objects_dirty":6716,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6716,"num_write_kb":429824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.265744"},"pg_stats":[{"pgid":"1.1d","version":"76'79","reported_seq":485,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.422662+0000","last_change":"2024-09-21T12:32:38.427111+0000","last_active":"2024-09-21T12:36:44.422662+0000","last_peered":"2024-09-21T12:36:44.422662+0000","last_clean":"2024-09-21T12:36:44.422662+0000","last_became_active":"2024-09-21T12:32:38.426383+0000","last_became_peered":"2024-09-21T12:32:38.426383+0000","last_unstale":"2024-09-21T12:36:44.422662+0000","last_undegraded":"2024-09-21T12:36:44.422662+0000","last_fullsized":"2024-09-21T12:36:44.422662+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:32.060116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:32.060116+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:14.142065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":797,"reported_epoch":430,"state":"active+undersized","last_fresh":"2024-09-21T12:36:40.300245+0000","last_change":"2024-09-21T12:34:34.340741+0000","last_active":"2024-09-21T12:36:40.300245+0000","last_peered":"2024-09-21T12:36:40.300245+0000","last_clean":"2024-09-21T12:34:29.289357+0000","last_became_active":"2024-09-21T12:34:34.340741+0000","last_became_peered":"2024-09-21T12:34:34.340741+0000","last_unstale":"2024-09-21T12:36:40.300245+0000","last_undegraded":"2024-09-21T12:36:40.300245+0000","last_fullsized":"2024-09-21T12:34:34.332209+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:29.086830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:29.086830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:31.924550+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":185,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.419648+0000","last_change":"2024-09-21T12:32:38.443995+0000","last_active":"2024-09-21T12:36:44.419648+0000","last_peered":"2024-09-21T12:36:44.419648+0000","last_clean":"2024-09-21T12:36:44.419648+0000","last_became_active":"2024-09-21T12:32:38.419081+0000","last_became_peered":"2024-09-21T12:32:38.419081+0000","last_unstale":"2024-09-21T12:36:44.419648+0000","last_undegraded":"2024-09-21T12:36:44.419648+0000","last_fullsized":"2024-09-21T12:36:44.419648+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:29:08.671723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:29:08.671723+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:18.917504+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":797,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.420478+0000","last_change":"2024-09-21T12:36:00.136989+0000","last_active":"2024-09-21T12:36:44.420478+0000","last_peered":"2024-09-21T12:36:44.420478+0000","last_clean":"2024-09-21T12:36:44.420478+0000","last_became_active":"2024-09-21T12:36:00.136673+0000","last_became_peered":"2024-09-21T12:36:00.136673+0000","last_unstale":"2024-09-21T12:36:44.420478+0000","last_undegraded":"2024-09-21T12:36:44.420478+0000","last_fullsized":"2024-09-21T12:36:44.420478+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:54.969048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:35:54.969048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:20.733223+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":191,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.729701+0000","last_change":"2024-09-21T12:36:44.729701+0000","last_active":"2024-09-21T12:36:44.729701+0000","last_peered":"2024-09-21T12:36:44.729701+0000","last_clean":"2024-09-21T12:36:44.729701+0000","last_became_active":"2024-09-21T12:32:38.425588+0000","last_became_peered":"2024-09-21T12:32:38.425588+0000","last_unstale":"2024-09-21T12:36:44.729701+0000","last_undegraded":"2024-09-21T12:36:44.729701+0000","last_fullsized":"2024-09-21T12:36:44.729701+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:44.729655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:44.729655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:58.553964+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"76'79","reported_seq":44,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.424073+0000","last_change":"2024-09-21T12:36:00.154588+0000","last_active":"2024-09-21T12:36:44.424073+0000","last_peered":"2024-09-21T12:36:44.424073+0000","last_clean":"2024-09-21T12:36:44.424073+0000","last_became_active":"2024-09-21T12:36:00.154209+0000","last_became_peered":"2024-09-21T12:36:00.154209+0000","last_unstale":"2024-09-21T12:36:44.424073+0000","last_undegraded":"2024-09-21T12:36:44.424073+0000","last_fullsized":"2024-09-21T12:36:44.424073+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:35:54.010909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:35:54.010909+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:12.176158+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":787,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.119087+0000","last_change":"2024-09-21T12:36:00.139483+0000","last_active":"2024-09-21T12:36:47.119087+0000","last_peered":"2024-09-21T12:36:47.119087+0000","last_clean":"2024-09-21T12:36:47.119087+0000","last_became_active":"2024-09-21T12:36:00.139167+0000","last_became_peered":"2024-09-21T12:36:00.139167+0000","last_unstale":"2024-09-21T12:36:47.119087+0000","last_undegraded":"2024-09-21T12:36:47.119087+0000","last_fullsized":"2024-09-21T12:36:47.119087+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:05.904553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:34:05.904553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":799,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T12:36:47.121129+0000","last_change":"2024-09-21T12:34:34.346389+0000","last_active":"2024-09-21T12:36:47.121129+0000","last_peered":"2024-09-21T12:36:47.121129+0000","last_clean":"2024-09-21T12:34:29.294384+0000","last_became_active":"2024-09-21T12:34:34.346389+0000","last_became_peered":"2024-09-21T12:34:34.346389+0000","last_unstale":"2024-09-21T12:36:47.121129+0000","last_undegraded":"2024-09-21T12:36:47.121129+0000","last_fullsized":"2024-09-21T12:34:34.336359+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:06.698999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:06.698999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:34.545818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":836,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:47.119283+0000","last_change":"2024-09-21T12:34:34.337065+0000","last_active":"2024-09-21T12:36:47.119283+0000","last_peered":"2024-09-21T12:36:47.119283+0000","last_clean":"2024-09-21T12:34:29.291199+0000","last_became_active":"2024-09-21T12:34:34.337065+0000","last_became_peered":"2024-09-21T12:34:34.337065+0000","last_unstale":"2024-09-21T12:36:47.119283+0000","last_undegraded":"2024-09-21T12:34:34.335679+0000","last_fullsized":"2024-09-21T12:34:34.335428+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:31:48.294300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:48.294300+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:39.494442+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":183,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.419549+0000","last_change":"2024-09-21T12:36:35.622295+0000","last_active":"2024-09-21T12:36:44.419549+0000","last_peered":"2024-09-21T12:36:44.419549+0000","last_clean":"2024-09-21T12:36:44.419549+0000","last_became_active":"2024-09-21T12:32:38.417724+0000","last_became_peered":"2024-09-21T12:32:38.417724+0000","last_unstale":"2024-09-21T12:36:44.419549+0000","last_undegraded":"2024-09-21T12:36:44.419549+0000","last_fullsized":"2024-09-21T12:36:44.419549+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:35.622232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:35.622232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:53.247230+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":183,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.421331+0000","last_change":"2024-09-21T12:33:32.373658+0000","last_active":"2024-09-21T12:36:44.421331+0000","last_peered":"2024-09-21T12:36:44.421331+0000","last_clean":"2024-09-21T12:36:44.421331+0000","last_became_active":"2024-09-21T12:32:38.428077+0000","last_became_peered":"2024-09-21T12:32:38.428077+0000","last_unstale":"2024-09-21T12:36:44.421331+0000","last_undegraded":"2024-09-21T12:36:44.421331+0000","last_fullsized":"2024-09-21T12:36:44.421331+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:32.372832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:32.372832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":201,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.420483+0000","last_change":"2024-09-21T12:36:42.653161+0000","last_active":"2024-09-21T12:36:44.420483+0000","last_peered":"2024-09-21T12:36:44.420483+0000","last_clean":"2024-09-21T12:36:44.420483+0000","last_became_active":"2024-09-21T12:32:38.426121+0000","last_became_peered":"2024-09-21T12:32:38.426121+0000","last_unstale":"2024-09-21T12:36:44.420483+0000","last_undegraded":"2024-09-21T12:36:44.420483+0000","last_fullsized":"2024-09-21T12:36:44.420483+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:36:42.653075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:42.653075+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:44.149637+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":836,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.427936+0000","last_change":"2024-09-21T12:36:00.138341+0000","last_active":"2024-09-21T12:36:44.427936+0000","last_peered":"2024-09-21T12:36:44.427936+0000","last_clean":"2024-09-21T12:36:44.427936+0000","last_became_active":"2024-09-21T12:36:00.137364+0000","last_became_peered":"2024-09-21T12:36:00.137364+0000","last_unstale":"2024-09-21T12:36:44.427936+0000","last_undegraded":"2024-09-21T12:36:44.427936+0000","last_fullsized":"2024-09-21T12:36:44.427936+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:13.198376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:13.198376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:24.428313+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":801,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-21T12:36:40.295454+0000","last_change":"2024-09-21T12:36:38.752470+0000","last_active":"2024-09-21T12:36:40.295454+0000","last_peered":"2024-09-21T12:36:40.295454+0000","last_clean":"2024-09-21T12:36:40.295454+0000","last_became_active":"2024-09-21T12:36:00.134723+0000","last_became_peered":"2024-09-21T12:36:00.134723+0000","last_unstale":"2024-09-21T12:36:40.295454+0000","last_undegraded":"2024-09-21T12:36:40.295454+0000","last_fullsized":"2024-09-21T12:36:40.295454+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:38.752410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:38.752410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:46.920310+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"76'79","reported_seq":572,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.120925+0000","last_change":"2024-09-21T12:36:00.148161+0000","last_active":"2024-09-21T12:36:47.120925+0000","last_peered":"2024-09-21T12:36:47.120925+0000","last_clean":"2024-09-21T12:36:47.120925+0000","last_became_active":"2024-09-21T12:36:00.147795+0000","last_became_peered":"2024-09-21T12:36:00.147795+0000","last_unstale":"2024-09-21T12:36:47.120925+0000","last_undegraded":"2024-09-21T12:36:47.120925+0000","last_fullsized":"2024-09-21T12:36:47.120925+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:36.752562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:36.752562+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":183,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.420654+0000","last_change":"2024-09-21T12:35:14.244213+0000","last_active":"2024-09-21T12:36:44.420654+0000","last_peered":"2024-09-21T12:36:44.420654+0000","last_clean":"2024-09-21T12:36:44.420654+0000","last_became_active":"2024-09-21T12:32:38.426142+0000","last_became_peered":"2024-09-21T12:32:38.426142+0000","last_unstale":"2024-09-21T12:36:44.420654+0000","last_undegraded":"2024-09-21T12:36:44.420654+0000","last_fullsized":"2024-09-21T12:36:44.420654+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:14.243954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:14.243954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":189,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.420202+0000","last_change":"2024-09-21T12:33:26.395059+0000","last_active":"2024-09-21T12:36:44.420202+0000","last_peered":"2024-09-21T12:36:44.420202+0000","last_clean":"2024-09-21T12:36:44.420202+0000","last_became_active":"2024-09-21T12:32:38.422912+0000","last_became_peered":"2024-09-21T12:32:38.422912+0000","last_unstale":"2024-09-21T12:36:44.420202+0000","last_undegraded":"2024-09-21T12:36:44.420202+0000","last_fullsized":"2024-09-21T12:36:44.420202+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'7","last_scrub_stamp":"2024-09-21T12:33:26.394982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:26.394982+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":44,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.424916+0000","last_change":"2024-09-21T12:36:00.143221+0000","last_active":"2024-09-21T12:36:44.424916+0000","last_peered":"2024-09-21T12:36:44.424916+0000","last_clean":"2024-09-21T12:36:44.424916+0000","last_became_active":"2024-09-21T12:36:00.142106+0000","last_became_peered":"2024-09-21T12:36:00.142106+0000","last_unstale":"2024-09-21T12:36:44.424916+0000","last_undegraded":"2024-09-21T12:36:44.424916+0000","last_fullsized":"2024-09-21T12:36:44.424916+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:09.558418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:35:09.558418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":796,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.427655+0000","last_change":"2024-09-21T12:33:36.125346+0000","last_active":"2024-09-21T12:36:44.427655+0000","last_peered":"2024-09-21T12:36:44.427655+0000","last_clean":"2024-09-21T12:36:44.427655+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:36:44.427655+0000","last_undegraded":"2024-09-21T12:36:44.427655+0000","last_fullsized":"2024-09-21T12:36:44.427655+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:36.124095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:36.124095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:19.399768+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"76'79","reported_seq":620,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.120459+0000","last_change":"2024-09-21T12:34:48.959991+0000","last_active":"2024-09-21T12:36:47.120459+0000","last_peered":"2024-09-21T12:36:47.120459+0000","last_clean":"2024-09-21T12:36:47.120459+0000","last_became_active":"2024-09-21T12:27:16.957569+0000","last_became_peered":"2024-09-21T12:27:16.957569+0000","last_unstale":"2024-09-21T12:36:47.120459+0000","last_undegraded":"2024-09-21T12:36:47.120459+0000","last_fullsized":"2024-09-21T12:36:47.120459+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:48.959916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:48.959916+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":184,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.421513+0000","last_change":"2024-09-21T12:34:05.193021+0000","last_active":"2024-09-21T12:36:44.421513+0000","last_peered":"2024-09-21T12:36:44.421513+0000","last_clean":"2024-09-21T12:36:44.421513+0000","last_became_active":"2024-09-21T12:32:38.428997+0000","last_became_peered":"2024-09-21T12:32:38.428997+0000","last_unstale":"2024-09-21T12:36:44.421513+0000","last_undegraded":"2024-09-21T12:36:44.421513+0000","last_fullsized":"2024-09-21T12:36:44.421513+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:05.192971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:34:05.192971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":856,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-21T12:36:40.296852+0000","last_change":"2024-09-21T12:36:00.127510+0000","last_active":"2024-09-21T12:36:40.296852+0000","last_peered":"2024-09-21T12:36:40.296852+0000","last_clean":"2024-09-21T12:36:40.296852+0000","last_became_active":"2024-09-21T12:36:00.127153+0000","last_became_peered":"2024-09-21T12:36:00.127153+0000","last_unstale":"2024-09-21T12:36:40.296852+0000","last_undegraded":"2024-09-21T12:36:40.296852+0000","last_fullsized":"2024-09-21T12:36:40.296852+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:56.016840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:56.016840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:07.001046+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":840,"reported_epoch":431,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:44.428033+0000","last_change":"2024-09-21T12:34:34.341728+0000","last_active":"2024-09-21T12:36:44.428033+0000","last_peered":"2024-09-21T12:36:44.428033+0000","last_clean":"2024-09-21T12:34:29.305282+0000","last_became_active":"2024-09-21T12:34:34.341728+0000","last_became_peered":"2024-09-21T12:34:34.341728+0000","last_unstale":"2024-09-21T12:36:44.428033+0000","last_undegraded":"2024-09-21T12:34:34.336150+0000","last_fullsized":"2024-09-21T12:34:34.336049+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:33:01.668584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:01.668584+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:47.258634+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":819,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.422918+0000","last_change":"2024-09-21T12:35:50.957724+0000","last_active":"2024-09-21T12:36:44.422918+0000","last_peered":"2024-09-21T12:36:44.422918+0000","last_clean":"2024-09-21T12:36:44.422918+0000","last_became_active":"2024-09-21T12:32:38.425958+0000","last_became_peered":"2024-09-21T12:32:38.425958+0000","last_unstale":"2024-09-21T12:36:44.422918+0000","last_undegraded":"2024-09-21T12:36:44.422918+0000","last_fullsized":"2024-09-21T12:36:44.422918+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:50.957677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:35:50.957677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:02.566364+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"76'79","reported_seq":747,"reported_epoch":431,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:44.427663+0000","last_change":"2024-09-21T12:36:01.149228+0000","last_active":"2024-09-21T12:36:44.427663+0000","last_peered":"2024-09-21T12:36:44.427663+0000","last_clean":"2024-09-21T12:36:44.427663+0000","last_became_active":"2024-09-21T12:36:01.148686+0000","last_became_peered":"2024-09-21T12:36:01.148686+0000","last_unstale":"2024-09-21T12:36:44.427663+0000","last_undegraded":"2024-09-21T12:36:44.427663+0000","last_fullsized":"2024-09-21T12:36:44.427663+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:27.294674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:27.294674+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:14.322523+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7,1],"avail_no_missing":["5","1","7"],"object_location_counts":[{"shards":"1,5,7","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":803,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.119923+0000","last_change":"2024-09-21T12:35:06.944278+0000","last_active":"2024-09-21T12:36:47.119923+0000","last_peered":"2024-09-21T12:36:47.119923+0000","last_clean":"2024-09-21T12:36:47.119923+0000","last_became_active":"2024-09-21T12:32:02.033276+0000","last_became_peered":"2024-09-21T12:32:02.033276+0000","last_unstale":"2024-09-21T12:36:47.119923+0000","last_undegraded":"2024-09-21T12:36:47.119923+0000","last_fullsized":"2024-09-21T12:36:47.119923+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:06.944209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:06.944209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":179,"reported_epoch":431,"state":"active+undersized","last_fresh":"2024-09-21T12:36:44.420010+0000","last_change":"2024-09-21T12:34:34.349252+0000","last_active":"2024-09-21T12:36:44.420010+0000","last_peered":"2024-09-21T12:36:44.420010+0000","last_clean":"2024-09-21T12:34:29.278130+0000","last_became_active":"2024-09-21T12:34:34.349252+0000","last_became_peered":"2024-09-21T12:34:34.349252+0000","last_unstale":"2024-09-21T12:36:44.420010+0000","last_undegraded":"2024-09-21T12:36:44.420010+0000","last_fullsized":"2024-09-21T12:34:34.329566+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:45.725968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.725968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:14.888145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":897,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.119448+0000","last_change":"2024-09-21T12:35:59.641966+0000","last_active":"2024-09-21T12:36:47.119448+0000","last_peered":"2024-09-21T12:36:47.119448+0000","last_clean":"2024-09-21T12:36:47.119448+0000","last_became_active":"2024-09-21T12:32:38.409163+0000","last_became_peered":"2024-09-21T12:32:38.409163+0000","last_unstale":"2024-09-21T12:36:47.119448+0000","last_undegraded":"2024-09-21T12:36:47.119448+0000","last_fullsized":"2024-09-21T12:36:47.119448+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T12:35:59.641895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:59.641895+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:19.348495+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":187,"reported_epoch":431,"state":"active+undersized","last_fresh":"2024-09-21T12:36:44.420266+0000","last_change":"2024-09-21T12:34:34.349550+0000","last_active":"2024-09-21T12:36:44.420266+0000","last_peered":"2024-09-21T12:36:44.420266+0000","last_clean":"2024-09-21T12:34:29.281006+0000","last_became_active":"2024-09-21T12:34:34.349550+0000","last_became_peered":"2024-09-21T12:34:34.349550+0000","last_unstale":"2024-09-21T12:36:44.420266+0000","last_undegraded":"2024-09-21T12:36:44.420266+0000","last_fullsized":"2024-09-21T12:34:34.331038+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:53.270634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:32:53.270634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:37.640748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"76'79","reported_seq":542,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.427841+0000","last_change":"2024-09-21T12:36:41.538607+0000","last_active":"2024-09-21T12:36:44.427841+0000","last_peered":"2024-09-21T12:36:44.427841+0000","last_clean":"2024-09-21T12:36:44.427841+0000","last_became_active":"2024-09-21T12:32:38.414204+0000","last_became_peered":"2024-09-21T12:32:38.414204+0000","last_unstale":"2024-09-21T12:36:44.427841+0000","last_undegraded":"2024-09-21T12:36:44.427841+0000","last_fullsized":"2024-09-21T12:36:44.427841+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:41.538559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:41.538559+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:06.792208+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":770,"reported_epoch":431,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:44.423956+0000","last_change":"2024-09-21T12:36:37.839177+0000","last_active":"2024-09-21T12:36:44.423956+0000","last_peered":"2024-09-21T12:36:44.423956+0000","last_clean":"2024-09-21T12:36:44.423956+0000","last_became_active":"2024-09-21T12:36:01.126897+0000","last_became_peered":"2024-09-21T12:36:01.126897+0000","last_unstale":"2024-09-21T12:36:44.423956+0000","last_undegraded":"2024-09-21T12:36:44.423956+0000","last_fullsized":"2024-09-21T12:36:44.423956+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:36:37.839024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:37.839024+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:04.241490+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5,3],"avail_no_missing":["7","3","5"],"object_location_counts":[{"shards":"3,5,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":845,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.561097+0000","last_change":"2024-09-21T12:36:44.561097+0000","last_active":"2024-09-21T12:36:44.561097+0000","last_peered":"2024-09-21T12:36:44.561097+0000","last_clean":"2024-09-21T12:36:44.561097+0000","last_became_active":"2024-09-21T12:32:38.425335+0000","last_became_peered":"2024-09-21T12:32:38.425335+0000","last_unstale":"2024-09-21T12:36:44.561097+0000","last_undegraded":"2024-09-21T12:36:44.561097+0000","last_fullsized":"2024-09-21T12:36:44.561097+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:36:44.561030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:36:44.561030+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:57.359274+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":912,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-21T12:36:40.297393+0000","last_change":"2024-09-21T12:36:00.173918+0000","last_active":"2024-09-21T12:36:40.297393+0000","last_peered":"2024-09-21T12:36:40.297393+0000","last_clean":"2024-09-21T12:36:40.297393+0000","last_became_active":"2024-09-21T12:36:00.148761+0000","last_became_peered":"2024-09-21T12:36:00.148761+0000","last_unstale":"2024-09-21T12:36:40.297393+0000","last_undegraded":"2024-09-21T12:36:40.297393+0000","last_fullsized":"2024-09-21T12:36:40.297393+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:32:57.082525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:57.082525+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:19.889528+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"415'18","reported_seq":251,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.421149+0000","last_change":"2024-09-21T12:36:00.148175+0000","last_active":"2024-09-21T12:36:44.421149+0000","last_peered":"2024-09-21T12:36:44.421149+0000","last_clean":"2024-09-21T12:36:44.421149+0000","last_became_active":"2024-09-21T12:36:00.147676+0000","last_became_peered":"2024-09-21T12:36:00.147676+0000","last_unstale":"2024-09-21T12:36:44.421149+0000","last_undegraded":"2024-09-21T12:36:44.421149+0000","last_fullsized":"2024-09-21T12:36:44.421149+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'15","last_scrub_stamp":"2024-09-21T12:35:11.182313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:35:11.182313+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"431'5444","reported_seq":7680,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:46.609391+0000","last_change":"2024-09-21T12:36:13.108581+0000","last_active":"2024-09-21T12:36:46.609391+0000","last_peered":"2024-09-21T12:36:46.609391+0000","last_clean":"2024-09-21T12:36:46.609391+0000","last_became_active":"2024-09-21T12:36:13.105161+0000","last_became_peered":"2024-09-21T12:36:13.105161+0000","last_unstale":"2024-09-21T12:36:46.609391+0000","last_undegraded":"2024-09-21T12:36:46.609391+0000","last_fullsized":"2024-09-21T12:36:46.609391+0000","mapping_epoch":420,"log_start":"416'1600","ondisk_log_start":"416'1600","created":405,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3844,"log_dups_size":0,"ondisk_log_size":3844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:51.244739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356777984,"num_objects":5444,"num_object_clones":0,"num_object_copies":16332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5444,"num_write_kb":348416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1078,"num_bytes_recovered":70647808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"76'79","reported_seq":347,"reported_epoch":431,"state":"active+undersized","last_fresh":"2024-09-21T12:36:44.427930+0000","last_change":"2024-09-21T12:34:34.347159+0000","last_active":"2024-09-21T12:36:44.427930+0000","last_peered":"2024-09-21T12:36:44.427930+0000","last_clean":"2024-09-21T12:34:29.305642+0000","last_became_active":"2024-09-21T12:34:34.347159+0000","last_became_peered":"2024-09-21T12:34:34.347159+0000","last_unstale":"2024-09-21T12:36:44.427930+0000","last_undegraded":"2024-09-21T12:36:44.427930+0000","last_fullsized":"2024-09-21T12:34:34.340605+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:18.151644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:18.151644+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:28.449536+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":451,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:47.120052+0000","last_change":"2024-09-21T12:34:34.345532+0000","last_active":"2024-09-21T12:36:47.120052+0000","last_peered":"2024-09-21T12:36:47.120052+0000","last_clean":"2024-09-21T12:34:29.295616+0000","last_became_active":"2024-09-21T12:34:34.345532+0000","last_became_peered":"2024-09-21T12:34:34.345532+0000","last_unstale":"2024-09-21T12:36:47.120052+0000","last_undegraded":"2024-09-21T12:34:34.334307+0000","last_fullsized":"2024-09-21T12:34:34.334181+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:31:56.722241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:56.722241+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":268,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.122723+0000","last_change":"2024-09-21T12:36:00.140935+0000","last_active":"2024-09-21T12:36:47.122723+0000","last_peered":"2024-09-21T12:36:47.122723+0000","last_clean":"2024-09-21T12:36:47.122723+0000","last_became_active":"2024-09-21T12:36:00.140455+0000","last_became_peered":"2024-09-21T12:36:00.140455+0000","last_unstale":"2024-09-21T12:36:47.122723+0000","last_undegraded":"2024-09-21T12:36:47.122723+0000","last_fullsized":"2024-09-21T12:36:47.122723+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:05.703464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:05.703464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:21.552824+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"352'15","reported_seq":237,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.421464+0000","last_change":"2024-09-21T12:35:50.989423+0000","last_active":"2024-09-21T12:36:44.421464+0000","last_peered":"2024-09-21T12:36:44.421464+0000","last_clean":"2024-09-21T12:36:44.421464+0000","last_became_active":"2024-09-21T12:32:38.429092+0000","last_became_peered":"2024-09-21T12:32:38.429092+0000","last_unstale":"2024-09-21T12:36:44.421464+0000","last_undegraded":"2024-09-21T12:36:44.421464+0000","last_fullsized":"2024-09-21T12:36:44.421464+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'15","last_scrub_stamp":"2024-09-21T12:35:50.989377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:35:50.989377+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:19.146860+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":280,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.422832+0000","last_change":"2024-09-21T12:35:24.989679+0000","last_active":"2024-09-21T12:36:44.422832+0000","last_peered":"2024-09-21T12:36:44.422832+0000","last_clean":"2024-09-21T12:36:44.422832+0000","last_became_active":"2024-09-21T12:31:39.054600+0000","last_became_peered":"2024-09-21T12:31:39.054600+0000","last_unstale":"2024-09-21T12:36:44.422832+0000","last_undegraded":"2024-09-21T12:36:44.422832+0000","last_fullsized":"2024-09-21T12:36:44.422832+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:35:24.989615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:24.989615+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:51.845745+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.f","version":"431'5626","reported_seq":5690,"reported_epoch":431,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:46.841980+0000","last_change":"2024-09-21T12:36:01.154474+0000","last_active":"2024-09-21T12:36:46.841980+0000","last_peered":"2024-09-21T12:36:46.841980+0000","last_clean":"2024-09-21T12:36:46.841980+0000","last_became_active":"2024-09-21T12:36:01.133058+0000","last_became_peered":"2024-09-21T12:36:01.133058+0000","last_unstale":"2024-09-21T12:36:46.841980+0000","last_undegraded":"2024-09-21T12:36:46.841980+0000","last_fullsized":"2024-09-21T12:36:46.841980+0000","mapping_epoch":415,"log_start":"416'1400","ondisk_log_start":"416'1400","created":405,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4226,"log_dups_size":0,"ondisk_log_size":4226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:53.471384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368705536,"num_objects":5626,"num_object_clones":0,"num_object_copies":16878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5626,"num_objects_unfound":0,"num_objects_dirty":5626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5626,"num_write_kb":360064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,0],"avail_no_missing":["7","0","1"],"object_location_counts":[{"shards":"0,1,7","objects":5626}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"76'79","reported_seq":187,"reported_epoch":431,"state":"active+undersized","last_fresh":"2024-09-21T12:36:44.421164+0000","last_change":"2024-09-21T12:34:34.348261+0000","last_active":"2024-09-21T12:36:44.421164+0000","last_peered":"2024-09-21T12:36:44.421164+0000","last_clean":"2024-09-21T12:34:29.279263+0000","last_became_active":"2024-09-21T12:34:34.348261+0000","last_became_peered":"2024-09-21T12:34:34.348261+0000","last_unstale":"2024-09-21T12:36:44.421164+0000","last_undegraded":"2024-09-21T12:36:44.421164+0000","last_fullsized":"2024-09-21T12:34:34.332145+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:00.343916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:00.343916+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:39.681388+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":949,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.123213+0000","last_change":"2024-09-21T12:36:00.163117+0000","last_active":"2024-09-21T12:36:47.123213+0000","last_peered":"2024-09-21T12:36:47.123213+0000","last_clean":"2024-09-21T12:36:47.123213+0000","last_became_active":"2024-09-21T12:36:00.139849+0000","last_became_peered":"2024-09-21T12:36:00.139849+0000","last_unstale":"2024-09-21T12:36:47.123213+0000","last_undegraded":"2024-09-21T12:36:47.123213+0000","last_fullsized":"2024-09-21T12:36:47.123213+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T12:35:12.790809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:12.790809+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":800,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.120485+0000","last_change":"2024-09-21T12:36:00.152050+0000","last_active":"2024-09-21T12:36:47.120485+0000","last_peered":"2024-09-21T12:36:47.120485+0000","last_clean":"2024-09-21T12:36:47.120485+0000","last_became_active":"2024-09-21T12:36:00.151488+0000","last_became_peered":"2024-09-21T12:36:00.151488+0000","last_unstale":"2024-09-21T12:36:47.120485+0000","last_undegraded":"2024-09-21T12:36:47.120485+0000","last_fullsized":"2024-09-21T12:36:47.120485+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:22.752960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:22.752960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":197,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.419866+0000","last_change":"2024-09-21T12:36:00.137014+0000","last_active":"2024-09-21T12:36:44.419866+0000","last_peered":"2024-09-21T12:36:44.419866+0000","last_clean":"2024-09-21T12:36:44.419866+0000","last_became_active":"2024-09-21T12:36:00.136650+0000","last_became_peered":"2024-09-21T12:36:00.136650+0000","last_unstale":"2024-09-21T12:36:44.419866+0000","last_undegraded":"2024-09-21T12:36:44.419866+0000","last_fullsized":"2024-09-21T12:36:44.419866+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:12.216104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:12.216104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":179,"reported_epoch":431,"state":"active+undersized","last_fresh":"2024-09-21T12:36:44.420602+0000","last_change":"2024-09-21T12:34:34.347165+0000","last_active":"2024-09-21T12:36:44.420602+0000","last_peered":"2024-09-21T12:36:44.420602+0000","last_clean":"2024-09-21T12:34:29.278468+0000","last_became_active":"2024-09-21T12:34:34.347165+0000","last_became_peered":"2024-09-21T12:34:34.347165+0000","last_unstale":"2024-09-21T12:36:44.420602+0000","last_undegraded":"2024-09-21T12:36:44.420602+0000","last_fullsized":"2024-09-21T12:34:34.331768+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:05.143736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:29:05.143736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:19.903019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"431'5546","reported_seq":7660,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:46.836377+0000","last_change":"2024-09-21T12:36:08.103615+0000","last_active":"2024-09-21T12:36:46.836377+0000","last_peered":"2024-09-21T12:36:46.836377+0000","last_clean":"2024-09-21T12:36:46.836377+0000","last_became_active":"2024-09-21T12:36:08.101796+0000","last_became_peered":"2024-09-21T12:36:08.101796+0000","last_unstale":"2024-09-21T12:36:46.836377+0000","last_undegraded":"2024-09-21T12:36:46.836377+0000","last_fullsized":"2024-09-21T12:36:46.836377+0000","mapping_epoch":417,"log_start":"416'1400","ondisk_log_start":"416'1400","created":405,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4146,"log_dups_size":0,"ondisk_log_size":4146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:16.601019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363462656,"num_objects":5546,"num_object_clones":0,"num_object_copies":16638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5546,"num_write_kb":354944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1016,"num_bytes_recovered":66584576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"76'79","reported_seq":735,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.121301+0000","last_change":"2024-09-21T12:36:43.462949+0000","last_active":"2024-09-21T12:36:47.121301+0000","last_peered":"2024-09-21T12:36:47.121301+0000","last_clean":"2024-09-21T12:36:47.121301+0000","last_became_active":"2024-09-21T12:36:00.155047+0000","last_became_peered":"2024-09-21T12:36:00.155047+0000","last_unstale":"2024-09-21T12:36:47.121301+0000","last_undegraded":"2024-09-21T12:36:47.121301+0000","last_fullsized":"2024-09-21T12:36:47.121301+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:43.462874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:43.462874+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:44.145911+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":191,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.419628+0000","last_change":"2024-09-21T12:32:38.457999+0000","last_active":"2024-09-21T12:36:44.419628+0000","last_peered":"2024-09-21T12:36:44.419628+0000","last_clean":"2024-09-21T12:36:44.419628+0000","last_became_active":"2024-09-21T12:32:38.425453+0000","last_became_peered":"2024-09-21T12:32:38.425453+0000","last_unstale":"2024-09-21T12:36:44.419628+0000","last_undegraded":"2024-09-21T12:36:44.419628+0000","last_fullsized":"2024-09-21T12:36:44.419628+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-21T12:26:51.463698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:51.463698+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:07.901778+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":40,"num_read_kb":33,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":879,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-21T12:36:40.297887+0000","last_change":"2024-09-21T12:36:39.754233+0000","last_active":"2024-09-21T12:36:40.297887+0000","last_peered":"2024-09-21T12:36:40.297887+0000","last_clean":"2024-09-21T12:36:40.297887+0000","last_became_active":"2024-09-21T12:32:38.424412+0000","last_became_peered":"2024-09-21T12:32:38.424412+0000","last_unstale":"2024-09-21T12:36:40.297887+0000","last_undegraded":"2024-09-21T12:36:40.297887+0000","last_fullsized":"2024-09-21T12:36:40.297887+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:36:39.754173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:36:39.754173+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:02.184909+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"352'12","reported_seq":1038,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.123041+0000","last_change":"2024-09-21T12:34:07.827409+0000","last_active":"2024-09-21T12:36:47.123041+0000","last_peered":"2024-09-21T12:36:47.123041+0000","last_clean":"2024-09-21T12:36:47.123041+0000","last_became_active":"2024-09-21T12:32:38.409177+0000","last_became_peered":"2024-09-21T12:32:38.409177+0000","last_unstale":"2024-09-21T12:36:47.123041+0000","last_undegraded":"2024-09-21T12:36:47.123041+0000","last_fullsized":"2024-09-21T12:36:47.123041+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'12","last_scrub_stamp":"2024-09-21T12:34:07.827315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:07.827315+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":770,"reported_epoch":431,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:44.424582+0000","last_change":"2024-09-21T12:36:01.138656+0000","last_active":"2024-09-21T12:36:44.424582+0000","last_peered":"2024-09-21T12:36:44.424582+0000","last_clean":"2024-09-21T12:36:44.424582+0000","last_became_active":"2024-09-21T12:36:01.135064+0000","last_became_peered":"2024-09-21T12:36:01.135064+0000","last_unstale":"2024-09-21T12:36:44.424582+0000","last_undegraded":"2024-09-21T12:36:44.424582+0000","last_fullsized":"2024-09-21T12:36:44.424582+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:59.406201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:32:59.406201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:25.816939+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6,1],"avail_no_missing":["7","1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"431'5353","reported_seq":5413,"reported_epoch":431,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:46.878004+0000","last_change":"2024-09-21T12:35:53.507442+0000","last_active":"2024-09-21T12:36:46.878004+0000","last_peered":"2024-09-21T12:36:46.878004+0000","last_clean":"2024-09-21T12:35:46.767214+0000","last_became_active":"2024-09-21T12:35:47.791328+0000","last_became_peered":"2024-09-21T12:35:47.791328+0000","last_unstale":"2024-09-21T12:36:46.878004+0000","last_undegraded":"2024-09-21T12:35:53.507160+0000","last_fullsized":"2024-09-21T12:35:47.787870+0000","mapping_epoch":405,"log_start":"416'1200","ondisk_log_start":"416'1200","created":405,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4153,"log_dups_size":0,"ondisk_log_size":4153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:03.374268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350814208,"num_objects":5353,"num_object_clones":0,"num_object_copies":16059,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5353,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5353,"num_write_kb":342592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5353}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"76'79","reported_seq":268,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.120339+0000","last_change":"2024-09-21T12:36:00.151734+0000","last_active":"2024-09-21T12:36:47.120339+0000","last_peered":"2024-09-21T12:36:47.120339+0000","last_clean":"2024-09-21T12:36:47.120339+0000","last_became_active":"2024-09-21T12:36:00.151076+0000","last_became_peered":"2024-09-21T12:36:00.151076+0000","last_unstale":"2024-09-21T12:36:47.120339+0000","last_undegraded":"2024-09-21T12:36:47.120339+0000","last_fullsized":"2024-09-21T12:36:47.120339+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:31:50.320834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:31:50.320834+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:13.696624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":334,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.420841+0000","last_change":"2024-09-21T12:36:33.481707+0000","last_active":"2024-09-21T12:36:44.420841+0000","last_peered":"2024-09-21T12:36:44.420841+0000","last_clean":"2024-09-21T12:36:44.420841+0000","last_became_active":"2024-09-21T12:32:38.424412+0000","last_became_peered":"2024-09-21T12:32:38.424412+0000","last_unstale":"2024-09-21T12:36:44.420841+0000","last_undegraded":"2024-09-21T12:36:44.420841+0000","last_fullsized":"2024-09-21T12:36:44.420841+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:36:33.481627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:33.481627+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:00.222164+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":805,"reported_epoch":431,"state":"active+undersized","last_fresh":"2024-09-21T12:36:44.428181+0000","last_change":"2024-09-21T12:34:34.339986+0000","last_active":"2024-09-21T12:36:44.428181+0000","last_peered":"2024-09-21T12:36:44.428181+0000","last_clean":"2024-09-21T12:34:29.305656+0000","last_became_active":"2024-09-21T12:34:34.339986+0000","last_became_peered":"2024-09-21T12:34:34.339986+0000","last_unstale":"2024-09-21T12:36:44.428181+0000","last_undegraded":"2024-09-21T12:36:44.428181+0000","last_fullsized":"2024-09-21T12:34:34.336347+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:19.629083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:19.629083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:43.542005+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":994,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-21T12:36:40.300996+0000","last_change":"2024-09-21T12:34:17.207726+0000","last_active":"2024-09-21T12:36:40.300996+0000","last_peered":"2024-09-21T12:36:40.300996+0000","last_clean":"2024-09-21T12:36:40.300996+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:36:40.300996+0000","last_undegraded":"2024-09-21T12:36:40.300996+0000","last_fullsized":"2024-09-21T12:36:40.300996+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:34:17.207117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:17.207117+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:42.051141+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":778,"reported_epoch":430,"state":"active+undersized","last_fresh":"2024-09-21T12:36:40.295145+0000","last_change":"2024-09-21T12:34:34.335260+0000","last_active":"2024-09-21T12:36:40.295145+0000","last_peered":"2024-09-21T12:36:40.295145+0000","last_clean":"2024-09-21T12:34:29.288868+0000","last_became_active":"2024-09-21T12:34:34.335260+0000","last_became_peered":"2024-09-21T12:34:34.335260+0000","last_unstale":"2024-09-21T12:36:40.295145+0000","last_undegraded":"2024-09-21T12:36:40.295145+0000","last_fullsized":"2024-09-21T12:34:34.331914+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:37.850100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:37.850100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:41.003975+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"431'5620","reported_seq":5680,"reported_epoch":431,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:46.583183+0000","last_change":"2024-09-21T12:35:53.515408+0000","last_active":"2024-09-21T12:36:46.583183+0000","last_peered":"2024-09-21T12:36:46.583183+0000","last_clean":"2024-09-21T12:35:46.767214+0000","last_became_active":"2024-09-21T12:35:47.791289+0000","last_became_peered":"2024-09-21T12:35:47.791289+0000","last_unstale":"2024-09-21T12:36:46.583183+0000","last_undegraded":"2024-09-21T12:35:53.506389+0000","last_fullsized":"2024-09-21T12:35:47.788809+0000","mapping_epoch":405,"log_start":"416'1800","ondisk_log_start":"416'1800","created":405,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3820,"log_dups_size":0,"ondisk_log_size":3820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:03.677425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368312320,"num_objects":5620,"num_object_clones":0,"num_object_copies":16860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5620,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5620,"num_write_kb":359680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":5620}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"76'79","reported_seq":467,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-21T12:36:40.296849+0000","last_change":"2024-09-21T12:32:52.986242+0000","last_active":"2024-09-21T12:36:40.296849+0000","last_peered":"2024-09-21T12:36:40.296849+0000","last_clean":"2024-09-21T12:36:40.296849+0000","last_became_active":"2024-09-21T12:29:05.835884+0000","last_became_peered":"2024-09-21T12:29:05.835884+0000","last_unstale":"2024-09-21T12:36:40.296849+0000","last_undegraded":"2024-09-21T12:36:40.296849+0000","last_fullsized":"2024-09-21T12:36:40.296849+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:52.986175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:52.986175+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:24.882111+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":890,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-21T12:36:40.300858+0000","last_change":"2024-09-21T12:36:36.774685+0000","last_active":"2024-09-21T12:36:40.300858+0000","last_peered":"2024-09-21T12:36:40.300858+0000","last_clean":"2024-09-21T12:36:40.300858+0000","last_became_active":"2024-09-21T12:36:00.127312+0000","last_became_peered":"2024-09-21T12:36:00.127312+0000","last_unstale":"2024-09-21T12:36:40.300858+0000","last_undegraded":"2024-09-21T12:36:40.300858+0000","last_fullsized":"2024-09-21T12:36:40.300858+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'29","last_scrub_stamp":"2024-09-21T12:36:36.774599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:36.774599+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:43.150659+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":781,"reported_epoch":431,"state":"active+undersized","last_fresh":"2024-09-21T12:36:44.423259+0000","last_change":"2024-09-21T12:34:34.337276+0000","last_active":"2024-09-21T12:36:44.423259+0000","last_peered":"2024-09-21T12:36:44.423259+0000","last_clean":"2024-09-21T12:34:29.286433+0000","last_became_active":"2024-09-21T12:34:34.337276+0000","last_became_peered":"2024-09-21T12:34:34.337276+0000","last_unstale":"2024-09-21T12:36:44.423259+0000","last_undegraded":"2024-09-21T12:36:44.423259+0000","last_fullsized":"2024-09-21T12:34:34.330247+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:38.250806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:38.250806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:29.240702+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":371,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-21T12:36:40.297413+0000","last_change":"2024-09-21T12:36:00.924159+0000","last_active":"2024-09-21T12:36:40.297413+0000","last_peered":"2024-09-21T12:36:40.297413+0000","last_clean":"2024-09-21T12:36:40.297413+0000","last_became_active":"2024-09-21T12:29:05.825869+0000","last_became_peered":"2024-09-21T12:29:05.825869+0000","last_unstale":"2024-09-21T12:36:40.297413+0000","last_undegraded":"2024-09-21T12:36:40.297413+0000","last_fullsized":"2024-09-21T12:36:40.297413+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:00.924084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:00.924084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:28.472139+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":777,"reported_epoch":431,"state":"active+undersized","last_fresh":"2024-09-21T12:36:44.420314+0000","last_change":"2024-09-21T12:34:34.339352+0000","last_active":"2024-09-21T12:36:44.420314+0000","last_peered":"2024-09-21T12:36:44.420314+0000","last_clean":"2024-09-21T12:34:29.284095+0000","last_became_active":"2024-09-21T12:34:34.339352+0000","last_became_peered":"2024-09-21T12:34:34.339352+0000","last_unstale":"2024-09-21T12:36:44.420314+0000","last_undegraded":"2024-09-21T12:36:44.420314+0000","last_fullsized":"2024-09-21T12:34:34.329336+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:05.510620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:05.510620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:25.203926+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"432'5498","reported_seq":5563,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.806142+0000","last_change":"2024-09-21T12:35:47.791371+0000","last_active":"2024-09-21T12:36:47.806142+0000","last_peered":"2024-09-21T12:36:47.806142+0000","last_clean":"2024-09-21T12:36:47.806142+0000","last_became_active":"2024-09-21T12:35:47.791120+0000","last_became_peered":"2024-09-21T12:35:47.791120+0000","last_unstale":"2024-09-21T12:36:47.806142+0000","last_undegraded":"2024-09-21T12:36:47.806142+0000","last_fullsized":"2024-09-21T12:36:47.806142+0000","mapping_epoch":405,"log_start":"416'1700","ondisk_log_start":"416'1700","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3798,"log_dups_size":0,"ondisk_log_size":3798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:02.202057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360316928,"num_objects":5498,"num_object_clones":0,"num_object_copies":16494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5498,"num_write_kb":351872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"76'79","reported_seq":579,"reported_epoch":430,"state":"active+undersized","last_fresh":"2024-09-21T12:36:40.301057+0000","last_change":"2024-09-21T12:34:34.337922+0000","last_active":"2024-09-21T12:36:40.301057+0000","last_peered":"2024-09-21T12:36:40.301057+0000","last_clean":"2024-09-21T12:34:29.289671+0000","last_became_active":"2024-09-21T12:34:34.337922+0000","last_became_peered":"2024-09-21T12:34:34.337922+0000","last_unstale":"2024-09-21T12:36:40.301057+0000","last_undegraded":"2024-09-21T12:36:40.301057+0000","last_fullsized":"2024-09-21T12:34:34.332211+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:34.055720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:34.055720+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:32.875426+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"389'41","reported_seq":1043,"reported_epoch":431,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:44.422911+0000","last_change":"2024-09-21T12:34:34.339316+0000","last_active":"2024-09-21T12:36:44.422911+0000","last_peered":"2024-09-21T12:36:44.422911+0000","last_clean":"2024-09-21T12:34:30.529659+0000","last_became_active":"2024-09-21T12:34:34.339316+0000","last_became_peered":"2024-09-21T12:34:34.339316+0000","last_unstale":"2024-09-21T12:36:44.422911+0000","last_undegraded":"2024-09-21T12:34:34.330286+0000","last_fullsized":"2024-09-21T12:34:34.330177+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'40","last_scrub_stamp":"2024-09-21T12:32:25.139435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:25.139435+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:40.440122+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":183,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.420369+0000","last_change":"2024-09-21T12:35:09.182021+0000","last_active":"2024-09-21T12:36:44.420369+0000","last_peered":"2024-09-21T12:36:44.420369+0000","last_clean":"2024-09-21T12:36:44.420369+0000","last_became_active":"2024-09-21T12:32:38.426309+0000","last_became_peered":"2024-09-21T12:32:38.426309+0000","last_unstale":"2024-09-21T12:36:44.420369+0000","last_undegraded":"2024-09-21T12:36:44.420369+0000","last_fullsized":"2024-09-21T12:36:44.420369+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:09.181952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:09.181952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":511,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.421663+0000","last_change":"2024-09-21T12:35:27.114524+0000","last_active":"2024-09-21T12:36:44.421663+0000","last_peered":"2024-09-21T12:36:44.421663+0000","last_clean":"2024-09-21T12:36:44.421663+0000","last_became_active":"2024-09-21T12:26:54.083584+0000","last_became_peered":"2024-09-21T12:26:54.083584+0000","last_unstale":"2024-09-21T12:36:44.421663+0000","last_undegraded":"2024-09-21T12:36:44.421663+0000","last_fullsized":"2024-09-21T12:36:44.421663+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T12:35:27.114442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:35:27.114442+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":821,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-21T12:36:40.295441+0000","last_change":"2024-09-21T12:33:00.087826+0000","last_active":"2024-09-21T12:36:40.295441+0000","last_peered":"2024-09-21T12:36:40.295441+0000","last_clean":"2024-09-21T12:36:40.295441+0000","last_became_active":"2024-09-21T12:32:38.424402+0000","last_became_peered":"2024-09-21T12:32:38.424402+0000","last_unstale":"2024-09-21T12:36:40.295441+0000","last_undegraded":"2024-09-21T12:36:40.295441+0000","last_fullsized":"2024-09-21T12:36:40.295441+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:00.087746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:33:00.087746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:10.889168+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":824,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.120821+0000","last_change":"2024-09-21T12:36:00.144733+0000","last_active":"2024-09-21T12:36:47.120821+0000","last_peered":"2024-09-21T12:36:47.120821+0000","last_clean":"2024-09-21T12:36:47.120821+0000","last_became_active":"2024-09-21T12:36:00.144526+0000","last_became_peered":"2024-09-21T12:36:00.144526+0000","last_unstale":"2024-09-21T12:36:47.120821+0000","last_undegraded":"2024-09-21T12:36:47.120821+0000","last_fullsized":"2024-09-21T12:36:47.120821+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:59.899499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:59.899499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"432'5631","reported_seq":5696,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.816676+0000","last_change":"2024-09-21T12:35:47.848839+0000","last_active":"2024-09-21T12:36:47.816676+0000","last_peered":"2024-09-21T12:36:47.816676+0000","last_clean":"2024-09-21T12:36:47.816676+0000","last_became_active":"2024-09-21T12:35:47.848543+0000","last_became_peered":"2024-09-21T12:35:47.848543+0000","last_unstale":"2024-09-21T12:36:47.816676+0000","last_undegraded":"2024-09-21T12:36:47.816676+0000","last_fullsized":"2024-09-21T12:36:47.816676+0000","mapping_epoch":405,"log_start":"416'1800","ondisk_log_start":"416'1800","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3831,"log_dups_size":0,"ondisk_log_size":3831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:50.416027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369033216,"num_objects":5631,"num_object_clones":0,"num_object_copies":16893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5631,"num_write_kb":360384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'79","reported_seq":208,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.424992+0000","last_change":"2024-09-21T12:36:01.132046+0000","last_active":"2024-09-21T12:36:44.424992+0000","last_peered":"2024-09-21T12:36:44.424992+0000","last_clean":"2024-09-21T12:36:44.424992+0000","last_became_active":"2024-09-21T12:36:01.131622+0000","last_became_peered":"2024-09-21T12:36:01.131622+0000","last_unstale":"2024-09-21T12:36:44.424992+0000","last_undegraded":"2024-09-21T12:36:44.424992+0000","last_fullsized":"2024-09-21T12:36:44.424992+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:30:39.179014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:30:39.179014+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:24.773542+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":288,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.423587+0000","last_change":"2024-09-21T12:35:18.061854+0000","last_active":"2024-09-21T12:36:44.423587+0000","last_peered":"2024-09-21T12:36:44.423587+0000","last_clean":"2024-09-21T12:36:44.423587+0000","last_became_active":"2024-09-21T12:31:39.057316+0000","last_became_peered":"2024-09-21T12:31:39.057316+0000","last_unstale":"2024-09-21T12:36:44.423587+0000","last_undegraded":"2024-09-21T12:36:44.423587+0000","last_fullsized":"2024-09-21T12:36:44.423587+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'17","last_scrub_stamp":"2024-09-21T12:35:18.061753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:18.061753+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":814,"reported_epoch":431,"state":"active+undersized","last_fresh":"2024-09-21T12:36:44.420066+0000","last_change":"2024-09-21T12:34:34.343641+0000","last_active":"2024-09-21T12:36:44.420066+0000","last_peered":"2024-09-21T12:36:44.420066+0000","last_clean":"2024-09-21T12:34:29.286935+0000","last_became_active":"2024-09-21T12:34:34.343641+0000","last_became_peered":"2024-09-21T12:34:34.343641+0000","last_unstale":"2024-09-21T12:36:44.420066+0000","last_undegraded":"2024-09-21T12:36:44.420066+0000","last_fullsized":"2024-09-21T12:34:34.329247+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:00.211563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:34:00.211563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:30.227747+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":124,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:47.119492+0000","last_change":"2024-09-21T12:34:34.338167+0000","last_active":"2024-09-21T12:36:47.119492+0000","last_peered":"2024-09-21T12:36:47.119492+0000","last_clean":"2024-09-21T12:34:29.291114+0000","last_became_active":"2024-09-21T12:34:34.338167+0000","last_became_peered":"2024-09-21T12:34:34.338167+0000","last_unstale":"2024-09-21T12:36:47.119492+0000","last_undegraded":"2024-09-21T12:34:34.335226+0000","last_fullsized":"2024-09-21T12:34:34.335117+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:32:20.845428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:32:20.845428+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:19.597743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":122,"reported_epoch":431,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:44.428170+0000","last_change":"2024-09-21T12:36:01.125293+0000","last_active":"2024-09-21T12:36:44.428170+0000","last_peered":"2024-09-21T12:36:44.428170+0000","last_clean":"2024-09-21T12:36:44.428170+0000","last_became_active":"2024-09-21T12:36:01.124868+0000","last_became_peered":"2024-09-21T12:36:01.124868+0000","last_unstale":"2024-09-21T12:36:44.428170+0000","last_undegraded":"2024-09-21T12:36:44.428170+0000","last_fullsized":"2024-09-21T12:36:44.428170+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:02.295255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:02.295255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:20.382099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7,4],"avail_no_missing":["5","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":13,"reported_epoch":430,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:40.306247+0000","last_change":"2024-09-21T12:36:40.306122+0000","last_active":"2024-09-21T12:36:40.306247+0000","last_peered":"2024-09-21T12:36:40.306247+0000","last_clean":"2024-09-21T12:36:40.306247+0000","last_became_active":"2024-09-21T12:36:40.305602+0000","last_became_peered":"2024-09-21T12:36:40.305602+0000","last_unstale":"2024-09-21T12:36:40.306247+0000","last_undegraded":"2024-09-21T12:36:40.306247+0000","last_fullsized":"2024-09-21T12:36:40.306247+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:07.541665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:07.541665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6,3],"avail_no_missing":["0","3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"432'5600","reported_seq":5665,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.789217+0000","last_change":"2024-09-21T12:35:47.791375+0000","last_active":"2024-09-21T12:36:47.789217+0000","last_peered":"2024-09-21T12:36:47.789217+0000","last_clean":"2024-09-21T12:36:47.789217+0000","last_became_active":"2024-09-21T12:35:47.791073+0000","last_became_peered":"2024-09-21T12:35:47.791073+0000","last_unstale":"2024-09-21T12:36:47.789217+0000","last_undegraded":"2024-09-21T12:36:47.789217+0000","last_fullsized":"2024-09-21T12:36:47.789217+0000","mapping_epoch":405,"log_start":"416'1800","ondisk_log_start":"416'1800","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3800,"log_dups_size":0,"ondisk_log_size":3800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:52.928312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367001600,"num_objects":5600,"num_object_clones":0,"num_object_copies":16800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5600,"num_write_kb":358400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"76'79","reported_seq":750,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.432717+0000","last_change":"2024-09-21T12:36:41.495668+0000","last_active":"2024-09-21T12:36:44.432717+0000","last_peered":"2024-09-21T12:36:44.432717+0000","last_clean":"2024-09-21T12:36:44.432717+0000","last_became_active":"2024-09-21T12:32:38.424710+0000","last_became_peered":"2024-09-21T12:32:38.424710+0000","last_unstale":"2024-09-21T12:36:44.432717+0000","last_undegraded":"2024-09-21T12:36:44.432717+0000","last_fullsized":"2024-09-21T12:36:44.432717+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:41.495532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:41.495532+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:56.228849+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":821,"reported_epoch":430,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:40.297489+0000","last_change":"2024-09-21T12:34:34.334240+0000","last_active":"2024-09-21T12:36:40.297489+0000","last_peered":"2024-09-21T12:36:40.297489+0000","last_clean":"2024-09-21T12:34:29.290312+0000","last_became_active":"2024-09-21T12:34:34.334240+0000","last_became_peered":"2024-09-21T12:34:34.334240+0000","last_unstale":"2024-09-21T12:36:40.297489+0000","last_undegraded":"2024-09-21T12:34:34.331492+0000","last_fullsized":"2024-09-21T12:34:34.331281+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:28:45.782955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.782955+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:47.864929+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":835,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-21T12:36:40.298530+0000","last_change":"2024-09-21T12:32:53.934634+0000","last_active":"2024-09-21T12:36:40.298530+0000","last_peered":"2024-09-21T12:36:40.298530+0000","last_clean":"2024-09-21T12:36:40.298530+0000","last_became_active":"2024-09-21T12:32:38.411004+0000","last_became_peered":"2024-09-21T12:32:38.411004+0000","last_unstale":"2024-09-21T12:36:40.298530+0000","last_undegraded":"2024-09-21T12:36:40.298530+0000","last_fullsized":"2024-09-21T12:36:40.298530+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:53.934583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:53.934583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:17.842220+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":103,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:45.220123+0000","last_change":"2024-09-21T12:36:27.852222+0000","last_active":"2024-09-21T12:36:45.220123+0000","last_peered":"2024-09-21T12:36:45.220123+0000","last_clean":"2024-09-21T12:36:45.220123+0000","last_became_active":"2024-09-21T12:36:00.144053+0000","last_became_peered":"2024-09-21T12:36:00.144053+0000","last_unstale":"2024-09-21T12:36:45.220123+0000","last_undegraded":"2024-09-21T12:36:45.220123+0000","last_fullsized":"2024-09-21T12:36:45.220123+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-21T12:36:27.852030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:36:27.852030+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:48.419358+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":53,"num_read_kb":48,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":232,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.428472+0000","last_change":"2024-09-21T12:36:36.590854+0000","last_active":"2024-09-21T12:36:44.428472+0000","last_peered":"2024-09-21T12:36:44.428472+0000","last_clean":"2024-09-21T12:36:44.428472+0000","last_became_active":"2024-09-21T12:32:02.030240+0000","last_became_peered":"2024-09-21T12:32:02.030240+0000","last_unstale":"2024-09-21T12:36:44.428472+0000","last_undegraded":"2024-09-21T12:36:44.428472+0000","last_fullsized":"2024-09-21T12:36:44.428472+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:36.590782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:36.590782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:38.226971+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":276,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.422539+0000","last_change":"2024-09-21T12:35:22.978507+0000","last_active":"2024-09-21T12:36:44.422539+0000","last_peered":"2024-09-21T12:36:44.422539+0000","last_clean":"2024-09-21T12:36:44.422539+0000","last_became_active":"2024-09-21T12:31:39.060160+0000","last_became_peered":"2024-09-21T12:31:39.060160+0000","last_unstale":"2024-09-21T12:36:44.422539+0000","last_undegraded":"2024-09-21T12:36:44.422539+0000","last_fullsized":"2024-09-21T12:36:44.422539+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:22.978438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:22.978438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"432'5599","reported_seq":5664,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.811434+0000","last_change":"2024-09-21T12:35:47.849177+0000","last_active":"2024-09-21T12:36:47.811434+0000","last_peered":"2024-09-21T12:36:47.811434+0000","last_clean":"2024-09-21T12:36:47.811434+0000","last_became_active":"2024-09-21T12:35:47.848768+0000","last_became_peered":"2024-09-21T12:35:47.848768+0000","last_unstale":"2024-09-21T12:36:47.811434+0000","last_undegraded":"2024-09-21T12:36:47.811434+0000","last_fullsized":"2024-09-21T12:36:47.811434+0000","mapping_epoch":405,"log_start":"417'1800","ondisk_log_start":"417'1800","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3799,"log_dups_size":0,"ondisk_log_size":3799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:56.898376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366936064,"num_objects":5599,"num_object_clones":0,"num_object_copies":16797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5599,"num_write_kb":358336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"415'35","reported_seq":309,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.421456+0000","last_change":"2024-09-21T12:36:43.681619+0000","last_active":"2024-09-21T12:36:44.421456+0000","last_peered":"2024-09-21T12:36:44.421456+0000","last_clean":"2024-09-21T12:36:44.421456+0000","last_became_active":"2024-09-21T12:36:00.139342+0000","last_became_peered":"2024-09-21T12:36:00.139342+0000","last_unstale":"2024-09-21T12:36:44.421456+0000","last_undegraded":"2024-09-21T12:36:44.421456+0000","last_fullsized":"2024-09-21T12:36:44.421456+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'35","last_scrub_stamp":"2024-09-21T12:36:43.681471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:43.681471+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:50.711798+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":800,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.120545+0000","last_change":"2024-09-21T12:34:01.815496+0000","last_active":"2024-09-21T12:36:47.120545+0000","last_peered":"2024-09-21T12:36:47.120545+0000","last_clean":"2024-09-21T12:36:47.120545+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:36:47.120545+0000","last_undegraded":"2024-09-21T12:36:47.120545+0000","last_fullsized":"2024-09-21T12:36:47.120545+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:34:01.815445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:34:01.815445+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":44,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.425653+0000","last_change":"2024-09-21T12:36:00.144564+0000","last_active":"2024-09-21T12:36:44.425653+0000","last_peered":"2024-09-21T12:36:44.425653+0000","last_clean":"2024-09-21T12:36:44.425653+0000","last_became_active":"2024-09-21T12:36:00.134878+0000","last_became_peered":"2024-09-21T12:36:00.134878+0000","last_unstale":"2024-09-21T12:36:44.425653+0000","last_undegraded":"2024-09-21T12:36:44.425653+0000","last_fullsized":"2024-09-21T12:36:44.425653+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:35:24.687711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:35:24.687711+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:37:11.734028+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26834,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":9,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26834,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":205,"reported_epoch":431,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:44.423558+0000","last_change":"2024-09-21T12:36:01.130402+0000","last_active":"2024-09-21T12:36:44.423558+0000","last_peered":"2024-09-21T12:36:44.423558+0000","last_clean":"2024-09-21T12:36:44.423558+0000","last_became_active":"2024-09-21T12:36:01.129957+0000","last_became_peered":"2024-09-21T12:36:01.129957+0000","last_unstale":"2024-09-21T12:36:44.423558+0000","last_undegraded":"2024-09-21T12:36:44.423558+0000","last_fullsized":"2024-09-21T12:36:44.423558+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:36.684459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:36.684459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:21.752176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5,0],"avail_no_missing":["7","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":642,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.119841+0000","last_change":"2024-09-21T12:36:00.653180+0000","last_active":"2024-09-21T12:36:47.119841+0000","last_peered":"2024-09-21T12:36:47.119841+0000","last_clean":"2024-09-21T12:36:47.119841+0000","last_became_active":"2024-09-21T12:32:38.409187+0000","last_became_peered":"2024-09-21T12:32:38.409187+0000","last_unstale":"2024-09-21T12:36:47.119841+0000","last_undegraded":"2024-09-21T12:36:47.119841+0000","last_fullsized":"2024-09-21T12:36:47.119841+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:00.653130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:00.653130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:10.867677+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1","version":"432'5519","reported_seq":5584,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.816106+0000","last_change":"2024-09-21T12:35:47.788808+0000","last_active":"2024-09-21T12:36:47.816106+0000","last_peered":"2024-09-21T12:36:47.816106+0000","last_clean":"2024-09-21T12:36:47.816106+0000","last_became_active":"2024-09-21T12:35:47.788318+0000","last_became_peered":"2024-09-21T12:35:47.788318+0000","last_unstale":"2024-09-21T12:36:47.816106+0000","last_undegraded":"2024-09-21T12:36:47.816106+0000","last_fullsized":"2024-09-21T12:36:47.816106+0000","mapping_epoch":405,"log_start":"416'1700","ondisk_log_start":"416'1700","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3819,"log_dups_size":0,"ondisk_log_size":3819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:02.330107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":361693184,"num_objects":5519,"num_object_clones":0,"num_object_copies":16557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5519,"num_write_kb":353216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"76'79","reported_seq":739,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.432969+0000","last_change":"2024-09-21T12:33:04.508642+0000","last_active":"2024-09-21T12:36:44.432969+0000","last_peered":"2024-09-21T12:36:44.432969+0000","last_clean":"2024-09-21T12:36:44.432969+0000","last_became_active":"2024-09-21T12:27:14.954351+0000","last_became_peered":"2024-09-21T12:27:14.954351+0000","last_unstale":"2024-09-21T12:36:44.432969+0000","last_undegraded":"2024-09-21T12:36:44.432969+0000","last_fullsized":"2024-09-21T12:36:44.432969+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:04.508319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:04.508319+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:23.209734+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":203,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.421609+0000","last_change":"2024-09-21T12:36:41.666876+0000","last_active":"2024-09-21T12:36:44.421609+0000","last_peered":"2024-09-21T12:36:44.421609+0000","last_clean":"2024-09-21T12:36:44.421609+0000","last_became_active":"2024-09-21T12:32:38.419714+0000","last_became_peered":"2024-09-21T12:32:38.419714+0000","last_unstale":"2024-09-21T12:36:44.421609+0000","last_undegraded":"2024-09-21T12:36:44.421609+0000","last_fullsized":"2024-09-21T12:36:44.421609+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:36:41.666812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:41.666812+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:50.318849+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":207,"reported_epoch":431,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:44.419721+0000","last_change":"2024-09-21T12:36:40.708599+0000","last_active":"2024-09-21T12:36:44.419721+0000","last_peered":"2024-09-21T12:36:44.419721+0000","last_clean":"2024-09-21T12:36:44.419721+0000","last_became_active":"2024-09-21T12:36:01.130874+0000","last_became_peered":"2024-09-21T12:36:01.130874+0000","last_unstale":"2024-09-21T12:36:44.419721+0000","last_undegraded":"2024-09-21T12:36:44.419721+0000","last_fullsized":"2024-09-21T12:36:44.419721+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:40.708469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:36:40.708469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:10.539508+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7,1],"avail_no_missing":["4","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":183,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.421025+0000","last_change":"2024-09-21T12:33:59.196744+0000","last_active":"2024-09-21T12:36:44.421025+0000","last_peered":"2024-09-21T12:36:44.421025+0000","last_clean":"2024-09-21T12:36:44.421025+0000","last_became_active":"2024-09-21T12:32:38.425683+0000","last_became_peered":"2024-09-21T12:32:38.425683+0000","last_unstale":"2024-09-21T12:36:44.421025+0000","last_undegraded":"2024-09-21T12:36:44.421025+0000","last_fullsized":"2024-09-21T12:36:44.421025+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:59.196667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:33:59.196667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":782,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.121387+0000","last_change":"2024-09-21T12:35:24.786713+0000","last_active":"2024-09-21T12:36:47.121387+0000","last_peered":"2024-09-21T12:36:47.121387+0000","last_clean":"2024-09-21T12:36:47.121387+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:36:47.121387+0000","last_undegraded":"2024-09-21T12:36:47.121387+0000","last_fullsized":"2024-09-21T12:36:47.121387+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:24.786609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:24.786609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"432'5563","reported_seq":5625,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:47.807152+0000","last_change":"2024-09-21T12:35:53.505439+0000","last_active":"2024-09-21T12:36:47.807152+0000","last_peered":"2024-09-21T12:36:47.807152+0000","last_clean":"2024-09-21T12:35:46.767214+0000","last_became_active":"2024-09-21T12:35:47.787239+0000","last_became_peered":"2024-09-21T12:35:47.787239+0000","last_unstale":"2024-09-21T12:36:47.807152+0000","last_undegraded":"2024-09-21T12:35:53.504658+0000","last_fullsized":"2024-09-21T12:35:47.783070+0000","mapping_epoch":405,"log_start":"416'1700","ondisk_log_start":"416'1700","created":405,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3863,"log_dups_size":0,"ondisk_log_size":3863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:16.320842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364576768,"num_objects":5563,"num_object_clones":0,"num_object_copies":16689,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5563,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5563,"num_write_kb":356032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":5563}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"76'79","reported_seq":122,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T12:36:47.122802+0000","last_change":"2024-09-21T12:34:34.338192+0000","last_active":"2024-09-21T12:36:47.122802+0000","last_peered":"2024-09-21T12:36:47.122802+0000","last_clean":"2024-09-21T12:34:29.289953+0000","last_became_active":"2024-09-21T12:34:34.338192+0000","last_became_peered":"2024-09-21T12:34:34.338192+0000","last_unstale":"2024-09-21T12:36:47.122802+0000","last_undegraded":"2024-09-21T12:36:47.122802+0000","last_fullsized":"2024-09-21T12:34:34.334873+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:17.035216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:17.035216+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":205,"reported_epoch":431,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:44.420949+0000","last_change":"2024-09-21T12:34:34.341886+0000","last_active":"2024-09-21T12:36:44.420949+0000","last_peered":"2024-09-21T12:36:44.420949+0000","last_clean":"2024-09-21T12:34:29.279382+0000","last_became_active":"2024-09-21T12:34:34.341886+0000","last_became_peered":"2024-09-21T12:34:34.341886+0000","last_unstale":"2024-09-21T12:36:44.420949+0000","last_undegraded":"2024-09-21T12:34:34.333296+0000","last_fullsized":"2024-09-21T12:34:34.332270+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'31","last_scrub_stamp":"2024-09-21T12:33:08.360012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:08.360012+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:25.815464+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":191,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.420179+0000","last_change":"2024-09-21T12:35:18.088890+0000","last_active":"2024-09-21T12:36:44.420179+0000","last_peered":"2024-09-21T12:36:44.420179+0000","last_clean":"2024-09-21T12:36:44.420179+0000","last_became_active":"2024-09-21T12:32:38.424451+0000","last_became_peered":"2024-09-21T12:32:38.424451+0000","last_unstale":"2024-09-21T12:36:44.420179+0000","last_undegraded":"2024-09-21T12:36:44.420179+0000","last_fullsized":"2024-09-21T12:36:44.420179+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:18.088831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:18.088831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":813,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.422629+0000","last_change":"2024-09-21T12:36:43.539109+0000","last_active":"2024-09-21T12:36:44.422629+0000","last_peered":"2024-09-21T12:36:44.422629+0000","last_clean":"2024-09-21T12:36:44.422629+0000","last_became_active":"2024-09-21T12:32:38.425177+0000","last_became_peered":"2024-09-21T12:32:38.425177+0000","last_unstale":"2024-09-21T12:36:44.422629+0000","last_undegraded":"2024-09-21T12:36:44.422629+0000","last_fullsized":"2024-09-21T12:36:44.422629+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:43.539053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:43.539053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:59.470609+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":732,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T12:36:47.119671+0000","last_change":"2024-09-21T12:34:34.345710+0000","last_active":"2024-09-21T12:36:47.119671+0000","last_peered":"2024-09-21T12:36:47.119671+0000","last_clean":"2024-09-21T12:34:29.294810+0000","last_became_active":"2024-09-21T12:34:34.345710+0000","last_became_peered":"2024-09-21T12:34:34.345710+0000","last_unstale":"2024-09-21T12:36:47.119671+0000","last_undegraded":"2024-09-21T12:36:47.119671+0000","last_fullsized":"2024-09-21T12:34:34.334512+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:20.005906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:34:20.005906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3","version":"431'5457","reported_seq":7640,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:46.934793+0000","last_change":"2024-09-21T12:36:13.109138+0000","last_active":"2024-09-21T12:36:46.934793+0000","last_peered":"2024-09-21T12:36:46.934793+0000","last_clean":"2024-09-21T12:36:46.934793+0000","last_became_active":"2024-09-21T12:36:13.105381+0000","last_became_peered":"2024-09-21T12:36:13.105381+0000","last_unstale":"2024-09-21T12:36:46.934793+0000","last_undegraded":"2024-09-21T12:36:46.934793+0000","last_fullsized":"2024-09-21T12:36:46.934793+0000","mapping_epoch":420,"log_start":"413'900","ondisk_log_start":"413'900","created":405,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4557,"log_dups_size":0,"ondisk_log_size":4557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:57.457946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":357629952,"num_objects":5457,"num_object_clones":0,"num_object_copies":16371,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5457,"num_write_kb":349248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1051,"num_bytes_recovered":68878336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"76'79","reported_seq":856,"reported_epoch":431,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:44.423649+0000","last_change":"2024-09-21T12:36:01.128488+0000","last_active":"2024-09-21T12:36:44.423649+0000","last_peered":"2024-09-21T12:36:44.423649+0000","last_clean":"2024-09-21T12:36:44.423649+0000","last_became_active":"2024-09-21T12:36:01.127851+0000","last_became_peered":"2024-09-21T12:36:01.127851+0000","last_unstale":"2024-09-21T12:36:44.423649+0000","last_undegraded":"2024-09-21T12:36:44.423649+0000","last_fullsized":"2024-09-21T12:36:44.423649+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:05.500994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:05.500994+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:23.785504+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,5],"avail_no_missing":["7","3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":191,"reported_epoch":431,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:44.420148+0000","last_change":"2024-09-21T12:34:34.347138+0000","last_active":"2024-09-21T12:36:44.420148+0000","last_peered":"2024-09-21T12:36:44.420148+0000","last_clean":"2024-09-21T12:34:29.280929+0000","last_became_active":"2024-09-21T12:34:34.347138+0000","last_became_peered":"2024-09-21T12:34:34.347138+0000","last_unstale":"2024-09-21T12:36:44.420148+0000","last_undegraded":"2024-09-21T12:34:34.332331+0000","last_fullsized":"2024-09-21T12:34:34.330784+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:32:26.690403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:26.690403+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:22.898893+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":814,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.119649+0000","last_change":"2024-09-21T12:32:38.404305+0000","last_active":"2024-09-21T12:36:47.119649+0000","last_peered":"2024-09-21T12:36:47.119649+0000","last_clean":"2024-09-21T12:36:47.119649+0000","last_became_active":"2024-09-21T12:32:38.403997+0000","last_became_peered":"2024-09-21T12:32:38.403997+0000","last_unstale":"2024-09-21T12:36:47.119649+0000","last_undegraded":"2024-09-21T12:36:47.119649+0000","last_fullsized":"2024-09-21T12:36:47.119649+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:47.248389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:31:47.248389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:18.590240+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"416'14","reported_seq":168,"reported_epoch":431,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:44.424858+0000","last_change":"2024-09-21T12:36:31.838103+0000","last_active":"2024-09-21T12:36:44.424858+0000","last_peered":"2024-09-21T12:36:44.424858+0000","last_clean":"2024-09-21T12:36:44.424858+0000","last_became_active":"2024-09-21T12:36:01.130569+0000","last_became_peered":"2024-09-21T12:36:01.130569+0000","last_unstale":"2024-09-21T12:36:44.424858+0000","last_undegraded":"2024-09-21T12:36:44.424858+0000","last_fullsized":"2024-09-21T12:36:44.424858+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'14","last_scrub_stamp":"2024-09-21T12:36:31.838024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:31.838024+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:59.227849+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5,0],"avail_no_missing":["7","0","5"],"object_location_counts":[{"shards":"0,5,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":187,"reported_epoch":431,"state":"active+undersized","last_fresh":"2024-09-21T12:36:44.421252+0000","last_change":"2024-09-21T12:34:34.348303+0000","last_active":"2024-09-21T12:36:44.421252+0000","last_peered":"2024-09-21T12:36:44.421252+0000","last_clean":"2024-09-21T12:34:29.279670+0000","last_became_active":"2024-09-21T12:34:34.348303+0000","last_became_peered":"2024-09-21T12:34:34.348303+0000","last_unstale":"2024-09-21T12:36:44.421252+0000","last_undegraded":"2024-09-21T12:36:44.421252+0000","last_fullsized":"2024-09-21T12:34:34.331636+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:06.369436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:06.369436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:24.813776+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"431'5528","reported_seq":5589,"reported_epoch":431,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:46.900487+0000","last_change":"2024-09-21T12:36:01.140837+0000","last_active":"2024-09-21T12:36:46.900487+0000","last_peered":"2024-09-21T12:36:46.900487+0000","last_clean":"2024-09-21T12:36:46.900487+0000","last_became_active":"2024-09-21T12:36:01.132703+0000","last_became_peered":"2024-09-21T12:36:01.132703+0000","last_unstale":"2024-09-21T12:36:46.900487+0000","last_undegraded":"2024-09-21T12:36:46.900487+0000","last_fullsized":"2024-09-21T12:36:46.900487+0000","mapping_epoch":415,"log_start":"416'1300","ondisk_log_start":"416'1300","created":405,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4228,"log_dups_size":0,"ondisk_log_size":4228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:52.238296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362283008,"num_objects":5528,"num_object_clones":0,"num_object_copies":16584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5528,"num_objects_unfound":0,"num_objects_dirty":5528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5528,"num_write_kb":353792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6,1],"avail_no_missing":["7","1","6"],"object_location_counts":[{"shards":"1,6,7","objects":5528}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"76'79","reported_seq":195,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.421707+0000","last_change":"2024-09-21T12:36:01.159899+0000","last_active":"2024-09-21T12:36:44.421707+0000","last_peered":"2024-09-21T12:36:44.421707+0000","last_clean":"2024-09-21T12:36:44.421707+0000","last_became_active":"2024-09-21T12:36:01.159481+0000","last_became_peered":"2024-09-21T12:36:01.159481+0000","last_unstale":"2024-09-21T12:36:44.421707+0000","last_undegraded":"2024-09-21T12:36:44.421707+0000","last_fullsized":"2024-09-21T12:36:44.421707+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:49.155069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:49.155069+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":906,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-21T12:36:40.771296+0000","last_change":"2024-09-21T12:36:40.771296+0000","last_active":"2024-09-21T12:36:40.771296+0000","last_peered":"2024-09-21T12:36:40.771296+0000","last_clean":"2024-09-21T12:36:40.771296+0000","last_became_active":"2024-09-21T12:32:38.424376+0000","last_became_peered":"2024-09-21T12:32:38.424376+0000","last_unstale":"2024-09-21T12:36:40.771296+0000","last_undegraded":"2024-09-21T12:36:40.771296+0000","last_fullsized":"2024-09-21T12:36:40.771296+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:36:40.771222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:40.771222+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:42.363964+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":478,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.121910+0000","last_change":"2024-09-21T12:34:36.079650+0000","last_active":"2024-09-21T12:36:47.121910+0000","last_peered":"2024-09-21T12:36:47.121910+0000","last_clean":"2024-09-21T12:36:47.121910+0000","last_became_active":"2024-09-21T12:26:54.084228+0000","last_became_peered":"2024-09-21T12:26:54.084228+0000","last_unstale":"2024-09-21T12:36:47.121910+0000","last_undegraded":"2024-09-21T12:36:47.121910+0000","last_fullsized":"2024-09-21T12:36:47.121910+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:36.079583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:34:36.079583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":750,"reported_epoch":431,"state":"active+undersized","last_fresh":"2024-09-21T12:36:44.420393+0000","last_change":"2024-09-21T12:34:34.346822+0000","last_active":"2024-09-21T12:36:44.420393+0000","last_peered":"2024-09-21T12:36:44.420393+0000","last_clean":"2024-09-21T12:34:29.278339+0000","last_became_active":"2024-09-21T12:34:34.346822+0000","last_became_peered":"2024-09-21T12:34:34.346822+0000","last_unstale":"2024-09-21T12:36:44.420393+0000","last_undegraded":"2024-09-21T12:36:44.420393+0000","last_fullsized":"2024-09-21T12:34:34.330172+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:47.361115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:47.361115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:10.874140+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":797,"reported_epoch":430,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:40.298922+0000","last_change":"2024-09-21T12:36:34.842873+0000","last_active":"2024-09-21T12:36:40.298922+0000","last_peered":"2024-09-21T12:36:40.298922+0000","last_clean":"2024-09-21T12:36:40.298922+0000","last_became_active":"2024-09-21T12:36:01.130370+0000","last_became_peered":"2024-09-21T12:36:01.130370+0000","last_unstale":"2024-09-21T12:36:40.298922+0000","last_undegraded":"2024-09-21T12:36:40.298922+0000","last_fullsized":"2024-09-21T12:36:40.298922+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:34.842666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:34.842666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:56.753346+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7,3],"avail_no_missing":["0","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"431'5432","reported_seq":7597,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:46.913219+0000","last_change":"2024-09-21T12:36:12.108511+0000","last_active":"2024-09-21T12:36:46.913219+0000","last_peered":"2024-09-21T12:36:46.913219+0000","last_clean":"2024-09-21T12:36:46.913219+0000","last_became_active":"2024-09-21T12:36:12.104630+0000","last_became_peered":"2024-09-21T12:36:12.104630+0000","last_unstale":"2024-09-21T12:36:46.913219+0000","last_undegraded":"2024-09-21T12:36:46.913219+0000","last_fullsized":"2024-09-21T12:36:46.913219+0000","mapping_epoch":419,"log_start":"416'1200","ondisk_log_start":"416'1200","created":405,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4232,"log_dups_size":0,"ondisk_log_size":4232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:06.896900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355991552,"num_objects":5432,"num_object_clones":0,"num_object_copies":16296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5432,"num_write_kb":347648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1047,"num_bytes_recovered":68616192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"76'79","reported_seq":723,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.428626+0000","last_change":"2024-09-21T12:35:57.172169+0000","last_active":"2024-09-21T12:36:44.428626+0000","last_peered":"2024-09-21T12:36:44.428626+0000","last_clean":"2024-09-21T12:36:44.428626+0000","last_became_active":"2024-09-21T12:32:38.414877+0000","last_became_peered":"2024-09-21T12:32:38.414877+0000","last_unstale":"2024-09-21T12:36:44.428626+0000","last_undegraded":"2024-09-21T12:36:44.428626+0000","last_fullsized":"2024-09-21T12:36:44.428626+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:35:57.172103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:35:57.172103+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:06.814669+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":960,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.426242+0000","last_change":"2024-09-21T12:36:00.174641+0000","last_active":"2024-09-21T12:36:44.426242+0000","last_peered":"2024-09-21T12:36:44.426242+0000","last_clean":"2024-09-21T12:36:44.426242+0000","last_became_active":"2024-09-21T12:36:00.148392+0000","last_became_peered":"2024-09-21T12:36:00.148392+0000","last_unstale":"2024-09-21T12:36:44.426242+0000","last_undegraded":"2024-09-21T12:36:44.426242+0000","last_fullsized":"2024-09-21T12:36:44.426242+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'35","last_scrub_stamp":"2024-09-21T12:34:46.956618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:34:46.956618+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:17.045549+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":44,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.424386+0000","last_change":"2024-09-21T12:36:00.145588+0000","last_active":"2024-09-21T12:36:44.424386+0000","last_peered":"2024-09-21T12:36:44.424386+0000","last_clean":"2024-09-21T12:36:44.424386+0000","last_became_active":"2024-09-21T12:36:00.145287+0000","last_became_peered":"2024-09-21T12:36:00.145287+0000","last_unstale":"2024-09-21T12:36:44.424386+0000","last_undegraded":"2024-09-21T12:36:44.424386+0000","last_fullsized":"2024-09-21T12:36:44.424386+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:58.197610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:58.197610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:14.038278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":686,"reported_epoch":430,"state":"active+undersized","last_fresh":"2024-09-21T12:36:40.301055+0000","last_change":"2024-09-21T12:34:34.343106+0000","last_active":"2024-09-21T12:36:40.301055+0000","last_peered":"2024-09-21T12:36:40.301055+0000","last_clean":"2024-09-21T12:34:29.291366+0000","last_became_active":"2024-09-21T12:34:34.343106+0000","last_became_peered":"2024-09-21T12:34:34.343106+0000","last_unstale":"2024-09-21T12:36:40.301055+0000","last_undegraded":"2024-09-21T12:36:40.301055+0000","last_fullsized":"2024-09-21T12:34:34.333287+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:20.243774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:20.243774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:37.780982+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":770,"reported_epoch":431,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:44.425678+0000","last_change":"2024-09-21T12:36:28.804571+0000","last_active":"2024-09-21T12:36:44.425678+0000","last_peered":"2024-09-21T12:36:44.425678+0000","last_clean":"2024-09-21T12:36:44.425678+0000","last_became_active":"2024-09-21T12:36:01.135641+0000","last_became_peered":"2024-09-21T12:36:01.135641+0000","last_unstale":"2024-09-21T12:36:44.425678+0000","last_undegraded":"2024-09-21T12:36:44.425678+0000","last_fullsized":"2024-09-21T12:36:44.425678+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:28.804459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:28.804459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:45.512216+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,6],"avail_no_missing":["7","3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"430'5139","reported_seq":5196,"reported_epoch":430,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:43.397019+0000","last_change":"2024-09-21T12:35:53.511573+0000","last_active":"2024-09-21T12:36:43.397019+0000","last_peered":"2024-09-21T12:36:43.397019+0000","last_clean":"2024-09-21T12:35:46.767214+0000","last_became_active":"2024-09-21T12:35:47.793577+0000","last_became_peered":"2024-09-21T12:35:47.793577+0000","last_unstale":"2024-09-21T12:36:43.397019+0000","last_undegraded":"2024-09-21T12:35:53.504929+0000","last_fullsized":"2024-09-21T12:35:47.782699+0000","mapping_epoch":405,"log_start":"411'523","ondisk_log_start":"411'523","created":405,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4616,"log_dups_size":0,"ondisk_log_size":4616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:01.401191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336789504,"num_objects":5139,"num_object_clones":0,"num_object_copies":15417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5139,"num_write_kb":328896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":5139}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"76'79","reported_seq":748,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.121619+0000","last_change":"2024-09-21T12:36:31.563764+0000","last_active":"2024-09-21T12:36:47.121619+0000","last_peered":"2024-09-21T12:36:47.121619+0000","last_clean":"2024-09-21T12:36:47.121619+0000","last_became_active":"2024-09-21T12:27:14.951375+0000","last_became_peered":"2024-09-21T12:27:14.951375+0000","last_unstale":"2024-09-21T12:36:47.121619+0000","last_undegraded":"2024-09-21T12:36:47.121619+0000","last_fullsized":"2024-09-21T12:36:47.121619+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:31.563694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:31.563694+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:35.930543+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":195,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.420047+0000","last_change":"2024-09-21T12:35:12.197153+0000","last_active":"2024-09-21T12:36:44.420047+0000","last_peered":"2024-09-21T12:36:44.420047+0000","last_clean":"2024-09-21T12:36:44.420047+0000","last_became_active":"2024-09-21T12:32:38.421609+0000","last_became_peered":"2024-09-21T12:32:38.421609+0000","last_unstale":"2024-09-21T12:36:44.420047+0000","last_undegraded":"2024-09-21T12:36:44.420047+0000","last_fullsized":"2024-09-21T12:36:44.420047+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T12:35:12.197060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:12.197060+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":44,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.423613+0000","last_change":"2024-09-21T12:36:00.151126+0000","last_active":"2024-09-21T12:36:44.423613+0000","last_peered":"2024-09-21T12:36:44.423613+0000","last_clean":"2024-09-21T12:36:44.423613+0000","last_became_active":"2024-09-21T12:36:00.150832+0000","last_became_peered":"2024-09-21T12:36:00.150832+0000","last_unstale":"2024-09-21T12:36:44.423613+0000","last_undegraded":"2024-09-21T12:36:44.423613+0000","last_fullsized":"2024-09-21T12:36:44.423613+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:19.611537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:19.611537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":108,"reported_epoch":430,"state":"active+undersized","last_fresh":"2024-09-21T12:36:40.295536+0000","last_change":"2024-09-21T12:34:34.333741+0000","last_active":"2024-09-21T12:36:40.295536+0000","last_peered":"2024-09-21T12:36:40.295536+0000","last_clean":"2024-09-21T12:34:29.289730+0000","last_became_active":"2024-09-21T12:34:34.333741+0000","last_became_peered":"2024-09-21T12:34:34.333741+0000","last_unstale":"2024-09-21T12:36:40.295536+0000","last_undegraded":"2024-09-21T12:36:40.295536+0000","last_fullsized":"2024-09-21T12:34:34.330975+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:23.769748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:32:23.769748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:04.957773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":810,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.119270+0000","last_change":"2024-09-21T12:36:00.139665+0000","last_active":"2024-09-21T12:36:47.119270+0000","last_peered":"2024-09-21T12:36:47.119270+0000","last_clean":"2024-09-21T12:36:47.119270+0000","last_became_active":"2024-09-21T12:36:00.139300+0000","last_became_peered":"2024-09-21T12:36:00.139300+0000","last_unstale":"2024-09-21T12:36:47.119270+0000","last_undegraded":"2024-09-21T12:36:47.119270+0000","last_fullsized":"2024-09-21T12:36:47.119270+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:04.876656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:34:04.876656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"431'5560","reported_seq":5623,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:46.602661+0000","last_change":"2024-09-21T12:35:47.849125+0000","last_active":"2024-09-21T12:36:46.602661+0000","last_peered":"2024-09-21T12:36:46.602661+0000","last_clean":"2024-09-21T12:36:46.602661+0000","last_became_active":"2024-09-21T12:35:47.848708+0000","last_became_peered":"2024-09-21T12:35:47.848708+0000","last_unstale":"2024-09-21T12:36:46.602661+0000","last_undegraded":"2024-09-21T12:36:46.602661+0000","last_fullsized":"2024-09-21T12:36:46.602661+0000","mapping_epoch":405,"log_start":"416'1700","ondisk_log_start":"416'1700","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3860,"log_dups_size":0,"ondisk_log_size":3860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:47.277063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364380160,"num_objects":5560,"num_object_clones":0,"num_object_copies":16680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5560,"num_write_kb":355840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"76'79","reported_seq":224,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.424702+0000","last_change":"2024-09-21T12:36:36.859482+0000","last_active":"2024-09-21T12:36:44.424702+0000","last_peered":"2024-09-21T12:36:44.424702+0000","last_clean":"2024-09-21T12:36:44.424702+0000","last_became_active":"2024-09-21T12:36:01.141807+0000","last_became_peered":"2024-09-21T12:36:01.141807+0000","last_unstale":"2024-09-21T12:36:44.424702+0000","last_undegraded":"2024-09-21T12:36:44.424702+0000","last_fullsized":"2024-09-21T12:36:44.424702+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:36.859412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:36.859412+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:54.672933+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":325,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.427583+0000","last_change":"2024-09-21T12:36:00.171576+0000","last_active":"2024-09-21T12:36:44.427583+0000","last_peered":"2024-09-21T12:36:44.427583+0000","last_clean":"2024-09-21T12:36:44.427583+0000","last_became_active":"2024-09-21T12:36:00.137597+0000","last_became_peered":"2024-09-21T12:36:00.137597+0000","last_unstale":"2024-09-21T12:36:44.427583+0000","last_undegraded":"2024-09-21T12:36:44.427583+0000","last_fullsized":"2024-09-21T12:36:44.427583+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:35:52.226406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:52.226406+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:52.655311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":50,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.424983+0000","last_change":"2024-09-21T12:36:00.161052+0000","last_active":"2024-09-21T12:36:44.424983+0000","last_peered":"2024-09-21T12:36:44.424983+0000","last_clean":"2024-09-21T12:36:44.424983+0000","last_became_active":"2024-09-21T12:36:00.148899+0000","last_became_peered":"2024-09-21T12:36:00.148899+0000","last_unstale":"2024-09-21T12:36:44.424983+0000","last_undegraded":"2024-09-21T12:36:44.424983+0000","last_fullsized":"2024-09-21T12:36:44.424983+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-21T12:35:51.961868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:51.961868+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:54.987102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"415'16","reported_seq":58,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.423372+0000","last_change":"2024-09-21T12:36:00.165364+0000","last_active":"2024-09-21T12:36:44.423372+0000","last_peered":"2024-09-21T12:36:44.423372+0000","last_clean":"2024-09-21T12:36:44.423372+0000","last_became_active":"2024-09-21T12:36:00.143091+0000","last_became_peered":"2024-09-21T12:36:00.143091+0000","last_unstale":"2024-09-21T12:36:44.423372+0000","last_undegraded":"2024-09-21T12:36:44.423372+0000","last_fullsized":"2024-09-21T12:36:44.423372+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'14","last_scrub_stamp":"2024-09-21T12:35:06.542700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:35:06.542700+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":492,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.426081+0000","last_change":"2024-09-21T12:36:34.998909+0000","last_active":"2024-09-21T12:36:44.426081+0000","last_peered":"2024-09-21T12:36:44.426081+0000","last_clean":"2024-09-21T12:36:44.426081+0000","last_became_active":"2024-09-21T12:26:54.089155+0000","last_became_peered":"2024-09-21T12:26:54.089155+0000","last_unstale":"2024-09-21T12:36:44.426081+0000","last_undegraded":"2024-09-21T12:36:44.426081+0000","last_fullsized":"2024-09-21T12:36:44.426081+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:34.998795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:34.998795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:47.855846+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"76'79","reported_seq":522,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.427767+0000","last_change":"2024-09-21T12:32:26.698667+0000","last_active":"2024-09-21T12:36:44.427767+0000","last_peered":"2024-09-21T12:36:44.427767+0000","last_clean":"2024-09-21T12:36:44.427767+0000","last_became_active":"2024-09-21T12:31:39.060422+0000","last_became_peered":"2024-09-21T12:31:39.060422+0000","last_unstale":"2024-09-21T12:36:44.427767+0000","last_undegraded":"2024-09-21T12:36:44.427767+0000","last_fullsized":"2024-09-21T12:36:44.427767+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:26.698622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:26.698622+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:01.158707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":839,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.426252+0000","last_change":"2024-09-21T12:35:59.183091+0000","last_active":"2024-09-21T12:36:44.426252+0000","last_peered":"2024-09-21T12:36:44.426252+0000","last_clean":"2024-09-21T12:36:44.426252+0000","last_became_active":"2024-09-21T12:32:02.026693+0000","last_became_peered":"2024-09-21T12:32:02.026693+0000","last_unstale":"2024-09-21T12:36:44.426252+0000","last_undegraded":"2024-09-21T12:36:44.426252+0000","last_fullsized":"2024-09-21T12:36:44.426252+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:59.183047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:59.183047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:03.346089+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":297,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.120245+0000","last_change":"2024-09-21T12:36:01.139003+0000","last_active":"2024-09-21T12:36:47.120245+0000","last_peered":"2024-09-21T12:36:47.120245+0000","last_clean":"2024-09-21T12:36:47.120245+0000","last_became_active":"2024-09-21T12:36:01.138626+0000","last_became_peered":"2024-09-21T12:36:01.138626+0000","last_unstale":"2024-09-21T12:36:47.120245+0000","last_undegraded":"2024-09-21T12:36:47.120245+0000","last_fullsized":"2024-09-21T12:36:47.120245+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T12:34:09.872320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:34:09.872320+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":487,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.120693+0000","last_change":"2024-09-21T12:36:42.510836+0000","last_active":"2024-09-21T12:36:47.120693+0000","last_peered":"2024-09-21T12:36:47.120693+0000","last_clean":"2024-09-21T12:36:47.120693+0000","last_became_active":"2024-09-21T12:26:54.084109+0000","last_became_peered":"2024-09-21T12:26:54.084109+0000","last_unstale":"2024-09-21T12:36:47.120693+0000","last_undegraded":"2024-09-21T12:36:47.120693+0000","last_fullsized":"2024-09-21T12:36:47.120693+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:42.510764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:42.510764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:51.665123+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":44,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.424295+0000","last_change":"2024-09-21T12:36:00.149496+0000","last_active":"2024-09-21T12:36:44.424295+0000","last_peered":"2024-09-21T12:36:44.424295+0000","last_clean":"2024-09-21T12:36:44.424295+0000","last_became_active":"2024-09-21T12:36:00.149086+0000","last_became_peered":"2024-09-21T12:36:00.149086+0000","last_unstale":"2024-09-21T12:36:44.424295+0000","last_undegraded":"2024-09-21T12:36:44.424295+0000","last_fullsized":"2024-09-21T12:36:44.424295+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:14.566963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:14.566963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'79","reported_seq":770,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.432814+0000","last_change":"2024-09-21T12:36:39.411851+0000","last_active":"2024-09-21T12:36:44.432814+0000","last_peered":"2024-09-21T12:36:44.432814+0000","last_clean":"2024-09-21T12:36:44.432814+0000","last_became_active":"2024-09-21T12:32:38.425495+0000","last_became_peered":"2024-09-21T12:32:38.425495+0000","last_unstale":"2024-09-21T12:36:44.432814+0000","last_undegraded":"2024-09-21T12:36:44.432814+0000","last_fullsized":"2024-09-21T12:36:44.432814+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:39.411785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:39.411785+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:43.123682+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":860,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-21T12:36:40.297795+0000","last_change":"2024-09-21T12:36:00.142630+0000","last_active":"2024-09-21T12:36:40.297795+0000","last_peered":"2024-09-21T12:36:40.297795+0000","last_clean":"2024-09-21T12:36:40.297795+0000","last_became_active":"2024-09-21T12:36:00.142274+0000","last_became_peered":"2024-09-21T12:36:00.142274+0000","last_unstale":"2024-09-21T12:36:40.297795+0000","last_undegraded":"2024-09-21T12:36:40.297795+0000","last_fullsized":"2024-09-21T12:36:40.297795+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:32:19.786928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:19.786928+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:01.953764+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":44,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.423416+0000","last_change":"2024-09-21T12:36:00.140779+0000","last_active":"2024-09-21T12:36:44.423416+0000","last_peered":"2024-09-21T12:36:44.423416+0000","last_clean":"2024-09-21T12:36:44.423416+0000","last_became_active":"2024-09-21T12:36:00.140477+0000","last_became_peered":"2024-09-21T12:36:00.140477+0000","last_unstale":"2024-09-21T12:36:44.423416+0000","last_undegraded":"2024-09-21T12:36:44.423416+0000","last_fullsized":"2024-09-21T12:36:44.423416+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:27.798236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:27.798236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:51.374052+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":791,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.120001+0000","last_change":"2024-09-21T12:36:00.151444+0000","last_active":"2024-09-21T12:36:47.120001+0000","last_peered":"2024-09-21T12:36:47.120001+0000","last_clean":"2024-09-21T12:36:47.120001+0000","last_became_active":"2024-09-21T12:36:00.150975+0000","last_became_peered":"2024-09-21T12:36:00.150975+0000","last_unstale":"2024-09-21T12:36:47.120001+0000","last_undegraded":"2024-09-21T12:36:47.120001+0000","last_fullsized":"2024-09-21T12:36:47.120001+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:58.720834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:32:58.720834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:20.560140+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":766,"reported_epoch":431,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:44.420462+0000","last_change":"2024-09-21T12:36:01.138863+0000","last_active":"2024-09-21T12:36:44.420462+0000","last_peered":"2024-09-21T12:36:44.420462+0000","last_clean":"2024-09-21T12:36:44.420462+0000","last_became_active":"2024-09-21T12:36:01.138205+0000","last_became_peered":"2024-09-21T12:36:01.138205+0000","last_unstale":"2024-09-21T12:36:44.420462+0000","last_undegraded":"2024-09-21T12:36:44.420462+0000","last_fullsized":"2024-09-21T12:36:44.420462+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:55.936030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:55.936030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:09.145667+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7,0],"avail_no_missing":["4","0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.10","version":"76'79","reported_seq":52,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.425476+0000","last_change":"2024-09-21T12:36:00.829177+0000","last_active":"2024-09-21T12:36:44.425476+0000","last_peered":"2024-09-21T12:36:44.425476+0000","last_clean":"2024-09-21T12:36:44.425476+0000","last_became_active":"2024-09-21T12:36:00.140266+0000","last_became_peered":"2024-09-21T12:36:00.140266+0000","last_unstale":"2024-09-21T12:36:44.425476+0000","last_undegraded":"2024-09-21T12:36:44.425476+0000","last_fullsized":"2024-09-21T12:36:44.425476+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:00.829119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:00.829119+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:19.526550+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":544,"reported_epoch":431,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:44.424963+0000","last_change":"2024-09-21T12:36:01.130787+0000","last_active":"2024-09-21T12:36:44.424963+0000","last_peered":"2024-09-21T12:36:44.424963+0000","last_clean":"2024-09-21T12:36:44.424963+0000","last_became_active":"2024-09-21T12:36:01.129843+0000","last_became_peered":"2024-09-21T12:36:01.129843+0000","last_unstale":"2024-09-21T12:36:44.424963+0000","last_undegraded":"2024-09-21T12:36:44.424963+0000","last_fullsized":"2024-09-21T12:36:44.424963+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:35:54.238498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:54.238498+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:14.079285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5,6],"avail_no_missing":["7","5","6"],"object_location_counts":[{"shards":"5,6,7","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":844,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-21T12:36:40.297576+0000","last_change":"2024-09-21T12:36:00.127553+0000","last_active":"2024-09-21T12:36:40.297576+0000","last_peered":"2024-09-21T12:36:40.297576+0000","last_clean":"2024-09-21T12:36:40.297576+0000","last_became_active":"2024-09-21T12:36:00.127225+0000","last_became_peered":"2024-09-21T12:36:00.127225+0000","last_unstale":"2024-09-21T12:36:40.297576+0000","last_undegraded":"2024-09-21T12:36:40.297576+0000","last_fullsized":"2024-09-21T12:36:40.297576+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:22.225122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:22.225122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:41.794207+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"416'15","reported_seq":341,"reported_epoch":432,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:47.119010+0000","last_change":"2024-09-21T12:36:01.139488+0000","last_active":"2024-09-21T12:36:47.119010+0000","last_peered":"2024-09-21T12:36:47.119010+0000","last_clean":"2024-09-21T12:36:47.119010+0000","last_became_active":"2024-09-21T12:36:01.138017+0000","last_became_peered":"2024-09-21T12:36:01.138017+0000","last_unstale":"2024-09-21T12:36:47.119010+0000","last_undegraded":"2024-09-21T12:36:47.119010+0000","last_fullsized":"2024-09-21T12:36:47.119010+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'10","last_scrub_stamp":"2024-09-21T12:34:14.003545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:14.003545+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7,1],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":187,"reported_epoch":431,"state":"active+undersized","last_fresh":"2024-09-21T12:36:44.420781+0000","last_change":"2024-09-21T12:34:34.340859+0000","last_active":"2024-09-21T12:36:44.420781+0000","last_peered":"2024-09-21T12:36:44.420781+0000","last_clean":"2024-09-21T12:34:29.279088+0000","last_became_active":"2024-09-21T12:34:34.340859+0000","last_became_peered":"2024-09-21T12:34:34.340859+0000","last_unstale":"2024-09-21T12:36:44.420781+0000","last_undegraded":"2024-09-21T12:36:44.420781+0000","last_fullsized":"2024-09-21T12:34:34.330912+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:05.370263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:05.370263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:28.761559+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"76'79","reported_seq":611,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-21T12:36:40.299777+0000","last_change":"2024-09-21T12:35:17.231922+0000","last_active":"2024-09-21T12:36:40.299777+0000","last_peered":"2024-09-21T12:36:40.299777+0000","last_clean":"2024-09-21T12:36:40.299777+0000","last_became_active":"2024-09-21T12:32:02.034387+0000","last_became_peered":"2024-09-21T12:32:02.034387+0000","last_unstale":"2024-09-21T12:36:40.299777+0000","last_undegraded":"2024-09-21T12:36:40.299777+0000","last_fullsized":"2024-09-21T12:36:40.299777+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:35:17.231860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:35:17.231860+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:40.387153+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":879,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.121771+0000","last_change":"2024-09-21T12:36:36.521884+0000","last_active":"2024-09-21T12:36:47.121771+0000","last_peered":"2024-09-21T12:36:47.121771+0000","last_clean":"2024-09-21T12:36:47.121771+0000","last_became_active":"2024-09-21T12:36:00.155033+0000","last_became_peered":"2024-09-21T12:36:00.155033+0000","last_unstale":"2024-09-21T12:36:47.121771+0000","last_undegraded":"2024-09-21T12:36:47.121771+0000","last_fullsized":"2024-09-21T12:36:47.121771+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:36:36.521810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:36.521810+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:05.336949+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":810,"reported_epoch":431,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:44.424670+0000","last_change":"2024-09-21T12:36:01.139279+0000","last_active":"2024-09-21T12:36:44.424670+0000","last_peered":"2024-09-21T12:36:44.424670+0000","last_clean":"2024-09-21T12:36:44.424670+0000","last_became_active":"2024-09-21T12:36:01.135527+0000","last_became_peered":"2024-09-21T12:36:01.135527+0000","last_unstale":"2024-09-21T12:36:44.424670+0000","last_undegraded":"2024-09-21T12:36:44.424670+0000","last_fullsized":"2024-09-21T12:36:44.424670+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:32.630261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:32.630261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:17.760444+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4,6],"avail_no_missing":["7","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":999,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.428338+0000","last_change":"2024-09-21T12:36:29.482398+0000","last_active":"2024-09-21T12:36:44.428338+0000","last_peered":"2024-09-21T12:36:44.428338+0000","last_clean":"2024-09-21T12:36:44.428338+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:36:44.428338+0000","last_undegraded":"2024-09-21T12:36:44.428338+0000","last_fullsized":"2024-09-21T12:36:44.428338+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:36:29.482171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:29.482171+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:37.228329+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":800,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-21T12:36:40.296894+0000","last_change":"2024-09-21T12:36:00.149113+0000","last_active":"2024-09-21T12:36:40.296894+0000","last_peered":"2024-09-21T12:36:40.296894+0000","last_clean":"2024-09-21T12:36:40.296894+0000","last_became_active":"2024-09-21T12:36:00.148332+0000","last_became_peered":"2024-09-21T12:36:00.148332+0000","last_unstale":"2024-09-21T12:36:40.296894+0000","last_undegraded":"2024-09-21T12:36:40.296894+0000","last_fullsized":"2024-09-21T12:36:40.296894+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:26.208487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:26.208487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"76'79","reported_seq":469,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.423192+0000","last_change":"2024-09-21T12:36:28.244064+0000","last_active":"2024-09-21T12:36:44.423192+0000","last_peered":"2024-09-21T12:36:44.423192+0000","last_clean":"2024-09-21T12:36:44.423192+0000","last_became_active":"2024-09-21T12:27:16.940347+0000","last_became_peered":"2024-09-21T12:27:16.940347+0000","last_unstale":"2024-09-21T12:36:44.423192+0000","last_undegraded":"2024-09-21T12:36:44.423192+0000","last_fullsized":"2024-09-21T12:36:44.423192+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:28.243938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:28.243938+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:53.995371+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":195,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.420521+0000","last_change":"2024-09-21T12:34:00.191190+0000","last_active":"2024-09-21T12:36:44.420521+0000","last_peered":"2024-09-21T12:36:44.420521+0000","last_clean":"2024-09-21T12:36:44.420521+0000","last_became_active":"2024-09-21T12:32:38.421008+0000","last_became_peered":"2024-09-21T12:32:38.421008+0000","last_unstale":"2024-09-21T12:36:44.420521+0000","last_undegraded":"2024-09-21T12:36:44.420521+0000","last_fullsized":"2024-09-21T12:36:44.420521+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:34:00.191106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:34:00.191106+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":195,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.420369+0000","last_change":"2024-09-21T12:36:01.124070+0000","last_active":"2024-09-21T12:36:44.420369+0000","last_peered":"2024-09-21T12:36:44.420369+0000","last_clean":"2024-09-21T12:36:44.420369+0000","last_became_active":"2024-09-21T12:36:01.123399+0000","last_became_peered":"2024-09-21T12:36:01.123399+0000","last_unstale":"2024-09-21T12:36:44.420369+0000","last_undegraded":"2024-09-21T12:36:44.420369+0000","last_fullsized":"2024-09-21T12:36:44.420369+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:30.406915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:30.406915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":802,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.422190+0000","last_change":"2024-09-21T12:36:34.472774+0000","last_active":"2024-09-21T12:36:44.422190+0000","last_peered":"2024-09-21T12:36:44.422190+0000","last_clean":"2024-09-21T12:36:44.422190+0000","last_became_active":"2024-09-21T12:31:50.040754+0000","last_became_peered":"2024-09-21T12:31:50.040754+0000","last_unstale":"2024-09-21T12:36:44.422190+0000","last_undegraded":"2024-09-21T12:36:44.422190+0000","last_fullsized":"2024-09-21T12:36:44.422190+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:34.472702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:34.472702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:47.750215+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":254,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.119137+0000","last_change":"2024-09-21T12:31:53.359206+0000","last_active":"2024-09-21T12:36:47.119137+0000","last_peered":"2024-09-21T12:36:47.119137+0000","last_clean":"2024-09-21T12:36:47.119137+0000","last_became_active":"2024-09-21T12:31:50.036150+0000","last_became_peered":"2024-09-21T12:31:50.036150+0000","last_unstale":"2024-09-21T12:36:47.119137+0000","last_undegraded":"2024-09-21T12:36:47.119137+0000","last_fullsized":"2024-09-21T12:36:47.119137+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:53.359159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:31:53.359159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:17.628056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"76'79","reported_seq":502,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:45.476276+0000","last_change":"2024-09-21T12:36:45.476276+0000","last_active":"2024-09-21T12:36:45.476276+0000","last_peered":"2024-09-21T12:36:45.476276+0000","last_clean":"2024-09-21T12:36:45.476276+0000","last_became_active":"2024-09-21T12:27:16.921880+0000","last_became_peered":"2024-09-21T12:27:16.921880+0000","last_unstale":"2024-09-21T12:36:45.476276+0000","last_undegraded":"2024-09-21T12:36:45.476276+0000","last_fullsized":"2024-09-21T12:36:45.476276+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:45.476087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:45.476087+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:11.672214+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":282,"reported_epoch":431,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:44.421471+0000","last_change":"2024-09-21T12:34:34.345279+0000","last_active":"2024-09-21T12:36:44.421471+0000","last_peered":"2024-09-21T12:36:44.421471+0000","last_clean":"2024-09-21T12:34:29.286797+0000","last_became_active":"2024-09-21T12:34:34.345279+0000","last_became_peered":"2024-09-21T12:34:34.345279+0000","last_unstale":"2024-09-21T12:36:44.421471+0000","last_undegraded":"2024-09-21T12:34:34.331120+0000","last_fullsized":"2024-09-21T12:34:34.331029+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:33:02.544885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:02.544885+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:44.914352+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":44,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.424593+0000","last_change":"2024-09-21T12:36:00.147458+0000","last_active":"2024-09-21T12:36:44.424593+0000","last_peered":"2024-09-21T12:36:44.424593+0000","last_clean":"2024-09-21T12:36:44.424593+0000","last_became_active":"2024-09-21T12:36:00.147063+0000","last_became_peered":"2024-09-21T12:36:00.147063+0000","last_unstale":"2024-09-21T12:36:44.424593+0000","last_undegraded":"2024-09-21T12:36:44.424593+0000","last_fullsized":"2024-09-21T12:36:44.424593+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:28.849200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:28.849200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":786,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.119351+0000","last_change":"2024-09-21T12:34:11.934886+0000","last_active":"2024-09-21T12:36:47.119351+0000","last_peered":"2024-09-21T12:36:47.119351+0000","last_clean":"2024-09-21T12:36:47.119351+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:36:47.119351+0000","last_undegraded":"2024-09-21T12:36:47.119351+0000","last_fullsized":"2024-09-21T12:36:47.119351+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:11.934826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:11.934826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":773,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-21T12:36:40.295759+0000","last_change":"2024-09-21T12:35:59.959597+0000","last_active":"2024-09-21T12:36:40.295759+0000","last_peered":"2024-09-21T12:36:40.295759+0000","last_clean":"2024-09-21T12:36:40.295759+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:36:40.295759+0000","last_undegraded":"2024-09-21T12:36:40.295759+0000","last_fullsized":"2024-09-21T12:36:40.295759+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:59.959545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:59.959545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:04.514896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"76'79","reported_seq":279,"reported_epoch":431,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:44.423743+0000","last_change":"2024-09-21T12:36:43.792972+0000","last_active":"2024-09-21T12:36:44.423743+0000","last_peered":"2024-09-21T12:36:44.423743+0000","last_clean":"2024-09-21T12:36:44.423743+0000","last_became_active":"2024-09-21T12:36:01.135246+0000","last_became_peered":"2024-09-21T12:36:01.135246+0000","last_unstale":"2024-09-21T12:36:44.423743+0000","last_undegraded":"2024-09-21T12:36:44.423743+0000","last_fullsized":"2024-09-21T12:36:44.423743+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:43.792355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:43.792355+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:48.500764+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4,1],"avail_no_missing":["7","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":867,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.120000+0000","last_change":"2024-09-21T12:36:41.506309+0000","last_active":"2024-09-21T12:36:47.120000+0000","last_peered":"2024-09-21T12:36:47.120000+0000","last_clean":"2024-09-21T12:36:47.120000+0000","last_became_active":"2024-09-21T12:36:00.139682+0000","last_became_peered":"2024-09-21T12:36:00.139682+0000","last_unstale":"2024-09-21T12:36:47.120000+0000","last_undegraded":"2024-09-21T12:36:47.120000+0000","last_fullsized":"2024-09-21T12:36:47.120000+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:36:41.506231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:41.506231+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:10.569673+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":830,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.428433+0000","last_change":"2024-09-21T12:36:34.526618+0000","last_active":"2024-09-21T12:36:44.428433+0000","last_peered":"2024-09-21T12:36:44.428433+0000","last_clean":"2024-09-21T12:36:44.428433+0000","last_became_active":"2024-09-21T12:36:00.151584+0000","last_became_peered":"2024-09-21T12:36:00.151584+0000","last_unstale":"2024-09-21T12:36:44.428433+0000","last_undegraded":"2024-09-21T12:36:44.428433+0000","last_fullsized":"2024-09-21T12:36:44.428433+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:34.526551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:36:34.526551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:39.862324+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":44,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.425385+0000","last_change":"2024-09-21T12:36:00.145015+0000","last_active":"2024-09-21T12:36:44.425385+0000","last_peered":"2024-09-21T12:36:44.425385+0000","last_clean":"2024-09-21T12:36:44.425385+0000","last_became_active":"2024-09-21T12:36:00.144521+0000","last_became_peered":"2024-09-21T12:36:00.144521+0000","last_unstale":"2024-09-21T12:36:44.425385+0000","last_undegraded":"2024-09-21T12:36:44.425385+0000","last_fullsized":"2024-09-21T12:36:44.425385+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:47.316065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:47.316065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":782,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.119550+0000","last_change":"2024-09-21T12:35:51.628370+0000","last_active":"2024-09-21T12:36:47.119550+0000","last_peered":"2024-09-21T12:36:47.119550+0000","last_clean":"2024-09-21T12:36:47.119550+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:36:47.119550+0000","last_undegraded":"2024-09-21T12:36:47.119550+0000","last_fullsized":"2024-09-21T12:36:47.119550+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:51.628326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:51.628326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:05.095925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"76'79","reported_seq":44,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.423557+0000","last_change":"2024-09-21T12:36:00.151654+0000","last_active":"2024-09-21T12:36:44.423557+0000","last_peered":"2024-09-21T12:36:44.423557+0000","last_clean":"2024-09-21T12:36:44.423557+0000","last_became_active":"2024-09-21T12:36:00.151043+0000","last_became_peered":"2024-09-21T12:36:00.151043+0000","last_unstale":"2024-09-21T12:36:44.423557+0000","last_undegraded":"2024-09-21T12:36:44.423557+0000","last_fullsized":"2024-09-21T12:36:44.423557+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:35:51.976777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:35:51.976777+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:17.768008+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":802,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.123129+0000","last_change":"2024-09-21T12:35:14.789148+0000","last_active":"2024-09-21T12:36:47.123129+0000","last_peered":"2024-09-21T12:36:47.123129+0000","last_clean":"2024-09-21T12:36:47.123129+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:36:47.123129+0000","last_undegraded":"2024-09-21T12:36:47.123129+0000","last_fullsized":"2024-09-21T12:36:47.123129+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'8","last_scrub_stamp":"2024-09-21T12:35:14.789070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:14.789070+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":806,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-21T12:36:40.300453+0000","last_change":"2024-09-21T12:35:27.237430+0000","last_active":"2024-09-21T12:36:40.300453+0000","last_peered":"2024-09-21T12:36:40.300453+0000","last_clean":"2024-09-21T12:36:40.300453+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:36:40.300453+0000","last_undegraded":"2024-09-21T12:36:40.300453+0000","last_fullsized":"2024-09-21T12:36:40.300453+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:35:27.236980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:27.236980+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:49.341969+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":827,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.426081+0000","last_change":"2024-09-21T12:36:00.135914+0000","last_active":"2024-09-21T12:36:44.426081+0000","last_peered":"2024-09-21T12:36:44.426081+0000","last_clean":"2024-09-21T12:36:44.426081+0000","last_became_active":"2024-09-21T12:36:00.135537+0000","last_became_peered":"2024-09-21T12:36:00.135537+0000","last_unstale":"2024-09-21T12:36:44.426081+0000","last_undegraded":"2024-09-21T12:36:44.426081+0000","last_fullsized":"2024-09-21T12:36:44.426081+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:40.861493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:40.861493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:21.056349+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":806,"reported_epoch":431,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:44.425106+0000","last_change":"2024-09-21T12:36:38.838788+0000","last_active":"2024-09-21T12:36:44.425106+0000","last_peered":"2024-09-21T12:36:44.425106+0000","last_clean":"2024-09-21T12:36:44.425106+0000","last_became_active":"2024-09-21T12:36:01.134484+0000","last_became_peered":"2024-09-21T12:36:01.134484+0000","last_unstale":"2024-09-21T12:36:44.425106+0000","last_undegraded":"2024-09-21T12:36:44.425106+0000","last_fullsized":"2024-09-21T12:36:44.425106+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:38.838688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:38.838688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:55.540359+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4,3],"avail_no_missing":["7","3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"76'79","reported_seq":197,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.420267+0000","last_change":"2024-09-21T12:36:00.137642+0000","last_active":"2024-09-21T12:36:44.420267+0000","last_peered":"2024-09-21T12:36:44.420267+0000","last_clean":"2024-09-21T12:36:44.420267+0000","last_became_active":"2024-09-21T12:36:00.137290+0000","last_became_peered":"2024-09-21T12:36:00.137290+0000","last_unstale":"2024-09-21T12:36:44.420267+0000","last_undegraded":"2024-09-21T12:36:44.420267+0000","last_fullsized":"2024-09-21T12:36:44.420267+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:43.304606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:43.304606+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":808,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.422752+0000","last_change":"2024-09-21T12:35:48.922444+0000","last_active":"2024-09-21T12:36:44.422752+0000","last_peered":"2024-09-21T12:36:44.422752+0000","last_clean":"2024-09-21T12:36:44.422752+0000","last_became_active":"2024-09-21T12:32:38.426822+0000","last_became_peered":"2024-09-21T12:32:38.426822+0000","last_unstale":"2024-09-21T12:36:44.422752+0000","last_undegraded":"2024-09-21T12:36:44.422752+0000","last_fullsized":"2024-09-21T12:36:44.422752+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:48.922389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:48.922389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:56.376647+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":807,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T12:36:47.123352+0000","last_change":"2024-09-21T12:35:30.730640+0000","last_active":"2024-09-21T12:36:47.123352+0000","last_peered":"2024-09-21T12:36:47.123352+0000","last_clean":"2024-09-21T12:36:47.123352+0000","last_became_active":"2024-09-21T12:31:39.052486+0000","last_became_peered":"2024-09-21T12:31:39.052486+0000","last_unstale":"2024-09-21T12:36:47.123352+0000","last_undegraded":"2024-09-21T12:36:47.123352+0000","last_fullsized":"2024-09-21T12:36:47.123352+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:30.730574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:30.730574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:36:50.338601+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":820,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-09-21T12:36:44.427098+0000","last_change":"2024-09-21T12:34:44.919918+0000","last_active":"2024-09-21T12:36:44.427098+0000","last_peered":"2024-09-21T12:36:44.427098+0000","last_clean":"2024-09-21T12:36:44.427098+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:36:44.427098+0000","last_undegraded":"2024-09-21T12:36:44.427098+0000","last_fullsized":"2024-09-21T12:36:44.427098+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:34:44.919796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:34:44.919796+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:24.048666+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":826,"reported_epoch":430,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:40.296999+0000","last_change":"2024-09-21T12:36:28.911452+0000","last_active":"2024-09-21T12:36:40.296999+0000","last_peered":"2024-09-21T12:36:40.296999+0000","last_clean":"2024-09-21T12:36:40.296999+0000","last_became_active":"2024-09-21T12:36:01.130870+0000","last_became_peered":"2024-09-21T12:36:01.130870+0000","last_unstale":"2024-09-21T12:36:40.296999+0000","last_undegraded":"2024-09-21T12:36:40.296999+0000","last_fullsized":"2024-09-21T12:36:40.296999+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:28.911328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:28.911328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:34.674874+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7,1],"avail_no_missing":["0","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":5774704640,"num_objects":88115,"num_object_clones":0,"num_object_copies":264345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21675,"num_objects_misplaced":11154,"num_objects_unfound":0,"num_objects_dirty":88115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88115,"num_write_kb":5639360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4193,"num_bytes_recovered":274792448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15833694208,"data_stored":15833694208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64592,"ondisk_log_size":64592,"up":42,"acting":44,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":83,"acting":90,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106,"ondisk_log_size":106,"up":87,"acting":92,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":54,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1105,"num_read_kb":862,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":472,"num_bytes_recovered":9372,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":11216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":687,"ondisk_log_size":687,"up":87,"acting":88,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":598,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":88,"acting":90,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":53,"num_read_kb":48,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":617202,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":9,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":289522,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2048000,"data_stored":2033792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2449,"ondisk_log_size":2449,"up":86,"acting":89,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234186,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2847768,"kb_used_data":2112712,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":90897384,"statfs":{"total":95995035648,"available":93078921216,"internally_reserved":0,"allocated":2163417088,"data_stored":2162910619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":752679370},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463122,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3486472,"kb_used_data":2765304,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":90258680,"statfs":{"total":95995035648,"available":92424888320,"internally_reserved":0,"allocated":2831671296,"data_stored":2831180128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":738458054},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724757,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2090908,"kb_used_data":1405836,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":91654244,"statfs":{"total":95995035648,"available":93853945856,"internally_reserved":0,"allocated":1439576064,"data_stored":1439066435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953693,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1759324,"kb_used_data":1413004,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":91985828,"statfs":{"total":95995035648,"available":94193487872,"internally_reserved":0,"allocated":1446916096,"data_stored":1446404507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149923,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3351232,"kb_used_data":2539820,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":90393920,"statfs":{"total":95995035648,"available":92563374080,"internally_reserved":0,"allocated":2600775680,"data_stored":2600242245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":830863815},"hb_peers":[0,1,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411534,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2808848,"kb_used_data":2272320,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":90936304,"statfs":{"total":95995035648,"available":93118775296,"internally_reserved":0,"allocated":2326855680,"data_stored":2326574609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575087,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4004924,"kb_used_data":3161580,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":89740228,"statfs":{"total":95995035648,"available":91893993472,"internally_reserved":0,"allocated":3237457920,"data_stored":3236914970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":863566277},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738612,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2792148,"kb_used_data":1971008,"kb_used_omap":1,"kb_used_meta":821118,"kb_avail":90953004,"statfs":{"total":95995035648,"available":93135876096,"internally_reserved":0,"allocated":2018312192,"data_stored":2017805122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":840825285},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2038235136,"data_stored":2038235136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3241345024,"data_stored":3241345024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2619932672,"data_stored":2619932672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1450049536,"data_stored":1450049536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1447231488,"data_stored":1447231488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2870149120,"data_stored":2870149120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2166751232,"data_stored":2166751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:36:50.612 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T12:36:50.612 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pg_num 51 2024-09-21T12:36:50.619 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:50.649 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:50.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:50.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:50.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:50 smithi183 ceph-mon[33277]: from='client.26228 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:36:50.751 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:50.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:50.795 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:50.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:50.896 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:50.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:50.923 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:50.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:51.024 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:51.065 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:51.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:51.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:51.167 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:51.204 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:51.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:51.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:51.306 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:51.346 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:51.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:51.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:51.437 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:36:51.437 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:36:51.448 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:51.477 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:36:51.477 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:36:51.479 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T12:36:51.500 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:51.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:51.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:51.508 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 58 15 94914 94899 102.248 90.9375 0.0124084 0.00977544 2024-09-21T12:36:51.515 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:36:51.515 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:36:51.553 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:36:51.554 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:36:51.590 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:36:51.591 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:36:51.602 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:51.626 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:36:51.626 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:36:51.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:51.637 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:51.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:51.660 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:36:51.660 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:36:51.688 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:51 smithi114 ceph-mon[40867]: pgmap v846: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 75 MiB/s wr, 1.19k op/s; 22177/266283 objects degraded (8.328%); 11162/266283 objects misplaced (4.192%) 2024-09-21T12:36:51.688 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:51 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:36:51.688 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:51 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:36:51.688 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:51 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/4083880627' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:36:51.688 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:51 smithi114 ceph-mon[30324]: pgmap v846: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 75 MiB/s wr, 1.19k op/s; 22177/266283 objects degraded (8.328%); 11162/266283 objects misplaced (4.192%) 2024-09-21T12:36:51.688 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:51 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:36:51.688 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:51 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:36:51.688 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:51 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4083880627' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:36:51.689 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:36:51.690 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-21T12:36:51.690 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.690 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.26147.0:95121 11.a 11:53372a0c:::benchmark_data_smithi114_187622_object95120:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-21T12:36:51.690 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:36:51.670883+0000", 2024-09-21T12:36:51.690 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.015016763000000001, 2024-09-21T12:36:51.690 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.015046868999999999, 2024-09-21T12:36:51.690 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:36:51.690 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:36:51.690 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:36:51.690 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:36:51.690 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.26147", 2024-09-21T12:36:51.691 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:36:51.691 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 95121 2024-09-21T12:36:51.691 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.691 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:36:51.691 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.691 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:36:51.691 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.670883+0000", 2024-09-21T12:36:51.691 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:51.691 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.691 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.691 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:36:51.691 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.670883+0000", 2024-09-21T12:36:51.691 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:51.691 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.692 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.692 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:36:51.692 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.670884+0000", 2024-09-21T12:36:51.692 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1039999999999999e-06 2024-09-21T12:36:51.692 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.692 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.692 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:36:51.692 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.670907+0000", 2024-09-21T12:36:51.692 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3493999999999999e-05 2024-09-21T12:36:51.692 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.692 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.692 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:36:51.692 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.670916+0000", 2024-09-21T12:36:51.693 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6719999999999993e-06 2024-09-21T12:36:51.693 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.693 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.693 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:36:51.693 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.670922+0000", 2024-09-21T12:36:51.693 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2110000000000003e-06 2024-09-21T12:36:51.693 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.693 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.693 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:36:51.693 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.675481+0000", 2024-09-21T12:36:51.693 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0045589589999999996 2024-09-21T12:36:51.694 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.694 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.694 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:36:51.694 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.675620+0000", 2024-09-21T12:36:51.694 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013908799999999999 2024-09-21T12:36:51.694 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.694 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.694 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 5,6", 2024-09-21T12:36:51.694 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.675763+0000", 2024-09-21T12:36:51.694 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000142638 2024-09-21T12:36:51.694 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.694 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.695 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:36:51.695 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.681953+0000", 2024-09-21T12:36:51.695 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0061898969999999998 2024-09-21T12:36:51.695 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.695 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.695 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:36:51.695 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.681985+0000", 2024-09-21T12:36:51.695 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2558000000000002e-05 2024-09-21T12:36:51.695 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:51.695 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:51.695 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:51.696 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.696 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.696 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.26147.0:95128 11.2 11:4c5a2940:::benchmark_data_smithi114_187622_object95127:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-21T12:36:51.696 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:36:51.677288+0000", 2024-09-21T12:36:51.696 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0086113820000000008, 2024-09-21T12:36:51.696 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0087178120000000001, 2024-09-21T12:36:51.696 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:36:51.696 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:36:51.696 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:36:51.696 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:36:51.696 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.26147", 2024-09-21T12:36:51.696 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:36:51.696 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 95128 2024-09-21T12:36:51.696 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.697 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:36:51.697 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.697 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:36:51.697 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.677288+0000", 2024-09-21T12:36:51.697 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:51.697 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.697 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.697 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:36:51.697 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.677288+0000", 2024-09-21T12:36:51.697 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:51.697 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.697 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.697 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:36:51.697 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.677289+0000", 2024-09-21T12:36:51.697 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.206e-06 2024-09-21T12:36:51.698 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.698 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.698 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:36:51.698 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.677312+0000", 2024-09-21T12:36:51.698 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3351999999999999e-05 2024-09-21T12:36:51.698 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.698 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.698 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:36:51.698 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.677321+0000", 2024-09-21T12:36:51.698 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.8529999999999998e-06 2024-09-21T12:36:51.698 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.698 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.698 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:36:51.698 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.677328+0000", 2024-09-21T12:36:51.699 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3239999999999999e-06 2024-09-21T12:36:51.699 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.699 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.699 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:36:51.699 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.682010+0000", 2024-09-21T12:36:51.699 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0046822909999999999 2024-09-21T12:36:51.699 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.699 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.699 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:36:51.699 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.682146+0000", 2024-09-21T12:36:51.699 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000136508 2024-09-21T12:36:51.699 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.699 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.699 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0", 2024-09-21T12:36:51.699 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.682275+0000", 2024-09-21T12:36:51.700 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012870899999999999 2024-09-21T12:36:51.700 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:51.700 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:51.700 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:51.700 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.700 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.700 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.26147.0:95134 11.2 11:47d474d2:::benchmark_data_smithi114_187622_object95133:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-21T12:36:51.700 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:36:51.681042+0000", 2024-09-21T12:36:51.700 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0048574559999999996, 2024-09-21T12:36:51.700 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0050350000000000004, 2024-09-21T12:36:51.700 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:36:51.700 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:36:51.700 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:36:51.700 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:36:51.701 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.26147", 2024-09-21T12:36:51.701 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:36:51.701 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 95134 2024-09-21T12:36:51.701 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.701 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:36:51.701 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.701 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:36:51.701 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.681042+0000", 2024-09-21T12:36:51.701 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:51.701 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.701 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.701 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:36:51.701 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.681042+0000", 2024-09-21T12:36:51.702 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:51.702 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.702 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.702 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:36:51.702 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.681044+0000", 2024-09-21T12:36:51.702 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9920000000000002e-06 2024-09-21T12:36:51.702 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.702 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.702 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:36:51.702 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.681074+0000", 2024-09-21T12:36:51.702 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.9784000000000001e-05 2024-09-21T12:36:51.702 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.702 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.702 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:36:51.703 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.681087+0000", 2024-09-21T12:36:51.703 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3587000000000001e-05 2024-09-21T12:36:51.703 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.703 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.703 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:36:51.703 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.681095+0000", 2024-09-21T12:36:51.703 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9359999999999999e-06 2024-09-21T12:36:51.703 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.703 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.703 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:36:51.703 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.684583+0000", 2024-09-21T12:36:51.703 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0034874699999999999 2024-09-21T12:36:51.703 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.703 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.704 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:36:51.704 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.684721+0000", 2024-09-21T12:36:51.704 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000138283 2024-09-21T12:36:51.704 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.704 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.704 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0", 2024-09-21T12:36:51.704 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.684864+0000", 2024-09-21T12:36:51.704 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014271600000000001 2024-09-21T12:36:51.704 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:51.704 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:51.704 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:51.704 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.704 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.704 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.26147.0:95136 11.b 11:d5fdf68d:::benchmark_data_smithi114_187622_object95135:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-21T12:36:51.705 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:36:51.682797+0000", 2024-09-21T12:36:51.705 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0031018980000000001, 2024-09-21T12:36:51.705 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0033397069999999999, 2024-09-21T12:36:51.705 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:36:51.705 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:36:51.705 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:36:51.705 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:36:51.705 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.26147", 2024-09-21T12:36:51.705 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:36:51.705 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 95136 2024-09-21T12:36:51.705 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.705 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:36:51.705 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.705 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:36:51.706 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.682797+0000", 2024-09-21T12:36:51.706 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:51.706 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.706 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.706 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:36:51.706 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.682797+0000", 2024-09-21T12:36:51.706 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:51.706 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.706 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.706 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:36:51.706 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.682799+0000", 2024-09-21T12:36:51.706 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2500000000000001e-06 2024-09-21T12:36:51.706 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.707 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.707 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:36:51.707 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.682822+0000", 2024-09-21T12:36:51.707 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3595e-05 2024-09-21T12:36:51.707 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.707 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.707 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:36:51.707 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.682834+0000", 2024-09-21T12:36:51.707 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1413999999999999e-05 2024-09-21T12:36:51.707 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.708 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.708 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:36:51.708 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.682841+0000", 2024-09-21T12:36:51.708 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3969999999999997e-06 2024-09-21T12:36:51.708 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.708 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.708 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:36:51.708 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.682898+0000", 2024-09-21T12:36:51.708 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6798000000000001e-05 2024-09-21T12:36:51.708 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.709 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.709 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:36:51.709 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.683051+0000", 2024-09-21T12:36:51.709 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015285299999999999 2024-09-21T12:36:51.709 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.709 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.709 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,1", 2024-09-21T12:36:51.709 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.683481+0000", 2024-09-21T12:36:51.709 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00043000800000000001 2024-09-21T12:36:51.709 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:51.709 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:51.709 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:51.709 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.709 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.710 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.26147.0:95137 11.b 11.818d78b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-21T12:36:51.710 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:36:51.684469+0000", 2024-09-21T12:36:51.710 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.001430789, 2024-09-21T12:36:51.710 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00172182, 2024-09-21T12:36:51.710 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:36:51.710 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:36:51.710 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "queued for pg", 2024-09-21T12:36:51.710 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:36:51.710 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.26147", 2024-09-21T12:36:51.710 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:36:51.710 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 95137 2024-09-21T12:36:51.710 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.710 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:36:51.711 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.711 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:36:51.711 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.684469+0000", 2024-09-21T12:36:51.711 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:51.711 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.711 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.711 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:36:51.711 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.684469+0000", 2024-09-21T12:36:51.711 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:51.711 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.711 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.711 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:36:51.711 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.684470+0000", 2024-09-21T12:36:51.712 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.269e-06 2024-09-21T12:36:51.712 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.712 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.712 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:36:51.712 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.684493+0000", 2024-09-21T12:36:51.712 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2963999999999999e-05 2024-09-21T12:36:51.712 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.712 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.712 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:36:51.712 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.684509+0000", 2024-09-21T12:36:51.712 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6622000000000001e-05 2024-09-21T12:36:51.712 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.712 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.712 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:36:51.712 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.684517+0000", 2024-09-21T12:36:51.713 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3089999999999998e-06 2024-09-21T12:36:51.713 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:51.713 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:51.713 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:51.713 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.713 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.713 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.26147.0:95138 11.b 11.d775f84b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-21T12:36:51.713 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:36:51.684528+0000", 2024-09-21T12:36:51.713 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0013709499999999999, 2024-09-21T12:36:51.713 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001702847, 2024-09-21T12:36:51.713 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:36:51.713 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:36:51.713 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "queued for pg", 2024-09-21T12:36:51.713 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:36:51.713 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.26147", 2024-09-21T12:36:51.714 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:36:51.714 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 95138 2024-09-21T12:36:51.714 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.714 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:36:51.714 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.714 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:36:51.714 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.684528+0000", 2024-09-21T12:36:51.714 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:51.714 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.714 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.714 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:36:51.714 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.684528+0000", 2024-09-21T12:36:51.714 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:51.714 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.714 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.715 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:36:51.715 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.684529+0000", 2024-09-21T12:36:51.715 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.89e-07 2024-09-21T12:36:51.715 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.715 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.715 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:36:51.715 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.684548+0000", 2024-09-21T12:36:51.715 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8794e-05 2024-09-21T12:36:51.715 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.715 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.715 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:36:51.715 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.684554+0000", 2024-09-21T12:36:51.715 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5810000000000002e-06 2024-09-21T12:36:51.715 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.715 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.716 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:36:51.716 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.684559+0000", 2024-09-21T12:36:51.716 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1749999999999996e-06 2024-09-21T12:36:51.716 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:51.716 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:51.716 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:51.716 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.716 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.716 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.26147.0:95139 11.0 e434/420 11:09db012d:::benchmark_data_smithi114_187622_object95138:head v 434'5911, mlcod=434'5909)", 2024-09-21T12:36:51.716 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:36:51.685002+0000", 2024-09-21T12:36:51.716 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.000897371, 2024-09-21T12:36:51.716 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001271086, 2024-09-21T12:36:51.716 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:36:51.716 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:36:51.716 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:36:51.716 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:36:51.717 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.717 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:36:51.717 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.685002+0000", 2024-09-21T12:36:51.717 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:51.717 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.717 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.717 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:36:51.717 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.685002+0000", 2024-09-21T12:36:51.717 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:51.717 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.717 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.717 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:36:51.717 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.685002+0000", 2024-09-21T12:36:51.718 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5000000000000003e-07 2024-09-21T12:36:51.718 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.718 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.718 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:36:51.718 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.685030+0000", 2024-09-21T12:36:51.718 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.7061999999999999e-05 2024-09-21T12:36:51.718 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.718 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.718 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:36:51.718 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.685041+0000", 2024-09-21T12:36:51.718 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1146e-05 2024-09-21T12:36:51.718 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.718 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.719 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:36:51.719 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.685047+0000", 2024-09-21T12:36:51.719 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0719999999999996e-06 2024-09-21T12:36:51.719 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.719 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.719 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:36:51.719 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.685895+0000", 2024-09-21T12:36:51.719 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000848054 2024-09-21T12:36:51.719 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.719 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.719 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:36:51.719 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.685912+0000", 2024-09-21T12:36:51.719 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7611000000000001e-05 2024-09-21T12:36:51.720 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:51.720 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:51.720 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:51.720 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.720 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.720 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.26147.0:95118 11.2 11:4bea7752:::benchmark_data_smithi114_187622_object95117:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-21T12:36:51.720 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:36:51.669346+0000", 2024-09-21T12:36:51.720 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.016553258000000001, 2024-09-21T12:36:51.720 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.016977921999999999, 2024-09-21T12:36:51.720 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:36:51.720 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:36:51.720 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:36:51.720 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:36:51.720 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.26147", 2024-09-21T12:36:51.720 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:36:51.720 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 95118 2024-09-21T12:36:51.721 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.721 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:36:51.721 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.721 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:36:51.721 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.669346+0000", 2024-09-21T12:36:51.721 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:51.721 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.721 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.721 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:36:51.721 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.669346+0000", 2024-09-21T12:36:51.721 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:36:51.721 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.721 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.721 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:36:51.721 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.669348+0000", 2024-09-21T12:36:51.722 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0059999999999999e-06 2024-09-21T12:36:51.722 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.722 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.722 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:36:51.722 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.669376+0000", 2024-09-21T12:36:51.722 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.7744000000000001e-05 2024-09-21T12:36:51.722 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.722 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.722 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:36:51.722 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.669396+0000", 2024-09-21T12:36:51.722 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9843999999999998e-05 2024-09-21T12:36:51.722 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.722 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.722 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:36:51.723 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.669405+0000", 2024-09-21T12:36:51.723 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.8409999999999995e-06 2024-09-21T12:36:51.723 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.723 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.723 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:36:51.723 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.669505+0000", 2024-09-21T12:36:51.723 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.9547000000000001e-05 2024-09-21T12:36:51.723 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.723 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.723 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:36:51.723 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.669655+0000", 2024-09-21T12:36:51.723 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015001700000000001 2024-09-21T12:36:51.723 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.723 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.723 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0", 2024-09-21T12:36:51.723 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.669804+0000", 2024-09-21T12:36:51.724 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000148803 2024-09-21T12:36:51.724 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:36:51.724 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:36:51.724 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:36:51.724 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:51.681919+0000", 2024-09-21T12:36:51.724 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.012115101 2024-09-21T12:36:51.724 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:51.724 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:36:51.724 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:51.724 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:36:51.724 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:36:51.724 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 8 2024-09-21T12:36:51.724 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:36:51.738 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:51.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:51 smithi183 ceph-mon[33277]: pgmap v846: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 75 MiB/s wr, 1.19k op/s; 22177/266283 objects degraded (8.328%); 11162/266283 objects misplaced (4.192%) 2024-09-21T12:36:51.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:51 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:36:51.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:51 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:36:51.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:51 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/4083880627' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:36:51.769 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:51.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:51.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:51.838 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T12:36:51.872 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:51.900 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:51.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:51.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:52.002 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:52.033 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:52.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:52.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:52.135 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:52.159 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:52.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:52.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:52.261 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:52.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:52.300 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:52.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:52.401 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:52.439 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:52.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:52.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:52.508 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 59 16 96399 96383 102.087 92.75 0.0195323 0.00979147 2024-09-21T12:36:52.542 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:52.584 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:52.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:52.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:52.657 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:36:52 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:36:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:36:52.657 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:52 smithi114 ceph-mon[40867]: osdmap e434: 8 total, 7 up, 8 in 2024-09-21T12:36:52.657 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:52 smithi114 ceph-mon[40867]: pgmap v848: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 89 MiB/s wr, 1.43k op/s; 22689/270564 objects degraded (8.386%); 11162/270564 objects misplaced (4.125%) 2024-09-21T12:36:52.657 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:52 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 22177/266283 objects degraded (8.328%), 13 pgs degraded, 33 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:52.658 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:52 smithi114 ceph-mon[30324]: osdmap e434: 8 total, 7 up, 8 in 2024-09-21T12:36:52.658 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:52 smithi114 ceph-mon[30324]: pgmap v848: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 89 MiB/s wr, 1.43k op/s; 22689/270564 objects degraded (8.386%); 11162/270564 objects misplaced (4.125%) 2024-09-21T12:36:52.659 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:52 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 22177/266283 objects degraded (8.328%), 13 pgs degraded, 33 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:52.686 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:52.711 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:52.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:52.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:52.728 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T12:36:52.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:52 smithi183 ceph-mon[33277]: osdmap e434: 8 total, 7 up, 8 in 2024-09-21T12:36:52.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:52 smithi183 ceph-mon[33277]: pgmap v848: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 89 MiB/s wr, 1.43k op/s; 22689/270564 objects degraded (8.386%); 11162/270564 objects misplaced (4.125%) 2024-09-21T12:36:52.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:52 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 22177/266283 objects degraded (8.328%), 13 pgs degraded, 33 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:52.813 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:52.843 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:52.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:52.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:52.945 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:52.977 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:52.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:52.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:53.079 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:53.113 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:53.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:53.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:53.184 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T12:36:53.184 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T12:36:53.214 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:53.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:53.248 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:53.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:53.251 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:36:53.251 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:36:53.265 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:36:53.349 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:53.387 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:53.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:53.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:53.488 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:53.508 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-21T12:36:53.508163+0000 min lat: 0.00111509 max lat: 2.08308 avg lat: 0.00980241 2024-09-21T12:36:53.508 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T12:36:53.508 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 60 16 97921 97905 101.97 95.125 0.00447077 0.00980241 2024-09-21T12:36:53.522 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:53.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:53.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:53.625 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:53.661 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:53.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:53.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:53.763 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:53.794 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:53.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:53.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:53.896 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:53.927 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:53.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:53.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:54.028 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:54.070 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:54.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:54.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:54.172 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:54.225 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:54.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:54.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:54.305 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:36:54.305 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:36:54.305 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:36:54.306 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:36:54.306 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:36:54.327 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:54.355 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:54.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:54.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:54.457 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:54.486 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:54.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:54.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:54.508 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 61 16 99376 99360 101.789 90.9375 0.00711556 0.00981987 2024-09-21T12:36:54.588 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:54.626 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:54.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:54.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:54.728 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:54.751 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:54.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:54.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:54.852 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:54.886 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:54.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:54.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:54.988 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:55.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:55.027 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:55.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:55.128 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:55.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:55.162 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:55.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:55.185 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:36:55.234 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:54 smithi114 ceph-mon[40867]: pgmap v849: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 757 B/s rd, 98 MiB/s wr, 1.57k op/s; 24055/288813 objects degraded (8.329%); 12101/288813 objects misplaced (4.190%) 2024-09-21T12:36:55.234 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:54 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3768491358' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"} : dispatch 2024-09-21T12:36:55.234 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:54 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"} : dispatch 2024-09-21T12:36:55.234 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:54 smithi114 ceph-mon[30324]: pgmap v849: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 757 B/s rd, 98 MiB/s wr, 1.57k op/s; 24055/288813 objects degraded (8.329%); 12101/288813 objects misplaced (4.190%) 2024-09-21T12:36:55.234 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:54 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3768491358' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"} : dispatch 2024-09-21T12:36:55.234 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:54 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"} : dispatch 2024-09-21T12:36:55.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:54 smithi183 ceph-mon[33277]: pgmap v849: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 757 B/s rd, 98 MiB/s wr, 1.57k op/s; 24055/288813 objects degraded (8.329%); 12101/288813 objects misplaced (4.190%) 2024-09-21T12:36:55.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:54 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3768491358' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"} : dispatch 2024-09-21T12:36:55.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:54 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"} : dispatch 2024-09-21T12:36:55.263 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:55.291 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:55.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:55.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:55.393 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:55.435 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:55.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:55.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:55.510 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 62 16 100866 100850 101.647 93.125 0.0071082 0.00983372 2024-09-21T12:36:55.537 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:55.571 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:55.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:55.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:55.672 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:55.706 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:55.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:55.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:55.808 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:36:55.842 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:55.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:55.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:36:55.944 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:55.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:55.979 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:55.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:56.080 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:56.114 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:56.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:56.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:56.216 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:56.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:56.251 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:56.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:56.352 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:56.378 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:56.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:56.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:56.480 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:56.510 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 63 16 102424 102408 101.58 97.375 0.013514 0.00984045 2024-09-21T12:36:56.514 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:56.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:56.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:56.586 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: specified pg_num 51 < pg_num_max 32 2024-09-21T12:36:56.616 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:56.655 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:56.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:56.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:56.758 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:56.796 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:56.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:56.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:56.898 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:56.925 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:56.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:56.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:57.027 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:57.058 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:57.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:57.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:57.160 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:57.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:57.193 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:57.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:57.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:56 smithi183 ceph-mon[33277]: pgmap v850: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 62 MiB/s wr, 994 op/s; 24055/288813 objects degraded (8.329%); 12101/288813 objects misplaced (4.190%) 2024-09-21T12:36:57.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:56 smithi114 ceph-mon[30324]: pgmap v850: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 62 MiB/s wr, 994 op/s; 24055/288813 objects degraded (8.329%); 12101/288813 objects misplaced (4.190%) 2024-09-21T12:36:57.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:56 smithi114 ceph-mon[40867]: pgmap v850: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 62 MiB/s wr, 994 op/s; 24055/288813 objects degraded (8.329%); 12101/288813 objects misplaced (4.190%) 2024-09-21T12:36:57.294 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:57.328 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:57.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:57.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:57.373 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T12:36:57.430 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:57.458 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:57.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:57.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:57.510 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 64 16 104070 104054 101.599 102.875 0.0173864 0.00983827 2024-09-21T12:36:57.560 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:57.610 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:57.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:57.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:57.712 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:57.755 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:57.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:57.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:57.857 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:57.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:57.896 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:57.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:57.944 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:36:57 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:36:57.997 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:58.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:57 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 24055/288813 objects degraded (8.329%), 13 pgs degraded, 33 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:58.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:57 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 24055/288813 objects degraded (8.329%), 13 pgs degraded, 33 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:58.033 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:58.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:58.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:58.113 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T12:36:58.135 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:58.169 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:36:58.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:58.198 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:58.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:58.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:57 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 24055/288813 objects degraded (8.329%), 13 pgs degraded, 33 pgs undersized (PG_DEGRADED) 2024-09-21T12:36:58.298 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:58.345 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:58.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:58.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:58.448 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:58.486 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:58.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:58.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:58.510 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 65 16 105521 105505 101.431 90.6875 0.0163029 0.00985426 2024-09-21T12:36:58.588 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:58.621 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:58.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:58.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:58.722 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:36:58.752 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:58.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:58.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:36:58.853 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:58.867 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:36:58.868 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:36:58.874 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:36:58.900 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:58.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:58.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:58.904 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:36:58.904 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:36:58.906 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T12:36:58.937 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:36:58.938 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:36:58.975 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:36:58.976 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:36:58.976 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-21T12:36:59.001 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:59.008 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:36:59.008 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:36:59.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:59.037 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:59.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:59.046 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:36:59.046 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:36:59.077 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:36:59.077 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:36:59.138 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:59.168 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:59.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:59.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:59.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:58 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:36:59.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:58 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:36:59.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:58 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:36:59.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:58 smithi183 ceph-mon[33277]: pgmap v851: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.3 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 97 MiB/s wr, 1.55k op/s; 25541/310152 objects degraded (8.235%); 13018/310152 objects misplaced (4.197%) 2024-09-21T12:36:59.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:58 smithi183 ceph-mon[33277]: osdmap e435: 8 total, 7 up, 8 in 2024-09-21T12:36:59.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:58 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3375646044' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:36:59.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:58 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:36:59.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:58 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2620587900' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:36:59.270 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:36:59.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:58 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:36:59.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:58 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:36:59.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:58 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:36:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:58 smithi114 ceph-mon[40867]: pgmap v851: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.3 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 97 MiB/s wr, 1.55k op/s; 25541/310152 objects degraded (8.235%); 13018/310152 objects misplaced (4.197%) 2024-09-21T12:36:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:58 smithi114 ceph-mon[40867]: osdmap e435: 8 total, 7 up, 8 in 2024-09-21T12:36:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:58 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3375646044' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:36:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:58 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:36:59.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:58 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2620587900' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:36:59.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:58 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:36:59.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:58 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:36:59.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:58 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:36:59.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:58 smithi114 ceph-mon[30324]: pgmap v851: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.3 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 97 MiB/s wr, 1.55k op/s; 25541/310152 objects degraded (8.235%); 13018/310152 objects misplaced (4.197%) 2024-09-21T12:36:59.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:58 smithi114 ceph-mon[30324]: osdmap e435: 8 total, 7 up, 8 in 2024-09-21T12:36:59.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:58 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3375646044' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:36:59.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:58 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:36:59.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:58 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2620587900' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:36:59.308 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:59.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:59.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:36:59.410 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:36:59.445 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:36:59.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:59.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:36:59.510 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 66 16 107017 107001 101.31 93.5 0.0134649 0.00986644 2024-09-21T12:36:59.547 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:36:59.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:59.580 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:59.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:36:59.680 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:59.712 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:59.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:59.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:36:59.814 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:36:59.849 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:59.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:59.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:36:59.951 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:36:59.986 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:36:59.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:36:59.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:00.088 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:00.122 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:00.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:00.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:00.124 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T12:37:00.125 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T12:37:00.223 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:00.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:59 smithi183 ceph-mon[33277]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:37:00.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:59 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:37:00.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:36:59 smithi183 ceph-mon[33277]: osdmap e436: 8 total, 7 up, 8 in 2024-09-21T12:37:00.254 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:00.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:00.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:00.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:59 smithi114 ceph-mon[30324]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:37:00.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:59 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:37:00.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:36:59 smithi114 ceph-mon[30324]: osdmap e436: 8 total, 7 up, 8 in 2024-09-21T12:37:00.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:59 smithi114 ceph-mon[40867]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:37:00.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:59 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:37:00.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:36:59 smithi114 ceph-mon[40867]: osdmap e436: 8 total, 7 up, 8 in 2024-09-21T12:37:00.356 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:00.393 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:00.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:00.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:00.494 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:00.511 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 67 16 108679 108663 101.348 103.875 0.0170042 0.00986248 2024-09-21T12:37:00.537 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:00.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:00.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:00.639 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:00.670 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:00.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:00.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:00.772 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:00.810 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:00.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:00.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:00.912 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:00.943 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:00.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:00.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:01.044 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:01.076 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:01.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:01.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:01.118 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:00 smithi114 ceph-mon[30324]: pgmap v854: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.71k op/s; 26021/311592 objects degraded (8.351%); 13018/311592 objects misplaced (4.178%) 2024-09-21T12:37:01.118 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:00 smithi114 ceph-mon[40867]: pgmap v854: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.71k op/s; 26021/311592 objects degraded (8.351%); 13018/311592 objects misplaced (4.178%) 2024-09-21T12:37:01.177 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:01.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:01.219 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:01.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:01.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:00 smithi183 ceph-mon[33277]: pgmap v854: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.71k op/s; 26021/311592 objects degraded (8.351%); 13018/311592 objects misplaced (4.178%) 2024-09-21T12:37:01.286 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:37:01.286 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:37:01.320 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:01.354 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:01.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:01.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:01.456 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:01.490 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:01.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:01.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:01.511 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 68 16 110295 110279 101.343 101 0.00757958 0.00986325 2024-09-21T12:37:01.592 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:01.628 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:01.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:01.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:01.730 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:01.761 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:01.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:01.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:01.863 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:01.901 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:01.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:01.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:02.002 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:02.034 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:02.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:02.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:02.125 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:37:02.135 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:02.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:02.190 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:02.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:02.291 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:02.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:02.333 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:02.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:02.374 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 4, 7] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-09-21T12:37:02.374 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:37:02.374 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_4 2024-09-21T12:37:02.375 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:37:02.434 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:02.460 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:02.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:02.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:02.512 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 69 16 111722 111706 101.165 89.1875 0.00504063 0.00988073 2024-09-21T12:37:02.561 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:02.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:02.613 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:02.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:02.639 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T12:37:02.710 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:37:02.715 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:02.741 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:02.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:02.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:02.831 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T12:37:02.842 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:02.851 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:37:02 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:37:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:37:02.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:02.885 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:02.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:02.986 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:03.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:03.028 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:03.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:03.129 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:03.133 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:37:03.150 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:03.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:03.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:03.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:02 smithi183 ceph-mon[33277]: pgmap v855: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.14k op/s; 26506/316077 objects degraded (8.386%); 13018/316077 objects misplaced (4.119%) 2024-09-21T12:37:03.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:02 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 26506/316077 objects degraded (8.386%), 13 pgs degraded, 33 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:03.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:02 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:37:03.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:02 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:37:03.251 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:03.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:02 smithi114 ceph-mon[40867]: pgmap v855: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.14k op/s; 26506/316077 objects degraded (8.386%); 13018/316077 objects misplaced (4.119%) 2024-09-21T12:37:03.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:02 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 26506/316077 objects degraded (8.386%), 13 pgs degraded, 33 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:03.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:02 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:37:03.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:02 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:37:03.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:02 smithi114 ceph-mon[30324]: pgmap v855: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.14k op/s; 26506/316077 objects degraded (8.386%); 13018/316077 objects misplaced (4.119%) 2024-09-21T12:37:03.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:02 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 26506/316077 objects degraded (8.386%), 13 pgs degraded, 33 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:03.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:02 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:37:03.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:02 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:37:03.293 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:03.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:03.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:03.395 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:03.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:03.427 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:03.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:03.512 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 70 16 113044 113028 100.9 82.625 0.0473547 0.00990715 2024-09-21T12:37:03.528 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:03.556 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:03.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:03.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:03.657 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:03.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:03.691 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:03.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:03.792 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:03.822 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:03.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:03.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:03.924 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:03.958 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:03.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:03.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:04.059 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:04.101 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:04.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:04.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:04.169 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:37:04.169 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:37:04.203 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:04.240 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:04.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:04.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:04.342 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:04.376 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:04.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:04.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:04.478 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:04.512 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:04.513 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 71 16 114457 114441 100.723 88.3125 0.0130633 0.00992405 2024-09-21T12:37:04.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:04.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:04.614 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:04.644 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:04.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:04.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:04.746 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:04.783 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:04.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:04.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:04.880 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:37:04.886 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:04.914 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:04.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:04.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:05.016 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:05.044 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:05.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:05.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:05.055 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:37:05.056 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:37:05.092 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:37:05.092 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:37:05.093 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T12:37:05.125 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:37:05.126 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:37:05.146 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:05.159 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:37:05.159 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:37:05.185 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:05.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:05.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:05.188 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T12:37:05.194 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:37:05.194 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:37:05.225 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:04 smithi114 ceph-mon[30324]: pgmap v856: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 120 MiB/s wr, 1.93k op/s; 27887/335028 objects degraded (8.324%); 14000/335028 objects misplaced (4.179%) 2024-09-21T12:37:05.225 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:04 smithi114 ceph-mon[40867]: pgmap v856: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 120 MiB/s wr, 1.93k op/s; 27887/335028 objects degraded (8.324%); 14000/335028 objects misplaced (4.179%) 2024-09-21T12:37:05.225 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:37:05.225 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:37:05.257 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:37:05.257 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:37:05.258 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:04 smithi183 ceph-mon[33277]: pgmap v856: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 120 MiB/s wr, 1.93k op/s; 27887/335028 objects degraded (8.324%); 14000/335028 objects misplaced (4.179%) 2024-09-21T12:37:05.287 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:05.329 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:05.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:05.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:05.431 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:05.462 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:05.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:05.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:05.512 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 72 16 115796 115780 100.486 83.6875 0.00766791 0.00994737 2024-09-21T12:37:05.564 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:05.612 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:05.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:05.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:05.642 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:37:05.714 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:05.759 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:05.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:05.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:05.860 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:05.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:05.898 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:05.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:05.999 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:06.033 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:06.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:06.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:06.134 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:06.172 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:06.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:06.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:06.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:06 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3002165506' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:37:06.274 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:06.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:06 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3002165506' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:37:06.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:06 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3002165506' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:37:06.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:06.310 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:06.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:06.411 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:06.451 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:06.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:06.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:06.512 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 73 16 117290 117274 100.389 93.375 0.0119155 0.00995678 2024-09-21T12:37:06.554 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:06.583 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:06.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:06.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:06.679 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:37:06.679 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-21T12:37:06.679 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-21T12:37:06.679 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-21T12:37:06.679 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:37:06.684 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:06.708 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:06.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:06.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:06.810 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:06.847 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:06.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:06.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:06.949 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:06.982 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:06.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:06.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:07.062 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:37:07.065 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:37:07.084 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:07.117 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:07.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:07.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:07.218 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:07.252 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:07.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:07.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:07.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:07 smithi114 ceph-mon[30324]: pgmap v857: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 65 MiB/s wr, 1.04k op/s; 27887/335028 objects degraded (8.324%); 14000/335028 objects misplaced (4.179%) 2024-09-21T12:37:07.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:07 smithi114 ceph-mon[40867]: pgmap v857: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 65 MiB/s wr, 1.04k op/s; 27887/335028 objects degraded (8.324%); 14000/335028 objects misplaced (4.179%) 2024-09-21T12:37:07.354 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:07.358 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:07 smithi183 ceph-mon[33277]: pgmap v857: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 65 MiB/s wr, 1.04k op/s; 27887/335028 objects degraded (8.324%); 14000/335028 objects misplaced (4.179%) 2024-09-21T12:37:07.385 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:07.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:07.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:07.416 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T12:37:07.487 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:07.513 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 74 16 118729 118713 100.247 89.9375 0.0081358 0.00997103 2024-09-21T12:37:07.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:07.519 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:07.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:07.620 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:07.661 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:07.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:07.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:07.762 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:07.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:07.803 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:07.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:07.842 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:37:07.856 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:37:07.892 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":857,"stamp":"2024-09-21T12:37:05.882252+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7306380013,"num_objects":111676,"num_object_clones":0,"num_object_copies":335028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27887,"num_objects_misplaced":14000,"num_objects_unfound":0,"num_objects_dirty":111676,"num_whiteouts":0,"num_read":1192,"num_read_kb":948,"num_write":112204,"num_write_kb":7134647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4699,"num_bytes_recovered":275092355,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68112,"ondisk_log_size":68112,"up":481,"acting":501,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":554,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":27405036,"kb_used_data":21764768,"kb_used_omap":12,"kb_used_meta":5640115,"kb_avail":722556180,"statfs":{"total":767960285184,"available":739897528320,"internally_reserved":0,"allocated":22287122432,"data_stored":22282978351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":5775478320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":543424512,"num_objects":8292,"num_object_clones":0,"num_object_copies":24876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2346,"num_objects_misplaced":982,"num_objects_unfound":0,"num_objects_dirty":8292,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8292,"num_write_kb":530688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005566"},"pg_stats":[{"pgid":"1.1d","version":"76'79","reported_seq":495,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.987284+0000","last_change":"2024-09-21T12:32:38.427111+0000","last_active":"2024-09-21T12:36:58.987284+0000","last_peered":"2024-09-21T12:36:58.987284+0000","last_clean":"2024-09-21T12:36:58.987284+0000","last_became_active":"2024-09-21T12:32:38.426383+0000","last_became_peered":"2024-09-21T12:32:38.426383+0000","last_unstale":"2024-09-21T12:36:58.987284+0000","last_undegraded":"2024-09-21T12:36:58.987284+0000","last_fullsized":"2024-09-21T12:36:58.987284+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:32.060116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:32.060116+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:14.142065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":810,"reported_epoch":436,"state":"active+undersized","last_fresh":"2024-09-21T12:36:58.978448+0000","last_change":"2024-09-21T12:34:34.340741+0000","last_active":"2024-09-21T12:36:58.978448+0000","last_peered":"2024-09-21T12:36:58.978448+0000","last_clean":"2024-09-21T12:34:29.289357+0000","last_became_active":"2024-09-21T12:34:34.340741+0000","last_became_peered":"2024-09-21T12:34:34.340741+0000","last_unstale":"2024-09-21T12:36:58.978448+0000","last_undegraded":"2024-09-21T12:36:58.978448+0000","last_fullsized":"2024-09-21T12:34:34.332209+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:29.086830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:29.086830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:31.924550+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":195,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.968637+0000","last_change":"2024-09-21T12:32:38.443995+0000","last_active":"2024-09-21T12:36:58.968637+0000","last_peered":"2024-09-21T12:36:58.968637+0000","last_clean":"2024-09-21T12:36:58.968637+0000","last_became_active":"2024-09-21T12:32:38.419081+0000","last_became_peered":"2024-09-21T12:32:38.419081+0000","last_unstale":"2024-09-21T12:36:58.968637+0000","last_undegraded":"2024-09-21T12:36:58.968637+0000","last_fullsized":"2024-09-21T12:36:58.968637+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-21T12:29:08.671723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:29:08.671723+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:18.917504+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":807,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.986171+0000","last_change":"2024-09-21T12:36:00.136989+0000","last_active":"2024-09-21T12:36:58.986171+0000","last_peered":"2024-09-21T12:36:58.986171+0000","last_clean":"2024-09-21T12:36:58.986171+0000","last_became_active":"2024-09-21T12:36:00.136673+0000","last_became_peered":"2024-09-21T12:36:00.136673+0000","last_unstale":"2024-09-21T12:36:58.986171+0000","last_undegraded":"2024-09-21T12:36:58.986171+0000","last_fullsized":"2024-09-21T12:36:58.986171+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:54.969048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:35:54.969048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:20.733223+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":201,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.969276+0000","last_change":"2024-09-21T12:36:44.729701+0000","last_active":"2024-09-21T12:36:58.969276+0000","last_peered":"2024-09-21T12:36:58.969276+0000","last_clean":"2024-09-21T12:36:58.969276+0000","last_became_active":"2024-09-21T12:32:38.425588+0000","last_became_peered":"2024-09-21T12:32:38.425588+0000","last_unstale":"2024-09-21T12:36:58.969276+0000","last_undegraded":"2024-09-21T12:36:58.969276+0000","last_fullsized":"2024-09-21T12:36:58.969276+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:44.729655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:44.729655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:58.553964+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"76'79","reported_seq":54,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.982171+0000","last_change":"2024-09-21T12:36:00.154588+0000","last_active":"2024-09-21T12:36:58.982171+0000","last_peered":"2024-09-21T12:36:58.982171+0000","last_clean":"2024-09-21T12:36:58.982171+0000","last_became_active":"2024-09-21T12:36:00.154209+0000","last_became_peered":"2024-09-21T12:36:00.154209+0000","last_unstale":"2024-09-21T12:36:58.982171+0000","last_undegraded":"2024-09-21T12:36:58.982171+0000","last_fullsized":"2024-09-21T12:36:58.982171+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:35:54.010909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:35:54.010909+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:12.176158+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":795,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.981062+0000","last_change":"2024-09-21T12:36:00.139483+0000","last_active":"2024-09-21T12:36:58.981062+0000","last_peered":"2024-09-21T12:36:58.981062+0000","last_clean":"2024-09-21T12:36:58.981062+0000","last_became_active":"2024-09-21T12:36:00.139167+0000","last_became_peered":"2024-09-21T12:36:00.139167+0000","last_unstale":"2024-09-21T12:36:58.981062+0000","last_undegraded":"2024-09-21T12:36:58.981062+0000","last_fullsized":"2024-09-21T12:36:58.981062+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:05.904553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:34:05.904553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":807,"reported_epoch":436,"state":"active+undersized","last_fresh":"2024-09-21T12:36:58.981260+0000","last_change":"2024-09-21T12:34:34.346389+0000","last_active":"2024-09-21T12:36:58.981260+0000","last_peered":"2024-09-21T12:36:58.981260+0000","last_clean":"2024-09-21T12:34:29.294384+0000","last_became_active":"2024-09-21T12:34:34.346389+0000","last_became_peered":"2024-09-21T12:34:34.346389+0000","last_unstale":"2024-09-21T12:36:58.981260+0000","last_undegraded":"2024-09-21T12:36:58.981260+0000","last_fullsized":"2024-09-21T12:34:34.336359+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:06.698999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:06.698999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:34.545818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":844,"reported_epoch":436,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:58.977283+0000","last_change":"2024-09-21T12:34:34.337065+0000","last_active":"2024-09-21T12:36:58.977283+0000","last_peered":"2024-09-21T12:36:58.977283+0000","last_clean":"2024-09-21T12:34:29.291199+0000","last_became_active":"2024-09-21T12:34:34.337065+0000","last_became_peered":"2024-09-21T12:34:34.337065+0000","last_unstale":"2024-09-21T12:36:58.977283+0000","last_undegraded":"2024-09-21T12:34:34.335679+0000","last_fullsized":"2024-09-21T12:34:34.335428+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:31:48.294300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:48.294300+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:39.494442+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":193,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.968629+0000","last_change":"2024-09-21T12:36:35.622295+0000","last_active":"2024-09-21T12:36:58.968629+0000","last_peered":"2024-09-21T12:36:58.968629+0000","last_clean":"2024-09-21T12:36:58.968629+0000","last_became_active":"2024-09-21T12:32:38.417724+0000","last_became_peered":"2024-09-21T12:32:38.417724+0000","last_unstale":"2024-09-21T12:36:58.968629+0000","last_undegraded":"2024-09-21T12:36:58.968629+0000","last_fullsized":"2024-09-21T12:36:58.968629+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:35.622232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:35.622232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:53.247230+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":193,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.972652+0000","last_change":"2024-09-21T12:33:32.373658+0000","last_active":"2024-09-21T12:36:58.972652+0000","last_peered":"2024-09-21T12:36:58.972652+0000","last_clean":"2024-09-21T12:36:58.972652+0000","last_became_active":"2024-09-21T12:32:38.428077+0000","last_became_peered":"2024-09-21T12:32:38.428077+0000","last_unstale":"2024-09-21T12:36:58.972652+0000","last_undegraded":"2024-09-21T12:36:58.972652+0000","last_fullsized":"2024-09-21T12:36:58.972652+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:32.372832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:32.372832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:55.578090+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":211,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.969388+0000","last_change":"2024-09-21T12:36:42.653161+0000","last_active":"2024-09-21T12:36:58.969388+0000","last_peered":"2024-09-21T12:36:58.969388+0000","last_clean":"2024-09-21T12:36:58.969388+0000","last_became_active":"2024-09-21T12:32:38.426121+0000","last_became_peered":"2024-09-21T12:32:38.426121+0000","last_unstale":"2024-09-21T12:36:58.969388+0000","last_undegraded":"2024-09-21T12:36:58.969388+0000","last_fullsized":"2024-09-21T12:36:58.969388+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:36:42.653075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:42.653075+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:44.149637+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":846,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.985101+0000","last_change":"2024-09-21T12:36:00.138341+0000","last_active":"2024-09-21T12:36:58.985101+0000","last_peered":"2024-09-21T12:36:58.985101+0000","last_clean":"2024-09-21T12:36:58.985101+0000","last_became_active":"2024-09-21T12:36:00.137364+0000","last_became_peered":"2024-09-21T12:36:00.137364+0000","last_unstale":"2024-09-21T12:36:58.985101+0000","last_undegraded":"2024-09-21T12:36:58.985101+0000","last_fullsized":"2024-09-21T12:36:58.985101+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:13.198376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:13.198376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:24.428313+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":813,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.975072+0000","last_change":"2024-09-21T12:36:38.752470+0000","last_active":"2024-09-21T12:36:58.975072+0000","last_peered":"2024-09-21T12:36:58.975072+0000","last_clean":"2024-09-21T12:36:58.975072+0000","last_became_active":"2024-09-21T12:36:00.134723+0000","last_became_peered":"2024-09-21T12:36:00.134723+0000","last_unstale":"2024-09-21T12:36:58.975072+0000","last_undegraded":"2024-09-21T12:36:58.975072+0000","last_fullsized":"2024-09-21T12:36:58.975072+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:38.752410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:38.752410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:46.920310+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"76'79","reported_seq":580,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.981034+0000","last_change":"2024-09-21T12:36:00.148161+0000","last_active":"2024-09-21T12:36:58.981034+0000","last_peered":"2024-09-21T12:36:58.981034+0000","last_clean":"2024-09-21T12:36:58.981034+0000","last_became_active":"2024-09-21T12:36:00.147795+0000","last_became_peered":"2024-09-21T12:36:00.147795+0000","last_unstale":"2024-09-21T12:36:58.981034+0000","last_undegraded":"2024-09-21T12:36:58.981034+0000","last_fullsized":"2024-09-21T12:36:58.981034+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:36.752562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:36.752562+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:52.540202+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":193,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.972290+0000","last_change":"2024-09-21T12:35:14.244213+0000","last_active":"2024-09-21T12:36:58.972290+0000","last_peered":"2024-09-21T12:36:58.972290+0000","last_clean":"2024-09-21T12:36:58.972290+0000","last_became_active":"2024-09-21T12:32:38.426142+0000","last_became_peered":"2024-09-21T12:32:38.426142+0000","last_unstale":"2024-09-21T12:36:58.972290+0000","last_undegraded":"2024-09-21T12:36:58.972290+0000","last_fullsized":"2024-09-21T12:36:58.972290+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:14.243954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:14.243954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":199,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.969107+0000","last_change":"2024-09-21T12:33:26.395059+0000","last_active":"2024-09-21T12:36:58.969107+0000","last_peered":"2024-09-21T12:36:58.969107+0000","last_clean":"2024-09-21T12:36:58.969107+0000","last_became_active":"2024-09-21T12:32:38.422912+0000","last_became_peered":"2024-09-21T12:32:38.422912+0000","last_unstale":"2024-09-21T12:36:58.969107+0000","last_undegraded":"2024-09-21T12:36:58.969107+0000","last_fullsized":"2024-09-21T12:36:58.969107+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'7","last_scrub_stamp":"2024-09-21T12:33:26.394982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:26.394982+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:54.602228+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":54,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.981090+0000","last_change":"2024-09-21T12:36:00.143221+0000","last_active":"2024-09-21T12:36:58.981090+0000","last_peered":"2024-09-21T12:36:58.981090+0000","last_clean":"2024-09-21T12:36:58.981090+0000","last_became_active":"2024-09-21T12:36:00.142106+0000","last_became_peered":"2024-09-21T12:36:00.142106+0000","last_unstale":"2024-09-21T12:36:58.981090+0000","last_undegraded":"2024-09-21T12:36:58.981090+0000","last_fullsized":"2024-09-21T12:36:58.981090+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:09.558418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:35:09.558418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:51.721658+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":806,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.984792+0000","last_change":"2024-09-21T12:33:36.125346+0000","last_active":"2024-09-21T12:36:58.984792+0000","last_peered":"2024-09-21T12:36:58.984792+0000","last_clean":"2024-09-21T12:36:58.984792+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:36:58.984792+0000","last_undegraded":"2024-09-21T12:36:58.984792+0000","last_fullsized":"2024-09-21T12:36:58.984792+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:36.124095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:36.124095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:19.399768+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"76'79","reported_seq":629,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.977922+0000","last_change":"2024-09-21T12:34:48.959991+0000","last_active":"2024-09-21T12:36:58.977922+0000","last_peered":"2024-09-21T12:36:58.977922+0000","last_clean":"2024-09-21T12:36:58.977922+0000","last_became_active":"2024-09-21T12:27:16.957569+0000","last_became_peered":"2024-09-21T12:27:16.957569+0000","last_unstale":"2024-09-21T12:36:58.977922+0000","last_undegraded":"2024-09-21T12:36:58.977922+0000","last_fullsized":"2024-09-21T12:36:58.977922+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:48.959916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:48.959916+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":194,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.972239+0000","last_change":"2024-09-21T12:34:05.193021+0000","last_active":"2024-09-21T12:36:58.972239+0000","last_peered":"2024-09-21T12:36:58.972239+0000","last_clean":"2024-09-21T12:36:58.972239+0000","last_became_active":"2024-09-21T12:32:38.428997+0000","last_became_peered":"2024-09-21T12:32:38.428997+0000","last_unstale":"2024-09-21T12:36:58.972239+0000","last_undegraded":"2024-09-21T12:36:58.972239+0000","last_fullsized":"2024-09-21T12:36:58.972239+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:05.192971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:34:05.192971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":868,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.976176+0000","last_change":"2024-09-21T12:36:00.127510+0000","last_active":"2024-09-21T12:36:58.976176+0000","last_peered":"2024-09-21T12:36:58.976176+0000","last_clean":"2024-09-21T12:36:58.976176+0000","last_became_active":"2024-09-21T12:36:00.127153+0000","last_became_peered":"2024-09-21T12:36:00.127153+0000","last_unstale":"2024-09-21T12:36:58.976176+0000","last_undegraded":"2024-09-21T12:36:58.976176+0000","last_fullsized":"2024-09-21T12:36:58.976176+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:56.016840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:56.016840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:07.001046+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":850,"reported_epoch":436,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:58.985123+0000","last_change":"2024-09-21T12:34:34.341728+0000","last_active":"2024-09-21T12:36:58.985123+0000","last_peered":"2024-09-21T12:36:58.985123+0000","last_clean":"2024-09-21T12:34:29.305282+0000","last_became_active":"2024-09-21T12:34:34.341728+0000","last_became_peered":"2024-09-21T12:34:34.341728+0000","last_unstale":"2024-09-21T12:36:58.985123+0000","last_undegraded":"2024-09-21T12:34:34.336150+0000","last_fullsized":"2024-09-21T12:34:34.336049+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:33:01.668584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:01.668584+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:48.550794+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":829,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.987515+0000","last_change":"2024-09-21T12:35:50.957724+0000","last_active":"2024-09-21T12:36:58.987515+0000","last_peered":"2024-09-21T12:36:58.987515+0000","last_clean":"2024-09-21T12:36:58.987515+0000","last_became_active":"2024-09-21T12:32:38.425958+0000","last_became_peered":"2024-09-21T12:32:38.425958+0000","last_unstale":"2024-09-21T12:36:58.987515+0000","last_undegraded":"2024-09-21T12:36:58.987515+0000","last_fullsized":"2024-09-21T12:36:58.987515+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:50.957677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:35:50.957677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:02.566364+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"76'79","reported_seq":758,"reported_epoch":436,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:58.984877+0000","last_change":"2024-09-21T12:36:01.149228+0000","last_active":"2024-09-21T12:36:58.984877+0000","last_peered":"2024-09-21T12:36:58.984877+0000","last_clean":"2024-09-21T12:36:58.984877+0000","last_became_active":"2024-09-21T12:36:01.148686+0000","last_became_peered":"2024-09-21T12:36:01.148686+0000","last_unstale":"2024-09-21T12:36:58.984877+0000","last_undegraded":"2024-09-21T12:36:58.984877+0000","last_fullsized":"2024-09-21T12:36:58.984877+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:27.294674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:27.294674+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:14.322523+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7,1],"avail_no_missing":["5","1","7"],"object_location_counts":[{"shards":"1,5,7","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":812,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.981798+0000","last_change":"2024-09-21T12:35:06.944278+0000","last_active":"2024-09-21T12:36:58.981798+0000","last_peered":"2024-09-21T12:36:58.981798+0000","last_clean":"2024-09-21T12:36:58.981798+0000","last_became_active":"2024-09-21T12:32:02.033276+0000","last_became_peered":"2024-09-21T12:32:02.033276+0000","last_unstale":"2024-09-21T12:36:58.981798+0000","last_undegraded":"2024-09-21T12:36:58.981798+0000","last_fullsized":"2024-09-21T12:36:58.981798+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:06.944209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:06.944209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":189,"reported_epoch":436,"state":"active+undersized","last_fresh":"2024-09-21T12:36:58.968957+0000","last_change":"2024-09-21T12:34:34.349252+0000","last_active":"2024-09-21T12:36:58.968957+0000","last_peered":"2024-09-21T12:36:58.968957+0000","last_clean":"2024-09-21T12:34:29.278130+0000","last_became_active":"2024-09-21T12:34:34.349252+0000","last_became_peered":"2024-09-21T12:34:34.349252+0000","last_unstale":"2024-09-21T12:36:58.968957+0000","last_undegraded":"2024-09-21T12:36:58.968957+0000","last_fullsized":"2024-09-21T12:34:34.329566+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:45.725968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.725968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:14.888145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":905,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.982902+0000","last_change":"2024-09-21T12:35:59.641966+0000","last_active":"2024-09-21T12:36:58.982902+0000","last_peered":"2024-09-21T12:36:58.982902+0000","last_clean":"2024-09-21T12:36:58.982902+0000","last_became_active":"2024-09-21T12:32:38.409163+0000","last_became_peered":"2024-09-21T12:32:38.409163+0000","last_unstale":"2024-09-21T12:36:58.982902+0000","last_undegraded":"2024-09-21T12:36:58.982902+0000","last_fullsized":"2024-09-21T12:36:58.982902+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T12:35:59.641895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:59.641895+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:19.348495+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":197,"reported_epoch":436,"state":"active+undersized","last_fresh":"2024-09-21T12:36:58.969686+0000","last_change":"2024-09-21T12:34:34.349550+0000","last_active":"2024-09-21T12:36:58.969686+0000","last_peered":"2024-09-21T12:36:58.969686+0000","last_clean":"2024-09-21T12:34:29.281006+0000","last_became_active":"2024-09-21T12:34:34.349550+0000","last_became_peered":"2024-09-21T12:34:34.349550+0000","last_unstale":"2024-09-21T12:36:58.969686+0000","last_undegraded":"2024-09-21T12:36:58.969686+0000","last_fullsized":"2024-09-21T12:34:34.331038+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:53.270634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:32:53.270634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:37.640748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"76'79","reported_seq":552,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.984811+0000","last_change":"2024-09-21T12:36:41.538607+0000","last_active":"2024-09-21T12:36:58.984811+0000","last_peered":"2024-09-21T12:36:58.984811+0000","last_clean":"2024-09-21T12:36:58.984811+0000","last_became_active":"2024-09-21T12:32:38.414204+0000","last_became_peered":"2024-09-21T12:32:38.414204+0000","last_unstale":"2024-09-21T12:36:58.984811+0000","last_undegraded":"2024-09-21T12:36:58.984811+0000","last_fullsized":"2024-09-21T12:36:58.984811+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:41.538559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:41.538559+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:06.792208+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":780,"reported_epoch":436,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:58.982050+0000","last_change":"2024-09-21T12:36:37.839177+0000","last_active":"2024-09-21T12:36:58.982050+0000","last_peered":"2024-09-21T12:36:58.982050+0000","last_clean":"2024-09-21T12:36:58.982050+0000","last_became_active":"2024-09-21T12:36:01.126897+0000","last_became_peered":"2024-09-21T12:36:01.126897+0000","last_unstale":"2024-09-21T12:36:58.982050+0000","last_undegraded":"2024-09-21T12:36:58.982050+0000","last_fullsized":"2024-09-21T12:36:58.982050+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:36:37.839024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:37.839024+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:04.241490+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5,3],"avail_no_missing":["7","3","5"],"object_location_counts":[{"shards":"3,5,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":855,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.986307+0000","last_change":"2024-09-21T12:36:44.561097+0000","last_active":"2024-09-21T12:36:58.986307+0000","last_peered":"2024-09-21T12:36:58.986307+0000","last_clean":"2024-09-21T12:36:58.986307+0000","last_became_active":"2024-09-21T12:32:38.425335+0000","last_became_peered":"2024-09-21T12:32:38.425335+0000","last_unstale":"2024-09-21T12:36:58.986307+0000","last_undegraded":"2024-09-21T12:36:58.986307+0000","last_fullsized":"2024-09-21T12:36:58.986307+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:36:44.561030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:36:44.561030+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:57.359274+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":924,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.979581+0000","last_change":"2024-09-21T12:36:00.173918+0000","last_active":"2024-09-21T12:36:58.979581+0000","last_peered":"2024-09-21T12:36:58.979581+0000","last_clean":"2024-09-21T12:36:58.979581+0000","last_became_active":"2024-09-21T12:36:00.148761+0000","last_became_peered":"2024-09-21T12:36:00.148761+0000","last_unstale":"2024-09-21T12:36:58.979581+0000","last_undegraded":"2024-09-21T12:36:58.979581+0000","last_fullsized":"2024-09-21T12:36:58.979581+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:32:57.082525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:57.082525+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:19.889528+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"415'18","reported_seq":265,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.972740+0000","last_change":"2024-09-21T12:36:00.148175+0000","last_active":"2024-09-21T12:36:58.972740+0000","last_peered":"2024-09-21T12:36:58.972740+0000","last_clean":"2024-09-21T12:36:58.972740+0000","last_became_active":"2024-09-21T12:36:00.147676+0000","last_became_peered":"2024-09-21T12:36:00.147676+0000","last_unstale":"2024-09-21T12:36:58.972740+0000","last_undegraded":"2024-09-21T12:36:58.972740+0000","last_fullsized":"2024-09-21T12:36:58.972740+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'15","last_scrub_stamp":"2024-09-21T12:35:11.182313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:35:11.182313+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"436'6880","reported_seq":9126,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:37:01.607163+0000","last_change":"2024-09-21T12:36:13.108581+0000","last_active":"2024-09-21T12:37:01.607163+0000","last_peered":"2024-09-21T12:37:01.607163+0000","last_clean":"2024-09-21T12:37:01.607163+0000","last_became_active":"2024-09-21T12:36:13.105161+0000","last_became_peered":"2024-09-21T12:36:13.105161+0000","last_unstale":"2024-09-21T12:37:01.607163+0000","last_undegraded":"2024-09-21T12:37:01.607163+0000","last_fullsized":"2024-09-21T12:37:01.607163+0000","mapping_epoch":420,"log_start":"423'3000","ondisk_log_start":"423'3000","created":405,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3880,"log_dups_size":0,"ondisk_log_size":3880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450887680,"num_objects":6880,"num_object_clones":0,"num_object_copies":20640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6880,"num_write_kb":440320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1078,"num_bytes_recovered":70647808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"76'79","reported_seq":357,"reported_epoch":436,"state":"active+undersized","last_fresh":"2024-09-21T12:36:58.985771+0000","last_change":"2024-09-21T12:34:34.347159+0000","last_active":"2024-09-21T12:36:58.985771+0000","last_peered":"2024-09-21T12:36:58.985771+0000","last_clean":"2024-09-21T12:34:29.305642+0000","last_became_active":"2024-09-21T12:34:34.347159+0000","last_became_peered":"2024-09-21T12:34:34.347159+0000","last_unstale":"2024-09-21T12:36:58.985771+0000","last_undegraded":"2024-09-21T12:36:58.985771+0000","last_fullsized":"2024-09-21T12:34:34.340605+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:18.151644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:18.151644+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:28.449536+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":459,"reported_epoch":436,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:58.981932+0000","last_change":"2024-09-21T12:34:34.345532+0000","last_active":"2024-09-21T12:36:58.981932+0000","last_peered":"2024-09-21T12:36:58.981932+0000","last_clean":"2024-09-21T12:34:29.295616+0000","last_became_active":"2024-09-21T12:34:34.345532+0000","last_became_peered":"2024-09-21T12:34:34.345532+0000","last_unstale":"2024-09-21T12:36:58.981932+0000","last_undegraded":"2024-09-21T12:34:34.334307+0000","last_fullsized":"2024-09-21T12:34:34.334181+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:31:56.722241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:56.722241+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:50.482287+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":276,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.979576+0000","last_change":"2024-09-21T12:36:00.140935+0000","last_active":"2024-09-21T12:36:58.979576+0000","last_peered":"2024-09-21T12:36:58.979576+0000","last_clean":"2024-09-21T12:36:58.979576+0000","last_became_active":"2024-09-21T12:36:00.140455+0000","last_became_peered":"2024-09-21T12:36:00.140455+0000","last_unstale":"2024-09-21T12:36:58.979576+0000","last_undegraded":"2024-09-21T12:36:58.979576+0000","last_fullsized":"2024-09-21T12:36:58.979576+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:05.703464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:05.703464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:21.552824+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"352'15","reported_seq":250,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.972194+0000","last_change":"2024-09-21T12:35:50.989423+0000","last_active":"2024-09-21T12:36:58.972194+0000","last_peered":"2024-09-21T12:36:58.972194+0000","last_clean":"2024-09-21T12:36:58.972194+0000","last_became_active":"2024-09-21T12:32:38.429092+0000","last_became_peered":"2024-09-21T12:32:38.429092+0000","last_unstale":"2024-09-21T12:36:58.972194+0000","last_undegraded":"2024-09-21T12:36:58.972194+0000","last_fullsized":"2024-09-21T12:36:58.972194+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'15","last_scrub_stamp":"2024-09-21T12:35:50.989377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:35:50.989377+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:19.146860+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":290,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.987436+0000","last_change":"2024-09-21T12:35:24.989679+0000","last_active":"2024-09-21T12:36:58.987436+0000","last_peered":"2024-09-21T12:36:58.987436+0000","last_clean":"2024-09-21T12:36:58.987436+0000","last_became_active":"2024-09-21T12:31:39.054600+0000","last_became_peered":"2024-09-21T12:31:39.054600+0000","last_unstale":"2024-09-21T12:36:58.987436+0000","last_undegraded":"2024-09-21T12:36:58.987436+0000","last_fullsized":"2024-09-21T12:36:58.987436+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:35:24.989615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:24.989615+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.f","version":"436'7017","reported_seq":7091,"reported_epoch":436,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:37:01.918730+0000","last_change":"2024-09-21T12:36:01.154474+0000","last_active":"2024-09-21T12:37:01.918730+0000","last_peered":"2024-09-21T12:37:01.918730+0000","last_clean":"2024-09-21T12:37:01.918730+0000","last_became_active":"2024-09-21T12:36:01.133058+0000","last_became_peered":"2024-09-21T12:36:01.133058+0000","last_unstale":"2024-09-21T12:37:01.918730+0000","last_undegraded":"2024-09-21T12:37:01.918730+0000","last_fullsized":"2024-09-21T12:37:01.918730+0000","mapping_epoch":415,"log_start":"421'2700","ondisk_log_start":"421'2700","created":405,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4317,"log_dups_size":0,"ondisk_log_size":4317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459866112,"num_objects":7017,"num_object_clones":0,"num_object_copies":21051,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7017,"num_objects_unfound":0,"num_objects_dirty":7017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7017,"num_write_kb":449088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,0],"avail_no_missing":["7","0","1"],"object_location_counts":[{"shards":"0,1,7","objects":7017}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"76'79","reported_seq":198,"reported_epoch":436,"state":"active+undersized","last_fresh":"2024-09-21T12:36:58.972447+0000","last_change":"2024-09-21T12:34:34.348261+0000","last_active":"2024-09-21T12:36:58.972447+0000","last_peered":"2024-09-21T12:36:58.972447+0000","last_clean":"2024-09-21T12:34:29.279263+0000","last_became_active":"2024-09-21T12:34:34.348261+0000","last_became_peered":"2024-09-21T12:34:34.348261+0000","last_unstale":"2024-09-21T12:36:58.972447+0000","last_undegraded":"2024-09-21T12:36:58.972447+0000","last_fullsized":"2024-09-21T12:34:34.332145+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:00.343916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:00.343916+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:39.681388+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":957,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.986992+0000","last_change":"2024-09-21T12:36:00.163117+0000","last_active":"2024-09-21T12:36:58.986992+0000","last_peered":"2024-09-21T12:36:58.986992+0000","last_clean":"2024-09-21T12:36:58.986992+0000","last_became_active":"2024-09-21T12:36:00.139849+0000","last_became_peered":"2024-09-21T12:36:00.139849+0000","last_unstale":"2024-09-21T12:36:58.986992+0000","last_undegraded":"2024-09-21T12:36:58.986992+0000","last_fullsized":"2024-09-21T12:36:58.986992+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T12:35:12.790809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:12.790809+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":808,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.982389+0000","last_change":"2024-09-21T12:36:00.152050+0000","last_active":"2024-09-21T12:36:58.982389+0000","last_peered":"2024-09-21T12:36:58.982389+0000","last_clean":"2024-09-21T12:36:58.982389+0000","last_became_active":"2024-09-21T12:36:00.151488+0000","last_became_peered":"2024-09-21T12:36:00.151488+0000","last_unstale":"2024-09-21T12:36:58.982389+0000","last_undegraded":"2024-09-21T12:36:58.982389+0000","last_fullsized":"2024-09-21T12:36:58.982389+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:22.752960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:22.752960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":208,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.968793+0000","last_change":"2024-09-21T12:36:00.137014+0000","last_active":"2024-09-21T12:36:58.968793+0000","last_peered":"2024-09-21T12:36:58.968793+0000","last_clean":"2024-09-21T12:36:58.968793+0000","last_became_active":"2024-09-21T12:36:00.136650+0000","last_became_peered":"2024-09-21T12:36:00.136650+0000","last_unstale":"2024-09-21T12:36:58.968793+0000","last_undegraded":"2024-09-21T12:36:58.968793+0000","last_fullsized":"2024-09-21T12:36:58.968793+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:12.216104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:12.216104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":190,"reported_epoch":436,"state":"active+undersized","last_fresh":"2024-09-21T12:36:58.970978+0000","last_change":"2024-09-21T12:34:34.347165+0000","last_active":"2024-09-21T12:36:58.970978+0000","last_peered":"2024-09-21T12:36:58.970978+0000","last_clean":"2024-09-21T12:34:29.278468+0000","last_became_active":"2024-09-21T12:34:34.347165+0000","last_became_peered":"2024-09-21T12:34:34.347165+0000","last_unstale":"2024-09-21T12:36:58.970978+0000","last_undegraded":"2024-09-21T12:36:58.970978+0000","last_fullsized":"2024-09-21T12:34:34.331768+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:05.143736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:29:05.143736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:19.903019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"436'6992","reported_seq":9116,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:37:01.901704+0000","last_change":"2024-09-21T12:36:08.103615+0000","last_active":"2024-09-21T12:37:01.901704+0000","last_peered":"2024-09-21T12:37:01.901704+0000","last_clean":"2024-09-21T12:37:01.901704+0000","last_became_active":"2024-09-21T12:36:08.101796+0000","last_became_peered":"2024-09-21T12:36:08.101796+0000","last_unstale":"2024-09-21T12:37:01.901704+0000","last_undegraded":"2024-09-21T12:37:01.901704+0000","last_fullsized":"2024-09-21T12:37:01.901704+0000","mapping_epoch":417,"log_start":"423'2900","ondisk_log_start":"423'2900","created":405,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4092,"log_dups_size":0,"ondisk_log_size":4092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:16.601019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458227712,"num_objects":6992,"num_object_clones":0,"num_object_copies":20976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6992,"num_write_kb":447488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1016,"num_bytes_recovered":66584576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"76'79","reported_seq":744,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.978923+0000","last_change":"2024-09-21T12:36:43.462949+0000","last_active":"2024-09-21T12:36:58.978923+0000","last_peered":"2024-09-21T12:36:58.978923+0000","last_clean":"2024-09-21T12:36:58.978923+0000","last_became_active":"2024-09-21T12:36:00.155047+0000","last_became_peered":"2024-09-21T12:36:00.155047+0000","last_unstale":"2024-09-21T12:36:58.978923+0000","last_undegraded":"2024-09-21T12:36:58.978923+0000","last_fullsized":"2024-09-21T12:36:58.978923+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:43.462874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:43.462874+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:44.145911+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":202,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.968712+0000","last_change":"2024-09-21T12:32:38.457999+0000","last_active":"2024-09-21T12:36:58.968712+0000","last_peered":"2024-09-21T12:36:58.968712+0000","last_clean":"2024-09-21T12:36:58.968712+0000","last_became_active":"2024-09-21T12:32:38.425453+0000","last_became_peered":"2024-09-21T12:32:38.425453+0000","last_unstale":"2024-09-21T12:36:58.968712+0000","last_undegraded":"2024-09-21T12:36:58.968712+0000","last_fullsized":"2024-09-21T12:36:58.968712+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-21T12:26:51.463698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:51.463698+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:07.901778+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":40,"num_read_kb":33,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":891,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.979964+0000","last_change":"2024-09-21T12:36:39.754233+0000","last_active":"2024-09-21T12:36:58.979964+0000","last_peered":"2024-09-21T12:36:58.979964+0000","last_clean":"2024-09-21T12:36:58.979964+0000","last_became_active":"2024-09-21T12:32:38.424412+0000","last_became_peered":"2024-09-21T12:32:38.424412+0000","last_unstale":"2024-09-21T12:36:58.979964+0000","last_undegraded":"2024-09-21T12:36:58.979964+0000","last_fullsized":"2024-09-21T12:36:58.979964+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:36:39.754173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:36:39.754173+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:02.184909+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"352'12","reported_seq":1049,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.982879+0000","last_change":"2024-09-21T12:34:07.827409+0000","last_active":"2024-09-21T12:36:58.982879+0000","last_peered":"2024-09-21T12:36:58.982879+0000","last_clean":"2024-09-21T12:36:58.982879+0000","last_became_active":"2024-09-21T12:32:38.409177+0000","last_became_peered":"2024-09-21T12:32:38.409177+0000","last_unstale":"2024-09-21T12:36:58.982879+0000","last_undegraded":"2024-09-21T12:36:58.982879+0000","last_fullsized":"2024-09-21T12:36:58.982879+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'12","last_scrub_stamp":"2024-09-21T12:34:07.827315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:07.827315+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:56.601246+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":780,"reported_epoch":436,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:58.981541+0000","last_change":"2024-09-21T12:36:01.138656+0000","last_active":"2024-09-21T12:36:58.981541+0000","last_peered":"2024-09-21T12:36:58.981541+0000","last_clean":"2024-09-21T12:36:58.981541+0000","last_became_active":"2024-09-21T12:36:01.135064+0000","last_became_peered":"2024-09-21T12:36:01.135064+0000","last_unstale":"2024-09-21T12:36:58.981541+0000","last_undegraded":"2024-09-21T12:36:58.981541+0000","last_fullsized":"2024-09-21T12:36:58.981541+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:59.406201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:32:59.406201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:25.816939+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6,1],"avail_no_missing":["7","1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"436'6753","reported_seq":6823,"reported_epoch":436,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:37:01.888051+0000","last_change":"2024-09-21T12:35:53.507442+0000","last_active":"2024-09-21T12:37:01.888051+0000","last_peered":"2024-09-21T12:37:01.888051+0000","last_clean":"2024-09-21T12:35:46.767214+0000","last_became_active":"2024-09-21T12:35:47.791328+0000","last_became_peered":"2024-09-21T12:35:47.791328+0000","last_unstale":"2024-09-21T12:37:01.888051+0000","last_undegraded":"2024-09-21T12:35:53.507160+0000","last_fullsized":"2024-09-21T12:35:47.787870+0000","mapping_epoch":405,"log_start":"422'2600","ondisk_log_start":"422'2600","created":405,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4153,"log_dups_size":0,"ondisk_log_size":4153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:03.374268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442564608,"num_objects":6753,"num_object_clones":0,"num_object_copies":20259,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6753,"num_write_kb":432192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":6753}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"76'79","reported_seq":276,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.982234+0000","last_change":"2024-09-21T12:36:00.151734+0000","last_active":"2024-09-21T12:36:58.982234+0000","last_peered":"2024-09-21T12:36:58.982234+0000","last_clean":"2024-09-21T12:36:58.982234+0000","last_became_active":"2024-09-21T12:36:00.151076+0000","last_became_peered":"2024-09-21T12:36:00.151076+0000","last_unstale":"2024-09-21T12:36:58.982234+0000","last_undegraded":"2024-09-21T12:36:58.982234+0000","last_fullsized":"2024-09-21T12:36:58.982234+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:31:50.320834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:31:50.320834+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:13.696624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":344,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.986544+0000","last_change":"2024-09-21T12:36:33.481707+0000","last_active":"2024-09-21T12:36:58.986544+0000","last_peered":"2024-09-21T12:36:58.986544+0000","last_clean":"2024-09-21T12:36:58.986544+0000","last_became_active":"2024-09-21T12:32:38.424412+0000","last_became_peered":"2024-09-21T12:32:38.424412+0000","last_unstale":"2024-09-21T12:36:58.986544+0000","last_undegraded":"2024-09-21T12:36:58.986544+0000","last_fullsized":"2024-09-21T12:36:58.986544+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:36:33.481627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:33.481627+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:00.222164+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":816,"reported_epoch":436,"state":"active+undersized","last_fresh":"2024-09-21T12:36:58.986409+0000","last_change":"2024-09-21T12:34:34.339986+0000","last_active":"2024-09-21T12:36:58.986409+0000","last_peered":"2024-09-21T12:36:58.986409+0000","last_clean":"2024-09-21T12:34:29.305656+0000","last_became_active":"2024-09-21T12:34:34.339986+0000","last_became_peered":"2024-09-21T12:34:34.339986+0000","last_unstale":"2024-09-21T12:36:58.986409+0000","last_undegraded":"2024-09-21T12:36:58.986409+0000","last_fullsized":"2024-09-21T12:34:34.336347+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:19.629083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:19.629083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:43.542005+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":1019,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:37:00.560182+0000","last_change":"2024-09-21T12:36:44.788732+0000","last_active":"2024-09-21T12:37:00.560182+0000","last_peered":"2024-09-21T12:37:00.560182+0000","last_clean":"2024-09-21T12:37:00.560182+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:37:00.560182+0000","last_undegraded":"2024-09-21T12:37:00.560182+0000","last_fullsized":"2024-09-21T12:37:00.560182+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:36:44.788662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:44.788662+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:47.677328+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":790,"reported_epoch":436,"state":"active+undersized","last_fresh":"2024-09-21T12:36:58.974793+0000","last_change":"2024-09-21T12:34:34.335260+0000","last_active":"2024-09-21T12:36:58.974793+0000","last_peered":"2024-09-21T12:36:58.974793+0000","last_clean":"2024-09-21T12:34:29.288868+0000","last_became_active":"2024-09-21T12:34:34.335260+0000","last_became_peered":"2024-09-21T12:34:34.335260+0000","last_unstale":"2024-09-21T12:36:58.974793+0000","last_undegraded":"2024-09-21T12:36:58.974793+0000","last_fullsized":"2024-09-21T12:34:34.331914+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:37.850100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:37.850100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:43.750121+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"436'7105","reported_seq":7175,"reported_epoch":436,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:37:01.584037+0000","last_change":"2024-09-21T12:35:53.515408+0000","last_active":"2024-09-21T12:37:01.584037+0000","last_peered":"2024-09-21T12:37:01.584037+0000","last_clean":"2024-09-21T12:35:46.767214+0000","last_became_active":"2024-09-21T12:35:47.791289+0000","last_became_peered":"2024-09-21T12:35:47.791289+0000","last_unstale":"2024-09-21T12:37:01.584037+0000","last_undegraded":"2024-09-21T12:35:53.506389+0000","last_fullsized":"2024-09-21T12:35:47.788809+0000","mapping_epoch":405,"log_start":"424'3300","ondisk_log_start":"424'3300","created":405,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3805,"log_dups_size":0,"ondisk_log_size":3805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:03.677425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":465633280,"num_objects":7105,"num_object_clones":0,"num_object_copies":21315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7105,"num_write_kb":454720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":7105}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"76'79","reported_seq":479,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.979101+0000","last_change":"2024-09-21T12:32:52.986242+0000","last_active":"2024-09-21T12:36:58.979101+0000","last_peered":"2024-09-21T12:36:58.979101+0000","last_clean":"2024-09-21T12:36:58.979101+0000","last_became_active":"2024-09-21T12:29:05.835884+0000","last_became_peered":"2024-09-21T12:29:05.835884+0000","last_unstale":"2024-09-21T12:36:58.979101+0000","last_undegraded":"2024-09-21T12:36:58.979101+0000","last_fullsized":"2024-09-21T12:36:58.979101+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:52.986175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:52.986175+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:24.882111+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":903,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.978009+0000","last_change":"2024-09-21T12:36:36.774685+0000","last_active":"2024-09-21T12:36:58.978009+0000","last_peered":"2024-09-21T12:36:58.978009+0000","last_clean":"2024-09-21T12:36:58.978009+0000","last_became_active":"2024-09-21T12:36:00.127312+0000","last_became_peered":"2024-09-21T12:36:00.127312+0000","last_unstale":"2024-09-21T12:36:58.978009+0000","last_undegraded":"2024-09-21T12:36:58.978009+0000","last_fullsized":"2024-09-21T12:36:58.978009+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'29","last_scrub_stamp":"2024-09-21T12:36:36.774599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:36.774599+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:43.150659+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":791,"reported_epoch":436,"state":"active+undersized","last_fresh":"2024-09-21T12:36:58.992853+0000","last_change":"2024-09-21T12:34:34.337276+0000","last_active":"2024-09-21T12:36:58.992853+0000","last_peered":"2024-09-21T12:36:58.992853+0000","last_clean":"2024-09-21T12:34:29.286433+0000","last_became_active":"2024-09-21T12:34:34.337276+0000","last_became_peered":"2024-09-21T12:34:34.337276+0000","last_unstale":"2024-09-21T12:36:58.992853+0000","last_undegraded":"2024-09-21T12:36:58.992853+0000","last_fullsized":"2024-09-21T12:34:34.330247+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:38.250806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:38.250806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:29.240702+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":383,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.976310+0000","last_change":"2024-09-21T12:36:00.924159+0000","last_active":"2024-09-21T12:36:58.976310+0000","last_peered":"2024-09-21T12:36:58.976310+0000","last_clean":"2024-09-21T12:36:58.976310+0000","last_became_active":"2024-09-21T12:29:05.825869+0000","last_became_peered":"2024-09-21T12:29:05.825869+0000","last_unstale":"2024-09-21T12:36:58.976310+0000","last_undegraded":"2024-09-21T12:36:58.976310+0000","last_fullsized":"2024-09-21T12:36:58.976310+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:00.924084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:00.924084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:28.472139+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":788,"reported_epoch":436,"state":"active+undersized","last_fresh":"2024-09-21T12:36:58.986094+0000","last_change":"2024-09-21T12:34:34.339352+0000","last_active":"2024-09-21T12:36:58.986094+0000","last_peered":"2024-09-21T12:36:58.986094+0000","last_clean":"2024-09-21T12:34:29.284095+0000","last_became_active":"2024-09-21T12:34:34.339352+0000","last_became_peered":"2024-09-21T12:34:34.339352+0000","last_unstale":"2024-09-21T12:36:58.986094+0000","last_undegraded":"2024-09-21T12:36:58.986094+0000","last_fullsized":"2024-09-21T12:34:34.329336+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:05.510620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:05.510620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:25.203926+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"436'6951","reported_seq":7024,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:37:02.834376+0000","last_change":"2024-09-21T12:35:47.791371+0000","last_active":"2024-09-21T12:37:02.834376+0000","last_peered":"2024-09-21T12:37:02.834376+0000","last_clean":"2024-09-21T12:37:02.834376+0000","last_became_active":"2024-09-21T12:35:47.791120+0000","last_became_peered":"2024-09-21T12:35:47.791120+0000","last_unstale":"2024-09-21T12:37:02.834376+0000","last_undegraded":"2024-09-21T12:37:02.834376+0000","last_fullsized":"2024-09-21T12:37:02.834376+0000","mapping_epoch":405,"log_start":"423'3100","ondisk_log_start":"423'3100","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3851,"log_dups_size":0,"ondisk_log_size":3851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455540736,"num_objects":6951,"num_object_clones":0,"num_object_copies":20853,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6951,"num_write_kb":444864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"76'79","reported_seq":591,"reported_epoch":436,"state":"active+undersized","last_fresh":"2024-09-21T12:36:58.976367+0000","last_change":"2024-09-21T12:34:34.337922+0000","last_active":"2024-09-21T12:36:58.976367+0000","last_peered":"2024-09-21T12:36:58.976367+0000","last_clean":"2024-09-21T12:34:29.289671+0000","last_became_active":"2024-09-21T12:34:34.337922+0000","last_became_peered":"2024-09-21T12:34:34.337922+0000","last_unstale":"2024-09-21T12:36:58.976367+0000","last_undegraded":"2024-09-21T12:36:58.976367+0000","last_fullsized":"2024-09-21T12:34:34.332211+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:34.055720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:34.055720+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:32.875426+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"389'41","reported_seq":1056,"reported_epoch":436,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:58.992515+0000","last_change":"2024-09-21T12:34:34.339316+0000","last_active":"2024-09-21T12:36:58.992515+0000","last_peered":"2024-09-21T12:36:58.992515+0000","last_clean":"2024-09-21T12:34:30.529659+0000","last_became_active":"2024-09-21T12:34:34.339316+0000","last_became_peered":"2024-09-21T12:34:34.339316+0000","last_unstale":"2024-09-21T12:36:58.992515+0000","last_undegraded":"2024-09-21T12:34:34.330286+0000","last_fullsized":"2024-09-21T12:34:34.330177+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'40","last_scrub_stamp":"2024-09-21T12:32:25.139435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:25.139435+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:40.440122+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":194,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.969204+0000","last_change":"2024-09-21T12:35:09.182021+0000","last_active":"2024-09-21T12:36:58.969204+0000","last_peered":"2024-09-21T12:36:58.969204+0000","last_clean":"2024-09-21T12:36:58.969204+0000","last_became_active":"2024-09-21T12:32:38.426309+0000","last_became_peered":"2024-09-21T12:32:38.426309+0000","last_unstale":"2024-09-21T12:36:58.969204+0000","last_undegraded":"2024-09-21T12:36:58.969204+0000","last_fullsized":"2024-09-21T12:36:58.969204+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:09.181952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:09.181952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":521,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.986838+0000","last_change":"2024-09-21T12:35:27.114524+0000","last_active":"2024-09-21T12:36:58.986838+0000","last_peered":"2024-09-21T12:36:58.986838+0000","last_clean":"2024-09-21T12:36:58.986838+0000","last_became_active":"2024-09-21T12:26:54.083584+0000","last_became_peered":"2024-09-21T12:26:54.083584+0000","last_unstale":"2024-09-21T12:36:58.986838+0000","last_undegraded":"2024-09-21T12:36:58.986838+0000","last_fullsized":"2024-09-21T12:36:58.986838+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T12:35:27.114442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:35:27.114442+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:53.587329+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":833,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.976627+0000","last_change":"2024-09-21T12:33:00.087826+0000","last_active":"2024-09-21T12:36:58.976627+0000","last_peered":"2024-09-21T12:36:58.976627+0000","last_clean":"2024-09-21T12:36:58.976627+0000","last_became_active":"2024-09-21T12:32:38.424402+0000","last_became_peered":"2024-09-21T12:32:38.424402+0000","last_unstale":"2024-09-21T12:36:58.976627+0000","last_undegraded":"2024-09-21T12:36:58.976627+0000","last_fullsized":"2024-09-21T12:36:58.976627+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:00.087746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:33:00.087746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:10.889168+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":832,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.980928+0000","last_change":"2024-09-21T12:36:00.144733+0000","last_active":"2024-09-21T12:36:58.980928+0000","last_peered":"2024-09-21T12:36:58.980928+0000","last_clean":"2024-09-21T12:36:58.980928+0000","last_became_active":"2024-09-21T12:36:00.144526+0000","last_became_peered":"2024-09-21T12:36:00.144526+0000","last_unstale":"2024-09-21T12:36:58.980928+0000","last_undegraded":"2024-09-21T12:36:58.980928+0000","last_fullsized":"2024-09-21T12:36:58.980928+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:59.899499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:59.899499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:54.611578+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"436'7082","reported_seq":7154,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:37:02.822554+0000","last_change":"2024-09-21T12:35:47.848839+0000","last_active":"2024-09-21T12:37:02.822554+0000","last_peered":"2024-09-21T12:37:02.822554+0000","last_clean":"2024-09-21T12:37:02.822554+0000","last_became_active":"2024-09-21T12:35:47.848543+0000","last_became_peered":"2024-09-21T12:35:47.848543+0000","last_unstale":"2024-09-21T12:37:02.822554+0000","last_undegraded":"2024-09-21T12:37:02.822554+0000","last_fullsized":"2024-09-21T12:37:02.822554+0000","mapping_epoch":405,"log_start":"423'3200","ondisk_log_start":"423'3200","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3882,"log_dups_size":0,"ondisk_log_size":3882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464125952,"num_objects":7082,"num_object_clones":0,"num_object_copies":21246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7082,"num_write_kb":453248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'79","reported_seq":218,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.981181+0000","last_change":"2024-09-21T12:36:01.132046+0000","last_active":"2024-09-21T12:36:58.981181+0000","last_peered":"2024-09-21T12:36:58.981181+0000","last_clean":"2024-09-21T12:36:58.981181+0000","last_became_active":"2024-09-21T12:36:01.131622+0000","last_became_peered":"2024-09-21T12:36:01.131622+0000","last_unstale":"2024-09-21T12:36:58.981181+0000","last_undegraded":"2024-09-21T12:36:58.981181+0000","last_fullsized":"2024-09-21T12:36:58.981181+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:30:39.179014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:30:39.179014+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:24.773542+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":298,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.987120+0000","last_change":"2024-09-21T12:35:18.061854+0000","last_active":"2024-09-21T12:36:58.987120+0000","last_peered":"2024-09-21T12:36:58.987120+0000","last_clean":"2024-09-21T12:36:58.987120+0000","last_became_active":"2024-09-21T12:31:39.057316+0000","last_became_peered":"2024-09-21T12:31:39.057316+0000","last_unstale":"2024-09-21T12:36:58.987120+0000","last_undegraded":"2024-09-21T12:36:58.987120+0000","last_fullsized":"2024-09-21T12:36:58.987120+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'17","last_scrub_stamp":"2024-09-21T12:35:18.061753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:18.061753+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:48.554338+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":824,"reported_epoch":436,"state":"active+undersized","last_fresh":"2024-09-21T12:36:58.985185+0000","last_change":"2024-09-21T12:34:34.343641+0000","last_active":"2024-09-21T12:36:58.985185+0000","last_peered":"2024-09-21T12:36:58.985185+0000","last_clean":"2024-09-21T12:34:29.286935+0000","last_became_active":"2024-09-21T12:34:34.343641+0000","last_became_peered":"2024-09-21T12:34:34.343641+0000","last_unstale":"2024-09-21T12:36:58.985185+0000","last_undegraded":"2024-09-21T12:36:58.985185+0000","last_fullsized":"2024-09-21T12:34:34.329247+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:00.211563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:34:00.211563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:30.227747+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":132,"reported_epoch":436,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:58.977105+0000","last_change":"2024-09-21T12:34:34.338167+0000","last_active":"2024-09-21T12:36:58.977105+0000","last_peered":"2024-09-21T12:36:58.977105+0000","last_clean":"2024-09-21T12:34:29.291114+0000","last_became_active":"2024-09-21T12:34:34.338167+0000","last_became_peered":"2024-09-21T12:34:34.338167+0000","last_unstale":"2024-09-21T12:36:58.977105+0000","last_undegraded":"2024-09-21T12:34:34.335226+0000","last_fullsized":"2024-09-21T12:34:34.335117+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:32:20.845428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:32:20.845428+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:19.597743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":132,"reported_epoch":436,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:58.985297+0000","last_change":"2024-09-21T12:36:01.125293+0000","last_active":"2024-09-21T12:36:58.985297+0000","last_peered":"2024-09-21T12:36:58.985297+0000","last_clean":"2024-09-21T12:36:58.985297+0000","last_became_active":"2024-09-21T12:36:01.124868+0000","last_became_peered":"2024-09-21T12:36:01.124868+0000","last_unstale":"2024-09-21T12:36:58.985297+0000","last_undegraded":"2024-09-21T12:36:58.985297+0000","last_fullsized":"2024-09-21T12:36:58.985297+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:02.295255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:02.295255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:20.382099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7,4],"avail_no_missing":["5","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":25,"reported_epoch":436,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:58.976554+0000","last_change":"2024-09-21T12:36:40.306122+0000","last_active":"2024-09-21T12:36:58.976554+0000","last_peered":"2024-09-21T12:36:58.976554+0000","last_clean":"2024-09-21T12:36:58.976554+0000","last_became_active":"2024-09-21T12:36:40.305602+0000","last_became_peered":"2024-09-21T12:36:40.305602+0000","last_unstale":"2024-09-21T12:36:58.976554+0000","last_undegraded":"2024-09-21T12:36:58.976554+0000","last_fullsized":"2024-09-21T12:36:58.976554+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:07.541665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:07.541665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6,3],"avail_no_missing":["0","3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"436'6993","reported_seq":7066,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:37:02.822542+0000","last_change":"2024-09-21T12:35:47.791375+0000","last_active":"2024-09-21T12:37:02.822542+0000","last_peered":"2024-09-21T12:37:02.822542+0000","last_clean":"2024-09-21T12:37:02.822542+0000","last_became_active":"2024-09-21T12:35:47.791073+0000","last_became_peered":"2024-09-21T12:35:47.791073+0000","last_unstale":"2024-09-21T12:37:02.822542+0000","last_undegraded":"2024-09-21T12:37:02.822542+0000","last_fullsized":"2024-09-21T12:37:02.822542+0000","mapping_epoch":405,"log_start":"423'3100","ondisk_log_start":"423'3100","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3893,"log_dups_size":0,"ondisk_log_size":3893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458293248,"num_objects":6993,"num_object_clones":0,"num_object_copies":20979,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6993,"num_write_kb":447552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"76'79","reported_seq":761,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.988110+0000","last_change":"2024-09-21T12:36:41.495668+0000","last_active":"2024-09-21T12:36:58.988110+0000","last_peered":"2024-09-21T12:36:58.988110+0000","last_clean":"2024-09-21T12:36:58.988110+0000","last_became_active":"2024-09-21T12:32:38.424710+0000","last_became_peered":"2024-09-21T12:32:38.424710+0000","last_unstale":"2024-09-21T12:36:58.988110+0000","last_undegraded":"2024-09-21T12:36:58.988110+0000","last_fullsized":"2024-09-21T12:36:58.988110+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:41.495532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:41.495532+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:56.228849+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":833,"reported_epoch":436,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:58.979669+0000","last_change":"2024-09-21T12:34:34.334240+0000","last_active":"2024-09-21T12:36:58.979669+0000","last_peered":"2024-09-21T12:36:58.979669+0000","last_clean":"2024-09-21T12:34:29.290312+0000","last_became_active":"2024-09-21T12:34:34.334240+0000","last_became_peered":"2024-09-21T12:34:34.334240+0000","last_unstale":"2024-09-21T12:36:58.979669+0000","last_undegraded":"2024-09-21T12:34:34.331492+0000","last_fullsized":"2024-09-21T12:34:34.331281+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:28:45.782955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.782955+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:51.732517+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":848,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.974919+0000","last_change":"2024-09-21T12:32:53.934634+0000","last_active":"2024-09-21T12:36:58.974919+0000","last_peered":"2024-09-21T12:36:58.974919+0000","last_clean":"2024-09-21T12:36:58.974919+0000","last_became_active":"2024-09-21T12:32:38.411004+0000","last_became_peered":"2024-09-21T12:32:38.411004+0000","last_unstale":"2024-09-21T12:36:58.974919+0000","last_undegraded":"2024-09-21T12:36:58.974919+0000","last_fullsized":"2024-09-21T12:36:58.974919+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:53.934583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:53.934583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:17.842220+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":128,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:37:00.243754+0000","last_change":"2024-09-21T12:36:27.852222+0000","last_active":"2024-09-21T12:37:00.243754+0000","last_peered":"2024-09-21T12:37:00.243754+0000","last_clean":"2024-09-21T12:37:00.243754+0000","last_became_active":"2024-09-21T12:36:00.144053+0000","last_became_peered":"2024-09-21T12:36:00.144053+0000","last_unstale":"2024-09-21T12:37:00.243754+0000","last_undegraded":"2024-09-21T12:37:00.243754+0000","last_fullsized":"2024-09-21T12:37:00.243754+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-21T12:36:27.852030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:36:27.852030+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:48.419358+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":68,"num_read_kb":63,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":242,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.988094+0000","last_change":"2024-09-21T12:36:36.590854+0000","last_active":"2024-09-21T12:36:58.988094+0000","last_peered":"2024-09-21T12:36:58.988094+0000","last_clean":"2024-09-21T12:36:58.988094+0000","last_became_active":"2024-09-21T12:32:02.030240+0000","last_became_peered":"2024-09-21T12:32:02.030240+0000","last_unstale":"2024-09-21T12:36:58.988094+0000","last_undegraded":"2024-09-21T12:36:58.988094+0000","last_fullsized":"2024-09-21T12:36:58.988094+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:36.590782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:36.590782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:38.226971+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":286,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.992119+0000","last_change":"2024-09-21T12:35:22.978507+0000","last_active":"2024-09-21T12:36:58.992119+0000","last_peered":"2024-09-21T12:36:58.992119+0000","last_clean":"2024-09-21T12:36:58.992119+0000","last_became_active":"2024-09-21T12:31:39.060160+0000","last_became_peered":"2024-09-21T12:31:39.060160+0000","last_unstale":"2024-09-21T12:36:58.992119+0000","last_undegraded":"2024-09-21T12:36:58.992119+0000","last_fullsized":"2024-09-21T12:36:58.992119+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:22.978438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:22.978438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:54.617613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"436'7016","reported_seq":7089,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:37:02.822562+0000","last_change":"2024-09-21T12:35:47.849177+0000","last_active":"2024-09-21T12:37:02.822562+0000","last_peered":"2024-09-21T12:37:02.822562+0000","last_clean":"2024-09-21T12:37:02.822562+0000","last_became_active":"2024-09-21T12:35:47.848768+0000","last_became_peered":"2024-09-21T12:35:47.848768+0000","last_unstale":"2024-09-21T12:37:02.822562+0000","last_undegraded":"2024-09-21T12:37:02.822562+0000","last_fullsized":"2024-09-21T12:37:02.822562+0000","mapping_epoch":405,"log_start":"423'3200","ondisk_log_start":"423'3200","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3816,"log_dups_size":0,"ondisk_log_size":3816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459800576,"num_objects":7016,"num_object_clones":0,"num_object_copies":21048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7016,"num_write_kb":449024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"415'35","reported_seq":323,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.972786+0000","last_change":"2024-09-21T12:36:43.681619+0000","last_active":"2024-09-21T12:36:58.972786+0000","last_peered":"2024-09-21T12:36:58.972786+0000","last_clean":"2024-09-21T12:36:58.972786+0000","last_became_active":"2024-09-21T12:36:00.139342+0000","last_became_peered":"2024-09-21T12:36:00.139342+0000","last_unstale":"2024-09-21T12:36:58.972786+0000","last_undegraded":"2024-09-21T12:36:58.972786+0000","last_fullsized":"2024-09-21T12:36:58.972786+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'35","last_scrub_stamp":"2024-09-21T12:36:43.681471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:43.681471+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:50.711798+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":809,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.986408+0000","last_change":"2024-09-21T12:34:01.815496+0000","last_active":"2024-09-21T12:36:58.986408+0000","last_peered":"2024-09-21T12:36:58.986408+0000","last_clean":"2024-09-21T12:36:58.986408+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:36:58.986408+0000","last_undegraded":"2024-09-21T12:36:58.986408+0000","last_fullsized":"2024-09-21T12:36:58.986408+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:34:01.815445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:34:01.815445+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":54,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.981182+0000","last_change":"2024-09-21T12:36:00.144564+0000","last_active":"2024-09-21T12:36:58.981182+0000","last_peered":"2024-09-21T12:36:58.981182+0000","last_clean":"2024-09-21T12:36:58.981182+0000","last_became_active":"2024-09-21T12:36:00.134878+0000","last_became_peered":"2024-09-21T12:36:00.134878+0000","last_unstale":"2024-09-21T12:36:58.981182+0000","last_undegraded":"2024-09-21T12:36:58.981182+0000","last_fullsized":"2024-09-21T12:36:58.981182+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:35:24.687711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:35:24.687711+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T12:37:11.734028+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26834,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":9,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26834,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":215,"reported_epoch":436,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:58.981544+0000","last_change":"2024-09-21T12:36:01.130402+0000","last_active":"2024-09-21T12:36:58.981544+0000","last_peered":"2024-09-21T12:36:58.981544+0000","last_clean":"2024-09-21T12:36:58.981544+0000","last_became_active":"2024-09-21T12:36:01.129957+0000","last_became_peered":"2024-09-21T12:36:01.129957+0000","last_unstale":"2024-09-21T12:36:58.981544+0000","last_undegraded":"2024-09-21T12:36:58.981544+0000","last_fullsized":"2024-09-21T12:36:58.981544+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:36.684459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:36.684459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:21.752176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5,0],"avail_no_missing":["7","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":650,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.981668+0000","last_change":"2024-09-21T12:36:00.653180+0000","last_active":"2024-09-21T12:36:58.981668+0000","last_peered":"2024-09-21T12:36:58.981668+0000","last_clean":"2024-09-21T12:36:58.981668+0000","last_became_active":"2024-09-21T12:32:38.409187+0000","last_became_peered":"2024-09-21T12:32:38.409187+0000","last_unstale":"2024-09-21T12:36:58.981668+0000","last_undegraded":"2024-09-21T12:36:58.981668+0000","last_fullsized":"2024-09-21T12:36:58.981668+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:00.653130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:00.653130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:10.867677+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1","version":"436'6953","reported_seq":7024,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:37:02.823013+0000","last_change":"2024-09-21T12:35:47.788808+0000","last_active":"2024-09-21T12:37:02.823013+0000","last_peered":"2024-09-21T12:37:02.823013+0000","last_clean":"2024-09-21T12:37:02.823013+0000","last_became_active":"2024-09-21T12:35:47.788318+0000","last_became_peered":"2024-09-21T12:35:47.788318+0000","last_unstale":"2024-09-21T12:37:02.823013+0000","last_undegraded":"2024-09-21T12:37:02.823013+0000","last_fullsized":"2024-09-21T12:37:02.823013+0000","mapping_epoch":405,"log_start":"423'3100","ondisk_log_start":"423'3100","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3853,"log_dups_size":0,"ondisk_log_size":3853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455671808,"num_objects":6953,"num_object_clones":0,"num_object_copies":20859,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6953,"num_write_kb":444992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"76'79","reported_seq":749,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.988666+0000","last_change":"2024-09-21T12:33:04.508642+0000","last_active":"2024-09-21T12:36:58.988666+0000","last_peered":"2024-09-21T12:36:58.988666+0000","last_clean":"2024-09-21T12:36:58.988666+0000","last_became_active":"2024-09-21T12:27:14.954351+0000","last_became_peered":"2024-09-21T12:27:14.954351+0000","last_unstale":"2024-09-21T12:36:58.988666+0000","last_undegraded":"2024-09-21T12:36:58.988666+0000","last_fullsized":"2024-09-21T12:36:58.988666+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:04.508319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:04.508319+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:23.209734+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":213,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.972334+0000","last_change":"2024-09-21T12:36:41.666876+0000","last_active":"2024-09-21T12:36:58.972334+0000","last_peered":"2024-09-21T12:36:58.972334+0000","last_clean":"2024-09-21T12:36:58.972334+0000","last_became_active":"2024-09-21T12:32:38.419714+0000","last_became_peered":"2024-09-21T12:32:38.419714+0000","last_unstale":"2024-09-21T12:36:58.972334+0000","last_undegraded":"2024-09-21T12:36:58.972334+0000","last_fullsized":"2024-09-21T12:36:58.972334+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:36:41.666812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:41.666812+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:50.318849+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":218,"reported_epoch":436,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:58.968868+0000","last_change":"2024-09-21T12:36:40.708599+0000","last_active":"2024-09-21T12:36:58.968868+0000","last_peered":"2024-09-21T12:36:58.968868+0000","last_clean":"2024-09-21T12:36:58.968868+0000","last_became_active":"2024-09-21T12:36:01.130874+0000","last_became_peered":"2024-09-21T12:36:01.130874+0000","last_unstale":"2024-09-21T12:36:58.968868+0000","last_undegraded":"2024-09-21T12:36:58.968868+0000","last_fullsized":"2024-09-21T12:36:58.968868+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:40.708469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:36:40.708469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:10.539508+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7,1],"avail_no_missing":["4","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":193,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.970887+0000","last_change":"2024-09-21T12:33:59.196744+0000","last_active":"2024-09-21T12:36:58.970887+0000","last_peered":"2024-09-21T12:36:58.970887+0000","last_clean":"2024-09-21T12:36:58.970887+0000","last_became_active":"2024-09-21T12:32:38.425683+0000","last_became_peered":"2024-09-21T12:32:38.425683+0000","last_unstale":"2024-09-21T12:36:58.970887+0000","last_undegraded":"2024-09-21T12:36:58.970887+0000","last_fullsized":"2024-09-21T12:36:58.970887+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:59.196667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:33:59.196667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":790,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.979033+0000","last_change":"2024-09-21T12:35:24.786713+0000","last_active":"2024-09-21T12:36:58.979033+0000","last_peered":"2024-09-21T12:36:58.979033+0000","last_clean":"2024-09-21T12:36:58.979033+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:36:58.979033+0000","last_undegraded":"2024-09-21T12:36:58.979033+0000","last_fullsized":"2024-09-21T12:36:58.979033+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:24.786609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:24.786609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"436'7012","reported_seq":7082,"reported_epoch":436,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:37:02.837850+0000","last_change":"2024-09-21T12:35:53.505439+0000","last_active":"2024-09-21T12:37:02.837850+0000","last_peered":"2024-09-21T12:37:02.837850+0000","last_clean":"2024-09-21T12:35:46.767214+0000","last_became_active":"2024-09-21T12:35:47.787239+0000","last_became_peered":"2024-09-21T12:35:47.787239+0000","last_unstale":"2024-09-21T12:37:02.837850+0000","last_undegraded":"2024-09-21T12:35:53.504658+0000","last_fullsized":"2024-09-21T12:35:47.783070+0000","mapping_epoch":405,"log_start":"423'3200","ondisk_log_start":"423'3200","created":405,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3812,"log_dups_size":0,"ondisk_log_size":3812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:16.320842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459538432,"num_objects":7012,"num_object_clones":0,"num_object_copies":21036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7012,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7012,"num_write_kb":448768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":7012}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"76'79","reported_seq":130,"reported_epoch":436,"state":"active+undersized","last_fresh":"2024-09-21T12:36:58.979660+0000","last_change":"2024-09-21T12:34:34.338192+0000","last_active":"2024-09-21T12:36:58.979660+0000","last_peered":"2024-09-21T12:36:58.979660+0000","last_clean":"2024-09-21T12:34:29.289953+0000","last_became_active":"2024-09-21T12:34:34.338192+0000","last_became_peered":"2024-09-21T12:34:34.338192+0000","last_unstale":"2024-09-21T12:36:58.979660+0000","last_undegraded":"2024-09-21T12:36:58.979660+0000","last_fullsized":"2024-09-21T12:34:34.334873+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:17.035216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:17.035216+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":215,"reported_epoch":436,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:58.972190+0000","last_change":"2024-09-21T12:34:34.341886+0000","last_active":"2024-09-21T12:36:58.972190+0000","last_peered":"2024-09-21T12:36:58.972190+0000","last_clean":"2024-09-21T12:34:29.279382+0000","last_became_active":"2024-09-21T12:34:34.341886+0000","last_became_peered":"2024-09-21T12:34:34.341886+0000","last_unstale":"2024-09-21T12:36:58.972190+0000","last_undegraded":"2024-09-21T12:34:34.333296+0000","last_fullsized":"2024-09-21T12:34:34.332270+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'31","last_scrub_stamp":"2024-09-21T12:33:08.360012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:08.360012+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:25.815464+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":202,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.969046+0000","last_change":"2024-09-21T12:35:18.088890+0000","last_active":"2024-09-21T12:36:58.969046+0000","last_peered":"2024-09-21T12:36:58.969046+0000","last_clean":"2024-09-21T12:36:58.969046+0000","last_became_active":"2024-09-21T12:32:38.424451+0000","last_became_peered":"2024-09-21T12:32:38.424451+0000","last_unstale":"2024-09-21T12:36:58.969046+0000","last_undegraded":"2024-09-21T12:36:58.969046+0000","last_fullsized":"2024-09-21T12:36:58.969046+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:18.088831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:18.088831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":823,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.992206+0000","last_change":"2024-09-21T12:36:43.539109+0000","last_active":"2024-09-21T12:36:58.992206+0000","last_peered":"2024-09-21T12:36:58.992206+0000","last_clean":"2024-09-21T12:36:58.992206+0000","last_became_active":"2024-09-21T12:32:38.425177+0000","last_became_peered":"2024-09-21T12:32:38.425177+0000","last_unstale":"2024-09-21T12:36:58.992206+0000","last_undegraded":"2024-09-21T12:36:58.992206+0000","last_fullsized":"2024-09-21T12:36:58.992206+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:43.539053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:43.539053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:59.470609+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":740,"reported_epoch":436,"state":"active+undersized","last_fresh":"2024-09-21T12:36:58.981487+0000","last_change":"2024-09-21T12:34:34.345710+0000","last_active":"2024-09-21T12:36:58.981487+0000","last_peered":"2024-09-21T12:36:58.981487+0000","last_clean":"2024-09-21T12:34:29.294810+0000","last_became_active":"2024-09-21T12:34:34.345710+0000","last_became_peered":"2024-09-21T12:34:34.345710+0000","last_unstale":"2024-09-21T12:36:58.981487+0000","last_undegraded":"2024-09-21T12:36:58.981487+0000","last_fullsized":"2024-09-21T12:34:34.334512+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:20.005906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:34:20.005906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3","version":"436'6914","reported_seq":9105,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:37:01.926175+0000","last_change":"2024-09-21T12:36:13.109138+0000","last_active":"2024-09-21T12:37:01.926175+0000","last_peered":"2024-09-21T12:37:01.926175+0000","last_clean":"2024-09-21T12:37:01.926175+0000","last_became_active":"2024-09-21T12:36:13.105381+0000","last_became_peered":"2024-09-21T12:36:13.105381+0000","last_unstale":"2024-09-21T12:37:01.926175+0000","last_undegraded":"2024-09-21T12:37:01.926175+0000","last_fullsized":"2024-09-21T12:37:01.926175+0000","mapping_epoch":420,"log_start":"421'2400","ondisk_log_start":"421'2400","created":405,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4514,"log_dups_size":0,"ondisk_log_size":4514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:58.232691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453115904,"num_objects":6914,"num_object_clones":0,"num_object_copies":20742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6914,"num_write_kb":442496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1051,"num_bytes_recovered":68878336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"76'79","reported_seq":866,"reported_epoch":436,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:58.981150+0000","last_change":"2024-09-21T12:36:01.128488+0000","last_active":"2024-09-21T12:36:58.981150+0000","last_peered":"2024-09-21T12:36:58.981150+0000","last_clean":"2024-09-21T12:36:58.981150+0000","last_became_active":"2024-09-21T12:36:01.127851+0000","last_became_peered":"2024-09-21T12:36:01.127851+0000","last_unstale":"2024-09-21T12:36:58.981150+0000","last_undegraded":"2024-09-21T12:36:58.981150+0000","last_fullsized":"2024-09-21T12:36:58.981150+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:05.500994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:05.500994+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:23.785504+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,5],"avail_no_missing":["7","3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":201,"reported_epoch":436,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:58.969389+0000","last_change":"2024-09-21T12:34:34.347138+0000","last_active":"2024-09-21T12:36:58.969389+0000","last_peered":"2024-09-21T12:36:58.969389+0000","last_clean":"2024-09-21T12:34:29.280929+0000","last_became_active":"2024-09-21T12:34:34.347138+0000","last_became_peered":"2024-09-21T12:34:34.347138+0000","last_unstale":"2024-09-21T12:36:58.969389+0000","last_undegraded":"2024-09-21T12:34:34.332331+0000","last_fullsized":"2024-09-21T12:34:34.330784+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:32:26.690403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:26.690403+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:22.898893+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":822,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.981640+0000","last_change":"2024-09-21T12:32:38.404305+0000","last_active":"2024-09-21T12:36:58.981640+0000","last_peered":"2024-09-21T12:36:58.981640+0000","last_clean":"2024-09-21T12:36:58.981640+0000","last_became_active":"2024-09-21T12:32:38.403997+0000","last_became_peered":"2024-09-21T12:32:38.403997+0000","last_unstale":"2024-09-21T12:36:58.981640+0000","last_undegraded":"2024-09-21T12:36:58.981640+0000","last_fullsized":"2024-09-21T12:36:58.981640+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:47.248389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:31:47.248389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:18.590240+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"416'14","reported_seq":181,"reported_epoch":436,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:58.982429+0000","last_change":"2024-09-21T12:36:31.838103+0000","last_active":"2024-09-21T12:36:58.982429+0000","last_peered":"2024-09-21T12:36:58.982429+0000","last_clean":"2024-09-21T12:36:58.982429+0000","last_became_active":"2024-09-21T12:36:01.130569+0000","last_became_peered":"2024-09-21T12:36:01.130569+0000","last_unstale":"2024-09-21T12:36:58.982429+0000","last_undegraded":"2024-09-21T12:36:58.982429+0000","last_fullsized":"2024-09-21T12:36:58.982429+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'14","last_scrub_stamp":"2024-09-21T12:36:31.838024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:31.838024+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:59.227849+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5,0],"avail_no_missing":["7","0","5"],"object_location_counts":[{"shards":"0,5,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":197,"reported_epoch":436,"state":"active+undersized","last_fresh":"2024-09-21T12:36:58.970392+0000","last_change":"2024-09-21T12:34:34.348303+0000","last_active":"2024-09-21T12:36:58.970392+0000","last_peered":"2024-09-21T12:36:58.970392+0000","last_clean":"2024-09-21T12:34:29.279670+0000","last_became_active":"2024-09-21T12:34:34.348303+0000","last_became_peered":"2024-09-21T12:34:34.348303+0000","last_unstale":"2024-09-21T12:36:58.970392+0000","last_undegraded":"2024-09-21T12:36:58.970392+0000","last_fullsized":"2024-09-21T12:34:34.331636+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:06.369436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:06.369436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:24.813776+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"436'6975","reported_seq":7046,"reported_epoch":436,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:37:01.933187+0000","last_change":"2024-09-21T12:36:01.140837+0000","last_active":"2024-09-21T12:37:01.933187+0000","last_peered":"2024-09-21T12:37:01.933187+0000","last_clean":"2024-09-21T12:37:01.933187+0000","last_became_active":"2024-09-21T12:36:01.132703+0000","last_became_peered":"2024-09-21T12:36:01.132703+0000","last_unstale":"2024-09-21T12:37:01.933187+0000","last_undegraded":"2024-09-21T12:37:01.933187+0000","last_fullsized":"2024-09-21T12:37:01.933187+0000","mapping_epoch":415,"log_start":"422'2700","ondisk_log_start":"422'2700","created":405,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4275,"log_dups_size":0,"ondisk_log_size":4275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457113600,"num_objects":6975,"num_object_clones":0,"num_object_copies":20925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6975,"num_objects_unfound":0,"num_objects_dirty":6975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6975,"num_write_kb":446400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6,1],"avail_no_missing":["7","1","6"],"object_location_counts":[{"shards":"1,6,7","objects":6975}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"76'79","reported_seq":205,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.972442+0000","last_change":"2024-09-21T12:36:01.159899+0000","last_active":"2024-09-21T12:36:58.972442+0000","last_peered":"2024-09-21T12:36:58.972442+0000","last_clean":"2024-09-21T12:36:58.972442+0000","last_became_active":"2024-09-21T12:36:01.159481+0000","last_became_peered":"2024-09-21T12:36:01.159481+0000","last_unstale":"2024-09-21T12:36:58.972442+0000","last_undegraded":"2024-09-21T12:36:58.972442+0000","last_fullsized":"2024-09-21T12:36:58.972442+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:49.155069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:49.155069+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":918,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.977494+0000","last_change":"2024-09-21T12:36:40.771296+0000","last_active":"2024-09-21T12:36:58.977494+0000","last_peered":"2024-09-21T12:36:58.977494+0000","last_clean":"2024-09-21T12:36:58.977494+0000","last_became_active":"2024-09-21T12:32:38.424376+0000","last_became_peered":"2024-09-21T12:32:38.424376+0000","last_unstale":"2024-09-21T12:36:58.977494+0000","last_undegraded":"2024-09-21T12:36:58.977494+0000","last_fullsized":"2024-09-21T12:36:58.977494+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:36:40.771222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:40.771222+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:42.363964+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":486,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.979611+0000","last_change":"2024-09-21T12:34:36.079650+0000","last_active":"2024-09-21T12:36:58.979611+0000","last_peered":"2024-09-21T12:36:58.979611+0000","last_clean":"2024-09-21T12:36:58.979611+0000","last_became_active":"2024-09-21T12:26:54.084228+0000","last_became_peered":"2024-09-21T12:26:54.084228+0000","last_unstale":"2024-09-21T12:36:58.979611+0000","last_undegraded":"2024-09-21T12:36:58.979611+0000","last_fullsized":"2024-09-21T12:36:58.979611+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:36.079583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:34:36.079583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":760,"reported_epoch":436,"state":"active+undersized","last_fresh":"2024-09-21T12:36:58.969196+0000","last_change":"2024-09-21T12:34:34.346822+0000","last_active":"2024-09-21T12:36:58.969196+0000","last_peered":"2024-09-21T12:36:58.969196+0000","last_clean":"2024-09-21T12:34:29.278339+0000","last_became_active":"2024-09-21T12:34:34.346822+0000","last_became_peered":"2024-09-21T12:34:34.346822+0000","last_unstale":"2024-09-21T12:36:58.969196+0000","last_undegraded":"2024-09-21T12:36:58.969196+0000","last_fullsized":"2024-09-21T12:34:34.330172+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:47.361115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:47.361115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:10.874140+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":809,"reported_epoch":436,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:58.975015+0000","last_change":"2024-09-21T12:36:34.842873+0000","last_active":"2024-09-21T12:36:58.975015+0000","last_peered":"2024-09-21T12:36:58.975015+0000","last_clean":"2024-09-21T12:36:58.975015+0000","last_became_active":"2024-09-21T12:36:01.130370+0000","last_became_peered":"2024-09-21T12:36:01.130370+0000","last_unstale":"2024-09-21T12:36:58.975015+0000","last_undegraded":"2024-09-21T12:36:58.975015+0000","last_fullsized":"2024-09-21T12:36:58.975015+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:34.842666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:34.842666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:56.753346+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7,3],"avail_no_missing":["0","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"436'6840","reported_seq":9015,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:37:01.926136+0000","last_change":"2024-09-21T12:36:12.108511+0000","last_active":"2024-09-21T12:37:01.926136+0000","last_peered":"2024-09-21T12:37:01.926136+0000","last_clean":"2024-09-21T12:37:01.926136+0000","last_became_active":"2024-09-21T12:36:12.104630+0000","last_became_peered":"2024-09-21T12:36:12.104630+0000","last_unstale":"2024-09-21T12:37:01.926136+0000","last_undegraded":"2024-09-21T12:37:01.926136+0000","last_fullsized":"2024-09-21T12:37:01.926136+0000","mapping_epoch":419,"log_start":"422'2600","ondisk_log_start":"422'2600","created":405,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4240,"log_dups_size":0,"ondisk_log_size":4240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:06.896900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448266240,"num_objects":6840,"num_object_clones":0,"num_object_copies":20520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6840,"num_write_kb":437760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1047,"num_bytes_recovered":68616192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"76'79","reported_seq":733,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.988362+0000","last_change":"2024-09-21T12:35:57.172169+0000","last_active":"2024-09-21T12:36:58.988362+0000","last_peered":"2024-09-21T12:36:58.988362+0000","last_clean":"2024-09-21T12:36:58.988362+0000","last_became_active":"2024-09-21T12:32:38.414877+0000","last_became_peered":"2024-09-21T12:32:38.414877+0000","last_unstale":"2024-09-21T12:36:58.988362+0000","last_undegraded":"2024-09-21T12:36:58.988362+0000","last_fullsized":"2024-09-21T12:36:58.988362+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:35:57.172103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:35:57.172103+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:06.814669+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":970,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.973729+0000","last_change":"2024-09-21T12:36:00.174641+0000","last_active":"2024-09-21T12:36:58.973729+0000","last_peered":"2024-09-21T12:36:58.973729+0000","last_clean":"2024-09-21T12:36:58.973729+0000","last_became_active":"2024-09-21T12:36:00.148392+0000","last_became_peered":"2024-09-21T12:36:00.148392+0000","last_unstale":"2024-09-21T12:36:58.973729+0000","last_undegraded":"2024-09-21T12:36:58.973729+0000","last_fullsized":"2024-09-21T12:36:58.973729+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'35","last_scrub_stamp":"2024-09-21T12:34:46.956618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:34:46.956618+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:17.045549+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":54,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.982079+0000","last_change":"2024-09-21T12:36:00.145588+0000","last_active":"2024-09-21T12:36:58.982079+0000","last_peered":"2024-09-21T12:36:58.982079+0000","last_clean":"2024-09-21T12:36:58.982079+0000","last_became_active":"2024-09-21T12:36:00.145287+0000","last_became_peered":"2024-09-21T12:36:00.145287+0000","last_unstale":"2024-09-21T12:36:58.982079+0000","last_undegraded":"2024-09-21T12:36:58.982079+0000","last_fullsized":"2024-09-21T12:36:58.982079+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:58.197610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:58.197610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:14.038278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":698,"reported_epoch":436,"state":"active+undersized","last_fresh":"2024-09-21T12:36:58.977764+0000","last_change":"2024-09-21T12:34:34.343106+0000","last_active":"2024-09-21T12:36:58.977764+0000","last_peered":"2024-09-21T12:36:58.977764+0000","last_clean":"2024-09-21T12:34:29.291366+0000","last_became_active":"2024-09-21T12:34:34.343106+0000","last_became_peered":"2024-09-21T12:34:34.343106+0000","last_unstale":"2024-09-21T12:36:58.977764+0000","last_undegraded":"2024-09-21T12:36:58.977764+0000","last_fullsized":"2024-09-21T12:34:34.333287+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:20.243774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:20.243774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:37.780982+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":780,"reported_epoch":436,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:58.981199+0000","last_change":"2024-09-21T12:36:28.804571+0000","last_active":"2024-09-21T12:36:58.981199+0000","last_peered":"2024-09-21T12:36:58.981199+0000","last_clean":"2024-09-21T12:36:58.981199+0000","last_became_active":"2024-09-21T12:36:01.135641+0000","last_became_peered":"2024-09-21T12:36:01.135641+0000","last_unstale":"2024-09-21T12:36:58.981199+0000","last_undegraded":"2024-09-21T12:36:58.981199+0000","last_fullsized":"2024-09-21T12:36:58.981199+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:28.804459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:28.804459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:45.512216+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,6],"avail_no_missing":["7","3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"436'6962","reported_seq":7032,"reported_epoch":436,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:37:03.422871+0000","last_change":"2024-09-21T12:35:53.511573+0000","last_active":"2024-09-21T12:37:03.422871+0000","last_peered":"2024-09-21T12:37:03.422871+0000","last_clean":"2024-09-21T12:35:46.767214+0000","last_became_active":"2024-09-21T12:35:47.793577+0000","last_became_peered":"2024-09-21T12:35:47.793577+0000","last_unstale":"2024-09-21T12:37:03.422871+0000","last_undegraded":"2024-09-21T12:35:53.504929+0000","last_fullsized":"2024-09-21T12:35:47.782699+0000","mapping_epoch":405,"log_start":"421'2323","ondisk_log_start":"421'2323","created":405,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4639,"log_dups_size":0,"ondisk_log_size":4639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456261632,"num_objects":6962,"num_object_clones":0,"num_object_copies":20886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6962,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6962,"num_write_kb":445568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":6962}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"76'79","reported_seq":756,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.979303+0000","last_change":"2024-09-21T12:36:31.563764+0000","last_active":"2024-09-21T12:36:58.979303+0000","last_peered":"2024-09-21T12:36:58.979303+0000","last_clean":"2024-09-21T12:36:58.979303+0000","last_became_active":"2024-09-21T12:27:14.951375+0000","last_became_peered":"2024-09-21T12:27:14.951375+0000","last_unstale":"2024-09-21T12:36:58.979303+0000","last_undegraded":"2024-09-21T12:36:58.979303+0000","last_fullsized":"2024-09-21T12:36:58.979303+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:31.563694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:31.563694+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:35.930543+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":206,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.968959+0000","last_change":"2024-09-21T12:35:12.197153+0000","last_active":"2024-09-21T12:36:58.968959+0000","last_peered":"2024-09-21T12:36:58.968959+0000","last_clean":"2024-09-21T12:36:58.968959+0000","last_became_active":"2024-09-21T12:32:38.421609+0000","last_became_peered":"2024-09-21T12:32:38.421609+0000","last_unstale":"2024-09-21T12:36:58.968959+0000","last_undegraded":"2024-09-21T12:36:58.968959+0000","last_fullsized":"2024-09-21T12:36:58.968959+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T12:35:12.197060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:12.197060+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":54,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.982326+0000","last_change":"2024-09-21T12:36:00.151126+0000","last_active":"2024-09-21T12:36:58.982326+0000","last_peered":"2024-09-21T12:36:58.982326+0000","last_clean":"2024-09-21T12:36:58.982326+0000","last_became_active":"2024-09-21T12:36:00.150832+0000","last_became_peered":"2024-09-21T12:36:00.150832+0000","last_unstale":"2024-09-21T12:36:58.982326+0000","last_undegraded":"2024-09-21T12:36:58.982326+0000","last_fullsized":"2024-09-21T12:36:58.982326+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:19.611537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:19.611537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:54.751976+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":120,"reported_epoch":436,"state":"active+undersized","last_fresh":"2024-09-21T12:36:58.975155+0000","last_change":"2024-09-21T12:34:34.333741+0000","last_active":"2024-09-21T12:36:58.975155+0000","last_peered":"2024-09-21T12:36:58.975155+0000","last_clean":"2024-09-21T12:34:29.289730+0000","last_became_active":"2024-09-21T12:34:34.333741+0000","last_became_peered":"2024-09-21T12:34:34.333741+0000","last_unstale":"2024-09-21T12:36:58.975155+0000","last_undegraded":"2024-09-21T12:36:58.975155+0000","last_fullsized":"2024-09-21T12:34:34.330975+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:23.769748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:32:23.769748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:04.957773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":818,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.981272+0000","last_change":"2024-09-21T12:36:00.139665+0000","last_active":"2024-09-21T12:36:58.981272+0000","last_peered":"2024-09-21T12:36:58.981272+0000","last_clean":"2024-09-21T12:36:58.981272+0000","last_became_active":"2024-09-21T12:36:00.139300+0000","last_became_peered":"2024-09-21T12:36:00.139300+0000","last_unstale":"2024-09-21T12:36:58.981272+0000","last_undegraded":"2024-09-21T12:36:58.981272+0000","last_fullsized":"2024-09-21T12:36:58.981272+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:04.876656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:34:04.876656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"436'7032","reported_seq":7105,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:37:01.624100+0000","last_change":"2024-09-21T12:35:47.849125+0000","last_active":"2024-09-21T12:37:01.624100+0000","last_peered":"2024-09-21T12:37:01.624100+0000","last_clean":"2024-09-21T12:37:01.624100+0000","last_became_active":"2024-09-21T12:35:47.848708+0000","last_became_peered":"2024-09-21T12:35:47.848708+0000","last_unstale":"2024-09-21T12:37:01.624100+0000","last_undegraded":"2024-09-21T12:37:01.624100+0000","last_fullsized":"2024-09-21T12:37:01.624100+0000","mapping_epoch":405,"log_start":"423'3200","ondisk_log_start":"423'3200","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3832,"log_dups_size":0,"ondisk_log_size":3832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:57.634193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":460849152,"num_objects":7032,"num_object_clones":0,"num_object_copies":21096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7032,"num_write_kb":450048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"76'79","reported_seq":234,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.982166+0000","last_change":"2024-09-21T12:36:36.859482+0000","last_active":"2024-09-21T12:36:58.982166+0000","last_peered":"2024-09-21T12:36:58.982166+0000","last_clean":"2024-09-21T12:36:58.982166+0000","last_became_active":"2024-09-21T12:36:01.141807+0000","last_became_peered":"2024-09-21T12:36:01.141807+0000","last_unstale":"2024-09-21T12:36:58.982166+0000","last_undegraded":"2024-09-21T12:36:58.982166+0000","last_fullsized":"2024-09-21T12:36:58.982166+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:36.859412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:36.859412+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:54.672933+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":335,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.984764+0000","last_change":"2024-09-21T12:36:00.171576+0000","last_active":"2024-09-21T12:36:58.984764+0000","last_peered":"2024-09-21T12:36:58.984764+0000","last_clean":"2024-09-21T12:36:58.984764+0000","last_became_active":"2024-09-21T12:36:00.137597+0000","last_became_peered":"2024-09-21T12:36:00.137597+0000","last_unstale":"2024-09-21T12:36:58.984764+0000","last_undegraded":"2024-09-21T12:36:58.984764+0000","last_fullsized":"2024-09-21T12:36:58.984764+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:35:52.226406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:52.226406+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:53.571128+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":60,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.981999+0000","last_change":"2024-09-21T12:36:00.161052+0000","last_active":"2024-09-21T12:36:58.981999+0000","last_peered":"2024-09-21T12:36:58.981999+0000","last_clean":"2024-09-21T12:36:58.981999+0000","last_became_active":"2024-09-21T12:36:00.148899+0000","last_became_peered":"2024-09-21T12:36:00.148899+0000","last_unstale":"2024-09-21T12:36:58.981999+0000","last_undegraded":"2024-09-21T12:36:58.981999+0000","last_fullsized":"2024-09-21T12:36:58.981999+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-21T12:35:51.961868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:51.961868+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"415'16","reported_seq":71,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.981010+0000","last_change":"2024-09-21T12:36:00.165364+0000","last_active":"2024-09-21T12:36:58.981010+0000","last_peered":"2024-09-21T12:36:58.981010+0000","last_clean":"2024-09-21T12:36:58.981010+0000","last_became_active":"2024-09-21T12:36:00.143091+0000","last_became_peered":"2024-09-21T12:36:00.143091+0000","last_unstale":"2024-09-21T12:36:58.981010+0000","last_undegraded":"2024-09-21T12:36:58.981010+0000","last_fullsized":"2024-09-21T12:36:58.981010+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'14","last_scrub_stamp":"2024-09-21T12:35:06.542700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:35:06.542700+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:58.759100+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":502,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.973485+0000","last_change":"2024-09-21T12:36:34.998909+0000","last_active":"2024-09-21T12:36:58.973485+0000","last_peered":"2024-09-21T12:36:58.973485+0000","last_clean":"2024-09-21T12:36:58.973485+0000","last_became_active":"2024-09-21T12:26:54.089155+0000","last_became_peered":"2024-09-21T12:26:54.089155+0000","last_unstale":"2024-09-21T12:36:58.973485+0000","last_undegraded":"2024-09-21T12:36:58.973485+0000","last_fullsized":"2024-09-21T12:36:58.973485+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:34.998795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:34.998795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:47.855846+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"76'79","reported_seq":532,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.985004+0000","last_change":"2024-09-21T12:32:26.698667+0000","last_active":"2024-09-21T12:36:58.985004+0000","last_peered":"2024-09-21T12:36:58.985004+0000","last_clean":"2024-09-21T12:36:58.985004+0000","last_became_active":"2024-09-21T12:31:39.060422+0000","last_became_peered":"2024-09-21T12:31:39.060422+0000","last_unstale":"2024-09-21T12:36:58.985004+0000","last_undegraded":"2024-09-21T12:36:58.985004+0000","last_fullsized":"2024-09-21T12:36:58.985004+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:26.698622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:26.698622+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:01.158707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":849,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.975446+0000","last_change":"2024-09-21T12:35:59.183091+0000","last_active":"2024-09-21T12:36:58.975446+0000","last_peered":"2024-09-21T12:36:58.975446+0000","last_clean":"2024-09-21T12:36:58.975446+0000","last_became_active":"2024-09-21T12:32:02.026693+0000","last_became_peered":"2024-09-21T12:32:02.026693+0000","last_unstale":"2024-09-21T12:36:58.975446+0000","last_undegraded":"2024-09-21T12:36:58.975446+0000","last_fullsized":"2024-09-21T12:36:58.975446+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:59.183047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:59.183047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:03.346089+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":305,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.982141+0000","last_change":"2024-09-21T12:36:01.139003+0000","last_active":"2024-09-21T12:36:58.982141+0000","last_peered":"2024-09-21T12:36:58.982141+0000","last_clean":"2024-09-21T12:36:58.982141+0000","last_became_active":"2024-09-21T12:36:01.138626+0000","last_became_peered":"2024-09-21T12:36:01.138626+0000","last_unstale":"2024-09-21T12:36:58.982141+0000","last_undegraded":"2024-09-21T12:36:58.982141+0000","last_fullsized":"2024-09-21T12:36:58.982141+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T12:34:09.872320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:34:09.872320+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:58.661408+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":495,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.980781+0000","last_change":"2024-09-21T12:36:42.510836+0000","last_active":"2024-09-21T12:36:58.980781+0000","last_peered":"2024-09-21T12:36:58.980781+0000","last_clean":"2024-09-21T12:36:58.980781+0000","last_became_active":"2024-09-21T12:26:54.084109+0000","last_became_peered":"2024-09-21T12:26:54.084109+0000","last_unstale":"2024-09-21T12:36:58.980781+0000","last_undegraded":"2024-09-21T12:36:58.980781+0000","last_fullsized":"2024-09-21T12:36:58.980781+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:42.510764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:42.510764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:51.665123+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":54,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.981779+0000","last_change":"2024-09-21T12:36:00.149496+0000","last_active":"2024-09-21T12:36:58.981779+0000","last_peered":"2024-09-21T12:36:58.981779+0000","last_clean":"2024-09-21T12:36:58.981779+0000","last_became_active":"2024-09-21T12:36:00.149086+0000","last_became_peered":"2024-09-21T12:36:00.149086+0000","last_unstale":"2024-09-21T12:36:58.981779+0000","last_undegraded":"2024-09-21T12:36:58.981779+0000","last_fullsized":"2024-09-21T12:36:58.981779+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:14.566963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:14.566963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'79","reported_seq":780,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.988196+0000","last_change":"2024-09-21T12:36:39.411851+0000","last_active":"2024-09-21T12:36:58.988196+0000","last_peered":"2024-09-21T12:36:58.988196+0000","last_clean":"2024-09-21T12:36:58.988196+0000","last_became_active":"2024-09-21T12:32:38.425495+0000","last_became_peered":"2024-09-21T12:32:38.425495+0000","last_unstale":"2024-09-21T12:36:58.988196+0000","last_undegraded":"2024-09-21T12:36:58.988196+0000","last_fullsized":"2024-09-21T12:36:58.988196+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:39.411785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:39.411785+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:43.123682+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":872,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.979836+0000","last_change":"2024-09-21T12:36:00.142630+0000","last_active":"2024-09-21T12:36:58.979836+0000","last_peered":"2024-09-21T12:36:58.979836+0000","last_clean":"2024-09-21T12:36:58.979836+0000","last_became_active":"2024-09-21T12:36:00.142274+0000","last_became_peered":"2024-09-21T12:36:00.142274+0000","last_unstale":"2024-09-21T12:36:58.979836+0000","last_undegraded":"2024-09-21T12:36:58.979836+0000","last_fullsized":"2024-09-21T12:36:58.979836+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:32:19.786928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:19.786928+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:01.953764+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":54,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.980939+0000","last_change":"2024-09-21T12:36:00.140779+0000","last_active":"2024-09-21T12:36:58.980939+0000","last_peered":"2024-09-21T12:36:58.980939+0000","last_clean":"2024-09-21T12:36:58.980939+0000","last_became_active":"2024-09-21T12:36:00.140477+0000","last_became_peered":"2024-09-21T12:36:00.140477+0000","last_unstale":"2024-09-21T12:36:58.980939+0000","last_undegraded":"2024-09-21T12:36:58.980939+0000","last_fullsized":"2024-09-21T12:36:58.980939+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:27.798236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:27.798236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":799,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.981845+0000","last_change":"2024-09-21T12:36:00.151444+0000","last_active":"2024-09-21T12:36:58.981845+0000","last_peered":"2024-09-21T12:36:58.981845+0000","last_clean":"2024-09-21T12:36:58.981845+0000","last_became_active":"2024-09-21T12:36:00.150975+0000","last_became_peered":"2024-09-21T12:36:00.150975+0000","last_unstale":"2024-09-21T12:36:58.981845+0000","last_undegraded":"2024-09-21T12:36:58.981845+0000","last_fullsized":"2024-09-21T12:36:58.981845+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:58.720834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:32:58.720834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:20.560140+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":776,"reported_epoch":436,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:58.972095+0000","last_change":"2024-09-21T12:36:01.138863+0000","last_active":"2024-09-21T12:36:58.972095+0000","last_peered":"2024-09-21T12:36:58.972095+0000","last_clean":"2024-09-21T12:36:58.972095+0000","last_became_active":"2024-09-21T12:36:01.138205+0000","last_became_peered":"2024-09-21T12:36:01.138205+0000","last_unstale":"2024-09-21T12:36:58.972095+0000","last_undegraded":"2024-09-21T12:36:58.972095+0000","last_fullsized":"2024-09-21T12:36:58.972095+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:55.936030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:55.936030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:09.145667+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7,0],"avail_no_missing":["4","0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.10","version":"76'79","reported_seq":62,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.980994+0000","last_change":"2024-09-21T12:36:00.829177+0000","last_active":"2024-09-21T12:36:58.980994+0000","last_peered":"2024-09-21T12:36:58.980994+0000","last_clean":"2024-09-21T12:36:58.980994+0000","last_became_active":"2024-09-21T12:36:00.140266+0000","last_became_peered":"2024-09-21T12:36:00.140266+0000","last_unstale":"2024-09-21T12:36:58.980994+0000","last_undegraded":"2024-09-21T12:36:58.980994+0000","last_fullsized":"2024-09-21T12:36:58.980994+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:00.829119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:00.829119+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:19.526550+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":554,"reported_epoch":436,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:58.982307+0000","last_change":"2024-09-21T12:36:01.130787+0000","last_active":"2024-09-21T12:36:58.982307+0000","last_peered":"2024-09-21T12:36:58.982307+0000","last_clean":"2024-09-21T12:36:58.982307+0000","last_became_active":"2024-09-21T12:36:01.129843+0000","last_became_peered":"2024-09-21T12:36:01.129843+0000","last_unstale":"2024-09-21T12:36:58.982307+0000","last_undegraded":"2024-09-21T12:36:58.982307+0000","last_fullsized":"2024-09-21T12:36:58.982307+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:35:54.238498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:54.238498+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:14.079285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5,6],"avail_no_missing":["7","5","6"],"object_location_counts":[{"shards":"5,6,7","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":856,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.976455+0000","last_change":"2024-09-21T12:36:00.127553+0000","last_active":"2024-09-21T12:36:58.976455+0000","last_peered":"2024-09-21T12:36:58.976455+0000","last_clean":"2024-09-21T12:36:58.976455+0000","last_became_active":"2024-09-21T12:36:00.127225+0000","last_became_peered":"2024-09-21T12:36:00.127225+0000","last_unstale":"2024-09-21T12:36:58.976455+0000","last_undegraded":"2024-09-21T12:36:58.976455+0000","last_fullsized":"2024-09-21T12:36:58.976455+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:22.225122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:22.225122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"416'15","reported_seq":352,"reported_epoch":436,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:58.978813+0000","last_change":"2024-09-21T12:36:01.139488+0000","last_active":"2024-09-21T12:36:58.978813+0000","last_peered":"2024-09-21T12:36:58.978813+0000","last_clean":"2024-09-21T12:36:58.978813+0000","last_became_active":"2024-09-21T12:36:01.138017+0000","last_became_peered":"2024-09-21T12:36:01.138017+0000","last_unstale":"2024-09-21T12:36:58.978813+0000","last_undegraded":"2024-09-21T12:36:58.978813+0000","last_fullsized":"2024-09-21T12:36:58.978813+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'10","last_scrub_stamp":"2024-09-21T12:34:14.003545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:14.003545+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7,1],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":197,"reported_epoch":436,"state":"active+undersized","last_fresh":"2024-09-21T12:36:58.970291+0000","last_change":"2024-09-21T12:34:34.340859+0000","last_active":"2024-09-21T12:36:58.970291+0000","last_peered":"2024-09-21T12:36:58.970291+0000","last_clean":"2024-09-21T12:34:29.279088+0000","last_became_active":"2024-09-21T12:34:34.340859+0000","last_became_peered":"2024-09-21T12:34:34.340859+0000","last_unstale":"2024-09-21T12:36:58.970291+0000","last_undegraded":"2024-09-21T12:36:58.970291+0000","last_fullsized":"2024-09-21T12:34:34.330912+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:05.370263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:05.370263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:28.761559+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"76'79","reported_seq":623,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.976152+0000","last_change":"2024-09-21T12:35:17.231922+0000","last_active":"2024-09-21T12:36:58.976152+0000","last_peered":"2024-09-21T12:36:58.976152+0000","last_clean":"2024-09-21T12:36:58.976152+0000","last_became_active":"2024-09-21T12:32:02.034387+0000","last_became_peered":"2024-09-21T12:32:02.034387+0000","last_unstale":"2024-09-21T12:36:58.976152+0000","last_undegraded":"2024-09-21T12:36:58.976152+0000","last_fullsized":"2024-09-21T12:36:58.976152+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:35:17.231860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:35:17.231860+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":887,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.979459+0000","last_change":"2024-09-21T12:36:36.521884+0000","last_active":"2024-09-21T12:36:58.979459+0000","last_peered":"2024-09-21T12:36:58.979459+0000","last_clean":"2024-09-21T12:36:58.979459+0000","last_became_active":"2024-09-21T12:36:00.155033+0000","last_became_peered":"2024-09-21T12:36:00.155033+0000","last_unstale":"2024-09-21T12:36:58.979459+0000","last_undegraded":"2024-09-21T12:36:58.979459+0000","last_fullsized":"2024-09-21T12:36:58.979459+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:36:36.521810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:36.521810+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:05.336949+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":820,"reported_epoch":436,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:58.981625+0000","last_change":"2024-09-21T12:36:01.139279+0000","last_active":"2024-09-21T12:36:58.981625+0000","last_peered":"2024-09-21T12:36:58.981625+0000","last_clean":"2024-09-21T12:36:58.981625+0000","last_became_active":"2024-09-21T12:36:01.135527+0000","last_became_peered":"2024-09-21T12:36:01.135527+0000","last_unstale":"2024-09-21T12:36:58.981625+0000","last_undegraded":"2024-09-21T12:36:58.981625+0000","last_fullsized":"2024-09-21T12:36:58.981625+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:32.630261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:32.630261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:17.760444+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4,6],"avail_no_missing":["7","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":1012,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.986533+0000","last_change":"2024-09-21T12:36:29.482398+0000","last_active":"2024-09-21T12:36:58.986533+0000","last_peered":"2024-09-21T12:36:58.986533+0000","last_clean":"2024-09-21T12:36:58.986533+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:36:58.986533+0000","last_undegraded":"2024-09-21T12:36:58.986533+0000","last_fullsized":"2024-09-21T12:36:58.986533+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:36:29.482171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:29.482171+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:37.228329+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":812,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.979182+0000","last_change":"2024-09-21T12:36:00.149113+0000","last_active":"2024-09-21T12:36:58.979182+0000","last_peered":"2024-09-21T12:36:58.979182+0000","last_clean":"2024-09-21T12:36:58.979182+0000","last_became_active":"2024-09-21T12:36:00.148332+0000","last_became_peered":"2024-09-21T12:36:00.148332+0000","last_unstale":"2024-09-21T12:36:58.979182+0000","last_undegraded":"2024-09-21T12:36:58.979182+0000","last_fullsized":"2024-09-21T12:36:58.979182+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:26.208487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:26.208487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:56.731364+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"76'79","reported_seq":480,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.986408+0000","last_change":"2024-09-21T12:36:28.244064+0000","last_active":"2024-09-21T12:36:58.986408+0000","last_peered":"2024-09-21T12:36:58.986408+0000","last_clean":"2024-09-21T12:36:58.986408+0000","last_became_active":"2024-09-21T12:27:16.940347+0000","last_became_peered":"2024-09-21T12:27:16.940347+0000","last_unstale":"2024-09-21T12:36:58.986408+0000","last_undegraded":"2024-09-21T12:36:58.986408+0000","last_fullsized":"2024-09-21T12:36:58.986408+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:28.243938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:28.243938+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:53.995371+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":205,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.969208+0000","last_change":"2024-09-21T12:34:00.191190+0000","last_active":"2024-09-21T12:36:58.969208+0000","last_peered":"2024-09-21T12:36:58.969208+0000","last_clean":"2024-09-21T12:36:58.969208+0000","last_became_active":"2024-09-21T12:32:38.421008+0000","last_became_peered":"2024-09-21T12:32:38.421008+0000","last_unstale":"2024-09-21T12:36:58.969208+0000","last_undegraded":"2024-09-21T12:36:58.969208+0000","last_fullsized":"2024-09-21T12:36:58.969208+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:34:00.191106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:34:00.191106+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":205,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.970608+0000","last_change":"2024-09-21T12:36:01.124070+0000","last_active":"2024-09-21T12:36:58.970608+0000","last_peered":"2024-09-21T12:36:58.970608+0000","last_clean":"2024-09-21T12:36:58.970608+0000","last_became_active":"2024-09-21T12:36:01.123399+0000","last_became_peered":"2024-09-21T12:36:01.123399+0000","last_unstale":"2024-09-21T12:36:58.970608+0000","last_undegraded":"2024-09-21T12:36:58.970608+0000","last_fullsized":"2024-09-21T12:36:58.970608+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:30.406915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:30.406915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:49.594071+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":812,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.987072+0000","last_change":"2024-09-21T12:36:34.472774+0000","last_active":"2024-09-21T12:36:58.987072+0000","last_peered":"2024-09-21T12:36:58.987072+0000","last_clean":"2024-09-21T12:36:58.987072+0000","last_became_active":"2024-09-21T12:31:50.040754+0000","last_became_peered":"2024-09-21T12:31:50.040754+0000","last_unstale":"2024-09-21T12:36:58.987072+0000","last_undegraded":"2024-09-21T12:36:58.987072+0000","last_fullsized":"2024-09-21T12:36:58.987072+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:34.472702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:34.472702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:47.750215+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":262,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.976851+0000","last_change":"2024-09-21T12:31:53.359206+0000","last_active":"2024-09-21T12:36:58.976851+0000","last_peered":"2024-09-21T12:36:58.976851+0000","last_clean":"2024-09-21T12:36:58.976851+0000","last_became_active":"2024-09-21T12:31:50.036150+0000","last_became_peered":"2024-09-21T12:31:50.036150+0000","last_unstale":"2024-09-21T12:36:58.976851+0000","last_undegraded":"2024-09-21T12:36:58.976851+0000","last_fullsized":"2024-09-21T12:36:58.976851+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:53.359159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:31:53.359159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:17.628056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"76'79","reported_seq":513,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.988461+0000","last_change":"2024-09-21T12:36:45.476276+0000","last_active":"2024-09-21T12:36:58.988461+0000","last_peered":"2024-09-21T12:36:58.988461+0000","last_clean":"2024-09-21T12:36:58.988461+0000","last_became_active":"2024-09-21T12:27:16.921880+0000","last_became_peered":"2024-09-21T12:27:16.921880+0000","last_unstale":"2024-09-21T12:36:58.988461+0000","last_undegraded":"2024-09-21T12:36:58.988461+0000","last_fullsized":"2024-09-21T12:36:58.988461+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:45.476087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:45.476087+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:11.672214+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":292,"reported_epoch":436,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:36:58.986667+0000","last_change":"2024-09-21T12:34:34.345279+0000","last_active":"2024-09-21T12:36:58.986667+0000","last_peered":"2024-09-21T12:36:58.986667+0000","last_clean":"2024-09-21T12:34:29.286797+0000","last_became_active":"2024-09-21T12:34:34.345279+0000","last_became_peered":"2024-09-21T12:34:34.345279+0000","last_unstale":"2024-09-21T12:36:58.986667+0000","last_undegraded":"2024-09-21T12:34:34.331120+0000","last_fullsized":"2024-09-21T12:34:34.331029+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:33:02.544885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:02.544885+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:46.518783+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":54,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.982205+0000","last_change":"2024-09-21T12:36:00.147458+0000","last_active":"2024-09-21T12:36:58.982205+0000","last_peered":"2024-09-21T12:36:58.982205+0000","last_clean":"2024-09-21T12:36:58.982205+0000","last_became_active":"2024-09-21T12:36:00.147063+0000","last_became_peered":"2024-09-21T12:36:00.147063+0000","last_unstale":"2024-09-21T12:36:58.982205+0000","last_undegraded":"2024-09-21T12:36:58.982205+0000","last_fullsized":"2024-09-21T12:36:58.982205+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:28.849200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:28.849200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":794,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.981363+0000","last_change":"2024-09-21T12:34:11.934886+0000","last_active":"2024-09-21T12:36:58.981363+0000","last_peered":"2024-09-21T12:36:58.981363+0000","last_clean":"2024-09-21T12:36:58.981363+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:36:58.981363+0000","last_undegraded":"2024-09-21T12:36:58.981363+0000","last_fullsized":"2024-09-21T12:36:58.981363+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:11.934826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:11.934826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":785,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.975769+0000","last_change":"2024-09-21T12:35:59.959597+0000","last_active":"2024-09-21T12:36:58.975769+0000","last_peered":"2024-09-21T12:36:58.975769+0000","last_clean":"2024-09-21T12:36:58.975769+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:36:58.975769+0000","last_undegraded":"2024-09-21T12:36:58.975769+0000","last_fullsized":"2024-09-21T12:36:58.975769+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:59.959545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:59.959545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:04.514896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"76'79","reported_seq":289,"reported_epoch":436,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:58.981900+0000","last_change":"2024-09-21T12:36:43.792972+0000","last_active":"2024-09-21T12:36:58.981900+0000","last_peered":"2024-09-21T12:36:58.981900+0000","last_clean":"2024-09-21T12:36:58.981900+0000","last_became_active":"2024-09-21T12:36:01.135246+0000","last_became_peered":"2024-09-21T12:36:01.135246+0000","last_unstale":"2024-09-21T12:36:58.981900+0000","last_undegraded":"2024-09-21T12:36:58.981900+0000","last_fullsized":"2024-09-21T12:36:58.981900+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:43.792355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:43.792355+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:48.500764+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4,1],"avail_no_missing":["7","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":875,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.977568+0000","last_change":"2024-09-21T12:36:41.506309+0000","last_active":"2024-09-21T12:36:58.977568+0000","last_peered":"2024-09-21T12:36:58.977568+0000","last_clean":"2024-09-21T12:36:58.977568+0000","last_became_active":"2024-09-21T12:36:00.139682+0000","last_became_peered":"2024-09-21T12:36:00.139682+0000","last_unstale":"2024-09-21T12:36:58.977568+0000","last_undegraded":"2024-09-21T12:36:58.977568+0000","last_fullsized":"2024-09-21T12:36:58.977568+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:36:41.506231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:41.506231+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:10.569673+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":840,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.985506+0000","last_change":"2024-09-21T12:36:34.526618+0000","last_active":"2024-09-21T12:36:58.985506+0000","last_peered":"2024-09-21T12:36:58.985506+0000","last_clean":"2024-09-21T12:36:58.985506+0000","last_became_active":"2024-09-21T12:36:00.151584+0000","last_became_peered":"2024-09-21T12:36:00.151584+0000","last_unstale":"2024-09-21T12:36:58.985506+0000","last_undegraded":"2024-09-21T12:36:58.985506+0000","last_fullsized":"2024-09-21T12:36:58.985506+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:34.526551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:36:34.526551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:39.862324+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":54,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.980836+0000","last_change":"2024-09-21T12:36:00.145015+0000","last_active":"2024-09-21T12:36:58.980836+0000","last_peered":"2024-09-21T12:36:58.980836+0000","last_clean":"2024-09-21T12:36:58.980836+0000","last_became_active":"2024-09-21T12:36:00.144521+0000","last_became_peered":"2024-09-21T12:36:00.144521+0000","last_unstale":"2024-09-21T12:36:58.980836+0000","last_undegraded":"2024-09-21T12:36:58.980836+0000","last_fullsized":"2024-09-21T12:36:58.980836+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:47.316065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:47.316065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:48.738968+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":790,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.981555+0000","last_change":"2024-09-21T12:35:51.628370+0000","last_active":"2024-09-21T12:36:58.981555+0000","last_peered":"2024-09-21T12:36:58.981555+0000","last_clean":"2024-09-21T12:36:58.981555+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:36:58.981555+0000","last_undegraded":"2024-09-21T12:36:58.981555+0000","last_fullsized":"2024-09-21T12:36:58.981555+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:51.628326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:51.628326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:05.095925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"76'79","reported_seq":54,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.981764+0000","last_change":"2024-09-21T12:36:00.151654+0000","last_active":"2024-09-21T12:36:58.981764+0000","last_peered":"2024-09-21T12:36:58.981764+0000","last_clean":"2024-09-21T12:36:58.981764+0000","last_became_active":"2024-09-21T12:36:00.151043+0000","last_became_peered":"2024-09-21T12:36:00.151043+0000","last_unstale":"2024-09-21T12:36:58.981764+0000","last_undegraded":"2024-09-21T12:36:58.981764+0000","last_fullsized":"2024-09-21T12:36:58.981764+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:35:51.976777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:35:51.976777+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:17.768008+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":810,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.982931+0000","last_change":"2024-09-21T12:35:14.789148+0000","last_active":"2024-09-21T12:36:58.982931+0000","last_peered":"2024-09-21T12:36:58.982931+0000","last_clean":"2024-09-21T12:36:58.982931+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:36:58.982931+0000","last_undegraded":"2024-09-21T12:36:58.982931+0000","last_fullsized":"2024-09-21T12:36:58.982931+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'8","last_scrub_stamp":"2024-09-21T12:35:14.789070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:14.789070+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":818,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.978613+0000","last_change":"2024-09-21T12:35:27.237430+0000","last_active":"2024-09-21T12:36:58.978613+0000","last_peered":"2024-09-21T12:36:58.978613+0000","last_clean":"2024-09-21T12:36:58.978613+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:36:58.978613+0000","last_undegraded":"2024-09-21T12:36:58.978613+0000","last_fullsized":"2024-09-21T12:36:58.978613+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:35:27.236980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:27.236980+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":837,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.973655+0000","last_change":"2024-09-21T12:36:00.135914+0000","last_active":"2024-09-21T12:36:58.973655+0000","last_peered":"2024-09-21T12:36:58.973655+0000","last_clean":"2024-09-21T12:36:58.973655+0000","last_became_active":"2024-09-21T12:36:00.135537+0000","last_became_peered":"2024-09-21T12:36:00.135537+0000","last_unstale":"2024-09-21T12:36:58.973655+0000","last_undegraded":"2024-09-21T12:36:58.973655+0000","last_fullsized":"2024-09-21T12:36:58.973655+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:40.861493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:40.861493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:21.056349+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":816,"reported_epoch":436,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:58.981244+0000","last_change":"2024-09-21T12:36:38.838788+0000","last_active":"2024-09-21T12:36:58.981244+0000","last_peered":"2024-09-21T12:36:58.981244+0000","last_clean":"2024-09-21T12:36:58.981244+0000","last_became_active":"2024-09-21T12:36:01.134484+0000","last_became_peered":"2024-09-21T12:36:01.134484+0000","last_unstale":"2024-09-21T12:36:58.981244+0000","last_undegraded":"2024-09-21T12:36:58.981244+0000","last_fullsized":"2024-09-21T12:36:58.981244+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:38.838688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:38.838688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:55.540359+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4,3],"avail_no_missing":["7","3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"76'79","reported_seq":207,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.969100+0000","last_change":"2024-09-21T12:36:00.137642+0000","last_active":"2024-09-21T12:36:58.969100+0000","last_peered":"2024-09-21T12:36:58.969100+0000","last_clean":"2024-09-21T12:36:58.969100+0000","last_became_active":"2024-09-21T12:36:00.137290+0000","last_became_peered":"2024-09-21T12:36:00.137290+0000","last_unstale":"2024-09-21T12:36:58.969100+0000","last_undegraded":"2024-09-21T12:36:58.969100+0000","last_fullsized":"2024-09-21T12:36:58.969100+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:43.304606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:43.304606+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":818,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.987363+0000","last_change":"2024-09-21T12:35:48.922444+0000","last_active":"2024-09-21T12:36:58.987363+0000","last_peered":"2024-09-21T12:36:58.987363+0000","last_clean":"2024-09-21T12:36:58.987363+0000","last_became_active":"2024-09-21T12:32:38.426822+0000","last_became_peered":"2024-09-21T12:32:38.426822+0000","last_unstale":"2024-09-21T12:36:58.987363+0000","last_undegraded":"2024-09-21T12:36:58.987363+0000","last_fullsized":"2024-09-21T12:36:58.987363+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:48.922389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:48.922389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":815,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.987160+0000","last_change":"2024-09-21T12:35:30.730640+0000","last_active":"2024-09-21T12:36:58.987160+0000","last_peered":"2024-09-21T12:36:58.987160+0000","last_clean":"2024-09-21T12:36:58.987160+0000","last_became_active":"2024-09-21T12:31:39.052486+0000","last_became_peered":"2024-09-21T12:31:39.052486+0000","last_unstale":"2024-09-21T12:36:58.987160+0000","last_undegraded":"2024-09-21T12:36:58.987160+0000","last_fullsized":"2024-09-21T12:36:58.987160+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:30.730574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:30.730574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":831,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-21T12:36:58.973979+0000","last_change":"2024-09-21T12:34:44.919918+0000","last_active":"2024-09-21T12:36:58.973979+0000","last_peered":"2024-09-21T12:36:58.973979+0000","last_clean":"2024-09-21T12:36:58.973979+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:36:58.973979+0000","last_undegraded":"2024-09-21T12:36:58.973979+0000","last_fullsized":"2024-09-21T12:36:58.973979+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:34:44.919796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:34:44.919796+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:24.048666+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":838,"reported_epoch":436,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:36:58.979258+0000","last_change":"2024-09-21T12:36:28.911452+0000","last_active":"2024-09-21T12:36:58.979258+0000","last_peered":"2024-09-21T12:36:58.979258+0000","last_clean":"2024-09-21T12:36:58.979258+0000","last_became_active":"2024-09-21T12:36:01.130870+0000","last_became_peered":"2024-09-21T12:36:01.130870+0000","last_unstale":"2024-09-21T12:36:58.979258+0000","last_undegraded":"2024-09-21T12:36:58.979258+0000","last_fullsized":"2024-09-21T12:36:58.979258+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:28.911328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:28.911328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:34.674874+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7,1],"avail_no_missing":["0","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":7305756672,"num_objects":111477,"num_object_clones":0,"num_object_copies":334431,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27832,"num_objects_misplaced":13992,"num_objects_unfound":0,"num_objects_dirty":111477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":111477,"num_write_kb":7134528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4193,"num_bytes_recovered":274792448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20107952128,"data_stored":20107952128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64854,"ondisk_log_size":64854,"up":42,"acting":44,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":83,"acting":90,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106,"ondisk_log_size":106,"up":87,"acting":92,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":54,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1105,"num_read_kb":862,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":472,"num_bytes_recovered":9372,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":11216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":687,"ondisk_log_size":687,"up":87,"acting":88,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":598,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":88,"acting":90,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":68,"num_read_kb":63,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":617202,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":9,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":289522,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2048000,"data_stored":2033792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2449,"ondisk_log_size":2449,"up":86,"acting":89,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234189,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3389160,"kb_used_data":2658264,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":90355992,"statfs":{"total":95995035648,"available":92524535808,"internally_reserved":0,"allocated":2722062336,"data_stored":2721518617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":748419530},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463125,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4239640,"kb_used_data":3490696,"kb_used_omap":1,"kb_used_meta":748926,"kb_avail":89505512,"statfs":{"total":95995035648,"available":91653644288,"internally_reserved":0,"allocated":3574472704,"data_stored":3573944286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":766900678},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724760,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2463792,"kb_used_data":1774556,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":91281360,"statfs":{"total":95995035648,"available":93472112640,"internally_reserved":0,"allocated":1817145344,"data_stored":1816598465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953696,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2133932,"kb_used_data":1782044,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":91611220,"statfs":{"total":95995035648,"available":93809889280,"internally_reserved":0,"allocated":1824813056,"data_stored":1824264217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149926,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4014588,"kb_used_data":3167084,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":89730564,"statfs":{"total":95995035648,"available":91884097536,"internally_reserved":0,"allocated":3243094016,"data_stored":3242530801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":867826119},"hb_peers":[0,1,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411534,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2808848,"kb_used_data":2272320,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":90936304,"statfs":{"total":95995035648,"available":93118775296,"internally_reserved":0,"allocated":2326855680,"data_stored":2326574609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575090,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4810644,"kb_used_data":3940924,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":88934508,"statfs":{"total":95995035648,"available":91068936192,"internally_reserved":0,"allocated":4035506176,"data_stored":4034925976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":890567109},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738616,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3544432,"kb_used_data":2678880,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":90200720,"statfs":{"total":95995035648,"available":92365537280,"internally_reserved":0,"allocated":2743173120,"data_stored":2742621380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":886307269},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2765160448,"data_stored":2765160448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4092133376,"data_stored":4092133376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3246456832,"data_stored":3246456832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1831534592,"data_stored":1831534592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1824522240,"data_stored":1824522240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3618897920,"data_stored":3618897920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2729246720,"data_stored":2729246720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:37:07.899 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pool set unique_pool_4 pgp_num 16 2024-09-21T12:37:07.904 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:07.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:07.939 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:07.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:07.972 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:37:07 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:37:08.040 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:08.079 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:08.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:08.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:08.180 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:08.198 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:08 smithi114 ceph-mon[30324]: from='client.25831 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:37:08.199 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:08 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 27887/335028 objects degraded (8.324%), 13 pgs degraded, 33 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:08.202 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:08 smithi114 ceph-mon[40867]: from='client.25831 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:37:08.202 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:08 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 27887/335028 objects degraded (8.324%), 13 pgs degraded, 33 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:08.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:08 smithi183 ceph-mon[33277]: from='client.25831 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:37:08.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:08 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 27887/335028 objects degraded (8.324%), 13 pgs degraded, 33 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:08.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:08.241 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:08.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:08.342 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:08.378 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:08.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:08.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:08.480 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:08.512 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 75 15 120207 120192 100.143 92.4375 0.00463599 0.00998173 2024-09-21T12:37:08.513 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:08.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:08.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:08.615 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:08.644 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:08.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:08.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:08.746 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:08.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:08.781 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:08.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:08.882 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:08.915 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:08.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:08.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:09.017 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:09.030 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T12:37:09.031 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T12:37:09.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:09.054 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:09.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:09.155 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:09.187 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:09.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:09.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:09.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:09 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:37:09.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:09 smithi183 ceph-mon[33277]: pgmap v858: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.0 KiB/s rd, 93 MiB/s wr, 1.49k op/s; 29096/354519 objects degraded (8.207%); 14845/354519 objects misplaced (4.187%) 2024-09-21T12:37:09.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:09 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:37:09.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:09 smithi114 ceph-mon[40867]: pgmap v858: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.0 KiB/s rd, 93 MiB/s wr, 1.49k op/s; 29096/354519 objects degraded (8.207%); 14845/354519 objects misplaced (4.187%) 2024-09-21T12:37:09.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:09 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:37:09.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:09 smithi114 ceph-mon[30324]: pgmap v858: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.0 KiB/s rd, 93 MiB/s wr, 1.49k op/s; 29096/354519 objects degraded (8.207%); 14845/354519 objects misplaced (4.187%) 2024-09-21T12:37:09.288 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:09.337 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:37:09.342 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:09.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:09.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:09.444 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:09.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:09.483 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:09.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:09.517 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 76 16 121742 121726 100.086 95.875 0.00954835 0.00998722 2024-09-21T12:37:09.584 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:09.614 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:09.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:09.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:09.716 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:09.737 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:09.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:09.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:09.839 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:09.860 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:09.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:09.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:09.962 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:10.001 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:10.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:10.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:10.102 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:10.132 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-21T12:37:10.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:10.147 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:10.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:10.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:10 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1314345743' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:37:10.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:10 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:37:10.249 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:10.283 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:10.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:10.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:10.386 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:10.418 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:10.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:10.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:10.513 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 77 16 123041 123025 99.8407 81.1875 0.011869 0.0100119 2024-09-21T12:37:10.519 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:10 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1314345743' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:37:10.519 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:10 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:37:10.519 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:10 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1314345743' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:37:10.519 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:10 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:37:10.520 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:10.550 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:10.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:10.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:10.651 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:10.694 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:10.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:10.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:10.796 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:10.821 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:10.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:10.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:10.923 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:10.964 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:10.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:10.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:11.031 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:37:11.065 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:11.102 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:11.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:11.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:11.149 INFO:teuthology.orchestra.run.smithi114.stderr:set pool 11 pgp_num to 16 2024-09-21T12:37:11.189 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:11 smithi183 ceph-mon[33277]: pgmap v859: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 938 B/s rd, 85 MiB/s wr, 1.35k op/s; 29551/355884 objects degraded (8.304%); 14845/355884 objects misplaced (4.171%) 2024-09-21T12:37:11.189 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:11 smithi183 ceph-mon[33277]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:37:11.189 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:11 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:37:11.189 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:11 smithi183 ceph-mon[33277]: osdmap e437: 8 total, 7 up, 8 in 2024-09-21T12:37:11.190 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:11 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3855129281' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T12:37:11.190 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:11 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T12:37:11.190 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:11 smithi183 ceph-mon[33277]: 11.f scrub starts 2024-09-21T12:37:11.204 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:11.227 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:11.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:11.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:11.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:11 smithi114 ceph-mon[30324]: pgmap v859: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 938 B/s rd, 85 MiB/s wr, 1.35k op/s; 29551/355884 objects degraded (8.304%); 14845/355884 objects misplaced (4.171%) 2024-09-21T12:37:11.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:11 smithi114 ceph-mon[30324]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:37:11.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:11 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:37:11.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:11 smithi114 ceph-mon[30324]: osdmap e437: 8 total, 7 up, 8 in 2024-09-21T12:37:11.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:11 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3855129281' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T12:37:11.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:11 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T12:37:11.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:11 smithi114 ceph-mon[30324]: 11.f scrub starts 2024-09-21T12:37:11.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:11 smithi114 ceph-mon[40867]: pgmap v859: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 938 B/s rd, 85 MiB/s wr, 1.35k op/s; 29551/355884 objects degraded (8.304%); 14845/355884 objects misplaced (4.171%) 2024-09-21T12:37:11.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:11 smithi114 ceph-mon[40867]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:37:11.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:11 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:37:11.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:11 smithi114 ceph-mon[40867]: osdmap e437: 8 total, 7 up, 8 in 2024-09-21T12:37:11.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:11 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3855129281' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T12:37:11.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:11 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T12:37:11.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:11 smithi114 ceph-mon[40867]: 11.f scrub starts 2024-09-21T12:37:11.329 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:11.366 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:11.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:11.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:11.468 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:11.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:11.502 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:11.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:11.513 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 78 16 124263 124247 99.5398 76.375 0.0131754 0.0100423 2024-09-21T12:37:11.603 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:11.623 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:11.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:11.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:11.725 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:11.755 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:37:11.755 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-21T12:37:11.755 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-21T12:37:11.756 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:37:11.756 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.756 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:4736 10.1 10:8e69640a:::benchmark_data_smithi114_161690_object4735:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e352)", 2024-09-21T12:37:11.756 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:32:40.398397+0000", 2024-09-21T12:37:11.756 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 270.67780482500001, 2024-09-21T12:37:11.756 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0010828510000001, 2024-09-21T12:37:11.756 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:11.756 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:11.756 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-09-21T12:37:11.756 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:11.756 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:37:11.756 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:37:11.756 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 4736 2024-09-21T12:37:11.756 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.757 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:11.757 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.757 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:11.757 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398397+0000", 2024-09-21T12:37:11.757 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.757 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.757 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.757 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:11.757 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398397+0000", 2024-09-21T12:37:11.757 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.757 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.757 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.757 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:11.757 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398399+0000", 2024-09-21T12:37:11.758 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.548e-06 2024-09-21T12:37:11.758 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.758 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.758 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:11.758 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398520+0000", 2024-09-21T12:37:11.758 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012060799999999999 2024-09-21T12:37:11.758 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.758 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.758 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:11.758 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398532+0000", 2024-09-21T12:37:11.758 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2405999999999999e-05 2024-09-21T12:37:11.758 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.759 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.759 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:11.759 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398537+0000", 2024-09-21T12:37:11.759 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9269999999999996e-06 2024-09-21T12:37:11.759 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.759 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.759 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.759 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398583+0000", 2024-09-21T12:37:11.759 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6199999999999998e-05 2024-09-21T12:37:11.759 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.759 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.759 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:11.759 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398590+0000", 2024-09-21T12:37:11.759 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3459999999999996e-06 2024-09-21T12:37:11.759 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.760 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.760 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.760 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.395001+0000", 2024-09-21T12:37:11.760 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.99641102500000001 2024-09-21T12:37:11.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.760 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.760 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for missing object", 2024-09-21T12:37:11.760 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.395140+0000", 2024-09-21T12:37:11.760 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013893299999999999 2024-09-21T12:37:11.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.760 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.760 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.760 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.399384+0000", 2024-09-21T12:37:11.760 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0042432750000000003 2024-09-21T12:37:11.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.761 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.761 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:11.761 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.399480+0000", 2024-09-21T12:37:11.761 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6582999999999993e-05 2024-09-21T12:37:11.761 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.761 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:11.761 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.761 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.761 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.761 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:4740 10.1 10:87045276:::benchmark_data_smithi114_161690_object4739:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e352)", 2024-09-21T12:37:11.761 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:32:40.398550+0000", 2024-09-21T12:37:11.761 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 270.67765249299998, 2024-09-21T12:37:11.761 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.010872301, 2024-09-21T12:37:11.761 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:11.762 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:11.762 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-09-21T12:37:11.762 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:11.762 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:37:11.762 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:37:11.762 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 4740 2024-09-21T12:37:11.762 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.762 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:11.762 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.762 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:11.762 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398550+0000", 2024-09-21T12:37:11.762 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.762 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.762 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.763 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:11.763 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398550+0000", 2024-09-21T12:37:11.763 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.763 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.763 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.763 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:11.763 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398550+0000", 2024-09-21T12:37:11.763 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1399999999999997e-07 2024-09-21T12:37:11.763 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.763 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.763 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:11.763 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398589+0000", 2024-09-21T12:37:11.763 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8723000000000002e-05 2024-09-21T12:37:11.763 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.763 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.763 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:11.764 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398595+0000", 2024-09-21T12:37:11.764 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4670000000000001e-06 2024-09-21T12:37:11.764 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.764 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.764 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:11.764 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398599+0000", 2024-09-21T12:37:11.764 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8940000000000003e-06 2024-09-21T12:37:11.764 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.764 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.764 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.764 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398654+0000", 2024-09-21T12:37:11.764 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4842000000000001e-05 2024-09-21T12:37:11.764 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.764 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.765 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:11.765 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398660+0000", 2024-09-21T12:37:11.765 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2469999999999997e-06 2024-09-21T12:37:11.765 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.765 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.765 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.765 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.395377+0000", 2024-09-21T12:37:11.765 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.99671631000000005 2024-09-21T12:37:11.765 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.765 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.765 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for missing object", 2024-09-21T12:37:11.765 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.395480+0000", 2024-09-21T12:37:11.765 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000103106 2024-09-21T12:37:11.765 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.765 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.765 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.766 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.403495+0000", 2024-09-21T12:37:11.766 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0080156570000000007 2024-09-21T12:37:11.766 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.766 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.766 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for degraded object", 2024-09-21T12:37:11.766 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.403561+0000", 2024-09-21T12:37:11.766 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5333999999999995e-05 2024-09-21T12:37:11.766 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.766 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.766 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.766 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.409316+0000", 2024-09-21T12:37:11.766 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.005755273 2024-09-21T12:37:11.766 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.766 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.767 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:11.767 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.409422+0000", 2024-09-21T12:37:11.767 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010603399999999999 2024-09-21T12:37:11.767 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.767 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:11.767 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.767 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.767 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.767 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:4747 10.1 10:8672b373:::benchmark_data_smithi114_161690_object4746:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e352)", 2024-09-21T12:37:11.767 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:32:40.398608+0000", 2024-09-21T12:37:11.767 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 270.67759462499998, 2024-09-21T12:37:11.767 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.012158095, 2024-09-21T12:37:11.767 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:11.767 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:11.767 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-09-21T12:37:11.767 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:11.768 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:37:11.768 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:37:11.768 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 4747 2024-09-21T12:37:11.768 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.768 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:11.768 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.768 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:11.768 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398608+0000", 2024-09-21T12:37:11.768 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.768 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.768 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.768 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:11.768 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398608+0000", 2024-09-21T12:37:11.769 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.769 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.769 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.769 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:11.769 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398608+0000", 2024-09-21T12:37:11.769 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1899999999999998e-07 2024-09-21T12:37:11.769 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.769 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.769 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:11.769 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398666+0000", 2024-09-21T12:37:11.769 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8527999999999999e-05 2024-09-21T12:37:11.769 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.769 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.769 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:11.770 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398673+0000", 2024-09-21T12:37:11.770 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1020000000000002e-06 2024-09-21T12:37:11.770 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.770 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.770 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:11.770 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398677+0000", 2024-09-21T12:37:11.770 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1099999999999996e-06 2024-09-21T12:37:11.770 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.770 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.770 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.770 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398703+0000", 2024-09-21T12:37:11.770 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6556000000000001e-05 2024-09-21T12:37:11.770 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.770 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.770 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:11.771 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398707+0000", 2024-09-21T12:37:11.771 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7589999999999999e-06 2024-09-21T12:37:11.771 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.771 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.771 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.771 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.395516+0000", 2024-09-21T12:37:11.771 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.99680916900000005 2024-09-21T12:37:11.771 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.771 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.771 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for missing object", 2024-09-21T12:37:11.771 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.395627+0000", 2024-09-21T12:37:11.771 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000110518 2024-09-21T12:37:11.771 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.772 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.772 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.772 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.404280+0000", 2024-09-21T12:37:11.772 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0086530200000000008 2024-09-21T12:37:11.772 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.772 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.772 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for degraded object", 2024-09-21T12:37:11.772 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.404347+0000", 2024-09-21T12:37:11.772 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6889000000000003e-05 2024-09-21T12:37:11.772 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.772 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.772 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.773 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.410689+0000", 2024-09-21T12:37:11.773 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0063425820000000003 2024-09-21T12:37:11.773 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.773 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.773 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:11.773 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.410766+0000", 2024-09-21T12:37:11.773 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6543000000000002e-05 2024-09-21T12:37:11.773 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.773 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:11.773 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.773 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.773 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.773 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:4753 10.1 10:8cf8d9e3:::benchmark_data_smithi114_161690_object4752:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e352)", 2024-09-21T12:37:11.773 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:32:40.398685+0000", 2024-09-21T12:37:11.773 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 270.677517599, 2024-09-21T12:37:11.774 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0059854239999999, 2024-09-21T12:37:11.774 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:11.774 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:11.774 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:11.774 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:11.774 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:37:11.774 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:37:11.774 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 4753 2024-09-21T12:37:11.774 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.774 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:11.774 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.774 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:11.774 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398685+0000", 2024-09-21T12:37:11.774 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.774 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.775 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.775 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:11.775 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398685+0000", 2024-09-21T12:37:11.775 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.775 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.775 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.775 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:11.775 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398685+0000", 2024-09-21T12:37:11.775 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5600000000000001e-07 2024-09-21T12:37:11.775 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.775 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.775 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:11.775 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398717+0000", 2024-09-21T12:37:11.775 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2150999999999998e-05 2024-09-21T12:37:11.775 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.775 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.776 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:11.776 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398723+0000", 2024-09-21T12:37:11.776 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1680000000000001e-06 2024-09-21T12:37:11.776 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.776 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.776 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:11.776 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398727+0000", 2024-09-21T12:37:11.776 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3189999999999999e-06 2024-09-21T12:37:11.776 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.776 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.776 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.776 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398755+0000", 2024-09-21T12:37:11.776 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8285000000000001e-05 2024-09-21T12:37:11.776 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.777 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.777 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:11.777 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.398760+0000", 2024-09-21T12:37:11.777 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2850000000000002e-06 2024-09-21T12:37:11.777 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.777 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.777 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.777 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.395783+0000", 2024-09-21T12:37:11.777 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.99702310800000005 2024-09-21T12:37:11.777 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.777 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.777 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:11.777 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.395921+0000", 2024-09-21T12:37:11.777 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000137386 2024-09-21T12:37:11.778 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.778 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.778 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,7", 2024-09-21T12:37:11.778 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.396083+0000", 2024-09-21T12:37:11.778 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016184200000000001 2024-09-21T12:37:11.778 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.778 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.778 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:11.778 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.402513+0000", 2024-09-21T12:37:11.778 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0064306559999999999 2024-09-21T12:37:11.778 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.778 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.778 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:11.779 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.403401+0000", 2024-09-21T12:37:11.779 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00088756700000000002 2024-09-21T12:37:11.779 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.779 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.779 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:11.779 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.404477+0000", 2024-09-21T12:37:11.779 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001076495 2024-09-21T12:37:11.779 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.779 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.779 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:11.779 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.404545+0000", 2024-09-21T12:37:11.779 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8096999999999995e-05 2024-09-21T12:37:11.779 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.779 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.780 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:11.780 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.404670+0000", 2024-09-21T12:37:11.780 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000124709 2024-09-21T12:37:11.780 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.780 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:11.780 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.780 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.780 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.780 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:4758 10.e 10:7d84b820:::benchmark_data_smithi114_161690_object4757:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-09-21T12:37:11.780 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:32:40.406634+0000", 2024-09-21T12:37:11.780 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 270.66956812799998, 2024-09-21T12:37:11.780 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.99406411500000003, 2024-09-21T12:37:11.780 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:11.780 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:11.781 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:11.781 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:11.781 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:37:11.781 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:37:11.781 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 4758 2024-09-21T12:37:11.781 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.781 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:11.781 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.781 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:11.781 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.406634+0000", 2024-09-21T12:37:11.781 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.781 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.781 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.781 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:11.782 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.406634+0000", 2024-09-21T12:37:11.782 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.782 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.782 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.782 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:11.782 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.406636+0000", 2024-09-21T12:37:11.782 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7600000000000001e-06 2024-09-21T12:37:11.782 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.782 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.782 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:11.782 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.406748+0000", 2024-09-21T12:37:11.782 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000112218 2024-09-21T12:37:11.782 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.783 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.783 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:11.783 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.406758+0000", 2024-09-21T12:37:11.783 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7690000000000008e-06 2024-09-21T12:37:11.783 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.783 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.783 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:11.783 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.406765+0000", 2024-09-21T12:37:11.783 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0999999999999998e-06 2024-09-21T12:37:11.783 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.783 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.783 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.783 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.406841+0000", 2024-09-21T12:37:11.784 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6260999999999994e-05 2024-09-21T12:37:11.784 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.784 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.784 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:11.785 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.406852+0000", 2024-09-21T12:37:11.785 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0370999999999999e-05 2024-09-21T12:37:11.785 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.785 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.785 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.785 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.394417+0000", 2024-09-21T12:37:11.785 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.98756492699999998 2024-09-21T12:37:11.785 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.785 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.785 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:11.785 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.394551+0000", 2024-09-21T12:37:11.785 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013420399999999999 2024-09-21T12:37:11.785 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.785 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.786 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:37:11.786 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.394703+0000", 2024-09-21T12:37:11.786 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015263899999999999 2024-09-21T12:37:11.786 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.786 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.786 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:11.786 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.397860+0000", 2024-09-21T12:37:11.786 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0031562750000000001 2024-09-21T12:37:11.786 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.786 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.786 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:11.786 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.400191+0000", 2024-09-21T12:37:11.786 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0023308690000000002 2024-09-21T12:37:11.786 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.787 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.787 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:11.787 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.400377+0000", 2024-09-21T12:37:11.787 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000186305 2024-09-21T12:37:11.787 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.787 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.787 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:11.787 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.400654+0000", 2024-09-21T12:37:11.787 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00027736200000000002 2024-09-21T12:37:11.787 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.787 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.787 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:11.787 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.400698+0000", 2024-09-21T12:37:11.787 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4054999999999997e-05 2024-09-21T12:37:11.788 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.788 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:11.788 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.788 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.788 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.788 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:4774 10.e 10:72ca348c:::benchmark_data_smithi114_161690_object4773:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-09-21T12:37:11.788 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:32:40.414678+0000", 2024-09-21T12:37:11.788 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 270.66152382600001, 2024-09-21T12:37:11.788 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.98615683700000001, 2024-09-21T12:37:11.788 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:11.788 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:11.788 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:11.788 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:11.789 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:37:11.789 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:37:11.789 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 4774 2024-09-21T12:37:11.789 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.789 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:11.789 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.789 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:11.789 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.414678+0000", 2024-09-21T12:37:11.789 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.789 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.789 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.789 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:11.789 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.414678+0000", 2024-09-21T12:37:11.789 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.790 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.790 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.790 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:11.790 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.414680+0000", 2024-09-21T12:37:11.790 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7850000000000001e-06 2024-09-21T12:37:11.790 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.790 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.790 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:11.790 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.414722+0000", 2024-09-21T12:37:11.790 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1987000000000003e-05 2024-09-21T12:37:11.790 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.790 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.790 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:11.790 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.414732+0000", 2024-09-21T12:37:11.791 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7899999999999994e-06 2024-09-21T12:37:11.791 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.791 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.791 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:11.791 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.414737+0000", 2024-09-21T12:37:11.791 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.809e-06 2024-09-21T12:37:11.791 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.791 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.791 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.791 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.414784+0000", 2024-09-21T12:37:11.791 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7478000000000002e-05 2024-09-21T12:37:11.791 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.791 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.791 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:11.792 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.414791+0000", 2024-09-21T12:37:11.792 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5300000000000002e-06 2024-09-21T12:37:11.792 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.792 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.792 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.792 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.396518+0000", 2024-09-21T12:37:11.792 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.981727397 2024-09-21T12:37:11.792 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.792 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.792 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:11.792 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.396626+0000", 2024-09-21T12:37:11.792 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000108282 2024-09-21T12:37:11.792 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.792 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.793 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:37:11.793 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.396757+0000", 2024-09-21T12:37:11.793 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013024799999999999 2024-09-21T12:37:11.793 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.793 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.793 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:11.793 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.400175+0000", 2024-09-21T12:37:11.793 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0034185230000000001 2024-09-21T12:37:11.793 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.793 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.793 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:11.793 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.400306+0000", 2024-09-21T12:37:11.793 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013045400000000001 2024-09-21T12:37:11.794 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.794 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.794 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:11.794 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.400741+0000", 2024-09-21T12:37:11.794 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00043500800000000002 2024-09-21T12:37:11.794 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.794 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.794 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:11.794 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.400792+0000", 2024-09-21T12:37:11.794 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1277999999999999e-05 2024-09-21T12:37:11.794 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.794 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.794 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:11.794 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.400835+0000", 2024-09-21T12:37:11.795 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3268e-05 2024-09-21T12:37:11.795 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.795 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:11.795 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.795 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.795 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.795 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:4775 10.e 10:72b3c699:::benchmark_data_smithi114_161690_object4774:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-09-21T12:37:11.795 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:32:40.415654+0000", 2024-09-21T12:37:11.795 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 270.66054785699998, 2024-09-21T12:37:11.795 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.98873270000000002, 2024-09-21T12:37:11.795 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:11.795 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:11.795 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:11.795 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:11.796 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:37:11.796 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:37:11.796 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 4775 2024-09-21T12:37:11.796 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.796 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:11.796 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.796 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:11.796 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.415654+0000", 2024-09-21T12:37:11.796 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.796 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.796 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.796 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:11.796 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.415654+0000", 2024-09-21T12:37:11.797 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.797 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.797 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.797 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:11.797 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.415655+0000", 2024-09-21T12:37:11.797 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9400000000000004e-07 2024-09-21T12:37:11.797 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.797 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.797 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:11.797 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.415718+0000", 2024-09-21T12:37:11.797 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2335999999999994e-05 2024-09-21T12:37:11.797 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.797 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.797 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:11.797 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.415727+0000", 2024-09-21T12:37:11.798 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5459999999999997e-06 2024-09-21T12:37:11.798 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.798 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.798 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:11.798 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.415732+0000", 2024-09-21T12:37:11.798 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7430000000000002e-06 2024-09-21T12:37:11.798 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.798 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.798 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.798 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.415770+0000", 2024-09-21T12:37:11.798 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8055000000000001e-05 2024-09-21T12:37:11.798 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.798 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.799 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:11.799 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.415775+0000", 2024-09-21T12:37:11.799 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3310000000000003e-06 2024-09-21T12:37:11.799 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.799 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.799 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.799 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.397298+0000", 2024-09-21T12:37:11.799 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.98152272699999998 2024-09-21T12:37:11.799 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.799 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.799 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:11.799 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.397404+0000", 2024-09-21T12:37:11.799 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000105733 2024-09-21T12:37:11.799 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.800 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.800 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:37:11.800 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.397517+0000", 2024-09-21T12:37:11.800 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000113713 2024-09-21T12:37:11.800 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.800 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.800 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:11.800 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.400341+0000", 2024-09-21T12:37:11.800 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0028232130000000002 2024-09-21T12:37:11.800 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.800 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.800 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:11.800 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.400845+0000", 2024-09-21T12:37:11.801 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00050434899999999996 2024-09-21T12:37:11.801 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.801 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.801 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:11.801 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.404223+0000", 2024-09-21T12:37:11.801 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0033781430000000001 2024-09-21T12:37:11.801 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.801 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.801 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:11.801 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.404293+0000", 2024-09-21T12:37:11.801 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0326000000000004e-05 2024-09-21T12:37:11.801 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.801 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.801 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:11.802 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.404387+0000", 2024-09-21T12:37:11.802 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3591000000000006e-05 2024-09-21T12:37:11.802 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.802 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:11.802 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.802 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.802 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.802 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:4776 10.1 10:888d7384:::benchmark_data_smithi114_161690_object4775:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-09-21T12:37:11.802 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:32:40.417419+0000", 2024-09-21T12:37:11.802 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 270.65878278100001, 2024-09-21T12:37:11.802 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.98747165000000003, 2024-09-21T12:37:11.802 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:11.802 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:11.802 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:11.803 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:11.803 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:37:11.803 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:37:11.803 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 4776 2024-09-21T12:37:11.803 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.803 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:11.803 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.803 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:11.803 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.417419+0000", 2024-09-21T12:37:11.803 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.803 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.803 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.803 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:11.803 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.417419+0000", 2024-09-21T12:37:11.803 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.804 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.804 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.804 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:11.804 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.417420+0000", 2024-09-21T12:37:11.804 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7499999999999999e-07 2024-09-21T12:37:11.804 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.804 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.804 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:11.804 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.417438+0000", 2024-09-21T12:37:11.804 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8039000000000001e-05 2024-09-21T12:37:11.804 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.804 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.804 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:11.804 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.417448+0000", 2024-09-21T12:37:11.805 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2699999999999993e-06 2024-09-21T12:37:11.805 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.805 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.805 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:11.805 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.417452+0000", 2024-09-21T12:37:11.805 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8160000000000001e-06 2024-09-21T12:37:11.805 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.805 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.805 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.805 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.417495+0000", 2024-09-21T12:37:11.805 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3003000000000001e-05 2024-09-21T12:37:11.805 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.805 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.805 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:11.806 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.417502+0000", 2024-09-21T12:37:11.806 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5699999999999998e-06 2024-09-21T12:37:11.806 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.806 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.806 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.806 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.397050+0000", 2024-09-21T12:37:11.806 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.97954854000000002 2024-09-21T12:37:11.806 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.806 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.806 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:11.806 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.397324+0000", 2024-09-21T12:37:11.806 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00027303499999999998 2024-09-21T12:37:11.806 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.807 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.807 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,7", 2024-09-21T12:37:11.807 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.397998+0000", 2024-09-21T12:37:11.807 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00067407299999999997 2024-09-21T12:37:11.807 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.807 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.807 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:11.807 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.403408+0000", 2024-09-21T12:37:11.807 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0054104849999999996 2024-09-21T12:37:11.807 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.807 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.807 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:11.807 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.404399+0000", 2024-09-21T12:37:11.807 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0009913159999999999 2024-09-21T12:37:11.808 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.808 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.808 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:11.808 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.404714+0000", 2024-09-21T12:37:11.808 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00031463800000000001 2024-09-21T12:37:11.808 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.808 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.808 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:11.808 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.404775+0000", 2024-09-21T12:37:11.808 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1446999999999996e-05 2024-09-21T12:37:11.808 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.808 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.808 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:11.808 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.404891+0000", 2024-09-21T12:37:11.808 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000115643 2024-09-21T12:37:11.809 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.809 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:11.809 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.809 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.809 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.809 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:4800 10.1 10:8bf9335d:::benchmark_data_smithi114_161690_object4799:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-09-21T12:37:11.809 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:32:40.434905+0000", 2024-09-21T12:37:11.809 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 270.64129718100003, 2024-09-21T12:37:11.809 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.97031433600000006, 2024-09-21T12:37:11.809 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:11.809 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:11.809 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:11.809 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:11.809 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:37:11.810 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:37:11.810 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 4800 2024-09-21T12:37:11.810 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.810 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:11.810 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.810 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:11.810 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.434905+0000", 2024-09-21T12:37:11.810 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.810 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.810 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.810 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:11.810 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.434905+0000", 2024-09-21T12:37:11.810 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.811 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.811 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.811 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:11.811 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.434907+0000", 2024-09-21T12:37:11.811 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.643e-06 2024-09-21T12:37:11.811 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.811 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.811 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:11.811 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.435037+0000", 2024-09-21T12:37:11.811 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000130235 2024-09-21T12:37:11.811 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.811 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.811 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:11.812 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.435046+0000", 2024-09-21T12:37:11.812 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4320000000000005e-06 2024-09-21T12:37:11.812 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.812 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.812 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:11.812 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.435054+0000", 2024-09-21T12:37:11.812 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.199e-06 2024-09-21T12:37:11.812 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.812 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.812 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.812 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.435105+0000", 2024-09-21T12:37:11.812 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1756999999999999e-05 2024-09-21T12:37:11.812 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.813 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.813 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:11.813 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.435115+0000", 2024-09-21T12:37:11.813 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0010000000000001e-05 2024-09-21T12:37:11.813 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.813 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.813 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.813 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.399510+0000", 2024-09-21T12:37:11.813 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.96439424100000004 2024-09-21T12:37:11.813 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.813 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.813 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:11.813 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.399630+0000", 2024-09-21T12:37:11.814 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012031099999999999 2024-09-21T12:37:11.814 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.814 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.814 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,7", 2024-09-21T12:37:11.814 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401455+0000", 2024-09-21T12:37:11.814 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001824712 2024-09-21T12:37:11.814 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.814 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.814 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:11.814 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.403578+0000", 2024-09-21T12:37:11.814 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021233340000000002 2024-09-21T12:37:11.814 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.814 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.814 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:11.815 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.405023+0000", 2024-09-21T12:37:11.815 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001444695 2024-09-21T12:37:11.815 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.815 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.815 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:11.815 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.405067+0000", 2024-09-21T12:37:11.815 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4169999999999999e-05 2024-09-21T12:37:11.815 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.815 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.815 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:11.815 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.405115+0000", 2024-09-21T12:37:11.815 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8257999999999998e-05 2024-09-21T12:37:11.816 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.816 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.816 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:11.816 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.405219+0000", 2024-09-21T12:37:11.816 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010433900000000001 2024-09-21T12:37:11.816 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.816 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:11.816 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.816 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.816 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.816 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:4806 10.1 10:8e1cf431:::benchmark_data_smithi114_161690_object4805:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-09-21T12:37:11.816 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:32:40.440760+0000", 2024-09-21T12:37:11.816 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 270.635442592, 2024-09-21T12:37:11.816 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.96816857999999995, 2024-09-21T12:37:11.817 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:11.817 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:11.817 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:11.817 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:11.817 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:37:11.817 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:37:11.817 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 4806 2024-09-21T12:37:11.817 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.817 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:11.817 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.817 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:11.817 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.440760+0000", 2024-09-21T12:37:11.817 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.817 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.817 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.818 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:11.818 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.440760+0000", 2024-09-21T12:37:11.818 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.818 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.818 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.818 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:11.818 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.440761+0000", 2024-09-21T12:37:11.818 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1429999999999999e-06 2024-09-21T12:37:11.818 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.818 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.818 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:11.818 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.440832+0000", 2024-09-21T12:37:11.818 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1087000000000006e-05 2024-09-21T12:37:11.818 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.818 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.819 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:11.819 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.440842+0000", 2024-09-21T12:37:11.819 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8959999999999998e-06 2024-09-21T12:37:11.819 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.819 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.819 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:11.819 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.440849+0000", 2024-09-21T12:37:11.819 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9229999999999996e-06 2024-09-21T12:37:11.819 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.819 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.819 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.819 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.440893+0000", 2024-09-21T12:37:11.819 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4635000000000002e-05 2024-09-21T12:37:11.819 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.820 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.820 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:11.820 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.440899+0000", 2024-09-21T12:37:11.820 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6690000000000003e-06 2024-09-21T12:37:11.820 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.820 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.820 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.820 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.405253+0000", 2024-09-21T12:37:11.820 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.96435404800000002 2024-09-21T12:37:11.820 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.820 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.820 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:11.820 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.405423+0000", 2024-09-21T12:37:11.820 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000169906 2024-09-21T12:37:11.821 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.821 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.821 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,7", 2024-09-21T12:37:11.821 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.405582+0000", 2024-09-21T12:37:11.821 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015887200000000001 2024-09-21T12:37:11.821 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.821 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.821 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:11.821 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.407870+0000", 2024-09-21T12:37:11.821 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0022885280000000002 2024-09-21T12:37:11.821 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.821 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.821 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:11.821 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.407890+0000", 2024-09-21T12:37:11.822 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0069999999999999e-05 2024-09-21T12:37:11.822 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.822 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.822 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:11.822 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.408766+0000", 2024-09-21T12:37:11.822 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00087574899999999999 2024-09-21T12:37:11.822 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.822 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.822 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:11.822 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.408824+0000", 2024-09-21T12:37:11.822 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.821e-05 2024-09-21T12:37:11.822 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.822 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.823 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:11.823 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.408928+0000", 2024-09-21T12:37:11.823 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000103844 2024-09-21T12:37:11.823 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.823 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:11.823 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.823 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.823 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.823 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:4810 10.d 10:bb36af8e:::benchmark_data_smithi114_161690_object4809:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-09-21T12:37:11.823 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:32:41.401462+0000", 2024-09-21T12:37:11.823 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 269.67474046799998, 2024-09-21T12:37:11.823 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.63868821899999995, 2024-09-21T12:37:11.823 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:11.824 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:11.824 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:11.824 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:11.825 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:37:11.825 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:37:11.825 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 4810 2024-09-21T12:37:11.825 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.826 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:11.826 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.826 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:11.826 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401462+0000", 2024-09-21T12:37:11.826 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.826 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.826 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.826 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:11.826 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401462+0000", 2024-09-21T12:37:11.826 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.826 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.826 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.826 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:11.827 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401463+0000", 2024-09-21T12:37:11.827 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8999999999999995e-07 2024-09-21T12:37:11.827 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.827 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.827 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:11.827 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401497+0000", 2024-09-21T12:37:11.827 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4382999999999999e-05 2024-09-21T12:37:11.827 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.827 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.827 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:11.827 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401507+0000", 2024-09-21T12:37:11.827 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0462000000000001e-05 2024-09-21T12:37:11.827 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.827 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.828 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:11.828 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401514+0000", 2024-09-21T12:37:11.828 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8770000000000004e-06 2024-09-21T12:37:11.828 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.828 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.828 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.828 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401970+0000", 2024-09-21T12:37:11.828 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00045594199999999998 2024-09-21T12:37:11.828 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.828 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.828 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:11.828 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401980+0000", 2024-09-21T12:37:11.828 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5980000000000003e-06 2024-09-21T12:37:11.829 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.829 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.829 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.829 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:42.036422+0000", 2024-09-21T12:37:11.829 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.63444185200000003 2024-09-21T12:37:11.829 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.829 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.829 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:11.829 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:42.036595+0000", 2024-09-21T12:37:11.829 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000173229 2024-09-21T12:37:11.829 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.829 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.829 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,6", 2024-09-21T12:37:11.829 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:42.036791+0000", 2024-09-21T12:37:11.830 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019604099999999999 2024-09-21T12:37:11.830 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.830 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.830 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:11.830 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:42.038080+0000", 2024-09-21T12:37:11.830 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012889850000000001 2024-09-21T12:37:11.830 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.830 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.830 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:11.830 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:42.039965+0000", 2024-09-21T12:37:11.830 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001884957 2024-09-21T12:37:11.830 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.830 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.831 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:11.831 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:42.040040+0000", 2024-09-21T12:37:11.831 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4645000000000001e-05 2024-09-21T12:37:11.831 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.831 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.831 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:11.831 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:42.040101+0000", 2024-09-21T12:37:11.831 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1365000000000006e-05 2024-09-21T12:37:11.831 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.831 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.831 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:11.831 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:42.040150+0000", 2024-09-21T12:37:11.831 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9092999999999999e-05 2024-09-21T12:37:11.832 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.832 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:11.832 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.832 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.832 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.832 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:4824 10.d 10:bda3e2d2:::benchmark_data_smithi114_161690_object4823:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-09-21T12:37:11.832 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:32:41.407783+0000", 2024-09-21T12:37:11.832 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 269.66841896699998, 2024-09-21T12:37:11.832 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.632637691, 2024-09-21T12:37:11.832 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:11.832 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:11.832 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:11.832 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:11.833 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:37:11.833 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:37:11.833 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 4824 2024-09-21T12:37:11.833 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.833 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:11.833 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.833 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:11.833 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.407783+0000", 2024-09-21T12:37:11.833 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.833 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.833 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.833 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:11.833 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.407783+0000", 2024-09-21T12:37:11.833 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.833 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.834 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.834 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:11.834 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.407785+0000", 2024-09-21T12:37:11.834 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4300000000000001e-06 2024-09-21T12:37:11.834 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.834 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.834 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:11.834 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.407951+0000", 2024-09-21T12:37:11.834 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016585999999999999 2024-09-21T12:37:11.834 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.834 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.834 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:11.834 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.407968+0000", 2024-09-21T12:37:11.835 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7093999999999999e-05 2024-09-21T12:37:11.835 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.835 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.835 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:11.835 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.407975+0000", 2024-09-21T12:37:11.835 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7719999999999994e-06 2024-09-21T12:37:11.835 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.835 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.835 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.835 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.408039+0000", 2024-09-21T12:37:11.835 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3856999999999995e-05 2024-09-21T12:37:11.835 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.835 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.835 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:11.835 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.408049+0000", 2024-09-21T12:37:11.836 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0159999999999999e-05 2024-09-21T12:37:11.836 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.836 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.836 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.836 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:42.038129+0000", 2024-09-21T12:37:11.836 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.63007954499999996 2024-09-21T12:37:11.836 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.836 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.836 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:11.836 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:42.038306+0000", 2024-09-21T12:37:11.836 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017660899999999999 2024-09-21T12:37:11.836 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.836 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.836 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,6", 2024-09-21T12:37:11.836 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:42.038502+0000", 2024-09-21T12:37:11.836 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000195997 2024-09-21T12:37:11.837 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.837 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.837 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:11.837 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:42.040256+0000", 2024-09-21T12:37:11.837 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001754715 2024-09-21T12:37:11.837 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.837 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.837 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:11.837 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:42.040298+0000", 2024-09-21T12:37:11.837 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2159000000000001e-05 2024-09-21T12:37:11.837 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.837 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.837 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:11.838 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:42.040332+0000", 2024-09-21T12:37:11.838 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3847999999999998e-05 2024-09-21T12:37:11.838 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.838 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.838 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:11.838 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:42.040382+0000", 2024-09-21T12:37:11.838 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9273000000000001e-05 2024-09-21T12:37:11.838 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.838 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.838 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:11.838 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:42.040421+0000", 2024-09-21T12:37:11.838 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9372000000000002e-05 2024-09-21T12:37:11.838 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.838 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:11.839 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.839 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.839 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.839 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:4865 10.d 10:bda1db2a:::benchmark_data_smithi114_161690_object4864:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-09-21T12:37:11.839 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:32:41.422887+0000", 2024-09-21T12:37:11.839 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 269.65331519099999, 2024-09-21T12:37:11.839 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61825188200000003, 2024-09-21T12:37:11.839 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:11.839 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:11.839 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:11.839 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:11.839 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:37:11.840 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:37:11.840 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 4865 2024-09-21T12:37:11.840 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.840 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:11.840 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.840 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:11.840 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.422887+0000", 2024-09-21T12:37:11.840 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.840 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.840 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.840 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:11.840 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.422887+0000", 2024-09-21T12:37:11.840 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.840 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.840 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.841 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:11.841 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.422889+0000", 2024-09-21T12:37:11.841 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5209999999999999e-06 2024-09-21T12:37:11.841 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.841 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.841 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:11.841 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.422943+0000", 2024-09-21T12:37:11.841 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4295000000000001e-05 2024-09-21T12:37:11.841 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.841 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.841 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:11.841 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.422956+0000", 2024-09-21T12:37:11.841 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3254999999999999e-05 2024-09-21T12:37:11.841 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.841 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.842 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:11.842 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.422963+0000", 2024-09-21T12:37:11.842 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3610000000000003e-06 2024-09-21T12:37:11.842 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.842 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.842 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.842 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.423035+0000", 2024-09-21T12:37:11.842 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1481000000000002e-05 2024-09-21T12:37:11.842 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.842 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.842 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:11.842 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.423042+0000", 2024-09-21T12:37:11.842 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1019999999999999e-06 2024-09-21T12:37:11.843 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.843 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.843 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.843 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:42.039149+0000", 2024-09-21T12:37:11.843 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.61610706299999995 2024-09-21T12:37:11.843 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.843 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.843 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:11.843 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:42.039279+0000", 2024-09-21T12:37:11.843 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013017000000000001 2024-09-21T12:37:11.843 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.843 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.843 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,6", 2024-09-21T12:37:11.843 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:42.039468+0000", 2024-09-21T12:37:11.844 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000188567 2024-09-21T12:37:11.844 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.844 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.844 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:11.844 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:42.040442+0000", 2024-09-21T12:37:11.844 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00097431000000000004 2024-09-21T12:37:11.844 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.844 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.844 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:11.844 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:42.040677+0000", 2024-09-21T12:37:11.844 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00023520600000000001 2024-09-21T12:37:11.844 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.844 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.844 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:11.845 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:42.041036+0000", 2024-09-21T12:37:11.845 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00035842099999999999 2024-09-21T12:37:11.845 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.845 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.845 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:11.845 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:42.041097+0000", 2024-09-21T12:37:11.845 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1418e-05 2024-09-21T12:37:11.845 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.845 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.845 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:11.845 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:42.041139+0000", 2024-09-21T12:37:11.845 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1712000000000003e-05 2024-09-21T12:37:11.845 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.845 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:11.846 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.846 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.846 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.846 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:43124 10.e 10:799fecac:::benchmark_data_smithi114_161690_object43123:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-09-21T12:37:11.846 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:33:00.809160+0000", 2024-09-21T12:37:11.846 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 250.267042132, 2024-09-21T12:37:11.846 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.37125677299999998, 2024-09-21T12:37:11.846 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:11.846 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:11.846 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:11.846 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:11.846 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:37:11.846 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:37:11.846 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 43124 2024-09-21T12:37:11.847 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.847 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:11.847 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.847 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:11.847 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.809160+0000", 2024-09-21T12:37:11.847 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.847 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.847 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.847 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:11.847 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.809160+0000", 2024-09-21T12:37:11.847 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.847 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.847 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.847 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:11.848 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.809161+0000", 2024-09-21T12:37:11.848 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2049999999999999e-06 2024-09-21T12:37:11.848 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.848 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.848 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:11.848 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.809187+0000", 2024-09-21T12:37:11.848 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5267999999999999e-05 2024-09-21T12:37:11.848 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.848 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.848 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:11.848 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.809195+0000", 2024-09-21T12:37:11.848 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1210000000000007e-06 2024-09-21T12:37:11.848 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.849 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.849 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:11.849 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.809200+0000", 2024-09-21T12:37:11.849 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5949999999999998e-06 2024-09-21T12:37:11.849 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.849 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.849 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.849 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.809563+0000", 2024-09-21T12:37:11.849 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00036228600000000002 2024-09-21T12:37:11.849 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.849 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.849 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:11.849 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.809667+0000", 2024-09-21T12:37:11.849 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010447500000000001 2024-09-21T12:37:11.849 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.850 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.850 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:37:11.850 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.809786+0000", 2024-09-21T12:37:11.850 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011851800000000001 2024-09-21T12:37:11.850 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.850 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.850 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:11.850 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.814289+0000", 2024-09-21T12:37:11.850 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.004503239 2024-09-21T12:37:11.850 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.850 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.850 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:11.850 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.814308+0000", 2024-09-21T12:37:11.850 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9629000000000001e-05 2024-09-21T12:37:11.850 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.851 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.851 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:11.851 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:01.180319+0000", 2024-09-21T12:37:11.851 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.36601022300000002 2024-09-21T12:37:11.851 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.851 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.851 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:11.851 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:01.180373+0000", 2024-09-21T12:37:11.851 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4098999999999998e-05 2024-09-21T12:37:11.851 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.851 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.851 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:11.851 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:01.180417+0000", 2024-09-21T12:37:11.851 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4115e-05 2024-09-21T12:37:11.851 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.852 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:11.852 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.852 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.852 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.852 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:43140 10.e 10:7d319651:::benchmark_data_smithi114_161690_object43139:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-09-21T12:37:11.852 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:33:00.816984+0000", 2024-09-21T12:37:11.852 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 250.25921849100001, 2024-09-21T12:37:11.852 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.36359987700000002, 2024-09-21T12:37:11.852 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:11.852 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:11.852 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:11.852 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:11.852 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:37:11.852 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:37:11.853 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 43140 2024-09-21T12:37:11.853 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.853 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:11.853 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.853 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:11.853 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.816984+0000", 2024-09-21T12:37:11.853 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.853 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.853 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.853 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:11.853 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.816984+0000", 2024-09-21T12:37:11.853 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.853 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.853 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.853 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:11.854 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.816985+0000", 2024-09-21T12:37:11.854 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.421e-06 2024-09-21T12:37:11.854 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.854 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.854 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:11.854 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.817044+0000", 2024-09-21T12:37:11.854 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8888999999999998e-05 2024-09-21T12:37:11.854 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.854 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.854 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:11.854 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.817057+0000", 2024-09-21T12:37:11.854 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2639999999999999e-05 2024-09-21T12:37:11.854 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.854 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.855 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:11.855 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.817064+0000", 2024-09-21T12:37:11.855 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9410000000000001e-06 2024-09-21T12:37:11.855 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.855 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.855 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.855 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.817118+0000", 2024-09-21T12:37:11.855 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4262000000000003e-05 2024-09-21T12:37:11.855 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.855 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.855 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:11.855 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.817208+0000", 2024-09-21T12:37:11.855 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9741999999999996e-05 2024-09-21T12:37:11.855 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.855 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.856 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:37:11.856 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.817321+0000", 2024-09-21T12:37:11.856 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000113163 2024-09-21T12:37:11.856 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.856 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.856 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:11.856 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.820284+0000", 2024-09-21T12:37:11.856 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002962723 2024-09-21T12:37:11.856 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.856 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.856 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:11.856 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.821842+0000", 2024-09-21T12:37:11.856 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0015581810000000001 2024-09-21T12:37:11.856 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.856 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.857 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:11.857 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:01.180501+0000", 2024-09-21T12:37:11.857 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.35865960699999999 2024-09-21T12:37:11.857 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.857 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.857 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:11.857 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:01.180550+0000", 2024-09-21T12:37:11.857 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8902e-05 2024-09-21T12:37:11.857 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.857 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.857 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:11.857 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:01.180584+0000", 2024-09-21T12:37:11.857 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3405999999999998e-05 2024-09-21T12:37:11.857 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.858 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:11.858 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.858 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.858 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.858 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:160282 10.e 10:722e82d6:::benchmark_data_smithi114_161690_object33230:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-21T12:37:11.858 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:34:33.317417+0000", 2024-09-21T12:37:11.858 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 157.758784749, 2024-09-21T12:37:11.858 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0514934460000001, 2024-09-21T12:37:11.858 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:11.858 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:11.858 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-09-21T12:37:11.858 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:11.858 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:37:11.858 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:37:11.859 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 160282 2024-09-21T12:37:11.859 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.859 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:11.859 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.859 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:11.859 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.317417+0000", 2024-09-21T12:37:11.859 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.859 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.859 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.859 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:11.859 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.317417+0000", 2024-09-21T12:37:11.859 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.859 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.859 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.859 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:11.860 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.317419+0000", 2024-09-21T12:37:11.860 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1769999999999999e-06 2024-09-21T12:37:11.860 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.860 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.860 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:11.860 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.317422+0000", 2024-09-21T12:37:11.860 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8099999999999999e-06 2024-09-21T12:37:11.860 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.860 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.860 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:11.860 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.317437+0000", 2024-09-21T12:37:11.860 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4336e-05 2024-09-21T12:37:11.860 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.860 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.860 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:11.861 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.317444+0000", 2024-09-21T12:37:11.861 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.063e-06 2024-09-21T12:37:11.861 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.861 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.861 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.861 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.317507+0000", 2024-09-21T12:37:11.861 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3219000000000006e-05 2024-09-21T12:37:11.861 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.861 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.861 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:11.861 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.317583+0000", 2024-09-21T12:37:11.861 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5537999999999994e-05 2024-09-21T12:37:11.861 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.862 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.862 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:37:11.862 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.317645+0000", 2024-09-21T12:37:11.862 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2382e-05 2024-09-21T12:37:11.862 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.862 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.862 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:11.862 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.322598+0000", 2024-09-21T12:37:11.862 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0049531410000000003 2024-09-21T12:37:11.862 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.862 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.862 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:11.862 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.322662+0000", 2024-09-21T12:37:11.862 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3751999999999995e-05 2024-09-21T12:37:11.863 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.863 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.863 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.863 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.348898+0000", 2024-09-21T12:37:11.863 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.026236114000000001 2024-09-21T12:37:11.863 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.863 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.863 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:11.863 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.348909+0000", 2024-09-21T12:37:11.863 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1072e-05 2024-09-21T12:37:11.863 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.863 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.863 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.863 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.368825+0000", 2024-09-21T12:37:11.863 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.019915793 2024-09-21T12:37:11.864 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.864 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.864 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:11.864 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.368911+0000", 2024-09-21T12:37:11.864 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6049000000000002e-05 2024-09-21T12:37:11.864 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.864 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:11.864 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.864 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.864 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.864 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:160284 10.e 10:717d2c04:::benchmark_data_smithi114_161690_object33232:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-21T12:37:11.864 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:34:33.317465+0000", 2024-09-21T12:37:11.864 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 157.75873705399999, 2024-09-21T12:37:11.864 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0528247690000001, 2024-09-21T12:37:11.864 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:11.865 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:11.865 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-09-21T12:37:11.865 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:11.865 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:37:11.865 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:37:11.865 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 160284 2024-09-21T12:37:11.865 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.865 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:11.865 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.865 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:11.865 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.317465+0000", 2024-09-21T12:37:11.865 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.865 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.865 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.866 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:11.866 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.317465+0000", 2024-09-21T12:37:11.866 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.866 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.866 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.866 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:11.866 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.317466+0000", 2024-09-21T12:37:11.866 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7400000000000003e-07 2024-09-21T12:37:11.866 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.866 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.866 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:11.866 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.317469+0000", 2024-09-21T12:37:11.866 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.864e-06 2024-09-21T12:37:11.866 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.867 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.867 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:11.867 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.317475+0000", 2024-09-21T12:37:11.867 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7710000000000001e-06 2024-09-21T12:37:11.867 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.867 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.867 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:11.867 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.317481+0000", 2024-09-21T12:37:11.867 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3730000000000001e-06 2024-09-21T12:37:11.867 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.867 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.867 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.867 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.320417+0000", 2024-09-21T12:37:11.867 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0029362070000000001 2024-09-21T12:37:11.868 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.868 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.868 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:11.868 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.320517+0000", 2024-09-21T12:37:11.868 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100006 2024-09-21T12:37:11.868 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.868 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.868 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:37:11.868 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.320616+0000", 2024-09-21T12:37:11.868 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9077e-05 2024-09-21T12:37:11.868 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.868 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.868 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:11.868 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.322686+0000", 2024-09-21T12:37:11.869 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020694469999999999 2024-09-21T12:37:11.869 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.869 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.869 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:11.869 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.325499+0000", 2024-09-21T12:37:11.869 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002813875 2024-09-21T12:37:11.869 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.869 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.869 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.869 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.348947+0000", 2024-09-21T12:37:11.869 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.023447174000000001 2024-09-21T12:37:11.869 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.869 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.869 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:11.870 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.348952+0000", 2024-09-21T12:37:11.870 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.913e-06 2024-09-21T12:37:11.870 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.870 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.870 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.870 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.370194+0000", 2024-09-21T12:37:11.870 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.021241072 2024-09-21T12:37:11.870 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.870 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.870 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:11.870 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.370290+0000", 2024-09-21T12:37:11.870 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6416000000000007e-05 2024-09-21T12:37:11.870 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.870 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:11.871 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.871 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.871 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.871 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:160293 10.e 10:711c34e8:::benchmark_data_smithi114_161690_object33241:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-09-21T12:37:11.871 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:34:33.326225+0000", 2024-09-21T12:37:11.871 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 157.749977659, 2024-09-21T12:37:11.871 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.046270002, 2024-09-21T12:37:11.871 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:11.871 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:11.871 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-09-21T12:37:11.871 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:11.871 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:37:11.871 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:37:11.872 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 160293 2024-09-21T12:37:11.872 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.872 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:11.872 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.872 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:11.872 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.326225+0000", 2024-09-21T12:37:11.872 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.872 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.872 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.872 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:11.872 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.326225+0000", 2024-09-21T12:37:11.872 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.872 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.872 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.873 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:11.873 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.326225+0000", 2024-09-21T12:37:11.873 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9100000000000002e-07 2024-09-21T12:37:11.873 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.873 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.873 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:11.873 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.326230+0000", 2024-09-21T12:37:11.873 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1579999999999998e-06 2024-09-21T12:37:11.873 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.873 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.873 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:11.873 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.326238+0000", 2024-09-21T12:37:11.873 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8550000000000008e-06 2024-09-21T12:37:11.873 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.874 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.874 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:11.874 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.326243+0000", 2024-09-21T12:37:11.874 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7119999999999999e-06 2024-09-21T12:37:11.874 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.874 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.874 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.874 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.326290+0000", 2024-09-21T12:37:11.874 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6842000000000003e-05 2024-09-21T12:37:11.874 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.874 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.874 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:11.874 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.326373+0000", 2024-09-21T12:37:11.875 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3039000000000002e-05 2024-09-21T12:37:11.875 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.875 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.875 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:37:11.875 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.326451+0000", 2024-09-21T12:37:11.875 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7676999999999995e-05 2024-09-21T12:37:11.875 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.875 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.875 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:11.875 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.329310+0000", 2024-09-21T12:37:11.875 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0028591240000000002 2024-09-21T12:37:11.875 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.875 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.875 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:11.876 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.329323+0000", 2024-09-21T12:37:11.876 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.343e-05 2024-09-21T12:37:11.876 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.876 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.876 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.876 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.348979+0000", 2024-09-21T12:37:11.876 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.019655542000000002 2024-09-21T12:37:11.876 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.876 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.876 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:11.876 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.348984+0000", 2024-09-21T12:37:11.876 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1329999999999998e-06 2024-09-21T12:37:11.876 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.876 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.877 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.877 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.370315+0000", 2024-09-21T12:37:11.877 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0213312539999999 2024-09-21T12:37:11.877 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.877 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.877 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:11.877 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.372495+0000", 2024-09-21T12:37:11.877 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021793450000000001 2024-09-21T12:37:11.877 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.877 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:11.877 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.877 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.877 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.877 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:160312 10.e 10:7dfe521f:::benchmark_data_smithi114_161690_object33260:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-21T12:37:11.877 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:34:33.362633+0000", 2024-09-21T12:37:11.878 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 157.71356944300001, 2024-09-21T12:37:11.878 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.020352052, 2024-09-21T12:37:11.878 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:11.878 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:11.878 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:11.878 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:11.878 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:37:11.878 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:37:11.878 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 160312 2024-09-21T12:37:11.878 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.878 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:11.878 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.878 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:11.878 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.362633+0000", 2024-09-21T12:37:11.879 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.879 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.879 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.879 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:11.879 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.362633+0000", 2024-09-21T12:37:11.879 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.879 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.879 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.879 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:11.879 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.362635+0000", 2024-09-21T12:37:11.879 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.81e-06 2024-09-21T12:37:11.879 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.879 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.879 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:11.880 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.362643+0000", 2024-09-21T12:37:11.880 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2539999999999992e-06 2024-09-21T12:37:11.880 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.880 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.880 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:11.880 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.362666+0000", 2024-09-21T12:37:11.880 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3512000000000001e-05 2024-09-21T12:37:11.880 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.880 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.880 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:11.880 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.362679+0000", 2024-09-21T12:37:11.880 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2697000000000001e-05 2024-09-21T12:37:11.880 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.881 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.881 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.881 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.362779+0000", 2024-09-21T12:37:11.881 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100207 2024-09-21T12:37:11.881 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.881 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.881 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:11.881 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.362791+0000", 2024-09-21T12:37:11.881 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.216e-05 2024-09-21T12:37:11.881 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.881 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.881 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.881 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.373306+0000", 2024-09-21T12:37:11.882 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.010514082 2024-09-21T12:37:11.882 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.882 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.882 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:11.882 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.373404+0000", 2024-09-21T12:37:11.882 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8029999999999995e-05 2024-09-21T12:37:11.882 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.882 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.882 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 6", 2024-09-21T12:37:11.882 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.373510+0000", 2024-09-21T12:37:11.882 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010648499999999999 2024-09-21T12:37:11.882 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.882 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.882 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:11.883 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.382762+0000", 2024-09-21T12:37:11.883 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0092519230000000004 2024-09-21T12:37:11.883 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.883 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.883 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:11.883 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.382807+0000", 2024-09-21T12:37:11.883 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5046999999999997e-05 2024-09-21T12:37:11.883 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.883 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.883 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:11.883 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.382870+0000", 2024-09-21T12:37:11.883 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2860999999999993e-05 2024-09-21T12:37:11.883 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.883 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.883 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:11.884 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.382985+0000", 2024-09-21T12:37:11.884 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000114984 2024-09-21T12:37:11.884 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.884 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:11.884 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.884 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.884 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.884 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:32512 11.e 11:7a301a7a:::benchmark_data_smithi114_187622_object32511:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-09-21T12:37:11.884 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:36:10.029727+0000", 2024-09-21T12:37:11.884 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 61.046475096999998, 2024-09-21T12:37:11.884 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.133327006, 2024-09-21T12:37:11.884 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:11.884 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:11.884 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:11.885 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:11.885 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:37:11.885 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:37:11.885 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 32512 2024-09-21T12:37:11.885 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.885 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:11.885 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.885 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:11.885 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:10.029727+0000", 2024-09-21T12:37:11.885 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.885 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.885 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.885 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:11.885 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:10.029727+0000", 2024-09-21T12:37:11.885 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:11.886 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.886 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.886 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:11.886 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:10.029729+0000", 2024-09-21T12:37:11.886 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4920000000000001e-06 2024-09-21T12:37:11.886 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.886 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.886 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:11.886 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:10.029764+0000", 2024-09-21T12:37:11.886 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5259000000000002e-05 2024-09-21T12:37:11.886 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.886 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.886 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:11.886 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:10.029781+0000", 2024-09-21T12:37:11.886 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7377e-05 2024-09-21T12:37:11.887 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.887 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.887 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:11.887 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:10.029790+0000", 2024-09-21T12:37:11.887 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7839999999999992e-06 2024-09-21T12:37:11.887 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.887 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.887 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:11.887 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:10.029863+0000", 2024-09-21T12:37:11.887 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2720999999999997e-05 2024-09-21T12:37:11.887 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.887 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.887 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:11.887 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:10.029991+0000", 2024-09-21T12:37:11.888 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012798900000000001 2024-09-21T12:37:11.888 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.888 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.888 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:37:11.888 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:10.030119+0000", 2024-09-21T12:37:11.888 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012862999999999999 2024-09-21T12:37:11.888 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.888 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.888 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:11.888 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:10.031608+0000", 2024-09-21T12:37:11.888 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001488468 2024-09-21T12:37:11.888 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.888 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.888 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:11.889 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:10.032115+0000", 2024-09-21T12:37:11.889 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00050718300000000003 2024-09-21T12:37:11.889 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.889 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.889 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:11.889 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:10.162941+0000", 2024-09-21T12:37:11.889 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.130826308 2024-09-21T12:37:11.889 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.889 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.889 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:11.889 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:10.162992+0000", 2024-09-21T12:37:11.889 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0708999999999998e-05 2024-09-21T12:37:11.889 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:11.889 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:11.889 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:11.890 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:10.163054+0000", 2024-09-21T12:37:11.890 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2086000000000002e-05 2024-09-21T12:37:11.890 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.890 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:11.890 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.890 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:11.890 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:11.890 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:37:11.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:11.892 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:11.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:11.966 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:37:11.993 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:12.021 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:12.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:12.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:12.123 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:12.160 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:12.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:12.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:12.261 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:12.295 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:12.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:12.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:12.397 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:12.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:12.429 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:12.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:12.477 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:12 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-09-21T12:37:12.477 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:12 smithi114 ceph-mon[40867]: osdmap e438: 8 total, 7 up, 8 in 2024-09-21T12:37:12.477 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:12 smithi114 ceph-mon[40867]: 3.f scrub starts 2024-09-21T12:37:12.477 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:12 smithi114 ceph-mon[40867]: 3.f scrub ok 2024-09-21T12:37:12.478 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:12 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 29551/355884 objects degraded (8.304%), 13 pgs degraded, 33 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:12.478 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:12 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-09-21T12:37:12.478 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:12 smithi114 ceph-mon[30324]: osdmap e438: 8 total, 7 up, 8 in 2024-09-21T12:37:12.478 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:12 smithi114 ceph-mon[30324]: 3.f scrub starts 2024-09-21T12:37:12.478 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:12 smithi114 ceph-mon[30324]: 3.f scrub ok 2024-09-21T12:37:12.478 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:12 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 29551/355884 objects degraded (8.304%), 13 pgs degraded, 33 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:12 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-09-21T12:37:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:12 smithi183 ceph-mon[33277]: osdmap e438: 8 total, 7 up, 8 in 2024-09-21T12:37:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:12 smithi183 ceph-mon[33277]: 3.f scrub starts 2024-09-21T12:37:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:12 smithi183 ceph-mon[33277]: 3.f scrub ok 2024-09-21T12:37:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:12 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 29551/355884 objects degraded (8.304%), 13 pgs degraded, 33 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:12.513 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 79 16 125541 125525 99.2908 79.875 0.0193408 0.0100675 2024-09-21T12:37:12.530 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:12.563 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:12.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:12.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:12.665 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:12.695 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:12.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:12.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:12.775 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:37:12 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:37:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:37:12.797 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:12.803 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T12:37:12.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:12.829 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:12.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:12.931 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:12.962 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:12.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:12.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:12.978 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T12:37:13.064 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:13.087 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:13.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:13.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:13.178 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:37:13.189 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:13.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:13.235 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:13.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:13.336 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:13.380 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:13.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:13.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:13.430 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:13 smithi114 ceph-mon[40867]: pgmap v862: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 7.3 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.03k op/s; 29973/359766 objects degraded (8.331%); 14845/359766 objects misplaced (4.126%) 2024-09-21T12:37:13.430 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:13 smithi114 ceph-mon[40867]: 11.0 scrub starts 2024-09-21T12:37:13.430 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:13 smithi114 ceph-mon[30324]: pgmap v862: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 7.3 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.03k op/s; 29973/359766 objects degraded (8.331%); 14845/359766 objects misplaced (4.126%) 2024-09-21T12:37:13.430 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:13 smithi114 ceph-mon[30324]: 11.0 scrub starts 2024-09-21T12:37:13.467 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:13 smithi183 ceph-mon[33277]: pgmap v862: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 7.3 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.03k op/s; 29973/359766 objects degraded (8.331%); 14845/359766 objects misplaced (4.126%) 2024-09-21T12:37:13.467 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:13 smithi183 ceph-mon[33277]: 11.0 scrub starts 2024-09-21T12:37:13.481 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:13.513 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-21T12:37:13.513516+0000 min lat: 0.00111509 max lat: 2.08308 avg lat: 0.0100973 2024-09-21T12:37:13.514 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T12:37:13.514 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 80 16 126752 126736 98.9957 75.6875 0.0102892 0.0100973 2024-09-21T12:37:13.523 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:13.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:13.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:13.625 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:13.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:13.662 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:13.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:13.763 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:13.795 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:37:13.795 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:37:13.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:13.801 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:13.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:13.902 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:13.926 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:13.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:13.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:14.028 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:14.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:14.066 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:14.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:14.167 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:14.202 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:14.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:14.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:14.304 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:14.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:14.332 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:14.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:14.333 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:37:14.334 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:37:14.334 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:37:14.334 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:37:14.334 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:37:14.334 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:37:14.334 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T12:37:14.334 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:37:14.334 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:37:14.334 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:37:14.335 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:37:14.335 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:37:14.335 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:37:14.335 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:37:14.335 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:37:14.434 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:14.469 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:14.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:14.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:14.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:14 smithi183 ceph-mon[33277]: 5.14 scrub starts 2024-09-21T12:37:14.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:14 smithi183 ceph-mon[33277]: 5.14 scrub ok 2024-09-21T12:37:14.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:14 smithi183 ceph-mon[33277]: 11.3 scrub starts 2024-09-21T12:37:14.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:14 smithi183 ceph-mon[33277]: 11.f scrub ok 2024-09-21T12:37:14.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:14 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/947090133' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:37:14.513 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 81 16 127776 127760 98.5636 64 0.00489044 0.0101417 2024-09-21T12:37:14.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:14 smithi114 ceph-mon[40867]: 5.14 scrub starts 2024-09-21T12:37:14.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:14 smithi114 ceph-mon[40867]: 5.14 scrub ok 2024-09-21T12:37:14.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:14 smithi114 ceph-mon[40867]: 11.3 scrub starts 2024-09-21T12:37:14.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:14 smithi114 ceph-mon[40867]: 11.f scrub ok 2024-09-21T12:37:14.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:14 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/947090133' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:37:14.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:14 smithi114 ceph-mon[30324]: 5.14 scrub starts 2024-09-21T12:37:14.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:14 smithi114 ceph-mon[30324]: 5.14 scrub ok 2024-09-21T12:37:14.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:14 smithi114 ceph-mon[30324]: 11.3 scrub starts 2024-09-21T12:37:14.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:14 smithi114 ceph-mon[30324]: 11.f scrub ok 2024-09-21T12:37:14.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:14 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/947090133' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:37:14.571 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:14.592 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:14.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:14.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:14.694 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:14.714 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:14.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:14.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:14.815 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:14.836 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:14.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:14.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:14.937 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:14.971 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:14.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:14.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:15.073 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:15.111 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:15.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:15.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:15.212 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:15.247 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:15.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:15.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:15.349 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:15.381 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:15.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:15.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:15.482 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:15.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:15 smithi183 ceph-mon[33277]: pgmap v863: 178 pgs: 1 active+clean+remapped+scrubbing, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 7.6 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s; 31215/376443 objects degraded (8.292%); 15686/376443 objects misplaced (4.167%) 2024-09-21T12:37:15.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:15 smithi183 ceph-mon[33277]: 11.7 scrub starts 2024-09-21T12:37:15.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:15 smithi183 ceph-mon[33277]: 1.0 scrub starts 2024-09-21T12:37:15.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:15 smithi183 ceph-mon[33277]: 1.0 scrub ok 2024-09-21T12:37:15.514 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 82 16 128746 128730 98.1009 60.625 0.0117635 0.010189 2024-09-21T12:37:15.516 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:15.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:15.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:15.523 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:15 smithi114 ceph-mon[40867]: pgmap v863: 178 pgs: 1 active+clean+remapped+scrubbing, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 7.6 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s; 31215/376443 objects degraded (8.292%); 15686/376443 objects misplaced (4.167%) 2024-09-21T12:37:15.523 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:15 smithi114 ceph-mon[40867]: 11.7 scrub starts 2024-09-21T12:37:15.523 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:15 smithi114 ceph-mon[40867]: 1.0 scrub starts 2024-09-21T12:37:15.523 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:15 smithi114 ceph-mon[40867]: 1.0 scrub ok 2024-09-21T12:37:15.523 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:15 smithi114 ceph-mon[30324]: pgmap v863: 178 pgs: 1 active+clean+remapped+scrubbing, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 7.6 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s; 31215/376443 objects degraded (8.292%); 15686/376443 objects misplaced (4.167%) 2024-09-21T12:37:15.523 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:15 smithi114 ceph-mon[30324]: 11.7 scrub starts 2024-09-21T12:37:15.523 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:15 smithi114 ceph-mon[30324]: 1.0 scrub starts 2024-09-21T12:37:15.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:15 smithi114 ceph-mon[30324]: 1.0 scrub ok 2024-09-21T12:37:15.567 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T12:37:15.567 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T12:37:15.613 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:37:15.618 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:15.664 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:15.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:15.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:15.765 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:15.794 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:15.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:15.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:15.896 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:15.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:15.933 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:15.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:16.033 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:16.067 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:16.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:16.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:16.169 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:16.196 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:16.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:16.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:16.298 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:16.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:16.338 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:16.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:16.439 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:16.477 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:16.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:16.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:16 smithi183 ceph-mon[33277]: 11.0 scrub ok 2024-09-21T12:37:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:16 smithi183 ceph-mon[33277]: 11.4 scrub starts 2024-09-21T12:37:16.514 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 83 16 129968 129952 97.8391 76.375 0.0124983 0.0102168 2024-09-21T12:37:16.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:16 smithi114 ceph-mon[40867]: 11.0 scrub ok 2024-09-21T12:37:16.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:16 smithi114 ceph-mon[40867]: 11.4 scrub starts 2024-09-21T12:37:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:16 smithi114 ceph-mon[30324]: 11.0 scrub ok 2024-09-21T12:37:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:16 smithi114 ceph-mon[30324]: 11.4 scrub starts 2024-09-21T12:37:16.579 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:16.614 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:16.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:16.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:16.715 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:16.753 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:16.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:16.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:16.854 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:16.887 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:16.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:16.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:16.924 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 4, 7] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-09-21T12:37:16.924 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:37:16.924 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T12:37:16.924 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.890946 2024-09-21T12:37:16.924 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 5 0.890945750752006 2024-09-21T12:37:16.989 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:17.030 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:17.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:17.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:17.132 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:17.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:17.159 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:17.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:17.197 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-21T12:37:17.218 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:17 smithi114 ceph-mon[40867]: 11.3 scrub ok 2024-09-21T12:37:17.218 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:17 smithi114 ceph-mon[40867]: pgmap v864: 178 pgs: 1 active+clean+remapped+scrubbing, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 7.6 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 913 op/s; 31215/376443 objects degraded (8.292%); 15686/376443 objects misplaced (4.167%) 2024-09-21T12:37:17.218 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:17 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3334338737' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:37:17.218 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:17 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:37:17.218 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:17 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:37:17.218 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:17 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:37:17.218 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:17 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 31215/376443 objects degraded (8.292%), 13 pgs degraded, 33 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:17.219 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:17 smithi114 ceph-mon[30324]: 11.3 scrub ok 2024-09-21T12:37:17.219 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:17 smithi114 ceph-mon[30324]: pgmap v864: 178 pgs: 1 active+clean+remapped+scrubbing, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 7.6 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 913 op/s; 31215/376443 objects degraded (8.292%); 15686/376443 objects misplaced (4.167%) 2024-09-21T12:37:17.219 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:17 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3334338737' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:37:17.219 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:17 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:37:17.219 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:17 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:37:17.219 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:17 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:37:17.219 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:17 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 31215/376443 objects degraded (8.292%), 13 pgs degraded, 33 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:17.240 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T12:37:17.260 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:17.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:17.325 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:17.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:17.426 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:17.468 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:17.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:17.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:17.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:17 smithi183 ceph-mon[33277]: 11.3 scrub ok 2024-09-21T12:37:17.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:17 smithi183 ceph-mon[33277]: pgmap v864: 178 pgs: 1 active+clean+remapped+scrubbing, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 7.6 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 913 op/s; 31215/376443 objects degraded (8.292%); 15686/376443 objects misplaced (4.167%) 2024-09-21T12:37:17.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:17 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3334338737' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:37:17.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:17 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:37:17.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:17 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:37:17.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:17 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:37:17.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:17 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 31215/376443 objects degraded (8.292%), 13 pgs degraded, 33 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:17.514 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 84 16 131263 131247 97.6378 80.9375 0.010922 0.0102377 2024-09-21T12:37:17.569 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:37:17.570 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:17.604 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:17.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:17.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:17.698 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:37:17.705 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:17.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:17.749 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:17.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:17.850 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:17.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:17.889 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:17.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:17.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:37:17 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:37:17.990 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:18.032 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:18.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:18.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:18.117 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:37:18.134 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:18.169 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:18.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:18.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:18.271 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:18.294 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:18 smithi114 ceph-mon[30324]: 1.7 scrub starts 2024-09-21T12:37:18.294 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:18 smithi114 ceph-mon[30324]: 1.7 scrub ok 2024-09-21T12:37:18.294 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:18 smithi114 ceph-mon[30324]: 11.7 scrub ok 2024-09-21T12:37:18.294 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:18 smithi114 ceph-mon[30324]: 1.1d scrub starts 2024-09-21T12:37:18.294 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:18 smithi114 ceph-mon[30324]: 1.1d scrub ok 2024-09-21T12:37:18.294 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:18 smithi114 ceph-mon[30324]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:37:18.295 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:18 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:37:18.295 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:18 smithi114 ceph-mon[30324]: osdmap e439: 8 total, 7 up, 8 in 2024-09-21T12:37:18.295 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:18 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:37:18.295 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:18 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:37:18.295 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:18 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:37:18.295 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:18 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:37:18.295 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:18 smithi114 ceph-mon[40867]: 1.7 scrub starts 2024-09-21T12:37:18.295 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:18 smithi114 ceph-mon[40867]: 1.7 scrub ok 2024-09-21T12:37:18.295 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:18 smithi114 ceph-mon[40867]: 11.7 scrub ok 2024-09-21T12:37:18.295 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:18 smithi114 ceph-mon[40867]: 1.1d scrub starts 2024-09-21T12:37:18.295 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:18 smithi114 ceph-mon[40867]: 1.1d scrub ok 2024-09-21T12:37:18.295 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:18 smithi114 ceph-mon[40867]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:37:18.295 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:18 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:37:18.296 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:18 smithi114 ceph-mon[40867]: osdmap e439: 8 total, 7 up, 8 in 2024-09-21T12:37:18.296 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:18 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:37:18.296 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:18 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:37:18.296 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:18 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:37:18.296 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:18 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:37:18.308 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:18 smithi183 ceph-mon[33277]: 1.7 scrub starts 2024-09-21T12:37:18.308 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:18 smithi183 ceph-mon[33277]: 1.7 scrub ok 2024-09-21T12:37:18.308 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:18 smithi183 ceph-mon[33277]: 11.7 scrub ok 2024-09-21T12:37:18.308 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:18 smithi183 ceph-mon[33277]: 1.1d scrub starts 2024-09-21T12:37:18.308 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:18 smithi183 ceph-mon[33277]: 1.1d scrub ok 2024-09-21T12:37:18.309 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:18 smithi183 ceph-mon[33277]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:37:18.309 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:18 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:37:18.309 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:18 smithi183 ceph-mon[33277]: osdmap e439: 8 total, 7 up, 8 in 2024-09-21T12:37:18.309 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:18 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:37:18.309 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:18 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:37:18.309 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:18 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:37:18.309 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:18 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:37:18.315 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:18.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:18.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:18.349 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:37:18.349 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:37:18.417 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:18.424 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:37:18.447 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:18.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:18.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:18.514 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 85 16 132678 132662 97.5295 88.4375 0.0107136 0.0102493 2024-09-21T12:37:18.549 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:18.587 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:18.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:18.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:18.691 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:18.729 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:18.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:18.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:18.831 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:18.860 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:18.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:18.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:18.962 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:18.996 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:18.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:18.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:19.094 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T12:37:19.098 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:19.181 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:19.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:19.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:19.283 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:19.322 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:19.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:19.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:19.380 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:37:19.380 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:37:19.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:19 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:37:19.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:19 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:37:19.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:19 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:37:19.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:19 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:37:19.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:19 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:37:19.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:19 smithi114 ceph-mon[30324]: pgmap v866: 178 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped+scrubbing, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 124 active+clean; 8.0 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 131 B/s rd, 98 MiB/s wr, 1.57k op/s; 32265/392556 objects degraded (8.219%); 16428/392556 objects misplaced (4.185%) 2024-09-21T12:37:19.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:19 smithi114 ceph-mon[30324]: 4.a scrub starts 2024-09-21T12:37:19.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:19 smithi114 ceph-mon[30324]: 4.a scrub ok 2024-09-21T12:37:19.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:19 smithi114 ceph-mon[30324]: 11.4 scrub ok 2024-09-21T12:37:19.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:19 smithi114 ceph-mon[30324]: osdmap e440: 8 total, 7 up, 8 in 2024-09-21T12:37:19.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:19 smithi114 ceph-mon[30324]: 3.13 scrub starts 2024-09-21T12:37:19.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:19 smithi114 ceph-mon[30324]: 3.13 scrub ok 2024-09-21T12:37:19.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:19 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2643281061' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.890945750752006} : dispatch 2024-09-21T12:37:19.382 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:19 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:37:19.382 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:19 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:37:19.382 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:19 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:37:19.382 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:19 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:37:19.382 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:19 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:37:19.382 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:19 smithi114 ceph-mon[40867]: pgmap v866: 178 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped+scrubbing, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 124 active+clean; 8.0 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 131 B/s rd, 98 MiB/s wr, 1.57k op/s; 32265/392556 objects degraded (8.219%); 16428/392556 objects misplaced (4.185%) 2024-09-21T12:37:19.382 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:19 smithi114 ceph-mon[40867]: 4.a scrub starts 2024-09-21T12:37:19.382 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:19 smithi114 ceph-mon[40867]: 4.a scrub ok 2024-09-21T12:37:19.382 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:19 smithi114 ceph-mon[40867]: 11.4 scrub ok 2024-09-21T12:37:19.382 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:19 smithi114 ceph-mon[40867]: osdmap e440: 8 total, 7 up, 8 in 2024-09-21T12:37:19.382 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:19 smithi114 ceph-mon[40867]: 3.13 scrub starts 2024-09-21T12:37:19.383 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:19 smithi114 ceph-mon[40867]: 3.13 scrub ok 2024-09-21T12:37:19.383 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:19 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2643281061' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.890945750752006} : dispatch 2024-09-21T12:37:19.415 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:37:19.415 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:37:19.415 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T12:37:19.424 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:19.453 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:37:19.453 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:37:19.454 INFO:teuthology.orchestra.run.smithi114.stderr:set osd.5 primary-affinity to 0.890946 (8583892) 2024-09-21T12:37:19.459 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:19.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:19.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:19 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:37:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:19 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:37:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:19 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:37:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:19 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:37:19.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:19 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:37:19.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:19 smithi183 ceph-mon[33277]: pgmap v866: 178 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped+scrubbing, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 124 active+clean; 8.0 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 131 B/s rd, 98 MiB/s wr, 1.57k op/s; 32265/392556 objects degraded (8.219%); 16428/392556 objects misplaced (4.185%) 2024-09-21T12:37:19.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:19 smithi183 ceph-mon[33277]: 4.a scrub starts 2024-09-21T12:37:19.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:19 smithi183 ceph-mon[33277]: 4.a scrub ok 2024-09-21T12:37:19.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:19 smithi183 ceph-mon[33277]: 11.4 scrub ok 2024-09-21T12:37:19.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:19 smithi183 ceph-mon[33277]: osdmap e440: 8 total, 7 up, 8 in 2024-09-21T12:37:19.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:19 smithi183 ceph-mon[33277]: 3.13 scrub starts 2024-09-21T12:37:19.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:19 smithi183 ceph-mon[33277]: 3.13 scrub ok 2024-09-21T12:37:19.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:19 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2643281061' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.890945750752006} : dispatch 2024-09-21T12:37:19.491 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:37:19.495 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:37:19.495 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:37:19.514 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 86 16 134116 134100 97.4404 89.875 0.00709147 0.010257 2024-09-21T12:37:19.527 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:37:19.527 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:37:19.555 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:37:19.555 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:37:19.561 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:19.578 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:37:19.578 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:37:19.587 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:19.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:19.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:19.689 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:19.711 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:19.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:19.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:19.812 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:19.834 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:19.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:19.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:19.936 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:19.957 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:19.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:19.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:20.059 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:20.078 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T12:37:20.080 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:20.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:20.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:20.182 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:20.203 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:20.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:20.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:20.305 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:20.326 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:20.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:20.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:20.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:20 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1270738820' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:37:20.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:20 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2643281061' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.890945750752006}]': finished 2024-09-21T12:37:20.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:20 smithi114 ceph-mon[30324]: osdmap e441: 8 total, 7 up, 8 in 2024-09-21T12:37:20.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:20 smithi114 ceph-mon[30324]: 1.1c scrub starts 2024-09-21T12:37:20.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:20 smithi114 ceph-mon[30324]: 1.1c scrub ok 2024-09-21T12:37:20.382 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:20 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1270738820' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:37:20.382 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:20 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2643281061' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.890945750752006}]': finished 2024-09-21T12:37:20.383 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:20 smithi114 ceph-mon[40867]: osdmap e441: 8 total, 7 up, 8 in 2024-09-21T12:37:20.383 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:20 smithi114 ceph-mon[40867]: 1.1c scrub starts 2024-09-21T12:37:20.383 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:20 smithi114 ceph-mon[40867]: 1.1c scrub ok 2024-09-21T12:37:20.427 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:20.449 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:20.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:20.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:20 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1270738820' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:37:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:20 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2643281061' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.890945750752006}]': finished 2024-09-21T12:37:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:20 smithi183 ceph-mon[33277]: osdmap e441: 8 total, 7 up, 8 in 2024-09-21T12:37:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:20 smithi183 ceph-mon[33277]: 1.1c scrub starts 2024-09-21T12:37:20.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:20 smithi183 ceph-mon[33277]: 1.1c scrub ok 2024-09-21T12:37:20.514 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 87 16 134246 134230 96.4139 8.125 0.918979 0.0103683 2024-09-21T12:37:20.551 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:20.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:20.585 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:20.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:20.586 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:37:20.586 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-21T12:37:20.586 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-21T12:37:20.587 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:37:20.587 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.587 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:136741 9.f 9:f16b8898:::benchmark_data_smithi114_133214_object12707:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-09-21T12:37:20.587 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:30:12.044643+0000", 2024-09-21T12:37:20.587 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 428.10968947499998, 2024-09-21T12:37:20.587 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15204224699999999, 2024-09-21T12:37:20.587 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:20.587 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:20.587 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:20.587 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:20.587 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:37:20.588 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:37:20.588 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 136741 2024-09-21T12:37:20.588 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.588 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:20.588 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.588 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:20.588 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.044643+0000", 2024-09-21T12:37:20.588 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.588 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.588 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.588 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:20.588 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.044643+0000", 2024-09-21T12:37:20.588 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.588 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.589 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.589 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:20.589 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.044644+0000", 2024-09-21T12:37:20.589 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.043e-06 2024-09-21T12:37:20.589 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.589 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.589 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:20.589 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.044647+0000", 2024-09-21T12:37:20.589 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4199999999999999e-06 2024-09-21T12:37:20.589 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.589 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.589 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:20.589 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.044655+0000", 2024-09-21T12:37:20.590 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3649999999999998e-06 2024-09-21T12:37:20.590 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.590 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.590 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:20.590 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.044659+0000", 2024-09-21T12:37:20.590 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5800000000000002e-06 2024-09-21T12:37:20.590 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.590 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.590 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:20.590 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.044697+0000", 2024-09-21T12:37:20.590 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7499000000000001e-05 2024-09-21T12:37:20.590 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.590 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.590 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:20.590 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.044810+0000", 2024-09-21T12:37:20.591 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000113287 2024-09-21T12:37:20.591 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.591 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.591 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:37:20.591 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.044925+0000", 2024-09-21T12:37:20.591 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000115343 2024-09-21T12:37:20.591 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.591 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.591 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:20.591 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.046578+0000", 2024-09-21T12:37:20.591 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0016527989999999999 2024-09-21T12:37:20.591 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.591 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.591 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.592 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.154593+0000", 2024-09-21T12:37:20.592 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.108015135 2024-09-21T12:37:20.592 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.592 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.592 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.592 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.196609+0000", 2024-09-21T12:37:20.592 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.042015706999999999 2024-09-21T12:37:20.592 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.592 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.592 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:20.592 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.196649+0000", 2024-09-21T12:37:20.592 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9935999999999998e-05 2024-09-21T12:37:20.592 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.592 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.592 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:20.593 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:12.196685+0000", 2024-09-21T12:37:20.593 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6133000000000001e-05 2024-09-21T12:37:20.593 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.593 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:20.593 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.593 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.593 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.593 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:143984 9.f 9:fd48ee49:::benchmark_data_smithi114_133214_object19950:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-09-21T12:37:20.593 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:30:20.026313+0000", 2024-09-21T12:37:20.593 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 420.12801966500001, 2024-09-21T12:37:20.593 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.33116894200000002, 2024-09-21T12:37:20.593 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:20.593 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:20.593 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:20.593 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:20.594 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:37:20.594 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:37:20.594 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 143984 2024-09-21T12:37:20.594 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.594 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:20.594 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.594 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:20.594 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026313+0000", 2024-09-21T12:37:20.594 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.594 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.594 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.594 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:20.594 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026313+0000", 2024-09-21T12:37:20.594 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.595 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.595 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.595 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:20.595 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026314+0000", 2024-09-21T12:37:20.595 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.186e-06 2024-09-21T12:37:20.595 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.595 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.595 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:20.595 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026317+0000", 2024-09-21T12:37:20.595 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0989999999999999e-06 2024-09-21T12:37:20.595 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.595 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.595 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:20.596 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026329+0000", 2024-09-21T12:37:20.596 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1932e-05 2024-09-21T12:37:20.596 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.596 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.596 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:20.596 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026333+0000", 2024-09-21T12:37:20.596 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3370000000000001e-06 2024-09-21T12:37:20.596 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.596 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.596 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:20.596 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026364+0000", 2024-09-21T12:37:20.596 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0235000000000001e-05 2024-09-21T12:37:20.596 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.597 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.597 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:20.597 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026441+0000", 2024-09-21T12:37:20.597 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7663000000000005e-05 2024-09-21T12:37:20.597 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.597 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.597 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:37:20.597 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026511+0000", 2024-09-21T12:37:20.597 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9640000000000004e-05 2024-09-21T12:37:20.597 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.597 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.597 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:20.597 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.029298+0000", 2024-09-21T12:37:20.597 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0027875500000000002 2024-09-21T12:37:20.598 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.598 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.598 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.598 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.039048+0000", 2024-09-21T12:37:20.598 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0097491490000000004 2024-09-21T12:37:20.598 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.598 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.598 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.598 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.357393+0000", 2024-09-21T12:37:20.598 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31834596300000001 2024-09-21T12:37:20.598 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.598 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.598 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:20.598 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.357435+0000", 2024-09-21T12:37:20.599 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1078999999999998e-05 2024-09-21T12:37:20.599 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.599 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.599 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:20.599 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.357482+0000", 2024-09-21T12:37:20.599 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7108999999999999e-05 2024-09-21T12:37:20.599 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.599 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:20.599 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.599 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.599 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.599 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:143985 9.f 9:f12d363a:::benchmark_data_smithi114_133214_object19951:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-09-21T12:37:20.599 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:30:20.026340+0000", 2024-09-21T12:37:20.599 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 420.12799202000002, 2024-09-21T12:37:20.600 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.33955830100000001, 2024-09-21T12:37:20.600 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:20.600 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:20.600 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:20.600 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:20.600 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:37:20.600 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:37:20.600 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 143985 2024-09-21T12:37:20.600 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.600 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:20.600 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.600 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:20.600 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026340+0000", 2024-09-21T12:37:20.600 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.600 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.601 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.601 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:20.601 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026340+0000", 2024-09-21T12:37:20.601 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.601 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.601 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.601 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:20.601 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026341+0000", 2024-09-21T12:37:20.601 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.84e-07 2024-09-21T12:37:20.601 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.601 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.601 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:20.601 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026342+0000", 2024-09-21T12:37:20.602 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.671e-06 2024-09-21T12:37:20.602 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.602 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.602 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:20.602 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026346+0000", 2024-09-21T12:37:20.602 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5769999999999998e-06 2024-09-21T12:37:20.602 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.602 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.602 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:20.602 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.026350+0000", 2024-09-21T12:37:20.602 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8029999999999998e-06 2024-09-21T12:37:20.602 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.602 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.602 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:20.603 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.027962+0000", 2024-09-21T12:37:20.603 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0016119839999999999 2024-09-21T12:37:20.603 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.603 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.603 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:20.603 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.028067+0000", 2024-09-21T12:37:20.603 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010498299999999999 2024-09-21T12:37:20.603 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.603 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.603 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:37:20.603 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.028160+0000", 2024-09-21T12:37:20.603 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3731000000000001e-05 2024-09-21T12:37:20.603 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.604 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.604 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:20.604 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.030647+0000", 2024-09-21T12:37:20.604 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024862310000000002 2024-09-21T12:37:20.604 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.604 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.604 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.604 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.039088+0000", 2024-09-21T12:37:20.604 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0084414620000000003 2024-09-21T12:37:20.604 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.604 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.604 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.604 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.365834+0000", 2024-09-21T12:37:20.604 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.32674630799999999 2024-09-21T12:37:20.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.605 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.605 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:20.605 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.365865+0000", 2024-09-21T12:37:20.605 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0732999999999999e-05 2024-09-21T12:37:20.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.605 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.605 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:20.605 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.365899+0000", 2024-09-21T12:37:20.605 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3534000000000001e-05 2024-09-21T12:37:20.605 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.605 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:20.605 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.606 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.606 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:143986 9.f 9:f1d8e647:::benchmark_data_smithi114_133214_object19952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-09-21T12:37:20.606 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:30:20.027267+0000", 2024-09-21T12:37:20.606 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 420.12706528400003, 2024-09-21T12:37:20.606 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.34414182799999998, 2024-09-21T12:37:20.606 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:20.606 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:20.606 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:20.606 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:20.606 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:37:20.606 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:37:20.606 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 143986 2024-09-21T12:37:20.606 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.606 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:20.607 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.607 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:20.607 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.027267+0000", 2024-09-21T12:37:20.607 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.607 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.607 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:20.607 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.027267+0000", 2024-09-21T12:37:20.607 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.607 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.607 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:20.607 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.027268+0000", 2024-09-21T12:37:20.607 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.065e-06 2024-09-21T12:37:20.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.608 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:20.608 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.027272+0000", 2024-09-21T12:37:20.608 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.318e-06 2024-09-21T12:37:20.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.608 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:20.608 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.027282+0000", 2024-09-21T12:37:20.608 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4490000000000005e-06 2024-09-21T12:37:20.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.608 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:20.608 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.027287+0000", 2024-09-21T12:37:20.608 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9019999999999998e-06 2024-09-21T12:37:20.609 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.609 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.609 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:20.609 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.029320+0000", 2024-09-21T12:37:20.609 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020333090000000001 2024-09-21T12:37:20.609 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.609 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.609 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:20.609 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.029383+0000", 2024-09-21T12:37:20.609 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3274000000000004e-05 2024-09-21T12:37:20.609 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.609 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.609 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:37:20.609 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.029450+0000", 2024-09-21T12:37:20.609 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6649999999999994e-05 2024-09-21T12:37:20.610 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.610 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.610 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:20.610 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.030775+0000", 2024-09-21T12:37:20.610 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001325346 2024-09-21T12:37:20.610 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.610 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.610 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.610 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.039124+0000", 2024-09-21T12:37:20.610 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0083483350000000001 2024-09-21T12:37:20.610 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.610 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.610 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.610 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.371309+0000", 2024-09-21T12:37:20.611 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.33218562400000001 2024-09-21T12:37:20.611 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.611 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.611 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:20.611 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.371364+0000", 2024-09-21T12:37:20.611 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4336999999999998e-05 2024-09-21T12:37:20.611 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.611 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.611 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:20.611 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:20.371409+0000", 2024-09-21T12:37:20.611 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5219000000000002e-05 2024-09-21T12:37:20.611 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.611 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:20.611 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.611 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.612 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.612 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:148258 9.f 9:fa68205d:::benchmark_data_smithi114_133214_object24224:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-09-21T12:37:20.612 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:30:25.552566+0000", 2024-09-21T12:37:20.612 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 414.601766274, 2024-09-21T12:37:20.612 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.165969116, 2024-09-21T12:37:20.612 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:20.612 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:20.612 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:20.612 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:20.612 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:37:20.612 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:37:20.612 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 148258 2024-09-21T12:37:20.612 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.612 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:20.612 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.613 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:20.613 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.552566+0000", 2024-09-21T12:37:20.613 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.613 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.613 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.613 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:20.613 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.552566+0000", 2024-09-21T12:37:20.613 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.613 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.613 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.613 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:20.613 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.552568+0000", 2024-09-21T12:37:20.613 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.406e-06 2024-09-21T12:37:20.613 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.613 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.614 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:20.614 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.552573+0000", 2024-09-21T12:37:20.614 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0030000000000002e-06 2024-09-21T12:37:20.614 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.614 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.614 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:20.614 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.552583+0000", 2024-09-21T12:37:20.614 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0379e-05 2024-09-21T12:37:20.614 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.614 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.614 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:20.614 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.552589+0000", 2024-09-21T12:37:20.615 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2480000000000001e-06 2024-09-21T12:37:20.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.615 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.615 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:20.615 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.552647+0000", 2024-09-21T12:37:20.615 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7422e-05 2024-09-21T12:37:20.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.615 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.615 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:20.615 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.552765+0000", 2024-09-21T12:37:20.615 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000118627 2024-09-21T12:37:20.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.615 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.615 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:37:20.616 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.552887+0000", 2024-09-21T12:37:20.616 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000121557 2024-09-21T12:37:20.616 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.616 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.616 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:20.616 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.554567+0000", 2024-09-21T12:37:20.616 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0016804249999999999 2024-09-21T12:37:20.616 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.616 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.616 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.616 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.569714+0000", 2024-09-21T12:37:20.616 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.015146980000000001 2024-09-21T12:37:20.616 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.616 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.617 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.617 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.718424+0000", 2024-09-21T12:37:20.617 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14870934699999999 2024-09-21T12:37:20.617 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.617 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.617 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:20.617 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.718488+0000", 2024-09-21T12:37:20.617 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4362000000000005e-05 2024-09-21T12:37:20.617 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.617 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.617 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:20.617 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.718535+0000", 2024-09-21T12:37:20.617 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7360000000000001e-05 2024-09-21T12:37:20.618 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.618 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:20.618 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.618 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.618 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.618 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:148262 9.f 9:f36a2eaf:::benchmark_data_smithi114_133214_object24228:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-09-21T12:37:20.618 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:30:25.564038+0000", 2024-09-21T12:37:20.618 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 414.590294465, 2024-09-21T12:37:20.618 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.154638887, 2024-09-21T12:37:20.618 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:20.618 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:20.618 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:20.618 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:20.619 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:37:20.619 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:37:20.619 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 148262 2024-09-21T12:37:20.619 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.619 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:20.619 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.619 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:20.619 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.564038+0000", 2024-09-21T12:37:20.619 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.619 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.619 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.619 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:20.619 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.564038+0000", 2024-09-21T12:37:20.619 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.620 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.620 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.620 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:20.620 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.564039+0000", 2024-09-21T12:37:20.620 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.144e-06 2024-09-21T12:37:20.620 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.620 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.620 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:20.620 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.564043+0000", 2024-09-21T12:37:20.620 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.25e-06 2024-09-21T12:37:20.620 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.620 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.620 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:20.620 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.564054+0000", 2024-09-21T12:37:20.621 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0963e-05 2024-09-21T12:37:20.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.621 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:20.621 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.564060+0000", 2024-09-21T12:37:20.621 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.981e-06 2024-09-21T12:37:20.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.621 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:20.621 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.564107+0000", 2024-09-21T12:37:20.621 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6758999999999997e-05 2024-09-21T12:37:20.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.621 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:20.621 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.564242+0000", 2024-09-21T12:37:20.622 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013536700000000001 2024-09-21T12:37:20.622 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.622 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.622 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:37:20.622 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.564364+0000", 2024-09-21T12:37:20.622 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000122062 2024-09-21T12:37:20.622 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.622 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.622 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:20.622 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.566033+0000", 2024-09-21T12:37:20.622 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001668444 2024-09-21T12:37:20.622 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.622 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.623 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.623 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.569752+0000", 2024-09-21T12:37:20.623 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0037187560000000001 2024-09-21T12:37:20.623 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.623 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.623 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.623 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.718589+0000", 2024-09-21T12:37:20.623 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14883724800000001 2024-09-21T12:37:20.623 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.623 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.623 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:20.623 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.718637+0000", 2024-09-21T12:37:20.623 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7763000000000003e-05 2024-09-21T12:37:20.623 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.623 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.624 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:20.624 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.718677+0000", 2024-09-21T12:37:20.624 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.015e-05 2024-09-21T12:37:20.624 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.624 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:20.624 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.624 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:148264 9.f 9:f2a805cb:::benchmark_data_smithi114_133214_object24230:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-09-21T12:37:20.624 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:30:25.566769+0000", 2024-09-21T12:37:20.624 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 414.58756384700001, 2024-09-21T12:37:20.624 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15200875599999999, 2024-09-21T12:37:20.624 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:20.624 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:20.624 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:20.625 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:20.625 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:37:20.625 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:37:20.625 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 148264 2024-09-21T12:37:20.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.625 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:20.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:20.625 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.566769+0000", 2024-09-21T12:37:20.625 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:20.626 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.566769+0000", 2024-09-21T12:37:20.626 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.626 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:20.626 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.566770+0000", 2024-09-21T12:37:20.626 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.043e-06 2024-09-21T12:37:20.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.626 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:20.626 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.566773+0000", 2024-09-21T12:37:20.626 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8179999999999997e-06 2024-09-21T12:37:20.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.627 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:20.627 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.566783+0000", 2024-09-21T12:37:20.627 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4299999999999995e-06 2024-09-21T12:37:20.627 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.627 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.627 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:20.627 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.566794+0000", 2024-09-21T12:37:20.627 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0910999999999999e-05 2024-09-21T12:37:20.627 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.627 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.627 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:20.627 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.566853+0000", 2024-09-21T12:37:20.627 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8943999999999997e-05 2024-09-21T12:37:20.627 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.628 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.628 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:20.628 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.566971+0000", 2024-09-21T12:37:20.628 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000118113 2024-09-21T12:37:20.628 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.628 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.628 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:37:20.628 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.567091+0000", 2024-09-21T12:37:20.628 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000120232 2024-09-21T12:37:20.628 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.628 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.628 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:20.628 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.568779+0000", 2024-09-21T12:37:20.628 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001688134 2024-09-21T12:37:20.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.629 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.629 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.629 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.569859+0000", 2024-09-21T12:37:20.629 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010798330000000001 2024-09-21T12:37:20.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.629 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.629 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.629 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.718708+0000", 2024-09-21T12:37:20.629 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14884867800000001 2024-09-21T12:37:20.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.629 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.629 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:20.629 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.718752+0000", 2024-09-21T12:37:20.630 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4379999999999999e-05 2024-09-21T12:37:20.630 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.630 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.630 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:20.630 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:25.718777+0000", 2024-09-21T12:37:20.630 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5239999999999999e-05 2024-09-21T12:37:20.630 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.630 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:20.630 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.630 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.630 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.630 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:175495 9.f 9:f9909850:::benchmark_data_smithi114_133214_object51461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-21T12:37:20.630 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:30:56.789850+0000", 2024-09-21T12:37:20.631 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 383.36448189100003, 2024-09-21T12:37:20.631 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.16364801100000001, 2024-09-21T12:37:20.631 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:20.631 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:20.631 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:20.631 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:20.631 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:37:20.631 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:37:20.631 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 175495 2024-09-21T12:37:20.631 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.631 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:20.631 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.631 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:20.631 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.789850+0000", 2024-09-21T12:37:20.632 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.632 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.632 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.632 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:20.632 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.789850+0000", 2024-09-21T12:37:20.632 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.632 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.632 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.632 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:20.632 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.789852+0000", 2024-09-21T12:37:20.632 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.573e-06 2024-09-21T12:37:20.632 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.632 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.632 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:20.632 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.789857+0000", 2024-09-21T12:37:20.633 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0209999999999999e-06 2024-09-21T12:37:20.633 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.633 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.633 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:20.633 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.789868+0000", 2024-09-21T12:37:20.633 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0496000000000001e-05 2024-09-21T12:37:20.633 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.633 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.633 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:20.633 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.789874+0000", 2024-09-21T12:37:20.633 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7599999999999997e-06 2024-09-21T12:37:20.633 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.633 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.634 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:20.634 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.789923+0000", 2024-09-21T12:37:20.634 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8188e-05 2024-09-21T12:37:20.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.634 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.634 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:20.634 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.790047+0000", 2024-09-21T12:37:20.634 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012412999999999999 2024-09-21T12:37:20.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.634 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.634 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:37:20.634 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.790161+0000", 2024-09-21T12:37:20.634 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011474800000000001 2024-09-21T12:37:20.635 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.635 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.635 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:20.635 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.791943+0000", 2024-09-21T12:37:20.635 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0017820710000000001 2024-09-21T12:37:20.635 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.635 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.635 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.635 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.943029+0000", 2024-09-21T12:37:20.635 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15108565800000001 2024-09-21T12:37:20.635 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.635 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.635 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.635 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.953435+0000", 2024-09-21T12:37:20.636 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.010406227000000001 2024-09-21T12:37:20.636 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.636 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:20.636 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.953469+0000", 2024-09-21T12:37:20.637 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3969e-05 2024-09-21T12:37:20.637 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.637 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.637 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:20.638 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:30:56.953499+0000", 2024-09-21T12:37:20.638 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.917e-05 2024-09-21T12:37:20.638 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.638 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:20.638 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.638 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.638 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.638 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:216505 9.f 9:f5ab0cee:::benchmark_data_smithi114_133214_object92471:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e306)", 2024-09-21T12:37:20.638 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:33.894635+0000", 2024-09-21T12:37:20.638 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 346.25969700600001, 2024-09-21T12:37:20.638 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.38916874499999998, 2024-09-21T12:37:20.638 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:20.638 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:20.639 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:20.639 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:20.639 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:37:20.639 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:37:20.639 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 216505 2024-09-21T12:37:20.639 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.639 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:20.639 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.639 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:20.639 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.894635+0000", 2024-09-21T12:37:20.639 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.639 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.639 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.639 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:20.639 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.894635+0000", 2024-09-21T12:37:20.640 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.640 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:20.640 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.894637+0000", 2024-09-21T12:37:20.640 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.654e-06 2024-09-21T12:37:20.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.640 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:20.640 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.894642+0000", 2024-09-21T12:37:20.640 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0390000000000004e-06 2024-09-21T12:37:20.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.640 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:20.640 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.894658+0000", 2024-09-21T12:37:20.641 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6379999999999999e-05 2024-09-21T12:37:20.641 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.641 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.641 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:20.641 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.894666+0000", 2024-09-21T12:37:20.641 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0550000000000002e-06 2024-09-21T12:37:20.641 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.641 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.641 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:20.641 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.894723+0000", 2024-09-21T12:37:20.641 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7982000000000002e-05 2024-09-21T12:37:20.641 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.641 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.641 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:20.641 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.894802+0000", 2024-09-21T12:37:20.642 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8399000000000006e-05 2024-09-21T12:37:20.642 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.642 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.642 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:37:20.642 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.894874+0000", 2024-09-21T12:37:20.642 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2423999999999996e-05 2024-09-21T12:37:20.642 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.642 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.642 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:20.642 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.896880+0000", 2024-09-21T12:37:20.642 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020054180000000001 2024-09-21T12:37:20.642 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.642 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.642 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.643 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.900439+0000", 2024-09-21T12:37:20.643 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0035589409999999999 2024-09-21T12:37:20.643 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.643 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.643 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.643 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:34.283682+0000", 2024-09-21T12:37:20.643 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.38324365500000002 2024-09-21T12:37:20.643 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.643 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.643 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:20.643 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:34.283755+0000", 2024-09-21T12:37:20.643 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2329000000000005e-05 2024-09-21T12:37:20.643 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.644 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.644 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:20.644 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:34.283804+0000", 2024-09-21T12:37:20.644 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9468999999999997e-05 2024-09-21T12:37:20.644 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.644 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:20.644 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.644 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.644 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.644 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:216514 9.f 9:f58474dd:::benchmark_data_smithi114_133214_object92480:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e306)", 2024-09-21T12:37:20.644 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:33.900257+0000", 2024-09-21T12:37:20.644 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 346.25407569999999, 2024-09-21T12:37:20.644 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.38852348399999997, 2024-09-21T12:37:20.644 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:20.645 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:20.645 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:20.645 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:20.645 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:37:20.645 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:37:20.645 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 216514 2024-09-21T12:37:20.645 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.645 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:20.645 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.645 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:20.645 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.900257+0000", 2024-09-21T12:37:20.645 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.645 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.645 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.646 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:20.646 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.900257+0000", 2024-09-21T12:37:20.646 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.646 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.646 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.646 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:20.646 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.900258+0000", 2024-09-21T12:37:20.646 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.57e-06 2024-09-21T12:37:20.646 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.646 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.646 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:20.646 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.900264+0000", 2024-09-21T12:37:20.646 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2979999999999999e-06 2024-09-21T12:37:20.646 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:20.647 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.900279+0000", 2024-09-21T12:37:20.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5211000000000001e-05 2024-09-21T12:37:20.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:20.647 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.900286+0000", 2024-09-21T12:37:20.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2089999999999999e-06 2024-09-21T12:37:20.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:20.647 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.900470+0000", 2024-09-21T12:37:20.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00018402699999999999 2024-09-21T12:37:20.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:20.648 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.900607+0000", 2024-09-21T12:37:20.648 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013691800000000001 2024-09-21T12:37:20.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:37:20.648 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.900732+0000", 2024-09-21T12:37:20.648 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000125081 2024-09-21T12:37:20.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:20.649 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.902574+0000", 2024-09-21T12:37:20.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0018421290000000001 2024-09-21T12:37:20.649 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.649 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.649 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:33.903091+0000", 2024-09-21T12:37:20.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000517214 2024-09-21T12:37:20.649 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.649 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.649 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:34.288673+0000", 2024-09-21T12:37:20.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.38558214000000002 2024-09-21T12:37:20.649 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.650 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:20.650 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:34.288738+0000", 2024-09-21T12:37:20.650 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4135999999999997e-05 2024-09-21T12:37:20.650 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.650 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.650 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:20.650 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:34.288780+0000", 2024-09-21T12:37:20.650 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2551e-05 2024-09-21T12:37:20.650 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.650 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:20.650 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.650 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.651 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.651 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:231697 9.19 9:9abf7afe:::benchmark_data_smithi114_133214_object107663:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:37:20.651 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:50.030279+0000", 2024-09-21T12:37:20.651 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 330.12405327200003, 2024-09-21T12:37:20.651 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1184929130000001, 2024-09-21T12:37:20.651 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:20.651 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:20.651 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:20.651 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:20.651 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:37:20.651 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:37:20.651 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 231697 2024-09-21T12:37:20.651 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.651 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:20.652 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.652 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:20.652 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030279+0000", 2024-09-21T12:37:20.652 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.652 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.652 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.652 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:20.652 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030279+0000", 2024-09-21T12:37:20.652 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.652 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.652 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.652 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:20.653 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030281+0000", 2024-09-21T12:37:20.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.689e-06 2024-09-21T12:37:20.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.653 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.653 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:20.653 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030285+0000", 2024-09-21T12:37:20.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9859999999999996e-06 2024-09-21T12:37:20.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.653 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.653 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:20.653 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030295+0000", 2024-09-21T12:37:20.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0178e-05 2024-09-21T12:37:20.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.653 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.653 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:20.654 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030305+0000", 2024-09-21T12:37:20.654 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7860000000000008e-06 2024-09-21T12:37:20.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.654 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:20.654 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030481+0000", 2024-09-21T12:37:20.654 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000175768 2024-09-21T12:37:20.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.654 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:20.654 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030490+0000", 2024-09-21T12:37:20.654 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7669999999999998e-06 2024-09-21T12:37:20.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.655 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:20.655 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.144698+0000", 2024-09-21T12:37:20.655 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1142073050000001 2024-09-21T12:37:20.655 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.655 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.655 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:20.655 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.144803+0000", 2024-09-21T12:37:20.655 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010563100000000001 2024-09-21T12:37:20.655 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.655 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.655 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,6,7", 2024-09-21T12:37:20.655 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.145159+0000", 2024-09-21T12:37:20.655 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00035557900000000002 2024-09-21T12:37:20.655 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.656 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.656 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.656 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.148533+0000", 2024-09-21T12:37:20.656 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003373741 2024-09-21T12:37:20.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.656 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.656 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.656 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.148585+0000", 2024-09-21T12:37:20.656 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2278999999999999e-05 2024-09-21T12:37:20.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.656 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.656 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.656 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.148619+0000", 2024-09-21T12:37:20.656 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.417e-05 2024-09-21T12:37:20.657 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.657 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.657 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:20.657 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.148629+0000", 2024-09-21T12:37:20.657 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8779999999999993e-06 2024-09-21T12:37:20.657 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.657 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.657 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:20.657 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.148667+0000", 2024-09-21T12:37:20.657 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7728999999999997e-05 2024-09-21T12:37:20.657 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.657 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.657 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:20.657 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.148772+0000", 2024-09-21T12:37:20.657 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010542700000000001 2024-09-21T12:37:20.658 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.658 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:20.658 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.658 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.658 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.658 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:231705 9.19 9:9e8af994:::benchmark_data_smithi114_133214_object107671:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:37:20.658 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:50.030314+0000", 2024-09-21T12:37:20.658 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 330.12401797000001, 2024-09-21T12:37:20.658 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1228952539999999, 2024-09-21T12:37:20.658 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:20.658 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:20.658 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:20.658 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:20.658 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:37:20.659 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:37:20.659 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 231705 2024-09-21T12:37:20.659 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.659 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:20.659 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.659 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:20.659 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030314+0000", 2024-09-21T12:37:20.659 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.659 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.659 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.659 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:20.659 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030314+0000", 2024-09-21T12:37:20.659 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:20.660 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030315+0000", 2024-09-21T12:37:20.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9299999999999998e-07 2024-09-21T12:37:20.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:20.660 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030317+0000", 2024-09-21T12:37:20.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2450000000000001e-06 2024-09-21T12:37:20.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:20.660 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030323+0000", 2024-09-21T12:37:20.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8939999999999998e-06 2024-09-21T12:37:20.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.661 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:20.661 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030328+0000", 2024-09-21T12:37:20.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1800000000000004e-06 2024-09-21T12:37:20.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.661 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:20.661 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030512+0000", 2024-09-21T12:37:20.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00018370300000000001 2024-09-21T12:37:20.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:20.662 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030518+0000", 2024-09-21T12:37:20.662 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2319999999999998e-06 2024-09-21T12:37:20.662 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.662 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:20.662 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.148936+0000", 2024-09-21T12:37:20.662 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1184178929999999 2024-09-21T12:37:20.662 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.662 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:20.662 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.149021+0000", 2024-09-21T12:37:20.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4841999999999998e-05 2024-09-21T12:37:20.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.663 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.663 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,6,7", 2024-09-21T12:37:20.663 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.149286+0000", 2024-09-21T12:37:20.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00026551899999999998 2024-09-21T12:37:20.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.663 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.663 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.663 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153018+0000", 2024-09-21T12:37:20.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0037314340000000001 2024-09-21T12:37:20.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.663 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.663 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.664 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153057+0000", 2024-09-21T12:37:20.664 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9365e-05 2024-09-21T12:37:20.664 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.664 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.664 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:20.664 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153068+0000", 2024-09-21T12:37:20.664 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.11e-05 2024-09-21T12:37:20.664 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.664 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.664 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.664 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153107+0000", 2024-09-21T12:37:20.664 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8305e-05 2024-09-21T12:37:20.664 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.665 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.665 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:20.665 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153144+0000", 2024-09-21T12:37:20.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7110000000000002e-05 2024-09-21T12:37:20.665 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.665 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.665 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:20.665 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153210+0000", 2024-09-21T12:37:20.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5938999999999993e-05 2024-09-21T12:37:20.665 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.665 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:20.665 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.665 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.666 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:231765 9.19 9:9e977502:::benchmark_data_smithi114_133214_object107731:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:37:20.666 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:50.030334+0000", 2024-09-21T12:37:20.666 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 330.12399838099998, 2024-09-21T12:37:20.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.123057872, 2024-09-21T12:37:20.666 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:20.666 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:20.666 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:20.666 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:20.666 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:37:20.666 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:37:20.666 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 231765 2024-09-21T12:37:20.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.667 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:20.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:20.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030334+0000", 2024-09-21T12:37:20.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:20.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030334+0000", 2024-09-21T12:37:20.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:20.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030334+0000", 2024-09-21T12:37:20.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2300000000000002e-07 2024-09-21T12:37:20.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:20.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030337+0000", 2024-09-21T12:37:20.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0949999999999998e-06 2024-09-21T12:37:20.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:20.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030342+0000", 2024-09-21T12:37:20.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7710000000000003e-06 2024-09-21T12:37:20.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:20.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030347+0000", 2024-09-21T12:37:20.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.198e-06 2024-09-21T12:37:20.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:20.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030540+0000", 2024-09-21T12:37:20.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019217600000000001 2024-09-21T12:37:20.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:20.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:50.030546+0000", 2024-09-21T12:37:20.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.212e-06 2024-09-21T12:37:20.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:20.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.150826+0000", 2024-09-21T12:37:20.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.120280194 2024-09-21T12:37:20.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:20.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.150950+0000", 2024-09-21T12:37:20.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012343099999999999 2024-09-21T12:37:20.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,6,7", 2024-09-21T12:37:20.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.151239+0000", 2024-09-21T12:37:20.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00028982299999999998 2024-09-21T12:37:20.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:20.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153075+0000", 2024-09-21T12:37:20.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001835723 2024-09-21T12:37:20.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153243+0000", 2024-09-21T12:37:20.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000167973 2024-09-21T12:37:20.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153273+0000", 2024-09-21T12:37:20.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0176999999999999e-05 2024-09-21T12:37:20.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153296+0000", 2024-09-21T12:37:20.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2816e-05 2024-09-21T12:37:20.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:20.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153327+0000", 2024-09-21T12:37:20.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1031999999999997e-05 2024-09-21T12:37:20.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:20.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:51.153392+0000", 2024-09-21T12:37:20.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4827999999999997e-05 2024-09-21T12:37:20.672 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.673 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:20.673 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.673 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:235406 9.f 9:fff68ecc:::benchmark_data_smithi114_133214_object111372:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-09-21T12:37:20.673 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:56.217779+0000", 2024-09-21T12:37:20.673 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 323.936553178, 2024-09-21T12:37:20.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15964376999999999, 2024-09-21T12:37:20.673 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:20.673 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:20.673 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:20.673 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:20.673 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:37:20.673 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:37:20.673 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 235406 2024-09-21T12:37:20.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.674 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:20.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:20.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.217779+0000", 2024-09-21T12:37:20.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:20.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.217779+0000", 2024-09-21T12:37:20.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:20.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.217780+0000", 2024-09-21T12:37:20.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8000000000000005e-07 2024-09-21T12:37:20.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:20.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.217784+0000", 2024-09-21T12:37:20.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7349999999999998e-06 2024-09-21T12:37:20.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:20.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.217795+0000", 2024-09-21T12:37:20.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0934000000000001e-05 2024-09-21T12:37:20.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:20.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.217801+0000", 2024-09-21T12:37:20.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9399999999999999e-06 2024-09-21T12:37:20.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:20.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.223367+0000", 2024-09-21T12:37:20.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0055666259999999999 2024-09-21T12:37:20.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:20.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.223468+0000", 2024-09-21T12:37:20.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100368 2024-09-21T12:37:20.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:37:20.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.223558+0000", 2024-09-21T12:37:20.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9945999999999996e-05 2024-09-21T12:37:20.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.225567+0000", 2024-09-21T12:37:20.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020092069999999998 2024-09-21T12:37:20.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:20.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.227739+0000", 2024-09-21T12:37:20.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021725160000000002 2024-09-21T12:37:20.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.377351+0000", 2024-09-21T12:37:20.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.149611789 2024-09-21T12:37:20.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:20.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.377385+0000", 2024-09-21T12:37:20.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3680000000000003e-05 2024-09-21T12:37:20.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:20.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.377423+0000", 2024-09-21T12:37:20.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8248999999999999e-05 2024-09-21T12:37:20.679 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.679 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:20.679 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.679 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:235409 9.f 9:f26d45b2:::benchmark_data_smithi114_133214_object111375:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-09-21T12:37:20.679 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:56.218056+0000", 2024-09-21T12:37:20.679 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 323.93627671000002, 2024-09-21T12:37:20.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.16153879900000001, 2024-09-21T12:37:20.679 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:20.680 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:20.680 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:20.680 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:20.680 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:37:20.680 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:37:20.680 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 235409 2024-09-21T12:37:20.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.680 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:20.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:20.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.218056+0000", 2024-09-21T12:37:20.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:20.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.218056+0000", 2024-09-21T12:37:20.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:20.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.218056+0000", 2024-09-21T12:37:20.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9999999999999996e-07 2024-09-21T12:37:20.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:20.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.218060+0000", 2024-09-21T12:37:20.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6820000000000001e-06 2024-09-21T12:37:20.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:20.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.218072+0000", 2024-09-21T12:37:20.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1511999999999999e-05 2024-09-21T12:37:20.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:20.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.218078+0000", 2024-09-21T12:37:20.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0260000000000004e-06 2024-09-21T12:37:20.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:20.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.225599+0000", 2024-09-21T12:37:20.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0075208879999999999 2024-09-21T12:37:20.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:20.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.225706+0000", 2024-09-21T12:37:20.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010780099999999999 2024-09-21T12:37:20.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:37:20.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.225831+0000", 2024-09-21T12:37:20.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012434100000000001 2024-09-21T12:37:20.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:20.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.227745+0000", 2024-09-21T12:37:20.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0019139090000000001 2024-09-21T12:37:20.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.228076+0000", 2024-09-21T12:37:20.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00033095599999999998 2024-09-21T12:37:20.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.379487+0000", 2024-09-21T12:37:20.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15141185500000001 2024-09-21T12:37:20.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:20.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.379551+0000", 2024-09-21T12:37:20.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3971000000000001e-05 2024-09-21T12:37:20.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:20.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.379594+0000", 2024-09-21T12:37:20.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3058e-05 2024-09-21T12:37:20.685 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.685 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:20.685 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.685 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:235410 9.f 9:fc9ff039:::benchmark_data_smithi114_133214_object111376:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-09-21T12:37:20.685 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:56.219546+0000", 2024-09-21T12:37:20.685 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 323.934786132, 2024-09-21T12:37:20.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.160211877, 2024-09-21T12:37:20.686 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:20.686 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:20.686 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:20.686 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:20.686 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:37:20.686 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:37:20.686 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 235410 2024-09-21T12:37:20.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.686 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:20.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:20.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.219546+0000", 2024-09-21T12:37:20.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:20.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.219546+0000", 2024-09-21T12:37:20.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:20.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.219547+0000", 2024-09-21T12:37:20.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1429999999999999e-06 2024-09-21T12:37:20.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:20.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.219552+0000", 2024-09-21T12:37:20.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3170000000000003e-06 2024-09-21T12:37:20.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:20.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.219561+0000", 2024-09-21T12:37:20.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2119999999999992e-06 2024-09-21T12:37:20.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:20.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.219568+0000", 2024-09-21T12:37:20.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.764e-06 2024-09-21T12:37:20.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:20.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.226862+0000", 2024-09-21T12:37:20.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0072939019999999997 2024-09-21T12:37:20.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:20.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.226927+0000", 2024-09-21T12:37:20.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5598000000000005e-05 2024-09-21T12:37:20.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,3", 2024-09-21T12:37:20.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.227004+0000", 2024-09-21T12:37:20.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6323999999999996e-05 2024-09-21T12:37:20.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:20.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.227859+0000", 2024-09-21T12:37:20.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00085563900000000005 2024-09-21T12:37:20.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.372977+0000", 2024-09-21T12:37:20.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14511786199999999 2024-09-21T12:37:20.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.379660+0000", 2024-09-21T12:37:20.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0066829469999999998 2024-09-21T12:37:20.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:20.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.379717+0000", 2024-09-21T12:37:20.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6575e-05 2024-09-21T12:37:20.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.691 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:20.691 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:56.379758+0000", 2024-09-21T12:37:20.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1594000000000002e-05 2024-09-21T12:37:20.691 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.691 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:20.691 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.691 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:11440 11.5 11:ad0a4d3e:::benchmark_data_smithi114_187622_object11439:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-21T12:37:20.691 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:35:56.833542+0000", 2024-09-21T12:37:20.692 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 83.320790360999993, 2024-09-21T12:37:20.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.043062870000000003, 2024-09-21T12:37:20.692 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:20.692 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:20.692 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:20.692 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:20.692 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:37:20.692 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:37:20.692 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 11440 2024-09-21T12:37:20.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.692 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:20.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:20.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:56.833542+0000", 2024-09-21T12:37:20.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:20.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:56.833542+0000", 2024-09-21T12:37:20.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:20.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:56.833543+0000", 2024-09-21T12:37:20.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2529999999999999e-06 2024-09-21T12:37:20.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:20.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:56.833616+0000", 2024-09-21T12:37:20.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2308e-05 2024-09-21T12:37:20.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:20.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:56.833631+0000", 2024-09-21T12:37:20.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5051999999999999e-05 2024-09-21T12:37:20.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:20.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:56.833636+0000", 2024-09-21T12:37:20.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8490000000000002e-06 2024-09-21T12:37:20.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:20.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:56.833694+0000", 2024-09-21T12:37:20.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7728e-05 2024-09-21T12:37:20.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:20.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:56.833830+0000", 2024-09-21T12:37:20.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013600099999999999 2024-09-21T12:37:20.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,6", 2024-09-21T12:37:20.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:56.833943+0000", 2024-09-21T12:37:20.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000112667 2024-09-21T12:37:20.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:20.696 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:56.834608+0000", 2024-09-21T12:37:20.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00066520000000000001 2024-09-21T12:37:20.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.696 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:56.836964+0000", 2024-09-21T12:37:20.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0023560830000000001 2024-09-21T12:37:20.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.696 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:56.876516+0000", 2024-09-21T12:37:20.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.039551613999999999 2024-09-21T12:37:20.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:20.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:56.876558+0000", 2024-09-21T12:37:20.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2559999999999999e-05 2024-09-21T12:37:20.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:20.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:56.876605+0000", 2024-09-21T12:37:20.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6554999999999997e-05 2024-09-21T12:37:20.697 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.697 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:20.697 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.697 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:14937 11.5 11:a1921f22:::benchmark_data_smithi114_187622_object14936:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-21T12:37:20.698 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:35:58.117671+0000", 2024-09-21T12:37:20.698 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 82.036661839999994, 2024-09-21T12:37:20.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.076060639999999999, 2024-09-21T12:37:20.698 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:20.698 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:20.698 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:20.698 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:20.698 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:37:20.698 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:37:20.698 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 14937 2024-09-21T12:37:20.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.698 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:20.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:20.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.117671+0000", 2024-09-21T12:37:20.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:20.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.117671+0000", 2024-09-21T12:37:20.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:20.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.117672+0000", 2024-09-21T12:37:20.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.353e-06 2024-09-21T12:37:20.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.117745+0000", 2024-09-21T12:37:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3047999999999996e-05 2024-09-21T12:37:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.117759+0000", 2024-09-21T12:37:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4154999999999999e-05 2024-09-21T12:37:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.117767+0000", 2024-09-21T12:37:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7729999999999999e-06 2024-09-21T12:37:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:20.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.117850+0000", 2024-09-21T12:37:20.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3066999999999996e-05 2024-09-21T12:37:20.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:20.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.118033+0000", 2024-09-21T12:37:20.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00018278500000000001 2024-09-21T12:37:20.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,6", 2024-09-21T12:37:20.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.118179+0000", 2024-09-21T12:37:20.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000146086 2024-09-21T12:37:20.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:20.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.119115+0000", 2024-09-21T12:37:20.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00093581199999999995 2024-09-21T12:37:20.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.119645+0000", 2024-09-21T12:37:20.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000530483 2024-09-21T12:37:20.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:20.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.193628+0000", 2024-09-21T12:37:20.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.073983362999999996 2024-09-21T12:37:20.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:20.703 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.193688+0000", 2024-09-21T12:37:20.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9032e-05 2024-09-21T12:37:20.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:20.703 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.193731+0000", 2024-09-21T12:37:20.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3683000000000002e-05 2024-09-21T12:37:20.703 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.703 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:20.703 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.703 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.26147.0:123856 11.5 e438/419 11:a2674114:::benchmark_data_smithi114_187622_object123855:head v 438'7634, mlcod=437'7631)", 2024-09-21T12:37:20.704 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:37:11.180617+0000", 2024-09-21T12:37:20.704 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 8.9737153850000002, 2024-09-21T12:37:20.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.046053943999999999, 2024-09-21T12:37:20.704 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:20.704 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:20.704 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:20.704 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:20.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:20.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:11.180617+0000", 2024-09-21T12:37:20.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:20.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:11.180617+0000", 2024-09-21T12:37:20.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:20.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:11.180618+0000", 2024-09-21T12:37:20.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.26e-06 2024-09-21T12:37:20.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:20.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:11.180660+0000", 2024-09-21T12:37:20.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1829999999999998e-05 2024-09-21T12:37:20.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:20.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:11.221792+0000", 2024-09-21T12:37:20.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.041131460000000002 2024-09-21T12:37:20.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:20.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:11.221799+0000", 2024-09-21T12:37:20.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7710000000000006e-06 2024-09-21T12:37:20.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:20.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:11.222269+0000", 2024-09-21T12:37:20.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00046984899999999999 2024-09-21T12:37:20.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:20.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:11.222289+0000", 2024-09-21T12:37:20.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9647e-05 2024-09-21T12:37:20.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:20.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:11.226580+0000", 2024-09-21T12:37:20.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0042912760000000001 2024-09-21T12:37:20.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:20.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:11.226671+0000", 2024-09-21T12:37:20.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0851000000000001e-05 2024-09-21T12:37:20.707 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.707 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:20.707 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.707 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.26147.0:123861 11.8 e438/405 11:1b6426f5:::benchmark_data_smithi114_187622_object123860:head v 438'7837, mlcod=437'7835)", 2024-09-21T12:37:20.708 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:37:11.187527+0000", 2024-09-21T12:37:20.708 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 8.9668056929999995, 2024-09-21T12:37:20.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.043550698999999998, 2024-09-21T12:37:20.708 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:20.708 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:20.708 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:20.708 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:20.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:20.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:11.187527+0000", 2024-09-21T12:37:20.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:20.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:11.187527+0000", 2024-09-21T12:37:20.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:20.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:20.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:11.187527+0000", 2024-09-21T12:37:20.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8699999999999995e-07 2024-09-21T12:37:20.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:20.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:11.187663+0000", 2024-09-21T12:37:20.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013611600000000001 2024-09-21T12:37:20.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:20.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:11.221688+0000", 2024-09-21T12:37:20.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.034024242000000003 2024-09-21T12:37:20.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.710 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:20.710 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:11.221697+0000", 2024-09-21T12:37:20.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0140000000000004e-06 2024-09-21T12:37:20.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.710 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:20.710 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:11.223679+0000", 2024-09-21T12:37:20.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0019818840000000002 2024-09-21T12:37:20.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.710 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:20.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:11.223696+0000", 2024-09-21T12:37:20.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7598999999999999e-05 2024-09-21T12:37:20.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:20.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:11.231048+0000", 2024-09-21T12:37:20.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0073514660000000001 2024-09-21T12:37:20.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:20.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:20.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:20.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:11.231077+0000", 2024-09-21T12:37:20.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9791e-05 2024-09-21T12:37:20.711 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.711 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:20.711 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.712 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:20.712 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:20.712 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:37:20.712 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:20.745 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:20.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:20.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:20.847 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:20.879 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:20.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:20.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:20.981 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:21.017 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:21.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:21.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:21.118 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:21.156 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:21.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:21.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:21.258 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:21.292 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:21.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:21.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:21.393 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:21.432 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:21.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:21.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:21.514 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 88 16 135702 135686 96.3521 91 0.00854337 0.0103747 2024-09-21T12:37:21.534 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:21.565 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:21.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:21.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:21.667 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:21.682 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T12:37:21.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:21.700 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:21.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:21.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:21 smithi183 ceph-mon[33277]: 6.11 scrub starts 2024-09-21T12:37:21.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:21 smithi183 ceph-mon[33277]: 6.11 scrub ok 2024-09-21T12:37:21.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:21 smithi183 ceph-mon[33277]: pgmap v869: 178 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped+scrubbing, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 124 active+clean; 8.0 GiB data, 30 GiB used, 686 GiB / 715 GiB avail; 60 MiB/s wr, 958 op/s; 32651/393714 objects degraded (8.293%); 16428/393714 objects misplaced (4.173%) 2024-09-21T12:37:21.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:21 smithi183 ceph-mon[33277]: osdmap e442: 8 total, 7 up, 8 in 2024-09-21T12:37:21.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:21 smithi183 ceph-mon[33277]: 11.5 scrub starts 2024-09-21T12:37:21.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:21 smithi114 ceph-mon[40867]: 6.11 scrub starts 2024-09-21T12:37:21.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:21 smithi114 ceph-mon[40867]: 6.11 scrub ok 2024-09-21T12:37:21.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:21 smithi114 ceph-mon[40867]: pgmap v869: 178 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped+scrubbing, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 124 active+clean; 8.0 GiB data, 30 GiB used, 686 GiB / 715 GiB avail; 60 MiB/s wr, 958 op/s; 32651/393714 objects degraded (8.293%); 16428/393714 objects misplaced (4.173%) 2024-09-21T12:37:21.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:21 smithi114 ceph-mon[40867]: osdmap e442: 8 total, 7 up, 8 in 2024-09-21T12:37:21.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:21 smithi114 ceph-mon[40867]: 11.5 scrub starts 2024-09-21T12:37:21.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:21 smithi114 ceph-mon[30324]: 6.11 scrub starts 2024-09-21T12:37:21.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:21 smithi114 ceph-mon[30324]: 6.11 scrub ok 2024-09-21T12:37:21.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:21 smithi114 ceph-mon[30324]: pgmap v869: 178 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped+scrubbing, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 124 active+clean; 8.0 GiB data, 30 GiB used, 686 GiB / 715 GiB avail; 60 MiB/s wr, 958 op/s; 32651/393714 objects degraded (8.293%); 16428/393714 objects misplaced (4.173%) 2024-09-21T12:37:21.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:21 smithi114 ceph-mon[30324]: osdmap e442: 8 total, 7 up, 8 in 2024-09-21T12:37:21.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:21 smithi114 ceph-mon[30324]: 11.5 scrub starts 2024-09-21T12:37:21.802 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:21.832 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:21.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:21.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:21.934 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:21.964 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:21.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:21.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:22.068 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:22.110 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:37:22.113 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:22.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:22.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:22.215 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:22.238 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:22.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:22.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:22.340 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:22.368 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:22.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:22.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:22.470 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:22.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:22.505 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:22.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:22.514 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 89 16 136881 136865 96.0974 73.6875 0.00254665 0.0104003 2024-09-21T12:37:22.606 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:22.640 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:22.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:22.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:22.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:22 smithi183 ceph-mon[33277]: osdmap e443: 8 total, 7 up, 8 in 2024-09-21T12:37:22.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:22 smithi183 ceph-mon[33277]: 4.18 scrub starts 2024-09-21T12:37:22.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:22 smithi183 ceph-mon[33277]: 4.18 scrub ok 2024-09-21T12:37:22.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:22 smithi183 ceph-mon[33277]: 5.19 scrub starts 2024-09-21T12:37:22.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:22 smithi183 ceph-mon[33277]: 5.19 scrub ok 2024-09-21T12:37:22.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:22 smithi183 ceph-mon[33277]: pgmap v872: 178 pgs: 1 active+clean+remapped+scrubbing, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 8.1 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 23 MiB/s wr, 374 op/s; 33019/397047 objects degraded (8.316%); 16428/397047 objects misplaced (4.138%) 2024-09-21T12:37:22.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:22 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 32651/393714 objects degraded (8.293%), 13 pgs degraded, 33 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:22.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:22 smithi183 ceph-mon[33277]: osdmap e444: 8 total, 7 up, 8 in 2024-09-21T12:37:22.742 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:22.774 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:37:22 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:37:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:37:22.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:22 smithi114 ceph-mon[40867]: osdmap e443: 8 total, 7 up, 8 in 2024-09-21T12:37:22.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:22 smithi114 ceph-mon[40867]: 4.18 scrub starts 2024-09-21T12:37:22.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:22 smithi114 ceph-mon[40867]: 4.18 scrub ok 2024-09-21T12:37:22.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:22 smithi114 ceph-mon[40867]: 5.19 scrub starts 2024-09-21T12:37:22.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:22 smithi114 ceph-mon[40867]: 5.19 scrub ok 2024-09-21T12:37:22.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:22 smithi114 ceph-mon[40867]: pgmap v872: 178 pgs: 1 active+clean+remapped+scrubbing, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 8.1 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 23 MiB/s wr, 374 op/s; 33019/397047 objects degraded (8.316%); 16428/397047 objects misplaced (4.138%) 2024-09-21T12:37:22.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:22 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 32651/393714 objects degraded (8.293%), 13 pgs degraded, 33 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:22.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:22 smithi114 ceph-mon[40867]: osdmap e444: 8 total, 7 up, 8 in 2024-09-21T12:37:22.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:22 smithi114 ceph-mon[30324]: osdmap e443: 8 total, 7 up, 8 in 2024-09-21T12:37:22.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:22 smithi114 ceph-mon[30324]: 4.18 scrub starts 2024-09-21T12:37:22.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:22 smithi114 ceph-mon[30324]: 4.18 scrub ok 2024-09-21T12:37:22.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:22 smithi114 ceph-mon[30324]: 5.19 scrub starts 2024-09-21T12:37:22.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:22 smithi114 ceph-mon[30324]: 5.19 scrub ok 2024-09-21T12:37:22.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:22 smithi114 ceph-mon[30324]: pgmap v872: 178 pgs: 1 active+clean+remapped+scrubbing, 19 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 8.1 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 23 MiB/s wr, 374 op/s; 33019/397047 objects degraded (8.316%); 16428/397047 objects misplaced (4.138%) 2024-09-21T12:37:22.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:22 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 32651/393714 objects degraded (8.293%), 13 pgs degraded, 33 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:22.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:22 smithi114 ceph-mon[30324]: osdmap e444: 8 total, 7 up, 8 in 2024-09-21T12:37:22.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:22.789 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:22.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:22.890 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:22.923 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:22.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:22.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:23.024 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:23.051 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:23.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:23.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:23.153 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:23.188 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:23.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:23.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:23.290 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:23.320 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:23.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:23.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:23.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:23.454 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:23.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:23.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:23.515 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Total time run: 90.0046 2024-09-21T12:37:23.515 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Total writes made: 138118 2024-09-21T12:37:23.515 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Write size: 65536 2024-09-21T12:37:23.515 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Object size: 65536 2024-09-21T12:37:23.515 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Bandwidth (MB/sec): 95.9104 2024-09-21T12:37:23.515 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev Bandwidth: 35.0008 2024-09-21T12:37:23.515 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max bandwidth (MB/sec): 238.812 2024-09-21T12:37:23.515 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min bandwidth (MB/sec): 0 2024-09-21T12:37:23.515 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Average IOPS: 1534 2024-09-21T12:37:23.516 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev IOPS: 560.013 2024-09-21T12:37:23.516 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max IOPS: 3821 2024-09-21T12:37:23.516 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min IOPS: 0 2024-09-21T12:37:23.516 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Average Latency(s): 0.0104228 2024-09-21T12:37:23.516 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev Latency(s): 0.0306795 2024-09-21T12:37:23.516 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max latency(s): 2.08308 2024-09-21T12:37:23.516 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min latency(s): 0.00111509 2024-09-21T12:37:23.521 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T12:37:23.556 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:23.590 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:23.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:23.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:23.625 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:23 smithi183 ceph-mon[33277]: 5.4 scrub starts 2024-09-21T12:37:23.625 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:23 smithi183 ceph-mon[33277]: 5.4 scrub ok 2024-09-21T12:37:23.666 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:37:23.666 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:37:23.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.667 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.26147.0:138337 11.a e444/405 11:56b1ef90:::benchmark_data_smithi114_187622_object216:head v 444'8639, mlcod=444'8632)", 2024-09-21T12:37:23.667 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:37:23.657766+0000", 2024-09-21T12:37:23.667 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.00554493, 2024-09-21T12:37:23.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0055717680000000004, 2024-09-21T12:37:23.667 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:23.667 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:23.667 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:37:23.667 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:23.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:23.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.657766+0000", 2024-09-21T12:37:23.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:23.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:23.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.657766+0000", 2024-09-21T12:37:23.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:23.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:23.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.657767+0000", 2024-09-21T12:37:23.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7700000000000004e-07 2024-09-21T12:37:23.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:23.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.657772+0000", 2024-09-21T12:37:23.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5489999999999997e-06 2024-09-21T12:37:23.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:23.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.659393+0000", 2024-09-21T12:37:23.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001620874 2024-09-21T12:37:23.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:23.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.659402+0000", 2024-09-21T12:37:23.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1649999999999995e-06 2024-09-21T12:37:23.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:23.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.661670+0000", 2024-09-21T12:37:23.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0022679599999999999 2024-09-21T12:37:23.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:23.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.661689+0000", 2024-09-21T12:37:23.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9014000000000001e-05 2024-09-21T12:37:23.670 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:23.671 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:23.671 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:23.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.671 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.26147.0:138344 11.1 e444/405 11:89741acc:::benchmark_data_smithi114_187622_object223:head v 444'8655, mlcod=444'8653)", 2024-09-21T12:37:23.671 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:37:23.661202+0000", 2024-09-21T12:37:23.671 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.002108197, 2024-09-21T12:37:23.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021894190000000002, 2024-09-21T12:37:23.671 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:23.671 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:23.671 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:37:23.671 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:23.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:23.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.661202+0000", 2024-09-21T12:37:23.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:23.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:23.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.661202+0000", 2024-09-21T12:37:23.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:23.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:23.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.661203+0000", 2024-09-21T12:37:23.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0400000000000005e-07 2024-09-21T12:37:23.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:23.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.661210+0000", 2024-09-21T12:37:23.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4899999999999997e-06 2024-09-21T12:37:23.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:23.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.661222+0000", 2024-09-21T12:37:23.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2163e-05 2024-09-21T12:37:23.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:23.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.661229+0000", 2024-09-21T12:37:23.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1940000000000001e-06 2024-09-21T12:37:23.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:23.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.661295+0000", 2024-09-21T12:37:23.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5279999999999998e-05 2024-09-21T12:37:23.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:23.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.661315+0000", 2024-09-21T12:37:23.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0307999999999999e-05 2024-09-21T12:37:23.674 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:23.675 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:23.675 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:23.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.675 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.26147.0:138316 11.a e444/405 11:59079802:::benchmark_data_smithi114_187622_object195:head v 444'8634, mlcod=444'8631)", 2024-09-21T12:37:23.675 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:37:23.641757+0000", 2024-09-21T12:37:23.675 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.021554001, 2024-09-21T12:37:23.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.021681867000000001, 2024-09-21T12:37:23.675 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:23.675 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:23.675 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:23.675 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:23.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:23.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.641757+0000", 2024-09-21T12:37:23.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:23.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:23.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.641757+0000", 2024-09-21T12:37:23.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:23.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:23.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.641757+0000", 2024-09-21T12:37:23.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1800000000000002e-07 2024-09-21T12:37:23.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:23.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.641760+0000", 2024-09-21T12:37:23.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2569999999999999e-06 2024-09-21T12:37:23.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:23.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.641766+0000", 2024-09-21T12:37:23.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.93e-06 2024-09-21T12:37:23.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:23.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.641770+0000", 2024-09-21T12:37:23.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0230000000000003e-06 2024-09-21T12:37:23.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:23.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.647751+0000", 2024-09-21T12:37:23.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0059805209999999999 2024-09-21T12:37:23.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:23.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.647773+0000", 2024-09-21T12:37:23.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2124999999999999e-05 2024-09-21T12:37:23.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:23.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.663392+0000", 2024-09-21T12:37:23.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.01561941 2024-09-21T12:37:23.679 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:23.679 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:23.679 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:23.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.680 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.26147.0:138324 11.a e444/405 11:5edb762c:::benchmark_data_smithi114_187622_object203:head v 444'8635, mlcod=444'8631)", 2024-09-21T12:37:23.680 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:37:23.647916+0000", 2024-09-21T12:37:23.680 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.015395125000000001, 2024-09-21T12:37:23.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.01558535, 2024-09-21T12:37:23.680 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:23.680 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:23.680 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:37:23.680 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:23.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:23.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.647916+0000", 2024-09-21T12:37:23.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:23.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:23.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.647916+0000", 2024-09-21T12:37:23.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:23.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:23.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.647916+0000", 2024-09-21T12:37:23.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6899999999999997e-07 2024-09-21T12:37:23.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:23.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.647921+0000", 2024-09-21T12:37:23.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2109999999999997e-06 2024-09-21T12:37:23.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:23.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.647932+0000", 2024-09-21T12:37:23.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0730000000000001e-05 2024-09-21T12:37:23.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:23.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.647938+0000", 2024-09-21T12:37:23.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0689999999999998e-06 2024-09-21T12:37:23.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:23.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.650403+0000", 2024-09-21T12:37:23.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024645180000000002 2024-09-21T12:37:23.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:23.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.650421+0000", 2024-09-21T12:37:23.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7856999999999999e-05 2024-09-21T12:37:23.683 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:23.683 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:23.683 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:23.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.683 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.26147.0:138328 11.a e444/405 11:573e9c7a:::benchmark_data_smithi114_187622_object207:head v 444'8636, mlcod=444'8631)", 2024-09-21T12:37:23.683 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:37:23.650321+0000", 2024-09-21T12:37:23.684 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.012989729, 2024-09-21T12:37:23.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.013234154, 2024-09-21T12:37:23.684 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:23.684 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:23.684 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:37:23.684 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:23.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:23.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.650321+0000", 2024-09-21T12:37:23.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:23.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:23.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.650321+0000", 2024-09-21T12:37:23.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:23.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:23.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.650322+0000", 2024-09-21T12:37:23.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9299999999999997e-07 2024-09-21T12:37:23.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:23.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.650328+0000", 2024-09-21T12:37:23.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8660000000000002e-06 2024-09-21T12:37:23.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:23.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.650338+0000", 2024-09-21T12:37:23.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0754000000000001e-05 2024-09-21T12:37:23.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:23.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.650344+0000", 2024-09-21T12:37:23.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0360000000000003e-06 2024-09-21T12:37:23.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:23.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.652342+0000", 2024-09-21T12:37:23.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001997207 2024-09-21T12:37:23.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:23.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.652357+0000", 2024-09-21T12:37:23.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5735000000000001e-05 2024-09-21T12:37:23.687 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:23.687 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:23.687 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:23.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.688 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.26147.0:138332 11.a e444/405 11:5c483b4e:::benchmark_data_smithi114_187622_object211:head v 444'8637, mlcod=444'8632)", 2024-09-21T12:37:23.688 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:37:23.653270+0000", 2024-09-21T12:37:23.688 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.010040848, 2024-09-21T12:37:23.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.010312427000000001, 2024-09-21T12:37:23.688 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:23.688 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:23.688 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:37:23.688 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:23.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:23.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.653270+0000", 2024-09-21T12:37:23.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:23.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:23.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.653270+0000", 2024-09-21T12:37:23.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:23.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:23.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.653271+0000", 2024-09-21T12:37:23.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8000000000000004e-07 2024-09-21T12:37:23.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:23.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.653275+0000", 2024-09-21T12:37:23.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.092e-06 2024-09-21T12:37:23.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:23.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.653285+0000", 2024-09-21T12:37:23.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0006e-05 2024-09-21T12:37:23.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:23.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.653291+0000", 2024-09-21T12:37:23.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3339999999999998e-06 2024-09-21T12:37:23.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.691 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:23.691 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.656495+0000", 2024-09-21T12:37:23.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0032041299999999999 2024-09-21T12:37:23.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.691 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:23.691 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.656560+0000", 2024-09-21T12:37:23.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4224e-05 2024-09-21T12:37:23.691 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:23.691 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:23.691 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:23.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.692 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.26147.0:138336 11.a e444/405 11:562e1b41:::benchmark_data_smithi114_187622_object215:head v 444'8638, mlcod=444'8632)", 2024-09-21T12:37:23.692 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:37:23.655654+0000", 2024-09-21T12:37:23.692 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0076567680000000004, 2024-09-21T12:37:23.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0079558130000000008, 2024-09-21T12:37:23.692 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:23.692 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:23.692 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:37:23.692 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:23.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:23.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.655654+0000", 2024-09-21T12:37:23.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:23.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:23.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.655654+0000", 2024-09-21T12:37:23.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:23.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:23.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.655654+0000", 2024-09-21T12:37:23.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5900000000000002e-07 2024-09-21T12:37:23.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:23.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.655658+0000", 2024-09-21T12:37:23.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3900000000000002e-06 2024-09-21T12:37:23.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:23.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.655667+0000", 2024-09-21T12:37:23.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6409999999999997e-06 2024-09-21T12:37:23.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:23.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.655672+0000", 2024-09-21T12:37:23.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.561e-06 2024-09-21T12:37:23.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:23.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.659470+0000", 2024-09-21T12:37:23.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0037981349999999998 2024-09-21T12:37:23.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:23.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:23.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:23.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:23.659488+0000", 2024-09-21T12:37:23.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8326999999999999e-05 2024-09-21T12:37:23.695 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:23.695 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:23.695 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:23.695 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:23.695 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-21T12:37:23.695 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 7 2024-09-21T12:37:23.695 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:37:23.695 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:23.733 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:23.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:23.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:23.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:23 smithi114 ceph-mon[30324]: 5.4 scrub starts 2024-09-21T12:37:23.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:23 smithi114 ceph-mon[30324]: 5.4 scrub ok 2024-09-21T12:37:23.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:23 smithi114 ceph-mon[40867]: 5.4 scrub starts 2024-09-21T12:37:23.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:23 smithi114 ceph-mon[40867]: 5.4 scrub ok 2024-09-21T12:37:23.835 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:23.868 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:23.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:23.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:23.969 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:24.003 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:24.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:24.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:24.105 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:24.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:24.140 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:24.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:24.241 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:24.280 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:24.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:24.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:24.382 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:24.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:24.418 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:24.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:24.500 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T12:37:24.500 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T12:37:24.519 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:24.524 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:37:24.548 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:24.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:24.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:24.650 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:24.656 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T12:37:24.688 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:24.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:24.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:24.722 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:24 smithi114 ceph-mon[30324]: 11.5 scrub ok 2024-09-21T12:37:24.722 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:24 smithi114 ceph-mon[30324]: pgmap v874: 178 pgs: 1 active+clean+scrubbing, 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 124 active+clean; 8.3 GiB data, 31 GiB used, 685 GiB / 715 GiB avail; 1.1 KiB/s rd, 81 MiB/s wr, 1.29k op/s; 34084/410910 objects degraded (8.295%); 17161/410910 objects misplaced (4.176%) 2024-09-21T12:37:24.723 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:24 smithi114 ceph-mon[40867]: 11.5 scrub ok 2024-09-21T12:37:24.723 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:24 smithi114 ceph-mon[40867]: pgmap v874: 178 pgs: 1 active+clean+scrubbing, 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 124 active+clean; 8.3 GiB data, 31 GiB used, 685 GiB / 715 GiB avail; 1.1 KiB/s rd, 81 MiB/s wr, 1.29k op/s; 34084/410910 objects degraded (8.295%); 17161/410910 objects misplaced (4.176%) 2024-09-21T12:37:24.789 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:24.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:24.836 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:24.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:24.937 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:24.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:24 smithi183 ceph-mon[33277]: 11.5 scrub ok 2024-09-21T12:37:24.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:24 smithi183 ceph-mon[33277]: pgmap v874: 178 pgs: 1 active+clean+scrubbing, 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 124 active+clean; 8.3 GiB data, 31 GiB used, 685 GiB / 715 GiB avail; 1.1 KiB/s rd, 81 MiB/s wr, 1.29k op/s; 34084/410910 objects degraded (8.295%); 17161/410910 objects misplaced (4.176%) 2024-09-21T12:37:24.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:24.991 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:24.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:25.078 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:37:25.092 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:25.139 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:25.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:25.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:25.243 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:25.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:25.279 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:25.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:25.380 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:25.409 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:25.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:25.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:25.511 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:25.548 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:25.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:25.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:25.650 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:25.689 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:25.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:25.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:25.706 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:25 smithi183 ceph-mon[33277]: 3.1b scrub starts 2024-09-21T12:37:25.706 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:25 smithi183 ceph-mon[33277]: 3.1b scrub ok 2024-09-21T12:37:25.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:25 smithi114 ceph-mon[40867]: 3.1b scrub starts 2024-09-21T12:37:25.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:25 smithi114 ceph-mon[40867]: 3.1b scrub ok 2024-09-21T12:37:25.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:25 smithi114 ceph-mon[30324]: 3.1b scrub starts 2024-09-21T12:37:25.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:25 smithi114 ceph-mon[30324]: 3.1b scrub ok 2024-09-21T12:37:25.791 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:25.824 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:25.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:25.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:25.875 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:37:25.875 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-21T12:37:25.875 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-21T12:37:25.875 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-21T12:37:25.875 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:37:25.927 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:25.965 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:25.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:25.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:26.067 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:26.104 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:26.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:26.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:26.205 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:26.238 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:26.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:26.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:26.340 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:26.370 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:26.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:26.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:26.472 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:26.501 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:37:26.511 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:26.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:26.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:26.523 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-21T12:37:26.612 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:26.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:26.649 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:26.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:26.729 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:26 smithi183 ceph-mon[33277]: 3.1d scrub starts 2024-09-21T12:37:26.729 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:26 smithi183 ceph-mon[33277]: 3.1d scrub ok 2024-09-21T12:37:26.729 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:26 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/392613659' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:37:26.729 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:26 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:37:26.729 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:26 smithi183 ceph-mon[33277]: pgmap v875: 178 pgs: 1 active+clean+scrubbing, 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 124 active+clean; 8.3 GiB data, 31 GiB used, 685 GiB / 715 GiB avail; 853 B/s rd, 60 MiB/s wr, 955 op/s; 34084/410910 objects degraded (8.295%); 17161/410910 objects misplaced (4.176%) 2024-09-21T12:37:26.750 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:26.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:26 smithi114 ceph-mon[40867]: 3.1d scrub starts 2024-09-21T12:37:26.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:26 smithi114 ceph-mon[40867]: 3.1d scrub ok 2024-09-21T12:37:26.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:26 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/392613659' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:37:26.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:26 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:37:26.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:26 smithi114 ceph-mon[40867]: pgmap v875: 178 pgs: 1 active+clean+scrubbing, 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 124 active+clean; 8.3 GiB data, 31 GiB used, 685 GiB / 715 GiB avail; 853 B/s rd, 60 MiB/s wr, 955 op/s; 34084/410910 objects degraded (8.295%); 17161/410910 objects misplaced (4.176%) 2024-09-21T12:37:26.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:26 smithi114 ceph-mon[30324]: 3.1d scrub starts 2024-09-21T12:37:26.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:26 smithi114 ceph-mon[30324]: 3.1d scrub ok 2024-09-21T12:37:26.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:26 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/392613659' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:37:26.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:26 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:37:26.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:26 smithi114 ceph-mon[30324]: pgmap v875: 178 pgs: 1 active+clean+scrubbing, 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 124 active+clean; 8.3 GiB data, 31 GiB used, 685 GiB / 715 GiB avail; 853 B/s rd, 60 MiB/s wr, 955 op/s; 34084/410910 objects degraded (8.295%); 17161/410910 objects misplaced (4.176%) 2024-09-21T12:37:26.785 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T12:37:26.794 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:26.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:26.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:26.896 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:26.933 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:26.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:26.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:27.035 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:27.077 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:27.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:27.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:27.115 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 4, 7] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-09-21T12:37:27.115 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:37:27.115 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_4 2024-09-21T12:37:27.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:37:27.179 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:27.210 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:27.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:27.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:27.249 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:37:27.313 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:27.350 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:27.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:27.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:27.452 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:27.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:27.486 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:27.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:27.547 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:37:27.558 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:37:27.586 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:27.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:27.630 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:27.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:27.660 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:27 smithi183 ceph-mon[33277]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:37:27.660 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:27 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:37:27.660 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:27 smithi183 ceph-mon[33277]: osdmap e445: 8 total, 7 up, 8 in 2024-09-21T12:37:27.660 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:27 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 34084/410910 objects degraded (8.295%), 13 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:27.660 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:27 smithi183 ceph-mon[33277]: osdmap e446: 8 total, 7 up, 8 in 2024-09-21T12:37:27.683 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:27 smithi114 ceph-mon[40867]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:37:27.684 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:27 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:37:27.684 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:27 smithi114 ceph-mon[40867]: osdmap e445: 8 total, 7 up, 8 in 2024-09-21T12:37:27.684 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:27 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 34084/410910 objects degraded (8.295%), 13 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:27.684 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:27 smithi114 ceph-mon[40867]: osdmap e446: 8 total, 7 up, 8 in 2024-09-21T12:37:27.684 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:27 smithi114 ceph-mon[30324]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:37:27.684 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:27 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:37:27.684 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:27 smithi114 ceph-mon[30324]: osdmap e445: 8 total, 7 up, 8 in 2024-09-21T12:37:27.684 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:27 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 34084/410910 objects degraded (8.295%), 13 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:27.685 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:27 smithi114 ceph-mon[30324]: osdmap e446: 8 total, 7 up, 8 in 2024-09-21T12:37:27.731 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:27.766 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:27.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:27.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:27.863 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:37:27.864 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-21T12:37:27.864 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-21T12:37:27.864 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:37:27.864 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.864 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25022.0:223832 8.9 8:95f724a3:::benchmark_data_smithi114_104936_object93416:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-09-21T12:37:27.864 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:27:35.314953+0000", 2024-09-21T12:37:27.864 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 592.54349812700002, 2024-09-21T12:37:27.864 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13631865800000001, 2024-09-21T12:37:27.864 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:27.864 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:27.864 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:27.864 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:27.864 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25022", 2024-09-21T12:37:27.865 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/1784743428", 2024-09-21T12:37:27.865 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 223832 2024-09-21T12:37:27.865 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.865 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:27.865 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.865 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:27.865 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.314953+0000", 2024-09-21T12:37:27.865 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.865 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.865 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.865 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:27.865 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.314953+0000", 2024-09-21T12:37:27.865 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.865 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.865 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.866 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:27.866 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.314954+0000", 2024-09-21T12:37:27.866 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4249999999999999e-06 2024-09-21T12:37:27.866 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.866 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.866 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:27.866 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.314959+0000", 2024-09-21T12:37:27.866 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6149999999999999e-06 2024-09-21T12:37:27.866 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.866 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.866 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:27.866 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.314973+0000", 2024-09-21T12:37:27.866 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4134999999999999e-05 2024-09-21T12:37:27.866 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.867 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.867 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:27.867 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.314979+0000", 2024-09-21T12:37:27.867 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3489999999999997e-06 2024-09-21T12:37:27.867 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.867 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.867 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.867 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.315029+0000", 2024-09-21T12:37:27.867 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0272999999999998e-05 2024-09-21T12:37:27.867 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.867 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.867 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:27.867 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.315137+0000", 2024-09-21T12:37:27.867 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000107878 2024-09-21T12:37:27.868 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.868 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.868 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-21T12:37:27.868 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.315251+0000", 2024-09-21T12:37:27.868 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000113453 2024-09-21T12:37:27.868 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.868 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.868 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:27.868 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.317452+0000", 2024-09-21T12:37:27.868 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0022017080000000001 2024-09-21T12:37:27.868 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.868 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.868 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.868 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.317539+0000", 2024-09-21T12:37:27.868 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6935999999999996e-05 2024-09-21T12:37:27.869 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.869 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.869 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.869 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.451201+0000", 2024-09-21T12:37:27.869 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13366187199999999 2024-09-21T12:37:27.869 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.869 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.869 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:27.869 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.451237+0000", 2024-09-21T12:37:27.869 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6273999999999999e-05 2024-09-21T12:37:27.869 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.869 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.869 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:27.869 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:27:35.451271+0000", 2024-09-21T12:37:27.870 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3739999999999999e-05 2024-09-21T12:37:27.870 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.870 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:27.870 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.870 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.870 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.870 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:4793 10.0 10:0db935dd:::benchmark_data_smithi114_161690_object4792:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-09-21T12:37:27.870 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:32:40.429237+0000", 2024-09-21T12:37:27.870 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 287.42921341800002, 2024-09-21T12:37:27.870 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.97170812799999995, 2024-09-21T12:37:27.870 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:27.870 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:27.870 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:27.870 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:27.871 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:37:27.871 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:37:27.871 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 4793 2024-09-21T12:37:27.871 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.871 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:27.871 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.871 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:27.871 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.429237+0000", 2024-09-21T12:37:27.871 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.871 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.871 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.871 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:27.871 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.429237+0000", 2024-09-21T12:37:27.871 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.872 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.872 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.872 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:27.872 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.429241+0000", 2024-09-21T12:37:27.872 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4620000000000001e-06 2024-09-21T12:37:27.872 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.872 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.872 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:27.872 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.429321+0000", 2024-09-21T12:37:27.872 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0127e-05 2024-09-21T12:37:27.872 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.872 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.872 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:27.873 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.429332+0000", 2024-09-21T12:37:27.873 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.081e-05 2024-09-21T12:37:27.873 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.873 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.873 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:27.873 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.429337+0000", 2024-09-21T12:37:27.873 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2000000000000002e-06 2024-09-21T12:37:27.873 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.873 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.873 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.873 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.429383+0000", 2024-09-21T12:37:27.873 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6369999999999998e-05 2024-09-21T12:37:27.873 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.873 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.873 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:27.874 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.429390+0000", 2024-09-21T12:37:27.874 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0729999999999999e-06 2024-09-21T12:37:27.874 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.874 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.874 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.874 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.394738+0000", 2024-09-21T12:37:27.874 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.96534762399999996 2024-09-21T12:37:27.874 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.874 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.874 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:27.874 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.394860+0000", 2024-09-21T12:37:27.874 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012222200000000001 2024-09-21T12:37:27.875 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.875 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.875 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,4", 2024-09-21T12:37:27.875 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.394986+0000", 2024-09-21T12:37:27.875 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012562900000000001 2024-09-21T12:37:27.876 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.876 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.876 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:27.876 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.396494+0000", 2024-09-21T12:37:27.876 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001508704 2024-09-21T12:37:27.876 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.876 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.876 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.876 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.399005+0000", 2024-09-21T12:37:27.876 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0025109709999999999 2024-09-21T12:37:27.876 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.876 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.876 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.877 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.400827+0000", 2024-09-21T12:37:27.877 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001821533 2024-09-21T12:37:27.877 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.877 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.877 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:27.877 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.400894+0000", 2024-09-21T12:37:27.877 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7377000000000002e-05 2024-09-21T12:37:27.877 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.877 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.877 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:27.877 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.400945+0000", 2024-09-21T12:37:27.877 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1026000000000002e-05 2024-09-21T12:37:27.877 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.877 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:27.878 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.878 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.878 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.878 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:4801 10.0 10:0f176bf4:::benchmark_data_smithi114_161690_object4800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-09-21T12:37:27.878 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:32:40.434971+0000", 2024-09-21T12:37:27.878 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 287.42347940299999, 2024-09-21T12:37:27.878 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.96613722999999996, 2024-09-21T12:37:27.878 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:27.878 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:27.878 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:27.878 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:27.878 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:37:27.878 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:37:27.878 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 4801 2024-09-21T12:37:27.879 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.879 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:27.879 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.879 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:27.879 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.434971+0000", 2024-09-21T12:37:27.879 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.879 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.879 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.879 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:27.879 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.434971+0000", 2024-09-21T12:37:27.879 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.879 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.879 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.879 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:27.880 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.434978+0000", 2024-09-21T12:37:27.880 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9670000000000004e-06 2024-09-21T12:37:27.880 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.880 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.880 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:27.880 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.435070+0000", 2024-09-21T12:37:27.880 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1558999999999995e-05 2024-09-21T12:37:27.880 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.880 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.880 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:27.880 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.435086+0000", 2024-09-21T12:37:27.880 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6447e-05 2024-09-21T12:37:27.880 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.880 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.881 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:27.881 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.435095+0000", 2024-09-21T12:37:27.881 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6719999999999993e-06 2024-09-21T12:37:27.881 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.881 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.881 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.881 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.435169+0000", 2024-09-21T12:37:27.881 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3708999999999995e-05 2024-09-21T12:37:27.881 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.881 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.881 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:27.881 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.435179+0000", 2024-09-21T12:37:27.881 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0409e-05 2024-09-21T12:37:27.881 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.881 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.882 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.882 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.395628+0000", 2024-09-21T12:37:27.882 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.960448686 2024-09-21T12:37:27.882 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.882 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.882 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:27.882 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.395750+0000", 2024-09-21T12:37:27.882 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000121984 2024-09-21T12:37:27.882 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.882 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.882 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,4", 2024-09-21T12:37:27.883 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.395914+0000", 2024-09-21T12:37:27.883 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000163857 2024-09-21T12:37:27.883 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.883 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.883 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:27.883 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.396701+0000", 2024-09-21T12:37:27.883 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00078757800000000004 2024-09-21T12:37:27.883 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.883 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.883 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.883 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.399098+0000", 2024-09-21T12:37:27.883 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0023964110000000002 2024-09-21T12:37:27.883 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.883 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.884 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.884 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401013+0000", 2024-09-21T12:37:27.884 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0019155540000000001 2024-09-21T12:37:27.884 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.884 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.884 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:27.884 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401072+0000", 2024-09-21T12:37:27.884 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8999999999999998e-05 2024-09-21T12:37:27.884 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.884 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.884 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:27.884 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401108+0000", 2024-09-21T12:37:27.884 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6396999999999997e-05 2024-09-21T12:37:27.884 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.884 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:27.885 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.885 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.885 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.885 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:43123 10.f 10:f5348dda:::benchmark_data_smithi114_161690_object43122:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-09-21T12:37:27.885 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:33:00.807719+0000", 2024-09-21T12:37:27.885 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 267.05073191100001, 2024-09-21T12:37:27.885 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.37280072400000003, 2024-09-21T12:37:27.885 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:27.885 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:27.885 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:27.885 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:27.885 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:37:27.885 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:37:27.885 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 43123 2024-09-21T12:37:27.886 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.886 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:27.886 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.886 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:27.886 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.807719+0000", 2024-09-21T12:37:27.886 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.886 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.886 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.886 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:27.886 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.807719+0000", 2024-09-21T12:37:27.886 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.886 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.886 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.887 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:27.887 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.807720+0000", 2024-09-21T12:37:27.887 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.46e-06 2024-09-21T12:37:27.887 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.887 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.887 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:27.887 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.807752+0000", 2024-09-21T12:37:27.887 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1424000000000003e-05 2024-09-21T12:37:27.887 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.887 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.887 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:27.887 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.807766+0000", 2024-09-21T12:37:27.887 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4498000000000001e-05 2024-09-21T12:37:27.887 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.887 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.888 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:27.888 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.807774+0000", 2024-09-21T12:37:27.888 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9780000000000005e-06 2024-09-21T12:37:27.888 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.888 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.888 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.888 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.808091+0000", 2024-09-21T12:37:27.888 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00031702100000000001 2024-09-21T12:37:27.888 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.888 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.888 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:27.888 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.808233+0000", 2024-09-21T12:37:27.888 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014201 2024-09-21T12:37:27.888 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.888 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.889 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,3", 2024-09-21T12:37:27.889 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.808361+0000", 2024-09-21T12:37:27.889 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000127813 2024-09-21T12:37:27.889 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.889 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.889 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.889 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.810940+0000", 2024-09-21T12:37:27.889 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0025795179999999998 2024-09-21T12:37:27.889 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.889 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.889 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:27.889 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.812556+0000", 2024-09-21T12:37:27.889 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001615184 2024-09-21T12:37:27.889 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.889 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.890 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.890 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:01.180397+0000", 2024-09-21T12:37:27.890 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.36784113099999999 2024-09-21T12:37:27.890 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.890 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.890 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:27.890 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:01.180472+0000", 2024-09-21T12:37:27.890 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5068999999999996e-05 2024-09-21T12:37:27.890 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.890 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.890 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:27.890 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:01.180519+0000", 2024-09-21T12:37:27.890 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7617999999999997e-05 2024-09-21T12:37:27.890 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.891 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:27.891 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.891 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.891 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.891 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:43148 10.f 10:f122d938:::benchmark_data_smithi114_161690_object43147:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-09-21T12:37:27.891 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:33:00.820371+0000", 2024-09-21T12:37:27.891 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:27.892 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 267.03807935700002, 2024-09-21T12:37:27.892 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.36041693699999999, 2024-09-21T12:37:27.892 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:27.892 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:27.892 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:27.893 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:27.893 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:37:27.893 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:37:27.893 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 43148 2024-09-21T12:37:27.893 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.893 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:27.893 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.893 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:27.893 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.820371+0000", 2024-09-21T12:37:27.893 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.893 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.893 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.893 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:27.893 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.820371+0000", 2024-09-21T12:37:27.894 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.894 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.894 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.894 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:27.894 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.820373+0000", 2024-09-21T12:37:27.894 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5290000000000001e-06 2024-09-21T12:37:27.894 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.894 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.894 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:27.894 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.820402+0000", 2024-09-21T12:37:27.894 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9646e-05 2024-09-21T12:37:27.894 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.894 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.894 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:27.895 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.820415+0000", 2024-09-21T12:37:27.895 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2773e-05 2024-09-21T12:37:27.895 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.895 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.895 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:27.895 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.820423+0000", 2024-09-21T12:37:27.895 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5279999999999998e-06 2024-09-21T12:37:27.895 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.895 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.895 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.895 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.820517+0000", 2024-09-21T12:37:27.895 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4477999999999999e-05 2024-09-21T12:37:27.895 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.895 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.896 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:27.896 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.820658+0000", 2024-09-21T12:37:27.896 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000140507 2024-09-21T12:37:27.896 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.896 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.896 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,3", 2024-09-21T12:37:27.896 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.820792+0000", 2024-09-21T12:37:27.896 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013416799999999999 2024-09-21T12:37:27.896 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.896 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.896 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:27.896 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.822512+0000", 2024-09-21T12:37:27.896 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001720023 2024-09-21T12:37:27.896 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.897 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.897 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.897 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.822900+0000", 2024-09-21T12:37:27.897 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00038832100000000001 2024-09-21T12:37:27.897 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.897 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.897 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.897 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:01.180699+0000", 2024-09-21T12:37:27.897 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.35779872600000001 2024-09-21T12:37:27.897 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.897 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.897 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:27.897 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:01.180751+0000", 2024-09-21T12:37:27.897 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2394e-05 2024-09-21T12:37:27.898 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.898 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.898 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:27.898 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:01.180788+0000", 2024-09-21T12:37:27.898 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6844000000000002e-05 2024-09-21T12:37:27.898 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.898 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:27.898 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.898 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.898 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.898 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:64539 10.0 10:0a4a2714:::benchmark_data_smithi114_161690_object64538:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-09-21T12:37:27.898 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:33:16.091235+0000", 2024-09-21T12:37:27.898 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 251.76721526200001, 2024-09-21T12:37:27.898 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.12744955099999999, 2024-09-21T12:37:27.899 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:27.899 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:27.899 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:27.899 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:27.899 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:37:27.899 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:37:27.899 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 64539 2024-09-21T12:37:27.899 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.899 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:27.899 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.899 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:27.899 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.091235+0000", 2024-09-21T12:37:27.899 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.900 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.900 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.900 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:27.900 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.091235+0000", 2024-09-21T12:37:27.900 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.900 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.900 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.900 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:27.900 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.091237+0000", 2024-09-21T12:37:27.900 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7090000000000001e-06 2024-09-21T12:37:27.900 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.900 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.900 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:27.900 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.091267+0000", 2024-09-21T12:37:27.900 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9697999999999999e-05 2024-09-21T12:37:27.901 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.901 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.901 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:27.901 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.091278+0000", 2024-09-21T12:37:27.901 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1452e-05 2024-09-21T12:37:27.901 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.901 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.901 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:27.901 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.091284+0000", 2024-09-21T12:37:27.901 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.327e-06 2024-09-21T12:37:27.901 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.901 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.901 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.901 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.091360+0000", 2024-09-21T12:37:27.902 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6860000000000006e-05 2024-09-21T12:37:27.902 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.902 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.902 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:27.902 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.091457+0000", 2024-09-21T12:37:27.902 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6650999999999993e-05 2024-09-21T12:37:27.902 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.902 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.902 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,4", 2024-09-21T12:37:27.902 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.091542+0000", 2024-09-21T12:37:27.902 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5358999999999994e-05 2024-09-21T12:37:27.902 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.902 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.903 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:27.903 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.093589+0000", 2024-09-21T12:37:27.903 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020467150000000002 2024-09-21T12:37:27.903 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.903 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.903 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.903 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.093785+0000", 2024-09-21T12:37:27.903 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000195484 2024-09-21T12:37:27.903 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.903 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.903 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.903 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.218607+0000", 2024-09-21T12:37:27.903 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.124822291 2024-09-21T12:37:27.903 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.903 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.904 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:27.904 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.218646+0000", 2024-09-21T12:37:27.904 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8739000000000003e-05 2024-09-21T12:37:27.904 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.904 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.904 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:27.904 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.218685+0000", 2024-09-21T12:37:27.904 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9266e-05 2024-09-21T12:37:27.904 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.904 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:27.904 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.904 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.904 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.904 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:160307 10.f 10:f936c05c:::benchmark_data_smithi114_161690_object33255:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-21T12:37:27.905 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:34:33.358322+0000", 2024-09-21T12:37:27.905 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 174.50012876700001, 2024-09-21T12:37:27.905 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.010655659, 2024-09-21T12:37:27.905 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:27.905 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:27.905 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:27.905 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:27.905 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:37:27.905 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:37:27.905 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 160307 2024-09-21T12:37:27.905 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.905 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:27.905 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.905 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:27.906 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.358322+0000", 2024-09-21T12:37:27.906 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.906 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.906 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.906 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:27.906 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.358322+0000", 2024-09-21T12:37:27.906 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.906 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.906 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.906 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:27.906 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.358324+0000", 2024-09-21T12:37:27.906 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6819999999999999e-06 2024-09-21T12:37:27.906 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.907 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:27.907 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.358329+0000", 2024-09-21T12:37:27.907 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2530000000000004e-06 2024-09-21T12:37:27.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.907 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:27.907 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.358342+0000", 2024-09-21T12:37:27.907 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3258000000000001e-05 2024-09-21T12:37:27.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.907 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:27.907 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.358349+0000", 2024-09-21T12:37:27.908 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8419999999999999e-06 2024-09-21T12:37:27.908 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.908 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.908 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.908 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.358395+0000", 2024-09-21T12:37:27.908 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6254000000000001e-05 2024-09-21T12:37:27.908 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.908 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.908 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:27.908 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.358402+0000", 2024-09-21T12:37:27.908 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5540000000000003e-06 2024-09-21T12:37:27.908 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.908 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.908 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.909 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.364082+0000", 2024-09-21T12:37:27.909 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.005680286 2024-09-21T12:37:27.909 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.909 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.909 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:27.909 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.364229+0000", 2024-09-21T12:37:27.909 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014693700000000001 2024-09-21T12:37:27.909 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.909 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.909 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3", 2024-09-21T12:37:27.909 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.364330+0000", 2024-09-21T12:37:27.909 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010115000000000001 2024-09-21T12:37:27.909 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.909 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.910 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:27.910 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.368799+0000", 2024-09-21T12:37:27.910 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0044687160000000002 2024-09-21T12:37:27.910 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.910 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.910 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.910 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.368853+0000", 2024-09-21T12:37:27.910 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.376e-05 2024-09-21T12:37:27.910 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.910 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.910 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:27.910 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.368906+0000", 2024-09-21T12:37:27.910 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3761999999999998e-05 2024-09-21T12:37:27.910 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.910 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.911 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:27.911 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.368978+0000", 2024-09-21T12:37:27.911 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1204999999999993e-05 2024-09-21T12:37:27.911 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.911 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:27.911 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.911 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.911 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.911 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:16847 11.4 11:27dda178:::benchmark_data_smithi114_187622_object16846:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-21T12:37:27.911 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:35:58.924832+0000", 2024-09-21T12:37:27.911 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 88.933618315999993, 2024-09-21T12:37:27.911 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14433790599999999, 2024-09-21T12:37:27.911 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:27.912 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:27.912 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:27.912 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:27.912 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:37:27.912 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:37:27.912 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 16847 2024-09-21T12:37:27.912 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.912 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:27.912 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.912 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:27.912 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.924832+0000", 2024-09-21T12:37:27.912 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.912 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.912 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.913 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:27.913 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.924832+0000", 2024-09-21T12:37:27.913 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.913 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.913 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.913 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:27.913 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.924834+0000", 2024-09-21T12:37:27.913 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.722e-06 2024-09-21T12:37:27.913 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.913 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.913 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:27.913 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.924885+0000", 2024-09-21T12:37:27.913 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1369000000000002e-05 2024-09-21T12:37:27.913 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.913 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.914 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:27.914 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.924906+0000", 2024-09-21T12:37:27.914 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0724999999999999e-05 2024-09-21T12:37:27.914 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.914 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.914 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:27.914 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.924913+0000", 2024-09-21T12:37:27.914 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2980000000000003e-06 2024-09-21T12:37:27.914 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.914 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.914 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.914 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.924966+0000", 2024-09-21T12:37:27.914 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2994000000000001e-05 2024-09-21T12:37:27.914 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.915 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.915 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:27.915 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.925099+0000", 2024-09-21T12:37:27.915 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000132733 2024-09-21T12:37:27.915 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.915 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.915 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:37:27.915 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.925248+0000", 2024-09-21T12:37:27.915 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000149059 2024-09-21T12:37:27.915 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.915 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.915 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:27.915 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.926325+0000", 2024-09-21T12:37:27.915 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00107649 2024-09-21T12:37:27.915 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.916 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.916 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.916 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.929258+0000", 2024-09-21T12:37:27.916 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0029329260000000002 2024-09-21T12:37:27.916 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.916 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.916 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.916 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:59.068980+0000", 2024-09-21T12:37:27.916 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13972220599999999 2024-09-21T12:37:27.916 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.916 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.916 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:27.916 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:59.069048+0000", 2024-09-21T12:37:27.916 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8336000000000004e-05 2024-09-21T12:37:27.917 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.917 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.917 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:27.917 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:59.069170+0000", 2024-09-21T12:37:27.917 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012204799999999999 2024-09-21T12:37:27.917 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.917 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:27.917 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.917 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.917 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.917 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:17074 11.4 11.189f0fc4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-09-21T12:37:27.917 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:35:59.165067+0000", 2024-09-21T12:37:27.917 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 88.693384017, 2024-09-21T12:37:27.917 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.95632275099999997, 2024-09-21T12:37:27.918 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:27.918 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:27.918 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-09-21T12:37:27.918 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:27.918 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:37:27.918 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:37:27.918 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 17074 2024-09-21T12:37:27.918 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.918 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:27.918 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.918 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:27.918 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:59.165067+0000", 2024-09-21T12:37:27.918 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.918 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.919 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.919 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:27.919 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:59.165067+0000", 2024-09-21T12:37:27.919 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.919 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.919 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.919 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:27.919 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:59.165068+0000", 2024-09-21T12:37:27.919 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5650000000000001e-06 2024-09-21T12:37:27.919 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.919 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.919 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:27.919 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:59.165104+0000", 2024-09-21T12:37:27.919 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6115000000000002e-05 2024-09-21T12:37:27.920 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:37:27 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:37:27.921 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.921 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.921 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:27.921 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:59.165112+0000", 2024-09-21T12:37:27.921 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4159999999999998e-06 2024-09-21T12:37:27.921 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.921 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.921 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:27.921 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:59.165116+0000", 2024-09-21T12:37:27.921 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6609999999999999e-06 2024-09-21T12:37:27.921 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.922 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.922 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.922 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:59.165281+0000", 2024-09-21T12:37:27.922 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016428100000000001 2024-09-21T12:37:27.922 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.922 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.922 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:27.922 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:59.165325+0000", 2024-09-21T12:37:27.922 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3918999999999997e-05 2024-09-21T12:37:27.922 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.922 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.922 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.922 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:00.121378+0000", 2024-09-21T12:37:27.922 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.95605380299999998 2024-09-21T12:37:27.923 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.923 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.923 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:27.923 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:00.121389+0000", 2024-09-21T12:37:27.923 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0991e-05 2024-09-21T12:37:27.923 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.923 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:27.923 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.923 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.923 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.923 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:34459 11.3 11:ca74ae22:::benchmark_data_smithi114_187622_object34458:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-09-21T12:37:27.923 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:36:12.115486+0000", 2024-09-21T12:37:27.923 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 75.742964162000007, 2024-09-21T12:37:27.923 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.99501541699999996, 2024-09-21T12:37:27.924 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:27.924 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:27.924 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:27.924 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:27.924 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:37:27.924 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:37:27.924 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 34459 2024-09-21T12:37:27.924 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.924 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:27.924 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.924 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:27.924 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115486+0000", 2024-09-21T12:37:27.924 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.924 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.925 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.925 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:27.925 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115486+0000", 2024-09-21T12:37:27.925 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.925 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.925 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.925 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:27.925 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115488+0000", 2024-09-21T12:37:27.925 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4270000000000001e-06 2024-09-21T12:37:27.925 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.925 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.925 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:27.925 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115619+0000", 2024-09-21T12:37:27.925 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013127500000000001 2024-09-21T12:37:27.926 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.926 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.926 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:27.926 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115631+0000", 2024-09-21T12:37:27.926 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1894e-05 2024-09-21T12:37:27.926 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.926 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.926 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:27.926 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115640+0000", 2024-09-21T12:37:27.926 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8789999999999992e-06 2024-09-21T12:37:27.926 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.926 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.926 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.926 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115711+0000", 2024-09-21T12:37:27.927 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0730999999999998e-05 2024-09-21T12:37:27.927 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.927 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.927 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:27.927 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115720+0000", 2024-09-21T12:37:27.927 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0669999999999998e-06 2024-09-21T12:37:27.927 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.927 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.927 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.927 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.107366+0000", 2024-09-21T12:37:27.927 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.99164616100000003 2024-09-21T12:37:27.927 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.927 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.927 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:27.928 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.107531+0000", 2024-09-21T12:37:27.928 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016521400000000001 2024-09-21T12:37:27.928 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.928 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.928 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:37:27.928 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.107671+0000", 2024-09-21T12:37:27.928 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000139915 2024-09-21T12:37:27.928 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.928 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.928 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.928 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.110368+0000", 2024-09-21T12:37:27.928 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002696693 2024-09-21T12:37:27.928 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.928 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.928 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:27.929 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.110378+0000", 2024-09-21T12:37:27.929 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0263e-05 2024-09-21T12:37:27.929 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.929 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.929 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.929 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.110416+0000", 2024-09-21T12:37:27.929 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7540999999999999e-05 2024-09-21T12:37:27.929 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.929 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.929 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:27.929 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.110469+0000", 2024-09-21T12:37:27.929 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3103999999999998e-05 2024-09-21T12:37:27.929 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.929 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.929 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:27.930 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.110502+0000", 2024-09-21T12:37:27.930 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3253000000000002e-05 2024-09-21T12:37:27.930 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.930 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:27.930 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.930 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.930 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.930 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:34460 11.3 11:c0bd347b:::benchmark_data_smithi114_187622_object34459:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-09-21T12:37:27.930 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:36:12.115839+0000", 2024-09-21T12:37:27.930 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 75.742611956000005, 2024-09-21T12:37:27.930 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.99728419700000004, 2024-09-21T12:37:27.930 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:27.930 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:27.930 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:27.931 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:27.931 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:37:27.931 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:37:27.931 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 34460 2024-09-21T12:37:27.931 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.931 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:27.931 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.931 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:27.931 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115839+0000", 2024-09-21T12:37:27.931 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.931 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.931 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.931 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:27.931 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115839+0000", 2024-09-21T12:37:27.931 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.931 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.932 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.932 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:27.932 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115840+0000", 2024-09-21T12:37:27.932 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7000000000000003e-07 2024-09-21T12:37:27.932 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.932 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.932 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:27.932 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115906+0000", 2024-09-21T12:37:27.932 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6925e-05 2024-09-21T12:37:27.932 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.932 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.932 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:27.932 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115918+0000", 2024-09-21T12:37:27.932 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1385e-05 2024-09-21T12:37:27.933 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.933 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.933 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:27.933 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115925+0000", 2024-09-21T12:37:27.933 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8040000000000004e-06 2024-09-21T12:37:27.933 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.933 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.933 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.933 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115973+0000", 2024-09-21T12:37:27.933 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8804999999999998e-05 2024-09-21T12:37:27.933 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.933 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.933 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:27.933 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115986+0000", 2024-09-21T12:37:27.933 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2918000000000001e-05 2024-09-21T12:37:27.934 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.934 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.934 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.934 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.109359+0000", 2024-09-21T12:37:27.934 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.99337215800000001 2024-09-21T12:37:27.934 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.934 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.934 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:27.934 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.109454+0000", 2024-09-21T12:37:27.934 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5828999999999993e-05 2024-09-21T12:37:27.934 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.934 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.934 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:37:27.934 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.109547+0000", 2024-09-21T12:37:27.934 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2757999999999995e-05 2024-09-21T12:37:27.935 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.935 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.935 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:27.935 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.112914+0000", 2024-09-21T12:37:27.935 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0033664300000000001 2024-09-21T12:37:27.935 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.935 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.935 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.935 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.112976+0000", 2024-09-21T12:37:27.935 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2454999999999997e-05 2024-09-21T12:37:27.935 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.935 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.935 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.936 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.113046+0000", 2024-09-21T12:37:27.936 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9919000000000006e-05 2024-09-21T12:37:27.936 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.936 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.936 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:27.936 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.113096+0000", 2024-09-21T12:37:27.936 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9969000000000002e-05 2024-09-21T12:37:27.936 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.936 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.936 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:27.936 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.113123+0000", 2024-09-21T12:37:27.936 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6971999999999999e-05 2024-09-21T12:37:27.936 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.936 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:27.936 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.937 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.937 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.937 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:34461 11.3 11:c2a3c0ef:::benchmark_data_smithi114_187622_object34460:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-09-21T12:37:27.937 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:36:12.116148+0000", 2024-09-21T12:37:27.937 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 75.742302906999996, 2024-09-21T12:37:27.937 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0049646350000001, 2024-09-21T12:37:27.937 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:27.937 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:27.937 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:27.937 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:27.937 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:37:27.937 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:37:27.937 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 34461 2024-09-21T12:37:27.937 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.938 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:27.938 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.938 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:27.938 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.116148+0000", 2024-09-21T12:37:27.938 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.938 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.938 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.938 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:27.938 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.116148+0000", 2024-09-21T12:37:27.938 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.938 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.938 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.938 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:27.938 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.116148+0000", 2024-09-21T12:37:27.938 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5799999999999998e-07 2024-09-21T12:37:27.939 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.939 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.939 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:27.939 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.116220+0000", 2024-09-21T12:37:27.939 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1686000000000004e-05 2024-09-21T12:37:27.939 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.939 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.939 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:27.939 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.116230+0000", 2024-09-21T12:37:27.939 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8819999999999996e-06 2024-09-21T12:37:27.939 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.939 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.939 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:27.939 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.116236+0000", 2024-09-21T12:37:27.940 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2620000000000004e-06 2024-09-21T12:37:27.940 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.940 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.940 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.940 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.116276+0000", 2024-09-21T12:37:27.940 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9833000000000003e-05 2024-09-21T12:37:27.940 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.940 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.940 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:27.940 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.116284+0000", 2024-09-21T12:37:27.940 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5689999999999999e-06 2024-09-21T12:37:27.940 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.940 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.940 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.940 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.110628+0000", 2024-09-21T12:37:27.941 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.99434418899999999 2024-09-21T12:37:27.941 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.941 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.941 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:27.941 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.110769+0000", 2024-09-21T12:37:27.941 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000141354 2024-09-21T12:37:27.941 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.941 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.941 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:37:27.941 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.110926+0000", 2024-09-21T12:37:27.941 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015713899999999999 2024-09-21T12:37:27.941 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.941 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.941 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:27.941 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.113140+0000", 2024-09-21T12:37:27.942 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00221383 2024-09-21T12:37:27.942 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.942 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.942 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.942 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.113204+0000", 2024-09-21T12:37:27.942 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4039000000000001e-05 2024-09-21T12:37:27.942 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.942 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.942 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.942 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.121005+0000", 2024-09-21T12:37:27.942 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0078011649999999997 2024-09-21T12:37:27.942 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.942 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.943 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:27.943 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.121067+0000", 2024-09-21T12:37:27.943 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1832e-05 2024-09-21T12:37:27.943 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.943 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.943 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:27.943 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.121112+0000", 2024-09-21T12:37:27.943 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5096999999999999e-05 2024-09-21T12:37:27.943 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.943 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:27.943 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.943 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.943 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.943 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:34467 11.3 11:c8c61d4a:::benchmark_data_smithi114_187622_object34466:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-09-21T12:37:27.943 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:36:12.125970+0000", 2024-09-21T12:37:27.944 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 75.732480434999999, 2024-09-21T12:37:27.944 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.99832942599999996, 2024-09-21T12:37:27.944 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:27.944 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:27.944 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:27.944 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:27.944 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:37:27.944 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:37:27.944 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 34467 2024-09-21T12:37:27.944 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.944 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:27.944 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.944 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:27.944 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.125970+0000", 2024-09-21T12:37:27.944 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.945 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.945 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.945 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:27.945 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.125970+0000", 2024-09-21T12:37:27.945 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.945 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.945 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.945 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:27.945 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.125972+0000", 2024-09-21T12:37:27.945 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8050000000000001e-06 2024-09-21T12:37:27.945 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.945 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.945 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:27.945 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.126024+0000", 2024-09-21T12:37:27.945 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1699000000000001e-05 2024-09-21T12:37:27.946 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.946 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.946 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:27.946 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.126035+0000", 2024-09-21T12:37:27.946 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1764e-05 2024-09-21T12:37:27.946 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.946 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.946 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:27.946 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.126041+0000", 2024-09-21T12:37:27.946 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5999999999999997e-06 2024-09-21T12:37:27.946 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.946 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.946 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.947 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.126136+0000", 2024-09-21T12:37:27.947 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4429e-05 2024-09-21T12:37:27.947 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.947 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.947 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:27.947 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.126146+0000", 2024-09-21T12:37:27.947 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0387999999999999e-05 2024-09-21T12:37:27.947 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.947 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.947 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.947 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.111722+0000", 2024-09-21T12:37:27.947 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.98557621399999995 2024-09-21T12:37:27.948 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.948 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.948 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:27.948 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.111856+0000", 2024-09-21T12:37:27.948 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013419199999999999 2024-09-21T12:37:27.948 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.948 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.948 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:37:27.948 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.112031+0000", 2024-09-21T12:37:27.948 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000174439 2024-09-21T12:37:27.948 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.948 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.948 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.948 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.113254+0000", 2024-09-21T12:37:27.949 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012237319999999999 2024-09-21T12:37:27.949 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.949 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.949 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:27.949 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.113263+0000", 2024-09-21T12:37:27.949 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2369999999999992e-06 2024-09-21T12:37:27.949 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.949 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.949 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.949 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.124195+0000", 2024-09-21T12:37:27.949 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.010932457 2024-09-21T12:37:27.949 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.949 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.949 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:27.949 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.124258+0000", 2024-09-21T12:37:27.950 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3283999999999999e-05 2024-09-21T12:37:27.950 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.950 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.950 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:27.950 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.124300+0000", 2024-09-21T12:37:27.950 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1186000000000002e-05 2024-09-21T12:37:27.950 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.950 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:27.950 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.950 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.950 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.950 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:34474 11.3 11:c74a366c:::benchmark_data_smithi114_187622_object34473:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-09-21T12:37:27.950 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:36:12.128588+0000", 2024-09-21T12:37:27.950 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 75.729862476999998, 2024-09-21T12:37:27.951 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.99583373799999997, 2024-09-21T12:37:27.951 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:27.951 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:27.951 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:27.951 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:27.951 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:37:27.951 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:37:27.951 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 34474 2024-09-21T12:37:27.951 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.951 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:27.951 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.951 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:27.951 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.128588+0000", 2024-09-21T12:37:27.952 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.952 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.952 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.952 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:27.952 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.128588+0000", 2024-09-21T12:37:27.952 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.952 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.952 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.952 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:27.952 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.128590+0000", 2024-09-21T12:37:27.952 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5060000000000001e-06 2024-09-21T12:37:27.952 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.952 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.952 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:27.952 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.128660+0000", 2024-09-21T12:37:27.953 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9892000000000001e-05 2024-09-21T12:37:27.953 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.953 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.953 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:27.953 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.128673+0000", 2024-09-21T12:37:27.953 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3859000000000001e-05 2024-09-21T12:37:27.953 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.953 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.953 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:27.953 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.128680+0000", 2024-09-21T12:37:27.953 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.888e-06 2024-09-21T12:37:27.953 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.953 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.954 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.954 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.128741+0000", 2024-09-21T12:37:27.954 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0501999999999998e-05 2024-09-21T12:37:27.954 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.954 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.954 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:27.954 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.128751+0000", 2024-09-21T12:37:27.954 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0581000000000001e-05 2024-09-21T12:37:27.954 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.954 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.954 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.954 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.113279+0000", 2024-09-21T12:37:27.954 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.98452719099999997 2024-09-21T12:37:27.954 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.955 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.955 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:27.955 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.113377+0000", 2024-09-21T12:37:27.955 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8591999999999994e-05 2024-09-21T12:37:27.955 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.955 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.955 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:37:27.955 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.113469+0000", 2024-09-21T12:37:27.955 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1896000000000002e-05 2024-09-21T12:37:27.955 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.955 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.955 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.955 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.116868+0000", 2024-09-21T12:37:27.956 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0033991070000000002 2024-09-21T12:37:27.956 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.956 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.956 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:27.956 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.118613+0000", 2024-09-21T12:37:27.956 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0017451529999999999 2024-09-21T12:37:27.956 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.956 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.956 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.956 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.124339+0000", 2024-09-21T12:37:27.956 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0057257920000000004 2024-09-21T12:37:27.956 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.956 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.956 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:27.957 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.124386+0000", 2024-09-21T12:37:27.957 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6699000000000001e-05 2024-09-21T12:37:27.957 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.957 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.957 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:27.957 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.124422+0000", 2024-09-21T12:37:27.957 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.608e-05 2024-09-21T12:37:27.957 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.957 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:27.957 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.957 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.957 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.957 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:34483 11.3 11:ce099e96:::benchmark_data_smithi114_187622_object34482:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-09-21T12:37:27.957 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:36:12.139319+0000", 2024-09-21T12:37:27.957 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 75.719132080999998, 2024-09-21T12:37:27.958 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.98632494900000001, 2024-09-21T12:37:27.958 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:27.958 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:27.958 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:27.958 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:27.958 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:37:27.958 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:37:27.958 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 34483 2024-09-21T12:37:27.958 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.958 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:27.958 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.958 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:27.958 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.139319+0000", 2024-09-21T12:37:27.958 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.958 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.959 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.959 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:27.959 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.139319+0000", 2024-09-21T12:37:27.959 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.959 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.959 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.959 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:27.959 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.139321+0000", 2024-09-21T12:37:27.959 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9870000000000002e-06 2024-09-21T12:37:27.959 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.959 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.959 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:27.959 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.139360+0000", 2024-09-21T12:37:27.960 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9413999999999999e-05 2024-09-21T12:37:27.960 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.960 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.960 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:27.960 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.139376+0000", 2024-09-21T12:37:27.960 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5662999999999999e-05 2024-09-21T12:37:27.960 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.960 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.960 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:27.960 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.139384+0000", 2024-09-21T12:37:27.960 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2060000000000007e-06 2024-09-21T12:37:27.962 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.962 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.962 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.962 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.139441+0000", 2024-09-21T12:37:27.962 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7043999999999998e-05 2024-09-21T12:37:27.962 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.962 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.962 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:27.962 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.139453+0000", 2024-09-21T12:37:27.962 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1671999999999999e-05 2024-09-21T12:37:27.962 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.962 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.963 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.963 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.114270+0000", 2024-09-21T12:37:27.963 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.97481730600000005 2024-09-21T12:37:27.963 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.963 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.963 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:27.963 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.114384+0000", 2024-09-21T12:37:27.963 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011422299999999999 2024-09-21T12:37:27.963 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.963 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.963 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:37:27.963 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.114501+0000", 2024-09-21T12:37:27.963 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011684999999999999 2024-09-21T12:37:27.963 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.964 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.964 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.964 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.120883+0000", 2024-09-21T12:37:27.964 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0063821340000000002 2024-09-21T12:37:27.964 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.964 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.964 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:27.964 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.122831+0000", 2024-09-21T12:37:27.964 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0019476070000000001 2024-09-21T12:37:27.964 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.964 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.964 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.964 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.124475+0000", 2024-09-21T12:37:27.964 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001643931 2024-09-21T12:37:27.965 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.965 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.965 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:27.965 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.125606+0000", 2024-09-21T12:37:27.965 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001131284 2024-09-21T12:37:27.965 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.965 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.965 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:27.965 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.125644+0000", 2024-09-21T12:37:27.965 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7628e-05 2024-09-21T12:37:27.965 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.965 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:27.965 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.965 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.965 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.966 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:34502 11.3 11:c044b555:::benchmark_data_smithi114_187622_object34501:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-09-21T12:37:27.966 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:36:12.146909+0000", 2024-09-21T12:37:27.966 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 75.711541322000002, 2024-09-21T12:37:27.966 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.97888777299999996, 2024-09-21T12:37:27.966 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:27.966 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:27.966 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:27.966 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:27.966 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:37:27.966 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:37:27.966 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 34502 2024-09-21T12:37:27.966 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.966 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:27.966 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.967 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:27.967 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.146909+0000", 2024-09-21T12:37:27.967 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.967 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.967 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.967 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:27.967 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.146909+0000", 2024-09-21T12:37:27.967 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.967 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.967 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.967 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:27.967 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.146911+0000", 2024-09-21T12:37:27.967 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5409999999999999e-06 2024-09-21T12:37:27.967 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.967 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.968 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:27.968 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.146948+0000", 2024-09-21T12:37:27.968 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7619e-05 2024-09-21T12:37:27.968 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.968 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.968 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:27.968 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.146961+0000", 2024-09-21T12:37:27.968 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2254e-05 2024-09-21T12:37:27.968 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.968 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.968 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:27.968 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.146968+0000", 2024-09-21T12:37:27.968 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0469999999999996e-06 2024-09-21T12:37:27.968 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.969 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.969 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.969 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.147037+0000", 2024-09-21T12:37:27.969 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9210999999999999e-05 2024-09-21T12:37:27.969 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.969 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.969 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:27.969 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.147047+0000", 2024-09-21T12:37:27.969 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6460000000000005e-06 2024-09-21T12:37:27.969 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.969 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.969 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.969 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.115516+0000", 2024-09-21T12:37:27.969 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.96846887999999998 2024-09-21T12:37:27.970 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.970 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.970 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:27.970 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.115620+0000", 2024-09-21T12:37:27.970 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000104876 2024-09-21T12:37:27.970 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.970 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.970 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:37:27.970 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.115705+0000", 2024-09-21T12:37:27.970 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4994999999999995e-05 2024-09-21T12:37:27.970 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.970 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.970 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.970 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.120929+0000", 2024-09-21T12:37:27.970 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0052235110000000001 2024-09-21T12:37:27.971 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.971 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.971 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:27.971 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.122835+0000", 2024-09-21T12:37:27.971 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0019065779999999999 2024-09-21T12:37:27.971 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.971 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.971 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.971 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.125706+0000", 2024-09-21T12:37:27.971 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0028708330000000002 2024-09-21T12:37:27.971 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.971 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.971 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:27.972 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.125759+0000", 2024-09-21T12:37:27.972 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2596000000000003e-05 2024-09-21T12:37:27.972 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.972 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.972 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:27.972 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.125797+0000", 2024-09-21T12:37:27.972 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8186000000000003e-05 2024-09-21T12:37:27.972 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.972 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:27.972 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.972 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.972 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.972 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:34533 11.3 11:cd9680cc:::benchmark_data_smithi114_187622_object34532:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-09-21T12:37:27.972 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:36:12.224372+0000", 2024-09-21T12:37:27.973 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 75.634078329999994, 2024-09-21T12:37:27.973 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.90168315700000001, 2024-09-21T12:37:27.973 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:27.973 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:27.973 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:27.973 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:27.973 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:37:27.973 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:37:27.973 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 34533 2024-09-21T12:37:27.973 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.973 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:27.973 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.973 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:27.973 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.224372+0000", 2024-09-21T12:37:27.974 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.974 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.974 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.974 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:27.974 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.224372+0000", 2024-09-21T12:37:27.974 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.974 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.974 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.974 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:27.974 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.224382+0000", 2024-09-21T12:37:27.974 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7000000000000003e-06 2024-09-21T12:37:27.974 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.974 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.975 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:27.975 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.224432+0000", 2024-09-21T12:37:27.975 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0099000000000003e-05 2024-09-21T12:37:27.975 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.975 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.975 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:27.975 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.224443+0000", 2024-09-21T12:37:27.975 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0525e-05 2024-09-21T12:37:27.975 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.975 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.975 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:27.975 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.224449+0000", 2024-09-21T12:37:27.975 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9780000000000002e-06 2024-09-21T12:37:27.975 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.976 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.976 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.976 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.224500+0000", 2024-09-21T12:37:27.976 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1669000000000003e-05 2024-09-21T12:37:27.976 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.976 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.976 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:27.976 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.224507+0000", 2024-09-21T12:37:27.976 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5529999999999998e-06 2024-09-21T12:37:27.976 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.976 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.976 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.976 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.116961+0000", 2024-09-21T12:37:27.977 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.89245421599999997 2024-09-21T12:37:27.977 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.977 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.977 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:27.977 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.117083+0000", 2024-09-21T12:37:27.977 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012155 2024-09-21T12:37:27.977 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.977 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.977 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:37:27.977 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.117234+0000", 2024-09-21T12:37:27.977 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015182300000000001 2024-09-21T12:37:27.977 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.977 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.978 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:27.978 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.122842+0000", 2024-09-21T12:37:27.978 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0056078320000000001 2024-09-21T12:37:27.978 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.978 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.978 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.978 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.125852+0000", 2024-09-21T12:37:27.978 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0030097890000000001 2024-09-21T12:37:27.978 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.978 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.978 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.978 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.125972+0000", 2024-09-21T12:37:27.978 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012015000000000001 2024-09-21T12:37:27.979 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.979 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.979 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:27.979 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.126021+0000", 2024-09-21T12:37:27.979 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8993999999999999e-05 2024-09-21T12:37:27.979 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.979 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.979 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:27.979 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.126055+0000", 2024-09-21T12:37:27.979 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4279000000000002e-05 2024-09-21T12:37:27.979 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.979 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:27.979 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.979 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.980 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.980 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:34535 11.3 11:c8ed66ca:::benchmark_data_smithi114_187622_object34534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-09-21T12:37:27.980 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:36:12.227358+0000", 2024-09-21T12:37:27.980 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 75.631093140000004, 2024-09-21T12:37:27.980 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.90690378699999996, 2024-09-21T12:37:27.980 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:27.980 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:27.980 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:27.980 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:27.980 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:37:27.980 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:37:27.980 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 34535 2024-09-21T12:37:27.980 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.980 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:27.981 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.981 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:27.981 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.227358+0000", 2024-09-21T12:37:27.981 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.981 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.981 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.981 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:27.981 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.227358+0000", 2024-09-21T12:37:27.981 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.981 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.981 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.981 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:27.981 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.227359+0000", 2024-09-21T12:37:27.981 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2669999999999999e-06 2024-09-21T12:37:27.981 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.982 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.982 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:27.982 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.227388+0000", 2024-09-21T12:37:27.982 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9192000000000001e-05 2024-09-21T12:37:27.982 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.982 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.982 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:27.982 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.227399+0000", 2024-09-21T12:37:27.982 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1494e-05 2024-09-21T12:37:27.982 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.982 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.982 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:27.982 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.227405+0000", 2024-09-21T12:37:27.983 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5860000000000004e-06 2024-09-21T12:37:27.983 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.983 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.983 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.983 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.227450+0000", 2024-09-21T12:37:27.983 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.545e-05 2024-09-21T12:37:27.983 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.983 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.983 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:37:27.983 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.227456+0000", 2024-09-21T12:37:27.983 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.665e-06 2024-09-21T12:37:27.983 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.983 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.983 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.984 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.119175+0000", 2024-09-21T12:37:27.984 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.89171925100000005 2024-09-21T12:37:27.984 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.984 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.984 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:27.984 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.119306+0000", 2024-09-21T12:37:27.984 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013061099999999999 2024-09-21T12:37:27.984 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.984 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.984 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:37:27.984 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.119472+0000", 2024-09-21T12:37:27.984 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016553499999999999 2024-09-21T12:37:27.984 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.984 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.984 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:27.985 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.123128+0000", 2024-09-21T12:37:27.985 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003655957 2024-09-21T12:37:27.985 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.985 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.985 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.985 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.125911+0000", 2024-09-21T12:37:27.985 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0027830950000000002 2024-09-21T12:37:27.985 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.985 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.985 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.985 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.134151+0000", 2024-09-21T12:37:27.985 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0082405670000000007 2024-09-21T12:37:27.985 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.985 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.985 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:27.986 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.134217+0000", 2024-09-21T12:37:27.986 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5687999999999999e-05 2024-09-21T12:37:27.986 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.986 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.986 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:27.986 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.134261+0000", 2024-09-21T12:37:27.986 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4428999999999998e-05 2024-09-21T12:37:27.986 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.986 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:27.986 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.986 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.986 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.986 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:136649 11.3 11:cddb3060:::benchmark_data_smithi114_187622_object136648:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-09-21T12:37:27.987 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:37:22.242348+0000", 2024-09-21T12:37:27.987 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 5.6161026390000002, 2024-09-21T12:37:27.987 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.130686786, 2024-09-21T12:37:27.987 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:27.987 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:27.987 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:27.987 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:27.987 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:37:27.987 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:37:27.987 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 136649 2024-09-21T12:37:27.987 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.987 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:27.987 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.987 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:27.987 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.242348+0000", 2024-09-21T12:37:27.988 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.988 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.988 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.988 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:27.988 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.242348+0000", 2024-09-21T12:37:27.988 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.988 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.988 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.988 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:27.988 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.242349+0000", 2024-09-21T12:37:27.988 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2500000000000001e-06 2024-09-21T12:37:27.988 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.988 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.988 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:27.988 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.242395+0000", 2024-09-21T12:37:27.989 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6026999999999998e-05 2024-09-21T12:37:27.989 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.989 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.989 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:27.989 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.242405+0000", 2024-09-21T12:37:27.989 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5519999999999993e-06 2024-09-21T12:37:27.989 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.989 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.989 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:27.989 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.242410+0000", 2024-09-21T12:37:27.989 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2270000000000001e-06 2024-09-21T12:37:27.989 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.989 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.990 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.990 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.242455+0000", 2024-09-21T12:37:27.990 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4888999999999997e-05 2024-09-21T12:37:27.990 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.990 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.990 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:27.990 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.242553+0000", 2024-09-21T12:37:27.990 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8227000000000007e-05 2024-09-21T12:37:27.990 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.990 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.990 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:37:27.990 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.242638+0000", 2024-09-21T12:37:27.990 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4963000000000006e-05 2024-09-21T12:37:27.990 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.991 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.991 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:27.991 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.246352+0000", 2024-09-21T12:37:27.991 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0037137450000000001 2024-09-21T12:37:27.991 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.991 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.991 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.991 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.248100+0000", 2024-09-21T12:37:27.991 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0017482450000000001 2024-09-21T12:37:27.991 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.991 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.991 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.991 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.372877+0000", 2024-09-21T12:37:27.991 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.124776927 2024-09-21T12:37:27.992 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.992 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.992 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:27.992 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.372986+0000", 2024-09-21T12:37:27.992 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000108646 2024-09-21T12:37:27.992 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.992 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.992 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:27.992 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.373035+0000", 2024-09-21T12:37:27.992 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9088000000000002e-05 2024-09-21T12:37:27.992 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.992 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:27.992 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.992 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.993 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.993 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:136650 11.3 11:ce8eedbf:::benchmark_data_smithi114_187622_object136649:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-09-21T12:37:27.993 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:37:22.242424+0000", 2024-09-21T12:37:27.993 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 5.6160264269999995, 2024-09-21T12:37:27.993 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.130759292, 2024-09-21T12:37:27.993 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:27.993 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:27.993 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:37:27.993 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:27.993 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:37:27.993 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:37:27.993 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 136650 2024-09-21T12:37:27.993 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.993 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:27.993 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.994 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:27.994 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.242424+0000", 2024-09-21T12:37:27.994 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.994 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.994 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.994 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:27.994 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.242424+0000", 2024-09-21T12:37:27.994 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:27.994 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.994 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.994 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:27.994 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.242425+0000", 2024-09-21T12:37:27.994 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8500000000000002e-07 2024-09-21T12:37:27.995 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.995 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.995 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:27.995 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.242501+0000", 2024-09-21T12:37:27.995 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6328000000000005e-05 2024-09-21T12:37:27.995 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.995 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.995 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:27.995 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.242506+0000", 2024-09-21T12:37:27.995 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4689999999999996e-06 2024-09-21T12:37:27.995 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.995 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.995 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:27.996 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.242510+0000", 2024-09-21T12:37:27.996 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5920000000000001e-06 2024-09-21T12:37:27.996 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.996 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.996 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:27.996 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.244455+0000", 2024-09-21T12:37:27.996 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001944519 2024-09-21T12:37:27.996 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.996 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.996 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:27.996 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.244547+0000", 2024-09-21T12:37:27.996 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2243000000000004e-05 2024-09-21T12:37:27.996 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.996 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.997 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:37:27.997 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.244676+0000", 2024-09-21T12:37:27.997 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012941499999999999 2024-09-21T12:37:27.997 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.997 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.997 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:37:27.997 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.246433+0000", 2024-09-21T12:37:27.997 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0017571799999999999 2024-09-21T12:37:27.997 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.997 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.997 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.997 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.249840+0000", 2024-09-21T12:37:27.997 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003406999 2024-09-21T12:37:27.997 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.998 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.998 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:37:27.998 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.373095+0000", 2024-09-21T12:37:27.998 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.123254802 2024-09-21T12:37:27.998 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.998 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.998 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:37:27.998 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.373148+0000", 2024-09-21T12:37:27.998 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2561e-05 2024-09-21T12:37:27.998 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:27.998 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:27.998 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:37:27.998 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.373184+0000", 2024-09-21T12:37:27.998 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5799000000000001e-05 2024-09-21T12:37:27.999 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.999 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:27.999 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.999 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:27.999 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:27.999 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:37:28.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:28.000 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:28.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:28.101 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:28.122 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:28.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:28.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:28.223 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:28.256 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:28.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:28.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:28.358 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:28.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:28.393 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:28.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:28.494 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:28.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:28.527 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:28.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:28.628 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:28.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:28.663 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:28.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:28.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:28 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:37:28.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:28 smithi183 ceph-mon[33277]: pgmap v878: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 8.1 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 1.8 KiB/s rd, 62 MiB/s wr, 1.90k op/s; 33399/398700 objects degraded (8.377%); 16668/398700 objects misplaced (4.181%) 2024-09-21T12:37:28.764 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:28.768 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:28 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:37:28.768 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:28 smithi114 ceph-mon[40867]: pgmap v878: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 8.1 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 1.8 KiB/s rd, 62 MiB/s wr, 1.90k op/s; 33399/398700 objects degraded (8.377%); 16668/398700 objects misplaced (4.181%) 2024-09-21T12:37:28.768 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:28 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:37:28.768 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:28 smithi114 ceph-mon[30324]: pgmap v878: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 8.1 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 1.8 KiB/s rd, 62 MiB/s wr, 1.90k op/s; 33399/398700 objects degraded (8.377%); 16668/398700 objects misplaced (4.181%) 2024-09-21T12:37:28.801 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:28.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:28.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:28.858 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T12:37:28.902 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:28.940 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:28.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:28.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:29.042 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:29.077 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:29.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:29.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:29.179 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:29.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:29.218 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:29.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:29.252 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:37:29.319 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:29.357 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:29.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:29.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:29.424 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T12:37:29.459 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:29.487 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:29.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:29.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:29.589 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:29.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:29.628 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:29.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:29.729 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:29.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:29.761 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:29.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:29.862 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:29.885 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:37:29.885 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:37:29.885 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.885 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.26147.0:147888 11.e e446/417 11:771e79b7:::benchmark_data_smithi114_187622_object9767:head v 446'9282, mlcod=446'9279)", 2024-09-21T12:37:29.885 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:37:29.880858+0000", 2024-09-21T12:37:29.885 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.001615666, 2024-09-21T12:37:29.885 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001641254, 2024-09-21T12:37:29.886 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:29.886 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:29.886 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:37:29.886 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:29.886 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.886 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:29.886 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:29.880858+0000", 2024-09-21T12:37:29.886 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:29.886 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:29.886 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.886 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:29.886 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:29.880858+0000", 2024-09-21T12:37:29.886 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:29.887 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:29.887 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.887 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:29.887 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:29.880859+0000", 2024-09-21T12:37:29.887 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1900000000000001e-07 2024-09-21T12:37:29.887 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:29.887 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.887 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:29.887 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:29.880865+0000", 2024-09-21T12:37:29.887 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1959999999999996e-06 2024-09-21T12:37:29.887 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:29.887 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.887 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:29.887 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:29.880878+0000", 2024-09-21T12:37:29.888 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3302000000000001e-05 2024-09-21T12:37:29.888 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:29.888 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.888 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:29.888 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:29.880885+0000", 2024-09-21T12:37:29.888 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4790000000000001e-06 2024-09-21T12:37:29.888 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:29.888 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.888 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:29.888 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:29.880958+0000", 2024-09-21T12:37:29.888 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3221999999999998e-05 2024-09-21T12:37:29.888 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:29.888 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.888 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:29.888 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:29.880975+0000", 2024-09-21T12:37:29.888 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7051e-05 2024-09-21T12:37:29.889 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:29.889 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:29.889 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:29.889 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:29.889 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.889 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:147896 11.f 11:ffa8e00d:::benchmark_data_smithi114_187622_object9775:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e446)", 2024-09-21T12:37:29.889 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:37:29.881407+0000", 2024-09-21T12:37:29.889 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0010671000000000001, 2024-09-21T12:37:29.889 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0011492200000000001, 2024-09-21T12:37:29.889 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:29.889 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:29.889 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:37:29.889 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:29.889 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:37:29.889 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:37:29.890 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 147896 2024-09-21T12:37:29.890 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:29.890 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:29.890 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.890 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:29.890 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:29.881407+0000", 2024-09-21T12:37:29.890 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:29.890 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:29.890 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.890 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:29.890 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:29.881407+0000", 2024-09-21T12:37:29.890 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:29.890 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:29.890 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.890 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:29.891 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:29.881408+0000", 2024-09-21T12:37:29.891 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.562e-06 2024-09-21T12:37:29.891 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:29.891 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.891 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:29.891 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:29.881412+0000", 2024-09-21T12:37:29.891 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0389999999999998e-06 2024-09-21T12:37:29.891 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:29.891 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.891 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:29.891 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:29.881424+0000", 2024-09-21T12:37:29.891 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2165e-05 2024-09-21T12:37:29.891 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:29.891 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.891 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:29.892 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:29.881432+0000", 2024-09-21T12:37:29.892 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.588e-06 2024-09-21T12:37:29.892 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:29.892 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.892 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:37:29.892 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:29.882087+0000", 2024-09-21T12:37:29.892 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00065532899999999996 2024-09-21T12:37:29.892 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:29.892 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.892 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:37:29.892 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:29.882212+0000", 2024-09-21T12:37:29.892 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012478500000000001 2024-09-21T12:37:29.892 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:29.892 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.893 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1", 2024-09-21T12:37:29.893 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:29.882325+0000", 2024-09-21T12:37:29.893 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011294 2024-09-21T12:37:29.893 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:29.893 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:29.893 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:29.893 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:29.893 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.893 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.26147.0:147892 11.e e446/417)", 2024-09-21T12:37:29.893 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:37:29.881520+0000", 2024-09-21T12:37:29.893 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.00095378599999999998, 2024-09-21T12:37:29.893 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010951069999999999, 2024-09-21T12:37:29.893 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:29.893 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:29.893 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-09-21T12:37:29.894 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:29.894 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.894 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:29.894 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:29.881520+0000", 2024-09-21T12:37:29.894 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:29.894 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:29.894 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.894 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:29.894 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:29.881520+0000", 2024-09-21T12:37:29.894 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:29.894 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:29.894 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.894 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:29.894 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:29.881520+0000", 2024-09-21T12:37:29.894 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3999999999999997e-07 2024-09-21T12:37:29.895 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:29.895 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.895 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:29.895 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:29.881524+0000", 2024-09-21T12:37:29.895 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4010000000000001e-06 2024-09-21T12:37:29.895 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:29.895 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.895 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:29.895 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:29.881531+0000", 2024-09-21T12:37:29.895 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0230000000000004e-06 2024-09-21T12:37:29.895 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:29.895 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.895 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:29.895 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:29.881535+0000", 2024-09-21T12:37:29.895 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7759999999999997e-06 2024-09-21T12:37:29.896 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:29.896 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:29.896 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:29.896 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:29.896 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.896 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:147900 11.f 11.d576f71f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e446)", 2024-09-21T12:37:29.896 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:37:29.882222+0000", 2024-09-21T12:37:29.896 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.00025123799999999998, 2024-09-21T12:37:29.896 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00042781699999999999, 2024-09-21T12:37:29.896 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:37:29.896 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:37:29.896 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-09-21T12:37:29.896 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:37:29.896 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:37:29.896 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:37:29.896 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 147900 2024-09-21T12:37:29.897 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:29.897 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:37:29.897 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.897 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:37:29.897 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:29.882222+0000", 2024-09-21T12:37:29.897 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:29.897 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:29.897 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.897 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:37:29.897 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:29.882222+0000", 2024-09-21T12:37:29.897 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:37:29.897 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:29.897 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.897 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:37:29.897 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:29.882223+0000", 2024-09-21T12:37:29.897 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1500000000000003e-07 2024-09-21T12:37:29.898 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:29.898 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.898 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:37:29.898 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:29.882227+0000", 2024-09-21T12:37:29.898 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4089999999999999e-06 2024-09-21T12:37:29.898 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:29.898 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.898 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:37:29.898 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:29.882239+0000", 2024-09-21T12:37:29.898 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1843e-05 2024-09-21T12:37:29.898 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:37:29.898 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:37:29.898 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:37:29.898 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:29.882246+0000", 2024-09-21T12:37:29.899 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2169999999999997e-06 2024-09-21T12:37:29.899 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:29.899 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:37:29.899 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:29.899 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:37:29.899 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-21T12:37:29.899 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 4 2024-09-21T12:37:29.899 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:37:29.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:29.900 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:29.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:30.001 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:30.036 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:30.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:30.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:30.133 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:37:30.138 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:37:30.138 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:30.200 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:30.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:30.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:30.301 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:30.331 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:30.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:30.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:30.433 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:30.453 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:30.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:30.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:30.555 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:30.589 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:30.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:30.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:30.691 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:30.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:30.725 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:30.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:30.826 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:30.860 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:30.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:30.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:30.892 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T12:37:30.962 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:30.998 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:30.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:30.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:31.100 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:31.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:31.136 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:31.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:31.143 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:30 smithi114 ceph-mon[40867]: from='client.26288 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:37:31.143 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:30 smithi114 ceph-mon[40867]: pgmap v879: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 8.1 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 790 B/s rd, 30 MiB/s wr, 1.25k op/s; 32930/397293 objects degraded (8.289%); 16668/397293 objects misplaced (4.195%) 2024-09-21T12:37:31.143 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:30 smithi114 ceph-mon[30324]: from='client.26288 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:37:31.143 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:30 smithi114 ceph-mon[30324]: pgmap v879: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 8.1 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 790 B/s rd, 30 MiB/s wr, 1.25k op/s; 32930/397293 objects degraded (8.289%); 16668/397293 objects misplaced (4.195%) 2024-09-21T12:37:31.234 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:37:31.235 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:37:31.235 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:37:31.235 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:37:31.235 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:37:31.235 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:37:31.235 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:37:31.235 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:37:31.235 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T12:37:31.235 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:37:31.236 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:37:31.236 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:37:31.236 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:37:31.237 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:31.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:30 smithi183 ceph-mon[33277]: from='client.26288 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:37:31.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:30 smithi183 ceph-mon[33277]: pgmap v879: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 8.1 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 790 B/s rd, 30 MiB/s wr, 1.25k op/s; 32930/397293 objects degraded (8.289%); 16668/397293 objects misplaced (4.195%) 2024-09-21T12:37:31.252 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:37:31.252 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:37:31.271 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:31.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:31.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:31.347 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:37:31.373 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:31.407 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:31.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:31.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:31.509 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:31.545 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:31.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:31.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:31.647 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:31.680 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:31.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:31.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:31.781 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:31.814 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:31.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:31.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:31.916 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:31.944 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:31.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:31.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:32.009 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:37:32.009 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:37:32.046 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:32.079 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:32.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:32.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:32.100 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":878,"stamp":"2024-09-21T12:37:27.891948+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8697223731,"num_objects":132900,"num_object_clones":0,"num_object_copies":398700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33399,"num_objects_misplaced":16668,"num_objects_unfound":0,"num_objects_dirty":132900,"num_whiteouts":0,"num_read":1213,"num_read_kb":965,"num_write":144249,"num_write_kb":8839278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4697,"num_bytes_recovered":275065495,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69062,"ondisk_log_size":69062,"up":481,"acting":501,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":553,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":31731704,"kb_used_data":25871912,"kb_used_omap":12,"kb_used_meta":5859443,"kb_avail":718229512,"statfs":{"total":767960285184,"available":735467020288,"internally_reserved":0,"allocated":26492837888,"data_stored":26488241698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":6000070192},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,7,8,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36044824,"num_objects":551,"num_object_clones":0,"num_object_copies":1653,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":380,"num_objects_misplaced":240,"num_objects_unfound":0,"num_objects_dirty":551,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":11377,"num_write_kb":381633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002443"},"pg_stats":[{"pgid":"1.1d","version":"76'79","reported_seq":522,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.519789+0000","last_change":"2024-09-21T12:37:16.858542+0000","last_active":"2024-09-21T12:37:26.519789+0000","last_peered":"2024-09-21T12:37:26.519789+0000","last_clean":"2024-09-21T12:37:26.519789+0000","last_became_active":"2024-09-21T12:32:38.426383+0000","last_became_peered":"2024-09-21T12:32:38.426383+0000","last_unstale":"2024-09-21T12:37:26.519789+0000","last_undegraded":"2024-09-21T12:37:26.519789+0000","last_fullsized":"2024-09-21T12:37:26.519789+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:37:16.858472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:37:16.858472+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:31.571828+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":826,"reported_epoch":444,"state":"active+undersized","last_fresh":"2024-09-21T12:37:22.129165+0000","last_change":"2024-09-21T12:34:34.340741+0000","last_active":"2024-09-21T12:37:22.129165+0000","last_peered":"2024-09-21T12:37:22.129165+0000","last_clean":"2024-09-21T12:34:29.289357+0000","last_became_active":"2024-09-21T12:34:34.340741+0000","last_became_peered":"2024-09-21T12:34:34.340741+0000","last_unstale":"2024-09-21T12:37:22.129165+0000","last_undegraded":"2024-09-21T12:37:22.129165+0000","last_fullsized":"2024-09-21T12:34:34.332209+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:29.086830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:29.086830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:31.924550+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":221,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.529352+0000","last_change":"2024-09-21T12:37:21.750122+0000","last_active":"2024-09-21T12:37:26.529352+0000","last_peered":"2024-09-21T12:37:26.529352+0000","last_clean":"2024-09-21T12:37:26.529352+0000","last_became_active":"2024-09-21T12:32:38.419081+0000","last_became_peered":"2024-09-21T12:32:38.419081+0000","last_unstale":"2024-09-21T12:37:26.529352+0000","last_undegraded":"2024-09-21T12:37:26.529352+0000","last_fullsized":"2024-09-21T12:37:26.529352+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:37:21.750033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:37:21.750033+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:28.292019+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":834,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.519189+0000","last_change":"2024-09-21T12:37:21.767752+0000","last_active":"2024-09-21T12:37:26.519189+0000","last_peered":"2024-09-21T12:37:26.519189+0000","last_clean":"2024-09-21T12:37:26.519189+0000","last_became_active":"2024-09-21T12:36:00.136673+0000","last_became_peered":"2024-09-21T12:36:00.136673+0000","last_unstale":"2024-09-21T12:37:26.519189+0000","last_undegraded":"2024-09-21T12:37:26.519189+0000","last_fullsized":"2024-09-21T12:37:26.519189+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:21.767691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:37:21.767691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:33.819470+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":219,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.530311+0000","last_change":"2024-09-21T12:36:44.729701+0000","last_active":"2024-09-21T12:37:26.530311+0000","last_peered":"2024-09-21T12:37:26.530311+0000","last_clean":"2024-09-21T12:37:26.530311+0000","last_became_active":"2024-09-21T12:32:38.425588+0000","last_became_peered":"2024-09-21T12:32:38.425588+0000","last_unstale":"2024-09-21T12:37:26.530311+0000","last_undegraded":"2024-09-21T12:37:26.530311+0000","last_fullsized":"2024-09-21T12:37:26.530311+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:44.729655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:44.729655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:58.553964+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"76'79","reported_seq":80,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.520557+0000","last_change":"2024-09-21T12:37:19.644089+0000","last_active":"2024-09-21T12:37:26.520557+0000","last_peered":"2024-09-21T12:37:26.520557+0000","last_clean":"2024-09-21T12:37:26.520557+0000","last_became_active":"2024-09-21T12:36:00.154209+0000","last_became_peered":"2024-09-21T12:36:00.154209+0000","last_unstale":"2024-09-21T12:37:26.520557+0000","last_undegraded":"2024-09-21T12:37:26.520557+0000","last_fullsized":"2024-09-21T12:37:26.520557+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:37:19.644030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:37:19.644030+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:25.710101+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":815,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.135673+0000","last_change":"2024-09-21T12:36:00.139483+0000","last_active":"2024-09-21T12:37:27.135673+0000","last_peered":"2024-09-21T12:37:27.135673+0000","last_clean":"2024-09-21T12:37:27.135673+0000","last_became_active":"2024-09-21T12:36:00.139167+0000","last_became_peered":"2024-09-21T12:36:00.139167+0000","last_unstale":"2024-09-21T12:37:27.135673+0000","last_undegraded":"2024-09-21T12:37:27.135673+0000","last_fullsized":"2024-09-21T12:37:27.135673+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:05.904553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:34:05.904553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:03.748543+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":827,"reported_epoch":446,"state":"active+undersized","last_fresh":"2024-09-21T12:37:27.137040+0000","last_change":"2024-09-21T12:34:34.346389+0000","last_active":"2024-09-21T12:37:27.137040+0000","last_peered":"2024-09-21T12:37:27.137040+0000","last_clean":"2024-09-21T12:34:29.294384+0000","last_became_active":"2024-09-21T12:34:34.346389+0000","last_became_peered":"2024-09-21T12:34:34.346389+0000","last_unstale":"2024-09-21T12:37:27.137040+0000","last_undegraded":"2024-09-21T12:37:27.137040+0000","last_fullsized":"2024-09-21T12:34:34.336359+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:06.698999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:06.698999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:34.545818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":865,"reported_epoch":446,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:37:27.135224+0000","last_change":"2024-09-21T12:34:34.337065+0000","last_active":"2024-09-21T12:37:27.135224+0000","last_peered":"2024-09-21T12:37:27.135224+0000","last_clean":"2024-09-21T12:34:29.291199+0000","last_became_active":"2024-09-21T12:34:34.337065+0000","last_became_peered":"2024-09-21T12:34:34.337065+0000","last_unstale":"2024-09-21T12:37:27.135224+0000","last_undegraded":"2024-09-21T12:34:34.335679+0000","last_fullsized":"2024-09-21T12:34:34.335428+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:31:48.294300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:48.294300+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:39.494442+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":211,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.529272+0000","last_change":"2024-09-21T12:36:35.622295+0000","last_active":"2024-09-21T12:37:26.529272+0000","last_peered":"2024-09-21T12:37:26.529272+0000","last_clean":"2024-09-21T12:37:26.529272+0000","last_became_active":"2024-09-21T12:32:38.417724+0000","last_became_peered":"2024-09-21T12:32:38.417724+0000","last_unstale":"2024-09-21T12:37:26.529272+0000","last_undegraded":"2024-09-21T12:37:26.529272+0000","last_fullsized":"2024-09-21T12:37:26.529272+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:35.622232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:35.622232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:53.247230+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":372,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.525915+0000","last_change":"2024-09-21T12:37:25.297240+0000","last_active":"2024-09-21T12:37:26.525915+0000","last_peered":"2024-09-21T12:37:26.525915+0000","last_clean":"2024-09-21T12:37:26.525915+0000","last_became_active":"2024-09-21T12:37:20.467925+0000","last_became_peered":"2024-09-21T12:37:20.467925+0000","last_unstale":"2024-09-21T12:37:26.525915+0000","last_undegraded":"2024-09-21T12:37:26.525915+0000","last_fullsized":"2024-09-21T12:37:26.525915+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:25.297200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:37:25.297200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:38.647772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":229,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.530394+0000","last_change":"2024-09-21T12:36:42.653161+0000","last_active":"2024-09-21T12:37:26.530394+0000","last_peered":"2024-09-21T12:37:26.530394+0000","last_clean":"2024-09-21T12:37:26.530394+0000","last_became_active":"2024-09-21T12:32:38.426121+0000","last_became_peered":"2024-09-21T12:32:38.426121+0000","last_unstale":"2024-09-21T12:37:26.530394+0000","last_undegraded":"2024-09-21T12:37:26.530394+0000","last_fullsized":"2024-09-21T12:37:26.530394+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:36:42.653075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:42.653075+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:44.149637+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":864,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.524338+0000","last_change":"2024-09-21T12:36:00.138341+0000","last_active":"2024-09-21T12:37:26.524338+0000","last_peered":"2024-09-21T12:37:26.524338+0000","last_clean":"2024-09-21T12:37:26.524338+0000","last_became_active":"2024-09-21T12:36:00.137364+0000","last_became_peered":"2024-09-21T12:36:00.137364+0000","last_unstale":"2024-09-21T12:37:26.524338+0000","last_undegraded":"2024-09-21T12:37:26.524338+0000","last_fullsized":"2024-09-21T12:37:26.524338+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:13.198376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:13.198376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":830,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T12:37:22.127624+0000","last_change":"2024-09-21T12:36:38.752470+0000","last_active":"2024-09-21T12:37:22.127624+0000","last_peered":"2024-09-21T12:37:22.127624+0000","last_clean":"2024-09-21T12:37:22.127624+0000","last_became_active":"2024-09-21T12:36:00.134723+0000","last_became_peered":"2024-09-21T12:36:00.134723+0000","last_unstale":"2024-09-21T12:37:22.127624+0000","last_undegraded":"2024-09-21T12:37:22.127624+0000","last_fullsized":"2024-09-21T12:37:22.127624+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:38.752410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:38.752410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:46.920310+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"76'79","reported_seq":601,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.136912+0000","last_change":"2024-09-21T12:36:00.148161+0000","last_active":"2024-09-21T12:37:27.136912+0000","last_peered":"2024-09-21T12:37:27.136912+0000","last_clean":"2024-09-21T12:37:27.136912+0000","last_became_active":"2024-09-21T12:36:00.147795+0000","last_became_peered":"2024-09-21T12:36:00.147795+0000","last_unstale":"2024-09-21T12:37:27.136912+0000","last_undegraded":"2024-09-21T12:37:27.136912+0000","last_fullsized":"2024-09-21T12:37:27.136912+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:36.752562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:36.752562+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:52.540202+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":356,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.525269+0000","last_change":"2024-09-21T12:37:20.468922+0000","last_active":"2024-09-21T12:37:26.525269+0000","last_peered":"2024-09-21T12:37:26.525269+0000","last_clean":"2024-09-21T12:37:26.525269+0000","last_became_active":"2024-09-21T12:37:20.468468+0000","last_became_peered":"2024-09-21T12:37:20.468468+0000","last_unstale":"2024-09-21T12:37:26.525269+0000","last_undegraded":"2024-09-21T12:37:26.525269+0000","last_fullsized":"2024-09-21T12:37:26.525269+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:14.243954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:14.243954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":217,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.530137+0000","last_change":"2024-09-21T12:33:26.395059+0000","last_active":"2024-09-21T12:37:26.530137+0000","last_peered":"2024-09-21T12:37:26.530137+0000","last_clean":"2024-09-21T12:37:26.530137+0000","last_became_active":"2024-09-21T12:32:38.422912+0000","last_became_peered":"2024-09-21T12:32:38.422912+0000","last_unstale":"2024-09-21T12:37:26.530137+0000","last_undegraded":"2024-09-21T12:37:26.530137+0000","last_fullsized":"2024-09-21T12:37:26.530137+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'7","last_scrub_stamp":"2024-09-21T12:33:26.394982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:26.394982+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:54.602228+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":72,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.519362+0000","last_change":"2024-09-21T12:36:00.143221+0000","last_active":"2024-09-21T12:37:26.519362+0000","last_peered":"2024-09-21T12:37:26.519362+0000","last_clean":"2024-09-21T12:37:26.519362+0000","last_became_active":"2024-09-21T12:36:00.142106+0000","last_became_peered":"2024-09-21T12:36:00.142106+0000","last_unstale":"2024-09-21T12:37:26.519362+0000","last_undegraded":"2024-09-21T12:37:26.519362+0000","last_fullsized":"2024-09-21T12:37:26.519362+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:09.558418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:35:09.558418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:51.721658+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":825,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.524201+0000","last_change":"2024-09-21T12:33:36.125346+0000","last_active":"2024-09-21T12:37:26.524201+0000","last_peered":"2024-09-21T12:37:26.524201+0000","last_clean":"2024-09-21T12:37:26.524201+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:37:26.524201+0000","last_undegraded":"2024-09-21T12:37:26.524201+0000","last_fullsized":"2024-09-21T12:37:26.524201+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:36.124095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:36.124095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"76'79","reported_seq":649,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.137152+0000","last_change":"2024-09-21T12:34:48.959991+0000","last_active":"2024-09-21T12:37:27.137152+0000","last_peered":"2024-09-21T12:37:27.137152+0000","last_clean":"2024-09-21T12:37:27.137152+0000","last_became_active":"2024-09-21T12:27:16.957569+0000","last_became_peered":"2024-09-21T12:27:16.957569+0000","last_unstale":"2024-09-21T12:37:27.137152+0000","last_undegraded":"2024-09-21T12:37:27.137152+0000","last_fullsized":"2024-09-21T12:37:27.137152+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:48.959916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:48.959916+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":212,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.531163+0000","last_change":"2024-09-21T12:34:05.193021+0000","last_active":"2024-09-21T12:37:26.531163+0000","last_peered":"2024-09-21T12:37:26.531163+0000","last_clean":"2024-09-21T12:37:26.531163+0000","last_became_active":"2024-09-21T12:32:38.428997+0000","last_became_peered":"2024-09-21T12:32:38.428997+0000","last_unstale":"2024-09-21T12:37:26.531163+0000","last_undegraded":"2024-09-21T12:37:26.531163+0000","last_fullsized":"2024-09-21T12:37:26.531163+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:05.192971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:34:05.192971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:04.693690+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":884,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T12:37:22.128519+0000","last_change":"2024-09-21T12:36:00.127510+0000","last_active":"2024-09-21T12:37:22.128519+0000","last_peered":"2024-09-21T12:37:22.128519+0000","last_clean":"2024-09-21T12:37:22.128519+0000","last_became_active":"2024-09-21T12:36:00.127153+0000","last_became_peered":"2024-09-21T12:36:00.127153+0000","last_unstale":"2024-09-21T12:37:22.128519+0000","last_undegraded":"2024-09-21T12:37:22.128519+0000","last_fullsized":"2024-09-21T12:37:22.128519+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:56.016840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:56.016840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":868,"reported_epoch":445,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:37:26.524501+0000","last_change":"2024-09-21T12:34:34.341728+0000","last_active":"2024-09-21T12:37:26.524501+0000","last_peered":"2024-09-21T12:37:26.524501+0000","last_clean":"2024-09-21T12:34:29.305282+0000","last_became_active":"2024-09-21T12:34:34.341728+0000","last_became_peered":"2024-09-21T12:34:34.341728+0000","last_unstale":"2024-09-21T12:37:26.524501+0000","last_undegraded":"2024-09-21T12:34:34.336150+0000","last_fullsized":"2024-09-21T12:34:34.336049+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:33:01.668584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:01.668584+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:48.550794+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":847,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.520053+0000","last_change":"2024-09-21T12:35:50.957724+0000","last_active":"2024-09-21T12:37:26.520053+0000","last_peered":"2024-09-21T12:37:26.520053+0000","last_clean":"2024-09-21T12:37:26.520053+0000","last_became_active":"2024-09-21T12:32:38.425958+0000","last_became_peered":"2024-09-21T12:32:38.425958+0000","last_unstale":"2024-09-21T12:37:26.520053+0000","last_undegraded":"2024-09-21T12:37:26.520053+0000","last_fullsized":"2024-09-21T12:37:26.520053+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:50.957677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:35:50.957677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:07.760563+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"76'79","reported_seq":776,"reported_epoch":445,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:37:26.525570+0000","last_change":"2024-09-21T12:36:01.149228+0000","last_active":"2024-09-21T12:37:26.525570+0000","last_peered":"2024-09-21T12:37:26.525570+0000","last_clean":"2024-09-21T12:37:26.525570+0000","last_became_active":"2024-09-21T12:36:01.148686+0000","last_became_peered":"2024-09-21T12:36:01.148686+0000","last_unstale":"2024-09-21T12:37:26.525570+0000","last_undegraded":"2024-09-21T12:37:26.525570+0000","last_fullsized":"2024-09-21T12:37:26.525570+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:27.294674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:27.294674+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7,1],"avail_no_missing":["5","1","7"],"object_location_counts":[{"shards":"1,5,7","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":832,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.136975+0000","last_change":"2024-09-21T12:35:06.944278+0000","last_active":"2024-09-21T12:37:27.136975+0000","last_peered":"2024-09-21T12:37:27.136975+0000","last_clean":"2024-09-21T12:37:27.136975+0000","last_became_active":"2024-09-21T12:32:02.033276+0000","last_became_peered":"2024-09-21T12:32:02.033276+0000","last_unstale":"2024-09-21T12:37:27.136975+0000","last_undegraded":"2024-09-21T12:37:27.136975+0000","last_fullsized":"2024-09-21T12:37:27.136975+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:06.944209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:06.944209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":207,"reported_epoch":445,"state":"active+undersized","last_fresh":"2024-09-21T12:37:26.530065+0000","last_change":"2024-09-21T12:34:34.349252+0000","last_active":"2024-09-21T12:37:26.530065+0000","last_peered":"2024-09-21T12:37:26.530065+0000","last_clean":"2024-09-21T12:34:29.278130+0000","last_became_active":"2024-09-21T12:34:34.349252+0000","last_became_peered":"2024-09-21T12:34:34.349252+0000","last_unstale":"2024-09-21T12:37:26.530065+0000","last_undegraded":"2024-09-21T12:37:26.530065+0000","last_fullsized":"2024-09-21T12:34:34.329566+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:45.725968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.725968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:20.738724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":925,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.135907+0000","last_change":"2024-09-21T12:35:59.641966+0000","last_active":"2024-09-21T12:37:27.135907+0000","last_peered":"2024-09-21T12:37:27.135907+0000","last_clean":"2024-09-21T12:37:27.135907+0000","last_became_active":"2024-09-21T12:32:38.409163+0000","last_became_peered":"2024-09-21T12:32:38.409163+0000","last_unstale":"2024-09-21T12:37:27.135907+0000","last_undegraded":"2024-09-21T12:37:27.135907+0000","last_fullsized":"2024-09-21T12:37:27.135907+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T12:35:59.641895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:59.641895+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":215,"reported_epoch":445,"state":"active+undersized","last_fresh":"2024-09-21T12:37:26.531300+0000","last_change":"2024-09-21T12:34:34.349550+0000","last_active":"2024-09-21T12:37:26.531300+0000","last_peered":"2024-09-21T12:37:26.531300+0000","last_clean":"2024-09-21T12:34:29.281006+0000","last_became_active":"2024-09-21T12:34:34.349550+0000","last_became_peered":"2024-09-21T12:34:34.349550+0000","last_unstale":"2024-09-21T12:37:26.531300+0000","last_undegraded":"2024-09-21T12:37:26.531300+0000","last_fullsized":"2024-09-21T12:34:34.331038+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:53.270634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:32:53.270634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:37.640748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"76'79","reported_seq":571,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.524113+0000","last_change":"2024-09-21T12:36:41.538607+0000","last_active":"2024-09-21T12:37:26.524113+0000","last_peered":"2024-09-21T12:37:26.524113+0000","last_clean":"2024-09-21T12:37:26.524113+0000","last_became_active":"2024-09-21T12:32:38.414204+0000","last_became_peered":"2024-09-21T12:32:38.414204+0000","last_unstale":"2024-09-21T12:37:26.524113+0000","last_undegraded":"2024-09-21T12:37:26.524113+0000","last_fullsized":"2024-09-21T12:37:26.524113+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:41.538559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:41.538559+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:06.792208+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":798,"reported_epoch":445,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:37:26.520328+0000","last_change":"2024-09-21T12:36:37.839177+0000","last_active":"2024-09-21T12:37:26.520328+0000","last_peered":"2024-09-21T12:37:26.520328+0000","last_clean":"2024-09-21T12:37:26.520328+0000","last_became_active":"2024-09-21T12:36:01.126897+0000","last_became_peered":"2024-09-21T12:36:01.126897+0000","last_unstale":"2024-09-21T12:37:26.520328+0000","last_undegraded":"2024-09-21T12:37:26.520328+0000","last_fullsized":"2024-09-21T12:37:26.520328+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:36:37.839024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:37.839024+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:04.241490+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5,3],"avail_no_missing":["7","3","5"],"object_location_counts":[{"shards":"3,5,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":874,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.519296+0000","last_change":"2024-09-21T12:36:44.561097+0000","last_active":"2024-09-21T12:37:26.519296+0000","last_peered":"2024-09-21T12:37:26.519296+0000","last_clean":"2024-09-21T12:37:26.519296+0000","last_became_active":"2024-09-21T12:32:38.425335+0000","last_became_peered":"2024-09-21T12:32:38.425335+0000","last_unstale":"2024-09-21T12:37:26.519296+0000","last_undegraded":"2024-09-21T12:37:26.519296+0000","last_fullsized":"2024-09-21T12:37:26.519296+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:36:44.561030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:36:44.561030+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:57.359274+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":940,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T12:37:22.129393+0000","last_change":"2024-09-21T12:36:00.173918+0000","last_active":"2024-09-21T12:37:22.129393+0000","last_peered":"2024-09-21T12:37:22.129393+0000","last_clean":"2024-09-21T12:37:22.129393+0000","last_became_active":"2024-09-21T12:36:00.148761+0000","last_became_peered":"2024-09-21T12:36:00.148761+0000","last_unstale":"2024-09-21T12:37:22.129393+0000","last_undegraded":"2024-09-21T12:37:22.129393+0000","last_fullsized":"2024-09-21T12:37:22.129393+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:32:57.082525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:57.082525+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"415'18","reported_seq":288,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.532523+0000","last_change":"2024-09-21T12:36:00.148175+0000","last_active":"2024-09-21T12:37:26.532523+0000","last_peered":"2024-09-21T12:37:26.532523+0000","last_clean":"2024-09-21T12:37:26.532523+0000","last_became_active":"2024-09-21T12:36:00.147676+0000","last_became_peered":"2024-09-21T12:36:00.147676+0000","last_unstale":"2024-09-21T12:37:26.532523+0000","last_undegraded":"2024-09-21T12:37:26.532523+0000","last_fullsized":"2024-09-21T12:37:26.532523+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'15","last_scrub_stamp":"2024-09-21T12:35:11.182313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:35:11.182313+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"445'8890","reported_seq":11243,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.633145+0000","last_change":"2024-09-21T12:37:14.558326+0000","last_active":"2024-09-21T12:37:26.633145+0000","last_peered":"2024-09-21T12:37:26.633145+0000","last_clean":"2024-09-21T12:37:26.633145+0000","last_became_active":"2024-09-21T12:36:13.105161+0000","last_became_peered":"2024-09-21T12:36:13.105161+0000","last_unstale":"2024-09-21T12:37:26.633145+0000","last_undegraded":"2024-09-21T12:37:26.633145+0000","last_fullsized":"2024-09-21T12:37:26.633145+0000","mapping_epoch":420,"log_start":"430'5000","ondisk_log_start":"430'5000","created":405,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"438'7954","last_scrub_stamp":"2024-09-21T12:37:14.558192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:37:14.558192+0000","objects_scrubbed":7862,"log_size":3890,"log_dups_size":0,"ondisk_log_size":3890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:38.367536+0000","scrub_duration":2678,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":543555584,"num_objects":8294,"num_object_clones":0,"num_object_copies":24882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8890,"num_write_kb":549888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1078,"num_bytes_recovered":70647808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"76'79","reported_seq":375,"reported_epoch":445,"state":"active+undersized","last_fresh":"2024-09-21T12:37:26.525222+0000","last_change":"2024-09-21T12:34:34.347159+0000","last_active":"2024-09-21T12:37:26.525222+0000","last_peered":"2024-09-21T12:37:26.525222+0000","last_clean":"2024-09-21T12:34:29.305642+0000","last_became_active":"2024-09-21T12:34:34.347159+0000","last_became_peered":"2024-09-21T12:34:34.347159+0000","last_unstale":"2024-09-21T12:37:26.525222+0000","last_undegraded":"2024-09-21T12:37:26.525222+0000","last_fullsized":"2024-09-21T12:34:34.340605+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:18.151644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:18.151644+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:28.449536+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":479,"reported_epoch":446,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:37:27.137163+0000","last_change":"2024-09-21T12:34:34.345532+0000","last_active":"2024-09-21T12:37:27.137163+0000","last_peered":"2024-09-21T12:37:27.137163+0000","last_clean":"2024-09-21T12:34:29.295616+0000","last_became_active":"2024-09-21T12:34:34.345532+0000","last_became_peered":"2024-09-21T12:34:34.345532+0000","last_unstale":"2024-09-21T12:37:27.137163+0000","last_undegraded":"2024-09-21T12:34:34.334307+0000","last_fullsized":"2024-09-21T12:34:34.334181+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:31:56.722241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:31:56.722241+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:50.482287+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":296,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.135296+0000","last_change":"2024-09-21T12:36:00.140935+0000","last_active":"2024-09-21T12:37:27.135296+0000","last_peered":"2024-09-21T12:37:27.135296+0000","last_clean":"2024-09-21T12:37:27.135296+0000","last_became_active":"2024-09-21T12:36:00.140455+0000","last_became_peered":"2024-09-21T12:36:00.140455+0000","last_unstale":"2024-09-21T12:37:27.135296+0000","last_undegraded":"2024-09-21T12:37:27.135296+0000","last_fullsized":"2024-09-21T12:37:27.135296+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:05.703464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:05.703464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"352'15","reported_seq":274,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.531085+0000","last_change":"2024-09-21T12:35:50.989423+0000","last_active":"2024-09-21T12:37:26.531085+0000","last_peered":"2024-09-21T12:37:26.531085+0000","last_clean":"2024-09-21T12:37:26.531085+0000","last_became_active":"2024-09-21T12:32:38.429092+0000","last_became_peered":"2024-09-21T12:32:38.429092+0000","last_unstale":"2024-09-21T12:37:26.531085+0000","last_undegraded":"2024-09-21T12:37:26.531085+0000","last_fullsized":"2024-09-21T12:37:26.531085+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'15","last_scrub_stamp":"2024-09-21T12:35:50.989377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:35:50.989377+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":324,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.526133+0000","last_change":"2024-09-21T12:37:20.469356+0000","last_active":"2024-09-21T12:37:26.526133+0000","last_peered":"2024-09-21T12:37:26.526133+0000","last_clean":"2024-09-21T12:37:26.526133+0000","last_became_active":"2024-09-21T12:37:20.468591+0000","last_became_peered":"2024-09-21T12:37:20.468591+0000","last_unstale":"2024-09-21T12:37:26.526133+0000","last_undegraded":"2024-09-21T12:37:26.526133+0000","last_fullsized":"2024-09-21T12:37:26.526133+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:35:24.989615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:24.989615+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"445'9039","reported_seq":9218,"reported_epoch":445,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:37:27.013792+0000","last_change":"2024-09-21T12:37:13.555476+0000","last_active":"2024-09-21T12:37:27.013792+0000","last_peered":"2024-09-21T12:37:27.013792+0000","last_clean":"2024-09-21T12:37:27.013792+0000","last_became_active":"2024-09-21T12:36:01.133058+0000","last_became_peered":"2024-09-21T12:36:01.133058+0000","last_unstale":"2024-09-21T12:37:27.013792+0000","last_undegraded":"2024-09-21T12:37:27.013792+0000","last_fullsized":"2024-09-21T12:37:27.013792+0000","mapping_epoch":415,"log_start":"427'4800","ondisk_log_start":"427'4800","created":405,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"438'8003","last_scrub_stamp":"2024-09-21T12:37:13.555093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:37:13.555093+0000","objects_scrubbed":7888,"log_size":4239,"log_dups_size":0,"ondisk_log_size":4239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:28.857219+0000","scrub_duration":2942,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":548864000,"num_objects":8375,"num_object_clones":0,"num_object_copies":25125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8375,"num_objects_unfound":0,"num_objects_dirty":8375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9039,"num_write_kb":557248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,0],"avail_no_missing":["7","0","1"],"object_location_counts":[{"shards":"0,1,7","objects":8375}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"76'79","reported_seq":216,"reported_epoch":445,"state":"active+undersized","last_fresh":"2024-09-21T12:37:26.531644+0000","last_change":"2024-09-21T12:34:34.348261+0000","last_active":"2024-09-21T12:37:26.531644+0000","last_peered":"2024-09-21T12:37:26.531644+0000","last_clean":"2024-09-21T12:34:29.279263+0000","last_became_active":"2024-09-21T12:34:34.348261+0000","last_became_peered":"2024-09-21T12:34:34.348261+0000","last_unstale":"2024-09-21T12:37:26.531644+0000","last_undegraded":"2024-09-21T12:37:26.531644+0000","last_fullsized":"2024-09-21T12:34:34.332145+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:00.343916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:00.343916+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:39.681388+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":977,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.135846+0000","last_change":"2024-09-21T12:36:00.163117+0000","last_active":"2024-09-21T12:37:27.135846+0000","last_peered":"2024-09-21T12:37:27.135846+0000","last_clean":"2024-09-21T12:37:27.135846+0000","last_became_active":"2024-09-21T12:36:00.139849+0000","last_became_peered":"2024-09-21T12:36:00.139849+0000","last_unstale":"2024-09-21T12:37:27.135846+0000","last_undegraded":"2024-09-21T12:37:27.135846+0000","last_fullsized":"2024-09-21T12:37:27.135846+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T12:35:12.790809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:12.790809+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":828,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.137736+0000","last_change":"2024-09-21T12:36:00.152050+0000","last_active":"2024-09-21T12:37:27.137736+0000","last_peered":"2024-09-21T12:37:27.137736+0000","last_clean":"2024-09-21T12:37:27.137736+0000","last_became_active":"2024-09-21T12:36:00.151488+0000","last_became_peered":"2024-09-21T12:36:00.151488+0000","last_unstale":"2024-09-21T12:37:27.137736+0000","last_undegraded":"2024-09-21T12:37:27.137736+0000","last_fullsized":"2024-09-21T12:37:27.137736+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:22.752960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:22.752960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":226,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.529623+0000","last_change":"2024-09-21T12:36:00.137014+0000","last_active":"2024-09-21T12:37:26.529623+0000","last_peered":"2024-09-21T12:37:26.529623+0000","last_clean":"2024-09-21T12:37:26.529623+0000","last_became_active":"2024-09-21T12:36:00.136650+0000","last_became_peered":"2024-09-21T12:36:00.136650+0000","last_unstale":"2024-09-21T12:37:26.529623+0000","last_undegraded":"2024-09-21T12:37:26.529623+0000","last_fullsized":"2024-09-21T12:37:26.529623+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:12.216104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:12.216104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":208,"reported_epoch":445,"state":"active+undersized","last_fresh":"2024-09-21T12:37:26.530448+0000","last_change":"2024-09-21T12:34:34.347165+0000","last_active":"2024-09-21T12:37:26.530448+0000","last_peered":"2024-09-21T12:37:26.530448+0000","last_clean":"2024-09-21T12:34:29.278468+0000","last_became_active":"2024-09-21T12:34:34.347165+0000","last_became_peered":"2024-09-21T12:34:34.347165+0000","last_unstale":"2024-09-21T12:37:26.530448+0000","last_undegraded":"2024-09-21T12:37:26.530448+0000","last_fullsized":"2024-09-21T12:34:34.331768+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:29:05.143736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:29:05.143736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:22.708853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"445'9018","reported_seq":11161,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.894252+0000","last_change":"2024-09-21T12:36:08.103615+0000","last_active":"2024-09-21T12:37:26.894252+0000","last_peered":"2024-09-21T12:37:26.894252+0000","last_clean":"2024-09-21T12:37:26.894252+0000","last_became_active":"2024-09-21T12:36:08.101796+0000","last_became_peered":"2024-09-21T12:36:08.101796+0000","last_unstale":"2024-09-21T12:37:26.894252+0000","last_undegraded":"2024-09-21T12:37:26.894252+0000","last_fullsized":"2024-09-21T12:37:26.894252+0000","mapping_epoch":417,"log_start":"429'4900","ondisk_log_start":"429'4900","created":405,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4118,"log_dups_size":0,"ondisk_log_size":4118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":547487744,"num_objects":8354,"num_object_clones":0,"num_object_copies":25062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9018,"num_write_kb":555904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1016,"num_bytes_recovered":66584576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"76'79","reported_seq":764,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.135253+0000","last_change":"2024-09-21T12:36:43.462949+0000","last_active":"2024-09-21T12:37:27.135253+0000","last_peered":"2024-09-21T12:37:27.135253+0000","last_clean":"2024-09-21T12:37:27.135253+0000","last_became_active":"2024-09-21T12:36:00.155047+0000","last_became_peered":"2024-09-21T12:36:00.155047+0000","last_unstale":"2024-09-21T12:37:27.135253+0000","last_undegraded":"2024-09-21T12:37:27.135253+0000","last_fullsized":"2024-09-21T12:37:27.135253+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:43.462874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:43.462874+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:44.145911+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":220,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.529391+0000","last_change":"2024-09-21T12:32:38.457999+0000","last_active":"2024-09-21T12:37:26.529391+0000","last_peered":"2024-09-21T12:37:26.529391+0000","last_clean":"2024-09-21T12:37:26.529391+0000","last_became_active":"2024-09-21T12:32:38.425453+0000","last_became_peered":"2024-09-21T12:32:38.425453+0000","last_unstale":"2024-09-21T12:37:26.529391+0000","last_undegraded":"2024-09-21T12:37:26.529391+0000","last_fullsized":"2024-09-21T12:37:26.529391+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-21T12:26:51.463698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:51.463698+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:08.672611+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":40,"num_read_kb":33,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":907,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T12:37:22.129838+0000","last_change":"2024-09-21T12:36:39.754233+0000","last_active":"2024-09-21T12:37:22.129838+0000","last_peered":"2024-09-21T12:37:22.129838+0000","last_clean":"2024-09-21T12:37:22.129838+0000","last_became_active":"2024-09-21T12:32:38.424412+0000","last_became_peered":"2024-09-21T12:32:38.424412+0000","last_unstale":"2024-09-21T12:37:22.129838+0000","last_undegraded":"2024-09-21T12:37:22.129838+0000","last_fullsized":"2024-09-21T12:37:22.129838+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:36:39.754173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:36:39.754173+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:02.184909+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"352'12","reported_seq":1074,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.135647+0000","last_change":"2024-09-21T12:34:07.827409+0000","last_active":"2024-09-21T12:37:27.135647+0000","last_peered":"2024-09-21T12:37:27.135647+0000","last_clean":"2024-09-21T12:37:27.135647+0000","last_became_active":"2024-09-21T12:32:38.409177+0000","last_became_peered":"2024-09-21T12:32:38.409177+0000","last_unstale":"2024-09-21T12:37:27.135647+0000","last_undegraded":"2024-09-21T12:37:27.135647+0000","last_fullsized":"2024-09-21T12:37:27.135647+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'12","last_scrub_stamp":"2024-09-21T12:34:07.827315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:07.827315+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:56.601246+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":799,"reported_epoch":445,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:37:26.521840+0000","last_change":"2024-09-21T12:36:01.138656+0000","last_active":"2024-09-21T12:37:26.521840+0000","last_peered":"2024-09-21T12:37:26.521840+0000","last_clean":"2024-09-21T12:37:26.521840+0000","last_became_active":"2024-09-21T12:36:01.135064+0000","last_became_peered":"2024-09-21T12:36:01.135064+0000","last_unstale":"2024-09-21T12:37:26.521840+0000","last_undegraded":"2024-09-21T12:37:26.521840+0000","last_fullsized":"2024-09-21T12:37:26.521840+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:59.406201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:32:59.406201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6,1],"avail_no_missing":["7","1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"445'8700","reported_seq":8790,"reported_epoch":445,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:37:26.910363+0000","last_change":"2024-09-21T12:35:53.507442+0000","last_active":"2024-09-21T12:37:26.910363+0000","last_peered":"2024-09-21T12:37:26.910363+0000","last_clean":"2024-09-21T12:35:46.767214+0000","last_became_active":"2024-09-21T12:35:47.791328+0000","last_became_peered":"2024-09-21T12:35:47.791328+0000","last_unstale":"2024-09-21T12:37:26.910363+0000","last_undegraded":"2024-09-21T12:35:53.507160+0000","last_fullsized":"2024-09-21T12:35:47.787870+0000","mapping_epoch":405,"log_start":"428'4600","ondisk_log_start":"428'4600","created":405,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4100,"log_dups_size":0,"ondisk_log_size":4100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":531300376,"num_objects":8108,"num_object_clones":0,"num_object_copies":24324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8108,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8700,"num_write_kb":537793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":8108}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"76'79","reported_seq":304,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.137611+0000","last_change":"2024-09-21T12:37:16.761819+0000","last_active":"2024-09-21T12:37:27.137611+0000","last_peered":"2024-09-21T12:37:27.137611+0000","last_clean":"2024-09-21T12:37:27.137611+0000","last_became_active":"2024-09-21T12:36:00.151076+0000","last_became_peered":"2024-09-21T12:36:00.151076+0000","last_unstale":"2024-09-21T12:37:27.137611+0000","last_undegraded":"2024-09-21T12:37:27.137611+0000","last_fullsized":"2024-09-21T12:37:27.137611+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:37:16.761749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:37:16.761749+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:39.806827+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":362,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.520291+0000","last_change":"2024-09-21T12:36:33.481707+0000","last_active":"2024-09-21T12:37:26.520291+0000","last_peered":"2024-09-21T12:37:26.520291+0000","last_clean":"2024-09-21T12:37:26.520291+0000","last_became_active":"2024-09-21T12:32:38.424412+0000","last_became_peered":"2024-09-21T12:32:38.424412+0000","last_unstale":"2024-09-21T12:37:26.520291+0000","last_undegraded":"2024-09-21T12:37:26.520291+0000","last_fullsized":"2024-09-21T12:37:26.520291+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:36:33.481627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:33.481627+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:00.222164+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":834,"reported_epoch":445,"state":"active+undersized","last_fresh":"2024-09-21T12:37:26.525746+0000","last_change":"2024-09-21T12:34:34.339986+0000","last_active":"2024-09-21T12:37:26.525746+0000","last_peered":"2024-09-21T12:37:26.525746+0000","last_clean":"2024-09-21T12:34:29.305656+0000","last_became_active":"2024-09-21T12:34:34.339986+0000","last_became_peered":"2024-09-21T12:34:34.339986+0000","last_unstale":"2024-09-21T12:37:26.525746+0000","last_undegraded":"2024-09-21T12:37:26.525746+0000","last_fullsized":"2024-09-21T12:34:34.336347+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:19.629083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:19.629083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:43.542005+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":1039,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T12:37:22.132071+0000","last_change":"2024-09-21T12:36:44.788732+0000","last_active":"2024-09-21T12:37:22.132071+0000","last_peered":"2024-09-21T12:37:22.132071+0000","last_clean":"2024-09-21T12:37:22.132071+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:37:22.132071+0000","last_undegraded":"2024-09-21T12:37:22.132071+0000","last_fullsized":"2024-09-21T12:37:22.132071+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:36:44.788662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:44.788662+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:47.677328+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":806,"reported_epoch":444,"state":"active+undersized","last_fresh":"2024-09-21T12:37:22.127363+0000","last_change":"2024-09-21T12:34:34.335260+0000","last_active":"2024-09-21T12:37:22.127363+0000","last_peered":"2024-09-21T12:37:22.127363+0000","last_clean":"2024-09-21T12:34:29.288868+0000","last_became_active":"2024-09-21T12:34:34.335260+0000","last_became_peered":"2024-09-21T12:34:34.335260+0000","last_unstale":"2024-09-21T12:37:22.127363+0000","last_undegraded":"2024-09-21T12:37:22.127363+0000","last_fullsized":"2024-09-21T12:34:34.331914+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:37.850100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:28:37.850100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:43.750121+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"445'9060","reported_seq":9149,"reported_epoch":445,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:37:26.615193+0000","last_change":"2024-09-21T12:35:53.515408+0000","last_active":"2024-09-21T12:37:26.615193+0000","last_peered":"2024-09-21T12:37:26.615193+0000","last_clean":"2024-09-21T12:35:46.767214+0000","last_became_active":"2024-09-21T12:35:47.791289+0000","last_became_peered":"2024-09-21T12:35:47.791289+0000","last_unstale":"2024-09-21T12:37:26.615193+0000","last_undegraded":"2024-09-21T12:35:53.506389+0000","last_fullsized":"2024-09-21T12:35:47.788809+0000","mapping_epoch":405,"log_start":"430'5200","ondisk_log_start":"430'5200","created":405,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3860,"log_dups_size":0,"ondisk_log_size":3860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:19.767114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":555220992,"num_objects":8472,"num_object_clones":0,"num_object_copies":25416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8472,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9060,"num_write_kb":561024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":8472}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"76'79","reported_seq":495,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T12:37:22.128840+0000","last_change":"2024-09-21T12:32:52.986242+0000","last_active":"2024-09-21T12:37:22.128840+0000","last_peered":"2024-09-21T12:37:22.128840+0000","last_clean":"2024-09-21T12:37:22.128840+0000","last_became_active":"2024-09-21T12:29:05.835884+0000","last_became_peered":"2024-09-21T12:29:05.835884+0000","last_unstale":"2024-09-21T12:37:22.128840+0000","last_undegraded":"2024-09-21T12:37:22.128840+0000","last_fullsized":"2024-09-21T12:37:22.128840+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:52.986175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:52.986175+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:24.882111+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":919,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T12:37:22.128940+0000","last_change":"2024-09-21T12:36:36.774685+0000","last_active":"2024-09-21T12:37:22.128940+0000","last_peered":"2024-09-21T12:37:22.128940+0000","last_clean":"2024-09-21T12:37:22.128940+0000","last_became_active":"2024-09-21T12:36:00.127312+0000","last_became_peered":"2024-09-21T12:36:00.127312+0000","last_unstale":"2024-09-21T12:37:22.128940+0000","last_undegraded":"2024-09-21T12:37:22.128940+0000","last_fullsized":"2024-09-21T12:37:22.128940+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'29","last_scrub_stamp":"2024-09-21T12:36:36.774599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:36.774599+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:43.150659+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":810,"reported_epoch":445,"state":"active+undersized","last_fresh":"2024-09-21T12:37:26.522104+0000","last_change":"2024-09-21T12:34:34.337276+0000","last_active":"2024-09-21T12:37:26.522104+0000","last_peered":"2024-09-21T12:37:26.522104+0000","last_clean":"2024-09-21T12:34:29.286433+0000","last_became_active":"2024-09-21T12:34:34.337276+0000","last_became_peered":"2024-09-21T12:34:34.337276+0000","last_unstale":"2024-09-21T12:37:26.522104+0000","last_undegraded":"2024-09-21T12:37:26.522104+0000","last_fullsized":"2024-09-21T12:34:34.330247+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:38.250806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:38.250806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:29.240702+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":399,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T12:37:22.129246+0000","last_change":"2024-09-21T12:36:00.924159+0000","last_active":"2024-09-21T12:37:22.129246+0000","last_peered":"2024-09-21T12:37:22.129246+0000","last_clean":"2024-09-21T12:37:22.129246+0000","last_became_active":"2024-09-21T12:29:05.825869+0000","last_became_peered":"2024-09-21T12:29:05.825869+0000","last_unstale":"2024-09-21T12:37:22.129246+0000","last_undegraded":"2024-09-21T12:37:22.129246+0000","last_fullsized":"2024-09-21T12:37:22.129246+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:00.924084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:00.924084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:28.472139+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":806,"reported_epoch":445,"state":"active+undersized","last_fresh":"2024-09-21T12:37:26.518966+0000","last_change":"2024-09-21T12:34:34.339352+0000","last_active":"2024-09-21T12:37:26.518966+0000","last_peered":"2024-09-21T12:37:26.518966+0000","last_clean":"2024-09-21T12:34:29.284095+0000","last_became_active":"2024-09-21T12:34:34.339352+0000","last_became_peered":"2024-09-21T12:34:34.339352+0000","last_unstale":"2024-09-21T12:37:26.518966+0000","last_undegraded":"2024-09-21T12:37:26.518966+0000","last_fullsized":"2024-09-21T12:34:34.329336+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:05.510620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:05.510620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"446'8934","reported_seq":9028,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.846683+0000","last_change":"2024-09-21T12:35:47.791371+0000","last_active":"2024-09-21T12:37:27.846683+0000","last_peered":"2024-09-21T12:37:27.846683+0000","last_clean":"2024-09-21T12:37:27.846683+0000","last_became_active":"2024-09-21T12:35:47.791120+0000","last_became_peered":"2024-09-21T12:35:47.791120+0000","last_unstale":"2024-09-21T12:37:27.846683+0000","last_undegraded":"2024-09-21T12:37:27.846683+0000","last_fullsized":"2024-09-21T12:37:27.846683+0000","mapping_epoch":405,"log_start":"430'5000","ondisk_log_start":"430'5000","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3934,"log_dups_size":0,"ondisk_log_size":3934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":532414464,"num_objects":8124,"num_object_clones":0,"num_object_copies":24372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8934,"num_write_kb":545856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"76'79","reported_seq":251,"reported_epoch":445,"state":"active+undersized","last_fresh":"2024-09-21T12:37:26.523885+0000","last_change":"2024-09-21T12:37:20.466215+0000","last_active":"2024-09-21T12:37:26.523885+0000","last_peered":"2024-09-21T12:37:26.523885+0000","last_clean":"2024-09-21T12:26:52.048263+0000","last_became_active":"2024-09-21T12:37:20.466215+0000","last_became_peered":"2024-09-21T12:37:20.466215+0000","last_unstale":"2024-09-21T12:37:26.523885+0000","last_undegraded":"2024-09-21T12:37:26.523885+0000","last_fullsized":"2024-09-21T12:37:20.460391+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:34.055720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:34.055720+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"389'41","reported_seq":1080,"reported_epoch":445,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:37:26.520246+0000","last_change":"2024-09-21T12:34:34.339316+0000","last_active":"2024-09-21T12:37:26.520246+0000","last_peered":"2024-09-21T12:37:26.520246+0000","last_clean":"2024-09-21T12:34:30.529659+0000","last_became_active":"2024-09-21T12:34:34.339316+0000","last_became_peered":"2024-09-21T12:34:34.339316+0000","last_unstale":"2024-09-21T12:37:26.520246+0000","last_undegraded":"2024-09-21T12:34:34.330286+0000","last_fullsized":"2024-09-21T12:34:34.330177+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'40","last_scrub_stamp":"2024-09-21T12:32:25.139435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:25.139435+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:40.440122+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":212,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.530275+0000","last_change":"2024-09-21T12:35:09.182021+0000","last_active":"2024-09-21T12:37:26.530275+0000","last_peered":"2024-09-21T12:37:26.530275+0000","last_clean":"2024-09-21T12:37:26.530275+0000","last_became_active":"2024-09-21T12:32:38.426309+0000","last_became_peered":"2024-09-21T12:32:38.426309+0000","last_unstale":"2024-09-21T12:37:26.530275+0000","last_undegraded":"2024-09-21T12:37:26.530275+0000","last_fullsized":"2024-09-21T12:37:26.530275+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:09.181952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:09.181952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":540,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.521567+0000","last_change":"2024-09-21T12:35:27.114524+0000","last_active":"2024-09-21T12:37:26.521567+0000","last_peered":"2024-09-21T12:37:26.521567+0000","last_clean":"2024-09-21T12:37:26.521567+0000","last_became_active":"2024-09-21T12:26:54.083584+0000","last_became_peered":"2024-09-21T12:26:54.083584+0000","last_unstale":"2024-09-21T12:37:26.521567+0000","last_undegraded":"2024-09-21T12:37:26.521567+0000","last_fullsized":"2024-09-21T12:37:26.521567+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T12:35:27.114442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:35:27.114442+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:53.587329+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":850,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T12:37:22.127223+0000","last_change":"2024-09-21T12:33:00.087826+0000","last_active":"2024-09-21T12:37:22.127223+0000","last_peered":"2024-09-21T12:37:22.127223+0000","last_clean":"2024-09-21T12:37:22.127223+0000","last_became_active":"2024-09-21T12:32:38.424402+0000","last_became_peered":"2024-09-21T12:32:38.424402+0000","last_unstale":"2024-09-21T12:37:22.127223+0000","last_undegraded":"2024-09-21T12:37:22.127223+0000","last_fullsized":"2024-09-21T12:37:22.127223+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:00.087746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:33:00.087746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":852,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.136881+0000","last_change":"2024-09-21T12:36:00.144733+0000","last_active":"2024-09-21T12:37:27.136881+0000","last_peered":"2024-09-21T12:37:27.136881+0000","last_clean":"2024-09-21T12:37:27.136881+0000","last_became_active":"2024-09-21T12:36:00.144526+0000","last_became_peered":"2024-09-21T12:36:00.144526+0000","last_unstale":"2024-09-21T12:37:27.136881+0000","last_undegraded":"2024-09-21T12:37:27.136881+0000","last_fullsized":"2024-09-21T12:37:27.136881+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:59.899499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:59.899499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:54.611578+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"446'9171","reported_seq":9265,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.829245+0000","last_change":"2024-09-21T12:35:47.848839+0000","last_active":"2024-09-21T12:37:27.829245+0000","last_peered":"2024-09-21T12:37:27.829245+0000","last_clean":"2024-09-21T12:37:27.829245+0000","last_became_active":"2024-09-21T12:35:47.848543+0000","last_became_peered":"2024-09-21T12:35:47.848543+0000","last_unstale":"2024-09-21T12:37:27.829245+0000","last_undegraded":"2024-09-21T12:37:27.829245+0000","last_fullsized":"2024-09-21T12:37:27.829245+0000","mapping_epoch":405,"log_start":"430'5300","ondisk_log_start":"430'5300","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3871,"log_dups_size":0,"ondisk_log_size":3871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":544800768,"num_objects":8313,"num_object_clones":0,"num_object_copies":24939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9171,"num_write_kb":559488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'79","reported_seq":236,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.519453+0000","last_change":"2024-09-21T12:36:01.132046+0000","last_active":"2024-09-21T12:37:26.519453+0000","last_peered":"2024-09-21T12:37:26.519453+0000","last_clean":"2024-09-21T12:37:26.519453+0000","last_became_active":"2024-09-21T12:36:01.131622+0000","last_became_peered":"2024-09-21T12:36:01.131622+0000","last_unstale":"2024-09-21T12:37:26.519453+0000","last_undegraded":"2024-09-21T12:37:26.519453+0000","last_fullsized":"2024-09-21T12:37:26.519453+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:30:39.179014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:30:39.179014+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":316,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.522382+0000","last_change":"2024-09-21T12:35:18.061854+0000","last_active":"2024-09-21T12:37:26.522382+0000","last_peered":"2024-09-21T12:37:26.522382+0000","last_clean":"2024-09-21T12:37:26.522382+0000","last_became_active":"2024-09-21T12:31:39.057316+0000","last_became_peered":"2024-09-21T12:31:39.057316+0000","last_unstale":"2024-09-21T12:37:26.522382+0000","last_undegraded":"2024-09-21T12:37:26.522382+0000","last_fullsized":"2024-09-21T12:37:26.522382+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'17","last_scrub_stamp":"2024-09-21T12:35:18.061753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:18.061753+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:48.554338+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":842,"reported_epoch":445,"state":"active+undersized","last_fresh":"2024-09-21T12:37:26.519755+0000","last_change":"2024-09-21T12:34:34.343641+0000","last_active":"2024-09-21T12:37:26.519755+0000","last_peered":"2024-09-21T12:37:26.519755+0000","last_clean":"2024-09-21T12:34:29.286935+0000","last_became_active":"2024-09-21T12:34:34.343641+0000","last_became_peered":"2024-09-21T12:34:34.343641+0000","last_unstale":"2024-09-21T12:37:26.519755+0000","last_undegraded":"2024-09-21T12:37:26.519755+0000","last_fullsized":"2024-09-21T12:34:34.329247+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:00.211563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:34:00.211563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:30.227747+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":152,"reported_epoch":446,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:37:27.135042+0000","last_change":"2024-09-21T12:34:34.338167+0000","last_active":"2024-09-21T12:37:27.135042+0000","last_peered":"2024-09-21T12:37:27.135042+0000","last_clean":"2024-09-21T12:34:29.291114+0000","last_became_active":"2024-09-21T12:34:34.338167+0000","last_became_peered":"2024-09-21T12:34:34.338167+0000","last_unstale":"2024-09-21T12:37:27.135042+0000","last_undegraded":"2024-09-21T12:34:34.335226+0000","last_fullsized":"2024-09-21T12:34:34.335117+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:32:20.845428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:32:20.845428+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":150,"reported_epoch":445,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:37:26.524532+0000","last_change":"2024-09-21T12:36:01.125293+0000","last_active":"2024-09-21T12:37:26.524532+0000","last_peered":"2024-09-21T12:37:26.524532+0000","last_clean":"2024-09-21T12:37:26.524532+0000","last_became_active":"2024-09-21T12:36:01.124868+0000","last_became_peered":"2024-09-21T12:36:01.124868+0000","last_unstale":"2024-09-21T12:37:26.524532+0000","last_undegraded":"2024-09-21T12:37:26.524532+0000","last_fullsized":"2024-09-21T12:37:26.524532+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:02.295255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:02.295255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7,4],"avail_no_missing":["5","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":41,"reported_epoch":444,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:37:22.126985+0000","last_change":"2024-09-21T12:36:40.306122+0000","last_active":"2024-09-21T12:37:22.126985+0000","last_peered":"2024-09-21T12:37:22.126985+0000","last_clean":"2024-09-21T12:37:22.126985+0000","last_became_active":"2024-09-21T12:36:40.305602+0000","last_became_peered":"2024-09-21T12:36:40.305602+0000","last_unstale":"2024-09-21T12:37:22.126985+0000","last_undegraded":"2024-09-21T12:37:22.126985+0000","last_fullsized":"2024-09-21T12:37:22.126985+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:07.541665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:07.541665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6,3],"avail_no_missing":["0","3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"446'9046","reported_seq":9140,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.865408+0000","last_change":"2024-09-21T12:35:47.791375+0000","last_active":"2024-09-21T12:37:27.865408+0000","last_peered":"2024-09-21T12:37:27.865408+0000","last_clean":"2024-09-21T12:37:27.865408+0000","last_became_active":"2024-09-21T12:35:47.791073+0000","last_became_peered":"2024-09-21T12:35:47.791073+0000","last_unstale":"2024-09-21T12:37:27.865408+0000","last_undegraded":"2024-09-21T12:37:27.865408+0000","last_fullsized":"2024-09-21T12:37:27.865408+0000","mapping_epoch":405,"log_start":"430'5100","ondisk_log_start":"430'5100","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3946,"log_dups_size":0,"ondisk_log_size":3946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536084480,"num_objects":8180,"num_object_clones":0,"num_object_copies":24540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9046,"num_write_kb":551232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"76'79","reported_seq":779,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.520749+0000","last_change":"2024-09-21T12:36:41.495668+0000","last_active":"2024-09-21T12:37:26.520749+0000","last_peered":"2024-09-21T12:37:26.520749+0000","last_clean":"2024-09-21T12:37:26.520749+0000","last_became_active":"2024-09-21T12:32:38.424710+0000","last_became_peered":"2024-09-21T12:32:38.424710+0000","last_unstale":"2024-09-21T12:37:26.520749+0000","last_undegraded":"2024-09-21T12:37:26.520749+0000","last_fullsized":"2024-09-21T12:37:26.520749+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:41.495532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:41.495532+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:56.228849+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":849,"reported_epoch":444,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:37:22.129480+0000","last_change":"2024-09-21T12:34:34.334240+0000","last_active":"2024-09-21T12:37:22.129480+0000","last_peered":"2024-09-21T12:37:22.129480+0000","last_clean":"2024-09-21T12:34:29.290312+0000","last_became_active":"2024-09-21T12:34:34.334240+0000","last_became_peered":"2024-09-21T12:34:34.334240+0000","last_unstale":"2024-09-21T12:37:22.129480+0000","last_undegraded":"2024-09-21T12:34:34.331492+0000","last_fullsized":"2024-09-21T12:34:34.331281+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-21T12:28:45.782955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:28:45.782955+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:51.732517+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":864,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T12:37:22.130155+0000","last_change":"2024-09-21T12:32:53.934634+0000","last_active":"2024-09-21T12:37:22.130155+0000","last_peered":"2024-09-21T12:37:22.130155+0000","last_clean":"2024-09-21T12:37:22.130155+0000","last_became_active":"2024-09-21T12:32:38.411004+0000","last_became_peered":"2024-09-21T12:32:38.411004+0000","last_unstale":"2024-09-21T12:37:22.130155+0000","last_undegraded":"2024-09-21T12:37:22.130155+0000","last_fullsized":"2024-09-21T12:37:22.130155+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:53.934583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:53.934583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":171,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.519029+0000","last_change":"2024-09-21T12:36:27.852222+0000","last_active":"2024-09-21T12:37:26.519029+0000","last_peered":"2024-09-21T12:37:26.519029+0000","last_clean":"2024-09-21T12:37:26.519029+0000","last_became_active":"2024-09-21T12:36:00.144053+0000","last_became_peered":"2024-09-21T12:36:00.144053+0000","last_unstale":"2024-09-21T12:37:26.519029+0000","last_undegraded":"2024-09-21T12:37:26.519029+0000","last_fullsized":"2024-09-21T12:37:26.519029+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-21T12:36:27.852030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:36:27.852030+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:48.419358+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":93,"num_read_kb":88,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":260,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.524442+0000","last_change":"2024-09-21T12:36:36.590854+0000","last_active":"2024-09-21T12:37:26.524442+0000","last_peered":"2024-09-21T12:37:26.524442+0000","last_clean":"2024-09-21T12:37:26.524442+0000","last_became_active":"2024-09-21T12:32:02.030240+0000","last_became_peered":"2024-09-21T12:32:02.030240+0000","last_unstale":"2024-09-21T12:37:26.524442+0000","last_undegraded":"2024-09-21T12:37:26.524442+0000","last_fullsized":"2024-09-21T12:37:26.524442+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:36.590782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:36.590782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:38.226971+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":304,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.519735+0000","last_change":"2024-09-21T12:35:22.978507+0000","last_active":"2024-09-21T12:37:26.519735+0000","last_peered":"2024-09-21T12:37:26.519735+0000","last_clean":"2024-09-21T12:37:26.519735+0000","last_became_active":"2024-09-21T12:31:39.060160+0000","last_became_peered":"2024-09-21T12:31:39.060160+0000","last_unstale":"2024-09-21T12:37:26.519735+0000","last_undegraded":"2024-09-21T12:37:26.519735+0000","last_fullsized":"2024-09-21T12:37:26.519735+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:22.978438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:22.978438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:54.617613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"446'9050","reported_seq":9144,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.864271+0000","last_change":"2024-09-21T12:35:47.849177+0000","last_active":"2024-09-21T12:37:27.864271+0000","last_peered":"2024-09-21T12:37:27.864271+0000","last_clean":"2024-09-21T12:37:27.864271+0000","last_became_active":"2024-09-21T12:35:47.848768+0000","last_became_peered":"2024-09-21T12:35:47.848768+0000","last_unstale":"2024-09-21T12:37:27.864271+0000","last_undegraded":"2024-09-21T12:37:27.864271+0000","last_fullsized":"2024-09-21T12:37:27.864271+0000","mapping_epoch":405,"log_start":"430'5200","ondisk_log_start":"430'5200","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3850,"log_dups_size":0,"ondisk_log_size":3850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536739840,"num_objects":8190,"num_object_clones":0,"num_object_copies":24570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9050,"num_write_kb":551680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"415'35","reported_seq":346,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.531846+0000","last_change":"2024-09-21T12:36:43.681619+0000","last_active":"2024-09-21T12:37:26.531846+0000","last_peered":"2024-09-21T12:37:26.531846+0000","last_clean":"2024-09-21T12:37:26.531846+0000","last_became_active":"2024-09-21T12:36:00.139342+0000","last_became_peered":"2024-09-21T12:36:00.139342+0000","last_unstale":"2024-09-21T12:37:26.531846+0000","last_undegraded":"2024-09-21T12:37:26.531846+0000","last_fullsized":"2024-09-21T12:37:26.531846+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'35","last_scrub_stamp":"2024-09-21T12:36:43.681471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:43.681471+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:50.711798+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":829,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.137594+0000","last_change":"2024-09-21T12:34:01.815496+0000","last_active":"2024-09-21T12:37:27.137594+0000","last_peered":"2024-09-21T12:37:27.137594+0000","last_clean":"2024-09-21T12:37:27.137594+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:37:27.137594+0000","last_undegraded":"2024-09-21T12:37:27.137594+0000","last_fullsized":"2024-09-21T12:37:27.137594+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:34:01.815445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:34:01.815445+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":80,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.520258+0000","last_change":"2024-09-21T12:37:14.690524+0000","last_active":"2024-09-21T12:37:26.520258+0000","last_peered":"2024-09-21T12:37:26.520258+0000","last_clean":"2024-09-21T12:37:26.520258+0000","last_became_active":"2024-09-21T12:36:00.134878+0000","last_became_peered":"2024-09-21T12:36:00.134878+0000","last_unstale":"2024-09-21T12:37:26.520258+0000","last_undegraded":"2024-09-21T12:37:26.520258+0000","last_fullsized":"2024-09-21T12:37:26.520258+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:37:14.690456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:37:14.690456+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:27.199911+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":241,"reported_epoch":445,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:37:26.519443+0000","last_change":"2024-09-21T12:37:22.649664+0000","last_active":"2024-09-21T12:37:26.519443+0000","last_peered":"2024-09-21T12:37:26.519443+0000","last_clean":"2024-09-21T12:37:26.519443+0000","last_became_active":"2024-09-21T12:36:01.129957+0000","last_became_peered":"2024-09-21T12:36:01.129957+0000","last_unstale":"2024-09-21T12:37:26.519443+0000","last_undegraded":"2024-09-21T12:37:26.519443+0000","last_fullsized":"2024-09-21T12:37:26.519443+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:22.649558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:37:22.649558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:45.743489+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5,0],"avail_no_missing":["7","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":477,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.524753+0000","last_change":"2024-09-21T12:37:20.485399+0000","last_active":"2024-09-21T12:37:26.524753+0000","last_peered":"2024-09-21T12:37:26.524753+0000","last_clean":"2024-09-21T12:37:26.524753+0000","last_became_active":"2024-09-21T12:37:20.485126+0000","last_became_peered":"2024-09-21T12:37:20.485126+0000","last_unstale":"2024-09-21T12:37:26.524753+0000","last_undegraded":"2024-09-21T12:37:26.524753+0000","last_fullsized":"2024-09-21T12:37:26.524753+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:00.653130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:00.653130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"446'9023","reported_seq":9117,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.864775+0000","last_change":"2024-09-21T12:35:47.788808+0000","last_active":"2024-09-21T12:37:27.864775+0000","last_peered":"2024-09-21T12:37:27.864775+0000","last_clean":"2024-09-21T12:37:27.864775+0000","last_became_active":"2024-09-21T12:35:47.788318+0000","last_became_peered":"2024-09-21T12:35:47.788318+0000","last_unstale":"2024-09-21T12:37:27.864775+0000","last_undegraded":"2024-09-21T12:37:27.864775+0000","last_fullsized":"2024-09-21T12:37:27.864775+0000","mapping_epoch":405,"log_start":"430'5100","ondisk_log_start":"430'5100","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3923,"log_dups_size":0,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540344320,"num_objects":8245,"num_object_clones":0,"num_object_copies":24735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9023,"num_write_kb":552576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"76'79","reported_seq":767,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.520965+0000","last_change":"2024-09-21T12:33:04.508642+0000","last_active":"2024-09-21T12:37:26.520965+0000","last_peered":"2024-09-21T12:37:26.520965+0000","last_clean":"2024-09-21T12:37:26.520965+0000","last_became_active":"2024-09-21T12:27:14.954351+0000","last_became_peered":"2024-09-21T12:27:14.954351+0000","last_unstale":"2024-09-21T12:37:26.520965+0000","last_undegraded":"2024-09-21T12:37:26.520965+0000","last_fullsized":"2024-09-21T12:37:26.520965+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:04.508319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:04.508319+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":232,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.531236+0000","last_change":"2024-09-21T12:36:41.666876+0000","last_active":"2024-09-21T12:37:26.531236+0000","last_peered":"2024-09-21T12:37:26.531236+0000","last_clean":"2024-09-21T12:37:26.531236+0000","last_became_active":"2024-09-21T12:32:38.419714+0000","last_became_peered":"2024-09-21T12:32:38.419714+0000","last_unstale":"2024-09-21T12:37:26.531236+0000","last_undegraded":"2024-09-21T12:37:26.531236+0000","last_fullsized":"2024-09-21T12:37:26.531236+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:36:41.666812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:41.666812+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:50.318849+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":236,"reported_epoch":445,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:37:26.529474+0000","last_change":"2024-09-21T12:36:40.708599+0000","last_active":"2024-09-21T12:37:26.529474+0000","last_peered":"2024-09-21T12:37:26.529474+0000","last_clean":"2024-09-21T12:37:26.529474+0000","last_became_active":"2024-09-21T12:36:01.130874+0000","last_became_peered":"2024-09-21T12:36:01.130874+0000","last_unstale":"2024-09-21T12:37:26.529474+0000","last_undegraded":"2024-09-21T12:37:26.529474+0000","last_fullsized":"2024-09-21T12:37:26.529474+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:40.708469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:36:40.708469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:10.539508+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7,1],"avail_no_missing":["4","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":356,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.524974+0000","last_change":"2024-09-21T12:37:20.468577+0000","last_active":"2024-09-21T12:37:26.524974+0000","last_peered":"2024-09-21T12:37:26.524974+0000","last_clean":"2024-09-21T12:37:26.524974+0000","last_became_active":"2024-09-21T12:37:20.467986+0000","last_became_peered":"2024-09-21T12:37:20.467986+0000","last_unstale":"2024-09-21T12:37:26.524974+0000","last_undegraded":"2024-09-21T12:37:26.524974+0000","last_fullsized":"2024-09-21T12:37:26.524974+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:59.196667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:33:59.196667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":810,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.135338+0000","last_change":"2024-09-21T12:35:24.786713+0000","last_active":"2024-09-21T12:37:27.135338+0000","last_peered":"2024-09-21T12:37:27.135338+0000","last_clean":"2024-09-21T12:37:27.135338+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:37:27.135338+0000","last_undegraded":"2024-09-21T12:37:27.135338+0000","last_fullsized":"2024-09-21T12:37:27.135338+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:24.786609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:24.786609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"446'8982","reported_seq":9073,"reported_epoch":446,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:37:27.858379+0000","last_change":"2024-09-21T12:35:53.505439+0000","last_active":"2024-09-21T12:37:27.858379+0000","last_peered":"2024-09-21T12:37:27.858379+0000","last_clean":"2024-09-21T12:35:46.767214+0000","last_became_active":"2024-09-21T12:35:47.787239+0000","last_became_peered":"2024-09-21T12:35:47.787239+0000","last_unstale":"2024-09-21T12:37:27.858379+0000","last_undegraded":"2024-09-21T12:35:53.504658+0000","last_fullsized":"2024-09-21T12:35:47.783070+0000","mapping_epoch":405,"log_start":"430'5100","ondisk_log_start":"430'5100","created":405,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":3882,"log_dups_size":0,"ondisk_log_size":3882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536739840,"num_objects":8190,"num_object_clones":0,"num_object_copies":24570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8982,"num_write_kb":549504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":8190}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"76'79","reported_seq":150,"reported_epoch":446,"state":"active+undersized","last_fresh":"2024-09-21T12:37:27.135389+0000","last_change":"2024-09-21T12:34:34.338192+0000","last_active":"2024-09-21T12:37:27.135389+0000","last_peered":"2024-09-21T12:37:27.135389+0000","last_clean":"2024-09-21T12:34:29.289953+0000","last_became_active":"2024-09-21T12:34:34.338192+0000","last_became_peered":"2024-09-21T12:34:34.338192+0000","last_unstale":"2024-09-21T12:37:27.135389+0000","last_undegraded":"2024-09-21T12:37:27.135389+0000","last_fullsized":"2024-09-21T12:34:34.334873+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:17.035216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:17.035216+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:10.791491+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":234,"reported_epoch":445,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:37:26.531483+0000","last_change":"2024-09-21T12:34:34.341886+0000","last_active":"2024-09-21T12:37:26.531483+0000","last_peered":"2024-09-21T12:37:26.531483+0000","last_clean":"2024-09-21T12:34:29.279382+0000","last_became_active":"2024-09-21T12:34:34.341886+0000","last_became_peered":"2024-09-21T12:34:34.341886+0000","last_unstale":"2024-09-21T12:37:26.531483+0000","last_undegraded":"2024-09-21T12:34:34.333296+0000","last_fullsized":"2024-09-21T12:34:34.332270+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'31","last_scrub_stamp":"2024-09-21T12:33:08.360012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:08.360012+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":220,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.530202+0000","last_change":"2024-09-21T12:35:18.088890+0000","last_active":"2024-09-21T12:37:26.530202+0000","last_peered":"2024-09-21T12:37:26.530202+0000","last_clean":"2024-09-21T12:37:26.530202+0000","last_became_active":"2024-09-21T12:32:38.424451+0000","last_became_peered":"2024-09-21T12:32:38.424451+0000","last_unstale":"2024-09-21T12:37:26.530202+0000","last_undegraded":"2024-09-21T12:37:26.530202+0000","last_fullsized":"2024-09-21T12:37:26.530202+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:18.088831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:18.088831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":841,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.519824+0000","last_change":"2024-09-21T12:36:43.539109+0000","last_active":"2024-09-21T12:37:26.519824+0000","last_peered":"2024-09-21T12:37:26.519824+0000","last_clean":"2024-09-21T12:37:26.519824+0000","last_became_active":"2024-09-21T12:32:38.425177+0000","last_became_peered":"2024-09-21T12:32:38.425177+0000","last_unstale":"2024-09-21T12:37:26.519824+0000","last_undegraded":"2024-09-21T12:37:26.519824+0000","last_fullsized":"2024-09-21T12:37:26.519824+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:43.539053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:43.539053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:59.470609+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":760,"reported_epoch":446,"state":"active+undersized","last_fresh":"2024-09-21T12:37:27.136784+0000","last_change":"2024-09-21T12:34:34.345710+0000","last_active":"2024-09-21T12:37:27.136784+0000","last_peered":"2024-09-21T12:37:27.136784+0000","last_clean":"2024-09-21T12:34:29.294810+0000","last_became_active":"2024-09-21T12:34:34.345710+0000","last_became_peered":"2024-09-21T12:34:34.345710+0000","last_unstale":"2024-09-21T12:37:27.136784+0000","last_undegraded":"2024-09-21T12:37:27.136784+0000","last_fullsized":"2024-09-21T12:34:34.334512+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:20.005906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:34:20.005906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3","version":"445'8969","reported_seq":11269,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.965007+0000","last_change":"2024-09-21T12:37:15.781544+0000","last_active":"2024-09-21T12:37:26.965007+0000","last_peered":"2024-09-21T12:37:26.965007+0000","last_clean":"2024-09-21T12:37:26.965007+0000","last_became_active":"2024-09-21T12:36:13.105381+0000","last_became_peered":"2024-09-21T12:36:13.105381+0000","last_unstale":"2024-09-21T12:37:26.965007+0000","last_undegraded":"2024-09-21T12:37:26.965007+0000","last_fullsized":"2024-09-21T12:37:26.965007+0000","mapping_epoch":420,"log_start":"426'4400","ondisk_log_start":"426'4400","created":405,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"438'8074","last_scrub_stamp":"2024-09-21T12:37:15.781342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:37:15.781342+0000","objects_scrubbed":8004,"log_size":4569,"log_dups_size":0,"ondisk_log_size":4569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:44.284324+0000","scrub_duration":2612,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":542703616,"num_objects":8281,"num_object_clones":0,"num_object_copies":24843,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8969,"num_write_kb":552000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1051,"num_bytes_recovered":68878336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"76'79","reported_seq":885,"reported_epoch":445,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:37:26.520084+0000","last_change":"2024-09-21T12:36:01.128488+0000","last_active":"2024-09-21T12:37:26.520084+0000","last_peered":"2024-09-21T12:37:26.520084+0000","last_clean":"2024-09-21T12:37:26.520084+0000","last_became_active":"2024-09-21T12:36:01.127851+0000","last_became_peered":"2024-09-21T12:36:01.127851+0000","last_unstale":"2024-09-21T12:37:26.520084+0000","last_undegraded":"2024-09-21T12:37:26.520084+0000","last_fullsized":"2024-09-21T12:37:26.520084+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:29:05.500994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:29:05.500994+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:26.624519+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,5],"avail_no_missing":["7","3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":220,"reported_epoch":445,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:37:26.530630+0000","last_change":"2024-09-21T12:34:34.347138+0000","last_active":"2024-09-21T12:37:26.530630+0000","last_peered":"2024-09-21T12:37:26.530630+0000","last_clean":"2024-09-21T12:34:29.280929+0000","last_became_active":"2024-09-21T12:34:34.347138+0000","last_became_peered":"2024-09-21T12:34:34.347138+0000","last_unstale":"2024-09-21T12:37:26.530630+0000","last_undegraded":"2024-09-21T12:34:34.332331+0000","last_fullsized":"2024-09-21T12:34:34.330784+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:32:26.690403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:26.690403+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:24.672389+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":842,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.136683+0000","last_change":"2024-09-21T12:32:38.404305+0000","last_active":"2024-09-21T12:37:27.136683+0000","last_peered":"2024-09-21T12:37:27.136683+0000","last_clean":"2024-09-21T12:37:27.136683+0000","last_became_active":"2024-09-21T12:32:38.403997+0000","last_became_peered":"2024-09-21T12:32:38.403997+0000","last_unstale":"2024-09-21T12:37:27.136683+0000","last_undegraded":"2024-09-21T12:37:27.136683+0000","last_fullsized":"2024-09-21T12:37:27.136683+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:31:47.248389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:31:47.248389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:26.925030+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"416'14","reported_seq":204,"reported_epoch":445,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:37:26.521038+0000","last_change":"2024-09-21T12:36:31.838103+0000","last_active":"2024-09-21T12:37:26.521038+0000","last_peered":"2024-09-21T12:37:26.521038+0000","last_clean":"2024-09-21T12:37:26.521038+0000","last_became_active":"2024-09-21T12:36:01.130569+0000","last_became_peered":"2024-09-21T12:36:01.130569+0000","last_unstale":"2024-09-21T12:37:26.521038+0000","last_undegraded":"2024-09-21T12:37:26.521038+0000","last_fullsized":"2024-09-21T12:37:26.521038+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'14","last_scrub_stamp":"2024-09-21T12:36:31.838024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:31.838024+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:59.227849+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5,0],"avail_no_missing":["7","0","5"],"object_location_counts":[{"shards":"0,5,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":216,"reported_epoch":445,"state":"active+undersized","last_fresh":"2024-09-21T12:37:26.530939+0000","last_change":"2024-09-21T12:34:34.348303+0000","last_active":"2024-09-21T12:37:26.530939+0000","last_peered":"2024-09-21T12:37:26.530939+0000","last_clean":"2024-09-21T12:34:29.279670+0000","last_became_active":"2024-09-21T12:34:34.348303+0000","last_became_peered":"2024-09-21T12:34:34.348303+0000","last_unstale":"2024-09-21T12:37:26.530939+0000","last_undegraded":"2024-09-21T12:37:26.530939+0000","last_fullsized":"2024-09-21T12:34:34.331636+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:06.369436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:06.369436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"445'9027","reported_seq":9207,"reported_epoch":445,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:37:26.991090+0000","last_change":"2024-09-21T12:37:18.160092+0000","last_active":"2024-09-21T12:37:26.991090+0000","last_peered":"2024-09-21T12:37:26.991090+0000","last_clean":"2024-09-21T12:37:26.991090+0000","last_became_active":"2024-09-21T12:36:01.132703+0000","last_became_peered":"2024-09-21T12:36:01.132703+0000","last_unstale":"2024-09-21T12:37:26.991090+0000","last_undegraded":"2024-09-21T12:37:26.991090+0000","last_fullsized":"2024-09-21T12:37:26.991090+0000","mapping_epoch":415,"log_start":"429'4800","ondisk_log_start":"429'4800","created":405,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'8269","last_scrub_stamp":"2024-09-21T12:37:18.159924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:37:18.159924+0000","objects_scrubbed":8165,"log_size":4227,"log_dups_size":0,"ondisk_log_size":4227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:28.059203+0000","scrub_duration":2527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":542965760,"num_objects":8285,"num_object_clones":0,"num_object_copies":24855,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8285,"num_objects_unfound":0,"num_objects_dirty":8285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9027,"num_write_kb":553984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6,1],"avail_no_missing":["7","1","6"],"object_location_counts":[{"shards":"1,6,7","objects":8285}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"76'79","reported_seq":224,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.531315+0000","last_change":"2024-09-21T12:36:01.159899+0000","last_active":"2024-09-21T12:37:26.531315+0000","last_peered":"2024-09-21T12:37:26.531315+0000","last_clean":"2024-09-21T12:37:26.531315+0000","last_became_active":"2024-09-21T12:36:01.159481+0000","last_became_peered":"2024-09-21T12:36:01.159481+0000","last_unstale":"2024-09-21T12:37:26.531315+0000","last_undegraded":"2024-09-21T12:37:26.531315+0000","last_fullsized":"2024-09-21T12:37:26.531315+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:49.155069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:49.155069+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":934,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T12:37:22.128990+0000","last_change":"2024-09-21T12:36:40.771296+0000","last_active":"2024-09-21T12:37:22.128990+0000","last_peered":"2024-09-21T12:37:22.128990+0000","last_clean":"2024-09-21T12:37:22.128990+0000","last_became_active":"2024-09-21T12:32:38.424376+0000","last_became_peered":"2024-09-21T12:32:38.424376+0000","last_unstale":"2024-09-21T12:37:22.128990+0000","last_undegraded":"2024-09-21T12:37:22.128990+0000","last_fullsized":"2024-09-21T12:37:22.128990+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:36:40.771222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:40.771222+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:42.363964+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":351,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.525184+0000","last_change":"2024-09-21T12:37:20.485093+0000","last_active":"2024-09-21T12:37:26.525184+0000","last_peered":"2024-09-21T12:37:26.525184+0000","last_clean":"2024-09-21T12:37:26.525184+0000","last_became_active":"2024-09-21T12:37:20.484773+0000","last_became_peered":"2024-09-21T12:37:20.484773+0000","last_unstale":"2024-09-21T12:37:26.525184+0000","last_undegraded":"2024-09-21T12:37:26.525184+0000","last_fullsized":"2024-09-21T12:37:26.525184+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:36.079583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:34:36.079583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":778,"reported_epoch":445,"state":"active+undersized","last_fresh":"2024-09-21T12:37:26.530286+0000","last_change":"2024-09-21T12:34:34.346822+0000","last_active":"2024-09-21T12:37:26.530286+0000","last_peered":"2024-09-21T12:37:26.530286+0000","last_clean":"2024-09-21T12:34:29.278339+0000","last_became_active":"2024-09-21T12:34:34.346822+0000","last_became_peered":"2024-09-21T12:34:34.346822+0000","last_unstale":"2024-09-21T12:37:26.530286+0000","last_undegraded":"2024-09-21T12:37:26.530286+0000","last_fullsized":"2024-09-21T12:34:34.330172+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:28:47.361115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:28:47.361115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:12.595398+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":825,"reported_epoch":444,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:37:22.130243+0000","last_change":"2024-09-21T12:36:34.842873+0000","last_active":"2024-09-21T12:37:22.130243+0000","last_peered":"2024-09-21T12:37:22.130243+0000","last_clean":"2024-09-21T12:37:22.130243+0000","last_became_active":"2024-09-21T12:36:01.130370+0000","last_became_peered":"2024-09-21T12:36:01.130370+0000","last_unstale":"2024-09-21T12:37:22.130243+0000","last_undegraded":"2024-09-21T12:37:22.130243+0000","last_fullsized":"2024-09-21T12:37:22.130243+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:34.842666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:34.842666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:56.753346+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7,3],"avail_no_missing":["0","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"445'8871","reported_seq":11159,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.970423+0000","last_change":"2024-09-21T12:37:23.352215+0000","last_active":"2024-09-21T12:37:26.970423+0000","last_peered":"2024-09-21T12:37:26.970423+0000","last_clean":"2024-09-21T12:37:26.970423+0000","last_became_active":"2024-09-21T12:36:12.104630+0000","last_became_peered":"2024-09-21T12:36:12.104630+0000","last_unstale":"2024-09-21T12:37:26.970423+0000","last_undegraded":"2024-09-21T12:37:26.970423+0000","last_fullsized":"2024-09-21T12:37:26.970423+0000","mapping_epoch":419,"log_start":"427'4600","ondisk_log_start":"427'4600","created":405,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'8526","last_scrub_stamp":"2024-09-21T12:37:23.351998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:37:23.351998+0000","objects_scrubbed":8413,"log_size":4271,"log_dups_size":0,"ondisk_log_size":4271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:50.828150+0000","scrub_duration":2758,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":537591808,"num_objects":8203,"num_object_clones":0,"num_object_copies":24609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8871,"num_write_kb":546368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1047,"num_bytes_recovered":68616192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"76'79","reported_seq":751,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.524878+0000","last_change":"2024-09-21T12:35:57.172169+0000","last_active":"2024-09-21T12:37:26.524878+0000","last_peered":"2024-09-21T12:37:26.524878+0000","last_clean":"2024-09-21T12:37:26.524878+0000","last_became_active":"2024-09-21T12:32:38.414877+0000","last_became_peered":"2024-09-21T12:32:38.414877+0000","last_unstale":"2024-09-21T12:37:26.524878+0000","last_undegraded":"2024-09-21T12:37:26.524878+0000","last_fullsized":"2024-09-21T12:37:26.524878+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:35:57.172103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:35:57.172103+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:07.424536+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":996,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.523408+0000","last_change":"2024-09-21T12:37:18.028663+0000","last_active":"2024-09-21T12:37:26.523408+0000","last_peered":"2024-09-21T12:37:26.523408+0000","last_clean":"2024-09-21T12:37:26.523408+0000","last_became_active":"2024-09-21T12:36:00.148392+0000","last_became_peered":"2024-09-21T12:36:00.148392+0000","last_unstale":"2024-09-21T12:37:26.523408+0000","last_undegraded":"2024-09-21T12:37:26.523408+0000","last_fullsized":"2024-09-21T12:37:26.523408+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'35","last_scrub_stamp":"2024-09-21T12:37:18.028567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:37:18.028567+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:43.446555+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":72,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.520707+0000","last_change":"2024-09-21T12:36:00.145588+0000","last_active":"2024-09-21T12:37:26.520707+0000","last_peered":"2024-09-21T12:37:26.520707+0000","last_clean":"2024-09-21T12:37:26.520707+0000","last_became_active":"2024-09-21T12:36:00.145287+0000","last_became_peered":"2024-09-21T12:36:00.145287+0000","last_unstale":"2024-09-21T12:37:26.520707+0000","last_undegraded":"2024-09-21T12:37:26.520707+0000","last_fullsized":"2024-09-21T12:37:26.520707+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:58.197610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:58.197610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":714,"reported_epoch":444,"state":"active+undersized","last_fresh":"2024-09-21T12:37:22.131789+0000","last_change":"2024-09-21T12:34:34.343106+0000","last_active":"2024-09-21T12:37:22.131789+0000","last_peered":"2024-09-21T12:37:22.131789+0000","last_clean":"2024-09-21T12:34:29.291366+0000","last_became_active":"2024-09-21T12:34:34.343106+0000","last_became_peered":"2024-09-21T12:34:34.343106+0000","last_unstale":"2024-09-21T12:37:22.131789+0000","last_undegraded":"2024-09-21T12:37:22.131789+0000","last_fullsized":"2024-09-21T12:34:34.333287+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:20.243774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:20.243774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:37.780982+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":798,"reported_epoch":445,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:37:26.520527+0000","last_change":"2024-09-21T12:36:28.804571+0000","last_active":"2024-09-21T12:37:26.520527+0000","last_peered":"2024-09-21T12:37:26.520527+0000","last_clean":"2024-09-21T12:37:26.520527+0000","last_became_active":"2024-09-21T12:36:01.135641+0000","last_became_peered":"2024-09-21T12:36:01.135641+0000","last_unstale":"2024-09-21T12:37:26.520527+0000","last_undegraded":"2024-09-21T12:37:26.520527+0000","last_fullsized":"2024-09-21T12:37:26.520527+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:28.804459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:28.804459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:45.512216+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,6],"avail_no_missing":["7","3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"444'8574","reported_seq":8660,"reported_epoch":444,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:37:23.425857+0000","last_change":"2024-09-21T12:35:53.511573+0000","last_active":"2024-09-21T12:37:23.425857+0000","last_peered":"2024-09-21T12:37:23.425857+0000","last_clean":"2024-09-21T12:35:46.767214+0000","last_became_active":"2024-09-21T12:35:47.793577+0000","last_became_peered":"2024-09-21T12:35:47.793577+0000","last_unstale":"2024-09-21T12:37:23.425857+0000","last_undegraded":"2024-09-21T12:35:53.504929+0000","last_fullsized":"2024-09-21T12:35:47.782699+0000","mapping_epoch":405,"log_start":"426'3923","ondisk_log_start":"426'3923","created":405,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4651,"log_dups_size":0,"ondisk_log_size":4651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561905664,"num_objects":8574,"num_object_clones":0,"num_object_copies":25722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8574,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8574,"num_write_kb":548736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":8574}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"76'79","reported_seq":776,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.135580+0000","last_change":"2024-09-21T12:36:31.563764+0000","last_active":"2024-09-21T12:37:27.135580+0000","last_peered":"2024-09-21T12:37:27.135580+0000","last_clean":"2024-09-21T12:37:27.135580+0000","last_became_active":"2024-09-21T12:27:14.951375+0000","last_became_peered":"2024-09-21T12:27:14.951375+0000","last_unstale":"2024-09-21T12:37:27.135580+0000","last_undegraded":"2024-09-21T12:37:27.135580+0000","last_fullsized":"2024-09-21T12:37:27.135580+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:31.563694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:31.563694+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:35.930543+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":224,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.529749+0000","last_change":"2024-09-21T12:35:12.197153+0000","last_active":"2024-09-21T12:37:26.529749+0000","last_peered":"2024-09-21T12:37:26.529749+0000","last_clean":"2024-09-21T12:37:26.529749+0000","last_became_active":"2024-09-21T12:32:38.421609+0000","last_became_peered":"2024-09-21T12:32:38.421609+0000","last_unstale":"2024-09-21T12:37:26.529749+0000","last_undegraded":"2024-09-21T12:37:26.529749+0000","last_fullsized":"2024-09-21T12:37:26.529749+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T12:35:12.197060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:12.197060+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":72,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.520865+0000","last_change":"2024-09-21T12:36:00.151126+0000","last_active":"2024-09-21T12:37:26.520865+0000","last_peered":"2024-09-21T12:37:26.520865+0000","last_clean":"2024-09-21T12:37:26.520865+0000","last_became_active":"2024-09-21T12:36:00.150832+0000","last_became_peered":"2024-09-21T12:36:00.150832+0000","last_unstale":"2024-09-21T12:37:26.520865+0000","last_undegraded":"2024-09-21T12:37:26.520865+0000","last_fullsized":"2024-09-21T12:37:26.520865+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:19.611537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:19.611537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:54.751976+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":136,"reported_epoch":444,"state":"active+undersized","last_fresh":"2024-09-21T12:37:22.127702+0000","last_change":"2024-09-21T12:34:34.333741+0000","last_active":"2024-09-21T12:37:22.127702+0000","last_peered":"2024-09-21T12:37:22.127702+0000","last_clean":"2024-09-21T12:34:29.289730+0000","last_became_active":"2024-09-21T12:34:34.333741+0000","last_became_peered":"2024-09-21T12:34:34.333741+0000","last_unstale":"2024-09-21T12:37:22.127702+0000","last_undegraded":"2024-09-21T12:37:22.127702+0000","last_fullsized":"2024-09-21T12:34:34.330975+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:23.769748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:32:23.769748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:13.655604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":838,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.135790+0000","last_change":"2024-09-21T12:36:00.139665+0000","last_active":"2024-09-21T12:37:27.135790+0000","last_peered":"2024-09-21T12:37:27.135790+0000","last_clean":"2024-09-21T12:37:27.135790+0000","last_became_active":"2024-09-21T12:36:00.139300+0000","last_became_peered":"2024-09-21T12:36:00.139300+0000","last_unstale":"2024-09-21T12:37:27.135790+0000","last_undegraded":"2024-09-21T12:37:27.135790+0000","last_fullsized":"2024-09-21T12:37:27.135790+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:04.876656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:34:04.876656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:59.665165+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"445'9173","reported_seq":9363,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.871488+0000","last_change":"2024-09-21T12:37:20.472851+0000","last_active":"2024-09-21T12:37:26.871488+0000","last_peered":"2024-09-21T12:37:26.871488+0000","last_clean":"2024-09-21T12:37:26.871488+0000","last_became_active":"2024-09-21T12:37:20.468527+0000","last_became_peered":"2024-09-21T12:37:20.468527+0000","last_unstale":"2024-09-21T12:37:26.871488+0000","last_undegraded":"2024-09-21T12:37:26.871488+0000","last_fullsized":"2024-09-21T12:37:26.871488+0000","mapping_epoch":441,"log_start":"427'4700","ondisk_log_start":"427'4700","created":405,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"438'8315","last_scrub_stamp":"2024-09-21T12:37:16.765348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:37:16.765348+0000","objects_scrubbed":8225,"log_size":4473,"log_dups_size":0,"ondisk_log_size":4473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:31.757528+0000","scrub_duration":2844,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":557907968,"num_objects":8513,"num_object_clones":0,"num_object_copies":25539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9173,"num_write_kb":565952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"76'79","reported_seq":252,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.521033+0000","last_change":"2024-09-21T12:36:36.859482+0000","last_active":"2024-09-21T12:37:26.521033+0000","last_peered":"2024-09-21T12:37:26.521033+0000","last_clean":"2024-09-21T12:37:26.521033+0000","last_became_active":"2024-09-21T12:36:01.141807+0000","last_became_peered":"2024-09-21T12:36:01.141807+0000","last_unstale":"2024-09-21T12:37:26.521033+0000","last_undegraded":"2024-09-21T12:37:26.521033+0000","last_fullsized":"2024-09-21T12:37:26.521033+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:36.859412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:36.859412+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:54.672933+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":353,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.525486+0000","last_change":"2024-09-21T12:36:00.171576+0000","last_active":"2024-09-21T12:37:26.525486+0000","last_peered":"2024-09-21T12:37:26.525486+0000","last_clean":"2024-09-21T12:37:26.525486+0000","last_became_active":"2024-09-21T12:36:00.137597+0000","last_became_peered":"2024-09-21T12:36:00.137597+0000","last_unstale":"2024-09-21T12:37:26.525486+0000","last_undegraded":"2024-09-21T12:37:26.525486+0000","last_fullsized":"2024-09-21T12:37:26.525486+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:35:52.226406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:52.226406+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:53.571128+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":86,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.521326+0000","last_change":"2024-09-21T12:37:11.677270+0000","last_active":"2024-09-21T12:37:26.521326+0000","last_peered":"2024-09-21T12:37:26.521326+0000","last_clean":"2024-09-21T12:37:26.521326+0000","last_became_active":"2024-09-21T12:36:00.148899+0000","last_became_peered":"2024-09-21T12:36:00.148899+0000","last_unstale":"2024-09-21T12:37:26.521326+0000","last_undegraded":"2024-09-21T12:37:26.521326+0000","last_fullsized":"2024-09-21T12:37:26.521326+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-21T12:37:11.677196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:37:11.677196+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:30.541193+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"415'16","reported_seq":94,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.519716+0000","last_change":"2024-09-21T12:36:00.165364+0000","last_active":"2024-09-21T12:37:26.519716+0000","last_peered":"2024-09-21T12:37:26.519716+0000","last_clean":"2024-09-21T12:37:26.519716+0000","last_became_active":"2024-09-21T12:36:00.143091+0000","last_became_peered":"2024-09-21T12:36:00.143091+0000","last_unstale":"2024-09-21T12:37:26.519716+0000","last_undegraded":"2024-09-21T12:37:26.519716+0000","last_fullsized":"2024-09-21T12:37:26.519716+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'14","last_scrub_stamp":"2024-09-21T12:35:06.542700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:35:06.542700+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:58.759100+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":323,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.525050+0000","last_change":"2024-09-21T12:37:20.468878+0000","last_active":"2024-09-21T12:37:26.525050+0000","last_peered":"2024-09-21T12:37:26.525050+0000","last_clean":"2024-09-21T12:37:26.525050+0000","last_became_active":"2024-09-21T12:37:20.468350+0000","last_became_peered":"2024-09-21T12:37:20.468350+0000","last_unstale":"2024-09-21T12:37:26.525050+0000","last_undegraded":"2024-09-21T12:37:26.525050+0000","last_fullsized":"2024-09-21T12:37:26.525050+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:34.998795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:34.998795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:59.966596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"76'79","reported_seq":550,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.524840+0000","last_change":"2024-09-21T12:32:26.698667+0000","last_active":"2024-09-21T12:37:26.524840+0000","last_peered":"2024-09-21T12:37:26.524840+0000","last_clean":"2024-09-21T12:37:26.524840+0000","last_became_active":"2024-09-21T12:31:39.060422+0000","last_became_peered":"2024-09-21T12:31:39.060422+0000","last_unstale":"2024-09-21T12:37:26.524840+0000","last_undegraded":"2024-09-21T12:37:26.524840+0000","last_fullsized":"2024-09-21T12:37:26.524840+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:26.698622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:26.698622+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:02.504452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":867,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.523408+0000","last_change":"2024-09-21T12:35:59.183091+0000","last_active":"2024-09-21T12:37:26.523408+0000","last_peered":"2024-09-21T12:37:26.523408+0000","last_clean":"2024-09-21T12:37:26.523408+0000","last_became_active":"2024-09-21T12:32:02.026693+0000","last_became_peered":"2024-09-21T12:32:02.026693+0000","last_unstale":"2024-09-21T12:37:26.523408+0000","last_undegraded":"2024-09-21T12:37:26.523408+0000","last_fullsized":"2024-09-21T12:37:26.523408+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:59.183047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:59.183047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:04.182782+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":325,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.137549+0000","last_change":"2024-09-21T12:36:01.139003+0000","last_active":"2024-09-21T12:37:27.137549+0000","last_peered":"2024-09-21T12:37:27.137549+0000","last_clean":"2024-09-21T12:37:27.137549+0000","last_became_active":"2024-09-21T12:36:01.138626+0000","last_became_peered":"2024-09-21T12:36:01.138626+0000","last_unstale":"2024-09-21T12:37:27.137549+0000","last_undegraded":"2024-09-21T12:37:27.137549+0000","last_fullsized":"2024-09-21T12:37:27.137549+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T12:34:09.872320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:34:09.872320+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:58.661408+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":343,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.524638+0000","last_change":"2024-09-21T12:37:20.486113+0000","last_active":"2024-09-21T12:37:26.524638+0000","last_peered":"2024-09-21T12:37:26.524638+0000","last_clean":"2024-09-21T12:37:26.524638+0000","last_became_active":"2024-09-21T12:37:20.485806+0000","last_became_peered":"2024-09-21T12:37:20.485806+0000","last_unstale":"2024-09-21T12:37:26.524638+0000","last_undegraded":"2024-09-21T12:37:26.524638+0000","last_fullsized":"2024-09-21T12:37:26.524638+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:42.510764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:42.510764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:55.014354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":72,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.520684+0000","last_change":"2024-09-21T12:36:00.149496+0000","last_active":"2024-09-21T12:37:26.520684+0000","last_peered":"2024-09-21T12:37:26.520684+0000","last_clean":"2024-09-21T12:37:26.520684+0000","last_became_active":"2024-09-21T12:36:00.149086+0000","last_became_peered":"2024-09-21T12:36:00.149086+0000","last_unstale":"2024-09-21T12:37:26.520684+0000","last_undegraded":"2024-09-21T12:37:26.520684+0000","last_fullsized":"2024-09-21T12:37:26.520684+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:14.566963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:14.566963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:00.770630+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'79","reported_seq":798,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.520834+0000","last_change":"2024-09-21T12:36:39.411851+0000","last_active":"2024-09-21T12:37:26.520834+0000","last_peered":"2024-09-21T12:37:26.520834+0000","last_clean":"2024-09-21T12:37:26.520834+0000","last_became_active":"2024-09-21T12:32:38.425495+0000","last_became_peered":"2024-09-21T12:32:38.425495+0000","last_unstale":"2024-09-21T12:37:26.520834+0000","last_undegraded":"2024-09-21T12:37:26.520834+0000","last_fullsized":"2024-09-21T12:37:26.520834+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:39.411785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:39.411785+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:43.123682+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":889,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T12:37:22.129662+0000","last_change":"2024-09-21T12:36:00.142630+0000","last_active":"2024-09-21T12:37:22.129662+0000","last_peered":"2024-09-21T12:37:22.129662+0000","last_clean":"2024-09-21T12:37:22.129662+0000","last_became_active":"2024-09-21T12:36:00.142274+0000","last_became_peered":"2024-09-21T12:36:00.142274+0000","last_unstale":"2024-09-21T12:37:22.129662+0000","last_undegraded":"2024-09-21T12:37:22.129662+0000","last_fullsized":"2024-09-21T12:37:22.129662+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:32:19.786928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:32:19.786928+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:03.691123+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":72,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.519802+0000","last_change":"2024-09-21T12:36:00.140779+0000","last_active":"2024-09-21T12:37:26.519802+0000","last_peered":"2024-09-21T12:37:26.519802+0000","last_clean":"2024-09-21T12:37:26.519802+0000","last_became_active":"2024-09-21T12:36:00.140477+0000","last_became_peered":"2024-09-21T12:36:00.140477+0000","last_unstale":"2024-09-21T12:37:26.519802+0000","last_undegraded":"2024-09-21T12:37:26.519802+0000","last_fullsized":"2024-09-21T12:37:26.519802+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:27.798236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:27.798236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:02.709671+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":820,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.137078+0000","last_change":"2024-09-21T12:36:00.151444+0000","last_active":"2024-09-21T12:37:27.137078+0000","last_peered":"2024-09-21T12:37:27.137078+0000","last_clean":"2024-09-21T12:37:27.137078+0000","last_became_active":"2024-09-21T12:36:00.150975+0000","last_became_peered":"2024-09-21T12:36:00.150975+0000","last_unstale":"2024-09-21T12:37:27.137078+0000","last_undegraded":"2024-09-21T12:37:27.137078+0000","last_fullsized":"2024-09-21T12:37:27.137078+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:58.720834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:32:58.720834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":794,"reported_epoch":445,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:37:26.531836+0000","last_change":"2024-09-21T12:36:01.138863+0000","last_active":"2024-09-21T12:37:26.531836+0000","last_peered":"2024-09-21T12:37:26.531836+0000","last_clean":"2024-09-21T12:37:26.531836+0000","last_became_active":"2024-09-21T12:36:01.138205+0000","last_became_peered":"2024-09-21T12:36:01.138205+0000","last_unstale":"2024-09-21T12:37:26.531836+0000","last_undegraded":"2024-09-21T12:37:26.531836+0000","last_fullsized":"2024-09-21T12:37:26.531836+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:55.936030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:55.936030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7,0],"avail_no_missing":["4","0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.10","version":"76'79","reported_seq":40,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.525356+0000","last_change":"2024-09-21T12:37:20.468086+0000","last_active":"2024-09-21T12:37:26.525356+0000","last_peered":"2024-09-21T12:37:26.525356+0000","last_clean":"2024-09-21T12:37:26.525356+0000","last_became_active":"2024-09-21T12:37:20.467540+0000","last_became_peered":"2024-09-21T12:37:20.467540+0000","last_unstale":"2024-09-21T12:37:26.525356+0000","last_undegraded":"2024-09-21T12:37:26.525356+0000","last_fullsized":"2024-09-21T12:37:26.525356+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:00.829119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:00.829119+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":369,"reported_epoch":445,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:37:26.525459+0000","last_change":"2024-09-21T12:37:21.488541+0000","last_active":"2024-09-21T12:37:26.525459+0000","last_peered":"2024-09-21T12:37:26.525459+0000","last_clean":"2024-09-21T12:37:26.525459+0000","last_became_active":"2024-09-21T12:37:21.486097+0000","last_became_peered":"2024-09-21T12:37:21.486097+0000","last_unstale":"2024-09-21T12:37:26.525459+0000","last_undegraded":"2024-09-21T12:37:26.525459+0000","last_fullsized":"2024-09-21T12:37:26.525459+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:35:54.238498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:54.238498+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7,6],"avail_no_missing":["5","6","7"],"object_location_counts":[{"shards":"5,6,7","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":872,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T12:37:22.129400+0000","last_change":"2024-09-21T12:36:00.127553+0000","last_active":"2024-09-21T12:37:22.129400+0000","last_peered":"2024-09-21T12:37:22.129400+0000","last_clean":"2024-09-21T12:37:22.129400+0000","last_became_active":"2024-09-21T12:36:00.127225+0000","last_became_peered":"2024-09-21T12:36:00.127225+0000","last_unstale":"2024-09-21T12:37:22.129400+0000","last_undegraded":"2024-09-21T12:37:22.129400+0000","last_fullsized":"2024-09-21T12:37:22.129400+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:22.225122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:22.225122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"416'15","reported_seq":385,"reported_epoch":446,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:37:27.135049+0000","last_change":"2024-09-21T12:37:12.744856+0000","last_active":"2024-09-21T12:37:27.135049+0000","last_peered":"2024-09-21T12:37:27.135049+0000","last_clean":"2024-09-21T12:37:27.135049+0000","last_became_active":"2024-09-21T12:36:01.138017+0000","last_became_peered":"2024-09-21T12:36:01.138017+0000","last_unstale":"2024-09-21T12:37:27.135049+0000","last_undegraded":"2024-09-21T12:37:27.135049+0000","last_fullsized":"2024-09-21T12:37:27.135049+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'15","last_scrub_stamp":"2024-09-21T12:37:12.744729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:37:12.744729+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:14.529067+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7,1],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":215,"reported_epoch":445,"state":"active+undersized","last_fresh":"2024-09-21T12:37:26.532446+0000","last_change":"2024-09-21T12:34:34.340859+0000","last_active":"2024-09-21T12:37:26.532446+0000","last_peered":"2024-09-21T12:37:26.532446+0000","last_clean":"2024-09-21T12:34:29.279088+0000","last_became_active":"2024-09-21T12:34:34.340859+0000","last_became_peered":"2024-09-21T12:34:34.340859+0000","last_unstale":"2024-09-21T12:37:26.532446+0000","last_undegraded":"2024-09-21T12:37:26.532446+0000","last_fullsized":"2024-09-21T12:34:34.330912+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:05.370263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:05.370263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:28.761559+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"76'79","reported_seq":639,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T12:37:22.130474+0000","last_change":"2024-09-21T12:35:17.231922+0000","last_active":"2024-09-21T12:37:22.130474+0000","last_peered":"2024-09-21T12:37:22.130474+0000","last_clean":"2024-09-21T12:37:22.130474+0000","last_became_active":"2024-09-21T12:32:02.034387+0000","last_became_peered":"2024-09-21T12:32:02.034387+0000","last_unstale":"2024-09-21T12:37:22.130474+0000","last_undegraded":"2024-09-21T12:37:22.130474+0000","last_fullsized":"2024-09-21T12:37:22.130474+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:35:17.231860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:35:17.231860+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":907,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.136763+0000","last_change":"2024-09-21T12:36:36.521884+0000","last_active":"2024-09-21T12:37:27.136763+0000","last_peered":"2024-09-21T12:37:27.136763+0000","last_clean":"2024-09-21T12:37:27.136763+0000","last_became_active":"2024-09-21T12:36:00.155033+0000","last_became_peered":"2024-09-21T12:36:00.155033+0000","last_unstale":"2024-09-21T12:37:27.136763+0000","last_undegraded":"2024-09-21T12:37:27.136763+0000","last_fullsized":"2024-09-21T12:37:27.136763+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:36:36.521810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:36.521810+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:05.336949+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":846,"reported_epoch":445,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:37:26.522090+0000","last_change":"2024-09-21T12:37:18.659752+0000","last_active":"2024-09-21T12:37:26.522090+0000","last_peered":"2024-09-21T12:37:26.522090+0000","last_clean":"2024-09-21T12:37:26.522090+0000","last_became_active":"2024-09-21T12:36:01.135527+0000","last_became_peered":"2024-09-21T12:36:01.135527+0000","last_unstale":"2024-09-21T12:37:26.522090+0000","last_undegraded":"2024-09-21T12:37:26.522090+0000","last_fullsized":"2024-09-21T12:37:26.522090+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:18.659683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:37:18.659683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:43.495166+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4,6],"avail_no_missing":["7","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":1036,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.526044+0000","last_change":"2024-09-21T12:36:29.482398+0000","last_active":"2024-09-21T12:37:26.526044+0000","last_peered":"2024-09-21T12:37:26.526044+0000","last_clean":"2024-09-21T12:37:26.526044+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:37:26.526044+0000","last_undegraded":"2024-09-21T12:37:26.526044+0000","last_fullsized":"2024-09-21T12:37:26.526044+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:36:29.482171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:29.482171+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:37.228329+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":829,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T12:37:22.128937+0000","last_change":"2024-09-21T12:36:00.149113+0000","last_active":"2024-09-21T12:37:22.128937+0000","last_peered":"2024-09-21T12:37:22.128937+0000","last_clean":"2024-09-21T12:37:22.128937+0000","last_became_active":"2024-09-21T12:36:00.148332+0000","last_became_peered":"2024-09-21T12:36:00.148332+0000","last_unstale":"2024-09-21T12:37:22.128937+0000","last_undegraded":"2024-09-21T12:37:22.128937+0000","last_fullsized":"2024-09-21T12:37:22.128937+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:26.208487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:26.208487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:56.731364+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"76'79","reported_seq":498,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.522229+0000","last_change":"2024-09-21T12:36:28.244064+0000","last_active":"2024-09-21T12:37:26.522229+0000","last_peered":"2024-09-21T12:37:26.522229+0000","last_clean":"2024-09-21T12:37:26.522229+0000","last_became_active":"2024-09-21T12:27:16.940347+0000","last_became_peered":"2024-09-21T12:27:16.940347+0000","last_unstale":"2024-09-21T12:37:26.522229+0000","last_undegraded":"2024-09-21T12:37:26.522229+0000","last_fullsized":"2024-09-21T12:37:26.522229+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:28.243938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:28.243938+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:53.995371+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":224,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.530354+0000","last_change":"2024-09-21T12:34:00.191190+0000","last_active":"2024-09-21T12:37:26.530354+0000","last_peered":"2024-09-21T12:37:26.530354+0000","last_clean":"2024-09-21T12:37:26.530354+0000","last_became_active":"2024-09-21T12:32:38.421008+0000","last_became_peered":"2024-09-21T12:32:38.421008+0000","last_unstale":"2024-09-21T12:37:26.530354+0000","last_undegraded":"2024-09-21T12:37:26.530354+0000","last_fullsized":"2024-09-21T12:37:26.530354+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:34:00.191106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:34:00.191106+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:03.660808+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":224,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.531386+0000","last_change":"2024-09-21T12:36:01.124070+0000","last_active":"2024-09-21T12:37:26.531386+0000","last_peered":"2024-09-21T12:37:26.531386+0000","last_clean":"2024-09-21T12:37:26.531386+0000","last_became_active":"2024-09-21T12:36:01.123399+0000","last_became_peered":"2024-09-21T12:36:01.123399+0000","last_unstale":"2024-09-21T12:37:26.531386+0000","last_undegraded":"2024-09-21T12:37:26.531386+0000","last_fullsized":"2024-09-21T12:37:26.531386+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:30.406915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:30.406915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:49.594071+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":830,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.521823+0000","last_change":"2024-09-21T12:36:34.472774+0000","last_active":"2024-09-21T12:37:26.521823+0000","last_peered":"2024-09-21T12:37:26.521823+0000","last_clean":"2024-09-21T12:37:26.521823+0000","last_became_active":"2024-09-21T12:31:50.040754+0000","last_became_peered":"2024-09-21T12:31:50.040754+0000","last_unstale":"2024-09-21T12:37:26.521823+0000","last_undegraded":"2024-09-21T12:37:26.521823+0000","last_fullsized":"2024-09-21T12:37:26.521823+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:34.472702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:34.472702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:47.750215+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":290,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.134956+0000","last_change":"2024-09-21T12:37:19.857159+0000","last_active":"2024-09-21T12:37:27.134956+0000","last_peered":"2024-09-21T12:37:27.134956+0000","last_clean":"2024-09-21T12:37:27.134956+0000","last_became_active":"2024-09-21T12:31:50.036150+0000","last_became_peered":"2024-09-21T12:31:50.036150+0000","last_unstale":"2024-09-21T12:37:27.134956+0000","last_undegraded":"2024-09-21T12:37:27.134956+0000","last_fullsized":"2024-09-21T12:37:27.134956+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:19.857108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:37:19.857108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:23.382352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"76'79","reported_seq":531,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.525154+0000","last_change":"2024-09-21T12:36:45.476276+0000","last_active":"2024-09-21T12:37:26.525154+0000","last_peered":"2024-09-21T12:37:26.525154+0000","last_clean":"2024-09-21T12:37:26.525154+0000","last_became_active":"2024-09-21T12:27:16.921880+0000","last_became_peered":"2024-09-21T12:27:16.921880+0000","last_unstale":"2024-09-21T12:37:26.525154+0000","last_undegraded":"2024-09-21T12:37:26.525154+0000","last_fullsized":"2024-09-21T12:37:26.525154+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:45.476087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:45.476087+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:11.672214+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":310,"reported_epoch":445,"state":"active+undersized+degraded","last_fresh":"2024-09-21T12:37:26.520440+0000","last_change":"2024-09-21T12:34:34.345279+0000","last_active":"2024-09-21T12:37:26.520440+0000","last_peered":"2024-09-21T12:37:26.520440+0000","last_clean":"2024-09-21T12:34:29.286797+0000","last_became_active":"2024-09-21T12:34:34.345279+0000","last_became_peered":"2024-09-21T12:34:34.345279+0000","last_unstale":"2024-09-21T12:37:26.520440+0000","last_undegraded":"2024-09-21T12:34:34.331120+0000","last_fullsized":"2024-09-21T12:34:34.331029+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:33:02.544885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:02.544885+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:46.518783+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":72,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.520938+0000","last_change":"2024-09-21T12:36:00.147458+0000","last_active":"2024-09-21T12:37:26.520938+0000","last_peered":"2024-09-21T12:37:26.520938+0000","last_clean":"2024-09-21T12:37:26.520938+0000","last_became_active":"2024-09-21T12:36:00.147063+0000","last_became_peered":"2024-09-21T12:36:00.147063+0000","last_unstale":"2024-09-21T12:37:26.520938+0000","last_undegraded":"2024-09-21T12:37:26.520938+0000","last_fullsized":"2024-09-21T12:37:26.520938+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:28.849200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:28.849200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:01.749100+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":814,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.135890+0000","last_change":"2024-09-21T12:34:11.934886+0000","last_active":"2024-09-21T12:37:27.135890+0000","last_peered":"2024-09-21T12:37:27.135890+0000","last_clean":"2024-09-21T12:37:27.135890+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:37:27.135890+0000","last_undegraded":"2024-09-21T12:37:27.135890+0000","last_fullsized":"2024-09-21T12:37:27.135890+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:11.934826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:11.934826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":801,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T12:37:22.127849+0000","last_change":"2024-09-21T12:35:59.959597+0000","last_active":"2024-09-21T12:37:22.127849+0000","last_peered":"2024-09-21T12:37:22.127849+0000","last_clean":"2024-09-21T12:37:22.127849+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:37:22.127849+0000","last_undegraded":"2024-09-21T12:37:22.127849+0000","last_fullsized":"2024-09-21T12:37:22.127849+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:59.959545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:59.959545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"76'79","reported_seq":307,"reported_epoch":445,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:37:26.520035+0000","last_change":"2024-09-21T12:36:43.792972+0000","last_active":"2024-09-21T12:37:26.520035+0000","last_peered":"2024-09-21T12:37:26.520035+0000","last_clean":"2024-09-21T12:37:26.520035+0000","last_became_active":"2024-09-21T12:36:01.135246+0000","last_became_peered":"2024-09-21T12:36:01.135246+0000","last_unstale":"2024-09-21T12:37:26.520035+0000","last_undegraded":"2024-09-21T12:37:26.520035+0000","last_fullsized":"2024-09-21T12:37:26.520035+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:43.792355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:43.792355+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:48.500764+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4,1],"avail_no_missing":["7","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":895,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.136750+0000","last_change":"2024-09-21T12:36:41.506309+0000","last_active":"2024-09-21T12:37:27.136750+0000","last_peered":"2024-09-21T12:37:27.136750+0000","last_clean":"2024-09-21T12:37:27.136750+0000","last_became_active":"2024-09-21T12:36:00.139682+0000","last_became_peered":"2024-09-21T12:36:00.139682+0000","last_unstale":"2024-09-21T12:37:27.136750+0000","last_undegraded":"2024-09-21T12:37:27.136750+0000","last_fullsized":"2024-09-21T12:37:27.136750+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:36:41.506231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:41.506231+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:10.569673+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":859,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.524723+0000","last_change":"2024-09-21T12:36:34.526618+0000","last_active":"2024-09-21T12:37:26.524723+0000","last_peered":"2024-09-21T12:37:26.524723+0000","last_clean":"2024-09-21T12:37:26.524723+0000","last_became_active":"2024-09-21T12:36:00.151584+0000","last_became_peered":"2024-09-21T12:36:00.151584+0000","last_unstale":"2024-09-21T12:37:26.524723+0000","last_undegraded":"2024-09-21T12:37:26.524723+0000","last_fullsized":"2024-09-21T12:37:26.524723+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:34.526551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:36:34.526551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:39.862324+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":72,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.519879+0000","last_change":"2024-09-21T12:36:00.145015+0000","last_active":"2024-09-21T12:37:26.519879+0000","last_peered":"2024-09-21T12:37:26.519879+0000","last_clean":"2024-09-21T12:37:26.519879+0000","last_became_active":"2024-09-21T12:36:00.144521+0000","last_became_peered":"2024-09-21T12:36:00.144521+0000","last_unstale":"2024-09-21T12:37:26.519879+0000","last_undegraded":"2024-09-21T12:37:26.519879+0000","last_fullsized":"2024-09-21T12:37:26.519879+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:47.316065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:47.316065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:48.738968+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":810,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.136650+0000","last_change":"2024-09-21T12:35:51.628370+0000","last_active":"2024-09-21T12:37:27.136650+0000","last_peered":"2024-09-21T12:37:27.136650+0000","last_clean":"2024-09-21T12:37:27.136650+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:37:27.136650+0000","last_undegraded":"2024-09-21T12:37:27.136650+0000","last_fullsized":"2024-09-21T12:37:27.136650+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:51.628326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:51.628326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"76'79","reported_seq":72,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.520585+0000","last_change":"2024-09-21T12:36:00.151654+0000","last_active":"2024-09-21T12:37:26.520585+0000","last_peered":"2024-09-21T12:37:26.520585+0000","last_clean":"2024-09-21T12:37:26.520585+0000","last_became_active":"2024-09-21T12:36:00.151043+0000","last_became_peered":"2024-09-21T12:36:00.151043+0000","last_unstale":"2024-09-21T12:37:26.520585+0000","last_undegraded":"2024-09-21T12:37:26.520585+0000","last_fullsized":"2024-09-21T12:37:26.520585+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:35:51.976777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:35:51.976777+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":830,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.135748+0000","last_change":"2024-09-21T12:35:14.789148+0000","last_active":"2024-09-21T12:37:27.135748+0000","last_peered":"2024-09-21T12:37:27.135748+0000","last_clean":"2024-09-21T12:37:27.135748+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:37:27.135748+0000","last_undegraded":"2024-09-21T12:37:27.135748+0000","last_fullsized":"2024-09-21T12:37:27.135748+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'8","last_scrub_stamp":"2024-09-21T12:35:14.789070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:14.789070+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":835,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-21T12:37:22.129348+0000","last_change":"2024-09-21T12:35:27.237430+0000","last_active":"2024-09-21T12:37:22.129348+0000","last_peered":"2024-09-21T12:37:22.129348+0000","last_clean":"2024-09-21T12:37:22.129348+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:37:22.129348+0000","last_undegraded":"2024-09-21T12:37:22.129348+0000","last_fullsized":"2024-09-21T12:37:22.129348+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:35:27.236980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:27.236980+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":855,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.523948+0000","last_change":"2024-09-21T12:36:00.135914+0000","last_active":"2024-09-21T12:37:26.523948+0000","last_peered":"2024-09-21T12:37:26.523948+0000","last_clean":"2024-09-21T12:37:26.523948+0000","last_became_active":"2024-09-21T12:36:00.135537+0000","last_became_peered":"2024-09-21T12:36:00.135537+0000","last_unstale":"2024-09-21T12:37:26.523948+0000","last_undegraded":"2024-09-21T12:37:26.523948+0000","last_fullsized":"2024-09-21T12:37:26.523948+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:40.861493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:40.861493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":834,"reported_epoch":445,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:37:26.519550+0000","last_change":"2024-09-21T12:36:38.838788+0000","last_active":"2024-09-21T12:37:26.519550+0000","last_peered":"2024-09-21T12:37:26.519550+0000","last_clean":"2024-09-21T12:37:26.519550+0000","last_became_active":"2024-09-21T12:36:01.134484+0000","last_became_peered":"2024-09-21T12:36:01.134484+0000","last_unstale":"2024-09-21T12:37:26.519550+0000","last_undegraded":"2024-09-21T12:37:26.519550+0000","last_fullsized":"2024-09-21T12:37:26.519550+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:38.838688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:38.838688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:55.540359+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4,3],"avail_no_missing":["7","3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"76'79","reported_seq":225,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.530229+0000","last_change":"2024-09-21T12:36:00.137642+0000","last_active":"2024-09-21T12:37:26.530229+0000","last_peered":"2024-09-21T12:37:26.530229+0000","last_clean":"2024-09-21T12:37:26.530229+0000","last_became_active":"2024-09-21T12:36:00.137290+0000","last_became_peered":"2024-09-21T12:36:00.137290+0000","last_unstale":"2024-09-21T12:37:26.530229+0000","last_undegraded":"2024-09-21T12:37:26.530229+0000","last_fullsized":"2024-09-21T12:37:26.530229+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:43.304606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:43.304606+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":836,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.519882+0000","last_change":"2024-09-21T12:35:48.922444+0000","last_active":"2024-09-21T12:37:26.519882+0000","last_peered":"2024-09-21T12:37:26.519882+0000","last_clean":"2024-09-21T12:37:26.519882+0000","last_became_active":"2024-09-21T12:32:38.426822+0000","last_became_peered":"2024-09-21T12:32:38.426822+0000","last_unstale":"2024-09-21T12:37:26.519882+0000","last_undegraded":"2024-09-21T12:37:26.519882+0000","last_fullsized":"2024-09-21T12:37:26.519882+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:48.922389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:48.922389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:38:06.753792+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":836,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-09-21T12:37:27.136725+0000","last_change":"2024-09-21T12:35:30.730640+0000","last_active":"2024-09-21T12:37:27.136725+0000","last_peered":"2024-09-21T12:37:27.136725+0000","last_clean":"2024-09-21T12:37:27.136725+0000","last_became_active":"2024-09-21T12:31:39.052486+0000","last_became_peered":"2024-09-21T12:31:39.052486+0000","last_unstale":"2024-09-21T12:37:27.136725+0000","last_undegraded":"2024-09-21T12:37:27.136725+0000","last_fullsized":"2024-09-21T12:37:27.136725+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:30.730574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:30.730574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":849,"reported_epoch":445,"state":"active+clean","last_fresh":"2024-09-21T12:37:26.524145+0000","last_change":"2024-09-21T12:34:44.919918+0000","last_active":"2024-09-21T12:37:26.524145+0000","last_peered":"2024-09-21T12:37:26.524145+0000","last_clean":"2024-09-21T12:37:26.524145+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:37:26.524145+0000","last_undegraded":"2024-09-21T12:37:26.524145+0000","last_fullsized":"2024-09-21T12:37:26.524145+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:34:44.919796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:34:44.919796+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":854,"reported_epoch":444,"state":"active+clean+remapped","last_fresh":"2024-09-21T12:37:22.129040+0000","last_change":"2024-09-21T12:36:28.911452+0000","last_active":"2024-09-21T12:37:22.129040+0000","last_peered":"2024-09-21T12:37:22.129040+0000","last_clean":"2024-09-21T12:37:22.129040+0000","last_became_active":"2024-09-21T12:36:01.130870+0000","last_became_peered":"2024-09-21T12:36:01.130870+0000","last_unstale":"2024-09-21T12:37:22.129040+0000","last_undegraded":"2024-09-21T12:37:22.129040+0000","last_fullsized":"2024-09-21T12:37:22.129040+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:28.911328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:28.911328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:37:34.674874+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7,1],"avail_no_missing":["0","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":8696627224,"num_objects":132701,"num_object_clones":0,"num_object_copies":398103,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33344,"num_objects_misplaced":16660,"num_objects_unfound":0,"num_objects_dirty":132701,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143527,"num_write_kb":8839233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4193,"num_bytes_recovered":274792448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24045887488,"data_stored":24045879344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65804,"ondisk_log_size":65804,"up":42,"acting":44,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":83,"acting":90,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106,"ondisk_log_size":106,"up":87,"acting":92,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":54,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1105,"num_read_kb":862,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":472,"num_bytes_recovered":9372,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":11216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":687,"ondisk_log_size":687,"up":87,"acting":88,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":598,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":88,"acting":90,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":93,"num_read_kb":88,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2048000,"data_stored":2033792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2449,"ondisk_log_size":2449,"up":86,"acting":89,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234194,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4015100,"kb_used_data":3199336,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":89730052,"statfs":{"total":95995035648,"available":91883573248,"internally_reserved":0,"allocated":3276120064,"data_stored":3275509720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":835123658},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463130,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5063152,"kb_used_data":4233692,"kb_used_omap":1,"kb_used_meta":829438,"kb_avail":88682000,"statfs":{"total":95995035648,"available":90810368000,"internally_reserved":0,"allocated":4335300608,"data_stored":4334701493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":849344966},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724765,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2843244,"kb_used_data":2154012,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":90901908,"statfs":{"total":95995035648,"available":93083553792,"internally_reserved":0,"allocated":2205708288,"data_stored":2205102254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953701,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2517572,"kb_used_data":2147632,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":91227580,"statfs":{"total":95995035648,"available":93417041920,"internally_reserved":0,"allocated":2199175168,"data_stored":2198555632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149931,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4561808,"kb_used_data":3714300,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":89183344,"statfs":{"total":95995035648,"available":91323744256,"internally_reserved":0,"allocated":3803443200,"data_stored":3802813360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":867826119},"hb_peers":[0,1,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411534,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2808848,"kb_used_data":2272320,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":90936304,"statfs":{"total":95995035648,"available":93118775296,"internally_reserved":0,"allocated":2326855680,"data_stored":2326574609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575095,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5775948,"kb_used_data":4870140,"kb_used_omap":1,"kb_used_meta":905790,"kb_avail":87969204,"statfs":{"total":95995035648,"available":90080464896,"internally_reserved":0,"allocated":4987023360,"data_stored":4986384005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":927529413},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738620,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4146032,"kb_used_data":3280480,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":89599120,"statfs":{"total":95995035648,"available":91749498880,"internally_reserved":0,"allocated":3359211520,"data_stored":3358600625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":886307269},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3383033856,"data_stored":3383033856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4928700416,"data_stored":4928700416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3762487296,"data_stored":3762487296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2190217216,"data_stored":2190213144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2189295616,"data_stored":2189295616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4332589056,"data_stored":4332584984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3259564032,"data_stored":3259564032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:37:32.107 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pool set unique_pool_4 pgp_num 16 2024-09-21T12:37:32.180 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:32.205 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:32.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:32.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:32.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:31 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1147513365' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:37:32.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:31 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1147513365' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:37:32.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:31 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1147513365' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:37:32.307 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:32.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:32.345 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:32.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:32.446 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:32.481 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:32.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:32.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:32.582 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:32.615 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:32.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:32.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:32.717 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:32.752 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:32.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:32.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:32.794 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T12:37:32.795 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T12:37:32.854 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:32.855 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:37:32.871 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:37:32.881 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:32.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:32.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:32.941 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T12:37:32.967 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:37:32 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:37:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:37:32.983 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:33.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:33.023 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:33.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:33.124 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:33.160 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:33.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:33.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:33.224 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:32 smithi114 ceph-mon[40867]: pgmap v880: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 8.0 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 767 B/s rd, 11 MiB/s wr, 1.03k op/s; 32418/394296 objects degraded (8.222%); 16668/394296 objects misplaced (4.227%) 2024-09-21T12:37:33.224 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:32 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 32418/394296 objects degraded (8.222%), 13 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:33.224 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:32 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:37:33.224 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:32 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:37:33.224 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:32 smithi114 ceph-mon[30324]: pgmap v880: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 8.0 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 767 B/s rd, 11 MiB/s wr, 1.03k op/s; 32418/394296 objects degraded (8.222%); 16668/394296 objects misplaced (4.227%) 2024-09-21T12:37:33.224 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:32 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 32418/394296 objects degraded (8.222%), 13 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:33.224 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:32 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:37:33.224 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:32 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:37:33.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:32 smithi183 ceph-mon[33277]: pgmap v880: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 8.0 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 767 B/s rd, 11 MiB/s wr, 1.03k op/s; 32418/394296 objects degraded (8.222%); 16668/394296 objects misplaced (4.227%) 2024-09-21T12:37:33.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:32 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 32418/394296 objects degraded (8.222%), 13 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:33.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:32 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:37:33.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:32 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:37:33.262 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:33.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:33.293 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:33.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:33.345 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:37:33.395 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:33.430 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:33.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:33.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:33.531 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:33.560 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:33.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:33.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:33.661 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:33.697 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:33.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:33.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:33.798 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:33.826 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:33.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:33.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:33.928 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:33.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:33.965 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:33.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:33.975 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:37:33.975 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:37:34.067 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:34.098 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:34.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:34.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:34.200 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:34.231 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:34.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:34.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:34.332 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:34.363 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:34.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:34.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:34.465 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:34.500 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:34.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:34.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:34.602 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:34.635 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:34.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:34.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:34.711 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T12:37:34.737 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:34.796 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:37:34.803 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:34.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:34.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:34.905 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:34.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:34.931 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:34.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:34.981 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-21T12:37:34.983 INFO:teuthology.orchestra.run.smithi114.stderr:set pool 11 pgp_num to 16 2024-09-21T12:37:35.031 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:35.060 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:35.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:35.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:35.161 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:35.208 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:35.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:35.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:35.253 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:34 smithi114 ceph-mon[30324]: pgmap v881: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 7.6 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.4 KiB/s rd, 11 MiB/s wr, 1.88k op/s; 30966/373788 objects degraded (8.284%); 15695/373788 objects misplaced (4.199%) 2024-09-21T12:37:35.253 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:34 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1176337499' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:37:35.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:34 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:37:35.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:34 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/448942903' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T12:37:35.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:34 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T12:37:35.254 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:34 smithi114 ceph-mon[40867]: pgmap v881: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 7.6 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.4 KiB/s rd, 11 MiB/s wr, 1.88k op/s; 30966/373788 objects degraded (8.284%); 15695/373788 objects misplaced (4.199%) 2024-09-21T12:37:35.254 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:34 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1176337499' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:37:35.254 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:34 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:37:35.254 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:34 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/448942903' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T12:37:35.254 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:34 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T12:37:35.310 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:35.342 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:35.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:35.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:35.443 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:35.474 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:35.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:35.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:35.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:34 smithi183 ceph-mon[33277]: pgmap v881: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 7.6 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.4 KiB/s rd, 11 MiB/s wr, 1.88k op/s; 30966/373788 objects degraded (8.284%); 15695/373788 objects misplaced (4.199%) 2024-09-21T12:37:35.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:34 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1176337499' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:37:35.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:34 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:37:35.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:34 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/448942903' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T12:37:35.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:34 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T12:37:35.576 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:35.612 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:35.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:35.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:35.714 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:35.748 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:35.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:35.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:35.850 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:35.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:35.885 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:35.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:35.986 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:36.018 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:36.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:36.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:36.119 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:36.162 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:36.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:36.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:36.264 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:36.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:36 smithi114 ceph-mon[30324]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:37:36.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:36 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:37:36.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:36 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-09-21T12:37:36.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:36 smithi114 ceph-mon[30324]: osdmap e447: 8 total, 7 up, 8 in 2024-09-21T12:37:36.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:36 smithi114 ceph-mon[40867]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:37:36.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:36 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:37:36.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:36 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-09-21T12:37:36.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:36 smithi114 ceph-mon[40867]: osdmap e447: 8 total, 7 up, 8 in 2024-09-21T12:37:36.295 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:36.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:36.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:36.397 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:36.425 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:36.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:36.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:36.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:36 smithi183 ceph-mon[33277]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:37:36.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:36 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:37:36.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:36 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-09-21T12:37:36.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:36 smithi183 ceph-mon[33277]: osdmap e447: 8 total, 7 up, 8 in 2024-09-21T12:37:36.527 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:36.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:36.563 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:36.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:36.664 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:36.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:36.723 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:36.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:36.824 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:36.859 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:36.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:36.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:36.961 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:36.986 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:36.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:36.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:37.088 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:37.121 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:37.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:37.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:37.222 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:37.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:37 smithi183 ceph-mon[33277]: pgmap v883: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 7.6 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 583 B/s rd, 2.3 MiB/s wr, 1.27k op/s; 30966/373788 objects degraded (8.284%); 15695/373788 objects misplaced (4.199%) 2024-09-21T12:37:37.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:37.257 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:37.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:37.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:37 smithi114 ceph-mon[30324]: pgmap v883: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 7.6 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 583 B/s rd, 2.3 MiB/s wr, 1.27k op/s; 30966/373788 objects degraded (8.284%); 15695/373788 objects misplaced (4.199%) 2024-09-21T12:37:37.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:37 smithi114 ceph-mon[40867]: pgmap v883: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 7.6 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 583 B/s rd, 2.3 MiB/s wr, 1.27k op/s; 30966/373788 objects degraded (8.284%); 15695/373788 objects misplaced (4.199%) 2024-09-21T12:37:37.358 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:37.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:37.393 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:37.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:37.495 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:37.524 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:37.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:37.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:37.626 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:37.646 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:37.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:37.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:37.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:37.781 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:37.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:37.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:37.883 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:37.911 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:37.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:37.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:37.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:37:37 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:37:38.013 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:38.048 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:38.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:38.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:38.150 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:38.174 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:38.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:38.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:38.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:38 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 30966/373788 objects degraded (8.284%), 13 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:38.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:38 smithi114 ceph-mon[30324]: osdmap e448: 8 total, 7 up, 8 in 2024-09-21T12:37:38.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:38 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 30966/373788 objects degraded (8.284%), 13 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:38.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:38 smithi114 ceph-mon[40867]: osdmap e448: 8 total, 7 up, 8 in 2024-09-21T12:37:38.276 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:38.309 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:38.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:38.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:38.412 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:38.450 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:38.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:38.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:38.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:38 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 30966/373788 objects degraded (8.284%), 13 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:38.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:38 smithi183 ceph-mon[33277]: osdmap e448: 8 total, 7 up, 8 in 2024-09-21T12:37:38.552 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:38.585 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:38.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:38.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:38.686 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:38.724 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:38.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:38.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:38.826 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:38.855 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:38.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:38.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:38.956 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:38.988 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:38.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:38.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:39.090 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:39.124 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:39.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:39.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:39.226 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:39.261 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:39.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:39.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:39.363 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:39.394 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:39.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:39.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:39.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:39 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:37:39.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:39 smithi183 ceph-mon[33277]: pgmap v885: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 7.3 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s; 29941/360225 objects degraded (8.312%); 14695/360225 objects misplaced (4.079%) 2024-09-21T12:37:39.496 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:39.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:39 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:37:39.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:39 smithi114 ceph-mon[40867]: pgmap v885: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 7.3 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s; 29941/360225 objects degraded (8.312%); 14695/360225 objects misplaced (4.079%) 2024-09-21T12:37:39.515 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:39 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:37:39.516 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:39 smithi114 ceph-mon[30324]: pgmap v885: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 7.3 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s; 29941/360225 objects degraded (8.312%); 14695/360225 objects misplaced (4.079%) 2024-09-21T12:37:39.526 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:39.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:39.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:39.629 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:39.660 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:39.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:39.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:39.762 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:39.794 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:39.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:39.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:39.896 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:39.928 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:39.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:39.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:40.030 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:40.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:40.072 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:40.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:40.173 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:40.205 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:40.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:40.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:40.307 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:40.341 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:40.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:40.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:40.442 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:40.479 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:40.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:40.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:40.580 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:40.615 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:40.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:40.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:40.717 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:40.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:40.750 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:40.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:40.851 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:40.880 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:40.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:40.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:40.982 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:41.015 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:41.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:41.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:41.117 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:41.152 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:41.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:41.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:41.255 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:41.286 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:41.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:41.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:41.351 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:37:41.387 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:41.420 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:41.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:41.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:41.422 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:37:41.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:41 smithi183 ceph-mon[33277]: pgmap v886: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 7.1 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.85k op/s; 28939/349974 objects degraded (8.269%); 14695/349974 objects misplaced (4.199%) 2024-09-21T12:37:41.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:41 smithi114 ceph-mon[30324]: pgmap v886: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 7.1 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.85k op/s; 28939/349974 objects degraded (8.269%); 14695/349974 objects misplaced (4.199%) 2024-09-21T12:37:41.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:41 smithi114 ceph-mon[40867]: pgmap v886: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 7.1 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.85k op/s; 28939/349974 objects degraded (8.269%); 14695/349974 objects misplaced (4.199%) 2024-09-21T12:37:41.522 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:41.560 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:41.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:41.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:41.663 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:41.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:41.695 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:41.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:41.796 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:41.828 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:41.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:41.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:41.930 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:41.965 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:41.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:41.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:42.067 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:42.103 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:42.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:42.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:42.205 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:42.233 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:42.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:42.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:42.330 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:37:42.330 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:37:42.331 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:42 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 28939/349974 objects degraded (8.269%), 13 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:42.331 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:42 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 28939/349974 objects degraded (8.269%), 13 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:42.335 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:42.360 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:37:42.360 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:37:42.361 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T12:37:42.366 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:42.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:42.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:42.394 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:37:42.394 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:37:42.439 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:37:42.440 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:37:42.468 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:42.480 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:37:42.480 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:37:42.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:42 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 28939/349974 objects degraded (8.269%), 13 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:42.501 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:42.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:42.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:42.516 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:37:42.517 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:37:42.549 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:37:42.550 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-21T12:37:42.550 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:37:42.550 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.26147.0:167764 11.6 11:6ecfa687:::benchmark_data_smithi114_187622_object29643:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e448)", 2024-09-21T12:37:42.550 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:37:42.539048+0000", 2024-09-21T12:37:42.550 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0086656939999999998, 2024-09-21T12:37:42.550 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0086955160000000004, 2024-09-21T12:37:42.550 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:37:42.550 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:37:42.550 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:37:42.550 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:37:42.550 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.26147", 2024-09-21T12:37:42.551 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:37:42.551 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 167764 2024-09-21T12:37:42.551 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:37:42.551 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:37:42.551 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:37:42.551 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:37:42.551 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:37:42.539048+0000", 2024-09-21T12:37:42.551 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:37:42.551 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:37:42.551 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:37:42.551 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:37:42.551 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:37:42.539048+0000", 2024-09-21T12:37:42.551 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:37:42.551 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:37:42.551 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:37:42.552 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:37:42.552 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:37:42.539050+0000", 2024-09-21T12:37:42.552 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6789999999999999e-06 2024-09-21T12:37:42.552 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:37:42.552 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:37:42.552 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:37:42.552 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:37:42.539056+0000", 2024-09-21T12:37:42.552 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.993e-06 2024-09-21T12:37:42.552 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:37:42.552 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:37:42.552 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:37:42.552 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:37:42.539078+0000", 2024-09-21T12:37:42.552 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2869000000000001e-05 2024-09-21T12:37:42.552 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:37:42.553 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:37:42.553 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:37:42.553 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:37:42.539087+0000", 2024-09-21T12:37:42.553 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9299999999999992e-06 2024-09-21T12:37:42.553 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:37:42.553 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:37:42.553 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:37:42.553 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:37:42.539154+0000", 2024-09-21T12:37:42.553 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6489999999999995e-05 2024-09-21T12:37:42.553 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:37:42.553 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:37:42.553 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:37:42.553 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:37:42.539299+0000", 2024-09-21T12:37:42.553 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014559800000000001 2024-09-21T12:37:42.554 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:37:42.554 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:37:42.554 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1", 2024-09-21T12:37:42.554 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:37:42.539697+0000", 2024-09-21T12:37:42.554 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00039791300000000001 2024-09-21T12:37:42.554 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:37:42.554 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:37:42.554 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:37:42.554 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:37:42.542779+0000", 2024-09-21T12:37:42.554 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0030819279999999998 2024-09-21T12:37:42.554 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:37:42.554 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:37:42.554 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:37:42.555 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:37:42.555 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:37:42.555 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.26147.0:167769 11.2 e448/405 11:4a437339:::benchmark_data_smithi114_187622_object29648:head v 448'10456, mlcod=448'10454)", 2024-09-21T12:37:42.555 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:37:42.545141+0000", 2024-09-21T12:37:42.555 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0025725930000000002, 2024-09-21T12:37:42.555 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0026756850000000001, 2024-09-21T12:37:42.555 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:37:42.555 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:37:42.555 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:37:42.555 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:37:42.555 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:37:42.555 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:37:42.555 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:37:42.545141+0000", 2024-09-21T12:37:42.555 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:37:42.555 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:37:42.556 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:37:42.556 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:37:42.556 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:37:42.545141+0000", 2024-09-21T12:37:42.556 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:37:42.556 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:37:42.556 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:37:42.556 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:37:42.556 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:37:42.545142+0000", 2024-09-21T12:37:42.556 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.139e-06 2024-09-21T12:37:42.556 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:37:42.556 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:37:42.556 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:37:42.556 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:37:42.545148+0000", 2024-09-21T12:37:42.556 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1460000000000001e-06 2024-09-21T12:37:42.557 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:37:42.557 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:37:42.557 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:37:42.557 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:37:42.545164+0000", 2024-09-21T12:37:42.557 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5539000000000001e-05 2024-09-21T12:37:42.557 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:37:42.557 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:37:42.557 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:37:42.557 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:37:42.545171+0000", 2024-09-21T12:37:42.557 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0149999999999998e-06 2024-09-21T12:37:42.557 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:37:42.557 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:37:42.557 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:37:42.557 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:37:42.545222+0000", 2024-09-21T12:37:42.558 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1181999999999999e-05 2024-09-21T12:37:42.558 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:37:42.558 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:37:42.558 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:37:42.558 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:37:42.545241+0000", 2024-09-21T12:37:42.558 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.906e-05 2024-09-21T12:37:42.558 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:37:42.558 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:37:42.558 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:37:42.558 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:37:42.558 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:37:42.558 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.26147.0:167765 11.9 e448/405 11:94cb4223:::benchmark_data_smithi114_187622_object29644:head v 448'10526, mlcod=448'10524)", 2024-09-21T12:37:42.558 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:37:42.545191+0000", 2024-09-21T12:37:42.558 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0025225320000000001, 2024-09-21T12:37:42.559 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00267593, 2024-09-21T12:37:42.559 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:37:42.559 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:37:42.559 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:37:42.559 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:37:42.559 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:37:42.559 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:37:42.559 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:37:42.545191+0000", 2024-09-21T12:37:42.559 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:37:42.559 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:37:42.559 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:37:42.559 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:37:42.559 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:37:42.545191+0000", 2024-09-21T12:37:42.559 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:37:42.560 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:37:42.560 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:37:42.560 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:37:42.560 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:37:42.545191+0000", 2024-09-21T12:37:42.560 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.39e-07 2024-09-21T12:37:42.560 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:37:42.560 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:37:42.560 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:37:42.560 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:37:42.545195+0000", 2024-09-21T12:37:42.560 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.3550000000000001e-06 2024-09-21T12:37:42.560 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:37:42.560 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:37:42.560 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:37:42.560 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:37:42.545200+0000", 2024-09-21T12:37:42.561 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4190000000000001e-06 2024-09-21T12:37:42.561 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:37:42.561 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:37:42.561 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:37:42.561 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:37:42.545204+0000", 2024-09-21T12:37:42.561 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3490000000000002e-06 2024-09-21T12:37:42.561 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:37:42.561 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:37:42.561 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:37:42.561 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:37:42.545336+0000", 2024-09-21T12:37:42.561 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013183800000000001 2024-09-21T12:37:42.561 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:37:42.561 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:37:42.561 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:37:42.562 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:37:42.545353+0000", 2024-09-21T12:37:42.562 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7037999999999998e-05 2024-09-21T12:37:42.562 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:37:42.562 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:37:42.562 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:37:42.562 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:37:42.562 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-21T12:37:42.562 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 3 2024-09-21T12:37:42.562 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:37:42.562 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:37:42.563 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:37:42.602 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:42.632 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:42.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:42.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:42.734 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:42.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:42.766 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:42.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:42.774 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:37:42 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:37:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:37:42.867 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:42.898 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:42.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:42.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:43.000 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:43.030 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:43.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:43.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:43.132 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:43.169 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:43.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:43.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:43.271 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:43.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:43.308 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:43.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:43.325 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T12:37:43.409 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:43.446 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:43.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:43.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:43.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:43 smithi183 ceph-mon[33277]: pgmap v887: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.12k op/s; 28398/346911 objects degraded (8.186%); 14695/346911 objects misplaced (4.236%) 2024-09-21T12:37:43.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:43 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3946659251' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:37:43.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:43 smithi114 ceph-mon[30324]: pgmap v887: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.12k op/s; 28398/346911 objects degraded (8.186%); 14695/346911 objects misplaced (4.236%) 2024-09-21T12:37:43.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:43 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3946659251' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:37:43.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:43 smithi114 ceph-mon[40867]: pgmap v887: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.12k op/s; 28398/346911 objects degraded (8.186%); 14695/346911 objects misplaced (4.236%) 2024-09-21T12:37:43.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:43 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3946659251' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:37:43.546 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 4, 7] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-09-21T12:37:43.546 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-09-21T12:37:43.546 INFO:tasks.cephadm.osd.2:Restarting osd.2 (starting--it wasn't running)... 2024-09-21T12:37:43.547 DEBUG:teuthology.orchestra.run.smithi114:osd.2> sudo journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@osd.2.service 2024-09-21T12:37:43.547 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:43.550 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl start ceph-e2ad975c-7812-11ef-baf6-efdc52797490@osd.2 2024-09-21T12:37:43.594 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:43.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:43.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:43.657 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T12:37:43.696 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:43.750 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:43.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:43.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:43.853 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:43.882 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:43.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:43.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:43.984 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:44.020 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:44.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:44.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:44.024 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:43 smithi114 systemd[1]: Starting Ceph osd.2 for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:37:44.123 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:44.159 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:44.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:44.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:44.261 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:44.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:44.293 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:44.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:44.394 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:44.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:44.446 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:44.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:44.547 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:44.588 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:44.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:44.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:44.690 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:44.713 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:44.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:44.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:44.814 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:44.850 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:44.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:44.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:44.902 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T12:37:44.903 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T12:37:44.951 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:45.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:45.016 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:45.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:45.034 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:37:45.069 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:37:45.118 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:45.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:45.167 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:45.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:45.268 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:45.308 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:45.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:45.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:45.410 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:45.434 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:45 smithi114 podman[199617]: 2024-09-21 12:37:45.129404078 +0000 UTC m=+0.046591546 image pull 8f4d013db793795fef09ff5edf96e6b42ba8cd2c9158b2f0b7a5d7c1e657c063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472 2024-09-21T12:37:45.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:45 smithi114 ceph-mon[40867]: pgmap v888: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.76k op/s; 26986/326526 objects degraded (8.265%); 13721/326526 objects misplaced (4.202%) 2024-09-21T12:37:45.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:45 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:37:45.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:45 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:37:45.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:45 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:37:45.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:45 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:37:45.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:45 smithi114 ceph-mon[30324]: pgmap v888: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.76k op/s; 26986/326526 objects degraded (8.265%); 13721/326526 objects misplaced (4.202%) 2024-09-21T12:37:45.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:45 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:37:45.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:45 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:37:45.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:45 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:37:45.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:45 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:37:45.448 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:45.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:45.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:45.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:45 smithi183 ceph-mon[33277]: pgmap v888: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.76k op/s; 26986/326526 objects degraded (8.265%); 13721/326526 objects misplaced (4.202%) 2024-09-21T12:37:45.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:45 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:37:45.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:45 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:37:45.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:45 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:37:45.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:45 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:37:45.550 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:45.587 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:45.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:45.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:45.688 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:45.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:45.725 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:45.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:45.774 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:45 smithi114 podman[199617]: 2024-09-21 12:37:45.542291825 +0000 UTC m=+0.459479266 container create 6749bc589c8bf7e7bf03077e8a1560d0eaae176fed989b5588a76c22b12f7bad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, GIT_CLEAN=True) 2024-09-21T12:37:45.827 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:45.857 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:45.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:45.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:45.959 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:45.994 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:45.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:45.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:46.096 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:46.100 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:37:46.101 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:37:46.101 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:37:46.101 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:37:46.101 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:37:46.137 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:46.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:46.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:46.239 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:46.271 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:46.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:46.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:46.373 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:46.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:46.408 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:46.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:46.438 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:46 smithi114 ceph-mon[30324]: pgmap v889: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.71k op/s; 26986/326526 objects degraded (8.265%); 13721/326526 objects misplaced (4.202%) 2024-09-21T12:37:46.438 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:46 smithi114 ceph-mon[30324]: pgmap v890: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.40k op/s; 26986/326526 objects degraded (8.265%); 13721/326526 objects misplaced (4.202%) 2024-09-21T12:37:46.438 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:46 smithi114 ceph-mon[40867]: pgmap v889: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.71k op/s; 26986/326526 objects degraded (8.265%); 13721/326526 objects misplaced (4.202%) 2024-09-21T12:37:46.439 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:46 smithi114 ceph-mon[40867]: pgmap v890: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.40k op/s; 26986/326526 objects degraded (8.265%); 13721/326526 objects misplaced (4.202%) 2024-09-21T12:37:46.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:46 smithi183 ceph-mon[33277]: pgmap v889: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.71k op/s; 26986/326526 objects degraded (8.265%); 13721/326526 objects misplaced (4.202%) 2024-09-21T12:37:46.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:46 smithi183 ceph-mon[33277]: pgmap v890: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.40k op/s; 26986/326526 objects degraded (8.265%); 13721/326526 objects misplaced (4.202%) 2024-09-21T12:37:46.510 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:46.541 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:46.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:46.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:46.643 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:46.674 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:46.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:46.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:46.776 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:46.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:46.815 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:46.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:46.816 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T12:37:46.904 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:37:46.916 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:46.943 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:46.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:46.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:47.024 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:46 smithi114 podman[199617]: 2024-09-21 12:37:46.84761161 +0000 UTC m=+1.764799072 container init 6749bc589c8bf7e7bf03077e8a1560d0eaae176fed989b5588a76c22b12f7bad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2-activate, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.vendor=CentOS) 2024-09-21T12:37:47.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:46 smithi114 podman[199617]: 2024-09-21 12:37:46.861159929 +0000 UTC m=+1.778347372 container start 6749bc589c8bf7e7bf03077e8a1560d0eaae176fed989b5588a76c22b12f7bad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2-activate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True) 2024-09-21T12:37:47.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:46 smithi114 podman[199617]: 2024-09-21 12:37:46.982422707 +0000 UTC m=+1.899610158 container attach 6749bc589c8bf7e7bf03077e8a1560d0eaae176fed989b5588a76c22b12f7bad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2-activate, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.vendor=CentOS) 2024-09-21T12:37:47.045 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:47.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:47.087 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:47.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:47.188 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:47.242 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:47.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:47.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:47.344 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:47.378 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:47.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:47.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:47.480 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:47.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:47 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 26986/326526 objects degraded (8.265%), 13 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:47.518 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:47.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:47.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:47.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:47 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 26986/326526 objects degraded (8.265%), 13 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:47.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:47 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 26986/326526 objects degraded (8.265%), 13 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:47.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:47 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2-activate[199755]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T12:37:47.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:47 smithi114 bash[199617]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T12:37:47.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:47 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2-activate[199755]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T12:37:47.525 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:47 smithi114 bash[199617]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T12:37:47.620 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:47.652 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:47.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:47.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:47.754 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:47.785 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:47.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:47.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:47.887 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:47.922 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:47.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:47.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:47.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:37:47 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:37:47.999 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:37:48.023 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:48.077 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:37:48.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:48.082 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:48.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:48.183 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:48.202 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-21T12:37:48.235 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:48.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:48.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:48.336 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:48.366 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:48.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:48.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:48.468 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:48.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:48 smithi183 ceph-mon[33277]: pgmap v891: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.69k op/s; 26056/313524 objects degraded (8.311%); 12727/313524 objects misplaced (4.059%) 2024-09-21T12:37:48.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:48 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:37:48.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:48 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2384377716' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:37:48.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:48 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:37:48.508 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:48.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:48.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:48.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:48 smithi114 ceph-mon[40867]: pgmap v891: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.69k op/s; 26056/313524 objects degraded (8.311%); 12727/313524 objects misplaced (4.059%) 2024-09-21T12:37:48.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:48 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:37:48.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:48 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2384377716' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:37:48.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:48 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:37:48.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:48 smithi114 ceph-mon[30324]: pgmap v891: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.69k op/s; 26056/313524 objects degraded (8.311%); 12727/313524 objects misplaced (4.059%) 2024-09-21T12:37:48.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:48 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:37:48.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:48 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2384377716' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:37:48.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:48 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:37:48.610 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:48.642 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:48.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:48.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:48.744 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:48.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:48.777 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:48.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:48.878 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:48.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:48.912 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:48.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:49.013 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:49.047 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:49.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:49.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:49.149 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:49.176 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:49.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:49.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:49.278 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:49.316 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:49.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:49.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:49.413 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:49 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:37:49.413 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:49 smithi114 ceph-mon[30324]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:37:49.413 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:49 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:37:49.413 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:49 smithi114 ceph-mon[30324]: osdmap e449: 8 total, 7 up, 8 in 2024-09-21T12:37:49.413 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:49 smithi114 ceph-mon[30324]: 6.8 scrub starts 2024-09-21T12:37:49.413 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:49 smithi114 ceph-mon[30324]: 6.8 scrub ok 2024-09-21T12:37:49.413 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:49 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:37:49.414 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:49 smithi114 ceph-mon[40867]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:37:49.414 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:49 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:37:49.414 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:49 smithi114 ceph-mon[40867]: osdmap e449: 8 total, 7 up, 8 in 2024-09-21T12:37:49.414 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:49 smithi114 ceph-mon[40867]: 6.8 scrub starts 2024-09-21T12:37:49.414 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:49 smithi114 ceph-mon[40867]: 6.8 scrub ok 2024-09-21T12:37:49.417 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:49.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:49.459 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:49.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:49.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:49 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:37:49.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:49 smithi183 ceph-mon[33277]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:37:49.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:49 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:37:49.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:49 smithi183 ceph-mon[33277]: osdmap e449: 8 total, 7 up, 8 in 2024-09-21T12:37:49.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:49 smithi183 ceph-mon[33277]: 6.8 scrub starts 2024-09-21T12:37:49.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:49 smithi183 ceph-mon[33277]: 6.8 scrub ok 2024-09-21T12:37:49.560 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:49.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:49.592 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:49.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:49.693 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:49.712 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:37:49.712 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:37:49.722 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:49.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:49.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:49.756 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:37:49.756 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:37:49.757 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T12:37:49.803 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:37:49.803 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:37:49.824 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:49.839 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:37:49.839 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:37:49.860 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:49.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:49.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:49.875 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:37:49.875 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:37:49.907 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:37:49.907 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:37:49.947 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:37:49.947 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:37:49.962 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:50.010 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:50.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:50.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:50.112 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:50.140 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:50.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:50.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:50.242 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:50.280 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:50.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:50.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:50.314 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:37:50.314 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:37:50.382 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:50.419 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:50.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:50.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:50.486 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:50 smithi114 ceph-mon[30324]: 1.e scrub starts 2024-09-21T12:37:50.486 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:50 smithi114 ceph-mon[30324]: 1.e scrub ok 2024-09-21T12:37:50.486 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:50 smithi114 ceph-mon[30324]: pgmap v893: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 2.00k op/s; 25116/303573 objects degraded (8.273%); 12727/303573 objects misplaced (4.192%) 2024-09-21T12:37:50.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:50 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1051865057' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:37:50.487 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:50 smithi114 ceph-mon[40867]: 1.e scrub starts 2024-09-21T12:37:50.487 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:50 smithi114 ceph-mon[40867]: 1.e scrub ok 2024-09-21T12:37:50.487 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:50 smithi114 ceph-mon[40867]: pgmap v893: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 2.00k op/s; 25116/303573 objects degraded (8.273%); 12727/303573 objects misplaced (4.192%) 2024-09-21T12:37:50.487 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:50 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1051865057' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:37:50.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:50 smithi183 ceph-mon[33277]: 1.e scrub starts 2024-09-21T12:37:50.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:50 smithi183 ceph-mon[33277]: 1.e scrub ok 2024-09-21T12:37:50.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:50 smithi183 ceph-mon[33277]: pgmap v893: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 2.00k op/s; 25116/303573 objects degraded (8.273%); 12727/303573 objects misplaced (4.192%) 2024-09-21T12:37:50.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:50 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1051865057' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:37:50.522 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:50.560 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:50.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:50.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:50.662 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:50.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:50.697 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:50.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:50.754 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:50 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2-activate[199755]: --> Failed to activate via raw: did not find any matching OSD to activate 2024-09-21T12:37:50.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:50 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2-activate[199755]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T12:37:50.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:50 smithi114 bash[199617]: --> Failed to activate via raw: did not find any matching OSD to activate 2024-09-21T12:37:50.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:50 smithi114 bash[199617]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T12:37:50.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:50 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2-activate[199755]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T12:37:50.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:50 smithi114 bash[199617]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T12:37:50.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:50 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2-activate[199755]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T12:37:50.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:50 smithi114 bash[199617]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T12:37:50.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:50 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2-activate[199755]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2024-09-21T12:37:50.755 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:50 smithi114 bash[199617]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2024-09-21T12:37:50.798 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:50.840 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:50.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:50.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:50.942 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:50.974 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:50.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:50.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:51.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:50 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2-activate[199755]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-21T12:37:51.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:50 smithi114 bash[199617]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-21T12:37:51.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:50 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2-activate[199755]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2024-09-21T12:37:51.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:50 smithi114 bash[199617]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2024-09-21T12:37:51.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:50 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2-activate[199755]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-21T12:37:51.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:50 smithi114 bash[199617]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-21T12:37:51.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:50 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2-activate[199755]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T12:37:51.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:50 smithi114 bash[199617]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T12:37:51.025 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:50 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2-activate[199755]: --> ceph-volume lvm activate successful for osd ID: 2 2024-09-21T12:37:51.026 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:50 smithi114 bash[199617]: --> ceph-volume lvm activate successful for osd ID: 2 2024-09-21T12:37:51.026 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:50 smithi114 conmon[199755]: conmon 6749bc589c8bf7e7bf03 : Failed to open cgroups file: /sys/fs/cgroup/machine.slice/libpod-6749bc589c8bf7e7bf03077e8a1560d0eaae176fed989b5588a76c22b12f7bad.scope/memory.events 2024-09-21T12:37:51.076 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:51.110 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:51.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:51.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:51.155 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T12:37:51.170 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T12:37:51.211 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:51.240 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:51.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:51.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:51.341 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:51.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:51.403 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:51.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:51.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:51 smithi183 ceph-mon[33277]: 3.14 scrub starts 2024-09-21T12:37:51.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:51 smithi183 ceph-mon[33277]: 3.14 scrub ok 2024-09-21T12:37:51.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:51 smithi183 ceph-mon[33277]: 5.13 scrub starts 2024-09-21T12:37:51.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:51 smithi183 ceph-mon[33277]: 5.13 scrub ok 2024-09-21T12:37:51.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:51 smithi114 ceph-mon[40867]: 3.14 scrub starts 2024-09-21T12:37:51.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:51 smithi114 ceph-mon[40867]: 3.14 scrub ok 2024-09-21T12:37:51.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:51 smithi114 ceph-mon[40867]: 5.13 scrub starts 2024-09-21T12:37:51.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:51 smithi114 ceph-mon[40867]: 5.13 scrub ok 2024-09-21T12:37:51.495 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:51 smithi114 podman[199617]: 2024-09-21 12:37:51.133267895 +0000 UTC m=+6.050455351 container died 6749bc589c8bf7e7bf03077e8a1560d0eaae176fed989b5588a76c22b12f7bad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2-activate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T12:37:51.495 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:51 smithi114 ceph-mon[30324]: 3.14 scrub starts 2024-09-21T12:37:51.496 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:51 smithi114 ceph-mon[30324]: 3.14 scrub ok 2024-09-21T12:37:51.496 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:51 smithi114 ceph-mon[30324]: 5.13 scrub starts 2024-09-21T12:37:51.496 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:51 smithi114 ceph-mon[30324]: 5.13 scrub ok 2024-09-21T12:37:51.504 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:51.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:51.540 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:51.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:51.641 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:51.670 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:51.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:51.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:51.772 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:51.803 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:51.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:51.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:51.905 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:51.944 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:51.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:51.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:51.986 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T12:37:51.986 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T12:37:52.046 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:52.076 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:52.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:52.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:52.177 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:52.208 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:52.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:52.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:52.310 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:52.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:52.346 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:52.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:52.447 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:52.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:52.500 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:52.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:52.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:52 smithi114 ceph-mon[30324]: 3.3 scrub starts 2024-09-21T12:37:52.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:52 smithi114 ceph-mon[30324]: 3.3 scrub ok 2024-09-21T12:37:52.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:52 smithi114 ceph-mon[30324]: pgmap v894: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 707 B/s rd, 0 B/s wr, 1.06k op/s; 25116/303573 objects degraded (8.273%); 12727/303573 objects misplaced (4.192%) 2024-09-21T12:37:52.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:52 smithi114 ceph-mon[30324]: 5.10 scrub starts 2024-09-21T12:37:52.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:52 smithi114 ceph-mon[30324]: 5.10 scrub ok 2024-09-21T12:37:52.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:52 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 25116/303573 objects degraded (8.273%), 13 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:52.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:52 smithi114 ceph-mon[30324]: osdmap e450: 8 total, 7 up, 8 in 2024-09-21T12:37:52.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:52 smithi114 ceph-mon[40867]: 3.3 scrub starts 2024-09-21T12:37:52.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:52 smithi114 ceph-mon[40867]: 3.3 scrub ok 2024-09-21T12:37:52.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:52 smithi114 ceph-mon[40867]: pgmap v894: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 707 B/s rd, 0 B/s wr, 1.06k op/s; 25116/303573 objects degraded (8.273%); 12727/303573 objects misplaced (4.192%) 2024-09-21T12:37:52.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:52 smithi114 ceph-mon[40867]: 5.10 scrub starts 2024-09-21T12:37:52.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:52 smithi114 ceph-mon[40867]: 5.10 scrub ok 2024-09-21T12:37:52.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:52 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 25116/303573 objects degraded (8.273%), 13 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:52.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:52 smithi114 ceph-mon[40867]: osdmap e450: 8 total, 7 up, 8 in 2024-09-21T12:37:52.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:52 smithi114 podman[200553]: 2024-09-21 12:37:52.453821781 +0000 UTC m=+1.445983231 container remove 6749bc589c8bf7e7bf03077e8a1560d0eaae176fed989b5588a76c22b12f7bad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2-activate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-21T12:37:52.528 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:37:52.550 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:37:52.602 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:52.640 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:52.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:52.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:52.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:52 smithi183 ceph-mon[33277]: 3.3 scrub starts 2024-09-21T12:37:52.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:52 smithi183 ceph-mon[33277]: 3.3 scrub ok 2024-09-21T12:37:52.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:52 smithi183 ceph-mon[33277]: pgmap v894: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 707 B/s rd, 0 B/s wr, 1.06k op/s; 25116/303573 objects degraded (8.273%); 12727/303573 objects misplaced (4.192%) 2024-09-21T12:37:52.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:52 smithi183 ceph-mon[33277]: 5.10 scrub starts 2024-09-21T12:37:52.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:52 smithi183 ceph-mon[33277]: 5.10 scrub ok 2024-09-21T12:37:52.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:52 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 25116/303573 objects degraded (8.273%), 13 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:52.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:52 smithi183 ceph-mon[33277]: osdmap e450: 8 total, 7 up, 8 in 2024-09-21T12:37:52.742 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:52.776 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:52.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:52.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:52.877 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:52.913 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:52.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:52.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:53.014 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:53.024 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:37:52 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:37:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:37:53.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:53.053 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:53.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:53.154 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:53.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:53.175 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:53.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:53.276 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:53.307 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:53.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:53.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:53.409 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:53.442 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:53.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:53.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:53.544 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:53.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:53.578 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:53.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:53.679 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:53.704 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:53.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:53.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:53.806 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:53.842 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:53.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:53.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:53.944 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:53.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:53.978 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:53.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:53.987 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:37:54.079 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:54.108 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:54.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:54.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:54.210 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:54.241 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:54.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:54.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:54.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:54 smithi114 ceph-mon[30324]: 4.7 scrub starts 2024-09-21T12:37:54.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:54 smithi114 ceph-mon[30324]: 4.7 scrub ok 2024-09-21T12:37:54.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:54 smithi114 ceph-mon[30324]: 4.9 scrub starts 2024-09-21T12:37:54.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:54 smithi114 ceph-mon[30324]: 4.9 scrub ok 2024-09-21T12:37:54.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:54 smithi114 ceph-mon[30324]: pgmap v896: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.84k op/s; 23706/282252 objects degraded (8.399%); 11767/282252 objects misplaced (4.169%) 2024-09-21T12:37:54.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:54 smithi114 ceph-mon[40867]: 4.7 scrub starts 2024-09-21T12:37:54.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:54 smithi114 ceph-mon[40867]: 4.7 scrub ok 2024-09-21T12:37:54.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:54 smithi114 ceph-mon[40867]: 4.9 scrub starts 2024-09-21T12:37:54.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:54 smithi114 ceph-mon[40867]: 4.9 scrub ok 2024-09-21T12:37:54.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:54 smithi114 ceph-mon[40867]: pgmap v896: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.84k op/s; 23706/282252 objects degraded (8.399%); 11767/282252 objects misplaced (4.169%) 2024-09-21T12:37:54.343 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:54.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:54.381 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:54.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:54.482 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:54.513 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:54.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:54.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:54.615 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:54.649 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:54.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:54.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:54.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:54 smithi183 ceph-mon[33277]: 4.7 scrub starts 2024-09-21T12:37:54.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:54 smithi183 ceph-mon[33277]: 4.7 scrub ok 2024-09-21T12:37:54.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:54 smithi183 ceph-mon[33277]: 4.9 scrub starts 2024-09-21T12:37:54.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:54 smithi183 ceph-mon[33277]: 4.9 scrub ok 2024-09-21T12:37:54.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:54 smithi183 ceph-mon[33277]: pgmap v896: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.84k op/s; 23706/282252 objects degraded (8.399%); 11767/282252 objects misplaced (4.169%) 2024-09-21T12:37:54.752 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:54.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:54.789 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:54.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:54.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:54.925 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:54.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:54.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:55.027 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:55.064 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:55.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:55.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:55.166 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:55.201 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:55.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:55.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:55.303 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:55.337 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:55.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:55.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:55.439 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:55.460 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:55.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:55.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:55.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:55 smithi114 ceph-mon[40867]: 1.13 scrub starts 2024-09-21T12:37:55.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:55 smithi114 ceph-mon[40867]: 1.13 scrub ok 2024-09-21T12:37:55.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:55 smithi114 ceph-mon[40867]: 6.5 scrub starts 2024-09-21T12:37:55.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:55 smithi114 ceph-mon[40867]: 6.5 scrub ok 2024-09-21T12:37:55.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:55 smithi114 ceph-mon[40867]: 3.a scrub starts 2024-09-21T12:37:55.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:55 smithi114 ceph-mon[40867]: 3.a scrub ok 2024-09-21T12:37:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:55 smithi114 ceph-mon[30324]: 1.13 scrub starts 2024-09-21T12:37:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:55 smithi114 ceph-mon[30324]: 1.13 scrub ok 2024-09-21T12:37:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:55 smithi114 ceph-mon[30324]: 6.5 scrub starts 2024-09-21T12:37:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:55 smithi114 ceph-mon[30324]: 6.5 scrub ok 2024-09-21T12:37:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:55 smithi114 ceph-mon[30324]: 3.a scrub starts 2024-09-21T12:37:55.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:55 smithi114 ceph-mon[30324]: 3.a scrub ok 2024-09-21T12:37:55.562 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:55.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:55.591 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:55.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:55.692 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:55.723 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:55.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:55.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:55.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:55 smithi183 ceph-mon[33277]: 1.13 scrub starts 2024-09-21T12:37:55.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:55 smithi183 ceph-mon[33277]: 1.13 scrub ok 2024-09-21T12:37:55.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:55 smithi183 ceph-mon[33277]: 6.5 scrub starts 2024-09-21T12:37:55.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:55 smithi183 ceph-mon[33277]: 6.5 scrub ok 2024-09-21T12:37:55.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:55 smithi183 ceph-mon[33277]: 3.a scrub starts 2024-09-21T12:37:55.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:55 smithi183 ceph-mon[33277]: 3.a scrub ok 2024-09-21T12:37:55.825 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:55.846 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:55.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:55.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:55.948 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:55.983 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:55.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:55.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:56.085 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:56.120 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:56.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:56.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:56.222 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:56.250 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:56.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:56.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:56.305 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:37:56.305 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:37:56.352 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:56.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:56.384 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:56.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:56.485 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:56.516 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:56.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:56.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:56.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:56 smithi114 ceph-mon[30324]: 4.1b scrub starts 2024-09-21T12:37:56.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:56 smithi114 ceph-mon[30324]: 4.1b scrub ok 2024-09-21T12:37:56.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:56 smithi114 ceph-mon[30324]: 6.d scrub starts 2024-09-21T12:37:56.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:56 smithi114 ceph-mon[30324]: 6.d scrub ok 2024-09-21T12:37:56.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:56 smithi114 ceph-mon[30324]: pgmap v897: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.36k op/s; 23251/280887 objects degraded (8.278%); 11767/280887 objects misplaced (4.189%) 2024-09-21T12:37:56.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:56 smithi114 ceph-mon[40867]: 4.1b scrub starts 2024-09-21T12:37:56.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:56 smithi114 ceph-mon[40867]: 4.1b scrub ok 2024-09-21T12:37:56.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:56 smithi114 ceph-mon[40867]: 6.d scrub starts 2024-09-21T12:37:56.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:56 smithi114 ceph-mon[40867]: 6.d scrub ok 2024-09-21T12:37:56.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:56 smithi114 ceph-mon[40867]: pgmap v897: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.36k op/s; 23251/280887 objects degraded (8.278%); 11767/280887 objects misplaced (4.189%) 2024-09-21T12:37:56.618 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:56.651 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:56.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:56.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:56.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:56 smithi183 ceph-mon[33277]: 4.1b scrub starts 2024-09-21T12:37:56.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:56 smithi183 ceph-mon[33277]: 4.1b scrub ok 2024-09-21T12:37:56.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:56 smithi183 ceph-mon[33277]: 6.d scrub starts 2024-09-21T12:37:56.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:56 smithi183 ceph-mon[33277]: 6.d scrub ok 2024-09-21T12:37:56.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:56 smithi183 ceph-mon[33277]: pgmap v897: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.36k op/s; 23251/280887 objects degraded (8.278%); 11767/280887 objects misplaced (4.189%) 2024-09-21T12:37:56.753 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:56.788 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:56.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:56.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:56.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:56.912 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:56.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:56.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:57.014 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:57.035 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:57.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:57.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:57.137 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:57.158 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:57.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:57.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:57.260 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:57.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:57.283 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:57.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:57.384 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:57.419 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:57.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:57.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:57.520 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:57.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:57 smithi114 ceph-mon[40867]: 1.1e scrub starts 2024-09-21T12:37:57.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:57 smithi114 ceph-mon[40867]: 5.1f scrub starts 2024-09-21T12:37:57.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:57 smithi114 ceph-mon[40867]: 1.1e scrub ok 2024-09-21T12:37:57.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:57 smithi114 ceph-mon[40867]: 5.1f scrub ok 2024-09-21T12:37:57.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:57 smithi114 ceph-mon[40867]: 4.8 scrub starts 2024-09-21T12:37:57.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:57 smithi114 ceph-mon[40867]: 4.8 scrub ok 2024-09-21T12:37:57.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:57 smithi114 ceph-mon[40867]: 1.10 scrub starts 2024-09-21T12:37:57.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:57 smithi114 ceph-mon[40867]: 1.10 scrub ok 2024-09-21T12:37:57.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:57 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 23251/280887 objects degraded (8.278%), 13 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:57.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:57 smithi114 ceph-mon[30324]: 1.1e scrub starts 2024-09-21T12:37:57.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:57 smithi114 ceph-mon[30324]: 5.1f scrub starts 2024-09-21T12:37:57.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:57 smithi114 ceph-mon[30324]: 1.1e scrub ok 2024-09-21T12:37:57.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:57 smithi114 ceph-mon[30324]: 5.1f scrub ok 2024-09-21T12:37:57.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:57 smithi114 ceph-mon[30324]: 4.8 scrub starts 2024-09-21T12:37:57.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:57 smithi114 ceph-mon[30324]: 4.8 scrub ok 2024-09-21T12:37:57.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:57 smithi114 ceph-mon[30324]: 1.10 scrub starts 2024-09-21T12:37:57.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:57 smithi114 ceph-mon[30324]: 1.10 scrub ok 2024-09-21T12:37:57.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:57 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 23251/280887 objects degraded (8.278%), 13 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:57.555 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:57.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:57.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:57.657 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:57.677 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:57.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:57.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:57.684 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:57 smithi183 ceph-mon[33277]: 1.1e scrub starts 2024-09-21T12:37:57.685 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:57 smithi183 ceph-mon[33277]: 5.1f scrub starts 2024-09-21T12:37:57.685 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:57 smithi183 ceph-mon[33277]: 1.1e scrub ok 2024-09-21T12:37:57.685 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:57 smithi183 ceph-mon[33277]: 5.1f scrub ok 2024-09-21T12:37:57.685 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:57 smithi183 ceph-mon[33277]: 4.8 scrub starts 2024-09-21T12:37:57.685 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:57 smithi183 ceph-mon[33277]: 4.8 scrub ok 2024-09-21T12:37:57.685 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:57 smithi183 ceph-mon[33277]: 1.10 scrub starts 2024-09-21T12:37:57.685 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:57 smithi183 ceph-mon[33277]: 1.10 scrub ok 2024-09-21T12:37:57.685 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:57 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 23251/280887 objects degraded (8.278%), 13 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T12:37:57.779 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:57.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:57.815 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:57.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:57.916 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:57.947 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:57.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:57.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:57.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:37:57 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:37:58.048 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:58.078 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:58.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:58.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:58.180 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:58.213 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:58.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:58.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:58.315 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:58.346 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:58.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:58.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:58.448 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:37:58.487 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:58.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:58.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:37:58.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:58 smithi114 ceph-mon[40867]: 6.6 scrub starts 2024-09-21T12:37:58.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:58 smithi114 ceph-mon[40867]: 6.6 scrub ok 2024-09-21T12:37:58.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:58 smithi114 ceph-mon[40867]: pgmap v898: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.34k op/s; 22360/268905 objects degraded (8.315%); 10913/268905 objects misplaced (4.058%) 2024-09-21T12:37:58.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:58 smithi114 ceph-mon[40867]: 2.2 scrub starts 2024-09-21T12:37:58.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:58 smithi114 ceph-mon[40867]: 2.2 scrub ok 2024-09-21T12:37:58.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:58 smithi114 ceph-mon[40867]: 5.1a scrub starts 2024-09-21T12:37:58.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:58 smithi114 ceph-mon[40867]: 5.1a scrub ok 2024-09-21T12:37:58.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:58 smithi114 ceph-mon[30324]: 6.6 scrub starts 2024-09-21T12:37:58.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:58 smithi114 ceph-mon[30324]: 6.6 scrub ok 2024-09-21T12:37:58.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:58 smithi114 ceph-mon[30324]: pgmap v898: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.34k op/s; 22360/268905 objects degraded (8.315%); 10913/268905 objects misplaced (4.058%) 2024-09-21T12:37:58.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:58 smithi114 ceph-mon[30324]: 2.2 scrub starts 2024-09-21T12:37:58.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:58 smithi114 ceph-mon[30324]: 2.2 scrub ok 2024-09-21T12:37:58.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:58 smithi114 ceph-mon[30324]: 5.1a scrub starts 2024-09-21T12:37:58.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:58 smithi114 ceph-mon[30324]: 5.1a scrub ok 2024-09-21T12:37:58.589 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:58.619 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:58.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:58.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:58.681 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T12:37:58.721 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:58.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:58 smithi183 ceph-mon[33277]: 6.6 scrub starts 2024-09-21T12:37:58.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:58 smithi183 ceph-mon[33277]: 6.6 scrub ok 2024-09-21T12:37:58.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:58 smithi183 ceph-mon[33277]: pgmap v898: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.34k op/s; 22360/268905 objects degraded (8.315%); 10913/268905 objects misplaced (4.058%) 2024-09-21T12:37:58.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:58 smithi183 ceph-mon[33277]: 2.2 scrub starts 2024-09-21T12:37:58.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:58 smithi183 ceph-mon[33277]: 2.2 scrub ok 2024-09-21T12:37:58.740 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:58 smithi183 ceph-mon[33277]: 5.1a scrub starts 2024-09-21T12:37:58.740 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:58 smithi183 ceph-mon[33277]: 5.1a scrub ok 2024-09-21T12:37:58.754 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:58.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:58.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:58.857 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:58.896 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:58.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:58.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:37:58.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:58 smithi114 podman[201128]: 2024-09-21 12:37:58.693671322 +0000 UTC m=+0.043482406 image pull 8f4d013db793795fef09ff5edf96e6b42ba8cd2c9158b2f0b7a5d7c1e657c063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472 2024-09-21T12:37:58.979 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:58 smithi114 podman[201128]: 2024-09-21 12:37:58.818274413 +0000 UTC m=+0.168085480 container create 9a201988edd88a93759b283219b7fe6b24114181dd77304737ba093030e8770d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, ceph=True, org.label-schema.vendor=CentOS) 2024-09-21T12:37:58.997 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:59.032 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:59.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:59.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:59.134 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:37:59.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:59.171 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:59.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:37:59.272 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:37:59.315 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:59.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:59.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:37:59.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:59 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:37:59.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:59 smithi114 ceph-mon[40867]: 5.0 scrub starts 2024-09-21T12:37:59.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:59 smithi114 ceph-mon[40867]: 5.0 scrub ok 2024-09-21T12:37:59.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:59 smithi114 ceph-mon[40867]: 6.14 scrub starts 2024-09-21T12:37:59.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:59 smithi114 ceph-mon[40867]: 6.14 scrub ok 2024-09-21T12:37:59.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:59 smithi114 ceph-mon[40867]: 6.4 scrub starts 2024-09-21T12:37:59.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:37:59 smithi114 ceph-mon[40867]: 6.4 scrub ok 2024-09-21T12:37:59.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:59 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:37:59.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:59 smithi114 ceph-mon[30324]: 5.0 scrub starts 2024-09-21T12:37:59.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:59 smithi114 ceph-mon[30324]: 5.0 scrub ok 2024-09-21T12:37:59.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:59 smithi114 ceph-mon[30324]: 6.14 scrub starts 2024-09-21T12:37:59.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:59 smithi114 ceph-mon[30324]: 6.14 scrub ok 2024-09-21T12:37:59.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:59 smithi114 ceph-mon[30324]: 6.4 scrub starts 2024-09-21T12:37:59.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:37:59 smithi114 ceph-mon[30324]: 6.4 scrub ok 2024-09-21T12:37:59.374 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:59 smithi114 podman[201128]: 2024-09-21 12:37:59.078518004 +0000 UTC m=+0.428329070 container init 9a201988edd88a93759b283219b7fe6b24114181dd77304737ba093030e8770d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-21T12:37:59.374 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:59 smithi114 podman[201128]: 2024-09-21 12:37:59.086640098 +0000 UTC m=+0.436451162 container start 9a201988edd88a93759b283219b7fe6b24114181dd77304737ba093030e8770d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=) 2024-09-21T12:37:59.374 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:59 smithi114 ceph-osd[201221]: -- 172.21.15.114:0/3232103798 <== mon.0 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560cb62afa40 con 0x560cb7086000 2024-09-21T12:37:59.418 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:37:59.450 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:59.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:59.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:37:59.531 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T12:37:59.541 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:37:59.552 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:59.574 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:59.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:59.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:59.583 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:37:59.675 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:37:59.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:59 smithi114 bash[201128]: 9a201988edd88a93759b283219b7fe6b24114181dd77304737ba093030e8770d 2024-09-21T12:37:59.676 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:37:59 smithi114 systemd[1]: Started Ceph osd.2 for e2ad975c-7812-11ef-baf6-efdc52797490. 2024-09-21T12:37:59.705 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:37:59.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:59.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:37:59.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:59 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:37:59.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:59 smithi183 ceph-mon[33277]: 5.0 scrub starts 2024-09-21T12:37:59.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:59 smithi183 ceph-mon[33277]: 5.0 scrub ok 2024-09-21T12:37:59.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:59 smithi183 ceph-mon[33277]: 6.14 scrub starts 2024-09-21T12:37:59.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:59 smithi183 ceph-mon[33277]: 6.14 scrub ok 2024-09-21T12:37:59.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:59 smithi183 ceph-mon[33277]: 6.4 scrub starts 2024-09-21T12:37:59.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:37:59 smithi183 ceph-mon[33277]: 6.4 scrub ok 2024-09-21T12:37:59.807 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:37:59.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:59.830 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:59.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:37:59.931 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:37:59.952 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:37:59.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:37:59.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:00.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:00.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:00.076 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:00.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:00.177 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:00.199 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:00.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:00.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:00.288 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-21T12:38:00.301 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:00.323 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:00.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:00.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:00.339 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:38:00.339 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:38:00.339 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:38:00.339 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:38:00.339 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T12:38:00.348 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:38:00.348 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:38:00.425 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:00.446 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:38:00.446 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:38:00.461 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:00.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:00.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:00.465 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:00 smithi114 ceph-mon[30324]: pgmap v899: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s; 21544/260058 objects degraded (8.284%); 10913/260058 objects misplaced (4.196%) 2024-09-21T12:38:00.465 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:00 smithi114 ceph-mon[30324]: 3.e scrub starts 2024-09-21T12:38:00.465 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:00 smithi114 ceph-mon[30324]: 3.e scrub ok 2024-09-21T12:38:00.465 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:00 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4264677067' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:38:00.465 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:00 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:38:00.466 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:00 smithi114 ceph-mon[40867]: pgmap v899: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s; 21544/260058 objects degraded (8.284%); 10913/260058 objects misplaced (4.196%) 2024-09-21T12:38:00.466 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:00 smithi114 ceph-mon[40867]: 3.e scrub starts 2024-09-21T12:38:00.466 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:00 smithi114 ceph-mon[40867]: 3.e scrub ok 2024-09-21T12:38:00.466 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:00 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/4264677067' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:38:00.466 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:00 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:38:00.476 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:38:00.476 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:38:00.515 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:38:00.515 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:38:00.553 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:38:00.553 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:38:00.563 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:00.607 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:00.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:00.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:00.630 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:38:00.630 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:38:00.630 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:38:00.630 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:38:00.630 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:38:00.709 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:00.737 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:00.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:00 smithi183 ceph-mon[33277]: pgmap v899: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s; 21544/260058 objects degraded (8.284%); 10913/260058 objects misplaced (4.196%) 2024-09-21T12:38:00.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:00 smithi183 ceph-mon[33277]: 3.e scrub starts 2024-09-21T12:38:00.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:00 smithi183 ceph-mon[33277]: 3.e scrub ok 2024-09-21T12:38:00.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:00 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/4264677067' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:38:00.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:00 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:38:00.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:00.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:00.840 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:00.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:00.877 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:00.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:00.978 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:01.012 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:01.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:01.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:01.114 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:01.147 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:01.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:01.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:01.249 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:01.289 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:01.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:01.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:01.391 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:01.423 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:01.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:01.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:01.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[40867]: 6.16 scrub starts 2024-09-21T12:38:01.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[40867]: 6.16 scrub ok 2024-09-21T12:38:01.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[40867]: 4.17 scrub starts 2024-09-21T12:38:01.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[40867]: 4.17 scrub ok 2024-09-21T12:38:01.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[40867]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:38:01.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:38:01.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[40867]: osdmap e451: 8 total, 7 up, 8 in 2024-09-21T12:38:01.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3909120236' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:38:01.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[40867]: 6.7 scrub starts 2024-09-21T12:38:01.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[40867]: 6.7 scrub ok 2024-09-21T12:38:01.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[40867]: 5.12 scrub starts 2024-09-21T12:38:01.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[40867]: 5.12 scrub ok 2024-09-21T12:38:01.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[40867]: 1.11 scrub starts 2024-09-21T12:38:01.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[40867]: 1.11 scrub ok 2024-09-21T12:38:01.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[40867]: 5.9 scrub starts 2024-09-21T12:38:01.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[40867]: 5.9 scrub ok 2024-09-21T12:38:01.526 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2[201186]: 2024-09-21T12:38:01.120+0000 7ffb7bd72640 -1 Falling back to public interface 2024-09-21T12:38:01.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[30324]: 6.16 scrub starts 2024-09-21T12:38:01.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[30324]: 6.16 scrub ok 2024-09-21T12:38:01.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[30324]: 4.17 scrub starts 2024-09-21T12:38:01.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[30324]: 4.17 scrub ok 2024-09-21T12:38:01.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[30324]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:38:01.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:38:01.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[30324]: osdmap e451: 8 total, 7 up, 8 in 2024-09-21T12:38:01.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3909120236' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:38:01.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[30324]: 6.7 scrub starts 2024-09-21T12:38:01.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[30324]: 6.7 scrub ok 2024-09-21T12:38:01.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[30324]: 5.12 scrub starts 2024-09-21T12:38:01.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[30324]: 5.12 scrub ok 2024-09-21T12:38:01.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[30324]: 1.11 scrub starts 2024-09-21T12:38:01.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[30324]: 1.11 scrub ok 2024-09-21T12:38:01.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[30324]: 5.9 scrub starts 2024-09-21T12:38:01.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:01 smithi114 ceph-mon[30324]: 5.9 scrub ok 2024-09-21T12:38:01.527 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:01.558 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:01.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:01.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:01.629 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T12:38:01.660 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:01.689 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:01.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:01.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:01.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:01 smithi183 ceph-mon[33277]: 6.16 scrub starts 2024-09-21T12:38:01.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:01 smithi183 ceph-mon[33277]: 6.16 scrub ok 2024-09-21T12:38:01.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:01 smithi183 ceph-mon[33277]: 4.17 scrub starts 2024-09-21T12:38:01.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:01 smithi183 ceph-mon[33277]: 4.17 scrub ok 2024-09-21T12:38:01.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:01 smithi183 ceph-mon[33277]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:38:01.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:01 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:38:01.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:01 smithi183 ceph-mon[33277]: osdmap e451: 8 total, 7 up, 8 in 2024-09-21T12:38:01.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:01 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3909120236' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:38:01.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:01 smithi183 ceph-mon[33277]: 6.7 scrub starts 2024-09-21T12:38:01.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:01 smithi183 ceph-mon[33277]: 6.7 scrub ok 2024-09-21T12:38:01.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:01 smithi183 ceph-mon[33277]: 5.12 scrub starts 2024-09-21T12:38:01.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:01 smithi183 ceph-mon[33277]: 5.12 scrub ok 2024-09-21T12:38:01.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:01 smithi183 ceph-mon[33277]: 1.11 scrub starts 2024-09-21T12:38:01.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:01 smithi183 ceph-mon[33277]: 1.11 scrub ok 2024-09-21T12:38:01.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:01 smithi183 ceph-mon[33277]: 5.9 scrub starts 2024-09-21T12:38:01.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:01 smithi183 ceph-mon[33277]: 5.9 scrub ok 2024-09-21T12:38:01.791 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:01.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:01.826 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:01.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:01.927 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:01.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:01.967 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:01.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:02.068 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:02.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:02.100 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:02.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:02.201 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:02.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:02.235 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:02.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:02.336 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:02.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:02.373 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:02.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:02.474 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:02.501 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:02.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:02.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:02.603 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:02.635 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:02.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:02.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:02.682 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:02 smithi114 ceph-mon[30324]: 11.e scrub starts 2024-09-21T12:38:02.682 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:02 smithi114 ceph-mon[30324]: 6.b scrub starts 2024-09-21T12:38:02.682 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:02 smithi114 ceph-mon[30324]: 6.b scrub ok 2024-09-21T12:38:02.682 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:02 smithi114 ceph-mon[30324]: pgmap v901: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 567 B/s rd, 0 B/s wr, 1.13k op/s; 21544/260058 objects degraded (8.284%); 10913/260058 objects misplaced (4.196%) 2024-09-21T12:38:02.682 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:02 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:38:02.682 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:02 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:38:02.683 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:02 smithi114 ceph-mon[30324]: osdmap e452: 8 total, 7 up, 8 in 2024-09-21T12:38:02.683 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:02 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 21544/260058 objects degraded (8.284%), 13 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T12:38:02.683 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:38:02 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:38:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:38:02.683 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:02 smithi114 ceph-mon[40867]: 11.e scrub starts 2024-09-21T12:38:02.683 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:02 smithi114 ceph-mon[40867]: 6.b scrub starts 2024-09-21T12:38:02.683 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:02 smithi114 ceph-mon[40867]: 6.b scrub ok 2024-09-21T12:38:02.684 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:02 smithi114 ceph-mon[40867]: pgmap v901: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 567 B/s rd, 0 B/s wr, 1.13k op/s; 21544/260058 objects degraded (8.284%); 10913/260058 objects misplaced (4.196%) 2024-09-21T12:38:02.684 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:02 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:38:02.684 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:02 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:38:02.684 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:02 smithi114 ceph-mon[40867]: osdmap e452: 8 total, 7 up, 8 in 2024-09-21T12:38:02.684 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:02 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 21544/260058 objects degraded (8.284%), 13 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T12:38:02.736 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:02.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:02 smithi183 ceph-mon[33277]: 11.e scrub starts 2024-09-21T12:38:02.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:02 smithi183 ceph-mon[33277]: 6.b scrub starts 2024-09-21T12:38:02.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:02 smithi183 ceph-mon[33277]: 6.b scrub ok 2024-09-21T12:38:02.740 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:02 smithi183 ceph-mon[33277]: pgmap v901: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 5.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 567 B/s rd, 0 B/s wr, 1.13k op/s; 21544/260058 objects degraded (8.284%); 10913/260058 objects misplaced (4.196%) 2024-09-21T12:38:02.740 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:02 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:38:02.740 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:02 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:38:02.740 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:02 smithi183 ceph-mon[33277]: osdmap e452: 8 total, 7 up, 8 in 2024-09-21T12:38:02.740 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:02 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 21544/260058 objects degraded (8.284%), 13 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T12:38:02.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:02.770 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:02.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:02.871 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:02.906 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:02.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:02.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:03.008 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:03.031 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:03.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:03.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:03.132 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:03.164 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:03.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:03.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:03.266 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:03.299 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:03.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:03.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:03.401 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:03.430 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:03.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:03.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:03.532 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:03.565 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:03.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:03.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:03.586 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:03 smithi114 ceph-mon[30324]: 11.e scrub ok 2024-09-21T12:38:03.586 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:03 smithi114 ceph-mon[30324]: 3.2 scrub starts 2024-09-21T12:38:03.586 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:03 smithi114 ceph-mon[30324]: 3.2 scrub ok 2024-09-21T12:38:03.586 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:03 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:38:03.586 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:03 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:38:03.586 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:03 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:38:03.586 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:03 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:38:03.586 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:03 smithi114 ceph-mon[40867]: 11.e scrub ok 2024-09-21T12:38:03.587 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:03 smithi114 ceph-mon[40867]: 3.2 scrub starts 2024-09-21T12:38:03.587 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:03 smithi114 ceph-mon[40867]: 3.2 scrub ok 2024-09-21T12:38:03.587 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:03 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:38:03.587 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:03 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:38:03.587 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:03 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:38:03.587 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:03 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:38:03.592 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:38:03.636 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T12:38:03.636 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T12:38:03.668 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:03.711 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:03.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:03.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:03.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:03 smithi183 ceph-mon[33277]: 11.e scrub ok 2024-09-21T12:38:03.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:03 smithi183 ceph-mon[33277]: 3.2 scrub starts 2024-09-21T12:38:03.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:03 smithi183 ceph-mon[33277]: 3.2 scrub ok 2024-09-21T12:38:03.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:03 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:38:03.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:03 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:38:03.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:03 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:38:03.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:03 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:38:03.813 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:03.859 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:03.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:03.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:03.961 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:03.989 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:03.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:03.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:04.091 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:04.113 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:04.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:04.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:04.214 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:04.244 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:04.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:04.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:04.346 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:04.376 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:04.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:04.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:04.388 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:38:04.428 INFO:teuthology.orchestra.run.smithi114.stderr:no valid command found; 10 closest matches: 2024-09-21T12:38:04.428 INFO:teuthology.orchestra.run.smithi114.stderr:0 2024-09-21T12:38:04.428 INFO:teuthology.orchestra.run.smithi114.stderr:1 2024-09-21T12:38:04.428 INFO:teuthology.orchestra.run.smithi114.stderr:2 2024-09-21T12:38:04.428 INFO:teuthology.orchestra.run.smithi114.stderr:abort 2024-09-21T12:38:04.429 INFO:teuthology.orchestra.run.smithi114.stderr:assert 2024-09-21T12:38:04.429 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs debug_inject_read_zeros 2024-09-21T12:38:04.429 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs files list 2024-09-21T12:38:04.429 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs stats 2024-09-21T12:38:04.429 INFO:teuthology.orchestra.run.smithi114.stderr:bluestore allocator dump block 2024-09-21T12:38:04.429 INFO:teuthology.orchestra.run.smithi114.stderr:bluestore allocator fragmentation block 2024-09-21T12:38:04.429 INFO:teuthology.orchestra.run.smithi114.stderr:admin_socket: invalid command 2024-09-21T12:38:04.478 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:04.500 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T12:38:04.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:04.520 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:04.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:04.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:04 smithi114 ceph-mon[30324]: 5.e scrub starts 2024-09-21T12:38:04.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:04 smithi114 ceph-mon[30324]: 5.e scrub ok 2024-09-21T12:38:04.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:04 smithi114 ceph-mon[30324]: pgmap v903: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 4.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.62k op/s; 20332/242115 objects degraded (8.398%); 10136/242115 objects misplaced (4.186%) 2024-09-21T12:38:04.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:04 smithi114 ceph-mon[30324]: osdmap e453: 8 total, 7 up, 8 in 2024-09-21T12:38:04.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:04 smithi114 ceph-mon[30324]: 4.1a scrub starts 2024-09-21T12:38:04.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:04 smithi114 ceph-mon[30324]: 4.1a scrub ok 2024-09-21T12:38:04.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:04 smithi114 ceph-mon[30324]: 1.16 scrub starts 2024-09-21T12:38:04.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:04 smithi114 ceph-mon[30324]: 1.16 scrub ok 2024-09-21T12:38:04.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:04 smithi114 ceph-mon[30324]: 6.15 scrub starts 2024-09-21T12:38:04.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:04 smithi114 ceph-mon[30324]: 6.15 scrub ok 2024-09-21T12:38:04.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:04 smithi114 ceph-mon[40867]: 5.e scrub starts 2024-09-21T12:38:04.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:04 smithi114 ceph-mon[40867]: 5.e scrub ok 2024-09-21T12:38:04.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:04 smithi114 ceph-mon[40867]: pgmap v903: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 4.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.62k op/s; 20332/242115 objects degraded (8.398%); 10136/242115 objects misplaced (4.186%) 2024-09-21T12:38:04.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:04 smithi114 ceph-mon[40867]: osdmap e453: 8 total, 7 up, 8 in 2024-09-21T12:38:04.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:04 smithi114 ceph-mon[40867]: 4.1a scrub starts 2024-09-21T12:38:04.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:04 smithi114 ceph-mon[40867]: 4.1a scrub ok 2024-09-21T12:38:04.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:04 smithi114 ceph-mon[40867]: 1.16 scrub starts 2024-09-21T12:38:04.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:04 smithi114 ceph-mon[40867]: 1.16 scrub ok 2024-09-21T12:38:04.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:04 smithi114 ceph-mon[40867]: 6.15 scrub starts 2024-09-21T12:38:04.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:04 smithi114 ceph-mon[40867]: 6.15 scrub ok 2024-09-21T12:38:04.621 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:04.648 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:04.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:04.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:04.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:04 smithi183 ceph-mon[33277]: 5.e scrub starts 2024-09-21T12:38:04.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:04 smithi183 ceph-mon[33277]: 5.e scrub ok 2024-09-21T12:38:04.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:04 smithi183 ceph-mon[33277]: pgmap v903: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 4.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.62k op/s; 20332/242115 objects degraded (8.398%); 10136/242115 objects misplaced (4.186%) 2024-09-21T12:38:04.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:04 smithi183 ceph-mon[33277]: osdmap e453: 8 total, 7 up, 8 in 2024-09-21T12:38:04.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:04 smithi183 ceph-mon[33277]: 4.1a scrub starts 2024-09-21T12:38:04.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:04 smithi183 ceph-mon[33277]: 4.1a scrub ok 2024-09-21T12:38:04.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:04 smithi183 ceph-mon[33277]: 1.16 scrub starts 2024-09-21T12:38:04.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:04 smithi183 ceph-mon[33277]: 1.16 scrub ok 2024-09-21T12:38:04.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:04 smithi183 ceph-mon[33277]: 6.15 scrub starts 2024-09-21T12:38:04.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:04 smithi183 ceph-mon[33277]: 6.15 scrub ok 2024-09-21T12:38:04.750 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:04.786 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:04.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:04.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:04.887 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:04.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:04.922 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:04.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:04.930 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:38:04 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2[201186]: 2024-09-21T12:38:04.638+0000 7ffb7bd72640 -1 osd.2 387 log_to_monitors true 2024-09-21T12:38:05.024 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:05.054 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:05.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:05.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:05.156 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:05.192 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:05.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:05.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:05.294 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:05.333 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:05.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:05.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:05.430 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T12:38:05.430 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-09-21T12:38:05.435 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:05.462 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:05.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:05.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:05.564 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:05.594 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:05.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:05.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:05.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:05 smithi114 ceph-mon[30324]: 4.16 scrub starts 2024-09-21T12:38:05.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:05 smithi114 ceph-mon[30324]: from='osd.2 [v2:172.21.15.114:6818/1359699248,v1:172.21.15.114:6819/1359699248]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T12:38:05.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:05 smithi114 ceph-mon[30324]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T12:38:05.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:05 smithi114 ceph-mon[40867]: 4.16 scrub starts 2024-09-21T12:38:05.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:05 smithi114 ceph-mon[40867]: from='osd.2 [v2:172.21.15.114:6818/1359699248,v1:172.21.15.114:6819/1359699248]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T12:38:05.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:05 smithi114 ceph-mon[40867]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T12:38:05.622 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:38:05 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2[201186]: 2024-09-21T12:38:05.619+0000 7ffb6fb14640 -1 osd.2 454 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T12:38:05.638 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:38:05.695 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:05.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:05 smithi183 ceph-mon[33277]: 4.16 scrub starts 2024-09-21T12:38:05.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:05 smithi183 ceph-mon[33277]: from='osd.2 [v2:172.21.15.114:6818/1359699248,v1:172.21.15.114:6819/1359699248]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T12:38:05.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:05 smithi183 ceph-mon[33277]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T12:38:05.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:05.747 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:05.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:05.848 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:05.875 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:05.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:05.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:05.976 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:06.005 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:06.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:06.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:06.053 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:38:06.053 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 20, 2024-09-21T12:38:06.053 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 600, 2024-09-21T12:38:06.053 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-21T12:38:06.053 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.053 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:179611 9.16 9:6ab28fd3:::benchmark_data_smithi114_133214_object55577:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-21T12:38:06.054 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:01.026416+0000", 2024-09-21T12:38:06.054 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 425.021940771, 2024-09-21T12:38:06.054 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0040347110000001, 2024-09-21T12:38:06.054 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:38:06.054 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:38:06.054 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:38:06.054 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:38:06.054 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:38:06.054 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:38:06.054 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 179611 2024-09-21T12:38:06.054 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.054 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:38:06.054 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.054 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:38:06.055 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.026416+0000", 2024-09-21T12:38:06.055 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.055 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.055 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.055 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:38:06.055 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.026416+0000", 2024-09-21T12:38:06.055 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.055 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.055 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.055 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:38:06.055 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.026417+0000", 2024-09-21T12:38:06.055 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2240000000000001e-06 2024-09-21T12:38:06.055 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.055 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.055 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:38:06.056 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.026423+0000", 2024-09-21T12:38:06.056 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0679999999999996e-06 2024-09-21T12:38:06.056 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.056 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.056 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:38:06.056 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.026436+0000", 2024-09-21T12:38:06.056 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.34e-05 2024-09-21T12:38:06.056 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.056 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.056 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:38:06.056 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.026443+0000", 2024-09-21T12:38:06.056 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5079999999999999e-06 2024-09-21T12:38:06.056 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.056 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.056 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.056 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.026505+0000", 2024-09-21T12:38:06.057 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1450000000000003e-05 2024-09-21T12:38:06.057 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.057 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.057 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:38:06.057 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.026632+0000", 2024-09-21T12:38:06.057 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012670800000000001 2024-09-21T12:38:06.057 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.057 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.057 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,5,6", 2024-09-21T12:38:06.057 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.026929+0000", 2024-09-21T12:38:06.057 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00029765500000000002 2024-09-21T12:38:06.057 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.058 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.058 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.058 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.034732+0000", 2024-09-21T12:38:06.058 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.007802425 2024-09-21T12:38:06.058 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.058 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.058 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:38:06.058 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:01.034741+0000", 2024-09-21T12:38:06.058 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1020000000000003e-06 2024-09-21T12:38:06.058 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.058 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.058 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.058 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.030372+0000", 2024-09-21T12:38:06.058 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99563096699999998 2024-09-21T12:38:06.059 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.059 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.059 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:38:06.059 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:02.030451+0000", 2024-09-21T12:38:06.059 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9203999999999996e-05 2024-09-21T12:38:06.059 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.059 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:38:06.059 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.059 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.059 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.059 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:180848 9.12 9:4ecd9eaa:::benchmark_data_smithi114_133214_object56814:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-09-21T12:38:06.059 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:04.045860+0000", 2024-09-21T12:38:06.059 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 422.00249660899999, 2024-09-21T12:38:06.059 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99015821699999995, 2024-09-21T12:38:06.059 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:38:06.059 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:38:06.060 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:38:06.060 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:38:06.060 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:38:06.060 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:38:06.060 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 180848 2024-09-21T12:38:06.060 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.060 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:38:06.060 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.060 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:38:06.060 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.045860+0000", 2024-09-21T12:38:06.060 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.060 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.060 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.060 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:38:06.060 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.045860+0000", 2024-09-21T12:38:06.061 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.061 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.061 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.061 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:38:06.061 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.045862+0000", 2024-09-21T12:38:06.061 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.466e-06 2024-09-21T12:38:06.061 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.061 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.061 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:38:06.061 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.045866+0000", 2024-09-21T12:38:06.061 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.89e-06 2024-09-21T12:38:06.061 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.061 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.061 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:38:06.061 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.045887+0000", 2024-09-21T12:38:06.062 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0801e-05 2024-09-21T12:38:06.062 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.062 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.062 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:38:06.062 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.045894+0000", 2024-09-21T12:38:06.062 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2830000000000004e-06 2024-09-21T12:38:06.062 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.062 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.062 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.062 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.045948+0000", 2024-09-21T12:38:06.062 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3677999999999997e-05 2024-09-21T12:38:06.062 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.062 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.062 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:38:06.062 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:04.045956+0000", 2024-09-21T12:38:06.062 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1079999999999993e-06 2024-09-21T12:38:06.063 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.063 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.063 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.063 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:05.032414+0000", 2024-09-21T12:38:06.063 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98645813900000001 2024-09-21T12:38:06.063 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.063 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.063 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:38:06.063 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:05.032550+0000", 2024-09-21T12:38:06.063 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013655699999999999 2024-09-21T12:38:06.063 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.063 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.063 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,5", 2024-09-21T12:38:06.063 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:05.032837+0000", 2024-09-21T12:38:06.064 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00028715899999999999 2024-09-21T12:38:06.064 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.064 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.064 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.064 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:05.035846+0000", 2024-09-21T12:38:06.064 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0030088020000000001 2024-09-21T12:38:06.064 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.064 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.064 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:38:06.064 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:05.035860+0000", 2024-09-21T12:38:06.064 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3366e-05 2024-09-21T12:38:06.064 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.064 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.064 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.064 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:05.035895+0000", 2024-09-21T12:38:06.064 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5751999999999999e-05 2024-09-21T12:38:06.065 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.065 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.065 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:38:06.065 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:05.035973+0000", 2024-09-21T12:38:06.065 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7999999999999999e-05 2024-09-21T12:38:06.065 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.065 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.065 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:38:06.065 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:05.036019+0000", 2024-09-21T12:38:06.065 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5216000000000002e-05 2024-09-21T12:38:06.065 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.065 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:38:06.065 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.065 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.065 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.065 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231723 9.14 9:2de8fd01:::benchmark_data_smithi114_133214_object107689:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T12:38:06.066 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:49.066280+0000", 2024-09-21T12:38:06.066 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 376.98207743799998, 2024-09-21T12:38:06.066 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1384219230000001, 2024-09-21T12:38:06.066 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:38:06.066 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:38:06.066 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:38:06.066 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:38:06.066 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:38:06.066 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:38:06.066 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231723 2024-09-21T12:38:06.066 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.066 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:38:06.067 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.067 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:38:06.067 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.066280+0000", 2024-09-21T12:38:06.067 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.067 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.067 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.067 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:38:06.067 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.066280+0000", 2024-09-21T12:38:06.067 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.067 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.067 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.067 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:38:06.067 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.066281+0000", 2024-09-21T12:38:06.067 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5939999999999999e-06 2024-09-21T12:38:06.068 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.068 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.068 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:38:06.068 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.066286+0000", 2024-09-21T12:38:06.068 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0189999999999997e-06 2024-09-21T12:38:06.068 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.068 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.068 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:38:06.068 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.066298+0000", 2024-09-21T12:38:06.068 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2024999999999999e-05 2024-09-21T12:38:06.068 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.069 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.069 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:38:06.069 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.066305+0000", 2024-09-21T12:38:06.069 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9870000000000002e-06 2024-09-21T12:38:06.069 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.069 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.069 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.069 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.066371+0000", 2024-09-21T12:38:06.069 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6025999999999994e-05 2024-09-21T12:38:06.069 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.069 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.069 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:38:06.069 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:49.066382+0000", 2024-09-21T12:38:06.069 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0311e-05 2024-09-21T12:38:06.070 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.070 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.070 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.070 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.038211+0000", 2024-09-21T12:38:06.070 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97182923300000001 2024-09-21T12:38:06.070 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.070 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.070 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:38:06.070 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.038222+0000", 2024-09-21T12:38:06.070 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1032e-05 2024-09-21T12:38:06.070 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.070 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.070 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.070 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.201658+0000", 2024-09-21T12:38:06.070 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1634362899999999 2024-09-21T12:38:06.071 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.071 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.071 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:38:06.071 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.201745+0000", 2024-09-21T12:38:06.071 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6829000000000005e-05 2024-09-21T12:38:06.071 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.071 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.071 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,3,5", 2024-09-21T12:38:06.071 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.201998+0000", 2024-09-21T12:38:06.071 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000253212 2024-09-21T12:38:06.071 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.071 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.071 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:38:06.071 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.203777+0000", 2024-09-21T12:38:06.072 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001778565 2024-09-21T12:38:06.072 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.072 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.072 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.072 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.204187+0000", 2024-09-21T12:38:06.072 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000409873 2024-09-21T12:38:06.072 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.072 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.072 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.072 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.204239+0000", 2024-09-21T12:38:06.072 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2166999999999997e-05 2024-09-21T12:38:06.072 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.072 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.072 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.073 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.204276+0000", 2024-09-21T12:38:06.073 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.7181999999999997e-05 2024-09-21T12:38:06.073 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.073 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.073 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:38:06.073 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.204485+0000", 2024-09-21T12:38:06.073 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00020932399999999999 2024-09-21T12:38:06.073 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.073 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.073 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:38:06.073 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.204701+0000", 2024-09-21T12:38:06.073 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00021625400000000001 2024-09-21T12:38:06.073 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.074 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:38:06.074 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.074 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.074 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.074 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231668 9.7 9:e40b995a:::benchmark_data_smithi114_133214_object107634:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:38:06.074 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029737+0000", 2024-09-21T12:38:06.074 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 376.01862008900002, 2024-09-21T12:38:06.074 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1844050580000001, 2024-09-21T12:38:06.074 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:38:06.074 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:38:06.074 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:38:06.074 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:38:06.074 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:38:06.074 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:38:06.074 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231668 2024-09-21T12:38:06.075 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.075 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:38:06.075 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.075 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:38:06.075 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029737+0000", 2024-09-21T12:38:06.075 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.075 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.075 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.075 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:38:06.075 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029737+0000", 2024-09-21T12:38:06.075 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.075 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.075 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.075 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:38:06.076 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029739+0000", 2024-09-21T12:38:06.076 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9300000000000002e-06 2024-09-21T12:38:06.076 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.076 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.076 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:38:06.076 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029744+0000", 2024-09-21T12:38:06.076 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6970000000000001e-06 2024-09-21T12:38:06.076 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.076 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.076 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:38:06.076 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029764+0000", 2024-09-21T12:38:06.076 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0959e-05 2024-09-21T12:38:06.076 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.076 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.076 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:38:06.076 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029773+0000", 2024-09-21T12:38:06.077 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9390000000000003e-06 2024-09-21T12:38:06.077 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.077 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.077 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.077 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.030032+0000", 2024-09-21T12:38:06.077 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00025845499999999999 2024-09-21T12:38:06.077 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.077 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.077 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op must wait for map", 2024-09-21T12:38:06.077 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.030039+0000", 2024-09-21T12:38:06.077 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5310000000000004e-06 2024-09-21T12:38:06.077 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.077 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.077 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.078 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033252+0000", 2024-09-21T12:38:06.078 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0032126519999999999 2024-09-21T12:38:06.078 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.078 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.078 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:38:06.078 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033262+0000", 2024-09-21T12:38:06.078 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6039999999999999e-06 2024-09-21T12:38:06.078 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.078 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.078 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.078 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.214079+0000", 2024-09-21T12:38:06.078 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1808168320000001 2024-09-21T12:38:06.078 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.078 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.079 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:38:06.079 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.214142+0000", 2024-09-21T12:38:06.079 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3459000000000003e-05 2024-09-21T12:38:06.079 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.079 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:38:06.079 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.079 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.079 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.079 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231684 9.2 9:431c20d3:::benchmark_data_smithi114_133214_object107650:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:38:06.079 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029798+0000", 2024-09-21T12:38:06.079 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 376.01855922599998, 2024-09-21T12:38:06.079 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.174456956, 2024-09-21T12:38:06.079 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:38:06.079 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:38:06.080 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:38:06.080 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:38:06.080 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:38:06.080 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:38:06.080 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231684 2024-09-21T12:38:06.080 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.081 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:38:06.081 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.081 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:38:06.081 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029798+0000", 2024-09-21T12:38:06.081 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.081 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.081 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.081 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:38:06.081 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029798+0000", 2024-09-21T12:38:06.081 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.081 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.081 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.081 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:38:06.081 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029798+0000", 2024-09-21T12:38:06.082 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4600000000000005e-07 2024-09-21T12:38:06.082 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.082 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.082 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:38:06.082 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029802+0000", 2024-09-21T12:38:06.082 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.399e-06 2024-09-21T12:38:06.082 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.082 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.082 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:38:06.082 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029813+0000", 2024-09-21T12:38:06.082 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0927999999999999e-05 2024-09-21T12:38:06.082 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.082 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.082 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:38:06.083 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029819+0000", 2024-09-21T12:38:06.083 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3840000000000002e-06 2024-09-21T12:38:06.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.083 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.083 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.083 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029883+0000", 2024-09-21T12:38:06.083 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3903000000000001e-05 2024-09-21T12:38:06.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.083 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.083 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op must wait for map", 2024-09-21T12:38:06.083 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029891+0000", 2024-09-21T12:38:06.083 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1929999999999993e-06 2024-09-21T12:38:06.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.083 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.084 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.084 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033472+0000", 2024-09-21T12:38:06.084 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003581009 2024-09-21T12:38:06.084 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.084 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.084 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:38:06.084 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033481+0000", 2024-09-21T12:38:06.084 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5320000000000006e-06 2024-09-21T12:38:06.084 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.084 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.084 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.084 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.204173+0000", 2024-09-21T12:38:06.084 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.170692377 2024-09-21T12:38:06.084 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.085 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.085 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:38:06.085 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.204255+0000", 2024-09-21T12:38:06.085 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1685000000000001e-05 2024-09-21T12:38:06.085 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.085 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:38:06.085 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.085 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.085 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.085 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231688 9.7 9:e423c016:::benchmark_data_smithi114_133214_object107654:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:38:06.085 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029833+0000", 2024-09-21T12:38:06.085 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 376.01852383599999, 2024-09-21T12:38:06.086 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.184627359, 2024-09-21T12:38:06.086 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:38:06.086 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:38:06.086 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:38:06.086 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:38:06.086 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:38:06.086 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:38:06.086 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231688 2024-09-21T12:38:06.086 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.086 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:38:06.086 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.086 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:38:06.086 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029833+0000", 2024-09-21T12:38:06.086 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.087 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.087 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.087 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:38:06.087 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029833+0000", 2024-09-21T12:38:06.087 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.087 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.087 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.087 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:38:06.088 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029834+0000", 2024-09-21T12:38:06.088 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6600000000000002e-07 2024-09-21T12:38:06.088 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.088 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.088 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:38:06.088 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029836+0000", 2024-09-21T12:38:06.088 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.6450000000000001e-06 2024-09-21T12:38:06.088 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.088 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.088 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:38:06.088 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029843+0000", 2024-09-21T12:38:06.088 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6440000000000003e-06 2024-09-21T12:38:06.089 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.089 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.089 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:38:06.089 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029848+0000", 2024-09-21T12:38:06.089 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0549999999999999e-06 2024-09-21T12:38:06.089 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.089 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.089 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.089 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033286+0000", 2024-09-21T12:38:06.089 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0034382089999999998 2024-09-21T12:38:06.089 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.089 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.089 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:38:06.089 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033294+0000", 2024-09-21T12:38:06.089 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9279999999999993e-06 2024-09-21T12:38:06.090 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.090 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.090 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.090 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.214400+0000", 2024-09-21T12:38:06.090 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.181105869 2024-09-21T12:38:06.090 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.090 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.090 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:38:06.090 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.214461+0000", 2024-09-21T12:38:06.090 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0543e-05 2024-09-21T12:38:06.090 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.090 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:38:06.090 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.090 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.091 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.091 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231694 9.7 9:e4532815:::benchmark_data_smithi114_133214_object107660:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:38:06.091 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029855+0000", 2024-09-21T12:38:06.091 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 376.01850191099999, 2024-09-21T12:38:06.091 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.212233884, 2024-09-21T12:38:06.091 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:38:06.091 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:38:06.091 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:38:06.091 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:38:06.091 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:38:06.091 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:38:06.091 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231694 2024-09-21T12:38:06.091 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.092 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:38:06.092 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.092 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:38:06.092 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029855+0000", 2024-09-21T12:38:06.092 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.092 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.092 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.092 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:38:06.092 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029855+0000", 2024-09-21T12:38:06.092 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.092 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.092 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.092 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:38:06.092 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029856+0000", 2024-09-21T12:38:06.093 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4499999999999997e-07 2024-09-21T12:38:06.093 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.093 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.093 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:38:06.093 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029858+0000", 2024-09-21T12:38:06.093 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.131e-06 2024-09-21T12:38:06.093 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.093 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.093 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:38:06.093 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029864+0000", 2024-09-21T12:38:06.093 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1410000000000002e-06 2024-09-21T12:38:06.093 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.093 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.093 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:38:06.093 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029869+0000", 2024-09-21T12:38:06.093 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7829999999999998e-06 2024-09-21T12:38:06.094 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.094 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.094 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.094 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033319+0000", 2024-09-21T12:38:06.094 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003450304 2024-09-21T12:38:06.094 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.094 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.094 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:38:06.094 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033326+0000", 2024-09-21T12:38:06.094 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3590000000000002e-06 2024-09-21T12:38:06.094 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.094 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.094 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.094 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.214613+0000", 2024-09-21T12:38:06.095 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1812871789999999 2024-09-21T12:38:06.095 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.095 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.095 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:38:06.095 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.214822+0000", 2024-09-21T12:38:06.095 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00020822599999999999 2024-09-21T12:38:06.095 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.095 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.095 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,3,5", 2024-09-21T12:38:06.095 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.215218+0000", 2024-09-21T12:38:06.095 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000396728 2024-09-21T12:38:06.095 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.095 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.095 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:38:06.096 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.229559+0000", 2024-09-21T12:38:06.096 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.014340258 2024-09-21T12:38:06.096 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.096 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.096 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.096 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.232468+0000", 2024-09-21T12:38:06.096 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0029092469999999998 2024-09-21T12:38:06.096 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.096 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.096 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.096 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.232552+0000", 2024-09-21T12:38:06.096 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.3855000000000003e-05 2024-09-21T12:38:06.096 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.096 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.096 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.096 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.241712+0000", 2024-09-21T12:38:06.097 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0091604219999999997 2024-09-21T12:38:06.097 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.097 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.097 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:38:06.097 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.241968+0000", 2024-09-21T12:38:06.097 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00025556599999999999 2024-09-21T12:38:06.097 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.097 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.097 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:38:06.097 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.242089+0000", 2024-09-21T12:38:06.097 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012124 2024-09-21T12:38:06.097 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.097 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:38:06.097 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.097 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.098 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.098 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231696 9.2 9:4615e034:::benchmark_data_smithi114_133214_object107662:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:38:06.098 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029874+0000", 2024-09-21T12:38:06.098 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 376.01848293199998, 2024-09-21T12:38:06.098 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.181378748, 2024-09-21T12:38:06.098 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:38:06.098 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:38:06.098 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:38:06.098 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:38:06.098 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:38:06.098 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:38:06.098 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231696 2024-09-21T12:38:06.098 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.099 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:38:06.099 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.099 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:38:06.099 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029874+0000", 2024-09-21T12:38:06.099 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.099 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.099 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.099 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:38:06.099 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029874+0000", 2024-09-21T12:38:06.099 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.099 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.099 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.099 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:38:06.099 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029874+0000", 2024-09-21T12:38:06.099 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5600000000000001e-07 2024-09-21T12:38:06.099 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.100 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.100 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:38:06.100 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029876+0000", 2024-09-21T12:38:06.100 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.99e-06 2024-09-21T12:38:06.100 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.100 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.100 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:38:06.100 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029882+0000", 2024-09-21T12:38:06.100 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4750000000000001e-06 2024-09-21T12:38:06.100 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.100 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.100 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:38:06.100 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029887+0000", 2024-09-21T12:38:06.100 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2839999999999997e-06 2024-09-21T12:38:06.100 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.100 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.101 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.101 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033506+0000", 2024-09-21T12:38:06.101 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00361875 2024-09-21T12:38:06.101 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.101 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.101 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:38:06.101 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033514+0000", 2024-09-21T12:38:06.101 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6850000000000001e-06 2024-09-21T12:38:06.101 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.101 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.101 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.101 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.204539+0000", 2024-09-21T12:38:06.101 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1710255940000001 2024-09-21T12:38:06.101 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.101 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.102 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:38:06.102 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.204647+0000", 2024-09-21T12:38:06.102 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000107538 2024-09-21T12:38:06.102 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.102 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.102 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,3,7", 2024-09-21T12:38:06.102 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.204970+0000", 2024-09-21T12:38:06.102 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00032291999999999998 2024-09-21T12:38:06.102 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.102 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.102 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:38:06.102 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.207582+0000", 2024-09-21T12:38:06.102 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0026120230000000002 2024-09-21T12:38:06.103 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.103 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.103 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.103 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.210625+0000", 2024-09-21T12:38:06.103 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0030428959999999998 2024-09-21T12:38:06.103 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.103 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.103 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.103 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.210912+0000", 2024-09-21T12:38:06.103 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00028736700000000002 2024-09-21T12:38:06.103 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.103 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.103 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.104 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.211091+0000", 2024-09-21T12:38:06.104 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017910199999999999 2024-09-21T12:38:06.104 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.104 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.104 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:38:06.104 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.211150+0000", 2024-09-21T12:38:06.104 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8934999999999997e-05 2024-09-21T12:38:06.104 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.104 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.104 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:38:06.104 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.211253+0000", 2024-09-21T12:38:06.104 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000102833 2024-09-21T12:38:06.104 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.104 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:38:06.104 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.104 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.105 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.105 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231712 9.2 9:41d44c3f:::benchmark_data_smithi114_133214_object107678:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:38:06.105 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029893+0000", 2024-09-21T12:38:06.105 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 376.01846394799998, 2024-09-21T12:38:06.105 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.194669177, 2024-09-21T12:38:06.105 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:38:06.105 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:38:06.105 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:38:06.105 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:38:06.105 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:38:06.105 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:38:06.105 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231712 2024-09-21T12:38:06.105 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.105 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:38:06.105 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.105 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:38:06.106 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029893+0000", 2024-09-21T12:38:06.106 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.106 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.106 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.106 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:38:06.106 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029893+0000", 2024-09-21T12:38:06.106 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.106 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.106 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.106 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:38:06.106 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029893+0000", 2024-09-21T12:38:06.106 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5499999999999999e-07 2024-09-21T12:38:06.106 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.106 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.106 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:38:06.106 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029895+0000", 2024-09-21T12:38:06.107 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7859999999999999e-06 2024-09-21T12:38:06.107 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.107 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.107 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:38:06.107 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029901+0000", 2024-09-21T12:38:06.107 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7239999999999997e-06 2024-09-21T12:38:06.107 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.107 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.107 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:38:06.107 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029905+0000", 2024-09-21T12:38:06.107 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:38:06.108 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5210000000000004e-06 2024-09-21T12:38:06.108 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.108 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.108 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.108 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033538+0000", 2024-09-21T12:38:06.108 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0036328900000000002 2024-09-21T12:38:06.108 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.108 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.108 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:38:06.109 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033546+0000", 2024-09-21T12:38:06.109 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9170000000000006e-06 2024-09-21T12:38:06.109 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.109 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.109 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.109 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.207834+0000", 2024-09-21T12:38:06.109 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.17428818 2024-09-21T12:38:06.109 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.109 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.109 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:38:06.109 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.207946+0000", 2024-09-21T12:38:06.109 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000111501 2024-09-21T12:38:06.109 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.109 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.109 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,3,7", 2024-09-21T12:38:06.110 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.209056+0000", 2024-09-21T12:38:06.110 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0011098670000000001 2024-09-21T12:38:06.110 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.110 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.110 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.110 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.212301+0000", 2024-09-21T12:38:06.110 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0032456199999999998 2024-09-21T12:38:06.110 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.110 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.110 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:38:06.110 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.212318+0000", 2024-09-21T12:38:06.110 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7031999999999999e-05 2024-09-21T12:38:06.110 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.110 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.111 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.111 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.214089+0000", 2024-09-21T12:38:06.111 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001770726 2024-09-21T12:38:06.111 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.111 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.111 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.111 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.224405+0000", 2024-09-21T12:38:06.111 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010315658 2024-09-21T12:38:06.111 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.111 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.111 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:38:06.111 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.224461+0000", 2024-09-21T12:38:06.111 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6341999999999997e-05 2024-09-21T12:38:06.111 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.112 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.112 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:38:06.112 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.224562+0000", 2024-09-21T12:38:06.112 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010105799999999999 2024-09-21T12:38:06.112 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.112 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:38:06.112 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.112 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.112 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.112 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231746 9.7 9:e6f0ab86:::benchmark_data_smithi114_133214_object107712:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:38:06.112 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029911+0000", 2024-09-21T12:38:06.112 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 376.01844595300003, 2024-09-21T12:38:06.112 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2150179190000001, 2024-09-21T12:38:06.113 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:38:06.113 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:38:06.113 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:38:06.113 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:38:06.113 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:38:06.113 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:38:06.113 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231746 2024-09-21T12:38:06.113 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.113 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:38:06.113 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.113 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:38:06.113 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029911+0000", 2024-09-21T12:38:06.113 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.113 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.114 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.114 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:38:06.114 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029911+0000", 2024-09-21T12:38:06.114 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.114 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.114 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.114 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:38:06.114 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029911+0000", 2024-09-21T12:38:06.114 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4700000000000002e-07 2024-09-21T12:38:06.114 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.114 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.114 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:38:06.114 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029913+0000", 2024-09-21T12:38:06.114 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.889e-06 2024-09-21T12:38:06.115 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.115 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.115 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:38:06.115 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029919+0000", 2024-09-21T12:38:06.115 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3630000000000002e-06 2024-09-21T12:38:06.115 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.115 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.115 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:38:06.115 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029923+0000", 2024-09-21T12:38:06.115 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4379999999999997e-06 2024-09-21T12:38:06.115 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.115 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.115 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.116 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033349+0000", 2024-09-21T12:38:06.116 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0034264059999999999 2024-09-21T12:38:06.116 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.116 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.116 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:38:06.116 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033357+0000", 2024-09-21T12:38:06.116 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4590000000000001e-06 2024-09-21T12:38:06.116 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.116 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.116 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.116 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.225869+0000", 2024-09-21T12:38:06.116 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.192512056 2024-09-21T12:38:06.116 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.117 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.117 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:38:06.117 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.225992+0000", 2024-09-21T12:38:06.117 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012313800000000001 2024-09-21T12:38:06.117 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.117 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.117 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,3,5", 2024-09-21T12:38:06.117 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.226276+0000", 2024-09-21T12:38:06.117 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00028393899999999998 2024-09-21T12:38:06.117 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.117 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.117 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.117 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.232512+0000", 2024-09-21T12:38:06.117 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0062360330000000002 2024-09-21T12:38:06.117 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.118 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.118 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.118 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.232592+0000", 2024-09-21T12:38:06.118 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9430000000000004e-05 2024-09-21T12:38:06.118 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.118 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.118 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:38:06.118 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.241522+0000", 2024-09-21T12:38:06.118 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0089301380000000007 2024-09-21T12:38:06.118 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.118 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.118 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.118 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.244727+0000", 2024-09-21T12:38:06.118 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0032050389999999998 2024-09-21T12:38:06.119 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.119 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.119 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:38:06.119 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.244780+0000", 2024-09-21T12:38:06.119 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3777999999999999e-05 2024-09-21T12:38:06.119 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.119 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.119 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:38:06.119 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.244929+0000", 2024-09-21T12:38:06.119 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014846600000000001 2024-09-21T12:38:06.119 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.119 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:38:06.119 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.119 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.119 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.120 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231751 9.7 9:e24ce5a0:::benchmark_data_smithi114_133214_object107717:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:38:06.120 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029929+0000", 2024-09-21T12:38:06.120 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 376.01842833799998, 2024-09-21T12:38:06.120 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2152077539999999, 2024-09-21T12:38:06.120 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:38:06.120 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:38:06.120 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:38:06.120 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:38:06.120 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:38:06.120 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:38:06.120 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231751 2024-09-21T12:38:06.120 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.120 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:38:06.120 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.121 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:38:06.121 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029929+0000", 2024-09-21T12:38:06.121 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.121 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.121 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.121 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:38:06.121 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029929+0000", 2024-09-21T12:38:06.121 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.121 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.121 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.121 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:38:06.121 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029929+0000", 2024-09-21T12:38:06.121 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5199999999999998e-07 2024-09-21T12:38:06.122 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.122 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.122 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:38:06.122 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029931+0000", 2024-09-21T12:38:06.122 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0090000000000002e-06 2024-09-21T12:38:06.122 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.122 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.122 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:38:06.122 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029936+0000", 2024-09-21T12:38:06.122 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.215e-06 2024-09-21T12:38:06.122 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.122 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.122 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:38:06.122 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029940+0000", 2024-09-21T12:38:06.123 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.2570000000000001e-06 2024-09-21T12:38:06.123 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.123 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.123 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.123 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033382+0000", 2024-09-21T12:38:06.123 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0034412370000000002 2024-09-21T12:38:06.123 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.123 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.123 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:38:06.123 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033388+0000", 2024-09-21T12:38:06.123 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6780000000000003e-06 2024-09-21T12:38:06.123 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.123 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.123 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.124 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.233447+0000", 2024-09-21T12:38:06.124 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.200058297 2024-09-21T12:38:06.124 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.124 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.124 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:38:06.124 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.233585+0000", 2024-09-21T12:38:06.124 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000138429 2024-09-21T12:38:06.124 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.124 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.124 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,3,5", 2024-09-21T12:38:06.124 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.233890+0000", 2024-09-21T12:38:06.124 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00030514399999999998 2024-09-21T12:38:06.124 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.124 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.124 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.125 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.237017+0000", 2024-09-21T12:38:06.125 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003126789 2024-09-21T12:38:06.125 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.125 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.125 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.125 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.237064+0000", 2024-09-21T12:38:06.125 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7287000000000003e-05 2024-09-21T12:38:06.125 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.125 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.125 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:38:06.125 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.242303+0000", 2024-09-21T12:38:06.125 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.005238599 2024-09-21T12:38:06.125 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.125 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.126 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.126 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.244979+0000", 2024-09-21T12:38:06.126 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0026761549999999999 2024-09-21T12:38:06.126 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.126 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.126 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:38:06.126 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.245035+0000", 2024-09-21T12:38:06.126 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6159000000000002e-05 2024-09-21T12:38:06.126 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.126 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.126 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:38:06.126 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.245136+0000", 2024-09-21T12:38:06.126 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000101147 2024-09-21T12:38:06.126 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.126 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:38:06.127 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.127 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.127 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.127 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231758 9.7 9:e06abb50:::benchmark_data_smithi114_133214_object107724:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:38:06.127 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029948+0000", 2024-09-21T12:38:06.127 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 376.01840927299997, 2024-09-21T12:38:06.127 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2159567739999999, 2024-09-21T12:38:06.127 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:38:06.127 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:38:06.127 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:38:06.127 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:38:06.127 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:38:06.127 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:38:06.127 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231758 2024-09-21T12:38:06.128 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.128 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:38:06.128 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.128 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:38:06.128 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029948+0000", 2024-09-21T12:38:06.128 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.128 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.128 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.128 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:38:06.128 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029948+0000", 2024-09-21T12:38:06.128 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.128 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.128 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.128 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:38:06.128 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029948+0000", 2024-09-21T12:38:06.129 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.41e-07 2024-09-21T12:38:06.129 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.129 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.129 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:38:06.129 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029950+0000", 2024-09-21T12:38:06.129 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0470000000000001e-06 2024-09-21T12:38:06.129 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.129 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.129 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:38:06.129 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029956+0000", 2024-09-21T12:38:06.129 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7069999999999997e-06 2024-09-21T12:38:06.129 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.129 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.129 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:38:06.129 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029961+0000", 2024-09-21T12:38:06.130 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1320000000000002e-06 2024-09-21T12:38:06.130 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.130 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.130 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.130 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033413+0000", 2024-09-21T12:38:06.130 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0034517609999999998 2024-09-21T12:38:06.130 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.130 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.130 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:38:06.130 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033420+0000", 2024-09-21T12:38:06.130 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6240000000000002e-06 2024-09-21T12:38:06.130 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.130 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.130 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.131 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.235474+0000", 2024-09-21T12:38:06.131 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.202053832 2024-09-21T12:38:06.131 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.131 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.131 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:38:06.131 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.235573+0000", 2024-09-21T12:38:06.131 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.9216999999999996e-05 2024-09-21T12:38:06.131 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.131 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.131 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,3,5", 2024-09-21T12:38:06.131 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.235777+0000", 2024-09-21T12:38:06.131 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000203687 2024-09-21T12:38:06.131 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.131 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.132 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.132 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.239048+0000", 2024-09-21T12:38:06.132 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003271173 2024-09-21T12:38:06.132 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.132 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.132 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.132 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.241655+0000", 2024-09-21T12:38:06.132 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002607023 2024-09-21T12:38:06.132 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.132 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.132 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:38:06.132 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.242308+0000", 2024-09-21T12:38:06.132 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00065309099999999996 2024-09-21T12:38:06.132 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.132 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.133 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.133 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.245732+0000", 2024-09-21T12:38:06.133 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003423461 2024-09-21T12:38:06.133 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.133 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.133 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:38:06.133 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.245786+0000", 2024-09-21T12:38:06.133 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4289999999999997e-05 2024-09-21T12:38:06.133 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.133 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.133 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:38:06.133 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.245904+0000", 2024-09-21T12:38:06.133 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011838800000000001 2024-09-21T12:38:06.133 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.133 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:38:06.134 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.134 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.134 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.134 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231761 9.7 9:e362f0a9:::benchmark_data_smithi114_133214_object107727:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-09-21T12:38:06.134 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:50.029967+0000", 2024-09-21T12:38:06.134 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 376.01839047499999, 2024-09-21T12:38:06.134 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.217986754, 2024-09-21T12:38:06.134 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:38:06.134 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:38:06.134 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:38:06.134 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:38:06.134 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:38:06.134 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:38:06.135 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231761 2024-09-21T12:38:06.135 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.135 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:38:06.135 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.135 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:38:06.135 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029967+0000", 2024-09-21T12:38:06.135 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.135 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.135 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.135 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:38:06.135 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029967+0000", 2024-09-21T12:38:06.135 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.135 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.135 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.135 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:38:06.136 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029967+0000", 2024-09-21T12:38:06.136 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4900000000000001e-07 2024-09-21T12:38:06.136 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.136 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.136 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:38:06.136 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029969+0000", 2024-09-21T12:38:06.136 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.68e-06 2024-09-21T12:38:06.136 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.136 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.136 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:38:06.136 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029974+0000", 2024-09-21T12:38:06.136 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5899999999999998e-06 2024-09-21T12:38:06.136 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.136 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.136 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:38:06.137 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.029979+0000", 2024-09-21T12:38:06.137 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6530000000000002e-06 2024-09-21T12:38:06.137 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.137 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.137 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.137 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033442+0000", 2024-09-21T12:38:06.137 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0034634930000000002 2024-09-21T12:38:06.137 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.137 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.137 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:38:06.137 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:50.033449+0000", 2024-09-21T12:38:06.137 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9290000000000001e-06 2024-09-21T12:38:06.137 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.137 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.137 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.137 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.237181+0000", 2024-09-21T12:38:06.138 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2037321190000001 2024-09-21T12:38:06.138 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.138 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.138 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:38:06.138 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.237313+0000", 2024-09-21T12:38:06.138 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013190700000000001 2024-09-21T12:38:06.138 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.138 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.138 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,3,5", 2024-09-21T12:38:06.138 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.237544+0000", 2024-09-21T12:38:06.138 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00023036199999999999 2024-09-21T12:38:06.138 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.139 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.139 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.139 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.241579+0000", 2024-09-21T12:38:06.139 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.004035294 2024-09-21T12:38:06.139 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.139 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.139 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:38:06.139 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.242313+0000", 2024-09-21T12:38:06.139 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00073439600000000001 2024-09-21T12:38:06.139 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.139 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.139 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.139 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.245175+0000", 2024-09-21T12:38:06.139 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0028617299999999998 2024-09-21T12:38:06.139 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.140 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.140 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.140 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.247800+0000", 2024-09-21T12:38:06.140 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002624548 2024-09-21T12:38:06.140 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.140 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.140 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:38:06.140 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.247850+0000", 2024-09-21T12:38:06.140 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0158999999999999e-05 2024-09-21T12:38:06.140 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.140 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.140 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:38:06.140 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:51.247953+0000", 2024-09-21T12:38:06.140 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000103545 2024-09-21T12:38:06.141 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.141 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:38:06.141 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.141 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.141 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.141 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231982 9.14 9:2d540493:::benchmark_data_smithi114_133214_object107948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-09-21T12:38:06.141 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:52.048929+0000", 2024-09-21T12:38:06.141 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 373.99942834699999, 2024-09-21T12:38:06.141 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99998082899999996, 2024-09-21T12:38:06.141 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:38:06.141 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:38:06.141 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:38:06.141 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:38:06.141 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:38:06.141 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:38:06.142 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231982 2024-09-21T12:38:06.142 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.142 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:38:06.142 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.142 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:38:06.142 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.048929+0000", 2024-09-21T12:38:06.142 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.142 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.142 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.142 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:38:06.142 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.048929+0000", 2024-09-21T12:38:06.142 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.142 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.143 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.143 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:38:06.143 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.048931+0000", 2024-09-21T12:38:06.143 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.6589999999999999e-06 2024-09-21T12:38:06.143 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.143 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.143 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:38:06.143 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.048936+0000", 2024-09-21T12:38:06.143 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0869999999999997e-06 2024-09-21T12:38:06.143 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.143 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.144 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:38:06.144 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.048950+0000", 2024-09-21T12:38:06.144 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3976e-05 2024-09-21T12:38:06.144 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.144 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.144 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:38:06.144 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.048962+0000", 2024-09-21T12:38:06.144 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1225e-05 2024-09-21T12:38:06.144 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.144 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.144 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.144 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.049061+0000", 2024-09-21T12:38:06.144 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.9086999999999995e-05 2024-09-21T12:38:06.144 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.145 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.145 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:38:06.145 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.049068+0000", 2024-09-21T12:38:06.145 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6539999999999999e-06 2024-09-21T12:38:06.145 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.145 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.145 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.145 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.045781+0000", 2024-09-21T12:38:06.145 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99671266999999997 2024-09-21T12:38:06.145 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.145 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.145 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:38:06.145 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.045859+0000", 2024-09-21T12:38:06.145 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7891000000000004e-05 2024-09-21T12:38:06.145 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.145 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.146 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,5", 2024-09-21T12:38:06.146 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.046118+0000", 2024-09-21T12:38:06.146 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00025944599999999998 2024-09-21T12:38:06.146 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.146 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.146 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.146 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.048757+0000", 2024-09-21T12:38:06.146 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0026386069999999998 2024-09-21T12:38:06.146 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.146 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.146 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:38:06.146 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.048771+0000", 2024-09-21T12:38:06.146 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4440999999999999e-05 2024-09-21T12:38:06.146 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.146 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.146 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.147 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.048824+0000", 2024-09-21T12:38:06.147 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2460999999999998e-05 2024-09-21T12:38:06.147 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.147 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.147 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:38:06.147 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.048880+0000", 2024-09-21T12:38:06.147 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6067999999999999e-05 2024-09-21T12:38:06.147 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.147 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.147 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:38:06.147 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.048909+0000", 2024-09-21T12:38:06.147 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.9556999999999998e-05 2024-09-21T12:38:06.147 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.147 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:38:06.147 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.147 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.148 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.148 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.25436.0:231987 9.14 9:2dd07ba6:::benchmark_data_smithi114_133214_object107953:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-09-21T12:38:06.148 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:31:52.052061+0000", 2024-09-21T12:38:06.148 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:06.149 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 373.99629636499998, 2024-09-21T12:38:06.149 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99820888500000005, 2024-09-21T12:38:06.149 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:38:06.149 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:38:06.150 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:38:06.150 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:38:06.150 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.25436", 2024-09-21T12:38:06.150 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:38:06.150 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 231987 2024-09-21T12:38:06.150 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.150 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:38:06.150 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.150 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:38:06.150 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.052061+0000", 2024-09-21T12:38:06.150 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.150 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.151 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.151 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:38:06.151 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.052061+0000", 2024-09-21T12:38:06.151 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.151 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.151 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.151 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:38:06.151 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.052062+0000", 2024-09-21T12:38:06.151 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.026e-06 2024-09-21T12:38:06.151 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.151 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.151 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:38:06.151 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.052479+0000", 2024-09-21T12:38:06.151 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00041693600000000002 2024-09-21T12:38:06.152 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.152 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.152 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:38:06.152 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.052490+0000", 2024-09-21T12:38:06.152 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1270000000000001e-05 2024-09-21T12:38:06.152 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.152 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.152 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:38:06.152 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.052496+0000", 2024-09-21T12:38:06.152 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4330000000000001e-06 2024-09-21T12:38:06.152 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.152 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.152 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.152 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.052654+0000", 2024-09-21T12:38:06.153 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015731400000000001 2024-09-21T12:38:06.153 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.153 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.153 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:38:06.153 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:52.052663+0000", 2024-09-21T12:38:06.153 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9760000000000001e-06 2024-09-21T12:38:06.153 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.153 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.153 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.153 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.047676+0000", 2024-09-21T12:38:06.153 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99501326700000003 2024-09-21T12:38:06.153 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.153 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.153 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:38:06.154 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.047793+0000", 2024-09-21T12:38:06.154 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000117099 2024-09-21T12:38:06.154 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.154 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.154 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,5", 2024-09-21T12:38:06.154 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.047970+0000", 2024-09-21T12:38:06.154 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017714599999999999 2024-09-21T12:38:06.154 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.154 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.154 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.154 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.050154+0000", 2024-09-21T12:38:06.154 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0021837110000000001 2024-09-21T12:38:06.154 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.154 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.155 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:38:06.155 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.050164+0000", 2024-09-21T12:38:06.155 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0156000000000001e-05 2024-09-21T12:38:06.155 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.155 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.155 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.155 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.050195+0000", 2024-09-21T12:38:06.155 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.0966999999999997e-05 2024-09-21T12:38:06.155 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.155 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.155 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:38:06.155 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.050237+0000", 2024-09-21T12:38:06.155 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1909999999999997e-05 2024-09-21T12:38:06.155 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.155 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.156 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:38:06.156 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:31:53.050270+0000", 2024-09-21T12:38:06.156 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2673999999999999e-05 2024-09-21T12:38:06.156 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.156 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:38:06.156 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.156 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.156 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.156 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.14478.0:3149 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94620848901376 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-21T12:38:06.156 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:34:33.326939+0000", 2024-09-21T12:38:06.156 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 212.721418202, 2024-09-21T12:38:06.156 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.013910753, 2024-09-21T12:38:06.156 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:38:06.156 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:38:06.156 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:38:06.157 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:38:06.157 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.14478", 2024-09-21T12:38:06.157 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1371625176", 2024-09-21T12:38:06.157 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 3149 2024-09-21T12:38:06.157 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.157 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:38:06.157 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.157 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:38:06.157 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.326939+0000", 2024-09-21T12:38:06.157 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.157 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.157 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.157 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:38:06.157 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.326939+0000", 2024-09-21T12:38:06.157 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.158 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.158 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.158 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:38:06.158 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.326941+0000", 2024-09-21T12:38:06.158 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8059999999999999e-06 2024-09-21T12:38:06.158 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.158 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.158 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:38:06.158 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.326945+0000", 2024-09-21T12:38:06.158 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1860000000000002e-06 2024-09-21T12:38:06.158 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.158 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.158 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:38:06.158 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.326960+0000", 2024-09-21T12:38:06.158 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5400000000000002e-05 2024-09-21T12:38:06.158 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.159 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.159 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:38:06.159 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.326968+0000", 2024-09-21T12:38:06.159 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6709999999999999e-06 2024-09-21T12:38:06.159 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.159 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.159 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.159 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.327031+0000", 2024-09-21T12:38:06.159 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3319999999999997e-05 2024-09-21T12:38:06.159 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.159 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.159 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op must wait for map", 2024-09-21T12:38:06.159 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.327037+0000", 2024-09-21T12:38:06.159 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9340000000000003e-06 2024-09-21T12:38:06.159 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.159 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.160 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.160 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.343795+0000", 2024-09-21T12:38:06.160 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.016757859999999999 2024-09-21T12:38:06.160 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.160 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.160 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:38:06.160 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:33.343805+0000", 2024-09-21T12:38:06.160 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6500000000000008e-06 2024-09-21T12:38:06.160 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.160 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.160 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.160 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:34.339875+0000", 2024-09-21T12:38:06.160 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99607032600000001 2024-09-21T12:38:06.160 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.160 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.160 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:38:06.161 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:34.340032+0000", 2024-09-21T12:38:06.161 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015660500000000001 2024-09-21T12:38:06.161 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.161 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.161 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:38:06.161 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:34:34.340850+0000", 2024-09-21T12:38:06.161 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00081799499999999996 2024-09-21T12:38:06.161 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.161 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:38:06.161 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.161 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.161 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.161 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.26147.0:17050 11.f 11:fa27f951:::benchmark_data_smithi114_187622_object17049:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-21T12:38:06.161 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:35:59.107044+0000", 2024-09-21T12:38:06.162 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 126.94131271800001, 2024-09-21T12:38:06.162 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.018961, 2024-09-21T12:38:06.162 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:38:06.162 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:38:06.162 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:38:06.162 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:38:06.162 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.26147", 2024-09-21T12:38:06.162 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:38:06.162 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 17050 2024-09-21T12:38:06.162 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.162 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:38:06.162 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.162 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:38:06.163 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.107044+0000", 2024-09-21T12:38:06.163 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.163 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.163 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.163 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:38:06.163 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.107044+0000", 2024-09-21T12:38:06.163 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.163 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.163 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.163 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:38:06.163 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.107046+0000", 2024-09-21T12:38:06.163 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3859999999999999e-06 2024-09-21T12:38:06.164 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.164 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.164 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:38:06.164 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.107861+0000", 2024-09-21T12:38:06.164 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000814933 2024-09-21T12:38:06.164 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.164 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.164 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:38:06.164 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.107874+0000", 2024-09-21T12:38:06.164 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3322999999999999e-05 2024-09-21T12:38:06.164 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.164 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.164 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:38:06.164 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.107881+0000", 2024-09-21T12:38:06.165 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3939999999999998e-06 2024-09-21T12:38:06.165 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.165 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.165 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.165 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.107945+0000", 2024-09-21T12:38:06.165 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3542999999999998e-05 2024-09-21T12:38:06.165 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.165 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.165 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:38:06.165 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.108080+0000", 2024-09-21T12:38:06.165 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000135449 2024-09-21T12:38:06.165 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.165 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.166 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,7", 2024-09-21T12:38:06.166 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.108213+0000", 2024-09-21T12:38:06.166 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000132726 2024-09-21T12:38:06.166 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.166 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.166 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-21T12:38:06.166 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.110137+0000", 2024-09-21T12:38:06.166 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001923754 2024-09-21T12:38:06.166 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.166 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.166 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:38:06.166 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.119226+0000", 2024-09-21T12:38:06.166 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.009089125 2024-09-21T12:38:06.166 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.167 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.167 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.167 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.130527+0000", 2024-09-21T12:38:06.167 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.011300737 2024-09-21T12:38:06.167 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.167 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.167 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:38:06.167 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.130536+0000", 2024-09-21T12:38:06.167 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6870000000000006e-06 2024-09-21T12:38:06.167 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.167 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.167 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.167 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:00.125990+0000", 2024-09-21T12:38:06.167 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99545339099999997 2024-09-21T12:38:06.168 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.168 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.168 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:38:06.168 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:00.126005+0000", 2024-09-21T12:38:06.168 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5551999999999999e-05 2024-09-21T12:38:06.168 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.168 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:38:06.168 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.168 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.168 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.168 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.26147.0:17071 11.f 11.cb223a2f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-09-21T12:38:06.168 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:35:59.137176+0000", 2024-09-21T12:38:06.168 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 126.911180632, 2024-09-21T12:38:06.168 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98887491900000002, 2024-09-21T12:38:06.169 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:38:06.169 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:38:06.169 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:38:06.169 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:38:06.169 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.26147", 2024-09-21T12:38:06.169 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:38:06.169 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 17071 2024-09-21T12:38:06.169 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.169 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:38:06.169 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.169 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:38:06.169 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.137176+0000", 2024-09-21T12:38:06.169 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.169 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.169 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.170 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:38:06.170 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.137176+0000", 2024-09-21T12:38:06.170 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.170 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.170 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.170 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:38:06.170 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.137177+0000", 2024-09-21T12:38:06.170 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7000000000000004e-07 2024-09-21T12:38:06.170 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.170 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.170 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:38:06.170 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.137202+0000", 2024-09-21T12:38:06.170 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.4893999999999999e-05 2024-09-21T12:38:06.170 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.171 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.171 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:38:06.171 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.137210+0000", 2024-09-21T12:38:06.171 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1280000000000008e-06 2024-09-21T12:38:06.171 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.171 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.171 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:38:06.171 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.137216+0000", 2024-09-21T12:38:06.171 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9200000000000001e-06 2024-09-21T12:38:06.171 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.171 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.171 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.171 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.137309+0000", 2024-09-21T12:38:06.171 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.3009000000000003e-05 2024-09-21T12:38:06.172 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.172 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.172 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:38:06.172 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.137321+0000", 2024-09-21T12:38:06.172 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2292e-05 2024-09-21T12:38:06.172 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.172 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.172 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.172 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:00.126041+0000", 2024-09-21T12:38:06.172 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98872015899999999 2024-09-21T12:38:06.172 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.172 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.172 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:38:06.172 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:00.126051+0000", 2024-09-21T12:38:06.173 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.8470000000000008e-06 2024-09-21T12:38:06.173 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.173 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:38:06.173 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.173 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.173 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.173 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.26147.0:17075 11.f 11.800d527f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-09-21T12:38:06.173 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:35:59.139143+0000", 2024-09-21T12:38:06.173 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 126.909214044, 2024-09-21T12:38:06.173 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98694552000000002, 2024-09-21T12:38:06.173 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:38:06.173 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:38:06.173 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:38:06.173 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:38:06.173 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.26147", 2024-09-21T12:38:06.173 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:38:06.174 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 17075 2024-09-21T12:38:06.174 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.174 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:38:06.174 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.174 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:38:06.174 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.139143+0000", 2024-09-21T12:38:06.174 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.174 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.174 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.174 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:38:06.174 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.139143+0000", 2024-09-21T12:38:06.174 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.174 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.174 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.175 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:38:06.175 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.139144+0000", 2024-09-21T12:38:06.175 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3090000000000001e-06 2024-09-21T12:38:06.175 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.175 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.175 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:38:06.175 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.139170+0000", 2024-09-21T12:38:06.175 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5925e-05 2024-09-21T12:38:06.175 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.175 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.175 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:38:06.175 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.139187+0000", 2024-09-21T12:38:06.175 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6441000000000001e-05 2024-09-21T12:38:06.175 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.175 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.176 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:38:06.176 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.139195+0000", 2024-09-21T12:38:06.176 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.4279999999999996e-06 2024-09-21T12:38:06.176 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.176 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.176 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.176 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.139257+0000", 2024-09-21T12:38:06.176 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2242999999999993e-05 2024-09-21T12:38:06.176 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.176 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.176 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:38:06.176 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.139268+0000", 2024-09-21T12:38:06.176 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0281e-05 2024-09-21T12:38:06.176 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.177 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.177 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.177 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:00.126078+0000", 2024-09-21T12:38:06.177 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98681080300000001 2024-09-21T12:38:06.177 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.177 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.177 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:38:06.177 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:00.126088+0000", 2024-09-21T12:38:06.177 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.009e-05 2024-09-21T12:38:06.177 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.177 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:38:06.177 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.177 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.177 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.177 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.26147.0:17076 11.f 11.e217396f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-09-21T12:38:06.178 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:35:59.139218+0000", 2024-09-21T12:38:06.178 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 126.90913899900001, 2024-09-21T12:38:06.178 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98692602799999996, 2024-09-21T12:38:06.178 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:38:06.178 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:38:06.178 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-21T12:38:06.178 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-21T12:38:06.178 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.26147", 2024-09-21T12:38:06.178 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:38:06.178 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 17076 2024-09-21T12:38:06.178 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.178 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:38:06.178 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.178 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:38:06.178 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.139218+0000", 2024-09-21T12:38:06.179 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.179 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.179 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.179 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:38:06.179 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.139218+0000", 2024-09-21T12:38:06.179 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:38:06.179 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.179 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.179 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:38:06.179 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.139219+0000", 2024-09-21T12:38:06.179 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2399999999999998e-07 2024-09-21T12:38:06.179 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.179 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.179 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:38:06.179 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.139239+0000", 2024-09-21T12:38:06.180 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0856000000000002e-05 2024-09-21T12:38:06.180 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.180 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.180 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:38:06.180 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.139248+0000", 2024-09-21T12:38:06.180 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9080000000000001e-06 2024-09-21T12:38:06.180 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.180 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.180 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:38:06.180 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.139254+0000", 2024-09-21T12:38:06.180 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1919999999999996e-06 2024-09-21T12:38:06.180 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.180 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.180 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.180 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.139294+0000", 2024-09-21T12:38:06.180 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0222000000000002e-05 2024-09-21T12:38:06.181 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.181 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.181 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-21T12:38:06.181 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:35:59.139300+0000", 2024-09-21T12:38:06.181 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5810000000000002e-06 2024-09-21T12:38:06.181 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.181 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.181 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:38:06.181 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:00.126118+0000", 2024-09-21T12:38:06.181 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98681790800000002 2024-09-21T12:38:06.181 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:38:06.181 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:38:06.181 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:38:06.181 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:36:00.126144+0000", 2024-09-21T12:38:06.181 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5737000000000001e-05 2024-09-21T12:38:06.181 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.182 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:38:06.182 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.182 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:38:06.182 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:38:06.182 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:38:06.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:06.183 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:06.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:06.284 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:06.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:06.314 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:06.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:06.414 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:06.435 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:06.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:06.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:06.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:06 smithi114 ceph-mon[40867]: 4.16 scrub ok 2024-09-21T12:38:06.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:06 smithi114 ceph-mon[40867]: 6.1b scrub starts 2024-09-21T12:38:06.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:06 smithi114 ceph-mon[40867]: 6.1b scrub ok 2024-09-21T12:38:06.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:06 smithi114 ceph-mon[40867]: pgmap v905: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 4.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.07k op/s; 19900/240819 objects degraded (8.263%); 10136/240819 objects misplaced (4.209%) 2024-09-21T12:38:06.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:06 smithi114 ceph-mon[40867]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T12:38:06.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:06 smithi114 ceph-mon[40867]: from='osd.2 [v2:172.21.15.114:6818/1359699248,v1:172.21.15.114:6819/1359699248]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-21T12:38:06.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:06 smithi114 ceph-mon[40867]: osdmap e454: 8 total, 7 up, 8 in 2024-09-21T12:38:06.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:06 smithi114 ceph-mon[40867]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-21T12:38:06.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:06 smithi114 ceph-mon[40867]: 3.15 scrub starts 2024-09-21T12:38:06.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:06 smithi114 ceph-mon[40867]: 3.15 scrub ok 2024-09-21T12:38:06.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:06 smithi114 ceph-mon[30324]: 4.16 scrub ok 2024-09-21T12:38:06.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:06 smithi114 ceph-mon[30324]: 6.1b scrub starts 2024-09-21T12:38:06.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:06 smithi114 ceph-mon[30324]: 6.1b scrub ok 2024-09-21T12:38:06.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:06 smithi114 ceph-mon[30324]: pgmap v905: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 4.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.07k op/s; 19900/240819 objects degraded (8.263%); 10136/240819 objects misplaced (4.209%) 2024-09-21T12:38:06.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:06 smithi114 ceph-mon[30324]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T12:38:06.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:06 smithi114 ceph-mon[30324]: from='osd.2 [v2:172.21.15.114:6818/1359699248,v1:172.21.15.114:6819/1359699248]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-21T12:38:06.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:06 smithi114 ceph-mon[30324]: osdmap e454: 8 total, 7 up, 8 in 2024-09-21T12:38:06.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:06 smithi114 ceph-mon[30324]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-21T12:38:06.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:06 smithi114 ceph-mon[30324]: 3.15 scrub starts 2024-09-21T12:38:06.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:06 smithi114 ceph-mon[30324]: 3.15 scrub ok 2024-09-21T12:38:06.537 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:06.558 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:06.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:06.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:06.659 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:06.681 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:06.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:06.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:06.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:06 smithi183 ceph-mon[33277]: 4.16 scrub ok 2024-09-21T12:38:06.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:06 smithi183 ceph-mon[33277]: 6.1b scrub starts 2024-09-21T12:38:06.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:06 smithi183 ceph-mon[33277]: 6.1b scrub ok 2024-09-21T12:38:06.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:06 smithi183 ceph-mon[33277]: pgmap v905: 178 pgs: 20 active+clean+remapped, 20 active+undersized, 13 active+undersized+degraded, 125 active+clean; 4.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.07k op/s; 19900/240819 objects degraded (8.263%); 10136/240819 objects misplaced (4.209%) 2024-09-21T12:38:06.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:06 smithi183 ceph-mon[33277]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T12:38:06.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:06 smithi183 ceph-mon[33277]: from='osd.2 [v2:172.21.15.114:6818/1359699248,v1:172.21.15.114:6819/1359699248]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-21T12:38:06.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:06 smithi183 ceph-mon[33277]: osdmap e454: 8 total, 7 up, 8 in 2024-09-21T12:38:06.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:06 smithi183 ceph-mon[33277]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-21T12:38:06.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:06 smithi183 ceph-mon[33277]: 3.15 scrub starts 2024-09-21T12:38:06.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:06 smithi183 ceph-mon[33277]: 3.15 scrub ok 2024-09-21T12:38:06.782 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:06.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:06.804 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:06.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:06.906 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:06.927 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:06.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:06.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:07.028 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:07.050 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:07.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:07.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:07.152 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:07.179 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:07.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:07.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:07.225 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:38:07.281 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:07.304 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:07.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:07.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:07.406 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:07.431 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:07.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:07.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:07.533 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:07.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:07.555 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:07.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:07.656 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:07.678 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:07.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:07.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:07.695 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:07 smithi183 ceph-mon[33277]: 3.10 scrub starts 2024-09-21T12:38:07.695 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:07 smithi183 ceph-mon[33277]: 3.10 scrub ok 2024-09-21T12:38:07.695 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:07 smithi183 ceph-mon[33277]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T12:38:07.695 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:07 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T12:38:07.695 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:07 smithi183 ceph-mon[33277]: osd.2 [v2:172.21.15.114:6818/1359699248,v1:172.21.15.114:6819/1359699248] boot 2024-09-21T12:38:07.695 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:07 smithi183 ceph-mon[33277]: osdmap e455: 8 total, 8 up, 8 in 2024-09-21T12:38:07.695 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:07 smithi183 ceph-mon[33277]: 3.11 scrub starts 2024-09-21T12:38:07.695 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:07 smithi183 ceph-mon[33277]: 3.11 scrub ok 2024-09-21T12:38:07.695 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:07 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 19900/240819 objects degraded (8.263%), 13 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T12:38:07.695 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:07 smithi183 ceph-mon[33277]: osdmap e456: 8 total, 8 up, 8 in 2024-09-21T12:38:07.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:07 smithi114 ceph-mon[30324]: 3.10 scrub starts 2024-09-21T12:38:07.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:07 smithi114 ceph-mon[30324]: 3.10 scrub ok 2024-09-21T12:38:07.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:07 smithi114 ceph-mon[30324]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T12:38:07.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:07 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T12:38:07.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:07 smithi114 ceph-mon[30324]: osd.2 [v2:172.21.15.114:6818/1359699248,v1:172.21.15.114:6819/1359699248] boot 2024-09-21T12:38:07.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:07 smithi114 ceph-mon[30324]: osdmap e455: 8 total, 8 up, 8 in 2024-09-21T12:38:07.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:07 smithi114 ceph-mon[30324]: 3.11 scrub starts 2024-09-21T12:38:07.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:07 smithi114 ceph-mon[30324]: 3.11 scrub ok 2024-09-21T12:38:07.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:07 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 19900/240819 objects degraded (8.263%), 13 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T12:38:07.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:07 smithi114 ceph-mon[30324]: osdmap e456: 8 total, 8 up, 8 in 2024-09-21T12:38:07.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:07 smithi114 ceph-mon[40867]: 3.10 scrub starts 2024-09-21T12:38:07.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:07 smithi114 ceph-mon[40867]: 3.10 scrub ok 2024-09-21T12:38:07.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:07 smithi114 ceph-mon[40867]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T12:38:07.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:07 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T12:38:07.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:07 smithi114 ceph-mon[40867]: osd.2 [v2:172.21.15.114:6818/1359699248,v1:172.21.15.114:6819/1359699248] boot 2024-09-21T12:38:07.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:07 smithi114 ceph-mon[40867]: osdmap e455: 8 total, 8 up, 8 in 2024-09-21T12:38:07.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:07 smithi114 ceph-mon[40867]: 3.11 scrub starts 2024-09-21T12:38:07.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:07 smithi114 ceph-mon[40867]: 3.11 scrub ok 2024-09-21T12:38:07.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:07 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 19900/240819 objects degraded (8.263%), 13 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T12:38:07.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:07 smithi114 ceph-mon[40867]: osdmap e456: 8 total, 8 up, 8 in 2024-09-21T12:38:07.780 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:07.803 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:07.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:07.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:07.905 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:07.927 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:07.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:07.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:07.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:38:07 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:38:08.029 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:08.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:08.054 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:08.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:08.137 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-21T12:38:08.155 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:08.179 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:08.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:08.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:08.281 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:08.309 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:08.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:08.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:08.412 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:08.447 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:08.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:08.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:08.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:08 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:38:08.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:08 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:38:08.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:08 smithi114 ceph-mon[40867]: 11.f starting backfill to osd.2 from (0'0,0'0] MAX to 454'12762 2024-09-21T12:38:08.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:08 smithi114 ceph-mon[40867]: 11.4 starting backfill to osd.2 from (0'0,0'0] MAX to 454'12612 2024-09-21T12:38:08.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:08 smithi114 ceph-mon[40867]: pgmap v909: 178 pgs: 1 unknown, 16 remapped+peering, 19 peering, 4 active+clean+remapped, 7 active+undersized, 6 active+undersized+degraded, 125 active+clean; 4.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s; 9864/229854 objects degraded (4.291%); 1/229854 objects misplaced (0.000%) 2024-09-21T12:38:08.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:08 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3771420591' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:38:08.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:08 smithi114 ceph-mon[40867]: 4.4 scrub starts 2024-09-21T12:38:08.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:08 smithi114 ceph-mon[40867]: 4.4 scrub ok 2024-09-21T12:38:08.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:08 smithi114 ceph-mon[40867]: 4.6 scrub starts 2024-09-21T12:38:08.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:08 smithi114 ceph-mon[40867]: 4.6 scrub ok 2024-09-21T12:38:08.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:08 smithi114 ceph-mon[40867]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:38:08.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:08 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3771420591' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:38:08.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:08 smithi114 ceph-mon[40867]: 11.c starting backfill to osd.2 from (0'0,0'0] MAX to 454'12893 2024-09-21T12:38:08.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:08 smithi114 ceph-mon[40867]: osdmap e457: 8 total, 8 up, 8 in 2024-09-21T12:38:08.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:08 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:38:08.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:08 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:38:08.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:08 smithi114 ceph-mon[30324]: 11.f starting backfill to osd.2 from (0'0,0'0] MAX to 454'12762 2024-09-21T12:38:08.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:08 smithi114 ceph-mon[30324]: 11.4 starting backfill to osd.2 from (0'0,0'0] MAX to 454'12612 2024-09-21T12:38:08.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:08 smithi114 ceph-mon[30324]: pgmap v909: 178 pgs: 1 unknown, 16 remapped+peering, 19 peering, 4 active+clean+remapped, 7 active+undersized, 6 active+undersized+degraded, 125 active+clean; 4.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s; 9864/229854 objects degraded (4.291%); 1/229854 objects misplaced (0.000%) 2024-09-21T12:38:08.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:08 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3771420591' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:38:08.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:08 smithi114 ceph-mon[30324]: 4.4 scrub starts 2024-09-21T12:38:08.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:08 smithi114 ceph-mon[30324]: 4.4 scrub ok 2024-09-21T12:38:08.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:08 smithi114 ceph-mon[30324]: 4.6 scrub starts 2024-09-21T12:38:08.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:08 smithi114 ceph-mon[30324]: 4.6 scrub ok 2024-09-21T12:38:08.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:08 smithi114 ceph-mon[30324]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:38:08.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:08 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3771420591' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:38:08.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:08 smithi114 ceph-mon[30324]: 11.c starting backfill to osd.2 from (0'0,0'0] MAX to 454'12893 2024-09-21T12:38:08.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:08 smithi114 ceph-mon[30324]: osdmap e457: 8 total, 8 up, 8 in 2024-09-21T12:38:08.548 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:08.579 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:08.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:08.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:08.680 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:08.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:08.702 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:08.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:08.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:08 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:38:08.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:08 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:38:08.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:08 smithi183 ceph-mon[33277]: 11.f starting backfill to osd.2 from (0'0,0'0] MAX to 454'12762 2024-09-21T12:38:08.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:08 smithi183 ceph-mon[33277]: 11.4 starting backfill to osd.2 from (0'0,0'0] MAX to 454'12612 2024-09-21T12:38:08.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:08 smithi183 ceph-mon[33277]: pgmap v909: 178 pgs: 1 unknown, 16 remapped+peering, 19 peering, 4 active+clean+remapped, 7 active+undersized, 6 active+undersized+degraded, 125 active+clean; 4.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s; 9864/229854 objects degraded (4.291%); 1/229854 objects misplaced (0.000%) 2024-09-21T12:38:08.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:08 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3771420591' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:38:08.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:08 smithi183 ceph-mon[33277]: 4.4 scrub starts 2024-09-21T12:38:08.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:08 smithi183 ceph-mon[33277]: 4.4 scrub ok 2024-09-21T12:38:08.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:08 smithi183 ceph-mon[33277]: 4.6 scrub starts 2024-09-21T12:38:08.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:08 smithi183 ceph-mon[33277]: 4.6 scrub ok 2024-09-21T12:38:08.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:08 smithi183 ceph-mon[33277]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:38:08.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:08 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3771420591' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:38:08.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:08 smithi183 ceph-mon[33277]: 11.c starting backfill to osd.2 from (0'0,0'0] MAX to 454'12893 2024-09-21T12:38:08.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:08 smithi183 ceph-mon[33277]: osdmap e457: 8 total, 8 up, 8 in 2024-09-21T12:38:08.803 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:08.827 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:08.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:08.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:08.929 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:08.952 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:08.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:08.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:09.054 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:09.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:09.088 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:09.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:09.189 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:09.219 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:09.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:09.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:09.320 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:09.353 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:09.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:09.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:09.455 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:09.485 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:09.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:09.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:09.587 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:09.618 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:09.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:09.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:09.720 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:09.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:09 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:38:09.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:09 smithi183 ceph-mon[33277]: 11.d starting backfill to osd.2 from (0'0,0'0] MAX to 454'12254 2024-09-21T12:38:09.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:09 smithi183 ceph-mon[33277]: 11.2 starting backfill to osd.2 from (0'0,0'0] MAX to 454'12570 2024-09-21T12:38:09.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:09 smithi183 ceph-mon[33277]: 11.6 starting backfill to osd.2 from (0'0,0'0] MAX to 454'12509 2024-09-21T12:38:09.742 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:09.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:09.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:09.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:09 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:38:09.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:09 smithi114 ceph-mon[30324]: 11.d starting backfill to osd.2 from (0'0,0'0] MAX to 454'12254 2024-09-21T12:38:09.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:09 smithi114 ceph-mon[30324]: 11.2 starting backfill to osd.2 from (0'0,0'0] MAX to 454'12570 2024-09-21T12:38:09.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:09 smithi114 ceph-mon[30324]: 11.6 starting backfill to osd.2 from (0'0,0'0] MAX to 454'12509 2024-09-21T12:38:09.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:09 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:38:09.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:09 smithi114 ceph-mon[40867]: 11.d starting backfill to osd.2 from (0'0,0'0] MAX to 454'12254 2024-09-21T12:38:09.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:09 smithi114 ceph-mon[40867]: 11.2 starting backfill to osd.2 from (0'0,0'0] MAX to 454'12570 2024-09-21T12:38:09.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:09 smithi114 ceph-mon[40867]: 11.6 starting backfill to osd.2 from (0'0,0'0] MAX to 454'12509 2024-09-21T12:38:09.843 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:09.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:09.874 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:09.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:09.975 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:10.005 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:10.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:10.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:10.107 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:10.137 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:10.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:10.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:10.239 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:10.267 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:10.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:10.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:10.369 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:10.417 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:10.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:10.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:10.432 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T12:38:10.452 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T12:38:10.519 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:10.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:10.616 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:10.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:10.717 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:10.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:10 smithi183 ceph-mon[33277]: pgmap v911: 178 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 unknown, 17 remapped+peering, 20 peering, 139 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s; 4630/223662 objects degraded (2.070%); 0 B/s, 0 objects/s recovering 2024-09-21T12:38:10.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:10.750 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:10.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:10.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:10 smithi114 ceph-mon[30324]: pgmap v911: 178 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 unknown, 17 remapped+peering, 20 peering, 139 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s; 4630/223662 objects degraded (2.070%); 0 B/s, 0 objects/s recovering 2024-09-21T12:38:10.777 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:10 smithi114 ceph-mon[40867]: pgmap v911: 178 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 unknown, 17 remapped+peering, 20 peering, 139 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s; 4630/223662 objects degraded (2.070%); 0 B/s, 0 objects/s recovering 2024-09-21T12:38:10.851 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:10.879 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:10.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:10.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:10.981 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:11.012 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:11.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:11.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:11.114 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:11.141 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:11.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:11.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:11.243 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:11.273 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:11.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:11.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:11.375 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:11.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:11.404 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:11.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:11.504 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:11.534 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:11.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:11.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:11.636 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:11.674 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:11.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:11.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:11.776 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:11.804 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:11.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:11.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:11.826 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:38:11.886 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:38:11.906 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:11.929 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:38:11.930 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:38:11.934 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:11.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:11.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:11.967 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:38:11.967 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:38:11.997 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:38:11.998 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-21T12:38:12.033 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:38:12.033 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:38:12.035 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:12.063 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:12.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:12.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:12.064 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:38:12.065 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:38:12.100 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:38:12.101 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:38:12.132 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:38:12.132 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:38:12.165 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:12.174 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:38:12.174 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:38:12.193 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:12.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:12.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:12.295 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:12.337 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:12.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:12.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:12.439 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:12.476 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:12.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:12.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:12.577 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:12.616 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:12.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:12.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:12.644 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:38:12 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:38:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:38:12.644 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:12 smithi114 ceph-mon[30324]: pgmap v912: 178 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 unknown, 17 remapped+peering, 20 peering, 139 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 879 B/s rd, 0 B/s wr, 983 op/s; 4630/223662 objects degraded (2.070%); 0 B/s, 0 objects/s recovering 2024-09-21T12:38:12.644 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:12 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2349855265' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:38:12.644 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:12 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 4630/223662 objects degraded (2.070%), 1 pg degraded (PG_DEGRADED) 2024-09-21T12:38:12.644 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:12 smithi114 ceph-mon[30324]: osdmap e458: 8 total, 8 up, 8 in 2024-09-21T12:38:12.645 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:12 smithi114 ceph-mon[40867]: pgmap v912: 178 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 unknown, 17 remapped+peering, 20 peering, 139 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 879 B/s rd, 0 B/s wr, 983 op/s; 4630/223662 objects degraded (2.070%); 0 B/s, 0 objects/s recovering 2024-09-21T12:38:12.645 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:12 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2349855265' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:38:12.645 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:12 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 4630/223662 objects degraded (2.070%), 1 pg degraded (PG_DEGRADED) 2024-09-21T12:38:12.645 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:12 smithi114 ceph-mon[40867]: osdmap e458: 8 total, 8 up, 8 in 2024-09-21T12:38:12.718 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:12.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:12 smithi183 ceph-mon[33277]: pgmap v912: 178 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 unknown, 17 remapped+peering, 20 peering, 139 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 879 B/s rd, 0 B/s wr, 983 op/s; 4630/223662 objects degraded (2.070%); 0 B/s, 0 objects/s recovering 2024-09-21T12:38:12.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:12 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2349855265' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:38:12.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:12 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 4630/223662 objects degraded (2.070%), 1 pg degraded (PG_DEGRADED) 2024-09-21T12:38:12.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:12 smithi183 ceph-mon[33277]: osdmap e458: 8 total, 8 up, 8 in 2024-09-21T12:38:12.758 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:12.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:12.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:12.859 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:12.891 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:12.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:12.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:12.993 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:13.025 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:13.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:13.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:13.127 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:13.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:13.163 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:13.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:13.246 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T12:38:13.263 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:13.335 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:13.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:13.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:13.437 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:13.463 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:13.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:13.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:13.565 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:13.591 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:13.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:13.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:13.693 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:13.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:13.722 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:13.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:13.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:13 smithi183 ceph-mon[33277]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:38:13.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:13 smithi114 ceph-mon[30324]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:38:13.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:13 smithi114 ceph-mon[40867]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T12:38:13.824 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:13.849 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:13.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:13.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:13.952 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:13.972 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:13.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:13.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:14.074 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:14.101 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:14.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:14.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:14.203 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:14.233 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:14.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:14.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:14.334 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:14.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:14.359 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:14.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:14.460 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:14.494 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:14.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:14.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:14.596 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:14.625 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:14.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:14.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:14.726 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:14.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:14 smithi183 ceph-mon[33277]: pgmap v914: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 172 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 850 B/s rd, 0 B/s wr, 1.08k op/s; 15999/210354 objects degraded (7.606%); 7892/210354 objects misplaced (3.752%); 27 MiB/s, 431 objects/s recovering 2024-09-21T12:38:14.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:14 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:38:14.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:14 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:38:14.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:14.761 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:14.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:14.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:14 smithi114 ceph-mon[40867]: pgmap v914: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 172 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 850 B/s rd, 0 B/s wr, 1.08k op/s; 15999/210354 objects degraded (7.606%); 7892/210354 objects misplaced (3.752%); 27 MiB/s, 431 objects/s recovering 2024-09-21T12:38:14.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:14 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:38:14.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:14 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:38:14.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:14 smithi114 ceph-mon[30324]: pgmap v914: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 172 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 850 B/s rd, 0 B/s wr, 1.08k op/s; 15999/210354 objects degraded (7.606%); 7892/210354 objects misplaced (3.752%); 27 MiB/s, 431 objects/s recovering 2024-09-21T12:38:14.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:14 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:38:14.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:14 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:38:14.862 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:14.866 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:38:14.866 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:38:14.890 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:14.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:14.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:14.991 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:15.023 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:15.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:15.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:15.125 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:15.158 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:15.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:15.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:15.260 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:15.296 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:15.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:15.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:15.397 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:15.436 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:15.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:15.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:15.471 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:38:15.538 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:15.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:15.572 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:15.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:15.673 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:15.694 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:15.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:15.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:15.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:15 smithi183 ceph-mon[33277]: osdmap e459: 8 total, 8 up, 8 in 2024-09-21T12:38:15.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:15 smithi114 ceph-mon[30324]: osdmap e459: 8 total, 8 up, 8 in 2024-09-21T12:38:15.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:15 smithi114 ceph-mon[40867]: osdmap e459: 8 total, 8 up, 8 in 2024-09-21T12:38:15.795 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:15.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:15.846 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:15.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:15.947 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:15.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:15.985 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:15.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:16.086 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:16.120 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:16.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:16.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:16.222 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:16.250 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:16.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:16.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:16.352 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:16.379 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:16.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:16.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:16.480 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:16.509 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:16.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:16.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:16.612 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:16.632 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:16.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:16.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:16.734 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:16.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:16 smithi183 ceph-mon[33277]: pgmap v916: 178 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 172 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 730 B/s rd, 0 B/s wr, 727 op/s; 15636/209265 objects degraded (7.472%); 7892/209265 objects misplaced (3.771%); 23 MiB/s, 370 objects/s recovering 2024-09-21T12:38:16.764 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:16.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:16.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:16.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:16 smithi114 ceph-mon[30324]: pgmap v916: 178 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 172 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 730 B/s rd, 0 B/s wr, 727 op/s; 15636/209265 objects degraded (7.472%); 7892/209265 objects misplaced (3.771%); 23 MiB/s, 370 objects/s recovering 2024-09-21T12:38:16.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:16 smithi114 ceph-mon[40867]: pgmap v916: 178 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 172 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 730 B/s rd, 0 B/s wr, 727 op/s; 15636/209265 objects degraded (7.472%); 7892/209265 objects misplaced (3.771%); 23 MiB/s, 370 objects/s recovering 2024-09-21T12:38:16.866 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:16.887 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:16.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:16.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:16.988 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:17.024 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:17.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:17.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:17.126 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:17.166 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:17.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:17.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:17.268 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:17.302 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:17.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:17.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:17.404 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:17.427 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:17.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:17.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:17.529 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:17.559 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:17.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:17.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:17.661 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:17.694 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:17.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:17.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:17.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:17 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 15636/209265 objects degraded (7.472%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T12:38:17.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:17 smithi183 ceph-mon[33277]: osdmap e460: 8 total, 8 up, 8 in 2024-09-21T12:38:17.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:38:17 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:38:17.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:17 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 15636/209265 objects degraded (7.472%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T12:38:17.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:17 smithi114 ceph-mon[30324]: osdmap e460: 8 total, 8 up, 8 in 2024-09-21T12:38:17.755 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:17 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 15636/209265 objects degraded (7.472%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T12:38:17.755 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:17 smithi114 ceph-mon[40867]: osdmap e460: 8 total, 8 up, 8 in 2024-09-21T12:38:17.796 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:17.842 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:17.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:17.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:17.930 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:38:17.930 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:38:17.944 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:18.000 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:18.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:18.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:18.102 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:18.133 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:18.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:18.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:18.235 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:18.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:18.271 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:18.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:18.372 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:18.409 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:18.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:18.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:18.511 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:18.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:18.556 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:18.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:18.573 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T12:38:18.657 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:18.710 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:18.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:18.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:18.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:18 smithi183 ceph-mon[33277]: pgmap v918: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 172 active+clean; 4.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.34k op/s; 13585/199617 objects degraded (6.806%); 7238/199617 objects misplaced (3.626%); 43 MiB/s, 684 objects/s recovering 2024-09-21T12:38:18.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:18 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:38:18.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:18 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:38:18.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:18 smithi114 ceph-mon[30324]: pgmap v918: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 172 active+clean; 4.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.34k op/s; 13585/199617 objects degraded (6.806%); 7238/199617 objects misplaced (3.626%); 43 MiB/s, 684 objects/s recovering 2024-09-21T12:38:18.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:18 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:38:18.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:18 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:38:18.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:18 smithi114 ceph-mon[40867]: pgmap v918: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 172 active+clean; 4.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.34k op/s; 13585/199617 objects degraded (6.806%); 7238/199617 objects misplaced (3.626%); 43 MiB/s, 684 objects/s recovering 2024-09-21T12:38:18.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:18 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:38:18.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:18 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:38:18.813 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:18.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:18.859 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:18.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:18.960 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:18.973 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:38:18.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:18.987 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:18.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:19.051 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:38:19.088 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:19.117 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:19.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:19.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:19.218 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:19.260 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:19.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:19.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:19.362 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:19.403 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:19.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:19.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:19.505 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:19.540 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:19.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:19.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:19.641 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:19.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:19.676 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:19.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:19.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:19 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:38:19.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:19 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:38:19.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:19 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:38:19.777 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:19.813 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:19.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:19.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:19.916 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:19.960 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:19.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:19.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:20.062 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:20.067 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T12:38:20.105 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:20.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:20.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:20.207 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:20.232 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:20.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:20.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:20.333 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:20.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:20.387 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:20.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:20.488 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:20.517 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:20.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:20.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:20.620 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:20.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:20.650 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:20.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:20.651 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:20 smithi114 ceph-mon[30324]: pgmap v919: 178 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 172 active+clean; 3.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 729 B/s rd, 0 B/s wr, 1.15k op/s; 12362/192405 objects degraded (6.425%); 7238/192405 objects misplaced (3.762%); 23 MiB/s, 361 objects/s recovering 2024-09-21T12:38:20.651 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:20 smithi114 ceph-mon[40867]: pgmap v919: 178 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 172 active+clean; 3.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 729 B/s rd, 0 B/s wr, 1.15k op/s; 12362/192405 objects degraded (6.425%); 7238/192405 objects misplaced (3.762%); 23 MiB/s, 361 objects/s recovering 2024-09-21T12:38:20.661 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:38:20.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:20 smithi183 ceph-mon[33277]: pgmap v919: 178 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 172 active+clean; 3.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 729 B/s rd, 0 B/s wr, 1.15k op/s; 12362/192405 objects degraded (6.425%); 7238/192405 objects misplaced (3.762%); 23 MiB/s, 361 objects/s recovering 2024-09-21T12:38:20.751 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:20.777 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:38:20.778 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:38:20.799 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:20.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:20.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:20.814 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:38:20.814 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:38:20.849 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:38:20.850 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:38:20.901 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:20.906 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:38:20.906 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:38:20.925 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:20.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:20.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:20.956 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:38:20.956 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:38:20.998 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:38:20.998 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:38:21.018 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:38:21.018 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:38:21.027 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:21.035 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:38:21.035 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:38:21.060 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:21.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:21.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:21.162 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:21.189 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:21.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:21.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:21.290 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:21.322 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:21.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:21.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:21.424 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:21.448 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:21.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:21.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:21.521 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-21T12:38:21.550 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:21.578 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:21.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:21.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:21.680 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:21.708 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:21.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:21.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:21.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:21 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/141036676' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:38:21.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:21 smithi183 ceph-mon[33277]: pgmap v920: 178 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 172 active+clean; 3.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 748 op/s; 12362/192405 objects degraded (6.425%); 7238/192405 objects misplaced (3.762%); 16 MiB/s, 261 objects/s recovering 2024-09-21T12:38:21.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:21 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2877391299' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:38:21.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:21 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:38:21.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:21 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/141036676' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:38:21.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:21 smithi114 ceph-mon[30324]: pgmap v920: 178 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 172 active+clean; 3.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 748 op/s; 12362/192405 objects degraded (6.425%); 7238/192405 objects misplaced (3.762%); 16 MiB/s, 261 objects/s recovering 2024-09-21T12:38:21.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:21 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2877391299' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:38:21.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:21 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:38:21.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:21 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/141036676' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:38:21.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:21 smithi114 ceph-mon[40867]: pgmap v920: 178 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 172 active+clean; 3.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 748 op/s; 12362/192405 objects degraded (6.425%); 7238/192405 objects misplaced (3.762%); 16 MiB/s, 261 objects/s recovering 2024-09-21T12:38:21.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:21 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2877391299' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:38:21.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:21 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:38:21.810 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:21.843 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:21.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:21.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:21.946 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:21.983 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:21.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:21.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:22.084 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:22.120 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:22.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:22.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:22.222 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:22.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:22.255 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:22.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:22.356 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:22.378 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:22.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:22.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:22.479 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:22.523 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:22.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:22.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:22.624 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:22.660 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:22.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:22.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:22.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:22 smithi183 ceph-mon[33277]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:38:22.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:22 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:38:22.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:22 smithi183 ceph-mon[33277]: osdmap e461: 8 total, 8 up, 8 in 2024-09-21T12:38:22.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:22 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 12362/192405 objects degraded (6.425%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T12:38:22.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:22 smithi183 ceph-mon[33277]: osdmap e462: 8 total, 8 up, 8 in 2024-09-21T12:38:22.762 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:22.774 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:38:22 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:38:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:38:22.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:22 smithi114 ceph-mon[40867]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:38:22.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:22 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:38:22.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:22 smithi114 ceph-mon[40867]: osdmap e461: 8 total, 8 up, 8 in 2024-09-21T12:38:22.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:22 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 12362/192405 objects degraded (6.425%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T12:38:22.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:22 smithi114 ceph-mon[40867]: osdmap e462: 8 total, 8 up, 8 in 2024-09-21T12:38:22.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:22 smithi114 ceph-mon[30324]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:38:22.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:22 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:38:22.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:22 smithi114 ceph-mon[30324]: osdmap e461: 8 total, 8 up, 8 in 2024-09-21T12:38:22.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:22 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 12362/192405 objects degraded (6.425%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T12:38:22.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:22 smithi114 ceph-mon[30324]: osdmap e462: 8 total, 8 up, 8 in 2024-09-21T12:38:22.782 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:22.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:22.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:22.884 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:22.916 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:22.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:22.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:23.017 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:23.054 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:23.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:23.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:23.156 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:23.180 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:23.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:23.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:23.282 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:23.315 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:23.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:23.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:23.417 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:23.445 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:23.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:23.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:23.547 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:23.575 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:23.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:23.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:23.677 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:23.703 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:23.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:23.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:23.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:23 smithi114 ceph-mon[30324]: pgmap v923: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 172 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 850 B/s rd, 0 B/s wr, 1.31k op/s; 9982/176025 objects degraded (5.671%); 6562/176025 objects misplaced (3.728%); 24 MiB/s, 377 objects/s recovering 2024-09-21T12:38:23.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:23 smithi114 ceph-mon[40867]: pgmap v923: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 172 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 850 B/s rd, 0 B/s wr, 1.31k op/s; 9982/176025 objects degraded (5.671%); 6562/176025 objects misplaced (3.728%); 24 MiB/s, 377 objects/s recovering 2024-09-21T12:38:23.805 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:23.840 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:23.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:23.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:23.942 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:23.970 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:23.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:23.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:23.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:23 smithi183 ceph-mon[33277]: pgmap v923: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 172 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 850 B/s rd, 0 B/s wr, 1.31k op/s; 9982/176025 objects degraded (5.671%); 6562/176025 objects misplaced (3.728%); 24 MiB/s, 377 objects/s recovering 2024-09-21T12:38:24.072 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:24.113 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:24.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:24.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:24.215 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:24.240 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:24.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:24.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:24.341 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:24.373 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:24.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:24.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:24.474 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:24.506 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:24.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:24.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:24.608 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:24.634 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:24.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:24.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:24.736 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:24.772 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:24.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:24.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:24.873 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:24.894 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:24.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:24.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:24.996 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:25.031 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:25.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:25.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:25.132 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:25.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:25.162 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:25.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:25.262 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:25.302 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:25.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:25.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:25.404 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:25.435 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:25.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:25.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:25.536 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:25.557 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:25.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:25.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:25.659 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:25.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:25.695 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:25.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:25.796 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:25.832 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:25.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:25.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:25.934 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:25.961 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:25.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:25.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:26.063 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:26.095 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:26.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:26.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:26.197 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:26.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:26.229 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:26.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:26.330 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:26.362 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:26.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:26.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:26.464 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:26.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:26 smithi183 ceph-mon[33277]: pgmap v924: 178 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 172 active+clean; 3.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.02k op/s; 9651/175032 objects degraded (5.514%); 6562/175032 objects misplaced (3.749%); 18 MiB/s, 284 objects/s recovering 2024-09-21T12:38:26.495 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:26.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:26.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:26.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:26 smithi114 ceph-mon[40867]: pgmap v924: 178 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 172 active+clean; 3.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.02k op/s; 9651/175032 objects degraded (5.514%); 6562/175032 objects misplaced (3.749%); 18 MiB/s, 284 objects/s recovering 2024-09-21T12:38:26.502 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:26 smithi114 ceph-mon[30324]: pgmap v924: 178 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 172 active+clean; 3.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.02k op/s; 9651/175032 objects degraded (5.514%); 6562/175032 objects misplaced (3.749%); 18 MiB/s, 284 objects/s recovering 2024-09-21T12:38:26.597 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:26.626 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:26.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:26.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:26.728 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:26.769 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:26.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:26.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:26.870 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:26.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:26.919 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:26.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:27.020 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:27.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:27.058 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:27.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:27.159 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:27.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:27.210 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:27.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:27.311 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:27.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:27.346 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:27.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:27.447 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:27.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:27 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 9651/175032 objects degraded (5.514%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T12:38:27.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:27.489 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:27.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:27.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:27 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 9651/175032 objects degraded (5.514%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T12:38:27.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:27 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 9651/175032 objects degraded (5.514%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T12:38:27.591 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:27.618 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:27.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:27.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:27.720 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:27.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:27.758 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:27.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:27.834 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:38:27.860 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:27.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:27.907 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:27.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:27.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:38:27 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:38:28.008 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:28.039 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:28.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:28.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:28.141 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:28.171 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:28.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:28.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:28.235 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T12:38:28.273 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:28.303 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:28.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:28.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:28.404 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:28.439 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:28.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:28.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:28.498 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:28 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:38:28.498 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:28 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:38:28.498 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:28 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:38:28.498 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:28 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:38:28.498 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:28 smithi114 ceph-mon[30324]: pgmap v925: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 172 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 8099/165900 objects degraded (4.882%); 5952/165900 objects misplaced (3.588%); 23 MiB/s, 367 objects/s recovering 2024-09-21T12:38:28.498 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:28 smithi114 ceph-mon[30324]: osdmap e463: 8 total, 8 up, 8 in 2024-09-21T12:38:28.498 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:28 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/797389877' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:38:28.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:28 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:38:28.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:28 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:38:28.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:28 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:38:28.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:28 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:38:28.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:28 smithi114 ceph-mon[40867]: pgmap v925: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 172 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 8099/165900 objects degraded (4.882%); 5952/165900 objects misplaced (3.588%); 23 MiB/s, 367 objects/s recovering 2024-09-21T12:38:28.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:28 smithi114 ceph-mon[40867]: osdmap e463: 8 total, 8 up, 8 in 2024-09-21T12:38:28.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:28 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/797389877' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:38:28.540 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:28.558 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:38:28.567 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:28.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:28.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:28.669 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:28.678 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:38:28.678 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-21T12:38:28.678 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-21T12:38:28.678 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-21T12:38:28.678 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:38:28.718 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:28.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:28.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:28.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:28 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:38:28.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:28 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:38:28.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:28 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:38:28.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:28 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:38:28.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:28 smithi183 ceph-mon[33277]: pgmap v925: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 172 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 8099/165900 objects degraded (4.882%); 5952/165900 objects misplaced (3.588%); 23 MiB/s, 367 objects/s recovering 2024-09-21T12:38:28.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:28 smithi183 ceph-mon[33277]: osdmap e463: 8 total, 8 up, 8 in 2024-09-21T12:38:28.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:28 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/797389877' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:38:28.820 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:28.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:28.860 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:28.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:28.961 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:28.993 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:28.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:28.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:29.095 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:29.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:29.131 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:29.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:29.232 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:29.270 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:29.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:29.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:29.371 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:29.399 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:29.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:29.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:29.448 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T12:38:29.501 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:29.512 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:29 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:38:29.512 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:29 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/797389877' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T12:38:29.512 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:29 smithi114 ceph-mon[30324]: osdmap e464: 8 total, 8 up, 8 in 2024-09-21T12:38:29.513 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:29 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:38:29.513 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:29 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/797389877' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T12:38:29.513 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:29 smithi114 ceph-mon[40867]: osdmap e464: 8 total, 8 up, 8 in 2024-09-21T12:38:29.530 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:29.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:29.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:29.632 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:29.661 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:29.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:29.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:29 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:38:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:29 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/797389877' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T12:38:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:29 smithi183 ceph-mon[33277]: osdmap e464: 8 total, 8 up, 8 in 2024-09-21T12:38:29.762 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:29.764 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T12:38:29.789 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:29.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:29.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:29.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:29.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:29.968 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:29.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:29.983 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T12:38:30.040 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:38:30.041 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:38:30.069 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:30.154 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:30.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:30.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:30.257 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:30.288 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:30.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:30.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:30.365 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:30 smithi114 ceph-mon[40867]: pgmap v928: 178 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 172 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 729 B/s rd, 0 B/s wr, 1.13k op/s; 7138/158679 objects degraded (4.498%); 5952/158679 objects misplaced (3.751%); 19 MiB/s, 305 objects/s recovering 2024-09-21T12:38:30.366 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:30 smithi114 ceph-mon[30324]: pgmap v928: 178 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 172 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 729 B/s rd, 0 B/s wr, 1.13k op/s; 7138/158679 objects degraded (4.498%); 5952/158679 objects misplaced (3.751%); 19 MiB/s, 305 objects/s recovering 2024-09-21T12:38:30.389 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:30.413 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:30.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:30.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:30.515 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:30.548 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:30.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:30.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:30.649 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:30.683 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:30.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:30.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:30.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:30 smithi183 ceph-mon[33277]: pgmap v928: 178 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 172 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 729 B/s rd, 0 B/s wr, 1.13k op/s; 7138/158679 objects degraded (4.498%); 5952/158679 objects misplaced (3.751%); 19 MiB/s, 305 objects/s recovering 2024-09-21T12:38:30.784 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:30.805 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:30.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:30.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:30.906 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:30.934 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:30.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:30.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:31.036 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:31.066 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:31.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:31.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:31.168 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:31.190 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:31.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:31.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:31.284 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T12:38:31.291 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:31.313 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:31.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:31.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:31.414 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:31.437 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:31.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:31.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:31.539 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:31.559 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:31 smithi114 ceph-mon[30324]: osdmap e465: 8 total, 8 up, 8 in 2024-09-21T12:38:31.559 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:31 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3033646090' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:38:31.559 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:31 smithi114 ceph-mon[40867]: osdmap e465: 8 total, 8 up, 8 in 2024-09-21T12:38:31.559 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:31 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3033646090' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:38:31.585 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:31.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:31.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:31.687 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:31.713 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:31.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:31.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:31.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:31 smithi183 ceph-mon[33277]: osdmap e465: 8 total, 8 up, 8 in 2024-09-21T12:38:31.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:31 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3033646090' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:38:31.815 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:31.840 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:31.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:31.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:31.942 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:31.971 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:31.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:31.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:32.072 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:32.099 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:32.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:32.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:32.201 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:32.222 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:32.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:32.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:32.323 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:32.352 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:32.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:32.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:32.454 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:32.459 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:38:32.459 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:38:32.459 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:38:32.459 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:38:32.459 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:38:32.459 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:38:32.459 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:38:32.459 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:38:32.460 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:38:32.460 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:38:32.460 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:38:32.460 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:38:32.460 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:38:32.460 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:38:32.460 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:38:32.460 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:38:32.480 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:32.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:32.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:32.552 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:32 smithi114 ceph-mon[30324]: pgmap v930: 178 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 172 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 908 op/s; 7138/158679 objects degraded (4.498%); 5952/158679 objects misplaced (3.751%); 18 MiB/s, 281 objects/s recovering 2024-09-21T12:38:32.552 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:32 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3033646090' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T12:38:32.552 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:32 smithi114 ceph-mon[30324]: osdmap e466: 8 total, 8 up, 8 in 2024-09-21T12:38:32.552 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:32 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2573602479' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:38:32.552 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:32 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 7138/158679 objects degraded (4.498%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T12:38:32.552 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:32 smithi114 ceph-mon[40867]: pgmap v930: 178 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 172 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 908 op/s; 7138/158679 objects degraded (4.498%); 5952/158679 objects misplaced (3.751%); 18 MiB/s, 281 objects/s recovering 2024-09-21T12:38:32.552 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:32 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3033646090' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T12:38:32.553 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:32 smithi114 ceph-mon[40867]: osdmap e466: 8 total, 8 up, 8 in 2024-09-21T12:38:32.553 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:32 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2573602479' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:38:32.553 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:32 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 7138/158679 objects degraded (4.498%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T12:38:32.581 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:32.614 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:32.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:32.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:32.715 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:32.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:32 smithi183 ceph-mon[33277]: pgmap v930: 178 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 172 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 908 op/s; 7138/158679 objects degraded (4.498%); 5952/158679 objects misplaced (3.751%); 18 MiB/s, 281 objects/s recovering 2024-09-21T12:38:32.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:32 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3033646090' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T12:38:32.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:32 smithi183 ceph-mon[33277]: osdmap e466: 8 total, 8 up, 8 in 2024-09-21T12:38:32.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:32 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2573602479' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:38:32.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:32 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 7138/158679 objects degraded (4.498%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T12:38:32.744 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:32.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:32.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:32.846 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:32.885 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:32.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:32.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:32.986 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:33.015 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:33.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:33.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:33.024 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:38:32 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:38:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:38:33.117 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:33.145 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:33.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:33.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:33.247 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:33.268 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:33.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:33.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:33.369 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:33.395 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:33.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:33.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:33.497 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:33.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:33 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:38:33.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:33 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:38:33.535 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:33.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:33.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:33.637 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:33.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:33.676 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:33.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:33.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:33 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:38:33.776 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:33.801 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:33.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:33.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:33.903 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:33.931 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:33.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:33.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:33.987 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:38:33.993 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:38:34.033 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:34.063 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:34.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:34.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:34.164 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:34.204 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:34.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:34.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:34.306 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:34.340 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:34.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:34.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:34.441 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:34.477 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:34.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:34.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:34.575 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:34 smithi114 ceph-mon[30324]: pgmap v932: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 2.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 863 B/s rd, 0 B/s wr, 1.19k op/s; 4968/144732 objects degraded (3.433%); 5356/144732 objects misplaced (3.701%); 24 MiB/s, 387 objects/s recovering 2024-09-21T12:38:34.576 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:34 smithi114 ceph-mon[40867]: pgmap v932: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 2.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 863 B/s rd, 0 B/s wr, 1.19k op/s; 4968/144732 objects degraded (3.433%); 5356/144732 objects misplaced (3.701%); 24 MiB/s, 387 objects/s recovering 2024-09-21T12:38:34.576 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T12:38:34.578 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:34.619 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:34.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:34.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:34.720 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:34.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:34 smithi183 ceph-mon[33277]: pgmap v932: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 2.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 863 B/s rd, 0 B/s wr, 1.19k op/s; 4968/144732 objects degraded (3.433%); 5356/144732 objects misplaced (3.701%); 24 MiB/s, 387 objects/s recovering 2024-09-21T12:38:34.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:34.763 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:34.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:34.864 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:34.914 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:34.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:34.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:35.016 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:35.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:35.062 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:35.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:35.162 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:35.192 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:35.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:35.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:35.294 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:35.321 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:35.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:35.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:35.423 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:35.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:35.454 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:35.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:35.554 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:35.594 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:35.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:35.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:35.695 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:35.749 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:35.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:35.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:35.850 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:35.877 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:38:35.877 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:38:35.877 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:35.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:35.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:35.979 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:36.006 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:36.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:36.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:36.108 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:36.134 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:36.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:36.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:36.236 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:36.274 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:36.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:36.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:36.375 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:36.404 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:38:36.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:36.416 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:36.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:36.517 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:36.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:36.567 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:36.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:36.668 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:36.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:36.710 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:36.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:36.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:36 smithi183 ceph-mon[33277]: pgmap v933: 178 pgs: 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 2.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 738 B/s rd, 0 B/s wr, 766 op/s; 3923/143805 objects degraded (2.728%); 5356/143805 objects misplaced (3.724%); 24 MiB/s, 379 objects/s recovering 2024-09-21T12:38:36.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:36 smithi114 ceph-mon[30324]: pgmap v933: 178 pgs: 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 2.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 738 B/s rd, 0 B/s wr, 766 op/s; 3923/143805 objects degraded (2.728%); 5356/143805 objects misplaced (3.724%); 24 MiB/s, 379 objects/s recovering 2024-09-21T12:38:36.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:36 smithi114 ceph-mon[40867]: pgmap v933: 178 pgs: 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 2.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 738 B/s rd, 0 B/s wr, 766 op/s; 3923/143805 objects degraded (2.728%); 5356/143805 objects misplaced (3.724%); 24 MiB/s, 379 objects/s recovering 2024-09-21T12:38:36.812 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:36.844 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:36.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:36.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:36.946 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:36.998 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T12:38:37.000 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:37.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:37.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:37.041 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:38:37.102 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:37.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:37.161 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:37.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:37.262 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:37.315 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-21T12:38:37.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:37.319 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:37.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:37.345 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:38:37.420 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:37.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:37.469 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:37.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:37.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:37 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2632790869' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:38:37.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:37 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:38:37.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:37 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 3923/143805 objects degraded (2.728%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T12:38:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:37 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2632790869' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:38:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:37 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:38:37.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:37 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 3923/143805 objects degraded (2.728%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T12:38:37.571 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:37.607 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:37.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:37.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:37.709 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:37.726 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:37 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2632790869' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:38:37.727 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:37 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:38:37.727 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:37 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 3923/143805 objects degraded (2.728%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T12:38:37.738 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:37.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:37.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:37.840 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:37.866 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:37.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:37.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:37.967 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:37.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:38:37 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:38:38.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:38.010 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:38.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:38.111 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:38.145 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:38.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:38.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:38.247 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:38.278 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:38.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:38.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:38.332 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:38:38.332 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:38:38.346 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:38 smithi183 ceph-mon[33277]: pgmap v934: 178 pgs: 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 2.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s; 3388/134091 objects degraded (2.527%); 4706/134091 objects misplaced (3.510%); 24 MiB/s, 382 objects/s recovering 2024-09-21T12:38:38.346 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:38 smithi183 ceph-mon[33277]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:38:38.346 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:38 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:38:38.346 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:38 smithi183 ceph-mon[33277]: osdmap e467: 8 total, 8 up, 8 in 2024-09-21T12:38:38.346 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:38 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2911660006' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:38:38.355 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T12:38:38.380 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:38.428 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:38.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:38.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:38.530 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:38.562 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:38.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:38.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:38.664 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:38.697 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:38.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:38.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:38.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:38 smithi114 ceph-mon[40867]: pgmap v934: 178 pgs: 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 2.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s; 3388/134091 objects degraded (2.527%); 4706/134091 objects misplaced (3.510%); 24 MiB/s, 382 objects/s recovering 2024-09-21T12:38:38.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:38 smithi114 ceph-mon[40867]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:38:38.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:38 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:38:38.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:38 smithi114 ceph-mon[40867]: osdmap e467: 8 total, 8 up, 8 in 2024-09-21T12:38:38.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:38 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2911660006' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:38:38.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:38 smithi114 ceph-mon[30324]: pgmap v934: 178 pgs: 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 2.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s; 3388/134091 objects degraded (2.527%); 4706/134091 objects misplaced (3.510%); 24 MiB/s, 382 objects/s recovering 2024-09-21T12:38:38.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:38 smithi114 ceph-mon[30324]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:38:38.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:38 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:38:38.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:38 smithi114 ceph-mon[30324]: osdmap e467: 8 total, 8 up, 8 in 2024-09-21T12:38:38.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:38 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2911660006' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:38:38.799 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:38.827 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:38.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:38.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:38.929 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:38.958 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:38.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:38.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:39.060 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:39.094 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:39.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:39.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:39.195 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:39.218 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:39.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:39.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:39.319 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:39.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:39.357 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:39.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:39.458 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:39.494 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:39.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:39.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:39.596 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:39.631 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:39.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:39.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:39.733 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:39.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:39 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:38:39.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:39 smithi183 ceph-mon[33277]: 5.f scrub starts 2024-09-21T12:38:39.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:39 smithi183 ceph-mon[33277]: 5.f scrub ok 2024-09-21T12:38:39.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:39 smithi183 ceph-mon[33277]: 5.a scrub starts 2024-09-21T12:38:39.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:39 smithi183 ceph-mon[33277]: 5.a scrub ok 2024-09-21T12:38:39.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:39 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2911660006' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T12:38:39.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:39 smithi183 ceph-mon[33277]: osdmap e468: 8 total, 8 up, 8 in 2024-09-21T12:38:39.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:39 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:38:39.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:39 smithi114 ceph-mon[40867]: 5.f scrub starts 2024-09-21T12:38:39.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:39 smithi114 ceph-mon[40867]: 5.f scrub ok 2024-09-21T12:38:39.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:39 smithi114 ceph-mon[40867]: 5.a scrub starts 2024-09-21T12:38:39.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:39 smithi114 ceph-mon[40867]: 5.a scrub ok 2024-09-21T12:38:39.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:39 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2911660006' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T12:38:39.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:39 smithi114 ceph-mon[40867]: osdmap e468: 8 total, 8 up, 8 in 2024-09-21T12:38:39.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:39 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:38:39.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:39 smithi114 ceph-mon[30324]: 5.f scrub starts 2024-09-21T12:38:39.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:39 smithi114 ceph-mon[30324]: 5.f scrub ok 2024-09-21T12:38:39.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:39 smithi114 ceph-mon[30324]: 5.a scrub starts 2024-09-21T12:38:39.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:39 smithi114 ceph-mon[30324]: 5.a scrub ok 2024-09-21T12:38:39.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:39 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2911660006' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T12:38:39.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:39 smithi114 ceph-mon[30324]: osdmap e468: 8 total, 8 up, 8 in 2024-09-21T12:38:39.776 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:39.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:39.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:39.878 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:39.909 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:39.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:39.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:40.010 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:40.039 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:40.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:40.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:40.141 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:40.183 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:40.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:40.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:40.285 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:40.317 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:40.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:40.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:40.418 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:40.438 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:40.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:40.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:40.468 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T12:38:40.540 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:40.574 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:40.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:40.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:40.677 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:40.697 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:40.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:40.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:40.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:40 smithi183 ceph-mon[33277]: 4.15 scrub starts 2024-09-21T12:38:40.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:40 smithi183 ceph-mon[33277]: 4.15 scrub ok 2024-09-21T12:38:40.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:40 smithi183 ceph-mon[33277]: pgmap v937: 178 pgs: 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 2.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.38k op/s; 1956/126159 objects degraded (1.550%); 4706/126159 objects misplaced (3.730%); 33 MiB/s, 531 objects/s recovering 2024-09-21T12:38:40.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:40 smithi183 ceph-mon[33277]: 4.f scrub starts 2024-09-21T12:38:40.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:40 smithi183 ceph-mon[33277]: 4.f scrub ok 2024-09-21T12:38:40.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:40 smithi114 ceph-mon[40867]: 4.15 scrub starts 2024-09-21T12:38:40.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:40 smithi114 ceph-mon[40867]: 4.15 scrub ok 2024-09-21T12:38:40.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:40 smithi114 ceph-mon[40867]: pgmap v937: 178 pgs: 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 2.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.38k op/s; 1956/126159 objects degraded (1.550%); 4706/126159 objects misplaced (3.730%); 33 MiB/s, 531 objects/s recovering 2024-09-21T12:38:40.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:40 smithi114 ceph-mon[40867]: 4.f scrub starts 2024-09-21T12:38:40.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:40 smithi114 ceph-mon[40867]: 4.f scrub ok 2024-09-21T12:38:40.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:40 smithi114 ceph-mon[30324]: 4.15 scrub starts 2024-09-21T12:38:40.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:40 smithi114 ceph-mon[30324]: 4.15 scrub ok 2024-09-21T12:38:40.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:40 smithi114 ceph-mon[30324]: pgmap v937: 178 pgs: 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 2.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.38k op/s; 1956/126159 objects degraded (1.550%); 4706/126159 objects misplaced (3.730%); 33 MiB/s, 531 objects/s recovering 2024-09-21T12:38:40.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:40 smithi114 ceph-mon[30324]: 4.f scrub starts 2024-09-21T12:38:40.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:40 smithi114 ceph-mon[30324]: 4.f scrub ok 2024-09-21T12:38:40.798 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:40.822 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:40.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:40.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:40.924 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:40.949 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:40.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:40.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:40.978 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T12:38:41.051 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:41.075 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:41.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:41.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:41.177 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:41.200 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:41.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:41.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:41.257 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:38:41.302 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:41.326 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:41.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:41.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:41.428 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:41.455 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:41.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:41.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:41.557 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:41.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:41.590 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:41.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:41.666 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:41 smithi183 ceph-mon[33277]: osdmap e469: 8 total, 8 up, 8 in 2024-09-21T12:38:41.667 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:41 smithi183 ceph-mon[33277]: 4.19 scrub starts 2024-09-21T12:38:41.667 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:41 smithi183 ceph-mon[33277]: 4.19 scrub ok 2024-09-21T12:38:41.691 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:41.715 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:41.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:41.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:41.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:41 smithi114 ceph-mon[40867]: osdmap e469: 8 total, 8 up, 8 in 2024-09-21T12:38:41.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:41 smithi114 ceph-mon[40867]: 4.19 scrub starts 2024-09-21T12:38:41.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:41 smithi114 ceph-mon[40867]: 4.19 scrub ok 2024-09-21T12:38:41.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:41 smithi114 ceph-mon[30324]: osdmap e469: 8 total, 8 up, 8 in 2024-09-21T12:38:41.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:41 smithi114 ceph-mon[30324]: 4.19 scrub starts 2024-09-21T12:38:41.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:41 smithi114 ceph-mon[30324]: 4.19 scrub ok 2024-09-21T12:38:41.816 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:41.841 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:41.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:41.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:41.943 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:41.967 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:41.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:41.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:42.068 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:42.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:42.094 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:42.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:42.194 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:42.219 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:42.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:42.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:42.321 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:42.341 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:42.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:42.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:42.343 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:38:42.343 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:38:42.443 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:42.475 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:42.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:42.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:42.577 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:42.609 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:42.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:42.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:42.650 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:42 smithi183 ceph-mon[33277]: 1.b scrub starts 2024-09-21T12:38:42.650 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:42 smithi183 ceph-mon[33277]: 1.b scrub ok 2024-09-21T12:38:42.650 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:42 smithi183 ceph-mon[33277]: 1.f scrub starts 2024-09-21T12:38:42.650 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:42 smithi183 ceph-mon[33277]: 1.f scrub ok 2024-09-21T12:38:42.650 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:42 smithi183 ceph-mon[33277]: pgmap v939: 178 pgs: 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 2.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 980 op/s; 1956/126159 objects degraded (1.550%); 4706/126159 objects misplaced (3.730%); 16 MiB/s, 260 objects/s recovering 2024-09-21T12:38:42.650 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:42 smithi183 ceph-mon[33277]: 3.1a scrub starts 2024-09-21T12:38:42.651 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:42 smithi183 ceph-mon[33277]: 3.1a scrub ok 2024-09-21T12:38:42.651 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:42 smithi183 ceph-mon[33277]: osdmap e470: 8 total, 8 up, 8 in 2024-09-21T12:38:42.651 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:42 smithi183 ceph-mon[33277]: Health check update: Degraded data redundancy: 1956/126159 objects degraded (1.550%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T12:38:42.711 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:42.739 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:42.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:42.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:42.774 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:38:42 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:38:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:38:42.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:42 smithi114 ceph-mon[30324]: 1.b scrub starts 2024-09-21T12:38:42.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:42 smithi114 ceph-mon[30324]: 1.b scrub ok 2024-09-21T12:38:42.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:42 smithi114 ceph-mon[30324]: 1.f scrub starts 2024-09-21T12:38:42.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:42 smithi114 ceph-mon[30324]: 1.f scrub ok 2024-09-21T12:38:42.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:42 smithi114 ceph-mon[30324]: pgmap v939: 178 pgs: 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 2.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 980 op/s; 1956/126159 objects degraded (1.550%); 4706/126159 objects misplaced (3.730%); 16 MiB/s, 260 objects/s recovering 2024-09-21T12:38:42.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:42 smithi114 ceph-mon[30324]: 3.1a scrub starts 2024-09-21T12:38:42.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:42 smithi114 ceph-mon[30324]: 3.1a scrub ok 2024-09-21T12:38:42.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:42 smithi114 ceph-mon[30324]: osdmap e470: 8 total, 8 up, 8 in 2024-09-21T12:38:42.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:42 smithi114 ceph-mon[30324]: Health check update: Degraded data redundancy: 1956/126159 objects degraded (1.550%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T12:38:42.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:42 smithi114 ceph-mon[40867]: 1.b scrub starts 2024-09-21T12:38:42.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:42 smithi114 ceph-mon[40867]: 1.b scrub ok 2024-09-21T12:38:42.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:42 smithi114 ceph-mon[40867]: 1.f scrub starts 2024-09-21T12:38:42.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:42 smithi114 ceph-mon[40867]: 1.f scrub ok 2024-09-21T12:38:42.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:42 smithi114 ceph-mon[40867]: pgmap v939: 178 pgs: 1 active+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 173 active+clean; 2.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 980 op/s; 1956/126159 objects degraded (1.550%); 4706/126159 objects misplaced (3.730%); 16 MiB/s, 260 objects/s recovering 2024-09-21T12:38:42.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:42 smithi114 ceph-mon[40867]: 3.1a scrub starts 2024-09-21T12:38:42.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:42 smithi114 ceph-mon[40867]: 3.1a scrub ok 2024-09-21T12:38:42.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:42 smithi114 ceph-mon[40867]: osdmap e470: 8 total, 8 up, 8 in 2024-09-21T12:38:42.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:42 smithi114 ceph-mon[40867]: Health check update: Degraded data redundancy: 1956/126159 objects degraded (1.550%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T12:38:42.841 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:42.880 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:42.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:42.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:42.982 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:43.014 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T12:38:43.015 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:43.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:43.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:43.117 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:43.153 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:43.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:43.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:43.254 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:43.285 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:43.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:43.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:43.387 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:43.412 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:38:43.422 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:43.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:43.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:43.524 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:43.563 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:43.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:43.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:43.664 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:43.685 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:43.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:43.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:43.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:43 smithi183 ceph-mon[33277]: 4.1f scrub starts 2024-09-21T12:38:43.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:43 smithi183 ceph-mon[33277]: 4.1f scrub ok 2024-09-21T12:38:43.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:43 smithi183 ceph-mon[33277]: osdmap e471: 8 total, 8 up, 8 in 2024-09-21T12:38:43.745 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:43 smithi114 ceph-mon[30324]: 4.1f scrub starts 2024-09-21T12:38:43.745 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:43 smithi114 ceph-mon[30324]: 4.1f scrub ok 2024-09-21T12:38:43.745 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:43 smithi114 ceph-mon[30324]: osdmap e471: 8 total, 8 up, 8 in 2024-09-21T12:38:43.746 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:43 smithi114 ceph-mon[40867]: 4.1f scrub starts 2024-09-21T12:38:43.746 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:43 smithi114 ceph-mon[40867]: 4.1f scrub ok 2024-09-21T12:38:43.746 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:43 smithi114 ceph-mon[40867]: osdmap e471: 8 total, 8 up, 8 in 2024-09-21T12:38:43.787 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:43.795 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T12:38:43.804 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:38:43.820 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:43.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:43.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:43.922 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:43.947 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:43.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:43.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:44.049 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:44.071 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:44.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:44.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:44.172 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:44.196 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:44.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:44.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:44.298 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:44.320 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:44.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:44.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:44.422 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:44.448 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:44.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:44.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:44.549 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:44.578 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:44.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:44.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:44.680 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:44.709 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:44.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:44.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:44.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:44 smithi183 ceph-mon[33277]: pgmap v942: 178 pgs: 2 active+remapped+backfilling, 1 peering, 1 active+undersized+degraded+remapped+backfilling, 174 active+clean; 2.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 898 op/s; 1320/114276 objects degraded (1.155%); 2740/114276 objects misplaced (2.398%); 35 MiB/s, 565 objects/s recovering 2024-09-21T12:38:44.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:44 smithi183 ceph-mon[33277]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-21T12:38:44.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:44 smithi183 ceph-mon[33277]: osdmap e472: 8 total, 8 up, 8 in 2024-09-21T12:38:44.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:44 smithi114 ceph-mon[30324]: pgmap v942: 178 pgs: 2 active+remapped+backfilling, 1 peering, 1 active+undersized+degraded+remapped+backfilling, 174 active+clean; 2.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 898 op/s; 1320/114276 objects degraded (1.155%); 2740/114276 objects misplaced (2.398%); 35 MiB/s, 565 objects/s recovering 2024-09-21T12:38:44.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:44 smithi114 ceph-mon[30324]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-21T12:38:44.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:44 smithi114 ceph-mon[30324]: osdmap e472: 8 total, 8 up, 8 in 2024-09-21T12:38:44.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:44 smithi114 ceph-mon[40867]: pgmap v942: 178 pgs: 2 active+remapped+backfilling, 1 peering, 1 active+undersized+degraded+remapped+backfilling, 174 active+clean; 2.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 898 op/s; 1320/114276 objects degraded (1.155%); 2740/114276 objects misplaced (2.398%); 35 MiB/s, 565 objects/s recovering 2024-09-21T12:38:44.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:44 smithi114 ceph-mon[40867]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-21T12:38:44.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:44 smithi114 ceph-mon[40867]: osdmap e472: 8 total, 8 up, 8 in 2024-09-21T12:38:44.811 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:44.846 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:44.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:44.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:44.947 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:44.982 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:44.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:44.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:45.083 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:45.116 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:45.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:45.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:45.156 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:38:45.156 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:38:45.218 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:45.249 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:45.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:45.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:45.351 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:45.379 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:45.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:45.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:45.481 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:45.512 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:45.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:45.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:45.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:45 smithi114 ceph-mon[40867]: 11.3 scrub starts 2024-09-21T12:38:45.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:45 smithi114 ceph-mon[40867]: 11.3 scrub ok 2024-09-21T12:38:45.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:45 smithi114 ceph-mon[40867]: 3.b scrub starts 2024-09-21T12:38:45.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:45 smithi114 ceph-mon[40867]: 3.b scrub ok 2024-09-21T12:38:45.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:45 smithi114 ceph-mon[40867]: 6.2 scrub starts 2024-09-21T12:38:45.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:45 smithi114 ceph-mon[40867]: 6.2 scrub ok 2024-09-21T12:38:45.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:45 smithi114 ceph-mon[40867]: 5.1b scrub starts 2024-09-21T12:38:45.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:45 smithi114 ceph-mon[40867]: 5.1b scrub ok 2024-09-21T12:38:45.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:45 smithi114 ceph-mon[40867]: osdmap e473: 8 total, 8 up, 8 in 2024-09-21T12:38:45.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:45 smithi114 ceph-mon[40867]: 6.0 scrub starts 2024-09-21T12:38:45.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:45 smithi114 ceph-mon[40867]: 6.0 scrub ok 2024-09-21T12:38:45.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:45 smithi114 ceph-mon[30324]: 11.3 scrub starts 2024-09-21T12:38:45.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:45 smithi114 ceph-mon[30324]: 11.3 scrub ok 2024-09-21T12:38:45.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:45 smithi114 ceph-mon[30324]: 3.b scrub starts 2024-09-21T12:38:45.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:45 smithi114 ceph-mon[30324]: 3.b scrub ok 2024-09-21T12:38:45.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:45 smithi114 ceph-mon[30324]: 6.2 scrub starts 2024-09-21T12:38:45.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:45 smithi114 ceph-mon[30324]: 6.2 scrub ok 2024-09-21T12:38:45.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:45 smithi114 ceph-mon[30324]: 5.1b scrub starts 2024-09-21T12:38:45.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:45 smithi114 ceph-mon[30324]: 5.1b scrub ok 2024-09-21T12:38:45.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:45 smithi114 ceph-mon[30324]: osdmap e473: 8 total, 8 up, 8 in 2024-09-21T12:38:45.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:45 smithi114 ceph-mon[30324]: 6.0 scrub starts 2024-09-21T12:38:45.528 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:45 smithi114 ceph-mon[30324]: 6.0 scrub ok 2024-09-21T12:38:45.614 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:45.652 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:45.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:45.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:45.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:45 smithi183 ceph-mon[33277]: 11.3 scrub starts 2024-09-21T12:38:45.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:45 smithi183 ceph-mon[33277]: 11.3 scrub ok 2024-09-21T12:38:45.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:45 smithi183 ceph-mon[33277]: 3.b scrub starts 2024-09-21T12:38:45.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:45 smithi183 ceph-mon[33277]: 3.b scrub ok 2024-09-21T12:38:45.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:45 smithi183 ceph-mon[33277]: 6.2 scrub starts 2024-09-21T12:38:45.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:45 smithi183 ceph-mon[33277]: 6.2 scrub ok 2024-09-21T12:38:45.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:45 smithi183 ceph-mon[33277]: 5.1b scrub starts 2024-09-21T12:38:45.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:45 smithi183 ceph-mon[33277]: 5.1b scrub ok 2024-09-21T12:38:45.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:45 smithi183 ceph-mon[33277]: osdmap e473: 8 total, 8 up, 8 in 2024-09-21T12:38:45.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:45 smithi183 ceph-mon[33277]: 6.0 scrub starts 2024-09-21T12:38:45.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:45 smithi183 ceph-mon[33277]: 6.0 scrub ok 2024-09-21T12:38:45.753 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:45.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:45.802 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:45.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:45.892 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T12:38:45.903 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:45.937 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:45.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:45.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:46.039 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:46.066 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:46.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:46.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:46.102 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:38:46.168 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:46.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:46.204 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:46.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:46.273 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:38:46.306 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:46.334 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:46.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:46.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:46.435 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:46.448 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:38:46.448 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:38:46.463 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:46.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:46.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:46.485 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:38:46.485 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:38:46.520 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:38:46.520 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:38:46.562 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:38:46.562 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:38:46.565 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:46.594 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:46.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:46.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:46.596 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:38:46.596 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:38:46.631 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:38:46.631 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:38:46.632 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:46 smithi114 ceph-mon[40867]: 4.a scrub starts 2024-09-21T12:38:46.632 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:46 smithi114 ceph-mon[40867]: 4.a scrub ok 2024-09-21T12:38:46.632 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:46 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:38:46.632 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:46 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:38:46.632 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:46 smithi114 ceph-mon[40867]: pgmap v945: 178 pgs: 2 active+remapped+backfilling, 2 peering, 174 active+clean; 2.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 2740/113586 objects misplaced (2.412%); 53 MiB/s, 841 objects/s recovering 2024-09-21T12:38:46.632 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:46 smithi114 ceph-mon[40867]: 5.15 scrub starts 2024-09-21T12:38:46.632 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:46 smithi114 ceph-mon[40867]: 5.15 scrub ok 2024-09-21T12:38:46.632 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:46 smithi114 ceph-mon[40867]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1320/114276 objects degraded (1.155%), 1 pg degraded) 2024-09-21T12:38:46.632 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:46 smithi114 ceph-mon[40867]: osdmap e474: 8 total, 8 up, 8 in 2024-09-21T12:38:46.633 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:38:46.633 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:38:46.633 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:46 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:38:46.633 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:46 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:38:46.633 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:46 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2634356545' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:38:46.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:46 smithi114 ceph-mon[30324]: 4.a scrub starts 2024-09-21T12:38:46.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:46 smithi114 ceph-mon[30324]: 4.a scrub ok 2024-09-21T12:38:46.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:46 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:38:46.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:46 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:38:46.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:46 smithi114 ceph-mon[30324]: pgmap v945: 178 pgs: 2 active+remapped+backfilling, 2 peering, 174 active+clean; 2.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 2740/113586 objects misplaced (2.412%); 53 MiB/s, 841 objects/s recovering 2024-09-21T12:38:46.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:46 smithi114 ceph-mon[30324]: 5.15 scrub starts 2024-09-21T12:38:46.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:46 smithi114 ceph-mon[30324]: 5.15 scrub ok 2024-09-21T12:38:46.634 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:46 smithi114 ceph-mon[30324]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1320/114276 objects degraded (1.155%), 1 pg degraded) 2024-09-21T12:38:46.634 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:46 smithi114 ceph-mon[30324]: osdmap e474: 8 total, 8 up, 8 in 2024-09-21T12:38:46.634 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:38:46.634 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:38:46.634 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:46 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:38:46.634 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:46 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:38:46.634 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:46 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2634356545' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:38:46.668 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:38:46.668 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:38:46.696 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:46.706 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:46 smithi183 ceph-mon[33277]: 4.a scrub starts 2024-09-21T12:38:46.706 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:46 smithi183 ceph-mon[33277]: 4.a scrub ok 2024-09-21T12:38:46.706 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:46 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:38:46.706 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:46 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:38:46.706 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:46 smithi183 ceph-mon[33277]: pgmap v945: 178 pgs: 2 active+remapped+backfilling, 2 peering, 174 active+clean; 2.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 2740/113586 objects misplaced (2.412%); 53 MiB/s, 841 objects/s recovering 2024-09-21T12:38:46.706 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:46 smithi183 ceph-mon[33277]: 5.15 scrub starts 2024-09-21T12:38:46.706 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:46 smithi183 ceph-mon[33277]: 5.15 scrub ok 2024-09-21T12:38:46.706 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:46 smithi183 ceph-mon[33277]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1320/114276 objects degraded (1.155%), 1 pg degraded) 2024-09-21T12:38:46.707 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:46 smithi183 ceph-mon[33277]: osdmap e474: 8 total, 8 up, 8 in 2024-09-21T12:38:46.707 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:38:46.707 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:38:46.707 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:46 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:38:46.707 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:46 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:38:46.707 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:46 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2634356545' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:38:46.709 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:38:46.710 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:38:46.731 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:46.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:46.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:46.833 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:46.861 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:38:46.861 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-21T12:38:46.861 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 0 2024-09-21T12:38:46.861 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:38:46.871 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:46.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:46.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:46.974 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:47.007 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:47.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:47.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:47.109 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:47.140 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:47.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:47.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:47.241 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:47.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:47.284 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:47.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:47.385 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:47.406 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:47.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:47.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:47.507 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:47.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:47 smithi114 ceph-mon[30324]: 6.1 scrub starts 2024-09-21T12:38:47.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:47 smithi114 ceph-mon[30324]: 6.1 scrub ok 2024-09-21T12:38:47.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:47 smithi114 ceph-mon[30324]: 4.c scrub starts 2024-09-21T12:38:47.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:47 smithi114 ceph-mon[30324]: 4.c scrub ok 2024-09-21T12:38:47.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:47 smithi114 ceph-mon[30324]: pgmap v947: 178 pgs: 2 active+remapped+backfilling, 2 peering, 174 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 440 op/s; 2740/113586 objects misplaced (2.412%); 24 MiB/s, 376 objects/s recovering 2024-09-21T12:38:47.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:47 smithi114 ceph-mon[30324]: pgmap v948: 178 pgs: 2 active+remapped+backfilling, 2 peering, 174 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 73 op/s; 2740/113586 objects misplaced (2.412%); 25 MiB/s, 402 objects/s recovering 2024-09-21T12:38:47.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:47 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:38:47.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:47 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:38:47.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:47 smithi114 ceph-mon[30324]: 5.8 scrub starts 2024-09-21T12:38:47.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:47 smithi114 ceph-mon[30324]: osdmap e475: 8 total, 8 up, 8 in 2024-09-21T12:38:47.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:47 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/140227551' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:38:47.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:47 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:38:47.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:47 smithi114 ceph-mon[40867]: 6.1 scrub starts 2024-09-21T12:38:47.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:47 smithi114 ceph-mon[40867]: 6.1 scrub ok 2024-09-21T12:38:47.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:47 smithi114 ceph-mon[40867]: 4.c scrub starts 2024-09-21T12:38:47.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:47 smithi114 ceph-mon[40867]: 4.c scrub ok 2024-09-21T12:38:47.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:47 smithi114 ceph-mon[40867]: pgmap v947: 178 pgs: 2 active+remapped+backfilling, 2 peering, 174 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 440 op/s; 2740/113586 objects misplaced (2.412%); 24 MiB/s, 376 objects/s recovering 2024-09-21T12:38:47.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:47 smithi114 ceph-mon[40867]: pgmap v948: 178 pgs: 2 active+remapped+backfilling, 2 peering, 174 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 73 op/s; 2740/113586 objects misplaced (2.412%); 25 MiB/s, 402 objects/s recovering 2024-09-21T12:38:47.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:47 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:38:47.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:47 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:38:47.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:47 smithi114 ceph-mon[40867]: 5.8 scrub starts 2024-09-21T12:38:47.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:47 smithi114 ceph-mon[40867]: osdmap e475: 8 total, 8 up, 8 in 2024-09-21T12:38:47.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:47 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/140227551' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:38:47.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:47 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:38:47.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:47.549 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:47.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:47.650 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:47.686 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:47.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:47.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:47.737 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:47 smithi183 ceph-mon[33277]: 6.1 scrub starts 2024-09-21T12:38:47.737 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:47 smithi183 ceph-mon[33277]: 6.1 scrub ok 2024-09-21T12:38:47.737 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:47 smithi183 ceph-mon[33277]: 4.c scrub starts 2024-09-21T12:38:47.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:47 smithi183 ceph-mon[33277]: 4.c scrub ok 2024-09-21T12:38:47.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:47 smithi183 ceph-mon[33277]: pgmap v947: 178 pgs: 2 active+remapped+backfilling, 2 peering, 174 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 440 op/s; 2740/113586 objects misplaced (2.412%); 24 MiB/s, 376 objects/s recovering 2024-09-21T12:38:47.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:47 smithi183 ceph-mon[33277]: pgmap v948: 178 pgs: 2 active+remapped+backfilling, 2 peering, 174 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 73 op/s; 2740/113586 objects misplaced (2.412%); 25 MiB/s, 402 objects/s recovering 2024-09-21T12:38:47.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:47 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:38:47.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:47 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:38:47.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:47 smithi183 ceph-mon[33277]: 5.8 scrub starts 2024-09-21T12:38:47.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:47 smithi183 ceph-mon[33277]: osdmap e475: 8 total, 8 up, 8 in 2024-09-21T12:38:47.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:47 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/140227551' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:38:47.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:47 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:38:47.788 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:47.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:47.824 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:47.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:47.864 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:38:47.925 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:47.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:47.949 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:47.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:47.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:38:47 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:38:48.051 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:48.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:48.082 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:48.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:48.182 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:48.208 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:48.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:48.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:48.310 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:48.312 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-21T12:38:48.336 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:48.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:48.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:48.438 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:48.459 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:48.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:48.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:48.561 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:48.583 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:48.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:48.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:48.684 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:48.706 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:48.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:48.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:48.808 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:48.828 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:48.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:48.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:48.930 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:48.951 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:48.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:48.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:48.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:48 smithi183 ceph-mon[33277]: 5.8 scrub ok 2024-09-21T12:38:48.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:48 smithi183 ceph-mon[33277]: 5.5 scrub starts 2024-09-21T12:38:48.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:48 smithi183 ceph-mon[33277]: 5.5 scrub ok 2024-09-21T12:38:48.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:48 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:38:48.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:48 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:38:48.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:48 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:38:48.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:48 smithi183 ceph-mon[33277]: 4.12 scrub starts 2024-09-21T12:38:48.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:48 smithi183 ceph-mon[33277]: 4.12 scrub ok 2024-09-21T12:38:48.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:48 smithi183 ceph-mon[33277]: pgmap v950: 178 pgs: 2 active+remapped+backfilling, 1 peering, 175 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.10k op/s; 269/100689 objects misplaced (0.267%); 37 MiB/s, 598 objects/s recovering 2024-09-21T12:38:48.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:48 smithi183 ceph-mon[33277]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:38:48.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:48 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:38:48.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:48 smithi183 ceph-mon[33277]: osdmap e476: 8 total, 8 up, 8 in 2024-09-21T12:38:49.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:48 smithi114 ceph-mon[40867]: 5.8 scrub ok 2024-09-21T12:38:49.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:48 smithi114 ceph-mon[40867]: 5.5 scrub starts 2024-09-21T12:38:49.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:48 smithi114 ceph-mon[40867]: 5.5 scrub ok 2024-09-21T12:38:49.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:48 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:38:49.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:48 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:38:49.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:48 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:38:49.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:48 smithi114 ceph-mon[40867]: 4.12 scrub starts 2024-09-21T12:38:49.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:48 smithi114 ceph-mon[40867]: 4.12 scrub ok 2024-09-21T12:38:49.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:48 smithi114 ceph-mon[40867]: pgmap v950: 178 pgs: 2 active+remapped+backfilling, 1 peering, 175 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.10k op/s; 269/100689 objects misplaced (0.267%); 37 MiB/s, 598 objects/s recovering 2024-09-21T12:38:49.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:48 smithi114 ceph-mon[40867]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:38:49.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:48 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:38:49.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:48 smithi114 ceph-mon[40867]: osdmap e476: 8 total, 8 up, 8 in 2024-09-21T12:38:49.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:48 smithi114 ceph-mon[30324]: 5.8 scrub ok 2024-09-21T12:38:49.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:48 smithi114 ceph-mon[30324]: 5.5 scrub starts 2024-09-21T12:38:49.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:48 smithi114 ceph-mon[30324]: 5.5 scrub ok 2024-09-21T12:38:49.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:48 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:38:49.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:48 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:38:49.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:48 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:38:49.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:48 smithi114 ceph-mon[30324]: 4.12 scrub starts 2024-09-21T12:38:49.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:48 smithi114 ceph-mon[30324]: 4.12 scrub ok 2024-09-21T12:38:49.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:48 smithi114 ceph-mon[30324]: pgmap v950: 178 pgs: 2 active+remapped+backfilling, 1 peering, 175 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.10k op/s; 269/100689 objects misplaced (0.267%); 37 MiB/s, 598 objects/s recovering 2024-09-21T12:38:49.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:48 smithi114 ceph-mon[30324]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:38:49.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:48 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:38:49.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:48 smithi114 ceph-mon[30324]: osdmap e476: 8 total, 8 up, 8 in 2024-09-21T12:38:49.053 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:49.079 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:49.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:49.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:49.180 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:49.202 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:49.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:49.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:49.303 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:49.307 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T12:38:49.329 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:49.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:49.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:49.430 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:49.463 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:49.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:49.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:49.565 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:49.595 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:49.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:49.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:49.697 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:49.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:49.733 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:49.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:49.817 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:38:49.834 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:49.851 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:49 smithi114 ceph-mon[30324]: 3.1 scrub starts 2024-09-21T12:38:49.851 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:49 smithi114 ceph-mon[30324]: 3.1 scrub ok 2024-09-21T12:38:49.852 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:49 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/823407797' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:38:49.852 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:49 smithi114 ceph-mon[30324]: 1.9 scrub starts 2024-09-21T12:38:49.852 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:49 smithi114 ceph-mon[30324]: 1.9 scrub ok 2024-09-21T12:38:49.852 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:49 smithi114 ceph-mon[30324]: 3.4 scrub starts 2024-09-21T12:38:49.852 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:49 smithi114 ceph-mon[30324]: 3.4 scrub ok 2024-09-21T12:38:49.852 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:49 smithi114 ceph-mon[30324]: 5.1c scrub starts 2024-09-21T12:38:49.852 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:49 smithi114 ceph-mon[30324]: 5.1c scrub ok 2024-09-21T12:38:49.852 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:49 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/823407797' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T12:38:49.852 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:49 smithi114 ceph-mon[30324]: osdmap e477: 8 total, 8 up, 8 in 2024-09-21T12:38:49.853 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:49 smithi114 ceph-mon[40867]: 3.1 scrub starts 2024-09-21T12:38:49.853 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:49 smithi114 ceph-mon[40867]: 3.1 scrub ok 2024-09-21T12:38:49.853 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:49 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/823407797' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:38:49.853 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:49 smithi114 ceph-mon[40867]: 1.9 scrub starts 2024-09-21T12:38:49.853 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:49 smithi114 ceph-mon[40867]: 1.9 scrub ok 2024-09-21T12:38:49.853 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:49 smithi114 ceph-mon[40867]: 3.4 scrub starts 2024-09-21T12:38:49.853 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:49 smithi114 ceph-mon[40867]: 3.4 scrub ok 2024-09-21T12:38:49.853 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:49 smithi114 ceph-mon[40867]: 5.1c scrub starts 2024-09-21T12:38:49.853 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:49 smithi114 ceph-mon[40867]: 5.1c scrub ok 2024-09-21T12:38:49.853 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:49 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/823407797' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T12:38:49.853 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:49 smithi114 ceph-mon[40867]: osdmap e477: 8 total, 8 up, 8 in 2024-09-21T12:38:49.863 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:49.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:49.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:49.965 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:49.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:49 smithi183 ceph-mon[33277]: 3.1 scrub starts 2024-09-21T12:38:49.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:49 smithi183 ceph-mon[33277]: 3.1 scrub ok 2024-09-21T12:38:49.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:49 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/823407797' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:38:49.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:49 smithi183 ceph-mon[33277]: 1.9 scrub starts 2024-09-21T12:38:49.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:49 smithi183 ceph-mon[33277]: 1.9 scrub ok 2024-09-21T12:38:49.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:49 smithi183 ceph-mon[33277]: 3.4 scrub starts 2024-09-21T12:38:49.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:49 smithi183 ceph-mon[33277]: 3.4 scrub ok 2024-09-21T12:38:49.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:49 smithi183 ceph-mon[33277]: 5.1c scrub starts 2024-09-21T12:38:49.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:49 smithi183 ceph-mon[33277]: 5.1c scrub ok 2024-09-21T12:38:49.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:49 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/823407797' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T12:38:49.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:49 smithi183 ceph-mon[33277]: osdmap e477: 8 total, 8 up, 8 in 2024-09-21T12:38:49.993 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:49.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:49.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:50.095 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:50.126 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:50.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:50.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:50.175 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T12:38:50.228 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:50.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:50.267 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:50.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:50.368 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:50.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:50.407 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:50.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:50.509 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:50.551 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:50.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:50.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:50.579 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:38:50.654 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:50.687 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:50.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:50.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:50.748 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:50 smithi183 ceph-mon[33277]: 2.1 scrub starts 2024-09-21T12:38:50.748 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:50 smithi183 ceph-mon[33277]: 2.1 scrub ok 2024-09-21T12:38:50.748 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:50 smithi183 ceph-mon[33277]: 3.0 scrub starts 2024-09-21T12:38:50.749 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:50 smithi183 ceph-mon[33277]: 3.0 scrub ok 2024-09-21T12:38:50.749 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:50 smithi183 ceph-mon[33277]: pgmap v953: 178 pgs: 2 active+remapped+backfilling, 176 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 269/99762 objects misplaced (0.270%); 36 MiB/s, 581 objects/s recovering 2024-09-21T12:38:50.789 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:50.809 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:50.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:50.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:50.911 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:50.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:50.945 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:50.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:51.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:50 smithi114 ceph-mon[40867]: 2.1 scrub starts 2024-09-21T12:38:51.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:50 smithi114 ceph-mon[40867]: 2.1 scrub ok 2024-09-21T12:38:51.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:50 smithi114 ceph-mon[40867]: 3.0 scrub starts 2024-09-21T12:38:51.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:50 smithi114 ceph-mon[40867]: 3.0 scrub ok 2024-09-21T12:38:51.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:50 smithi114 ceph-mon[40867]: pgmap v953: 178 pgs: 2 active+remapped+backfilling, 176 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 269/99762 objects misplaced (0.270%); 36 MiB/s, 581 objects/s recovering 2024-09-21T12:38:51.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:50 smithi114 ceph-mon[30324]: 2.1 scrub starts 2024-09-21T12:38:51.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:50 smithi114 ceph-mon[30324]: 2.1 scrub ok 2024-09-21T12:38:51.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:50 smithi114 ceph-mon[30324]: 3.0 scrub starts 2024-09-21T12:38:51.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:50 smithi114 ceph-mon[30324]: 3.0 scrub ok 2024-09-21T12:38:51.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:50 smithi114 ceph-mon[30324]: pgmap v953: 178 pgs: 2 active+remapped+backfilling, 176 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 269/99762 objects misplaced (0.270%); 36 MiB/s, 581 objects/s recovering 2024-09-21T12:38:51.046 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:51.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:51.075 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:51.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:51.176 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:51.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:51.214 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:51.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:51.224 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:38:51.224 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-21T12:38:51.224 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-21T12:38:51.224 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-21T12:38:51.224 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:38:51.315 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:51.351 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:51.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:51.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:51.453 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:51.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:51.494 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:51.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:51.596 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:51.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:51.640 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:51.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:51.741 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:51.775 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:51.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:51.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:51.877 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:51.915 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:51.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:51.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:51.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:51 smithi183 ceph-mon[33277]: 5.17 scrub starts 2024-09-21T12:38:51.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:51 smithi183 ceph-mon[33277]: 5.17 scrub ok 2024-09-21T12:38:51.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:51 smithi183 ceph-mon[33277]: 1.2 scrub starts 2024-09-21T12:38:51.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:51 smithi183 ceph-mon[33277]: 1.2 scrub ok 2024-09-21T12:38:51.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:51 smithi183 ceph-mon[33277]: 11.c scrub starts 2024-09-21T12:38:51.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:51 smithi183 ceph-mon[33277]: 1.5 scrub starts 2024-09-21T12:38:51.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:51 smithi183 ceph-mon[33277]: 1.5 scrub ok 2024-09-21T12:38:52.016 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:52.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:51 smithi114 ceph-mon[30324]: 5.17 scrub starts 2024-09-21T12:38:52.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:51 smithi114 ceph-mon[30324]: 5.17 scrub ok 2024-09-21T12:38:52.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:51 smithi114 ceph-mon[30324]: 1.2 scrub starts 2024-09-21T12:38:52.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:51 smithi114 ceph-mon[30324]: 1.2 scrub ok 2024-09-21T12:38:52.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:51 smithi114 ceph-mon[30324]: 11.c scrub starts 2024-09-21T12:38:52.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:51 smithi114 ceph-mon[30324]: 1.5 scrub starts 2024-09-21T12:38:52.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:51 smithi114 ceph-mon[30324]: 1.5 scrub ok 2024-09-21T12:38:52.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:51 smithi114 ceph-mon[40867]: 5.17 scrub starts 2024-09-21T12:38:52.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:51 smithi114 ceph-mon[40867]: 5.17 scrub ok 2024-09-21T12:38:52.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:51 smithi114 ceph-mon[40867]: 1.2 scrub starts 2024-09-21T12:38:52.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:51 smithi114 ceph-mon[40867]: 1.2 scrub ok 2024-09-21T12:38:52.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:51 smithi114 ceph-mon[40867]: 11.c scrub starts 2024-09-21T12:38:52.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:51 smithi114 ceph-mon[40867]: 1.5 scrub starts 2024-09-21T12:38:52.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:51 smithi114 ceph-mon[40867]: 1.5 scrub ok 2024-09-21T12:38:52.043 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:52.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:52.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:52.078 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T12:38:52.144 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:52.171 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:52.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:52.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:52.216 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:38:52.273 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:52.301 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:52.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:52.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:52.403 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:52.450 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:52.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:52.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:52.551 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:52.558 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T12:38:52.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:52.579 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:52.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:52.608 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:38:52 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:38:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:38:52.680 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:52.711 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:52.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:52.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:52.813 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:52.848 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:52.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:52.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:52.874 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:52 smithi114 ceph-mon[40867]: 3.c scrub starts 2024-09-21T12:38:52.874 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:52 smithi114 ceph-mon[40867]: 3.c scrub ok 2024-09-21T12:38:52.874 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:52 smithi114 ceph-mon[40867]: 11.c scrub ok 2024-09-21T12:38:52.874 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:52 smithi114 ceph-mon[40867]: 6.3 scrub starts 2024-09-21T12:38:52.874 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:52 smithi114 ceph-mon[40867]: 6.3 scrub ok 2024-09-21T12:38:52.874 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:52 smithi114 ceph-mon[40867]: 6.1d scrub starts 2024-09-21T12:38:52.874 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:52 smithi114 ceph-mon[40867]: 6.1d scrub ok 2024-09-21T12:38:52.874 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:52 smithi114 ceph-mon[40867]: osdmap e478: 8 total, 8 up, 8 in 2024-09-21T12:38:52.874 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:52 smithi114 ceph-mon[40867]: pgmap v955: 178 pgs: 178 active+clean; 1.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 988 op/s; 3.3 MiB/s, 53 objects/s recovering 2024-09-21T12:38:52.874 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:52 smithi114 ceph-mon[30324]: 3.c scrub starts 2024-09-21T12:38:52.875 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:52 smithi114 ceph-mon[30324]: 3.c scrub ok 2024-09-21T12:38:52.875 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:52 smithi114 ceph-mon[30324]: 11.c scrub ok 2024-09-21T12:38:52.875 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:52 smithi114 ceph-mon[30324]: 6.3 scrub starts 2024-09-21T12:38:52.875 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:52 smithi114 ceph-mon[30324]: 6.3 scrub ok 2024-09-21T12:38:52.875 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:52 smithi114 ceph-mon[30324]: 6.1d scrub starts 2024-09-21T12:38:52.875 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:52 smithi114 ceph-mon[30324]: 6.1d scrub ok 2024-09-21T12:38:52.875 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:52 smithi114 ceph-mon[30324]: osdmap e478: 8 total, 8 up, 8 in 2024-09-21T12:38:52.875 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:52 smithi114 ceph-mon[30324]: pgmap v955: 178 pgs: 178 active+clean; 1.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 988 op/s; 3.3 MiB/s, 53 objects/s recovering 2024-09-21T12:38:52.943 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:38:52.950 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:52.960 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:38:52.978 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:38:52.978 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:38:52.983 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:52.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:52.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:52.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:52 smithi183 ceph-mon[33277]: 3.c scrub starts 2024-09-21T12:38:52.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:52 smithi183 ceph-mon[33277]: 3.c scrub ok 2024-09-21T12:38:52.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:52 smithi183 ceph-mon[33277]: 11.c scrub ok 2024-09-21T12:38:52.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:52 smithi183 ceph-mon[33277]: 6.3 scrub starts 2024-09-21T12:38:52.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:52 smithi183 ceph-mon[33277]: 6.3 scrub ok 2024-09-21T12:38:52.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:52 smithi183 ceph-mon[33277]: 6.1d scrub starts 2024-09-21T12:38:52.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:52 smithi183 ceph-mon[33277]: 6.1d scrub ok 2024-09-21T12:38:52.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:52 smithi183 ceph-mon[33277]: osdmap e478: 8 total, 8 up, 8 in 2024-09-21T12:38:52.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:52 smithi183 ceph-mon[33277]: pgmap v955: 178 pgs: 178 active+clean; 1.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 988 op/s; 3.3 MiB/s, 53 objects/s recovering 2024-09-21T12:38:53.014 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:38:53.014 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:38:53.044 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:38:53.044 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:38:53.074 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:38:53.074 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:38:53.084 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:53.106 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:38:53.107 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:38:53.122 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:53.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:53.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:53.143 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:38:53.143 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:38:53.179 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:38:53.179 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:38:53.187 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T12:38:53.218 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:38:53.218 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:38:53.225 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:53.264 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:53.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:53.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:53.366 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:53.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:53.419 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:53.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:53.520 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:53.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:53.561 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:53.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:53.662 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:53.675 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:38:53.675 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:38:53.699 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:53.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:53.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:53.800 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:53.824 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:53.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:53.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:53.925 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:53.928 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:53 smithi114 ceph-mon[30324]: 3.1c scrub starts 2024-09-21T12:38:53.928 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:53 smithi114 ceph-mon[30324]: 3.1c scrub ok 2024-09-21T12:38:53.928 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:53 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1674753407' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:38:53.928 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:53 smithi114 ceph-mon[30324]: 4.d scrub starts 2024-09-21T12:38:53.928 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:53 smithi114 ceph-mon[30324]: 4.d scrub ok 2024-09-21T12:38:53.929 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:53 smithi114 ceph-mon[40867]: 3.1c scrub starts 2024-09-21T12:38:53.929 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:53 smithi114 ceph-mon[40867]: 3.1c scrub ok 2024-09-21T12:38:53.929 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:53 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1674753407' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:38:53.929 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:53 smithi114 ceph-mon[40867]: 4.d scrub starts 2024-09-21T12:38:53.929 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:53 smithi114 ceph-mon[40867]: 4.d scrub ok 2024-09-21T12:38:53.958 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:53.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:53.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:53.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:53 smithi183 ceph-mon[33277]: 3.1c scrub starts 2024-09-21T12:38:53.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:53 smithi183 ceph-mon[33277]: 3.1c scrub ok 2024-09-21T12:38:53.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:53 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1674753407' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:38:53.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:53 smithi183 ceph-mon[33277]: 4.d scrub starts 2024-09-21T12:38:53.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:53 smithi183 ceph-mon[33277]: 4.d scrub ok 2024-09-21T12:38:54.060 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:54.093 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:54.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:54.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:54.195 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:54.220 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:54.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:54.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:54.322 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:54.336 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:38:54.344 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:54.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:54.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:54.445 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:54.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:54.483 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:54.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:54.583 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:54.616 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:54.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:54.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:54.627 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T12:38:54.717 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:54.751 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:54.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:54.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:54.852 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:54.884 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:54.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:54.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:54.931 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:54 smithi114 ceph-mon[40867]: 3.d scrub starts 2024-09-21T12:38:54.931 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:54 smithi114 ceph-mon[40867]: 3.d scrub ok 2024-09-21T12:38:54.931 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:54 smithi114 ceph-mon[40867]: 5.1d scrub starts 2024-09-21T12:38:54.931 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:54 smithi114 ceph-mon[40867]: 5.1d scrub ok 2024-09-21T12:38:54.931 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:54 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1457798478' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:38:54.931 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:54 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:38:54.931 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:54 smithi114 ceph-mon[40867]: pgmap v956: 178 pgs: 178 active+clean; 1.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 895 op/s; 2.8 MiB/s, 44 objects/s recovering 2024-09-21T12:38:54.931 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:54 smithi114 ceph-mon[40867]: 4.1e scrub starts 2024-09-21T12:38:54.931 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:54 smithi114 ceph-mon[40867]: 4.1e scrub ok 2024-09-21T12:38:54.931 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:54 smithi114 ceph-mon[30324]: 3.d scrub starts 2024-09-21T12:38:54.932 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:54 smithi114 ceph-mon[30324]: 3.d scrub ok 2024-09-21T12:38:54.932 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:54 smithi114 ceph-mon[30324]: 5.1d scrub starts 2024-09-21T12:38:54.932 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:54 smithi114 ceph-mon[30324]: 5.1d scrub ok 2024-09-21T12:38:54.932 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:54 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1457798478' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:38:54.932 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:54 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:38:54.932 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:54 smithi114 ceph-mon[30324]: pgmap v956: 178 pgs: 178 active+clean; 1.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 895 op/s; 2.8 MiB/s, 44 objects/s recovering 2024-09-21T12:38:54.932 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:54 smithi114 ceph-mon[30324]: 4.1e scrub starts 2024-09-21T12:38:54.932 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:54 smithi114 ceph-mon[30324]: 4.1e scrub ok 2024-09-21T12:38:54.986 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:54.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:54 smithi183 ceph-mon[33277]: 3.d scrub starts 2024-09-21T12:38:54.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:54 smithi183 ceph-mon[33277]: 3.d scrub ok 2024-09-21T12:38:54.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:54 smithi183 ceph-mon[33277]: 5.1d scrub starts 2024-09-21T12:38:54.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:54 smithi183 ceph-mon[33277]: 5.1d scrub ok 2024-09-21T12:38:54.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:54 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1457798478' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:38:54.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:54 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:38:54.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:54 smithi183 ceph-mon[33277]: pgmap v956: 178 pgs: 178 active+clean; 1.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 895 op/s; 2.8 MiB/s, 44 objects/s recovering 2024-09-21T12:38:54.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:54 smithi183 ceph-mon[33277]: 4.1e scrub starts 2024-09-21T12:38:54.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:54 smithi183 ceph-mon[33277]: 4.1e scrub ok 2024-09-21T12:38:55.017 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:55.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:55.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:55.118 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:55.159 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:55.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:55.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:55.239 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T12:38:55.261 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:55.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:55.316 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:55.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:55.417 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:55.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:55.452 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:55.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:55.553 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:55.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:55.602 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:55.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:55.631 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-21T12:38:55.655 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:38:55.703 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:55.738 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:55.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:55.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:55.839 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:55.846 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:55 smithi183 ceph-mon[33277]: 5.7 scrub starts 2024-09-21T12:38:55.846 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:55 smithi183 ceph-mon[33277]: 5.7 scrub ok 2024-09-21T12:38:55.847 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:55 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T12:38:55.847 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:55 smithi183 ceph-mon[33277]: osdmap e479: 8 total, 8 up, 8 in 2024-09-21T12:38:55.847 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:55 smithi183 ceph-mon[33277]: 1.15 scrub starts 2024-09-21T12:38:55.847 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:55 smithi183 ceph-mon[33277]: 1.15 scrub ok 2024-09-21T12:38:55.847 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:55 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/546019271' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:38:55.858 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T12:38:55.876 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:55.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:55.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:55.912 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:55 smithi114 ceph-mon[30324]: 5.7 scrub starts 2024-09-21T12:38:55.913 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:55 smithi114 ceph-mon[30324]: 5.7 scrub ok 2024-09-21T12:38:55.913 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:55 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T12:38:55.913 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:55 smithi114 ceph-mon[30324]: osdmap e479: 8 total, 8 up, 8 in 2024-09-21T12:38:55.913 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:55 smithi114 ceph-mon[30324]: 1.15 scrub starts 2024-09-21T12:38:55.913 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:55 smithi114 ceph-mon[30324]: 1.15 scrub ok 2024-09-21T12:38:55.913 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:55 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/546019271' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:38:55.913 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:55 smithi114 ceph-mon[40867]: 5.7 scrub starts 2024-09-21T12:38:55.913 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:55 smithi114 ceph-mon[40867]: 5.7 scrub ok 2024-09-21T12:38:55.913 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:55 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T12:38:55.913 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:55 smithi114 ceph-mon[40867]: osdmap e479: 8 total, 8 up, 8 in 2024-09-21T12:38:55.913 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:55 smithi114 ceph-mon[40867]: 1.15 scrub starts 2024-09-21T12:38:55.914 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:55 smithi114 ceph-mon[40867]: 1.15 scrub ok 2024-09-21T12:38:55.914 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:55 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/546019271' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T12:38:55.978 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:56.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:56.028 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:56.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:56.128 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:56.167 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:56.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:56.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:56.269 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:56.283 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:38:56.283 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:38:56.305 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:56.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:56.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:56.329 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:38:56.406 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:56.441 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:56.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:56.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:56.544 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:56.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:56.603 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:56.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:56.639 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:38:56.704 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:56.736 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:56.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:56.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:56.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:56 smithi114 ceph-mon[40867]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:38:56.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:56 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/546019271' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:38:56.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:56 smithi114 ceph-mon[40867]: osdmap e480: 8 total, 8 up, 8 in 2024-09-21T12:38:56.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:56 smithi114 ceph-mon[40867]: pgmap v959: 178 pgs: 178 active+clean; 1.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 844 op/s; 2.8 MiB/s, 44 objects/s recovering 2024-09-21T12:38:56.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:56 smithi114 ceph-mon[30324]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:38:56.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:56 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/546019271' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:38:56.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:56 smithi114 ceph-mon[30324]: osdmap e480: 8 total, 8 up, 8 in 2024-09-21T12:38:56.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:56 smithi114 ceph-mon[30324]: pgmap v959: 178 pgs: 178 active+clean; 1.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 844 op/s; 2.8 MiB/s, 44 objects/s recovering 2024-09-21T12:38:56.837 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:56.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:56.877 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:56.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:56.965 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:56 smithi183 ceph-mon[33277]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:38:56.965 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:56 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/546019271' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T12:38:56.965 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:56 smithi183 ceph-mon[33277]: osdmap e480: 8 total, 8 up, 8 in 2024-09-21T12:38:56.965 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:56 smithi183 ceph-mon[33277]: pgmap v959: 178 pgs: 178 active+clean; 1.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 844 op/s; 2.8 MiB/s, 44 objects/s recovering 2024-09-21T12:38:56.978 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:57.013 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T12:38:57.014 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:57.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:57.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:57.116 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:57.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:57.157 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:57.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:57.258 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:57.291 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:57.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:57.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:57.347 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:38:57.393 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:57.402 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:38:57.433 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:57.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:57.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:57.535 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:57.570 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:57.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:57.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:57.673 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:57.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:57.711 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:57.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:57.812 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:38:57.819 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:38:57 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:38:57.847 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:57.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:57.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:38:57.949 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:57.978 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-21T12:38:57.978 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:38:57.993 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:57.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:57.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:58.004 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:38:58.004 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:38:58.037 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:38:58.038 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:38:58.070 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:38:58.070 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:38:58.095 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:58.106 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:38:58.107 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:38:58.125 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:58.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:58.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:58.138 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:38:58.139 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:38:58.172 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:38:58.172 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:38:58.174 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T12:38:58.214 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:38:58.214 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:38:58.227 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:58.251 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:38:58.251 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:38:58.254 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:58.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:58.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:58.356 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:58.393 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:58.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:58.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:58.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:58 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2350577714' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:38:58.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:58 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:38:58.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:58 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3569812316' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:38:58.495 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:38:58.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:58 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2350577714' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:38:58.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:58 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:38:58.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:58 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3569812316' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:38:58.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:58 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2350577714' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:38:58.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:58 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:38:58.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:58 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3569812316' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:38:58.527 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:58.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:58.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:38:58.541 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T12:38:58.629 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:38:58.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:58.688 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:58.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:38:58.728 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T12:38:58.788 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:58.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:58.828 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:58.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:58.929 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:58.976 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:58.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:58.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:59.078 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:38:59.114 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:38:59.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 4 1 2024-09-21T12:38:59.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:59.126 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:59.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:38:59.227 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:38:59.258 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:59.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:59.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:38:59.360 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:59.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:59.400 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:59.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:59.450 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:59 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:38:59.451 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:59 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T12:38:59.451 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:59 smithi114 ceph-mon[30324]: osdmap e481: 8 total, 8 up, 8 in 2024-09-21T12:38:59.451 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:38:59 smithi114 ceph-mon[30324]: pgmap v961: 178 pgs: 178 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.35k op/s 2024-09-21T12:38:59.451 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:59 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:38:59.451 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:59 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T12:38:59.451 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:59 smithi114 ceph-mon[40867]: osdmap e481: 8 total, 8 up, 8 in 2024-09-21T12:38:59.451 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:38:59 smithi114 ceph-mon[40867]: pgmap v961: 178 pgs: 178 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.35k op/s 2024-09-21T12:38:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:59 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:38:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:59 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T12:38:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:59 smithi183 ceph-mon[33277]: osdmap e481: 8 total, 8 up, 8 in 2024-09-21T12:38:59.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:38:59 smithi183 ceph-mon[33277]: pgmap v961: 178 pgs: 178 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.35k op/s 2024-09-21T12:38:59.501 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:59.545 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:59.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:59.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:59.647 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:38:59.681 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:38:59.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:59.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:38:59.783 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:38:59.818 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:59.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:59.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:38:59.919 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:38:59.952 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:38:59.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:59.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:38:59.997 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:38:59.997 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-21T12:38:59.998 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-21T12:38:59.998 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:38:59.998 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:38:59.998 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:4793 10.0 10:0db935dd:::benchmark_data_smithi114_161690_object4792:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-09-21T12:38:59.998 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:32:40.429237+0000", 2024-09-21T12:38:59.998 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 379.563623914, 2024-09-21T12:38:59.998 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.97170812799999995, 2024-09-21T12:38:59.998 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:38:59.998 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:38:59.998 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:38:59.998 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:38:59.998 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:38:59.999 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:38:59.999 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 4793 2024-09-21T12:38:59.999 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:38:59.999 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:38:59.999 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:38:59.999 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:38:59.999 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.429237+0000", 2024-09-21T12:38:59.999 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:38:59.999 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:38:59.999 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:38:59.999 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:38:59.999 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.429237+0000", 2024-09-21T12:38:59.999 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.000 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.000 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.000 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:00.000 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.429241+0000", 2024-09-21T12:39:00.000 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4620000000000001e-06 2024-09-21T12:39:00.000 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.000 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.000 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:00.000 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.429321+0000", 2024-09-21T12:39:00.000 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0127e-05 2024-09-21T12:39:00.000 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.000 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.000 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:00.000 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.429332+0000", 2024-09-21T12:39:00.001 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.081e-05 2024-09-21T12:39:00.001 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.001 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.001 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:00.001 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.429337+0000", 2024-09-21T12:39:00.001 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2000000000000002e-06 2024-09-21T12:39:00.001 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.001 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.001 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.001 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.429383+0000", 2024-09-21T12:39:00.001 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6369999999999998e-05 2024-09-21T12:39:00.001 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.001 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.001 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:00.002 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.429390+0000", 2024-09-21T12:39:00.002 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0729999999999999e-06 2024-09-21T12:39:00.002 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.002 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.002 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.002 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.394738+0000", 2024-09-21T12:39:00.002 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.96534762399999996 2024-09-21T12:39:00.002 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.002 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.002 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:00.002 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.394860+0000", 2024-09-21T12:39:00.002 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012222200000000001 2024-09-21T12:39:00.003 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.003 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.003 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,4", 2024-09-21T12:39:00.003 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.394986+0000", 2024-09-21T12:39:00.003 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012562900000000001 2024-09-21T12:39:00.003 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.003 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.003 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:00.003 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.396494+0000", 2024-09-21T12:39:00.003 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001508704 2024-09-21T12:39:00.003 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.003 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.003 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.004 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.399005+0000", 2024-09-21T12:39:00.004 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0025109709999999999 2024-09-21T12:39:00.004 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.004 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.004 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.004 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.400827+0000", 2024-09-21T12:39:00.004 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001821533 2024-09-21T12:39:00.004 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.004 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.004 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:00.004 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.400894+0000", 2024-09-21T12:39:00.004 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7377000000000002e-05 2024-09-21T12:39:00.004 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.004 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.005 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:00.005 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.400945+0000", 2024-09-21T12:39:00.005 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1026000000000002e-05 2024-09-21T12:39:00.005 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.005 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:00.005 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.005 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.005 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.005 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:4801 10.0 10:0f176bf4:::benchmark_data_smithi114_161690_object4800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-09-21T12:39:00.005 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:32:40.434971+0000", 2024-09-21T12:39:00.005 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 379.55788989899997, 2024-09-21T12:39:00.005 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.96613722999999996, 2024-09-21T12:39:00.005 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:00.005 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:00.006 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:00.006 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:00.006 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:39:00.006 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:39:00.006 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 4801 2024-09-21T12:39:00.006 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.006 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:00.006 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.006 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:00.006 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.434971+0000", 2024-09-21T12:39:00.006 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.006 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.006 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.006 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:00.007 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.434971+0000", 2024-09-21T12:39:00.007 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.007 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.007 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.007 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:00.007 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.434978+0000", 2024-09-21T12:39:00.007 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9670000000000004e-06 2024-09-21T12:39:00.007 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.007 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.007 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:00.007 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.435070+0000", 2024-09-21T12:39:00.007 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1558999999999995e-05 2024-09-21T12:39:00.007 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.007 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.007 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:00.008 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.435086+0000", 2024-09-21T12:39:00.008 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6447e-05 2024-09-21T12:39:00.008 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.008 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.008 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:00.008 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.435095+0000", 2024-09-21T12:39:00.008 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6719999999999993e-06 2024-09-21T12:39:00.008 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.008 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.008 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.008 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.435169+0000", 2024-09-21T12:39:00.008 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3708999999999995e-05 2024-09-21T12:39:00.008 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.008 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.008 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:00.008 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:40.435179+0000", 2024-09-21T12:39:00.009 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0409e-05 2024-09-21T12:39:00.009 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.009 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.009 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.009 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.395628+0000", 2024-09-21T12:39:00.009 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.960448686 2024-09-21T12:39:00.009 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.009 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.009 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:00.009 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.395750+0000", 2024-09-21T12:39:00.009 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000121984 2024-09-21T12:39:00.009 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.009 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.009 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,4", 2024-09-21T12:39:00.009 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.395914+0000", 2024-09-21T12:39:00.010 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000163857 2024-09-21T12:39:00.010 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.010 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.010 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:00.010 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.396701+0000", 2024-09-21T12:39:00.010 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00078757800000000004 2024-09-21T12:39:00.010 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.010 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.010 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.010 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.399098+0000", 2024-09-21T12:39:00.010 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0023964110000000002 2024-09-21T12:39:00.010 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.010 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.010 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.010 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401013+0000", 2024-09-21T12:39:00.011 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0019155540000000001 2024-09-21T12:39:00.011 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.011 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.011 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:00.011 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401072+0000", 2024-09-21T12:39:00.011 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8999999999999998e-05 2024-09-21T12:39:00.011 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.011 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.011 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:00.011 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:32:41.401108+0000", 2024-09-21T12:39:00.011 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6396999999999997e-05 2024-09-21T12:39:00.011 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.011 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:00.012 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.012 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.012 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.012 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:43123 10.f 10:f5348dda:::benchmark_data_smithi114_161690_object43122:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-09-21T12:39:00.012 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:33:00.807719+0000", 2024-09-21T12:39:00.012 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 359.185142407, 2024-09-21T12:39:00.012 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.37280072400000003, 2024-09-21T12:39:00.012 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:00.012 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:00.012 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:00.012 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:00.012 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:39:00.012 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:39:00.012 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 43123 2024-09-21T12:39:00.013 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.013 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:00.013 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.013 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:00.013 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.807719+0000", 2024-09-21T12:39:00.013 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.013 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.013 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.013 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:00.013 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.807719+0000", 2024-09-21T12:39:00.013 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.013 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.013 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.013 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:00.013 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.807720+0000", 2024-09-21T12:39:00.014 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.46e-06 2024-09-21T12:39:00.014 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.014 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.014 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:00.014 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.807752+0000", 2024-09-21T12:39:00.014 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1424000000000003e-05 2024-09-21T12:39:00.014 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.014 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.014 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:00.014 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.807766+0000", 2024-09-21T12:39:00.014 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4498000000000001e-05 2024-09-21T12:39:00.014 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.014 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.014 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:00.015 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.807774+0000", 2024-09-21T12:39:00.015 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9780000000000005e-06 2024-09-21T12:39:00.015 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.015 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.015 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.015 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.808091+0000", 2024-09-21T12:39:00.015 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00031702100000000001 2024-09-21T12:39:00.015 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.015 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.015 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:00.015 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.808233+0000", 2024-09-21T12:39:00.015 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014201 2024-09-21T12:39:00.015 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.015 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.016 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,3", 2024-09-21T12:39:00.016 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.808361+0000", 2024-09-21T12:39:00.016 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000127813 2024-09-21T12:39:00.016 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.016 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.016 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.016 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.810940+0000", 2024-09-21T12:39:00.016 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0025795179999999998 2024-09-21T12:39:00.016 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.016 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.016 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:00.016 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.812556+0000", 2024-09-21T12:39:00.016 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001615184 2024-09-21T12:39:00.016 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.016 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.017 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.017 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:01.180397+0000", 2024-09-21T12:39:00.017 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.36784113099999999 2024-09-21T12:39:00.017 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.017 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.017 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:00.017 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:01.180472+0000", 2024-09-21T12:39:00.017 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5068999999999996e-05 2024-09-21T12:39:00.017 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.017 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.017 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:00.017 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:01.180519+0000", 2024-09-21T12:39:00.017 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7617999999999997e-05 2024-09-21T12:39:00.018 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.018 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:00.018 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.018 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.018 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.018 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:43148 10.f 10:f122d938:::benchmark_data_smithi114_161690_object43147:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-09-21T12:39:00.018 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:33:00.820371+0000", 2024-09-21T12:39:00.018 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 359.172489853, 2024-09-21T12:39:00.018 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.36041693699999999, 2024-09-21T12:39:00.018 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:00.018 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:00.019 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:00.019 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:00.019 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:39:00.019 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:39:00.019 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 43148 2024-09-21T12:39:00.019 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.019 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:00.019 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.019 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:00.019 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.820371+0000", 2024-09-21T12:39:00.019 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.019 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.019 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.020 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:00.020 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.820371+0000", 2024-09-21T12:39:00.020 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.020 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.020 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.020 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:00.020 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.820373+0000", 2024-09-21T12:39:00.020 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5290000000000001e-06 2024-09-21T12:39:00.020 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.020 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.020 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:00.020 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.820402+0000", 2024-09-21T12:39:00.020 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9646e-05 2024-09-21T12:39:00.020 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.021 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.021 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:00.021 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.820415+0000", 2024-09-21T12:39:00.021 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2773e-05 2024-09-21T12:39:00.021 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.021 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.021 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:00.021 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.820423+0000", 2024-09-21T12:39:00.021 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5279999999999998e-06 2024-09-21T12:39:00.021 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.021 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.021 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.022 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.820517+0000", 2024-09-21T12:39:00.022 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4477999999999999e-05 2024-09-21T12:39:00.022 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.022 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.022 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:00.022 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.820658+0000", 2024-09-21T12:39:00.022 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000140507 2024-09-21T12:39:00.022 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.022 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.022 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,3", 2024-09-21T12:39:00.022 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.820792+0000", 2024-09-21T12:39:00.022 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013416799999999999 2024-09-21T12:39:00.022 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.023 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.023 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:00.023 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.822512+0000", 2024-09-21T12:39:00.023 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001720023 2024-09-21T12:39:00.023 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.023 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.023 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.023 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:00.822900+0000", 2024-09-21T12:39:00.023 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00038832100000000001 2024-09-21T12:39:00.023 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.023 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.023 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.023 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:01.180699+0000", 2024-09-21T12:39:00.023 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.35779872600000001 2024-09-21T12:39:00.023 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.024 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.024 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:00.024 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:01.180751+0000", 2024-09-21T12:39:00.024 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2394e-05 2024-09-21T12:39:00.024 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.024 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.024 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:00.024 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:01.180788+0000", 2024-09-21T12:39:00.024 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6844000000000002e-05 2024-09-21T12:39:00.024 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.024 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:00.024 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.024 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.025 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.025 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:64539 10.0 10:0a4a2714:::benchmark_data_smithi114_161690_object64538:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-09-21T12:39:00.025 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:33:16.091235+0000", 2024-09-21T12:39:00.025 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 343.90162575800002, 2024-09-21T12:39:00.025 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.12744955099999999, 2024-09-21T12:39:00.026 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:00.026 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:00.026 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:00.026 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:00.026 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:39:00.026 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:39:00.026 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 64539 2024-09-21T12:39:00.026 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.026 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:00.026 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.026 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:00.026 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.091235+0000", 2024-09-21T12:39:00.026 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.027 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.027 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.027 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:00.027 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.091235+0000", 2024-09-21T12:39:00.027 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.027 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.027 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.027 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:00.027 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.091237+0000", 2024-09-21T12:39:00.027 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7090000000000001e-06 2024-09-21T12:39:00.027 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.027 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.027 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:00.027 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.091267+0000", 2024-09-21T12:39:00.028 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9697999999999999e-05 2024-09-21T12:39:00.028 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.028 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.028 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:00.028 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.091278+0000", 2024-09-21T12:39:00.028 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1452e-05 2024-09-21T12:39:00.028 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.028 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.028 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:00.028 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.091284+0000", 2024-09-21T12:39:00.028 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.327e-06 2024-09-21T12:39:00.028 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.028 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.029 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.029 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.091360+0000", 2024-09-21T12:39:00.029 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6860000000000006e-05 2024-09-21T12:39:00.029 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.029 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.029 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:00.029 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.091457+0000", 2024-09-21T12:39:00.029 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6650999999999993e-05 2024-09-21T12:39:00.029 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.029 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.029 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,4", 2024-09-21T12:39:00.029 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.091542+0000", 2024-09-21T12:39:00.029 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5358999999999994e-05 2024-09-21T12:39:00.029 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.030 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.030 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:00.030 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.093589+0000", 2024-09-21T12:39:00.030 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020467150000000002 2024-09-21T12:39:00.030 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.030 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.030 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.030 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.093785+0000", 2024-09-21T12:39:00.030 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000195484 2024-09-21T12:39:00.030 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.030 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.030 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.030 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.218607+0000", 2024-09-21T12:39:00.031 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.124822291 2024-09-21T12:39:00.031 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.031 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.031 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:00.031 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.218646+0000", 2024-09-21T12:39:00.031 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8739000000000003e-05 2024-09-21T12:39:00.031 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.031 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.031 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:00.031 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:33:16.218685+0000", 2024-09-21T12:39:00.031 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9266e-05 2024-09-21T12:39:00.031 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.031 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:00.031 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.032 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.032 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.032 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15924.0:160307 10.f 10:f936c05c:::benchmark_data_smithi114_161690_object33255:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-21T12:39:00.032 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:34:33.358322+0000", 2024-09-21T12:39:00.032 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 266.63453926300002, 2024-09-21T12:39:00.032 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.010655659, 2024-09-21T12:39:00.032 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:00.032 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:00.033 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:00.033 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:00.033 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15924", 2024-09-21T12:39:00.033 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/3591853", 2024-09-21T12:39:00.033 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 160307 2024-09-21T12:39:00.033 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.033 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:00.033 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.033 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:00.033 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.358322+0000", 2024-09-21T12:39:00.033 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.033 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.033 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.034 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:00.034 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.358322+0000", 2024-09-21T12:39:00.034 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.034 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.034 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.034 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:00.034 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.358324+0000", 2024-09-21T12:39:00.034 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6819999999999999e-06 2024-09-21T12:39:00.034 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.034 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.034 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:00.034 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.358329+0000", 2024-09-21T12:39:00.034 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2530000000000004e-06 2024-09-21T12:39:00.035 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.035 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.035 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:00.035 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.358342+0000", 2024-09-21T12:39:00.035 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3258000000000001e-05 2024-09-21T12:39:00.035 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.035 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.035 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:00.035 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.358349+0000", 2024-09-21T12:39:00.035 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8419999999999999e-06 2024-09-21T12:39:00.035 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.035 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.035 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.036 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.358395+0000", 2024-09-21T12:39:00.036 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6254000000000001e-05 2024-09-21T12:39:00.036 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.036 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.036 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:00.036 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:33.358402+0000", 2024-09-21T12:39:00.036 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5540000000000003e-06 2024-09-21T12:39:00.036 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.036 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.036 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.036 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.364082+0000", 2024-09-21T12:39:00.036 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.005680286 2024-09-21T12:39:00.036 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.036 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.037 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:00.037 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.364229+0000", 2024-09-21T12:39:00.037 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014693700000000001 2024-09-21T12:39:00.037 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.037 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.037 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3", 2024-09-21T12:39:00.037 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.364330+0000", 2024-09-21T12:39:00.037 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010115000000000001 2024-09-21T12:39:00.037 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.037 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.037 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:00.037 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.368799+0000", 2024-09-21T12:39:00.037 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0044687160000000002 2024-09-21T12:39:00.038 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.038 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.038 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.038 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.368853+0000", 2024-09-21T12:39:00.038 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.376e-05 2024-09-21T12:39:00.038 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.038 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.038 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:00.038 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.368906+0000", 2024-09-21T12:39:00.038 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3761999999999998e-05 2024-09-21T12:39:00.038 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.038 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.038 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:00.038 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:34:34.368978+0000", 2024-09-21T12:39:00.039 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1204999999999993e-05 2024-09-21T12:39:00.039 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.039 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:00.039 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.039 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.039 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.039 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:16847 11.4 11:27dda178:::benchmark_data_smithi114_187622_object16846:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-21T12:39:00.039 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:35:58.924832+0000", 2024-09-21T12:39:00.039 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 181.06802881199999, 2024-09-21T12:39:00.039 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14433790599999999, 2024-09-21T12:39:00.039 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:00.039 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:00.039 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:00.039 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:00.040 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:00.040 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:00.040 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 16847 2024-09-21T12:39:00.040 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.040 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:00.040 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.040 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:00.040 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.924832+0000", 2024-09-21T12:39:00.040 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.040 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.040 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.040 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:00.040 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.924832+0000", 2024-09-21T12:39:00.040 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.041 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.041 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.041 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:00.041 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.924834+0000", 2024-09-21T12:39:00.041 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.722e-06 2024-09-21T12:39:00.041 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.041 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.041 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:00.041 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.924885+0000", 2024-09-21T12:39:00.041 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1369000000000002e-05 2024-09-21T12:39:00.041 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.041 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.041 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:00.041 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.924906+0000", 2024-09-21T12:39:00.042 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0724999999999999e-05 2024-09-21T12:39:00.042 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.042 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.042 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:00.042 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.924913+0000", 2024-09-21T12:39:00.042 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2980000000000003e-06 2024-09-21T12:39:00.042 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.042 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.042 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.042 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.924966+0000", 2024-09-21T12:39:00.042 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2994000000000001e-05 2024-09-21T12:39:00.042 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.042 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.043 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:00.043 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.925099+0000", 2024-09-21T12:39:00.043 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000132733 2024-09-21T12:39:00.043 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.043 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.043 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:39:00.043 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.925248+0000", 2024-09-21T12:39:00.043 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000149059 2024-09-21T12:39:00.043 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.043 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.043 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:00.043 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.926325+0000", 2024-09-21T12:39:00.043 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00107649 2024-09-21T12:39:00.043 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.043 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.044 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.044 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:58.929258+0000", 2024-09-21T12:39:00.044 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0029329260000000002 2024-09-21T12:39:00.044 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.044 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.044 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.044 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:59.068980+0000", 2024-09-21T12:39:00.044 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13972220599999999 2024-09-21T12:39:00.044 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.044 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.044 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:00.044 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:59.069048+0000", 2024-09-21T12:39:00.044 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8336000000000004e-05 2024-09-21T12:39:00.044 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.045 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.045 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:00.045 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:59.069170+0000", 2024-09-21T12:39:00.045 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012204799999999999 2024-09-21T12:39:00.045 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.045 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:00.045 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.045 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.045 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.045 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:17074 11.4 11.189f0fc4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-09-21T12:39:00.045 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:35:59.165067+0000", 2024-09-21T12:39:00.045 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 180.82779451299999, 2024-09-21T12:39:00.045 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.95632275099999997, 2024-09-21T12:39:00.045 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:00.046 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:00.046 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-09-21T12:39:00.046 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:00.046 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:00.046 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:00.046 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 17074 2024-09-21T12:39:00.046 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.046 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:00.046 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.046 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:00.046 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:59.165067+0000", 2024-09-21T12:39:00.046 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.046 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.047 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.047 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:00.047 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:59.165067+0000", 2024-09-21T12:39:00.047 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.047 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.047 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.047 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:00.047 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:59.165068+0000", 2024-09-21T12:39:00.047 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5650000000000001e-06 2024-09-21T12:39:00.047 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.047 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.047 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:00.047 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:59.165104+0000", 2024-09-21T12:39:00.047 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6115000000000002e-05 2024-09-21T12:39:00.048 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.048 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.048 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:00.048 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:59.165112+0000", 2024-09-21T12:39:00.048 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4159999999999998e-06 2024-09-21T12:39:00.048 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.048 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.048 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:00.048 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:59.165116+0000", 2024-09-21T12:39:00.048 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6609999999999999e-06 2024-09-21T12:39:00.048 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.048 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.048 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.048 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:59.165281+0000", 2024-09-21T12:39:00.049 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016428100000000001 2024-09-21T12:39:00.049 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.049 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.049 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:00.049 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:35:59.165325+0000", 2024-09-21T12:39:00.049 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3918999999999997e-05 2024-09-21T12:39:00.049 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.049 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.049 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.049 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:00.121378+0000", 2024-09-21T12:39:00.049 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.95605380299999998 2024-09-21T12:39:00.049 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.049 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.049 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:00.050 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:00.121389+0000", 2024-09-21T12:39:00.050 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0991e-05 2024-09-21T12:39:00.050 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.050 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:00.050 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.050 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.050 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.050 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:34459 11.3 11:ca74ae22:::benchmark_data_smithi114_187622_object34458:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-09-21T12:39:00.050 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:36:12.115486+0000", 2024-09-21T12:39:00.050 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 167.87737465800001, 2024-09-21T12:39:00.050 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.99501541699999996, 2024-09-21T12:39:00.050 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:00.050 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:00.050 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:00.051 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:00.051 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:00.051 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:00.051 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 34459 2024-09-21T12:39:00.051 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.051 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:00.051 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.051 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:00.051 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115486+0000", 2024-09-21T12:39:00.051 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.051 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.051 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.051 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:00.052 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115486+0000", 2024-09-21T12:39:00.052 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.052 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.052 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.052 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:00.052 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115488+0000", 2024-09-21T12:39:00.052 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4270000000000001e-06 2024-09-21T12:39:00.052 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.052 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.052 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:00.052 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115619+0000", 2024-09-21T12:39:00.052 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013127500000000001 2024-09-21T12:39:00.052 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.053 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.053 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:00.053 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115631+0000", 2024-09-21T12:39:00.053 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1894e-05 2024-09-21T12:39:00.053 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.053 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.053 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:00.053 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115640+0000", 2024-09-21T12:39:00.053 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8789999999999992e-06 2024-09-21T12:39:00.053 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.053 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.053 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.053 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115711+0000", 2024-09-21T12:39:00.053 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0730999999999998e-05 2024-09-21T12:39:00.054 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.054 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.054 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:00.054 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115720+0000", 2024-09-21T12:39:00.054 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0669999999999998e-06 2024-09-21T12:39:00.054 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.054 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.054 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.054 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.107366+0000", 2024-09-21T12:39:00.054 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.99164616100000003 2024-09-21T12:39:00.054 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.054 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.055 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:00.055 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.107531+0000", 2024-09-21T12:39:00.055 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016521400000000001 2024-09-21T12:39:00.055 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.055 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.055 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:39:00.055 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.107671+0000", 2024-09-21T12:39:00.055 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000139915 2024-09-21T12:39:00.055 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.055 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.055 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.055 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.110368+0000", 2024-09-21T12:39:00.055 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002696693 2024-09-21T12:39:00.055 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.056 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.056 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:00.056 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.110378+0000", 2024-09-21T12:39:00.056 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0263e-05 2024-09-21T12:39:00.057 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.057 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.057 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.057 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.110416+0000", 2024-09-21T12:39:00.057 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7540999999999999e-05 2024-09-21T12:39:00.058 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.058 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.058 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:00.058 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.110469+0000", 2024-09-21T12:39:00.058 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3103999999999998e-05 2024-09-21T12:39:00.058 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.058 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.058 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:00.058 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.110502+0000", 2024-09-21T12:39:00.058 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3253000000000002e-05 2024-09-21T12:39:00.058 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.058 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:00.058 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.058 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.059 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.059 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:34460 11.3 11:c0bd347b:::benchmark_data_smithi114_187622_object34459:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-09-21T12:39:00.059 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:36:12.115839+0000", 2024-09-21T12:39:00.059 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 167.87702245200001, 2024-09-21T12:39:00.059 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.99728419700000004, 2024-09-21T12:39:00.059 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:00.059 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:00.059 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:00.059 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:00.059 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:00.059 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:00.059 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 34460 2024-09-21T12:39:00.059 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.060 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:00.060 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.060 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:00.060 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115839+0000", 2024-09-21T12:39:00.060 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.060 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.060 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.060 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:00.060 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115839+0000", 2024-09-21T12:39:00.060 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.060 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.060 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.060 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:00.060 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115840+0000", 2024-09-21T12:39:00.060 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7000000000000003e-07 2024-09-21T12:39:00.061 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.061 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.061 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:00.061 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115906+0000", 2024-09-21T12:39:00.061 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6925e-05 2024-09-21T12:39:00.061 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.061 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.061 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:00.061 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115918+0000", 2024-09-21T12:39:00.061 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1385e-05 2024-09-21T12:39:00.061 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.061 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.061 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:00.062 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115925+0000", 2024-09-21T12:39:00.062 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8040000000000004e-06 2024-09-21T12:39:00.062 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.062 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.062 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.062 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115973+0000", 2024-09-21T12:39:00.062 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8804999999999998e-05 2024-09-21T12:39:00.062 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.062 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.062 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:00.062 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.115986+0000", 2024-09-21T12:39:00.062 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2918000000000001e-05 2024-09-21T12:39:00.062 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.062 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.063 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.063 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.109359+0000", 2024-09-21T12:39:00.063 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.99337215800000001 2024-09-21T12:39:00.063 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.063 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.063 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:00.063 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.109454+0000", 2024-09-21T12:39:00.063 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5828999999999993e-05 2024-09-21T12:39:00.063 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.063 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.063 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:39:00.063 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.109547+0000", 2024-09-21T12:39:00.063 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2757999999999995e-05 2024-09-21T12:39:00.063 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.064 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.064 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:00.064 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.112914+0000", 2024-09-21T12:39:00.064 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0033664300000000001 2024-09-21T12:39:00.064 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.064 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.064 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.064 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.112976+0000", 2024-09-21T12:39:00.064 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2454999999999997e-05 2024-09-21T12:39:00.064 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.064 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.064 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.064 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.113046+0000", 2024-09-21T12:39:00.065 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9919000000000006e-05 2024-09-21T12:39:00.065 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.065 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.065 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:00.065 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.113096+0000", 2024-09-21T12:39:00.065 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9969000000000002e-05 2024-09-21T12:39:00.065 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.065 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.065 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:00.065 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.113123+0000", 2024-09-21T12:39:00.065 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6971999999999999e-05 2024-09-21T12:39:00.065 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.065 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:00.066 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.066 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.066 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.066 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:34461 11.3 11:c2a3c0ef:::benchmark_data_smithi114_187622_object34460:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-09-21T12:39:00.066 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:36:12.116148+0000", 2024-09-21T12:39:00.066 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 167.876713403, 2024-09-21T12:39:00.066 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0049646350000001, 2024-09-21T12:39:00.066 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:00.066 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:00.066 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:00.066 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:00.066 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:00.066 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:00.067 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 34461 2024-09-21T12:39:00.067 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.067 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:00.067 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.067 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:00.067 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.116148+0000", 2024-09-21T12:39:00.067 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.067 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.067 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.067 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:00.067 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.116148+0000", 2024-09-21T12:39:00.067 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.067 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.067 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.068 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:00.068 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.116148+0000", 2024-09-21T12:39:00.068 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5799999999999998e-07 2024-09-21T12:39:00.068 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.068 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.068 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:00.068 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.116220+0000", 2024-09-21T12:39:00.068 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1686000000000004e-05 2024-09-21T12:39:00.068 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.068 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.068 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:00.068 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.116230+0000", 2024-09-21T12:39:00.068 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8819999999999996e-06 2024-09-21T12:39:00.068 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.069 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.069 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:00.069 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.116236+0000", 2024-09-21T12:39:00.069 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2620000000000004e-06 2024-09-21T12:39:00.069 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.069 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.069 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.069 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:39:00.070 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:39:00.071 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:39:00.071 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.116276+0000", 2024-09-21T12:39:00.071 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9833000000000003e-05 2024-09-21T12:39:00.071 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.072 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:00.072 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.116284+0000", 2024-09-21T12:39:00.072 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5689999999999999e-06 2024-09-21T12:39:00.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.072 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.072 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.110628+0000", 2024-09-21T12:39:00.072 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.99434418899999999 2024-09-21T12:39:00.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.072 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:00.072 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.110769+0000", 2024-09-21T12:39:00.072 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000141354 2024-09-21T12:39:00.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.073 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.073 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:39:00.073 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.110926+0000", 2024-09-21T12:39:00.073 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015713899999999999 2024-09-21T12:39:00.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.073 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.073 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:00.073 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.113140+0000", 2024-09-21T12:39:00.073 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00221383 2024-09-21T12:39:00.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.073 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.073 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.073 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.113204+0000", 2024-09-21T12:39:00.074 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4039000000000001e-05 2024-09-21T12:39:00.074 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.074 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.074 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.074 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.121005+0000", 2024-09-21T12:39:00.074 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0078011649999999997 2024-09-21T12:39:00.074 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.074 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.074 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:00.074 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.121067+0000", 2024-09-21T12:39:00.074 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1832e-05 2024-09-21T12:39:00.074 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.074 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.074 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:00.074 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.121112+0000", 2024-09-21T12:39:00.075 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5096999999999999e-05 2024-09-21T12:39:00.075 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.075 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:00.075 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.075 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.075 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.075 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:34467 11.3 11:c8c61d4a:::benchmark_data_smithi114_187622_object34466:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-09-21T12:39:00.075 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:36:12.125970+0000", 2024-09-21T12:39:00.075 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 167.866890931, 2024-09-21T12:39:00.075 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.99832942599999996, 2024-09-21T12:39:00.075 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:00.075 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:00.075 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:00.075 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:00.076 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:00.076 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:00.076 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 34467 2024-09-21T12:39:00.076 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.076 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:00.076 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.076 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:00.076 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.125970+0000", 2024-09-21T12:39:00.076 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.076 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.076 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.076 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:00.076 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.125970+0000", 2024-09-21T12:39:00.077 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.077 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.077 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:00.077 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.125972+0000", 2024-09-21T12:39:00.077 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8050000000000001e-06 2024-09-21T12:39:00.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.077 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.077 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:00.077 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.126024+0000", 2024-09-21T12:39:00.077 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1699000000000001e-05 2024-09-21T12:39:00.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.077 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.077 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:00.078 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.126035+0000", 2024-09-21T12:39:00.078 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1764e-05 2024-09-21T12:39:00.078 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.078 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.078 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:00.078 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.126041+0000", 2024-09-21T12:39:00.078 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5999999999999997e-06 2024-09-21T12:39:00.078 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.078 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.078 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.078 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.126136+0000", 2024-09-21T12:39:00.078 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4429e-05 2024-09-21T12:39:00.078 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.079 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.079 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:00.079 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.126146+0000", 2024-09-21T12:39:00.079 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0387999999999999e-05 2024-09-21T12:39:00.079 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.079 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.079 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.079 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.111722+0000", 2024-09-21T12:39:00.079 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.98557621399999995 2024-09-21T12:39:00.079 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.079 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.079 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:00.080 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.111856+0000", 2024-09-21T12:39:00.080 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013419199999999999 2024-09-21T12:39:00.080 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.080 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.080 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:39:00.080 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.112031+0000", 2024-09-21T12:39:00.080 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000174439 2024-09-21T12:39:00.080 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.080 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.080 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.080 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.113254+0000", 2024-09-21T12:39:00.080 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012237319999999999 2024-09-21T12:39:00.080 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.080 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.081 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:00.081 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.113263+0000", 2024-09-21T12:39:00.081 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2369999999999992e-06 2024-09-21T12:39:00.081 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.081 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.081 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.081 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.124195+0000", 2024-09-21T12:39:00.081 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.010932457 2024-09-21T12:39:00.081 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.081 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.081 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:00.081 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.124258+0000", 2024-09-21T12:39:00.081 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3283999999999999e-05 2024-09-21T12:39:00.082 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.082 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.082 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:00.082 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.124300+0000", 2024-09-21T12:39:00.082 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1186000000000002e-05 2024-09-21T12:39:00.082 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.082 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:00.082 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.082 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.082 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.082 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:34474 11.3 11:c74a366c:::benchmark_data_smithi114_187622_object34473:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-09-21T12:39:00.082 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:36:12.128588+0000", 2024-09-21T12:39:00.082 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 167.864272973, 2024-09-21T12:39:00.083 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.99583373799999997, 2024-09-21T12:39:00.083 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:00.083 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:00.083 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:00.083 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:00.083 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:00.083 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:00.083 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 34474 2024-09-21T12:39:00.083 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.083 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:00.083 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.083 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:00.083 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.128588+0000", 2024-09-21T12:39:00.083 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.083 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.084 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.084 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:00.084 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.128588+0000", 2024-09-21T12:39:00.084 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.084 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.084 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.084 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:00.084 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.128590+0000", 2024-09-21T12:39:00.084 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5060000000000001e-06 2024-09-21T12:39:00.084 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.084 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.084 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:00.084 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.128660+0000", 2024-09-21T12:39:00.084 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9892000000000001e-05 2024-09-21T12:39:00.085 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.085 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.085 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:00.085 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.128673+0000", 2024-09-21T12:39:00.085 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3859000000000001e-05 2024-09-21T12:39:00.085 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.085 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.085 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:00.085 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.128680+0000", 2024-09-21T12:39:00.085 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.888e-06 2024-09-21T12:39:00.085 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.085 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.085 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.085 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.128741+0000", 2024-09-21T12:39:00.085 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0501999999999998e-05 2024-09-21T12:39:00.086 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.086 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.086 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:00.086 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.128751+0000", 2024-09-21T12:39:00.086 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0581000000000001e-05 2024-09-21T12:39:00.086 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.086 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.086 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.086 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.113279+0000", 2024-09-21T12:39:00.086 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.98452719099999997 2024-09-21T12:39:00.086 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.086 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.086 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:00.087 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.113377+0000", 2024-09-21T12:39:00.087 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8591999999999994e-05 2024-09-21T12:39:00.087 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.087 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.087 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:39:00.087 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.113469+0000", 2024-09-21T12:39:00.087 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1896000000000002e-05 2024-09-21T12:39:00.087 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.087 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.087 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.087 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.116868+0000", 2024-09-21T12:39:00.087 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0033991070000000002 2024-09-21T12:39:00.087 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.087 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.087 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:00.088 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.118613+0000", 2024-09-21T12:39:00.088 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0017451529999999999 2024-09-21T12:39:00.088 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.088 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.088 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.088 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.124339+0000", 2024-09-21T12:39:00.088 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0057257920000000004 2024-09-21T12:39:00.088 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.088 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.088 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:00.088 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.124386+0000", 2024-09-21T12:39:00.088 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6699000000000001e-05 2024-09-21T12:39:00.088 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.088 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.089 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:00.089 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.124422+0000", 2024-09-21T12:39:00.089 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.608e-05 2024-09-21T12:39:00.089 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.089 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:00.089 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.089 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.089 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.089 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:34483 11.3 11:ce099e96:::benchmark_data_smithi114_187622_object34482:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-09-21T12:39:00.089 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:36:12.139319+0000", 2024-09-21T12:39:00.089 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 167.85354257700001, 2024-09-21T12:39:00.089 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.98632494900000001, 2024-09-21T12:39:00.089 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:00.089 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:00.090 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:00.090 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:00.090 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:00.090 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:00.090 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 34483 2024-09-21T12:39:00.090 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.090 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:00.090 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.090 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:00.090 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.139319+0000", 2024-09-21T12:39:00.090 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.090 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.090 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.091 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:00.091 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.139319+0000", 2024-09-21T12:39:00.091 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.091 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.091 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.091 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:00.091 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.139321+0000", 2024-09-21T12:39:00.091 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9870000000000002e-06 2024-09-21T12:39:00.091 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.091 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.091 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:00.091 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.139360+0000", 2024-09-21T12:39:00.091 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9413999999999999e-05 2024-09-21T12:39:00.091 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.092 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.092 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:00.092 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.139376+0000", 2024-09-21T12:39:00.092 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5662999999999999e-05 2024-09-21T12:39:00.092 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.092 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.092 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:00.092 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.139384+0000", 2024-09-21T12:39:00.092 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2060000000000007e-06 2024-09-21T12:39:00.092 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.092 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.092 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.092 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.139441+0000", 2024-09-21T12:39:00.092 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7043999999999998e-05 2024-09-21T12:39:00.093 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.093 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.093 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:00.093 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.139453+0000", 2024-09-21T12:39:00.093 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1671999999999999e-05 2024-09-21T12:39:00.093 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.093 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.093 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.093 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.114270+0000", 2024-09-21T12:39:00.093 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.97481730600000005 2024-09-21T12:39:00.093 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.093 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.093 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:00.093 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.114384+0000", 2024-09-21T12:39:00.093 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011422299999999999 2024-09-21T12:39:00.093 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.094 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.094 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:39:00.094 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.114501+0000", 2024-09-21T12:39:00.094 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011684999999999999 2024-09-21T12:39:00.094 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.094 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.094 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.094 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.120883+0000", 2024-09-21T12:39:00.094 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0063821340000000002 2024-09-21T12:39:00.094 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.094 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.094 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:00.094 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.122831+0000", 2024-09-21T12:39:00.094 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0019476070000000001 2024-09-21T12:39:00.095 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.095 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.095 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.095 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.124475+0000", 2024-09-21T12:39:00.095 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001643931 2024-09-21T12:39:00.095 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.095 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.095 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:00.095 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.125606+0000", 2024-09-21T12:39:00.095 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001131284 2024-09-21T12:39:00.095 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.095 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.095 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:00.095 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.125644+0000", 2024-09-21T12:39:00.095 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7628e-05 2024-09-21T12:39:00.096 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.096 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:00.096 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.096 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.096 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.096 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:34502 11.3 11:c044b555:::benchmark_data_smithi114_187622_object34501:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-09-21T12:39:00.096 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:36:12.146909+0000", 2024-09-21T12:39:00.096 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 167.845951818, 2024-09-21T12:39:00.096 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.97888777299999996, 2024-09-21T12:39:00.096 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:00.096 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:00.097 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:00.097 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:00.097 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:00.097 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:00.097 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 34502 2024-09-21T12:39:00.097 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.097 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:00.097 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.097 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:00.097 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.146909+0000", 2024-09-21T12:39:00.097 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.097 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.097 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.098 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:00.098 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.146909+0000", 2024-09-21T12:39:00.098 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.098 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.098 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.098 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:00.098 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.146911+0000", 2024-09-21T12:39:00.098 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5409999999999999e-06 2024-09-21T12:39:00.098 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.098 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.098 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:00.098 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.146948+0000", 2024-09-21T12:39:00.098 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7619e-05 2024-09-21T12:39:00.098 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.099 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.099 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:00.099 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.146961+0000", 2024-09-21T12:39:00.099 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2254e-05 2024-09-21T12:39:00.099 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.099 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.099 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:00.099 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.146968+0000", 2024-09-21T12:39:00.099 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0469999999999996e-06 2024-09-21T12:39:00.099 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.099 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.099 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.099 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.147037+0000", 2024-09-21T12:39:00.099 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9210999999999999e-05 2024-09-21T12:39:00.100 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.100 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.100 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:00.100 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.147047+0000", 2024-09-21T12:39:00.100 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6460000000000005e-06 2024-09-21T12:39:00.100 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.100 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.100 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.100 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.115516+0000", 2024-09-21T12:39:00.100 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.96846887999999998 2024-09-21T12:39:00.100 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.100 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.100 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:00.100 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.115620+0000", 2024-09-21T12:39:00.101 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000104876 2024-09-21T12:39:00.101 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.101 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.101 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:39:00.101 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.115705+0000", 2024-09-21T12:39:00.101 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4994999999999995e-05 2024-09-21T12:39:00.101 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.101 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.101 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.101 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.120929+0000", 2024-09-21T12:39:00.101 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0052235110000000001 2024-09-21T12:39:00.101 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.101 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.101 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:00.102 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.122835+0000", 2024-09-21T12:39:00.102 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0019065779999999999 2024-09-21T12:39:00.102 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.102 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.102 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.102 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.125706+0000", 2024-09-21T12:39:00.102 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0028708330000000002 2024-09-21T12:39:00.102 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.102 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.102 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:00.102 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.125759+0000", 2024-09-21T12:39:00.102 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2596000000000003e-05 2024-09-21T12:39:00.103 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.103 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.103 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:00.103 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.125797+0000", 2024-09-21T12:39:00.103 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8186000000000003e-05 2024-09-21T12:39:00.103 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.103 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:00.103 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.103 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.103 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.103 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:34533 11.3 11:cd9680cc:::benchmark_data_smithi114_187622_object34532:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-09-21T12:39:00.103 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:36:12.224372+0000", 2024-09-21T12:39:00.103 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 167.76848882600001, 2024-09-21T12:39:00.103 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.90168315700000001, 2024-09-21T12:39:00.103 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:00.104 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:00.104 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:00.104 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:00.104 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:00.104 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:00.104 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 34533 2024-09-21T12:39:00.104 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.104 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:00.104 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.104 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:00.104 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.224372+0000", 2024-09-21T12:39:00.104 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.104 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.104 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.104 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:00.105 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.224372+0000", 2024-09-21T12:39:00.105 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.105 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.105 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.105 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:00.105 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.224382+0000", 2024-09-21T12:39:00.105 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7000000000000003e-06 2024-09-21T12:39:00.105 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.105 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.105 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:00.105 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.224432+0000", 2024-09-21T12:39:00.105 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0099000000000003e-05 2024-09-21T12:39:00.105 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.105 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.105 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:00.106 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.224443+0000", 2024-09-21T12:39:00.106 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0525e-05 2024-09-21T12:39:00.106 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.106 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.106 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:00.106 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.224449+0000", 2024-09-21T12:39:00.106 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9780000000000002e-06 2024-09-21T12:39:00.106 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.106 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.106 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.106 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.224500+0000", 2024-09-21T12:39:00.106 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1669000000000003e-05 2024-09-21T12:39:00.106 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.107 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.107 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:00.107 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.224507+0000", 2024-09-21T12:39:00.107 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5529999999999998e-06 2024-09-21T12:39:00.107 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.107 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.107 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.107 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.116961+0000", 2024-09-21T12:39:00.107 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.89245421599999997 2024-09-21T12:39:00.107 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.107 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.107 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:00.107 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.117083+0000", 2024-09-21T12:39:00.107 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012155 2024-09-21T12:39:00.108 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.108 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.108 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:39:00.108 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.117234+0000", 2024-09-21T12:39:00.108 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015182300000000001 2024-09-21T12:39:00.108 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.108 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.108 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:00.108 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.122842+0000", 2024-09-21T12:39:00.108 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0056078320000000001 2024-09-21T12:39:00.108 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.108 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.108 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.108 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.125852+0000", 2024-09-21T12:39:00.109 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0030097890000000001 2024-09-21T12:39:00.109 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.109 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.109 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.109 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.125972+0000", 2024-09-21T12:39:00.109 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012015000000000001 2024-09-21T12:39:00.109 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.109 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.109 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:00.109 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.126021+0000", 2024-09-21T12:39:00.109 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8993999999999999e-05 2024-09-21T12:39:00.109 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.109 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.109 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:00.109 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.126055+0000", 2024-09-21T12:39:00.110 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4279000000000002e-05 2024-09-21T12:39:00.110 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.110 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:00.110 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.110 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.110 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.110 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:34535 11.3 11:c8ed66ca:::benchmark_data_smithi114_187622_object34534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-09-21T12:39:00.110 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:36:12.227358+0000", 2024-09-21T12:39:00.110 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 167.76550363600001, 2024-09-21T12:39:00.110 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.90690378699999996, 2024-09-21T12:39:00.110 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:00.110 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:00.110 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:00.111 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:00.111 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:00.111 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:00.111 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 34535 2024-09-21T12:39:00.111 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.111 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:00.111 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.111 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:00.111 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.227358+0000", 2024-09-21T12:39:00.111 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.111 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.111 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.111 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:00.111 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.227358+0000", 2024-09-21T12:39:00.111 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.112 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.112 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.112 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:00.112 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.227359+0000", 2024-09-21T12:39:00.112 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2669999999999999e-06 2024-09-21T12:39:00.112 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.112 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.112 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:00.112 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.227388+0000", 2024-09-21T12:39:00.112 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9192000000000001e-05 2024-09-21T12:39:00.112 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.112 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.112 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:00.113 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.227399+0000", 2024-09-21T12:39:00.113 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1494e-05 2024-09-21T12:39:00.113 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.113 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.113 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:00.113 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.227405+0000", 2024-09-21T12:39:00.113 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5860000000000004e-06 2024-09-21T12:39:00.113 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.113 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.113 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.113 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.227450+0000", 2024-09-21T12:39:00.113 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.545e-05 2024-09-21T12:39:00.113 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.113 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.113 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:00.114 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:12.227456+0000", 2024-09-21T12:39:00.114 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.665e-06 2024-09-21T12:39:00.114 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.114 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.114 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.114 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.119175+0000", 2024-09-21T12:39:00.114 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.89171925100000005 2024-09-21T12:39:00.114 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.114 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.114 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:00.114 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.119306+0000", 2024-09-21T12:39:00.114 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013061099999999999 2024-09-21T12:39:00.114 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.114 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.115 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:39:00.115 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.119472+0000", 2024-09-21T12:39:00.115 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016553499999999999 2024-09-21T12:39:00.115 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.115 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.115 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:00.115 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.123128+0000", 2024-09-21T12:39:00.115 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003655957 2024-09-21T12:39:00.115 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.115 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.115 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.115 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.125911+0000", 2024-09-21T12:39:00.115 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0027830950000000002 2024-09-21T12:39:00.115 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.116 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.116 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.116 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.134151+0000", 2024-09-21T12:39:00.116 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0082405670000000007 2024-09-21T12:39:00.116 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.116 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.116 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:00.116 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.134217+0000", 2024-09-21T12:39:00.116 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5687999999999999e-05 2024-09-21T12:39:00.116 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.116 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.116 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:00.116 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:36:13.134261+0000", 2024-09-21T12:39:00.116 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4428999999999998e-05 2024-09-21T12:39:00.116 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.117 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:00.117 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.117 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.117 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.117 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:136649 11.3 11:cddb3060:::benchmark_data_smithi114_187622_object136648:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-09-21T12:39:00.117 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:37:22.242348+0000", 2024-09-21T12:39:00.117 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 97.750513135000006, 2024-09-21T12:39:00.117 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.130686786, 2024-09-21T12:39:00.117 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:00.117 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:00.117 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:00.117 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:00.117 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:00.117 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:00.118 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 136649 2024-09-21T12:39:00.118 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.118 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:00.118 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.118 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:00.118 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.242348+0000", 2024-09-21T12:39:00.118 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.118 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.118 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.118 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:00.118 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.242348+0000", 2024-09-21T12:39:00.118 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.118 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.118 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.119 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:00.119 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.242349+0000", 2024-09-21T12:39:00.119 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2500000000000001e-06 2024-09-21T12:39:00.119 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.119 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.119 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:00.119 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.242395+0000", 2024-09-21T12:39:00.119 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6026999999999998e-05 2024-09-21T12:39:00.119 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.119 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.119 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:00.119 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.242405+0000", 2024-09-21T12:39:00.120 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5519999999999993e-06 2024-09-21T12:39:00.120 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.120 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.120 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:00.120 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.242410+0000", 2024-09-21T12:39:00.120 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2270000000000001e-06 2024-09-21T12:39:00.120 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.120 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.120 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.120 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.242455+0000", 2024-09-21T12:39:00.120 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4888999999999997e-05 2024-09-21T12:39:00.120 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.120 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.121 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:00.121 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.242553+0000", 2024-09-21T12:39:00.121 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8227000000000007e-05 2024-09-21T12:39:00.121 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.121 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.121 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:39:00.121 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.242638+0000", 2024-09-21T12:39:00.121 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4963000000000006e-05 2024-09-21T12:39:00.121 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.121 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.121 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:00.121 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.246352+0000", 2024-09-21T12:39:00.121 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0037137450000000001 2024-09-21T12:39:00.121 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.122 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.122 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.122 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.248100+0000", 2024-09-21T12:39:00.122 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0017482450000000001 2024-09-21T12:39:00.122 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.122 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.122 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.122 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.372877+0000", 2024-09-21T12:39:00.122 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.124776927 2024-09-21T12:39:00.122 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.122 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.122 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:00.122 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.372986+0000", 2024-09-21T12:39:00.122 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000108646 2024-09-21T12:39:00.123 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.123 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.123 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:00.123 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.373035+0000", 2024-09-21T12:39:00.123 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9088000000000002e-05 2024-09-21T12:39:00.123 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.123 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:00.123 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.123 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.123 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.123 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:136650 11.3 11:ce8eedbf:::benchmark_data_smithi114_187622_object136649:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-09-21T12:39:00.123 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:37:22.242424+0000", 2024-09-21T12:39:00.123 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 97.750436922999995, 2024-09-21T12:39:00.123 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.130759292, 2024-09-21T12:39:00.124 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:00.124 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:00.124 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:00.124 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:00.124 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:00.124 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:00.124 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 136650 2024-09-21T12:39:00.124 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.124 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:00.124 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.124 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:00.124 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.242424+0000", 2024-09-21T12:39:00.124 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.124 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.125 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.125 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:00.125 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.242424+0000", 2024-09-21T12:39:00.125 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.125 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.125 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.125 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:00.125 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.242425+0000", 2024-09-21T12:39:00.125 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8500000000000002e-07 2024-09-21T12:39:00.125 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.125 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.125 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:00.125 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.242501+0000", 2024-09-21T12:39:00.125 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6328000000000005e-05 2024-09-21T12:39:00.125 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.126 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.126 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:00.126 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.242506+0000", 2024-09-21T12:39:00.126 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4689999999999996e-06 2024-09-21T12:39:00.126 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.126 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.126 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:00.126 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.242510+0000", 2024-09-21T12:39:00.126 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5920000000000001e-06 2024-09-21T12:39:00.126 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.126 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.126 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.126 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.244455+0000", 2024-09-21T12:39:00.127 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001944519 2024-09-21T12:39:00.127 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.127 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.127 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:00.127 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.244547+0000", 2024-09-21T12:39:00.127 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2243000000000004e-05 2024-09-21T12:39:00.127 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.127 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.127 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:39:00.127 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.244676+0000", 2024-09-21T12:39:00.127 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012941499999999999 2024-09-21T12:39:00.127 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.127 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.127 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:00.127 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.246433+0000", 2024-09-21T12:39:00.127 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0017571799999999999 2024-09-21T12:39:00.128 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.128 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.128 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.128 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.249840+0000", 2024-09-21T12:39:00.128 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003406999 2024-09-21T12:39:00.128 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.128 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.128 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.128 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.373095+0000", 2024-09-21T12:39:00.128 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.123254802 2024-09-21T12:39:00.128 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.128 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.128 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:00.128 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.373148+0000", 2024-09-21T12:39:00.128 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2561e-05 2024-09-21T12:39:00.129 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.129 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.129 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:00.129 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:37:22.373184+0000", 2024-09-21T12:39:00.129 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5799000000000001e-05 2024-09-21T12:39:00.129 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.129 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:00.129 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.129 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.129 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.129 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:202398 11.3 11:c2b8b545:::benchmark_data_smithi114_187622_object64277:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e457)", 2024-09-21T12:39:00.129 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:38:08.622842+0000", 2024-09-21T12:39:00.129 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 51.370018983999998, 2024-09-21T12:39:00.129 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.106672643, 2024-09-21T12:39:00.129 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:00.130 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:00.130 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:00.130 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:00.130 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:00.130 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:00.130 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 202398 2024-09-21T12:39:00.130 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.130 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:00.130 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.130 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:00.130 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:08.622842+0000", 2024-09-21T12:39:00.130 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.131 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.131 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.131 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:00.131 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:08.622842+0000", 2024-09-21T12:39:00.131 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:00.131 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.131 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.131 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:00.131 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:08.622843+0000", 2024-09-21T12:39:00.131 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.269e-06 2024-09-21T12:39:00.131 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.131 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.131 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:00.131 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:08.622848+0000", 2024-09-21T12:39:00.132 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1060000000000002e-06 2024-09-21T12:39:00.132 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.132 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.132 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:00.132 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:08.622858+0000", 2024-09-21T12:39:00.132 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.062e-05 2024-09-21T12:39:00.132 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.132 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.132 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:00.132 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:08.622863+0000", 2024-09-21T12:39:00.132 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2390000000000002e-06 2024-09-21T12:39:00.132 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.132 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.132 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:00.133 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:08.622920+0000", 2024-09-21T12:39:00.133 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6344999999999997e-05 2024-09-21T12:39:00.133 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.133 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.133 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:00.133 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:08.623031+0000", 2024-09-21T12:39:00.133 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000111537 2024-09-21T12:39:00.133 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.133 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.133 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,7", 2024-09-21T12:39:00.133 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:08.623134+0000", 2024-09-21T12:39:00.133 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010253 2024-09-21T12:39:00.133 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.133 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.133 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:00.133 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:08.627768+0000", 2024-09-21T12:39:00.134 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0046339620000000002 2024-09-21T12:39:00.134 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.134 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.134 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.134 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:08.628030+0000", 2024-09-21T12:39:00.134 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00026269200000000003 2024-09-21T12:39:00.134 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.134 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.134 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:00.134 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:08.729435+0000", 2024-09-21T12:39:00.134 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.101404929 2024-09-21T12:39:00.134 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.134 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.134 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:00.134 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:08.729477+0000", 2024-09-21T12:39:00.134 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1779000000000001e-05 2024-09-21T12:39:00.135 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:00.135 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:00.135 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:00.135 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:08.729515+0000", 2024-09-21T12:39:00.135 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7635000000000001e-05 2024-09-21T12:39:00.135 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.135 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:00.135 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.135 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:00.135 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:00.135 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:39:00.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:00.137 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:00.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:39:00.238 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:39:00.266 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:00.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:00.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:39:00.368 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:39:00.410 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:00.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:00.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:39:00.512 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:39:00.548 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:00.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:00.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:39:00.650 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:39:00.688 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:00.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:00.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:39:00.789 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:39:00.811 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:00.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:00.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:39:00.912 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:39:00.950 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:00.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:00.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:39:01.038 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T12:39:01.052 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:39:01.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:01.090 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:01.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:39:01.190 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:39:01.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:01.226 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:01.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:39:01.327 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:39:01.365 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T12:39:01.378 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:01.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:01.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:39:01.408 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:39:01.480 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:39:01.515 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:01.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:01.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:39:01.616 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:39:01.646 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:01.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:01.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:39:01.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:01 smithi183 ceph-mon[33277]: pgmap v962: 178 pgs: 178 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-21T12:39:01.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:01 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1258144194' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T12:39:01.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:01 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T12:39:01.748 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:39:01.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:01 smithi114 ceph-mon[30324]: pgmap v962: 178 pgs: 178 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-21T12:39:01.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:01 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1258144194' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T12:39:01.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:01 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T12:39:01.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:01 smithi114 ceph-mon[40867]: pgmap v962: 178 pgs: 178 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.04k op/s 2024-09-21T12:39:01.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:01 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1258144194' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T12:39:01.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:01 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T12:39:01.784 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:01.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:01.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:39:01.886 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:39:01.918 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:01.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:01.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:39:01.939 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T12:39:02.019 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:39:02.059 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:02.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:02.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:39:02.069 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:39:02.069 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:39:02.069 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.069 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:260663 11.f 11:f669eb07:::benchmark_data_smithi114_187622_object122542:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-09-21T12:39:02.069 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:39:02.060098+0000", 2024-09-21T12:39:02.069 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0065239249999999999, 2024-09-21T12:39:02.069 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0065518369999999996, 2024-09-21T12:39:02.070 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:02.070 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:02.070 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:39:02.070 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:02.070 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:02.070 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:02.070 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 260663 2024-09-21T12:39:02.070 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.070 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:02.070 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.070 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:02.070 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.060098+0000", 2024-09-21T12:39:02.070 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:02.070 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.071 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.071 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:02.071 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.060098+0000", 2024-09-21T12:39:02.071 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:02.071 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.071 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.071 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:02.071 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.060099+0000", 2024-09-21T12:39:02.071 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4950000000000001e-06 2024-09-21T12:39:02.071 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.071 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.071 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:02.071 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.060103+0000", 2024-09-21T12:39:02.071 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8010000000000001e-06 2024-09-21T12:39:02.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.072 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:02.072 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.060112+0000", 2024-09-21T12:39:02.072 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1079999999999999e-06 2024-09-21T12:39:02.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.072 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:02.072 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.060119+0000", 2024-09-21T12:39:02.072 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1010000000000003e-06 2024-09-21T12:39:02.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.072 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:02.072 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.060154+0000", 2024-09-21T12:39:02.073 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5389999999999998e-05 2024-09-21T12:39:02.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.073 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.073 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:02.073 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.060232+0000", 2024-09-21T12:39:02.073 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7197e-05 2024-09-21T12:39:02.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.073 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.073 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,2", 2024-09-21T12:39:02.073 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.060351+0000", 2024-09-21T12:39:02.073 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000118987 2024-09-21T12:39:02.073 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:02.073 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:02.073 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:02.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.073 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.074 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:260665 11.f 11:f426c5cc:::benchmark_data_smithi114_187622_object122544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-09-21T12:39:02.074 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:39:02.060962+0000", 2024-09-21T12:39:02.074 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0056594389999999996, 2024-09-21T12:39:02.074 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0057539999999999996, 2024-09-21T12:39:02.074 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:02.074 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:02.074 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:39:02.074 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:02.074 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:02.074 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:02.074 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 260665 2024-09-21T12:39:02.074 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.074 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:02.074 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.074 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:02.075 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.060962+0000", 2024-09-21T12:39:02.075 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:02.075 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.075 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.075 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:02.075 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.060962+0000", 2024-09-21T12:39:02.075 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:02.075 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.075 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.075 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:02.075 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.060963+0000", 2024-09-21T12:39:02.075 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5300000000000002e-07 2024-09-21T12:39:02.075 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.075 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.075 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:02.076 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.060967+0000", 2024-09-21T12:39:02.076 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.189e-06 2024-09-21T12:39:02.076 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.076 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.076 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:02.076 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.060981+0000", 2024-09-21T12:39:02.076 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3981e-05 2024-09-21T12:39:02.076 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.076 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.076 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:02.076 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.060988+0000", 2024-09-21T12:39:02.076 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9670000000000004e-06 2024-09-21T12:39:02.076 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.076 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.076 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:02.076 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.064527+0000", 2024-09-21T12:39:02.077 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0035383900000000002 2024-09-21T12:39:02.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.077 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.077 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:02.077 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.064654+0000", 2024-09-21T12:39:02.077 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000127107 2024-09-21T12:39:02.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.077 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.077 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,2", 2024-09-21T12:39:02.077 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.064778+0000", 2024-09-21T12:39:02.077 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000124289 2024-09-21T12:39:02.077 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:02.077 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:02.077 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:02.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.077 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.078 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop_reply(client.26147.0:260648 11.f e482/476)", 2024-09-21T12:39:02.078 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:39:02.061624+0000", 2024-09-21T12:39:02.078 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0049971929999999996, 2024-09-21T12:39:02.078 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0051454220000000002, 2024-09-21T12:39:02.078 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:02.078 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:02.078 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-09-21T12:39:02.078 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:02.078 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.078 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:02.078 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.061624+0000", 2024-09-21T12:39:02.078 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:02.078 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.078 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.079 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:02.079 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.061624+0000", 2024-09-21T12:39:02.079 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:02.079 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.079 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.079 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:02.079 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.061625+0000", 2024-09-21T12:39:02.079 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.47e-07 2024-09-21T12:39:02.079 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.079 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.079 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:02.079 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.061629+0000", 2024-09-21T12:39:02.079 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8530000000000002e-06 2024-09-21T12:39:02.079 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.079 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.079 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:02.080 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.061638+0000", 2024-09-21T12:39:02.080 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4369999999999996e-06 2024-09-21T12:39:02.080 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.080 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.080 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:02.080 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.061644+0000", 2024-09-21T12:39:02.080 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7980000000000002e-06 2024-09-21T12:39:02.080 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:02.080 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:02.080 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:02.080 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.080 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.080 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop_reply(client.26147.0:260656 11.f e482/476)", 2024-09-21T12:39:02.080 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:39:02.061829+0000", 2024-09-21T12:39:02.080 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.004792017, 2024-09-21T12:39:02.080 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0049751420000000001, 2024-09-21T12:39:02.081 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:02.081 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:02.081 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-09-21T12:39:02.081 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:02.081 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.081 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:02.081 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.061829+0000", 2024-09-21T12:39:02.081 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:02.081 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.081 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.081 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:02.081 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.061829+0000", 2024-09-21T12:39:02.081 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:02.081 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.082 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.082 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:02.082 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.061830+0000", 2024-09-21T12:39:02.082 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3599999999999999e-07 2024-09-21T12:39:02.082 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.082 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.082 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:02.082 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.061833+0000", 2024-09-21T12:39:02.082 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6419999999999999e-06 2024-09-21T12:39:02.082 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.083 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.083 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:02.083 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.061845+0000", 2024-09-21T12:39:02.083 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2145e-05 2024-09-21T12:39:02.083 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.083 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.084 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:02.084 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.061850+0000", 2024-09-21T12:39:02.084 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4410000000000001e-06 2024-09-21T12:39:02.084 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:02.084 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:02.084 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:02.084 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.084 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.084 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:260668 11.5 11:a57b3826:::benchmark_data_smithi114_187622_object122547:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-09-21T12:39:02.084 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:39:02.061937+0000", 2024-09-21T12:39:02.084 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0046844520000000004, 2024-09-21T12:39:02.084 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0049143479999999998, 2024-09-21T12:39:02.084 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:02.084 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:02.084 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:39:02.084 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:02.085 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:02.085 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:02.085 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 260668 2024-09-21T12:39:02.085 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.085 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:02.085 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.085 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:02.085 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.061937+0000", 2024-09-21T12:39:02.085 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:02.085 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.085 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.085 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:02.085 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.061937+0000", 2024-09-21T12:39:02.085 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:02.085 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.085 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.086 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:02.086 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.061938+0000", 2024-09-21T12:39:02.086 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.049e-06 2024-09-21T12:39:02.086 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.086 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.086 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:02.086 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.061943+0000", 2024-09-21T12:39:02.086 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4479999999999996e-06 2024-09-21T12:39:02.086 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.086 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.086 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:02.086 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.061953+0000", 2024-09-21T12:39:02.086 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0524e-05 2024-09-21T12:39:02.086 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.086 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.087 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:02.087 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.061960+0000", 2024-09-21T12:39:02.087 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7349999999999999e-06 2024-09-21T12:39:02.087 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.087 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.087 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:02.087 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.062009+0000", 2024-09-21T12:39:02.087 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9024999999999999e-05 2024-09-21T12:39:02.087 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.087 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.087 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:02.087 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.062106+0000", 2024-09-21T12:39:02.087 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7083000000000005e-05 2024-09-21T12:39:02.087 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.087 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.088 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,6", 2024-09-21T12:39:02.088 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.062218+0000", 2024-09-21T12:39:02.088 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011205 2024-09-21T12:39:02.088 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.088 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.088 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:02.088 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.066329+0000", 2024-09-21T12:39:02.088 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0041112099999999997 2024-09-21T12:39:02.088 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.088 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.088 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:02.088 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.066343+0000", 2024-09-21T12:39:02.088 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4218e-05 2024-09-21T12:39:02.088 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:02.088 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:02.088 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:02.089 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.089 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.089 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.26147.0:260667 11.0 e482/420 11:02b0cf55:::benchmark_data_smithi114_187622_object122546:head v 482'16216, mlcod=482'16214)", 2024-09-21T12:39:02.089 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:39:02.064393+0000", 2024-09-21T12:39:02.089 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0022287850000000001, 2024-09-21T12:39:02.089 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0025239389999999998, 2024-09-21T12:39:02.089 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:02.089 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:02.089 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-21T12:39:02.089 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:02.089 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.089 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:02.089 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.064393+0000", 2024-09-21T12:39:02.089 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:02.089 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.089 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.090 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:02.090 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.064393+0000", 2024-09-21T12:39:02.090 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:02.090 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.090 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.090 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:02.090 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.064394+0000", 2024-09-21T12:39:02.090 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3900000000000003e-07 2024-09-21T12:39:02.090 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.090 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.090 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:02.090 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.064399+0000", 2024-09-21T12:39:02.090 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8130000000000001e-06 2024-09-21T12:39:02.090 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.091 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.091 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:02.091 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.064409+0000", 2024-09-21T12:39:02.091 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4979999999999995e-06 2024-09-21T12:39:02.091 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.091 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.091 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:02.091 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.064415+0000", 2024-09-21T12:39:02.091 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1319999999999999e-06 2024-09-21T12:39:02.091 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.091 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.091 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:02.091 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.064479+0000", 2024-09-21T12:39:02.091 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3785e-05 2024-09-21T12:39:02.091 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.091 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.092 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:02.092 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.064498+0000", 2024-09-21T12:39:02.092 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8893e-05 2024-09-21T12:39:02.092 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:02.092 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:02.092 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:02.092 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.092 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.092 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:260673 11.5 11:a478f7a2:::benchmark_data_smithi114_187622_object122552:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-09-21T12:39:02.092 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:39:02.066364+0000", 2024-09-21T12:39:02.092 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.00025742400000000002, 2024-09-21T12:39:02.092 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00059975600000000001, 2024-09-21T12:39:02.092 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:02.092 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:02.092 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:39:02.093 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:02.093 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:02.093 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:02.093 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 260673 2024-09-21T12:39:02.093 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.093 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:02.093 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.093 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:02.093 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.066364+0000", 2024-09-21T12:39:02.093 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:02.093 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.093 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.093 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:02.093 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.066364+0000", 2024-09-21T12:39:02.093 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:02.093 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.094 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.094 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:02.094 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.066366+0000", 2024-09-21T12:39:02.094 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6780000000000001e-06 2024-09-21T12:39:02.094 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.094 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.094 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:02.094 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.066372+0000", 2024-09-21T12:39:02.094 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9009999999999999e-06 2024-09-21T12:39:02.094 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.094 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.094 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:02.094 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.066385+0000", 2024-09-21T12:39:02.094 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3789e-05 2024-09-21T12:39:02.094 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.095 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.095 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:02.095 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.066393+0000", 2024-09-21T12:39:02.095 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1659999999999997e-06 2024-09-21T12:39:02.095 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.095 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.095 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:02.095 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.066604+0000", 2024-09-21T12:39:02.095 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00021127799999999999 2024-09-21T12:39:02.095 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.095 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.095 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:02.095 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.066714+0000", 2024-09-21T12:39:02.095 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000110338 2024-09-21T12:39:02.096 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.096 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.096 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,6", 2024-09-21T12:39:02.096 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.066944+0000", 2024-09-21T12:39:02.096 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00023009699999999999 2024-09-21T12:39:02.096 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:02.096 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:02.096 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:02.096 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.096 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.096 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:260648 11.f 11:f2838291:::benchmark_data_smithi114_187622_object122527:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-09-21T12:39:02.096 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:39:02.047304+0000", 2024-09-21T12:39:02.096 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.019317398999999999, 2024-09-21T12:39:02.096 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.019720719000000001, 2024-09-21T12:39:02.096 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:02.096 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:02.097 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:39:02.097 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:02.097 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:02.097 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:02.097 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 260648 2024-09-21T12:39:02.097 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.097 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:02.097 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.097 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:02.097 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.047304+0000", 2024-09-21T12:39:02.097 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:02.097 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.097 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.097 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:02.098 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.047304+0000", 2024-09-21T12:39:02.098 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:02.098 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.098 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.098 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:02.098 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.047305+0000", 2024-09-21T12:39:02.098 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8800000000000001e-07 2024-09-21T12:39:02.098 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.098 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.098 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:02.098 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.047309+0000", 2024-09-21T12:39:02.098 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7629999999999998e-06 2024-09-21T12:39:02.098 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.099 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.099 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:02.099 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.047317+0000", 2024-09-21T12:39:02.099 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.354e-06 2024-09-21T12:39:02.099 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.099 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.099 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:02.099 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.047328+0000", 2024-09-21T12:39:02.099 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.058e-05 2024-09-21T12:39:02.100 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.100 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.100 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:02.100 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.047393+0000", 2024-09-21T12:39:02.100 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5740000000000004e-05 2024-09-21T12:39:02.100 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.100 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.100 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:02.100 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.047521+0000", 2024-09-21T12:39:02.100 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000127906 2024-09-21T12:39:02.100 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.100 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.100 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,2", 2024-09-21T12:39:02.101 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.047638+0000", 2024-09-21T12:39:02.101 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000116266 2024-09-21T12:39:02.101 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.101 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.101 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:02.101 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.050477+0000", 2024-09-21T12:39:02.101 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0028390099999999999 2024-09-21T12:39:02.101 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.101 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.101 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:02.101 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.053158+0000", 2024-09-21T12:39:02.101 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0026817730000000001 2024-09-21T12:39:02.101 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.101 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.102 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:02.102 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.066985+0000", 2024-09-21T12:39:02.102 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.013827134 2024-09-21T12:39:02.102 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:02.102 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:02.102 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:02.102 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.102 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.102 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:260656 11.f 11:ff82e151:::benchmark_data_smithi114_187622_object122535:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-09-21T12:39:02.102 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:39:02.054652+0000", 2024-09-21T12:39:02.102 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.01196966, 2024-09-21T12:39:02.102 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.012452123000000001, 2024-09-21T12:39:02.102 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:02.103 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:02.103 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:39:02.103 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:02.103 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:02.103 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:02.103 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 260656 2024-09-21T12:39:02.103 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.103 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:02.103 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.103 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:02.103 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.054652+0000", 2024-09-21T12:39:02.103 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:02.103 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.103 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.104 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:02.104 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.054652+0000", 2024-09-21T12:39:02.104 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:02.104 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.104 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.104 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:02.104 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.054653+0000", 2024-09-21T12:39:02.104 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.159e-06 2024-09-21T12:39:02.104 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.104 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.104 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:02.104 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.054657+0000", 2024-09-21T12:39:02.104 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7910000000000002e-06 2024-09-21T12:39:02.104 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.104 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.105 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:02.105 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.054667+0000", 2024-09-21T12:39:02.105 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7769999999999997e-06 2024-09-21T12:39:02.105 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.105 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.105 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:02.105 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.054672+0000", 2024-09-21T12:39:02.105 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3310000000000003e-06 2024-09-21T12:39:02.105 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.105 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.105 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:02.105 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.054728+0000", 2024-09-21T12:39:02.105 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5937999999999999e-05 2024-09-21T12:39:02.105 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.105 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.105 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:02.106 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.054867+0000", 2024-09-21T12:39:02.106 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013878600000000001 2024-09-21T12:39:02.106 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.106 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.106 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,2", 2024-09-21T12:39:02.106 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.054997+0000", 2024-09-21T12:39:02.106 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012999600000000001 2024-09-21T12:39:02.106 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.106 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.106 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:02.106 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.059652+0000", 2024-09-21T12:39:02.106 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0046554830000000002 2024-09-21T12:39:02.106 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.106 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.107 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:02.107 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.059752+0000", 2024-09-21T12:39:02.107 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100379 2024-09-21T12:39:02.107 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:02.107 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:02.107 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:02.107 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.107 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.107 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:260661 11.4 11:25657d58:::benchmark_data_smithi114_187622_object122540:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-09-21T12:39:02.107 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:39:02.055937+0000", 2024-09-21T12:39:02.107 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.010684726, 2024-09-21T12:39:02.107 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.01123935, 2024-09-21T12:39:02.107 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:02.107 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:02.107 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-21T12:39:02.108 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:02.108 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:02.108 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:02.109 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 260661 2024-09-21T12:39:02.110 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.110 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:02.110 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.110 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:02.110 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.055937+0000", 2024-09-21T12:39:02.110 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:02.111 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.111 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.111 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:02.111 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.055937+0000", 2024-09-21T12:39:02.111 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:02.111 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.112 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.112 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:02.112 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.055937+0000", 2024-09-21T12:39:02.112 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7399999999999999e-07 2024-09-21T12:39:02.112 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.112 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.112 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:02.113 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.055943+0000", 2024-09-21T12:39:02.113 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7089999999999999e-06 2024-09-21T12:39:02.113 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.113 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.113 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:02.113 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.055948+0000", 2024-09-21T12:39:02.114 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2109999999999997e-06 2024-09-21T12:39:02.114 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.114 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.114 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:02.114 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.055952+0000", 2024-09-21T12:39:02.114 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.647e-06 2024-09-21T12:39:02.115 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.115 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.115 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:02.115 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.056011+0000", 2024-09-21T12:39:02.115 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8931000000000002e-05 2024-09-21T12:39:02.115 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.116 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.116 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:02.116 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.056246+0000", 2024-09-21T12:39:02.116 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00023544399999999999 2024-09-21T12:39:02.116 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.116 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.116 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,6", 2024-09-21T12:39:02.116 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.056349+0000", 2024-09-21T12:39:02.117 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000103535 2024-09-21T12:39:02.117 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.117 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.117 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:02.117 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.059162+0000", 2024-09-21T12:39:02.117 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0028120409999999999 2024-09-21T12:39:02.117 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:02.118 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:02.118 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:02.118 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:39:02.059642+0000", 2024-09-21T12:39:02.118 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00048076899999999998 2024-09-21T12:39:02.118 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:02.118 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:02.118 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:02.118 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:02.119 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-21T12:39:02.119 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 10 2024-09-21T12:39:02.119 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:39:02.160 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:39:02.190 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:02.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:02.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:39:02.292 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:39:02.319 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:02.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:02.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:39:02.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:39:02.456 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:02.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:02.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:39:02.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:02 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T12:39:02.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:02 smithi114 ceph-mon[30324]: osdmap e482: 8 total, 8 up, 8 in 2024-09-21T12:39:02.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:02 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T12:39:02.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:02 smithi114 ceph-mon[40867]: osdmap e482: 8 total, 8 up, 8 in 2024-09-21T12:39:02.558 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:39:02.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:02.598 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:02.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:39:02.699 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:39:02.729 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:02.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:02.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:39:02.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:02 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T12:39:02.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:02 smithi183 ceph-mon[33277]: osdmap e482: 8 total, 8 up, 8 in 2024-09-21T12:39:02.831 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:39:02.863 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:02.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:02.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:39:02.964 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:39:02.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:02.993 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:02.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:39:03.024 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:39:02 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:39:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:39:03.035 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T12:39:03.094 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:39:03.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:03.134 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:03.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:39:03.235 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:39:03.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:03.276 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:03.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:39:03.377 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:39:03.415 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:03.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:03.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:39:03.434 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:39:03.517 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:39:03.554 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:03.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:03.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:39:03.655 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:03 smithi183 ceph-mon[33277]: pgmap v964: 178 pgs: 178 active+clean; 1.1 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-09-21T12:39:03.655 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:03 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:39:03.656 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:39:03.684 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:03.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:03.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:39:03.705 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:03 smithi114 ceph-mon[40867]: pgmap v964: 178 pgs: 178 active+clean; 1.1 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-09-21T12:39:03.705 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:03 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:39:03.706 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:03 smithi114 ceph-mon[30324]: pgmap v964: 178 pgs: 178 active+clean; 1.1 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-09-21T12:39:03.706 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:03 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:39:03.786 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:39:03.812 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:03.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:03.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:39:03.914 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:39:03.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:03.948 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:03.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:39:04.025 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:39:04.026 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-21T12:39:04.026 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-21T12:39:04.026 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-21T12:39:04.026 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:39:04.049 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:39:04.084 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:04.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:04.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:39:04.186 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T12:39:04.186 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:39:04.227 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:04.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:04.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:39:04.329 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:39:04.361 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:39:04.389 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:04 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3197920939' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:39:04.389 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:04 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:39:04.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:04.399 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:04.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:39:04.408 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-21T12:39:04.500 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:39:04.544 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:04.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:04.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:39:04.645 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:39:04.676 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:04.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:04.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:39:04.703 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:04 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3197920939' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:39:04.703 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:04 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:39:04.747 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T12:39:04.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:04 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3197920939' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:39:04.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:04 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:39:04.778 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:39:04.820 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:04.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:04.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:39:04.921 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:39:04.954 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:04.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:04.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:39:05.055 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:39:05.087 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:05.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:05.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:39:05.167 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:39:05.189 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:39:05.206 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:39:05.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:05.221 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:05.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:39:05.322 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:39:05.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:05.360 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:05.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:39:05.460 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:39:05.487 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:05.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:05.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:39:05.589 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:39:05.616 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:05 smithi114 ceph-mon[30324]: pgmap v965: 178 pgs: 178 active+clean; 936 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.63k op/s 2024-09-21T12:39:05.616 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:05 smithi114 ceph-mon[30324]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:39:05.616 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:05 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:39:05.616 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:05 smithi114 ceph-mon[30324]: osdmap e483: 8 total, 8 up, 8 in 2024-09-21T12:39:05.616 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:05 smithi114 ceph-mon[40867]: pgmap v965: 178 pgs: 178 active+clean; 936 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.63k op/s 2024-09-21T12:39:05.616 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:05 smithi114 ceph-mon[40867]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:39:05.616 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:05 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:39:05.616 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:05 smithi114 ceph-mon[40867]: osdmap e483: 8 total, 8 up, 8 in 2024-09-21T12:39:05.617 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:05 smithi183 ceph-mon[33277]: pgmap v965: 178 pgs: 178 active+clean; 936 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.63k op/s 2024-09-21T12:39:05.617 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:05 smithi183 ceph-mon[33277]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:39:05.617 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:05 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T12:39:05.617 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:05 smithi183 ceph-mon[33277]: osdmap e483: 8 total, 8 up, 8 in 2024-09-21T12:39:05.634 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:05.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:05.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:39:05.736 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:39:05.764 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:05.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:05.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:39:05.866 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:39:05.902 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:05.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:05.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:39:05.907 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:39:05.907 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:39:05.948 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:39:05.948 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:39:05.966 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:39:05.966 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:39:05.983 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:39:05.983 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:39:06.004 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:39:06.027 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:39:06.027 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:39:06.042 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:06.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:06.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:39:06.061 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:39:06.061 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:39:06.100 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:39:06.100 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:39:06.143 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:39:06.143 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:39:06.144 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:39:06.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:06.181 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:06.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:39:06.282 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:39:06.320 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:06.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:06.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:39:06.422 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:39:06.456 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-21T12:39:06.456 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-21T12:39:06.456 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-21T12:39:06.456 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-21T12:39:06.456 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.456 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:221955 9.d 9:baadbb95:::benchmark_data_smithi114_133214_object97921:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:39:06.456 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.080261+0000", 2024-09-21T12:39:06.456 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 447.70441556399999, 2024-09-21T12:39:06.457 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0418771250000001, 2024-09-21T12:39:06.457 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:06.457 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:06.457 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:06.457 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:06.457 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:39:06.457 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:39:06.457 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 221955 2024-09-21T12:39:06.457 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.457 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:06.457 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.457 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:06.457 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.080261+0000", 2024-09-21T12:39:06.457 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.458 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.458 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.458 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:06.458 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.080261+0000", 2024-09-21T12:39:06.458 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.458 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.458 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.458 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:06.458 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.080262+0000", 2024-09-21T12:39:06.458 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1230000000000001e-06 2024-09-21T12:39:06.458 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.458 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.458 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:06.459 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.080265+0000", 2024-09-21T12:39:06.459 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.58e-06 2024-09-21T12:39:06.459 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.459 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.459 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:06.459 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.080277+0000", 2024-09-21T12:39:06.459 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1610000000000001e-05 2024-09-21T12:39:06.459 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.459 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.459 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:06.459 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.080282+0000", 2024-09-21T12:39:06.459 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.366e-06 2024-09-21T12:39:06.459 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.459 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.460 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.460 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.080329+0000", 2024-09-21T12:39:06.460 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7267e-05 2024-09-21T12:39:06.460 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.460 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.460 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:06.460 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.080337+0000", 2024-09-21T12:39:06.460 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0299999999999994e-06 2024-09-21T12:39:06.460 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.460 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.460 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.460 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.056546+0000", 2024-09-21T12:39:06.460 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.97620854599999995 2024-09-21T12:39:06.460 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.461 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.461 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:06.461 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.056558+0000", 2024-09-21T12:39:06.461 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1619e-05 2024-09-21T12:39:06.461 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.461 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.461 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.461 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.115730+0000", 2024-09-21T12:39:06.461 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0591722560000001 2024-09-21T12:39:06.461 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.461 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.461 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:06.461 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.115840+0000", 2024-09-21T12:39:06.461 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000110555 2024-09-21T12:39:06.462 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.462 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.462 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:39:06.462 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.115924+0000", 2024-09-21T12:39:06.462 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3115999999999995e-05 2024-09-21T12:39:06.462 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.462 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.462 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.462 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.117996+0000", 2024-09-21T12:39:06.462 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020720320000000001 2024-09-21T12:39:06.462 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.462 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.463 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:06.463 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.121834+0000", 2024-09-21T12:39:06.463 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0038379970000000001 2024-09-21T12:39:06.463 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.463 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.463 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.463 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.121889+0000", 2024-09-21T12:39:06.463 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5245999999999999e-05 2024-09-21T12:39:06.463 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.463 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.463 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.463 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.121951+0000", 2024-09-21T12:39:06.463 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1906e-05 2024-09-21T12:39:06.464 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.464 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.464 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:06.464 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122014+0000", 2024-09-21T12:39:06.464 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3441999999999993e-05 2024-09-21T12:39:06.464 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.464 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.464 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:06.464 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122138+0000", 2024-09-21T12:39:06.464 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012343400000000001 2024-09-21T12:39:06.464 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.464 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:06.464 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.464 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.464 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.465 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:221957 9.d 9:bbdc0e00:::benchmark_data_smithi114_133214_object97923:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:39:06.465 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.082731+0000", 2024-09-21T12:39:06.465 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 447.70194556500002, 2024-09-21T12:39:06.465 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0396655529999999, 2024-09-21T12:39:06.465 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:06.465 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:06.465 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:06.465 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:06.465 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:39:06.465 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:39:06.465 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 221957 2024-09-21T12:39:06.465 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.465 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:06.466 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.466 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:06.466 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.082731+0000", 2024-09-21T12:39:06.466 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.466 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.466 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.466 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:06.466 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.082731+0000", 2024-09-21T12:39:06.466 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.466 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.466 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.466 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:06.466 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.082731+0000", 2024-09-21T12:39:06.466 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0699999999999996e-07 2024-09-21T12:39:06.467 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.467 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.467 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:06.467 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.082735+0000", 2024-09-21T12:39:06.467 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2720000000000002e-06 2024-09-21T12:39:06.467 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.467 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.467 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:06.467 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.082744+0000", 2024-09-21T12:39:06.467 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0410000000000003e-06 2024-09-21T12:39:06.467 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.467 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.467 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:06.467 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.082748+0000", 2024-09-21T12:39:06.467 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6280000000000004e-06 2024-09-21T12:39:06.468 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.468 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.468 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.468 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.082805+0000", 2024-09-21T12:39:06.468 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6938000000000003e-05 2024-09-21T12:39:06.468 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.468 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.468 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:06.468 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.082824+0000", 2024-09-21T12:39:06.468 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8493999999999999e-05 2024-09-21T12:39:06.468 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.468 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.468 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.468 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.056890+0000", 2024-09-21T12:39:06.468 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.97406623800000003 2024-09-21T12:39:06.469 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.469 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.469 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:06.469 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.056898+0000", 2024-09-21T12:39:06.469 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5159999999999994e-06 2024-09-21T12:39:06.469 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.469 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.469 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.469 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.118054+0000", 2024-09-21T12:39:06.469 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.061155555 2024-09-21T12:39:06.469 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.469 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.469 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:06.469 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.118246+0000", 2024-09-21T12:39:06.470 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019225500000000001 2024-09-21T12:39:06.470 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.470 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.470 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:39:06.470 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.118379+0000", 2024-09-21T12:39:06.470 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013250299999999999 2024-09-21T12:39:06.470 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.470 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.470 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:06.470 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122152+0000", 2024-09-21T12:39:06.470 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0037734909999999999 2024-09-21T12:39:06.470 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.470 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.471 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.471 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122183+0000", 2024-09-21T12:39:06.471 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0549999999999997e-05 2024-09-21T12:39:06.471 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.471 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.471 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.471 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122237+0000", 2024-09-21T12:39:06.471 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3735e-05 2024-09-21T12:39:06.471 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.471 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.471 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.471 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122292+0000", 2024-09-21T12:39:06.471 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5507000000000002e-05 2024-09-21T12:39:06.471 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.472 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.472 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:06.472 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122328+0000", 2024-09-21T12:39:06.472 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6149000000000002e-05 2024-09-21T12:39:06.472 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.472 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.472 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:06.472 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122396+0000", 2024-09-21T12:39:06.472 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7873999999999994e-05 2024-09-21T12:39:06.472 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.472 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:06.472 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.472 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.472 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.473 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:221969 9.d 9:b7382fd3:::benchmark_data_smithi114_133214_object97935:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:39:06.473 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.094992+0000", 2024-09-21T12:39:06.473 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 447.68968417100001, 2024-09-21T12:39:06.473 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0395933770000001, 2024-09-21T12:39:06.473 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:06.473 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:06.473 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:06.473 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:06.473 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:39:06.473 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:39:06.473 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 221969 2024-09-21T12:39:06.473 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.473 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:06.473 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.474 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:06.474 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.094992+0000", 2024-09-21T12:39:06.474 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.474 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.474 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.474 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:06.474 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.094992+0000", 2024-09-21T12:39:06.474 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.474 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.474 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.474 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:06.474 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.094993+0000", 2024-09-21T12:39:06.475 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.054e-06 2024-09-21T12:39:06.475 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.475 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.475 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:06.475 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.094998+0000", 2024-09-21T12:39:06.475 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7620000000000003e-06 2024-09-21T12:39:06.475 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.475 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.475 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:06.475 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095010+0000", 2024-09-21T12:39:06.475 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2091e-05 2024-09-21T12:39:06.475 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.475 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.475 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:06.476 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095018+0000", 2024-09-21T12:39:06.476 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8029999999999997e-06 2024-09-21T12:39:06.476 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.476 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.476 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.476 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095074+0000", 2024-09-21T12:39:06.476 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6013000000000001e-05 2024-09-21T12:39:06.476 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.476 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.476 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:06.476 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095081+0000", 2024-09-21T12:39:06.476 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9999999999999999e-06 2024-09-21T12:39:06.476 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.477 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.477 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.477 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.056923+0000", 2024-09-21T12:39:06.477 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.96184202600000002 2024-09-21T12:39:06.477 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.477 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.477 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:06.477 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.056930+0000", 2024-09-21T12:39:06.477 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4329999999999998e-06 2024-09-21T12:39:06.477 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.477 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.477 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.477 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122692+0000", 2024-09-21T12:39:06.477 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.065761918 2024-09-21T12:39:06.477 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.478 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.478 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:06.478 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122771+0000", 2024-09-21T12:39:06.478 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9456000000000006e-05 2024-09-21T12:39:06.478 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.478 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.478 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:39:06.478 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.122867+0000", 2024-09-21T12:39:06.478 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5639999999999999e-05 2024-09-21T12:39:06.478 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.478 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.478 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.478 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.129541+0000", 2024-09-21T12:39:06.478 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.006674386 2024-09-21T12:39:06.479 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.479 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.479 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.479 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.129579+0000", 2024-09-21T12:39:06.479 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7162000000000001e-05 2024-09-21T12:39:06.479 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.479 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.479 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.479 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.129693+0000", 2024-09-21T12:39:06.479 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011387099999999999 2024-09-21T12:39:06.480 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.480 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.480 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:06.480 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134406+0000", 2024-09-21T12:39:06.480 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0047137619999999998 2024-09-21T12:39:06.480 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.480 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.480 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:06.480 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134466+0000", 2024-09-21T12:39:06.480 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9317999999999997e-05 2024-09-21T12:39:06.480 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.480 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.480 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:06.480 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134585+0000", 2024-09-21T12:39:06.481 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000119682 2024-09-21T12:39:06.481 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.481 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:06.481 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.481 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.481 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.481 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:221970 9.d 9:b4ae3e82:::benchmark_data_smithi114_133214_object97936:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:39:06.481 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.095717+0000", 2024-09-21T12:39:06.481 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 447.68895887399998, 2024-09-21T12:39:06.481 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0390824689999998, 2024-09-21T12:39:06.481 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:06.481 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:06.481 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:06.482 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:06.482 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:39:06.482 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:39:06.482 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 221970 2024-09-21T12:39:06.482 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.482 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:06.482 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.482 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:06.482 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095717+0000", 2024-09-21T12:39:06.482 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.482 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.482 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.482 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:06.483 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095717+0000", 2024-09-21T12:39:06.483 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.483 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.483 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.483 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:06.483 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095718+0000", 2024-09-21T12:39:06.483 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4e-07 2024-09-21T12:39:06.483 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.483 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.483 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:06.483 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095723+0000", 2024-09-21T12:39:06.483 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.51e-06 2024-09-21T12:39:06.483 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.484 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.484 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:06.484 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095732+0000", 2024-09-21T12:39:06.484 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7620000000000007e-06 2024-09-21T12:39:06.484 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.484 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.484 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:06.484 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095738+0000", 2024-09-21T12:39:06.484 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2909999999999998e-06 2024-09-21T12:39:06.484 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.484 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.484 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.484 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095779+0000", 2024-09-21T12:39:06.484 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1356999999999997e-05 2024-09-21T12:39:06.485 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.485 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.485 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:06.485 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.095785+0000", 2024-09-21T12:39:06.485 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7610000000000004e-06 2024-09-21T12:39:06.485 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.485 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.485 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.485 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.056979+0000", 2024-09-21T12:39:06.485 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.96119413399999998 2024-09-21T12:39:06.485 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.485 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.485 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:06.485 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.056986+0000", 2024-09-21T12:39:06.485 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1770000000000001e-06 2024-09-21T12:39:06.486 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.486 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.486 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.486 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.125121+0000", 2024-09-21T12:39:06.486 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0681344740000001 2024-09-21T12:39:06.486 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.486 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.486 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:06.486 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.125232+0000", 2024-09-21T12:39:06.486 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000111875 2024-09-21T12:39:06.486 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.486 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.486 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:39:06.486 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.125348+0000", 2024-09-21T12:39:06.487 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000115696 2024-09-21T12:39:06.487 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.487 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.487 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.487 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.129616+0000", 2024-09-21T12:39:06.487 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0042675930000000001 2024-09-21T12:39:06.487 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.487 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.487 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.487 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.129653+0000", 2024-09-21T12:39:06.487 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7786e-05 2024-09-21T12:39:06.487 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.487 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.487 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.488 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.129734+0000", 2024-09-21T12:39:06.488 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1036000000000001e-05 2024-09-21T12:39:06.488 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.488 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.488 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:06.488 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134602+0000", 2024-09-21T12:39:06.488 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0048670479999999997 2024-09-21T12:39:06.488 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.488 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.488 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:06.489 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134664+0000", 2024-09-21T12:39:06.489 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2360999999999995e-05 2024-09-21T12:39:06.489 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.489 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.489 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:06.489 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134800+0000", 2024-09-21T12:39:06.489 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013576800000000001 2024-09-21T12:39:06.489 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.489 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:06.489 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.489 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.489 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.489 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:221972 9.d 9:b9fcc29c:::benchmark_data_smithi114_133214_object97938:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:39:06.489 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.096425+0000", 2024-09-21T12:39:06.490 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 447.68825093200002, 2024-09-21T12:39:06.490 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0387005569999999, 2024-09-21T12:39:06.490 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:06.490 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:06.490 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:06.490 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:06.490 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:39:06.490 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:39:06.490 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 221972 2024-09-21T12:39:06.490 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.490 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:06.490 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.490 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:06.490 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.096425+0000", 2024-09-21T12:39:06.491 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.491 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.491 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.491 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:06.491 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.096425+0000", 2024-09-21T12:39:06.491 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.491 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.491 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.491 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:06.491 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.096427+0000", 2024-09-21T12:39:06.491 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0099999999999998e-06 2024-09-21T12:39:06.491 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.491 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.491 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:06.491 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.096437+0000", 2024-09-21T12:39:06.492 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6530000000000006e-06 2024-09-21T12:39:06.492 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.492 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.492 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:06.492 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.096465+0000", 2024-09-21T12:39:06.492 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8048999999999999e-05 2024-09-21T12:39:06.492 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.492 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.492 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:06.492 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.096479+0000", 2024-09-21T12:39:06.492 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4517e-05 2024-09-21T12:39:06.493 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.493 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.493 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.493 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.096622+0000", 2024-09-21T12:39:06.493 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000142173 2024-09-21T12:39:06.493 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.493 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.493 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:06.493 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.096637+0000", 2024-09-21T12:39:06.493 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5792999999999999e-05 2024-09-21T12:39:06.493 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.493 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.493 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.493 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057015+0000", 2024-09-21T12:39:06.493 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.96037787799999996 2024-09-21T12:39:06.494 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.494 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.494 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:06.494 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057023+0000", 2024-09-21T12:39:06.494 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6580000000000002e-06 2024-09-21T12:39:06.494 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.494 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.494 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.494 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.127356+0000", 2024-09-21T12:39:06.494 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.070332735 2024-09-21T12:39:06.494 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.494 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.494 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:06.494 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.127461+0000", 2024-09-21T12:39:06.495 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000105509 2024-09-21T12:39:06.495 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.495 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.495 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:39:06.495 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.127579+0000", 2024-09-21T12:39:06.495 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000118016 2024-09-21T12:39:06.495 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.495 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.495 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.495 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134387+0000", 2024-09-21T12:39:06.495 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0068083450000000004 2024-09-21T12:39:06.495 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.495 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.495 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:06.495 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134811+0000", 2024-09-21T12:39:06.496 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00042396600000000001 2024-09-21T12:39:06.496 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.496 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.496 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.496 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134871+0000", 2024-09-21T12:39:06.496 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9574000000000003e-05 2024-09-21T12:39:06.496 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.496 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.496 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.496 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134949+0000", 2024-09-21T12:39:06.496 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8454000000000005e-05 2024-09-21T12:39:06.496 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.496 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.496 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:06.497 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135009+0000", 2024-09-21T12:39:06.497 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9676999999999998e-05 2024-09-21T12:39:06.497 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.497 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.497 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:06.497 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135126+0000", 2024-09-21T12:39:06.497 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011655 2024-09-21T12:39:06.497 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.497 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:06.497 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.497 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.497 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.497 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:221997 9.d 9:b9573347:::benchmark_data_smithi114_133214_object97963:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:39:06.497 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.121318+0000", 2024-09-21T12:39:06.498 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 447.66335807500002, 2024-09-21T12:39:06.498 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.01426215, 2024-09-21T12:39:06.498 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:06.498 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:06.498 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:06.498 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:06.498 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:39:06.498 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:39:06.498 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 221997 2024-09-21T12:39:06.498 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.498 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:06.498 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.498 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:06.498 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.121318+0000", 2024-09-21T12:39:06.499 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.499 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.499 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.499 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:06.499 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.121318+0000", 2024-09-21T12:39:06.499 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.499 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.499 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.499 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:06.499 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.121319+0000", 2024-09-21T12:39:06.499 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.364e-06 2024-09-21T12:39:06.499 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.499 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.499 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:06.500 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.121324+0000", 2024-09-21T12:39:06.500 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5929999999999999e-06 2024-09-21T12:39:06.500 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.500 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.500 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:06.500 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.121334+0000", 2024-09-21T12:39:06.500 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8309999999999996e-06 2024-09-21T12:39:06.500 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.500 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.501 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:06.501 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.121341+0000", 2024-09-21T12:39:06.501 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0910000000000004e-06 2024-09-21T12:39:06.501 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.501 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.501 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.501 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.121402+0000", 2024-09-21T12:39:06.501 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0875999999999998e-05 2024-09-21T12:39:06.501 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.501 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.501 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:06.501 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.121409+0000", 2024-09-21T12:39:06.501 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.498e-06 2024-09-21T12:39:06.501 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.502 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.502 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.502 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057045+0000", 2024-09-21T12:39:06.502 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.93563575200000004 2024-09-21T12:39:06.502 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.502 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.502 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:06.502 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057053+0000", 2024-09-21T12:39:06.502 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9359999999999999e-06 2024-09-21T12:39:06.502 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.502 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.502 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.502 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.129765+0000", 2024-09-21T12:39:06.503 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0727123919999999 2024-09-21T12:39:06.503 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.503 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.503 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:06.503 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.129890+0000", 2024-09-21T12:39:06.503 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012480799999999999 2024-09-21T12:39:06.503 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.503 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.503 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:39:06.503 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.130027+0000", 2024-09-21T12:39:06.503 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000136425 2024-09-21T12:39:06.503 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.503 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.503 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:06.504 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134824+0000", 2024-09-21T12:39:06.504 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0047976219999999997 2024-09-21T12:39:06.504 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.504 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.504 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.504 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135204+0000", 2024-09-21T12:39:06.504 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000380054 2024-09-21T12:39:06.504 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.504 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.504 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.504 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135259+0000", 2024-09-21T12:39:06.504 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5071000000000002e-05 2024-09-21T12:39:06.504 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.505 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.505 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.505 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135393+0000", 2024-09-21T12:39:06.505 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000133991 2024-09-21T12:39:06.505 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.505 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.505 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:06.505 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135457+0000", 2024-09-21T12:39:06.505 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3830000000000004e-05 2024-09-21T12:39:06.505 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.505 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.505 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:06.505 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135580+0000", 2024-09-21T12:39:06.505 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012301599999999999 2024-09-21T12:39:06.506 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.506 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:06.506 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.506 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.506 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.506 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:222014 9.d 9:bb4c3379:::benchmark_data_smithi114_133214_object97980:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:39:06.506 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.130717+0000", 2024-09-21T12:39:06.506 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 447.65395937800002, 2024-09-21T12:39:06.506 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0050891000000002, 2024-09-21T12:39:06.506 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:06.506 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:06.506 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:06.506 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:06.506 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:39:06.507 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:39:06.507 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 222014 2024-09-21T12:39:06.507 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.507 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:06.507 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.507 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:06.507 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.130717+0000", 2024-09-21T12:39:06.507 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.507 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.507 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.507 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:06.507 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.130717+0000", 2024-09-21T12:39:06.507 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.507 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.508 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.508 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:06.508 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.130718+0000", 2024-09-21T12:39:06.508 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.052e-06 2024-09-21T12:39:06.508 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.508 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.508 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:06.508 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.130722+0000", 2024-09-21T12:39:06.508 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8469999999999998e-06 2024-09-21T12:39:06.508 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.508 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.508 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:06.508 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.130730+0000", 2024-09-21T12:39:06.508 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5029999999999997e-06 2024-09-21T12:39:06.509 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.509 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.509 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:06.509 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.130737+0000", 2024-09-21T12:39:06.509 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6420000000000001e-06 2024-09-21T12:39:06.509 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.509 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.509 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.509 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.130793+0000", 2024-09-21T12:39:06.509 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6498e-05 2024-09-21T12:39:06.509 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.509 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.509 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:06.509 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.130802+0000", 2024-09-21T12:39:06.509 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0080000000000008e-06 2024-09-21T12:39:06.510 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.510 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.510 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.510 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057076+0000", 2024-09-21T12:39:06.510 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.92627378500000002 2024-09-21T12:39:06.510 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.510 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.510 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:06.510 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057083+0000", 2024-09-21T12:39:06.510 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7410000000000003e-06 2024-09-21T12:39:06.510 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.510 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.510 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.511 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.131785+0000", 2024-09-21T12:39:06.511 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0747017809999999 2024-09-21T12:39:06.511 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.511 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.511 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:06.511 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.131913+0000", 2024-09-21T12:39:06.511 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012813399999999999 2024-09-21T12:39:06.511 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.511 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.511 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:39:06.511 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.132034+0000", 2024-09-21T12:39:06.511 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000121236 2024-09-21T12:39:06.511 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.511 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.511 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:06.512 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.134886+0000", 2024-09-21T12:39:06.512 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0028518910000000001 2024-09-21T12:39:06.512 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.512 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.512 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.512 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135303+0000", 2024-09-21T12:39:06.512 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000417279 2024-09-21T12:39:06.512 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.512 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.512 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.512 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135347+0000", 2024-09-21T12:39:06.512 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4399e-05 2024-09-21T12:39:06.512 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.513 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.513 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.513 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135633+0000", 2024-09-21T12:39:06.513 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00028544300000000002 2024-09-21T12:39:06.513 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.513 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.513 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:06.514 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135690+0000", 2024-09-21T12:39:06.514 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6577999999999999e-05 2024-09-21T12:39:06.514 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.514 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.514 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:06.514 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135806+0000", 2024-09-21T12:39:06.514 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000116283 2024-09-21T12:39:06.515 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.515 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:06.515 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.515 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.515 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.515 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:222036 9.d 9:bf4c3996:::benchmark_data_smithi114_133214_object98002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:39:06.515 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.149247+0000", 2024-09-21T12:39:06.516 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 447.63542917900003, 2024-09-21T12:39:06.516 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9939442509999998, 2024-09-21T12:39:06.516 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:06.516 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:06.516 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:06.516 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:06.517 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:39:06.517 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:39:06.517 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 222036 2024-09-21T12:39:06.517 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.517 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:06.517 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.518 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:06.518 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.149247+0000", 2024-09-21T12:39:06.518 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.518 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.518 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.518 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:06.518 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.149247+0000", 2024-09-21T12:39:06.519 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.519 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.519 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.519 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:06.519 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.149248+0000", 2024-09-21T12:39:06.519 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3200000000000001e-06 2024-09-21T12:39:06.520 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.520 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.520 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:06.520 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.149253+0000", 2024-09-21T12:39:06.520 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2129999999999999e-06 2024-09-21T12:39:06.520 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.521 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.521 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:06.521 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.149263+0000", 2024-09-21T12:39:06.521 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2580000000000001e-06 2024-09-21T12:39:06.521 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.521 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.522 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:06.522 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.149269+0000", 2024-09-21T12:39:06.522 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5379999999999999e-06 2024-09-21T12:39:06.522 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.522 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.522 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.523 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.149332+0000", 2024-09-21T12:39:06.523 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2496999999999995e-05 2024-09-21T12:39:06.523 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.523 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.523 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:06.523 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.149343+0000", 2024-09-21T12:39:06.523 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1011e-05 2024-09-21T12:39:06.524 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.524 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.524 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.524 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057106+0000", 2024-09-21T12:39:06.524 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.90776296199999995 2024-09-21T12:39:06.524 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.525 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.525 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:06.525 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057113+0000", 2024-09-21T12:39:06.525 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4250000000000001e-06 2024-09-21T12:39:06.525 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.525 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.526 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.526 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135854+0000", 2024-09-21T12:39:06.526 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.078740493 2024-09-21T12:39:06.526 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.526 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.526 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:06.526 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.135971+0000", 2024-09-21T12:39:06.527 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000117704 2024-09-21T12:39:06.527 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.527 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.527 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:39:06.527 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.136102+0000", 2024-09-21T12:39:06.527 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013025100000000001 2024-09-21T12:39:06.528 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.528 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.528 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:06.528 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.142810+0000", 2024-09-21T12:39:06.528 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0067082419999999997 2024-09-21T12:39:06.528 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.528 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.529 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.529 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.142861+0000", 2024-09-21T12:39:06.529 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1381000000000002e-05 2024-09-21T12:39:06.529 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.529 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.529 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.529 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.142919+0000", 2024-09-21T12:39:06.529 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7482000000000003e-05 2024-09-21T12:39:06.530 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.530 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.530 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.530 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.143004+0000", 2024-09-21T12:39:06.530 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5746000000000002e-05 2024-09-21T12:39:06.530 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.530 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.531 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:06.531 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.143072+0000", 2024-09-21T12:39:06.531 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8009000000000006e-05 2024-09-21T12:39:06.531 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.531 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.531 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:06.531 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.143191+0000", 2024-09-21T12:39:06.531 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000118719 2024-09-21T12:39:06.531 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.531 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:06.531 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.531 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.532 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.532 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:222038 9.d 9:b5ba6170:::benchmark_data_smithi114_133214_object98004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:39:06.532 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.150309+0000", 2024-09-21T12:39:06.532 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 447.634367285, 2024-09-21T12:39:06.532 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.995585494, 2024-09-21T12:39:06.532 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:06.532 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:06.532 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:06.532 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:06.532 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:39:06.532 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:39:06.532 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 222038 2024-09-21T12:39:06.533 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.533 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:06.533 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.533 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:06.533 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.150309+0000", 2024-09-21T12:39:06.533 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.533 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.533 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.533 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:06.533 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.150309+0000", 2024-09-21T12:39:06.533 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.533 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.533 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.533 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:06.533 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.150310+0000", 2024-09-21T12:39:06.534 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8800000000000001e-07 2024-09-21T12:39:06.534 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.534 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.534 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:06.534 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.150313+0000", 2024-09-21T12:39:06.534 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.58e-06 2024-09-21T12:39:06.534 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.534 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.534 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:06.534 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.150322+0000", 2024-09-21T12:39:06.534 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2600000000000005e-06 2024-09-21T12:39:06.534 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.534 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.535 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:06.535 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.150328+0000", 2024-09-21T12:39:06.535 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3790000000000003e-06 2024-09-21T12:39:06.535 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.535 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.535 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.535 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.150385+0000", 2024-09-21T12:39:06.535 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7188000000000002e-05 2024-09-21T12:39:06.535 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.535 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.535 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:06.535 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.150394+0000", 2024-09-21T12:39:06.535 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1360000000000003e-06 2024-09-21T12:39:06.535 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.536 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.536 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.536 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057143+0000", 2024-09-21T12:39:06.536 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.90674834100000001 2024-09-21T12:39:06.536 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.536 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.536 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:06.536 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057150+0000", 2024-09-21T12:39:06.536 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.481e-06 2024-09-21T12:39:06.536 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.536 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.536 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.536 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.138249+0000", 2024-09-21T12:39:06.536 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0810986 2024-09-21T12:39:06.537 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.537 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.537 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:06.537 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.138359+0000", 2024-09-21T12:39:06.537 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011010699999999999 2024-09-21T12:39:06.537 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.537 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.537 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:39:06.537 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.138473+0000", 2024-09-21T12:39:06.537 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000114271 2024-09-21T12:39:06.537 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.537 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.537 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:06.538 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.142879+0000", 2024-09-21T12:39:06.538 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0044062709999999998 2024-09-21T12:39:06.538 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.538 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.538 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.538 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.142962+0000", 2024-09-21T12:39:06.538 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2497999999999995e-05 2024-09-21T12:39:06.538 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.538 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.538 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.538 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.143245+0000", 2024-09-21T12:39:06.539 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00028279200000000003 2024-09-21T12:39:06.539 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.539 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.539 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.539 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.145676+0000", 2024-09-21T12:39:06.539 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024311929999999999 2024-09-21T12:39:06.539 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.539 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.539 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:06.539 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.145735+0000", 2024-09-21T12:39:06.539 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9159000000000001e-05 2024-09-21T12:39:06.539 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.539 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.539 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:06.539 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.145894+0000", 2024-09-21T12:39:06.540 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015935 2024-09-21T12:39:06.540 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.540 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:06.540 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.540 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.540 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.540 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25436.0:222057 9.d 9:bd682287:::benchmark_data_smithi114_133214_object98023:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T12:39:06.540 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:31:38.194600+0000", 2024-09-21T12:39:06.540 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 447.59007604999999, 2024-09-21T12:39:06.540 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9643363250000001, 2024-09-21T12:39:06.540 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:06.540 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:06.540 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:06.540 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:06.541 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25436", 2024-09-21T12:39:06.541 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/2726987823", 2024-09-21T12:39:06.541 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 222057 2024-09-21T12:39:06.541 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.541 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:06.541 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.541 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:06.541 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.194600+0000", 2024-09-21T12:39:06.541 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.541 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.541 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.541 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:06.541 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.194600+0000", 2024-09-21T12:39:06.541 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.541 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.542 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.542 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:06.542 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.194601+0000", 2024-09-21T12:39:06.542 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.15e-06 2024-09-21T12:39:06.542 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.542 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.542 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:06.542 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.194605+0000", 2024-09-21T12:39:06.542 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1500000000000001e-06 2024-09-21T12:39:06.542 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.542 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.542 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:06.542 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.194616+0000", 2024-09-21T12:39:06.542 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0759e-05 2024-09-21T12:39:06.543 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.543 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.543 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:06.543 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.194622+0000", 2024-09-21T12:39:06.543 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.614e-06 2024-09-21T12:39:06.543 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.543 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.543 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.543 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.194683+0000", 2024-09-21T12:39:06.543 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0934000000000003e-05 2024-09-21T12:39:06.543 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.543 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.543 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:06.543 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:38.194690+0000", 2024-09-21T12:39:06.544 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3189999999999997e-06 2024-09-21T12:39:06.544 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.544 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.544 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.544 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057173+0000", 2024-09-21T12:39:06.544 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.862483322 2024-09-21T12:39:06.544 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.544 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.544 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:06.544 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:39.057180+0000", 2024-09-21T12:39:06.544 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0600000000000002e-06 2024-09-21T12:39:06.544 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.544 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.544 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.544 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.140485+0000", 2024-09-21T12:39:06.545 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0833051250000001 2024-09-21T12:39:06.545 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.545 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.545 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:06.545 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.140621+0000", 2024-09-21T12:39:06.545 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013514899999999999 2024-09-21T12:39:06.545 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.545 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.545 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T12:39:06.545 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.140741+0000", 2024-09-21T12:39:06.545 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012082399999999999 2024-09-21T12:39:06.545 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.545 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.545 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.546 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.145628+0000", 2024-09-21T12:39:06.546 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0048861720000000003 2024-09-21T12:39:06.546 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.546 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.546 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.546 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.145926+0000", 2024-09-21T12:39:06.546 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00029853099999999999 2024-09-21T12:39:06.546 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.546 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.546 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:06.546 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.150100+0000", 2024-09-21T12:39:06.546 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0041734930000000003 2024-09-21T12:39:06.546 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.547 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.547 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.547 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.158801+0000", 2024-09-21T12:39:06.547 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0087013130000000004 2024-09-21T12:39:06.547 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.547 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.547 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:06.547 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.158849+0000", 2024-09-21T12:39:06.547 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7604e-05 2024-09-21T12:39:06.547 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.547 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.547 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:06.547 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:31:40.158936+0000", 2024-09-21T12:39:06.547 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7806000000000006e-05 2024-09-21T12:39:06.547 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.548 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:06.548 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.548 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.548 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.548 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:244122 11.4 11:229138af:::benchmark_data_smithi114_187622_object106001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e475)", 2024-09-21T12:39:06.548 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:38:48.302317+0000", 2024-09-21T12:39:06.548 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 17.482359438, 2024-09-21T12:39:06.548 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.082307211, 2024-09-21T12:39:06.548 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:06.548 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:06.548 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-09-21T12:39:06.548 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:06.548 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:06.548 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:06.549 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 244122 2024-09-21T12:39:06.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.549 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:06.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.549 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:06.549 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.302317+0000", 2024-09-21T12:39:06.549 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.549 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:06.549 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.302317+0000", 2024-09-21T12:39:06.549 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.550 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:06.550 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.302318+0000", 2024-09-21T12:39:06.550 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.84e-06 2024-09-21T12:39:06.550 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.550 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.550 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:06.550 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.302323+0000", 2024-09-21T12:39:06.550 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3900000000000003e-06 2024-09-21T12:39:06.550 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.550 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.550 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:06.550 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.302335+0000", 2024-09-21T12:39:06.550 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1973999999999999e-05 2024-09-21T12:39:06.551 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.551 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.551 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:06.551 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.302342+0000", 2024-09-21T12:39:06.551 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2899999999999997e-06 2024-09-21T12:39:06.551 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.551 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.551 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.551 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.302382+0000", 2024-09-21T12:39:06.551 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9459000000000003e-05 2024-09-21T12:39:06.551 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.551 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.551 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:06.551 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.302492+0000", 2024-09-21T12:39:06.552 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011003800000000001 2024-09-21T12:39:06.552 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.552 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.552 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,2,6", 2024-09-21T12:39:06.552 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.302582+0000", 2024-09-21T12:39:06.552 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9893000000000001e-05 2024-09-21T12:39:06.552 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.552 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.552 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.552 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.323515+0000", 2024-09-21T12:39:06.552 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.020933552000000001 2024-09-21T12:39:06.552 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.552 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.552 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:06.553 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.323544+0000", 2024-09-21T12:39:06.553 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8989999999999999e-05 2024-09-21T12:39:06.553 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.553 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.553 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.553 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.329450+0000", 2024-09-21T12:39:06.553 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0059058959999999999 2024-09-21T12:39:06.553 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.553 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.553 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:06.553 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.329461+0000", 2024-09-21T12:39:06.553 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1131999999999999e-05 2024-09-21T12:39:06.553 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.553 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.554 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.554 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.384544+0000", 2024-09-21T12:39:06.554 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0550825079999999 2024-09-21T12:39:06.554 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.554 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.554 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:06.554 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.384624+0000", 2024-09-21T12:39:06.554 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0248999999999997e-05 2024-09-21T12:39:06.554 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.554 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:06.554 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.554 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.554 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.554 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:244128 11.f 11:fc8d1a4d:::benchmark_data_smithi114_187622_object106007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e475)", 2024-09-21T12:39:06.555 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:38:48.324019+0000", 2024-09-21T12:39:06.555 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 17.460656894, 2024-09-21T12:39:06.555 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1288967160000001, 2024-09-21T12:39:06.555 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:06.555 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:06.555 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-09-21T12:39:06.555 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:06.555 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:06.555 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:06.555 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 244128 2024-09-21T12:39:06.555 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.555 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:06.555 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.555 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:06.556 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.324019+0000", 2024-09-21T12:39:06.556 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.556 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.556 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.556 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:06.556 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.324019+0000", 2024-09-21T12:39:06.556 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.556 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.556 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.556 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:06.556 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.324020+0000", 2024-09-21T12:39:06.556 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.19e-06 2024-09-21T12:39:06.556 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.556 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.557 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:06.557 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.324024+0000", 2024-09-21T12:39:06.557 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.002e-06 2024-09-21T12:39:06.557 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.557 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.557 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:06.557 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.324034+0000", 2024-09-21T12:39:06.557 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1349999999999998e-06 2024-09-21T12:39:06.557 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.557 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.557 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:06.557 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.324040+0000", 2024-09-21T12:39:06.557 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4930000000000003e-06 2024-09-21T12:39:06.557 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.558 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.558 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.558 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.324448+0000", 2024-09-21T12:39:06.558 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00040823600000000003 2024-09-21T12:39:06.558 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.558 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.558 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:06.558 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.324569+0000", 2024-09-21T12:39:06.558 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000120445 2024-09-21T12:39:06.558 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.558 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.558 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1,2", 2024-09-21T12:39:06.558 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.324691+0000", 2024-09-21T12:39:06.558 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012256800000000001 2024-09-21T12:39:06.559 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.559 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.559 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.559 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.331205+0000", 2024-09-21T12:39:06.559 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0065136459999999997 2024-09-21T12:39:06.559 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.559 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.559 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:06.559 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.331214+0000", 2024-09-21T12:39:06.559 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9209999999999998e-06 2024-09-21T12:39:06.559 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.559 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.559 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.559 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.433717+0000", 2024-09-21T12:39:06.560 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.102503204 2024-09-21T12:39:06.560 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.560 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.560 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for missing object", 2024-09-21T12:39:06.560 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.433837+0000", 2024-09-21T12:39:06.560 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000119484 2024-09-21T12:39:06.560 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.560 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.560 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.560 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.452832+0000", 2024-09-21T12:39:06.560 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.018995821999999999 2024-09-21T12:39:06.560 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.560 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.560 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:06.560 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.452916+0000", 2024-09-21T12:39:06.561 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3570000000000001e-05 2024-09-21T12:39:06.561 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.561 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:06.561 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.561 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.561 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.561 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:244134 11.f 11:f9508737:::benchmark_data_smithi114_187622_object106013:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e475)", 2024-09-21T12:39:06.561 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:38:48.324053+0000", 2024-09-21T12:39:06.561 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 17.460622828999998, 2024-09-21T12:39:06.561 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.129067083, 2024-09-21T12:39:06.561 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:06.561 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:06.561 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:06.562 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:06.562 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:06.562 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:06.562 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 244134 2024-09-21T12:39:06.562 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.562 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:06.562 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.562 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:06.562 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.324053+0000", 2024-09-21T12:39:06.562 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.562 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.562 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.563 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:06.563 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.324053+0000", 2024-09-21T12:39:06.563 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.563 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.563 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.563 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:06.563 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.324054+0000", 2024-09-21T12:39:06.563 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.58e-07 2024-09-21T12:39:06.563 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.563 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.563 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:06.563 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.324056+0000", 2024-09-21T12:39:06.563 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1019999999999999e-06 2024-09-21T12:39:06.564 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.564 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.564 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:06.564 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.324062+0000", 2024-09-21T12:39:06.564 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8950000000000003e-06 2024-09-21T12:39:06.564 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.564 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.564 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:06.564 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.324066+0000", 2024-09-21T12:39:06.564 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8849999999999998e-06 2024-09-21T12:39:06.564 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.564 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.564 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.565 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.331608+0000", 2024-09-21T12:39:06.565 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0075410379999999999 2024-09-21T12:39:06.565 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.565 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.565 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:06.565 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.331615+0000", 2024-09-21T12:39:06.565 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3760000000000002e-06 2024-09-21T12:39:06.565 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.565 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.565 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.565 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.434101+0000", 2024-09-21T12:39:06.565 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1024859359999999 2024-09-21T12:39:06.565 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.565 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.566 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:06.566 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.434266+0000", 2024-09-21T12:39:06.566 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016564100000000001 2024-09-21T12:39:06.566 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.566 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.566 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,2", 2024-09-21T12:39:06.566 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.434391+0000", 2024-09-21T12:39:06.566 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012449600000000001 2024-09-21T12:39:06.566 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.566 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.566 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:06.566 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.441566+0000", 2024-09-21T12:39:06.566 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0071752810000000004 2024-09-21T12:39:06.567 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.567 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.567 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.567 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.447974+0000", 2024-09-21T12:39:06.567 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0064073990000000003 2024-09-21T12:39:06.567 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.567 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.567 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.567 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.452967+0000", 2024-09-21T12:39:06.567 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0049938070000000003 2024-09-21T12:39:06.567 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.567 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.568 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:06.568 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.453024+0000", 2024-09-21T12:39:06.568 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6348000000000003e-05 2024-09-21T12:39:06.568 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.568 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.568 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:06.568 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.453120+0000", 2024-09-21T12:39:06.568 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6521000000000006e-05 2024-09-21T12:39:06.568 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.568 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:06.568 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.568 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.568 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.568 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:244136 11.f 11:fcb176f7:::benchmark_data_smithi114_187622_object106015:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-09-21T12:39:06.569 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:38:48.324073+0000", 2024-09-21T12:39:06.569 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 17.460603249999998, 2024-09-21T12:39:06.569 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1292736889999999, 2024-09-21T12:39:06.569 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:06.569 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:06.569 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:06.569 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:06.569 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:06.569 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:06.569 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 244136 2024-09-21T12:39:06.569 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.569 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:06.570 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.570 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:06.570 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.324073+0000", 2024-09-21T12:39:06.570 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.570 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.570 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.571 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:06.571 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.324073+0000", 2024-09-21T12:39:06.571 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.571 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.571 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.571 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:06.572 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.324073+0000", 2024-09-21T12:39:06.572 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6800000000000001e-07 2024-09-21T12:39:06.572 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.572 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.572 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:06.572 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.324075+0000", 2024-09-21T12:39:06.572 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1179999999999999e-06 2024-09-21T12:39:06.572 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.572 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.572 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:06.572 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.324081+0000", 2024-09-21T12:39:06.572 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2909999999999998e-06 2024-09-21T12:39:06.572 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.572 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.573 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:06.573 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.324085+0000", 2024-09-21T12:39:06.573 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5519999999999998e-06 2024-09-21T12:39:06.573 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.573 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.573 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.573 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.331637+0000", 2024-09-21T12:39:06.573 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.007551955 2024-09-21T12:39:06.573 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.573 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.573 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:06.573 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.331643+0000", 2024-09-21T12:39:06.573 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8749999999999997e-06 2024-09-21T12:39:06.573 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.574 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.574 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.574 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.436723+0000", 2024-09-21T12:39:06.574 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.105079951 2024-09-21T12:39:06.574 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.574 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.574 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:06.574 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.436845+0000", 2024-09-21T12:39:06.574 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012229400000000001 2024-09-21T12:39:06.574 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.574 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.574 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,2", 2024-09-21T12:39:06.574 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.436967+0000", 2024-09-21T12:39:06.575 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012204799999999999 2024-09-21T12:39:06.575 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.575 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.575 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:06.575 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.441573+0000", 2024-09-21T12:39:06.575 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0046058829999999999 2024-09-21T12:39:06.575 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.575 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.575 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.575 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.448035+0000", 2024-09-21T12:39:06.575 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0064614750000000004 2024-09-21T12:39:06.575 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.575 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.575 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.576 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.453181+0000", 2024-09-21T12:39:06.576 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.005146851 2024-09-21T12:39:06.576 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.576 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.576 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:06.576 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.453253+0000", 2024-09-21T12:39:06.576 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1091000000000001e-05 2024-09-21T12:39:06.576 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.576 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.576 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:06.576 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.453347+0000", 2024-09-21T12:39:06.576 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3937000000000005e-05 2024-09-21T12:39:06.576 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.576 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:06.577 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.577 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.577 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.577 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:244142 11.f 11:f635f6a1:::benchmark_data_smithi114_187622_object106021:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-09-21T12:39:06.577 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:38:48.326220+0000", 2024-09-21T12:39:06.577 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 17.458456163000001, 2024-09-21T12:39:06.577 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.127454502, 2024-09-21T12:39:06.577 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:06.577 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:06.577 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:06.577 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:06.577 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:06.577 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:06.577 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 244142 2024-09-21T12:39:06.578 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.578 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:06.578 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.578 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:06.578 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.326220+0000", 2024-09-21T12:39:06.578 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.578 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.578 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.578 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:06.578 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.326220+0000", 2024-09-21T12:39:06.578 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.578 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.578 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.578 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:06.579 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.326221+0000", 2024-09-21T12:39:06.579 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.477e-06 2024-09-21T12:39:06.579 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.579 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.579 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:06.579 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.326228+0000", 2024-09-21T12:39:06.579 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2580000000000001e-06 2024-09-21T12:39:06.579 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.579 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.579 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:06.579 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.326278+0000", 2024-09-21T12:39:06.580 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0229999999999998e-05 2024-09-21T12:39:06.580 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.580 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.580 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:06.580 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.326287+0000", 2024-09-21T12:39:06.580 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5230000000000001e-06 2024-09-21T12:39:06.580 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.580 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.580 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.580 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.331664+0000", 2024-09-21T12:39:06.580 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0053762619999999997 2024-09-21T12:39:06.581 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.581 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.581 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:06.581 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.331669+0000", 2024-09-21T12:39:06.581 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6930000000000004e-06 2024-09-21T12:39:06.581 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.581 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.581 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.581 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.438802+0000", 2024-09-21T12:39:06.581 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1071322480000001 2024-09-21T12:39:06.581 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.581 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.581 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:06.581 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.438881+0000", 2024-09-21T12:39:06.582 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9845000000000005e-05 2024-09-21T12:39:06.582 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.582 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.582 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,2", 2024-09-21T12:39:06.582 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.438956+0000", 2024-09-21T12:39:06.582 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4603000000000004e-05 2024-09-21T12:39:06.582 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.582 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.582 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:06.582 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.447837+0000", 2024-09-21T12:39:06.582 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0088805399999999993 2024-09-21T12:39:06.583 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.583 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.583 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.583 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.452424+0000", 2024-09-21T12:39:06.583 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.004587832 2024-09-21T12:39:06.583 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.583 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.583 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.583 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.453409+0000", 2024-09-21T12:39:06.583 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00098480099999999999 2024-09-21T12:39:06.583 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.583 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.583 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:06.583 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.453473+0000", 2024-09-21T12:39:06.583 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3996000000000002e-05 2024-09-21T12:39:06.584 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.584 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.584 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:06.584 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.453674+0000", 2024-09-21T12:39:06.584 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000201194 2024-09-21T12:39:06.584 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.584 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:06.584 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.584 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.584 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.584 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:244151 11.f 11:fc60e970:::benchmark_data_smithi114_187622_object106030:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-09-21T12:39:06.585 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:38:48.338110+0000", 2024-09-21T12:39:06.585 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 17.446566089000001, 2024-09-21T12:39:06.585 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.119553579, 2024-09-21T12:39:06.585 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:06.585 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:06.585 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:06.585 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:06.585 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:06.586 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:06.586 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 244151 2024-09-21T12:39:06.586 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.586 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:06.586 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.586 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:06.586 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.338110+0000", 2024-09-21T12:39:06.586 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.586 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.586 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.586 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:06.586 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.338110+0000", 2024-09-21T12:39:06.587 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.587 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.587 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.587 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:06.587 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.338112+0000", 2024-09-21T12:39:06.587 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7430000000000001e-06 2024-09-21T12:39:06.587 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.587 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.587 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:06.587 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.338116+0000", 2024-09-21T12:39:06.587 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2849999999999996e-06 2024-09-21T12:39:06.587 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.587 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.588 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:06.588 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.338127+0000", 2024-09-21T12:39:06.588 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0859000000000001e-05 2024-09-21T12:39:06.588 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.588 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.588 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:06.588 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.338134+0000", 2024-09-21T12:39:06.588 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2749999999999998e-06 2024-09-21T12:39:06.588 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.588 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.588 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.589 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.338187+0000", 2024-09-21T12:39:06.589 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3266000000000001e-05 2024-09-21T12:39:06.589 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.589 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.589 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:06.589 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.338193+0000", 2024-09-21T12:39:06.589 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0580000000000003e-06 2024-09-21T12:39:06.589 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.589 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.589 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.589 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.440281+0000", 2024-09-21T12:39:06.589 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1020872129999999 2024-09-21T12:39:06.590 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.590 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.590 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:06.590 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.440387+0000", 2024-09-21T12:39:06.590 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000106031 2024-09-21T12:39:06.590 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.590 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.590 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,2", 2024-09-21T12:39:06.590 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.440505+0000", 2024-09-21T12:39:06.590 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011824099999999999 2024-09-21T12:39:06.590 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.590 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.590 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:06.590 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.447986+0000", 2024-09-21T12:39:06.591 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0074813359999999999 2024-09-21T12:39:06.591 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.591 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.591 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.591 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.452488+0000", 2024-09-21T12:39:06.591 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0045021899999999997 2024-09-21T12:39:06.591 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.591 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.591 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.591 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.457519+0000", 2024-09-21T12:39:06.591 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0050301729999999998 2024-09-21T12:39:06.591 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.591 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.592 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:06.592 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.457570+0000", 2024-09-21T12:39:06.592 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1424000000000001e-05 2024-09-21T12:39:06.592 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.592 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.592 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:06.592 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.457664+0000", 2024-09-21T12:39:06.592 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3485000000000004e-05 2024-09-21T12:39:06.592 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.592 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:06.592 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.592 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.593 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.593 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:244171 11.f 11:fc59bcf5:::benchmark_data_smithi114_187622_object106050:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-09-21T12:39:06.593 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:38:48.356437+0000", 2024-09-21T12:39:06.593 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 17.428238753999999, 2024-09-21T12:39:06.593 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.101464166, 2024-09-21T12:39:06.593 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:06.593 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:06.593 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:06.593 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:06.593 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:06.593 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:06.593 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 244171 2024-09-21T12:39:06.594 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.594 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:06.594 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.594 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:06.594 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.356437+0000", 2024-09-21T12:39:06.594 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.594 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.594 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.594 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:06.594 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.356437+0000", 2024-09-21T12:39:06.594 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.594 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.594 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.595 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:06.595 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.356439+0000", 2024-09-21T12:39:06.595 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5719999999999999e-06 2024-09-21T12:39:06.595 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.595 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.595 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:06.595 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.356445+0000", 2024-09-21T12:39:06.595 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8980000000000001e-06 2024-09-21T12:39:06.595 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.595 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.595 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:06.595 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.356457+0000", 2024-09-21T12:39:06.595 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2330000000000001e-05 2024-09-21T12:39:06.595 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.596 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.596 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:06.596 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.356464+0000", 2024-09-21T12:39:06.596 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6220000000000003e-06 2024-09-21T12:39:06.596 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.596 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.596 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.596 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.356539+0000", 2024-09-21T12:39:06.596 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5146000000000002e-05 2024-09-21T12:39:06.596 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.596 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.596 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:06.596 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.356569+0000", 2024-09-21T12:39:06.596 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0334000000000001e-05 2024-09-21T12:39:06.597 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.597 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.597 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.597 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.441610+0000", 2024-09-21T12:39:06.597 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.085040666 2024-09-21T12:39:06.597 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.598 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.598 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:06.598 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.441680+0000", 2024-09-21T12:39:06.598 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9913999999999995e-05 2024-09-21T12:39:06.598 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.598 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.599 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,2", 2024-09-21T12:39:06.599 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.441748+0000", 2024-09-21T12:39:06.599 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8644000000000002e-05 2024-09-21T12:39:06.599 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.599 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.599 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:06.599 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.447994+0000", 2024-09-21T12:39:06.600 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0062452499999999999 2024-09-21T12:39:06.600 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.600 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.600 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.600 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.453796+0000", 2024-09-21T12:39:06.600 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0058019050000000004 2024-09-21T12:39:06.600 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.600 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.600 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.600 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.457726+0000", 2024-09-21T12:39:06.600 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0039306539999999996 2024-09-21T12:39:06.600 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.600 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.600 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:06.601 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.457785+0000", 2024-09-21T12:39:06.601 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8859e-05 2024-09-21T12:39:06.601 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.601 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.601 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:06.601 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.457901+0000", 2024-09-21T12:39:06.601 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000116372 2024-09-21T12:39:06.601 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.601 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:06.602 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.602 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.602 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.602 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:244185 11.f 11:f4c1d1ed:::benchmark_data_smithi114_187622_object106064:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-09-21T12:39:06.602 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:38:48.367044+0000", 2024-09-21T12:39:06.602 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 17.417632013999999, 2024-09-21T12:39:06.602 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0966253749999999, 2024-09-21T12:39:06.602 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:06.602 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:06.602 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:06.602 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:06.602 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:06.602 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:06.603 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 244185 2024-09-21T12:39:06.603 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.603 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:06.603 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.603 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:06.603 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.367044+0000", 2024-09-21T12:39:06.603 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.603 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.603 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.603 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:06.603 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.367044+0000", 2024-09-21T12:39:06.603 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.603 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.603 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.604 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:06.604 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.367045+0000", 2024-09-21T12:39:06.604 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.339e-06 2024-09-21T12:39:06.604 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.604 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.604 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:06.604 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.367050+0000", 2024-09-21T12:39:06.604 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3050000000000003e-06 2024-09-21T12:39:06.604 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.604 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.604 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:06.604 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.367060+0000", 2024-09-21T12:39:06.604 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0294e-05 2024-09-21T12:39:06.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.605 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.605 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:06.605 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.367066+0000", 2024-09-21T12:39:06.605 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9379999999999997e-06 2024-09-21T12:39:06.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.605 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.605 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.605 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.367125+0000", 2024-09-21T12:39:06.605 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9156000000000001e-05 2024-09-21T12:39:06.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.605 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.605 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:06.605 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.367135+0000", 2024-09-21T12:39:06.606 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5650000000000007e-06 2024-09-21T12:39:06.606 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.606 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.606 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.606 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.442924+0000", 2024-09-21T12:39:06.606 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.075789661 2024-09-21T12:39:06.606 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.606 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.606 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:06.606 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.443026+0000", 2024-09-21T12:39:06.606 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000102 2024-09-21T12:39:06.606 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.606 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.607 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,2", 2024-09-21T12:39:06.607 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.443262+0000", 2024-09-21T12:39:06.607 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000236004 2024-09-21T12:39:06.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.607 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.607 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:06.607 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.448001+0000", 2024-09-21T12:39:06.607 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0047384280000000003 2024-09-21T12:39:06.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.607 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.607 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.607 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.456851+0000", 2024-09-21T12:39:06.607 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0088497780000000009 2024-09-21T12:39:06.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.608 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.608 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.463517+0000", 2024-09-21T12:39:06.608 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0066664510000000003 2024-09-21T12:39:06.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.608 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:06.608 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.463553+0000", 2024-09-21T12:39:06.608 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6412999999999998e-05 2024-09-21T12:39:06.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.608 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:06.608 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.463669+0000", 2024-09-21T12:39:06.608 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000116043 2024-09-21T12:39:06.609 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.609 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:06.609 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.609 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.609 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.609 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:244204 11.f 11:f1b68334:::benchmark_data_smithi114_187622_object106083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-09-21T12:39:06.609 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:38:48.385381+0000", 2024-09-21T12:39:06.609 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 17.399295464000001, 2024-09-21T12:39:06.609 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0920128280000001, 2024-09-21T12:39:06.609 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:06.609 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:06.609 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:06.609 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:06.610 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:06.610 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:06.610 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 244204 2024-09-21T12:39:06.610 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.610 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:06.610 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.610 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:06.610 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.385381+0000", 2024-09-21T12:39:06.610 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.611 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.611 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.611 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:06.611 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.385381+0000", 2024-09-21T12:39:06.611 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.611 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.611 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.611 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:06.611 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.385382+0000", 2024-09-21T12:39:06.611 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.668e-06 2024-09-21T12:39:06.611 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.611 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.611 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:06.611 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.385388+0000", 2024-09-21T12:39:06.612 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0759999999999999e-06 2024-09-21T12:39:06.612 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.612 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.612 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:06.612 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.385402+0000", 2024-09-21T12:39:06.612 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3926e-05 2024-09-21T12:39:06.612 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.612 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.612 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:06.612 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.385409+0000", 2024-09-21T12:39:06.612 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1980000000000004e-06 2024-09-21T12:39:06.612 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.612 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.612 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.613 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.385465+0000", 2024-09-21T12:39:06.613 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5436999999999998e-05 2024-09-21T12:39:06.613 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.613 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.613 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:06.613 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.385472+0000", 2024-09-21T12:39:06.613 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1890000000000001e-06 2024-09-21T12:39:06.613 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.613 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.614 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.614 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.445975+0000", 2024-09-21T12:39:06.614 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.060503298 2024-09-21T12:39:06.614 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.614 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.614 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:06.614 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.446049+0000", 2024-09-21T12:39:06.614 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3831000000000005e-05 2024-09-21T12:39:06.614 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.614 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.614 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,2", 2024-09-21T12:39:06.614 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.446129+0000", 2024-09-21T12:39:06.614 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9939999999999997e-05 2024-09-21T12:39:06.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.615 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.615 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:06.615 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.449990+0000", 2024-09-21T12:39:06.615 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0038609529999999999 2024-09-21T12:39:06.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.615 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.615 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.615 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.456876+0000", 2024-09-21T12:39:06.615 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0068861269999999997 2024-09-21T12:39:06.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.615 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.615 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.615 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.477336+0000", 2024-09-21T12:39:06.616 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.020459672000000002 2024-09-21T12:39:06.616 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.616 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.616 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:06.616 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.477368+0000", 2024-09-21T12:39:06.616 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1893000000000001e-05 2024-09-21T12:39:06.616 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.616 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.616 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:06.617 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.477393+0000", 2024-09-21T12:39:06.617 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5619999999999999e-05 2024-09-21T12:39:06.617 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.617 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:06.617 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.617 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.617 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.618 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.26147.0:244211 11.f 11:ffabf915:::benchmark_data_smithi114_187622_object106090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-09-21T12:39:06.618 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-21T12:38:48.399890+0000", 2024-09-21T12:39:06.618 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 17.384785685000001, 2024-09-21T12:39:06.618 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.07960828, 2024-09-21T12:39:06.618 INFO:teuthology.orchestra.run.smithi183.stdout: "continuous": false, 2024-09-21T12:39:06.618 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-21T12:39:06.618 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:06.619 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-21T12:39:06.619 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.26147", 2024-09-21T12:39:06.619 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.114:0/138975550", 2024-09-21T12:39:06.619 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 244211 2024-09-21T12:39:06.619 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.619 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-21T12:39:06.620 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.620 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-21T12:39:06.620 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.399890+0000", 2024-09-21T12:39:06.620 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.620 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.620 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.620 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-21T12:39:06.621 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.399890+0000", 2024-09-21T12:39:06.621 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-21T12:39:06.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.621 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-21T12:39:06.621 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.399892+0000", 2024-09-21T12:39:06.621 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2899999999999999e-06 2024-09-21T12:39:06.622 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.622 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.622 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-21T12:39:06.622 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.399896+0000", 2024-09-21T12:39:06.622 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3970000000000004e-06 2024-09-21T12:39:06.622 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.623 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.623 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-21T12:39:06.623 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.399908+0000", 2024-09-21T12:39:06.623 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1566000000000001e-05 2024-09-21T12:39:06.623 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.623 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.623 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-21T12:39:06.624 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.399915+0000", 2024-09-21T12:39:06.624 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5950000000000001e-06 2024-09-21T12:39:06.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.624 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.624 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.399963+0000", 2024-09-21T12:39:06.625 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7808e-05 2024-09-21T12:39:06.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-21T12:39:06.625 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:48.399970+0000", 2024-09-21T12:39:06.625 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7390000000000002e-06 2024-09-21T12:39:06.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-21T12:39:06.625 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.448066+0000", 2024-09-21T12:39:06.626 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.048095878 2024-09-21T12:39:06.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.626 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-21T12:39:06.626 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.448141+0000", 2024-09-21T12:39:06.626 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5658e-05 2024-09-21T12:39:06.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.626 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,2", 2024-09-21T12:39:06.626 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.448214+0000", 2024-09-21T12:39:06.626 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2957e-05 2024-09-21T12:39:06.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.626 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-21T12:39:06.627 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.452438+0000", 2024-09-21T12:39:06.627 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00422391 2024-09-21T12:39:06.627 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.627 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.627 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.627 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.460562+0000", 2024-09-21T12:39:06.627 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0081239290000000002 2024-09-21T12:39:06.627 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.627 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.627 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-21T12:39:06.627 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.479362+0000", 2024-09-21T12:39:06.627 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.018799865999999998 2024-09-21T12:39:06.628 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.628 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.628 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-21T12:39:06.628 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.479412+0000", 2024-09-21T12:39:06.628 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0025999999999998e-05 2024-09-21T12:39:06.628 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-21T12:39:06.628 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-21T12:39:06.628 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-21T12:39:06.628 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-21T12:38:49.479499+0000", 2024-09-21T12:39:06.628 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6661000000000003e-05 2024-09-21T12:39:06.628 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.628 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:06.628 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.628 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-21T12:39:06.629 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-21T12:39:06.629 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-21T12:39:06.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:06.630 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:06.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:39:06.731 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:39:06.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:06 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1782361453' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:39:06.765 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:06.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:06.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:39:06.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:06 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1782361453' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:39:06.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:06 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1782361453' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:39:06.866 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:39:06.904 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:06.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:06.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:39:07.007 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:39:07.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:07.042 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:07.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:39:07.143 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:39:07.163 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T12:39:07.178 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:07.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:07.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:39:07.195 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T12:39:07.279 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:39:07.310 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:07.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:07.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:39:07.387 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T12:39:07.412 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:39:07.416 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:07 smithi114 ceph-mon[40867]: pgmap v967: 178 pgs: 178 active+clean; 936 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 891 op/s 2024-09-21T12:39:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:07 smithi114 ceph-mon[30324]: pgmap v967: 178 pgs: 178 active+clean; 936 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 891 op/s 2024-09-21T12:39:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:07 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/737672490' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:39:07.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:07.437 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:07.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:39:07.538 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:39:07.571 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:07.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:07.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:39:07.672 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:39:07.721 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:07.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:07.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:39:07.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:07 smithi183 ceph-mon[33277]: pgmap v967: 178 pgs: 178 active+clean; 936 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 891 op/s 2024-09-21T12:39:07.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:07 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/737672490' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:39:07.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:07 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/737672490' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T12:39:07.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:07 smithi183 ceph-mon[33277]: osdmap e484: 8 total, 8 up, 8 in 2024-09-21T12:39:07.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:07 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/737672490' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:39:07.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:07 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/737672490' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T12:39:07.774 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:07 smithi114 ceph-mon[40867]: osdmap e484: 8 total, 8 up, 8 in 2024-09-21T12:39:07.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:07 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/737672490' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T12:39:07.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:07 smithi114 ceph-mon[30324]: osdmap e484: 8 total, 8 up, 8 in 2024-09-21T12:39:07.823 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:39:07.861 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:07.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:07.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:39:07.963 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:39:07.992 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:07.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:07.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:39:08.093 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:39:08.130 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:08.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:08.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:39:08.231 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:39:08.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:39:07 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:39:08.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:08.268 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:08.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:39:08.369 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:39:08.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:08.404 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:08.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:39:08.505 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:39:08.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:08.537 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:08.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:39:08.638 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:39:08.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:08.675 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:08.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:39:08.777 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:39:08.814 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:08.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:08.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:39:09.036 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:39:09.063 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:09.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:09.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:39:09.166 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:39:09.199 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:09.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:09.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:39:09.300 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:39:09.325 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:09.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:09.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:39:09.427 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:39:09.455 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:09.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:09.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:39:09.557 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:39:09.579 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:09.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:09.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:39:09.681 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:39:09.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:09.711 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:09.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:39:09.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:09 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:39:09.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:09 smithi183 ceph-mon[33277]: pgmap v969: 178 pgs: 178 active+clean; 550 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.86k op/s 2024-09-21T12:39:09.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:09 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:39:09.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:09 smithi114 ceph-mon[30324]: pgmap v969: 178 pgs: 178 active+clean; 550 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.86k op/s 2024-09-21T12:39:09.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:09 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:39:09.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:09 smithi114 ceph-mon[40867]: pgmap v969: 178 pgs: 178 active+clean; 550 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.86k op/s 2024-09-21T12:39:09.812 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:39:09.851 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:09.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:09.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:39:09.953 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:39:09.986 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:09.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:09.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:39:10.087 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:39:10.125 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:10.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:10.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:39:10.226 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:39:10.259 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:10.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:10.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:39:10.361 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:39:10.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:10.391 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:10.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:39:10.492 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:39:10.517 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:10.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:10.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:39:10.619 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:39:10.641 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:10.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:10.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:39:10.743 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:39:10.770 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:10.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:10.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:39:10.872 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:39:10.910 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:10.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:10.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:39:11.012 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:39:11.037 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:11.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:11.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:39:11.139 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:39:11.171 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:11.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:11.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:39:11.272 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:39:11.306 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:11.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:11.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:39:11.408 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:39:11.439 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:11.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:11.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:39:11.541 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:39:11.572 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:11.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:11.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:39:11.674 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:39:11.706 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:11.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:11.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:39:11.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:11 smithi183 ceph-mon[33277]: pgmap v970: 178 pgs: 178 active+clean; 522 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.20k op/s 2024-09-21T12:39:11.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:11 smithi114 ceph-mon[30324]: pgmap v970: 178 pgs: 178 active+clean; 522 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.20k op/s 2024-09-21T12:39:11.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:11 smithi114 ceph-mon[40867]: pgmap v970: 178 pgs: 178 active+clean; 522 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.20k op/s 2024-09-21T12:39:11.808 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:39:11.844 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:11.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:11.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:39:11.946 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:39:11.975 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:11.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:11.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:39:11.981 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T12:39:11.981 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:39:12.077 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:39:12.119 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:12.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:12.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:39:12.221 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:39:12.264 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:12.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:12.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:39:12.365 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:39:12.392 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:12.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:12.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:39:12.494 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:39:12.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:12.546 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:12.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:39:12.604 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:39:12.647 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:39:12.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:12.675 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:12.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:39:12.681 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:39:12.688 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:39:12.767 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:39:12 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:39:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:39:12.777 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:39:12.824 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:12.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:12.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:39:12.926 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:39:12.962 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:12.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:12.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:39:13.064 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:39:13.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:13.099 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:13.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:39:13.200 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:39:13.242 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:13.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:13.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:39:13.296 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:13 smithi114 ceph-mon[40867]: pgmap v971: 178 pgs: 178 active+clean; 281 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-09-21T12:39:13.296 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:13 smithi114 ceph-mon[30324]: pgmap v971: 178 pgs: 178 active+clean; 281 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-09-21T12:39:13.344 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:39:13.387 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:13.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:13.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:39:13.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:13 smithi183 ceph-mon[33277]: pgmap v971: 178 pgs: 178 active+clean; 281 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-09-21T12:39:13.488 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:39:13.517 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:13.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:13.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:39:13.618 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:39:13.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:13.659 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:13.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:39:13.708 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Removed 138118 objects 2024-09-21T12:39:13.708 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Clean up completed and total clean up time :110.187 2024-09-21T12:39:13.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:39:13.782 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:13.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:13.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:39:13.853 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:39:13.856 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:39:13.883 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:39:13.905 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:13.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:13.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:39:13.960 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:39:13.960 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:39:14.007 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:39:14.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:14.030 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:14.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:39:14.131 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:39:14.152 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:14.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:14.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:39:14.254 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:39:14.279 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:14.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:14.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:39:14.380 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:39:14.404 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:14.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:14.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:39:14.506 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:39:14.528 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:14.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:14.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:39:14.554 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T12:39:14.629 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:39:14.650 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:14.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:14.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:39:14.738 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-09-21T12:39:14.738 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-09-21T12:39:14.752 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:39:14.778 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:14.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:14.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:39:14.880 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:39:14.901 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:14.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:14.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:39:15.002 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:39:15.028 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:15.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:15.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:39:15.030 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T12:39:15.130 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:39:15.151 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:15.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:15.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:39:15.252 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:39:15.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:15.274 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:15.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:39:15.376 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:39:15.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:15.403 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:15.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:39:15.504 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:39:15.530 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:15.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:15.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:39:15.582 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":971,"stamp":"2024-09-21T12:39:12.307720+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":294787635,"num_objects":4689,"num_object_clones":0,"num_object_copies":14067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4689,"num_whiteouts":0,"num_read":1318,"num_read_kb":1070,"num_write":272470,"num_write_kb":8839598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22917,"num_bytes_recovered":1468803759,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71688,"ondisk_log_size":71688,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":535,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6937424,"kb_used_data":1021896,"kb_used_omap":18,"kb_used_meta":5915053,"kb_avail":743023792,"statfs":{"total":767960285184,"available":760856363008,"internally_reserved":0,"allocated":1046421504,"data_stored":1040014529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19199,"internal_metadata":6057014529},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-687079424,"num_objects":-10484,"num_object_clones":0,"num_object_copies":-31452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10484,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10484,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.007639"},"pg_stats":[{"pgid":"1.1d","version":"76'79","reported_seq":600,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.169498+0000","last_change":"2024-09-21T12:37:16.858542+0000","last_active":"2024-09-21T12:39:07.169498+0000","last_peered":"2024-09-21T12:39:07.169498+0000","last_clean":"2024-09-21T12:39:07.169498+0000","last_became_active":"2024-09-21T12:32:38.426383+0000","last_became_peered":"2024-09-21T12:32:38.426383+0000","last_unstale":"2024-09-21T12:39:07.169498+0000","last_undegraded":"2024-09-21T12:39:07.169498+0000","last_fullsized":"2024-09-21T12:39:07.169498+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:37:16.858472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:37:16.858472+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":914,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.182161+0000","last_change":"2024-09-21T12:38:07.168168+0000","last_active":"2024-09-21T12:39:07.182161+0000","last_peered":"2024-09-21T12:39:07.182161+0000","last_clean":"2024-09-21T12:39:07.182161+0000","last_became_active":"2024-09-21T12:38:07.167857+0000","last_became_peered":"2024-09-21T12:38:07.167857+0000","last_unstale":"2024-09-21T12:39:07.182161+0000","last_undegraded":"2024-09-21T12:39:07.182161+0000","last_fullsized":"2024-09-21T12:39:07.182161+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:29.086830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:29.086830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:02.550603+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":300,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.165522+0000","last_change":"2024-09-21T12:37:21.750122+0000","last_active":"2024-09-21T12:39:07.165522+0000","last_peered":"2024-09-21T12:39:07.165522+0000","last_clean":"2024-09-21T12:39:07.165522+0000","last_became_active":"2024-09-21T12:32:38.419081+0000","last_became_peered":"2024-09-21T12:32:38.419081+0000","last_unstale":"2024-09-21T12:39:07.165522+0000","last_undegraded":"2024-09-21T12:39:07.165522+0000","last_fullsized":"2024-09-21T12:39:07.165522+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:37:21.750033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:37:21.750033+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":912,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.167652+0000","last_change":"2024-09-21T12:37:21.767752+0000","last_active":"2024-09-21T12:39:07.167652+0000","last_peered":"2024-09-21T12:39:07.167652+0000","last_clean":"2024-09-21T12:39:07.167652+0000","last_became_active":"2024-09-21T12:36:00.136673+0000","last_became_peered":"2024-09-21T12:36:00.136673+0000","last_unstale":"2024-09-21T12:39:07.167652+0000","last_undegraded":"2024-09-21T12:39:07.167652+0000","last_fullsized":"2024-09-21T12:39:07.167652+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:21.767691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:37:21.767691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":298,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.166239+0000","last_change":"2024-09-21T12:36:44.729701+0000","last_active":"2024-09-21T12:39:07.166239+0000","last_peered":"2024-09-21T12:39:07.166239+0000","last_clean":"2024-09-21T12:39:07.166239+0000","last_became_active":"2024-09-21T12:32:38.425588+0000","last_became_peered":"2024-09-21T12:32:38.425588+0000","last_unstale":"2024-09-21T12:39:07.166239+0000","last_undegraded":"2024-09-21T12:39:07.166239+0000","last_fullsized":"2024-09-21T12:39:07.166239+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:44.729655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:44.729655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"76'79","reported_seq":159,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.171558+0000","last_change":"2024-09-21T12:37:19.644089+0000","last_active":"2024-09-21T12:39:07.171558+0000","last_peered":"2024-09-21T12:39:07.171558+0000","last_clean":"2024-09-21T12:39:07.171558+0000","last_became_active":"2024-09-21T12:36:00.154209+0000","last_became_peered":"2024-09-21T12:36:00.154209+0000","last_unstale":"2024-09-21T12:39:07.171558+0000","last_undegraded":"2024-09-21T12:39:07.171558+0000","last_fullsized":"2024-09-21T12:39:07.171558+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:37:19.644030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:37:19.644030+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":900,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.174610+0000","last_change":"2024-09-21T12:38:04.831151+0000","last_active":"2024-09-21T12:39:07.174610+0000","last_peered":"2024-09-21T12:39:07.174610+0000","last_clean":"2024-09-21T12:39:07.174610+0000","last_became_active":"2024-09-21T12:36:00.139167+0000","last_became_peered":"2024-09-21T12:36:00.139167+0000","last_unstale":"2024-09-21T12:39:07.174610+0000","last_undegraded":"2024-09-21T12:39:07.174610+0000","last_fullsized":"2024-09-21T12:39:07.174610+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:04.830956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:38:04.830956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:31.539751+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":911,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.173438+0000","last_change":"2024-09-21T12:38:07.159232+0000","last_active":"2024-09-21T12:39:07.173438+0000","last_peered":"2024-09-21T12:39:07.173438+0000","last_clean":"2024-09-21T12:39:07.173438+0000","last_became_active":"2024-09-21T12:38:07.157964+0000","last_became_peered":"2024-09-21T12:38:07.157964+0000","last_unstale":"2024-09-21T12:39:07.173438+0000","last_undegraded":"2024-09-21T12:39:07.173438+0000","last_fullsized":"2024-09-21T12:39:07.173438+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:06.698999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:06.698999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:59.290352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":956,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.174569+0000","last_change":"2024-09-21T12:38:41.050094+0000","last_active":"2024-09-21T12:39:07.174569+0000","last_peered":"2024-09-21T12:39:07.174569+0000","last_clean":"2024-09-21T12:39:07.174569+0000","last_became_active":"2024-09-21T12:38:07.168250+0000","last_became_peered":"2024-09-21T12:38:07.168250+0000","last_unstale":"2024-09-21T12:39:07.174569+0000","last_undegraded":"2024-09-21T12:39:07.174569+0000","last_fullsized":"2024-09-21T12:39:07.174569+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:38:41.050017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:41.050017+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:53.559093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":290,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.165493+0000","last_change":"2024-09-21T12:36:35.622295+0000","last_active":"2024-09-21T12:39:07.165493+0000","last_peered":"2024-09-21T12:39:07.165493+0000","last_clean":"2024-09-21T12:39:07.165493+0000","last_became_active":"2024-09-21T12:32:38.417724+0000","last_became_peered":"2024-09-21T12:32:38.417724+0000","last_unstale":"2024-09-21T12:39:07.165493+0000","last_undegraded":"2024-09-21T12:39:07.165493+0000","last_fullsized":"2024-09-21T12:39:07.165493+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:35.622232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:35.622232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":450,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.180778+0000","last_change":"2024-09-21T12:37:25.297240+0000","last_active":"2024-09-21T12:39:07.180778+0000","last_peered":"2024-09-21T12:39:07.180778+0000","last_clean":"2024-09-21T12:39:07.180778+0000","last_became_active":"2024-09-21T12:37:20.467925+0000","last_became_peered":"2024-09-21T12:37:20.467925+0000","last_unstale":"2024-09-21T12:39:07.180778+0000","last_undegraded":"2024-09-21T12:39:07.180778+0000","last_fullsized":"2024-09-21T12:39:07.180778+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:25.297200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:37:25.297200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":316,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.166330+0000","last_change":"2024-09-21T12:38:03.431702+0000","last_active":"2024-09-21T12:39:07.166330+0000","last_peered":"2024-09-21T12:39:07.166330+0000","last_clean":"2024-09-21T12:39:07.166330+0000","last_became_active":"2024-09-21T12:32:38.426121+0000","last_became_peered":"2024-09-21T12:32:38.426121+0000","last_unstale":"2024-09-21T12:39:07.166330+0000","last_undegraded":"2024-09-21T12:39:07.166330+0000","last_fullsized":"2024-09-21T12:39:07.166330+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:38:03.431620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:03.431620+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:18.419545+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":951,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.180447+0000","last_change":"2024-09-21T12:38:44.245362+0000","last_active":"2024-09-21T12:39:07.180447+0000","last_peered":"2024-09-21T12:39:07.180447+0000","last_clean":"2024-09-21T12:39:07.180447+0000","last_became_active":"2024-09-21T12:36:00.137364+0000","last_became_peered":"2024-09-21T12:36:00.137364+0000","last_unstale":"2024-09-21T12:39:07.180447+0000","last_undegraded":"2024-09-21T12:39:07.180447+0000","last_fullsized":"2024-09-21T12:39:07.180447+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:44.245309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:44.245309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:02.710227+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":910,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181390+0000","last_change":"2024-09-21T12:36:38.752470+0000","last_active":"2024-09-21T12:39:07.181390+0000","last_peered":"2024-09-21T12:39:07.181390+0000","last_clean":"2024-09-21T12:39:07.181390+0000","last_became_active":"2024-09-21T12:36:00.134723+0000","last_became_peered":"2024-09-21T12:36:00.134723+0000","last_unstale":"2024-09-21T12:39:07.181390+0000","last_undegraded":"2024-09-21T12:39:07.181390+0000","last_fullsized":"2024-09-21T12:39:07.181390+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:38.752410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:38.752410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"76'79","reported_seq":685,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.173241+0000","last_change":"2024-09-21T12:37:56.010046+0000","last_active":"2024-09-21T12:39:07.173241+0000","last_peered":"2024-09-21T12:39:07.173241+0000","last_clean":"2024-09-21T12:39:07.173241+0000","last_became_active":"2024-09-21T12:36:00.147795+0000","last_became_peered":"2024-09-21T12:36:00.147795+0000","last_unstale":"2024-09-21T12:39:07.173241+0000","last_undegraded":"2024-09-21T12:39:07.173241+0000","last_fullsized":"2024-09-21T12:39:07.173241+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:37:56.009970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:37:56.009970+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:07.284213+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":442,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.180581+0000","last_change":"2024-09-21T12:38:52.309280+0000","last_active":"2024-09-21T12:39:07.180581+0000","last_peered":"2024-09-21T12:39:07.180581+0000","last_clean":"2024-09-21T12:39:07.180581+0000","last_became_active":"2024-09-21T12:37:20.468468+0000","last_became_peered":"2024-09-21T12:37:20.468468+0000","last_unstale":"2024-09-21T12:39:07.180581+0000","last_undegraded":"2024-09-21T12:39:07.180581+0000","last_fullsized":"2024-09-21T12:39:07.180581+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:52.309224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:52.309224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:56.584480+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":304,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.166295+0000","last_change":"2024-09-21T12:37:55.044998+0000","last_active":"2024-09-21T12:39:07.166295+0000","last_peered":"2024-09-21T12:39:07.166295+0000","last_clean":"2024-09-21T12:39:07.166295+0000","last_became_active":"2024-09-21T12:32:38.422912+0000","last_became_peered":"2024-09-21T12:32:38.422912+0000","last_unstale":"2024-09-21T12:39:07.166295+0000","last_undegraded":"2024-09-21T12:39:07.166295+0000","last_fullsized":"2024-09-21T12:39:07.166295+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'7","last_scrub_stamp":"2024-09-21T12:37:55.044927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:37:55.044927+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":159,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.170983+0000","last_change":"2024-09-21T12:37:57.609800+0000","last_active":"2024-09-21T12:39:07.170983+0000","last_peered":"2024-09-21T12:39:07.170983+0000","last_clean":"2024-09-21T12:39:07.170983+0000","last_became_active":"2024-09-21T12:36:00.142106+0000","last_became_peered":"2024-09-21T12:36:00.142106+0000","last_unstale":"2024-09-21T12:39:07.170983+0000","last_undegraded":"2024-09-21T12:39:07.170983+0000","last_fullsized":"2024-09-21T12:39:07.170983+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:57.609734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:37:57.609734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":903,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.180057+0000","last_change":"2024-09-21T12:33:36.125346+0000","last_active":"2024-09-21T12:39:07.180057+0000","last_peered":"2024-09-21T12:39:07.180057+0000","last_clean":"2024-09-21T12:39:07.180057+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:39:07.180057+0000","last_undegraded":"2024-09-21T12:39:07.180057+0000","last_fullsized":"2024-09-21T12:39:07.180057+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:36.124095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:36.124095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:34.160912+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"76'79","reported_seq":725,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.175182+0000","last_change":"2024-09-21T12:34:48.959991+0000","last_active":"2024-09-21T12:39:07.175182+0000","last_peered":"2024-09-21T12:39:07.175182+0000","last_clean":"2024-09-21T12:39:07.175182+0000","last_became_active":"2024-09-21T12:27:16.957569+0000","last_became_peered":"2024-09-21T12:27:16.957569+0000","last_unstale":"2024-09-21T12:39:07.175182+0000","last_undegraded":"2024-09-21T12:39:07.175182+0000","last_fullsized":"2024-09-21T12:39:07.175182+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:48.959916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:48.959916+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":291,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.167291+0000","last_change":"2024-09-21T12:34:05.193021+0000","last_active":"2024-09-21T12:39:07.167291+0000","last_peered":"2024-09-21T12:39:07.167291+0000","last_clean":"2024-09-21T12:39:07.167291+0000","last_became_active":"2024-09-21T12:32:38.428997+0000","last_became_peered":"2024-09-21T12:32:38.428997+0000","last_unstale":"2024-09-21T12:39:07.167291+0000","last_undegraded":"2024-09-21T12:39:07.167291+0000","last_fullsized":"2024-09-21T12:39:07.167291+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:05.192971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:34:05.192971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:03.415980+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":973,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.182335+0000","last_change":"2024-09-21T12:37:23.610348+0000","last_active":"2024-09-21T12:39:07.182335+0000","last_peered":"2024-09-21T12:39:07.182335+0000","last_clean":"2024-09-21T12:39:07.182335+0000","last_became_active":"2024-09-21T12:36:00.127153+0000","last_became_peered":"2024-09-21T12:36:00.127153+0000","last_unstale":"2024-09-21T12:39:07.182335+0000","last_undegraded":"2024-09-21T12:39:07.182335+0000","last_fullsized":"2024-09-21T12:39:07.182335+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:23.610263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:37:23.610263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":953,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.180298+0000","last_change":"2024-09-21T12:38:07.162167+0000","last_active":"2024-09-21T12:39:07.180298+0000","last_peered":"2024-09-21T12:39:07.180298+0000","last_clean":"2024-09-21T12:39:07.180298+0000","last_became_active":"2024-09-21T12:38:07.161813+0000","last_became_peered":"2024-09-21T12:38:07.161813+0000","last_unstale":"2024-09-21T12:39:07.180298+0000","last_undegraded":"2024-09-21T12:39:07.180298+0000","last_fullsized":"2024-09-21T12:39:07.180298+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:33:01.668584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:01.668584+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:19.184023+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":934,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.169707+0000","last_change":"2024-09-21T12:38:53.956579+0000","last_active":"2024-09-21T12:39:07.169707+0000","last_peered":"2024-09-21T12:39:07.169707+0000","last_clean":"2024-09-21T12:39:07.169707+0000","last_became_active":"2024-09-21T12:32:38.425958+0000","last_became_peered":"2024-09-21T12:32:38.425958+0000","last_unstale":"2024-09-21T12:39:07.169707+0000","last_undegraded":"2024-09-21T12:39:07.169707+0000","last_fullsized":"2024-09-21T12:39:07.169707+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:53.956515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:53.956515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:08.408871+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"76'79","reported_seq":860,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.180249+0000","last_change":"2024-09-21T12:38:08.157982+0000","last_active":"2024-09-21T12:39:07.180249+0000","last_peered":"2024-09-21T12:39:07.180249+0000","last_clean":"2024-09-21T12:39:07.180249+0000","last_became_active":"2024-09-21T12:38:08.157632+0000","last_became_peered":"2024-09-21T12:38:08.157632+0000","last_unstale":"2024-09-21T12:39:07.180249+0000","last_undegraded":"2024-09-21T12:39:07.180249+0000","last_fullsized":"2024-09-21T12:39:07.180249+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:27.294674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:27.294674+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:21.171485+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":909,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.173706+0000","last_change":"2024-09-21T12:35:06.944278+0000","last_active":"2024-09-21T12:39:07.173706+0000","last_peered":"2024-09-21T12:39:07.173706+0000","last_clean":"2024-09-21T12:39:07.173706+0000","last_became_active":"2024-09-21T12:32:02.033276+0000","last_became_peered":"2024-09-21T12:32:02.033276+0000","last_unstale":"2024-09-21T12:39:07.173706+0000","last_undegraded":"2024-09-21T12:39:07.173706+0000","last_fullsized":"2024-09-21T12:39:07.173706+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:06.944209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:06.944209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":301,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.165903+0000","last_change":"2024-09-21T12:38:41.208373+0000","last_active":"2024-09-21T12:39:07.165903+0000","last_peered":"2024-09-21T12:39:07.165903+0000","last_clean":"2024-09-21T12:39:07.165903+0000","last_became_active":"2024-09-21T12:38:07.159387+0000","last_became_peered":"2024-09-21T12:38:07.159387+0000","last_unstale":"2024-09-21T12:39:07.165903+0000","last_undegraded":"2024-09-21T12:39:07.165903+0000","last_fullsized":"2024-09-21T12:39:07.165903+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:41.208301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:41.208301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:51.906773+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":1002,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.175556+0000","last_change":"2024-09-21T12:35:59.641966+0000","last_active":"2024-09-21T12:39:07.175556+0000","last_peered":"2024-09-21T12:39:07.175556+0000","last_clean":"2024-09-21T12:39:07.175556+0000","last_became_active":"2024-09-21T12:32:38.409163+0000","last_became_peered":"2024-09-21T12:32:38.409163+0000","last_unstale":"2024-09-21T12:39:07.175556+0000","last_undegraded":"2024-09-21T12:39:07.175556+0000","last_fullsized":"2024-09-21T12:39:07.175556+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T12:35:59.641895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:59.641895+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":309,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.166486+0000","last_change":"2024-09-21T12:38:49.267225+0000","last_active":"2024-09-21T12:39:07.166486+0000","last_peered":"2024-09-21T12:39:07.166486+0000","last_clean":"2024-09-21T12:39:07.166486+0000","last_became_active":"2024-09-21T12:38:07.161634+0000","last_became_peered":"2024-09-21T12:38:07.161634+0000","last_unstale":"2024-09-21T12:39:07.166486+0000","last_undegraded":"2024-09-21T12:39:07.166486+0000","last_fullsized":"2024-09-21T12:39:07.166486+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:49.267162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:49.267162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:09.895158+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"76'79","reported_seq":649,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.180241+0000","last_change":"2024-09-21T12:36:41.538607+0000","last_active":"2024-09-21T12:39:07.180241+0000","last_peered":"2024-09-21T12:39:07.180241+0000","last_clean":"2024-09-21T12:39:07.180241+0000","last_became_active":"2024-09-21T12:32:38.414204+0000","last_became_peered":"2024-09-21T12:32:38.414204+0000","last_unstale":"2024-09-21T12:39:07.180241+0000","last_undegraded":"2024-09-21T12:39:07.180241+0000","last_fullsized":"2024-09-21T12:39:07.180241+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:41.538559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:41.538559+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:04.209095+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":883,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.171444+0000","last_change":"2024-09-21T12:38:08.166042+0000","last_active":"2024-09-21T12:39:07.171444+0000","last_peered":"2024-09-21T12:39:07.171444+0000","last_clean":"2024-09-21T12:39:07.171444+0000","last_became_active":"2024-09-21T12:38:08.165695+0000","last_became_peered":"2024-09-21T12:38:08.165695+0000","last_unstale":"2024-09-21T12:39:07.171444+0000","last_undegraded":"2024-09-21T12:39:07.171444+0000","last_fullsized":"2024-09-21T12:39:07.171444+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:36:37.839024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:37.839024+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":952,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.167826+0000","last_change":"2024-09-21T12:36:44.561097+0000","last_active":"2024-09-21T12:39:07.167826+0000","last_peered":"2024-09-21T12:39:07.167826+0000","last_clean":"2024-09-21T12:39:07.167826+0000","last_became_active":"2024-09-21T12:32:38.425335+0000","last_became_peered":"2024-09-21T12:32:38.425335+0000","last_unstale":"2024-09-21T12:39:07.167826+0000","last_undegraded":"2024-09-21T12:39:07.167826+0000","last_fullsized":"2024-09-21T12:39:07.167826+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:36:44.561030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:36:44.561030+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:57.870150+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":1029,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181432+0000","last_change":"2024-09-21T12:38:54.562392+0000","last_active":"2024-09-21T12:39:07.181432+0000","last_peered":"2024-09-21T12:39:07.181432+0000","last_clean":"2024-09-21T12:39:07.181432+0000","last_became_active":"2024-09-21T12:36:00.148761+0000","last_became_peered":"2024-09-21T12:36:00.148761+0000","last_unstale":"2024-09-21T12:39:07.181432+0000","last_undegraded":"2024-09-21T12:39:07.181432+0000","last_fullsized":"2024-09-21T12:39:07.181432+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:38:54.562317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:54.562317+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:07.141362+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"415'18","reported_seq":396,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.166959+0000","last_change":"2024-09-21T12:37:56.019259+0000","last_active":"2024-09-21T12:39:07.166959+0000","last_peered":"2024-09-21T12:39:07.166959+0000","last_clean":"2024-09-21T12:39:07.166959+0000","last_became_active":"2024-09-21T12:36:00.147676+0000","last_became_peered":"2024-09-21T12:36:00.147676+0000","last_unstale":"2024-09-21T12:39:07.166959+0000","last_undegraded":"2024-09-21T12:39:07.166959+0000","last_fullsized":"2024-09-21T12:39:07.166959+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'18","last_scrub_stamp":"2024-09-21T12:37:56.019080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:37:56.019080+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:18.873114+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"484'17022","reported_seq":19455,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:11.868437+0000","last_change":"2024-09-21T12:37:14.558326+0000","last_active":"2024-09-21T12:39:11.868437+0000","last_peered":"2024-09-21T12:39:11.868437+0000","last_clean":"2024-09-21T12:39:11.868437+0000","last_became_active":"2024-09-21T12:36:13.105161+0000","last_became_peered":"2024-09-21T12:36:13.105161+0000","last_unstale":"2024-09-21T12:39:11.868437+0000","last_undegraded":"2024-09-21T12:39:11.868437+0000","last_fullsized":"2024-09-21T12:39:11.868437+0000","mapping_epoch":420,"log_start":"457'12800","ondisk_log_start":"457'12800","created":405,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"438'7954","last_scrub_stamp":"2024-09-21T12:37:14.558192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:37:14.558192+0000","objects_scrubbed":7862,"log_size":4222,"log_dups_size":0,"ondisk_log_size":4222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:11.142794+0000","scrub_duration":2678,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10616832,"num_objects":162,"num_object_clones":0,"num_object_copies":486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17022,"num_write_kb":549888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1078,"num_bytes_recovered":70647808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"76'79","reported_seq":461,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.180670+0000","last_change":"2024-09-21T12:38:07.165980+0000","last_active":"2024-09-21T12:39:07.180670+0000","last_peered":"2024-09-21T12:39:07.180670+0000","last_clean":"2024-09-21T12:39:07.180670+0000","last_became_active":"2024-09-21T12:38:07.165640+0000","last_became_peered":"2024-09-21T12:38:07.165640+0000","last_unstale":"2024-09-21T12:39:07.180670+0000","last_undegraded":"2024-09-21T12:39:07.180670+0000","last_fullsized":"2024-09-21T12:39:07.180670+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:18.151644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:18.151644+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:26.197946+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":571,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.173840+0000","last_change":"2024-09-21T12:38:40.030449+0000","last_active":"2024-09-21T12:39:07.173840+0000","last_peered":"2024-09-21T12:39:07.173840+0000","last_clean":"2024-09-21T12:39:07.173840+0000","last_became_active":"2024-09-21T12:38:07.166374+0000","last_became_peered":"2024-09-21T12:38:07.166374+0000","last_unstale":"2024-09-21T12:39:07.173840+0000","last_undegraded":"2024-09-21T12:39:07.173840+0000","last_fullsized":"2024-09-21T12:39:07.173840+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:38:40.030357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:40.030357+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:09.020801+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":373,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.182606+0000","last_change":"2024-09-21T12:36:00.140935+0000","last_active":"2024-09-21T12:39:07.182606+0000","last_peered":"2024-09-21T12:39:07.182606+0000","last_clean":"2024-09-21T12:39:07.182606+0000","last_became_active":"2024-09-21T12:36:00.140455+0000","last_became_peered":"2024-09-21T12:36:00.140455+0000","last_unstale":"2024-09-21T12:39:07.182606+0000","last_undegraded":"2024-09-21T12:39:07.182606+0000","last_fullsized":"2024-09-21T12:39:07.182606+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:05.703464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:05.703464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:57.233717+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"352'15","reported_seq":381,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.167263+0000","last_change":"2024-09-21T12:38:02.464559+0000","last_active":"2024-09-21T12:39:07.167263+0000","last_peered":"2024-09-21T12:39:07.167263+0000","last_clean":"2024-09-21T12:39:07.167263+0000","last_became_active":"2024-09-21T12:32:38.429092+0000","last_became_peered":"2024-09-21T12:32:38.429092+0000","last_unstale":"2024-09-21T12:39:07.167263+0000","last_undegraded":"2024-09-21T12:39:07.167263+0000","last_fullsized":"2024-09-21T12:39:07.167263+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'15","last_scrub_stamp":"2024-09-21T12:38:02.464486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:02.464486+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:27.762390+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":410,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.180934+0000","last_change":"2024-09-21T12:37:55.128975+0000","last_active":"2024-09-21T12:39:07.180934+0000","last_peered":"2024-09-21T12:39:07.180934+0000","last_clean":"2024-09-21T12:39:07.180934+0000","last_became_active":"2024-09-21T12:37:20.468591+0000","last_became_peered":"2024-09-21T12:37:20.468591+0000","last_unstale":"2024-09-21T12:39:07.180934+0000","last_undegraded":"2024-09-21T12:39:07.180934+0000","last_fullsized":"2024-09-21T12:39:07.180934+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:37:55.128928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:37:55.128928+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:08.552968+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"484'17294","reported_seq":22653,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:12.076552+0000","last_change":"2024-09-21T12:38:49.464266+0000","last_active":"2024-09-21T12:39:12.076552+0000","last_peered":"2024-09-21T12:39:12.076552+0000","last_clean":"2024-09-21T12:39:12.076552+0000","last_became_active":"2024-09-21T12:38:49.404712+0000","last_became_peered":"2024-09-21T12:38:49.404712+0000","last_unstale":"2024-09-21T12:39:12.076552+0000","last_undegraded":"2024-09-21T12:39:12.076552+0000","last_fullsized":"2024-09-21T12:39:12.076552+0000","mapping_epoch":476,"log_start":"457'13000","ondisk_log_start":"457'13000","created":405,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"438'8003","last_scrub_stamp":"2024-09-21T12:37:13.555093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:37:13.555093+0000","objects_scrubbed":7888,"log_size":4294,"log_dups_size":0,"ondisk_log_size":4294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2942,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17294,"num_write_kb":557248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2537,"num_bytes_recovered":166199296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"76'79","reported_seq":310,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.166823+0000","last_change":"2024-09-21T12:38:51.310242+0000","last_active":"2024-09-21T12:39:07.166823+0000","last_peered":"2024-09-21T12:39:07.166823+0000","last_clean":"2024-09-21T12:39:07.166823+0000","last_became_active":"2024-09-21T12:38:07.159280+0000","last_became_peered":"2024-09-21T12:38:07.159280+0000","last_unstale":"2024-09-21T12:39:07.166823+0000","last_undegraded":"2024-09-21T12:39:07.166823+0000","last_fullsized":"2024-09-21T12:39:07.166823+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:38:51.310157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:38:51.310157+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:11.944946+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":1054,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.183030+0000","last_change":"2024-09-21T12:36:00.163117+0000","last_active":"2024-09-21T12:39:07.183030+0000","last_peered":"2024-09-21T12:39:07.183030+0000","last_clean":"2024-09-21T12:39:07.183030+0000","last_became_active":"2024-09-21T12:36:00.139849+0000","last_became_peered":"2024-09-21T12:36:00.139849+0000","last_unstale":"2024-09-21T12:39:07.183030+0000","last_undegraded":"2024-09-21T12:39:07.183030+0000","last_fullsized":"2024-09-21T12:39:07.183030+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T12:35:12.790809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:12.790809+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":905,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.174162+0000","last_change":"2024-09-21T12:36:00.152050+0000","last_active":"2024-09-21T12:39:07.174162+0000","last_peered":"2024-09-21T12:39:07.174162+0000","last_clean":"2024-09-21T12:39:07.174162+0000","last_became_active":"2024-09-21T12:36:00.151488+0000","last_became_peered":"2024-09-21T12:36:00.151488+0000","last_unstale":"2024-09-21T12:39:07.174162+0000","last_undegraded":"2024-09-21T12:39:07.174162+0000","last_fullsized":"2024-09-21T12:39:07.174162+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:22.752960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:22.752960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":305,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.165800+0000","last_change":"2024-09-21T12:36:00.137014+0000","last_active":"2024-09-21T12:39:07.165800+0000","last_peered":"2024-09-21T12:39:07.165800+0000","last_clean":"2024-09-21T12:39:07.165800+0000","last_became_active":"2024-09-21T12:36:00.136650+0000","last_became_peered":"2024-09-21T12:36:00.136650+0000","last_unstale":"2024-09-21T12:39:07.165800+0000","last_undegraded":"2024-09-21T12:39:07.165800+0000","last_fullsized":"2024-09-21T12:39:07.165800+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:12.216104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:12.216104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:04.402566+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":302,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.166431+0000","last_change":"2024-09-21T12:38:44.156366+0000","last_active":"2024-09-21T12:39:07.166431+0000","last_peered":"2024-09-21T12:39:07.166431+0000","last_clean":"2024-09-21T12:39:07.166431+0000","last_became_active":"2024-09-21T12:38:07.166919+0000","last_became_peered":"2024-09-21T12:38:07.166919+0000","last_unstale":"2024-09-21T12:39:07.166431+0000","last_undegraded":"2024-09-21T12:39:07.166431+0000","last_fullsized":"2024-09-21T12:39:07.166431+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:44.156307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:38:44.156307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:06.667086+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"484'17239","reported_seq":19523,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:11.951713+0000","last_change":"2024-09-21T12:38:01.860989+0000","last_active":"2024-09-21T12:39:11.951713+0000","last_peered":"2024-09-21T12:39:11.951713+0000","last_clean":"2024-09-21T12:39:11.951713+0000","last_became_active":"2024-09-21T12:36:08.101796+0000","last_became_peered":"2024-09-21T12:36:08.101796+0000","last_unstale":"2024-09-21T12:39:11.951713+0000","last_undegraded":"2024-09-21T12:39:11.951713+0000","last_fullsized":"2024-09-21T12:39:11.951713+0000","mapping_epoch":417,"log_start":"458'13100","ondisk_log_start":"458'13100","created":405,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"452'12252","last_scrub_stamp":"2024-09-21T12:38:01.860921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:38:01.860921+0000","objects_scrubbed":5200,"log_size":4139,"log_dups_size":0,"ondisk_log_size":4139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:22.227817+0000","scrub_duration":1407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8716288,"num_objects":133,"num_object_clones":0,"num_object_copies":399,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17239,"num_write_kb":555904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1016,"num_bytes_recovered":66584576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"76'79","reported_seq":841,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.175866+0000","last_change":"2024-09-21T12:36:43.462949+0000","last_active":"2024-09-21T12:39:07.175866+0000","last_peered":"2024-09-21T12:39:07.175866+0000","last_clean":"2024-09-21T12:39:07.175866+0000","last_became_active":"2024-09-21T12:36:00.155047+0000","last_became_peered":"2024-09-21T12:36:00.155047+0000","last_unstale":"2024-09-21T12:39:07.175866+0000","last_undegraded":"2024-09-21T12:39:07.175866+0000","last_fullsized":"2024-09-21T12:39:07.175866+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:43.462874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:43.462874+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":299,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.165690+0000","last_change":"2024-09-21T12:32:38.457999+0000","last_active":"2024-09-21T12:39:07.165690+0000","last_peered":"2024-09-21T12:39:07.165690+0000","last_clean":"2024-09-21T12:39:07.165690+0000","last_became_active":"2024-09-21T12:32:38.425453+0000","last_became_peered":"2024-09-21T12:32:38.425453+0000","last_unstale":"2024-09-21T12:39:07.165690+0000","last_undegraded":"2024-09-21T12:39:07.165690+0000","last_fullsized":"2024-09-21T12:39:07.165690+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-21T12:26:51.463698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:51.463698+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:33.319669+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":40,"num_read_kb":33,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":988,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.184270+0000","last_change":"2024-09-21T12:36:39.754233+0000","last_active":"2024-09-21T12:39:07.184270+0000","last_peered":"2024-09-21T12:39:07.184270+0000","last_clean":"2024-09-21T12:39:07.184270+0000","last_became_active":"2024-09-21T12:32:38.424412+0000","last_became_peered":"2024-09-21T12:32:38.424412+0000","last_unstale":"2024-09-21T12:39:07.184270+0000","last_undegraded":"2024-09-21T12:39:07.184270+0000","last_fullsized":"2024-09-21T12:39:07.184270+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:36:39.754173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:36:39.754173+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"352'12","reported_seq":1179,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.182841+0000","last_change":"2024-09-21T12:37:57.970843+0000","last_active":"2024-09-21T12:39:07.182841+0000","last_peered":"2024-09-21T12:39:07.182841+0000","last_clean":"2024-09-21T12:39:07.182841+0000","last_became_active":"2024-09-21T12:32:38.409177+0000","last_became_peered":"2024-09-21T12:32:38.409177+0000","last_unstale":"2024-09-21T12:39:07.182841+0000","last_undegraded":"2024-09-21T12:39:07.182841+0000","last_fullsized":"2024-09-21T12:39:07.182841+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'12","last_scrub_stamp":"2024-09-21T12:37:57.970636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:37:57.970636+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:08.843912+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":892,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.171247+0000","last_change":"2024-09-21T12:38:51.740285+0000","last_active":"2024-09-21T12:39:07.171247+0000","last_peered":"2024-09-21T12:39:07.171247+0000","last_clean":"2024-09-21T12:39:07.171247+0000","last_became_active":"2024-09-21T12:38:08.154409+0000","last_became_peered":"2024-09-21T12:38:08.154409+0000","last_unstale":"2024-09-21T12:39:07.171247+0000","last_undegraded":"2024-09-21T12:39:07.171247+0000","last_fullsized":"2024-09-21T12:39:07.171247+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:51.740136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:38:51.740136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:54.159637+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"484'16688","reported_seq":24434,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:11.927513+0000","last_change":"2024-09-21T12:38:31.314916+0000","last_active":"2024-09-21T12:39:11.927513+0000","last_peered":"2024-09-21T12:39:11.927513+0000","last_clean":"2024-09-21T12:39:11.927513+0000","last_became_active":"2024-09-21T12:38:31.294527+0000","last_became_peered":"2024-09-21T12:38:31.294527+0000","last_unstale":"2024-09-21T12:39:11.927513+0000","last_undegraded":"2024-09-21T12:39:11.927513+0000","last_fullsized":"2024-09-21T12:39:11.927513+0000","mapping_epoch":465,"log_start":"457'12500","ondisk_log_start":"457'12500","created":405,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4188,"log_dups_size":0,"ondisk_log_size":4188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:06.427301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7798808,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":120,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16688,"num_write_kb":537793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3781,"num_bytes_recovered":247726104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"76'79","reported_seq":381,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.174011+0000","last_change":"2024-09-21T12:37:16.761819+0000","last_active":"2024-09-21T12:39:07.174011+0000","last_peered":"2024-09-21T12:39:07.174011+0000","last_clean":"2024-09-21T12:39:07.174011+0000","last_became_active":"2024-09-21T12:36:00.151076+0000","last_became_peered":"2024-09-21T12:36:00.151076+0000","last_unstale":"2024-09-21T12:39:07.174011+0000","last_undegraded":"2024-09-21T12:39:07.174011+0000","last_fullsized":"2024-09-21T12:39:07.174011+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:37:16.761749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:37:16.761749+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":441,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.168071+0000","last_change":"2024-09-21T12:36:33.481707+0000","last_active":"2024-09-21T12:39:07.168071+0000","last_peered":"2024-09-21T12:39:07.168071+0000","last_clean":"2024-09-21T12:39:07.168071+0000","last_became_active":"2024-09-21T12:32:38.424412+0000","last_became_peered":"2024-09-21T12:32:38.424412+0000","last_unstale":"2024-09-21T12:39:07.168071+0000","last_undegraded":"2024-09-21T12:39:07.168071+0000","last_fullsized":"2024-09-21T12:39:07.168071+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:36:33.481627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:33.481627+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:07.097608+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":920,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.180687+0000","last_change":"2024-09-21T12:38:07.165927+0000","last_active":"2024-09-21T12:39:07.180687+0000","last_peered":"2024-09-21T12:39:07.180687+0000","last_clean":"2024-09-21T12:39:07.180687+0000","last_became_active":"2024-09-21T12:38:07.165614+0000","last_became_peered":"2024-09-21T12:38:07.165614+0000","last_unstale":"2024-09-21T12:39:07.180687+0000","last_undegraded":"2024-09-21T12:39:07.180687+0000","last_fullsized":"2024-09-21T12:39:07.180687+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:19.629083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:19.629083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:10.095481+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":1141,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.182887+0000","last_change":"2024-09-21T12:36:44.788732+0000","last_active":"2024-09-21T12:39:07.182887+0000","last_peered":"2024-09-21T12:39:07.182887+0000","last_clean":"2024-09-21T12:39:07.182887+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:39:07.182887+0000","last_undegraded":"2024-09-21T12:39:07.182887+0000","last_fullsized":"2024-09-21T12:39:07.182887+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:36:44.788662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:44.788662+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":902,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181133+0000","last_change":"2024-09-21T12:38:44.590928+0000","last_active":"2024-09-21T12:39:07.181133+0000","last_peered":"2024-09-21T12:39:07.181133+0000","last_clean":"2024-09-21T12:39:07.181133+0000","last_became_active":"2024-09-21T12:38:07.163353+0000","last_became_peered":"2024-09-21T12:38:07.163353+0000","last_unstale":"2024-09-21T12:39:07.181133+0000","last_undegraded":"2024-09-21T12:39:07.181133+0000","last_fullsized":"2024-09-21T12:39:07.181133+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:44.590868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:38:44.590868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:06.202952+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"484'17386","reported_seq":24341,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:11.870174+0000","last_change":"2024-09-21T12:38:51.655564+0000","last_active":"2024-09-21T12:39:11.870174+0000","last_peered":"2024-09-21T12:39:11.870174+0000","last_clean":"2024-09-21T12:39:11.870174+0000","last_became_active":"2024-09-21T12:38:42.397498+0000","last_became_peered":"2024-09-21T12:38:42.397498+0000","last_unstale":"2024-09-21T12:39:11.870174+0000","last_undegraded":"2024-09-21T12:39:11.870174+0000","last_fullsized":"2024-09-21T12:39:11.870174+0000","mapping_epoch":470,"log_start":"458'13200","ondisk_log_start":"458'13200","created":405,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'15790","last_scrub_stamp":"2024-09-21T12:38:51.655304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:38:51.655304+0000","objects_scrubbed":1772,"log_size":4186,"log_dups_size":0,"ondisk_log_size":4186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:08.589625+0000","scrub_duration":718,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9568256,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17386,"num_write_kb":561024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3370,"num_bytes_recovered":220856320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"76'79","reported_seq":576,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181090+0000","last_change":"2024-09-21T12:32:52.986242+0000","last_active":"2024-09-21T12:39:07.181090+0000","last_peered":"2024-09-21T12:39:07.181090+0000","last_clean":"2024-09-21T12:39:07.181090+0000","last_became_active":"2024-09-21T12:29:05.835884+0000","last_became_peered":"2024-09-21T12:29:05.835884+0000","last_unstale":"2024-09-21T12:39:07.181090+0000","last_undegraded":"2024-09-21T12:39:07.181090+0000","last_fullsized":"2024-09-21T12:39:07.181090+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:52.986175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:52.986175+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:57.609852+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":1000,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.182668+0000","last_change":"2024-09-21T12:36:36.774685+0000","last_active":"2024-09-21T12:39:07.182668+0000","last_peered":"2024-09-21T12:39:07.182668+0000","last_clean":"2024-09-21T12:39:07.182668+0000","last_became_active":"2024-09-21T12:36:00.127312+0000","last_became_peered":"2024-09-21T12:36:00.127312+0000","last_unstale":"2024-09-21T12:39:07.182668+0000","last_undegraded":"2024-09-21T12:39:07.182668+0000","last_fullsized":"2024-09-21T12:39:07.182668+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'29","last_scrub_stamp":"2024-09-21T12:36:36.774599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:36.774599+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":904,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.170817+0000","last_change":"2024-09-21T12:38:48.953684+0000","last_active":"2024-09-21T12:39:07.170817+0000","last_peered":"2024-09-21T12:39:07.170817+0000","last_clean":"2024-09-21T12:39:07.170817+0000","last_became_active":"2024-09-21T12:38:07.163439+0000","last_became_peered":"2024-09-21T12:38:07.163439+0000","last_unstale":"2024-09-21T12:39:07.170817+0000","last_undegraded":"2024-09-21T12:39:07.170817+0000","last_fullsized":"2024-09-21T12:39:07.170817+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:48.953635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:48.953635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:17.502212+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":480,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181361+0000","last_change":"2024-09-21T12:36:00.924159+0000","last_active":"2024-09-21T12:39:07.181361+0000","last_peered":"2024-09-21T12:39:07.181361+0000","last_clean":"2024-09-21T12:39:07.181361+0000","last_became_active":"2024-09-21T12:29:05.825869+0000","last_became_peered":"2024-09-21T12:29:05.825869+0000","last_unstale":"2024-09-21T12:39:07.181361+0000","last_undegraded":"2024-09-21T12:39:07.181361+0000","last_fullsized":"2024-09-21T12:39:07.181361+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:00.924084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:00.924084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":900,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.167565+0000","last_change":"2024-09-21T12:38:45.894780+0000","last_active":"2024-09-21T12:39:07.167565+0000","last_peered":"2024-09-21T12:39:07.167565+0000","last_clean":"2024-09-21T12:39:07.167565+0000","last_became_active":"2024-09-21T12:38:07.158992+0000","last_became_peered":"2024-09-21T12:38:07.158992+0000","last_unstale":"2024-09-21T12:39:07.167565+0000","last_undegraded":"2024-09-21T12:39:07.167565+0000","last_fullsized":"2024-09-21T12:39:07.167565+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:45.894717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:38:45.894717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:57.342298+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"484'16583","reported_seq":16755,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:08.130947+0000","last_change":"2024-09-21T12:35:47.791371+0000","last_active":"2024-09-21T12:39:08.130947+0000","last_peered":"2024-09-21T12:39:08.130947+0000","last_clean":"2024-09-21T12:39:08.130947+0000","last_became_active":"2024-09-21T12:35:47.791120+0000","last_became_peered":"2024-09-21T12:35:47.791120+0000","last_unstale":"2024-09-21T12:39:08.130947+0000","last_undegraded":"2024-09-21T12:39:08.130947+0000","last_fullsized":"2024-09-21T12:39:08.130947+0000","mapping_epoch":405,"log_start":"457'12500","ondisk_log_start":"457'12500","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4083,"log_dups_size":0,"ondisk_log_size":4083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31129600,"num_objects":475,"num_object_clones":0,"num_object_copies":1425,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16583,"num_write_kb":545856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"76'79","reported_seq":336,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.179930+0000","last_change":"2024-09-21T12:38:07.162380+0000","last_active":"2024-09-21T12:39:07.179930+0000","last_peered":"2024-09-21T12:39:07.179930+0000","last_clean":"2024-09-21T12:39:07.179930+0000","last_became_active":"2024-09-21T12:38:07.162110+0000","last_became_peered":"2024-09-21T12:38:07.162110+0000","last_unstale":"2024-09-21T12:39:07.179930+0000","last_undegraded":"2024-09-21T12:39:07.179930+0000","last_fullsized":"2024-09-21T12:39:07.179930+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:34.055720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:34.055720+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:24.198020+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"457'42","reported_seq":1196,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:10.589926+0000","last_change":"2024-09-21T12:38:07.440187+0000","last_active":"2024-09-21T12:39:10.589926+0000","last_peered":"2024-09-21T12:39:10.589926+0000","last_clean":"2024-09-21T12:39:10.589926+0000","last_became_active":"2024-09-21T12:38:07.164073+0000","last_became_peered":"2024-09-21T12:38:07.164073+0000","last_unstale":"2024-09-21T12:39:10.589926+0000","last_undegraded":"2024-09-21T12:39:10.589926+0000","last_fullsized":"2024-09-21T12:39:10.589926+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'41","last_scrub_stamp":"2024-09-21T12:38:07.440124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:07.440124+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:24.803396+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":299,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.166401+0000","last_change":"2024-09-21T12:37:50.899946+0000","last_active":"2024-09-21T12:39:07.166401+0000","last_peered":"2024-09-21T12:39:07.166401+0000","last_clean":"2024-09-21T12:39:07.166401+0000","last_became_active":"2024-09-21T12:32:38.426309+0000","last_became_peered":"2024-09-21T12:32:38.426309+0000","last_unstale":"2024-09-21T12:39:07.166401+0000","last_undegraded":"2024-09-21T12:39:07.166401+0000","last_fullsized":"2024-09-21T12:39:07.166401+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:50.899879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:37:50.899879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":626,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.168291+0000","last_change":"2024-09-21T12:37:57.480538+0000","last_active":"2024-09-21T12:39:07.168291+0000","last_peered":"2024-09-21T12:39:07.168291+0000","last_clean":"2024-09-21T12:39:07.168291+0000","last_became_active":"2024-09-21T12:26:54.083584+0000","last_became_peered":"2024-09-21T12:26:54.083584+0000","last_unstale":"2024-09-21T12:39:07.168291+0000","last_undegraded":"2024-09-21T12:39:07.168291+0000","last_fullsized":"2024-09-21T12:39:07.168291+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T12:37:57.480458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:37:57.480458+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:19.124474+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":938,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.183385+0000","last_change":"2024-09-21T12:38:47.559548+0000","last_active":"2024-09-21T12:39:07.183385+0000","last_peered":"2024-09-21T12:39:07.183385+0000","last_clean":"2024-09-21T12:39:07.183385+0000","last_became_active":"2024-09-21T12:32:38.424402+0000","last_became_peered":"2024-09-21T12:32:38.424402+0000","last_unstale":"2024-09-21T12:39:07.183385+0000","last_undegraded":"2024-09-21T12:39:07.183385+0000","last_fullsized":"2024-09-21T12:39:07.183385+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:47.559490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:47.559490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:03.876249+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":937,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.173142+0000","last_change":"2024-09-21T12:37:56.975879+0000","last_active":"2024-09-21T12:39:07.173142+0000","last_peered":"2024-09-21T12:39:07.173142+0000","last_clean":"2024-09-21T12:39:07.173142+0000","last_became_active":"2024-09-21T12:36:00.144526+0000","last_became_peered":"2024-09-21T12:36:00.144526+0000","last_unstale":"2024-09-21T12:39:07.173142+0000","last_undegraded":"2024-09-21T12:39:07.173142+0000","last_fullsized":"2024-09-21T12:39:07.173142+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:56.975826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:37:56.975826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"484'17005","reported_seq":17176,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:08.134350+0000","last_change":"2024-09-21T12:35:47.848839+0000","last_active":"2024-09-21T12:39:08.134350+0000","last_peered":"2024-09-21T12:39:08.134350+0000","last_clean":"2024-09-21T12:39:08.134350+0000","last_became_active":"2024-09-21T12:35:47.848543+0000","last_became_peered":"2024-09-21T12:35:47.848543+0000","last_unstale":"2024-09-21T12:39:08.134350+0000","last_undegraded":"2024-09-21T12:39:08.134350+0000","last_fullsized":"2024-09-21T12:39:08.134350+0000","mapping_epoch":405,"log_start":"457'12900","ondisk_log_start":"457'12900","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4105,"log_dups_size":0,"ondisk_log_size":4105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31391744,"num_objects":479,"num_object_clones":0,"num_object_copies":1437,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17005,"num_write_kb":559488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'79","reported_seq":323,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.171005+0000","last_change":"2024-09-21T12:38:50.763337+0000","last_active":"2024-09-21T12:39:07.171005+0000","last_peered":"2024-09-21T12:39:07.171005+0000","last_clean":"2024-09-21T12:39:07.171005+0000","last_became_active":"2024-09-21T12:36:01.131622+0000","last_became_peered":"2024-09-21T12:36:01.131622+0000","last_unstale":"2024-09-21T12:39:07.171005+0000","last_undegraded":"2024-09-21T12:39:07.171005+0000","last_fullsized":"2024-09-21T12:39:07.171005+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:38:50.763274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:38:50.763274+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:05.037000+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":403,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.170255+0000","last_change":"2024-09-21T12:37:52.583825+0000","last_active":"2024-09-21T12:39:07.170255+0000","last_peered":"2024-09-21T12:39:07.170255+0000","last_clean":"2024-09-21T12:39:07.170255+0000","last_became_active":"2024-09-21T12:31:39.057316+0000","last_became_peered":"2024-09-21T12:31:39.057316+0000","last_unstale":"2024-09-21T12:39:07.170255+0000","last_undegraded":"2024-09-21T12:39:07.170255+0000","last_fullsized":"2024-09-21T12:39:07.170255+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'17","last_scrub_stamp":"2024-09-21T12:37:52.583730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:37:52.583730+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:08.182509+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":936,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.167353+0000","last_change":"2024-09-21T12:38:49.953105+0000","last_active":"2024-09-21T12:39:07.167353+0000","last_peered":"2024-09-21T12:39:07.167353+0000","last_clean":"2024-09-21T12:39:07.167353+0000","last_became_active":"2024-09-21T12:38:07.154643+0000","last_became_peered":"2024-09-21T12:38:07.154643+0000","last_unstale":"2024-09-21T12:39:07.167353+0000","last_undegraded":"2024-09-21T12:39:07.167353+0000","last_fullsized":"2024-09-21T12:39:07.167353+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:49.952845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:49.952845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:51.858894+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":244,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.174482+0000","last_change":"2024-09-21T12:38:49.202301+0000","last_active":"2024-09-21T12:39:07.174482+0000","last_peered":"2024-09-21T12:39:07.174482+0000","last_clean":"2024-09-21T12:39:07.174482+0000","last_became_active":"2024-09-21T12:38:07.168680+0000","last_became_peered":"2024-09-21T12:38:07.168680+0000","last_unstale":"2024-09-21T12:39:07.174482+0000","last_undegraded":"2024-09-21T12:39:07.174482+0000","last_fullsized":"2024-09-21T12:39:07.174482+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:38:49.202240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:38:49.202240+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:56.553523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":235,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.180887+0000","last_change":"2024-09-21T12:38:08.157936+0000","last_active":"2024-09-21T12:39:07.180887+0000","last_peered":"2024-09-21T12:39:07.180887+0000","last_clean":"2024-09-21T12:39:07.180887+0000","last_became_active":"2024-09-21T12:38:08.157608+0000","last_became_peered":"2024-09-21T12:38:08.157608+0000","last_unstale":"2024-09-21T12:39:07.180887+0000","last_undegraded":"2024-09-21T12:39:07.180887+0000","last_fullsized":"2024-09-21T12:39:07.180887+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:02.295255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:02.295255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:25.229168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":136,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.183301+0000","last_change":"2024-09-21T12:38:08.166386+0000","last_active":"2024-09-21T12:39:07.183301+0000","last_peered":"2024-09-21T12:39:07.183301+0000","last_clean":"2024-09-21T12:39:07.183301+0000","last_became_active":"2024-09-21T12:38:08.165465+0000","last_became_peered":"2024-09-21T12:38:08.165465+0000","last_unstale":"2024-09-21T12:39:07.183301+0000","last_undegraded":"2024-09-21T12:39:07.183301+0000","last_fullsized":"2024-09-21T12:39:07.183301+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:53.671046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:37:53.671046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:20.092756+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"484'16780","reported_seq":16951,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:08.105703+0000","last_change":"2024-09-21T12:35:47.791375+0000","last_active":"2024-09-21T12:39:08.105703+0000","last_peered":"2024-09-21T12:39:08.105703+0000","last_clean":"2024-09-21T12:39:08.105703+0000","last_became_active":"2024-09-21T12:35:47.791073+0000","last_became_peered":"2024-09-21T12:35:47.791073+0000","last_unstale":"2024-09-21T12:39:08.105703+0000","last_undegraded":"2024-09-21T12:39:08.105703+0000","last_fullsized":"2024-09-21T12:39:08.105703+0000","mapping_epoch":405,"log_start":"457'12700","ondisk_log_start":"457'12700","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4080,"log_dups_size":0,"ondisk_log_size":4080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29229056,"num_objects":446,"num_object_clones":0,"num_object_copies":1338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16780,"num_write_kb":551232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"76'79","reported_seq":858,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.177660+0000","last_change":"2024-09-21T12:36:41.495668+0000","last_active":"2024-09-21T12:39:07.177660+0000","last_peered":"2024-09-21T12:39:07.177660+0000","last_clean":"2024-09-21T12:39:07.177660+0000","last_became_active":"2024-09-21T12:32:38.424710+0000","last_became_peered":"2024-09-21T12:32:38.424710+0000","last_unstale":"2024-09-21T12:39:07.177660+0000","last_undegraded":"2024-09-21T12:39:07.177660+0000","last_fullsized":"2024-09-21T12:39:07.177660+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:41.495532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:41.495532+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:55.878434+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":945,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.183554+0000","last_change":"2024-09-21T12:38:07.517859+0000","last_active":"2024-09-21T12:39:07.183554+0000","last_peered":"2024-09-21T12:39:07.183554+0000","last_clean":"2024-09-21T12:39:07.183554+0000","last_became_active":"2024-09-21T12:38:07.166839+0000","last_became_peered":"2024-09-21T12:38:07.166839+0000","last_unstale":"2024-09-21T12:39:07.183554+0000","last_undegraded":"2024-09-21T12:39:07.183554+0000","last_fullsized":"2024-09-21T12:39:07.183554+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'24","last_scrub_stamp":"2024-09-21T12:38:07.517773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:07.517773+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:28.299185+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":953,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181635+0000","last_change":"2024-09-21T12:38:48.558827+0000","last_active":"2024-09-21T12:39:07.181635+0000","last_peered":"2024-09-21T12:39:07.181635+0000","last_clean":"2024-09-21T12:39:07.181635+0000","last_became_active":"2024-09-21T12:32:38.411004+0000","last_became_peered":"2024-09-21T12:32:38.411004+0000","last_unstale":"2024-09-21T12:39:07.181635+0000","last_undegraded":"2024-09-21T12:39:07.181635+0000","last_fullsized":"2024-09-21T12:39:07.181635+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:48.558779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:48.558779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:01.839466+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":355,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:10.516184+0000","last_change":"2024-09-21T12:36:27.852222+0000","last_active":"2024-09-21T12:39:10.516184+0000","last_peered":"2024-09-21T12:39:10.516184+0000","last_clean":"2024-09-21T12:39:10.516184+0000","last_became_active":"2024-09-21T12:36:00.144053+0000","last_became_peered":"2024-09-21T12:36:00.144053+0000","last_unstale":"2024-09-21T12:39:10.516184+0000","last_undegraded":"2024-09-21T12:39:10.516184+0000","last_fullsized":"2024-09-21T12:39:10.516184+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-21T12:36:27.852030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:36:27.852030+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:58.750782+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":198,"num_read_kb":193,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":347,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.180938+0000","last_change":"2024-09-21T12:38:54.293655+0000","last_active":"2024-09-21T12:39:07.180938+0000","last_peered":"2024-09-21T12:39:07.180938+0000","last_clean":"2024-09-21T12:39:07.180938+0000","last_became_active":"2024-09-21T12:32:02.030240+0000","last_became_peered":"2024-09-21T12:32:02.030240+0000","last_unstale":"2024-09-21T12:39:07.180938+0000","last_undegraded":"2024-09-21T12:39:07.180938+0000","last_fullsized":"2024-09-21T12:39:07.180938+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:54.293611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:54.293611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:20.703408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":391,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.167485+0000","last_change":"2024-09-21T12:37:58.528094+0000","last_active":"2024-09-21T12:39:07.167485+0000","last_peered":"2024-09-21T12:39:07.167485+0000","last_clean":"2024-09-21T12:39:07.167485+0000","last_became_active":"2024-09-21T12:31:39.060160+0000","last_became_peered":"2024-09-21T12:31:39.060160+0000","last_unstale":"2024-09-21T12:39:07.167485+0000","last_undegraded":"2024-09-21T12:39:07.167485+0000","last_fullsized":"2024-09-21T12:39:07.167485+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:58.528033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:37:58.528033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"484'16752","reported_seq":16924,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:08.099134+0000","last_change":"2024-09-21T12:35:47.849177+0000","last_active":"2024-09-21T12:39:08.099134+0000","last_peered":"2024-09-21T12:39:08.099134+0000","last_clean":"2024-09-21T12:39:08.099134+0000","last_became_active":"2024-09-21T12:35:47.848768+0000","last_became_peered":"2024-09-21T12:35:47.848768+0000","last_unstale":"2024-09-21T12:39:08.099134+0000","last_undegraded":"2024-09-21T12:39:08.099134+0000","last_fullsized":"2024-09-21T12:39:08.099134+0000","mapping_epoch":405,"log_start":"454'12600","ondisk_log_start":"454'12600","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4152,"log_dups_size":0,"ondisk_log_size":4152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31981568,"num_objects":488,"num_object_clones":0,"num_object_copies":1464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16752,"num_write_kb":551680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"415'35","reported_seq":446,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.166949+0000","last_change":"2024-09-21T12:36:43.681619+0000","last_active":"2024-09-21T12:39:07.166949+0000","last_peered":"2024-09-21T12:39:07.166949+0000","last_clean":"2024-09-21T12:39:07.166949+0000","last_became_active":"2024-09-21T12:36:00.139342+0000","last_became_peered":"2024-09-21T12:36:00.139342+0000","last_unstale":"2024-09-21T12:39:07.166949+0000","last_undegraded":"2024-09-21T12:39:07.166949+0000","last_fullsized":"2024-09-21T12:39:07.166949+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'35","last_scrub_stamp":"2024-09-21T12:36:43.681471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:43.681471+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:09.405650+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":914,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.180089+0000","last_change":"2024-09-21T12:38:01.901094+0000","last_active":"2024-09-21T12:39:07.180089+0000","last_peered":"2024-09-21T12:39:07.180089+0000","last_clean":"2024-09-21T12:39:07.180089+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:39:07.180089+0000","last_undegraded":"2024-09-21T12:39:07.180089+0000","last_fullsized":"2024-09-21T12:39:07.180089+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:38:01.898469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:01.898469+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:08.392980+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":159,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.170944+0000","last_change":"2024-09-21T12:37:14.690524+0000","last_active":"2024-09-21T12:39:07.170944+0000","last_peered":"2024-09-21T12:39:07.170944+0000","last_clean":"2024-09-21T12:39:07.170944+0000","last_became_active":"2024-09-21T12:36:00.134878+0000","last_became_peered":"2024-09-21T12:36:00.134878+0000","last_unstale":"2024-09-21T12:39:07.170944+0000","last_undegraded":"2024-09-21T12:39:07.170944+0000","last_fullsized":"2024-09-21T12:39:07.170944+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:37:14.690456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:37:14.690456+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":326,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.171240+0000","last_change":"2024-09-21T12:38:08.164955+0000","last_active":"2024-09-21T12:39:07.171240+0000","last_peered":"2024-09-21T12:39:07.171240+0000","last_clean":"2024-09-21T12:39:07.171240+0000","last_became_active":"2024-09-21T12:38:08.164554+0000","last_became_peered":"2024-09-21T12:38:08.164554+0000","last_unstale":"2024-09-21T12:39:07.171240+0000","last_undegraded":"2024-09-21T12:39:07.171240+0000","last_fullsized":"2024-09-21T12:39:07.171240+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:22.649558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:37:22.649558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":563,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181134+0000","last_change":"2024-09-21T12:38:00.438673+0000","last_active":"2024-09-21T12:39:07.181134+0000","last_peered":"2024-09-21T12:39:07.181134+0000","last_clean":"2024-09-21T12:39:07.181134+0000","last_became_active":"2024-09-21T12:37:20.485126+0000","last_became_peered":"2024-09-21T12:37:20.485126+0000","last_unstale":"2024-09-21T12:39:07.181134+0000","last_undegraded":"2024-09-21T12:39:07.181134+0000","last_fullsized":"2024-09-21T12:39:07.181134+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:00.438586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:38:00.438586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":331,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"484'16806","reported_seq":16978,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:08.117545+0000","last_change":"2024-09-21T12:35:47.788808+0000","last_active":"2024-09-21T12:39:08.117545+0000","last_peered":"2024-09-21T12:39:08.117545+0000","last_clean":"2024-09-21T12:39:08.117545+0000","last_became_active":"2024-09-21T12:35:47.788318+0000","last_became_peered":"2024-09-21T12:35:47.788318+0000","last_unstale":"2024-09-21T12:39:08.117545+0000","last_undegraded":"2024-09-21T12:39:08.117545+0000","last_fullsized":"2024-09-21T12:39:08.117545+0000","mapping_epoch":405,"log_start":"457'12700","ondisk_log_start":"457'12700","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4106,"log_dups_size":0,"ondisk_log_size":4106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30277632,"num_objects":462,"num_object_clones":0,"num_object_copies":1386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16806,"num_write_kb":552576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"76'79","reported_seq":854,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.177930+0000","last_change":"2024-09-21T12:38:40.853940+0000","last_active":"2024-09-21T12:39:07.177930+0000","last_peered":"2024-09-21T12:39:07.177930+0000","last_clean":"2024-09-21T12:39:07.177930+0000","last_became_active":"2024-09-21T12:27:14.954351+0000","last_became_peered":"2024-09-21T12:27:14.954351+0000","last_unstale":"2024-09-21T12:39:07.177930+0000","last_undegraded":"2024-09-21T12:39:07.177930+0000","last_fullsized":"2024-09-21T12:39:07.177930+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:38:40.853870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:38:40.853870+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:57.441950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":310,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.167315+0000","last_change":"2024-09-21T12:36:41.666876+0000","last_active":"2024-09-21T12:39:07.167315+0000","last_peered":"2024-09-21T12:39:07.167315+0000","last_clean":"2024-09-21T12:39:07.167315+0000","last_became_active":"2024-09-21T12:32:38.419714+0000","last_became_peered":"2024-09-21T12:32:38.419714+0000","last_unstale":"2024-09-21T12:39:07.167315+0000","last_undegraded":"2024-09-21T12:39:07.167315+0000","last_fullsized":"2024-09-21T12:39:07.167315+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:36:41.666812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:41.666812+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":321,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.165830+0000","last_change":"2024-09-21T12:38:08.148418+0000","last_active":"2024-09-21T12:39:07.165830+0000","last_peered":"2024-09-21T12:39:07.165830+0000","last_clean":"2024-09-21T12:39:07.165830+0000","last_became_active":"2024-09-21T12:38:08.148109+0000","last_became_peered":"2024-09-21T12:38:08.148109+0000","last_unstale":"2024-09-21T12:39:07.165830+0000","last_undegraded":"2024-09-21T12:39:07.165830+0000","last_fullsized":"2024-09-21T12:39:07.165830+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:40.708469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:36:40.708469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":442,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181217+0000","last_change":"2024-09-21T12:38:38.201658+0000","last_active":"2024-09-21T12:39:07.181217+0000","last_peered":"2024-09-21T12:39:07.181217+0000","last_clean":"2024-09-21T12:39:07.181217+0000","last_became_active":"2024-09-21T12:37:20.467986+0000","last_became_peered":"2024-09-21T12:37:20.467986+0000","last_unstale":"2024-09-21T12:39:07.181217+0000","last_undegraded":"2024-09-21T12:39:07.181217+0000","last_fullsized":"2024-09-21T12:39:07.181217+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:38.201448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:38.201448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:46.239927+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":887,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.175952+0000","last_change":"2024-09-21T12:35:24.786713+0000","last_active":"2024-09-21T12:39:07.175952+0000","last_peered":"2024-09-21T12:39:07.175952+0000","last_clean":"2024-09-21T12:39:07.175952+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:39:07.175952+0000","last_undegraded":"2024-09-21T12:39:07.175952+0000","last_fullsized":"2024-09-21T12:39:07.175952+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:24.786609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:24.786609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"484'16687","reported_seq":23592,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:08.127467+0000","last_change":"2024-09-21T12:38:41.459562+0000","last_active":"2024-09-21T12:39:08.127467+0000","last_peered":"2024-09-21T12:39:08.127467+0000","last_clean":"2024-09-21T12:39:08.127467+0000","last_became_active":"2024-09-21T12:38:41.419928+0000","last_became_peered":"2024-09-21T12:38:41.419928+0000","last_unstale":"2024-09-21T12:39:08.127467+0000","last_undegraded":"2024-09-21T12:39:08.127467+0000","last_fullsized":"2024-09-21T12:39:08.127467+0000","mapping_epoch":469,"log_start":"457'12600","ondisk_log_start":"457'12600","created":405,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4087,"log_dups_size":0,"ondisk_log_size":4087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31784960,"num_objects":485,"num_object_clones":0,"num_object_copies":1455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16687,"num_write_kb":549504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3360,"num_bytes_recovered":220135424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"76'79","reported_seq":234,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.182697+0000","last_change":"2024-09-21T12:38:07.162337+0000","last_active":"2024-09-21T12:39:07.182697+0000","last_peered":"2024-09-21T12:39:07.182697+0000","last_clean":"2024-09-21T12:39:07.182697+0000","last_became_active":"2024-09-21T12:38:07.161730+0000","last_became_peered":"2024-09-21T12:38:07.161730+0000","last_unstale":"2024-09-21T12:39:07.182697+0000","last_undegraded":"2024-09-21T12:39:07.182697+0000","last_fullsized":"2024-09-21T12:39:07.182697+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:17.035216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:17.035216+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:05.361038+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":327,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.166680+0000","last_change":"2024-09-21T12:38:53.268874+0000","last_active":"2024-09-21T12:39:07.166680+0000","last_peered":"2024-09-21T12:39:07.166680+0000","last_clean":"2024-09-21T12:39:07.166680+0000","last_became_active":"2024-09-21T12:38:07.163226+0000","last_became_peered":"2024-09-21T12:38:07.163226+0000","last_unstale":"2024-09-21T12:39:07.166680+0000","last_undegraded":"2024-09-21T12:39:07.166680+0000","last_fullsized":"2024-09-21T12:39:07.166680+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'31","last_scrub_stamp":"2024-09-21T12:38:53.268744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:53.268744+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:08.857112+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":307,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.166018+0000","last_change":"2024-09-21T12:37:53.993984+0000","last_active":"2024-09-21T12:39:07.166018+0000","last_peered":"2024-09-21T12:39:07.166018+0000","last_clean":"2024-09-21T12:39:07.166018+0000","last_became_active":"2024-09-21T12:32:38.424451+0000","last_became_peered":"2024-09-21T12:32:38.424451+0000","last_unstale":"2024-09-21T12:39:07.166018+0000","last_undegraded":"2024-09-21T12:39:07.166018+0000","last_fullsized":"2024-09-21T12:39:07.166018+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:53.993920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:37:53.993920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":920,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.170299+0000","last_change":"2024-09-21T12:36:43.539109+0000","last_active":"2024-09-21T12:39:07.170299+0000","last_peered":"2024-09-21T12:39:07.170299+0000","last_clean":"2024-09-21T12:39:07.170299+0000","last_became_active":"2024-09-21T12:32:38.425177+0000","last_became_peered":"2024-09-21T12:32:38.425177+0000","last_unstale":"2024-09-21T12:39:07.170299+0000","last_undegraded":"2024-09-21T12:39:07.170299+0000","last_fullsized":"2024-09-21T12:39:07.170299+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:43.539053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:43.539053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:05.078879+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":844,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.173536+0000","last_change":"2024-09-21T12:38:07.160870+0000","last_active":"2024-09-21T12:39:07.173536+0000","last_peered":"2024-09-21T12:39:07.173536+0000","last_clean":"2024-09-21T12:39:07.173536+0000","last_became_active":"2024-09-21T12:38:07.160538+0000","last_became_peered":"2024-09-21T12:38:07.160538+0000","last_unstale":"2024-09-21T12:39:07.173536+0000","last_undegraded":"2024-09-21T12:39:07.173536+0000","last_fullsized":"2024-09-21T12:39:07.173536+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:20.005906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:34:20.005906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:01.287896+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3","version":"484'17139","reported_seq":19550,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:12.009927+0000","last_change":"2024-09-21T12:38:43.975447+0000","last_active":"2024-09-21T12:39:12.009927+0000","last_peered":"2024-09-21T12:39:12.009927+0000","last_clean":"2024-09-21T12:39:12.009927+0000","last_became_active":"2024-09-21T12:36:13.105381+0000","last_became_peered":"2024-09-21T12:36:13.105381+0000","last_unstale":"2024-09-21T12:39:12.009927+0000","last_undegraded":"2024-09-21T12:39:12.009927+0000","last_fullsized":"2024-09-21T12:39:12.009927+0000","mapping_epoch":420,"log_start":"453'12400","ondisk_log_start":"453'12400","created":405,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"472'14907","last_scrub_stamp":"2024-09-21T12:38:43.975362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:38:43.975362+0000","objects_scrubbed":2350,"log_size":4739,"log_dups_size":0,"ondisk_log_size":4739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:46.744558+0000","scrub_duration":422,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7274496,"num_objects":111,"num_object_clones":0,"num_object_copies":333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17139,"num_write_kb":552000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1051,"num_bytes_recovered":68878336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"76'79","reported_seq":977,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.171044+0000","last_change":"2024-09-21T12:38:48.834565+0000","last_active":"2024-09-21T12:39:07.171044+0000","last_peered":"2024-09-21T12:39:07.171044+0000","last_clean":"2024-09-21T12:39:07.171044+0000","last_became_active":"2024-09-21T12:38:08.156755+0000","last_became_peered":"2024-09-21T12:38:08.156755+0000","last_unstale":"2024-09-21T12:39:07.171044+0000","last_undegraded":"2024-09-21T12:39:07.171044+0000","last_fullsized":"2024-09-21T12:39:07.171044+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:38:48.834502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:38:48.834502+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:17.209408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":313,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.166368+0000","last_change":"2024-09-21T12:38:46.208578+0000","last_active":"2024-09-21T12:39:07.166368+0000","last_peered":"2024-09-21T12:39:07.166368+0000","last_clean":"2024-09-21T12:39:07.166368+0000","last_became_active":"2024-09-21T12:38:07.162180+0000","last_became_peered":"2024-09-21T12:38:07.162180+0000","last_unstale":"2024-09-21T12:39:07.166368+0000","last_undegraded":"2024-09-21T12:39:07.166368+0000","last_fullsized":"2024-09-21T12:39:07.166368+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:38:46.208493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:46.208493+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:10.147140+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":927,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.175138+0000","last_change":"2024-09-21T12:38:44.144209+0000","last_active":"2024-09-21T12:39:07.175138+0000","last_peered":"2024-09-21T12:39:07.175138+0000","last_clean":"2024-09-21T12:39:07.175138+0000","last_became_active":"2024-09-21T12:32:38.403997+0000","last_became_peered":"2024-09-21T12:32:38.403997+0000","last_unstale":"2024-09-21T12:39:07.175138+0000","last_undegraded":"2024-09-21T12:39:07.175138+0000","last_fullsized":"2024-09-21T12:39:07.175138+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:44.144166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:44.144166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:55.330893+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"457'16","reported_seq":312,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.171713+0000","last_change":"2024-09-21T12:38:08.167263+0000","last_active":"2024-09-21T12:39:07.171713+0000","last_peered":"2024-09-21T12:39:07.171713+0000","last_clean":"2024-09-21T12:39:07.171713+0000","last_became_active":"2024-09-21T12:38:08.165321+0000","last_became_peered":"2024-09-21T12:38:08.165321+0000","last_unstale":"2024-09-21T12:39:07.171713+0000","last_undegraded":"2024-09-21T12:39:07.171713+0000","last_fullsized":"2024-09-21T12:39:07.171713+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'14","last_scrub_stamp":"2024-09-21T12:36:31.838024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:31.838024+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":301,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.167215+0000","last_change":"2024-09-21T12:38:07.164086+0000","last_active":"2024-09-21T12:39:07.167215+0000","last_peered":"2024-09-21T12:39:07.167215+0000","last_clean":"2024-09-21T12:39:07.167215+0000","last_became_active":"2024-09-21T12:38:07.163242+0000","last_became_peered":"2024-09-21T12:38:07.163242+0000","last_unstale":"2024-09-21T12:39:07.167215+0000","last_undegraded":"2024-09-21T12:39:07.167215+0000","last_fullsized":"2024-09-21T12:39:07.167215+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:06.369436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:06.369436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:59.467726+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"484'17181","reported_seq":22533,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:12.078536+0000","last_change":"2024-09-21T12:38:49.384862+0000","last_active":"2024-09-21T12:39:12.078536+0000","last_peered":"2024-09-21T12:39:12.078536+0000","last_clean":"2024-09-21T12:39:12.078536+0000","last_became_active":"2024-09-21T12:38:49.364306+0000","last_became_peered":"2024-09-21T12:38:49.364306+0000","last_unstale":"2024-09-21T12:39:12.078536+0000","last_undegraded":"2024-09-21T12:39:12.078536+0000","last_fullsized":"2024-09-21T12:39:12.078536+0000","mapping_epoch":476,"log_start":"458'12900","ondisk_log_start":"458'12900","created":405,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'8269","last_scrub_stamp":"2024-09-21T12:37:18.159924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:37:18.159924+0000","objects_scrubbed":8165,"log_size":4281,"log_dups_size":0,"ondisk_log_size":4281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8585216,"num_objects":131,"num_object_clones":0,"num_object_copies":393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17181,"num_write_kb":553984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2533,"num_bytes_recovered":166002688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"76'79","reported_seq":310,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.167382+0000","last_change":"2024-09-21T12:37:48.990420+0000","last_active":"2024-09-21T12:39:07.167382+0000","last_peered":"2024-09-21T12:39:07.167382+0000","last_clean":"2024-09-21T12:39:07.167382+0000","last_became_active":"2024-09-21T12:36:01.159481+0000","last_became_peered":"2024-09-21T12:36:01.159481+0000","last_unstale":"2024-09-21T12:39:07.167382+0000","last_undegraded":"2024-09-21T12:39:07.167382+0000","last_fullsized":"2024-09-21T12:39:07.167382+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:37:48.990356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:37:48.990356+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":1015,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.182759+0000","last_change":"2024-09-21T12:36:40.771296+0000","last_active":"2024-09-21T12:39:07.182759+0000","last_peered":"2024-09-21T12:39:07.182759+0000","last_clean":"2024-09-21T12:39:07.182759+0000","last_became_active":"2024-09-21T12:32:38.424376+0000","last_became_peered":"2024-09-21T12:32:38.424376+0000","last_unstale":"2024-09-21T12:39:07.182759+0000","last_undegraded":"2024-09-21T12:39:07.182759+0000","last_fullsized":"2024-09-21T12:39:07.182759+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:36:40.771222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:40.771222+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":437,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.180663+0000","last_change":"2024-09-21T12:38:51.327197+0000","last_active":"2024-09-21T12:39:07.180663+0000","last_peered":"2024-09-21T12:39:07.180663+0000","last_clean":"2024-09-21T12:39:07.180663+0000","last_became_active":"2024-09-21T12:37:20.484773+0000","last_became_peered":"2024-09-21T12:37:20.484773+0000","last_unstale":"2024-09-21T12:39:07.180663+0000","last_undegraded":"2024-09-21T12:39:07.180663+0000","last_fullsized":"2024-09-21T12:39:07.180663+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:51.327127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:51.327127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:13.918622+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":872,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.166160+0000","last_change":"2024-09-21T12:38:38.252977+0000","last_active":"2024-09-21T12:39:07.166160+0000","last_peered":"2024-09-21T12:39:07.166160+0000","last_clean":"2024-09-21T12:39:07.166160+0000","last_became_active":"2024-09-21T12:38:07.165226+0000","last_became_peered":"2024-09-21T12:38:07.165226+0000","last_unstale":"2024-09-21T12:39:07.166160+0000","last_undegraded":"2024-09-21T12:39:07.166160+0000","last_fullsized":"2024-09-21T12:39:07.166160+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:38.252885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:38.252885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:06.853610+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":912,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181705+0000","last_change":"2024-09-21T12:38:08.164968+0000","last_active":"2024-09-21T12:39:07.181705+0000","last_peered":"2024-09-21T12:39:07.181705+0000","last_clean":"2024-09-21T12:39:07.181705+0000","last_became_active":"2024-09-21T12:38:08.164626+0000","last_became_peered":"2024-09-21T12:38:08.164626+0000","last_unstale":"2024-09-21T12:39:07.181705+0000","last_undegraded":"2024-09-21T12:39:07.181705+0000","last_fullsized":"2024-09-21T12:39:07.181705+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:34.842666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:34.842666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"484'16947","reported_seq":19314,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:12.063890+0000","last_change":"2024-09-21T12:37:23.352215+0000","last_active":"2024-09-21T12:39:12.063890+0000","last_peered":"2024-09-21T12:39:12.063890+0000","last_clean":"2024-09-21T12:39:12.063890+0000","last_became_active":"2024-09-21T12:36:12.104630+0000","last_became_peered":"2024-09-21T12:36:12.104630+0000","last_unstale":"2024-09-21T12:39:12.063890+0000","last_undegraded":"2024-09-21T12:39:12.063890+0000","last_fullsized":"2024-09-21T12:39:12.063890+0000","mapping_epoch":419,"log_start":"457'12700","ondisk_log_start":"457'12700","created":405,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'8526","last_scrub_stamp":"2024-09-21T12:37:23.351998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:37:23.351998+0000","objects_scrubbed":8413,"log_size":4247,"log_dups_size":0,"ondisk_log_size":4247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2758,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8323072,"num_objects":127,"num_object_clones":0,"num_object_copies":381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16947,"num_write_kb":546368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1047,"num_bytes_recovered":68616192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"76'79","reported_seq":838,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181175+0000","last_change":"2024-09-21T12:38:41.145390+0000","last_active":"2024-09-21T12:39:07.181175+0000","last_peered":"2024-09-21T12:39:07.181175+0000","last_clean":"2024-09-21T12:39:07.181175+0000","last_became_active":"2024-09-21T12:32:38.414877+0000","last_became_peered":"2024-09-21T12:32:38.414877+0000","last_unstale":"2024-09-21T12:39:07.181175+0000","last_undegraded":"2024-09-21T12:39:07.181175+0000","last_fullsized":"2024-09-21T12:39:07.181175+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:38:41.145341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:38:41.145341+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:00.633321+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":1083,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.173203+0000","last_change":"2024-09-21T12:38:44.587607+0000","last_active":"2024-09-21T12:39:07.173203+0000","last_peered":"2024-09-21T12:39:07.173203+0000","last_clean":"2024-09-21T12:39:07.173203+0000","last_became_active":"2024-09-21T12:36:00.148392+0000","last_became_peered":"2024-09-21T12:36:00.148392+0000","last_unstale":"2024-09-21T12:39:07.173203+0000","last_undegraded":"2024-09-21T12:39:07.173203+0000","last_fullsized":"2024-09-21T12:39:07.173203+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'35","last_scrub_stamp":"2024-09-21T12:38:44.587516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:44.587516+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:45.419078+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":159,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.171404+0000","last_change":"2024-09-21T12:38:53.700983+0000","last_active":"2024-09-21T12:39:07.171404+0000","last_peered":"2024-09-21T12:39:07.171404+0000","last_clean":"2024-09-21T12:39:07.171404+0000","last_became_active":"2024-09-21T12:36:00.145287+0000","last_became_peered":"2024-09-21T12:36:00.145287+0000","last_unstale":"2024-09-21T12:39:07.171404+0000","last_undegraded":"2024-09-21T12:39:07.171404+0000","last_fullsized":"2024-09-21T12:39:07.171404+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:53.700939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:53.700939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:55.132890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":802,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.183037+0000","last_change":"2024-09-21T12:38:07.162955+0000","last_active":"2024-09-21T12:39:07.183037+0000","last_peered":"2024-09-21T12:39:07.183037+0000","last_clean":"2024-09-21T12:39:07.183037+0000","last_became_active":"2024-09-21T12:38:07.162684+0000","last_became_peered":"2024-09-21T12:38:07.162684+0000","last_unstale":"2024-09-21T12:39:07.183037+0000","last_undegraded":"2024-09-21T12:39:07.183037+0000","last_fullsized":"2024-09-21T12:39:07.183037+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:20.243774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:20.243774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:05.650260+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":891,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.171049+0000","last_change":"2024-09-21T12:38:08.164642+0000","last_active":"2024-09-21T12:39:07.171049+0000","last_peered":"2024-09-21T12:39:07.171049+0000","last_clean":"2024-09-21T12:39:07.171049+0000","last_became_active":"2024-09-21T12:38:08.164339+0000","last_became_peered":"2024-09-21T12:38:08.164339+0000","last_unstale":"2024-09-21T12:39:07.171049+0000","last_undegraded":"2024-09-21T12:39:07.171049+0000","last_fullsized":"2024-09-21T12:39:07.171049+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:48.619605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:37:48.619605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"484'16704","reported_seq":22169,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:08.525592+0000","last_change":"2024-09-21T12:38:44.468122+0000","last_active":"2024-09-21T12:39:08.525592+0000","last_peered":"2024-09-21T12:39:08.525592+0000","last_clean":"2024-09-21T12:39:08.525592+0000","last_became_active":"2024-09-21T12:38:44.441467+0000","last_became_peered":"2024-09-21T12:38:44.441467+0000","last_unstale":"2024-09-21T12:39:08.525592+0000","last_undegraded":"2024-09-21T12:39:08.525592+0000","last_fullsized":"2024-09-21T12:39:08.525592+0000","mapping_epoch":472,"log_start":"450'11823","ondisk_log_start":"450'11823","created":405,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4881,"log_dups_size":0,"ondisk_log_size":4881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29753344,"num_objects":454,"num_object_clones":0,"num_object_copies":1362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16704,"num_write_kb":549056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2640,"num_bytes_recovered":172883968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"76'79","reported_seq":853,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.176162+0000","last_change":"2024-09-21T12:36:31.563764+0000","last_active":"2024-09-21T12:39:07.176162+0000","last_peered":"2024-09-21T12:39:07.176162+0000","last_clean":"2024-09-21T12:39:07.176162+0000","last_became_active":"2024-09-21T12:27:14.951375+0000","last_became_peered":"2024-09-21T12:27:14.951375+0000","last_unstale":"2024-09-21T12:39:07.176162+0000","last_undegraded":"2024-09-21T12:39:07.176162+0000","last_fullsized":"2024-09-21T12:39:07.176162+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:31.563694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:31.563694+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":311,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.165996+0000","last_change":"2024-09-21T12:37:52.992717+0000","last_active":"2024-09-21T12:39:07.165996+0000","last_peered":"2024-09-21T12:39:07.165996+0000","last_clean":"2024-09-21T12:39:07.165996+0000","last_became_active":"2024-09-21T12:32:38.421609+0000","last_became_peered":"2024-09-21T12:32:38.421609+0000","last_unstale":"2024-09-21T12:39:07.165996+0000","last_undegraded":"2024-09-21T12:39:07.165996+0000","last_fullsized":"2024-09-21T12:39:07.165996+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T12:37:52.992635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:37:52.992635+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":159,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.171506+0000","last_change":"2024-09-21T12:37:59.566316+0000","last_active":"2024-09-21T12:39:07.171506+0000","last_peered":"2024-09-21T12:39:07.171506+0000","last_clean":"2024-09-21T12:39:07.171506+0000","last_became_active":"2024-09-21T12:36:00.150832+0000","last_became_peered":"2024-09-21T12:36:00.150832+0000","last_unstale":"2024-09-21T12:39:07.171506+0000","last_undegraded":"2024-09-21T12:39:07.171506+0000","last_fullsized":"2024-09-21T12:39:07.171506+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:59.566268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:37:59.566268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:26.751431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":232,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181497+0000","last_change":"2024-09-21T12:38:46.602449+0000","last_active":"2024-09-21T12:39:07.181497+0000","last_peered":"2024-09-21T12:39:07.181497+0000","last_clean":"2024-09-21T12:39:07.181497+0000","last_became_active":"2024-09-21T12:38:07.161907+0000","last_became_peered":"2024-09-21T12:38:07.161907+0000","last_unstale":"2024-09-21T12:39:07.181497+0000","last_undegraded":"2024-09-21T12:39:07.181497+0000","last_fullsized":"2024-09-21T12:39:07.181497+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:46.597509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:46.597509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:50.838226+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":923,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.174789+0000","last_change":"2024-09-21T12:38:00.902377+0000","last_active":"2024-09-21T12:39:07.174789+0000","last_peered":"2024-09-21T12:39:07.174789+0000","last_clean":"2024-09-21T12:39:07.174789+0000","last_became_active":"2024-09-21T12:36:00.139300+0000","last_became_peered":"2024-09-21T12:36:00.139300+0000","last_unstale":"2024-09-21T12:39:07.174789+0000","last_undegraded":"2024-09-21T12:39:07.174789+0000","last_fullsized":"2024-09-21T12:39:07.174789+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:00.901368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:38:00.901368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:19.772138+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"484'17535","reported_seq":17804,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:11.911943+0000","last_change":"2024-09-21T12:37:20.472851+0000","last_active":"2024-09-21T12:39:11.911943+0000","last_peered":"2024-09-21T12:39:11.911943+0000","last_clean":"2024-09-21T12:39:11.911943+0000","last_became_active":"2024-09-21T12:37:20.468527+0000","last_became_peered":"2024-09-21T12:37:20.468527+0000","last_unstale":"2024-09-21T12:39:11.911943+0000","last_undegraded":"2024-09-21T12:39:11.911943+0000","last_fullsized":"2024-09-21T12:39:11.911943+0000","mapping_epoch":441,"log_start":"457'12900","ondisk_log_start":"457'12900","created":405,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"438'8315","last_scrub_stamp":"2024-09-21T12:37:16.765348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:37:16.765348+0000","objects_scrubbed":8225,"log_size":4635,"log_dups_size":0,"ondisk_log_size":4635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2844,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9895936,"num_objects":151,"num_object_clones":0,"num_object_copies":453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17535,"num_write_kb":565952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"76'79","reported_seq":331,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.171597+0000","last_change":"2024-09-21T12:36:36.859482+0000","last_active":"2024-09-21T12:39:07.171597+0000","last_peered":"2024-09-21T12:39:07.171597+0000","last_clean":"2024-09-21T12:39:07.171597+0000","last_became_active":"2024-09-21T12:36:01.141807+0000","last_became_peered":"2024-09-21T12:36:01.141807+0000","last_unstale":"2024-09-21T12:39:07.171597+0000","last_undegraded":"2024-09-21T12:39:07.171597+0000","last_fullsized":"2024-09-21T12:39:07.171597+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:36.859412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:36.859412+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:02.659187+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":440,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.180141+0000","last_change":"2024-09-21T12:37:56.066173+0000","last_active":"2024-09-21T12:39:07.180141+0000","last_peered":"2024-09-21T12:39:07.180141+0000","last_clean":"2024-09-21T12:39:07.180141+0000","last_became_active":"2024-09-21T12:36:00.137597+0000","last_became_peered":"2024-09-21T12:36:00.137597+0000","last_unstale":"2024-09-21T12:39:07.180141+0000","last_undegraded":"2024-09-21T12:39:07.180141+0000","last_fullsized":"2024-09-21T12:39:07.180141+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:37:56.066042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:37:56.066042+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:17.082865+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":165,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.171654+0000","last_change":"2024-09-21T12:37:11.677270+0000","last_active":"2024-09-21T12:39:07.171654+0000","last_peered":"2024-09-21T12:39:07.171654+0000","last_clean":"2024-09-21T12:39:07.171654+0000","last_became_active":"2024-09-21T12:36:00.148899+0000","last_became_peered":"2024-09-21T12:36:00.148899+0000","last_unstale":"2024-09-21T12:39:07.171654+0000","last_undegraded":"2024-09-21T12:39:07.171654+0000","last_fullsized":"2024-09-21T12:39:07.171654+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-21T12:37:11.677196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:37:11.677196+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"415'16","reported_seq":202,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.170795+0000","last_change":"2024-09-21T12:38:00.562505+0000","last_active":"2024-09-21T12:39:07.170795+0000","last_peered":"2024-09-21T12:39:07.170795+0000","last_clean":"2024-09-21T12:39:07.170795+0000","last_became_active":"2024-09-21T12:36:00.143091+0000","last_became_peered":"2024-09-21T12:36:00.143091+0000","last_unstale":"2024-09-21T12:39:07.170795+0000","last_undegraded":"2024-09-21T12:39:07.170795+0000","last_fullsized":"2024-09-21T12:39:07.170795+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'16","last_scrub_stamp":"2024-09-21T12:38:00.562427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:00.562427+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:16.512959+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":401,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.180513+0000","last_change":"2024-09-21T12:37:20.468878+0000","last_active":"2024-09-21T12:39:07.180513+0000","last_peered":"2024-09-21T12:39:07.180513+0000","last_clean":"2024-09-21T12:39:07.180513+0000","last_became_active":"2024-09-21T12:37:20.468350+0000","last_became_peered":"2024-09-21T12:37:20.468350+0000","last_unstale":"2024-09-21T12:39:07.180513+0000","last_undegraded":"2024-09-21T12:39:07.180513+0000","last_fullsized":"2024-09-21T12:39:07.180513+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:34.998795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:34.998795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:59.304496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"76'79","reported_seq":629,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.180190+0000","last_change":"2024-09-21T12:32:26.698667+0000","last_active":"2024-09-21T12:39:07.180190+0000","last_peered":"2024-09-21T12:39:07.180190+0000","last_clean":"2024-09-21T12:39:07.180190+0000","last_became_active":"2024-09-21T12:31:39.060422+0000","last_became_peered":"2024-09-21T12:31:39.060422+0000","last_unstale":"2024-09-21T12:39:07.180190+0000","last_undegraded":"2024-09-21T12:39:07.180190+0000","last_fullsized":"2024-09-21T12:39:07.180190+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:26.698622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:26.698622+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:11.139311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":954,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.173074+0000","last_change":"2024-09-21T12:38:05.816377+0000","last_active":"2024-09-21T12:39:07.173074+0000","last_peered":"2024-09-21T12:39:07.173074+0000","last_clean":"2024-09-21T12:39:07.173074+0000","last_became_active":"2024-09-21T12:32:02.026693+0000","last_became_peered":"2024-09-21T12:32:02.026693+0000","last_unstale":"2024-09-21T12:39:07.173074+0000","last_undegraded":"2024-09-21T12:39:07.173074+0000","last_fullsized":"2024-09-21T12:39:07.173074+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:05.816312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:05.816312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:17.640983+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":410,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.173965+0000","last_change":"2024-09-21T12:37:59.919992+0000","last_active":"2024-09-21T12:39:07.173965+0000","last_peered":"2024-09-21T12:39:07.173965+0000","last_clean":"2024-09-21T12:39:07.173965+0000","last_became_active":"2024-09-21T12:36:01.138626+0000","last_became_peered":"2024-09-21T12:36:01.138626+0000","last_unstale":"2024-09-21T12:39:07.173965+0000","last_undegraded":"2024-09-21T12:39:07.173965+0000","last_fullsized":"2024-09-21T12:39:07.173965+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T12:37:59.919947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:37:59.919947+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":421,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.180976+0000","last_change":"2024-09-21T12:37:20.486113+0000","last_active":"2024-09-21T12:39:07.180976+0000","last_peered":"2024-09-21T12:39:07.180976+0000","last_clean":"2024-09-21T12:39:07.180976+0000","last_became_active":"2024-09-21T12:37:20.485806+0000","last_became_peered":"2024-09-21T12:37:20.485806+0000","last_unstale":"2024-09-21T12:39:07.180976+0000","last_undegraded":"2024-09-21T12:39:07.180976+0000","last_fullsized":"2024-09-21T12:39:07.180976+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:42.510764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:42.510764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:57.271996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":159,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.171332+0000","last_change":"2024-09-21T12:38:03.579433+0000","last_active":"2024-09-21T12:39:07.171332+0000","last_peered":"2024-09-21T12:39:07.171332+0000","last_clean":"2024-09-21T12:39:07.171332+0000","last_became_active":"2024-09-21T12:36:00.149086+0000","last_became_peered":"2024-09-21T12:36:00.149086+0000","last_unstale":"2024-09-21T12:39:07.171332+0000","last_undegraded":"2024-09-21T12:39:07.171332+0000","last_fullsized":"2024-09-21T12:39:07.171332+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:03.579275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:38:03.579275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:08.467030+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'79","reported_seq":885,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.177777+0000","last_change":"2024-09-21T12:37:53.523886+0000","last_active":"2024-09-21T12:39:07.177777+0000","last_peered":"2024-09-21T12:39:07.177777+0000","last_clean":"2024-09-21T12:39:07.177777+0000","last_became_active":"2024-09-21T12:32:38.425495+0000","last_became_peered":"2024-09-21T12:32:38.425495+0000","last_unstale":"2024-09-21T12:39:07.177777+0000","last_undegraded":"2024-09-21T12:39:07.177777+0000","last_fullsized":"2024-09-21T12:39:07.177777+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:37:53.523818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:37:53.523818+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:07.695443+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":977,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.183654+0000","last_change":"2024-09-21T12:38:04.577293+0000","last_active":"2024-09-21T12:39:07.183654+0000","last_peered":"2024-09-21T12:39:07.183654+0000","last_clean":"2024-09-21T12:39:07.183654+0000","last_became_active":"2024-09-21T12:36:00.142274+0000","last_became_peered":"2024-09-21T12:36:00.142274+0000","last_unstale":"2024-09-21T12:39:07.183654+0000","last_undegraded":"2024-09-21T12:39:07.183654+0000","last_fullsized":"2024-09-21T12:39:07.183654+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:38:04.577219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:04.577219+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:22.773773+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":159,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.170851+0000","last_change":"2024-09-21T12:38:06.554109+0000","last_active":"2024-09-21T12:39:07.170851+0000","last_peered":"2024-09-21T12:39:07.170851+0000","last_clean":"2024-09-21T12:39:07.170851+0000","last_became_active":"2024-09-21T12:36:00.140477+0000","last_became_peered":"2024-09-21T12:36:00.140477+0000","last_unstale":"2024-09-21T12:39:07.170851+0000","last_undegraded":"2024-09-21T12:39:07.170851+0000","last_fullsized":"2024-09-21T12:39:07.170851+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:06.554062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:06.554062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:08.951635+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":904,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.173752+0000","last_change":"2024-09-21T12:38:50.213195+0000","last_active":"2024-09-21T12:39:07.173752+0000","last_peered":"2024-09-21T12:39:07.173752+0000","last_clean":"2024-09-21T12:39:07.173752+0000","last_became_active":"2024-09-21T12:36:00.150975+0000","last_became_peered":"2024-09-21T12:36:00.150975+0000","last_unstale":"2024-09-21T12:39:07.173752+0000","last_undegraded":"2024-09-21T12:39:07.173752+0000","last_fullsized":"2024-09-21T12:39:07.173752+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:50.213138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:50.213138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:09.450880+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":887,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.166659+0000","last_change":"2024-09-21T12:38:08.150215+0000","last_active":"2024-09-21T12:39:07.166659+0000","last_peered":"2024-09-21T12:39:07.166659+0000","last_clean":"2024-09-21T12:39:07.166659+0000","last_became_active":"2024-09-21T12:38:08.149697+0000","last_became_peered":"2024-09-21T12:38:08.149697+0000","last_unstale":"2024-09-21T12:39:07.166659+0000","last_undegraded":"2024-09-21T12:39:07.166659+0000","last_fullsized":"2024-09-21T12:39:07.166659+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:57.998679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:37:57.998679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:18.259205+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.10","version":"76'79","reported_seq":126,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.179911+0000","last_change":"2024-09-21T12:37:57.062032+0000","last_active":"2024-09-21T12:39:07.179911+0000","last_peered":"2024-09-21T12:39:07.179911+0000","last_clean":"2024-09-21T12:39:07.179911+0000","last_became_active":"2024-09-21T12:37:20.467540+0000","last_became_peered":"2024-09-21T12:37:20.467540+0000","last_unstale":"2024-09-21T12:39:07.179911+0000","last_undegraded":"2024-09-21T12:39:07.179911+0000","last_fullsized":"2024-09-21T12:39:07.179911+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:37:57.061984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:37:57.061984+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":462,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.180320+0000","last_change":"2024-09-21T12:38:39.157782+0000","last_active":"2024-09-21T12:39:07.180320+0000","last_peered":"2024-09-21T12:39:07.180320+0000","last_clean":"2024-09-21T12:39:07.180320+0000","last_became_active":"2024-09-21T12:38:08.164375+0000","last_became_peered":"2024-09-21T12:38:08.164375+0000","last_unstale":"2024-09-21T12:39:07.180320+0000","last_undegraded":"2024-09-21T12:39:07.180320+0000","last_fullsized":"2024-09-21T12:39:07.180320+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:38:39.157727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:39.157727+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:00.959002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":953,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181500+0000","last_change":"2024-09-21T12:36:00.127553+0000","last_active":"2024-09-21T12:39:07.181500+0000","last_peered":"2024-09-21T12:39:07.181500+0000","last_clean":"2024-09-21T12:39:07.181500+0000","last_became_active":"2024-09-21T12:36:00.127225+0000","last_became_peered":"2024-09-21T12:36:00.127225+0000","last_unstale":"2024-09-21T12:39:07.181500+0000","last_undegraded":"2024-09-21T12:39:07.181500+0000","last_fullsized":"2024-09-21T12:39:07.181500+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:22.225122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:22.225122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:06.616447+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"457'17","reported_seq":490,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.175768+0000","last_change":"2024-09-21T12:38:08.172081+0000","last_active":"2024-09-21T12:39:07.175768+0000","last_peered":"2024-09-21T12:39:07.175768+0000","last_clean":"2024-09-21T12:39:07.175768+0000","last_became_active":"2024-09-21T12:38:08.171750+0000","last_became_peered":"2024-09-21T12:38:08.171750+0000","last_unstale":"2024-09-21T12:39:07.175768+0000","last_undegraded":"2024-09-21T12:39:07.175768+0000","last_fullsized":"2024-09-21T12:39:07.175768+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'15","last_scrub_stamp":"2024-09-21T12:37:12.744729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:37:12.744729+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":301,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.166730+0000","last_change":"2024-09-21T12:38:07.163535+0000","last_active":"2024-09-21T12:39:07.166730+0000","last_peered":"2024-09-21T12:39:07.166730+0000","last_clean":"2024-09-21T12:39:07.166730+0000","last_became_active":"2024-09-21T12:38:07.163162+0000","last_became_peered":"2024-09-21T12:38:07.163162+0000","last_unstale":"2024-09-21T12:39:07.166730+0000","last_undegraded":"2024-09-21T12:39:07.166730+0000","last_fullsized":"2024-09-21T12:39:07.166730+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:05.370263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:05.370263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:58.482168+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"76'79","reported_seq":728,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181827+0000","last_change":"2024-09-21T12:38:00.556820+0000","last_active":"2024-09-21T12:39:07.181827+0000","last_peered":"2024-09-21T12:39:07.181827+0000","last_clean":"2024-09-21T12:39:07.181827+0000","last_became_active":"2024-09-21T12:32:02.034387+0000","last_became_peered":"2024-09-21T12:32:02.034387+0000","last_unstale":"2024-09-21T12:39:07.181827+0000","last_undegraded":"2024-09-21T12:39:07.181827+0000","last_fullsized":"2024-09-21T12:39:07.181827+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:38:00.556755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:38:00.556755+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:12.610054+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":984,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.176306+0000","last_change":"2024-09-21T12:36:36.521884+0000","last_active":"2024-09-21T12:39:07.176306+0000","last_peered":"2024-09-21T12:39:07.176306+0000","last_clean":"2024-09-21T12:39:07.176306+0000","last_became_active":"2024-09-21T12:36:00.155033+0000","last_became_peered":"2024-09-21T12:36:00.155033+0000","last_unstale":"2024-09-21T12:39:07.176306+0000","last_undegraded":"2024-09-21T12:39:07.176306+0000","last_fullsized":"2024-09-21T12:39:07.176306+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:36:36.521810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:36.521810+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":931,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.171268+0000","last_change":"2024-09-21T12:38:08.165195+0000","last_active":"2024-09-21T12:39:07.171268+0000","last_peered":"2024-09-21T12:39:07.171268+0000","last_clean":"2024-09-21T12:39:07.171268+0000","last_became_active":"2024-09-21T12:38:08.164312+0000","last_became_peered":"2024-09-21T12:38:08.164312+0000","last_unstale":"2024-09-21T12:39:07.171268+0000","last_undegraded":"2024-09-21T12:39:07.171268+0000","last_fullsized":"2024-09-21T12:39:07.171268+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:18.659683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:37:18.659683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":1143,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.180926+0000","last_change":"2024-09-21T12:38:45.275249+0000","last_active":"2024-09-21T12:39:07.180926+0000","last_peered":"2024-09-21T12:39:07.180926+0000","last_clean":"2024-09-21T12:39:07.180926+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:39:07.180926+0000","last_undegraded":"2024-09-21T12:39:07.180926+0000","last_fullsized":"2024-09-21T12:39:07.180926+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:38:45.275110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:45.275110+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:10.836660+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":917,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181188+0000","last_change":"2024-09-21T12:37:59.582535+0000","last_active":"2024-09-21T12:39:07.181188+0000","last_peered":"2024-09-21T12:39:07.181188+0000","last_clean":"2024-09-21T12:39:07.181188+0000","last_became_active":"2024-09-21T12:36:00.148332+0000","last_became_peered":"2024-09-21T12:36:00.148332+0000","last_unstale":"2024-09-21T12:39:07.181188+0000","last_undegraded":"2024-09-21T12:39:07.181188+0000","last_fullsized":"2024-09-21T12:39:07.181188+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:59.582483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:37:59.582483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"76'79","reported_seq":584,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.176523+0000","last_change":"2024-09-21T12:38:03.539003+0000","last_active":"2024-09-21T12:39:07.176523+0000","last_peered":"2024-09-21T12:39:07.176523+0000","last_clean":"2024-09-21T12:39:07.176523+0000","last_became_active":"2024-09-21T12:27:16.940347+0000","last_became_peered":"2024-09-21T12:27:16.940347+0000","last_unstale":"2024-09-21T12:39:07.176523+0000","last_undegraded":"2024-09-21T12:39:07.176523+0000","last_fullsized":"2024-09-21T12:39:07.176523+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:38:03.538881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:38:03.538881+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:30.585103+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":302,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.166487+0000","last_change":"2024-09-21T12:34:00.191190+0000","last_active":"2024-09-21T12:39:07.166487+0000","last_peered":"2024-09-21T12:39:07.166487+0000","last_clean":"2024-09-21T12:39:07.166487+0000","last_became_active":"2024-09-21T12:32:38.421008+0000","last_became_peered":"2024-09-21T12:32:38.421008+0000","last_unstale":"2024-09-21T12:39:07.166487+0000","last_undegraded":"2024-09-21T12:39:07.166487+0000","last_fullsized":"2024-09-21T12:39:07.166487+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:34:00.191106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:34:00.191106+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:01.433046+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":310,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.166581+0000","last_change":"2024-09-21T12:37:49.940962+0000","last_active":"2024-09-21T12:39:07.166581+0000","last_peered":"2024-09-21T12:39:07.166581+0000","last_clean":"2024-09-21T12:39:07.166581+0000","last_became_active":"2024-09-21T12:36:01.123399+0000","last_became_peered":"2024-09-21T12:36:01.123399+0000","last_unstale":"2024-09-21T12:39:07.166581+0000","last_undegraded":"2024-09-21T12:39:07.166581+0000","last_fullsized":"2024-09-21T12:39:07.166581+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:49.940903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:37:49.940903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:18.658566+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":917,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.168526+0000","last_change":"2024-09-21T12:38:00.521861+0000","last_active":"2024-09-21T12:39:07.168526+0000","last_peered":"2024-09-21T12:39:07.168526+0000","last_clean":"2024-09-21T12:39:07.168526+0000","last_became_active":"2024-09-21T12:31:50.040754+0000","last_became_peered":"2024-09-21T12:31:50.040754+0000","last_unstale":"2024-09-21T12:39:07.168526+0000","last_undegraded":"2024-09-21T12:39:07.168526+0000","last_fullsized":"2024-09-21T12:39:07.168526+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:00.521803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:00.521803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:12.854669+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":367,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.174400+0000","last_change":"2024-09-21T12:37:19.857159+0000","last_active":"2024-09-21T12:39:07.174400+0000","last_peered":"2024-09-21T12:39:07.174400+0000","last_clean":"2024-09-21T12:39:07.174400+0000","last_became_active":"2024-09-21T12:31:50.036150+0000","last_became_peered":"2024-09-21T12:31:50.036150+0000","last_unstale":"2024-09-21T12:39:07.174400+0000","last_undegraded":"2024-09-21T12:39:07.174400+0000","last_fullsized":"2024-09-21T12:39:07.174400+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:19.857108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:37:19.857108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"76'79","reported_seq":609,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181300+0000","last_change":"2024-09-21T12:36:45.476276+0000","last_active":"2024-09-21T12:39:07.181300+0000","last_peered":"2024-09-21T12:39:07.181300+0000","last_clean":"2024-09-21T12:39:07.181300+0000","last_became_active":"2024-09-21T12:27:16.921880+0000","last_became_peered":"2024-09-21T12:27:16.921880+0000","last_unstale":"2024-09-21T12:39:07.181300+0000","last_undegraded":"2024-09-21T12:39:07.181300+0000","last_fullsized":"2024-09-21T12:39:07.181300+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:45.476087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:45.476087+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:05.179564+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":404,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.168206+0000","last_change":"2024-09-21T12:38:47.987378+0000","last_active":"2024-09-21T12:39:07.168206+0000","last_peered":"2024-09-21T12:39:07.168206+0000","last_clean":"2024-09-21T12:39:07.168206+0000","last_became_active":"2024-09-21T12:38:07.153467+0000","last_became_peered":"2024-09-21T12:38:07.153467+0000","last_unstale":"2024-09-21T12:39:07.168206+0000","last_undegraded":"2024-09-21T12:39:07.168206+0000","last_fullsized":"2024-09-21T12:39:07.168206+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:38:47.987308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:47.987308+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:49.248198+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":159,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.171562+0000","last_change":"2024-09-21T12:38:05.529592+0000","last_active":"2024-09-21T12:39:07.171562+0000","last_peered":"2024-09-21T12:39:07.171562+0000","last_clean":"2024-09-21T12:39:07.171562+0000","last_became_active":"2024-09-21T12:36:00.147063+0000","last_became_peered":"2024-09-21T12:36:00.147063+0000","last_unstale":"2024-09-21T12:39:07.171562+0000","last_undegraded":"2024-09-21T12:39:07.171562+0000","last_fullsized":"2024-09-21T12:39:07.171562+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:05.529479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:05.529479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:29.024607+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":899,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.174865+0000","last_change":"2024-09-21T12:37:50.035009+0000","last_active":"2024-09-21T12:39:07.174865+0000","last_peered":"2024-09-21T12:39:07.174865+0000","last_clean":"2024-09-21T12:39:07.174865+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:39:07.174865+0000","last_undegraded":"2024-09-21T12:39:07.174865+0000","last_fullsized":"2024-09-21T12:39:07.174865+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:50.034945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:37:50.034945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:12.785791+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":882,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181586+0000","last_change":"2024-09-21T12:35:59.959597+0000","last_active":"2024-09-21T12:39:07.181586+0000","last_peered":"2024-09-21T12:39:07.181586+0000","last_clean":"2024-09-21T12:39:07.181586+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:39:07.181586+0000","last_undegraded":"2024-09-21T12:39:07.181586+0000","last_fullsized":"2024-09-21T12:39:07.181586+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:59.959545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:59.959545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"76'79","reported_seq":392,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.171336+0000","last_change":"2024-09-21T12:38:08.157233+0000","last_active":"2024-09-21T12:39:07.171336+0000","last_peered":"2024-09-21T12:39:07.171336+0000","last_clean":"2024-09-21T12:39:07.171336+0000","last_became_active":"2024-09-21T12:38:08.156834+0000","last_became_peered":"2024-09-21T12:38:08.156834+0000","last_unstale":"2024-09-21T12:39:07.171336+0000","last_undegraded":"2024-09-21T12:39:07.171336+0000","last_fullsized":"2024-09-21T12:39:07.171336+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:43.792355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:43.792355+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:57.778271+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":972,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.174767+0000","last_change":"2024-09-21T12:36:41.506309+0000","last_active":"2024-09-21T12:39:07.174767+0000","last_peered":"2024-09-21T12:39:07.174767+0000","last_clean":"2024-09-21T12:39:07.174767+0000","last_became_active":"2024-09-21T12:36:00.139682+0000","last_became_peered":"2024-09-21T12:36:00.139682+0000","last_unstale":"2024-09-21T12:39:07.174767+0000","last_undegraded":"2024-09-21T12:39:07.174767+0000","last_fullsized":"2024-09-21T12:39:07.174767+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:36:41.506231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:41.506231+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":937,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181063+0000","last_change":"2024-09-21T12:36:34.526618+0000","last_active":"2024-09-21T12:39:07.181063+0000","last_peered":"2024-09-21T12:39:07.181063+0000","last_clean":"2024-09-21T12:39:07.181063+0000","last_became_active":"2024-09-21T12:36:00.151584+0000","last_became_peered":"2024-09-21T12:36:00.151584+0000","last_unstale":"2024-09-21T12:39:07.181063+0000","last_undegraded":"2024-09-21T12:39:07.181063+0000","last_fullsized":"2024-09-21T12:39:07.181063+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:34.526551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:36:34.526551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:56.283290+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":159,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.170621+0000","last_change":"2024-09-21T12:37:51.569522+0000","last_active":"2024-09-21T12:39:07.170621+0000","last_peered":"2024-09-21T12:39:07.170621+0000","last_clean":"2024-09-21T12:39:07.170621+0000","last_became_active":"2024-09-21T12:36:00.144521+0000","last_became_peered":"2024-09-21T12:36:00.144521+0000","last_unstale":"2024-09-21T12:39:07.170621+0000","last_undegraded":"2024-09-21T12:39:07.170621+0000","last_fullsized":"2024-09-21T12:39:07.170621+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:51.569462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:37:51.569462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":887,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.175053+0000","last_change":"2024-09-21T12:35:51.628370+0000","last_active":"2024-09-21T12:39:07.175053+0000","last_peered":"2024-09-21T12:39:07.175053+0000","last_clean":"2024-09-21T12:39:07.175053+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:39:07.175053+0000","last_undegraded":"2024-09-21T12:39:07.175053+0000","last_fullsized":"2024-09-21T12:39:07.175053+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:51.628326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:51.628326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"76'79","reported_seq":159,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.171303+0000","last_change":"2024-09-21T12:38:54.700596+0000","last_active":"2024-09-21T12:39:07.171303+0000","last_peered":"2024-09-21T12:39:07.171303+0000","last_clean":"2024-09-21T12:39:07.171303+0000","last_became_active":"2024-09-21T12:36:00.151043+0000","last_became_peered":"2024-09-21T12:36:00.151043+0000","last_unstale":"2024-09-21T12:39:07.171303+0000","last_undegraded":"2024-09-21T12:39:07.171303+0000","last_fullsized":"2024-09-21T12:39:07.171303+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:38:54.700420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:38:54.700420+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:56.059368+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":907,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.182935+0000","last_change":"2024-09-21T12:35:14.789148+0000","last_active":"2024-09-21T12:39:07.182935+0000","last_peered":"2024-09-21T12:39:07.182935+0000","last_clean":"2024-09-21T12:39:07.182935+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:39:07.182935+0000","last_undegraded":"2024-09-21T12:39:07.182935+0000","last_fullsized":"2024-09-21T12:39:07.182935+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'8","last_scrub_stamp":"2024-09-21T12:35:14.789070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:14.789070+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":915,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.182239+0000","last_change":"2024-09-21T12:35:27.237430+0000","last_active":"2024-09-21T12:39:07.182239+0000","last_peered":"2024-09-21T12:39:07.182239+0000","last_clean":"2024-09-21T12:39:07.182239+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:39:07.182239+0000","last_undegraded":"2024-09-21T12:39:07.182239+0000","last_fullsized":"2024-09-21T12:39:07.182239+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:35:27.236980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:27.236980+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":934,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.173409+0000","last_change":"2024-09-21T12:36:00.135914+0000","last_active":"2024-09-21T12:39:07.173409+0000","last_peered":"2024-09-21T12:39:07.173409+0000","last_clean":"2024-09-21T12:39:07.173409+0000","last_became_active":"2024-09-21T12:36:00.135537+0000","last_became_peered":"2024-09-21T12:36:00.135537+0000","last_unstale":"2024-09-21T12:39:07.173409+0000","last_undegraded":"2024-09-21T12:39:07.173409+0000","last_fullsized":"2024-09-21T12:39:07.173409+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:40.861493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:40.861493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:34.475132+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":919,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.171069+0000","last_change":"2024-09-21T12:38:08.164865+0000","last_active":"2024-09-21T12:39:07.171069+0000","last_peered":"2024-09-21T12:39:07.171069+0000","last_clean":"2024-09-21T12:39:07.171069+0000","last_became_active":"2024-09-21T12:38:08.164535+0000","last_became_peered":"2024-09-21T12:38:08.164535+0000","last_unstale":"2024-09-21T12:39:07.171069+0000","last_undegraded":"2024-09-21T12:39:07.171069+0000","last_fullsized":"2024-09-21T12:39:07.171069+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:38.838688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:38.838688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:04.705707+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"76'79","reported_seq":304,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.166070+0000","last_change":"2024-09-21T12:36:00.137642+0000","last_active":"2024-09-21T12:39:07.166070+0000","last_peered":"2024-09-21T12:39:07.166070+0000","last_clean":"2024-09-21T12:39:07.166070+0000","last_became_active":"2024-09-21T12:36:00.137290+0000","last_became_peered":"2024-09-21T12:36:00.137290+0000","last_unstale":"2024-09-21T12:39:07.166070+0000","last_undegraded":"2024-09-21T12:39:07.166070+0000","last_fullsized":"2024-09-21T12:39:07.166070+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:43.304606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:43.304606+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:05.436780+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":923,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.169347+0000","last_change":"2024-09-21T12:38:51.954801+0000","last_active":"2024-09-21T12:39:07.169347+0000","last_peered":"2024-09-21T12:39:07.169347+0000","last_clean":"2024-09-21T12:39:07.169347+0000","last_became_active":"2024-09-21T12:32:38.426822+0000","last_became_peered":"2024-09-21T12:32:38.426822+0000","last_unstale":"2024-09-21T12:39:07.169347+0000","last_undegraded":"2024-09-21T12:39:07.169347+0000","last_fullsized":"2024-09-21T12:39:07.169347+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:51.954739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:38:51.954739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:53.948595+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":912,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.183174+0000","last_change":"2024-09-21T12:35:30.730640+0000","last_active":"2024-09-21T12:39:07.183174+0000","last_peered":"2024-09-21T12:39:07.183174+0000","last_clean":"2024-09-21T12:39:07.183174+0000","last_became_active":"2024-09-21T12:31:39.052486+0000","last_became_peered":"2024-09-21T12:31:39.052486+0000","last_unstale":"2024-09-21T12:39:07.183174+0000","last_undegraded":"2024-09-21T12:39:07.183174+0000","last_fullsized":"2024-09-21T12:39:07.183174+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:30.730574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:30.730574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":936,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.173453+0000","last_change":"2024-09-21T12:38:41.504397+0000","last_active":"2024-09-21T12:39:07.173453+0000","last_peered":"2024-09-21T12:39:07.173453+0000","last_clean":"2024-09-21T12:39:07.173453+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:39:07.173453+0000","last_undegraded":"2024-09-21T12:39:07.173453+0000","last_fullsized":"2024-09-21T12:39:07.173453+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:38:41.504320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:41.504320+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:48.172302+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":941,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181268+0000","last_change":"2024-09-21T12:38:08.175436+0000","last_active":"2024-09-21T12:39:07.181268+0000","last_peered":"2024-09-21T12:39:07.181268+0000","last_clean":"2024-09-21T12:39:07.181268+0000","last_became_active":"2024-09-21T12:38:08.174864+0000","last_became_peered":"2024-09-21T12:38:08.174864+0000","last_unstale":"2024-09-21T12:39:07.181268+0000","last_undegraded":"2024-09-21T12:39:07.181268+0000","last_fullsized":"2024-09-21T12:39:07.181268+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:28.911328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:28.911328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":294191128,"num_objects":4490,"num_object_clones":0,"num_object_copies":13470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":271748,"num_write_kb":8839553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22413,"num_bytes_recovered":1468530712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":830681088,"data_stored":830668872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68425,"ondisk_log_size":68425,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1105,"num_read_kb":862,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":472,"num_bytes_recovered":9372,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3597,"internal_metadata":0},"log_size":688,"ondisk_log_size":688,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":598,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":198,"num_read_kb":193,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2449,"ondisk_log_size":2449,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234215,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":907572,"kb_used_data":68388,"kb_used_omap":1,"kb_used_meta":839166,"kb_avail":92837580,"statfs":{"total":95995035648,"available":95065681920,"internally_reserved":0,"allocated":70029312,"data_stored":69233727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":859306442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463151,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":971300,"kb_used_data":91860,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":92773852,"statfs":{"total":95995035648,"available":95000424448,"internally_reserved":0,"allocated":94064640,"data_stored":93284270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":900528582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724786,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":835704,"kb_used_data":60328,"kb_used_omap":1,"kb_used_meta":775294,"kb_avail":92909448,"statfs":{"total":95995035648,"available":95139274752,"internally_reserved":0,"allocated":61775872,"data_stored":60977127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":793901509},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953722,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":469756,"kb_used_data":51244,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":93275396,"statfs":{"total":95995035648,"available":95514005504,"internally_reserved":0,"allocated":52473856,"data_stored":51669079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149951,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1135624,"kb_used_data":242292,"kb_used_omap":1,"kb_used_meta":893310,"kb_avail":92609528,"statfs":{"total":95995035648,"available":94832156672,"internally_reserved":0,"allocated":248107008,"data_stored":247295620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":914749895},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":455,"seq":1954210119694,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":359400,"kb_used_data":134936,"kb_used_omap":7,"kb_used_meta":224184,"kb_avail":93385752,"statfs":{"total":95995035648,"available":95627010048,"internally_reserved":0,"allocated":138174464,"data_stored":137390474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8042,"internal_metadata":229564566},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575116,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1082388,"kb_used_data":112708,"kb_used_omap":1,"kb_used_meta":969662,"kb_avail":92662764,"statfs":{"total":95995035648,"available":94886670336,"internally_reserved":0,"allocated":115412992,"data_stored":114584478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":992934341},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738641,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1175680,"kb_used_data":260140,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":92569472,"statfs":{"total":95995035648,"available":94791139328,"internally_reserved":0,"allocated":266383360,"data_stored":265579754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":937490885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3597,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":235339776,"data_stored":235339776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":76742656,"data_stored":76742656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":108662784,"data_stored":108658712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":215875584,"data_stored":215875584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":35262464,"data_stored":35258392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":39518208,"data_stored":39518208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":70324224,"data_stored":70320152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":48955392,"data_stored":48955392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:39:15.592 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:39:15.614 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:15 smithi114 ceph-mon[40867]: from='client.25975 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:39:15.614 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:15 smithi114 ceph-mon[40867]: pgmap v972: 178 pgs: 178 active+clean; 95 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-09-21T12:39:15.614 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:15 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/861299946' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:39:15.614 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:15 smithi114 ceph-mon[30324]: from='client.25975 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:39:15.615 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:15 smithi114 ceph-mon[30324]: pgmap v972: 178 pgs: 178 active+clean; 95 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-09-21T12:39:15.615 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:15 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/861299946' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:39:15.632 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:39:15.657 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:15.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:15.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:39:15.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:15 smithi183 ceph-mon[33277]: from='client.25975 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:39:15.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:15 smithi183 ceph-mon[33277]: pgmap v972: 178 pgs: 178 active+clean; 95 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-09-21T12:39:15.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:15 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/861299946' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:39:15.746 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-21T12:39:15.759 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:39:15.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:15.783 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:15.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:39:15.884 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:39:15.906 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:15.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:15.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:39:16.008 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:39:16.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:16.035 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:16.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:39:16.072 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:39:16.073 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:39:16.073 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:39:16.073 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:39:16.136 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:39:16.162 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:16.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:16.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:39:16.263 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:39:16.288 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:16.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:16.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:39:16.390 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:39:16.411 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:16.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:16.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:39:16.502 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:16 smithi114 ceph-mon[40867]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:39:16.502 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:16 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/861299946' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:39:16.502 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:16 smithi114 ceph-mon[40867]: osdmap e485: 8 total, 8 up, 8 in 2024-09-21T12:39:16.502 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:16 smithi114 ceph-mon[30324]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:39:16.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:16 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/861299946' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:39:16.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:16 smithi114 ceph-mon[30324]: osdmap e485: 8 total, 8 up, 8 in 2024-09-21T12:39:16.513 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:39:16.534 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:16.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:16.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:39:16.636 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:39:16.657 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:16.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:16.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:39:16.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:16 smithi183 ceph-mon[33277]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T12:39:16.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:16 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/861299946' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:39:16.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:16 smithi183 ceph-mon[33277]: osdmap e485: 8 total, 8 up, 8 in 2024-09-21T12:39:16.758 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:39:16.784 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:16.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:16.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:39:16.886 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:39:16.907 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:16.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:16.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:39:17.009 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:39:17.035 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:17.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:17.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:39:17.137 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:39:17.159 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:17.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:17.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:39:17.262 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:39:17.285 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:17.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:17.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:39:17.386 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:39:17.410 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:17.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:17.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:39:17.512 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:39:17.538 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:17.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:17.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:39:17.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:17 smithi114 ceph-mon[30324]: 1.12 scrub starts 2024-09-21T12:39:17.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:17 smithi114 ceph-mon[30324]: 1.12 scrub ok 2024-09-21T12:39:17.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:17 smithi114 ceph-mon[30324]: 4.18 scrub starts 2024-09-21T12:39:17.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:17 smithi114 ceph-mon[30324]: 4.18 scrub ok 2024-09-21T12:39:17.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:17 smithi114 ceph-mon[30324]: pgmap v974: 178 pgs: 178 active+clean; 95 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 559 B/s rd, 0 B/s wr, 1.06k op/s 2024-09-21T12:39:17.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:17 smithi114 ceph-mon[30324]: osdmap e486: 8 total, 8 up, 8 in 2024-09-21T12:39:17.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:17 smithi114 ceph-mon[40867]: 1.12 scrub starts 2024-09-21T12:39:17.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:17 smithi114 ceph-mon[40867]: 1.12 scrub ok 2024-09-21T12:39:17.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:17 smithi114 ceph-mon[40867]: 4.18 scrub starts 2024-09-21T12:39:17.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:17 smithi114 ceph-mon[40867]: 4.18 scrub ok 2024-09-21T12:39:17.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:17 smithi114 ceph-mon[40867]: pgmap v974: 178 pgs: 178 active+clean; 95 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 559 B/s rd, 0 B/s wr, 1.06k op/s 2024-09-21T12:39:17.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:17 smithi114 ceph-mon[40867]: osdmap e486: 8 total, 8 up, 8 in 2024-09-21T12:39:17.640 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:39:17.663 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:17.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:17.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:39:17.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:17 smithi183 ceph-mon[33277]: 1.12 scrub starts 2024-09-21T12:39:17.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:17 smithi183 ceph-mon[33277]: 1.12 scrub ok 2024-09-21T12:39:17.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:17 smithi183 ceph-mon[33277]: 4.18 scrub starts 2024-09-21T12:39:17.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:17 smithi183 ceph-mon[33277]: 4.18 scrub ok 2024-09-21T12:39:17.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:17 smithi183 ceph-mon[33277]: pgmap v974: 178 pgs: 178 active+clean; 95 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 559 B/s rd, 0 B/s wr, 1.06k op/s 2024-09-21T12:39:17.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:17 smithi183 ceph-mon[33277]: osdmap e486: 8 total, 8 up, 8 in 2024-09-21T12:39:17.764 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:39:17.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:17.787 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:17.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:39:17.888 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:39:17.914 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:17.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:17.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:39:18.010 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:39:18.013 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:39:18.016 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:39:18.041 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:18.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:18.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:39:18.144 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:39:18.168 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:18.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:18.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:39:18.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:39:17 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:39:18.269 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:39:18.292 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:18.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:18.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:39:18.393 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1 2024-09-21T12:39:18.414 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:18.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:18.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.1' 2024-09-21T12:39:18.516 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5 2024-09-21T12:39:18.540 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:18.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:18.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.5' 2024-09-21T12:39:18.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:18 smithi114 ceph-mon[40867]: 5.19 scrub starts 2024-09-21T12:39:18.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:18 smithi114 ceph-mon[40867]: 5.19 scrub ok 2024-09-21T12:39:18.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:18 smithi114 ceph-mon[40867]: 11.7 scrub starts 2024-09-21T12:39:18.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:18 smithi114 ceph-mon[40867]: 11.7 scrub ok 2024-09-21T12:39:18.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:18 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:39:18.567 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:18 smithi114 ceph-mon[30324]: 5.19 scrub starts 2024-09-21T12:39:18.567 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:18 smithi114 ceph-mon[30324]: 5.19 scrub ok 2024-09-21T12:39:18.567 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:18 smithi114 ceph-mon[30324]: 11.7 scrub starts 2024-09-21T12:39:18.567 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:18 smithi114 ceph-mon[30324]: 11.7 scrub ok 2024-09-21T12:39:18.567 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:18 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:39:18.641 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:39:18.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:18.665 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:18.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:39:18.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:18 smithi183 ceph-mon[33277]: 5.19 scrub starts 2024-09-21T12:39:18.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:18 smithi183 ceph-mon[33277]: 5.19 scrub ok 2024-09-21T12:39:18.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:18 smithi183 ceph-mon[33277]: 11.7 scrub starts 2024-09-21T12:39:18.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:18 smithi183 ceph-mon[33277]: 11.7 scrub ok 2024-09-21T12:39:18.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:18 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:39:18.766 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:39:18.789 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:18.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:18.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:39:18.891 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:39:18.914 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:18.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:18.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:39:19.016 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:39:19.040 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:19.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:19.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:39:19.141 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:39:19.165 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:19.165 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-21T12:39:19.165 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T12:39:19.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:19.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:39:19.267 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7 2024-09-21T12:39:19.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:19.290 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:19.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.7' 2024-09-21T12:39:19.383 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'unique_pool_4' removed 2024-09-21T12:39:19.391 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0 2024-09-21T12:39:19.417 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:19 smithi114 ceph-mon[40867]: 11.6 scrub starts 2024-09-21T12:39:19.417 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:19 smithi114 ceph-mon[40867]: 11.6 scrub ok 2024-09-21T12:39:19.417 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:19 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:39:19.417 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:19 smithi114 ceph-mon[40867]: from='client.16533 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:39:19.417 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:19 smithi114 ceph-mon[40867]: 6.1f scrub starts 2024-09-21T12:39:19.417 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:19 smithi114 ceph-mon[40867]: 6.1f scrub ok 2024-09-21T12:39:19.417 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:19 smithi114 ceph-mon[40867]: 3.1d scrub starts 2024-09-21T12:39:19.417 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:19 smithi114 ceph-mon[40867]: 3.1d scrub ok 2024-09-21T12:39:19.417 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:19 smithi114 ceph-mon[40867]: pgmap v976: 178 pgs: 178 active+clean; 1.7 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-09-21T12:39:19.418 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:19 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2654492222' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T12:39:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:19 smithi114 ceph-mon[30324]: 11.6 scrub starts 2024-09-21T12:39:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:19 smithi114 ceph-mon[30324]: 11.6 scrub ok 2024-09-21T12:39:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:19 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:39:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:19 smithi114 ceph-mon[30324]: from='client.16533 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:39:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:19 smithi114 ceph-mon[30324]: 6.1f scrub starts 2024-09-21T12:39:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:19 smithi114 ceph-mon[30324]: 6.1f scrub ok 2024-09-21T12:39:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:19 smithi114 ceph-mon[30324]: 3.1d scrub starts 2024-09-21T12:39:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:19 smithi114 ceph-mon[30324]: 3.1d scrub ok 2024-09-21T12:39:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:19 smithi114 ceph-mon[30324]: pgmap v976: 178 pgs: 178 active+clean; 1.7 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-09-21T12:39:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:19 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2654492222' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T12:39:19.425 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:19.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:19.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.0' 2024-09-21T12:39:19.527 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:39:19.548 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:19.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:19.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:39:19.649 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2 2024-09-21T12:39:19.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:19.671 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:19.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.2' 2024-09-21T12:39:19.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:19 smithi183 ceph-mon[33277]: 11.6 scrub starts 2024-09-21T12:39:19.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:19 smithi183 ceph-mon[33277]: 11.6 scrub ok 2024-09-21T12:39:19.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:19 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:39:19.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:19 smithi183 ceph-mon[33277]: from='client.16533 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:39:19.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:19 smithi183 ceph-mon[33277]: 6.1f scrub starts 2024-09-21T12:39:19.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:19 smithi183 ceph-mon[33277]: 6.1f scrub ok 2024-09-21T12:39:19.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:19 smithi183 ceph-mon[33277]: 3.1d scrub starts 2024-09-21T12:39:19.740 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:19 smithi183 ceph-mon[33277]: 3.1d scrub ok 2024-09-21T12:39:19.740 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:19 smithi183 ceph-mon[33277]: pgmap v976: 178 pgs: 178 active+clean; 1.7 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-09-21T12:39:19.740 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:19 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2654492222' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T12:39:19.772 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3 2024-09-21T12:39:19.794 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-21T12:39:19.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:19.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.3' 2024-09-21T12:39:19.895 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:39:19.918 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:19.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:19.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:39:20.019 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:39:20.042 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:20.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:20.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:39:20.143 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6 2024-09-21T12:39:20.167 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:20.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:20.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.6' 2024-09-21T12:39:20.269 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4 2024-09-21T12:39:20.291 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-21T12:39:20.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:39:20.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd.4' 2024-09-21T12:39:20.334 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-21T12:39:20.343 INFO:tasks.thrashosds:joining thrashosds 2024-09-21T12:39:20.654 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:20 smithi183 ceph-mon[33277]: 4.1b scrub starts 2024-09-21T12:39:20.655 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:20 smithi183 ceph-mon[33277]: 4.1b scrub ok 2024-09-21T12:39:20.655 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:20 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2654492222' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T12:39:20.655 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:20 smithi183 ceph-mon[33277]: osdmap e487: 8 total, 8 up, 8 in 2024-09-21T12:39:20.655 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:20 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/223982952' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:39:20.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:20 smithi114 ceph-mon[30324]: 4.1b scrub starts 2024-09-21T12:39:20.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:20 smithi114 ceph-mon[30324]: 4.1b scrub ok 2024-09-21T12:39:20.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:20 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2654492222' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T12:39:20.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:20 smithi114 ceph-mon[30324]: osdmap e487: 8 total, 8 up, 8 in 2024-09-21T12:39:20.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:20 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/223982952' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:39:20.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:20 smithi114 ceph-mon[40867]: 4.1b scrub starts 2024-09-21T12:39:20.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:20 smithi114 ceph-mon[40867]: 4.1b scrub ok 2024-09-21T12:39:20.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:20 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2654492222' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T12:39:20.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:20 smithi114 ceph-mon[40867]: osdmap e487: 8 total, 8 up, 8 in 2024-09-21T12:39:20.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:20 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/223982952' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T12:39:20.915 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":975,"stamp":"2024-09-21T12:39:17.163976+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20453915,"num_objects":502,"num_object_clones":0,"num_object_copies":1506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":502,"num_whiteouts":0,"num_read":1323,"num_read_kb":1075,"num_write":276657,"num_write_kb":8839598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22917,"num_bytes_recovered":1468803759,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71875,"ondisk_log_size":71875,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":535,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6148492,"kb_used_data":186040,"kb_used_omap":18,"kb_used_meta":5962221,"kb_avail":743812724,"statfs":{"total":767960285184,"available":761664229376,"internally_reserved":0,"allocated":190504960,"data_stored":184090031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19199,"internal_metadata":6105314561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-556007448,"num_objects":-8485,"num_object_clones":0,"num_object_copies":-25455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8485,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8485,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.859429"},"pg_stats":[{"pgid":"1.1d","version":"76'79","reported_seq":602,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370327+0000","last_change":"2024-09-21T12:37:16.858542+0000","last_active":"2024-09-21T12:39:15.370327+0000","last_peered":"2024-09-21T12:39:15.370327+0000","last_clean":"2024-09-21T12:39:15.370327+0000","last_became_active":"2024-09-21T12:32:38.426383+0000","last_became_peered":"2024-09-21T12:32:38.426383+0000","last_unstale":"2024-09-21T12:39:15.370327+0000","last_undegraded":"2024-09-21T12:39:15.370327+0000","last_fullsized":"2024-09-21T12:39:15.370327+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:37:16.858472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:37:16.858472+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:14.113783+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":914,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.182161+0000","last_change":"2024-09-21T12:38:07.168168+0000","last_active":"2024-09-21T12:39:07.182161+0000","last_peered":"2024-09-21T12:39:07.182161+0000","last_clean":"2024-09-21T12:39:07.182161+0000","last_became_active":"2024-09-21T12:38:07.167857+0000","last_became_peered":"2024-09-21T12:38:07.167857+0000","last_unstale":"2024-09-21T12:39:07.182161+0000","last_undegraded":"2024-09-21T12:39:07.182161+0000","last_fullsized":"2024-09-21T12:39:07.182161+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:29.086830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:29.086830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:02.550603+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":310,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.266371+0000","last_change":"2024-09-21T12:39:16.266371+0000","last_active":"2024-09-21T12:39:16.266371+0000","last_peered":"2024-09-21T12:39:16.266371+0000","last_clean":"2024-09-21T12:39:16.266371+0000","last_became_active":"2024-09-21T12:32:38.419081+0000","last_became_peered":"2024-09-21T12:32:38.419081+0000","last_unstale":"2024-09-21T12:39:16.266371+0000","last_undegraded":"2024-09-21T12:39:16.266371+0000","last_fullsized":"2024-09-21T12:39:16.266371+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:39:16.266285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:16.266285+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:30.189485+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":914,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370592+0000","last_change":"2024-09-21T12:37:21.767752+0000","last_active":"2024-09-21T12:39:15.370592+0000","last_peered":"2024-09-21T12:39:15.370592+0000","last_clean":"2024-09-21T12:39:15.370592+0000","last_became_active":"2024-09-21T12:36:00.136673+0000","last_became_peered":"2024-09-21T12:36:00.136673+0000","last_unstale":"2024-09-21T12:39:15.370592+0000","last_undegraded":"2024-09-21T12:39:15.370592+0000","last_fullsized":"2024-09-21T12:39:15.370592+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:21.767691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:37:21.767691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":300,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.221850+0000","last_change":"2024-09-21T12:36:44.729701+0000","last_active":"2024-09-21T12:39:16.221850+0000","last_peered":"2024-09-21T12:39:16.221850+0000","last_clean":"2024-09-21T12:39:16.221850+0000","last_became_active":"2024-09-21T12:32:38.425588+0000","last_became_peered":"2024-09-21T12:32:38.425588+0000","last_unstale":"2024-09-21T12:39:16.221850+0000","last_undegraded":"2024-09-21T12:39:16.221850+0000","last_fullsized":"2024-09-21T12:39:16.221850+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:44.729655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:44.729655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:13.322426+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"76'79","reported_seq":161,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370659+0000","last_change":"2024-09-21T12:37:19.644089+0000","last_active":"2024-09-21T12:39:15.370659+0000","last_peered":"2024-09-21T12:39:15.370659+0000","last_clean":"2024-09-21T12:39:15.370659+0000","last_became_active":"2024-09-21T12:36:00.154209+0000","last_became_peered":"2024-09-21T12:36:00.154209+0000","last_unstale":"2024-09-21T12:39:15.370659+0000","last_undegraded":"2024-09-21T12:39:15.370659+0000","last_fullsized":"2024-09-21T12:39:15.370659+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:37:19.644030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:37:19.644030+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":900,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.174610+0000","last_change":"2024-09-21T12:38:04.831151+0000","last_active":"2024-09-21T12:39:07.174610+0000","last_peered":"2024-09-21T12:39:07.174610+0000","last_clean":"2024-09-21T12:39:07.174610+0000","last_became_active":"2024-09-21T12:36:00.139167+0000","last_became_peered":"2024-09-21T12:36:00.139167+0000","last_unstale":"2024-09-21T12:39:07.174610+0000","last_undegraded":"2024-09-21T12:39:07.174610+0000","last_fullsized":"2024-09-21T12:39:07.174610+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:04.830956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:38:04.830956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:31.539751+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":911,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.173438+0000","last_change":"2024-09-21T12:38:07.159232+0000","last_active":"2024-09-21T12:39:07.173438+0000","last_peered":"2024-09-21T12:39:07.173438+0000","last_clean":"2024-09-21T12:39:07.173438+0000","last_became_active":"2024-09-21T12:38:07.157964+0000","last_became_peered":"2024-09-21T12:38:07.157964+0000","last_unstale":"2024-09-21T12:39:07.173438+0000","last_undegraded":"2024-09-21T12:39:07.173438+0000","last_fullsized":"2024-09-21T12:39:07.173438+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:06.698999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:06.698999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:59.290352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":956,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.174569+0000","last_change":"2024-09-21T12:38:41.050094+0000","last_active":"2024-09-21T12:39:07.174569+0000","last_peered":"2024-09-21T12:39:07.174569+0000","last_clean":"2024-09-21T12:39:07.174569+0000","last_became_active":"2024-09-21T12:38:07.168250+0000","last_became_peered":"2024-09-21T12:38:07.168250+0000","last_unstale":"2024-09-21T12:39:07.174569+0000","last_undegraded":"2024-09-21T12:39:07.174569+0000","last_fullsized":"2024-09-21T12:39:07.174569+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:38:41.050017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:41.050017+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:53.559093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":292,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.221133+0000","last_change":"2024-09-21T12:36:35.622295+0000","last_active":"2024-09-21T12:39:16.221133+0000","last_peered":"2024-09-21T12:39:16.221133+0000","last_clean":"2024-09-21T12:39:16.221133+0000","last_became_active":"2024-09-21T12:32:38.417724+0000","last_became_peered":"2024-09-21T12:32:38.417724+0000","last_unstale":"2024-09-21T12:39:16.221133+0000","last_undegraded":"2024-09-21T12:39:16.221133+0000","last_fullsized":"2024-09-21T12:39:16.221133+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:35.622232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:35.622232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:12.340388+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":452,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.917305+0000","last_change":"2024-09-21T12:37:25.297240+0000","last_active":"2024-09-21T12:39:15.917305+0000","last_peered":"2024-09-21T12:39:15.917305+0000","last_clean":"2024-09-21T12:39:15.917305+0000","last_became_active":"2024-09-21T12:37:20.467925+0000","last_became_peered":"2024-09-21T12:37:20.467925+0000","last_unstale":"2024-09-21T12:39:15.917305+0000","last_undegraded":"2024-09-21T12:39:15.917305+0000","last_fullsized":"2024-09-21T12:39:15.917305+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:25.297200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:37:25.297200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":318,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.221943+0000","last_change":"2024-09-21T12:38:03.431702+0000","last_active":"2024-09-21T12:39:16.221943+0000","last_peered":"2024-09-21T12:39:16.221943+0000","last_clean":"2024-09-21T12:39:16.221943+0000","last_became_active":"2024-09-21T12:32:38.426121+0000","last_became_peered":"2024-09-21T12:32:38.426121+0000","last_unstale":"2024-09-21T12:39:16.221943+0000","last_undegraded":"2024-09-21T12:39:16.221943+0000","last_fullsized":"2024-09-21T12:39:16.221943+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:38:03.431620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:03.431620+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:18.419545+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":953,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.916886+0000","last_change":"2024-09-21T12:38:44.245362+0000","last_active":"2024-09-21T12:39:15.916886+0000","last_peered":"2024-09-21T12:39:15.916886+0000","last_clean":"2024-09-21T12:39:15.916886+0000","last_became_active":"2024-09-21T12:36:00.137364+0000","last_became_peered":"2024-09-21T12:36:00.137364+0000","last_unstale":"2024-09-21T12:39:15.916886+0000","last_undegraded":"2024-09-21T12:39:15.916886+0000","last_fullsized":"2024-09-21T12:39:15.916886+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:44.245309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:44.245309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:02.710227+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":910,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181390+0000","last_change":"2024-09-21T12:36:38.752470+0000","last_active":"2024-09-21T12:39:07.181390+0000","last_peered":"2024-09-21T12:39:07.181390+0000","last_clean":"2024-09-21T12:39:07.181390+0000","last_became_active":"2024-09-21T12:36:00.134723+0000","last_became_peered":"2024-09-21T12:36:00.134723+0000","last_unstale":"2024-09-21T12:39:07.181390+0000","last_undegraded":"2024-09-21T12:39:07.181390+0000","last_fullsized":"2024-09-21T12:39:07.181390+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:38.752410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:38.752410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"76'79","reported_seq":685,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.173241+0000","last_change":"2024-09-21T12:37:56.010046+0000","last_active":"2024-09-21T12:39:07.173241+0000","last_peered":"2024-09-21T12:39:07.173241+0000","last_clean":"2024-09-21T12:39:07.173241+0000","last_became_active":"2024-09-21T12:36:00.147795+0000","last_became_peered":"2024-09-21T12:36:00.147795+0000","last_unstale":"2024-09-21T12:39:07.173241+0000","last_undegraded":"2024-09-21T12:39:07.173241+0000","last_fullsized":"2024-09-21T12:39:07.173241+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:37:56.009970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:37:56.009970+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:07.284213+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":444,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.917089+0000","last_change":"2024-09-21T12:38:52.309280+0000","last_active":"2024-09-21T12:39:15.917089+0000","last_peered":"2024-09-21T12:39:15.917089+0000","last_clean":"2024-09-21T12:39:15.917089+0000","last_became_active":"2024-09-21T12:37:20.468468+0000","last_became_peered":"2024-09-21T12:37:20.468468+0000","last_unstale":"2024-09-21T12:39:15.917089+0000","last_undegraded":"2024-09-21T12:39:15.917089+0000","last_fullsized":"2024-09-21T12:39:15.917089+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:52.309224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:52.309224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:56.584480+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":306,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.221660+0000","last_change":"2024-09-21T12:37:55.044998+0000","last_active":"2024-09-21T12:39:16.221660+0000","last_peered":"2024-09-21T12:39:16.221660+0000","last_clean":"2024-09-21T12:39:16.221660+0000","last_became_active":"2024-09-21T12:32:38.422912+0000","last_became_peered":"2024-09-21T12:32:38.422912+0000","last_unstale":"2024-09-21T12:39:16.221660+0000","last_undegraded":"2024-09-21T12:39:16.221660+0000","last_fullsized":"2024-09-21T12:39:16.221660+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'7","last_scrub_stamp":"2024-09-21T12:37:55.044927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:37:55.044927+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":161,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370079+0000","last_change":"2024-09-21T12:37:57.609800+0000","last_active":"2024-09-21T12:39:15.370079+0000","last_peered":"2024-09-21T12:39:15.370079+0000","last_clean":"2024-09-21T12:39:15.370079+0000","last_became_active":"2024-09-21T12:36:00.142106+0000","last_became_peered":"2024-09-21T12:36:00.142106+0000","last_unstale":"2024-09-21T12:39:15.370079+0000","last_undegraded":"2024-09-21T12:39:15.370079+0000","last_fullsized":"2024-09-21T12:39:15.370079+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:57.609734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:37:57.609734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":905,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.916603+0000","last_change":"2024-09-21T12:33:36.125346+0000","last_active":"2024-09-21T12:39:15.916603+0000","last_peered":"2024-09-21T12:39:15.916603+0000","last_clean":"2024-09-21T12:39:15.916603+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:39:15.916603+0000","last_undegraded":"2024-09-21T12:39:15.916603+0000","last_fullsized":"2024-09-21T12:39:15.916603+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:36.124095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:36.124095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:34.160912+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"76'79","reported_seq":725,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.175182+0000","last_change":"2024-09-21T12:34:48.959991+0000","last_active":"2024-09-21T12:39:07.175182+0000","last_peered":"2024-09-21T12:39:07.175182+0000","last_clean":"2024-09-21T12:39:07.175182+0000","last_became_active":"2024-09-21T12:27:16.957569+0000","last_became_peered":"2024-09-21T12:27:16.957569+0000","last_unstale":"2024-09-21T12:39:07.175182+0000","last_undegraded":"2024-09-21T12:39:07.175182+0000","last_fullsized":"2024-09-21T12:39:07.175182+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:48.959916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:48.959916+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":293,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.222705+0000","last_change":"2024-09-21T12:34:05.193021+0000","last_active":"2024-09-21T12:39:16.222705+0000","last_peered":"2024-09-21T12:39:16.222705+0000","last_clean":"2024-09-21T12:39:16.222705+0000","last_became_active":"2024-09-21T12:32:38.428997+0000","last_became_peered":"2024-09-21T12:32:38.428997+0000","last_unstale":"2024-09-21T12:39:16.222705+0000","last_undegraded":"2024-09-21T12:39:16.222705+0000","last_fullsized":"2024-09-21T12:39:16.222705+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:05.192971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:34:05.192971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:03.415980+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":973,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.182335+0000","last_change":"2024-09-21T12:37:23.610348+0000","last_active":"2024-09-21T12:39:07.182335+0000","last_peered":"2024-09-21T12:39:07.182335+0000","last_clean":"2024-09-21T12:39:07.182335+0000","last_became_active":"2024-09-21T12:36:00.127153+0000","last_became_peered":"2024-09-21T12:36:00.127153+0000","last_unstale":"2024-09-21T12:39:07.182335+0000","last_undegraded":"2024-09-21T12:39:07.182335+0000","last_fullsized":"2024-09-21T12:39:07.182335+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:23.610263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:37:23.610263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":956,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.916862+0000","last_change":"2024-09-21T12:38:07.162167+0000","last_active":"2024-09-21T12:39:15.916862+0000","last_peered":"2024-09-21T12:39:15.916862+0000","last_clean":"2024-09-21T12:39:15.916862+0000","last_became_active":"2024-09-21T12:38:07.161813+0000","last_became_peered":"2024-09-21T12:38:07.161813+0000","last_unstale":"2024-09-21T12:39:15.916862+0000","last_undegraded":"2024-09-21T12:39:15.916862+0000","last_fullsized":"2024-09-21T12:39:15.916862+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:33:01.668584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:01.668584+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:19.184023+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":936,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370641+0000","last_change":"2024-09-21T12:38:53.956579+0000","last_active":"2024-09-21T12:39:15.370641+0000","last_peered":"2024-09-21T12:39:15.370641+0000","last_clean":"2024-09-21T12:39:15.370641+0000","last_became_active":"2024-09-21T12:32:38.425958+0000","last_became_peered":"2024-09-21T12:32:38.425958+0000","last_unstale":"2024-09-21T12:39:15.370641+0000","last_undegraded":"2024-09-21T12:39:15.370641+0000","last_fullsized":"2024-09-21T12:39:15.370641+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:53.956515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:53.956515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:08.408871+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"76'79","reported_seq":862,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.916612+0000","last_change":"2024-09-21T12:38:08.157982+0000","last_active":"2024-09-21T12:39:15.916612+0000","last_peered":"2024-09-21T12:39:15.916612+0000","last_clean":"2024-09-21T12:39:15.916612+0000","last_became_active":"2024-09-21T12:38:08.157632+0000","last_became_peered":"2024-09-21T12:38:08.157632+0000","last_unstale":"2024-09-21T12:39:15.916612+0000","last_undegraded":"2024-09-21T12:39:15.916612+0000","last_fullsized":"2024-09-21T12:39:15.916612+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:27.294674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:27.294674+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:21.171485+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":909,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.173706+0000","last_change":"2024-09-21T12:35:06.944278+0000","last_active":"2024-09-21T12:39:07.173706+0000","last_peered":"2024-09-21T12:39:07.173706+0000","last_clean":"2024-09-21T12:39:07.173706+0000","last_became_active":"2024-09-21T12:32:02.033276+0000","last_became_peered":"2024-09-21T12:32:02.033276+0000","last_unstale":"2024-09-21T12:39:07.173706+0000","last_undegraded":"2024-09-21T12:39:07.173706+0000","last_fullsized":"2024-09-21T12:39:07.173706+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:06.944209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:06.944209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":303,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.221458+0000","last_change":"2024-09-21T12:38:41.208373+0000","last_active":"2024-09-21T12:39:16.221458+0000","last_peered":"2024-09-21T12:39:16.221458+0000","last_clean":"2024-09-21T12:39:16.221458+0000","last_became_active":"2024-09-21T12:38:07.159387+0000","last_became_peered":"2024-09-21T12:38:07.159387+0000","last_unstale":"2024-09-21T12:39:16.221458+0000","last_undegraded":"2024-09-21T12:39:16.221458+0000","last_fullsized":"2024-09-21T12:39:16.221458+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:41.208301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:41.208301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:51.906773+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":1002,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.175556+0000","last_change":"2024-09-21T12:35:59.641966+0000","last_active":"2024-09-21T12:39:07.175556+0000","last_peered":"2024-09-21T12:39:07.175556+0000","last_clean":"2024-09-21T12:39:07.175556+0000","last_became_active":"2024-09-21T12:32:38.409163+0000","last_became_peered":"2024-09-21T12:32:38.409163+0000","last_unstale":"2024-09-21T12:39:07.175556+0000","last_undegraded":"2024-09-21T12:39:07.175556+0000","last_fullsized":"2024-09-21T12:39:07.175556+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T12:35:59.641895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:59.641895+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":311,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.222481+0000","last_change":"2024-09-21T12:38:49.267225+0000","last_active":"2024-09-21T12:39:16.222481+0000","last_peered":"2024-09-21T12:39:16.222481+0000","last_clean":"2024-09-21T12:39:16.222481+0000","last_became_active":"2024-09-21T12:38:07.161634+0000","last_became_peered":"2024-09-21T12:38:07.161634+0000","last_unstale":"2024-09-21T12:39:16.222481+0000","last_undegraded":"2024-09-21T12:39:16.222481+0000","last_fullsized":"2024-09-21T12:39:16.222481+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:49.267162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:49.267162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:09.895158+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"76'79","reported_seq":651,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.916688+0000","last_change":"2024-09-21T12:36:41.538607+0000","last_active":"2024-09-21T12:39:15.916688+0000","last_peered":"2024-09-21T12:39:15.916688+0000","last_clean":"2024-09-21T12:39:15.916688+0000","last_became_active":"2024-09-21T12:32:38.414204+0000","last_became_peered":"2024-09-21T12:32:38.414204+0000","last_unstale":"2024-09-21T12:39:15.916688+0000","last_undegraded":"2024-09-21T12:39:15.916688+0000","last_fullsized":"2024-09-21T12:39:15.916688+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:41.538559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:41.538559+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:04.209095+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":885,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370588+0000","last_change":"2024-09-21T12:38:08.166042+0000","last_active":"2024-09-21T12:39:15.370588+0000","last_peered":"2024-09-21T12:39:15.370588+0000","last_clean":"2024-09-21T12:39:15.370588+0000","last_became_active":"2024-09-21T12:38:08.165695+0000","last_became_peered":"2024-09-21T12:38:08.165695+0000","last_unstale":"2024-09-21T12:39:15.370588+0000","last_undegraded":"2024-09-21T12:39:15.370588+0000","last_fullsized":"2024-09-21T12:39:15.370588+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:36:37.839024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:37.839024+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":954,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.371812+0000","last_change":"2024-09-21T12:36:44.561097+0000","last_active":"2024-09-21T12:39:15.371812+0000","last_peered":"2024-09-21T12:39:15.371812+0000","last_clean":"2024-09-21T12:39:15.371812+0000","last_became_active":"2024-09-21T12:32:38.425335+0000","last_became_peered":"2024-09-21T12:32:38.425335+0000","last_unstale":"2024-09-21T12:39:15.371812+0000","last_undegraded":"2024-09-21T12:39:15.371812+0000","last_fullsized":"2024-09-21T12:39:15.371812+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:36:44.561030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:36:44.561030+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:57.870150+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":1029,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181432+0000","last_change":"2024-09-21T12:38:54.562392+0000","last_active":"2024-09-21T12:39:07.181432+0000","last_peered":"2024-09-21T12:39:07.181432+0000","last_clean":"2024-09-21T12:39:07.181432+0000","last_became_active":"2024-09-21T12:36:00.148761+0000","last_became_peered":"2024-09-21T12:36:00.148761+0000","last_unstale":"2024-09-21T12:39:07.181432+0000","last_undegraded":"2024-09-21T12:39:07.181432+0000","last_fullsized":"2024-09-21T12:39:07.181432+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:38:54.562317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:54.562317+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:07.141362+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"415'18","reported_seq":399,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.222829+0000","last_change":"2024-09-21T12:37:56.019259+0000","last_active":"2024-09-21T12:39:16.222829+0000","last_peered":"2024-09-21T12:39:16.222829+0000","last_clean":"2024-09-21T12:39:16.222829+0000","last_became_active":"2024-09-21T12:36:00.147676+0000","last_became_peered":"2024-09-21T12:36:00.147676+0000","last_unstale":"2024-09-21T12:39:16.222829+0000","last_undegraded":"2024-09-21T12:39:16.222829+0000","last_fullsized":"2024-09-21T12:39:16.222829+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'18","last_scrub_stamp":"2024-09-21T12:37:56.019080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:37:56.019080+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:18.873114+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"484'17184","reported_seq":19619,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.369836+0000","last_change":"2024-09-21T12:37:14.558326+0000","last_active":"2024-09-21T12:39:15.369836+0000","last_peered":"2024-09-21T12:39:15.369836+0000","last_clean":"2024-09-21T12:39:15.369836+0000","last_became_active":"2024-09-21T12:36:13.105161+0000","last_became_peered":"2024-09-21T12:36:13.105161+0000","last_unstale":"2024-09-21T12:39:15.369836+0000","last_undegraded":"2024-09-21T12:39:15.369836+0000","last_fullsized":"2024-09-21T12:39:15.369836+0000","mapping_epoch":420,"log_start":"458'13000","ondisk_log_start":"458'13000","created":405,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"438'7954","last_scrub_stamp":"2024-09-21T12:37:14.558192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:37:14.558192+0000","objects_scrubbed":7862,"log_size":4184,"log_dups_size":0,"ondisk_log_size":4184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:11.142794+0000","scrub_duration":2678,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17184,"num_write_kb":549888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1078,"num_bytes_recovered":70647808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"76'79","reported_seq":463,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.917056+0000","last_change":"2024-09-21T12:38:07.165980+0000","last_active":"2024-09-21T12:39:15.917056+0000","last_peered":"2024-09-21T12:39:15.917056+0000","last_clean":"2024-09-21T12:39:15.917056+0000","last_became_active":"2024-09-21T12:38:07.165640+0000","last_became_peered":"2024-09-21T12:38:07.165640+0000","last_unstale":"2024-09-21T12:39:15.917056+0000","last_undegraded":"2024-09-21T12:39:15.917056+0000","last_fullsized":"2024-09-21T12:39:15.917056+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:18.151644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:18.151644+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:26.197946+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":571,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.173840+0000","last_change":"2024-09-21T12:38:40.030449+0000","last_active":"2024-09-21T12:39:07.173840+0000","last_peered":"2024-09-21T12:39:07.173840+0000","last_clean":"2024-09-21T12:39:07.173840+0000","last_became_active":"2024-09-21T12:38:07.166374+0000","last_became_peered":"2024-09-21T12:38:07.166374+0000","last_unstale":"2024-09-21T12:39:07.173840+0000","last_undegraded":"2024-09-21T12:39:07.173840+0000","last_fullsized":"2024-09-21T12:39:07.173840+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:38:40.030357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:40.030357+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:09.020801+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":373,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.182606+0000","last_change":"2024-09-21T12:36:00.140935+0000","last_active":"2024-09-21T12:39:07.182606+0000","last_peered":"2024-09-21T12:39:07.182606+0000","last_clean":"2024-09-21T12:39:07.182606+0000","last_became_active":"2024-09-21T12:36:00.140455+0000","last_became_peered":"2024-09-21T12:36:00.140455+0000","last_unstale":"2024-09-21T12:39:07.182606+0000","last_undegraded":"2024-09-21T12:39:07.182606+0000","last_fullsized":"2024-09-21T12:39:07.182606+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:05.703464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:05.703464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:57.233717+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"352'15","reported_seq":384,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.222576+0000","last_change":"2024-09-21T12:38:02.464559+0000","last_active":"2024-09-21T12:39:16.222576+0000","last_peered":"2024-09-21T12:39:16.222576+0000","last_clean":"2024-09-21T12:39:16.222576+0000","last_became_active":"2024-09-21T12:32:38.429092+0000","last_became_peered":"2024-09-21T12:32:38.429092+0000","last_unstale":"2024-09-21T12:39:16.222576+0000","last_undegraded":"2024-09-21T12:39:16.222576+0000","last_fullsized":"2024-09-21T12:39:16.222576+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'15","last_scrub_stamp":"2024-09-21T12:38:02.464486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:02.464486+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:27.762390+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":412,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.917597+0000","last_change":"2024-09-21T12:37:55.128975+0000","last_active":"2024-09-21T12:39:15.917597+0000","last_peered":"2024-09-21T12:39:15.917597+0000","last_clean":"2024-09-21T12:39:15.917597+0000","last_became_active":"2024-09-21T12:37:20.468591+0000","last_became_peered":"2024-09-21T12:37:20.468591+0000","last_unstale":"2024-09-21T12:39:15.917597+0000","last_undegraded":"2024-09-21T12:39:15.917597+0000","last_fullsized":"2024-09-21T12:39:15.917597+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:37:55.128928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:37:55.128928+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"484'17414","reported_seq":22775,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.371304+0000","last_change":"2024-09-21T12:38:49.464266+0000","last_active":"2024-09-21T12:39:15.371304+0000","last_peered":"2024-09-21T12:39:15.371304+0000","last_clean":"2024-09-21T12:39:15.371304+0000","last_became_active":"2024-09-21T12:38:49.404712+0000","last_became_peered":"2024-09-21T12:38:49.404712+0000","last_unstale":"2024-09-21T12:39:15.371304+0000","last_undegraded":"2024-09-21T12:39:15.371304+0000","last_fullsized":"2024-09-21T12:39:15.371304+0000","mapping_epoch":476,"log_start":"458'13100","ondisk_log_start":"458'13100","created":405,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"438'8003","last_scrub_stamp":"2024-09-21T12:37:13.555093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:37:13.555093+0000","objects_scrubbed":7888,"log_size":4314,"log_dups_size":0,"ondisk_log_size":4314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2942,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17414,"num_write_kb":557248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2537,"num_bytes_recovered":166199296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"76'79","reported_seq":312,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.223070+0000","last_change":"2024-09-21T12:38:51.310242+0000","last_active":"2024-09-21T12:39:16.223070+0000","last_peered":"2024-09-21T12:39:16.223070+0000","last_clean":"2024-09-21T12:39:16.223070+0000","last_became_active":"2024-09-21T12:38:07.159280+0000","last_became_peered":"2024-09-21T12:38:07.159280+0000","last_unstale":"2024-09-21T12:39:16.223070+0000","last_undegraded":"2024-09-21T12:39:16.223070+0000","last_fullsized":"2024-09-21T12:39:16.223070+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:38:51.310157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:38:51.310157+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:11.944946+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":1054,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.183030+0000","last_change":"2024-09-21T12:36:00.163117+0000","last_active":"2024-09-21T12:39:07.183030+0000","last_peered":"2024-09-21T12:39:07.183030+0000","last_clean":"2024-09-21T12:39:07.183030+0000","last_became_active":"2024-09-21T12:36:00.139849+0000","last_became_peered":"2024-09-21T12:36:00.139849+0000","last_unstale":"2024-09-21T12:39:07.183030+0000","last_undegraded":"2024-09-21T12:39:07.183030+0000","last_fullsized":"2024-09-21T12:39:07.183030+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T12:35:12.790809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:12.790809+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":905,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.174162+0000","last_change":"2024-09-21T12:36:00.152050+0000","last_active":"2024-09-21T12:39:07.174162+0000","last_peered":"2024-09-21T12:39:07.174162+0000","last_clean":"2024-09-21T12:39:07.174162+0000","last_became_active":"2024-09-21T12:36:00.151488+0000","last_became_peered":"2024-09-21T12:36:00.151488+0000","last_unstale":"2024-09-21T12:39:07.174162+0000","last_undegraded":"2024-09-21T12:39:07.174162+0000","last_fullsized":"2024-09-21T12:39:07.174162+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:22.752960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:22.752960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":307,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.221270+0000","last_change":"2024-09-21T12:36:00.137014+0000","last_active":"2024-09-21T12:39:16.221270+0000","last_peered":"2024-09-21T12:39:16.221270+0000","last_clean":"2024-09-21T12:39:16.221270+0000","last_became_active":"2024-09-21T12:36:00.136650+0000","last_became_peered":"2024-09-21T12:36:00.136650+0000","last_unstale":"2024-09-21T12:39:16.221270+0000","last_undegraded":"2024-09-21T12:39:16.221270+0000","last_fullsized":"2024-09-21T12:39:16.221270+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:12.216104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:12.216104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:04.402566+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":304,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.222040+0000","last_change":"2024-09-21T12:38:44.156366+0000","last_active":"2024-09-21T12:39:16.222040+0000","last_peered":"2024-09-21T12:39:16.222040+0000","last_clean":"2024-09-21T12:39:16.222040+0000","last_became_active":"2024-09-21T12:38:07.166919+0000","last_became_peered":"2024-09-21T12:38:07.166919+0000","last_unstale":"2024-09-21T12:39:16.222040+0000","last_undegraded":"2024-09-21T12:39:16.222040+0000","last_fullsized":"2024-09-21T12:39:16.222040+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:44.156307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:38:44.156307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:06.667086+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"484'17372","reported_seq":19658,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.222227+0000","last_change":"2024-09-21T12:38:01.860989+0000","last_active":"2024-09-21T12:39:16.222227+0000","last_peered":"2024-09-21T12:39:16.222227+0000","last_clean":"2024-09-21T12:39:16.222227+0000","last_became_active":"2024-09-21T12:36:08.101796+0000","last_became_peered":"2024-09-21T12:36:08.101796+0000","last_unstale":"2024-09-21T12:39:16.222227+0000","last_undegraded":"2024-09-21T12:39:16.222227+0000","last_fullsized":"2024-09-21T12:39:16.222227+0000","mapping_epoch":417,"log_start":"459'13200","ondisk_log_start":"459'13200","created":405,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"452'12252","last_scrub_stamp":"2024-09-21T12:38:01.860921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:38:01.860921+0000","objects_scrubbed":5200,"log_size":4172,"log_dups_size":0,"ondisk_log_size":4172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:22.227817+0000","scrub_duration":1407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17372,"num_write_kb":555904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1016,"num_bytes_recovered":66584576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"76'79","reported_seq":841,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.175866+0000","last_change":"2024-09-21T12:36:43.462949+0000","last_active":"2024-09-21T12:39:07.175866+0000","last_peered":"2024-09-21T12:39:07.175866+0000","last_clean":"2024-09-21T12:39:07.175866+0000","last_became_active":"2024-09-21T12:36:00.155047+0000","last_became_peered":"2024-09-21T12:36:00.155047+0000","last_unstale":"2024-09-21T12:39:07.175866+0000","last_undegraded":"2024-09-21T12:39:07.175866+0000","last_fullsized":"2024-09-21T12:39:07.175866+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:43.462874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:43.462874+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":301,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.221129+0000","last_change":"2024-09-21T12:32:38.457999+0000","last_active":"2024-09-21T12:39:16.221129+0000","last_peered":"2024-09-21T12:39:16.221129+0000","last_clean":"2024-09-21T12:39:16.221129+0000","last_became_active":"2024-09-21T12:32:38.425453+0000","last_became_peered":"2024-09-21T12:32:38.425453+0000","last_unstale":"2024-09-21T12:39:16.221129+0000","last_undegraded":"2024-09-21T12:39:16.221129+0000","last_fullsized":"2024-09-21T12:39:16.221129+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-21T12:26:51.463698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:51.463698+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:33.319669+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":40,"num_read_kb":33,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":988,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.184270+0000","last_change":"2024-09-21T12:36:39.754233+0000","last_active":"2024-09-21T12:39:07.184270+0000","last_peered":"2024-09-21T12:39:07.184270+0000","last_clean":"2024-09-21T12:39:07.184270+0000","last_became_active":"2024-09-21T12:32:38.424412+0000","last_became_peered":"2024-09-21T12:32:38.424412+0000","last_unstale":"2024-09-21T12:39:07.184270+0000","last_undegraded":"2024-09-21T12:39:07.184270+0000","last_fullsized":"2024-09-21T12:39:07.184270+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:36:39.754173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:36:39.754173+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"352'12","reported_seq":1180,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.182884+0000","last_change":"2024-09-21T12:37:57.970843+0000","last_active":"2024-09-21T12:39:07.182884+0000","last_peered":"2024-09-21T12:39:07.182884+0000","last_clean":"2024-09-21T12:39:07.182884+0000","last_became_active":"2024-09-21T12:32:38.409177+0000","last_became_peered":"2024-09-21T12:32:38.409177+0000","last_unstale":"2024-09-21T12:39:07.182884+0000","last_undegraded":"2024-09-21T12:39:07.182884+0000","last_fullsized":"2024-09-21T12:39:07.182884+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'12","last_scrub_stamp":"2024-09-21T12:37:57.970636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:37:57.970636+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":894,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370513+0000","last_change":"2024-09-21T12:38:51.740285+0000","last_active":"2024-09-21T12:39:15.370513+0000","last_peered":"2024-09-21T12:39:15.370513+0000","last_clean":"2024-09-21T12:39:15.370513+0000","last_became_active":"2024-09-21T12:38:08.154409+0000","last_became_peered":"2024-09-21T12:38:08.154409+0000","last_unstale":"2024-09-21T12:39:15.370513+0000","last_undegraded":"2024-09-21T12:39:15.370513+0000","last_fullsized":"2024-09-21T12:39:15.370513+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:51.740136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:38:51.740136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:54.159637+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"484'16808","reported_seq":24556,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.222857+0000","last_change":"2024-09-21T12:38:31.314916+0000","last_active":"2024-09-21T12:39:16.222857+0000","last_peered":"2024-09-21T12:39:16.222857+0000","last_clean":"2024-09-21T12:39:16.222857+0000","last_became_active":"2024-09-21T12:38:31.294527+0000","last_became_peered":"2024-09-21T12:38:31.294527+0000","last_unstale":"2024-09-21T12:39:16.222857+0000","last_undegraded":"2024-09-21T12:39:16.222857+0000","last_fullsized":"2024-09-21T12:39:16.222857+0000","mapping_epoch":465,"log_start":"458'12600","ondisk_log_start":"458'12600","created":405,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4208,"log_dups_size":0,"ondisk_log_size":4208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:06.427301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16808,"num_write_kb":537793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3781,"num_bytes_recovered":247726104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"76'79","reported_seq":381,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.174011+0000","last_change":"2024-09-21T12:37:16.761819+0000","last_active":"2024-09-21T12:39:07.174011+0000","last_peered":"2024-09-21T12:39:07.174011+0000","last_clean":"2024-09-21T12:39:07.174011+0000","last_became_active":"2024-09-21T12:36:00.151076+0000","last_became_peered":"2024-09-21T12:36:00.151076+0000","last_unstale":"2024-09-21T12:39:07.174011+0000","last_undegraded":"2024-09-21T12:39:07.174011+0000","last_fullsized":"2024-09-21T12:39:07.174011+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:37:16.761749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:37:16.761749+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":443,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370161+0000","last_change":"2024-09-21T12:36:33.481707+0000","last_active":"2024-09-21T12:39:15.370161+0000","last_peered":"2024-09-21T12:39:15.370161+0000","last_clean":"2024-09-21T12:39:15.370161+0000","last_became_active":"2024-09-21T12:32:38.424412+0000","last_became_peered":"2024-09-21T12:32:38.424412+0000","last_unstale":"2024-09-21T12:39:15.370161+0000","last_undegraded":"2024-09-21T12:39:15.370161+0000","last_fullsized":"2024-09-21T12:39:15.370161+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:36:33.481627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:33.481627+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:07.097608+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":922,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.917297+0000","last_change":"2024-09-21T12:38:07.165927+0000","last_active":"2024-09-21T12:39:15.917297+0000","last_peered":"2024-09-21T12:39:15.917297+0000","last_clean":"2024-09-21T12:39:15.917297+0000","last_became_active":"2024-09-21T12:38:07.165614+0000","last_became_peered":"2024-09-21T12:38:07.165614+0000","last_unstale":"2024-09-21T12:39:15.917297+0000","last_undegraded":"2024-09-21T12:39:15.917297+0000","last_fullsized":"2024-09-21T12:39:15.917297+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:19.629083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:19.629083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:11.161960+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":1142,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:08.537670+0000","last_change":"2024-09-21T12:36:44.788732+0000","last_active":"2024-09-21T12:39:08.537670+0000","last_peered":"2024-09-21T12:39:08.537670+0000","last_clean":"2024-09-21T12:39:08.537670+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:39:08.537670+0000","last_undegraded":"2024-09-21T12:39:08.537670+0000","last_fullsized":"2024-09-21T12:39:08.537670+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:36:44.788662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:44.788662+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":902,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181133+0000","last_change":"2024-09-21T12:38:44.590928+0000","last_active":"2024-09-21T12:39:07.181133+0000","last_peered":"2024-09-21T12:39:07.181133+0000","last_clean":"2024-09-21T12:39:07.181133+0000","last_became_active":"2024-09-21T12:38:07.163353+0000","last_became_peered":"2024-09-21T12:38:07.163353+0000","last_unstale":"2024-09-21T12:39:07.181133+0000","last_undegraded":"2024-09-21T12:39:07.181133+0000","last_fullsized":"2024-09-21T12:39:07.181133+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:44.590868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:38:44.590868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:06.202952+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"484'17532","reported_seq":24489,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370489+0000","last_change":"2024-09-21T12:38:51.655564+0000","last_active":"2024-09-21T12:39:15.370489+0000","last_peered":"2024-09-21T12:39:15.370489+0000","last_clean":"2024-09-21T12:39:15.370489+0000","last_became_active":"2024-09-21T12:38:42.397498+0000","last_became_peered":"2024-09-21T12:38:42.397498+0000","last_unstale":"2024-09-21T12:39:15.370489+0000","last_undegraded":"2024-09-21T12:39:15.370489+0000","last_fullsized":"2024-09-21T12:39:15.370489+0000","mapping_epoch":470,"log_start":"458'13300","ondisk_log_start":"458'13300","created":405,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'15790","last_scrub_stamp":"2024-09-21T12:38:51.655304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:38:51.655304+0000","objects_scrubbed":1772,"log_size":4232,"log_dups_size":0,"ondisk_log_size":4232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:08.589625+0000","scrub_duration":718,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17532,"num_write_kb":561024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3370,"num_bytes_recovered":220856320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"76'79","reported_seq":576,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181090+0000","last_change":"2024-09-21T12:32:52.986242+0000","last_active":"2024-09-21T12:39:07.181090+0000","last_peered":"2024-09-21T12:39:07.181090+0000","last_clean":"2024-09-21T12:39:07.181090+0000","last_became_active":"2024-09-21T12:29:05.835884+0000","last_became_peered":"2024-09-21T12:29:05.835884+0000","last_unstale":"2024-09-21T12:39:07.181090+0000","last_undegraded":"2024-09-21T12:39:07.181090+0000","last_fullsized":"2024-09-21T12:39:07.181090+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:52.986175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:52.986175+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:57.609852+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":1000,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.182668+0000","last_change":"2024-09-21T12:36:36.774685+0000","last_active":"2024-09-21T12:39:07.182668+0000","last_peered":"2024-09-21T12:39:07.182668+0000","last_clean":"2024-09-21T12:39:07.182668+0000","last_became_active":"2024-09-21T12:36:00.127312+0000","last_became_peered":"2024-09-21T12:36:00.127312+0000","last_unstale":"2024-09-21T12:39:07.182668+0000","last_undegraded":"2024-09-21T12:39:07.182668+0000","last_fullsized":"2024-09-21T12:39:07.182668+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'29","last_scrub_stamp":"2024-09-21T12:36:36.774599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:36.774599+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":906,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370516+0000","last_change":"2024-09-21T12:38:48.953684+0000","last_active":"2024-09-21T12:39:15.370516+0000","last_peered":"2024-09-21T12:39:15.370516+0000","last_clean":"2024-09-21T12:39:15.370516+0000","last_became_active":"2024-09-21T12:38:07.163439+0000","last_became_peered":"2024-09-21T12:38:07.163439+0000","last_unstale":"2024-09-21T12:39:15.370516+0000","last_undegraded":"2024-09-21T12:39:15.370516+0000","last_fullsized":"2024-09-21T12:39:15.370516+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:48.953635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:48.953635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:17.502212+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":480,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181361+0000","last_change":"2024-09-21T12:36:00.924159+0000","last_active":"2024-09-21T12:39:07.181361+0000","last_peered":"2024-09-21T12:39:07.181361+0000","last_clean":"2024-09-21T12:39:07.181361+0000","last_became_active":"2024-09-21T12:29:05.825869+0000","last_became_peered":"2024-09-21T12:29:05.825869+0000","last_unstale":"2024-09-21T12:39:07.181361+0000","last_undegraded":"2024-09-21T12:39:07.181361+0000","last_fullsized":"2024-09-21T12:39:07.181361+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:00.924084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:00.924084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":902,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.369923+0000","last_change":"2024-09-21T12:38:45.894780+0000","last_active":"2024-09-21T12:39:15.369923+0000","last_peered":"2024-09-21T12:39:15.369923+0000","last_clean":"2024-09-21T12:39:15.369923+0000","last_became_active":"2024-09-21T12:38:07.158992+0000","last_became_peered":"2024-09-21T12:38:07.158992+0000","last_unstale":"2024-09-21T12:39:15.369923+0000","last_undegraded":"2024-09-21T12:39:15.369923+0000","last_fullsized":"2024-09-21T12:39:15.369923+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:45.894717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:38:45.894717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:57.342298+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"484'17013","reported_seq":17185,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:13.122960+0000","last_change":"2024-09-21T12:35:47.791371+0000","last_active":"2024-09-21T12:39:13.122960+0000","last_peered":"2024-09-21T12:39:13.122960+0000","last_clean":"2024-09-21T12:39:13.122960+0000","last_became_active":"2024-09-21T12:35:47.791120+0000","last_became_peered":"2024-09-21T12:35:47.791120+0000","last_unstale":"2024-09-21T12:39:13.122960+0000","last_undegraded":"2024-09-21T12:39:13.122960+0000","last_fullsized":"2024-09-21T12:39:13.122960+0000","mapping_epoch":405,"log_start":"458'12900","ondisk_log_start":"458'12900","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4113,"log_dups_size":0,"ondisk_log_size":4113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2949120,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17013,"num_write_kb":545856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"76'79","reported_seq":338,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.916438+0000","last_change":"2024-09-21T12:38:07.162380+0000","last_active":"2024-09-21T12:39:15.916438+0000","last_peered":"2024-09-21T12:39:15.916438+0000","last_clean":"2024-09-21T12:39:15.916438+0000","last_became_active":"2024-09-21T12:38:07.162110+0000","last_became_peered":"2024-09-21T12:38:07.162110+0000","last_unstale":"2024-09-21T12:39:15.916438+0000","last_undegraded":"2024-09-21T12:39:15.916438+0000","last_fullsized":"2024-09-21T12:39:15.916438+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:34.055720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:34.055720+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:24.198020+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"457'42","reported_seq":1199,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370457+0000","last_change":"2024-09-21T12:38:07.440187+0000","last_active":"2024-09-21T12:39:15.370457+0000","last_peered":"2024-09-21T12:39:15.370457+0000","last_clean":"2024-09-21T12:39:15.370457+0000","last_became_active":"2024-09-21T12:38:07.164073+0000","last_became_peered":"2024-09-21T12:38:07.164073+0000","last_unstale":"2024-09-21T12:39:15.370457+0000","last_undegraded":"2024-09-21T12:39:15.370457+0000","last_fullsized":"2024-09-21T12:39:15.370457+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'41","last_scrub_stamp":"2024-09-21T12:38:07.440124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:07.440124+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:24.803396+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":301,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.221813+0000","last_change":"2024-09-21T12:37:50.899946+0000","last_active":"2024-09-21T12:39:16.221813+0000","last_peered":"2024-09-21T12:39:16.221813+0000","last_clean":"2024-09-21T12:39:16.221813+0000","last_became_active":"2024-09-21T12:32:38.426309+0000","last_became_peered":"2024-09-21T12:32:38.426309+0000","last_unstale":"2024-09-21T12:39:16.221813+0000","last_undegraded":"2024-09-21T12:39:16.221813+0000","last_fullsized":"2024-09-21T12:39:16.221813+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:50.899879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:37:50.899879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":628,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370586+0000","last_change":"2024-09-21T12:37:57.480538+0000","last_active":"2024-09-21T12:39:15.370586+0000","last_peered":"2024-09-21T12:39:15.370586+0000","last_clean":"2024-09-21T12:39:15.370586+0000","last_became_active":"2024-09-21T12:26:54.083584+0000","last_became_peered":"2024-09-21T12:26:54.083584+0000","last_unstale":"2024-09-21T12:39:15.370586+0000","last_undegraded":"2024-09-21T12:39:15.370586+0000","last_fullsized":"2024-09-21T12:39:15.370586+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T12:37:57.480458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:37:57.480458+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:19.124474+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":939,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.183424+0000","last_change":"2024-09-21T12:38:47.559548+0000","last_active":"2024-09-21T12:39:07.183424+0000","last_peered":"2024-09-21T12:39:07.183424+0000","last_clean":"2024-09-21T12:39:07.183424+0000","last_became_active":"2024-09-21T12:32:38.424402+0000","last_became_peered":"2024-09-21T12:32:38.424402+0000","last_unstale":"2024-09-21T12:39:07.183424+0000","last_undegraded":"2024-09-21T12:39:07.183424+0000","last_fullsized":"2024-09-21T12:39:07.183424+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:47.559490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:47.559490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:03.876249+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":937,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.173142+0000","last_change":"2024-09-21T12:37:56.975879+0000","last_active":"2024-09-21T12:39:07.173142+0000","last_peered":"2024-09-21T12:39:07.173142+0000","last_clean":"2024-09-21T12:39:07.173142+0000","last_became_active":"2024-09-21T12:36:00.144526+0000","last_became_peered":"2024-09-21T12:36:00.144526+0000","last_unstale":"2024-09-21T12:39:07.173142+0000","last_undegraded":"2024-09-21T12:39:07.173142+0000","last_fullsized":"2024-09-21T12:39:07.173142+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:56.975826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:37:56.975826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"484'17435","reported_seq":17607,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:13.099553+0000","last_change":"2024-09-21T12:35:47.848839+0000","last_active":"2024-09-21T12:39:13.099553+0000","last_peered":"2024-09-21T12:39:13.099553+0000","last_clean":"2024-09-21T12:39:13.099553+0000","last_became_active":"2024-09-21T12:35:47.848543+0000","last_became_peered":"2024-09-21T12:35:47.848543+0000","last_unstale":"2024-09-21T12:39:13.099553+0000","last_undegraded":"2024-09-21T12:39:13.099553+0000","last_fullsized":"2024-09-21T12:39:13.099553+0000","mapping_epoch":405,"log_start":"459'13300","ondisk_log_start":"459'13300","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4135,"log_dups_size":0,"ondisk_log_size":4135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3211264,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17435,"num_write_kb":559488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'79","reported_seq":325,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370147+0000","last_change":"2024-09-21T12:38:50.763337+0000","last_active":"2024-09-21T12:39:15.370147+0000","last_peered":"2024-09-21T12:39:15.370147+0000","last_clean":"2024-09-21T12:39:15.370147+0000","last_became_active":"2024-09-21T12:36:01.131622+0000","last_became_peered":"2024-09-21T12:36:01.131622+0000","last_unstale":"2024-09-21T12:39:15.370147+0000","last_undegraded":"2024-09-21T12:39:15.370147+0000","last_fullsized":"2024-09-21T12:39:15.370147+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:38:50.763274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:38:50.763274+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:05.037000+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":405,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.371037+0000","last_change":"2024-09-21T12:37:52.583825+0000","last_active":"2024-09-21T12:39:15.371037+0000","last_peered":"2024-09-21T12:39:15.371037+0000","last_clean":"2024-09-21T12:39:15.371037+0000","last_became_active":"2024-09-21T12:31:39.057316+0000","last_became_peered":"2024-09-21T12:31:39.057316+0000","last_unstale":"2024-09-21T12:39:15.371037+0000","last_undegraded":"2024-09-21T12:39:15.371037+0000","last_fullsized":"2024-09-21T12:39:15.371037+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'17","last_scrub_stamp":"2024-09-21T12:37:52.583730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:37:52.583730+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":938,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.369582+0000","last_change":"2024-09-21T12:38:49.953105+0000","last_active":"2024-09-21T12:39:15.369582+0000","last_peered":"2024-09-21T12:39:15.369582+0000","last_clean":"2024-09-21T12:39:15.369582+0000","last_became_active":"2024-09-21T12:38:07.154643+0000","last_became_peered":"2024-09-21T12:38:07.154643+0000","last_unstale":"2024-09-21T12:39:15.369582+0000","last_undegraded":"2024-09-21T12:39:15.369582+0000","last_fullsized":"2024-09-21T12:39:15.369582+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:49.952845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:49.952845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:51.858894+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":244,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.174482+0000","last_change":"2024-09-21T12:38:49.202301+0000","last_active":"2024-09-21T12:39:07.174482+0000","last_peered":"2024-09-21T12:39:07.174482+0000","last_clean":"2024-09-21T12:39:07.174482+0000","last_became_active":"2024-09-21T12:38:07.168680+0000","last_became_peered":"2024-09-21T12:38:07.168680+0000","last_unstale":"2024-09-21T12:39:07.174482+0000","last_undegraded":"2024-09-21T12:39:07.174482+0000","last_fullsized":"2024-09-21T12:39:07.174482+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:38:49.202240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:38:49.202240+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:56.553523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":237,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.917334+0000","last_change":"2024-09-21T12:38:08.157936+0000","last_active":"2024-09-21T12:39:15.917334+0000","last_peered":"2024-09-21T12:39:15.917334+0000","last_clean":"2024-09-21T12:39:15.917334+0000","last_became_active":"2024-09-21T12:38:08.157608+0000","last_became_peered":"2024-09-21T12:38:08.157608+0000","last_unstale":"2024-09-21T12:39:15.917334+0000","last_undegraded":"2024-09-21T12:39:15.917334+0000","last_fullsized":"2024-09-21T12:39:15.917334+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:02.295255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:02.295255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:25.229168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":136,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.183301+0000","last_change":"2024-09-21T12:38:08.166386+0000","last_active":"2024-09-21T12:39:07.183301+0000","last_peered":"2024-09-21T12:39:07.183301+0000","last_clean":"2024-09-21T12:39:07.183301+0000","last_became_active":"2024-09-21T12:38:08.165465+0000","last_became_peered":"2024-09-21T12:38:08.165465+0000","last_unstale":"2024-09-21T12:39:07.183301+0000","last_undegraded":"2024-09-21T12:39:07.183301+0000","last_fullsized":"2024-09-21T12:39:07.183301+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:53.671046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:37:53.671046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:20.092756+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"484'17185","reported_seq":17357,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:13.135328+0000","last_change":"2024-09-21T12:35:47.791375+0000","last_active":"2024-09-21T12:39:13.135328+0000","last_peered":"2024-09-21T12:39:13.135328+0000","last_clean":"2024-09-21T12:39:13.135328+0000","last_became_active":"2024-09-21T12:35:47.791073+0000","last_became_peered":"2024-09-21T12:35:47.791073+0000","last_unstale":"2024-09-21T12:39:13.135328+0000","last_undegraded":"2024-09-21T12:39:13.135328+0000","last_fullsized":"2024-09-21T12:39:13.135328+0000","mapping_epoch":405,"log_start":"459'13100","ondisk_log_start":"459'13100","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4085,"log_dups_size":0,"ondisk_log_size":4085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2686976,"num_objects":41,"num_object_clones":0,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17185,"num_write_kb":551232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"76'79","reported_seq":860,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370649+0000","last_change":"2024-09-21T12:36:41.495668+0000","last_active":"2024-09-21T12:39:15.370649+0000","last_peered":"2024-09-21T12:39:15.370649+0000","last_clean":"2024-09-21T12:39:15.370649+0000","last_became_active":"2024-09-21T12:32:38.424710+0000","last_became_peered":"2024-09-21T12:32:38.424710+0000","last_unstale":"2024-09-21T12:39:15.370649+0000","last_undegraded":"2024-09-21T12:39:15.370649+0000","last_fullsized":"2024-09-21T12:39:15.370649+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:41.495532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:41.495532+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:55.878434+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":945,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.183554+0000","last_change":"2024-09-21T12:38:07.517859+0000","last_active":"2024-09-21T12:39:07.183554+0000","last_peered":"2024-09-21T12:39:07.183554+0000","last_clean":"2024-09-21T12:39:07.183554+0000","last_became_active":"2024-09-21T12:38:07.166839+0000","last_became_peered":"2024-09-21T12:38:07.166839+0000","last_unstale":"2024-09-21T12:39:07.183554+0000","last_undegraded":"2024-09-21T12:39:07.183554+0000","last_fullsized":"2024-09-21T12:39:07.183554+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'24","last_scrub_stamp":"2024-09-21T12:38:07.517773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:07.517773+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:28.299185+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":953,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181635+0000","last_change":"2024-09-21T12:38:48.558827+0000","last_active":"2024-09-21T12:39:07.181635+0000","last_peered":"2024-09-21T12:39:07.181635+0000","last_clean":"2024-09-21T12:39:07.181635+0000","last_became_active":"2024-09-21T12:32:38.411004+0000","last_became_peered":"2024-09-21T12:32:38.411004+0000","last_unstale":"2024-09-21T12:39:07.181635+0000","last_undegraded":"2024-09-21T12:39:07.181635+0000","last_fullsized":"2024-09-21T12:39:07.181635+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:48.558779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:48.558779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:01.839466+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":362,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.522321+0000","last_change":"2024-09-21T12:36:27.852222+0000","last_active":"2024-09-21T12:39:15.522321+0000","last_peered":"2024-09-21T12:39:15.522321+0000","last_clean":"2024-09-21T12:39:15.522321+0000","last_became_active":"2024-09-21T12:36:00.144053+0000","last_became_peered":"2024-09-21T12:36:00.144053+0000","last_unstale":"2024-09-21T12:39:15.522321+0000","last_undegraded":"2024-09-21T12:39:15.522321+0000","last_fullsized":"2024-09-21T12:39:15.522321+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-21T12:36:27.852030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:36:27.852030+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:58.750782+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":203,"num_read_kb":198,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":349,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.917437+0000","last_change":"2024-09-21T12:38:54.293655+0000","last_active":"2024-09-21T12:39:15.917437+0000","last_peered":"2024-09-21T12:39:15.917437+0000","last_clean":"2024-09-21T12:39:15.917437+0000","last_became_active":"2024-09-21T12:32:02.030240+0000","last_became_peered":"2024-09-21T12:32:02.030240+0000","last_unstale":"2024-09-21T12:39:15.917437+0000","last_undegraded":"2024-09-21T12:39:15.917437+0000","last_fullsized":"2024-09-21T12:39:15.917437+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:54.293611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:54.293611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:20.703408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":393,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.369977+0000","last_change":"2024-09-21T12:37:58.528094+0000","last_active":"2024-09-21T12:39:15.369977+0000","last_peered":"2024-09-21T12:39:15.369977+0000","last_clean":"2024-09-21T12:39:15.369977+0000","last_became_active":"2024-09-21T12:31:39.060160+0000","last_became_peered":"2024-09-21T12:31:39.060160+0000","last_unstale":"2024-09-21T12:39:15.369977+0000","last_undegraded":"2024-09-21T12:39:15.369977+0000","last_fullsized":"2024-09-21T12:39:15.369977+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:58.528033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:37:58.528033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"484'17193","reported_seq":17365,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:13.134237+0000","last_change":"2024-09-21T12:35:47.849177+0000","last_active":"2024-09-21T12:39:13.134237+0000","last_peered":"2024-09-21T12:39:13.134237+0000","last_clean":"2024-09-21T12:39:13.134237+0000","last_became_active":"2024-09-21T12:35:47.848768+0000","last_became_peered":"2024-09-21T12:35:47.848768+0000","last_unstale":"2024-09-21T12:39:13.134237+0000","last_undegraded":"2024-09-21T12:39:13.134237+0000","last_fullsized":"2024-09-21T12:39:13.134237+0000","mapping_epoch":405,"log_start":"459'13100","ondisk_log_start":"459'13100","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4093,"log_dups_size":0,"ondisk_log_size":4093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3080192,"num_objects":47,"num_object_clones":0,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17193,"num_write_kb":551680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"415'35","reported_seq":449,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.223177+0000","last_change":"2024-09-21T12:36:43.681619+0000","last_active":"2024-09-21T12:39:16.223177+0000","last_peered":"2024-09-21T12:39:16.223177+0000","last_clean":"2024-09-21T12:39:16.223177+0000","last_became_active":"2024-09-21T12:36:00.139342+0000","last_became_peered":"2024-09-21T12:36:00.139342+0000","last_unstale":"2024-09-21T12:39:16.223177+0000","last_undegraded":"2024-09-21T12:39:16.223177+0000","last_fullsized":"2024-09-21T12:39:16.223177+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'35","last_scrub_stamp":"2024-09-21T12:36:43.681471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:43.681471+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:09.405650+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":914,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.180089+0000","last_change":"2024-09-21T12:38:01.901094+0000","last_active":"2024-09-21T12:39:07.180089+0000","last_peered":"2024-09-21T12:39:07.180089+0000","last_clean":"2024-09-21T12:39:07.180089+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:39:07.180089+0000","last_undegraded":"2024-09-21T12:39:07.180089+0000","last_fullsized":"2024-09-21T12:39:07.180089+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:38:01.898469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:01.898469+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:08.392980+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":161,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370089+0000","last_change":"2024-09-21T12:37:14.690524+0000","last_active":"2024-09-21T12:39:15.370089+0000","last_peered":"2024-09-21T12:39:15.370089+0000","last_clean":"2024-09-21T12:39:15.370089+0000","last_became_active":"2024-09-21T12:36:00.134878+0000","last_became_peered":"2024-09-21T12:36:00.134878+0000","last_unstale":"2024-09-21T12:39:15.370089+0000","last_undegraded":"2024-09-21T12:39:15.370089+0000","last_fullsized":"2024-09-21T12:39:15.370089+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:37:14.690456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:37:14.690456+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":328,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370270+0000","last_change":"2024-09-21T12:38:08.164955+0000","last_active":"2024-09-21T12:39:15.370270+0000","last_peered":"2024-09-21T12:39:15.370270+0000","last_clean":"2024-09-21T12:39:15.370270+0000","last_became_active":"2024-09-21T12:38:08.164554+0000","last_became_peered":"2024-09-21T12:38:08.164554+0000","last_unstale":"2024-09-21T12:39:15.370270+0000","last_undegraded":"2024-09-21T12:39:15.370270+0000","last_fullsized":"2024-09-21T12:39:15.370270+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:22.649558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:37:22.649558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":565,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.917608+0000","last_change":"2024-09-21T12:38:00.438673+0000","last_active":"2024-09-21T12:39:15.917608+0000","last_peered":"2024-09-21T12:39:15.917608+0000","last_clean":"2024-09-21T12:39:15.917608+0000","last_became_active":"2024-09-21T12:37:20.485126+0000","last_became_peered":"2024-09-21T12:37:20.485126+0000","last_unstale":"2024-09-21T12:39:15.917608+0000","last_undegraded":"2024-09-21T12:39:15.917608+0000","last_fullsized":"2024-09-21T12:39:15.917608+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:00.438586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:38:00.438586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":331,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"484'17223","reported_seq":17395,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:13.135330+0000","last_change":"2024-09-21T12:35:47.788808+0000","last_active":"2024-09-21T12:39:13.135330+0000","last_peered":"2024-09-21T12:39:13.135330+0000","last_clean":"2024-09-21T12:39:13.135330+0000","last_became_active":"2024-09-21T12:35:47.788318+0000","last_became_peered":"2024-09-21T12:35:47.788318+0000","last_unstale":"2024-09-21T12:39:13.135330+0000","last_undegraded":"2024-09-21T12:39:13.135330+0000","last_fullsized":"2024-09-21T12:39:13.135330+0000","mapping_epoch":405,"log_start":"459'13100","ondisk_log_start":"459'13100","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4123,"log_dups_size":0,"ondisk_log_size":4123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2949120,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17223,"num_write_kb":552576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"76'79","reported_seq":856,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370902+0000","last_change":"2024-09-21T12:38:40.853940+0000","last_active":"2024-09-21T12:39:15.370902+0000","last_peered":"2024-09-21T12:39:15.370902+0000","last_clean":"2024-09-21T12:39:15.370902+0000","last_became_active":"2024-09-21T12:27:14.954351+0000","last_became_peered":"2024-09-21T12:27:14.954351+0000","last_unstale":"2024-09-21T12:39:15.370902+0000","last_undegraded":"2024-09-21T12:39:15.370902+0000","last_fullsized":"2024-09-21T12:39:15.370902+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:38:40.853870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:38:40.853870+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:57.441950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":312,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.222736+0000","last_change":"2024-09-21T12:36:41.666876+0000","last_active":"2024-09-21T12:39:16.222736+0000","last_peered":"2024-09-21T12:39:16.222736+0000","last_clean":"2024-09-21T12:39:16.222736+0000","last_became_active":"2024-09-21T12:32:38.419714+0000","last_became_peered":"2024-09-21T12:32:38.419714+0000","last_unstale":"2024-09-21T12:39:16.222736+0000","last_undegraded":"2024-09-21T12:39:16.222736+0000","last_fullsized":"2024-09-21T12:39:16.222736+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:36:41.666812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:41.666812+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:07.411686+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":323,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.221163+0000","last_change":"2024-09-21T12:38:08.148418+0000","last_active":"2024-09-21T12:39:16.221163+0000","last_peered":"2024-09-21T12:39:16.221163+0000","last_clean":"2024-09-21T12:39:16.221163+0000","last_became_active":"2024-09-21T12:38:08.148109+0000","last_became_peered":"2024-09-21T12:38:08.148109+0000","last_unstale":"2024-09-21T12:39:16.221163+0000","last_undegraded":"2024-09-21T12:39:16.221163+0000","last_fullsized":"2024-09-21T12:39:16.221163+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:40.708469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:36:40.708469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:15.280609+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":444,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.917698+0000","last_change":"2024-09-21T12:38:38.201658+0000","last_active":"2024-09-21T12:39:15.917698+0000","last_peered":"2024-09-21T12:39:15.917698+0000","last_clean":"2024-09-21T12:39:15.917698+0000","last_became_active":"2024-09-21T12:37:20.467986+0000","last_became_peered":"2024-09-21T12:37:20.467986+0000","last_unstale":"2024-09-21T12:39:15.917698+0000","last_undegraded":"2024-09-21T12:39:15.917698+0000","last_fullsized":"2024-09-21T12:39:15.917698+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:38.201448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:38.201448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:46.239927+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":887,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.175952+0000","last_change":"2024-09-21T12:35:24.786713+0000","last_active":"2024-09-21T12:39:07.175952+0000","last_peered":"2024-09-21T12:39:07.175952+0000","last_clean":"2024-09-21T12:39:07.175952+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:39:07.175952+0000","last_undegraded":"2024-09-21T12:39:07.175952+0000","last_fullsized":"2024-09-21T12:39:07.175952+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:24.786609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:24.786609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"484'17114","reported_seq":24019,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:13.129747+0000","last_change":"2024-09-21T12:38:41.459562+0000","last_active":"2024-09-21T12:39:13.129747+0000","last_peered":"2024-09-21T12:39:13.129747+0000","last_clean":"2024-09-21T12:39:13.129747+0000","last_became_active":"2024-09-21T12:38:41.419928+0000","last_became_peered":"2024-09-21T12:38:41.419928+0000","last_unstale":"2024-09-21T12:39:13.129747+0000","last_undegraded":"2024-09-21T12:39:13.129747+0000","last_fullsized":"2024-09-21T12:39:13.129747+0000","mapping_epoch":469,"log_start":"458'13000","ondisk_log_start":"458'13000","created":405,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4114,"log_dups_size":0,"ondisk_log_size":4114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3801088,"num_objects":58,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17114,"num_write_kb":549504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3360,"num_bytes_recovered":220135424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"76'79","reported_seq":234,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.182697+0000","last_change":"2024-09-21T12:38:07.162337+0000","last_active":"2024-09-21T12:39:07.182697+0000","last_peered":"2024-09-21T12:39:07.182697+0000","last_clean":"2024-09-21T12:39:07.182697+0000","last_became_active":"2024-09-21T12:38:07.161730+0000","last_became_peered":"2024-09-21T12:38:07.161730+0000","last_unstale":"2024-09-21T12:39:07.182697+0000","last_undegraded":"2024-09-21T12:39:07.182697+0000","last_fullsized":"2024-09-21T12:39:07.182697+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:17.035216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:17.035216+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:05.361038+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":329,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.222965+0000","last_change":"2024-09-21T12:38:53.268874+0000","last_active":"2024-09-21T12:39:16.222965+0000","last_peered":"2024-09-21T12:39:16.222965+0000","last_clean":"2024-09-21T12:39:16.222965+0000","last_became_active":"2024-09-21T12:38:07.163226+0000","last_became_peered":"2024-09-21T12:38:07.163226+0000","last_unstale":"2024-09-21T12:39:16.222965+0000","last_undegraded":"2024-09-21T12:39:16.222965+0000","last_fullsized":"2024-09-21T12:39:16.222965+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'31","last_scrub_stamp":"2024-09-21T12:38:53.268744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:53.268744+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:08.857112+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":309,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.221560+0000","last_change":"2024-09-21T12:37:53.993984+0000","last_active":"2024-09-21T12:39:16.221560+0000","last_peered":"2024-09-21T12:39:16.221560+0000","last_clean":"2024-09-21T12:39:16.221560+0000","last_became_active":"2024-09-21T12:32:38.424451+0000","last_became_peered":"2024-09-21T12:32:38.424451+0000","last_unstale":"2024-09-21T12:39:16.221560+0000","last_undegraded":"2024-09-21T12:39:16.221560+0000","last_fullsized":"2024-09-21T12:39:16.221560+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:53.993920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:37:53.993920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":922,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370142+0000","last_change":"2024-09-21T12:36:43.539109+0000","last_active":"2024-09-21T12:39:15.370142+0000","last_peered":"2024-09-21T12:39:15.370142+0000","last_clean":"2024-09-21T12:39:15.370142+0000","last_became_active":"2024-09-21T12:32:38.425177+0000","last_became_peered":"2024-09-21T12:32:38.425177+0000","last_unstale":"2024-09-21T12:39:15.370142+0000","last_undegraded":"2024-09-21T12:39:15.370142+0000","last_fullsized":"2024-09-21T12:39:15.370142+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:43.539053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:43.539053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:05.078879+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":844,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.173536+0000","last_change":"2024-09-21T12:38:07.160870+0000","last_active":"2024-09-21T12:39:07.173536+0000","last_peered":"2024-09-21T12:39:07.173536+0000","last_clean":"2024-09-21T12:39:07.173536+0000","last_became_active":"2024-09-21T12:38:07.160538+0000","last_became_peered":"2024-09-21T12:38:07.160538+0000","last_unstale":"2024-09-21T12:39:07.173536+0000","last_undegraded":"2024-09-21T12:39:07.173536+0000","last_fullsized":"2024-09-21T12:39:07.173536+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:20.005906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:34:20.005906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:01.287896+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3","version":"484'17250","reported_seq":19663,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.221802+0000","last_change":"2024-09-21T12:38:43.975447+0000","last_active":"2024-09-21T12:39:16.221802+0000","last_peered":"2024-09-21T12:39:16.221802+0000","last_clean":"2024-09-21T12:39:16.221802+0000","last_became_active":"2024-09-21T12:36:13.105381+0000","last_became_peered":"2024-09-21T12:36:13.105381+0000","last_unstale":"2024-09-21T12:39:16.221802+0000","last_undegraded":"2024-09-21T12:39:16.221802+0000","last_fullsized":"2024-09-21T12:39:16.221802+0000","mapping_epoch":420,"log_start":"454'12500","ondisk_log_start":"454'12500","created":405,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"472'14907","last_scrub_stamp":"2024-09-21T12:38:43.975362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:38:43.975362+0000","objects_scrubbed":2350,"log_size":4750,"log_dups_size":0,"ondisk_log_size":4750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:46.744558+0000","scrub_duration":422,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17250,"num_write_kb":552000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1051,"num_bytes_recovered":68878336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"76'79","reported_seq":979,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370241+0000","last_change":"2024-09-21T12:38:48.834565+0000","last_active":"2024-09-21T12:39:15.370241+0000","last_peered":"2024-09-21T12:39:15.370241+0000","last_clean":"2024-09-21T12:39:15.370241+0000","last_became_active":"2024-09-21T12:38:08.156755+0000","last_became_peered":"2024-09-21T12:38:08.156755+0000","last_unstale":"2024-09-21T12:39:15.370241+0000","last_undegraded":"2024-09-21T12:39:15.370241+0000","last_fullsized":"2024-09-21T12:39:15.370241+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:38:48.834502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:38:48.834502+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:17.209408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":315,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.222380+0000","last_change":"2024-09-21T12:38:46.208578+0000","last_active":"2024-09-21T12:39:16.222380+0000","last_peered":"2024-09-21T12:39:16.222380+0000","last_clean":"2024-09-21T12:39:16.222380+0000","last_became_active":"2024-09-21T12:38:07.162180+0000","last_became_peered":"2024-09-21T12:38:07.162180+0000","last_unstale":"2024-09-21T12:39:16.222380+0000","last_undegraded":"2024-09-21T12:39:16.222380+0000","last_fullsized":"2024-09-21T12:39:16.222380+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:38:46.208493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:46.208493+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:10.147140+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":927,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.175138+0000","last_change":"2024-09-21T12:38:44.144209+0000","last_active":"2024-09-21T12:39:07.175138+0000","last_peered":"2024-09-21T12:39:07.175138+0000","last_clean":"2024-09-21T12:39:07.175138+0000","last_became_active":"2024-09-21T12:32:38.403997+0000","last_became_peered":"2024-09-21T12:32:38.403997+0000","last_unstale":"2024-09-21T12:39:07.175138+0000","last_undegraded":"2024-09-21T12:39:07.175138+0000","last_fullsized":"2024-09-21T12:39:07.175138+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:44.144166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:44.144166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:55.330893+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"457'16","reported_seq":315,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.371309+0000","last_change":"2024-09-21T12:38:08.167263+0000","last_active":"2024-09-21T12:39:15.371309+0000","last_peered":"2024-09-21T12:39:15.371309+0000","last_clean":"2024-09-21T12:39:15.371309+0000","last_became_active":"2024-09-21T12:38:08.165321+0000","last_became_peered":"2024-09-21T12:38:08.165321+0000","last_unstale":"2024-09-21T12:39:15.371309+0000","last_undegraded":"2024-09-21T12:39:15.371309+0000","last_fullsized":"2024-09-21T12:39:15.371309+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'14","last_scrub_stamp":"2024-09-21T12:36:31.838024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:31.838024+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:11.567448+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":303,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.222471+0000","last_change":"2024-09-21T12:38:07.164086+0000","last_active":"2024-09-21T12:39:16.222471+0000","last_peered":"2024-09-21T12:39:16.222471+0000","last_clean":"2024-09-21T12:39:16.222471+0000","last_became_active":"2024-09-21T12:38:07.163242+0000","last_became_peered":"2024-09-21T12:38:07.163242+0000","last_unstale":"2024-09-21T12:39:16.222471+0000","last_undegraded":"2024-09-21T12:39:16.222471+0000","last_fullsized":"2024-09-21T12:39:16.222471+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:06.369436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:06.369436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:59.467726+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"484'17312","reported_seq":22666,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370225+0000","last_change":"2024-09-21T12:38:49.384862+0000","last_active":"2024-09-21T12:39:15.370225+0000","last_peered":"2024-09-21T12:39:15.370225+0000","last_clean":"2024-09-21T12:39:15.370225+0000","last_became_active":"2024-09-21T12:38:49.364306+0000","last_became_peered":"2024-09-21T12:38:49.364306+0000","last_unstale":"2024-09-21T12:39:15.370225+0000","last_undegraded":"2024-09-21T12:39:15.370225+0000","last_fullsized":"2024-09-21T12:39:15.370225+0000","mapping_epoch":476,"log_start":"458'13000","ondisk_log_start":"458'13000","created":405,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'8269","last_scrub_stamp":"2024-09-21T12:37:18.159924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:37:18.159924+0000","objects_scrubbed":8165,"log_size":4312,"log_dups_size":0,"ondisk_log_size":4312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17312,"num_write_kb":553984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2533,"num_bytes_recovered":166002688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"76'79","reported_seq":312,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.222808+0000","last_change":"2024-09-21T12:37:48.990420+0000","last_active":"2024-09-21T12:39:16.222808+0000","last_peered":"2024-09-21T12:39:16.222808+0000","last_clean":"2024-09-21T12:39:16.222808+0000","last_became_active":"2024-09-21T12:36:01.159481+0000","last_became_peered":"2024-09-21T12:36:01.159481+0000","last_unstale":"2024-09-21T12:39:16.222808+0000","last_undegraded":"2024-09-21T12:39:16.222808+0000","last_fullsized":"2024-09-21T12:39:16.222808+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:37:48.990356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:37:48.990356+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":1015,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.182759+0000","last_change":"2024-09-21T12:36:40.771296+0000","last_active":"2024-09-21T12:39:07.182759+0000","last_peered":"2024-09-21T12:39:07.182759+0000","last_clean":"2024-09-21T12:39:07.182759+0000","last_became_active":"2024-09-21T12:32:38.424376+0000","last_became_peered":"2024-09-21T12:32:38.424376+0000","last_unstale":"2024-09-21T12:39:07.182759+0000","last_undegraded":"2024-09-21T12:39:07.182759+0000","last_fullsized":"2024-09-21T12:39:07.182759+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:36:40.771222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:40.771222+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":439,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.917187+0000","last_change":"2024-09-21T12:38:51.327197+0000","last_active":"2024-09-21T12:39:15.917187+0000","last_peered":"2024-09-21T12:39:15.917187+0000","last_clean":"2024-09-21T12:39:15.917187+0000","last_became_active":"2024-09-21T12:37:20.484773+0000","last_became_peered":"2024-09-21T12:37:20.484773+0000","last_unstale":"2024-09-21T12:39:15.917187+0000","last_undegraded":"2024-09-21T12:39:15.917187+0000","last_fullsized":"2024-09-21T12:39:15.917187+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:51.327127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:51.327127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:13.918622+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":874,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.221753+0000","last_change":"2024-09-21T12:38:38.252977+0000","last_active":"2024-09-21T12:39:16.221753+0000","last_peered":"2024-09-21T12:39:16.221753+0000","last_clean":"2024-09-21T12:39:16.221753+0000","last_became_active":"2024-09-21T12:38:07.165226+0000","last_became_peered":"2024-09-21T12:38:07.165226+0000","last_unstale":"2024-09-21T12:39:16.221753+0000","last_undegraded":"2024-09-21T12:39:16.221753+0000","last_fullsized":"2024-09-21T12:39:16.221753+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:38.252885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:38.252885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:06.853610+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":912,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181705+0000","last_change":"2024-09-21T12:38:08.164968+0000","last_active":"2024-09-21T12:39:07.181705+0000","last_peered":"2024-09-21T12:39:07.181705+0000","last_clean":"2024-09-21T12:39:07.181705+0000","last_became_active":"2024-09-21T12:38:08.164626+0000","last_became_peered":"2024-09-21T12:38:08.164626+0000","last_unstale":"2024-09-21T12:39:07.181705+0000","last_undegraded":"2024-09-21T12:39:07.181705+0000","last_fullsized":"2024-09-21T12:39:07.181705+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:34.842666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:34.842666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"484'17074","reported_seq":19443,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370973+0000","last_change":"2024-09-21T12:37:23.352215+0000","last_active":"2024-09-21T12:39:15.370973+0000","last_peered":"2024-09-21T12:39:15.370973+0000","last_clean":"2024-09-21T12:39:15.370973+0000","last_became_active":"2024-09-21T12:36:12.104630+0000","last_became_peered":"2024-09-21T12:36:12.104630+0000","last_unstale":"2024-09-21T12:39:15.370973+0000","last_undegraded":"2024-09-21T12:39:15.370973+0000","last_fullsized":"2024-09-21T12:39:15.370973+0000","mapping_epoch":419,"log_start":"458'12800","ondisk_log_start":"458'12800","created":405,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"444'8526","last_scrub_stamp":"2024-09-21T12:37:23.351998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:37:23.351998+0000","objects_scrubbed":8413,"log_size":4274,"log_dups_size":0,"ondisk_log_size":4274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2758,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17074,"num_write_kb":546368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1047,"num_bytes_recovered":68616192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"76'79","reported_seq":840,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.917672+0000","last_change":"2024-09-21T12:38:41.145390+0000","last_active":"2024-09-21T12:39:15.917672+0000","last_peered":"2024-09-21T12:39:15.917672+0000","last_clean":"2024-09-21T12:39:15.917672+0000","last_became_active":"2024-09-21T12:32:38.414877+0000","last_became_peered":"2024-09-21T12:32:38.414877+0000","last_unstale":"2024-09-21T12:39:15.917672+0000","last_undegraded":"2024-09-21T12:39:15.917672+0000","last_fullsized":"2024-09-21T12:39:15.917672+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:38:41.145341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:38:41.145341+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:00.633321+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":1085,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.221905+0000","last_change":"2024-09-21T12:38:44.587607+0000","last_active":"2024-09-21T12:39:16.221905+0000","last_peered":"2024-09-21T12:39:16.221905+0000","last_clean":"2024-09-21T12:39:16.221905+0000","last_became_active":"2024-09-21T12:36:00.148392+0000","last_became_peered":"2024-09-21T12:36:00.148392+0000","last_unstale":"2024-09-21T12:39:16.221905+0000","last_undegraded":"2024-09-21T12:39:16.221905+0000","last_fullsized":"2024-09-21T12:39:16.221905+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'35","last_scrub_stamp":"2024-09-21T12:38:44.587516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:44.587516+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:45.419078+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":161,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370935+0000","last_change":"2024-09-21T12:38:53.700983+0000","last_active":"2024-09-21T12:39:15.370935+0000","last_peered":"2024-09-21T12:39:15.370935+0000","last_clean":"2024-09-21T12:39:15.370935+0000","last_became_active":"2024-09-21T12:36:00.145287+0000","last_became_peered":"2024-09-21T12:36:00.145287+0000","last_unstale":"2024-09-21T12:39:15.370935+0000","last_undegraded":"2024-09-21T12:39:15.370935+0000","last_fullsized":"2024-09-21T12:39:15.370935+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:53.700939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:53.700939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:55.132890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":802,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.183037+0000","last_change":"2024-09-21T12:38:07.162955+0000","last_active":"2024-09-21T12:39:07.183037+0000","last_peered":"2024-09-21T12:39:07.183037+0000","last_clean":"2024-09-21T12:39:07.183037+0000","last_became_active":"2024-09-21T12:38:07.162684+0000","last_became_peered":"2024-09-21T12:38:07.162684+0000","last_unstale":"2024-09-21T12:39:07.183037+0000","last_undegraded":"2024-09-21T12:39:07.183037+0000","last_fullsized":"2024-09-21T12:39:07.183037+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:20.243774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:20.243774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:05.650260+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":893,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370459+0000","last_change":"2024-09-21T12:38:08.164642+0000","last_active":"2024-09-21T12:39:15.370459+0000","last_peered":"2024-09-21T12:39:15.370459+0000","last_clean":"2024-09-21T12:39:15.370459+0000","last_became_active":"2024-09-21T12:38:08.164339+0000","last_became_peered":"2024-09-21T12:38:08.164339+0000","last_unstale":"2024-09-21T12:39:15.370459+0000","last_undegraded":"2024-09-21T12:39:15.370459+0000","last_fullsized":"2024-09-21T12:39:15.370459+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:48.619605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:37:48.619605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"484'17140","reported_seq":22605,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:13.484734+0000","last_change":"2024-09-21T12:38:44.468122+0000","last_active":"2024-09-21T12:39:13.484734+0000","last_peered":"2024-09-21T12:39:13.484734+0000","last_clean":"2024-09-21T12:39:13.484734+0000","last_became_active":"2024-09-21T12:38:44.441467+0000","last_became_peered":"2024-09-21T12:38:44.441467+0000","last_unstale":"2024-09-21T12:39:13.484734+0000","last_undegraded":"2024-09-21T12:39:13.484734+0000","last_fullsized":"2024-09-21T12:39:13.484734+0000","mapping_epoch":472,"log_start":"453'12223","ondisk_log_start":"453'12223","created":405,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:35:46.767214+0000","objects_scrubbed":0,"log_size":4917,"log_dups_size":0,"ondisk_log_size":4917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1179648,"num_objects":18,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17140,"num_write_kb":549056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2640,"num_bytes_recovered":172883968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"76'79","reported_seq":853,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.176162+0000","last_change":"2024-09-21T12:36:31.563764+0000","last_active":"2024-09-21T12:39:07.176162+0000","last_peered":"2024-09-21T12:39:07.176162+0000","last_clean":"2024-09-21T12:39:07.176162+0000","last_became_active":"2024-09-21T12:27:14.951375+0000","last_became_peered":"2024-09-21T12:27:14.951375+0000","last_unstale":"2024-09-21T12:39:07.176162+0000","last_undegraded":"2024-09-21T12:39:07.176162+0000","last_fullsized":"2024-09-21T12:39:07.176162+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:31.563694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:31.563694+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":313,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.221449+0000","last_change":"2024-09-21T12:37:52.992717+0000","last_active":"2024-09-21T12:39:16.221449+0000","last_peered":"2024-09-21T12:39:16.221449+0000","last_clean":"2024-09-21T12:39:16.221449+0000","last_became_active":"2024-09-21T12:32:38.421609+0000","last_became_peered":"2024-09-21T12:32:38.421609+0000","last_unstale":"2024-09-21T12:39:16.221449+0000","last_undegraded":"2024-09-21T12:39:16.221449+0000","last_fullsized":"2024-09-21T12:39:16.221449+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T12:37:52.992635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:37:52.992635+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":161,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.371079+0000","last_change":"2024-09-21T12:37:59.566316+0000","last_active":"2024-09-21T12:39:15.371079+0000","last_peered":"2024-09-21T12:39:15.371079+0000","last_clean":"2024-09-21T12:39:15.371079+0000","last_became_active":"2024-09-21T12:36:00.150832+0000","last_became_peered":"2024-09-21T12:36:00.150832+0000","last_unstale":"2024-09-21T12:39:15.371079+0000","last_undegraded":"2024-09-21T12:39:15.371079+0000","last_fullsized":"2024-09-21T12:39:15.371079+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:59.566268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:37:59.566268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:26.751431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":232,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181497+0000","last_change":"2024-09-21T12:38:46.602449+0000","last_active":"2024-09-21T12:39:07.181497+0000","last_peered":"2024-09-21T12:39:07.181497+0000","last_clean":"2024-09-21T12:39:07.181497+0000","last_became_active":"2024-09-21T12:38:07.161907+0000","last_became_peered":"2024-09-21T12:38:07.161907+0000","last_unstale":"2024-09-21T12:39:07.181497+0000","last_undegraded":"2024-09-21T12:39:07.181497+0000","last_fullsized":"2024-09-21T12:39:07.181497+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:46.597509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:46.597509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:50.838226+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":923,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.174789+0000","last_change":"2024-09-21T12:38:00.902377+0000","last_active":"2024-09-21T12:39:07.174789+0000","last_peered":"2024-09-21T12:39:07.174789+0000","last_clean":"2024-09-21T12:39:07.174789+0000","last_became_active":"2024-09-21T12:36:00.139300+0000","last_became_peered":"2024-09-21T12:36:00.139300+0000","last_unstale":"2024-09-21T12:39:07.174789+0000","last_undegraded":"2024-09-21T12:39:07.174789+0000","last_fullsized":"2024-09-21T12:39:07.174789+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:00.901368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:38:00.901368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:19.772138+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"484'17686","reported_seq":17957,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.917349+0000","last_change":"2024-09-21T12:37:20.472851+0000","last_active":"2024-09-21T12:39:15.917349+0000","last_peered":"2024-09-21T12:39:15.917349+0000","last_clean":"2024-09-21T12:39:15.917349+0000","last_became_active":"2024-09-21T12:37:20.468527+0000","last_became_peered":"2024-09-21T12:37:20.468527+0000","last_unstale":"2024-09-21T12:39:15.917349+0000","last_undegraded":"2024-09-21T12:39:15.917349+0000","last_fullsized":"2024-09-21T12:39:15.917349+0000","mapping_epoch":441,"log_start":"457'13100","ondisk_log_start":"457'13100","created":405,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"438'8315","last_scrub_stamp":"2024-09-21T12:37:16.765348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:35:46.767214+0000","last_clean_scrub_stamp":"2024-09-21T12:37:16.765348+0000","objects_scrubbed":8225,"log_size":4586,"log_dups_size":0,"ondisk_log_size":4586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2844,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17686,"num_write_kb":565952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"76'79","reported_seq":333,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.371144+0000","last_change":"2024-09-21T12:36:36.859482+0000","last_active":"2024-09-21T12:39:15.371144+0000","last_peered":"2024-09-21T12:39:15.371144+0000","last_clean":"2024-09-21T12:39:15.371144+0000","last_became_active":"2024-09-21T12:36:01.141807+0000","last_became_peered":"2024-09-21T12:36:01.141807+0000","last_unstale":"2024-09-21T12:39:15.371144+0000","last_undegraded":"2024-09-21T12:39:15.371144+0000","last_fullsized":"2024-09-21T12:39:15.371144+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:36.859412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:36.859412+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:02.659187+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":442,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.916597+0000","last_change":"2024-09-21T12:37:56.066173+0000","last_active":"2024-09-21T12:39:15.916597+0000","last_peered":"2024-09-21T12:39:15.916597+0000","last_clean":"2024-09-21T12:39:15.916597+0000","last_became_active":"2024-09-21T12:36:00.137597+0000","last_became_peered":"2024-09-21T12:36:00.137597+0000","last_unstale":"2024-09-21T12:39:15.916597+0000","last_undegraded":"2024-09-21T12:39:15.916597+0000","last_fullsized":"2024-09-21T12:39:15.916597+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:37:56.066042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:37:56.066042+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:17.082865+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":167,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.371077+0000","last_change":"2024-09-21T12:37:11.677270+0000","last_active":"2024-09-21T12:39:15.371077+0000","last_peered":"2024-09-21T12:39:15.371077+0000","last_clean":"2024-09-21T12:39:15.371077+0000","last_became_active":"2024-09-21T12:36:00.148899+0000","last_became_peered":"2024-09-21T12:36:00.148899+0000","last_unstale":"2024-09-21T12:39:15.371077+0000","last_undegraded":"2024-09-21T12:39:15.371077+0000","last_fullsized":"2024-09-21T12:39:15.371077+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-21T12:37:11.677196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:37:11.677196+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"415'16","reported_seq":205,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.369959+0000","last_change":"2024-09-21T12:38:00.562505+0000","last_active":"2024-09-21T12:39:15.369959+0000","last_peered":"2024-09-21T12:39:15.369959+0000","last_clean":"2024-09-21T12:39:15.369959+0000","last_became_active":"2024-09-21T12:36:00.143091+0000","last_became_peered":"2024-09-21T12:36:00.143091+0000","last_unstale":"2024-09-21T12:39:15.369959+0000","last_undegraded":"2024-09-21T12:39:15.369959+0000","last_fullsized":"2024-09-21T12:39:15.369959+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'16","last_scrub_stamp":"2024-09-21T12:38:00.562427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:00.562427+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:16.512959+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":403,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.916900+0000","last_change":"2024-09-21T12:37:20.468878+0000","last_active":"2024-09-21T12:39:15.916900+0000","last_peered":"2024-09-21T12:39:15.916900+0000","last_clean":"2024-09-21T12:39:15.916900+0000","last_became_active":"2024-09-21T12:37:20.468350+0000","last_became_peered":"2024-09-21T12:37:20.468350+0000","last_unstale":"2024-09-21T12:39:15.916900+0000","last_undegraded":"2024-09-21T12:39:15.916900+0000","last_fullsized":"2024-09-21T12:39:15.916900+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:34.998795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:34.998795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:59.304496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"76'79","reported_seq":639,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.243982+0000","last_change":"2024-09-21T12:39:16.243982+0000","last_active":"2024-09-21T12:39:16.243982+0000","last_peered":"2024-09-21T12:39:16.243982+0000","last_clean":"2024-09-21T12:39:16.243982+0000","last_became_active":"2024-09-21T12:31:39.060422+0000","last_became_peered":"2024-09-21T12:31:39.060422+0000","last_unstale":"2024-09-21T12:39:16.243982+0000","last_undegraded":"2024-09-21T12:39:16.243982+0000","last_fullsized":"2024-09-21T12:39:16.243982+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:16.243927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:16.243927+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:40.547837+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":956,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.222377+0000","last_change":"2024-09-21T12:38:05.816377+0000","last_active":"2024-09-21T12:39:16.222377+0000","last_peered":"2024-09-21T12:39:16.222377+0000","last_clean":"2024-09-21T12:39:16.222377+0000","last_became_active":"2024-09-21T12:32:02.026693+0000","last_became_peered":"2024-09-21T12:32:02.026693+0000","last_unstale":"2024-09-21T12:39:16.222377+0000","last_undegraded":"2024-09-21T12:39:16.222377+0000","last_fullsized":"2024-09-21T12:39:16.222377+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:05.816312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:05.816312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:17.640983+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":410,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.173965+0000","last_change":"2024-09-21T12:37:59.919992+0000","last_active":"2024-09-21T12:39:07.173965+0000","last_peered":"2024-09-21T12:39:07.173965+0000","last_clean":"2024-09-21T12:39:07.173965+0000","last_became_active":"2024-09-21T12:36:01.138626+0000","last_became_peered":"2024-09-21T12:36:01.138626+0000","last_unstale":"2024-09-21T12:39:07.173965+0000","last_undegraded":"2024-09-21T12:39:07.173965+0000","last_fullsized":"2024-09-21T12:39:07.173965+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T12:37:59.919947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:37:59.919947+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":423,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.917342+0000","last_change":"2024-09-21T12:37:20.486113+0000","last_active":"2024-09-21T12:39:15.917342+0000","last_peered":"2024-09-21T12:39:15.917342+0000","last_clean":"2024-09-21T12:39:15.917342+0000","last_became_active":"2024-09-21T12:37:20.485806+0000","last_became_peered":"2024-09-21T12:37:20.485806+0000","last_unstale":"2024-09-21T12:39:15.917342+0000","last_undegraded":"2024-09-21T12:39:15.917342+0000","last_fullsized":"2024-09-21T12:39:15.917342+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:42.510764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:42.510764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:57.271996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":161,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370837+0000","last_change":"2024-09-21T12:38:03.579433+0000","last_active":"2024-09-21T12:39:15.370837+0000","last_peered":"2024-09-21T12:39:15.370837+0000","last_clean":"2024-09-21T12:39:15.370837+0000","last_became_active":"2024-09-21T12:36:00.149086+0000","last_became_peered":"2024-09-21T12:36:00.149086+0000","last_unstale":"2024-09-21T12:39:15.370837+0000","last_undegraded":"2024-09-21T12:39:15.370837+0000","last_fullsized":"2024-09-21T12:39:15.370837+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:03.579275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:38:03.579275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'79","reported_seq":887,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370742+0000","last_change":"2024-09-21T12:37:53.523886+0000","last_active":"2024-09-21T12:39:15.370742+0000","last_peered":"2024-09-21T12:39:15.370742+0000","last_clean":"2024-09-21T12:39:15.370742+0000","last_became_active":"2024-09-21T12:32:38.425495+0000","last_became_peered":"2024-09-21T12:32:38.425495+0000","last_unstale":"2024-09-21T12:39:15.370742+0000","last_undegraded":"2024-09-21T12:39:15.370742+0000","last_fullsized":"2024-09-21T12:39:15.370742+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:37:53.523818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:37:53.523818+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":977,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.183654+0000","last_change":"2024-09-21T12:38:04.577293+0000","last_active":"2024-09-21T12:39:07.183654+0000","last_peered":"2024-09-21T12:39:07.183654+0000","last_clean":"2024-09-21T12:39:07.183654+0000","last_became_active":"2024-09-21T12:36:00.142274+0000","last_became_peered":"2024-09-21T12:36:00.142274+0000","last_unstale":"2024-09-21T12:39:07.183654+0000","last_undegraded":"2024-09-21T12:39:07.183654+0000","last_fullsized":"2024-09-21T12:39:07.183654+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:38:04.577219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:04.577219+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:22.773773+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":161,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.369911+0000","last_change":"2024-09-21T12:38:06.554109+0000","last_active":"2024-09-21T12:39:15.369911+0000","last_peered":"2024-09-21T12:39:15.369911+0000","last_clean":"2024-09-21T12:39:15.369911+0000","last_became_active":"2024-09-21T12:36:00.140477+0000","last_became_peered":"2024-09-21T12:36:00.140477+0000","last_unstale":"2024-09-21T12:39:15.369911+0000","last_undegraded":"2024-09-21T12:39:15.369911+0000","last_fullsized":"2024-09-21T12:39:15.369911+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:06.554062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:06.554062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":904,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.173752+0000","last_change":"2024-09-21T12:38:50.213195+0000","last_active":"2024-09-21T12:39:07.173752+0000","last_peered":"2024-09-21T12:39:07.173752+0000","last_clean":"2024-09-21T12:39:07.173752+0000","last_became_active":"2024-09-21T12:36:00.150975+0000","last_became_peered":"2024-09-21T12:36:00.150975+0000","last_unstale":"2024-09-21T12:39:07.173752+0000","last_undegraded":"2024-09-21T12:39:07.173752+0000","last_fullsized":"2024-09-21T12:39:07.173752+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:50.213138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:50.213138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:09.450880+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":889,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.222965+0000","last_change":"2024-09-21T12:38:08.150215+0000","last_active":"2024-09-21T12:39:16.222965+0000","last_peered":"2024-09-21T12:39:16.222965+0000","last_clean":"2024-09-21T12:39:16.222965+0000","last_became_active":"2024-09-21T12:38:08.149697+0000","last_became_peered":"2024-09-21T12:38:08.149697+0000","last_unstale":"2024-09-21T12:39:16.222965+0000","last_undegraded":"2024-09-21T12:39:16.222965+0000","last_fullsized":"2024-09-21T12:39:16.222965+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:57.998679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:37:57.998679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:18.259205+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.10","version":"76'79","reported_seq":128,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.916420+0000","last_change":"2024-09-21T12:37:57.062032+0000","last_active":"2024-09-21T12:39:15.916420+0000","last_peered":"2024-09-21T12:39:15.916420+0000","last_clean":"2024-09-21T12:39:15.916420+0000","last_became_active":"2024-09-21T12:37:20.467540+0000","last_became_peered":"2024-09-21T12:37:20.467540+0000","last_unstale":"2024-09-21T12:39:15.916420+0000","last_undegraded":"2024-09-21T12:39:15.916420+0000","last_fullsized":"2024-09-21T12:39:15.916420+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:37:57.061984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:37:57.061984+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":464,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.917182+0000","last_change":"2024-09-21T12:38:39.157782+0000","last_active":"2024-09-21T12:39:15.917182+0000","last_peered":"2024-09-21T12:39:15.917182+0000","last_clean":"2024-09-21T12:39:15.917182+0000","last_became_active":"2024-09-21T12:38:08.164375+0000","last_became_peered":"2024-09-21T12:38:08.164375+0000","last_unstale":"2024-09-21T12:39:15.917182+0000","last_undegraded":"2024-09-21T12:39:15.917182+0000","last_fullsized":"2024-09-21T12:39:15.917182+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:38:39.157727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:39.157727+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:00.959002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":953,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181500+0000","last_change":"2024-09-21T12:36:00.127553+0000","last_active":"2024-09-21T12:39:07.181500+0000","last_peered":"2024-09-21T12:39:07.181500+0000","last_clean":"2024-09-21T12:39:07.181500+0000","last_became_active":"2024-09-21T12:36:00.127225+0000","last_became_peered":"2024-09-21T12:36:00.127225+0000","last_unstale":"2024-09-21T12:39:07.181500+0000","last_undegraded":"2024-09-21T12:39:07.181500+0000","last_fullsized":"2024-09-21T12:39:07.181500+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:22.225122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:22.225122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:06.616447+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"457'17","reported_seq":491,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.175810+0000","last_change":"2024-09-21T12:38:08.172081+0000","last_active":"2024-09-21T12:39:07.175810+0000","last_peered":"2024-09-21T12:39:07.175810+0000","last_clean":"2024-09-21T12:39:07.175810+0000","last_became_active":"2024-09-21T12:38:08.171750+0000","last_became_peered":"2024-09-21T12:38:08.171750+0000","last_unstale":"2024-09-21T12:39:07.175810+0000","last_undegraded":"2024-09-21T12:39:07.175810+0000","last_fullsized":"2024-09-21T12:39:07.175810+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'15","last_scrub_stamp":"2024-09-21T12:37:12.744729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:37:12.744729+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":303,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.222575+0000","last_change":"2024-09-21T12:38:07.163535+0000","last_active":"2024-09-21T12:39:16.222575+0000","last_peered":"2024-09-21T12:39:16.222575+0000","last_clean":"2024-09-21T12:39:16.222575+0000","last_became_active":"2024-09-21T12:38:07.163162+0000","last_became_peered":"2024-09-21T12:38:07.163162+0000","last_unstale":"2024-09-21T12:39:16.222575+0000","last_undegraded":"2024-09-21T12:39:16.222575+0000","last_fullsized":"2024-09-21T12:39:16.222575+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:05.370263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:05.370263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:58.482168+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"76'79","reported_seq":728,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181827+0000","last_change":"2024-09-21T12:38:00.556820+0000","last_active":"2024-09-21T12:39:07.181827+0000","last_peered":"2024-09-21T12:39:07.181827+0000","last_clean":"2024-09-21T12:39:07.181827+0000","last_became_active":"2024-09-21T12:32:02.034387+0000","last_became_peered":"2024-09-21T12:32:02.034387+0000","last_unstale":"2024-09-21T12:39:07.181827+0000","last_undegraded":"2024-09-21T12:39:07.181827+0000","last_fullsized":"2024-09-21T12:39:07.181827+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:38:00.556755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:38:00.556755+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:12.610054+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":984,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.176306+0000","last_change":"2024-09-21T12:36:36.521884+0000","last_active":"2024-09-21T12:39:07.176306+0000","last_peered":"2024-09-21T12:39:07.176306+0000","last_clean":"2024-09-21T12:39:07.176306+0000","last_became_active":"2024-09-21T12:36:00.155033+0000","last_became_peered":"2024-09-21T12:36:00.155033+0000","last_unstale":"2024-09-21T12:39:07.176306+0000","last_undegraded":"2024-09-21T12:39:07.176306+0000","last_fullsized":"2024-09-21T12:39:07.176306+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:36:36.521810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:36.521810+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":933,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370725+0000","last_change":"2024-09-21T12:38:08.165195+0000","last_active":"2024-09-21T12:39:15.370725+0000","last_peered":"2024-09-21T12:39:15.370725+0000","last_clean":"2024-09-21T12:39:15.370725+0000","last_became_active":"2024-09-21T12:38:08.164312+0000","last_became_peered":"2024-09-21T12:38:08.164312+0000","last_unstale":"2024-09-21T12:39:15.370725+0000","last_undegraded":"2024-09-21T12:39:15.370725+0000","last_fullsized":"2024-09-21T12:39:15.370725+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:18.659683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:37:18.659683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":1146,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.917522+0000","last_change":"2024-09-21T12:38:45.275249+0000","last_active":"2024-09-21T12:39:15.917522+0000","last_peered":"2024-09-21T12:39:15.917522+0000","last_clean":"2024-09-21T12:39:15.917522+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:39:15.917522+0000","last_undegraded":"2024-09-21T12:39:15.917522+0000","last_fullsized":"2024-09-21T12:39:15.917522+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:38:45.275110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:45.275110+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:10.836660+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":917,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181188+0000","last_change":"2024-09-21T12:37:59.582535+0000","last_active":"2024-09-21T12:39:07.181188+0000","last_peered":"2024-09-21T12:39:07.181188+0000","last_clean":"2024-09-21T12:39:07.181188+0000","last_became_active":"2024-09-21T12:36:00.148332+0000","last_became_peered":"2024-09-21T12:36:00.148332+0000","last_unstale":"2024-09-21T12:39:07.181188+0000","last_undegraded":"2024-09-21T12:39:07.181188+0000","last_fullsized":"2024-09-21T12:39:07.181188+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:59.582483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:37:59.582483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"76'79","reported_seq":586,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.369861+0000","last_change":"2024-09-21T12:38:03.539003+0000","last_active":"2024-09-21T12:39:15.369861+0000","last_peered":"2024-09-21T12:39:15.369861+0000","last_clean":"2024-09-21T12:39:15.369861+0000","last_became_active":"2024-09-21T12:27:16.940347+0000","last_became_peered":"2024-09-21T12:27:16.940347+0000","last_unstale":"2024-09-21T12:39:15.369861+0000","last_undegraded":"2024-09-21T12:39:15.369861+0000","last_fullsized":"2024-09-21T12:39:15.369861+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:38:03.538881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:38:03.538881+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:30.585103+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":304,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.221912+0000","last_change":"2024-09-21T12:34:00.191190+0000","last_active":"2024-09-21T12:39:16.221912+0000","last_peered":"2024-09-21T12:39:16.221912+0000","last_clean":"2024-09-21T12:39:16.221912+0000","last_became_active":"2024-09-21T12:32:38.421008+0000","last_became_peered":"2024-09-21T12:32:38.421008+0000","last_unstale":"2024-09-21T12:39:16.221912+0000","last_undegraded":"2024-09-21T12:39:16.221912+0000","last_fullsized":"2024-09-21T12:39:16.221912+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:34:00.191106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:34:00.191106+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:01.433046+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":312,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.222751+0000","last_change":"2024-09-21T12:37:49.940962+0000","last_active":"2024-09-21T12:39:16.222751+0000","last_peered":"2024-09-21T12:39:16.222751+0000","last_clean":"2024-09-21T12:39:16.222751+0000","last_became_active":"2024-09-21T12:36:01.123399+0000","last_became_peered":"2024-09-21T12:36:01.123399+0000","last_unstale":"2024-09-21T12:39:16.222751+0000","last_undegraded":"2024-09-21T12:39:16.222751+0000","last_fullsized":"2024-09-21T12:39:16.222751+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:49.940903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:37:49.940903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:18.658566+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":919,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370773+0000","last_change":"2024-09-21T12:38:00.521861+0000","last_active":"2024-09-21T12:39:15.370773+0000","last_peered":"2024-09-21T12:39:15.370773+0000","last_clean":"2024-09-21T12:39:15.370773+0000","last_became_active":"2024-09-21T12:31:50.040754+0000","last_became_peered":"2024-09-21T12:31:50.040754+0000","last_unstale":"2024-09-21T12:39:15.370773+0000","last_undegraded":"2024-09-21T12:39:15.370773+0000","last_fullsized":"2024-09-21T12:39:15.370773+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:00.521803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:00.521803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":367,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.174400+0000","last_change":"2024-09-21T12:37:19.857159+0000","last_active":"2024-09-21T12:39:07.174400+0000","last_peered":"2024-09-21T12:39:07.174400+0000","last_clean":"2024-09-21T12:39:07.174400+0000","last_became_active":"2024-09-21T12:31:50.036150+0000","last_became_peered":"2024-09-21T12:31:50.036150+0000","last_unstale":"2024-09-21T12:39:07.174400+0000","last_undegraded":"2024-09-21T12:39:07.174400+0000","last_fullsized":"2024-09-21T12:39:07.174400+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:19.857108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:37:19.857108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"76'79","reported_seq":611,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.917764+0000","last_change":"2024-09-21T12:36:45.476276+0000","last_active":"2024-09-21T12:39:15.917764+0000","last_peered":"2024-09-21T12:39:15.917764+0000","last_clean":"2024-09-21T12:39:15.917764+0000","last_became_active":"2024-09-21T12:27:16.921880+0000","last_became_peered":"2024-09-21T12:27:16.921880+0000","last_unstale":"2024-09-21T12:39:15.917764+0000","last_undegraded":"2024-09-21T12:39:15.917764+0000","last_fullsized":"2024-09-21T12:39:15.917764+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:45.476087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:45.476087+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:05.179564+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":406,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370485+0000","last_change":"2024-09-21T12:38:47.987378+0000","last_active":"2024-09-21T12:39:15.370485+0000","last_peered":"2024-09-21T12:39:15.370485+0000","last_clean":"2024-09-21T12:39:15.370485+0000","last_became_active":"2024-09-21T12:38:07.153467+0000","last_became_peered":"2024-09-21T12:38:07.153467+0000","last_unstale":"2024-09-21T12:39:15.370485+0000","last_undegraded":"2024-09-21T12:39:15.370485+0000","last_fullsized":"2024-09-21T12:39:15.370485+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:38:47.987308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:47.987308+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:49.248198+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":161,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.371137+0000","last_change":"2024-09-21T12:38:05.529592+0000","last_active":"2024-09-21T12:39:15.371137+0000","last_peered":"2024-09-21T12:39:15.371137+0000","last_clean":"2024-09-21T12:39:15.371137+0000","last_became_active":"2024-09-21T12:36:00.147063+0000","last_became_peered":"2024-09-21T12:36:00.147063+0000","last_unstale":"2024-09-21T12:39:15.371137+0000","last_undegraded":"2024-09-21T12:39:15.371137+0000","last_fullsized":"2024-09-21T12:39:15.371137+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:05.529479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:05.529479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:29.024607+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":899,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.174865+0000","last_change":"2024-09-21T12:37:50.035009+0000","last_active":"2024-09-21T12:39:07.174865+0000","last_peered":"2024-09-21T12:39:07.174865+0000","last_clean":"2024-09-21T12:39:07.174865+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:39:07.174865+0000","last_undegraded":"2024-09-21T12:39:07.174865+0000","last_fullsized":"2024-09-21T12:39:07.174865+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:50.034945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:37:50.034945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:12.785791+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":882,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181586+0000","last_change":"2024-09-21T12:35:59.959597+0000","last_active":"2024-09-21T12:39:07.181586+0000","last_peered":"2024-09-21T12:39:07.181586+0000","last_clean":"2024-09-21T12:39:07.181586+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:39:07.181586+0000","last_undegraded":"2024-09-21T12:39:07.181586+0000","last_fullsized":"2024-09-21T12:39:07.181586+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:59.959545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:59.959545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"76'79","reported_seq":394,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370336+0000","last_change":"2024-09-21T12:38:08.157233+0000","last_active":"2024-09-21T12:39:15.370336+0000","last_peered":"2024-09-21T12:39:15.370336+0000","last_clean":"2024-09-21T12:39:15.370336+0000","last_became_active":"2024-09-21T12:38:08.156834+0000","last_became_peered":"2024-09-21T12:38:08.156834+0000","last_unstale":"2024-09-21T12:39:15.370336+0000","last_undegraded":"2024-09-21T12:39:15.370336+0000","last_fullsized":"2024-09-21T12:39:15.370336+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:43.792355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:43.792355+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:57.778271+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":972,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.174767+0000","last_change":"2024-09-21T12:36:41.506309+0000","last_active":"2024-09-21T12:39:07.174767+0000","last_peered":"2024-09-21T12:39:07.174767+0000","last_clean":"2024-09-21T12:39:07.174767+0000","last_became_active":"2024-09-21T12:36:00.139682+0000","last_became_peered":"2024-09-21T12:36:00.139682+0000","last_unstale":"2024-09-21T12:39:07.174767+0000","last_undegraded":"2024-09-21T12:39:07.174767+0000","last_fullsized":"2024-09-21T12:39:07.174767+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:36:41.506231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:41.506231+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":939,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.917465+0000","last_change":"2024-09-21T12:36:34.526618+0000","last_active":"2024-09-21T12:39:15.917465+0000","last_peered":"2024-09-21T12:39:15.917465+0000","last_clean":"2024-09-21T12:39:15.917465+0000","last_became_active":"2024-09-21T12:36:00.151584+0000","last_became_peered":"2024-09-21T12:36:00.151584+0000","last_unstale":"2024-09-21T12:39:15.917465+0000","last_undegraded":"2024-09-21T12:39:15.917465+0000","last_fullsized":"2024-09-21T12:39:15.917465+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:34.526551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:36:34.526551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:56.283290+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":161,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.369742+0000","last_change":"2024-09-21T12:37:51.569522+0000","last_active":"2024-09-21T12:39:15.369742+0000","last_peered":"2024-09-21T12:39:15.369742+0000","last_clean":"2024-09-21T12:39:15.369742+0000","last_became_active":"2024-09-21T12:36:00.144521+0000","last_became_peered":"2024-09-21T12:36:00.144521+0000","last_unstale":"2024-09-21T12:39:15.369742+0000","last_undegraded":"2024-09-21T12:39:15.369742+0000","last_fullsized":"2024-09-21T12:39:15.369742+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:51.569462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:37:51.569462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":887,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.175053+0000","last_change":"2024-09-21T12:35:51.628370+0000","last_active":"2024-09-21T12:39:07.175053+0000","last_peered":"2024-09-21T12:39:07.175053+0000","last_clean":"2024-09-21T12:39:07.175053+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:39:07.175053+0000","last_undegraded":"2024-09-21T12:39:07.175053+0000","last_fullsized":"2024-09-21T12:39:07.175053+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:51.628326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:51.628326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"76'79","reported_seq":161,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370806+0000","last_change":"2024-09-21T12:38:54.700596+0000","last_active":"2024-09-21T12:39:15.370806+0000","last_peered":"2024-09-21T12:39:15.370806+0000","last_clean":"2024-09-21T12:39:15.370806+0000","last_became_active":"2024-09-21T12:36:00.151043+0000","last_became_peered":"2024-09-21T12:36:00.151043+0000","last_unstale":"2024-09-21T12:39:15.370806+0000","last_undegraded":"2024-09-21T12:39:15.370806+0000","last_fullsized":"2024-09-21T12:39:15.370806+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:38:54.700420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:38:54.700420+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:56.059368+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":907,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.182935+0000","last_change":"2024-09-21T12:35:14.789148+0000","last_active":"2024-09-21T12:39:07.182935+0000","last_peered":"2024-09-21T12:39:07.182935+0000","last_clean":"2024-09-21T12:39:07.182935+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:39:07.182935+0000","last_undegraded":"2024-09-21T12:39:07.182935+0000","last_fullsized":"2024-09-21T12:39:07.182935+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'8","last_scrub_stamp":"2024-09-21T12:35:14.789070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:14.789070+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":915,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.182239+0000","last_change":"2024-09-21T12:35:27.237430+0000","last_active":"2024-09-21T12:39:07.182239+0000","last_peered":"2024-09-21T12:39:07.182239+0000","last_clean":"2024-09-21T12:39:07.182239+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:39:07.182239+0000","last_undegraded":"2024-09-21T12:39:07.182239+0000","last_fullsized":"2024-09-21T12:39:07.182239+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:35:27.236980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:27.236980+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":936,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.221658+0000","last_change":"2024-09-21T12:36:00.135914+0000","last_active":"2024-09-21T12:39:16.221658+0000","last_peered":"2024-09-21T12:39:16.221658+0000","last_clean":"2024-09-21T12:39:16.221658+0000","last_became_active":"2024-09-21T12:36:00.135537+0000","last_became_peered":"2024-09-21T12:36:00.135537+0000","last_unstale":"2024-09-21T12:39:16.221658+0000","last_undegraded":"2024-09-21T12:39:16.221658+0000","last_fullsized":"2024-09-21T12:39:16.221658+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:40.861493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:40.861493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:34.475132+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":921,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370238+0000","last_change":"2024-09-21T12:38:08.164865+0000","last_active":"2024-09-21T12:39:15.370238+0000","last_peered":"2024-09-21T12:39:15.370238+0000","last_clean":"2024-09-21T12:39:15.370238+0000","last_became_active":"2024-09-21T12:38:08.164535+0000","last_became_peered":"2024-09-21T12:38:08.164535+0000","last_unstale":"2024-09-21T12:39:15.370238+0000","last_undegraded":"2024-09-21T12:39:15.370238+0000","last_fullsized":"2024-09-21T12:39:15.370238+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:38.838688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:38.838688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:04.705707+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"76'79","reported_seq":306,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.221654+0000","last_change":"2024-09-21T12:36:00.137642+0000","last_active":"2024-09-21T12:39:16.221654+0000","last_peered":"2024-09-21T12:39:16.221654+0000","last_clean":"2024-09-21T12:39:16.221654+0000","last_became_active":"2024-09-21T12:36:00.137290+0000","last_became_peered":"2024-09-21T12:36:00.137290+0000","last_unstale":"2024-09-21T12:39:16.221654+0000","last_undegraded":"2024-09-21T12:39:16.221654+0000","last_fullsized":"2024-09-21T12:39:16.221654+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:43.304606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:43.304606+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:05.436780+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":925,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:15.370498+0000","last_change":"2024-09-21T12:38:51.954801+0000","last_active":"2024-09-21T12:39:15.370498+0000","last_peered":"2024-09-21T12:39:15.370498+0000","last_clean":"2024-09-21T12:39:15.370498+0000","last_became_active":"2024-09-21T12:32:38.426822+0000","last_became_peered":"2024-09-21T12:32:38.426822+0000","last_unstale":"2024-09-21T12:39:15.370498+0000","last_undegraded":"2024-09-21T12:39:15.370498+0000","last_fullsized":"2024-09-21T12:39:15.370498+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:51.954739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:38:51.954739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:53.948595+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":912,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.183174+0000","last_change":"2024-09-21T12:35:30.730640+0000","last_active":"2024-09-21T12:39:07.183174+0000","last_peered":"2024-09-21T12:39:07.183174+0000","last_clean":"2024-09-21T12:39:07.183174+0000","last_became_active":"2024-09-21T12:31:39.052486+0000","last_became_peered":"2024-09-21T12:31:39.052486+0000","last_unstale":"2024-09-21T12:39:07.183174+0000","last_undegraded":"2024-09-21T12:39:07.183174+0000","last_fullsized":"2024-09-21T12:39:07.183174+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:30.730574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:30.730574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":938,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-21T12:39:16.222162+0000","last_change":"2024-09-21T12:38:41.504397+0000","last_active":"2024-09-21T12:39:16.222162+0000","last_peered":"2024-09-21T12:39:16.222162+0000","last_clean":"2024-09-21T12:39:16.222162+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:39:16.222162+0000","last_undegraded":"2024-09-21T12:39:16.222162+0000","last_fullsized":"2024-09-21T12:39:16.222162+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:38:41.504320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:41.504320+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:48.172302+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":941,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-21T12:39:07.181268+0000","last_change":"2024-09-21T12:38:08.175436+0000","last_active":"2024-09-21T12:39:07.181268+0000","last_peered":"2024-09-21T12:39:07.181268+0000","last_clean":"2024-09-21T12:39:07.181268+0000","last_became_active":"2024-09-21T12:38:08.174864+0000","last_became_peered":"2024-09-21T12:38:08.174864+0000","last_unstale":"2024-09-21T12:39:07.181268+0000","last_undegraded":"2024-09-21T12:39:07.181268+0000","last_fullsized":"2024-09-21T12:39:07.181268+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:28.911328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:28.911328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":19857408,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":275935,"num_write_kb":8839553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22413,"num_bytes_recovered":1468530712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":107151360,"data_stored":107151360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68612,"ondisk_log_size":68612,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1105,"num_read_kb":862,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":472,"num_bytes_recovered":9372,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3597,"internal_metadata":0},"log_size":688,"ondisk_log_size":688,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":598,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":203,"num_read_kb":198,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2449,"ondisk_log_size":2449,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234216,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":845572,"kb_used_data":2224,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":92899580,"statfs":{"total":95995035648,"available":95129169920,"internally_reserved":0,"allocated":2277376,"data_stored":1477757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":863566282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463152,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":885872,"kb_used_data":2268,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":92859280,"statfs":{"total":95995035648,"available":95087902720,"internally_reserved":0,"allocated":2322432,"data_stored":1542100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":904788422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724787,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":781704,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":779454,"kb_avail":92963448,"statfs":{"total":95995035648,"available":95194570752,"internally_reserved":0,"allocated":2281472,"data_stored":1478693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":798161349},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953723,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":408008,"kb_used_data":1972,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":93337144,"statfs":{"total":95995035648,"available":95577235456,"internally_reserved":0,"allocated":2019328,"data_stored":1214589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149952,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":965892,"kb_used_data":60084,"kb_used_omap":1,"kb_used_meta":905790,"kb_avail":92779260,"statfs":{"total":95995035648,"available":95005962240,"internally_reserved":0,"allocated":61526016,"data_stored":60714628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":927529415},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":455,"seq":1954210119695,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":230564,"kb_used_data":2196,"kb_used_omap":7,"kb_used_meta":228344,"kb_avail":93514588,"statfs":{"total":95995035648,"available":95758938112,"internally_reserved":0,"allocated":2248704,"data_stored":1468786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8042,"internal_metadata":233824406},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575117,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":975844,"kb_used_data":2000,"kb_used_omap":1,"kb_used_meta":973822,"kb_avail":92769308,"statfs":{"total":95995035648,"available":94995771392,"internally_reserved":0,"allocated":2048000,"data_stored":1215452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":997194181},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738642,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1055036,"kb_used_data":113068,"kb_used_omap":1,"kb_used_meta":941886,"kb_avail":92690116,"statfs":{"total":95995035648,"available":94914678784,"internally_reserved":0,"allocated":115781632,"data_stored":114978026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":964491717},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3597,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":85458944,"data_stored":85458944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":21692416,"data_stored":21692416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:39:20.922 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T12:39:20.922 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T12:39:20.922 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-21T12:39:20.922 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:39:21.178 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-21T12:39:21.178 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-21T12:39:21.178 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-21T12:39:21.178 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-21T12:39:21.178 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-21T12:39:21.179 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T12:39:21.179 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T12:39:21.179 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T12:39:21.179 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T12:39:21.179 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T12:39:21.179 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-21T12:39:21.179 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-21T12:39:21.180 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-21T12:39:21.180 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T12:39:21.180 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T12:39:21.180 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T12:39:21.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:21 smithi114 ceph-mon[40867]: 3.10 scrub starts 2024-09-21T12:39:21.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:21 smithi114 ceph-mon[40867]: 3.10 scrub ok 2024-09-21T12:39:21.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:21 smithi114 ceph-mon[40867]: 3.7 scrub starts 2024-09-21T12:39:21.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:21 smithi114 ceph-mon[40867]: 3.7 scrub ok 2024-09-21T12:39:21.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:21 smithi114 ceph-mon[40867]: pgmap v978: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T12:39:21.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:21 smithi114 ceph-mon[30324]: 3.10 scrub starts 2024-09-21T12:39:21.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:21 smithi114 ceph-mon[30324]: 3.10 scrub ok 2024-09-21T12:39:21.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:21 smithi114 ceph-mon[30324]: 3.7 scrub starts 2024-09-21T12:39:21.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:21 smithi114 ceph-mon[30324]: 3.7 scrub ok 2024-09-21T12:39:21.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:21 smithi114 ceph-mon[30324]: pgmap v978: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T12:39:21.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:21 smithi183 ceph-mon[33277]: 3.10 scrub starts 2024-09-21T12:39:21.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:21 smithi183 ceph-mon[33277]: 3.10 scrub ok 2024-09-21T12:39:21.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:21 smithi183 ceph-mon[33277]: 3.7 scrub starts 2024-09-21T12:39:21.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:21 smithi183 ceph-mon[33277]: 3.7 scrub ok 2024-09-21T12:39:21.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:21 smithi183 ceph-mon[33277]: pgmap v978: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T12:39:22.282 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T12:39:22.778 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:39:22.902 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:39:22 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:39:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:39:23.323 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-21T12:39:23.324 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 20, 2024-09-21T12:39:23.324 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 600, 2024-09-21T12:39:23.324 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-21T12:39:23.324 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.324 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "pg_backfill(progress 11.d e 457/457 lb 11:b03faa9a:::benchmark_data_smithi114_187622_object101344:head)", 2024-09-21T12:39:23.324 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:38:08.641956+0000", 2024-09-21T12:39:23.324 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 74.643525957999998, 2024-09-21T12:39:23.324 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.086870960999999997, 2024-09-21T12:39:23.324 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:39:23.324 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:39:23.324 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:39:23.324 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:39:23.325 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.325 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:39:23.325 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:08.641956+0000", 2024-09-21T12:39:23.325 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.325 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.325 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.325 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:39:23.325 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:08.641956+0000", 2024-09-21T12:39:23.325 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.325 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.325 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.325 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:39:23.325 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:08.641956+0000", 2024-09-21T12:39:23.325 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5999999999999999e-07 2024-09-21T12:39:23.326 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.326 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.326 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:39:23.326 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:08.641961+0000", 2024-09-21T12:39:23.326 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8389999999999997e-06 2024-09-21T12:39:23.326 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.326 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.326 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:39:23.326 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:08.641967+0000", 2024-09-21T12:39:23.326 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2480000000000001e-06 2024-09-21T12:39:23.326 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.326 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.327 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:39:23.327 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:08.641971+0000", 2024-09-21T12:39:23.327 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.692e-06 2024-09-21T12:39:23.327 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.327 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.327 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:39:23.327 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:08.728720+0000", 2024-09-21T12:39:23.327 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.086748757999999995 2024-09-21T12:39:23.327 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.327 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.327 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:39:23.328 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:08.728752+0000", 2024-09-21T12:39:23.328 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2738999999999999e-05 2024-09-21T12:39:23.328 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.328 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.328 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:39:23.328 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:08.728827+0000", 2024-09-21T12:39:23.328 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4225000000000002e-05 2024-09-21T12:39:23.328 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.328 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:39:23.328 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.328 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.329 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.329 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "MOSDPGPush(11.d 457/456 [PushOp(11:b0408a0b:::benchmark_data_smithi114_187622_object125687:head, version: 438'7656, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(11:b0408a0b:::benchmark_data_smithi114_187622_object125687:head@438'7656, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T12:39:23.329 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:38:08.641981+0000", 2024-09-21T12:39:23.329 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 74.643500750000001, 2024-09-21T12:39:23.329 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.088394362000000004, 2024-09-21T12:39:23.329 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:39:23.329 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:39:23.329 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:39:23.329 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:39:23.329 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.329 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:39:23.330 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:08.641981+0000", 2024-09-21T12:39:23.330 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.330 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.330 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.330 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:39:23.330 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:08.641981+0000", 2024-09-21T12:39:23.330 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.330 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.330 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.330 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:39:23.330 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:08.641981+0000", 2024-09-21T12:39:23.331 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.7500000000000001e-07 2024-09-21T12:39:23.331 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.331 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.331 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:39:23.331 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:08.642012+0000", 2024-09-21T12:39:23.331 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.1208999999999999e-05 2024-09-21T12:39:23.331 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.331 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.331 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:39:23.331 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:08.729837+0000", 2024-09-21T12:39:23.331 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.087824719999999995 2024-09-21T12:39:23.332 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.332 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.332 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:39:23.332 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:08.729943+0000", 2024-09-21T12:39:23.332 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000105683 2024-09-21T12:39:23.332 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.332 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.332 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:39:23.332 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:08.730047+0000", 2024-09-21T12:39:23.332 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000104299 2024-09-21T12:39:23.332 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.333 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.333 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:39:23.333 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:08.730061+0000", 2024-09-21T12:39:23.333 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3468999999999999e-05 2024-09-21T12:39:23.333 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.333 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.333 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:39:23.333 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:08.730375+0000", 2024-09-21T12:39:23.333 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00031470700000000001 2024-09-21T12:39:23.333 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.333 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:39:23.333 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.334 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.334 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.334 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "MOSDPGPush(11.d 460/456 [PushOp(11:b9b8fe2a:::benchmark_data_smithi114_187622_object84895:head, version: 431'5165, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(11:b9b8fe2a:::benchmark_data_smithi114_187622_object84895:head@431'5165, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T12:39:23.334 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:38:21.443638+0000", 2024-09-21T12:39:23.334 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 61.841843552, 2024-09-21T12:39:23.334 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.070681666000000004, 2024-09-21T12:39:23.334 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:39:23.334 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:39:23.334 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:39:23.334 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:39:23.334 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.335 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:39:23.335 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:21.443638+0000", 2024-09-21T12:39:23.335 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.335 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.335 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.335 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:39:23.335 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:21.443638+0000", 2024-09-21T12:39:23.335 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.335 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.335 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.335 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:39:23.336 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:21.443639+0000", 2024-09-21T12:39:23.336 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7199999999999999e-07 2024-09-21T12:39:23.336 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.336 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.336 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:39:23.336 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:21.443700+0000", 2024-09-21T12:39:23.336 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1829999999999996e-05 2024-09-21T12:39:23.336 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.336 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.336 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:39:23.336 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:21.443712+0000", 2024-09-21T12:39:23.336 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1812e-05 2024-09-21T12:39:23.337 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.337 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.337 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:39:23.337 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:21.443721+0000", 2024-09-21T12:39:23.337 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9160000000000007e-06 2024-09-21T12:39:23.337 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.337 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.337 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:39:23.337 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:21.514112+0000", 2024-09-21T12:39:23.337 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.070390615000000004 2024-09-21T12:39:23.338 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.338 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.338 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:39:23.338 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:21.514124+0000", 2024-09-21T12:39:23.338 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2374e-05 2024-09-21T12:39:23.338 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.338 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.338 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:39:23.338 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:21.514320+0000", 2024-09-21T12:39:23.338 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00019564699999999999 2024-09-21T12:39:23.338 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.338 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:39:23.339 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.339 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.339 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.339 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "pg_backfill(progress 11.d e 462/462 lb 11:bbe54ecd:::benchmark_data_smithi114_187622_object122159:head)", 2024-09-21T12:39:23.339 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:38:24.237624+0000", 2024-09-21T12:39:23.339 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 59.047857677000003, 2024-09-21T12:39:23.339 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.11110545099999999, 2024-09-21T12:39:23.339 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:39:23.339 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:39:23.339 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:39:23.339 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:39:23.339 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.339 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:39:23.340 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.237624+0000", 2024-09-21T12:39:23.340 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.340 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.340 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.340 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:39:23.340 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.237624+0000", 2024-09-21T12:39:23.340 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.340 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.340 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.340 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:39:23.340 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.237624+0000", 2024-09-21T12:39:23.340 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.0199999999999998e-07 2024-09-21T12:39:23.341 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.341 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.341 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:39:23.341 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.237628+0000", 2024-09-21T12:39:23.341 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8689999999999997e-06 2024-09-21T12:39:23.341 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.341 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.341 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:39:23.341 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.237632+0000", 2024-09-21T12:39:23.341 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1779999999999996e-06 2024-09-21T12:39:23.341 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.341 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.341 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:39:23.341 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.237635+0000", 2024-09-21T12:39:23.342 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.016e-06 2024-09-21T12:39:23.342 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.342 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.342 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:39:23.342 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.348646+0000", 2024-09-21T12:39:23.342 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.11101103900000001 2024-09-21T12:39:23.342 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.342 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.342 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:39:23.342 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.348662+0000", 2024-09-21T12:39:23.342 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6035000000000001e-05 2024-09-21T12:39:23.343 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.343 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.343 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:39:23.343 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.348729+0000", 2024-09-21T12:39:23.343 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7012000000000001e-05 2024-09-21T12:39:23.343 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.343 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:39:23.343 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.343 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.343 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.343 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "MOSDPGPush(11.c 462/456 [PushOp(11:363e308b:::benchmark_data_smithi114_187622_object124173:head, version: 438'7903, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(11:363e308b:::benchmark_data_smithi114_187622_object124173:head@438'7903, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T12:39:23.343 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:38:24.837129+0000", 2024-09-21T12:39:23.344 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 58.448352894000003, 2024-09-21T12:39:23.344 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.087017722000000006, 2024-09-21T12:39:23.344 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:39:23.344 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:39:23.344 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:39:23.344 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:39:23.344 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.344 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:39:23.344 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.837129+0000", 2024-09-21T12:39:23.344 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.344 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.344 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.345 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:39:23.345 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.837129+0000", 2024-09-21T12:39:23.345 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.345 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.345 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.345 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:39:23.345 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.837130+0000", 2024-09-21T12:39:23.345 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.47e-07 2024-09-21T12:39:23.345 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.345 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.345 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:39:23.346 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.837156+0000", 2024-09-21T12:39:23.346 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.6404999999999999e-05 2024-09-21T12:39:23.346 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.346 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.346 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:39:23.346 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.837176+0000", 2024-09-21T12:39:23.346 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0384e-05 2024-09-21T12:39:23.346 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.346 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.346 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:39:23.346 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.837204+0000", 2024-09-21T12:39:23.346 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.7412000000000001e-05 2024-09-21T12:39:23.347 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.347 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.347 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:39:23.347 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.837311+0000", 2024-09-21T12:39:23.347 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000106877 2024-09-21T12:39:23.347 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.347 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.347 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:39:23.347 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.837325+0000", 2024-09-21T12:39:23.347 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3949e-05 2024-09-21T12:39:23.347 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.347 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.348 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:39:23.348 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.924146+0000", 2024-09-21T12:39:23.348 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.086821748000000004 2024-09-21T12:39:23.348 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.348 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:39:23.348 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.348 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.348 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.348 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.26147.0:220766 11.d e462/456 11:bc65a9b9:::benchmark_data_smithi114_187622_object82645:head v 462'13435, mlcod=462'13432)", 2024-09-21T12:39:23.348 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:38:24.837355+0000", 2024-09-21T12:39:23.348 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 58.448126997999999, 2024-09-21T12:39:23.348 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.085993523000000002, 2024-09-21T12:39:23.349 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:39:23.349 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:39:23.349 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:23.349 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:39:23.349 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.349 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:39:23.349 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.837355+0000", 2024-09-21T12:39:23.349 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.349 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.349 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.349 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:39:23.349 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.837355+0000", 2024-09-21T12:39:23.349 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.350 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.350 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.350 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:39:23.350 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.837355+0000", 2024-09-21T12:39:23.350 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8299999999999996e-07 2024-09-21T12:39:23.350 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.350 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.350 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:39:23.350 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.837361+0000", 2024-09-21T12:39:23.350 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6659999999999996e-06 2024-09-21T12:39:23.350 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.350 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.351 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:39:23.351 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.837371+0000", 2024-09-21T12:39:23.351 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0144e-05 2024-09-21T12:39:23.351 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.351 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.351 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:39:23.351 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.837377+0000", 2024-09-21T12:39:23.351 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0240000000000003e-06 2024-09-21T12:39:23.351 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.351 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.351 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:39:23.352 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.837437+0000", 2024-09-21T12:39:23.352 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9517e-05 2024-09-21T12:39:23.352 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.352 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.352 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:39:23.352 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.837454+0000", 2024-09-21T12:39:23.352 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.736e-05 2024-09-21T12:39:23.352 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.352 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.352 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:39:23.352 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.923312+0000", 2024-09-21T12:39:23.353 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.085858379999999998 2024-09-21T12:39:23.353 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.353 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.353 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:39:23.353 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.923348+0000", 2024-09-21T12:39:23.353 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5648999999999997e-05 2024-09-21T12:39:23.353 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.353 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:39:23.353 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.353 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.353 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.353 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.26147.0:220769 11.f e462/455 11:f85b3579:::benchmark_data_smithi114_187622_object82648:head v 462'14002, mlcod=462'14000)", 2024-09-21T12:39:23.354 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:38:24.837802+0000", 2024-09-21T12:39:23.354 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 58.447679229000002, 2024-09-21T12:39:23.354 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.091694224000000005, 2024-09-21T12:39:23.354 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:39:23.354 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:39:23.354 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:23.354 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:39:23.354 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.354 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:39:23.354 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.837802+0000", 2024-09-21T12:39:23.354 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.355 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.355 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.355 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:39:23.355 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.837802+0000", 2024-09-21T12:39:23.355 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.355 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.355 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.355 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:39:23.355 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.837804+0000", 2024-09-21T12:39:23.355 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.871e-06 2024-09-21T12:39:23.355 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.356 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.356 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:39:23.356 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.837809+0000", 2024-09-21T12:39:23.356 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3699999999999997e-06 2024-09-21T12:39:23.356 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.356 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.356 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:39:23.356 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.837817+0000", 2024-09-21T12:39:23.356 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6710000000000005e-06 2024-09-21T12:39:23.356 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.356 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.356 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:39:23.357 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.837823+0000", 2024-09-21T12:39:23.357 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5180000000000004e-06 2024-09-21T12:39:23.357 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.357 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.357 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:39:23.357 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.923464+0000", 2024-09-21T12:39:23.357 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.085641445999999996 2024-09-21T12:39:23.357 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.357 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.357 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:39:23.357 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.923485+0000", 2024-09-21T12:39:23.358 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1024000000000001e-05 2024-09-21T12:39:23.358 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.358 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.358 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:39:23.358 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.929458+0000", 2024-09-21T12:39:23.358 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0059723149999999997 2024-09-21T12:39:23.358 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.358 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.358 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:39:23.358 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.929497+0000", 2024-09-21T12:39:23.358 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.9008999999999999e-05 2024-09-21T12:39:23.359 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.359 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:39:23.359 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.359 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.359 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.359 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "pg_backfill(progress 11.c e 462/462 lb 11:363e0256:::benchmark_data_smithi114_187622_object92972:head)", 2024-09-21T12:39:23.359 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:38:24.837922+0000", 2024-09-21T12:39:23.359 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 58.447559349999999, 2024-09-21T12:39:23.359 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.094369772000000005, 2024-09-21T12:39:23.359 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:39:23.359 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:39:23.359 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:39:23.360 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:39:23.360 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.360 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:39:23.360 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.837922+0000", 2024-09-21T12:39:23.360 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.360 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.360 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.360 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:39:23.360 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.837922+0000", 2024-09-21T12:39:23.360 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.360 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.360 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.360 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:39:23.361 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.837923+0000", 2024-09-21T12:39:23.361 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4000000000000001e-07 2024-09-21T12:39:23.361 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.361 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.361 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:39:23.361 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.837929+0000", 2024-09-21T12:39:23.361 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5370000000000003e-06 2024-09-21T12:39:23.361 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.361 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.361 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:39:23.361 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.923461+0000", 2024-09-21T12:39:23.362 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.085531880000000005 2024-09-21T12:39:23.362 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.362 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.362 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:39:23.362 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.923473+0000", 2024-09-21T12:39:23.362 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2068e-05 2024-09-21T12:39:23.362 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.362 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.362 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:39:23.362 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.932255+0000", 2024-09-21T12:39:23.362 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0087813349999999995 2024-09-21T12:39:23.362 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.363 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.363 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:39:23.363 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.932268+0000", 2024-09-21T12:39:23.363 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3725e-05 2024-09-21T12:39:23.363 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.363 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.363 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:39:23.363 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.932292+0000", 2024-09-21T12:39:23.363 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3587e-05 2024-09-21T12:39:23.363 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.363 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:39:23.363 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.363 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.364 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.364 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.26147.0:220768 11.6 e462/456 11:608fba61:::benchmark_data_smithi114_187622_object82647:head v 462'13693, mlcod=462'13689)", 2024-09-21T12:39:23.364 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:38:24.840180+0000", 2024-09-21T12:39:23.364 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 58.445301252999997, 2024-09-21T12:39:23.364 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.087888645000000001, 2024-09-21T12:39:23.364 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:39:23.364 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:39:23.364 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:23.364 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:39:23.364 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.364 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:39:23.364 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.840180+0000", 2024-09-21T12:39:23.364 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.365 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.365 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.365 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:39:23.365 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.840180+0000", 2024-09-21T12:39:23.365 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.365 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.365 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.365 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:39:23.365 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.840181+0000", 2024-09-21T12:39:23.365 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0890000000000001e-06 2024-09-21T12:39:23.365 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.365 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.365 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:39:23.366 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.840189+0000", 2024-09-21T12:39:23.366 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1069999999999999e-06 2024-09-21T12:39:23.366 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.366 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.366 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:39:23.366 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.923143+0000", 2024-09-21T12:39:23.366 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.082954779000000006 2024-09-21T12:39:23.366 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.366 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.366 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:39:23.366 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.923164+0000", 2024-09-21T12:39:23.366 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1033999999999999e-05 2024-09-21T12:39:23.366 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.367 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.367 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:39:23.367 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.923251+0000", 2024-09-21T12:39:23.367 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6589999999999996e-05 2024-09-21T12:39:23.367 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.367 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.367 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:39:23.367 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.923275+0000", 2024-09-21T12:39:23.367 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3699000000000001e-05 2024-09-21T12:39:23.367 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.367 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.367 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:39:23.367 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.927976+0000", 2024-09-21T12:39:23.367 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0047010999999999997 2024-09-21T12:39:23.368 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.368 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.368 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:39:23.368 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.928069+0000", 2024-09-21T12:39:23.368 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.3246999999999997e-05 2024-09-21T12:39:23.368 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.368 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:39:23.368 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.368 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.368 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.368 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "pg_backfill(progress 11.d e 462/462 lb 11:bc3314ac:::benchmark_data_smithi114_187622_object121293:head)", 2024-09-21T12:39:23.368 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:38:24.841686+0000", 2024-09-21T12:39:23.368 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 58.443796091000003, 2024-09-21T12:39:23.368 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.081744204000000001, 2024-09-21T12:39:23.369 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:39:23.369 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:39:23.369 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:39:23.369 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:39:23.369 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.369 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:39:23.369 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.841686+0000", 2024-09-21T12:39:23.369 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.369 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.369 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.369 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:39:23.369 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.841686+0000", 2024-09-21T12:39:23.369 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.370 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.370 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.370 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:39:23.370 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.841686+0000", 2024-09-21T12:39:23.370 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6200000000000006e-07 2024-09-21T12:39:23.370 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.370 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.370 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:39:23.370 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.841695+0000", 2024-09-21T12:39:23.370 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.259e-06 2024-09-21T12:39:23.370 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.370 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.371 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:39:23.371 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.923309+0000", 2024-09-21T12:39:23.371 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.081613904000000001 2024-09-21T12:39:23.371 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.371 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.371 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:39:23.371 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.923318+0000", 2024-09-21T12:39:23.371 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.3929999999999997e-06 2024-09-21T12:39:23.371 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.371 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.371 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:39:23.372 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.923377+0000", 2024-09-21T12:39:23.372 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9308999999999997e-05 2024-09-21T12:39:23.372 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.372 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.372 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:39:23.372 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.923391+0000", 2024-09-21T12:39:23.372 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3939e-05 2024-09-21T12:39:23.372 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.372 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.372 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:39:23.372 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:24.923430+0000", 2024-09-21T12:39:23.372 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8438e-05 2024-09-21T12:39:23.373 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.373 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:39:23.373 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.373 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.373 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.373 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.26147.0:221327 11.c e462/456 11:30baa3d2:::benchmark_data_smithi114_187622_object83206:head v 462'14112, mlcod=462'14108)", 2024-09-21T12:39:23.373 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:38:25.400872+0000", 2024-09-21T12:39:23.373 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 57.884609722, 2024-09-21T12:39:23.373 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.24199574500000001, 2024-09-21T12:39:23.373 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:39:23.373 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:39:23.373 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:23.373 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:39:23.374 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.374 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:39:23.374 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.400872+0000", 2024-09-21T12:39:23.374 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.374 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.374 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.374 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:39:23.374 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.400872+0000", 2024-09-21T12:39:23.374 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.374 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.374 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.374 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:39:23.375 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.400872+0000", 2024-09-21T12:39:23.375 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.6399999999999998e-07 2024-09-21T12:39:23.375 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.375 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.375 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:39:23.375 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.400877+0000", 2024-09-21T12:39:23.375 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3170000000000003e-06 2024-09-21T12:39:23.375 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.375 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.375 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:39:23.375 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.400885+0000", 2024-09-21T12:39:23.375 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.0220000000000004e-06 2024-09-21T12:39:23.375 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.376 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.376 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:39:23.376 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.400890+0000", 2024-09-21T12:39:23.376 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3449999999999996e-06 2024-09-21T12:39:23.376 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.376 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.376 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:39:23.376 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.401620+0000", 2024-09-21T12:39:23.376 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00072971500000000003 2024-09-21T12:39:23.376 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.376 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.376 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:39:23.377 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.401636+0000", 2024-09-21T12:39:23.377 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6663999999999998e-05 2024-09-21T12:39:23.377 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.377 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.377 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:39:23.377 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.642787+0000", 2024-09-21T12:39:23.377 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.241150587 2024-09-21T12:39:23.377 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.377 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.377 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:39:23.377 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.642868+0000", 2024-09-21T12:39:23.377 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.0730999999999997e-05 2024-09-21T12:39:23.378 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.378 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:39:23.378 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.378 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.378 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.378 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.26147.0:221335 11.d e462/456 11:bf5e9ccf:::benchmark_data_smithi114_187622_object83214:head v 462'13470, mlcod=462'13468)", 2024-09-21T12:39:23.378 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:38:25.402424+0000", 2024-09-21T12:39:23.378 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 57.883057338, 2024-09-21T12:39:23.378 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.237281769, 2024-09-21T12:39:23.378 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:39:23.378 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:39:23.378 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:23.379 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:39:23.379 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.379 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:39:23.379 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.402424+0000", 2024-09-21T12:39:23.379 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.379 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.379 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.379 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:39:23.379 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.402424+0000", 2024-09-21T12:39:23.379 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.379 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.379 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.379 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:39:23.380 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.402425+0000", 2024-09-21T12:39:23.380 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.3500000000000005e-07 2024-09-21T12:39:23.380 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.380 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.380 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:39:23.380 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.402431+0000", 2024-09-21T12:39:23.380 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4909999999999996e-06 2024-09-21T12:39:23.380 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.380 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.380 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:39:23.380 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.402441+0000", 2024-09-21T12:39:23.380 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0152999999999999e-05 2024-09-21T12:39:23.380 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.381 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.381 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:39:23.381 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.402447+0000", 2024-09-21T12:39:23.381 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1310000000000003e-06 2024-09-21T12:39:23.381 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.381 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.381 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:39:23.381 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.402507+0000", 2024-09-21T12:39:23.381 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0349000000000001e-05 2024-09-21T12:39:23.381 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.381 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.381 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:39:23.382 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.402525+0000", 2024-09-21T12:39:23.382 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7374999999999999e-05 2024-09-21T12:39:23.382 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.382 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.382 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:39:23.382 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.639667+0000", 2024-09-21T12:39:23.382 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.23714231999999999 2024-09-21T12:39:23.382 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.382 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.382 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:39:23.382 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.639706+0000", 2024-09-21T12:39:23.382 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.9115000000000001e-05 2024-09-21T12:39:23.383 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.383 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:39:23.383 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.383 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.383 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.383 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.26147.0:221331 11.c e462/456 11:3bdc3e7d:::benchmark_data_smithi114_187622_object83210:head v 462'14113, mlcod=462'14110)", 2024-09-21T12:39:23.383 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:38:25.403990+0000", 2024-09-21T12:39:23.383 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 57.881491994999998, 2024-09-21T12:39:23.383 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.23900700799999999, 2024-09-21T12:39:23.383 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:39:23.383 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:39:23.383 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:23.384 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:39:23.384 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.384 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:39:23.384 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.403990+0000", 2024-09-21T12:39:23.384 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.384 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.384 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.384 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:39:23.384 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.403990+0000", 2024-09-21T12:39:23.384 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.384 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.384 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.384 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:39:23.385 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.403990+0000", 2024-09-21T12:39:23.385 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4499999999999996e-07 2024-09-21T12:39:23.385 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.385 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.385 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:39:23.385 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.403997+0000", 2024-09-21T12:39:23.385 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1580000000000002e-06 2024-09-21T12:39:23.385 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.385 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.385 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:39:23.385 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.404006+0000", 2024-09-21T12:39:23.385 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7909999999999999e-06 2024-09-21T12:39:23.386 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.386 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.386 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:39:23.386 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.404012+0000", 2024-09-21T12:39:23.386 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7919999999999998e-06 2024-09-21T12:39:23.386 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.386 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.386 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:39:23.386 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.404081+0000", 2024-09-21T12:39:23.386 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8527000000000003e-05 2024-09-21T12:39:23.386 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.386 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.387 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:39:23.387 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.404099+0000", 2024-09-21T12:39:23.387 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7869000000000001e-05 2024-09-21T12:39:23.387 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.387 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.387 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:39:23.387 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.642914+0000", 2024-09-21T12:39:23.387 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.23881539199999999 2024-09-21T12:39:23.387 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.388 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.388 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:39:23.388 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.642997+0000", 2024-09-21T12:39:23.388 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.2733999999999998e-05 2024-09-21T12:39:23.388 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.388 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:39:23.388 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.388 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.388 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.388 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.26147.0:221340 11.4 e462/455 11:293cf4f3:::benchmark_data_smithi114_187622_object83219:head v 462'13860, mlcod=462'13858)", 2024-09-21T12:39:23.388 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:38:25.404727+0000", 2024-09-21T12:39:23.389 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 57.880754965999998, 2024-09-21T12:39:23.389 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.246590103, 2024-09-21T12:39:23.389 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:39:23.389 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:39:23.389 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:23.389 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:39:23.389 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.389 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:39:23.389 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.404727+0000", 2024-09-21T12:39:23.389 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.389 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.389 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.390 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:39:23.390 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.404727+0000", 2024-09-21T12:39:23.390 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.390 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.390 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.390 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:39:23.390 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.404727+0000", 2024-09-21T12:39:23.390 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8000000000000005e-07 2024-09-21T12:39:23.390 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.390 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.390 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:39:23.391 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.404737+0000", 2024-09-21T12:39:23.391 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0780000000000002e-06 2024-09-21T12:39:23.391 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.391 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.391 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:39:23.391 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.404747+0000", 2024-09-21T12:39:23.391 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0514e-05 2024-09-21T12:39:23.391 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.391 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.391 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:39:23.391 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.404753+0000", 2024-09-21T12:39:23.392 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.195e-06 2024-09-21T12:39:23.392 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.392 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.392 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:39:23.392 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.405644+0000", 2024-09-21T12:39:23.392 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000891146 2024-09-21T12:39:23.392 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.392 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.392 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:39:23.392 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.405663+0000", 2024-09-21T12:39:23.392 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9057000000000001e-05 2024-09-21T12:39:23.392 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.392 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.393 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:39:23.393 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.651238+0000", 2024-09-21T12:39:23.393 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.24557435799999999 2024-09-21T12:39:23.393 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.393 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.393 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:39:23.393 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.651317+0000", 2024-09-21T12:39:23.393 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8974999999999995e-05 2024-09-21T12:39:23.393 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.393 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:39:23.393 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.394 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.394 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.394 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.26147.0:221341 11.c e462/456 11:3cec178a:::benchmark_data_smithi114_187622_object83220:head v 462'14114, mlcod=462'14110)", 2024-09-21T12:39:23.394 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:38:25.406104+0000", 2024-09-21T12:39:23.394 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 57.879377388999998, 2024-09-21T12:39:23.394 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.24555158299999999, 2024-09-21T12:39:23.394 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:39:23.394 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:39:23.394 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:23.394 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:39:23.394 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.394 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:39:23.394 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.406104+0000", 2024-09-21T12:39:23.395 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.395 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.395 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.395 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:39:23.395 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.406104+0000", 2024-09-21T12:39:23.395 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.395 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.395 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.395 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:39:23.395 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.406105+0000", 2024-09-21T12:39:23.395 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0800000000000004e-07 2024-09-21T12:39:23.395 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.395 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.396 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:39:23.396 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.406110+0000", 2024-09-21T12:39:23.396 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6720000000000003e-06 2024-09-21T12:39:23.396 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.396 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.396 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:39:23.396 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.406120+0000", 2024-09-21T12:39:23.396 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0034000000000001e-05 2024-09-21T12:39:23.396 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.396 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.396 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:39:23.396 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.406126+0000", 2024-09-21T12:39:23.397 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3130000000000004e-06 2024-09-21T12:39:23.397 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.397 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.397 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:39:23.397 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.639325+0000", 2024-09-21T12:39:23.397 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.23319917200000001 2024-09-21T12:39:23.397 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.397 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.397 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:39:23.397 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.639350+0000", 2024-09-21T12:39:23.397 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5332000000000001e-05 2024-09-21T12:39:23.397 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.397 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.398 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:39:23.398 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.651575+0000", 2024-09-21T12:39:23.398 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.01222432 2024-09-21T12:39:23.398 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.398 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.398 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:39:23.398 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.651656+0000", 2024-09-21T12:39:23.398 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1032000000000006e-05 2024-09-21T12:39:23.398 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.398 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:39:23.398 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.398 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.398 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.399 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.26147.0:221342 11.f e462/455 11:fbf7b516:::benchmark_data_smithi114_187622_object83221:head v 462'14039, mlcod=462'14037)", 2024-09-21T12:39:23.399 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:38:25.406284+0000", 2024-09-21T12:39:23.399 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 57.879197417999997, 2024-09-21T12:39:23.399 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.233098098, 2024-09-21T12:39:23.399 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:39:23.399 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:39:23.399 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:23.399 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:39:23.399 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.399 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:39:23.399 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.406284+0000", 2024-09-21T12:39:23.399 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.399 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.400 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.400 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:39:23.400 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.406284+0000", 2024-09-21T12:39:23.400 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.400 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.400 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.400 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:39:23.400 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.406285+0000", 2024-09-21T12:39:23.400 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5300000000000004e-07 2024-09-21T12:39:23.400 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.400 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.400 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:39:23.400 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.406290+0000", 2024-09-21T12:39:23.400 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9320000000000004e-06 2024-09-21T12:39:23.401 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.401 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.401 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:39:23.401 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.406299+0000", 2024-09-21T12:39:23.401 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.4590000000000004e-06 2024-09-21T12:39:23.401 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.401 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.401 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:39:23.401 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.406305+0000", 2024-09-21T12:39:23.401 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7939999999999999e-06 2024-09-21T12:39:23.401 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.401 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.402 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:39:23.402 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.406370+0000", 2024-09-21T12:39:23.402 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4850000000000004e-05 2024-09-21T12:39:23.402 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.402 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.402 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:39:23.402 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.406387+0000", 2024-09-21T12:39:23.402 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7609e-05 2024-09-21T12:39:23.402 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.402 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.403 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:39:23.403 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.639332+0000", 2024-09-21T12:39:23.403 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.23294486 2024-09-21T12:39:23.403 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.403 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.403 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:39:23.403 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.639382+0000", 2024-09-21T12:39:23.403 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9941000000000002e-05 2024-09-21T12:39:23.403 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.403 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:39:23.403 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.404 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.404 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.404 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "MOSDPGPush(11.d 462/456 [PushOp(11:bc9e879e:::benchmark_data_smithi114_187622_object135866:head, version: 443'8285, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(11:bc9e879e:::benchmark_data_smithi114_187622_object135866:head@443'8285, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T12:39:23.404 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:38:25.408384+0000", 2024-09-21T12:39:23.404 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 57.877097904999999, 2024-09-21T12:39:23.404 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.23121749899999999, 2024-09-21T12:39:23.404 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:39:23.404 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:39:23.404 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:39:23.404 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:39:23.404 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.404 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:39:23.405 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.408384+0000", 2024-09-21T12:39:23.405 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.405 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.405 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.405 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:39:23.405 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.408384+0000", 2024-09-21T12:39:23.405 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.405 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.405 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.405 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:39:23.405 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.408393+0000", 2024-09-21T12:39:23.405 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6569999999999992e-06 2024-09-21T12:39:23.406 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.406 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.406 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:39:23.406 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.408435+0000", 2024-09-21T12:39:23.406 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1813000000000001e-05 2024-09-21T12:39:23.406 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.406 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.406 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:39:23.406 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.408457+0000", 2024-09-21T12:39:23.406 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1849e-05 2024-09-21T12:39:23.406 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.407 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.407 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:39:23.407 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.408474+0000", 2024-09-21T12:39:23.407 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6707000000000001e-05 2024-09-21T12:39:23.407 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.407 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.407 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:39:23.407 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.639360+0000", 2024-09-21T12:39:23.407 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.23088613799999999 2024-09-21T12:39:23.407 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.407 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.407 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:39:23.408 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.639373+0000", 2024-09-21T12:39:23.408 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2873e-05 2024-09-21T12:39:23.408 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.408 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.408 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:39:23.408 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.639601+0000", 2024-09-21T12:39:23.408 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000228462 2024-09-21T12:39:23.408 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.408 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:39:23.408 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.409 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.409 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.409 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "pg_backfill(progress 11.d e 462/462 lb 11:bc9ca29f:::benchmark_data_smithi114_187622_object100075:head)", 2024-09-21T12:39:23.409 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:38:25.408503+0000", 2024-09-21T12:39:23.409 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 57.876978309000002, 2024-09-21T12:39:23.409 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.231160226, 2024-09-21T12:39:23.409 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:39:23.409 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:39:23.409 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-21T12:39:23.409 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:39:23.409 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.409 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:39:23.409 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.408503+0000", 2024-09-21T12:39:23.410 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.410 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.410 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.410 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:39:23.410 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.408503+0000", 2024-09-21T12:39:23.410 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.410 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.410 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.410 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:39:23.410 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.408504+0000", 2024-09-21T12:39:23.410 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5999999999999999e-07 2024-09-21T12:39:23.411 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.411 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.411 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:39:23.411 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.408510+0000", 2024-09-21T12:39:23.411 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8690000000000001e-06 2024-09-21T12:39:23.411 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.411 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.411 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:39:23.411 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.408517+0000", 2024-09-21T12:39:23.411 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9809999999999997e-06 2024-09-21T12:39:23.411 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.411 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.412 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:39:23.412 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.639303+0000", 2024-09-21T12:39:23.412 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.23078696300000001 2024-09-21T12:39:23.412 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.412 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.412 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:39:23.412 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.639621+0000", 2024-09-21T12:39:23.412 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00031778999999999998 2024-09-21T12:39:23.412 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.412 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.412 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:39:23.413 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.639631+0000", 2024-09-21T12:39:23.413 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.3340000000000008e-06 2024-09-21T12:39:23.413 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.413 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.413 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:39:23.413 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.639664+0000", 2024-09-21T12:39:23.413 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2929000000000003e-05 2024-09-21T12:39:23.413 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.413 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:39:23.413 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.413 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.413 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.413 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.26147.0:221343 11.c e462/456 11:3eb5fd8b:::benchmark_data_smithi114_187622_object83222:head v 462'14115, mlcod=462'14110)", 2024-09-21T12:39:23.414 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:38:25.409434+0000", 2024-09-21T12:39:23.414 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 57.876047229999998, 2024-09-21T12:39:23.414 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.24230114899999999, 2024-09-21T12:39:23.414 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:39:23.414 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:39:23.414 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:23.414 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:39:23.414 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.414 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:39:23.414 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.409434+0000", 2024-09-21T12:39:23.414 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.415 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.415 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.415 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:39:23.415 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.409434+0000", 2024-09-21T12:39:23.415 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.415 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.415 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.415 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:39:23.415 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.409435+0000", 2024-09-21T12:39:23.415 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4700000000000001e-07 2024-09-21T12:39:23.415 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.415 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.416 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:39:23.416 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.409440+0000", 2024-09-21T12:39:23.416 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1939999999999997e-06 2024-09-21T12:39:23.416 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.416 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.416 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:39:23.416 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.639299+0000", 2024-09-21T12:39:23.416 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.22985879000000001 2024-09-21T12:39:23.416 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.416 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.416 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:39:23.416 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.639310+0000", 2024-09-21T12:39:23.417 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1172000000000001e-05 2024-09-21T12:39:23.417 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.417 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.417 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:39:23.417 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.641296+0000", 2024-09-21T12:39:23.417 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0019855379999999998 2024-09-21T12:39:23.417 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.417 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.417 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:39:23.417 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.641314+0000", 2024-09-21T12:39:23.417 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8014999999999999e-05 2024-09-21T12:39:23.417 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.418 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.418 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:39:23.418 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.651692+0000", 2024-09-21T12:39:23.418 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010378469 2024-09-21T12:39:23.418 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.418 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.418 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:39:23.418 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.651736+0000", 2024-09-21T12:39:23.418 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3223999999999998e-05 2024-09-21T12:39:23.418 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.418 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:39:23.419 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.419 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.419 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.26147.0:221369 11.6 e462/456 11:69679ad3:::benchmark_data_smithi114_187622_object83248:head v 462'13725, mlcod=462'13723)", 2024-09-21T12:39:23.419 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-21T12:38:25.433660+0000", 2024-09-21T12:39:23.419 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 57.851821843000003, 2024-09-21T12:39:23.419 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.20931754999999999, 2024-09-21T12:39:23.419 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-21T12:39:23.419 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-21T12:39:23.419 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T12:39:23.419 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-21T12:39:23.419 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.419 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-21T12:39:23.420 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.433660+0000", 2024-09-21T12:39:23.420 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.420 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.420 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.420 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-21T12:39:23.420 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.433660+0000", 2024-09-21T12:39:23.420 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-21T12:39:23.420 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.420 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.420 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-21T12:39:23.420 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.433661+0000", 2024-09-21T12:39:23.420 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3960000000000001e-06 2024-09-21T12:39:23.421 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.421 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.421 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-21T12:39:23.421 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.433669+0000", 2024-09-21T12:39:23.421 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1089999999999998e-06 2024-09-21T12:39:23.421 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.421 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.421 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-21T12:39:23.421 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.639328+0000", 2024-09-21T12:39:23.421 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.205658797 2024-09-21T12:39:23.421 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.422 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.422 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-21T12:39:23.422 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.639338+0000", 2024-09-21T12:39:23.422 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.5359999999999998e-06 2024-09-21T12:39:23.422 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.422 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.422 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-21T12:39:23.422 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.639425+0000", 2024-09-21T12:39:23.422 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.7836000000000004e-05 2024-09-21T12:39:23.422 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.422 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.423 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-21T12:39:23.423 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.639445+0000", 2024-09-21T12:39:23.423 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9085000000000001e-05 2024-09-21T12:39:23.423 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.423 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.423 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-21T12:39:23.423 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.642942+0000", 2024-09-21T12:39:23.423 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0034969839999999999 2024-09-21T12:39:23.423 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-21T12:39:23.423 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-21T12:39:23.423 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-21T12:39:23.423 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-21T12:38:25.642977+0000", 2024-09-21T12:39:23.424 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5806999999999998e-05 2024-09-21T12:39:23.424 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.424 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:39:23.424 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.424 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-21T12:39:23.424 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-21T12:39:23.424 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-21T12:39:23.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:23 smithi183 ceph-mon[33277]: 5.2 scrub starts 2024-09-21T12:39:23.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:23 smithi183 ceph-mon[33277]: 5.2 scrub ok 2024-09-21T12:39:23.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:23 smithi183 ceph-mon[33277]: osdmap e488: 8 total, 8 up, 8 in 2024-09-21T12:39:23.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:23 smithi183 ceph-mon[33277]: pgmap v980: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T12:39:23.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:23 smithi183 ceph-mon[33277]: 4.9 scrub starts 2024-09-21T12:39:23.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:23 smithi183 ceph-mon[33277]: 4.9 scrub ok 2024-09-21T12:39:23.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:23 smithi183 ceph-mon[33277]: 6.1c scrub starts 2024-09-21T12:39:23.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:23 smithi183 ceph-mon[33277]: 6.1c scrub ok 2024-09-21T12:39:23.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:23 smithi114 ceph-mon[30324]: 5.2 scrub starts 2024-09-21T12:39:23.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:23 smithi114 ceph-mon[30324]: 5.2 scrub ok 2024-09-21T12:39:23.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:23 smithi114 ceph-mon[30324]: osdmap e488: 8 total, 8 up, 8 in 2024-09-21T12:39:23.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:23 smithi114 ceph-mon[30324]: pgmap v980: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T12:39:23.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:23 smithi114 ceph-mon[30324]: 4.9 scrub starts 2024-09-21T12:39:23.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:23 smithi114 ceph-mon[30324]: 4.9 scrub ok 2024-09-21T12:39:23.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:23 smithi114 ceph-mon[30324]: 6.1c scrub starts 2024-09-21T12:39:23.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:23 smithi114 ceph-mon[30324]: 6.1c scrub ok 2024-09-21T12:39:23.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:23 smithi114 ceph-mon[40867]: 5.2 scrub starts 2024-09-21T12:39:23.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:23 smithi114 ceph-mon[40867]: 5.2 scrub ok 2024-09-21T12:39:23.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:23 smithi114 ceph-mon[40867]: osdmap e488: 8 total, 8 up, 8 in 2024-09-21T12:39:23.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:23 smithi114 ceph-mon[40867]: pgmap v980: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T12:39:23.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:23 smithi114 ceph-mon[40867]: 4.9 scrub starts 2024-09-21T12:39:23.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:23 smithi114 ceph-mon[40867]: 4.9 scrub ok 2024-09-21T12:39:23.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:23 smithi114 ceph-mon[40867]: 6.1c scrub starts 2024-09-21T12:39:23.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:23 smithi114 ceph-mon[40867]: 6.1c scrub ok 2024-09-21T12:39:23.535 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:39:24.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:24 smithi183 ceph-mon[33277]: 4.1c scrub starts 2024-09-21T12:39:24.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:24 smithi183 ceph-mon[33277]: 4.1c scrub ok 2024-09-21T12:39:24.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:24 smithi114 ceph-mon[30324]: 4.1c scrub starts 2024-09-21T12:39:24.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:24 smithi114 ceph-mon[30324]: 4.1c scrub ok 2024-09-21T12:39:24.504 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:24 smithi114 ceph-mon[40867]: 4.1c scrub starts 2024-09-21T12:39:24.504 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:24 smithi114 ceph-mon[40867]: 4.1c scrub ok 2024-09-21T12:39:24.733 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T12:39:24.922 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:39:24.924 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:39:25.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:25 smithi114 ceph-mon[30324]: pgmap v981: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 573 B/s rd, 0 op/s 2024-09-21T12:39:25.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:25 smithi114 ceph-mon[30324]: 1.1c scrub starts 2024-09-21T12:39:25.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:25 smithi114 ceph-mon[30324]: 1.1c scrub ok 2024-09-21T12:39:25.376 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:25 smithi114 ceph-mon[40867]: pgmap v981: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 573 B/s rd, 0 op/s 2024-09-21T12:39:25.376 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:25 smithi114 ceph-mon[40867]: 1.1c scrub starts 2024-09-21T12:39:25.376 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:25 smithi114 ceph-mon[40867]: 1.1c scrub ok 2024-09-21T12:39:25.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:25 smithi183 ceph-mon[33277]: pgmap v981: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 573 B/s rd, 0 op/s 2024-09-21T12:39:25.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:25 smithi183 ceph-mon[33277]: 1.1c scrub starts 2024-09-21T12:39:25.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:25 smithi183 ceph-mon[33277]: 1.1c scrub ok 2024-09-21T12:39:25.847 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":981,"stamp":"2024-09-21T12:39:24.311266+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":1327,"num_read_kb":1079,"num_write":722,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":504,"num_bytes_recovered":273047,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3263,"ondisk_log_size":3263,"up":486,"acting":486,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":516,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6068204,"kb_used_data":82180,"kb_used_omap":18,"kb_used_meta":5985837,"kb_avail":743893012,"statfs":{"total":767960285184,"available":761746444288,"internally_reserved":0,"allocated":84152320,"data_stored":77627365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19199,"internal_metadata":6129497345},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":62,"apply_latency_ms":62,"commit_latency_ns":62000000,"apply_latency_ns":62000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19857408,"num_objects":-303,"num_object_clones":0,"num_object_copies":-909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-303,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":-275935,"num_write_kb":-8839553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-22413,"num_bytes_recovered":-1468530712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.147290"},"pg_stats":[{"pgid":"1.1d","version":"76'79","reported_seq":606,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.385134+0000","last_change":"2024-09-21T12:37:16.858542+0000","last_active":"2024-09-21T12:39:19.385134+0000","last_peered":"2024-09-21T12:39:19.385134+0000","last_clean":"2024-09-21T12:39:19.385134+0000","last_became_active":"2024-09-21T12:32:38.426383+0000","last_became_peered":"2024-09-21T12:32:38.426383+0000","last_unstale":"2024-09-21T12:39:19.385134+0000","last_undegraded":"2024-09-21T12:39:19.385134+0000","last_fullsized":"2024-09-21T12:39:19.385134+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:37:16.858472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:37:16.858472+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:14.113783+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":922,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.394057+0000","last_change":"2024-09-21T12:38:07.168168+0000","last_active":"2024-09-21T12:39:22.394057+0000","last_peered":"2024-09-21T12:39:22.394057+0000","last_clean":"2024-09-21T12:39:22.394057+0000","last_became_active":"2024-09-21T12:38:07.167857+0000","last_became_peered":"2024-09-21T12:38:07.167857+0000","last_unstale":"2024-09-21T12:39:22.394057+0000","last_undegraded":"2024-09-21T12:39:22.394057+0000","last_fullsized":"2024-09-21T12:39:22.394057+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:29.086830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:29.086830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:02.550603+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"276'15","reported_seq":314,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.383808+0000","last_change":"2024-09-21T12:39:16.266371+0000","last_active":"2024-09-21T12:39:19.383808+0000","last_peered":"2024-09-21T12:39:19.383808+0000","last_clean":"2024-09-21T12:39:19.383808+0000","last_became_active":"2024-09-21T12:32:38.419081+0000","last_became_peered":"2024-09-21T12:32:38.419081+0000","last_unstale":"2024-09-21T12:39:19.383808+0000","last_undegraded":"2024-09-21T12:39:19.383808+0000","last_fullsized":"2024-09-21T12:39:19.383808+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:39:16.266285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:16.266285+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:30.189485+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":926,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.384837+0000","last_change":"2024-09-21T12:39:17.175795+0000","last_active":"2024-09-21T12:39:19.384837+0000","last_peered":"2024-09-21T12:39:19.384837+0000","last_clean":"2024-09-21T12:39:19.384837+0000","last_became_active":"2024-09-21T12:36:00.136673+0000","last_became_peered":"2024-09-21T12:36:00.136673+0000","last_unstale":"2024-09-21T12:39:19.384837+0000","last_undegraded":"2024-09-21T12:39:19.384837+0000","last_fullsized":"2024-09-21T12:39:19.384837+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:17.175740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:17.175740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:46.137502+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":304,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.384309+0000","last_change":"2024-09-21T12:36:44.729701+0000","last_active":"2024-09-21T12:39:19.384309+0000","last_peered":"2024-09-21T12:39:19.384309+0000","last_clean":"2024-09-21T12:39:19.384309+0000","last_became_active":"2024-09-21T12:32:38.425588+0000","last_became_peered":"2024-09-21T12:32:38.425588+0000","last_unstale":"2024-09-21T12:39:19.384309+0000","last_undegraded":"2024-09-21T12:39:19.384309+0000","last_fullsized":"2024-09-21T12:39:19.384309+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:44.729655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:44.729655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:13.322426+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"76'79","reported_seq":165,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:20.299531+0000","last_change":"2024-09-21T12:37:19.644089+0000","last_active":"2024-09-21T12:39:20.299531+0000","last_peered":"2024-09-21T12:39:20.299531+0000","last_clean":"2024-09-21T12:39:20.299531+0000","last_became_active":"2024-09-21T12:36:00.154209+0000","last_became_peered":"2024-09-21T12:36:00.154209+0000","last_unstale":"2024-09-21T12:39:20.299531+0000","last_undegraded":"2024-09-21T12:39:20.299531+0000","last_fullsized":"2024-09-21T12:39:20.299531+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:37:19.644030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:37:19.644030+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":908,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.326465+0000","last_change":"2024-09-21T12:38:04.831151+0000","last_active":"2024-09-21T12:39:22.326465+0000","last_peered":"2024-09-21T12:39:22.326465+0000","last_clean":"2024-09-21T12:39:22.326465+0000","last_became_active":"2024-09-21T12:36:00.139167+0000","last_became_peered":"2024-09-21T12:36:00.139167+0000","last_unstale":"2024-09-21T12:39:22.326465+0000","last_undegraded":"2024-09-21T12:39:22.326465+0000","last_fullsized":"2024-09-21T12:39:22.326465+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:04.830956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:38:04.830956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:31.539751+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":919,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.327359+0000","last_change":"2024-09-21T12:38:07.159232+0000","last_active":"2024-09-21T12:39:22.327359+0000","last_peered":"2024-09-21T12:39:22.327359+0000","last_clean":"2024-09-21T12:39:22.327359+0000","last_became_active":"2024-09-21T12:38:07.157964+0000","last_became_peered":"2024-09-21T12:38:07.157964+0000","last_unstale":"2024-09-21T12:39:22.327359+0000","last_undegraded":"2024-09-21T12:39:22.327359+0000","last_fullsized":"2024-09-21T12:39:22.327359+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:06.698999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:06.698999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:59.290352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"276'27","reported_seq":964,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.326426+0000","last_change":"2024-09-21T12:38:41.050094+0000","last_active":"2024-09-21T12:39:22.326426+0000","last_peered":"2024-09-21T12:39:22.326426+0000","last_clean":"2024-09-21T12:39:22.326426+0000","last_became_active":"2024-09-21T12:38:07.168250+0000","last_became_peered":"2024-09-21T12:38:07.168250+0000","last_unstale":"2024-09-21T12:39:22.326426+0000","last_undegraded":"2024-09-21T12:39:22.326426+0000","last_fullsized":"2024-09-21T12:39:22.326426+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:38:41.050017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:41.050017+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:53.559093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":296,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.383797+0000","last_change":"2024-09-21T12:36:35.622295+0000","last_active":"2024-09-21T12:39:19.383797+0000","last_peered":"2024-09-21T12:39:19.383797+0000","last_clean":"2024-09-21T12:39:19.383797+0000","last_became_active":"2024-09-21T12:32:38.417724+0000","last_became_peered":"2024-09-21T12:32:38.417724+0000","last_unstale":"2024-09-21T12:39:19.383797+0000","last_undegraded":"2024-09-21T12:39:19.383797+0000","last_fullsized":"2024-09-21T12:39:19.383797+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:35.622232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:35.622232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:12.340388+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":464,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.487081+0000","last_change":"2024-09-21T12:39:18.247341+0000","last_active":"2024-09-21T12:39:19.487081+0000","last_peered":"2024-09-21T12:39:19.487081+0000","last_clean":"2024-09-21T12:39:19.487081+0000","last_became_active":"2024-09-21T12:37:20.467925+0000","last_became_peered":"2024-09-21T12:37:20.467925+0000","last_unstale":"2024-09-21T12:39:19.487081+0000","last_undegraded":"2024-09-21T12:39:19.487081+0000","last_fullsized":"2024-09-21T12:39:19.487081+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:18.247283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:18.247283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:30.819585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"276'15","reported_seq":322,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.384406+0000","last_change":"2024-09-21T12:38:03.431702+0000","last_active":"2024-09-21T12:39:19.384406+0000","last_peered":"2024-09-21T12:39:19.384406+0000","last_clean":"2024-09-21T12:39:19.384406+0000","last_became_active":"2024-09-21T12:32:38.426121+0000","last_became_peered":"2024-09-21T12:32:38.426121+0000","last_unstale":"2024-09-21T12:39:19.384406+0000","last_undegraded":"2024-09-21T12:39:19.384406+0000","last_fullsized":"2024-09-21T12:39:19.384406+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:38:03.431620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:03.431620+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":957,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.483513+0000","last_change":"2024-09-21T12:38:44.245362+0000","last_active":"2024-09-21T12:39:19.483513+0000","last_peered":"2024-09-21T12:39:19.483513+0000","last_clean":"2024-09-21T12:39:19.483513+0000","last_became_active":"2024-09-21T12:36:00.137364+0000","last_became_peered":"2024-09-21T12:36:00.137364+0000","last_unstale":"2024-09-21T12:39:19.483513+0000","last_undegraded":"2024-09-21T12:39:19.483513+0000","last_fullsized":"2024-09-21T12:39:19.483513+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:44.245309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:44.245309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:02.710227+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":918,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.393364+0000","last_change":"2024-09-21T12:36:38.752470+0000","last_active":"2024-09-21T12:39:22.393364+0000","last_peered":"2024-09-21T12:39:22.393364+0000","last_clean":"2024-09-21T12:39:22.393364+0000","last_became_active":"2024-09-21T12:36:00.134723+0000","last_became_peered":"2024-09-21T12:36:00.134723+0000","last_unstale":"2024-09-21T12:39:22.393364+0000","last_undegraded":"2024-09-21T12:39:22.393364+0000","last_fullsized":"2024-09-21T12:39:22.393364+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:38.752410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:38.752410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"76'79","reported_seq":694,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.327244+0000","last_change":"2024-09-21T12:37:56.010046+0000","last_active":"2024-09-21T12:39:22.327244+0000","last_peered":"2024-09-21T12:39:22.327244+0000","last_clean":"2024-09-21T12:39:22.327244+0000","last_became_active":"2024-09-21T12:36:00.147795+0000","last_became_peered":"2024-09-21T12:36:00.147795+0000","last_unstale":"2024-09-21T12:39:22.327244+0000","last_undegraded":"2024-09-21T12:39:22.327244+0000","last_fullsized":"2024-09-21T12:39:22.327244+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:37:56.009970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:37:56.009970+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":448,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.483965+0000","last_change":"2024-09-21T12:38:52.309280+0000","last_active":"2024-09-21T12:39:19.483965+0000","last_peered":"2024-09-21T12:39:19.483965+0000","last_clean":"2024-09-21T12:39:19.483965+0000","last_became_active":"2024-09-21T12:37:20.468468+0000","last_became_peered":"2024-09-21T12:37:20.468468+0000","last_unstale":"2024-09-21T12:39:19.483965+0000","last_undegraded":"2024-09-21T12:39:19.483965+0000","last_fullsized":"2024-09-21T12:39:19.483965+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:52.309224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:52.309224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:56.584480+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"276'7","reported_seq":318,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.384192+0000","last_change":"2024-09-21T12:39:19.280732+0000","last_active":"2024-09-21T12:39:19.384192+0000","last_peered":"2024-09-21T12:39:19.384192+0000","last_clean":"2024-09-21T12:39:19.384192+0000","last_became_active":"2024-09-21T12:32:38.422912+0000","last_became_peered":"2024-09-21T12:32:38.422912+0000","last_unstale":"2024-09-21T12:39:19.384192+0000","last_undegraded":"2024-09-21T12:39:19.384192+0000","last_fullsized":"2024-09-21T12:39:19.384192+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'7","last_scrub_stamp":"2024-09-21T12:39:19.280679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:19.280679+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:31.804604+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":165,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:20.299439+0000","last_change":"2024-09-21T12:37:57.609800+0000","last_active":"2024-09-21T12:39:20.299439+0000","last_peered":"2024-09-21T12:39:20.299439+0000","last_clean":"2024-09-21T12:39:20.299439+0000","last_became_active":"2024-09-21T12:36:00.142106+0000","last_became_peered":"2024-09-21T12:36:00.142106+0000","last_unstale":"2024-09-21T12:39:20.299439+0000","last_undegraded":"2024-09-21T12:39:20.299439+0000","last_fullsized":"2024-09-21T12:39:20.299439+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:57.609734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:37:57.609734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":909,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.484372+0000","last_change":"2024-09-21T12:33:36.125346+0000","last_active":"2024-09-21T12:39:19.484372+0000","last_peered":"2024-09-21T12:39:19.484372+0000","last_clean":"2024-09-21T12:39:19.484372+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:39:19.484372+0000","last_undegraded":"2024-09-21T12:39:19.484372+0000","last_fullsized":"2024-09-21T12:39:19.484372+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:36.124095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:36.124095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:34.160912+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"76'79","reported_seq":734,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.326831+0000","last_change":"2024-09-21T12:34:48.959991+0000","last_active":"2024-09-21T12:39:22.326831+0000","last_peered":"2024-09-21T12:39:22.326831+0000","last_clean":"2024-09-21T12:39:22.326831+0000","last_became_active":"2024-09-21T12:27:16.957569+0000","last_became_peered":"2024-09-21T12:27:16.957569+0000","last_unstale":"2024-09-21T12:39:22.326831+0000","last_undegraded":"2024-09-21T12:39:22.326831+0000","last_fullsized":"2024-09-21T12:39:22.326831+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:48.959916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:48.959916+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:13.209070+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":297,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.385066+0000","last_change":"2024-09-21T12:34:05.193021+0000","last_active":"2024-09-21T12:39:19.385066+0000","last_peered":"2024-09-21T12:39:19.385066+0000","last_clean":"2024-09-21T12:39:19.385066+0000","last_became_active":"2024-09-21T12:32:38.428997+0000","last_became_peered":"2024-09-21T12:32:38.428997+0000","last_unstale":"2024-09-21T12:39:19.385066+0000","last_undegraded":"2024-09-21T12:39:19.385066+0000","last_fullsized":"2024-09-21T12:39:19.385066+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:05.192971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:34:05.192971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:03.415980+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":981,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.393648+0000","last_change":"2024-09-21T12:37:23.610348+0000","last_active":"2024-09-21T12:39:22.393648+0000","last_peered":"2024-09-21T12:39:22.393648+0000","last_clean":"2024-09-21T12:39:22.393648+0000","last_became_active":"2024-09-21T12:36:00.127153+0000","last_became_peered":"2024-09-21T12:36:00.127153+0000","last_unstale":"2024-09-21T12:39:22.393648+0000","last_undegraded":"2024-09-21T12:39:22.393648+0000","last_fullsized":"2024-09-21T12:39:22.393648+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:23.610263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:37:23.610263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"276'27","reported_seq":960,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.483656+0000","last_change":"2024-09-21T12:38:07.162167+0000","last_active":"2024-09-21T12:39:19.483656+0000","last_peered":"2024-09-21T12:39:19.483656+0000","last_clean":"2024-09-21T12:39:19.483656+0000","last_became_active":"2024-09-21T12:38:07.161813+0000","last_became_peered":"2024-09-21T12:38:07.161813+0000","last_unstale":"2024-09-21T12:39:19.483656+0000","last_undegraded":"2024-09-21T12:39:19.483656+0000","last_fullsized":"2024-09-21T12:39:19.483656+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:33:01.668584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:33:01.668584+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":940,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.385362+0000","last_change":"2024-09-21T12:38:53.956579+0000","last_active":"2024-09-21T12:39:19.385362+0000","last_peered":"2024-09-21T12:39:19.385362+0000","last_clean":"2024-09-21T12:39:19.385362+0000","last_became_active":"2024-09-21T12:32:38.425958+0000","last_became_peered":"2024-09-21T12:32:38.425958+0000","last_unstale":"2024-09-21T12:39:19.385362+0000","last_undegraded":"2024-09-21T12:39:19.385362+0000","last_fullsized":"2024-09-21T12:39:19.385362+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:53.956515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:53.956515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:08.408871+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"76'79","reported_seq":867,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.486063+0000","last_change":"2024-09-21T12:38:08.157982+0000","last_active":"2024-09-21T12:39:19.486063+0000","last_peered":"2024-09-21T12:39:19.486063+0000","last_clean":"2024-09-21T12:39:19.486063+0000","last_became_active":"2024-09-21T12:38:08.157632+0000","last_became_peered":"2024-09-21T12:38:08.157632+0000","last_unstale":"2024-09-21T12:39:19.486063+0000","last_undegraded":"2024-09-21T12:39:19.486063+0000","last_fullsized":"2024-09-21T12:39:19.486063+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:27.294674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:27.294674+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":925,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.327244+0000","last_change":"2024-09-21T12:39:18.225857+0000","last_active":"2024-09-21T12:39:22.327244+0000","last_peered":"2024-09-21T12:39:22.327244+0000","last_clean":"2024-09-21T12:39:22.327244+0000","last_became_active":"2024-09-21T12:32:02.033276+0000","last_became_peered":"2024-09-21T12:32:02.033276+0000","last_unstale":"2024-09-21T12:39:22.327244+0000","last_undegraded":"2024-09-21T12:39:22.327244+0000","last_fullsized":"2024-09-21T12:39:22.327244+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:18.225813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:18.225813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:37.203577+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":307,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.384091+0000","last_change":"2024-09-21T12:38:41.208373+0000","last_active":"2024-09-21T12:39:19.384091+0000","last_peered":"2024-09-21T12:39:19.384091+0000","last_clean":"2024-09-21T12:39:19.384091+0000","last_became_active":"2024-09-21T12:38:07.159387+0000","last_became_peered":"2024-09-21T12:38:07.159387+0000","last_unstale":"2024-09-21T12:39:19.384091+0000","last_undegraded":"2024-09-21T12:39:19.384091+0000","last_fullsized":"2024-09-21T12:39:19.384091+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:41.208301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:41.208301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:51.906773+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"276'18","reported_seq":1010,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.326955+0000","last_change":"2024-09-21T12:35:59.641966+0000","last_active":"2024-09-21T12:39:22.326955+0000","last_peered":"2024-09-21T12:39:22.326955+0000","last_clean":"2024-09-21T12:39:22.326955+0000","last_became_active":"2024-09-21T12:32:38.409163+0000","last_became_peered":"2024-09-21T12:32:38.409163+0000","last_unstale":"2024-09-21T12:39:22.326955+0000","last_undegraded":"2024-09-21T12:39:22.326955+0000","last_fullsized":"2024-09-21T12:39:22.326955+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T12:35:59.641895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:59.641895+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":315,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.384536+0000","last_change":"2024-09-21T12:38:49.267225+0000","last_active":"2024-09-21T12:39:19.384536+0000","last_peered":"2024-09-21T12:39:19.384536+0000","last_clean":"2024-09-21T12:39:19.384536+0000","last_became_active":"2024-09-21T12:38:07.161634+0000","last_became_peered":"2024-09-21T12:38:07.161634+0000","last_unstale":"2024-09-21T12:39:19.384536+0000","last_undegraded":"2024-09-21T12:39:19.384536+0000","last_fullsized":"2024-09-21T12:39:19.384536+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:49.267162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:49.267162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:09.895158+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"76'79","reported_seq":655,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.483329+0000","last_change":"2024-09-21T12:36:41.538607+0000","last_active":"2024-09-21T12:39:19.483329+0000","last_peered":"2024-09-21T12:39:19.483329+0000","last_clean":"2024-09-21T12:39:19.483329+0000","last_became_active":"2024-09-21T12:32:38.414204+0000","last_became_peered":"2024-09-21T12:32:38.414204+0000","last_unstale":"2024-09-21T12:39:19.483329+0000","last_undegraded":"2024-09-21T12:39:19.483329+0000","last_fullsized":"2024-09-21T12:39:19.483329+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:41.538559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:41.538559+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:04.209095+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":889,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:20.299668+0000","last_change":"2024-09-21T12:38:08.166042+0000","last_active":"2024-09-21T12:39:20.299668+0000","last_peered":"2024-09-21T12:39:20.299668+0000","last_clean":"2024-09-21T12:39:20.299668+0000","last_became_active":"2024-09-21T12:38:08.165695+0000","last_became_peered":"2024-09-21T12:38:08.165695+0000","last_unstale":"2024-09-21T12:39:20.299668+0000","last_undegraded":"2024-09-21T12:39:20.299668+0000","last_fullsized":"2024-09-21T12:39:20.299668+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:36:37.839024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:37.839024+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":958,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.384969+0000","last_change":"2024-09-21T12:36:44.561097+0000","last_active":"2024-09-21T12:39:19.384969+0000","last_peered":"2024-09-21T12:39:19.384969+0000","last_clean":"2024-09-21T12:39:19.384969+0000","last_became_active":"2024-09-21T12:32:38.425335+0000","last_became_peered":"2024-09-21T12:32:38.425335+0000","last_unstale":"2024-09-21T12:39:19.384969+0000","last_undegraded":"2024-09-21T12:39:19.384969+0000","last_fullsized":"2024-09-21T12:39:19.384969+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:36:44.561030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:36:44.561030+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:57.870150+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"276'20","reported_seq":1037,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.393909+0000","last_change":"2024-09-21T12:38:54.562392+0000","last_active":"2024-09-21T12:39:22.393909+0000","last_peered":"2024-09-21T12:39:22.393909+0000","last_clean":"2024-09-21T12:39:22.393909+0000","last_became_active":"2024-09-21T12:36:00.148761+0000","last_became_peered":"2024-09-21T12:36:00.148761+0000","last_unstale":"2024-09-21T12:39:22.393909+0000","last_undegraded":"2024-09-21T12:39:22.393909+0000","last_fullsized":"2024-09-21T12:39:22.393909+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:38:54.562317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:54.562317+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:07.141362+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"415'18","reported_seq":404,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.385345+0000","last_change":"2024-09-21T12:37:56.019259+0000","last_active":"2024-09-21T12:39:19.385345+0000","last_peered":"2024-09-21T12:39:19.385345+0000","last_clean":"2024-09-21T12:39:19.385345+0000","last_became_active":"2024-09-21T12:36:00.147676+0000","last_became_peered":"2024-09-21T12:36:00.147676+0000","last_unstale":"2024-09-21T12:39:19.385345+0000","last_undegraded":"2024-09-21T12:39:19.385345+0000","last_fullsized":"2024-09-21T12:39:19.385345+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'18","last_scrub_stamp":"2024-09-21T12:37:56.019080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:37:56.019080+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"76'79","reported_seq":467,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.494936+0000","last_change":"2024-09-21T12:38:07.165980+0000","last_active":"2024-09-21T12:39:19.494936+0000","last_peered":"2024-09-21T12:39:19.494936+0000","last_clean":"2024-09-21T12:39:19.494936+0000","last_became_active":"2024-09-21T12:38:07.165640+0000","last_became_peered":"2024-09-21T12:38:07.165640+0000","last_unstale":"2024-09-21T12:39:19.494936+0000","last_undegraded":"2024-09-21T12:39:19.494936+0000","last_fullsized":"2024-09-21T12:39:19.494936+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:18.151644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:18.151644+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:26.197946+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"276'27","reported_seq":579,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.327330+0000","last_change":"2024-09-21T12:38:40.030449+0000","last_active":"2024-09-21T12:39:22.327330+0000","last_peered":"2024-09-21T12:39:22.327330+0000","last_clean":"2024-09-21T12:39:22.327330+0000","last_became_active":"2024-09-21T12:38:07.166374+0000","last_became_peered":"2024-09-21T12:38:07.166374+0000","last_unstale":"2024-09-21T12:39:22.327330+0000","last_undegraded":"2024-09-21T12:39:22.327330+0000","last_fullsized":"2024-09-21T12:39:22.327330+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:38:40.030357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:40.030357+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:09.020801+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":381,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.326259+0000","last_change":"2024-09-21T12:36:00.140935+0000","last_active":"2024-09-21T12:39:22.326259+0000","last_peered":"2024-09-21T12:39:22.326259+0000","last_clean":"2024-09-21T12:39:22.326259+0000","last_became_active":"2024-09-21T12:36:00.140455+0000","last_became_peered":"2024-09-21T12:36:00.140455+0000","last_unstale":"2024-09-21T12:39:22.326259+0000","last_undegraded":"2024-09-21T12:39:22.326259+0000","last_fullsized":"2024-09-21T12:39:22.326259+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:05.703464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:33:05.703464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:57.233717+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"352'15","reported_seq":389,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.385020+0000","last_change":"2024-09-21T12:38:02.464559+0000","last_active":"2024-09-21T12:39:19.385020+0000","last_peered":"2024-09-21T12:39:19.385020+0000","last_clean":"2024-09-21T12:39:19.385020+0000","last_became_active":"2024-09-21T12:32:38.429092+0000","last_became_peered":"2024-09-21T12:32:38.429092+0000","last_unstale":"2024-09-21T12:39:19.385020+0000","last_undegraded":"2024-09-21T12:39:19.385020+0000","last_fullsized":"2024-09-21T12:39:19.385020+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'15","last_scrub_stamp":"2024-09-21T12:38:02.464486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:02.464486+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:27.762390+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":416,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.487331+0000","last_change":"2024-09-21T12:37:55.128975+0000","last_active":"2024-09-21T12:39:19.487331+0000","last_peered":"2024-09-21T12:39:19.487331+0000","last_clean":"2024-09-21T12:39:19.487331+0000","last_became_active":"2024-09-21T12:37:20.468591+0000","last_became_peered":"2024-09-21T12:37:20.468591+0000","last_unstale":"2024-09-21T12:39:19.487331+0000","last_undegraded":"2024-09-21T12:39:19.487331+0000","last_fullsized":"2024-09-21T12:39:19.487331+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:37:55.128928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:37:55.128928+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"76'79","reported_seq":316,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.384708+0000","last_change":"2024-09-21T12:38:51.310242+0000","last_active":"2024-09-21T12:39:19.384708+0000","last_peered":"2024-09-21T12:39:19.384708+0000","last_clean":"2024-09-21T12:39:19.384708+0000","last_became_active":"2024-09-21T12:38:07.159280+0000","last_became_peered":"2024-09-21T12:38:07.159280+0000","last_unstale":"2024-09-21T12:39:19.384708+0000","last_undegraded":"2024-09-21T12:39:19.384708+0000","last_fullsized":"2024-09-21T12:39:19.384708+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:38:51.310157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:38:51.310157+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:11.944946+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"276'30","reported_seq":1062,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.326541+0000","last_change":"2024-09-21T12:36:00.163117+0000","last_active":"2024-09-21T12:39:22.326541+0000","last_peered":"2024-09-21T12:39:22.326541+0000","last_clean":"2024-09-21T12:39:22.326541+0000","last_became_active":"2024-09-21T12:36:00.139849+0000","last_became_peered":"2024-09-21T12:36:00.139849+0000","last_unstale":"2024-09-21T12:39:22.326541+0000","last_undegraded":"2024-09-21T12:39:22.326541+0000","last_fullsized":"2024-09-21T12:39:22.326541+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T12:35:12.790809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:12.790809+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:15.189776+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":921,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.327501+0000","last_change":"2024-09-21T12:39:20.300208+0000","last_active":"2024-09-21T12:39:22.327501+0000","last_peered":"2024-09-21T12:39:22.327501+0000","last_clean":"2024-09-21T12:39:22.327501+0000","last_became_active":"2024-09-21T12:36:00.151488+0000","last_became_peered":"2024-09-21T12:36:00.151488+0000","last_unstale":"2024-09-21T12:39:22.327501+0000","last_undegraded":"2024-09-21T12:39:22.327501+0000","last_fullsized":"2024-09-21T12:39:22.327501+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:20.300049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:20.300049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:33.837202+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":311,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.383992+0000","last_change":"2024-09-21T12:36:00.137014+0000","last_active":"2024-09-21T12:39:19.383992+0000","last_peered":"2024-09-21T12:39:19.383992+0000","last_clean":"2024-09-21T12:39:19.383992+0000","last_became_active":"2024-09-21T12:36:00.136650+0000","last_became_peered":"2024-09-21T12:36:00.136650+0000","last_unstale":"2024-09-21T12:39:19.383992+0000","last_undegraded":"2024-09-21T12:39:19.383992+0000","last_fullsized":"2024-09-21T12:39:19.383992+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:12.216104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:12.216104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:04.402566+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":308,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.384418+0000","last_change":"2024-09-21T12:38:44.156366+0000","last_active":"2024-09-21T12:39:19.384418+0000","last_peered":"2024-09-21T12:39:19.384418+0000","last_clean":"2024-09-21T12:39:19.384418+0000","last_became_active":"2024-09-21T12:38:07.166919+0000","last_became_peered":"2024-09-21T12:38:07.166919+0000","last_unstale":"2024-09-21T12:39:19.384418+0000","last_undegraded":"2024-09-21T12:39:19.384418+0000","last_fullsized":"2024-09-21T12:39:19.384418+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:44.156307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:38:44.156307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:06.667086+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"76'79","reported_seq":849,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.326776+0000","last_change":"2024-09-21T12:36:43.462949+0000","last_active":"2024-09-21T12:39:22.326776+0000","last_peered":"2024-09-21T12:39:22.326776+0000","last_clean":"2024-09-21T12:39:22.326776+0000","last_became_active":"2024-09-21T12:36:00.155047+0000","last_became_peered":"2024-09-21T12:36:00.155047+0000","last_unstale":"2024-09-21T12:39:22.326776+0000","last_undegraded":"2024-09-21T12:39:22.326776+0000","last_fullsized":"2024-09-21T12:39:22.326776+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:43.462874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:43.462874+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"276'28","reported_seq":305,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.383888+0000","last_change":"2024-09-21T12:32:38.457999+0000","last_active":"2024-09-21T12:39:19.383888+0000","last_peered":"2024-09-21T12:39:19.383888+0000","last_clean":"2024-09-21T12:39:19.383888+0000","last_became_active":"2024-09-21T12:32:38.425453+0000","last_became_peered":"2024-09-21T12:32:38.425453+0000","last_unstale":"2024-09-21T12:39:19.383888+0000","last_undegraded":"2024-09-21T12:39:19.383888+0000","last_fullsized":"2024-09-21T12:39:19.383888+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-21T12:26:51.463698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:26:51.463698+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:33.319669+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":40,"num_read_kb":33,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":996,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.394138+0000","last_change":"2024-09-21T12:36:39.754233+0000","last_active":"2024-09-21T12:39:22.394138+0000","last_peered":"2024-09-21T12:39:22.394138+0000","last_clean":"2024-09-21T12:39:22.394138+0000","last_became_active":"2024-09-21T12:32:38.424412+0000","last_became_peered":"2024-09-21T12:32:38.424412+0000","last_unstale":"2024-09-21T12:39:22.394138+0000","last_undegraded":"2024-09-21T12:39:22.394138+0000","last_fullsized":"2024-09-21T12:39:22.394138+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:36:39.754173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:36:39.754173+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"352'12","reported_seq":1190,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.326375+0000","last_change":"2024-09-21T12:37:57.970843+0000","last_active":"2024-09-21T12:39:22.326375+0000","last_peered":"2024-09-21T12:39:22.326375+0000","last_clean":"2024-09-21T12:39:22.326375+0000","last_became_active":"2024-09-21T12:32:38.409177+0000","last_became_peered":"2024-09-21T12:32:38.409177+0000","last_unstale":"2024-09-21T12:39:22.326375+0000","last_undegraded":"2024-09-21T12:39:22.326375+0000","last_fullsized":"2024-09-21T12:39:22.326375+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'12","last_scrub_stamp":"2024-09-21T12:37:57.970636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:37:57.970636+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":898,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:20.299504+0000","last_change":"2024-09-21T12:38:51.740285+0000","last_active":"2024-09-21T12:39:20.299504+0000","last_peered":"2024-09-21T12:39:20.299504+0000","last_clean":"2024-09-21T12:39:20.299504+0000","last_became_active":"2024-09-21T12:38:08.154409+0000","last_became_peered":"2024-09-21T12:38:08.154409+0000","last_unstale":"2024-09-21T12:39:20.299504+0000","last_undegraded":"2024-09-21T12:39:20.299504+0000","last_fullsized":"2024-09-21T12:39:20.299504+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:51.740136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:38:51.740136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:54.159637+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"76'79","reported_seq":389,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.327426+0000","last_change":"2024-09-21T12:37:16.761819+0000","last_active":"2024-09-21T12:39:22.327426+0000","last_peered":"2024-09-21T12:39:22.327426+0000","last_clean":"2024-09-21T12:39:22.327426+0000","last_became_active":"2024-09-21T12:36:00.151076+0000","last_became_peered":"2024-09-21T12:36:00.151076+0000","last_unstale":"2024-09-21T12:39:22.327426+0000","last_undegraded":"2024-09-21T12:39:22.327426+0000","last_fullsized":"2024-09-21T12:39:22.327426+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:37:16.761749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:37:16.761749+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"276'20","reported_seq":447,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.386230+0000","last_change":"2024-09-21T12:36:33.481707+0000","last_active":"2024-09-21T12:39:19.386230+0000","last_peered":"2024-09-21T12:39:19.386230+0000","last_clean":"2024-09-21T12:39:19.386230+0000","last_became_active":"2024-09-21T12:32:38.424412+0000","last_became_peered":"2024-09-21T12:32:38.424412+0000","last_unstale":"2024-09-21T12:39:19.386230+0000","last_undegraded":"2024-09-21T12:39:19.386230+0000","last_fullsized":"2024-09-21T12:39:19.386230+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:36:33.481627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:33.481627+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:07.097608+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":926,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.487006+0000","last_change":"2024-09-21T12:38:07.165927+0000","last_active":"2024-09-21T12:39:19.487006+0000","last_peered":"2024-09-21T12:39:19.487006+0000","last_clean":"2024-09-21T12:39:19.487006+0000","last_became_active":"2024-09-21T12:38:07.165614+0000","last_became_peered":"2024-09-21T12:38:07.165614+0000","last_unstale":"2024-09-21T12:39:19.487006+0000","last_undegraded":"2024-09-21T12:39:19.487006+0000","last_fullsized":"2024-09-21T12:39:19.487006+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:32:19.629083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:32:19.629083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:11.161960+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":1152,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.393836+0000","last_change":"2024-09-21T12:36:44.788732+0000","last_active":"2024-09-21T12:39:22.393836+0000","last_peered":"2024-09-21T12:39:22.393836+0000","last_clean":"2024-09-21T12:39:22.393836+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:39:22.393836+0000","last_undegraded":"2024-09-21T12:39:22.393836+0000","last_fullsized":"2024-09-21T12:39:22.393836+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:36:44.788662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:44.788662+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":910,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.393287+0000","last_change":"2024-09-21T12:38:44.590928+0000","last_active":"2024-09-21T12:39:22.393287+0000","last_peered":"2024-09-21T12:39:22.393287+0000","last_clean":"2024-09-21T12:39:22.393287+0000","last_became_active":"2024-09-21T12:38:07.163353+0000","last_became_peered":"2024-09-21T12:38:07.163353+0000","last_unstale":"2024-09-21T12:39:22.393287+0000","last_undegraded":"2024-09-21T12:39:22.393287+0000","last_fullsized":"2024-09-21T12:39:22.393287+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:44.590868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:38:44.590868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:06.202952+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"76'79","reported_seq":584,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.393765+0000","last_change":"2024-09-21T12:32:52.986242+0000","last_active":"2024-09-21T12:39:22.393765+0000","last_peered":"2024-09-21T12:39:22.393765+0000","last_clean":"2024-09-21T12:39:22.393765+0000","last_became_active":"2024-09-21T12:29:05.835884+0000","last_became_peered":"2024-09-21T12:29:05.835884+0000","last_unstale":"2024-09-21T12:39:22.393765+0000","last_undegraded":"2024-09-21T12:39:22.393765+0000","last_fullsized":"2024-09-21T12:39:22.393765+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:32:52.986175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:32:52.986175+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:57.609852+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"276'29","reported_seq":1008,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.393727+0000","last_change":"2024-09-21T12:36:36.774685+0000","last_active":"2024-09-21T12:39:22.393727+0000","last_peered":"2024-09-21T12:39:22.393727+0000","last_clean":"2024-09-21T12:39:22.393727+0000","last_became_active":"2024-09-21T12:36:00.127312+0000","last_became_peered":"2024-09-21T12:36:00.127312+0000","last_unstale":"2024-09-21T12:39:22.393727+0000","last_undegraded":"2024-09-21T12:39:22.393727+0000","last_fullsized":"2024-09-21T12:39:22.393727+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'29","last_scrub_stamp":"2024-09-21T12:36:36.774599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:36.774599+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":910,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.386744+0000","last_change":"2024-09-21T12:38:48.953684+0000","last_active":"2024-09-21T12:39:19.386744+0000","last_peered":"2024-09-21T12:39:19.386744+0000","last_clean":"2024-09-21T12:39:19.386744+0000","last_became_active":"2024-09-21T12:38:07.163439+0000","last_became_peered":"2024-09-21T12:38:07.163439+0000","last_unstale":"2024-09-21T12:39:19.386744+0000","last_undegraded":"2024-09-21T12:39:19.386744+0000","last_fullsized":"2024-09-21T12:39:19.386744+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:48.953635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:48.953635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:17.502212+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":496,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.393504+0000","last_change":"2024-09-21T12:39:21.815712+0000","last_active":"2024-09-21T12:39:22.393504+0000","last_peered":"2024-09-21T12:39:22.393504+0000","last_clean":"2024-09-21T12:39:22.393504+0000","last_became_active":"2024-09-21T12:29:05.825869+0000","last_became_peered":"2024-09-21T12:29:05.825869+0000","last_unstale":"2024-09-21T12:39:22.393504+0000","last_undegraded":"2024-09-21T12:39:22.393504+0000","last_fullsized":"2024-09-21T12:39:22.393504+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":257,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:21.815656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:21.815656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:23.341727+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":906,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.384753+0000","last_change":"2024-09-21T12:38:45.894780+0000","last_active":"2024-09-21T12:39:19.384753+0000","last_peered":"2024-09-21T12:39:19.384753+0000","last_clean":"2024-09-21T12:39:19.384753+0000","last_became_active":"2024-09-21T12:38:07.158992+0000","last_became_peered":"2024-09-21T12:38:07.158992+0000","last_unstale":"2024-09-21T12:39:19.384753+0000","last_undegraded":"2024-09-21T12:39:19.384753+0000","last_fullsized":"2024-09-21T12:39:19.384753+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:45.894717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:38:45.894717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:57.342298+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"76'79","reported_seq":342,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.484212+0000","last_change":"2024-09-21T12:38:07.162380+0000","last_active":"2024-09-21T12:39:19.484212+0000","last_peered":"2024-09-21T12:39:19.484212+0000","last_clean":"2024-09-21T12:39:19.484212+0000","last_became_active":"2024-09-21T12:38:07.162110+0000","last_became_peered":"2024-09-21T12:38:07.162110+0000","last_unstale":"2024-09-21T12:39:19.484212+0000","last_undegraded":"2024-09-21T12:39:19.484212+0000","last_fullsized":"2024-09-21T12:39:19.484212+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:33:34.055720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:33:34.055720+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:24.198020+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"457'42","reported_seq":1204,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.386690+0000","last_change":"2024-09-21T12:38:07.440187+0000","last_active":"2024-09-21T12:39:19.386690+0000","last_peered":"2024-09-21T12:39:19.386690+0000","last_clean":"2024-09-21T12:39:19.386690+0000","last_became_active":"2024-09-21T12:38:07.164073+0000","last_became_peered":"2024-09-21T12:38:07.164073+0000","last_unstale":"2024-09-21T12:39:19.386690+0000","last_undegraded":"2024-09-21T12:39:19.386690+0000","last_fullsized":"2024-09-21T12:39:19.386690+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'41","last_scrub_stamp":"2024-09-21T12:38:07.440124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:07.440124+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:24.803396+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":305,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.384283+0000","last_change":"2024-09-21T12:37:50.899946+0000","last_active":"2024-09-21T12:39:19.384283+0000","last_peered":"2024-09-21T12:39:19.384283+0000","last_clean":"2024-09-21T12:39:19.384283+0000","last_became_active":"2024-09-21T12:32:38.426309+0000","last_became_peered":"2024-09-21T12:32:38.426309+0000","last_unstale":"2024-09-21T12:39:19.384283+0000","last_undegraded":"2024-09-21T12:39:19.384283+0000","last_fullsized":"2024-09-21T12:39:19.384283+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:50.899879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:37:50.899879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":632,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.386330+0000","last_change":"2024-09-21T12:37:57.480538+0000","last_active":"2024-09-21T12:39:19.386330+0000","last_peered":"2024-09-21T12:39:19.386330+0000","last_clean":"2024-09-21T12:39:19.386330+0000","last_became_active":"2024-09-21T12:26:54.083584+0000","last_became_peered":"2024-09-21T12:26:54.083584+0000","last_unstale":"2024-09-21T12:39:19.386330+0000","last_undegraded":"2024-09-21T12:39:19.386330+0000","last_fullsized":"2024-09-21T12:39:19.386330+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T12:37:57.480458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:37:57.480458+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":947,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.393734+0000","last_change":"2024-09-21T12:38:47.559548+0000","last_active":"2024-09-21T12:39:22.393734+0000","last_peered":"2024-09-21T12:39:22.393734+0000","last_clean":"2024-09-21T12:39:22.393734+0000","last_became_active":"2024-09-21T12:32:38.424402+0000","last_became_peered":"2024-09-21T12:32:38.424402+0000","last_unstale":"2024-09-21T12:39:22.393734+0000","last_undegraded":"2024-09-21T12:39:22.393734+0000","last_fullsized":"2024-09-21T12:39:22.393734+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:47.559490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:47.559490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:03.876249+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":945,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.327186+0000","last_change":"2024-09-21T12:37:56.975879+0000","last_active":"2024-09-21T12:39:22.327186+0000","last_peered":"2024-09-21T12:39:22.327186+0000","last_clean":"2024-09-21T12:39:22.327186+0000","last_became_active":"2024-09-21T12:36:00.144526+0000","last_became_peered":"2024-09-21T12:36:00.144526+0000","last_unstale":"2024-09-21T12:39:22.327186+0000","last_undegraded":"2024-09-21T12:39:22.327186+0000","last_fullsized":"2024-09-21T12:39:22.327186+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:56.975826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:37:56.975826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'79","reported_seq":329,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:20.300665+0000","last_change":"2024-09-21T12:38:50.763337+0000","last_active":"2024-09-21T12:39:20.300665+0000","last_peered":"2024-09-21T12:39:20.300665+0000","last_clean":"2024-09-21T12:39:20.300665+0000","last_became_active":"2024-09-21T12:36:01.131622+0000","last_became_peered":"2024-09-21T12:36:01.131622+0000","last_unstale":"2024-09-21T12:39:20.300665+0000","last_undegraded":"2024-09-21T12:39:20.300665+0000","last_fullsized":"2024-09-21T12:39:20.300665+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:38:50.763274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:38:50.763274+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:05.037000+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"276'17","reported_seq":409,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.386486+0000","last_change":"2024-09-21T12:37:52.583825+0000","last_active":"2024-09-21T12:39:19.386486+0000","last_peered":"2024-09-21T12:39:19.386486+0000","last_clean":"2024-09-21T12:39:19.386486+0000","last_became_active":"2024-09-21T12:31:39.057316+0000","last_became_peered":"2024-09-21T12:31:39.057316+0000","last_unstale":"2024-09-21T12:39:19.386486+0000","last_undegraded":"2024-09-21T12:39:19.386486+0000","last_fullsized":"2024-09-21T12:39:19.386486+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'17","last_scrub_stamp":"2024-09-21T12:37:52.583730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:37:52.583730+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":942,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.384276+0000","last_change":"2024-09-21T12:38:49.953105+0000","last_active":"2024-09-21T12:39:19.384276+0000","last_peered":"2024-09-21T12:39:19.384276+0000","last_clean":"2024-09-21T12:39:19.384276+0000","last_became_active":"2024-09-21T12:38:07.154643+0000","last_became_peered":"2024-09-21T12:38:07.154643+0000","last_unstale":"2024-09-21T12:39:19.384276+0000","last_undegraded":"2024-09-21T12:39:19.384276+0000","last_fullsized":"2024-09-21T12:39:19.384276+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:49.952845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:49.952845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:51.858894+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":252,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.326370+0000","last_change":"2024-09-21T12:38:49.202301+0000","last_active":"2024-09-21T12:39:22.326370+0000","last_peered":"2024-09-21T12:39:22.326370+0000","last_clean":"2024-09-21T12:39:22.326370+0000","last_became_active":"2024-09-21T12:38:07.168680+0000","last_became_peered":"2024-09-21T12:38:07.168680+0000","last_unstale":"2024-09-21T12:39:22.326370+0000","last_undegraded":"2024-09-21T12:39:22.326370+0000","last_fullsized":"2024-09-21T12:39:22.326370+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:38:49.202240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:38:49.202240+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:56.553523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":241,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.484546+0000","last_change":"2024-09-21T12:38:08.157936+0000","last_active":"2024-09-21T12:39:19.484546+0000","last_peered":"2024-09-21T12:39:19.484546+0000","last_clean":"2024-09-21T12:39:19.484546+0000","last_became_active":"2024-09-21T12:38:08.157608+0000","last_became_peered":"2024-09-21T12:38:08.157608+0000","last_unstale":"2024-09-21T12:39:19.484546+0000","last_undegraded":"2024-09-21T12:39:19.484546+0000","last_fullsized":"2024-09-21T12:39:19.484546+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:02.295255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:02.295255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:25.229168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":144,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.393556+0000","last_change":"2024-09-21T12:38:08.166386+0000","last_active":"2024-09-21T12:39:22.393556+0000","last_peered":"2024-09-21T12:39:22.393556+0000","last_clean":"2024-09-21T12:39:22.393556+0000","last_became_active":"2024-09-21T12:38:08.165465+0000","last_became_peered":"2024-09-21T12:38:08.165465+0000","last_unstale":"2024-09-21T12:39:22.393556+0000","last_undegraded":"2024-09-21T12:39:22.393556+0000","last_fullsized":"2024-09-21T12:39:22.393556+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:53.671046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:37:53.671046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"76'79","reported_seq":864,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.386630+0000","last_change":"2024-09-21T12:36:41.495668+0000","last_active":"2024-09-21T12:39:19.386630+0000","last_peered":"2024-09-21T12:39:19.386630+0000","last_clean":"2024-09-21T12:39:19.386630+0000","last_became_active":"2024-09-21T12:32:38.424710+0000","last_became_peered":"2024-09-21T12:32:38.424710+0000","last_unstale":"2024-09-21T12:39:19.386630+0000","last_undegraded":"2024-09-21T12:39:19.386630+0000","last_fullsized":"2024-09-21T12:39:19.386630+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:41.495532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:41.495532+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:55.878434+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"276'24","reported_seq":953,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.394223+0000","last_change":"2024-09-21T12:38:07.517859+0000","last_active":"2024-09-21T12:39:22.394223+0000","last_peered":"2024-09-21T12:39:22.394223+0000","last_clean":"2024-09-21T12:39:22.394223+0000","last_became_active":"2024-09-21T12:38:07.166839+0000","last_became_peered":"2024-09-21T12:38:07.166839+0000","last_unstale":"2024-09-21T12:39:22.394223+0000","last_undegraded":"2024-09-21T12:39:22.394223+0000","last_fullsized":"2024-09-21T12:39:22.394223+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'24","last_scrub_stamp":"2024-09-21T12:38:07.517773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:07.517773+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:28.299185+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":961,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.393312+0000","last_change":"2024-09-21T12:38:48.558827+0000","last_active":"2024-09-21T12:39:22.393312+0000","last_peered":"2024-09-21T12:39:22.393312+0000","last_clean":"2024-09-21T12:39:22.393312+0000","last_became_active":"2024-09-21T12:32:38.411004+0000","last_became_peered":"2024-09-21T12:32:38.411004+0000","last_unstale":"2024-09-21T12:39:22.393312+0000","last_undegraded":"2024-09-21T12:39:22.393312+0000","last_fullsized":"2024-09-21T12:39:22.393312+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:48.558779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:48.558779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:01.839466+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":371,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:20.529077+0000","last_change":"2024-09-21T12:36:27.852222+0000","last_active":"2024-09-21T12:39:20.529077+0000","last_peered":"2024-09-21T12:39:20.529077+0000","last_clean":"2024-09-21T12:39:20.529077+0000","last_became_active":"2024-09-21T12:36:00.144053+0000","last_became_peered":"2024-09-21T12:36:00.144053+0000","last_unstale":"2024-09-21T12:39:20.529077+0000","last_undegraded":"2024-09-21T12:39:20.529077+0000","last_fullsized":"2024-09-21T12:39:20.529077+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-21T12:36:27.852030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:36:27.852030+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:58.750782+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":208,"num_read_kb":203,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":353,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.486212+0000","last_change":"2024-09-21T12:38:54.293655+0000","last_active":"2024-09-21T12:39:19.486212+0000","last_peered":"2024-09-21T12:39:19.486212+0000","last_clean":"2024-09-21T12:39:19.486212+0000","last_became_active":"2024-09-21T12:32:02.030240+0000","last_became_peered":"2024-09-21T12:32:02.030240+0000","last_unstale":"2024-09-21T12:39:19.486212+0000","last_undegraded":"2024-09-21T12:39:19.486212+0000","last_fullsized":"2024-09-21T12:39:19.486212+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:54.293611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:54.293611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:20.703408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":397,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.386143+0000","last_change":"2024-09-21T12:37:58.528094+0000","last_active":"2024-09-21T12:39:19.386143+0000","last_peered":"2024-09-21T12:39:19.386143+0000","last_clean":"2024-09-21T12:39:19.386143+0000","last_became_active":"2024-09-21T12:31:39.060160+0000","last_became_peered":"2024-09-21T12:31:39.060160+0000","last_unstale":"2024-09-21T12:39:19.386143+0000","last_undegraded":"2024-09-21T12:39:19.386143+0000","last_fullsized":"2024-09-21T12:39:19.386143+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:58.528033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:37:58.528033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"415'35","reported_seq":454,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.384905+0000","last_change":"2024-09-21T12:36:43.681619+0000","last_active":"2024-09-21T12:39:19.384905+0000","last_peered":"2024-09-21T12:39:19.384905+0000","last_clean":"2024-09-21T12:39:19.384905+0000","last_became_active":"2024-09-21T12:36:00.139342+0000","last_became_peered":"2024-09-21T12:36:00.139342+0000","last_unstale":"2024-09-21T12:39:19.384905+0000","last_undegraded":"2024-09-21T12:39:19.384905+0000","last_fullsized":"2024-09-21T12:39:19.384905+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'35","last_scrub_stamp":"2024-09-21T12:36:43.681471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:43.681471+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:09.405650+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":922,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.327214+0000","last_change":"2024-09-21T12:38:01.901094+0000","last_active":"2024-09-21T12:39:22.327214+0000","last_peered":"2024-09-21T12:39:22.327214+0000","last_clean":"2024-09-21T12:39:22.327214+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:39:22.327214+0000","last_undegraded":"2024-09-21T12:39:22.327214+0000","last_fullsized":"2024-09-21T12:39:22.327214+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:38:01.898469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:01.898469+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":165,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:20.299376+0000","last_change":"2024-09-21T12:37:14.690524+0000","last_active":"2024-09-21T12:39:20.299376+0000","last_peered":"2024-09-21T12:39:20.299376+0000","last_clean":"2024-09-21T12:39:20.299376+0000","last_became_active":"2024-09-21T12:36:00.134878+0000","last_became_peered":"2024-09-21T12:36:00.134878+0000","last_unstale":"2024-09-21T12:39:20.299376+0000","last_undegraded":"2024-09-21T12:39:20.299376+0000","last_fullsized":"2024-09-21T12:39:20.299376+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:37:14.690456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:37:14.690456+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":332,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:20.300057+0000","last_change":"2024-09-21T12:38:08.164955+0000","last_active":"2024-09-21T12:39:20.300057+0000","last_peered":"2024-09-21T12:39:20.300057+0000","last_clean":"2024-09-21T12:39:20.300057+0000","last_became_active":"2024-09-21T12:38:08.164554+0000","last_became_peered":"2024-09-21T12:38:08.164554+0000","last_unstale":"2024-09-21T12:39:20.300057+0000","last_undegraded":"2024-09-21T12:39:20.300057+0000","last_fullsized":"2024-09-21T12:39:20.300057+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:22.649558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:37:22.649558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":569,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.486464+0000","last_change":"2024-09-21T12:38:00.438673+0000","last_active":"2024-09-21T12:39:19.486464+0000","last_peered":"2024-09-21T12:39:19.486464+0000","last_clean":"2024-09-21T12:39:19.486464+0000","last_became_active":"2024-09-21T12:37:20.485126+0000","last_became_peered":"2024-09-21T12:37:20.485126+0000","last_unstale":"2024-09-21T12:39:19.486464+0000","last_undegraded":"2024-09-21T12:39:19.486464+0000","last_fullsized":"2024-09-21T12:39:19.486464+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:00.438586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:38:00.438586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":331,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"76'79","reported_seq":860,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.386928+0000","last_change":"2024-09-21T12:38:40.853940+0000","last_active":"2024-09-21T12:39:19.386928+0000","last_peered":"2024-09-21T12:39:19.386928+0000","last_clean":"2024-09-21T12:39:19.386928+0000","last_became_active":"2024-09-21T12:27:14.954351+0000","last_became_peered":"2024-09-21T12:27:14.954351+0000","last_unstale":"2024-09-21T12:39:19.386928+0000","last_undegraded":"2024-09-21T12:39:19.386928+0000","last_fullsized":"2024-09-21T12:39:19.386928+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:38:40.853870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:38:40.853870+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:57.441950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"276'19","reported_seq":316,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.385144+0000","last_change":"2024-09-21T12:36:41.666876+0000","last_active":"2024-09-21T12:39:19.385144+0000","last_peered":"2024-09-21T12:39:19.385144+0000","last_clean":"2024-09-21T12:39:19.385144+0000","last_became_active":"2024-09-21T12:32:38.419714+0000","last_became_peered":"2024-09-21T12:32:38.419714+0000","last_unstale":"2024-09-21T12:39:19.385144+0000","last_undegraded":"2024-09-21T12:39:19.385144+0000","last_fullsized":"2024-09-21T12:39:19.385144+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:36:41.666812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:41.666812+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:07.411686+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":327,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.383936+0000","last_change":"2024-09-21T12:38:08.148418+0000","last_active":"2024-09-21T12:39:19.383936+0000","last_peered":"2024-09-21T12:39:19.383936+0000","last_clean":"2024-09-21T12:39:19.383936+0000","last_became_active":"2024-09-21T12:38:08.148109+0000","last_became_peered":"2024-09-21T12:38:08.148109+0000","last_unstale":"2024-09-21T12:39:19.383936+0000","last_undegraded":"2024-09-21T12:39:19.383936+0000","last_fullsized":"2024-09-21T12:39:19.383936+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:40.708469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:36:40.708469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:15.280609+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":448,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.486613+0000","last_change":"2024-09-21T12:38:38.201658+0000","last_active":"2024-09-21T12:39:19.486613+0000","last_peered":"2024-09-21T12:39:19.486613+0000","last_clean":"2024-09-21T12:39:19.486613+0000","last_became_active":"2024-09-21T12:37:20.467986+0000","last_became_peered":"2024-09-21T12:37:20.467986+0000","last_unstale":"2024-09-21T12:39:19.486613+0000","last_undegraded":"2024-09-21T12:39:19.486613+0000","last_fullsized":"2024-09-21T12:39:19.486613+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:38.201448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:38.201448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:46.239927+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":895,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.326855+0000","last_change":"2024-09-21T12:35:24.786713+0000","last_active":"2024-09-21T12:39:22.326855+0000","last_peered":"2024-09-21T12:39:22.326855+0000","last_clean":"2024-09-21T12:39:22.326855+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:39:22.326855+0000","last_undegraded":"2024-09-21T12:39:22.326855+0000","last_fullsized":"2024-09-21T12:39:22.326855+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:24.786609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:24.786609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:14.189862+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"76'79","reported_seq":242,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.326270+0000","last_change":"2024-09-21T12:38:07.162337+0000","last_active":"2024-09-21T12:39:22.326270+0000","last_peered":"2024-09-21T12:39:22.326270+0000","last_clean":"2024-09-21T12:39:22.326270+0000","last_became_active":"2024-09-21T12:38:07.161730+0000","last_became_peered":"2024-09-21T12:38:07.161730+0000","last_unstale":"2024-09-21T12:39:22.326270+0000","last_undegraded":"2024-09-21T12:39:22.326270+0000","last_fullsized":"2024-09-21T12:39:22.326270+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:17.035216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:17.035216+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:05.361038+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"276'31","reported_seq":333,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.384554+0000","last_change":"2024-09-21T12:38:53.268874+0000","last_active":"2024-09-21T12:39:19.384554+0000","last_peered":"2024-09-21T12:39:19.384554+0000","last_clean":"2024-09-21T12:39:19.384554+0000","last_became_active":"2024-09-21T12:38:07.163226+0000","last_became_peered":"2024-09-21T12:38:07.163226+0000","last_unstale":"2024-09-21T12:39:19.384554+0000","last_undegraded":"2024-09-21T12:39:19.384554+0000","last_fullsized":"2024-09-21T12:39:19.384554+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'31","last_scrub_stamp":"2024-09-21T12:38:53.268744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:53.268744+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:08.857112+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":313,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.384166+0000","last_change":"2024-09-21T12:37:53.993984+0000","last_active":"2024-09-21T12:39:19.384166+0000","last_peered":"2024-09-21T12:39:19.384166+0000","last_clean":"2024-09-21T12:39:19.384166+0000","last_became_active":"2024-09-21T12:32:38.424451+0000","last_became_peered":"2024-09-21T12:32:38.424451+0000","last_unstale":"2024-09-21T12:39:19.384166+0000","last_undegraded":"2024-09-21T12:39:19.384166+0000","last_fullsized":"2024-09-21T12:39:19.384166+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:53.993920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:37:53.993920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":926,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.386231+0000","last_change":"2024-09-21T12:36:43.539109+0000","last_active":"2024-09-21T12:39:19.386231+0000","last_peered":"2024-09-21T12:39:19.386231+0000","last_clean":"2024-09-21T12:39:19.386231+0000","last_became_active":"2024-09-21T12:32:38.425177+0000","last_became_peered":"2024-09-21T12:32:38.425177+0000","last_unstale":"2024-09-21T12:39:19.386231+0000","last_undegraded":"2024-09-21T12:39:19.386231+0000","last_fullsized":"2024-09-21T12:39:19.386231+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:43.539053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:43.539053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:05.078879+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":852,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.327155+0000","last_change":"2024-09-21T12:38:07.160870+0000","last_active":"2024-09-21T12:39:22.327155+0000","last_peered":"2024-09-21T12:39:22.327155+0000","last_clean":"2024-09-21T12:39:22.327155+0000","last_became_active":"2024-09-21T12:38:07.160538+0000","last_became_peered":"2024-09-21T12:38:07.160538+0000","last_unstale":"2024-09-21T12:39:22.327155+0000","last_undegraded":"2024-09-21T12:39:22.327155+0000","last_fullsized":"2024-09-21T12:39:22.327155+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:20.005906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:34:20.005906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:01.287896+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"76'79","reported_seq":983,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:20.300418+0000","last_change":"2024-09-21T12:38:48.834565+0000","last_active":"2024-09-21T12:39:20.300418+0000","last_peered":"2024-09-21T12:39:20.300418+0000","last_clean":"2024-09-21T12:39:20.300418+0000","last_became_active":"2024-09-21T12:38:08.156755+0000","last_became_peered":"2024-09-21T12:38:08.156755+0000","last_unstale":"2024-09-21T12:39:20.300418+0000","last_undegraded":"2024-09-21T12:39:20.300418+0000","last_fullsized":"2024-09-21T12:39:20.300418+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:38:48.834502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:38:48.834502+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:17.209408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"276'20","reported_seq":319,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.384445+0000","last_change":"2024-09-21T12:38:46.208578+0000","last_active":"2024-09-21T12:39:19.384445+0000","last_peered":"2024-09-21T12:39:19.384445+0000","last_clean":"2024-09-21T12:39:19.384445+0000","last_became_active":"2024-09-21T12:38:07.162180+0000","last_became_peered":"2024-09-21T12:38:07.162180+0000","last_unstale":"2024-09-21T12:39:19.384445+0000","last_undegraded":"2024-09-21T12:39:19.384445+0000","last_fullsized":"2024-09-21T12:39:19.384445+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:38:46.208493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:46.208493+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:10.147140+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":935,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.326796+0000","last_change":"2024-09-21T12:38:44.144209+0000","last_active":"2024-09-21T12:39:22.326796+0000","last_peered":"2024-09-21T12:39:22.326796+0000","last_clean":"2024-09-21T12:39:22.326796+0000","last_became_active":"2024-09-21T12:32:38.403997+0000","last_became_peered":"2024-09-21T12:32:38.403997+0000","last_unstale":"2024-09-21T12:39:22.326796+0000","last_undegraded":"2024-09-21T12:39:22.326796+0000","last_fullsized":"2024-09-21T12:39:22.326796+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:44.144166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:44.144166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:55.330893+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"457'16","reported_seq":320,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:20.300121+0000","last_change":"2024-09-21T12:38:08.167263+0000","last_active":"2024-09-21T12:39:20.300121+0000","last_peered":"2024-09-21T12:39:20.300121+0000","last_clean":"2024-09-21T12:39:20.300121+0000","last_became_active":"2024-09-21T12:38:08.165321+0000","last_became_peered":"2024-09-21T12:38:08.165321+0000","last_unstale":"2024-09-21T12:39:20.300121+0000","last_undegraded":"2024-09-21T12:39:20.300121+0000","last_fullsized":"2024-09-21T12:39:20.300121+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'14","last_scrub_stamp":"2024-09-21T12:36:31.838024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:31.838024+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:11.567448+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":307,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.384894+0000","last_change":"2024-09-21T12:38:07.164086+0000","last_active":"2024-09-21T12:39:19.384894+0000","last_peered":"2024-09-21T12:39:19.384894+0000","last_clean":"2024-09-21T12:39:19.384894+0000","last_became_active":"2024-09-21T12:38:07.163242+0000","last_became_peered":"2024-09-21T12:38:07.163242+0000","last_unstale":"2024-09-21T12:39:19.384894+0000","last_undegraded":"2024-09-21T12:39:19.384894+0000","last_fullsized":"2024-09-21T12:39:19.384894+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:06.369436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:06.369436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:59.467726+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.e","version":"76'79","reported_seq":316,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.385228+0000","last_change":"2024-09-21T12:37:48.990420+0000","last_active":"2024-09-21T12:39:19.385228+0000","last_peered":"2024-09-21T12:39:19.385228+0000","last_clean":"2024-09-21T12:39:19.385228+0000","last_became_active":"2024-09-21T12:36:01.159481+0000","last_became_peered":"2024-09-21T12:36:01.159481+0000","last_unstale":"2024-09-21T12:39:19.385228+0000","last_undegraded":"2024-09-21T12:39:19.385228+0000","last_fullsized":"2024-09-21T12:39:19.385228+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:37:48.990356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:37:48.990356+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"276'15","reported_seq":1023,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.393777+0000","last_change":"2024-09-21T12:36:40.771296+0000","last_active":"2024-09-21T12:39:22.393777+0000","last_peered":"2024-09-21T12:39:22.393777+0000","last_clean":"2024-09-21T12:39:22.393777+0000","last_became_active":"2024-09-21T12:32:38.424376+0000","last_became_peered":"2024-09-21T12:32:38.424376+0000","last_unstale":"2024-09-21T12:39:22.393777+0000","last_undegraded":"2024-09-21T12:39:22.393777+0000","last_fullsized":"2024-09-21T12:39:22.393777+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:36:40.771222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:40.771222+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":443,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.484078+0000","last_change":"2024-09-21T12:38:51.327197+0000","last_active":"2024-09-21T12:39:19.484078+0000","last_peered":"2024-09-21T12:39:19.484078+0000","last_clean":"2024-09-21T12:39:19.484078+0000","last_became_active":"2024-09-21T12:37:20.484773+0000","last_became_peered":"2024-09-21T12:37:20.484773+0000","last_unstale":"2024-09-21T12:39:19.484078+0000","last_undegraded":"2024-09-21T12:39:19.484078+0000","last_fullsized":"2024-09-21T12:39:19.484078+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:51.327127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:51.327127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:13.918622+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":878,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.384279+0000","last_change":"2024-09-21T12:38:38.252977+0000","last_active":"2024-09-21T12:39:19.384279+0000","last_peered":"2024-09-21T12:39:19.384279+0000","last_clean":"2024-09-21T12:39:19.384279+0000","last_became_active":"2024-09-21T12:38:07.165226+0000","last_became_peered":"2024-09-21T12:38:07.165226+0000","last_unstale":"2024-09-21T12:39:19.384279+0000","last_undegraded":"2024-09-21T12:39:19.384279+0000","last_fullsized":"2024-09-21T12:39:19.384279+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:38.252885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:38.252885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:06.853610+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":920,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.393369+0000","last_change":"2024-09-21T12:38:08.164968+0000","last_active":"2024-09-21T12:39:22.393369+0000","last_peered":"2024-09-21T12:39:22.393369+0000","last_clean":"2024-09-21T12:39:22.393369+0000","last_became_active":"2024-09-21T12:38:08.164626+0000","last_became_peered":"2024-09-21T12:38:08.164626+0000","last_unstale":"2024-09-21T12:39:22.393369+0000","last_undegraded":"2024-09-21T12:39:22.393369+0000","last_fullsized":"2024-09-21T12:39:22.393369+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:34.842666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:34.842666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"76'79","reported_seq":844,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.486538+0000","last_change":"2024-09-21T12:38:41.145390+0000","last_active":"2024-09-21T12:39:19.486538+0000","last_peered":"2024-09-21T12:39:19.486538+0000","last_clean":"2024-09-21T12:39:19.486538+0000","last_became_active":"2024-09-21T12:32:38.414877+0000","last_became_peered":"2024-09-21T12:32:38.414877+0000","last_unstale":"2024-09-21T12:39:19.486538+0000","last_undegraded":"2024-09-21T12:39:19.486538+0000","last_fullsized":"2024-09-21T12:39:19.486538+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:38:41.145341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:38:41.145341+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:00.633321+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"276'35","reported_seq":1089,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.481981+0000","last_change":"2024-09-21T12:38:44.587607+0000","last_active":"2024-09-21T12:39:19.481981+0000","last_peered":"2024-09-21T12:39:19.481981+0000","last_clean":"2024-09-21T12:39:19.481981+0000","last_became_active":"2024-09-21T12:36:00.148392+0000","last_became_peered":"2024-09-21T12:36:00.148392+0000","last_unstale":"2024-09-21T12:39:19.481981+0000","last_undegraded":"2024-09-21T12:39:19.481981+0000","last_fullsized":"2024-09-21T12:39:19.481981+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'35","last_scrub_stamp":"2024-09-21T12:38:44.587516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:44.587516+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:45.419078+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":165,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:20.300238+0000","last_change":"2024-09-21T12:38:53.700983+0000","last_active":"2024-09-21T12:39:20.300238+0000","last_peered":"2024-09-21T12:39:20.300238+0000","last_clean":"2024-09-21T12:39:20.300238+0000","last_became_active":"2024-09-21T12:36:00.145287+0000","last_became_peered":"2024-09-21T12:36:00.145287+0000","last_unstale":"2024-09-21T12:39:20.300238+0000","last_undegraded":"2024-09-21T12:39:20.300238+0000","last_fullsized":"2024-09-21T12:39:20.300238+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:53.700939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:53.700939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:55.132890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":810,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.393962+0000","last_change":"2024-09-21T12:38:07.162955+0000","last_active":"2024-09-21T12:39:22.393962+0000","last_peered":"2024-09-21T12:39:22.393962+0000","last_clean":"2024-09-21T12:39:22.393962+0000","last_became_active":"2024-09-21T12:38:07.162684+0000","last_became_peered":"2024-09-21T12:38:07.162684+0000","last_unstale":"2024-09-21T12:39:22.393962+0000","last_undegraded":"2024-09-21T12:39:22.393962+0000","last_fullsized":"2024-09-21T12:39:22.393962+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:20.243774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:20.243774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:05.650260+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":897,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:20.302042+0000","last_change":"2024-09-21T12:38:08.164642+0000","last_active":"2024-09-21T12:39:20.302042+0000","last_peered":"2024-09-21T12:39:20.302042+0000","last_clean":"2024-09-21T12:39:20.302042+0000","last_became_active":"2024-09-21T12:38:08.164339+0000","last_became_peered":"2024-09-21T12:38:08.164339+0000","last_unstale":"2024-09-21T12:39:20.302042+0000","last_undegraded":"2024-09-21T12:39:20.302042+0000","last_fullsized":"2024-09-21T12:39:20.302042+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:48.619605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:37:48.619605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"76'79","reported_seq":861,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.326984+0000","last_change":"2024-09-21T12:36:31.563764+0000","last_active":"2024-09-21T12:39:22.326984+0000","last_peered":"2024-09-21T12:39:22.326984+0000","last_clean":"2024-09-21T12:39:22.326984+0000","last_became_active":"2024-09-21T12:27:14.951375+0000","last_became_peered":"2024-09-21T12:27:14.951375+0000","last_unstale":"2024-09-21T12:39:22.326984+0000","last_undegraded":"2024-09-21T12:39:22.326984+0000","last_fullsized":"2024-09-21T12:39:22.326984+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:31.563694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:31.563694+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"276'18","reported_seq":317,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.384087+0000","last_change":"2024-09-21T12:37:52.992717+0000","last_active":"2024-09-21T12:39:19.384087+0000","last_peered":"2024-09-21T12:39:19.384087+0000","last_clean":"2024-09-21T12:39:19.384087+0000","last_became_active":"2024-09-21T12:32:38.421609+0000","last_became_peered":"2024-09-21T12:32:38.421609+0000","last_unstale":"2024-09-21T12:39:19.384087+0000","last_undegraded":"2024-09-21T12:39:19.384087+0000","last_fullsized":"2024-09-21T12:39:19.384087+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T12:37:52.992635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:37:52.992635+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":165,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:20.300664+0000","last_change":"2024-09-21T12:37:59.566316+0000","last_active":"2024-09-21T12:39:20.300664+0000","last_peered":"2024-09-21T12:39:20.300664+0000","last_clean":"2024-09-21T12:39:20.300664+0000","last_became_active":"2024-09-21T12:36:00.150832+0000","last_became_peered":"2024-09-21T12:36:00.150832+0000","last_unstale":"2024-09-21T12:39:20.300664+0000","last_undegraded":"2024-09-21T12:39:20.300664+0000","last_fullsized":"2024-09-21T12:39:20.300664+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:59.566268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:37:59.566268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:26.751431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":240,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.393395+0000","last_change":"2024-09-21T12:38:46.602449+0000","last_active":"2024-09-21T12:39:22.393395+0000","last_peered":"2024-09-21T12:39:22.393395+0000","last_clean":"2024-09-21T12:39:22.393395+0000","last_became_active":"2024-09-21T12:38:07.161907+0000","last_became_peered":"2024-09-21T12:38:07.161907+0000","last_unstale":"2024-09-21T12:39:22.393395+0000","last_undegraded":"2024-09-21T12:39:22.393395+0000","last_fullsized":"2024-09-21T12:39:22.393395+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:46.597509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:46.597509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:50.838226+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":931,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.326516+0000","last_change":"2024-09-21T12:38:00.902377+0000","last_active":"2024-09-21T12:39:22.326516+0000","last_peered":"2024-09-21T12:39:22.326516+0000","last_clean":"2024-09-21T12:39:22.326516+0000","last_became_active":"2024-09-21T12:36:00.139300+0000","last_became_peered":"2024-09-21T12:36:00.139300+0000","last_unstale":"2024-09-21T12:39:22.326516+0000","last_undegraded":"2024-09-21T12:39:22.326516+0000","last_fullsized":"2024-09-21T12:39:22.326516+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:00.901368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:38:00.901368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"76'79","reported_seq":337,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:20.301143+0000","last_change":"2024-09-21T12:36:36.859482+0000","last_active":"2024-09-21T12:39:20.301143+0000","last_peered":"2024-09-21T12:39:20.301143+0000","last_clean":"2024-09-21T12:39:20.301143+0000","last_became_active":"2024-09-21T12:36:01.141807+0000","last_became_peered":"2024-09-21T12:36:01.141807+0000","last_unstale":"2024-09-21T12:39:20.301143+0000","last_undegraded":"2024-09-21T12:39:20.301143+0000","last_fullsized":"2024-09-21T12:39:20.301143+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:36.859412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:36.859412+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:02.659187+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"276'27","reported_seq":446,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.485947+0000","last_change":"2024-09-21T12:37:56.066173+0000","last_active":"2024-09-21T12:39:19.485947+0000","last_peered":"2024-09-21T12:39:19.485947+0000","last_clean":"2024-09-21T12:39:19.485947+0000","last_became_active":"2024-09-21T12:36:00.137597+0000","last_became_peered":"2024-09-21T12:36:00.137597+0000","last_unstale":"2024-09-21T12:39:19.485947+0000","last_undegraded":"2024-09-21T12:39:19.485947+0000","last_fullsized":"2024-09-21T12:39:19.485947+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:37:56.066042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:37:56.066042+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":171,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:20.299667+0000","last_change":"2024-09-21T12:37:11.677270+0000","last_active":"2024-09-21T12:39:20.299667+0000","last_peered":"2024-09-21T12:39:20.299667+0000","last_clean":"2024-09-21T12:39:20.299667+0000","last_became_active":"2024-09-21T12:36:00.148899+0000","last_became_peered":"2024-09-21T12:36:00.148899+0000","last_unstale":"2024-09-21T12:39:20.299667+0000","last_undegraded":"2024-09-21T12:39:20.299667+0000","last_fullsized":"2024-09-21T12:39:20.299667+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-21T12:37:11.677196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:37:11.677196+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"415'16","reported_seq":210,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:20.299983+0000","last_change":"2024-09-21T12:38:00.562505+0000","last_active":"2024-09-21T12:39:20.299983+0000","last_peered":"2024-09-21T12:39:20.299983+0000","last_clean":"2024-09-21T12:39:20.299983+0000","last_became_active":"2024-09-21T12:36:00.143091+0000","last_became_peered":"2024-09-21T12:36:00.143091+0000","last_unstale":"2024-09-21T12:39:20.299983+0000","last_undegraded":"2024-09-21T12:39:20.299983+0000","last_fullsized":"2024-09-21T12:39:20.299983+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'16","last_scrub_stamp":"2024-09-21T12:38:00.562427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:00.562427+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":407,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.487820+0000","last_change":"2024-09-21T12:37:20.468878+0000","last_active":"2024-09-21T12:39:19.487820+0000","last_peered":"2024-09-21T12:39:19.487820+0000","last_clean":"2024-09-21T12:39:19.487820+0000","last_became_active":"2024-09-21T12:37:20.468350+0000","last_became_peered":"2024-09-21T12:37:20.468350+0000","last_unstale":"2024-09-21T12:39:19.487820+0000","last_undegraded":"2024-09-21T12:39:19.487820+0000","last_fullsized":"2024-09-21T12:39:19.487820+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:34.998795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:34.998795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:59.304496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"76'79","reported_seq":643,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.487655+0000","last_change":"2024-09-21T12:39:16.243982+0000","last_active":"2024-09-21T12:39:19.487655+0000","last_peered":"2024-09-21T12:39:19.487655+0000","last_clean":"2024-09-21T12:39:19.487655+0000","last_became_active":"2024-09-21T12:31:39.060422+0000","last_became_peered":"2024-09-21T12:31:39.060422+0000","last_unstale":"2024-09-21T12:39:19.487655+0000","last_undegraded":"2024-09-21T12:39:19.487655+0000","last_fullsized":"2024-09-21T12:39:19.487655+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:16.243927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:16.243927+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:40.547837+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":968,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.481714+0000","last_change":"2024-09-21T12:39:19.481714+0000","last_active":"2024-09-21T12:39:19.481714+0000","last_peered":"2024-09-21T12:39:19.481714+0000","last_clean":"2024-09-21T12:39:19.481714+0000","last_became_active":"2024-09-21T12:32:02.026693+0000","last_became_peered":"2024-09-21T12:32:02.026693+0000","last_unstale":"2024-09-21T12:39:19.481714+0000","last_undegraded":"2024-09-21T12:39:19.481714+0000","last_fullsized":"2024-09-21T12:39:19.481714+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:19.481652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:19.481652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:34.846777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"276'12","reported_seq":418,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.327363+0000","last_change":"2024-09-21T12:37:59.919992+0000","last_active":"2024-09-21T12:39:22.327363+0000","last_peered":"2024-09-21T12:39:22.327363+0000","last_clean":"2024-09-21T12:39:22.327363+0000","last_became_active":"2024-09-21T12:36:01.138626+0000","last_became_peered":"2024-09-21T12:36:01.138626+0000","last_unstale":"2024-09-21T12:39:22.327363+0000","last_undegraded":"2024-09-21T12:39:22.327363+0000","last_fullsized":"2024-09-21T12:39:22.327363+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T12:37:59.919947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:37:59.919947+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":427,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.484647+0000","last_change":"2024-09-21T12:37:20.486113+0000","last_active":"2024-09-21T12:39:19.484647+0000","last_peered":"2024-09-21T12:39:19.484647+0000","last_clean":"2024-09-21T12:39:19.484647+0000","last_became_active":"2024-09-21T12:37:20.485806+0000","last_became_peered":"2024-09-21T12:37:20.485806+0000","last_unstale":"2024-09-21T12:39:19.484647+0000","last_undegraded":"2024-09-21T12:39:19.484647+0000","last_fullsized":"2024-09-21T12:39:19.484647+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:42.510764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:42.510764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:57.271996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":165,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:20.300382+0000","last_change":"2024-09-21T12:38:03.579433+0000","last_active":"2024-09-21T12:39:20.300382+0000","last_peered":"2024-09-21T12:39:20.300382+0000","last_clean":"2024-09-21T12:39:20.300382+0000","last_became_active":"2024-09-21T12:36:00.149086+0000","last_became_peered":"2024-09-21T12:36:00.149086+0000","last_unstale":"2024-09-21T12:39:20.300382+0000","last_undegraded":"2024-09-21T12:39:20.300382+0000","last_fullsized":"2024-09-21T12:39:20.300382+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:03.579275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:38:03.579275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'79","reported_seq":891,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.386632+0000","last_change":"2024-09-21T12:37:53.523886+0000","last_active":"2024-09-21T12:39:19.386632+0000","last_peered":"2024-09-21T12:39:19.386632+0000","last_clean":"2024-09-21T12:39:19.386632+0000","last_became_active":"2024-09-21T12:32:38.425495+0000","last_became_peered":"2024-09-21T12:32:38.425495+0000","last_unstale":"2024-09-21T12:39:19.386632+0000","last_undegraded":"2024-09-21T12:39:19.386632+0000","last_fullsized":"2024-09-21T12:39:19.386632+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:37:53.523818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:37:53.523818+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"276'15","reported_seq":985,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.394043+0000","last_change":"2024-09-21T12:38:04.577293+0000","last_active":"2024-09-21T12:39:22.394043+0000","last_peered":"2024-09-21T12:39:22.394043+0000","last_clean":"2024-09-21T12:39:22.394043+0000","last_became_active":"2024-09-21T12:36:00.142274+0000","last_became_peered":"2024-09-21T12:36:00.142274+0000","last_unstale":"2024-09-21T12:39:22.394043+0000","last_undegraded":"2024-09-21T12:39:22.394043+0000","last_fullsized":"2024-09-21T12:39:22.394043+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:38:04.577219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:04.577219+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:22.773773+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":165,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:20.299750+0000","last_change":"2024-09-21T12:38:06.554109+0000","last_active":"2024-09-21T12:39:20.299750+0000","last_peered":"2024-09-21T12:39:20.299750+0000","last_clean":"2024-09-21T12:39:20.299750+0000","last_became_active":"2024-09-21T12:36:00.140477+0000","last_became_peered":"2024-09-21T12:36:00.140477+0000","last_unstale":"2024-09-21T12:39:20.299750+0000","last_undegraded":"2024-09-21T12:39:20.299750+0000","last_fullsized":"2024-09-21T12:39:20.299750+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:06.554062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:06.554062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":912,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.327273+0000","last_change":"2024-09-21T12:38:50.213195+0000","last_active":"2024-09-21T12:39:22.327273+0000","last_peered":"2024-09-21T12:39:22.327273+0000","last_clean":"2024-09-21T12:39:22.327273+0000","last_became_active":"2024-09-21T12:36:00.150975+0000","last_became_peered":"2024-09-21T12:36:00.150975+0000","last_unstale":"2024-09-21T12:39:22.327273+0000","last_undegraded":"2024-09-21T12:39:22.327273+0000","last_fullsized":"2024-09-21T12:39:22.327273+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:50.213138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:50.213138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:09.450880+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":893,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.384740+0000","last_change":"2024-09-21T12:38:08.150215+0000","last_active":"2024-09-21T12:39:19.384740+0000","last_peered":"2024-09-21T12:39:19.384740+0000","last_clean":"2024-09-21T12:39:19.384740+0000","last_became_active":"2024-09-21T12:38:08.149697+0000","last_became_peered":"2024-09-21T12:38:08.149697+0000","last_unstale":"2024-09-21T12:39:19.384740+0000","last_undegraded":"2024-09-21T12:39:19.384740+0000","last_fullsized":"2024-09-21T12:39:19.384740+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:57.998679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:37:57.998679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.10","version":"76'79","reported_seq":132,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.485755+0000","last_change":"2024-09-21T12:37:57.062032+0000","last_active":"2024-09-21T12:39:19.485755+0000","last_peered":"2024-09-21T12:39:19.485755+0000","last_clean":"2024-09-21T12:39:19.485755+0000","last_became_active":"2024-09-21T12:37:20.467540+0000","last_became_peered":"2024-09-21T12:37:20.467540+0000","last_unstale":"2024-09-21T12:39:19.485755+0000","last_undegraded":"2024-09-21T12:39:19.485755+0000","last_fullsized":"2024-09-21T12:39:19.485755+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:37:57.061984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:37:57.061984+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"276'15","reported_seq":468,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.486861+0000","last_change":"2024-09-21T12:38:39.157782+0000","last_active":"2024-09-21T12:39:19.486861+0000","last_peered":"2024-09-21T12:39:19.486861+0000","last_clean":"2024-09-21T12:39:19.486861+0000","last_became_active":"2024-09-21T12:38:08.164375+0000","last_became_peered":"2024-09-21T12:38:08.164375+0000","last_unstale":"2024-09-21T12:39:19.486861+0000","last_undegraded":"2024-09-21T12:39:19.486861+0000","last_fullsized":"2024-09-21T12:39:19.486861+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:38:39.157727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:39.157727+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:00.959002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":961,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.393641+0000","last_change":"2024-09-21T12:36:00.127553+0000","last_active":"2024-09-21T12:39:22.393641+0000","last_peered":"2024-09-21T12:39:22.393641+0000","last_clean":"2024-09-21T12:39:22.393641+0000","last_became_active":"2024-09-21T12:36:00.127225+0000","last_became_peered":"2024-09-21T12:36:00.127225+0000","last_unstale":"2024-09-21T12:39:22.393641+0000","last_undegraded":"2024-09-21T12:39:22.393641+0000","last_fullsized":"2024-09-21T12:39:22.393641+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:22.225122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:22.225122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:06.616447+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"457'17","reported_seq":501,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.326738+0000","last_change":"2024-09-21T12:38:08.172081+0000","last_active":"2024-09-21T12:39:22.326738+0000","last_peered":"2024-09-21T12:39:22.326738+0000","last_clean":"2024-09-21T12:39:22.326738+0000","last_became_active":"2024-09-21T12:38:08.171750+0000","last_became_peered":"2024-09-21T12:38:08.171750+0000","last_unstale":"2024-09-21T12:39:22.326738+0000","last_undegraded":"2024-09-21T12:39:22.326738+0000","last_fullsized":"2024-09-21T12:39:22.326738+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"416'15","last_scrub_stamp":"2024-09-21T12:37:12.744729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:37:12.744729+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":307,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.385093+0000","last_change":"2024-09-21T12:38:07.163535+0000","last_active":"2024-09-21T12:39:19.385093+0000","last_peered":"2024-09-21T12:39:19.385093+0000","last_clean":"2024-09-21T12:39:19.385093+0000","last_became_active":"2024-09-21T12:38:07.163162+0000","last_became_peered":"2024-09-21T12:38:07.163162+0000","last_unstale":"2024-09-21T12:39:19.385093+0000","last_undegraded":"2024-09-21T12:39:19.385093+0000","last_fullsized":"2024-09-21T12:39:19.385093+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:33:05.370263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:33:05.370263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:58.482168+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"76'79","reported_seq":736,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.393479+0000","last_change":"2024-09-21T12:38:00.556820+0000","last_active":"2024-09-21T12:39:22.393479+0000","last_peered":"2024-09-21T12:39:22.393479+0000","last_clean":"2024-09-21T12:39:22.393479+0000","last_became_active":"2024-09-21T12:32:02.034387+0000","last_became_peered":"2024-09-21T12:32:02.034387+0000","last_unstale":"2024-09-21T12:39:22.393479+0000","last_undegraded":"2024-09-21T12:39:22.393479+0000","last_fullsized":"2024-09-21T12:39:22.393479+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:38:00.556755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:38:00.556755+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"276'20","reported_seq":992,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.327064+0000","last_change":"2024-09-21T12:36:36.521884+0000","last_active":"2024-09-21T12:39:22.327064+0000","last_peered":"2024-09-21T12:39:22.327064+0000","last_clean":"2024-09-21T12:39:22.327064+0000","last_became_active":"2024-09-21T12:36:00.155033+0000","last_became_peered":"2024-09-21T12:36:00.155033+0000","last_unstale":"2024-09-21T12:39:22.327064+0000","last_undegraded":"2024-09-21T12:39:22.327064+0000","last_fullsized":"2024-09-21T12:39:22.327064+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:36:36.521810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:36.521810+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":937,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:20.299462+0000","last_change":"2024-09-21T12:38:08.165195+0000","last_active":"2024-09-21T12:39:20.299462+0000","last_peered":"2024-09-21T12:39:20.299462+0000","last_clean":"2024-09-21T12:39:20.299462+0000","last_became_active":"2024-09-21T12:38:08.164312+0000","last_became_peered":"2024-09-21T12:38:08.164312+0000","last_unstale":"2024-09-21T12:39:20.299462+0000","last_undegraded":"2024-09-21T12:39:20.299462+0000","last_fullsized":"2024-09-21T12:39:20.299462+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:18.659683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:37:18.659683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":1151,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.487239+0000","last_change":"2024-09-21T12:38:45.275249+0000","last_active":"2024-09-21T12:39:19.487239+0000","last_peered":"2024-09-21T12:39:19.487239+0000","last_clean":"2024-09-21T12:39:19.487239+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:39:19.487239+0000","last_undegraded":"2024-09-21T12:39:19.487239+0000","last_fullsized":"2024-09-21T12:39:19.487239+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:38:45.275110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:45.275110+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:10.836660+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":925,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.393817+0000","last_change":"2024-09-21T12:37:59.582535+0000","last_active":"2024-09-21T12:39:22.393817+0000","last_peered":"2024-09-21T12:39:22.393817+0000","last_clean":"2024-09-21T12:39:22.393817+0000","last_became_active":"2024-09-21T12:36:00.148332+0000","last_became_peered":"2024-09-21T12:36:00.148332+0000","last_unstale":"2024-09-21T12:39:22.393817+0000","last_undegraded":"2024-09-21T12:39:22.393817+0000","last_fullsized":"2024-09-21T12:39:22.393817+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:59.582483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:37:59.582483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"76'79","reported_seq":591,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.388222+0000","last_change":"2024-09-21T12:38:03.539003+0000","last_active":"2024-09-21T12:39:19.388222+0000","last_peered":"2024-09-21T12:39:19.388222+0000","last_clean":"2024-09-21T12:39:19.388222+0000","last_became_active":"2024-09-21T12:27:16.940347+0000","last_became_peered":"2024-09-21T12:27:16.940347+0000","last_unstale":"2024-09-21T12:39:19.388222+0000","last_undegraded":"2024-09-21T12:39:19.388222+0000","last_fullsized":"2024-09-21T12:39:19.388222+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:38:03.538881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:38:03.538881+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:30.585103+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"276'19","reported_seq":308,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.384295+0000","last_change":"2024-09-21T12:34:00.191190+0000","last_active":"2024-09-21T12:39:19.384295+0000","last_peered":"2024-09-21T12:39:19.384295+0000","last_clean":"2024-09-21T12:39:19.384295+0000","last_became_active":"2024-09-21T12:32:38.421008+0000","last_became_peered":"2024-09-21T12:32:38.421008+0000","last_unstale":"2024-09-21T12:39:19.384295+0000","last_undegraded":"2024-09-21T12:39:19.384295+0000","last_fullsized":"2024-09-21T12:39:19.384295+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:34:00.191106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:34:00.191106+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:01.433046+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":316,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.384620+0000","last_change":"2024-09-21T12:37:49.940962+0000","last_active":"2024-09-21T12:39:19.384620+0000","last_peered":"2024-09-21T12:39:19.384620+0000","last_clean":"2024-09-21T12:39:19.384620+0000","last_became_active":"2024-09-21T12:36:01.123399+0000","last_became_peered":"2024-09-21T12:36:01.123399+0000","last_unstale":"2024-09-21T12:39:19.384620+0000","last_undegraded":"2024-09-21T12:39:19.384620+0000","last_fullsized":"2024-09-21T12:39:19.384620+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:49.940903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:37:49.940903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":923,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.386436+0000","last_change":"2024-09-21T12:38:00.521861+0000","last_active":"2024-09-21T12:39:19.386436+0000","last_peered":"2024-09-21T12:39:19.386436+0000","last_clean":"2024-09-21T12:39:19.386436+0000","last_became_active":"2024-09-21T12:31:50.040754+0000","last_became_peered":"2024-09-21T12:31:50.040754+0000","last_unstale":"2024-09-21T12:39:19.386436+0000","last_undegraded":"2024-09-21T12:39:19.386436+0000","last_fullsized":"2024-09-21T12:39:19.386436+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:00.521803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:00.521803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":375,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.326327+0000","last_change":"2024-09-21T12:37:19.857159+0000","last_active":"2024-09-21T12:39:22.326327+0000","last_peered":"2024-09-21T12:39:22.326327+0000","last_clean":"2024-09-21T12:39:22.326327+0000","last_became_active":"2024-09-21T12:31:50.036150+0000","last_became_peered":"2024-09-21T12:31:50.036150+0000","last_unstale":"2024-09-21T12:39:22.326327+0000","last_undegraded":"2024-09-21T12:39:22.326327+0000","last_fullsized":"2024-09-21T12:39:22.326327+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:19.857108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:37:19.857108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"76'79","reported_seq":616,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.486766+0000","last_change":"2024-09-21T12:36:45.476276+0000","last_active":"2024-09-21T12:39:19.486766+0000","last_peered":"2024-09-21T12:39:19.486766+0000","last_clean":"2024-09-21T12:39:19.486766+0000","last_became_active":"2024-09-21T12:27:16.921880+0000","last_became_peered":"2024-09-21T12:27:16.921880+0000","last_unstale":"2024-09-21T12:39:19.486766+0000","last_undegraded":"2024-09-21T12:39:19.486766+0000","last_fullsized":"2024-09-21T12:39:19.486766+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:45.476087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:45.476087+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:05.179564+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"276'15","reported_seq":410,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.386201+0000","last_change":"2024-09-21T12:38:47.987378+0000","last_active":"2024-09-21T12:39:19.386201+0000","last_peered":"2024-09-21T12:39:19.386201+0000","last_clean":"2024-09-21T12:39:19.386201+0000","last_became_active":"2024-09-21T12:38:07.153467+0000","last_became_peered":"2024-09-21T12:38:07.153467+0000","last_unstale":"2024-09-21T12:39:19.386201+0000","last_undegraded":"2024-09-21T12:39:19.386201+0000","last_fullsized":"2024-09-21T12:39:19.386201+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:38:47.987308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:47.987308+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:49.248198+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":165,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:20.300192+0000","last_change":"2024-09-21T12:38:05.529592+0000","last_active":"2024-09-21T12:39:20.300192+0000","last_peered":"2024-09-21T12:39:20.300192+0000","last_clean":"2024-09-21T12:39:20.300192+0000","last_became_active":"2024-09-21T12:36:00.147063+0000","last_became_peered":"2024-09-21T12:36:00.147063+0000","last_unstale":"2024-09-21T12:39:20.300192+0000","last_undegraded":"2024-09-21T12:39:20.300192+0000","last_fullsized":"2024-09-21T12:39:20.300192+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:05.529479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:05.529479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:29.024607+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":907,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.326577+0000","last_change":"2024-09-21T12:37:50.035009+0000","last_active":"2024-09-21T12:39:22.326577+0000","last_peered":"2024-09-21T12:39:22.326577+0000","last_clean":"2024-09-21T12:39:22.326577+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:39:22.326577+0000","last_undegraded":"2024-09-21T12:39:22.326577+0000","last_fullsized":"2024-09-21T12:39:22.326577+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:50.034945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:37:50.034945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":890,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.393453+0000","last_change":"2024-09-21T12:35:59.959597+0000","last_active":"2024-09-21T12:39:22.393453+0000","last_peered":"2024-09-21T12:39:22.393453+0000","last_clean":"2024-09-21T12:39:22.393453+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:39:22.393453+0000","last_undegraded":"2024-09-21T12:39:22.393453+0000","last_fullsized":"2024-09-21T12:39:22.393453+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:59.959545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:59.959545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:14.642421+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"76'79","reported_seq":398,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:20.299862+0000","last_change":"2024-09-21T12:38:08.157233+0000","last_active":"2024-09-21T12:39:20.299862+0000","last_peered":"2024-09-21T12:39:20.299862+0000","last_clean":"2024-09-21T12:39:20.299862+0000","last_became_active":"2024-09-21T12:38:08.156834+0000","last_became_peered":"2024-09-21T12:38:08.156834+0000","last_unstale":"2024-09-21T12:39:20.299862+0000","last_undegraded":"2024-09-21T12:39:20.299862+0000","last_fullsized":"2024-09-21T12:39:20.299862+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:36:43.792355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:36:43.792355+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:57.778271+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"276'19","reported_seq":980,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.326563+0000","last_change":"2024-09-21T12:36:41.506309+0000","last_active":"2024-09-21T12:39:22.326563+0000","last_peered":"2024-09-21T12:39:22.326563+0000","last_clean":"2024-09-21T12:39:22.326563+0000","last_became_active":"2024-09-21T12:36:00.139682+0000","last_became_peered":"2024-09-21T12:36:00.139682+0000","last_unstale":"2024-09-21T12:39:22.326563+0000","last_undegraded":"2024-09-21T12:39:22.326563+0000","last_fullsized":"2024-09-21T12:39:22.326563+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:36:41.506231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:36:41.506231+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":943,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.484660+0000","last_change":"2024-09-21T12:36:34.526618+0000","last_active":"2024-09-21T12:39:19.484660+0000","last_peered":"2024-09-21T12:39:19.484660+0000","last_clean":"2024-09-21T12:39:19.484660+0000","last_became_active":"2024-09-21T12:36:00.151584+0000","last_became_peered":"2024-09-21T12:36:00.151584+0000","last_unstale":"2024-09-21T12:39:19.484660+0000","last_undegraded":"2024-09-21T12:39:19.484660+0000","last_fullsized":"2024-09-21T12:39:19.484660+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:34.526551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:36:34.526551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:56.283290+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":165,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:20.299905+0000","last_change":"2024-09-21T12:37:51.569522+0000","last_active":"2024-09-21T12:39:20.299905+0000","last_peered":"2024-09-21T12:39:20.299905+0000","last_clean":"2024-09-21T12:39:20.299905+0000","last_became_active":"2024-09-21T12:36:00.144521+0000","last_became_peered":"2024-09-21T12:36:00.144521+0000","last_unstale":"2024-09-21T12:39:20.299905+0000","last_undegraded":"2024-09-21T12:39:20.299905+0000","last_fullsized":"2024-09-21T12:39:20.299905+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:37:51.569462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:37:51.569462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":895,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.326859+0000","last_change":"2024-09-21T12:35:51.628370+0000","last_active":"2024-09-21T12:39:22.326859+0000","last_peered":"2024-09-21T12:39:22.326859+0000","last_clean":"2024-09-21T12:39:22.326859+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:39:22.326859+0000","last_undegraded":"2024-09-21T12:39:22.326859+0000","last_fullsized":"2024-09-21T12:39:22.326859+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:51.628326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:35:51.628326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"76'79","reported_seq":165,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:20.300623+0000","last_change":"2024-09-21T12:38:54.700596+0000","last_active":"2024-09-21T12:39:20.300623+0000","last_peered":"2024-09-21T12:39:20.300623+0000","last_clean":"2024-09-21T12:39:20.300623+0000","last_became_active":"2024-09-21T12:36:00.151043+0000","last_became_peered":"2024-09-21T12:36:00.151043+0000","last_unstale":"2024-09-21T12:39:20.300623+0000","last_undegraded":"2024-09-21T12:39:20.300623+0000","last_fullsized":"2024-09-21T12:39:20.300623+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:38:54.700420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:38:54.700420+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:56.059368+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"276'8","reported_seq":915,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.326421+0000","last_change":"2024-09-21T12:35:14.789148+0000","last_active":"2024-09-21T12:39:22.326421+0000","last_peered":"2024-09-21T12:39:22.326421+0000","last_clean":"2024-09-21T12:39:22.326421+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:39:22.326421+0000","last_undegraded":"2024-09-21T12:39:22.326421+0000","last_fullsized":"2024-09-21T12:39:22.326421+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'8","last_scrub_stamp":"2024-09-21T12:35:14.789070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:35:14.789070+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:12.255626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":923,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.394087+0000","last_change":"2024-09-21T12:35:27.237430+0000","last_active":"2024-09-21T12:39:22.394087+0000","last_peered":"2024-09-21T12:39:22.394087+0000","last_clean":"2024-09-21T12:39:22.394087+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:39:22.394087+0000","last_undegraded":"2024-09-21T12:39:22.394087+0000","last_fullsized":"2024-09-21T12:39:22.394087+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:35:27.236980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:27.236980+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:07.615108+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":940,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.481351+0000","last_change":"2024-09-21T12:36:00.135914+0000","last_active":"2024-09-21T12:39:19.481351+0000","last_peered":"2024-09-21T12:39:19.481351+0000","last_clean":"2024-09-21T12:39:19.481351+0000","last_became_active":"2024-09-21T12:36:00.135537+0000","last_became_peered":"2024-09-21T12:36:00.135537+0000","last_unstale":"2024-09-21T12:39:19.481351+0000","last_undegraded":"2024-09-21T12:39:19.481351+0000","last_fullsized":"2024-09-21T12:39:19.481351+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:34:40.861493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:34:40.861493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:34.475132+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":925,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:20.299438+0000","last_change":"2024-09-21T12:38:08.164865+0000","last_active":"2024-09-21T12:39:20.299438+0000","last_peered":"2024-09-21T12:39:20.299438+0000","last_clean":"2024-09-21T12:39:20.299438+0000","last_became_active":"2024-09-21T12:38:08.164535+0000","last_became_peered":"2024-09-21T12:38:08.164535+0000","last_unstale":"2024-09-21T12:39:20.299438+0000","last_undegraded":"2024-09-21T12:39:20.299438+0000","last_fullsized":"2024-09-21T12:39:20.299438+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:38.838688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:36:38.838688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:04.705707+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"76'79","reported_seq":310,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.384189+0000","last_change":"2024-09-21T12:36:00.137642+0000","last_active":"2024-09-21T12:39:19.384189+0000","last_peered":"2024-09-21T12:39:19.384189+0000","last_clean":"2024-09-21T12:39:19.384189+0000","last_became_active":"2024-09-21T12:36:00.137290+0000","last_became_peered":"2024-09-21T12:36:00.137290+0000","last_unstale":"2024-09-21T12:39:19.384189+0000","last_undegraded":"2024-09-21T12:39:19.384189+0000","last_fullsized":"2024-09-21T12:39:19.384189+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:34:43.304606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:34:43.304606+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:05.436780+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":929,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.385224+0000","last_change":"2024-09-21T12:38:51.954801+0000","last_active":"2024-09-21T12:39:19.385224+0000","last_peered":"2024-09-21T12:39:19.385224+0000","last_clean":"2024-09-21T12:39:19.385224+0000","last_became_active":"2024-09-21T12:32:38.426822+0000","last_became_peered":"2024-09-21T12:32:38.426822+0000","last_unstale":"2024-09-21T12:39:19.385224+0000","last_undegraded":"2024-09-21T12:39:19.385224+0000","last_fullsized":"2024-09-21T12:39:19.385224+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:51.954739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:38:51.954739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:53.948595+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":920,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.326714+0000","last_change":"2024-09-21T12:35:30.730640+0000","last_active":"2024-09-21T12:39:22.326714+0000","last_peered":"2024-09-21T12:39:22.326714+0000","last_clean":"2024-09-21T12:39:22.326714+0000","last_became_active":"2024-09-21T12:31:39.052486+0000","last_became_peered":"2024-09-21T12:31:39.052486+0000","last_unstale":"2024-09-21T12:39:22.326714+0000","last_undegraded":"2024-09-21T12:39:22.326714+0000","last_fullsized":"2024-09-21T12:39:22.326714+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:35:30.730574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:35:30.730574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"276'19","reported_seq":942,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-21T12:39:19.481390+0000","last_change":"2024-09-21T12:38:41.504397+0000","last_active":"2024-09-21T12:39:19.481390+0000","last_peered":"2024-09-21T12:39:19.481390+0000","last_clean":"2024-09-21T12:39:19.481390+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:39:19.481390+0000","last_undegraded":"2024-09-21T12:39:19.481390+0000","last_fullsized":"2024-09-21T12:39:19.481390+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:38:41.504320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:41.504320+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:39:48.172302+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":949,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-21T12:39:22.393905+0000","last_change":"2024-09-21T12:38:08.175436+0000","last_active":"2024-09-21T12:39:22.393905+0000","last_peered":"2024-09-21T12:39:22.393905+0000","last_clean":"2024-09-21T12:39:22.393905+0000","last_became_active":"2024-09-21T12:38:08.174864+0000","last_became_peered":"2024-09-21T12:38:08.174864+0000","last_unstale":"2024-09-21T12:39:22.393905+0000","last_undegraded":"2024-09-21T12:39:22.393905+0000","last_fullsized":"2024-09-21T12:39:22.393905+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:28.911328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:36:28.911328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1105,"num_read_kb":862,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":472,"num_bytes_recovered":9372,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3597,"internal_metadata":0},"log_size":688,"ondisk_log_size":688,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":598,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":208,"num_read_kb":203,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2449,"ondisk_log_size":2449,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234217,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":845600,"kb_used_data":2252,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":92899552,"statfs":{"total":95995035648,"available":95129141248,"internally_reserved":0,"allocated":2306048,"data_stored":1494273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":863566282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463153,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":885900,"kb_used_data":2296,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":92859252,"statfs":{"total":95995035648,"available":95087874048,"internally_reserved":0,"allocated":2351104,"data_stored":1558616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":904788422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724788,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":781732,"kb_used_data":2256,"kb_used_omap":1,"kb_used_meta":779454,"kb_avail":92963420,"statfs":{"total":95995035648,"available":95194542080,"internally_reserved":0,"allocated":2310144,"data_stored":1495209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":798161349},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953724,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":408040,"kb_used_data":2000,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":93337112,"statfs":{"total":95995035648,"available":95577202688,"internally_reserved":0,"allocated":2048000,"data_stored":1231105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149954,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":926208,"kb_used_data":2344,"kb_used_omap":1,"kb_used_meta":923838,"kb_avail":92818944,"statfs":{"total":95995035648,"available":95046598656,"internally_reserved":0,"allocated":2400256,"data_stored":1568279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":946010567},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":455,"seq":1954210119696,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":230592,"kb_used_data":2220,"kb_used_omap":7,"kb_used_meta":228344,"kb_avail":93514560,"statfs":{"total":95995035648,"available":95758909440,"internally_reserved":0,"allocated":2273280,"data_stored":1485294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8042,"internal_metadata":233824406},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575118,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":993920,"kb_used_data":2028,"kb_used_omap":1,"kb_used_meta":991870,"kb_avail":92751232,"statfs":{"total":95995035648,"available":94977261568,"internally_reserved":0,"allocated":2076672,"data_stored":1231968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1015675333},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738644,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":996212,"kb_used_data":66784,"kb_used_omap":1,"kb_used_meta":929406,"kb_avail":92748940,"statfs":{"total":95995035648,"available":94974914560,"internally_reserved":0,"allocated":68386816,"data_stored":67562621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":951712197},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3597,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:39:25.853 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T12:39:25.853 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pg_num 61 2024-09-21T12:39:25.854 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:39:26.342 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:39:26.463 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:26 smithi114 ceph-mon[40867]: from='client.26468 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:39:26.463 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:26 smithi114 ceph-mon[40867]: 3.18 scrub starts 2024-09-21T12:39:26.463 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:26 smithi114 ceph-mon[40867]: 3.18 scrub ok 2024-09-21T12:39:26.464 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:26 smithi114 ceph-mon[30324]: from='client.26468 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:39:26.464 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:26 smithi114 ceph-mon[30324]: 3.18 scrub starts 2024-09-21T12:39:26.464 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:26 smithi114 ceph-mon[30324]: 3.18 scrub ok 2024-09-21T12:39:26.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:26 smithi183 ceph-mon[33277]: from='client.26468 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:39:26.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:26 smithi183 ceph-mon[33277]: 3.18 scrub starts 2024-09-21T12:39:26.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:26 smithi183 ceph-mon[33277]: 3.18 scrub ok 2024-09-21T12:39:26.985 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: specified pg_num 61 < pg_num_max 32 2024-09-21T12:39:27.184 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-21T12:39:27.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:27 smithi114 ceph-mon[40867]: 1.18 scrub starts 2024-09-21T12:39:27.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:27 smithi114 ceph-mon[40867]: 1.18 scrub ok 2024-09-21T12:39:27.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:27 smithi114 ceph-mon[40867]: pgmap v982: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-21T12:39:27.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:27 smithi114 ceph-mon[40867]: 6.13 scrub starts 2024-09-21T12:39:27.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:27 smithi114 ceph-mon[40867]: 6.13 scrub ok 2024-09-21T12:39:27.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:27 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3793694441' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:39:27.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:27 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/176463157' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "61"} : dispatch 2024-09-21T12:39:27.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:27 smithi114 ceph-mon[30324]: 1.18 scrub starts 2024-09-21T12:39:27.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:27 smithi114 ceph-mon[30324]: 1.18 scrub ok 2024-09-21T12:39:27.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:27 smithi114 ceph-mon[30324]: pgmap v982: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-21T12:39:27.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:27 smithi114 ceph-mon[30324]: 6.13 scrub starts 2024-09-21T12:39:27.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:27 smithi114 ceph-mon[30324]: 6.13 scrub ok 2024-09-21T12:39:27.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:27 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3793694441' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:39:27.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:27 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/176463157' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "61"} : dispatch 2024-09-21T12:39:27.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:27 smithi183 ceph-mon[33277]: 1.18 scrub starts 2024-09-21T12:39:27.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:27 smithi183 ceph-mon[33277]: 1.18 scrub ok 2024-09-21T12:39:27.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:27 smithi183 ceph-mon[33277]: pgmap v982: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-21T12:39:27.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:27 smithi183 ceph-mon[33277]: 6.13 scrub starts 2024-09-21T12:39:27.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:27 smithi183 ceph-mon[33277]: 6.13 scrub ok 2024-09-21T12:39:27.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:27 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3793694441' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T12:39:27.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:27 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/176463157' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "61"} : dispatch 2024-09-21T12:39:27.995 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T12:39:28.186 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:39:27 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:39:28.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:28 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3793694441' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:39:28.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:28 smithi114 ceph-mon[30324]: osdmap e489: 8 total, 8 up, 8 in 2024-09-21T12:39:28.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:28 smithi114 ceph-mon[30324]: 4.1d scrub starts 2024-09-21T12:39:28.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:28 smithi114 ceph-mon[30324]: 4.1d scrub ok 2024-09-21T12:39:28.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:28 smithi114 ceph-mon[30324]: 1.e scrub starts 2024-09-21T12:39:28.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:28 smithi114 ceph-mon[30324]: 1.e scrub ok 2024-09-21T12:39:28.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:28 smithi114 ceph-mon[30324]: 1.0 scrub starts 2024-09-21T12:39:28.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:28 smithi114 ceph-mon[30324]: 1.0 scrub ok 2024-09-21T12:39:28.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:28 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3793694441' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:39:28.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:28 smithi114 ceph-mon[40867]: osdmap e489: 8 total, 8 up, 8 in 2024-09-21T12:39:28.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:28 smithi114 ceph-mon[40867]: 4.1d scrub starts 2024-09-21T12:39:28.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:28 smithi114 ceph-mon[40867]: 4.1d scrub ok 2024-09-21T12:39:28.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:28 smithi114 ceph-mon[40867]: 1.e scrub starts 2024-09-21T12:39:28.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:28 smithi114 ceph-mon[40867]: 1.e scrub ok 2024-09-21T12:39:28.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:28 smithi114 ceph-mon[40867]: 1.0 scrub starts 2024-09-21T12:39:28.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:28 smithi114 ceph-mon[40867]: 1.0 scrub ok 2024-09-21T12:39:28.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:28 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3793694441' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T12:39:28.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:28 smithi183 ceph-mon[33277]: osdmap e489: 8 total, 8 up, 8 in 2024-09-21T12:39:28.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:28 smithi183 ceph-mon[33277]: 4.1d scrub starts 2024-09-21T12:39:28.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:28 smithi183 ceph-mon[33277]: 4.1d scrub ok 2024-09-21T12:39:28.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:28 smithi183 ceph-mon[33277]: 1.e scrub starts 2024-09-21T12:39:28.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:28 smithi183 ceph-mon[33277]: 1.e scrub ok 2024-09-21T12:39:28.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:28 smithi183 ceph-mon[33277]: 1.0 scrub starts 2024-09-21T12:39:28.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:28 smithi183 ceph-mon[33277]: 1.0 scrub ok 2024-09-21T12:39:28.497 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T12:39:28.579 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:39:29.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:29 smithi183 ceph-mon[33277]: 4.b scrub starts 2024-09-21T12:39:29.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:29 smithi183 ceph-mon[33277]: 4.b scrub ok 2024-09-21T12:39:29.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:29 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:39:29.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:29 smithi183 ceph-mon[33277]: 6.4 scrub starts 2024-09-21T12:39:29.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:29 smithi183 ceph-mon[33277]: 6.4 scrub ok 2024-09-21T12:39:29.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:29 smithi183 ceph-mon[33277]: pgmap v984: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:39:29.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:29 smithi183 ceph-mon[33277]: 1.c scrub starts 2024-09-21T12:39:29.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:29 smithi183 ceph-mon[33277]: 1.c scrub ok 2024-09-21T12:39:29.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:29 smithi114 ceph-mon[30324]: 4.b scrub starts 2024-09-21T12:39:29.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:29 smithi114 ceph-mon[30324]: 4.b scrub ok 2024-09-21T12:39:29.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:29 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:39:29.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:29 smithi114 ceph-mon[30324]: 6.4 scrub starts 2024-09-21T12:39:29.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:29 smithi114 ceph-mon[30324]: 6.4 scrub ok 2024-09-21T12:39:29.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:29 smithi114 ceph-mon[30324]: pgmap v984: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:39:29.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:29 smithi114 ceph-mon[30324]: 1.c scrub starts 2024-09-21T12:39:29.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:29 smithi114 ceph-mon[30324]: 1.c scrub ok 2024-09-21T12:39:29.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:29 smithi114 ceph-mon[40867]: 4.b scrub starts 2024-09-21T12:39:29.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:29 smithi114 ceph-mon[40867]: 4.b scrub ok 2024-09-21T12:39:29.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:29 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:39:29.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:29 smithi114 ceph-mon[40867]: 6.4 scrub starts 2024-09-21T12:39:29.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:29 smithi114 ceph-mon[40867]: 6.4 scrub ok 2024-09-21T12:39:29.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:29 smithi114 ceph-mon[40867]: pgmap v984: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:39:29.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:29 smithi114 ceph-mon[40867]: 1.c scrub starts 2024-09-21T12:39:29.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:29 smithi114 ceph-mon[40867]: 1.c scrub ok 2024-09-21T12:39:30.211 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-21T12:39:30.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:30 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/23617584' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:39:30.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:30 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:39:30.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:30 smithi183 ceph-mon[33277]: 3.3 scrub starts 2024-09-21T12:39:30.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:30 smithi183 ceph-mon[33277]: 3.3 scrub ok 2024-09-21T12:39:30.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:30 smithi183 ceph-mon[33277]: 3.f scrub starts 2024-09-21T12:39:30.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:30 smithi183 ceph-mon[33277]: 3.f scrub ok 2024-09-21T12:39:30.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:30 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/23617584' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:39:30.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:30 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:39:30.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:30 smithi114 ceph-mon[30324]: 3.3 scrub starts 2024-09-21T12:39:30.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:30 smithi114 ceph-mon[30324]: 3.3 scrub ok 2024-09-21T12:39:30.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:30 smithi114 ceph-mon[30324]: 3.f scrub starts 2024-09-21T12:39:30.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:30 smithi114 ceph-mon[30324]: 3.f scrub ok 2024-09-21T12:39:30.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:30 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/23617584' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:39:30.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:30 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T12:39:30.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:30 smithi114 ceph-mon[40867]: 3.3 scrub starts 2024-09-21T12:39:30.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:30 smithi114 ceph-mon[40867]: 3.3 scrub ok 2024-09-21T12:39:30.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:30 smithi114 ceph-mon[40867]: 3.f scrub starts 2024-09-21T12:39:30.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:30 smithi114 ceph-mon[40867]: 3.f scrub ok 2024-09-21T12:39:31.274 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:31 smithi114 ceph-mon[40867]: 1.13 scrub starts 2024-09-21T12:39:31.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:31 smithi114 ceph-mon[40867]: 1.13 scrub ok 2024-09-21T12:39:31.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:31 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:39:31.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:31 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:39:31.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:31 smithi114 ceph-mon[40867]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:39:31.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:31 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:39:31.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:31 smithi114 ceph-mon[40867]: osdmap e490: 8 total, 8 up, 8 in 2024-09-21T12:39:31.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:31 smithi114 ceph-mon[40867]: 1.1 scrub starts 2024-09-21T12:39:31.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:31 smithi114 ceph-mon[40867]: 1.1 scrub ok 2024-09-21T12:39:31.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:31 smithi114 ceph-mon[40867]: 1.4 scrub starts 2024-09-21T12:39:31.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:31 smithi114 ceph-mon[40867]: 1.4 scrub ok 2024-09-21T12:39:31.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:31 smithi114 ceph-mon[40867]: pgmap v986: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T12:39:31.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:31 smithi114 ceph-mon[30324]: 1.13 scrub starts 2024-09-21T12:39:31.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:31 smithi114 ceph-mon[30324]: 1.13 scrub ok 2024-09-21T12:39:31.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:31 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:39:31.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:31 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:39:31.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:31 smithi114 ceph-mon[30324]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:39:31.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:31 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:39:31.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:31 smithi114 ceph-mon[30324]: osdmap e490: 8 total, 8 up, 8 in 2024-09-21T12:39:31.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:31 smithi114 ceph-mon[30324]: 1.1 scrub starts 2024-09-21T12:39:31.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:31 smithi114 ceph-mon[30324]: 1.1 scrub ok 2024-09-21T12:39:31.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:31 smithi114 ceph-mon[30324]: 1.4 scrub starts 2024-09-21T12:39:31.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:31 smithi114 ceph-mon[30324]: 1.4 scrub ok 2024-09-21T12:39:31.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:31 smithi114 ceph-mon[30324]: pgmap v986: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T12:39:31.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:31 smithi183 ceph-mon[33277]: 1.13 scrub starts 2024-09-21T12:39:31.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:31 smithi183 ceph-mon[33277]: 1.13 scrub ok 2024-09-21T12:39:31.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:31 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:39:31.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:31 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:39:31.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:31 smithi183 ceph-mon[33277]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T12:39:31.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:31 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T12:39:31.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:31 smithi183 ceph-mon[33277]: osdmap e490: 8 total, 8 up, 8 in 2024-09-21T12:39:31.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:31 smithi183 ceph-mon[33277]: 1.1 scrub starts 2024-09-21T12:39:31.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:31 smithi183 ceph-mon[33277]: 1.1 scrub ok 2024-09-21T12:39:31.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:31 smithi183 ceph-mon[33277]: 1.4 scrub starts 2024-09-21T12:39:31.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:31 smithi183 ceph-mon[33277]: 1.4 scrub ok 2024-09-21T12:39:31.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:31 smithi183 ceph-mon[33277]: pgmap v986: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T12:39:32.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:32 smithi183 ceph-mon[33277]: 4.3 scrub starts 2024-09-21T12:39:32.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:32 smithi183 ceph-mon[33277]: 4.3 scrub ok 2024-09-21T12:39:32.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:32 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:39:32.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:32 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:39:32.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:32 smithi183 ceph-mon[33277]: osdmap e491: 8 total, 8 up, 8 in 2024-09-21T12:39:32.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:32 smithi183 ceph-mon[33277]: 5.6 scrub starts 2024-09-21T12:39:32.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:32 smithi183 ceph-mon[33277]: 5.6 scrub ok 2024-09-21T12:39:32.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:32 smithi183 ceph-mon[33277]: 4.14 scrub starts 2024-09-21T12:39:32.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:32 smithi183 ceph-mon[33277]: 4.14 scrub ok 2024-09-21T12:39:32.490 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:32 smithi183 ceph-mon[33277]: osdmap e492: 8 total, 8 up, 8 in 2024-09-21T12:39:32.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:32 smithi114 ceph-mon[30324]: 4.3 scrub starts 2024-09-21T12:39:32.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:32 smithi114 ceph-mon[30324]: 4.3 scrub ok 2024-09-21T12:39:32.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:32 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:39:32.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:32 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:39:32.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:32 smithi114 ceph-mon[30324]: osdmap e491: 8 total, 8 up, 8 in 2024-09-21T12:39:32.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:32 smithi114 ceph-mon[30324]: 5.6 scrub starts 2024-09-21T12:39:32.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:32 smithi114 ceph-mon[30324]: 5.6 scrub ok 2024-09-21T12:39:32.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:32 smithi114 ceph-mon[30324]: 4.14 scrub starts 2024-09-21T12:39:32.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:32 smithi114 ceph-mon[30324]: 4.14 scrub ok 2024-09-21T12:39:32.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:32 smithi114 ceph-mon[30324]: osdmap e492: 8 total, 8 up, 8 in 2024-09-21T12:39:32.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:32 smithi114 ceph-mon[40867]: 4.3 scrub starts 2024-09-21T12:39:32.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:32 smithi114 ceph-mon[40867]: 4.3 scrub ok 2024-09-21T12:39:32.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:32 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:39:32.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:32 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:39:32.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:32 smithi114 ceph-mon[40867]: osdmap e491: 8 total, 8 up, 8 in 2024-09-21T12:39:32.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:32 smithi114 ceph-mon[40867]: 5.6 scrub starts 2024-09-21T12:39:32.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:32 smithi114 ceph-mon[40867]: 5.6 scrub ok 2024-09-21T12:39:32.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:32 smithi114 ceph-mon[40867]: 4.14 scrub starts 2024-09-21T12:39:32.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:32 smithi114 ceph-mon[40867]: 4.14 scrub ok 2024-09-21T12:39:32.528 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:32 smithi114 ceph-mon[40867]: osdmap e492: 8 total, 8 up, 8 in 2024-09-21T12:39:33.025 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:39:32 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:39:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:39:33.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:33 smithi183 ceph-mon[33277]: 3.a scrub starts 2024-09-21T12:39:33.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:33 smithi183 ceph-mon[33277]: 3.a scrub ok 2024-09-21T12:39:33.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:33 smithi183 ceph-mon[33277]: 6.18 scrub starts 2024-09-21T12:39:33.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:33 smithi183 ceph-mon[33277]: 6.18 scrub ok 2024-09-21T12:39:33.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:33 smithi183 ceph-mon[33277]: 1.a scrub starts 2024-09-21T12:39:33.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:33 smithi183 ceph-mon[33277]: 1.a scrub ok 2024-09-21T12:39:33.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:33 smithi183 ceph-mon[33277]: pgmap v989: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 999 B/s rd, 0 op/s 2024-09-21T12:39:33.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:33 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:39:33.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:33 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:39:33.498 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-21T12:39:33.498 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pgp_num 61 2024-09-21T12:39:33.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:33 smithi114 ceph-mon[30324]: 3.a scrub starts 2024-09-21T12:39:33.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:33 smithi114 ceph-mon[30324]: 3.a scrub ok 2024-09-21T12:39:33.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:33 smithi114 ceph-mon[30324]: 6.18 scrub starts 2024-09-21T12:39:33.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:33 smithi114 ceph-mon[30324]: 6.18 scrub ok 2024-09-21T12:39:33.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:33 smithi114 ceph-mon[30324]: 1.a scrub starts 2024-09-21T12:39:33.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:33 smithi114 ceph-mon[30324]: 1.a scrub ok 2024-09-21T12:39:33.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:33 smithi114 ceph-mon[30324]: pgmap v989: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 999 B/s rd, 0 op/s 2024-09-21T12:39:33.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:33 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:39:33.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:33 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:39:33.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:33 smithi114 ceph-mon[40867]: 3.a scrub starts 2024-09-21T12:39:33.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:33 smithi114 ceph-mon[40867]: 3.a scrub ok 2024-09-21T12:39:33.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:33 smithi114 ceph-mon[40867]: 6.18 scrub starts 2024-09-21T12:39:33.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:33 smithi114 ceph-mon[40867]: 6.18 scrub ok 2024-09-21T12:39:33.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:33 smithi114 ceph-mon[40867]: 1.a scrub starts 2024-09-21T12:39:33.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:33 smithi114 ceph-mon[40867]: 1.a scrub ok 2024-09-21T12:39:33.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:33 smithi114 ceph-mon[40867]: pgmap v989: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 999 B/s rd, 0 op/s 2024-09-21T12:39:33.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:33 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:39:33.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:33 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:39:33.757 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:39:34.399 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: specified pgp_num 61 > pg_num 31 2024-09-21T12:39:34.399 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:34 smithi114 ceph-mon[30324]: 5.3 scrub starts 2024-09-21T12:39:34.399 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:34 smithi114 ceph-mon[30324]: 5.3 scrub ok 2024-09-21T12:39:34.399 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:34 smithi114 ceph-mon[30324]: 4.7 scrub starts 2024-09-21T12:39:34.399 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:34 smithi114 ceph-mon[30324]: 4.7 scrub ok 2024-09-21T12:39:34.400 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:34 smithi114 ceph-mon[40867]: 5.3 scrub starts 2024-09-21T12:39:34.400 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:34 smithi114 ceph-mon[40867]: 5.3 scrub ok 2024-09-21T12:39:34.400 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:34 smithi114 ceph-mon[40867]: 4.7 scrub starts 2024-09-21T12:39:34.400 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:34 smithi114 ceph-mon[40867]: 4.7 scrub ok 2024-09-21T12:39:34.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:34 smithi183 ceph-mon[33277]: 5.3 scrub starts 2024-09-21T12:39:34.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:34 smithi183 ceph-mon[33277]: 5.3 scrub ok 2024-09-21T12:39:34.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:34 smithi183 ceph-mon[33277]: 4.7 scrub starts 2024-09-21T12:39:34.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:34 smithi183 ceph-mon[33277]: 4.7 scrub ok 2024-09-21T12:39:35.214 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T12:39:35.215 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-21T12:39:35.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:35 smithi183 ceph-mon[33277]: pgmap v990: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:39:35.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:35 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/4290288158' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-21T12:39:35.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:35 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-21T12:39:35.502 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:39:35.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:35 smithi114 ceph-mon[40867]: pgmap v990: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:39:35.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:35 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/4290288158' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-21T12:39:35.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:35 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-21T12:39:35.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:35 smithi114 ceph-mon[30324]: pgmap v990: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:39:35.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:35 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/4290288158' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-21T12:39:35.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:35 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-21T12:39:35.914 INFO:teuthology.orchestra.run.smithi114.stdout:mon.a: {} 2024-09-21T12:39:35.915 INFO:teuthology.orchestra.run.smithi114.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T12:39:35.924 INFO:teuthology.orchestra.run.smithi114.stdout:mon.b: {} 2024-09-21T12:39:35.925 INFO:teuthology.orchestra.run.smithi114.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T12:39:35.932 INFO:teuthology.orchestra.run.smithi114.stdout:mon.c: {} 2024-09-21T12:39:35.932 INFO:teuthology.orchestra.run.smithi114.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T12:39:36.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:36 smithi114 ceph-mon[40867]: 4.1 scrub starts 2024-09-21T12:39:36.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:36 smithi114 ceph-mon[40867]: 4.1 scrub ok 2024-09-21T12:39:36.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:36 smithi114 ceph-mon[40867]: 6.9 scrub starts 2024-09-21T12:39:36.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:36 smithi114 ceph-mon[40867]: 6.9 scrub ok 2024-09-21T12:39:36.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:36 smithi114 ceph-mon[40867]: 6.19 scrub starts 2024-09-21T12:39:36.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:36 smithi114 ceph-mon[40867]: 6.19 scrub ok 2024-09-21T12:39:36.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:36 smithi114 ceph-mon[40867]: 4.11 scrub starts 2024-09-21T12:39:36.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:36 smithi114 ceph-mon[40867]: 4.11 scrub ok 2024-09-21T12:39:36.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:36 smithi114 ceph-mon[40867]: 5.4 scrub starts 2024-09-21T12:39:36.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:36 smithi114 ceph-mon[40867]: 5.4 scrub ok 2024-09-21T12:39:36.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:36 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:39:36.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:36 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:39:36.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:36 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2642988096' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T12:39:36.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:36 smithi114 ceph-mon[30324]: 4.1 scrub starts 2024-09-21T12:39:36.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:36 smithi114 ceph-mon[30324]: 4.1 scrub ok 2024-09-21T12:39:36.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:36 smithi114 ceph-mon[30324]: 6.9 scrub starts 2024-09-21T12:39:36.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:36 smithi114 ceph-mon[30324]: 6.9 scrub ok 2024-09-21T12:39:36.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:36 smithi114 ceph-mon[30324]: 6.19 scrub starts 2024-09-21T12:39:36.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:36 smithi114 ceph-mon[30324]: 6.19 scrub ok 2024-09-21T12:39:36.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:36 smithi114 ceph-mon[30324]: 4.11 scrub starts 2024-09-21T12:39:36.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:36 smithi114 ceph-mon[30324]: 4.11 scrub ok 2024-09-21T12:39:36.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:36 smithi114 ceph-mon[30324]: 5.4 scrub starts 2024-09-21T12:39:36.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:36 smithi114 ceph-mon[30324]: 5.4 scrub ok 2024-09-21T12:39:36.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:36 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:39:36.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:36 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:39:36.528 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:36 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2642988096' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T12:39:36.705 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T12:39:36.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:36 smithi183 ceph-mon[33277]: 4.1 scrub starts 2024-09-21T12:39:36.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:36 smithi183 ceph-mon[33277]: 4.1 scrub ok 2024-09-21T12:39:36.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:36 smithi183 ceph-mon[33277]: 6.9 scrub starts 2024-09-21T12:39:36.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:36 smithi183 ceph-mon[33277]: 6.9 scrub ok 2024-09-21T12:39:36.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:36 smithi183 ceph-mon[33277]: 6.19 scrub starts 2024-09-21T12:39:36.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:36 smithi183 ceph-mon[33277]: 6.19 scrub ok 2024-09-21T12:39:36.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:36 smithi183 ceph-mon[33277]: 4.11 scrub starts 2024-09-21T12:39:36.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:36 smithi183 ceph-mon[33277]: 4.11 scrub ok 2024-09-21T12:39:36.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:36 smithi183 ceph-mon[33277]: 5.4 scrub starts 2024-09-21T12:39:36.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:36 smithi183 ceph-mon[33277]: 5.4 scrub ok 2024-09-21T12:39:36.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:36 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:39:36.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:36 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:39:36.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:36 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2642988096' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T12:39:36.950 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:39:37.366 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:37 smithi114 ceph-mon[40867]: 5.12 scrub starts 2024-09-21T12:39:37.366 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:37 smithi114 ceph-mon[40867]: 5.12 scrub ok 2024-09-21T12:39:37.366 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:37 smithi114 ceph-mon[40867]: 6.7 scrub starts 2024-09-21T12:39:37.367 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:37 smithi114 ceph-mon[40867]: 6.7 scrub ok 2024-09-21T12:39:37.367 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:37 smithi114 ceph-mon[40867]: osdmap e493: 8 total, 8 up, 8 in 2024-09-21T12:39:37.367 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:37 smithi114 ceph-mon[40867]: pgmap v992: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:39:37.367 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:37 smithi114 ceph-mon[40867]: osdmap e494: 8 total, 8 up, 8 in 2024-09-21T12:39:37.367 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:37 smithi114 ceph-mon[30324]: 5.12 scrub starts 2024-09-21T12:39:37.367 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:37 smithi114 ceph-mon[30324]: 5.12 scrub ok 2024-09-21T12:39:37.367 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:37 smithi114 ceph-mon[30324]: 6.7 scrub starts 2024-09-21T12:39:37.367 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:37 smithi114 ceph-mon[30324]: 6.7 scrub ok 2024-09-21T12:39:37.367 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:37 smithi114 ceph-mon[30324]: osdmap e493: 8 total, 8 up, 8 in 2024-09-21T12:39:37.367 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:37 smithi114 ceph-mon[30324]: pgmap v992: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:39:37.367 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:37 smithi114 ceph-mon[30324]: osdmap e494: 8 total, 8 up, 8 in 2024-09-21T12:39:37.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:37 smithi183 ceph-mon[33277]: 5.12 scrub starts 2024-09-21T12:39:37.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:37 smithi183 ceph-mon[33277]: 5.12 scrub ok 2024-09-21T12:39:37.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:37 smithi183 ceph-mon[33277]: 6.7 scrub starts 2024-09-21T12:39:37.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:37 smithi183 ceph-mon[33277]: 6.7 scrub ok 2024-09-21T12:39:37.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:37 smithi183 ceph-mon[33277]: osdmap e493: 8 total, 8 up, 8 in 2024-09-21T12:39:37.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:37 smithi183 ceph-mon[33277]: pgmap v992: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:39:37.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:37 smithi183 ceph-mon[33277]: osdmap e494: 8 total, 8 up, 8 in 2024-09-21T12:39:38.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:39:37 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:39:38.250 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T12:39:38.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:38 smithi114 ceph-mon[40867]: 5.1e scrub starts 2024-09-21T12:39:38.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:38 smithi114 ceph-mon[40867]: 5.1e scrub ok 2024-09-21T12:39:38.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:38 smithi114 ceph-mon[40867]: 2.2 scrub starts 2024-09-21T12:39:38.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:38 smithi114 ceph-mon[40867]: 2.2 scrub ok 2024-09-21T12:39:38.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:38 smithi114 ceph-mon[40867]: 5.11 scrub starts 2024-09-21T12:39:38.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:38 smithi114 ceph-mon[40867]: 5.11 scrub ok 2024-09-21T12:39:38.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:38 smithi114 ceph-mon[40867]: 1.10 scrub starts 2024-09-21T12:39:38.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:38 smithi114 ceph-mon[40867]: 1.10 scrub ok 2024-09-21T12:39:38.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:38 smithi114 ceph-mon[40867]: 6.11 scrub starts 2024-09-21T12:39:38.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:38 smithi114 ceph-mon[40867]: 6.11 scrub ok 2024-09-21T12:39:38.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:38 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1301124970' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:39:38.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:38 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:39:38.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:38 smithi114 ceph-mon[30324]: 5.1e scrub starts 2024-09-21T12:39:38.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:38 smithi114 ceph-mon[30324]: 5.1e scrub ok 2024-09-21T12:39:38.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:38 smithi114 ceph-mon[30324]: 2.2 scrub starts 2024-09-21T12:39:38.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:38 smithi114 ceph-mon[30324]: 2.2 scrub ok 2024-09-21T12:39:38.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:38 smithi114 ceph-mon[30324]: 5.11 scrub starts 2024-09-21T12:39:38.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:38 smithi114 ceph-mon[30324]: 5.11 scrub ok 2024-09-21T12:39:38.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:38 smithi114 ceph-mon[30324]: 1.10 scrub starts 2024-09-21T12:39:38.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:38 smithi114 ceph-mon[30324]: 1.10 scrub ok 2024-09-21T12:39:38.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:38 smithi114 ceph-mon[30324]: 6.11 scrub starts 2024-09-21T12:39:38.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:38 smithi114 ceph-mon[30324]: 6.11 scrub ok 2024-09-21T12:39:38.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:38 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1301124970' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:39:38.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:38 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:39:38.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:38 smithi183 ceph-mon[33277]: 5.1e scrub starts 2024-09-21T12:39:38.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:38 smithi183 ceph-mon[33277]: 5.1e scrub ok 2024-09-21T12:39:38.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:38 smithi183 ceph-mon[33277]: 2.2 scrub starts 2024-09-21T12:39:38.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:38 smithi183 ceph-mon[33277]: 2.2 scrub ok 2024-09-21T12:39:38.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:38 smithi183 ceph-mon[33277]: 5.11 scrub starts 2024-09-21T12:39:38.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:38 smithi183 ceph-mon[33277]: 5.11 scrub ok 2024-09-21T12:39:38.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:38 smithi183 ceph-mon[33277]: 1.10 scrub starts 2024-09-21T12:39:38.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:38 smithi183 ceph-mon[33277]: 1.10 scrub ok 2024-09-21T12:39:38.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:38 smithi183 ceph-mon[33277]: 6.11 scrub starts 2024-09-21T12:39:38.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:38 smithi183 ceph-mon[33277]: 6.11 scrub ok 2024-09-21T12:39:38.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:38 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1301124970' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:39:38.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:38 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:39:39.030 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 0 1 2024-09-21T12:39:39.282 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:39:39.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:39 smithi114 ceph-mon[30324]: 6.14 scrub starts 2024-09-21T12:39:39.639 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:39 smithi114 ceph-mon[30324]: 6.14 scrub ok 2024-09-21T12:39:39.639 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:39 smithi114 ceph-mon[30324]: 3.6 scrub starts 2024-09-21T12:39:39.639 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:39 smithi114 ceph-mon[30324]: 3.6 scrub ok 2024-09-21T12:39:39.639 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:39 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:39:39.639 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:39 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T12:39:39.639 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:39 smithi114 ceph-mon[30324]: osdmap e495: 8 total, 8 up, 8 in 2024-09-21T12:39:39.639 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:39 smithi114 ceph-mon[30324]: pgmap v995: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:39:39.639 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:39 smithi114 ceph-mon[40867]: 6.14 scrub starts 2024-09-21T12:39:39.639 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:39 smithi114 ceph-mon[40867]: 6.14 scrub ok 2024-09-21T12:39:39.640 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:39 smithi114 ceph-mon[40867]: 3.6 scrub starts 2024-09-21T12:39:39.640 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:39 smithi114 ceph-mon[40867]: 3.6 scrub ok 2024-09-21T12:39:39.640 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:39 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:39:39.640 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:39 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T12:39:39.640 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:39 smithi114 ceph-mon[40867]: osdmap e495: 8 total, 8 up, 8 in 2024-09-21T12:39:39.640 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:39 smithi114 ceph-mon[40867]: pgmap v995: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:39:39.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:39 smithi183 ceph-mon[33277]: 6.14 scrub starts 2024-09-21T12:39:39.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:39 smithi183 ceph-mon[33277]: 6.14 scrub ok 2024-09-21T12:39:39.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:39 smithi183 ceph-mon[33277]: 3.6 scrub starts 2024-09-21T12:39:39.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:39 smithi183 ceph-mon[33277]: 3.6 scrub ok 2024-09-21T12:39:39.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:39 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:39:39.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:39 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T12:39:39.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:39 smithi183 ceph-mon[33277]: osdmap e495: 8 total, 8 up, 8 in 2024-09-21T12:39:39.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:39 smithi183 ceph-mon[33277]: pgmap v995: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:39:40.257 INFO:teuthology.orchestra.run.smithi114.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-21T12:39:40.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:40 smithi114 ceph-mon[40867]: 4.1a scrub starts 2024-09-21T12:39:40.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:40 smithi114 ceph-mon[40867]: 4.1a scrub ok 2024-09-21T12:39:40.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:40 smithi114 ceph-mon[40867]: 4.4 scrub starts 2024-09-21T12:39:40.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:40 smithi114 ceph-mon[40867]: 4.4 scrub ok 2024-09-21T12:39:40.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:40 smithi114 ceph-mon[40867]: 5.14 scrub starts 2024-09-21T12:39:40.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:40 smithi114 ceph-mon[40867]: 5.14 scrub ok 2024-09-21T12:39:40.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:40 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2911304060' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:39:40.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:40 smithi114 ceph-mon[30324]: 4.1a scrub starts 2024-09-21T12:39:40.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:40 smithi114 ceph-mon[30324]: 4.1a scrub ok 2024-09-21T12:39:40.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:40 smithi114 ceph-mon[30324]: 4.4 scrub starts 2024-09-21T12:39:40.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:40 smithi114 ceph-mon[30324]: 4.4 scrub ok 2024-09-21T12:39:40.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:40 smithi114 ceph-mon[30324]: 5.14 scrub starts 2024-09-21T12:39:40.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:40 smithi114 ceph-mon[30324]: 5.14 scrub ok 2024-09-21T12:39:40.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:40 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2911304060' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:39:40.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:40 smithi183 ceph-mon[33277]: 4.1a scrub starts 2024-09-21T12:39:40.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:40 smithi183 ceph-mon[33277]: 4.1a scrub ok 2024-09-21T12:39:40.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:40 smithi183 ceph-mon[33277]: 4.4 scrub starts 2024-09-21T12:39:40.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:40 smithi183 ceph-mon[33277]: 4.4 scrub ok 2024-09-21T12:39:40.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:40 smithi183 ceph-mon[33277]: 5.14 scrub starts 2024-09-21T12:39:40.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:40 smithi183 ceph-mon[33277]: 5.14 scrub ok 2024-09-21T12:39:40.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:40 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2911304060' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T12:39:41.047 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T12:39:41.301 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:39:41.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:41 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2911304060' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-21T12:39:41.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:41 smithi114 ceph-mon[30324]: osdmap e496: 8 total, 8 up, 8 in 2024-09-21T12:39:41.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:41 smithi114 ceph-mon[30324]: pgmap v997: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:39:41.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:41 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2911304060' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-21T12:39:41.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:41 smithi114 ceph-mon[40867]: osdmap e496: 8 total, 8 up, 8 in 2024-09-21T12:39:41.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:41 smithi114 ceph-mon[40867]: pgmap v997: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:39:41.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:41 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2911304060' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-21T12:39:41.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:41 smithi183 ceph-mon[33277]: osdmap e496: 8 total, 8 up, 8 in 2024-09-21T12:39:41.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:41 smithi183 ceph-mon[33277]: pgmap v997: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:39:42.279 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T12:39:42.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:42 smithi114 ceph-mon[40867]: 1.7 scrub starts 2024-09-21T12:39:42.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:42 smithi114 ceph-mon[40867]: 1.7 scrub ok 2024-09-21T12:39:42.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:42 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1394474785' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:39:42.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:42 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:39:42.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:42 smithi114 ceph-mon[30324]: 1.7 scrub starts 2024-09-21T12:39:42.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:42 smithi114 ceph-mon[30324]: 1.7 scrub ok 2024-09-21T12:39:42.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:42 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1394474785' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:39:42.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:42 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:39:42.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:42 smithi183 ceph-mon[33277]: 1.7 scrub starts 2024-09-21T12:39:42.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:42 smithi183 ceph-mon[33277]: 1.7 scrub ok 2024-09-21T12:39:42.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:42 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1394474785' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:39:42.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:42 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:39:42.856 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:39:42 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:39:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:39:43.266 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 1 1 2024-09-21T12:39:43.513 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:39:43.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:43 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T12:39:43.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:43 smithi114 ceph-mon[30324]: osdmap e497: 8 total, 8 up, 8 in 2024-09-21T12:39:43.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:43 smithi114 ceph-mon[30324]: pgmap v999: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-09-21T12:39:43.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:43 smithi114 ceph-mon[30324]: 6.6 scrub starts 2024-09-21T12:39:43.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:43 smithi114 ceph-mon[30324]: 6.6 scrub ok 2024-09-21T12:39:43.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:43 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:39:43.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:43 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:39:43.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:43 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T12:39:43.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:43 smithi114 ceph-mon[40867]: osdmap e497: 8 total, 8 up, 8 in 2024-09-21T12:39:43.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:43 smithi114 ceph-mon[40867]: pgmap v999: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-09-21T12:39:43.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:43 smithi114 ceph-mon[40867]: 6.6 scrub starts 2024-09-21T12:39:43.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:43 smithi114 ceph-mon[40867]: 6.6 scrub ok 2024-09-21T12:39:43.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:43 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:39:43.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:43 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:39:43.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:43 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T12:39:43.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:43 smithi183 ceph-mon[33277]: osdmap e497: 8 total, 8 up, 8 in 2024-09-21T12:39:43.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:43 smithi183 ceph-mon[33277]: pgmap v999: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-09-21T12:39:43.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:43 smithi183 ceph-mon[33277]: 6.6 scrub starts 2024-09-21T12:39:43.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:43 smithi183 ceph-mon[33277]: 6.6 scrub ok 2024-09-21T12:39:43.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:43 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:39:43.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:43 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:39:44.285 INFO:teuthology.orchestra.run.smithi114.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-21T12:39:44.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:44 smithi114 ceph-mon[30324]: osdmap e498: 8 total, 8 up, 8 in 2024-09-21T12:39:44.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:44 smithi114 ceph-mon[30324]: 3.13 scrub starts 2024-09-21T12:39:44.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:44 smithi114 ceph-mon[30324]: 3.13 scrub ok 2024-09-21T12:39:44.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:44 smithi114 ceph-mon[30324]: 1.1e scrub starts 2024-09-21T12:39:44.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:44 smithi114 ceph-mon[30324]: 1.1e scrub ok 2024-09-21T12:39:44.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:44 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1556353761' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:39:44.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:44 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:39:44.657 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:44 smithi114 ceph-mon[40867]: osdmap e498: 8 total, 8 up, 8 in 2024-09-21T12:39:44.657 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:44 smithi114 ceph-mon[40867]: 3.13 scrub starts 2024-09-21T12:39:44.657 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:44 smithi114 ceph-mon[40867]: 3.13 scrub ok 2024-09-21T12:39:44.657 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:44 smithi114 ceph-mon[40867]: 1.1e scrub starts 2024-09-21T12:39:44.657 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:44 smithi114 ceph-mon[40867]: 1.1e scrub ok 2024-09-21T12:39:44.657 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:44 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1556353761' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:39:44.657 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:44 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:39:44.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:44 smithi183 ceph-mon[33277]: osdmap e498: 8 total, 8 up, 8 in 2024-09-21T12:39:44.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:44 smithi183 ceph-mon[33277]: 3.13 scrub starts 2024-09-21T12:39:44.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:44 smithi183 ceph-mon[33277]: 3.13 scrub ok 2024-09-21T12:39:44.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:44 smithi183 ceph-mon[33277]: 1.1e scrub starts 2024-09-21T12:39:44.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:44 smithi183 ceph-mon[33277]: 1.1e scrub ok 2024-09-21T12:39:44.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:44 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1556353761' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:39:44.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:44 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T12:39:44.969 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T12:39:45.215 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:39:45.596 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:45 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T12:39:45.597 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:45 smithi114 ceph-mon[40867]: 6.d scrub starts 2024-09-21T12:39:45.597 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:45 smithi114 ceph-mon[40867]: 6.d scrub ok 2024-09-21T12:39:45.597 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:45 smithi114 ceph-mon[40867]: osdmap e499: 8 total, 8 up, 8 in 2024-09-21T12:39:45.597 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:45 smithi114 ceph-mon[40867]: pgmap v1002: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:39:45.597 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:45 smithi114 ceph-mon[40867]: 5.10 scrub starts 2024-09-21T12:39:45.597 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:45 smithi114 ceph-mon[40867]: 5.10 scrub ok 2024-09-21T12:39:45.597 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:45 smithi114 ceph-mon[40867]: 4.17 scrub starts 2024-09-21T12:39:45.597 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:45 smithi114 ceph-mon[40867]: 4.17 scrub ok 2024-09-21T12:39:45.597 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:45 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T12:39:45.597 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:45 smithi114 ceph-mon[30324]: 6.d scrub starts 2024-09-21T12:39:45.598 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:45 smithi114 ceph-mon[30324]: 6.d scrub ok 2024-09-21T12:39:45.598 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:45 smithi114 ceph-mon[30324]: osdmap e499: 8 total, 8 up, 8 in 2024-09-21T12:39:45.598 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:45 smithi114 ceph-mon[30324]: pgmap v1002: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:39:45.598 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:45 smithi114 ceph-mon[30324]: 5.10 scrub starts 2024-09-21T12:39:45.598 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:45 smithi114 ceph-mon[30324]: 5.10 scrub ok 2024-09-21T12:39:45.598 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:45 smithi114 ceph-mon[30324]: 4.17 scrub starts 2024-09-21T12:39:45.598 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:45 smithi114 ceph-mon[30324]: 4.17 scrub ok 2024-09-21T12:39:45.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:45 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T12:39:45.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:45 smithi183 ceph-mon[33277]: 6.d scrub starts 2024-09-21T12:39:45.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:45 smithi183 ceph-mon[33277]: 6.d scrub ok 2024-09-21T12:39:45.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:45 smithi183 ceph-mon[33277]: osdmap e499: 8 total, 8 up, 8 in 2024-09-21T12:39:45.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:45 smithi183 ceph-mon[33277]: pgmap v1002: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:39:45.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:45 smithi183 ceph-mon[33277]: 5.10 scrub starts 2024-09-21T12:39:45.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:45 smithi183 ceph-mon[33277]: 5.10 scrub ok 2024-09-21T12:39:45.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:45 smithi183 ceph-mon[33277]: 4.17 scrub starts 2024-09-21T12:39:45.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:45 smithi183 ceph-mon[33277]: 4.17 scrub ok 2024-09-21T12:39:46.301 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T12:39:46.674 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:46 smithi114 ceph-mon[30324]: 3.14 scrub starts 2024-09-21T12:39:46.675 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:46 smithi114 ceph-mon[30324]: 3.14 scrub ok 2024-09-21T12:39:46.675 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:46 smithi114 ceph-mon[30324]: 3.1b scrub starts 2024-09-21T12:39:46.675 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:46 smithi114 ceph-mon[30324]: 3.1b scrub ok 2024-09-21T12:39:46.675 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:46 smithi114 ceph-mon[30324]: 1.16 scrub starts 2024-09-21T12:39:46.675 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:46 smithi114 ceph-mon[30324]: 1.16 scrub ok 2024-09-21T12:39:46.675 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:46 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3589786445' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:39:46.675 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:46 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:39:46.675 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:46 smithi114 ceph-mon[40867]: 3.14 scrub starts 2024-09-21T12:39:46.675 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:46 smithi114 ceph-mon[40867]: 3.14 scrub ok 2024-09-21T12:39:46.675 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:46 smithi114 ceph-mon[40867]: 3.1b scrub starts 2024-09-21T12:39:46.676 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:46 smithi114 ceph-mon[40867]: 3.1b scrub ok 2024-09-21T12:39:46.676 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:46 smithi114 ceph-mon[40867]: 1.16 scrub starts 2024-09-21T12:39:46.676 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:46 smithi114 ceph-mon[40867]: 1.16 scrub ok 2024-09-21T12:39:46.676 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:46 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3589786445' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:39:46.676 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:46 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:39:46.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:46 smithi183 ceph-mon[33277]: 3.14 scrub starts 2024-09-21T12:39:46.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:46 smithi183 ceph-mon[33277]: 3.14 scrub ok 2024-09-21T12:39:46.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:46 smithi183 ceph-mon[33277]: 3.1b scrub starts 2024-09-21T12:39:46.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:46 smithi183 ceph-mon[33277]: 3.1b scrub ok 2024-09-21T12:39:46.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:46 smithi183 ceph-mon[33277]: 1.16 scrub starts 2024-09-21T12:39:46.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:46 smithi183 ceph-mon[33277]: 1.16 scrub ok 2024-09-21T12:39:46.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:46 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3589786445' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:39:46.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:46 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:39:47.047 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 3 1 2024-09-21T12:39:47.306 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:39:47.582 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:47 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T12:39:47.583 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:47 smithi114 ceph-mon[30324]: osdmap e500: 8 total, 8 up, 8 in 2024-09-21T12:39:47.583 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:47 smithi114 ceph-mon[30324]: pgmap v1004: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail 2024-09-21T12:39:47.583 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:47 smithi114 ceph-mon[30324]: 6.8 scrub starts 2024-09-21T12:39:47.583 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:47 smithi114 ceph-mon[30324]: 6.8 scrub ok 2024-09-21T12:39:47.583 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:47 smithi114 ceph-mon[30324]: 3.2 scrub starts 2024-09-21T12:39:47.583 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:47 smithi114 ceph-mon[30324]: 3.2 scrub ok 2024-09-21T12:39:47.583 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:47 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:39:47.583 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:47 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:39:47.583 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:47 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:39:47.583 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:47 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:39:47.583 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:47 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T12:39:47.584 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:47 smithi114 ceph-mon[40867]: osdmap e500: 8 total, 8 up, 8 in 2024-09-21T12:39:47.584 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:47 smithi114 ceph-mon[40867]: pgmap v1004: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail 2024-09-21T12:39:47.584 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:47 smithi114 ceph-mon[40867]: 6.8 scrub starts 2024-09-21T12:39:47.584 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:47 smithi114 ceph-mon[40867]: 6.8 scrub ok 2024-09-21T12:39:47.584 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:47 smithi114 ceph-mon[40867]: 3.2 scrub starts 2024-09-21T12:39:47.584 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:47 smithi114 ceph-mon[40867]: 3.2 scrub ok 2024-09-21T12:39:47.584 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:47 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:39:47.584 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:47 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:39:47.584 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:47 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:39:47.584 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:47 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:39:47.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:47 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T12:39:47.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:47 smithi183 ceph-mon[33277]: osdmap e500: 8 total, 8 up, 8 in 2024-09-21T12:39:47.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:47 smithi183 ceph-mon[33277]: pgmap v1004: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail 2024-09-21T12:39:47.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:47 smithi183 ceph-mon[33277]: 6.8 scrub starts 2024-09-21T12:39:47.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:47 smithi183 ceph-mon[33277]: 6.8 scrub ok 2024-09-21T12:39:47.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:47 smithi183 ceph-mon[33277]: 3.2 scrub starts 2024-09-21T12:39:47.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:47 smithi183 ceph-mon[33277]: 3.2 scrub ok 2024-09-21T12:39:47.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:47 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:39:47.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:47 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:39:47.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:47 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:39:47.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:47 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:39:48.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:39:47 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:39:48.309 INFO:teuthology.orchestra.run.smithi114.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-21T12:39:48.596 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:48 smithi114 ceph-mon[40867]: pgmap v1005: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T12:39:48.596 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:48 smithi114 ceph-mon[40867]: pgmap v1006: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T12:39:48.596 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:48 smithi114 ceph-mon[40867]: 5.1a scrub starts 2024-09-21T12:39:48.596 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:48 smithi114 ceph-mon[40867]: 5.1a scrub ok 2024-09-21T12:39:48.596 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:48 smithi114 ceph-mon[40867]: 5.0 scrub starts 2024-09-21T12:39:48.596 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:48 smithi114 ceph-mon[40867]: 5.0 scrub ok 2024-09-21T12:39:48.597 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:48 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:39:48.597 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:48 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/595584492' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:39:48.597 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:48 smithi114 ceph-mon[30324]: pgmap v1005: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T12:39:48.597 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:48 smithi114 ceph-mon[30324]: pgmap v1006: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T12:39:48.597 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:48 smithi114 ceph-mon[30324]: 5.1a scrub starts 2024-09-21T12:39:48.597 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:48 smithi114 ceph-mon[30324]: 5.1a scrub ok 2024-09-21T12:39:48.597 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:48 smithi114 ceph-mon[30324]: 5.0 scrub starts 2024-09-21T12:39:48.597 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:48 smithi114 ceph-mon[30324]: 5.0 scrub ok 2024-09-21T12:39:48.597 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:48 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:39:48.597 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:48 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/595584492' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:39:48.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:48 smithi183 ceph-mon[33277]: pgmap v1005: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T12:39:48.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:48 smithi183 ceph-mon[33277]: pgmap v1006: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T12:39:48.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:48 smithi183 ceph-mon[33277]: 5.1a scrub starts 2024-09-21T12:39:48.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:48 smithi183 ceph-mon[33277]: 5.1a scrub ok 2024-09-21T12:39:48.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:48 smithi183 ceph-mon[33277]: 5.0 scrub starts 2024-09-21T12:39:48.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:48 smithi183 ceph-mon[33277]: 5.0 scrub ok 2024-09-21T12:39:48.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:48 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:39:48.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:48 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/595584492' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T12:39:49.155 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 4 1 2024-09-21T12:39:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:39:49.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:49 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:39:49.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:49 smithi114 ceph-mon[30324]: 4.a scrub starts 2024-09-21T12:39:49.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:49 smithi114 ceph-mon[30324]: 4.a scrub ok 2024-09-21T12:39:49.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:49 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/595584492' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-21T12:39:49.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:49 smithi114 ceph-mon[30324]: osdmap e501: 8 total, 8 up, 8 in 2024-09-21T12:39:49.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:49 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:39:49.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:49 smithi114 ceph-mon[40867]: 4.a scrub starts 2024-09-21T12:39:49.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:49 smithi114 ceph-mon[40867]: 4.a scrub ok 2024-09-21T12:39:49.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:49 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/595584492' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-21T12:39:49.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:49 smithi114 ceph-mon[40867]: osdmap e501: 8 total, 8 up, 8 in 2024-09-21T12:39:49.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:49 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:39:49.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:49 smithi183 ceph-mon[33277]: 4.a scrub starts 2024-09-21T12:39:49.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:49 smithi183 ceph-mon[33277]: 4.a scrub ok 2024-09-21T12:39:49.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:49 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/595584492' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-21T12:39:49.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:49 smithi183 ceph-mon[33277]: osdmap e501: 8 total, 8 up, 8 in 2024-09-21T12:39:50.323 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T12:39:50.639 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:50 smithi183 ceph-mon[33277]: pgmap v1008: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T12:39:50.640 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:50 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2663938394' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T12:39:50.708 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:50 smithi114 ceph-mon[40867]: pgmap v1008: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T12:39:50.708 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:50 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2663938394' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T12:39:50.708 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:50 smithi114 ceph-mon[30324]: pgmap v1008: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T12:39:50.709 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:50 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2663938394' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T12:39:51.122 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 4 1 2024-09-21T12:39:51.374 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:39:51.495 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:51 smithi114 ceph-mon[30324]: 4.8 scrub starts 2024-09-21T12:39:51.495 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:51 smithi114 ceph-mon[30324]: 4.8 scrub ok 2024-09-21T12:39:51.495 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:51 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2663938394' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T12:39:51.495 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:51 smithi114 ceph-mon[30324]: osdmap e502: 8 total, 8 up, 8 in 2024-09-21T12:39:51.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:51 smithi114 ceph-mon[40867]: 4.8 scrub starts 2024-09-21T12:39:51.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:51 smithi114 ceph-mon[40867]: 4.8 scrub ok 2024-09-21T12:39:51.496 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:51 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2663938394' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T12:39:51.496 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:51 smithi114 ceph-mon[40867]: osdmap e502: 8 total, 8 up, 8 in 2024-09-21T12:39:51.656 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:51 smithi183 ceph-mon[33277]: 4.8 scrub starts 2024-09-21T12:39:51.657 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:51 smithi183 ceph-mon[33277]: 4.8 scrub ok 2024-09-21T12:39:51.657 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:51 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2663938394' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T12:39:51.657 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:51 smithi183 ceph-mon[33277]: osdmap e502: 8 total, 8 up, 8 in 2024-09-21T12:39:52.334 INFO:teuthology.orchestra.run.smithi114.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-21T12:39:52.692 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:39:52 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:39:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:39:52.693 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:52 smithi114 ceph-mon[30324]: 4.1f scrub starts 2024-09-21T12:39:52.693 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:52 smithi114 ceph-mon[30324]: 4.1f scrub ok 2024-09-21T12:39:52.693 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:52 smithi114 ceph-mon[30324]: pgmap v1010: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T12:39:52.693 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:52 smithi114 ceph-mon[30324]: 5.f scrub starts 2024-09-21T12:39:52.693 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:52 smithi114 ceph-mon[30324]: 5.f scrub ok 2024-09-21T12:39:52.693 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:52 smithi114 ceph-mon[30324]: 5.13 scrub starts 2024-09-21T12:39:52.693 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:52 smithi114 ceph-mon[30324]: 5.13 scrub ok 2024-09-21T12:39:52.693 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:52 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2251172154' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T12:39:52.693 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:52 smithi114 ceph-mon[40867]: 4.1f scrub starts 2024-09-21T12:39:52.693 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:52 smithi114 ceph-mon[40867]: 4.1f scrub ok 2024-09-21T12:39:52.694 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:52 smithi114 ceph-mon[40867]: pgmap v1010: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T12:39:52.694 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:52 smithi114 ceph-mon[40867]: 5.f scrub starts 2024-09-21T12:39:52.694 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:52 smithi114 ceph-mon[40867]: 5.f scrub ok 2024-09-21T12:39:52.694 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:52 smithi114 ceph-mon[40867]: 5.13 scrub starts 2024-09-21T12:39:52.694 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:52 smithi114 ceph-mon[40867]: 5.13 scrub ok 2024-09-21T12:39:52.694 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:52 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2251172154' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T12:39:52.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:52 smithi183 ceph-mon[33277]: 4.1f scrub starts 2024-09-21T12:39:52.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:52 smithi183 ceph-mon[33277]: 4.1f scrub ok 2024-09-21T12:39:52.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:52 smithi183 ceph-mon[33277]: pgmap v1010: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T12:39:52.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:52 smithi183 ceph-mon[33277]: 5.f scrub starts 2024-09-21T12:39:52.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:52 smithi183 ceph-mon[33277]: 5.f scrub ok 2024-09-21T12:39:52.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:52 smithi183 ceph-mon[33277]: 5.13 scrub starts 2024-09-21T12:39:52.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:52 smithi183 ceph-mon[33277]: 5.13 scrub ok 2024-09-21T12:39:52.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:52 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2251172154' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T12:39:53.130 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T12:39:53.377 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:39:53.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:53 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2251172154' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-21T12:39:53.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:53 smithi114 ceph-mon[40867]: osdmap e503: 8 total, 8 up, 8 in 2024-09-21T12:39:53.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:53 smithi114 ceph-mon[40867]: 6.b scrub starts 2024-09-21T12:39:53.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:53 smithi114 ceph-mon[40867]: 6.b scrub ok 2024-09-21T12:39:53.498 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:53 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2251172154' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-21T12:39:53.499 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:53 smithi114 ceph-mon[30324]: osdmap e503: 8 total, 8 up, 8 in 2024-09-21T12:39:53.499 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:53 smithi114 ceph-mon[30324]: 6.b scrub starts 2024-09-21T12:39:53.499 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:53 smithi114 ceph-mon[30324]: 6.b scrub ok 2024-09-21T12:39:53.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:53 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2251172154' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-21T12:39:53.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:53 smithi183 ceph-mon[33277]: osdmap e503: 8 total, 8 up, 8 in 2024-09-21T12:39:53.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:53 smithi183 ceph-mon[33277]: 6.b scrub starts 2024-09-21T12:39:53.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:53 smithi183 ceph-mon[33277]: 6.b scrub ok 2024-09-21T12:39:54.347 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T12:39:54.677 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:54 smithi114 ceph-mon[30324]: 6.16 scrub starts 2024-09-21T12:39:54.677 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:54 smithi114 ceph-mon[30324]: 6.16 scrub ok 2024-09-21T12:39:54.677 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:54 smithi114 ceph-mon[30324]: pgmap v1012: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:39:54.677 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:54 smithi114 ceph-mon[30324]: 4.12 scrub starts 2024-09-21T12:39:54.677 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:54 smithi114 ceph-mon[30324]: 4.12 scrub ok 2024-09-21T12:39:54.677 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:54 smithi114 ceph-mon[30324]: 6.15 scrub starts 2024-09-21T12:39:54.678 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:54 smithi114 ceph-mon[30324]: 6.15 scrub ok 2024-09-21T12:39:54.678 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:54 smithi114 ceph-mon[30324]: 6.1b scrub starts 2024-09-21T12:39:54.678 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:54 smithi114 ceph-mon[30324]: 6.1b scrub ok 2024-09-21T12:39:54.678 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:54 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1039549539' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:39:54.678 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:54 smithi114 ceph-mon[40867]: 6.16 scrub starts 2024-09-21T12:39:54.678 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:54 smithi114 ceph-mon[40867]: 6.16 scrub ok 2024-09-21T12:39:54.678 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:54 smithi114 ceph-mon[40867]: pgmap v1012: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:39:54.678 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:54 smithi114 ceph-mon[40867]: 4.12 scrub starts 2024-09-21T12:39:54.678 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:54 smithi114 ceph-mon[40867]: 4.12 scrub ok 2024-09-21T12:39:54.678 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:54 smithi114 ceph-mon[40867]: 6.15 scrub starts 2024-09-21T12:39:54.678 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:54 smithi114 ceph-mon[40867]: 6.15 scrub ok 2024-09-21T12:39:54.678 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:54 smithi114 ceph-mon[40867]: 6.1b scrub starts 2024-09-21T12:39:54.679 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:54 smithi114 ceph-mon[40867]: 6.1b scrub ok 2024-09-21T12:39:54.679 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:54 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1039549539' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:39:54.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:54 smithi183 ceph-mon[33277]: 6.16 scrub starts 2024-09-21T12:39:54.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:54 smithi183 ceph-mon[33277]: 6.16 scrub ok 2024-09-21T12:39:54.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:54 smithi183 ceph-mon[33277]: pgmap v1012: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:39:54.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:54 smithi183 ceph-mon[33277]: 4.12 scrub starts 2024-09-21T12:39:54.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:54 smithi183 ceph-mon[33277]: 4.12 scrub ok 2024-09-21T12:39:54.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:54 smithi183 ceph-mon[33277]: 6.15 scrub starts 2024-09-21T12:39:54.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:54 smithi183 ceph-mon[33277]: 6.15 scrub ok 2024-09-21T12:39:54.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:54 smithi183 ceph-mon[33277]: 6.1b scrub starts 2024-09-21T12:39:54.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:54 smithi183 ceph-mon[33277]: 6.1b scrub ok 2024-09-21T12:39:54.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:54 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1039549539' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:39:55.097 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 5 1 2024-09-21T12:39:55.342 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:39:55.613 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:55 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:39:55.613 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:55 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:39:55.613 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:55 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1039549539' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T12:39:55.613 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:55 smithi114 ceph-mon[30324]: osdmap e504: 8 total, 8 up, 8 in 2024-09-21T12:39:55.613 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:55 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:39:55.613 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:55 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:39:55.613 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:55 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:39:55.613 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:55 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:39:55.613 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:55 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1039549539' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T12:39:55.613 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:55 smithi114 ceph-mon[40867]: osdmap e504: 8 total, 8 up, 8 in 2024-09-21T12:39:55.614 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:55 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:39:55.614 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:55 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:39:55.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:55 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:39:55.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:55 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:39:55.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:55 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1039549539' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T12:39:55.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:55 smithi183 ceph-mon[33277]: osdmap e504: 8 total, 8 up, 8 in 2024-09-21T12:39:55.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:55 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:39:55.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:55 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:39:56.374 INFO:teuthology.orchestra.run.smithi114.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-21T12:39:56.735 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:56 smithi114 ceph-mon[30324]: pgmap v1014: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:39:56.736 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:56 smithi114 ceph-mon[30324]: osdmap e505: 8 total, 8 up, 8 in 2024-09-21T12:39:56.736 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:56 smithi114 ceph-mon[30324]: 3.11 scrub starts 2024-09-21T12:39:56.736 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:56 smithi114 ceph-mon[30324]: 3.11 scrub ok 2024-09-21T12:39:56.736 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:56 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1862900093' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:39:56.736 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:56 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:39:56.736 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:56 smithi114 ceph-mon[40867]: pgmap v1014: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:39:56.736 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:56 smithi114 ceph-mon[40867]: osdmap e505: 8 total, 8 up, 8 in 2024-09-21T12:39:56.736 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:56 smithi114 ceph-mon[40867]: 3.11 scrub starts 2024-09-21T12:39:56.736 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:56 smithi114 ceph-mon[40867]: 3.11 scrub ok 2024-09-21T12:39:56.736 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:56 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1862900093' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:39:56.736 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:56 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:39:56.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:56 smithi183 ceph-mon[33277]: pgmap v1014: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:39:56.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:56 smithi183 ceph-mon[33277]: osdmap e505: 8 total, 8 up, 8 in 2024-09-21T12:39:56.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:56 smithi183 ceph-mon[33277]: 3.11 scrub starts 2024-09-21T12:39:56.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:56 smithi183 ceph-mon[33277]: 3.11 scrub ok 2024-09-21T12:39:56.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:56 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1862900093' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:39:56.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:56 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T12:39:57.157 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T12:39:57.380 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:57 smithi114 ceph-mon[30324]: 1.11 scrub starts 2024-09-21T12:39:57.380 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:57 smithi114 ceph-mon[30324]: 1.11 scrub ok 2024-09-21T12:39:57.380 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:57 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-21T12:39:57.380 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:57 smithi114 ceph-mon[30324]: osdmap e506: 8 total, 8 up, 8 in 2024-09-21T12:39:57.380 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:57 smithi114 ceph-mon[30324]: 5.9 scrub starts 2024-09-21T12:39:57.380 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:57 smithi114 ceph-mon[30324]: 5.9 scrub ok 2024-09-21T12:39:57.413 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:39:57.661 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:57 smithi114 ceph-mon[40867]: 1.11 scrub starts 2024-09-21T12:39:57.662 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:57 smithi114 ceph-mon[40867]: 1.11 scrub ok 2024-09-21T12:39:57.662 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:57 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-21T12:39:57.662 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:57 smithi114 ceph-mon[40867]: osdmap e506: 8 total, 8 up, 8 in 2024-09-21T12:39:57.662 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:57 smithi114 ceph-mon[40867]: 5.9 scrub starts 2024-09-21T12:39:57.662 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:57 smithi114 ceph-mon[40867]: 5.9 scrub ok 2024-09-21T12:39:57.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:57 smithi183 ceph-mon[33277]: 1.11 scrub starts 2024-09-21T12:39:57.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:57 smithi183 ceph-mon[33277]: 1.11 scrub ok 2024-09-21T12:39:57.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:57 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-21T12:39:57.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:57 smithi183 ceph-mon[33277]: osdmap e506: 8 total, 8 up, 8 in 2024-09-21T12:39:57.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:57 smithi183 ceph-mon[33277]: 5.9 scrub starts 2024-09-21T12:39:57.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:57 smithi183 ceph-mon[33277]: 5.9 scrub ok 2024-09-21T12:39:58.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:39:57 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:39:58.387 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T12:39:58.736 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:58 smithi114 ceph-mon[40867]: 6.5 scrub starts 2024-09-21T12:39:58.737 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:58 smithi114 ceph-mon[40867]: 6.5 scrub ok 2024-09-21T12:39:58.737 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:58 smithi114 ceph-mon[40867]: 3.16 scrub starts 2024-09-21T12:39:58.737 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:58 smithi114 ceph-mon[40867]: 3.16 scrub ok 2024-09-21T12:39:58.737 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:58 smithi114 ceph-mon[40867]: pgmap v1017: 162 pgs: 2 peering, 160 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T12:39:58.737 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:58 smithi114 ceph-mon[40867]: osdmap e507: 8 total, 8 up, 8 in 2024-09-21T12:39:58.737 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:58 smithi114 ceph-mon[40867]: 1.3 scrub starts 2024-09-21T12:39:58.737 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:58 smithi114 ceph-mon[40867]: 1.3 scrub ok 2024-09-21T12:39:58.737 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:58 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2968686206' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:39:58.737 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:58 smithi114 ceph-mon[40867]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T12:39:58.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:58 smithi114 ceph-mon[30324]: 6.5 scrub starts 2024-09-21T12:39:58.737 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:58 smithi114 ceph-mon[30324]: 6.5 scrub ok 2024-09-21T12:39:58.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:58 smithi114 ceph-mon[30324]: 3.16 scrub starts 2024-09-21T12:39:58.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:58 smithi114 ceph-mon[30324]: 3.16 scrub ok 2024-09-21T12:39:58.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:58 smithi114 ceph-mon[30324]: pgmap v1017: 162 pgs: 2 peering, 160 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T12:39:58.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:58 smithi114 ceph-mon[30324]: osdmap e507: 8 total, 8 up, 8 in 2024-09-21T12:39:58.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:58 smithi114 ceph-mon[30324]: 1.3 scrub starts 2024-09-21T12:39:58.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:58 smithi114 ceph-mon[30324]: 1.3 scrub ok 2024-09-21T12:39:58.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:58 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2968686206' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:39:58.738 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:58 smithi114 ceph-mon[30324]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T12:39:58.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:58 smithi183 ceph-mon[33277]: 6.5 scrub starts 2024-09-21T12:39:58.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:58 smithi183 ceph-mon[33277]: 6.5 scrub ok 2024-09-21T12:39:58.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:58 smithi183 ceph-mon[33277]: 3.16 scrub starts 2024-09-21T12:39:58.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:58 smithi183 ceph-mon[33277]: 3.16 scrub ok 2024-09-21T12:39:58.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:58 smithi183 ceph-mon[33277]: pgmap v1017: 162 pgs: 2 peering, 160 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T12:39:58.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:58 smithi183 ceph-mon[33277]: osdmap e507: 8 total, 8 up, 8 in 2024-09-21T12:39:58.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:58 smithi183 ceph-mon[33277]: 1.3 scrub starts 2024-09-21T12:39:58.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:58 smithi183 ceph-mon[33277]: 1.3 scrub ok 2024-09-21T12:39:58.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:58 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2968686206' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:39:58.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:58 smithi183 ceph-mon[33277]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T12:39:59.183 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 6 1 2024-09-21T12:39:59.433 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:39:59.726 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:59 smithi114 ceph-mon[40867]: 1.6 scrub starts 2024-09-21T12:39:59.727 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:59 smithi114 ceph-mon[40867]: 1.6 scrub ok 2024-09-21T12:39:59.727 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:59 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:39:59.727 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:59 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2968686206' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T12:39:59.727 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:39:59 smithi114 ceph-mon[40867]: osdmap e508: 8 total, 8 up, 8 in 2024-09-21T12:39:59.727 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:59 smithi114 ceph-mon[30324]: 1.6 scrub starts 2024-09-21T12:39:59.727 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:59 smithi114 ceph-mon[30324]: 1.6 scrub ok 2024-09-21T12:39:59.727 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:59 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:39:59.727 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:59 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2968686206' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T12:39:59.727 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:39:59 smithi114 ceph-mon[30324]: osdmap e508: 8 total, 8 up, 8 in 2024-09-21T12:39:59.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:59 smithi183 ceph-mon[33277]: 1.6 scrub starts 2024-09-21T12:39:59.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:59 smithi183 ceph-mon[33277]: 1.6 scrub ok 2024-09-21T12:39:59.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:59 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:39:59.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:59 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2968686206' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T12:39:59.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:39:59 smithi183 ceph-mon[33277]: osdmap e508: 8 total, 8 up, 8 in 2024-09-21T12:40:00.403 INFO:teuthology.orchestra.run.smithi114.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-21T12:40:00.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:00 smithi183 ceph-mon[33277]: pgmap v1020: 162 pgs: 2 peering, 160 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:40:00.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:00 smithi183 ceph-mon[33277]: 3.8 scrub starts 2024-09-21T12:40:00.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:00 smithi183 ceph-mon[33277]: 3.8 scrub ok 2024-09-21T12:40:00.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:00 smithi183 ceph-mon[33277]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); Reduced data availability: 1 pg inactive, 2 pgs peering 2024-09-21T12:40:00.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:00 smithi183 ceph-mon[33277]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-21T12:40:00.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:00 smithi183 ceph-mon[33277]: daemon osd.2 on smithi114 is in unknown state 2024-09-21T12:40:00.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:00 smithi183 ceph-mon[33277]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 2 pgs peering 2024-09-21T12:40:00.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:00 smithi183 ceph-mon[33277]: pg 1.8 is stuck peering for 7m, current state peering, last acting [5,2,3] 2024-09-21T12:40:00.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:00 smithi183 ceph-mon[33277]: pg 2.2 is stuck peering for 13m, current state peering, last acting [5,1,6] 2024-09-21T12:40:00.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:00 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2358149718' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:40:00.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:00 smithi114 ceph-mon[30324]: pgmap v1020: 162 pgs: 2 peering, 160 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:40:00.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:00 smithi114 ceph-mon[30324]: 3.8 scrub starts 2024-09-21T12:40:00.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:00 smithi114 ceph-mon[30324]: 3.8 scrub ok 2024-09-21T12:40:00.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:00 smithi114 ceph-mon[30324]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); Reduced data availability: 1 pg inactive, 2 pgs peering 2024-09-21T12:40:00.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:00 smithi114 ceph-mon[30324]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-21T12:40:00.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:00 smithi114 ceph-mon[30324]: daemon osd.2 on smithi114 is in unknown state 2024-09-21T12:40:00.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:00 smithi114 ceph-mon[30324]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 2 pgs peering 2024-09-21T12:40:00.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:00 smithi114 ceph-mon[30324]: pg 1.8 is stuck peering for 7m, current state peering, last acting [5,2,3] 2024-09-21T12:40:00.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:00 smithi114 ceph-mon[30324]: pg 2.2 is stuck peering for 13m, current state peering, last acting [5,1,6] 2024-09-21T12:40:00.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:00 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2358149718' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:40:00.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:00 smithi114 ceph-mon[40867]: pgmap v1020: 162 pgs: 2 peering, 160 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:40:00.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:00 smithi114 ceph-mon[40867]: 3.8 scrub starts 2024-09-21T12:40:00.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:00 smithi114 ceph-mon[40867]: 3.8 scrub ok 2024-09-21T12:40:00.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:00 smithi114 ceph-mon[40867]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); Reduced data availability: 1 pg inactive, 2 pgs peering 2024-09-21T12:40:00.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:00 smithi114 ceph-mon[40867]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-21T12:40:00.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:00 smithi114 ceph-mon[40867]: daemon osd.2 on smithi114 is in unknown state 2024-09-21T12:40:00.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:00 smithi114 ceph-mon[40867]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 2 pgs peering 2024-09-21T12:40:00.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:00 smithi114 ceph-mon[40867]: pg 1.8 is stuck peering for 7m, current state peering, last acting [5,2,3] 2024-09-21T12:40:00.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:00 smithi114 ceph-mon[40867]: pg 2.2 is stuck peering for 13m, current state peering, last acting [5,1,6] 2024-09-21T12:40:00.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:00 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2358149718' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-21T12:40:01.214 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T12:40:01.472 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:40:01.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:01 smithi114 ceph-mon[30324]: 3.19 scrub starts 2024-09-21T12:40:01.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:01 smithi114 ceph-mon[30324]: 3.19 scrub ok 2024-09-21T12:40:01.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:01 smithi114 ceph-mon[30324]: 4.16 scrub starts 2024-09-21T12:40:01.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:01 smithi114 ceph-mon[30324]: 4.16 scrub ok 2024-09-21T12:40:01.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:01 smithi114 ceph-mon[30324]: 1.8 scrub starts 2024-09-21T12:40:01.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:01 smithi114 ceph-mon[30324]: 1.8 scrub ok 2024-09-21T12:40:01.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:01 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2358149718' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-21T12:40:01.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:01 smithi114 ceph-mon[30324]: osdmap e509: 8 total, 8 up, 8 in 2024-09-21T12:40:01.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:01 smithi114 ceph-mon[30324]: 3.1e scrub starts 2024-09-21T12:40:01.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:01 smithi114 ceph-mon[30324]: 3.1e scrub ok 2024-09-21T12:40:01.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:01 smithi114 ceph-mon[40867]: 3.19 scrub starts 2024-09-21T12:40:01.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:01 smithi114 ceph-mon[40867]: 3.19 scrub ok 2024-09-21T12:40:01.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:01 smithi114 ceph-mon[40867]: 4.16 scrub starts 2024-09-21T12:40:01.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:01 smithi114 ceph-mon[40867]: 4.16 scrub ok 2024-09-21T12:40:01.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:01 smithi114 ceph-mon[40867]: 1.8 scrub starts 2024-09-21T12:40:01.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:01 smithi114 ceph-mon[40867]: 1.8 scrub ok 2024-09-21T12:40:01.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:01 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2358149718' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-21T12:40:01.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:01 smithi114 ceph-mon[40867]: osdmap e509: 8 total, 8 up, 8 in 2024-09-21T12:40:01.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:01 smithi114 ceph-mon[40867]: 3.1e scrub starts 2024-09-21T12:40:01.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:01 smithi114 ceph-mon[40867]: 3.1e scrub ok 2024-09-21T12:40:01.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:01 smithi183 ceph-mon[33277]: 3.19 scrub starts 2024-09-21T12:40:01.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:01 smithi183 ceph-mon[33277]: 3.19 scrub ok 2024-09-21T12:40:01.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:01 smithi183 ceph-mon[33277]: 4.16 scrub starts 2024-09-21T12:40:01.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:01 smithi183 ceph-mon[33277]: 4.16 scrub ok 2024-09-21T12:40:01.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:01 smithi183 ceph-mon[33277]: 1.8 scrub starts 2024-09-21T12:40:01.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:01 smithi183 ceph-mon[33277]: 1.8 scrub ok 2024-09-21T12:40:01.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:01 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2358149718' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-21T12:40:01.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:01 smithi183 ceph-mon[33277]: osdmap e509: 8 total, 8 up, 8 in 2024-09-21T12:40:01.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:01 smithi183 ceph-mon[33277]: 3.1e scrub starts 2024-09-21T12:40:01.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:01 smithi183 ceph-mon[33277]: 3.1e scrub ok 2024-09-21T12:40:02.416 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T12:40:02.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:02 smithi183 ceph-mon[33277]: 6.17 scrub starts 2024-09-21T12:40:02.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:02 smithi183 ceph-mon[33277]: 6.17 scrub ok 2024-09-21T12:40:02.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:02 smithi183 ceph-mon[33277]: pgmap v1022: 162 pgs: 2 peering, 160 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T12:40:02.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:02 smithi183 ceph-mon[33277]: osdmap e510: 8 total, 8 up, 8 in 2024-09-21T12:40:02.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:02 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2144619255' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:40:02.774 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:40:02 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:40:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:40:02.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:02 smithi114 ceph-mon[40867]: 6.17 scrub starts 2024-09-21T12:40:02.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:02 smithi114 ceph-mon[40867]: 6.17 scrub ok 2024-09-21T12:40:02.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:02 smithi114 ceph-mon[40867]: pgmap v1022: 162 pgs: 2 peering, 160 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T12:40:02.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:02 smithi114 ceph-mon[40867]: osdmap e510: 8 total, 8 up, 8 in 2024-09-21T12:40:02.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:02 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2144619255' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:40:02.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:02 smithi114 ceph-mon[30324]: 6.17 scrub starts 2024-09-21T12:40:02.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:02 smithi114 ceph-mon[30324]: 6.17 scrub ok 2024-09-21T12:40:02.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:02 smithi114 ceph-mon[30324]: pgmap v1022: 162 pgs: 2 peering, 160 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T12:40:02.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:02 smithi114 ceph-mon[30324]: osdmap e510: 8 total, 8 up, 8 in 2024-09-21T12:40:02.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:02 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2144619255' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:40:03.349 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 7 1 2024-09-21T12:40:03.600 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:40:03.720 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:03 smithi114 ceph-mon[40867]: 5.16 scrub starts 2024-09-21T12:40:03.721 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:03 smithi114 ceph-mon[40867]: 5.16 scrub ok 2024-09-21T12:40:03.721 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:03 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2144619255' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T12:40:03.721 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:03 smithi114 ceph-mon[40867]: osdmap e511: 8 total, 8 up, 8 in 2024-09-21T12:40:03.721 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:03 smithi114 ceph-mon[40867]: 6.f scrub starts 2024-09-21T12:40:03.721 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:03 smithi114 ceph-mon[40867]: 6.f scrub ok 2024-09-21T12:40:03.721 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:03 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:40:03.721 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:03 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:40:03.721 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:03 smithi114 ceph-mon[30324]: 5.16 scrub starts 2024-09-21T12:40:03.721 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:03 smithi114 ceph-mon[30324]: 5.16 scrub ok 2024-09-21T12:40:03.721 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:03 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2144619255' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T12:40:03.721 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:03 smithi114 ceph-mon[30324]: osdmap e511: 8 total, 8 up, 8 in 2024-09-21T12:40:03.722 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:03 smithi114 ceph-mon[30324]: 6.f scrub starts 2024-09-21T12:40:03.722 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:03 smithi114 ceph-mon[30324]: 6.f scrub ok 2024-09-21T12:40:03.722 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:03 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:40:03.722 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:03 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:40:03.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:03 smithi183 ceph-mon[33277]: 5.16 scrub starts 2024-09-21T12:40:03.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:03 smithi183 ceph-mon[33277]: 5.16 scrub ok 2024-09-21T12:40:03.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:03 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2144619255' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T12:40:03.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:03 smithi183 ceph-mon[33277]: osdmap e511: 8 total, 8 up, 8 in 2024-09-21T12:40:03.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:03 smithi183 ceph-mon[33277]: 6.f scrub starts 2024-09-21T12:40:03.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:03 smithi183 ceph-mon[33277]: 6.f scrub ok 2024-09-21T12:40:03.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:03 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:40:03.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:03 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:40:04.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:04 smithi114 ceph-mon[30324]: 3.0 scrub starts 2024-09-21T12:40:04.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:04 smithi114 ceph-mon[30324]: 3.0 scrub ok 2024-09-21T12:40:04.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:04 smithi114 ceph-mon[30324]: 3.1f scrub starts 2024-09-21T12:40:04.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:04 smithi114 ceph-mon[30324]: 3.1f scrub ok 2024-09-21T12:40:04.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:04 smithi114 ceph-mon[30324]: pgmap v1025: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T12:40:04.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:04 smithi114 ceph-mon[30324]: 4.19 scrub starts 2024-09-21T12:40:04.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:04 smithi114 ceph-mon[30324]: 4.19 scrub ok 2024-09-21T12:40:04.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:04 smithi114 ceph-mon[30324]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-21T12:40:04.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:04 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/977112299' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:40:04.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:04 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:40:04.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:04 smithi114 ceph-mon[40867]: 3.0 scrub starts 2024-09-21T12:40:04.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:04 smithi114 ceph-mon[40867]: 3.0 scrub ok 2024-09-21T12:40:04.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:04 smithi114 ceph-mon[40867]: 3.1f scrub starts 2024-09-21T12:40:04.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:04 smithi114 ceph-mon[40867]: 3.1f scrub ok 2024-09-21T12:40:04.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:04 smithi114 ceph-mon[40867]: pgmap v1025: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T12:40:04.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:04 smithi114 ceph-mon[40867]: 4.19 scrub starts 2024-09-21T12:40:04.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:04 smithi114 ceph-mon[40867]: 4.19 scrub ok 2024-09-21T12:40:04.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:04 smithi114 ceph-mon[40867]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-21T12:40:04.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:04 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/977112299' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:40:04.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:04 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:40:04.682 INFO:teuthology.orchestra.run.smithi114.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-21T12:40:04.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:04 smithi183 ceph-mon[33277]: 3.0 scrub starts 2024-09-21T12:40:04.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:04 smithi183 ceph-mon[33277]: 3.0 scrub ok 2024-09-21T12:40:04.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:04 smithi183 ceph-mon[33277]: 3.1f scrub starts 2024-09-21T12:40:04.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:04 smithi183 ceph-mon[33277]: 3.1f scrub ok 2024-09-21T12:40:04.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:04 smithi183 ceph-mon[33277]: pgmap v1025: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T12:40:04.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:04 smithi183 ceph-mon[33277]: 4.19 scrub starts 2024-09-21T12:40:04.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:04 smithi183 ceph-mon[33277]: 4.19 scrub ok 2024-09-21T12:40:04.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:04 smithi183 ceph-mon[33277]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-21T12:40:04.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:04 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/977112299' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:40:04.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:04 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T12:40:05.359 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T12:40:05.607 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:40:05.984 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:05 smithi114 ceph-mon[30324]: 3.b scrub starts 2024-09-21T12:40:05.984 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:05 smithi114 ceph-mon[30324]: 3.b scrub ok 2024-09-21T12:40:05.984 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:05 smithi114 ceph-mon[30324]: 1.14 scrub starts 2024-09-21T12:40:05.984 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:05 smithi114 ceph-mon[30324]: 1.14 scrub ok 2024-09-21T12:40:05.985 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:05 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-21T12:40:05.985 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:05 smithi114 ceph-mon[30324]: osdmap e512: 8 total, 8 up, 8 in 2024-09-21T12:40:05.985 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:05 smithi114 ceph-mon[30324]: 4.6 scrub starts 2024-09-21T12:40:05.985 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:05 smithi114 ceph-mon[30324]: 4.6 scrub ok 2024-09-21T12:40:05.985 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:05 smithi114 ceph-mon[30324]: pgmap v1027: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T12:40:05.985 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:05 smithi114 ceph-mon[40867]: 3.b scrub starts 2024-09-21T12:40:05.985 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:05 smithi114 ceph-mon[40867]: 3.b scrub ok 2024-09-21T12:40:05.985 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:05 smithi114 ceph-mon[40867]: 1.14 scrub starts 2024-09-21T12:40:05.985 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:05 smithi114 ceph-mon[40867]: 1.14 scrub ok 2024-09-21T12:40:05.985 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:05 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-21T12:40:05.985 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:05 smithi114 ceph-mon[40867]: osdmap e512: 8 total, 8 up, 8 in 2024-09-21T12:40:05.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:05 smithi114 ceph-mon[40867]: 4.6 scrub starts 2024-09-21T12:40:05.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:05 smithi114 ceph-mon[40867]: 4.6 scrub ok 2024-09-21T12:40:05.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:05 smithi114 ceph-mon[40867]: pgmap v1027: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T12:40:05.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:05 smithi183 ceph-mon[33277]: 3.b scrub starts 2024-09-21T12:40:05.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:05 smithi183 ceph-mon[33277]: 3.b scrub ok 2024-09-21T12:40:05.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:05 smithi183 ceph-mon[33277]: 1.14 scrub starts 2024-09-21T12:40:05.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:05 smithi183 ceph-mon[33277]: 1.14 scrub ok 2024-09-21T12:40:05.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:05 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-21T12:40:05.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:05 smithi183 ceph-mon[33277]: osdmap e512: 8 total, 8 up, 8 in 2024-09-21T12:40:05.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:05 smithi183 ceph-mon[33277]: 4.6 scrub starts 2024-09-21T12:40:05.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:05 smithi183 ceph-mon[33277]: 4.6 scrub ok 2024-09-21T12:40:05.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:05 smithi183 ceph-mon[33277]: pgmap v1027: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T12:40:06.689 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T12:40:06.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:06 smithi183 ceph-mon[33277]: 2.0 scrub starts 2024-09-21T12:40:06.988 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:06 smithi183 ceph-mon[33277]: 2.0 scrub ok 2024-09-21T12:40:06.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:06 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3452110876' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:40:06.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:06 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:40:07.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:06 smithi114 ceph-mon[30324]: 2.0 scrub starts 2024-09-21T12:40:07.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:06 smithi114 ceph-mon[30324]: 2.0 scrub ok 2024-09-21T12:40:07.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:06 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3452110876' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:40:07.024 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:06 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:40:07.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:06 smithi114 ceph-mon[40867]: 2.0 scrub starts 2024-09-21T12:40:07.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:06 smithi114 ceph-mon[40867]: 2.0 scrub ok 2024-09-21T12:40:07.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:06 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3452110876' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:40:07.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:06 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:40:07.477 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 2 1 2024-09-21T12:40:07.724 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:40:07.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:07 smithi114 ceph-mon[30324]: 5.c scrub starts 2024-09-21T12:40:07.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:07 smithi114 ceph-mon[30324]: 5.c scrub ok 2024-09-21T12:40:07.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:07 smithi114 ceph-mon[30324]: 2.1 scrub starts 2024-09-21T12:40:07.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:07 smithi114 ceph-mon[30324]: 2.1 scrub ok 2024-09-21T12:40:07.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:07 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T12:40:07.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:07 smithi114 ceph-mon[30324]: osdmap e513: 8 total, 8 up, 8 in 2024-09-21T12:40:07.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:07 smithi114 ceph-mon[30324]: 5.b scrub starts 2024-09-21T12:40:07.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:07 smithi114 ceph-mon[30324]: 5.b scrub ok 2024-09-21T12:40:07.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:07 smithi114 ceph-mon[30324]: osdmap e514: 8 total, 8 up, 8 in 2024-09-21T12:40:07.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:07 smithi114 ceph-mon[30324]: pgmap v1030: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T12:40:07.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:07 smithi114 ceph-mon[40867]: 5.c scrub starts 2024-09-21T12:40:07.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:07 smithi114 ceph-mon[40867]: 5.c scrub ok 2024-09-21T12:40:07.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:07 smithi114 ceph-mon[40867]: 2.1 scrub starts 2024-09-21T12:40:07.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:07 smithi114 ceph-mon[40867]: 2.1 scrub ok 2024-09-21T12:40:07.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:07 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T12:40:07.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:07 smithi114 ceph-mon[40867]: osdmap e513: 8 total, 8 up, 8 in 2024-09-21T12:40:07.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:07 smithi114 ceph-mon[40867]: 5.b scrub starts 2024-09-21T12:40:07.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:07 smithi114 ceph-mon[40867]: 5.b scrub ok 2024-09-21T12:40:07.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:07 smithi114 ceph-mon[40867]: osdmap e514: 8 total, 8 up, 8 in 2024-09-21T12:40:07.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:07 smithi114 ceph-mon[40867]: pgmap v1030: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T12:40:07.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:40:07 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:40:07.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:07 smithi183 ceph-mon[33277]: 5.c scrub starts 2024-09-21T12:40:07.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:07 smithi183 ceph-mon[33277]: 5.c scrub ok 2024-09-21T12:40:07.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:07 smithi183 ceph-mon[33277]: 2.1 scrub starts 2024-09-21T12:40:07.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:07 smithi183 ceph-mon[33277]: 2.1 scrub ok 2024-09-21T12:40:07.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:07 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T12:40:07.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:07 smithi183 ceph-mon[33277]: osdmap e513: 8 total, 8 up, 8 in 2024-09-21T12:40:07.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:07 smithi183 ceph-mon[33277]: 5.b scrub starts 2024-09-21T12:40:07.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:07 smithi183 ceph-mon[33277]: 5.b scrub ok 2024-09-21T12:40:07.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:07 smithi183 ceph-mon[33277]: osdmap e514: 8 total, 8 up, 8 in 2024-09-21T12:40:07.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:07 smithi183 ceph-mon[33277]: pgmap v1030: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T12:40:09.197 INFO:teuthology.orchestra.run.smithi114.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-21T12:40:09.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:09 smithi183 ceph-mon[33277]: 4.2 scrub starts 2024-09-21T12:40:09.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:09 smithi183 ceph-mon[33277]: 4.2 scrub ok 2024-09-21T12:40:09.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:09 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:40:09.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:09 smithi183 ceph-mon[33277]: 3.12 scrub starts 2024-09-21T12:40:09.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:09 smithi183 ceph-mon[33277]: 3.12 scrub ok 2024-09-21T12:40:09.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:09 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2286320224' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:40:09.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:09 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:40:09.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:09 smithi114 ceph-mon[30324]: 4.2 scrub starts 2024-09-21T12:40:09.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:09 smithi114 ceph-mon[30324]: 4.2 scrub ok 2024-09-21T12:40:09.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:09 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:40:09.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:09 smithi114 ceph-mon[30324]: 3.12 scrub starts 2024-09-21T12:40:09.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:09 smithi114 ceph-mon[30324]: 3.12 scrub ok 2024-09-21T12:40:09.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:09 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2286320224' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:40:09.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:09 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:40:09.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:09 smithi114 ceph-mon[40867]: 4.2 scrub starts 2024-09-21T12:40:09.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:09 smithi114 ceph-mon[40867]: 4.2 scrub ok 2024-09-21T12:40:09.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:09 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:40:09.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:09 smithi114 ceph-mon[40867]: 3.12 scrub starts 2024-09-21T12:40:09.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:09 smithi114 ceph-mon[40867]: 3.12 scrub ok 2024-09-21T12:40:09.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:09 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2286320224' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:40:09.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:09 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T12:40:09.895 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-21T12:40:09.895 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-21T12:40:09.895 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-21T12:40:09.895 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-21T12:40:09.895 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-21T12:40:09.895 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T12:40:10.147 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:40:10.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:10 smithi183 ceph-mon[33277]: 3.17 scrub starts 2024-09-21T12:40:10.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:10 smithi183 ceph-mon[33277]: 3.17 scrub ok 2024-09-21T12:40:10.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:10 smithi183 ceph-mon[33277]: 4.5 scrub starts 2024-09-21T12:40:10.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:10 smithi183 ceph-mon[33277]: 4.5 scrub ok 2024-09-21T12:40:10.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:10 smithi183 ceph-mon[33277]: 6.a scrub starts 2024-09-21T12:40:10.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:10 smithi183 ceph-mon[33277]: 6.a scrub ok 2024-09-21T12:40:10.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:10 smithi183 ceph-mon[33277]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-21T12:40:10.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:10 smithi183 ceph-mon[33277]: osdmap e515: 8 total, 8 up, 8 in 2024-09-21T12:40:10.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:10 smithi183 ceph-mon[33277]: pgmap v1032: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T12:40:10.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:10 smithi114 ceph-mon[30324]: 3.17 scrub starts 2024-09-21T12:40:10.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:10 smithi114 ceph-mon[30324]: 3.17 scrub ok 2024-09-21T12:40:10.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:10 smithi114 ceph-mon[30324]: 4.5 scrub starts 2024-09-21T12:40:10.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:10 smithi114 ceph-mon[30324]: 4.5 scrub ok 2024-09-21T12:40:10.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:10 smithi114 ceph-mon[30324]: 6.a scrub starts 2024-09-21T12:40:10.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:10 smithi114 ceph-mon[30324]: 6.a scrub ok 2024-09-21T12:40:10.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:10 smithi114 ceph-mon[30324]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-21T12:40:10.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:10 smithi114 ceph-mon[30324]: osdmap e515: 8 total, 8 up, 8 in 2024-09-21T12:40:10.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:10 smithi114 ceph-mon[30324]: pgmap v1032: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T12:40:10.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:10 smithi114 ceph-mon[40867]: 3.17 scrub starts 2024-09-21T12:40:10.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:10 smithi114 ceph-mon[40867]: 3.17 scrub ok 2024-09-21T12:40:10.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:10 smithi114 ceph-mon[40867]: 4.5 scrub starts 2024-09-21T12:40:10.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:10 smithi114 ceph-mon[40867]: 4.5 scrub ok 2024-09-21T12:40:10.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:10 smithi114 ceph-mon[40867]: 6.a scrub starts 2024-09-21T12:40:10.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:10 smithi114 ceph-mon[40867]: 6.a scrub ok 2024-09-21T12:40:10.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:10 smithi114 ceph-mon[40867]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-21T12:40:10.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:10 smithi114 ceph-mon[40867]: osdmap e515: 8 total, 8 up, 8 in 2024-09-21T12:40:10.527 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:10 smithi114 ceph-mon[40867]: pgmap v1032: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T12:40:10.813 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:40:10.813 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":516,"fsid":"e2ad975c-7812-11ef-baf6-efdc52797490","created":"2024-09-21T12:14:24.181227+0000","modified":"2024-09-21T12:40:10.190682+0000","last_up_change":"2024-09-21T12:38:06.326515+0000","last_in_change":"2024-09-21T12:35:59.095504+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":19,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T12:17:38.357647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"218","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"216","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.059999942779541,"score_stable":2.059999942779541,"optimal_score":1,"raw_score_acting":2.059999942779541,"raw_score_stable":2.059999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-21T12:20:12.761046+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":61,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-21T12:20:13.713971+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-21T12:20:15.368765+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-21T12:20:17.314275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-21T12:20:19.370218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5715e521-c7dc-41dd-a075-5c96a88fa9b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":472,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3555869816},{"type":"v1","addr":"172.21.15.114:6803","nonce":3555869816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3555869816},{"type":"v1","addr":"172.21.15.114:6805","nonce":3555869816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3555869816},{"type":"v1","addr":"172.21.15.114:6809","nonce":3555869816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3555869816},{"type":"v1","addr":"172.21.15.114:6807","nonce":3555869816}]},"public_addr":"172.21.15.114:6803/3555869816","cluster_addr":"172.21.15.114:6805/3555869816","heartbeat_back_addr":"172.21.15.114:6809/3555869816","heartbeat_front_addr":"172.21.15.114:6807/3555869816","state":["exists","up"]},{"osd":1,"uuid":"1fe160a7-5972-4e3e-99ce-15e2311ba4c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":470,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":2744663863},{"type":"v1","addr":"172.21.15.114:6811","nonce":2744663863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":2744663863},{"type":"v1","addr":"172.21.15.114:6813","nonce":2744663863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":2744663863},{"type":"v1","addr":"172.21.15.114:6817","nonce":2744663863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":2744663863},{"type":"v1","addr":"172.21.15.114:6815","nonce":2744663863}]},"public_addr":"172.21.15.114:6811/2744663863","cluster_addr":"172.21.15.114:6813/2744663863","heartbeat_back_addr":"172.21.15.114:6817/2744663863","heartbeat_front_addr":"172.21.15.114:6815/2744663863","state":["exists","up"]},{"osd":2,"uuid":"6a63581f-7603-49ae-9ab3-b37e7c91ab91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":387,"up_from":455,"up_thru":515,"down_at":388,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1359699248},{"type":"v1","addr":"172.21.15.114:6819","nonce":1359699248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1359699248},{"type":"v1","addr":"172.21.15.114:6821","nonce":1359699248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":1359699248},{"type":"v1","addr":"172.21.15.114:6825","nonce":1359699248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1359699248},{"type":"v1","addr":"172.21.15.114:6823","nonce":1359699248}]},"public_addr":"172.21.15.114:6819/1359699248","cluster_addr":"172.21.15.114:6821/1359699248","heartbeat_back_addr":"172.21.15.114:6825/1359699248","heartbeat_front_addr":"172.21.15.114:6823/1359699248","state":["exists","up"]},{"osd":3,"uuid":"0ad68cdb-ae5c-4090-8a8d-b67c1177ae42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":469,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":1312590318},{"type":"v1","addr":"172.21.15.114:6827","nonce":1312590318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":1312590318},{"type":"v1","addr":"172.21.15.114:6829","nonce":1312590318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1312590318},{"type":"v1","addr":"172.21.15.114:6833","nonce":1312590318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":1312590318},{"type":"v1","addr":"172.21.15.114:6831","nonce":1312590318}]},"public_addr":"172.21.15.114:6827/1312590318","cluster_addr":"172.21.15.114:6829/1312590318","heartbeat_back_addr":"172.21.15.114:6833/1312590318","heartbeat_front_addr":"172.21.15.114:6831/1312590318","state":["exists","up"]},{"osd":4,"uuid":"26bb8a84-e41d-4425-8491-2dea9764b7cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":465,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":1391644836},{"type":"v1","addr":"172.21.15.183:6801","nonce":1391644836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":1391644836},{"type":"v1","addr":"172.21.15.183:6803","nonce":1391644836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":1391644836},{"type":"v1","addr":"172.21.15.183:6807","nonce":1391644836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":1391644836},{"type":"v1","addr":"172.21.15.183:6805","nonce":1391644836}]},"public_addr":"172.21.15.183:6801/1391644836","cluster_addr":"172.21.15.183:6803/1391644836","heartbeat_back_addr":"172.21.15.183:6807/1391644836","heartbeat_front_addr":"172.21.15.183:6805/1391644836","state":["exists","up"]},{"osd":5,"uuid":"33f96871-0869-4281-82b9-bd518987c551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":506,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":3187643473},{"type":"v1","addr":"172.21.15.183:6809","nonce":3187643473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":3187643473},{"type":"v1","addr":"172.21.15.183:6811","nonce":3187643473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":3187643473},{"type":"v1","addr":"172.21.15.183:6815","nonce":3187643473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":3187643473},{"type":"v1","addr":"172.21.15.183:6813","nonce":3187643473}]},"public_addr":"172.21.15.183:6809/3187643473","cluster_addr":"172.21.15.183:6811/3187643473","heartbeat_back_addr":"172.21.15.183:6815/3187643473","heartbeat_front_addr":"172.21.15.183:6813/3187643473","state":["exists","up"]},{"osd":6,"uuid":"e09c72b8-ad5a-4779-a874-b149fce6ed70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":509,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3025199631},{"type":"v1","addr":"172.21.15.183:6817","nonce":3025199631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3025199631},{"type":"v1","addr":"172.21.15.183:6819","nonce":3025199631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":3025199631},{"type":"v1","addr":"172.21.15.183:6823","nonce":3025199631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3025199631},{"type":"v1","addr":"172.21.15.183:6821","nonce":3025199631}]},"public_addr":"172.21.15.183:6817/3025199631","cluster_addr":"172.21.15.183:6819/3025199631","heartbeat_back_addr":"172.21.15.183:6823/3025199631","heartbeat_front_addr":"172.21.15.183:6821/3025199631","state":["exists","up"]},{"osd":7,"uuid":"1a24194d-1af6-485f-8856-98844a02dd9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":476,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3203337442},{"type":"v1","addr":"172.21.15.183:6825","nonce":3203337442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":3203337442},{"type":"v1","addr":"172.21.15.183:6827","nonce":3203337442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3203337442},{"type":"v1","addr":"172.21.15.183:6831","nonce":3203337442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3203337442},{"type":"v1","addr":"172.21.15.183:6829","nonce":3203337442}]},"public_addr":"172.21.15.183:6825/3203337442","cluster_addr":"172.21.15.183:6827/3203337442","heartbeat_back_addr":"172.21.15.183:6831/3203337442","heartbeat_front_addr":"172.21.15.183:6829/3203337442","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:39:31.082597+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:39:29.994681+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:39:35.704346+0000","dead_epoch":387},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:39:54.482733+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:39:53.485560+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:39:42.503988+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:38:46.394641+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:38:45.052604+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"6.5","osds":[2,0,6]}],"pg_upmap_items":[{"pgid":"4.3","mappings":[{"from":0,"to":5}]},{"pgid":"4.c","mappings":[{"from":3,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.15","mappings":[{"from":3,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":5}]},{"pgid":"5.1d","mappings":[{"from":0,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6801/237768393":"2024-09-22T12:20:45.528856+0000","172.21.15.114:6800/237768393":"2024-09-22T12:20:45.528856+0000","172.21.15.114:0/898990943":"2024-09-22T12:14:53.998653+0000","172.21.15.114:0/3866305956":"2024-09-22T12:20:45.528856+0000","172.21.15.114:0/3598782088":"2024-09-22T12:20:45.528856+0000","172.21.15.114:0/2620151541":"2024-09-22T12:15:20.045217+0000","172.21.15.114:6800/1725148595":"2024-09-22T12:14:53.998653+0000","172.21.15.114:0/442404805":"2024-09-22T12:14:53.998653+0000","172.21.15.114:6801/1725148595":"2024-09-22T12:14:53.998653+0000","172.21.15.114:0/4173909019":"2024-09-22T12:20:45.528856+0000","172.21.15.114:0/2106467614":"2024-09-22T12:15:20.045217+0000","172.21.15.114:0/139803357":"2024-09-22T12:15:20.045217+0000","172.21.15.114:6801/770932721":"2024-09-22T12:15:20.045217+0000","172.21.15.114:6800/770932721":"2024-09-22T12:15:20.045217+0000","172.21.15.114:0/1457547294":"2024-09-22T12:20:45.528856+0000","172.21.15.114:0/3416259228":"2024-09-22T12:14:53.998653+0000","172.21.15.114:0/995404371":"2024-09-22T12:20:45.528856+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T12:40:11.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:11 smithi183 ceph-mon[33277]: 6.1d scrub starts 2024-09-21T12:40:11.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:11 smithi183 ceph-mon[33277]: 6.1d scrub ok 2024-09-21T12:40:11.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:11 smithi183 ceph-mon[33277]: 3.1 scrub starts 2024-09-21T12:40:11.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:11 smithi183 ceph-mon[33277]: 3.1 scrub ok 2024-09-21T12:40:11.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:11 smithi183 ceph-mon[33277]: 1.1b scrub starts 2024-09-21T12:40:11.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:11 smithi183 ceph-mon[33277]: 1.1b scrub ok 2024-09-21T12:40:11.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:11 smithi183 ceph-mon[33277]: osdmap e516: 8 total, 8 up, 8 in 2024-09-21T12:40:11.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:11 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1691622328' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T12:40:11.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:11 smithi114 ceph-mon[40867]: 6.1d scrub starts 2024-09-21T12:40:11.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:11 smithi114 ceph-mon[40867]: 6.1d scrub ok 2024-09-21T12:40:11.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:11 smithi114 ceph-mon[40867]: 3.1 scrub starts 2024-09-21T12:40:11.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:11 smithi114 ceph-mon[40867]: 3.1 scrub ok 2024-09-21T12:40:11.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:11 smithi114 ceph-mon[40867]: 1.1b scrub starts 2024-09-21T12:40:11.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:11 smithi114 ceph-mon[40867]: 1.1b scrub ok 2024-09-21T12:40:11.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:11 smithi114 ceph-mon[40867]: osdmap e516: 8 total, 8 up, 8 in 2024-09-21T12:40:11.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:11 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1691622328' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T12:40:11.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:11 smithi114 ceph-mon[30324]: 6.1d scrub starts 2024-09-21T12:40:11.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:11 smithi114 ceph-mon[30324]: 6.1d scrub ok 2024-09-21T12:40:11.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:11 smithi114 ceph-mon[30324]: 3.1 scrub starts 2024-09-21T12:40:11.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:11 smithi114 ceph-mon[30324]: 3.1 scrub ok 2024-09-21T12:40:11.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:11 smithi114 ceph-mon[30324]: 1.1b scrub starts 2024-09-21T12:40:11.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:11 smithi114 ceph-mon[30324]: 1.1b scrub ok 2024-09-21T12:40:11.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:11 smithi114 ceph-mon[30324]: osdmap e516: 8 total, 8 up, 8 in 2024-09-21T12:40:11.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:11 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1691622328' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T12:40:11.615 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-21T12:40:11.615 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T12:40:11.867 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:40:12.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:12 smithi114 ceph-mon[30324]: pgmap v1034: 162 pgs: 1 activating, 161 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 48 B/s, 3 objects/s recovering 2024-09-21T12:40:12.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:12 smithi114 ceph-mon[30324]: 5.17 scrub starts 2024-09-21T12:40:12.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:12 smithi114 ceph-mon[30324]: 5.17 scrub ok 2024-09-21T12:40:12.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:12 smithi114 ceph-mon[30324]: 1.d scrub starts 2024-09-21T12:40:12.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:12 smithi114 ceph-mon[30324]: 1.d scrub ok 2024-09-21T12:40:12.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:12 smithi114 ceph-mon[30324]: 5.d scrub starts 2024-09-21T12:40:12.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:12 smithi114 ceph-mon[30324]: 5.d scrub ok 2024-09-21T12:40:12.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:12 smithi114 ceph-mon[40867]: pgmap v1034: 162 pgs: 1 activating, 161 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 48 B/s, 3 objects/s recovering 2024-09-21T12:40:12.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:12 smithi114 ceph-mon[40867]: 5.17 scrub starts 2024-09-21T12:40:12.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:12 smithi114 ceph-mon[40867]: 5.17 scrub ok 2024-09-21T12:40:12.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:12 smithi114 ceph-mon[40867]: 1.d scrub starts 2024-09-21T12:40:12.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:12 smithi114 ceph-mon[40867]: 1.d scrub ok 2024-09-21T12:40:12.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:12 smithi114 ceph-mon[40867]: 5.d scrub starts 2024-09-21T12:40:12.276 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:12 smithi114 ceph-mon[40867]: 5.d scrub ok 2024-09-21T12:40:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:12 smithi183 ceph-mon[33277]: pgmap v1034: 162 pgs: 1 activating, 161 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 48 B/s, 3 objects/s recovering 2024-09-21T12:40:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:12 smithi183 ceph-mon[33277]: 5.17 scrub starts 2024-09-21T12:40:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:12 smithi183 ceph-mon[33277]: 5.17 scrub ok 2024-09-21T12:40:12.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:12 smithi183 ceph-mon[33277]: 1.d scrub starts 2024-09-21T12:40:12.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:12 smithi183 ceph-mon[33277]: 1.d scrub ok 2024-09-21T12:40:12.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:12 smithi183 ceph-mon[33277]: 5.d scrub starts 2024-09-21T12:40:12.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:12 smithi183 ceph-mon[33277]: 5.d scrub ok 2024-09-21T12:40:12.564 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:40:12.565 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":516,"fsid":"e2ad975c-7812-11ef-baf6-efdc52797490","created":"2024-09-21T12:14:24.181227+0000","modified":"2024-09-21T12:40:10.190682+0000","last_up_change":"2024-09-21T12:38:06.326515+0000","last_in_change":"2024-09-21T12:35:59.095504+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":19,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T12:17:38.357647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"218","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"216","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.059999942779541,"score_stable":2.059999942779541,"optimal_score":1,"raw_score_acting":2.059999942779541,"raw_score_stable":2.059999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-21T12:20:12.761046+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":61,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-21T12:20:13.713971+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-21T12:20:15.368765+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-21T12:20:17.314275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-21T12:20:19.370218+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5715e521-c7dc-41dd-a075-5c96a88fa9b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":472,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3555869816},{"type":"v1","addr":"172.21.15.114:6803","nonce":3555869816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3555869816},{"type":"v1","addr":"172.21.15.114:6805","nonce":3555869816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3555869816},{"type":"v1","addr":"172.21.15.114:6809","nonce":3555869816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3555869816},{"type":"v1","addr":"172.21.15.114:6807","nonce":3555869816}]},"public_addr":"172.21.15.114:6803/3555869816","cluster_addr":"172.21.15.114:6805/3555869816","heartbeat_back_addr":"172.21.15.114:6809/3555869816","heartbeat_front_addr":"172.21.15.114:6807/3555869816","state":["exists","up"]},{"osd":1,"uuid":"1fe160a7-5972-4e3e-99ce-15e2311ba4c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":470,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":2744663863},{"type":"v1","addr":"172.21.15.114:6811","nonce":2744663863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":2744663863},{"type":"v1","addr":"172.21.15.114:6813","nonce":2744663863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":2744663863},{"type":"v1","addr":"172.21.15.114:6817","nonce":2744663863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":2744663863},{"type":"v1","addr":"172.21.15.114:6815","nonce":2744663863}]},"public_addr":"172.21.15.114:6811/2744663863","cluster_addr":"172.21.15.114:6813/2744663863","heartbeat_back_addr":"172.21.15.114:6817/2744663863","heartbeat_front_addr":"172.21.15.114:6815/2744663863","state":["exists","up"]},{"osd":2,"uuid":"6a63581f-7603-49ae-9ab3-b37e7c91ab91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":387,"up_from":455,"up_thru":515,"down_at":388,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1359699248},{"type":"v1","addr":"172.21.15.114:6819","nonce":1359699248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1359699248},{"type":"v1","addr":"172.21.15.114:6821","nonce":1359699248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":1359699248},{"type":"v1","addr":"172.21.15.114:6825","nonce":1359699248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1359699248},{"type":"v1","addr":"172.21.15.114:6823","nonce":1359699248}]},"public_addr":"172.21.15.114:6819/1359699248","cluster_addr":"172.21.15.114:6821/1359699248","heartbeat_back_addr":"172.21.15.114:6825/1359699248","heartbeat_front_addr":"172.21.15.114:6823/1359699248","state":["exists","up"]},{"osd":3,"uuid":"0ad68cdb-ae5c-4090-8a8d-b67c1177ae42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":469,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":1312590318},{"type":"v1","addr":"172.21.15.114:6827","nonce":1312590318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":1312590318},{"type":"v1","addr":"172.21.15.114:6829","nonce":1312590318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1312590318},{"type":"v1","addr":"172.21.15.114:6833","nonce":1312590318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":1312590318},{"type":"v1","addr":"172.21.15.114:6831","nonce":1312590318}]},"public_addr":"172.21.15.114:6827/1312590318","cluster_addr":"172.21.15.114:6829/1312590318","heartbeat_back_addr":"172.21.15.114:6833/1312590318","heartbeat_front_addr":"172.21.15.114:6831/1312590318","state":["exists","up"]},{"osd":4,"uuid":"26bb8a84-e41d-4425-8491-2dea9764b7cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":465,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":1391644836},{"type":"v1","addr":"172.21.15.183:6801","nonce":1391644836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":1391644836},{"type":"v1","addr":"172.21.15.183:6803","nonce":1391644836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":1391644836},{"type":"v1","addr":"172.21.15.183:6807","nonce":1391644836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":1391644836},{"type":"v1","addr":"172.21.15.183:6805","nonce":1391644836}]},"public_addr":"172.21.15.183:6801/1391644836","cluster_addr":"172.21.15.183:6803/1391644836","heartbeat_back_addr":"172.21.15.183:6807/1391644836","heartbeat_front_addr":"172.21.15.183:6805/1391644836","state":["exists","up"]},{"osd":5,"uuid":"33f96871-0869-4281-82b9-bd518987c551","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":506,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":3187643473},{"type":"v1","addr":"172.21.15.183:6809","nonce":3187643473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":3187643473},{"type":"v1","addr":"172.21.15.183:6811","nonce":3187643473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":3187643473},{"type":"v1","addr":"172.21.15.183:6815","nonce":3187643473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":3187643473},{"type":"v1","addr":"172.21.15.183:6813","nonce":3187643473}]},"public_addr":"172.21.15.183:6809/3187643473","cluster_addr":"172.21.15.183:6811/3187643473","heartbeat_back_addr":"172.21.15.183:6815/3187643473","heartbeat_front_addr":"172.21.15.183:6813/3187643473","state":["exists","up"]},{"osd":6,"uuid":"e09c72b8-ad5a-4779-a874-b149fce6ed70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":509,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3025199631},{"type":"v1","addr":"172.21.15.183:6817","nonce":3025199631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3025199631},{"type":"v1","addr":"172.21.15.183:6819","nonce":3025199631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":3025199631},{"type":"v1","addr":"172.21.15.183:6823","nonce":3025199631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3025199631},{"type":"v1","addr":"172.21.15.183:6821","nonce":3025199631}]},"public_addr":"172.21.15.183:6817/3025199631","cluster_addr":"172.21.15.183:6819/3025199631","heartbeat_back_addr":"172.21.15.183:6823/3025199631","heartbeat_front_addr":"172.21.15.183:6821/3025199631","state":["exists","up"]},{"osd":7,"uuid":"1a24194d-1af6-485f-8856-98844a02dd9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":476,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3203337442},{"type":"v1","addr":"172.21.15.183:6825","nonce":3203337442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":3203337442},{"type":"v1","addr":"172.21.15.183:6827","nonce":3203337442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3203337442},{"type":"v1","addr":"172.21.15.183:6831","nonce":3203337442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3203337442},{"type":"v1","addr":"172.21.15.183:6829","nonce":3203337442}]},"public_addr":"172.21.15.183:6825/3203337442","cluster_addr":"172.21.15.183:6827/3203337442","heartbeat_back_addr":"172.21.15.183:6831/3203337442","heartbeat_front_addr":"172.21.15.183:6829/3203337442","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:39:31.082597+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:39:29.994681+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:39:35.704346+0000","dead_epoch":387},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:39:54.482733+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:39:53.485560+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:39:42.503988+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:38:46.394641+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:38:45.052604+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"6.5","osds":[2,0,6]}],"pg_upmap_items":[{"pgid":"4.3","mappings":[{"from":0,"to":5}]},{"pgid":"4.c","mappings":[{"from":3,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.15","mappings":[{"from":3,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":5}]},{"pgid":"5.1d","mappings":[{"from":0,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6801/237768393":"2024-09-22T12:20:45.528856+0000","172.21.15.114:6800/237768393":"2024-09-22T12:20:45.528856+0000","172.21.15.114:0/898990943":"2024-09-22T12:14:53.998653+0000","172.21.15.114:0/3866305956":"2024-09-22T12:20:45.528856+0000","172.21.15.114:0/3598782088":"2024-09-22T12:20:45.528856+0000","172.21.15.114:0/2620151541":"2024-09-22T12:15:20.045217+0000","172.21.15.114:6800/1725148595":"2024-09-22T12:14:53.998653+0000","172.21.15.114:0/442404805":"2024-09-22T12:14:53.998653+0000","172.21.15.114:6801/1725148595":"2024-09-22T12:14:53.998653+0000","172.21.15.114:0/4173909019":"2024-09-22T12:20:45.528856+0000","172.21.15.114:0/2106467614":"2024-09-22T12:15:20.045217+0000","172.21.15.114:0/139803357":"2024-09-22T12:15:20.045217+0000","172.21.15.114:6801/770932721":"2024-09-22T12:15:20.045217+0000","172.21.15.114:6800/770932721":"2024-09-22T12:15:20.045217+0000","172.21.15.114:0/1457547294":"2024-09-22T12:20:45.528856+0000","172.21.15.114:0/3416259228":"2024-09-22T12:14:53.998653+0000","172.21.15.114:0/995404371":"2024-09-22T12:20:45.528856+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T12:40:12.916 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:40:12 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:40:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:40:13.296 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-21T12:40:13.296 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-21T12:40:13.297 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-21T12:40:13.297 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-21T12:40:13.297 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-21T12:40:13.298 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-21T12:40:13.298 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-21T12:40:13.299 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-21T12:40:13.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:13 smithi183 ceph-mon[33277]: 3.5 scrub starts 2024-09-21T12:40:13.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:13 smithi183 ceph-mon[33277]: 3.5 scrub ok 2024-09-21T12:40:13.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:13 smithi183 ceph-mon[33277]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-21T12:40:13.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:13 smithi183 ceph-mon[33277]: 4.10 scrub starts 2024-09-21T12:40:13.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:13 smithi183 ceph-mon[33277]: 4.10 scrub ok 2024-09-21T12:40:13.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:13 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1588790249' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T12:40:13.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:13 smithi183 ceph-mon[33277]: 6.12 scrub starts 2024-09-21T12:40:13.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:13 smithi183 ceph-mon[33277]: 6.12 scrub ok 2024-09-21T12:40:13.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:13 smithi114 ceph-mon[40867]: 3.5 scrub starts 2024-09-21T12:40:13.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:13 smithi114 ceph-mon[40867]: 3.5 scrub ok 2024-09-21T12:40:13.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:13 smithi114 ceph-mon[40867]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-21T12:40:13.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:13 smithi114 ceph-mon[40867]: 4.10 scrub starts 2024-09-21T12:40:13.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:13 smithi114 ceph-mon[40867]: 4.10 scrub ok 2024-09-21T12:40:13.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:13 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1588790249' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T12:40:13.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:13 smithi114 ceph-mon[40867]: 6.12 scrub starts 2024-09-21T12:40:13.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:13 smithi114 ceph-mon[40867]: 6.12 scrub ok 2024-09-21T12:40:13.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:13 smithi114 ceph-mon[30324]: 3.5 scrub starts 2024-09-21T12:40:13.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:13 smithi114 ceph-mon[30324]: 3.5 scrub ok 2024-09-21T12:40:13.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:13 smithi114 ceph-mon[30324]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-21T12:40:13.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:13 smithi114 ceph-mon[30324]: 4.10 scrub starts 2024-09-21T12:40:13.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:13 smithi114 ceph-mon[30324]: 4.10 scrub ok 2024-09-21T12:40:13.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:13 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1588790249' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T12:40:13.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:13 smithi114 ceph-mon[30324]: 6.12 scrub starts 2024-09-21T12:40:13.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:13 smithi114 ceph-mon[30324]: 6.12 scrub ok 2024-09-21T12:40:13.830 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:40:13.832 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:40:13.832 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:40:13.834 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:40:13.838 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:40:13.839 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:40:13.839 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:40:13.843 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:40:14.261 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:14 smithi114 ceph-mon[40867]: pgmap v1035: 162 pgs: 1 activating, 161 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1016 B/s rd, 0 op/s; 36 B/s, 2 objects/s recovering 2024-09-21T12:40:14.262 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:14 smithi114 ceph-mon[40867]: 1.19 scrub starts 2024-09-21T12:40:14.262 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:14 smithi114 ceph-mon[40867]: 1.19 scrub ok 2024-09-21T12:40:14.262 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:14 smithi114 ceph-mon[30324]: pgmap v1035: 162 pgs: 1 activating, 161 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1016 B/s rd, 0 op/s; 36 B/s, 2 objects/s recovering 2024-09-21T12:40:14.262 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:14 smithi114 ceph-mon[30324]: 1.19 scrub starts 2024-09-21T12:40:14.262 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:14 smithi114 ceph-mon[30324]: 1.19 scrub ok 2024-09-21T12:40:14.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:14 smithi183 ceph-mon[33277]: pgmap v1035: 162 pgs: 1 activating, 161 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1016 B/s rd, 0 op/s; 36 B/s, 2 objects/s recovering 2024-09-21T12:40:14.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:14 smithi183 ceph-mon[33277]: 1.19 scrub starts 2024-09-21T12:40:14.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:14 smithi183 ceph-mon[33277]: 1.19 scrub ok 2024-09-21T12:40:15.431 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:15 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:40:15.431 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:15 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:40:15.431 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:15 smithi114 ceph-mon[40867]: osdmap e517: 8 total, 8 up, 8 in 2024-09-21T12:40:15.431 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:15 smithi114 ceph-mon[40867]: 6.c scrub starts 2024-09-21T12:40:15.431 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:15 smithi114 ceph-mon[40867]: 6.c scrub ok 2024-09-21T12:40:15.431 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:15 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:40:15.431 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:15 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:40:15.432 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:15 smithi114 ceph-mon[30324]: osdmap e517: 8 total, 8 up, 8 in 2024-09-21T12:40:15.432 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:15 smithi114 ceph-mon[30324]: 6.c scrub starts 2024-09-21T12:40:15.432 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:15 smithi114 ceph-mon[30324]: 6.c scrub ok 2024-09-21T12:40:15.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:15 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:40:15.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:15 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:40:15.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:15 smithi183 ceph-mon[33277]: osdmap e517: 8 total, 8 up, 8 in 2024-09-21T12:40:15.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:15 smithi183 ceph-mon[33277]: 6.c scrub starts 2024-09-21T12:40:15.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:15 smithi183 ceph-mon[33277]: 6.c scrub ok 2024-09-21T12:40:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:16 smithi183 ceph-mon[33277]: 1.1d scrub starts 2024-09-21T12:40:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:16 smithi183 ceph-mon[33277]: 1.1d scrub ok 2024-09-21T12:40:16.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:16 smithi183 ceph-mon[33277]: purged_snaps scrub starts 2024-09-21T12:40:16.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:16 smithi183 ceph-mon[33277]: purged_snaps scrub ok 2024-09-21T12:40:16.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:16 smithi183 ceph-mon[33277]: pgmap v1038: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 36 B/s, 2 objects/s recovering 2024-09-21T12:40:16.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:16 smithi183 ceph-mon[33277]: osdmap e518: 8 total, 8 up, 8 in 2024-09-21T12:40:16.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:16 smithi183 ceph-mon[33277]: 4.0 scrub starts 2024-09-21T12:40:16.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:16 smithi183 ceph-mon[33277]: 4.0 scrub ok 2024-09-21T12:40:16.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:16 smithi183 ceph-mon[33277]: 5.a scrub starts 2024-09-21T12:40:16.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:16 smithi183 ceph-mon[33277]: 5.a scrub ok 2024-09-21T12:40:16.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:16 smithi183 ceph-mon[33277]: 3.e scrub starts 2024-09-21T12:40:16.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:16 smithi183 ceph-mon[33277]: 3.e scrub ok 2024-09-21T12:40:16.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:16 smithi114 ceph-mon[30324]: 1.1d scrub starts 2024-09-21T12:40:16.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:16 smithi114 ceph-mon[30324]: 1.1d scrub ok 2024-09-21T12:40:16.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:16 smithi114 ceph-mon[30324]: purged_snaps scrub starts 2024-09-21T12:40:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:16 smithi114 ceph-mon[30324]: purged_snaps scrub ok 2024-09-21T12:40:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:16 smithi114 ceph-mon[30324]: pgmap v1038: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 36 B/s, 2 objects/s recovering 2024-09-21T12:40:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:16 smithi114 ceph-mon[30324]: osdmap e518: 8 total, 8 up, 8 in 2024-09-21T12:40:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:16 smithi114 ceph-mon[30324]: 4.0 scrub starts 2024-09-21T12:40:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:16 smithi114 ceph-mon[30324]: 4.0 scrub ok 2024-09-21T12:40:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:16 smithi114 ceph-mon[30324]: 5.a scrub starts 2024-09-21T12:40:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:16 smithi114 ceph-mon[30324]: 5.a scrub ok 2024-09-21T12:40:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:16 smithi114 ceph-mon[30324]: 3.e scrub starts 2024-09-21T12:40:16.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:16 smithi114 ceph-mon[30324]: 3.e scrub ok 2024-09-21T12:40:16.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:16 smithi114 ceph-mon[40867]: 1.1d scrub starts 2024-09-21T12:40:16.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:16 smithi114 ceph-mon[40867]: 1.1d scrub ok 2024-09-21T12:40:16.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:16 smithi114 ceph-mon[40867]: purged_snaps scrub starts 2024-09-21T12:40:16.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:16 smithi114 ceph-mon[40867]: purged_snaps scrub ok 2024-09-21T12:40:16.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:16 smithi114 ceph-mon[40867]: pgmap v1038: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 36 B/s, 2 objects/s recovering 2024-09-21T12:40:16.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:16 smithi114 ceph-mon[40867]: osdmap e518: 8 total, 8 up, 8 in 2024-09-21T12:40:16.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:16 smithi114 ceph-mon[40867]: 4.0 scrub starts 2024-09-21T12:40:16.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:16 smithi114 ceph-mon[40867]: 4.0 scrub ok 2024-09-21T12:40:16.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:16 smithi114 ceph-mon[40867]: 5.a scrub starts 2024-09-21T12:40:16.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:16 smithi114 ceph-mon[40867]: 5.a scrub ok 2024-09-21T12:40:16.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:16 smithi114 ceph-mon[40867]: 3.e scrub starts 2024-09-21T12:40:16.526 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:16 smithi114 ceph-mon[40867]: 3.e scrub ok 2024-09-21T12:40:16.763 INFO:teuthology.orchestra.run.smithi114.stdout:231928234228 2024-09-21T12:40:16.764 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-21T12:40:17.256 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:17 smithi114 ceph-mon[40867]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-09-21T12:40:17.257 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:17 smithi114 ceph-mon[40867]: 4.f scrub starts 2024-09-21T12:40:17.257 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:17 smithi114 ceph-mon[40867]: 4.f scrub ok 2024-09-21T12:40:17.257 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:17 smithi114 ceph-mon[30324]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-09-21T12:40:17.257 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:17 smithi114 ceph-mon[30324]: 4.f scrub starts 2024-09-21T12:40:17.257 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:17 smithi114 ceph-mon[30324]: 4.f scrub ok 2024-09-21T12:40:17.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:17 smithi183 ceph-mon[33277]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-09-21T12:40:17.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:17 smithi183 ceph-mon[33277]: 4.f scrub starts 2024-09-21T12:40:17.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:17 smithi183 ceph-mon[33277]: 4.f scrub ok 2024-09-21T12:40:18.233 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:40:17 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:40:18.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:18 smithi183 ceph-mon[33277]: pgmap v1039: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s; 31 B/s, 1 objects/s recovering 2024-09-21T12:40:18.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:18 smithi183 ceph-mon[33277]: 3.15 scrub starts 2024-09-21T12:40:18.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:18 smithi183 ceph-mon[33277]: 3.15 scrub ok 2024-09-21T12:40:18.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:18 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:40:18.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:18 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:40:18.521 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:18 smithi114 ceph-mon[30324]: pgmap v1039: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s; 31 B/s, 1 objects/s recovering 2024-09-21T12:40:18.522 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:18 smithi114 ceph-mon[30324]: 3.15 scrub starts 2024-09-21T12:40:18.522 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:18 smithi114 ceph-mon[30324]: 3.15 scrub ok 2024-09-21T12:40:18.522 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:18 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:40:18.522 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:18 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:40:18.522 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:18 smithi114 ceph-mon[40867]: pgmap v1039: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s; 31 B/s, 1 objects/s recovering 2024-09-21T12:40:18.522 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:18 smithi114 ceph-mon[40867]: 3.15 scrub starts 2024-09-21T12:40:18.522 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:18 smithi114 ceph-mon[40867]: 3.15 scrub ok 2024-09-21T12:40:18.522 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:18 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:40:18.522 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:18 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:40:18.549 INFO:teuthology.orchestra.run.smithi114.stdout:137438953735 2024-09-21T12:40:18.550 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-21T12:40:19.503 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:19 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:40:19.504 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:19 smithi114 ceph-mon[40867]: 5.1f scrub starts 2024-09-21T12:40:19.504 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:19 smithi114 ceph-mon[40867]: 5.1f scrub ok 2024-09-21T12:40:19.504 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:19 smithi114 ceph-mon[40867]: 6.3 scrub starts 2024-09-21T12:40:19.504 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:19 smithi114 ceph-mon[40867]: 6.3 scrub ok 2024-09-21T12:40:19.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:19 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:40:19.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:19 smithi114 ceph-mon[30324]: 5.1f scrub starts 2024-09-21T12:40:19.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:19 smithi114 ceph-mon[30324]: 5.1f scrub ok 2024-09-21T12:40:19.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:19 smithi114 ceph-mon[30324]: 6.3 scrub starts 2024-09-21T12:40:19.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:19 smithi114 ceph-mon[30324]: 6.3 scrub ok 2024-09-21T12:40:19.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:19 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:40:19.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:19 smithi183 ceph-mon[33277]: 5.1f scrub starts 2024-09-21T12:40:19.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:19 smithi183 ceph-mon[33277]: 5.1f scrub ok 2024-09-21T12:40:19.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:19 smithi183 ceph-mon[33277]: 6.3 scrub starts 2024-09-21T12:40:19.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:19 smithi183 ceph-mon[33277]: 6.3 scrub ok 2024-09-21T12:40:19.911 INFO:teuthology.orchestra.run.smithi114.stdout:34359738656 2024-09-21T12:40:19.911 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-21T12:40:20.402 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:20 smithi114 ceph-mon[40867]: 6.10 scrub starts 2024-09-21T12:40:20.402 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:20 smithi114 ceph-mon[40867]: 6.10 scrub ok 2024-09-21T12:40:20.402 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:20 smithi114 ceph-mon[40867]: pgmap v1040: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:40:20.402 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:20 smithi114 ceph-mon[40867]: 5.8 scrub starts 2024-09-21T12:40:20.402 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:20 smithi114 ceph-mon[40867]: 5.8 scrub ok 2024-09-21T12:40:20.402 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:20 smithi114 ceph-mon[40867]: 3.d scrub starts 2024-09-21T12:40:20.402 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:20 smithi114 ceph-mon[40867]: 3.d scrub ok 2024-09-21T12:40:20.403 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:20 smithi114 ceph-mon[30324]: 6.10 scrub starts 2024-09-21T12:40:20.403 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:20 smithi114 ceph-mon[30324]: 6.10 scrub ok 2024-09-21T12:40:20.403 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:20 smithi114 ceph-mon[30324]: pgmap v1040: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:40:20.403 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:20 smithi114 ceph-mon[30324]: 5.8 scrub starts 2024-09-21T12:40:20.403 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:20 smithi114 ceph-mon[30324]: 5.8 scrub ok 2024-09-21T12:40:20.403 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:20 smithi114 ceph-mon[30324]: 3.d scrub starts 2024-09-21T12:40:20.403 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:20 smithi114 ceph-mon[30324]: 3.d scrub ok 2024-09-21T12:40:20.640 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:20 smithi183 ceph-mon[33277]: 6.10 scrub starts 2024-09-21T12:40:20.640 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:20 smithi183 ceph-mon[33277]: 6.10 scrub ok 2024-09-21T12:40:20.640 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:20 smithi183 ceph-mon[33277]: pgmap v1040: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:40:20.640 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:20 smithi183 ceph-mon[33277]: 5.8 scrub starts 2024-09-21T12:40:20.640 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:20 smithi183 ceph-mon[33277]: 5.8 scrub ok 2024-09-21T12:40:20.640 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:20 smithi183 ceph-mon[33277]: 3.d scrub starts 2024-09-21T12:40:20.640 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:20 smithi183 ceph-mon[33277]: 3.d scrub ok 2024-09-21T12:40:20.796 INFO:teuthology.orchestra.run.smithi114.stdout:1954210119707 2024-09-21T12:40:20.796 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-21T12:40:21.366 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:21 smithi114 ceph-mon[30324]: 6.1 scrub starts 2024-09-21T12:40:21.366 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:21 smithi114 ceph-mon[30324]: 6.1 scrub ok 2024-09-21T12:40:21.366 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:21 smithi114 ceph-mon[30324]: 1.17 scrub starts 2024-09-21T12:40:21.366 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:21 smithi114 ceph-mon[30324]: 1.17 scrub ok 2024-09-21T12:40:21.367 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:21 smithi114 ceph-mon[40867]: 6.1 scrub starts 2024-09-21T12:40:21.367 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:21 smithi114 ceph-mon[40867]: 6.1 scrub ok 2024-09-21T12:40:21.367 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:21 smithi114 ceph-mon[40867]: 1.17 scrub starts 2024-09-21T12:40:21.367 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:21 smithi114 ceph-mon[40867]: 1.17 scrub ok 2024-09-21T12:40:21.395 INFO:teuthology.orchestra.run.smithi114.stdout:111669149965 2024-09-21T12:40:21.396 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-21T12:40:21.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:21 smithi183 ceph-mon[33277]: 6.1 scrub starts 2024-09-21T12:40:21.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:21 smithi183 ceph-mon[33277]: 6.1 scrub ok 2024-09-21T12:40:21.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:21 smithi183 ceph-mon[33277]: 1.17 scrub starts 2024-09-21T12:40:21.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:21 smithi183 ceph-mon[33277]: 1.17 scrub ok 2024-09-21T12:40:22.472 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:22 smithi114 ceph-mon[40867]: pgmap v1041: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T12:40:22.472 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:22 smithi114 ceph-mon[40867]: 3.1c scrub starts 2024-09-21T12:40:22.472 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:22 smithi114 ceph-mon[40867]: 3.1c scrub ok 2024-09-21T12:40:22.473 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:22 smithi114 ceph-mon[30324]: pgmap v1041: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T12:40:22.473 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:22 smithi114 ceph-mon[30324]: 3.1c scrub starts 2024-09-21T12:40:22.473 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:22 smithi114 ceph-mon[30324]: 3.1c scrub ok 2024-09-21T12:40:22.502 INFO:teuthology.orchestra.run.smithi114.stdout:167503724800 2024-09-21T12:40:22.502 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-21T12:40:22.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:22 smithi183 ceph-mon[33277]: pgmap v1041: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T12:40:22.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:22 smithi183 ceph-mon[33277]: 3.1c scrub starts 2024-09-21T12:40:22.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:22 smithi183 ceph-mon[33277]: 3.1c scrub ok 2024-09-21T12:40:22.774 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:40:22 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:40:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:40:22.920 INFO:teuthology.orchestra.run.smithi114.stdout:201863463165 2024-09-21T12:40:22.920 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-21T12:40:23.382 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:40:23.424 INFO:teuthology.orchestra.run.smithi114.stdout:55834575130 2024-09-21T12:40:23.424 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-21T12:40:23.447 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:40:23.448 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:40:23.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:23 smithi114 ceph-mon[30324]: 6.e scrub starts 2024-09-21T12:40:23.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:23 smithi114 ceph-mon[30324]: 6.e scrub ok 2024-09-21T12:40:23.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:23 smithi114 ceph-mon[30324]: 1.f scrub starts 2024-09-21T12:40:23.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:23 smithi114 ceph-mon[30324]: 1.f scrub ok 2024-09-21T12:40:23.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:23 smithi114 ceph-mon[30324]: 5.5 scrub starts 2024-09-21T12:40:23.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:23 smithi114 ceph-mon[30324]: 5.5 scrub ok 2024-09-21T12:40:23.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:23 smithi114 ceph-mon[30324]: 1.15 scrub starts 2024-09-21T12:40:23.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:23 smithi114 ceph-mon[30324]: 1.15 scrub ok 2024-09-21T12:40:23.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:23 smithi114 ceph-mon[40867]: 6.e scrub starts 2024-09-21T12:40:23.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:23 smithi114 ceph-mon[40867]: 6.e scrub ok 2024-09-21T12:40:23.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:23 smithi114 ceph-mon[40867]: 1.f scrub starts 2024-09-21T12:40:23.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:23 smithi114 ceph-mon[40867]: 1.f scrub ok 2024-09-21T12:40:23.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:23 smithi114 ceph-mon[40867]: 5.5 scrub starts 2024-09-21T12:40:23.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:23 smithi114 ceph-mon[40867]: 5.5 scrub ok 2024-09-21T12:40:23.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:23 smithi114 ceph-mon[40867]: 1.15 scrub starts 2024-09-21T12:40:23.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:23 smithi114 ceph-mon[40867]: 1.15 scrub ok 2024-09-21T12:40:23.579 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:40:23.685 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:40:23.685 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:40:23.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:23 smithi183 ceph-mon[33277]: 6.e scrub starts 2024-09-21T12:40:23.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:23 smithi183 ceph-mon[33277]: 6.e scrub ok 2024-09-21T12:40:23.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:23 smithi183 ceph-mon[33277]: 1.f scrub starts 2024-09-21T12:40:23.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:23 smithi183 ceph-mon[33277]: 1.f scrub ok 2024-09-21T12:40:23.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:23 smithi183 ceph-mon[33277]: 5.5 scrub starts 2024-09-21T12:40:23.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:23 smithi183 ceph-mon[33277]: 5.5 scrub ok 2024-09-21T12:40:23.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:23 smithi183 ceph-mon[33277]: 1.15 scrub starts 2024-09-21T12:40:23.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:23 smithi183 ceph-mon[33277]: 1.15 scrub ok 2024-09-21T12:40:24.349 INFO:teuthology.orchestra.run.smithi114.stdout:231928234230 2024-09-21T12:40:24.349 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:24 smithi114 ceph-mon[30324]: 6.0 scrub starts 2024-09-21T12:40:24.349 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:24 smithi114 ceph-mon[30324]: 6.0 scrub ok 2024-09-21T12:40:24.350 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:24 smithi114 ceph-mon[30324]: pgmap v1042: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 9.8 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-21T12:40:24.350 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:24 smithi114 ceph-mon[40867]: 6.0 scrub starts 2024-09-21T12:40:24.350 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:24 smithi114 ceph-mon[40867]: 6.0 scrub ok 2024-09-21T12:40:24.350 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:24 smithi114 ceph-mon[40867]: pgmap v1042: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 9.8 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-21T12:40:24.626 INFO:teuthology.orchestra.run.smithi114.stdout:111669149967 2024-09-21T12:40:24.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:24 smithi183 ceph-mon[33277]: 6.0 scrub starts 2024-09-21T12:40:24.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:24 smithi183 ceph-mon[33277]: 6.0 scrub ok 2024-09-21T12:40:24.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:24 smithi183 ceph-mon[33277]: pgmap v1042: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 9.8 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-21T12:40:25.420 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149965 got 111669149967 for osd.3 2024-09-21T12:40:25.420 DEBUG:teuthology.parallel:result is None 2024-09-21T12:40:25.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:25 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1630208898' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T12:40:25.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:25 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3665703703' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T12:40:25.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:25 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1630208898' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T12:40:25.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:25 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3665703703' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T12:40:25.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:25 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1630208898' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T12:40:25.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:25 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3665703703' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T12:40:26.266 INFO:teuthology.orchestra.run.smithi114.stdout:1954210119709 2024-09-21T12:40:26.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:26 smithi114 ceph-mon[30324]: 4.13 scrub starts 2024-09-21T12:40:26.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:26 smithi114 ceph-mon[30324]: 4.13 scrub ok 2024-09-21T12:40:26.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:26 smithi114 ceph-mon[30324]: pgmap v1043: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s 2024-09-21T12:40:26.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:26 smithi114 ceph-mon[30324]: 6.1e scrub starts 2024-09-21T12:40:26.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:26 smithi114 ceph-mon[30324]: 6.1e scrub ok 2024-09-21T12:40:26.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:26 smithi114 ceph-mon[40867]: 4.13 scrub starts 2024-09-21T12:40:26.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:26 smithi114 ceph-mon[40867]: 4.13 scrub ok 2024-09-21T12:40:26.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:26 smithi114 ceph-mon[40867]: pgmap v1043: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s 2024-09-21T12:40:26.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:26 smithi114 ceph-mon[40867]: 6.1e scrub starts 2024-09-21T12:40:26.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:26 smithi114 ceph-mon[40867]: 6.1e scrub ok 2024-09-21T12:40:26.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:26 smithi183 ceph-mon[33277]: 4.13 scrub starts 2024-09-21T12:40:26.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:26 smithi183 ceph-mon[33277]: 4.13 scrub ok 2024-09-21T12:40:26.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:26 smithi183 ceph-mon[33277]: pgmap v1043: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 24 op/s 2024-09-21T12:40:26.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:26 smithi183 ceph-mon[33277]: 6.1e scrub starts 2024-09-21T12:40:26.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:26 smithi183 ceph-mon[33277]: 6.1e scrub ok 2024-09-21T12:40:26.916 INFO:teuthology.orchestra.run.smithi114.stdout:167503724801 2024-09-21T12:40:27.187 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1954210119707 got 1954210119709 for osd.2 2024-09-21T12:40:27.187 DEBUG:teuthology.parallel:result is None 2024-09-21T12:40:27.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:27 smithi114 ceph-mon[40867]: 1.b scrub starts 2024-09-21T12:40:27.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:27 smithi114 ceph-mon[40867]: 1.b scrub ok 2024-09-21T12:40:27.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:27 smithi114 ceph-mon[40867]: 4.15 scrub starts 2024-09-21T12:40:27.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:27 smithi114 ceph-mon[40867]: 4.15 scrub ok 2024-09-21T12:40:27.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:27 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/1409403994' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T12:40:27.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:27 smithi114 ceph-mon[40867]: 5.1 scrub starts 2024-09-21T12:40:27.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:27 smithi114 ceph-mon[40867]: 5.1 scrub ok 2024-09-21T12:40:27.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:27 smithi114 ceph-mon[40867]: 1.2 scrub starts 2024-09-21T12:40:27.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:27 smithi114 ceph-mon[40867]: 1.2 scrub ok 2024-09-21T12:40:27.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:27 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/999534834' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T12:40:27.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:27 smithi114 ceph-mon[30324]: 1.b scrub starts 2024-09-21T12:40:27.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:27 smithi114 ceph-mon[30324]: 1.b scrub ok 2024-09-21T12:40:27.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:27 smithi114 ceph-mon[30324]: 4.15 scrub starts 2024-09-21T12:40:27.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:27 smithi114 ceph-mon[30324]: 4.15 scrub ok 2024-09-21T12:40:27.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:27 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/1409403994' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T12:40:27.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:27 smithi114 ceph-mon[30324]: 5.1 scrub starts 2024-09-21T12:40:27.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:27 smithi114 ceph-mon[30324]: 5.1 scrub ok 2024-09-21T12:40:27.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:27 smithi114 ceph-mon[30324]: 1.2 scrub starts 2024-09-21T12:40:27.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:27 smithi114 ceph-mon[30324]: 1.2 scrub ok 2024-09-21T12:40:27.526 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:27 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/999534834' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T12:40:27.606 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234228 got 231928234230 for osd.7 2024-09-21T12:40:27.606 DEBUG:teuthology.parallel:result is None 2024-09-21T12:40:27.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:27 smithi183 ceph-mon[33277]: 1.b scrub starts 2024-09-21T12:40:27.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:27 smithi183 ceph-mon[33277]: 1.b scrub ok 2024-09-21T12:40:27.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:27 smithi183 ceph-mon[33277]: 4.15 scrub starts 2024-09-21T12:40:27.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:27 smithi183 ceph-mon[33277]: 4.15 scrub ok 2024-09-21T12:40:27.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:27 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/1409403994' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T12:40:27.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:27 smithi183 ceph-mon[33277]: 5.1 scrub starts 2024-09-21T12:40:27.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:27 smithi183 ceph-mon[33277]: 5.1 scrub ok 2024-09-21T12:40:27.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:27 smithi183 ceph-mon[33277]: 1.2 scrub starts 2024-09-21T12:40:27.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:27 smithi183 ceph-mon[33277]: 1.2 scrub ok 2024-09-21T12:40:27.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:27 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/999534834' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T12:40:27.971 INFO:teuthology.orchestra.run.smithi114.stdout:34359738657 2024-09-21T12:40:28.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:40:27 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:40:28.468 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:40:28.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:28 smithi114 ceph-mon[40867]: pgmap v1044: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-09-21T12:40:28.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:28 smithi114 ceph-mon[40867]: 1.1a scrub starts 2024-09-21T12:40:28.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:28 smithi114 ceph-mon[40867]: 1.1a scrub ok 2024-09-21T12:40:28.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:28 smithi114 ceph-mon[40867]: 4.1c scrub starts 2024-09-21T12:40:28.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:28 smithi114 ceph-mon[40867]: 4.1c scrub ok 2024-09-21T12:40:28.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:28 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/3981567941' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T12:40:28.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:28 smithi114 ceph-mon[30324]: pgmap v1044: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-09-21T12:40:28.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:28 smithi114 ceph-mon[30324]: 1.1a scrub starts 2024-09-21T12:40:28.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:28 smithi114 ceph-mon[30324]: 1.1a scrub ok 2024-09-21T12:40:28.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:28 smithi114 ceph-mon[30324]: 4.1c scrub starts 2024-09-21T12:40:28.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:28 smithi114 ceph-mon[30324]: 4.1c scrub ok 2024-09-21T12:40:28.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:28 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/3981567941' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T12:40:28.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:28 smithi183 ceph-mon[33277]: pgmap v1044: 162 pgs: 162 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-09-21T12:40:28.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:28 smithi183 ceph-mon[33277]: 1.1a scrub starts 2024-09-21T12:40:28.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:28 smithi183 ceph-mon[33277]: 1.1a scrub ok 2024-09-21T12:40:28.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:28 smithi183 ceph-mon[33277]: 4.1c scrub starts 2024-09-21T12:40:28.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:28 smithi183 ceph-mon[33277]: 4.1c scrub ok 2024-09-21T12:40:28.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:28 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/3981567941' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T12:40:28.876 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724800 got 167503724801 for osd.5 2024-09-21T12:40:28.876 DEBUG:teuthology.parallel:result is None 2024-09-21T12:40:29.488 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738656 got 34359738657 for osd.0 2024-09-21T12:40:29.488 DEBUG:teuthology.parallel:result is None 2024-09-21T12:40:29.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:29 smithi114 ceph-mon[30324]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:40:29.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:29 smithi114 ceph-mon[30324]: 4.1e scrub starts 2024-09-21T12:40:29.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:29 smithi114 ceph-mon[30324]: 4.1e scrub ok 2024-09-21T12:40:29.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:29 smithi114 ceph-mon[30324]: 1.9 scrub starts 2024-09-21T12:40:29.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:29 smithi114 ceph-mon[30324]: 1.9 scrub ok 2024-09-21T12:40:29.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:29 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/812419786' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T12:40:29.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:29 smithi114 ceph-mon[40867]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:40:29.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:29 smithi114 ceph-mon[40867]: 4.1e scrub starts 2024-09-21T12:40:29.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:29 smithi114 ceph-mon[40867]: 4.1e scrub ok 2024-09-21T12:40:29.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:29 smithi114 ceph-mon[40867]: 1.9 scrub starts 2024-09-21T12:40:29.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:29 smithi114 ceph-mon[40867]: 1.9 scrub ok 2024-09-21T12:40:29.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:29 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/812419786' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T12:40:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:29 smithi183 ceph-mon[33277]: from='client.24422 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:40:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:29 smithi183 ceph-mon[33277]: 4.1e scrub starts 2024-09-21T12:40:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:29 smithi183 ceph-mon[33277]: 4.1e scrub ok 2024-09-21T12:40:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:29 smithi183 ceph-mon[33277]: 1.9 scrub starts 2024-09-21T12:40:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:29 smithi183 ceph-mon[33277]: 1.9 scrub ok 2024-09-21T12:40:29.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:29 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/812419786' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T12:40:30.104 INFO:teuthology.orchestra.run.smithi114.stdout:137438953738 2024-09-21T12:40:30.150 INFO:teuthology.orchestra.run.smithi114.stdout:201863463167 2024-09-21T12:40:30.320 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:40:30.408 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:30 smithi114 ceph-mon[40867]: 4.e scrub starts 2024-09-21T12:40:30.408 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:30 smithi114 ceph-mon[40867]: 4.e scrub ok 2024-09-21T12:40:30.408 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:30 smithi114 ceph-mon[40867]: pgmap v1045: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-21T12:40:30.408 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:30 smithi114 ceph-mon[40867]: 5.18 scrub starts 2024-09-21T12:40:30.408 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:30 smithi114 ceph-mon[40867]: 5.18 scrub ok 2024-09-21T12:40:30.408 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:30 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/2948886443' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T12:40:30.409 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:30 smithi114 ceph-mon[30324]: 4.e scrub starts 2024-09-21T12:40:30.409 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:30 smithi114 ceph-mon[30324]: 4.e scrub ok 2024-09-21T12:40:30.409 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:30 smithi114 ceph-mon[30324]: pgmap v1045: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-21T12:40:30.409 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:30 smithi114 ceph-mon[30324]: 5.18 scrub starts 2024-09-21T12:40:30.409 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:30 smithi114 ceph-mon[30324]: 5.18 scrub ok 2024-09-21T12:40:30.409 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:30 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/2948886443' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T12:40:30.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:30 smithi183 ceph-mon[33277]: 4.e scrub starts 2024-09-21T12:40:30.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:30 smithi183 ceph-mon[33277]: 4.e scrub ok 2024-09-21T12:40:30.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:30 smithi183 ceph-mon[33277]: pgmap v1045: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-21T12:40:30.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:30 smithi183 ceph-mon[33277]: 5.18 scrub starts 2024-09-21T12:40:30.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:30 smithi183 ceph-mon[33277]: 5.18 scrub ok 2024-09-21T12:40:30.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:30 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/2948886443' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T12:40:30.753 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953735 got 137438953738 for osd.4 2024-09-21T12:40:30.753 DEBUG:teuthology.parallel:result is None 2024-09-21T12:40:31.533 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:31 smithi114 ceph-mon[40867]: 6.1a scrub starts 2024-09-21T12:40:31.534 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:31 smithi114 ceph-mon[40867]: 6.1a scrub ok 2024-09-21T12:40:31.534 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:31 smithi114 ceph-mon[40867]: 1.0 scrub starts 2024-09-21T12:40:31.534 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:31 smithi114 ceph-mon[40867]: 1.0 scrub ok 2024-09-21T12:40:31.534 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:31 smithi114 ceph-mon[30324]: 6.1a scrub starts 2024-09-21T12:40:31.534 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:31 smithi114 ceph-mon[30324]: 6.1a scrub ok 2024-09-21T12:40:31.534 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:31 smithi114 ceph-mon[30324]: 1.0 scrub starts 2024-09-21T12:40:31.534 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:31 smithi114 ceph-mon[30324]: 1.0 scrub ok 2024-09-21T12:40:31.575 INFO:teuthology.orchestra.run.smithi114.stdout:55834575132 2024-09-21T12:40:31.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:31 smithi183 ceph-mon[33277]: 6.1a scrub starts 2024-09-21T12:40:31.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:31 smithi183 ceph-mon[33277]: 6.1a scrub ok 2024-09-21T12:40:31.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:31 smithi183 ceph-mon[33277]: 1.0 scrub starts 2024-09-21T12:40:31.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:31 smithi183 ceph-mon[33277]: 1.0 scrub ok 2024-09-21T12:40:31.971 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863463165 got 201863463167 for osd.6 2024-09-21T12:40:31.971 DEBUG:teuthology.parallel:result is None 2024-09-21T12:40:32.549 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:32 smithi114 ceph-mon[40867]: pgmap v1046: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-21T12:40:32.549 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:32 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/214139133' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T12:40:32.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:32 smithi114 ceph-mon[30324]: pgmap v1046: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-21T12:40:32.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:32 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/214139133' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T12:40:32.653 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575130 got 55834575132 for osd.1 2024-09-21T12:40:32.653 DEBUG:teuthology.parallel:result is None 2024-09-21T12:40:32.653 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T12:40:32.653 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:40:32.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:32 smithi183 ceph-mon[33277]: pgmap v1046: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-21T12:40:32.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:32 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/214139133' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T12:40:32.904 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:40:33.024 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:40:32 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: ::ffff:172.21.15.183 - - [21/Sep/2024:12:40:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T12:40:33.560 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:40:33.560 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:33 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:40:33.561 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:33 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:40:33.566 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:40:33.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:33 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T12:40:34.358 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":1047,"stamp":"2024-09-21T12:40:33.228097+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":1827,"num_read_kb":1579,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":493,"num_bytes_recovered":272805,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3552,"ondisk_log_size":3552,"up":486,"acting":486,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":486,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6031516,"kb_used_data":10844,"kb_used_omap":18,"kb_used_meta":6020525,"kb_avail":743929700,"statfs":{"total":767960285184,"available":761784012800,"internally_reserved":0,"allocated":11104256,"data_stored":7220921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19199,"internal_metadata":6165017857},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":444,"num_read_kb":444,"num_write":286,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003026"},"pg_stats":[{"pgid":"1.1d","version":"76'79","reported_seq":677,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.226603+0000","last_change":"2024-09-21T12:40:14.582628+0000","last_active":"2024-09-21T12:40:15.226603+0000","last_peered":"2024-09-21T12:40:15.226603+0000","last_clean":"2024-09-21T12:40:15.226603+0000","last_became_active":"2024-09-21T12:32:38.426383+0000","last_became_peered":"2024-09-21T12:32:38.426383+0000","last_unstale":"2024-09-21T12:40:15.226603+0000","last_undegraded":"2024-09-21T12:40:15.226603+0000","last_fullsized":"2024-09-21T12:40:15.226603+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:40:14.582579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:40:14.582579+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:30.420194+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":990,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.226558+0000","last_change":"2024-09-21T12:40:02.821040+0000","last_active":"2024-09-21T12:40:15.226558+0000","last_peered":"2024-09-21T12:40:15.226558+0000","last_clean":"2024-09-21T12:40:15.226558+0000","last_became_active":"2024-09-21T12:38:07.167857+0000","last_became_peered":"2024-09-21T12:38:07.167857+0000","last_unstale":"2024-09-21T12:40:15.226558+0000","last_undegraded":"2024-09-21T12:40:15.226558+0000","last_fullsized":"2024-09-21T12:40:15.226558+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:02.820974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:40:02.820974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:21.756947+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"518'21","reported_seq":385,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:22.977153+0000","last_change":"2024-09-21T12:39:16.266371+0000","last_active":"2024-09-21T12:40:22.977153+0000","last_peered":"2024-09-21T12:40:22.977153+0000","last_clean":"2024-09-21T12:40:22.977153+0000","last_became_active":"2024-09-21T12:32:38.419081+0000","last_became_peered":"2024-09-21T12:32:38.419081+0000","last_unstale":"2024-09-21T12:40:22.977153+0000","last_undegraded":"2024-09-21T12:40:22.977153+0000","last_fullsized":"2024-09-21T12:40:22.977153+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:39:16.266285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:16.266285+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:30.189485+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":989,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.224138+0000","last_change":"2024-09-21T12:39:17.175795+0000","last_active":"2024-09-21T12:40:15.224138+0000","last_peered":"2024-09-21T12:40:15.224138+0000","last_clean":"2024-09-21T12:40:15.224138+0000","last_became_active":"2024-09-21T12:36:00.136673+0000","last_became_peered":"2024-09-21T12:36:00.136673+0000","last_unstale":"2024-09-21T12:40:15.224138+0000","last_undegraded":"2024-09-21T12:40:15.224138+0000","last_fullsized":"2024-09-21T12:40:15.224138+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:17.175740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:17.175740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:46.137502+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":374,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:30.447423+0000","last_change":"2024-09-21T12:40:30.447423+0000","last_active":"2024-09-21T12:40:30.447423+0000","last_peered":"2024-09-21T12:40:30.447423+0000","last_clean":"2024-09-21T12:40:30.447423+0000","last_became_active":"2024-09-21T12:32:38.425588+0000","last_became_peered":"2024-09-21T12:32:38.425588+0000","last_unstale":"2024-09-21T12:40:30.447423+0000","last_undegraded":"2024-09-21T12:40:30.447423+0000","last_fullsized":"2024-09-21T12:40:30.447423+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:30.447365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:40:30.447365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:49.853295+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"76'79","reported_seq":233,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.530150+0000","last_change":"2024-09-21T12:39:24.527707+0000","last_active":"2024-09-21T12:40:15.530150+0000","last_peered":"2024-09-21T12:40:15.530150+0000","last_clean":"2024-09-21T12:40:15.530150+0000","last_became_active":"2024-09-21T12:36:00.154209+0000","last_became_peered":"2024-09-21T12:36:00.154209+0000","last_unstale":"2024-09-21T12:40:15.530150+0000","last_undegraded":"2024-09-21T12:40:15.530150+0000","last_fullsized":"2024-09-21T12:40:15.530150+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:24.527622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:24.527622+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:37.668270+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":976,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.527996+0000","last_change":"2024-09-21T12:39:53.588395+0000","last_active":"2024-09-21T12:40:15.527996+0000","last_peered":"2024-09-21T12:40:15.527996+0000","last_clean":"2024-09-21T12:40:15.527996+0000","last_became_active":"2024-09-21T12:36:00.139167+0000","last_became_peered":"2024-09-21T12:36:00.139167+0000","last_unstale":"2024-09-21T12:40:15.527996+0000","last_undegraded":"2024-09-21T12:40:15.527996+0000","last_fullsized":"2024-09-21T12:40:15.527996+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:53.588345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:53.588345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:05.269035+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":987,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.528741+0000","last_change":"2024-09-21T12:40:00.466457+0000","last_active":"2024-09-21T12:40:15.528741+0000","last_peered":"2024-09-21T12:40:15.528741+0000","last_clean":"2024-09-21T12:40:15.528741+0000","last_became_active":"2024-09-21T12:38:07.157964+0000","last_became_peered":"2024-09-21T12:38:07.157964+0000","last_unstale":"2024-09-21T12:40:15.528741+0000","last_undegraded":"2024-09-21T12:40:15.528741+0000","last_fullsized":"2024-09-21T12:40:15.528741+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:00.466413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:40:00.466413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:25.941476+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"518'39","reported_seq":1050,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.171553+0000","last_change":"2024-09-21T12:40:03.491219+0000","last_active":"2024-09-21T12:40:23.171553+0000","last_peered":"2024-09-21T12:40:23.171553+0000","last_clean":"2024-09-21T12:40:23.171553+0000","last_became_active":"2024-09-21T12:38:07.168250+0000","last_became_peered":"2024-09-21T12:38:07.168250+0000","last_unstale":"2024-09-21T12:40:23.171553+0000","last_undegraded":"2024-09-21T12:40:23.171553+0000","last_fullsized":"2024-09-21T12:40:23.171553+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:40:03.491164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:40:03.491164+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:05.177020+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":366,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:29.428119+0000","last_change":"2024-09-21T12:40:29.428119+0000","last_active":"2024-09-21T12:40:29.428119+0000","last_peered":"2024-09-21T12:40:29.428119+0000","last_clean":"2024-09-21T12:40:29.428119+0000","last_became_active":"2024-09-21T12:32:38.417724+0000","last_became_peered":"2024-09-21T12:32:38.417724+0000","last_unstale":"2024-09-21T12:40:29.428119+0000","last_undegraded":"2024-09-21T12:40:29.428119+0000","last_fullsized":"2024-09-21T12:40:29.428119+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:29.428061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:40:29.428061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:32.240082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":527,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.703145+0000","last_change":"2024-09-21T12:39:18.247341+0000","last_active":"2024-09-21T12:40:15.703145+0000","last_peered":"2024-09-21T12:40:15.703145+0000","last_clean":"2024-09-21T12:40:15.703145+0000","last_became_active":"2024-09-21T12:37:20.467925+0000","last_became_peered":"2024-09-21T12:37:20.467925+0000","last_unstale":"2024-09-21T12:40:15.703145+0000","last_undegraded":"2024-09-21T12:40:15.703145+0000","last_fullsized":"2024-09-21T12:40:15.703145+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:18.247283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:18.247283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:30.819585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"518'21","reported_seq":401,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.077851+0000","last_change":"2024-09-21T12:39:38.519883+0000","last_active":"2024-09-21T12:40:23.077851+0000","last_peered":"2024-09-21T12:40:23.077851+0000","last_clean":"2024-09-21T12:40:23.077851+0000","last_became_active":"2024-09-21T12:32:38.426121+0000","last_became_peered":"2024-09-21T12:32:38.426121+0000","last_unstale":"2024-09-21T12:40:23.077851+0000","last_undegraded":"2024-09-21T12:40:23.077851+0000","last_fullsized":"2024-09-21T12:40:23.077851+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:39:38.519813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:38.519813+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:41.134696+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1020,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.702124+0000","last_change":"2024-09-21T12:38:44.245362+0000","last_active":"2024-09-21T12:40:15.702124+0000","last_peered":"2024-09-21T12:40:15.702124+0000","last_clean":"2024-09-21T12:40:15.702124+0000","last_became_active":"2024-09-21T12:36:00.137364+0000","last_became_peered":"2024-09-21T12:36:00.137364+0000","last_unstale":"2024-09-21T12:40:15.702124+0000","last_undegraded":"2024-09-21T12:40:15.702124+0000","last_fullsized":"2024-09-21T12:40:15.702124+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:44.245309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:44.245309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":987,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.225313+0000","last_change":"2024-09-21T12:39:31.655725+0000","last_active":"2024-09-21T12:40:15.225313+0000","last_peered":"2024-09-21T12:40:15.225313+0000","last_clean":"2024-09-21T12:40:15.225313+0000","last_became_active":"2024-09-21T12:36:00.134723+0000","last_became_peered":"2024-09-21T12:36:00.134723+0000","last_unstale":"2024-09-21T12:40:15.225313+0000","last_undegraded":"2024-09-21T12:40:15.225313+0000","last_fullsized":"2024-09-21T12:40:15.225313+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:31.655654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:31.655654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:48.861074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"76'79","reported_seq":761,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.528601+0000","last_change":"2024-09-21T12:39:43.470032+0000","last_active":"2024-09-21T12:40:15.528601+0000","last_peered":"2024-09-21T12:40:15.528601+0000","last_clean":"2024-09-21T12:40:15.528601+0000","last_became_active":"2024-09-21T12:36:00.147795+0000","last_became_peered":"2024-09-21T12:36:00.147795+0000","last_unstale":"2024-09-21T12:40:15.528601+0000","last_undegraded":"2024-09-21T12:40:15.528601+0000","last_fullsized":"2024-09-21T12:40:15.528601+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:43.469978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:43.469978+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:05.783906+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":519,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:22.084456+0000","last_change":"2024-09-21T12:40:22.084456+0000","last_active":"2024-09-21T12:40:22.084456+0000","last_peered":"2024-09-21T12:40:22.084456+0000","last_clean":"2024-09-21T12:40:22.084456+0000","last_became_active":"2024-09-21T12:37:20.468468+0000","last_became_peered":"2024-09-21T12:37:20.468468+0000","last_unstale":"2024-09-21T12:40:22.084456+0000","last_undegraded":"2024-09-21T12:40:22.084456+0000","last_fullsized":"2024-09-21T12:40:22.084456+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:22.084399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:40:22.084399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:22.686670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"518'9","reported_seq":383,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.057245+0000","last_change":"2024-09-21T12:39:19.280732+0000","last_active":"2024-09-21T12:40:23.057245+0000","last_peered":"2024-09-21T12:40:23.057245+0000","last_clean":"2024-09-21T12:40:23.057245+0000","last_became_active":"2024-09-21T12:32:38.422912+0000","last_became_peered":"2024-09-21T12:32:38.422912+0000","last_unstale":"2024-09-21T12:40:23.057245+0000","last_undegraded":"2024-09-21T12:40:23.057245+0000","last_fullsized":"2024-09-21T12:40:23.057245+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'7","last_scrub_stamp":"2024-09-21T12:39:19.280679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:19.280679+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:31.804604+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":233,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.529233+0000","last_change":"2024-09-21T12:39:47.460848+0000","last_active":"2024-09-21T12:40:15.529233+0000","last_peered":"2024-09-21T12:40:15.529233+0000","last_clean":"2024-09-21T12:40:15.529233+0000","last_became_active":"2024-09-21T12:36:00.142106+0000","last_became_peered":"2024-09-21T12:36:00.142106+0000","last_unstale":"2024-09-21T12:40:15.529233+0000","last_undegraded":"2024-09-21T12:40:15.529233+0000","last_fullsized":"2024-09-21T12:40:15.529233+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:47.460786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:47.460786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:09.399546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":980,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.701485+0000","last_change":"2024-09-21T12:39:35.183449+0000","last_active":"2024-09-21T12:40:15.701485+0000","last_peered":"2024-09-21T12:40:15.701485+0000","last_clean":"2024-09-21T12:40:15.701485+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:40:15.701485+0000","last_undegraded":"2024-09-21T12:40:15.701485+0000","last_fullsized":"2024-09-21T12:40:15.701485+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:35.183355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:35.183355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:41.588221+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"76'79","reported_seq":801,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.527981+0000","last_change":"2024-09-21T12:40:13.517471+0000","last_active":"2024-09-21T12:40:15.527981+0000","last_peered":"2024-09-21T12:40:15.527981+0000","last_clean":"2024-09-21T12:40:15.527981+0000","last_became_active":"2024-09-21T12:27:16.957569+0000","last_became_peered":"2024-09-21T12:27:16.957569+0000","last_unstale":"2024-09-21T12:40:15.527981+0000","last_undegraded":"2024-09-21T12:40:15.527981+0000","last_fullsized":"2024-09-21T12:40:15.527981+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:40:13.517297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:40:13.517297+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:33.783424+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":366,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:25.437571+0000","last_change":"2024-09-21T12:40:25.437571+0000","last_active":"2024-09-21T12:40:25.437571+0000","last_peered":"2024-09-21T12:40:25.437571+0000","last_clean":"2024-09-21T12:40:25.437571+0000","last_became_active":"2024-09-21T12:32:38.428997+0000","last_became_peered":"2024-09-21T12:32:38.428997+0000","last_unstale":"2024-09-21T12:40:25.437571+0000","last_undegraded":"2024-09-21T12:40:25.437571+0000","last_fullsized":"2024-09-21T12:40:25.437571+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:25.437512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:40:25.437512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:35.814112+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1050,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.225652+0000","last_change":"2024-09-21T12:39:44.734228+0000","last_active":"2024-09-21T12:40:15.225652+0000","last_peered":"2024-09-21T12:40:15.225652+0000","last_clean":"2024-09-21T12:40:15.225652+0000","last_became_active":"2024-09-21T12:36:00.127153+0000","last_became_peered":"2024-09-21T12:36:00.127153+0000","last_unstale":"2024-09-21T12:40:15.225652+0000","last_undegraded":"2024-09-21T12:40:15.225652+0000","last_fullsized":"2024-09-21T12:40:15.225652+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:44.734176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:44.734176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:53.381683+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"518'39","reported_seq":808,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:27.701492+0000","last_change":"2024-09-21T12:40:27.701492+0000","last_active":"2024-09-21T12:40:27.701492+0000","last_peered":"2024-09-21T12:40:27.701492+0000","last_clean":"2024-09-21T12:40:27.701492+0000","last_became_active":"2024-09-21T12:40:10.205351+0000","last_became_peered":"2024-09-21T12:40:10.205351+0000","last_unstale":"2024-09-21T12:40:27.701492+0000","last_undegraded":"2024-09-21T12:40:27.701492+0000","last_fullsized":"2024-09-21T12:40:27.701492+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"518'39","last_scrub_stamp":"2024-09-21T12:40:27.701423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:40:27.701423+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:50.494609+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1003,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.226829+0000","last_change":"2024-09-21T12:38:53.956579+0000","last_active":"2024-09-21T12:40:15.226829+0000","last_peered":"2024-09-21T12:40:15.226829+0000","last_clean":"2024-09-21T12:40:15.226829+0000","last_became_active":"2024-09-21T12:32:38.425958+0000","last_became_peered":"2024-09-21T12:32:38.425958+0000","last_unstale":"2024-09-21T12:40:15.226829+0000","last_undegraded":"2024-09-21T12:40:15.226829+0000","last_fullsized":"2024-09-21T12:40:15.226829+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:53.956515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:53.956515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"76'79","reported_seq":937,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.701288+0000","last_change":"2024-09-21T12:39:26.166502+0000","last_active":"2024-09-21T12:40:15.701288+0000","last_peered":"2024-09-21T12:40:15.701288+0000","last_clean":"2024-09-21T12:40:15.701288+0000","last_became_active":"2024-09-21T12:38:08.157632+0000","last_became_peered":"2024-09-21T12:38:08.157632+0000","last_unstale":"2024-09-21T12:40:15.701288+0000","last_undegraded":"2024-09-21T12:40:15.701288+0000","last_fullsized":"2024-09-21T12:40:15.701288+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:26.166402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:26.166402+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:46.517049+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":984,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.528581+0000","last_change":"2024-09-21T12:39:18.225857+0000","last_active":"2024-09-21T12:40:15.528581+0000","last_peered":"2024-09-21T12:40:15.528581+0000","last_clean":"2024-09-21T12:40:15.528581+0000","last_became_active":"2024-09-21T12:32:02.033276+0000","last_became_peered":"2024-09-21T12:32:02.033276+0000","last_unstale":"2024-09-21T12:40:15.528581+0000","last_undegraded":"2024-09-21T12:40:15.528581+0000","last_fullsized":"2024-09-21T12:40:15.528581+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:18.225813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:18.225813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:37.203577+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":369,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.221932+0000","last_change":"2024-09-21T12:38:41.208373+0000","last_active":"2024-09-21T12:40:15.221932+0000","last_peered":"2024-09-21T12:40:15.221932+0000","last_clean":"2024-09-21T12:40:15.221932+0000","last_became_active":"2024-09-21T12:38:07.159387+0000","last_became_peered":"2024-09-21T12:38:07.159387+0000","last_unstale":"2024-09-21T12:40:15.221932+0000","last_undegraded":"2024-09-21T12:40:15.221932+0000","last_fullsized":"2024-09-21T12:40:15.221932+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:41.208301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:41.208301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"518'24","reported_seq":1087,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.173326+0000","last_change":"2024-09-21T12:39:27.355062+0000","last_active":"2024-09-21T12:40:23.173326+0000","last_peered":"2024-09-21T12:40:23.173326+0000","last_clean":"2024-09-21T12:40:23.173326+0000","last_became_active":"2024-09-21T12:32:38.409163+0000","last_became_peered":"2024-09-21T12:32:38.409163+0000","last_unstale":"2024-09-21T12:40:23.173326+0000","last_undegraded":"2024-09-21T12:40:23.173326+0000","last_fullsized":"2024-09-21T12:40:23.173326+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T12:39:27.355002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:27.355002+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:51.502466+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":377,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.222516+0000","last_change":"2024-09-21T12:38:49.267225+0000","last_active":"2024-09-21T12:40:15.222516+0000","last_peered":"2024-09-21T12:40:15.222516+0000","last_clean":"2024-09-21T12:40:15.222516+0000","last_became_active":"2024-09-21T12:38:07.161634+0000","last_became_peered":"2024-09-21T12:38:07.161634+0000","last_unstale":"2024-09-21T12:40:15.222516+0000","last_undegraded":"2024-09-21T12:40:15.222516+0000","last_fullsized":"2024-09-21T12:40:15.222516+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:49.267162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:49.267162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"76'79","reported_seq":726,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.701702+0000","last_change":"2024-09-21T12:40:10.186296+0000","last_active":"2024-09-21T12:40:15.701702+0000","last_peered":"2024-09-21T12:40:15.701702+0000","last_clean":"2024-09-21T12:40:15.701702+0000","last_became_active":"2024-09-21T12:32:38.414204+0000","last_became_peered":"2024-09-21T12:32:38.414204+0000","last_unstale":"2024-09-21T12:40:15.701702+0000","last_undegraded":"2024-09-21T12:40:15.701702+0000","last_fullsized":"2024-09-21T12:40:15.701702+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:40:10.186225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:40:10.186225+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:23.922014+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":958,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.529941+0000","last_change":"2024-09-21T12:39:22.535974+0000","last_active":"2024-09-21T12:40:15.529941+0000","last_peered":"2024-09-21T12:40:15.529941+0000","last_clean":"2024-09-21T12:40:15.529941+0000","last_became_active":"2024-09-21T12:38:08.165695+0000","last_became_peered":"2024-09-21T12:38:08.165695+0000","last_unstale":"2024-09-21T12:40:15.529941+0000","last_undegraded":"2024-09-21T12:40:15.529941+0000","last_fullsized":"2024-09-21T12:40:15.529941+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:39:22.535916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:22.535916+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:48.068876+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":1029,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.224275+0000","last_change":"2024-09-21T12:39:59.453047+0000","last_active":"2024-09-21T12:40:15.224275+0000","last_peered":"2024-09-21T12:40:15.224275+0000","last_clean":"2024-09-21T12:40:15.224275+0000","last_became_active":"2024-09-21T12:32:38.425335+0000","last_became_peered":"2024-09-21T12:32:38.425335+0000","last_unstale":"2024-09-21T12:40:15.224275+0000","last_undegraded":"2024-09-21T12:40:15.224275+0000","last_fullsized":"2024-09-21T12:40:15.224275+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:39:59.452988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:59.452988+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:28.155192+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"518'30","reported_seq":1121,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:27.928984+0000","last_change":"2024-09-21T12:40:27.928984+0000","last_active":"2024-09-21T12:40:27.928984+0000","last_peered":"2024-09-21T12:40:27.928984+0000","last_clean":"2024-09-21T12:40:27.928984+0000","last_became_active":"2024-09-21T12:36:00.148761+0000","last_became_peered":"2024-09-21T12:36:00.148761+0000","last_unstale":"2024-09-21T12:40:27.928984+0000","last_undegraded":"2024-09-21T12:40:27.928984+0000","last_fullsized":"2024-09-21T12:40:27.928984+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"518'30","last_scrub_stamp":"2024-09-21T12:40:27.928904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:40:27.928904+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:51.385029+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"510'19","reported_seq":614,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:25.602643+0000","last_change":"2024-09-21T12:40:17.861797+0000","last_active":"2024-09-21T12:40:25.602643+0000","last_peered":"2024-09-21T12:40:25.602643+0000","last_clean":"2024-09-21T12:40:25.602643+0000","last_became_active":"2024-09-21T12:40:01.428161+0000","last_became_peered":"2024-09-21T12:40:01.428161+0000","last_unstale":"2024-09-21T12:40:25.602643+0000","last_undegraded":"2024-09-21T12:40:25.602643+0000","last_fullsized":"2024-09-21T12:40:25.602643+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"510'19","last_scrub_stamp":"2024-09-21T12:40:17.861729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:40:17.861729+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:25.090781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"76'79","reported_seq":537,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.702842+0000","last_change":"2024-09-21T12:39:32.209673+0000","last_active":"2024-09-21T12:40:15.702842+0000","last_peered":"2024-09-21T12:40:15.702842+0000","last_clean":"2024-09-21T12:40:15.702842+0000","last_became_active":"2024-09-21T12:38:07.165640+0000","last_became_peered":"2024-09-21T12:38:07.165640+0000","last_unstale":"2024-09-21T12:40:15.702842+0000","last_undegraded":"2024-09-21T12:40:15.702842+0000","last_fullsized":"2024-09-21T12:40:15.702842+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:32.209596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:32.209596+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:35.994443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"518'39","reported_seq":630,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.179674+0000","last_change":"2024-09-21T12:40:16.616707+0000","last_active":"2024-09-21T12:40:23.179674+0000","last_peered":"2024-09-21T12:40:23.179674+0000","last_clean":"2024-09-21T12:40:23.179674+0000","last_became_active":"2024-09-21T12:40:10.205854+0000","last_became_peered":"2024-09-21T12:40:10.205854+0000","last_unstale":"2024-09-21T12:40:23.179674+0000","last_undegraded":"2024-09-21T12:40:23.179674+0000","last_fullsized":"2024-09-21T12:40:23.179674+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:40:16.616651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:40:16.616651+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:25.142385+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":449,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.527553+0000","last_change":"2024-09-21T12:39:59.459892+0000","last_active":"2024-09-21T12:40:15.527553+0000","last_peered":"2024-09-21T12:40:15.527553+0000","last_clean":"2024-09-21T12:40:15.527553+0000","last_became_active":"2024-09-21T12:36:00.140455+0000","last_became_peered":"2024-09-21T12:36:00.140455+0000","last_unstale":"2024-09-21T12:40:15.527553+0000","last_undegraded":"2024-09-21T12:40:15.527553+0000","last_fullsized":"2024-09-21T12:40:15.527553+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:59.459843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:59.459843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:19.062658+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"352'15","reported_seq":465,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:25.602892+0000","last_change":"2024-09-21T12:38:02.464559+0000","last_active":"2024-09-21T12:40:25.602892+0000","last_peered":"2024-09-21T12:40:25.602892+0000","last_clean":"2024-09-21T12:40:25.602892+0000","last_became_active":"2024-09-21T12:32:38.429092+0000","last_became_peered":"2024-09-21T12:32:38.429092+0000","last_unstale":"2024-09-21T12:40:25.602892+0000","last_undegraded":"2024-09-21T12:40:25.602892+0000","last_fullsized":"2024-09-21T12:40:25.602892+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'15","last_scrub_stamp":"2024-09-21T12:38:02.464486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:02.464486+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":487,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.703385+0000","last_change":"2024-09-21T12:39:44.280084+0000","last_active":"2024-09-21T12:40:15.703385+0000","last_peered":"2024-09-21T12:40:15.703385+0000","last_clean":"2024-09-21T12:40:15.703385+0000","last_became_active":"2024-09-21T12:37:20.468591+0000","last_became_peered":"2024-09-21T12:37:20.468591+0000","last_unstale":"2024-09-21T12:40:15.703385+0000","last_undegraded":"2024-09-21T12:40:15.703385+0000","last_fullsized":"2024-09-21T12:40:15.703385+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:39:44.280022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:44.280022+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:57.269368+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"76'79","reported_seq":377,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.222395+0000","last_change":"2024-09-21T12:38:51.310242+0000","last_active":"2024-09-21T12:40:15.222395+0000","last_peered":"2024-09-21T12:40:15.222395+0000","last_clean":"2024-09-21T12:40:15.222395+0000","last_became_active":"2024-09-21T12:38:07.159280+0000","last_became_peered":"2024-09-21T12:38:07.159280+0000","last_unstale":"2024-09-21T12:40:15.222395+0000","last_undegraded":"2024-09-21T12:40:15.222395+0000","last_fullsized":"2024-09-21T12:40:15.222395+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:38:51.310157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:38:51.310157+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"518'42","reported_seq":1148,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.136358+0000","last_change":"2024-09-21T12:40:15.531899+0000","last_active":"2024-09-21T12:40:23.136358+0000","last_peered":"2024-09-21T12:40:23.136358+0000","last_clean":"2024-09-21T12:40:23.136358+0000","last_became_active":"2024-09-21T12:36:00.139849+0000","last_became_peered":"2024-09-21T12:36:00.139849+0000","last_unstale":"2024-09-21T12:40:23.136358+0000","last_undegraded":"2024-09-21T12:40:23.136358+0000","last_fullsized":"2024-09-21T12:40:23.136358+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T12:40:15.531825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:40:15.531825+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:24.163047+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":981,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.528791+0000","last_change":"2024-09-21T12:39:20.300208+0000","last_active":"2024-09-21T12:40:15.528791+0000","last_peered":"2024-09-21T12:40:15.528791+0000","last_clean":"2024-09-21T12:40:15.528791+0000","last_became_active":"2024-09-21T12:36:00.151488+0000","last_became_peered":"2024-09-21T12:36:00.151488+0000","last_unstale":"2024-09-21T12:40:15.528791+0000","last_undegraded":"2024-09-21T12:40:15.528791+0000","last_fullsized":"2024-09-21T12:40:15.528791+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:20.300049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:20.300049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:33.837202+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":380,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:26.442021+0000","last_change":"2024-09-21T12:40:26.442021+0000","last_active":"2024-09-21T12:40:26.442021+0000","last_peered":"2024-09-21T12:40:26.442021+0000","last_clean":"2024-09-21T12:40:26.442021+0000","last_became_active":"2024-09-21T12:36:00.136650+0000","last_became_peered":"2024-09-21T12:36:00.136650+0000","last_unstale":"2024-09-21T12:40:26.442021+0000","last_undegraded":"2024-09-21T12:40:26.442021+0000","last_fullsized":"2024-09-21T12:40:26.442021+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:26.441959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:40:26.441959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:29.201734+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":369,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.222214+0000","last_change":"2024-09-21T12:38:44.156366+0000","last_active":"2024-09-21T12:40:15.222214+0000","last_peered":"2024-09-21T12:40:15.222214+0000","last_clean":"2024-09-21T12:40:15.222214+0000","last_became_active":"2024-09-21T12:38:07.166919+0000","last_became_peered":"2024-09-21T12:38:07.166919+0000","last_unstale":"2024-09-21T12:40:15.222214+0000","last_undegraded":"2024-09-21T12:40:15.222214+0000","last_fullsized":"2024-09-21T12:40:15.222214+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:44.156307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:38:44.156307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"76'79","reported_seq":916,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.528241+0000","last_change":"2024-09-21T12:39:30.289586+0000","last_active":"2024-09-21T12:40:15.528241+0000","last_peered":"2024-09-21T12:40:15.528241+0000","last_clean":"2024-09-21T12:40:15.528241+0000","last_became_active":"2024-09-21T12:36:00.155047+0000","last_became_peered":"2024-09-21T12:36:00.155047+0000","last_unstale":"2024-09-21T12:40:15.528241+0000","last_undegraded":"2024-09-21T12:40:15.528241+0000","last_fullsized":"2024-09-21T12:40:15.528241+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:30.289532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:30.289532+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:44.792256+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"518'42","reported_seq":395,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.162971+0000","last_change":"2024-09-21T12:39:34.562769+0000","last_active":"2024-09-21T12:40:23.162971+0000","last_peered":"2024-09-21T12:40:23.162971+0000","last_clean":"2024-09-21T12:40:23.162971+0000","last_became_active":"2024-09-21T12:32:38.425453+0000","last_became_peered":"2024-09-21T12:32:38.425453+0000","last_unstale":"2024-09-21T12:40:23.162971+0000","last_undegraded":"2024-09-21T12:40:23.162971+0000","last_fullsized":"2024-09-21T12:40:23.162971+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'28","last_scrub_stamp":"2024-09-21T12:39:34.562691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:34.562691+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:41.975070+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":1065,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.226252+0000","last_change":"2024-09-21T12:39:37.668319+0000","last_active":"2024-09-21T12:40:15.226252+0000","last_peered":"2024-09-21T12:40:15.226252+0000","last_clean":"2024-09-21T12:40:15.226252+0000","last_became_active":"2024-09-21T12:32:38.424412+0000","last_became_peered":"2024-09-21T12:32:38.424412+0000","last_unstale":"2024-09-21T12:40:15.226252+0000","last_undegraded":"2024-09-21T12:40:15.226252+0000","last_fullsized":"2024-09-21T12:40:15.226252+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:39:37.668271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:37.668271+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:06.741317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"352'12","reported_seq":1272,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:28.151395+0000","last_change":"2024-09-21T12:39:47.512406+0000","last_active":"2024-09-21T12:40:28.151395+0000","last_peered":"2024-09-21T12:40:28.151395+0000","last_clean":"2024-09-21T12:40:28.151395+0000","last_became_active":"2024-09-21T12:32:38.409177+0000","last_became_peered":"2024-09-21T12:32:38.409177+0000","last_unstale":"2024-09-21T12:40:28.151395+0000","last_undegraded":"2024-09-21T12:40:28.151395+0000","last_fullsized":"2024-09-21T12:40:28.151395+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'12","last_scrub_stamp":"2024-09-21T12:39:47.512357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:47.512357+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:17.319208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":966,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:18.648172+0000","last_change":"2024-09-21T12:40:18.648172+0000","last_active":"2024-09-21T12:40:18.648172+0000","last_peered":"2024-09-21T12:40:18.648172+0000","last_clean":"2024-09-21T12:40:18.648172+0000","last_became_active":"2024-09-21T12:38:08.154409+0000","last_became_peered":"2024-09-21T12:38:08.154409+0000","last_unstale":"2024-09-21T12:40:18.648172+0000","last_undegraded":"2024-09-21T12:40:18.648172+0000","last_fullsized":"2024-09-21T12:40:18.648172+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:18.648117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:40:18.648117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:21.869154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"76'79","reported_seq":474,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.702457+0000","last_change":"2024-09-21T12:40:01.428877+0000","last_active":"2024-09-21T12:40:15.702457+0000","last_peered":"2024-09-21T12:40:15.702457+0000","last_clean":"2024-09-21T12:40:15.702457+0000","last_became_active":"2024-09-21T12:40:01.428481+0000","last_became_peered":"2024-09-21T12:40:01.428481+0000","last_unstale":"2024-09-21T12:40:15.702457+0000","last_undegraded":"2024-09-21T12:40:15.702457+0000","last_fullsized":"2024-09-21T12:40:15.702457+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:41.480738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:41.480738+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:51.477238+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"518'30","reported_seq":533,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.151322+0000","last_change":"2024-09-21T12:40:07.491352+0000","last_active":"2024-09-21T12:40:23.151322+0000","last_peered":"2024-09-21T12:40:23.151322+0000","last_clean":"2024-09-21T12:40:23.151322+0000","last_became_active":"2024-09-21T12:32:38.424412+0000","last_became_peered":"2024-09-21T12:32:38.424412+0000","last_unstale":"2024-09-21T12:40:23.151322+0000","last_undegraded":"2024-09-21T12:40:23.151322+0000","last_fullsized":"2024-09-21T12:40:23.151322+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:40:07.491287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:40:07.491287+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:07.651957+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":996,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.703112+0000","last_change":"2024-09-21T12:40:12.112607+0000","last_active":"2024-09-21T12:40:15.703112+0000","last_peered":"2024-09-21T12:40:15.703112+0000","last_clean":"2024-09-21T12:40:15.703112+0000","last_became_active":"2024-09-21T12:38:07.165614+0000","last_became_peered":"2024-09-21T12:38:07.165614+0000","last_unstale":"2024-09-21T12:40:15.703112+0000","last_undegraded":"2024-09-21T12:40:15.703112+0000","last_fullsized":"2024-09-21T12:40:15.703112+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:12.112551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:40:12.112551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:25.965104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":1233,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:20.602497+0000","last_change":"2024-09-21T12:39:32.632252+0000","last_active":"2024-09-21T12:40:20.602497+0000","last_peered":"2024-09-21T12:40:20.602497+0000","last_clean":"2024-09-21T12:40:20.602497+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:40:20.602497+0000","last_undegraded":"2024-09-21T12:40:20.602497+0000","last_fullsized":"2024-09-21T12:40:20.602497+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:39:32.632187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:32.632187+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:51.554132+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":979,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:22.930917+0000","last_change":"2024-09-21T12:40:22.930855+0000","last_active":"2024-09-21T12:40:22.930917+0000","last_peered":"2024-09-21T12:40:22.930917+0000","last_clean":"2024-09-21T12:40:22.930917+0000","last_became_active":"2024-09-21T12:38:07.163353+0000","last_became_peered":"2024-09-21T12:38:07.163353+0000","last_unstale":"2024-09-21T12:40:22.930917+0000","last_undegraded":"2024-09-21T12:40:22.930917+0000","last_fullsized":"2024-09-21T12:40:22.930917+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:22.930784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:40:22.930784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:33.007973+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"76'79","reported_seq":474,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.702083+0000","last_change":"2024-09-21T12:40:01.424167+0000","last_active":"2024-09-21T12:40:15.702083+0000","last_peered":"2024-09-21T12:40:15.702083+0000","last_clean":"2024-09-21T12:40:15.702083+0000","last_became_active":"2024-09-21T12:40:01.423746+0000","last_became_peered":"2024-09-21T12:40:01.423746+0000","last_unstale":"2024-09-21T12:40:15.702083+0000","last_undegraded":"2024-09-21T12:40:15.702083+0000","last_fullsized":"2024-09-21T12:40:15.702083+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:57.712232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:57.712232+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:00.968556+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"518'39","reported_seq":1091,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.145409+0000","last_change":"2024-09-21T12:39:30.640700+0000","last_active":"2024-09-21T12:40:23.145409+0000","last_peered":"2024-09-21T12:40:23.145409+0000","last_clean":"2024-09-21T12:40:23.145409+0000","last_became_active":"2024-09-21T12:36:00.127312+0000","last_became_peered":"2024-09-21T12:36:00.127312+0000","last_unstale":"2024-09-21T12:40:23.145409+0000","last_undegraded":"2024-09-21T12:40:23.145409+0000","last_fullsized":"2024-09-21T12:40:23.145409+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'29","last_scrub_stamp":"2024-09-21T12:39:30.640595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:30.640595+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:52.831790+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":972,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.226585+0000","last_change":"2024-09-21T12:38:48.953684+0000","last_active":"2024-09-21T12:40:15.226585+0000","last_peered":"2024-09-21T12:40:15.226585+0000","last_clean":"2024-09-21T12:40:15.226585+0000","last_became_active":"2024-09-21T12:38:07.163439+0000","last_became_peered":"2024-09-21T12:38:07.163439+0000","last_unstale":"2024-09-21T12:40:15.226585+0000","last_undegraded":"2024-09-21T12:40:15.226585+0000","last_fullsized":"2024-09-21T12:40:15.226585+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:48.953635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:48.953635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:17.502212+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":449,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.702596+0000","last_change":"2024-09-21T12:40:01.429852+0000","last_active":"2024-09-21T12:40:15.702596+0000","last_peered":"2024-09-21T12:40:15.702596+0000","last_clean":"2024-09-21T12:40:15.702596+0000","last_became_active":"2024-09-21T12:40:01.428162+0000","last_became_peered":"2024-09-21T12:40:01.428162+0000","last_unstale":"2024-09-21T12:40:15.702596+0000","last_undegraded":"2024-09-21T12:40:15.702596+0000","last_fullsized":"2024-09-21T12:40:15.702596+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:21.815656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:21.815656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:47.004063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":976,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:19.539373+0000","last_change":"2024-09-21T12:40:19.539373+0000","last_active":"2024-09-21T12:40:19.539373+0000","last_peered":"2024-09-21T12:40:19.539373+0000","last_clean":"2024-09-21T12:40:19.539373+0000","last_became_active":"2024-09-21T12:38:07.158992+0000","last_became_peered":"2024-09-21T12:38:07.158992+0000","last_unstale":"2024-09-21T12:40:19.539373+0000","last_undegraded":"2024-09-21T12:40:19.539373+0000","last_fullsized":"2024-09-21T12:40:19.539373+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:19.539305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:40:19.539305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:38.716869+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"76'79","reported_seq":413,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.701238+0000","last_change":"2024-09-21T12:39:30.224719+0000","last_active":"2024-09-21T12:40:15.701238+0000","last_peered":"2024-09-21T12:40:15.701238+0000","last_clean":"2024-09-21T12:40:15.701238+0000","last_became_active":"2024-09-21T12:38:07.162110+0000","last_became_peered":"2024-09-21T12:38:07.162110+0000","last_unstale":"2024-09-21T12:40:15.701238+0000","last_undegraded":"2024-09-21T12:40:15.701238+0000","last_fullsized":"2024-09-21T12:40:15.701238+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:30.224638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:30.224638+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:47.399797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"518'54","reported_seq":1306,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:25.602625+0000","last_change":"2024-09-21T12:39:39.041771+0000","last_active":"2024-09-21T12:40:25.602625+0000","last_peered":"2024-09-21T12:40:25.602625+0000","last_clean":"2024-09-21T12:40:25.602625+0000","last_became_active":"2024-09-21T12:38:07.164073+0000","last_became_peered":"2024-09-21T12:38:07.164073+0000","last_unstale":"2024-09-21T12:40:25.602625+0000","last_undegraded":"2024-09-21T12:40:25.602625+0000","last_fullsized":"2024-09-21T12:40:25.602625+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'42","last_scrub_stamp":"2024-09-21T12:39:39.041700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:39.041700+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:45.259714+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":374,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.222550+0000","last_change":"2024-09-21T12:39:29.418691+0000","last_active":"2024-09-21T12:40:15.222550+0000","last_peered":"2024-09-21T12:40:15.222550+0000","last_clean":"2024-09-21T12:40:15.222550+0000","last_became_active":"2024-09-21T12:32:38.426309+0000","last_became_peered":"2024-09-21T12:32:38.426309+0000","last_unstale":"2024-09-21T12:40:15.222550+0000","last_undegraded":"2024-09-21T12:40:15.222550+0000","last_fullsized":"2024-09-21T12:40:15.222550+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:29.418625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:29.418625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:51.060636+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":362,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.701664+0000","last_change":"2024-09-21T12:39:57.394717+0000","last_active":"2024-09-21T12:40:15.701664+0000","last_peered":"2024-09-21T12:40:15.701664+0000","last_clean":"2024-09-21T12:40:15.701664+0000","last_became_active":"2024-09-21T12:39:57.394373+0000","last_became_peered":"2024-09-21T12:39:57.394373+0000","last_unstale":"2024-09-21T12:40:15.701664+0000","last_undegraded":"2024-09-21T12:40:15.701664+0000","last_fullsized":"2024-09-21T12:40:15.701664+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T12:39:37.041444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:39:37.041444+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:01.747850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011011149999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1015,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:21.966066+0000","last_change":"2024-09-21T12:40:21.966066+0000","last_active":"2024-09-21T12:40:21.966066+0000","last_peered":"2024-09-21T12:40:21.966066+0000","last_clean":"2024-09-21T12:40:21.966066+0000","last_became_active":"2024-09-21T12:32:38.424402+0000","last_became_peered":"2024-09-21T12:32:38.424402+0000","last_unstale":"2024-09-21T12:40:21.966066+0000","last_undegraded":"2024-09-21T12:40:21.966066+0000","last_fullsized":"2024-09-21T12:40:21.966066+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:21.966001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:40:21.966001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:50.192762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1013,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.528573+0000","last_change":"2024-09-21T12:39:42.489044+0000","last_active":"2024-09-21T12:40:15.528573+0000","last_peered":"2024-09-21T12:40:15.528573+0000","last_clean":"2024-09-21T12:40:15.528573+0000","last_became_active":"2024-09-21T12:36:00.144526+0000","last_became_peered":"2024-09-21T12:36:00.144526+0000","last_unstale":"2024-09-21T12:40:15.528573+0000","last_undegraded":"2024-09-21T12:40:15.528573+0000","last_fullsized":"2024-09-21T12:40:15.528573+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:42.488986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:42.488986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:55.906511+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'79","reported_seq":397,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:26.728419+0000","last_change":"2024-09-21T12:40:26.728419+0000","last_active":"2024-09-21T12:40:26.728419+0000","last_peered":"2024-09-21T12:40:26.728419+0000","last_clean":"2024-09-21T12:40:26.728419+0000","last_became_active":"2024-09-21T12:36:01.131622+0000","last_became_peered":"2024-09-21T12:36:01.131622+0000","last_unstale":"2024-09-21T12:40:26.728419+0000","last_undegraded":"2024-09-21T12:40:26.728419+0000","last_fullsized":"2024-09-21T12:40:26.728419+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:40:26.728360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:40:26.728360+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:33.440314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"518'21","reported_seq":486,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.097162+0000","last_change":"2024-09-21T12:39:32.975957+0000","last_active":"2024-09-21T12:40:23.097162+0000","last_peered":"2024-09-21T12:40:23.097162+0000","last_clean":"2024-09-21T12:40:23.097162+0000","last_became_active":"2024-09-21T12:31:39.057316+0000","last_became_peered":"2024-09-21T12:31:39.057316+0000","last_unstale":"2024-09-21T12:40:23.097162+0000","last_undegraded":"2024-09-21T12:40:23.097162+0000","last_fullsized":"2024-09-21T12:40:23.097162+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'17","last_scrub_stamp":"2024-09-21T12:39:32.975879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:32.975879+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:54.712182+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1013,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.223964+0000","last_change":"2024-09-21T12:40:02.507166+0000","last_active":"2024-09-21T12:40:15.223964+0000","last_peered":"2024-09-21T12:40:15.223964+0000","last_clean":"2024-09-21T12:40:15.223964+0000","last_became_active":"2024-09-21T12:38:07.154643+0000","last_became_peered":"2024-09-21T12:38:07.154643+0000","last_unstale":"2024-09-21T12:40:15.223964+0000","last_undegraded":"2024-09-21T12:40:15.223964+0000","last_fullsized":"2024-09-21T12:40:15.223964+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:02.507122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:40:02.507122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:20.157120+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":730,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.626461+0000","last_change":"2024-09-21T12:40:10.205849+0000","last_active":"2024-09-21T12:40:15.626461+0000","last_peered":"2024-09-21T12:40:15.626461+0000","last_clean":"2024-09-21T12:40:15.626461+0000","last_became_active":"2024-09-21T12:40:10.205183+0000","last_became_peered":"2024-09-21T12:40:10.205183+0000","last_unstale":"2024-09-21T12:40:15.626461+0000","last_undegraded":"2024-09-21T12:40:15.626461+0000","last_fullsized":"2024-09-21T12:40:15.626461+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:40:06.454218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:40:06.454218+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:22.969232+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0019042449999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":721,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.626672+0000","last_change":"2024-09-21T12:40:10.207972+0000","last_active":"2024-09-21T12:40:15.626672+0000","last_peered":"2024-09-21T12:40:15.626672+0000","last_clean":"2024-09-21T12:40:15.626672+0000","last_became_active":"2024-09-21T12:40:10.207682+0000","last_became_peered":"2024-09-21T12:40:10.207682+0000","last_unstale":"2024-09-21T12:40:15.626672+0000","last_undegraded":"2024-09-21T12:40:15.626672+0000","last_fullsized":"2024-09-21T12:40:15.626672+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:31.255364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:31.255364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:47.361606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":88,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.626563+0000","last_change":"2024-09-21T12:40:10.432876+0000","last_active":"2024-09-21T12:40:15.626563+0000","last_peered":"2024-09-21T12:40:15.626563+0000","last_clean":"2024-09-21T12:40:15.626563+0000","last_became_active":"2024-09-21T12:40:10.432646+0000","last_became_peered":"2024-09-21T12:40:10.432646+0000","last_unstale":"2024-09-21T12:40:15.626563+0000","last_undegraded":"2024-09-21T12:40:15.626563+0000","last_fullsized":"2024-09-21T12:40:15.626563+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:56.707155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:56.707155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:21.936398+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"76'79","reported_seq":934,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.224646+0000","last_change":"2024-09-21T12:39:57.434316+0000","last_active":"2024-09-21T12:40:15.224646+0000","last_peered":"2024-09-21T12:40:15.224646+0000","last_clean":"2024-09-21T12:40:15.224646+0000","last_became_active":"2024-09-21T12:32:38.424710+0000","last_became_peered":"2024-09-21T12:32:38.424710+0000","last_unstale":"2024-09-21T12:40:15.224646+0000","last_undegraded":"2024-09-21T12:40:15.224646+0000","last_fullsized":"2024-09-21T12:40:15.224646+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:57.434269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:57.434269+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:09.700806+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"518'36","reported_seq":1040,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.107585+0000","last_change":"2024-09-21T12:40:04.868633+0000","last_active":"2024-09-21T12:40:23.107585+0000","last_peered":"2024-09-21T12:40:23.107585+0000","last_clean":"2024-09-21T12:40:23.107585+0000","last_became_active":"2024-09-21T12:38:07.166839+0000","last_became_peered":"2024-09-21T12:38:07.166839+0000","last_unstale":"2024-09-21T12:40:23.107585+0000","last_undegraded":"2024-09-21T12:40:23.107585+0000","last_fullsized":"2024-09-21T12:40:23.107585+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'24","last_scrub_stamp":"2024-09-21T12:40:04.868559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:40:04.868559+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:06.609155+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1029,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.225646+0000","last_change":"2024-09-21T12:40:09.884209+0000","last_active":"2024-09-21T12:40:15.225646+0000","last_peered":"2024-09-21T12:40:15.225646+0000","last_clean":"2024-09-21T12:40:15.225646+0000","last_became_active":"2024-09-21T12:32:38.411004+0000","last_became_peered":"2024-09-21T12:32:38.411004+0000","last_unstale":"2024-09-21T12:40:15.225646+0000","last_undegraded":"2024-09-21T12:40:15.225646+0000","last_fullsized":"2024-09-21T12:40:15.225646+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:09.884153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:40:09.884153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:12.598597+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":510,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:30.639912+0000","last_change":"2024-09-21T12:40:05.623574+0000","last_active":"2024-09-21T12:40:30.639912+0000","last_peered":"2024-09-21T12:40:30.639912+0000","last_clean":"2024-09-21T12:40:30.639912+0000","last_became_active":"2024-09-21T12:36:00.144053+0000","last_became_peered":"2024-09-21T12:36:00.144053+0000","last_unstale":"2024-09-21T12:40:30.639912+0000","last_undegraded":"2024-09-21T12:40:30.639912+0000","last_fullsized":"2024-09-21T12:40:30.639912+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-21T12:40:05.623495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:40:05.623495+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:34.144320+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":278,"num_read_kb":273,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":415,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.703266+0000","last_change":"2024-09-21T12:38:54.293655+0000","last_active":"2024-09-21T12:40:15.703266+0000","last_peered":"2024-09-21T12:40:15.703266+0000","last_clean":"2024-09-21T12:40:15.703266+0000","last_became_active":"2024-09-21T12:32:02.030240+0000","last_became_peered":"2024-09-21T12:32:02.030240+0000","last_unstale":"2024-09-21T12:40:15.703266+0000","last_undegraded":"2024-09-21T12:40:15.703266+0000","last_fullsized":"2024-09-21T12:40:15.703266+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:54.293611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:54.293611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:20.703408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":468,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.224530+0000","last_change":"2024-09-21T12:39:27.990436+0000","last_active":"2024-09-21T12:40:15.224530+0000","last_peered":"2024-09-21T12:40:15.224530+0000","last_clean":"2024-09-21T12:40:15.224530+0000","last_became_active":"2024-09-21T12:31:39.060160+0000","last_became_peered":"2024-09-21T12:31:39.060160+0000","last_unstale":"2024-09-21T12:40:15.224530+0000","last_undegraded":"2024-09-21T12:40:15.224530+0000","last_fullsized":"2024-09-21T12:40:15.224530+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:27.990369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:27.990369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:57.401228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"518'48","reported_seq":197,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:25.602758+0000","last_change":"2024-09-21T12:40:08.959291+0000","last_active":"2024-09-21T12:40:25.602758+0000","last_peered":"2024-09-21T12:40:25.602758+0000","last_clean":"2024-09-21T12:40:25.602758+0000","last_became_active":"2024-09-21T12:40:01.426627+0000","last_became_peered":"2024-09-21T12:40:01.426627+0000","last_unstale":"2024-09-21T12:40:25.602758+0000","last_undegraded":"2024-09-21T12:40:25.602758+0000","last_fullsized":"2024-09-21T12:40:25.602758+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"510'36","last_scrub_stamp":"2024-09-21T12:40:08.959223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:40:08.959223+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:09.101029+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":989,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.527983+0000","last_change":"2024-09-21T12:39:46.503605+0000","last_active":"2024-09-21T12:40:15.527983+0000","last_peered":"2024-09-21T12:40:15.527983+0000","last_clean":"2024-09-21T12:40:15.527983+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:40:15.527983+0000","last_undegraded":"2024-09-21T12:40:15.527983+0000","last_fullsized":"2024-09-21T12:40:15.527983+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:39:46.503556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:46.503556+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:47.330227+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":242,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:30.741404+0000","last_change":"2024-09-21T12:40:30.741404+0000","last_active":"2024-09-21T12:40:30.741404+0000","last_peered":"2024-09-21T12:40:30.741404+0000","last_clean":"2024-09-21T12:40:30.741404+0000","last_became_active":"2024-09-21T12:36:00.134878+0000","last_became_peered":"2024-09-21T12:36:00.134878+0000","last_unstale":"2024-09-21T12:40:30.741404+0000","last_undegraded":"2024-09-21T12:40:30.741404+0000","last_fullsized":"2024-09-21T12:40:30.741404+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:40:30.741347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:40:30.741347+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:32.440076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":400,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.529342+0000","last_change":"2024-09-21T12:39:35.396535+0000","last_active":"2024-09-21T12:40:15.529342+0000","last_peered":"2024-09-21T12:40:15.529342+0000","last_clean":"2024-09-21T12:40:15.529342+0000","last_became_active":"2024-09-21T12:38:08.164554+0000","last_became_peered":"2024-09-21T12:38:08.164554+0000","last_unstale":"2024-09-21T12:40:15.529342+0000","last_undegraded":"2024-09-21T12:40:15.529342+0000","last_fullsized":"2024-09-21T12:40:15.529342+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:35.396482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:35.396482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:01.813844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":640,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.703494+0000","last_change":"2024-09-21T12:39:36.225811+0000","last_active":"2024-09-21T12:40:15.703494+0000","last_peered":"2024-09-21T12:40:15.703494+0000","last_clean":"2024-09-21T12:40:15.703494+0000","last_became_active":"2024-09-21T12:37:20.485126+0000","last_became_peered":"2024-09-21T12:37:20.485126+0000","last_unstale":"2024-09-21T12:40:15.703494+0000","last_undegraded":"2024-09-21T12:40:15.703494+0000","last_fullsized":"2024-09-21T12:40:15.703494+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:36.225755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:36.225755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:43.496265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"76'79","reported_seq":931,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:25.526647+0000","last_change":"2024-09-21T12:40:25.526647+0000","last_active":"2024-09-21T12:40:25.526647+0000","last_peered":"2024-09-21T12:40:25.526647+0000","last_clean":"2024-09-21T12:40:25.526647+0000","last_became_active":"2024-09-21T12:27:14.954351+0000","last_became_peered":"2024-09-21T12:27:14.954351+0000","last_unstale":"2024-09-21T12:40:25.526647+0000","last_undegraded":"2024-09-21T12:40:25.526647+0000","last_fullsized":"2024-09-21T12:40:25.526647+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:40:25.526597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:40:25.526597+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:50.760074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"518'27","reported_seq":398,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:28.472915+0000","last_change":"2024-09-21T12:40:28.472915+0000","last_active":"2024-09-21T12:40:28.472915+0000","last_peered":"2024-09-21T12:40:28.472915+0000","last_clean":"2024-09-21T12:40:28.472915+0000","last_became_active":"2024-09-21T12:32:38.419714+0000","last_became_peered":"2024-09-21T12:32:38.419714+0000","last_unstale":"2024-09-21T12:40:28.472915+0000","last_undegraded":"2024-09-21T12:40:28.472915+0000","last_fullsized":"2024-09-21T12:40:28.472915+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"518'27","last_scrub_stamp":"2024-09-21T12:40:28.472846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:40:28.472846+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:42.813306+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":388,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.221832+0000","last_change":"2024-09-21T12:38:08.148418+0000","last_active":"2024-09-21T12:40:15.221832+0000","last_peered":"2024-09-21T12:40:15.221832+0000","last_clean":"2024-09-21T12:40:15.221832+0000","last_became_active":"2024-09-21T12:38:08.148109+0000","last_became_peered":"2024-09-21T12:38:08.148109+0000","last_unstale":"2024-09-21T12:40:15.221832+0000","last_undegraded":"2024-09-21T12:40:15.221832+0000","last_fullsized":"2024-09-21T12:40:15.221832+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:40.708469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:36:40.708469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:15.280609+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":519,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.703603+0000","last_change":"2024-09-21T12:39:51.277137+0000","last_active":"2024-09-21T12:40:15.703603+0000","last_peered":"2024-09-21T12:40:15.703603+0000","last_clean":"2024-09-21T12:40:15.703603+0000","last_became_active":"2024-09-21T12:37:20.467986+0000","last_became_peered":"2024-09-21T12:37:20.467986+0000","last_unstale":"2024-09-21T12:40:15.703603+0000","last_undegraded":"2024-09-21T12:40:15.703603+0000","last_fullsized":"2024-09-21T12:40:15.703603+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:51.277082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:51.277082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:20.277970+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":962,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.528271+0000","last_change":"2024-09-21T12:40:14.522626+0000","last_active":"2024-09-21T12:40:15.528271+0000","last_peered":"2024-09-21T12:40:15.528271+0000","last_clean":"2024-09-21T12:40:15.528271+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:40:15.528271+0000","last_undegraded":"2024-09-21T12:40:15.528271+0000","last_fullsized":"2024-09-21T12:40:15.528271+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:14.522582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:40:14.522582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:26.810882+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"76'79","reported_seq":322,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.701150+0000","last_change":"2024-09-21T12:40:00.283573+0000","last_active":"2024-09-21T12:40:15.701150+0000","last_peered":"2024-09-21T12:40:15.701150+0000","last_clean":"2024-09-21T12:40:15.701150+0000","last_became_active":"2024-09-21T12:39:57.394768+0000","last_became_peered":"2024-09-21T12:39:57.394768+0000","last_unstale":"2024-09-21T12:40:15.701150+0000","last_undegraded":"2024-09-21T12:40:15.701150+0000","last_fullsized":"2024-09-21T12:40:15.701150+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:40:00.283488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:40:00.283488+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:13.411294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"518'45","reported_seq":416,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.157284+0000","last_change":"2024-09-21T12:38:53.268874+0000","last_active":"2024-09-21T12:40:23.157284+0000","last_peered":"2024-09-21T12:40:23.157284+0000","last_clean":"2024-09-21T12:40:23.157284+0000","last_became_active":"2024-09-21T12:38:07.163226+0000","last_became_peered":"2024-09-21T12:38:07.163226+0000","last_unstale":"2024-09-21T12:40:23.157284+0000","last_undegraded":"2024-09-21T12:40:23.157284+0000","last_fullsized":"2024-09-21T12:40:23.157284+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'31","last_scrub_stamp":"2024-09-21T12:38:53.268744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:53.268744+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":486,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.702782+0000","last_change":"2024-09-21T12:40:01.422904+0000","last_active":"2024-09-21T12:40:15.702782+0000","last_peered":"2024-09-21T12:40:15.702782+0000","last_clean":"2024-09-21T12:40:15.702782+0000","last_became_active":"2024-09-21T12:40:01.422156+0000","last_became_peered":"2024-09-21T12:40:01.422156+0000","last_unstale":"2024-09-21T12:40:15.702782+0000","last_undegraded":"2024-09-21T12:40:15.702782+0000","last_fullsized":"2024-09-21T12:40:15.702782+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:31.453523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:31.453523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:41.618032+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":996,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.224619+0000","last_change":"2024-09-21T12:40:06.448316+0000","last_active":"2024-09-21T12:40:15.224619+0000","last_peered":"2024-09-21T12:40:15.224619+0000","last_clean":"2024-09-21T12:40:15.224619+0000","last_became_active":"2024-09-21T12:32:38.425177+0000","last_became_peered":"2024-09-21T12:32:38.425177+0000","last_unstale":"2024-09-21T12:40:15.224619+0000","last_undegraded":"2024-09-21T12:40:15.224619+0000","last_fullsized":"2024-09-21T12:40:15.224619+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:06.448265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:40:06.448265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:13.299279+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":794,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.626981+0000","last_change":"2024-09-21T12:40:10.206824+0000","last_active":"2024-09-21T12:40:15.626981+0000","last_peered":"2024-09-21T12:40:15.626981+0000","last_clean":"2024-09-21T12:40:15.626981+0000","last_became_active":"2024-09-21T12:40:10.206622+0000","last_became_peered":"2024-09-21T12:40:10.206622+0000","last_unstale":"2024-09-21T12:40:15.626981+0000","last_undegraded":"2024-09-21T12:40:15.626981+0000","last_fullsized":"2024-09-21T12:40:15.626981+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:02.515408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:40:02.515408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:24.884220+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"76'79","reported_seq":1052,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:28.660249+0000","last_change":"2024-09-21T12:40:28.660249+0000","last_active":"2024-09-21T12:40:28.660249+0000","last_peered":"2024-09-21T12:40:28.660249+0000","last_clean":"2024-09-21T12:40:28.660249+0000","last_became_active":"2024-09-21T12:38:08.156755+0000","last_became_peered":"2024-09-21T12:38:08.156755+0000","last_unstale":"2024-09-21T12:40:28.660249+0000","last_undegraded":"2024-09-21T12:40:28.660249+0000","last_fullsized":"2024-09-21T12:40:28.660249+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:40:28.660176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:40:28.660176+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:35.645247+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"518'30","reported_seq":396,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.117843+0000","last_change":"2024-09-21T12:38:46.208578+0000","last_active":"2024-09-21T12:40:23.117843+0000","last_peered":"2024-09-21T12:40:23.117843+0000","last_clean":"2024-09-21T12:40:23.117843+0000","last_became_active":"2024-09-21T12:38:07.162180+0000","last_became_peered":"2024-09-21T12:38:07.162180+0000","last_unstale":"2024-09-21T12:40:23.117843+0000","last_undegraded":"2024-09-21T12:40:23.117843+0000","last_fullsized":"2024-09-21T12:40:23.117843+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:38:46.208493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:46.208493+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1003,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.528190+0000","last_change":"2024-09-21T12:40:04.488713+0000","last_active":"2024-09-21T12:40:15.528190+0000","last_peered":"2024-09-21T12:40:15.528190+0000","last_clean":"2024-09-21T12:40:15.528190+0000","last_became_active":"2024-09-21T12:32:38.403997+0000","last_became_peered":"2024-09-21T12:32:38.403997+0000","last_unstale":"2024-09-21T12:40:15.528190+0000","last_undegraded":"2024-09-21T12:40:15.528190+0000","last_fullsized":"2024-09-21T12:40:15.528190+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:04.488665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:40:04.488665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:27.871575+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"516'17","reported_seq":909,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:20.602532+0000","last_change":"2024-09-21T12:40:11.697861+0000","last_active":"2024-09-21T12:40:20.602532+0000","last_peered":"2024-09-21T12:40:20.602532+0000","last_clean":"2024-09-21T12:40:20.602532+0000","last_became_active":"2024-09-21T12:40:10.207745+0000","last_became_peered":"2024-09-21T12:40:10.207745+0000","last_unstale":"2024-09-21T12:40:20.602532+0000","last_undegraded":"2024-09-21T12:40:20.602532+0000","last_fullsized":"2024-09-21T12:40:20.602532+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'17","last_scrub_stamp":"2024-09-21T12:40:11.697810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:40:11.697810+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:23.355909+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":377,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:21.558597+0000","last_change":"2024-09-21T12:40:21.558597+0000","last_active":"2024-09-21T12:40:21.558597+0000","last_peered":"2024-09-21T12:40:21.558597+0000","last_clean":"2024-09-21T12:40:21.558597+0000","last_became_active":"2024-09-21T12:38:07.163242+0000","last_became_peered":"2024-09-21T12:38:07.163242+0000","last_unstale":"2024-09-21T12:40:21.558597+0000","last_undegraded":"2024-09-21T12:40:21.558597+0000","last_fullsized":"2024-09-21T12:40:21.558597+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:21.558535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:40:21.558535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:49.676545+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.e","version":"76'79","reported_seq":386,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.222842+0000","last_change":"2024-09-21T12:39:27.391527+0000","last_active":"2024-09-21T12:40:15.222842+0000","last_peered":"2024-09-21T12:40:15.222842+0000","last_clean":"2024-09-21T12:40:15.222842+0000","last_became_active":"2024-09-21T12:36:01.159481+0000","last_became_peered":"2024-09-21T12:36:01.159481+0000","last_unstale":"2024-09-21T12:40:15.222842+0000","last_undegraded":"2024-09-21T12:40:15.222842+0000","last_fullsized":"2024-09-21T12:40:15.222842+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:27.391477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:27.391477+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:55.811814+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"518'21","reported_seq":1100,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.141621+0000","last_change":"2024-09-21T12:39:27.747950+0000","last_active":"2024-09-21T12:40:23.141621+0000","last_peered":"2024-09-21T12:40:23.141621+0000","last_clean":"2024-09-21T12:40:23.141621+0000","last_became_active":"2024-09-21T12:32:38.424376+0000","last_became_peered":"2024-09-21T12:32:38.424376+0000","last_unstale":"2024-09-21T12:40:23.141621+0000","last_undegraded":"2024-09-21T12:40:23.141621+0000","last_fullsized":"2024-09-21T12:40:23.141621+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:39:27.747889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:27.747889+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:54.802430+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":506,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.703006+0000","last_change":"2024-09-21T12:38:51.327197+0000","last_active":"2024-09-21T12:40:15.703006+0000","last_peered":"2024-09-21T12:40:15.703006+0000","last_clean":"2024-09-21T12:40:15.703006+0000","last_became_active":"2024-09-21T12:37:20.484773+0000","last_became_peered":"2024-09-21T12:37:20.484773+0000","last_unstale":"2024-09-21T12:40:15.703006+0000","last_undegraded":"2024-09-21T12:40:15.703006+0000","last_fullsized":"2024-09-21T12:40:15.703006+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:51.327127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:51.327127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":822,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.627670+0000","last_change":"2024-09-21T12:40:15.627670+0000","last_active":"2024-09-21T12:40:15.627670+0000","last_peered":"2024-09-21T12:40:15.627670+0000","last_clean":"2024-09-21T12:40:15.627670+0000","last_became_active":"2024-09-21T12:40:10.206786+0000","last_became_peered":"2024-09-21T12:40:10.206786+0000","last_unstale":"2024-09-21T12:40:15.627670+0000","last_undegraded":"2024-09-21T12:40:15.627670+0000","last_fullsized":"2024-09-21T12:40:15.627670+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:15.627628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:40:15.627628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:41.093031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":989,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.225599+0000","last_change":"2024-09-21T12:39:34.662753+0000","last_active":"2024-09-21T12:40:15.225599+0000","last_peered":"2024-09-21T12:40:15.225599+0000","last_clean":"2024-09-21T12:40:15.225599+0000","last_became_active":"2024-09-21T12:38:08.164626+0000","last_became_peered":"2024-09-21T12:38:08.164626+0000","last_unstale":"2024-09-21T12:40:15.225599+0000","last_undegraded":"2024-09-21T12:40:15.225599+0000","last_fullsized":"2024-09-21T12:40:15.225599+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:34.662706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:34.662706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:39.901409+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"76'79","reported_seq":474,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:21.898417+0000","last_change":"2024-09-21T12:40:21.898417+0000","last_active":"2024-09-21T12:40:21.898417+0000","last_peered":"2024-09-21T12:40:21.898417+0000","last_clean":"2024-09-21T12:40:21.898417+0000","last_became_active":"2024-09-21T12:40:01.425715+0000","last_became_peered":"2024-09-21T12:40:01.425715+0000","last_unstale":"2024-09-21T12:40:21.898417+0000","last_undegraded":"2024-09-21T12:40:21.898417+0000","last_fullsized":"2024-09-21T12:40:21.898417+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:40:21.898359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:40:21.898359+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:30.363313+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"518'51","reported_seq":1183,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.177322+0000","last_change":"2024-09-21T12:39:48.101801+0000","last_active":"2024-09-21T12:40:23.177322+0000","last_peered":"2024-09-21T12:40:23.177322+0000","last_clean":"2024-09-21T12:40:23.177322+0000","last_became_active":"2024-09-21T12:36:00.148392+0000","last_became_peered":"2024-09-21T12:36:00.148392+0000","last_unstale":"2024-09-21T12:40:23.177322+0000","last_undegraded":"2024-09-21T12:40:23.177322+0000","last_fullsized":"2024-09-21T12:40:23.177322+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'35","last_scrub_stamp":"2024-09-21T12:39:48.101688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:48.101688+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:13.498714+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":234,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:19.630007+0000","last_change":"2024-09-21T12:40:19.629930+0000","last_active":"2024-09-21T12:40:19.630007+0000","last_peered":"2024-09-21T12:40:19.630007+0000","last_clean":"2024-09-21T12:40:19.630007+0000","last_became_active":"2024-09-21T12:36:00.145287+0000","last_became_peered":"2024-09-21T12:36:00.145287+0000","last_unstale":"2024-09-21T12:40:19.630007+0000","last_undegraded":"2024-09-21T12:40:19.630007+0000","last_fullsized":"2024-09-21T12:40:19.630007+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:19.629864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:40:19.629864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:48.903131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":752,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.626481+0000","last_change":"2024-09-21T12:40:10.205161+0000","last_active":"2024-09-21T12:40:15.626481+0000","last_peered":"2024-09-21T12:40:15.626481+0000","last_clean":"2024-09-21T12:40:15.626481+0000","last_became_active":"2024-09-21T12:40:10.204864+0000","last_became_peered":"2024-09-21T12:40:10.204864+0000","last_unstale":"2024-09-21T12:40:15.626481+0000","last_undegraded":"2024-09-21T12:40:15.626481+0000","last_fullsized":"2024-09-21T12:40:15.626481+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:06.884082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:40:06.884082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:27.693979+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":966,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.529749+0000","last_change":"2024-09-21T12:39:46.420589+0000","last_active":"2024-09-21T12:40:15.529749+0000","last_peered":"2024-09-21T12:40:15.529749+0000","last_clean":"2024-09-21T12:40:15.529749+0000","last_became_active":"2024-09-21T12:38:08.164339+0000","last_became_peered":"2024-09-21T12:38:08.164339+0000","last_unstale":"2024-09-21T12:40:15.529749+0000","last_undegraded":"2024-09-21T12:40:15.529749+0000","last_fullsized":"2024-09-21T12:40:15.529749+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:46.420527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:46.420527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:12.973409+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"76'79","reported_seq":929,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.528421+0000","last_change":"2024-09-21T12:39:28.377091+0000","last_active":"2024-09-21T12:40:15.528421+0000","last_peered":"2024-09-21T12:40:15.528421+0000","last_clean":"2024-09-21T12:40:15.528421+0000","last_became_active":"2024-09-21T12:27:14.951375+0000","last_became_peered":"2024-09-21T12:27:14.951375+0000","last_unstale":"2024-09-21T12:40:15.528421+0000","last_undegraded":"2024-09-21T12:40:15.528421+0000","last_fullsized":"2024-09-21T12:40:15.528421+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:28.377033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:28.377033+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:43.162251+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"518'24","reported_seq":395,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:22.995204+0000","last_change":"2024-09-21T12:39:22.328834+0000","last_active":"2024-09-21T12:40:22.995204+0000","last_peered":"2024-09-21T12:40:22.995204+0000","last_clean":"2024-09-21T12:40:22.995204+0000","last_became_active":"2024-09-21T12:32:38.421609+0000","last_became_peered":"2024-09-21T12:32:38.421609+0000","last_unstale":"2024-09-21T12:40:22.995204+0000","last_undegraded":"2024-09-21T12:40:22.995204+0000","last_fullsized":"2024-09-21T12:40:22.995204+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T12:39:22.328772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:22.328772+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:27.373604+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":234,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.708307+0000","last_change":"2024-09-21T12:40:15.708258+0000","last_active":"2024-09-21T12:40:15.708307+0000","last_peered":"2024-09-21T12:40:15.708307+0000","last_clean":"2024-09-21T12:40:15.708307+0000","last_became_active":"2024-09-21T12:36:00.150832+0000","last_became_peered":"2024-09-21T12:36:00.150832+0000","last_unstale":"2024-09-21T12:40:15.708307+0000","last_undegraded":"2024-09-21T12:40:15.708307+0000","last_fullsized":"2024-09-21T12:40:15.708307+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:15.708200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:40:15.708200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:18.215154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":721,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:19.627051+0000","last_change":"2024-09-21T12:40:19.627051+0000","last_active":"2024-09-21T12:40:19.627051+0000","last_peered":"2024-09-21T12:40:19.627051+0000","last_clean":"2024-09-21T12:40:19.627051+0000","last_became_active":"2024-09-21T12:40:10.202413+0000","last_became_peered":"2024-09-21T12:40:10.202413+0000","last_unstale":"2024-09-21T12:40:19.627051+0000","last_undegraded":"2024-09-21T12:40:19.627051+0000","last_fullsized":"2024-09-21T12:40:19.627051+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:19.626989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:40:19.626989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:31.397975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":998,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.527949+0000","last_change":"2024-09-21T12:39:52.544099+0000","last_active":"2024-09-21T12:40:15.527949+0000","last_peered":"2024-09-21T12:40:15.527949+0000","last_clean":"2024-09-21T12:40:15.527949+0000","last_became_active":"2024-09-21T12:36:00.139300+0000","last_became_peered":"2024-09-21T12:36:00.139300+0000","last_unstale":"2024-09-21T12:40:15.527949+0000","last_undegraded":"2024-09-21T12:40:15.527949+0000","last_fullsized":"2024-09-21T12:40:15.527949+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:52.544055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:52.544055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:21.056635+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"76'79","reported_seq":405,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.530501+0000","last_change":"2024-09-21T12:40:11.657863+0000","last_active":"2024-09-21T12:40:15.530501+0000","last_peered":"2024-09-21T12:40:15.530501+0000","last_clean":"2024-09-21T12:40:15.530501+0000","last_became_active":"2024-09-21T12:36:01.141807+0000","last_became_peered":"2024-09-21T12:36:01.141807+0000","last_unstale":"2024-09-21T12:40:15.530501+0000","last_undegraded":"2024-09-21T12:40:15.530501+0000","last_fullsized":"2024-09-21T12:40:15.530501+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:40:11.657791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:40:11.657791+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:36.142388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"518'39","reported_seq":534,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.115627+0000","last_change":"2024-09-21T12:39:50.287772+0000","last_active":"2024-09-21T12:40:23.115627+0000","last_peered":"2024-09-21T12:40:23.115627+0000","last_clean":"2024-09-21T12:40:23.115627+0000","last_became_active":"2024-09-21T12:36:00.137597+0000","last_became_peered":"2024-09-21T12:36:00.137597+0000","last_unstale":"2024-09-21T12:40:23.115627+0000","last_undegraded":"2024-09-21T12:40:23.115627+0000","last_fullsized":"2024-09-21T12:40:23.115627+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:39:50.287686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:50.287686+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:19.547933+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":240,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.530840+0000","last_change":"2024-09-21T12:39:29.483527+0000","last_active":"2024-09-21T12:40:15.530840+0000","last_peered":"2024-09-21T12:40:15.530840+0000","last_clean":"2024-09-21T12:40:15.530840+0000","last_became_active":"2024-09-21T12:36:00.148899+0000","last_became_peered":"2024-09-21T12:36:00.148899+0000","last_unstale":"2024-09-21T12:40:15.530840+0000","last_undegraded":"2024-09-21T12:40:15.530840+0000","last_fullsized":"2024-09-21T12:40:15.530840+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-21T12:39:29.483457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:29.483457+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:55.417541+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"415'16","reported_seq":293,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:25.602654+0000","last_change":"2024-09-21T12:39:56.532997+0000","last_active":"2024-09-21T12:40:25.602654+0000","last_peered":"2024-09-21T12:40:25.602654+0000","last_clean":"2024-09-21T12:40:25.602654+0000","last_became_active":"2024-09-21T12:36:00.143091+0000","last_became_peered":"2024-09-21T12:36:00.143091+0000","last_unstale":"2024-09-21T12:40:25.602654+0000","last_undegraded":"2024-09-21T12:40:25.602654+0000","last_fullsized":"2024-09-21T12:40:25.602654+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'16","last_scrub_stamp":"2024-09-21T12:39:56.532873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:56.532873+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:04.686343+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":478,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.702130+0000","last_change":"2024-09-21T12:40:09.147282+0000","last_active":"2024-09-21T12:40:15.702130+0000","last_peered":"2024-09-21T12:40:15.702130+0000","last_clean":"2024-09-21T12:40:15.702130+0000","last_became_active":"2024-09-21T12:37:20.468350+0000","last_became_peered":"2024-09-21T12:37:20.468350+0000","last_unstale":"2024-09-21T12:40:15.702130+0000","last_undegraded":"2024-09-21T12:40:15.702130+0000","last_fullsized":"2024-09-21T12:40:15.702130+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:09.147222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:40:09.147222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:25.515342+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"76'79","reported_seq":705,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.701894+0000","last_change":"2024-09-21T12:39:16.243982+0000","last_active":"2024-09-21T12:40:15.701894+0000","last_peered":"2024-09-21T12:40:15.701894+0000","last_clean":"2024-09-21T12:40:15.701894+0000","last_became_active":"2024-09-21T12:31:39.060422+0000","last_became_peered":"2024-09-21T12:31:39.060422+0000","last_unstale":"2024-09-21T12:40:15.701894+0000","last_undegraded":"2024-09-21T12:40:15.701894+0000","last_fullsized":"2024-09-21T12:40:15.701894+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:16.243927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:16.243927+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:40.547837+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1029,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.701714+0000","last_change":"2024-09-21T12:39:19.481714+0000","last_active":"2024-09-21T12:40:15.701714+0000","last_peered":"2024-09-21T12:40:15.701714+0000","last_clean":"2024-09-21T12:40:15.701714+0000","last_became_active":"2024-09-21T12:32:02.026693+0000","last_became_peered":"2024-09-21T12:32:02.026693+0000","last_unstale":"2024-09-21T12:40:15.701714+0000","last_undegraded":"2024-09-21T12:40:15.701714+0000","last_fullsized":"2024-09-21T12:40:15.701714+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:19.481652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:19.481652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:34.846777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"518'18","reported_seq":494,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.130981+0000","last_change":"2024-09-21T12:39:44.517980+0000","last_active":"2024-09-21T12:40:23.130981+0000","last_peered":"2024-09-21T12:40:23.130981+0000","last_clean":"2024-09-21T12:40:23.130981+0000","last_became_active":"2024-09-21T12:36:01.138626+0000","last_became_peered":"2024-09-21T12:36:01.138626+0000","last_unstale":"2024-09-21T12:40:23.130981+0000","last_undegraded":"2024-09-21T12:40:23.130981+0000","last_fullsized":"2024-09-21T12:40:23.130981+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T12:39:44.517927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:44.517927+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:14.117178+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":498,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.703205+0000","last_change":"2024-09-21T12:40:02.297488+0000","last_active":"2024-09-21T12:40:15.703205+0000","last_peered":"2024-09-21T12:40:15.703205+0000","last_clean":"2024-09-21T12:40:15.703205+0000","last_became_active":"2024-09-21T12:37:20.485806+0000","last_became_peered":"2024-09-21T12:37:20.485806+0000","last_unstale":"2024-09-21T12:40:15.703205+0000","last_undegraded":"2024-09-21T12:40:15.703205+0000","last_fullsized":"2024-09-21T12:40:15.703205+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:02.297429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:40:02.297429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:04.429618+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":233,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.530187+0000","last_change":"2024-09-21T12:39:53.520465+0000","last_active":"2024-09-21T12:40:15.530187+0000","last_peered":"2024-09-21T12:40:15.530187+0000","last_clean":"2024-09-21T12:40:15.530187+0000","last_became_active":"2024-09-21T12:36:00.149086+0000","last_became_peered":"2024-09-21T12:36:00.149086+0000","last_unstale":"2024-09-21T12:40:15.530187+0000","last_undegraded":"2024-09-21T12:40:15.530187+0000","last_fullsized":"2024-09-21T12:40:15.530187+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:53.520419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:53.520419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:23.000283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'79","reported_seq":961,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.224735+0000","last_change":"2024-09-21T12:39:29.959881+0000","last_active":"2024-09-21T12:40:15.224735+0000","last_peered":"2024-09-21T12:40:15.224735+0000","last_clean":"2024-09-21T12:40:15.224735+0000","last_became_active":"2024-09-21T12:32:38.425495+0000","last_became_peered":"2024-09-21T12:32:38.425495+0000","last_unstale":"2024-09-21T12:40:15.224735+0000","last_undegraded":"2024-09-21T12:40:15.224735+0000","last_fullsized":"2024-09-21T12:40:15.224735+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:29.959790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:29.959790+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:37.092161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"518'21","reported_seq":1063,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.008343+0000","last_change":"2024-09-21T12:39:59.759617+0000","last_active":"2024-09-21T12:40:23.008343+0000","last_peered":"2024-09-21T12:40:23.008343+0000","last_clean":"2024-09-21T12:40:23.008343+0000","last_became_active":"2024-09-21T12:36:00.142274+0000","last_became_peered":"2024-09-21T12:36:00.142274+0000","last_unstale":"2024-09-21T12:40:23.008343+0000","last_undegraded":"2024-09-21T12:40:23.008343+0000","last_fullsized":"2024-09-21T12:40:23.008343+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:39:59.759566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:59.759566+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:03.918859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":233,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.529131+0000","last_change":"2024-09-21T12:39:55.510514+0000","last_active":"2024-09-21T12:40:15.529131+0000","last_peered":"2024-09-21T12:40:15.529131+0000","last_clean":"2024-09-21T12:40:15.529131+0000","last_became_active":"2024-09-21T12:36:00.140477+0000","last_became_peered":"2024-09-21T12:36:00.140477+0000","last_unstale":"2024-09-21T12:40:15.529131+0000","last_undegraded":"2024-09-21T12:40:15.529131+0000","last_fullsized":"2024-09-21T12:40:15.529131+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:55.510453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:55.510453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:03.619575+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":980,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.528602+0000","last_change":"2024-09-21T12:40:11.504657+0000","last_active":"2024-09-21T12:40:15.528602+0000","last_peered":"2024-09-21T12:40:15.528602+0000","last_clean":"2024-09-21T12:40:15.528602+0000","last_became_active":"2024-09-21T12:36:00.150975+0000","last_became_peered":"2024-09-21T12:36:00.150975+0000","last_unstale":"2024-09-21T12:40:15.528602+0000","last_undegraded":"2024-09-21T12:40:15.528602+0000","last_fullsized":"2024-09-21T12:40:15.528602+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:11.504601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:40:11.504601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:34.882501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":839,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.626778+0000","last_change":"2024-09-21T12:40:10.207798+0000","last_active":"2024-09-21T12:40:15.626778+0000","last_peered":"2024-09-21T12:40:15.626778+0000","last_clean":"2024-09-21T12:40:15.626778+0000","last_became_active":"2024-09-21T12:40:10.207553+0000","last_became_peered":"2024-09-21T12:40:10.207553+0000","last_unstale":"2024-09-21T12:40:15.626778+0000","last_undegraded":"2024-09-21T12:40:15.626778+0000","last_fullsized":"2024-09-21T12:40:15.626778+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:37.528714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:37.528714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:05.401724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"76'79","reported_seq":203,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.700940+0000","last_change":"2024-09-21T12:39:37.274653+0000","last_active":"2024-09-21T12:40:15.700940+0000","last_peered":"2024-09-21T12:40:15.700940+0000","last_clean":"2024-09-21T12:40:15.700940+0000","last_became_active":"2024-09-21T12:37:20.467540+0000","last_became_peered":"2024-09-21T12:37:20.467540+0000","last_unstale":"2024-09-21T12:40:15.700940+0000","last_undegraded":"2024-09-21T12:40:15.700940+0000","last_fullsized":"2024-09-21T12:40:15.700940+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:37.274594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:37.274594+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:37.391359+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"518'21","reported_seq":547,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:26.156955+0000","last_change":"2024-09-21T12:40:26.156955+0000","last_active":"2024-09-21T12:40:26.156955+0000","last_peered":"2024-09-21T12:40:26.156955+0000","last_clean":"2024-09-21T12:40:26.156955+0000","last_became_active":"2024-09-21T12:38:08.164375+0000","last_became_peered":"2024-09-21T12:38:08.164375+0000","last_unstale":"2024-09-21T12:40:26.156955+0000","last_undegraded":"2024-09-21T12:40:26.156955+0000","last_fullsized":"2024-09-21T12:40:26.156955+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"518'21","last_scrub_stamp":"2024-09-21T12:40:26.156873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:40:26.156873+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:31.934630+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1029,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.225069+0000","last_change":"2024-09-21T12:40:07.879745+0000","last_active":"2024-09-21T12:40:15.225069+0000","last_peered":"2024-09-21T12:40:15.225069+0000","last_clean":"2024-09-21T12:40:15.225069+0000","last_became_active":"2024-09-21T12:36:00.127225+0000","last_became_peered":"2024-09-21T12:36:00.127225+0000","last_unstale":"2024-09-21T12:40:15.225069+0000","last_undegraded":"2024-09-21T12:40:15.225069+0000","last_fullsized":"2024-09-21T12:40:15.225069+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:07.879698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:40:07.879698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:30.244960+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"457'17","reported_seq":582,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:25.602750+0000","last_change":"2024-09-21T12:39:39.398127+0000","last_active":"2024-09-21T12:40:25.602750+0000","last_peered":"2024-09-21T12:40:25.602750+0000","last_clean":"2024-09-21T12:40:25.602750+0000","last_became_active":"2024-09-21T12:38:08.171750+0000","last_became_peered":"2024-09-21T12:38:08.171750+0000","last_unstale":"2024-09-21T12:40:25.602750+0000","last_undegraded":"2024-09-21T12:40:25.602750+0000","last_fullsized":"2024-09-21T12:40:25.602750+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'17","last_scrub_stamp":"2024-09-21T12:39:39.398076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:39.398076+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:50.554325+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":377,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.222984+0000","last_change":"2024-09-21T12:40:00.587195+0000","last_active":"2024-09-21T12:40:15.222984+0000","last_peered":"2024-09-21T12:40:15.222984+0000","last_clean":"2024-09-21T12:40:15.222984+0000","last_became_active":"2024-09-21T12:38:07.163162+0000","last_became_peered":"2024-09-21T12:38:07.163162+0000","last_unstale":"2024-09-21T12:40:15.222984+0000","last_undegraded":"2024-09-21T12:40:15.222984+0000","last_fullsized":"2024-09-21T12:40:15.222984+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:00.587144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:40:00.587144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:25.922211+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"76'79","reported_seq":804,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.225758+0000","last_change":"2024-09-21T12:39:55.721332+0000","last_active":"2024-09-21T12:40:15.225758+0000","last_peered":"2024-09-21T12:40:15.225758+0000","last_clean":"2024-09-21T12:40:15.225758+0000","last_became_active":"2024-09-21T12:32:02.034387+0000","last_became_peered":"2024-09-21T12:32:02.034387+0000","last_unstale":"2024-09-21T12:40:15.225758+0000","last_undegraded":"2024-09-21T12:40:15.225758+0000","last_fullsized":"2024-09-21T12:40:15.225758+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:55.721282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:55.721282+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:00.025864+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"518'30","reported_seq":1074,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.147352+0000","last_change":"2024-09-21T12:39:31.262960+0000","last_active":"2024-09-21T12:40:23.147352+0000","last_peered":"2024-09-21T12:40:23.147352+0000","last_clean":"2024-09-21T12:40:23.147352+0000","last_became_active":"2024-09-21T12:36:00.155033+0000","last_became_peered":"2024-09-21T12:36:00.155033+0000","last_unstale":"2024-09-21T12:40:23.147352+0000","last_undegraded":"2024-09-21T12:40:23.147352+0000","last_fullsized":"2024-09-21T12:40:23.147352+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:39:31.262908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:31.262908+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:37.406495+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1005,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.530024+0000","last_change":"2024-09-21T12:39:43.400476+0000","last_active":"2024-09-21T12:40:15.530024+0000","last_peered":"2024-09-21T12:40:15.530024+0000","last_clean":"2024-09-21T12:40:15.530024+0000","last_became_active":"2024-09-21T12:38:08.164312+0000","last_became_peered":"2024-09-21T12:38:08.164312+0000","last_unstale":"2024-09-21T12:40:15.530024+0000","last_undegraded":"2024-09-21T12:40:15.530024+0000","last_fullsized":"2024-09-21T12:40:15.530024+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:43.400427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:43.400427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:47.864285+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":1228,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:25.602652+0000","last_change":"2024-09-21T12:38:45.275249+0000","last_active":"2024-09-21T12:40:25.602652+0000","last_peered":"2024-09-21T12:40:25.602652+0000","last_clean":"2024-09-21T12:40:25.602652+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:40:25.602652+0000","last_undegraded":"2024-09-21T12:40:25.602652+0000","last_fullsized":"2024-09-21T12:40:25.602652+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:38:45.275110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:45.275110+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":994,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.225793+0000","last_change":"2024-09-21T12:39:52.722587+0000","last_active":"2024-09-21T12:40:15.225793+0000","last_peered":"2024-09-21T12:40:15.225793+0000","last_clean":"2024-09-21T12:40:15.225793+0000","last_became_active":"2024-09-21T12:36:00.148332+0000","last_became_peered":"2024-09-21T12:36:00.148332+0000","last_unstale":"2024-09-21T12:40:15.225793+0000","last_undegraded":"2024-09-21T12:40:15.225793+0000","last_fullsized":"2024-09-21T12:40:15.225793+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:52.722539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:52.722539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:07.068423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"76'79","reported_seq":661,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.226714+0000","last_change":"2024-09-21T12:39:45.196000+0000","last_active":"2024-09-21T12:40:15.226714+0000","last_peered":"2024-09-21T12:40:15.226714+0000","last_clean":"2024-09-21T12:40:15.226714+0000","last_became_active":"2024-09-21T12:27:16.940347+0000","last_became_peered":"2024-09-21T12:27:16.940347+0000","last_unstale":"2024-09-21T12:40:15.226714+0000","last_undegraded":"2024-09-21T12:40:15.226714+0000","last_fullsized":"2024-09-21T12:40:15.226714+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:45.195949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:45.195949+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:07.860174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"518'27","reported_seq":390,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:24.485509+0000","last_change":"2024-09-21T12:40:24.485509+0000","last_active":"2024-09-21T12:40:24.485509+0000","last_peered":"2024-09-21T12:40:24.485509+0000","last_clean":"2024-09-21T12:40:24.485509+0000","last_became_active":"2024-09-21T12:32:38.421008+0000","last_became_peered":"2024-09-21T12:32:38.421008+0000","last_unstale":"2024-09-21T12:40:24.485509+0000","last_undegraded":"2024-09-21T12:40:24.485509+0000","last_fullsized":"2024-09-21T12:40:24.485509+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"518'27","last_scrub_stamp":"2024-09-21T12:40:24.485444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:40:24.485444+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:27.449665+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":386,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.222611+0000","last_change":"2024-09-21T12:39:44.540849+0000","last_active":"2024-09-21T12:40:15.222611+0000","last_peered":"2024-09-21T12:40:15.222611+0000","last_clean":"2024-09-21T12:40:15.222611+0000","last_became_active":"2024-09-21T12:36:01.123399+0000","last_became_peered":"2024-09-21T12:36:01.123399+0000","last_unstale":"2024-09-21T12:40:15.222611+0000","last_undegraded":"2024-09-21T12:40:15.222611+0000","last_fullsized":"2024-09-21T12:40:15.222611+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:44.540771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:44.540771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:03.012331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":993,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.224880+0000","last_change":"2024-09-21T12:39:36.009428+0000","last_active":"2024-09-21T12:40:15.224880+0000","last_peered":"2024-09-21T12:40:15.224880+0000","last_clean":"2024-09-21T12:40:15.224880+0000","last_became_active":"2024-09-21T12:31:50.040754+0000","last_became_peered":"2024-09-21T12:31:50.040754+0000","last_unstale":"2024-09-21T12:40:15.224880+0000","last_undegraded":"2024-09-21T12:40:15.224880+0000","last_fullsized":"2024-09-21T12:40:15.224880+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:36.009386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:36.009386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:39.667415+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":442,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.527610+0000","last_change":"2024-09-21T12:39:37.375691+0000","last_active":"2024-09-21T12:40:15.527610+0000","last_peered":"2024-09-21T12:40:15.527610+0000","last_clean":"2024-09-21T12:40:15.527610+0000","last_became_active":"2024-09-21T12:31:50.036150+0000","last_became_peered":"2024-09-21T12:31:50.036150+0000","last_unstale":"2024-09-21T12:40:15.527610+0000","last_undegraded":"2024-09-21T12:40:15.527610+0000","last_fullsized":"2024-09-21T12:40:15.527610+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:37.375649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:37.375649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:01.292220+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"76'79","reported_seq":686,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:20.119278+0000","last_change":"2024-09-21T12:40:20.119278+0000","last_active":"2024-09-21T12:40:20.119278+0000","last_peered":"2024-09-21T12:40:20.119278+0000","last_clean":"2024-09-21T12:40:20.119278+0000","last_became_active":"2024-09-21T12:27:16.921880+0000","last_became_peered":"2024-09-21T12:27:16.921880+0000","last_unstale":"2024-09-21T12:40:20.119278+0000","last_undegraded":"2024-09-21T12:40:20.119278+0000","last_fullsized":"2024-09-21T12:40:20.119278+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:40:20.119215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:40:20.119215+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:46.961114+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"518'21","reported_seq":489,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.071594+0000","last_change":"2024-09-21T12:39:53.405733+0000","last_active":"2024-09-21T12:40:23.071594+0000","last_peered":"2024-09-21T12:40:23.071594+0000","last_clean":"2024-09-21T12:40:23.071594+0000","last_became_active":"2024-09-21T12:38:07.153467+0000","last_became_peered":"2024-09-21T12:38:07.153467+0000","last_unstale":"2024-09-21T12:40:23.071594+0000","last_undegraded":"2024-09-21T12:40:23.071594+0000","last_fullsized":"2024-09-21T12:40:23.071594+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:39:53.405674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:53.405674+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:06.710449+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":233,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:17.674382+0000","last_change":"2024-09-21T12:40:17.674382+0000","last_active":"2024-09-21T12:40:17.674382+0000","last_peered":"2024-09-21T12:40:17.674382+0000","last_clean":"2024-09-21T12:40:17.674382+0000","last_became_active":"2024-09-21T12:36:00.147063+0000","last_became_peered":"2024-09-21T12:36:00.147063+0000","last_unstale":"2024-09-21T12:40:17.674382+0000","last_undegraded":"2024-09-21T12:40:17.674382+0000","last_fullsized":"2024-09-21T12:40:17.674382+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:17.674314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:40:17.674314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:44.172604+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":974,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.528062+0000","last_change":"2024-09-21T12:39:51.512200+0000","last_active":"2024-09-21T12:40:15.528062+0000","last_peered":"2024-09-21T12:40:15.528062+0000","last_clean":"2024-09-21T12:40:15.528062+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:40:15.528062+0000","last_undegraded":"2024-09-21T12:40:15.528062+0000","last_fullsized":"2024-09-21T12:40:15.528062+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:51.512151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:51.512151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:58.225728+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":958,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:18.933242+0000","last_change":"2024-09-21T12:40:18.933242+0000","last_active":"2024-09-21T12:40:18.933242+0000","last_peered":"2024-09-21T12:40:18.933242+0000","last_clean":"2024-09-21T12:40:18.933242+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:40:18.933242+0000","last_undegraded":"2024-09-21T12:40:18.933242+0000","last_fullsized":"2024-09-21T12:40:18.933242+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:18.933195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:40:18.933195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:30.295146+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"76'79","reported_seq":418,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.626527+0000","last_change":"2024-09-21T12:40:10.207981+0000","last_active":"2024-09-21T12:40:15.626527+0000","last_peered":"2024-09-21T12:40:15.626527+0000","last_clean":"2024-09-21T12:40:15.626527+0000","last_became_active":"2024-09-21T12:40:10.207654+0000","last_became_peered":"2024-09-21T12:40:10.207654+0000","last_unstale":"2024-09-21T12:40:15.626527+0000","last_undegraded":"2024-09-21T12:40:15.626527+0000","last_fullsized":"2024-09-21T12:40:15.626527+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:40:04.593922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:40:04.593922+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:28.407895+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"518'27","reported_seq":1059,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.165269+0000","last_change":"2024-09-21T12:39:35.321714+0000","last_active":"2024-09-21T12:40:23.165269+0000","last_peered":"2024-09-21T12:40:23.165269+0000","last_clean":"2024-09-21T12:40:23.165269+0000","last_became_active":"2024-09-21T12:36:00.139682+0000","last_became_peered":"2024-09-21T12:36:00.139682+0000","last_unstale":"2024-09-21T12:40:23.165269+0000","last_undegraded":"2024-09-21T12:40:23.165269+0000","last_fullsized":"2024-09-21T12:40:23.165269+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:39:35.321662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:35.321662+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:43.243787+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1014,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.703282+0000","last_change":"2024-09-21T12:39:57.149886+0000","last_active":"2024-09-21T12:40:15.703282+0000","last_peered":"2024-09-21T12:40:15.703282+0000","last_clean":"2024-09-21T12:40:15.703282+0000","last_became_active":"2024-09-21T12:36:00.151584+0000","last_became_peered":"2024-09-21T12:36:00.151584+0000","last_unstale":"2024-09-21T12:40:15.703282+0000","last_undegraded":"2024-09-21T12:40:15.703282+0000","last_fullsized":"2024-09-21T12:40:15.703282+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:57.149832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:57.149832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:10.622809+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":233,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.528993+0000","last_change":"2024-09-21T12:39:44.411449+0000","last_active":"2024-09-21T12:40:15.528993+0000","last_peered":"2024-09-21T12:40:15.528993+0000","last_clean":"2024-09-21T12:40:15.528993+0000","last_became_active":"2024-09-21T12:36:00.144521+0000","last_became_peered":"2024-09-21T12:36:00.144521+0000","last_unstale":"2024-09-21T12:40:15.528993+0000","last_undegraded":"2024-09-21T12:40:15.528993+0000","last_fullsized":"2024-09-21T12:40:15.528993+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:44.411387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:44.411387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:46.423608+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":962,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.528127+0000","last_change":"2024-09-21T12:39:26.328424+0000","last_active":"2024-09-21T12:40:15.528127+0000","last_peered":"2024-09-21T12:40:15.528127+0000","last_clean":"2024-09-21T12:40:15.528127+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:40:15.528127+0000","last_undegraded":"2024-09-21T12:40:15.528127+0000","last_fullsized":"2024-09-21T12:40:15.528127+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:26.328369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:26.328369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:54.895336+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"76'79","reported_seq":233,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:22.667030+0000","last_change":"2024-09-21T12:40:22.667030+0000","last_active":"2024-09-21T12:40:22.667030+0000","last_peered":"2024-09-21T12:40:22.667030+0000","last_clean":"2024-09-21T12:40:22.667030+0000","last_became_active":"2024-09-21T12:36:00.151043+0000","last_became_peered":"2024-09-21T12:36:00.151043+0000","last_unstale":"2024-09-21T12:40:22.667030+0000","last_undegraded":"2024-09-21T12:40:22.667030+0000","last_fullsized":"2024-09-21T12:40:22.667030+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:40:22.666967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:40:22.666967+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:47.557026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"518'12","reported_seq":988,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:22.997810+0000","last_change":"2024-09-21T12:40:12.545354+0000","last_active":"2024-09-21T12:40:22.997810+0000","last_peered":"2024-09-21T12:40:22.997810+0000","last_clean":"2024-09-21T12:40:22.997810+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:40:22.997810+0000","last_undegraded":"2024-09-21T12:40:22.997810+0000","last_fullsized":"2024-09-21T12:40:22.997810+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'8","last_scrub_stamp":"2024-09-21T12:40:12.545286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:40:12.545286+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:26.007593+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":992,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.226640+0000","last_change":"2024-09-21T12:40:08.835848+0000","last_active":"2024-09-21T12:40:15.226640+0000","last_peered":"2024-09-21T12:40:15.226640+0000","last_clean":"2024-09-21T12:40:15.226640+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:40:15.226640+0000","last_undegraded":"2024-09-21T12:40:15.226640+0000","last_fullsized":"2024-09-21T12:40:15.226640+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:40:08.835779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:40:08.835779+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:10.451789+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1009,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.701436+0000","last_change":"2024-09-21T12:39:37.252584+0000","last_active":"2024-09-21T12:40:15.701436+0000","last_peered":"2024-09-21T12:40:15.701436+0000","last_clean":"2024-09-21T12:40:15.701436+0000","last_became_active":"2024-09-21T12:36:00.135537+0000","last_became_peered":"2024-09-21T12:36:00.135537+0000","last_unstale":"2024-09-21T12:40:15.701436+0000","last_undegraded":"2024-09-21T12:40:15.701436+0000","last_fullsized":"2024-09-21T12:40:15.701436+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:37.252520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:37.252520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:49.355760+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":993,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.529504+0000","last_change":"2024-09-21T12:40:12.706791+0000","last_active":"2024-09-21T12:40:15.529504+0000","last_peered":"2024-09-21T12:40:15.529504+0000","last_clean":"2024-09-21T12:40:15.529504+0000","last_became_active":"2024-09-21T12:38:08.164535+0000","last_became_peered":"2024-09-21T12:38:08.164535+0000","last_unstale":"2024-09-21T12:40:15.529504+0000","last_undegraded":"2024-09-21T12:40:15.529504+0000","last_fullsized":"2024-09-21T12:40:15.529504+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:12.706723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:40:12.706723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:35.409788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"76'79","reported_seq":380,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:27.422230+0000","last_change":"2024-09-21T12:40:27.422230+0000","last_active":"2024-09-21T12:40:27.422230+0000","last_peered":"2024-09-21T12:40:27.422230+0000","last_clean":"2024-09-21T12:40:27.422230+0000","last_became_active":"2024-09-21T12:36:00.137290+0000","last_became_peered":"2024-09-21T12:36:00.137290+0000","last_unstale":"2024-09-21T12:40:27.422230+0000","last_undegraded":"2024-09-21T12:40:27.422230+0000","last_fullsized":"2024-09-21T12:40:27.422230+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:40:27.422170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:40:27.422170+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:54.236977+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1000,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.226740+0000","last_change":"2024-09-21T12:40:09.562035+0000","last_active":"2024-09-21T12:40:15.226740+0000","last_peered":"2024-09-21T12:40:15.226740+0000","last_clean":"2024-09-21T12:40:15.226740+0000","last_became_active":"2024-09-21T12:32:38.426822+0000","last_became_peered":"2024-09-21T12:32:38.426822+0000","last_unstale":"2024-09-21T12:40:15.226740+0000","last_undegraded":"2024-09-21T12:40:15.226740+0000","last_fullsized":"2024-09-21T12:40:15.226740+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:09.561964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:40:09.561964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:23.954625+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":988,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.527888+0000","last_change":"2024-09-21T12:39:25.339333+0000","last_active":"2024-09-21T12:40:15.527888+0000","last_peered":"2024-09-21T12:40:15.527888+0000","last_clean":"2024-09-21T12:40:15.527888+0000","last_became_active":"2024-09-21T12:31:39.052486+0000","last_became_peered":"2024-09-21T12:31:39.052486+0000","last_unstale":"2024-09-21T12:40:15.527888+0000","last_undegraded":"2024-09-21T12:40:15.527888+0000","last_fullsized":"2024-09-21T12:40:15.527888+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:25.339277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:25.339277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:39.704442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"518'27","reported_seq":1023,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.167172+0000","last_change":"2024-09-21T12:39:51.135469+0000","last_active":"2024-09-21T12:40:23.167172+0000","last_peered":"2024-09-21T12:40:23.167172+0000","last_clean":"2024-09-21T12:40:23.167172+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:40:23.167172+0000","last_undegraded":"2024-09-21T12:40:23.167172+0000","last_fullsized":"2024-09-21T12:40:23.167172+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:39:51.135405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:51.135405+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:08.006842+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1018,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.225929+0000","last_change":"2024-09-21T12:39:36.668705+0000","last_active":"2024-09-21T12:40:15.225929+0000","last_peered":"2024-09-21T12:40:15.225929+0000","last_clean":"2024-09-21T12:40:15.225929+0000","last_became_active":"2024-09-21T12:38:08.174864+0000","last_became_peered":"2024-09-21T12:38:08.174864+0000","last_unstale":"2024-09-21T12:40:15.225929+0000","last_undegraded":"2024-09-21T12:40:15.225929+0000","last_fullsized":"2024-09-21T12:40:15.225929+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:36.668657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:36.668657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:55.188047+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1535,"num_read_kb":1292,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":462,"num_bytes_recovered":9130,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3597,"internal_metadata":0},"log_size":975,"ondisk_log_size":975,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":598,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":278,"num_read_kb":273,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2449,"ondisk_log_size":2449,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234232,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":848896,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":92896256,"statfs":{"total":95995035648,"available":95125766144,"internally_reserved":0,"allocated":1425408,"data_stored":943751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":867826122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463168,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":889196,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":92855956,"statfs":{"total":95995035648,"available":95084498944,"internally_reserved":0,"allocated":1470464,"data_stored":1008094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":909048262},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724803,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":780868,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":779454,"kb_avail":92964284,"statfs":{"total":95995035648,"available":95195426816,"internally_reserved":0,"allocated":1429504,"data_stored":944687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":798161349},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953739,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":425220,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93319932,"statfs":{"total":95995035648,"available":95559610368,"internally_reserved":0,"allocated":1167360,"data_stored":680583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149969,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":929488,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":927998,"kb_avail":92815664,"statfs":{"total":95995035648,"available":95043239936,"internally_reserved":0,"allocated":1507328,"data_stored":1009868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":950270407},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":455,"seq":1954210119711,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":233904,"kb_used_data":1376,"kb_used_omap":7,"kb_used_meta":232504,"kb_avail":93511248,"statfs":{"total":95995035648,"available":95755517952,"internally_reserved":0,"allocated":1409024,"data_stored":943034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8042,"internal_metadata":238084246},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575133,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":993056,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":991870,"kb_avail":92752096,"statfs":{"total":95995035648,"available":94978146304,"internally_reserved":0,"allocated":1196032,"data_stored":681446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1015675333},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738658,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":930888,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":929406,"kb_avail":92814264,"statfs":{"total":95995035648,"available":95041806336,"internally_reserved":0,"allocated":1499136,"data_stored":1009458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":951712197},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3597,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:40:34.365 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:40:34.612 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:40:34.733 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:34 smithi114 ceph-mon[30324]: 5.1d scrub starts 2024-09-21T12:40:34.733 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:34 smithi114 ceph-mon[30324]: 5.1d scrub ok 2024-09-21T12:40:34.733 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:34 smithi114 ceph-mon[30324]: pgmap v1047: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-21T12:40:34.733 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:34 smithi114 ceph-mon[30324]: 3.9 scrub starts 2024-09-21T12:40:34.733 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:34 smithi114 ceph-mon[30324]: 3.9 scrub ok 2024-09-21T12:40:34.733 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:34 smithi114 ceph-mon[40867]: 5.1d scrub starts 2024-09-21T12:40:34.733 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:34 smithi114 ceph-mon[40867]: 5.1d scrub ok 2024-09-21T12:40:34.734 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:34 smithi114 ceph-mon[40867]: pgmap v1047: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-21T12:40:34.734 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:34 smithi114 ceph-mon[40867]: 3.9 scrub starts 2024-09-21T12:40:34.734 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:34 smithi114 ceph-mon[40867]: 3.9 scrub ok 2024-09-21T12:40:34.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:34 smithi183 ceph-mon[33277]: 5.1d scrub starts 2024-09-21T12:40:34.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:34 smithi183 ceph-mon[33277]: 5.1d scrub ok 2024-09-21T12:40:34.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:34 smithi183 ceph-mon[33277]: pgmap v1047: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-09-21T12:40:34.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:34 smithi183 ceph-mon[33277]: 3.9 scrub starts 2024-09-21T12:40:34.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:34 smithi183 ceph-mon[33277]: 3.9 scrub ok 2024-09-21T12:40:35.284 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-21T12:40:35.286 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-21T12:40:35.659 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:35 smithi114 ceph-mon[40867]: from='client.26714 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:40:35.659 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:35 smithi114 ceph-mon[40867]: 3.4 scrub starts 2024-09-21T12:40:35.659 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:35 smithi114 ceph-mon[40867]: 3.4 scrub ok 2024-09-21T12:40:35.659 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:35 smithi114 ceph-mon[40867]: 5.1b scrub starts 2024-09-21T12:40:35.659 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:35 smithi114 ceph-mon[40867]: 5.1b scrub ok 2024-09-21T12:40:35.659 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:35 smithi114 ceph-mon[40867]: 3.7 scrub starts 2024-09-21T12:40:35.659 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:35 smithi114 ceph-mon[40867]: 3.7 scrub ok 2024-09-21T12:40:35.659 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:35 smithi114 ceph-mon[30324]: from='client.26714 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:40:35.659 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:35 smithi114 ceph-mon[30324]: 3.4 scrub starts 2024-09-21T12:40:35.659 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:35 smithi114 ceph-mon[30324]: 3.4 scrub ok 2024-09-21T12:40:35.660 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:35 smithi114 ceph-mon[30324]: 5.1b scrub starts 2024-09-21T12:40:35.660 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:35 smithi114 ceph-mon[30324]: 5.1b scrub ok 2024-09-21T12:40:35.660 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:35 smithi114 ceph-mon[30324]: 3.7 scrub starts 2024-09-21T12:40:35.660 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:35 smithi114 ceph-mon[30324]: 3.7 scrub ok 2024-09-21T12:40:35.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:35 smithi183 ceph-mon[33277]: from='client.26714 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:40:35.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:35 smithi183 ceph-mon[33277]: 3.4 scrub starts 2024-09-21T12:40:35.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:35 smithi183 ceph-mon[33277]: 3.4 scrub ok 2024-09-21T12:40:35.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:35 smithi183 ceph-mon[33277]: 5.1b scrub starts 2024-09-21T12:40:35.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:35 smithi183 ceph-mon[33277]: 5.1b scrub ok 2024-09-21T12:40:35.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:35 smithi183 ceph-mon[33277]: 3.7 scrub starts 2024-09-21T12:40:35.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:35 smithi183 ceph-mon[33277]: 3.7 scrub ok 2024-09-21T12:40:36.084 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":1048,"stamp":"2024-09-21T12:40:35.228687+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":1827,"num_read_kb":1579,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":493,"num_bytes_recovered":272805,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3552,"ondisk_log_size":3552,"up":486,"acting":486,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":486,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6031516,"kb_used_data":10844,"kb_used_omap":18,"kb_used_meta":6020525,"kb_avail":743929700,"statfs":{"total":767960285184,"available":761784012800,"internally_reserved":0,"allocated":11104256,"data_stored":7220921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19199,"internal_metadata":6165017857},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":361,"num_read_kb":361,"num_write":234,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003141"},"pg_stats":[{"pgid":"1.1d","version":"76'79","reported_seq":677,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.226603+0000","last_change":"2024-09-21T12:40:14.582628+0000","last_active":"2024-09-21T12:40:15.226603+0000","last_peered":"2024-09-21T12:40:15.226603+0000","last_clean":"2024-09-21T12:40:15.226603+0000","last_became_active":"2024-09-21T12:32:38.426383+0000","last_became_peered":"2024-09-21T12:32:38.426383+0000","last_unstale":"2024-09-21T12:40:15.226603+0000","last_undegraded":"2024-09-21T12:40:15.226603+0000","last_fullsized":"2024-09-21T12:40:15.226603+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:40:14.582579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:40:14.582579+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:30.420194+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":990,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.226558+0000","last_change":"2024-09-21T12:40:02.821040+0000","last_active":"2024-09-21T12:40:15.226558+0000","last_peered":"2024-09-21T12:40:15.226558+0000","last_clean":"2024-09-21T12:40:15.226558+0000","last_became_active":"2024-09-21T12:38:07.167857+0000","last_became_peered":"2024-09-21T12:38:07.167857+0000","last_unstale":"2024-09-21T12:40:15.226558+0000","last_undegraded":"2024-09-21T12:40:15.226558+0000","last_fullsized":"2024-09-21T12:40:15.226558+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:02.820974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:40:02.820974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:21.756947+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"518'21","reported_seq":385,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:22.977153+0000","last_change":"2024-09-21T12:39:16.266371+0000","last_active":"2024-09-21T12:40:22.977153+0000","last_peered":"2024-09-21T12:40:22.977153+0000","last_clean":"2024-09-21T12:40:22.977153+0000","last_became_active":"2024-09-21T12:32:38.419081+0000","last_became_peered":"2024-09-21T12:32:38.419081+0000","last_unstale":"2024-09-21T12:40:22.977153+0000","last_undegraded":"2024-09-21T12:40:22.977153+0000","last_fullsized":"2024-09-21T12:40:22.977153+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:39:16.266285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:16.266285+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:30.189485+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":989,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.224138+0000","last_change":"2024-09-21T12:39:17.175795+0000","last_active":"2024-09-21T12:40:15.224138+0000","last_peered":"2024-09-21T12:40:15.224138+0000","last_clean":"2024-09-21T12:40:15.224138+0000","last_became_active":"2024-09-21T12:36:00.136673+0000","last_became_peered":"2024-09-21T12:36:00.136673+0000","last_unstale":"2024-09-21T12:40:15.224138+0000","last_undegraded":"2024-09-21T12:40:15.224138+0000","last_fullsized":"2024-09-21T12:40:15.224138+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:17.175740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:17.175740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:46.137502+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":374,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:30.447423+0000","last_change":"2024-09-21T12:40:30.447423+0000","last_active":"2024-09-21T12:40:30.447423+0000","last_peered":"2024-09-21T12:40:30.447423+0000","last_clean":"2024-09-21T12:40:30.447423+0000","last_became_active":"2024-09-21T12:32:38.425588+0000","last_became_peered":"2024-09-21T12:32:38.425588+0000","last_unstale":"2024-09-21T12:40:30.447423+0000","last_undegraded":"2024-09-21T12:40:30.447423+0000","last_fullsized":"2024-09-21T12:40:30.447423+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:30.447365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:40:30.447365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:49.853295+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"76'79","reported_seq":233,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.530150+0000","last_change":"2024-09-21T12:39:24.527707+0000","last_active":"2024-09-21T12:40:15.530150+0000","last_peered":"2024-09-21T12:40:15.530150+0000","last_clean":"2024-09-21T12:40:15.530150+0000","last_became_active":"2024-09-21T12:36:00.154209+0000","last_became_peered":"2024-09-21T12:36:00.154209+0000","last_unstale":"2024-09-21T12:40:15.530150+0000","last_undegraded":"2024-09-21T12:40:15.530150+0000","last_fullsized":"2024-09-21T12:40:15.530150+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:24.527622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:24.527622+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:37.668270+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":976,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.527996+0000","last_change":"2024-09-21T12:39:53.588395+0000","last_active":"2024-09-21T12:40:15.527996+0000","last_peered":"2024-09-21T12:40:15.527996+0000","last_clean":"2024-09-21T12:40:15.527996+0000","last_became_active":"2024-09-21T12:36:00.139167+0000","last_became_peered":"2024-09-21T12:36:00.139167+0000","last_unstale":"2024-09-21T12:40:15.527996+0000","last_undegraded":"2024-09-21T12:40:15.527996+0000","last_fullsized":"2024-09-21T12:40:15.527996+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:53.588345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:53.588345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:05.269035+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":987,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.528741+0000","last_change":"2024-09-21T12:40:00.466457+0000","last_active":"2024-09-21T12:40:15.528741+0000","last_peered":"2024-09-21T12:40:15.528741+0000","last_clean":"2024-09-21T12:40:15.528741+0000","last_became_active":"2024-09-21T12:38:07.157964+0000","last_became_peered":"2024-09-21T12:38:07.157964+0000","last_unstale":"2024-09-21T12:40:15.528741+0000","last_undegraded":"2024-09-21T12:40:15.528741+0000","last_fullsized":"2024-09-21T12:40:15.528741+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:00.466413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:40:00.466413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:25.941476+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"518'39","reported_seq":1050,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.171553+0000","last_change":"2024-09-21T12:40:03.491219+0000","last_active":"2024-09-21T12:40:23.171553+0000","last_peered":"2024-09-21T12:40:23.171553+0000","last_clean":"2024-09-21T12:40:23.171553+0000","last_became_active":"2024-09-21T12:38:07.168250+0000","last_became_peered":"2024-09-21T12:38:07.168250+0000","last_unstale":"2024-09-21T12:40:23.171553+0000","last_undegraded":"2024-09-21T12:40:23.171553+0000","last_fullsized":"2024-09-21T12:40:23.171553+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:40:03.491164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:40:03.491164+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:05.177020+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":366,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:29.428119+0000","last_change":"2024-09-21T12:40:29.428119+0000","last_active":"2024-09-21T12:40:29.428119+0000","last_peered":"2024-09-21T12:40:29.428119+0000","last_clean":"2024-09-21T12:40:29.428119+0000","last_became_active":"2024-09-21T12:32:38.417724+0000","last_became_peered":"2024-09-21T12:32:38.417724+0000","last_unstale":"2024-09-21T12:40:29.428119+0000","last_undegraded":"2024-09-21T12:40:29.428119+0000","last_fullsized":"2024-09-21T12:40:29.428119+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:29.428061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:40:29.428061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:32.240082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":527,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.703145+0000","last_change":"2024-09-21T12:39:18.247341+0000","last_active":"2024-09-21T12:40:15.703145+0000","last_peered":"2024-09-21T12:40:15.703145+0000","last_clean":"2024-09-21T12:40:15.703145+0000","last_became_active":"2024-09-21T12:37:20.467925+0000","last_became_peered":"2024-09-21T12:37:20.467925+0000","last_unstale":"2024-09-21T12:40:15.703145+0000","last_undegraded":"2024-09-21T12:40:15.703145+0000","last_fullsized":"2024-09-21T12:40:15.703145+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:18.247283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:18.247283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:30.819585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"518'21","reported_seq":401,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.077851+0000","last_change":"2024-09-21T12:39:38.519883+0000","last_active":"2024-09-21T12:40:23.077851+0000","last_peered":"2024-09-21T12:40:23.077851+0000","last_clean":"2024-09-21T12:40:23.077851+0000","last_became_active":"2024-09-21T12:32:38.426121+0000","last_became_peered":"2024-09-21T12:32:38.426121+0000","last_unstale":"2024-09-21T12:40:23.077851+0000","last_undegraded":"2024-09-21T12:40:23.077851+0000","last_fullsized":"2024-09-21T12:40:23.077851+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:39:38.519813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:38.519813+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:41.134696+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1020,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.702124+0000","last_change":"2024-09-21T12:38:44.245362+0000","last_active":"2024-09-21T12:40:15.702124+0000","last_peered":"2024-09-21T12:40:15.702124+0000","last_clean":"2024-09-21T12:40:15.702124+0000","last_became_active":"2024-09-21T12:36:00.137364+0000","last_became_peered":"2024-09-21T12:36:00.137364+0000","last_unstale":"2024-09-21T12:40:15.702124+0000","last_undegraded":"2024-09-21T12:40:15.702124+0000","last_fullsized":"2024-09-21T12:40:15.702124+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:44.245309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:44.245309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":987,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.225313+0000","last_change":"2024-09-21T12:39:31.655725+0000","last_active":"2024-09-21T12:40:15.225313+0000","last_peered":"2024-09-21T12:40:15.225313+0000","last_clean":"2024-09-21T12:40:15.225313+0000","last_became_active":"2024-09-21T12:36:00.134723+0000","last_became_peered":"2024-09-21T12:36:00.134723+0000","last_unstale":"2024-09-21T12:40:15.225313+0000","last_undegraded":"2024-09-21T12:40:15.225313+0000","last_fullsized":"2024-09-21T12:40:15.225313+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:31.655654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:31.655654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:48.861074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"76'79","reported_seq":761,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.528601+0000","last_change":"2024-09-21T12:39:43.470032+0000","last_active":"2024-09-21T12:40:15.528601+0000","last_peered":"2024-09-21T12:40:15.528601+0000","last_clean":"2024-09-21T12:40:15.528601+0000","last_became_active":"2024-09-21T12:36:00.147795+0000","last_became_peered":"2024-09-21T12:36:00.147795+0000","last_unstale":"2024-09-21T12:40:15.528601+0000","last_undegraded":"2024-09-21T12:40:15.528601+0000","last_fullsized":"2024-09-21T12:40:15.528601+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:43.469978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:43.469978+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:05.783906+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":519,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:22.084456+0000","last_change":"2024-09-21T12:40:22.084456+0000","last_active":"2024-09-21T12:40:22.084456+0000","last_peered":"2024-09-21T12:40:22.084456+0000","last_clean":"2024-09-21T12:40:22.084456+0000","last_became_active":"2024-09-21T12:37:20.468468+0000","last_became_peered":"2024-09-21T12:37:20.468468+0000","last_unstale":"2024-09-21T12:40:22.084456+0000","last_undegraded":"2024-09-21T12:40:22.084456+0000","last_fullsized":"2024-09-21T12:40:22.084456+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:22.084399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:40:22.084399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:22.686670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"518'9","reported_seq":383,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.057245+0000","last_change":"2024-09-21T12:39:19.280732+0000","last_active":"2024-09-21T12:40:23.057245+0000","last_peered":"2024-09-21T12:40:23.057245+0000","last_clean":"2024-09-21T12:40:23.057245+0000","last_became_active":"2024-09-21T12:32:38.422912+0000","last_became_peered":"2024-09-21T12:32:38.422912+0000","last_unstale":"2024-09-21T12:40:23.057245+0000","last_undegraded":"2024-09-21T12:40:23.057245+0000","last_fullsized":"2024-09-21T12:40:23.057245+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'7","last_scrub_stamp":"2024-09-21T12:39:19.280679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:19.280679+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:31.804604+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":233,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.529233+0000","last_change":"2024-09-21T12:39:47.460848+0000","last_active":"2024-09-21T12:40:15.529233+0000","last_peered":"2024-09-21T12:40:15.529233+0000","last_clean":"2024-09-21T12:40:15.529233+0000","last_became_active":"2024-09-21T12:36:00.142106+0000","last_became_peered":"2024-09-21T12:36:00.142106+0000","last_unstale":"2024-09-21T12:40:15.529233+0000","last_undegraded":"2024-09-21T12:40:15.529233+0000","last_fullsized":"2024-09-21T12:40:15.529233+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:47.460786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:47.460786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:09.399546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":980,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.701485+0000","last_change":"2024-09-21T12:39:35.183449+0000","last_active":"2024-09-21T12:40:15.701485+0000","last_peered":"2024-09-21T12:40:15.701485+0000","last_clean":"2024-09-21T12:40:15.701485+0000","last_became_active":"2024-09-21T12:20:21.311055+0000","last_became_peered":"2024-09-21T12:20:21.311055+0000","last_unstale":"2024-09-21T12:40:15.701485+0000","last_undegraded":"2024-09-21T12:40:15.701485+0000","last_fullsized":"2024-09-21T12:40:15.701485+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:35.183355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:35.183355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:41.588221+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"76'79","reported_seq":801,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.527981+0000","last_change":"2024-09-21T12:40:13.517471+0000","last_active":"2024-09-21T12:40:15.527981+0000","last_peered":"2024-09-21T12:40:15.527981+0000","last_clean":"2024-09-21T12:40:15.527981+0000","last_became_active":"2024-09-21T12:27:16.957569+0000","last_became_peered":"2024-09-21T12:27:16.957569+0000","last_unstale":"2024-09-21T12:40:15.527981+0000","last_undegraded":"2024-09-21T12:40:15.527981+0000","last_fullsized":"2024-09-21T12:40:15.527981+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:40:13.517297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:40:13.517297+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:33.783424+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":366,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:25.437571+0000","last_change":"2024-09-21T12:40:25.437571+0000","last_active":"2024-09-21T12:40:25.437571+0000","last_peered":"2024-09-21T12:40:25.437571+0000","last_clean":"2024-09-21T12:40:25.437571+0000","last_became_active":"2024-09-21T12:32:38.428997+0000","last_became_peered":"2024-09-21T12:32:38.428997+0000","last_unstale":"2024-09-21T12:40:25.437571+0000","last_undegraded":"2024-09-21T12:40:25.437571+0000","last_fullsized":"2024-09-21T12:40:25.437571+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:25.437512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:40:25.437512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:35.814112+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1050,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.225652+0000","last_change":"2024-09-21T12:39:44.734228+0000","last_active":"2024-09-21T12:40:15.225652+0000","last_peered":"2024-09-21T12:40:15.225652+0000","last_clean":"2024-09-21T12:40:15.225652+0000","last_became_active":"2024-09-21T12:36:00.127153+0000","last_became_peered":"2024-09-21T12:36:00.127153+0000","last_unstale":"2024-09-21T12:40:15.225652+0000","last_undegraded":"2024-09-21T12:40:15.225652+0000","last_fullsized":"2024-09-21T12:40:15.225652+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:44.734176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:44.734176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:53.381683+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"518'39","reported_seq":808,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:27.701492+0000","last_change":"2024-09-21T12:40:27.701492+0000","last_active":"2024-09-21T12:40:27.701492+0000","last_peered":"2024-09-21T12:40:27.701492+0000","last_clean":"2024-09-21T12:40:27.701492+0000","last_became_active":"2024-09-21T12:40:10.205351+0000","last_became_peered":"2024-09-21T12:40:10.205351+0000","last_unstale":"2024-09-21T12:40:27.701492+0000","last_undegraded":"2024-09-21T12:40:27.701492+0000","last_fullsized":"2024-09-21T12:40:27.701492+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"518'39","last_scrub_stamp":"2024-09-21T12:40:27.701423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:40:27.701423+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:50.494609+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1003,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.226829+0000","last_change":"2024-09-21T12:38:53.956579+0000","last_active":"2024-09-21T12:40:15.226829+0000","last_peered":"2024-09-21T12:40:15.226829+0000","last_clean":"2024-09-21T12:40:15.226829+0000","last_became_active":"2024-09-21T12:32:38.425958+0000","last_became_peered":"2024-09-21T12:32:38.425958+0000","last_unstale":"2024-09-21T12:40:15.226829+0000","last_undegraded":"2024-09-21T12:40:15.226829+0000","last_fullsized":"2024-09-21T12:40:15.226829+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:53.956515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:53.956515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"76'79","reported_seq":937,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.701288+0000","last_change":"2024-09-21T12:39:26.166502+0000","last_active":"2024-09-21T12:40:15.701288+0000","last_peered":"2024-09-21T12:40:15.701288+0000","last_clean":"2024-09-21T12:40:15.701288+0000","last_became_active":"2024-09-21T12:38:08.157632+0000","last_became_peered":"2024-09-21T12:38:08.157632+0000","last_unstale":"2024-09-21T12:40:15.701288+0000","last_undegraded":"2024-09-21T12:40:15.701288+0000","last_fullsized":"2024-09-21T12:40:15.701288+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:26.166402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:26.166402+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:46.517049+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":984,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.528581+0000","last_change":"2024-09-21T12:39:18.225857+0000","last_active":"2024-09-21T12:40:15.528581+0000","last_peered":"2024-09-21T12:40:15.528581+0000","last_clean":"2024-09-21T12:40:15.528581+0000","last_became_active":"2024-09-21T12:32:02.033276+0000","last_became_peered":"2024-09-21T12:32:02.033276+0000","last_unstale":"2024-09-21T12:40:15.528581+0000","last_undegraded":"2024-09-21T12:40:15.528581+0000","last_fullsized":"2024-09-21T12:40:15.528581+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:18.225813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:18.225813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:37.203577+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":369,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.221932+0000","last_change":"2024-09-21T12:38:41.208373+0000","last_active":"2024-09-21T12:40:15.221932+0000","last_peered":"2024-09-21T12:40:15.221932+0000","last_clean":"2024-09-21T12:40:15.221932+0000","last_became_active":"2024-09-21T12:38:07.159387+0000","last_became_peered":"2024-09-21T12:38:07.159387+0000","last_unstale":"2024-09-21T12:40:15.221932+0000","last_undegraded":"2024-09-21T12:40:15.221932+0000","last_fullsized":"2024-09-21T12:40:15.221932+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:41.208301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:41.208301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"518'24","reported_seq":1087,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.173326+0000","last_change":"2024-09-21T12:39:27.355062+0000","last_active":"2024-09-21T12:40:23.173326+0000","last_peered":"2024-09-21T12:40:23.173326+0000","last_clean":"2024-09-21T12:40:23.173326+0000","last_became_active":"2024-09-21T12:32:38.409163+0000","last_became_peered":"2024-09-21T12:32:38.409163+0000","last_unstale":"2024-09-21T12:40:23.173326+0000","last_undegraded":"2024-09-21T12:40:23.173326+0000","last_fullsized":"2024-09-21T12:40:23.173326+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T12:39:27.355002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:27.355002+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:51.502466+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":377,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.222516+0000","last_change":"2024-09-21T12:38:49.267225+0000","last_active":"2024-09-21T12:40:15.222516+0000","last_peered":"2024-09-21T12:40:15.222516+0000","last_clean":"2024-09-21T12:40:15.222516+0000","last_became_active":"2024-09-21T12:38:07.161634+0000","last_became_peered":"2024-09-21T12:38:07.161634+0000","last_unstale":"2024-09-21T12:40:15.222516+0000","last_undegraded":"2024-09-21T12:40:15.222516+0000","last_fullsized":"2024-09-21T12:40:15.222516+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:49.267162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:49.267162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"76'79","reported_seq":726,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.701702+0000","last_change":"2024-09-21T12:40:10.186296+0000","last_active":"2024-09-21T12:40:15.701702+0000","last_peered":"2024-09-21T12:40:15.701702+0000","last_clean":"2024-09-21T12:40:15.701702+0000","last_became_active":"2024-09-21T12:32:38.414204+0000","last_became_peered":"2024-09-21T12:32:38.414204+0000","last_unstale":"2024-09-21T12:40:15.701702+0000","last_undegraded":"2024-09-21T12:40:15.701702+0000","last_fullsized":"2024-09-21T12:40:15.701702+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:40:10.186225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:40:10.186225+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:23.922014+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":958,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.529941+0000","last_change":"2024-09-21T12:39:22.535974+0000","last_active":"2024-09-21T12:40:15.529941+0000","last_peered":"2024-09-21T12:40:15.529941+0000","last_clean":"2024-09-21T12:40:15.529941+0000","last_became_active":"2024-09-21T12:38:08.165695+0000","last_became_peered":"2024-09-21T12:38:08.165695+0000","last_unstale":"2024-09-21T12:40:15.529941+0000","last_undegraded":"2024-09-21T12:40:15.529941+0000","last_fullsized":"2024-09-21T12:40:15.529941+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:39:22.535916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:22.535916+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:48.068876+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"59'1","reported_seq":1029,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.224275+0000","last_change":"2024-09-21T12:39:59.453047+0000","last_active":"2024-09-21T12:40:15.224275+0000","last_peered":"2024-09-21T12:40:15.224275+0000","last_clean":"2024-09-21T12:40:15.224275+0000","last_became_active":"2024-09-21T12:32:38.425335+0000","last_became_peered":"2024-09-21T12:32:38.425335+0000","last_unstale":"2024-09-21T12:40:15.224275+0000","last_undegraded":"2024-09-21T12:40:15.224275+0000","last_fullsized":"2024-09-21T12:40:15.224275+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:39:59.452988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:59.452988+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:28.155192+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"518'30","reported_seq":1121,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:27.928984+0000","last_change":"2024-09-21T12:40:27.928984+0000","last_active":"2024-09-21T12:40:27.928984+0000","last_peered":"2024-09-21T12:40:27.928984+0000","last_clean":"2024-09-21T12:40:27.928984+0000","last_became_active":"2024-09-21T12:36:00.148761+0000","last_became_peered":"2024-09-21T12:36:00.148761+0000","last_unstale":"2024-09-21T12:40:27.928984+0000","last_undegraded":"2024-09-21T12:40:27.928984+0000","last_fullsized":"2024-09-21T12:40:27.928984+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"518'30","last_scrub_stamp":"2024-09-21T12:40:27.928904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:40:27.928904+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:51.385029+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"510'19","reported_seq":614,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:25.602643+0000","last_change":"2024-09-21T12:40:17.861797+0000","last_active":"2024-09-21T12:40:25.602643+0000","last_peered":"2024-09-21T12:40:25.602643+0000","last_clean":"2024-09-21T12:40:25.602643+0000","last_became_active":"2024-09-21T12:40:01.428161+0000","last_became_peered":"2024-09-21T12:40:01.428161+0000","last_unstale":"2024-09-21T12:40:25.602643+0000","last_undegraded":"2024-09-21T12:40:25.602643+0000","last_fullsized":"2024-09-21T12:40:25.602643+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"510'19","last_scrub_stamp":"2024-09-21T12:40:17.861729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:40:17.861729+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:25.090781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"76'79","reported_seq":537,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.702842+0000","last_change":"2024-09-21T12:39:32.209673+0000","last_active":"2024-09-21T12:40:15.702842+0000","last_peered":"2024-09-21T12:40:15.702842+0000","last_clean":"2024-09-21T12:40:15.702842+0000","last_became_active":"2024-09-21T12:38:07.165640+0000","last_became_peered":"2024-09-21T12:38:07.165640+0000","last_unstale":"2024-09-21T12:40:15.702842+0000","last_undegraded":"2024-09-21T12:40:15.702842+0000","last_fullsized":"2024-09-21T12:40:15.702842+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:32.209596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:32.209596+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:35.994443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"518'39","reported_seq":630,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.179674+0000","last_change":"2024-09-21T12:40:16.616707+0000","last_active":"2024-09-21T12:40:23.179674+0000","last_peered":"2024-09-21T12:40:23.179674+0000","last_clean":"2024-09-21T12:40:23.179674+0000","last_became_active":"2024-09-21T12:40:10.205854+0000","last_became_peered":"2024-09-21T12:40:10.205854+0000","last_unstale":"2024-09-21T12:40:23.179674+0000","last_undegraded":"2024-09-21T12:40:23.179674+0000","last_fullsized":"2024-09-21T12:40:23.179674+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:40:16.616651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:40:16.616651+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:25.142385+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":449,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.527553+0000","last_change":"2024-09-21T12:39:59.459892+0000","last_active":"2024-09-21T12:40:15.527553+0000","last_peered":"2024-09-21T12:40:15.527553+0000","last_clean":"2024-09-21T12:40:15.527553+0000","last_became_active":"2024-09-21T12:36:00.140455+0000","last_became_peered":"2024-09-21T12:36:00.140455+0000","last_unstale":"2024-09-21T12:40:15.527553+0000","last_undegraded":"2024-09-21T12:40:15.527553+0000","last_fullsized":"2024-09-21T12:40:15.527553+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:59.459843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:59.459843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:19.062658+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"352'15","reported_seq":465,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:25.602892+0000","last_change":"2024-09-21T12:38:02.464559+0000","last_active":"2024-09-21T12:40:25.602892+0000","last_peered":"2024-09-21T12:40:25.602892+0000","last_clean":"2024-09-21T12:40:25.602892+0000","last_became_active":"2024-09-21T12:32:38.429092+0000","last_became_peered":"2024-09-21T12:32:38.429092+0000","last_unstale":"2024-09-21T12:40:25.602892+0000","last_undegraded":"2024-09-21T12:40:25.602892+0000","last_fullsized":"2024-09-21T12:40:25.602892+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'15","last_scrub_stamp":"2024-09-21T12:38:02.464486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:02.464486+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"66'1","reported_seq":487,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.703385+0000","last_change":"2024-09-21T12:39:44.280084+0000","last_active":"2024-09-21T12:40:15.703385+0000","last_peered":"2024-09-21T12:40:15.703385+0000","last_clean":"2024-09-21T12:40:15.703385+0000","last_became_active":"2024-09-21T12:37:20.468591+0000","last_became_peered":"2024-09-21T12:37:20.468591+0000","last_unstale":"2024-09-21T12:40:15.703385+0000","last_undegraded":"2024-09-21T12:40:15.703385+0000","last_fullsized":"2024-09-21T12:40:15.703385+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T12:39:44.280022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:44.280022+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:57.269368+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"76'79","reported_seq":377,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.222395+0000","last_change":"2024-09-21T12:38:51.310242+0000","last_active":"2024-09-21T12:40:15.222395+0000","last_peered":"2024-09-21T12:40:15.222395+0000","last_clean":"2024-09-21T12:40:15.222395+0000","last_became_active":"2024-09-21T12:38:07.159280+0000","last_became_peered":"2024-09-21T12:38:07.159280+0000","last_unstale":"2024-09-21T12:40:15.222395+0000","last_undegraded":"2024-09-21T12:40:15.222395+0000","last_fullsized":"2024-09-21T12:40:15.222395+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:38:51.310157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:38:51.310157+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"518'42","reported_seq":1148,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.136358+0000","last_change":"2024-09-21T12:40:15.531899+0000","last_active":"2024-09-21T12:40:23.136358+0000","last_peered":"2024-09-21T12:40:23.136358+0000","last_clean":"2024-09-21T12:40:23.136358+0000","last_became_active":"2024-09-21T12:36:00.139849+0000","last_became_peered":"2024-09-21T12:36:00.139849+0000","last_unstale":"2024-09-21T12:40:23.136358+0000","last_undegraded":"2024-09-21T12:40:23.136358+0000","last_fullsized":"2024-09-21T12:40:23.136358+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'30","last_scrub_stamp":"2024-09-21T12:40:15.531825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:40:15.531825+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:24.163047+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":981,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.528791+0000","last_change":"2024-09-21T12:39:20.300208+0000","last_active":"2024-09-21T12:40:15.528791+0000","last_peered":"2024-09-21T12:40:15.528791+0000","last_clean":"2024-09-21T12:40:15.528791+0000","last_became_active":"2024-09-21T12:36:00.151488+0000","last_became_peered":"2024-09-21T12:36:00.151488+0000","last_unstale":"2024-09-21T12:40:15.528791+0000","last_undegraded":"2024-09-21T12:40:15.528791+0000","last_fullsized":"2024-09-21T12:40:15.528791+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:20.300049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:20.300049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:33.837202+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":380,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:26.442021+0000","last_change":"2024-09-21T12:40:26.442021+0000","last_active":"2024-09-21T12:40:26.442021+0000","last_peered":"2024-09-21T12:40:26.442021+0000","last_clean":"2024-09-21T12:40:26.442021+0000","last_became_active":"2024-09-21T12:36:00.136650+0000","last_became_peered":"2024-09-21T12:36:00.136650+0000","last_unstale":"2024-09-21T12:40:26.442021+0000","last_undegraded":"2024-09-21T12:40:26.442021+0000","last_fullsized":"2024-09-21T12:40:26.442021+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:26.441959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:40:26.441959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:29.201734+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":369,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.222214+0000","last_change":"2024-09-21T12:38:44.156366+0000","last_active":"2024-09-21T12:40:15.222214+0000","last_peered":"2024-09-21T12:40:15.222214+0000","last_clean":"2024-09-21T12:40:15.222214+0000","last_became_active":"2024-09-21T12:38:07.166919+0000","last_became_peered":"2024-09-21T12:38:07.166919+0000","last_unstale":"2024-09-21T12:40:15.222214+0000","last_undegraded":"2024-09-21T12:40:15.222214+0000","last_fullsized":"2024-09-21T12:40:15.222214+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:44.156307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:38:44.156307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"76'79","reported_seq":916,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.528241+0000","last_change":"2024-09-21T12:39:30.289586+0000","last_active":"2024-09-21T12:40:15.528241+0000","last_peered":"2024-09-21T12:40:15.528241+0000","last_clean":"2024-09-21T12:40:15.528241+0000","last_became_active":"2024-09-21T12:36:00.155047+0000","last_became_peered":"2024-09-21T12:36:00.155047+0000","last_unstale":"2024-09-21T12:40:15.528241+0000","last_undegraded":"2024-09-21T12:40:15.528241+0000","last_fullsized":"2024-09-21T12:40:15.528241+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:30.289532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:30.289532+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:44.792256+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"518'42","reported_seq":395,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.162971+0000","last_change":"2024-09-21T12:39:34.562769+0000","last_active":"2024-09-21T12:40:23.162971+0000","last_peered":"2024-09-21T12:40:23.162971+0000","last_clean":"2024-09-21T12:40:23.162971+0000","last_became_active":"2024-09-21T12:32:38.425453+0000","last_became_peered":"2024-09-21T12:32:38.425453+0000","last_unstale":"2024-09-21T12:40:23.162971+0000","last_undegraded":"2024-09-21T12:40:23.162971+0000","last_fullsized":"2024-09-21T12:40:23.162971+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'28","last_scrub_stamp":"2024-09-21T12:39:34.562691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:34.562691+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:41.975070+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"59'1","reported_seq":1065,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.226252+0000","last_change":"2024-09-21T12:39:37.668319+0000","last_active":"2024-09-21T12:40:15.226252+0000","last_peered":"2024-09-21T12:40:15.226252+0000","last_clean":"2024-09-21T12:40:15.226252+0000","last_became_active":"2024-09-21T12:32:38.424412+0000","last_became_peered":"2024-09-21T12:32:38.424412+0000","last_unstale":"2024-09-21T12:40:15.226252+0000","last_undegraded":"2024-09-21T12:40:15.226252+0000","last_fullsized":"2024-09-21T12:40:15.226252+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:39:37.668271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:37.668271+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:06.741317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"352'12","reported_seq":1272,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:28.151395+0000","last_change":"2024-09-21T12:39:47.512406+0000","last_active":"2024-09-21T12:40:28.151395+0000","last_peered":"2024-09-21T12:40:28.151395+0000","last_clean":"2024-09-21T12:40:28.151395+0000","last_became_active":"2024-09-21T12:32:38.409177+0000","last_became_peered":"2024-09-21T12:32:38.409177+0000","last_unstale":"2024-09-21T12:40:28.151395+0000","last_undegraded":"2024-09-21T12:40:28.151395+0000","last_fullsized":"2024-09-21T12:40:28.151395+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'12","last_scrub_stamp":"2024-09-21T12:39:47.512357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:47.512357+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:17.319208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":966,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:18.648172+0000","last_change":"2024-09-21T12:40:18.648172+0000","last_active":"2024-09-21T12:40:18.648172+0000","last_peered":"2024-09-21T12:40:18.648172+0000","last_clean":"2024-09-21T12:40:18.648172+0000","last_became_active":"2024-09-21T12:38:08.154409+0000","last_became_peered":"2024-09-21T12:38:08.154409+0000","last_unstale":"2024-09-21T12:40:18.648172+0000","last_undegraded":"2024-09-21T12:40:18.648172+0000","last_fullsized":"2024-09-21T12:40:18.648172+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:18.648117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:40:18.648117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:21.869154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"76'79","reported_seq":474,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.702457+0000","last_change":"2024-09-21T12:40:01.428877+0000","last_active":"2024-09-21T12:40:15.702457+0000","last_peered":"2024-09-21T12:40:15.702457+0000","last_clean":"2024-09-21T12:40:15.702457+0000","last_became_active":"2024-09-21T12:40:01.428481+0000","last_became_peered":"2024-09-21T12:40:01.428481+0000","last_unstale":"2024-09-21T12:40:15.702457+0000","last_undegraded":"2024-09-21T12:40:15.702457+0000","last_fullsized":"2024-09-21T12:40:15.702457+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:41.480738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:41.480738+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:51.477238+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"518'30","reported_seq":533,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.151322+0000","last_change":"2024-09-21T12:40:07.491352+0000","last_active":"2024-09-21T12:40:23.151322+0000","last_peered":"2024-09-21T12:40:23.151322+0000","last_clean":"2024-09-21T12:40:23.151322+0000","last_became_active":"2024-09-21T12:32:38.424412+0000","last_became_peered":"2024-09-21T12:32:38.424412+0000","last_unstale":"2024-09-21T12:40:23.151322+0000","last_undegraded":"2024-09-21T12:40:23.151322+0000","last_fullsized":"2024-09-21T12:40:23.151322+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:40:07.491287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:40:07.491287+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:07.651957+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":996,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.703112+0000","last_change":"2024-09-21T12:40:12.112607+0000","last_active":"2024-09-21T12:40:15.703112+0000","last_peered":"2024-09-21T12:40:15.703112+0000","last_clean":"2024-09-21T12:40:15.703112+0000","last_became_active":"2024-09-21T12:38:07.165614+0000","last_became_peered":"2024-09-21T12:38:07.165614+0000","last_unstale":"2024-09-21T12:40:15.703112+0000","last_undegraded":"2024-09-21T12:40:15.703112+0000","last_fullsized":"2024-09-21T12:40:15.703112+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:12.112551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:40:12.112551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:25.965104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":1234,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:25.602624+0000","last_change":"2024-09-21T12:39:32.632252+0000","last_active":"2024-09-21T12:40:25.602624+0000","last_peered":"2024-09-21T12:40:25.602624+0000","last_clean":"2024-09-21T12:40:25.602624+0000","last_became_active":"2024-09-21T12:20:19.307183+0000","last_became_peered":"2024-09-21T12:20:19.307183+0000","last_unstale":"2024-09-21T12:40:25.602624+0000","last_undegraded":"2024-09-21T12:40:25.602624+0000","last_fullsized":"2024-09-21T12:40:25.602624+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:39:32.632187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:32.632187+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:51.554132+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":979,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:22.930917+0000","last_change":"2024-09-21T12:40:22.930855+0000","last_active":"2024-09-21T12:40:22.930917+0000","last_peered":"2024-09-21T12:40:22.930917+0000","last_clean":"2024-09-21T12:40:22.930917+0000","last_became_active":"2024-09-21T12:38:07.163353+0000","last_became_peered":"2024-09-21T12:38:07.163353+0000","last_unstale":"2024-09-21T12:40:22.930917+0000","last_undegraded":"2024-09-21T12:40:22.930917+0000","last_fullsized":"2024-09-21T12:40:22.930917+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:22.930784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:40:22.930784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:33.007973+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"76'79","reported_seq":474,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.702083+0000","last_change":"2024-09-21T12:40:01.424167+0000","last_active":"2024-09-21T12:40:15.702083+0000","last_peered":"2024-09-21T12:40:15.702083+0000","last_clean":"2024-09-21T12:40:15.702083+0000","last_became_active":"2024-09-21T12:40:01.423746+0000","last_became_peered":"2024-09-21T12:40:01.423746+0000","last_unstale":"2024-09-21T12:40:15.702083+0000","last_undegraded":"2024-09-21T12:40:15.702083+0000","last_fullsized":"2024-09-21T12:40:15.702083+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:57.712232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:57.712232+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:00.968556+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"518'39","reported_seq":1091,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.145409+0000","last_change":"2024-09-21T12:39:30.640700+0000","last_active":"2024-09-21T12:40:23.145409+0000","last_peered":"2024-09-21T12:40:23.145409+0000","last_clean":"2024-09-21T12:40:23.145409+0000","last_became_active":"2024-09-21T12:36:00.127312+0000","last_became_peered":"2024-09-21T12:36:00.127312+0000","last_unstale":"2024-09-21T12:40:23.145409+0000","last_undegraded":"2024-09-21T12:40:23.145409+0000","last_fullsized":"2024-09-21T12:40:23.145409+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'29","last_scrub_stamp":"2024-09-21T12:39:30.640595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:30.640595+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:52.831790+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":972,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.226585+0000","last_change":"2024-09-21T12:38:48.953684+0000","last_active":"2024-09-21T12:40:15.226585+0000","last_peered":"2024-09-21T12:40:15.226585+0000","last_clean":"2024-09-21T12:40:15.226585+0000","last_became_active":"2024-09-21T12:38:07.163439+0000","last_became_peered":"2024-09-21T12:38:07.163439+0000","last_unstale":"2024-09-21T12:40:15.226585+0000","last_undegraded":"2024-09-21T12:40:15.226585+0000","last_fullsized":"2024-09-21T12:40:15.226585+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:48.953635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:48.953635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:17.502212+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":449,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.702596+0000","last_change":"2024-09-21T12:40:01.429852+0000","last_active":"2024-09-21T12:40:15.702596+0000","last_peered":"2024-09-21T12:40:15.702596+0000","last_clean":"2024-09-21T12:40:15.702596+0000","last_became_active":"2024-09-21T12:40:01.428162+0000","last_became_peered":"2024-09-21T12:40:01.428162+0000","last_unstale":"2024-09-21T12:40:15.702596+0000","last_undegraded":"2024-09-21T12:40:15.702596+0000","last_fullsized":"2024-09-21T12:40:15.702596+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:21.815656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:21.815656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:47.004063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":976,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:19.539373+0000","last_change":"2024-09-21T12:40:19.539373+0000","last_active":"2024-09-21T12:40:19.539373+0000","last_peered":"2024-09-21T12:40:19.539373+0000","last_clean":"2024-09-21T12:40:19.539373+0000","last_became_active":"2024-09-21T12:38:07.158992+0000","last_became_peered":"2024-09-21T12:38:07.158992+0000","last_unstale":"2024-09-21T12:40:19.539373+0000","last_undegraded":"2024-09-21T12:40:19.539373+0000","last_fullsized":"2024-09-21T12:40:19.539373+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:19.539305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:40:19.539305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:38.716869+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"76'79","reported_seq":413,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.701238+0000","last_change":"2024-09-21T12:39:30.224719+0000","last_active":"2024-09-21T12:40:15.701238+0000","last_peered":"2024-09-21T12:40:15.701238+0000","last_clean":"2024-09-21T12:40:15.701238+0000","last_became_active":"2024-09-21T12:38:07.162110+0000","last_became_peered":"2024-09-21T12:38:07.162110+0000","last_unstale":"2024-09-21T12:40:15.701238+0000","last_undegraded":"2024-09-21T12:40:15.701238+0000","last_fullsized":"2024-09-21T12:40:15.701238+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:30.224638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:30.224638+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:47.399797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"518'54","reported_seq":1306,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:25.602625+0000","last_change":"2024-09-21T12:39:39.041771+0000","last_active":"2024-09-21T12:40:25.602625+0000","last_peered":"2024-09-21T12:40:25.602625+0000","last_clean":"2024-09-21T12:40:25.602625+0000","last_became_active":"2024-09-21T12:38:07.164073+0000","last_became_peered":"2024-09-21T12:38:07.164073+0000","last_unstale":"2024-09-21T12:40:25.602625+0000","last_undegraded":"2024-09-21T12:40:25.602625+0000","last_fullsized":"2024-09-21T12:40:25.602625+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'42","last_scrub_stamp":"2024-09-21T12:39:39.041700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:39.041700+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:45.259714+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":374,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.222550+0000","last_change":"2024-09-21T12:39:29.418691+0000","last_active":"2024-09-21T12:40:15.222550+0000","last_peered":"2024-09-21T12:40:15.222550+0000","last_clean":"2024-09-21T12:40:15.222550+0000","last_became_active":"2024-09-21T12:32:38.426309+0000","last_became_peered":"2024-09-21T12:32:38.426309+0000","last_unstale":"2024-09-21T12:40:15.222550+0000","last_undegraded":"2024-09-21T12:40:15.222550+0000","last_fullsized":"2024-09-21T12:40:15.222550+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:29.418625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:29.418625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:51.060636+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"61'2","reported_seq":362,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.701664+0000","last_change":"2024-09-21T12:39:57.394717+0000","last_active":"2024-09-21T12:40:15.701664+0000","last_peered":"2024-09-21T12:40:15.701664+0000","last_clean":"2024-09-21T12:40:15.701664+0000","last_became_active":"2024-09-21T12:39:57.394373+0000","last_became_peered":"2024-09-21T12:39:57.394373+0000","last_unstale":"2024-09-21T12:40:15.701664+0000","last_undegraded":"2024-09-21T12:40:15.701664+0000","last_fullsized":"2024-09-21T12:40:15.701664+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T12:39:37.041444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:39:37.041444+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:01.747850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0011011149999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1015,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:21.966066+0000","last_change":"2024-09-21T12:40:21.966066+0000","last_active":"2024-09-21T12:40:21.966066+0000","last_peered":"2024-09-21T12:40:21.966066+0000","last_clean":"2024-09-21T12:40:21.966066+0000","last_became_active":"2024-09-21T12:32:38.424402+0000","last_became_peered":"2024-09-21T12:32:38.424402+0000","last_unstale":"2024-09-21T12:40:21.966066+0000","last_undegraded":"2024-09-21T12:40:21.966066+0000","last_fullsized":"2024-09-21T12:40:21.966066+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:21.966001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:40:21.966001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:50.192762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1013,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.528573+0000","last_change":"2024-09-21T12:39:42.489044+0000","last_active":"2024-09-21T12:40:15.528573+0000","last_peered":"2024-09-21T12:40:15.528573+0000","last_clean":"2024-09-21T12:40:15.528573+0000","last_became_active":"2024-09-21T12:36:00.144526+0000","last_became_peered":"2024-09-21T12:36:00.144526+0000","last_unstale":"2024-09-21T12:40:15.528573+0000","last_undegraded":"2024-09-21T12:40:15.528573+0000","last_fullsized":"2024-09-21T12:40:15.528573+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:42.488986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:42.488986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:55.906511+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"76'79","reported_seq":397,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:26.728419+0000","last_change":"2024-09-21T12:40:26.728419+0000","last_active":"2024-09-21T12:40:26.728419+0000","last_peered":"2024-09-21T12:40:26.728419+0000","last_clean":"2024-09-21T12:40:26.728419+0000","last_became_active":"2024-09-21T12:36:01.131622+0000","last_became_peered":"2024-09-21T12:36:01.131622+0000","last_unstale":"2024-09-21T12:40:26.728419+0000","last_undegraded":"2024-09-21T12:40:26.728419+0000","last_fullsized":"2024-09-21T12:40:26.728419+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:40:26.728360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:40:26.728360+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:33.440314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"518'21","reported_seq":486,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.097162+0000","last_change":"2024-09-21T12:39:32.975957+0000","last_active":"2024-09-21T12:40:23.097162+0000","last_peered":"2024-09-21T12:40:23.097162+0000","last_clean":"2024-09-21T12:40:23.097162+0000","last_became_active":"2024-09-21T12:31:39.057316+0000","last_became_peered":"2024-09-21T12:31:39.057316+0000","last_unstale":"2024-09-21T12:40:23.097162+0000","last_undegraded":"2024-09-21T12:40:23.097162+0000","last_fullsized":"2024-09-21T12:40:23.097162+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'17","last_scrub_stamp":"2024-09-21T12:39:32.975879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:32.975879+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:54.712182+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1013,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.223964+0000","last_change":"2024-09-21T12:40:02.507166+0000","last_active":"2024-09-21T12:40:15.223964+0000","last_peered":"2024-09-21T12:40:15.223964+0000","last_clean":"2024-09-21T12:40:15.223964+0000","last_became_active":"2024-09-21T12:38:07.154643+0000","last_became_peered":"2024-09-21T12:38:07.154643+0000","last_unstale":"2024-09-21T12:40:15.223964+0000","last_undegraded":"2024-09-21T12:40:15.223964+0000","last_fullsized":"2024-09-21T12:40:15.223964+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:02.507122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:40:02.507122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:20.157120+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"59'1","reported_seq":730,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.626461+0000","last_change":"2024-09-21T12:40:10.205849+0000","last_active":"2024-09-21T12:40:15.626461+0000","last_peered":"2024-09-21T12:40:15.626461+0000","last_clean":"2024-09-21T12:40:15.626461+0000","last_became_active":"2024-09-21T12:40:10.205183+0000","last_became_peered":"2024-09-21T12:40:10.205183+0000","last_unstale":"2024-09-21T12:40:15.626461+0000","last_undegraded":"2024-09-21T12:40:15.626461+0000","last_fullsized":"2024-09-21T12:40:15.626461+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:40:06.454218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:40:06.454218+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:22.969232+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0019042449999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":721,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.626672+0000","last_change":"2024-09-21T12:40:10.207972+0000","last_active":"2024-09-21T12:40:15.626672+0000","last_peered":"2024-09-21T12:40:15.626672+0000","last_clean":"2024-09-21T12:40:15.626672+0000","last_became_active":"2024-09-21T12:40:10.207682+0000","last_became_peered":"2024-09-21T12:40:10.207682+0000","last_unstale":"2024-09-21T12:40:15.626672+0000","last_undegraded":"2024-09-21T12:40:15.626672+0000","last_fullsized":"2024-09-21T12:40:15.626672+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:31.255364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:31.255364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:47.361606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":88,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.626563+0000","last_change":"2024-09-21T12:40:10.432876+0000","last_active":"2024-09-21T12:40:15.626563+0000","last_peered":"2024-09-21T12:40:15.626563+0000","last_clean":"2024-09-21T12:40:15.626563+0000","last_became_active":"2024-09-21T12:40:10.432646+0000","last_became_peered":"2024-09-21T12:40:10.432646+0000","last_unstale":"2024-09-21T12:40:15.626563+0000","last_undegraded":"2024-09-21T12:40:15.626563+0000","last_fullsized":"2024-09-21T12:40:15.626563+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:56.707155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:56.707155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:21.936398+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"76'79","reported_seq":934,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.224646+0000","last_change":"2024-09-21T12:39:57.434316+0000","last_active":"2024-09-21T12:40:15.224646+0000","last_peered":"2024-09-21T12:40:15.224646+0000","last_clean":"2024-09-21T12:40:15.224646+0000","last_became_active":"2024-09-21T12:32:38.424710+0000","last_became_peered":"2024-09-21T12:32:38.424710+0000","last_unstale":"2024-09-21T12:40:15.224646+0000","last_undegraded":"2024-09-21T12:40:15.224646+0000","last_fullsized":"2024-09-21T12:40:15.224646+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:57.434269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:57.434269+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:09.700806+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"518'36","reported_seq":1040,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.107585+0000","last_change":"2024-09-21T12:40:04.868633+0000","last_active":"2024-09-21T12:40:23.107585+0000","last_peered":"2024-09-21T12:40:23.107585+0000","last_clean":"2024-09-21T12:40:23.107585+0000","last_became_active":"2024-09-21T12:38:07.166839+0000","last_became_peered":"2024-09-21T12:38:07.166839+0000","last_unstale":"2024-09-21T12:40:23.107585+0000","last_undegraded":"2024-09-21T12:40:23.107585+0000","last_fullsized":"2024-09-21T12:40:23.107585+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'24","last_scrub_stamp":"2024-09-21T12:40:04.868559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:40:04.868559+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:06.609155+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1029,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.225646+0000","last_change":"2024-09-21T12:40:09.884209+0000","last_active":"2024-09-21T12:40:15.225646+0000","last_peered":"2024-09-21T12:40:15.225646+0000","last_clean":"2024-09-21T12:40:15.225646+0000","last_became_active":"2024-09-21T12:32:38.411004+0000","last_became_peered":"2024-09-21T12:32:38.411004+0000","last_unstale":"2024-09-21T12:40:15.225646+0000","last_undegraded":"2024-09-21T12:40:15.225646+0000","last_fullsized":"2024-09-21T12:40:15.225646+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:09.884153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:40:09.884153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:12.598597+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":510,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:30.639912+0000","last_change":"2024-09-21T12:40:05.623574+0000","last_active":"2024-09-21T12:40:30.639912+0000","last_peered":"2024-09-21T12:40:30.639912+0000","last_clean":"2024-09-21T12:40:30.639912+0000","last_became_active":"2024-09-21T12:36:00.144053+0000","last_became_peered":"2024-09-21T12:36:00.144053+0000","last_unstale":"2024-09-21T12:40:30.639912+0000","last_undegraded":"2024-09-21T12:40:30.639912+0000","last_fullsized":"2024-09-21T12:40:30.639912+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-21T12:40:05.623495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:13.272075+0000","last_clean_scrub_stamp":"2024-09-21T12:40:05.623495+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:34.144320+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00083785000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":278,"num_read_kb":273,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":415,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.703266+0000","last_change":"2024-09-21T12:38:54.293655+0000","last_active":"2024-09-21T12:40:15.703266+0000","last_peered":"2024-09-21T12:40:15.703266+0000","last_clean":"2024-09-21T12:40:15.703266+0000","last_became_active":"2024-09-21T12:32:02.030240+0000","last_became_peered":"2024-09-21T12:32:02.030240+0000","last_unstale":"2024-09-21T12:40:15.703266+0000","last_undegraded":"2024-09-21T12:40:15.703266+0000","last_fullsized":"2024-09-21T12:40:15.703266+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:54.293611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:54.293611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:20.703408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":468,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.224530+0000","last_change":"2024-09-21T12:39:27.990436+0000","last_active":"2024-09-21T12:40:15.224530+0000","last_peered":"2024-09-21T12:40:15.224530+0000","last_clean":"2024-09-21T12:40:15.224530+0000","last_became_active":"2024-09-21T12:31:39.060160+0000","last_became_peered":"2024-09-21T12:31:39.060160+0000","last_unstale":"2024-09-21T12:40:15.224530+0000","last_undegraded":"2024-09-21T12:40:15.224530+0000","last_fullsized":"2024-09-21T12:40:15.224530+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:27.990369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:27.990369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:57.401228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"518'48","reported_seq":197,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:25.602758+0000","last_change":"2024-09-21T12:40:08.959291+0000","last_active":"2024-09-21T12:40:25.602758+0000","last_peered":"2024-09-21T12:40:25.602758+0000","last_clean":"2024-09-21T12:40:25.602758+0000","last_became_active":"2024-09-21T12:40:01.426627+0000","last_became_peered":"2024-09-21T12:40:01.426627+0000","last_unstale":"2024-09-21T12:40:25.602758+0000","last_undegraded":"2024-09-21T12:40:25.602758+0000","last_fullsized":"2024-09-21T12:40:25.602758+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"510'36","last_scrub_stamp":"2024-09-21T12:40:08.959223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:40:08.959223+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:09.101029+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"59'1","reported_seq":989,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.527983+0000","last_change":"2024-09-21T12:39:46.503605+0000","last_active":"2024-09-21T12:40:15.527983+0000","last_peered":"2024-09-21T12:40:15.527983+0000","last_clean":"2024-09-21T12:40:15.527983+0000","last_became_active":"2024-09-21T12:20:15.298815+0000","last_became_peered":"2024-09-21T12:20:15.298815+0000","last_unstale":"2024-09-21T12:40:15.527983+0000","last_undegraded":"2024-09-21T12:40:15.527983+0000","last_fullsized":"2024-09-21T12:40:15.527983+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:39:46.503556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:46.503556+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:47.330227+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'79","reported_seq":242,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:30.741404+0000","last_change":"2024-09-21T12:40:30.741404+0000","last_active":"2024-09-21T12:40:30.741404+0000","last_peered":"2024-09-21T12:40:30.741404+0000","last_clean":"2024-09-21T12:40:30.741404+0000","last_became_active":"2024-09-21T12:36:00.134878+0000","last_became_peered":"2024-09-21T12:36:00.134878+0000","last_unstale":"2024-09-21T12:40:30.741404+0000","last_undegraded":"2024-09-21T12:40:30.741404+0000","last_fullsized":"2024-09-21T12:40:30.741404+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:40:30.741347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:40:30.741347+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:32.440076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":400,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.529342+0000","last_change":"2024-09-21T12:39:35.396535+0000","last_active":"2024-09-21T12:40:15.529342+0000","last_peered":"2024-09-21T12:40:15.529342+0000","last_clean":"2024-09-21T12:40:15.529342+0000","last_became_active":"2024-09-21T12:38:08.164554+0000","last_became_peered":"2024-09-21T12:38:08.164554+0000","last_unstale":"2024-09-21T12:40:15.529342+0000","last_undegraded":"2024-09-21T12:40:15.529342+0000","last_fullsized":"2024-09-21T12:40:15.529342+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:35.396482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:35.396482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:01.813844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":640,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.703494+0000","last_change":"2024-09-21T12:39:36.225811+0000","last_active":"2024-09-21T12:40:15.703494+0000","last_peered":"2024-09-21T12:40:15.703494+0000","last_clean":"2024-09-21T12:40:15.703494+0000","last_became_active":"2024-09-21T12:37:20.485126+0000","last_became_peered":"2024-09-21T12:37:20.485126+0000","last_unstale":"2024-09-21T12:40:15.703494+0000","last_undegraded":"2024-09-21T12:40:15.703494+0000","last_fullsized":"2024-09-21T12:40:15.703494+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:36.225755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:36.225755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:43.496265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"76'79","reported_seq":931,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:25.526647+0000","last_change":"2024-09-21T12:40:25.526647+0000","last_active":"2024-09-21T12:40:25.526647+0000","last_peered":"2024-09-21T12:40:25.526647+0000","last_clean":"2024-09-21T12:40:25.526647+0000","last_became_active":"2024-09-21T12:27:14.954351+0000","last_became_peered":"2024-09-21T12:27:14.954351+0000","last_unstale":"2024-09-21T12:40:25.526647+0000","last_undegraded":"2024-09-21T12:40:25.526647+0000","last_fullsized":"2024-09-21T12:40:25.526647+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:40:25.526597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:40:25.526597+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:50.760074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"518'27","reported_seq":398,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:28.472915+0000","last_change":"2024-09-21T12:40:28.472915+0000","last_active":"2024-09-21T12:40:28.472915+0000","last_peered":"2024-09-21T12:40:28.472915+0000","last_clean":"2024-09-21T12:40:28.472915+0000","last_became_active":"2024-09-21T12:32:38.419714+0000","last_became_peered":"2024-09-21T12:32:38.419714+0000","last_unstale":"2024-09-21T12:40:28.472915+0000","last_undegraded":"2024-09-21T12:40:28.472915+0000","last_fullsized":"2024-09-21T12:40:28.472915+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"518'27","last_scrub_stamp":"2024-09-21T12:40:28.472846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:40:28.472846+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:42.813306+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":388,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.221832+0000","last_change":"2024-09-21T12:38:08.148418+0000","last_active":"2024-09-21T12:40:15.221832+0000","last_peered":"2024-09-21T12:40:15.221832+0000","last_clean":"2024-09-21T12:40:15.221832+0000","last_became_active":"2024-09-21T12:38:08.148109+0000","last_became_peered":"2024-09-21T12:38:08.148109+0000","last_unstale":"2024-09-21T12:40:15.221832+0000","last_undegraded":"2024-09-21T12:40:15.221832+0000","last_fullsized":"2024-09-21T12:40:15.221832+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:36:40.708469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:36:40.708469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:15.280609+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":519,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.703603+0000","last_change":"2024-09-21T12:39:51.277137+0000","last_active":"2024-09-21T12:40:15.703603+0000","last_peered":"2024-09-21T12:40:15.703603+0000","last_clean":"2024-09-21T12:40:15.703603+0000","last_became_active":"2024-09-21T12:37:20.467986+0000","last_became_peered":"2024-09-21T12:37:20.467986+0000","last_unstale":"2024-09-21T12:40:15.703603+0000","last_undegraded":"2024-09-21T12:40:15.703603+0000","last_fullsized":"2024-09-21T12:40:15.703603+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:51.277082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:51.277082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:20.277970+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":962,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.528271+0000","last_change":"2024-09-21T12:40:14.522626+0000","last_active":"2024-09-21T12:40:15.528271+0000","last_peered":"2024-09-21T12:40:15.528271+0000","last_clean":"2024-09-21T12:40:15.528271+0000","last_became_active":"2024-09-21T12:20:21.592812+0000","last_became_peered":"2024-09-21T12:20:21.592812+0000","last_unstale":"2024-09-21T12:40:15.528271+0000","last_undegraded":"2024-09-21T12:40:15.528271+0000","last_fullsized":"2024-09-21T12:40:15.528271+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:14.522582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:40:14.522582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:26.810882+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"76'79","reported_seq":322,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.701150+0000","last_change":"2024-09-21T12:40:00.283573+0000","last_active":"2024-09-21T12:40:15.701150+0000","last_peered":"2024-09-21T12:40:15.701150+0000","last_clean":"2024-09-21T12:40:15.701150+0000","last_became_active":"2024-09-21T12:39:57.394768+0000","last_became_peered":"2024-09-21T12:39:57.394768+0000","last_unstale":"2024-09-21T12:40:15.701150+0000","last_undegraded":"2024-09-21T12:40:15.701150+0000","last_fullsized":"2024-09-21T12:40:15.701150+0000","mapping_epoch":506,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:40:00.283488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:40:00.283488+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:13.411294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"518'45","reported_seq":416,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.157284+0000","last_change":"2024-09-21T12:38:53.268874+0000","last_active":"2024-09-21T12:40:23.157284+0000","last_peered":"2024-09-21T12:40:23.157284+0000","last_clean":"2024-09-21T12:40:23.157284+0000","last_became_active":"2024-09-21T12:38:07.163226+0000","last_became_peered":"2024-09-21T12:38:07.163226+0000","last_unstale":"2024-09-21T12:40:23.157284+0000","last_undegraded":"2024-09-21T12:40:23.157284+0000","last_fullsized":"2024-09-21T12:40:23.157284+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'31","last_scrub_stamp":"2024-09-21T12:38:53.268744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:53.268744+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":486,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.702782+0000","last_change":"2024-09-21T12:40:01.422904+0000","last_active":"2024-09-21T12:40:15.702782+0000","last_peered":"2024-09-21T12:40:15.702782+0000","last_clean":"2024-09-21T12:40:15.702782+0000","last_became_active":"2024-09-21T12:40:01.422156+0000","last_became_peered":"2024-09-21T12:40:01.422156+0000","last_unstale":"2024-09-21T12:40:15.702782+0000","last_undegraded":"2024-09-21T12:40:15.702782+0000","last_fullsized":"2024-09-21T12:40:15.702782+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:31.453523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:31.453523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:41.618032+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":996,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.224619+0000","last_change":"2024-09-21T12:40:06.448316+0000","last_active":"2024-09-21T12:40:15.224619+0000","last_peered":"2024-09-21T12:40:15.224619+0000","last_clean":"2024-09-21T12:40:15.224619+0000","last_became_active":"2024-09-21T12:32:38.425177+0000","last_became_peered":"2024-09-21T12:32:38.425177+0000","last_unstale":"2024-09-21T12:40:15.224619+0000","last_undegraded":"2024-09-21T12:40:15.224619+0000","last_fullsized":"2024-09-21T12:40:15.224619+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:06.448265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:40:06.448265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:13.299279+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":794,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.626981+0000","last_change":"2024-09-21T12:40:10.206824+0000","last_active":"2024-09-21T12:40:15.626981+0000","last_peered":"2024-09-21T12:40:15.626981+0000","last_clean":"2024-09-21T12:40:15.626981+0000","last_became_active":"2024-09-21T12:40:10.206622+0000","last_became_peered":"2024-09-21T12:40:10.206622+0000","last_unstale":"2024-09-21T12:40:15.626981+0000","last_undegraded":"2024-09-21T12:40:15.626981+0000","last_fullsized":"2024-09-21T12:40:15.626981+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:02.515408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:40:02.515408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:24.884220+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"76'79","reported_seq":1052,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:28.660249+0000","last_change":"2024-09-21T12:40:28.660249+0000","last_active":"2024-09-21T12:40:28.660249+0000","last_peered":"2024-09-21T12:40:28.660249+0000","last_clean":"2024-09-21T12:40:28.660249+0000","last_became_active":"2024-09-21T12:38:08.156755+0000","last_became_peered":"2024-09-21T12:38:08.156755+0000","last_unstale":"2024-09-21T12:40:28.660249+0000","last_undegraded":"2024-09-21T12:40:28.660249+0000","last_fullsized":"2024-09-21T12:40:28.660249+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:40:28.660176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:40:28.660176+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:35.645247+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"518'30","reported_seq":396,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.117843+0000","last_change":"2024-09-21T12:38:46.208578+0000","last_active":"2024-09-21T12:40:23.117843+0000","last_peered":"2024-09-21T12:40:23.117843+0000","last_clean":"2024-09-21T12:40:23.117843+0000","last_became_active":"2024-09-21T12:38:07.162180+0000","last_became_peered":"2024-09-21T12:38:07.162180+0000","last_unstale":"2024-09-21T12:40:23.117843+0000","last_undegraded":"2024-09-21T12:40:23.117843+0000","last_fullsized":"2024-09-21T12:40:23.117843+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:38:46.208493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:38:46.208493+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1003,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.528190+0000","last_change":"2024-09-21T12:40:04.488713+0000","last_active":"2024-09-21T12:40:15.528190+0000","last_peered":"2024-09-21T12:40:15.528190+0000","last_clean":"2024-09-21T12:40:15.528190+0000","last_became_active":"2024-09-21T12:32:38.403997+0000","last_became_peered":"2024-09-21T12:32:38.403997+0000","last_unstale":"2024-09-21T12:40:15.528190+0000","last_undegraded":"2024-09-21T12:40:15.528190+0000","last_fullsized":"2024-09-21T12:40:15.528190+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:04.488665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:40:04.488665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:27.871575+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"516'17","reported_seq":909,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:20.602532+0000","last_change":"2024-09-21T12:40:11.697861+0000","last_active":"2024-09-21T12:40:20.602532+0000","last_peered":"2024-09-21T12:40:20.602532+0000","last_clean":"2024-09-21T12:40:20.602532+0000","last_became_active":"2024-09-21T12:40:10.207745+0000","last_became_peered":"2024-09-21T12:40:10.207745+0000","last_unstale":"2024-09-21T12:40:20.602532+0000","last_undegraded":"2024-09-21T12:40:20.602532+0000","last_fullsized":"2024-09-21T12:40:20.602532+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'17","last_scrub_stamp":"2024-09-21T12:40:11.697810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:40:11.697810+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:23.355909+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":377,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:21.558597+0000","last_change":"2024-09-21T12:40:21.558597+0000","last_active":"2024-09-21T12:40:21.558597+0000","last_peered":"2024-09-21T12:40:21.558597+0000","last_clean":"2024-09-21T12:40:21.558597+0000","last_became_active":"2024-09-21T12:38:07.163242+0000","last_became_peered":"2024-09-21T12:38:07.163242+0000","last_unstale":"2024-09-21T12:40:21.558597+0000","last_undegraded":"2024-09-21T12:40:21.558597+0000","last_fullsized":"2024-09-21T12:40:21.558597+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:21.558535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:40:21.558535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:49.676545+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.e","version":"76'79","reported_seq":386,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.222842+0000","last_change":"2024-09-21T12:39:27.391527+0000","last_active":"2024-09-21T12:40:15.222842+0000","last_peered":"2024-09-21T12:40:15.222842+0000","last_clean":"2024-09-21T12:40:15.222842+0000","last_became_active":"2024-09-21T12:36:01.159481+0000","last_became_peered":"2024-09-21T12:36:01.159481+0000","last_unstale":"2024-09-21T12:40:15.222842+0000","last_undegraded":"2024-09-21T12:40:15.222842+0000","last_fullsized":"2024-09-21T12:40:15.222842+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:27.391477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:27.391477+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:55.811814+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"518'21","reported_seq":1101,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.142402+0000","last_change":"2024-09-21T12:39:27.747950+0000","last_active":"2024-09-21T12:40:23.142402+0000","last_peered":"2024-09-21T12:40:23.142402+0000","last_clean":"2024-09-21T12:40:23.142402+0000","last_became_active":"2024-09-21T12:32:38.424376+0000","last_became_peered":"2024-09-21T12:32:38.424376+0000","last_unstale":"2024-09-21T12:40:23.142402+0000","last_undegraded":"2024-09-21T12:40:23.142402+0000","last_fullsized":"2024-09-21T12:40:23.142402+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:39:27.747889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:27.747889+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:54.802430+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":506,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.703006+0000","last_change":"2024-09-21T12:38:51.327197+0000","last_active":"2024-09-21T12:40:15.703006+0000","last_peered":"2024-09-21T12:40:15.703006+0000","last_clean":"2024-09-21T12:40:15.703006+0000","last_became_active":"2024-09-21T12:37:20.484773+0000","last_became_peered":"2024-09-21T12:37:20.484773+0000","last_unstale":"2024-09-21T12:40:15.703006+0000","last_undegraded":"2024-09-21T12:40:15.703006+0000","last_fullsized":"2024-09-21T12:40:15.703006+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:38:51.327127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:38:51.327127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":822,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.627670+0000","last_change":"2024-09-21T12:40:15.627670+0000","last_active":"2024-09-21T12:40:15.627670+0000","last_peered":"2024-09-21T12:40:15.627670+0000","last_clean":"2024-09-21T12:40:15.627670+0000","last_became_active":"2024-09-21T12:40:10.206786+0000","last_became_peered":"2024-09-21T12:40:10.206786+0000","last_unstale":"2024-09-21T12:40:15.627670+0000","last_undegraded":"2024-09-21T12:40:15.627670+0000","last_fullsized":"2024-09-21T12:40:15.627670+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:15.627628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:40:15.627628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:41.093031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":989,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.225599+0000","last_change":"2024-09-21T12:39:34.662753+0000","last_active":"2024-09-21T12:40:15.225599+0000","last_peered":"2024-09-21T12:40:15.225599+0000","last_clean":"2024-09-21T12:40:15.225599+0000","last_became_active":"2024-09-21T12:38:08.164626+0000","last_became_peered":"2024-09-21T12:38:08.164626+0000","last_unstale":"2024-09-21T12:40:15.225599+0000","last_undegraded":"2024-09-21T12:40:15.225599+0000","last_fullsized":"2024-09-21T12:40:15.225599+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:34.662706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:34.662706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:39.901409+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"76'79","reported_seq":474,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:21.898417+0000","last_change":"2024-09-21T12:40:21.898417+0000","last_active":"2024-09-21T12:40:21.898417+0000","last_peered":"2024-09-21T12:40:21.898417+0000","last_clean":"2024-09-21T12:40:21.898417+0000","last_became_active":"2024-09-21T12:40:01.425715+0000","last_became_peered":"2024-09-21T12:40:01.425715+0000","last_unstale":"2024-09-21T12:40:21.898417+0000","last_undegraded":"2024-09-21T12:40:21.898417+0000","last_fullsized":"2024-09-21T12:40:21.898417+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:40:21.898359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:40:21.898359+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:30.363313+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"518'51","reported_seq":1183,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.177322+0000","last_change":"2024-09-21T12:39:48.101801+0000","last_active":"2024-09-21T12:40:23.177322+0000","last_peered":"2024-09-21T12:40:23.177322+0000","last_clean":"2024-09-21T12:40:23.177322+0000","last_became_active":"2024-09-21T12:36:00.148392+0000","last_became_peered":"2024-09-21T12:36:00.148392+0000","last_unstale":"2024-09-21T12:40:23.177322+0000","last_undegraded":"2024-09-21T12:40:23.177322+0000","last_fullsized":"2024-09-21T12:40:23.177322+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'35","last_scrub_stamp":"2024-09-21T12:39:48.101688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:48.101688+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:13.498714+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":234,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:19.630007+0000","last_change":"2024-09-21T12:40:19.629930+0000","last_active":"2024-09-21T12:40:19.630007+0000","last_peered":"2024-09-21T12:40:19.630007+0000","last_clean":"2024-09-21T12:40:19.630007+0000","last_became_active":"2024-09-21T12:36:00.145287+0000","last_became_peered":"2024-09-21T12:36:00.145287+0000","last_unstale":"2024-09-21T12:40:19.630007+0000","last_undegraded":"2024-09-21T12:40:19.630007+0000","last_fullsized":"2024-09-21T12:40:19.630007+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:19.629864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:40:19.629864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:48.903131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":752,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.626481+0000","last_change":"2024-09-21T12:40:10.205161+0000","last_active":"2024-09-21T12:40:15.626481+0000","last_peered":"2024-09-21T12:40:15.626481+0000","last_clean":"2024-09-21T12:40:15.626481+0000","last_became_active":"2024-09-21T12:40:10.204864+0000","last_became_peered":"2024-09-21T12:40:10.204864+0000","last_unstale":"2024-09-21T12:40:15.626481+0000","last_undegraded":"2024-09-21T12:40:15.626481+0000","last_fullsized":"2024-09-21T12:40:15.626481+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:06.884082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:40:06.884082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:27.693979+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":966,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.529749+0000","last_change":"2024-09-21T12:39:46.420589+0000","last_active":"2024-09-21T12:40:15.529749+0000","last_peered":"2024-09-21T12:40:15.529749+0000","last_clean":"2024-09-21T12:40:15.529749+0000","last_became_active":"2024-09-21T12:38:08.164339+0000","last_became_peered":"2024-09-21T12:38:08.164339+0000","last_unstale":"2024-09-21T12:40:15.529749+0000","last_undegraded":"2024-09-21T12:40:15.529749+0000","last_fullsized":"2024-09-21T12:40:15.529749+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:46.420527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:46.420527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:12.973409+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"76'79","reported_seq":929,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.528421+0000","last_change":"2024-09-21T12:39:28.377091+0000","last_active":"2024-09-21T12:40:15.528421+0000","last_peered":"2024-09-21T12:40:15.528421+0000","last_clean":"2024-09-21T12:40:15.528421+0000","last_became_active":"2024-09-21T12:27:14.951375+0000","last_became_peered":"2024-09-21T12:27:14.951375+0000","last_unstale":"2024-09-21T12:40:15.528421+0000","last_undegraded":"2024-09-21T12:40:15.528421+0000","last_fullsized":"2024-09-21T12:40:15.528421+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:28.377033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:28.377033+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:43.162251+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"518'24","reported_seq":395,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:22.995204+0000","last_change":"2024-09-21T12:39:22.328834+0000","last_active":"2024-09-21T12:40:22.995204+0000","last_peered":"2024-09-21T12:40:22.995204+0000","last_clean":"2024-09-21T12:40:22.995204+0000","last_became_active":"2024-09-21T12:32:38.421609+0000","last_became_peered":"2024-09-21T12:32:38.421609+0000","last_unstale":"2024-09-21T12:40:22.995204+0000","last_undegraded":"2024-09-21T12:40:22.995204+0000","last_fullsized":"2024-09-21T12:40:22.995204+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'18","last_scrub_stamp":"2024-09-21T12:39:22.328772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:22.328772+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:27.373604+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":234,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.708307+0000","last_change":"2024-09-21T12:40:15.708258+0000","last_active":"2024-09-21T12:40:15.708307+0000","last_peered":"2024-09-21T12:40:15.708307+0000","last_clean":"2024-09-21T12:40:15.708307+0000","last_became_active":"2024-09-21T12:36:00.150832+0000","last_became_peered":"2024-09-21T12:36:00.150832+0000","last_unstale":"2024-09-21T12:40:15.708307+0000","last_undegraded":"2024-09-21T12:40:15.708307+0000","last_fullsized":"2024-09-21T12:40:15.708307+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:15.708200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:40:15.708200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:18.215154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":721,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:19.627051+0000","last_change":"2024-09-21T12:40:19.627051+0000","last_active":"2024-09-21T12:40:19.627051+0000","last_peered":"2024-09-21T12:40:19.627051+0000","last_clean":"2024-09-21T12:40:19.627051+0000","last_became_active":"2024-09-21T12:40:10.202413+0000","last_became_peered":"2024-09-21T12:40:10.202413+0000","last_unstale":"2024-09-21T12:40:19.627051+0000","last_undegraded":"2024-09-21T12:40:19.627051+0000","last_fullsized":"2024-09-21T12:40:19.627051+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:19.626989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:40:19.626989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:31.397975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":998,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.527949+0000","last_change":"2024-09-21T12:39:52.544099+0000","last_active":"2024-09-21T12:40:15.527949+0000","last_peered":"2024-09-21T12:40:15.527949+0000","last_clean":"2024-09-21T12:40:15.527949+0000","last_became_active":"2024-09-21T12:36:00.139300+0000","last_became_peered":"2024-09-21T12:36:00.139300+0000","last_unstale":"2024-09-21T12:40:15.527949+0000","last_undegraded":"2024-09-21T12:40:15.527949+0000","last_fullsized":"2024-09-21T12:40:15.527949+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:52.544055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:52.544055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:21.056635+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"76'79","reported_seq":405,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.530501+0000","last_change":"2024-09-21T12:40:11.657863+0000","last_active":"2024-09-21T12:40:15.530501+0000","last_peered":"2024-09-21T12:40:15.530501+0000","last_clean":"2024-09-21T12:40:15.530501+0000","last_became_active":"2024-09-21T12:36:01.141807+0000","last_became_peered":"2024-09-21T12:36:01.141807+0000","last_unstale":"2024-09-21T12:40:15.530501+0000","last_undegraded":"2024-09-21T12:40:15.530501+0000","last_fullsized":"2024-09-21T12:40:15.530501+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:40:11.657791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:40:11.657791+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:36.142388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"518'39","reported_seq":534,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.115627+0000","last_change":"2024-09-21T12:39:50.287772+0000","last_active":"2024-09-21T12:40:23.115627+0000","last_peered":"2024-09-21T12:40:23.115627+0000","last_clean":"2024-09-21T12:40:23.115627+0000","last_became_active":"2024-09-21T12:36:00.137597+0000","last_became_peered":"2024-09-21T12:36:00.137597+0000","last_unstale":"2024-09-21T12:40:23.115627+0000","last_undegraded":"2024-09-21T12:40:23.115627+0000","last_fullsized":"2024-09-21T12:40:23.115627+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'27","last_scrub_stamp":"2024-09-21T12:39:50.287686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:50.287686+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:19.547933+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"59'2","reported_seq":240,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.530840+0000","last_change":"2024-09-21T12:39:29.483527+0000","last_active":"2024-09-21T12:40:15.530840+0000","last_peered":"2024-09-21T12:40:15.530840+0000","last_clean":"2024-09-21T12:40:15.530840+0000","last_became_active":"2024-09-21T12:36:00.148899+0000","last_became_peered":"2024-09-21T12:36:00.148899+0000","last_unstale":"2024-09-21T12:40:15.530840+0000","last_undegraded":"2024-09-21T12:40:15.530840+0000","last_fullsized":"2024-09-21T12:40:15.530840+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-21T12:39:29.483457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:29.483457+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:55.417541+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"415'16","reported_seq":293,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:25.602654+0000","last_change":"2024-09-21T12:39:56.532997+0000","last_active":"2024-09-21T12:40:25.602654+0000","last_peered":"2024-09-21T12:40:25.602654+0000","last_clean":"2024-09-21T12:40:25.602654+0000","last_became_active":"2024-09-21T12:36:00.143091+0000","last_became_peered":"2024-09-21T12:36:00.143091+0000","last_unstale":"2024-09-21T12:40:25.602654+0000","last_undegraded":"2024-09-21T12:40:25.602654+0000","last_fullsized":"2024-09-21T12:40:25.602654+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'16","last_scrub_stamp":"2024-09-21T12:39:56.532873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:56.532873+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:04.686343+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":478,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.702130+0000","last_change":"2024-09-21T12:40:09.147282+0000","last_active":"2024-09-21T12:40:15.702130+0000","last_peered":"2024-09-21T12:40:15.702130+0000","last_clean":"2024-09-21T12:40:15.702130+0000","last_became_active":"2024-09-21T12:37:20.468350+0000","last_became_peered":"2024-09-21T12:37:20.468350+0000","last_unstale":"2024-09-21T12:40:15.702130+0000","last_undegraded":"2024-09-21T12:40:15.702130+0000","last_fullsized":"2024-09-21T12:40:15.702130+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:09.147222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:40:09.147222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:25.515342+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"76'79","reported_seq":705,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.701894+0000","last_change":"2024-09-21T12:39:16.243982+0000","last_active":"2024-09-21T12:40:15.701894+0000","last_peered":"2024-09-21T12:40:15.701894+0000","last_clean":"2024-09-21T12:40:15.701894+0000","last_became_active":"2024-09-21T12:31:39.060422+0000","last_became_peered":"2024-09-21T12:31:39.060422+0000","last_unstale":"2024-09-21T12:40:15.701894+0000","last_undegraded":"2024-09-21T12:40:15.701894+0000","last_fullsized":"2024-09-21T12:40:15.701894+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:16.243927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:16.243927+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:40.547837+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1029,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.701714+0000","last_change":"2024-09-21T12:39:19.481714+0000","last_active":"2024-09-21T12:40:15.701714+0000","last_peered":"2024-09-21T12:40:15.701714+0000","last_clean":"2024-09-21T12:40:15.701714+0000","last_became_active":"2024-09-21T12:32:02.026693+0000","last_became_peered":"2024-09-21T12:32:02.026693+0000","last_unstale":"2024-09-21T12:40:15.701714+0000","last_undegraded":"2024-09-21T12:40:15.701714+0000","last_fullsized":"2024-09-21T12:40:15.701714+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:19.481652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:19.481652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:34.846777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"518'18","reported_seq":494,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.130981+0000","last_change":"2024-09-21T12:39:44.517980+0000","last_active":"2024-09-21T12:40:23.130981+0000","last_peered":"2024-09-21T12:40:23.130981+0000","last_clean":"2024-09-21T12:40:23.130981+0000","last_became_active":"2024-09-21T12:36:01.138626+0000","last_became_peered":"2024-09-21T12:36:01.138626+0000","last_unstale":"2024-09-21T12:40:23.130981+0000","last_undegraded":"2024-09-21T12:40:23.130981+0000","last_fullsized":"2024-09-21T12:40:23.130981+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'12","last_scrub_stamp":"2024-09-21T12:39:44.517927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:44.517927+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:14.117178+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":498,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.703205+0000","last_change":"2024-09-21T12:40:02.297488+0000","last_active":"2024-09-21T12:40:15.703205+0000","last_peered":"2024-09-21T12:40:15.703205+0000","last_clean":"2024-09-21T12:40:15.703205+0000","last_became_active":"2024-09-21T12:37:20.485806+0000","last_became_peered":"2024-09-21T12:37:20.485806+0000","last_unstale":"2024-09-21T12:40:15.703205+0000","last_undegraded":"2024-09-21T12:40:15.703205+0000","last_fullsized":"2024-09-21T12:40:15.703205+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:02.297429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:40:02.297429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:04.429618+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":233,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.530187+0000","last_change":"2024-09-21T12:39:53.520465+0000","last_active":"2024-09-21T12:40:15.530187+0000","last_peered":"2024-09-21T12:40:15.530187+0000","last_clean":"2024-09-21T12:40:15.530187+0000","last_became_active":"2024-09-21T12:36:00.149086+0000","last_became_peered":"2024-09-21T12:36:00.149086+0000","last_unstale":"2024-09-21T12:40:15.530187+0000","last_undegraded":"2024-09-21T12:40:15.530187+0000","last_fullsized":"2024-09-21T12:40:15.530187+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:53.520419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:53.520419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:23.000283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"76'79","reported_seq":961,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.224735+0000","last_change":"2024-09-21T12:39:29.959881+0000","last_active":"2024-09-21T12:40:15.224735+0000","last_peered":"2024-09-21T12:40:15.224735+0000","last_clean":"2024-09-21T12:40:15.224735+0000","last_became_active":"2024-09-21T12:32:38.425495+0000","last_became_peered":"2024-09-21T12:32:38.425495+0000","last_unstale":"2024-09-21T12:40:15.224735+0000","last_undegraded":"2024-09-21T12:40:15.224735+0000","last_fullsized":"2024-09-21T12:40:15.224735+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:29.959790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:29.959790+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:37.092161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"518'21","reported_seq":1063,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.008343+0000","last_change":"2024-09-21T12:39:59.759617+0000","last_active":"2024-09-21T12:40:23.008343+0000","last_peered":"2024-09-21T12:40:23.008343+0000","last_clean":"2024-09-21T12:40:23.008343+0000","last_became_active":"2024-09-21T12:36:00.142274+0000","last_became_peered":"2024-09-21T12:36:00.142274+0000","last_unstale":"2024-09-21T12:40:23.008343+0000","last_undegraded":"2024-09-21T12:40:23.008343+0000","last_fullsized":"2024-09-21T12:40:23.008343+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:39:59.759566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:59.759566+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:03.918859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":233,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.529131+0000","last_change":"2024-09-21T12:39:55.510514+0000","last_active":"2024-09-21T12:40:15.529131+0000","last_peered":"2024-09-21T12:40:15.529131+0000","last_clean":"2024-09-21T12:40:15.529131+0000","last_became_active":"2024-09-21T12:36:00.140477+0000","last_became_peered":"2024-09-21T12:36:00.140477+0000","last_unstale":"2024-09-21T12:40:15.529131+0000","last_undegraded":"2024-09-21T12:40:15.529131+0000","last_fullsized":"2024-09-21T12:40:15.529131+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:55.510453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:55.510453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:03.619575+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":980,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.528602+0000","last_change":"2024-09-21T12:40:11.504657+0000","last_active":"2024-09-21T12:40:15.528602+0000","last_peered":"2024-09-21T12:40:15.528602+0000","last_clean":"2024-09-21T12:40:15.528602+0000","last_became_active":"2024-09-21T12:36:00.150975+0000","last_became_peered":"2024-09-21T12:36:00.150975+0000","last_unstale":"2024-09-21T12:40:15.528602+0000","last_undegraded":"2024-09-21T12:40:15.528602+0000","last_fullsized":"2024-09-21T12:40:15.528602+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:11.504601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:40:11.504601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:34.882501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":839,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.626778+0000","last_change":"2024-09-21T12:40:10.207798+0000","last_active":"2024-09-21T12:40:15.626778+0000","last_peered":"2024-09-21T12:40:15.626778+0000","last_clean":"2024-09-21T12:40:15.626778+0000","last_became_active":"2024-09-21T12:40:10.207553+0000","last_became_peered":"2024-09-21T12:40:10.207553+0000","last_unstale":"2024-09-21T12:40:15.626778+0000","last_undegraded":"2024-09-21T12:40:15.626778+0000","last_fullsized":"2024-09-21T12:40:15.626778+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:37.528714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:37.528714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:05.401724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"76'79","reported_seq":203,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.700940+0000","last_change":"2024-09-21T12:39:37.274653+0000","last_active":"2024-09-21T12:40:15.700940+0000","last_peered":"2024-09-21T12:40:15.700940+0000","last_clean":"2024-09-21T12:40:15.700940+0000","last_became_active":"2024-09-21T12:37:20.467540+0000","last_became_peered":"2024-09-21T12:37:20.467540+0000","last_unstale":"2024-09-21T12:40:15.700940+0000","last_undegraded":"2024-09-21T12:40:15.700940+0000","last_fullsized":"2024-09-21T12:40:15.700940+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:37.274594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:37.274594+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:37.391359+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"518'21","reported_seq":547,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:26.156955+0000","last_change":"2024-09-21T12:40:26.156955+0000","last_active":"2024-09-21T12:40:26.156955+0000","last_peered":"2024-09-21T12:40:26.156955+0000","last_clean":"2024-09-21T12:40:26.156955+0000","last_became_active":"2024-09-21T12:38:08.164375+0000","last_became_peered":"2024-09-21T12:38:08.164375+0000","last_unstale":"2024-09-21T12:40:26.156955+0000","last_undegraded":"2024-09-21T12:40:26.156955+0000","last_fullsized":"2024-09-21T12:40:26.156955+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"518'21","last_scrub_stamp":"2024-09-21T12:40:26.156873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:40:26.156873+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:31.934630+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1029,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.225069+0000","last_change":"2024-09-21T12:40:07.879745+0000","last_active":"2024-09-21T12:40:15.225069+0000","last_peered":"2024-09-21T12:40:15.225069+0000","last_clean":"2024-09-21T12:40:15.225069+0000","last_became_active":"2024-09-21T12:36:00.127225+0000","last_became_peered":"2024-09-21T12:36:00.127225+0000","last_unstale":"2024-09-21T12:40:15.225069+0000","last_undegraded":"2024-09-21T12:40:15.225069+0000","last_fullsized":"2024-09-21T12:40:15.225069+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:07.879698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:40:07.879698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:30.244960+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"457'17","reported_seq":582,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:25.602750+0000","last_change":"2024-09-21T12:39:39.398127+0000","last_active":"2024-09-21T12:40:25.602750+0000","last_peered":"2024-09-21T12:40:25.602750+0000","last_clean":"2024-09-21T12:40:25.602750+0000","last_became_active":"2024-09-21T12:38:08.171750+0000","last_became_peered":"2024-09-21T12:38:08.171750+0000","last_unstale":"2024-09-21T12:40:25.602750+0000","last_undegraded":"2024-09-21T12:40:25.602750+0000","last_fullsized":"2024-09-21T12:40:25.602750+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'17","last_scrub_stamp":"2024-09-21T12:39:39.398076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:39.398076+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:50.554325+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":377,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.222984+0000","last_change":"2024-09-21T12:40:00.587195+0000","last_active":"2024-09-21T12:40:15.222984+0000","last_peered":"2024-09-21T12:40:15.222984+0000","last_clean":"2024-09-21T12:40:15.222984+0000","last_became_active":"2024-09-21T12:38:07.163162+0000","last_became_peered":"2024-09-21T12:38:07.163162+0000","last_unstale":"2024-09-21T12:40:15.222984+0000","last_undegraded":"2024-09-21T12:40:15.222984+0000","last_fullsized":"2024-09-21T12:40:15.222984+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:00.587144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:40:00.587144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:25.922211+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"76'79","reported_seq":804,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.225758+0000","last_change":"2024-09-21T12:39:55.721332+0000","last_active":"2024-09-21T12:40:15.225758+0000","last_peered":"2024-09-21T12:40:15.225758+0000","last_clean":"2024-09-21T12:40:15.225758+0000","last_became_active":"2024-09-21T12:32:02.034387+0000","last_became_peered":"2024-09-21T12:32:02.034387+0000","last_unstale":"2024-09-21T12:40:15.225758+0000","last_undegraded":"2024-09-21T12:40:15.225758+0000","last_fullsized":"2024-09-21T12:40:15.225758+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:55.721282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:55.721282+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:00.025864+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"518'30","reported_seq":1074,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.147352+0000","last_change":"2024-09-21T12:39:31.262960+0000","last_active":"2024-09-21T12:40:23.147352+0000","last_peered":"2024-09-21T12:40:23.147352+0000","last_clean":"2024-09-21T12:40:23.147352+0000","last_became_active":"2024-09-21T12:36:00.155033+0000","last_became_peered":"2024-09-21T12:36:00.155033+0000","last_unstale":"2024-09-21T12:40:23.147352+0000","last_undegraded":"2024-09-21T12:40:23.147352+0000","last_fullsized":"2024-09-21T12:40:23.147352+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'20","last_scrub_stamp":"2024-09-21T12:39:31.262908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:31.262908+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:37.406495+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1005,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.530024+0000","last_change":"2024-09-21T12:39:43.400476+0000","last_active":"2024-09-21T12:40:15.530024+0000","last_peered":"2024-09-21T12:40:15.530024+0000","last_clean":"2024-09-21T12:40:15.530024+0000","last_became_active":"2024-09-21T12:38:08.164312+0000","last_became_peered":"2024-09-21T12:38:08.164312+0000","last_unstale":"2024-09-21T12:40:15.530024+0000","last_undegraded":"2024-09-21T12:40:15.530024+0000","last_fullsized":"2024-09-21T12:40:15.530024+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:43.400427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:43.400427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:47.864285+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":1228,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:25.602652+0000","last_change":"2024-09-21T12:38:45.275249+0000","last_active":"2024-09-21T12:40:25.602652+0000","last_peered":"2024-09-21T12:40:25.602652+0000","last_clean":"2024-09-21T12:40:25.602652+0000","last_became_active":"2024-09-21T12:20:19.314576+0000","last_became_peered":"2024-09-21T12:20:19.314576+0000","last_unstale":"2024-09-21T12:40:25.602652+0000","last_undegraded":"2024-09-21T12:40:25.602652+0000","last_fullsized":"2024-09-21T12:40:25.602652+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-21T12:38:45.275110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:38:45.275110+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":994,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.225793+0000","last_change":"2024-09-21T12:39:52.722587+0000","last_active":"2024-09-21T12:40:15.225793+0000","last_peered":"2024-09-21T12:40:15.225793+0000","last_clean":"2024-09-21T12:40:15.225793+0000","last_became_active":"2024-09-21T12:36:00.148332+0000","last_became_peered":"2024-09-21T12:36:00.148332+0000","last_unstale":"2024-09-21T12:40:15.225793+0000","last_undegraded":"2024-09-21T12:40:15.225793+0000","last_fullsized":"2024-09-21T12:40:15.225793+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:52.722539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:52.722539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:07.068423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"76'79","reported_seq":661,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.226714+0000","last_change":"2024-09-21T12:39:45.196000+0000","last_active":"2024-09-21T12:40:15.226714+0000","last_peered":"2024-09-21T12:40:15.226714+0000","last_clean":"2024-09-21T12:40:15.226714+0000","last_became_active":"2024-09-21T12:27:16.940347+0000","last_became_peered":"2024-09-21T12:27:16.940347+0000","last_unstale":"2024-09-21T12:40:15.226714+0000","last_undegraded":"2024-09-21T12:40:15.226714+0000","last_fullsized":"2024-09-21T12:40:15.226714+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:39:45.195949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:39:45.195949+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:07.860174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"518'27","reported_seq":390,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:24.485509+0000","last_change":"2024-09-21T12:40:24.485509+0000","last_active":"2024-09-21T12:40:24.485509+0000","last_peered":"2024-09-21T12:40:24.485509+0000","last_clean":"2024-09-21T12:40:24.485509+0000","last_became_active":"2024-09-21T12:32:38.421008+0000","last_became_peered":"2024-09-21T12:32:38.421008+0000","last_unstale":"2024-09-21T12:40:24.485509+0000","last_undegraded":"2024-09-21T12:40:24.485509+0000","last_fullsized":"2024-09-21T12:40:24.485509+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"518'27","last_scrub_stamp":"2024-09-21T12:40:24.485444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:40:24.485444+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:27.449665+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":386,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.222611+0000","last_change":"2024-09-21T12:39:44.540849+0000","last_active":"2024-09-21T12:40:15.222611+0000","last_peered":"2024-09-21T12:40:15.222611+0000","last_clean":"2024-09-21T12:40:15.222611+0000","last_became_active":"2024-09-21T12:36:01.123399+0000","last_became_peered":"2024-09-21T12:36:01.123399+0000","last_unstale":"2024-09-21T12:40:15.222611+0000","last_undegraded":"2024-09-21T12:40:15.222611+0000","last_fullsized":"2024-09-21T12:40:15.222611+0000","mapping_epoch":415,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":416,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:44.540771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:44.540771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:03.012331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":993,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.224880+0000","last_change":"2024-09-21T12:39:36.009428+0000","last_active":"2024-09-21T12:40:15.224880+0000","last_peered":"2024-09-21T12:40:15.224880+0000","last_clean":"2024-09-21T12:40:15.224880+0000","last_became_active":"2024-09-21T12:31:50.040754+0000","last_became_peered":"2024-09-21T12:31:50.040754+0000","last_unstale":"2024-09-21T12:40:15.224880+0000","last_undegraded":"2024-09-21T12:40:15.224880+0000","last_fullsized":"2024-09-21T12:40:15.224880+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:36.009386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:36.009386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:39.667415+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":442,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.527610+0000","last_change":"2024-09-21T12:39:37.375691+0000","last_active":"2024-09-21T12:40:15.527610+0000","last_peered":"2024-09-21T12:40:15.527610+0000","last_clean":"2024-09-21T12:40:15.527610+0000","last_became_active":"2024-09-21T12:31:50.036150+0000","last_became_peered":"2024-09-21T12:31:50.036150+0000","last_unstale":"2024-09-21T12:40:15.527610+0000","last_undegraded":"2024-09-21T12:40:15.527610+0000","last_fullsized":"2024-09-21T12:40:15.527610+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:37.375649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:37.375649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:01.292220+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"76'79","reported_seq":686,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:20.119278+0000","last_change":"2024-09-21T12:40:20.119278+0000","last_active":"2024-09-21T12:40:20.119278+0000","last_peered":"2024-09-21T12:40:20.119278+0000","last_clean":"2024-09-21T12:40:20.119278+0000","last_became_active":"2024-09-21T12:27:16.921880+0000","last_became_peered":"2024-09-21T12:27:16.921880+0000","last_unstale":"2024-09-21T12:40:20.119278+0000","last_undegraded":"2024-09-21T12:40:20.119278+0000","last_fullsized":"2024-09-21T12:40:20.119278+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:40:20.119215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:40:20.119215+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:46.961114+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"518'21","reported_seq":489,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.071594+0000","last_change":"2024-09-21T12:39:53.405733+0000","last_active":"2024-09-21T12:40:23.071594+0000","last_peered":"2024-09-21T12:40:23.071594+0000","last_clean":"2024-09-21T12:40:23.071594+0000","last_became_active":"2024-09-21T12:38:07.153467+0000","last_became_peered":"2024-09-21T12:38:07.153467+0000","last_unstale":"2024-09-21T12:40:23.071594+0000","last_undegraded":"2024-09-21T12:40:23.071594+0000","last_fullsized":"2024-09-21T12:40:23.071594+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'15","last_scrub_stamp":"2024-09-21T12:39:53.405674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:53.405674+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:06.710449+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":233,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:17.674382+0000","last_change":"2024-09-21T12:40:17.674382+0000","last_active":"2024-09-21T12:40:17.674382+0000","last_peered":"2024-09-21T12:40:17.674382+0000","last_clean":"2024-09-21T12:40:17.674382+0000","last_became_active":"2024-09-21T12:36:00.147063+0000","last_became_peered":"2024-09-21T12:36:00.147063+0000","last_unstale":"2024-09-21T12:40:17.674382+0000","last_undegraded":"2024-09-21T12:40:17.674382+0000","last_fullsized":"2024-09-21T12:40:17.674382+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:17.674314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:40:17.674314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:44.172604+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":974,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.528062+0000","last_change":"2024-09-21T12:39:51.512200+0000","last_active":"2024-09-21T12:40:15.528062+0000","last_peered":"2024-09-21T12:40:15.528062+0000","last_clean":"2024-09-21T12:40:15.528062+0000","last_became_active":"2024-09-21T12:20:19.310986+0000","last_became_peered":"2024-09-21T12:20:19.310986+0000","last_unstale":"2024-09-21T12:40:15.528062+0000","last_undegraded":"2024-09-21T12:40:15.528062+0000","last_fullsized":"2024-09-21T12:40:15.528062+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:51.512151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:51.512151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:58.225728+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":958,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:18.933242+0000","last_change":"2024-09-21T12:40:18.933242+0000","last_active":"2024-09-21T12:40:18.933242+0000","last_peered":"2024-09-21T12:40:18.933242+0000","last_clean":"2024-09-21T12:40:18.933242+0000","last_became_active":"2024-09-21T12:20:21.307682+0000","last_became_peered":"2024-09-21T12:20:21.307682+0000","last_unstale":"2024-09-21T12:40:18.933242+0000","last_undegraded":"2024-09-21T12:40:18.933242+0000","last_fullsized":"2024-09-21T12:40:18.933242+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:18.933195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:40:18.933195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:30.295146+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"76'79","reported_seq":418,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.626527+0000","last_change":"2024-09-21T12:40:10.207981+0000","last_active":"2024-09-21T12:40:15.626527+0000","last_peered":"2024-09-21T12:40:15.626527+0000","last_clean":"2024-09-21T12:40:15.626527+0000","last_became_active":"2024-09-21T12:40:10.207654+0000","last_became_peered":"2024-09-21T12:40:10.207654+0000","last_unstale":"2024-09-21T12:40:15.626527+0000","last_undegraded":"2024-09-21T12:40:15.626527+0000","last_fullsized":"2024-09-21T12:40:15.626527+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:40:04.593922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:40:04.593922+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:28.407895+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"518'27","reported_seq":1059,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.165269+0000","last_change":"2024-09-21T12:39:35.321714+0000","last_active":"2024-09-21T12:40:23.165269+0000","last_peered":"2024-09-21T12:40:23.165269+0000","last_clean":"2024-09-21T12:40:23.165269+0000","last_became_active":"2024-09-21T12:36:00.139682+0000","last_became_peered":"2024-09-21T12:36:00.139682+0000","last_unstale":"2024-09-21T12:40:23.165269+0000","last_undegraded":"2024-09-21T12:40:23.165269+0000","last_fullsized":"2024-09-21T12:40:23.165269+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:39:35.321662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:35.321662+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:43.243787+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1014,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.703282+0000","last_change":"2024-09-21T12:39:57.149886+0000","last_active":"2024-09-21T12:40:15.703282+0000","last_peered":"2024-09-21T12:40:15.703282+0000","last_clean":"2024-09-21T12:40:15.703282+0000","last_became_active":"2024-09-21T12:36:00.151584+0000","last_became_peered":"2024-09-21T12:36:00.151584+0000","last_unstale":"2024-09-21T12:40:15.703282+0000","last_undegraded":"2024-09-21T12:40:15.703282+0000","last_fullsized":"2024-09-21T12:40:15.703282+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:57.149832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:57.149832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:10.622809+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":233,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.528993+0000","last_change":"2024-09-21T12:39:44.411449+0000","last_active":"2024-09-21T12:40:15.528993+0000","last_peered":"2024-09-21T12:40:15.528993+0000","last_clean":"2024-09-21T12:40:15.528993+0000","last_became_active":"2024-09-21T12:36:00.144521+0000","last_became_peered":"2024-09-21T12:36:00.144521+0000","last_unstale":"2024-09-21T12:40:15.528993+0000","last_undegraded":"2024-09-21T12:40:15.528993+0000","last_fullsized":"2024-09-21T12:40:15.528993+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:44.411387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:44.411387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:46.423608+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":962,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.528127+0000","last_change":"2024-09-21T12:39:26.328424+0000","last_active":"2024-09-21T12:40:15.528127+0000","last_peered":"2024-09-21T12:40:15.528127+0000","last_clean":"2024-09-21T12:40:15.528127+0000","last_became_active":"2024-09-21T12:20:21.595358+0000","last_became_peered":"2024-09-21T12:20:21.595358+0000","last_unstale":"2024-09-21T12:40:15.528127+0000","last_undegraded":"2024-09-21T12:40:15.528127+0000","last_fullsized":"2024-09-21T12:40:15.528127+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:26.328369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:39:26.328369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:54.895336+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"76'79","reported_seq":233,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:22.667030+0000","last_change":"2024-09-21T12:40:22.667030+0000","last_active":"2024-09-21T12:40:22.667030+0000","last_peered":"2024-09-21T12:40:22.667030+0000","last_clean":"2024-09-21T12:40:22.667030+0000","last_became_active":"2024-09-21T12:36:00.151043+0000","last_became_peered":"2024-09-21T12:36:00.151043+0000","last_unstale":"2024-09-21T12:40:22.667030+0000","last_undegraded":"2024-09-21T12:40:22.667030+0000","last_fullsized":"2024-09-21T12:40:22.667030+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:40:22.666967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:40:22.666967+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:47.557026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"518'12","reported_seq":988,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:22.997810+0000","last_change":"2024-09-21T12:40:12.545354+0000","last_active":"2024-09-21T12:40:22.997810+0000","last_peered":"2024-09-21T12:40:22.997810+0000","last_clean":"2024-09-21T12:40:22.997810+0000","last_became_active":"2024-09-21T12:20:17.303696+0000","last_became_peered":"2024-09-21T12:20:17.303696+0000","last_unstale":"2024-09-21T12:40:22.997810+0000","last_undegraded":"2024-09-21T12:40:22.997810+0000","last_fullsized":"2024-09-21T12:40:22.997810+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'8","last_scrub_stamp":"2024-09-21T12:40:12.545286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:40:12.545286+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:26.007593+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"59'1","reported_seq":992,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.226640+0000","last_change":"2024-09-21T12:40:08.835848+0000","last_active":"2024-09-21T12:40:15.226640+0000","last_peered":"2024-09-21T12:40:15.226640+0000","last_clean":"2024-09-21T12:40:15.226640+0000","last_became_active":"2024-09-21T12:20:15.298119+0000","last_became_peered":"2024-09-21T12:20:15.298119+0000","last_unstale":"2024-09-21T12:40:15.226640+0000","last_undegraded":"2024-09-21T12:40:15.226640+0000","last_fullsized":"2024-09-21T12:40:15.226640+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-21T12:40:08.835779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:40:08.835779+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:10.451789+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1009,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.701436+0000","last_change":"2024-09-21T12:39:37.252584+0000","last_active":"2024-09-21T12:40:15.701436+0000","last_peered":"2024-09-21T12:40:15.701436+0000","last_clean":"2024-09-21T12:40:15.701436+0000","last_became_active":"2024-09-21T12:36:00.135537+0000","last_became_peered":"2024-09-21T12:36:00.135537+0000","last_unstale":"2024-09-21T12:40:15.701436+0000","last_undegraded":"2024-09-21T12:40:15.701436+0000","last_fullsized":"2024-09-21T12:40:15.701436+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:37.252520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:37.252520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:49.355760+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":993,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.529504+0000","last_change":"2024-09-21T12:40:12.706791+0000","last_active":"2024-09-21T12:40:15.529504+0000","last_peered":"2024-09-21T12:40:15.529504+0000","last_clean":"2024-09-21T12:40:15.529504+0000","last_became_active":"2024-09-21T12:38:08.164535+0000","last_became_peered":"2024-09-21T12:38:08.164535+0000","last_unstale":"2024-09-21T12:40:15.529504+0000","last_undegraded":"2024-09-21T12:40:15.529504+0000","last_fullsized":"2024-09-21T12:40:15.529504+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:12.706723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:40:12.706723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:35.409788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"76'79","reported_seq":380,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:27.422230+0000","last_change":"2024-09-21T12:40:27.422230+0000","last_active":"2024-09-21T12:40:27.422230+0000","last_peered":"2024-09-21T12:40:27.422230+0000","last_clean":"2024-09-21T12:40:27.422230+0000","last_became_active":"2024-09-21T12:36:00.137290+0000","last_became_peered":"2024-09-21T12:36:00.137290+0000","last_unstale":"2024-09-21T12:40:27.422230+0000","last_undegraded":"2024-09-21T12:40:27.422230+0000","last_fullsized":"2024-09-21T12:40:27.422230+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'79","last_scrub_stamp":"2024-09-21T12:40:27.422170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:17:39.165404+0000","last_clean_scrub_stamp":"2024-09-21T12:40:27.422170+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:54.236977+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1000,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.226740+0000","last_change":"2024-09-21T12:40:09.562035+0000","last_active":"2024-09-21T12:40:15.226740+0000","last_peered":"2024-09-21T12:40:15.226740+0000","last_clean":"2024-09-21T12:40:15.226740+0000","last_became_active":"2024-09-21T12:32:38.426822+0000","last_became_peered":"2024-09-21T12:32:38.426822+0000","last_unstale":"2024-09-21T12:40:15.226740+0000","last_undegraded":"2024-09-21T12:40:15.226740+0000","last_fullsized":"2024-09-21T12:40:15.226740+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:40:09.561964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:20.292557+0000","last_clean_scrub_stamp":"2024-09-21T12:40:09.561964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:23.954625+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":988,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.527888+0000","last_change":"2024-09-21T12:39:25.339333+0000","last_active":"2024-09-21T12:40:15.527888+0000","last_peered":"2024-09-21T12:40:15.527888+0000","last_clean":"2024-09-21T12:40:15.527888+0000","last_became_active":"2024-09-21T12:31:39.052486+0000","last_became_peered":"2024-09-21T12:31:39.052486+0000","last_unstale":"2024-09-21T12:40:15.527888+0000","last_undegraded":"2024-09-21T12:40:15.527888+0000","last_fullsized":"2024-09-21T12:40:15.527888+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:25.339277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:14.277394+0000","last_clean_scrub_stamp":"2024-09-21T12:39:25.339277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:39.704442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"518'27","reported_seq":1023,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:23.167172+0000","last_change":"2024-09-21T12:39:51.135469+0000","last_active":"2024-09-21T12:40:23.167172+0000","last_peered":"2024-09-21T12:40:23.167172+0000","last_clean":"2024-09-21T12:40:23.167172+0000","last_became_active":"2024-09-21T12:20:17.298718+0000","last_became_peered":"2024-09-21T12:20:17.298718+0000","last_unstale":"2024-09-21T12:40:23.167172+0000","last_undegraded":"2024-09-21T12:40:23.167172+0000","last_fullsized":"2024-09-21T12:40:23.167172+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'19","last_scrub_stamp":"2024-09-21T12:39:51.135405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:16.284875+0000","last_clean_scrub_stamp":"2024-09-21T12:39:51.135405+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:41:08.006842+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1018,"reported_epoch":518,"state":"active+clean","last_fresh":"2024-09-21T12:40:15.225929+0000","last_change":"2024-09-21T12:39:36.668705+0000","last_active":"2024-09-21T12:40:15.225929+0000","last_peered":"2024-09-21T12:40:15.225929+0000","last_clean":"2024-09-21T12:40:15.225929+0000","last_became_active":"2024-09-21T12:38:08.174864+0000","last_became_peered":"2024-09-21T12:38:08.174864+0000","last_unstale":"2024-09-21T12:40:15.225929+0000","last_undegraded":"2024-09-21T12:40:15.225929+0000","last_fullsized":"2024-09-21T12:40:15.225929+0000","mapping_epoch":456,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:39:36.668657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:20:18.288890+0000","last_clean_scrub_stamp":"2024-09-21T12:39:36.668657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T12:40:55.188047+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1535,"num_read_kb":1292,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":462,"num_bytes_recovered":9130,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3597,"internal_metadata":0},"log_size":975,"ondisk_log_size":975,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":598,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":278,"num_read_kb":273,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2449,"ondisk_log_size":2449,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234232,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":848896,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":92896256,"statfs":{"total":95995035648,"available":95125766144,"internally_reserved":0,"allocated":1425408,"data_stored":943751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":867826122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463168,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":889196,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":92855956,"statfs":{"total":95995035648,"available":95084498944,"internally_reserved":0,"allocated":1470464,"data_stored":1008094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":909048262},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724803,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":780868,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":779454,"kb_avail":92964284,"statfs":{"total":95995035648,"available":95195426816,"internally_reserved":0,"allocated":1429504,"data_stored":944687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":798161349},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953739,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":425220,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93319932,"statfs":{"total":95995035648,"available":95559610368,"internally_reserved":0,"allocated":1167360,"data_stored":680583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149969,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":929488,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":927998,"kb_avail":92815664,"statfs":{"total":95995035648,"available":95043239936,"internally_reserved":0,"allocated":1507328,"data_stored":1009868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":950270407},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":455,"seq":1954210119711,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":233904,"kb_used_data":1376,"kb_used_omap":7,"kb_used_meta":232504,"kb_avail":93511248,"statfs":{"total":95995035648,"available":95755517952,"internally_reserved":0,"allocated":1409024,"data_stored":943034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8042,"internal_metadata":238084246},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575133,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":993056,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":991870,"kb_avail":92752096,"statfs":{"total":95995035648,"available":94978146304,"internally_reserved":0,"allocated":1196032,"data_stored":681446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1015675333},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738659,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":930888,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":929406,"kb_avail":92814264,"statfs":{"total":95995035648,"available":95041806336,"internally_reserved":0,"allocated":1499136,"data_stored":1009458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":951712197},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3597,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:40:36.091 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T12:40:36.092 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301 shell --fsid e2ad975c-7812-11ef-baf6-efdc52797490 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-21T12:40:36.338 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:40:36.458 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:36 smithi114 ceph-mon[40867]: pgmap v1048: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-21T12:40:36.459 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:36 smithi114 ceph-mon[40867]: from='client.16803 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:40:36.459 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:36 smithi114 ceph-mon[30324]: pgmap v1048: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-21T12:40:36.459 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:36 smithi114 ceph-mon[30324]: from='client.16803 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:40:36.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:36 smithi183 ceph-mon[33277]: pgmap v1048: 162 pgs: 162 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-09-21T12:40:36.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:36 smithi183 ceph-mon[33277]: from='client.16803 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:40:37.300 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:40:37 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:40:37] ENGINE Bus STOPPING 2024-09-21T12:40:37.300 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:40:37 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:40:37] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T12:40:37.301 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:40:37 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:40:37] ENGINE Bus STOPPED 2024-09-21T12:40:37.301 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:40:37 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:40:37] ENGINE Bus STARTING 2024-09-21T12:40:37.301 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:40:37 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:40:37] ENGINE Serving on http://:::9283 2024-09-21T12:40:37.301 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:40:37 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: [21/Sep/2024:12:40:37] ENGINE Bus STARTED 2024-09-21T12:40:37.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:37 smithi183 ceph-mon[33277]: 5.e scrub starts 2024-09-21T12:40:37.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:37 smithi183 ceph-mon[33277]: 5.e scrub ok 2024-09-21T12:40:37.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:37 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/525955078' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-21T12:40:37.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:37 smithi183 ceph-mon[33277]: from='client.? 172.21.15.114:0/525955078' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-21T12:40:37.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:37 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:40:37.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:37 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:40:37.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:37 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:40:37.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:37 smithi183 ceph-mon[33277]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:40:37.739 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:37 smithi183 ceph-mon[33277]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T12:40:37.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:37 smithi114 ceph-mon[30324]: 5.e scrub starts 2024-09-21T12:40:37.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:37 smithi114 ceph-mon[30324]: 5.e scrub ok 2024-09-21T12:40:37.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:37 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/525955078' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-21T12:40:37.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:37 smithi114 ceph-mon[30324]: from='client.? 172.21.15.114:0/525955078' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-21T12:40:37.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:37 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:40:37.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:37 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:40:37.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:37 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:40:37.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:37 smithi114 ceph-mon[30324]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:40:37.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:37 smithi114 ceph-mon[30324]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T12:40:37.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:37 smithi114 ceph-mon[40867]: 5.e scrub starts 2024-09-21T12:40:37.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:37 smithi114 ceph-mon[40867]: 5.e scrub ok 2024-09-21T12:40:37.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:37 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/525955078' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-21T12:40:37.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:37 smithi114 ceph-mon[40867]: from='client.? 172.21.15.114:0/525955078' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-21T12:40:37.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:37 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T12:40:37.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:37 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T12:40:37.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:37 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T12:40:37.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:37 smithi114 ceph-mon[40867]: from='mgr.24470 ' entity='mgr.y' 2024-09-21T12:40:37.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:37 smithi114 ceph-mon[40867]: from='mgr.24470 172.21.15.114:0/4049545892' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T12:40:37.822 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-21T12:40:37.832 INFO:tasks.cephadm:Teardown begin 2024-09-21T12:40:37.834 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T12:40:37.883 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T12:40:37.926 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-21T12:40:37.926 DEBUG:teuthology.orchestra.run.smithi114:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-21T12:40:37.954 DEBUG:teuthology.orchestra.run.smithi183:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-21T12:40:38.002 INFO:tasks.cephadm:Stopping all daemons... 2024-09-21T12:40:38.002 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-21T12:40:38.002 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-e2ad975c-7812-11ef-baf6-efdc52797490@mon.a 2024-09-21T12:40:38.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:40:37 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug there is no tcmu-runner data available 2024-09-21T12:40:38.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:38 smithi114 systemd[1]: Stopping Ceph mon.a for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:40:38.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:38 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-a[30301]: 2024-09-21T12:40:38.297+0000 7f4a03e90640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T12:40:38.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:38 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-a[30301]: 2024-09-21T12:40:38.297+0000 7f4a03e90640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-21T12:40:38.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:38 smithi114 podman[217664]: 2024-09-21 12:40:38.492623352 +0000 UTC m=+0.297248169 container died 19957648bd4ed923f5f31e20868af18b01a8a96b64d2d0418433545927aa7586 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=) 2024-09-21T12:40:39.140 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:38 smithi114 podman[217664]: 2024-09-21 12:40:38.862739758 +0000 UTC m=+0.667364582 container cleanup 19957648bd4ed923f5f31e20868af18b01a8a96b64d2d0418433545927aa7586 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T12:40:39.140 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 21 12:40:38 smithi114 bash[217664]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-a 2024-09-21T12:40:39.267 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@mon.a.service' 2024-09-21T12:40:39.305 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T12:40:39.305 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-21T12:40:39.306 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-21T12:40:39.306 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-e2ad975c-7812-11ef-baf6-efdc52797490@mon.c 2024-09-21T12:40:39.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:39 smithi114 systemd[1]: Stopping Ceph mon.c for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:40:40.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:39 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-c[40844]: 2024-09-21T12:40:39.677+0000 7fba37042640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T12:40:40.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:39 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-c[40844]: 2024-09-21T12:40:39.677+0000 7fba37042640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-21T12:40:40.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:39 smithi114 podman[217795]: 2024-09-21 12:40:39.821574214 +0000 UTC m=+0.292814485 container died 78eecbc43ac75f9fce635b95e97306f07684f0aff5df76da86fe71daf4b30d47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-c, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-21T12:40:40.445 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:40 smithi114 podman[217795]: 2024-09-21 12:40:40.284891611 +0000 UTC m=+0.756131869 container cleanup 78eecbc43ac75f9fce635b95e97306f07684f0aff5df76da86fe71daf4b30d47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-21T12:40:40.445 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:40 smithi114 bash[217795]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-c 2024-09-21T12:40:40.705 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 21 12:40:40 smithi114 podman[217813]: 2024-09-21 12:40:40.444906791 +0000 UTC m=+0.617903033 container remove 78eecbc43ac75f9fce635b95e97306f07684f0aff5df76da86fe71daf4b30d47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-c, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-21T12:40:40.709 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@mon.c.service' 2024-09-21T12:40:40.743 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T12:40:40.743 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-21T12:40:40.743 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-21T12:40:40.743 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-e2ad975c-7812-11ef-baf6-efdc52797490@mon.b 2024-09-21T12:40:41.051 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:40 smithi183 systemd[1]: Stopping Ceph mon.b for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:40:41.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-b[33254]: 2024-09-21T12:40:41.050+0000 7fcce0dce640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T12:40:41.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-b[33254]: 2024-09-21T12:40:41.050+0000 7fcce0dce640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-21T12:40:41.488 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:41 smithi183 podman[153371]: 2024-09-21 12:40:41.214059261 +0000 UTC m=+0.261932162 container died 91b61105f05cb8bea9357a5f3059bf8731eaba160867238f99ca65557b992694 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, CEPH_POINT_RELEASE=) 2024-09-21T12:40:41.976 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:41 smithi183 podman[153371]: 2024-09-21 12:40:41.690981229 +0000 UTC m=+0.738854169 container cleanup 91b61105f05cb8bea9357a5f3059bf8731eaba160867238f99ca65557b992694 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-b, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-21T12:40:41.977 INFO:journalctl@ceph.mon.b.smithi183.stdout:Sep 21 12:40:41 smithi183 bash[153371]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mon-b 2024-09-21T12:40:42.100 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@mon.b.service' 2024-09-21T12:40:42.150 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T12:40:42.150 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-21T12:40:42.150 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-21T12:40:42.150 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-e2ad975c-7812-11ef-baf6-efdc52797490@mgr.y 2024-09-21T12:40:42.450 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:40:42 smithi114 systemd[1]: Stopping Ceph mgr.y for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:40:42.750 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:40:42 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: 2024-09-21T12:40:42.448+0000 7f0e015af640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T12:40:42.750 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:40:42 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y[30575]: 2024-09-21T12:40:42.448+0000 7f0e015af640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-09-21T12:40:42.750 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:40:42 smithi114 podman[217927]: 2024-09-21 12:40:42.481180108 +0000 UTC m=+0.139786038 container died b5b5207883523b18469b1bbaa814349423722e255898bc59245c63c90d67c322 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-21T12:40:43.024 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:40:42 smithi114 podman[217927]: 2024-09-21 12:40:42.923120892 +0000 UTC m=+0.581726814 container cleanup b5b5207883523b18469b1bbaa814349423722e255898bc59245c63c90d67c322 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5c1cdc3c470dae78e91a0ed2bcf13509a046d301, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T12:40:43.024 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 21 12:40:42 smithi114 bash[217927]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-y 2024-09-21T12:40:43.394 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@mgr.y.service' 2024-09-21T12:40:43.429 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T12:40:43.429 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-21T12:40:43.429 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-21T12:40:43.429 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-e2ad975c-7812-11ef-baf6-efdc52797490@mgr.x 2024-09-21T12:40:43.738 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Sep 21 12:40:43 smithi183 systemd[1]: Stopping Ceph mgr.x for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:40:44.031 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Sep 21 12:40:43 smithi183 podman[153504]: 2024-09-21 12:40:43.770954107 +0000 UTC m=+0.147826824 container died 4b8441eb92187ba7467a7e43f79f2558574cb864b77a4eb795b06bc4d05f2bfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-x, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.vendor=CentOS) 2024-09-21T12:40:44.398 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Sep 21 12:40:44 smithi183 podman[153504]: 2024-09-21 12:40:44.187615503 +0000 UTC m=+0.564488219 container cleanup 4b8441eb92187ba7467a7e43f79f2558574cb864b77a4eb795b06bc4d05f2bfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-x, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-21T12:40:44.398 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Sep 21 12:40:44 smithi183 bash[153504]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-x 2024-09-21T12:40:44.626 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@mgr.x.service' 2024-09-21T12:40:44.649 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Sep 21 12:40:44 smithi183 podman[153521]: 2024-09-21 12:40:44.397611326 +0000 UTC m=+0.629113329 container remove 4b8441eb92187ba7467a7e43f79f2558574cb864b77a4eb795b06bc4d05f2bfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-mgr-x, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-21T12:40:44.650 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Sep 21 12:40:44 smithi183 systemd[1]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-21T12:40:44.650 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Sep 21 12:40:44 smithi183 systemd[1]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490@mgr.x.service: Failed with result 'exit-code'. 2024-09-21T12:40:44.650 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Sep 21 12:40:44 smithi183 systemd[1]: Stopped Ceph mgr.x for e2ad975c-7812-11ef-baf6-efdc52797490. 2024-09-21T12:40:44.650 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Sep 21 12:40:44 smithi183 systemd[1]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490@mgr.x.service: Consumed 12.742s CPU time. 2024-09-21T12:40:44.661 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T12:40:44.662 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-21T12:40:44.662 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-21T12:40:44.662 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-e2ad975c-7812-11ef-baf6-efdc52797490@osd.0 2024-09-21T12:40:44.956 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 21 12:40:44 smithi114 systemd[1]: Stopping Ceph osd.0 for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:40:45.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 21 12:40:44 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-0[44577]: 2024-09-21T12:40:44.954+0000 7fad3128e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T12:40:45.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 21 12:40:44 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-0[44577]: 2024-09-21T12:40:44.954+0000 7fad3128e640 -1 osd.0 518 *** Got signal Terminated *** 2024-09-21T12:40:45.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 21 12:40:44 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-0[44577]: 2024-09-21T12:40:44.954+0000 7fad3128e640 -1 osd.0 518 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T12:40:50.775 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 21 12:40:50 smithi114 podman[218060]: 2024-09-21 12:40:50.502981052 +0000 UTC m=+5.651667673 container died 5c0d96f54bc27e105a0b986c6a2bc9b02eaa8e7262788197e7dfd496a1b33f9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3) 2024-09-21T12:40:51.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 21 12:40:50 smithi114 podman[218060]: 2024-09-21 12:40:50.986244118 +0000 UTC m=+6.134930735 container cleanup 5c0d96f54bc27e105a0b986c6a2bc9b02eaa8e7262788197e7dfd496a1b33f9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-21T12:40:51.159 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 21 12:40:50 smithi114 bash[218060]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-0 2024-09-21T12:40:51.524 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 21 12:40:51 smithi114 podman[218081]: 2024-09-21 12:40:51.338739057 +0000 UTC m=+0.833419936 container remove 5c0d96f54bc27e105a0b986c6a2bc9b02eaa8e7262788197e7dfd496a1b33f9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-21T12:40:52.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 21 12:40:51 smithi114 podman[218187]: 2024-09-21 12:40:51.651666767 +0000 UTC m=+0.019068621 image pull 8f4d013db793795fef09ff5edf96e6b42ba8cd2c9158b2f0b7a5d7c1e657c063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472 2024-09-21T12:40:52.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 21 12:40:51 smithi114 podman[218187]: 2024-09-21 12:40:51.772973505 +0000 UTC m=+0.140375361 container create b4e654185621e51009bec133a318c47d4ddac3814feffc37f17c694ac99d13d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-0-deactivate, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T12:40:52.018 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 21 12:40:52 smithi114 podman[218187]: 2024-09-21 12:40:52.017361172 +0000 UTC m=+0.384763045 container init b4e654185621e51009bec133a318c47d4ddac3814feffc37f17c694ac99d13d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-21T12:40:52.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 21 12:40:52 smithi114 podman[218187]: 2024-09-21 12:40:52.022492866 +0000 UTC m=+0.389894722 container start b4e654185621e51009bec133a318c47d4ddac3814feffc37f17c694ac99d13d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-0-deactivate, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True) 2024-09-21T12:40:52.274 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 21 12:40:52 smithi114 podman[218187]: 2024-09-21 12:40:52.100659039 +0000 UTC m=+0.468060980 container attach b4e654185621e51009bec133a318c47d4ddac3814feffc37f17c694ac99d13d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-0-deactivate, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-21T12:40:52.549 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 21 12:40:52 smithi114 podman[218187]: 2024-09-21 12:40:52.290001357 +0000 UTC m=+0.657403220 container died b4e654185621e51009bec133a318c47d4ddac3814feffc37f17c694ac99d13d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-0-deactivate, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T12:40:52.987 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@osd.0.service' 2024-09-21T12:40:53.026 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T12:40:53.027 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-21T12:40:53.027 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-21T12:40:53.027 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-e2ad975c-7812-11ef-baf6-efdc52797490@osd.1 2024-09-21T12:40:53.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 21 12:40:53 smithi114 systemd[1]: Stopping Ceph osd.1 for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:40:53.774 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 21 12:40:53 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-1[50411]: 2024-09-21T12:40:53.384+0000 7f8467c37640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T12:40:53.774 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 21 12:40:53 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-1[50411]: 2024-09-21T12:40:53.384+0000 7f8467c37640 -1 osd.1 518 *** Got signal Terminated *** 2024-09-21T12:40:53.775 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 21 12:40:53 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-1[50411]: 2024-09-21T12:40:53.384+0000 7f8467c37640 -1 osd.1 518 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T12:40:59.242 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 21 12:40:58 smithi114 podman[218318]: 2024-09-21 12:40:58.94473955 +0000 UTC m=+5.689807384 container died 375d2d524fc4843de1dbbf7c96ef6f4e828ec24a5c0dc45a15b5d2a771060228 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-1, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-21T12:40:59.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 21 12:40:59 smithi114 podman[218318]: 2024-09-21 12:40:59.424627187 +0000 UTC m=+6.169695008 container cleanup 375d2d524fc4843de1dbbf7c96ef6f4e828ec24a5c0dc45a15b5d2a771060228 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-1, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-21T12:40:59.524 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 21 12:40:59 smithi114 bash[218318]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-1 2024-09-21T12:41:00.017 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 21 12:40:59 smithi114 podman[218337]: 2024-09-21 12:40:59.709518419 +0000 UTC m=+0.752459232 container remove 375d2d524fc4843de1dbbf7c96ef6f4e828ec24a5c0dc45a15b5d2a771060228 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-1, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-21T12:41:00.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 21 12:41:00 smithi114 podman[218433]: 2024-09-21 12:40:59.919681804 +0000 UTC m=+0.015925070 image pull 8f4d013db793795fef09ff5edf96e6b42ba8cd2c9158b2f0b7a5d7c1e657c063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472 2024-09-21T12:41:00.274 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 21 12:41:00 smithi114 podman[218433]: 2024-09-21 12:41:00.036088756 +0000 UTC m=+0.132332007 container create 1157e92b777717dbe12045ff2a5a56b07bd8394da6e12b1029a3ab14bcf5fad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-1-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T12:41:00.774 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 21 12:41:00 smithi114 podman[218433]: 2024-09-21 12:41:00.313361101 +0000 UTC m=+0.409604357 container init 1157e92b777717dbe12045ff2a5a56b07bd8394da6e12b1029a3ab14bcf5fad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-21T12:41:00.774 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 21 12:41:00 smithi114 podman[218433]: 2024-09-21 12:41:00.31819194 +0000 UTC m=+0.414435188 container start 1157e92b777717dbe12045ff2a5a56b07bd8394da6e12b1029a3ab14bcf5fad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-21T12:41:00.775 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 21 12:41:00 smithi114 podman[218433]: 2024-09-21 12:41:00.421423328 +0000 UTC m=+0.517666578 container attach 1157e92b777717dbe12045ff2a5a56b07bd8394da6e12b1029a3ab14bcf5fad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-1-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD) 2024-09-21T12:41:00.775 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 21 12:41:00 smithi114 podman[218433]: 2024-09-21 12:41:00.505353052 +0000 UTC m=+0.601596307 container died 1157e92b777717dbe12045ff2a5a56b07bd8394da6e12b1029a3ab14bcf5fad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-1-deactivate, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-21T12:41:01.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@osd.1.service' 2024-09-21T12:41:01.462 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T12:41:01.462 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-21T12:41:01.462 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-21T12:41:01.463 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-e2ad975c-7812-11ef-baf6-efdc52797490@osd.2 2024-09-21T12:41:01.774 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:41:01 smithi114 systemd[1]: Stopping Ceph osd.2 for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:41:02.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:41:01 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2[201186]: 2024-09-21T12:41:01.821+0000 7ffb78d0a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T12:41:02.274 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:41:01 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2[201186]: 2024-09-21T12:41:01.821+0000 7ffb78d0a640 -1 osd.2 518 *** Got signal Terminated *** 2024-09-21T12:41:02.275 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:41:01 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2[201186]: 2024-09-21T12:41:01.821+0000 7ffb78d0a640 -1 osd.2 518 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T12:41:07.447 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:41:07 smithi114 podman[218562]: 2024-09-21 12:41:07.179676802 +0000 UTC m=+5.494939920 container died 9a201988edd88a93759b283219b7fe6b24114181dd77304737ba093030e8770d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T12:41:07.774 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:41:07 smithi114 podman[218562]: 2024-09-21 12:41:07.552996677 +0000 UTC m=+5.868259780 container cleanup 9a201988edd88a93759b283219b7fe6b24114181dd77304737ba093030e8770d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2) 2024-09-21T12:41:07.774 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:41:07 smithi114 bash[218562]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2 2024-09-21T12:41:08.222 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:41:07 smithi114 podman[218581]: 2024-09-21 12:41:07.862774877 +0000 UTC m=+0.682233259 container remove 9a201988edd88a93759b283219b7fe6b24114181dd77304737ba093030e8770d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-21T12:41:08.483 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:41:08 smithi114 podman[218671]: 2024-09-21 12:41:08.124808438 +0000 UTC m=+0.015755319 image pull 8f4d013db793795fef09ff5edf96e6b42ba8cd2c9158b2f0b7a5d7c1e657c063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472 2024-09-21T12:41:08.484 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:41:08 smithi114 podman[218671]: 2024-09-21 12:41:08.255320329 +0000 UTC m=+0.146267201 container create 67531cdb522c989ab587c872978dd8c671c6169b1720efbb1e21abbbf223c965 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2-deactivate, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3) 2024-09-21T12:41:08.484 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:41:08 smithi114 podman[218671]: 2024-09-21 12:41:08.483095737 +0000 UTC m=+0.374042616 container init 67531cdb522c989ab587c872978dd8c671c6169b1720efbb1e21abbbf223c965 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2-deactivate, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-21T12:41:08.774 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:41:08 smithi114 podman[218671]: 2024-09-21 12:41:08.487756974 +0000 UTC m=+0.378703852 container start 67531cdb522c989ab587c872978dd8c671c6169b1720efbb1e21abbbf223c965 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2-deactivate, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-21T12:41:08.774 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:41:08 smithi114 podman[218671]: 2024-09-21 12:41:08.591121506 +0000 UTC m=+0.482068378 container attach 67531cdb522c989ab587c872978dd8c671c6169b1720efbb1e21abbbf223c965 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-21T12:41:08.774 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 21 12:41:08 smithi114 podman[218671]: 2024-09-21 12:41:08.672386717 +0000 UTC m=+0.563333592 container died 67531cdb522c989ab587c872978dd8c671c6169b1720efbb1e21abbbf223c965 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-2-deactivate, ceph=True, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-21T12:41:09.351 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@osd.2.service' 2024-09-21T12:41:09.390 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T12:41:09.390 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-21T12:41:09.390 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-21T12:41:09.390 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-e2ad975c-7812-11ef-baf6-efdc52797490@osd.3 2024-09-21T12:41:09.718 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 21 12:41:09 smithi114 systemd[1]: Stopping Ceph osd.3 for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:41:10.024 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 21 12:41:09 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-3[62933]: 2024-09-21T12:41:09.716+0000 7f200adf2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T12:41:10.024 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 21 12:41:09 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-3[62933]: 2024-09-21T12:41:09.716+0000 7f200adf2640 -1 osd.3 518 *** Got signal Terminated *** 2024-09-21T12:41:10.024 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 21 12:41:09 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-3[62933]: 2024-09-21T12:41:09.716+0000 7f200adf2640 -1 osd.3 518 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T12:41:11.739 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:11 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:11.485+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.826418+0000 front 2024-09-21T12:40:45.826280+0000 (oldest deadline 2024-09-21T12:41:10.526217+0000) 2024-09-21T12:41:11.739 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:11 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:11.726+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:12.710 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:12 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:12.451+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.826418+0000 front 2024-09-21T12:40:45.826280+0000 (oldest deadline 2024-09-21T12:41:10.526217+0000) 2024-09-21T12:41:12.988 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:12 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:12.708+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:13.744 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:13 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:13.490+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.826418+0000 front 2024-09-21T12:40:45.826280+0000 (oldest deadline 2024-09-21T12:41:10.526217+0000) 2024-09-21T12:41:14.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:13 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:13.742+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:14.238 INFO:journalctl@ceph.osd.4.smithi183.stdout:Sep 21 12:41:13 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-4[37819]: 2024-09-21T12:41:13.790+0000 7f57d0341640 -1 osd.4 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:47.933219+0000 front 2024-09-21T12:40:47.933202+0000 (oldest deadline 2024-09-21T12:41:13.233094+0000) 2024-09-21T12:41:14.239 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:13 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:13.979+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:48.892221+0000 front 2024-09-21T12:40:48.892179+0000 (oldest deadline 2024-09-21T12:41:12.992108+0000) 2024-09-21T12:41:14.739 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:14 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:14.481+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.826418+0000 front 2024-09-21T12:40:45.826280+0000 (oldest deadline 2024-09-21T12:41:10.526217+0000) 2024-09-21T12:41:15.238 INFO:journalctl@ceph.osd.4.smithi183.stdout:Sep 21 12:41:14 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-4[37819]: 2024-09-21T12:41:14.768+0000 7f57d0341640 -1 osd.4 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:47.933219+0000 front 2024-09-21T12:40:47.933202+0000 (oldest deadline 2024-09-21T12:41:13.233094+0000) 2024-09-21T12:41:15.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:14 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:14.740+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:15.239 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:14 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:14.989+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:48.892221+0000 front 2024-09-21T12:40:48.892179+0000 (oldest deadline 2024-09-21T12:41:12.992108+0000) 2024-09-21T12:41:15.682 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 21 12:41:15 smithi114 podman[218800]: 2024-09-21 12:41:15.365763375 +0000 UTC m=+5.754027597 container died e372729ce419204c94663db88bb9477ba022d78a457d8c14493b15f3e080aeea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-3, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-21T12:41:15.738 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:15 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:15.466+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.826418+0000 front 2024-09-21T12:40:45.826280+0000 (oldest deadline 2024-09-21T12:41:10.526217+0000) 2024-09-21T12:41:16.024 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 21 12:41:15 smithi114 podman[218800]: 2024-09-21 12:41:15.755365479 +0000 UTC m=+6.143629704 container cleanup e372729ce419204c94663db88bb9477ba022d78a457d8c14493b15f3e080aeea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-3, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, GIT_BRANCH=HEAD) 2024-09-21T12:41:16.024 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 21 12:41:15 smithi114 bash[218800]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-3 2024-09-21T12:41:16.238 INFO:journalctl@ceph.osd.4.smithi183.stdout:Sep 21 12:41:15 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-4[37819]: 2024-09-21T12:41:15.796+0000 7f57d0341640 -1 osd.4 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:47.933219+0000 front 2024-09-21T12:40:47.933202+0000 (oldest deadline 2024-09-21T12:41:13.233094+0000) 2024-09-21T12:41:16.238 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:15 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:15.943+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:48.892221+0000 front 2024-09-21T12:40:48.892179+0000 (oldest deadline 2024-09-21T12:41:12.992108+0000) 2024-09-21T12:41:16.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:15 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:15.780+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:16.499 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 21 12:41:16 smithi114 podman[218819]: 2024-09-21 12:41:16.190088949 +0000 UTC m=+0.801782931 container remove e372729ce419204c94663db88bb9477ba022d78a457d8c14493b15f3e080aeea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-3, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-21T12:41:16.738 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:16 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:16.464+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.826418+0000 front 2024-09-21T12:40:45.826280+0000 (oldest deadline 2024-09-21T12:41:10.526217+0000) 2024-09-21T12:41:16.774 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 21 12:41:16 smithi114 podman[218914]: 2024-09-21 12:41:16.401640283 +0000 UTC m=+0.016042811 image pull 8f4d013db793795fef09ff5edf96e6b42ba8cd2c9158b2f0b7a5d7c1e657c063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472 2024-09-21T12:41:16.774 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 21 12:41:16 smithi114 podman[218914]: 2024-09-21 12:41:16.499083947 +0000 UTC m=+0.113486462 container create 5134ca715e865e43a3e3b601f013e2ffdcb908d7bfe5498b697751e61475843b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-3-deactivate, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-21T12:41:16.775 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 21 12:41:16 smithi114 podman[218914]: 2024-09-21 12:41:16.709620405 +0000 UTC m=+0.324022925 container init 5134ca715e865e43a3e3b601f013e2ffdcb908d7bfe5498b697751e61475843b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-3-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T12:41:16.775 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 21 12:41:16 smithi114 podman[218914]: 2024-09-21 12:41:16.715128768 +0000 UTC m=+0.329531285 container start 5134ca715e865e43a3e3b601f013e2ffdcb908d7bfe5498b697751e61475843b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-21T12:41:17.205 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 21 12:41:16 smithi114 podman[218914]: 2024-09-21 12:41:16.801401342 +0000 UTC m=+0.415803867 container attach 5134ca715e865e43a3e3b601f013e2ffdcb908d7bfe5498b697751e61475843b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-3-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-21T12:41:17.205 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 21 12:41:16 smithi114 podman[218914]: 2024-09-21 12:41:16.911111337 +0000 UTC m=+0.525513862 container died 5134ca715e865e43a3e3b601f013e2ffdcb908d7bfe5498b697751e61475843b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True) 2024-09-21T12:41:17.238 INFO:journalctl@ceph.osd.4.smithi183.stdout:Sep 21 12:41:16 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-4[37819]: 2024-09-21T12:41:16.821+0000 7f57d0341640 -1 osd.4 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:47.933219+0000 front 2024-09-21T12:40:47.933202+0000 (oldest deadline 2024-09-21T12:41:13.233094+0000) 2024-09-21T12:41:17.238 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:16 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:16.957+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:48.892221+0000 front 2024-09-21T12:40:48.892179+0000 (oldest deadline 2024-09-21T12:41:12.992108+0000) 2024-09-21T12:41:17.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:16 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:16.825+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:17.502 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@osd.3.service' 2024-09-21T12:41:17.541 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T12:41:17.542 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-21T12:41:17.542 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-21T12:41:17.542 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-e2ad975c-7812-11ef-baf6-efdc52797490@osd.4 2024-09-21T12:41:17.834 INFO:journalctl@ceph.osd.4.smithi183.stdout:Sep 21 12:41:17 smithi183 systemd[1]: Stopping Ceph osd.4 for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:41:17.834 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:17 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:17.490+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.826418+0000 front 2024-09-21T12:40:45.826280+0000 (oldest deadline 2024-09-21T12:41:10.526217+0000) 2024-09-21T12:41:18.238 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:17 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:17.928+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:48.892221+0000 front 2024-09-21T12:40:48.892179+0000 (oldest deadline 2024-09-21T12:41:12.992108+0000) 2024-09-21T12:41:18.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:17 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:17.835+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:18.239 INFO:journalctl@ceph.osd.4.smithi183.stdout:Sep 21 12:41:17 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-4[37819]: 2024-09-21T12:41:17.832+0000 7f57d7d34640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T12:41:18.239 INFO:journalctl@ceph.osd.4.smithi183.stdout:Sep 21 12:41:17 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-4[37819]: 2024-09-21T12:41:17.832+0000 7f57d7d34640 -1 osd.4 518 *** Got signal Terminated *** 2024-09-21T12:41:18.239 INFO:journalctl@ceph.osd.4.smithi183.stdout:Sep 21 12:41:17 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-4[37819]: 2024-09-21T12:41:17.832+0000 7f57d7d34640 -1 osd.4 518 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T12:41:18.239 INFO:journalctl@ceph.osd.4.smithi183.stdout:Sep 21 12:41:17 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-4[37819]: 2024-09-21T12:41:17.836+0000 7f57d0341640 -1 osd.4 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:47.933219+0000 front 2024-09-21T12:40:47.933202+0000 (oldest deadline 2024-09-21T12:41:13.233094+0000) 2024-09-21T12:41:18.738 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:18 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:18.475+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.826418+0000 front 2024-09-21T12:40:45.826280+0000 (oldest deadline 2024-09-21T12:41:10.526217+0000) 2024-09-21T12:41:19.238 INFO:journalctl@ceph.osd.4.smithi183.stdout:Sep 21 12:41:18 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-4[37819]: 2024-09-21T12:41:18.866+0000 7f57d0341640 -1 osd.4 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:47.933219+0000 front 2024-09-21T12:40:47.933202+0000 (oldest deadline 2024-09-21T12:41:13.233094+0000) 2024-09-21T12:41:19.238 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:18 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:18.950+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:48.892221+0000 front 2024-09-21T12:40:48.892179+0000 (oldest deadline 2024-09-21T12:41:12.992108+0000) 2024-09-21T12:41:19.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:18 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:18.837+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:19.738 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:19 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:19.442+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.826418+0000 front 2024-09-21T12:40:45.826280+0000 (oldest deadline 2024-09-21T12:41:10.526217+0000) 2024-09-21T12:41:19.738 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:19 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:19.442+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.527419+0000 front 2024-09-21T12:40:57.527370+0000 (oldest deadline 2024-09-21T12:41:19.227279+0000) 2024-09-21T12:41:20.238 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:19 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:19.951+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:48.892221+0000 front 2024-09-21T12:40:48.892179+0000 (oldest deadline 2024-09-21T12:41:12.992108+0000) 2024-09-21T12:41:20.238 INFO:journalctl@ceph.osd.4.smithi183.stdout:Sep 21 12:41:19 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-4[37819]: 2024-09-21T12:41:19.882+0000 7f57d0341640 -1 osd.4 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:47.933219+0000 front 2024-09-21T12:40:47.933202+0000 (oldest deadline 2024-09-21T12:41:13.233094+0000) 2024-09-21T12:41:20.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:19 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:19.842+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:20.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:19 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:19.842+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.534184+0000 front 2024-09-21T12:40:57.534200+0000 (oldest deadline 2024-09-21T12:41:19.834191+0000) 2024-09-21T12:41:20.738 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:20 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:20.420+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.826418+0000 front 2024-09-21T12:40:45.826280+0000 (oldest deadline 2024-09-21T12:41:10.526217+0000) 2024-09-21T12:41:20.738 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:20 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:20.420+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.527419+0000 front 2024-09-21T12:40:57.527370+0000 (oldest deadline 2024-09-21T12:41:19.227279+0000) 2024-09-21T12:41:21.238 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:20 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:20.917+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:48.892221+0000 front 2024-09-21T12:40:48.892179+0000 (oldest deadline 2024-09-21T12:41:12.992108+0000) 2024-09-21T12:41:21.239 INFO:journalctl@ceph.osd.4.smithi183.stdout:Sep 21 12:41:20 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-4[37819]: 2024-09-21T12:41:20.928+0000 7f57d0341640 -1 osd.4 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:47.933219+0000 front 2024-09-21T12:40:47.933202+0000 (oldest deadline 2024-09-21T12:41:13.233094+0000) 2024-09-21T12:41:21.239 INFO:journalctl@ceph.osd.4.smithi183.stdout:Sep 21 12:41:20 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-4[37819]: 2024-09-21T12:41:20.928+0000 7f57d0341640 -1 osd.4 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:55.534121+0000 front 2024-09-21T12:40:55.534175+0000 (oldest deadline 2024-09-21T12:41:20.734506+0000) 2024-09-21T12:41:21.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:20 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:20.833+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:21.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:20 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:20.833+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.534184+0000 front 2024-09-21T12:40:57.534200+0000 (oldest deadline 2024-09-21T12:41:19.834191+0000) 2024-09-21T12:41:21.738 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:21 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:21.434+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.826418+0000 front 2024-09-21T12:40:45.826280+0000 (oldest deadline 2024-09-21T12:41:10.526217+0000) 2024-09-21T12:41:21.738 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:21 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:21.434+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.527419+0000 front 2024-09-21T12:40:57.527370+0000 (oldest deadline 2024-09-21T12:41:19.227279+0000) 2024-09-21T12:41:22.238 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:21 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:21.964+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:48.892221+0000 front 2024-09-21T12:40:48.892179+0000 (oldest deadline 2024-09-21T12:41:12.992108+0000) 2024-09-21T12:41:22.239 INFO:journalctl@ceph.osd.4.smithi183.stdout:Sep 21 12:41:21 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-4[37819]: 2024-09-21T12:41:21.935+0000 7f57d0341640 -1 osd.4 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:47.933219+0000 front 2024-09-21T12:40:47.933202+0000 (oldest deadline 2024-09-21T12:41:13.233094+0000) 2024-09-21T12:41:22.239 INFO:journalctl@ceph.osd.4.smithi183.stdout:Sep 21 12:41:21 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-4[37819]: 2024-09-21T12:41:21.935+0000 7f57d0341640 -1 osd.4 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:55.534121+0000 front 2024-09-21T12:40:55.534175+0000 (oldest deadline 2024-09-21T12:41:20.734506+0000) 2024-09-21T12:41:22.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:21 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:21.878+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:22.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:21 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:21.878+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.534184+0000 front 2024-09-21T12:40:57.534200+0000 (oldest deadline 2024-09-21T12:41:19.834191+0000) 2024-09-21T12:41:22.503 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:22 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:22.391+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.826418+0000 front 2024-09-21T12:40:45.826280+0000 (oldest deadline 2024-09-21T12:41:10.526217+0000) 2024-09-21T12:41:22.503 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:22 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:22.391+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.527419+0000 front 2024-09-21T12:40:57.527370+0000 (oldest deadline 2024-09-21T12:41:19.227279+0000) 2024-09-21T12:41:23.238 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:22 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:22.946+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:48.892221+0000 front 2024-09-21T12:40:48.892179+0000 (oldest deadline 2024-09-21T12:41:12.992108+0000) 2024-09-21T12:41:23.238 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:22 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:22.946+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:52.992665+0000 front 2024-09-21T12:40:52.992650+0000 (oldest deadline 2024-09-21T12:41:22.392957+0000) 2024-09-21T12:41:23.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:22 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:22.913+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:23.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:22 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:22.913+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.534184+0000 front 2024-09-21T12:40:57.534200+0000 (oldest deadline 2024-09-21T12:41:19.834191+0000) 2024-09-21T12:41:23.559 INFO:journalctl@ceph.osd.4.smithi183.stdout:Sep 21 12:41:23 smithi183 podman[153649]: 2024-09-21 12:41:23.257834903 +0000 UTC m=+5.511721725 container died eca3e821edd9d5f35575c011757ae144ade8ca88882d52c38aa74bd73449d4b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-4, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3) 2024-09-21T12:41:23.560 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:23 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:23.382+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.826418+0000 front 2024-09-21T12:40:45.826280+0000 (oldest deadline 2024-09-21T12:41:10.526217+0000) 2024-09-21T12:41:23.560 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:23 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:23.382+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.527419+0000 front 2024-09-21T12:40:57.527370+0000 (oldest deadline 2024-09-21T12:41:19.227279+0000) 2024-09-21T12:41:23.951 INFO:journalctl@ceph.osd.4.smithi183.stdout:Sep 21 12:41:23 smithi183 podman[153649]: 2024-09-21 12:41:23.722670882 +0000 UTC m=+5.976557701 container cleanup eca3e821edd9d5f35575c011757ae144ade8ca88882d52c38aa74bd73449d4b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-4, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, ceph=True, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-21T12:41:23.952 INFO:journalctl@ceph.osd.4.smithi183.stdout:Sep 21 12:41:23 smithi183 bash[153649]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-4 2024-09-21T12:41:24.238 INFO:journalctl@ceph.osd.4.smithi183.stdout:Sep 21 12:41:23 smithi183 podman[153680]: 2024-09-21 12:41:23.982698983 +0000 UTC m=+0.722920781 container remove eca3e821edd9d5f35575c011757ae144ade8ca88882d52c38aa74bd73449d4b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-4, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-21T12:41:24.238 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:23 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:23.978+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:48.892221+0000 front 2024-09-21T12:40:48.892179+0000 (oldest deadline 2024-09-21T12:41:12.992108+0000) 2024-09-21T12:41:24.238 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:23 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:23.978+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:52.992665+0000 front 2024-09-21T12:40:52.992650+0000 (oldest deadline 2024-09-21T12:41:22.392957+0000) 2024-09-21T12:41:24.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:23 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:23.950+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:24.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:23 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:23.950+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.534184+0000 front 2024-09-21T12:40:57.534200+0000 (oldest deadline 2024-09-21T12:41:19.834191+0000) 2024-09-21T12:41:24.553 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:24 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:24.405+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.826418+0000 front 2024-09-21T12:40:45.826280+0000 (oldest deadline 2024-09-21T12:41:10.526217+0000) 2024-09-21T12:41:24.553 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:24 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:24.405+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.527419+0000 front 2024-09-21T12:40:57.527370+0000 (oldest deadline 2024-09-21T12:41:19.227279+0000) 2024-09-21T12:41:24.553 INFO:journalctl@ceph.osd.4.smithi183.stdout:Sep 21 12:41:24 smithi183 podman[153775]: 2024-09-21 12:41:24.19569113 +0000 UTC m=+0.016009963 image pull 8f4d013db793795fef09ff5edf96e6b42ba8cd2c9158b2f0b7a5d7c1e657c063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472 2024-09-21T12:41:24.554 INFO:journalctl@ceph.osd.4.smithi183.stdout:Sep 21 12:41:24 smithi183 podman[153775]: 2024-09-21 12:41:24.325378862 +0000 UTC m=+0.145697717 container create 723aaef570dfc4c1d41671db5166e832625c2eabd0b1c3b2cca2e8a244c97ed6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T12:41:24.815 INFO:journalctl@ceph.osd.4.smithi183.stdout:Sep 21 12:41:24 smithi183 podman[153775]: 2024-09-21 12:41:24.552648183 +0000 UTC m=+0.372967011 container init 723aaef570dfc4c1d41671db5166e832625c2eabd0b1c3b2cca2e8a244c97ed6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-4-deactivate, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-21T12:41:24.815 INFO:journalctl@ceph.osd.4.smithi183.stdout:Sep 21 12:41:24 smithi183 podman[153775]: 2024-09-21 12:41:24.557637948 +0000 UTC m=+0.377956767 container start 723aaef570dfc4c1d41671db5166e832625c2eabd0b1c3b2cca2e8a244c97ed6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-4-deactivate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-21T12:41:24.815 INFO:journalctl@ceph.osd.4.smithi183.stdout:Sep 21 12:41:24 smithi183 podman[153775]: 2024-09-21 12:41:24.669309916 +0000 UTC m=+0.489628742 container attach 723aaef570dfc4c1d41671db5166e832625c2eabd0b1c3b2cca2e8a244c97ed6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-4-deactivate, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-21T12:41:25.107 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:24 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:24.993+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:48.892221+0000 front 2024-09-21T12:40:48.892179+0000 (oldest deadline 2024-09-21T12:41:12.992108+0000) 2024-09-21T12:41:25.107 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:24 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:24.993+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:52.992665+0000 front 2024-09-21T12:40:52.992650+0000 (oldest deadline 2024-09-21T12:41:22.392957+0000) 2024-09-21T12:41:25.107 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:24 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:24.968+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:25.107 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:24 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:24.968+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.534184+0000 front 2024-09-21T12:40:57.534200+0000 (oldest deadline 2024-09-21T12:41:19.834191+0000) 2024-09-21T12:41:25.107 INFO:journalctl@ceph.osd.4.smithi183.stdout:Sep 21 12:41:24 smithi183 podman[153775]: 2024-09-21 12:41:24.814045024 +0000 UTC m=+0.634363848 container died 723aaef570dfc4c1d41671db5166e832625c2eabd0b1c3b2cca2e8a244c97ed6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-4-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T12:41:25.397 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@osd.4.service' 2024-09-21T12:41:25.437 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T12:41:25.438 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-21T12:41:25.438 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-21T12:41:25.438 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-e2ad975c-7812-11ef-baf6-efdc52797490@osd.5 2024-09-21T12:41:25.738 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:25.398+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.826418+0000 front 2024-09-21T12:40:45.826280+0000 (oldest deadline 2024-09-21T12:41:10.526217+0000) 2024-09-21T12:41:25.738 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:25.398+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.527419+0000 front 2024-09-21T12:40:57.527370+0000 (oldest deadline 2024-09-21T12:41:19.227279+0000) 2024-09-21T12:41:25.739 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:25 smithi183 systemd[1]: Stopping Ceph osd.5 for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:41:26.238 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:26.019+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:48.892221+0000 front 2024-09-21T12:40:48.892179+0000 (oldest deadline 2024-09-21T12:41:12.992108+0000) 2024-09-21T12:41:26.238 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:26.019+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:52.992665+0000 front 2024-09-21T12:40:52.992650+0000 (oldest deadline 2024-09-21T12:41:22.392957+0000) 2024-09-21T12:41:26.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:25.971+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:26.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:25.971+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.534184+0000 front 2024-09-21T12:40:57.534200+0000 (oldest deadline 2024-09-21T12:41:19.834191+0000) 2024-09-21T12:41:26.239 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:25.771+0000 7fa16cc85640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T12:41:26.239 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:25.771+0000 7fa16cc85640 -1 osd.5 518 *** Got signal Terminated *** 2024-09-21T12:41:26.239 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:25 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:25.771+0000 7fa16cc85640 -1 osd.5 518 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T12:41:26.738 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:26.442+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.826418+0000 front 2024-09-21T12:40:45.826280+0000 (oldest deadline 2024-09-21T12:41:10.526217+0000) 2024-09-21T12:41:26.738 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:26.442+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.527419+0000 front 2024-09-21T12:40:57.527370+0000 (oldest deadline 2024-09-21T12:41:19.227279+0000) 2024-09-21T12:41:27.238 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:27 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:27.061+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:48.892221+0000 front 2024-09-21T12:40:48.892179+0000 (oldest deadline 2024-09-21T12:41:12.992108+0000) 2024-09-21T12:41:27.238 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:27 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:27.061+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:52.992665+0000 front 2024-09-21T12:40:52.992650+0000 (oldest deadline 2024-09-21T12:41:22.392957+0000) 2024-09-21T12:41:27.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:26.960+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:27.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:26.960+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.534184+0000 front 2024-09-21T12:40:57.534200+0000 (oldest deadline 2024-09-21T12:41:19.834191+0000) 2024-09-21T12:41:27.239 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:41:26 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=cleanup t=2024-09-21T12:41:26.971232429Z level=info msg="Completed cleanup jobs" duration=12.177542ms 2024-09-21T12:41:27.239 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:41:27 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=plugins.update.checker t=2024-09-21T12:41:27.105703415Z level=info msg="Update check succeeded" duration=68.66392ms 2024-09-21T12:41:27.738 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:27 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:27.443+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.826418+0000 front 2024-09-21T12:40:45.826280+0000 (oldest deadline 2024-09-21T12:41:10.526217+0000) 2024-09-21T12:41:27.738 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:27 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:27.443+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.527419+0000 front 2024-09-21T12:40:57.527370+0000 (oldest deadline 2024-09-21T12:41:19.227279+0000) 2024-09-21T12:41:28.238 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:28 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:28.062+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:48.892221+0000 front 2024-09-21T12:40:48.892179+0000 (oldest deadline 2024-09-21T12:41:12.992108+0000) 2024-09-21T12:41:28.238 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:28 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:28.062+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:52.992665+0000 front 2024-09-21T12:40:52.992650+0000 (oldest deadline 2024-09-21T12:41:22.392957+0000) 2024-09-21T12:41:28.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:27 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:27.951+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:28.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:27 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:27.951+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.534184+0000 front 2024-09-21T12:40:57.534200+0000 (oldest deadline 2024-09-21T12:41:19.834191+0000) 2024-09-21T12:41:28.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:27 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:27.951+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-21T12:41:01.936148+0000 front 2024-09-21T12:41:01.936188+0000 (oldest deadline 2024-09-21T12:41:27.236125+0000) 2024-09-21T12:41:28.738 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:28 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:28.397+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.826418+0000 front 2024-09-21T12:40:45.826280+0000 (oldest deadline 2024-09-21T12:41:10.526217+0000) 2024-09-21T12:41:28.738 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:28 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:28.397+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.527419+0000 front 2024-09-21T12:40:57.527370+0000 (oldest deadline 2024-09-21T12:41:19.227279+0000) 2024-09-21T12:41:29.238 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:29 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:29.086+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:48.892221+0000 front 2024-09-21T12:40:48.892179+0000 (oldest deadline 2024-09-21T12:41:12.992108+0000) 2024-09-21T12:41:29.238 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:29 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:29.086+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:52.992665+0000 front 2024-09-21T12:40:52.992650+0000 (oldest deadline 2024-09-21T12:41:22.392957+0000) 2024-09-21T12:41:29.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:28 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:28.955+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:29.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:28 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:28.955+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.534184+0000 front 2024-09-21T12:40:57.534200+0000 (oldest deadline 2024-09-21T12:41:19.834191+0000) 2024-09-21T12:41:29.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:28 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:28.955+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-21T12:41:01.936148+0000 front 2024-09-21T12:41:01.936188+0000 (oldest deadline 2024-09-21T12:41:27.236125+0000) 2024-09-21T12:41:29.738 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:29 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:29.380+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.826418+0000 front 2024-09-21T12:40:45.826280+0000 (oldest deadline 2024-09-21T12:41:10.526217+0000) 2024-09-21T12:41:29.738 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:29 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:29.380+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.527419+0000 front 2024-09-21T12:40:57.527370+0000 (oldest deadline 2024-09-21T12:41:19.227279+0000) 2024-09-21T12:41:30.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:29 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:29.977+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:30.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:29 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:29.977+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.534184+0000 front 2024-09-21T12:40:57.534200+0000 (oldest deadline 2024-09-21T12:41:19.834191+0000) 2024-09-21T12:41:30.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:29 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:29.977+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-21T12:41:01.936148+0000 front 2024-09-21T12:41:01.936188+0000 (oldest deadline 2024-09-21T12:41:27.236125+0000) 2024-09-21T12:41:30.239 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:30 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:30.127+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:48.892221+0000 front 2024-09-21T12:40:48.892179+0000 (oldest deadline 2024-09-21T12:41:12.992108+0000) 2024-09-21T12:41:30.239 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:30 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:30.127+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:52.992665+0000 front 2024-09-21T12:40:52.992650+0000 (oldest deadline 2024-09-21T12:41:22.392957+0000) 2024-09-21T12:41:30.239 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:30 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:30.127+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-21T12:41:03.493921+0000 front 2024-09-21T12:41:03.493908+0000 (oldest deadline 2024-09-21T12:41:29.393725+0000) 2024-09-21T12:41:30.738 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:30 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:30.360+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.826418+0000 front 2024-09-21T12:40:45.826280+0000 (oldest deadline 2024-09-21T12:41:10.526217+0000) 2024-09-21T12:41:30.738 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:30 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:30.360+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.527419+0000 front 2024-09-21T12:40:57.527370+0000 (oldest deadline 2024-09-21T12:41:19.227279+0000) 2024-09-21T12:41:30.738 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:30 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5[43658]: 2024-09-21T12:41:30.360+0000 7fa16929a640 -1 osd.5 518 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-21T12:41:04.428465+0000 front 2024-09-21T12:41:04.428530+0000 (oldest deadline 2024-09-21T12:41:30.328479+0000) 2024-09-21T12:41:31.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:30 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:30.974+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:31.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:30 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:30.974+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.534184+0000 front 2024-09-21T12:40:57.534200+0000 (oldest deadline 2024-09-21T12:41:19.834191+0000) 2024-09-21T12:41:31.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:30 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:30.974+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-21T12:41:01.936148+0000 front 2024-09-21T12:41:01.936188+0000 (oldest deadline 2024-09-21T12:41:27.236125+0000) 2024-09-21T12:41:31.239 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:31 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:31.134+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:48.892221+0000 front 2024-09-21T12:40:48.892179+0000 (oldest deadline 2024-09-21T12:41:12.992108+0000) 2024-09-21T12:41:31.239 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:31 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:31.134+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:52.992665+0000 front 2024-09-21T12:40:52.992650+0000 (oldest deadline 2024-09-21T12:41:22.392957+0000) 2024-09-21T12:41:31.239 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:31 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:31.134+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-21T12:41:03.493921+0000 front 2024-09-21T12:41:03.493908+0000 (oldest deadline 2024-09-21T12:41:29.393725+0000) 2024-09-21T12:41:31.645 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:31 smithi183 podman[153905]: 2024-09-21 12:41:31.384175875 +0000 UTC m=+5.725126736 container died 7fefb9c82058b1fa8f0094aa2eb2ae0d9b2d9f0ec81a5c175b4cea251ed2be99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-21T12:41:31.927 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:31 smithi183 podman[153905]: 2024-09-21 12:41:31.742223355 +0000 UTC m=+6.083174217 container cleanup 7fefb9c82058b1fa8f0094aa2eb2ae0d9b2d9f0ec81a5c175b4cea251ed2be99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3) 2024-09-21T12:41:31.927 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:31 smithi183 bash[153905]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5 2024-09-21T12:41:32.238 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:32 smithi183 podman[153926]: 2024-09-21 12:41:32.01869144 +0000 UTC m=+0.628158502 container remove 7fefb9c82058b1fa8f0094aa2eb2ae0d9b2d9f0ec81a5c175b4cea251ed2be99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-21T12:41:32.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:31 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:31.924+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:32.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:31 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:31.924+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.534184+0000 front 2024-09-21T12:40:57.534200+0000 (oldest deadline 2024-09-21T12:41:19.834191+0000) 2024-09-21T12:41:32.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:31 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:31.924+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-21T12:41:01.936148+0000 front 2024-09-21T12:41:01.936188+0000 (oldest deadline 2024-09-21T12:41:27.236125+0000) 2024-09-21T12:41:32.239 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:32 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:32.156+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:48.892221+0000 front 2024-09-21T12:40:48.892179+0000 (oldest deadline 2024-09-21T12:41:12.992108+0000) 2024-09-21T12:41:32.239 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:32 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:32.156+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:52.992665+0000 front 2024-09-21T12:40:52.992650+0000 (oldest deadline 2024-09-21T12:41:22.392957+0000) 2024-09-21T12:41:32.239 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:32 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:32.156+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-21T12:41:03.493921+0000 front 2024-09-21T12:41:03.493908+0000 (oldest deadline 2024-09-21T12:41:29.393725+0000) 2024-09-21T12:41:32.607 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:32 smithi183 podman[154017]: 2024-09-21 12:41:32.253156376 +0000 UTC m=+0.016433464 image pull 8f4d013db793795fef09ff5edf96e6b42ba8cd2c9158b2f0b7a5d7c1e657c063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472 2024-09-21T12:41:32.607 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:32 smithi183 podman[154017]: 2024-09-21 12:41:32.386492642 +0000 UTC m=+0.149787009 container create c6b8417c11dee757cf2c2e95df47bdc7d293010c1b14d1a08c0e1b946b7f51ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5-deactivate, org.label-schema.build-date=20240731, ceph=True, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-21T12:41:32.891 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:32 smithi183 podman[154017]: 2024-09-21 12:41:32.606206442 +0000 UTC m=+0.369483536 container init c6b8417c11dee757cf2c2e95df47bdc7d293010c1b14d1a08c0e1b946b7f51ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T12:41:32.891 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:32 smithi183 podman[154017]: 2024-09-21 12:41:32.61172203 +0000 UTC m=+0.374999126 container start c6b8417c11dee757cf2c2e95df47bdc7d293010c1b14d1a08c0e1b946b7f51ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS) 2024-09-21T12:41:32.891 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:32 smithi183 podman[154017]: 2024-09-21 12:41:32.67210111 +0000 UTC m=+0.435378206 container attach c6b8417c11dee757cf2c2e95df47bdc7d293010c1b14d1a08c0e1b946b7f51ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5-deactivate, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T12:41:32.892 INFO:journalctl@ceph.osd.5.smithi183.stdout:Sep 21 12:41:32 smithi183 podman[154017]: 2024-09-21 12:41:32.79626924 +0000 UTC m=+0.559546328 container died c6b8417c11dee757cf2c2e95df47bdc7d293010c1b14d1a08c0e1b946b7f51ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3) 2024-09-21T12:41:33.143 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:32 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:32.889+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:33.143 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:32 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:32.889+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.534184+0000 front 2024-09-21T12:40:57.534200+0000 (oldest deadline 2024-09-21T12:41:19.834191+0000) 2024-09-21T12:41:33.143 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:32 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:32.889+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-21T12:41:01.936148+0000 front 2024-09-21T12:41:01.936188+0000 (oldest deadline 2024-09-21T12:41:27.236125+0000) 2024-09-21T12:41:33.355 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@osd.5.service' 2024-09-21T12:41:33.394 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T12:41:33.394 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-21T12:41:33.394 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-21T12:41:33.394 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-e2ad975c-7812-11ef-baf6-efdc52797490@osd.6 2024-09-21T12:41:33.455 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:33 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:33.140+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:48.892221+0000 front 2024-09-21T12:40:48.892179+0000 (oldest deadline 2024-09-21T12:41:12.992108+0000) 2024-09-21T12:41:33.455 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:33 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:33.140+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:52.992665+0000 front 2024-09-21T12:40:52.992650+0000 (oldest deadline 2024-09-21T12:41:22.392957+0000) 2024-09-21T12:41:33.455 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:33 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:33.140+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-21T12:41:03.493921+0000 front 2024-09-21T12:41:03.493908+0000 (oldest deadline 2024-09-21T12:41:29.393725+0000) 2024-09-21T12:41:33.732 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:33 smithi183 systemd[1]: Stopping Ceph osd.6 for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:41:33.988 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:33 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:33.847+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:33.988 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:33 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:33.847+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.534184+0000 front 2024-09-21T12:40:57.534200+0000 (oldest deadline 2024-09-21T12:41:19.834191+0000) 2024-09-21T12:41:33.988 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:33 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:33.847+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-21T12:41:01.936148+0000 front 2024-09-21T12:41:01.936188+0000 (oldest deadline 2024-09-21T12:41:27.236125+0000) 2024-09-21T12:41:33.989 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:33 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:33.729+0000 7f3be72f0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T12:41:33.989 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:33 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:33.729+0000 7f3be72f0640 -1 osd.6 518 *** Got signal Terminated *** 2024-09-21T12:41:33.989 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:33 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:33.729+0000 7f3be72f0640 -1 osd.6 518 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T12:41:34.488 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:34.134+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:48.892221+0000 front 2024-09-21T12:40:48.892179+0000 (oldest deadline 2024-09-21T12:41:12.992108+0000) 2024-09-21T12:41:34.489 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:34.134+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:52.992665+0000 front 2024-09-21T12:40:52.992650+0000 (oldest deadline 2024-09-21T12:41:22.392957+0000) 2024-09-21T12:41:34.489 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:34.134+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-21T12:41:03.493921+0000 front 2024-09-21T12:41:03.493908+0000 (oldest deadline 2024-09-21T12:41:29.393725+0000) 2024-09-21T12:41:34.489 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:41:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:41:34.262Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://172.21.15.114:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 172.21.15.114:8765: connect: connection refused" 2024-09-21T12:41:34.489 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:41:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:41:34.263Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.114:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.114:8765: connect: connection refused" 2024-09-21T12:41:34.489 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:41:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:41:34.265Z caller=refresh.go:90 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.114:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.114:8765: connect: connection refused" 2024-09-21T12:41:34.489 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:41:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:41:34.266Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.114:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.114:8765: connect: connection refused" 2024-09-21T12:41:34.489 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:41:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:41:34.269Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.114:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.114:8765: connect: connection refused" 2024-09-21T12:41:34.489 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:41:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:41:34.270Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.114:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.114:8765: connect: connection refused" 2024-09-21T12:41:34.489 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:41:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:41:34.271Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=smb msg="Unable to refresh target groups" err="Get \"http://172.21.15.114:8765/sd/prometheus/sd-config?service=smb\": dial tcp 172.21.15.114:8765: connect: connection refused" 2024-09-21T12:41:35.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:34.860+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:35.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:34.860+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.534184+0000 front 2024-09-21T12:40:57.534200+0000 (oldest deadline 2024-09-21T12:41:19.834191+0000) 2024-09-21T12:41:35.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:34.860+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-21T12:41:01.936148+0000 front 2024-09-21T12:41:01.936188+0000 (oldest deadline 2024-09-21T12:41:27.236125+0000) 2024-09-21T12:41:35.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:34 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:34.860+0000 7f23923f8640 -1 osd.7 518 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24403.0:1236 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e518) 2024-09-21T12:41:35.239 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:35 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:35.090+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:48.892221+0000 front 2024-09-21T12:40:48.892179+0000 (oldest deadline 2024-09-21T12:41:12.992108+0000) 2024-09-21T12:41:35.239 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:35 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:35.090+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:52.992665+0000 front 2024-09-21T12:40:52.992650+0000 (oldest deadline 2024-09-21T12:41:22.392957+0000) 2024-09-21T12:41:35.239 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:35 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:35.090+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-21T12:41:03.493921+0000 front 2024-09-21T12:41:03.493908+0000 (oldest deadline 2024-09-21T12:41:29.393725+0000) 2024-09-21T12:41:36.238 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:36 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:36.092+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:48.892221+0000 front 2024-09-21T12:40:48.892179+0000 (oldest deadline 2024-09-21T12:41:12.992108+0000) 2024-09-21T12:41:36.239 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:36 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:36.092+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:52.992665+0000 front 2024-09-21T12:40:52.992650+0000 (oldest deadline 2024-09-21T12:41:22.392957+0000) 2024-09-21T12:41:36.239 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:36 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:36.092+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-21T12:41:03.493921+0000 front 2024-09-21T12:41:03.493908+0000 (oldest deadline 2024-09-21T12:41:29.393725+0000) 2024-09-21T12:41:36.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:35 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:35.899+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:36.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:35 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:35.899+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.534184+0000 front 2024-09-21T12:40:57.534200+0000 (oldest deadline 2024-09-21T12:41:19.834191+0000) 2024-09-21T12:41:36.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:35 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:35.899+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-21T12:41:01.936148+0000 front 2024-09-21T12:41:01.936188+0000 (oldest deadline 2024-09-21T12:41:27.236125+0000) 2024-09-21T12:41:36.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:35 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:35.899+0000 7f23923f8640 -1 osd.7 518 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24403.0:1236 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e518) 2024-09-21T12:41:37.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:36 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:36.925+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:37.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:36 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:36.925+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.534184+0000 front 2024-09-21T12:40:57.534200+0000 (oldest deadline 2024-09-21T12:41:19.834191+0000) 2024-09-21T12:41:37.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:36 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:36.925+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-21T12:41:01.936148+0000 front 2024-09-21T12:41:01.936188+0000 (oldest deadline 2024-09-21T12:41:27.236125+0000) 2024-09-21T12:41:37.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:36 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:36.925+0000 7f23923f8640 -1 osd.7 518 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24403.0:1236 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e518) 2024-09-21T12:41:37.239 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:37 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:37.043+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:48.892221+0000 front 2024-09-21T12:40:48.892179+0000 (oldest deadline 2024-09-21T12:41:12.992108+0000) 2024-09-21T12:41:37.239 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:37 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:37.043+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:52.992665+0000 front 2024-09-21T12:40:52.992650+0000 (oldest deadline 2024-09-21T12:41:22.392957+0000) 2024-09-21T12:41:37.239 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:37 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:37.043+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-21T12:41:03.493921+0000 front 2024-09-21T12:41:03.493908+0000 (oldest deadline 2024-09-21T12:41:29.393725+0000) 2024-09-21T12:41:37.239 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:37 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:37.043+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6830 osd.3 since back 2024-09-21T12:41:12.294626+0000 front 2024-09-21T12:41:12.294551+0000 (oldest deadline 2024-09-21T12:41:36.994564+0000) 2024-09-21T12:41:38.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:37 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:37.922+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:38.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:37 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:37.922+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.534184+0000 front 2024-09-21T12:40:57.534200+0000 (oldest deadline 2024-09-21T12:41:19.834191+0000) 2024-09-21T12:41:38.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:37 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:37.922+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-21T12:41:01.936148+0000 front 2024-09-21T12:41:01.936188+0000 (oldest deadline 2024-09-21T12:41:27.236125+0000) 2024-09-21T12:41:38.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:37 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:37.922+0000 7f23923f8640 -1 osd.7 518 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24403.0:1236 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e518) 2024-09-21T12:41:38.239 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:38 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:38.052+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:48.892221+0000 front 2024-09-21T12:40:48.892179+0000 (oldest deadline 2024-09-21T12:41:12.992108+0000) 2024-09-21T12:41:38.239 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:38 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:38.052+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:52.992665+0000 front 2024-09-21T12:40:52.992650+0000 (oldest deadline 2024-09-21T12:41:22.392957+0000) 2024-09-21T12:41:38.239 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:38 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:38.052+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-21T12:41:03.493921+0000 front 2024-09-21T12:41:03.493908+0000 (oldest deadline 2024-09-21T12:41:29.393725+0000) 2024-09-21T12:41:38.239 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:38 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6[49555]: 2024-09-21T12:41:38.052+0000 7f3bdf8fd640 -1 osd.6 518 heartbeat_check: no reply from 172.21.15.114:6830 osd.3 since back 2024-09-21T12:41:12.294626+0000 front 2024-09-21T12:41:12.294551+0000 (oldest deadline 2024-09-21T12:41:36.994564+0000) 2024-09-21T12:41:39.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:38 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:38.880+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:39.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:38 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:38.880+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.534184+0000 front 2024-09-21T12:40:57.534200+0000 (oldest deadline 2024-09-21T12:41:19.834191+0000) 2024-09-21T12:41:39.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:38 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:38.880+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-21T12:41:01.936148+0000 front 2024-09-21T12:41:01.936188+0000 (oldest deadline 2024-09-21T12:41:27.236125+0000) 2024-09-21T12:41:39.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:38 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:38.880+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6830 osd.3 since back 2024-09-21T12:41:08.937139+0000 front 2024-09-21T12:41:08.937108+0000 (oldest deadline 2024-09-21T12:41:38.337585+0000) 2024-09-21T12:41:39.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:38 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:38.880+0000 7f23923f8640 -1 osd.7 518 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24403.0:1236 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e518) 2024-09-21T12:41:39.540 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:39 smithi183 podman[154145]: 2024-09-21 12:41:39.278174706 +0000 UTC m=+5.641263705 container died 96769b2d1a98450c23973d0e80278c8bfd59edb92ed971efdee827fbef8704a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6, ceph=True, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-21T12:41:39.873 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:39 smithi183 podman[154145]: 2024-09-21 12:41:39.638236764 +0000 UTC m=+6.001325761 container cleanup 96769b2d1a98450c23973d0e80278c8bfd59edb92ed971efdee827fbef8704a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.license=GPLv2) 2024-09-21T12:41:39.874 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:39 smithi183 bash[154145]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6 2024-09-21T12:41:40.223 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:39 smithi183 podman[154167]: 2024-09-21 12:41:39.914650361 +0000 UTC m=+0.631853916 container remove 96769b2d1a98450c23973d0e80278c8bfd59edb92ed971efdee827fbef8704a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2) 2024-09-21T12:41:40.224 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:39 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:39.871+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:40.224 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:39 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:39.871+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.534184+0000 front 2024-09-21T12:40:57.534200+0000 (oldest deadline 2024-09-21T12:41:19.834191+0000) 2024-09-21T12:41:40.224 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:39 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:39.871+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-21T12:41:01.936148+0000 front 2024-09-21T12:41:01.936188+0000 (oldest deadline 2024-09-21T12:41:27.236125+0000) 2024-09-21T12:41:40.224 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:39 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:39.871+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6830 osd.3 since back 2024-09-21T12:41:08.937139+0000 front 2024-09-21T12:41:08.937108+0000 (oldest deadline 2024-09-21T12:41:38.337585+0000) 2024-09-21T12:41:40.224 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:39 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:39.872+0000 7f23923f8640 -1 osd.7 518 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24403.0:1236 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e518) 2024-09-21T12:41:40.489 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:40 smithi183 podman[154259]: 2024-09-21 12:41:40.125519618 +0000 UTC m=+0.016032364 image pull 8f4d013db793795fef09ff5edf96e6b42ba8cd2c9158b2f0b7a5d7c1e657c063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472 2024-09-21T12:41:40.489 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:40 smithi183 podman[154259]: 2024-09-21 12:41:40.240546852 +0000 UTC m=+0.131059587 container create 71fd0dcb3f4a9a7d0cff0beaadf0e0276a6d2761db029f8be13237b7d7cbcc21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T12:41:40.489 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:40 smithi183 podman[154259]: 2024-09-21 12:41:40.442765368 +0000 UTC m=+0.333278111 container init 71fd0dcb3f4a9a7d0cff0beaadf0e0276a6d2761db029f8be13237b7d7cbcc21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3) 2024-09-21T12:41:40.489 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:40 smithi183 podman[154259]: 2024-09-21 12:41:40.447541365 +0000 UTC m=+0.338054102 container start 71fd0dcb3f4a9a7d0cff0beaadf0e0276a6d2761db029f8be13237b7d7cbcc21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T12:41:40.912 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:40 smithi183 podman[154259]: 2024-09-21 12:41:40.534108865 +0000 UTC m=+0.424621620 container attach 71fd0dcb3f4a9a7d0cff0beaadf0e0276a6d2761db029f8be13237b7d7cbcc21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-21T12:41:40.913 INFO:journalctl@ceph.osd.6.smithi183.stdout:Sep 21 12:41:40 smithi183 podman[154259]: 2024-09-21 12:41:40.638646651 +0000 UTC m=+0.529159395 container died 71fd0dcb3f4a9a7d0cff0beaadf0e0276a6d2761db029f8be13237b7d7cbcc21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-6-deactivate, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-21T12:41:41.172 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:40 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:40.911+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:41.172 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:40 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:40.911+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.534184+0000 front 2024-09-21T12:40:57.534200+0000 (oldest deadline 2024-09-21T12:41:19.834191+0000) 2024-09-21T12:41:41.172 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:40 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:40.911+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-21T12:41:01.936148+0000 front 2024-09-21T12:41:01.936188+0000 (oldest deadline 2024-09-21T12:41:27.236125+0000) 2024-09-21T12:41:41.172 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:40 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:40.911+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6830 osd.3 since back 2024-09-21T12:41:08.937139+0000 front 2024-09-21T12:41:08.937108+0000 (oldest deadline 2024-09-21T12:41:38.337585+0000) 2024-09-21T12:41:41.196 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@osd.6.service' 2024-09-21T12:41:41.275 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T12:41:41.276 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-21T12:41:41.276 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-21T12:41:41.276 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-e2ad975c-7812-11ef-baf6-efdc52797490@osd.7 2024-09-21T12:41:41.491 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:41 smithi183 systemd[1]: Stopping Ceph osd.7 for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:41:41.916 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:41.625+0000 7f2399deb640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T12:41:41.916 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:41.625+0000 7f2399deb640 -1 osd.7 518 *** Got signal Terminated *** 2024-09-21T12:41:41.916 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:41.625+0000 7f2399deb640 -1 osd.7 518 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T12:41:42.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:41.914+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:42.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:41.914+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.534184+0000 front 2024-09-21T12:40:57.534200+0000 (oldest deadline 2024-09-21T12:41:19.834191+0000) 2024-09-21T12:41:42.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:41.914+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-21T12:41:01.936148+0000 front 2024-09-21T12:41:01.936188+0000 (oldest deadline 2024-09-21T12:41:27.236125+0000) 2024-09-21T12:41:42.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:41 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:41.914+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6830 osd.3 since back 2024-09-21T12:41:08.937139+0000 front 2024-09-21T12:41:08.937108+0000 (oldest deadline 2024-09-21T12:41:38.337585+0000) 2024-09-21T12:41:43.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:42 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:42.907+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:43.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:42 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:42.907+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.534184+0000 front 2024-09-21T12:40:57.534200+0000 (oldest deadline 2024-09-21T12:41:19.834191+0000) 2024-09-21T12:41:43.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:42 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:42.907+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-21T12:41:01.936148+0000 front 2024-09-21T12:41:01.936188+0000 (oldest deadline 2024-09-21T12:41:27.236125+0000) 2024-09-21T12:41:43.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:42 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:42.907+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6830 osd.3 since back 2024-09-21T12:41:08.937139+0000 front 2024-09-21T12:41:08.937108+0000 (oldest deadline 2024-09-21T12:41:38.337585+0000) 2024-09-21T12:41:44.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:43 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:43.884+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:44.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:43 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:43.884+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.534184+0000 front 2024-09-21T12:40:57.534200+0000 (oldest deadline 2024-09-21T12:41:19.834191+0000) 2024-09-21T12:41:44.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:43 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:43.884+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-21T12:41:01.936148+0000 front 2024-09-21T12:41:01.936188+0000 (oldest deadline 2024-09-21T12:41:27.236125+0000) 2024-09-21T12:41:44.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:43 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:43.884+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6830 osd.3 since back 2024-09-21T12:41:08.937139+0000 front 2024-09-21T12:41:08.937108+0000 (oldest deadline 2024-09-21T12:41:38.337585+0000) 2024-09-21T12:41:45.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:44 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:44.897+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:45.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:44 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:44.897+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.534184+0000 front 2024-09-21T12:40:57.534200+0000 (oldest deadline 2024-09-21T12:41:19.834191+0000) 2024-09-21T12:41:45.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:44 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:44.897+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-21T12:41:01.936148+0000 front 2024-09-21T12:41:01.936188+0000 (oldest deadline 2024-09-21T12:41:27.236125+0000) 2024-09-21T12:41:45.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:44 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:44.897+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6830 osd.3 since back 2024-09-21T12:41:08.937139+0000 front 2024-09-21T12:41:08.937108+0000 (oldest deadline 2024-09-21T12:41:38.337585+0000) 2024-09-21T12:41:45.239 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:44 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:44.897+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.183:6804 osd.4 since back 2024-09-21T12:41:18.838205+0000 front 2024-09-21T12:41:18.838272+0000 (oldest deadline 2024-09-21T12:41:44.138302+0000) 2024-09-21T12:41:46.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:45 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:45.849+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-21T12:40:45.733198+0000 front 2024-09-21T12:40:45.733319+0000 (oldest deadline 2024-09-21T12:41:11.633215+0000) 2024-09-21T12:41:46.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:45 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:45.849+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-21T12:40:57.534184+0000 front 2024-09-21T12:40:57.534200+0000 (oldest deadline 2024-09-21T12:41:19.834191+0000) 2024-09-21T12:41:46.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:45 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:45.849+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-21T12:41:01.936148+0000 front 2024-09-21T12:41:01.936188+0000 (oldest deadline 2024-09-21T12:41:27.236125+0000) 2024-09-21T12:41:46.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:45 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:45.849+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.114:6830 osd.3 since back 2024-09-21T12:41:08.937139+0000 front 2024-09-21T12:41:08.937108+0000 (oldest deadline 2024-09-21T12:41:38.337585+0000) 2024-09-21T12:41:46.238 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:45 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7[55441]: 2024-09-21T12:41:45.849+0000 7f23923f8640 -1 osd.7 518 heartbeat_check: no reply from 172.21.15.183:6804 osd.4 since back 2024-09-21T12:41:18.838205+0000 front 2024-09-21T12:41:18.838272+0000 (oldest deadline 2024-09-21T12:41:44.138302+0000) 2024-09-21T12:41:47.359 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:47 smithi183 podman[154386]: 2024-09-21 12:41:47.093224543 +0000 UTC m=+5.595203017 container died 2fd941f10bb84af5c270038efb24439261f2cedfff3542937d0e461f44c2de80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2) 2024-09-21T12:41:47.691 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:47 smithi183 podman[154386]: 2024-09-21 12:41:47.448147249 +0000 UTC m=+5.950125751 container cleanup 2fd941f10bb84af5c270038efb24439261f2cedfff3542937d0e461f44c2de80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-21T12:41:47.692 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:47 smithi183 bash[154386]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7 2024-09-21T12:41:47.988 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:47 smithi183 podman[154407]: 2024-09-21 12:41:47.690823244 +0000 UTC m=+0.598422823 container remove 2fd941f10bb84af5c270038efb24439261f2cedfff3542937d0e461f44c2de80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-21T12:41:48.361 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:48 smithi183 podman[154501]: 2024-09-21 12:41:47.954797071 +0000 UTC m=+0.015598372 image pull 8f4d013db793795fef09ff5edf96e6b42ba8cd2c9158b2f0b7a5d7c1e657c063 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472 2024-09-21T12:41:48.361 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:48 smithi183 podman[154501]: 2024-09-21 12:41:48.066563332 +0000 UTC m=+0.127364661 container create be932605161710dc27aa4ca69db75f616db97753a08c7a8c827edbfc21aab56a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-21T12:41:48.361 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:48 smithi183 podman[154501]: 2024-09-21 12:41:48.276722233 +0000 UTC m=+0.337523520 container init be932605161710dc27aa4ca69db75f616db97753a08c7a8c827edbfc21aab56a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7-deactivate, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T12:41:48.361 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:48 smithi183 podman[154501]: 2024-09-21 12:41:48.281687595 +0000 UTC m=+0.342488888 container start be932605161710dc27aa4ca69db75f616db97753a08c7a8c827edbfc21aab56a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, maintainer=Guillaume Abrioux ) 2024-09-21T12:41:48.361 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:48 smithi183 podman[154501]: 2024-09-21 12:41:48.360104137 +0000 UTC m=+0.420905433 container attach be932605161710dc27aa4ca69db75f616db97753a08c7a8c827edbfc21aab56a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.schema-version=1.0) 2024-09-21T12:41:48.738 INFO:journalctl@ceph.osd.7.smithi183.stdout:Sep 21 12:41:48 smithi183 podman[154501]: 2024-09-21 12:41:48.465617089 +0000 UTC m=+0.526418387 container died be932605161710dc27aa4ca69db75f616db97753a08c7a8c827edbfc21aab56a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-osd-7-deactivate, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T12:41:49.050 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@osd.7.service' 2024-09-21T12:41:49.088 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T12:41:49.088 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-21T12:41:49.088 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-09-21T12:41:49.088 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-e2ad975c-7812-11ef-baf6-efdc52797490@rgw.foo.a 2024-09-21T12:41:49.408 INFO:journalctl@ceph.rgw.foo.a.smithi114.stdout:Sep 21 12:41:49 smithi114 systemd[1]: Stopping Ceph rgw.foo.a for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:41:49.774 INFO:journalctl@ceph.rgw.foo.a.smithi114.stdout:Sep 21 12:41:49 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-rgw-foo-a[67501]: 2024-09-21T12:41:49.406+0000 7f71b6a13640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T12:41:49.774 INFO:journalctl@ceph.rgw.foo.a.smithi114.stdout:Sep 21 12:41:49 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-rgw-foo-a[67501]: 2024-09-21T12:41:49.406+0000 7f71ba27f880 -1 shutting down 2024-09-21T12:41:59.714 INFO:journalctl@ceph.rgw.foo.a.smithi114.stdout:Sep 21 12:41:59 smithi114 bash[219043]: time="2024-09-21T12:41:59Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-e2ad975c-7812-11ef-baf6-efdc52797490-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-09-21T12:41:59.714 INFO:journalctl@ceph.rgw.foo.a.smithi114.stdout:Sep 21 12:41:59 smithi114 podman[219043]: 2024-09-21 12:41:59.429748062 +0000 UTC m=+10.146579977 container died c3e2161c3c5d822b12fc83776b098fd2bedb1a447c0de82cf6be9a569b0416f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-rgw-foo-a, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=) 2024-09-21T12:42:00.024 INFO:journalctl@ceph.rgw.foo.a.smithi114.stdout:Sep 21 12:41:59 smithi114 podman[219043]: 2024-09-21 12:41:59.887122299 +0000 UTC m=+10.603954207 container cleanup c3e2161c3c5d822b12fc83776b098fd2bedb1a447c0de82cf6be9a569b0416f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-rgw-foo-a, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-21T12:42:00.024 INFO:journalctl@ceph.rgw.foo.a.smithi114.stdout:Sep 21 12:41:59 smithi114 bash[219043]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490-rgw-foo-a 2024-09-21T12:42:00.321 INFO:journalctl@ceph.rgw.foo.a.smithi114.stdout:Sep 21 12:42:00 smithi114 podman[219077]: 2024-09-21 12:42:00.213678926 +0000 UTC m=+0.775586218 container remove c3e2161c3c5d822b12fc83776b098fd2bedb1a447c0de82cf6be9a569b0416f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-rgw-foo-a, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T12:42:00.321 INFO:journalctl@ceph.rgw.foo.a.smithi114.stdout:Sep 21 12:42:00 smithi114 systemd[1]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-09-21T12:42:00.339 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@rgw.foo.a.service' 2024-09-21T12:42:00.375 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T12:42:00.376 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-09-21T12:42:00.376 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-21T12:42:00.376 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-e2ad975c-7812-11ef-baf6-efdc52797490@prometheus.a 2024-09-21T12:42:00.726 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:42:00 smithi183 systemd[1]: Stopping Ceph prometheus.a for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:42:00.983 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:42:00 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:42:00.725Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-21T12:42:00.983 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:42:00 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:42:00.726Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-21T12:42:00.983 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:42:00 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:42:00.726Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-21T12:42:00.983 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:42:00 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:42:00.726Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-21T12:42:00.983 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:42:00 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:42:00.726Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-21T12:42:00.983 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:42:00 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:42:00.726Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-21T12:42:00.984 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:42:00 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:42:00.726Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-21T12:42:00.984 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:42:00 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:42:00.726Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-21T12:42:00.984 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:42:00 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:42:00.726Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-21T12:42:00.984 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:42:00 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:42:00.727Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-21T12:42:00.984 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:42:00 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:42:00.727Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-21T12:42:00.984 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:42:00 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a[68673]: ts=2024-09-21T12:42:00.727Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-21T12:42:00.984 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:42:00 smithi183 podman[154629]: 2024-09-21 12:42:00.736306011 +0000 UTC m=+0.119651668 container died cb6f35f47c5b55547c82f9315d527a12248fa8f6c0293d11c3a09cfce41966e4 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:42:01.238 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:42:01 smithi183 podman[154629]: 2024-09-21 12:42:01.12275964 +0000 UTC m=+0.506105287 container cleanup cb6f35f47c5b55547c82f9315d527a12248fa8f6c0293d11c3a09cfce41966e4 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:42:01.238 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Sep 21 12:42:01 smithi183 bash[154629]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490-prometheus-a 2024-09-21T12:42:01.534 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ad975c-7812-11ef-baf6-efdc52797490@prometheus.a.service' 2024-09-21T12:42:01.571 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T12:42:01.571 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-21T12:42:01.571 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e2ad975c-7812-11ef-baf6-efdc52797490 --force --keep-logs 2024-09-21T12:42:01.773 INFO:teuthology.orchestra.run.smithi114.stdout:Deleting cluster with fsid: e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T12:42:03.655 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:42:03 smithi114 systemd[1]: Stopping Ceph alertmanager.a for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:42:03.928 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:42:03 smithi114 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a[85236]: ts=2024-09-21T12:42:03.653Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-21T12:42:03.929 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:42:03 smithi114 podman[219568]: 2024-09-21 12:42:03.69457404 +0000 UTC m=+0.162164657 container died 9a9b92f1127acee778994c000cc07647c2820e8abac5f18d3ca6d6209e105fd1 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T12:42:04.274 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:42:04 smithi114 podman[219568]: 2024-09-21 12:42:04.101395136 +0000 UTC m=+0.568985784 container cleanup 9a9b92f1127acee778994c000cc07647c2820e8abac5f18d3ca6d6209e105fd1 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T12:42:04.274 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:42:04 smithi114 bash[219568]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a 2024-09-21T12:42:04.758 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:42:04 smithi114 podman[219585]: 2024-09-21 12:42:04.361015378 +0000 UTC m=+0.661112603 container remove 9a9b92f1127acee778994c000cc07647c2820e8abac5f18d3ca6d6209e105fd1 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T12:42:04.759 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:42:04 smithi114 podman[219585]: 2024-09-21 12:42:04.436694742 +0000 UTC m=+0.736791974 volume remove c6e2b695ba50f3c7f023453a430b0d38c59732691ca126ab94c3cf9ac39bd566 2024-09-21T12:42:04.759 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:42:04 smithi114 systemd[1]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-21T12:42:04.759 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:42:04 smithi114 systemd[1]: Stopped Ceph alertmanager.a for e2ad975c-7812-11ef-baf6-efdc52797490. 2024-09-21T12:42:04.759 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 21 12:42:04 smithi114 systemd[1]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490@alertmanager.a.service: Consumed 1.666s CPU time. 2024-09-21T12:42:05.024 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:42:04 smithi114 systemd[1]: Stopping Ceph node-exporter.a for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:42:05.025 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:42:05 smithi114 podman[219705]: 2024-09-21 12:42:05.003212382 +0000 UTC m=+0.107464065 container died e900873487e17a09d8531d17b30f0fbb191b9f26ab1a5a399d2b6eb73ebcc78f (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T12:42:05.767 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:42:05 smithi114 podman[219705]: 2024-09-21 12:42:05.481595806 +0000 UTC m=+0.585847478 container cleanup e900873487e17a09d8531d17b30f0fbb191b9f26ab1a5a399d2b6eb73ebcc78f (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T12:42:05.767 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:42:05 smithi114 bash[219705]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a 2024-09-21T12:42:06.024 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:42:05 smithi114 podman[219723]: 2024-09-21 12:42:05.766245535 +0000 UTC m=+0.762726446 container remove e900873487e17a09d8531d17b30f0fbb191b9f26ab1a5a399d2b6eb73ebcc78f (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T12:42:06.024 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:42:05 smithi114 systemd[1]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-21T12:42:06.024 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:42:05 smithi114 systemd[1]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-21T12:42:06.024 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:42:05 smithi114 systemd[1]: Stopped Ceph node-exporter.a for e2ad975c-7812-11ef-baf6-efdc52797490. 2024-09-21T12:42:06.024 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 21 12:42:05 smithi114 systemd[1]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490@node-exporter.a.service: Consumed 4.851s CPU time. 2024-09-21T12:42:06.589 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e2ad975c-7812-11ef-baf6-efdc52797490 --force --keep-logs 2024-09-21T12:42:06.791 INFO:teuthology.orchestra.run.smithi183.stdout:Deleting cluster with fsid: e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T12:42:08.488 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:42:08 smithi183 systemd[1]: Stopping Ceph iscsi.iscsi.a for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:42:08.489 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:42:08 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a[60370]: debug Shutdown received 2024-09-21T12:42:18.561 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:42:18 smithi183 bash[155060]: time="2024-09-21T12:42:18Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-09-21T12:42:18.561 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:42:18 smithi183 podman[155060]: 2024-09-21 12:42:18.309912798 +0000 UTC m=+10.126477851 container died a66333b3bcc4f3e5c8f2081bf5087d59688d55c891deb99605273d0c03cfd592 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T12:42:18.886 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:42:18 smithi183 podman[155060]: 2024-09-21 12:42:18.725802343 +0000 UTC m=+10.542367395 container cleanup a66333b3bcc4f3e5c8f2081bf5087d59688d55c891deb99605273d0c03cfd592 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, io.buildah.version=1.37.2) 2024-09-21T12:42:18.886 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:42:18 smithi183 bash[155060]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a 2024-09-21T12:42:19.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:42:18 smithi183 podman[155084]: 2024-09-21 12:42:18.885564786 +0000 UTC m=+0.573871406 container remove a66333b3bcc4f3e5c8f2081bf5087d59688d55c891deb99605273d0c03cfd592 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3855b8c483c85027fea24c112434955a9e8e6f629214baaca7b99a8fabb55472, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-iscsi-iscsi-a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-21-0813-5c1cdc3, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-21T12:42:19.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:42:18 smithi183 systemd[1]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-09-21T12:42:19.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:42:19 smithi183 systemd[1]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-09-21T12:42:19.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:42:19 smithi183 systemd[1]: Stopped Ceph iscsi.iscsi.a for e2ad975c-7812-11ef-baf6-efdc52797490. 2024-09-21T12:42:19.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Sep 21 12:42:19 smithi183 systemd[1]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490@iscsi.iscsi.a.service: Consumed 2.631s CPU time. 2024-09-21T12:42:20.988 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:42:20 smithi183 systemd[1]: Stopping Ceph grafana.a for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:42:20.988 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:42:20 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=server t=2024-09-21T12:42:20.803256383Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-09-21T12:42:20.989 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:42:20 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=tracing t=2024-09-21T12:42:20.803634415Z level=info msg="Closing tracing" 2024-09-21T12:42:20.989 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:42:20 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=grafana-apiserver t=2024-09-21T12:42:20.803711541Z level=info msg="StorageObjectCountTracker pruner is exiting" 2024-09-21T12:42:20.989 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:42:20 smithi183 ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a[63267]: logger=ticker t=2024-09-21T12:42:20.803764678Z level=info msg=stopped last_tick=2024-09-21T12:42:20Z 2024-09-21T12:42:20.989 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:42:20 smithi183 podman[155361]: 2024-09-21 12:42:20.813685876 +0000 UTC m=+0.103934904 container died 439a4d10dbdea341607cd280b3b6402e0f43128eef27b63c09047fb0224a8d09 (image=quay.io/ceph/grafana:10.4.8, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a, maintainer=Grafana Labs ) 2024-09-21T12:42:21.452 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:42:21 smithi183 podman[155361]: 2024-09-21 12:42:21.242195778 +0000 UTC m=+0.532444791 container cleanup 439a4d10dbdea341607cd280b3b6402e0f43128eef27b63c09047fb0224a8d09 (image=quay.io/ceph/grafana:10.4.8, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a, maintainer=Grafana Labs ) 2024-09-21T12:42:21.453 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:42:21 smithi183 bash[155361]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a 2024-09-21T12:42:21.738 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:42:21 smithi183 podman[155378]: 2024-09-21 12:42:21.451728657 +0000 UTC m=+0.636835639 container remove 439a4d10dbdea341607cd280b3b6402e0f43128eef27b63c09047fb0224a8d09 (image=quay.io/ceph/grafana:10.4.8, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-grafana-a, maintainer=Grafana Labs ) 2024-09-21T12:42:21.738 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:42:21 smithi183 systemd[1]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490@grafana.a.service: Deactivated successfully. 2024-09-21T12:42:21.738 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:42:21 smithi183 systemd[1]: Stopped Ceph grafana.a for e2ad975c-7812-11ef-baf6-efdc52797490. 2024-09-21T12:42:21.738 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Sep 21 12:42:21 smithi183 systemd[1]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490@grafana.a.service: Consumed 14.632s CPU time. 2024-09-21T12:42:22.118 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:42:21 smithi183 systemd[1]: Stopping Ceph node-exporter.b for e2ad975c-7812-11ef-baf6-efdc52797490... 2024-09-21T12:42:22.414 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:42:22 smithi183 podman[155499]: 2024-09-21 12:42:22.117110254 +0000 UTC m=+0.116568086 container died 03eac822cd8878e9b4cff3808345c4315c1205d493e36c65894ba389e69633c4 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T12:42:22.697 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:42:22 smithi183 podman[155499]: 2024-09-21 12:42:22.512667306 +0000 UTC m=+0.512125143 container cleanup 03eac822cd8878e9b4cff3808345c4315c1205d493e36c65894ba389e69633c4 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T12:42:22.697 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:42:22 smithi183 bash[155499]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b 2024-09-21T12:42:22.988 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:42:22 smithi183 podman[155516]: 2024-09-21 12:42:22.696848086 +0000 UTC m=+0.583606121 container remove 03eac822cd8878e9b4cff3808345c4315c1205d493e36c65894ba389e69633c4 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-e2ad975c-7812-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T12:42:22.988 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:42:22 smithi183 systemd[1]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-21T12:42:22.988 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:42:22 smithi183 systemd[1]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-21T12:42:22.989 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:42:22 smithi183 systemd[1]: Stopped Ceph node-exporter.b for e2ad975c-7812-11ef-baf6-efdc52797490. 2024-09-21T12:42:22.989 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Sep 21 12:42:22 smithi183 systemd[1]: ceph-e2ad975c-7812-11ef-baf6-efdc52797490@node-exporter.b.service: Consumed 4.259s CPU time. 2024-09-21T12:42:23.584 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T12:42:23.616 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T12:42:23.643 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-21T12:42:23.645 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/crash to /home/teuthworker/archive/gabrioux-2024-09-21_10:07:29-orch:cephadm-wip-guits-main-2024-09-21-0813-distro-default-smithi/7913913/remote/smithi114/crash 2024-09-21T12:42:23.646 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/crash -- . 2024-09-21T12:42:23.704 INFO:teuthology.orchestra.run.smithi114.stderr:tar: /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T12:42:23.704 INFO:teuthology.orchestra.run.smithi114.stderr:tar: Error is not recoverable: exiting now 2024-09-21T12:42:23.706 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/crash to /home/teuthworker/archive/gabrioux-2024-09-21_10:07:29-orch:cephadm-wip-guits-main-2024-09-21-0813-distro-default-smithi/7913913/remote/smithi183/crash 2024-09-21T12:42:23.707 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/crash -- . 2024-09-21T12:42:23.745 INFO:teuthology.orchestra.run.smithi183.stderr:tar: /var/lib/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T12:42:23.745 INFO:teuthology.orchestra.run.smithi183.stderr:tar: Error is not recoverable: exiting now 2024-09-21T12:42:23.746 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-21T12:42:23.746 DEBUG:teuthology.orchestra.run.smithi114:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v CEPHADM_STRAY_DAEMON | egrep -v CEPHADM_FAILED_DAEMON | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-21T12:42:23.852 INFO:tasks.cephadm:Compressing logs... 2024-09-21T12:42:23.852 DEBUG:teuthology.orchestra.run.smithi114:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T12:42:23.854 DEBUG:teuthology.orchestra.run.smithi183:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T12:42:23.897 INFO:teuthology.orchestra.run.smithi114.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-osd.0.log 2024-09-21T12:42:23.897 INFO:teuthology.orchestra.run.smithi114.stderr:: No such file or directory 2024-09-21T12:42:23.897 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-osd.1.log 2024-09-21T12:42:23.897 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-volume.log 2024-09-21T12:42:23.897 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph.audit.log 2024-09-21T12:42:23.897 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-mon.c.log 2024-09-21T12:42:23.897 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph.cephadm.log 2024-09-21T12:42:23.897 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-mon.a.log 2024-09-21T12:42:23.897 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-osd.2.log 2024-09-21T12:42:23.897 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-mgr.y.log 2024-09-21T12:42:23.898 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph.log 2024-09-21T12:42:23.898 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-client.rgw.foo.a.log 2024-09-21T12:42:23.898 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph.log: gzip -5 --verbose -- /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-osd.3.log 2024-09-21T12:42:23.898 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/objectstore_tool.7.log 2024-09-21T12:42:23.898 INFO:teuthology.orchestra.run.smithi114.stderr: 59.0% -- replaced with /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-client.rgw.foo.a.log.gz 2024-09-21T12:42:23.898 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T12:42:23.905 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/objectstore_tool.7.log: /var/log/ceph/cephadm.log: 91.7% -- replaced with /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph.log.gz 2024-09-21T12:42:23.911 INFO:teuthology.orchestra.run.smithi183.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose: No such file or directory -- 2024-09-21T12:42:23.911 INFO:teuthology.orchestra.run.smithi183.stderr: /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/tcmu-runner.log 2024-09-21T12:42:23.911 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-osd.7.log 2024-09-21T12:42:23.912 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-volume.log 2024-09-21T12:42:23.913 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/tcmu-runner.log: 63.5% -- replaced with /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/tcmu-runner.log.gz 2024-09-21T12:42:23.913 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-osd.4.log 2024-09-21T12:42:23.913 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph.audit.log 2024-09-21T12:42:23.915 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph.cephadm.log 2024-09-21T12:42:23.915 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-osd.6.log 2024-09-21T12:42:23.915 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-osd.5.log 2024-09-21T12:42:23.915 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph.log 2024-09-21T12:42:23.915 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-mgr.x.log 2024-09-21T12:42:23.915 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph.log: gzip -5 --verbose -- /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-mon.b.log 2024-09-21T12:42:23.915 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T12:42:23.997 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-mon.b.log: /var/log/ceph/cephadm.log: 91.7% -- replaced with /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-mgr.x.log.gz 2024-09-21T12:42:24.007 INFO:teuthology.orchestra.run.smithi183.stderr: 92.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T12:42:24.013 INFO:teuthology.orchestra.run.smithi183.stderr: 79.8% -- replaced with /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph.cephadm.log.gz 2024-09-21T12:42:24.183 INFO:teuthology.orchestra.run.smithi183.stderr: 90.0% -- replaced with /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph.audit.log.gz 2024-09-21T12:42:24.184 INFO:teuthology.orchestra.run.smithi114.stderr: 89.2% -- replaced with /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph.cephadm.log.gz 2024-09-21T12:42:24.234 INFO:teuthology.orchestra.run.smithi183.stderr: 91.5% -- replaced with /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-volume.log.gz 2024-09-21T12:42:24.240 INFO:teuthology.orchestra.run.smithi114.stderr: 94.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T12:42:24.287 INFO:teuthology.orchestra.run.smithi183.stderr: 85.3% -- replaced with /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph.log.gz 2024-09-21T12:42:24.323 INFO:teuthology.orchestra.run.smithi114.stderr: 94.0% -- replaced with /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph.audit.log.gz 2024-09-21T12:42:24.946 INFO:teuthology.orchestra.run.smithi114.stderr: 91.5% -- replaced with /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-volume.log.gz 2024-09-21T12:42:25.062 INFO:teuthology.orchestra.run.smithi183.stderr: 91.0% -- replaced with /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-mon.b.log.gz 2024-09-21T12:42:26.017 INFO:teuthology.orchestra.run.smithi114.stderr: 90.1% -- replaced with /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-mgr.y.log.gz 2024-09-21T12:42:26.074 INFO:teuthology.orchestra.run.smithi114.stderr: 94.2% -- replaced with /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/objectstore_tool.7.log.gz 2024-09-21T12:42:29.002 INFO:teuthology.orchestra.run.smithi114.stderr: 91.0% -- replaced with /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-mon.c.log.gz 2024-09-21T12:42:44.521 INFO:teuthology.orchestra.run.smithi114.stderr: 90.4% -- replaced with /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-mon.a.log.gz 2024-09-21T12:43:46.101 INFO:teuthology.orchestra.run.smithi183.stderr: 93.6% -- replaced with /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-osd.4.log.gz 2024-09-21T12:44:12.005 INFO:teuthology.orchestra.run.smithi183.stderr: 93.9% -- replaced with /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-osd.6.log.gz 2024-09-21T12:45:20.327 INFO:teuthology.orchestra.run.smithi183.stderr: 94.0% -- replaced with /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-osd.5.log.gz 2024-09-21T12:45:56.731 INFO:teuthology.orchestra.run.smithi183.stderr: 93.3% -- replaced with /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-osd.7.log.gz 2024-09-21T12:45:56.733 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-09-21T12:45:56.733 INFO:teuthology.orchestra.run.smithi183.stderr:real 3m32.864s 2024-09-21T12:45:56.733 INFO:teuthology.orchestra.run.smithi183.stderr:user 4m10.875s 2024-09-21T12:45:56.733 INFO:teuthology.orchestra.run.smithi183.stderr:sys 0m10.532s 2024-09-21T12:56:29.291 INFO:teuthology.orchestra.run.smithi114.stderr: 93.4% -- replaced with /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-osd.2.log.gz 2024-09-21T13:01:49.727 INFO:teuthology.orchestra.run.smithi114.stderr: 93.0% -- replaced with /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-osd.0.log.gz 2024-09-21T13:02:03.758 INFO:teuthology.orchestra.run.smithi114.stderr: 93.0% -- replaced with /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-osd.1.log.gz 2024-09-21T13:02:39.050 INFO:teuthology.orchestra.run.smithi114.stderr: 93.2% -- replaced with /var/log/ceph/e2ad975c-7812-11ef-baf6-efdc52797490/ceph-osd.3.log.gz 2024-09-21T13:02:39.052 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-21T13:02:39.052 INFO:teuthology.orchestra.run.smithi114.stderr:real 20m15.184s 2024-09-21T13:02:39.052 INFO:teuthology.orchestra.run.smithi114.stderr:user 6m54.380s 2024-09-21T13:02:39.052 INFO:teuthology.orchestra.run.smithi114.stderr:sys 0m21.703s 2024-09-21T13:02:39.053 INFO:tasks.cephadm:Archiving logs... 2024-09-21T13:02:39.053 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-09-21_10:07:29-orch:cephadm-wip-guits-main-2024-09-21-0813-distro-default-smithi/7913913/remote/smithi114/log 2024-09-21T13:02:39.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T13:03:38.750 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-09-21_10:07:29-orch:cephadm-wip-guits-main-2024-09-21-0813-distro-default-smithi/7913913/remote/smithi183/log 2024-09-21T13:03:38.751 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T13:04:13.557 INFO:tasks.cephadm:Removing cluster... 2024-09-21T13:04:13.557 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e2ad975c-7812-11ef-baf6-efdc52797490 --force 2024-09-21T13:04:13.796 INFO:teuthology.orchestra.run.smithi114.stdout:Deleting cluster with fsid: e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T13:04:14.475 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e2ad975c-7812-11ef-baf6-efdc52797490 --force 2024-09-21T13:04:14.677 INFO:teuthology.orchestra.run.smithi183.stdout:Deleting cluster with fsid: e2ad975c-7812-11ef-baf6-efdc52797490 2024-09-21T13:04:15.243 INFO:tasks.cephadm:Removing cephadm ... 2024-09-21T13:04:15.243 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T13:04:15.261 DEBUG:teuthology.orchestra.run.smithi183:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T13:04:15.276 INFO:tasks.cephadm:Teardown complete 2024-09-21T13:04:15.276 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-21T13:04:15.285 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-21T13:04:15.286 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T13:04:15.303 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T13:04:15.414 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-21T13:04:15.423 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-21T13:04:15.423 DEBUG:teuthology.orchestra.run.smithi114:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T13:04:15.426 DEBUG:teuthology.orchestra.run.smithi183:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T13:04:15.439 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-09-21T13:04:15.440 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2024-09-21T13:04:15.469 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T13:04:15.469 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-09-21T13:04:15.469 INFO:teuthology.orchestra.run.smithi114.stdout:^* hv01.front.sepia.ceph.com 3 6 377 23 +174us[ +161us] +/- 57ms 2024-09-21T13:04:15.469 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 29 -816us[ -829us] +/- 56ms 2024-09-21T13:04:15.469 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 +630us[ +630us] +/- 68ms 2024-09-21T13:04:15.469 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T13:04:15.504 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T13:04:15.504 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2024-09-21T13:04:15.504 INFO:teuthology.orchestra.run.smithi183.stdout:^* hv01.front.sepia.ceph.com 3 6 377 25 +180us[ +170us] +/- 57ms 2024-09-21T13:04:15.504 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 25 -797us[ -797us] +/- 56ms 2024-09-21T13:04:15.504 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 +642us[ +642us] +/- 68ms 2024-09-21T13:04:15.504 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T13:04:15.505 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-21T13:04:15.531 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-21T13:04:15.532 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-21T13:04:15.554 DEBUG:teuthology.orchestra.run.smithi114:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T13:04:15.663 DEBUG:teuthology.orchestra.run.smithi183:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T13:04:15.731 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T13:04:15.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:04:15.759 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T13:04:15.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:04:15.785 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-21T13:04:15.794 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-21T13:04:15.908 INFO:teuthology.task.internal:Duration was 3420.585355 seconds 2024-09-21T13:04:15.908 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-21T13:04:16.163 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-21T13:04:16.163 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T13:04:16.166 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T13:04:16.217 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T13:04:16.234 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T13:04:16.586 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-21T13:04:16.586 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi114.front.sepia.ceph.com 2024-09-21T13:04:16.586 DEBUG:teuthology.orchestra.run.smithi114:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T13:04:16.626 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi183.front.sepia.ceph.com 2024-09-21T13:04:16.626 DEBUG:teuthology.orchestra.run.smithi183:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T13:04:16.658 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-21T13:04:16.658 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T13:04:16.669 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T13:04:16.725 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-21T13:04:16.725 DEBUG:teuthology.orchestra.run.smithi114:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T13:04:16.737 DEBUG:teuthology.orchestra.run.smithi183:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T13:04:18.243 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-21T13:04:18.254 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-21T13:04:18.254 DEBUG:teuthology.orchestra.run.smithi114:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T13:04:18.279 DEBUG:teuthology.orchestra.run.smithi183:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T13:04:18.305 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-21T13:04:18.371 DEBUG:teuthology.orchestra.run.smithi114:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T13:04:18.374 DEBUG:teuthology.orchestra.run.smithi183:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T13:04:18.397 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = core 2024-09-21T13:04:18.411 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = core 2024-09-21T13:04:18.575 DEBUG:teuthology.orchestra.run.smithi114:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T13:04:18.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:04:18.590 DEBUG:teuthology.orchestra.run.smithi183:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T13:04:18.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:04:18.631 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-21T13:04:18.647 INFO:teuthology.task.internal:Transferring archived files... 2024-09-21T13:04:18.647 DEBUG:teuthology.misc:Transferring archived files from smithi114:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-09-21_10:07:29-orch:cephadm-wip-guits-main-2024-09-21-0813-distro-default-smithi/7913913/remote/smithi114 2024-09-21T13:04:18.647 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T13:04:18.712 DEBUG:teuthology.misc:Transferring archived files from smithi183:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-09-21_10:07:29-orch:cephadm-wip-guits-main-2024-09-21-0813-distro-default-smithi/7913913/remote/smithi183 2024-09-21T13:04:18.713 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T13:04:18.752 INFO:teuthology.task.internal:Removing archive directory... 2024-09-21T13:04:18.752 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T13:04:18.754 DEBUG:teuthology.orchestra.run.smithi183:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T13:04:18.800 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-21T13:04:18.810 INFO:teuthology.task.internal:Not uploading archives. 2024-09-21T13:04:18.811 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-21T13:04:18.874 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-21T13:04:18.875 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T13:04:18.877 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T13:04:18.889 INFO:teuthology.orchestra.run.smithi114.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 13:04 /home/ubuntu/cephtest 2024-09-21T13:04:18.892 INFO:teuthology.orchestra.run.smithi183.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 13:04 /home/ubuntu/cephtest 2024-09-21T13:04:18.893 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-21T13:04:18.961 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-21T13:04:19.236 INFO:teuthology.run:Summary data: description: orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} duration: 3420.58535528183 flavor: default owner: scheduled_gabrioux@teuthology success: true 2024-09-21T13:04:19.236 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T13:04:19.320 INFO:teuthology.run:pass